2024-06-29T14:10:59.537 INFO:root:teuthology version: 0.0.1.dev329+g544fecb 2024-06-29T14:10:59.537 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_mchangir@teuthology --name mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi --archive /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} -- /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/orig.config.yaml 2024-06-29T14:10:59.584 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-29T14:10:59.664 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923 branch: wip-mchangir-qa-handle-correct-CephManager-instantiation-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} email: mchangir@redhat.com first_in_suite: false job_id: '7778923' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-qa-handle-correct-CephManager-instantiation-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: pid file: '' global: ms die on skipped message: false mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: cf146c2f9dac0bd5aed66008255af2113db872f7 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: cf146c2f9dac0bd5aed66008255af2113db872f7 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: qa-handle-correct-CephManager-instantiation sha1: 019bb938d4977149d100ab4c3d2c706147295470 owner: scheduled_mchangir@teuthology priority: 25 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1683 sha1: cf146c2f9dac0bd5aed66008255af2113db872f7 sleep_before_teardown: 0 suite: fs suite_branch: qa-handle-correct-CephManager-instantiation suite_path: /home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa suite_relpath: qa suite_repo: https://github.com/mchangir/ceph.git suite_sha1: 019bb938d4977149d100ab4c3d2c706147295470 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVzEtjeLZZBhzDv+SpBG7iSBNbAYaYmtwJ3f5TkpQkrkcW6XYhdOIU+sAX+Mx0iQQ6AlUmiY0XJleEBZr57khA= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9+d3Vrq9TnDck0IEV5e9v2Ww4f02ZHHfnR663z9oyc/t+Hd3gvLHgkwIfWuV13+10q5S4Bv4OTrxayVIYvZNc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 544fecbcd55f3d2b6f478823823ce40cbefef1d4 timestamp: 2024-06-29_14:01:10 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698093 2024-06-29T14:10:59.664 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa; will attempt to use it 2024-06-29T14:10:59.665 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks 2024-06-29T14:10:59.665 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-29T14:10:59.666 INFO:teuthology.task.internal:Checking packages... 2024-06-29T14:10:59.687 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cf146c2f9dac0bd5aed66008255af2113db872f7' 2024-06-29T14:10:59.687 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-29T14:10:59.687 INFO:teuthology.packaging:ref: None 2024-06-29T14:10:59.687 INFO:teuthology.packaging:tag: None 2024-06-29T14:10:59.687 INFO:teuthology.packaging:branch: wip-mchangir-qa-handle-correct-CephManager-instantiation-debug 2024-06-29T14:10:59.687 INFO:teuthology.packaging:sha1: cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:10:59.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-qa-handle-correct-CephManager-instantiation-debug 2024-06-29T14:10:59.834 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-4577.gcf146c2f 2024-06-29T14:10:59.836 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-29T14:10:59.841 INFO:teuthology.task.internal:no buildpackages task found 2024-06-29T14:10:59.842 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-29T14:10:59.879 INFO:teuthology.task.internal:Saving configuration 2024-06-29T14:10:59.897 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-29T14:10:59.905 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-29T14:10:59.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-29 14:05:16.150771', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVzEtjeLZZBhzDv+SpBG7iSBNbAYaYmtwJ3f5TkpQkrkcW6XYhdOIU+sAX+Mx0iQQ6AlUmiY0XJleEBZr57khA='} 2024-06-29T14:10:59.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-29 14:05:16.149256', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9+d3Vrq9TnDck0IEV5e9v2Ww4f02ZHHfnR663z9oyc/t+Hd3gvLHgkwIfWuV13+10q5S4Bv4OTrxayVIYvZNc='} 2024-06-29T14:10:59.955 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-29T14:10:59.961 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-06-29T14:10:59.961 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-06-29T14:10:59.961 INFO:teuthology.run_tasks:Running task console_log... 2024-06-29T14:11:00.038 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f50fa633ac0>, signals=[15]) 2024-06-29T14:11:00.038 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-29T14:11:00.044 INFO:teuthology.task.internal:Opening connections... 2024-06-29T14:11:00.044 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:11:00.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:11:00.122 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T14:11:00.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:11:00.198 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-29T14:11:00.205 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-06-29T14:11:00.224 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-06-29T14:11:00.224 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-06-29T14:11:00.282 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-06-29T14:11:00.282 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-06-29T14:11:00.282 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-06-29T14:11:00.282 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-06-29T14:11:00.282 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-06-29T14:11:00.282 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-06-29T14:11:00.282 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-29T14:11:00.282 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-06-29T14:11:00.283 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-06-29T14:11:00.283 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-29T14:11:00.283 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-06-29T14:11:00.283 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-29T14:11:00.283 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-29T14:11:00.283 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-29T14:11:00.283 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-06-29T14:11:00.302 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-06-29T14:11:00.320 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-06-29T14:11:00.320 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-06-29T14:11:00.377 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-06-29T14:11:00.377 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-06-29T14:11:00.377 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-06-29T14:11:00.377 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-06-29T14:11:00.377 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-06-29T14:11:00.377 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-06-29T14:11:00.377 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-29T14:11:00.377 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-06-29T14:11:00.378 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-06-29T14:11:00.378 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-29T14:11:00.378 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-06-29T14:11:00.378 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-29T14:11:00.378 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-29T14:11:00.378 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-29T14:11:00.378 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-06-29T14:11:00.403 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-29T14:11:00.412 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-29T14:11:00.487 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-29T14:11:00.487 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-06-29T14:11:00.489 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-06-29T14:11:00.505 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-29T14:11:00.512 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-29T14:11:00.513 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-06-29T14:11:00.547 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-06-29T14:11:00.646 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-29T14:11:00.700 INFO:teuthology.run_tasks:Running task kernel... 2024-06-29T14:11:00.714 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-06-29T14:11:00.714 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-06-29T14:11:00.714 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-06-29T14:11:00.715 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-29T14:11:00.715 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-29T14:11:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:11:00.731 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-06-29T14:11:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:11:00.733 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-06-29T14:11:00.788 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-437.el9.x86_64 2024-06-29T14:11:00.788 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-437.el9.x86_64 2024-06-29T14:11:00.788 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-06-29T14:11:00.790 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-437.el9.x86_64 2024-06-29T14:11:00.790 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-437.el9.x86_64 2024-06-29T14:11:00.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-06-29T14:11:04.596 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.1 MB 00:02 2024-06-29T14:11:04.621 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.1 MB 00:02 2024-06-29T14:11:06.347 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 35 MB/s | 20 MB 00:00 2024-06-29T14:11:06.347 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-06-29T14:11:11.339 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.4 MB 00:01 2024-06-29T14:11:12.306 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.4 MB 00:02 2024-06-29T14:11:12.840 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 17 kB 00:00 2024-06-29T14:11:14.010 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-06-29T14:11:14.231 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 17 kB 00:00 2024-06-29T14:11:15.436 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-06-29T14:11:19.341 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-06-29T14:11:20.883 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-06-29T14:11:21.422 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-29T14:11:21.423 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-29T14:11:21.515 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-29T14:11:21.526 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-29T14:11:21.526 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-06-29T14:11:21.526 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-29T14:11:21.526 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-29T14:11:21.526 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-467.el9 baseos 1.2 M 2024-06-29T14:11:21.526 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-29T14:11:21.526 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-467.el9 baseos 16 M 2024-06-29T14:11:21.526 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-467.el9 baseos 35 M 2024-06-29T14:11:21.527 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-467.el9 baseos 29 M 2024-06-29T14:11:21.527 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:11:21.527 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-29T14:11:21.527 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-29T14:11:21.527 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-06-29T14:11:21.527 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:11:21.527 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 82 M 2024-06-29T14:11:21.528 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 125 M 2024-06-29T14:11:21.528 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-29T14:11:22.001 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-467.el9.x86_64.rpm 3.1 MB/s | 1.2 MB 00:00 2024-06-29T14:11:23.056 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-29T14:11:23.056 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-29T14:11:23.138 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-06-29T14:11:23.147 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-06-29T14:11:23.147 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-06-29T14:11:23.147 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-06-29T14:11:23.147 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-06-29T14:11:23.147 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-467.el9 baseos 1.2 M 2024-06-29T14:11:23.147 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-06-29T14:11:23.148 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-467.el9 baseos 16 M 2024-06-29T14:11:23.148 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-467.el9 baseos 35 M 2024-06-29T14:11:23.148 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-467.el9 baseos 29 M 2024-06-29T14:11:23.148 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:11:23.148 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-06-29T14:11:23.148 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-06-29T14:11:23.148 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-06-29T14:11:23.148 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:11:23.149 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 82 M 2024-06-29T14:11:23.149 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 125 M 2024-06-29T14:11:23.149 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-06-29T14:11:23.476 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-467.el9.x86_64.rpm 4.8 MB/s | 1.2 MB 00:00 2024-06-29T14:11:27.684 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-467.el9.x86_64.rpm 2.7 MB/s | 16 MB 00:06 2024-06-29T14:11:28.526 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-467.el9.x86_64.rpm 3.1 MB/s | 16 MB 00:05 2024-06-29T14:11:29.192 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-core-5.14.0-467.el9.x86_6 4.1 MB/s | 29 MB 00:07 2024-06-29T14:11:29.284 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-core-5.14.0-467.el9.x86_6 5.0 MB/s | 29 MB 00:05 2024-06-29T14:11:35.709 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-5.14.0-467.el9.x86_64.rpm 2.8 MB/s | 35 MB 00:12 2024-06-29T14:11:35.709 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-06-29T14:11:35.709 INFO:teuthology.orchestra.run.smithi059.stdout:Total 6.5 MB/s | 82 MB 00:12 2024-06-29T14:11:36.459 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-06-29T14:11:36.917 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-06-29T14:11:36.917 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-06-29T14:11:37.458 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-5.14.0-467.el9.x86_64.rpm 2.2 MB/s | 35 MB 00:15 2024-06-29T14:11:37.458 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-29T14:11:37.458 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.2 MB/s | 82 MB 00:15 2024-06-29T14:11:37.927 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-06-29T14:11:37.928 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-06-29T14:11:38.178 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-29T14:11:38.528 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-29T14:11:38.528 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-29T14:11:39.495 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-29T14:11:39.495 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-29T14:11:40.493 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-06-29T14:11:41.159 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-467.el9.x86_64 1/4 2024-06-29T14:11:41.243 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:11:41.958 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-29T14:11:42.819 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-467.el9.x86_64 1/4 2024-06-29T14:11:42.916 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:11:44.062 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:11:44.211 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:11:45.610 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:11:45.758 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:11:49.363 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:11:49.419 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:11:50.565 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:11:50.673 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:11:54.597 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:11:55.504 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:25.287 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:25.327 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:25.662 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:25.662 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-467.el9.x86_64 1/4 2024-06-29T14:12:25.662 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:12:25.662 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:12:25.706 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:25.748 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:26.153 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-467.el9.x86_64 1/4 2024-06-29T14:12:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:12:26.154 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:12:26.470 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:26.470 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:12:26.470 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-29T14:12:26.471 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.471 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.471 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.471 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.471 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:12:26.471 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-29T14:12:26.855 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-06-29T14:12:26.926 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.927 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-29T14:12:26.927 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-467.el9.x86_64 2024-06-29T14:12:26.927 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-29T14:12:26.927 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-29T14:12:26.927 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-06-29T14:12:26.927 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-06-29T14:12:26.927 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-06-29T14:12:26.972 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:26.972 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:12:26.972 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-06-29T14:12:26.972 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.972 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.973 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.973 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.973 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:12:26.973 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-06-29T14:12:26.981 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-29T14:12:26.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-29T14:12:26.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-06-29T14:12:27.012 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-06-29T14:12:27.012 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-06-29T14:12:27.012 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-06-29T14:12:27.012 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-06-29T14:12:27.012 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-06-29T14:12:27.012 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:12:27.013 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-29T14:12:27.014 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-29T14:12:27.014 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-29T14:12:27.014 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-06-29T14:12:27.014 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:12:27.014 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-06-29T14:12:27.015 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-29T14:12:27.015 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-467.el9.x86_64 2024-06-29T14:12:27.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-29T14:12:27.358 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-06-29T14:12:27.451 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-29T14:12:27.451 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-29T14:12:27.452 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-467.el9.x86_64 2024-06-29T14:12:27.452 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-29T14:12:27.452 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-29T14:12:27.452 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-06-29T14:12:27.452 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-06-29T14:12:27.452 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-06-29T14:12:27.508 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-29T14:12:27.508 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-29T14:12:27.508 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-06-29T14:12:27.540 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-06-29T14:12:27.540 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-06-29T14:12:27.540 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2024-06-29T14:12:27.540 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 68.el9 2024-06-29T14:12:27.540 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-06-29T14:12:27.540 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-29T14:12:27.540 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2024-06-29T14:12:27.540 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8264254 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-29T14:12:27.541 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-06-29T14:12:27.542 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:12:27.542 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-06-29T14:12:27.542 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-29T14:12:27.543 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-467.el9.x86_64 2024-06-29T14:12:27.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-29T14:12:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-06-29T14:12:28.203 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-06-29T14:12:29.081 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-29T14:12:29.117 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-06-29T14:12:29.120 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-06-29T14:12:29.140 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.RaXHp27pg2 2024-06-29T14:12:29.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RaXHp27pg2 2024-06-29T14:12:29.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.RaXHp27pg2 2024-06-29T14:12:29.344 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.RaXHp27pg2 is 6KB 2024-06-29T14:12:29.392 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.RaXHp27pg2 2024-06-29T14:12:29.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-06-29T14:12:29.469 INFO:teuthology.orchestra.run.smithi153.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-29T14:12:29.470 INFO:teuthology.orchestra.run.smithi153.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-29T14:12:29.470 INFO:teuthology.orchestra.run.smithi153.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-29T14:12:29.470 INFO:teuthology.orchestra.run.smithi153.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64.conf 2024-06-29T14:12:29.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64 2024-06-29T14:12:29.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-06-29T14:12:29.666 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-29T14:12:29.700 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-06-29T14:12:29.702 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-06-29T14:12:29.719 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.0JhaUfVo9J 2024-06-29T14:12:29.719 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0JhaUfVo9J 2024-06-29T14:12:29.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.0JhaUfVo9J 2024-06-29T14:12:29.927 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.0JhaUfVo9J is 6KB 2024-06-29T14:12:29.977 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.0JhaUfVo9J 2024-06-29T14:12:29.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-06-29T14:12:30.054 INFO:teuthology.orchestra.run.smithi059.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-29T14:12:30.055 INFO:teuthology.orchestra.run.smithi059.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-29T14:12:30.055 INFO:teuthology.orchestra.run.smithi059.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-29T14:12:30.055 INFO:teuthology.orchestra.run.smithi059.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64.conf 2024-06-29T14:12:30.056 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64 2024-06-29T14:12:30.193 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-06-29T14:12:59.649 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-06-29T14:12:59.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-29T14:12:59.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:13:00.197 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-06-29T14:13:00.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-06-29T14:13:00.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:13:34.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-06-29T14:13:43.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-29T14:13:43.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:00.217 DEBUG:teuthology.orchestra.remote:timed out 2024-06-29T14:14:01.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-06-29T14:14:09.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-06-29T14:14:09.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:12.599 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-06-29T14:14:12.949 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-06-29T14:14:12.950 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-467.el9.x86_64"... 2024-06-29T14:14:12.950 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-06-29T14:14:12.965 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-467.el9.x86_64 2024-06-29T14:14:12.966 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-467.el9.x86_64 vs 5.14.0-467.el9.x86_64 2024-06-29T14:14:12.966 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-29T14:14:12.966 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-29T14:14:13.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-29T14:14:13.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:13.967 DEBUG:teuthology.parallel:result is None 2024-06-29T14:14:16.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-06-29T14:14:31.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-29T14:14:31.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:31.868 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-06-29T14:14:32.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-29T14:14:32.236 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-467.el9.x86_64"... 2024-06-29T14:14:32.236 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-06-29T14:14:32.253 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-467.el9.x86_64 2024-06-29T14:14:32.253 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-467.el9.x86_64 vs 5.14.0-467.el9.x86_64 2024-06-29T14:14:32.253 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-29T14:14:32.253 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-29T14:14:33.255 DEBUG:teuthology.parallel:result is None 2024-06-29T14:14:33.255 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-29T14:14:33.263 INFO:teuthology.task.internal:Creating test directory... 2024-06-29T14:14:33.263 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-29T14:14:33.267 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-29T14:14:33.286 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-29T14:14:33.305 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-29T14:14:33.369 INFO:teuthology.task.internal:Creating archive directory... 2024-06-29T14:14:33.370 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-29T14:14:33.372 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-29T14:14:33.406 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-29T14:14:33.412 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-29T14:14:33.412 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-29T14:14:33.442 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-29T14:14:33.511 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-29T14:14:33.529 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-29T14:14:33.564 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-29T14:14:33.589 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-29T14:14:33.592 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-29T14:14:33.599 INFO:teuthology.task.internal:Configuring sudo... 2024-06-29T14:14:33.599 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-29T14:14:33.601 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-29T14:14:33.630 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-29T14:14:33.668 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-29T14:14:33.668 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-29T14:14:33.671 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-29T14:14:33.689 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-29T14:14:33.774 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-29T14:14:33.824 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-29T14:14:33.908 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-29T14:14:33.950 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:14:33.951 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-29T14:14:34.017 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-29T14:14:34.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-29T14:14:34.114 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-29T14:14:34.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-29T14:14:34.242 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:14:34.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-29T14:14:34.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-06-29T14:14:34.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-06-29T14:14:34.367 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-29T14:14:34.389 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-29T14:14:34.873 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-29T14:14:34.882 INFO:teuthology.task.internal:Starting timer... 2024-06-29T14:14:34.883 INFO:teuthology.run_tasks:Running task pcp... 2024-06-29T14:14:34.924 INFO:teuthology.run_tasks:Running task selinux... 2024-06-29T14:14:34.954 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-29T14:14:34.955 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-06-29T14:14:35.026 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-06-29T14:14:35.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-06-29T14:14:35.089 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-06-29T14:14:35.091 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-29T14:14:35.091 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-06-29T14:14:35.111 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-06-29T14:14:35.111 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-06-29T14:14:35.136 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-06-29T14:14:35.136 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-06-29T14:14:35.136 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-29T14:14:35.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:14:35.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-29T14:14:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:14:35.209 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-29T14:14:35.209 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-06-29T14:14:35.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-06-29T14:14:35.291 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-29T14:14:35.301 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-29T14:14:35.312 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-29T14:14:35.313 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-29T14:18:59.190 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-06-29T14:18:59.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-06-29T14:18:59.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:18:59.277 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-06-29T14:18:59.365 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-06-29T14:18:59.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-29T14:18:59.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:18:59.466 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-06-29T14:18:59.546 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-29T14:18:59.546 INFO:teuthology.run_tasks:Running task clock... 2024-06-29T14:18:59.558 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-29T14:18:59.558 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-29T14:18:59.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-29T14:18:59.561 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-29T14:18:59.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-29T14:18:59.595 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-29T14:18:59.612 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-29T14:18:59.616 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-29T14:18:59.631 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-29T14:18:59.639 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-06-29T14:18:59.651 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-06-29T14:18:59.655 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-06-29T14:18:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-06-29T14:18:59.667 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-29T14:18:59.680 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-29T14:18:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-29T14:18:59.694 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-29T14:18:59.741 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-06-29T14:18:59.745 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-29T14:18:59.745 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-06-29T14:18:59.745 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:18:59.745 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:18:59.745 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:18:59.745 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:18:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-06-29T14:18:59.754 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-29T14:18:59.754 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-06-29T14:18:59.754 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:18:59.754 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:18:59.754 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:18:59.754 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:18:59.755 INFO:teuthology.run_tasks:Running task install... 2024-06-29T14:18:59.764 DEBUG:teuthology.task.install:project ceph 2024-06-29T14:18:59.765 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'}} 2024-06-29T14:18:59.765 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'} 2024-06-29T14:18:59.765 INFO:teuthology.task.install:Using flavor: default 2024-06-29T14:18:59.777 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-29T14:18:59.777 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-06-29T14:18:59.778 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7', 'tag': None, 'wait_for_package': False} 2024-06-29T14:18:59.779 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:18:59.782 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7', 'tag': None, 'wait_for_package': False} 2024-06-29T14:18:59.783 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:19:00.026 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/ 2024-06-29T14:19:00.026 INFO:teuthology.task.install.rpm:Package version is 19.0.0-4577.gcf146c2f 2024-06-29T14:19:00.029 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/ 2024-06-29T14:19:00.030 INFO:teuthology.task.install.rpm:Package version is 19.0.0-4577.gcf146c2f 2024-06-29T14:19:00.141 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-29T14:19:00.141 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:19:00.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-29T14:19:00.167 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-29T14:19:00.167 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cf146c2f9dac0bd5aed66008255af2113db872f7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-29T14:19:00.201 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-29T14:19:00.201 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:19:00.201 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-29T14:19:00.230 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-29T14:19:00.230 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cf146c2f9dac0bd5aed66008255af2113db872f7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-29T14:19:00.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-29T14:19:00.300 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-29T14:19:00.307 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-29T14:19:00.372 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-06-29T14:19:00.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-06-29T14:19:00.379 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-29T14:19:00.446 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-06-29T14:19:00.447 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-06-29T14:19:00.604 INFO:teuthology.orchestra.run.smithi153.stdout:45 files removed 2024-06-29T14:19:00.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-29T14:19:00.688 INFO:teuthology.orchestra.run.smithi059.stdout:45 files removed 2024-06-29T14:19:00.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-29T14:19:01.269 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 238 kB/s | 84 kB 00:00 2024-06-29T14:19:01.375 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 231 kB/s | 84 kB 00:00 2024-06-29T14:19:01.686 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 65 kB/s | 24 kB 00:00 2024-06-29T14:19:01.809 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 63 kB/s | 24 kB 00:00 2024-06-29T14:19:02.089 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-06-29T14:19:02.218 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-06-29T14:19:04.901 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.1 MB 00:02 2024-06-29T14:19:05.061 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.1 MB 00:02 2024-06-29T14:19:13.677 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-06-29T14:19:14.636 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 20 MB 00:08 2024-06-29T14:19:18.019 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.4 MB 00:00 2024-06-29T14:19:19.553 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 17 kB 00:00 2024-06-29T14:19:19.637 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.4 MB 00:01 2024-06-29T14:19:20.699 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-06-29T14:19:21.221 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 106 kB/s | 17 kB 00:00 2024-06-29T14:19:22.375 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-06-29T14:19:26.180 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-06-29T14:19:27.723 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-06-29T14:19:28.741 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-29T14:19:28.742 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-29T14:19:28.754 INFO:teuthology.orchestra.run.smithi059.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-29T14:19:28.755 INFO:teuthology.orchestra.run.smithi059.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-29T14:19:28.756 INFO:teuthology.orchestra.run.smithi059.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-29T14:19:28.757 INFO:teuthology.orchestra.run.smithi059.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-29T14:19:28.758 INFO:teuthology.orchestra.run.smithi059.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-29T14:19:28.761 INFO:teuthology.orchestra.run.smithi059.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-29T14:19:28.762 INFO:teuthology.orchestra.run.smithi059.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-29T14:19:28.831 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-06-29T14:19:28.854 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-06-29T14:19:28.854 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-06-29T14:19:28.854 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-06-29T14:19:28.854 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-06-29T14:19:28.854 INFO:teuthology.orchestra.run.smithi059.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-29T14:19:28.854 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 6.6 k 2024-06-29T14:19:28.854 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 5.4 M 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 866 k 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 147 k 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 1.5 M 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 143 k 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 14 M 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 7.4 M 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 49 k 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 10 M 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 43 M 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 274 k 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 764 k 2024-06-29T14:19:28.855 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-mirror x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 227 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-shell x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 27 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-top noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 18 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 32 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 759 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 127 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-06-29T14:19:28.856 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 161 k 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 321 k 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 301 k 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 99 k 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 86 k 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.0 M 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 169 k 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-devel x86_64 6.4.0-1.el9 appstream 95 k 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-06-29T14:19:28.857 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager x86_64 1:1.48.0-1.el9 baseos 2.3 M 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-libnm x86_64 1:1.48.0-1.el9 baseos 1.8 M 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-team x86_64 1:1.48.0-1.el9 baseos 41 k 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-tui x86_64 1:1.48.0-1.el9 baseos 248 k 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.3 M 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.1 M 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs x86_64 6.4.0-1.el9 baseos 1.1 M 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-06-29T14:19:28.858 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-29T14:19:28.859 INFO:teuthology.orchestra.run.smithi059.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-29T14:19:28.859 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-29T14:19:28.859 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 20 M 2024-06-29T14:19:28.859 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 34 k 2024-06-29T14:19:28.859 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 2.3 M 2024-06-29T14:19:28.859 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 277 k 2024-06-29T14:19:28.859 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 4.9 M 2024-06-29T14:19:28.859 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 18 M 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 16 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 25 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 158 k 2024-06-29T14:19:28.860 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 500 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 5.2 M 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-29T14:19:28.861 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 45 k 2024-06-29T14:19:28.862 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 136 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.30.0-1.el9 epel 213 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-29T14:19:28.863 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-29T14:19:28.864 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-29T14:19:28.865 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-29T14:19:28.866 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-06-29T14:19:28.867 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-29T14:19:28.868 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:19:28.868 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-06-29T14:19:28.868 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-06-29T14:19:28.868 INFO:teuthology.orchestra.run.smithi059.stdout:Install 148 Packages 2024-06-29T14:19:28.868 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 12 Packages 2024-06-29T14:19:28.868 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:19:28.869 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 231 M 2024-06-29T14:19:28.869 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-06-29T14:19:29.681 INFO:teuthology.orchestra.run.smithi059.stdout:(1/160): ceph-19.0.0-4577.gcf146c2f.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-06-29T14:19:30.199 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-29T14:19:30.200 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-29T14:19:30.211 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-29T14:19:30.212 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-29T14:19:30.213 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-29T14:19:30.214 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-29T14:19:30.215 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-29T14:19:30.218 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-29T14:19:30.219 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-29T14:19:30.283 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-29T14:19:30.306 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-06-29T14:19:30.306 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-29T14:19:30.306 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 6.6 k 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 5.4 M 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 866 k 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 147 k 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 1.5 M 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 143 k 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 14 M 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 7.4 M 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 49 k 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 10 M 2024-06-29T14:19:30.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 43 M 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 274 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 764 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 227 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 27 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 18 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 32 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 759 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 127 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-06-29T14:19:30.308 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 161 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 321 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 301 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 99 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 86 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.0 M 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 169 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.4.0-1.el9 appstream 95 k 2024-06-29T14:19:30.309 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager x86_64 1:1.48.0-1.el9 baseos 2.3 M 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm x86_64 1:1.48.0-1.el9 baseos 1.8 M 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team x86_64 1:1.48.0-1.el9 baseos 41 k 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui x86_64 1:1.48.0-1.el9 baseos 248 k 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.3 M 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.1 M 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-06-29T14:19:30.310 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs x86_64 6.4.0-1.el9 baseos 1.1 M 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 20 M 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 34 k 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 2.3 M 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 277 k 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 4.9 M 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 18 M 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 16 k 2024-06-29T14:19:30.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 25 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 158 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-29T14:19:30.312 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 500 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 5.2 M 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-29T14:19:30.313 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-29T14:19:30.314 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 45 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 136 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.30.0-1.el9 epel 213 k 2024-06-29T14:19:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-29T14:19:30.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-29T14:19:30.317 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-29T14:19:30.318 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-29T14:19:30.319 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout:Install 148 Packages 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 12 Packages 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 231 M 2024-06-29T14:19:30.320 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-29T14:19:30.423 INFO:teuthology.orchestra.run.smithi059.stdout:(2/160): ceph-fuse-19.0.0-4577.gcf146c2f.el9.x8 1.1 MB/s | 866 kB 00:00 2024-06-29T14:19:30.548 INFO:teuthology.orchestra.run.smithi059.stdout:(3/160): ceph-immutable-object-cache-19.0.0-457 1.1 MB/s | 147 kB 00:00 2024-06-29T14:19:30.682 INFO:teuthology.orchestra.run.smithi059.stdout:(4/160): ceph-base-19.0.0-4577.gcf146c2f.el9.x8 4.4 MB/s | 5.4 MB 00:01 2024-06-29T14:19:30.799 INFO:teuthology.orchestra.run.smithi059.stdout:(5/160): ceph-mgr-19.0.0-4577.gcf146c2f.el9.x86 13 MB/s | 1.5 MB 00:00 2024-06-29T14:19:30.966 INFO:teuthology.orchestra.run.smithi059.stdout:(6/160): ceph-mds-19.0.0-4577.gcf146c2f.el9.x86 5.4 MB/s | 2.3 MB 00:00 2024-06-29T14:19:31.066 INFO:teuthology.orchestra.run.smithi059.stdout:(7/160): ceph-mon-19.0.0-4577.gcf146c2f.el9.x86 18 MB/s | 4.9 MB 00:00 2024-06-29T14:19:31.373 INFO:teuthology.orchestra.run.smithi153.stdout:(1/160): ceph-19.0.0-4577.gcf146c2f.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-06-29T14:19:31.621 INFO:teuthology.orchestra.run.smithi059.stdout:(8/160): ceph-radosgw-19.0.0-4577.gcf146c2f.el9 18 MB/s | 10 MB 00:00 2024-06-29T14:19:31.692 INFO:teuthology.orchestra.run.smithi059.stdout:(9/160): ceph-selinux-19.0.0-4577.gcf146c2f.el9 358 kB/s | 25 kB 00:00 2024-06-29T14:19:31.749 INFO:teuthology.orchestra.run.smithi153.stdout:(2/160): ceph-fuse-19.0.0-4577.gcf146c2f.el9.x8 2.3 MB/s | 866 kB 00:00 2024-06-29T14:19:31.882 INFO:teuthology.orchestra.run.smithi153.stdout:(3/160): ceph-immutable-object-cache-19.0.0-457 1.1 MB/s | 147 kB 00:00 2024-06-29T14:19:32.466 INFO:teuthology.orchestra.run.smithi153.stdout:(4/160): ceph-mds-19.0.0-4577.gcf146c2f.el9.x86 3.9 MB/s | 2.3 MB 00:00 2024-06-29T14:19:32.684 INFO:teuthology.orchestra.run.smithi059.stdout:(10/160): ceph-osd-19.0.0-4577.gcf146c2f.el9.x8 10 MB/s | 18 MB 00:01 2024-06-29T14:19:32.691 INFO:teuthology.orchestra.run.smithi153.stdout:(5/160): ceph-base-19.0.0-4577.gcf146c2f.el9.x8 3.4 MB/s | 5.4 MB 00:01 2024-06-29T14:19:32.734 INFO:teuthology.orchestra.run.smithi059.stdout:(11/160): cephfs-mirror-19.0.0-4577.gcf146c2f.e 4.5 MB/s | 227 kB 00:00 2024-06-29T14:19:32.784 INFO:teuthology.orchestra.run.smithi059.stdout:(12/160): cephfs-shell-19.0.0-4577.gcf146c2f.el 540 kB/s | 27 kB 00:00 2024-06-29T14:19:32.826 INFO:teuthology.orchestra.run.smithi059.stdout:(13/160): libcephfs-devel-19.0.0-4577.gcf146c2f 759 kB/s | 32 kB 00:00 2024-06-29T14:19:32.909 INFO:teuthology.orchestra.run.smithi059.stdout:(14/160): libcephfs2-19.0.0-4577.gcf146c2f.el9. 8.9 MB/s | 759 kB 00:00 2024-06-29T14:19:32.960 INFO:teuthology.orchestra.run.smithi059.stdout:(15/160): libcephsqlite-19.0.0-4577.gcf146c2f.e 3.1 MB/s | 158 kB 00:00 2024-06-29T14:19:33.002 INFO:teuthology.orchestra.run.smithi059.stdout:(16/160): librados-devel-19.0.0-4577.gcf146c2f. 3.0 MB/s | 127 kB 00:00 2024-06-29T14:19:33.033 INFO:teuthology.orchestra.run.smithi153.stdout:(6/160): ceph-mon-19.0.0-4577.gcf146c2f.el9.x86 14 MB/s | 4.9 MB 00:00 2024-06-29T14:19:33.052 INFO:teuthology.orchestra.run.smithi059.stdout:(17/160): libradosstriper1-19.0.0-4577.gcf146c2 9.8 MB/s | 500 kB 00:00 2024-06-29T14:19:33.275 INFO:teuthology.orchestra.run.smithi153.stdout:(7/160): ceph-mgr-19.0.0-4577.gcf146c2f.el9.x86 1.9 MB/s | 1.5 MB 00:00 2024-06-29T14:19:33.403 INFO:teuthology.orchestra.run.smithi059.stdout:(18/160): librgw2-19.0.0-4577.gcf146c2f.el9.x86 15 MB/s | 5.2 MB 00:00 2024-06-29T14:19:33.461 INFO:teuthology.orchestra.run.smithi059.stdout:(19/160): python3-ceph-argparse-19.0.0-4577.gcf 778 kB/s | 45 kB 00:00 2024-06-29T14:19:33.511 INFO:teuthology.orchestra.run.smithi059.stdout:(20/160): python3-ceph-common-19.0.0-4577.gcf14 2.6 MB/s | 136 kB 00:00 2024-06-29T14:19:33.570 INFO:teuthology.orchestra.run.smithi059.stdout:(21/160): python3-cephfs-19.0.0-4577.gcf146c2f. 2.7 MB/s | 161 kB 00:00 2024-06-29T14:19:33.637 INFO:teuthology.orchestra.run.smithi059.stdout:(22/160): python3-rados-19.0.0-4577.gcf146c2f.e 4.7 MB/s | 321 kB 00:00 2024-06-29T14:19:33.696 INFO:teuthology.orchestra.run.smithi059.stdout:(23/160): python3-rbd-19.0.0-4577.gcf146c2f.el9 5.0 MB/s | 301 kB 00:00 2024-06-29T14:19:33.746 INFO:teuthology.orchestra.run.smithi059.stdout:(24/160): python3-rgw-19.0.0-4577.gcf146c2f.el9 1.9 MB/s | 99 kB 00:00 2024-06-29T14:19:33.775 INFO:teuthology.orchestra.run.smithi153.stdout:(8/160): ceph-osd-19.0.0-4577.gcf146c2f.el9.x86 24 MB/s | 18 MB 00:00 2024-06-29T14:19:33.788 INFO:teuthology.orchestra.run.smithi059.stdout:(25/160): rbd-fuse-19.0.0-4577.gcf146c2f.el9.x8 2.0 MB/s | 86 kB 00:00 2024-06-29T14:19:33.817 INFO:teuthology.orchestra.run.smithi153.stdout:(9/160): ceph-selinux-19.0.0-4577.gcf146c2f.el9 609 kB/s | 25 kB 00:00 2024-06-29T14:19:34.063 INFO:teuthology.orchestra.run.smithi059.stdout:(26/160): rbd-mirror-19.0.0-4577.gcf146c2f.el9. 11 MB/s | 3.0 MB 00:00 2024-06-29T14:19:34.147 INFO:teuthology.orchestra.run.smithi059.stdout:(27/160): rbd-nbd-19.0.0-4577.gcf146c2f.el9.x86 2.0 MB/s | 169 kB 00:00 2024-06-29T14:19:34.222 INFO:teuthology.orchestra.run.smithi059.stdout:(28/160): ceph-grafana-dashboards-19.0.0-4577.g 458 kB/s | 34 kB 00:00 2024-06-29T14:19:34.298 INFO:teuthology.orchestra.run.smithi059.stdout:(29/160): ceph-mgr-cephadm-19.0.0-4577.gcf146c2 1.9 MB/s | 143 kB 00:00 2024-06-29T14:19:35.390 INFO:teuthology.orchestra.run.smithi059.stdout:(30/160): ceph-mgr-dashboard-19.0.0-4577.gcf146 13 MB/s | 14 MB 00:01 2024-06-29T14:19:35.509 INFO:teuthology.orchestra.run.smithi153.stdout:(10/160): ceph-test-19.0.0-4577.gcf146c2f.el9.x 26 MB/s | 43 MB 00:01 2024-06-29T14:19:35.559 INFO:teuthology.orchestra.run.smithi153.stdout:(11/160): cephfs-mirror-19.0.0-4577.gcf146c2f.e 4.5 MB/s | 227 kB 00:00 2024-06-29T14:19:35.609 INFO:teuthology.orchestra.run.smithi153.stdout:(12/160): cephfs-shell-19.0.0-4577.gcf146c2f.el 544 kB/s | 27 kB 00:00 2024-06-29T14:19:35.660 INFO:teuthology.orchestra.run.smithi153.stdout:(13/160): libcephfs-devel-19.0.0-4577.gcf146c2f 635 kB/s | 32 kB 00:00 2024-06-29T14:19:35.723 INFO:teuthology.orchestra.run.smithi059.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 22 MB/s | 7.4 MB 00:00 2024-06-29T14:19:35.735 INFO:teuthology.orchestra.run.smithi153.stdout:(14/160): libcephfs2-19.0.0-4577.gcf146c2f.el9. 9.9 MB/s | 759 kB 00:00 2024-06-29T14:19:35.773 INFO:teuthology.orchestra.run.smithi059.stdout:(32/160): ceph-mgr-modules-core-19.0.0-4577.gcf 5.5 MB/s | 277 kB 00:00 2024-06-29T14:19:35.785 INFO:teuthology.orchestra.run.smithi153.stdout:(15/160): libcephsqlite-19.0.0-4577.gcf146c2f.e 3.1 MB/s | 158 kB 00:00 2024-06-29T14:19:35.824 INFO:teuthology.orchestra.run.smithi059.stdout:(33/160): ceph-mgr-rook-19.0.0-4577.gcf146c2f.e 980 kB/s | 49 kB 00:00 2024-06-29T14:19:35.835 INFO:teuthology.orchestra.run.smithi153.stdout:(16/160): librados-devel-19.0.0-4577.gcf146c2f. 2.5 MB/s | 127 kB 00:00 2024-06-29T14:19:35.858 INFO:teuthology.orchestra.run.smithi059.stdout:(34/160): ceph-prometheus-alerts-19.0.0-4577.gc 494 kB/s | 16 kB 00:00 2024-06-29T14:19:35.886 INFO:teuthology.orchestra.run.smithi153.stdout:(17/160): libradosstriper1-19.0.0-4577.gcf146c2 9.8 MB/s | 500 kB 00:00 2024-06-29T14:19:35.908 INFO:teuthology.orchestra.run.smithi059.stdout:(35/160): ceph-volume-19.0.0-4577.gcf146c2f.el9 5.4 MB/s | 274 kB 00:00 2024-06-29T14:19:36.033 INFO:teuthology.orchestra.run.smithi059.stdout:(36/160): cephadm-19.0.0-4577.gcf146c2f.el9.noa 6.0 MB/s | 764 kB 00:00 2024-06-29T14:19:36.067 INFO:teuthology.orchestra.run.smithi059.stdout:(37/160): cephfs-top-19.0.0-4577.gcf146c2f.el9. 546 kB/s | 18 kB 00:00 2024-06-29T14:19:36.103 INFO:teuthology.orchestra.run.smithi153.stdout:(18/160): librgw2-19.0.0-4577.gcf146c2f.el9.x86 24 MB/s | 5.2 MB 00:00 2024-06-29T14:19:36.145 INFO:teuthology.orchestra.run.smithi153.stdout:(19/160): python3-ceph-argparse-19.0.0-4577.gcf 1.1 MB/s | 45 kB 00:00 2024-06-29T14:19:36.195 INFO:teuthology.orchestra.run.smithi153.stdout:(20/160): python3-ceph-common-19.0.0-4577.gcf14 2.7 MB/s | 136 kB 00:00 2024-06-29T14:19:36.246 INFO:teuthology.orchestra.run.smithi153.stdout:(21/160): python3-cephfs-19.0.0-4577.gcf146c2f. 3.1 MB/s | 161 kB 00:00 2024-06-29T14:19:36.296 INFO:teuthology.orchestra.run.smithi153.stdout:(22/160): python3-rados-19.0.0-4577.gcf146c2f.e 6.3 MB/s | 321 kB 00:00 2024-06-29T14:19:36.300 INFO:teuthology.orchestra.run.smithi059.stdout:(38/160): ceph-common-19.0.0-4577.gcf146c2f.el9 2.9 MB/s | 20 MB 00:06 2024-06-29T14:19:36.317 INFO:teuthology.orchestra.run.smithi059.stdout:(39/160): fuse-2.9.9-15.el9.x86_64.rpm 321 kB/s | 80 kB 00:00 2024-06-29T14:19:36.346 INFO:teuthology.orchestra.run.smithi153.stdout:(23/160): python3-rbd-19.0.0-4577.gcf146c2f.el9 5.9 MB/s | 301 kB 00:00 2024-06-29T14:19:36.422 INFO:teuthology.orchestra.run.smithi153.stdout:(24/160): python3-rgw-19.0.0-4577.gcf146c2f.el9 1.3 MB/s | 99 kB 00:00 2024-06-29T14:19:36.468 INFO:teuthology.orchestra.run.smithi059.stdout:(40/160): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-06-29T14:19:36.472 INFO:teuthology.orchestra.run.smithi153.stdout:(25/160): rbd-fuse-19.0.0-4577.gcf146c2f.el9.x8 1.7 MB/s | 86 kB 00:00 2024-06-29T14:19:36.593 INFO:teuthology.orchestra.run.smithi059.stdout:(41/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-06-29T14:19:36.597 INFO:teuthology.orchestra.run.smithi153.stdout:(26/160): rbd-mirror-19.0.0-4577.gcf146c2f.el9. 24 MB/s | 3.0 MB 00:00 2024-06-29T14:19:36.643 INFO:teuthology.orchestra.run.smithi059.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-06-29T14:19:36.647 INFO:teuthology.orchestra.run.smithi153.stdout:(27/160): rbd-nbd-19.0.0-4577.gcf146c2f.el9.x86 3.3 MB/s | 169 kB 00:00 2024-06-29T14:19:36.694 INFO:teuthology.orchestra.run.smithi059.stdout:(43/160): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-06-29T14:19:36.698 INFO:teuthology.orchestra.run.smithi153.stdout:(28/160): ceph-grafana-dashboards-19.0.0-4577.g 684 kB/s | 34 kB 00:00 2024-06-29T14:19:36.744 INFO:teuthology.orchestra.run.smithi059.stdout:(44/160): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-06-29T14:19:36.806 INFO:teuthology.orchestra.run.smithi153.stdout:(29/160): ceph-radosgw-19.0.0-4577.gcf146c2f.el 2.9 MB/s | 10 MB 00:03 2024-06-29T14:19:36.832 INFO:teuthology.orchestra.run.smithi153.stdout:(30/160): ceph-mgr-cephadm-19.0.0-4577.gcf146c2 1.0 MB/s | 143 kB 00:00 2024-06-29T14:19:37.065 INFO:teuthology.orchestra.run.smithi153.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 32 MB/s | 7.4 MB 00:00 2024-06-29T14:19:37.116 INFO:teuthology.orchestra.run.smithi153.stdout:(32/160): ceph-mgr-modules-core-19.0.0-4577.gcf 5.5 MB/s | 277 kB 00:00 2024-06-29T14:19:37.136 INFO:teuthology.orchestra.run.smithi059.stdout:(45/160): ceph-test-19.0.0-4577.gcf146c2f.el9.x 8.0 MB/s | 43 MB 00:05 2024-06-29T14:19:37.157 INFO:teuthology.orchestra.run.smithi153.stdout:(33/160): ceph-mgr-rook-19.0.0-4577.gcf146c2f.e 1.2 MB/s | 49 kB 00:00 2024-06-29T14:19:37.161 INFO:teuthology.orchestra.run.smithi059.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 299 kB/s | 125 kB 00:00 2024-06-29T14:19:37.203 INFO:teuthology.orchestra.run.smithi059.stdout:(47/160): python3-cryptography-36.0.1-4.el9.x86 2.0 MB/s | 1.2 MB 00:00 2024-06-29T14:19:37.208 INFO:teuthology.orchestra.run.smithi153.stdout:(34/160): ceph-prometheus-alerts-19.0.0-4577.gc 328 kB/s | 16 kB 00:00 2024-06-29T14:19:37.258 INFO:teuthology.orchestra.run.smithi153.stdout:(35/160): ceph-volume-19.0.0-4577.gcf146c2f.el9 5.4 MB/s | 274 kB 00:00 2024-06-29T14:19:37.317 INFO:teuthology.orchestra.run.smithi153.stdout:(36/160): cephadm-19.0.0-4577.gcf146c2f.el9.noa 13 MB/s | 764 kB 00:00 2024-06-29T14:19:37.354 INFO:teuthology.orchestra.run.smithi059.stdout:(48/160): python3-urllib3-1.26.5-5.el9.noarch.r 989 kB/s | 215 kB 00:00 2024-06-29T14:19:37.367 INFO:teuthology.orchestra.run.smithi153.stdout:(37/160): cephfs-top-19.0.0-4577.gcf146c2f.el9. 366 kB/s | 18 kB 00:00 2024-06-29T14:19:37.546 INFO:teuthology.orchestra.run.smithi059.stdout:(49/160): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 689 kB 00:00 2024-06-29T14:19:37.567 INFO:teuthology.orchestra.run.smithi153.stdout:(38/160): fuse-2.9.9-15.el9.x86_64.rpm 402 kB/s | 80 kB 00:00 2024-06-29T14:19:37.596 INFO:teuthology.orchestra.run.smithi059.stdout:(50/160): boost-program-options-1.75.0-8.el9.x8 442 kB/s | 107 kB 00:00 2024-06-29T14:19:37.621 INFO:teuthology.orchestra.run.smithi059.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 289 kB/s | 22 kB 00:00 2024-06-29T14:19:37.646 INFO:teuthology.orchestra.run.smithi059.stdout:(52/160): bison-3.7.4-5.el9.x86_64.rpm 2.1 MB/s | 943 kB 00:00 2024-06-29T14:19:37.688 INFO:teuthology.orchestra.run.smithi059.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 488 kB/s | 33 kB 00:00 2024-06-29T14:19:37.763 INFO:teuthology.orchestra.run.smithi059.stdout:(54/160): flexiblas-openblas-openmp-3.0.4-8.el9 243 kB/s | 18 kB 00:00 2024-06-29T14:19:37.856 INFO:teuthology.orchestra.run.smithi059.stdout:(55/160): flex-2.6.4-9.el9.x86_64.rpm 1.2 MB/s | 313 kB 00:00 2024-06-29T14:19:37.876 INFO:teuthology.orchestra.run.smithi153.stdout:(39/160): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-06-29T14:19:37.881 INFO:teuthology.orchestra.run.smithi059.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-06-29T14:19:37.923 INFO:teuthology.orchestra.run.smithi059.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-06-29T14:19:37.951 INFO:teuthology.orchestra.run.smithi153.stdout:(40/160): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-06-29T14:19:37.956 INFO:teuthology.orchestra.run.smithi059.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-06-29T14:19:38.018 INFO:teuthology.orchestra.run.smithi153.stdout:(41/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-06-29T14:19:38.073 INFO:teuthology.orchestra.run.smithi059.stdout:(59/160): libtool-2.4.6-46.el9.x86_64.rpm 4.9 MB/s | 581 kB 00:00 2024-06-29T14:19:38.099 INFO:teuthology.orchestra.run.smithi059.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-06-29T14:19:38.141 INFO:teuthology.orchestra.run.smithi059.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-06-29T14:19:38.174 INFO:teuthology.orchestra.run.smithi059.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-06-29T14:19:38.208 INFO:teuthology.orchestra.run.smithi059.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-06-29T14:19:38.210 INFO:teuthology.orchestra.run.smithi153.stdout:(42/160): python3-cryptography-36.0.1-4.el9.x86 6.4 MB/s | 1.2 MB 00:00 2024-06-29T14:19:38.259 INFO:teuthology.orchestra.run.smithi059.stdout:(64/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.9 MB/s | 3.0 MB 00:00 2024-06-29T14:19:38.275 INFO:teuthology.orchestra.run.smithi059.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 525 kB/s | 35 kB 00:00 2024-06-29T14:19:38.301 INFO:teuthology.orchestra.run.smithi059.stdout:(66/160): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-06-29T14:19:38.385 INFO:teuthology.orchestra.run.smithi153.stdout:(43/160): ceph-common-19.0.0-4577.gcf146c2f.el9 2.7 MB/s | 20 MB 00:07 2024-06-29T14:19:38.411 INFO:teuthology.orchestra.run.smithi153.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 531 kB/s | 106 kB 00:00 2024-06-29T14:19:38.503 INFO:teuthology.orchestra.run.smithi153.stdout:(45/160): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-06-29T14:19:38.586 INFO:teuthology.orchestra.run.smithi153.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 674 kB/s | 135 kB 00:00 2024-06-29T14:19:38.653 INFO:teuthology.orchestra.run.smithi153.stdout:(47/160): python3-urllib3-1.26.5-5.el9.noarch.r 1.4 MB/s | 215 kB 00:00 2024-06-29T14:19:38.684 INFO:teuthology.orchestra.run.smithi059.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 4.1 MB/s | 1.6 MB 00:00 2024-06-29T14:19:38.829 INFO:teuthology.orchestra.run.smithi153.stdout:(48/160): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 689 kB 00:00 2024-06-29T14:19:38.879 INFO:teuthology.orchestra.run.smithi153.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-06-29T14:19:38.921 INFO:teuthology.orchestra.run.smithi153.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 523 kB/s | 22 kB 00:00 2024-06-29T14:19:38.954 INFO:teuthology.orchestra.run.smithi153.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-06-29T14:19:38.980 INFO:teuthology.orchestra.run.smithi153.stdout:(52/160): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-06-29T14:19:38.997 INFO:teuthology.orchestra.run.smithi153.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 779 kB/s | 33 kB 00:00 2024-06-29T14:19:39.047 INFO:teuthology.orchestra.run.smithi153.stdout:(54/160): flexiblas-openblas-openmp-3.0.4-8.el9 366 kB/s | 18 kB 00:00 2024-06-29T14:19:39.097 INFO:teuthology.orchestra.run.smithi153.stdout:(55/160): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-06-29T14:19:39.148 INFO:teuthology.orchestra.run.smithi153.stdout:(56/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-06-29T14:19:39.173 INFO:teuthology.orchestra.run.smithi153.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-06-29T14:19:39.198 INFO:teuthology.orchestra.run.smithi153.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-06-29T14:19:39.209 INFO:teuthology.orchestra.run.smithi059.stdout:(68/160): postgresql-contrib-13.14-1.el9.x86_64 1.7 MB/s | 911 kB 00:00 2024-06-29T14:19:39.248 INFO:teuthology.orchestra.run.smithi153.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-06-29T14:19:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 6.8 MB/s | 581 kB 00:00 2024-06-29T14:19:39.307 INFO:teuthology.orchestra.run.smithi153.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-06-29T14:19:39.333 INFO:teuthology.orchestra.run.smithi153.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-06-29T14:19:39.334 INFO:teuthology.orchestra.run.smithi059.stdout:(69/160): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-06-29T14:19:39.366 INFO:teuthology.orchestra.run.smithi153.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-06-29T14:19:39.392 INFO:teuthology.orchestra.run.smithi153.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-06-29T14:19:39.417 INFO:teuthology.orchestra.run.smithi153.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 35 kB 00:00 2024-06-29T14:19:39.550 INFO:teuthology.orchestra.run.smithi153.stdout:(66/160): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-06-29T14:19:39.659 INFO:teuthology.orchestra.run.smithi153.stdout:(67/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-06-29T14:19:39.717 INFO:teuthology.orchestra.run.smithi153.stdout:(68/160): postgresql-13.14-1.el9.x86_64.rpm 9.6 MB/s | 1.6 MB 00:00 2024-06-29T14:19:39.760 INFO:teuthology.orchestra.run.smithi153.stdout:(69/160): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-06-29T14:19:39.785 INFO:teuthology.orchestra.run.smithi153.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-06-29T14:19:39.885 INFO:teuthology.orchestra.run.smithi153.stdout:(71/160): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-06-29T14:19:39.961 INFO:teuthology.orchestra.run.smithi153.stdout:(72/160): postgresql-server-13.14-1.el9.x86_64. 29 MB/s | 5.8 MB 00:00 2024-06-29T14:19:39.977 INFO:teuthology.orchestra.run.smithi153.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 950 kB/s | 87 kB 00:00 2024-06-29T14:19:40.010 INFO:teuthology.orchestra.run.smithi059.stdout:(70/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 2.6 MB/s | 4.4 MB 00:01 2024-06-29T14:19:40.153 INFO:teuthology.orchestra.run.smithi153.stdout:(74/160): ceph-mgr-dashboard-19.0.0-4577.gcf146 4.2 MB/s | 14 MB 00:03 2024-06-29T14:19:40.178 INFO:teuthology.orchestra.run.smithi153.stdout:(75/160): python3-colorama-0.4.6-3.el9.noarch.r 306 kB/s | 61 kB 00:00 2024-06-29T14:19:40.261 INFO:teuthology.orchestra.run.smithi153.stdout:(76/160): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-06-29T14:19:40.287 INFO:teuthology.orchestra.run.smithi153.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-06-29T14:19:40.312 INFO:teuthology.orchestra.run.smithi153.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-06-29T14:19:40.346 INFO:teuthology.orchestra.run.smithi153.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 325 kB/s | 19 kB 00:00 2024-06-29T14:19:40.371 INFO:teuthology.orchestra.run.smithi153.stdout:(80/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-06-29T14:19:40.388 INFO:teuthology.orchestra.run.smithi153.stdout:(81/160): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-06-29T14:19:40.413 INFO:teuthology.orchestra.run.smithi153.stdout:(82/160): python3-markupsafe-1.1.1-12.el9.x86_6 519 kB/s | 35 kB 00:00 2024-06-29T14:19:40.464 INFO:teuthology.orchestra.run.smithi153.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-06-29T14:19:40.489 INFO:teuthology.orchestra.run.smithi153.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-06-29T14:19:40.531 INFO:teuthology.orchestra.run.smithi153.stdout:(85/160): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-06-29T14:19:40.556 INFO:teuthology.orchestra.run.smithi153.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-06-29T14:19:40.593 INFO:teuthology.orchestra.run.smithi059.stdout:(71/160): protobuf-3.14.0-13.el9.x86_64.rpm 1.7 MB/s | 1.0 MB 00:00 2024-06-29T14:19:40.631 INFO:teuthology.orchestra.run.smithi153.stdout:(87/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-06-29T14:19:40.656 INFO:teuthology.orchestra.run.smithi153.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-06-29T14:19:40.682 INFO:teuthology.orchestra.run.smithi153.stdout:(89/160): python3-protobuf-3.14.0-13.el9.noarch 1.7 MB/s | 269 kB 00:00 2024-06-29T14:19:40.685 INFO:teuthology.orchestra.run.smithi059.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 957 kB/s | 87 kB 00:00 2024-06-29T14:19:40.707 INFO:teuthology.orchestra.run.smithi153.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-29T14:19:40.732 INFO:teuthology.orchestra.run.smithi153.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-06-29T14:19:40.757 INFO:teuthology.orchestra.run.smithi153.stdout:(92/160): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 279 kB 00:00 2024-06-29T14:19:40.808 INFO:teuthology.orchestra.run.smithi153.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-06-29T14:19:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 587 kB/s | 44 kB 00:00 2024-06-29T14:19:40.867 INFO:teuthology.orchestra.run.smithi153.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-06-29T14:19:40.892 INFO:teuthology.orchestra.run.smithi153.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 610 kB/s | 36 kB 00:00 2024-06-29T14:19:40.917 INFO:teuthology.orchestra.run.smithi153.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-06-29T14:19:40.942 INFO:teuthology.orchestra.run.smithi153.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-06-29T14:19:40.976 INFO:teuthology.orchestra.run.smithi153.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 985 kB/s | 58 kB 00:00 2024-06-29T14:19:41.001 INFO:teuthology.orchestra.run.smithi153.stdout:(100/160): xfsprogs-devel-6.4.0-1.el9.x86_64.rp 1.6 MB/s | 95 kB 00:00 2024-06-29T14:19:41.035 INFO:teuthology.orchestra.run.smithi153.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-06-29T14:19:41.077 INFO:teuthology.orchestra.run.smithi153.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 593 kB/s | 45 kB 00:00 2024-06-29T14:19:41.094 INFO:teuthology.orchestra.run.smithi059.stdout:(73/160): openblas-openmp-0.3.21-2.el9.x86_64.r 1.7 MB/s | 4.7 MB 00:02 2024-06-29T14:19:41.161 INFO:teuthology.orchestra.run.smithi059.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 921 kB/s | 61 kB 00:00 2024-06-29T14:19:41.286 INFO:teuthology.orchestra.run.smithi153.stdout:(103/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-06-29T14:19:41.311 INFO:teuthology.orchestra.run.smithi059.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-06-29T14:19:41.352 INFO:teuthology.orchestra.run.smithi153.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 8.1 MB/s | 548 kB 00:00 2024-06-29T14:19:41.386 INFO:teuthology.orchestra.run.smithi153.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-06-29T14:19:41.411 INFO:teuthology.orchestra.run.smithi153.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 784 kB/s | 19 kB 00:00 2024-06-29T14:19:41.512 INFO:teuthology.orchestra.run.smithi153.stdout:(107/160): libarrow-9.0.0-11.el9.x86_64.rpm 44 MB/s | 4.4 MB 00:00 2024-06-29T14:19:41.512 INFO:teuthology.orchestra.run.smithi059.stdout:(76/160): python3-babel-2.9.1-2.el9.noarch.rpm 7.2 MB/s | 6.0 MB 00:00 2024-06-29T14:19:41.537 INFO:teuthology.orchestra.run.smithi059.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 1.1 MB/s | 248 kB 00:00 2024-06-29T14:19:41.570 INFO:teuthology.orchestra.run.smithi059.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 439 kB/s | 26 kB 00:00 2024-06-29T14:19:41.587 INFO:teuthology.orchestra.run.smithi153.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 988 kB/s | 26 kB 00:00 2024-06-29T14:19:41.595 INFO:teuthology.orchestra.run.smithi059.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-06-29T14:19:41.612 INFO:teuthology.orchestra.run.smithi153.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-29T14:19:41.629 INFO:teuthology.orchestra.run.smithi059.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-06-29T14:19:41.637 INFO:teuthology.orchestra.run.smithi153.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.8 MB/s | 67 kB 00:00 2024-06-29T14:19:41.655 INFO:teuthology.orchestra.run.smithi059.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 592 kB/s | 35 kB 00:00 2024-06-29T14:19:41.663 INFO:teuthology.orchestra.run.smithi153.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 6.1 MB/s | 151 kB 00:00 2024-06-29T14:19:41.696 INFO:teuthology.orchestra.run.smithi153.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 25 MB/s | 839 kB 00:00 2024-06-29T14:19:41.730 INFO:teuthology.orchestra.run.smithi153.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-06-29T14:19:41.755 INFO:teuthology.orchestra.run.smithi153.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-29T14:19:41.780 INFO:teuthology.orchestra.run.smithi153.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-06-29T14:19:41.788 INFO:teuthology.orchestra.run.smithi059.stdout:(82/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-06-29T14:19:41.806 INFO:teuthology.orchestra.run.smithi153.stdout:(116/160): python3-certifi-2021.10.8-2.el9.noar 599 kB/s | 15 kB 00:00 2024-06-29T14:19:41.831 INFO:teuthology.orchestra.run.smithi153.stdout:(117/160): python3-cheroot-8.6.0-4.el9.noarch.r 6.7 MB/s | 172 kB 00:00 2024-06-29T14:19:41.847 INFO:teuthology.orchestra.run.smithi059.stdout:(83/160): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-06-29T14:19:41.856 INFO:teuthology.orchestra.run.smithi153.stdout:(118/160): python3-cherrypy-18.6.1-2.el9.noarch 14 MB/s | 358 kB 00:00 2024-06-29T14:19:41.881 INFO:teuthology.orchestra.run.smithi059.stdout:(84/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 250 kB 00:00 2024-06-29T14:19:41.882 INFO:teuthology.orchestra.run.smithi153.stdout:(119/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 12 MB/s | 303 kB 00:00 2024-06-29T14:19:41.897 INFO:teuthology.orchestra.run.smithi059.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-06-29T14:19:41.907 INFO:teuthology.orchestra.run.smithi153.stdout:(120/160): lua-devel-5.4.4-4.el9.x86_64.rpm 26 kB/s | 22 kB 00:00 2024-06-29T14:19:41.939 INFO:teuthology.orchestra.run.smithi059.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 723 kB/s | 42 kB 00:00 2024-06-29T14:19:41.957 INFO:teuthology.orchestra.run.smithi153.stdout:(121/160): python3-google-auth-2.30.0-1.el9.noa 2.8 MB/s | 213 kB 00:00 2024-06-29T14:19:41.965 INFO:teuthology.orchestra.run.smithi059.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 3.9 MB/s | 269 kB 00:00 2024-06-29T14:19:41.983 INFO:teuthology.orchestra.run.smithi153.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 6.2 MB/s | 144 kB 00:00 2024-06-29T14:19:42.008 INFO:teuthology.orchestra.run.smithi153.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 427 kB/s | 11 kB 00:00 2024-06-29T14:19:42.023 INFO:teuthology.orchestra.run.smithi059.stdout:(88/160): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-06-29T14:19:42.033 INFO:teuthology.orchestra.run.smithi153.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 707 kB/s | 18 kB 00:00 2024-06-29T14:19:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 927 kB/s | 23 kB 00:00 2024-06-29T14:19:42.073 INFO:teuthology.orchestra.run.smithi059.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-06-29T14:19:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 772 kB/s | 19 kB 00:00 2024-06-29T14:19:42.099 INFO:teuthology.orchestra.run.smithi059.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-06-29T14:19:42.109 INFO:teuthology.orchestra.run.smithi153.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 792 kB/s | 20 kB 00:00 2024-06-29T14:19:42.124 INFO:teuthology.orchestra.run.smithi059.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-06-29T14:19:42.151 INFO:teuthology.orchestra.run.smithi153.stdout:(128/160): python3-kubernetes-26.1.0-2.el9.noar 24 MB/s | 1.0 MB 00:00 2024-06-29T14:19:42.168 INFO:teuthology.orchestra.run.smithi153.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 2.7 MB/s | 46 kB 00:00 2024-06-29T14:19:42.200 INFO:teuthology.orchestra.run.smithi059.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 555 kB/s | 42 kB 00:00 2024-06-29T14:19:42.201 INFO:teuthology.orchestra.run.smithi153.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-06-29T14:19:42.218 INFO:teuthology.orchestra.run.smithi153.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 3.4 MB/s | 58 kB 00:00 2024-06-29T14:19:42.244 INFO:teuthology.orchestra.run.smithi153.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 11 MB/s | 272 kB 00:00 2024-06-29T14:19:42.258 INFO:teuthology.orchestra.run.smithi059.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 759 kB/s | 44 kB 00:00 2024-06-29T14:19:42.261 INFO:teuthology.orchestra.run.smithi153.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 993 kB/s | 16 kB 00:00 2024-06-29T14:19:42.286 INFO:teuthology.orchestra.run.smithi153.stdout:(134/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-06-29T14:19:42.303 INFO:teuthology.orchestra.run.smithi153.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.6 MB/s | 28 kB 00:00 2024-06-29T14:19:42.345 INFO:teuthology.orchestra.run.smithi153.stdout:(136/160): python3-grpcio-1.46.7-10.el9.x86_64. 4.7 MB/s | 2.0 MB 00:00 2024-06-29T14:19:42.370 INFO:teuthology.orchestra.run.smithi153.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 459 kB/s | 31 kB 00:00 2024-06-29T14:19:42.404 INFO:teuthology.orchestra.run.smithi153.stdout:(138/160): protobuf-compiler-3.14.0-13.el9.x86_ 651 kB/s | 863 kB 00:01 2024-06-29T14:19:42.429 INFO:teuthology.orchestra.run.smithi153.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-06-29T14:19:42.480 INFO:teuthology.orchestra.run.smithi153.stdout:(140/160): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-06-29T14:19:42.500 INFO:teuthology.orchestra.run.smithi059.stdout:(94/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-06-29T14:19:42.505 INFO:teuthology.orchestra.run.smithi153.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 995 kB/s | 51 kB 00:00 2024-06-29T14:19:42.525 INFO:teuthology.orchestra.run.smithi059.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 827 kB/s | 221 kB 00:00 2024-06-29T14:19:42.530 INFO:teuthology.orchestra.run.smithi153.stdout:(142/160): python3-tempora-5.0.0-2.el9.noarch.r 284 kB/s | 36 kB 00:00 2024-06-29T14:19:42.551 INFO:teuthology.orchestra.run.smithi059.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 715 kB/s | 36 kB 00:00 2024-06-29T14:19:42.555 INFO:teuthology.orchestra.run.smithi153.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-06-29T14:19:42.581 INFO:teuthology.orchestra.run.smithi153.stdout:(144/160): python3-webob-1.8.7-6.el9.noarch.rpm 2.3 MB/s | 230 kB 00:00 2024-06-29T14:19:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 806 kB/s | 47 kB 00:00 2024-06-29T14:19:42.606 INFO:teuthology.orchestra.run.smithi153.stdout:(145/160): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-06-29T14:19:42.609 INFO:teuthology.orchestra.run.smithi059.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-06-29T14:19:42.631 INFO:teuthology.orchestra.run.smithi153.stdout:(146/160): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-06-29T14:19:42.635 INFO:teuthology.orchestra.run.smithi059.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-06-29T14:19:42.658 INFO:teuthology.orchestra.run.smithi153.stdout:(147/160): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-06-29T14:19:42.660 INFO:teuthology.orchestra.run.smithi059.stdout:(100/160): xfsprogs-devel-6.4.0-1.el9.x86_64.rp 1.8 MB/s | 95 kB 00:00 2024-06-29T14:19:42.707 INFO:teuthology.orchestra.run.smithi153.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-06-29T14:19:42.710 INFO:teuthology.orchestra.run.smithi059.stdout:(101/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-06-29T14:19:42.736 INFO:teuthology.orchestra.run.smithi059.stdout:(102/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 634 kB/s | 64 kB 00:00 2024-06-29T14:19:42.844 INFO:teuthology.orchestra.run.smithi059.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-06-29T14:19:42.915 INFO:teuthology.orchestra.run.smithi153.stdout:(149/160): librbd1-19.0.0-4577.gcf146c2f.el9.x8 12 MB/s | 3.1 MB 00:00 2024-06-29T14:19:42.919 INFO:teuthology.orchestra.run.smithi059.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 7.2 MB/s | 548 kB 00:00 2024-06-29T14:19:42.945 INFO:teuthology.orchestra.run.smithi059.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-06-29T14:19:42.970 INFO:teuthology.orchestra.run.smithi059.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 774 kB/s | 19 kB 00:00 2024-06-29T14:19:43.070 INFO:teuthology.orchestra.run.smithi059.stdout:(107/160): libarrow-9.0.0-11.el9.x86_64.rpm 44 MB/s | 4.4 MB 00:00 2024-06-29T14:19:43.096 INFO:teuthology.orchestra.run.smithi059.stdout:(108/160): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-06-29T14:19:43.138 INFO:teuthology.orchestra.run.smithi059.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 388 kB/s | 26 kB 00:00 2024-06-29T14:19:43.213 INFO:teuthology.orchestra.run.smithi059.stdout:(110/160): postgresql-server-13.14-1.el9.x86_64 1.5 MB/s | 5.8 MB 00:03 2024-06-29T14:19:43.241 INFO:teuthology.orchestra.run.smithi153.stdout:(150/160): NetworkManager-1.48.0-1.el9.x86_64.r 4.3 MB/s | 2.3 MB 00:00 2024-06-29T14:19:43.263 INFO:teuthology.orchestra.run.smithi059.stdout:(111/160): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2024-06-29T14:19:43.297 INFO:teuthology.orchestra.run.smithi059.stdout:(112/160): libunwind-1.6.2-1.el9.x86_64.rpm 430 kB/s | 67 kB 00:00 2024-06-29T14:19:43.307 INFO:teuthology.orchestra.run.smithi153.stdout:(151/160): NetworkManager-team-1.48.0-1.el9.x86 620 kB/s | 41 kB 00:00 2024-06-29T14:19:43.322 INFO:teuthology.orchestra.run.smithi059.stdout:(113/160): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-06-29T14:19:43.348 INFO:teuthology.orchestra.run.smithi059.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-29T14:19:43.373 INFO:teuthology.orchestra.run.smithi059.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-06-29T14:19:43.391 INFO:teuthology.orchestra.run.smithi153.stdout:(152/160): librados2-19.0.0-4577.gcf146c2f.el9. 4.4 MB/s | 3.3 MB 00:00 2024-06-29T14:19:43.398 INFO:teuthology.orchestra.run.smithi059.stdout:(116/160): python3-certifi-2021.10.8-2.el9.noar 602 kB/s | 15 kB 00:00 2024-06-29T14:19:43.423 INFO:teuthology.orchestra.run.smithi059.stdout:(117/160): python3-asyncssh-2.13.2-5.el9.noarch 4.2 MB/s | 548 kB 00:00 2024-06-29T14:19:43.433 INFO:teuthology.orchestra.run.smithi153.stdout:(153/160): NetworkManager-libnm-1.48.0-1.el9.x8 3.5 MB/s | 1.8 MB 00:00 2024-06-29T14:19:43.449 INFO:teuthology.orchestra.run.smithi059.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-06-29T14:19:43.483 INFO:teuthology.orchestra.run.smithi059.stdout:(119/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.8 MB/s | 839 kB 00:00 2024-06-29T14:19:43.491 INFO:teuthology.orchestra.run.smithi153.stdout:(154/160): NetworkManager-tui-1.48.0-1.el9.x86_ 1.3 MB/s | 248 kB 00:00 2024-06-29T14:19:43.508 INFO:teuthology.orchestra.run.smithi059.stdout:(120/160): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-06-29T14:19:43.517 INFO:teuthology.orchestra.run.smithi153.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 2.4 MB/s | 207 kB 00:00 2024-06-29T14:19:43.533 INFO:teuthology.orchestra.run.smithi059.stdout:(121/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-06-29T14:19:43.542 INFO:teuthology.orchestra.run.smithi153.stdout:(156/160): elfutils-debuginfod-client-0.191-4.e 243 kB/s | 37 kB 00:00 2024-06-29T14:19:43.558 INFO:teuthology.orchestra.run.smithi059.stdout:(122/160): python3-google-auth-2.30.0-1.el9.noa 2.8 MB/s | 213 kB 00:00 2024-06-29T14:19:43.583 INFO:teuthology.orchestra.run.smithi059.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-06-29T14:19:43.609 INFO:teuthology.orchestra.run.smithi059.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 707 kB/s | 18 kB 00:00 2024-06-29T14:19:43.634 INFO:teuthology.orchestra.run.smithi059.stdout:(125/160): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-06-29T14:19:43.659 INFO:teuthology.orchestra.run.smithi059.stdout:(126/160): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-06-29T14:19:43.685 INFO:teuthology.orchestra.run.smithi059.stdout:(127/160): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-06-29T14:19:43.692 INFO:teuthology.orchestra.run.smithi153.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 1.3 MB/s | 260 kB 00:00 2024-06-29T14:19:43.718 INFO:teuthology.orchestra.run.smithi059.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-06-29T14:19:43.744 INFO:teuthology.orchestra.run.smithi059.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-29T14:19:43.826 INFO:teuthology.orchestra.run.smithi153.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 4.4 MB/s | 1.4 MB 00:00 2024-06-29T14:19:43.856 INFO:teuthology.orchestra.run.smithi059.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 706 kB/s | 79 kB 00:00 2024-06-29T14:19:43.903 INFO:teuthology.orchestra.run.smithi059.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.2 MB/s | 58 kB 00:00 2024-06-29T14:19:43.936 INFO:teuthology.orchestra.run.smithi059.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-06-29T14:19:43.962 INFO:teuthology.orchestra.run.smithi059.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 655 kB/s | 16 kB 00:00 2024-06-29T14:19:43.987 INFO:teuthology.orchestra.run.smithi059.stdout:(134/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.6 MB/s | 90 kB 00:00 2024-06-29T14:19:44.012 INFO:teuthology.orchestra.run.smithi059.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-06-29T14:19:44.038 INFO:teuthology.orchestra.run.smithi059.stdout:(136/160): python3-repoze-lru-0.7-16.el9.noarch 1.2 MB/s | 31 kB 00:00 2024-06-29T14:19:44.071 INFO:teuthology.orchestra.run.smithi059.stdout:(137/160): python3-kubernetes-26.1.0-2.el9.noar 2.6 MB/s | 1.0 MB 00:00 2024-06-29T14:19:44.097 INFO:teuthology.orchestra.run.smithi059.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-06-29T14:19:44.122 INFO:teuthology.orchestra.run.smithi059.stdout:(139/160): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-06-29T14:19:44.147 INFO:teuthology.orchestra.run.smithi059.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-06-29T14:19:44.172 INFO:teuthology.orchestra.run.smithi059.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 2.0 MB/s | 51 kB 00:00 2024-06-29T14:19:44.198 INFO:teuthology.orchestra.run.smithi059.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-06-29T14:19:44.223 INFO:teuthology.orchestra.run.smithi059.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-06-29T14:19:44.240 INFO:teuthology.orchestra.run.smithi059.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-06-29T14:19:44.276 INFO:teuthology.orchestra.run.smithi153.stdout:(159/160): xfsprogs-6.4.0-1.el9.x86_64.rpm 1.9 MB/s | 1.1 MB 00:00 2024-06-29T14:19:44.282 INFO:teuthology.orchestra.run.smithi059.stdout:(145/160): python3-grpcio-1.46.7-10.el9.x86_64. 2.6 MB/s | 2.0 MB 00:00 2024-06-29T14:19:44.307 INFO:teuthology.orchestra.run.smithi059.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-06-29T14:19:44.332 INFO:teuthology.orchestra.run.smithi059.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-06-29T14:19:44.383 INFO:teuthology.orchestra.run.smithi059.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-06-29T14:19:44.410 INFO:teuthology.orchestra.run.smithi153.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 2.8 MB/s | 2.4 MB 00:00 2024-06-29T14:19:44.415 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-29T14:19:44.415 INFO:teuthology.orchestra.run.smithi153.stdout:Total 16 MB/s | 231 MB 00:14 2024-06-29T14:19:44.700 INFO:teuthology.orchestra.run.smithi059.stdout:(149/160): librbd1-19.0.0-4577.gcf146c2f.el9.x8 8.4 MB/s | 3.1 MB 00:00 2024-06-29T14:19:44.750 INFO:teuthology.orchestra.run.smithi059.stdout:(150/160): NetworkManager-1.48.0-1.el9.x86_64.r 6.8 MB/s | 2.3 MB 00:00 2024-06-29T14:19:44.808 INFO:teuthology.orchestra.run.smithi059.stdout:(151/160): librados2-19.0.0-4577.gcf146c2f.el9. 6.7 MB/s | 3.3 MB 00:00 2024-06-29T14:19:44.834 INFO:teuthology.orchestra.run.smithi059.stdout:(152/160): NetworkManager-team-1.48.0-1.el9.x86 494 kB/s | 41 kB 00:00 2024-06-29T14:19:44.884 INFO:teuthology.orchestra.run.smithi059.stdout:(153/160): elfutils-debuginfod-client-0.191-4.e 734 kB/s | 37 kB 00:00 2024-06-29T14:19:44.934 INFO:teuthology.orchestra.run.smithi059.stdout:(154/160): elfutils-libelf-0.191-4.el9.x86_64.r 4.0 MB/s | 207 kB 00:00 2024-06-29T14:19:44.976 INFO:teuthology.orchestra.run.smithi059.stdout:(155/160): NetworkManager-tui-1.48.0-1.el9.x86_ 1.4 MB/s | 248 kB 00:00 2024-06-29T14:19:44.993 INFO:teuthology.orchestra.run.smithi059.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 4.3 MB/s | 260 kB 00:00 2024-06-29T14:19:45.102 INFO:teuthology.orchestra.run.smithi059.stdout:(157/160): NetworkManager-libnm-1.48.0-1.el9.x8 4.5 MB/s | 1.8 MB 00:00 2024-06-29T14:19:45.219 INFO:teuthology.orchestra.run.smithi059.stdout:(158/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-06-29T14:19:45.261 INFO:teuthology.orchestra.run.smithi059.stdout:(159/160): openssl-3.2.2-2.el9.x86_64.rpm 4.8 MB/s | 1.4 MB 00:00 2024-06-29T14:19:45.528 INFO:teuthology.orchestra.run.smithi059.stdout:(160/160): xfsprogs-6.4.0-1.el9.x86_64.rpm 2.5 MB/s | 1.1 MB 00:00 2024-06-29T14:19:45.534 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-06-29T14:19:45.534 INFO:teuthology.orchestra.run.smithi059.stdout:Total 14 MB/s | 231 MB 00:16 2024-06-29T14:19:45.678 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-29T14:19:45.881 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-29T14:19:45.881 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-29T14:19:46.861 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-06-29T14:19:47.087 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-06-29T14:19:47.088 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-06-29T14:19:48.058 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-29T14:19:48.058 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-29T14:19:49.379 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-06-29T14:19:49.380 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-06-29T14:19:54.658 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-29T14:19:54.756 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-06-29T14:19:54.803 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-06-29T14:19:54.857 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-06-29T14:19:54.897 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-06-29T14:19:54.923 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-06-29T14:19:54.975 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-06-29T14:19:55.318 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-06-29T14:19:55.440 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:19:55.566 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:19:55.592 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:19:55.683 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:19:55.712 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 10/172 2024-06-29T14:19:55.764 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-06-29T14:19:55.938 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-06-29T14:19:55.986 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-06-29T14:19:56.014 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:19:56.110 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-06-29T14:19:56.119 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:19:56.148 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 15/172 2024-06-29T14:19:56.190 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-06-29T14:19:56.194 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 15/172 2024-06-29T14:19:56.215 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-06-29T14:19:56.234 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-06-29T14:19:56.238 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-06-29T14:19:56.299 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-06-29T14:19:56.333 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-06-29T14:19:56.358 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-06-29T14:19:56.405 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-06-29T14:19:56.442 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-06-29T14:19:56.549 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-06-29T14:19:56.760 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-06-29T14:19:56.785 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsprogs-6.4.0-1.el9.x86_64 18/172 2024-06-29T14:19:56.948 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-libnm-1:1.48.0-1.el9.x86_64 19/172 2024-06-29T14:19:56.961 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:19:57.000 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.0-1.el9.x86_64 19/172 2024-06-29T14:19:57.102 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:19:57.198 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:19:57.210 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:19:57.235 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:19:57.329 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:19:57.357 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 10/172 2024-06-29T14:19:57.396 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-06-29T14:19:57.406 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:19:57.517 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-06-29T14:19:57.548 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-06-29T14:19:57.582 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-06-29T14:19:57.586 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-06-29T14:19:57.631 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-06-29T14:19:57.657 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:19:57.663 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-06-29T14:19:57.707 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-06-29T14:19:57.740 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-06-29T14:19:57.748 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:19:57.775 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 15/172 2024-06-29T14:19:57.783 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-06-29T14:19:57.816 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-06-29T14:19:57.830 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 15/172 2024-06-29T14:19:57.851 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-06-29T14:19:57.852 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-06-29T14:19:57.878 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-06-29T14:19:57.922 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-06-29T14:19:57.975 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/172 2024-06-29T14:19:58.005 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-06-29T14:19:58.054 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-06-29T14:19:58.064 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-06-29T14:19:58.141 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-06-29T14:19:58.162 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-06-29T14:19:58.176 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9. 35/172 2024-06-29T14:19:58.220 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el 36/172 2024-06-29T14:19:58.248 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_6 37/172 2024-06-29T14:19:58.318 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-06-29T14:19:58.371 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-06-29T14:19:58.423 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : xfsprogs-6.4.0-1.el9.x86_64 18/172 2024-06-29T14:19:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-06-29T14:19:58.550 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-libnm-1:1.48.0-1.el9.x86_64 19/172 2024-06-29T14:19:58.586 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-06-29T14:19:58.619 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.0-1.el9.x86_64 19/172 2024-06-29T14:19:58.640 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-06-29T14:19:58.706 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:19:58.787 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:19:58.996 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:19:59.105 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-06-29T14:19:59.134 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-06-29T14:19:59.169 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-06-29T14:19:59.170 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-06-29T14:19:59.244 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-06-29T14:19:59.252 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-06-29T14:19:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-06-29T14:19:59.295 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-06-29T14:19:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-06-29T14:19:59.335 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-06-29T14:19:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-06-29T14:19:59.378 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-06-29T14:19:59.398 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_6 48/172 2024-06-29T14:19:59.411 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-06-29T14:19:59.438 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-06-29T14:19:59.440 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-06-29T14:19:59.510 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-06-29T14:19:59.570 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/172 2024-06-29T14:19:59.593 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-06-29T14:19:59.600 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-06-29T14:19:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-06-29T14:19:59.648 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-06-29T14:19:59.730 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-06-29T14:19:59.734 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-06-29T14:19:59.762 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9. 35/172 2024-06-29T14:19:59.791 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-06-29T14:19:59.806 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el 36/172 2024-06-29T14:19:59.834 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_6 37/172 2024-06-29T14:19:59.863 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-06-29T14:19:59.894 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-06-29T14:19:59.907 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-06-29T14:19:59.940 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-06-29T14:19:59.949 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-06-29T14:19:59.986 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-06-29T14:20:00.022 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-06-29T14:20:00.027 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-06-29T14:20:00.066 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-06-29T14:20:00.108 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-06-29T14:20:00.181 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-06-29T14:20:00.236 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-06-29T14:20:00.243 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-06-29T14:20:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-06-29T14:20:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 63/172 2024-06-29T14:20:00.431 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-06-29T14:20:00.468 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.30.0-1.el9.noarch 65/172 2024-06-29T14:20:00.564 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-06-29T14:20:00.735 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-06-29T14:20:00.795 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-06-29T14:20:00.817 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-06-29T14:20:00.883 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-06-29T14:20:00.918 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-06-29T14:20:00.932 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-06-29T14:20:00.955 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-06-29T14:20:01.003 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-06-29T14:20:01.041 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_6 48/172 2024-06-29T14:20:01.071 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-06-29T14:20:01.084 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-06-29T14:20:01.116 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-06-29T14:20:01.143 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-06-29T14:20:01.178 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-06-29T14:20:01.217 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-06-29T14:20:01.238 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-06-29T14:20:01.257 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-06-29T14:20:01.286 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-06-29T14:20:01.287 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-06-29T14:20:01.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-06-29T14:20:01.355 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-06-29T14:20:01.381 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-06-29T14:20:01.396 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-06-29T14:20:01.430 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-06-29T14:20:01.443 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-06-29T14:20:01.516 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-06-29T14:20:01.517 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-06-29T14:20:01.554 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-06-29T14:20:01.569 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-06-29T14:20:01.590 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-06-29T14:20:01.600 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-06-29T14:20:01.638 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-06-29T14:20:01.673 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-06-29T14:20:01.702 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-06-29T14:20:01.717 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-06-29T14:20:01.751 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-06-29T14:20:01.759 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-06-29T14:20:01.902 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-06-29T14:20:01.944 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-06-29T14:20:01.986 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 63/172 2024-06-29T14:20:02.087 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-06-29T14:20:02.118 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.30.0-1.el9.noarch 65/172 2024-06-29T14:20:02.159 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-06-29T14:20:02.203 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-06-29T14:20:02.227 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-06-29T14:20:02.312 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-06-29T14:20:02.442 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-06-29T14:20:02.468 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-06-29T14:20:02.476 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-06-29T14:20:02.504 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-06-29T14:20:02.516 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-06-29T14:20:02.547 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-06-29T14:20:02.549 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-06-29T14:20:02.587 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-06-29T14:20:02.626 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-06-29T14:20:02.647 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-06-29T14:20:02.672 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-06-29T14:20:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-06-29T14:20:02.707 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-06-29T14:20:02.740 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-06-29T14:20:02.741 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-06-29T14:20:02.782 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-06-29T14:20:02.816 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-06-29T14:20:02.827 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-06-29T14:20:02.860 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-06-29T14:20:02.906 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-06-29T14:20:02.964 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-06-29T14:20:02.994 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-06-29T14:20:03.032 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-06-29T14:20:03.067 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-06-29T14:20:03.103 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-06-29T14:20:03.146 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-06-29T14:20:03.163 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-06-29T14:20:03.792 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-06-29T14:20:03.914 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-06-29T14:20:03.931 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-06-29T14:20:03.992 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-06-29T14:20:04.026 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-06-29T14:20:04.131 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-06-29T14:20:04.191 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-06-29T14:20:04.236 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-06-29T14:20:04.335 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-06-29T14:20:04.370 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-06-29T14:20:04.553 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-06-29T14:20:04.576 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-06-29T14:20:04.656 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-06-29T14:20:05.154 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-06-29T14:20:05.237 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-06-29T14:20:05.265 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-06-29T14:20:05.298 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-06-29T14:20:05.344 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-06-29T14:20:05.454 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-06-29T14:20:05.616 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-06-29T14:20:05.669 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 104/172 2024-06-29T14:20:05.751 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 104/172 2024-06-29T14:20:05.783 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 105/172 2024-06-29T14:20:05.826 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-06-29T14:20:06.313 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-06-29T14:20:06.364 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 108/172 2024-06-29T14:20:06.419 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 108/172 2024-06-29T14:20:06.449 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 109/172 2024-06-29T14:20:06.777 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-06-29T14:20:06.854 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-06-29T14:20:06.875 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-06-29T14:20:06.899 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-06-29T14:20:07.233 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-06-29T14:20:07.270 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 104/172 2024-06-29T14:20:07.361 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 104/172 2024-06-29T14:20:07.391 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 105/172 2024-06-29T14:20:07.429 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-06-29T14:20:07.966 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-06-29T14:20:07.997 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 108/172 2024-06-29T14:20:08.053 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 108/172 2024-06-29T14:20:08.082 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 109/172 2024-06-29T14:20:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:08.713 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:09.274 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:09.307 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 111/172 2024-06-29T14:20:09.406 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 111/172 2024-06-29T14:20:09.406 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-29T14:20:09.406 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-29T14:20:09.407 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:09.433 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 112/172 2024-06-29T14:20:10.466 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:10.505 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:11.110 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:11.150 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 111/172 2024-06-29T14:20:11.241 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 111/172 2024-06-29T14:20:11.242 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-29T14:20:11.242 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-29T14:20:11.242 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:11.276 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 112/172 2024-06-29T14:20:20.721 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 112/172 2024-06-29T14:20:20.721 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-06-29T14:20:20.721 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-06-29T14:20:20.721 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-06-29T14:20:20.721 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-06-29T14:20:20.721 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-06-29T14:20:20.722 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-06-29T14:20:20.722 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-06-29T14:20:20.722 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:20.907 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 113/172 2024-06-29T14:20:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 113/172 2024-06-29T14:20:20.925 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:20.925 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-29T14:20:20.925 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:22.493 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 114/172 2024-06-29T14:20:22.520 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 114/172 2024-06-29T14:20:22.520 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:22.520 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-29T14:20:22.520 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-29T14:20:22.520 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-29T14:20:22.520 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:22.743 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 115/172 2024-06-29T14:20:22.767 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 115/172 2024-06-29T14:20:22.767 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:22.767 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-29T14:20:22.767 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-29T14:20:22.767 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-29T14:20:22.767 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:23.158 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 112/172 2024-06-29T14:20:23.159 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-06-29T14:20:23.159 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-06-29T14:20:23.159 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-06-29T14:20:23.159 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-06-29T14:20:23.159 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-06-29T14:20:23.160 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-06-29T14:20:23.160 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-06-29T14:20:23.160 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:23.227 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 116/172 2024-06-29T14:20:23.251 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 116/172 2024-06-29T14:20:23.251 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:23.251 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-29T14:20:23.251 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-29T14:20:23.251 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-29T14:20:23.251 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:23.356 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 113/172 2024-06-29T14:20:23.357 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-06-29T14:20:23.372 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 113/172 2024-06-29T14:20:23.373 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:23.373 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-29T14:20:23.373 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:23.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-06-29T14:20:23.470 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-06-29T14:20:23.656 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-06-29T14:20:23.703 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-06-29T14:20:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-06-29T14:20:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-06-29T14:20:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-06-29T14:20:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-06-29T14:20:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-06-29T14:20:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 127/172 2024-06-29T14:20:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 128/172 2024-06-29T14:20:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 128/172 2024-06-29T14:20:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.e 129/172 2024-06-29T14:20:24.682 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f. 130/172 2024-06-29T14:20:24.712 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 131/172 2024-06-29T14:20:25.001 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 114/172 2024-06-29T14:20:25.029 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 114/172 2024-06-29T14:20:25.030 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:25.030 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-29T14:20:25.030 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-29T14:20:25.030 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-29T14:20:25.030 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:25.259 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 115/172 2024-06-29T14:20:25.284 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 115/172 2024-06-29T14:20:25.284 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:25.285 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-29T14:20:25.285 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-29T14:20:25.285 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-29T14:20:25.285 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:25.769 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 116/172 2024-06-29T14:20:25.793 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 116/172 2024-06-29T14:20:25.794 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:25.794 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-29T14:20:25.794 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-29T14:20:25.794 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-29T14:20:25.794 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:25.908 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-06-29T14:20:25.987 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-06-29T14:20:26.054 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-06-29T14:20:26.250 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-06-29T14:20:26.303 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-06-29T14:20:26.384 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-06-29T14:20:26.482 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-06-29T14:20:26.623 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-06-29T14:20:26.669 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-06-29T14:20:26.702 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-06-29T14:20:26.728 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-15.el9.x86_64 127/172 2024-06-29T14:20:27.191 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 128/172 2024-06-29T14:20:27.226 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 128/172 2024-06-29T14:20:27.263 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.e 129/172 2024-06-29T14:20:27.326 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f. 130/172 2024-06-29T14:20:27.353 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 131/172 2024-06-29T14:20:27.834 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 131/172 2024-06-29T14:20:27.900 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 132/172 2024-06-29T14:20:28.868 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 132/172 2024-06-29T14:20:28.908 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 133/172 2024-06-29T14:20:29.018 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 133/172 2024-06-29T14:20:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el 134/172 2024-06-29T14:20:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 135/172 2024-06-29T14:20:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 135/172 2024-06-29T14:20:29.235 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:29.235 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-29T14:20:29.235 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-29T14:20:29.235 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-29T14:20:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:29.278 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 136/172 2024-06-29T14:20:29.290 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 136/172 2024-06-29T14:20:29.387 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 137/172 2024-06-29T14:20:29.439 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 138/172 2024-06-29T14:20:30.381 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-06-29T14:20:30.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 140/172 2024-06-29T14:20:30.437 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 140/172 2024-06-29T14:20:30.437 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:30.437 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-29T14:20:30.437 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-29T14:20:30.437 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-29T14:20:30.438 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:30.481 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-4577.gcf146 141/172 2024-06-29T14:20:30.503 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-4577.gcf146 141/172 2024-06-29T14:20:30.503 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:30.504 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-29T14:20:30.504 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:30.548 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 142/172 2024-06-29T14:20:30.571 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 142/172 2024-06-29T14:20:30.571 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:30.571 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-29T14:20:30.571 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-29T14:20:30.572 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-29T14:20:30.572 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:30.673 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 131/172 2024-06-29T14:20:30.734 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 132/172 2024-06-29T14:20:30.882 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 143/172 2024-06-29T14:20:30.905 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 143/172 2024-06-29T14:20:30.905 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:30.906 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-29T14:20:30.906 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-29T14:20:30.906 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-29T14:20:30.906 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:31.750 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 132/172 2024-06-29T14:20:31.776 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 133/172 2024-06-29T14:20:31.895 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 133/172 2024-06-29T14:20:32.066 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el 134/172 2024-06-29T14:20:32.094 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 135/172 2024-06-29T14:20:32.120 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 135/172 2024-06-29T14:20:32.120 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:32.120 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-29T14:20:32.120 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-29T14:20:32.120 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-29T14:20:32.120 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:32.170 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 136/172 2024-06-29T14:20:32.182 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 136/172 2024-06-29T14:20:32.284 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 137/172 2024-06-29T14:20:32.343 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 138/172 2024-06-29T14:20:33.333 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-06-29T14:20:33.372 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 140/172 2024-06-29T14:20:33.398 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 140/172 2024-06-29T14:20:33.398 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:33.398 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-29T14:20:33.398 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-29T14:20:33.398 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-29T14:20:33.398 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:33.439 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-4577.gcf146 141/172 2024-06-29T14:20:33.464 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-4577.gcf146 141/172 2024-06-29T14:20:33.464 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:33.464 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-29T14:20:33.464 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:33.515 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 142/172 2024-06-29T14:20:33.539 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 142/172 2024-06-29T14:20:33.540 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:33.540 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-29T14:20:33.540 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-29T14:20:33.540 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-29T14:20:33.540 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:33.874 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 143/172 2024-06-29T14:20:33.900 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 143/172 2024-06-29T14:20:33.901 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:33.901 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-29T14:20:33.901 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-29T14:20:33.901 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-29T14:20:33.901 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:35.169 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 144/172 2024-06-29T14:20:35.210 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 145/172 2024-06-29T14:20:35.250 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 146/172 2024-06-29T14:20:35.300 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.4.0-1.el9.x86_64 147/172 2024-06-29T14:20:35.327 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-06-29T14:20:35.518 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 149/172 2024-06-29T14:20:35.564 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-06-29T14:20:35.597 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_ 151/172 2024-06-29T14:20:35.629 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-06-29T14:20:35.658 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-team-1:1.48.0-1.el9.x86_64 153/172 2024-06-29T14:20:35.705 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-tui-1:1.48.0-1.el9.x86_64 154/172 2024-06-29T14:20:36.315 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:20:36.361 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:20:36.380 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:20:37.202 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 156/172 2024-06-29T14:20:37.299 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-06-29T14:20:37.340 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-06-29T14:20:37.424 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-06-29T14:20:37.460 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-06-29T14:20:37.486 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-06-29T14:20:37.510 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-06-29T14:20:37.537 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-06-29T14:20:37.537 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-06-29T14:20:37.621 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-06-29T14:20:37.654 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-06-29T14:20:37.654 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-06-29T14:20:37.738 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-06-29T14:20:37.764 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-06-29T14:20:37.770 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:20:37.770 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:20:37.966 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:20:37.966 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-06-29T14:20:38.050 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-06-29T14:20:38.074 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-06-29T14:20:38.106 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-06-29T14:20:38.207 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:20:38.436 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 144/172 2024-06-29T14:20:38.495 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 145/172 2024-06-29T14:20:38.535 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 146/172 2024-06-29T14:20:38.576 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xfsprogs-devel-6.4.0-1.el9.x86_64 147/172 2024-06-29T14:20:38.602 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-06-29T14:20:38.794 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 149/172 2024-06-29T14:20:38.832 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-06-29T14:20:38.864 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_ 151/172 2024-06-29T14:20:38.905 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-06-29T14:20:38.934 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-team-1:1.48.0-1.el9.x86_64 153/172 2024-06-29T14:20:38.972 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-tui-1:1.48.0-1.el9.x86_64 154/172 2024-06-29T14:20:39.571 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:20:39.611 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:20:39.633 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:20:40.549 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 156/172 2024-06-29T14:20:40.632 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-06-29T14:20:40.674 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-06-29T14:20:40.759 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-06-29T14:20:40.793 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-06-29T14:20:40.819 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-06-29T14:20:40.843 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-06-29T14:20:40.870 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-06-29T14:20:40.870 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-06-29T14:20:40.971 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-06-29T14:20:41.004 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-06-29T14:20:41.004 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-06-29T14:20:41.080 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-06-29T14:20:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-06-29T14:20:41.111 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:20:41.112 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:20:41.299 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:20:41.299 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-06-29T14:20:41.383 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-06-29T14:20:41.424 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-06-29T14:20:41.475 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-06-29T14:20:41.553 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:20:44.311 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:20:44.311 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 1/172 2024-06-29T14:20:44.311 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 2/172 2024-06-29T14:20:44.311 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 3/172 2024-06-29T14:20:44.311 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 4/172 2024-06-29T14:20:44.311 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-4577.gcf146 5/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 6/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 7/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 10/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 11/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 12/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 13/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_ 15/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 16/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 17/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_6 18/172 2024-06-29T14:20:44.312 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 19/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 20/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el 21/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9. 22/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_6 23/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 24/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 25/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 26/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 27/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 28/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 29/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f. 30/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 31/172 2024-06-29T14:20:44.313 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 32/172 2024-06-29T14:20:44.314 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 33/172 2024-06-29T14:20:44.314 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el 34/172 2024-06-29T14:20:44.314 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 35/172 2024-06-29T14:20:44.314 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.e 36/172 2024-06-29T14:20:44.314 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 37/172 2024-06-29T14:20:44.314 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 38/172 2024-06-29T14:20:44.314 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 39/172 2024-06-29T14:20:44.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/172 2024-06-29T14:20:44.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-06-29T14:20:44.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-06-29T14:20:44.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-06-29T14:20:44.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-06-29T14:20:44.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-06-29T14:20:44.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-06-29T14:20:44.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-06-29T14:20:44.316 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-06-29T14:20:44.317 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-06-29T14:20:44.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-06-29T14:20:44.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-06-29T14:20:44.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-06-29T14:20:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-06-29T14:20:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-06-29T14:20:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-06-29T14:20:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-06-29T14:20:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-06-29T14:20:44.319 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-06-29T14:20:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-06-29T14:20:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-06-29T14:20:44.320 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-06-29T14:20:44.322 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.4.0-1.el9.x86_64 101/172 2024-06-29T14:20:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-06-29T14:20:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-06-29T14:20:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-06-29T14:20:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-06-29T14:20:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-06-29T14:20:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-06-29T14:20:44.323 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-06-29T14:20:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-06-29T14:20:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-06-29T14:20:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-06-29T14:20:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-06-29T14:20:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-06-29T14:20:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-06-29T14:20:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.30.0-1.el9.noarch 122/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-06-29T14:20:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-06-29T14:20:44.326 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-06-29T14:20:44.327 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 149/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 151/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.48.0-1.el9.x86_64 153/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.48.0-1.el9.x86_64 155/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.48.0-1.el9.x86_64 157/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.48.0-1.el9.x86_64 159/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-06-29T14:20:44.328 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-06-29T14:20:44.329 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-06-29T14:20:44.329 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-06-29T14:20:44.329 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-06-29T14:20:44.329 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-06-29T14:20:44.329 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-06-29T14:20:44.329 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-06-29T14:20:44.329 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-06-29T14:20:44.329 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-6.4.0-1.el9.x86_64 171/172 2024-06-29T14:20:47.218 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:20:47.218 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:47.218 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-06-29T14:20:47.218 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-1:1.48.0-1.el9.x86_64 2024-06-29T14:20:47.218 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm-1:1.48.0-1.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team-1:1.48.0-1.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui-1:1.48.0-1.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-6.4.0-1.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-06-29T14:20:47.219 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.220 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-06-29T14:20:47.221 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-29T14:20:47.222 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-46.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-29T14:20:47.223 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.14-1.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-29T14:20:47.224 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.30.0-1.el9.noarch 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-29T14:20:47.225 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-29T14:20:47.226 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-29T14:20:47.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-29T14:20:47.228 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.4.0-1.el9.x86_64 2024-06-29T14:20:47.229 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-29T14:20:47.230 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-29T14:20:47.230 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:47.230 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-29T14:20:47.487 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:20:47.487 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 1/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 2/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 3/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 4/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-4577.gcf146 5/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 6/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 7/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 10/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 11/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 12/172 2024-06-29T14:20:47.488 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 13/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_ 15/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 16/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 17/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_6 18/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 19/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 20/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el 21/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9. 22/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_6 23/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 24/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 25/172 2024-06-29T14:20:47.489 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 26/172 2024-06-29T14:20:47.490 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 27/172 2024-06-29T14:20:47.490 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 28/172 2024-06-29T14:20:47.502 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 29/172 2024-06-29T14:20:47.502 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f. 30/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 31/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 32/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 33/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el 34/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 35/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.e 36/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 37/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 38/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 39/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-06-29T14:20:47.503 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-06-29T14:20:47.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-06-29T14:20:47.505 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-06-29T14:20:47.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-06-29T14:20:47.507 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-devel-6.4.0-1.el9.x86_64 101/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-06-29T14:20:47.508 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-06-29T14:20:47.509 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.30.0-1.el9.noarch 122/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-06-29T14:20:47.510 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-06-29T14:20:47.511 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 149/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 151/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-1:1.48.0-1.el9.x86_64 153/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-libnm-1:1.48.0-1.el9.x86_64 155/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-team-1:1.48.0-1.el9.x86_64 157/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-tui-1:1.48.0-1.el9.x86_64 159/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-06-29T14:20:47.512 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-06-29T14:20:47.513 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-6.4.0-1.el9.x86_64 171/172 2024-06-29T14:20:47.657 DEBUG:teuthology.parallel:result is None 2024-06-29T14:20:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:20:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:50.372 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-06-29T14:20:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-1:1.48.0-1.el9.x86_64 2024-06-29T14:20:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-libnm-1:1.48.0-1.el9.x86_64 2024-06-29T14:20:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-team-1:1.48.0-1.el9.x86_64 2024-06-29T14:20:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-tui-1:1.48.0-1.el9.x86_64 2024-06-29T14:20:50.372 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-6.4.0-1.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: automake-1.16.2-8.el9.noarch 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: bison-3.7.4-5.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-29T14:20:50.373 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.374 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: flex-2.6.4-9.el9.x86_64 2024-06-29T14:20:50.375 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-29T14:20:50.376 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: libtool-2.4.6-46.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-29T14:20:50.377 INFO:teuthology.orchestra.run.smithi059.stdout: lua-5.4.4-4.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-13.14-1.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-29T14:20:50.378 INFO:teuthology.orchestra.run.smithi059.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-29T14:20:50.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.30.0-1.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-29T14:20:50.380 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-29T14:20:50.381 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.382 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-29T14:20:50.383 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: re2-1:20211101-3.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-devel-6.4.0-1.el9.x86_64 2024-06-29T14:20:50.384 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-29T14:20:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-29T14:20:50.385 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:50.385 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-06-29T14:20:50.798 DEBUG:teuthology.parallel:result is None 2024-06-29T14:20:50.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:20:51.034 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-29T14:20:51.059 INFO:teuthology.orchestra.run.smithi059.stdout:19.0.0-4577.gcf146c2f.el9 2024-06-29T14:20:51.059 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4577.gcf146c2f.el9 2024-06-29T14:20:51.059 INFO:teuthology.task.install:The correct ceph version 19.0.0-4577.gcf146c2f is installed. 2024-06-29T14:20:51.061 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:20:51.290 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-29T14:20:51.318 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-4577.gcf146c2f.el9 2024-06-29T14:20:51.318 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4577.gcf146c2f.el9 2024-06-29T14:20:51.318 INFO:teuthology.task.install:The correct ceph version 19.0.0-4577.gcf146c2f is installed. 2024-06-29T14:20:51.320 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-29T14:20:51.320 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:51.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-29T14:20:51.354 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:20:51.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-29T14:20:51.389 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-29T14:20:51.389 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:51.389 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-06-29T14:20:51.424 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-29T14:20:51.492 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:20:51.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-06-29T14:20:51.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-29T14:20:51.589 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-29T14:20:51.589 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:51.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-29T14:20:51.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-29T14:20:51.686 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:20:51.686 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-29T14:20:51.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-29T14:20:51.779 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-29T14:20:51.779 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:51.779 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-06-29T14:20:51.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-29T14:20:51.874 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:20:51.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-06-29T14:20:51.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-29T14:20:51.966 INFO:teuthology.run_tasks:Running task ceph... 2024-06-29T14:20:52.045 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-06-29T14:20:52.045 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-06-29T14:20:52.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-06-29T14:20:52.073 INFO:tasks.ceph:Disabling ceph logrotate... 2024-06-29T14:20:52.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-29T14:20:52.116 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-29T14:20:52.141 INFO:tasks.ceph:Creating extra log directories... 2024-06-29T14:20:52.141 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-29T14:20:52.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-29T14:20:52.215 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-06-29T14:20:52.215 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-06-29T14:20:52.215 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-06-29T14:20:52.215 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:52.215 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-06-29T14:20:52.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-06-29T14:20:52.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-06-29T14:20:52.405 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:20:52.405 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-06-29T14:20:52.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-06-29T14:20:52.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-06-29T14:20:52.558 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-06-29T14:20:52.558 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-06-29T14:20:52.558 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923', 'branch': 'wip-mchangir-qa-handle-correct-CephManager-instantiation-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snap-schedule}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7778923', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-qa-handle-correct-CephManager-instantiation-debug'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'qa-handle-correct-CephManager-instantiation', 'sha1': '019bb938d4977149d100ab4c3d2c706147295470'}}, 'owner': 'scheduled_mchangir@teuthology', 'priority': 25, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1683, 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'qa-handle-correct-CephManager-instantiation', 'suite_path': '/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mchangir/ceph.git', 'suite_sha1': '019bb938d4977149d100ab4c3d2c706147295470', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVzEtjeLZZBhzDv+SpBG7iSBNbAYaYmtwJ3f5TkpQkrkcW6XYhdOIU+sAX+Mx0iQQ6AlUmiY0XJleEBZr57khA=', 'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9+d3Vrq9TnDck0IEV5e9v2Ww4f02ZHHfnR663z9oyc/t+Hd3gvLHgkwIfWuV13+10q5S4Bv4OTrxayVIYvZNc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '544fecbcd55f3d2b6f478823823ce40cbefef1d4', 'timestamp': '2024-06-29_14:01:10', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698093'} 2024-06-29T14:20:52.559 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-29T14:20:52.561 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-29T14:20:52.608 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-06-29T14:20:52.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-06-29T14:20:52.673 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:52.673 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-06-29T14:20:52.697 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-29T14:20:52.697 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-06-29 14:20:41.910185897 +0000 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-06-29 14:18:20.371123860 +0000 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-06-29 14:18:20.371123860 +0000 2024-06-29T14:20:52.753 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-06-29 14:18:20.371123860 +0000 2024-06-29T14:20:52.754 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-29T14:20:52.817 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-06-29T14:20:52.817 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-06-29T14:20:52.817 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000258213 s, 2.0 MB/s 2024-06-29T14:20:52.818 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-29T14:20:52.876 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-06-29T14:20:52.931 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-29T14:20:52.931 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:20:52.931 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-06-29T14:20:52.931 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:20:52.932 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:20:52.932 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-06-29 14:20:41.911185885 +0000 2024-06-29T14:20:52.932 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-06-29 14:18:20.714119927 +0000 2024-06-29T14:20:52.932 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-06-29 14:18:20.714119927 +0000 2024-06-29T14:20:52.932 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-06-29 14:18:20.714119927 +0000 2024-06-29T14:20:52.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-29T14:20:52.995 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-06-29T14:20:52.996 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-06-29T14:20:52.996 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000193406 s, 2.6 MB/s 2024-06-29T14:20:52.997 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-29T14:20:53.053 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-06-29T14:20:53.110 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-29T14:20:53.110 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:20:53.110 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-29T14:20:53.110 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:20:53.110 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:20:53.110 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-06-29 14:20:41.911185885 +0000 2024-06-29T14:20:53.111 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-06-29 14:18:21.062115937 +0000 2024-06-29T14:20:53.111 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-06-29 14:18:21.062115937 +0000 2024-06-29T14:20:53.111 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-06-29 14:18:21.062115937 +0000 2024-06-29T14:20:53.111 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-29T14:20:53.174 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-06-29T14:20:53.174 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-06-29T14:20:53.174 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000201466 s, 2.5 MB/s 2024-06-29T14:20:53.176 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-29T14:20:53.232 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-06-29T14:20:53.288 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-29T14:20:53.289 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:20:53.289 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-06-29T14:20:53.289 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:20:53.289 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:20:53.289 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-06-29 14:20:41.911185885 +0000 2024-06-29T14:20:53.289 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-06-29 14:18:21.398112084 +0000 2024-06-29T14:20:53.289 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-06-29 14:18:21.398112084 +0000 2024-06-29T14:20:53.289 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-06-29 14:18:21.398112084 +0000 2024-06-29T14:20:53.289 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-29T14:20:53.353 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-06-29T14:20:53.353 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-06-29T14:20:53.353 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000201191 s, 2.5 MB/s 2024-06-29T14:20:53.354 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-29T14:20:53.411 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:20:53.411 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:20:53.411 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-06-29T14:20:53.431 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-29T14:20:53.431 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-06-29T14:20:53.489 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-29T14:20:53.490 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:20:53.490 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-06-29T14:20:53.490 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:20:53.490 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:20:53.490 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-29 14:20:38.581880465 +0000 2024-06-29T14:20:53.490 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-29 14:18:19.469530855 +0000 2024-06-29T14:20:53.490 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-29 14:18:19.469530855 +0000 2024-06-29T14:20:53.490 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-29 14:18:19.469530855 +0000 2024-06-29T14:20:53.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-29T14:20:53.557 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-29T14:20:53.557 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-29T14:20:53.557 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000222516 s, 2.3 MB/s 2024-06-29T14:20:53.558 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-29T14:20:53.617 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-06-29T14:20:53.674 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-29T14:20:53.674 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:20:53.674 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-29T14:20:53.674 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:20:53.674 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:20:53.675 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-29 14:20:38.581880465 +0000 2024-06-29T14:20:53.675 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-29 14:18:19.794525446 +0000 2024-06-29T14:20:53.675 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-29 14:18:19.794525446 +0000 2024-06-29T14:20:53.675 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-29 14:18:19.794525446 +0000 2024-06-29T14:20:53.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-29T14:20:53.740 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-29T14:20:53.740 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-29T14:20:53.740 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000205081 s, 2.5 MB/s 2024-06-29T14:20:53.741 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-29T14:20:53.798 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-06-29T14:20:53.854 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-29T14:20:53.855 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:20:53.855 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-29T14:20:53.855 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:20:53.855 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:20:53.855 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-29 14:20:38.581880465 +0000 2024-06-29T14:20:53.855 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-29 14:18:20.123519971 +0000 2024-06-29T14:20:53.855 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-29 14:18:20.123519971 +0000 2024-06-29T14:20:53.855 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-29 14:18:20.123519971 +0000 2024-06-29T14:20:53.855 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-29T14:20:53.919 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-29T14:20:53.919 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-29T14:20:53.919 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000240837 s, 2.1 MB/s 2024-06-29T14:20:53.920 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-29T14:20:53.977 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-06-29T14:20:54.033 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-29T14:20:54.033 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:20:54.033 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-29T14:20:54.033 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:20:54.033 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:20:54.033 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-29 14:20:38.582880449 +0000 2024-06-29T14:20:54.033 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-29 14:18:20.464514296 +0000 2024-06-29T14:20:54.034 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-29 14:18:20.464514296 +0000 2024-06-29T14:20:54.034 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-29 14:18:20.464514296 +0000 2024-06-29T14:20:54.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-29T14:20:54.097 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-29T14:20:54.097 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-29T14:20:54.097 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000200896 s, 2.5 MB/s 2024-06-29T14:20:54.098 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-29T14:20:54.154 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:20:54.154 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-29T14:20:54.154 INFO:tasks.ceph:Generating config... 2024-06-29T14:20:54.156 INFO:tasks.ceph:[client] pid file = 2024-06-29T14:20:54.156 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-06-29T14:20:54.156 INFO:tasks.ceph:[mgr] debug client = 20 2024-06-29T14:20:54.156 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-06-29T14:20:54.156 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-06-29T14:20:54.156 INFO:tasks.ceph:[mgr] debug ms = 1 2024-06-29T14:20:54.156 INFO:tasks.ceph:[mon] debug mon = 20 2024-06-29T14:20:54.156 INFO:tasks.ceph:[mon] debug ms = 1 2024-06-29T14:20:54.156 INFO:tasks.ceph:[mon] debug paxos = 20 2024-06-29T14:20:54.156 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-06-29T14:20:54.156 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] debug ms = 1 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] debug osd = 20 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-06-29T14:20:54.157 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-06-29T14:20:54.157 INFO:tasks.ceph:Setting up mon.a... 2024-06-29T14:20:54.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-06-29T14:20:54.224 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.keyring 2024-06-29T14:20:54.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-06-29T14:20:54.287 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-29T14:20:54.313 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.59'), ('mon.b', '172.21.15.153'), ('mon.c', '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]')] 2024-06-29T14:20:54.314 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.59,172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-06-29T14:20:54.315 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:54.315 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-06-29T14:20:54.370 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.59 --add b 172.21.15.153 --addv c '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid 150b8fd8-1bf5-44f0-b8d0-0083a48c0071 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = quincy 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:fsid 150b8fd8-1bf5-44f0-b8d0-0083a48c0071 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2024-06-29T14:20:54.471315+0000 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-06-29T14:20:54.471315+0000 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 17 (quincy) 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 1 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-06-29T14:20:54.469 INFO:teuthology.orchestra.run.smithi059.stdout:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-06-29T14:20:54.470 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-06-29T14:20:54.471 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-06-29T14:20:54.488 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 150b8fd8-1bf5-44f0-b8d0-0083a48c0071... 2024-06-29T14:20:54.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-29T14:20:54.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-29T14:20:54.572 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-06-29T14:20:54.572 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-06-29T14:20:54.572 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-29T14:20:54.573 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-06-29T14:20:54.574 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790]" 2024-06-29T14:20:54.575 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on skipped message = False 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 150b8fd8-1bf5-44f0-b8d0-0083a48c0071 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-06-29T14:20:54.576 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-06-29T14:20:54.577 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-06-29T14:20:54.577 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.577 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-06-29T14:20:54.577 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-06-29T14:20:54.577 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.578 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-06-29T14:20:54.578 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.578 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-06-29T14:20:54.578 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.578 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-06-29T14:20:54.578 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.579 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-06-29T14:20:54.579 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-06-29T14:20:54.579 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.579 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-06-29T14:20:54.579 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-06-29T14:20:54.579 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-06-29T14:20:54.580 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-06-29T14:20:54.580 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 20 2024-06-29T14:20:54.580 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 20 2024-06-29T14:20:54.580 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-06-29T14:20:54.580 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-06-29T14:20:54.580 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 10 2024-06-29T14:20:54.581 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-06-29T14:20:54.581 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-06-29T14:20:54.581 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-06-29T14:20:54.581 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-06-29T14:20:54.582 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-06-29T14:20:54.582 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.582 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-06-29T14:20:54.582 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-06-29T14:20:54.582 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-06-29T14:20:54.582 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-06-29T14:20:54.582 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-06-29T14:20:54.583 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-06-29T14:20:54.583 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-06-29T14:20:54.583 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-06-29T14:20:54.583 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-06-29T14:20:54.583 INFO:teuthology.orchestra.run.smithi059.stdout: debug finisher = 20 2024-06-29T14:20:54.583 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.583 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-06-29T14:20:54.583 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-06-29T14:20:54.584 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-06-29T14:20:54.584 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-06-29T14:20:54.584 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-06-29T14:20:54.584 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-06-29T14:20:54.584 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-06-29T14:20:54.584 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-29T14:20:54.585 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-29T14:20:54.586 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.586 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-06-29T14:20:54.586 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-06-29T14:20:54.586 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.586 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-06-29T14:20:54.586 INFO:teuthology.orchestra.run.smithi059.stdout: mon down mkfs grace = 300 2024-06-29T14:20:54.586 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:20:54.586 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = "" 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-06-29T14:20:54.587 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-06-29T14:20:54.589 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-06-29T14:20:54.589 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-06-29T14:20:54.589 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-29T14:20:54.589 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-06-29T14:20:54.589 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.589 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-06-29T14:20:54.589 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.589 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-29T14:20:54.590 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.591 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "172.21.15.59,172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790]" 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on skipped message = False 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 150b8fd8-1bf5-44f0-b8d0-0083a48c0071 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-06-29T14:20:54.592 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-06-29T14:20:54.593 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-06-29T14:20:54.594 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-06-29T14:20:54.595 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: debug client = 20 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: debug finisher = 20 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-06-29T14:20:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-06-29T14:20:54.597 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: mon down mkfs grace = 300 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = "" 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2024-06-29T14:20:54.598 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2024-06-29T14:20:54.599 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.a] 2024-06-29T14:20:54.599 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.c] 2024-06-29T14:20:54.599 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.b] 2024-06-29T14:20:54.599 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.d] 2024-06-29T14:20:54.599 INFO:tasks.ceph:Creating admin key on mon.a... 2024-06-29T14:20:54.599 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-06-29T14:20:54.694 INFO:tasks.ceph:Copying monmap to all nodes... 2024-06-29T14:20:54.694 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:54.694 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-06-29T14:20:54.710 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:54.710 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-06-29T14:20:54.765 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:20:54.765 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:54.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-29T14:20:54.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-29T14:20:54.841 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:20:54.841 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:20:54.898 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T14:20:54.899 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:20:54.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-29T14:20:54.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-29T14:20:54.935 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:20:54.935 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:20:54.991 INFO:tasks.ceph:Setting up mon nodes... 2024-06-29T14:20:54.991 INFO:tasks.ceph:Setting up mgr nodes... 2024-06-29T14:20:54.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-06-29T14:20:55.054 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-06-29T14:20:55.057 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-06-29T14:20:55.129 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-06-29T14:20:55.132 INFO:tasks.ceph:Setting up mds nodes... 2024-06-29T14:20:55.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-06-29T14:20:55.194 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-06-29T14:20:55.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-06-29T14:20:55.223 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-06-29T14:20:55.325 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-06-29T14:20:55.328 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-06-29T14:20:55.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-06-29T14:20:55.414 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-06-29T14:20:55.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-06-29T14:20:55.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-06-29T14:20:55.543 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-06-29T14:20:55.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-06-29T14:20:55.571 INFO:tasks.ceph_client:Setting up client nodes... 2024-06-29T14:20:55.572 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-29T14:20:55.624 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-06-29T14:20:55.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-06-29T14:20:55.689 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-06-29T14:20:55.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-29T14:20:55.753 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-06-29T14:20:55.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-06-29T14:20:55.818 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-06-29T14:20:55.831 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-06-29T14:20:55.831 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-29T14:20:55.831 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-06-29T14:20:55.857 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:20:55.858 INFO:tasks.ceph:role: osd.0 2024-06-29T14:20:55.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:20:55.858 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-29T14:20:55.931 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:20:55.931 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:20:55.931 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:20:55.932 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:20:55.932 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:20:55.932 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:20:55.932 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:20:55.932 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:20:55.932 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:20:55.932 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:20:56.448 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-06-29T14:20:56.451 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-06-29T14:20:56.451 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-06-29T14:20:56.593 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-06-29T14:20:56.626 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-06-29T14:20:56.692 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:20:56.693 INFO:tasks.ceph:role: osd.1 2024-06-29T14:20:56.693 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:20:56.693 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-29T14:20:56.767 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:20:56.767 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:20:56.767 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:20:56.767 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:20:56.767 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:20:56.767 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:20:56.767 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:20:56.768 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:20:56.768 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:20:56.768 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:20:57.208 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-06-29T14:20:57.210 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-06-29T14:20:57.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-06-29T14:20:57.290 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-06-29T14:20:57.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-06-29T14:20:57.425 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:20:57.426 INFO:tasks.ceph:role: osd.2 2024-06-29T14:20:57.426 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:20:57.426 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-29T14:20:57.498 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:20:57.498 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:20:57.499 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:20:57.499 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:20:57.499 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:20:57.499 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:20:57.499 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:20:57.499 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:20:57.499 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:20:57.499 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:20:57.983 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-06-29T14:20:57.985 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-06-29T14:20:57.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-06-29T14:20:58.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-06-29T14:20:58.162 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-06-29T14:20:58.229 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:20:58.229 INFO:tasks.ceph:role: osd.3 2024-06-29T14:20:58.229 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:20:58.229 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-29T14:20:58.301 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:20:58.302 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:20:58.302 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:20:58.302 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:20:58.302 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:20:58.302 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:20:58.302 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:20:58.302 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:20:58.303 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:20:58.303 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:20:58.814 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-06-29T14:20:58.816 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-06-29T14:20:58.816 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-06-29T14:20:58.954 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-06-29T14:20:58.993 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:20:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:20:59.107+0000 7faa4a20f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-06-29T14:20:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:20:59.107+0000 7faa4a20f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-06-29T14:20:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:20:59.107+0000 7faa4a20f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-29T14:20:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:20:59.107+0000 7faa4a20f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-29T14:20:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:20:59.107+0000 7faa4a20f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-29T14:20:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:20:59.107+0000 7faa4a20f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-06-29T14:21:01.974 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-29T14:21:02.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:02.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:02.109+0000 7f103f5d6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-06-29T14:21:02.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:02.109+0000 7f103f5d6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-06-29T14:21:02.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:02.109+0000 7f103f5d6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-29T14:21:02.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:02.109+0000 7f103f5d6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-29T14:21:02.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:02.109+0000 7f103f5d6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-29T14:21:02.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:02.110+0000 7f103f5d6640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-06-29T14:21:04.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-29T14:21:04.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:05.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:05.095+0000 7f6294823640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-06-29T14:21:05.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:05.095+0000 7f6294823640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-06-29T14:21:05.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:05.095+0000 7f6294823640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-29T14:21:05.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:05.096+0000 7f6294823640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-29T14:21:05.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:05.096+0000 7f6294823640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-29T14:21:05.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:05.096+0000 7f6294823640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-06-29T14:21:07.947 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-29T14:21:07.975 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:08.078+0000 7f35e1c63640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-06-29T14:21:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:08.079+0000 7f35e1c63640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-06-29T14:21:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:08.079+0000 7f35e1c63640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-29T14:21:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:08.079+0000 7f35e1c63640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-29T14:21:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:08.079+0000 7f35e1c63640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-29T14:21:08.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:08.079+0000 7f35e1c63640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-06-29T14:21:10.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-29T14:21:10.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-06-29T14:21:10.988 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:10.988 INFO:tasks.ceph:role: osd.4 2024-06-29T14:21:10.988 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T14:21:10.989 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-29T14:21:11.065 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:11.065 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:11.065 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:11.065 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:11.065 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:11.065 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:11.065 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:11.066 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:11.066 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:11.066 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:11.577 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-06-29T14:21:11.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-06-29T14:21:11.580 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-06-29T14:21:11.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-06-29T14:21:11.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-06-29T14:21:11.826 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:11.826 INFO:tasks.ceph:role: osd.5 2024-06-29T14:21:11.826 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T14:21:11.826 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-29T14:21:11.898 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:11.899 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:12.384 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-06-29T14:21:12.386 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-06-29T14:21:12.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-06-29T14:21:12.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-06-29T14:21:12.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-06-29T14:21:12.578 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:12.578 INFO:tasks.ceph:role: osd.6 2024-06-29T14:21:12.579 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T14:21:12.579 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-29T14:21:12.651 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:12.652 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:13.174 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-06-29T14:21:13.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-06-29T14:21:13.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-06-29T14:21:13.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-06-29T14:21:13.358 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-06-29T14:21:13.423 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:13.423 INFO:tasks.ceph:role: osd.7 2024-06-29T14:21:13.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T14:21:13.423 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-29T14:21:13.495 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:13.496 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:13.496 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:13.496 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:13.496 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:14.003 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-06-29T14:21:14.005 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-06-29T14:21:14.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-06-29T14:21:14.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-06-29T14:21:14.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:14.266+0000 7f87fce2c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-06-29T14:21:14.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:14.267+0000 7f87fce2c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-06-29T14:21:14.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:14.267+0000 7f87fce2c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-29T14:21:14.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:14.267+0000 7f87fce2c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-29T14:21:14.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:14.267+0000 7f87fce2c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-29T14:21:14.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:14.267+0000 7f87fce2c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-06-29T14:21:17.112 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-29T14:21:17.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:17.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:17.238+0000 7f51f3b57640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-06-29T14:21:17.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:17.238+0000 7f51f3b57640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-06-29T14:21:17.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:17.238+0000 7f51f3b57640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-29T14:21:17.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:17.238+0000 7f51f3b57640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-29T14:21:17.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:17.238+0000 7f51f3b57640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-29T14:21:17.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:17.239+0000 7f51f3b57640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-06-29T14:21:20.087 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-29T14:21:20.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:20.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:20.211+0000 7f839aa0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-06-29T14:21:20.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:20.212+0000 7f839aa0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-06-29T14:21:20.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:20.212+0000 7f839aa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-29T14:21:20.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:20.212+0000 7f839aa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-29T14:21:20.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:20.212+0000 7f839aa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-29T14:21:20.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:20.212+0000 7f839aa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-06-29T14:21:22.560 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:21:22.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:21:23.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-29T14:21:23.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:23.183+0000 7f090bc15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-06-29T14:21:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:23.183+0000 7f090bc15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-06-29T14:21:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:23.183+0000 7f090bc15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-29T14:21:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:23.183+0000 7f090bc15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-29T14:21:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:23.183+0000 7f090bc15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-29T14:21:23.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:23.184+0000 7f090bc15640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-06-29T14:21:26.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-29T14:21:26.059 INFO:tasks.ceph:Reading keys from all nodes... 2024-06-29T14:21:26.059 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.059 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-06-29T14:21:26.091 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.091 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-06-29T14:21:26.161 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-06-29T14:21:26.230 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.230 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-06-29T14:21:26.298 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-06-29T14:21:26.367 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-06-29T14:21:26.435 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.436 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-06-29T14:21:26.503 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:26.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-06-29T14:21:26.532 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:26.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-06-29T14:21:26.601 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:26.601 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-06-29T14:21:26.669 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:26.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-06-29T14:21:26.736 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:26.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-06-29T14:21:26.803 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:26.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-06-29T14:21:26.869 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:26.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-06-29T14:21:26.934 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.935 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-06-29T14:21:26.954 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:26.954 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-06-29T14:21:27.014 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:27.014 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-06-29T14:21:27.030 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:27.030 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-06-29T14:21:27.085 INFO:tasks.ceph:Adding keys to all mons... 2024-06-29T14:21:27.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-29T14:21:27.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-29T14:21:27.129 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.y] 2024-06-29T14:21:27.130 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg== 2024-06-29T14:21:27.130 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-06-29T14:21:27.130 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w== 2024-06-29T14:21:27.130 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-06-29T14:21:27.131 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg== 2024-06-29T14:21:27.131 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-06-29T14:21:27.131 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag== 2024-06-29T14:21:27.131 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-06-29T14:21:27.131 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA== 2024-06-29T14:21:27.132 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-06-29T14:21:27.132 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ== 2024-06-29T14:21:27.132 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2024-06-29T14:21:27.132 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw== 2024-06-29T14:21:27.132 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-06-29T14:21:27.133 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw== 2024-06-29T14:21:27.133 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-06-29T14:21:27.133 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q== 2024-06-29T14:21:27.133 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-06-29T14:21:27.133 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA== 2024-06-29T14:21:27.134 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.4] 2024-06-29T14:21:27.134 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw== 2024-06-29T14:21:27.134 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.5] 2024-06-29T14:21:27.134 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q== 2024-06-29T14:21:27.134 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.6] 2024-06-29T14:21:27.134 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA== 2024-06-29T14:21:27.135 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.7] 2024-06-29T14:21:27.135 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg== 2024-06-29T14:21:27.135 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:21:27.135 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:21:27.135 INFO:teuthology.orchestra.run.smithi059.stdout:[client.2] 2024-06-29T14:21:27.136 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q== 2024-06-29T14:21:27.136 INFO:teuthology.orchestra.run.smithi059.stdout:[client.1] 2024-06-29T14:21:27.136 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA== 2024-06-29T14:21:27.136 INFO:teuthology.orchestra.run.smithi059.stdout:[client.3] 2024-06-29T14:21:27.136 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA== 2024-06-29T14:21:27.148 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.y] 2024-06-29T14:21:27.148 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg== 2024-06-29T14:21:27.148 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.a] 2024-06-29T14:21:27.148 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w== 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.c] 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg== 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag== 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA== 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ== 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw== 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw== 2024-06-29T14:21:27.149 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.b] 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q== 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.d] 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA== 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw== 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q== 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.6] 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA== 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.7] 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg== 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-06-29T14:21:27.150 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:21:27.151 INFO:teuthology.orchestra.run.smithi153.stdout:[client.2] 2024-06-29T14:21:27.151 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q== 2024-06-29T14:21:27.151 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-06-29T14:21:27.151 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA== 2024-06-29T14:21:27.151 INFO:teuthology.orchestra.run.smithi153.stdout:[client.3] 2024-06-29T14:21:27.151 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA== 2024-06-29T14:21:27.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-29T14:21:27.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-29T14:21:27.253 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:21:27.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:21:27.323 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:21:27.326 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:21:27.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.392 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.517 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.519 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.583 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.644 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-29T14:21:27.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-29T14:21:27.707 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:21:27.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:21:27.774 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:21:27.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:21:27.840 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.843 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.904 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.967 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:27.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:28.031 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:28.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:21:28.095 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:21:28.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:21:28.162 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:21:28.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:21:28.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:21:28.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:21:28.294 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:21:28.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:21:28.361 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-06-29T14:21:28.361 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-06-29T14:21:28.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-29T14:21:28.493 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-06-29T14:21:28.519 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-06-29T14:21:28.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-29T14:21:28.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-06-29T14:21:28.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-06-29T14:21:28.741 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-29T14:21:28.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-06-29T14:21:28.873 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:28.876 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:28.928 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-06-29T14:21:28.929 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-29T14:21:28.929 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-06-29T14:21:28.932 INFO:tasks.ceph.mon.a:Started 2024-06-29T14:21:28.932 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-29T14:21:28.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-06-29T14:21:28.970 INFO:tasks.ceph.mon.b:Started 2024-06-29T14:21:28.971 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-29T14:21:28.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-06-29T14:21:28.972 INFO:tasks.ceph.mon.c:Started 2024-06-29T14:21:28.973 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-06-29T14:21:28.973 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-06-29T14:21:28.973 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-06-29T14:21:28.975 INFO:tasks.ceph.mgr.y:Started 2024-06-29T14:21:28.975 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-06-29T14:21:28.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-06-29T14:21:28.978 INFO:tasks.ceph.mgr.x:Started 2024-06-29T14:21:28.978 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-06-29T14:21:28.978 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-06-29T14:21:28.980 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-06-29T14:21:28.980 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set client debug_client 20 2024-06-29T14:21:29.023 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-06-29T14:21:29.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set client debug_ms 1 2024-06-29T14:21:29.066 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-06-29T14:21:29.066 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-06-29T14:21:29.109 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-06-29T14:21:29.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-06-29T14:21:29.153 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-06-29T14:21:29.153 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-06-29T14:21:29.198 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-06-29T14:21:29.199 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-06-29T14:21:29.242 INFO:tasks.ceph:setting config [mds] debug_ms = 1 2024-06-29T14:21:29.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds debug_ms 1 2024-06-29T14:21:29.285 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-06-29T14:21:29.285 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-06-29T14:21:29.328 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-06-29T14:21:29.328 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-06-29T14:21:29.372 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-06-29T14:21:29.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-06-29T14:21:29.417 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-06-29T14:21:29.417 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-06-29T14:21:29.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.446+0000 7f703dccf640 1 -- 172.21.15.59:0/3711254868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7028004800 msgr2=0x7f7028004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.446+0000 7f703dccf640 1 --2- 172.21.15.59:0/3711254868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7028004800 0x7f7028004c10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f702c006f10 tx=0x7f7028005830 comp rx=0 tx=0).stop 2024-06-29T14:21:29.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.446+0000 7f703d4ce640 1 -- 172.21.15.59:0/3711254868 reap_dead start 2024-06-29T14:21:29.446 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-06-29T14:21:29.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-06-29T14:21:29.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.447+0000 7f703dccf640 1 -- 172.21.15.59:0/3711254868 shutdown_connections 2024-06-29T14:21:29.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.447+0000 7f703dccf640 1 -- 172.21.15.59:0/3711254868 >> 172.21.15.59:0/3711254868 conn(0x7f703001e3f0 msgr2=0x7f7030039e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.448+0000 7f88a5ffb640 1 -- 172.21.15.59:0/280553606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88a8004150 msgr2=0x7f88c01f3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.448+0000 7f88a5ffb640 1 --2- 172.21.15.59:0/280553606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88a8004150 0x7f88c01f3c20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88b0006610 tx=0x7f88b0005ae0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.448+0000 7f88a5ffb640 1 -- 172.21.15.59:0/280553606 shutdown_connections 2024-06-29T14:21:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.448+0000 7f88a5ffb640 1 --2- 172.21.15.59:0/280553606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88c0147c10 0x7f88c015d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.448+0000 7f88a5ffb640 1 --2- 172.21.15.59:0/280553606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88c0147800 0x7f88c015cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.448+0000 7f88a5ffb640 1 --2- 172.21.15.59:0/280553606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88a8004150 0x7f88c01f3c20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.448+0000 7f88a5ffb640 1 -- 172.21.15.59:0/280553606 >> 172.21.15.59:0/280553606 conn(0x7f88c00b1be0 msgr2=0x7f88c015b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.448+0000 7f88a5ffb640 1 -- 172.21.15.59:0/280553606 shutdown_connections 2024-06-29T14:21:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.449+0000 7f88a5ffb640 1 -- 172.21.15.59:0/280553606 wait complete. 2024-06-29T14:21:29.451 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-06-29T14:21:29.452 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-06-29T14:21:29.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.452+0000 7f154f1f5640 1 -- 172.21.15.59:0/1274440601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f15480b25a0 con 0x7f1548161110 2024-06-29T14:21:29.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.454+0000 7f703dccf640 1 -- 172.21.15.59:0/3711254868 shutdown_connections 2024-06-29T14:21:29.458 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-06-29T14:21:29.458 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-06-29T14:21:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.460+0000 7f333428f640 1 -- 172.21.15.59:0/844674694 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f332c0734e0 con 0x7f332c154810 2024-06-29T14:21:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.459+0000 7f703dccf640 1 -- 172.21.15.59:0/3711254868 wait complete. 2024-06-29T14:21:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.462+0000 7f703dccf640 1 Processor -- start 2024-06-29T14:21:29.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.462+0000 7f703dccf640 1 -- start start 2024-06-29T14:21:29.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.466+0000 7f703dccf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7028004800 0x7f70300d4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.470 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-06-29T14:21:29.471 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-06-29T14:21:29.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.467+0000 7fcd177fe640 1 -- 172.21.15.59:0/3561211361 <== mon.2 v2:172.21.15.153:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7fcd280129c0 con 0x7fcd30160380 2024-06-29T14:21:29.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.469+0000 7fc160ff9640 1 -- 172.21.15.59:0/1631207689 <== mon.0 v2:172.21.15.59:3300/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7fc15c017440 con 0x7fc1640d1a60 2024-06-29T14:21:29.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.468+0000 7f3322ffd640 1 -- 172.21.15.59:0/844674694 <== mon.2 v2:172.21.15.153:3301/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f331c011ac0 con 0x7f332c154810 2024-06-29T14:21:29.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.467+0000 7f15457fa640 1 -- 172.21.15.59:0/1274440601 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v3)=0 v3) ==== 123+0+0 (secure 0 0 0) 0x7f1540011a70 con 0x7f1548161110 2024-06-29T14:21:29.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.468+0000 7f15457fa640 1 -- 172.21.15.59:0/1274440601 <== mon.0 v2:172.21.15.59:3300/0 9 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f1540011d90 con 0x7f1548161110 2024-06-29T14:21:29.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.470+0000 7f1532ffd640 1 -- 172.21.15.59:0/1274440601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1548161110 msgr2=0x7f15480ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.470+0000 7f1532ffd640 1 --2- 172.21.15.59:0/1274440601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1548161110 0x7f15480ce220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f154000e990 tx=0x7f154000ee50 comp rx=0 tx=0).stop 2024-06-29T14:21:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.470+0000 7f1532ffd640 1 -- 172.21.15.59:0/1274440601 shutdown_connections 2024-06-29T14:21:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.470+0000 7f1532ffd640 1 --2- 172.21.15.59:0/1274440601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1548161110 0x7f15480ce220 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.470+0000 7f1532ffd640 1 --2- 172.21.15.59:0/1274440601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15480d32b0 0x7f1548160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.470+0000 7f1532ffd640 1 --2- 172.21.15.59:0/1274440601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15480b27c0 0x7f15481579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.470+0000 7f1532ffd640 1 -- 172.21.15.59:0/1274440601 >> 172.21.15.59:0/1274440601 conn(0x7f15480d3ec0 msgr2=0x7f154815b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.471+0000 7f1532ffd640 1 -- 172.21.15.59:0/1274440601 shutdown_connections 2024-06-29T14:21:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.471+0000 7f1532ffd640 1 -- 172.21.15.59:0/1274440601 wait complete. 2024-06-29T14:21:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.469+0000 7fe1c27fc640 1 -- 172.21.15.59:0/2315216118 <== mon.0 v2:172.21.15.59:3300/0 8 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7fe1c400aa80 con 0x7fe1cc1586b0 2024-06-29T14:21:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.467+0000 7f703cccd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7028004800 0x7f70300d4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.467+0000 7f703cccd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7028004800 0x7f70300d4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42364/0 (socket says 172.21.15.59:42364) 2024-06-29T14:21:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.467+0000 7f703cccd640 1 -- 172.21.15.59:0/3494931036 learned_addr learned my addr 172.21.15.59:0/3494931036 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.467+0000 7f703dccf640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70300d2660 0x7f7030050750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.468+0000 7f703dccf640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70300520f0 0x7f7030036b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.468+0000 7f703dccf640 1 -- 172.21.15.59:0/3494931036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f70300066a0 con 0x7f7028004800 2024-06-29T14:21:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.468+0000 7f7037fff640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70300d2660 0x7f7030050750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.468+0000 7f703d4ce640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70300520f0 0x7f7030036b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.468+0000 7f703dccf640 1 -- 172.21.15.59:0/3494931036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f70300dfa40 con 0x7f70300d2660 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.468+0000 7f703dccf640 1 -- 172.21.15.59:0/3494931036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f70300536c0 con 0x7f70300520f0 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.473+0000 7f703cccd640 1 -- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70300520f0 msgr2=0x7f7030036b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.473+0000 7f703cccd640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70300520f0 0x7f7030036b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.473+0000 7f703cccd640 1 -- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70300d2660 msgr2=0x7f7030050750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.473+0000 7f703cccd640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70300d2660 0x7f7030050750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.473+0000 7f703cccd640 1 -- 172.21.15.59:0/3494931036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7030052500 con 0x7f7028004800 2024-06-29T14:21:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.474+0000 7f703cccd640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7028004800 0x7f70300d4ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f703806b240 tx=0x7f703806a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.478 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-06-29T14:21:29.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-06-29T14:21:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.477+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7038075020 con 0x7f7028004800 2024-06-29T14:21:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.477+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(2 keys) ==== 54+0+0 (secure 0 0 0) 0x7f70380751c0 con 0x7f7028004800 2024-06-29T14:21:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.477+0000 7f703dccf640 1 -- 172.21.15.59:0/3494931036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7030051050 con 0x7f7028004800 2024-06-29T14:21:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.478+0000 7f703dccf640 1 -- 172.21.15.59:0/3494931036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7030051610 con 0x7f7028004800 2024-06-29T14:21:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.478+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70380693d0 con 0x7f7028004800 2024-06-29T14:21:29.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.483+0000 7fcd177fe640 1 -- 172.21.15.59:0/3561211361 <== mon.2 v2:172.21.15.153:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fcd28012b90 con 0x7fcd30160380 2024-06-29T14:21:29.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.483+0000 7fe1c27fc640 1 -- 172.21.15.59:0/2315216118 <== mon.0 v2:172.21.15.59:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fe1c400ac80 con 0x7fe1cc1586b0 2024-06-29T14:21:29.485 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-06-29T14:21:29.485 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-06-29T14:21:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.484+0000 7f3322ffd640 1 -- 172.21.15.59:0/844674694 <== mon.2 v2:172.21.15.153:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f331c011cf0 con 0x7f332c154810 2024-06-29T14:21:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.485+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7038096420 con 0x7f7028004800 2024-06-29T14:21:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.485+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7038091020 con 0x7f7028004800 2024-06-29T14:21:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.485+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 6 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f7038070800 con 0x7f7028004800 2024-06-29T14:21:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.484+0000 7f3322ffd640 1 -- 172.21.15.59:0/844674694 <== mon.2 v2:172.21.15.153:3301/0 10 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7f331c0233f0 con 0x7f332c154810 2024-06-29T14:21:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.486+0000 7fc160ff9640 1 -- 172.21.15.59:0/1631207689 <== mon.0 v2:172.21.15.59:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fc15c0175f0 con 0x7fc1640d1a60 2024-06-29T14:21:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.490+0000 7fcd157fa640 1 -- 172.21.15.59:0/3561211361 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7fcd08005310 con 0x7fcd30160380 2024-06-29T14:21:29.495 DEBUG:tasks.ceph:set 18 configs 2024-06-29T14:21:29.495 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.503+0000 7f703dccf640 1 -- 172.21.15.59:0/3494931036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7010005290 con 0x7f7028004800 2024-06-29T14:21:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.505+0000 7fa8a2935640 1 -- 172.21.15.59:0/2947565686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c06e1f0 msgr2=0x7fa89c0ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.505+0000 7fa8a2935640 1 --2- 172.21.15.59:0/2947565686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c06e1f0 0x7fa89c0ceec0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa88c009830 tx=0x7fa88c02a3f0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.506+0000 7fa8a2935640 1 -- 172.21.15.59:0/2947565686 shutdown_connections 2024-06-29T14:21:29.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.506+0000 7fa8a2935640 1 --2- 172.21.15.59:0/2947565686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c06e1f0 0x7fa89c0ceec0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.506+0000 7fa8a2935640 1 -- 172.21.15.59:0/2947565686 >> 172.21.15.59:0/2947565686 conn(0x7fa89c0ccbe0 msgr2=0x7fa89c0ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.505+0000 7f333428f640 1 -- 172.21.15.59:0/844674694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f332c154810 msgr2=0x7f332c154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.505+0000 7f333428f640 1 --2- 172.21.15.59:0/844674694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f332c154810 0x7f332c154c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f331c00bef0 tx=0x7f331c0039c0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.507+0000 7f333428f640 1 -- 172.21.15.59:0/844674694 shutdown_connections 2024-06-29T14:21:29.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.507+0000 7f333428f640 1 --2- 172.21.15.59:0/844674694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f332c159b30 0x7f332c159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.507+0000 7f333428f640 1 --2- 172.21.15.59:0/844674694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f332c154810 0x7f332c154c20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.507+0000 7f333428f640 1 --2- 172.21.15.59:0/844674694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3318003ac0 0x7f332c06d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.507+0000 7f333428f640 1 -- 172.21.15.59:0/844674694 >> 172.21.15.59:0/844674694 conn(0x7f332c0b1be0 msgr2=0x7f332c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.506+0000 7fa8a2935640 1 -- 172.21.15.59:0/2947565686 shutdown_connections 2024-06-29T14:21:29.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.506+0000 7fa8a2935640 1 -- 172.21.15.59:0/2947565686 wait complete. 2024-06-29T14:21:29.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.509+0000 7fa8a2935640 1 Processor -- start 2024-06-29T14:21:29.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.510+0000 7fcd177fe640 1 -- 172.21.15.59:0/3561211361 <== mon.2 v2:172.21.15.153:3301/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7fcd28012de0 con 0x7fcd30160380 2024-06-29T14:21:29.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.510+0000 7fcd177fe640 1 -- 172.21.15.59:0/3561211361 <== mon.2 v2:172.21.15.153:3301/0 11 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7fcd28024440 con 0x7fcd30160380 2024-06-29T14:21:29.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.510+0000 7fc160ff9640 1 -- 172.21.15.59:0/1631207689 <== mon.0 v2:172.21.15.59:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7fc15c00f400 con 0x7fc1640d1a60 2024-06-29T14:21:29.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.510+0000 7fa8a2935640 1 -- start start 2024-06-29T14:21:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.512+0000 7f333428f640 1 -- 172.21.15.59:0/844674694 shutdown_connections 2024-06-29T14:21:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.512+0000 7f333428f640 1 -- 172.21.15.59:0/844674694 wait complete. 2024-06-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.512+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f70380709d0 con 0x7f7028004800 2024-06-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fa8a2935640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c06e1f0 0x7fa89c160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fcd157fa640 1 -- 172.21.15.59:0/3561211361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd30160380 msgr2=0x7fcd30140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fcd157fa640 1 --2- 172.21.15.59:0/3561211361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd30160380 0x7fcd30140d90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd280040c0 tx=0x7fcd28004580 comp rx=0 tx=0).stop 2024-06-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fcd157fa640 1 -- 172.21.15.59:0/3561211361 shutdown_connections 2024-06-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fcd157fa640 1 --2- 172.21.15.59:0/3561211361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd30160380 0x7fcd30140d90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fcd157fa640 1 --2- 172.21.15.59:0/3561211361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd30145710 0x7fcd30140850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fcd157fa640 1 --2- 172.21.15.59:0/3561211361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd30072550 0x7fcd3015aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fcd157fa640 1 -- 172.21.15.59:0/3561211361 >> 172.21.15.59:0/3561211361 conn(0x7fcd300b24c0 msgr2=0x7fcd3015e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fa8a2935640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa89c073590 0x7fa89c14d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fa89bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c06e1f0 0x7fa89c160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fa89bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c06e1f0 0x7fa89c160940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:40946/0 (socket says 172.21.15.59:40946) 2024-06-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fa89bfff640 1 -- 172.21.15.59:0/1731374113 learned_addr learned my addr 172.21.15.59:0/1731374113 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.513+0000 7fa8a2935640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c161d80 0x7fa89c0adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.514+0000 7fcd157fa640 1 -- 172.21.15.59:0/3561211361 shutdown_connections 2024-06-29T14:21:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.514+0000 7fcd157fa640 1 -- 172.21.15.59:0/3561211361 wait complete. 2024-06-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.515+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa89c0742c0 con 0x7fa89c073590 2024-06-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.515+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa89c162a90 con 0x7fa89c161d80 2024-06-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.515+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa89c15c7c0 con 0x7fa89c06e1f0 2024-06-29T14:21:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.515+0000 7fa8a0ed2640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c161d80 0x7fa89c0adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.521+0000 7fa8a0ed2640 1 -- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c06e1f0 msgr2=0x7fa89c160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.521+0000 7fa8a0ed2640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c06e1f0 0x7fa89c160940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.521+0000 7fa8a0ed2640 1 -- 172.21.15.59:0/1731374113 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa89c073590 msgr2=0x7fa89c14d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.521+0000 7fa8a0ed2640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa89c073590 0x7fa89c14d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.521+0000 7fa8a0ed2640 1 -- 172.21.15.59:0/1731374113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa89c1558c0 con 0x7fa89c161d80 2024-06-29T14:21:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.522+0000 7fa8a0ed2640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c161d80 0x7fa89c0adea0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa89000b3e0 tx=0x7fa89000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.522+0000 7fa898ff9640 1 -- 172.21.15.59:0/1731374113 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa890015020 con 0x7fa89c161d80 2024-06-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.523+0000 7fa898ff9640 1 -- 172.21.15.59:0/1731374113 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7fa890011850 con 0x7fa89c161d80 2024-06-29T14:21:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.523+0000 7fa898ff9640 1 -- 172.21.15.59:0/1731374113 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa890011ca0 con 0x7fa89c161d80 2024-06-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.523+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa89c14a630 con 0x7fa89c161d80 2024-06-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.523+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa89c14abf0 con 0x7fa89c161d80 2024-06-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.523+0000 7fa898ff9640 1 -- 172.21.15.59:0/1731374113 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa890018020 con 0x7fa89c161d80 2024-06-29T14:21:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.523+0000 7fa898ff9640 1 -- 172.21.15.59:0/1731374113 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa89001a450 con 0x7fa89c161d80 2024-06-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.527+0000 7fe1c27fc640 1 -- 172.21.15.59:0/2315216118 <== mon.0 v2:172.21.15.59:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7fe1c400f3e0 con 0x7fe1cc1586b0 2024-06-29T14:21:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.528+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa87c005290 con 0x7fa89c161d80 2024-06-29T14:21:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.539+0000 7fa898ff9640 1 -- 172.21.15.59:0/1731374113 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fa89001f070 con 0x7fa89c161d80 2024-06-29T14:21:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.550+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 8 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f7038075370 con 0x7f7028004800 2024-06-29T14:21:29.560 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.629+0000 7fc159ffb640 1 -- 172.21.15.59:0/1631207689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7fc13c0052f0 con 0x7fc1640d1a60 2024-06-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.636+0000 7fc160ff9640 1 -- 172.21.15.59:0/1631207689 <== mon.0 v2:172.21.15.59:3300/0 11 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v6)=0 v6) ==== 114+0+0 (secure 0 0 0) 0x7fc15c004ab0 con 0x7fc1640d1a60 2024-06-29T14:21:29.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.636+0000 7fc160ff9640 1 -- 172.21.15.59:0/1631207689 <== mon.0 v2:172.21.15.59:3300/0 12 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc15c004e00 con 0x7fc1640d1a60 2024-06-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.637+0000 7fe1c27fc640 1 -- 172.21.15.59:0/2315216118 <== mon.0 v2:172.21.15.59:3300/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1c400f8c0 con 0x7fe1cc1586b0 2024-06-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.639+0000 7fa898ff9640 1 -- 172.21.15.59:0/1731374113 <== mon.1 v2:172.21.15.153:3300/0 7 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8900035f0 con 0x7fa89c161d80 2024-06-29T14:21:29.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.640+0000 7fc159ffb640 1 -- 172.21.15.59:0/1631207689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640d1a60 msgr2=0x7fc164154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.640+0000 7fc159ffb640 1 --2- 172.21.15.59:0/1631207689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640d1a60 0x7fc164154550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc1640738c0 tx=0x7fc15c00c6e0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.643+0000 7fc159ffb640 1 -- 172.21.15.59:0/1631207689 shutdown_connections 2024-06-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.643+0000 7fc159ffb640 1 --2- 172.21.15.59:0/1631207689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640d1a60 0x7fc164154550 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.643+0000 7fc159ffb640 1 --2- 172.21.15.59:0/1631207689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1640b2fa0 0x7fc16415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.643+0000 7fc159ffb640 1 --2- 172.21.15.59:0/1631207689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1640b2270 0x7fc164147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.643+0000 7fc159ffb640 1 -- 172.21.15.59:0/1631207689 >> 172.21.15.59:0/1631207689 conn(0x7fc164073e10 msgr2=0x7fc1640d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.643+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 9 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7038096710 con 0x7f7028004800 2024-06-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.643+0000 7fc159ffb640 1 -- 172.21.15.59:0/1631207689 shutdown_connections 2024-06-29T14:21:29.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.643+0000 7fc159ffb640 1 -- 172.21.15.59:0/1631207689 wait complete. 2024-06-29T14:21:29.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.648+0000 7fe1affff640 1 -- 172.21.15.59:0/2315216118 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fe1a80052f0 con 0x7fe1cc1586b0 2024-06-29T14:21:29.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.666+0000 7fe1c27fc640 1 -- 172.21.15.59:0/2315216118 <== mon.0 v2:172.21.15.59:3300/0 12 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v7)=0 v7) ==== 108+0+0 (secure 0 0 0) 0x7fe1cc1405a0 con 0x7fe1cc1586b0 2024-06-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.676+0000 7fe1d3bd4640 1 -- 172.21.15.59:0/2315216118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1cc1586b0 msgr2=0x7fe1cc158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.676+0000 7fe1d3bd4640 1 --2- 172.21.15.59:0/2315216118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1cc1586b0 0x7fe1cc158ac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc0696f0 tx=0x7fe1c400cc50 comp rx=0 tx=0).stop 2024-06-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.679+0000 7fe1d3bd4640 1 -- 172.21.15.59:0/2315216118 shutdown_connections 2024-06-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.679+0000 7fe1d3bd4640 1 --2- 172.21.15.59:0/2315216118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1cc1586b0 0x7fe1cc158ac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.679+0000 7fe1d3bd4640 1 --2- 172.21.15.59:0/2315216118 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1cc13fc50 0x7fe1cc140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.679+0000 7fe1d3bd4640 1 --2- 172.21.15.59:0/2315216118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1cc157dd0 0x7fe1cc159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.679+0000 7fe1d3bd4640 1 -- 172.21.15.59:0/2315216118 >> 172.21.15.59:0/2315216118 conn(0x7fe1cc0b6fb0 msgr2=0x7fe1cc15f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.683+0000 7fe1d3bd4640 1 -- 172.21.15.59:0/2315216118 shutdown_connections 2024-06-29T14:21:29.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.683+0000 7fe1d3bd4640 1 -- 172.21.15.59:0/2315216118 wait complete. 2024-06-29T14:21:29.702 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.702 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.702 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.702 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.702+0000 7f92664d6640 1 -- 172.21.15.59:0/1547432399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92600ad5d0 msgr2=0x7f926015a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.702+0000 7f92664d6640 1 --2- 172.21.15.59:0/1547432399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92600ad5d0 0x7f926015a3e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f92600ae540 tx=0x7f925402a5c0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.704+0000 7f92664d6640 1 -- 172.21.15.59:0/1547432399 shutdown_connections 2024-06-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.704+0000 7f92664d6640 1 --2- 172.21.15.59:0/1547432399 >> v2:172.21.15.59:3300/0 conn(0x7f92600b4a30 0x7f92600cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.704+0000 7f92664d6640 1 --2- 172.21.15.59:0/1547432399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92600ad5d0 0x7f926015a3e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.704+0000 7f92664d6640 1 -- 172.21.15.59:0/1547432399 >> 172.21.15.59:0/1547432399 conn(0x7f92600b1be0 msgr2=0x7f92600b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.704+0000 7f92664d6640 1 -- 172.21.15.59:0/1547432399 shutdown_connections 2024-06-29T14:21:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.704+0000 7f92664d6640 1 -- 172.21.15.59:0/1547432399 wait complete. 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.707+0000 7f92664d6640 1 Processor -- start 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.707+0000 7f92664d6640 1 -- start start 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.710+0000 7f92664d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92600b4310 0x7f92600c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.710+0000 7f925ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92600b4310 0x7f92600c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.710+0000 7f925ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92600b4310 0x7f92600c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42404/0 (socket says 172.21.15.59:42404) 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.710+0000 7f925ffff640 1 -- 172.21.15.59:0/3432879379 learned_addr learned my addr 172.21.15.59:0/3432879379 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.710+0000 7f92664d6640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92600b4a30 0x7f9260153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.710+0000 7f92664d6640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92600c4e50 0x7f9260157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f92664d6640 1 -- 172.21.15.59:0/3432879379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f92600ce220 con 0x7f92600b4310 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f92664d6640 1 -- 172.21.15.59:0/3432879379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92600ce0a0 con 0x7f92600c4e50 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f92664d6640 1 -- 172.21.15.59:0/3432879379 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f92600ce3a0 con 0x7f92600b4a30 2024-06-29T14:21:29.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f9264a73640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92600c4e50 0x7f9260157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f925ffff640 1 -- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92600b4a30 msgr2=0x7f9260153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f925ffff640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92600b4a30 0x7f9260153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f925ffff640 1 -- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92600c4e50 msgr2=0x7f9260157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f925ffff640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92600c4e50 0x7f9260157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.711+0000 7f925ffff640 1 -- 172.21.15.59:0/3432879379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926006f730 con 0x7f92600b4310 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.712+0000 7f925ffff640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92600b4310 0x7f92600c6ad0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f925000c760 tx=0x7f925000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.713+0000 7f925cff9640 1 -- 172.21.15.59:0/3432879379 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500050d0 con 0x7f92600b4310 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.713+0000 7f925cff9640 1 -- 172.21.15.59:0/3432879379 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9250005270 con 0x7f92600b4310 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.713+0000 7f925cff9640 1 -- 172.21.15.59:0/3432879379 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925000a640 con 0x7f92600b4310 2024-06-29T14:21:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.714+0000 7f92664d6640 1 -- 172.21.15.59:0/3432879379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92600c5a30 con 0x7f92600b4310 2024-06-29T14:21:29.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.720+0000 7f92664d6640 1 -- 172.21.15.59:0/3432879379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92600c5ff0 con 0x7f92600b4310 2024-06-29T14:21:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.724+0000 7f925cff9640 1 -- 172.21.15.59:0/3432879379 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f925000ab70 con 0x7f92600b4310 2024-06-29T14:21:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.724+0000 7f925cff9640 1 -- 172.21.15.59:0/3432879379 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9250012780 con 0x7f92600b4310 2024-06-29T14:21:29.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.724+0000 7f924e7fc640 1 -- 172.21.15.59:0/3432879379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92600ce850 con 0x7f92600b4310 2024-06-29T14:21:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.728+0000 7f925cff9640 1 -- 172.21.15.59:0/3432879379 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f9250020020 con 0x7f92600b4310 2024-06-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.744+0000 7f9ceea74640 1 -- 172.21.15.59:0/1132781329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 msgr2=0x7f9ce80cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.744+0000 7f9ceea74640 1 --2- 172.21.15.59:0/1132781329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 0x7f9ce80cd9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009830 tx=0x7f9cdc02c590 comp rx=0 tx=0).stop 2024-06-29T14:21:29.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.751+0000 7f9ceea74640 1 -- 172.21.15.59:0/1132781329 shutdown_connections 2024-06-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.751+0000 7f9ceea74640 1 --2- 172.21.15.59:0/1132781329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 0x7f9ce80cd9b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.751+0000 7f9ceea74640 1 --2- 172.21.15.59:0/1132781329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce816bd00 0x7f9ce8165020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.751+0000 7f9ceea74640 1 -- 172.21.15.59:0/1132781329 >> 172.21.15.59:0/1132781329 conn(0x7f9ce80ccbe0 msgr2=0x7f9ce80ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.752+0000 7f9ceea74640 1 -- 172.21.15.59:0/1132781329 shutdown_connections 2024-06-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.752+0000 7f9ceea74640 1 -- 172.21.15.59:0/1132781329 wait complete. 2024-06-29T14:21:29.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.755+0000 7f9ceea74640 1 Processor -- start 2024-06-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.755+0000 7f9ceea74640 1 -- start start 2024-06-29T14:21:29.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.758+0000 7f9ceea74640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 0x7f9ce815e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.762+0000 7f9ceea74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce80cdef0 0x7f9ce815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.762+0000 7f9cec810640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 0x7f9ce815e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.762+0000 7f9cec810640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 0x7f9ce815e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:42116/0 (socket says 172.21.15.59:42116) 2024-06-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.762+0000 7f9ce7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce80cdef0 0x7f9ce815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.763+0000 7f9ce7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce80cdef0 0x7f9ce815fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42420/0 (socket says 172.21.15.59:42420) 2024-06-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.763+0000 7f9cec810640 1 -- 172.21.15.59:0/1163787978 learned_addr learned my addr 172.21.15.59:0/1163787978 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.763+0000 7f9ceea74640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce816bd00 0x7f9ce815fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.763+0000 7f9ceea74640 1 -- 172.21.15.59:0/1163787978 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce80725e0 con 0x7f9ce80cdef0 2024-06-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.763+0000 7f9ceea74640 1 -- 172.21.15.59:0/1163787978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8072460 con 0x7f9ce808b2d0 2024-06-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.763+0000 7f9ced011640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce816bd00 0x7f9ce815fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.763+0000 7f9ceea74640 1 -- 172.21.15.59:0/1163787978 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9ce8072760 con 0x7f9ce816bd00 2024-06-29T14:21:29.764 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.767+0000 7f9cec810640 1 -- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce816bd00 msgr2=0x7f9ce815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.767+0000 7f9cec810640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce816bd00 0x7f9ce815fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.767+0000 7f9cec810640 1 -- 172.21.15.59:0/1163787978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce80cdef0 msgr2=0x7f9ce815fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.767+0000 7f9cec810640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce80cdef0 0x7f9ce815fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.767+0000 7f9cec810640 1 -- 172.21.15.59:0/1163787978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce815c1f0 con 0x7f9ce808b2d0 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.767+0000 7f9ce7fff640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce80cdef0 0x7f9ce815fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.767+0000 7f9cec810640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 0x7f9ce815e1e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400f930 tx=0x7f9cd400fdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.768+0000 7f9ce57fa640 1 -- 172.21.15.59:0/1163787978 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400de30 con 0x7f9ce808b2d0 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.768+0000 7f9ce57fa640 1 -- 172.21.15.59:0/1163787978 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cd4005020 con 0x7f9ce808b2d0 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.768+0000 7f9ceea74640 1 -- 172.21.15.59:0/1163787978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce815c4d0 con 0x7f9ce808b2d0 2024-06-29T14:21:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.768+0000 7f9ceea74640 1 -- 172.21.15.59:0/1163787978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce815ca90 con 0x7f9ce808b2d0 2024-06-29T14:21:29.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.770+0000 7f9ce57fa640 1 -- 172.21.15.59:0/1163787978 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4005450 con 0x7f9ce808b2d0 2024-06-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.771+0000 7f9ce57fa640 1 -- 172.21.15.59:0/1163787978 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9cd4004b30 con 0x7f9ce808b2d0 2024-06-29T14:21:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.771+0000 7f9ce57fa640 1 -- 172.21.15.59:0/1163787978 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9cd4017730 con 0x7f9ce808b2d0 2024-06-29T14:21:29.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.779+0000 7f9cd2ffd640 1 -- 172.21.15.59:0/1163787978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc8005290 con 0x7f9ce808b2d0 2024-06-29T14:21:29.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.781+0000 7f9ce57fa640 1 -- 172.21.15.59:0/1163787978 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f9cd4037030 con 0x7f9ce808b2d0 2024-06-29T14:21:29.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.797+0000 7f703dccf640 1 -- 172.21.15.59:0/3494931036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f7010005580 con 0x7f7028004800 2024-06-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.802+0000 7faea1391640 1 -- 172.21.15.59:0/3030873556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0b3800 msgr2=0x7fae9c0b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.802+0000 7faea1391640 1 --2- 172.21.15.59:0/3030873556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0b3800 0x7fae9c0b3c10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009830 tx=0x7fae8c02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.804+0000 7f1cd5847640 1 -- 172.21.15.59:0/12722848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd0161d80 msgr2=0x7f1cd0162190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.804+0000 7f1cd5847640 1 --2- 172.21.15.59:0/12722848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd0161d80 0x7f1cd0162190 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8005fd0 tx=0x7f1cb802a560 comp rx=0 tx=0).stop 2024-06-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.804+0000 7f1cd5847640 1 -- 172.21.15.59:0/12722848 shutdown_connections 2024-06-29T14:21:29.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.804+0000 7f1cd5847640 1 --2- 172.21.15.59:0/12722848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd0161d80 0x7f1cd0162190 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.804+0000 7f1cd5847640 1 --2- 172.21.15.59:0/12722848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd006e1f0 0x7f1cd00ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.804+0000 7f1cd5847640 1 -- 172.21.15.59:0/12722848 >> 172.21.15.59:0/12722848 conn(0x7f1cd00ccbe0 msgr2=0x7f1cd00ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.805+0000 7f1cd5847640 1 -- 172.21.15.59:0/12722848 shutdown_connections 2024-06-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.805+0000 7f1cd5847640 1 -- 172.21.15.59:0/12722848 wait complete. 2024-06-29T14:21:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.808+0000 7f1cd5847640 1 Processor -- start 2024-06-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.808+0000 7f1cd5847640 1 -- start start 2024-06-29T14:21:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7faea1391640 1 -- 172.21.15.59:0/3030873556 shutdown_connections 2024-06-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7faea1391640 1 --2- 172.21.15.59:0/3030873556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0b3800 0x7fae9c0b3c10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7faea1391640 1 --2- 172.21.15.59:0/3030873556 >> v2:172.21.15.153:3300/0 conn(0x7fae9c0cf380 0x7fae9c0ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7faea1391640 1 -- 172.21.15.59:0/3030873556 >> 172.21.15.59:0/3030873556 conn(0x7fae9c0b24c0 msgr2=0x7fae9c0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7f01e75eb640 1 -- 172.21.15.59:0/3559638319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01cc0053c0 msgr2=0x7f01cc00ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7f01e75eb640 1 --2- 172.21.15.59:0/3559638319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01cc0053c0 0x7f01cc00ba40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f01e00cf990 tx=0x7f01cc005830 comp rx=0 tx=0).stop 2024-06-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7f01e5b88640 1 -- 172.21.15.59:0/3559638319 reap_dead start 2024-06-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7f01e75eb640 1 -- 172.21.15.59:0/3559638319 shutdown_connections 2024-06-29T14:21:29.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.809+0000 7f01e75eb640 1 -- 172.21.15.59:0/3559638319 >> 172.21.15.59:0/3559638319 conn(0x7f01e00ccbe0 msgr2=0x7f01e00ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f1cd5847640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cd006e1f0 0x7f1cd0154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f70357fa640 1 -- 172.21.15.59:0/3494931036 <== mon.0 v2:172.21.15.59:3300/0 10 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v8)=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7f7038070bd0 con 0x7f7028004800 2024-06-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f1cceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cd006e1f0 0x7f1cd0154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f1cceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cd006e1f0 0x7f1cd0154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42440/0 (socket says 172.21.15.59:42440) 2024-06-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f1cceffd640 1 -- 172.21.15.59:0/1106899989 learned_addr learned my addr 172.21.15.59:0/1106899989 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f1cd5847640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd0161d80 0x7f1cd013b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f1cd5847640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd01626d0 0x7f1cd014c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f1cd5847640 1 -- 172.21.15.59:0/1106899989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd0074220 con 0x7f1cd006e1f0 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.812+0000 7f1cc7fff640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd0161d80 0x7f1cd013b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.813+0000 7f1cd5847640 1 -- 172.21.15.59:0/1106899989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd00740a0 con 0x7f1cd0161d80 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.813+0000 7f1ccf7fe640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd01626d0 0x7f1cd014c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.813+0000 7f1cd5847640 1 -- 172.21.15.59:0/1106899989 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd00743a0 con 0x7f1cd01626d0 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.813+0000 7f7022ffd640 1 -- 172.21.15.59:0/3494931036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7028004800 msgr2=0x7f70300d4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.813+0000 7f7022ffd640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7028004800 0x7f70300d4ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f703806b240 tx=0x7f703806a580 comp rx=0 tx=0).stop 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.813+0000 7f7022ffd640 1 -- 172.21.15.59:0/3494931036 shutdown_connections 2024-06-29T14:21:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.814+0000 7f7022ffd640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70300520f0 0x7f7030036b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.814+0000 7f7022ffd640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70300d2660 0x7f7030050750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.814+0000 7f7022ffd640 1 --2- 172.21.15.59:0/3494931036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7028004800 0x7f70300d4ce0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.814+0000 7f7022ffd640 1 -- 172.21.15.59:0/3494931036 >> 172.21.15.59:0/3494931036 conn(0x7f703001e3f0 msgr2=0x7f7030032730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.813+0000 7faea1391640 1 -- 172.21.15.59:0/3030873556 shutdown_connections 2024-06-29T14:21:29.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.814+0000 7faea1391640 1 -- 172.21.15.59:0/3030873556 wait complete. 2024-06-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.814+0000 7f7022ffd640 1 -- 172.21.15.59:0/3494931036 shutdown_connections 2024-06-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.814+0000 7f7022ffd640 1 -- 172.21.15.59:0/3494931036 wait complete. 2024-06-29T14:21:29.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.815+0000 7f01e75eb640 1 -- 172.21.15.59:0/3559638319 shutdown_connections 2024-06-29T14:21:29.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.817+0000 7faea1391640 1 Processor -- start 2024-06-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.817+0000 7faea1391640 1 -- start start 2024-06-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.820+0000 7faea1391640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae9c0b3800 0x7fae9c145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.818+0000 7f01e75eb640 1 -- 172.21.15.59:0/3559638319 wait complete. 2024-06-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.820+0000 7f01e75eb640 1 Processor -- start 2024-06-29T14:21:29.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.821+0000 7f01e75eb640 1 -- start start 2024-06-29T14:21:29.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.823+0000 7fae9affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae9c0b3800 0x7fae9c145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.823+0000 7fae9affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae9c0b3800 0x7fae9c145b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42456/0 (socket says 172.21.15.59:42456) 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.823+0000 7fae9affd640 1 -- 172.21.15.59:0/2700960762 learned_addr learned my addr 172.21.15.59:0/2700960762 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7faea1391640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0cccc0 0x7fae9c146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7faea1391640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c0cf380 0x7fae9c148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7fae9a7fc640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0cccc0 0x7fae9c146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7fae9b7fe640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c0cf380 0x7fae9c148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c15e270 con 0x7fae9c0b3800 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c15e0f0 con 0x7fae9c0cf380 2024-06-29T14:21:29.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fae9c15e3f0 con 0x7fae9c0cccc0 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.825+0000 7fae9affd640 1 -- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0cccc0 msgr2=0x7fae9c146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.825+0000 7fae9affd640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0cccc0 0x7fae9c146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.825+0000 7fae9affd640 1 -- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c0cf380 msgr2=0x7fae9c148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.825+0000 7fae9affd640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c0cf380 0x7fae9c148920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.825+0000 7fae9affd640 1 -- 172.21.15.59:0/2700960762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9c0abf10 con 0x7fae9c0b3800 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.816+0000 7f1ccf7fe640 1 -- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd0161d80 msgr2=0x7f1cd013b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.816+0000 7f1ccf7fe640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd0161d80 0x7f1cd013b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.816+0000 7f1ccf7fe640 1 -- 172.21.15.59:0/1106899989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cd006e1f0 msgr2=0x7f1cd0154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7f1ccf7fe640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cd006e1f0 0x7f1cd0154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.824+0000 7f1ccf7fe640 1 -- 172.21.15.59:0/1106899989 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd01524d0 con 0x7f1cd01626d0 2024-06-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.825+0000 7f1cceffd640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cd006e1f0 0x7f1cd0154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.826+0000 7fae9affd640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae9c0b3800 0x7fae9c145b10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fae9400f7e0 tx=0x7fae9400fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.826+0000 7fae9a7fc640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0cccc0 0x7fae9c146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:21:29.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.826+0000 7f01e75eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01cc0053c0 0x7f01e0202160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.826+0000 7f1ccf7fe640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd01626d0 0x7f1cd014c750 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1cc800c730 tx=0x7f1cc800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.826+0000 7fae837fe640 1 -- 172.21.15.59:0/2700960762 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9401c070 con 0x7fae9c0b3800 2024-06-29T14:21:29.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7f01e75eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e0070310 0x7f01e01534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7fae837fe640 1 -- 172.21.15.59:0/2700960762 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae9400ca50 con 0x7fae9c0b3800 2024-06-29T14:21:29.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7fae837fe640 1 -- 172.21.15.59:0/2700960762 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae94017430 con 0x7fae9c0b3800 2024-06-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7f01e75eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e0159600 0x7f01e0159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7f01e4b86640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e0070310 0x7f01e01534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7f01e4b86640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e0070310 0x7f01e01534d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:42196/0 (socket says 172.21.15.59:42196) 2024-06-29T14:21:29.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7f01e4b86640 1 -- 172.21.15.59:0/155597184 learned_addr learned my addr 172.21.15.59:0/155597184 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7f01e75eb640 1 -- 172.21.15.59:0/155597184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f01e0073ca0 con 0x7f01e0159600 2024-06-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7f01e75eb640 1 -- 172.21.15.59:0/155597184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01e0073e20 con 0x7f01e0070310 2024-06-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.827+0000 7f01e75eb640 1 -- 172.21.15.59:0/155597184 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f01e0073fa0 con 0x7f01cc0053c0 2024-06-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7f01e4b86640 1 -- 172.21.15.59:0/155597184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01cc0053c0 msgr2=0x7f01e0202160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7f01e4b86640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01cc0053c0 0x7f01e0202160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7f01e4b86640 1 -- 172.21.15.59:0/155597184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e0159600 msgr2=0x7f01e0159a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7f01e5b88640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e0159600 0x7f01e0159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9c0ab150 con 0x7fae9c0b3800 2024-06-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7f01e4b86640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e0159600 0x7f01e0159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7f01e4b86640 1 -- 172.21.15.59:0/155597184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e0159f50 con 0x7f01e0070310 2024-06-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.829+0000 7f01e5b88640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e0159600 0x7f01e0159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.829+0000 7fae837fe640 1 -- 172.21.15.59:0/2700960762 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fae9401f070 con 0x7fae9c0b3800 2024-06-29T14:21:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.829+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae9c0ab710 con 0x7fae9c0b3800 2024-06-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.829+0000 7fae837fe640 1 -- 172.21.15.59:0/2700960762 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fae94017a10 con 0x7fae9c0b3800 2024-06-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.829+0000 7f01e4b86640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e0070310 0x7f01e01534d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f01d800c9e0 tx=0x7f01d800cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7fa19bfff640 1 -- 172.21.15.59:0/2165550899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa19c158620 msgr2=0x7fa19c06c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.828+0000 7fa19bfff640 1 --2- 172.21.15.59:0/2165550899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa19c158620 0x7fa19c06c330 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009830 tx=0x7fa18c02a410 comp rx=0 tx=0).stop 2024-06-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.830+0000 7f1cccff9640 1 -- 172.21.15.59:0/1106899989 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8003b70 con 0x7f1cd01626d0 2024-06-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.831+0000 7f1cccff9640 1 -- 172.21.15.59:0/1106899989 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1cc8003d10 con 0x7f1cd01626d0 2024-06-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.831+0000 7f1cd5847640 1 -- 172.21.15.59:0/1106899989 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd01527b0 con 0x7f1cd01626d0 2024-06-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.832+0000 7f1cccff9640 1 -- 172.21.15.59:0/1106899989 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8005330 con 0x7f1cd01626d0 2024-06-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.832+0000 7f1cccff9640 1 -- 172.21.15.59:0/1106899989 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1cc8005510 con 0x7f1cd01626d0 2024-06-29T14:21:29.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.832+0000 7f1cd5847640 1 -- 172.21.15.59:0/1106899989 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd0152d70 con 0x7f1cd01626d0 2024-06-29T14:21:29.833 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '1'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7f1cccff9640 1 -- 172.21.15.59:0/1106899989 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1cc8012800 con 0x7f1cd01626d0 2024-06-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7fa19bfff640 1 -- 172.21.15.59:0/2165550899 shutdown_connections 2024-06-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7fa19bfff640 1 --2- 172.21.15.59:0/2165550899 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa19c0b47d0 0x7fa19c06f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7fa19bfff640 1 --2- 172.21.15.59:0/2165550899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa19c158620 0x7fa19c06c330 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7fa19bfff640 1 -- 172.21.15.59:0/2165550899 >> 172.21.15.59:0/2165550899 conn(0x7fa19c0b30f0 msgr2=0x7fa19c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7f01d5ffb640 1 -- 172.21.15.59:0/155597184 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8007d20 con 0x7f01e0070310 2024-06-29T14:21:29.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae9c15e970 con 0x7fae9c0b3800 2024-06-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7f01e75eb640 1 -- 172.21.15.59:0/155597184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e02029b0 con 0x7f01e0070310 2024-06-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.834+0000 7f01e75eb640 1 -- 172.21.15.59:0/155597184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e0202f70 con 0x7f01e0070310 2024-06-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.836+0000 7f01d5ffb640 1 -- 172.21.15.59:0/155597184 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01d800f450 con 0x7f01e0070310 2024-06-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.836+0000 7f01d5ffb640 1 -- 172.21.15.59:0/155597184 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d800f8a0 con 0x7f01e0070310 2024-06-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.836+0000 7f1cd5847640 1 -- 172.21.15.59:0/1106899989 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd00ce140 con 0x7f1cd01626d0 2024-06-29T14:21:29.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.838+0000 7fa19bfff640 1 -- 172.21.15.59:0/2165550899 shutdown_connections 2024-06-29T14:21:29.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.838+0000 7fae837fe640 1 -- 172.21.15.59:0/2700960762 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fae94027070 con 0x7fae9c0b3800 2024-06-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.839+0000 7f01d5ffb640 1 -- 172.21.15.59:0/155597184 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f01d800fa80 con 0x7f01e0070310 2024-06-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.839+0000 7f01d5ffb640 1 -- 172.21.15.59:0/155597184 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f01d8021730 con 0x7f01e0070310 2024-06-29T14:21:29.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.839+0000 7f01cb7fe640 1 -- 172.21.15.59:0/155597184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c0005290 con 0x7f01e0070310 2024-06-29T14:21:29.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.839+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fa87c005850 con 0x7fa89c161d80 2024-06-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.841+0000 7f1cccff9640 1 -- 172.21.15.59:0/1106899989 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f1cc8020020 con 0x7f1cd01626d0 2024-06-29T14:21:29.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.842+0000 7f01d5ffb640 1 -- 172.21.15.59:0/155597184 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f01e0159f50 con 0x7f01e0070310 2024-06-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.839+0000 7fa19bfff640 1 -- 172.21.15.59:0/2165550899 wait complete. 2024-06-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.843+0000 7fa19bfff640 1 Processor -- start 2024-06-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.843+0000 7fa19bfff640 1 -- start start 2024-06-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa19c06c870 0x7fa19c0a9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa19c158620 0x7fa19c15d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c08b5b0 0x7fa19c15d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19bfff640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c0b0c10 con 0x7fa19c08b5b0 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19bfff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c0b0a90 con 0x7fa19c06c870 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19bfff640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa19c0b0d90 con 0x7fa19c158620 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c08b5b0 0x7fa19c15d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c08b5b0 0x7fa19c15d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42496/0 (socket says 172.21.15.59:42496) 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.845+0000 7fa19b7fe640 1 -- 172.21.15.59:0/3128841411 learned_addr learned my addr 172.21.15.59:0/3128841411 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.846+0000 7fa19a7fc640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa19c158620 0x7fa19c15d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.846+0000 7fa19affd640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa19c06c870 0x7fa19c0a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.847+0000 7fa19a7fc640 1 -- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa19c06c870 msgr2=0x7fa19c0a9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.847+0000 7fa19a7fc640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa19c06c870 0x7fa19c0a9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.847+0000 7fa19a7fc640 1 -- 172.21.15.59:0/3128841411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c08b5b0 msgr2=0x7fa19c15d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.847+0000 7fa19a7fc640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c08b5b0 0x7fa19c15d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.847+0000 7fa19a7fc640 1 -- 172.21.15.59:0/3128841411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19c0a9a50 con 0x7fa19c158620 2024-06-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.848+0000 7fa19a7fc640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa19c158620 0x7fa19c15d010 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa19400bdf0 tx=0x7fa194009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.848+0000 7f7f52885640 1 -- 172.21.15.59:0/2747006441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f44006000 msgr2=0x7f7f44006410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.848+0000 7f7f52885640 1 --2- 172.21.15.59:0/2747006441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f44006000 0x7f7f44006410 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7f44053510 tx=0x7f7f4c0a6660 comp rx=0 tx=0).stop 2024-06-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f2f32c81640 1 -- 172.21.15.59:0/1569110463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c0cf8f0 msgr2=0x7f2f2c0cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f2f32c81640 1 --2- 172.21.15.59:0/1569110463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c0cf8f0 0x7f2f2c0cfd00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c074220 tx=0x7f2f1c02a560 comp rx=0 tx=0).stop 2024-06-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f2f32c81640 1 -- 172.21.15.59:0/1569110463 shutdown_connections 2024-06-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f2f32c81640 1 --2- 172.21.15.59:0/1569110463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c0cf8f0 0x7f2f2c0cfd00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f2f32c81640 1 --2- 172.21.15.59:0/1569110463 >> v2:172.21.15.153:3300/0 conn(0x7f2f2c0afda0 0x7f2f2c0ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f2f32c81640 1 -- 172.21.15.59:0/1569110463 >> 172.21.15.59:0/1569110463 conn(0x7f2f2c0b2ab0 msgr2=0x7f2f2c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f7f52885640 1 -- 172.21.15.59:0/2747006441 shutdown_connections 2024-06-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f7f52885640 1 --2- 172.21.15.59:0/2747006441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f44006000 0x7f7f44006410 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f7f52885640 1 --2- 172.21.15.59:0/2747006441 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f4401dd80 0x7f7f44053820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f7f52885640 1 -- 172.21.15.59:0/2747006441 >> 172.21.15.59:0/2747006441 conn(0x7f7f440362b0 msgr2=0x7f7f440366c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f7f52885640 1 -- 172.21.15.59:0/2747006441 shutdown_connections 2024-06-29T14:21:29.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.849+0000 7f7f52885640 1 -- 172.21.15.59:0/2747006441 wait complete. 2024-06-29T14:21:29.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.852+0000 7fa898ff9640 1 -- 172.21.15.59:0/1731374113 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v9)=0 v9) ==== 106+0+0 (secure 0 0 0) 0x7fa89c1558c0 con 0x7fa89c161d80 2024-06-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.853+0000 7f7f52885640 1 Processor -- start 2024-06-29T14:21:29.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.853+0000 7f2f32c81640 1 -- 172.21.15.59:0/1569110463 shutdown_connections 2024-06-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.853+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c161d80 msgr2=0x7fa89c0adea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.853+0000 7fa8a2935640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c161d80 0x7fa89c0adea0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa89000b3e0 tx=0x7fa89000b8a0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.854+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 shutdown_connections 2024-06-29T14:21:29.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.854+0000 7fa8a2935640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa89c161d80 0x7fa89c0adea0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.854+0000 7fa8a2935640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa89c073590 0x7fa89c14d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.854+0000 7fa8a2935640 1 --2- 172.21.15.59:0/1731374113 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa89c06e1f0 0x7fa89c160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.854+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 >> 172.21.15.59:0/1731374113 conn(0x7fa89c0ccbe0 msgr2=0x7fa89c162190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.854+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 shutdown_connections 2024-06-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.854+0000 7fa8a2935640 1 -- 172.21.15.59:0/1731374113 wait complete. 2024-06-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.854+0000 7fa1837fe640 1 -- 172.21.15.59:0/3128841411 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194019070 con 0x7fa19c158620 2024-06-29T14:21:29.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.856+0000 7f2f32c81640 1 -- 172.21.15.59:0/1569110463 wait complete. 2024-06-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.856+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19c08a510 con 0x7fa19c158620 2024-06-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.856+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa19c08aad0 con 0x7fa19c158620 2024-06-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.857+0000 7fa1837fe640 1 -- 172.21.15.59:0/3128841411 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa194014440 con 0x7fa19c158620 2024-06-29T14:21:29.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.857+0000 7fa1837fe640 1 -- 172.21.15.59:0/3128841411 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194014870 con 0x7fa19c158620 2024-06-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.859+0000 7fa1837fe640 1 -- 172.21.15.59:0/3128841411 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa19401c070 con 0x7fa19c158620 2024-06-29T14:21:29.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.859+0000 7fa1837fe640 1 -- 172.21.15.59:0/3128841411 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa19401d3b0 con 0x7fa19c158620 2024-06-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.861+0000 7f7f52885640 1 -- start start 2024-06-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f52885640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f44006000 0x7f7f440d1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f52885640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4401dd80 0x7f7f4404fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f52885640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f440dc3f0 0x7f7f44050310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f52885640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7f440e03b0 con 0x7f7f4401dd80 2024-06-29T14:21:29.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f52885640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f440e0230 con 0x7f7f44006000 2024-06-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f52885640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7f44006950 con 0x7f7f440dc3f0 2024-06-29T14:21:29.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa17c005290 con 0x7fa19c158620 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f51082640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4401dd80 0x7f7f4404fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f51082640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4401dd80 0x7f7f4404fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42524/0 (socket says 172.21.15.59:42524) 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.863+0000 7f7f51082640 1 -- 172.21.15.59:0/571796944 learned_addr learned my addr 172.21.15.59:0/571796944 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f7f51883640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f44006000 0x7f7f440d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f7f51082640 1 -- 172.21.15.59:0/571796944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f440dc3f0 msgr2=0x7f7f44050310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f7f51082640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f440dc3f0 0x7f7f44050310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f7f51082640 1 -- 172.21.15.59:0/571796944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f44006000 msgr2=0x7f7f440d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f7f51082640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f44006000 0x7f7f440d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f7f51082640 1 -- 172.21.15.59:0/571796944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4404d2e0 con 0x7f7f4401dd80 2024-06-29T14:21:29.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f7f51883640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f44006000 0x7f7f440d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f2f32c81640 1 Processor -- start 2024-06-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.864+0000 7f7f51082640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4401dd80 0x7f7f4404fdd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c069a10 tx=0x7f7f4c071220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.865+0000 7f7f427fc640 1 -- 172.21.15.59:0/571796944 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c071d50 con 0x7f7f4401dd80 2024-06-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.865+0000 7f7f427fc640 1 -- 172.21.15.59:0/571796944 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f4c0714d0 con 0x7f7f4401dd80 2024-06-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.865+0000 7f7f427fc640 1 -- 172.21.15.59:0/571796944 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0b8620 con 0x7f7f4401dd80 2024-06-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.865+0000 7f2f32c81640 1 -- start start 2024-06-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.866+0000 7f2b329ac640 1 -- 172.21.15.59:0/3248033148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b14004800 msgr2=0x7f2b14004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.866+0000 7f2b329ac640 1 --2- 172.21.15.59:0/3248033148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b14004800 0x7f2b14004c10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c710 tx=0x7f2b14005830 comp rx=0 tx=0).stop 2024-06-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.866+0000 7f7f52885640 1 -- 172.21.15.59:0/571796944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4404d750 con 0x7f7f4401dd80 2024-06-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.866+0000 7f7f52885640 1 -- 172.21.15.59:0/571796944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f4404dd10 con 0x7f7f4401dd80 2024-06-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.867+0000 7f2f32c81640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c0afda0 0x7f2f2c14e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.867+0000 7f2f32c81640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f2c161960 0x7f2f2c154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f32c81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c1550d0 0x7f2f2c155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f2bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f2c161960 0x7f2f2c154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f2bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f2c161960 0x7f2f2c154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:41096/0 (socket says 172.21.15.59:41096) 2024-06-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f2bfff640 1 -- 172.21.15.59:0/1429097785 learned_addr learned my addr 172.21.15.59:0/1429097785 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7fa1837fe640 1 -- 172.21.15.59:0/3128841411 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fa19401e440 con 0x7fa19c158620 2024-06-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c0b4d20 con 0x7f2f2c1550d0 2024-06-29T14:21:29.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c0b4ba0 con 0x7f2f2c0afda0 2024-06-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2f2c0b4ea0 con 0x7f2f2c161960 2024-06-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2b30f49640 1 -- 172.21.15.59:0/3248033148 reap_dead start 2024-06-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2b329ac640 1 -- 172.21.15.59:0/3248033148 shutdown_connections 2024-06-29T14:21:29.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2b329ac640 1 -- 172.21.15.59:0/3248033148 >> 172.21.15.59:0/3248033148 conn(0x7f2b2c0ccbe0 msgr2=0x7f2b2c0ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.869+0000 7f7f427fc640 1 -- 172.21.15.59:0/571796944 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7f4c0b8b50 con 0x7f7f4401dd80 2024-06-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.869+0000 7f7f427fc640 1 -- 172.21.15.59:0/571796944 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7f4c0adbb0 con 0x7f7f4401dd80 2024-06-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f30a1d640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c0afda0 0x7f2f2c14e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.868+0000 7f2f3121e640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c1550d0 0x7f2f2c155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.869+0000 7f2f3121e640 1 -- 172.21.15.59:0/1429097785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f2c161960 msgr2=0x7f2f2c154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.869+0000 7f2f3121e640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f2c161960 0x7f2f2c154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.869+0000 7f2f3121e640 1 -- 172.21.15.59:0/1429097785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c0afda0 msgr2=0x7f2f2c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.869+0000 7f2f3121e640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c0afda0 0x7f2f2c14e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.869+0000 7f2f3121e640 1 -- 172.21.15.59:0/1429097785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c153200 con 0x7f2f2c1550d0 2024-06-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.869+0000 7f2f3121e640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c1550d0 0x7f2f2c155dc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0b5f70 tx=0x7f2f1c004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.870+0000 7f2f2bfff640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f2c161960 0x7f2f2c154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.870+0000 7f7f52885640 1 -- 172.21.15.59:0/571796944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f4400be80 con 0x7f7f4401dd80 2024-06-29T14:21:29.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.872+0000 7f2b329ac640 1 -- 172.21.15.59:0/3248033148 shutdown_connections 2024-06-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.876+0000 7fa433d24640 1 -- 172.21.15.59:0/1319360408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0ceec0 msgr2=0x7fa42c162490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.876+0000 7fa433d24640 1 --2- 172.21.15.59:0/1319360408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0ceec0 0x7fa42c162490 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa424005fd0 tx=0x7fa42402a680 comp rx=0 tx=0).stop 2024-06-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.876+0000 7fa433d24640 1 -- 172.21.15.59:0/1319360408 shutdown_connections 2024-06-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.876+0000 7fa433d24640 1 --2- 172.21.15.59:0/1319360408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0ceec0 0x7fa42c162490 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.876+0000 7fa433d24640 1 -- 172.21.15.59:0/1319360408 >> 172.21.15.59:0/1319360408 conn(0x7fa42c0ccbe0 msgr2=0x7fa42c0ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.872+0000 7f2f297fa640 1 -- 172.21.15.59:0/1429097785 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c041070 con 0x7f2f2c1550d0 2024-06-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.872+0000 7f2f297fa640 1 -- 172.21.15.59:0/1429097785 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f1c003c30 con 0x7f2f2c1550d0 2024-06-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.872+0000 7f2f297fa640 1 -- 172.21.15.59:0/1429097785 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c03b440 con 0x7f2f2c1550d0 2024-06-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.876+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2c1534e0 con 0x7f2f2c1550d0 2024-06-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.877+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2c153aa0 con 0x7f2f2c1550d0 2024-06-29T14:21:29.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.876+0000 7f7f427fc640 1 -- 172.21.15.59:0/571796944 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f7f4c0c1460 con 0x7f7f4401dd80 2024-06-29T14:21:29.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.876+0000 7f2b329ac640 1 -- 172.21.15.59:0/3248033148 wait complete. 2024-06-29T14:21:29.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.879+0000 7f2b329ac640 1 Processor -- start 2024-06-29T14:21:29.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.880+0000 7f2b329ac640 1 -- start start 2024-06-29T14:21:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.881+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f2c0b53c0 con 0x7f2f2c1550d0 2024-06-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.882+0000 7f2b329ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b2c0b3360 0x7f2b2c201730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.882+0000 7f2b329ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b2c201c70 0x7f2b2c0b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.882+0000 7f2b329ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b2c0b1200 0x7f2b2c203920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.882+0000 7f2b329ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c073e10 con 0x7f2b2c201c70 2024-06-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.882+0000 7f2b329ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c073c90 con 0x7f2b2c0b1200 2024-06-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.882+0000 7f2b329ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2b2c073f90 con 0x7f2b2c0b3360 2024-06-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.883+0000 7f2b2bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b2c0b3360 0x7f2b2c201730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.883+0000 7f2b2b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b2c201c70 0x7f2b2c0b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.883+0000 7f2b2bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b2c0b3360 0x7f2b2c201730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:41106/0 (socket says 172.21.15.59:41106) 2024-06-29T14:21:29.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.883+0000 7f2b2bfff640 1 -- 172.21.15.59:0/2207338486 learned_addr learned my addr 172.21.15.59:0/2207338486 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.884 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.884+0000 7f2b2b7fe640 1 -- 172.21.15.59:0/2207338486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b2c0b3360 msgr2=0x7f2b2c201730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.884+0000 7f2b2b7fe640 1 --2- 172.21.15.59:0/2207338486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b2c0b3360 0x7f2b2c201730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.884+0000 7f2b2b7fe640 1 -- 172.21.15.59:0/2207338486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b2c0b1200 msgr2=0x7f2b2c203920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.884+0000 7f2b2b7fe640 1 --2- 172.21.15.59:0/2207338486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b2c0b1200 0x7f2b2c203920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.884+0000 7f2b2b7fe640 1 -- 172.21.15.59:0/2207338486 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2c0b1610 con 0x7f2b2c201c70 2024-06-29T14:21:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.885+0000 7f2b2b7fe640 1 --2- 172.21.15.59:0/2207338486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b2c201c70 0x7f2b2c0b0cc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002f000 tx=0x7f2b20004850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.887+0000 7f2b28ff9640 1 -- 172.21.15.59:0/2207338486 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20004e50 con 0x7f2b2c201c70 2024-06-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.887+0000 7f2b329ac640 1 -- 172.21.15.59:0/2207338486 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2c202140 con 0x7f2b2c201c70 2024-06-29T14:21:29.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.887+0000 7f2b329ac640 1 -- 172.21.15.59:0/2207338486 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b2c203120 con 0x7f2b2c201c70 2024-06-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.888+0000 7f2f297fa640 1 -- 172.21.15.59:0/1429097785 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2f1c044070 con 0x7f2f2c1550d0 2024-06-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.888+0000 7f2f297fa640 1 -- 172.21.15.59:0/1429097785 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f1c0507e0 con 0x7f2f2c1550d0 2024-06-29T14:21:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.889+0000 7f2f297fa640 1 -- 172.21.15.59:0/1429097785 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f2f1c03bd70 con 0x7f2f2c1550d0 2024-06-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.888+0000 7f2b28ff9640 1 -- 172.21.15.59:0/2207338486 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b20042070 con 0x7f2b2c201c70 2024-06-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.888+0000 7f2b28ff9640 1 -- 172.21.15.59:0/2207338486 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b200336b0 con 0x7f2b2c201c70 2024-06-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.888+0000 7f2b28ff9640 1 -- 172.21.15.59:0/2207338486 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2b20033d00 con 0x7f2b2c201c70 2024-06-29T14:21:29.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.888+0000 7f2b28ff9640 1 -- 172.21.15.59:0/2207338486 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2b2003d560 con 0x7f2b2c201c70 2024-06-29T14:21:29.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.892+0000 7f2b127fc640 1 -- 172.21.15.59:0/2207338486 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b0c005290 con 0x7f2b2c201c70 2024-06-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.897+0000 7fa433d24640 1 -- 172.21.15.59:0/1319360408 shutdown_connections 2024-06-29T14:21:29.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.897+0000 7fa433d24640 1 -- 172.21.15.59:0/1319360408 wait complete. 2024-06-29T14:21:29.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.897+0000 7f2b28ff9640 1 -- 172.21.15.59:0/2207338486 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f2b2003d7b0 con 0x7f2b2c201c70 2024-06-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.901+0000 7fa433d24640 1 Processor -- start 2024-06-29T14:21:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.904+0000 7fa433d24640 1 -- start start 2024-06-29T14:21:29.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.906+0000 7fa433d24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c06e1f0 0x7fa42c1501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa431ac0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c06e1f0 0x7fa42c1501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa431ac0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c06e1f0 0x7fa42c1501c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42562/0 (socket says 172.21.15.59:42562) 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa431ac0640 1 -- 172.21.15.59:0/1812756805 learned_addr learned my addr 172.21.15.59:0/1812756805 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa433d24640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c073590 0x7fa42c152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa433d24640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0ceec0 0x7fa42c06f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c15c7c0 con 0x7fa42c06e1f0 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa4312bf640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c073590 0x7fa42c152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa4322c1640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0ceec0 0x7fa42c06f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c0742c0 con 0x7fa42c0ceec0 2024-06-29T14:21:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.907+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa42c15c940 con 0x7fa42c073590 2024-06-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.908+0000 7fa431ac0640 1 -- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c073590 msgr2=0x7fa42c152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.908+0000 7fa431ac0640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c073590 0x7fa42c152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.908+0000 7fa431ac0640 1 -- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0ceec0 msgr2=0x7fa42c06f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.908+0000 7fa431ac0640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0ceec0 0x7fa42c06f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.908+0000 7fa431ac0640 1 -- 172.21.15.59:0/1812756805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c1580c0 con 0x7fa42c06e1f0 2024-06-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.908+0000 7fa431ac0640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c06e1f0 0x7fa42c1501c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa42800c7d0 tx=0x7fa42800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.908+0000 7fa4312bf640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c073590 0x7fa42c152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:21:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.909+0000 7fa4322c1640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0ceec0 0x7fa42c06f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.909+0000 7fa4227fc640 1 -- 172.21.15.59:0/1812756805 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428017070 con 0x7fa42c06e1f0 2024-06-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.909+0000 7fa4227fc640 1 -- 172.21.15.59:0/1812756805 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa428005360 con 0x7fa42c06e1f0 2024-06-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.909+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c1583a0 con 0x7fa42c06e1f0 2024-06-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.909+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c158960 con 0x7fa42c06e1f0 2024-06-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.909+0000 7fa4227fc640 1 -- 172.21.15.59:0/1812756805 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42800a880 con 0x7fa42c06e1f0 2024-06-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.909+0000 7fa4227fc640 1 -- 172.21.15.59:0/1812756805 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa42801a070 con 0x7fa42c06e1f0 2024-06-29T14:21:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.910+0000 7fa4227fc640 1 -- 172.21.15.59:0/1812756805 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa42801c780 con 0x7fa42c06e1f0 2024-06-29T14:21:29.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.920+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa414005290 con 0x7fa42c06e1f0 2024-06-29T14:21:29.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.928+0000 7fa4227fc640 1 -- 172.21.15.59:0/1812756805 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fa4280035f0 con 0x7fa42c06e1f0 2024-06-29T14:21:29.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.936+0000 7f924e7fc640 1 -- 172.21.15.59:0/3432879379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f92600b2090 con 0x7f92600b4310 2024-06-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.944+0000 7f925cff9640 1 -- 172.21.15.59:0/3432879379 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v10)=0 v10) ==== 115+0+0 (secure 0 0 0) 0x7f926006f730 con 0x7f92600b4310 2024-06-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.945+0000 7f924e7fc640 1 -- 172.21.15.59:0/3432879379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92600b4310 msgr2=0x7f92600c6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.945+0000 7f924e7fc640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92600b4310 0x7f92600c6ad0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f925000c760 tx=0x7f925000cc20 comp rx=0 tx=0).stop 2024-06-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.945+0000 7f924e7fc640 1 -- 172.21.15.59:0/3432879379 shutdown_connections 2024-06-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.945+0000 7f924e7fc640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92600c4e50 0x7f9260157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.945+0000 7f924e7fc640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92600b4a30 0x7f9260153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.945+0000 7f924e7fc640 1 --2- 172.21.15.59:0/3432879379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92600b4310 0x7f92600c6ad0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.945+0000 7f924e7fc640 1 -- 172.21.15.59:0/3432879379 >> 172.21.15.59:0/3432879379 conn(0x7f92600b1be0 msgr2=0x7f92600aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.946+0000 7f924e7fc640 1 -- 172.21.15.59:0/3432879379 shutdown_connections 2024-06-29T14:21:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.946+0000 7f924e7fc640 1 -- 172.21.15.59:0/3432879379 wait complete. 2024-06-29T14:21:29.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.962+0000 7f9cd2ffd640 1 -- 172.21.15.59:0/1163787978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f9cc8005580 con 0x7f9ce808b2d0 2024-06-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.968+0000 7f9ce57fa640 1 -- 172.21.15.59:0/1163787978 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v11)=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7f9cd4025030 con 0x7f9ce808b2d0 2024-06-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.970+0000 7f9cd2ffd640 1 -- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 msgr2=0x7f9ce815e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:29.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.970+0000 7f9cd2ffd640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 0x7f9ce815e1e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400f930 tx=0x7f9cd400fdf0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.970 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi059') 2024-06-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.972+0000 7f9cd2ffd640 1 -- 172.21.15.59:0/1163787978 shutdown_connections 2024-06-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.972+0000 7f9cd2ffd640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce816bd00 0x7f9ce815fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.972+0000 7f9cd2ffd640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce80cdef0 0x7f9ce815fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.972+0000 7f9cd2ffd640 1 --2- 172.21.15.59:0/1163787978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce808b2d0 0x7f9ce815e1e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.972+0000 7f9cd2ffd640 1 -- 172.21.15.59:0/1163787978 >> 172.21.15.59:0/1163787978 conn(0x7f9ce80ccbe0 msgr2=0x7f9ce80ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.972+0000 7f9cd2ffd640 1 -- 172.21.15.59:0/1163787978 shutdown_connections 2024-06-29T14:21:29.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.972+0000 7f9cd2ffd640 1 -- 172.21.15.59:0/1163787978 wait complete. 2024-06-29T14:21:29.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:29.993+0000 7f1cd5847640 1 -- 172.21.15.59:0/1106899989 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f1cd0155120 con 0x7f1cd01626d0 2024-06-29T14:21:30.000 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi059') 2024-06-29T14:21:30.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.003+0000 7f1cccff9640 1 -- 172.21.15.59:0/1106899989 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v12)=0 v12) ==== 120+0+0 (secure 0 0 0) 0x7f1cc8012a00 con 0x7f1cd01626d0 2024-06-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.004+0000 7f1cc5ffb640 1 -- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd01626d0 msgr2=0x7f1cd014c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.004+0000 7f1cc5ffb640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd01626d0 0x7f1cd014c750 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1cc800c730 tx=0x7f1cc800cbf0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.005+0000 7f01cb7fe640 1 -- 172.21.15.59:0/155597184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f01c0005580 con 0x7f01e0070310 2024-06-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.005+0000 7f1cc5ffb640 1 -- 172.21.15.59:0/1106899989 shutdown_connections 2024-06-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.005+0000 7f1cc5ffb640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cd01626d0 0x7f1cd014c750 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.005+0000 7f1cc5ffb640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cd0161d80 0x7f1cd013b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.005+0000 7f1cc5ffb640 1 --2- 172.21.15.59:0/1106899989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cd006e1f0 0x7f1cd0154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.005+0000 7f1cc5ffb640 1 -- 172.21.15.59:0/1106899989 >> 172.21.15.59:0/1106899989 conn(0x7f1cd00ccbe0 msgr2=0x7f1cd00ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.005+0000 7f1cc5ffb640 1 -- 172.21.15.59:0/1106899989 shutdown_connections 2024-06-29T14:21:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.005+0000 7f1cc5ffb640 1 -- 172.21.15.59:0/1106899989 wait complete. 2024-06-29T14:21:30.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.013+0000 7f01d5ffb640 1 -- 172.21.15.59:0/155597184 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v13)=0 v13) ==== 125+0+0 (secure 0 0 0) 0x7f01d8014030 con 0x7f01e0070310 2024-06-29T14:21:30.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.015+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fae9c06d850 con 0x7fae9c0b3800 2024-06-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.016+0000 7f01cb7fe640 1 -- 172.21.15.59:0/155597184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e0070310 msgr2=0x7f01e01534d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.016+0000 7f01cb7fe640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e0070310 0x7f01e01534d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f01d800c9e0 tx=0x7f01d800cea0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.017+0000 7f01cb7fe640 1 -- 172.21.15.59:0/155597184 shutdown_connections 2024-06-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.017+0000 7f01cb7fe640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e0159600 0x7f01e0159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.017+0000 7f01cb7fe640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01e0070310 0x7f01e01534d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.017+0000 7f01cb7fe640 1 --2- 172.21.15.59:0/155597184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01cc0053c0 0x7f01e0202160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.017+0000 7f01cb7fe640 1 -- 172.21.15.59:0/155597184 >> 172.21.15.59:0/155597184 conn(0x7f01e00ccbe0 msgr2=0x7f01e00b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.018+0000 7f01cb7fe640 1 -- 172.21.15.59:0/155597184 shutdown_connections 2024-06-29T14:21:30.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.020+0000 7fae837fe640 1 -- 172.21.15.59:0/2700960762 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v14)=0 v14) ==== 119+0+0 (secure 0 0 0) 0x7fae94017c10 con 0x7fae9c0b3800 2024-06-29T14:21:30.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.020+0000 7f01cb7fe640 1 -- 172.21.15.59:0/155597184 wait complete. 2024-06-29T14:21:30.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.026+0000 7f7f52885640 1 -- 172.21.15.59:0/571796944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f7f44032b50 con 0x7f7f4401dd80 2024-06-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae9c0b3800 msgr2=0x7fae9c145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae9c0b3800 0x7fae9c145b10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fae9400f7e0 tx=0x7fae9400fca0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 shutdown_connections 2024-06-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae9c0cf380 0x7fae9c148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae9c0cccc0 0x7fae9c146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 --2- 172.21.15.59:0/2700960762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae9c0b3800 0x7fae9c145b10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 >> 172.21.15.59:0/2700960762 conn(0x7fae9c0b24c0 msgr2=0x7fae9c157300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 shutdown_connections 2024-06-29T14:21:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.028+0000 7faea1391640 1 -- 172.21.15.59:0/2700960762 wait complete. 2024-06-29T14:21:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.033+0000 7f7f427fc640 1 -- 172.21.15.59:0/571796944 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v15)=0 v15) ==== 120+0+0 (secure 0 0 0) 0x7f7f4c0c1640 con 0x7f7f4401dd80 2024-06-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 -- 172.21.15.59:0/571796944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4401dd80 msgr2=0x7f7f4404fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4401dd80 0x7f7f4404fdd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c069a10 tx=0x7f7f4c071220 comp rx=0 tx=0).stop 2024-06-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 -- 172.21.15.59:0/571796944 shutdown_connections 2024-06-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f440dc3f0 0x7f7f44050310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4401dd80 0x7f7f4404fdd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 --2- 172.21.15.59:0/571796944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f44006000 0x7f7f440d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 -- 172.21.15.59:0/571796944 >> 172.21.15.59:0/571796944 conn(0x7f7f440362b0 msgr2=0x7f7f44035730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 -- 172.21.15.59:0/571796944 shutdown_connections 2024-06-29T14:21:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.036+0000 7f7f2ffff640 1 -- 172.21.15.59:0/571796944 wait complete. 2024-06-29T14:21:30.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.058+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fa17c005580 con 0x7fa19c158620 2024-06-29T14:21:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.064+0000 7fa1837fe640 1 -- 172.21.15.59:0/3128841411 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v16)=0 v16) ==== 121+0+0 (secure 0 0 0) 0x7fa19401e620 con 0x7fa19c158620 2024-06-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa19c158620 msgr2=0x7fa19c15d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa19c158620 0x7fa19c15d010 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa19400bdf0 tx=0x7fa194009f70 comp rx=0 tx=0).stop 2024-06-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 shutdown_connections 2024-06-29T14:21:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa19c08b5b0 0x7fa19c15d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa19c158620 0x7fa19c15d010 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 --2- 172.21.15.59:0/3128841411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa19c06c870 0x7fa19c0a9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 >> 172.21.15.59:0/3128841411 conn(0x7fa19c0b30f0 msgr2=0x7fa19c06e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 shutdown_connections 2024-06-29T14:21:30.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.065+0000 7fa19bfff640 1 -- 172.21.15.59:0/3128841411 wait complete. 2024-06-29T14:21:30.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.074+0000 7f2b127fc640 1 -- 172.21.15.59:0/2207338486 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f2b0c005580 con 0x7f2b2c201c70 2024-06-29T14:21:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.077+0000 7f2b28ff9640 1 -- 172.21.15.59:0/2207338486 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v17)=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7f2b2003d990 con 0x7f2b2c201c70 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 -- 172.21.15.59:0/2207338486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b2c201c70 msgr2=0x7f2b2c0b0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 --2- 172.21.15.59:0/2207338486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b2c201c70 0x7f2b2c0b0cc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002f000 tx=0x7f2b20004850 comp rx=0 tx=0).stop 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 -- 172.21.15.59:0/2207338486 shutdown_connections 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 --2- 172.21.15.59:0/2207338486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b2c0b1200 0x7f2b2c203920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 --2- 172.21.15.59:0/2207338486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b2c201c70 0x7f2b2c0b0cc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 --2- 172.21.15.59:0/2207338486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b2c0b3360 0x7f2b2c201730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 -- 172.21.15.59:0/2207338486 >> 172.21.15.59:0/2207338486 conn(0x7f2b2c0ccbe0 msgr2=0x7f2b2c06f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 -- 172.21.15.59:0/2207338486 shutdown_connections 2024-06-29T14:21:30.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.078+0000 7f2b127fc640 1 -- 172.21.15.59:0/2207338486 wait complete. 2024-06-29T14:21:30.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.102+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f2f2c0cef40 con 0x7f2f2c1550d0 2024-06-29T14:21:30.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.104+0000 7f2f297fa640 1 -- 172.21.15.59:0/1429097785 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v18)=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f2f1c031970 con 0x7f2f2c1550d0 2024-06-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c1550d0 msgr2=0x7f2f2c155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c1550d0 0x7f2f2c155dc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0b5f70 tx=0x7f2f1c004550 comp rx=0 tx=0).stop 2024-06-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 shutdown_connections 2024-06-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f2c1550d0 0x7f2f2c155dc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f2c161960 0x7f2f2c154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 --2- 172.21.15.59:0/1429097785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f2c0afda0 0x7f2f2c14e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 >> 172.21.15.59:0/1429097785 conn(0x7f2f2c0b2ab0 msgr2=0x7f2f2c157000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 shutdown_connections 2024-06-29T14:21:30.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.106+0000 7f2f32c81640 1 -- 172.21.15.59:0/1429097785 wait complete. 2024-06-29T14:21:30.120 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi059') 2024-06-29T14:21:30.121 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi059') 2024-06-29T14:21:30.121 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi059') 2024-06-29T14:21:30.121 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi059') 2024-06-29T14:21:30.121 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi059') 2024-06-29T14:21:30.121 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi059') 2024-06-29T14:21:30.121 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi059') 2024-06-29T14:21:30.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.123+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fa414005850 con 0x7fa42c06e1f0 2024-06-29T14:21:30.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.125+0000 7fa4227fc640 1 -- 172.21.15.59:0/1812756805 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v19)=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7fa4280037d0 con 0x7fa42c06e1f0 2024-06-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c06e1f0 msgr2=0x7fa42c1501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c06e1f0 0x7fa42c1501c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa42800c7d0 tx=0x7fa42800cc90 comp rx=0 tx=0).stop 2024-06-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 shutdown_connections 2024-06-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0ceec0 0x7fa42c06f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c073590 0x7fa42c152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 --2- 172.21.15.59:0/1812756805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c06e1f0 0x7fa42c1501c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 >> 172.21.15.59:0/1812756805 conn(0x7fa42c0ccbe0 msgr2=0x7fa42c156690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 shutdown_connections 2024-06-29T14:21:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.126+0000 7fa433d24640 1 -- 172.21.15.59:0/1812756805 wait complete. 2024-06-29T14:21:30.136 INFO:tasks.ceph:Setting crush tunables to default 2024-06-29T14:21:30.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd crush tunables default 2024-06-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.296+0000 7fc6e5d62640 1 -- 172.21.15.59:0/35846614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e00cff60 msgr2=0x7fc6e00af810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.296+0000 7fc6e5d62640 1 --2- 172.21.15.59:0/35846614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e00cff60 0x7fc6e00af810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc6e015f1c0 tx=0x7fc6d002a670 comp rx=0 tx=0).stop 2024-06-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.296+0000 7fc6e5d62640 1 -- 172.21.15.59:0/35846614 shutdown_connections 2024-06-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.296+0000 7fc6e5d62640 1 --2- 172.21.15.59:0/35846614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e00cff60 0x7fc6e00af810 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.296+0000 7fc6e5d62640 1 -- 172.21.15.59:0/35846614 >> 172.21.15.59:0/35846614 conn(0x7fc6e00b2eb0 msgr2=0x7fc6e00cd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.296+0000 7fc6e5d62640 1 -- 172.21.15.59:0/35846614 shutdown_connections 2024-06-29T14:21:30.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.296+0000 7fc6e5d62640 1 -- 172.21.15.59:0/35846614 wait complete. 2024-06-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.298+0000 7fc6e5d62640 1 Processor -- start 2024-06-29T14:21:30.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.298+0000 7fc6e5d62640 1 -- start start 2024-06-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6e5d62640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e00cff60 0x7fc6e014b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6e5d62640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e0158870 0x7fc6e01564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:30.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6e5d62640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e01f2720 0x7fc6e014eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6e5d62640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e015b960 con 0x7fc6e0158870 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6e5d62640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e015b7e0 con 0x7fc6e01f2720 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6e5d62640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e015bae0 con 0x7fc6e00cff60 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e0158870 0x7fc6e01564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e0158870 0x7fc6e01564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42590/0 (socket says 172.21.15.59:42590) 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 -- 172.21.15.59:0/3361811922 learned_addr learned my addr 172.21.15.59:0/3361811922 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6dffff640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e01f2720 0x7fc6e014eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6df7fe640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e00cff60 0x7fc6e014b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 -- 172.21.15.59:0/3361811922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e00cff60 msgr2=0x7fc6e014b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e00cff60 0x7fc6e014b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 -- 172.21.15.59:0/3361811922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e01f2720 msgr2=0x7fc6e014eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e01f2720 0x7fc6e014eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 -- 172.21.15.59:0/3361811922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e01569e0 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.300+0000 7fc6d6ffd640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e0158870 0x7fc6e01564a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c3a0 tx=0x7fc6c800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.301+0000 7fc6dcff9640 1 -- 172.21.15.59:0/3361811922 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8017070 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.301+0000 7fc6dcff9640 1 -- 172.21.15.59:0/3361811922 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6c80035f0 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.301+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e01f3710 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.301+0000 7fc6dcff9640 1 -- 172.21.15.59:0/3361811922 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8003a40 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.301+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e01f2e70 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.301+0000 7fc6dcff9640 1 -- 172.21.15.59:0/3361811922 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc6c801a070 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.301+0000 7fc6dcff9640 1 -- 172.21.15.59:0/3361811922 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc6c8012740 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.302+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e015bf90 con 0x7fc6e0158870 2024-06-29T14:21:30.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.304+0000 7fc6dcff9640 1 -- 172.21.15.59:0/3361811922 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fc6c8021020 con 0x7fc6e0158870 2024-06-29T14:21:30.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.372+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc6e00b4b00 con 0x7fc6e0158870 2024-06-29T14:21:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.373+0000 7fc6dcff9640 1 -- 172.21.15.59:0/3361811922 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fc6e006eea0 con 0x7fc6e0158870 2024-06-29T14:21:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-06-29T14:21:30.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e0158870 msgr2=0x7fc6e01564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e0158870 0x7fc6e01564a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c3a0 tx=0x7fc6c800c860 comp rx=0 tx=0).stop 2024-06-29T14:21:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 shutdown_connections 2024-06-29T14:21:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6e01f2720 0x7fc6e014eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6e0158870 0x7fc6e01564a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 --2- 172.21.15.59:0/3361811922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6e00cff60 0x7fc6e014b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 >> 172.21.15.59:0/3361811922 conn(0x7fc6e00b2eb0 msgr2=0x7fc6e00b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 shutdown_connections 2024-06-29T14:21:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:30.374+0000 7fc6e5d62640 1 -- 172.21.15.59:0/3361811922 wait complete. 2024-06-29T14:21:30.384 INFO:tasks.ceph:check_enable_crimson: False 2024-06-29T14:21:30.384 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-06-29T14:21:30.385 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:30.385 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-06-29T14:21:30.412 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:30.412 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-06-29T14:21:30.478 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:30.479 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-06-29T14:21:30.545 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:21:30.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-06-29T14:21:30.610 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:30.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-06-29T14:21:30.637 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:30.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-06-29T14:21:30.702 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:30.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-06-29T14:21:30.767 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:21:30.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-06-29T14:21:30.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new f02c84e6-3893-40f1-a178-d9b3cafe2129 0 2024-06-29T14:21:30.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.990+0000 7fb1785a5640 1 -- 172.21.15.153:0/264927482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cea30 msgr2=0x7fb17015b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.990+0000 7fb1785a5640 1 --2- 172.21.15.153:0/264927482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cea30 0x7fb17015b660 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb160005fd0 tx=0x7fb16002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.990+0000 7fb1785a5640 1 -- 172.21.15.153:0/264927482 shutdown_connections 2024-06-29T14:21:30.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.990+0000 7fb1785a5640 1 --2- 172.21.15.153:0/264927482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1700ccfe0 0x7fb17015bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.990+0000 7fb1785a5640 1 --2- 172.21.15.153:0/264927482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cea30 0x7fb17015b660 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.990+0000 7fb1785a5640 1 -- 172.21.15.153:0/264927482 >> 172.21.15.153:0/264927482 conn(0x7fb1700b1be0 msgr2=0x7fb1700b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:30.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.990+0000 7fb1785a5640 1 -- 172.21.15.153:0/264927482 shutdown_connections 2024-06-29T14:21:30.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.990+0000 7fb1785a5640 1 -- 172.21.15.153:0/264927482 wait complete. 2024-06-29T14:21:30.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.992+0000 7fb1785a5640 1 Processor -- start 2024-06-29T14:21:30.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.992+0000 7fb1785a5640 1 -- start start 2024-06-29T14:21:30.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb1785a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700cea30 0x7fb170153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:30.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb1785a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cee40 0x7fb1700acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:30.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb1785a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb170143ba0 0x7fb1700ad400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:30.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb1785a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb170151800 con 0x7fb1700cea30 2024-06-29T14:21:30.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb1785a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb170151680 con 0x7fb1700cee40 2024-06-29T14:21:30.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb1785a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb170151980 con 0x7fb170143ba0 2024-06-29T14:21:30.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb175b40640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cee40 0x7fb1700acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb176b42640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb170143ba0 0x7fb1700ad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb175b40640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cee40 0x7fb1700acec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47532/0 (socket says 172.21.15.153:47532) 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb176b42640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb170143ba0 0x7fb1700ad400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50076/0 (socket says 172.21.15.153:50076) 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb175b40640 1 -- 172.21.15.153:0/1372936861 learned_addr learned my addr 172.21.15.153:0/1372936861 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.993+0000 7fb176341640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700cea30 0x7fb170153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb175b40640 1 -- 172.21.15.153:0/1372936861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb170143ba0 msgr2=0x7fb1700ad400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb175b40640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb170143ba0 0x7fb1700ad400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb175b40640 1 -- 172.21.15.153:0/1372936861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700cea30 msgr2=0x7fb170153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb175b40640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700cea30 0x7fb170153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb175b40640 1 -- 172.21.15.153:0/1372936861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17009c6a0 con 0x7fb1700cee40 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb176b42640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb170143ba0 0x7fb1700ad400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb175b40640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cee40 0x7fb1700acec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00c3a0 tx=0x7fb16c00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:30.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb166ffd640 1 -- 172.21.15.153:0/1372936861 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c017070 con 0x7fb1700cee40 2024-06-29T14:21:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb166ffd640 1 -- 172.21.15.153:0/1372936861 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb16c0053c0 con 0x7fb1700cee40 2024-06-29T14:21:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17009cbd0 con 0x7fb1700cee40 2024-06-29T14:21:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb166ffd640 1 -- 172.21.15.153:0/1372936861 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c003850 con 0x7fb1700cee40 2024-06-29T14:21:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb170152e00 con 0x7fb1700cee40 2024-06-29T14:21:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb166ffd640 1 -- 172.21.15.153:0/1372936861 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb16c01a070 con 0x7fb1700cee40 2024-06-29T14:21:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.994+0000 7fb166ffd640 1 -- 172.21.15.153:0/1372936861 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fb16c01c440 con 0x7fb1700cee40 2024-06-29T14:21:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.995+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb17015b660 con 0x7fb1700cee40 2024-06-29T14:21:30.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:30.997+0000 7fb166ffd640 1 -- 172.21.15.153:0/1372936861 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fb16c021020 con 0x7fb1700cee40 2024-06-29T14:21:31.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.070+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f02c84e6-3893-40f1-a178-d9b3cafe2129", "id": 0} v 0) -- 0x7fb17009c8f0 con 0x7fb1700cee40 2024-06-29T14:21:31.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.074+0000 7fb166ffd640 1 -- 172.21.15.153:0/1372936861 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f02c84e6-3893-40f1-a178-d9b3cafe2129", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fb16c011650 con 0x7fb1700cee40 2024-06-29T14:21:31.072 INFO:teuthology.orchestra.run.smithi153.stdout:0 2024-06-29T14:21:31.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cee40 msgr2=0x7fb1700acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cee40 0x7fb1700acec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00c3a0 tx=0x7fb16c00c860 comp rx=0 tx=0).stop 2024-06-29T14:21:31.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 shutdown_connections 2024-06-29T14:21:31.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb170143ba0 0x7fb1700ad400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700cee40 0x7fb1700acec0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 --2- 172.21.15.153:0/1372936861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700cea30 0x7fb170153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 >> 172.21.15.153:0/1372936861 conn(0x7fb1700b1be0 msgr2=0x7fb17015db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 shutdown_connections 2024-06-29T14:21:31.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.075+0000 7fb1785a5640 1 -- 172.21.15.153:0/1372936861 wait complete. 2024-06-29T14:21:31.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 1fd5fe92-f046-4d7f-b0e5-6caecf0c6201 1 2024-06-29T14:21:31.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.199+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3999208377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6806e190 msgr2=0x7fbd6815ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.199+0000 7fbd6f3eb640 1 --2- 172.21.15.153:0/3999208377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6806e190 0x7fbd6815ebb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009830 tx=0x7fbd6402a480 comp rx=0 tx=0).stop 2024-06-29T14:21:31.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.199+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3999208377 shutdown_connections 2024-06-29T14:21:31.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.199+0000 7fbd6f3eb640 1 --2- 172.21.15.153:0/3999208377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6815f0f0 0x7fbd680b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.199+0000 7fbd6f3eb640 1 --2- 172.21.15.153:0/3999208377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6806e190 0x7fbd6815ebb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.199+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3999208377 >> 172.21.15.153:0/3999208377 conn(0x7fbd68070100 msgr2=0x7fbd68070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.199+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3999208377 shutdown_connections 2024-06-29T14:21:31.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.199+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3999208377 wait complete. 2024-06-29T14:21:31.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.201+0000 7fbd6f3eb640 1 Processor -- start 2024-06-29T14:21:31.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.201+0000 7fbd6f3eb640 1 -- start start 2024-06-29T14:21:31.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6f3eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6815b150 0x7fbd680ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6f3eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd6815f0f0 0x7fbd681528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6f3eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd68134720 0x7fbd680b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6f3eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6815f680 con 0x7fbd6815f0f0 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6f3eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6815f500 con 0x7fbd68134720 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6f3eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbd680b5af0 con 0x7fbd6815b150 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6d187640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6815b150 0x7fbd680ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6d187640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6815b150 0x7fbd680ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50086/0 (socket says 172.21.15.153:50086) 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6d187640 1 -- 172.21.15.153:0/3691150737 learned_addr learned my addr 172.21.15.153:0/3691150737 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6d988640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd68134720 0x7fbd680b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.202+0000 7fbd6c986640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd6815f0f0 0x7fbd681528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd6d187640 1 -- 172.21.15.153:0/3691150737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd68134720 msgr2=0x7fbd680b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd6d187640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd68134720 0x7fbd680b4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd6d187640 1 -- 172.21.15.153:0/3691150737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd6815f0f0 msgr2=0x7fbd681528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd6d187640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd6815f0f0 0x7fbd681528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd6d187640 1 -- 172.21.15.153:0/3691150737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd681f4830 con 0x7fbd6815b150 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd6d187640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6815b150 0x7fbd680ad8b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800c3a0 tx=0x7fbd5800c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd5dffb640 1 -- 172.21.15.153:0/3691150737 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58017070 con 0x7fbd6815b150 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd5dffb640 1 -- 172.21.15.153:0/3691150737 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd580053c0 con 0x7fbd6815b150 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd680ae100 con 0x7fbd6815b150 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd5dffb640 1 -- 172.21.15.153:0/3691150737 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58003850 con 0x7fbd6815b150 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd680ae6c0 con 0x7fbd6815b150 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd5dffb640 1 -- 172.21.15.153:0/3691150737 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbd5801a070 con 0x7fbd6815b150 2024-06-29T14:21:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.203+0000 7fbd5dffb640 1 -- 172.21.15.153:0/3691150737 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fbd58011cf0 con 0x7fbd6815b150 2024-06-29T14:21:31.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.204+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd680b6010 con 0x7fbd6815b150 2024-06-29T14:21:31.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.207+0000 7fbd5dffb640 1 -- 172.21.15.153:0/3691150737 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fbd58021070 con 0x7fbd6815b150 2024-06-29T14:21:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.279+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1fd5fe92-f046-4d7f-b0e5-6caecf0c6201", "id": 1} v 0) -- 0x7fbd680b3b70 con 0x7fbd6815b150 2024-06-29T14:21:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.283+0000 7fbd5dffb640 1 -- 172.21.15.153:0/3691150737 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1fd5fe92-f046-4d7f-b0e5-6caecf0c6201", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fbd58011590 con 0x7fbd6815b150 2024-06-29T14:21:31.281 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6815b150 msgr2=0x7fbd680ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6815b150 0x7fbd680ad8b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800c3a0 tx=0x7fbd5800c860 comp rx=0 tx=0).stop 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 shutdown_connections 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd68134720 0x7fbd680b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd6815f0f0 0x7fbd681528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 --2- 172.21.15.153:0/3691150737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6815b150 0x7fbd680ad8b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 >> 172.21.15.153:0/3691150737 conn(0x7fbd68070100 msgr2=0x7fbd680725c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 shutdown_connections 2024-06-29T14:21:31.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.284+0000 7fbd6f3eb640 1 -- 172.21.15.153:0/3691150737 wait complete. 2024-06-29T14:21:31.297 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new e9c69d5d-fbda-4514-a952-731694e1abb7 2 2024-06-29T14:21:31.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 -- 172.21.15.153:0/1776461078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815c5a0 msgr2=0x7f5aa815c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 --2- 172.21.15.153:0/1776461078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815c5a0 0x7f5aa815c9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5aa806f380 tx=0x7f5aa002aa30 comp rx=0 tx=0).stop 2024-06-29T14:21:31.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 -- 172.21.15.153:0/1776461078 shutdown_connections 2024-06-29T14:21:31.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 --2- 172.21.15.153:0/1776461078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815c5a0 0x7f5aa815c9b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 --2- 172.21.15.153:0/1776461078 >> v2:172.21.15.153:3300/0 conn(0x7f5aa80b5320 0x7f5aa80b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 --2- 172.21.15.153:0/1776461078 >> v2:172.21.15.59:3300/0 conn(0x7f5aa815d580 0x7f5aa8142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 -- 172.21.15.153:0/1776461078 >> 172.21.15.153:0/1776461078 conn(0x7f5aa8073e50 msgr2=0x7f5aa80b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 -- 172.21.15.153:0/1776461078 shutdown_connections 2024-06-29T14:21:31.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.413+0000 7f5aaf21f640 1 -- 172.21.15.153:0/1776461078 wait complete. 2024-06-29T14:21:31.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.415+0000 7f5aaf21f640 1 Processor -- start 2024-06-29T14:21:31.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.415+0000 7f5aaf21f640 1 -- start start 2024-06-29T14:21:31.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aaf21f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5aa80b5320 0x7f5aa80cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aaf21f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aa815c5a0 0x7f5aa81f8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aaf21f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815d580 0x7f5aa81f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aaf21f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa806e4e0 con 0x7f5aa80b5320 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aaf21f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa806e360 con 0x7f5aa815c5a0 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aaf21f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa806e660 con 0x7f5aa815d580 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aa7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aa815c5a0 0x7f5aa81f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aa7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aa815c5a0 0x7f5aa81f8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47554/0 (socket says 172.21.15.153:47554) 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aa7fff640 1 -- 172.21.15.153:0/4026631107 learned_addr learned my addr 172.21.15.153:0/4026631107 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aad7bc640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815d580 0x7f5aa81f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.416+0000 7f5aacfbb640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5aa80b5320 0x7f5aa80cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aad7bc640 1 -- 172.21.15.153:0/4026631107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aa815c5a0 msgr2=0x7f5aa81f8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aad7bc640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aa815c5a0 0x7f5aa81f8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aad7bc640 1 -- 172.21.15.153:0/4026631107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5aa80b5320 msgr2=0x7f5aa80cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aad7bc640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5aa80b5320 0x7f5aa80cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aad7bc640 1 -- 172.21.15.153:0/4026631107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa80c9d80 con 0x7f5aa815d580 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aacfbb640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5aa80b5320 0x7f5aa80cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aad7bc640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815d580 0x7f5aa81f8880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80b5e20 tx=0x7f5aa0003ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aa57fa640 1 -- 172.21.15.153:0/4026631107 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0033050 con 0x7f5aa815d580 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aa57fa640 1 -- 172.21.15.153:0/4026631107 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5aa0004450 con 0x7f5aa815d580 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa80ca060 con 0x7f5aa815d580 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aa57fa640 1 -- 172.21.15.153:0/4026631107 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0004840 con 0x7f5aa815d580 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa80ca620 con 0x7f5aa815d580 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aa57fa640 1 -- 172.21.15.153:0/4026631107 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5aa0036070 con 0x7f5aa815d580 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.417+0000 7f5aa57fa640 1 -- 172.21.15.153:0/4026631107 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f5aa0038db0 con 0x7f5aa815d580 2024-06-29T14:21:31.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.418+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa815c9b0 con 0x7f5aa815d580 2024-06-29T14:21:31.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.420+0000 7f5aa57fa640 1 -- 172.21.15.153:0/4026631107 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f5aa0062030 con 0x7f5aa815d580 2024-06-29T14:21:31.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.493+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e9c69d5d-fbda-4514-a952-731694e1abb7", "id": 2} v 0) -- 0x7f5aa80745e0 con 0x7f5aa815d580 2024-06-29T14:21:31.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.501+0000 7f5aa57fa640 1 -- 172.21.15.153:0/4026631107 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e9c69d5d-fbda-4514-a952-731694e1abb7", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f5aa0050030 con 0x7f5aa815d580 2024-06-29T14:21:31.500 INFO:teuthology.orchestra.run.smithi153.stdout:2 2024-06-29T14:21:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815d580 msgr2=0x7f5aa81f8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815d580 0x7f5aa81f8880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80b5e20 tx=0x7f5aa0003ef0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 shutdown_connections 2024-06-29T14:21:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5aa815d580 0x7f5aa81f8880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aa815c5a0 0x7f5aa81f8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 --2- 172.21.15.153:0/4026631107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5aa80b5320 0x7f5aa80cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 >> 172.21.15.153:0/4026631107 conn(0x7f5aa8073e50 msgr2=0x7f5aa8143350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 shutdown_connections 2024-06-29T14:21:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.502+0000 7f5aaf21f640 1 -- 172.21.15.153:0/4026631107 wait complete. 2024-06-29T14:21:31.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new aea314fe-1780-4b91-8f6c-b2b5125046b3 3 2024-06-29T14:21:31.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.628+0000 7f5c38fe0640 1 -- 172.21.15.153:0/2477529087 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 msgr2=0x7f5c34073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.628+0000 7f5c38fe0640 1 --2- 172.21.15.153:0/2477529087 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 0x7f5c34073d40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5c340cf9b0 tx=0x7f5c2802a560 comp rx=0 tx=0).stop 2024-06-29T14:21:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.629+0000 7f5c38fe0640 1 -- 172.21.15.153:0/2477529087 shutdown_connections 2024-06-29T14:21:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.629+0000 7f5c38fe0640 1 --2- 172.21.15.153:0/2477529087 >> v2:172.21.15.153:3300/0 conn(0x7f5c3413dda0 0x7f5c34074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.629+0000 7f5c38fe0640 1 --2- 172.21.15.153:0/2477529087 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 0x7f5c34073d40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.629+0000 7f5c38fe0640 1 --2- 172.21.15.153:0/2477529087 >> v2:172.21.15.59:3300/0 conn(0x7f5c3415b2c0 0x7f5c34140d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.629+0000 7f5c38fe0640 1 -- 172.21.15.153:0/2477529087 >> 172.21.15.153:0/2477529087 conn(0x7f5c340cd4c0 msgr2=0x7f5c340cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.629+0000 7f5c38fe0640 1 -- 172.21.15.153:0/2477529087 shutdown_connections 2024-06-29T14:21:31.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.629+0000 7f5c38fe0640 1 -- 172.21.15.153:0/2477529087 wait complete. 2024-06-29T14:21:31.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.631+0000 7f5c38fe0640 1 Processor -- start 2024-06-29T14:21:31.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.631+0000 7f5c38fe0640 1 -- start start 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c38fe0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 0x7f5c34137180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c38fe0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c3413dda0 0x7f5c3415a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c38fe0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c3415b2c0 0x7f5c3415a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c38fe0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3415f2d0 con 0x7f5c3415b2c0 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c38fe0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3415f150 con 0x7f5c3413dda0 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c38fe0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3415f450 con 0x7f5c3413d990 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c3259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 0x7f5c34137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c31d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c3413dda0 0x7f5c3415a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c31d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c3413dda0 0x7f5c3415a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47576/0 (socket says 172.21.15.153:47576) 2024-06-29T14:21:31.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c3259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 0x7f5c34137180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50124/0 (socket says 172.21.15.153:50124) 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c31d9b640 1 -- 172.21.15.153:0/3613147197 learned_addr learned my addr 172.21.15.153:0/3613147197 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c32d9d640 1 --2- 172.21.15.153:0/3613147197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c3415b2c0 0x7f5c3415a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c31d9b640 1 -- 172.21.15.153:0/3613147197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 msgr2=0x7f5c34137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c31d9b640 1 --2- 172.21.15.153:0/3613147197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 0x7f5c34137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c31d9b640 1 -- 172.21.15.153:0/3613147197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c3415b2c0 msgr2=0x7f5c3415a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c31d9b640 1 --2- 172.21.15.153:0/3613147197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c3415b2c0 0x7f5c3415a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.632+0000 7f5c31d9b640 1 -- 172.21.15.153:0/3613147197 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c34155c10 con 0x7f5c3413dda0 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.633+0000 7f5c31d9b640 1 --2- 172.21.15.153:0/3613147197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c3413dda0 0x7f5c3415a200 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c28000c00 tx=0x7f5c280038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.633+0000 7f5c22ffd640 1 -- 172.21.15.153:0/3613147197 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2803e030 con 0x7f5c3413dda0 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.633+0000 7f5c22ffd640 1 -- 172.21.15.153:0/3613147197 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c28003ca0 con 0x7f5c3413dda0 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.633+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c34155ef0 con 0x7f5c3413dda0 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.633+0000 7f5c22ffd640 1 -- 172.21.15.153:0/3613147197 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28035040 con 0x7f5c3413dda0 2024-06-29T14:21:31.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.633+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c341564b0 con 0x7f5c3413dda0 2024-06-29T14:21:31.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.633+0000 7f5c22ffd640 1 -- 172.21.15.153:0/3613147197 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5c28041030 con 0x7f5c3413dda0 2024-06-29T14:21:31.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.633+0000 7f5c22ffd640 1 -- 172.21.15.153:0/3613147197 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f5c28043b20 con 0x7f5c3413dda0 2024-06-29T14:21:31.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.634+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c34140d10 con 0x7f5c3413dda0 2024-06-29T14:21:31.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.638+0000 7f5c22ffd640 1 -- 172.21.15.153:0/3613147197 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f5c28043e00 con 0x7f5c3413dda0 2024-06-29T14:21:31.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.710+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd new", "uuid": "aea314fe-1780-4b91-8f6c-b2b5125046b3", "id": 3} v 0) -- 0x7f5c340697f0 con 0x7f5c3413dda0 2024-06-29T14:21:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.714+0000 7f5c22ffd640 1 -- 172.21.15.153:0/3613147197 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aea314fe-1780-4b91-8f6c-b2b5125046b3", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f5c28004ee0 con 0x7f5c3413dda0 2024-06-29T14:21:31.713 INFO:teuthology.orchestra.run.smithi153.stdout:3 2024-06-29T14:21:31.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c3413dda0 msgr2=0x7f5c3415a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 --2- 172.21.15.153:0/3613147197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c3413dda0 0x7f5c3415a200 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c28000c00 tx=0x7f5c280038a0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 shutdown_connections 2024-06-29T14:21:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 --2- 172.21.15.153:0/3613147197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c3415b2c0 0x7f5c3415a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 --2- 172.21.15.153:0/3613147197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c3413dda0 0x7f5c3415a200 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 --2- 172.21.15.153:0/3613147197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c3413d990 0x7f5c34137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 >> 172.21.15.153:0/3613147197 conn(0x7f5c340cd4c0 msgr2=0x7f5c340cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 shutdown_connections 2024-06-29T14:21:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.715+0000 7f5c38fe0640 1 -- 172.21.15.153:0/3613147197 wait complete. 2024-06-29T14:21:31.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 5532cdeb-7ff7-4174-bf7a-7f3f90cc4009 4 2024-06-29T14:21:31.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.841+0000 7f0500fc7640 1 -- 172.21.15.153:0/3880809638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04fc0cf4d0 msgr2=0x7f04fc0cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.841+0000 7f0500fc7640 1 --2- 172.21.15.153:0/3880809638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04fc0cf4d0 0x7f04fc0cce70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f04f0005fd0 tx=0x7f04f002a540 comp rx=0 tx=0).stop 2024-06-29T14:21:31.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.841+0000 7f0500fc7640 1 -- 172.21.15.153:0/3880809638 shutdown_connections 2024-06-29T14:21:31.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.841+0000 7f0500fc7640 1 --2- 172.21.15.153:0/3880809638 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04fc0b2ee0 0x7f04fc0b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.841+0000 7f0500fc7640 1 --2- 172.21.15.153:0/3880809638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04fc0cf4d0 0x7f04fc0cce70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.841+0000 7f0500fc7640 1 -- 172.21.15.153:0/3880809638 >> 172.21.15.153:0/3880809638 conn(0x7f04fc0b1be0 msgr2=0x7f04fc0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.841+0000 7f0500fc7640 1 -- 172.21.15.153:0/3880809638 shutdown_connections 2024-06-29T14:21:31.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.841+0000 7f0500fc7640 1 -- 172.21.15.153:0/3880809638 wait complete. 2024-06-29T14:21:31.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.843+0000 7f0500fc7640 1 Processor -- start 2024-06-29T14:21:31.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.843+0000 7f0500fc7640 1 -- start start 2024-06-29T14:21:31.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f0500fc7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04fc073120 0x7f04fc09aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f0500fc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04fc0b2ee0 0x7f04fc1f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f0500fc7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04fc0cf4d0 0x7f04fc0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f0500fc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc15faa0 con 0x7f04fc0b2ee0 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f0500fc7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc15f920 con 0x7f04fc0cf4d0 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f0500fc7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f04fc15fc20 con 0x7f04fc073120 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04fc073120 0x7f04fc09aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04fc0cf4d0 0x7f04fc0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04fc073120 0x7f04fc09aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50140/0 (socket says 172.21.15.153:50140) 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 -- 172.21.15.153:0/1308554148 learned_addr learned my addr 172.21.15.153:0/1308554148 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04f9d9b640 1 --2- 172.21.15.153:0/1308554148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04fc0b2ee0 0x7f04fc1f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 -- 172.21.15.153:0/1308554148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04fc0cf4d0 msgr2=0x7f04fc0ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 --2- 172.21.15.153:0/1308554148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04fc0cf4d0 0x7f04fc0ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 -- 172.21.15.153:0/1308554148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04fc0b2ee0 msgr2=0x7f04fc1f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 --2- 172.21.15.153:0/1308554148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04fc0b2ee0 0x7f04fc1f4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 -- 172.21.15.153:0/1308554148 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04fc0ca6c0 con 0x7f04fc073120 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04fa59c640 1 --2- 172.21.15.153:0/1308554148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04fc073120 0x7f04fc09aa40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f04f0006010 tx=0x7f04f00038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04eaffd640 1 -- 172.21.15.153:0/1308554148 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f003e030 con 0x7f04fc073120 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04eaffd640 1 -- 172.21.15.153:0/1308554148 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04f0003d90 con 0x7f04fc073120 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04fc15c960 con 0x7f04fc073120 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.845+0000 7f04eaffd640 1 -- 172.21.15.153:0/1308554148 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f0035040 con 0x7f04fc073120 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.846+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04fc15cf20 con 0x7f04fc073120 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.846+0000 7f04eaffd640 1 -- 172.21.15.153:0/1308554148 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f04f0041030 con 0x7f04fc073120 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.846+0000 7f04eaffd640 1 -- 172.21.15.153:0/1308554148 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f04f0043c40 con 0x7f04fc073120 2024-06-29T14:21:31.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.847+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04fc0b32f0 con 0x7f04fc073120 2024-06-29T14:21:31.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.849+0000 7f04eaffd640 1 -- 172.21.15.153:0/1308554148 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f04f0004f50 con 0x7f04fc073120 2024-06-29T14:21:31.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.920+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5532cdeb-7ff7-4174-bf7a-7f3f90cc4009", "id": 4} v 0) -- 0x7f04fc15f410 con 0x7f04fc073120 2024-06-29T14:21:31.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.924+0000 7f04eaffd640 1 -- 172.21.15.153:0/1308554148 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5532cdeb-7ff7-4174-bf7a-7f3f90cc4009", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f04f0043730 con 0x7f04fc073120 2024-06-29T14:21:31.923 INFO:teuthology.orchestra.run.smithi153.stdout:4 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04fc073120 msgr2=0x7f04fc09aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 --2- 172.21.15.153:0/1308554148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04fc073120 0x7f04fc09aa40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f04f0006010 tx=0x7f04f00038a0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 shutdown_connections 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 --2- 172.21.15.153:0/1308554148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04fc0cf4d0 0x7f04fc0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 --2- 172.21.15.153:0/1308554148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04fc0b2ee0 0x7f04fc1f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 --2- 172.21.15.153:0/1308554148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04fc073120 0x7f04fc09aa40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 >> 172.21.15.153:0/1308554148 conn(0x7f04fc0b1be0 msgr2=0x7f04fc0ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 shutdown_connections 2024-06-29T14:21:31.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:31.925+0000 7f0500fc7640 1 -- 172.21.15.153:0/1308554148 wait complete. 2024-06-29T14:21:31.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new aea8df17-2e12-4559-bd03-8c496f34482c 5 2024-06-29T14:21:32.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.050+0000 7f8d147bd640 1 -- 172.21.15.153:0/593000526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 msgr2=0x7f8cf8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.050+0000 7f8d147bd640 1 --2- 172.21.15.153:0/593000526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 0x7f8cf8003f30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc00c290 tx=0x7f8cf8005830 comp rx=0 tx=0).stop 2024-06-29T14:21:32.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.050+0000 7f8d147bd640 1 -- 172.21.15.153:0/593000526 shutdown_connections 2024-06-29T14:21:32.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.050+0000 7f8d147bd640 1 --2- 172.21.15.153:0/593000526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 0x7f8cf8003f30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.050+0000 7f8d147bd640 1 -- 172.21.15.153:0/593000526 >> 172.21.15.153:0/593000526 conn(0x7f8d0c072d80 msgr2=0x7f8d0c073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:32.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.050+0000 7f8d147bd640 1 -- 172.21.15.153:0/593000526 shutdown_connections 2024-06-29T14:21:32.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.050+0000 7f8d147bd640 1 -- 172.21.15.153:0/593000526 wait complete. 2024-06-29T14:21:32.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.052+0000 7f8d147bd640 1 Processor -- start 2024-06-29T14:21:32.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.052+0000 7f8d147bd640 1 -- start start 2024-06-29T14:21:32.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d147bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 0x7f8d0c1efd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d147bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d0c13c870 0x7f8d0c1f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d147bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d0c1f62c0 0x7f8d0c1f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d147bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c0b3e00 con 0x7f8d0c1f62c0 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d147bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c0b3c80 con 0x7f8cf8003ac0 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d147bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8d0c0b3f80 con 0x7f8d0c13c870 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d12559640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 0x7f8d0c1efd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d11d58640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d0c13c870 0x7f8d0c1f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d12559640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 0x7f8d0c1efd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47606/0 (socket says 172.21.15.153:47606) 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d12559640 1 -- 172.21.15.153:0/377902034 learned_addr learned my addr 172.21.15.153:0/377902034 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d12d5a640 1 --2- 172.21.15.153:0/377902034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d0c1f62c0 0x7f8d0c1f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d11d58640 1 -- 172.21.15.153:0/377902034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 msgr2=0x7f8d0c1efd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d11d58640 1 --2- 172.21.15.153:0/377902034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 0x7f8d0c1efd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d11d58640 1 -- 172.21.15.153:0/377902034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d0c1f62c0 msgr2=0x7f8d0c1f3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d11d58640 1 --2- 172.21.15.153:0/377902034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d0c1f62c0 0x7f8d0c1f3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d11d58640 1 -- 172.21.15.153:0/377902034 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0c13e2f0 con 0x7f8d0c13c870 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d11d58640 1 --2- 172.21.15.153:0/377902034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d0c13c870 0x7f8d0c1f5d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8d08003d60 tx=0x7f8d08001630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d02ffd640 1 -- 172.21.15.153:0/377902034 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08001ce0 con 0x7f8d0c13c870 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d02ffd640 1 -- 172.21.15.153:0/377902034 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d08016020 con 0x7f8d0c13c870 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0c1f66d0 con 0x7f8d0c13c870 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d02ffd640 1 -- 172.21.15.153:0/377902034 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08011670 con 0x7f8d0c13c870 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.054+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0c13d530 con 0x7f8d0c13c870 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.055+0000 7f8d02ffd640 1 -- 172.21.15.153:0/377902034 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d08011ba0 con 0x7f8d0c13c870 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.055+0000 7f8d02ffd640 1 -- 172.21.15.153:0/377902034 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f8d0801b990 con 0x7f8d0c13c870 2024-06-29T14:21:32.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.056+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d0c06d090 con 0x7f8d0c13c870 2024-06-29T14:21:32.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.058+0000 7f8d02ffd640 1 -- 172.21.15.153:0/377902034 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f8d08020020 con 0x7f8d0c13c870 2024-06-29T14:21:32.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.130+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd new", "uuid": "aea8df17-2e12-4559-bd03-8c496f34482c", "id": 5} v 0) -- 0x7f8d0c13cf70 con 0x7f8d0c13c870 2024-06-29T14:21:32.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.135+0000 7f8d02ffd640 1 -- 172.21.15.153:0/377902034 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aea8df17-2e12-4559-bd03-8c496f34482c", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f8d0c1f66d0 con 0x7f8d0c13c870 2024-06-29T14:21:32.133 INFO:teuthology.orchestra.run.smithi153.stdout:5 2024-06-29T14:21:32.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.135+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d0c13c870 msgr2=0x7f8d0c1f5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.135+0000 7f8d147bd640 1 --2- 172.21.15.153:0/377902034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d0c13c870 0x7f8d0c1f5d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8d08003d60 tx=0x7f8d08001630 comp rx=0 tx=0).stop 2024-06-29T14:21:32.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.136+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 shutdown_connections 2024-06-29T14:21:32.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.136+0000 7f8d147bd640 1 --2- 172.21.15.153:0/377902034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d0c1f62c0 0x7f8d0c1f3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.136+0000 7f8d147bd640 1 --2- 172.21.15.153:0/377902034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d0c13c870 0x7f8d0c1f5d80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.136+0000 7f8d147bd640 1 --2- 172.21.15.153:0/377902034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cf8003ac0 0x7f8d0c1efd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.136+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 >> 172.21.15.153:0/377902034 conn(0x7f8d0c072d80 msgr2=0x7f8d0c15d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:32.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.136+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 shutdown_connections 2024-06-29T14:21:32.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.136+0000 7f8d147bd640 1 -- 172.21.15.153:0/377902034 wait complete. 2024-06-29T14:21:32.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 737193be-7224-4e5f-9857-bc0349f5a29a 6 2024-06-29T14:21:32.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.261+0000 7f875332a640 1 -- 172.21.15.153:0/13986844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15c5a0 msgr2=0x7f874c15c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.261+0000 7f875332a640 1 --2- 172.21.15.153:0/13986844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15c5a0 0x7f874c15c9b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8744005fd0 tx=0x7f874402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.261+0000 7f875332a640 1 -- 172.21.15.153:0/13986844 shutdown_connections 2024-06-29T14:21:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.261+0000 7f875332a640 1 --2- 172.21.15.153:0/13986844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15c5a0 0x7f874c15c9b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.261+0000 7f875332a640 1 --2- 172.21.15.153:0/13986844 >> v2:172.21.15.59:3300/0 conn(0x7f874c0b5320 0x7f874c0b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.261+0000 7f875332a640 1 -- 172.21.15.153:0/13986844 >> 172.21.15.153:0/13986844 conn(0x7f874c073e50 msgr2=0x7f874c0b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.262+0000 7f875332a640 1 -- 172.21.15.153:0/13986844 shutdown_connections 2024-06-29T14:21:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.262+0000 7f875332a640 1 -- 172.21.15.153:0/13986844 wait complete. 2024-06-29T14:21:32.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.263+0000 7f875332a640 1 Processor -- start 2024-06-29T14:21:32.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.263+0000 7f875332a640 1 -- start start 2024-06-29T14:21:32.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f875332a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874c0b5320 0x7f874c153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f875332a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874c15c5a0 0x7f874c1f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f875332a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15d580 0x7f874c1450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f875332a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f874c06e6f0 con 0x7f874c15c5a0 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f875332a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f874c06e570 con 0x7f874c0b5320 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f875332a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f874c06e870 con 0x7f874c15d580 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87510c6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874c0b5320 0x7f874c153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87518c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15d580 0x7f874c1450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87518c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15d580 0x7f874c1450f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50176/0 (socket says 172.21.15.153:50176) 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87518c7640 1 -- 172.21.15.153:0/3661948696 learned_addr learned my addr 172.21.15.153:0/3661948696 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87508c5640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874c15c5a0 0x7f874c1f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87518c7640 1 -- 172.21.15.153:0/3661948696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874c0b5320 msgr2=0x7f874c153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87518c7640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874c0b5320 0x7f874c153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87518c7640 1 -- 172.21.15.153:0/3661948696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874c15c5a0 msgr2=0x7f874c1f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87518c7640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874c15c5a0 0x7f874c1f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87518c7640 1 -- 172.21.15.153:0/3661948696 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874c0c6bd0 con 0x7f874c15d580 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.265+0000 7f87510c6640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874c0b5320 0x7f874c153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:21:32.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.266+0000 7f87518c7640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15d580 0x7f874c1450f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8744006010 tx=0x7f8744031650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:32.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.266+0000 7f8741ffb640 1 -- 172.21.15.153:0/3661948696 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744003e00 con 0x7f874c15d580 2024-06-29T14:21:32.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.266+0000 7f8741ffb640 1 -- 172.21.15.153:0/3661948696 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8744042070 con 0x7f874c15d580 2024-06-29T14:21:32.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.266+0000 7f8741ffb640 1 -- 172.21.15.153:0/3661948696 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744031d00 con 0x7f874c15d580 2024-06-29T14:21:32.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.266+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f874c153ad0 con 0x7f874c15d580 2024-06-29T14:21:32.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.267+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f874c1f6e80 con 0x7f874c15d580 2024-06-29T14:21:32.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.267+0000 7f8741ffb640 1 -- 172.21.15.153:0/3661948696 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f874403eb40 con 0x7f874c15d580 2024-06-29T14:21:32.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.267+0000 7f8741ffb640 1 -- 172.21.15.153:0/3661948696 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f8744004630 con 0x7f874c15d580 2024-06-29T14:21:32.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.268+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f874c06ed40 con 0x7f874c15d580 2024-06-29T14:21:32.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.270+0000 7f8741ffb640 1 -- 172.21.15.153:0/3661948696 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f8744033070 con 0x7f874c15d580 2024-06-29T14:21:32.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.341+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd new", "uuid": "737193be-7224-4e5f-9857-bc0349f5a29a", "id": 6} v 0) -- 0x7f874c074260 con 0x7f874c15d580 2024-06-29T14:21:32.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.345+0000 7f8741ffb640 1 -- 172.21.15.153:0/3661948696 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "737193be-7224-4e5f-9857-bc0349f5a29a", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f8744033250 con 0x7f874c15d580 2024-06-29T14:21:32.343 INFO:teuthology.orchestra.run.smithi153.stdout:6 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15d580 msgr2=0x7f874c1450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15d580 0x7f874c1450f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8744006010 tx=0x7f8744031650 comp rx=0 tx=0).stop 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 shutdown_connections 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874c15d580 0x7f874c1450f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874c15c5a0 0x7f874c1f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 --2- 172.21.15.153:0/3661948696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874c0b5320 0x7f874c153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 >> 172.21.15.153:0/3661948696 conn(0x7f874c073e50 msgr2=0x7f874c143190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 shutdown_connections 2024-06-29T14:21:32.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.346+0000 7f875332a640 1 -- 172.21.15.153:0/3661948696 wait complete. 2024-06-29T14:21:32.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 5fe7aecb-2572-4365-9ee9-9f272916b7ec 7 2024-06-29T14:21:32.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.469+0000 7f88d9dee640 1 -- 172.21.15.153:0/2275802616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d406efa0 msgr2=0x7f88d40b58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.469+0000 7f88d9dee640 1 --2- 172.21.15.153:0/2275802616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d406efa0 0x7f88d40b58d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88d4071ec0 tx=0x7f88bc02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.470+0000 7f88d9dee640 1 -- 172.21.15.153:0/2275802616 shutdown_connections 2024-06-29T14:21:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.470+0000 7f88d9dee640 1 --2- 172.21.15.153:0/2275802616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d406efa0 0x7f88d40b58d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.470+0000 7f88d9dee640 1 --2- 172.21.15.153:0/2275802616 >> v2:172.21.15.59:3300/0 conn(0x7f88d4072cb0 0x7f88d41439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.470+0000 7f88d9dee640 1 -- 172.21.15.153:0/2275802616 >> 172.21.15.153:0/2275802616 conn(0x7f88d40b30c0 msgr2=0x7f88d40cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.470+0000 7f88d9dee640 1 -- 172.21.15.153:0/2275802616 shutdown_connections 2024-06-29T14:21:32.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.470+0000 7f88d9dee640 1 -- 172.21.15.153:0/2275802616 wait complete. 2024-06-29T14:21:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.471+0000 7f88d9dee640 1 Processor -- start 2024-06-29T14:21:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.472+0000 7f88d9dee640 1 -- start start 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d9dee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88d406efa0 0x7f88d4159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d9dee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88d4072cb0 0x7f88d413c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d9dee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d40cf860 0x7f88d415e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d9dee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f88d415bb70 con 0x7f88d406efa0 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d9dee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88d415b9f0 con 0x7f88d40cf860 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d9dee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f88d415bcf0 con 0x7f88d4072cb0 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88d4072cb0 0x7f88d413c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d40cf860 0x7f88d415e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88d4072cb0 0x7f88d413c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50190/0 (socket says 172.21.15.153:50190) 2024-06-29T14:21:32.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 -- 172.21.15.153:0/2001905684 learned_addr learned my addr 172.21.15.153:0/2001905684 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d37fe640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88d406efa0 0x7f88d4159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 -- 172.21.15.153:0/2001905684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d40cf860 msgr2=0x7f88d415e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d40cf860 0x7f88d415e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 -- 172.21.15.153:0/2001905684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88d406efa0 msgr2=0x7f88d4159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88d406efa0 0x7f88d4159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 -- 172.21.15.153:0/2001905684 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d41e51b0 con 0x7f88d4072cb0 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88d3fff640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d40cf860 0x7f88d415e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.473+0000 7f88caffd640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88d4072cb0 0x7f88d413c8d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f88bc034040 tx=0x7f88bc003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.474+0000 7f88d0ff9640 1 -- 172.21.15.153:0/2001905684 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88bc03e030 con 0x7f88d4072cb0 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.474+0000 7f88d0ff9640 1 -- 172.21.15.153:0/2001905684 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88bc003d10 con 0x7f88d4072cb0 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.474+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d41e5490 con 0x7f88d4072cb0 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.474+0000 7f88d0ff9640 1 -- 172.21.15.153:0/2001905684 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88bc035040 con 0x7f88d4072cb0 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.474+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d41e5a50 con 0x7f88d4072cb0 2024-06-29T14:21:32.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.474+0000 7f88d0ff9640 1 -- 172.21.15.153:0/2001905684 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f88bc041030 con 0x7f88d4072cb0 2024-06-29T14:21:32.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.474+0000 7f88d0ff9640 1 -- 172.21.15.153:0/2001905684 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f88bc031400 con 0x7f88d4072cb0 2024-06-29T14:21:32.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.475+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d415c1a0 con 0x7f88d4072cb0 2024-06-29T14:21:32.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.477+0000 7f88d0ff9640 1 -- 172.21.15.153:0/2001905684 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f88bc043410 con 0x7f88d4072cb0 2024-06-29T14:21:32.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.550+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5fe7aecb-2572-4365-9ee9-9f272916b7ec", "id": 7} v 0) -- 0x7f88d4140df0 con 0x7f88d4072cb0 2024-06-29T14:21:32.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.553+0000 7f88d0ff9640 1 -- 172.21.15.153:0/2001905684 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5fe7aecb-2572-4365-9ee9-9f272916b7ec", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f88bc004ee0 con 0x7f88d4072cb0 2024-06-29T14:21:32.553 INFO:teuthology.orchestra.run.smithi153.stdout:7 2024-06-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88d4072cb0 msgr2=0x7f88d413c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88d4072cb0 0x7f88d413c8d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f88bc034040 tx=0x7f88bc003820 comp rx=0 tx=0).stop 2024-06-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 shutdown_connections 2024-06-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88d40cf860 0x7f88d415e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88d4072cb0 0x7f88d413c8d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 --2- 172.21.15.153:0/2001905684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88d406efa0 0x7f88d4159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 >> 172.21.15.153:0/2001905684 conn(0x7f88d40b30c0 msgr2=0x7f88d40b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:32.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 shutdown_connections 2024-06-29T14:21:32.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:21:32.554+0000 7f88d9dee640 1 -- 172.21.15.153:0/2001905684 wait complete. 2024-06-29T14:21:32.565 INFO:tasks.ceph.osd.0:Restarting daemon 2024-06-29T14:21:32.565 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-06-29T14:21:32.567 INFO:tasks.ceph.osd.0:Started 2024-06-29T14:21:32.567 INFO:tasks.ceph.osd.1:Restarting daemon 2024-06-29T14:21:32.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-06-29T14:21:32.570 INFO:tasks.ceph.osd.1:Started 2024-06-29T14:21:32.570 INFO:tasks.ceph.osd.2:Restarting daemon 2024-06-29T14:21:32.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-06-29T14:21:32.572 INFO:tasks.ceph.osd.2:Started 2024-06-29T14:21:32.572 INFO:tasks.ceph.osd.3:Restarting daemon 2024-06-29T14:21:32.572 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-06-29T14:21:32.575 INFO:tasks.ceph.osd.3:Started 2024-06-29T14:21:32.575 INFO:tasks.ceph.osd.4:Restarting daemon 2024-06-29T14:21:32.575 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-06-29T14:21:32.578 INFO:tasks.ceph.osd.4:Started 2024-06-29T14:21:32.578 INFO:tasks.ceph.osd.5:Restarting daemon 2024-06-29T14:21:32.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-06-29T14:21:32.580 INFO:tasks.ceph.osd.5:Started 2024-06-29T14:21:32.580 INFO:tasks.ceph.osd.6:Restarting daemon 2024-06-29T14:21:32.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-06-29T14:21:32.583 INFO:tasks.ceph.osd.6:Started 2024-06-29T14:21:32.583 INFO:tasks.ceph.osd.7:Restarting daemon 2024-06-29T14:21:32.583 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-06-29T14:21:32.585 INFO:tasks.ceph.osd.7:Started 2024-06-29T14:21:32.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:21:32.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.735+0000 7f70595d2640 1 -- 172.21.15.59:0/3343165718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 msgr2=0x7f705415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.735+0000 7f70595d2640 1 --2- 172.21.15.59:0/3343165718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 0x7f705415f9b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f704c005fd0 tx=0x7f704c02aa90 comp rx=0 tx=0).stop 2024-06-29T14:21:32.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.735+0000 7f70595d2640 1 -- 172.21.15.59:0/3343165718 shutdown_connections 2024-06-29T14:21:32.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.735+0000 7f70595d2640 1 --2- 172.21.15.59:0/3343165718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 0x7f705415f9b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.735+0000 7f70595d2640 1 --2- 172.21.15.59:0/3343165718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70540b40a0 0x7f705415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.735+0000 7f70595d2640 1 -- 172.21.15.59:0/3343165718 >> 172.21.15.59:0/3343165718 conn(0x7f70540d3ee0 msgr2=0x7f70540d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:32.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.735+0000 7f70595d2640 1 -- 172.21.15.59:0/3343165718 shutdown_connections 2024-06-29T14:21:32.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.735+0000 7f70595d2640 1 -- 172.21.15.59:0/3343165718 wait complete. 2024-06-29T14:21:32.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.737+0000 7f70595d2640 1 Processor -- start 2024-06-29T14:21:32.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.737+0000 7f70595d2640 1 -- start start 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70595d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70540d23c0 0x7f70541e6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70595d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 0x7f70541f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70595d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705414edd0 0x7f7054156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70595d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7054160280 con 0x7f705415ee10 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 0x7f70541f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70595d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7054160100 con 0x7f705414edd0 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70595d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7054160400 con 0x7f70540d23c0 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 0x7f70541f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37166/0 (socket says 172.21.15.59:37166) 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 -- 172.21.15.59:0/2505468674 learned_addr learned my addr 172.21.15.59:0/2505468674 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:32.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f7052ffd640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70540d23c0 0x7f70541e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70537fe640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705414edd0 0x7f7054156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 -- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70540d23c0 msgr2=0x7f70541e6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70540d23c0 0x7f70541e6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 -- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705414edd0 msgr2=0x7f7054156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705414edd0 0x7f7054156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 -- 172.21.15.59:0/2505468674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7054154400 con 0x7f705415ee10 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70537fe640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705414edd0 0x7f7054156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70527fc640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 0x7f70541f4a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f703c00e7c0 tx=0x7f703c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f704b7fe640 1 -- 172.21.15.59:0/2505468674 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c00ccc0 con 0x7f705415ee10 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f704b7fe640 1 -- 172.21.15.59:0/2505468674 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f703c010430 con 0x7f705415ee10 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70541546e0 con 0x7f705415ee10 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.740+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7054154ca0 con 0x7f705415ee10 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.739+0000 7f704b7fe640 1 -- 172.21.15.59:0/2505468674 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c010860 con 0x7f705415ee10 2024-06-29T14:21:32.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.740+0000 7f704b7fe640 1 -- 172.21.15.59:0/2505468674 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f703c010a00 con 0x7f705415ee10 2024-06-29T14:21:32.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.740+0000 7f704b7fe640 1 -- 172.21.15.59:0/2505468674 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f703c018c90 con 0x7f705415ee10 2024-06-29T14:21:32.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.741+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f705415e8d0 con 0x7f705415ee10 2024-06-29T14:21:32.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.743+0000 7f704b7fe640 1 -- 172.21.15.59:0/2505468674 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f703c018420 con 0x7f705415ee10 2024-06-29T14:21:32.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.820+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70540c7370 con 0x7f705415ee10 2024-06-29T14:21:32.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.820+0000 7f704b7fe640 1 -- 172.21.15.59:0/2505468674 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f703c018600 con 0x7f705415ee10 2024-06-29T14:21:32.819 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:21:32.820 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":10,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:32.552755+0000","last_up_change":"0.000000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 msgr2=0x7f70541f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 0x7f70541f4a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f703c00e7c0 tx=0x7f703c00ec80 comp rx=0 tx=0).stop 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 shutdown_connections 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f705414edd0 0x7f7054156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f705415ee10 0x7f70541f4a20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 --2- 172.21.15.59:0/2505468674 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70540d23c0 0x7f70541e6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 >> 172.21.15.59:0/2505468674 conn(0x7f70540d3ee0 msgr2=0x7f7054157fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 shutdown_connections 2024-06-29T14:21:32.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:32.821+0000 7f70595d2640 1 -- 172.21.15.59:0/2505468674 wait complete. 2024-06-29T14:21:32.831 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-06-29T14:21:32.831 INFO:tasks.ceph:Waiting for OSDs to come up 2024-06-29T14:21:33.188 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-06-29T14:21:33.188+0000 7fe501701640 -1 Falling back to public interface 2024-06-29T14:21:33.193 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-06-29T14:21:33.193+0000 7fa1084d9640 -1 Falling back to public interface 2024-06-29T14:21:33.201 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-06-29T14:21:33.201+0000 7f7f16c18640 -1 Falling back to public interface 2024-06-29T14:21:33.209 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-06-29T14:21:33.210+0000 7f410e263640 -1 Falling back to public interface 2024-06-29T14:21:33.213 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-06-29T14:21:33.215+0000 7fdc3c20e640 -1 Falling back to public interface 2024-06-29T14:21:33.217 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-06-29T14:21:33.218+0000 7f68aada7640 -1 Falling back to public interface 2024-06-29T14:21:33.227 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-06-29T14:21:33.228+0000 7f3bf3133640 -1 Falling back to public interface 2024-06-29T14:21:33.232 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-06-29T14:21:33.233+0000 7fd44a018640 -1 Falling back to public interface 2024-06-29T14:21:33.638 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-29T14:21:33.749 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.750+0000 7f8e8409e640 1 -- 172.21.15.59:0/3844576212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c0b6940 msgr2=0x7f8e7c145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:33.750 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.750+0000 7f8e8409e640 1 --2- 172.21.15.59:0/3844576212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c0b6940 0x7f8e7c145400 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8e78005fd0 tx=0x7f8e7802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:21:33.750 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.750+0000 7f8e8409e640 1 -- 172.21.15.59:0/3844576212 shutdown_connections 2024-06-29T14:21:33.750 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.750+0000 7f8e8409e640 1 --2- 172.21.15.59:0/3844576212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e7c15cbd0 0x7f8e7c0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:33.750 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.750+0000 7f8e8409e640 1 --2- 172.21.15.59:0/3844576212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c0b6940 0x7f8e7c145400 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:33.750 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.750+0000 7f8e8409e640 1 -- 172.21.15.59:0/3844576212 >> 172.21.15.59:0/3844576212 conn(0x7f8e7c06c8f0 msgr2=0x7f8e7c0b6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:33.750 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.750+0000 7f8e8409e640 1 -- 172.21.15.59:0/3844576212 shutdown_connections 2024-06-29T14:21:33.750 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.750+0000 7f8e8409e640 1 -- 172.21.15.59:0/3844576212 wait complete. 2024-06-29T14:21:33.751 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.752+0000 7f8e8409e640 1 Processor -- start 2024-06-29T14:21:33.751 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.752+0000 7f8e8409e640 1 -- start start 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.753+0000 7f8e8409e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e7c0b6940 0x7f8e7c139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.753+0000 7f8e8409e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e7c0cf710 0x7f8e7c1f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.753+0000 7f8e8409e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c15cbd0 0x7f8e7c1f5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.753+0000 7f8e8409e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c15d0f0 con 0x7f8e7c15cbd0 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.753+0000 7f8e8409e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c0ce680 con 0x7f8e7c0b6940 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.753+0000 7f8e8409e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8e7c15da20 con 0x7f8e7c0cf710 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e8263b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c15cbd0 0x7f8e7c1f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e7c0cf710 0x7f8e7c1f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e7c0cf710 0x7f8e7c1f4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:52358/0 (socket says 172.21.15.59:52358) 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e8263b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c15cbd0 0x7f8e7c1f5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37186/0 (socket says 172.21.15.59:37186) 2024-06-29T14:21:33.754 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 -- 172.21.15.59:0/2276250698 learned_addr learned my addr 172.21.15.59:0/2276250698 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81e3a640 1 --2- 172.21.15.59:0/2276250698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e7c0b6940 0x7f8e7c139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 -- 172.21.15.59:0/2276250698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e7c0b6940 msgr2=0x7f8e7c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 --2- 172.21.15.59:0/2276250698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e7c0b6940 0x7f8e7c139fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 -- 172.21.15.59:0/2276250698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c15cbd0 msgr2=0x7f8e7c1f5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 --2- 172.21.15.59:0/2276250698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c15cbd0 0x7f8e7c1f5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 -- 172.21.15.59:0/2276250698 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7c0a9bf0 con 0x7f8e7c0cf710 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.754+0000 7f8e81639640 1 --2- 172.21.15.59:0/2276250698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e7c0cf710 0x7f8e7c1f4b40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009fd0 tx=0x7f8e6c00ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.755+0000 7f8e727fc640 1 -- 172.21.15.59:0/2276250698 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00ccc0 con 0x7f8e7c0cf710 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.755+0000 7f8e727fc640 1 -- 172.21.15.59:0/2276250698 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e6c00ee70 con 0x7f8e7c0cf710 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.755+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7c0a9ed0 con 0x7f8e7c0cf710 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.755+0000 7f8e727fc640 1 -- 172.21.15.59:0/2276250698 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00ce60 con 0x7f8e7c0cf710 2024-06-29T14:21:33.755 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.755+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e7c0aa490 con 0x7f8e7c0cf710 2024-06-29T14:21:33.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.756+0000 7f8e727fc640 1 -- 172.21.15.59:0/2276250698 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 2) ==== 49122+0+0 (secure 0 0 0) 0x7f8e6c005260 con 0x7f8e7c0cf710 2024-06-29T14:21:33.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.756+0000 7f8e727fc640 1 -- 172.21.15.59:0/2276250698 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f8e6c051170 con 0x7f8e7c0cf710 2024-06-29T14:21:33.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.756+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e7c15df40 con 0x7f8e7c0cf710 2024-06-29T14:21:33.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.758+0000 7f8e727fc640 1 -- 172.21.15.59:0/2276250698 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f8e6c0047a0 con 0x7f8e7c0cf710 2024-06-29T14:21:33.820 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.821+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e7c0b7310 con 0x7f8e7c0cf710 2024-06-29T14:21:33.821 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.822+0000 7f8e727fc640 1 -- 172.21.15.59:0/2276250698 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f8e6c0049e0 con 0x7f8e7c0cf710 2024-06-29T14:21:33.821 INFO:teuthology.misc.health.smithi059.stdout: 2024-06-29T14:21:33.821 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:32.552755+0000","last_up_change":"0.000000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e7c0cf710 msgr2=0x7f8e7c1f4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 --2- 172.21.15.59:0/2276250698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e7c0cf710 0x7f8e7c1f4b40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009fd0 tx=0x7f8e6c00ec20 comp rx=0 tx=0).stop 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 shutdown_connections 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 --2- 172.21.15.59:0/2276250698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e7c15cbd0 0x7f8e7c1f5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 --2- 172.21.15.59:0/2276250698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e7c0cf710 0x7f8e7c1f4b40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 --2- 172.21.15.59:0/2276250698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e7c0b6940 0x7f8e7c139fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 >> 172.21.15.59:0/2276250698 conn(0x7f8e7c06c8f0 msgr2=0x7f8e7c144ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 shutdown_connections 2024-06-29T14:21:33.823 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:33.823+0000 7f8e8409e640 1 -- 172.21.15.59:0/2276250698 wait complete. 2024-06-29T14:21:33.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-29T14:21:34.732 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:21:34.732+0000 7fb086b36640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-06-29T14:21:34.732 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:21:34.733+0000 7fb086b36640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-29T14:21:36.630 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-06-29T14:21:36.631+0000 7fe501701640 -1 osd.4 0 log_to_monitors true 2024-06-29T14:21:36.672 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-06-29T14:21:36.673+0000 7f410e263640 -1 osd.2 0 log_to_monitors true 2024-06-29T14:21:36.687 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-06-29T14:21:36.688+0000 7fa1084d9640 -1 osd.6 0 log_to_monitors true 2024-06-29T14:21:36.717 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-06-29T14:21:36.718+0000 7f7f16c18640 -1 osd.7 0 log_to_monitors true 2024-06-29T14:21:36.724 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-06-29T14:21:36.725+0000 7fdc3c20e640 -1 osd.0 0 log_to_monitors true 2024-06-29T14:21:36.750 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-06-29T14:21:36.752+0000 7f3bf3133640 -1 osd.1 0 log_to_monitors true 2024-06-29T14:21:36.754 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-06-29T14:21:36.755+0000 7f68aada7640 -1 osd.3 0 log_to_monitors true 2024-06-29T14:21:36.754 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-06-29T14:21:36.755+0000 7fd44a018640 -1 osd.5 0 log_to_monitors true 2024-06-29T14:21:38.554 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-06-29T14:21:38.555+0000 7f410a1f9640 -1 osd.2 0 waiting for initial osdmap 2024-06-29T14:21:38.568 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-06-29T14:21:38.569+0000 7f4101ff4640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:21:39.387 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-06-29T14:21:39.387+0000 7f7f125e7640 -1 osd.7 0 waiting for initial osdmap 2024-06-29T14:21:39.405 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-06-29T14:21:39.406+0000 7f7f0a3e2640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:21:39.421 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-06-29T14:21:39.422+0000 7fa10446f640 -1 osd.6 0 waiting for initial osdmap 2024-06-29T14:21:39.433 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-06-29T14:21:39.434+0000 7fa0fc26a640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:21:39.762 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-06-29T14:21:39.763+0000 7fe4fd697640 -1 osd.4 0 waiting for initial osdmap 2024-06-29T14:21:39.774 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-06-29T14:21:39.775+0000 7fe4f5492640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:21:39.911 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-06-29T14:21:39.911+0000 7fd4461b4640 -1 osd.5 0 waiting for initial osdmap 2024-06-29T14:21:39.922 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-06-29T14:21:39.924+0000 7fd43d79b640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:21:40.444 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-06-29T14:21:40.445+0000 7f3bef0c9640 -1 osd.1 0 waiting for initial osdmap 2024-06-29T14:21:40.457 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-06-29T14:21:40.458+0000 7f3be6ec4640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:21:40.461 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-06-29T14:21:40.463+0000 7f68a6d3d640 -1 osd.3 0 waiting for initial osdmap 2024-06-29T14:21:40.480 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-06-29T14:21:40.481+0000 7f689eb38640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:21:40.520 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-06-29T14:21:40.521+0000 7fdc37b04640 -1 osd.0 0 waiting for initial osdmap 2024-06-29T14:21:40.533 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-06-29T14:21:40.534+0000 7fdc2f8ff640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:21:40.643 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-29T14:21:40.751 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.752+0000 7f0bb54e4640 1 -- 172.21.15.59:0/2193950454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb00cf9e0 msgr2=0x7f0bb015ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:40.752 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.752+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/2193950454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb00cf9e0 0x7f0bb015ddb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0073af0 tx=0x7f0ba802aac0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.752 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.752+0000 7f0bb54e4640 1 -- 172.21.15.59:0/2193950454 shutdown_connections 2024-06-29T14:21:40.752 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.752+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/2193950454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb00cf9e0 0x7f0bb015ddb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.752 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.752+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/2193950454 >> v2:172.21.15.153:3300/0 conn(0x7f0bb00d1510 0x7f0bb0152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.752 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.752+0000 7f0bb54e4640 1 -- 172.21.15.59:0/2193950454 >> 172.21.15.59:0/2193950454 conn(0x7f0bb006c980 msgr2=0x7f0bb00b6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:40.752 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.752+0000 7f0bb54e4640 1 -- 172.21.15.59:0/2193950454 shutdown_connections 2024-06-29T14:21:40.752 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.752+0000 7f0bb54e4640 1 -- 172.21.15.59:0/2193950454 wait complete. 2024-06-29T14:21:40.753 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.754+0000 7f0bb54e4640 1 Processor -- start 2024-06-29T14:21:40.753 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.754+0000 7f0bb54e4640 1 -- start start 2024-06-29T14:21:40.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.755+0000 7f0bb54e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb00d1510 0x7f0bb00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:40.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.755+0000 7f0bb54e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb015c350 0x7f0bb015efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:40.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0bb54e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bb0141e30 0x7f0bb015f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:40.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0bb54e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb015e680 con 0x7f0bb015c350 2024-06-29T14:21:40.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0bb54e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb015e500 con 0x7f0bb00d1510 2024-06-29T14:21:40.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0bb54e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0bb015e800 con 0x7f0bb0141e30 2024-06-29T14:21:40.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb015c350 0x7f0bb015efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:40.756 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb015c350 0x7f0bb015efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37226/0 (socket says 172.21.15.59:37226) 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 -- 172.21.15.59:0/827580442 learned_addr learned my addr 172.21.15.59:0/827580442 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0baeffd640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb00d1510 0x7f0bb00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0baf7fe640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bb0141e30 0x7f0bb015f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 -- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bb0141e30 msgr2=0x7f0bb015f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bb0141e30 0x7f0bb015f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 -- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb00d1510 msgr2=0x7f0bb00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb00d1510 0x7f0bb00ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 -- 172.21.15.59:0/827580442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb0142db0 con 0x7f0bb015c350 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0baf7fe640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bb0141e30 0x7f0bb015f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba7fff640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb015c350 0x7f0bb015efa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c3d0 tx=0x7f0b9800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba77fe640 1 -- 172.21.15.59:0/827580442 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98017070 con 0x7f0bb015c350 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba77fe640 1 -- 172.21.15.59:0/827580442 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b980053c0 con 0x7f0bb015c350 2024-06-29T14:21:40.757 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb0142510 con 0x7f0bb015c350 2024-06-29T14:21:40.758 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0ba77fe640 1 -- 172.21.15.59:0/827580442 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98003850 con 0x7f0bb015c350 2024-06-29T14:21:40.758 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.756+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb0142ad0 con 0x7f0bb015c350 2024-06-29T14:21:40.758 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.757+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bb015ed90 con 0x7f0bb015c350 2024-06-29T14:21:40.758 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.758+0000 7f0ba77fe640 1 -- 172.21.15.59:0/827580442 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) ==== 98145+0+0 (secure 0 0 0) 0x7f0b9801a070 con 0x7f0bb015c350 2024-06-29T14:21:40.759 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.759+0000 7f0ba77fe640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0b90076240 0x7f0b90078750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:40.759 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.759+0000 7f0ba77fe640 1 -- 172.21.15.59:0/827580442 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3631+0+0 (secure 0 0 0) 0x7f0b98096180 con 0x7f0bb015c350 2024-06-29T14:21:40.759 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.759+0000 7f0baeffd640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0b90076240 0x7f0b90078750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:40.759 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.759+0000 7f0baeffd640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0b90076240 0x7f0b90078750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009b30 tx=0x7f0ba0005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:40.762 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.764+0000 7f0ba77fe640 1 -- 172.21.15.59:0/827580442 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b9805fd90 con 0x7f0bb015c350 2024-06-29T14:21:40.955 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.956+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bb01422a0 con 0x7f0bb015c350 2024-06-29T14:21:40.955 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.956+0000 7f0ba77fe640 1 -- 172.21.15.59:0/827580442 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+9522 (secure 0 0 0) 0x7f0b980683d0 con 0x7f0bb015c350 2024-06-29T14:21:40.956 INFO:teuthology.misc.health.smithi059.stdout: 2024-06-29T14:21:40.956 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":14,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:40.453124+0000","last_up_change":"2024-06-29T14:21:40.453124+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0b90076240 msgr2=0x7f0b90078750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0b90076240 0x7f0b90078750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009b30 tx=0x7f0ba0005f90 comp rx=0 tx=0).stop 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb015c350 msgr2=0x7f0bb015efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb015c350 0x7f0bb015efa0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c3d0 tx=0x7f0b9800c890 comp rx=0 tx=0).stop 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 shutdown_connections 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0b90076240 0x7f0b90078750 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bb0141e30 0x7f0bb015f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bb015c350 0x7f0bb015efa0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.958 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 --2- 172.21.15.59:0/827580442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb00d1510 0x7f0bb00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:40.959 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 >> 172.21.15.59:0/827580442 conn(0x7f0bb006c980 msgr2=0x7f0bb0071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:40.959 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 shutdown_connections 2024-06-29T14:21:40.959 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:40.959+0000 7f0bb54e4640 1 -- 172.21.15.59:0/827580442 wait complete. 2024-06-29T14:21:40.968 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-06-29T14:21:47.775 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-29T14:21:47.899 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.899+0000 7f079917a640 1 -- 172.21.15.59:0/2395835384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c003c30 msgr2=0x7f077c004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:47.899 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.899+0000 7f079917a640 1 --2- 172.21.15.59:0/2395835384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c003c30 0x7f077c004080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0784006f80 tx=0x7f077c005830 comp rx=0 tx=0).stop 2024-06-29T14:21:47.899 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.899+0000 7f079917a640 1 -- 172.21.15.59:0/2395835384 shutdown_connections 2024-06-29T14:21:47.899 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.899+0000 7f079917a640 1 --2- 172.21.15.59:0/2395835384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c003c30 0x7f077c004080 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:47.899 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.899+0000 7f079917a640 1 -- 172.21.15.59:0/2395835384 >> 172.21.15.59:0/2395835384 conn(0x7f07940b5d00 msgr2=0x7f07940d3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:47.900 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.900+0000 7f079917a640 1 -- 172.21.15.59:0/2395835384 shutdown_connections 2024-06-29T14:21:47.900 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.900+0000 7f079917a640 1 -- 172.21.15.59:0/2395835384 wait complete. 2024-06-29T14:21:47.901 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.902+0000 7f079917a640 1 Processor -- start 2024-06-29T14:21:47.901 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.902+0000 7f079917a640 1 -- start start 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079917a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c003c30 0x7f07941f4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079917a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07941f5260 0x7f07940c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079917a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07940c99b0 0x7f07940caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079917a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f07940d3350 con 0x7f07941f5260 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079917a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f07940d34d0 con 0x7f077c003c30 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079917a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f07940d3650 con 0x7f07940c99b0 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07941f5260 0x7f07940c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07941f5260 0x7f07940c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39716/0 (socket says 172.21.15.59:39716) 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079259c640 1 -- 172.21.15.59:0/2119062107 learned_addr learned my addr 172.21.15.59:0/2119062107 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:47.904 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f0792d9d640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c003c30 0x7f07941f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.903+0000 7f079359e640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07940c99b0 0x7f07940caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f079259c640 1 -- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07940c99b0 msgr2=0x7f07940caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f079259c640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07940c99b0 0x7f07940caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f079259c640 1 -- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c003c30 msgr2=0x7f07941f4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f079259c640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c003c30 0x7f07941f4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f079259c640 1 -- 172.21.15.59:0/2119062107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07940c9dc0 con 0x7f07941f5260 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f079259c640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07941f5260 0x7f07940c9470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0788004d90 tx=0x7f0788001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f077b7fe640 1 -- 172.21.15.59:0/2119062107 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788015020 con 0x7f07941f5260 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f077b7fe640 1 -- 172.21.15.59:0/2119062107 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0788011980 con 0x7f07941f5260 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07940c7780 con 0x7f07941f5260 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f077b7fe640 1 -- 172.21.15.59:0/2119062107 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788011dd0 con 0x7f07941f5260 2024-06-29T14:21:47.905 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.904+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07940c7d40 con 0x7f07941f5260 2024-06-29T14:21:47.906 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.905+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07940c74f0 con 0x7f07941f5260 2024-06-29T14:21:47.906 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.906+0000 7f077b7fe640 1 -- 172.21.15.59:0/2119062107 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0788018020 con 0x7f07941f5260 2024-06-29T14:21:47.907 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.907+0000 7f077b7fe640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f07800762d0 0x7f07800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:47.907 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.907+0000 7f077b7fe640 1 -- 172.21.15.59:0/2119062107 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f0788097d50 con 0x7f07941f5260 2024-06-29T14:21:47.907 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.907+0000 7f0792d9d640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f07800762d0 0x7f07800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:47.907 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.907+0000 7f0792d9d640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f07800762d0 0x7f07800787e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0784033000 tx=0x7f07940d4980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:47.910 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:47.911+0000 7f077b7fe640 1 -- 172.21.15.59:0/2119062107 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f078809b030 con 0x7f07941f5260 2024-06-29T14:21:48.102 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.103+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07940ceea0 con 0x7f07941f5260 2024-06-29T14:21:48.102 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.103+0000 7f077b7fe640 1 -- 172.21.15.59:0/2119062107 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12896 (secure 0 0 0) 0x7f078801f020 con 0x7f07941f5260 2024-06-29T14:21:48.103 INFO:teuthology.misc.health.smithi059.stdout: 2024-06-29T14:21:48.103 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":18,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:44.463274+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f07800762d0 msgr2=0x7f07800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f07800762d0 0x7f07800787e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0784033000 tx=0x7f07940d4980 comp rx=0 tx=0).stop 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07941f5260 msgr2=0x7f07940c9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07941f5260 0x7f07940c9470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0788004d90 tx=0x7f0788001800 comp rx=0 tx=0).stop 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 shutdown_connections 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f07800762d0 0x7f07800787e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07940c99b0 0x7f07940caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07941f5260 0x7f07940c9470 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 --2- 172.21.15.59:0/2119062107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c003c30 0x7f07941f4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 >> 172.21.15.59:0/2119062107 conn(0x7f07940b5d00 msgr2=0x7f07940b7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 shutdown_connections 2024-06-29T14:21:48.105 INFO:teuthology.misc.health.smithi059.stderr:2024-06-29T14:21:48.105+0000 7f079917a640 1 -- 172.21.15.59:0/2119062107 wait complete. 2024-06-29T14:21:48.114 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-06-29T14:21:48.115 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-06-29T14:21:48.115 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:21:48.115 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:21:48.117 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:21:48.118 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:21:48.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:21:48.120 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:21:48.120 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:21:48.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:21:48.123 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:21:48.123 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:21:48.123 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:21:48.125 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:21:48.126 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-06-29T14:21:48.126 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:21:48.207 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:21:48.207 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:21:48.222 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:21:48.223 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:21:48.223 INFO:tasks.ceph.mds.d.smithi153.stderr:2024-06-29T14:21:48.225+0000 7f028829e640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 -- 172.21.15.59:0/559839595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 msgr2=0x7f2d1015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 --2- 172.21.15.59:0/559839595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 0x7f2d1015e8d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2d100d04f0 tx=0x7f2d0402a560 comp rx=0 tx=0).stop 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 -- 172.21.15.59:0/559839595 shutdown_connections 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 --2- 172.21.15.59:0/559839595 >> v2:172.21.15.153:3300/0 conn(0x7f2d1015ee10 0x7f2d1015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 --2- 172.21.15.59:0/559839595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 0x7f2d1015e8d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 --2- 172.21.15.59:0/559839595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100d23c0 0x7f2d100b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 -- 172.21.15.59:0/559839595 >> 172.21.15.59:0/559839595 conn(0x7f2d100d3ee0 msgr2=0x7f2d100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 -- 172.21.15.59:0/559839595 shutdown_connections 2024-06-29T14:21:48.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.249+0000 7f2d167e0640 1 -- 172.21.15.59:0/559839595 wait complete. 2024-06-29T14:21:48.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.251+0000 7f2d167e0640 1 Processor -- start 2024-06-29T14:21:48.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.251+0000 7f2d167e0640 1 -- start start 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d167e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 0x7f2d1014eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d167e0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100d23c0 0x7f2d1014a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d167e0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015ee10 0x7f2d10154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 0x7f2d1014eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 0x7f2d1014eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39790/0 (socket says 172.21.15.59:39790) 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d167e0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2d10160070 con 0x7f2d100b3ed0 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d167e0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1015fef0 con 0x7f2d1015ee10 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d167e0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2d101601f0 con 0x7f2d100d23c0 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 -- 172.21.15.59:0/1673931949 learned_addr learned my addr 172.21.15.59:0/1673931949 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:48.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d14d7d640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015ee10 0x7f2d10154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0f7fe640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100d23c0 0x7f2d1014a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 -- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100d23c0 msgr2=0x7f2d1014a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100d23c0 0x7f2d1014a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 -- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015ee10 msgr2=0x7f2d10154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015ee10 0x7f2d10154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 -- 172.21.15.59:0/1673931949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1014f400 con 0x7f2d100b3ed0 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0ffff640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 0x7f2d1014eec0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000d6b0 tx=0x7f2d0000db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0cff9640 1 -- 172.21.15.59:0/1673931949 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00017020 con 0x7f2d100b3ed0 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1014fd40 con 0x7f2d100b3ed0 2024-06-29T14:21:48.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0cff9640 1 -- 172.21.15.59:0/1673931949 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d00004320 con 0x7f2d100b3ed0 2024-06-29T14:21:48.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.253+0000 7f2d0cff9640 1 -- 172.21.15.59:0/1673931949 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00009a80 con 0x7f2d100b3ed0 2024-06-29T14:21:48.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.254+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d10150300 con 0x7f2d100b3ed0 2024-06-29T14:21:48.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.255+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d100b3990 con 0x7f2d100b3ed0 2024-06-29T14:21:48.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.255+0000 7f2d0cff9640 1 -- 172.21.15.59:0/1673931949 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d0001a020 con 0x7f2d100b3ed0 2024-06-29T14:21:48.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.256+0000 7f2d0cff9640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.256+0000 7f2d0cff9640 1 -- 172.21.15.59:0/1673931949 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f2d00097640 con 0x7f2d100b3ed0 2024-06-29T14:21:48.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.256+0000 7f2d0f7fe640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.257+0000 7f2d0f7fe640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2d04033000 tx=0x7f2d04004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:48.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.261+0000 7f2d0cff9640 1 -- 172.21.15.59:0/1673931949 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d00021020 con 0x7f2d100b3ed0 2024-06-29T14:21:48.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.452+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d100d2950 con 0x7f2d100b3ed0 2024-06-29T14:21:48.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.453+0000 7f2d0cff9640 1 -- 172.21.15.59:0/1673931949 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12896 (secure 0 0 0) 0x7f2d00061300 con 0x7f2d100b3ed0 2024-06-29T14:21:48.452 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:21:48.452 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":18,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:44.463274+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:21:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 msgr2=0x7f2cfc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2d04033000 tx=0x7f2d04004730 comp rx=0 tx=0).stop 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 msgr2=0x7f2d1014eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 0x7f2d1014eec0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000d6b0 tx=0x7f2d0000db70 comp rx=0 tx=0).stop 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 shutdown_connections 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015ee10 0x7f2d10154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d100d23c0 0x7f2d1014a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 --2- 172.21.15.59:0/1673931949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d100b3ed0 0x7f2d1014eec0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 >> 172.21.15.59:0/1673931949 conn(0x7f2d100d3ee0 msgr2=0x7f2d100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 shutdown_connections 2024-06-29T14:21:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.455+0000 7f2d167e0640 1 -- 172.21.15.59:0/1673931949 wait complete. 2024-06-29T14:21:48.466 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:21:48.466 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:21:48.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.587+0000 7fb0c4866640 1 -- 172.21.15.59:0/1439135095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 msgr2=0x7fb0c00d26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.587+0000 7fb0c4866640 1 --2- 172.21.15.59:0/1439135095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 0x7fb0c00d26a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00d4ba0 tx=0x7fb0a802a4d0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.587+0000 7fb0c4866640 1 -- 172.21.15.59:0/1439135095 shutdown_connections 2024-06-29T14:21:48.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.587+0000 7fb0c4866640 1 --2- 172.21.15.59:0/1439135095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 0x7fb0c00d26a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.587+0000 7fb0c4866640 1 --2- 172.21.15.59:0/1439135095 >> v2:172.21.15.153:3300/0 conn(0x7fb0c00b2980 0x7fb0c015dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.587+0000 7fb0c4866640 1 -- 172.21.15.59:0/1439135095 >> 172.21.15.59:0/1439135095 conn(0x7fb0c00d3b20 msgr2=0x7fb0c00d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:48.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.587+0000 7fb0c4866640 1 -- 172.21.15.59:0/1439135095 shutdown_connections 2024-06-29T14:21:48.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.587+0000 7fb0c4866640 1 -- 172.21.15.59:0/1439135095 wait complete. 2024-06-29T14:21:48.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.589+0000 7fb0c4866640 1 Processor -- start 2024-06-29T14:21:48.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.589+0000 7fb0c4866640 1 -- start start 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0c4866640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c00b2980 0x7fb0c00cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0c4866640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 0x7fb0c00ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0c4866640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c00d2be0 0x7fb0c00c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0c4866640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01620a0 con 0x7fb0c00cefb0 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0c4866640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c00b4db0 con 0x7fb0c00b2980 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0c4866640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb0c0162220 con 0x7fb0c00d2be0 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0bdd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 0x7fb0c00ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0bdd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 0x7fb0c00ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39806/0 (socket says 172.21.15.59:39806) 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.590+0000 7fb0bdd9b640 1 -- 172.21.15.59:0/2972223520 learned_addr learned my addr 172.21.15.59:0/2972223520 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:48.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0bed9d640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c00d2be0 0x7fb0c00c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0be59c640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c00b2980 0x7fb0c00cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0bdd9b640 1 -- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c00d2be0 msgr2=0x7fb0c00c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0bdd9b640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c00d2be0 0x7fb0c00c86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0bdd9b640 1 -- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c00b2980 msgr2=0x7fb0c00cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0bdd9b640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c00b2980 0x7fb0c00cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0bdd9b640 1 -- 172.21.15.59:0/2972223520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c00ca870 con 0x7fb0c00cefb0 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0bdd9b640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 0x7fb0c00ccfd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8034040 tx=0x7fb0a8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0b6ffd640 1 -- 172.21.15.59:0/2972223520 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a803e030 con 0x7fb0c00cefb0 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0b6ffd640 1 -- 172.21.15.59:0/2972223520 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0a8003e00 con 0x7fb0c00cefb0 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c00cab50 con 0x7fb0c00cefb0 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0b6ffd640 1 -- 172.21.15.59:0/2972223520 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8035040 con 0x7fb0c00cefb0 2024-06-29T14:21:48.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.591+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c00cb110 con 0x7fb0c00cefb0 2024-06-29T14:21:48.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.593+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c0162740 con 0x7fb0c00cefb0 2024-06-29T14:21:48.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.593+0000 7fb0b6ffd640 1 -- 172.21.15.59:0/2972223520 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb0a8041030 con 0x7fb0c00cefb0 2024-06-29T14:21:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.593+0000 7fb0b6ffd640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb0a00762d0 0x7fb0a00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.594+0000 7fb0b6ffd640 1 -- 172.21.15.59:0/2972223520 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fb0a80b7b50 con 0x7fb0c00cefb0 2024-06-29T14:21:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.594+0000 7fb0be59c640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb0a00762d0 0x7fb0a00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.594+0000 7fb0be59c640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb0a00762d0 0x7fb0a00787e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0005fd0 tx=0x7fb0b00095e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:48.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.598+0000 7fb0b6ffd640 1 -- 172.21.15.59:0/2972223520 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb0a8081130 con 0x7fb0c00cefb0 2024-06-29T14:21:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.792+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb0c00d4400 con 0x7fb0c00cefb0 2024-06-29T14:21:48.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.792+0000 7fb0b6ffd640 1 -- 172.21.15.59:0/2972223520 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7fb0a8085430 con 0x7fb0c00cefb0 2024-06-29T14:21:48.792 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:21:48.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb0a00762d0 msgr2=0x7fb0a00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb0a00762d0 0x7fb0a00787e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0005fd0 tx=0x7fb0b00095e0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 msgr2=0x7fb0c00ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 0x7fb0c00ccfd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8034040 tx=0x7fb0a8003910 comp rx=0 tx=0).stop 2024-06-29T14:21:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 shutdown_connections 2024-06-29T14:21:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb0a00762d0 0x7fb0a00787e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb0c00d2be0 0x7fb0c00c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0c00cefb0 0x7fb0c00ccfd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 --2- 172.21.15.59:0/2972223520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0c00b2980 0x7fb0c00cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 >> 172.21.15.59:0/2972223520 conn(0x7fb0c00d3b20 msgr2=0x7fb0c01604a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 shutdown_connections 2024-06-29T14:21:48.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.795+0000 7fb0c4866640 1 -- 172.21.15.59:0/2972223520 wait complete. 2024-06-29T14:21:48.805 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:21:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.923+0000 7f58f8d4a640 1 -- 172.21.15.59:0/2082884471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415c2f0 msgr2=0x7f58f415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.923+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/2082884471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415c2f0 0x7f58f415c700 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58f40b1c40 tx=0x7f58e402a560 comp rx=0 tx=0).stop 2024-06-29T14:21:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.923+0000 7f58f8d4a640 1 -- 172.21.15.59:0/2082884471 shutdown_connections 2024-06-29T14:21:48.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.923+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/2082884471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415c2f0 0x7f58f415c700 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.923+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/2082884471 >> v2:172.21.15.153:3300/0 conn(0x7f58f40b3740 0x7f58f40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.923+0000 7f58f8d4a640 1 -- 172.21.15.59:0/2082884471 >> 172.21.15.59:0/2082884471 conn(0x7f58f406c950 msgr2=0x7f58f4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:48.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.923+0000 7f58f8d4a640 1 -- 172.21.15.59:0/2082884471 shutdown_connections 2024-06-29T14:21:48.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.923+0000 7f58f8d4a640 1 -- 172.21.15.59:0/2082884471 wait complete. 2024-06-29T14:21:48.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.925+0000 7f58f8d4a640 1 Processor -- start 2024-06-29T14:21:48.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.925+0000 7f58f8d4a640 1 -- start start 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f8d4a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40b3740 0x7f58f40ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f8d4a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f415c2f0 0x7f58f40afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f8d4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415b460 0x7f58f40d1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f8d4a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f58f40b5fd0 con 0x7f58f415b460 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f8d4a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58f40b5e50 con 0x7f58f40b3740 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f8d4a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f58f40b6150 con 0x7f58f415c2f0 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415b460 0x7f58f40d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415b460 0x7f58f40d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39822/0 (socket says 172.21.15.59:39822) 2024-06-29T14:21:48.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.926+0000 7f58f2d9d640 1 -- 172.21.15.59:0/1107562737 learned_addr learned my addr 172.21.15.59:0/1107562737 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:48.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f1d9b640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f415c2f0 0x7f58f40afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f259c640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40b3740 0x7f58f40ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f2d9d640 1 -- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f415c2f0 msgr2=0x7f58f40afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f2d9d640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f415c2f0 0x7f58f40afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f2d9d640 1 -- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40b3740 msgr2=0x7f58f40ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:48.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f2d9d640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40b3740 0x7f58f40ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:48.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f2d9d640 1 -- 172.21.15.59:0/1107562737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f40abf70 con 0x7f58f415b460 2024-06-29T14:21:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f259c640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40b3740 0x7f58f40ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f1d9b640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f415c2f0 0x7f58f40afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f2d9d640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415b460 0x7f58f40d1900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58e4034040 tx=0x7f58e40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58eaffd640 1 -- 172.21.15.59:0/1107562737 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e403e030 con 0x7f58f415b460 2024-06-29T14:21:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58eaffd640 1 -- 172.21.15.59:0/1107562737 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58e40047a0 con 0x7f58f415b460 2024-06-29T14:21:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f40ac250 con 0x7f58f415b460 2024-06-29T14:21:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58eaffd640 1 -- 172.21.15.59:0/1107562737 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4035040 con 0x7f58f415b460 2024-06-29T14:21:48.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.927+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f414c3d0 con 0x7f58f415b460 2024-06-29T14:21:48.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.928+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f40d2a80 con 0x7f58f415b460 2024-06-29T14:21:48.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.929+0000 7f58eaffd640 1 -- 172.21.15.59:0/1107562737 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f58e4041030 con 0x7f58f415b460 2024-06-29T14:21:48.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.929+0000 7f58eaffd640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58d40762d0 0x7f58d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:48.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.930+0000 7f58eaffd640 1 -- 172.21.15.59:0/1107562737 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f58e40b7cc0 con 0x7f58f415b460 2024-06-29T14:21:48.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.930+0000 7f58f259c640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58d40762d0 0x7f58d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:48.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.930+0000 7f58f259c640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58d40762d0 0x7f58d40787e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009fc0 tx=0x7f58ec009550 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:48.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:48.934+0000 7f58eaffd640 1 -- 172.21.15.59:0/1107562737 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f58f40d2a80 con 0x7f58f415b460 2024-06-29T14:21:49.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.164+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58f40b6670 con 0x7f58f415b460 2024-06-29T14:21:49.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.165+0000 7f58eaffd640 1 -- 172.21.15.59:0/1107562737 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1431 (secure 0 0 0) 0x7f58e40818e0 con 0x7f58f415b460 2024-06-29T14:21:49.164 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:21:49.164 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":2,"btime":"2024-06-29T14:21:48:214900+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-29T14:21:49.165 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 2 2024-06-29T14:21:49.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58d40762d0 msgr2=0x7f58d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:49.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58d40762d0 0x7f58d40787e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009fc0 tx=0x7f58ec009550 comp rx=0 tx=0).stop 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415b460 msgr2=0x7f58f40d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415b460 0x7f58f40d1900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58e4034040 tx=0x7f58e40039e0 comp rx=0 tx=0).stop 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 shutdown_connections 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58d40762d0 0x7f58d40787e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f415b460 0x7f58f40d1900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58f415c2f0 0x7f58f40afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 --2- 172.21.15.59:0/1107562737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58f40b3740 0x7f58f40ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 >> 172.21.15.59:0/1107562737 conn(0x7f58f406c950 msgr2=0x7f58f4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 shutdown_connections 2024-06-29T14:21:49.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:49.167+0000 7f58f8d4a640 1 -- 172.21.15.59:0/1107562737 wait complete. 2024-06-29T14:21:51.177 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:21:51.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.311+0000 7fafed488640 1 -- 172.21.15.59:0/1462901087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80ce950 msgr2=0x7fafe81622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.311+0000 7fafed488640 1 --2- 172.21.15.59:0/1462901087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80ce950 0x7fafe81622c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009830 tx=0x7fafd002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.311+0000 7fafed488640 1 -- 172.21.15.59:0/1462901087 shutdown_connections 2024-06-29T14:21:51.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.311+0000 7fafed488640 1 --2- 172.21.15.59:0/1462901087 >> v2:172.21.15.153:3300/0 conn(0x7fafe8162800 0x7fafe8162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.311+0000 7fafed488640 1 --2- 172.21.15.59:0/1462901087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80ce950 0x7fafe81622c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.311+0000 7fafed488640 1 -- 172.21.15.59:0/1462901087 >> 172.21.15.59:0/1462901087 conn(0x7fafe80b5610 msgr2=0x7fafe80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:51.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.311+0000 7fafed488640 1 -- 172.21.15.59:0/1462901087 shutdown_connections 2024-06-29T14:21:51.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.311+0000 7fafed488640 1 -- 172.21.15.59:0/1462901087 wait complete. 2024-06-29T14:21:51.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.313+0000 7fafed488640 1 Processor -- start 2024-06-29T14:21:51.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.313+0000 7fafed488640 1 -- start start 2024-06-29T14:21:51.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.314+0000 7fafed488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe808c190 0x7fafe815c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.314+0000 7fafed488640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe80ce950 0x7fafe80cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.314+0000 7fafed488640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8162800 0x7fafe81f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.314+0000 7fafed488640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fafe80b7f60 con 0x7fafe808c190 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.314+0000 7fafed488640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafe80b7de0 con 0x7fafe8162800 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.314+0000 7fafed488640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fafe80b80e0 con 0x7fafe80ce950 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe808c190 0x7fafe815c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe808c190 0x7fafe815c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39836/0 (socket says 172.21.15.59:39836) 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 -- 172.21.15.59:0/3588206148 learned_addr learned my addr 172.21.15.59:0/3588206148 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe67fc640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe80ce950 0x7fafe80cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe77fe640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8162800 0x7fafe81f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 -- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe80ce950 msgr2=0x7fafe80cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe80ce950 0x7fafe80cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 -- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8162800 msgr2=0x7fafe81f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8162800 0x7fafe81f4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 -- 172.21.15.59:0/3588206148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe80cb060 con 0x7fafe808c190 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe77fe640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8162800 0x7fafe81f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafe6ffd640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe808c190 0x7fafe815c1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafd800c7a0 tx=0x7fafd800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafdf7fe640 1 -- 172.21.15.59:0/3588206148 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd8017070 con 0x7fafe808c190 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafdf7fe640 1 -- 172.21.15.59:0/3588206148 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafd8005070 con 0x7fafe808c190 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe80cb340 con 0x7fafe808c190 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafdf7fe640 1 -- 172.21.15.59:0/3588206148 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd80054a0 con 0x7fafe808c190 2024-06-29T14:21:51.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.315+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe80cb900 con 0x7fafe808c190 2024-06-29T14:21:51.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.317+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe81622c0 con 0x7fafe808c190 2024-06-29T14:21:51.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.317+0000 7fafdf7fe640 1 -- 172.21.15.59:0/3588206148 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fafd801a070 con 0x7fafe808c190 2024-06-29T14:21:51.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.318+0000 7fafdf7fe640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fafc80762d0 0x7fafc80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:51.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.318+0000 7fafdf7fe640 1 -- 172.21.15.59:0/3588206148 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fafd8096b80 con 0x7fafe808c190 2024-06-29T14:21:51.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.318+0000 7fafe67fc640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fafc80762d0 0x7fafc80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:51.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.319+0000 7fafe67fc640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fafc80762d0 0x7fafc80787e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fafd0034000 tx=0x7fafd0004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:51.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.323+0000 7fafdf7fe640 1 -- 172.21.15.59:0/3588206148 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fafd8060830 con 0x7fafe808c190 2024-06-29T14:21:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.551+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafe8075bb0 con 0x7fafe808c190 2024-06-29T14:21:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.552+0000 7fafdf7fe640 1 -- 172.21.15.59:0/3588206148 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3909 (secure 0 0 0) 0x7fafd8060130 con 0x7fafe808c190 2024-06-29T14:21:51.551 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:21:51.551 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":3,"btime":"2024-06-29T14:21:49:216174+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4375,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-06-29T14:21:51.551 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 3 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fafc80762d0 msgr2=0x7fafc80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fafc80762d0 0x7fafc80787e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fafd0034000 tx=0x7fafd0004730 comp rx=0 tx=0).stop 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe808c190 msgr2=0x7fafe815c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe808c190 0x7fafe815c1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafd800c7a0 tx=0x7fafd800cc60 comp rx=0 tx=0).stop 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 shutdown_connections 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fafc80762d0 0x7fafc80787e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8162800 0x7fafe81f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe80ce950 0x7fafe80cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 --2- 172.21.15.59:0/3588206148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe808c190 0x7fafe815c1b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 >> 172.21.15.59:0/3588206148 conn(0x7fafe80b5610 msgr2=0x7fafe80c7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:51.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 shutdown_connections 2024-06-29T14:21:51.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.554+0000 7fafed488640 1 -- 172.21.15.59:0/3588206148 wait complete. 2024-06-29T14:21:51.564 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:21:51.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.683+0000 7f439d7d2640 1 -- 172.21.15.59:0/3196819085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398162190 msgr2=0x7f43980ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.683+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3196819085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398162190 0x7f43980ce550 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4388009830 tx=0x7f438802a480 comp rx=0 tx=0).stop 2024-06-29T14:21:51.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.683+0000 7f439d7d2640 1 -- 172.21.15.59:0/3196819085 shutdown_connections 2024-06-29T14:21:51.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.683+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3196819085 >> v2:172.21.15.153:3300/0 conn(0x7f439815e0a0 0x7f43980b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.683+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3196819085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4398162190 0x7f43980ce550 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.683+0000 7f439d7d2640 1 -- 172.21.15.59:0/3196819085 >> 172.21.15.59:0/3196819085 conn(0x7f43980d45b0 msgr2=0x7f43980d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.683+0000 7f439d7d2640 1 -- 172.21.15.59:0/3196819085 shutdown_connections 2024-06-29T14:21:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.683+0000 7f439d7d2640 1 -- 172.21.15.59:0/3196819085 wait complete. 2024-06-29T14:21:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.685+0000 7f439d7d2640 1 Processor -- start 2024-06-29T14:21:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.685+0000 7f439d7d2640 1 -- start start 2024-06-29T14:21:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f439d7d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439815e0a0 0x7f43981564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f439d7d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398162770 0x7f43981586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f439d7d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4398156dd0 0x7f4398158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f439d7d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f43980b43a0 con 0x7f4398156dd0 2024-06-29T14:21:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f439d7d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43980b4220 con 0x7f4398162770 2024-06-29T14:21:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f439d7d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f43980b4520 con 0x7f439815e0a0 2024-06-29T14:21:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f43977fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4398156dd0 0x7f4398158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f43977fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4398156dd0 0x7f4398158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39852/0 (socket says 172.21.15.59:39852) 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f43977fe640 1 -- 172.21.15.59:0/3598782595 learned_addr learned my addr 172.21.15.59:0/3598782595 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f4396ffd640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439815e0a0 0x7f43981564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f43967fc640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398162770 0x7f43981586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f43977fe640 1 -- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439815e0a0 msgr2=0x7f43981564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.686+0000 7f43977fe640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439815e0a0 0x7f43981564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f43977fe640 1 -- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398162770 msgr2=0x7f43981586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f43977fe640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398162770 0x7f43981586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f43977fe640 1 -- 172.21.15.59:0/3598782595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4398154050 con 0x7f4398156dd0 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f43977fe640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4398156dd0 0x7f4398158c30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f439000ebf0 tx=0x7f439000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f438f7fe640 1 -- 172.21.15.59:0/3598782595 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4390003b70 con 0x7f4398156dd0 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f438f7fe640 1 -- 172.21.15.59:0/3598782595 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4390003d10 con 0x7f4398156dd0 2024-06-29T14:21:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43980ccf30 con 0x7f4398156dd0 2024-06-29T14:21:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f438f7fe640 1 -- 172.21.15.59:0/3598782595 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4390004280 con 0x7f4398156dd0 2024-06-29T14:21:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.687+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43981545d0 con 0x7f4398156dd0 2024-06-29T14:21:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.688+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43980d0aa0 con 0x7f4398156dd0 2024-06-29T14:21:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.689+0000 7f438f7fe640 1 -- 172.21.15.59:0/3598782595 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4390004420 con 0x7f4398156dd0 2024-06-29T14:21:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.689+0000 7f438f7fe640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43780762d0 0x7f43780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.689+0000 7f438f7fe640 1 -- 172.21.15.59:0/3598782595 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f4390097ee0 con 0x7f4398156dd0 2024-06-29T14:21:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.689+0000 7f4396ffd640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43780762d0 0x7f43780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.690+0000 7f4396ffd640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43780762d0 0x7f43780787e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4380007fd0 tx=0x7f4380001390 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:51.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.693+0000 7f438f7fe640 1 -- 172.21.15.59:0/3598782595 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f43980d0aa0 con 0x7f4398156dd0 2024-06-29T14:21:51.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.887+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43980cd210 con 0x7f4398156dd0 2024-06-29T14:21:51.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.888+0000 7f438f7fe640 1 -- 172.21.15.59:0/3598782595 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12896 (secure 0 0 0) 0x7f43980cd210 con 0x7f4398156dd0 2024-06-29T14:21:51.887 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:21:51.887 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":18,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:44.463274+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:21:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43780762d0 msgr2=0x7f43780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43780762d0 0x7f43780787e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4380007fd0 tx=0x7f4380001390 comp rx=0 tx=0).stop 2024-06-29T14:21:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4398156dd0 msgr2=0x7f4398158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4398156dd0 0x7f4398158c30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f439000ebf0 tx=0x7f439000c6a0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 shutdown_connections 2024-06-29T14:21:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43780762d0 0x7f43780787e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4398156dd0 0x7f4398158c30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4398162770 0x7f43981586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 --2- 172.21.15.59:0/3598782595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439815e0a0 0x7f43981564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 >> 172.21.15.59:0/3598782595 conn(0x7f43980d45b0 msgr2=0x7f439806c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 shutdown_connections 2024-06-29T14:21:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:51.891+0000 7f439d7d2640 1 -- 172.21.15.59:0/3598782595 wait complete. 2024-06-29T14:21:51.901 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:21:51.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:21:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.022+0000 7f55b8154640 1 -- 172.21.15.59:0/3109004455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b015efe0 msgr2=0x7f55b015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.022+0000 7f55b8154640 1 --2- 172.21.15.59:0/3109004455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b015efe0 0x7f55b015f9b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55a8005fd0 tx=0x7f55a802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.023+0000 7f55b8154640 1 -- 172.21.15.59:0/3109004455 shutdown_connections 2024-06-29T14:21:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.023+0000 7f55b8154640 1 --2- 172.21.15.59:0/3109004455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b015efe0 0x7f55b015f9b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.023+0000 7f55b8154640 1 -- 172.21.15.59:0/3109004455 >> 172.21.15.59:0/3109004455 conn(0x7f55b00d3ee0 msgr2=0x7f55b00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.023+0000 7f55b8154640 1 -- 172.21.15.59:0/3109004455 shutdown_connections 2024-06-29T14:21:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.023+0000 7f55b8154640 1 -- 172.21.15.59:0/3109004455 wait complete. 2024-06-29T14:21:52.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.024+0000 7f55b8154640 1 Processor -- start 2024-06-29T14:21:52.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.024+0000 7f55b8154640 1 -- start start 2024-06-29T14:21:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b8154640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b00d23c0 0x7f55b014c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b8154640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55b015efe0 0x7f55b00ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b8154640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55b014dfe0 0x7f55b00cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b8154640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f55b0160420 con 0x7f55b00d23c0 2024-06-29T14:21:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b8154640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55b01602a0 con 0x7f55b015efe0 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b8154640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f55b01605a0 con 0x7f55b014dfe0 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b00d23c0 0x7f55b014c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b00d23c0 0x7f55b014c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39870/0 (socket says 172.21.15.59:39870) 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 -- 172.21.15.59:0/2990235154 learned_addr learned my addr 172.21.15.59:0/2990235154 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b66f1640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55b014dfe0 0x7f55b00cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b56ef640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55b015efe0 0x7f55b00ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 -- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55b014dfe0 msgr2=0x7f55b00cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55b014dfe0 0x7f55b00cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 -- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55b015efe0 msgr2=0x7f55b00ca0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55b015efe0 0x7f55b00ca0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 -- 172.21.15.59:0/2990235154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b0153740 con 0x7f55b00d23c0 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b56ef640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55b015efe0 0x7f55b00ca0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b66f1640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55b014dfe0 0x7f55b00cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b5ef0640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b00d23c0 0x7f55b014c1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00c3c0 tx=0x7f55ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55a67fc640 1 -- 172.21.15.59:0/2990235154 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac017070 con 0x7f55b00d23c0 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55a67fc640 1 -- 172.21.15.59:0/2990235154 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55ac0035f0 con 0x7f55b00d23c0 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b00cc6e0 con 0x7f55b00d23c0 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55a67fc640 1 -- 172.21.15.59:0/2990235154 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac003a20 con 0x7f55b00d23c0 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.027+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b0153260 con 0x7f55b00d23c0 2024-06-29T14:21:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.028+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b015f9b0 con 0x7f55b00d23c0 2024-06-29T14:21:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.028+0000 7f55a67fc640 1 -- 172.21.15.59:0/2990235154 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f55ac01a070 con 0x7f55b00d23c0 2024-06-29T14:21:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.029+0000 7f55a67fc640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55940762d0 0x7f55940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.029+0000 7f55a67fc640 1 -- 172.21.15.59:0/2990235154 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f55ac096800 con 0x7f55b00d23c0 2024-06-29T14:21:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.029+0000 7f55b56ef640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55940762d0 0x7f55940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.030+0000 7f55b56ef640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55940762d0 0x7f55940787e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55a0006fd0 tx=0x7f55a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.034+0000 7f55a67fc640 1 -- 172.21.15.59:0/2990235154 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f55ac060420 con 0x7f55b00d23c0 2024-06-29T14:21:52.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.227+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f55b00c7370 con 0x7f55b00d23c0 2024-06-29T14:21:52.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.230+0000 7f55a67fc640 1 -- 172.21.15.59:0/2990235154 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7f55ac05fd20 con 0x7f55b00d23c0 2024-06-29T14:21:52.228 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55940762d0 msgr2=0x7f55940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55940762d0 0x7f55940787e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55a0006fd0 tx=0x7f55a0008040 comp rx=0 tx=0).stop 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b00d23c0 msgr2=0x7f55b014c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b00d23c0 0x7f55b014c1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00c3c0 tx=0x7f55ac00c880 comp rx=0 tx=0).stop 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 shutdown_connections 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55940762d0 0x7f55940787e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55b014dfe0 0x7f55b00cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55b015efe0 0x7f55b00ca0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 --2- 172.21.15.59:0/2990235154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55b00d23c0 0x7f55b014c1a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 >> 172.21.15.59:0/2990235154 conn(0x7f55b00d3ee0 msgr2=0x7f55b00b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:52.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 shutdown_connections 2024-06-29T14:21:52.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.232+0000 7f55b8154640 1 -- 172.21.15.59:0/2990235154 wait complete. 2024-06-29T14:21:52.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:21:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 -- 172.21.15.59:0/1020513247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c08c190 msgr2=0x7fdb6c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 --2- 172.21.15.59:0/1020513247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c08c190 0x7fdb6c0cf080 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0cfbc0 tx=0x7fdb5c02aa10 comp rx=0 tx=0).stop 2024-06-29T14:21:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 -- 172.21.15.59:0/1020513247 shutdown_connections 2024-06-29T14:21:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 --2- 172.21.15.59:0/1020513247 >> v2:172.21.15.153:3300/0 conn(0x7fdb6c0ce950 0x7fdb6c0d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 --2- 172.21.15.59:0/1020513247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6c0cf5c0 0x7fdb6c0d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 --2- 172.21.15.59:0/1020513247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c08c190 0x7fdb6c0cf080 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 -- 172.21.15.59:0/1020513247 >> 172.21.15.59:0/1020513247 conn(0x7fdb6c0b5610 msgr2=0x7fdb6c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 -- 172.21.15.59:0/1020513247 shutdown_connections 2024-06-29T14:21:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.361+0000 7fdb71089640 1 -- 172.21.15.59:0/1020513247 wait complete. 2024-06-29T14:21:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.363+0000 7fdb71089640 1 Processor -- start 2024-06-29T14:21:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.363+0000 7fdb71089640 1 -- start start 2024-06-29T14:21:52.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb71089640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6c08c190 0x7fdb6c150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb71089640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c0ce950 0x7fdb6c14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb71089640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb6c0cf5c0 0x7fdb6c14f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb71089640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c162440 con 0x7fdb6c0ce950 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb71089640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6c1622c0 con 0x7fdb6c0cf5c0 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb71089640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6c1625c0 con 0x7fdb6c08c190 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb6a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c0ce950 0x7fdb6c14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb6a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c0ce950 0x7fdb6c14ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39886/0 (socket says 172.21.15.59:39886) 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb6a59c640 1 -- 172.21.15.59:0/1237955439 learned_addr learned my addr 172.21.15.59:0/1237955439 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb6b59e640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb6c0cf5c0 0x7fdb6c14f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.364+0000 7fdb6ad9d640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6c08c190 0x7fdb6c150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb6a59c640 1 -- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6c08c190 msgr2=0x7fdb6c150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb6a59c640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6c08c190 0x7fdb6c150c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb6a59c640 1 -- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb6c0cf5c0 msgr2=0x7fdb6c14f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb6a59c640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb6c0cf5c0 0x7fdb6c14f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb6a59c640 1 -- 172.21.15.59:0/1237955439 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6c14f710 con 0x7fdb6c0ce950 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb6b59e640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb6c0cf5c0 0x7fdb6c14f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb6ad9d640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6c08c190 0x7fdb6c150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb6a59c640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c0ce950 0x7fdb6c14ea90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c970 tx=0x7fdb5400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb637fe640 1 -- 172.21.15.59:0/1237955439 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54007bb0 con 0x7fdb6c0ce950 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb637fe640 1 -- 172.21.15.59:0/1237955439 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb54007e40 con 0x7fdb6c0ce950 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6c14fb90 con 0x7fdb6c0ce950 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb637fe640 1 -- 172.21.15.59:0/1237955439 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400f690 con 0x7fdb6c0ce950 2024-06-29T14:21:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.365+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb6c150150 con 0x7fdb6c0ce950 2024-06-29T14:21:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.366+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb6c1521c0 con 0x7fdb6c0ce950 2024-06-29T14:21:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.367+0000 7fdb637fe640 1 -- 172.21.15.59:0/1237955439 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdb5400f830 con 0x7fdb6c0ce950 2024-06-29T14:21:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.369+0000 7fdb637fe640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdb4c0762d0 0x7fdb4c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.369+0000 7fdb637fe640 1 -- 172.21.15.59:0/1237955439 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fdb54097c60 con 0x7fdb6c0ce950 2024-06-29T14:21:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.369+0000 7fdb6ad9d640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdb4c0762d0 0x7fdb4c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.370+0000 7fdb6ad9d640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdb4c0762d0 0x7fdb4c0787e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c035000 tx=0x7fdb5c004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:52.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.373+0000 7fdb637fe640 1 -- 172.21.15.59:0/1237955439 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdb54014030 con 0x7fdb6c0ce950 2024-06-29T14:21:52.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.564+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdb6c0cf080 con 0x7fdb6c0ce950 2024-06-29T14:21:52.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.564+0000 7fdb637fe640 1 -- 172.21.15.59:0/1237955439 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7fdb54061890 con 0x7fdb6c0ce950 2024-06-29T14:21:52.563 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:21:52.564 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:21:52.564 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:21:52.564 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:21:52.564 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:21:52.564 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:21:52.564 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:21:52.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdb4c0762d0 msgr2=0x7fdb4c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdb4c0762d0 0x7fdb4c0787e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c035000 tx=0x7fdb5c004ab0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c0ce950 msgr2=0x7fdb6c14ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c0ce950 0x7fdb6c14ea90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c970 tx=0x7fdb5400ce30 comp rx=0 tx=0).stop 2024-06-29T14:21:52.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 shutdown_connections 2024-06-29T14:21:52.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdb4c0762d0 0x7fdb4c0787e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb6c0cf5c0 0x7fdb6c14f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb6c0ce950 0x7fdb6c14ea90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 --2- 172.21.15.59:0/1237955439 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6c08c190 0x7fdb6c150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 >> 172.21.15.59:0/1237955439 conn(0x7fdb6c0b5610 msgr2=0x7fdb6c0c9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:52.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 shutdown_connections 2024-06-29T14:21:52.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.567+0000 7fdb71089640 1 -- 172.21.15.59:0/1237955439 wait complete. 2024-06-29T14:21:52.577 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:21:52.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:21:52.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:21:52.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:21:52.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.700+0000 7fb7b884e640 1 -- 172.21.15.59:0/74498635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a0005250 msgr2=0x7fb7a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.700+0000 7fb7b884e640 1 --2- 172.21.15.59:0/74498635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a0005250 0x7fb7a000b9c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40cfd00 tx=0x7fb7a0005830 comp rx=0 tx=0).stop 2024-06-29T14:21:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.700+0000 7fb7b884e640 1 -- 172.21.15.59:0/74498635 shutdown_connections 2024-06-29T14:21:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.700+0000 7fb7b884e640 1 --2- 172.21.15.59:0/74498635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a0005250 0x7fb7a000b9c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.700+0000 7fb7b884e640 1 -- 172.21.15.59:0/74498635 >> 172.21.15.59:0/74498635 conn(0x7fb7b4073e80 msgr2=0x7fb7b4074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.700+0000 7fb7b884e640 1 -- 172.21.15.59:0/74498635 shutdown_connections 2024-06-29T14:21:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.700+0000 7fb7b884e640 1 -- 172.21.15.59:0/74498635 wait complete. 2024-06-29T14:21:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.702+0000 7fb7b884e640 1 Processor -- start 2024-06-29T14:21:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.702+0000 7fb7b884e640 1 -- start start 2024-06-29T14:21:52.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b884e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a0005250 0x7fb7b40cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b884e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40c76d0 0x7fb7b40b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b884e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7b40b5230 0x7fb7b40ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b884e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b40d1bf0 con 0x7fb7b40c76d0 2024-06-29T14:21:52.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b884e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b40d01e0 con 0x7fb7a0005250 2024-06-29T14:21:52.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b884e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b40d1d70 con 0x7fb7b40b5230 2024-06-29T14:21:52.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40c76d0 0x7fb7b40b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40c76d0 0x7fb7b40b4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51930/0 (socket says 172.21.15.59:51930) 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 -- 172.21.15.59:0/3368614256 learned_addr learned my addr 172.21.15.59:0/3368614256 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b259c640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a0005250 0x7fb7b40cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b2d9d640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7b40b5230 0x7fb7b40ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 -- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7b40b5230 msgr2=0x7fb7b40ac680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7b40b5230 0x7fb7b40ac680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 -- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a0005250 msgr2=0x7fb7b40cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a0005250 0x7fb7b40cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 -- 172.21.15.59:0/3368614256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b40b5640 con 0x7fb7b40c76d0 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7a9d9b640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40c76d0 0x7fb7b40b4cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb79c003f00 tx=0x7fb79c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:52.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7ab7fe640 1 -- 172.21.15.59:0/3368614256 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c015070 con 0x7fb7b40c76d0 2024-06-29T14:21:52.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7ab7fe640 1 -- 172.21.15.59:0/3368614256 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb79c011840 con 0x7fb7b40c76d0 2024-06-29T14:21:52.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7ab7fe640 1 -- 172.21.15.59:0/3368614256 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c011c70 con 0x7fb7b40c76d0 2024-06-29T14:21:52.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b40af720 con 0x7fb7b40c76d0 2024-06-29T14:21:52.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.705+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b40afce0 con 0x7fb7b40c76d0 2024-06-29T14:21:52.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.707+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b40ce970 con 0x7fb7b40c76d0 2024-06-29T14:21:52.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.707+0000 7fb7ab7fe640 1 -- 172.21.15.59:0/3368614256 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb79c018070 con 0x7fb7b40c76d0 2024-06-29T14:21:52.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.707+0000 7fb7ab7fe640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7940762d0 0x7fb7940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:52.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.707+0000 7fb7ab7fe640 1 -- 172.21.15.59:0/3368614256 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fb79c097f00 con 0x7fb7b40c76d0 2024-06-29T14:21:52.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.707+0000 7fb7b259c640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7940762d0 0x7fb7940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:52.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.708+0000 7fb7b259c640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7940762d0 0x7fb7940787e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009bd0 tx=0x7fb7a4005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:52.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.711+0000 7fb7ab7fe640 1 -- 172.21.15.59:0/3368614256 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7b40ce970 con 0x7fb7b40c76d0 2024-06-29T14:21:52.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:52.908+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb7b40af4b0 con 0x7fb7b40c76d0 2024-06-29T14:21:53.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.228+0000 7fb7ab7fe640 1 -- 172.21.15.59:0/3368614256 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) ==== 138+0+0 (secure 0 0 0) 0x7fb79c0614d0 con 0x7fb7b40c76d0 2024-06-29T14:21:53.227 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:21:53.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7940762d0 msgr2=0x7fb7940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:53.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7940762d0 0x7fb7940787e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009bd0 tx=0x7fb7a4005f90 comp rx=0 tx=0).stop 2024-06-29T14:21:53.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40c76d0 msgr2=0x7fb7b40b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:53.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40c76d0 0x7fb7b40b4cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb79c003f00 tx=0x7fb79c001630 comp rx=0 tx=0).stop 2024-06-29T14:21:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 shutdown_connections 2024-06-29T14:21:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7940762d0 0x7fb7940787e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7b40b5230 0x7fb7b40ac680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40c76d0 0x7fb7b40b4cf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 --2- 172.21.15.59:0/3368614256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a0005250 0x7fb7b40cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 >> 172.21.15.59:0/3368614256 conn(0x7fb7b4073e80 msgr2=0x7fb7b40b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 shutdown_connections 2024-06-29T14:21:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.231+0000 7fb7b884e640 1 -- 172.21.15.59:0/3368614256 wait complete. 2024-06-29T14:21:53.245 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-06-29T14:21:53.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.414+0000 7fa6550df640 1 -- 172.21.15.59:0/1433646479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa63c005250 msgr2=0x7fa63c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:53.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.414+0000 7fa6550df640 1 --2- 172.21.15.59:0/1433646479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa63c005250 0x7fa63c00b9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c610 tx=0x7fa63c005830 comp rx=0 tx=0).stop 2024-06-29T14:21:53.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.414+0000 7fa6550df640 1 -- 172.21.15.59:0/1433646479 shutdown_connections 2024-06-29T14:21:53.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.414+0000 7fa6550df640 1 --2- 172.21.15.59:0/1433646479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa63c005250 0x7fa63c00b9c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:53.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.414+0000 7fa6550df640 1 -- 172.21.15.59:0/1433646479 >> 172.21.15.59:0/1433646479 conn(0x7fa650073ef0 msgr2=0x7fa650074300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:53.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.414+0000 7fa6550df640 1 -- 172.21.15.59:0/1433646479 shutdown_connections 2024-06-29T14:21:53.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.414+0000 7fa6550df640 1 -- 172.21.15.59:0/1433646479 wait complete. 2024-06-29T14:21:53.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.416+0000 7fa6550df640 1 Processor -- start 2024-06-29T14:21:53.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.416+0000 7fa6550df640 1 -- start start 2024-06-29T14:21:53.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa6550df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa63c005250 0x7fa650130d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:53.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa6550df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6501429f0 0x7fa650144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa6550df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa650144f60 0x7fa650133050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa6550df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa650159c10 con 0x7fa6501429f0 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa6550df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa650159a90 con 0x7fa63c005250 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa6550df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa65015f900 con 0x7fa650144f60 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6501429f0 0x7fa650144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6501429f0 0x7fa650144a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51960/0 (socket says 172.21.15.59:51960) 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 -- 172.21.15.59:0/672469080 learned_addr learned my addr 172.21.15.59:0/672469080 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64f59e640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa650144f60 0x7fa650133050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64ed9d640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa63c005250 0x7fa650130d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 -- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa650144f60 msgr2=0x7fa650133050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:53.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa650144f60 0x7fa650133050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 -- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa63c005250 msgr2=0x7fa650130d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa63c005250 0x7fa650130d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 -- 172.21.15.59:0/672469080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa650145370 con 0x7fa6501429f0 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64ed9d640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa63c005250 0x7fa650130d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.418+0000 7fa64e59c640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6501429f0 0x7fa650144a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa638003e10 tx=0x7fa638001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.419+0000 7fa6477fe640 1 -- 172.21.15.59:0/672469080 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638011640 con 0x7fa6501429f0 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.419+0000 7fa6477fe640 1 -- 172.21.15.59:0/672469080 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa638016020 con 0x7fa6501429f0 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.419+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa650141f50 con 0x7fa6501429f0 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.419+0000 7fa6477fe640 1 -- 172.21.15.59:0/672469080 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638011a60 con 0x7fa6501429f0 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.419+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa650144140 con 0x7fa6501429f0 2024-06-29T14:21:53.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.420+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa650143bd0 con 0x7fa6501429f0 2024-06-29T14:21:53.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.421+0000 7fa6477fe640 1 -- 172.21.15.59:0/672469080 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa638011c00 con 0x7fa6501429f0 2024-06-29T14:21:53.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.421+0000 7fa6477fe640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa630076300 0x7fa630078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:53.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.421+0000 7fa6477fe640 1 -- 172.21.15.59:0/672469080 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5227+0+0 (secure 0 0 0) 0x7fa638097f90 con 0x7fa6501429f0 2024-06-29T14:21:53.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.421+0000 7fa64ed9d640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa630076300 0x7fa630078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:53.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.422+0000 7fa64ed9d640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa630076300 0x7fa630078810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa648009460 tx=0x7fa648005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:53.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.426+0000 7fa6477fe640 1 -- 172.21.15.59:0/672469080 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa638020020 con 0x7fa6501429f0 2024-06-29T14:21:53.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:53.617+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fa650160350 con 0x7fa6501429f0 2024-06-29T14:21:54.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.231+0000 7fa6477fe640 1 -- 172.21.15.59:0/672469080 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v20) ==== 171+0+0 (secure 0 0 0) 0x7fa650145370 con 0x7fa6501429f0 2024-06-29T14:21:54.229 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:21:54.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.234+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa630076300 msgr2=0x7fa630078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:54.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.234+0000 7fa6550df640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa630076300 0x7fa630078810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa648009460 tx=0x7fa648005f90 comp rx=0 tx=0).stop 2024-06-29T14:21:54.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6501429f0 msgr2=0x7fa650144a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:54.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6501429f0 0x7fa650144a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa638003e10 tx=0x7fa638001630 comp rx=0 tx=0).stop 2024-06-29T14:21:54.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 shutdown_connections 2024-06-29T14:21:54.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa630076300 0x7fa630078810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa650144f60 0x7fa650133050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6501429f0 0x7fa650144a20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 --2- 172.21.15.59:0/672469080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa63c005250 0x7fa650130d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 >> 172.21.15.59:0/672469080 conn(0x7fa650073ef0 msgr2=0x7fa650074300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:54.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 shutdown_connections 2024-06-29T14:21:54.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.235+0000 7fa6550df640 1 -- 172.21.15.59:0/672469080 wait complete. 2024-06-29T14:21:54.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:21:54.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.389+0000 7fb57f73c640 1 -- 172.21.15.59:0/4145772792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781448e0 msgr2=0x7fb5780cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:54.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.389+0000 7fb57f73c640 1 --2- 172.21.15.59:0/4145772792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781448e0 0x7fb5780cfbe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb5780ccee0 tx=0x7fb57002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.389+0000 7fb57f73c640 1 -- 172.21.15.59:0/4145772792 shutdown_connections 2024-06-29T14:21:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.389+0000 7fb57f73c640 1 --2- 172.21.15.59:0/4145772792 >> v2:172.21.15.153:3300/0 conn(0x7fb5780d0120 0x7fb5780cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.389+0000 7fb57f73c640 1 --2- 172.21.15.59:0/4145772792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781448e0 0x7fb5780cfbe0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.389+0000 7fb57f73c640 1 -- 172.21.15.59:0/4145772792 >> 172.21.15.59:0/4145772792 conn(0x7fb5780b2ab0 msgr2=0x7fb5780b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.389+0000 7fb57f73c640 1 -- 172.21.15.59:0/4145772792 shutdown_connections 2024-06-29T14:21:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.389+0000 7fb57f73c640 1 -- 172.21.15.59:0/4145772792 wait complete. 2024-06-29T14:21:54.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.391+0000 7fb57f73c640 1 Processor -- start 2024-06-29T14:21:54.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.391+0000 7fb57f73c640 1 -- start start 2024-06-29T14:21:54.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57f73c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5780cd7c0 0x7fb57815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:54.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57f73c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5780d0120 0x7fb5781f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:54.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57f73c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5781448e0 0x7fb57813d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:54.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57f73c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb57815fe30 con 0x7fb5780cd7c0 2024-06-29T14:21:54.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57f73c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb57806dc90 con 0x7fb5780d0120 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57f73c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb57815ffb0 con 0x7fb5781448e0 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57d4d8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5780cd7c0 0x7fb57815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57d4d8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5780cd7c0 0x7fb57815e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51970/0 (socket says 172.21.15.59:51970) 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57d4d8640 1 -- 172.21.15.59:0/2484375127 learned_addr learned my addr 172.21.15.59:0/2484375127 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57dcd9640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5781448e0 0x7fb57813d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.392+0000 7fb57ccd7640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5780d0120 0x7fb5781f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb57d4d8640 1 -- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5781448e0 msgr2=0x7fb57813d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb57d4d8640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5781448e0 0x7fb57813d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb57d4d8640 1 -- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5780d0120 msgr2=0x7fb5781f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb57d4d8640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5780d0120 0x7fb5781f2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb57d4d8640 1 -- 172.21.15.59:0/2484375127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5781f36e0 con 0x7fb5780cd7c0 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb57d4d8640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5780cd7c0 0x7fb57815e3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5700099c0 tx=0x7fb5700039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb575ffb640 1 -- 172.21.15.59:0/2484375127 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57003e030 con 0x7fb5780cd7c0 2024-06-29T14:21:54.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb575ffb640 1 -- 172.21.15.59:0/2484375127 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5700047a0 con 0x7fb5780cd7c0 2024-06-29T14:21:54.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5781f39c0 con 0x7fb5780cd7c0 2024-06-29T14:21:54.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb575ffb640 1 -- 172.21.15.59:0/2484375127 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb570035040 con 0x7fb5780cd7c0 2024-06-29T14:21:54.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.393+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5780a96f0 con 0x7fb5780cd7c0 2024-06-29T14:21:54.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.394+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5780cfdf0 con 0x7fb5780cd7c0 2024-06-29T14:21:54.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.395+0000 7fb575ffb640 1 -- 172.21.15.59:0/2484375127 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb570041030 con 0x7fb5780cd7c0 2024-06-29T14:21:54.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.396+0000 7fb575ffb640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5640762d0 0x7fb5640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:54.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.396+0000 7fb575ffb640 1 -- 172.21.15.59:0/2484375127 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5609+0+0 (secure 0 0 0) 0x7fb5700b7ef0 con 0x7fb5780cd7c0 2024-06-29T14:21:54.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.396+0000 7fb57ccd7640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5640762d0 0x7fb5640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:54.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.397+0000 7fb57ccd7640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5640762d0 0x7fb5640787e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb5780afd40 tx=0x7fb56c00b2e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:54.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.400+0000 7fb575ffb640 1 -- 172.21.15.59:0/2484375127 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb5780cfdf0 con 0x7fb5780cd7c0 2024-06-29T14:21:54.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:54.635+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb5780cf320 con 0x7fb5780cd7c0 2024-06-29T14:21:55.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.260+0000 7fb575ffb640 1 -- 172.21.15.59:0/2484375127 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fb5780cf320 con 0x7fb5780cd7c0 2024-06-29T14:21:55.258 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 2 and data pool 3 2024-06-29T14:21:55.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5640762d0 msgr2=0x7fb5640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:55.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5640762d0 0x7fb5640787e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb5780afd40 tx=0x7fb56c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5780cd7c0 msgr2=0x7fb57815e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:55.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5780cd7c0 0x7fb57815e3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5700099c0 tx=0x7fb5700039e0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 shutdown_connections 2024-06-29T14:21:55.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5640762d0 0x7fb5640787e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5781448e0 0x7fb57813d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5780d0120 0x7fb5781f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 --2- 172.21.15.59:0/2484375127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5780cd7c0 0x7fb57815e3c0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5700099c0 tx=0x7fb5700039e0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.265+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 >> 172.21.15.59:0/2484375127 conn(0x7fb5780b2ab0 msgr2=0x7fb578144e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:55.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.266+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 shutdown_connections 2024-06-29T14:21:55.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.266+0000 7fb57f73c640 1 -- 172.21.15.59:0/2484375127 wait complete. 2024-06-29T14:21:55.292 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-06-29T14:21:55.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-06-29T14:21:55.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.425+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/3545998474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ea40b2980 msgr2=0x7f9ea40b6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.425+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/3545998474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ea40b2980 0x7f9ea40b6f70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009830 tx=0x7f9e8c02a480 comp rx=0 tx=0).stop 2024-06-29T14:21:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.425+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/3545998474 shutdown_connections 2024-06-29T14:21:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.425+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/3545998474 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ea40d2eb0 0x7f9ea40b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.425+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/3545998474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ea40b2980 0x7f9ea40b6f70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.425+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/3545998474 >> 172.21.15.59:0/3545998474 conn(0x7f9ea40d0f90 msgr2=0x7f9ea40d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.425+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/3545998474 shutdown_connections 2024-06-29T14:21:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.425+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/3545998474 wait complete. 2024-06-29T14:21:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.427+0000 7f9ea8b0a640 1 Processor -- start 2024-06-29T14:21:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.427+0000 7f9ea8b0a640 1 -- start start 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.428+0000 7f9ea8b0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ea40b2980 0x7f9ea4157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.428+0000 7f9ea8b0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea40d1c30 0x7f9ea41f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.428+0000 7f9ea8b0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ea40d2eb0 0x7f9ea4155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.428+0000 7f9ea8b0a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea40b4730 con 0x7f9ea40d1c30 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.428+0000 7f9ea8b0a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea40b45b0 con 0x7f9ea40b2980 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.428+0000 7f9ea8b0a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea40b48b0 con 0x7f9ea40d2eb0 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea40d1c30 0x7f9ea41f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea40d1c30 0x7f9ea41f4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51972/0 (socket says 172.21.15.59:51972) 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 -- 172.21.15.59:0/773482818 learned_addr learned my addr 172.21.15.59:0/773482818 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea2d9d640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ea40d2eb0 0x7f9ea4155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea259c640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ea40b2980 0x7f9ea4157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 -- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ea40d2eb0 msgr2=0x7f9ea4155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ea40d2eb0 0x7f9ea4155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 -- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ea40b2980 msgr2=0x7f9ea4157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ea40b2980 0x7f9ea4157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 -- 172.21.15.59:0/773482818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea415d110 con 0x7f9ea40d1c30 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea1d9b640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea40d1c30 0x7f9ea41f4790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009950 tx=0x7f9e8c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9e9affd640 1 -- 172.21.15.59:0/773482818 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c005c90 con 0x7f9ea40d1c30 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9e9affd640 1 -- 172.21.15.59:0/773482818 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e8c031e50 con 0x7f9ea40d1c30 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea415d3f0 con 0x7f9ea40d1c30 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9e9affd640 1 -- 172.21.15.59:0/773482818 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c005260 con 0x7f9ea40d1c30 2024-06-29T14:21:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.429+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea415d9b0 con 0x7f9ea40d1c30 2024-06-29T14:21:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.430+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea40d2bf0 con 0x7f9ea40d1c30 2024-06-29T14:21:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.431+0000 7f9e9affd640 1 -- 172.21.15.59:0/773482818 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9e8c005400 con 0x7f9ea40d1c30 2024-06-29T14:21:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.431+0000 7f9e9affd640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e840762d0 0x7f9e840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.432+0000 7f9e9affd640 1 -- 172.21.15.59:0/773482818 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9e8c0b86e0 con 0x7f9ea40d1c30 2024-06-29T14:21:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.432+0000 7f9ea259c640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e840762d0 0x7f9e840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.432+0000 7f9ea259c640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e840762d0 0x7f9e840787e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c007fd0 tx=0x7f9e9c001370 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:55.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.436+0000 7f9e9affd640 1 -- 172.21.15.59:0/773482818 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9ea40d2bf0 con 0x7f9ea40d1c30 2024-06-29T14:21:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:55.636+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f9ea40b6f70 con 0x7f9ea40d1c30 2024-06-29T14:21:56.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.262+0000 7f9e9affd640 1 -- 172.21.15.59:0/773482818 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v22) ==== 157+0+0 (secure 0 0 0) 0x7f9e8c03f030 con 0x7f9ea40d1c30 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e840762d0 msgr2=0x7f9e840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e840762d0 0x7f9e840787e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c007fd0 tx=0x7f9e9c001370 comp rx=0 tx=0).stop 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea40d1c30 msgr2=0x7f9ea41f4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea40d1c30 0x7f9ea41f4790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009950 tx=0x7f9e8c033c20 comp rx=0 tx=0).stop 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 shutdown_connections 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e840762d0 0x7f9e840787e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ea40d2eb0 0x7f9ea4155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea40d1c30 0x7f9ea41f4790 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 --2- 172.21.15.59:0/773482818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ea40b2980 0x7f9ea4157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.265+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 >> 172.21.15.59:0/773482818 conn(0x7f9ea40d0f90 msgr2=0x7f9ea40b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.266+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 shutdown_connections 2024-06-29T14:21:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.266+0000 7f9ea8b0a640 1 -- 172.21.15.59:0/773482818 wait complete. 2024-06-29T14:21:56.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:21:56.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.413+0000 7f20c9d4a640 1 -- 172.21.15.59:0/1149334648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b4ba0 msgr2=0x7f20c40d2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:56.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.413+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/1149334648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b4ba0 0x7f20c40d2250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f20bc005fd0 tx=0x7f20bc02a8e0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.413+0000 7f20c9d4a640 1 -- 172.21.15.59:0/1149334648 shutdown_connections 2024-06-29T14:21:56.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.413+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/1149334648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b4ba0 0x7f20c40d2250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.413+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/1149334648 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20c40cf4e0 0x7f20c415e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.413+0000 7f20c9d4a640 1 -- 172.21.15.59:0/1149334648 >> 172.21.15.59:0/1149334648 conn(0x7f20c40d4050 msgr2=0x7f20c40b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:56.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.413+0000 7f20c9d4a640 1 -- 172.21.15.59:0/1149334648 shutdown_connections 2024-06-29T14:21:56.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.413+0000 7f20c9d4a640 1 -- 172.21.15.59:0/1149334648 wait complete. 2024-06-29T14:21:56.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.415+0000 7f20c9d4a640 1 Processor -- start 2024-06-29T14:21:56.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.415+0000 7f20c9d4a640 1 -- start start 2024-06-29T14:21:56.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c9d4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b2740 0x7f20c40cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:56.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c9d4a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c40cf4e0 0x7f20c413d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c9d4a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20c4158e60 0x7f20c413dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c9d4a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f20c40d2c00 con 0x7f20c40b2740 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c9d4a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f20c40d2a80 con 0x7f20c40cf4e0 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c9d4a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f20c40d2d80 con 0x7f20c4158e60 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b2740 0x7f20c40cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b2740 0x7f20c40cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51990/0 (socket says 172.21.15.59:51990) 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c37fe640 1 -- 172.21.15.59:0/894659588 learned_addr learned my addr 172.21.15.59:0/894659588 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20c3fff640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20c4158e60 0x7f20c413dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.416+0000 7f20baffd640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c40cf4e0 0x7f20c413d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c37fe640 1 -- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20c4158e60 msgr2=0x7f20c413dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c37fe640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20c4158e60 0x7f20c413dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c37fe640 1 -- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c40cf4e0 msgr2=0x7f20c413d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c37fe640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c40cf4e0 0x7f20c413d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c37fe640 1 -- 172.21.15.59:0/894659588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c4157750 con 0x7f20c40b2740 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c37fe640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b2740 0x7f20c40cca50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20b400d6d0 tx=0x7f20b400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c0ff9640 1 -- 172.21.15.59:0/894659588 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4017070 con 0x7f20c40b2740 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c0ff9640 1 -- 172.21.15.59:0/894659588 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20b4005330 con 0x7f20c40b2740 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c4157a30 con 0x7f20c40b2740 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c0ff9640 1 -- 172.21.15.59:0/894659588 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4009820 con 0x7f20c40b2740 2024-06-29T14:21:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.417+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c4157ff0 con 0x7f20c40b2740 2024-06-29T14:21:56.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.418+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c40d31c0 con 0x7f20c40b2740 2024-06-29T14:21:56.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.419+0000 7f20c0ff9640 1 -- 172.21.15.59:0/894659588 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f20b4009bd0 con 0x7f20c40b2740 2024-06-29T14:21:56.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.419+0000 7f20c0ff9640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f20a40762d0 0x7f20a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:56.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.419+0000 7f20c0ff9640 1 -- 172.21.15.59:0/894659588 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6007+0+0 (secure 0 0 0) 0x7f20b4097b70 con 0x7f20c40b2740 2024-06-29T14:21:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.420+0000 7f20baffd640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f20a40762d0 0x7f20a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.420+0000 7f20baffd640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f20a40762d0 0x7f20a40787e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20c415ca80 tx=0x7f20ac009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:56.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.424+0000 7f20c0ff9640 1 -- 172.21.15.59:0/894659588 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f20b4060cd0 con 0x7f20c40b2740 2024-06-29T14:21:56.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:56.616+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f20c40b3490 con 0x7f20c40b2740 2024-06-29T14:21:58.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.290+0000 7f20c0ff9640 1 -- 172.21.15.59:0/894659588 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v24) ==== 228+0+0 (secure 0 0 0) 0x7f20b4069310 con 0x7f20c40b2740 2024-06-29T14:21:58.289 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data_ec' created 2024-06-29T14:21:58.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f20a40762d0 msgr2=0x7f20a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:58.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f20a40762d0 0x7f20a40787e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20c415ca80 tx=0x7f20ac009060 comp rx=0 tx=0).stop 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b2740 msgr2=0x7f20c40cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b2740 0x7f20c40cca50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20b400d6d0 tx=0x7f20b400db90 comp rx=0 tx=0).stop 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 shutdown_connections 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f20a40762d0 0x7f20a40787e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20c4158e60 0x7f20c413dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c40cf4e0 0x7f20c413d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 --2- 172.21.15.59:0/894659588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c40b2740 0x7f20c40cca50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 >> 172.21.15.59:0/894659588 conn(0x7f20c40d4050 msgr2=0x7f20c40ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 shutdown_connections 2024-06-29T14:21:58.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.294+0000 7f20c9d4a640 1 -- 172.21.15.59:0/894659588 wait complete. 2024-06-29T14:21:58.315 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-06-29T14:21:58.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.445+0000 7f81dc69e640 1 -- 172.21.15.59:0/1209824391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d415f1e0 msgr2=0x7f81d40b3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:58.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.445+0000 7f81dc69e640 1 --2- 172.21.15.59:0/1209824391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d415f1e0 0x7f81d40b3be0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009830 tx=0x7f81cc02a510 comp rx=0 tx=0).stop 2024-06-29T14:21:58.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.445+0000 7f81dc69e640 1 -- 172.21.15.59:0/1209824391 shutdown_connections 2024-06-29T14:21:58.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.445+0000 7f81dc69e640 1 --2- 172.21.15.59:0/1209824391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d415f1e0 0x7f81d40b3be0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:58.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.445+0000 7f81dc69e640 1 --2- 172.21.15.59:0/1209824391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d40745a0 0x7f81d4143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:58.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.445+0000 7f81dc69e640 1 -- 172.21.15.59:0/1209824391 >> 172.21.15.59:0/1209824391 conn(0x7f81d40cd630 msgr2=0x7f81d40cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:58.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.445+0000 7f81dc69e640 1 -- 172.21.15.59:0/1209824391 shutdown_connections 2024-06-29T14:21:58.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.445+0000 7f81dc69e640 1 -- 172.21.15.59:0/1209824391 wait complete. 2024-06-29T14:21:58.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.447+0000 7f81dc69e640 1 Processor -- start 2024-06-29T14:21:58.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.447+0000 7f81dc69e640 1 -- start start 2024-06-29T14:21:58.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.448+0000 7f81dc69e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d41440a0 0x7f81d415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:58.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.448+0000 7f81dc69e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81d415f1e0 0x7f81d40ab930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:58.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.448+0000 7f81dc69e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d413e720 0x7f81d4133b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.448+0000 7f81dc69e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f81d40b45f0 con 0x7f81d415f1e0 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.448+0000 7f81dc69e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81d40b4470 con 0x7f81d413e720 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.448+0000 7f81dc69e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f81d40b4770 con 0x7f81d41440a0 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.448+0000 7f81d9c39640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81d415f1e0 0x7f81d40ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d9c39640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81d415f1e0 0x7f81d40ab930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52002/0 (socket says 172.21.15.59:52002) 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d9c39640 1 -- 172.21.15.59:0/391895038 learned_addr learned my addr 172.21.15.59:0/391895038 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81dac3b640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d413e720 0x7f81d4133b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81da43a640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d41440a0 0x7f81d415a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d9c39640 1 -- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d41440a0 msgr2=0x7f81d415a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d9c39640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d41440a0 0x7f81d415a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d9c39640 1 -- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d413e720 msgr2=0x7f81d4133b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d9c39640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d413e720 0x7f81d4133b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d9c39640 1 -- 172.21.15.59:0/391895038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d40ac2c0 con 0x7f81d415f1e0 2024-06-29T14:21:58.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d9c39640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81d415f1e0 0x7f81d40ab930 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f81c800e9f0 tx=0x7f81c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:58.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d2ffd640 1 -- 172.21.15.59:0/391895038 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c800cc60 con 0x7f81d415f1e0 2024-06-29T14:21:58.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d2ffd640 1 -- 172.21.15.59:0/391895038 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81c800ce00 con 0x7f81d415f1e0 2024-06-29T14:21:58.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d40ac5a0 con 0x7f81d415f1e0 2024-06-29T14:21:58.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81d2ffd640 1 -- 172.21.15.59:0/391895038 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8004280 con 0x7f81d415f1e0 2024-06-29T14:21:58.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.449+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d40acb60 con 0x7f81d415f1e0 2024-06-29T14:21:58.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.450+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d40b4d00 con 0x7f81d415f1e0 2024-06-29T14:21:58.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.451+0000 7f81d2ffd640 1 -- 172.21.15.59:0/391895038 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f81c80035f0 con 0x7f81d415f1e0 2024-06-29T14:21:58.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.452+0000 7f81d2ffd640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f81c00762d0 0x7f81c00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:58.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.452+0000 7f81d2ffd640 1 -- 172.21.15.59:0/391895038 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7f81c80984b0 con 0x7f81d415f1e0 2024-06-29T14:21:58.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.452+0000 7f81da43a640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f81c00762d0 0x7f81c00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:58.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.453+0000 7f81da43a640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f81c00762d0 0x7f81c00787e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f81c4002200 tx=0x7f81c40019f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:58.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.456+0000 7f81d2ffd640 1 -- 172.21.15.59:0/391895038 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f81c8054af0 con 0x7f81d415f1e0 2024-06-29T14:21:58.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:58.646+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f81d40cffb0 con 0x7f81d415f1e0 2024-06-29T14:21:59.223 INFO:tasks.ceph.mds.b.smithi153.stderr:2024-06-29T14:21:59.225+0000 7f05276ca640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:21:59.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.292+0000 7f81d2ffd640 1 -- 172.21.15.59:0/391895038 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v25) ==== 169+0+0 (secure 0 0 0) 0x7f81c8061410 con 0x7f81d415f1e0 2024-06-29T14:21:59.290 INFO:teuthology.orchestra.run.smithi059.stderr:set pool 4 allow_ec_overwrites to true 2024-06-29T14:21:59.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f81c00762d0 msgr2=0x7f81c00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:59.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f81c00762d0 0x7f81c00787e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f81c4002200 tx=0x7f81c40019f0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81d415f1e0 msgr2=0x7f81d40ab930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:59.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81d415f1e0 0x7f81d40ab930 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f81c800e9f0 tx=0x7f81c800eeb0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 shutdown_connections 2024-06-29T14:21:59.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f81c00762d0 0x7f81c00787e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81d413e720 0x7f81d4133b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81d415f1e0 0x7f81d40ab930 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 --2- 172.21.15.59:0/391895038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f81d41440a0 0x7f81d415a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 >> 172.21.15.59:0/391895038 conn(0x7f81d40cd630 msgr2=0x7f81d40b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:59.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 shutdown_connections 2024-06-29T14:21:59.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.298+0000 7f81dc69e640 1 -- 172.21.15.59:0/391895038 wait complete. 2024-06-29T14:21:59.325 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-06-29T14:21:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 -- 172.21.15.59:0/2091942921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280b3ed0 msgr2=0x7fdf2815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 --2- 172.21.15.59:0/2091942921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280b3ed0 0x7fdf2815e8d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdf280d04f0 tx=0x7fdf1c02a560 comp rx=0 tx=0).stop 2024-06-29T14:21:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 -- 172.21.15.59:0/2091942921 shutdown_connections 2024-06-29T14:21:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 --2- 172.21.15.59:0/2091942921 >> v2:172.21.15.153:3300/0 conn(0x7fdf2815ee10 0x7fdf2815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 --2- 172.21.15.59:0/2091942921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280b3ed0 0x7fdf2815e8d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 --2- 172.21.15.59:0/2091942921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf280d23c0 0x7fdf280b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 -- 172.21.15.59:0/2091942921 >> 172.21.15.59:0/2091942921 conn(0x7fdf280d3ee0 msgr2=0x7fdf280d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:21:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 -- 172.21.15.59:0/2091942921 shutdown_connections 2024-06-29T14:21:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.454+0000 7fdf2e562640 1 -- 172.21.15.59:0/2091942921 wait complete. 2024-06-29T14:21:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.455+0000 7fdf2e562640 1 Processor -- start 2024-06-29T14:21:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.456+0000 7fdf2e562640 1 -- start start 2024-06-29T14:21:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf2e562640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280b3ed0 0x7fdf28157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf2e562640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280d23c0 0x7fdf2815ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf2e562640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf2815ee10 0x7fdf281ea310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf2e562640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdf28160070 con 0x7fdf280d23c0 2024-06-29T14:21:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf2e562640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2815fef0 con 0x7fdf280b3ed0 2024-06-29T14:21:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf2e562640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdf281601f0 con 0x7fdf2815ee10 2024-06-29T14:21:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280d23c0 0x7fdf2815ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280d23c0 0x7fdf2815ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52014/0 (socket says 172.21.15.59:52014) 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf277fe640 1 -- 172.21.15.59:0/829728498 learned_addr learned my addr 172.21.15.59:0/829728498 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf2caff640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf2815ee10 0x7fdf281ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf27fff640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280b3ed0 0x7fdf28157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf277fe640 1 -- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf2815ee10 msgr2=0x7fdf281ea310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf277fe640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf2815ee10 0x7fdf281ea310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf277fe640 1 -- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280b3ed0 msgr2=0x7fdf28157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf277fe640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280b3ed0 0x7fdf28157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.457+0000 7fdf277fe640 1 -- 172.21.15.59:0/829728498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf2815a3b0 con 0x7fdf280d23c0 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.458+0000 7fdf277fe640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280d23c0 0x7fdf2815ce50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c034040 tx=0x7fdf1c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.458+0000 7fdf24ff9640 1 -- 172.21.15.59:0/829728498 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c03e030 con 0x7fdf280d23c0 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.458+0000 7fdf24ff9640 1 -- 172.21.15.59:0/829728498 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf1c0047a0 con 0x7fdf280d23c0 2024-06-29T14:21:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.458+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf2815af60 con 0x7fdf280d23c0 2024-06-29T14:21:59.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.458+0000 7fdf24ff9640 1 -- 172.21.15.59:0/829728498 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c035040 con 0x7fdf280d23c0 2024-06-29T14:21:59.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.458+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf2815b520 con 0x7fdf280d23c0 2024-06-29T14:21:59.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.459+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf280b3990 con 0x7fdf280d23c0 2024-06-29T14:21:59.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.460+0000 7fdf24ff9640 1 -- 172.21.15.59:0/829728498 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdf1c041030 con 0x7fdf280d23c0 2024-06-29T14:21:59.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.460+0000 7fdf24ff9640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf140762d0 0x7fdf140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:21:59.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.460+0000 7fdf24ff9640 1 -- 172.21.15.59:0/829728498 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6500+0+0 (secure 0 0 0) 0x7fdf1c0b8310 con 0x7fdf280d23c0 2024-06-29T14:21:59.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.461+0000 7fdf27fff640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf140762d0 0x7fdf140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:21:59.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.461+0000 7fdf27fff640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf140762d0 0x7fdf140787e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdf280d2e40 tx=0x7fdf18005f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:21:59.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.465+0000 7fdf24ff9640 1 -- 172.21.15.59:0/829728498 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf1c081270 con 0x7fdf280d23c0 2024-06-29T14:21:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:21:59.697+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fdf280b7d40 con 0x7fdf280d23c0 2024-06-29T14:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.307+0000 7fdf24ff9640 1 -- 172.21.15.59:0/829728498 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7fdf1c085570 con 0x7fdf280d23c0 2024-06-29T14:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:added data pool 4 to fsmap 2024-06-29T14:22:00.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf140762d0 msgr2=0x7fdf140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf140762d0 0x7fdf140787e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdf280d2e40 tx=0x7fdf18005f10 comp rx=0 tx=0).stop 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280d23c0 msgr2=0x7fdf2815ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280d23c0 0x7fdf2815ce50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c034040 tx=0x7fdf1c0039e0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 shutdown_connections 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf140762d0 0x7fdf140787e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf2815ee10 0x7fdf281ea310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf280d23c0 0x7fdf2815ce50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 --2- 172.21.15.59:0/829728498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280b3ed0 0x7fdf28157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 >> 172.21.15.59:0/829728498 conn(0x7fdf280d3ee0 msgr2=0x7fdf280b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 shutdown_connections 2024-06-29T14:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.310+0000 7fdf2e562640 1 -- 172.21.15.59:0/829728498 wait complete. 2024-06-29T14:22:00.322 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:00.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.449+0000 7fc4fca46640 1 -- 172.21.15.59:0/2576109655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80b2270 msgr2=0x7fc4f815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.449+0000 7fc4fca46640 1 --2- 172.21.15.59:0/2576109655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80b2270 0x7fc4f815bc20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8074c50 tx=0x7fc4e802a9e0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.449+0000 7fc4fca46640 1 -- 172.21.15.59:0/2576109655 shutdown_connections 2024-06-29T14:22:00.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.449+0000 7fc4fca46640 1 --2- 172.21.15.59:0/2576109655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4f80b2fa0 0x7fc4f80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.449+0000 7fc4fca46640 1 --2- 172.21.15.59:0/2576109655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80b2270 0x7fc4f815bc20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.449+0000 7fc4fca46640 1 -- 172.21.15.59:0/2576109655 >> 172.21.15.59:0/2576109655 conn(0x7fc4f8073e10 msgr2=0x7fc4f8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:00.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.449+0000 7fc4fca46640 1 -- 172.21.15.59:0/2576109655 shutdown_connections 2024-06-29T14:22:00.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.449+0000 7fc4fca46640 1 -- 172.21.15.59:0/2576109655 wait complete. 2024-06-29T14:22:00.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.451+0000 7fc4fca46640 1 Processor -- start 2024-06-29T14:22:00.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.451+0000 7fc4fca46640 1 -- start start 2024-06-29T14:22:00.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4fca46640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4f80b2270 0x7fc4f814c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:00.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4fca46640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4f80b2fa0 0x7fc4f815e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:00.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4fca46640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80d1a60 0x7fc4f815ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:00.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4fca46640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f80b6cc0 con 0x7fc4f80d1a60 2024-06-29T14:22:00.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4fca46640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f80b6b40 con 0x7fc4f80b2270 2024-06-29T14:22:00.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4fca46640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f80cfe90 con 0x7fc4f80b2fa0 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80d1a60 0x7fc4f815ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80d1a60 0x7fc4f815ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52022/0 (socket says 172.21.15.59:52022) 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 -- 172.21.15.59:0/783402566 learned_addr learned my addr 172.21.15.59:0/783402566 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f5d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4f80b2fa0 0x7fc4f815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f659c640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4f80b2270 0x7fc4f814c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 -- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4f80b2fa0 msgr2=0x7fc4f815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4f80b2fa0 0x7fc4f815e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 -- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4f80b2270 msgr2=0x7fc4f814c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4f80b2270 0x7fc4f814c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 -- 172.21.15.59:0/783402566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f80af880 con 0x7fc4f80d1a60 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.453+0000 7fc4f6d9d640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80d1a60 0x7fc4f815ed40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000b3e0 tx=0x7fc4f000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.454+0000 7fc4eeffd640 1 -- 172.21.15.59:0/783402566 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0015030 con 0x7fc4f80d1a60 2024-06-29T14:22:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.454+0000 7fc4eeffd640 1 -- 172.21.15.59:0/783402566 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4f0004110 con 0x7fc4f80d1a60 2024-06-29T14:22:00.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.454+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f80afb60 con 0x7fc4f80d1a60 2024-06-29T14:22:00.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.454+0000 7fc4eeffd640 1 -- 172.21.15.59:0/783402566 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0011630 con 0x7fc4f80d1a60 2024-06-29T14:22:00.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.454+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f80b0120 con 0x7fc4f80d1a60 2024-06-29T14:22:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.455+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f80d0420 con 0x7fc4f80d1a60 2024-06-29T14:22:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.455+0000 7fc4eeffd640 1 -- 172.21.15.59:0/783402566 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc4f0018070 con 0x7fc4f80d1a60 2024-06-29T14:22:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.456+0000 7fc4eeffd640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc4d80762d0 0x7fc4d80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.456+0000 7fc4eeffd640 1 -- 172.21.15.59:0/783402566 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fc4f0097500 con 0x7fc4f80d1a60 2024-06-29T14:22:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.456+0000 7fc4f659c640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc4d80762d0 0x7fc4d80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.457+0000 7fc4f659c640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc4d80762d0 0x7fc4d80787e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8035000 tx=0x7fc4e8004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:00.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.461+0000 7fc4eeffd640 1 -- 172.21.15.59:0/783402566 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc4f0060430 con 0x7fc4f80d1a60 2024-06-29T14:22:00.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.692+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4f815b470 con 0x7fc4f80d1a60 2024-06-29T14:22:00.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.693+0000 7fc4eeffd640 1 -- 172.21.15.59:0/783402566 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5222 (secure 0 0 0) 0x7fc4f0064730 con 0x7fc4f80d1a60 2024-06-29T14:22:00.691 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:00.691 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":7,"btime":"2024-06-29T14:22:00:301000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:21:59.698797+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:00.691 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 7 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.694+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc4d80762d0 msgr2=0x7fc4d80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.694+0000 7fc4fca46640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc4d80762d0 0x7fc4d80787e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8035000 tx=0x7fc4e8004ab0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.694+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80d1a60 msgr2=0x7fc4f815ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.694+0000 7fc4fca46640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80d1a60 0x7fc4f815ed40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000b3e0 tx=0x7fc4f000b8a0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.695+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 shutdown_connections 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.695+0000 7fc4fca46640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc4d80762d0 0x7fc4d80787e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.695+0000 7fc4fca46640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4f80d1a60 0x7fc4f815ed40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.695+0000 7fc4fca46640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4f80b2fa0 0x7fc4f815e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.695+0000 7fc4fca46640 1 --2- 172.21.15.59:0/783402566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4f80b2270 0x7fc4f814c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.695+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 >> 172.21.15.59:0/783402566 conn(0x7fc4f8073e10 msgr2=0x7fc4f8070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:00.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.695+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 shutdown_connections 2024-06-29T14:22:00.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.695+0000 7fc4fca46640 1 -- 172.21.15.59:0/783402566 wait complete. 2024-06-29T14:22:00.704 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:00.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.823+0000 7f82bfc47640 1 -- 172.21.15.59:0/198571463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b8073bd0 msgr2=0x7f82b806ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.823+0000 7f82bfc47640 1 --2- 172.21.15.59:0/198571463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b8073bd0 0x7f82b806ee50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f82b0005fd0 tx=0x7f82b002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.823+0000 7f82bfc47640 1 -- 172.21.15.59:0/198571463 shutdown_connections 2024-06-29T14:22:00.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.823+0000 7f82bfc47640 1 --2- 172.21.15.59:0/198571463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b8073bd0 0x7f82b806ee50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.823+0000 7f82bfc47640 1 --2- 172.21.15.59:0/198571463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b80ce6c0 0x7f82b815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.823+0000 7f82bfc47640 1 -- 172.21.15.59:0/198571463 >> 172.21.15.59:0/198571463 conn(0x7f82b806cef0 msgr2=0x7f82b806d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:00.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.823+0000 7f82bfc47640 1 -- 172.21.15.59:0/198571463 shutdown_connections 2024-06-29T14:22:00.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.823+0000 7f82bfc47640 1 -- 172.21.15.59:0/198571463 wait complete. 2024-06-29T14:22:00.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.825+0000 7f82bfc47640 1 Processor -- start 2024-06-29T14:22:00.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.825+0000 7f82bfc47640 1 -- start start 2024-06-29T14:22:00.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bfc47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b80ce6c0 0x7f82b81582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:00.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bfc47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b8144760 0x7f82b80b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bfc47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b81f6840 0x7f82b81377b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bfc47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f82b806f9c0 con 0x7f82b80ce6c0 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bfc47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82b806f840 con 0x7f82b81f6840 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bfc47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f82b806fb40 con 0x7f82b8144760 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bd9e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b80ce6c0 0x7f82b81582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bd9e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b80ce6c0 0x7f82b81582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52034/0 (socket says 172.21.15.59:52034) 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.826+0000 7f82bd9e3640 1 -- 172.21.15.59:0/3795985468 learned_addr learned my addr 172.21.15.59:0/3795985468 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bd1e2640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b8144760 0x7f82b80b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82be1e4640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b81f6840 0x7f82b81377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bd9e3640 1 -- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b8144760 msgr2=0x7f82b80b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bd9e3640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b8144760 0x7f82b80b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bd9e3640 1 -- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b81f6840 msgr2=0x7f82b81377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bd9e3640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b81f6840 0x7f82b81377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bd9e3640 1 -- 172.21.15.59:0/3795985468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b80b4470 con 0x7f82b80ce6c0 2024-06-29T14:22:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82be1e4640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b81f6840 0x7f82b81377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bd9e3640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b80ce6c0 0x7f82b81582c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f82b400e910 tx=0x7f82b400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3795985468 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b400cd20 con 0x7f82b80ce6c0 2024-06-29T14:22:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3795985468 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82b4010430 con 0x7f82b80ce6c0 2024-06-29T14:22:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3795985468 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4010860 con 0x7f82b80ce6c0 2024-06-29T14:22:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b8136fd0 con 0x7f82b80ce6c0 2024-06-29T14:22:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.827+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b81f74e0 con 0x7f82b80ce6c0 2024-06-29T14:22:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.828+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b806fff0 con 0x7f82b80ce6c0 2024-06-29T14:22:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.829+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3795985468 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f82b401f020 con 0x7f82b80ce6c0 2024-06-29T14:22:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.829+0000 7f82ae7fc640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f829c0762d0 0x7f829c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.829+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3795985468 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f82b4098310 con 0x7f82b80ce6c0 2024-06-29T14:22:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.830+0000 7f82bd1e2640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f829c0762d0 0x7f829c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.830+0000 7f82bd1e2640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f829c0762d0 0x7f829c0787e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f82b8143840 tx=0x7f82a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:00.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:00.834+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3795985468 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f82b4061820 con 0x7f82b80ce6c0 2024-06-29T14:22:01.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.024+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82b81f76b0 con 0x7f82b80ce6c0 2024-06-29T14:22:01.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.025+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3795985468 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7f82b4061120 con 0x7f82b80ce6c0 2024-06-29T14:22:01.026 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:01.026 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:01.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f829c0762d0 msgr2=0x7f829c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f829c0762d0 0x7f829c0787e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f82b8143840 tx=0x7f82a8009040 comp rx=0 tx=0).stop 2024-06-29T14:22:01.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b80ce6c0 msgr2=0x7f82b81582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b80ce6c0 0x7f82b81582c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f82b400e910 tx=0x7f82b400edd0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 shutdown_connections 2024-06-29T14:22:01.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f829c0762d0 0x7f829c0787e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b81f6840 0x7f82b81377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b8144760 0x7f82b80b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 --2- 172.21.15.59:0/3795985468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b80ce6c0 0x7f82b81582c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 >> 172.21.15.59:0/3795985468 conn(0x7f82b806cef0 msgr2=0x7f82b813b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:01.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 shutdown_connections 2024-06-29T14:22:01.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.027+0000 7f82bfc47640 1 -- 172.21.15.59:0/3795985468 wait complete. 2024-06-29T14:22:01.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:01.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.155+0000 7f5f6a812640 1 -- 172.21.15.59:0/1208438494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 msgr2=0x7f5f640ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.155+0000 7f5f6a812640 1 --2- 172.21.15.59:0/1208438494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 0x7f5f640ce550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009860 tx=0x7f5f5402a480 comp rx=0 tx=0).stop 2024-06-29T14:22:01.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.155+0000 7f5f6a812640 1 -- 172.21.15.59:0/1208438494 shutdown_connections 2024-06-29T14:22:01.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.155+0000 7f5f6a812640 1 --2- 172.21.15.59:0/1208438494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 0x7f5f640ce550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.155+0000 7f5f6a812640 1 -- 172.21.15.59:0/1208438494 >> 172.21.15.59:0/1208438494 conn(0x7f5f640d45b0 msgr2=0x7f5f640d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:01.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.156+0000 7f5f6a812640 1 -- 172.21.15.59:0/1208438494 shutdown_connections 2024-06-29T14:22:01.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.156+0000 7f5f6a812640 1 -- 172.21.15.59:0/1208438494 wait complete. 2024-06-29T14:22:01.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.157+0000 7f5f6a812640 1 Processor -- start 2024-06-29T14:22:01.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.158+0000 7f5f6a812640 1 -- start start 2024-06-29T14:22:01.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f6a812640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f6415e0a0 0x7f5f641537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f6a812640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 0x7f5f64151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f6a812640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640cc2d0 0x7f5f641f7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f6a812640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5f640b4570 con 0x7f5f6415e0a0 2024-06-29T14:22:01.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f6a812640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f640b43f0 con 0x7f5f640cc2d0 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f6a812640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5f640b46f0 con 0x7f5f64162770 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f6415e0a0 0x7f5f641537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f6415e0a0 0x7f5f641537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52044/0 (socket says 172.21.15.59:52044) 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 -- 172.21.15.59:0/3939126704 learned_addr learned my addr 172.21.15.59:0/3939126704 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f637fe640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 0x7f5f64151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f68daf640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640cc2d0 0x7f5f641f7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 -- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 msgr2=0x7f5f64151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 0x7f5f64151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 -- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640cc2d0 msgr2=0x7f5f641f7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640cc2d0 0x7f5f641f7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 -- 172.21.15.59:0/3939126704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f64153ce0 con 0x7f5f6415e0a0 2024-06-29T14:22:01.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f637fe640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 0x7f5f64151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:01.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f68daf640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640cc2d0 0x7f5f641f7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:01.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.159+0000 7f5f63fff640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f6415e0a0 0x7f5f641537a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f5402aa70 tx=0x7f5f54003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:01.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.160+0000 7f5f60ff9640 1 -- 172.21.15.59:0/3939126704 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f54040020 con 0x7f5f6415e0a0 2024-06-29T14:22:01.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.160+0000 7f5f60ff9640 1 -- 172.21.15.59:0/3939126704 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f54038040 con 0x7f5f6415e0a0 2024-06-29T14:22:01.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.160+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f640cb230 con 0x7f5f6415e0a0 2024-06-29T14:22:01.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.160+0000 7f5f60ff9640 1 -- 172.21.15.59:0/3939126704 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f54037070 con 0x7f5f6415e0a0 2024-06-29T14:22:01.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.160+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f640cb7f0 con 0x7f5f6415e0a0 2024-06-29T14:22:01.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.161+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f64069ea0 con 0x7f5f6415e0a0 2024-06-29T14:22:01.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.161+0000 7f5f60ff9640 1 -- 172.21.15.59:0/3939126704 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5f54043020 con 0x7f5f6415e0a0 2024-06-29T14:22:01.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.162+0000 7f5f60ff9640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5f500762d0 0x7f5f500787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.162+0000 7f5f60ff9640 1 -- 172.21.15.59:0/3939126704 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f5f540b7ee0 con 0x7f5f6415e0a0 2024-06-29T14:22:01.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.162+0000 7f5f637fe640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5f500762d0 0x7f5f500787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.163+0000 7f5f637fe640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5f500762d0 0x7f5f500787e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5f640cdf80 tx=0x7f5f58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:01.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.166+0000 7f5f60ff9640 1 -- 172.21.15.59:0/3939126704 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5f64069ea0 con 0x7f5f6415e0a0 2024-06-29T14:22:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.359+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f640ce550 con 0x7f5f6415e0a0 2024-06-29T14:22:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.361+0000 7f5f60ff9640 1 -- 172.21.15.59:0/3939126704 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7f5f5404a030 con 0x7f5f6415e0a0 2024-06-29T14:22:01.359 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:01.359 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5f500762d0 msgr2=0x7f5f500787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5f500762d0 0x7f5f500787e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5f640cdf80 tx=0x7f5f58009040 comp rx=0 tx=0).stop 2024-06-29T14:22:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f6415e0a0 msgr2=0x7f5f641537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f6415e0a0 0x7f5f641537a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f5402aa70 tx=0x7f5f54003b90 comp rx=0 tx=0).stop 2024-06-29T14:22:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 shutdown_connections 2024-06-29T14:22:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5f500762d0 0x7f5f500787e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640cc2d0 0x7f5f641f7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f64162770 0x7f5f64151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 --2- 172.21.15.59:0/3939126704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f6415e0a0 0x7f5f641537a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 >> 172.21.15.59:0/3939126704 conn(0x7f5f640d45b0 msgr2=0x7f5f64152f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 shutdown_connections 2024-06-29T14:22:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.363+0000 7f5f6a812640 1 -- 172.21.15.59:0/3939126704 wait complete. 2024-06-29T14:22:01.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.510+0000 7fc046ac5640 1 -- 172.21.15.59:0/4272878199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc02c004d40 msgr2=0x7fc02c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.510+0000 7fc046ac5640 1 --2- 172.21.15.59:0/4272878199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc02c004d40 0x7fc02c005150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc038009b50 tx=0x7fc02c005830 comp rx=0 tx=0).stop 2024-06-29T14:22:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.510+0000 7fc045062640 1 -- 172.21.15.59:0/4272878199 reap_dead start 2024-06-29T14:22:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.510+0000 7fc046ac5640 1 -- 172.21.15.59:0/4272878199 shutdown_connections 2024-06-29T14:22:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.510+0000 7fc046ac5640 1 -- 172.21.15.59:0/4272878199 >> 172.21.15.59:0/4272878199 conn(0x7fc0400d3b20 msgr2=0x7fc0400d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.510+0000 7fc046ac5640 1 -- 172.21.15.59:0/4272878199 shutdown_connections 2024-06-29T14:22:01.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.510+0000 7fc046ac5640 1 -- 172.21.15.59:0/4272878199 wait complete. 2024-06-29T14:22:01.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.512+0000 7fc046ac5640 1 Processor -- start 2024-06-29T14:22:01.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.512+0000 7fc046ac5640 1 -- start start 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc046ac5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c004d40 0x7fc0401622b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc046ac5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0401f7ec0 0x7fc040147710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc046ac5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc04015f4a0 0x7fc04015f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc046ac5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc0400b4010 con 0x7fc0401f7ec0 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc046ac5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0400b3e90 con 0x7fc02c004d40 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc046ac5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc0400b4190 con 0x7fc04015f4a0 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc03ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0401f7ec0 0x7fc040147710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc03ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0401f7ec0 0x7fc040147710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52074/0 (socket says 172.21.15.59:52074) 2024-06-29T14:22:01.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc03ffff640 1 -- 172.21.15.59:0/1608134739 learned_addr learned my addr 172.21.15.59:0/1608134739 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:01.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc045062640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc04015f4a0 0x7fc04015f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc044861640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c004d40 0x7fc0401622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc03ffff640 1 -- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc04015f4a0 msgr2=0x7fc04015f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc03ffff640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc04015f4a0 0x7fc04015f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc03ffff640 1 -- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c004d40 msgr2=0x7fc0401622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.513+0000 7fc03ffff640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c004d40 0x7fc0401622b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.514+0000 7fc03ffff640 1 -- 172.21.15.59:0/1608134739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04015eff0 con 0x7fc0401f7ec0 2024-06-29T14:22:01.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.514+0000 7fc03ffff640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0401f7ec0 0x7fc040147710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc030011d90 tx=0x7fc03000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:01.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.514+0000 7fc03d7fa640 1 -- 172.21.15.59:0/1608134739 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030003d80 con 0x7fc0401f7ec0 2024-06-29T14:22:01.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.514+0000 7fc03d7fa640 1 -- 172.21.15.59:0/1608134739 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc030004050 con 0x7fc0401f7ec0 2024-06-29T14:22:01.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.514+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc040159640 con 0x7fc0401f7ec0 2024-06-29T14:22:01.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.514+0000 7fc03d7fa640 1 -- 172.21.15.59:0/1608134739 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030004420 con 0x7fc0401f7ec0 2024-06-29T14:22:01.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.514+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc040147d50 con 0x7fc0401f7ec0 2024-06-29T14:22:01.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.515+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc04015e040 con 0x7fc0401f7ec0 2024-06-29T14:22:01.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.516+0000 7fc03d7fa640 1 -- 172.21.15.59:0/1608134739 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc0300045c0 con 0x7fc0401f7ec0 2024-06-29T14:22:01.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.516+0000 7fc03d7fa640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0200762d0 0x7fc0200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.516+0000 7fc03d7fa640 1 -- 172.21.15.59:0/1608134739 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fc030099150 con 0x7fc0401f7ec0 2024-06-29T14:22:01.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.516+0000 7fc044861640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0200762d0 0x7fc0200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.517+0000 7fc044861640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0200762d0 0x7fc0200787e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc028006fd0 tx=0x7fc028006cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:01.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.520+0000 7fc03d7fa640 1 -- 172.21.15.59:0/1608134739 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0300626e0 con 0x7fc0401f7ec0 2024-06-29T14:22:01.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.750+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0400b9190 con 0x7fc0401f7ec0 2024-06-29T14:22:01.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.751+0000 7fc03d7fa640 1 -- 172.21.15.59:0/1608134739 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5222 (secure 0 0 0) 0x7fc030061fe0 con 0x7fc0401f7ec0 2024-06-29T14:22:01.749 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:01.749 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":7,"btime":"2024-06-29T14:22:00:301000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:21:59.698797+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:01.749 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 7 2024-06-29T14:22:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0200762d0 msgr2=0x7fc0200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0200762d0 0x7fc0200787e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc028006fd0 tx=0x7fc028006cc0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0401f7ec0 msgr2=0x7fc040147710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0401f7ec0 0x7fc040147710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc030011d90 tx=0x7fc03000f790 comp rx=0 tx=0).stop 2024-06-29T14:22:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 shutdown_connections 2024-06-29T14:22:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0200762d0 0x7fc0200787e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc04015f4a0 0x7fc04015f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0401f7ec0 0x7fc040147710 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 --2- 172.21.15.59:0/1608134739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc02c004d40 0x7fc0401622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 >> 172.21.15.59:0/1608134739 conn(0x7fc0400d3b20 msgr2=0x7fc04015a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 shutdown_connections 2024-06-29T14:22:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.753+0000 7fc046ac5640 1 -- 172.21.15.59:0/1608134739 wait complete. 2024-06-29T14:22:01.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.881+0000 7fe914288640 1 -- 172.21.15.59:0/849974372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe90c162190 msgr2=0x7fe90c0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.881+0000 7fe914288640 1 --2- 172.21.15.59:0/849974372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe90c162190 0x7fe90c0b38e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe904005fd0 tx=0x7fe90402a540 comp rx=0 tx=0).stop 2024-06-29T14:22:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.881+0000 7fe914288640 1 -- 172.21.15.59:0/849974372 shutdown_connections 2024-06-29T14:22:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.881+0000 7fe914288640 1 --2- 172.21.15.59:0/849974372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe90c162190 0x7fe90c0b38e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.881+0000 7fe914288640 1 --2- 172.21.15.59:0/849974372 >> v2:172.21.15.153:3300/0 conn(0x7fe90c162770 0x7fe90c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.882+0000 7fe914288640 1 -- 172.21.15.59:0/849974372 >> 172.21.15.59:0/849974372 conn(0x7fe90c0d45b0 msgr2=0x7fe90c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.882+0000 7fe914288640 1 -- 172.21.15.59:0/849974372 shutdown_connections 2024-06-29T14:22:01.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.882+0000 7fe914288640 1 -- 172.21.15.59:0/849974372 wait complete. 2024-06-29T14:22:01.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.883+0000 7fe914288640 1 Processor -- start 2024-06-29T14:22:01.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.883+0000 7fe914288640 1 -- start start 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe914288640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe90c15e0a0 0x7fe90c152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe914288640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c162770 0x7fe90c0cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe914288640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe90c0cad50 0x7fe90c0ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe914288640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c0b4260 con 0x7fe90c162770 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe914288640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c0b40e0 con 0x7fe90c0cad50 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe914288640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe90c0b43e0 con 0x7fe90c15e0a0 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c162770 0x7fe90c0cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c162770 0x7fe90c0cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52078/0 (socket says 172.21.15.59:52078) 2024-06-29T14:22:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe912024640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe90c15e0a0 0x7fe90c152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 -- 172.21.15.59:0/1361733379 learned_addr learned my addr 172.21.15.59:0/1361733379 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe912825640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe90c0cad50 0x7fe90c0ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 -- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe90c15e0a0 msgr2=0x7fe90c152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe90c15e0a0 0x7fe90c152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 -- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe90c0cad50 msgr2=0x7fe90c0ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe90c0cad50 0x7fe90c0ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 -- 172.21.15.59:0/1361733379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c0cc320 con 0x7fe90c162770 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.885+0000 7fe911823640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c162770 0x7fe90c0cc700 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe90000c3a0 tx=0x7fe90000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.886+0000 7fe90affd640 1 -- 172.21.15.59:0/1361733379 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe900017070 con 0x7fe90c162770 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.886+0000 7fe90affd640 1 -- 172.21.15.59:0/1361733379 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9000053c0 con 0x7fe90c162770 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.886+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90c0cb280 con 0x7fe90c162770 2024-06-29T14:22:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.886+0000 7fe90affd640 1 -- 172.21.15.59:0/1361733379 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe900003850 con 0x7fe90c162770 2024-06-29T14:22:01.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.886+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90c0cb840 con 0x7fe90c162770 2024-06-29T14:22:01.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.887+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90c0b4970 con 0x7fe90c162770 2024-06-29T14:22:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.887+0000 7fe90affd640 1 -- 172.21.15.59:0/1361733379 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe90001a070 con 0x7fe90c162770 2024-06-29T14:22:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.888+0000 7fe90affd640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8f80762d0 0x7fe8f80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.888+0000 7fe90affd640 1 -- 172.21.15.59:0/1361733379 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe900096f10 con 0x7fe90c162770 2024-06-29T14:22:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.888+0000 7fe912024640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8f80762d0 0x7fe8f80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.889+0000 7fe912024640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8f80762d0 0x7fe8f80787e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc005fd0 tx=0x7fe8fc009500 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:01.893+0000 7fe90affd640 1 -- 172.21.15.59:0/1361733379 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe90005fe50 con 0x7fe90c162770 2024-06-29T14:22:02.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.087+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe90c0b3c00 con 0x7fe90c162770 2024-06-29T14:22:02.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.088+0000 7fe90affd640 1 -- 172.21.15.59:0/1361733379 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7fe900064150 con 0x7fe90c162770 2024-06-29T14:22:02.087 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:02.087 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.090+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8f80762d0 msgr2=0x7fe8f80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.090+0000 7fe914288640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8f80762d0 0x7fe8f80787e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc005fd0 tx=0x7fe8fc009500 comp rx=0 tx=0).stop 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.090+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c162770 msgr2=0x7fe90c0cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.090+0000 7fe914288640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c162770 0x7fe90c0cc700 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe90000c3a0 tx=0x7fe90000c860 comp rx=0 tx=0).stop 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.091+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 shutdown_connections 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.091+0000 7fe914288640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8f80762d0 0x7fe8f80787e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.091+0000 7fe914288640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe90c0cad50 0x7fe90c0ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.091+0000 7fe914288640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c162770 0x7fe90c0cc700 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.091+0000 7fe914288640 1 --2- 172.21.15.59:0/1361733379 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe90c15e0a0 0x7fe90c152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.091+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 >> 172.21.15.59:0/1361733379 conn(0x7fe90c0d45b0 msgr2=0x7fe90c15c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.091+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 shutdown_connections 2024-06-29T14:22:02.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.091+0000 7fe914288640 1 -- 172.21.15.59:0/1361733379 wait complete. 2024-06-29T14:22:02.503 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:21:54.635958+0000', 'modified': '2024-06-29T14:21:59.698797+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4375': {'gid': 4375, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6836/2240732059', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 2240732059}, {'type': 'v1', 'addr': '172.21.15.153:6836', 'nonce': 2240732059}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4375, 'qdb_cluster': [4375]} 2024-06-29T14:22:02.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:22:02.504 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T14:22:02.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.627+0000 7f591e3fc640 1 -- 172.21.15.59:0/824649495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180b2270 msgr2=0x7f591815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.627+0000 7f591e3fc640 1 --2- 172.21.15.59:0/824649495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180b2270 0x7f591815bc20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5918074c50 tx=0x7f590802a530 comp rx=0 tx=0).stop 2024-06-29T14:22:02.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.627+0000 7f591e3fc640 1 -- 172.21.15.59:0/824649495 shutdown_connections 2024-06-29T14:22:02.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.627+0000 7f591e3fc640 1 --2- 172.21.15.59:0/824649495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59180b2fa0 0x7f59180d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.627+0000 7f591e3fc640 1 --2- 172.21.15.59:0/824649495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180b2270 0x7f591815bc20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.627+0000 7f591e3fc640 1 -- 172.21.15.59:0/824649495 >> 172.21.15.59:0/824649495 conn(0x7f5918073e10 msgr2=0x7f5918074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:02.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.627+0000 7f591e3fc640 1 -- 172.21.15.59:0/824649495 shutdown_connections 2024-06-29T14:22:02.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.627+0000 7f591e3fc640 1 -- 172.21.15.59:0/824649495 wait complete. 2024-06-29T14:22:02.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.629+0000 7f591e3fc640 1 Processor -- start 2024-06-29T14:22:02.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.629+0000 7f591e3fc640 1 -- start start 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591e3fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59180b2270 0x7f59181f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591e3fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59180b2fa0 0x7f59180b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591e3fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180d1a60 0x7f59181f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591e3fc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f59180b6cc0 con 0x7f59180d1a60 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591e3fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59180b6b40 con 0x7f59180b2270 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591e3fc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f59180cfe90 con 0x7f59180b2fa0 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180d1a60 0x7f59181f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180d1a60 0x7f59181f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52090/0 (socket says 172.21.15.59:52090) 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 -- 172.21.15.59:0/3841787983 learned_addr learned my addr 172.21.15.59:0/3841787983 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f59177fe640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59180b2fa0 0x7f59180b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 -- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59180b2fa0 msgr2=0x7f59180b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59180b2fa0 0x7f59180b3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 -- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59180b2270 msgr2=0x7f59181f5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f5917fff640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59180b2270 0x7f59181f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59180b2270 0x7f59181f5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 -- 172.21.15.59:0/3841787983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59180af4c0 con 0x7f59180d1a60 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591c999640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180d1a60 0x7f59181f1a60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f591000b930 tx=0x7f591000bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f5914ff9640 1 -- 172.21.15.59:0/3841787983 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5910019070 con 0x7f59180d1a60 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f5914ff9640 1 -- 172.21.15.59:0/3841787983 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f591000cc50 con 0x7f59180d1a60 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f5914ff9640 1 -- 172.21.15.59:0/3841787983 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59100146a0 con 0x7f59180d1a60 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591e3fc640 1 -- 172.21.15.59:0/3841787983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59181f64d0 con 0x7f59180d1a60 2024-06-29T14:22:02.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.631+0000 7f591e3fc640 1 -- 172.21.15.59:0/3841787983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59181f6a90 con 0x7f59180d1a60 2024-06-29T14:22:02.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.633+0000 7f5914ff9640 1 -- 172.21.15.59:0/3841787983 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f591001c070 con 0x7f59180d1a60 2024-06-29T14:22:02.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.634+0000 7f5914ff9640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5904076300 0x7f5904078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.634+0000 7f5914ff9640 1 -- 172.21.15.59:0/3841787983 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f59100977c0 con 0x7f59180d1a60 2024-06-29T14:22:02.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.634+0000 7f5917fff640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5904076300 0x7f5904078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.634+0000 7f5917fff640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5904076300 0x7f5904078810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5908033000 tx=0x7f5908004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:02.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.635+0000 7f5925399740 1 Processor -- start 2024-06-29T14:22:02.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.635+0000 7f5925399740 1 -- start start 2024-06-29T14:22:02.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5925399740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6c19270 0x5561a6fbd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5925399740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a6fbb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5925399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a6fe5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5925399740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5561a6f81bc0 con 0x5561a6c19270 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5925399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5561a6f30630 con 0x5561a6bf8590 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5925399740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5561a6f1fbd0 con 0x5561a6bf03b0 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5917fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6c19270 0x5561a6fbd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5917fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6c19270 0x5561a6fbd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52102/0 (socket says 172.21.15.59:52102) 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5917fff640 1 -- 172.21.15.59:0/2274718996 learned_addr learned my addr 172.21.15.59:0/2274718996 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f59177fe640 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a6fbb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f591c999640 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a6fe5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5917fff640 1 -- 172.21.15.59:0/2274718996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 msgr2=0x5561a6fbb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5917fff640 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a6fbb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5917fff640 1 -- 172.21.15.59:0/2274718996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 msgr2=0x5561a6fe5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5917fff640 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a6fe5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.636+0000 7f5917fff640 1 -- 172.21.15.59:0/2274718996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561a6f1cca0 con 0x5561a6c19270 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.637+0000 7f5917fff640 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6c19270 0x5561a6fbd8e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f590802ab20 tx=0x7f590803b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f842493baebb75b server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.637+0000 7f58fdffb640 1 -- 172.21.15.59:0/2274718996 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908035050 con 0x5561a6c19270 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.637+0000 7f58fdffb640 1 -- 172.21.15.59:0/2274718996 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f590804c070 con 0x5561a6c19270 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.637+0000 7f58fdffb640 1 -- 172.21.15.59:0/2274718996 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908003d80 con 0x5561a6c19270 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.637+0000 7f5925399740 1 -- 172.21.15.59:0/2274718996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6c19270 msgr2=0x5561a6fbd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.637+0000 7f5925399740 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6c19270 0x5561a6fbd8e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f590802ab20 tx=0x7f590803b000 comp rx=0 tx=0).stop 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.638+0000 7f5925399740 1 -- 172.21.15.59:0/2274718996 shutdown_connections 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.638+0000 7f5925399740 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a6fe5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.638+0000 7f5925399740 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a6fbb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.638+0000 7f5925399740 1 --2- 172.21.15.59:0/2274718996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6c19270 0x5561a6fbd8e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.638+0000 7f5925399740 1 -- 172.21.15.59:0/2274718996 >> 172.21.15.59:0/2274718996 conn(0x5561a6c0d3c0 msgr2=0x5561a6fbcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.638+0000 7f5925399740 1 -- 172.21.15.59:0/2274718996 shutdown_connections 2024-06-29T14:22:02.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.638+0000 7f5925399740 1 -- 172.21.15.59:0/2274718996 wait complete. 2024-06-29T14:22:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.639+0000 7f5925399740 1 -- start start 2024-06-29T14:22:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.639+0000 7f5925399740 1 Processor -- start 2024-06-29T14:22:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f5925399740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7003c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f5925399740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f5925399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a701ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f5925399740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5561a6ebdae0 con 0x5561a6bf8590 2024-06-29T14:22:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f5925399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5561a6f23c10 con 0x5561a6c19270 2024-06-29T14:22:02.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f5925399740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5561a6c93620 con 0x5561a6bf03b0 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f59177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f5925399740 10 client.0 _ll_register_callbacks cb 0x5561a6eca480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f59177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52116/0 (socket says 172.21.15.59:52116) 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f59177fe640 1 -- 172.21.15.59:0/851251023 learned_addr learned my addr 172.21.15.59:0/851251023 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f5917fff640 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7003c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.640+0000 7f591c999640 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a701ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f59177fe640 1 -- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 msgr2=0x5561a7003c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f59177fe640 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7003c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f59177fe640 1 -- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 msgr2=0x5561a701ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:02.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f59177fe640 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a701ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f59177fe640 1 -- 172.21.15.59:0/851251023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561a6c84850 con 0x5561a6bf8590 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f59177fe640 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff20e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f590c0106e0 tx=0x7f590c010ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9049c739767c2fb server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f58f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f58f6ffd640 1 -- 172.21.15.59:0/851251023 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c01d060 con 0x5561a6bf8590 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f58f6ffd640 1 -- 172.21.15.59:0/851251023 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f590c00ed40 con 0x5561a6bf8590 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 1 -- 172.21.15.59:0/851251023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f590804c070 con 0x5561a6bf8590 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f58f6ffd640 1 -- 172.21.15.59:0/851251023 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f590c023050 con 0x5561a6bf8590 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 10 client.4491 fetch_fsmap learned FSMap version 7 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 1 -- 172.21.15.59:0/851251023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x5561a6cfb6b0 con 0x5561a6bf8590 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f58f6ffd640 1 -- 172.21.15.59:0/851251023 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f590804c070 con 0x5561a6bf8590 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f58f6ffd640 1 -- 172.21.15.59:0/851251023 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 7) ==== 3510+0+0 (secure 0 0 0) 0x7f590c026020 con 0x5561a6bf8590 2024-06-29T14:22:02.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 10 client.4491 fetch_fsmap finished waiting for FSMap version 7 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 10 client.4491 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 20 client.4491 populate_metadata read hostname 'smithi059' 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x5561a6bd7e40 0x5561a6fe7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 4 client.4491 mds_command: new command op to 4375 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.641+0000 7f5925399740 1 -- 172.21.15.59:0/851251023 --> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561a6f32580 con 0x5561a6bd7e40 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.642+0000 7f5917fff640 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x5561a6bd7e40 0x5561a6fe7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.642+0000 7f58f6ffd640 10 client.4491 ms_handle_connect on v2:172.21.15.153:6834/2240732059 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.642+0000 7f5917fff640 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x5561a6bd7e40 0x5561a6fe7f00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed256a34b237980 server_cookie=a70687191f0790be in_seq=0 out_seq=0 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f58f6ffd640 1 -- 172.21.15.59:0/851251023 <== mds.0 v2:172.21.15.153:6834/2240732059 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5561a6f32580 con 0x5561a6bd7e40 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f58f6ffd640 10 client.4491 handle_command_reply: tid=0 2024-06-29T14:22:02.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 client.4491 shutdown 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 -- 172.21.15.59:0/851251023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 msgr2=0x5561a6ff20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff20e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f590c0106e0 tx=0x7f590c010ba0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 -- 172.21.15.59:0/851251023 shutdown_connections 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x5561a6bd7e40 0x5561a6fe7f00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a701ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff20e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f58f6ffd640 0 client.4491 ms_handle_reset on v2:172.21.15.153:6834/2240732059 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 --2- 172.21.15.59:0/851251023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7003c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 -- 172.21.15.59:0/851251023 >> 172.21.15.59:0/851251023 conn(0x5561a6c0d3c0 msgr2=0x5561a6fb8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.643+0000 7f5925399740 1 -- 172.21.15.59:0/851251023 shutdown_connections 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.644+0000 7f5925399740 1 -- 172.21.15.59:0/851251023 wait complete. 2024-06-29T14:22:02.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.644+0000 7f5925399740 20 client.4491 trim_cache size 0 max 16384 2024-06-29T14:22:02.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.663+0000 7f5925399740 1 Processor -- start 2024-06-29T14:22:02.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.663+0000 7f5925399740 1 -- start start 2024-06-29T14:22:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f5925399740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7004600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f5925399740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f5925399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a6fbb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f5925399740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f59181fd670 con 0x5561a6bf8590 2024-06-29T14:22:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f5925399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f591806f950 con 0x5561a6c19270 2024-06-29T14:22:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f5925399740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f59181fd020 con 0x5561a6bf03b0 2024-06-29T14:22:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f59177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f59177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52132/0 (socket says 172.21.15.59:52132) 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f59177fe640 1 -- 172.21.15.59:0/1723266402 learned_addr learned my addr 172.21.15.59:0/1723266402 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f5917fff640 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7004600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f591c999640 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a6fbb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f59177fe640 1 -- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 msgr2=0x5561a7004600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f59177fe640 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7004600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f59177fe640 1 -- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 msgr2=0x5561a6fbb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f59177fe640 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a6fbb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f59177fe640 1 -- 172.21.15.59:0/1723266402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561a6fbc4f0 con 0x5561a6bf8590 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f5917fff640 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7004600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.664+0000 7f591c999640 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a6fbb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f59177fe640 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff5030 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f590c000c00 tx=0x7f590c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1307c7ea7bc0eb8 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f58fd7fa640 1 -- 172.21.15.59:0/1723266402 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c02b070 con 0x5561a6bf8590 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f58fd7fa640 1 -- 172.21.15.59:0/1723266402 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f590c01e890 con 0x5561a6bf8590 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f58fd7fa640 1 -- 172.21.15.59:0/1723266402 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c00eda0 con 0x5561a6bf8590 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 -- 172.21.15.59:0/1723266402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 msgr2=0x5561a6ff5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff5030 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f590c000c00 tx=0x7f590c0057b0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 -- 172.21.15.59:0/1723266402 shutdown_connections 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6c19270 0x5561a6fbb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf8590 0x5561a6ff5030 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 --2- 172.21.15.59:0/1723266402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6bf03b0 0x5561a7004600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 -- 172.21.15.59:0/1723266402 >> 172.21.15.59:0/1723266402 conn(0x5561a6c0d3c0 msgr2=0x5561a6fb5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 -- 172.21.15.59:0/1723266402 shutdown_connections 2024-06-29T14:22:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.665+0000 7f5925399740 1 -- 172.21.15.59:0/1723266402 wait complete. 2024-06-29T14:22:02.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.666+0000 7f5925399740 1 -- start start 2024-06-29T14:22:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.667+0000 7f5925399740 1 Processor -- start 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5925399740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf03b0 0x5561a6ff3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5925399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a707bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5925399740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6c19270 0x5561a7088fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5925399740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5561a6fe9020 con 0x5561a6bf03b0 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5925399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5561a6ff3540 con 0x5561a6bf8590 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5925399740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5561a6fde8d0 con 0x5561a6c19270 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5925399740 10 client.0 _ll_register_callbacks cb 0x7f590c00b540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf03b0 0x5561a6ff3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf03b0 0x5561a6ff3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52142/0 (socket says 172.21.15.59:52142) 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 -- 172.21.15.59:0/762426351 learned_addr learned my addr 172.21.15.59:0/762426351 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f591c999640 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6c19270 0x5561a7088fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f59177fe640 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a707bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 -- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6c19270 msgr2=0x5561a7088fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6c19270 0x5561a7088fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 -- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 msgr2=0x5561a707bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a707bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 -- 172.21.15.59:0/762426351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561a707bff0 con 0x5561a6bf03b0 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f59177fe640 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a707bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f5917fff640 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf03b0 0x5561a6ff3dc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f590802ab20 tx=0x7f59080035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b825ae62864998 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.668+0000 7f58f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f58f6ffd640 1 -- 172.21.15.59:0/762426351 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590805f060 con 0x5561a6bf03b0 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f58f6ffd640 1 -- 172.21.15.59:0/762426351 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f59080047a0 con 0x5561a6bf03b0 2024-06-29T14:22:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 1 -- 172.21.15.59:0/762426351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f590c01e890 con 0x5561a6bf03b0 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f58f6ffd640 1 -- 172.21.15.59:0/762426351 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f5908065050 con 0x5561a6bf03b0 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 10 client.4497 fetch_fsmap learned FSMap version 7 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 1 -- 172.21.15.59:0/762426351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x5561a7088830 con 0x5561a6bf03b0 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f58f6ffd640 1 -- 172.21.15.59:0/762426351 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f590c01e890 con 0x5561a6bf03b0 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f58f6ffd640 1 -- 172.21.15.59:0/762426351 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 7) ==== 3510+0+0 (secure 0 0 0) 0x7f5908068020 con 0x5561a6bf03b0 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 10 client.4497 fetch_fsmap finished waiting for FSMap version 7 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 10 client.4497 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 20 client.4497 populate_metadata read hostname 'smithi059' 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x5561a6fbb3c0 0x5561a7019660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 4 client.4497 mds_command: new command op to 4375 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f5925399740 1 -- 172.21.15.59:0/762426351 --> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] -- command(tid 0: {"prefix": "status"}) -- 0x5561a6f2e020 con 0x5561a6fbb3c0 2024-06-29T14:22:02.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.669+0000 7f59177fe640 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x5561a6fbb3c0 0x5561a7019660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f59177fe640 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x5561a6fbb3c0 0x5561a7019660 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39bbfb0c61ceb5a2 server_cookie=6b3f95c98a091f00 in_seq=0 out_seq=0 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f58f6ffd640 10 client.4497 ms_handle_connect on v2:172.21.15.153:6834/2240732059 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f58f6ffd640 1 -- 172.21.15.59:0/762426351 <== mds.0 v2:172.21.15.153:6834/2240732059 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5561a6f2e020 con 0x5561a6fbb3c0 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f58f6ffd640 10 client.4497 handle_command_reply: tid=0 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 client.4497 shutdown 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 -- 172.21.15.59:0/762426351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf03b0 msgr2=0x5561a6ff3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf03b0 0x5561a6ff3dc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f590802ab20 tx=0x7f59080035f0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 -- 172.21.15.59:0/762426351 shutdown_connections 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x5561a6fbb3c0 0x5561a7019660 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5561a6c19270 0x5561a7088fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5561a6bf8590 0x5561a707bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f58f6ffd640 0 client.4497 ms_handle_reset on v2:172.21.15.153:6834/2240732059 2024-06-29T14:22:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 --2- 172.21.15.59:0/762426351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5561a6bf03b0 0x5561a6ff3dc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 -- 172.21.15.59:0/762426351 >> 172.21.15.59:0/762426351 conn(0x5561a6c0d3c0 msgr2=0x5561a6fb5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 -- 172.21.15.59:0/762426351 shutdown_connections 2024-06-29T14:22:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 1 -- 172.21.15.59:0/762426351 wait complete. 2024-06-29T14:22:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.670+0000 7f5925399740 20 client.4497 trim_cache size 0 max 16384 2024-06-29T14:22:02.675 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:22:02.675 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:22:02.675 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 4375, 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 7.4085925269999997, 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 7, 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 26, 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 26, 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 14.443497761 2024-06-29T14:22:02.676 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:22:02.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 -- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5904076300 msgr2=0x7f5904078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5904076300 0x7f5904078810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5908033000 tx=0x7f5908004730 comp rx=0 tx=0).stop 2024-06-29T14:22:02.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 -- 172.21.15.59:0/3841787983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180d1a60 msgr2=0x7f59181f1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180d1a60 0x7f59181f1a60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f591000b930 tx=0x7f591000bdf0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 -- 172.21.15.59:0/3841787983 shutdown_connections 2024-06-29T14:22:02.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5904076300 0x7f5904078810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59180d1a60 0x7f59181f1a60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59180b2fa0 0x7f59180b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 --2- 172.21.15.59:0/3841787983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59180b2270 0x7f59181f5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 -- 172.21.15.59:0/3841787983 >> 172.21.15.59:0/3841787983 conn(0x7f5918073e10 msgr2=0x7f59180cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 -- 172.21.15.59:0/3841787983 shutdown_connections 2024-06-29T14:22:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.671+0000 7f591e3fc640 1 -- 172.21.15.59:0/3841787983 wait complete. 2024-06-29T14:22:02.680 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:22:02.680 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-06-29T14:22:02.680 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:22:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.800+0000 7f6655d24640 1 -- 172.21.15.59:0/3083766337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66500b40a0 msgr2=0x7f665015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.800+0000 7f6655d24640 1 --2- 172.21.15.59:0/3083766337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66500b40a0 0x7f665015e8d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6644005fd0 tx=0x7f664402a5c0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.800+0000 7f6655d24640 1 -- 172.21.15.59:0/3083766337 shutdown_connections 2024-06-29T14:22:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.800+0000 7f6655d24640 1 --2- 172.21.15.59:0/3083766337 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f665015ee10 0x7f665015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.800+0000 7f6655d24640 1 --2- 172.21.15.59:0/3083766337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66500b40a0 0x7f665015e8d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.800+0000 7f6655d24640 1 -- 172.21.15.59:0/3083766337 >> 172.21.15.59:0/3083766337 conn(0x7f66500d3ee0 msgr2=0x7f66500d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.800+0000 7f6655d24640 1 -- 172.21.15.59:0/3083766337 shutdown_connections 2024-06-29T14:22:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.800+0000 7f6655d24640 1 -- 172.21.15.59:0/3083766337 wait complete. 2024-06-29T14:22:02.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.802+0000 7f6655d24640 1 Processor -- start 2024-06-29T14:22:02.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.802+0000 7f6655d24640 1 -- start start 2024-06-29T14:22:02.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f6655d24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66500d23c0 0x7f66501410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f6655d24640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f665015ee10 0x7f6650152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f6655d24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6650142550 0x7f66501441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f6655d24640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6650160280 con 0x7f66500d23c0 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f6655d24640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6650160100 con 0x7f6650142550 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f6655d24640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6650160400 con 0x7f665015ee10 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f664f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66500d23c0 0x7f66501410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f664f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66500d23c0 0x7f66501410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40434/0 (socket says 172.21.15.59:40434) 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.803+0000 7f664f7fe640 1 -- 172.21.15.59:0/186893773 learned_addr learned my addr 172.21.15.59:0/186893773 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664ffff640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6650142550 0x7f66501441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664effd640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f665015ee10 0x7f6650152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664f7fe640 1 -- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f665015ee10 msgr2=0x7f6650152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664f7fe640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f665015ee10 0x7f6650152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664f7fe640 1 -- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6650142550 msgr2=0x7f66501441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664f7fe640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6650142550 0x7f66501441d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664f7fe640 1 -- 172.21.15.59:0/186893773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6650141790 con 0x7f66500d23c0 2024-06-29T14:22:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664ffff640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6650142550 0x7f66501441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f664f7fe640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66500d23c0 0x7f66501410f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f664000c370 tx=0x7f664000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f6637fff640 1 -- 172.21.15.59:0/186893773 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6640017070 con 0x7f66500d23c0 2024-06-29T14:22:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6650141a70 con 0x7f66500d23c0 2024-06-29T14:22:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f6637fff640 1 -- 172.21.15.59:0/186893773 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66400053c0 con 0x7f66500d23c0 2024-06-29T14:22:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f6637fff640 1 -- 172.21.15.59:0/186893773 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6640003850 con 0x7f66500d23c0 2024-06-29T14:22:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.804+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6650142030 con 0x7f66500d23c0 2024-06-29T14:22:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.805+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665015e8d0 con 0x7f66500d23c0 2024-06-29T14:22:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.806+0000 7f6637fff640 1 -- 172.21.15.59:0/186893773 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f664001a070 con 0x7f66500d23c0 2024-06-29T14:22:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.806+0000 7f6637fff640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f663c0762d0 0x7f663c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.806+0000 7f6637fff640 1 -- 172.21.15.59:0/186893773 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6640096e30 con 0x7f66500d23c0 2024-06-29T14:22:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.807+0000 7f664effd640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f663c0762d0 0x7f663c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.807+0000 7f664effd640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f663c0762d0 0x7f663c0787e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6644033000 tx=0x7f6644004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:02.811+0000 7f6637fff640 1 -- 172.21.15.59:0/186893773 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f664005fd80 con 0x7f66500d23c0 2024-06-29T14:22:03.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.043+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f66500d2950 con 0x7f66500d23c0 2024-06-29T14:22:03.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.044+0000 7f6637fff640 1 -- 172.21.15.59:0/186893773 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f66400683c0 con 0x7f66500d23c0 2024-06-29T14:22:03.042 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-29T14:22:03.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f663c0762d0 msgr2=0x7f663c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f663c0762d0 0x7f663c0787e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6644033000 tx=0x7f6644004730 comp rx=0 tx=0).stop 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66500d23c0 msgr2=0x7f66501410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66500d23c0 0x7f66501410f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f664000c370 tx=0x7f664000c830 comp rx=0 tx=0).stop 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 shutdown_connections 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f663c0762d0 0x7f663c0787e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6650142550 0x7f66501441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f665015ee10 0x7f6650152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 --2- 172.21.15.59:0/186893773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66500d23c0 0x7f66501410f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:03.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 >> 172.21.15.59:0/186893773 conn(0x7f66500d3ee0 msgr2=0x7f665006cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:03.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 shutdown_connections 2024-06-29T14:22:03.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03.046+0000 7f6655d24640 1 -- 172.21.15.59:0/186893773 wait complete. 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:22:03.055 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-06-29T14:22:03.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:03.056 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:22:03.072 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:03.072 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:03.072 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:03.072 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:03.072 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:03.072 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:03.072 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:03.072 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42728sec preferred_lft 42728sec 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:03.073 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:03.074 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:03.074 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:22:03.074 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-29T14:22:03.074 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:03.075 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:03.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add name ceph-brx type bridge 2024-06-29T14:22:03.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr flush dev ceph-brx 2024-06-29T14:22:03.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set ceph-brx up 2024-06-29T14:22:03.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-29T14:22:03.075 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:03.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:03.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:03.282 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:22:03.310 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:22:03.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:03.311 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:22:03.375 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:22:03.375 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:22:03.375 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:22:03.375 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:22:03.376 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:22:03.376 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:03.376 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:03.376 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:22:03.376 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:22:03.376 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:22:03.376 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:03.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:03.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:03.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:03.694 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:22:03.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:03.710 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list-id 2024-06-29T14:22:03.766 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:03.766 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:03.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-29T14:22:03.767 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-06-29T14:22:03.767 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:03.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:03.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:03.891 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-06-29T14:22:03.891 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:03.891 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:03.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-06-29T14:22:03.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-29T14:22:03.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-06-29T14:22:03.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-06-29T14:22:03.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-06-29T14:22:03.891 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:03.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:04.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:04.060 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:04.060 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:04.060 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:22:04.060 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set dev brx.0 master ceph-brx 2024-06-29T14:22:04.060 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:04.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:04.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:04.142 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-06-29T14:22:04.142 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:04.142 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-06-29T14:22:04.158 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-06-29T14:22:04.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:04.159 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-06-29T14:22:04.215 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-06-29T14:22:04.281 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-06-29T14:22:04.337 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-06-29T14:22:04.337 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-06-29T14:22:04.338 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-06-29T14:22:04.339 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-06-29T14:22:04.340 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-06-29T14:22:04.340 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-06-29T14:22:04.340 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-29T14:22:04.340 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-29T14:22:04.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:04.340 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-06-29T14:22:04.393 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-06-29T14:22:04.393 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-06-29T14:22:04.436 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-06-29T14:22:04.461 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-06-29T14:22:04.519 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-06-29T14:22:04.519 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-06-29T14:22:04.519 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-06-29T14:22:04.519 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-06-29T14:22:04.519 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-06-29T14:22:04.520 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-29T14:22:04.521 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-29T14:22:04.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:04.522 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-06-29T14:22:04.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi059.stderr:2024-06-29T14:22:04.554+0000 7fad14a46480 -1 init, newargv = 0x5616c988dcf0 newargc=15 2024-06-29T14:22:04.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi059.stderr:ceph-fuse[45045]: starting ceph client 2024-06-29T14:22:04.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi059.stderr:ceph-fuse[45045]: starting fuse 2024-06-29T14:22:04.568 INFO:teuthology.orchestra.run.smithi059.stdout:41 2024-06-29T14:22:04.569 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-06-29T14:22:04.569 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-06-29T14:22:04.570 DEBUG:teuthology.orchestra.run.smithi059:> 2024-06-29T14:22:04.571 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.admin")) 2024-06-29T14:22:04.571 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-06-29T14:22:04.690 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.admin.45045.asok 2024-06-29T14:22:04.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:04.701 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.45045.asok 2024-06-29T14:22:04.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:04.701 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.45045.asok status 2024-06-29T14:22:04.848 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:22:04.848 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-06-29T14:22:04.848 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "cf146c2f9dac0bd5aed66008255af2113db872f7", 2024-06-29T14:22:04.848 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-4577-gcf146c2f (cf146c2f9dac0bd5aed66008255af2113db872f7) squid (dev)", 2024-06-29T14:22:04.848 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "admin", 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "45045", 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 4509, 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 4509 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:22:04.849 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 129816505 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 129816505 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.4509 192.168.0.1:0/129816505", 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/129816505", 2024-06-29T14:22:04.850 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-06-29T14:22:04.851 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 7, 2024-06-29T14:22:04.851 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 26, 2024-06-29T14:22:04.851 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-06-29T14:22:04.851 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-06-29T14:22:04.851 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-06-29T14:22:04.851 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:22:04.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:04.857 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-06-29T14:22:04.857 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-06-29T14:22:04.873 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-06-29T14:22:04.873 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-06-29T14:22:04.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:04.874 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-06-29T14:22:04.940 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-06-29T14:22:05.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:05.054 INFO:teuthology.orchestra.run.smithi059.stdout:# file: . 2024-06-29T14:22:05.054 INFO:teuthology.orchestra.run.smithi059.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-06-29T14:22:05.054 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:05.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:05.060 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:05.060 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:05.077 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:05.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 -- 172.21.15.59:0/3254915598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 msgr2=0x7f4b08142be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 --2- 172.21.15.59:0/3254915598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 0x7f4b08142be0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b080b4ff0 tx=0x7f4b0002a560 comp rx=0 tx=0).stop 2024-06-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 -- 172.21.15.59:0/3254915598 shutdown_connections 2024-06-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 --2- 172.21.15.59:0/3254915598 >> v2:172.21.15.153:3300/0 conn(0x7f4b0806c270 0x7f4b0806c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 --2- 172.21.15.59:0/3254915598 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b080cccc0 0x7f4b080cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 --2- 172.21.15.59:0/3254915598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 0x7f4b08142be0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 -- 172.21.15.59:0/3254915598 >> 172.21.15.59:0/3254915598 conn(0x7f4b080b24c0 msgr2=0x7f4b080cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 -- 172.21.15.59:0/3254915598 shutdown_connections 2024-06-29T14:22:05.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.235+0000 7f4b10695640 1 -- 172.21.15.59:0/3254915598 wait complete. 2024-06-29T14:22:05.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.237+0000 7f4b10695640 1 Processor -- start 2024-06-29T14:22:05.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.237+0000 7f4b10695640 1 -- start start 2024-06-29T14:22:05.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.238+0000 7f4b10695640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b0806c270 0x7f4b0813c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.238+0000 7f4b10695640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b080cccc0 0x7f4b08157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.238+0000 7f4b10695640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 0x7f4b081f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.238+0000 7f4b10695640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4b080ce6e0 con 0x7f4b081425e0 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.238+0000 7f4b10695640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0806cdd0 con 0x7f4b080cccc0 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.238+0000 7f4b10695640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4b080ce860 con 0x7f4b0806c270 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.238+0000 7f4b0ec32640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 0x7f4b081f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0ec32640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 0x7f4b081f2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40448/0 (socket says 172.21.15.59:40448) 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0ec32640 1 -- 172.21.15.59:0/1788361473 learned_addr learned my addr 172.21.15.59:0/1788361473 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0dc30640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b080cccc0 0x7f4b08157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0e431640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b0806c270 0x7f4b0813c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0ec32640 1 -- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b0806c270 msgr2=0x7f4b0813c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0ec32640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b0806c270 0x7f4b0813c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0ec32640 1 -- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b080cccc0 msgr2=0x7f4b08157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0ec32640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b080cccc0 0x7f4b08157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0ec32640 1 -- 172.21.15.59:0/1788361473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0813f1a0 con 0x7f4b081425e0 2024-06-29T14:22:05.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b0ec32640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 0x7f4b081f2fb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e990 tx=0x7f4af800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:05.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b06ffd640 1 -- 172.21.15.59:0/1788361473 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8004d20 con 0x7f4b081425e0 2024-06-29T14:22:05.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b06ffd640 1 -- 172.21.15.59:0/1788361473 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4af800caa0 con 0x7f4b081425e0 2024-06-29T14:22:05.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0813f480 con 0x7f4b081425e0 2024-06-29T14:22:05.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b06ffd640 1 -- 172.21.15.59:0/1788361473 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8005020 con 0x7f4b081425e0 2024-06-29T14:22:05.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.239+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b0813fa40 con 0x7f4b081425e0 2024-06-29T14:22:05.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.240+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b080cd0d0 con 0x7f4b081425e0 2024-06-29T14:22:05.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.242+0000 7f4b06ffd640 1 -- 172.21.15.59:0/1788361473 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4af80051c0 con 0x7f4b081425e0 2024-06-29T14:22:05.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.242+0000 7f4b06ffd640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4af40762d0 0x7f4af40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.242+0000 7f4b06ffd640 1 -- 172.21.15.59:0/1788361473 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f4af8098740 con 0x7f4b081425e0 2024-06-29T14:22:05.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.243+0000 7f4b0e431640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4af40762d0 0x7f4af40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.243+0000 7f4b0e431640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4af40762d0 0x7f4af40787e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b00033000 tx=0x7f4b00004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:05.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.246+0000 7f4b06ffd640 1 -- 172.21.15.59:0/1788361473 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4af8016030 con 0x7f4b081425e0 2024-06-29T14:22:05.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.438+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b0815bc80 con 0x7f4b081425e0 2024-06-29T14:22:05.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.439+0000 7f4b06ffd640 1 -- 172.21.15.59:0/1788361473 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7f4af8061680 con 0x7f4b081425e0 2024-06-29T14:22:05.437 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:05.437 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:05.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4af40762d0 msgr2=0x7f4af40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4af40762d0 0x7f4af40787e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b00033000 tx=0x7f4b00004730 comp rx=0 tx=0).stop 2024-06-29T14:22:05.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 msgr2=0x7f4b081f2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 0x7f4b081f2fb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e990 tx=0x7f4af800ee50 comp rx=0 tx=0).stop 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 shutdown_connections 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4af40762d0 0x7f4af40787e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b081425e0 0x7f4b081f2fb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b080cccc0 0x7f4b08157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 --2- 172.21.15.59:0/1788361473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4b0806c270 0x7f4b0813c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 >> 172.21.15.59:0/1788361473 conn(0x7f4b080b24c0 msgr2=0x7f4b080b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 shutdown_connections 2024-06-29T14:22:05.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.441+0000 7f4b10695640 1 -- 172.21.15.59:0/1788361473 wait complete. 2024-06-29T14:22:05.450 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:05.451 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:05.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 -- 172.21.15.59:0/425463571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 msgr2=0x7f368013edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 --2- 172.21.15.59:0/425463571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 0x7f368013edc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3680069670 tx=0x7f366802a4d0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 -- 172.21.15.59:0/425463571 shutdown_connections 2024-06-29T14:22:05.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 --2- 172.21.15.59:0/425463571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36801414c0 0x7f36801418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 --2- 172.21.15.59:0/425463571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 0x7f368013edc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 --2- 172.21.15.59:0/425463571 >> v2:172.21.15.153:3300/0 conn(0x7f368015ceb0 0x7f36800b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 -- 172.21.15.59:0/425463571 >> 172.21.15.59:0/425463571 conn(0x7f36800b6fb0 msgr2=0x7f36800b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:05.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 -- 172.21.15.59:0/425463571 shutdown_connections 2024-06-29T14:22:05.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.574+0000 7f36869fc640 1 -- 172.21.15.59:0/425463571 wait complete. 2024-06-29T14:22:05.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.576+0000 7f36869fc640 1 Processor -- start 2024-06-29T14:22:05.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.576+0000 7f36869fc640 1 -- start start 2024-06-29T14:22:05.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f36869fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 0x7f3680139670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f36869fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36801414c0 0x7f368014a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f36869fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f368015ceb0 0x7f36801f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f36869fc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f36800b28c0 con 0x7f368013e9b0 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f36869fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36800b2740 con 0x7f368015ceb0 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f36869fc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f36800b2a40 con 0x7f36801414c0 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f367ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 0x7f3680139670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f367ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 0x7f3680139670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40472/0 (socket says 172.21.15.59:40472) 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f367ffff640 1 -- 172.21.15.59:0/1819469478 learned_addr learned my addr 172.21.15.59:0/1819469478 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f3677fff640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36801414c0 0x7f368014a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f3684f99640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f368015ceb0 0x7f36801f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.577+0000 7f367ffff640 1 -- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36801414c0 msgr2=0x7f368014a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f367ffff640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36801414c0 0x7f368014a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f367ffff640 1 -- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f368015ceb0 msgr2=0x7f36801f29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f367ffff640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f368015ceb0 0x7f36801f29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f367ffff640 1 -- 172.21.15.59:0/1819469478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3680133bc0 con 0x7f368013e9b0 2024-06-29T14:22:05.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f367ffff640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 0x7f3680139670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f367000d740 tx=0x7f367000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:05.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f367d7fa640 1 -- 172.21.15.59:0/1819469478 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670017070 con 0x7f368013e9b0 2024-06-29T14:22:05.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f367d7fa640 1 -- 172.21.15.59:0/1819469478 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3670009d20 con 0x7f368013e9b0 2024-06-29T14:22:05.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3680133e60 con 0x7f368013e9b0 2024-06-29T14:22:05.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f367d7fa640 1 -- 172.21.15.59:0/1819469478 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36700106c0 con 0x7f368013e9b0 2024-06-29T14:22:05.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.578+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3680134420 con 0x7f368013e9b0 2024-06-29T14:22:05.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.579+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36800b2f60 con 0x7f368013e9b0 2024-06-29T14:22:05.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.580+0000 7f367d7fa640 1 -- 172.21.15.59:0/1819469478 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f367001a070 con 0x7f368013e9b0 2024-06-29T14:22:05.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.580+0000 7f367d7fa640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f36600762d0 0x7f36600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.581+0000 7f367d7fa640 1 -- 172.21.15.59:0/1819469478 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f3670097b00 con 0x7f368013e9b0 2024-06-29T14:22:05.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.581+0000 7f3677fff640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f36600762d0 0x7f36600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.581+0000 7f3677fff640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f36600762d0 0x7f36600787e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3668033000 tx=0x7f3668004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:05.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.585+0000 7f367d7fa640 1 -- 172.21.15.59:0/1819469478 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3670060ad0 con 0x7f368013e9b0 2024-06-29T14:22:05.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.774+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f368006d7f0 con 0x7f368013e9b0 2024-06-29T14:22:05.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.775+0000 7f367d7fa640 1 -- 172.21.15.59:0/1819469478 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f3670064dd0 con 0x7f368013e9b0 2024-06-29T14:22:05.773 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:05.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f36600762d0 msgr2=0x7f36600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f36600762d0 0x7f36600787e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3668033000 tx=0x7f3668004730 comp rx=0 tx=0).stop 2024-06-29T14:22:05.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 msgr2=0x7f3680139670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 0x7f3680139670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f367000d740 tx=0x7f367000dc00 comp rx=0 tx=0).stop 2024-06-29T14:22:05.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 shutdown_connections 2024-06-29T14:22:05.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f36600762d0 0x7f36600787e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f368015ceb0 0x7f36801f29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36801414c0 0x7f368014a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 --2- 172.21.15.59:0/1819469478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f368013e9b0 0x7f3680139670 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 >> 172.21.15.59:0/1819469478 conn(0x7f36800b6fb0 msgr2=0x7f36800b3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:05.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 shutdown_connections 2024-06-29T14:22:05.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.777+0000 7f36869fc640 1 -- 172.21.15.59:0/1819469478 wait complete. 2024-06-29T14:22:05.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:05.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.909+0000 7fa60c222640 1 -- 172.21.15.59:0/1826249578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 msgr2=0x7fa5ec003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.909+0000 7fa60c222640 1 --2- 172.21.15.59:0/1826249578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 0x7fa5ec003e80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009b50 tx=0x7fa5ec005830 comp rx=0 tx=0).stop 2024-06-29T14:22:05.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.909+0000 7fa60a7bf640 1 -- 172.21.15.59:0/1826249578 reap_dead start 2024-06-29T14:22:05.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.909+0000 7fa60c222640 1 -- 172.21.15.59:0/1826249578 shutdown_connections 2024-06-29T14:22:05.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.909+0000 7fa60c222640 1 -- 172.21.15.59:0/1826249578 >> 172.21.15.59:0/1826249578 conn(0x7fa6040d45b0 msgr2=0x7fa6040d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:05.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.909+0000 7fa60c222640 1 -- 172.21.15.59:0/1826249578 shutdown_connections 2024-06-29T14:22:05.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.909+0000 7fa60c222640 1 -- 172.21.15.59:0/1826249578 wait complete. 2024-06-29T14:22:05.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.911+0000 7fa60c222640 1 Processor -- start 2024-06-29T14:22:05.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.911+0000 7fa60c222640 1 -- start start 2024-06-29T14:22:05.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa60c222640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 0x7fa6040c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa60c222640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6041e78c0 0x7fa604150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa60c222640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604159290 0x7fa6041596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa60c222640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa6040b4170 con 0x7fa5ec003a50 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa60c222640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6040b3ff0 con 0x7fa604159290 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa60c222640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa6040b42f0 con 0x7fa6041e78c0 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa609fbe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 0x7fa6040c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa609fbe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 0x7fa6040c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40502/0 (socket says 172.21.15.59:40502) 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa609fbe640 1 -- 172.21.15.59:0/1506018934 learned_addr learned my addr 172.21.15.59:0/1506018934 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa6097bd640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6041e78c0 0x7fa604150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa60a7bf640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604159290 0x7fa6041596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa609fbe640 1 -- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6041e78c0 msgr2=0x7fa604150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa609fbe640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6041e78c0 0x7fa604150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa609fbe640 1 -- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604159290 msgr2=0x7fa6041596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa609fbe640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604159290 0x7fa6041596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.912+0000 7fa609fbe640 1 -- 172.21.15.59:0/1506018934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa604159be0 con 0x7fa5ec003a50 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.913+0000 7fa609fbe640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 0x7fa6040c95b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009c70 tx=0x7fa5fc033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.913+0000 7fa6027fc640 1 -- 172.21.15.59:0/1506018934 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc038040 con 0x7fa5ec003a50 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.913+0000 7fa6027fc640 1 -- 172.21.15.59:0/1506018934 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5fc004020 con 0x7fa5ec003a50 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.913+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6040c9af0 con 0x7fa5ec003a50 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.913+0000 7fa6027fc640 1 -- 172.21.15.59:0/1506018934 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc004470 con 0x7fa5ec003a50 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.913+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa604157150 con 0x7fa5ec003a50 2024-06-29T14:22:05.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.914+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6040b4880 con 0x7fa5ec003a50 2024-06-29T14:22:05.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.915+0000 7fa6027fc640 1 -- 172.21.15.59:0/1506018934 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa5fc045020 con 0x7fa5ec003a50 2024-06-29T14:22:05.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.915+0000 7fa6027fc640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa5f0076300 0x7fa5f0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:05.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.915+0000 7fa6027fc640 1 -- 172.21.15.59:0/1506018934 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fa5fc0b8470 con 0x7fa5ec003a50 2024-06-29T14:22:05.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.915+0000 7fa6097bd640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa5f0076300 0x7fa5f0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:05.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.916+0000 7fa6097bd640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa5f0076300 0x7fa5f0078810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8001880 tx=0x7fa5f8000ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:05.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:05.920+0000 7fa6027fc640 1 -- 172.21.15.59:0/1506018934 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa604159be0 con 0x7fa5ec003a50 2024-06-29T14:22:06.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.108+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa60406c380 con 0x7fa5ec003a50 2024-06-29T14:22:06.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.108+0000 7fa6027fc640 1 -- 172.21.15.59:0/1506018934 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fa5fc081a90 con 0x7fa5ec003a50 2024-06-29T14:22:06.106 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:06.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa5f0076300 msgr2=0x7fa5f0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa5f0076300 0x7fa5f0078810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8001880 tx=0x7fa5f8000ed0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 msgr2=0x7fa6040c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 0x7fa6040c95b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009c70 tx=0x7fa5fc033670 comp rx=0 tx=0).stop 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 shutdown_connections 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa5f0076300 0x7fa5f0078810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604159290 0x7fa6041596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6041e78c0 0x7fa604150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 --2- 172.21.15.59:0/1506018934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5ec003a50 0x7fa6040c95b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 >> 172.21.15.59:0/1506018934 conn(0x7fa6040d45b0 msgr2=0x7fa60415cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 shutdown_connections 2024-06-29T14:22:06.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.110+0000 7fa60c222640 1 -- 172.21.15.59:0/1506018934 wait complete. 2024-06-29T14:22:06.119 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:06.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.242+0000 7f1515d3c640 1 -- 172.21.15.59:0/288755098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1510162770 msgr2=0x7f15100ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.242+0000 7f1515d3c640 1 --2- 172.21.15.59:0/288755098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1510162770 0x7f15100ce550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1500009830 tx=0x7f150002a480 comp rx=0 tx=0).stop 2024-06-29T14:22:06.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.242+0000 7f1515d3c640 1 -- 172.21.15.59:0/288755098 shutdown_connections 2024-06-29T14:22:06.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.242+0000 7f1515d3c640 1 --2- 172.21.15.59:0/288755098 >> v2:172.21.15.153:3300/0 conn(0x7f1510162190 0x7f15100b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.242+0000 7f1515d3c640 1 --2- 172.21.15.59:0/288755098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1510162770 0x7f15100ce550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.242+0000 7f1515d3c640 1 -- 172.21.15.59:0/288755098 >> 172.21.15.59:0/288755098 conn(0x7f15100d45b0 msgr2=0x7f15100d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:06.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.242+0000 7f1515d3c640 1 -- 172.21.15.59:0/288755098 shutdown_connections 2024-06-29T14:22:06.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.242+0000 7f1515d3c640 1 -- 172.21.15.59:0/288755098 wait complete. 2024-06-29T14:22:06.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.244+0000 7f1515d3c640 1 Processor -- start 2024-06-29T14:22:06.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.244+0000 7f1515d3c640 1 -- start start 2024-06-29T14:22:06.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f1515d3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151015e0a0 0x7f151015e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f1515d3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1510162770 0x7f151014fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f1515d3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15101e8a40 0x7f15101e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f1515d3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f15100b43a0 con 0x7f15101e8a40 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f1515d3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15100b4220 con 0x7f1510162770 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f1515d3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f15100b4520 con 0x7f151015e0a0 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f150ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15101e8a40 0x7f15101e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f150ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15101e8a40 0x7f15101e8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40516/0 (socket says 172.21.15.59:40516) 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f150ffff640 1 -- 172.21.15.59:0/652180427 learned_addr learned my addr 172.21.15.59:0/652180427 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f150f7fe640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151015e0a0 0x7f151015e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.245+0000 7f150effd640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1510162770 0x7f151014fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f150ffff640 1 -- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151015e0a0 msgr2=0x7f151015e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f150ffff640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151015e0a0 0x7f151015e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f150ffff640 1 -- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1510162770 msgr2=0x7f151014fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f150ffff640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1510162770 0x7f151014fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f150ffff640 1 -- 172.21.15.59:0/652180427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151015ea80 con 0x7f15101e8a40 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f150ffff640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15101e8a40 0x7f15101e8e50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f150800ea10 tx=0x7f150800eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f14f7fff640 1 -- 172.21.15.59:0/652180427 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150800cde0 con 0x7f15101e8a40 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f14f7fff640 1 -- 172.21.15.59:0/652180427 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1508004c10 con 0x7f15101e8a40 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15101e9390 con 0x7f15101e8a40 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f14f7fff640 1 -- 172.21.15.59:0/652180427 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508005020 con 0x7f15101e8a40 2024-06-29T14:22:06.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.246+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1510161b60 con 0x7f15101e8a40 2024-06-29T14:22:06.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.247+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15101615f0 con 0x7f15101e8a40 2024-06-29T14:22:06.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.248+0000 7f14f7fff640 1 -- 172.21.15.59:0/652180427 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f15080051c0 con 0x7f15101e8a40 2024-06-29T14:22:06.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.248+0000 7f14f7fff640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14fc076300 0x7f14fc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.249+0000 7f14f7fff640 1 -- 172.21.15.59:0/652180427 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1508098760 con 0x7f15101e8a40 2024-06-29T14:22:06.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.249+0000 7f150f7fe640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14fc076300 0x7f14fc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.249+0000 7f150f7fe640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14fc076300 0x7f14fc078810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1500034000 tx=0x7f1500004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.253+0000 7f14f7fff640 1 -- 172.21.15.59:0/652180427 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f15101e9390 con 0x7f15101e8a40 2024-06-29T14:22:06.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.441+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f15100b49f0 con 0x7f15101e8a40 2024-06-29T14:22:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.442+0000 7f14f7fff640 1 -- 172.21.15.59:0/652180427 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f1508061690 con 0x7f15101e8a40 2024-06-29T14:22:06.440 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:06.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14fc076300 msgr2=0x7f14fc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14fc076300 0x7f14fc078810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1500034000 tx=0x7f1500004730 comp rx=0 tx=0).stop 2024-06-29T14:22:06.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15101e8a40 msgr2=0x7f15101e8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15101e8a40 0x7f15101e8e50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f150800ea10 tx=0x7f150800eed0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 shutdown_connections 2024-06-29T14:22:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14fc076300 0x7f14fc078810 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15101e8a40 0x7f15101e8e50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1510162770 0x7f151014fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 --2- 172.21.15.59:0/652180427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f151015e0a0 0x7f151015e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 >> 172.21.15.59:0/652180427 conn(0x7f15100d45b0 msgr2=0x7f15100d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 shutdown_connections 2024-06-29T14:22:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.444+0000 7f1515d3c640 1 -- 172.21.15.59:0/652180427 wait complete. 2024-06-29T14:22:06.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:06.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.575+0000 7f6d72deb640 1 -- 172.21.15.59:0/2788442804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c143840 msgr2=0x7f6d6c06e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.575+0000 7f6d72deb640 1 --2- 172.21.15.59:0/2788442804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c143840 0x7f6d6c06e4d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d54005fd0 tx=0x7f6d5402a820 comp rx=0 tx=0).stop 2024-06-29T14:22:06.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.575+0000 7f6d72deb640 1 -- 172.21.15.59:0/2788442804 shutdown_connections 2024-06-29T14:22:06.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.575+0000 7f6d72deb640 1 --2- 172.21.15.59:0/2788442804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c143840 0x7f6d6c06e4d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.575+0000 7f6d72deb640 1 -- 172.21.15.59:0/2788442804 >> 172.21.15.59:0/2788442804 conn(0x7f6d6c0b2ab0 msgr2=0x7f6d6c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:06.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.575+0000 7f6d72deb640 1 -- 172.21.15.59:0/2788442804 shutdown_connections 2024-06-29T14:22:06.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.576+0000 7f6d72deb640 1 -- 172.21.15.59:0/2788442804 wait complete. 2024-06-29T14:22:06.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.577+0000 7f6d72deb640 1 Processor -- start 2024-06-29T14:22:06.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.577+0000 7f6d72deb640 1 -- start start 2024-06-29T14:22:06.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d72deb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d6c06ea10 0x7f6d6c158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d72deb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c0d0130 0x7f6d6c1f5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d72deb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d6c143840 0x7f6d6c1eed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d72deb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c0cf820 con 0x7f6d6c0d0130 2024-06-29T14:22:06.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d72deb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c0cf6a0 con 0x7f6d6c06ea10 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d72deb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6d6c0cf9a0 con 0x7f6d6c143840 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c0d0130 0x7f6d6c1f5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c0d0130 0x7f6d6c1f5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40524/0 (socket says 172.21.15.59:40524) 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 -- 172.21.15.59:0/58375793 learned_addr learned my addr 172.21.15.59:0/58375793 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d70b87640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d6c06ea10 0x7f6d6c158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d71388640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d6c143840 0x7f6d6c1eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 -- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d6c143840 msgr2=0x7f6d6c1eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d6c143840 0x7f6d6c1eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 -- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d6c06ea10 msgr2=0x7f6d6c158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d6c06ea10 0x7f6d6c158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 -- 172.21.15.59:0/58375793 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6c159780 con 0x7f6d6c0d0130 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d71388640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d6c143840 0x7f6d6c1eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.579+0000 7f6d6bfff640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c0d0130 0x7f6d6c1f5840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d54009cd0 tx=0x7f6d54032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:06.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.580+0000 7f6d697fa640 1 -- 172.21.15.59:0/58375793 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54037040 con 0x7f6d6c0d0130 2024-06-29T14:22:06.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.580+0000 7f6d697fa640 1 -- 172.21.15.59:0/58375793 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d54042070 con 0x7f6d6c0d0130 2024-06-29T14:22:06.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.580+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d6c1f4d60 con 0x7f6d6c0d0130 2024-06-29T14:22:06.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.580+0000 7f6d697fa640 1 -- 172.21.15.59:0/58375793 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54004280 con 0x7f6d6c0d0130 2024-06-29T14:22:06.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.580+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d6c1f5320 con 0x7f6d6c0d0130 2024-06-29T14:22:06.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.581+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d6c0cfe90 con 0x7f6d6c0d0130 2024-06-29T14:22:06.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.582+0000 7f6d697fa640 1 -- 172.21.15.59:0/58375793 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6d54004420 con 0x7f6d6c0d0130 2024-06-29T14:22:06.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.582+0000 7f6d697fa640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f6d4c0762d0 0x7f6d4c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.582+0000 7f6d697fa640 1 -- 172.21.15.59:0/58375793 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6d540b8810 con 0x7f6d6c0d0130 2024-06-29T14:22:06.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.582+0000 7f6d70b87640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f6d4c0762d0 0x7f6d4c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.583+0000 7f6d70b87640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f6d4c0762d0 0x7f6d4c0787e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c006fd0 tx=0x7f6d5c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:06.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.587+0000 7f6d697fa640 1 -- 172.21.15.59:0/58375793 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6d54081db0 con 0x7f6d6c0d0130 2024-06-29T14:22:06.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.776+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6d6c159510 con 0x7f6d6c0d0130 2024-06-29T14:22:06.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.777+0000 7f6d697fa640 1 -- 172.21.15.59:0/58375793 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f6d540816b0 con 0x7f6d6c0d0130 2024-06-29T14:22:06.775 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.778+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f6d4c0762d0 msgr2=0x7f6d4c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f6d4c0762d0 0x7f6d4c0787e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c006fd0 tx=0x7f6d5c009060 comp rx=0 tx=0).stop 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c0d0130 msgr2=0x7f6d6c1f5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c0d0130 0x7f6d6c1f5840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d54009cd0 tx=0x7f6d54032d10 comp rx=0 tx=0).stop 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 shutdown_connections 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f6d4c0762d0 0x7f6d4c0787e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d6c143840 0x7f6d6c1eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d6c0d0130 0x7f6d6c1f5840 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 --2- 172.21.15.59:0/58375793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d6c06ea10 0x7f6d6c158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 >> 172.21.15.59:0/58375793 conn(0x7f6d6c0b2ab0 msgr2=0x7f6d6c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 shutdown_connections 2024-06-29T14:22:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.779+0000 7f6d72deb640 1 -- 172.21.15.59:0/58375793 wait complete. 2024-06-29T14:22:06.788 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:22:06.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.909+0000 7f463dbc7640 1 -- 172.21.15.59:0/246775608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46380ce650 msgr2=0x7f46380b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.909+0000 7f463dbc7640 1 --2- 172.21.15.59:0/246775608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46380ce650 0x7f46380b3990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f462c009830 tx=0x7f462c02a500 comp rx=0 tx=0).stop 2024-06-29T14:22:06.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.909+0000 7f463dbc7640 1 -- 172.21.15.59:0/246775608 shutdown_connections 2024-06-29T14:22:06.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.909+0000 7f463dbc7640 1 --2- 172.21.15.59:0/246775608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46380ce650 0x7f46380b3990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.909+0000 7f463dbc7640 1 --2- 172.21.15.59:0/246775608 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463815e290 0x7f46380ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.909+0000 7f463dbc7640 1 -- 172.21.15.59:0/246775608 >> 172.21.15.59:0/246775608 conn(0x7f46380d3ee0 msgr2=0x7f46380d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:06.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.909+0000 7f463dbc7640 1 -- 172.21.15.59:0/246775608 shutdown_connections 2024-06-29T14:22:06.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.909+0000 7f463dbc7640 1 -- 172.21.15.59:0/246775608 wait complete. 2024-06-29T14:22:06.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.911+0000 7f463dbc7640 1 Processor -- start 2024-06-29T14:22:06.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.911+0000 7f463dbc7640 1 -- start start 2024-06-29T14:22:06.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f463dbc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46380b3ed0 0x7f463815e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f463dbc7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46380ce650 0x7f4638149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f463dbc7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463815e290 0x7f4638158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f463dbc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f46380d1f20 con 0x7f46380b3ed0 2024-06-29T14:22:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f463dbc7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46380d1da0 con 0x7f46380ce650 2024-06-29T14:22:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f463dbc7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46380d20a0 con 0x7f463815e290 2024-06-29T14:22:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f46377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46380b3ed0 0x7f463815e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f46377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46380b3ed0 0x7f463815e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40540/0 (socket says 172.21.15.59:40540) 2024-06-29T14:22:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.912+0000 7f46377fe640 1 -- 172.21.15.59:0/3010989338 learned_addr learned my addr 172.21.15.59:0/3010989338 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f4637fff640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463815e290 0x7f4638158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f4636ffd640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46380ce650 0x7f4638149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f46377fe640 1 -- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463815e290 msgr2=0x7f4638158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f46377fe640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463815e290 0x7f4638158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f46377fe640 1 -- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46380ce650 msgr2=0x7f4638149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f46377fe640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46380ce650 0x7f4638149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f46377fe640 1 -- 172.21.15.59:0/3010989338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463815eef0 con 0x7f46380b3ed0 2024-06-29T14:22:06.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f46377fe640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46380b3ed0 0x7f463815e6a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f462800e9f0 tx=0x7f462800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:06.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f461ffff640 1 -- 172.21.15.59:0/3010989338 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462800cd80 con 0x7f46380b3ed0 2024-06-29T14:22:06.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f461ffff640 1 -- 172.21.15.59:0/3010989338 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4628004020 con 0x7f46380b3ed0 2024-06-29T14:22:06.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463815f1d0 con 0x7f46380b3ed0 2024-06-29T14:22:06.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f461ffff640 1 -- 172.21.15.59:0/3010989338 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4628004470 con 0x7f46380b3ed0 2024-06-29T14:22:06.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.913+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463815f790 con 0x7f46380b3ed0 2024-06-29T14:22:06.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.914+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46380b3990 con 0x7f46380b3ed0 2024-06-29T14:22:06.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.915+0000 7f461ffff640 1 -- 172.21.15.59:0/3010989338 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f46280035f0 con 0x7f46380b3ed0 2024-06-29T14:22:06.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.916+0000 7f461ffff640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f46240762d0 0x7f46240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:06.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.916+0000 7f461ffff640 1 -- 172.21.15.59:0/3010989338 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f46280984a0 con 0x7f46380b3ed0 2024-06-29T14:22:06.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.916+0000 7f4636ffd640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f46240762d0 0x7f46240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:06.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.916+0000 7f4636ffd640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f46240762d0 0x7f46240787e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f462c034000 tx=0x7f462c004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:06.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:06.920+0000 7f461ffff640 1 -- 172.21.15.59:0/3010989338 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f46280613d0 con 0x7f46380b3ed0 2024-06-29T14:22:07.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.123+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f463815c6e0 con 0x7f46380b3ed0 2024-06-29T14:22:07.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.124+0000 7f461ffff640 1 -- 172.21.15.59:0/3010989338 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f46280656d0 con 0x7f46380b3ed0 2024-06-29T14:22:07.122 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-06-29T14:22:07.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f46240762d0 msgr2=0x7f46240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f46240762d0 0x7f46240787e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f462c034000 tx=0x7f462c004730 comp rx=0 tx=0).stop 2024-06-29T14:22:07.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46380b3ed0 msgr2=0x7f463815e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46380b3ed0 0x7f463815e6a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f462800e9f0 tx=0x7f462800eeb0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 shutdown_connections 2024-06-29T14:22:07.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f46240762d0 0x7f46240787e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463815e290 0x7f4638158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46380ce650 0x7f4638149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 --2- 172.21.15.59:0/3010989338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46380b3ed0 0x7f463815e6a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 >> 172.21.15.59:0/3010989338 conn(0x7f46380d3ee0 msgr2=0x7f46380b8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:07.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 shutdown_connections 2024-06-29T14:22:07.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.126+0000 7f463dbc7640 1 -- 172.21.15.59:0/3010989338 wait complete. 2024-06-29T14:22:07.135 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:07.135 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:07.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:07.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.310+0000 7f2d629fc640 1 -- 172.21.15.59:0/1547389707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162190 msgr2=0x7f2d5c0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.310+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1547389707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162190 0x7f2d5c0b38e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d54005fd0 tx=0x7f2d5402aa10 comp rx=0 tx=0).stop 2024-06-29T14:22:07.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.311+0000 7f2d629fc640 1 -- 172.21.15.59:0/1547389707 shutdown_connections 2024-06-29T14:22:07.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.311+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1547389707 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162190 0x7f2d5c0b38e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.311+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1547389707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5c162770 0x7f2d5c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.311+0000 7f2d629fc640 1 -- 172.21.15.59:0/1547389707 >> 172.21.15.59:0/1547389707 conn(0x7f2d5c0d45b0 msgr2=0x7f2d5c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:07.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.311+0000 7f2d629fc640 1 -- 172.21.15.59:0/1547389707 shutdown_connections 2024-06-29T14:22:07.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.311+0000 7f2d629fc640 1 -- 172.21.15.59:0/1547389707 wait complete. 2024-06-29T14:22:07.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.312+0000 7f2d629fc640 1 Processor -- start 2024-06-29T14:22:07.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.313+0000 7f2d629fc640 1 -- start start 2024-06-29T14:22:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d629fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5c15e0a0 0x7f2d5c1f8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d629fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162770 0x7f2d5c15af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d629fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5c15e540 0x7f2d5c15e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d629fc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c0b43a0 con 0x7f2d5c162770 2024-06-29T14:22:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d629fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c0b4220 con 0x7f2d5c15e0a0 2024-06-29T14:22:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d629fc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5c0b4520 con 0x7f2d5c15e540 2024-06-29T14:22:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162770 0x7f2d5c15af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d5bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5c15e0a0 0x7f2d5c1f8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162770 0x7f2d5c15af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40550/0 (socket says 172.21.15.59:40550) 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 -- 172.21.15.59:0/1257096412 learned_addr learned my addr 172.21.15.59:0/1257096412 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d60f99640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5c15e540 0x7f2d5c15e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 -- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5c15e540 msgr2=0x7f2d5c15e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5c15e540 0x7f2d5c15e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 -- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5c15e0a0 msgr2=0x7f2d5c1f8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5c15e0a0 0x7f2d5c1f8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 -- 172.21.15.59:0/1257096412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5c15ee90 con 0x7f2d5c162770 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.314+0000 7f2d53fff640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162770 0x7f2d5c15af50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400c3c0 tx=0x7f2d4400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.315+0000 7f2d597fa640 1 -- 172.21.15.59:0/1257096412 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44017070 con 0x7f2d5c162770 2024-06-29T14:22:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.315+0000 7f2d597fa640 1 -- 172.21.15.59:0/1257096412 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d440053e0 con 0x7f2d5c162770 2024-06-29T14:22:07.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.315+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5c0ca480 con 0x7f2d5c162770 2024-06-29T14:22:07.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.315+0000 7f2d597fa640 1 -- 172.21.15.59:0/1257096412 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44003870 con 0x7f2d5c162770 2024-06-29T14:22:07.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.315+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d5c0c8710 con 0x7f2d5c162770 2024-06-29T14:22:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.316+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d5c0b49d0 con 0x7f2d5c162770 2024-06-29T14:22:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.317+0000 7f2d597fa640 1 -- 172.21.15.59:0/1257096412 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d4401a070 con 0x7f2d5c162770 2024-06-29T14:22:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.318+0000 7f2d597fa640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2d3c0762d0 0x7f2d3c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.318+0000 7f2d597fa640 1 -- 172.21.15.59:0/1257096412 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2d44096ee0 con 0x7f2d5c162770 2024-06-29T14:22:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.318+0000 7f2d5bfff640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2d3c0762d0 0x7f2d3c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.319+0000 7f2d5bfff640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2d3c0762d0 0x7f2d3c0787e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009b30 tx=0x7f2d4c005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:07.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.322+0000 7f2d597fa640 1 -- 172.21.15.59:0/1257096412 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d5c15ee90 con 0x7f2d5c162770 2024-06-29T14:22:07.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.517+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d5c0cd160 con 0x7f2d5c162770 2024-06-29T14:22:07.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.518+0000 7f2d597fa640 1 -- 172.21.15.59:0/1257096412 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7f2d4405fe60 con 0x7f2d5c162770 2024-06-29T14:22:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:07.517 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2d3c0762d0 msgr2=0x7f2d3c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2d3c0762d0 0x7f2d3c0787e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009b30 tx=0x7f2d4c005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162770 msgr2=0x7f2d5c15af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162770 0x7f2d5c15af50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400c3c0 tx=0x7f2d4400c880 comp rx=0 tx=0).stop 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 shutdown_connections 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2d3c0762d0 0x7f2d3c0787e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5c15e540 0x7f2d5c15e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d5c162770 0x7f2d5c15af50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 --2- 172.21.15.59:0/1257096412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5c15e0a0 0x7f2d5c1f8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 >> 172.21.15.59:0/1257096412 conn(0x7f2d5c0d45b0 msgr2=0x7f2d5c15bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 shutdown_connections 2024-06-29T14:22:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.520+0000 7f2d629fc640 1 -- 172.21.15.59:0/1257096412 wait complete. 2024-06-29T14:22:07.530 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:07.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:07.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.652+0000 7f2509240640 1 -- 172.21.15.59:0/1018313762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25040ce950 msgr2=0x7f25040d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.652+0000 7f2509240640 1 --2- 172.21.15.59:0/1018313762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25040ce950 0x7f25040d2130 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005fd0 tx=0x7f24fc02aa90 comp rx=0 tx=0).stop 2024-06-29T14:22:07.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.652+0000 7f2509240640 1 -- 172.21.15.59:0/1018313762 shutdown_connections 2024-06-29T14:22:07.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.652+0000 7f2509240640 1 --2- 172.21.15.59:0/1018313762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25040ce950 0x7f25040d2130 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.652+0000 7f2509240640 1 --2- 172.21.15.59:0/1018313762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f250408c190 0x7f25040cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.652+0000 7f2509240640 1 -- 172.21.15.59:0/1018313762 >> 172.21.15.59:0/1018313762 conn(0x7f25040b5610 msgr2=0x7f25040b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:07.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.653+0000 7f2509240640 1 -- 172.21.15.59:0/1018313762 shutdown_connections 2024-06-29T14:22:07.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.653+0000 7f2509240640 1 -- 172.21.15.59:0/1018313762 wait complete. 2024-06-29T14:22:07.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.655+0000 7f2509240640 1 Processor -- start 2024-06-29T14:22:07.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.655+0000 7f2509240640 1 -- start start 2024-06-29T14:22:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2509240640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f250408c190 0x7f250414cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2509240640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25040ce950 0x7f2504161910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2509240640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25040cf5c0 0x7f25041599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2509240640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f25040b7fa0 con 0x7f250408c190 2024-06-29T14:22:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2509240640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f25040b7e20 con 0x7f25040ce950 2024-06-29T14:22:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2509240640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f25040b8120 con 0x7f25040cf5c0 2024-06-29T14:22:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2502d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f250408c190 0x7f250414cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2502d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f250408c190 0x7f250414cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40570/0 (socket says 172.21.15.59:40570) 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2502d9d640 1 -- 172.21.15.59:0/3646458267 learned_addr learned my addr 172.21.15.59:0/3646458267 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f250259c640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25040ce950 0x7f2504161910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f250359e640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25040cf5c0 0x7f25041599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2502d9d640 1 -- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25040cf5c0 msgr2=0x7f25041599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2502d9d640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25040cf5c0 0x7f25041599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2502d9d640 1 -- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25040ce950 msgr2=0x7f2504161910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2502d9d640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25040ce950 0x7f2504161910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.656+0000 7f2502d9d640 1 -- 172.21.15.59:0/3646458267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2504156e70 con 0x7f250408c190 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.657+0000 7f2502d9d640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f250408c190 0x7f250414cd00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f24f400e7c0 tx=0x7f24f400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.657+0000 7f24eb7fe640 1 -- 172.21.15.59:0/3646458267 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f400ccc0 con 0x7f250408c190 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.657+0000 7f24eb7fe640 1 -- 172.21.15.59:0/3646458267 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24f4010430 con 0x7f250408c190 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.657+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f250415f970 con 0x7f250408c190 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.657+0000 7f24eb7fe640 1 -- 172.21.15.59:0/3646458267 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f4010860 con 0x7f250408c190 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.657+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f250415a500 con 0x7f250408c190 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.658+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f250415a760 con 0x7f250408c190 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.658+0000 7f24eb7fe640 1 -- 172.21.15.59:0/3646458267 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f24f4010a00 con 0x7f250408c190 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.659+0000 7f24eb7fe640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f24f0076300 0x7f24f0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.659+0000 7f24eb7fe640 1 -- 172.21.15.59:0/3646458267 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f24f40982e0 con 0x7f250408c190 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.659+0000 7f250259c640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f24f0076300 0x7f24f0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.660+0000 7f250259c640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f24f0076300 0x7f24f0078810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f25040c87d0 tx=0x7f24f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:07.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.663+0000 7f24eb7fe640 1 -- 172.21.15.59:0/3646458267 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f24f40612b0 con 0x7f250408c190 2024-06-29T14:22:07.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.855+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f25040cf080 con 0x7f250408c190 2024-06-29T14:22:07.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.856+0000 7f24eb7fe640 1 -- 172.21.15.59:0/3646458267 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f24f40655b0 con 0x7f250408c190 2024-06-29T14:22:07.854 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:07.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f24f0076300 msgr2=0x7f24f0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f24f0076300 0x7f24f0078810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f25040c87d0 tx=0x7f24f8009040 comp rx=0 tx=0).stop 2024-06-29T14:22:07.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f250408c190 msgr2=0x7f250414cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f250408c190 0x7f250414cd00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f24f400e7c0 tx=0x7f24f400ec80 comp rx=0 tx=0).stop 2024-06-29T14:22:07.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 shutdown_connections 2024-06-29T14:22:07.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f24f0076300 0x7f24f0078810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25040cf5c0 0x7f25041599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25040ce950 0x7f2504161910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 --2- 172.21.15.59:0/3646458267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f250408c190 0x7f250414cd00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 >> 172.21.15.59:0/3646458267 conn(0x7f25040b5610 msgr2=0x7f25040c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:07.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 shutdown_connections 2024-06-29T14:22:07.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.858+0000 7f2509240640 1 -- 172.21.15.59:0/3646458267 wait complete. 2024-06-29T14:22:07.867 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:07.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.989+0000 7f2dc16c3640 1 -- 172.21.15.59:0/4232904023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0d23c0 msgr2=0x7f2dbc0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.989+0000 7f2dc16c3640 1 --2- 172.21.15.59:0/4232904023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0d23c0 0x7f2dbc0b3990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2db4009830 tx=0x7f2db402a510 comp rx=0 tx=0).stop 2024-06-29T14:22:07.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.989+0000 7f2dc16c3640 1 -- 172.21.15.59:0/4232904023 shutdown_connections 2024-06-29T14:22:07.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.989+0000 7f2dc16c3640 1 --2- 172.21.15.59:0/4232904023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0d23c0 0x7f2dbc0b3990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.989+0000 7f2dc16c3640 1 -- 172.21.15.59:0/4232904023 >> 172.21.15.59:0/4232904023 conn(0x7f2dbc0d3ee0 msgr2=0x7f2dbc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:07.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.989+0000 7f2dc16c3640 1 -- 172.21.15.59:0/4232904023 shutdown_connections 2024-06-29T14:22:07.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.989+0000 7f2dc16c3640 1 -- 172.21.15.59:0/4232904023 wait complete. 2024-06-29T14:22:07.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.991+0000 7f2dc16c3640 1 Processor -- start 2024-06-29T14:22:07.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.991+0000 7f2dc16c3640 1 -- start start 2024-06-29T14:22:07.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dc16c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0b3ed0 0x7f2dbc0cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dc16c3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dbc0d23c0 0x7f2dbc152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dc16c3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dbc15efe0 0x7f2dbc153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dc16c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc160450 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dc16c3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc1602d0 con 0x7f2dbc0d23c0 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dc16c3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2dbc1605d0 con 0x7f2dbc15efe0 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dbaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0b3ed0 0x7f2dbc0cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dbaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0b3ed0 0x7f2dbc0cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40586/0 (socket says 172.21.15.59:40586) 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dbaffd640 1 -- 172.21.15.59:0/3930603936 learned_addr learned my addr 172.21.15.59:0/3930603936 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dba7fc640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dbc0d23c0 0x7f2dbc152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.992+0000 7f2dbb7fe640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dbc15efe0 0x7f2dbc153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2dbaffd640 1 -- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dbc15efe0 msgr2=0x7f2dbc153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2dbaffd640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dbc15efe0 0x7f2dbc153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2dbaffd640 1 -- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dbc0d23c0 msgr2=0x7f2dbc152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:07.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2dbaffd640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dbc0d23c0 0x7f2dbc152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:07.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2dbaffd640 1 -- 172.21.15.59:0/3930603936 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dbc0cda20 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2dbaffd640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0b3ed0 0x7f2dbc0cd4e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2db402ab00 tx=0x7f2db4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:07.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2da37fe640 1 -- 172.21.15.59:0/3930603936 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db4031d60 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2da37fe640 1 -- 172.21.15.59:0/3930603936 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2db4033e10 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dbc150200 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2da37fe640 1 -- 172.21.15.59:0/3930603936 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db4005260 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.993+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dbc150780 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.994+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dbc0cabc0 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.995+0000 7f2da37fe640 1 -- 172.21.15.59:0/3930603936 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2db4005400 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.995+0000 7f2da37fe640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2da80762d0 0x7f2da80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:07.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.996+0000 7f2da37fe640 1 -- 172.21.15.59:0/3930603936 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2db40b8d10 con 0x7f2dbc0b3ed0 2024-06-29T14:22:07.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.996+0000 7f2dba7fc640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2da80762d0 0x7f2da80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:07.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:07.996+0000 7f2dba7fc640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2da80762d0 0x7f2da80787e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0c7370 tx=0x7f2db0009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:07.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.000+0000 7f2da37fe640 1 -- 172.21.15.59:0/3930603936 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2db40be020 con 0x7f2dbc0b3ed0 2024-06-29T14:22:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.190+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2dbc0689f0 con 0x7f2dbc0b3ed0 2024-06-29T14:22:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.190+0000 7f2da37fe640 1 -- 172.21.15.59:0/3930603936 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f2dbc0689f0 con 0x7f2dbc0b3ed0 2024-06-29T14:22:08.188 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2da80762d0 msgr2=0x7f2da80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2da80762d0 0x7f2da80787e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0c7370 tx=0x7f2db0009060 comp rx=0 tx=0).stop 2024-06-29T14:22:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0b3ed0 msgr2=0x7f2dbc0cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0b3ed0 0x7f2dbc0cd4e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2db402ab00 tx=0x7f2db4033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 shutdown_connections 2024-06-29T14:22:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2da80762d0 0x7f2da80787e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dbc15efe0 0x7f2dbc153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dbc0d23c0 0x7f2dbc152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 --2- 172.21.15.59:0/3930603936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc0b3ed0 0x7f2dbc0cd4e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 >> 172.21.15.59:0/3930603936 conn(0x7f2dbc0d3ee0 msgr2=0x7f2dbc0b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 shutdown_connections 2024-06-29T14:22:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.192+0000 7f2dc16c3640 1 -- 172.21.15.59:0/3930603936 wait complete. 2024-06-29T14:22:08.202 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:08.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.325+0000 7f21e0e09640 1 -- 172.21.15.59:0/963701694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21dc162190 msgr2=0x7f21dc0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.325+0000 7f21e0e09640 1 --2- 172.21.15.59:0/963701694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21dc162190 0x7f21dc0ce550 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009830 tx=0x7f21c402a480 comp rx=0 tx=0).stop 2024-06-29T14:22:08.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.325+0000 7f21e0e09640 1 -- 172.21.15.59:0/963701694 shutdown_connections 2024-06-29T14:22:08.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.325+0000 7f21e0e09640 1 --2- 172.21.15.59:0/963701694 >> v2:172.21.15.153:3300/0 conn(0x7f21dc15e0a0 0x7f21dc0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.325+0000 7f21e0e09640 1 --2- 172.21.15.59:0/963701694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21dc162190 0x7f21dc0ce550 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.325+0000 7f21e0e09640 1 -- 172.21.15.59:0/963701694 >> 172.21.15.59:0/963701694 conn(0x7f21dc0d45b0 msgr2=0x7f21dc0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:08.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.325+0000 7f21e0e09640 1 -- 172.21.15.59:0/963701694 shutdown_connections 2024-06-29T14:22:08.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.325+0000 7f21e0e09640 1 -- 172.21.15.59:0/963701694 wait complete. 2024-06-29T14:22:08.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.327+0000 7f21e0e09640 1 Processor -- start 2024-06-29T14:22:08.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.327+0000 7f21e0e09640 1 -- start start 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.328+0000 7f21e0e09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21dc15e0a0 0x7f21dc158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.328+0000 7f21e0e09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21dc162770 0x7f21dc152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.328+0000 7f21e0e09640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc155fb0 0x7f21dc153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.328+0000 7f21e0e09640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f21dc0b43b0 con 0x7f21dc155fb0 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.328+0000 7f21e0e09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f21dc0b4230 con 0x7f21dc162770 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.328+0000 7f21e0e09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f21dc0b4530 con 0x7f21dc15e0a0 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.328+0000 7f21dad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc155fb0 0x7f21dc153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21dad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc155fb0 0x7f21dc153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40594/0 (socket says 172.21.15.59:40594) 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21dad9d640 1 -- 172.21.15.59:0/3137768654 learned_addr learned my addr 172.21.15.59:0/3137768654 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:08.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21d9d9b640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21dc162770 0x7f21dc152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21da59c640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21dc15e0a0 0x7f21dc158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21dad9d640 1 -- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21dc15e0a0 msgr2=0x7f21dc158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21dad9d640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21dc15e0a0 0x7f21dc158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21dad9d640 1 -- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21dc162770 msgr2=0x7f21dc152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21dad9d640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21dc162770 0x7f21dc152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21dad9d640 1 -- 172.21.15.59:0/3137768654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21dc157100 con 0x7f21dc155fb0 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21d9d9b640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21dc162770 0x7f21dc152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21dad9d640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc155fb0 0x7f21dc153b80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00ea40 tx=0x7f21cc00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21d2ffd640 1 -- 172.21.15.59:0/3137768654 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc00ce40 con 0x7f21dc155fb0 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21d2ffd640 1 -- 172.21.15.59:0/3137768654 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21cc004c10 con 0x7f21dc155fb0 2024-06-29T14:22:08.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21dc1573c0 con 0x7f21dc155fb0 2024-06-29T14:22:08.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21d2ffd640 1 -- 172.21.15.59:0/3137768654 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc005020 con 0x7f21dc155fb0 2024-06-29T14:22:08.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.329+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21dc157980 con 0x7f21dc155fb0 2024-06-29T14:22:08.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.330+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21dc06a840 con 0x7f21dc155fb0 2024-06-29T14:22:08.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.331+0000 7f21d2ffd640 1 -- 172.21.15.59:0/3137768654 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f21cc0051c0 con 0x7f21dc155fb0 2024-06-29T14:22:08.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.331+0000 7f21d2ffd640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f21bc0762d0 0x7f21bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.332+0000 7f21d2ffd640 1 -- 172.21.15.59:0/3137768654 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f21cc098840 con 0x7f21dc155fb0 2024-06-29T14:22:08.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.332+0000 7f21da59c640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f21bc0762d0 0x7f21bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.332+0000 7f21da59c640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f21bc0762d0 0x7f21bc0787e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0b5480 tx=0x7f21d4007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:08.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.336+0000 7f21d2ffd640 1 -- 172.21.15.59:0/3137768654 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f21dc06a840 con 0x7f21dc155fb0 2024-06-29T14:22:08.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.525+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f21dc0b49e0 con 0x7f21dc155fb0 2024-06-29T14:22:08.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.526+0000 7f21d2ffd640 1 -- 172.21.15.59:0/3137768654 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f21cc061d50 con 0x7f21dc155fb0 2024-06-29T14:22:08.524 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:08.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f21bc0762d0 msgr2=0x7f21bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f21bc0762d0 0x7f21bc0787e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0b5480 tx=0x7f21d4007eb0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc155fb0 msgr2=0x7f21dc153b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc155fb0 0x7f21dc153b80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00ea40 tx=0x7f21cc00ef00 comp rx=0 tx=0).stop 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 shutdown_connections 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f21bc0762d0 0x7f21bc0787e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc155fb0 0x7f21dc153b80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21dc162770 0x7f21dc152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 --2- 172.21.15.59:0/3137768654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21dc15e0a0 0x7f21dc158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 >> 172.21.15.59:0/3137768654 conn(0x7f21dc0d45b0 msgr2=0x7f21dc0b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 shutdown_connections 2024-06-29T14:22:08.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.528+0000 7f21e0e09640 1 -- 172.21.15.59:0/3137768654 wait complete. 2024-06-29T14:22:08.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.660+0000 7fcc01562640 1 -- 172.21.15.59:0/1637050984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 msgr2=0x7fcbfc0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.660+0000 7fcc01562640 1 --2- 172.21.15.59:0/1637050984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 0x7fcbfc0b3990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc161d40 tx=0x7fcbec02aac0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.661+0000 7fcc01562640 1 -- 172.21.15.59:0/1637050984 shutdown_connections 2024-06-29T14:22:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.661+0000 7fcc01562640 1 --2- 172.21.15.59:0/1637050984 >> v2:172.21.15.153:3300/0 conn(0x7fcbfc15efe0 0x7fcbfc15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.661+0000 7fcc01562640 1 --2- 172.21.15.59:0/1637050984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 0x7fcbfc0b3990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.661+0000 7fcc01562640 1 -- 172.21.15.59:0/1637050984 >> 172.21.15.59:0/1637050984 conn(0x7fcbfc0d3ee0 msgr2=0x7fcbfc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.661+0000 7fcc01562640 1 -- 172.21.15.59:0/1637050984 shutdown_connections 2024-06-29T14:22:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.661+0000 7fcc01562640 1 -- 172.21.15.59:0/1637050984 wait complete. 2024-06-29T14:22:08.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.662+0000 7fcc01562640 1 Processor -- start 2024-06-29T14:22:08.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.663+0000 7fcc01562640 1 -- start start 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcc01562640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbfc0b3ed0 0x7fcbfc151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcc01562640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 0x7fcbfc1f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcc01562640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbfc15efe0 0x7fcbfc1eaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcc01562640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc160280 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcc01562640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc160100 con 0x7fcbfc15efe0 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcc01562640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcbfc160400 con 0x7fcbfc0b3ed0 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 0x7fcbfc1f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 0x7fcbfc1f6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40608/0 (socket says 172.21.15.59:40608) 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbfc0b3ed0 0x7fcbfc151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 -- 172.21.15.59:0/1202929353 learned_addr learned my addr 172.21.15.59:0/1202929353 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfb7fe640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbfc15efe0 0x7fcbfc1eaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 -- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbfc0b3ed0 msgr2=0x7fcbfc151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbfc0b3ed0 0x7fcbfc151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 -- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbfc15efe0 msgr2=0x7fcbfc1eaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbfc15efe0 0x7fcbfc1eaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 -- 172.21.15.59:0/1202929353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc1f65d0 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.664+0000 7fcbfa7fc640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 0x7fcbfc1f6090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400c400 tx=0x7fcbe400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.665+0000 7fcbf37fe640 1 -- 172.21.15.59:0/1202929353 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4017070 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.665+0000 7fcbf37fe640 1 -- 172.21.15.59:0/1202929353 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbe40053c0 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.665+0000 7fcbf37fe640 1 -- 172.21.15.59:0/1202929353 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4003830 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.665+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc152380 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.665+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc152940 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.666+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbfc06d5a0 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.667+0000 7fcbf37fe640 1 -- 172.21.15.59:0/1202929353 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcbe401a070 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.667+0000 7fcbf37fe640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcbdc0762d0 0x7fcbdc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.667+0000 7fcbf37fe640 1 -- 172.21.15.59:0/1202929353 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fcbe4096d40 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.667+0000 7fcbfaffd640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcbdc0762d0 0x7fcbdc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.668+0000 7fcbfaffd640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcbdc0762d0 0x7fcbdc0787e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcbec035000 tx=0x7fcbec038040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:08.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.671+0000 7fcbf37fe640 1 -- 172.21.15.59:0/1202929353 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcbfc06d5a0 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.863+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcbfc0b3990 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.863+0000 7fcbf37fe640 1 -- 172.21.15.59:0/1202929353 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fcbfc0b42e0 con 0x7fcbfc0d23c0 2024-06-29T14:22:08.861 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:08.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcbdc0762d0 msgr2=0x7fcbdc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcbdc0762d0 0x7fcbdc0787e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcbec035000 tx=0x7fcbec038040 comp rx=0 tx=0).stop 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 msgr2=0x7fcbfc1f6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 0x7fcbfc1f6090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400c400 tx=0x7fcbe400c8c0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 shutdown_connections 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcbdc0762d0 0x7fcbdc0787e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbfc15efe0 0x7fcbfc1eaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbfc0d23c0 0x7fcbfc1f6090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 --2- 172.21.15.59:0/1202929353 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbfc0b3ed0 0x7fcbfc151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 >> 172.21.15.59:0/1202929353 conn(0x7fcbfc0d3ee0 msgr2=0x7fcbfc0d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 shutdown_connections 2024-06-29T14:22:08.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.865+0000 7fcc01562640 1 -- 172.21.15.59:0/1202929353 wait complete. 2024-06-29T14:22:08.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:22:08.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.996+0000 7f2aede09640 1 -- 172.21.15.59:0/3542166584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae808c190 msgr2=0x7f2ae80cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.996+0000 7f2aede09640 1 --2- 172.21.15.59:0/3542166584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae808c190 0x7f2ae80cf080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8005fd0 tx=0x7f2ad802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.996+0000 7f2aede09640 1 -- 172.21.15.59:0/3542166584 shutdown_connections 2024-06-29T14:22:08.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.996+0000 7f2aede09640 1 --2- 172.21.15.59:0/3542166584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ae80cf5c0 0x7f2ae80d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.996+0000 7f2aede09640 1 --2- 172.21.15.59:0/3542166584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae808c190 0x7f2ae80cf080 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.996+0000 7f2aede09640 1 -- 172.21.15.59:0/3542166584 >> 172.21.15.59:0/3542166584 conn(0x7f2ae80b5610 msgr2=0x7f2ae80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:08.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.996+0000 7f2aede09640 1 -- 172.21.15.59:0/3542166584 shutdown_connections 2024-06-29T14:22:08.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.996+0000 7f2aede09640 1 -- 172.21.15.59:0/3542166584 wait complete. 2024-06-29T14:22:08.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.998+0000 7f2aede09640 1 Processor -- start 2024-06-29T14:22:08.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.998+0000 7f2aede09640 1 -- start start 2024-06-29T14:22:08.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2aede09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ae808c190 0x7f2ae80aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2aede09640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae80ce950 0x7f2ae815f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2aede09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ae80cf5c0 0x7f2ae8150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2aede09640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae80b7fa0 con 0x7f2ae80ce950 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2aede09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae80b7e20 con 0x7f2ae808c190 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2aede09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae80b8120 con 0x7f2ae80cf5c0 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2ae6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae80ce950 0x7f2ae815f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2ae6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae80ce950 0x7f2ae815f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40618/0 (socket says 172.21.15.59:40618) 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2ae6ffd640 1 -- 172.21.15.59:0/3613276481 learned_addr learned my addr 172.21.15.59:0/3613276481 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2ae77fe640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ae808c190 0x7f2ae80aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:08.999+0000 7f2ae7fff640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ae80cf5c0 0x7f2ae8150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2ae6ffd640 1 -- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ae80cf5c0 msgr2=0x7f2ae8150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2ae6ffd640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ae80cf5c0 0x7f2ae8150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2ae6ffd640 1 -- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ae808c190 msgr2=0x7f2ae80aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2ae6ffd640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ae808c190 0x7f2ae80aafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:08.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2ae6ffd640 1 -- 172.21.15.59:0/3613276481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae8160410 con 0x7f2ae80ce950 2024-06-29T14:22:09.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2ae6ffd640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae80ce950 0x7f2ae815f7d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00e7c0 tx=0x7f2adc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:09.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2acffff640 1 -- 172.21.15.59:0/3613276481 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc00ccc0 con 0x7f2ae80ce950 2024-06-29T14:22:09.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2acffff640 1 -- 172.21.15.59:0/3613276481 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2adc010430 con 0x7f2ae80ce950 2024-06-29T14:22:09.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2acffff640 1 -- 172.21.15.59:0/3613276481 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc010860 con 0x7f2ae80ce950 2024-06-29T14:22:09.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae81606b0 con 0x7f2ae80ce950 2024-06-29T14:22:09.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.000+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae8151440 con 0x7f2ae80ce950 2024-06-29T14:22:09.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.001+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae80cf080 con 0x7f2ae80ce950 2024-06-29T14:22:09.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.002+0000 7f2acffff640 1 -- 172.21.15.59:0/3613276481 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2adc010a00 con 0x7f2ae80ce950 2024-06-29T14:22:09.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.002+0000 7f2acffff640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2ad40762d0 0x7f2ad40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.003+0000 7f2acffff640 1 -- 172.21.15.59:0/3613276481 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2adc0983d0 con 0x7f2ae80ce950 2024-06-29T14:22:09.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.003+0000 7f2ae77fe640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2ad40762d0 0x7f2ad40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.003+0000 7f2ae77fe640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2ad40762d0 0x7f2ad40787e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8033000 tx=0x7f2ad8004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:09.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.007+0000 7f2acffff640 1 -- 172.21.15.59:0/3613276481 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2adc0613a0 con 0x7f2ae80ce950 2024-06-29T14:22:09.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.210+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2ae815fe60 con 0x7f2ae80ce950 2024-06-29T14:22:09.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.211+0000 7f2acffff640 1 -- 172.21.15.59:0/3613276481 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f2adc0656a0 con 0x7f2ae80ce950 2024-06-29T14:22:09.209 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-06-29T14:22:09.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2ad40762d0 msgr2=0x7f2ad40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2ad40762d0 0x7f2ad40787e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8033000 tx=0x7f2ad8004730 comp rx=0 tx=0).stop 2024-06-29T14:22:09.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae80ce950 msgr2=0x7f2ae815f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae80ce950 0x7f2ae815f7d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00e7c0 tx=0x7f2adc00ec80 comp rx=0 tx=0).stop 2024-06-29T14:22:09.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 shutdown_connections 2024-06-29T14:22:09.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2ad40762d0 0x7f2ad40787e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ae80cf5c0 0x7f2ae8150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ae80ce950 0x7f2ae815f7d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 --2- 172.21.15.59:0/3613276481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ae808c190 0x7f2ae80aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 >> 172.21.15.59:0/3613276481 conn(0x7f2ae80b5610 msgr2=0x7f2ae80d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:09.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 shutdown_connections 2024-06-29T14:22:09.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.213+0000 7f2aede09640 1 -- 172.21.15.59:0/3613276481 wait complete. 2024-06-29T14:22:09.222 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-06-29T14:22:09.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:09.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-06-29T14:22:09.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi059.stderr:ceph-fuse[45045]: fuse finished with error 0 and tester_r 0 2024-06-29T14:22:09.248 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:22:09.332 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:22:09.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:09.333 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-06-29T14:22:09.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.604+0000 7fb4584cc640 1 -- 172.21.15.59:0/3937596801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 msgr2=0x7fb4500b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.604+0000 7fb4584cc640 1 --2- 172.21.15.59:0/3937596801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 0x7fb4500b3990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb440009830 tx=0x7fb44002a500 comp rx=0 tx=0).stop 2024-06-29T14:22:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.604+0000 7fb4584cc640 1 -- 172.21.15.59:0/3937596801 shutdown_connections 2024-06-29T14:22:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.604+0000 7fb4584cc640 1 --2- 172.21.15.59:0/3937596801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 0x7fb4500b3990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.604+0000 7fb4584cc640 1 --2- 172.21.15.59:0/3937596801 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb45015e290 0x7fb4500ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.604+0000 7fb4584cc640 1 -- 172.21.15.59:0/3937596801 >> 172.21.15.59:0/3937596801 conn(0x7fb4500d3ee0 msgr2=0x7fb4500d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.604+0000 7fb4584cc640 1 -- 172.21.15.59:0/3937596801 shutdown_connections 2024-06-29T14:22:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.604+0000 7fb4584cc640 1 -- 172.21.15.59:0/3937596801 wait complete. 2024-06-29T14:22:09.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.606+0000 7fb4584cc640 1 Processor -- start 2024-06-29T14:22:09.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.606+0000 7fb4584cc640 1 -- start start 2024-06-29T14:22:09.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.607+0000 7fb4584cc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4500b3ed0 0x7fb450154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.607+0000 7fb4584cc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 0x7fb45014e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.607+0000 7fb4584cc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45015e290 0x7fb4501f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.607+0000 7fb4584cc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb4500d1f20 con 0x7fb4500ce650 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.607+0000 7fb4584cc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4500d1da0 con 0x7fb45015e290 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.607+0000 7fb4584cc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb4500d20a0 con 0x7fb4500b3ed0 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 0x7fb45014e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 0x7fb45014e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40636/0 (socket says 172.21.15.59:40636) 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 -- 172.21.15.59:0/4257159515 learned_addr learned my addr 172.21.15.59:0/4257159515 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb456268640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4500b3ed0 0x7fb450154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb456a69640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45015e290 0x7fb4501f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 -- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4500b3ed0 msgr2=0x7fb450154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4500b3ed0 0x7fb450154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 -- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45015e290 msgr2=0x7fb4501f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45015e290 0x7fb4501f5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 -- 172.21.15.59:0/4257159515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb450155620 con 0x7fb4500ce650 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb456268640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4500b3ed0 0x7fb450154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb456a69640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45015e290 0x7fb4501f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb455a67640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 0x7fb45014e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb440009950 tx=0x7fb440033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb446ffd640 1 -- 172.21.15.59:0/4257159515 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440031d60 con 0x7fb4500ce650 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb446ffd640 1 -- 172.21.15.59:0/4257159515 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb440033e10 con 0x7fb4500ce650 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb446ffd640 1 -- 172.21.15.59:0/4257159515 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440005260 con 0x7fb4500ce650 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb450155060 con 0x7fb4500ce650 2024-06-29T14:22:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.608+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45014e950 con 0x7fb4500ce650 2024-06-29T14:22:09.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.609+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4500b3990 con 0x7fb4500ce650 2024-06-29T14:22:09.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.610+0000 7fb446ffd640 1 -- 172.21.15.59:0/4257159515 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb440005400 con 0x7fb4500ce650 2024-06-29T14:22:09.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.611+0000 7fb446ffd640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb434076300 0x7fb434078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.611+0000 7fb446ffd640 1 -- 172.21.15.59:0/4257159515 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb4400b8d60 con 0x7fb4500ce650 2024-06-29T14:22:09.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.611+0000 7fb456268640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb434076300 0x7fb434078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.611+0000 7fb456268640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb434076300 0x7fb434078810 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00ba60 tx=0x7fb44c005cf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:09.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.615+0000 7fb446ffd640 1 -- 172.21.15.59:0/4257159515 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb44003f030 con 0x7fb4500ce650 2024-06-29T14:22:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.808+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb45015c4f0 con 0x7fb4500ce650 2024-06-29T14:22:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.809+0000 7fb446ffd640 1 -- 172.21.15.59:0/4257159515 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7fb440082390 con 0x7fb4500ce650 2024-06-29T14:22:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:09.808 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:09.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb434076300 msgr2=0x7fb434078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb434076300 0x7fb434078810 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00ba60 tx=0x7fb44c005cf0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 msgr2=0x7fb45014e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 0x7fb45014e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb440009950 tx=0x7fb440033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 shutdown_connections 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb434076300 0x7fb434078810 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45015e290 0x7fb4501f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4500ce650 0x7fb45014e090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 --2- 172.21.15.59:0/4257159515 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4500b3ed0 0x7fb450154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 >> 172.21.15.59:0/4257159515 conn(0x7fb4500d3ee0 msgr2=0x7fb4500b9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 shutdown_connections 2024-06-29T14:22:09.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.811+0000 7fb4584cc640 1 -- 172.21.15.59:0/4257159515 wait complete. 2024-06-29T14:22:09.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:09.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.942+0000 7fdf8a512640 1 -- 172.21.15.59:0/3295600819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 msgr2=0x7fdf6c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.942+0000 7fdf8a512640 1 --2- 172.21.15.59:0/3295600819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 0x7fdf6c004c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdf840b4df0 tx=0x7fdf6c005830 comp rx=0 tx=0).stop 2024-06-29T14:22:09.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.942+0000 7fdf88aaf640 1 -- 172.21.15.59:0/3295600819 reap_dead start 2024-06-29T14:22:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.942+0000 7fdf8a512640 1 -- 172.21.15.59:0/3295600819 shutdown_connections 2024-06-29T14:22:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.942+0000 7fdf8a512640 1 -- 172.21.15.59:0/3295600819 >> 172.21.15.59:0/3295600819 conn(0x7fdf840d3ee0 msgr2=0x7fdf840d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.942+0000 7fdf8a512640 1 -- 172.21.15.59:0/3295600819 shutdown_connections 2024-06-29T14:22:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.942+0000 7fdf8a512640 1 -- 172.21.15.59:0/3295600819 wait complete. 2024-06-29T14:22:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.944+0000 7fdf8a512640 1 Processor -- start 2024-06-29T14:22:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.944+0000 7fdf8a512640 1 -- start start 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf8a512640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 0x7fdf84150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf8a512640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf840b1620 0x7fdf840b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf8a512640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf840b2e40 0x7fdf840aa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf8a512640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdf840b3b10 con 0x7fdf6c004800 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf8a512640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf840b3990 con 0x7fdf840b2e40 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf8a512640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdf840b3c90 con 0x7fdf840b1620 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf83fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 0x7fdf84150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf83fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 0x7fdf84150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40662/0 (socket says 172.21.15.59:40662) 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.945+0000 7fdf83fff640 1 -- 172.21.15.59:0/1059076667 learned_addr learned my addr 172.21.15.59:0/1059076667 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf837fe640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf840b1620 0x7fdf840b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf88aaf640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf840b2e40 0x7fdf840aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf83fff640 1 -- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf840b1620 msgr2=0x7fdf840b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf83fff640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf840b1620 0x7fdf840b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf83fff640 1 -- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf840b2e40 msgr2=0x7fdf840aa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf83fff640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf840b2e40 0x7fdf840aa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf83fff640 1 -- 172.21.15.59:0/1059076667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf840b3250 con 0x7fdf6c004800 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf837fe640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf840b1620 0x7fdf840b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf88aaf640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf840b2e40 0x7fdf840aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf83fff640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 0x7fdf84150fa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdf74011ae0 tx=0x7fdf74009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf80ff9640 1 -- 172.21.15.59:0/1059076667 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7401e070 con 0x7fdf6c004800 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf80ff9640 1 -- 172.21.15.59:0/1059076667 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf7400fc70 con 0x7fdf6c004800 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf840b20a0 con 0x7fdf6c004800 2024-06-29T14:22:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf80ff9640 1 -- 172.21.15.59:0/1059076667 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74019410 con 0x7fdf6c004800 2024-06-29T14:22:09.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.946+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf840ac590 con 0x7fdf6c004800 2024-06-29T14:22:09.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.947+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf840b25c0 con 0x7fdf6c004800 2024-06-29T14:22:09.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.948+0000 7fdf80ff9640 1 -- 172.21.15.59:0/1059076667 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdf74021070 con 0x7fdf6c004800 2024-06-29T14:22:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.948+0000 7fdf80ff9640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf700762d0 0x7fdf700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.949+0000 7fdf80ff9640 1 -- 172.21.15.59:0/1059076667 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fdf74098360 con 0x7fdf6c004800 2024-06-29T14:22:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.949+0000 7fdf837fe640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf700762d0 0x7fdf700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.949+0000 7fdf837fe640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf700762d0 0x7fdf700787e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdf78035000 tx=0x7fdf7802d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:09.953+0000 7fdf80ff9640 1 -- 172.21.15.59:0/1059076667 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf840b3250 con 0x7fdf6c004800 2024-06-29T14:22:10.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.143+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdf840b9290 con 0x7fdf6c004800 2024-06-29T14:22:10.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.144+0000 7fdf80ff9640 1 -- 172.21.15.59:0/1059076667 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7fdf74061210 con 0x7fdf6c004800 2024-06-29T14:22:10.142 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:10.142 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:10.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf700762d0 msgr2=0x7fdf700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:10.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf700762d0 0x7fdf700787e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdf78035000 tx=0x7fdf7802d040 comp rx=0 tx=0).stop 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 msgr2=0x7fdf84150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 0x7fdf84150fa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdf74011ae0 tx=0x7fdf74009eb0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 shutdown_connections 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf700762d0 0x7fdf700787e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf840b2e40 0x7fdf840aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf840b1620 0x7fdf840b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 --2- 172.21.15.59:0/1059076667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c004800 0x7fdf84150fa0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 >> 172.21.15.59:0/1059076667 conn(0x7fdf840d3ee0 msgr2=0x7fdf8415c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 shutdown_connections 2024-06-29T14:22:10.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.146+0000 7fdf8a512640 1 -- 172.21.15.59:0/1059076667 wait complete. 2024-06-29T14:22:10.156 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:22:10.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.280+0000 7f886a34a640 1 -- 172.21.15.59:0/3245556077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88640b3990 msgr2=0x7f88640b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:10.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.280+0000 7f886a34a640 1 --2- 172.21.15.59:0/3245556077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88640b3990 0x7f88640b3da0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8858009830 tx=0x7f885802a510 comp rx=0 tx=0).stop 2024-06-29T14:22:10.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.281+0000 7f886a34a640 1 -- 172.21.15.59:0/3245556077 shutdown_connections 2024-06-29T14:22:10.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.281+0000 7f886a34a640 1 --2- 172.21.15.59:0/3245556077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88640b73f0 0x7f88640b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.281+0000 7f886a34a640 1 --2- 172.21.15.59:0/3245556077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88640b3990 0x7f88640b3da0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.281+0000 7f886a34a640 1 -- 172.21.15.59:0/3245556077 >> 172.21.15.59:0/3245556077 conn(0x7f88640d3ee0 msgr2=0x7f88640d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.281+0000 7f886a34a640 1 -- 172.21.15.59:0/3245556077 shutdown_connections 2024-06-29T14:22:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.281+0000 7f886a34a640 1 -- 172.21.15.59:0/3245556077 wait complete. 2024-06-29T14:22:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.282+0000 7f886a34a640 1 Processor -- start 2024-06-29T14:22:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.283+0000 7f886a34a640 1 -- start start 2024-06-29T14:22:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f886a34a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88640b3990 0x7f88641f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f886a34a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88640b73f0 0x7f88640b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f886a34a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f886415e290 0x7f88640d0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f886a34a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f88640b7d40 con 0x7f88640b73f0 2024-06-29T14:22:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f886a34a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88640cea30 con 0x7f886415e290 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f886a34a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f88640b7ec0 con 0x7f88640b3990 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88640b73f0 0x7f88640b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88640b73f0 0x7f88640b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40666/0 (socket says 172.21.15.59:40666) 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88637fe640 1 -- 172.21.15.59:0/2092499088 learned_addr learned my addr 172.21.15.59:0/2092499088 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f8863fff640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88640b3990 0x7f88641f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88688e7640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f886415e290 0x7f88640d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88637fe640 1 -- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88640b3990 msgr2=0x7f88641f5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88637fe640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88640b3990 0x7f88641f5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88637fe640 1 -- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f886415e290 msgr2=0x7f88640d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88637fe640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f886415e290 0x7f88640d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.284+0000 7f88637fe640 1 -- 172.21.15.59:0/2092499088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88640aaf10 con 0x7f88640b73f0 2024-06-29T14:22:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.285+0000 7f88637fe640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88640b73f0 0x7f88640b1d70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f885802ab00 tx=0x7f8858033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.285+0000 7f8860ff9640 1 -- 172.21.15.59:0/2092499088 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858031d60 con 0x7f88640b73f0 2024-06-29T14:22:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.285+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8864160d60 con 0x7f88640b73f0 2024-06-29T14:22:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.285+0000 7f8860ff9640 1 -- 172.21.15.59:0/2092499088 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8858033e10 con 0x7f88640b73f0 2024-06-29T14:22:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.285+0000 7f8860ff9640 1 -- 172.21.15.59:0/2092499088 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858005260 con 0x7f88640b73f0 2024-06-29T14:22:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.285+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8864161320 con 0x7f88640b73f0 2024-06-29T14:22:10.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.286+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88640b8420 con 0x7f88640b73f0 2024-06-29T14:22:10.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.287+0000 7f8860ff9640 1 -- 172.21.15.59:0/2092499088 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8858005400 con 0x7f88640b73f0 2024-06-29T14:22:10.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.287+0000 7f8860ff9640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88500762d0 0x7f88500787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:10.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.287+0000 7f8860ff9640 1 -- 172.21.15.59:0/2092499088 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f88580b8c00 con 0x7f88640b73f0 2024-06-29T14:22:10.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.287+0000 7f8863fff640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88500762d0 0x7f88500787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:10.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.288+0000 7f8863fff640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88500762d0 0x7f88500787e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f88540015e0 tx=0x7f8854009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.291+0000 7f8860ff9640 1 -- 172.21.15.59:0/2092499088 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f88580bf080 con 0x7f88640b73f0 2024-06-29T14:22:10.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:10.520+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f88641569b0 con 0x7f88640b73f0 2024-06-29T14:22:11.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.518+0000 7f8860ff9640 1 -- 172.21.15.59:0/2092499088 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f885803f030 con 0x7f88640b73f0 2024-06-29T14:22:11.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88500762d0 msgr2=0x7f88500787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:11.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88500762d0 0x7f88500787e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f88540015e0 tx=0x7f8854009040 comp rx=0 tx=0).stop 2024-06-29T14:22:11.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88640b73f0 msgr2=0x7f88640b1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:11.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88640b73f0 0x7f88640b1d70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f885802ab00 tx=0x7f8858033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:11.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 shutdown_connections 2024-06-29T14:22:11.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88500762d0 0x7f88500787e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:11.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f886415e290 0x7f88640d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:11.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88640b73f0 0x7f88640b1d70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:11.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 --2- 172.21.15.59:0/2092499088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88640b3990 0x7f88641f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:11.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 >> 172.21.15.59:0/2092499088 conn(0x7f88640d3ee0 msgr2=0x7f88640ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:11.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 shutdown_connections 2024-06-29T14:22:11.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.521+0000 7f886a34a640 1 -- 172.21.15.59:0/2092499088 wait complete. 2024-06-29T14:22:11.533 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-06-29T14:22:11.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:22:11.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.664+0000 7f50a7026640 1 -- 172.21.15.59:0/2441188826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 msgr2=0x7f50a00b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.664+0000 7f50a7026640 1 --2- 172.21.15.59:0/2441188826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 0x7f50a00b38e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5098005fd0 tx=0x7f509802aa10 comp rx=0 tx=0).stop 2024-06-29T14:22:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.664+0000 7f50a7026640 1 -- 172.21.15.59:0/2441188826 shutdown_connections 2024-06-29T14:22:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.664+0000 7f50a7026640 1 --2- 172.21.15.59:0/2441188826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 0x7f50a00b38e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.664+0000 7f50a7026640 1 --2- 172.21.15.59:0/2441188826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a0162190 0x7f50a00ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.664+0000 7f50a7026640 1 -- 172.21.15.59:0/2441188826 >> 172.21.15.59:0/2441188826 conn(0x7f50a00d45b0 msgr2=0x7f50a00d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.664+0000 7f50a7026640 1 -- 172.21.15.59:0/2441188826 shutdown_connections 2024-06-29T14:22:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.664+0000 7f50a7026640 1 -- 172.21.15.59:0/2441188826 wait complete. 2024-06-29T14:22:11.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.666+0000 7f50a7026640 1 Processor -- start 2024-06-29T14:22:11.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.666+0000 7f50a7026640 1 -- start start 2024-06-29T14:22:11.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.667+0000 7f50a7026640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 0x7f50a015e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.667+0000 7f50a7026640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a0162770 0x7f50a01f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a7026640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a01f8a00 0x7f50a015aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a7026640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f50a00b43a0 con 0x7f50a015e0a0 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a7026640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50a00b4220 con 0x7f50a0162770 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a7026640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f50a00b4520 con 0x7f50a01f8a00 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 0x7f50a015e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 0x7f50a015e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40686/0 (socket says 172.21.15.59:40686) 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 -- 172.21.15.59:0/1879076054 learned_addr learned my addr 172.21.15.59:0/1879076054 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a55c3640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a01f8a00 0x7f50a015aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f509ffff640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a0162770 0x7f50a01f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 -- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a01f8a00 msgr2=0x7f50a015aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a01f8a00 0x7f50a015aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:11.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 -- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a0162770 msgr2=0x7f50a01f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a0162770 0x7f50a01f84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 -- 172.21.15.59:0/1879076054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a015ef60 con 0x7f50a015e0a0 2024-06-29T14:22:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a4dc2640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 0x7f50a015e660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f509000c370 tx=0x7f509000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f509d7fa640 1 -- 172.21.15.59:0/1879076054 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50900052d0 con 0x7f50a015e0a0 2024-06-29T14:22:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f509d7fa640 1 -- 172.21.15.59:0/1879076054 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5090007020 con 0x7f50a015e0a0 2024-06-29T14:22:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a015f240 con 0x7f50a015e0a0 2024-06-29T14:22:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.668+0000 7f509d7fa640 1 -- 172.21.15.59:0/1879076054 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090007470 con 0x7f50a015e0a0 2024-06-29T14:22:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.669+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a015f800 con 0x7f50a015e0a0 2024-06-29T14:22:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.670+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a00b49d0 con 0x7f50a015e0a0 2024-06-29T14:22:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.670+0000 7f509d7fa640 1 -- 172.21.15.59:0/1879076054 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f50900067d0 con 0x7f50a015e0a0 2024-06-29T14:22:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.671+0000 7f509d7fa640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f50800762d0 0x7f50800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.671+0000 7f509d7fa640 1 -- 172.21.15.59:0/1879076054 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f50900970f0 con 0x7f50a015e0a0 2024-06-29T14:22:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.671+0000 7f509ffff640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f50800762d0 0x7f50800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:11.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.672+0000 7f509ffff640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f50800762d0 0x7f50800787e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f50a00ccf80 tx=0x7f508800b2e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:11.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.676+0000 7f509d7fa640 1 -- 172.21.15.59:0/1879076054 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f509005ff10 con 0x7f50a015e0a0 2024-06-29T14:22:11.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:11.908+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f50a00b8f10 con 0x7f50a015e0a0 2024-06-29T14:22:12.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.523+0000 7f509d7fa640 1 -- 172.21.15.59:0/1879076054 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f5090064210 con 0x7f50a015e0a0 2024-06-29T14:22:12.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f50800762d0 msgr2=0x7f50800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:12.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f50800762d0 0x7f50800787e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f50a00ccf80 tx=0x7f508800b2e0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 msgr2=0x7f50a015e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:12.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 0x7f50a015e660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f509000c370 tx=0x7f509000c830 comp rx=0 tx=0).stop 2024-06-29T14:22:12.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 shutdown_connections 2024-06-29T14:22:12.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f50800762d0 0x7f50800787e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a01f8a00 0x7f50a015aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a0162770 0x7f50a01f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 --2- 172.21.15.59:0/1879076054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a015e0a0 0x7f50a015e660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 >> 172.21.15.59:0/1879076054 conn(0x7f50a00d45b0 msgr2=0x7f50a006db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:12.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 shutdown_connections 2024-06-29T14:22:12.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.526+0000 7f50a7026640 1 -- 172.21.15.59:0/1879076054 wait complete. 2024-06-29T14:22:12.538 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:22:12.538 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:12.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.666+0000 7f7a4083f640 1 -- 172.21.15.59:0/4149760976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c0ce950 msgr2=0x7f7a3c1622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:12.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.666+0000 7f7a4083f640 1 --2- 172.21.15.59:0/4149760976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c0ce950 0x7f7a3c1622c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009860 tx=0x7f7a3002a540 comp rx=0 tx=0).stop 2024-06-29T14:22:12.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.667+0000 7f7a4083f640 1 -- 172.21.15.59:0/4149760976 shutdown_connections 2024-06-29T14:22:12.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.667+0000 7f7a4083f640 1 --2- 172.21.15.59:0/4149760976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a3c162800 0x7f7a3c162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.667+0000 7f7a4083f640 1 --2- 172.21.15.59:0/4149760976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c0ce950 0x7f7a3c1622c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.667+0000 7f7a4083f640 1 -- 172.21.15.59:0/4149760976 >> 172.21.15.59:0/4149760976 conn(0x7f7a3c0b5610 msgr2=0x7f7a3c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:12.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.667+0000 7f7a4083f640 1 -- 172.21.15.59:0/4149760976 shutdown_connections 2024-06-29T14:22:12.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.667+0000 7f7a4083f640 1 -- 172.21.15.59:0/4149760976 wait complete. 2024-06-29T14:22:12.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.668+0000 7f7a4083f640 1 Processor -- start 2024-06-29T14:22:12.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.669+0000 7f7a4083f640 1 -- start start 2024-06-29T14:22:12.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a4083f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a3c08c190 0x7f7a3c0b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:12.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a4083f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a3c0ce950 0x7f7a3c15bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:12.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a4083f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c162800 0x7f7a3c161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:12.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a4083f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c0b7f60 con 0x7f7a3c162800 2024-06-29T14:22:12.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a4083f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c0b7de0 con 0x7f7a3c08c190 2024-06-29T14:22:12.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a4083f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3c0b80e0 con 0x7f7a3c0ce950 2024-06-29T14:22:12.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c162800 0x7f7a3c161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:12.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c162800 0x7f7a3c161200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40710/0 (socket says 172.21.15.59:40710) 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3ad9d640 1 -- 172.21.15.59:0/2749711637 learned_addr learned my addr 172.21.15.59:0/2749711637 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a39d9b640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a3c0ce950 0x7f7a3c15bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3a59c640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a3c08c190 0x7f7a3c0b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3ad9d640 1 -- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a3c0ce950 msgr2=0x7f7a3c15bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3ad9d640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a3c0ce950 0x7f7a3c15bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3ad9d640 1 -- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a3c08c190 msgr2=0x7f7a3c0b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3ad9d640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a3c08c190 0x7f7a3c0b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.670+0000 7f7a3ad9d640 1 -- 172.21.15.59:0/2749711637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3c0aa700 con 0x7f7a3c162800 2024-06-29T14:22:12.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.671+0000 7f7a3ad9d640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c162800 0x7f7a3c161200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400e9f0 tx=0x7f7a3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:12.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.671+0000 7f7a2affd640 1 -- 172.21.15.59:0/2749711637 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34004d20 con 0x7f7a3c162800 2024-06-29T14:22:12.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.671+0000 7f7a2affd640 1 -- 172.21.15.59:0/2749711637 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a3400cb20 con 0x7f7a3c162800 2024-06-29T14:22:12.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.671+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a3c0aa9e0 con 0x7f7a3c162800 2024-06-29T14:22:12.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.671+0000 7f7a2affd640 1 -- 172.21.15.59:0/2749711637 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34005020 con 0x7f7a3c162800 2024-06-29T14:22:12.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.671+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3c0aafa0 con 0x7f7a3c162800 2024-06-29T14:22:12.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.672+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a3c1622c0 con 0x7f7a3c162800 2024-06-29T14:22:12.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.673+0000 7f7a2affd640 1 -- 172.21.15.59:0/2749711637 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7a340051c0 con 0x7f7a3c162800 2024-06-29T14:22:12.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.673+0000 7f7a2affd640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7a1c076300 0x7f7a1c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:12.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.674+0000 7f7a2affd640 1 -- 172.21.15.59:0/2749711637 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f7a34098270 con 0x7f7a3c162800 2024-06-29T14:22:12.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.674+0000 7f7a3a59c640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7a1c076300 0x7f7a1c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:12.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.674+0000 7f7a3a59c640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7a1c076300 0x7f7a1c078810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c0046b0 tx=0x7f7a2c007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:12.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.678+0000 7f7a2affd640 1 -- 172.21.15.59:0/2749711637 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7a34016030 con 0x7f7a3c162800 2024-06-29T14:22:12.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.907+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a3c0b5c30 con 0x7f7a3c162800 2024-06-29T14:22:12.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.907+0000 7f7a2affd640 1 -- 172.21.15.59:0/2749711637 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5222 (secure 0 0 0) 0x7f7a340611b0 con 0x7f7a3c162800 2024-06-29T14:22:12.905 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:12.906 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":9,"btime":"2024-06-29T14:22:12:518854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:22:11.909450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:12.906 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 9 2024-06-29T14:22:12.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7a1c076300 msgr2=0x7f7a1c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:12.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7a1c076300 0x7f7a1c078810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c0046b0 tx=0x7f7a2c007eb0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c162800 msgr2=0x7f7a3c161200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:12.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c162800 0x7f7a3c161200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400e9f0 tx=0x7f7a3400eeb0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 shutdown_connections 2024-06-29T14:22:12.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7a1c076300 0x7f7a1c078810 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a3c162800 0x7f7a3c161200 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a3c0ce950 0x7f7a3c15bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 --2- 172.21.15.59:0/2749711637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a3c08c190 0x7f7a3c0b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:12.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 >> 172.21.15.59:0/2749711637 conn(0x7f7a3c0b5610 msgr2=0x7f7a3c06d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:12.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 shutdown_connections 2024-06-29T14:22:12.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:12.910+0000 7f7a4083f640 1 -- 172.21.15.59:0/2749711637 wait complete. 2024-06-29T14:22:12.923 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:13.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.048+0000 7fb8023ef640 1 -- 172.21.15.59:0/3235328284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e4003a70 msgr2=0x7fb7e4003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.048+0000 7fb8023ef640 1 --2- 172.21.15.59:0/3235328284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e4003a70 0x7fb7e4003ea0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0017c0 tx=0x7fb7e4005830 comp rx=0 tx=0).stop 2024-06-29T14:22:13.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.048+0000 7fb8023ef640 1 -- 172.21.15.59:0/3235328284 shutdown_connections 2024-06-29T14:22:13.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.048+0000 7fb8023ef640 1 --2- 172.21.15.59:0/3235328284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e4003a70 0x7fb7e4003ea0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.048+0000 7fb8023ef640 1 -- 172.21.15.59:0/3235328284 >> 172.21.15.59:0/3235328284 conn(0x7fb7fc0b06e0 msgr2=0x7fb7fc0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:13.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.048+0000 7fb8023ef640 1 -- 172.21.15.59:0/3235328284 shutdown_connections 2024-06-29T14:22:13.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.048+0000 7fb8023ef640 1 -- 172.21.15.59:0/3235328284 wait complete. 2024-06-29T14:22:13.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.049+0000 7fb8023ef640 1 Processor -- start 2024-06-29T14:22:13.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.050+0000 7fb8023ef640 1 -- start start 2024-06-29T14:22:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb8023ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e4003a70 0x7fb7fc1f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb8023ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7fc1f6250 0x7fb7fc1432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb8023ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7fc143800 0x7fb7fc1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb8023ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc06e740 con 0x7fb7fc1f6250 2024-06-29T14:22:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb8023ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc06e5c0 con 0x7fb7e4003a70 2024-06-29T14:22:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb8023ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb7fc0b1b00 con 0x7fb7fc143800 2024-06-29T14:22:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7fc1f6250 0x7fb7fc1432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7fc1f6250 0x7fb7fc1432c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51574/0 (socket says 172.21.15.59:51574) 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fb7fe640 1 -- 172.21.15.59:0/2703173788 learned_addr learned my addr 172.21.15.59:0/2703173788 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fbfff640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e4003a70 0x7fb7fc1f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb80098c640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7fc143800 0x7fb7fc1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fb7fe640 1 -- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7fc143800 msgr2=0x7fb7fc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fb7fe640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7fc143800 0x7fb7fc1566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fb7fe640 1 -- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e4003a70 msgr2=0x7fb7fc1f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fb7fe640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e4003a70 0x7fb7fc1f5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.051+0000 7fb7fb7fe640 1 -- 172.21.15.59:0/2703173788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc159a70 con 0x7fb7fc1f6250 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.052+0000 7fb7fb7fe640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7fc1f6250 0x7fb7fc1432c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0006480 tx=0x7fb7f00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.052+0000 7fb7f8ff9640 1 -- 172.21.15.59:0/2703173788 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0015070 con 0x7fb7fc1f6250 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.052+0000 7fb7f8ff9640 1 -- 172.21.15.59:0/2703173788 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7f0011880 con 0x7fb7fc1f6250 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.052+0000 7fb7f8ff9640 1 -- 172.21.15.59:0/2703173788 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0011cd0 con 0x7fb7fc1f6250 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.052+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7fc1558e0 con 0x7fb7fc1f6250 2024-06-29T14:22:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.052+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7fc155ea0 con 0x7fb7fc1f6250 2024-06-29T14:22:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.053+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7fc1597e0 con 0x7fb7fc1f6250 2024-06-29T14:22:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.054+0000 7fb7f8ff9640 1 -- 172.21.15.59:0/2703173788 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb7f0018070 con 0x7fb7fc1f6250 2024-06-29T14:22:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.054+0000 7fb7f8ff9640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7e80762d0 0x7fb7e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.054+0000 7fb7f8ff9640 1 -- 172.21.15.59:0/2703173788 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb7f0098650 con 0x7fb7fc1f6250 2024-06-29T14:22:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.054+0000 7fb7fbfff640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7e80762d0 0x7fb7e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.055+0000 7fb7fbfff640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7e80762d0 0x7fb7e80787e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec032000 tx=0x7fb7ec02c000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:13.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.059+0000 7fb7f8ff9640 1 -- 172.21.15.59:0/2703173788 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7f00615a0 con 0x7fb7fc1f6250 2024-06-29T14:22:13.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.251+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7fc0b2060 con 0x7fb7fc1f6250 2024-06-29T14:22:13.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.252+0000 7fb7f8ff9640 1 -- 172.21.15.59:0/2703173788 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7fb7f00658a0 con 0x7fb7fc1f6250 2024-06-29T14:22:13.251 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:13.251 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.254+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7e80762d0 msgr2=0x7fb7e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.254+0000 7fb8023ef640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7e80762d0 0x7fb7e80787e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec032000 tx=0x7fb7ec02c000 comp rx=0 tx=0).stop 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7fc1f6250 msgr2=0x7fb7fc1432c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7fc1f6250 0x7fb7fc1432c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0006480 tx=0x7fb7f00039c0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 shutdown_connections 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7e80762d0 0x7fb7e80787e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7fc143800 0x7fb7fc1566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7fc1f6250 0x7fb7fc1432c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 --2- 172.21.15.59:0/2703173788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e4003a70 0x7fb7fc1f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 >> 172.21.15.59:0/2703173788 conn(0x7fb7fc0b06e0 msgr2=0x7fb7fc0b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 shutdown_connections 2024-06-29T14:22:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.255+0000 7fb8023ef640 1 -- 172.21.15.59:0/2703173788 wait complete. 2024-06-29T14:22:13.264 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:22:13.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:22:13.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.385+0000 7efd29638640 1 -- 172.21.15.59:0/4146723406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 msgr2=0x7efd100040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.385+0000 7efd29638640 1 --2- 172.21.15.59:0/4146723406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 0x7efd100040f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efd1c006f70 tx=0x7efd10005830 comp rx=0 tx=0).stop 2024-06-29T14:22:13.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.385+0000 7efd29638640 1 -- 172.21.15.59:0/4146723406 shutdown_connections 2024-06-29T14:22:13.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.385+0000 7efd29638640 1 --2- 172.21.15.59:0/4146723406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 0x7efd100040f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.385+0000 7efd29638640 1 -- 172.21.15.59:0/4146723406 >> 172.21.15.59:0/4146723406 conn(0x7efd240b1ea0 msgr2=0x7efd240b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:13.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.385+0000 7efd29638640 1 -- 172.21.15.59:0/4146723406 shutdown_connections 2024-06-29T14:22:13.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.385+0000 7efd29638640 1 -- 172.21.15.59:0/4146723406 wait complete. 2024-06-29T14:22:13.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.387+0000 7efd29638640 1 Processor -- start 2024-06-29T14:22:13.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.387+0000 7efd29638640 1 -- start start 2024-06-29T14:22:13.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd29638640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 0x7efd2414d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd29638640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd2406c4f0 0x7efd2414c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd29638640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd2406df50 0x7efd2406e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd29638640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efd240acfa0 con 0x7efd10003ca0 2024-06-29T14:22:13.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd29638640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd240740d0 con 0x7efd2406df50 2024-06-29T14:22:13.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd29638640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efd240ad120 con 0x7efd2406c4f0 2024-06-29T14:22:13.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 0x7efd2414d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 0x7efd2414d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51586/0 (socket says 172.21.15.59:51586) 2024-06-29T14:22:13.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 -- 172.21.15.59:0/1954716384 learned_addr learned my addr 172.21.15.59:0/1954716384 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:13.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd227fc640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd2406c4f0 0x7efd2414c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd237fe640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd2406df50 0x7efd2406e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 -- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd2406c4f0 msgr2=0x7efd2414c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd2406c4f0 0x7efd2414c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 -- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd2406df50 msgr2=0x7efd2406e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd2406df50 0x7efd2406e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 -- 172.21.15.59:0/1954716384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2406ceb0 con 0x7efd10003ca0 2024-06-29T14:22:13.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd237fe640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd2406df50 0x7efd2406e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:13.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd22ffd640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 0x7efd2414d230 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efd1c031040 tx=0x7efd1c001c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:13.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd1b7fe640 1 -- 172.21.15.59:0/1954716384 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c032030 con 0x7efd10003ca0 2024-06-29T14:22:13.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd1b7fe640 1 -- 172.21.15.59:0/1954716384 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd1c0038e0 con 0x7efd10003ca0 2024-06-29T14:22:13.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.389+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2406d190 con 0x7efd10003ca0 2024-06-29T14:22:13.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.390+0000 7efd1b7fe640 1 -- 172.21.15.59:0/1954716384 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c003cf0 con 0x7efd10003ca0 2024-06-29T14:22:13.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.390+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2406d750 con 0x7efd10003ca0 2024-06-29T14:22:13.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.391+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd240ad680 con 0x7efd10003ca0 2024-06-29T14:22:13.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.391+0000 7efd1b7fe640 1 -- 172.21.15.59:0/1954716384 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efd1c035020 con 0x7efd10003ca0 2024-06-29T14:22:13.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.392+0000 7efd1b7fe640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efd040762d0 0x7efd040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.392+0000 7efd1b7fe640 1 -- 172.21.15.59:0/1954716384 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7efd1c0b7e90 con 0x7efd10003ca0 2024-06-29T14:22:13.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.392+0000 7efd227fc640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efd040762d0 0x7efd040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.392+0000 7efd227fc640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efd040762d0 0x7efd040787e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efd241f60a0 tx=0x7efd0c006b10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:13.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.396+0000 7efd1b7fe640 1 -- 172.21.15.59:0/1954716384 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd1c047030 con 0x7efd10003ca0 2024-06-29T14:22:13.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.584+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd240cc780 con 0x7efd040762d0 2024-06-29T14:22:13.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.591+0000 7efd1b7fe640 1 -- 172.21.15.59:0/1954716384 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7efd240cc780 con 0x7efd040762d0 2024-06-29T14:22:13.592 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:13.594 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efd040762d0 msgr2=0x7efd040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efd040762d0 0x7efd040787e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efd241f60a0 tx=0x7efd0c006b10 comp rx=0 tx=0).stop 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 msgr2=0x7efd2414d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 0x7efd2414d230 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efd1c031040 tx=0x7efd1c001c30 comp rx=0 tx=0).stop 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 shutdown_connections 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efd040762d0 0x7efd040787e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd2406df50 0x7efd2406e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd2406c4f0 0x7efd2414c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 --2- 172.21.15.59:0/1954716384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10003ca0 0x7efd2414d230 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 >> 172.21.15.59:0/1954716384 conn(0x7efd240b1ea0 msgr2=0x7efd240b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:13.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 shutdown_connections 2024-06-29T14:22:13.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.594+0000 7efd29638640 1 -- 172.21.15.59:0/1954716384 wait complete. 2024-06-29T14:22:13.603 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-06-29T14:22:11.734756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219504,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755216,"statfs":{"total":773094113280,"available":772869341184,"internally_reserved":0,"allocated":3416064,"data_stored":2049459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.431725"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640964+0000","last_change":"2024-06-29T14:21:54.260091+0000","last_active":"2024-06-29T14:22:00.640964+0000","last_peered":"2024-06-29T14:22:00.640964+0000","last_clean":"2024-06-29T14:22:00.640964+0000","last_became_active":"2024-06-29T14:21:54.259733+0000","last_became_peered":"2024-06-29T14:21:54.259733+0000","last_unstale":"2024-06-29T14:22:00.640964+0000","last_undegraded":"2024-06-29T14:22:00.640964+0000","last_fullsized":"2024-06-29T14:22:00.640964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:31.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305128+0000","last_change":"2024-06-29T14:21:55.286512+0000","last_active":"2024-06-29T14:22:00.305128+0000","last_peered":"2024-06-29T14:22:00.305128+0000","last_clean":"2024-06-29T14:22:00.305128+0000","last_became_active":"2024-06-29T14:21:55.286126+0000","last_became_peered":"2024-06-29T14:21:55.286126+0000","last_unstale":"2024-06-29T14:22:00.305128+0000","last_undegraded":"2024-06-29T14:22:00.305128+0000","last_fullsized":"2024-06-29T14:22:00.305128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304598+0000","last_change":"2024-06-29T14:21:59.338143+0000","last_active":"2024-06-29T14:22:00.304598+0000","last_peered":"2024-06-29T14:22:00.304598+0000","last_clean":"2024-06-29T14:22:00.304598+0000","last_became_active":"2024-06-29T14:21:59.337328+0000","last_became_peered":"2024-06-29T14:21:59.337328+0000","last_unstale":"2024-06-29T14:22:00.304598+0000","last_undegraded":"2024-06-29T14:22:00.304598+0000","last_fullsized":"2024-06-29T14:22:00.304598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:23.956178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305340+0000","last_change":"2024-06-29T14:21:54.257884+0000","last_active":"2024-06-29T14:22:00.305340+0000","last_peered":"2024-06-29T14:22:00.305340+0000","last_clean":"2024-06-29T14:22:00.305340+0000","last_became_active":"2024-06-29T14:21:54.257540+0000","last_became_peered":"2024-06-29T14:21:54.257540+0000","last_unstale":"2024-06-29T14:22:00.305340+0000","last_undegraded":"2024-06-29T14:22:00.305340+0000","last_fullsized":"2024-06-29T14:22:00.305340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:54.713087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640589+0000","last_change":"2024-06-29T14:21:55.285781+0000","last_active":"2024-06-29T14:22:00.640589+0000","last_peered":"2024-06-29T14:22:00.640589+0000","last_clean":"2024-06-29T14:22:00.640589+0000","last_became_active":"2024-06-29T14:21:55.285358+0000","last_became_peered":"2024-06-29T14:21:55.285358+0000","last_unstale":"2024-06-29T14:22:00.640589+0000","last_undegraded":"2024-06-29T14:22:00.640589+0000","last_fullsized":"2024-06-29T14:22:00.640589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306447+0000","last_change":"2024-06-29T14:21:59.318024+0000","last_active":"2024-06-29T14:22:00.306447+0000","last_peered":"2024-06-29T14:22:00.306447+0000","last_clean":"2024-06-29T14:22:00.306447+0000","last_became_active":"2024-06-29T14:21:59.317356+0000","last_became_peered":"2024-06-29T14:21:59.317356+0000","last_unstale":"2024-06-29T14:22:00.306447+0000","last_undegraded":"2024-06-29T14:22:00.306447+0000","last_fullsized":"2024-06-29T14:22:00.306447+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:00.209167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640906+0000","last_change":"2024-06-29T14:21:54.258802+0000","last_active":"2024-06-29T14:22:00.640906+0000","last_peered":"2024-06-29T14:22:00.640906+0000","last_clean":"2024-06-29T14:22:00.640906+0000","last_became_active":"2024-06-29T14:21:54.258184+0000","last_became_peered":"2024-06-29T14:21:54.258184+0000","last_unstale":"2024-06-29T14:22:00.640906+0000","last_undegraded":"2024-06-29T14:22:00.640906+0000","last_fullsized":"2024-06-29T14:22:00.640906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:07:56.943579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640388+0000","last_change":"2024-06-29T14:21:55.296893+0000","last_active":"2024-06-29T14:22:00.640388+0000","last_peered":"2024-06-29T14:22:00.640388+0000","last_clean":"2024-06-29T14:22:00.640388+0000","last_became_active":"2024-06-29T14:21:55.296552+0000","last_became_peered":"2024-06-29T14:21:55.296552+0000","last_unstale":"2024-06-29T14:22:00.640388+0000","last_undegraded":"2024-06-29T14:22:00.640388+0000","last_fullsized":"2024-06-29T14:22:00.640388+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:57.029702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307221+0000","last_change":"2024-06-29T14:21:59.319287+0000","last_active":"2024-06-29T14:22:00.307221+0000","last_peered":"2024-06-29T14:22:00.307221+0000","last_clean":"2024-06-29T14:22:00.307221+0000","last_became_active":"2024-06-29T14:21:59.318807+0000","last_became_peered":"2024-06-29T14:21:59.318807+0000","last_unstale":"2024-06-29T14:22:00.307221+0000","last_undegraded":"2024-06-29T14:22:00.307221+0000","last_fullsized":"2024-06-29T14:22:00.307221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:20.883221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640313+0000","last_change":"2024-06-29T14:21:54.264911+0000","last_active":"2024-06-29T14:22:00.640313+0000","last_peered":"2024-06-29T14:22:00.640313+0000","last_clean":"2024-06-29T14:22:00.640313+0000","last_became_active":"2024-06-29T14:21:54.264532+0000","last_became_peered":"2024-06-29T14:21:54.264532+0000","last_unstale":"2024-06-29T14:22:00.640313+0000","last_undegraded":"2024-06-29T14:22:00.640313+0000","last_fullsized":"2024-06-29T14:22:00.640313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:10.799778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640442+0000","last_change":"2024-06-29T14:21:55.295115+0000","last_active":"2024-06-29T14:22:00.640442+0000","last_peered":"2024-06-29T14:22:00.640442+0000","last_clean":"2024-06-29T14:22:00.640442+0000","last_became_active":"2024-06-29T14:21:55.294772+0000","last_became_peered":"2024-06-29T14:21:55.294772+0000","last_unstale":"2024-06-29T14:22:00.640442+0000","last_undegraded":"2024-06-29T14:22:00.640442+0000","last_fullsized":"2024-06-29T14:22:00.640442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:17.149243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307463+0000","last_change":"2024-06-29T14:21:59.312924+0000","last_active":"2024-06-29T14:22:00.307463+0000","last_peered":"2024-06-29T14:22:00.307463+0000","last_clean":"2024-06-29T14:22:00.307463+0000","last_became_active":"2024-06-29T14:21:59.312393+0000","last_became_peered":"2024-06-29T14:21:59.312393+0000","last_unstale":"2024-06-29T14:22:00.307463+0000","last_undegraded":"2024-06-29T14:22:00.307463+0000","last_fullsized":"2024-06-29T14:22:00.307463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:32.630422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305149+0000","last_change":"2024-06-29T14:21:55.294165+0000","last_active":"2024-06-29T14:22:00.305149+0000","last_peered":"2024-06-29T14:22:00.305149+0000","last_clean":"2024-06-29T14:22:00.305149+0000","last_became_active":"2024-06-29T14:21:55.293388+0000","last_became_peered":"2024-06-29T14:21:55.293388+0000","last_unstale":"2024-06-29T14:22:00.305149+0000","last_undegraded":"2024-06-29T14:22:00.305149+0000","last_fullsized":"2024-06-29T14:22:00.305149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:50:36.883008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641127+0000","last_change":"2024-06-29T14:21:54.256784+0000","last_active":"2024-06-29T14:22:00.641127+0000","last_peered":"2024-06-29T14:22:00.641127+0000","last_clean":"2024-06-29T14:22:00.641127+0000","last_became_active":"2024-06-29T14:21:54.256434+0000","last_became_peered":"2024-06-29T14:21:54.256434+0000","last_unstale":"2024-06-29T14:22:00.641127+0000","last_undegraded":"2024-06-29T14:22:00.641127+0000","last_fullsized":"2024-06-29T14:22:00.641127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:06.963975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306723+0000","last_change":"2024-06-29T14:21:59.317621+0000","last_active":"2024-06-29T14:22:00.306723+0000","last_peered":"2024-06-29T14:22:00.306723+0000","last_clean":"2024-06-29T14:22:00.306723+0000","last_became_active":"2024-06-29T14:21:59.316736+0000","last_became_peered":"2024-06-29T14:21:59.316736+0000","last_unstale":"2024-06-29T14:22:00.306723+0000","last_undegraded":"2024-06-29T14:22:00.306723+0000","last_fullsized":"2024-06-29T14:22:00.306723+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:43.083466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304951+0000","last_change":"2024-06-29T14:21:55.296036+0000","last_active":"2024-06-29T14:22:00.304951+0000","last_peered":"2024-06-29T14:22:00.304951+0000","last_clean":"2024-06-29T14:22:00.304951+0000","last_became_active":"2024-06-29T14:21:55.294958+0000","last_became_peered":"2024-06-29T14:21:55.294958+0000","last_unstale":"2024-06-29T14:22:00.304951+0000","last_undegraded":"2024-06-29T14:22:00.304951+0000","last_fullsized":"2024-06-29T14:22:00.304951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:25.523198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641393+0000","last_change":"2024-06-29T14:21:54.249131+0000","last_active":"2024-06-29T14:22:00.641393+0000","last_peered":"2024-06-29T14:22:00.641393+0000","last_clean":"2024-06-29T14:22:00.641393+0000","last_became_active":"2024-06-29T14:21:54.248727+0000","last_became_peered":"2024-06-29T14:21:54.248727+0000","last_unstale":"2024-06-29T14:22:00.641393+0000","last_undegraded":"2024-06-29T14:22:00.641393+0000","last_fullsized":"2024-06-29T14:22:00.641393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:43.007566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306263+0000","last_change":"2024-06-29T14:21:59.318902+0000","last_active":"2024-06-29T14:22:00.306263+0000","last_peered":"2024-06-29T14:22:00.306263+0000","last_clean":"2024-06-29T14:22:00.306263+0000","last_became_active":"2024-06-29T14:21:59.318388+0000","last_became_peered":"2024-06-29T14:21:59.318388+0000","last_unstale":"2024-06-29T14:22:00.306263+0000","last_undegraded":"2024-06-29T14:22:00.306263+0000","last_fullsized":"2024-06-29T14:22:00.306263+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:13.544231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305926+0000","last_change":"2024-06-29T14:21:55.282042+0000","last_active":"2024-06-29T14:22:00.305926+0000","last_peered":"2024-06-29T14:22:00.305926+0000","last_clean":"2024-06-29T14:22:00.305926+0000","last_became_active":"2024-06-29T14:21:55.281259+0000","last_became_peered":"2024-06-29T14:21:55.281259+0000","last_unstale":"2024-06-29T14:22:00.305926+0000","last_undegraded":"2024-06-29T14:22:00.305926+0000","last_fullsized":"2024-06-29T14:22:00.305926+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640041+0000","last_change":"2024-06-29T14:21:54.255093+0000","last_active":"2024-06-29T14:22:00.640041+0000","last_peered":"2024-06-29T14:22:00.640041+0000","last_clean":"2024-06-29T14:22:00.640041+0000","last_became_active":"2024-06-29T14:21:54.254010+0000","last_became_peered":"2024-06-29T14:21:54.254010+0000","last_unstale":"2024-06-29T14:22:00.640041+0000","last_undegraded":"2024-06-29T14:22:00.640041+0000","last_fullsized":"2024-06-29T14:22:00.640041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:24.835712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.642067+0000","last_change":"2024-06-29T14:21:59.337086+0000","last_active":"2024-06-29T14:22:00.642067+0000","last_peered":"2024-06-29T14:22:00.642067+0000","last_clean":"2024-06-29T14:22:00.642067+0000","last_became_active":"2024-06-29T14:21:59.336659+0000","last_became_peered":"2024-06-29T14:21:59.336659+0000","last_unstale":"2024-06-29T14:22:00.642067+0000","last_undegraded":"2024-06-29T14:22:00.642067+0000","last_fullsized":"2024-06-29T14:22:00.642067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:47.549181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306644+0000","last_change":"2024-06-29T14:21:55.289199+0000","last_active":"2024-06-29T14:22:00.306644+0000","last_peered":"2024-06-29T14:22:00.306644+0000","last_clean":"2024-06-29T14:22:00.306644+0000","last_became_active":"2024-06-29T14:21:55.288787+0000","last_became_peered":"2024-06-29T14:21:55.288787+0000","last_unstale":"2024-06-29T14:22:00.306644+0000","last_undegraded":"2024-06-29T14:22:00.306644+0000","last_fullsized":"2024-06-29T14:22:00.306644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:33.390422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640397+0000","last_change":"2024-06-29T14:21:54.260787+0000","last_active":"2024-06-29T14:22:00.640397+0000","last_peered":"2024-06-29T14:22:00.640397+0000","last_clean":"2024-06-29T14:22:00.640397+0000","last_became_active":"2024-06-29T14:21:54.260352+0000","last_became_peered":"2024-06-29T14:21:54.260352+0000","last_unstale":"2024-06-29T14:22:00.640397+0000","last_undegraded":"2024-06-29T14:22:00.640397+0000","last_fullsized":"2024-06-29T14:22:00.640397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:40.455851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305896+0000","last_change":"2024-06-29T14:21:59.316811+0000","last_active":"2024-06-29T14:22:00.305896+0000","last_peered":"2024-06-29T14:22:00.305896+0000","last_clean":"2024-06-29T14:22:00.305896+0000","last_became_active":"2024-06-29T14:21:59.316067+0000","last_became_peered":"2024-06-29T14:21:59.316067+0000","last_unstale":"2024-06-29T14:22:00.305896+0000","last_undegraded":"2024-06-29T14:22:00.305896+0000","last_fullsized":"2024-06-29T14:22:00.305896+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:04.444868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640138+0000","last_change":"2024-06-29T14:21:55.293209+0000","last_active":"2024-06-29T14:22:00.640138+0000","last_peered":"2024-06-29T14:22:00.640138+0000","last_clean":"2024-06-29T14:22:00.640138+0000","last_became_active":"2024-06-29T14:21:55.291574+0000","last_became_peered":"2024-06-29T14:21:55.291574+0000","last_unstale":"2024-06-29T14:22:00.640138+0000","last_undegraded":"2024-06-29T14:22:00.640138+0000","last_fullsized":"2024-06-29T14:22:00.640138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:16.165478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305885+0000","last_change":"2024-06-29T14:21:54.256099+0000","last_active":"2024-06-29T14:22:00.305885+0000","last_peered":"2024-06-29T14:22:00.305885+0000","last_clean":"2024-06-29T14:22:00.305885+0000","last_became_active":"2024-06-29T14:21:54.254022+0000","last_became_peered":"2024-06-29T14:21:54.254022+0000","last_unstale":"2024-06-29T14:22:00.305885+0000","last_undegraded":"2024-06-29T14:22:00.305885+0000","last_fullsized":"2024-06-29T14:22:00.305885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:32.388331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305919+0000","last_change":"2024-06-29T14:21:59.337078+0000","last_active":"2024-06-29T14:22:00.305919+0000","last_peered":"2024-06-29T14:22:00.305919+0000","last_clean":"2024-06-29T14:22:00.305919+0000","last_became_active":"2024-06-29T14:21:59.336616+0000","last_became_peered":"2024-06-29T14:21:59.336616+0000","last_unstale":"2024-06-29T14:22:00.305919+0000","last_undegraded":"2024-06-29T14:22:00.305919+0000","last_fullsized":"2024-06-29T14:22:00.305919+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:33:53.623576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305915+0000","last_change":"2024-06-29T14:21:55.293645+0000","last_active":"2024-06-29T14:22:00.305915+0000","last_peered":"2024-06-29T14:22:00.305915+0000","last_clean":"2024-06-29T14:22:00.305915+0000","last_became_active":"2024-06-29T14:21:55.291467+0000","last_became_peered":"2024-06-29T14:21:55.291467+0000","last_unstale":"2024-06-29T14:22:00.305915+0000","last_undegraded":"2024-06-29T14:22:00.305915+0000","last_fullsized":"2024-06-29T14:22:00.305915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.505918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641935+0000","last_change":"2024-06-29T14:21:54.244171+0000","last_active":"2024-06-29T14:22:00.641935+0000","last_peered":"2024-06-29T14:22:00.641935+0000","last_clean":"2024-06-29T14:22:00.641935+0000","last_became_active":"2024-06-29T14:21:54.243791+0000","last_became_peered":"2024-06-29T14:21:54.243791+0000","last_unstale":"2024-06-29T14:22:00.641935+0000","last_undegraded":"2024-06-29T14:22:00.641935+0000","last_fullsized":"2024-06-29T14:22:00.641935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:24.194470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304468+0000","last_change":"2024-06-29T14:21:59.336963+0000","last_active":"2024-06-29T14:22:00.304468+0000","last_peered":"2024-06-29T14:22:00.304468+0000","last_clean":"2024-06-29T14:22:00.304468+0000","last_became_active":"2024-06-29T14:21:59.336450+0000","last_became_peered":"2024-06-29T14:21:59.336450+0000","last_unstale":"2024-06-29T14:22:00.304468+0000","last_undegraded":"2024-06-29T14:22:00.304468+0000","last_fullsized":"2024-06-29T14:22:00.304468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:00.347317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641725+0000","last_change":"2024-06-29T14:21:55.271686+0000","last_active":"2024-06-29T14:22:00.641725+0000","last_peered":"2024-06-29T14:22:00.641725+0000","last_clean":"2024-06-29T14:22:00.641725+0000","last_became_active":"2024-06-29T14:21:55.271310+0000","last_became_peered":"2024-06-29T14:21:55.271310+0000","last_unstale":"2024-06-29T14:22:00.641725+0000","last_undegraded":"2024-06-29T14:22:00.641725+0000","last_fullsized":"2024-06-29T14:22:00.641725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641232+0000","last_change":"2024-06-29T14:21:54.260566+0000","last_active":"2024-06-29T14:22:00.641232+0000","last_peered":"2024-06-29T14:22:00.641232+0000","last_clean":"2024-06-29T14:22:00.641232+0000","last_became_active":"2024-06-29T14:21:54.260089+0000","last_became_peered":"2024-06-29T14:21:54.260089+0000","last_unstale":"2024-06-29T14:22:00.641232+0000","last_undegraded":"2024-06-29T14:22:00.641232+0000","last_fullsized":"2024-06-29T14:22:00.641232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:28.957956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640300+0000","last_change":"2024-06-29T14:21:59.338874+0000","last_active":"2024-06-29T14:22:00.640300+0000","last_peered":"2024-06-29T14:22:00.640300+0000","last_clean":"2024-06-29T14:22:00.640300+0000","last_became_active":"2024-06-29T14:21:59.338388+0000","last_became_peered":"2024-06-29T14:21:59.338388+0000","last_unstale":"2024-06-29T14:22:00.640300+0000","last_undegraded":"2024-06-29T14:22:00.640300+0000","last_fullsized":"2024-06-29T14:22:00.640300+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:21.205918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641110+0000","last_change":"2024-06-29T14:21:55.284217+0000","last_active":"2024-06-29T14:22:00.641110+0000","last_peered":"2024-06-29T14:22:00.641110+0000","last_clean":"2024-06-29T14:22:00.641110+0000","last_became_active":"2024-06-29T14:21:55.283906+0000","last_became_peered":"2024-06-29T14:21:55.283906+0000","last_unstale":"2024-06-29T14:22:00.641110+0000","last_undegraded":"2024-06-29T14:22:00.641110+0000","last_fullsized":"2024-06-29T14:22:00.641110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:39.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641864+0000","last_change":"2024-06-29T14:21:54.257367+0000","last_active":"2024-06-29T14:22:00.641864+0000","last_peered":"2024-06-29T14:22:00.641864+0000","last_clean":"2024-06-29T14:22:00.641864+0000","last_became_active":"2024-06-29T14:21:54.257040+0000","last_became_peered":"2024-06-29T14:21:54.257040+0000","last_unstale":"2024-06-29T14:22:00.641864+0000","last_undegraded":"2024-06-29T14:22:00.641864+0000","last_fullsized":"2024-06-29T14:22:00.641864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:53.425388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639860+0000","last_change":"2024-06-29T14:21:59.323943+0000","last_active":"2024-06-29T14:22:00.639860+0000","last_peered":"2024-06-29T14:22:00.639860+0000","last_clean":"2024-06-29T14:22:00.639860+0000","last_became_active":"2024-06-29T14:21:59.323400+0000","last_became_peered":"2024-06-29T14:21:59.323400+0000","last_unstale":"2024-06-29T14:22:00.639860+0000","last_undegraded":"2024-06-29T14:22:00.639860+0000","last_fullsized":"2024-06-29T14:22:00.639860+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:01.086377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307356+0000","last_change":"2024-06-29T14:21:55.292906+0000","last_active":"2024-06-29T14:22:00.307356+0000","last_peered":"2024-06-29T14:22:00.307356+0000","last_clean":"2024-06-29T14:22:00.307356+0000","last_became_active":"2024-06-29T14:21:55.292578+0000","last_became_peered":"2024-06-29T14:21:55.292578+0000","last_unstale":"2024-06-29T14:22:00.307356+0000","last_undegraded":"2024-06-29T14:22:00.307356+0000","last_fullsized":"2024-06-29T14:22:00.307356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:11.801869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640191+0000","last_change":"2024-06-29T14:21:54.264501+0000","last_active":"2024-06-29T14:22:00.640191+0000","last_peered":"2024-06-29T14:22:00.640191+0000","last_clean":"2024-06-29T14:22:00.640191+0000","last_became_active":"2024-06-29T14:21:54.264182+0000","last_became_peered":"2024-06-29T14:21:54.264182+0000","last_unstale":"2024-06-29T14:22:00.640191+0000","last_undegraded":"2024-06-29T14:22:00.640191+0000","last_fullsized":"2024-06-29T14:22:00.640191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:53.425388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641814+0000","last_change":"2024-06-29T14:21:59.324707+0000","last_active":"2024-06-29T14:22:00.641814+0000","last_peered":"2024-06-29T14:22:00.641814+0000","last_clean":"2024-06-29T14:22:00.641814+0000","last_became_active":"2024-06-29T14:21:59.324141+0000","last_became_peered":"2024-06-29T14:21:59.324141+0000","last_unstale":"2024-06-29T14:22:00.641814+0000","last_undegraded":"2024-06-29T14:22:00.641814+0000","last_fullsized":"2024-06-29T14:22:00.641814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:43.889397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639772+0000","last_change":"2024-06-29T14:21:55.274636+0000","last_active":"2024-06-29T14:22:00.639772+0000","last_peered":"2024-06-29T14:22:00.639772+0000","last_clean":"2024-06-29T14:22:00.639772+0000","last_became_active":"2024-06-29T14:21:55.271330+0000","last_became_peered":"2024-06-29T14:21:55.271330+0000","last_unstale":"2024-06-29T14:22:00.639772+0000","last_undegraded":"2024-06-29T14:22:00.639772+0000","last_fullsized":"2024-06-29T14:22:00.639772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640544+0000","last_change":"2024-06-29T14:21:54.247945+0000","last_active":"2024-06-29T14:22:00.640544+0000","last_peered":"2024-06-29T14:22:00.640544+0000","last_clean":"2024-06-29T14:22:00.640544+0000","last_became_active":"2024-06-29T14:21:54.247592+0000","last_became_peered":"2024-06-29T14:21:54.247592+0000","last_unstale":"2024-06-29T14:22:00.640544+0000","last_undegraded":"2024-06-29T14:22:00.640544+0000","last_fullsized":"2024-06-29T14:22:00.640544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:03.449985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.642194+0000","last_change":"2024-06-29T14:21:59.321043+0000","last_active":"2024-06-29T14:22:00.642194+0000","last_peered":"2024-06-29T14:22:00.642194+0000","last_clean":"2024-06-29T14:22:00.642194+0000","last_became_active":"2024-06-29T14:21:59.320359+0000","last_became_peered":"2024-06-29T14:21:59.320359+0000","last_unstale":"2024-06-29T14:22:00.642194+0000","last_undegraded":"2024-06-29T14:22:00.642194+0000","last_fullsized":"2024-06-29T14:22:00.642194+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:22.691854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639595+0000","last_change":"2024-06-29T14:21:55.296697+0000","last_active":"2024-06-29T14:22:00.639595+0000","last_peered":"2024-06-29T14:22:00.639595+0000","last_clean":"2024-06-29T14:22:00.639595+0000","last_became_active":"2024-06-29T14:21:55.296392+0000","last_became_peered":"2024-06-29T14:21:55.296392+0000","last_unstale":"2024-06-29T14:22:00.639595+0000","last_undegraded":"2024-06-29T14:22:00.639595+0000","last_fullsized":"2024-06-29T14:22:00.639595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:17.069875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639911+0000","last_change":"2024-06-29T14:21:54.257791+0000","last_active":"2024-06-29T14:22:00.639911+0000","last_peered":"2024-06-29T14:22:00.639911+0000","last_clean":"2024-06-29T14:22:00.639911+0000","last_became_active":"2024-06-29T14:21:54.257395+0000","last_became_peered":"2024-06-29T14:21:54.257395+0000","last_unstale":"2024-06-29T14:22:00.639911+0000","last_undegraded":"2024-06-29T14:22:00.639911+0000","last_fullsized":"2024-06-29T14:22:00.639911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:33.159720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640058+0000","last_change":"2024-06-29T14:21:59.337618+0000","last_active":"2024-06-29T14:22:00.640058+0000","last_peered":"2024-06-29T14:22:00.640058+0000","last_clean":"2024-06-29T14:22:00.640058+0000","last_became_active":"2024-06-29T14:21:59.336503+0000","last_became_peered":"2024-06-29T14:21:59.336503+0000","last_unstale":"2024-06-29T14:22:00.640058+0000","last_undegraded":"2024-06-29T14:22:00.640058+0000","last_fullsized":"2024-06-29T14:22:00.640058+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:01.086377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304940+0000","last_change":"2024-06-29T14:21:55.284345+0000","last_active":"2024-06-29T14:22:00.304940+0000","last_peered":"2024-06-29T14:22:00.304940+0000","last_clean":"2024-06-29T14:22:00.304940+0000","last_became_active":"2024-06-29T14:21:55.283603+0000","last_became_peered":"2024-06-29T14:21:55.283603+0000","last_unstale":"2024-06-29T14:22:00.304940+0000","last_undegraded":"2024-06-29T14:22:00.304940+0000","last_fullsized":"2024-06-29T14:22:00.304940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639902+0000","last_change":"2024-06-29T14:21:54.259298+0000","last_active":"2024-06-29T14:22:00.639902+0000","last_peered":"2024-06-29T14:22:00.639902+0000","last_clean":"2024-06-29T14:22:00.639902+0000","last_became_active":"2024-06-29T14:21:54.258971+0000","last_became_peered":"2024-06-29T14:21:54.258971+0000","last_unstale":"2024-06-29T14:22:00.639902+0000","last_undegraded":"2024-06-29T14:22:00.639902+0000","last_fullsized":"2024-06-29T14:22:00.639902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:34.839629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":117,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640060+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:22:00.640060+0000","last_peered":"2024-06-29T14:22:00.640060+0000","last_clean":"2024-06-29T14:22:00.640060+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:22:00.640060+0000","last_undegraded":"2024-06-29T14:22:00.640060+0000","last_fullsized":"2024-06-29T14:22:00.640060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305693+0000","last_change":"2024-06-29T14:21:59.337431+0000","last_active":"2024-06-29T14:22:00.305693+0000","last_peered":"2024-06-29T14:22:00.305693+0000","last_clean":"2024-06-29T14:22:00.305693+0000","last_became_active":"2024-06-29T14:21:59.337008+0000","last_became_peered":"2024-06-29T14:21:59.337008+0000","last_unstale":"2024-06-29T14:22:00.305693+0000","last_undegraded":"2024-06-29T14:22:00.305693+0000","last_fullsized":"2024-06-29T14:22:00.305693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:56.220106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640737+0000","last_change":"2024-06-29T14:21:55.287255+0000","last_active":"2024-06-29T14:22:00.640737+0000","last_peered":"2024-06-29T14:22:00.640737+0000","last_clean":"2024-06-29T14:22:00.640737+0000","last_became_active":"2024-06-29T14:21:55.286563+0000","last_became_peered":"2024-06-29T14:21:55.286563+0000","last_unstale":"2024-06-29T14:22:00.640737+0000","last_undegraded":"2024-06-29T14:22:00.640737+0000","last_fullsized":"2024-06-29T14:22:00.640737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:40.442251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640792+0000","last_change":"2024-06-29T14:21:54.264774+0000","last_active":"2024-06-29T14:22:00.640792+0000","last_peered":"2024-06-29T14:22:00.640792+0000","last_clean":"2024-06-29T14:22:00.640792+0000","last_became_active":"2024-06-29T14:21:54.264395+0000","last_became_peered":"2024-06-29T14:21:54.264395+0000","last_unstale":"2024-06-29T14:22:00.640792+0000","last_undegraded":"2024-06-29T14:22:00.640792+0000","last_fullsized":"2024-06-29T14:22:00.640792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:49.746280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305477+0000","last_change":"2024-06-29T14:21:59.336774+0000","last_active":"2024-06-29T14:22:00.305477+0000","last_peered":"2024-06-29T14:22:00.305477+0000","last_clean":"2024-06-29T14:22:00.305477+0000","last_became_active":"2024-06-29T14:21:59.336326+0000","last_became_peered":"2024-06-29T14:21:59.336326+0000","last_unstale":"2024-06-29T14:22:00.305477+0000","last_undegraded":"2024-06-29T14:22:00.305477+0000","last_fullsized":"2024-06-29T14:22:00.305477+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:20.883221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640544+0000","last_change":"2024-06-29T14:21:55.289532+0000","last_active":"2024-06-29T14:22:00.640544+0000","last_peered":"2024-06-29T14:22:00.640544+0000","last_clean":"2024-06-29T14:22:00.640544+0000","last_became_active":"2024-06-29T14:21:55.289210+0000","last_became_peered":"2024-06-29T14:21:55.289210+0000","last_unstale":"2024-06-29T14:22:00.640544+0000","last_undegraded":"2024-06-29T14:22:00.640544+0000","last_fullsized":"2024-06-29T14:22:00.640544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:12.073214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641944+0000","last_change":"2024-06-29T14:21:54.246332+0000","last_active":"2024-06-29T14:22:00.641944+0000","last_peered":"2024-06-29T14:22:00.641944+0000","last_clean":"2024-06-29T14:22:00.641944+0000","last_became_active":"2024-06-29T14:21:54.245974+0000","last_became_peered":"2024-06-29T14:21:54.245974+0000","last_unstale":"2024-06-29T14:22:00.641944+0000","last_undegraded":"2024-06-29T14:22:00.641944+0000","last_fullsized":"2024-06-29T14:22:00.641944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:50:54.491455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640419+0000","last_change":"2024-06-29T14:21:59.318499+0000","last_active":"2024-06-29T14:22:00.640419+0000","last_peered":"2024-06-29T14:22:00.640419+0000","last_clean":"2024-06-29T14:22:00.640419+0000","last_became_active":"2024-06-29T14:21:59.317918+0000","last_became_peered":"2024-06-29T14:21:59.317918+0000","last_unstale":"2024-06-29T14:22:00.640419+0000","last_undegraded":"2024-06-29T14:22:00.640419+0000","last_fullsized":"2024-06-29T14:22:00.640419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:43.889397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641394+0000","last_change":"2024-06-29T14:21:55.294388+0000","last_active":"2024-06-29T14:22:00.641394+0000","last_peered":"2024-06-29T14:22:00.641394+0000","last_clean":"2024-06-29T14:22:00.641394+0000","last_became_active":"2024-06-29T14:21:55.294090+0000","last_became_peered":"2024-06-29T14:21:55.294090+0000","last_unstale":"2024-06-29T14:22:00.641394+0000","last_undegraded":"2024-06-29T14:22:00.641394+0000","last_fullsized":"2024-06-29T14:22:00.641394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:39.325162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641102+0000","last_change":"2024-06-29T14:21:54.252699+0000","last_active":"2024-06-29T14:22:00.641102+0000","last_peered":"2024-06-29T14:22:00.641102+0000","last_clean":"2024-06-29T14:22:00.641102+0000","last_became_active":"2024-06-29T14:21:54.252365+0000","last_became_peered":"2024-06-29T14:21:54.252365+0000","last_unstale":"2024-06-29T14:22:00.641102+0000","last_undegraded":"2024-06-29T14:22:00.641102+0000","last_fullsized":"2024-06-29T14:22:00.641102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:32.388331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640359+0000","last_change":"2024-06-29T14:21:59.337020+0000","last_active":"2024-06-29T14:22:00.640359+0000","last_peered":"2024-06-29T14:22:00.640359+0000","last_clean":"2024-06-29T14:22:00.640359+0000","last_became_active":"2024-06-29T14:21:59.336457+0000","last_became_peered":"2024-06-29T14:21:59.336457+0000","last_unstale":"2024-06-29T14:22:00.640359+0000","last_undegraded":"2024-06-29T14:22:00.640359+0000","last_fullsized":"2024-06-29T14:22:00.640359+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:25.501876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640773+0000","last_change":"2024-06-29T14:21:55.297155+0000","last_active":"2024-06-29T14:22:00.640773+0000","last_peered":"2024-06-29T14:22:00.640773+0000","last_clean":"2024-06-29T14:22:00.640773+0000","last_became_active":"2024-06-29T14:21:55.296790+0000","last_became_peered":"2024-06-29T14:21:55.296790+0000","last_unstale":"2024-06-29T14:22:00.640773+0000","last_undegraded":"2024-06-29T14:22:00.640773+0000","last_fullsized":"2024-06-29T14:22:00.640773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:52.163431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306370+0000","last_change":"2024-06-29T14:21:54.261995+0000","last_active":"2024-06-29T14:22:00.306370+0000","last_peered":"2024-06-29T14:22:00.306370+0000","last_clean":"2024-06-29T14:22:00.306370+0000","last_became_active":"2024-06-29T14:21:54.261607+0000","last_became_peered":"2024-06-29T14:21:54.261607+0000","last_unstale":"2024-06-29T14:22:00.306370+0000","last_undegraded":"2024-06-29T14:22:00.306370+0000","last_fullsized":"2024-06-29T14:22:00.306370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:59.466378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305836+0000","last_change":"2024-06-29T14:21:59.316601+0000","last_active":"2024-06-29T14:22:00.305836+0000","last_peered":"2024-06-29T14:22:00.305836+0000","last_clean":"2024-06-29T14:22:00.305836+0000","last_became_active":"2024-06-29T14:21:59.315986+0000","last_became_peered":"2024-06-29T14:21:59.315986+0000","last_unstale":"2024-06-29T14:22:00.305836+0000","last_undegraded":"2024-06-29T14:22:00.305836+0000","last_fullsized":"2024-06-29T14:22:00.305836+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:45.085147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641018+0000","last_change":"2024-06-29T14:21:55.291524+0000","last_active":"2024-06-29T14:22:00.641018+0000","last_peered":"2024-06-29T14:22:00.641018+0000","last_clean":"2024-06-29T14:22:00.641018+0000","last_became_active":"2024-06-29T14:21:55.291206+0000","last_became_peered":"2024-06-29T14:21:55.291206+0000","last_unstale":"2024-06-29T14:22:00.641018+0000","last_undegraded":"2024-06-29T14:22:00.641018+0000","last_fullsized":"2024-06-29T14:22:00.641018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:16.826546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640410+0000","last_change":"2024-06-29T14:21:54.245875+0000","last_active":"2024-06-29T14:22:00.640410+0000","last_peered":"2024-06-29T14:22:00.640410+0000","last_clean":"2024-06-29T14:22:00.640410+0000","last_became_active":"2024-06-29T14:21:54.245444+0000","last_became_peered":"2024-06-29T14:21:54.245444+0000","last_unstale":"2024-06-29T14:22:00.640410+0000","last_undegraded":"2024-06-29T14:22:00.640410+0000","last_fullsized":"2024-06-29T14:22:00.640410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:49.347688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639988+0000","last_change":"2024-06-29T14:21:59.317611+0000","last_active":"2024-06-29T14:22:00.639988+0000","last_peered":"2024-06-29T14:22:00.639988+0000","last_clean":"2024-06-29T14:22:00.639988+0000","last_became_active":"2024-06-29T14:21:59.317081+0000","last_became_peered":"2024-06-29T14:21:59.317081+0000","last_unstale":"2024-06-29T14:22:00.639988+0000","last_undegraded":"2024-06-29T14:22:00.639988+0000","last_fullsized":"2024-06-29T14:22:00.639988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:16.129888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305414+0000","last_change":"2024-06-29T14:21:55.291070+0000","last_active":"2024-06-29T14:22:00.305414+0000","last_peered":"2024-06-29T14:22:00.305414+0000","last_clean":"2024-06-29T14:22:00.305414+0000","last_became_active":"2024-06-29T14:21:55.290468+0000","last_became_peered":"2024-06-29T14:21:55.290468+0000","last_unstale":"2024-06-29T14:22:00.305414+0000","last_undegraded":"2024-06-29T14:22:00.305414+0000","last_fullsized":"2024-06-29T14:22:00.305414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:15.443094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305532+0000","last_change":"2024-06-29T14:21:54.257111+0000","last_active":"2024-06-29T14:22:00.305532+0000","last_peered":"2024-06-29T14:22:00.305532+0000","last_clean":"2024-06-29T14:22:00.305532+0000","last_became_active":"2024-06-29T14:21:54.255536+0000","last_became_peered":"2024-06-29T14:21:54.255536+0000","last_unstale":"2024-06-29T14:22:00.305532+0000","last_undegraded":"2024-06-29T14:22:00.305532+0000","last_fullsized":"2024-06-29T14:22:00.305532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:28:41.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305863+0000","last_change":"2024-06-29T14:21:59.338565+0000","last_active":"2024-06-29T14:22:00.305863+0000","last_peered":"2024-06-29T14:22:00.305863+0000","last_clean":"2024-06-29T14:22:00.305863+0000","last_became_active":"2024-06-29T14:21:59.337920+0000","last_became_peered":"2024-06-29T14:21:59.337920+0000","last_unstale":"2024-06-29T14:22:00.305863+0000","last_undegraded":"2024-06-29T14:22:00.305863+0000","last_fullsized":"2024-06-29T14:22:00.305863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:46.890912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640755+0000","last_change":"2024-06-29T14:21:55.294165+0000","last_active":"2024-06-29T14:22:00.640755+0000","last_peered":"2024-06-29T14:22:00.640755+0000","last_clean":"2024-06-29T14:22:00.640755+0000","last_became_active":"2024-06-29T14:21:55.293742+0000","last_became_peered":"2024-06-29T14:21:55.293742+0000","last_unstale":"2024-06-29T14:22:00.640755+0000","last_undegraded":"2024-06-29T14:22:00.640755+0000","last_fullsized":"2024-06-29T14:22:00.640755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:35.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640160+0000","last_change":"2024-06-29T14:21:54.261554+0000","last_active":"2024-06-29T14:22:00.640160+0000","last_peered":"2024-06-29T14:22:00.640160+0000","last_clean":"2024-06-29T14:22:00.640160+0000","last_became_active":"2024-06-29T14:21:54.261215+0000","last_became_peered":"2024-06-29T14:21:54.261215+0000","last_unstale":"2024-06-29T14:22:00.640160+0000","last_undegraded":"2024-06-29T14:22:00.640160+0000","last_fullsized":"2024-06-29T14:22:00.640160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:43.007566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305468+0000","last_change":"2024-06-29T14:21:59.338884+0000","last_active":"2024-06-29T14:22:00.305468+0000","last_peered":"2024-06-29T14:22:00.305468+0000","last_clean":"2024-06-29T14:22:00.305468+0000","last_became_active":"2024-06-29T14:21:59.336402+0000","last_became_peered":"2024-06-29T14:21:59.336402+0000","last_unstale":"2024-06-29T14:22:00.305468+0000","last_undegraded":"2024-06-29T14:22:00.305468+0000","last_fullsized":"2024-06-29T14:22:00.305468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:00.545575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640161+0000","last_change":"2024-06-29T14:21:55.288628+0000","last_active":"2024-06-29T14:22:00.640161+0000","last_peered":"2024-06-29T14:22:00.640161+0000","last_clean":"2024-06-29T14:22:00.640161+0000","last_became_active":"2024-06-29T14:21:55.288079+0000","last_became_peered":"2024-06-29T14:21:55.288079+0000","last_unstale":"2024-06-29T14:22:00.640161+0000","last_undegraded":"2024-06-29T14:22:00.640161+0000","last_fullsized":"2024-06-29T14:22:00.640161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:44.658689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640560+0000","last_change":"2024-06-29T14:21:54.263230+0000","last_active":"2024-06-29T14:22:00.640560+0000","last_peered":"2024-06-29T14:22:00.640560+0000","last_clean":"2024-06-29T14:22:00.640560+0000","last_became_active":"2024-06-29T14:21:54.262952+0000","last_became_peered":"2024-06-29T14:21:54.262952+0000","last_unstale":"2024-06-29T14:22:00.640560+0000","last_undegraded":"2024-06-29T14:22:00.640560+0000","last_fullsized":"2024-06-29T14:22:00.640560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:48.032837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640470+0000","last_change":"2024-06-29T14:21:59.337082+0000","last_active":"2024-06-29T14:22:00.640470+0000","last_peered":"2024-06-29T14:22:00.640470+0000","last_clean":"2024-06-29T14:22:00.640470+0000","last_became_active":"2024-06-29T14:21:59.336506+0000","last_became_peered":"2024-06-29T14:21:59.336506+0000","last_unstale":"2024-06-29T14:22:00.640470+0000","last_undegraded":"2024-06-29T14:22:00.640470+0000","last_fullsized":"2024-06-29T14:22:00.640470+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:00.347317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640609+0000","last_change":"2024-06-29T14:21:55.299489+0000","last_active":"2024-06-29T14:22:00.640609+0000","last_peered":"2024-06-29T14:22:00.640609+0000","last_clean":"2024-06-29T14:22:00.640609+0000","last_became_active":"2024-06-29T14:21:55.299181+0000","last_became_peered":"2024-06-29T14:21:55.299181+0000","last_unstale":"2024-06-29T14:22:00.640609+0000","last_undegraded":"2024-06-29T14:22:00.640609+0000","last_fullsized":"2024-06-29T14:22:00.640609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:31.165204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:05.053670+0000","last_change":"2024-06-29T14:21:54.251485+0000","last_active":"2024-06-29T14:22:05.053670+0000","last_peered":"2024-06-29T14:22:05.053670+0000","last_clean":"2024-06-29T14:22:05.053670+0000","last_became_active":"2024-06-29T14:21:54.250203+0000","last_became_peered":"2024-06-29T14:21:54.250203+0000","last_unstale":"2024-06-29T14:22:05.053670+0000","last_undegraded":"2024-06-29T14:22:05.053670+0000","last_fullsized":"2024-06-29T14:22:05.053670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:03.449985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641539+0000","last_change":"2024-06-29T14:21:59.324423+0000","last_active":"2024-06-29T14:22:00.641539+0000","last_peered":"2024-06-29T14:22:00.641539+0000","last_clean":"2024-06-29T14:22:00.641539+0000","last_became_active":"2024-06-29T14:21:59.323663+0000","last_became_peered":"2024-06-29T14:21:59.323663+0000","last_unstale":"2024-06-29T14:22:00.641539+0000","last_undegraded":"2024-06-29T14:22:00.641539+0000","last_fullsized":"2024-06-29T14:22:00.641539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:21.205918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307509+0000","last_change":"2024-06-29T14:21:55.292602+0000","last_active":"2024-06-29T14:22:00.307509+0000","last_peered":"2024-06-29T14:22:00.307509+0000","last_clean":"2024-06-29T14:22:00.307509+0000","last_became_active":"2024-06-29T14:21:55.292047+0000","last_became_peered":"2024-06-29T14:21:55.292047+0000","last_unstale":"2024-06-29T14:22:00.307509+0000","last_undegraded":"2024-06-29T14:22:00.307509+0000","last_fullsized":"2024-06-29T14:22:00.307509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:15.123791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641934+0000","last_change":"2024-06-29T14:21:54.264727+0000","last_active":"2024-06-29T14:22:00.641934+0000","last_peered":"2024-06-29T14:22:00.641934+0000","last_clean":"2024-06-29T14:22:00.641934+0000","last_became_active":"2024-06-29T14:21:54.264355+0000","last_became_peered":"2024-06-29T14:21:54.264355+0000","last_unstale":"2024-06-29T14:22:00.641934+0000","last_undegraded":"2024-06-29T14:22:00.641934+0000","last_fullsized":"2024-06-29T14:22:00.641934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:06.025267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306568+0000","last_change":"2024-06-29T14:21:59.318452+0000","last_active":"2024-06-29T14:22:00.306568+0000","last_peered":"2024-06-29T14:22:00.306568+0000","last_clean":"2024-06-29T14:22:00.306568+0000","last_became_active":"2024-06-29T14:21:59.317634+0000","last_became_peered":"2024-06-29T14:21:59.317634+0000","last_unstale":"2024-06-29T14:22:00.306568+0000","last_undegraded":"2024-06-29T14:22:00.306568+0000","last_fullsized":"2024-06-29T14:22:00.306568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:46.219297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640791+0000","last_change":"2024-06-29T14:21:55.284276+0000","last_active":"2024-06-29T14:22:00.640791+0000","last_peered":"2024-06-29T14:22:00.640791+0000","last_clean":"2024-06-29T14:22:00.640791+0000","last_became_active":"2024-06-29T14:21:55.283929+0000","last_became_peered":"2024-06-29T14:21:55.283929+0000","last_unstale":"2024-06-29T14:22:00.640791+0000","last_undegraded":"2024-06-29T14:22:00.640791+0000","last_fullsized":"2024-06-29T14:22:00.640791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640975+0000","last_change":"2024-06-29T14:21:54.260674+0000","last_active":"2024-06-29T14:22:00.640975+0000","last_peered":"2024-06-29T14:22:00.640975+0000","last_clean":"2024-06-29T14:22:00.640975+0000","last_became_active":"2024-06-29T14:21:54.258991+0000","last_became_peered":"2024-06-29T14:21:54.258991+0000","last_unstale":"2024-06-29T14:22:00.640975+0000","last_undegraded":"2024-06-29T14:22:00.640975+0000","last_fullsized":"2024-06-29T14:22:00.640975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:50:54.491455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305903+0000","last_change":"2024-06-29T14:21:59.319141+0000","last_active":"2024-06-29T14:22:00.305903+0000","last_peered":"2024-06-29T14:22:00.305903+0000","last_clean":"2024-06-29T14:22:00.305903+0000","last_became_active":"2024-06-29T14:21:59.318645+0000","last_became_peered":"2024-06-29T14:21:59.318645+0000","last_unstale":"2024-06-29T14:22:00.305903+0000","last_undegraded":"2024-06-29T14:22:00.305903+0000","last_fullsized":"2024-06-29T14:22:00.305903+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:50.697736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640894+0000","last_change":"2024-06-29T14:21:55.293028+0000","last_active":"2024-06-29T14:22:00.640894+0000","last_peered":"2024-06-29T14:22:00.640894+0000","last_clean":"2024-06-29T14:22:00.640894+0000","last_became_active":"2024-06-29T14:21:55.292709+0000","last_became_peered":"2024-06-29T14:21:55.292709+0000","last_unstale":"2024-06-29T14:22:00.640894+0000","last_undegraded":"2024-06-29T14:22:00.640894+0000","last_fullsized":"2024-06-29T14:22:00.640894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:30.912272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305837+0000","last_change":"2024-06-29T14:21:54.262667+0000","last_active":"2024-06-29T14:22:00.305837+0000","last_peered":"2024-06-29T14:22:00.305837+0000","last_clean":"2024-06-29T14:22:00.305837+0000","last_became_active":"2024-06-29T14:21:54.262286+0000","last_became_peered":"2024-06-29T14:21:54.262286+0000","last_unstale":"2024-06-29T14:22:00.305837+0000","last_undegraded":"2024-06-29T14:22:00.305837+0000","last_fullsized":"2024-06-29T14:22:00.305837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:49.746280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304956+0000","last_change":"2024-06-29T14:21:59.337794+0000","last_active":"2024-06-29T14:22:00.304956+0000","last_peered":"2024-06-29T14:22:00.304956+0000","last_clean":"2024-06-29T14:22:00.304956+0000","last_became_active":"2024-06-29T14:21:59.336659+0000","last_became_peered":"2024-06-29T14:21:59.336659+0000","last_unstale":"2024-06-29T14:22:00.304956+0000","last_undegraded":"2024-06-29T14:22:00.304956+0000","last_fullsized":"2024-06-29T14:22:00.304956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:15.282310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639863+0000","last_change":"2024-06-29T14:21:59.337985+0000","last_active":"2024-06-29T14:22:00.639863+0000","last_peered":"2024-06-29T14:22:00.639863+0000","last_clean":"2024-06-29T14:22:00.639863+0000","last_became_active":"2024-06-29T14:21:59.337142+0000","last_became_peered":"2024-06-29T14:21:59.337142+0000","last_unstale":"2024-06-29T14:22:00.639863+0000","last_undegraded":"2024-06-29T14:22:00.639863+0000","last_fullsized":"2024-06-29T14:22:00.639863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:26.400192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639809+0000","last_change":"2024-06-29T14:21:55.291354+0000","last_active":"2024-06-29T14:22:00.639809+0000","last_peered":"2024-06-29T14:22:00.639809+0000","last_clean":"2024-06-29T14:22:00.639809+0000","last_became_active":"2024-06-29T14:21:55.290328+0000","last_became_peered":"2024-06-29T14:21:55.290328+0000","last_unstale":"2024-06-29T14:22:00.639809+0000","last_undegraded":"2024-06-29T14:22:00.639809+0000","last_fullsized":"2024-06-29T14:22:00.639809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639839+0000","last_change":"2024-06-29T14:21:54.254791+0000","last_active":"2024-06-29T14:22:00.639839+0000","last_peered":"2024-06-29T14:22:00.639839+0000","last_clean":"2024-06-29T14:22:00.639839+0000","last_became_active":"2024-06-29T14:21:54.254345+0000","last_became_peered":"2024-06-29T14:21:54.254345+0000","last_unstale":"2024-06-29T14:22:00.639839+0000","last_undegraded":"2024-06-29T14:22:00.639839+0000","last_fullsized":"2024-06-29T14:22:00.639839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:06.025267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305684+0000","last_change":"2024-06-29T14:21:59.317911+0000","last_active":"2024-06-29T14:22:00.305684+0000","last_peered":"2024-06-29T14:22:00.305684+0000","last_clean":"2024-06-29T14:22:00.305684+0000","last_became_active":"2024-06-29T14:21:59.317469+0000","last_became_peered":"2024-06-29T14:21:59.317469+0000","last_unstale":"2024-06-29T14:22:00.305684+0000","last_undegraded":"2024-06-29T14:22:00.305684+0000","last_fullsized":"2024-06-29T14:22:00.305684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:32.630422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305774+0000","last_change":"2024-06-29T14:21:55.297533+0000","last_active":"2024-06-29T14:22:00.305774+0000","last_peered":"2024-06-29T14:22:00.305774+0000","last_clean":"2024-06-29T14:22:00.305774+0000","last_became_active":"2024-06-29T14:21:55.296487+0000","last_became_peered":"2024-06-29T14:21:55.296487+0000","last_unstale":"2024-06-29T14:22:00.305774+0000","last_undegraded":"2024-06-29T14:22:00.305774+0000","last_fullsized":"2024-06-29T14:22:00.305774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305104+0000","last_change":"2024-06-29T14:21:54.253349+0000","last_active":"2024-06-29T14:22:00.305104+0000","last_peered":"2024-06-29T14:22:00.305104+0000","last_clean":"2024-06-29T14:22:00.305104+0000","last_became_active":"2024-06-29T14:21:54.252606+0000","last_became_peered":"2024-06-29T14:21:54.252606+0000","last_unstale":"2024-06-29T14:22:00.305104+0000","last_undegraded":"2024-06-29T14:22:00.305104+0000","last_fullsized":"2024-06-29T14:22:00.305104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:20.986619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306038+0000","last_change":"2024-06-29T14:21:54.254902+0000","last_active":"2024-06-29T14:22:00.306038+0000","last_peered":"2024-06-29T14:22:00.306038+0000","last_clean":"2024-06-29T14:22:00.306038+0000","last_became_active":"2024-06-29T14:21:54.254549+0000","last_became_peered":"2024-06-29T14:21:54.254549+0000","last_unstale":"2024-06-29T14:22:00.306038+0000","last_undegraded":"2024-06-29T14:22:00.306038+0000","last_fullsized":"2024-06-29T14:22:00.306038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:49.347688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640490+0000","last_change":"2024-06-29T14:21:55.297816+0000","last_active":"2024-06-29T14:22:00.640490+0000","last_peered":"2024-06-29T14:22:00.640490+0000","last_clean":"2024-06-29T14:22:00.640490+0000","last_became_active":"2024-06-29T14:21:55.297080+0000","last_became_peered":"2024-06-29T14:21:55.297080+0000","last_unstale":"2024-06-29T14:22:00.640490+0000","last_undegraded":"2024-06-29T14:22:00.640490+0000","last_fullsized":"2024-06-29T14:22:00.640490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640695+0000","last_change":"2024-06-29T14:21:59.324236+0000","last_active":"2024-06-29T14:22:00.640695+0000","last_peered":"2024-06-29T14:22:00.640695+0000","last_clean":"2024-06-29T14:22:00.640695+0000","last_became_active":"2024-06-29T14:21:59.323673+0000","last_became_peered":"2024-06-29T14:21:59.323673+0000","last_unstale":"2024-06-29T14:22:00.640695+0000","last_undegraded":"2024-06-29T14:22:00.640695+0000","last_fullsized":"2024-06-29T14:22:00.640695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:45.514616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305150+0000","last_change":"2024-06-29T14:21:54.257451+0000","last_active":"2024-06-29T14:22:00.305150+0000","last_peered":"2024-06-29T14:22:00.305150+0000","last_clean":"2024-06-29T14:22:00.305150+0000","last_became_active":"2024-06-29T14:21:54.257055+0000","last_became_peered":"2024-06-29T14:21:54.257055+0000","last_unstale":"2024-06-29T14:22:00.305150+0000","last_undegraded":"2024-06-29T14:22:00.305150+0000","last_fullsized":"2024-06-29T14:22:00.305150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:10:39.689935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307544+0000","last_change":"2024-06-29T14:21:55.291560+0000","last_active":"2024-06-29T14:22:00.307544+0000","last_peered":"2024-06-29T14:22:00.307544+0000","last_clean":"2024-06-29T14:22:00.307544+0000","last_became_active":"2024-06-29T14:21:55.291263+0000","last_became_peered":"2024-06-29T14:21:55.291263+0000","last_unstale":"2024-06-29T14:22:00.307544+0000","last_undegraded":"2024-06-29T14:22:00.307544+0000","last_fullsized":"2024-06-29T14:22:00.307544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:50.748371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305606+0000","last_change":"2024-06-29T14:21:59.319249+0000","last_active":"2024-06-29T14:22:00.305606+0000","last_peered":"2024-06-29T14:22:00.305606+0000","last_clean":"2024-06-29T14:22:00.305606+0000","last_became_active":"2024-06-29T14:21:59.316597+0000","last_became_peered":"2024-06-29T14:21:59.316597+0000","last_unstale":"2024-06-29T14:22:00.305606+0000","last_undegraded":"2024-06-29T14:22:00.305606+0000","last_fullsized":"2024-06-29T14:22:00.305606+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:54.923537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305108+0000","last_change":"2024-06-29T14:21:55.291254+0000","last_active":"2024-06-29T14:22:00.305108+0000","last_peered":"2024-06-29T14:22:00.305108+0000","last_clean":"2024-06-29T14:22:00.305108+0000","last_became_active":"2024-06-29T14:21:55.290924+0000","last_became_peered":"2024-06-29T14:21:55.290924+0000","last_unstale":"2024-06-29T14:22:00.305108+0000","last_undegraded":"2024-06-29T14:22:00.305108+0000","last_fullsized":"2024-06-29T14:22:00.305108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:11.308698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640426+0000","last_change":"2024-06-29T14:21:55.280475+0000","last_active":"2024-06-29T14:22:00.640426+0000","last_peered":"2024-06-29T14:22:00.640426+0000","last_clean":"2024-06-29T14:22:00.640426+0000","last_became_active":"2024-06-29T14:21:55.277502+0000","last_became_peered":"2024-06-29T14:21:55.277502+0000","last_unstale":"2024-06-29T14:22:00.640426+0000","last_undegraded":"2024-06-29T14:22:00.640426+0000","last_fullsized":"2024-06-29T14:22:00.640426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:11.801869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640065+0000","last_change":"2024-06-29T14:21:55.285270+0000","last_active":"2024-06-29T14:22:00.640065+0000","last_peered":"2024-06-29T14:22:00.640065+0000","last_clean":"2024-06-29T14:22:00.640065+0000","last_became_active":"2024-06-29T14:21:55.284164+0000","last_became_peered":"2024-06-29T14:21:55.284164+0000","last_unstale":"2024-06-29T14:22:00.640065+0000","last_undegraded":"2024-06-29T14:22:00.640065+0000","last_fullsized":"2024-06-29T14:22:00.640065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:25.523198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640375+0000","last_change":"2024-06-29T14:21:55.287457+0000","last_active":"2024-06-29T14:22:00.640375+0000","last_peered":"2024-06-29T14:22:00.640375+0000","last_clean":"2024-06-29T14:22:00.640375+0000","last_became_active":"2024-06-29T14:21:55.287070+0000","last_became_peered":"2024-06-29T14:21:55.287070+0000","last_unstale":"2024-06-29T14:22:00.640375+0000","last_undegraded":"2024-06-29T14:22:00.640375+0000","last_fullsized":"2024-06-29T14:22:00.640375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305575+0000","last_change":"2024-06-29T14:21:55.298926+0000","last_active":"2024-06-29T14:22:00.305575+0000","last_peered":"2024-06-29T14:22:00.305575+0000","last_clean":"2024-06-29T14:22:00.305575+0000","last_became_active":"2024-06-29T14:21:55.298615+0000","last_became_peered":"2024-06-29T14:21:55.298615+0000","last_unstale":"2024-06-29T14:22:00.305575+0000","last_undegraded":"2024-06-29T14:22:00.305575+0000","last_fullsized":"2024-06-29T14:22:00.305575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641774+0000","last_change":"2024-06-29T14:21:55.290890+0000","last_active":"2024-06-29T14:22:00.641774+0000","last_peered":"2024-06-29T14:22:00.641774+0000","last_clean":"2024-06-29T14:22:00.641774+0000","last_became_active":"2024-06-29T14:21:55.290069+0000","last_became_peered":"2024-06-29T14:21:55.290069+0000","last_unstale":"2024-06-29T14:22:00.641774+0000","last_undegraded":"2024-06-29T14:22:00.641774+0000","last_fullsized":"2024-06-29T14:22:00.641774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306202+0000","last_change":"2024-06-29T14:21:55.277969+0000","last_active":"2024-06-29T14:22:00.306202+0000","last_peered":"2024-06-29T14:22:00.306202+0000","last_clean":"2024-06-29T14:22:00.306202+0000","last_became_active":"2024-06-29T14:21:55.277629+0000","last_became_peered":"2024-06-29T14:21:55.277629+0000","last_unstale":"2024-06-29T14:22:00.306202+0000","last_undegraded":"2024-06-29T14:22:00.306202+0000","last_fullsized":"2024-06-29T14:22:00.306202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:00.388193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305827+0000","last_change":"2024-06-29T14:21:55.297394+0000","last_active":"2024-06-29T14:22:00.305827+0000","last_peered":"2024-06-29T14:22:00.305827+0000","last_clean":"2024-06-29T14:22:00.305827+0000","last_became_active":"2024-06-29T14:21:55.297087+0000","last_became_peered":"2024-06-29T14:21:55.297087+0000","last_unstale":"2024-06-29T14:22:00.305827+0000","last_undegraded":"2024-06-29T14:22:00.305827+0000","last_fullsized":"2024-06-29T14:22:00.305827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:43.242914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639702+0000","last_change":"2024-06-29T14:21:55.284207+0000","last_active":"2024-06-29T14:22:00.639702+0000","last_peered":"2024-06-29T14:22:00.639702+0000","last_clean":"2024-06-29T14:22:00.639702+0000","last_became_active":"2024-06-29T14:21:55.283869+0000","last_became_peered":"2024-06-29T14:21:55.283869+0000","last_unstale":"2024-06-29T14:22:00.639702+0000","last_undegraded":"2024-06-29T14:22:00.639702+0000","last_fullsized":"2024-06-29T14:22:00.639702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304555+0000","last_change":"2024-06-29T14:21:55.298136+0000","last_active":"2024-06-29T14:22:00.304555+0000","last_peered":"2024-06-29T14:22:00.304555+0000","last_clean":"2024-06-29T14:22:00.304555+0000","last_became_active":"2024-06-29T14:21:55.297852+0000","last_became_peered":"2024-06-29T14:21:55.297852+0000","last_unstale":"2024-06-29T14:22:00.304555+0000","last_undegraded":"2024-06-29T14:22:00.304555+0000","last_fullsized":"2024-06-29T14:22:00.304555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640027+0000","last_change":"2024-06-29T14:21:55.291932+0000","last_active":"2024-06-29T14:22:00.640027+0000","last_peered":"2024-06-29T14:22:00.640027+0000","last_clean":"2024-06-29T14:22:00.640027+0000","last_became_active":"2024-06-29T14:21:55.291605+0000","last_became_peered":"2024-06-29T14:21:55.291605+0000","last_unstale":"2024-06-29T14:22:00.640027+0000","last_undegraded":"2024-06-29T14:22:00.640027+0000","last_fullsized":"2024-06-29T14:22:00.640027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.505918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640145+0000","last_change":"2024-06-29T14:21:55.296088+0000","last_active":"2024-06-29T14:22:00.640145+0000","last_peered":"2024-06-29T14:22:00.640145+0000","last_clean":"2024-06-29T14:22:00.640145+0000","last_became_active":"2024-06-29T14:21:55.295752+0000","last_became_peered":"2024-06-29T14:21:55.295752+0000","last_unstale":"2024-06-29T14:22:00.640145+0000","last_undegraded":"2024-06-29T14:22:00.640145+0000","last_fullsized":"2024-06-29T14:22:00.640145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306344+0000","last_change":"2024-06-29T14:21:55.293717+0000","last_active":"2024-06-29T14:22:00.306344+0000","last_peered":"2024-06-29T14:22:00.306344+0000","last_clean":"2024-06-29T14:22:00.306344+0000","last_became_active":"2024-06-29T14:21:55.293421+0000","last_became_peered":"2024-06-29T14:21:55.293421+0000","last_unstale":"2024-06-29T14:22:00.306344+0000","last_undegraded":"2024-06-29T14:22:00.306344+0000","last_fullsized":"2024-06-29T14:22:00.306344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:08:58.083131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641151+0000","last_change":"2024-06-29T14:21:55.288925+0000","last_active":"2024-06-29T14:22:00.641151+0000","last_peered":"2024-06-29T14:22:00.641151+0000","last_clean":"2024-06-29T14:22:00.641151+0000","last_became_active":"2024-06-29T14:21:55.288585+0000","last_became_peered":"2024-06-29T14:21:55.288585+0000","last_unstale":"2024-06-29T14:22:00.641151+0000","last_undegraded":"2024-06-29T14:22:00.641151+0000","last_fullsized":"2024-06-29T14:22:00.641151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:35.844291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641027+0000","last_change":"2024-06-29T14:21:55.271843+0000","last_active":"2024-06-29T14:22:00.641027+0000","last_peered":"2024-06-29T14:22:00.641027+0000","last_clean":"2024-06-29T14:22:00.641027+0000","last_became_active":"2024-06-29T14:21:55.271494+0000","last_became_peered":"2024-06-29T14:21:55.271494+0000","last_unstale":"2024-06-29T14:22:00.641027+0000","last_undegraded":"2024-06-29T14:22:00.641027+0000","last_fullsized":"2024-06-29T14:22:00.641027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:29.960047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640632+0000","last_change":"2024-06-29T14:21:55.278149+0000","last_active":"2024-06-29T14:22:00.640632+0000","last_peered":"2024-06-29T14:22:00.640632+0000","last_clean":"2024-06-29T14:22:00.640632+0000","last_became_active":"2024-06-29T14:21:55.277816+0000","last_became_peered":"2024-06-29T14:21:55.277816+0000","last_unstale":"2024-06-29T14:22:00.640632+0000","last_undegraded":"2024-06-29T14:22:00.640632+0000","last_fullsized":"2024-06-29T14:22:00.640632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:14:18.590915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639577+0000","last_change":"2024-06-29T14:21:55.296366+0000","last_active":"2024-06-29T14:22:00.639577+0000","last_peered":"2024-06-29T14:22:00.639577+0000","last_clean":"2024-06-29T14:22:00.639577+0000","last_became_active":"2024-06-29T14:21:55.296040+0000","last_became_peered":"2024-06-29T14:21:55.296040+0000","last_unstale":"2024-06-29T14:22:00.639577+0000","last_undegraded":"2024-06-29T14:22:00.639577+0000","last_fullsized":"2024-06-29T14:22:00.639577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639931+0000","last_change":"2024-06-29T14:21:55.291183+0000","last_active":"2024-06-29T14:22:00.639931+0000","last_peered":"2024-06-29T14:22:00.639931+0000","last_clean":"2024-06-29T14:22:00.639931+0000","last_became_active":"2024-06-29T14:21:55.289884+0000","last_became_peered":"2024-06-29T14:21:55.289884+0000","last_unstale":"2024-06-29T14:22:00.639931+0000","last_undegraded":"2024-06-29T14:22:00.639931+0000","last_fullsized":"2024-06-29T14:22:00.639931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640038+0000","last_change":"2024-06-29T14:21:55.293964+0000","last_active":"2024-06-29T14:22:00.640038+0000","last_peered":"2024-06-29T14:22:00.640038+0000","last_clean":"2024-06-29T14:22:00.640038+0000","last_became_active":"2024-06-29T14:21:55.291885+0000","last_became_peered":"2024-06-29T14:21:55.291885+0000","last_unstale":"2024-06-29T14:22:00.640038+0000","last_undegraded":"2024-06-29T14:22:00.640038+0000","last_fullsized":"2024-06-29T14:22:00.640038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:20.427848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304827+0000","last_change":"2024-06-29T14:21:55.288765+0000","last_active":"2024-06-29T14:22:00.304827+0000","last_peered":"2024-06-29T14:22:00.304827+0000","last_clean":"2024-06-29T14:22:00.304827+0000","last_became_active":"2024-06-29T14:21:55.288452+0000","last_became_peered":"2024-06-29T14:21:55.288452+0000","last_unstale":"2024-06-29T14:22:00.304827+0000","last_undegraded":"2024-06-29T14:22:00.304827+0000","last_fullsized":"2024-06-29T14:22:00.304827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:30.912272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307365+0000","last_change":"2024-06-29T14:21:55.295376+0000","last_active":"2024-06-29T14:22:00.307365+0000","last_peered":"2024-06-29T14:22:00.307365+0000","last_clean":"2024-06-29T14:22:00.307365+0000","last_became_active":"2024-06-29T14:21:55.294928+0000","last_became_peered":"2024-06-29T14:21:55.294928+0000","last_unstale":"2024-06-29T14:22:00.307365+0000","last_undegraded":"2024-06-29T14:22:00.307365+0000","last_fullsized":"2024-06-29T14:22:00.307365+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:28:42.614313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305773+0000","last_change":"2024-06-29T14:21:55.281068+0000","last_active":"2024-06-29T14:22:00.305773+0000","last_peered":"2024-06-29T14:22:00.305773+0000","last_clean":"2024-06-29T14:22:00.305773+0000","last_became_active":"2024-06-29T14:21:55.280780+0000","last_became_peered":"2024-06-29T14:21:55.280780+0000","last_unstale":"2024-06-29T14:22:00.305773+0000","last_undegraded":"2024-06-29T14:22:00.305773+0000","last_fullsized":"2024-06-29T14:22:00.305773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:36:38.990506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306085+0000","last_change":"2024-06-29T14:21:55.280506+0000","last_active":"2024-06-29T14:22:00.306085+0000","last_peered":"2024-06-29T14:22:00.306085+0000","last_clean":"2024-06-29T14:22:00.306085+0000","last_became_active":"2024-06-29T14:21:55.280221+0000","last_became_peered":"2024-06-29T14:21:55.280221+0000","last_unstale":"2024-06-29T14:22:00.306085+0000","last_undegraded":"2024-06-29T14:22:00.306085+0000","last_fullsized":"2024-06-29T14:22:00.306085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:41.028472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641311+0000","last_change":"2024-06-29T14:21:55.287649+0000","last_active":"2024-06-29T14:22:00.641311+0000","last_peered":"2024-06-29T14:22:00.641311+0000","last_clean":"2024-06-29T14:22:00.641311+0000","last_became_active":"2024-06-29T14:21:55.286171+0000","last_became_peered":"2024-06-29T14:21:55.286171+0000","last_unstale":"2024-06-29T14:22:00.641311+0000","last_undegraded":"2024-06-29T14:22:00.641311+0000","last_fullsized":"2024-06-29T14:22:00.641311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:30.219397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304571+0000","last_change":"2024-06-29T14:21:55.285603+0000","last_active":"2024-06-29T14:22:00.304571+0000","last_peered":"2024-06-29T14:22:00.304571+0000","last_clean":"2024-06-29T14:22:00.304571+0000","last_became_active":"2024-06-29T14:21:55.285284+0000","last_became_peered":"2024-06-29T14:21:55.285284+0000","last_unstale":"2024-06-29T14:22:00.304571+0000","last_undegraded":"2024-06-29T14:22:00.304571+0000","last_fullsized":"2024-06-29T14:22:00.304571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:44.658689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640973+0000","last_change":"2024-06-29T14:21:55.287794+0000","last_active":"2024-06-29T14:22:00.640973+0000","last_peered":"2024-06-29T14:22:00.640973+0000","last_clean":"2024-06-29T14:22:00.640973+0000","last_became_active":"2024-06-29T14:21:55.287028+0000","last_became_peered":"2024-06-29T14:21:55.287028+0000","last_unstale":"2024-06-29T14:22:00.640973+0000","last_undegraded":"2024-06-29T14:22:00.640973+0000","last_fullsized":"2024-06-29T14:22:00.640973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:39.026791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639887+0000","last_change":"2024-06-29T14:21:55.289581+0000","last_active":"2024-06-29T14:22:00.639887+0000","last_peered":"2024-06-29T14:22:00.639887+0000","last_clean":"2024-06-29T14:22:00.639887+0000","last_became_active":"2024-06-29T14:21:55.289097+0000","last_became_peered":"2024-06-29T14:21:55.289097+0000","last_unstale":"2024-06-29T14:22:00.639887+0000","last_undegraded":"2024-06-29T14:22:00.639887+0000","last_fullsized":"2024-06-29T14:22:00.639887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:43.242914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640505+0000","last_change":"2024-06-29T14:21:55.288463+0000","last_active":"2024-06-29T14:22:00.640505+0000","last_peered":"2024-06-29T14:22:00.640505+0000","last_clean":"2024-06-29T14:22:00.640505+0000","last_became_active":"2024-06-29T14:21:55.288128+0000","last_became_peered":"2024-06-29T14:21:55.288128+0000","last_unstale":"2024-06-29T14:22:00.640505+0000","last_undegraded":"2024-06-29T14:22:00.640505+0000","last_fullsized":"2024-06-29T14:22:00.640505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:28.573747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640255+0000","last_change":"2024-06-29T14:21:55.290100+0000","last_active":"2024-06-29T14:22:00.640255+0000","last_peered":"2024-06-29T14:22:00.640255+0000","last_clean":"2024-06-29T14:22:00.640255+0000","last_became_active":"2024-06-29T14:21:55.289525+0000","last_became_peered":"2024-06-29T14:21:55.289525+0000","last_unstale":"2024-06-29T14:22:00.640255+0000","last_undegraded":"2024-06-29T14:22:00.640255+0000","last_fullsized":"2024-06-29T14:22:00.640255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:22.343517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641269+0000","last_change":"2024-06-29T14:21:55.287921+0000","last_active":"2024-06-29T14:22:00.641269+0000","last_peered":"2024-06-29T14:22:00.641269+0000","last_clean":"2024-06-29T14:22:00.641269+0000","last_became_active":"2024-06-29T14:21:55.287383+0000","last_became_peered":"2024-06-29T14:21:55.287383+0000","last_unstale":"2024-06-29T14:22:00.641269+0000","last_undegraded":"2024-06-29T14:22:00.641269+0000","last_fullsized":"2024-06-29T14:22:00.641269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:05.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641908+0000","last_change":"2024-06-29T14:21:55.285210+0000","last_active":"2024-06-29T14:22:00.641908+0000","last_peered":"2024-06-29T14:22:00.641908+0000","last_clean":"2024-06-29T14:22:00.641908+0000","last_became_active":"2024-06-29T14:21:55.284857+0000","last_became_peered":"2024-06-29T14:21:55.284857+0000","last_unstale":"2024-06-29T14:22:00.641908+0000","last_undegraded":"2024-06-29T14:22:00.641908+0000","last_fullsized":"2024-06-29T14:22:00.641908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641229+0000","last_change":"2024-06-29T14:21:55.287889+0000","last_active":"2024-06-29T14:22:00.641229+0000","last_peered":"2024-06-29T14:22:00.641229+0000","last_clean":"2024-06-29T14:22:00.641229+0000","last_became_active":"2024-06-29T14:21:55.286286+0000","last_became_peered":"2024-06-29T14:21:55.286286+0000","last_unstale":"2024-06-29T14:22:00.641229+0000","last_undegraded":"2024-06-29T14:22:00.641229+0000","last_fullsized":"2024-06-29T14:22:00.641229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:23.065793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":111149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":698454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":868352,"data_stored":698398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":111290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":107574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:22:13.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:22:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.764+0000 7f38d46b2640 1 -- 172.21.15.59:0/1771010093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80053f0 msgr2=0x7f38b800ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.764+0000 7f38d46b2640 1 --2- 172.21.15.59:0/1771010093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80053f0 0x7f38b800ba70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0d01f0 tx=0x7f38b8005830 comp rx=0 tx=0).stop 2024-06-29T14:22:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.764+0000 7f38d46b2640 1 -- 172.21.15.59:0/1771010093 shutdown_connections 2024-06-29T14:22:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.764+0000 7f38d46b2640 1 --2- 172.21.15.59:0/1771010093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80053f0 0x7f38b800ba70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.764+0000 7f38d46b2640 1 -- 172.21.15.59:0/1771010093 >> 172.21.15.59:0/1771010093 conn(0x7f38cc0b5610 msgr2=0x7f38cc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.764+0000 7f38d46b2640 1 -- 172.21.15.59:0/1771010093 shutdown_connections 2024-06-29T14:22:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.764+0000 7f38d46b2640 1 -- 172.21.15.59:0/1771010093 wait complete. 2024-06-29T14:22:13.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.766+0000 7f38d46b2640 1 Processor -- start 2024-06-29T14:22:13.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.766+0000 7f38d46b2640 1 -- start start 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d46b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38cc0abc90 0x7f38cc0aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d46b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38cc0cd740 0x7f38cc0cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d46b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc0cb110 0x7f38cc0cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d46b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc162d90 con 0x7f38cc0cb110 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d46b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc162c10 con 0x7f38cc0cd740 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d46b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f38cc162f10 con 0x7f38cc0abc90 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d2c4f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc0cb110 0x7f38cc0cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d2c4f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc0cb110 0x7f38cc0cb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51598/0 (socket says 172.21.15.59:51598) 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d2c4f640 1 -- 172.21.15.59:0/3050951750 learned_addr learned my addr 172.21.15.59:0/3050951750 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:13.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.767+0000 7f38d244e640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38cc0abc90 0x7f38cc0aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d1c4d640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38cc0cd740 0x7f38cc0cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d2c4f640 1 -- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38cc0abc90 msgr2=0x7f38cc0aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d2c4f640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38cc0abc90 0x7f38cc0aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d2c4f640 1 -- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38cc0cd740 msgr2=0x7f38cc0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d2c4f640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38cc0cd740 0x7f38cc0cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d2c4f640 1 -- 172.21.15.59:0/3050951750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc0ac0a0 con 0x7f38cc0cb110 2024-06-29T14:22:13.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d2c4f640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc0cb110 0x7f38cc0cb520 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f38c4001be0 tx=0x7f38c4003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:13.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38c2ffd640 1 -- 172.21.15.59:0/3050951750 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4039070 con 0x7f38cc0cb110 2024-06-29T14:22:13.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38c2ffd640 1 -- 172.21.15.59:0/3050951750 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38c402bab0 con 0x7f38cc0cb110 2024-06-29T14:22:13.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38cc0aabf0 con 0x7f38cc0cb110 2024-06-29T14:22:13.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38c2ffd640 1 -- 172.21.15.59:0/3050951750 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4002080 con 0x7f38cc0cb110 2024-06-29T14:22:13.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.768+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38cc0ab1b0 con 0x7f38cc0cb110 2024-06-29T14:22:13.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.769+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38cc0cf240 con 0x7f38cc0cb110 2024-06-29T14:22:13.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.770+0000 7f38c2ffd640 1 -- 172.21.15.59:0/3050951750 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f38c4006140 con 0x7f38cc0cb110 2024-06-29T14:22:13.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.770+0000 7f38c2ffd640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f38b00762d0 0x7f38b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:13.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.770+0000 7f38c2ffd640 1 -- 172.21.15.59:0/3050951750 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f38c40b75d0 con 0x7f38cc0cb110 2024-06-29T14:22:13.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.771+0000 7f38d244e640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f38b00762d0 0x7f38b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:13.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.771+0000 7f38d244e640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f38b00762d0 0x7f38b00787e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0688b0 tx=0x7f38c8006b70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:13.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.775+0000 7f38c2ffd640 1 -- 172.21.15.59:0/3050951750 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f38c4080bf0 con 0x7f38cc0cb110 2024-06-29T14:22:13.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.962+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f38cc0c7780 con 0x7f38b00762d0 2024-06-29T14:22:13.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.968+0000 7f38c2ffd640 1 -- 172.21.15.59:0/3050951750 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7f38cc0c7780 con 0x7f38b00762d0 2024-06-29T14:22:13.970 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:13.972 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f38b00762d0 msgr2=0x7f38b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f38b00762d0 0x7f38b00787e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0688b0 tx=0x7f38c8006b70 comp rx=0 tx=0).stop 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc0cb110 msgr2=0x7f38cc0cb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc0cb110 0x7f38cc0cb520 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f38c4001be0 tx=0x7f38c4003b80 comp rx=0 tx=0).stop 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 shutdown_connections 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f38b00762d0 0x7f38b00787e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38cc0cb110 0x7f38cc0cb520 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38cc0cd740 0x7f38cc0cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 --2- 172.21.15.59:0/3050951750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38cc0abc90 0x7f38cc0aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 >> 172.21.15.59:0/3050951750 conn(0x7f38cc0b5610 msgr2=0x7f38cc06da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 shutdown_connections 2024-06-29T14:22:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:13.972+0000 7f38d46b2640 1 -- 172.21.15.59:0/3050951750 wait complete. 2024-06-29T14:22:13.981 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-06-29T14:22:13.735307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219504,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755216,"statfs":{"total":773094113280,"available":772869341184,"internally_reserved":0,"allocated":3416064,"data_stored":2049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003643"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640964+0000","last_change":"2024-06-29T14:21:54.260091+0000","last_active":"2024-06-29T14:22:00.640964+0000","last_peered":"2024-06-29T14:22:00.640964+0000","last_clean":"2024-06-29T14:22:00.640964+0000","last_became_active":"2024-06-29T14:21:54.259733+0000","last_became_peered":"2024-06-29T14:21:54.259733+0000","last_unstale":"2024-06-29T14:22:00.640964+0000","last_undegraded":"2024-06-29T14:22:00.640964+0000","last_fullsized":"2024-06-29T14:22:00.640964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:31.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305128+0000","last_change":"2024-06-29T14:21:55.286512+0000","last_active":"2024-06-29T14:22:00.305128+0000","last_peered":"2024-06-29T14:22:00.305128+0000","last_clean":"2024-06-29T14:22:00.305128+0000","last_became_active":"2024-06-29T14:21:55.286126+0000","last_became_peered":"2024-06-29T14:21:55.286126+0000","last_unstale":"2024-06-29T14:22:00.305128+0000","last_undegraded":"2024-06-29T14:22:00.305128+0000","last_fullsized":"2024-06-29T14:22:00.305128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304598+0000","last_change":"2024-06-29T14:21:59.338143+0000","last_active":"2024-06-29T14:22:00.304598+0000","last_peered":"2024-06-29T14:22:00.304598+0000","last_clean":"2024-06-29T14:22:00.304598+0000","last_became_active":"2024-06-29T14:21:59.337328+0000","last_became_peered":"2024-06-29T14:21:59.337328+0000","last_unstale":"2024-06-29T14:22:00.304598+0000","last_undegraded":"2024-06-29T14:22:00.304598+0000","last_fullsized":"2024-06-29T14:22:00.304598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:23.956178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305340+0000","last_change":"2024-06-29T14:21:54.257884+0000","last_active":"2024-06-29T14:22:00.305340+0000","last_peered":"2024-06-29T14:22:00.305340+0000","last_clean":"2024-06-29T14:22:00.305340+0000","last_became_active":"2024-06-29T14:21:54.257540+0000","last_became_peered":"2024-06-29T14:21:54.257540+0000","last_unstale":"2024-06-29T14:22:00.305340+0000","last_undegraded":"2024-06-29T14:22:00.305340+0000","last_fullsized":"2024-06-29T14:22:00.305340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:54.713087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640589+0000","last_change":"2024-06-29T14:21:55.285781+0000","last_active":"2024-06-29T14:22:00.640589+0000","last_peered":"2024-06-29T14:22:00.640589+0000","last_clean":"2024-06-29T14:22:00.640589+0000","last_became_active":"2024-06-29T14:21:55.285358+0000","last_became_peered":"2024-06-29T14:21:55.285358+0000","last_unstale":"2024-06-29T14:22:00.640589+0000","last_undegraded":"2024-06-29T14:22:00.640589+0000","last_fullsized":"2024-06-29T14:22:00.640589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306447+0000","last_change":"2024-06-29T14:21:59.318024+0000","last_active":"2024-06-29T14:22:00.306447+0000","last_peered":"2024-06-29T14:22:00.306447+0000","last_clean":"2024-06-29T14:22:00.306447+0000","last_became_active":"2024-06-29T14:21:59.317356+0000","last_became_peered":"2024-06-29T14:21:59.317356+0000","last_unstale":"2024-06-29T14:22:00.306447+0000","last_undegraded":"2024-06-29T14:22:00.306447+0000","last_fullsized":"2024-06-29T14:22:00.306447+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:00.209167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640906+0000","last_change":"2024-06-29T14:21:54.258802+0000","last_active":"2024-06-29T14:22:00.640906+0000","last_peered":"2024-06-29T14:22:00.640906+0000","last_clean":"2024-06-29T14:22:00.640906+0000","last_became_active":"2024-06-29T14:21:54.258184+0000","last_became_peered":"2024-06-29T14:21:54.258184+0000","last_unstale":"2024-06-29T14:22:00.640906+0000","last_undegraded":"2024-06-29T14:22:00.640906+0000","last_fullsized":"2024-06-29T14:22:00.640906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:07:56.943579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640388+0000","last_change":"2024-06-29T14:21:55.296893+0000","last_active":"2024-06-29T14:22:00.640388+0000","last_peered":"2024-06-29T14:22:00.640388+0000","last_clean":"2024-06-29T14:22:00.640388+0000","last_became_active":"2024-06-29T14:21:55.296552+0000","last_became_peered":"2024-06-29T14:21:55.296552+0000","last_unstale":"2024-06-29T14:22:00.640388+0000","last_undegraded":"2024-06-29T14:22:00.640388+0000","last_fullsized":"2024-06-29T14:22:00.640388+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:57.029702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307221+0000","last_change":"2024-06-29T14:21:59.319287+0000","last_active":"2024-06-29T14:22:00.307221+0000","last_peered":"2024-06-29T14:22:00.307221+0000","last_clean":"2024-06-29T14:22:00.307221+0000","last_became_active":"2024-06-29T14:21:59.318807+0000","last_became_peered":"2024-06-29T14:21:59.318807+0000","last_unstale":"2024-06-29T14:22:00.307221+0000","last_undegraded":"2024-06-29T14:22:00.307221+0000","last_fullsized":"2024-06-29T14:22:00.307221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:20.883221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640313+0000","last_change":"2024-06-29T14:21:54.264911+0000","last_active":"2024-06-29T14:22:00.640313+0000","last_peered":"2024-06-29T14:22:00.640313+0000","last_clean":"2024-06-29T14:22:00.640313+0000","last_became_active":"2024-06-29T14:21:54.264532+0000","last_became_peered":"2024-06-29T14:21:54.264532+0000","last_unstale":"2024-06-29T14:22:00.640313+0000","last_undegraded":"2024-06-29T14:22:00.640313+0000","last_fullsized":"2024-06-29T14:22:00.640313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:10.799778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640442+0000","last_change":"2024-06-29T14:21:55.295115+0000","last_active":"2024-06-29T14:22:00.640442+0000","last_peered":"2024-06-29T14:22:00.640442+0000","last_clean":"2024-06-29T14:22:00.640442+0000","last_became_active":"2024-06-29T14:21:55.294772+0000","last_became_peered":"2024-06-29T14:21:55.294772+0000","last_unstale":"2024-06-29T14:22:00.640442+0000","last_undegraded":"2024-06-29T14:22:00.640442+0000","last_fullsized":"2024-06-29T14:22:00.640442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:17.149243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307463+0000","last_change":"2024-06-29T14:21:59.312924+0000","last_active":"2024-06-29T14:22:00.307463+0000","last_peered":"2024-06-29T14:22:00.307463+0000","last_clean":"2024-06-29T14:22:00.307463+0000","last_became_active":"2024-06-29T14:21:59.312393+0000","last_became_peered":"2024-06-29T14:21:59.312393+0000","last_unstale":"2024-06-29T14:22:00.307463+0000","last_undegraded":"2024-06-29T14:22:00.307463+0000","last_fullsized":"2024-06-29T14:22:00.307463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:32.630422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305149+0000","last_change":"2024-06-29T14:21:55.294165+0000","last_active":"2024-06-29T14:22:00.305149+0000","last_peered":"2024-06-29T14:22:00.305149+0000","last_clean":"2024-06-29T14:22:00.305149+0000","last_became_active":"2024-06-29T14:21:55.293388+0000","last_became_peered":"2024-06-29T14:21:55.293388+0000","last_unstale":"2024-06-29T14:22:00.305149+0000","last_undegraded":"2024-06-29T14:22:00.305149+0000","last_fullsized":"2024-06-29T14:22:00.305149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:50:36.883008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641127+0000","last_change":"2024-06-29T14:21:54.256784+0000","last_active":"2024-06-29T14:22:00.641127+0000","last_peered":"2024-06-29T14:22:00.641127+0000","last_clean":"2024-06-29T14:22:00.641127+0000","last_became_active":"2024-06-29T14:21:54.256434+0000","last_became_peered":"2024-06-29T14:21:54.256434+0000","last_unstale":"2024-06-29T14:22:00.641127+0000","last_undegraded":"2024-06-29T14:22:00.641127+0000","last_fullsized":"2024-06-29T14:22:00.641127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:06.963975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306723+0000","last_change":"2024-06-29T14:21:59.317621+0000","last_active":"2024-06-29T14:22:00.306723+0000","last_peered":"2024-06-29T14:22:00.306723+0000","last_clean":"2024-06-29T14:22:00.306723+0000","last_became_active":"2024-06-29T14:21:59.316736+0000","last_became_peered":"2024-06-29T14:21:59.316736+0000","last_unstale":"2024-06-29T14:22:00.306723+0000","last_undegraded":"2024-06-29T14:22:00.306723+0000","last_fullsized":"2024-06-29T14:22:00.306723+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:43.083466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304951+0000","last_change":"2024-06-29T14:21:55.296036+0000","last_active":"2024-06-29T14:22:00.304951+0000","last_peered":"2024-06-29T14:22:00.304951+0000","last_clean":"2024-06-29T14:22:00.304951+0000","last_became_active":"2024-06-29T14:21:55.294958+0000","last_became_peered":"2024-06-29T14:21:55.294958+0000","last_unstale":"2024-06-29T14:22:00.304951+0000","last_undegraded":"2024-06-29T14:22:00.304951+0000","last_fullsized":"2024-06-29T14:22:00.304951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:25.523198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641393+0000","last_change":"2024-06-29T14:21:54.249131+0000","last_active":"2024-06-29T14:22:00.641393+0000","last_peered":"2024-06-29T14:22:00.641393+0000","last_clean":"2024-06-29T14:22:00.641393+0000","last_became_active":"2024-06-29T14:21:54.248727+0000","last_became_peered":"2024-06-29T14:21:54.248727+0000","last_unstale":"2024-06-29T14:22:00.641393+0000","last_undegraded":"2024-06-29T14:22:00.641393+0000","last_fullsized":"2024-06-29T14:22:00.641393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:43.007566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306263+0000","last_change":"2024-06-29T14:21:59.318902+0000","last_active":"2024-06-29T14:22:00.306263+0000","last_peered":"2024-06-29T14:22:00.306263+0000","last_clean":"2024-06-29T14:22:00.306263+0000","last_became_active":"2024-06-29T14:21:59.318388+0000","last_became_peered":"2024-06-29T14:21:59.318388+0000","last_unstale":"2024-06-29T14:22:00.306263+0000","last_undegraded":"2024-06-29T14:22:00.306263+0000","last_fullsized":"2024-06-29T14:22:00.306263+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:13.544231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305926+0000","last_change":"2024-06-29T14:21:55.282042+0000","last_active":"2024-06-29T14:22:00.305926+0000","last_peered":"2024-06-29T14:22:00.305926+0000","last_clean":"2024-06-29T14:22:00.305926+0000","last_became_active":"2024-06-29T14:21:55.281259+0000","last_became_peered":"2024-06-29T14:21:55.281259+0000","last_unstale":"2024-06-29T14:22:00.305926+0000","last_undegraded":"2024-06-29T14:22:00.305926+0000","last_fullsized":"2024-06-29T14:22:00.305926+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640041+0000","last_change":"2024-06-29T14:21:54.255093+0000","last_active":"2024-06-29T14:22:00.640041+0000","last_peered":"2024-06-29T14:22:00.640041+0000","last_clean":"2024-06-29T14:22:00.640041+0000","last_became_active":"2024-06-29T14:21:54.254010+0000","last_became_peered":"2024-06-29T14:21:54.254010+0000","last_unstale":"2024-06-29T14:22:00.640041+0000","last_undegraded":"2024-06-29T14:22:00.640041+0000","last_fullsized":"2024-06-29T14:22:00.640041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:24.835712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.642067+0000","last_change":"2024-06-29T14:21:59.337086+0000","last_active":"2024-06-29T14:22:00.642067+0000","last_peered":"2024-06-29T14:22:00.642067+0000","last_clean":"2024-06-29T14:22:00.642067+0000","last_became_active":"2024-06-29T14:21:59.336659+0000","last_became_peered":"2024-06-29T14:21:59.336659+0000","last_unstale":"2024-06-29T14:22:00.642067+0000","last_undegraded":"2024-06-29T14:22:00.642067+0000","last_fullsized":"2024-06-29T14:22:00.642067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:47.549181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306644+0000","last_change":"2024-06-29T14:21:55.289199+0000","last_active":"2024-06-29T14:22:00.306644+0000","last_peered":"2024-06-29T14:22:00.306644+0000","last_clean":"2024-06-29T14:22:00.306644+0000","last_became_active":"2024-06-29T14:21:55.288787+0000","last_became_peered":"2024-06-29T14:21:55.288787+0000","last_unstale":"2024-06-29T14:22:00.306644+0000","last_undegraded":"2024-06-29T14:22:00.306644+0000","last_fullsized":"2024-06-29T14:22:00.306644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:33.390422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640397+0000","last_change":"2024-06-29T14:21:54.260787+0000","last_active":"2024-06-29T14:22:00.640397+0000","last_peered":"2024-06-29T14:22:00.640397+0000","last_clean":"2024-06-29T14:22:00.640397+0000","last_became_active":"2024-06-29T14:21:54.260352+0000","last_became_peered":"2024-06-29T14:21:54.260352+0000","last_unstale":"2024-06-29T14:22:00.640397+0000","last_undegraded":"2024-06-29T14:22:00.640397+0000","last_fullsized":"2024-06-29T14:22:00.640397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:40.455851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305896+0000","last_change":"2024-06-29T14:21:59.316811+0000","last_active":"2024-06-29T14:22:00.305896+0000","last_peered":"2024-06-29T14:22:00.305896+0000","last_clean":"2024-06-29T14:22:00.305896+0000","last_became_active":"2024-06-29T14:21:59.316067+0000","last_became_peered":"2024-06-29T14:21:59.316067+0000","last_unstale":"2024-06-29T14:22:00.305896+0000","last_undegraded":"2024-06-29T14:22:00.305896+0000","last_fullsized":"2024-06-29T14:22:00.305896+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:04.444868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640138+0000","last_change":"2024-06-29T14:21:55.293209+0000","last_active":"2024-06-29T14:22:00.640138+0000","last_peered":"2024-06-29T14:22:00.640138+0000","last_clean":"2024-06-29T14:22:00.640138+0000","last_became_active":"2024-06-29T14:21:55.291574+0000","last_became_peered":"2024-06-29T14:21:55.291574+0000","last_unstale":"2024-06-29T14:22:00.640138+0000","last_undegraded":"2024-06-29T14:22:00.640138+0000","last_fullsized":"2024-06-29T14:22:00.640138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:16.165478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305885+0000","last_change":"2024-06-29T14:21:54.256099+0000","last_active":"2024-06-29T14:22:00.305885+0000","last_peered":"2024-06-29T14:22:00.305885+0000","last_clean":"2024-06-29T14:22:00.305885+0000","last_became_active":"2024-06-29T14:21:54.254022+0000","last_became_peered":"2024-06-29T14:21:54.254022+0000","last_unstale":"2024-06-29T14:22:00.305885+0000","last_undegraded":"2024-06-29T14:22:00.305885+0000","last_fullsized":"2024-06-29T14:22:00.305885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:32.388331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305919+0000","last_change":"2024-06-29T14:21:59.337078+0000","last_active":"2024-06-29T14:22:00.305919+0000","last_peered":"2024-06-29T14:22:00.305919+0000","last_clean":"2024-06-29T14:22:00.305919+0000","last_became_active":"2024-06-29T14:21:59.336616+0000","last_became_peered":"2024-06-29T14:21:59.336616+0000","last_unstale":"2024-06-29T14:22:00.305919+0000","last_undegraded":"2024-06-29T14:22:00.305919+0000","last_fullsized":"2024-06-29T14:22:00.305919+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:33:53.623576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305915+0000","last_change":"2024-06-29T14:21:55.293645+0000","last_active":"2024-06-29T14:22:00.305915+0000","last_peered":"2024-06-29T14:22:00.305915+0000","last_clean":"2024-06-29T14:22:00.305915+0000","last_became_active":"2024-06-29T14:21:55.291467+0000","last_became_peered":"2024-06-29T14:21:55.291467+0000","last_unstale":"2024-06-29T14:22:00.305915+0000","last_undegraded":"2024-06-29T14:22:00.305915+0000","last_fullsized":"2024-06-29T14:22:00.305915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.505918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641935+0000","last_change":"2024-06-29T14:21:54.244171+0000","last_active":"2024-06-29T14:22:00.641935+0000","last_peered":"2024-06-29T14:22:00.641935+0000","last_clean":"2024-06-29T14:22:00.641935+0000","last_became_active":"2024-06-29T14:21:54.243791+0000","last_became_peered":"2024-06-29T14:21:54.243791+0000","last_unstale":"2024-06-29T14:22:00.641935+0000","last_undegraded":"2024-06-29T14:22:00.641935+0000","last_fullsized":"2024-06-29T14:22:00.641935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:24.194470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304468+0000","last_change":"2024-06-29T14:21:59.336963+0000","last_active":"2024-06-29T14:22:00.304468+0000","last_peered":"2024-06-29T14:22:00.304468+0000","last_clean":"2024-06-29T14:22:00.304468+0000","last_became_active":"2024-06-29T14:21:59.336450+0000","last_became_peered":"2024-06-29T14:21:59.336450+0000","last_unstale":"2024-06-29T14:22:00.304468+0000","last_undegraded":"2024-06-29T14:22:00.304468+0000","last_fullsized":"2024-06-29T14:22:00.304468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:00.347317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641725+0000","last_change":"2024-06-29T14:21:55.271686+0000","last_active":"2024-06-29T14:22:00.641725+0000","last_peered":"2024-06-29T14:22:00.641725+0000","last_clean":"2024-06-29T14:22:00.641725+0000","last_became_active":"2024-06-29T14:21:55.271310+0000","last_became_peered":"2024-06-29T14:21:55.271310+0000","last_unstale":"2024-06-29T14:22:00.641725+0000","last_undegraded":"2024-06-29T14:22:00.641725+0000","last_fullsized":"2024-06-29T14:22:00.641725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641232+0000","last_change":"2024-06-29T14:21:54.260566+0000","last_active":"2024-06-29T14:22:00.641232+0000","last_peered":"2024-06-29T14:22:00.641232+0000","last_clean":"2024-06-29T14:22:00.641232+0000","last_became_active":"2024-06-29T14:21:54.260089+0000","last_became_peered":"2024-06-29T14:21:54.260089+0000","last_unstale":"2024-06-29T14:22:00.641232+0000","last_undegraded":"2024-06-29T14:22:00.641232+0000","last_fullsized":"2024-06-29T14:22:00.641232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:28.957956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640300+0000","last_change":"2024-06-29T14:21:59.338874+0000","last_active":"2024-06-29T14:22:00.640300+0000","last_peered":"2024-06-29T14:22:00.640300+0000","last_clean":"2024-06-29T14:22:00.640300+0000","last_became_active":"2024-06-29T14:21:59.338388+0000","last_became_peered":"2024-06-29T14:21:59.338388+0000","last_unstale":"2024-06-29T14:22:00.640300+0000","last_undegraded":"2024-06-29T14:22:00.640300+0000","last_fullsized":"2024-06-29T14:22:00.640300+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:21.205918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641110+0000","last_change":"2024-06-29T14:21:55.284217+0000","last_active":"2024-06-29T14:22:00.641110+0000","last_peered":"2024-06-29T14:22:00.641110+0000","last_clean":"2024-06-29T14:22:00.641110+0000","last_became_active":"2024-06-29T14:21:55.283906+0000","last_became_peered":"2024-06-29T14:21:55.283906+0000","last_unstale":"2024-06-29T14:22:00.641110+0000","last_undegraded":"2024-06-29T14:22:00.641110+0000","last_fullsized":"2024-06-29T14:22:00.641110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:39.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641864+0000","last_change":"2024-06-29T14:21:54.257367+0000","last_active":"2024-06-29T14:22:00.641864+0000","last_peered":"2024-06-29T14:22:00.641864+0000","last_clean":"2024-06-29T14:22:00.641864+0000","last_became_active":"2024-06-29T14:21:54.257040+0000","last_became_peered":"2024-06-29T14:21:54.257040+0000","last_unstale":"2024-06-29T14:22:00.641864+0000","last_undegraded":"2024-06-29T14:22:00.641864+0000","last_fullsized":"2024-06-29T14:22:00.641864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:53.425388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639860+0000","last_change":"2024-06-29T14:21:59.323943+0000","last_active":"2024-06-29T14:22:00.639860+0000","last_peered":"2024-06-29T14:22:00.639860+0000","last_clean":"2024-06-29T14:22:00.639860+0000","last_became_active":"2024-06-29T14:21:59.323400+0000","last_became_peered":"2024-06-29T14:21:59.323400+0000","last_unstale":"2024-06-29T14:22:00.639860+0000","last_undegraded":"2024-06-29T14:22:00.639860+0000","last_fullsized":"2024-06-29T14:22:00.639860+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:01.086377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307356+0000","last_change":"2024-06-29T14:21:55.292906+0000","last_active":"2024-06-29T14:22:00.307356+0000","last_peered":"2024-06-29T14:22:00.307356+0000","last_clean":"2024-06-29T14:22:00.307356+0000","last_became_active":"2024-06-29T14:21:55.292578+0000","last_became_peered":"2024-06-29T14:21:55.292578+0000","last_unstale":"2024-06-29T14:22:00.307356+0000","last_undegraded":"2024-06-29T14:22:00.307356+0000","last_fullsized":"2024-06-29T14:22:00.307356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:11.801869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640191+0000","last_change":"2024-06-29T14:21:54.264501+0000","last_active":"2024-06-29T14:22:00.640191+0000","last_peered":"2024-06-29T14:22:00.640191+0000","last_clean":"2024-06-29T14:22:00.640191+0000","last_became_active":"2024-06-29T14:21:54.264182+0000","last_became_peered":"2024-06-29T14:21:54.264182+0000","last_unstale":"2024-06-29T14:22:00.640191+0000","last_undegraded":"2024-06-29T14:22:00.640191+0000","last_fullsized":"2024-06-29T14:22:00.640191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:53.425388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641814+0000","last_change":"2024-06-29T14:21:59.324707+0000","last_active":"2024-06-29T14:22:00.641814+0000","last_peered":"2024-06-29T14:22:00.641814+0000","last_clean":"2024-06-29T14:22:00.641814+0000","last_became_active":"2024-06-29T14:21:59.324141+0000","last_became_peered":"2024-06-29T14:21:59.324141+0000","last_unstale":"2024-06-29T14:22:00.641814+0000","last_undegraded":"2024-06-29T14:22:00.641814+0000","last_fullsized":"2024-06-29T14:22:00.641814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:43.889397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639772+0000","last_change":"2024-06-29T14:21:55.274636+0000","last_active":"2024-06-29T14:22:00.639772+0000","last_peered":"2024-06-29T14:22:00.639772+0000","last_clean":"2024-06-29T14:22:00.639772+0000","last_became_active":"2024-06-29T14:21:55.271330+0000","last_became_peered":"2024-06-29T14:21:55.271330+0000","last_unstale":"2024-06-29T14:22:00.639772+0000","last_undegraded":"2024-06-29T14:22:00.639772+0000","last_fullsized":"2024-06-29T14:22:00.639772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640544+0000","last_change":"2024-06-29T14:21:54.247945+0000","last_active":"2024-06-29T14:22:00.640544+0000","last_peered":"2024-06-29T14:22:00.640544+0000","last_clean":"2024-06-29T14:22:00.640544+0000","last_became_active":"2024-06-29T14:21:54.247592+0000","last_became_peered":"2024-06-29T14:21:54.247592+0000","last_unstale":"2024-06-29T14:22:00.640544+0000","last_undegraded":"2024-06-29T14:22:00.640544+0000","last_fullsized":"2024-06-29T14:22:00.640544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:03.449985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.642194+0000","last_change":"2024-06-29T14:21:59.321043+0000","last_active":"2024-06-29T14:22:00.642194+0000","last_peered":"2024-06-29T14:22:00.642194+0000","last_clean":"2024-06-29T14:22:00.642194+0000","last_became_active":"2024-06-29T14:21:59.320359+0000","last_became_peered":"2024-06-29T14:21:59.320359+0000","last_unstale":"2024-06-29T14:22:00.642194+0000","last_undegraded":"2024-06-29T14:22:00.642194+0000","last_fullsized":"2024-06-29T14:22:00.642194+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:22.691854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639595+0000","last_change":"2024-06-29T14:21:55.296697+0000","last_active":"2024-06-29T14:22:00.639595+0000","last_peered":"2024-06-29T14:22:00.639595+0000","last_clean":"2024-06-29T14:22:00.639595+0000","last_became_active":"2024-06-29T14:21:55.296392+0000","last_became_peered":"2024-06-29T14:21:55.296392+0000","last_unstale":"2024-06-29T14:22:00.639595+0000","last_undegraded":"2024-06-29T14:22:00.639595+0000","last_fullsized":"2024-06-29T14:22:00.639595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:17.069875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639911+0000","last_change":"2024-06-29T14:21:54.257791+0000","last_active":"2024-06-29T14:22:00.639911+0000","last_peered":"2024-06-29T14:22:00.639911+0000","last_clean":"2024-06-29T14:22:00.639911+0000","last_became_active":"2024-06-29T14:21:54.257395+0000","last_became_peered":"2024-06-29T14:21:54.257395+0000","last_unstale":"2024-06-29T14:22:00.639911+0000","last_undegraded":"2024-06-29T14:22:00.639911+0000","last_fullsized":"2024-06-29T14:22:00.639911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:33.159720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640058+0000","last_change":"2024-06-29T14:21:59.337618+0000","last_active":"2024-06-29T14:22:00.640058+0000","last_peered":"2024-06-29T14:22:00.640058+0000","last_clean":"2024-06-29T14:22:00.640058+0000","last_became_active":"2024-06-29T14:21:59.336503+0000","last_became_peered":"2024-06-29T14:21:59.336503+0000","last_unstale":"2024-06-29T14:22:00.640058+0000","last_undegraded":"2024-06-29T14:22:00.640058+0000","last_fullsized":"2024-06-29T14:22:00.640058+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:01.086377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304940+0000","last_change":"2024-06-29T14:21:55.284345+0000","last_active":"2024-06-29T14:22:00.304940+0000","last_peered":"2024-06-29T14:22:00.304940+0000","last_clean":"2024-06-29T14:22:00.304940+0000","last_became_active":"2024-06-29T14:21:55.283603+0000","last_became_peered":"2024-06-29T14:21:55.283603+0000","last_unstale":"2024-06-29T14:22:00.304940+0000","last_undegraded":"2024-06-29T14:22:00.304940+0000","last_fullsized":"2024-06-29T14:22:00.304940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639902+0000","last_change":"2024-06-29T14:21:54.259298+0000","last_active":"2024-06-29T14:22:00.639902+0000","last_peered":"2024-06-29T14:22:00.639902+0000","last_clean":"2024-06-29T14:22:00.639902+0000","last_became_active":"2024-06-29T14:21:54.258971+0000","last_became_peered":"2024-06-29T14:21:54.258971+0000","last_unstale":"2024-06-29T14:22:00.639902+0000","last_undegraded":"2024-06-29T14:22:00.639902+0000","last_fullsized":"2024-06-29T14:22:00.639902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:34.839629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":117,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640060+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:22:00.640060+0000","last_peered":"2024-06-29T14:22:00.640060+0000","last_clean":"2024-06-29T14:22:00.640060+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:22:00.640060+0000","last_undegraded":"2024-06-29T14:22:00.640060+0000","last_fullsized":"2024-06-29T14:22:00.640060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305693+0000","last_change":"2024-06-29T14:21:59.337431+0000","last_active":"2024-06-29T14:22:00.305693+0000","last_peered":"2024-06-29T14:22:00.305693+0000","last_clean":"2024-06-29T14:22:00.305693+0000","last_became_active":"2024-06-29T14:21:59.337008+0000","last_became_peered":"2024-06-29T14:21:59.337008+0000","last_unstale":"2024-06-29T14:22:00.305693+0000","last_undegraded":"2024-06-29T14:22:00.305693+0000","last_fullsized":"2024-06-29T14:22:00.305693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:56.220106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640737+0000","last_change":"2024-06-29T14:21:55.287255+0000","last_active":"2024-06-29T14:22:00.640737+0000","last_peered":"2024-06-29T14:22:00.640737+0000","last_clean":"2024-06-29T14:22:00.640737+0000","last_became_active":"2024-06-29T14:21:55.286563+0000","last_became_peered":"2024-06-29T14:21:55.286563+0000","last_unstale":"2024-06-29T14:22:00.640737+0000","last_undegraded":"2024-06-29T14:22:00.640737+0000","last_fullsized":"2024-06-29T14:22:00.640737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:40.442251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640792+0000","last_change":"2024-06-29T14:21:54.264774+0000","last_active":"2024-06-29T14:22:00.640792+0000","last_peered":"2024-06-29T14:22:00.640792+0000","last_clean":"2024-06-29T14:22:00.640792+0000","last_became_active":"2024-06-29T14:21:54.264395+0000","last_became_peered":"2024-06-29T14:21:54.264395+0000","last_unstale":"2024-06-29T14:22:00.640792+0000","last_undegraded":"2024-06-29T14:22:00.640792+0000","last_fullsized":"2024-06-29T14:22:00.640792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:49.746280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305477+0000","last_change":"2024-06-29T14:21:59.336774+0000","last_active":"2024-06-29T14:22:00.305477+0000","last_peered":"2024-06-29T14:22:00.305477+0000","last_clean":"2024-06-29T14:22:00.305477+0000","last_became_active":"2024-06-29T14:21:59.336326+0000","last_became_peered":"2024-06-29T14:21:59.336326+0000","last_unstale":"2024-06-29T14:22:00.305477+0000","last_undegraded":"2024-06-29T14:22:00.305477+0000","last_fullsized":"2024-06-29T14:22:00.305477+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:20.883221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640544+0000","last_change":"2024-06-29T14:21:55.289532+0000","last_active":"2024-06-29T14:22:00.640544+0000","last_peered":"2024-06-29T14:22:00.640544+0000","last_clean":"2024-06-29T14:22:00.640544+0000","last_became_active":"2024-06-29T14:21:55.289210+0000","last_became_peered":"2024-06-29T14:21:55.289210+0000","last_unstale":"2024-06-29T14:22:00.640544+0000","last_undegraded":"2024-06-29T14:22:00.640544+0000","last_fullsized":"2024-06-29T14:22:00.640544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:12.073214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641944+0000","last_change":"2024-06-29T14:21:54.246332+0000","last_active":"2024-06-29T14:22:00.641944+0000","last_peered":"2024-06-29T14:22:00.641944+0000","last_clean":"2024-06-29T14:22:00.641944+0000","last_became_active":"2024-06-29T14:21:54.245974+0000","last_became_peered":"2024-06-29T14:21:54.245974+0000","last_unstale":"2024-06-29T14:22:00.641944+0000","last_undegraded":"2024-06-29T14:22:00.641944+0000","last_fullsized":"2024-06-29T14:22:00.641944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:50:54.491455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640419+0000","last_change":"2024-06-29T14:21:59.318499+0000","last_active":"2024-06-29T14:22:00.640419+0000","last_peered":"2024-06-29T14:22:00.640419+0000","last_clean":"2024-06-29T14:22:00.640419+0000","last_became_active":"2024-06-29T14:21:59.317918+0000","last_became_peered":"2024-06-29T14:21:59.317918+0000","last_unstale":"2024-06-29T14:22:00.640419+0000","last_undegraded":"2024-06-29T14:22:00.640419+0000","last_fullsized":"2024-06-29T14:22:00.640419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:43.889397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641394+0000","last_change":"2024-06-29T14:21:55.294388+0000","last_active":"2024-06-29T14:22:00.641394+0000","last_peered":"2024-06-29T14:22:00.641394+0000","last_clean":"2024-06-29T14:22:00.641394+0000","last_became_active":"2024-06-29T14:21:55.294090+0000","last_became_peered":"2024-06-29T14:21:55.294090+0000","last_unstale":"2024-06-29T14:22:00.641394+0000","last_undegraded":"2024-06-29T14:22:00.641394+0000","last_fullsized":"2024-06-29T14:22:00.641394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:39.325162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641102+0000","last_change":"2024-06-29T14:21:54.252699+0000","last_active":"2024-06-29T14:22:00.641102+0000","last_peered":"2024-06-29T14:22:00.641102+0000","last_clean":"2024-06-29T14:22:00.641102+0000","last_became_active":"2024-06-29T14:21:54.252365+0000","last_became_peered":"2024-06-29T14:21:54.252365+0000","last_unstale":"2024-06-29T14:22:00.641102+0000","last_undegraded":"2024-06-29T14:22:00.641102+0000","last_fullsized":"2024-06-29T14:22:00.641102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:32.388331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640359+0000","last_change":"2024-06-29T14:21:59.337020+0000","last_active":"2024-06-29T14:22:00.640359+0000","last_peered":"2024-06-29T14:22:00.640359+0000","last_clean":"2024-06-29T14:22:00.640359+0000","last_became_active":"2024-06-29T14:21:59.336457+0000","last_became_peered":"2024-06-29T14:21:59.336457+0000","last_unstale":"2024-06-29T14:22:00.640359+0000","last_undegraded":"2024-06-29T14:22:00.640359+0000","last_fullsized":"2024-06-29T14:22:00.640359+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:25.501876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640773+0000","last_change":"2024-06-29T14:21:55.297155+0000","last_active":"2024-06-29T14:22:00.640773+0000","last_peered":"2024-06-29T14:22:00.640773+0000","last_clean":"2024-06-29T14:22:00.640773+0000","last_became_active":"2024-06-29T14:21:55.296790+0000","last_became_peered":"2024-06-29T14:21:55.296790+0000","last_unstale":"2024-06-29T14:22:00.640773+0000","last_undegraded":"2024-06-29T14:22:00.640773+0000","last_fullsized":"2024-06-29T14:22:00.640773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:52.163431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306370+0000","last_change":"2024-06-29T14:21:54.261995+0000","last_active":"2024-06-29T14:22:00.306370+0000","last_peered":"2024-06-29T14:22:00.306370+0000","last_clean":"2024-06-29T14:22:00.306370+0000","last_became_active":"2024-06-29T14:21:54.261607+0000","last_became_peered":"2024-06-29T14:21:54.261607+0000","last_unstale":"2024-06-29T14:22:00.306370+0000","last_undegraded":"2024-06-29T14:22:00.306370+0000","last_fullsized":"2024-06-29T14:22:00.306370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:59.466378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305836+0000","last_change":"2024-06-29T14:21:59.316601+0000","last_active":"2024-06-29T14:22:00.305836+0000","last_peered":"2024-06-29T14:22:00.305836+0000","last_clean":"2024-06-29T14:22:00.305836+0000","last_became_active":"2024-06-29T14:21:59.315986+0000","last_became_peered":"2024-06-29T14:21:59.315986+0000","last_unstale":"2024-06-29T14:22:00.305836+0000","last_undegraded":"2024-06-29T14:22:00.305836+0000","last_fullsized":"2024-06-29T14:22:00.305836+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:45.085147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641018+0000","last_change":"2024-06-29T14:21:55.291524+0000","last_active":"2024-06-29T14:22:00.641018+0000","last_peered":"2024-06-29T14:22:00.641018+0000","last_clean":"2024-06-29T14:22:00.641018+0000","last_became_active":"2024-06-29T14:21:55.291206+0000","last_became_peered":"2024-06-29T14:21:55.291206+0000","last_unstale":"2024-06-29T14:22:00.641018+0000","last_undegraded":"2024-06-29T14:22:00.641018+0000","last_fullsized":"2024-06-29T14:22:00.641018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:16.826546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640410+0000","last_change":"2024-06-29T14:21:54.245875+0000","last_active":"2024-06-29T14:22:00.640410+0000","last_peered":"2024-06-29T14:22:00.640410+0000","last_clean":"2024-06-29T14:22:00.640410+0000","last_became_active":"2024-06-29T14:21:54.245444+0000","last_became_peered":"2024-06-29T14:21:54.245444+0000","last_unstale":"2024-06-29T14:22:00.640410+0000","last_undegraded":"2024-06-29T14:22:00.640410+0000","last_fullsized":"2024-06-29T14:22:00.640410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:49.347688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639988+0000","last_change":"2024-06-29T14:21:59.317611+0000","last_active":"2024-06-29T14:22:00.639988+0000","last_peered":"2024-06-29T14:22:00.639988+0000","last_clean":"2024-06-29T14:22:00.639988+0000","last_became_active":"2024-06-29T14:21:59.317081+0000","last_became_peered":"2024-06-29T14:21:59.317081+0000","last_unstale":"2024-06-29T14:22:00.639988+0000","last_undegraded":"2024-06-29T14:22:00.639988+0000","last_fullsized":"2024-06-29T14:22:00.639988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:16.129888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305414+0000","last_change":"2024-06-29T14:21:55.291070+0000","last_active":"2024-06-29T14:22:00.305414+0000","last_peered":"2024-06-29T14:22:00.305414+0000","last_clean":"2024-06-29T14:22:00.305414+0000","last_became_active":"2024-06-29T14:21:55.290468+0000","last_became_peered":"2024-06-29T14:21:55.290468+0000","last_unstale":"2024-06-29T14:22:00.305414+0000","last_undegraded":"2024-06-29T14:22:00.305414+0000","last_fullsized":"2024-06-29T14:22:00.305414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:15.443094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305532+0000","last_change":"2024-06-29T14:21:54.257111+0000","last_active":"2024-06-29T14:22:00.305532+0000","last_peered":"2024-06-29T14:22:00.305532+0000","last_clean":"2024-06-29T14:22:00.305532+0000","last_became_active":"2024-06-29T14:21:54.255536+0000","last_became_peered":"2024-06-29T14:21:54.255536+0000","last_unstale":"2024-06-29T14:22:00.305532+0000","last_undegraded":"2024-06-29T14:22:00.305532+0000","last_fullsized":"2024-06-29T14:22:00.305532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:28:41.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305863+0000","last_change":"2024-06-29T14:21:59.338565+0000","last_active":"2024-06-29T14:22:00.305863+0000","last_peered":"2024-06-29T14:22:00.305863+0000","last_clean":"2024-06-29T14:22:00.305863+0000","last_became_active":"2024-06-29T14:21:59.337920+0000","last_became_peered":"2024-06-29T14:21:59.337920+0000","last_unstale":"2024-06-29T14:22:00.305863+0000","last_undegraded":"2024-06-29T14:22:00.305863+0000","last_fullsized":"2024-06-29T14:22:00.305863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:46.890912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640755+0000","last_change":"2024-06-29T14:21:55.294165+0000","last_active":"2024-06-29T14:22:00.640755+0000","last_peered":"2024-06-29T14:22:00.640755+0000","last_clean":"2024-06-29T14:22:00.640755+0000","last_became_active":"2024-06-29T14:21:55.293742+0000","last_became_peered":"2024-06-29T14:21:55.293742+0000","last_unstale":"2024-06-29T14:22:00.640755+0000","last_undegraded":"2024-06-29T14:22:00.640755+0000","last_fullsized":"2024-06-29T14:22:00.640755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:35.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640160+0000","last_change":"2024-06-29T14:21:54.261554+0000","last_active":"2024-06-29T14:22:00.640160+0000","last_peered":"2024-06-29T14:22:00.640160+0000","last_clean":"2024-06-29T14:22:00.640160+0000","last_became_active":"2024-06-29T14:21:54.261215+0000","last_became_peered":"2024-06-29T14:21:54.261215+0000","last_unstale":"2024-06-29T14:22:00.640160+0000","last_undegraded":"2024-06-29T14:22:00.640160+0000","last_fullsized":"2024-06-29T14:22:00.640160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:43.007566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305468+0000","last_change":"2024-06-29T14:21:59.338884+0000","last_active":"2024-06-29T14:22:00.305468+0000","last_peered":"2024-06-29T14:22:00.305468+0000","last_clean":"2024-06-29T14:22:00.305468+0000","last_became_active":"2024-06-29T14:21:59.336402+0000","last_became_peered":"2024-06-29T14:21:59.336402+0000","last_unstale":"2024-06-29T14:22:00.305468+0000","last_undegraded":"2024-06-29T14:22:00.305468+0000","last_fullsized":"2024-06-29T14:22:00.305468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:00.545575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640161+0000","last_change":"2024-06-29T14:21:55.288628+0000","last_active":"2024-06-29T14:22:00.640161+0000","last_peered":"2024-06-29T14:22:00.640161+0000","last_clean":"2024-06-29T14:22:00.640161+0000","last_became_active":"2024-06-29T14:21:55.288079+0000","last_became_peered":"2024-06-29T14:21:55.288079+0000","last_unstale":"2024-06-29T14:22:00.640161+0000","last_undegraded":"2024-06-29T14:22:00.640161+0000","last_fullsized":"2024-06-29T14:22:00.640161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:44.658689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640560+0000","last_change":"2024-06-29T14:21:54.263230+0000","last_active":"2024-06-29T14:22:00.640560+0000","last_peered":"2024-06-29T14:22:00.640560+0000","last_clean":"2024-06-29T14:22:00.640560+0000","last_became_active":"2024-06-29T14:21:54.262952+0000","last_became_peered":"2024-06-29T14:21:54.262952+0000","last_unstale":"2024-06-29T14:22:00.640560+0000","last_undegraded":"2024-06-29T14:22:00.640560+0000","last_fullsized":"2024-06-29T14:22:00.640560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:48.032837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640470+0000","last_change":"2024-06-29T14:21:59.337082+0000","last_active":"2024-06-29T14:22:00.640470+0000","last_peered":"2024-06-29T14:22:00.640470+0000","last_clean":"2024-06-29T14:22:00.640470+0000","last_became_active":"2024-06-29T14:21:59.336506+0000","last_became_peered":"2024-06-29T14:21:59.336506+0000","last_unstale":"2024-06-29T14:22:00.640470+0000","last_undegraded":"2024-06-29T14:22:00.640470+0000","last_fullsized":"2024-06-29T14:22:00.640470+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:00.347317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640609+0000","last_change":"2024-06-29T14:21:55.299489+0000","last_active":"2024-06-29T14:22:00.640609+0000","last_peered":"2024-06-29T14:22:00.640609+0000","last_clean":"2024-06-29T14:22:00.640609+0000","last_became_active":"2024-06-29T14:21:55.299181+0000","last_became_peered":"2024-06-29T14:21:55.299181+0000","last_unstale":"2024-06-29T14:22:00.640609+0000","last_undegraded":"2024-06-29T14:22:00.640609+0000","last_fullsized":"2024-06-29T14:22:00.640609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:31.165204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:05.053670+0000","last_change":"2024-06-29T14:21:54.251485+0000","last_active":"2024-06-29T14:22:05.053670+0000","last_peered":"2024-06-29T14:22:05.053670+0000","last_clean":"2024-06-29T14:22:05.053670+0000","last_became_active":"2024-06-29T14:21:54.250203+0000","last_became_peered":"2024-06-29T14:21:54.250203+0000","last_unstale":"2024-06-29T14:22:05.053670+0000","last_undegraded":"2024-06-29T14:22:05.053670+0000","last_fullsized":"2024-06-29T14:22:05.053670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:03.449985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641539+0000","last_change":"2024-06-29T14:21:59.324423+0000","last_active":"2024-06-29T14:22:00.641539+0000","last_peered":"2024-06-29T14:22:00.641539+0000","last_clean":"2024-06-29T14:22:00.641539+0000","last_became_active":"2024-06-29T14:21:59.323663+0000","last_became_peered":"2024-06-29T14:21:59.323663+0000","last_unstale":"2024-06-29T14:22:00.641539+0000","last_undegraded":"2024-06-29T14:22:00.641539+0000","last_fullsized":"2024-06-29T14:22:00.641539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:21.205918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307509+0000","last_change":"2024-06-29T14:21:55.292602+0000","last_active":"2024-06-29T14:22:00.307509+0000","last_peered":"2024-06-29T14:22:00.307509+0000","last_clean":"2024-06-29T14:22:00.307509+0000","last_became_active":"2024-06-29T14:21:55.292047+0000","last_became_peered":"2024-06-29T14:21:55.292047+0000","last_unstale":"2024-06-29T14:22:00.307509+0000","last_undegraded":"2024-06-29T14:22:00.307509+0000","last_fullsized":"2024-06-29T14:22:00.307509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:15.123791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641934+0000","last_change":"2024-06-29T14:21:54.264727+0000","last_active":"2024-06-29T14:22:00.641934+0000","last_peered":"2024-06-29T14:22:00.641934+0000","last_clean":"2024-06-29T14:22:00.641934+0000","last_became_active":"2024-06-29T14:21:54.264355+0000","last_became_peered":"2024-06-29T14:21:54.264355+0000","last_unstale":"2024-06-29T14:22:00.641934+0000","last_undegraded":"2024-06-29T14:22:00.641934+0000","last_fullsized":"2024-06-29T14:22:00.641934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:06.025267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306568+0000","last_change":"2024-06-29T14:21:59.318452+0000","last_active":"2024-06-29T14:22:00.306568+0000","last_peered":"2024-06-29T14:22:00.306568+0000","last_clean":"2024-06-29T14:22:00.306568+0000","last_became_active":"2024-06-29T14:21:59.317634+0000","last_became_peered":"2024-06-29T14:21:59.317634+0000","last_unstale":"2024-06-29T14:22:00.306568+0000","last_undegraded":"2024-06-29T14:22:00.306568+0000","last_fullsized":"2024-06-29T14:22:00.306568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:46.219297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640791+0000","last_change":"2024-06-29T14:21:55.284276+0000","last_active":"2024-06-29T14:22:00.640791+0000","last_peered":"2024-06-29T14:22:00.640791+0000","last_clean":"2024-06-29T14:22:00.640791+0000","last_became_active":"2024-06-29T14:21:55.283929+0000","last_became_peered":"2024-06-29T14:21:55.283929+0000","last_unstale":"2024-06-29T14:22:00.640791+0000","last_undegraded":"2024-06-29T14:22:00.640791+0000","last_fullsized":"2024-06-29T14:22:00.640791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640975+0000","last_change":"2024-06-29T14:21:54.260674+0000","last_active":"2024-06-29T14:22:00.640975+0000","last_peered":"2024-06-29T14:22:00.640975+0000","last_clean":"2024-06-29T14:22:00.640975+0000","last_became_active":"2024-06-29T14:21:54.258991+0000","last_became_peered":"2024-06-29T14:21:54.258991+0000","last_unstale":"2024-06-29T14:22:00.640975+0000","last_undegraded":"2024-06-29T14:22:00.640975+0000","last_fullsized":"2024-06-29T14:22:00.640975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:50:54.491455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305903+0000","last_change":"2024-06-29T14:21:59.319141+0000","last_active":"2024-06-29T14:22:00.305903+0000","last_peered":"2024-06-29T14:22:00.305903+0000","last_clean":"2024-06-29T14:22:00.305903+0000","last_became_active":"2024-06-29T14:21:59.318645+0000","last_became_peered":"2024-06-29T14:21:59.318645+0000","last_unstale":"2024-06-29T14:22:00.305903+0000","last_undegraded":"2024-06-29T14:22:00.305903+0000","last_fullsized":"2024-06-29T14:22:00.305903+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:50.697736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640894+0000","last_change":"2024-06-29T14:21:55.293028+0000","last_active":"2024-06-29T14:22:00.640894+0000","last_peered":"2024-06-29T14:22:00.640894+0000","last_clean":"2024-06-29T14:22:00.640894+0000","last_became_active":"2024-06-29T14:21:55.292709+0000","last_became_peered":"2024-06-29T14:21:55.292709+0000","last_unstale":"2024-06-29T14:22:00.640894+0000","last_undegraded":"2024-06-29T14:22:00.640894+0000","last_fullsized":"2024-06-29T14:22:00.640894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:30.912272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305837+0000","last_change":"2024-06-29T14:21:54.262667+0000","last_active":"2024-06-29T14:22:00.305837+0000","last_peered":"2024-06-29T14:22:00.305837+0000","last_clean":"2024-06-29T14:22:00.305837+0000","last_became_active":"2024-06-29T14:21:54.262286+0000","last_became_peered":"2024-06-29T14:21:54.262286+0000","last_unstale":"2024-06-29T14:22:00.305837+0000","last_undegraded":"2024-06-29T14:22:00.305837+0000","last_fullsized":"2024-06-29T14:22:00.305837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:49.746280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304956+0000","last_change":"2024-06-29T14:21:59.337794+0000","last_active":"2024-06-29T14:22:00.304956+0000","last_peered":"2024-06-29T14:22:00.304956+0000","last_clean":"2024-06-29T14:22:00.304956+0000","last_became_active":"2024-06-29T14:21:59.336659+0000","last_became_peered":"2024-06-29T14:21:59.336659+0000","last_unstale":"2024-06-29T14:22:00.304956+0000","last_undegraded":"2024-06-29T14:22:00.304956+0000","last_fullsized":"2024-06-29T14:22:00.304956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:15.282310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639863+0000","last_change":"2024-06-29T14:21:59.337985+0000","last_active":"2024-06-29T14:22:00.639863+0000","last_peered":"2024-06-29T14:22:00.639863+0000","last_clean":"2024-06-29T14:22:00.639863+0000","last_became_active":"2024-06-29T14:21:59.337142+0000","last_became_peered":"2024-06-29T14:21:59.337142+0000","last_unstale":"2024-06-29T14:22:00.639863+0000","last_undegraded":"2024-06-29T14:22:00.639863+0000","last_fullsized":"2024-06-29T14:22:00.639863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:26.400192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639809+0000","last_change":"2024-06-29T14:21:55.291354+0000","last_active":"2024-06-29T14:22:00.639809+0000","last_peered":"2024-06-29T14:22:00.639809+0000","last_clean":"2024-06-29T14:22:00.639809+0000","last_became_active":"2024-06-29T14:21:55.290328+0000","last_became_peered":"2024-06-29T14:21:55.290328+0000","last_unstale":"2024-06-29T14:22:00.639809+0000","last_undegraded":"2024-06-29T14:22:00.639809+0000","last_fullsized":"2024-06-29T14:22:00.639809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639839+0000","last_change":"2024-06-29T14:21:54.254791+0000","last_active":"2024-06-29T14:22:00.639839+0000","last_peered":"2024-06-29T14:22:00.639839+0000","last_clean":"2024-06-29T14:22:00.639839+0000","last_became_active":"2024-06-29T14:21:54.254345+0000","last_became_peered":"2024-06-29T14:21:54.254345+0000","last_unstale":"2024-06-29T14:22:00.639839+0000","last_undegraded":"2024-06-29T14:22:00.639839+0000","last_fullsized":"2024-06-29T14:22:00.639839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:06.025267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305684+0000","last_change":"2024-06-29T14:21:59.317911+0000","last_active":"2024-06-29T14:22:00.305684+0000","last_peered":"2024-06-29T14:22:00.305684+0000","last_clean":"2024-06-29T14:22:00.305684+0000","last_became_active":"2024-06-29T14:21:59.317469+0000","last_became_peered":"2024-06-29T14:21:59.317469+0000","last_unstale":"2024-06-29T14:22:00.305684+0000","last_undegraded":"2024-06-29T14:22:00.305684+0000","last_fullsized":"2024-06-29T14:22:00.305684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:32.630422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305774+0000","last_change":"2024-06-29T14:21:55.297533+0000","last_active":"2024-06-29T14:22:00.305774+0000","last_peered":"2024-06-29T14:22:00.305774+0000","last_clean":"2024-06-29T14:22:00.305774+0000","last_became_active":"2024-06-29T14:21:55.296487+0000","last_became_peered":"2024-06-29T14:21:55.296487+0000","last_unstale":"2024-06-29T14:22:00.305774+0000","last_undegraded":"2024-06-29T14:22:00.305774+0000","last_fullsized":"2024-06-29T14:22:00.305774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305104+0000","last_change":"2024-06-29T14:21:54.253349+0000","last_active":"2024-06-29T14:22:00.305104+0000","last_peered":"2024-06-29T14:22:00.305104+0000","last_clean":"2024-06-29T14:22:00.305104+0000","last_became_active":"2024-06-29T14:21:54.252606+0000","last_became_peered":"2024-06-29T14:21:54.252606+0000","last_unstale":"2024-06-29T14:22:00.305104+0000","last_undegraded":"2024-06-29T14:22:00.305104+0000","last_fullsized":"2024-06-29T14:22:00.305104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:20.986619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306038+0000","last_change":"2024-06-29T14:21:54.254902+0000","last_active":"2024-06-29T14:22:00.306038+0000","last_peered":"2024-06-29T14:22:00.306038+0000","last_clean":"2024-06-29T14:22:00.306038+0000","last_became_active":"2024-06-29T14:21:54.254549+0000","last_became_peered":"2024-06-29T14:21:54.254549+0000","last_unstale":"2024-06-29T14:22:00.306038+0000","last_undegraded":"2024-06-29T14:22:00.306038+0000","last_fullsized":"2024-06-29T14:22:00.306038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:49.347688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640490+0000","last_change":"2024-06-29T14:21:55.297816+0000","last_active":"2024-06-29T14:22:00.640490+0000","last_peered":"2024-06-29T14:22:00.640490+0000","last_clean":"2024-06-29T14:22:00.640490+0000","last_became_active":"2024-06-29T14:21:55.297080+0000","last_became_peered":"2024-06-29T14:21:55.297080+0000","last_unstale":"2024-06-29T14:22:00.640490+0000","last_undegraded":"2024-06-29T14:22:00.640490+0000","last_fullsized":"2024-06-29T14:22:00.640490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640695+0000","last_change":"2024-06-29T14:21:59.324236+0000","last_active":"2024-06-29T14:22:00.640695+0000","last_peered":"2024-06-29T14:22:00.640695+0000","last_clean":"2024-06-29T14:22:00.640695+0000","last_became_active":"2024-06-29T14:21:59.323673+0000","last_became_peered":"2024-06-29T14:21:59.323673+0000","last_unstale":"2024-06-29T14:22:00.640695+0000","last_undegraded":"2024-06-29T14:22:00.640695+0000","last_fullsized":"2024-06-29T14:22:00.640695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:45.514616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305150+0000","last_change":"2024-06-29T14:21:54.257451+0000","last_active":"2024-06-29T14:22:00.305150+0000","last_peered":"2024-06-29T14:22:00.305150+0000","last_clean":"2024-06-29T14:22:00.305150+0000","last_became_active":"2024-06-29T14:21:54.257055+0000","last_became_peered":"2024-06-29T14:21:54.257055+0000","last_unstale":"2024-06-29T14:22:00.305150+0000","last_undegraded":"2024-06-29T14:22:00.305150+0000","last_fullsized":"2024-06-29T14:22:00.305150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:10:39.689935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307544+0000","last_change":"2024-06-29T14:21:55.291560+0000","last_active":"2024-06-29T14:22:00.307544+0000","last_peered":"2024-06-29T14:22:00.307544+0000","last_clean":"2024-06-29T14:22:00.307544+0000","last_became_active":"2024-06-29T14:21:55.291263+0000","last_became_peered":"2024-06-29T14:21:55.291263+0000","last_unstale":"2024-06-29T14:22:00.307544+0000","last_undegraded":"2024-06-29T14:22:00.307544+0000","last_fullsized":"2024-06-29T14:22:00.307544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:50.748371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305606+0000","last_change":"2024-06-29T14:21:59.319249+0000","last_active":"2024-06-29T14:22:00.305606+0000","last_peered":"2024-06-29T14:22:00.305606+0000","last_clean":"2024-06-29T14:22:00.305606+0000","last_became_active":"2024-06-29T14:21:59.316597+0000","last_became_peered":"2024-06-29T14:21:59.316597+0000","last_unstale":"2024-06-29T14:22:00.305606+0000","last_undegraded":"2024-06-29T14:22:00.305606+0000","last_fullsized":"2024-06-29T14:22:00.305606+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:54.923537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305108+0000","last_change":"2024-06-29T14:21:55.291254+0000","last_active":"2024-06-29T14:22:00.305108+0000","last_peered":"2024-06-29T14:22:00.305108+0000","last_clean":"2024-06-29T14:22:00.305108+0000","last_became_active":"2024-06-29T14:21:55.290924+0000","last_became_peered":"2024-06-29T14:21:55.290924+0000","last_unstale":"2024-06-29T14:22:00.305108+0000","last_undegraded":"2024-06-29T14:22:00.305108+0000","last_fullsized":"2024-06-29T14:22:00.305108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:11.308698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640426+0000","last_change":"2024-06-29T14:21:55.280475+0000","last_active":"2024-06-29T14:22:00.640426+0000","last_peered":"2024-06-29T14:22:00.640426+0000","last_clean":"2024-06-29T14:22:00.640426+0000","last_became_active":"2024-06-29T14:21:55.277502+0000","last_became_peered":"2024-06-29T14:21:55.277502+0000","last_unstale":"2024-06-29T14:22:00.640426+0000","last_undegraded":"2024-06-29T14:22:00.640426+0000","last_fullsized":"2024-06-29T14:22:00.640426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:11.801869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640065+0000","last_change":"2024-06-29T14:21:55.285270+0000","last_active":"2024-06-29T14:22:00.640065+0000","last_peered":"2024-06-29T14:22:00.640065+0000","last_clean":"2024-06-29T14:22:00.640065+0000","last_became_active":"2024-06-29T14:21:55.284164+0000","last_became_peered":"2024-06-29T14:21:55.284164+0000","last_unstale":"2024-06-29T14:22:00.640065+0000","last_undegraded":"2024-06-29T14:22:00.640065+0000","last_fullsized":"2024-06-29T14:22:00.640065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:25.523198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640375+0000","last_change":"2024-06-29T14:21:55.287457+0000","last_active":"2024-06-29T14:22:00.640375+0000","last_peered":"2024-06-29T14:22:00.640375+0000","last_clean":"2024-06-29T14:22:00.640375+0000","last_became_active":"2024-06-29T14:21:55.287070+0000","last_became_peered":"2024-06-29T14:21:55.287070+0000","last_unstale":"2024-06-29T14:22:00.640375+0000","last_undegraded":"2024-06-29T14:22:00.640375+0000","last_fullsized":"2024-06-29T14:22:00.640375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305575+0000","last_change":"2024-06-29T14:21:55.298926+0000","last_active":"2024-06-29T14:22:00.305575+0000","last_peered":"2024-06-29T14:22:00.305575+0000","last_clean":"2024-06-29T14:22:00.305575+0000","last_became_active":"2024-06-29T14:21:55.298615+0000","last_became_peered":"2024-06-29T14:21:55.298615+0000","last_unstale":"2024-06-29T14:22:00.305575+0000","last_undegraded":"2024-06-29T14:22:00.305575+0000","last_fullsized":"2024-06-29T14:22:00.305575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641774+0000","last_change":"2024-06-29T14:21:55.290890+0000","last_active":"2024-06-29T14:22:00.641774+0000","last_peered":"2024-06-29T14:22:00.641774+0000","last_clean":"2024-06-29T14:22:00.641774+0000","last_became_active":"2024-06-29T14:21:55.290069+0000","last_became_peered":"2024-06-29T14:21:55.290069+0000","last_unstale":"2024-06-29T14:22:00.641774+0000","last_undegraded":"2024-06-29T14:22:00.641774+0000","last_fullsized":"2024-06-29T14:22:00.641774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306202+0000","last_change":"2024-06-29T14:21:55.277969+0000","last_active":"2024-06-29T14:22:00.306202+0000","last_peered":"2024-06-29T14:22:00.306202+0000","last_clean":"2024-06-29T14:22:00.306202+0000","last_became_active":"2024-06-29T14:21:55.277629+0000","last_became_peered":"2024-06-29T14:21:55.277629+0000","last_unstale":"2024-06-29T14:22:00.306202+0000","last_undegraded":"2024-06-29T14:22:00.306202+0000","last_fullsized":"2024-06-29T14:22:00.306202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:00.388193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305827+0000","last_change":"2024-06-29T14:21:55.297394+0000","last_active":"2024-06-29T14:22:00.305827+0000","last_peered":"2024-06-29T14:22:00.305827+0000","last_clean":"2024-06-29T14:22:00.305827+0000","last_became_active":"2024-06-29T14:21:55.297087+0000","last_became_peered":"2024-06-29T14:21:55.297087+0000","last_unstale":"2024-06-29T14:22:00.305827+0000","last_undegraded":"2024-06-29T14:22:00.305827+0000","last_fullsized":"2024-06-29T14:22:00.305827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:43.242914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639702+0000","last_change":"2024-06-29T14:21:55.284207+0000","last_active":"2024-06-29T14:22:00.639702+0000","last_peered":"2024-06-29T14:22:00.639702+0000","last_clean":"2024-06-29T14:22:00.639702+0000","last_became_active":"2024-06-29T14:21:55.283869+0000","last_became_peered":"2024-06-29T14:21:55.283869+0000","last_unstale":"2024-06-29T14:22:00.639702+0000","last_undegraded":"2024-06-29T14:22:00.639702+0000","last_fullsized":"2024-06-29T14:22:00.639702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304555+0000","last_change":"2024-06-29T14:21:55.298136+0000","last_active":"2024-06-29T14:22:00.304555+0000","last_peered":"2024-06-29T14:22:00.304555+0000","last_clean":"2024-06-29T14:22:00.304555+0000","last_became_active":"2024-06-29T14:21:55.297852+0000","last_became_peered":"2024-06-29T14:21:55.297852+0000","last_unstale":"2024-06-29T14:22:00.304555+0000","last_undegraded":"2024-06-29T14:22:00.304555+0000","last_fullsized":"2024-06-29T14:22:00.304555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640027+0000","last_change":"2024-06-29T14:21:55.291932+0000","last_active":"2024-06-29T14:22:00.640027+0000","last_peered":"2024-06-29T14:22:00.640027+0000","last_clean":"2024-06-29T14:22:00.640027+0000","last_became_active":"2024-06-29T14:21:55.291605+0000","last_became_peered":"2024-06-29T14:21:55.291605+0000","last_unstale":"2024-06-29T14:22:00.640027+0000","last_undegraded":"2024-06-29T14:22:00.640027+0000","last_fullsized":"2024-06-29T14:22:00.640027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.505918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640145+0000","last_change":"2024-06-29T14:21:55.296088+0000","last_active":"2024-06-29T14:22:00.640145+0000","last_peered":"2024-06-29T14:22:00.640145+0000","last_clean":"2024-06-29T14:22:00.640145+0000","last_became_active":"2024-06-29T14:21:55.295752+0000","last_became_peered":"2024-06-29T14:21:55.295752+0000","last_unstale":"2024-06-29T14:22:00.640145+0000","last_undegraded":"2024-06-29T14:22:00.640145+0000","last_fullsized":"2024-06-29T14:22:00.640145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306344+0000","last_change":"2024-06-29T14:21:55.293717+0000","last_active":"2024-06-29T14:22:00.306344+0000","last_peered":"2024-06-29T14:22:00.306344+0000","last_clean":"2024-06-29T14:22:00.306344+0000","last_became_active":"2024-06-29T14:21:55.293421+0000","last_became_peered":"2024-06-29T14:21:55.293421+0000","last_unstale":"2024-06-29T14:22:00.306344+0000","last_undegraded":"2024-06-29T14:22:00.306344+0000","last_fullsized":"2024-06-29T14:22:00.306344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:08:58.083131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641151+0000","last_change":"2024-06-29T14:21:55.288925+0000","last_active":"2024-06-29T14:22:00.641151+0000","last_peered":"2024-06-29T14:22:00.641151+0000","last_clean":"2024-06-29T14:22:00.641151+0000","last_became_active":"2024-06-29T14:21:55.288585+0000","last_became_peered":"2024-06-29T14:21:55.288585+0000","last_unstale":"2024-06-29T14:22:00.641151+0000","last_undegraded":"2024-06-29T14:22:00.641151+0000","last_fullsized":"2024-06-29T14:22:00.641151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:35.844291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641027+0000","last_change":"2024-06-29T14:21:55.271843+0000","last_active":"2024-06-29T14:22:00.641027+0000","last_peered":"2024-06-29T14:22:00.641027+0000","last_clean":"2024-06-29T14:22:00.641027+0000","last_became_active":"2024-06-29T14:21:55.271494+0000","last_became_peered":"2024-06-29T14:21:55.271494+0000","last_unstale":"2024-06-29T14:22:00.641027+0000","last_undegraded":"2024-06-29T14:22:00.641027+0000","last_fullsized":"2024-06-29T14:22:00.641027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:29.960047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640632+0000","last_change":"2024-06-29T14:21:55.278149+0000","last_active":"2024-06-29T14:22:00.640632+0000","last_peered":"2024-06-29T14:22:00.640632+0000","last_clean":"2024-06-29T14:22:00.640632+0000","last_became_active":"2024-06-29T14:21:55.277816+0000","last_became_peered":"2024-06-29T14:21:55.277816+0000","last_unstale":"2024-06-29T14:22:00.640632+0000","last_undegraded":"2024-06-29T14:22:00.640632+0000","last_fullsized":"2024-06-29T14:22:00.640632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:14:18.590915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639577+0000","last_change":"2024-06-29T14:21:55.296366+0000","last_active":"2024-06-29T14:22:00.639577+0000","last_peered":"2024-06-29T14:22:00.639577+0000","last_clean":"2024-06-29T14:22:00.639577+0000","last_became_active":"2024-06-29T14:21:55.296040+0000","last_became_peered":"2024-06-29T14:21:55.296040+0000","last_unstale":"2024-06-29T14:22:00.639577+0000","last_undegraded":"2024-06-29T14:22:00.639577+0000","last_fullsized":"2024-06-29T14:22:00.639577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639931+0000","last_change":"2024-06-29T14:21:55.291183+0000","last_active":"2024-06-29T14:22:00.639931+0000","last_peered":"2024-06-29T14:22:00.639931+0000","last_clean":"2024-06-29T14:22:00.639931+0000","last_became_active":"2024-06-29T14:21:55.289884+0000","last_became_peered":"2024-06-29T14:21:55.289884+0000","last_unstale":"2024-06-29T14:22:00.639931+0000","last_undegraded":"2024-06-29T14:22:00.639931+0000","last_fullsized":"2024-06-29T14:22:00.639931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640038+0000","last_change":"2024-06-29T14:21:55.293964+0000","last_active":"2024-06-29T14:22:00.640038+0000","last_peered":"2024-06-29T14:22:00.640038+0000","last_clean":"2024-06-29T14:22:00.640038+0000","last_became_active":"2024-06-29T14:21:55.291885+0000","last_became_peered":"2024-06-29T14:21:55.291885+0000","last_unstale":"2024-06-29T14:22:00.640038+0000","last_undegraded":"2024-06-29T14:22:00.640038+0000","last_fullsized":"2024-06-29T14:22:00.640038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:20.427848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304827+0000","last_change":"2024-06-29T14:21:55.288765+0000","last_active":"2024-06-29T14:22:00.304827+0000","last_peered":"2024-06-29T14:22:00.304827+0000","last_clean":"2024-06-29T14:22:00.304827+0000","last_became_active":"2024-06-29T14:21:55.288452+0000","last_became_peered":"2024-06-29T14:21:55.288452+0000","last_unstale":"2024-06-29T14:22:00.304827+0000","last_undegraded":"2024-06-29T14:22:00.304827+0000","last_fullsized":"2024-06-29T14:22:00.304827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:30.912272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307365+0000","last_change":"2024-06-29T14:21:55.295376+0000","last_active":"2024-06-29T14:22:00.307365+0000","last_peered":"2024-06-29T14:22:00.307365+0000","last_clean":"2024-06-29T14:22:00.307365+0000","last_became_active":"2024-06-29T14:21:55.294928+0000","last_became_peered":"2024-06-29T14:21:55.294928+0000","last_unstale":"2024-06-29T14:22:00.307365+0000","last_undegraded":"2024-06-29T14:22:00.307365+0000","last_fullsized":"2024-06-29T14:22:00.307365+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:28:42.614313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305773+0000","last_change":"2024-06-29T14:21:55.281068+0000","last_active":"2024-06-29T14:22:00.305773+0000","last_peered":"2024-06-29T14:22:00.305773+0000","last_clean":"2024-06-29T14:22:00.305773+0000","last_became_active":"2024-06-29T14:21:55.280780+0000","last_became_peered":"2024-06-29T14:21:55.280780+0000","last_unstale":"2024-06-29T14:22:00.305773+0000","last_undegraded":"2024-06-29T14:22:00.305773+0000","last_fullsized":"2024-06-29T14:22:00.305773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:36:38.990506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306085+0000","last_change":"2024-06-29T14:21:55.280506+0000","last_active":"2024-06-29T14:22:00.306085+0000","last_peered":"2024-06-29T14:22:00.306085+0000","last_clean":"2024-06-29T14:22:00.306085+0000","last_became_active":"2024-06-29T14:21:55.280221+0000","last_became_peered":"2024-06-29T14:21:55.280221+0000","last_unstale":"2024-06-29T14:22:00.306085+0000","last_undegraded":"2024-06-29T14:22:00.306085+0000","last_fullsized":"2024-06-29T14:22:00.306085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:41.028472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641311+0000","last_change":"2024-06-29T14:21:55.287649+0000","last_active":"2024-06-29T14:22:00.641311+0000","last_peered":"2024-06-29T14:22:00.641311+0000","last_clean":"2024-06-29T14:22:00.641311+0000","last_became_active":"2024-06-29T14:21:55.286171+0000","last_became_peered":"2024-06-29T14:21:55.286171+0000","last_unstale":"2024-06-29T14:22:00.641311+0000","last_undegraded":"2024-06-29T14:22:00.641311+0000","last_fullsized":"2024-06-29T14:22:00.641311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:30.219397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304571+0000","last_change":"2024-06-29T14:21:55.285603+0000","last_active":"2024-06-29T14:22:00.304571+0000","last_peered":"2024-06-29T14:22:00.304571+0000","last_clean":"2024-06-29T14:22:00.304571+0000","last_became_active":"2024-06-29T14:21:55.285284+0000","last_became_peered":"2024-06-29T14:21:55.285284+0000","last_unstale":"2024-06-29T14:22:00.304571+0000","last_undegraded":"2024-06-29T14:22:00.304571+0000","last_fullsized":"2024-06-29T14:22:00.304571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:44.658689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640973+0000","last_change":"2024-06-29T14:21:55.287794+0000","last_active":"2024-06-29T14:22:00.640973+0000","last_peered":"2024-06-29T14:22:00.640973+0000","last_clean":"2024-06-29T14:22:00.640973+0000","last_became_active":"2024-06-29T14:21:55.287028+0000","last_became_peered":"2024-06-29T14:21:55.287028+0000","last_unstale":"2024-06-29T14:22:00.640973+0000","last_undegraded":"2024-06-29T14:22:00.640973+0000","last_fullsized":"2024-06-29T14:22:00.640973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:39.026791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639887+0000","last_change":"2024-06-29T14:21:55.289581+0000","last_active":"2024-06-29T14:22:00.639887+0000","last_peered":"2024-06-29T14:22:00.639887+0000","last_clean":"2024-06-29T14:22:00.639887+0000","last_became_active":"2024-06-29T14:21:55.289097+0000","last_became_peered":"2024-06-29T14:21:55.289097+0000","last_unstale":"2024-06-29T14:22:00.639887+0000","last_undegraded":"2024-06-29T14:22:00.639887+0000","last_fullsized":"2024-06-29T14:22:00.639887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:43.242914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640505+0000","last_change":"2024-06-29T14:21:55.288463+0000","last_active":"2024-06-29T14:22:00.640505+0000","last_peered":"2024-06-29T14:22:00.640505+0000","last_clean":"2024-06-29T14:22:00.640505+0000","last_became_active":"2024-06-29T14:21:55.288128+0000","last_became_peered":"2024-06-29T14:21:55.288128+0000","last_unstale":"2024-06-29T14:22:00.640505+0000","last_undegraded":"2024-06-29T14:22:00.640505+0000","last_fullsized":"2024-06-29T14:22:00.640505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:28.573747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640255+0000","last_change":"2024-06-29T14:21:55.290100+0000","last_active":"2024-06-29T14:22:00.640255+0000","last_peered":"2024-06-29T14:22:00.640255+0000","last_clean":"2024-06-29T14:22:00.640255+0000","last_became_active":"2024-06-29T14:21:55.289525+0000","last_became_peered":"2024-06-29T14:21:55.289525+0000","last_unstale":"2024-06-29T14:22:00.640255+0000","last_undegraded":"2024-06-29T14:22:00.640255+0000","last_fullsized":"2024-06-29T14:22:00.640255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:22.343517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641269+0000","last_change":"2024-06-29T14:21:55.287921+0000","last_active":"2024-06-29T14:22:00.641269+0000","last_peered":"2024-06-29T14:22:00.641269+0000","last_clean":"2024-06-29T14:22:00.641269+0000","last_became_active":"2024-06-29T14:21:55.287383+0000","last_became_peered":"2024-06-29T14:21:55.287383+0000","last_unstale":"2024-06-29T14:22:00.641269+0000","last_undegraded":"2024-06-29T14:22:00.641269+0000","last_fullsized":"2024-06-29T14:22:00.641269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:05.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641908+0000","last_change":"2024-06-29T14:21:55.285210+0000","last_active":"2024-06-29T14:22:00.641908+0000","last_peered":"2024-06-29T14:22:00.641908+0000","last_clean":"2024-06-29T14:22:00.641908+0000","last_became_active":"2024-06-29T14:21:55.284857+0000","last_became_peered":"2024-06-29T14:21:55.284857+0000","last_unstale":"2024-06-29T14:22:00.641908+0000","last_undegraded":"2024-06-29T14:22:00.641908+0000","last_fullsized":"2024-06-29T14:22:00.641908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641229+0000","last_change":"2024-06-29T14:21:55.287889+0000","last_active":"2024-06-29T14:22:00.641229+0000","last_peered":"2024-06-29T14:22:00.641229+0000","last_clean":"2024-06-29T14:22:00.641229+0000","last_became_active":"2024-06-29T14:21:55.286286+0000","last_became_peered":"2024-06-29T14:21:55.286286+0000","last_unstale":"2024-06-29T14:22:00.641229+0000","last_undegraded":"2024-06-29T14:22:00.641229+0000","last_fullsized":"2024-06-29T14:22:00.641229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:23.065793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509449,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":111290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":698454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":868352,"data_stored":698398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":111290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":107574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:22:13.986 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:22:13.987 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-06-29T14:22:13.987 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-06-29T14:22:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.145+0000 7f9e8cfee640 1 -- 172.21.15.59:0/1741123980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e88162770 msgr2=0x7f9e880ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.145+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/1741123980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e88162770 0x7f9e880ce550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9e80009830 tx=0x7f9e8002a470 comp rx=0 tx=0).stop 2024-06-29T14:22:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.145+0000 7f9e8cfee640 1 -- 172.21.15.59:0/1741123980 shutdown_connections 2024-06-29T14:22:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.145+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/1741123980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e8815e0a0 0x7f9e88161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.145+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/1741123980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e88162770 0x7f9e880ce550 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.145+0000 7f9e8cfee640 1 -- 172.21.15.59:0/1741123980 >> 172.21.15.59:0/1741123980 conn(0x7f9e880d45b0 msgr2=0x7f9e880d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.146+0000 7f9e8cfee640 1 -- 172.21.15.59:0/1741123980 shutdown_connections 2024-06-29T14:22:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.146+0000 7f9e8cfee640 1 -- 172.21.15.59:0/1741123980 wait complete. 2024-06-29T14:22:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.147+0000 7f9e8cfee640 1 Processor -- start 2024-06-29T14:22:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.147+0000 7f9e8cfee640 1 -- start start 2024-06-29T14:22:14.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e8cfee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e8815e0a0 0x7f9e880cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e8cfee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e88161fc0 0x7f9e88152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e8cfee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e88162770 0x7f9e881532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e8cfee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9e880d2d80 con 0x7f9e88162770 2024-06-29T14:22:14.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e8cfee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e880b4b00 con 0x7f9e8815e0a0 2024-06-29T14:22:14.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e8cfee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9e880d2f00 con 0x7f9e88161fc0 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e86d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e88162770 0x7f9e881532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e86d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e88162770 0x7f9e881532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51602/0 (socket says 172.21.15.59:51602) 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e86d9d640 1 -- 172.21.15.59:0/3340021609 learned_addr learned my addr 172.21.15.59:0/3340021609 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e85d9b640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e88161fc0 0x7f9e88152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e8659c640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e8815e0a0 0x7f9e880cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e86d9d640 1 -- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e88161fc0 msgr2=0x7f9e88152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e86d9d640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e88161fc0 0x7f9e88152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e86d9d640 1 -- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e8815e0a0 msgr2=0x7f9e880cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e86d9d640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e8815e0a0 0x7f9e880cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.149+0000 7f9e86d9d640 1 -- 172.21.15.59:0/3340021609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e88154630 con 0x7f9e88162770 2024-06-29T14:22:14.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.150+0000 7f9e86d9d640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e88162770 0x7f9e881532a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800caa0 tx=0x7f9e7800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:14.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.150+0000 7f9e7effd640 1 -- 172.21.15.59:0/3340021609 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78016020 con 0x7f9e88162770 2024-06-29T14:22:14.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.150+0000 7f9e7effd640 1 -- 172.21.15.59:0/3340021609 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e7800ad10 con 0x7f9e88162770 2024-06-29T14:22:14.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.150+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e88154910 con 0x7f9e88162770 2024-06-29T14:22:14.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.150+0000 7f9e7effd640 1 -- 172.21.15.59:0/3340021609 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78012680 con 0x7f9e88162770 2024-06-29T14:22:14.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.150+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e880cd170 con 0x7f9e88162770 2024-06-29T14:22:14.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.151+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e880d33d0 con 0x7f9e88162770 2024-06-29T14:22:14.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.152+0000 7f9e7effd640 1 -- 172.21.15.59:0/3340021609 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9e78019020 con 0x7f9e88162770 2024-06-29T14:22:14.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.152+0000 7f9e7effd640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e680762d0 0x7f9e680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.152+0000 7f9e7effd640 1 -- 172.21.15.59:0/3340021609 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f9e78097f40 con 0x7f9e88162770 2024-06-29T14:22:14.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.152+0000 7f9e8659c640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e680762d0 0x7f9e680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.153+0000 7f9e8659c640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e680762d0 0x7f9e680787e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9e80034000 tx=0x7f9e80004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:14.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.157+0000 7f9e7effd640 1 -- 172.21.15.59:0/3340021609 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9e78020020 con 0x7f9e88162770 2024-06-29T14:22:14.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.348+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f9e88154ae0 con 0x7f9e88162770 2024-06-29T14:22:14.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.348+0000 7f9e7effd640 1 -- 172.21.15.59:0/3340021609 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v27) ==== 86+0+160 (secure 0 0 0) 0x7f9e881623d0 con 0x7f9e88162770 2024-06-29T14:22:14.346 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:14.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e680762d0 msgr2=0x7f9e680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e680762d0 0x7f9e680787e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9e80034000 tx=0x7f9e80004730 comp rx=0 tx=0).stop 2024-06-29T14:22:14.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e88162770 msgr2=0x7f9e881532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e88162770 0x7f9e881532a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800caa0 tx=0x7f9e7800cf60 comp rx=0 tx=0).stop 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 shutdown_connections 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e680762d0 0x7f9e680787e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e88162770 0x7f9e881532a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e88161fc0 0x7f9e88152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 --2- 172.21.15.59:0/3340021609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e8815e0a0 0x7f9e880cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 >> 172.21.15.59:0/3340021609 conn(0x7f9e880d45b0 msgr2=0x7f9e880b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 shutdown_connections 2024-06-29T14:22:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.350+0000 7f9e8cfee640 1 -- 172.21.15.59:0/3340021609 wait complete. 2024-06-29T14:22:14.359 INFO:teuthology.orchestra.run.smithi059.stdout:[{"version":27,"timestamp":"2024-06-29T14:21:40.523965+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"4928.900892"}]}] 2024-06-29T14:22:14.360 INFO:tasks.ceph_manager:config epoch is 27 2024-06-29T14:22:14.360 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-29T14:22:14.360 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-06-29T14:22:14.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-06-29T14:22:14.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.479+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1016724791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162190 msgr2=0x7ff5640ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.479+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1016724791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162190 0x7ff5640ce550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff558009830 tx=0x7ff55802a410 comp rx=0 tx=0).stop 2024-06-29T14:22:14.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.479+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1016724791 shutdown_connections 2024-06-29T14:22:14.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.479+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1016724791 >> v2:172.21.15.153:3300/0 conn(0x7ff56415e0a0 0x7ff5640b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.479+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1016724791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162190 0x7ff5640ce550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.479+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1016724791 >> 172.21.15.59:0/1016724791 conn(0x7ff5640d45b0 msgr2=0x7ff5640d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.480+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1016724791 shutdown_connections 2024-06-29T14:22:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.480+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1016724791 wait complete. 2024-06-29T14:22:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.481+0000 7ff56b7f0640 1 Processor -- start 2024-06-29T14:22:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.481+0000 7ff56b7f0640 1 -- start start 2024-06-29T14:22:14.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56b7f0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff56415e0a0 0x7ff56415bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56b7f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162770 0x7ff5641599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56b7f0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5641f4160 0x7ff564157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56b7f0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff5640b43a0 con 0x7ff56415e0a0 2024-06-29T14:22:14.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56b7f0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff5640b4220 con 0x7ff5641f4160 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56b7f0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff5640b4520 con 0x7ff564162770 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff56415e0a0 0x7ff56415bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff56415e0a0 0x7ff56415bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51612/0 (socket says 172.21.15.59:51612) 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 -- 172.21.15.59:0/1522046887 learned_addr learned my addr 172.21.15.59:0/1522046887 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff568d8b640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162770 0x7ff5641599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff569d8d640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5641f4160 0x7ff564157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 -- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162770 msgr2=0x7ff5641599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162770 0x7ff5641599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 -- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5641f4160 msgr2=0x7ff564157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5641f4160 0x7ff564157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 -- 172.21.15.59:0/1522046887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5640c7e20 con 0x7ff56415e0a0 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff568d8b640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162770 0x7ff5641599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56958c640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff56415e0a0 0x7ff56415bdc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00c430 tx=0x7ff55c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff561ffb640 1 -- 172.21.15.59:0/1522046887 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c017070 con 0x7ff56415e0a0 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff561ffb640 1 -- 172.21.15.59:0/1522046887 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff55c0035f0 con 0x7ff56415e0a0 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.483+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff56415b2e0 con 0x7ff56415e0a0 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.484+0000 7ff561ffb640 1 -- 172.21.15.59:0/1522046887 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c003a40 con 0x7ff56415e0a0 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.484+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff56415b8a0 con 0x7ff56415e0a0 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.485+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5640b4a10 con 0x7ff56415e0a0 2024-06-29T14:22:14.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.485+0000 7ff561ffb640 1 -- 172.21.15.59:0/1522046887 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff55c01a070 con 0x7ff56415e0a0 2024-06-29T14:22:14.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.486+0000 7ff561ffb640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5500762d0 0x7ff5500787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.486+0000 7ff561ffb640 1 -- 172.21.15.59:0/1522046887 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff55c096ff0 con 0x7ff56415e0a0 2024-06-29T14:22:14.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.486+0000 7ff568d8b640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5500762d0 0x7ff5500787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.487+0000 7ff568d8b640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5500762d0 0x7ff5500787e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff558034000 tx=0x7ff558004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:14.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.490+0000 7ff561ffb640 1 -- 172.21.15.59:0/1522046887 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff55c060520 con 0x7ff56415e0a0 2024-06-29T14:22:14.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.736+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff56415d400 con 0x7ff56415e0a0 2024-06-29T14:22:14.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.739+0000 7ff561ffb640 1 -- 172.21.15.59:0/1522046887 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+189525 (secure 0 0 0) 0x7ff55c05fe20 con 0x7ff56415e0a0 2024-06-29T14:22:14.739 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:14.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5500762d0 msgr2=0x7ff5500787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5500762d0 0x7ff5500787e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff558034000 tx=0x7ff558004730 comp rx=0 tx=0).stop 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff56415e0a0 msgr2=0x7ff56415bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff56415e0a0 0x7ff56415bdc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00c430 tx=0x7ff55c00c8f0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 shutdown_connections 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5500762d0 0x7ff5500787e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5641f4160 0x7ff564157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff564162770 0x7ff5641599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 --2- 172.21.15.59:0/1522046887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff56415e0a0 0x7ff56415bdc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 >> 172.21.15.59:0/1522046887 conn(0x7ff5640d45b0 msgr2=0x7ff5640b7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 shutdown_connections 2024-06-29T14:22:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.742+0000 7ff56b7f0640 1 -- 172.21.15.59:0/1522046887 wait complete. 2024-06-29T14:22:14.751 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":654263470},{"type":"v1","addr":"172.21.15.153:6833","nonce":654263470}]},"active_addr":"172.21.15.153:6833/654263470","active_change":"2024-06-29T14:21:33.718425+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2185789752}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3207465637}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1180005807}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2990390183}]}]} 2024-06-29T14:22:14.756 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-06-29T14:22:14.756 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-06-29T14:22:14.756 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:14.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 -- 172.21.15.59:0/2687986436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465015e0a0 msgr2=0x7f4650161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 --2- 172.21.15.59:0/2687986436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465015e0a0 0x7f4650161a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f465008c200 tx=0x7f464402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 -- 172.21.15.59:0/2687986436 shutdown_connections 2024-06-29T14:22:14.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 --2- 172.21.15.59:0/2687986436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4650161fc0 0x7f46500d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 --2- 172.21.15.59:0/2687986436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465015e0a0 0x7f4650161a80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 --2- 172.21.15.59:0/2687986436 >> v2:172.21.15.153:3300/0 conn(0x7f4650162770 0x7f46500ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 -- 172.21.15.59:0/2687986436 >> 172.21.15.59:0/2687986436 conn(0x7f46500d45b0 msgr2=0x7f46500d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:14.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 -- 172.21.15.59:0/2687986436 shutdown_connections 2024-06-29T14:22:14.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.877+0000 7f4656889640 1 -- 172.21.15.59:0/2687986436 wait complete. 2024-06-29T14:22:14.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.879+0000 7f4656889640 1 Processor -- start 2024-06-29T14:22:14.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.879+0000 7f4656889640 1 -- start start 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4656889640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f465015e0a0 0x7f46500cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4656889640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4650161fc0 0x7f465015a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4656889640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650162770 0x7f465006d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4656889640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f46500d2dc0 con 0x7f4650162770 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4656889640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46500b31a0 con 0x7f4650161fc0 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4656889640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46500d2f40 con 0x7f465015e0a0 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650162770 0x7f465006d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650162770 0x7f465006d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51626/0 (socket says 172.21.15.59:51626) 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 -- 172.21.15.59:0/4166062247 learned_addr learned my addr 172.21.15.59:0/4166062247 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f464ffff640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f465015e0a0 0x7f46500cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f464f7fe640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4650161fc0 0x7f465015a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 -- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f465015e0a0 msgr2=0x7f46500cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f465015e0a0 0x7f46500cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 -- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4650161fc0 msgr2=0x7f465015a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4650161fc0 0x7f465015a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 -- 172.21.15.59:0/4166062247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465015a630 con 0x7f4650162770 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f4654e26640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650162770 0x7f465006d230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f464800e9d0 tx=0x7f464800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.881+0000 7f464cff9640 1 -- 172.21.15.59:0/4166062247 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464800cde0 con 0x7f4650162770 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.882+0000 7f464cff9640 1 -- 172.21.15.59:0/4166062247 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4648004b20 con 0x7f4650162770 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.882+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46500cbc00 con 0x7f4650162770 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.882+0000 7f464cff9640 1 -- 172.21.15.59:0/4166062247 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648005020 con 0x7f4650162770 2024-06-29T14:22:14.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.882+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46501f5000 con 0x7f4650162770 2024-06-29T14:22:14.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.883+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46500cb920 con 0x7f4650162770 2024-06-29T14:22:14.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.883+0000 7f464cff9640 1 -- 172.21.15.59:0/4166062247 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f46480051c0 con 0x7f4650162770 2024-06-29T14:22:14.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.884+0000 7f464cff9640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f463c0762d0 0x7f463c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:14.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.884+0000 7f464cff9640 1 -- 172.21.15.59:0/4166062247 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f46480985d0 con 0x7f4650162770 2024-06-29T14:22:14.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.884+0000 7f464ffff640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f463c0762d0 0x7f463c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:14.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.885+0000 7f464ffff640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f463c0762d0 0x7f463c0787e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4640009b30 tx=0x7f4640005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:14.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:14.889+0000 7f464cff9640 1 -- 172.21.15.59:0/4166062247 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f465015a630 con 0x7f4650162770 2024-06-29T14:22:15.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.082+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f465006de10 con 0x7f4650162770 2024-06-29T14:22:15.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.083+0000 7f464cff9640 1 -- 172.21.15.59:0/4166062247 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7f4648061470 con 0x7f4650162770 2024-06-29T14:22:15.081 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:15.082 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:15.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f463c0762d0 msgr2=0x7f463c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f463c0762d0 0x7f463c0787e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4640009b30 tx=0x7f4640005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:15.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650162770 msgr2=0x7f465006d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650162770 0x7f465006d230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f464800e9d0 tx=0x7f464800ee90 comp rx=0 tx=0).stop 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 shutdown_connections 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f463c0762d0 0x7f463c0787e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650162770 0x7f465006d230 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4650161fc0 0x7f465015a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 --2- 172.21.15.59:0/4166062247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f465015e0a0 0x7f46500cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 >> 172.21.15.59:0/4166062247 conn(0x7f46500d45b0 msgr2=0x7f46500b7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 shutdown_connections 2024-06-29T14:22:15.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.085+0000 7f4656889640 1 -- 172.21.15.59:0/4166062247 wait complete. 2024-06-29T14:22:15.095 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-06-29T14:22:15.095 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.214+0000 7f09d1c3d640 1 -- 172.21.15.59:0/2877013668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2270 msgr2=0x7f09cc15bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.214+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/2877013668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2270 0x7f09cc15bc20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f09bc005fd0 tx=0x7f09bc02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.214+0000 7f09d1c3d640 1 -- 172.21.15.59:0/2877013668 shutdown_connections 2024-06-29T14:22:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.214+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/2877013668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09cc0b2fa0 0x7f09cc0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.214+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/2877013668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2270 0x7f09cc15bc20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.214+0000 7f09d1c3d640 1 -- 172.21.15.59:0/2877013668 >> 172.21.15.59:0/2877013668 conn(0x7f09cc073e10 msgr2=0x7f09cc074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.214+0000 7f09d1c3d640 1 -- 172.21.15.59:0/2877013668 shutdown_connections 2024-06-29T14:22:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.214+0000 7f09d1c3d640 1 -- 172.21.15.59:0/2877013668 wait complete. 2024-06-29T14:22:15.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.216+0000 7f09d1c3d640 1 Processor -- start 2024-06-29T14:22:15.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.216+0000 7f09d1c3d640 1 -- start start 2024-06-29T14:22:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09d1c3d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09cc0b2270 0x7f09cc1f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09d1c3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2fa0 0x7f09cc1f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09d1c3d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09cc0d1a60 0x7f09cc06db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09d1c3d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0b6cc0 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09d1c3d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0b6b40 con 0x7f09cc0b2270 2024-06-29T14:22:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09d1c3d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f09cc0cfe90 con 0x7f09cc0d1a60 2024-06-29T14:22:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09caffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2fa0 0x7f09cc1f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09caffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2fa0 0x7f09cc1f8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51642/0 (socket says 172.21.15.59:51642) 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09caffd640 1 -- 172.21.15.59:0/27060731 learned_addr learned my addr 172.21.15.59:0/27060731 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09cb7fe640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09cc0b2270 0x7f09cc1f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09cbfff640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09cc0d1a60 0x7f09cc06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09caffd640 1 -- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09cc0d1a60 msgr2=0x7f09cc06db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09caffd640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09cc0d1a60 0x7f09cc06db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09caffd640 1 -- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09cc0b2270 msgr2=0x7f09cc1f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09caffd640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09cc0b2270 0x7f09cc1f8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.218+0000 7f09caffd640 1 -- 172.21.15.59:0/27060731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc15e030 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.219+0000 7f09caffd640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2fa0 0x7f09cc1f8640 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09c000e790 tx=0x7f09c000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.219+0000 7f09b3fff640 1 -- 172.21.15.59:0/27060731 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c000eea0 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.219+0000 7f09b3fff640 1 -- 172.21.15.59:0/27060731 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09c000cdb0 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.219+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc15e310 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.219+0000 7f09b3fff640 1 -- 172.21.15.59:0/27060731 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0010690 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.219+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc15e8d0 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.220+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09cc0d0400 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.221+0000 7f09b3fff640 1 -- 172.21.15.59:0/27060731 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f09c0010830 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.221+0000 7f09b3fff640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f09b80762d0 0x7f09b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.221+0000 7f09b3fff640 1 -- 172.21.15.59:0/27060731 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f09c0098450 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.221+0000 7f09cb7fe640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f09b80762d0 0x7f09b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.222+0000 7f09cb7fe640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f09b80762d0 0x7f09b80787e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f09bc033000 tx=0x7f09bc004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.225+0000 7f09b3fff640 1 -- 172.21.15.59:0/27060731 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f09c0061310 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.420+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09cc070280 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.421+0000 7f09b3fff640 1 -- 172.21.15.59:0/27060731 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7f09c0065610 con 0x7f09cc0b2fa0 2024-06-29T14:22:15.420 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:15.420 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:15.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f09b80762d0 msgr2=0x7f09b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f09b80762d0 0x7f09b80787e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f09bc033000 tx=0x7f09bc004730 comp rx=0 tx=0).stop 2024-06-29T14:22:15.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2fa0 msgr2=0x7f09cc1f8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2fa0 0x7f09cc1f8640 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09c000e790 tx=0x7f09c000ec50 comp rx=0 tx=0).stop 2024-06-29T14:22:15.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 shutdown_connections 2024-06-29T14:22:15.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f09b80762d0 0x7f09b80787e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09cc0d1a60 0x7f09cc06db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09cc0b2fa0 0x7f09cc1f8640 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 --2- 172.21.15.59:0/27060731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09cc0b2270 0x7f09cc1f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 >> 172.21.15.59:0/27060731 conn(0x7f09cc073e10 msgr2=0x7f09cc0d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 shutdown_connections 2024-06-29T14:22:15.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.424+0000 7f09d1c3d640 1 -- 172.21.15.59:0/27060731 wait complete. 2024-06-29T14:22:15.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-06-29T14:22:15.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-06-29T14:22:15.435 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-06-29T14:22:15.435 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-06-29T14:22:15.436 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-06-29T14:22:15.436 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-06-29T14:22:15.436 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-06-29T14:22:15.437 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-06-29T14:22:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.669+0000 7ffa0df82640 1 -- 172.21.15.59:0/2253012840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa08166800 msgr2=0x7ffa08168c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.670+0000 7ffa0df82640 1 --2- 172.21.15.59:0/2253012840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa08166800 0x7ffa08168c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffa00005fd0 tx=0x7ffa0002a770 comp rx=0 tx=0).stop 2024-06-29T14:22:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.669+0000 7f9be7fff640 1 -- 172.21.15.59:0/2026661473 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be802e050 con 0x7f9bf00c4710 2024-06-29T14:22:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.670+0000 7f9bf795b640 1 -- 172.21.15.59:0/2026661473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf00c4710 msgr2=0x7f9bf0072b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.670+0000 7f9bf795b640 1 --2- 172.21.15.59:0/2026661473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf00c4710 0x7f9bf0072b20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9be8005fd0 tx=0x7f9be802a840 comp rx=0 tx=0).stop 2024-06-29T14:22:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.670+0000 7f4235aa5640 1 -- 172.21.15.59:0/132500839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4230072b20 msgr2=0x7f4230072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.670+0000 7f4235aa5640 1 --2- 172.21.15.59:0/132500839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4230072b20 0x7f4230072f30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4224009830 tx=0x7f422402a5b0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.670+0000 7f9bf795b640 1 -- 172.21.15.59:0/2026661473 shutdown_connections 2024-06-29T14:22:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.670+0000 7f9bf795b640 1 --2- 172.21.15.59:0/2026661473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf00c4710 0x7f9bf0072b20 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9be8005fd0 tx=0x7f9be802a840 comp rx=0 tx=0).stop 2024-06-29T14:22:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.670+0000 7f9bf795b640 1 -- 172.21.15.59:0/2026661473 >> 172.21.15.59:0/2026661473 conn(0x7f9bf00b1c90 msgr2=0x7f9bf00cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7ffa0df82640 1 -- 172.21.15.59:0/2253012840 shutdown_connections 2024-06-29T14:22:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7ffa0df82640 1 --2- 172.21.15.59:0/2253012840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa08166800 0x7ffa08168c20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7ffa0df82640 1 -- 172.21.15.59:0/2253012840 >> 172.21.15.59:0/2253012840 conn(0x7ffa080af930 msgr2=0x7ffa080afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7f4235aa5640 1 -- 172.21.15.59:0/132500839 shutdown_connections 2024-06-29T14:22:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7f4235aa5640 1 --2- 172.21.15.59:0/132500839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4230072b20 0x7f4230072f30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7f4235aa5640 1 -- 172.21.15.59:0/132500839 >> 172.21.15.59:0/132500839 conn(0x7f42300b2100 msgr2=0x7f42300cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7f9bf795b640 1 -- 172.21.15.59:0/2026661473 shutdown_connections 2024-06-29T14:22:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7f9bf795b640 1 -- 172.21.15.59:0/2026661473 wait complete. 2024-06-29T14:22:15.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7ffa0df82640 1 -- 172.21.15.59:0/2253012840 shutdown_connections 2024-06-29T14:22:15.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7ffa0df82640 1 -- 172.21.15.59:0/2253012840 wait complete. 2024-06-29T14:22:15.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7f4235aa5640 1 -- 172.21.15.59:0/132500839 shutdown_connections 2024-06-29T14:22:15.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.671+0000 7f4235aa5640 1 -- 172.21.15.59:0/132500839 wait complete. 2024-06-29T14:22:15.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.673+0000 7ff541ffb640 1 -- 172.21.15.59:0/3983829668 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534031c30 con 0x7ff5440aef30 2024-06-29T14:22:15.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.674+0000 7f9bf795b640 1 Processor -- start 2024-06-29T14:22:15.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.674+0000 7f9bf795b640 1 -- start start 2024-06-29T14:22:15.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.674+0000 7ffa0df82640 1 Processor -- start 2024-06-29T14:22:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.674+0000 7ff5494c5640 1 -- 172.21.15.59:0/3983829668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5440aef30 msgr2=0x7ff5440ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.674+0000 7ff5494c5640 1 --2- 172.21.15.59:0/3983829668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5440aef30 0x7ff5440ad8b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff534009860 tx=0x7ff53402a530 comp rx=0 tx=0).stop 2024-06-29T14:22:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.675+0000 7ff5494c5640 1 -- 172.21.15.59:0/3983829668 shutdown_connections 2024-06-29T14:22:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.675+0000 7ff5494c5640 1 --2- 172.21.15.59:0/3983829668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5440aef30 0x7ff5440ad8b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.675+0000 7ff5494c5640 1 -- 172.21.15.59:0/3983829668 >> 172.21.15.59:0/3983829668 conn(0x7ff5440ccbe0 msgr2=0x7ff5440ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.674+0000 7f4235aa5640 1 Processor -- start 2024-06-29T14:22:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.674+0000 7f4235aa5640 1 -- start start 2024-06-29T14:22:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.675+0000 7ffa0df82640 1 -- start start 2024-06-29T14:22:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.675+0000 7ff5494c5640 1 -- 172.21.15.59:0/3983829668 shutdown_connections 2024-06-29T14:22:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.675+0000 7ff5494c5640 1 -- 172.21.15.59:0/3983829668 wait complete. 2024-06-29T14:22:15.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.675+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1831212478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d60004f40 msgr2=0x7f1d6000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.675+0000 7f1d7c0fd640 1 --2- 172.21.15.59:0/1831212478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d60004f40 0x7f1d6000b9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d74074cf0 tx=0x7f1d60005830 comp rx=0 tx=0).stop 2024-06-29T14:22:15.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.676+0000 7f1e054ef640 1 -- 172.21.15.59:0/2944036581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de8003ac0 msgr2=0x7f1de8003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.676+0000 7f1e054ef640 1 --2- 172.21.15.59:0/2944036581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de8003ac0 0x7f1de8003ef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1df0006a50 tx=0x7f1de8005830 comp rx=0 tx=0).stop 2024-06-29T14:22:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.676+0000 7f1e054ef640 1 -- 172.21.15.59:0/2944036581 shutdown_connections 2024-06-29T14:22:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.676+0000 7f1e054ef640 1 --2- 172.21.15.59:0/2944036581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de8003ac0 0x7f1de8003ef0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.676+0000 7f1e054ef640 1 -- 172.21.15.59:0/2944036581 >> 172.21.15.59:0/2944036581 conn(0x7f1e00072d80 msgr2=0x7f1e00073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f1e054ef640 1 -- 172.21.15.59:0/2944036581 shutdown_connections 2024-06-29T14:22:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f1e054ef640 1 -- 172.21.15.59:0/2944036581 wait complete. 2024-06-29T14:22:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f9bf795b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf0073060 0x7f9bf0179ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f9bf795b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9bf00b0c80 0x7f9bf0162110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f4235aa5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4230072b20 0x7f42300c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f4235aa5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4230143c70 0x7f4230131710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f4235aa5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f423015c220 0x7f4230159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f2673338640 1 -- 172.21.15.59:0/4009289116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2658004cc0 msgr2=0x7f26580050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f2673338640 1 --2- 172.21.15.59:0/4009289116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2658004cc0 0x7f26580050f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f266800c390 tx=0x7f2658005830 comp rx=0 tx=0).stop 2024-06-29T14:22:15.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.676+0000 7f1d7a69a640 1 -- 172.21.15.59:0/1831212478 reap_dead start 2024-06-29T14:22:15.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f9bf56f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf0073060 0x7f9bf0179ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f26718d5640 1 -- 172.21.15.59:0/4009289116 reap_dead start 2024-06-29T14:22:15.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f9bf56f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf0073060 0x7f9bf0179ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:45906/0 (socket says 172.21.15.59:45906) 2024-06-29T14:22:15.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f4235aa5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f423015d0d0 con 0x7f423015c220 2024-06-29T14:22:15.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f4235aa5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f423015cf50 con 0x7f4230072b20 2024-06-29T14:22:15.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f4235aa5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f423015d250 con 0x7f4230143c70 2024-06-29T14:22:15.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f422effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4230143c70 0x7f4230131710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f422effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4230143c70 0x7f4230131710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:48958/0 (socket says 172.21.15.59:48958) 2024-06-29T14:22:15.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f422effd640 1 -- 172.21.15.59:0/1261977374 learned_addr learned my addr 172.21.15.59:0/1261977374 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f422ffff640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f423015c220 0x7f4230159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f9bf56f7640 1 -- 172.21.15.59:0/2093296386 learned_addr learned my addr 172.21.15.59:0/2093296386 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f2673338640 1 -- 172.21.15.59:0/4009289116 shutdown_connections 2024-06-29T14:22:15.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f2673338640 1 -- 172.21.15.59:0/4009289116 >> 172.21.15.59:0/4009289116 conn(0x7f266c0d3ee0 msgr2=0x7f266c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f9bf795b640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf017a010 0x7f9bf0162650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1831212478 shutdown_connections 2024-06-29T14:22:15.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.677+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1831212478 >> 172.21.15.59:0/1831212478 conn(0x7f1d74072d80 msgr2=0x7f1d74073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f422f7fe640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4230072b20 0x7f42300c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1831212478 shutdown_connections 2024-06-29T14:22:15.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1831212478 wait complete. 2024-06-29T14:22:15.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f2673338640 1 -- 172.21.15.59:0/4009289116 shutdown_connections 2024-06-29T14:22:15.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f2673338640 1 -- 172.21.15.59:0/4009289116 wait complete. 2024-06-29T14:22:15.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7ffa0df82640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa080ce4c0 0x7ffa080acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7ffa0df82640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0815e1e0 0x7ffa08165df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7ffa0df82640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa08166800 0x7ffa081f8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7ffa0df82640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ffa08169160 con 0x7ffa08166800 2024-06-29T14:22:15.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7ffa0df82640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa080cf620 con 0x7ffa0815e1e0 2024-06-29T14:22:15.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7ffa0df82640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffa081692e0 con 0x7ffa080ce4c0 2024-06-29T14:22:15.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7ff5494c5640 1 Processor -- start 2024-06-29T14:22:15.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa0cf80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa080ce4c0 0x7ffa080acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa0cf80640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa080ce4c0 0x7ffa080acd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:48986/0 (socket says 172.21.15.59:48986) 2024-06-29T14:22:15.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa0cf80640 1 -- 172.21.15.59:0/3705450413 learned_addr learned my addr 172.21.15.59:0/3705450413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ff5494c5640 1 -- start start 2024-06-29T14:22:15.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f9bf4ef6640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9bf00b0c80 0x7f9bf0162110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf00b4180 con 0x7f9bf017a010 2024-06-29T14:22:15.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f422effd640 1 -- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4230072b20 msgr2=0x7f42300c50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f422effd640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4230072b20 0x7f42300c50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f422effd640 1 -- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f423015c220 msgr2=0x7f4230159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f422effd640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f423015c220 0x7f4230159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f422effd640 1 -- 172.21.15.59:0/1261977374 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4230158dc0 con 0x7f4230143c70 2024-06-29T14:22:15.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.678+0000 7f422ffff640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f423015c220 0x7f4230159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:15.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f422effd640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4230143c70 0x7f4230131710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4224000c00 tx=0x7f4224033ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f9bf5ef8640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf017a010 0x7f9bf0162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf00b4000 con 0x7f9bf0073060 2024-06-29T14:22:15.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf00b4300 con 0x7f9bf00b0c80 2024-06-29T14:22:15.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa07fff640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0815e1e0 0x7ffa08165df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa0cf80640 1 -- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0815e1e0 msgr2=0x7ffa08165df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa0cf80640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0815e1e0 0x7ffa08165df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa0cf80640 1 -- 172.21.15.59:0/3705450413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa08166800 msgr2=0x7ffa081f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa0cf80640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa08166800 0x7ffa081f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7ffa0cf80640 1 -- 172.21.15.59:0/3705450413 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa080ad940 con 0x7ffa080ce4c0 2024-06-29T14:22:15.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f1e054ef640 1 Processor -- start 2024-06-29T14:22:15.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f1ec359e640 1 -- 172.21.15.59:0/3035830191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40b4020 msgr2=0x7f1ec41637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f1ec359e640 1 --2- 172.21.15.59:0/3035830191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40b4020 0x7f1ec41637b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8005fd0 tx=0x7f1eb802b780 comp rx=0 tx=0).stop 2024-06-29T14:22:15.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f9bf56f7640 1 -- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9bf00b0c80 msgr2=0x7f9bf0162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f9bf56f7640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9bf00b0c80 0x7f9bf0162110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f9bf56f7640 1 -- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf017a010 msgr2=0x7f9bf0162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f9bf56f7640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf017a010 0x7f9bf0162650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f9bf56f7640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf006cb40 con 0x7f9bf0073060 2024-06-29T14:22:15.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f1e054ef640 1 -- start start 2024-06-29T14:22:15.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f9bf56f7640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf0073060 0x7f9bf0179ad0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00c730 tx=0x7f9bec00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4224031d60 con 0x7f4230143c70 2024-06-29T14:22:15.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42240330d0 con 0x7f4230143c70 2024-06-29T14:22:15.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4224033500 con 0x7f4230143c70 2024-06-29T14:22:15.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f4235aa5640 1 -- 172.21.15.59:0/1261977374 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42301590a0 con 0x7f4230143c70 2024-06-29T14:22:15.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.679+0000 7f4235aa5640 1 -- 172.21.15.59:0/1261977374 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4230159660 con 0x7f4230143c70 2024-06-29T14:22:15.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.680+0000 7f4235aa5640 1 -- 172.21.15.59:0/1261977374 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4230072f30 con 0x7f4230143c70 2024-06-29T14:22:15.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f1ec359e640 1 -- 172.21.15.59:0/3035830191 shutdown_connections 2024-06-29T14:22:15.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f1ec359e640 1 --2- 172.21.15.59:0/3035830191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40b4020 0x7f1ec41637b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f1ec359e640 1 --2- 172.21.15.59:0/3035830191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40cdfa0 0x7f1ec40ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f1ec359e640 1 -- 172.21.15.59:0/3035830191 >> 172.21.15.59:0/3035830191 conn(0x7f1ec40b21a0 msgr2=0x7f1ec4071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7ffa0cf80640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa080ce4c0 0x7ffa080acd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800c530 tx=0x7ff9f800c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7ff5494c5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5440aef30 0x7ff54414e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7ff5494c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5440b3450 0x7ff5441608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec016020 con 0x7f9bf0073060 2024-06-29T14:22:15.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9bec00a950 con 0x7f9bf0073060 2024-06-29T14:22:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bec00ada0 con 0x7f9bf0073060 2024-06-29T14:22:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf006ce20 con 0x7f9bf0073060 2024-06-29T14:22:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f1d7c0fd640 1 Processor -- start 2024-06-29T14:22:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf006d3e0 con 0x7f9bf0073060 2024-06-29T14:22:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f1e054ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de8003ac0 0x7f1e001558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f1e054ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e0015a110 0x7f1e001f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f1e054ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e0015a520 0x7f1e001f41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f1e054ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1e000b6b30 con 0x7f1e0015a520 2024-06-29T14:22:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f1e054ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e001411e0 con 0x7f1de8003ac0 2024-06-29T14:22:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f1e054ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1e0015b660 con 0x7f1e0015a110 2024-06-29T14:22:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f2673338640 1 Processor -- start 2024-06-29T14:22:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f2673338640 1 -- start start 2024-06-29T14:22:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f1ec359e640 1 -- 172.21.15.59:0/3035830191 shutdown_connections 2024-06-29T14:22:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7ff5494c5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5440ceec0 0x7ff5441525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7ff5494c5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff544169470 con 0x7ff5440b3450 2024-06-29T14:22:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7ff5494c5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff5441692f0 con 0x7ff5440aef30 2024-06-29T14:22:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7ff5494c5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff5441695f0 con 0x7ff5440ceec0 2024-06-29T14:22:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f1d7c0fd640 1 -- start start 2024-06-29T14:22:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7f1ec359e640 1 -- 172.21.15.59:0/3035830191 wait complete. 2024-06-29T14:22:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dff7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e0015a520 0x7f1e001f41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dff7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e0015a520 0x7f1e001f41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51718/0 (socket says 172.21.15.59:51718) 2024-06-29T14:22:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dff7fe640 1 -- 172.21.15.59:0/683002344 learned_addr learned my addr 172.21.15.59:0/683002344 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f42240035f0 con 0x7f4230143c70 2024-06-29T14:22:15.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f4217fff640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f421c0763a0 0x7f421c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f42240b8ed0 con 0x7f4230143c70 2024-06-29T14:22:15.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f4217fff640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7f421c07fb60 0x7f421c081ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 --> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42240317a0 con 0x7f421c07fb60 2024-06-29T14:22:15.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f42240b92c0 con 0x7f4230143c70 2024-06-29T14:22:15.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dfe7fc640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e0015a110 0x7f1e001f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dff7fe640 1 -- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e0015a110 msgr2=0x7f1e001f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9bd0001030 con 0x7f9bf0073060 2024-06-29T14:22:15.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f422ffff640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7f421c07fb60 0x7f421c081ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dff7fe640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e0015a110 0x7f1e001f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dff7fe640 1 -- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de8003ac0 msgr2=0x7f1e001558d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dff7fe640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1de8003ac0 0x7f1e001558d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.683+0000 7f1dff7fe640 1 -- 172.21.15.59:0/683002344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e00156240 con 0x7f1e0015a520 2024-06-29T14:22:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7ff542ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5440aef30 0x7ff54414e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7ff542ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5440aef30 0x7ff54414e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:45928/0 (socket says 172.21.15.59:45928) 2024-06-29T14:22:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7ff542ffd640 1 -- 172.21.15.59:0/1748974415 learned_addr learned my addr 172.21.15.59:0/1748974415 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f1d7c0fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d741f4e30 0x7f1d7406f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f1dff7fe640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e0015a520 0x7f1e001f41d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1e000b4800 tx=0x7f1df800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f2673338640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2658004cc0 0x7f266c0b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f2673338640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f266c151ea0 0x7f266c153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f1d7c0fd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7406fb50 0x7f1d7406e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df800cd00 con 0x7f1e0015a520 2024-06-29T14:22:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f2673338640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c1522b0 0x7f266c154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f2673338640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f266c0b3da0 con 0x7f266c1522b0 2024-06-29T14:22:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f2673338640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f266c0cea30 con 0x7f266c151ea0 2024-06-29T14:22:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f2673338640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f266c0b3f20 con 0x7f2658004cc0 2024-06-29T14:22:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f26710d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2658004cc0 0x7f266c0b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f26710d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2658004cc0 0x7f266c0b2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:49004/0 (socket says 172.21.15.59:49004) 2024-06-29T14:22:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f26710d4640 1 -- 172.21.15.59:0/2150498686 learned_addr learned my addr 172.21.15.59:0/2150498686 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1df8018070 con 0x7f1e0015a520 2024-06-29T14:22:15.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e00156520 con 0x7f1e0015a520 2024-06-29T14:22:15.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f422ffff640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7f421c07fb60 0x7f421c081ff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f26708d3640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f266c151ea0 0x7f266c153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e00156ae0 con 0x7f1e0015a520 2024-06-29T14:22:15.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9bec019020 con 0x7f9bf0073060 2024-06-29T14:22:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8003bd0 con 0x7f1e0015a520 2024-06-29T14:22:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f26718d5640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c1522b0 0x7f266c154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f26710d4640 1 -- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f266c151ea0 msgr2=0x7f266c153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f26710d4640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f266c151ea0 0x7f266c153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f26710d4640 1 -- 172.21.15.59:0/2150498686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c1522b0 msgr2=0x7f266c154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f26710d4640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c1522b0 0x7f266c154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f26710d4640 1 -- 172.21.15.59:0/2150498686 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266c15f900 con 0x7f2658004cc0 2024-06-29T14:22:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f26718d5640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c1522b0 0x7f266c154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ff542ffd640 1 -- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5440ceec0 msgr2=0x7ff5441525d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:22:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ff542ffd640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5440ceec0 0x7ff5441525d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ff542ffd640 1 -- 172.21.15.59:0/1748974415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5440b3450 msgr2=0x7ff5441608e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:22:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ff542ffd640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5440b3450 0x7ff5441608e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ff542ffd640 1 -- 172.21.15.59:0/1748974415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5441613e0 con 0x7ff5440aef30 2024-06-29T14:22:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ff542ffd640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5440aef30 0x7ff54414e510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff53402ab20 tx=0x7ff534033ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79e99640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d741f4e30 0x7f1d7406f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79e99640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d741f4e30 0x7f1d7406f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:45950/0 (socket says 172.21.15.59:45950) 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d7c0fd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d741f5240 0x7f1d7406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d7c0fd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1d740b3b00 con 0x7f1d7406fb50 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d7c0fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d740d0300 con 0x7f1d741f4e30 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d7c0fd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1d740b3c80 con 0x7f1d741f5240 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79e99640 1 -- 172.21.15.59:0/1864618578 learned_addr learned my addr 172.21.15.59:0/1864618578 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79698640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7406fb50 0x7f1d7406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79e99640 1 -- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d741f5240 msgr2=0x7f1d7406e8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79e99640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d741f5240 0x7f1d7406e8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79e99640 1 -- 172.21.15.59:0/1864618578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7406fb50 msgr2=0x7f1d7406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79e99640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7406fb50 0x7f1d7406e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7f1d79e99640 1 -- 172.21.15.59:0/1864618578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7406d2d0 con 0x7f1d741f4e30 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8005480 con 0x7ffa080ce4c0 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9f80035f0 con 0x7ffa080ce4c0 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8003a60 con 0x7ffa080ce4c0 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7ffa0df82640 1 -- 172.21.15.59:0/3705450413 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa080adc20 con 0x7ffa080ce4c0 2024-06-29T14:22:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.681+0000 7ffa0df82640 1 -- 172.21.15.59:0/3705450413 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa080ae1e0 con 0x7ffa080ce4c0 2024-06-29T14:22:15.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.682+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa080cddb0 con 0x7ffa080ce4c0 2024-06-29T14:22:15.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.684+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff9f8003c00 con 0x7ffa080ce4c0 2024-06-29T14:22:15.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ffa057fa640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff9f4076540 0x7ff9f4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff9f8097cd0 con 0x7ffa080ce4c0 2024-06-29T14:22:15.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534031d60 con 0x7ff5440aef30 2024-06-29T14:22:15.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ffa057fa640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7ff9f407fd10 0x7ff9f40821a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 --> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9f803be50 con 0x7ff9f407fd10 2024-06-29T14:22:15.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.685+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff9f80980c0 con 0x7ffa080ce4c0 2024-06-29T14:22:15.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff534005020 con 0x7ff5440aef30 2024-06-29T14:22:15.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534005450 con 0x7ff5440aef30 2024-06-29T14:22:15.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1de0001030 con 0x7f1e0015a520 2024-06-29T14:22:15.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f26710d4640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2658004cc0 0x7f266c0b2340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f266800c4b0 tx=0x7f2668001460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 <== osd.3 v2:172.21.15.59:6816/2018446268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42240317a0 con 0x7f421c07fb60 2024-06-29T14:22:15.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f422f7fe640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f421c0763a0 0x7f421c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7ffa07fff640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff9f4076540 0x7ff9f4078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f9be5ffb640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9bd40763a0 0x7f9bd40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f9bec097980 con 0x7f9bf0073060 2024-06-29T14:22:15.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2668001620 con 0x7f2658004cc0 2024-06-29T14:22:15.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7ffa0d781640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7ff9f407fd10 0x7ff9f40821a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f9be5ffb640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f9bd407fb60 0x7f9bd4081ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9bec01fe70 con 0x7f9bd407fb60 2024-06-29T14:22:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9bec09c2a0 con 0x7f9bf0073060 2024-06-29T14:22:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26680017c0 con 0x7f2658004cc0 2024-06-29T14:22:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2668003850 con 0x7f2658004cc0 2024-06-29T14:22:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.687+0000 7ff5494c5640 1 -- 172.21.15.59:0/1748974415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5441f87f0 con 0x7ff5440aef30 2024-06-29T14:22:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.687+0000 7ff5494c5640 1 -- 172.21.15.59:0/1748974415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54415b8c0 con 0x7ff5440aef30 2024-06-29T14:22:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f1d79e99640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d741f4e30 0x7f1d7406f610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00fda0 tx=0x7f1d6c00d710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.687+0000 7ffa07fff640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff9f4076540 0x7ff9f4078a50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc000f60 tx=0x7ff9fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.686+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f266c153130 con 0x7f2658004cc0 2024-06-29T14:22:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.687+0000 7f422f7fe640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f421c0763a0 0x7f421c0788b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42200046b0 tx=0x7f4220004620 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.688+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f266c1536f0 con 0x7f2658004cc0 2024-06-29T14:22:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.688+0000 7ffa0d781640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7ff9f407fd10 0x7ff9f40821a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.688+0000 7f1d79698640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7406fb50 0x7f1d7406e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:15.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.688+0000 7f9bf4ef6640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9bd40763a0 0x7f9bd40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.688+0000 7f9bf5ef8640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f9bd407fb60 0x7f9bd4081ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.688+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff51c001030 con 0x7ff5440aef30 2024-06-29T14:22:15.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c01d070 con 0x7f1d741f4e30 2024-06-29T14:22:15.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d6c018640 con 0x7f1d741f4e30 2024-06-29T14:22:15.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c018a70 con 0x7f1d741f4e30 2024-06-29T14:22:15.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1df8004030 con 0x7f1e0015a520 2024-06-29T14:22:15.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f266c15fbe0 con 0x7f2658004cc0 2024-06-29T14:22:15.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7f9bf4ef6640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9bd40763a0 0x7f9bd40788b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9be8034000 tx=0x7f9be8004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 <== osd.0 v2:172.21.15.59:6808/1075583359 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff9f803be50 con 0x7ff9f407fd10 2024-06-29T14:22:15.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7f9bf5ef8640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f9bd407fb60 0x7f9bd4081ff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.689+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1864618578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7406d5b0 con 0x7f1d741f4e30 2024-06-29T14:22:15.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.687+0000 7f1ec359e640 1 Processor -- start 2024-06-29T14:22:15.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.687+0000 7f1ec359e640 1 -- start start 2024-06-29T14:22:15.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec359e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40b4020 0x7f1ec4153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40cdfa0 0x7f1ec414ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec415a3e0 0x7f1ec41516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec359e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec415b070 con 0x7f1ec40cdfa0 2024-06-29T14:22:15.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec359e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec406fd40 con 0x7f1ec415a3e0 2024-06-29T14:22:15.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec359e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec4163cf0 con 0x7f1ec40b4020 2024-06-29T14:22:15.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff5340047a0 con 0x7ff5440aef30 2024-06-29T14:22:15.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f26680039f0 con 0x7f2658004cc0 2024-06-29T14:22:15.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1864618578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7406db70 con 0x7f1d741f4e30 2024-06-29T14:22:15.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1de77fe640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1dec076300 0x7f1dec078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1df8098920 con 0x7f1e0015a520 2024-06-29T14:22:15.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f1de77fe640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f1dec07fac0 0x7f1dec081f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7ff52b7fe640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5300763a0 0x7ff5300788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 --> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1df80020b0 con 0x7f1dec07fac0 2024-06-29T14:22:15.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1df8098d10 con 0x7f1e0015a520 2024-06-29T14:22:15.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec415a3e0 0x7f1ec41516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff5340b8b20 con 0x7ff5440aef30 2024-06-29T14:22:15.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 <== osd.1 v2:172.21.15.59:6824/1021101403 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9bec01fe70 con 0x7f9bd407fb60 2024-06-29T14:22:15.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec415a3e0 0x7f1ec41516f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:45974/0 (socket says 172.21.15.59:45974) 2024-06-29T14:22:15.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f2661ffb640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f26500762d0 0x7f26500787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f26680b7b60 con 0x7f2658004cc0 2024-06-29T14:22:15.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f2661ffb640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7f265007faa0 0x7f2650081f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 --> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2650082600 con 0x7f265007faa0 2024-06-29T14:22:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f26680bc2a0 con 0x7f2658004cc0 2024-06-29T14:22:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.690+0000 7f1ec2d9d640 1 -- 172.21.15.59:0/2289931248 learned_addr learned my addr 172.21.15.59:0/2289931248 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7ff52b7fe640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7ff53007fb70 0x7ff530082000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 --> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5340317a0 con 0x7ff53007fb70 2024-06-29T14:22:15.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff5340bd2a0 con 0x7ff5440aef30 2024-06-29T14:22:15.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.691+0000 7f1ec259c640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40b4020 0x7f1ec4153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1864618578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1d7406edf0 con 0x7f1d741f4e30 2024-06-29T14:22:15.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1ec2d9d640 1 -- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40b4020 msgr2=0x7f1ec4153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f26708d3640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f26500762d0 0x7f26500787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1ec2d9d640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40b4020 0x7f1ec4153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1ec2d9d640 1 -- 172.21.15.59:0/2289931248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40cdfa0 msgr2=0x7f1ec414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1ec1d9b640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40cdfa0 0x7f1ec414ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7ff5427fc640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5300763a0 0x7ff5300788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7ff5437fe640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7ff53007fb70 0x7ff530082000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1ec2d9d640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40cdfa0 0x7f1ec414ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1ec2d9d640 1 -- 172.21.15.59:0/2289931248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec414d3a0 con 0x7f1ec415a3e0 2024-06-29T14:22:15.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1ec1d9b640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40cdfa0 0x7f1ec414ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7ff5427fc640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5300763a0 0x7ff5300788b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff5440cbbf0 tx=0x7ff538007a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1dfeffd640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1dec076300 0x7f1dec078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f26718d5640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7f265007faa0 0x7f2650081f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f1ec2d9d640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec415a3e0 0x7f1ec41516f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8035040 tx=0x7f1eb8033f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f26708d3640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f26500762d0 0x7f26500787e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f266c0752d0 tx=0x7f265c005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7ff5437fe640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7ff53007fb70 0x7ff530082000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1dfe7fc640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f1dec07fac0 0x7f1dec081f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8041070 con 0x7f1ec415a3e0 2024-06-29T14:22:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1eb803da20 con 0x7f1ec415a3e0 2024-06-29T14:22:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb803de50 con 0x7f1ec415a3e0 2024-06-29T14:22:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1dfeffd640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1dec076300 0x7f1dec078810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1e0013e190 tx=0x7f1e000b5b00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1d6c020070 con 0x7f1d741f4e30 2024-06-29T14:22:15.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f1ec359e640 1 -- 172.21.15.59:0/2289931248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec4152460 con 0x7f1ec415a3e0 2024-06-29T14:22:15.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.692+0000 7f1dfe7fc640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f1dec07fac0 0x7f1dec081f50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.693+0000 7f1ec359e640 1 -- 172.21.15.59:0/2289931248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec4152a20 con 0x7f1ec415a3e0 2024-06-29T14:22:15.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7f26718d5640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7f265007faa0 0x7f2650081f30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 <== osd.2 v2:172.21.15.59:6800/2051957706 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1df80020b0 con 0x7f1dec07fac0 2024-06-29T14:22:15.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 <== osd.6 v2:172.21.15.153:6808/4247178163 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5340317a0 con 0x7ff53007fb70 2024-06-29T14:22:15.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7f1d6a7fc640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1d58076320 0x7f1d58078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1d6c0982b0 con 0x7f1d741f4e30 2024-06-29T14:22:15.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7f1d6a7fc640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f1d5807fa70 0x7f1d58081f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 --> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d6c00a0c0 con 0x7f1d5807fa70 2024-06-29T14:22:15.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1d6c09f050 con 0x7f1d741f4e30 2024-06-29T14:22:15.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.695+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 <== osd.4 v2:172.21.15.153:6800/2158079959 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2650082600 con 0x7f265007faa0 2024-06-29T14:22:15.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.694+0000 7f1ec359e640 1 -- 172.21.15.59:0/2289931248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1ec40ce3b0 con 0x7f1ec415a3e0 2024-06-29T14:22:15.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.695+0000 7f1d79698640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1d58076320 0x7f1d58078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.695+0000 7f1d79698640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1d58076320 0x7f1d58078830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d70005fd0 tx=0x7f1d70005e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.695+0000 7f1d7a69a640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f1d5807fa70 0x7f1d58081f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.696+0000 7f1d7a69a640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f1d5807fa70 0x7f1d58081f00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.696+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1eb8044070 con 0x7f1ec415a3e0 2024-06-29T14:22:15.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.697+0000 7f1eb2ffd640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1ea4076470 0x7f1ea4078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.697+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1eb80b9340 con 0x7f1ec415a3e0 2024-06-29T14:22:15.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.697+0000 7f1eb2ffd640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7f1ea407fc30 0x7f1ea40820c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:15.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.697+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 --> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1eb8007e80 con 0x7f1ea407fc30 2024-06-29T14:22:15.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.697+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1eb80b9730 con 0x7f1ec415a3e0 2024-06-29T14:22:15.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.697+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 <== osd.7 v2:172.21.15.153:6816/1688984748 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1d6c00a0c0 con 0x7f1d5807fa70 2024-06-29T14:22:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.697+0000 7f1ec1d9b640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7f1ea407fc30 0x7f1ea40820c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.697+0000 7f1ec259c640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1ea4076470 0x7f1ea4078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.698+0000 7f1ec1d9b640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7f1ea407fc30 0x7f1ea40820c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.698+0000 7f1ec259c640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1ea4076470 0x7f1ea4078980 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4006fd0 tx=0x7f1eb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.699+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 <== osd.5 v2:172.21.15.153:6824/4044954575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1eb8007e80 con 0x7f1ea407fc30 2024-06-29T14:22:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.717+0000 7f4235aa5640 1 -- 172.21.15.59:0/1261977374 --> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4230073b10 con 0x7f421c07fb60 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.718+0000 7f4217fff640 1 -- 172.21.15.59:0/1261977374 <== osd.3 v2:172.21.15.59:6816/2018446268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4230073b10 con 0x7f421c07fb60 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.718+0000 7f4215ffb640 1 -- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7f421c07fb60 msgr2=0x7f421c081ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.718+0000 7f4215ffb640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7f421c07fb60 0x7f421c081ff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.718+0000 7f4215ffb640 1 -- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f421c0763a0 msgr2=0x7f421c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.718+0000 7f4215ffb640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f421c0763a0 0x7f421c0788b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42200046b0 tx=0x7f4220004620 comp rx=0 tx=0).stop 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.718+0000 7f4215ffb640 1 -- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4230143c70 msgr2=0x7f4230131710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.718+0000 7f4215ffb640 1 --2- 172.21.15.59:0/1261977374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4230143c70 0x7f4230131710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4224000c00 tx=0x7f4224033ce0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.718+0000 7f422ffff640 1 -- 172.21.15.59:0/1261977374 reap_dead start 2024-06-29T14:22:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.719+0000 7f4215ffb640 1 -- 172.21.15.59:0/1261977374 shutdown_connections 2024-06-29T14:22:15.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.719+0000 7f4215ffb640 1 -- 172.21.15.59:0/1261977374 >> 172.21.15.59:0/1261977374 conn(0x7f42300b2100 msgr2=0x7f42300b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.719+0000 7f4215ffb640 1 -- 172.21.15.59:0/1261977374 shutdown_connections 2024-06-29T14:22:15.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.719+0000 7f4215ffb640 1 -- 172.21.15.59:0/1261977374 wait complete. 2024-06-29T14:22:15.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.724+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 --> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ffa0815ec70 con 0x7ff9f407fd10 2024-06-29T14:22:15.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.724+0000 7ffa057fa640 1 -- 172.21.15.59:0/3705450413 <== osd.0 v2:172.21.15.59:6808/1075583359 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ffa0815ec70 con 0x7ff9f407fd10 2024-06-29T14:22:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.725+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 --> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1de0002d50 con 0x7f1dec07fac0 2024-06-29T14:22:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 --> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9bd0002dd0 con 0x7f9bd407fb60 2024-06-29T14:22:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7ff9f407fd10 msgr2=0x7ff9f40821a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7ff9eeffd640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7ff9f407fd10 0x7ff9f40821a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff9f4076540 msgr2=0x7ff9f4078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7ff9eeffd640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff9f4076540 0x7ff9f4078a50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc000f60 tx=0x7ff9fc009040 comp rx=0 tx=0).stop 2024-06-29T14:22:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa080ce4c0 msgr2=0x7ffa080acd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7ff9eeffd640 1 --2- 172.21.15.59:0/3705450413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa080ce4c0 0x7ffa080acd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800c530 tx=0x7ff9f800c9f0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7ffa0d781640 1 -- 172.21.15.59:0/3705450413 reap_dead start 2024-06-29T14:22:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7f1de77fe640 1 -- 172.21.15.59:0/683002344 <== osd.2 v2:172.21.15.59:6800/2051957706 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1de0002d50 con 0x7f1dec07fac0 2024-06-29T14:22:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.726+0000 7f9be5ffb640 1 -- 172.21.15.59:0/2093296386 <== osd.1 v2:172.21.15.59:6824/1021101403 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9bd0002dd0 con 0x7f9bd407fb60 2024-06-29T14:22:15.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f1dec07fac0 msgr2=0x7f1dec081f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1e054ef640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f1dec07fac0 0x7f1dec081f50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 shutdown_connections 2024-06-29T14:22:15.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 >> 172.21.15.59:0/3705450413 conn(0x7ffa080af930 msgr2=0x7ffa0815f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 shutdown_connections 2024-06-29T14:22:15.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7ff9eeffd640 1 -- 172.21.15.59:0/3705450413 wait complete. 2024-06-29T14:22:15.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1dec076300 msgr2=0x7f1dec078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1e054ef640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1dec076300 0x7f1dec078810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1e0013e190 tx=0x7f1e000b5b00 comp rx=0 tx=0).stop 2024-06-29T14:22:15.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e0015a520 msgr2=0x7f1e001f41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1e054ef640 1 --2- 172.21.15.59:0/683002344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e0015a520 0x7f1e001f41d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1e000b4800 tx=0x7f1df800c6a0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f9bd407fb60 msgr2=0x7f9bd4081ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f9bf795b640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f9bd407fb60 0x7f9bd4081ff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9bd40763a0 msgr2=0x7f9bd40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf795b640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9bd40763a0 0x7f9bd40788b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9be8034000 tx=0x7f9be8004ab0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf0073060 msgr2=0x7f9bf0179ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1dff7fe640 1 -- 172.21.15.59:0/683002344 reap_dead start 2024-06-29T14:22:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 shutdown_connections 2024-06-29T14:22:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.727+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 >> 172.21.15.59:0/683002344 conn(0x7f1e00072d80 msgr2=0x7f1e0015dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf795b640 1 --2- 172.21.15.59:0/2093296386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf0073060 0x7f9bf0179ad0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00c730 tx=0x7f9bec00cbf0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf5ef8640 1 -- 172.21.15.59:0/2093296386 reap_dead start 2024-06-29T14:22:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 shutdown_connections 2024-06-29T14:22:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 >> 172.21.15.59:0/2093296386 conn(0x7f9bf00b1c90 msgr2=0x7f9bf00cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 shutdown_connections 2024-06-29T14:22:15.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f1e054ef640 1 -- 172.21.15.59:0/683002344 wait complete. 2024-06-29T14:22:15.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 --> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f266c1f52f0 con 0x7f265007faa0 2024-06-29T14:22:15.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 shutdown_connections 2024-06-29T14:22:15.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f9bf795b640 1 -- 172.21.15.59:0/2093296386 wait complete. 2024-06-29T14:22:15.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7f2661ffb640 1 -- 172.21.15.59:0/2150498686 <== osd.4 v2:172.21.15.153:6800/2158079959 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f266c1f52f0 con 0x7f265007faa0 2024-06-29T14:22:15.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.728+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 --> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff51c002dd0 con 0x7ff53007fb70 2024-06-29T14:22:15.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.729+0000 7ff52b7fe640 1 -- 172.21.15.59:0/1748974415 <== osd.6 v2:172.21.15.153:6808/4247178163 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff51c002dd0 con 0x7ff53007fb70 2024-06-29T14:22:15.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.729+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7f265007faa0 msgr2=0x7f2650081f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.729+0000 7f2673338640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7f265007faa0 0x7f2650081f30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.729+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7ff53007fb70 msgr2=0x7ff530082000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.729+0000 7ff5297fa640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7ff53007fb70 0x7ff530082000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f26500762d0 msgr2=0x7f26500787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f2673338640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f26500762d0 0x7f26500787e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f266c0752d0 tx=0x7f265c005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:15.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2658004cc0 msgr2=0x7f266c0b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f2673338640 1 --2- 172.21.15.59:0/2150498686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2658004cc0 0x7f266c0b2340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f266800c4b0 tx=0x7f2668001460 comp rx=0 tx=0).stop 2024-06-29T14:22:15.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f26718d5640 1 -- 172.21.15.59:0/2150498686 reap_dead start 2024-06-29T14:22:15.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f1d7c0fd640 1 -- 172.21.15.59:0/1864618578 --> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1d74161ae0 con 0x7f1d5807fa70 2024-06-29T14:22:15.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5300763a0 msgr2=0x7ff5300788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5297fa640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff5300763a0 0x7ff5300788b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff5440cbbf0 tx=0x7ff538007a10 comp rx=0 tx=0).stop 2024-06-29T14:22:15.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5440aef30 msgr2=0x7ff54414e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5297fa640 1 --2- 172.21.15.59:0/1748974415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5440aef30 0x7ff54414e510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff53402ab20 tx=0x7ff534033ce0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5437fe640 1 -- 172.21.15.59:0/1748974415 reap_dead start 2024-06-29T14:22:15.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 shutdown_connections 2024-06-29T14:22:15.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 >> 172.21.15.59:0/1748974415 conn(0x7ff5440ccbe0 msgr2=0x7ff544073730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 shutdown_connections 2024-06-29T14:22:15.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 shutdown_connections 2024-06-29T14:22:15.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 >> 172.21.15.59:0/2150498686 conn(0x7f266c0d3ee0 msgr2=0x7f266c15af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7ff5297fa640 1 -- 172.21.15.59:0/1748974415 wait complete. 2024-06-29T14:22:15.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 shutdown_connections 2024-06-29T14:22:15.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f2673338640 1 -- 172.21.15.59:0/2150498686 wait complete. 2024-06-29T14:22:15.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.730+0000 7f1d6a7fc640 1 -- 172.21.15.59:0/1864618578 <== osd.7 v2:172.21.15.153:6816/1688984748 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1d74161ae0 con 0x7f1d5807fa70 2024-06-29T14:22:15.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.731+0000 7f1d57fff640 1 -- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f1d5807fa70 msgr2=0x7f1d58081f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.731+0000 7f1d57fff640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f1d5807fa70 0x7f1d58081f00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d57fff640 1 -- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1d58076320 msgr2=0x7f1d58078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d57fff640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1d58076320 0x7f1d58078830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d70005fd0 tx=0x7f1d70005e80 comp rx=0 tx=0).stop 2024-06-29T14:22:15.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d57fff640 1 -- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d741f4e30 msgr2=0x7f1d7406f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d57fff640 1 --2- 172.21.15.59:0/1864618578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d741f4e30 0x7f1d7406f610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00fda0 tx=0x7f1d6c00d710 comp rx=0 tx=0).stop 2024-06-29T14:22:15.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d7a69a640 1 -- 172.21.15.59:0/1864618578 reap_dead start 2024-06-29T14:22:15.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d57fff640 1 -- 172.21.15.59:0/1864618578 shutdown_connections 2024-06-29T14:22:15.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d57fff640 1 -- 172.21.15.59:0/1864618578 >> 172.21.15.59:0/1864618578 conn(0x7f1d74072d80 msgr2=0x7f1d740b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d57fff640 1 -- 172.21.15.59:0/1864618578 shutdown_connections 2024-06-29T14:22:15.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.732+0000 7f1d57fff640 1 -- 172.21.15.59:0/1864618578 wait complete. 2024-06-29T14:22:15.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.733+0000 7f1ec359e640 1 -- 172.21.15.59:0/2289931248 --> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1ec4072570 con 0x7f1ea407fc30 2024-06-29T14:22:15.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.733+0000 7f1eb2ffd640 1 -- 172.21.15.59:0/2289931248 <== osd.5 v2:172.21.15.153:6824/4044954575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1ec4072570 con 0x7f1ea407fc30 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.734+0000 7f1eb0ff9640 1 -- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7f1ea407fc30 msgr2=0x7f1ea40820c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.734+0000 7f1eb0ff9640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7f1ea407fc30 0x7f1ea40820c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.735+0000 7f1eb0ff9640 1 -- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1ea4076470 msgr2=0x7f1ea4078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.735+0000 7f1eb0ff9640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1ea4076470 0x7f1ea4078980 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4006fd0 tx=0x7f1eb4008040 comp rx=0 tx=0).stop 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.735+0000 7f1eb0ff9640 1 -- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec415a3e0 msgr2=0x7f1ec41516f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.735+0000 7f1eb0ff9640 1 --2- 172.21.15.59:0/2289931248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec415a3e0 0x7f1ec41516f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8035040 tx=0x7f1eb8033f80 comp rx=0 tx=0).stop 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.735+0000 7f1ec2d9d640 1 -- 172.21.15.59:0/2289931248 reap_dead start 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.735+0000 7f1eb0ff9640 1 -- 172.21.15.59:0/2289931248 shutdown_connections 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.735+0000 7f1eb0ff9640 1 -- 172.21.15.59:0/2289931248 >> 172.21.15.59:0/2289931248 conn(0x7f1ec40b21a0 msgr2=0x7f1ec40b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.736+0000 7f1eb0ff9640 1 -- 172.21.15.59:0/2289931248 shutdown_connections 2024-06-29T14:22:15.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:15.736+0000 7f1eb0ff9640 1 -- 172.21.15.59:0/2289931248 wait complete. 2024-06-29T14:22:15.758 INFO:teuthology.orchestra.run.smithi059.stdout:64424509450 2024-06-29T14:22:15.758 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-29T14:22:15.759 INFO:teuthology.orchestra.run.smithi059.stdout:64424509451 2024-06-29T14:22:15.759 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-29T14:22:15.760 INFO:teuthology.orchestra.run.smithi059.stdout:64424509449 2024-06-29T14:22:15.760 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-29T14:22:15.761 INFO:teuthology.orchestra.run.smithi059.stdout:55834574858 2024-06-29T14:22:15.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-29T14:22:15.762 INFO:teuthology.orchestra.run.smithi059.stdout:60129542154 2024-06-29T14:22:15.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-29T14:22:15.763 INFO:teuthology.orchestra.run.smithi059.stdout:60129542152 2024-06-29T14:22:15.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-29T14:22:15.764 INFO:teuthology.orchestra.run.smithi059.stdout:60129542153 2024-06-29T14:22:15.764 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-29T14:22:15.765 INFO:teuthology.orchestra.run.smithi059.stdout:60129542153 2024-06-29T14:22:15.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-29T14:22:16.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.030+0000 7f096e77e640 1 -- 172.21.15.59:0/1927018123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0954004800 msgr2=0x7f0954004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.030+0000 7f096e77e640 1 --2- 172.21.15.59:0/1927018123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0954004800 0x7f0954004c10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f095800c310 tx=0x7f0954005830 comp rx=0 tx=0).stop 2024-06-29T14:22:16.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/1263824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed00051a0 msgr2=0x7f4ed000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f4eeb2d6640 1 --2- 172.21.15.59:0/1263824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed00051a0 0x7f4ed000b9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4ee406b240 tx=0x7f4ed0005830 comp rx=0 tx=0).stop 2024-06-29T14:22:16.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f096e77e640 1 -- 172.21.15.59:0/1927018123 shutdown_connections 2024-06-29T14:22:16.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f096e77e640 1 --2- 172.21.15.59:0/1927018123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0954004800 0x7f0954004c10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f096e77e640 1 -- 172.21.15.59:0/1927018123 >> 172.21.15.59:0/1927018123 conn(0x7f096001db60 msgr2=0x7f0960039600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f096e77e640 1 -- 172.21.15.59:0/1927018123 shutdown_connections 2024-06-29T14:22:16.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f096e77e640 1 -- 172.21.15.59:0/1927018123 wait complete. 2024-06-29T14:22:16.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/1263824 shutdown_connections 2024-06-29T14:22:16.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f4eeb2d6640 1 --2- 172.21.15.59:0/1263824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ed00051a0 0x7f4ed000b9c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.031+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/1263824 >> 172.21.15.59:0/1263824 conn(0x7f4edc0362c0 msgr2=0x7f4edc0366d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.032+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/1263824 shutdown_connections 2024-06-29T14:22:16.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.032+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/1263824 wait complete. 2024-06-29T14:22:16.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.034+0000 7f873ce85640 1 -- 172.21.15.59:0/1467010349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 msgr2=0x7f8720003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.034+0000 7f873ce85640 1 --2- 172.21.15.59:0/1467010349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 0x7f8720003ee0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f872c00aa40 tx=0x7f8720005830 comp rx=0 tx=0).stop 2024-06-29T14:22:16.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.034+0000 7f8737fff640 1 -- 172.21.15.59:0/1467010349 reap_dead start 2024-06-29T14:22:16.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.034+0000 7f873ce85640 1 -- 172.21.15.59:0/1467010349 shutdown_connections 2024-06-29T14:22:16.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.034+0000 7f873ce85640 1 -- 172.21.15.59:0/1467010349 >> 172.21.15.59:0/1467010349 conn(0x7f87380700f0 msgr2=0x7f87380719b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.034+0000 7f873ce85640 1 -- 172.21.15.59:0/1467010349 shutdown_connections 2024-06-29T14:22:16.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.034+0000 7f096e77e640 1 Processor -- start 2024-06-29T14:22:16.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.035+0000 7f096e77e640 1 -- start start 2024-06-29T14:22:16.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.034+0000 7f873ce85640 1 -- 172.21.15.59:0/1467010349 wait complete. 2024-06-29T14:22:16.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.035+0000 7fe87a865640 1 -- 172.21.15.59:0/3783774103 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe86c008dc0 msgr2=0x7fe86c0e1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.035+0000 7fe87a865640 1 --2- 172.21.15.59:0/3783774103 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe86c008dc0 0x7fe86c0e1110 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe860009910 tx=0x7fe86002a5b0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.035+0000 7fe87a865640 1 -- 172.21.15.59:0/3783774103 shutdown_connections 2024-06-29T14:22:16.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.035+0000 7fe87a865640 1 --2- 172.21.15.59:0/3783774103 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe86c008dc0 0x7fe86c0e1110 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.035+0000 7fe87a865640 1 -- 172.21.15.59:0/3783774103 >> 172.21.15.59:0/3783774103 conn(0x7fe86c01eca0 msgr2=0x7fe86c03a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.035+0000 7fe87a865640 1 -- 172.21.15.59:0/3783774103 shutdown_connections 2024-06-29T14:22:16.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.035+0000 7fe87a865640 1 -- 172.21.15.59:0/3783774103 wait complete. 2024-06-29T14:22:16.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096e77e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0954004800 0x7f096001e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096e77e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f096004d580 0x7f096004a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1959057558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7000b0e0 msgr2=0x7f5a7000b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096e77e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0960021940 0x7f0960021d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096d77c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0954004800 0x7f096001e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096e77e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f096003a0a0 con 0x7f0954004800 2024-06-29T14:22:16.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096d77c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0954004800 0x7f096001e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51788/0 (socket says 172.21.15.59:51788) 2024-06-29T14:22:16.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096d77c640 1 -- 172.21.15.59:0/271890025 learned_addr learned my addr 172.21.15.59:0/271890025 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:16.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096e77e640 1 -- 172.21.15.59:0/271890025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0960039f20 con 0x7f0960021940 2024-06-29T14:22:16.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096e77e640 1 -- 172.21.15.59:0/271890025 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f09600075d0 con 0x7f096004d580 2024-06-29T14:22:16.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096df7d640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0960021940 0x7f0960021d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7ffb2bfff640 1 -- 172.21.15.59:0/3204400947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 msgr2=0x7ffb240079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1959057558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7000b0e0 0x7f5a7000b4f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a700533d0 tx=0x7f5a6c02a4f0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/3204400947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 0x7ffb240079a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400f390 tx=0x7ffb2002a580 comp rx=0 tx=0).stop 2024-06-29T14:22:16.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f096cf7b640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f096004d580 0x7f096004a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7ffb2bfff640 1 -- 172.21.15.59:0/3204400947 shutdown_connections 2024-06-29T14:22:16.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1959057558 shutdown_connections 2024-06-29T14:22:16.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1959057558 >> v2:172.21.15.153:3300/0 conn(0x7f5a70007b70 0x7f5a70007f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1959057558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7000b0e0 0x7f5a7000b4f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.037+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1959057558 >> 172.21.15.59:0/1959057558 conn(0x7f5a7001ec70 msgr2=0x7f5a7003a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/3204400947 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 0x7ffb240079a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/3204400947 >> v2:172.21.15.59:3300/0 conn(0x7ffb24054730 0x7ffb240df7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f873ce85640 1 Processor -- start 2024-06-29T14:22:16.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7ffb2bfff640 1 -- 172.21.15.59:0/3204400947 >> 172.21.15.59:0/3204400947 conn(0x7ffb2401e3f0 msgr2=0x7ffb24039e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f873ce85640 1 -- start start 2024-06-29T14:22:16.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f096df7d640 1 -- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f096004d580 msgr2=0x7f096004a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f096df7d640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f096004d580 0x7f096004a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f096df7d640 1 -- 172.21.15.59:0/271890025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0954004800 msgr2=0x7f096001e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f096df7d640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0954004800 0x7f096001e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f096df7d640 1 -- 172.21.15.59:0/271890025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0960022290 con 0x7f0960021940 2024-06-29T14:22:16.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7ffb2bfff640 1 -- 172.21.15.59:0/3204400947 shutdown_connections 2024-06-29T14:22:16.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7ffb2bfff640 1 -- 172.21.15.59:0/3204400947 wait complete. 2024-06-29T14:22:16.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1959057558 shutdown_connections 2024-06-29T14:22:16.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f60f2272640 1 -- 172.21.15.59:0/4089664942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec15a400 msgr2=0x7f60ec0cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.038+0000 7f60f2272640 1 --2- 172.21.15.59:0/4089664942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec15a400 0x7f60ec0cf120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f60e00098c0 tx=0x7f60e002c790 comp rx=0 tx=0).stop 2024-06-29T14:22:16.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f096df7d640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0960021940 0x7f0960021d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f09680516e0 tx=0x7f096806a640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f3d34a26640 1 -- 172.21.15.59:0/739704632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d30162510 msgr2=0x7f3d30168b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f3d34a26640 1 --2- 172.21.15.59:0/739704632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d30162510 0x7f3d30168b30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3d200097e0 tx=0x7f3d2002a380 comp rx=0 tx=0).stop 2024-06-29T14:22:16.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7fe87a865640 1 Processor -- start 2024-06-29T14:22:16.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f096d77c640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0954004800 0x7f096001e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:16.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f095dffb640 1 -- 172.21.15.59:0/271890025 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0968063d20 con 0x7f0960021940 2024-06-29T14:22:16.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f3d34a26640 1 -- 172.21.15.59:0/739704632 shutdown_connections 2024-06-29T14:22:16.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f3d34a26640 1 --2- 172.21.15.59:0/739704632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d30162510 0x7f3d30168b30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1959057558 wait complete. 2024-06-29T14:22:16.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f3d34a26640 1 -- 172.21.15.59:0/739704632 >> 172.21.15.59:0/739704632 conn(0x7f3d300d3aa0 msgr2=0x7f3d300d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7fe87a865640 1 -- start start 2024-06-29T14:22:16.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f095dffb640 1 -- 172.21.15.59:0/271890025 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0968075100 con 0x7f0960021940 2024-06-29T14:22:16.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f3d34a26640 1 -- 172.21.15.59:0/739704632 shutdown_connections 2024-06-29T14:22:16.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f096e77e640 1 -- 172.21.15.59:0/271890025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096004b8b0 con 0x7f0960021940 2024-06-29T14:22:16.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f3d34a26640 1 -- 172.21.15.59:0/739704632 wait complete. 2024-06-29T14:22:16.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.039+0000 7f096e77e640 1 -- 172.21.15.59:0/271890025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096004be70 con 0x7f0960021940 2024-06-29T14:22:16.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.040+0000 7f095dffb640 1 -- 172.21.15.59:0/271890025 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09680691d0 con 0x7f0960021940 2024-06-29T14:22:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7f873ce85640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 0x7f87381683b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe87a865640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c008dc0 0x7fe86c0de030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe87a865640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c0e1650 0x7fe86c185f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7f873ce85640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8738165900 0x7f87381688f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe87a865640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe86c186490 0x7fe86c18a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe87a865640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c008450 con 0x7fe86c008dc0 2024-06-29T14:22:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe87a865640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0082d0 con 0x7fe86c0e1650 2024-06-29T14:22:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe87a865640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c0085d0 con 0x7fe86c186490 2024-06-29T14:22:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe873fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c008dc0 0x7fe86c0de030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe873fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c008dc0 0x7fe86c0de030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51800/0 (socket says 172.21.15.59:51800) 2024-06-29T14:22:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe873fff640 1 -- 172.21.15.59:0/4125961603 learned_addr learned my addr 172.21.15.59:0/4125961603 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:16.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7fe8737fe640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c0e1650 0x7fe86c185f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7f87377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 0x7f87381683b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7fe878e02640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe86c186490 0x7fe86c18a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7ffb2bfff640 1 Processor -- start 2024-06-29T14:22:16.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7fe873fff640 1 -- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe86c186490 msgr2=0x7fe86c18a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7fe873fff640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe86c186490 0x7fe86c18a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7fe873fff640 1 -- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c0e1650 msgr2=0x7fe86c185f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7fe873fff640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c0e1650 0x7fe86c185f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7fe873fff640 1 -- 172.21.15.59:0/4125961603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c0549b0 con 0x7fe86c008dc0 2024-06-29T14:22:16.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f095dffb640 1 -- 172.21.15.59:0/271890025 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0968069390 con 0x7f0960021940 2024-06-29T14:22:16.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7ffb2bfff640 1 -- start start 2024-06-29T14:22:16.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.041+0000 7f87377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 0x7f87381683b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51804/0 (socket says 172.21.15.59:51804) 2024-06-29T14:22:16.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f873ce85640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8738165dc0 0x7f8738205740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f873ce85640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f87380ae670 con 0x7f8720003a70 2024-06-29T14:22:16.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f873ce85640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87380ae4f0 con 0x7f8738165900 2024-06-29T14:22:16.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f873ce85640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f87380ae7f0 con 0x7f8738165dc0 2024-06-29T14:22:16.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f8737fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8738165dc0 0x7f8738205740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f60f2272640 1 -- 172.21.15.59:0/4089664942 shutdown_connections 2024-06-29T14:22:16.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f60f2272640 1 --2- 172.21.15.59:0/4089664942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60ec06c1e0 0x7f60ec06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f60f2272640 1 --2- 172.21.15.59:0/4089664942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec15a400 0x7f60ec0cf120 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f60f2272640 1 -- 172.21.15.59:0/4089664942 >> 172.21.15.59:0/4089664942 conn(0x7f60ec0b24c0 msgr2=0x7f60ec0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.040+0000 7f4eeb2d6640 1 Processor -- start 2024-06-29T14:22:16.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.040+0000 7f4eeb2d6640 1 -- start start 2024-06-29T14:22:16.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7fe873fff640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c008dc0 0x7fe86c0de030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe86002aba0 tx=0x7fe860004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f8737fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8738165dc0 0x7f8738205740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:49070/0 (socket says 172.21.15.59:49070) 2024-06-29T14:22:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.042+0000 7f8737fff640 1 -- 172.21.15.59:0/3921858657 learned_addr learned my addr 172.21.15.59:0/3921858657 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f5a7d3ac640 1 Processor -- start 2024-06-29T14:22:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f3d34a26640 1 Processor -- start 2024-06-29T14:22:16.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f8736ffd640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8738165900 0x7f87381688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f3d34a26640 1 -- start start 2024-06-29T14:22:16.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f8737fff640 1 -- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8738165900 msgr2=0x7f87381688f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7fe870ff9640 1 -- 172.21.15.59:0/4125961603 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86003f070 con 0x7fe86c008dc0 2024-06-29T14:22:16.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7fe870ff9640 1 -- 172.21.15.59:0/4125961603 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe860004260 con 0x7fe86c008dc0 2024-06-29T14:22:16.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7fe870ff9640 1 -- 172.21.15.59:0/4125961603 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860031600 con 0x7fe86c008dc0 2024-06-29T14:22:16.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7fe87a865640 1 -- 172.21.15.59:0/4125961603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c054c90 con 0x7fe86c008dc0 2024-06-29T14:22:16.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7fe87a865640 1 -- 172.21.15.59:0/4125961603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c055250 con 0x7fe86c008dc0 2024-06-29T14:22:16.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f5a7d3ac640 1 -- start start 2024-06-29T14:22:16.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f095dffb640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f094c0763a0 0x7f094c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f8737fff640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8738165900 0x7f87381688f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f4eeb2d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ed00051a0 0x7f4edc0de230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f8737fff640 1 -- 172.21.15.59:0/3921858657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 msgr2=0x7f87381683b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f4eeb2d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4edc038c60 0x7f4edc04bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f4eeb2d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4edc039070 0x7f4edc037600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f4eeb2d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4edc054470 con 0x7f4ed00051a0 2024-06-29T14:22:16.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f8737fff640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 0x7f87381683b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f8737fff640 1 -- 172.21.15.59:0/3921858657 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873816ca30 con 0x7f8738165dc0 2024-06-29T14:22:16.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f4eeb2d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4edc0542f0 con 0x7f4edc038c60 2024-06-29T14:22:16.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f4eeb2d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4edc0e0240 con 0x7f4edc039070 2024-06-29T14:22:16.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.043+0000 7f096e77e640 1 -- 172.21.15.59:0/271890025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0948005290 con 0x7f0960021940 2024-06-29T14:22:16.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f096d77c640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f094c0763a0 0x7f094c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f095dffb640 1 -- 172.21.15.59:0/271890025 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0968112060 con 0x7f0960021940 2024-06-29T14:22:16.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f4ee9873640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4edc039070 0x7f4edc037600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f4ee9873640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4edc039070 0x7f4edc037600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:49086/0 (socket says 172.21.15.59:49086) 2024-06-29T14:22:16.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f4ee9873640 1 -- 172.21.15.59:0/2974506445 learned_addr learned my addr 172.21.15.59:0/2974506445 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:16.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f60f2272640 1 -- 172.21.15.59:0/4089664942 shutdown_connections 2024-06-29T14:22:16.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f60f2272640 1 -- 172.21.15.59:0/4089664942 wait complete. 2024-06-29T14:22:16.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.044+0000 7f096d77c640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f094c0763a0 0x7f094c0788b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f096400b690 tx=0x7f096400b620 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f8737fff640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8738165dc0 0x7f8738205740 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87300016b0 tx=0x7f8730006420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f4ee9072640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ed00051a0 0x7f4edc0de230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7fe87a865640 1 -- 172.21.15.59:0/4125961603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86c0e1110 con 0x7fe86c008dc0 2024-06-29T14:22:16.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f871ffff640 1 -- 172.21.15.59:0/3921858657 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730015020 con 0x7f8738165dc0 2024-06-29T14:22:16.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f871ffff640 1 -- 172.21.15.59:0/3921858657 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8730010980 con 0x7f8738165dc0 2024-06-29T14:22:16.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f3d34a26640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d300b2a30 0x7f3d300b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f871ffff640 1 -- 172.21.15.59:0/3921858657 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730010d90 con 0x7f8738165dc0 2024-06-29T14:22:16.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87381665a0 con 0x7f8738165dc0 2024-06-29T14:22:16.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f5a7d3ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a70007b70 0x7f5a700d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb24007ee0 0x7ffb2404e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f4ee9072640 1 -- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4edc039070 msgr2=0x7f4edc037600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f4ee9072640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4edc039070 0x7f4edc037600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f4ee9072640 1 -- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4edc038c60 msgr2=0x7f4edc04bd10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:16.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f4ee9072640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4edc038c60 0x7f4edc04bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.045+0000 7f4ee9072640 1 -- 172.21.15.59:0/2974506445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4edc0db4b0 con 0x7f4ed00051a0 2024-06-29T14:22:16.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 0x7ffb240dad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f873816bf50 con 0x7f8738165dc0 2024-06-29T14:22:16.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb24054730 0x7ffb240db250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2bfff640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ffb24006150 con 0x7ffb24054730 2024-06-29T14:22:16.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f87377fe640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 0x7f87381683b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:16.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2bfff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffb24005fd0 con 0x7ffb2403a5a0 2024-06-29T14:22:16.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2bfff640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffb240062d0 con 0x7ffb24007ee0 2024-06-29T14:22:16.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f4ee9072640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ed00051a0 0x7f4edc0de230 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4edc055710 tx=0x7f4ed400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f3d34a26640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d301619f0 0x7f3d30157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f3d34a26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d30162510 0x7f3d301f80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f3d34a26640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3d30169280 con 0x7f3d300b2a30 2024-06-29T14:22:16.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7fe870ff9640 1 -- 172.21.15.59:0/4125961603 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe860042070 con 0x7fe86c008dc0 2024-06-29T14:22:16.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f5a7d3ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70008d70 0x7f5a7004ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2a7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 0x7ffb240dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f3d2e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d300b2a30 0x7f3d300b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f5a76ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a70007b70 0x7f5a700d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f3d2e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d300b2a30 0x7f3d300b7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51824/0 (socket says 172.21.15.59:51824) 2024-06-29T14:22:16.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f5a76ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a70007b70 0x7f5a700d0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:46064/0 (socket says 172.21.15.59:46064) 2024-06-29T14:22:16.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2a7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 0x7ffb240dad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:46068/0 (socket says 172.21.15.59:46068) 2024-06-29T14:22:16.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7ffb2a7fc640 1 -- 172.21.15.59:0/2068155458 learned_addr learned my addr 172.21.15.59:0/2068155458 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:16.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f4ed9ffb640 1 -- 172.21.15.59:0/2974506445 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4017070 con 0x7f4ed00051a0 2024-06-29T14:22:16.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f3d2e59c640 1 -- 172.21.15.59:0/505657816 learned_addr learned my addr 172.21.15.59:0/505657816 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:16.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3d300b4f00 con 0x7f3d30162510 2024-06-29T14:22:16.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3d30169400 con 0x7f3d301619f0 2024-06-29T14:22:16.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f5a76ffd640 1 -- 172.21.15.59:0/1717974785 learned_addr learned my addr 172.21.15.59:0/1717974785 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:16.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f4ed9ffb640 1 -- 172.21.15.59:0/2974506445 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ed40053c0 con 0x7f4ed00051a0 2024-06-29T14:22:16.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.046+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a7000b0e0 0x7f5a70049020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4edc0dd470 con 0x7f4ed00051a0 2024-06-29T14:22:16.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7fe870ff9640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8580762d0 0x7fe8580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7ffb2a7fc640 1 -- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb24007ee0 msgr2=0x7ffb2404e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7ffb2a7fc640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb24007ee0 0x7ffb2404e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7ffb2a7fc640 1 -- 172.21.15.59:0/2068155458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb24054730 msgr2=0x7ffb240db250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:22:16.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7ffb2a7fc640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb24054730 0x7ffb240db250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7ffb2a7fc640 1 -- 172.21.15.59:0/2068155458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2404e9f0 con 0x7ffb2403a5a0 2024-06-29T14:22:16.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7fe870ff9640 1 -- 172.21.15.59:0/4125961603 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe8600b8820 con 0x7fe86c008dc0 2024-06-29T14:22:16.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7fe8737fe640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8580762d0 0x7fe8580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f3d2e59c640 1 -- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d301619f0 msgr2=0x7f3d30157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7ffb2a7fc640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 0x7ffb240dad10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb20009780 tx=0x7ffb200316c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f3d2e59c640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d301619f0 0x7f3d30157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4edc0dda30 con 0x7f4ed00051a0 2024-06-29T14:22:16.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f4ed9ffb640 1 -- 172.21.15.59:0/2974506445 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4003850 con 0x7f4ed00051a0 2024-06-29T14:22:16.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f3d2e59c640 1 -- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d30162510 msgr2=0x7f3d301f80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f3d2e59c640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d30162510 0x7f3d301f80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f3d2e59c640 1 -- 172.21.15.59:0/505657816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d300b8410 con 0x7f3d300b2a30 2024-06-29T14:22:16.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7fe8737fe640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8580762d0 0x7fe8580787e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0343c0 tx=0x7fe868009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f3d2e59c640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d300b2a30 0x7f3d300b7950 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400c730 tx=0x7f3d2400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f5a767fc640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70008d70 0x7f5a7004ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.047+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700dddc0 con 0x7f5a70008d70 2024-06-29T14:22:16.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700ddc40 con 0x7f5a70007b70 2024-06-29T14:22:16.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5a700084c0 con 0x7f5a7000b0e0 2024-06-29T14:22:16.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f5a777fe640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a7000b0e0 0x7f5a70049020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f873816dd80 con 0x7f8738165dc0 2024-06-29T14:22:16.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f60f2272640 1 Processor -- start 2024-06-29T14:22:16.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f871ffff640 1 -- 172.21.15.59:0/3921858657 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8730018020 con 0x7f8738165dc0 2024-06-29T14:22:16.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.048+0000 7f60f2272640 1 -- start start 2024-06-29T14:22:16.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4edc00bf20 con 0x7f4ed00051a0 2024-06-29T14:22:16.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f5a76ffd640 1 -- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a7000b0e0 msgr2=0x7f5a70049020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f5a76ffd640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a7000b0e0 0x7f5a70049020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f5a76ffd640 1 -- 172.21.15.59:0/1717974785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70008d70 msgr2=0x7f5a7004ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f5a76ffd640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70008d70 0x7f5a7004ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f5a76ffd640 1 -- 172.21.15.59:0/1717974785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7004f320 con 0x7f5a70007b70 2024-06-29T14:22:16.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f5a767fc640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70008d70 0x7f5a7004ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:16.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f871ffff640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8724076320 0x7f8724078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f5a76ffd640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a70007b70 0x7f5a700d0ec0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800c410 tx=0x7f5a6800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f87377fe640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8724076320 0x7f8724078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.049+0000 7f871ffff640 1 -- 172.21.15.59:0/3921858657 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f87300982b0 con 0x7f8738165dc0 2024-06-29T14:22:16.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f87377fe640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8724076320 0x7f8724078830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f872c033000 tx=0x7f872c004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60f2272640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec06c1e0 0x7f60ec14d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60f2272640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60ec073f30 0x7f60ec14c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60f2272640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60ec15a400 0x7f60ec14b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7ffb137fe640 1 -- 172.21.15.59:0/2068155458 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20031d60 con 0x7ffb2403a5a0 2024-06-29T14:22:16.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7ffb137fe640 1 -- 172.21.15.59:0/2068155458 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb20042070 con 0x7ffb2403a5a0 2024-06-29T14:22:16.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb24034a60 con 0x7ffb2403a5a0 2024-06-29T14:22:16.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7ffb137fe640 1 -- 172.21.15.59:0/2068155458 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20005260 con 0x7ffb2403a5a0 2024-06-29T14:22:16.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb24035020 con 0x7ffb2403a5a0 2024-06-29T14:22:16.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f5a5f7fe640 1 -- 172.21.15.59:0/1717974785 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68017070 con 0x7f5a70007b70 2024-06-29T14:22:16.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60ebfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec06c1e0 0x7f60ec14d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60eb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60ec073f30 0x7f60ec14c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60eb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60ec073f30 0x7f60ec14c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:46082/0 (socket says 172.21.15.59:46082) 2024-06-29T14:22:16.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60eb7fe640 1 -- 172.21.15.59:0/1317671607 learned_addr learned my addr 172.21.15.59:0/1317671607 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:16.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec0cce40 con 0x7f60ec06c1e0 2024-06-29T14:22:16.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60ebfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec06c1e0 0x7f60ec14d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51836/0 (socket says 172.21.15.59:51836) 2024-06-29T14:22:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec0cccc0 con 0x7f60ec073f30 2024-06-29T14:22:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f60ec0ccfc0 con 0x7f60ec15a400 2024-06-29T14:22:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f60f080f640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60ec15a400 0x7f60ec14b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.050+0000 7f5a5f7fe640 1 -- 172.21.15.59:0/1717974785 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a680035f0 con 0x7f5a70007b70 2024-06-29T14:22:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7004b250 con 0x7f5a70007b70 2024-06-29T14:22:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a7004c920 con 0x7f5a70007b70 2024-06-29T14:22:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f095dffb640 1 -- 172.21.15.59:0/271890025 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0960022290 con 0x7f0960021940 2024-06-29T14:22:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f5a5f7fe640 1 -- 172.21.15.59:0/1717974785 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68017070 con 0x7f5a70007b70 2024-06-29T14:22:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f60ebfff640 1 -- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60ec15a400 msgr2=0x7f60ec14b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f60ebfff640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60ec15a400 0x7f60ec14b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f60ebfff640 1 -- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60ec073f30 msgr2=0x7f60ec14c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f60ebfff640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60ec073f30 0x7f60ec14c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f60ebfff640 1 -- 172.21.15.59:0/1317671607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec14c050 con 0x7f60ec06c1e0 2024-06-29T14:22:16.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.051+0000 7f60ebfff640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec06c1e0 0x7f60ec14d6a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00c3f0 tx=0x7f60dc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.052+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb2404ebc0 con 0x7ffb2403a5a0 2024-06-29T14:22:16.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.052+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a70007f80 con 0x7f5a70007b70 2024-06-29T14:22:16.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.053+0000 7ffb137fe640 1 -- 172.21.15.59:0/2068155458 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffb20005400 con 0x7ffb2403a5a0 2024-06-29T14:22:16.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.054+0000 7f5a5f7fe640 1 -- 172.21.15.59:0/1717974785 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5a68007020 con 0x7f5a70007b70 2024-06-29T14:22:16.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.054+0000 7fe870ff9640 1 -- 172.21.15.59:0/4125961603 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe8600817f0 con 0x7fe86c008dc0 2024-06-29T14:22:16.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.054+0000 7f3d1effd640 1 -- 172.21.15.59:0/505657816 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24017070 con 0x7f3d300b2a30 2024-06-29T14:22:16.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.054+0000 7f60e8ff9640 1 -- 172.21.15.59:0/1317671607 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc011790 con 0x7f60ec06c1e0 2024-06-29T14:22:16.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f60e8ff9640 1 -- 172.21.15.59:0/1317671607 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60dc017070 con 0x7f60ec06c1e0 2024-06-29T14:22:16.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f60e8ff9640 1 -- 172.21.15.59:0/1317671607 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc011ca0 con 0x7f60ec06c1e0 2024-06-29T14:22:16.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.054+0000 7f3d1effd640 1 -- 172.21.15.59:0/505657816 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d24005360 con 0x7f3d300b2a30 2024-06-29T14:22:16.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d300b86f0 con 0x7f3d300b2a30 2024-06-29T14:22:16.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d300b8cb0 con 0x7f3d300b2a30 2024-06-29T14:22:16.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f5a5f7fe640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5a64076500 0x7f5a64078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f5a5f7fe640 1 -- 172.21.15.59:0/1717974785 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f5a680969b0 con 0x7f5a70007b70 2024-06-29T14:22:16.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.054+0000 7ffb137fe640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ffb18076320 0x7ffb18078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.054+0000 7ffb137fe640 1 -- 172.21.15.59:0/2068155458 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ffb200bcb30 con 0x7ffb2403a5a0 2024-06-29T14:22:16.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7ffb2affd640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ffb18076320 0x7ffb18078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f3d1effd640 1 -- 172.21.15.59:0/505657816 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24017070 con 0x7f3d300b2a30 2024-06-29T14:22:16.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ec14ba90 con 0x7f60ec06c1e0 2024-06-29T14:22:16.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.055+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ec15b160 con 0x7f60ec06c1e0 2024-06-29T14:22:16.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.056+0000 7f5a767fc640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5a64076500 0x7f5a64078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.056+0000 7f871ffff640 1 -- 172.21.15.59:0/3921858657 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f873001f030 con 0x7f8738165dc0 2024-06-29T14:22:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.056+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3015dd00 con 0x7f3d300b2a30 2024-06-29T14:22:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.057+0000 7f5a767fc640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5a64076500 0x7f5a64078a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c033000 tx=0x7f5a6c004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.057+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ec06de40 con 0x7f60ec06c1e0 2024-06-29T14:22:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.058+0000 7ffb2affd640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ffb18076320 0x7ffb18078830 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c005ab0 tx=0x7ffb1c005a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.058+0000 7f4ed9ffb640 1 -- 172.21.15.59:0/2974506445 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4ed401a070 con 0x7f4ed00051a0 2024-06-29T14:22:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.058+0000 7f4ed9ffb640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4ec80763d0 0x7f4ec80788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.059+0000 7f4ed9ffb640 1 -- 172.21.15.59:0/2974506445 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f4ed40971a0 con 0x7f4ed00051a0 2024-06-29T14:22:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.059+0000 7f4ee8871640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4ec80763d0 0x7f4ec80788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.059+0000 7ffb137fe640 1 -- 172.21.15.59:0/2068155458 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffb200860b0 con 0x7ffb2403a5a0 2024-06-29T14:22:16.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.059+0000 7f4ee8871640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4ec80763d0 0x7f4ec80788e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4edc038c10 tx=0x7f4ee000c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.061+0000 7f4ed9ffb640 1 -- 172.21.15.59:0/2974506445 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ed4021020 con 0x7f4ed00051a0 2024-06-29T14:22:16.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.063+0000 7f60e8ff9640 1 -- 172.21.15.59:0/1317671607 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f60dc026430 con 0x7f60ec06c1e0 2024-06-29T14:22:16.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.064+0000 7f60e8ff9640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f60d80763d0 0x7f60d80788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.064+0000 7f60e8ff9640 1 -- 172.21.15.59:0/1317671607 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f60dc0983a0 con 0x7f60ec06c1e0 2024-06-29T14:22:16.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.064+0000 7f3d1effd640 1 -- 172.21.15.59:0/505657816 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3d2401a070 con 0x7f3d300b2a30 2024-06-29T14:22:16.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.065+0000 7f3d1effd640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3d100764d0 0x7f3d100789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:16.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.065+0000 7f60eb7fe640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f60d80763d0 0x7f60d80788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.065+0000 7f3d2dd9b640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3d100764d0 0x7f3d100789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:16.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.065+0000 7f3d1effd640 1 -- 172.21.15.59:0/505657816 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f3d2405c830 con 0x7f3d300b2a30 2024-06-29T14:22:16.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.064+0000 7f5a5f7fe640 1 -- 172.21.15.59:0/1717974785 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5a6805f870 con 0x7f5a70007b70 2024-06-29T14:22:16.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.066+0000 7f3d2dd9b640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3d100764d0 0x7f3d100789e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3d301f8020 tx=0x7f3d28007dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.066+0000 7f60eb7fe640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f60d80763d0 0x7f60d80788e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f60ec164210 tx=0x7f60e00046c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:16.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.069+0000 7f60e8ff9640 1 -- 172.21.15.59:0/1317671607 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f60ec15a810 con 0x7f60ec06c1e0 2024-06-29T14:22:16.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.078+0000 7f3d1effd640 1 -- 172.21.15.59:0/505657816 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3d24060240 con 0x7f3d300b2a30 2024-06-29T14:22:16.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.500+0000 7f096e77e640 1 -- 172.21.15.59:0/271890025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0948005580 con 0x7f0960021940 2024-06-29T14:22:16.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.501+0000 7f095dffb640 1 -- 172.21.15.59:0/271890025 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f09680db010 con 0x7f0960021940 2024-06-29T14:22:16.499 INFO:teuthology.orchestra.run.smithi059.stdout:64424509449 2024-06-29T14:22:16.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.504+0000 7fe87a865640 1 -- 172.21.15.59:0/4125961603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe86c0db0e0 con 0x7fe86c008dc0 2024-06-29T14:22:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.504+0000 7f09537fe640 1 -- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f094c0763a0 msgr2=0x7f094c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.504+0000 7f09537fe640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f094c0763a0 0x7f094c0788b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f096400b690 tx=0x7f096400b620 comp rx=0 tx=0).stop 2024-06-29T14:22:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.504+0000 7f09537fe640 1 -- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0960021940 msgr2=0x7f0960021d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.504+0000 7fe870ff9640 1 -- 172.21.15.59:0/4125961603 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe860085af0 con 0x7fe86c008dc0 2024-06-29T14:22:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.505+0000 7f09537fe640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0960021940 0x7f0960021d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f09680516e0 tx=0x7f096806a640 comp rx=0 tx=0).stop 2024-06-29T14:22:16.503 INFO:teuthology.orchestra.run.smithi059.stdout:55834574856 2024-06-29T14:22:16.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.505+0000 7f09537fe640 1 -- 172.21.15.59:0/271890025 shutdown_connections 2024-06-29T14:22:16.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.505+0000 7f09537fe640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f094c0763a0 0x7f094c0788b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.505+0000 7f09537fe640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0960021940 0x7f0960021d50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.505+0000 7f09537fe640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f096004d580 0x7f096004a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.505+0000 7f09537fe640 1 --2- 172.21.15.59:0/271890025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0954004800 0x7f096001e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.505+0000 7f09537fe640 1 -- 172.21.15.59:0/271890025 >> 172.21.15.59:0/271890025 conn(0x7f096001db60 msgr2=0x7f09600e2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.505+0000 7f09537fe640 1 -- 172.21.15.59:0/271890025 shutdown_connections 2024-06-29T14:22:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.506+0000 7f09537fe640 1 -- 172.21.15.59:0/271890025 wait complete. 2024-06-29T14:22:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.508+0000 7fe85e7fc640 1 -- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8580762d0 msgr2=0x7fe8580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.508+0000 7fe85e7fc640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8580762d0 0x7fe8580787e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0343c0 tx=0x7fe868009040 comp rx=0 tx=0).stop 2024-06-29T14:22:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.508+0000 7fe85e7fc640 1 -- 172.21.15.59:0/4125961603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c008dc0 msgr2=0x7fe86c0de030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.508+0000 7fe85e7fc640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c008dc0 0x7fe86c0de030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe86002aba0 tx=0x7fe860004dc0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.508+0000 7fe85e7fc640 1 -- 172.21.15.59:0/4125961603 shutdown_connections 2024-06-29T14:22:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.508+0000 7fe85e7fc640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe8580762d0 0x7fe8580787e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.508+0000 7fe85e7fc640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe86c186490 0x7fe86c18a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.509+0000 7fe85e7fc640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c0e1650 0x7fe86c185f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.509+0000 7fe85e7fc640 1 --2- 172.21.15.59:0/4125961603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c008dc0 0x7fe86c0de030 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.509+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f87380ce7c0 con 0x7f8738165dc0 2024-06-29T14:22:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.509+0000 7fe85e7fc640 1 -- 172.21.15.59:0/4125961603 >> 172.21.15.59:0/4125961603 conn(0x7fe86c01eca0 msgr2=0x7fe86c034950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.509+0000 7fe85e7fc640 1 -- 172.21.15.59:0/4125961603 shutdown_connections 2024-06-29T14:22:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.509+0000 7fe85e7fc640 1 -- 172.21.15.59:0/4125961603 wait complete. 2024-06-29T14:22:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.509+0000 7f871ffff640 1 -- 172.21.15.59:0/3921858657 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f873816aee0 con 0x7f8738165dc0 2024-06-29T14:22:16.508 INFO:teuthology.orchestra.run.smithi059.stdout:64424509450 2024-06-29T14:22:16.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.512+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4edc032b60 con 0x7f4ed00051a0 2024-06-29T14:22:16.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8724076320 msgr2=0x7f8724078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8724076320 0x7f8724078830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f872c033000 tx=0x7f872c004730 comp rx=0 tx=0).stop 2024-06-29T14:22:16.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f4ed9ffb640 1 -- 172.21.15.59:0/2974506445 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4ed40601e0 con 0x7f4ed00051a0 2024-06-29T14:22:16.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ffb240dfae0 con 0x7ffb2403a5a0 2024-06-29T14:22:16.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8738165dc0 msgr2=0x7f8738205740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8738165dc0 0x7f8738205740 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87300016b0 tx=0x7f8730006420 comp rx=0 tx=0).stop 2024-06-29T14:22:16.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 shutdown_connections 2024-06-29T14:22:16.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8724076320 0x7f8724078830 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.512 INFO:teuthology.orchestra.run.smithi059.stdout:60129542153 2024-06-29T14:22:16.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8738165dc0 0x7f8738205740 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8738165900 0x7f87381688f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 --2- 172.21.15.59:0/3921858657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8720003a70 0x7f87381683b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.513+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 >> 172.21.15.59:0/3921858657 conn(0x7f87380700f0 msgr2=0x7f8738073640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.514+0000 7ffb137fe640 1 -- 172.21.15.59:0/2068155458 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffb2008e0a0 con 0x7ffb2403a5a0 2024-06-29T14:22:16.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.514+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 shutdown_connections 2024-06-29T14:22:16.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.514+0000 7f873ce85640 1 -- 172.21.15.59:0/3921858657 wait complete. 2024-06-29T14:22:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:64424509448 2024-06-29T14:22:16.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.514+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5a700dbd50 con 0x7f5a70007b70 2024-06-29T14:22:16.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.515+0000 7f5a5f7fe640 1 -- 172.21.15.59:0/1717974785 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5a68063b70 con 0x7f5a70007b70 2024-06-29T14:22:16.515 INFO:teuthology.orchestra.run.smithi059.stdout:60129542152 2024-06-29T14:22:16.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.517+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ffb18076320 msgr2=0x7ffb18078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4ec80763d0 msgr2=0x7f4ec80788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4ec80763d0 0x7f4ec80788e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4edc038c10 tx=0x7f4ee000c040 comp rx=0 tx=0).stop 2024-06-29T14:22:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ffb18076320 0x7ffb18078830 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c005ab0 tx=0x7ffb1c005a20 comp rx=0 tx=0).stop 2024-06-29T14:22:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ed00051a0 msgr2=0x7f4edc0de230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ed00051a0 0x7f4edc0de230 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4edc055710 tx=0x7f4ed400c890 comp rx=0 tx=0).stop 2024-06-29T14:22:16.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 msgr2=0x7ffb240dad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 0x7ffb240dad10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb20009780 tx=0x7ffb200316c0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 shutdown_connections 2024-06-29T14:22:16.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4ec80763d0 0x7f4ec80788e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4edc039070 0x7f4edc037600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4edc038c60 0x7f4edc04bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 --2- 172.21.15.59:0/2974506445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ed00051a0 0x7f4edc0de230 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 >> 172.21.15.59:0/2974506445 conn(0x7f4edc0362c0 msgr2=0x7f4edc0053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 shutdown_connections 2024-06-29T14:22:16.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 shutdown_connections 2024-06-29T14:22:16.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7f4eeb2d6640 1 -- 172.21.15.59:0/2974506445 wait complete. 2024-06-29T14:22:16.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ffb18076320 0x7ffb18078830 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.518+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb24054730 0x7ffb240db250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb2403a5a0 0x7ffb240dad10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7ffb2bfff640 1 --2- 172.21.15.59:0/2068155458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffb24007ee0 0x7ffb2404e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 >> 172.21.15.59:0/2068155458 conn(0x7ffb2401e3f0 msgr2=0x7ffb24005630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 shutdown_connections 2024-06-29T14:22:16.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5a64076500 msgr2=0x7f5a64078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5a64076500 0x7f5a64078a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c033000 tx=0x7f5a6c004730 comp rx=0 tx=0).stop 2024-06-29T14:22:16.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7ffb2bfff640 1 -- 172.21.15.59:0/2068155458 wait complete. 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a70007b70 msgr2=0x7f5a700d0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a70007b70 0x7f5a700d0ec0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800c410 tx=0x7f5a6800c8d0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 shutdown_connections 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5a64076500 0x7f5a64078a10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a7000b0e0 0x7f5a70049020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a70008d70 0x7f5a7004ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 --2- 172.21.15.59:0/1717974785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a70007b70 0x7f5a700d0ec0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 >> 172.21.15.59:0/1717974785 conn(0x7f5a7001ec70 msgr2=0x7f5a70054bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 shutdown_connections 2024-06-29T14:22:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.519+0000 7f5a7d3ac640 1 -- 172.21.15.59:0/1717974785 wait complete. 2024-06-29T14:22:16.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.522+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f60ec0b0c10 con 0x7f60ec06c1e0 2024-06-29T14:22:16.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.523+0000 7f60e8ff9640 1 -- 172.21.15.59:0/1317671607 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f60dc061910 con 0x7f60ec06c1e0 2024-06-29T14:22:16.523 INFO:teuthology.orchestra.run.smithi059.stdout:60129542152 2024-06-29T14:22:16.524 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.3 2024-06-29T14:22:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f60d80763d0 msgr2=0x7f60d80788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f60d80763d0 0x7f60d80788e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f60ec164210 tx=0x7f60e00046c0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec06c1e0 msgr2=0x7f60ec14d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec06c1e0 0x7f60ec14d6a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00c3f0 tx=0x7f60dc00c8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 shutdown_connections 2024-06-29T14:22:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f60d80763d0 0x7f60d80788e0 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f60ec164210 tx=0x7f60e00046c0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60ec15a400 0x7f60ec14b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60ec073f30 0x7f60ec14c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 --2- 172.21.15.59:0/1317671607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60ec06c1e0 0x7f60ec14d6a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.527+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 >> 172.21.15.59:0/1317671607 conn(0x7f60ec0b24c0 msgr2=0x7f60ec0b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.528+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 shutdown_connections 2024-06-29T14:22:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.528+0000 7f60f2272640 1 -- 172.21.15.59:0/1317671607 wait complete. 2024-06-29T14:22:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.532+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3d300ce120 con 0x7f3d300b2a30 2024-06-29T14:22:16.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.2 2024-06-29T14:22:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.535+0000 7f3d1effd640 1 -- 172.21.15.59:0/505657816 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3d24064540 con 0x7f3d300b2a30 2024-06-29T14:22:16.533 INFO:teuthology.orchestra.run.smithi059.stdout:60129542151 2024-06-29T14:22:16.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509450 for osd.0 2024-06-29T14:22:16.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3d100764d0 msgr2=0x7f3d100789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3d100764d0 0x7f3d100789e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3d301f8020 tx=0x7f3d28007dd0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d300b2a30 msgr2=0x7f3d300b7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:16.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d300b2a30 0x7f3d300b7950 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400c730 tx=0x7f3d2400cbf0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 shutdown_connections 2024-06-29T14:22:16.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3d100764d0 0x7f3d100789e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d30162510 0x7f3d301f80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d301619f0 0x7f3d30157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 --2- 172.21.15.59:0/505657816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d300b2a30 0x7f3d300b7950 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 >> 172.21.15.59:0/505657816 conn(0x7f3d300d3aa0 msgr2=0x7f3d300d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.538+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 shutdown_connections 2024-06-29T14:22:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:16.539+0000 7f3d34a26640 1 -- 172.21.15.59:0/505657816 wait complete. 2024-06-29T14:22:16.540 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.7 2024-06-29T14:22:16.541 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.6 2024-06-29T14:22:16.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.1 2024-06-29T14:22:16.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.5 2024-06-29T14:22:16.551 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.4 2024-06-29T14:22:17.525 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-29T14:22:17.532 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-29T14:22:17.534 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-29T14:22:17.541 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-29T14:22:17.541 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-29T14:22:17.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-29T14:22:17.546 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-29T14:22:17.552 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-29T14:22:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.740+0000 7fe3c7fff640 1 -- 172.21.15.59:0/1897092955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c80b2fb0 msgr2=0x7fe3c80b33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.740+0000 7fe3c7fff640 1 --2- 172.21.15.59:0/1897092955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c80b2fb0 0x7fe3c80b33c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80b2610 tx=0x7fe3c002aa70 comp rx=0 tx=0).stop 2024-06-29T14:22:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.741+0000 7fe3c7fff640 1 -- 172.21.15.59:0/1897092955 shutdown_connections 2024-06-29T14:22:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.741+0000 7fe3c7fff640 1 --2- 172.21.15.59:0/1897092955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c80b2fb0 0x7fe3c80b33c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.741+0000 7fe3c7fff640 1 --2- 172.21.15.59:0/1897092955 >> v2:172.21.15.153:3300/0 conn(0x7fe3c8141d40 0x7fe3c8142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.741+0000 7fe3c7fff640 1 -- 172.21.15.59:0/1897092955 >> 172.21.15.59:0/1897092955 conn(0x7fe3c80af930 msgr2=0x7fe3c80afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:17.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.741+0000 7fe3c7fff640 1 -- 172.21.15.59:0/1897092955 shutdown_connections 2024-06-29T14:22:17.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.741+0000 7fe3c7fff640 1 -- 172.21.15.59:0/1897092955 wait complete. 2024-06-29T14:22:17.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.744+0000 7fe3c7fff640 1 Processor -- start 2024-06-29T14:22:17.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.745+0000 7fe3c7fff640 1 -- start start 2024-06-29T14:22:17.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.747+0000 7fe3c7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c80b2fb0 0x7fe3c806d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.748+0000 7fe3c7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3c8141d40 0x7fe3c80ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.748+0000 7fe3c7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c8142690 0x7fe3c81f8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.748+0000 7fe3c67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3c8141d40 0x7fe3c80ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.748+0000 7fe3c67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3c8141d40 0x7fe3c80ac5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:49126/0 (socket says 172.21.15.59:49126) 2024-06-29T14:22:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.748+0000 7fe3c77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c8142690 0x7fe3c81f8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.749+0000 7fe3c77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c8142690 0x7fe3c81f8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51854/0 (socket says 172.21.15.59:51854) 2024-06-29T14:22:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.749+0000 7fe3c67fc640 1 -- 172.21.15.59:0/349229887 learned_addr learned my addr 172.21.15.59:0/349229887 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.749+0000 7fe3c7fff640 1 -- 172.21.15.59:0/349229887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c80cd6e0 con 0x7fe3c8142690 2024-06-29T14:22:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.749+0000 7fe3c7fff640 1 -- 172.21.15.59:0/349229887 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c80cd560 con 0x7fe3c80b2fb0 2024-06-29T14:22:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.749+0000 7fe3c7fff640 1 -- 172.21.15.59:0/349229887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c80cd860 con 0x7fe3c8141d40 2024-06-29T14:22:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.750+0000 7fe3c67fc640 1 -- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c80b2fb0 msgr2=0x7fe3c806d020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.750+0000 7fe3c67fc640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c80b2fb0 0x7fe3c806d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.750+0000 7fe3c67fc640 1 -- 172.21.15.59:0/349229887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c8142690 msgr2=0x7fe3c81f8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.750+0000 7fe3c67fc640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c8142690 0x7fe3c81f8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.750+0000 7fe3c67fc640 1 -- 172.21.15.59:0/349229887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c806d560 con 0x7fe3c8141d40 2024-06-29T14:22:17.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.750+0000 7fe3c77fe640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c8142690 0x7fe3c81f8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:17.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.751+0000 7fe3c67fc640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3c8141d40 0x7fe3c80ac5c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800c430 tx=0x7fe3b800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.751+0000 7fe3af7fe640 1 -- 172.21.15.59:0/349229887 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8017070 con 0x7fe3c8141d40 2024-06-29T14:22:17.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.751+0000 7fe3c7fff640 1 -- 172.21.15.59:0/349229887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c80ad5b0 con 0x7fe3c8141d40 2024-06-29T14:22:17.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.751+0000 7fe3c7fff640 1 -- 172.21.15.59:0/349229887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c80adb70 con 0x7fe3c8141d40 2024-06-29T14:22:17.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.752+0000 7fe3af7fe640 1 -- 172.21.15.59:0/349229887 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3b80053c0 con 0x7fe3c8141d40 2024-06-29T14:22:17.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.752+0000 7fe3af7fe640 1 -- 172.21.15.59:0/349229887 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8003830 con 0x7fe3c8141d40 2024-06-29T14:22:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.756+0000 7fe3af7fe640 1 -- 172.21.15.59:0/349229887 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe3b801a070 con 0x7fe3c8141d40 2024-06-29T14:22:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.757+0000 7fe3af7fe640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3b40763a0 0x7fe3b40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.757+0000 7fe3c6ffd640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3b40763a0 0x7fe3b40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.758+0000 7fe3af7fe640 1 -- 172.21.15.59:0/349229887 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe3b8096d60 con 0x7fe3c8141d40 2024-06-29T14:22:17.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.758+0000 7fe3c6ffd640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3b40763a0 0x7fe3b40788b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009610 tx=0x7fe3bc006b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.761+0000 7fe3c7fff640 1 -- 172.21.15.59:0/349229887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a8005290 con 0x7fe3c8141d40 2024-06-29T14:22:17.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.766+0000 7fe3af7fe640 1 -- 172.21.15.59:0/349229887 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe3b805fda0 con 0x7fe3c8141d40 2024-06-29T14:22:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.803+0000 7f55b30e1640 1 -- 172.21.15.59:0/3511834549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a40374f0 msgr2=0x7f55a40e1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.803+0000 7f55b30e1640 1 --2- 172.21.15.59:0/3511834549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a40374f0 0x7f55a40e1ae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f559c009830 tx=0x7f559c02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.803+0000 7f55b30e1640 1 -- 172.21.15.59:0/3511834549 shutdown_connections 2024-06-29T14:22:17.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.803+0000 7f55b30e1640 1 --2- 172.21.15.59:0/3511834549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a40374f0 0x7f55a40e1ae0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.803+0000 7f55b30e1640 1 -- 172.21.15.59:0/3511834549 >> 172.21.15.59:0/3511834549 conn(0x7f55a401e3f0 msgr2=0x7f55a4039e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:17.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.803+0000 7f55b30e1640 1 -- 172.21.15.59:0/3511834549 shutdown_connections 2024-06-29T14:22:17.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.804+0000 7f55b30e1640 1 -- 172.21.15.59:0/3511834549 wait complete. 2024-06-29T14:22:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.806+0000 7f0ab3fff640 1 -- 172.21.15.59:0/3259079063 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8006c50 con 0x7f0aa8004800 2024-06-29T14:22:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.806+0000 7f0ac348f640 1 -- 172.21.15.59:0/3259079063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa8004800 msgr2=0x7f0aa8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.806+0000 7f0ac348f640 1 --2- 172.21.15.59:0/3259079063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa8004800 0x7f0aa8004c10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8001050 tx=0x7f0aa8005830 comp rx=0 tx=0).stop 2024-06-29T14:22:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.806+0000 7f0ac348f640 1 -- 172.21.15.59:0/3259079063 shutdown_connections 2024-06-29T14:22:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.806+0000 7f0ac348f640 1 --2- 172.21.15.59:0/3259079063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa8004800 0x7f0aa8004c10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.806+0000 7f0ac348f640 1 -- 172.21.15.59:0/3259079063 >> 172.21.15.59:0/3259079063 conn(0x7f0abc072d80 msgr2=0x7f0abc073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.807+0000 7f0ac348f640 1 -- 172.21.15.59:0/3259079063 shutdown_connections 2024-06-29T14:22:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.807+0000 7fb7abb11640 1 -- 172.21.15.59:0/3871166442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40ad5d0 msgr2=0x7fb7a415a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.807+0000 7fb7abb11640 1 --2- 172.21.15.59:0/3871166442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40ad5d0 0x7fb7a415a3e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0005fd0 tx=0x7fb7a002a560 comp rx=0 tx=0).stop 2024-06-29T14:22:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.807+0000 7f0ac348f640 1 -- 172.21.15.59:0/3259079063 wait complete. 2024-06-29T14:22:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.807+0000 7fb7abb11640 1 -- 172.21.15.59:0/3871166442 shutdown_connections 2024-06-29T14:22:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.807+0000 7fb7abb11640 1 --2- 172.21.15.59:0/3871166442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7a40b4a30 0x7fb7a40cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.807+0000 7fb7abb11640 1 --2- 172.21.15.59:0/3871166442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40ad5d0 0x7fb7a415a3e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.807+0000 7fb7abb11640 1 -- 172.21.15.59:0/3871166442 >> 172.21.15.59:0/3871166442 conn(0x7fb7a40b1be0 msgr2=0x7fb7a40b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.808+0000 7fb7abb11640 1 -- 172.21.15.59:0/3871166442 shutdown_connections 2024-06-29T14:22:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.808+0000 7fb7abb11640 1 -- 172.21.15.59:0/3871166442 wait complete. 2024-06-29T14:22:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.808+0000 7f55b30e1640 1 Processor -- start 2024-06-29T14:22:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.808+0000 7f55b30e1640 1 -- start start 2024-06-29T14:22:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.808+0000 7fe79ffff640 1 -- 172.21.15.59:0/1719960789 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000a610 con 0x7fe7a80cd3d0 2024-06-29T14:22:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fe7af918640 1 -- 172.21.15.59:0/1719960789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cd3d0 msgr2=0x7fe7a80b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fe7af918640 1 --2- 172.21.15.59:0/1719960789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cd3d0 0x7fe7a80b3c10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe7a806f820 tx=0x7fe7a002b4b0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fc1795ef640 1 -- 172.21.15.59:0/2488422659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 msgr2=0x7fc16000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fc1795ef640 1 --2- 172.21.15.59:0/2488422659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 0x7fc16000b9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c710 tx=0x7fc160005830 comp rx=0 tx=0).stop 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fc1795ef640 1 -- 172.21.15.59:0/2488422659 shutdown_connections 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fc1795ef640 1 --2- 172.21.15.59:0/2488422659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 0x7fc16000b9c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fc1795ef640 1 -- 172.21.15.59:0/2488422659 >> 172.21.15.59:0/2488422659 conn(0x7fc1740ccbe0 msgr2=0x7fc1740ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fe7af918640 1 -- 172.21.15.59:0/1719960789 shutdown_connections 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fe7af918640 1 --2- 172.21.15.59:0/1719960789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cd3d0 0x7fe7a80b3c10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fe7af918640 1 --2- 172.21.15.59:0/1719960789 >> v2:172.21.15.59:3300/0 conn(0x7fe7a80cf380 0x7fe7a80ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.809+0000 7fe7af918640 1 -- 172.21.15.59:0/1719960789 >> 172.21.15.59:0/1719960789 conn(0x7fe7a80b24c0 msgr2=0x7fe7a80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7fc1795ef640 1 -- 172.21.15.59:0/2488422659 shutdown_connections 2024-06-29T14:22:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f328513f640 1 -- 172.21.15.59:0/1923647788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3270004140 msgr2=0x7f327000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7fc1795ef640 1 -- 172.21.15.59:0/2488422659 wait complete. 2024-06-29T14:22:17.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f328513f640 1 --2- 172.21.15.59:0/1923647788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3270004140 0x7f327000b9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f328006b370 tx=0x7f3270005830 comp rx=0 tx=0).stop 2024-06-29T14:22:17.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f0ac348f640 1 Processor -- start 2024-06-29T14:22:17.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f328513f640 1 -- 172.21.15.59:0/1923647788 shutdown_connections 2024-06-29T14:22:17.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f328513f640 1 --2- 172.21.15.59:0/1923647788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3270004140 0x7f327000b9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f0ac348f640 1 -- start start 2024-06-29T14:22:17.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f328513f640 1 -- 172.21.15.59:0/1923647788 >> 172.21.15.59:0/1923647788 conn(0x7f3278058dd0 msgr2=0x7f32780591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:17.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f328513f640 1 -- 172.21.15.59:0/1923647788 shutdown_connections 2024-06-29T14:22:17.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f328513f640 1 -- 172.21.15.59:0/1923647788 wait complete. 2024-06-29T14:22:17.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.810+0000 7f55b30e1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a40374f0 0x7f55a40d7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b30e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a40dfd30 0x7f55a40d6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b30e1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a4052b80 0x7f55a404e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b20df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a40374f0 0x7f55a40d7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b20df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a40374f0 0x7f55a40d7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51952/0 (socket says 172.21.15.59:51952) 2024-06-29T14:22:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7fb7abb11640 1 Processor -- start 2024-06-29T14:22:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7fb7abb11640 1 -- start start 2024-06-29T14:22:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7fe7af918640 1 -- 172.21.15.59:0/1719960789 shutdown_connections 2024-06-29T14:22:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7fe7af918640 1 -- 172.21.15.59:0/1719960789 wait complete. 2024-06-29T14:22:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f0ac348f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0aa8004800 0x7f0abc155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f0ac348f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0abc13e390 0x7f0abc13ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b30e1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f55a40e2020 con 0x7f55a40374f0 2024-06-29T14:22:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b30e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55a4034920 con 0x7f55a40dfd30 2024-06-29T14:22:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b30e1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f55a40e21a0 con 0x7f55a4052b80 2024-06-29T14:22:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b20df640 1 -- 172.21.15.59:0/3347313631 learned_addr learned my addr 172.21.15.59:0/3347313631 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b28e0640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a4052b80 0x7f55a404e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.811+0000 7f55b20df640 1 -- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a4052b80 msgr2=0x7f55a404e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f55b20df640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a4052b80 0x7f55a404e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f55b20df640 1 -- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a40dfd30 msgr2=0x7f55a40d6940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f55b20df640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a40dfd30 0x7f55a40d6940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f55b20df640 1 -- 172.21.15.59:0/3347313631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a40532f0 con 0x7f55a40374f0 2024-06-29T14:22:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f55b20df640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a40374f0 0x7f55a40d7490 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f55a8007d40 tx=0x7f55a800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f55a2ffd640 1 -- 172.21.15.59:0/3347313631 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8021070 con 0x7f55a40374f0 2024-06-29T14:22:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f0ac348f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0abc13b140 0x7f0abc13cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f0ac348f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0abc162210 con 0x7f0aa8004800 2024-06-29T14:22:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f0ac348f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0abc162090 con 0x7f0abc13b140 2024-06-29T14:22:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f0ac348f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0abc162390 con 0x7f0abc13e390 2024-06-29T14:22:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.812+0000 7f0ac0a2a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0abc13e390 0x7f0abc13ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f7d8a481640 1 -- 172.21.15.59:0/190543722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d840cf100 msgr2=0x7f7d8415b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f328513f640 1 Processor -- start 2024-06-29T14:22:17.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f0ac122b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0aa8004800 0x7f0abc155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f0ac122b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0aa8004800 0x7f0abc155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51954/0 (socket says 172.21.15.59:51954) 2024-06-29T14:22:17.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f0ac122b640 1 -- 172.21.15.59:0/2984869405 learned_addr learned my addr 172.21.15.59:0/2984869405 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:17.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7fc1795ef640 1 Processor -- start 2024-06-29T14:22:17.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fc1795ef640 1 -- start start 2024-06-29T14:22:17.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f328513f640 1 -- start start 2024-06-29T14:22:17.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7abb11640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a40b4310 0x7fb7a40c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7f0ac122b640 1 -- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0abc13e390 msgr2=0x7f0abc13ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7f0ac122b640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0abc13e390 0x7f0abc13ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7f0ac122b640 1 -- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0abc13b140 msgr2=0x7f0abc13cfd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:17.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7f0ac122b640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0abc13b140 0x7f0abc13cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7f0ac122b640 1 -- 172.21.15.59:0/2984869405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0abc13b550 con 0x7f0aa8004800 2024-06-29T14:22:17.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7abb11640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7a40b4a30 0x7fb7a414bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7a98ad640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a40b4310 0x7fb7a40c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fe7af918640 1 Processor -- start 2024-06-29T14:22:17.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7f0ac122b640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0aa8004800 0x7f0abc155bc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0aac004d90 tx=0x7f0aac001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7abb11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40afa30 0x7fb7a40afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7a98ad640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a40b4310 0x7fb7a40c53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:46190/0 (socket says 172.21.15.59:46190) 2024-06-29T14:22:17.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7a90ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7a40b4a30 0x7fb7a414bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7abb11640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a40ce220 con 0x7fb7a40afa30 2024-06-29T14:22:17.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7abb11640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a40ce0a0 con 0x7fb7a40b4310 2024-06-29T14:22:17.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7abb11640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb7a40ce3a0 con 0x7fb7a40b4a30 2024-06-29T14:22:17.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fb7a90ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7a40b4a30 0x7fb7a414bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:49178/0 (socket says 172.21.15.59:49178) 2024-06-29T14:22:17.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7fb7a98ad640 1 -- 172.21.15.59:0/786734648 learned_addr learned my addr 172.21.15.59:0/786734648 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:17.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7fb7aa0ae640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40afa30 0x7fb7a40afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7fe7af918640 1 -- start start 2024-06-29T14:22:17.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7f0ab1ffb640 1 -- 172.21.15.59:0/2984869405 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac001d40 con 0x7f0aa8004800 2024-06-29T14:22:17.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0abc06edf0 con 0x7f0aa8004800 2024-06-29T14:22:17.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f55a2ffd640 1 -- 172.21.15.59:0/3347313631 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55a800ae40 con 0x7f55a40374f0 2024-06-29T14:22:17.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f55a2ffd640 1 -- 172.21.15.59:0/3347313631 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8021070 con 0x7f55a40374f0 2024-06-29T14:22:17.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.813+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a40535d0 con 0x7f55a40374f0 2024-06-29T14:22:17.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0abc06f3b0 con 0x7f0aa8004800 2024-06-29T14:22:17.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7fb7a90ac640 1 -- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a40b4310 msgr2=0x7fb7a40c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7fb7a90ac640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a40b4310 0x7fb7a40c53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7fb7a90ac640 1 -- 172.21.15.59:0/786734648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40afa30 msgr2=0x7fb7a40afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.814+0000 7f7d8a481640 1 --2- 172.21.15.59:0/190543722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d840cf100 0x7f7d8415b750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009860 tx=0x7f7d7402a460 comp rx=0 tx=0).stop 2024-06-29T14:22:17.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7f7d8a481640 1 -- 172.21.15.59:0/190543722 shutdown_connections 2024-06-29T14:22:17.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7f7d8a481640 1 --2- 172.21.15.59:0/190543722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d840cf100 0x7f7d8415b750 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7f7d8a481640 1 -- 172.21.15.59:0/190543722 >> 172.21.15.59:0/190543722 conn(0x7f7d84072d80 msgr2=0x7f7d84073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:17.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7f0ab1ffb640 1 -- 172.21.15.59:0/2984869405 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0aac016070 con 0x7f0aa8004800 2024-06-29T14:22:17.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7fb7a90ac640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40afa30 0x7fb7a40afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7fb7a90ac640 1 -- 172.21.15.59:0/786734648 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a40c5b30 con 0x7fb7a40b4a30 2024-06-29T14:22:17.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.815+0000 7fb7aa0ae640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40afa30 0x7fb7a40afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:17.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f328513f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3270004140 0x7f32780dfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7fb7a90ac640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7a40b4a30 0x7fb7a414bc20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0031000 tx=0x7fb7a00039e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f328513f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32780db200 0x7f3278051100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f328513f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3278051640 0x7f3278038b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f328513f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3278053a50 con 0x7f3270004140 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f328513f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32780538d0 con 0x7f3278051640 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f328513f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3278053bd0 con 0x7f32780db200 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f327f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32780db200 0x7f3278051100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f327f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32780db200 0x7f3278051100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:49192/0 (socket says 172.21.15.59:49192) 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f327f7fe640 1 -- 172.21.15.59:0/2427284297 learned_addr learned my addr 172.21.15.59:0/2427284297 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f327ffff640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3270004140 0x7f32780dfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f32863a1640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3278051640 0x7f3278038b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f327f7fe640 1 -- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3278051640 msgr2=0x7f3278038b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7f327f7fe640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3278051640 0x7f3278038b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7f327f7fe640 1 -- 172.21.15.59:0/2427284297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3270004140 msgr2=0x7f32780dfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.816+0000 7f55a2ffd640 1 -- 172.21.15.59:0/3347313631 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f55a8024070 con 0x7f55a40374f0 2024-06-29T14:22:17.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fe7af918640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a80b3800 0x7fe7a814a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7f327f7fe640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3270004140 0x7f32780dfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7f327f7fe640 1 -- 172.21.15.59:0/2427284297 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32780dd140 con 0x7f32780db200 2024-06-29T14:22:17.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7f327ffff640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3270004140 0x7f32780dfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:17.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fb7967fc640 1 -- 172.21.15.59:0/786734648 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a003e030 con 0x7fb7a40b4a30 2024-06-29T14:22:17.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fe7af918640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cf380 0x7fe7a8155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7f327f7fe640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32780db200 0x7f3278051100 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f326c00fb50 tx=0x7f326c00d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fb7967fc640 1 -- 172.21.15.59:0/786734648 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7a00047a0 con 0x7fb7a40b4a30 2024-06-29T14:22:17.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fb7967fc640 1 -- 172.21.15.59:0/786734648 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0035040 con 0x7fb7a40b4a30 2024-06-29T14:22:17.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fb7abb11640 1 -- 172.21.15.59:0/786734648 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a40b0380 con 0x7fb7a40b4a30 2024-06-29T14:22:17.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fb7abb11640 1 -- 172.21.15.59:0/786734648 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a41f0f10 con 0x7fb7a40b4a30 2024-06-29T14:22:17.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fc1795ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 0x7fc17415e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fe7af918640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7a81e4f60 0x7fe7a81555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7fe7ad6b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a80b3800 0x7fe7a814a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7ad6b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a80b3800 0x7fe7a814a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51982/0 (socket says 172.21.15.59:51982) 2024-06-29T14:22:17.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7f55a2ffd640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55900766d0 0x7f5590078be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.817+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a4053b90 con 0x7f55a40374f0 2024-06-29T14:22:17.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7f327cff9640 1 -- 172.21.15.59:0/2427284297 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c010630 con 0x7f32780db200 2024-06-29T14:22:17.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7f55b18de640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55900766d0 0x7f5590078be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7f327cff9640 1 -- 172.21.15.59:0/2427284297 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f326c014070 con 0x7f32780db200 2024-06-29T14:22:17.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7f327cff9640 1 -- 172.21.15.59:0/2427284297 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c010a50 con 0x7f32780db200 2024-06-29T14:22:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fc172ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 0x7fc17415e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32780e0190 con 0x7f32780db200 2024-06-29T14:22:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32780dc080 con 0x7f32780db200 2024-06-29T14:22:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7f55a2ffd640 1 -- 172.21.15.59:0/3347313631 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f55a8019070 con 0x7f55a40374f0 2024-06-29T14:22:17.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fc172ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 0x7fc17415e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51998/0 (socket says 172.21.15.59:51998) 2024-06-29T14:22:17.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fc172ffd640 1 -- 172.21.15.59:0/3931939990 learned_addr learned my addr 172.21.15.59:0/3931939990 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:17.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fc1795ef640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1741f5770 0x7fc1740a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7fc1795ef640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc17413ef40 0x7fc1740b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc174071f80 con 0x7fc160005270 2024-06-29T14:22:17.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc174071e00 con 0x7fc1741f5770 2024-06-29T14:22:17.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc174072100 con 0x7fc17413ef40 2024-06-29T14:22:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a9c005290 con 0x7f0aa8004800 2024-06-29T14:22:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7f0ab1ffb640 1 -- 172.21.15.59:0/2984869405 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac010b20 con 0x7f0aa8004800 2024-06-29T14:22:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7f0ab1ffb640 1 -- 172.21.15.59:0/2984869405 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0aac010cc0 con 0x7f0aa8004800 2024-06-29T14:22:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7fb787fff640 1 -- 172.21.15.59:0/786734648 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb780005290 con 0x7fb7a40b4a30 2024-06-29T14:22:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7af918640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a8070020 con 0x7fe7a80b3800 2024-06-29T14:22:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7af918640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a806fea0 con 0x7fe7a80cf380 2024-06-29T14:22:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7aceb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cf380 0x7fe7a8155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7aceb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cf380 0x7fe7a8155090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:46210/0 (socket says 172.21.15.59:46210) 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7aceb3640 1 -- 172.21.15.59:0/327159006 learned_addr learned my addr 172.21.15.59:0/327159006 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7adeb5640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7a81e4f60 0x7fe7a81555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a815dee0 con 0x7fe7a81e4f60 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7adeb5640 1 -- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cf380 msgr2=0x7fe7a8155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7adeb5640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cf380 0x7fe7a8155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.818+0000 7fe7adeb5640 1 -- 172.21.15.59:0/327159006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a80b3800 msgr2=0x7fe7a814a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7fe7adeb5640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a80b3800 0x7fe7a814a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7fe7adeb5640 1 -- 172.21.15.59:0/327159006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a814add0 con 0x7fe7a81e4f60 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.819+0000 7fe7ad6b4640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a80b3800 0x7fe7a814a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-06-29T14:22:17.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7f55b18de640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55900766d0 0x7f5590078be0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f55ac06b240 tx=0x7f55ac06bb70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7f0ab1ffb640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0aa00763d0 0x7f0aa00788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fb7967fc640 1 -- 172.21.15.59:0/786734648 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb7a0041030 con 0x7fb7a40b4a30 2024-06-29T14:22:17.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fe7adeb5640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7a81e4f60 0x7fe7a81555d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0031000 tx=0x7fe7a0008aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fe79dffb640 1 -- 172.21.15.59:0/327159006 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0052020 con 0x7fe7a81e4f60 2024-06-29T14:22:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a81e5550 con 0x7fe7a81e4f60 2024-06-29T14:22:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fe79dffb640 1 -- 172.21.15.59:0/327159006 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7a0048040 con 0x7fe7a81e4f60 2024-06-29T14:22:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fe79dffb640 1 -- 172.21.15.59:0/327159006 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000a370 con 0x7fe7a81e4f60 2024-06-29T14:22:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3260005290 con 0x7f32780db200 2024-06-29T14:22:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7f0ac0a2a640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0aa00763d0 0x7f0aa00788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a81e5b10 con 0x7fe7a81e4f60 2024-06-29T14:22:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7f0ab1ffb640 1 -- 172.21.15.59:0/2984869405 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0aac0981a0 con 0x7f0aa8004800 2024-06-29T14:22:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fc1727fc640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1741f5770 0x7fc1740a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fc1737fe640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc17413ef40 0x7fc1740b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fc172ffd640 1 -- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc17413ef40 msgr2=0x7fc1740b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fc172ffd640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc17413ef40 0x7fc1740b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fc172ffd640 1 -- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1741f5770 msgr2=0x7fc1740a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fc172ffd640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1741f5770 0x7fc1740a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.820+0000 7fc172ffd640 1 -- 172.21.15.59:0/3931939990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17415db00 con 0x7fc160005270 2024-06-29T14:22:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7fc172ffd640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 0x7fc17415e920 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc16c010a90 tx=0x7fc16c010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7fb7967fc640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7880763a0 0x7fb7880788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7f0ac0a2a640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0aa00763d0 0x7f0aa00788e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8004780 tx=0x7f0ab8001890 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7fb7967fc640 1 -- 172.21.15.59:0/786734648 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb7a00b8180 con 0x7fb7a40b4a30 2024-06-29T14:22:17.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a400a860 con 0x7f55a40374f0 2024-06-29T14:22:17.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7f7d8a481640 1 -- 172.21.15.59:0/190543722 shutdown_connections 2024-06-29T14:22:17.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.821+0000 7f327cff9640 1 -- 172.21.15.59:0/2427284297 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f326c016030 con 0x7f32780db200 2024-06-29T14:22:17.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.822+0000 7f7d8a481640 1 -- 172.21.15.59:0/190543722 wait complete. 2024-06-29T14:22:17.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.822+0000 7fb7a98ad640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7880763a0 0x7fb7880788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.822+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7a80b3c10 con 0x7fe7a81e4f60 2024-06-29T14:22:17.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.822+0000 7f327cff9640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32640763d0 0x7f32640788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.822+0000 7fb7a98ad640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7880763a0 0x7fb7880788b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb798001680 tx=0x7fb798009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.822+0000 7f327cff9640 1 -- 172.21.15.59:0/2427284297 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f326c0980c0 con 0x7f32780db200 2024-06-29T14:22:17.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.822+0000 7f327ffff640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32640763d0 0x7f32640788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.823+0000 7f327ffff640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32640763d0 0x7f32640788e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3274000c00 tx=0x7f3274001e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.824+0000 7f7d8a481640 1 Processor -- start 2024-06-29T14:22:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.824+0000 7f7d8a481640 1 -- start start 2024-06-29T14:22:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.825+0000 7f7d8a481640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d840cf100 0x7f7d841f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d8a481640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d840cf510 0x7f7d8413b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.825+0000 7fc16b7fe640 1 -- 172.21.15.59:0/3931939990 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c01f070 con 0x7fc160005270 2024-06-29T14:22:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17415dde0 con 0x7fc160005270 2024-06-29T14:22:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d8a481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d84140080 0x7f7d84140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7fb7967fc640 1 -- 172.21.15.59:0/786734648 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7a40b0380 con 0x7fb7a40b4a30 2024-06-29T14:22:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f0ab1ffb640 1 -- 172.21.15.59:0/2984869405 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0abc13b550 con 0x7f0aa8004800 2024-06-29T14:22:17.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1741f4f70 con 0x7fc160005270 2024-06-29T14:22:17.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d837fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d840cf510 0x7f7d8413b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d837fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d840cf510 0x7f7d8413b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:46230/0 (socket says 172.21.15.59:46230) 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d88a1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d84140080 0x7f7d84140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d88a1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d84140080 0x7f7d84140490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52010/0 (socket says 172.21.15.59:52010) 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d8a481640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8406ece0 con 0x7f7d84140080 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d8a481640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8406eb60 con 0x7f7d840cf510 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d8a481640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8406ee60 con 0x7f7d840cf100 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d837fe640 1 -- 172.21.15.59:0/1403601635 learned_addr learned my addr 172.21.15.59:0/1403601635 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.826+0000 7f7d83fff640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d840cf100 0x7f7d841f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.827+0000 7f7d83fff640 1 -- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d840cf510 msgr2=0x7f7d8413b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.827+0000 7f7d83fff640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d840cf510 0x7f7d8413b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.827+0000 7f7d83fff640 1 -- 172.21.15.59:0/1403601635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d84140080 msgr2=0x7f7d84140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.827+0000 7f7d83fff640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d84140080 0x7f7d84140490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.827+0000 7f7d83fff640 1 -- 172.21.15.59:0/1403601635 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d841409d0 con 0x7f7d840cf100 2024-06-29T14:22:17.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.827+0000 7f7d88a1e640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d84140080 0x7f7d84140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:17.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.827+0000 7f7d83fff640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d840cf100 0x7f7d841f0fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d74031000 tx=0x7f7d74033830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.827+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc17415b310 con 0x7fc160005270 2024-06-29T14:22:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.828+0000 7fe79dffb640 1 -- 172.21.15.59:0/327159006 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe7a000a510 con 0x7fe7a81e4f60 2024-06-29T14:22:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.829+0000 7fe79dffb640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe78c0762d0 0x7fe78c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.829+0000 7fe7ad6b4640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe78c0762d0 0x7fe78c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.830+0000 7fe7ad6b4640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe78c0762d0 0x7fe78c0787e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4005fd0 tx=0x7fe7a4009260 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.830+0000 7fe79dffb640 1 -- 172.21.15.59:0/327159006 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe7a004fe30 con 0x7fe7a81e4f60 2024-06-29T14:22:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.830+0000 7f7d80ff9640 1 -- 172.21.15.59:0/1403601635 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74040020 con 0x7f7d840cf100 2024-06-29T14:22:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.831+0000 7f7d8a481640 1 -- 172.21.15.59:0/1403601635 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d840acdb0 con 0x7f7d840cf100 2024-06-29T14:22:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.831+0000 7f7d8a481640 1 -- 172.21.15.59:0/1403601635 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d840acad0 con 0x7f7d840cf100 2024-06-29T14:22:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.831+0000 7f327cff9640 1 -- 172.21.15.59:0/2427284297 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f32780e0190 con 0x7f32780db200 2024-06-29T14:22:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.832+0000 7f55a2ffd640 1 -- 172.21.15.59:0/3347313631 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f55a8068380 con 0x7f55a40374f0 2024-06-29T14:22:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.833+0000 7f7d6a7fc640 1 -- 172.21.15.59:0/1403601635 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d64005290 con 0x7f7d840cf100 2024-06-29T14:22:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.833+0000 7fc16b7fe640 1 -- 172.21.15.59:0/3931939990 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc16c023400 con 0x7fc160005270 2024-06-29T14:22:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.833+0000 7fc16b7fe640 1 -- 172.21.15.59:0/3931939990 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c031060 con 0x7fc160005270 2024-06-29T14:22:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.835+0000 7fc16b7fe640 1 -- 172.21.15.59:0/3931939990 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc16c023870 con 0x7fc160005270 2024-06-29T14:22:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.836+0000 7fe79dffb640 1 -- 172.21.15.59:0/327159006 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe7a005c020 con 0x7fe7a81e4f60 2024-06-29T14:22:17.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.836+0000 7fc16b7fe640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1540762d0 0x7fc1540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.836+0000 7f7d80ff9640 1 -- 172.21.15.59:0/1403601635 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d74033030 con 0x7f7d840cf100 2024-06-29T14:22:17.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.836+0000 7f7d80ff9640 1 -- 172.21.15.59:0/1403601635 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74037040 con 0x7f7d840cf100 2024-06-29T14:22:17.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.836+0000 7fc16b7fe640 1 -- 172.21.15.59:0/3931939990 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fc16c0670a0 con 0x7fc160005270 2024-06-29T14:22:17.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.837+0000 7fc1727fc640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1540762d0 0x7fc1540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.839+0000 7f7d80ff9640 1 -- 172.21.15.59:0/1403601635 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7d74043020 con 0x7f7d840cf100 2024-06-29T14:22:17.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.840+0000 7f7d80ff9640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d70076300 0x7f7d70078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:17.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.840+0000 7f7d80ff9640 1 -- 172.21.15.59:0/1403601635 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f7d740b8400 con 0x7f7d840cf100 2024-06-29T14:22:17.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.841+0000 7f7d80ff9640 1 -- 172.21.15.59:0/1403601635 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7d7404a030 con 0x7f7d840cf100 2024-06-29T14:22:17.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.841+0000 7f7d837fe640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d70076300 0x7f7d70078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:17.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.841+0000 7fc1727fc640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1540762d0 0x7fc1540787e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc164006fd0 tx=0x7fc164006a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:17.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.842+0000 7fc16b7fe640 1 -- 172.21.15.59:0/3931939990 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc16c06b1b0 con 0x7fc160005270 2024-06-29T14:22:17.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:17.855+0000 7f7d837fe640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d70076300 0x7f7d70078810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d84141170 tx=0x7f7d78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:18.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.196+0000 7fe3c7fff640 1 -- 172.21.15.59:0/349229887 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe3a8005580 con 0x7fe3c8141d40 2024-06-29T14:22:18.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.197+0000 7fe3af7fe640 1 -- 172.21.15.59:0/349229887 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe3b80640a0 con 0x7fe3c8141d40 2024-06-29T14:22:18.195 INFO:teuthology.orchestra.run.smithi059.stdout:64424509450 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.200+0000 7fe3ad7fa640 1 -- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3b40763a0 msgr2=0x7fe3b40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.200+0000 7fe3ad7fa640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3b40763a0 0x7fe3b40788b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009610 tx=0x7fe3bc006b80 comp rx=0 tx=0).stop 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.200+0000 7fe3ad7fa640 1 -- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3c8141d40 msgr2=0x7fe3c80ac5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.200+0000 7fe3ad7fa640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3c8141d40 0x7fe3c80ac5c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800c430 tx=0x7fe3b800c8f0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.200+0000 7fe3ad7fa640 1 -- 172.21.15.59:0/349229887 shutdown_connections 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.200+0000 7fe3ad7fa640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3b40763a0 0x7fe3b40788b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.201+0000 7fe3ad7fa640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3c8142690 0x7fe3c81f8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.201+0000 7fe3ad7fa640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3c8141d40 0x7fe3c80ac5c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.201+0000 7fe3ad7fa640 1 --2- 172.21.15.59:0/349229887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3c80b2fb0 0x7fe3c806d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.201+0000 7fe3ad7fa640 1 -- 172.21.15.59:0/349229887 >> 172.21.15.59:0/349229887 conn(0x7fe3c80af930 msgr2=0x7fe3c8072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.201+0000 7fe3ad7fa640 1 -- 172.21.15.59:0/349229887 shutdown_connections 2024-06-29T14:22:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.201+0000 7fe3ad7fa640 1 -- 172.21.15.59:0/349229887 wait complete. 2024-06-29T14:22:18.216 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.3 2024-06-29T14:22:18.217 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:18.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.243+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe7a8152b10 con 0x7fe7a81e4f60 2024-06-29T14:22:18.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.244+0000 7fe79dffb640 1 -- 172.21.15.59:0/327159006 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe7a0093610 con 0x7fe7a81e4f60 2024-06-29T14:22:18.242 INFO:teuthology.orchestra.run.smithi059.stdout:60129542153 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.246+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe78c0762d0 msgr2=0x7fe78c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.246+0000 7fe7af918640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe78c0762d0 0x7fe78c0787e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4005fd0 tx=0x7fe7a4009260 comp rx=0 tx=0).stop 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7a81e4f60 msgr2=0x7fe7a81555d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7a81e4f60 0x7fe7a81555d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0031000 tx=0x7fe7a0008aa0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 shutdown_connections 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe78c0762d0 0x7fe78c0787e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7a81e4f60 0x7fe7a81555d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7a80cf380 0x7fe7a8155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 --2- 172.21.15.59:0/327159006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a80b3800 0x7fe7a814a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 >> 172.21.15.59:0/327159006 conn(0x7fe7a80b24c0 msgr2=0x7fe7a8068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 shutdown_connections 2024-06-29T14:22:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.247+0000 7fe7af918640 1 -- 172.21.15.59:0/327159006 wait complete. 2024-06-29T14:22:18.259 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.5 2024-06-29T14:22:18.259 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:18.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.266+0000 7fb787fff640 1 -- 172.21.15.59:0/786734648 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb780005580 con 0x7fb7a40b4a30 2024-06-29T14:22:18.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.267+0000 7fb7967fc640 1 -- 172.21.15.59:0/786734648 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb7a0081510 con 0x7fb7a40b4a30 2024-06-29T14:22:18.265 INFO:teuthology.orchestra.run.smithi059.stdout:60129542154 2024-06-29T14:22:18.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.268+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0a9c005580 con 0x7f0aa8004800 2024-06-29T14:22:18.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.269+0000 7f0ab1ffb640 1 -- 172.21.15.59:0/2984869405 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0aac0618a0 con 0x7f0aa8004800 2024-06-29T14:22:18.267 INFO:teuthology.orchestra.run.smithi059.stdout:64424509451 2024-06-29T14:22:18.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 -- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7880763a0 msgr2=0x7fb7880788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7880763a0 0x7fb7880788b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb798001680 tx=0x7fb798009040 comp rx=0 tx=0).stop 2024-06-29T14:22:18.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 -- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7a40b4a30 msgr2=0x7fb7a414bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7a40b4a30 0x7fb7a414bc20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0031000 tx=0x7fb7a00039e0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 -- 172.21.15.59:0/786734648 shutdown_connections 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb7880763a0 0x7fb7880788b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7a40afa30 0x7fb7a40afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7a40b4a30 0x7fb7a414bc20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 --2- 172.21.15.59:0/786734648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7a40b4310 0x7fb7a40c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 -- 172.21.15.59:0/786734648 >> 172.21.15.59:0/786734648 conn(0x7fb7a40b1be0 msgr2=0x7fb7a41599d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 -- 172.21.15.59:0/786734648 shutdown_connections 2024-06-29T14:22:18.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.270+0000 7fb787fff640 1 -- 172.21.15.59:0/786734648 wait complete. 2024-06-29T14:22:18.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0aa00763d0 msgr2=0x7f0aa00788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0aa00763d0 0x7f0aa00788e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8004780 tx=0x7f0ab8001890 comp rx=0 tx=0).stop 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0aa8004800 msgr2=0x7f0abc155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0aa8004800 0x7f0abc155bc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0aac004d90 tx=0x7f0aac001630 comp rx=0 tx=0).stop 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 shutdown_connections 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f0aa00763d0 0x7f0aa00788e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0abc13b140 0x7f0abc13cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0abc13e390 0x7f0abc13ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 --2- 172.21.15.59:0/2984869405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0aa8004800 0x7f0abc155bc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 >> 172.21.15.59:0/2984869405 conn(0x7f0abc072d80 msgr2=0x7f0abc0cfa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 shutdown_connections 2024-06-29T14:22:18.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.271+0000 7f0ac348f640 1 -- 172.21.15.59:0/2984869405 wait complete. 2024-06-29T14:22:18.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.274+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f3260005580 con 0x7f32780db200 2024-06-29T14:22:18.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.274+0000 7f327cff9640 1 -- 172.21.15.59:0/2427284297 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f32780dd140 con 0x7f32780db200 2024-06-29T14:22:18.272 INFO:teuthology.orchestra.run.smithi059.stdout:60129542153 2024-06-29T14:22:18.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32640763d0 msgr2=0x7f32640788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32640763d0 0x7f32640788e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3274000c00 tx=0x7f3274001e40 comp rx=0 tx=0).stop 2024-06-29T14:22:18.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32780db200 msgr2=0x7f3278051100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32780db200 0x7f3278051100 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f326c00fb50 tx=0x7f326c00d710 comp rx=0 tx=0).stop 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 shutdown_connections 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32640763d0 0x7f32640788e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3278051640 0x7f3278038b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32780db200 0x7f3278051100 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 --2- 172.21.15.59:0/2427284297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3270004140 0x7f32780dfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 >> 172.21.15.59:0/2427284297 conn(0x7f3278058dd0 msgr2=0x7f32780527c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 shutdown_connections 2024-06-29T14:22:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.277+0000 7f328513f640 1 -- 172.21.15.59:0/2427284297 wait complete. 2024-06-29T14:22:18.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.282+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f55a4032720 con 0x7f55a40374f0 2024-06-29T14:22:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.282+0000 7f55a2ffd640 1 -- 172.21.15.59:0/3347313631 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f55a8070400 con 0x7f55a40374f0 2024-06-29T14:22:18.280 INFO:teuthology.orchestra.run.smithi059.stdout:60129542152 2024-06-29T14:22:18.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.285+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55900766d0 msgr2=0x7f5590078be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.285+0000 7f55b30e1640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55900766d0 0x7f5590078be0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f55ac06b240 tx=0x7f55ac06bb70 comp rx=0 tx=0).stop 2024-06-29T14:22:18.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.285+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a40374f0 msgr2=0x7f55a40d7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.285+0000 7f55b30e1640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a40374f0 0x7f55a40d7490 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f55a8007d40 tx=0x7f55a800a430 comp rx=0 tx=0).stop 2024-06-29T14:22:18.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509451 for osd.0 2024-06-29T14:22:18.285 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:18.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.286+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 shutdown_connections 2024-06-29T14:22:18.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.286+0000 7f55b30e1640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f55900766d0 0x7f5590078be0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.286+0000 7f55b30e1640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a4052b80 0x7f55a404e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.286+0000 7f55b30e1640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a40dfd30 0x7f55a40d6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.286+0000 7f55b30e1640 1 --2- 172.21.15.59:0/3347313631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a40374f0 0x7f55a40d7490 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.286+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 >> 172.21.15.59:0/3347313631 conn(0x7f55a401e3f0 msgr2=0x7f55a4032f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.286+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 shutdown_connections 2024-06-29T14:22:18.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.286+0000 7f55b30e1640 1 -- 172.21.15.59:0/3347313631 wait complete. 2024-06-29T14:22:18.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.287+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc1740cd520 con 0x7fc160005270 2024-06-29T14:22:18.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.287+0000 7fc16b7fe640 1 -- 172.21.15.59:0/3931939990 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc16c06aab0 con 0x7fc160005270 2024-06-29T14:22:18.286 INFO:teuthology.orchestra.run.smithi059.stdout:64424509449 2024-06-29T14:22:18.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.6 2024-06-29T14:22:18.288 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.290+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1540762d0 msgr2=0x7fc1540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.290+0000 7fc1795ef640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1540762d0 0x7fc1540787e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc164006fd0 tx=0x7fc164006a60 comp rx=0 tx=0).stop 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.290+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 msgr2=0x7fc17415e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.290+0000 7fc1795ef640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 0x7fc17415e920 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc16c010a90 tx=0x7fc16c010f50 comp rx=0 tx=0).stop 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 shutdown_connections 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7fc1795ef640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1540762d0 0x7fc1540787e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7fc1795ef640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc17413ef40 0x7fc1740b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7fc1795ef640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1741f5770 0x7fc1740a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7fc1795ef640 1 --2- 172.21.15.59:0/3931939990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc160005270 0x7fc17415e920 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 >> 172.21.15.59:0/3931939990 conn(0x7fc1740ccbe0 msgr2=0x7fc174073a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 shutdown_connections 2024-06-29T14:22:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7fc1795ef640 1 -- 172.21.15.59:0/3931939990 wait complete. 2024-06-29T14:22:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.291+0000 7f7d6a7fc640 1 -- 172.21.15.59:0/1403601635 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7d64005850 con 0x7f7d840cf100 2024-06-29T14:22:18.291 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.7 2024-06-29T14:22:18.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.292+0000 7f7d80ff9640 1 -- 172.21.15.59:0/1403601635 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7d841409d0 con 0x7f7d840cf100 2024-06-29T14:22:18.291 INFO:teuthology.orchestra.run.smithi059.stdout:55834574858 2024-06-29T14:22:18.291 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 -- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d70076300 msgr2=0x7f7d70078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d70076300 0x7f7d70078810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7d84141170 tx=0x7f7d78009040 comp rx=0 tx=0).stop 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 -- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d840cf100 msgr2=0x7f7d841f0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d840cf100 0x7f7d841f0fd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d74031000 tx=0x7f7d74033830 comp rx=0 tx=0).stop 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 -- 172.21.15.59:0/1403601635 shutdown_connections 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d70076300 0x7f7d70078810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d84140080 0x7f7d84140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d840cf510 0x7f7d8413b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.295+0000 7f7d6a7fc640 1 --2- 172.21.15.59:0/1403601635 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d840cf100 0x7f7d841f0fd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.296+0000 7f7d6a7fc640 1 -- 172.21.15.59:0/1403601635 >> 172.21.15.59:0/1403601635 conn(0x7f7d84072d80 msgr2=0x7f7d8413d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.296+0000 7f7d6a7fc640 1 -- 172.21.15.59:0/1403601635 shutdown_connections 2024-06-29T14:22:18.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.296+0000 7f7d6a7fc640 1 -- 172.21.15.59:0/1403601635 wait complete. 2024-06-29T14:22:18.301 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.4 2024-06-29T14:22:18.301 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:18.302 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.1 2024-06-29T14:22:18.302 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:18.307 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.2 2024-06-29T14:22:18.307 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:18.307 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-29T14:22:18.307 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:22:18.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.467+0000 7fb125fd2640 1 -- 172.21.15.59:0/3438755235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200b2270 msgr2=0x7fb12015bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.467+0000 7fb125fd2640 1 --2- 172.21.15.59:0/3438755235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200b2270 0x7fb12015bc20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1200cf310 tx=0x7fb11002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.467+0000 7fb125fd2640 1 -- 172.21.15.59:0/3438755235 shutdown_connections 2024-06-29T14:22:18.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.467+0000 7fb125fd2640 1 --2- 172.21.15.59:0/3438755235 >> v2:172.21.15.153:3300/0 conn(0x7fb1200d1c30 0x7fb1200b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.467+0000 7fb125fd2640 1 --2- 172.21.15.59:0/3438755235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200b2270 0x7fb12015bc20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.467+0000 7fb125fd2640 1 -- 172.21.15.59:0/3438755235 >> 172.21.15.59:0/3438755235 conn(0x7fb120073e10 msgr2=0x7fb120074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.467+0000 7fb125fd2640 1 -- 172.21.15.59:0/3438755235 shutdown_connections 2024-06-29T14:22:18.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.467+0000 7fb125fd2640 1 -- 172.21.15.59:0/3438755235 wait complete. 2024-06-29T14:22:18.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.469+0000 7fb125fd2640 1 Processor -- start 2024-06-29T14:22:18.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.469+0000 7fb125fd2640 1 -- start start 2024-06-29T14:22:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb125fd2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1200b2270 0x7fb1200ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb125fd2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1200b2fa0 0x7fb1200af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb125fd2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200d1c30 0x7fb1200ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb125fd2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb1200b6400 con 0x7fb1200d1c30 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb125fd2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1200b6280 con 0x7fb1200b2270 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb125fd2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb1200b6580 con 0x7fb1200b2fa0 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200d1c30 0x7fb1200ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200d1c30 0x7fb1200ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52020/0 (socket says 172.21.15.59:52020) 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 -- 172.21.15.59:0/2153485761 learned_addr learned my addr 172.21.15.59:0/2153485761 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11f7fe640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1200b2270 0x7fb1200ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11effd640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1200b2fa0 0x7fb1200af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 -- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1200b2fa0 msgr2=0x7fb1200af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1200b2fa0 0x7fb1200af1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 -- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1200b2270 msgr2=0x7fb1200ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1200b2270 0x7fb1200ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 -- 172.21.15.59:0/2153485761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1200afa60 con 0x7fb1200d1c30 2024-06-29T14:22:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb11ffff640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200d1c30 0x7fb1200ac0c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb11800eab0 tx=0x7fb11800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb117fff640 1 -- 172.21.15.59:0/2153485761 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11800cea0 con 0x7fb1200d1c30 2024-06-29T14:22:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb117fff640 1 -- 172.21.15.59:0/2153485761 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb118004b20 con 0x7fb1200d1c30 2024-06-29T14:22:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.471+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1200afd40 con 0x7fb1200d1c30 2024-06-29T14:22:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.472+0000 7fb117fff640 1 -- 172.21.15.59:0/2153485761 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118005020 con 0x7fb1200d1c30 2024-06-29T14:22:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.472+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1200ab640 con 0x7fb1200d1c30 2024-06-29T14:22:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.473+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1200b6aa0 con 0x7fb1200d1c30 2024-06-29T14:22:18.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.473+0000 7fb117fff640 1 -- 172.21.15.59:0/2153485761 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb1180051c0 con 0x7fb1200d1c30 2024-06-29T14:22:18.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.474+0000 7fb117fff640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb100076300 0x7fb100078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:18.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.474+0000 7fb117fff640 1 -- 172.21.15.59:0/2153485761 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb118098660 con 0x7fb1200d1c30 2024-06-29T14:22:18.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.474+0000 7fb11f7fe640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb100076300 0x7fb100078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:18.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.475+0000 7fb11f7fe640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb100076300 0x7fb100078810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb110033000 tx=0x7fb110004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:18.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.478+0000 7fb117fff640 1 -- 172.21.15.59:0/2153485761 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb118061c80 con 0x7fb1200d1c30 2024-06-29T14:22:18.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.666+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb12015be30 con 0x7fb100076300 2024-06-29T14:22:18.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.672+0000 7fb117fff640 1 -- 172.21.15.59:0/2153485761 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7fb12015be30 con 0x7fb100076300 2024-06-29T14:22:18.673 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:18.675 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb100076300 msgr2=0x7fb100078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb100076300 0x7fb100078810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb110033000 tx=0x7fb110004730 comp rx=0 tx=0).stop 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200d1c30 msgr2=0x7fb1200ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200d1c30 0x7fb1200ac0c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb11800eab0 tx=0x7fb11800ef70 comp rx=0 tx=0).stop 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 shutdown_connections 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb100076300 0x7fb100078810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1200d1c30 0x7fb1200ac0c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1200b2fa0 0x7fb1200af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 --2- 172.21.15.59:0/2153485761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1200b2270 0x7fb1200ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 >> 172.21.15.59:0/2153485761 conn(0x7fb120073e10 msgr2=0x7fb120070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 shutdown_connections 2024-06-29T14:22:18.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.676+0000 7fb125fd2640 1 -- 172.21.15.59:0/2153485761 wait complete. 2024-06-29T14:22:18.686 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-06-29T14:22:17.736469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219504,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755216,"statfs":{"total":773094113280,"available":772869341184,"internally_reserved":0,"allocated":3416064,"data_stored":2049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640964+0000","last_change":"2024-06-29T14:21:54.260091+0000","last_active":"2024-06-29T14:22:00.640964+0000","last_peered":"2024-06-29T14:22:00.640964+0000","last_clean":"2024-06-29T14:22:00.640964+0000","last_became_active":"2024-06-29T14:21:54.259733+0000","last_became_peered":"2024-06-29T14:21:54.259733+0000","last_unstale":"2024-06-29T14:22:00.640964+0000","last_undegraded":"2024-06-29T14:22:00.640964+0000","last_fullsized":"2024-06-29T14:22:00.640964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:31.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305128+0000","last_change":"2024-06-29T14:21:55.286512+0000","last_active":"2024-06-29T14:22:00.305128+0000","last_peered":"2024-06-29T14:22:00.305128+0000","last_clean":"2024-06-29T14:22:00.305128+0000","last_became_active":"2024-06-29T14:21:55.286126+0000","last_became_peered":"2024-06-29T14:21:55.286126+0000","last_unstale":"2024-06-29T14:22:00.305128+0000","last_undegraded":"2024-06-29T14:22:00.305128+0000","last_fullsized":"2024-06-29T14:22:00.305128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304598+0000","last_change":"2024-06-29T14:21:59.338143+0000","last_active":"2024-06-29T14:22:00.304598+0000","last_peered":"2024-06-29T14:22:00.304598+0000","last_clean":"2024-06-29T14:22:00.304598+0000","last_became_active":"2024-06-29T14:21:59.337328+0000","last_became_peered":"2024-06-29T14:21:59.337328+0000","last_unstale":"2024-06-29T14:22:00.304598+0000","last_undegraded":"2024-06-29T14:22:00.304598+0000","last_fullsized":"2024-06-29T14:22:00.304598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:23.956178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305340+0000","last_change":"2024-06-29T14:21:54.257884+0000","last_active":"2024-06-29T14:22:00.305340+0000","last_peered":"2024-06-29T14:22:00.305340+0000","last_clean":"2024-06-29T14:22:00.305340+0000","last_became_active":"2024-06-29T14:21:54.257540+0000","last_became_peered":"2024-06-29T14:21:54.257540+0000","last_unstale":"2024-06-29T14:22:00.305340+0000","last_undegraded":"2024-06-29T14:22:00.305340+0000","last_fullsized":"2024-06-29T14:22:00.305340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:54.713087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640589+0000","last_change":"2024-06-29T14:21:55.285781+0000","last_active":"2024-06-29T14:22:00.640589+0000","last_peered":"2024-06-29T14:22:00.640589+0000","last_clean":"2024-06-29T14:22:00.640589+0000","last_became_active":"2024-06-29T14:21:55.285358+0000","last_became_peered":"2024-06-29T14:21:55.285358+0000","last_unstale":"2024-06-29T14:22:00.640589+0000","last_undegraded":"2024-06-29T14:22:00.640589+0000","last_fullsized":"2024-06-29T14:22:00.640589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306447+0000","last_change":"2024-06-29T14:21:59.318024+0000","last_active":"2024-06-29T14:22:00.306447+0000","last_peered":"2024-06-29T14:22:00.306447+0000","last_clean":"2024-06-29T14:22:00.306447+0000","last_became_active":"2024-06-29T14:21:59.317356+0000","last_became_peered":"2024-06-29T14:21:59.317356+0000","last_unstale":"2024-06-29T14:22:00.306447+0000","last_undegraded":"2024-06-29T14:22:00.306447+0000","last_fullsized":"2024-06-29T14:22:00.306447+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:00.209167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640906+0000","last_change":"2024-06-29T14:21:54.258802+0000","last_active":"2024-06-29T14:22:00.640906+0000","last_peered":"2024-06-29T14:22:00.640906+0000","last_clean":"2024-06-29T14:22:00.640906+0000","last_became_active":"2024-06-29T14:21:54.258184+0000","last_became_peered":"2024-06-29T14:21:54.258184+0000","last_unstale":"2024-06-29T14:22:00.640906+0000","last_undegraded":"2024-06-29T14:22:00.640906+0000","last_fullsized":"2024-06-29T14:22:00.640906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:07:56.943579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640388+0000","last_change":"2024-06-29T14:21:55.296893+0000","last_active":"2024-06-29T14:22:00.640388+0000","last_peered":"2024-06-29T14:22:00.640388+0000","last_clean":"2024-06-29T14:22:00.640388+0000","last_became_active":"2024-06-29T14:21:55.296552+0000","last_became_peered":"2024-06-29T14:21:55.296552+0000","last_unstale":"2024-06-29T14:22:00.640388+0000","last_undegraded":"2024-06-29T14:22:00.640388+0000","last_fullsized":"2024-06-29T14:22:00.640388+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:57.029702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307221+0000","last_change":"2024-06-29T14:21:59.319287+0000","last_active":"2024-06-29T14:22:00.307221+0000","last_peered":"2024-06-29T14:22:00.307221+0000","last_clean":"2024-06-29T14:22:00.307221+0000","last_became_active":"2024-06-29T14:21:59.318807+0000","last_became_peered":"2024-06-29T14:21:59.318807+0000","last_unstale":"2024-06-29T14:22:00.307221+0000","last_undegraded":"2024-06-29T14:22:00.307221+0000","last_fullsized":"2024-06-29T14:22:00.307221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:20.883221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640313+0000","last_change":"2024-06-29T14:21:54.264911+0000","last_active":"2024-06-29T14:22:00.640313+0000","last_peered":"2024-06-29T14:22:00.640313+0000","last_clean":"2024-06-29T14:22:00.640313+0000","last_became_active":"2024-06-29T14:21:54.264532+0000","last_became_peered":"2024-06-29T14:21:54.264532+0000","last_unstale":"2024-06-29T14:22:00.640313+0000","last_undegraded":"2024-06-29T14:22:00.640313+0000","last_fullsized":"2024-06-29T14:22:00.640313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:10.799778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640442+0000","last_change":"2024-06-29T14:21:55.295115+0000","last_active":"2024-06-29T14:22:00.640442+0000","last_peered":"2024-06-29T14:22:00.640442+0000","last_clean":"2024-06-29T14:22:00.640442+0000","last_became_active":"2024-06-29T14:21:55.294772+0000","last_became_peered":"2024-06-29T14:21:55.294772+0000","last_unstale":"2024-06-29T14:22:00.640442+0000","last_undegraded":"2024-06-29T14:22:00.640442+0000","last_fullsized":"2024-06-29T14:22:00.640442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:17.149243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307463+0000","last_change":"2024-06-29T14:21:59.312924+0000","last_active":"2024-06-29T14:22:00.307463+0000","last_peered":"2024-06-29T14:22:00.307463+0000","last_clean":"2024-06-29T14:22:00.307463+0000","last_became_active":"2024-06-29T14:21:59.312393+0000","last_became_peered":"2024-06-29T14:21:59.312393+0000","last_unstale":"2024-06-29T14:22:00.307463+0000","last_undegraded":"2024-06-29T14:22:00.307463+0000","last_fullsized":"2024-06-29T14:22:00.307463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:32.630422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305149+0000","last_change":"2024-06-29T14:21:55.294165+0000","last_active":"2024-06-29T14:22:00.305149+0000","last_peered":"2024-06-29T14:22:00.305149+0000","last_clean":"2024-06-29T14:22:00.305149+0000","last_became_active":"2024-06-29T14:21:55.293388+0000","last_became_peered":"2024-06-29T14:21:55.293388+0000","last_unstale":"2024-06-29T14:22:00.305149+0000","last_undegraded":"2024-06-29T14:22:00.305149+0000","last_fullsized":"2024-06-29T14:22:00.305149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:50:36.883008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641127+0000","last_change":"2024-06-29T14:21:54.256784+0000","last_active":"2024-06-29T14:22:00.641127+0000","last_peered":"2024-06-29T14:22:00.641127+0000","last_clean":"2024-06-29T14:22:00.641127+0000","last_became_active":"2024-06-29T14:21:54.256434+0000","last_became_peered":"2024-06-29T14:21:54.256434+0000","last_unstale":"2024-06-29T14:22:00.641127+0000","last_undegraded":"2024-06-29T14:22:00.641127+0000","last_fullsized":"2024-06-29T14:22:00.641127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:06.963975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306723+0000","last_change":"2024-06-29T14:21:59.317621+0000","last_active":"2024-06-29T14:22:00.306723+0000","last_peered":"2024-06-29T14:22:00.306723+0000","last_clean":"2024-06-29T14:22:00.306723+0000","last_became_active":"2024-06-29T14:21:59.316736+0000","last_became_peered":"2024-06-29T14:21:59.316736+0000","last_unstale":"2024-06-29T14:22:00.306723+0000","last_undegraded":"2024-06-29T14:22:00.306723+0000","last_fullsized":"2024-06-29T14:22:00.306723+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:43.083466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304951+0000","last_change":"2024-06-29T14:21:55.296036+0000","last_active":"2024-06-29T14:22:00.304951+0000","last_peered":"2024-06-29T14:22:00.304951+0000","last_clean":"2024-06-29T14:22:00.304951+0000","last_became_active":"2024-06-29T14:21:55.294958+0000","last_became_peered":"2024-06-29T14:21:55.294958+0000","last_unstale":"2024-06-29T14:22:00.304951+0000","last_undegraded":"2024-06-29T14:22:00.304951+0000","last_fullsized":"2024-06-29T14:22:00.304951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:25.523198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641393+0000","last_change":"2024-06-29T14:21:54.249131+0000","last_active":"2024-06-29T14:22:00.641393+0000","last_peered":"2024-06-29T14:22:00.641393+0000","last_clean":"2024-06-29T14:22:00.641393+0000","last_became_active":"2024-06-29T14:21:54.248727+0000","last_became_peered":"2024-06-29T14:21:54.248727+0000","last_unstale":"2024-06-29T14:22:00.641393+0000","last_undegraded":"2024-06-29T14:22:00.641393+0000","last_fullsized":"2024-06-29T14:22:00.641393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:43.007566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306263+0000","last_change":"2024-06-29T14:21:59.318902+0000","last_active":"2024-06-29T14:22:00.306263+0000","last_peered":"2024-06-29T14:22:00.306263+0000","last_clean":"2024-06-29T14:22:00.306263+0000","last_became_active":"2024-06-29T14:21:59.318388+0000","last_became_peered":"2024-06-29T14:21:59.318388+0000","last_unstale":"2024-06-29T14:22:00.306263+0000","last_undegraded":"2024-06-29T14:22:00.306263+0000","last_fullsized":"2024-06-29T14:22:00.306263+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:13.544231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305926+0000","last_change":"2024-06-29T14:21:55.282042+0000","last_active":"2024-06-29T14:22:00.305926+0000","last_peered":"2024-06-29T14:22:00.305926+0000","last_clean":"2024-06-29T14:22:00.305926+0000","last_became_active":"2024-06-29T14:21:55.281259+0000","last_became_peered":"2024-06-29T14:21:55.281259+0000","last_unstale":"2024-06-29T14:22:00.305926+0000","last_undegraded":"2024-06-29T14:22:00.305926+0000","last_fullsized":"2024-06-29T14:22:00.305926+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640041+0000","last_change":"2024-06-29T14:21:54.255093+0000","last_active":"2024-06-29T14:22:00.640041+0000","last_peered":"2024-06-29T14:22:00.640041+0000","last_clean":"2024-06-29T14:22:00.640041+0000","last_became_active":"2024-06-29T14:21:54.254010+0000","last_became_peered":"2024-06-29T14:21:54.254010+0000","last_unstale":"2024-06-29T14:22:00.640041+0000","last_undegraded":"2024-06-29T14:22:00.640041+0000","last_fullsized":"2024-06-29T14:22:00.640041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:24.835712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.642067+0000","last_change":"2024-06-29T14:21:59.337086+0000","last_active":"2024-06-29T14:22:00.642067+0000","last_peered":"2024-06-29T14:22:00.642067+0000","last_clean":"2024-06-29T14:22:00.642067+0000","last_became_active":"2024-06-29T14:21:59.336659+0000","last_became_peered":"2024-06-29T14:21:59.336659+0000","last_unstale":"2024-06-29T14:22:00.642067+0000","last_undegraded":"2024-06-29T14:22:00.642067+0000","last_fullsized":"2024-06-29T14:22:00.642067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:47.549181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306644+0000","last_change":"2024-06-29T14:21:55.289199+0000","last_active":"2024-06-29T14:22:00.306644+0000","last_peered":"2024-06-29T14:22:00.306644+0000","last_clean":"2024-06-29T14:22:00.306644+0000","last_became_active":"2024-06-29T14:21:55.288787+0000","last_became_peered":"2024-06-29T14:21:55.288787+0000","last_unstale":"2024-06-29T14:22:00.306644+0000","last_undegraded":"2024-06-29T14:22:00.306644+0000","last_fullsized":"2024-06-29T14:22:00.306644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:33.390422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640397+0000","last_change":"2024-06-29T14:21:54.260787+0000","last_active":"2024-06-29T14:22:00.640397+0000","last_peered":"2024-06-29T14:22:00.640397+0000","last_clean":"2024-06-29T14:22:00.640397+0000","last_became_active":"2024-06-29T14:21:54.260352+0000","last_became_peered":"2024-06-29T14:21:54.260352+0000","last_unstale":"2024-06-29T14:22:00.640397+0000","last_undegraded":"2024-06-29T14:22:00.640397+0000","last_fullsized":"2024-06-29T14:22:00.640397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:40.455851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305896+0000","last_change":"2024-06-29T14:21:59.316811+0000","last_active":"2024-06-29T14:22:00.305896+0000","last_peered":"2024-06-29T14:22:00.305896+0000","last_clean":"2024-06-29T14:22:00.305896+0000","last_became_active":"2024-06-29T14:21:59.316067+0000","last_became_peered":"2024-06-29T14:21:59.316067+0000","last_unstale":"2024-06-29T14:22:00.305896+0000","last_undegraded":"2024-06-29T14:22:00.305896+0000","last_fullsized":"2024-06-29T14:22:00.305896+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:04.444868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640138+0000","last_change":"2024-06-29T14:21:55.293209+0000","last_active":"2024-06-29T14:22:00.640138+0000","last_peered":"2024-06-29T14:22:00.640138+0000","last_clean":"2024-06-29T14:22:00.640138+0000","last_became_active":"2024-06-29T14:21:55.291574+0000","last_became_peered":"2024-06-29T14:21:55.291574+0000","last_unstale":"2024-06-29T14:22:00.640138+0000","last_undegraded":"2024-06-29T14:22:00.640138+0000","last_fullsized":"2024-06-29T14:22:00.640138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:16.165478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305885+0000","last_change":"2024-06-29T14:21:54.256099+0000","last_active":"2024-06-29T14:22:00.305885+0000","last_peered":"2024-06-29T14:22:00.305885+0000","last_clean":"2024-06-29T14:22:00.305885+0000","last_became_active":"2024-06-29T14:21:54.254022+0000","last_became_peered":"2024-06-29T14:21:54.254022+0000","last_unstale":"2024-06-29T14:22:00.305885+0000","last_undegraded":"2024-06-29T14:22:00.305885+0000","last_fullsized":"2024-06-29T14:22:00.305885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:32.388331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305919+0000","last_change":"2024-06-29T14:21:59.337078+0000","last_active":"2024-06-29T14:22:00.305919+0000","last_peered":"2024-06-29T14:22:00.305919+0000","last_clean":"2024-06-29T14:22:00.305919+0000","last_became_active":"2024-06-29T14:21:59.336616+0000","last_became_peered":"2024-06-29T14:21:59.336616+0000","last_unstale":"2024-06-29T14:22:00.305919+0000","last_undegraded":"2024-06-29T14:22:00.305919+0000","last_fullsized":"2024-06-29T14:22:00.305919+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:33:53.623576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305915+0000","last_change":"2024-06-29T14:21:55.293645+0000","last_active":"2024-06-29T14:22:00.305915+0000","last_peered":"2024-06-29T14:22:00.305915+0000","last_clean":"2024-06-29T14:22:00.305915+0000","last_became_active":"2024-06-29T14:21:55.291467+0000","last_became_peered":"2024-06-29T14:21:55.291467+0000","last_unstale":"2024-06-29T14:22:00.305915+0000","last_undegraded":"2024-06-29T14:22:00.305915+0000","last_fullsized":"2024-06-29T14:22:00.305915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.505918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641935+0000","last_change":"2024-06-29T14:21:54.244171+0000","last_active":"2024-06-29T14:22:00.641935+0000","last_peered":"2024-06-29T14:22:00.641935+0000","last_clean":"2024-06-29T14:22:00.641935+0000","last_became_active":"2024-06-29T14:21:54.243791+0000","last_became_peered":"2024-06-29T14:21:54.243791+0000","last_unstale":"2024-06-29T14:22:00.641935+0000","last_undegraded":"2024-06-29T14:22:00.641935+0000","last_fullsized":"2024-06-29T14:22:00.641935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:24.194470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304468+0000","last_change":"2024-06-29T14:21:59.336963+0000","last_active":"2024-06-29T14:22:00.304468+0000","last_peered":"2024-06-29T14:22:00.304468+0000","last_clean":"2024-06-29T14:22:00.304468+0000","last_became_active":"2024-06-29T14:21:59.336450+0000","last_became_peered":"2024-06-29T14:21:59.336450+0000","last_unstale":"2024-06-29T14:22:00.304468+0000","last_undegraded":"2024-06-29T14:22:00.304468+0000","last_fullsized":"2024-06-29T14:22:00.304468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:00.347317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641725+0000","last_change":"2024-06-29T14:21:55.271686+0000","last_active":"2024-06-29T14:22:00.641725+0000","last_peered":"2024-06-29T14:22:00.641725+0000","last_clean":"2024-06-29T14:22:00.641725+0000","last_became_active":"2024-06-29T14:21:55.271310+0000","last_became_peered":"2024-06-29T14:21:55.271310+0000","last_unstale":"2024-06-29T14:22:00.641725+0000","last_undegraded":"2024-06-29T14:22:00.641725+0000","last_fullsized":"2024-06-29T14:22:00.641725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641232+0000","last_change":"2024-06-29T14:21:54.260566+0000","last_active":"2024-06-29T14:22:00.641232+0000","last_peered":"2024-06-29T14:22:00.641232+0000","last_clean":"2024-06-29T14:22:00.641232+0000","last_became_active":"2024-06-29T14:21:54.260089+0000","last_became_peered":"2024-06-29T14:21:54.260089+0000","last_unstale":"2024-06-29T14:22:00.641232+0000","last_undegraded":"2024-06-29T14:22:00.641232+0000","last_fullsized":"2024-06-29T14:22:00.641232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:28.957956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640300+0000","last_change":"2024-06-29T14:21:59.338874+0000","last_active":"2024-06-29T14:22:00.640300+0000","last_peered":"2024-06-29T14:22:00.640300+0000","last_clean":"2024-06-29T14:22:00.640300+0000","last_became_active":"2024-06-29T14:21:59.338388+0000","last_became_peered":"2024-06-29T14:21:59.338388+0000","last_unstale":"2024-06-29T14:22:00.640300+0000","last_undegraded":"2024-06-29T14:22:00.640300+0000","last_fullsized":"2024-06-29T14:22:00.640300+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:21.205918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641110+0000","last_change":"2024-06-29T14:21:55.284217+0000","last_active":"2024-06-29T14:22:00.641110+0000","last_peered":"2024-06-29T14:22:00.641110+0000","last_clean":"2024-06-29T14:22:00.641110+0000","last_became_active":"2024-06-29T14:21:55.283906+0000","last_became_peered":"2024-06-29T14:21:55.283906+0000","last_unstale":"2024-06-29T14:22:00.641110+0000","last_undegraded":"2024-06-29T14:22:00.641110+0000","last_fullsized":"2024-06-29T14:22:00.641110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:39.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641864+0000","last_change":"2024-06-29T14:21:54.257367+0000","last_active":"2024-06-29T14:22:00.641864+0000","last_peered":"2024-06-29T14:22:00.641864+0000","last_clean":"2024-06-29T14:22:00.641864+0000","last_became_active":"2024-06-29T14:21:54.257040+0000","last_became_peered":"2024-06-29T14:21:54.257040+0000","last_unstale":"2024-06-29T14:22:00.641864+0000","last_undegraded":"2024-06-29T14:22:00.641864+0000","last_fullsized":"2024-06-29T14:22:00.641864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:53.425388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639860+0000","last_change":"2024-06-29T14:21:59.323943+0000","last_active":"2024-06-29T14:22:00.639860+0000","last_peered":"2024-06-29T14:22:00.639860+0000","last_clean":"2024-06-29T14:22:00.639860+0000","last_became_active":"2024-06-29T14:21:59.323400+0000","last_became_peered":"2024-06-29T14:21:59.323400+0000","last_unstale":"2024-06-29T14:22:00.639860+0000","last_undegraded":"2024-06-29T14:22:00.639860+0000","last_fullsized":"2024-06-29T14:22:00.639860+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:01.086377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307356+0000","last_change":"2024-06-29T14:21:55.292906+0000","last_active":"2024-06-29T14:22:00.307356+0000","last_peered":"2024-06-29T14:22:00.307356+0000","last_clean":"2024-06-29T14:22:00.307356+0000","last_became_active":"2024-06-29T14:21:55.292578+0000","last_became_peered":"2024-06-29T14:21:55.292578+0000","last_unstale":"2024-06-29T14:22:00.307356+0000","last_undegraded":"2024-06-29T14:22:00.307356+0000","last_fullsized":"2024-06-29T14:22:00.307356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:11.801869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640191+0000","last_change":"2024-06-29T14:21:54.264501+0000","last_active":"2024-06-29T14:22:00.640191+0000","last_peered":"2024-06-29T14:22:00.640191+0000","last_clean":"2024-06-29T14:22:00.640191+0000","last_became_active":"2024-06-29T14:21:54.264182+0000","last_became_peered":"2024-06-29T14:21:54.264182+0000","last_unstale":"2024-06-29T14:22:00.640191+0000","last_undegraded":"2024-06-29T14:22:00.640191+0000","last_fullsized":"2024-06-29T14:22:00.640191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:53.425388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641814+0000","last_change":"2024-06-29T14:21:59.324707+0000","last_active":"2024-06-29T14:22:00.641814+0000","last_peered":"2024-06-29T14:22:00.641814+0000","last_clean":"2024-06-29T14:22:00.641814+0000","last_became_active":"2024-06-29T14:21:59.324141+0000","last_became_peered":"2024-06-29T14:21:59.324141+0000","last_unstale":"2024-06-29T14:22:00.641814+0000","last_undegraded":"2024-06-29T14:22:00.641814+0000","last_fullsized":"2024-06-29T14:22:00.641814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:43.889397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639772+0000","last_change":"2024-06-29T14:21:55.274636+0000","last_active":"2024-06-29T14:22:00.639772+0000","last_peered":"2024-06-29T14:22:00.639772+0000","last_clean":"2024-06-29T14:22:00.639772+0000","last_became_active":"2024-06-29T14:21:55.271330+0000","last_became_peered":"2024-06-29T14:21:55.271330+0000","last_unstale":"2024-06-29T14:22:00.639772+0000","last_undegraded":"2024-06-29T14:22:00.639772+0000","last_fullsized":"2024-06-29T14:22:00.639772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640544+0000","last_change":"2024-06-29T14:21:54.247945+0000","last_active":"2024-06-29T14:22:00.640544+0000","last_peered":"2024-06-29T14:22:00.640544+0000","last_clean":"2024-06-29T14:22:00.640544+0000","last_became_active":"2024-06-29T14:21:54.247592+0000","last_became_peered":"2024-06-29T14:21:54.247592+0000","last_unstale":"2024-06-29T14:22:00.640544+0000","last_undegraded":"2024-06-29T14:22:00.640544+0000","last_fullsized":"2024-06-29T14:22:00.640544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:03.449985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.642194+0000","last_change":"2024-06-29T14:21:59.321043+0000","last_active":"2024-06-29T14:22:00.642194+0000","last_peered":"2024-06-29T14:22:00.642194+0000","last_clean":"2024-06-29T14:22:00.642194+0000","last_became_active":"2024-06-29T14:21:59.320359+0000","last_became_peered":"2024-06-29T14:21:59.320359+0000","last_unstale":"2024-06-29T14:22:00.642194+0000","last_undegraded":"2024-06-29T14:22:00.642194+0000","last_fullsized":"2024-06-29T14:22:00.642194+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:22.691854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639595+0000","last_change":"2024-06-29T14:21:55.296697+0000","last_active":"2024-06-29T14:22:00.639595+0000","last_peered":"2024-06-29T14:22:00.639595+0000","last_clean":"2024-06-29T14:22:00.639595+0000","last_became_active":"2024-06-29T14:21:55.296392+0000","last_became_peered":"2024-06-29T14:21:55.296392+0000","last_unstale":"2024-06-29T14:22:00.639595+0000","last_undegraded":"2024-06-29T14:22:00.639595+0000","last_fullsized":"2024-06-29T14:22:00.639595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:17.069875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639911+0000","last_change":"2024-06-29T14:21:54.257791+0000","last_active":"2024-06-29T14:22:00.639911+0000","last_peered":"2024-06-29T14:22:00.639911+0000","last_clean":"2024-06-29T14:22:00.639911+0000","last_became_active":"2024-06-29T14:21:54.257395+0000","last_became_peered":"2024-06-29T14:21:54.257395+0000","last_unstale":"2024-06-29T14:22:00.639911+0000","last_undegraded":"2024-06-29T14:22:00.639911+0000","last_fullsized":"2024-06-29T14:22:00.639911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:33.159720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640058+0000","last_change":"2024-06-29T14:21:59.337618+0000","last_active":"2024-06-29T14:22:00.640058+0000","last_peered":"2024-06-29T14:22:00.640058+0000","last_clean":"2024-06-29T14:22:00.640058+0000","last_became_active":"2024-06-29T14:21:59.336503+0000","last_became_peered":"2024-06-29T14:21:59.336503+0000","last_unstale":"2024-06-29T14:22:00.640058+0000","last_undegraded":"2024-06-29T14:22:00.640058+0000","last_fullsized":"2024-06-29T14:22:00.640058+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:01.086377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304940+0000","last_change":"2024-06-29T14:21:55.284345+0000","last_active":"2024-06-29T14:22:00.304940+0000","last_peered":"2024-06-29T14:22:00.304940+0000","last_clean":"2024-06-29T14:22:00.304940+0000","last_became_active":"2024-06-29T14:21:55.283603+0000","last_became_peered":"2024-06-29T14:21:55.283603+0000","last_unstale":"2024-06-29T14:22:00.304940+0000","last_undegraded":"2024-06-29T14:22:00.304940+0000","last_fullsized":"2024-06-29T14:22:00.304940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639902+0000","last_change":"2024-06-29T14:21:54.259298+0000","last_active":"2024-06-29T14:22:00.639902+0000","last_peered":"2024-06-29T14:22:00.639902+0000","last_clean":"2024-06-29T14:22:00.639902+0000","last_became_active":"2024-06-29T14:21:54.258971+0000","last_became_peered":"2024-06-29T14:21:54.258971+0000","last_unstale":"2024-06-29T14:22:00.639902+0000","last_undegraded":"2024-06-29T14:22:00.639902+0000","last_fullsized":"2024-06-29T14:22:00.639902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:34.839629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":117,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640060+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:22:00.640060+0000","last_peered":"2024-06-29T14:22:00.640060+0000","last_clean":"2024-06-29T14:22:00.640060+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:22:00.640060+0000","last_undegraded":"2024-06-29T14:22:00.640060+0000","last_fullsized":"2024-06-29T14:22:00.640060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305693+0000","last_change":"2024-06-29T14:21:59.337431+0000","last_active":"2024-06-29T14:22:00.305693+0000","last_peered":"2024-06-29T14:22:00.305693+0000","last_clean":"2024-06-29T14:22:00.305693+0000","last_became_active":"2024-06-29T14:21:59.337008+0000","last_became_peered":"2024-06-29T14:21:59.337008+0000","last_unstale":"2024-06-29T14:22:00.305693+0000","last_undegraded":"2024-06-29T14:22:00.305693+0000","last_fullsized":"2024-06-29T14:22:00.305693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:56.220106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640737+0000","last_change":"2024-06-29T14:21:55.287255+0000","last_active":"2024-06-29T14:22:00.640737+0000","last_peered":"2024-06-29T14:22:00.640737+0000","last_clean":"2024-06-29T14:22:00.640737+0000","last_became_active":"2024-06-29T14:21:55.286563+0000","last_became_peered":"2024-06-29T14:21:55.286563+0000","last_unstale":"2024-06-29T14:22:00.640737+0000","last_undegraded":"2024-06-29T14:22:00.640737+0000","last_fullsized":"2024-06-29T14:22:00.640737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:40.442251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640792+0000","last_change":"2024-06-29T14:21:54.264774+0000","last_active":"2024-06-29T14:22:00.640792+0000","last_peered":"2024-06-29T14:22:00.640792+0000","last_clean":"2024-06-29T14:22:00.640792+0000","last_became_active":"2024-06-29T14:21:54.264395+0000","last_became_peered":"2024-06-29T14:21:54.264395+0000","last_unstale":"2024-06-29T14:22:00.640792+0000","last_undegraded":"2024-06-29T14:22:00.640792+0000","last_fullsized":"2024-06-29T14:22:00.640792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:49.746280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305477+0000","last_change":"2024-06-29T14:21:59.336774+0000","last_active":"2024-06-29T14:22:00.305477+0000","last_peered":"2024-06-29T14:22:00.305477+0000","last_clean":"2024-06-29T14:22:00.305477+0000","last_became_active":"2024-06-29T14:21:59.336326+0000","last_became_peered":"2024-06-29T14:21:59.336326+0000","last_unstale":"2024-06-29T14:22:00.305477+0000","last_undegraded":"2024-06-29T14:22:00.305477+0000","last_fullsized":"2024-06-29T14:22:00.305477+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:20.883221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640544+0000","last_change":"2024-06-29T14:21:55.289532+0000","last_active":"2024-06-29T14:22:00.640544+0000","last_peered":"2024-06-29T14:22:00.640544+0000","last_clean":"2024-06-29T14:22:00.640544+0000","last_became_active":"2024-06-29T14:21:55.289210+0000","last_became_peered":"2024-06-29T14:21:55.289210+0000","last_unstale":"2024-06-29T14:22:00.640544+0000","last_undegraded":"2024-06-29T14:22:00.640544+0000","last_fullsized":"2024-06-29T14:22:00.640544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:12.073214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641944+0000","last_change":"2024-06-29T14:21:54.246332+0000","last_active":"2024-06-29T14:22:00.641944+0000","last_peered":"2024-06-29T14:22:00.641944+0000","last_clean":"2024-06-29T14:22:00.641944+0000","last_became_active":"2024-06-29T14:21:54.245974+0000","last_became_peered":"2024-06-29T14:21:54.245974+0000","last_unstale":"2024-06-29T14:22:00.641944+0000","last_undegraded":"2024-06-29T14:22:00.641944+0000","last_fullsized":"2024-06-29T14:22:00.641944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:50:54.491455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640419+0000","last_change":"2024-06-29T14:21:59.318499+0000","last_active":"2024-06-29T14:22:00.640419+0000","last_peered":"2024-06-29T14:22:00.640419+0000","last_clean":"2024-06-29T14:22:00.640419+0000","last_became_active":"2024-06-29T14:21:59.317918+0000","last_became_peered":"2024-06-29T14:21:59.317918+0000","last_unstale":"2024-06-29T14:22:00.640419+0000","last_undegraded":"2024-06-29T14:22:00.640419+0000","last_fullsized":"2024-06-29T14:22:00.640419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:43.889397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641394+0000","last_change":"2024-06-29T14:21:55.294388+0000","last_active":"2024-06-29T14:22:00.641394+0000","last_peered":"2024-06-29T14:22:00.641394+0000","last_clean":"2024-06-29T14:22:00.641394+0000","last_became_active":"2024-06-29T14:21:55.294090+0000","last_became_peered":"2024-06-29T14:21:55.294090+0000","last_unstale":"2024-06-29T14:22:00.641394+0000","last_undegraded":"2024-06-29T14:22:00.641394+0000","last_fullsized":"2024-06-29T14:22:00.641394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:39.325162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641102+0000","last_change":"2024-06-29T14:21:54.252699+0000","last_active":"2024-06-29T14:22:00.641102+0000","last_peered":"2024-06-29T14:22:00.641102+0000","last_clean":"2024-06-29T14:22:00.641102+0000","last_became_active":"2024-06-29T14:21:54.252365+0000","last_became_peered":"2024-06-29T14:21:54.252365+0000","last_unstale":"2024-06-29T14:22:00.641102+0000","last_undegraded":"2024-06-29T14:22:00.641102+0000","last_fullsized":"2024-06-29T14:22:00.641102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:32.388331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640359+0000","last_change":"2024-06-29T14:21:59.337020+0000","last_active":"2024-06-29T14:22:00.640359+0000","last_peered":"2024-06-29T14:22:00.640359+0000","last_clean":"2024-06-29T14:22:00.640359+0000","last_became_active":"2024-06-29T14:21:59.336457+0000","last_became_peered":"2024-06-29T14:21:59.336457+0000","last_unstale":"2024-06-29T14:22:00.640359+0000","last_undegraded":"2024-06-29T14:22:00.640359+0000","last_fullsized":"2024-06-29T14:22:00.640359+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:25.501876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640773+0000","last_change":"2024-06-29T14:21:55.297155+0000","last_active":"2024-06-29T14:22:00.640773+0000","last_peered":"2024-06-29T14:22:00.640773+0000","last_clean":"2024-06-29T14:22:00.640773+0000","last_became_active":"2024-06-29T14:21:55.296790+0000","last_became_peered":"2024-06-29T14:21:55.296790+0000","last_unstale":"2024-06-29T14:22:00.640773+0000","last_undegraded":"2024-06-29T14:22:00.640773+0000","last_fullsized":"2024-06-29T14:22:00.640773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:52.163431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306370+0000","last_change":"2024-06-29T14:21:54.261995+0000","last_active":"2024-06-29T14:22:00.306370+0000","last_peered":"2024-06-29T14:22:00.306370+0000","last_clean":"2024-06-29T14:22:00.306370+0000","last_became_active":"2024-06-29T14:21:54.261607+0000","last_became_peered":"2024-06-29T14:21:54.261607+0000","last_unstale":"2024-06-29T14:22:00.306370+0000","last_undegraded":"2024-06-29T14:22:00.306370+0000","last_fullsized":"2024-06-29T14:22:00.306370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:59.466378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305836+0000","last_change":"2024-06-29T14:21:59.316601+0000","last_active":"2024-06-29T14:22:00.305836+0000","last_peered":"2024-06-29T14:22:00.305836+0000","last_clean":"2024-06-29T14:22:00.305836+0000","last_became_active":"2024-06-29T14:21:59.315986+0000","last_became_peered":"2024-06-29T14:21:59.315986+0000","last_unstale":"2024-06-29T14:22:00.305836+0000","last_undegraded":"2024-06-29T14:22:00.305836+0000","last_fullsized":"2024-06-29T14:22:00.305836+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:45.085147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641018+0000","last_change":"2024-06-29T14:21:55.291524+0000","last_active":"2024-06-29T14:22:00.641018+0000","last_peered":"2024-06-29T14:22:00.641018+0000","last_clean":"2024-06-29T14:22:00.641018+0000","last_became_active":"2024-06-29T14:21:55.291206+0000","last_became_peered":"2024-06-29T14:21:55.291206+0000","last_unstale":"2024-06-29T14:22:00.641018+0000","last_undegraded":"2024-06-29T14:22:00.641018+0000","last_fullsized":"2024-06-29T14:22:00.641018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:16.826546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640410+0000","last_change":"2024-06-29T14:21:54.245875+0000","last_active":"2024-06-29T14:22:00.640410+0000","last_peered":"2024-06-29T14:22:00.640410+0000","last_clean":"2024-06-29T14:22:00.640410+0000","last_became_active":"2024-06-29T14:21:54.245444+0000","last_became_peered":"2024-06-29T14:21:54.245444+0000","last_unstale":"2024-06-29T14:22:00.640410+0000","last_undegraded":"2024-06-29T14:22:00.640410+0000","last_fullsized":"2024-06-29T14:22:00.640410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:49.347688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639988+0000","last_change":"2024-06-29T14:21:59.317611+0000","last_active":"2024-06-29T14:22:00.639988+0000","last_peered":"2024-06-29T14:22:00.639988+0000","last_clean":"2024-06-29T14:22:00.639988+0000","last_became_active":"2024-06-29T14:21:59.317081+0000","last_became_peered":"2024-06-29T14:21:59.317081+0000","last_unstale":"2024-06-29T14:22:00.639988+0000","last_undegraded":"2024-06-29T14:22:00.639988+0000","last_fullsized":"2024-06-29T14:22:00.639988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:16.129888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305414+0000","last_change":"2024-06-29T14:21:55.291070+0000","last_active":"2024-06-29T14:22:00.305414+0000","last_peered":"2024-06-29T14:22:00.305414+0000","last_clean":"2024-06-29T14:22:00.305414+0000","last_became_active":"2024-06-29T14:21:55.290468+0000","last_became_peered":"2024-06-29T14:21:55.290468+0000","last_unstale":"2024-06-29T14:22:00.305414+0000","last_undegraded":"2024-06-29T14:22:00.305414+0000","last_fullsized":"2024-06-29T14:22:00.305414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:15.443094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305532+0000","last_change":"2024-06-29T14:21:54.257111+0000","last_active":"2024-06-29T14:22:00.305532+0000","last_peered":"2024-06-29T14:22:00.305532+0000","last_clean":"2024-06-29T14:22:00.305532+0000","last_became_active":"2024-06-29T14:21:54.255536+0000","last_became_peered":"2024-06-29T14:21:54.255536+0000","last_unstale":"2024-06-29T14:22:00.305532+0000","last_undegraded":"2024-06-29T14:22:00.305532+0000","last_fullsized":"2024-06-29T14:22:00.305532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:28:41.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305863+0000","last_change":"2024-06-29T14:21:59.338565+0000","last_active":"2024-06-29T14:22:00.305863+0000","last_peered":"2024-06-29T14:22:00.305863+0000","last_clean":"2024-06-29T14:22:00.305863+0000","last_became_active":"2024-06-29T14:21:59.337920+0000","last_became_peered":"2024-06-29T14:21:59.337920+0000","last_unstale":"2024-06-29T14:22:00.305863+0000","last_undegraded":"2024-06-29T14:22:00.305863+0000","last_fullsized":"2024-06-29T14:22:00.305863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:46.890912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640755+0000","last_change":"2024-06-29T14:21:55.294165+0000","last_active":"2024-06-29T14:22:00.640755+0000","last_peered":"2024-06-29T14:22:00.640755+0000","last_clean":"2024-06-29T14:22:00.640755+0000","last_became_active":"2024-06-29T14:21:55.293742+0000","last_became_peered":"2024-06-29T14:21:55.293742+0000","last_unstale":"2024-06-29T14:22:00.640755+0000","last_undegraded":"2024-06-29T14:22:00.640755+0000","last_fullsized":"2024-06-29T14:22:00.640755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:35.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640160+0000","last_change":"2024-06-29T14:21:54.261554+0000","last_active":"2024-06-29T14:22:00.640160+0000","last_peered":"2024-06-29T14:22:00.640160+0000","last_clean":"2024-06-29T14:22:00.640160+0000","last_became_active":"2024-06-29T14:21:54.261215+0000","last_became_peered":"2024-06-29T14:21:54.261215+0000","last_unstale":"2024-06-29T14:22:00.640160+0000","last_undegraded":"2024-06-29T14:22:00.640160+0000","last_fullsized":"2024-06-29T14:22:00.640160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:43.007566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305468+0000","last_change":"2024-06-29T14:21:59.338884+0000","last_active":"2024-06-29T14:22:00.305468+0000","last_peered":"2024-06-29T14:22:00.305468+0000","last_clean":"2024-06-29T14:22:00.305468+0000","last_became_active":"2024-06-29T14:21:59.336402+0000","last_became_peered":"2024-06-29T14:21:59.336402+0000","last_unstale":"2024-06-29T14:22:00.305468+0000","last_undegraded":"2024-06-29T14:22:00.305468+0000","last_fullsized":"2024-06-29T14:22:00.305468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:00.545575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640161+0000","last_change":"2024-06-29T14:21:55.288628+0000","last_active":"2024-06-29T14:22:00.640161+0000","last_peered":"2024-06-29T14:22:00.640161+0000","last_clean":"2024-06-29T14:22:00.640161+0000","last_became_active":"2024-06-29T14:21:55.288079+0000","last_became_peered":"2024-06-29T14:21:55.288079+0000","last_unstale":"2024-06-29T14:22:00.640161+0000","last_undegraded":"2024-06-29T14:22:00.640161+0000","last_fullsized":"2024-06-29T14:22:00.640161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:44.658689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640560+0000","last_change":"2024-06-29T14:21:54.263230+0000","last_active":"2024-06-29T14:22:00.640560+0000","last_peered":"2024-06-29T14:22:00.640560+0000","last_clean":"2024-06-29T14:22:00.640560+0000","last_became_active":"2024-06-29T14:21:54.262952+0000","last_became_peered":"2024-06-29T14:21:54.262952+0000","last_unstale":"2024-06-29T14:22:00.640560+0000","last_undegraded":"2024-06-29T14:22:00.640560+0000","last_fullsized":"2024-06-29T14:22:00.640560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:48.032837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640470+0000","last_change":"2024-06-29T14:21:59.337082+0000","last_active":"2024-06-29T14:22:00.640470+0000","last_peered":"2024-06-29T14:22:00.640470+0000","last_clean":"2024-06-29T14:22:00.640470+0000","last_became_active":"2024-06-29T14:21:59.336506+0000","last_became_peered":"2024-06-29T14:21:59.336506+0000","last_unstale":"2024-06-29T14:22:00.640470+0000","last_undegraded":"2024-06-29T14:22:00.640470+0000","last_fullsized":"2024-06-29T14:22:00.640470+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:00.347317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640609+0000","last_change":"2024-06-29T14:21:55.299489+0000","last_active":"2024-06-29T14:22:00.640609+0000","last_peered":"2024-06-29T14:22:00.640609+0000","last_clean":"2024-06-29T14:22:00.640609+0000","last_became_active":"2024-06-29T14:21:55.299181+0000","last_became_peered":"2024-06-29T14:21:55.299181+0000","last_unstale":"2024-06-29T14:22:00.640609+0000","last_undegraded":"2024-06-29T14:22:00.640609+0000","last_fullsized":"2024-06-29T14:22:00.640609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:31.165204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:05.053670+0000","last_change":"2024-06-29T14:21:54.251485+0000","last_active":"2024-06-29T14:22:05.053670+0000","last_peered":"2024-06-29T14:22:05.053670+0000","last_clean":"2024-06-29T14:22:05.053670+0000","last_became_active":"2024-06-29T14:21:54.250203+0000","last_became_peered":"2024-06-29T14:21:54.250203+0000","last_unstale":"2024-06-29T14:22:05.053670+0000","last_undegraded":"2024-06-29T14:22:05.053670+0000","last_fullsized":"2024-06-29T14:22:05.053670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:03.449985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641539+0000","last_change":"2024-06-29T14:21:59.324423+0000","last_active":"2024-06-29T14:22:00.641539+0000","last_peered":"2024-06-29T14:22:00.641539+0000","last_clean":"2024-06-29T14:22:00.641539+0000","last_became_active":"2024-06-29T14:21:59.323663+0000","last_became_peered":"2024-06-29T14:21:59.323663+0000","last_unstale":"2024-06-29T14:22:00.641539+0000","last_undegraded":"2024-06-29T14:22:00.641539+0000","last_fullsized":"2024-06-29T14:22:00.641539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:21.205918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307509+0000","last_change":"2024-06-29T14:21:55.292602+0000","last_active":"2024-06-29T14:22:00.307509+0000","last_peered":"2024-06-29T14:22:00.307509+0000","last_clean":"2024-06-29T14:22:00.307509+0000","last_became_active":"2024-06-29T14:21:55.292047+0000","last_became_peered":"2024-06-29T14:21:55.292047+0000","last_unstale":"2024-06-29T14:22:00.307509+0000","last_undegraded":"2024-06-29T14:22:00.307509+0000","last_fullsized":"2024-06-29T14:22:00.307509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:15.123791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641934+0000","last_change":"2024-06-29T14:21:54.264727+0000","last_active":"2024-06-29T14:22:00.641934+0000","last_peered":"2024-06-29T14:22:00.641934+0000","last_clean":"2024-06-29T14:22:00.641934+0000","last_became_active":"2024-06-29T14:21:54.264355+0000","last_became_peered":"2024-06-29T14:21:54.264355+0000","last_unstale":"2024-06-29T14:22:00.641934+0000","last_undegraded":"2024-06-29T14:22:00.641934+0000","last_fullsized":"2024-06-29T14:22:00.641934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:06.025267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306568+0000","last_change":"2024-06-29T14:21:59.318452+0000","last_active":"2024-06-29T14:22:00.306568+0000","last_peered":"2024-06-29T14:22:00.306568+0000","last_clean":"2024-06-29T14:22:00.306568+0000","last_became_active":"2024-06-29T14:21:59.317634+0000","last_became_peered":"2024-06-29T14:21:59.317634+0000","last_unstale":"2024-06-29T14:22:00.306568+0000","last_undegraded":"2024-06-29T14:22:00.306568+0000","last_fullsized":"2024-06-29T14:22:00.306568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:46.219297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640791+0000","last_change":"2024-06-29T14:21:55.284276+0000","last_active":"2024-06-29T14:22:00.640791+0000","last_peered":"2024-06-29T14:22:00.640791+0000","last_clean":"2024-06-29T14:22:00.640791+0000","last_became_active":"2024-06-29T14:21:55.283929+0000","last_became_peered":"2024-06-29T14:21:55.283929+0000","last_unstale":"2024-06-29T14:22:00.640791+0000","last_undegraded":"2024-06-29T14:22:00.640791+0000","last_fullsized":"2024-06-29T14:22:00.640791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640975+0000","last_change":"2024-06-29T14:21:54.260674+0000","last_active":"2024-06-29T14:22:00.640975+0000","last_peered":"2024-06-29T14:22:00.640975+0000","last_clean":"2024-06-29T14:22:00.640975+0000","last_became_active":"2024-06-29T14:21:54.258991+0000","last_became_peered":"2024-06-29T14:21:54.258991+0000","last_unstale":"2024-06-29T14:22:00.640975+0000","last_undegraded":"2024-06-29T14:22:00.640975+0000","last_fullsized":"2024-06-29T14:22:00.640975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:50:54.491455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305903+0000","last_change":"2024-06-29T14:21:59.319141+0000","last_active":"2024-06-29T14:22:00.305903+0000","last_peered":"2024-06-29T14:22:00.305903+0000","last_clean":"2024-06-29T14:22:00.305903+0000","last_became_active":"2024-06-29T14:21:59.318645+0000","last_became_peered":"2024-06-29T14:21:59.318645+0000","last_unstale":"2024-06-29T14:22:00.305903+0000","last_undegraded":"2024-06-29T14:22:00.305903+0000","last_fullsized":"2024-06-29T14:22:00.305903+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:50.697736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640894+0000","last_change":"2024-06-29T14:21:55.293028+0000","last_active":"2024-06-29T14:22:00.640894+0000","last_peered":"2024-06-29T14:22:00.640894+0000","last_clean":"2024-06-29T14:22:00.640894+0000","last_became_active":"2024-06-29T14:21:55.292709+0000","last_became_peered":"2024-06-29T14:21:55.292709+0000","last_unstale":"2024-06-29T14:22:00.640894+0000","last_undegraded":"2024-06-29T14:22:00.640894+0000","last_fullsized":"2024-06-29T14:22:00.640894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:30.912272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305837+0000","last_change":"2024-06-29T14:21:54.262667+0000","last_active":"2024-06-29T14:22:00.305837+0000","last_peered":"2024-06-29T14:22:00.305837+0000","last_clean":"2024-06-29T14:22:00.305837+0000","last_became_active":"2024-06-29T14:21:54.262286+0000","last_became_peered":"2024-06-29T14:21:54.262286+0000","last_unstale":"2024-06-29T14:22:00.305837+0000","last_undegraded":"2024-06-29T14:22:00.305837+0000","last_fullsized":"2024-06-29T14:22:00.305837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:49.746280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304956+0000","last_change":"2024-06-29T14:21:59.337794+0000","last_active":"2024-06-29T14:22:00.304956+0000","last_peered":"2024-06-29T14:22:00.304956+0000","last_clean":"2024-06-29T14:22:00.304956+0000","last_became_active":"2024-06-29T14:21:59.336659+0000","last_became_peered":"2024-06-29T14:21:59.336659+0000","last_unstale":"2024-06-29T14:22:00.304956+0000","last_undegraded":"2024-06-29T14:22:00.304956+0000","last_fullsized":"2024-06-29T14:22:00.304956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:15.282310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639863+0000","last_change":"2024-06-29T14:21:59.337985+0000","last_active":"2024-06-29T14:22:00.639863+0000","last_peered":"2024-06-29T14:22:00.639863+0000","last_clean":"2024-06-29T14:22:00.639863+0000","last_became_active":"2024-06-29T14:21:59.337142+0000","last_became_peered":"2024-06-29T14:21:59.337142+0000","last_unstale":"2024-06-29T14:22:00.639863+0000","last_undegraded":"2024-06-29T14:22:00.639863+0000","last_fullsized":"2024-06-29T14:22:00.639863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:26.400192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639809+0000","last_change":"2024-06-29T14:21:55.291354+0000","last_active":"2024-06-29T14:22:00.639809+0000","last_peered":"2024-06-29T14:22:00.639809+0000","last_clean":"2024-06-29T14:22:00.639809+0000","last_became_active":"2024-06-29T14:21:55.290328+0000","last_became_peered":"2024-06-29T14:21:55.290328+0000","last_unstale":"2024-06-29T14:22:00.639809+0000","last_undegraded":"2024-06-29T14:22:00.639809+0000","last_fullsized":"2024-06-29T14:22:00.639809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639839+0000","last_change":"2024-06-29T14:21:54.254791+0000","last_active":"2024-06-29T14:22:00.639839+0000","last_peered":"2024-06-29T14:22:00.639839+0000","last_clean":"2024-06-29T14:22:00.639839+0000","last_became_active":"2024-06-29T14:21:54.254345+0000","last_became_peered":"2024-06-29T14:21:54.254345+0000","last_unstale":"2024-06-29T14:22:00.639839+0000","last_undegraded":"2024-06-29T14:22:00.639839+0000","last_fullsized":"2024-06-29T14:22:00.639839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:06.025267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305684+0000","last_change":"2024-06-29T14:21:59.317911+0000","last_active":"2024-06-29T14:22:00.305684+0000","last_peered":"2024-06-29T14:22:00.305684+0000","last_clean":"2024-06-29T14:22:00.305684+0000","last_became_active":"2024-06-29T14:21:59.317469+0000","last_became_peered":"2024-06-29T14:21:59.317469+0000","last_unstale":"2024-06-29T14:22:00.305684+0000","last_undegraded":"2024-06-29T14:22:00.305684+0000","last_fullsized":"2024-06-29T14:22:00.305684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:32.630422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305774+0000","last_change":"2024-06-29T14:21:55.297533+0000","last_active":"2024-06-29T14:22:00.305774+0000","last_peered":"2024-06-29T14:22:00.305774+0000","last_clean":"2024-06-29T14:22:00.305774+0000","last_became_active":"2024-06-29T14:21:55.296487+0000","last_became_peered":"2024-06-29T14:21:55.296487+0000","last_unstale":"2024-06-29T14:22:00.305774+0000","last_undegraded":"2024-06-29T14:22:00.305774+0000","last_fullsized":"2024-06-29T14:22:00.305774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305104+0000","last_change":"2024-06-29T14:21:54.253349+0000","last_active":"2024-06-29T14:22:00.305104+0000","last_peered":"2024-06-29T14:22:00.305104+0000","last_clean":"2024-06-29T14:22:00.305104+0000","last_became_active":"2024-06-29T14:21:54.252606+0000","last_became_peered":"2024-06-29T14:21:54.252606+0000","last_unstale":"2024-06-29T14:22:00.305104+0000","last_undegraded":"2024-06-29T14:22:00.305104+0000","last_fullsized":"2024-06-29T14:22:00.305104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:20.986619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306038+0000","last_change":"2024-06-29T14:21:54.254902+0000","last_active":"2024-06-29T14:22:00.306038+0000","last_peered":"2024-06-29T14:22:00.306038+0000","last_clean":"2024-06-29T14:22:00.306038+0000","last_became_active":"2024-06-29T14:21:54.254549+0000","last_became_peered":"2024-06-29T14:21:54.254549+0000","last_unstale":"2024-06-29T14:22:00.306038+0000","last_undegraded":"2024-06-29T14:22:00.306038+0000","last_fullsized":"2024-06-29T14:22:00.306038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:49.347688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640490+0000","last_change":"2024-06-29T14:21:55.297816+0000","last_active":"2024-06-29T14:22:00.640490+0000","last_peered":"2024-06-29T14:22:00.640490+0000","last_clean":"2024-06-29T14:22:00.640490+0000","last_became_active":"2024-06-29T14:21:55.297080+0000","last_became_peered":"2024-06-29T14:21:55.297080+0000","last_unstale":"2024-06-29T14:22:00.640490+0000","last_undegraded":"2024-06-29T14:22:00.640490+0000","last_fullsized":"2024-06-29T14:22:00.640490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640695+0000","last_change":"2024-06-29T14:21:59.324236+0000","last_active":"2024-06-29T14:22:00.640695+0000","last_peered":"2024-06-29T14:22:00.640695+0000","last_clean":"2024-06-29T14:22:00.640695+0000","last_became_active":"2024-06-29T14:21:59.323673+0000","last_became_peered":"2024-06-29T14:21:59.323673+0000","last_unstale":"2024-06-29T14:22:00.640695+0000","last_undegraded":"2024-06-29T14:22:00.640695+0000","last_fullsized":"2024-06-29T14:22:00.640695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:45.514616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305150+0000","last_change":"2024-06-29T14:21:54.257451+0000","last_active":"2024-06-29T14:22:00.305150+0000","last_peered":"2024-06-29T14:22:00.305150+0000","last_clean":"2024-06-29T14:22:00.305150+0000","last_became_active":"2024-06-29T14:21:54.257055+0000","last_became_peered":"2024-06-29T14:21:54.257055+0000","last_unstale":"2024-06-29T14:22:00.305150+0000","last_undegraded":"2024-06-29T14:22:00.305150+0000","last_fullsized":"2024-06-29T14:22:00.305150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:10:39.689935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307544+0000","last_change":"2024-06-29T14:21:55.291560+0000","last_active":"2024-06-29T14:22:00.307544+0000","last_peered":"2024-06-29T14:22:00.307544+0000","last_clean":"2024-06-29T14:22:00.307544+0000","last_became_active":"2024-06-29T14:21:55.291263+0000","last_became_peered":"2024-06-29T14:21:55.291263+0000","last_unstale":"2024-06-29T14:22:00.307544+0000","last_undegraded":"2024-06-29T14:22:00.307544+0000","last_fullsized":"2024-06-29T14:22:00.307544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:50.748371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305606+0000","last_change":"2024-06-29T14:21:59.319249+0000","last_active":"2024-06-29T14:22:00.305606+0000","last_peered":"2024-06-29T14:22:00.305606+0000","last_clean":"2024-06-29T14:22:00.305606+0000","last_became_active":"2024-06-29T14:21:59.316597+0000","last_became_peered":"2024-06-29T14:21:59.316597+0000","last_unstale":"2024-06-29T14:22:00.305606+0000","last_undegraded":"2024-06-29T14:22:00.305606+0000","last_fullsized":"2024-06-29T14:22:00.305606+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:54.923537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305108+0000","last_change":"2024-06-29T14:21:55.291254+0000","last_active":"2024-06-29T14:22:00.305108+0000","last_peered":"2024-06-29T14:22:00.305108+0000","last_clean":"2024-06-29T14:22:00.305108+0000","last_became_active":"2024-06-29T14:21:55.290924+0000","last_became_peered":"2024-06-29T14:21:55.290924+0000","last_unstale":"2024-06-29T14:22:00.305108+0000","last_undegraded":"2024-06-29T14:22:00.305108+0000","last_fullsized":"2024-06-29T14:22:00.305108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:11.308698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640426+0000","last_change":"2024-06-29T14:21:55.280475+0000","last_active":"2024-06-29T14:22:00.640426+0000","last_peered":"2024-06-29T14:22:00.640426+0000","last_clean":"2024-06-29T14:22:00.640426+0000","last_became_active":"2024-06-29T14:21:55.277502+0000","last_became_peered":"2024-06-29T14:21:55.277502+0000","last_unstale":"2024-06-29T14:22:00.640426+0000","last_undegraded":"2024-06-29T14:22:00.640426+0000","last_fullsized":"2024-06-29T14:22:00.640426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:11.801869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640065+0000","last_change":"2024-06-29T14:21:55.285270+0000","last_active":"2024-06-29T14:22:00.640065+0000","last_peered":"2024-06-29T14:22:00.640065+0000","last_clean":"2024-06-29T14:22:00.640065+0000","last_became_active":"2024-06-29T14:21:55.284164+0000","last_became_peered":"2024-06-29T14:21:55.284164+0000","last_unstale":"2024-06-29T14:22:00.640065+0000","last_undegraded":"2024-06-29T14:22:00.640065+0000","last_fullsized":"2024-06-29T14:22:00.640065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:25.523198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640375+0000","last_change":"2024-06-29T14:21:55.287457+0000","last_active":"2024-06-29T14:22:00.640375+0000","last_peered":"2024-06-29T14:22:00.640375+0000","last_clean":"2024-06-29T14:22:00.640375+0000","last_became_active":"2024-06-29T14:21:55.287070+0000","last_became_peered":"2024-06-29T14:21:55.287070+0000","last_unstale":"2024-06-29T14:22:00.640375+0000","last_undegraded":"2024-06-29T14:22:00.640375+0000","last_fullsized":"2024-06-29T14:22:00.640375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305575+0000","last_change":"2024-06-29T14:21:55.298926+0000","last_active":"2024-06-29T14:22:00.305575+0000","last_peered":"2024-06-29T14:22:00.305575+0000","last_clean":"2024-06-29T14:22:00.305575+0000","last_became_active":"2024-06-29T14:21:55.298615+0000","last_became_peered":"2024-06-29T14:21:55.298615+0000","last_unstale":"2024-06-29T14:22:00.305575+0000","last_undegraded":"2024-06-29T14:22:00.305575+0000","last_fullsized":"2024-06-29T14:22:00.305575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641774+0000","last_change":"2024-06-29T14:21:55.290890+0000","last_active":"2024-06-29T14:22:00.641774+0000","last_peered":"2024-06-29T14:22:00.641774+0000","last_clean":"2024-06-29T14:22:00.641774+0000","last_became_active":"2024-06-29T14:21:55.290069+0000","last_became_peered":"2024-06-29T14:21:55.290069+0000","last_unstale":"2024-06-29T14:22:00.641774+0000","last_undegraded":"2024-06-29T14:22:00.641774+0000","last_fullsized":"2024-06-29T14:22:00.641774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306202+0000","last_change":"2024-06-29T14:21:55.277969+0000","last_active":"2024-06-29T14:22:00.306202+0000","last_peered":"2024-06-29T14:22:00.306202+0000","last_clean":"2024-06-29T14:22:00.306202+0000","last_became_active":"2024-06-29T14:21:55.277629+0000","last_became_peered":"2024-06-29T14:21:55.277629+0000","last_unstale":"2024-06-29T14:22:00.306202+0000","last_undegraded":"2024-06-29T14:22:00.306202+0000","last_fullsized":"2024-06-29T14:22:00.306202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:00.388193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305827+0000","last_change":"2024-06-29T14:21:55.297394+0000","last_active":"2024-06-29T14:22:00.305827+0000","last_peered":"2024-06-29T14:22:00.305827+0000","last_clean":"2024-06-29T14:22:00.305827+0000","last_became_active":"2024-06-29T14:21:55.297087+0000","last_became_peered":"2024-06-29T14:21:55.297087+0000","last_unstale":"2024-06-29T14:22:00.305827+0000","last_undegraded":"2024-06-29T14:22:00.305827+0000","last_fullsized":"2024-06-29T14:22:00.305827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:43.242914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639702+0000","last_change":"2024-06-29T14:21:55.284207+0000","last_active":"2024-06-29T14:22:00.639702+0000","last_peered":"2024-06-29T14:22:00.639702+0000","last_clean":"2024-06-29T14:22:00.639702+0000","last_became_active":"2024-06-29T14:21:55.283869+0000","last_became_peered":"2024-06-29T14:21:55.283869+0000","last_unstale":"2024-06-29T14:22:00.639702+0000","last_undegraded":"2024-06-29T14:22:00.639702+0000","last_fullsized":"2024-06-29T14:22:00.639702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304555+0000","last_change":"2024-06-29T14:21:55.298136+0000","last_active":"2024-06-29T14:22:00.304555+0000","last_peered":"2024-06-29T14:22:00.304555+0000","last_clean":"2024-06-29T14:22:00.304555+0000","last_became_active":"2024-06-29T14:21:55.297852+0000","last_became_peered":"2024-06-29T14:21:55.297852+0000","last_unstale":"2024-06-29T14:22:00.304555+0000","last_undegraded":"2024-06-29T14:22:00.304555+0000","last_fullsized":"2024-06-29T14:22:00.304555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640027+0000","last_change":"2024-06-29T14:21:55.291932+0000","last_active":"2024-06-29T14:22:00.640027+0000","last_peered":"2024-06-29T14:22:00.640027+0000","last_clean":"2024-06-29T14:22:00.640027+0000","last_became_active":"2024-06-29T14:21:55.291605+0000","last_became_peered":"2024-06-29T14:21:55.291605+0000","last_unstale":"2024-06-29T14:22:00.640027+0000","last_undegraded":"2024-06-29T14:22:00.640027+0000","last_fullsized":"2024-06-29T14:22:00.640027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.505918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640145+0000","last_change":"2024-06-29T14:21:55.296088+0000","last_active":"2024-06-29T14:22:00.640145+0000","last_peered":"2024-06-29T14:22:00.640145+0000","last_clean":"2024-06-29T14:22:00.640145+0000","last_became_active":"2024-06-29T14:21:55.295752+0000","last_became_peered":"2024-06-29T14:21:55.295752+0000","last_unstale":"2024-06-29T14:22:00.640145+0000","last_undegraded":"2024-06-29T14:22:00.640145+0000","last_fullsized":"2024-06-29T14:22:00.640145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306344+0000","last_change":"2024-06-29T14:21:55.293717+0000","last_active":"2024-06-29T14:22:00.306344+0000","last_peered":"2024-06-29T14:22:00.306344+0000","last_clean":"2024-06-29T14:22:00.306344+0000","last_became_active":"2024-06-29T14:21:55.293421+0000","last_became_peered":"2024-06-29T14:21:55.293421+0000","last_unstale":"2024-06-29T14:22:00.306344+0000","last_undegraded":"2024-06-29T14:22:00.306344+0000","last_fullsized":"2024-06-29T14:22:00.306344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:08:58.083131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641151+0000","last_change":"2024-06-29T14:21:55.288925+0000","last_active":"2024-06-29T14:22:00.641151+0000","last_peered":"2024-06-29T14:22:00.641151+0000","last_clean":"2024-06-29T14:22:00.641151+0000","last_became_active":"2024-06-29T14:21:55.288585+0000","last_became_peered":"2024-06-29T14:21:55.288585+0000","last_unstale":"2024-06-29T14:22:00.641151+0000","last_undegraded":"2024-06-29T14:22:00.641151+0000","last_fullsized":"2024-06-29T14:22:00.641151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:35.844291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641027+0000","last_change":"2024-06-29T14:21:55.271843+0000","last_active":"2024-06-29T14:22:00.641027+0000","last_peered":"2024-06-29T14:22:00.641027+0000","last_clean":"2024-06-29T14:22:00.641027+0000","last_became_active":"2024-06-29T14:21:55.271494+0000","last_became_peered":"2024-06-29T14:21:55.271494+0000","last_unstale":"2024-06-29T14:22:00.641027+0000","last_undegraded":"2024-06-29T14:22:00.641027+0000","last_fullsized":"2024-06-29T14:22:00.641027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:29.960047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640632+0000","last_change":"2024-06-29T14:21:55.278149+0000","last_active":"2024-06-29T14:22:00.640632+0000","last_peered":"2024-06-29T14:22:00.640632+0000","last_clean":"2024-06-29T14:22:00.640632+0000","last_became_active":"2024-06-29T14:21:55.277816+0000","last_became_peered":"2024-06-29T14:21:55.277816+0000","last_unstale":"2024-06-29T14:22:00.640632+0000","last_undegraded":"2024-06-29T14:22:00.640632+0000","last_fullsized":"2024-06-29T14:22:00.640632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:14:18.590915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639577+0000","last_change":"2024-06-29T14:21:55.296366+0000","last_active":"2024-06-29T14:22:00.639577+0000","last_peered":"2024-06-29T14:22:00.639577+0000","last_clean":"2024-06-29T14:22:00.639577+0000","last_became_active":"2024-06-29T14:21:55.296040+0000","last_became_peered":"2024-06-29T14:21:55.296040+0000","last_unstale":"2024-06-29T14:22:00.639577+0000","last_undegraded":"2024-06-29T14:22:00.639577+0000","last_fullsized":"2024-06-29T14:22:00.639577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639931+0000","last_change":"2024-06-29T14:21:55.291183+0000","last_active":"2024-06-29T14:22:00.639931+0000","last_peered":"2024-06-29T14:22:00.639931+0000","last_clean":"2024-06-29T14:22:00.639931+0000","last_became_active":"2024-06-29T14:21:55.289884+0000","last_became_peered":"2024-06-29T14:21:55.289884+0000","last_unstale":"2024-06-29T14:22:00.639931+0000","last_undegraded":"2024-06-29T14:22:00.639931+0000","last_fullsized":"2024-06-29T14:22:00.639931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640038+0000","last_change":"2024-06-29T14:21:55.293964+0000","last_active":"2024-06-29T14:22:00.640038+0000","last_peered":"2024-06-29T14:22:00.640038+0000","last_clean":"2024-06-29T14:22:00.640038+0000","last_became_active":"2024-06-29T14:21:55.291885+0000","last_became_peered":"2024-06-29T14:21:55.291885+0000","last_unstale":"2024-06-29T14:22:00.640038+0000","last_undegraded":"2024-06-29T14:22:00.640038+0000","last_fullsized":"2024-06-29T14:22:00.640038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:20.427848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304827+0000","last_change":"2024-06-29T14:21:55.288765+0000","last_active":"2024-06-29T14:22:00.304827+0000","last_peered":"2024-06-29T14:22:00.304827+0000","last_clean":"2024-06-29T14:22:00.304827+0000","last_became_active":"2024-06-29T14:21:55.288452+0000","last_became_peered":"2024-06-29T14:21:55.288452+0000","last_unstale":"2024-06-29T14:22:00.304827+0000","last_undegraded":"2024-06-29T14:22:00.304827+0000","last_fullsized":"2024-06-29T14:22:00.304827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:30.912272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307365+0000","last_change":"2024-06-29T14:21:55.295376+0000","last_active":"2024-06-29T14:22:00.307365+0000","last_peered":"2024-06-29T14:22:00.307365+0000","last_clean":"2024-06-29T14:22:00.307365+0000","last_became_active":"2024-06-29T14:21:55.294928+0000","last_became_peered":"2024-06-29T14:21:55.294928+0000","last_unstale":"2024-06-29T14:22:00.307365+0000","last_undegraded":"2024-06-29T14:22:00.307365+0000","last_fullsized":"2024-06-29T14:22:00.307365+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:28:42.614313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305773+0000","last_change":"2024-06-29T14:21:55.281068+0000","last_active":"2024-06-29T14:22:00.305773+0000","last_peered":"2024-06-29T14:22:00.305773+0000","last_clean":"2024-06-29T14:22:00.305773+0000","last_became_active":"2024-06-29T14:21:55.280780+0000","last_became_peered":"2024-06-29T14:21:55.280780+0000","last_unstale":"2024-06-29T14:22:00.305773+0000","last_undegraded":"2024-06-29T14:22:00.305773+0000","last_fullsized":"2024-06-29T14:22:00.305773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:36:38.990506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306085+0000","last_change":"2024-06-29T14:21:55.280506+0000","last_active":"2024-06-29T14:22:00.306085+0000","last_peered":"2024-06-29T14:22:00.306085+0000","last_clean":"2024-06-29T14:22:00.306085+0000","last_became_active":"2024-06-29T14:21:55.280221+0000","last_became_peered":"2024-06-29T14:21:55.280221+0000","last_unstale":"2024-06-29T14:22:00.306085+0000","last_undegraded":"2024-06-29T14:22:00.306085+0000","last_fullsized":"2024-06-29T14:22:00.306085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:41.028472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641311+0000","last_change":"2024-06-29T14:21:55.287649+0000","last_active":"2024-06-29T14:22:00.641311+0000","last_peered":"2024-06-29T14:22:00.641311+0000","last_clean":"2024-06-29T14:22:00.641311+0000","last_became_active":"2024-06-29T14:21:55.286171+0000","last_became_peered":"2024-06-29T14:21:55.286171+0000","last_unstale":"2024-06-29T14:22:00.641311+0000","last_undegraded":"2024-06-29T14:22:00.641311+0000","last_fullsized":"2024-06-29T14:22:00.641311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:30.219397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304571+0000","last_change":"2024-06-29T14:21:55.285603+0000","last_active":"2024-06-29T14:22:00.304571+0000","last_peered":"2024-06-29T14:22:00.304571+0000","last_clean":"2024-06-29T14:22:00.304571+0000","last_became_active":"2024-06-29T14:21:55.285284+0000","last_became_peered":"2024-06-29T14:21:55.285284+0000","last_unstale":"2024-06-29T14:22:00.304571+0000","last_undegraded":"2024-06-29T14:22:00.304571+0000","last_fullsized":"2024-06-29T14:22:00.304571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:44.658689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640973+0000","last_change":"2024-06-29T14:21:55.287794+0000","last_active":"2024-06-29T14:22:00.640973+0000","last_peered":"2024-06-29T14:22:00.640973+0000","last_clean":"2024-06-29T14:22:00.640973+0000","last_became_active":"2024-06-29T14:21:55.287028+0000","last_became_peered":"2024-06-29T14:21:55.287028+0000","last_unstale":"2024-06-29T14:22:00.640973+0000","last_undegraded":"2024-06-29T14:22:00.640973+0000","last_fullsized":"2024-06-29T14:22:00.640973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:39.026791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639887+0000","last_change":"2024-06-29T14:21:55.289581+0000","last_active":"2024-06-29T14:22:00.639887+0000","last_peered":"2024-06-29T14:22:00.639887+0000","last_clean":"2024-06-29T14:22:00.639887+0000","last_became_active":"2024-06-29T14:21:55.289097+0000","last_became_peered":"2024-06-29T14:21:55.289097+0000","last_unstale":"2024-06-29T14:22:00.639887+0000","last_undegraded":"2024-06-29T14:22:00.639887+0000","last_fullsized":"2024-06-29T14:22:00.639887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:43.242914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640505+0000","last_change":"2024-06-29T14:21:55.288463+0000","last_active":"2024-06-29T14:22:00.640505+0000","last_peered":"2024-06-29T14:22:00.640505+0000","last_clean":"2024-06-29T14:22:00.640505+0000","last_became_active":"2024-06-29T14:21:55.288128+0000","last_became_peered":"2024-06-29T14:21:55.288128+0000","last_unstale":"2024-06-29T14:22:00.640505+0000","last_undegraded":"2024-06-29T14:22:00.640505+0000","last_fullsized":"2024-06-29T14:22:00.640505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:28.573747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640255+0000","last_change":"2024-06-29T14:21:55.290100+0000","last_active":"2024-06-29T14:22:00.640255+0000","last_peered":"2024-06-29T14:22:00.640255+0000","last_clean":"2024-06-29T14:22:00.640255+0000","last_became_active":"2024-06-29T14:21:55.289525+0000","last_became_peered":"2024-06-29T14:21:55.289525+0000","last_unstale":"2024-06-29T14:22:00.640255+0000","last_undegraded":"2024-06-29T14:22:00.640255+0000","last_fullsized":"2024-06-29T14:22:00.640255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:22.343517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641269+0000","last_change":"2024-06-29T14:21:55.287921+0000","last_active":"2024-06-29T14:22:00.641269+0000","last_peered":"2024-06-29T14:22:00.641269+0000","last_clean":"2024-06-29T14:22:00.641269+0000","last_became_active":"2024-06-29T14:21:55.287383+0000","last_became_peered":"2024-06-29T14:21:55.287383+0000","last_unstale":"2024-06-29T14:22:00.641269+0000","last_undegraded":"2024-06-29T14:22:00.641269+0000","last_fullsized":"2024-06-29T14:22:00.641269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:05.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641908+0000","last_change":"2024-06-29T14:21:55.285210+0000","last_active":"2024-06-29T14:22:00.641908+0000","last_peered":"2024-06-29T14:22:00.641908+0000","last_clean":"2024-06-29T14:22:00.641908+0000","last_became_active":"2024-06-29T14:21:55.284857+0000","last_became_peered":"2024-06-29T14:21:55.284857+0000","last_unstale":"2024-06-29T14:22:00.641908+0000","last_undegraded":"2024-06-29T14:22:00.641908+0000","last_fullsized":"2024-06-29T14:22:00.641908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641229+0000","last_change":"2024-06-29T14:21:55.287889+0000","last_active":"2024-06-29T14:22:00.641229+0000","last_peered":"2024-06-29T14:22:00.641229+0000","last_clean":"2024-06-29T14:22:00.641229+0000","last_became_active":"2024-06-29T14:21:55.286286+0000","last_became_peered":"2024-06-29T14:21:55.286286+0000","last_unstale":"2024-06-29T14:22:00.641229+0000","last_undegraded":"2024-06-29T14:22:00.641229+0000","last_fullsized":"2024-06-29T14:22:00.641229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:23.065793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509451,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":111290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509452,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":698454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":868352,"data_stored":698398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":111290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":107574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:22:18.691 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:22:18.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.846+0000 7fe7d1d63640 1 -- 172.21.15.59:0/2090838781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 msgr2=0x7fe7cc0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.846+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/2090838781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 0x7fe7cc0cf080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009900 tx=0x7fe7bc02a610 comp rx=0 tx=0).stop 2024-06-29T14:22:18.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.846+0000 7fe7d1d63640 1 -- 172.21.15.59:0/2090838781 shutdown_connections 2024-06-29T14:22:18.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.846+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/2090838781 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7cc0ce950 0x7fe7cc0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.846+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/2090838781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 0x7fe7cc0cf080 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.846+0000 7fe7d1d63640 1 -- 172.21.15.59:0/2090838781 >> 172.21.15.59:0/2090838781 conn(0x7fe7cc0b5610 msgr2=0x7fe7cc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:18.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.846+0000 7fe7d1d63640 1 -- 172.21.15.59:0/2090838781 shutdown_connections 2024-06-29T14:22:18.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.846+0000 7fe7d1d63640 1 -- 172.21.15.59:0/2090838781 wait complete. 2024-06-29T14:22:18.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.848+0000 7fe7d1d63640 1 Processor -- start 2024-06-29T14:22:18.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.848+0000 7fe7d1d63640 1 -- start start 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7d1d63640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 0x7fe7cc1e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7d1d63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7cc0ce950 0x7fe7cc0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7d1d63640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7cc0cf5c0 0x7fe7cc0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7d1d63640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc0b7f60 con 0x7fe7cc08c190 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7d1d63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc0b7de0 con 0x7fe7cc0ce950 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7d1d63640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe7cc0b80e0 con 0x7fe7cc0cf5c0 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 0x7fe7cc1e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 0x7fe7cc1e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52028/0 (socket says 172.21.15.59:52028) 2024-06-29T14:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7cb7fe640 1 -- 172.21.15.59:0/1581066178 learned_addr learned my addr 172.21.15.59:0/1581066178 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7cbfff640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7cc0cf5c0 0x7fe7cc0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.849+0000 7fe7caffd640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7cc0ce950 0x7fe7cc0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7cb7fe640 1 -- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7cc0cf5c0 msgr2=0x7fe7cc0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7cb7fe640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7cc0cf5c0 0x7fe7cc0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7cb7fe640 1 -- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7cc0ce950 msgr2=0x7fe7cc0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7cb7fe640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7cc0ce950 0x7fe7cc0ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7cb7fe640 1 -- 172.21.15.59:0/1581066178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc1e8430 con 0x7fe7cc08c190 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7cbfff640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7cc0cf5c0 0x7fe7cc0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7caffd640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7cc0ce950 0x7fe7cc0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:18.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7cb7fe640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 0x7fe7cc1e7970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc000c00 tx=0x7fe7bc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:18.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7b3fff640 1 -- 172.21.15.59:0/1581066178 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc03e030 con 0x7fe7cc08c190 2024-06-29T14:22:18.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7b3fff640 1 -- 172.21.15.59:0/1581066178 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7bc003ca0 con 0x7fe7cc08c190 2024-06-29T14:22:18.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7b3fff640 1 -- 172.21.15.59:0/1581066178 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc035040 con 0x7fe7cc08c190 2024-06-29T14:22:18.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc1e8710 con 0x7fe7cc08c190 2024-06-29T14:22:18.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.850+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc1e7eb0 con 0x7fe7cc08c190 2024-06-29T14:22:18.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.851+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7cc1e8950 con 0x7fe7cc08c190 2024-06-29T14:22:18.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.852+0000 7fe7b3fff640 1 -- 172.21.15.59:0/1581066178 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe7bc041030 con 0x7fe7cc08c190 2024-06-29T14:22:18.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.852+0000 7fe7b3fff640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe7b80762d0 0x7fe7b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:18.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.853+0000 7fe7b3fff640 1 -- 172.21.15.59:0/1581066178 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe7bc0b8220 con 0x7fe7cc08c190 2024-06-29T14:22:18.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.853+0000 7fe7caffd640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe7b80762d0 0x7fe7b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:18.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.853+0000 7fe7caffd640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe7b80762d0 0x7fe7b80787e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc1f8920 tx=0x7fe7c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:18.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:18.857+0000 7fe7b3fff640 1 -- 172.21.15.59:0/1581066178 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe7cc1e8950 con 0x7fe7cc08c190 2024-06-29T14:22:19.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.049+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe7cc0cf080 con 0x7fe7b80762d0 2024-06-29T14:22:19.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.055+0000 7fe7b3fff640 1 -- 172.21.15.59:0/1581066178 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7fe7cc0cf080 con 0x7fe7b80762d0 2024-06-29T14:22:19.061 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:19.064 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:22:19.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe7b80762d0 msgr2=0x7fe7b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe7b80762d0 0x7fe7b80787e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc1f8920 tx=0x7fe7c0009040 comp rx=0 tx=0).stop 2024-06-29T14:22:19.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 msgr2=0x7fe7cc1e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 0x7fe7cc1e7970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc000c00 tx=0x7fe7bc0038a0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 shutdown_connections 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe7b80762d0 0x7fe7b80787e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7cc0cf5c0 0x7fe7cc0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7cc0ce950 0x7fe7cc0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 --2- 172.21.15.59:0/1581066178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7cc08c190 0x7fe7cc1e7970 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 >> 172.21.15.59:0/1581066178 conn(0x7fe7cc0b5610 msgr2=0x7fe7cc1552a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 shutdown_connections 2024-06-29T14:22:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.059+0000 7fe7d1d63640 1 -- 172.21.15.59:0/1581066178 wait complete. 2024-06-29T14:22:19.068 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-06-29T14:22:17.736469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219504,"kb_used_data":3336,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755216,"statfs":{"total":773094113280,"available":772869341184,"internally_reserved":0,"allocated":3416064,"data_stored":2049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640964+0000","last_change":"2024-06-29T14:21:54.260091+0000","last_active":"2024-06-29T14:22:00.640964+0000","last_peered":"2024-06-29T14:22:00.640964+0000","last_clean":"2024-06-29T14:22:00.640964+0000","last_became_active":"2024-06-29T14:21:54.259733+0000","last_became_peered":"2024-06-29T14:21:54.259733+0000","last_unstale":"2024-06-29T14:22:00.640964+0000","last_undegraded":"2024-06-29T14:22:00.640964+0000","last_fullsized":"2024-06-29T14:22:00.640964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:31.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305128+0000","last_change":"2024-06-29T14:21:55.286512+0000","last_active":"2024-06-29T14:22:00.305128+0000","last_peered":"2024-06-29T14:22:00.305128+0000","last_clean":"2024-06-29T14:22:00.305128+0000","last_became_active":"2024-06-29T14:21:55.286126+0000","last_became_peered":"2024-06-29T14:21:55.286126+0000","last_unstale":"2024-06-29T14:22:00.305128+0000","last_undegraded":"2024-06-29T14:22:00.305128+0000","last_fullsized":"2024-06-29T14:22:00.305128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304598+0000","last_change":"2024-06-29T14:21:59.338143+0000","last_active":"2024-06-29T14:22:00.304598+0000","last_peered":"2024-06-29T14:22:00.304598+0000","last_clean":"2024-06-29T14:22:00.304598+0000","last_became_active":"2024-06-29T14:21:59.337328+0000","last_became_peered":"2024-06-29T14:21:59.337328+0000","last_unstale":"2024-06-29T14:22:00.304598+0000","last_undegraded":"2024-06-29T14:22:00.304598+0000","last_fullsized":"2024-06-29T14:22:00.304598+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:23.956178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305340+0000","last_change":"2024-06-29T14:21:54.257884+0000","last_active":"2024-06-29T14:22:00.305340+0000","last_peered":"2024-06-29T14:22:00.305340+0000","last_clean":"2024-06-29T14:22:00.305340+0000","last_became_active":"2024-06-29T14:21:54.257540+0000","last_became_peered":"2024-06-29T14:21:54.257540+0000","last_unstale":"2024-06-29T14:22:00.305340+0000","last_undegraded":"2024-06-29T14:22:00.305340+0000","last_fullsized":"2024-06-29T14:22:00.305340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:54.713087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640589+0000","last_change":"2024-06-29T14:21:55.285781+0000","last_active":"2024-06-29T14:22:00.640589+0000","last_peered":"2024-06-29T14:22:00.640589+0000","last_clean":"2024-06-29T14:22:00.640589+0000","last_became_active":"2024-06-29T14:21:55.285358+0000","last_became_peered":"2024-06-29T14:21:55.285358+0000","last_unstale":"2024-06-29T14:22:00.640589+0000","last_undegraded":"2024-06-29T14:22:00.640589+0000","last_fullsized":"2024-06-29T14:22:00.640589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306447+0000","last_change":"2024-06-29T14:21:59.318024+0000","last_active":"2024-06-29T14:22:00.306447+0000","last_peered":"2024-06-29T14:22:00.306447+0000","last_clean":"2024-06-29T14:22:00.306447+0000","last_became_active":"2024-06-29T14:21:59.317356+0000","last_became_peered":"2024-06-29T14:21:59.317356+0000","last_unstale":"2024-06-29T14:22:00.306447+0000","last_undegraded":"2024-06-29T14:22:00.306447+0000","last_fullsized":"2024-06-29T14:22:00.306447+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:00.209167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640906+0000","last_change":"2024-06-29T14:21:54.258802+0000","last_active":"2024-06-29T14:22:00.640906+0000","last_peered":"2024-06-29T14:22:00.640906+0000","last_clean":"2024-06-29T14:22:00.640906+0000","last_became_active":"2024-06-29T14:21:54.258184+0000","last_became_peered":"2024-06-29T14:21:54.258184+0000","last_unstale":"2024-06-29T14:22:00.640906+0000","last_undegraded":"2024-06-29T14:22:00.640906+0000","last_fullsized":"2024-06-29T14:22:00.640906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:07:56.943579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640388+0000","last_change":"2024-06-29T14:21:55.296893+0000","last_active":"2024-06-29T14:22:00.640388+0000","last_peered":"2024-06-29T14:22:00.640388+0000","last_clean":"2024-06-29T14:22:00.640388+0000","last_became_active":"2024-06-29T14:21:55.296552+0000","last_became_peered":"2024-06-29T14:21:55.296552+0000","last_unstale":"2024-06-29T14:22:00.640388+0000","last_undegraded":"2024-06-29T14:22:00.640388+0000","last_fullsized":"2024-06-29T14:22:00.640388+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:57.029702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307221+0000","last_change":"2024-06-29T14:21:59.319287+0000","last_active":"2024-06-29T14:22:00.307221+0000","last_peered":"2024-06-29T14:22:00.307221+0000","last_clean":"2024-06-29T14:22:00.307221+0000","last_became_active":"2024-06-29T14:21:59.318807+0000","last_became_peered":"2024-06-29T14:21:59.318807+0000","last_unstale":"2024-06-29T14:22:00.307221+0000","last_undegraded":"2024-06-29T14:22:00.307221+0000","last_fullsized":"2024-06-29T14:22:00.307221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:20.883221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640313+0000","last_change":"2024-06-29T14:21:54.264911+0000","last_active":"2024-06-29T14:22:00.640313+0000","last_peered":"2024-06-29T14:22:00.640313+0000","last_clean":"2024-06-29T14:22:00.640313+0000","last_became_active":"2024-06-29T14:21:54.264532+0000","last_became_peered":"2024-06-29T14:21:54.264532+0000","last_unstale":"2024-06-29T14:22:00.640313+0000","last_undegraded":"2024-06-29T14:22:00.640313+0000","last_fullsized":"2024-06-29T14:22:00.640313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:10.799778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640442+0000","last_change":"2024-06-29T14:21:55.295115+0000","last_active":"2024-06-29T14:22:00.640442+0000","last_peered":"2024-06-29T14:22:00.640442+0000","last_clean":"2024-06-29T14:22:00.640442+0000","last_became_active":"2024-06-29T14:21:55.294772+0000","last_became_peered":"2024-06-29T14:21:55.294772+0000","last_unstale":"2024-06-29T14:22:00.640442+0000","last_undegraded":"2024-06-29T14:22:00.640442+0000","last_fullsized":"2024-06-29T14:22:00.640442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:17.149243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307463+0000","last_change":"2024-06-29T14:21:59.312924+0000","last_active":"2024-06-29T14:22:00.307463+0000","last_peered":"2024-06-29T14:22:00.307463+0000","last_clean":"2024-06-29T14:22:00.307463+0000","last_became_active":"2024-06-29T14:21:59.312393+0000","last_became_peered":"2024-06-29T14:21:59.312393+0000","last_unstale":"2024-06-29T14:22:00.307463+0000","last_undegraded":"2024-06-29T14:22:00.307463+0000","last_fullsized":"2024-06-29T14:22:00.307463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:32.630422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305149+0000","last_change":"2024-06-29T14:21:55.294165+0000","last_active":"2024-06-29T14:22:00.305149+0000","last_peered":"2024-06-29T14:22:00.305149+0000","last_clean":"2024-06-29T14:22:00.305149+0000","last_became_active":"2024-06-29T14:21:55.293388+0000","last_became_peered":"2024-06-29T14:21:55.293388+0000","last_unstale":"2024-06-29T14:22:00.305149+0000","last_undegraded":"2024-06-29T14:22:00.305149+0000","last_fullsized":"2024-06-29T14:22:00.305149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:50:36.883008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641127+0000","last_change":"2024-06-29T14:21:54.256784+0000","last_active":"2024-06-29T14:22:00.641127+0000","last_peered":"2024-06-29T14:22:00.641127+0000","last_clean":"2024-06-29T14:22:00.641127+0000","last_became_active":"2024-06-29T14:21:54.256434+0000","last_became_peered":"2024-06-29T14:21:54.256434+0000","last_unstale":"2024-06-29T14:22:00.641127+0000","last_undegraded":"2024-06-29T14:22:00.641127+0000","last_fullsized":"2024-06-29T14:22:00.641127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:06.963975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306723+0000","last_change":"2024-06-29T14:21:59.317621+0000","last_active":"2024-06-29T14:22:00.306723+0000","last_peered":"2024-06-29T14:22:00.306723+0000","last_clean":"2024-06-29T14:22:00.306723+0000","last_became_active":"2024-06-29T14:21:59.316736+0000","last_became_peered":"2024-06-29T14:21:59.316736+0000","last_unstale":"2024-06-29T14:22:00.306723+0000","last_undegraded":"2024-06-29T14:22:00.306723+0000","last_fullsized":"2024-06-29T14:22:00.306723+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:43.083466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304951+0000","last_change":"2024-06-29T14:21:55.296036+0000","last_active":"2024-06-29T14:22:00.304951+0000","last_peered":"2024-06-29T14:22:00.304951+0000","last_clean":"2024-06-29T14:22:00.304951+0000","last_became_active":"2024-06-29T14:21:55.294958+0000","last_became_peered":"2024-06-29T14:21:55.294958+0000","last_unstale":"2024-06-29T14:22:00.304951+0000","last_undegraded":"2024-06-29T14:22:00.304951+0000","last_fullsized":"2024-06-29T14:22:00.304951+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:25.523198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641393+0000","last_change":"2024-06-29T14:21:54.249131+0000","last_active":"2024-06-29T14:22:00.641393+0000","last_peered":"2024-06-29T14:22:00.641393+0000","last_clean":"2024-06-29T14:22:00.641393+0000","last_became_active":"2024-06-29T14:21:54.248727+0000","last_became_peered":"2024-06-29T14:21:54.248727+0000","last_unstale":"2024-06-29T14:22:00.641393+0000","last_undegraded":"2024-06-29T14:22:00.641393+0000","last_fullsized":"2024-06-29T14:22:00.641393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:43.007566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306263+0000","last_change":"2024-06-29T14:21:59.318902+0000","last_active":"2024-06-29T14:22:00.306263+0000","last_peered":"2024-06-29T14:22:00.306263+0000","last_clean":"2024-06-29T14:22:00.306263+0000","last_became_active":"2024-06-29T14:21:59.318388+0000","last_became_peered":"2024-06-29T14:21:59.318388+0000","last_unstale":"2024-06-29T14:22:00.306263+0000","last_undegraded":"2024-06-29T14:22:00.306263+0000","last_fullsized":"2024-06-29T14:22:00.306263+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:13.544231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305926+0000","last_change":"2024-06-29T14:21:55.282042+0000","last_active":"2024-06-29T14:22:00.305926+0000","last_peered":"2024-06-29T14:22:00.305926+0000","last_clean":"2024-06-29T14:22:00.305926+0000","last_became_active":"2024-06-29T14:21:55.281259+0000","last_became_peered":"2024-06-29T14:21:55.281259+0000","last_unstale":"2024-06-29T14:22:00.305926+0000","last_undegraded":"2024-06-29T14:22:00.305926+0000","last_fullsized":"2024-06-29T14:22:00.305926+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640041+0000","last_change":"2024-06-29T14:21:54.255093+0000","last_active":"2024-06-29T14:22:00.640041+0000","last_peered":"2024-06-29T14:22:00.640041+0000","last_clean":"2024-06-29T14:22:00.640041+0000","last_became_active":"2024-06-29T14:21:54.254010+0000","last_became_peered":"2024-06-29T14:21:54.254010+0000","last_unstale":"2024-06-29T14:22:00.640041+0000","last_undegraded":"2024-06-29T14:22:00.640041+0000","last_fullsized":"2024-06-29T14:22:00.640041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:24.835712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.642067+0000","last_change":"2024-06-29T14:21:59.337086+0000","last_active":"2024-06-29T14:22:00.642067+0000","last_peered":"2024-06-29T14:22:00.642067+0000","last_clean":"2024-06-29T14:22:00.642067+0000","last_became_active":"2024-06-29T14:21:59.336659+0000","last_became_peered":"2024-06-29T14:21:59.336659+0000","last_unstale":"2024-06-29T14:22:00.642067+0000","last_undegraded":"2024-06-29T14:22:00.642067+0000","last_fullsized":"2024-06-29T14:22:00.642067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:47.549181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306644+0000","last_change":"2024-06-29T14:21:55.289199+0000","last_active":"2024-06-29T14:22:00.306644+0000","last_peered":"2024-06-29T14:22:00.306644+0000","last_clean":"2024-06-29T14:22:00.306644+0000","last_became_active":"2024-06-29T14:21:55.288787+0000","last_became_peered":"2024-06-29T14:21:55.288787+0000","last_unstale":"2024-06-29T14:22:00.306644+0000","last_undegraded":"2024-06-29T14:22:00.306644+0000","last_fullsized":"2024-06-29T14:22:00.306644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:33.390422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640397+0000","last_change":"2024-06-29T14:21:54.260787+0000","last_active":"2024-06-29T14:22:00.640397+0000","last_peered":"2024-06-29T14:22:00.640397+0000","last_clean":"2024-06-29T14:22:00.640397+0000","last_became_active":"2024-06-29T14:21:54.260352+0000","last_became_peered":"2024-06-29T14:21:54.260352+0000","last_unstale":"2024-06-29T14:22:00.640397+0000","last_undegraded":"2024-06-29T14:22:00.640397+0000","last_fullsized":"2024-06-29T14:22:00.640397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:40.455851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305896+0000","last_change":"2024-06-29T14:21:59.316811+0000","last_active":"2024-06-29T14:22:00.305896+0000","last_peered":"2024-06-29T14:22:00.305896+0000","last_clean":"2024-06-29T14:22:00.305896+0000","last_became_active":"2024-06-29T14:21:59.316067+0000","last_became_peered":"2024-06-29T14:21:59.316067+0000","last_unstale":"2024-06-29T14:22:00.305896+0000","last_undegraded":"2024-06-29T14:22:00.305896+0000","last_fullsized":"2024-06-29T14:22:00.305896+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:04.444868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640138+0000","last_change":"2024-06-29T14:21:55.293209+0000","last_active":"2024-06-29T14:22:00.640138+0000","last_peered":"2024-06-29T14:22:00.640138+0000","last_clean":"2024-06-29T14:22:00.640138+0000","last_became_active":"2024-06-29T14:21:55.291574+0000","last_became_peered":"2024-06-29T14:21:55.291574+0000","last_unstale":"2024-06-29T14:22:00.640138+0000","last_undegraded":"2024-06-29T14:22:00.640138+0000","last_fullsized":"2024-06-29T14:22:00.640138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:16.165478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305885+0000","last_change":"2024-06-29T14:21:54.256099+0000","last_active":"2024-06-29T14:22:00.305885+0000","last_peered":"2024-06-29T14:22:00.305885+0000","last_clean":"2024-06-29T14:22:00.305885+0000","last_became_active":"2024-06-29T14:21:54.254022+0000","last_became_peered":"2024-06-29T14:21:54.254022+0000","last_unstale":"2024-06-29T14:22:00.305885+0000","last_undegraded":"2024-06-29T14:22:00.305885+0000","last_fullsized":"2024-06-29T14:22:00.305885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:32.388331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305919+0000","last_change":"2024-06-29T14:21:59.337078+0000","last_active":"2024-06-29T14:22:00.305919+0000","last_peered":"2024-06-29T14:22:00.305919+0000","last_clean":"2024-06-29T14:22:00.305919+0000","last_became_active":"2024-06-29T14:21:59.336616+0000","last_became_peered":"2024-06-29T14:21:59.336616+0000","last_unstale":"2024-06-29T14:22:00.305919+0000","last_undegraded":"2024-06-29T14:22:00.305919+0000","last_fullsized":"2024-06-29T14:22:00.305919+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:33:53.623576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305915+0000","last_change":"2024-06-29T14:21:55.293645+0000","last_active":"2024-06-29T14:22:00.305915+0000","last_peered":"2024-06-29T14:22:00.305915+0000","last_clean":"2024-06-29T14:22:00.305915+0000","last_became_active":"2024-06-29T14:21:55.291467+0000","last_became_peered":"2024-06-29T14:21:55.291467+0000","last_unstale":"2024-06-29T14:22:00.305915+0000","last_undegraded":"2024-06-29T14:22:00.305915+0000","last_fullsized":"2024-06-29T14:22:00.305915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.505918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641935+0000","last_change":"2024-06-29T14:21:54.244171+0000","last_active":"2024-06-29T14:22:00.641935+0000","last_peered":"2024-06-29T14:22:00.641935+0000","last_clean":"2024-06-29T14:22:00.641935+0000","last_became_active":"2024-06-29T14:21:54.243791+0000","last_became_peered":"2024-06-29T14:21:54.243791+0000","last_unstale":"2024-06-29T14:22:00.641935+0000","last_undegraded":"2024-06-29T14:22:00.641935+0000","last_fullsized":"2024-06-29T14:22:00.641935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:24.194470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304468+0000","last_change":"2024-06-29T14:21:59.336963+0000","last_active":"2024-06-29T14:22:00.304468+0000","last_peered":"2024-06-29T14:22:00.304468+0000","last_clean":"2024-06-29T14:22:00.304468+0000","last_became_active":"2024-06-29T14:21:59.336450+0000","last_became_peered":"2024-06-29T14:21:59.336450+0000","last_unstale":"2024-06-29T14:22:00.304468+0000","last_undegraded":"2024-06-29T14:22:00.304468+0000","last_fullsized":"2024-06-29T14:22:00.304468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:00.347317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641725+0000","last_change":"2024-06-29T14:21:55.271686+0000","last_active":"2024-06-29T14:22:00.641725+0000","last_peered":"2024-06-29T14:22:00.641725+0000","last_clean":"2024-06-29T14:22:00.641725+0000","last_became_active":"2024-06-29T14:21:55.271310+0000","last_became_peered":"2024-06-29T14:21:55.271310+0000","last_unstale":"2024-06-29T14:22:00.641725+0000","last_undegraded":"2024-06-29T14:22:00.641725+0000","last_fullsized":"2024-06-29T14:22:00.641725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641232+0000","last_change":"2024-06-29T14:21:54.260566+0000","last_active":"2024-06-29T14:22:00.641232+0000","last_peered":"2024-06-29T14:22:00.641232+0000","last_clean":"2024-06-29T14:22:00.641232+0000","last_became_active":"2024-06-29T14:21:54.260089+0000","last_became_peered":"2024-06-29T14:21:54.260089+0000","last_unstale":"2024-06-29T14:22:00.641232+0000","last_undegraded":"2024-06-29T14:22:00.641232+0000","last_fullsized":"2024-06-29T14:22:00.641232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:28.957956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640300+0000","last_change":"2024-06-29T14:21:59.338874+0000","last_active":"2024-06-29T14:22:00.640300+0000","last_peered":"2024-06-29T14:22:00.640300+0000","last_clean":"2024-06-29T14:22:00.640300+0000","last_became_active":"2024-06-29T14:21:59.338388+0000","last_became_peered":"2024-06-29T14:21:59.338388+0000","last_unstale":"2024-06-29T14:22:00.640300+0000","last_undegraded":"2024-06-29T14:22:00.640300+0000","last_fullsized":"2024-06-29T14:22:00.640300+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:21.205918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641110+0000","last_change":"2024-06-29T14:21:55.284217+0000","last_active":"2024-06-29T14:22:00.641110+0000","last_peered":"2024-06-29T14:22:00.641110+0000","last_clean":"2024-06-29T14:22:00.641110+0000","last_became_active":"2024-06-29T14:21:55.283906+0000","last_became_peered":"2024-06-29T14:21:55.283906+0000","last_unstale":"2024-06-29T14:22:00.641110+0000","last_undegraded":"2024-06-29T14:22:00.641110+0000","last_fullsized":"2024-06-29T14:22:00.641110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:39.832722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641864+0000","last_change":"2024-06-29T14:21:54.257367+0000","last_active":"2024-06-29T14:22:00.641864+0000","last_peered":"2024-06-29T14:22:00.641864+0000","last_clean":"2024-06-29T14:22:00.641864+0000","last_became_active":"2024-06-29T14:21:54.257040+0000","last_became_peered":"2024-06-29T14:21:54.257040+0000","last_unstale":"2024-06-29T14:22:00.641864+0000","last_undegraded":"2024-06-29T14:22:00.641864+0000","last_fullsized":"2024-06-29T14:22:00.641864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:53.425388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639860+0000","last_change":"2024-06-29T14:21:59.323943+0000","last_active":"2024-06-29T14:22:00.639860+0000","last_peered":"2024-06-29T14:22:00.639860+0000","last_clean":"2024-06-29T14:22:00.639860+0000","last_became_active":"2024-06-29T14:21:59.323400+0000","last_became_peered":"2024-06-29T14:21:59.323400+0000","last_unstale":"2024-06-29T14:22:00.639860+0000","last_undegraded":"2024-06-29T14:22:00.639860+0000","last_fullsized":"2024-06-29T14:22:00.639860+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:01.086377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307356+0000","last_change":"2024-06-29T14:21:55.292906+0000","last_active":"2024-06-29T14:22:00.307356+0000","last_peered":"2024-06-29T14:22:00.307356+0000","last_clean":"2024-06-29T14:22:00.307356+0000","last_became_active":"2024-06-29T14:21:55.292578+0000","last_became_peered":"2024-06-29T14:21:55.292578+0000","last_unstale":"2024-06-29T14:22:00.307356+0000","last_undegraded":"2024-06-29T14:22:00.307356+0000","last_fullsized":"2024-06-29T14:22:00.307356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:11.801869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640191+0000","last_change":"2024-06-29T14:21:54.264501+0000","last_active":"2024-06-29T14:22:00.640191+0000","last_peered":"2024-06-29T14:22:00.640191+0000","last_clean":"2024-06-29T14:22:00.640191+0000","last_became_active":"2024-06-29T14:21:54.264182+0000","last_became_peered":"2024-06-29T14:21:54.264182+0000","last_unstale":"2024-06-29T14:22:00.640191+0000","last_undegraded":"2024-06-29T14:22:00.640191+0000","last_fullsized":"2024-06-29T14:22:00.640191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:53.425388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641814+0000","last_change":"2024-06-29T14:21:59.324707+0000","last_active":"2024-06-29T14:22:00.641814+0000","last_peered":"2024-06-29T14:22:00.641814+0000","last_clean":"2024-06-29T14:22:00.641814+0000","last_became_active":"2024-06-29T14:21:59.324141+0000","last_became_peered":"2024-06-29T14:21:59.324141+0000","last_unstale":"2024-06-29T14:22:00.641814+0000","last_undegraded":"2024-06-29T14:22:00.641814+0000","last_fullsized":"2024-06-29T14:22:00.641814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:43.889397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639772+0000","last_change":"2024-06-29T14:21:55.274636+0000","last_active":"2024-06-29T14:22:00.639772+0000","last_peered":"2024-06-29T14:22:00.639772+0000","last_clean":"2024-06-29T14:22:00.639772+0000","last_became_active":"2024-06-29T14:21:55.271330+0000","last_became_peered":"2024-06-29T14:21:55.271330+0000","last_unstale":"2024-06-29T14:22:00.639772+0000","last_undegraded":"2024-06-29T14:22:00.639772+0000","last_fullsized":"2024-06-29T14:22:00.639772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640544+0000","last_change":"2024-06-29T14:21:54.247945+0000","last_active":"2024-06-29T14:22:00.640544+0000","last_peered":"2024-06-29T14:22:00.640544+0000","last_clean":"2024-06-29T14:22:00.640544+0000","last_became_active":"2024-06-29T14:21:54.247592+0000","last_became_peered":"2024-06-29T14:21:54.247592+0000","last_unstale":"2024-06-29T14:22:00.640544+0000","last_undegraded":"2024-06-29T14:22:00.640544+0000","last_fullsized":"2024-06-29T14:22:00.640544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:03.449985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.642194+0000","last_change":"2024-06-29T14:21:59.321043+0000","last_active":"2024-06-29T14:22:00.642194+0000","last_peered":"2024-06-29T14:22:00.642194+0000","last_clean":"2024-06-29T14:22:00.642194+0000","last_became_active":"2024-06-29T14:21:59.320359+0000","last_became_peered":"2024-06-29T14:21:59.320359+0000","last_unstale":"2024-06-29T14:22:00.642194+0000","last_undegraded":"2024-06-29T14:22:00.642194+0000","last_fullsized":"2024-06-29T14:22:00.642194+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:22.691854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639595+0000","last_change":"2024-06-29T14:21:55.296697+0000","last_active":"2024-06-29T14:22:00.639595+0000","last_peered":"2024-06-29T14:22:00.639595+0000","last_clean":"2024-06-29T14:22:00.639595+0000","last_became_active":"2024-06-29T14:21:55.296392+0000","last_became_peered":"2024-06-29T14:21:55.296392+0000","last_unstale":"2024-06-29T14:22:00.639595+0000","last_undegraded":"2024-06-29T14:22:00.639595+0000","last_fullsized":"2024-06-29T14:22:00.639595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:17.069875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639911+0000","last_change":"2024-06-29T14:21:54.257791+0000","last_active":"2024-06-29T14:22:00.639911+0000","last_peered":"2024-06-29T14:22:00.639911+0000","last_clean":"2024-06-29T14:22:00.639911+0000","last_became_active":"2024-06-29T14:21:54.257395+0000","last_became_peered":"2024-06-29T14:21:54.257395+0000","last_unstale":"2024-06-29T14:22:00.639911+0000","last_undegraded":"2024-06-29T14:22:00.639911+0000","last_fullsized":"2024-06-29T14:22:00.639911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:33.159720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640058+0000","last_change":"2024-06-29T14:21:59.337618+0000","last_active":"2024-06-29T14:22:00.640058+0000","last_peered":"2024-06-29T14:22:00.640058+0000","last_clean":"2024-06-29T14:22:00.640058+0000","last_became_active":"2024-06-29T14:21:59.336503+0000","last_became_peered":"2024-06-29T14:21:59.336503+0000","last_unstale":"2024-06-29T14:22:00.640058+0000","last_undegraded":"2024-06-29T14:22:00.640058+0000","last_fullsized":"2024-06-29T14:22:00.640058+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:01.086377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304940+0000","last_change":"2024-06-29T14:21:55.284345+0000","last_active":"2024-06-29T14:22:00.304940+0000","last_peered":"2024-06-29T14:22:00.304940+0000","last_clean":"2024-06-29T14:22:00.304940+0000","last_became_active":"2024-06-29T14:21:55.283603+0000","last_became_peered":"2024-06-29T14:21:55.283603+0000","last_unstale":"2024-06-29T14:22:00.304940+0000","last_undegraded":"2024-06-29T14:22:00.304940+0000","last_fullsized":"2024-06-29T14:22:00.304940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639902+0000","last_change":"2024-06-29T14:21:54.259298+0000","last_active":"2024-06-29T14:22:00.639902+0000","last_peered":"2024-06-29T14:22:00.639902+0000","last_clean":"2024-06-29T14:22:00.639902+0000","last_became_active":"2024-06-29T14:21:54.258971+0000","last_became_peered":"2024-06-29T14:21:54.258971+0000","last_unstale":"2024-06-29T14:22:00.639902+0000","last_undegraded":"2024-06-29T14:22:00.639902+0000","last_fullsized":"2024-06-29T14:22:00.639902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:34.839629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":117,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640060+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:22:00.640060+0000","last_peered":"2024-06-29T14:22:00.640060+0000","last_clean":"2024-06-29T14:22:00.640060+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:22:00.640060+0000","last_undegraded":"2024-06-29T14:22:00.640060+0000","last_fullsized":"2024-06-29T14:22:00.640060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305693+0000","last_change":"2024-06-29T14:21:59.337431+0000","last_active":"2024-06-29T14:22:00.305693+0000","last_peered":"2024-06-29T14:22:00.305693+0000","last_clean":"2024-06-29T14:22:00.305693+0000","last_became_active":"2024-06-29T14:21:59.337008+0000","last_became_peered":"2024-06-29T14:21:59.337008+0000","last_unstale":"2024-06-29T14:22:00.305693+0000","last_undegraded":"2024-06-29T14:22:00.305693+0000","last_fullsized":"2024-06-29T14:22:00.305693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:56.220106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640737+0000","last_change":"2024-06-29T14:21:55.287255+0000","last_active":"2024-06-29T14:22:00.640737+0000","last_peered":"2024-06-29T14:22:00.640737+0000","last_clean":"2024-06-29T14:22:00.640737+0000","last_became_active":"2024-06-29T14:21:55.286563+0000","last_became_peered":"2024-06-29T14:21:55.286563+0000","last_unstale":"2024-06-29T14:22:00.640737+0000","last_undegraded":"2024-06-29T14:22:00.640737+0000","last_fullsized":"2024-06-29T14:22:00.640737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:40.442251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640792+0000","last_change":"2024-06-29T14:21:54.264774+0000","last_active":"2024-06-29T14:22:00.640792+0000","last_peered":"2024-06-29T14:22:00.640792+0000","last_clean":"2024-06-29T14:22:00.640792+0000","last_became_active":"2024-06-29T14:21:54.264395+0000","last_became_peered":"2024-06-29T14:21:54.264395+0000","last_unstale":"2024-06-29T14:22:00.640792+0000","last_undegraded":"2024-06-29T14:22:00.640792+0000","last_fullsized":"2024-06-29T14:22:00.640792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:49.746280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305477+0000","last_change":"2024-06-29T14:21:59.336774+0000","last_active":"2024-06-29T14:22:00.305477+0000","last_peered":"2024-06-29T14:22:00.305477+0000","last_clean":"2024-06-29T14:22:00.305477+0000","last_became_active":"2024-06-29T14:21:59.336326+0000","last_became_peered":"2024-06-29T14:21:59.336326+0000","last_unstale":"2024-06-29T14:22:00.305477+0000","last_undegraded":"2024-06-29T14:22:00.305477+0000","last_fullsized":"2024-06-29T14:22:00.305477+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:20.883221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640544+0000","last_change":"2024-06-29T14:21:55.289532+0000","last_active":"2024-06-29T14:22:00.640544+0000","last_peered":"2024-06-29T14:22:00.640544+0000","last_clean":"2024-06-29T14:22:00.640544+0000","last_became_active":"2024-06-29T14:21:55.289210+0000","last_became_peered":"2024-06-29T14:21:55.289210+0000","last_unstale":"2024-06-29T14:22:00.640544+0000","last_undegraded":"2024-06-29T14:22:00.640544+0000","last_fullsized":"2024-06-29T14:22:00.640544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:12.073214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641944+0000","last_change":"2024-06-29T14:21:54.246332+0000","last_active":"2024-06-29T14:22:00.641944+0000","last_peered":"2024-06-29T14:22:00.641944+0000","last_clean":"2024-06-29T14:22:00.641944+0000","last_became_active":"2024-06-29T14:21:54.245974+0000","last_became_peered":"2024-06-29T14:21:54.245974+0000","last_unstale":"2024-06-29T14:22:00.641944+0000","last_undegraded":"2024-06-29T14:22:00.641944+0000","last_fullsized":"2024-06-29T14:22:00.641944+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:50:54.491455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640419+0000","last_change":"2024-06-29T14:21:59.318499+0000","last_active":"2024-06-29T14:22:00.640419+0000","last_peered":"2024-06-29T14:22:00.640419+0000","last_clean":"2024-06-29T14:22:00.640419+0000","last_became_active":"2024-06-29T14:21:59.317918+0000","last_became_peered":"2024-06-29T14:21:59.317918+0000","last_unstale":"2024-06-29T14:22:00.640419+0000","last_undegraded":"2024-06-29T14:22:00.640419+0000","last_fullsized":"2024-06-29T14:22:00.640419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:23:43.889397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641394+0000","last_change":"2024-06-29T14:21:55.294388+0000","last_active":"2024-06-29T14:22:00.641394+0000","last_peered":"2024-06-29T14:22:00.641394+0000","last_clean":"2024-06-29T14:22:00.641394+0000","last_became_active":"2024-06-29T14:21:55.294090+0000","last_became_peered":"2024-06-29T14:21:55.294090+0000","last_unstale":"2024-06-29T14:22:00.641394+0000","last_undegraded":"2024-06-29T14:22:00.641394+0000","last_fullsized":"2024-06-29T14:22:00.641394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:39.325162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641102+0000","last_change":"2024-06-29T14:21:54.252699+0000","last_active":"2024-06-29T14:22:00.641102+0000","last_peered":"2024-06-29T14:22:00.641102+0000","last_clean":"2024-06-29T14:22:00.641102+0000","last_became_active":"2024-06-29T14:21:54.252365+0000","last_became_peered":"2024-06-29T14:21:54.252365+0000","last_unstale":"2024-06-29T14:22:00.641102+0000","last_undegraded":"2024-06-29T14:22:00.641102+0000","last_fullsized":"2024-06-29T14:22:00.641102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:32.388331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640359+0000","last_change":"2024-06-29T14:21:59.337020+0000","last_active":"2024-06-29T14:22:00.640359+0000","last_peered":"2024-06-29T14:22:00.640359+0000","last_clean":"2024-06-29T14:22:00.640359+0000","last_became_active":"2024-06-29T14:21:59.336457+0000","last_became_peered":"2024-06-29T14:21:59.336457+0000","last_unstale":"2024-06-29T14:22:00.640359+0000","last_undegraded":"2024-06-29T14:22:00.640359+0000","last_fullsized":"2024-06-29T14:22:00.640359+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:25.501876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640773+0000","last_change":"2024-06-29T14:21:55.297155+0000","last_active":"2024-06-29T14:22:00.640773+0000","last_peered":"2024-06-29T14:22:00.640773+0000","last_clean":"2024-06-29T14:22:00.640773+0000","last_became_active":"2024-06-29T14:21:55.296790+0000","last_became_peered":"2024-06-29T14:21:55.296790+0000","last_unstale":"2024-06-29T14:22:00.640773+0000","last_undegraded":"2024-06-29T14:22:00.640773+0000","last_fullsized":"2024-06-29T14:22:00.640773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:52.163431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306370+0000","last_change":"2024-06-29T14:21:54.261995+0000","last_active":"2024-06-29T14:22:00.306370+0000","last_peered":"2024-06-29T14:22:00.306370+0000","last_clean":"2024-06-29T14:22:00.306370+0000","last_became_active":"2024-06-29T14:21:54.261607+0000","last_became_peered":"2024-06-29T14:21:54.261607+0000","last_unstale":"2024-06-29T14:22:00.306370+0000","last_undegraded":"2024-06-29T14:22:00.306370+0000","last_fullsized":"2024-06-29T14:22:00.306370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:59.466378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305836+0000","last_change":"2024-06-29T14:21:59.316601+0000","last_active":"2024-06-29T14:22:00.305836+0000","last_peered":"2024-06-29T14:22:00.305836+0000","last_clean":"2024-06-29T14:22:00.305836+0000","last_became_active":"2024-06-29T14:21:59.315986+0000","last_became_peered":"2024-06-29T14:21:59.315986+0000","last_unstale":"2024-06-29T14:22:00.305836+0000","last_undegraded":"2024-06-29T14:22:00.305836+0000","last_fullsized":"2024-06-29T14:22:00.305836+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:45.085147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641018+0000","last_change":"2024-06-29T14:21:55.291524+0000","last_active":"2024-06-29T14:22:00.641018+0000","last_peered":"2024-06-29T14:22:00.641018+0000","last_clean":"2024-06-29T14:22:00.641018+0000","last_became_active":"2024-06-29T14:21:55.291206+0000","last_became_peered":"2024-06-29T14:21:55.291206+0000","last_unstale":"2024-06-29T14:22:00.641018+0000","last_undegraded":"2024-06-29T14:22:00.641018+0000","last_fullsized":"2024-06-29T14:22:00.641018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:16.826546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640410+0000","last_change":"2024-06-29T14:21:54.245875+0000","last_active":"2024-06-29T14:22:00.640410+0000","last_peered":"2024-06-29T14:22:00.640410+0000","last_clean":"2024-06-29T14:22:00.640410+0000","last_became_active":"2024-06-29T14:21:54.245444+0000","last_became_peered":"2024-06-29T14:21:54.245444+0000","last_unstale":"2024-06-29T14:22:00.640410+0000","last_undegraded":"2024-06-29T14:22:00.640410+0000","last_fullsized":"2024-06-29T14:22:00.640410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:49.347688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639988+0000","last_change":"2024-06-29T14:21:59.317611+0000","last_active":"2024-06-29T14:22:00.639988+0000","last_peered":"2024-06-29T14:22:00.639988+0000","last_clean":"2024-06-29T14:22:00.639988+0000","last_became_active":"2024-06-29T14:21:59.317081+0000","last_became_peered":"2024-06-29T14:21:59.317081+0000","last_unstale":"2024-06-29T14:22:00.639988+0000","last_undegraded":"2024-06-29T14:22:00.639988+0000","last_fullsized":"2024-06-29T14:22:00.639988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:16.129888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305414+0000","last_change":"2024-06-29T14:21:55.291070+0000","last_active":"2024-06-29T14:22:00.305414+0000","last_peered":"2024-06-29T14:22:00.305414+0000","last_clean":"2024-06-29T14:22:00.305414+0000","last_became_active":"2024-06-29T14:21:55.290468+0000","last_became_peered":"2024-06-29T14:21:55.290468+0000","last_unstale":"2024-06-29T14:22:00.305414+0000","last_undegraded":"2024-06-29T14:22:00.305414+0000","last_fullsized":"2024-06-29T14:22:00.305414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:15.443094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305532+0000","last_change":"2024-06-29T14:21:54.257111+0000","last_active":"2024-06-29T14:22:00.305532+0000","last_peered":"2024-06-29T14:22:00.305532+0000","last_clean":"2024-06-29T14:22:00.305532+0000","last_became_active":"2024-06-29T14:21:54.255536+0000","last_became_peered":"2024-06-29T14:21:54.255536+0000","last_unstale":"2024-06-29T14:22:00.305532+0000","last_undegraded":"2024-06-29T14:22:00.305532+0000","last_fullsized":"2024-06-29T14:22:00.305532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:28:41.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305863+0000","last_change":"2024-06-29T14:21:59.338565+0000","last_active":"2024-06-29T14:22:00.305863+0000","last_peered":"2024-06-29T14:22:00.305863+0000","last_clean":"2024-06-29T14:22:00.305863+0000","last_became_active":"2024-06-29T14:21:59.337920+0000","last_became_peered":"2024-06-29T14:21:59.337920+0000","last_unstale":"2024-06-29T14:22:00.305863+0000","last_undegraded":"2024-06-29T14:22:00.305863+0000","last_fullsized":"2024-06-29T14:22:00.305863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:46.890912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640755+0000","last_change":"2024-06-29T14:21:55.294165+0000","last_active":"2024-06-29T14:22:00.640755+0000","last_peered":"2024-06-29T14:22:00.640755+0000","last_clean":"2024-06-29T14:22:00.640755+0000","last_became_active":"2024-06-29T14:21:55.293742+0000","last_became_peered":"2024-06-29T14:21:55.293742+0000","last_unstale":"2024-06-29T14:22:00.640755+0000","last_undegraded":"2024-06-29T14:22:00.640755+0000","last_fullsized":"2024-06-29T14:22:00.640755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:35.044667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640160+0000","last_change":"2024-06-29T14:21:54.261554+0000","last_active":"2024-06-29T14:22:00.640160+0000","last_peered":"2024-06-29T14:22:00.640160+0000","last_clean":"2024-06-29T14:22:00.640160+0000","last_became_active":"2024-06-29T14:21:54.261215+0000","last_became_peered":"2024-06-29T14:21:54.261215+0000","last_unstale":"2024-06-29T14:22:00.640160+0000","last_undegraded":"2024-06-29T14:22:00.640160+0000","last_fullsized":"2024-06-29T14:22:00.640160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:43.007566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305468+0000","last_change":"2024-06-29T14:21:59.338884+0000","last_active":"2024-06-29T14:22:00.305468+0000","last_peered":"2024-06-29T14:22:00.305468+0000","last_clean":"2024-06-29T14:22:00.305468+0000","last_became_active":"2024-06-29T14:21:59.336402+0000","last_became_peered":"2024-06-29T14:21:59.336402+0000","last_unstale":"2024-06-29T14:22:00.305468+0000","last_undegraded":"2024-06-29T14:22:00.305468+0000","last_fullsized":"2024-06-29T14:22:00.305468+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:00.545575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640161+0000","last_change":"2024-06-29T14:21:55.288628+0000","last_active":"2024-06-29T14:22:00.640161+0000","last_peered":"2024-06-29T14:22:00.640161+0000","last_clean":"2024-06-29T14:22:00.640161+0000","last_became_active":"2024-06-29T14:21:55.288079+0000","last_became_peered":"2024-06-29T14:21:55.288079+0000","last_unstale":"2024-06-29T14:22:00.640161+0000","last_undegraded":"2024-06-29T14:22:00.640161+0000","last_fullsized":"2024-06-29T14:22:00.640161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:44.658689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640560+0000","last_change":"2024-06-29T14:21:54.263230+0000","last_active":"2024-06-29T14:22:00.640560+0000","last_peered":"2024-06-29T14:22:00.640560+0000","last_clean":"2024-06-29T14:22:00.640560+0000","last_became_active":"2024-06-29T14:21:54.262952+0000","last_became_peered":"2024-06-29T14:21:54.262952+0000","last_unstale":"2024-06-29T14:22:00.640560+0000","last_undegraded":"2024-06-29T14:22:00.640560+0000","last_fullsized":"2024-06-29T14:22:00.640560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:48.032837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640470+0000","last_change":"2024-06-29T14:21:59.337082+0000","last_active":"2024-06-29T14:22:00.640470+0000","last_peered":"2024-06-29T14:22:00.640470+0000","last_clean":"2024-06-29T14:22:00.640470+0000","last_became_active":"2024-06-29T14:21:59.336506+0000","last_became_peered":"2024-06-29T14:21:59.336506+0000","last_unstale":"2024-06-29T14:22:00.640470+0000","last_undegraded":"2024-06-29T14:22:00.640470+0000","last_fullsized":"2024-06-29T14:22:00.640470+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:00.347317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640609+0000","last_change":"2024-06-29T14:21:55.299489+0000","last_active":"2024-06-29T14:22:00.640609+0000","last_peered":"2024-06-29T14:22:00.640609+0000","last_clean":"2024-06-29T14:22:00.640609+0000","last_became_active":"2024-06-29T14:21:55.299181+0000","last_became_peered":"2024-06-29T14:21:55.299181+0000","last_unstale":"2024-06-29T14:22:00.640609+0000","last_undegraded":"2024-06-29T14:22:00.640609+0000","last_fullsized":"2024-06-29T14:22:00.640609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:31.165204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:05.053670+0000","last_change":"2024-06-29T14:21:54.251485+0000","last_active":"2024-06-29T14:22:05.053670+0000","last_peered":"2024-06-29T14:22:05.053670+0000","last_clean":"2024-06-29T14:22:05.053670+0000","last_became_active":"2024-06-29T14:21:54.250203+0000","last_became_peered":"2024-06-29T14:21:54.250203+0000","last_unstale":"2024-06-29T14:22:05.053670+0000","last_undegraded":"2024-06-29T14:22:05.053670+0000","last_fullsized":"2024-06-29T14:22:05.053670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:03.449985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641539+0000","last_change":"2024-06-29T14:21:59.324423+0000","last_active":"2024-06-29T14:22:00.641539+0000","last_peered":"2024-06-29T14:22:00.641539+0000","last_clean":"2024-06-29T14:22:00.641539+0000","last_became_active":"2024-06-29T14:21:59.323663+0000","last_became_peered":"2024-06-29T14:21:59.323663+0000","last_unstale":"2024-06-29T14:22:00.641539+0000","last_undegraded":"2024-06-29T14:22:00.641539+0000","last_fullsized":"2024-06-29T14:22:00.641539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:21.205918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307509+0000","last_change":"2024-06-29T14:21:55.292602+0000","last_active":"2024-06-29T14:22:00.307509+0000","last_peered":"2024-06-29T14:22:00.307509+0000","last_clean":"2024-06-29T14:22:00.307509+0000","last_became_active":"2024-06-29T14:21:55.292047+0000","last_became_peered":"2024-06-29T14:21:55.292047+0000","last_unstale":"2024-06-29T14:22:00.307509+0000","last_undegraded":"2024-06-29T14:22:00.307509+0000","last_fullsized":"2024-06-29T14:22:00.307509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:15.123791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641934+0000","last_change":"2024-06-29T14:21:54.264727+0000","last_active":"2024-06-29T14:22:00.641934+0000","last_peered":"2024-06-29T14:22:00.641934+0000","last_clean":"2024-06-29T14:22:00.641934+0000","last_became_active":"2024-06-29T14:21:54.264355+0000","last_became_peered":"2024-06-29T14:21:54.264355+0000","last_unstale":"2024-06-29T14:22:00.641934+0000","last_undegraded":"2024-06-29T14:22:00.641934+0000","last_fullsized":"2024-06-29T14:22:00.641934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:06.025267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306568+0000","last_change":"2024-06-29T14:21:59.318452+0000","last_active":"2024-06-29T14:22:00.306568+0000","last_peered":"2024-06-29T14:22:00.306568+0000","last_clean":"2024-06-29T14:22:00.306568+0000","last_became_active":"2024-06-29T14:21:59.317634+0000","last_became_peered":"2024-06-29T14:21:59.317634+0000","last_unstale":"2024-06-29T14:22:00.306568+0000","last_undegraded":"2024-06-29T14:22:00.306568+0000","last_fullsized":"2024-06-29T14:22:00.306568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:46.219297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640791+0000","last_change":"2024-06-29T14:21:55.284276+0000","last_active":"2024-06-29T14:22:00.640791+0000","last_peered":"2024-06-29T14:22:00.640791+0000","last_clean":"2024-06-29T14:22:00.640791+0000","last_became_active":"2024-06-29T14:21:55.283929+0000","last_became_peered":"2024-06-29T14:21:55.283929+0000","last_unstale":"2024-06-29T14:22:00.640791+0000","last_undegraded":"2024-06-29T14:22:00.640791+0000","last_fullsized":"2024-06-29T14:22:00.640791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640975+0000","last_change":"2024-06-29T14:21:54.260674+0000","last_active":"2024-06-29T14:22:00.640975+0000","last_peered":"2024-06-29T14:22:00.640975+0000","last_clean":"2024-06-29T14:22:00.640975+0000","last_became_active":"2024-06-29T14:21:54.258991+0000","last_became_peered":"2024-06-29T14:21:54.258991+0000","last_unstale":"2024-06-29T14:22:00.640975+0000","last_undegraded":"2024-06-29T14:22:00.640975+0000","last_fullsized":"2024-06-29T14:22:00.640975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:50:54.491455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305903+0000","last_change":"2024-06-29T14:21:59.319141+0000","last_active":"2024-06-29T14:22:00.305903+0000","last_peered":"2024-06-29T14:22:00.305903+0000","last_clean":"2024-06-29T14:22:00.305903+0000","last_became_active":"2024-06-29T14:21:59.318645+0000","last_became_peered":"2024-06-29T14:21:59.318645+0000","last_unstale":"2024-06-29T14:22:00.305903+0000","last_undegraded":"2024-06-29T14:22:00.305903+0000","last_fullsized":"2024-06-29T14:22:00.305903+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:50.697736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640894+0000","last_change":"2024-06-29T14:21:55.293028+0000","last_active":"2024-06-29T14:22:00.640894+0000","last_peered":"2024-06-29T14:22:00.640894+0000","last_clean":"2024-06-29T14:22:00.640894+0000","last_became_active":"2024-06-29T14:21:55.292709+0000","last_became_peered":"2024-06-29T14:21:55.292709+0000","last_unstale":"2024-06-29T14:22:00.640894+0000","last_undegraded":"2024-06-29T14:22:00.640894+0000","last_fullsized":"2024-06-29T14:22:00.640894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:30.912272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305837+0000","last_change":"2024-06-29T14:21:54.262667+0000","last_active":"2024-06-29T14:22:00.305837+0000","last_peered":"2024-06-29T14:22:00.305837+0000","last_clean":"2024-06-29T14:22:00.305837+0000","last_became_active":"2024-06-29T14:21:54.262286+0000","last_became_peered":"2024-06-29T14:21:54.262286+0000","last_unstale":"2024-06-29T14:22:00.305837+0000","last_undegraded":"2024-06-29T14:22:00.305837+0000","last_fullsized":"2024-06-29T14:22:00.305837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:49.746280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304956+0000","last_change":"2024-06-29T14:21:59.337794+0000","last_active":"2024-06-29T14:22:00.304956+0000","last_peered":"2024-06-29T14:22:00.304956+0000","last_clean":"2024-06-29T14:22:00.304956+0000","last_became_active":"2024-06-29T14:21:59.336659+0000","last_became_peered":"2024-06-29T14:21:59.336659+0000","last_unstale":"2024-06-29T14:22:00.304956+0000","last_undegraded":"2024-06-29T14:22:00.304956+0000","last_fullsized":"2024-06-29T14:22:00.304956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:15.282310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639863+0000","last_change":"2024-06-29T14:21:59.337985+0000","last_active":"2024-06-29T14:22:00.639863+0000","last_peered":"2024-06-29T14:22:00.639863+0000","last_clean":"2024-06-29T14:22:00.639863+0000","last_became_active":"2024-06-29T14:21:59.337142+0000","last_became_peered":"2024-06-29T14:21:59.337142+0000","last_unstale":"2024-06-29T14:22:00.639863+0000","last_undegraded":"2024-06-29T14:22:00.639863+0000","last_fullsized":"2024-06-29T14:22:00.639863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:26.400192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639809+0000","last_change":"2024-06-29T14:21:55.291354+0000","last_active":"2024-06-29T14:22:00.639809+0000","last_peered":"2024-06-29T14:22:00.639809+0000","last_clean":"2024-06-29T14:22:00.639809+0000","last_became_active":"2024-06-29T14:21:55.290328+0000","last_became_peered":"2024-06-29T14:21:55.290328+0000","last_unstale":"2024-06-29T14:22:00.639809+0000","last_undegraded":"2024-06-29T14:22:00.639809+0000","last_fullsized":"2024-06-29T14:22:00.639809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639839+0000","last_change":"2024-06-29T14:21:54.254791+0000","last_active":"2024-06-29T14:22:00.639839+0000","last_peered":"2024-06-29T14:22:00.639839+0000","last_clean":"2024-06-29T14:22:00.639839+0000","last_became_active":"2024-06-29T14:21:54.254345+0000","last_became_peered":"2024-06-29T14:21:54.254345+0000","last_unstale":"2024-06-29T14:22:00.639839+0000","last_undegraded":"2024-06-29T14:22:00.639839+0000","last_fullsized":"2024-06-29T14:22:00.639839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:06.025267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305684+0000","last_change":"2024-06-29T14:21:59.317911+0000","last_active":"2024-06-29T14:22:00.305684+0000","last_peered":"2024-06-29T14:22:00.305684+0000","last_clean":"2024-06-29T14:22:00.305684+0000","last_became_active":"2024-06-29T14:21:59.317469+0000","last_became_peered":"2024-06-29T14:21:59.317469+0000","last_unstale":"2024-06-29T14:22:00.305684+0000","last_undegraded":"2024-06-29T14:22:00.305684+0000","last_fullsized":"2024-06-29T14:22:00.305684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:32.630422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305774+0000","last_change":"2024-06-29T14:21:55.297533+0000","last_active":"2024-06-29T14:22:00.305774+0000","last_peered":"2024-06-29T14:22:00.305774+0000","last_clean":"2024-06-29T14:22:00.305774+0000","last_became_active":"2024-06-29T14:21:55.296487+0000","last_became_peered":"2024-06-29T14:21:55.296487+0000","last_unstale":"2024-06-29T14:22:00.305774+0000","last_undegraded":"2024-06-29T14:22:00.305774+0000","last_fullsized":"2024-06-29T14:22:00.305774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:05.557562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305104+0000","last_change":"2024-06-29T14:21:54.253349+0000","last_active":"2024-06-29T14:22:00.305104+0000","last_peered":"2024-06-29T14:22:00.305104+0000","last_clean":"2024-06-29T14:22:00.305104+0000","last_became_active":"2024-06-29T14:21:54.252606+0000","last_became_peered":"2024-06-29T14:21:54.252606+0000","last_unstale":"2024-06-29T14:22:00.305104+0000","last_undegraded":"2024-06-29T14:22:00.305104+0000","last_fullsized":"2024-06-29T14:22:00.305104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:20.986619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306038+0000","last_change":"2024-06-29T14:21:54.254902+0000","last_active":"2024-06-29T14:22:00.306038+0000","last_peered":"2024-06-29T14:22:00.306038+0000","last_clean":"2024-06-29T14:22:00.306038+0000","last_became_active":"2024-06-29T14:21:54.254549+0000","last_became_peered":"2024-06-29T14:21:54.254549+0000","last_unstale":"2024-06-29T14:22:00.306038+0000","last_undegraded":"2024-06-29T14:22:00.306038+0000","last_fullsized":"2024-06-29T14:22:00.306038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:49.347688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640490+0000","last_change":"2024-06-29T14:21:55.297816+0000","last_active":"2024-06-29T14:22:00.640490+0000","last_peered":"2024-06-29T14:22:00.640490+0000","last_clean":"2024-06-29T14:22:00.640490+0000","last_became_active":"2024-06-29T14:21:55.297080+0000","last_became_peered":"2024-06-29T14:21:55.297080+0000","last_unstale":"2024-06-29T14:22:00.640490+0000","last_undegraded":"2024-06-29T14:22:00.640490+0000","last_fullsized":"2024-06-29T14:22:00.640490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640695+0000","last_change":"2024-06-29T14:21:59.324236+0000","last_active":"2024-06-29T14:22:00.640695+0000","last_peered":"2024-06-29T14:22:00.640695+0000","last_clean":"2024-06-29T14:22:00.640695+0000","last_became_active":"2024-06-29T14:21:59.323673+0000","last_became_peered":"2024-06-29T14:21:59.323673+0000","last_unstale":"2024-06-29T14:22:00.640695+0000","last_undegraded":"2024-06-29T14:22:00.640695+0000","last_fullsized":"2024-06-29T14:22:00.640695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:45.514616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305150+0000","last_change":"2024-06-29T14:21:54.257451+0000","last_active":"2024-06-29T14:22:00.305150+0000","last_peered":"2024-06-29T14:22:00.305150+0000","last_clean":"2024-06-29T14:22:00.305150+0000","last_became_active":"2024-06-29T14:21:54.257055+0000","last_became_peered":"2024-06-29T14:21:54.257055+0000","last_unstale":"2024-06-29T14:22:00.305150+0000","last_undegraded":"2024-06-29T14:22:00.305150+0000","last_fullsized":"2024-06-29T14:22:00.305150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:53.227131+0000","last_clean_scrub_stamp":"2024-06-29T14:21:53.227131+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:10:39.689935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307544+0000","last_change":"2024-06-29T14:21:55.291560+0000","last_active":"2024-06-29T14:22:00.307544+0000","last_peered":"2024-06-29T14:22:00.307544+0000","last_clean":"2024-06-29T14:22:00.307544+0000","last_became_active":"2024-06-29T14:21:55.291263+0000","last_became_peered":"2024-06-29T14:21:55.291263+0000","last_unstale":"2024-06-29T14:22:00.307544+0000","last_undegraded":"2024-06-29T14:22:00.307544+0000","last_fullsized":"2024-06-29T14:22:00.307544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:50.748371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305606+0000","last_change":"2024-06-29T14:21:59.319249+0000","last_active":"2024-06-29T14:22:00.305606+0000","last_peered":"2024-06-29T14:22:00.305606+0000","last_clean":"2024-06-29T14:22:00.305606+0000","last_became_active":"2024-06-29T14:21:59.316597+0000","last_became_peered":"2024-06-29T14:21:59.316597+0000","last_unstale":"2024-06-29T14:22:00.305606+0000","last_undegraded":"2024-06-29T14:22:00.305606+0000","last_fullsized":"2024-06-29T14:22:00.305606+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:58.285896+0000","last_clean_scrub_stamp":"2024-06-29T14:21:58.285896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:54.923537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305108+0000","last_change":"2024-06-29T14:21:55.291254+0000","last_active":"2024-06-29T14:22:00.305108+0000","last_peered":"2024-06-29T14:22:00.305108+0000","last_clean":"2024-06-29T14:22:00.305108+0000","last_became_active":"2024-06-29T14:21:55.290924+0000","last_became_peered":"2024-06-29T14:21:55.290924+0000","last_unstale":"2024-06-29T14:22:00.305108+0000","last_undegraded":"2024-06-29T14:22:00.305108+0000","last_fullsized":"2024-06-29T14:22:00.305108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:11.308698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640426+0000","last_change":"2024-06-29T14:21:55.280475+0000","last_active":"2024-06-29T14:22:00.640426+0000","last_peered":"2024-06-29T14:22:00.640426+0000","last_clean":"2024-06-29T14:22:00.640426+0000","last_became_active":"2024-06-29T14:21:55.277502+0000","last_became_peered":"2024-06-29T14:21:55.277502+0000","last_unstale":"2024-06-29T14:22:00.640426+0000","last_undegraded":"2024-06-29T14:22:00.640426+0000","last_fullsized":"2024-06-29T14:22:00.640426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:11.801869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640065+0000","last_change":"2024-06-29T14:21:55.285270+0000","last_active":"2024-06-29T14:22:00.640065+0000","last_peered":"2024-06-29T14:22:00.640065+0000","last_clean":"2024-06-29T14:22:00.640065+0000","last_became_active":"2024-06-29T14:21:55.284164+0000","last_became_peered":"2024-06-29T14:21:55.284164+0000","last_unstale":"2024-06-29T14:22:00.640065+0000","last_undegraded":"2024-06-29T14:22:00.640065+0000","last_fullsized":"2024-06-29T14:22:00.640065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:25.523198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640375+0000","last_change":"2024-06-29T14:21:55.287457+0000","last_active":"2024-06-29T14:22:00.640375+0000","last_peered":"2024-06-29T14:22:00.640375+0000","last_clean":"2024-06-29T14:22:00.640375+0000","last_became_active":"2024-06-29T14:21:55.287070+0000","last_became_peered":"2024-06-29T14:21:55.287070+0000","last_unstale":"2024-06-29T14:22:00.640375+0000","last_undegraded":"2024-06-29T14:22:00.640375+0000","last_fullsized":"2024-06-29T14:22:00.640375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:35:58.872433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305575+0000","last_change":"2024-06-29T14:21:55.298926+0000","last_active":"2024-06-29T14:22:00.305575+0000","last_peered":"2024-06-29T14:22:00.305575+0000","last_clean":"2024-06-29T14:22:00.305575+0000","last_became_active":"2024-06-29T14:21:55.298615+0000","last_became_peered":"2024-06-29T14:21:55.298615+0000","last_unstale":"2024-06-29T14:22:00.305575+0000","last_undegraded":"2024-06-29T14:22:00.305575+0000","last_fullsized":"2024-06-29T14:22:00.305575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641774+0000","last_change":"2024-06-29T14:21:55.290890+0000","last_active":"2024-06-29T14:22:00.641774+0000","last_peered":"2024-06-29T14:22:00.641774+0000","last_clean":"2024-06-29T14:22:00.641774+0000","last_became_active":"2024-06-29T14:21:55.290069+0000","last_became_peered":"2024-06-29T14:21:55.290069+0000","last_unstale":"2024-06-29T14:22:00.641774+0000","last_undegraded":"2024-06-29T14:22:00.641774+0000","last_fullsized":"2024-06-29T14:22:00.641774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306202+0000","last_change":"2024-06-29T14:21:55.277969+0000","last_active":"2024-06-29T14:22:00.306202+0000","last_peered":"2024-06-29T14:22:00.306202+0000","last_clean":"2024-06-29T14:22:00.306202+0000","last_became_active":"2024-06-29T14:21:55.277629+0000","last_became_peered":"2024-06-29T14:21:55.277629+0000","last_unstale":"2024-06-29T14:22:00.306202+0000","last_undegraded":"2024-06-29T14:22:00.306202+0000","last_fullsized":"2024-06-29T14:22:00.306202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:00.388193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305827+0000","last_change":"2024-06-29T14:21:55.297394+0000","last_active":"2024-06-29T14:22:00.305827+0000","last_peered":"2024-06-29T14:22:00.305827+0000","last_clean":"2024-06-29T14:22:00.305827+0000","last_became_active":"2024-06-29T14:21:55.297087+0000","last_became_peered":"2024-06-29T14:21:55.297087+0000","last_unstale":"2024-06-29T14:22:00.305827+0000","last_undegraded":"2024-06-29T14:22:00.305827+0000","last_fullsized":"2024-06-29T14:22:00.305827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:43.242914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639702+0000","last_change":"2024-06-29T14:21:55.284207+0000","last_active":"2024-06-29T14:22:00.639702+0000","last_peered":"2024-06-29T14:22:00.639702+0000","last_clean":"2024-06-29T14:22:00.639702+0000","last_became_active":"2024-06-29T14:21:55.283869+0000","last_became_peered":"2024-06-29T14:21:55.283869+0000","last_unstale":"2024-06-29T14:22:00.639702+0000","last_undegraded":"2024-06-29T14:22:00.639702+0000","last_fullsized":"2024-06-29T14:22:00.639702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304555+0000","last_change":"2024-06-29T14:21:55.298136+0000","last_active":"2024-06-29T14:22:00.304555+0000","last_peered":"2024-06-29T14:22:00.304555+0000","last_clean":"2024-06-29T14:22:00.304555+0000","last_became_active":"2024-06-29T14:21:55.297852+0000","last_became_peered":"2024-06-29T14:21:55.297852+0000","last_unstale":"2024-06-29T14:22:00.304555+0000","last_undegraded":"2024-06-29T14:22:00.304555+0000","last_fullsized":"2024-06-29T14:22:00.304555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:48.092296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640027+0000","last_change":"2024-06-29T14:21:55.291932+0000","last_active":"2024-06-29T14:22:00.640027+0000","last_peered":"2024-06-29T14:22:00.640027+0000","last_clean":"2024-06-29T14:22:00.640027+0000","last_became_active":"2024-06-29T14:21:55.291605+0000","last_became_peered":"2024-06-29T14:21:55.291605+0000","last_unstale":"2024-06-29T14:22:00.640027+0000","last_undegraded":"2024-06-29T14:22:00.640027+0000","last_fullsized":"2024-06-29T14:22:00.640027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.505918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640145+0000","last_change":"2024-06-29T14:21:55.296088+0000","last_active":"2024-06-29T14:22:00.640145+0000","last_peered":"2024-06-29T14:22:00.640145+0000","last_clean":"2024-06-29T14:22:00.640145+0000","last_became_active":"2024-06-29T14:21:55.295752+0000","last_became_peered":"2024-06-29T14:21:55.295752+0000","last_unstale":"2024-06-29T14:22:00.640145+0000","last_undegraded":"2024-06-29T14:22:00.640145+0000","last_fullsized":"2024-06-29T14:22:00.640145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306344+0000","last_change":"2024-06-29T14:21:55.293717+0000","last_active":"2024-06-29T14:22:00.306344+0000","last_peered":"2024-06-29T14:22:00.306344+0000","last_clean":"2024-06-29T14:22:00.306344+0000","last_became_active":"2024-06-29T14:21:55.293421+0000","last_became_peered":"2024-06-29T14:21:55.293421+0000","last_unstale":"2024-06-29T14:22:00.306344+0000","last_undegraded":"2024-06-29T14:22:00.306344+0000","last_fullsized":"2024-06-29T14:22:00.306344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:08:58.083131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641151+0000","last_change":"2024-06-29T14:21:55.288925+0000","last_active":"2024-06-29T14:22:00.641151+0000","last_peered":"2024-06-29T14:22:00.641151+0000","last_clean":"2024-06-29T14:22:00.641151+0000","last_became_active":"2024-06-29T14:21:55.288585+0000","last_became_peered":"2024-06-29T14:21:55.288585+0000","last_unstale":"2024-06-29T14:22:00.641151+0000","last_undegraded":"2024-06-29T14:22:00.641151+0000","last_fullsized":"2024-06-29T14:22:00.641151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:35.844291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641027+0000","last_change":"2024-06-29T14:21:55.271843+0000","last_active":"2024-06-29T14:22:00.641027+0000","last_peered":"2024-06-29T14:22:00.641027+0000","last_clean":"2024-06-29T14:22:00.641027+0000","last_became_active":"2024-06-29T14:21:55.271494+0000","last_became_peered":"2024-06-29T14:21:55.271494+0000","last_unstale":"2024-06-29T14:22:00.641027+0000","last_undegraded":"2024-06-29T14:22:00.641027+0000","last_fullsized":"2024-06-29T14:22:00.641027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:29.960047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640632+0000","last_change":"2024-06-29T14:21:55.278149+0000","last_active":"2024-06-29T14:22:00.640632+0000","last_peered":"2024-06-29T14:22:00.640632+0000","last_clean":"2024-06-29T14:22:00.640632+0000","last_became_active":"2024-06-29T14:21:55.277816+0000","last_became_peered":"2024-06-29T14:21:55.277816+0000","last_unstale":"2024-06-29T14:22:00.640632+0000","last_undegraded":"2024-06-29T14:22:00.640632+0000","last_fullsized":"2024-06-29T14:22:00.640632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:14:18.590915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639577+0000","last_change":"2024-06-29T14:21:55.296366+0000","last_active":"2024-06-29T14:22:00.639577+0000","last_peered":"2024-06-29T14:22:00.639577+0000","last_clean":"2024-06-29T14:22:00.639577+0000","last_became_active":"2024-06-29T14:21:55.296040+0000","last_became_peered":"2024-06-29T14:21:55.296040+0000","last_unstale":"2024-06-29T14:22:00.639577+0000","last_undegraded":"2024-06-29T14:22:00.639577+0000","last_fullsized":"2024-06-29T14:22:00.639577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:28.570152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639931+0000","last_change":"2024-06-29T14:21:55.291183+0000","last_active":"2024-06-29T14:22:00.639931+0000","last_peered":"2024-06-29T14:22:00.639931+0000","last_clean":"2024-06-29T14:22:00.639931+0000","last_became_active":"2024-06-29T14:21:55.289884+0000","last_became_peered":"2024-06-29T14:21:55.289884+0000","last_unstale":"2024-06-29T14:22:00.639931+0000","last_undegraded":"2024-06-29T14:22:00.639931+0000","last_fullsized":"2024-06-29T14:22:00.639931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:32.414343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640038+0000","last_change":"2024-06-29T14:21:55.293964+0000","last_active":"2024-06-29T14:22:00.640038+0000","last_peered":"2024-06-29T14:22:00.640038+0000","last_clean":"2024-06-29T14:22:00.640038+0000","last_became_active":"2024-06-29T14:21:55.291885+0000","last_became_peered":"2024-06-29T14:21:55.291885+0000","last_unstale":"2024-06-29T14:22:00.640038+0000","last_undegraded":"2024-06-29T14:22:00.640038+0000","last_fullsized":"2024-06-29T14:22:00.640038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:20.427848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304827+0000","last_change":"2024-06-29T14:21:55.288765+0000","last_active":"2024-06-29T14:22:00.304827+0000","last_peered":"2024-06-29T14:22:00.304827+0000","last_clean":"2024-06-29T14:22:00.304827+0000","last_became_active":"2024-06-29T14:21:55.288452+0000","last_became_peered":"2024-06-29T14:21:55.288452+0000","last_unstale":"2024-06-29T14:22:00.304827+0000","last_undegraded":"2024-06-29T14:22:00.304827+0000","last_fullsized":"2024-06-29T14:22:00.304827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:30.912272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.307365+0000","last_change":"2024-06-29T14:21:55.295376+0000","last_active":"2024-06-29T14:22:00.307365+0000","last_peered":"2024-06-29T14:22:00.307365+0000","last_clean":"2024-06-29T14:22:00.307365+0000","last_became_active":"2024-06-29T14:21:55.294928+0000","last_became_peered":"2024-06-29T14:21:55.294928+0000","last_unstale":"2024-06-29T14:22:00.307365+0000","last_undegraded":"2024-06-29T14:22:00.307365+0000","last_fullsized":"2024-06-29T14:22:00.307365+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:28:42.614313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.305773+0000","last_change":"2024-06-29T14:21:55.281068+0000","last_active":"2024-06-29T14:22:00.305773+0000","last_peered":"2024-06-29T14:22:00.305773+0000","last_clean":"2024-06-29T14:22:00.305773+0000","last_became_active":"2024-06-29T14:21:55.280780+0000","last_became_peered":"2024-06-29T14:21:55.280780+0000","last_unstale":"2024-06-29T14:22:00.305773+0000","last_undegraded":"2024-06-29T14:22:00.305773+0000","last_fullsized":"2024-06-29T14:22:00.305773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:36:38.990506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.306085+0000","last_change":"2024-06-29T14:21:55.280506+0000","last_active":"2024-06-29T14:22:00.306085+0000","last_peered":"2024-06-29T14:22:00.306085+0000","last_clean":"2024-06-29T14:22:00.306085+0000","last_became_active":"2024-06-29T14:21:55.280221+0000","last_became_peered":"2024-06-29T14:21:55.280221+0000","last_unstale":"2024-06-29T14:22:00.306085+0000","last_undegraded":"2024-06-29T14:22:00.306085+0000","last_fullsized":"2024-06-29T14:22:00.306085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:41.028472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641311+0000","last_change":"2024-06-29T14:21:55.287649+0000","last_active":"2024-06-29T14:22:00.641311+0000","last_peered":"2024-06-29T14:22:00.641311+0000","last_clean":"2024-06-29T14:22:00.641311+0000","last_became_active":"2024-06-29T14:21:55.286171+0000","last_became_peered":"2024-06-29T14:21:55.286171+0000","last_unstale":"2024-06-29T14:22:00.641311+0000","last_undegraded":"2024-06-29T14:22:00.641311+0000","last_fullsized":"2024-06-29T14:22:00.641311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:30.219397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.304571+0000","last_change":"2024-06-29T14:21:55.285603+0000","last_active":"2024-06-29T14:22:00.304571+0000","last_peered":"2024-06-29T14:22:00.304571+0000","last_clean":"2024-06-29T14:22:00.304571+0000","last_became_active":"2024-06-29T14:21:55.285284+0000","last_became_peered":"2024-06-29T14:21:55.285284+0000","last_unstale":"2024-06-29T14:22:00.304571+0000","last_undegraded":"2024-06-29T14:22:00.304571+0000","last_fullsized":"2024-06-29T14:22:00.304571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:44.658689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640973+0000","last_change":"2024-06-29T14:21:55.287794+0000","last_active":"2024-06-29T14:22:00.640973+0000","last_peered":"2024-06-29T14:22:00.640973+0000","last_clean":"2024-06-29T14:22:00.640973+0000","last_became_active":"2024-06-29T14:21:55.287028+0000","last_became_peered":"2024-06-29T14:21:55.287028+0000","last_unstale":"2024-06-29T14:22:00.640973+0000","last_undegraded":"2024-06-29T14:22:00.640973+0000","last_fullsized":"2024-06-29T14:22:00.640973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:39.026791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.639887+0000","last_change":"2024-06-29T14:21:55.289581+0000","last_active":"2024-06-29T14:22:00.639887+0000","last_peered":"2024-06-29T14:22:00.639887+0000","last_clean":"2024-06-29T14:22:00.639887+0000","last_became_active":"2024-06-29T14:21:55.289097+0000","last_became_peered":"2024-06-29T14:21:55.289097+0000","last_unstale":"2024-06-29T14:22:00.639887+0000","last_undegraded":"2024-06-29T14:22:00.639887+0000","last_fullsized":"2024-06-29T14:22:00.639887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:43.242914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640505+0000","last_change":"2024-06-29T14:21:55.288463+0000","last_active":"2024-06-29T14:22:00.640505+0000","last_peered":"2024-06-29T14:22:00.640505+0000","last_clean":"2024-06-29T14:22:00.640505+0000","last_became_active":"2024-06-29T14:21:55.288128+0000","last_became_peered":"2024-06-29T14:21:55.288128+0000","last_unstale":"2024-06-29T14:22:00.640505+0000","last_undegraded":"2024-06-29T14:22:00.640505+0000","last_fullsized":"2024-06-29T14:22:00.640505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:28.573747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.640255+0000","last_change":"2024-06-29T14:21:55.290100+0000","last_active":"2024-06-29T14:22:00.640255+0000","last_peered":"2024-06-29T14:22:00.640255+0000","last_clean":"2024-06-29T14:22:00.640255+0000","last_became_active":"2024-06-29T14:21:55.289525+0000","last_became_peered":"2024-06-29T14:21:55.289525+0000","last_unstale":"2024-06-29T14:22:00.640255+0000","last_undegraded":"2024-06-29T14:22:00.640255+0000","last_fullsized":"2024-06-29T14:22:00.640255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:22.343517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641269+0000","last_change":"2024-06-29T14:21:55.287921+0000","last_active":"2024-06-29T14:22:00.641269+0000","last_peered":"2024-06-29T14:22:00.641269+0000","last_clean":"2024-06-29T14:22:00.641269+0000","last_became_active":"2024-06-29T14:21:55.287383+0000","last_became_peered":"2024-06-29T14:21:55.287383+0000","last_unstale":"2024-06-29T14:22:00.641269+0000","last_undegraded":"2024-06-29T14:22:00.641269+0000","last_fullsized":"2024-06-29T14:22:00.641269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:05.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641908+0000","last_change":"2024-06-29T14:21:55.285210+0000","last_active":"2024-06-29T14:22:00.641908+0000","last_peered":"2024-06-29T14:22:00.641908+0000","last_clean":"2024-06-29T14:22:00.641908+0000","last_became_active":"2024-06-29T14:21:55.284857+0000","last_became_peered":"2024-06-29T14:21:55.284857+0000","last_unstale":"2024-06-29T14:22:00.641908+0000","last_undegraded":"2024-06-29T14:22:00.641908+0000","last_fullsized":"2024-06-29T14:22:00.641908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:02.942194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-29T14:22:00.641229+0000","last_change":"2024-06-29T14:21:55.287889+0000","last_active":"2024-06-29T14:22:00.641229+0000","last_peered":"2024-06-29T14:22:00.641229+0000","last_clean":"2024-06-29T14:22:00.641229+0000","last_became_active":"2024-06-29T14:21:55.286286+0000","last_became_peered":"2024-06-29T14:21:55.286286+0000","last_unstale":"2024-06-29T14:22:00.641229+0000","last_undegraded":"2024-06-29T14:22:00.641229+0000","last_fullsized":"2024-06-29T14:22:00.641229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:54.229222+0000","last_clean_scrub_stamp":"2024-06-29T14:21:54.229222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:23.065793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509451,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":111290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509452,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":876544,"data_stored":698454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":868352,"data_stored":698398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":111290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":107574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:22:19.073 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-29T14:22:19.073 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-29T14:22:19.073 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-06-29T14:22:19.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-06-29T14:22:19.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.228+0000 7fefda6ce640 1 -- 172.21.15.59:0/2606509810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 msgr2=0x7fefd413edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.228+0000 7fefda6ce640 1 --2- 172.21.15.59:0/2606509810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 0x7fefd413edc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fefc8009830 tx=0x7fefc802a530 comp rx=0 tx=0).stop 2024-06-29T14:22:19.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.228+0000 7fefda6ce640 1 -- 172.21.15.59:0/2606509810 shutdown_connections 2024-06-29T14:22:19.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.228+0000 7fefda6ce640 1 --2- 172.21.15.59:0/2606509810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 0x7fefd413edc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.228+0000 7fefda6ce640 1 -- 172.21.15.59:0/2606509810 >> 172.21.15.59:0/2606509810 conn(0x7fefd40b6fb0 msgr2=0x7fefd40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.228+0000 7fefda6ce640 1 -- 172.21.15.59:0/2606509810 shutdown_connections 2024-06-29T14:22:19.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.228+0000 7fefda6ce640 1 -- 172.21.15.59:0/2606509810 wait complete. 2024-06-29T14:22:19.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.229+0000 7fefda6ce640 1 Processor -- start 2024-06-29T14:22:19.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.230+0000 7fefda6ce640 1 -- start start 2024-06-29T14:22:19.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefda6ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 0x7fefd406f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefda6ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefd41414c0 0x7fefd415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefda6ce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefd415ceb0 0x7fefd4130eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefda6ce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fefd40b2900 con 0x7fefd413e9b0 2024-06-29T14:22:19.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefda6ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fefd40b2780 con 0x7fefd41414c0 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefda6ce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fefd40b2a80 con 0x7fefd415ceb0 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 0x7fefd406f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 0x7fefd406f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52036/0 (socket says 172.21.15.59:52036) 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd3fff640 1 -- 172.21.15.59:0/2656742840 learned_addr learned my addr 172.21.15.59:0/2656742840 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd37fe640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefd41414c0 0x7fefd415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd8c6b640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefd415ceb0 0x7fefd4130eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd3fff640 1 -- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefd415ceb0 msgr2=0x7fefd4130eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd3fff640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefd415ceb0 0x7fefd4130eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd3fff640 1 -- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefd41414c0 msgr2=0x7fefd415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd3fff640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefd41414c0 0x7fefd415b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.231+0000 7fefd3fff640 1 -- 172.21.15.59:0/2656742840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd415bcc0 con 0x7fefd413e9b0 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.232+0000 7fefd3fff640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 0x7fefd406f170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fefc400c400 tx=0x7fefc400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.232+0000 7fefd0ff9640 1 -- 172.21.15.59:0/2656742840 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc4017070 con 0x7fefd413e9b0 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.232+0000 7fefd0ff9640 1 -- 172.21.15.59:0/2656742840 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefc40053c0 con 0x7fefd413e9b0 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.232+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd406e0d0 con 0x7fefd413e9b0 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.232+0000 7fefd0ff9640 1 -- 172.21.15.59:0/2656742840 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc4003850 con 0x7fefd413e9b0 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.232+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd406e690 con 0x7fefd413e9b0 2024-06-29T14:22:19.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.233+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd40b2f30 con 0x7fefd413e9b0 2024-06-29T14:22:19.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.234+0000 7fefd0ff9640 1 -- 172.21.15.59:0/2656742840 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fefc401a070 con 0x7fefd413e9b0 2024-06-29T14:22:19.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.234+0000 7fefd0ff9640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fefc00762d0 0x7fefc00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.234+0000 7fefd0ff9640 1 -- 172.21.15.59:0/2656742840 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fefc4096f20 con 0x7fefd413e9b0 2024-06-29T14:22:19.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.235+0000 7fefd37fe640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fefc00762d0 0x7fefc00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.235+0000 7fefd37fe640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fefc00762d0 0x7fefc00787e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fefc8034000 tx=0x7fefc8004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.239+0000 7fefd0ff9640 1 -- 172.21.15.59:0/2656742840 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fefc4060540 con 0x7fefd413e9b0 2024-06-29T14:22:19.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.471+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fefd40714b0 con 0x7fefd413e9b0 2024-06-29T14:22:19.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.471+0000 7fefd0ff9640 1 -- 172.21.15.59:0/2656742840 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fefc405fe40 con 0x7fefd413e9b0 2024-06-29T14:22:19.470 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:19.470 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-29T14:22:19.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fefc00762d0 msgr2=0x7fefc00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fefc00762d0 0x7fefc00787e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fefc8034000 tx=0x7fefc8004730 comp rx=0 tx=0).stop 2024-06-29T14:22:19.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 msgr2=0x7fefd406f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 0x7fefd406f170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fefc400c400 tx=0x7fefc400c8c0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 shutdown_connections 2024-06-29T14:22:19.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fefc00762d0 0x7fefc00787e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefd415ceb0 0x7fefd4130eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefd41414c0 0x7fefd415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 --2- 172.21.15.59:0/2656742840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefd413e9b0 0x7fefd406f170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 >> 172.21.15.59:0/2656742840 conn(0x7fefd40b6fb0 msgr2=0x7fefd40b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 shutdown_connections 2024-06-29T14:22:19.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.474+0000 7fefda6ce640 1 -- 172.21.15.59:0/2656742840 wait complete. 2024-06-29T14:22:19.483 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-06-29T14:22:19.483 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.603+0000 7f279247f640 1 -- 172.21.15.59:0/1267759446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 msgr2=0x7f2774003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.603+0000 7f279247f640 1 --2- 172.21.15.59:0/1267759446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 0x7f2774003ee0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c2e0 tx=0x7f2774005830 comp rx=0 tx=0).stop 2024-06-29T14:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.603+0000 7f2790a1c640 1 -- 172.21.15.59:0/1267759446 reap_dead start 2024-06-29T14:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.603+0000 7f279247f640 1 -- 172.21.15.59:0/1267759446 shutdown_connections 2024-06-29T14:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.603+0000 7f279247f640 1 -- 172.21.15.59:0/1267759446 >> 172.21.15.59:0/1267759446 conn(0x7f278c073e10 msgr2=0x7f278c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.603+0000 7f279247f640 1 -- 172.21.15.59:0/1267759446 shutdown_connections 2024-06-29T14:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.603+0000 7f279247f640 1 -- 172.21.15.59:0/1267759446 wait complete. 2024-06-29T14:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.605+0000 7f279247f640 1 Processor -- start 2024-06-29T14:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.605+0000 7f279247f640 1 -- start start 2024-06-29T14:22:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f279247f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 0x7f278c15d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f279247f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f278c15d990 0x7f278c0a9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f279247f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f278c0a9cc0 0x7f278c0aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f279247f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f278c0d18b0 con 0x7f2774003a70 2024-06-29T14:22:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f279247f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f278c0d1730 con 0x7f278c0a9cc0 2024-06-29T14:22:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f279247f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f278c15fa40 con 0x7f278c15d990 2024-06-29T14:22:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f278bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 0x7f278c15d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f278bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 0x7f278c15d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52046/0 (socket says 172.21.15.59:52046) 2024-06-29T14:22:19.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f278bfff640 1 -- 172.21.15.59:0/2323904408 learned_addr learned my addr 172.21.15.59:0/2323904408 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:19.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f2790a1c640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f278c0a9cc0 0x7f278c0aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.606+0000 7f278b7fe640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f278c15d990 0x7f278c0a9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f278bfff640 1 -- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f278c15d990 msgr2=0x7f278c0a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f278bfff640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f278c15d990 0x7f278c0a9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f278bfff640 1 -- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f278c0a9cc0 msgr2=0x7f278c0aa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f278bfff640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f278c0a9cc0 0x7f278c0aa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f278bfff640 1 -- 172.21.15.59:0/2323904408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278c0aa610 con 0x7f2774003a70 2024-06-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f278bfff640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 0x7f278c15d450 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c400 tx=0x7f277c006ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f2788ff9640 1 -- 172.21.15.59:0/2323904408 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c031040 con 0x7f2774003a70 2024-06-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f2788ff9640 1 -- 172.21.15.59:0/2323904408 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f277c003610 con 0x7f2774003a70 2024-06-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278c0ac090 con 0x7f2774003a70 2024-06-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f2788ff9640 1 -- 172.21.15.59:0/2323904408 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c003a40 con 0x7f2774003a70 2024-06-29T14:22:19.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.607+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f278c15e460 con 0x7f2774003a70 2024-06-29T14:22:19.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.608+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278c0b5b30 con 0x7f2774003a70 2024-06-29T14:22:19.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.609+0000 7f2788ff9640 1 -- 172.21.15.59:0/2323904408 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f277c034070 con 0x7f2774003a70 2024-06-29T14:22:19.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.609+0000 7f2788ff9640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f27780762d0 0x7f27780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.610+0000 7f2788ff9640 1 -- 172.21.15.59:0/2323904408 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f277c0b7e10 con 0x7f2774003a70 2024-06-29T14:22:19.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.610+0000 7f278b7fe640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f27780762d0 0x7f27780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.610+0000 7f278b7fe640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f27780762d0 0x7f27780787e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f278c15f870 tx=0x7f2780005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:19.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.614+0000 7f2788ff9640 1 -- 172.21.15.59:0/2323904408 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f277c0bb030 con 0x7f2774003a70 2024-06-29T14:22:19.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.806+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f278c158dd0 con 0x7f2774003a70 2024-06-29T14:22:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.807+0000 7f2788ff9640 1 -- 172.21.15.59:0/2323904408 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7f277c081410 con 0x7f2774003a70 2024-06-29T14:22:19.806 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:19.806 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.809+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f27780762d0 msgr2=0x7f27780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.809+0000 7f279247f640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f27780762d0 0x7f27780787e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f278c15f870 tx=0x7f2780005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.809+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 msgr2=0x7f278c15d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.809+0000 7f279247f640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 0x7f278c15d450 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c400 tx=0x7f277c006ce0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.810+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 shutdown_connections 2024-06-29T14:22:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.810+0000 7f279247f640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f27780762d0 0x7f27780787e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.810+0000 7f279247f640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f278c0a9cc0 0x7f278c0aa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.810+0000 7f279247f640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f278c15d990 0x7f278c0a9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.810+0000 7f279247f640 1 --2- 172.21.15.59:0/2323904408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2774003a70 0x7f278c15d450 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.810+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 >> 172.21.15.59:0/2323904408 conn(0x7f278c073e10 msgr2=0x7f278c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.810+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 shutdown_connections 2024-06-29T14:22:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.810+0000 7f279247f640 1 -- 172.21.15.59:0/2323904408 wait complete. 2024-06-29T14:22:19.819 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:19.819 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:19.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.942+0000 7f15b1be7640 1 -- 172.21.15.59:0/2000355089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0b4ee0 msgr2=0x7f15ac06e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.942+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2000355089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0b4ee0 0x7f15ac06e4d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0753d0 tx=0x7f15a002a530 comp rx=0 tx=0).stop 2024-06-29T14:22:19.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.942+0000 7f15b1be7640 1 -- 172.21.15.59:0/2000355089 shutdown_connections 2024-06-29T14:22:19.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.942+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2000355089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0b4ee0 0x7f15ac06e4d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.942+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2000355089 >> v2:172.21.15.153:3300/0 conn(0x7f15ac0d0130 0x7f15ac143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.942+0000 7f15b1be7640 1 -- 172.21.15.59:0/2000355089 >> 172.21.15.59:0/2000355089 conn(0x7f15ac0b2ab0 msgr2=0x7f15ac0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.942+0000 7f15b1be7640 1 -- 172.21.15.59:0/2000355089 shutdown_connections 2024-06-29T14:22:19.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.942+0000 7f15b1be7640 1 -- 172.21.15.59:0/2000355089 wait complete. 2024-06-29T14:22:19.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.944+0000 7f15b1be7640 1 Processor -- start 2024-06-29T14:22:19.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.944+0000 7f15b1be7640 1 -- start start 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.945+0000 7f15b1be7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ac06ea10 0x7f15ac0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.945+0000 7f15b1be7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ac0b4ee0 0x7f15ac13d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15b1be7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0d0130 0x7f15ac0ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15b1be7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac15ed80 con 0x7f15ac0d0130 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15b1be7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac06f030 con 0x7f15ac06ea10 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15b1be7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f15ac15ef00 con 0x7f15ac0b4ee0 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0d0130 0x7f15ac0ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0d0130 0x7f15ac0ac590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52054/0 (socket says 172.21.15.59:52054) 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 -- 172.21.15.59:0/2087925649 learned_addr learned my addr 172.21.15.59:0/2087925649 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:19.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15aaffd640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ac0b4ee0 0x7f15ac13d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15ab7fe640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ac06ea10 0x7f15ac0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 -- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ac0b4ee0 msgr2=0x7f15ac13d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ac0b4ee0 0x7f15ac13d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 -- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ac06ea10 msgr2=0x7f15ac0af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ac06ea10 0x7f15ac0af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 -- 172.21.15.59:0/2087925649 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac0ab4f0 con 0x7f15ac0d0130 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15abfff640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0d0130 0x7f15ac0ac590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f15a400c9f0 tx=0x7f15a400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f1593fff640 1 -- 172.21.15.59:0/2087925649 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4016020 con 0x7f15ac0d0130 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f1593fff640 1 -- 172.21.15.59:0/2087925649 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15a400ac60 con 0x7f15ac0d0130 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15ac0ab7d0 con 0x7f15ac0d0130 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f1593fff640 1 -- 172.21.15.59:0/2087925649 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4012680 con 0x7f15ac0d0130 2024-06-29T14:22:19.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.946+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15ac0abd90 con 0x7f15ac0d0130 2024-06-29T14:22:19.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.947+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15ac0ab260 con 0x7f15ac0d0130 2024-06-29T14:22:19.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.948+0000 7f1593fff640 1 -- 172.21.15.59:0/2087925649 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f15a4019020 con 0x7f15ac0d0130 2024-06-29T14:22:19.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.949+0000 7f1593fff640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1598076300 0x7f1598078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.949+0000 7f1593fff640 1 -- 172.21.15.59:0/2087925649 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f15a40981d0 con 0x7f15ac0d0130 2024-06-29T14:22:19.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.949+0000 7f15ab7fe640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1598076300 0x7f1598078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.949+0000 7f15ab7fe640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1598076300 0x7f1598078810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f159c009ac0 tx=0x7f159c005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:19.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:19.953+0000 7f1593fff640 1 -- 172.21.15.59:0/2087925649 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f15a4020020 con 0x7f15ac0d0130 2024-06-29T14:22:20.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.142+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15ac0afb30 con 0x7f15ac0d0130 2024-06-29T14:22:20.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.142+0000 7f1593fff640 1 -- 172.21.15.59:0/2087925649 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f15a40616c0 con 0x7f15ac0d0130 2024-06-29T14:22:20.140 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:20.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1598076300 msgr2=0x7f1598078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1598076300 0x7f1598078810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f159c009ac0 tx=0x7f159c005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0d0130 msgr2=0x7f15ac0ac590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0d0130 0x7f15ac0ac590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f15a400c9f0 tx=0x7f15a400ceb0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 shutdown_connections 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1598076300 0x7f1598078810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15ac0d0130 0x7f15ac0ac590 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15ac0b4ee0 0x7f15ac13d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 --2- 172.21.15.59:0/2087925649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15ac06ea10 0x7f15ac0af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 >> 172.21.15.59:0/2087925649 conn(0x7f15ac0b2ab0 msgr2=0x7f15ac140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 shutdown_connections 2024-06-29T14:22:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.144+0000 7f15b1be7640 1 -- 172.21.15.59:0/2087925649 wait complete. 2024-06-29T14:22:20.154 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:20.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.276+0000 7f983a723640 1 -- 172.21.15.59:0/1770376562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340b40a0 msgr2=0x7f983415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.276+0000 7f983a723640 1 --2- 172.21.15.59:0/1770376562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340b40a0 0x7f983415e8d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9828005fd0 tx=0x7f982802a670 comp rx=0 tx=0).stop 2024-06-29T14:22:20.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.276+0000 7f983a723640 1 -- 172.21.15.59:0/1770376562 shutdown_connections 2024-06-29T14:22:20.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.276+0000 7f983a723640 1 --2- 172.21.15.59:0/1770376562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340b40a0 0x7f983415e8d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.276+0000 7f983a723640 1 -- 172.21.15.59:0/1770376562 >> 172.21.15.59:0/1770376562 conn(0x7f98340d3ee0 msgr2=0x7f98340d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.276+0000 7f983a723640 1 -- 172.21.15.59:0/1770376562 shutdown_connections 2024-06-29T14:22:20.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.276+0000 7f983a723640 1 -- 172.21.15.59:0/1770376562 wait complete. 2024-06-29T14:22:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.278+0000 7f983a723640 1 Processor -- start 2024-06-29T14:22:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.278+0000 7f983a723640 1 -- start start 2024-06-29T14:22:20.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.279+0000 7f983a723640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340d23c0 0x7f983414d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.279+0000 7f983a723640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f983415ee10 0x7f9834156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f983a723640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9834154170 0x7f98341573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f983a723640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9834160450 con 0x7f98340d23c0 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f983a723640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98341602d0 con 0x7f9834154170 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f983a723640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f98341605d0 con 0x7f983415ee10 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340d23c0 0x7f983414d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340d23c0 0x7f983414d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52076/0 (socket says 172.21.15.59:52076) 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 -- 172.21.15.59:0/2726722807 learned_addr learned my addr 172.21.15.59:0/2726722807 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f98337fe640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f983415ee10 0x7f9834156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9838cc0640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9834154170 0x7f98341573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 -- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f983415ee10 msgr2=0x7f9834156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f983415ee10 0x7f9834156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 -- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9834154170 msgr2=0x7f98341573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9834154170 0x7f98341573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 -- 172.21.15.59:0/2726722807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98341533d0 con 0x7f98340d23c0 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f98337fe640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f983415ee10 0x7f9834156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9838cc0640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9834154170 0x7f98341573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9833fff640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340d23c0 0x7f983414d800 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f982400c370 tx=0x7f982400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9830ff9640 1 -- 172.21.15.59:0/2726722807 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824017070 con 0x7f98340d23c0 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9830ff9640 1 -- 172.21.15.59:0/2726722807 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98240035f0 con 0x7f98340d23c0 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98340d0f30 con 0x7f98340d23c0 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f9830ff9640 1 -- 172.21.15.59:0/2726722807 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824003a40 con 0x7f98340d23c0 2024-06-29T14:22:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.280+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9834153970 con 0x7f98340d23c0 2024-06-29T14:22:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.282+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f983415e8d0 con 0x7f98340d23c0 2024-06-29T14:22:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.282+0000 7f9830ff9640 1 -- 172.21.15.59:0/2726722807 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f982401a070 con 0x7f98340d23c0 2024-06-29T14:22:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.283+0000 7f9830ff9640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f98200762d0 0x7f98200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.283+0000 7f9830ff9640 1 -- 172.21.15.59:0/2726722807 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f9824096e50 con 0x7f98340d23c0 2024-06-29T14:22:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.283+0000 7f98337fe640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f98200762d0 0x7f98200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.284+0000 7f98337fe640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f98200762d0 0x7f98200787e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9828033000 tx=0x7f9828004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.288+0000 7f9830ff9640 1 -- 172.21.15.59:0/2726722807 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f98240603e0 con 0x7f98340d23c0 2024-06-29T14:22:20.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.478+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f98340d2950 con 0x7f98340d23c0 2024-06-29T14:22:20.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.478+0000 7f9830ff9640 1 -- 172.21.15.59:0/2726722807 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f982405fce0 con 0x7f98340d23c0 2024-06-29T14:22:20.477 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f98200762d0 msgr2=0x7f98200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f98200762d0 0x7f98200787e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9828033000 tx=0x7f9828004730 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340d23c0 msgr2=0x7f983414d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340d23c0 0x7f983414d800 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f982400c370 tx=0x7f982400c830 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 shutdown_connections 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f98200762d0 0x7f98200787e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9834154170 0x7f98341573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f983415ee10 0x7f9834156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 --2- 172.21.15.59:0/2726722807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98340d23c0 0x7f983414d800 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.480+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 >> 172.21.15.59:0/2726722807 conn(0x7f98340d3ee0 msgr2=0x7f98340b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.481+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 shutdown_connections 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.481+0000 7f983a723640 1 -- 172.21.15.59:0/2726722807 wait complete. 2024-06-29T14:22:20.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.613+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/1890194807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d4004800 msgr2=0x7ff3d4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.613+0000 7ff3ed9d2640 1 --2- 172.21.15.59:0/1890194807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d4004800 0x7ff3d4004c10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff3e808c6c0 tx=0x7ff3d4005830 comp rx=0 tx=0).stop 2024-06-29T14:22:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.613+0000 7ff3e77fe640 1 -- 172.21.15.59:0/1890194807 reap_dead start 2024-06-29T14:22:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.613+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/1890194807 shutdown_connections 2024-06-29T14:22:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.613+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/1890194807 >> 172.21.15.59:0/1890194807 conn(0x7ff3e80d45b0 msgr2=0x7ff3e80d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.613+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/1890194807 shutdown_connections 2024-06-29T14:22:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.613+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/1890194807 wait complete. 2024-06-29T14:22:20.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.615+0000 7ff3ed9d2640 1 Processor -- start 2024-06-29T14:22:20.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.615+0000 7ff3ed9d2640 1 -- start start 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3ed9d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3d4004800 0x7ff3e81f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3ed9d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e80c8cf0 0x7ff3e80c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3ed9d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3e8160eb0 0x7ff3e81612c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3ed9d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e80ce6d0 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3ed9d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e80ce550 con 0x7ff3d4004800 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3ed9d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e80ce850 con 0x7ff3e8160eb0 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e80c8cf0 0x7ff3e80c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e80c8cf0 0x7ff3e80c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52086/0 (socket says 172.21.15.59:52086) 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 -- 172.21.15.59:0/3481031734 learned_addr learned my addr 172.21.15.59:0/3481031734 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:20.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3e77fe640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3e8160eb0 0x7ff3e81612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3e6ffd640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3d4004800 0x7ff3e81f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 -- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3e8160eb0 msgr2=0x7ff3e81612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3e8160eb0 0x7ff3e81612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 -- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3d4004800 msgr2=0x7ff3e81f5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3d4004800 0x7ff3e81f5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 -- 172.21.15.59:0/3481031734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e8161800 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3e77fe640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3e8160eb0 0x7ff3e81612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3de7fc640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e80c8cf0 0x7ff3e80c9100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0036040 tx=0x7ff3d0033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3e4ff9640 1 -- 172.21.15.59:0/3481031734 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0038040 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3e4ff9640 1 -- 172.21.15.59:0/3481031734 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3d0041070 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.617+0000 7ff3e4ff9640 1 -- 172.21.15.59:0/3481031734 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0033db0 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.618+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e80c9940 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.618+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e80c76b0 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.618+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e80b38e0 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.619+0000 7ff3e4ff9640 1 -- 172.21.15.59:0/3481031734 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff3d002b820 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.620+0000 7ff3e4ff9640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff3c80762d0 0x7ff3c80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.620+0000 7ff3e4ff9640 1 -- 172.21.15.59:0/3481031734 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff3d00b8570 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.620+0000 7ff3e6ffd640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff3c80762d0 0x7ff3c80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.621+0000 7ff3e6ffd640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff3c80762d0 0x7ff3c80787e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8006460 tx=0x7ff3d8006b10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.624+0000 7ff3e4ff9640 1 -- 172.21.15.59:0/3481031734 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff3e8161800 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.815+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff3e80b8c40 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.816+0000 7ff3e4ff9640 1 -- 172.21.15.59:0/3481031734 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7ff3d0081b80 con 0x7ff3e80c8cf0 2024-06-29T14:22:20.814 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff3c80762d0 msgr2=0x7ff3c80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff3c80762d0 0x7ff3c80787e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8006460 tx=0x7ff3d8006b10 comp rx=0 tx=0).stop 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e80c8cf0 msgr2=0x7ff3e80c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e80c8cf0 0x7ff3e80c9100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0036040 tx=0x7ff3d0033670 comp rx=0 tx=0).stop 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 shutdown_connections 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff3c80762d0 0x7ff3c80787e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3e8160eb0 0x7ff3e81612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e80c8cf0 0x7ff3e80c9100 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 --2- 172.21.15.59:0/3481031734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3d4004800 0x7ff3e81f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.817+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 >> 172.21.15.59:0/3481031734 conn(0x7ff3e80d45b0 msgr2=0x7ff3e80b9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.818+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 shutdown_connections 2024-06-29T14:22:20.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.818+0000 7ff3ed9d2640 1 -- 172.21.15.59:0/3481031734 wait complete. 2024-06-29T14:22:20.827 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.950+0000 7f728181f640 1 -- 172.21.15.59:0/2108189527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c0d23c0 msgr2=0x7f727c0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.950+0000 7f728181f640 1 --2- 172.21.15.59:0/2108189527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c0d23c0 0x7f727c0b3990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f726c005fd0 tx=0x7f726c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.950+0000 7f728181f640 1 -- 172.21.15.59:0/2108189527 shutdown_connections 2024-06-29T14:22:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.950+0000 7f728181f640 1 --2- 172.21.15.59:0/2108189527 >> v2:172.21.15.153:3300/0 conn(0x7f727c0b3ed0 0x7f727c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.950+0000 7f728181f640 1 --2- 172.21.15.59:0/2108189527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c0d23c0 0x7f727c0b3990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.950+0000 7f728181f640 1 -- 172.21.15.59:0/2108189527 >> 172.21.15.59:0/2108189527 conn(0x7f727c0d3ee0 msgr2=0x7f727c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.950+0000 7f728181f640 1 -- 172.21.15.59:0/2108189527 shutdown_connections 2024-06-29T14:22:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.950+0000 7f728181f640 1 -- 172.21.15.59:0/2108189527 wait complete. 2024-06-29T14:22:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.952+0000 7f728181f640 1 Processor -- start 2024-06-29T14:22:20.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.952+0000 7f728181f640 1 -- start start 2024-06-29T14:22:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f728181f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f727c0b3ed0 0x7f727c1e9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f728181f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f727c0d23c0 0x7f727c1e9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f728181f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c15ee10 0x7f727c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f728181f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f727c160240 con 0x7f727c0d23c0 2024-06-29T14:22:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f728181f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f727c1600c0 con 0x7f727c0b3ed0 2024-06-29T14:22:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f728181f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f727c1603c0 con 0x7f727c15ee10 2024-06-29T14:22:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f727a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f727c0d23c0 0x7f727c1e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f727a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f727c0d23c0 0x7f727c1e9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52092/0 (socket says 172.21.15.59:52092) 2024-06-29T14:22:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f727a7fc640 1 -- 172.21.15.59:0/1380308051 learned_addr learned my addr 172.21.15.59:0/1380308051 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f727affd640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f727c0b3ed0 0x7f727c1e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.953+0000 7f727b7fe640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c15ee10 0x7f727c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f727a7fc640 1 -- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c15ee10 msgr2=0x7f727c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f727a7fc640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c15ee10 0x7f727c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f727a7fc640 1 -- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f727c0b3ed0 msgr2=0x7f727c1e9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f727a7fc640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f727c0b3ed0 0x7f727c1e9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f727a7fc640 1 -- 172.21.15.59:0/1380308051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727c1e9fd0 con 0x7f727c0d23c0 2024-06-29T14:22:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f727affd640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f727c0b3ed0 0x7f727c1e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f727b7fe640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c15ee10 0x7f727c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:20.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f727a7fc640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f727c0d23c0 0x7f727c1e9790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f727000e9f0 tx=0x7f727000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f7263fff640 1 -- 172.21.15.59:0/1380308051 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727000cd80 con 0x7f727c0d23c0 2024-06-29T14:22:20.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f7263fff640 1 -- 172.21.15.59:0/1380308051 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7270004020 con 0x7f727c0d23c0 2024-06-29T14:22:20.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f727c1ea2b0 con 0x7f727c0d23c0 2024-06-29T14:22:20.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f7263fff640 1 -- 172.21.15.59:0/1380308051 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270004470 con 0x7f727c0d23c0 2024-06-29T14:22:20.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.954+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f727c1ea870 con 0x7f727c0d23c0 2024-06-29T14:22:20.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.955+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727c1eacb0 con 0x7f727c0d23c0 2024-06-29T14:22:20.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.956+0000 7f7263fff640 1 -- 172.21.15.59:0/1380308051 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f72700035f0 con 0x7f727c0d23c0 2024-06-29T14:22:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.957+0000 7f7263fff640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7268076300 0x7f7268078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.957+0000 7f7263fff640 1 -- 172.21.15.59:0/1380308051 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f7270098200 con 0x7f727c0d23c0 2024-06-29T14:22:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.957+0000 7f727affd640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7268076300 0x7f7268078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.958+0000 7f727affd640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7268076300 0x7f7268078810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f726c033000 tx=0x7f726c004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:20.961+0000 7f7263fff640 1 -- 172.21.15.59:0/1380308051 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7270061260 con 0x7f727c0d23c0 2024-06-29T14:22:21.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.151+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f727c1e9d60 con 0x7f727c0d23c0 2024-06-29T14:22:21.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.152+0000 7f7263fff640 1 -- 172.21.15.59:0/1380308051 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f7270065560 con 0x7f727c0d23c0 2024-06-29T14:22:21.150 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7268076300 msgr2=0x7f7268078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7268076300 0x7f7268078810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f726c033000 tx=0x7f726c004730 comp rx=0 tx=0).stop 2024-06-29T14:22:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f727c0d23c0 msgr2=0x7f727c1e9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f727c0d23c0 0x7f727c1e9790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f727000e9f0 tx=0x7f727000eeb0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 shutdown_connections 2024-06-29T14:22:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7268076300 0x7f7268078810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f727c15ee10 0x7f727c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f727c0d23c0 0x7f727c1e9790 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 --2- 172.21.15.59:0/1380308051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f727c0b3ed0 0x7f727c1e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 >> 172.21.15.59:0/1380308051 conn(0x7f727c0d3ee0 msgr2=0x7f727c06d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 shutdown_connections 2024-06-29T14:22:21.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.154+0000 7f728181f640 1 -- 172.21.15.59:0/1380308051 wait complete. 2024-06-29T14:22:21.163 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:21.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.282+0000 7f72e81eb640 1 -- 172.21.15.59:0/641473897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e015ee10 msgr2=0x7f72e015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.282+0000 7f72e81eb640 1 --2- 172.21.15.59:0/641473897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e015ee10 0x7f72e015f9b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f72e008c4f0 tx=0x7f72d802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.282+0000 7f72e81eb640 1 -- 172.21.15.59:0/641473897 shutdown_connections 2024-06-29T14:22:21.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.282+0000 7f72e81eb640 1 --2- 172.21.15.59:0/641473897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e015ee10 0x7f72e015f9b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.283+0000 7f72e81eb640 1 --2- 172.21.15.59:0/641473897 >> v2:172.21.15.153:3300/0 conn(0x7f72e00b40a0 0x7f72e015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.283+0000 7f72e81eb640 1 -- 172.21.15.59:0/641473897 >> 172.21.15.59:0/641473897 conn(0x7f72e00d3ee0 msgr2=0x7f72e00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.283+0000 7f72e81eb640 1 -- 172.21.15.59:0/641473897 shutdown_connections 2024-06-29T14:22:21.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.283+0000 7f72e81eb640 1 -- 172.21.15.59:0/641473897 wait complete. 2024-06-29T14:22:21.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.284+0000 7f72e81eb640 1 Processor -- start 2024-06-29T14:22:21.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.284+0000 7f72e81eb640 1 -- start start 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e81eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e00d23c0 0x7f72e015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e81eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72e015ee10 0x7f72e00c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e81eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e00c8e80 0x7f72e015ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e81eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0160280 con 0x7f72e00d23c0 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e81eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0160100 con 0x7f72e00c8e80 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e81eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f72e0160400 con 0x7f72e015ee10 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e00d23c0 0x7f72e015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e00d23c0 0x7f72e015d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52112/0 (socket says 172.21.15.59:52112) 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 -- 172.21.15.59:0/4263184162 learned_addr learned my addr 172.21.15.59:0/4263184162 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 -- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72e015ee10 msgr2=0x7f72e00c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72e015ee10 0x7f72e00c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 -- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e00c8e80 msgr2=0x7f72e015ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e00c8e80 0x7f72e015ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 -- 172.21.15.59:0/4263184162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e00cac00 con 0x7f72e00d23c0 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72e5f87640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e00d23c0 0x7f72e015d4f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00c390 tx=0x7f72dc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.286+0000 7f72d67fc640 1 -- 172.21.15.59:0/4263184162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc017070 con 0x7f72e00d23c0 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.287+0000 7f72d67fc640 1 -- 172.21.15.59:0/4263184162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72dc0035f0 con 0x7f72e00d23c0 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.287+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e00c78a0 con 0x7f72e00d23c0 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.287+0000 7f72d67fc640 1 -- 172.21.15.59:0/4263184162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc003a20 con 0x7f72e00d23c0 2024-06-29T14:22:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.287+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e00c7e60 con 0x7f72e00d23c0 2024-06-29T14:22:21.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.288+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e006cf10 con 0x7f72e00d23c0 2024-06-29T14:22:21.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.288+0000 7f72d67fc640 1 -- 172.21.15.59:0/4263184162 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f72dc01a070 con 0x7f72e00d23c0 2024-06-29T14:22:21.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.289+0000 7f72d67fc640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f72c40762d0 0x7f72c40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.289+0000 7f72d67fc640 1 -- 172.21.15.59:0/4263184162 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f72dc096ed0 con 0x7f72e00d23c0 2024-06-29T14:22:21.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.289+0000 7f72e5786640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f72c40762d0 0x7f72c40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.290+0000 7f72e5786640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f72c40762d0 0x7f72c40787e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f72d00058c0 tx=0x7f72d0005850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.293+0000 7f72d67fc640 1 -- 172.21.15.59:0/4263184162 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f72e006cf10 con 0x7f72e00d23c0 2024-06-29T14:22:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.521+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72e00d3600 con 0x7f72e00d23c0 2024-06-29T14:22:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.522+0000 7f72d67fc640 1 -- 172.21.15.59:0/4263184162 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5222 (secure 0 0 0) 0x7f72e00d3600 con 0x7f72e00d23c0 2024-06-29T14:22:21.520 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:21.520 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":9,"btime":"2024-06-29T14:22:12:518854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:22:11.909450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:21.520 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 9 2024-06-29T14:22:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f72c40762d0 msgr2=0x7f72c40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f72c40762d0 0x7f72c40787e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f72d00058c0 tx=0x7f72d0005850 comp rx=0 tx=0).stop 2024-06-29T14:22:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e00d23c0 msgr2=0x7f72e015d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e00d23c0 0x7f72e015d4f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00c390 tx=0x7f72dc00c850 comp rx=0 tx=0).stop 2024-06-29T14:22:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 shutdown_connections 2024-06-29T14:22:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f72c40762d0 0x7f72c40787e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72e00c8e80 0x7f72e015ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72e015ee10 0x7f72e00c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 --2- 172.21.15.59:0/4263184162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e00d23c0 0x7f72e015d4f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 >> 172.21.15.59:0/4263184162 conn(0x7f72e00d3ee0 msgr2=0x7f72e006d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 shutdown_connections 2024-06-29T14:22:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.524+0000 7f72e81eb640 1 -- 172.21.15.59:0/4263184162 wait complete. 2024-06-29T14:22:21.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.653+0000 7f3e43f7f640 1 -- 172.21.15.59:0/2995655695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c162190 msgr2=0x7f3e3c0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.653+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/2995655695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c162190 0x7f3e3c0b38e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0d3aa0 tx=0x7f3e3402a940 comp rx=0 tx=0).stop 2024-06-29T14:22:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.653+0000 7f3e43f7f640 1 -- 172.21.15.59:0/2995655695 shutdown_connections 2024-06-29T14:22:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.653+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/2995655695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c162190 0x7f3e3c0b38e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.653+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/2995655695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e3c162770 0x7f3e3c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.653+0000 7f3e43f7f640 1 -- 172.21.15.59:0/2995655695 >> 172.21.15.59:0/2995655695 conn(0x7f3e3c0d45b0 msgr2=0x7f3e3c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.654+0000 7f3e43f7f640 1 -- 172.21.15.59:0/2995655695 shutdown_connections 2024-06-29T14:22:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.654+0000 7f3e43f7f640 1 -- 172.21.15.59:0/2995655695 wait complete. 2024-06-29T14:22:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.655+0000 7f3e43f7f640 1 Processor -- start 2024-06-29T14:22:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.655+0000 7f3e43f7f640 1 -- start start 2024-06-29T14:22:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e43f7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e3c15e0a0 0x7f3e3c14f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e43f7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c162770 0x7f3e3c0cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e43f7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c0cc1b0 0x7f3e3c160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e43f7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3c0b43a0 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e43f7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3c0b4220 con 0x7f3e3c162770 2024-06-29T14:22:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e43f7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3e3c0b4520 con 0x7f3e3c15e0a0 2024-06-29T14:22:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c0cc1b0 0x7f3e3c160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c0cc1b0 0x7f3e3c160370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52134/0 (socket says 172.21.15.59:52134) 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 -- 172.21.15.59:0/583646017 learned_addr learned my addr 172.21.15.59:0/583646017 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4151a640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c162770 0x7f3e3c0cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e41d1b640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e3c15e0a0 0x7f3e3c14f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 -- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e3c15e0a0 msgr2=0x7f3e3c14f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e3c15e0a0 0x7f3e3c14f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 -- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c162770 msgr2=0x7f3e3c0cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c162770 0x7f3e3c0cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 -- 172.21.15.59:0/583646017 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3c0cc5c0 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4151a640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c162770 0x7f3e3c0cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e41d1b640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e3c15e0a0 0x7f3e3c14f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.657+0000 7f3e4251c640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c0cc1b0 0x7f3e3c160370 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3e34033040 tx=0x7f3e34031da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.658+0000 7f3e327fc640 1 -- 172.21.15.59:0/583646017 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e34034050 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.658+0000 7f3e327fc640 1 -- 172.21.15.59:0/583646017 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e34004540 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.658+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3c1610b0 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.658+0000 7f3e327fc640 1 -- 172.21.15.59:0/583646017 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e34044480 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.658+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e3c161670 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.659+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e3c0b49d0 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.659+0000 7f3e327fc640 1 -- 172.21.15.59:0/583646017 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3e34037020 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.660+0000 7f3e327fc640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e20076300 0x7f3e20078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.660+0000 7f3e327fc640 1 -- 172.21.15.59:0/583646017 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f3e340b8ef0 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.660+0000 7f3e41d1b640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e20076300 0x7f3e20078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.661+0000 7f3e41d1b640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e20076300 0x7f3e20078810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e38005fd0 tx=0x7f3e38005e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.665+0000 7f3e327fc640 1 -- 172.21.15.59:0/583646017 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3e3c0cc5c0 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.858+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e3c0b4d90 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.859+0000 7f3e327fc640 1 -- 172.21.15.59:0/583646017 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7f3e34082510 con 0x7f3e3c0cc1b0 2024-06-29T14:22:21.857 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:21.857 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:21.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e20076300 msgr2=0x7f3e20078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e20076300 0x7f3e20078810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e38005fd0 tx=0x7f3e38005e20 comp rx=0 tx=0).stop 2024-06-29T14:22:21.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c0cc1b0 msgr2=0x7f3e3c160370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c0cc1b0 0x7f3e3c160370 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3e34033040 tx=0x7f3e34031da0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 shutdown_connections 2024-06-29T14:22:21.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e20076300 0x7f3e20078810 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c0cc1b0 0x7f3e3c160370 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e3c162770 0x7f3e3c0cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 --2- 172.21.15.59:0/583646017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e3c15e0a0 0x7f3e3c14f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 >> 172.21.15.59:0/583646017 conn(0x7f3e3c0d45b0 msgr2=0x7f3e3c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 shutdown_connections 2024-06-29T14:22:21.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.861+0000 7f3e43f7f640 1 -- 172.21.15.59:0/583646017 wait complete. 2024-06-29T14:22:21.870 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:21.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.993+0000 7fdf70f03640 1 -- 172.21.15.59:0/2402021956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c144470 msgr2=0x7fdf6c0b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.993+0000 7fdf70f03640 1 --2- 172.21.15.59:0/2402021956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c144470 0x7fdf6c0b1b00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009830 tx=0x7fdf6402a510 comp rx=0 tx=0).stop 2024-06-29T14:22:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.993+0000 7fdf70f03640 1 -- 172.21.15.59:0/2402021956 shutdown_connections 2024-06-29T14:22:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.993+0000 7fdf70f03640 1 --2- 172.21.15.59:0/2402021956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c0b2040 0x7fdf6c06e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.993+0000 7fdf70f03640 1 --2- 172.21.15.59:0/2402021956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c144470 0x7fdf6c0b1b00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.993+0000 7fdf70f03640 1 -- 172.21.15.59:0/2402021956 >> 172.21.15.59:0/2402021956 conn(0x7fdf6c0b06e0 msgr2=0x7fdf6c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.993+0000 7fdf70f03640 1 -- 172.21.15.59:0/2402021956 shutdown_connections 2024-06-29T14:22:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.993+0000 7fdf70f03640 1 -- 172.21.15.59:0/2402021956 wait complete. 2024-06-29T14:22:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.995+0000 7fdf70f03640 1 Processor -- start 2024-06-29T14:22:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.995+0000 7fdf70f03640 1 -- start start 2024-06-29T14:22:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.996+0000 7fdf70f03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c06e840 0x7fdf6c157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.996+0000 7fdf70f03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c0b2040 0x7fdf6c1f5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf70f03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c144470 0x7fdf6c1f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf70f03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c0d0360 con 0x7fdf6c0b2040 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf70f03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c0d01e0 con 0x7fdf6c144470 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf70f03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6c15ee80 con 0x7fdf6c06e840 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c0b2040 0x7fdf6c1f5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c0b2040 0x7fdf6c1f5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52150/0 (socket says 172.21.15.59:52150) 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 -- 172.21.15.59:0/1489680788 learned_addr learned my addr 172.21.15.59:0/1489680788 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf6a59c640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c06e840 0x7fdf6c157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf6ad9d640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c144470 0x7fdf6c1f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 -- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c06e840 msgr2=0x7fdf6c157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c06e840 0x7fdf6c157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 -- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c144470 msgr2=0x7fdf6c1f5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c144470 0x7fdf6c1f5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 -- 172.21.15.59:0/1489680788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c13c910 con 0x7fdf6c0b2040 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf69d9b640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c0b2040 0x7fdf6c1f5120 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400c970 tx=0x7fdf5400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1489680788 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54007bb0 con 0x7fdf6c0b2040 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1489680788 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf54007e40 con 0x7fdf6c0b2040 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6c13e710 con 0x7fdf6c0b2040 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1489680788 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54005280 con 0x7fdf6c0b2040 2024-06-29T14:22:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.997+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6c13ecd0 con 0x7fdf6c0b2040 2024-06-29T14:22:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.998+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6c15f3a0 con 0x7fdf6c0b2040 2024-06-29T14:22:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:21.999+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1489680788 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdf54005420 con 0x7fdf6c0b2040 2024-06-29T14:22:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.000+0000 7fdf62ffd640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf4c0762d0 0x7fdf4c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.000+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1489680788 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fdf54097e70 con 0x7fdf6c0b2040 2024-06-29T14:22:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.000+0000 7fdf6a59c640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf4c0762d0 0x7fdf4c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.000+0000 7fdf6a59c640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf4c0762d0 0x7fdf4c0787e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdf64034000 tx=0x7fdf64004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.004+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1489680788 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf6c13c910 con 0x7fdf6c0b2040 2024-06-29T14:22:22.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.193+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdf6c143db0 con 0x7fdf6c0b2040 2024-06-29T14:22:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.194+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1489680788 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fdf54014030 con 0x7fdf6c0b2040 2024-06-29T14:22:22.192 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf4c0762d0 msgr2=0x7fdf4c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf4c0762d0 0x7fdf4c0787e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdf64034000 tx=0x7fdf64004730 comp rx=0 tx=0).stop 2024-06-29T14:22:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c0b2040 msgr2=0x7fdf6c1f5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c0b2040 0x7fdf6c1f5120 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400c970 tx=0x7fdf5400ce30 comp rx=0 tx=0).stop 2024-06-29T14:22:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 shutdown_connections 2024-06-29T14:22:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fdf4c0762d0 0x7fdf4c0787e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c144470 0x7fdf6c1f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c0b2040 0x7fdf6c1f5120 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 --2- 172.21.15.59:0/1489680788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c06e840 0x7fdf6c157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 >> 172.21.15.59:0/1489680788 conn(0x7fdf6c0b06e0 msgr2=0x7fdf6c15b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:22.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 shutdown_connections 2024-06-29T14:22:22.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.196+0000 7fdf70f03640 1 -- 172.21.15.59:0/1489680788 wait complete. 2024-06-29T14:22:22.205 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.328+0000 7f25c7eef640 1 -- 172.21.15.59:0/1581869930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c00cd9b0 msgr2=0x7f25c015f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.328+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1581869930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c00cd9b0 0x7f25c015f240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f25c00cc020 tx=0x7f25bc02a560 comp rx=0 tx=0).stop 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.328+0000 7f25c7eef640 1 -- 172.21.15.59:0/1581869930 shutdown_connections 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.328+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1581869930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25c006d830 0x7f25c015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.328+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1581869930 >> v2:172.21.15.153:3300/0 conn(0x7f25c01444c0 0x7f25c006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.328+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1581869930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c00cd9b0 0x7f25c015f240 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.328+0000 7f25c7eef640 1 -- 172.21.15.59:0/1581869930 >> 172.21.15.59:0/1581869930 conn(0x7f25c00b24c0 msgr2=0x7f25c00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.328+0000 7f25c7eef640 1 -- 172.21.15.59:0/1581869930 shutdown_connections 2024-06-29T14:22:22.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.329+0000 7f25c7eef640 1 -- 172.21.15.59:0/1581869930 wait complete. 2024-06-29T14:22:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.330+0000 7f25c7eef640 1 Processor -- start 2024-06-29T14:22:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.330+0000 7f25c7eef640 1 -- start start 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c7eef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c006d830 0x7f25c0136490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c7eef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25c00cd9b0 0x7f25c0139290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c7eef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25c01444c0 0x7f25c00af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c7eef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f25c015b960 con 0x7f25c006d830 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c7eef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f25c015b7e0 con 0x7f25c01444c0 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c7eef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f25c015bae0 con 0x7f25c00cd9b0 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c006d830 0x7f25c0136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c006d830 0x7f25c0136490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52168/0 (socket says 172.21.15.59:52168) 2024-06-29T14:22:22.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 -- 172.21.15.59:0/1277007012 learned_addr learned my addr 172.21.15.59:0/1277007012 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c548a640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25c00cd9b0 0x7f25c0139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c648c640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25c01444c0 0x7f25c00af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 -- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25c00cd9b0 msgr2=0x7f25c0139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25c00cd9b0 0x7f25c0139290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 -- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25c01444c0 msgr2=0x7f25c00af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25c01444c0 0x7f25c00af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 -- 172.21.15.59:0/1277007012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c0137b30 con 0x7f25c006d830 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c648c640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25c01444c0 0x7f25c00af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c548a640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25c00cd9b0 0x7f25c0139290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25c5c8b640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c006d830 0x7f25c0136490 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f25bc034040 tx=0x7f25bc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25b67fc640 1 -- 172.21.15.59:0/1277007012 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc03e030 con 0x7f25c006d830 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.332+0000 7f25b67fc640 1 -- 172.21.15.59:0/1277007012 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25bc003d10 con 0x7f25c006d830 2024-06-29T14:22:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.333+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c0137e10 con 0x7f25c006d830 2024-06-29T14:22:22.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.333+0000 7f25b67fc640 1 -- 172.21.15.59:0/1277007012 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc035040 con 0x7f25c006d830 2024-06-29T14:22:22.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.333+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c01383d0 con 0x7f25c006d830 2024-06-29T14:22:22.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.334+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c00cf8b0 con 0x7f25c006d830 2024-06-29T14:22:22.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.334+0000 7f25b67fc640 1 -- 172.21.15.59:0/1277007012 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f25bc041030 con 0x7f25c006d830 2024-06-29T14:22:22.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.335+0000 7f25b67fc640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f25a4076300 0x7f25a4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.335+0000 7f25b67fc640 1 -- 172.21.15.59:0/1277007012 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f25bc0b81b0 con 0x7f25c006d830 2024-06-29T14:22:22.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.335+0000 7f25c548a640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f25a4076300 0x7f25a4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.336+0000 7f25c548a640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f25a4076300 0x7f25a4078810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f25c0138d20 tx=0x7f25b0005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:22.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.339+0000 7f25b67fc640 1 -- 172.21.15.59:0/1277007012 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f25c00cf8b0 con 0x7f25c006d830 2024-06-29T14:22:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.529+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f25c013ae50 con 0x7f25c006d830 2024-06-29T14:22:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.529+0000 7f25b67fc640 1 -- 172.21.15.59:0/1277007012 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f25c013ae50 con 0x7f25c006d830 2024-06-29T14:22:22.530 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f25a4076300 msgr2=0x7f25a4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f25a4076300 0x7f25a4078810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f25c0138d20 tx=0x7f25b0005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c006d830 msgr2=0x7f25c0136490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c006d830 0x7f25c0136490 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f25bc034040 tx=0x7f25bc003820 comp rx=0 tx=0).stop 2024-06-29T14:22:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 shutdown_connections 2024-06-29T14:22:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f25a4076300 0x7f25a4078810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25c01444c0 0x7f25c00af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25c00cd9b0 0x7f25c0139290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 --2- 172.21.15.59:0/1277007012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25c006d830 0x7f25c0136490 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 >> 172.21.15.59:0/1277007012 conn(0x7f25c00b24c0 msgr2=0x7f25c015ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 shutdown_connections 2024-06-29T14:22:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.531+0000 7f25c7eef640 1 -- 172.21.15.59:0/1277007012 wait complete. 2024-06-29T14:22:22.540 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:22.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.663+0000 7fd882c89640 1 -- 172.21.15.59:0/2296187147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 msgr2=0x7fd87c15bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.663+0000 7fd882c89640 1 --2- 172.21.15.59:0/2296187147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 0x7fd87c15bc20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd874009830 tx=0x7fd87402a510 comp rx=0 tx=0).stop 2024-06-29T14:22:22.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.663+0000 7fd882c89640 1 -- 172.21.15.59:0/2296187147 shutdown_connections 2024-06-29T14:22:22.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.663+0000 7fd882c89640 1 --2- 172.21.15.59:0/2296187147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd87c0b2fa0 0x7fd87c0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.663+0000 7fd882c89640 1 --2- 172.21.15.59:0/2296187147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 0x7fd87c15bc20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.663+0000 7fd882c89640 1 -- 172.21.15.59:0/2296187147 >> 172.21.15.59:0/2296187147 conn(0x7fd87c073e10 msgr2=0x7fd87c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:22.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.663+0000 7fd882c89640 1 -- 172.21.15.59:0/2296187147 shutdown_connections 2024-06-29T14:22:22.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.663+0000 7fd882c89640 1 -- 172.21.15.59:0/2296187147 wait complete. 2024-06-29T14:22:22.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.665+0000 7fd882c89640 1 Processor -- start 2024-06-29T14:22:22.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.665+0000 7fd882c89640 1 -- start start 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.666+0000 7fd882c89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 0x7fd87c0b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.666+0000 7fd882c89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd87c0b2fa0 0x7fd87c143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd882c89640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd87c0d1a60 0x7fd87c1436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd882c89640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c0b6cc0 con 0x7fd87c0b2270 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd882c89640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c0b6b40 con 0x7fd87c0b2fa0 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd882c89640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd87c0cfe90 con 0x7fd87c0d1a60 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 0x7fd87c0b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 0x7fd87c0b0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52184/0 (socket says 172.21.15.59:52184) 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 -- 172.21.15.59:0/3688025242 learned_addr learned my addr 172.21.15.59:0/3688025242 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:22.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 -- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd87c0d1a60 msgr2=0x7fd87c1436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd881226640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd87c0d1a60 0x7fd87c1436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd87c0d1a60 0x7fd87c1436b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd87bfff640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd87c0b2fa0 0x7fd87c143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 -- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd87c0b2fa0 msgr2=0x7fd87c143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd87c0b2fa0 0x7fd87c143170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 -- 172.21.15.59:0/3688025242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87c1e5820 con 0x7fd87c0b2270 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd87bfff640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd87c0b2fa0 0x7fd87c143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd880a25640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 0x7fd87c0b0570 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd874009950 tx=0x7fd874033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd8797fa640 1 -- 172.21.15.59:0/3688025242 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd874005c90 con 0x7fd87c0b2270 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd8797fa640 1 -- 172.21.15.59:0/3688025242 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd874031e50 con 0x7fd87c0b2270 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87c141210 con 0x7fd87c0b2270 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd8797fa640 1 -- 172.21.15.59:0/3688025242 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8740332b0 con 0x7fd87c0b2270 2024-06-29T14:22:22.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.667+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd87c1417d0 con 0x7fd87c0b2270 2024-06-29T14:22:22.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.668+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd87c1e59f0 con 0x7fd87c0b2270 2024-06-29T14:22:22.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.669+0000 7fd8797fa640 1 -- 172.21.15.59:0/3688025242 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd874033450 con 0x7fd87c0b2270 2024-06-29T14:22:22.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.670+0000 7fd8797fa640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd85c076300 0x7fd85c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.670+0000 7fd8797fa640 1 -- 172.21.15.59:0/3688025242 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd8740b8dd0 con 0x7fd87c0b2270 2024-06-29T14:22:22.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.670+0000 7fd87bfff640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd85c076300 0x7fd85c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.671+0000 7fd87bfff640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd85c076300 0x7fd85c078810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd87c1f5f10 tx=0x7fd86c009360 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:22.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.674+0000 7fd8797fa640 1 -- 172.21.15.59:0/3688025242 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd87403f030 con 0x7fd87c0b2270 2024-06-29T14:22:22.767 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:22:22.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:22:22.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.864+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd87c0d0400 con 0x7fd87c0b2270 2024-06-29T14:22:22.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.864+0000 7fd8797fa640 1 -- 172.21.15.59:0/3688025242 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fd8740ba0a0 con 0x7fd87c0b2270 2024-06-29T14:22:22.862 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:22.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd85c076300 msgr2=0x7fd85c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd85c076300 0x7fd85c078810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd87c1f5f10 tx=0x7fd86c009360 comp rx=0 tx=0).stop 2024-06-29T14:22:22.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 msgr2=0x7fd87c0b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 0x7fd87c0b0570 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd874009950 tx=0x7fd874033c90 comp rx=0 tx=0).stop 2024-06-29T14:22:22.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 shutdown_connections 2024-06-29T14:22:22.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd85c076300 0x7fd85c078810 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd87c0d1a60 0x7fd87c1436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd87c0b2fa0 0x7fd87c143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 --2- 172.21.15.59:0/3688025242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd87c0b2270 0x7fd87c0b0570 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 >> 172.21.15.59:0/3688025242 conn(0x7fd87c073e10 msgr2=0x7fd87c15b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:22.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 shutdown_connections 2024-06-29T14:22:22.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:22.866+0000 7fd882c89640 1 -- 172.21.15.59:0/3688025242 wait complete. 2024-06-29T14:22:22.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:23.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.038+0000 7f994b081640 1 -- 172.21.15.59:0/1038906851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99441414c0 msgr2=0x7f99441418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.038+0000 7f994b081640 1 --2- 172.21.15.59:0/1038906851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99441414c0 0x7f99441418d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9934005fd0 tx=0x7f993402a970 comp rx=0 tx=0).stop 2024-06-29T14:22:23.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.038+0000 7f994b081640 1 -- 172.21.15.59:0/1038906851 shutdown_connections 2024-06-29T14:22:23.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.038+0000 7f994b081640 1 --2- 172.21.15.59:0/1038906851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99441414c0 0x7f99441418d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.038+0000 7f994b081640 1 --2- 172.21.15.59:0/1038906851 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f994413e9b0 0x7f994413edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.038+0000 7f994b081640 1 -- 172.21.15.59:0/1038906851 >> 172.21.15.59:0/1038906851 conn(0x7f99440b6fb0 msgr2=0x7f99440b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:23.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.038+0000 7f994b081640 1 -- 172.21.15.59:0/1038906851 shutdown_connections 2024-06-29T14:22:23.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.038+0000 7f994b081640 1 -- 172.21.15.59:0/1038906851 wait complete. 2024-06-29T14:22:23.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.040+0000 7f994b081640 1 Processor -- start 2024-06-29T14:22:23.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.040+0000 7f994b081640 1 -- start start 2024-06-29T14:22:23.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.041+0000 7f994b081640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994413e9b0 0x7f9944157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.041+0000 7f994b081640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99441414c0 0x7f994413b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.041+0000 7f994b081640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994415ceb0 0x7f99441534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.041+0000 7f994b081640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f99440b2a80 con 0x7f994413e9b0 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.041+0000 7f994b081640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99440b2900 con 0x7f994415ceb0 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.041+0000 7f994b081640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f99440b2c00 con 0x7f99441414c0 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994413e9b0 0x7f9944157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994413e9b0 0x7f9944157a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35052/0 (socket says 172.21.15.59:35052) 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 -- 172.21.15.59:0/2940488668 learned_addr learned my addr 172.21.15.59:0/2940488668 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f994961e640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994415ceb0 0x7f99441534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9943fff640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99441414c0 0x7f994413b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 -- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99441414c0 msgr2=0x7f994413b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99441414c0 0x7f994413b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 -- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994415ceb0 msgr2=0x7f99441534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994415ceb0 0x7f99441534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 -- 172.21.15.59:0/2940488668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994413b5e0 con 0x7f994413e9b0 2024-06-29T14:22:23.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9943fff640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99441414c0 0x7f994413b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:23.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f9948e1d640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994413e9b0 0x7f9944157a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f993c00b420 tx=0x7f993c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:23.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f99417fa640 1 -- 172.21.15.59:0/2940488668 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c011830 con 0x7f994413e9b0 2024-06-29T14:22:23.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f99417fa640 1 -- 172.21.15.59:0/2940488668 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f993c016070 con 0x7f994413e9b0 2024-06-29T14:22:23.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f99417fa640 1 -- 172.21.15.59:0/2940488668 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c011d60 con 0x7f994413e9b0 2024-06-29T14:22:23.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9944156c50 con 0x7f994413e9b0 2024-06-29T14:22:23.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.042+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9944157210 con 0x7f994413e9b0 2024-06-29T14:22:23.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.044+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f994413edc0 con 0x7f994413e9b0 2024-06-29T14:22:23.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.044+0000 7f99417fa640 1 -- 172.21.15.59:0/2940488668 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f993c01b460 con 0x7f994413e9b0 2024-06-29T14:22:23.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.045+0000 7f99417fa640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f99240762d0 0x7f99240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.045+0000 7f99417fa640 1 -- 172.21.15.59:0/2940488668 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f993c097680 con 0x7f994413e9b0 2024-06-29T14:22:23.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.045+0000 7f9943fff640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f99240762d0 0x7f99240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.046+0000 7f9943fff640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f99240762d0 0x7f99240787e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f99440cfcf0 tx=0x7f992c005f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.049+0000 7f99417fa640 1 -- 172.21.15.59:0/2940488668 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f993c020020 con 0x7f994413e9b0 2024-06-29T14:22:23.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.242+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f99440cf760 con 0x7f994413e9b0 2024-06-29T14:22:23.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.243+0000 7f99417fa640 1 -- 172.21.15.59:0/2940488668 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f993c060b70 con 0x7f994413e9b0 2024-06-29T14:22:23.241 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:23.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f99240762d0 msgr2=0x7f99240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f99240762d0 0x7f99240787e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f99440cfcf0 tx=0x7f992c005f10 comp rx=0 tx=0).stop 2024-06-29T14:22:23.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994413e9b0 msgr2=0x7f9944157a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994413e9b0 0x7f9944157a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f993c00b420 tx=0x7f993c00b8e0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 shutdown_connections 2024-06-29T14:22:23.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f99240762d0 0x7f99240787e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f994415ceb0 0x7f99441534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99441414c0 0x7f994413b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 --2- 172.21.15.59:0/2940488668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994413e9b0 0x7f9944157a10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 >> 172.21.15.59:0/2940488668 conn(0x7f99440b6fb0 msgr2=0x7f994413fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:23.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 shutdown_connections 2024-06-29T14:22:23.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.245+0000 7f994b081640 1 -- 172.21.15.59:0/2940488668 wait complete. 2024-06-29T14:22:23.254 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:23.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.376+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/1499467399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78162770 msgr2=0x7f7c780ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.376+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/1499467399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78162770 0x7f7c780ce550 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7c68005fd0 tx=0x7f7c6802aa10 comp rx=0 tx=0).stop 2024-06-29T14:22:23.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.376+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/1499467399 shutdown_connections 2024-06-29T14:22:23.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.376+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/1499467399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c7815e0a0 0x7f7c78161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.376+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/1499467399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78162770 0x7f7c780ce550 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.376+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/1499467399 >> 172.21.15.59:0/1499467399 conn(0x7f7c780d45b0 msgr2=0x7f7c780d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:23.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.376+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/1499467399 shutdown_connections 2024-06-29T14:22:23.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.376+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/1499467399 wait complete. 2024-06-29T14:22:23.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.378+0000 7f7c7cfbd640 1 Processor -- start 2024-06-29T14:22:23.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.378+0000 7f7c7cfbd640 1 -- start start 2024-06-29T14:22:23.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.379+0000 7f7c7cfbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c7815e0a0 0x7f7c7815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.379+0000 7f7c7cfbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78161fc0 0x7f7c78152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.379+0000 7f7c7cfbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c78162770 0x7f7c7806d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.379+0000 7f7c7cfbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7c780d2d80 con 0x7f7c78161fc0 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.379+0000 7f7c7cfbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c780b4b00 con 0x7f7c7815e0a0 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.379+0000 7f7c7cfbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7c780d2f00 con 0x7f7c78162770 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78161fc0 0x7f7c78152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78161fc0 0x7f7c78152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35070/0 (socket says 172.21.15.59:35070) 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 -- 172.21.15.59:0/3358382968 learned_addr learned my addr 172.21.15.59:0/3358382968 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c76d9d640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c78162770 0x7f7c7806d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c7659c640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c7815e0a0 0x7f7c7815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 -- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c78162770 msgr2=0x7f7c7806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c78162770 0x7f7c7806d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 -- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c7815e0a0 msgr2=0x7f7c7815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c7815e0a0 0x7f7c7815ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 -- 172.21.15.59:0/3358382968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c780cd830 con 0x7f7c78161fc0 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c7659c640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c7815e0a0 0x7f7c7815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c75d9b640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78161fc0 0x7f7c78152aa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000d6a0 tx=0x7f7c6000db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c6effd640 1 -- 172.21.15.59:0/3358382968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60017070 con 0x7f7c78161fc0 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c6effd640 1 -- 172.21.15.59:0/3358382968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c60004310 con 0x7f7c78161fc0 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c780cdb10 con 0x7f7c78161fc0 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c6effd640 1 -- 172.21.15.59:0/3358382968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60005280 con 0x7f7c78161fc0 2024-06-29T14:22:23.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.380+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c7815fdf0 con 0x7f7c78161fc0 2024-06-29T14:22:23.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.381+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c780cdd50 con 0x7f7c78161fc0 2024-06-29T14:22:23.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.382+0000 7f7c6effd640 1 -- 172.21.15.59:0/3358382968 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7c60005420 con 0x7f7c78161fc0 2024-06-29T14:22:23.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.382+0000 7f7c6effd640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7c580762d0 0x7f7c580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.383+0000 7f7c6effd640 1 -- 172.21.15.59:0/3358382968 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f7c60097f00 con 0x7f7c78161fc0 2024-06-29T14:22:23.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.383+0000 7f7c7659c640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7c580762d0 0x7f7c580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.383+0000 7f7c7659c640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7c580762d0 0x7f7c580787e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7c68035000 tx=0x7f7c68004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:23.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.387+0000 7f7c6effd640 1 -- 172.21.15.59:0/3358382968 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7c60020020 con 0x7f7c78161fc0 2024-06-29T14:22:23.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.615+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c780d3410 con 0x7f7c78161fc0 2024-06-29T14:22:23.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.615+0000 7f7c6effd640 1 -- 172.21.15.59:0/3358382968 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5222 (secure 0 0 0) 0x7f7c600613f0 con 0x7f7c78161fc0 2024-06-29T14:22:23.614 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:23.614 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":9,"btime":"2024-06-29T14:22:12:518854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:22:11.909450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:23.614 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 9 2024-06-29T14:22:23.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7c580762d0 msgr2=0x7f7c580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7c580762d0 0x7f7c580787e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7c68035000 tx=0x7f7c68004ab0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78161fc0 msgr2=0x7f7c78152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78161fc0 0x7f7c78152aa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000d6a0 tx=0x7f7c6000db60 comp rx=0 tx=0).stop 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 shutdown_connections 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7c580762d0 0x7f7c580787e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c78162770 0x7f7c7806d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c78161fc0 0x7f7c78152aa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 --2- 172.21.15.59:0/3358382968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c7815e0a0 0x7f7c7815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 >> 172.21.15.59:0/3358382968 conn(0x7f7c780d45b0 msgr2=0x7f7c780d2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 shutdown_connections 2024-06-29T14:22:23.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.617+0000 7f7c7cfbd640 1 -- 172.21.15.59:0/3358382968 wait complete. 2024-06-29T14:22:23.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.746+0000 7fd744ac0640 1 -- 172.21.15.59:0/2076963532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400b2270 msgr2=0x7fd74015bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.746+0000 7fd744ac0640 1 --2- 172.21.15.59:0/2076963532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400b2270 0x7fd74015bc20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd7400b2680 tx=0x7fd73002a560 comp rx=0 tx=0).stop 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.746+0000 7fd744ac0640 1 -- 172.21.15.59:0/2076963532 shutdown_connections 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.746+0000 7fd744ac0640 1 --2- 172.21.15.59:0/2076963532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400d1a60 0x7fd7400cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.746+0000 7fd744ac0640 1 --2- 172.21.15.59:0/2076963532 >> v2:172.21.15.153:3300/0 conn(0x7fd7400b2fa0 0x7fd7400d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.746+0000 7fd744ac0640 1 --2- 172.21.15.59:0/2076963532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400b2270 0x7fd74015bc20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.746+0000 7fd744ac0640 1 -- 172.21.15.59:0/2076963532 >> 172.21.15.59:0/2076963532 conn(0x7fd740073e10 msgr2=0x7fd740074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.747+0000 7fd744ac0640 1 -- 172.21.15.59:0/2076963532 shutdown_connections 2024-06-29T14:22:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.747+0000 7fd744ac0640 1 -- 172.21.15.59:0/2076963532 wait complete. 2024-06-29T14:22:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.748+0000 7fd744ac0640 1 Processor -- start 2024-06-29T14:22:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.749+0000 7fd744ac0640 1 -- start start 2024-06-29T14:22:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd744ac0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7400b2270 0x7fd74014c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd744ac0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400b2fa0 0x7fd7400c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd744ac0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400d1a60 0x7fd7400c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd744ac0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd7400cfe40 con 0x7fd7400d1a60 2024-06-29T14:22:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd744ac0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd7400cfcc0 con 0x7fd7400b2270 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd744ac0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd7400cffc0 con 0x7fd7400b2fa0 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400d1a60 0x7fd7400c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400d1a60 0x7fd7400c6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35100/0 (socket says 172.21.15.59:35100) 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73ed9d640 1 -- 172.21.15.59:0/441779692 learned_addr learned my addr 172.21.15.59:0/441779692 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73e59c640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7400b2270 0x7fd74014c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd737fff640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400b2fa0 0x7fd7400c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73ed9d640 1 -- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400b2fa0 msgr2=0x7fd7400c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73ed9d640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400b2fa0 0x7fd7400c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73ed9d640 1 -- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7400b2270 msgr2=0x7fd74014c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73ed9d640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7400b2270 0x7fd74014c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.750+0000 7fd73ed9d640 1 -- 172.21.15.59:0/441779692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74014c870 con 0x7fd7400d1a60 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.751+0000 7fd73ed9d640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400d1a60 0x7fd7400c6ca0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd73800ca90 tx=0x7fd73800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.751+0000 7fd736ffd640 1 -- 172.21.15.59:0/441779692 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738007df0 con 0x7fd7400d1a60 2024-06-29T14:22:23.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.751+0000 7fd736ffd640 1 -- 172.21.15.59:0/441779692 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd73800f450 con 0x7fd7400d1a60 2024-06-29T14:22:23.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.751+0000 7fd736ffd640 1 -- 172.21.15.59:0/441779692 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73800f8c0 con 0x7fd7400d1a60 2024-06-29T14:22:23.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.751+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7400aec20 con 0x7fd7400d1a60 2024-06-29T14:22:23.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.751+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7400af1e0 con 0x7fd7400d1a60 2024-06-29T14:22:23.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.752+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7400d0440 con 0x7fd7400d1a60 2024-06-29T14:22:23.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.752+0000 7fd736ffd640 1 -- 172.21.15.59:0/441779692 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd73800fa60 con 0x7fd7400d1a60 2024-06-29T14:22:23.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.753+0000 7fd736ffd640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd7200762d0 0x7fd7200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:23.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.753+0000 7fd736ffd640 1 -- 172.21.15.59:0/441779692 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd7380985b0 con 0x7fd7400d1a60 2024-06-29T14:22:23.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.753+0000 7fd73e59c640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd7200762d0 0x7fd7200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:23.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.754+0000 7fd73e59c640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd7200762d0 0x7fd7200787e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd730033000 tx=0x7fd730004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.757+0000 7fd736ffd640 1 -- 172.21.15.59:0/441779692 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd738014030 con 0x7fd7400d1a60 2024-06-29T14:22:23.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.947+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7400b6bd0 con 0x7fd7400d1a60 2024-06-29T14:22:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.948+0000 7fd736ffd640 1 -- 172.21.15.59:0/441779692 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7fd7380614f0 con 0x7fd7400d1a60 2024-06-29T14:22:23.946 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:23.946 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd7200762d0 msgr2=0x7fd7200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd7200762d0 0x7fd7200787e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd730033000 tx=0x7fd730004730 comp rx=0 tx=0).stop 2024-06-29T14:22:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400d1a60 msgr2=0x7fd7400c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400d1a60 0x7fd7400c6ca0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd73800ca90 tx=0x7fd73800cf50 comp rx=0 tx=0).stop 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 shutdown_connections 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd7200762d0 0x7fd7200787e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400d1a60 0x7fd7400c6ca0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400b2fa0 0x7fd7400c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 --2- 172.21.15.59:0/441779692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7400b2270 0x7fd74014c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 >> 172.21.15.59:0/441779692 conn(0x7fd740073e10 msgr2=0x7fd740074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 shutdown_connections 2024-06-29T14:22:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:23.950+0000 7fd744ac0640 1 -- 172.21.15.59:0/441779692 wait complete. 2024-06-29T14:22:23.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 -- 172.21.15.59:0/3615245620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07181458a0 msgr2=0x7f07180cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 --2- 172.21.15.59:0/3615245620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07181458a0 0x7f07180cf870 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f07180d0730 tx=0x7f070802aa30 comp rx=0 tx=0).stop 2024-06-29T14:22:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 -- 172.21.15.59:0/3615245620 shutdown_connections 2024-06-29T14:22:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 --2- 172.21.15.59:0/3615245620 >> v2:172.21.15.153:3300/0 conn(0x7f071815c2f0 0x7f071815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 --2- 172.21.15.59:0/3615245620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07180b3740 0x7f07180b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 --2- 172.21.15.59:0/3615245620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07181458a0 0x7f07180cf870 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 -- 172.21.15.59:0/3615245620 >> 172.21.15.59:0/3615245620 conn(0x7f071806c950 msgr2=0x7f0718073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 -- 172.21.15.59:0/3615245620 shutdown_connections 2024-06-29T14:22:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.082+0000 7f071c9d2640 1 -- 172.21.15.59:0/3615245620 wait complete. 2024-06-29T14:22:24.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.084+0000 7f071c9d2640 1 Processor -- start 2024-06-29T14:22:24.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.084+0000 7f071c9d2640 1 -- start start 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f071c9d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07180b3740 0x7f071815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f071c9d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f071815c2f0 0x7f07180709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f071c9d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0718070ee0 0x7f07180c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f071c9d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f071815fe40 con 0x7f071815c2f0 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f071c9d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f071815fcc0 con 0x7f07180b3740 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f071c9d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f071815ffc0 con 0x7f0718070ee0 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f071815c2f0 0x7f07180709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f071815c2f0 0x7f07180709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35110/0 (socket says 172.21.15.59:35110) 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 -- 172.21.15.59:0/2590235810 learned_addr learned my addr 172.21.15.59:0/2590235810 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0716d9d640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0718070ee0 0x7f07180c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f071659c640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07180b3740 0x7f071815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 -- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0718070ee0 msgr2=0x7f07180c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0718070ee0 0x7f07180c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 -- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07180b3740 msgr2=0x7f071815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07180b3740 0x7f071815b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 -- 172.21.15.59:0/2590235810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07181f4b00 con 0x7f071815c2f0 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0716d9d640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0718070ee0 0x7f07180c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f0715d9b640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f071815c2f0 0x7f07180709a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f070000d660 tx=0x7f070000db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f070effd640 1 -- 172.21.15.59:0/2590235810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700017070 con 0x7f071815c2f0 2024-06-29T14:22:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f070effd640 1 -- 172.21.15.59:0/2590235810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0700005310 con 0x7f071815c2f0 2024-06-29T14:22:24.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0718071640 con 0x7f071815c2f0 2024-06-29T14:22:24.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.086+0000 7f070effd640 1 -- 172.21.15.59:0/2590235810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700009820 con 0x7f071815c2f0 2024-06-29T14:22:24.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.087+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071806db80 con 0x7f071815c2f0 2024-06-29T14:22:24.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.087+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07180713a0 con 0x7f071815c2f0 2024-06-29T14:22:24.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.088+0000 7f070effd640 1 -- 172.21.15.59:0/2590235810 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f07000099c0 con 0x7f071815c2f0 2024-06-29T14:22:24.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.089+0000 7f070effd640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f06f8076300 0x7f06f8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.089+0000 7f070effd640 1 -- 172.21.15.59:0/2590235810 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0700097a50 con 0x7f071815c2f0 2024-06-29T14:22:24.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.089+0000 7f071659c640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f06f8076300 0x7f06f8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.090+0000 7f071659c640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f06f8076300 0x7f06f8078810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0708035000 tx=0x7f0708004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:24.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.093+0000 7f070effd640 1 -- 172.21.15.59:0/2590235810 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f07180713a0 con 0x7f071815c2f0 2024-06-29T14:22:24.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.321+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07181458a0 con 0x7f071815c2f0 2024-06-29T14:22:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.322+0000 7f070effd640 1 -- 172.21.15.59:0/2590235810 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5222 (secure 0 0 0) 0x7f07181f4b00 con 0x7f071815c2f0 2024-06-29T14:22:24.320 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:24.320 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":9,"btime":"2024-06-29T14:22:12:518854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:22:11.909450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 9 2024-06-29T14:22:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f06f8076300 msgr2=0x7f06f8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f06f8076300 0x7f06f8078810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0708035000 tx=0x7f0708004ab0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f071815c2f0 msgr2=0x7f07180709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f071815c2f0 0x7f07180709a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f070000d660 tx=0x7f070000db20 comp rx=0 tx=0).stop 2024-06-29T14:22:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 shutdown_connections 2024-06-29T14:22:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f06f8076300 0x7f06f8078810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0718070ee0 0x7f07180c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f071815c2f0 0x7f07180709a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 --2- 172.21.15.59:0/2590235810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07180b3740 0x7f071815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 >> 172.21.15.59:0/2590235810 conn(0x7f071806c950 msgr2=0x7f071815e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 shutdown_connections 2024-06-29T14:22:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.324+0000 7f071c9d2640 1 -- 172.21.15.59:0/2590235810 wait complete. 2024-06-29T14:22:24.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:21:54.635958+0000', 'modified': '2024-06-29T14:22:11.909450+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4375': {'gid': 4375, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6836/2240732059', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 2240732059}, {'type': 'v1', 'addr': '172.21.15.153:6836', 'nonce': 2240732059}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4375, 'qdb_cluster': [4375]} 2024-06-29T14:22:24.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:22:24.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T14:22:24.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.860+0000 7f71c6d4a640 1 -- 172.21.15.59:0/1181934458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71ac004d40 msgr2=0x7f71ac0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.860+0000 7f71c6d4a640 1 --2- 172.21.15.59:0/1181934458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71ac004d40 0x7f71ac0051f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f71b0005fd0 tx=0x7f71ac005830 comp rx=0 tx=0).stop 2024-06-29T14:22:24.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.860+0000 7f71c52e7640 1 -- 172.21.15.59:0/1181934458 reap_dead start 2024-06-29T14:22:24.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.860+0000 7f71c6d4a640 1 -- 172.21.15.59:0/1181934458 shutdown_connections 2024-06-29T14:22:24.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.860+0000 7f71c6d4a640 1 -- 172.21.15.59:0/1181934458 >> 172.21.15.59:0/1181934458 conn(0x7f71c00b5610 msgr2=0x7f71c00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:24.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.861+0000 7f71c6d4a640 1 -- 172.21.15.59:0/1181934458 shutdown_connections 2024-06-29T14:22:24.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.861+0000 7f71c6d4a640 1 -- 172.21.15.59:0/1181934458 wait complete. 2024-06-29T14:22:24.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.862+0000 7f71c6d4a640 1 Processor -- start 2024-06-29T14:22:24.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.862+0000 7f71c6d4a640 1 -- start start 2024-06-29T14:22:24.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71c6d4a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71ac004d40 0x7f71c015b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71c6d4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71c0165aa0 0x7f71c0165eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71c6d4a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71c015c0a0 0x7f71c015c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71c6d4a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0162df0 con 0x7f71c0165aa0 2024-06-29T14:22:24.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71c6d4a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0162c70 con 0x7f71ac004d40 2024-06-29T14:22:24.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71c6d4a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f71c0162f70 con 0x7f71c015c0a0 2024-06-29T14:22:24.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71c0165aa0 0x7f71c0165eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71c0165aa0 0x7f71c0165eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35136/0 (socket says 172.21.15.59:35136) 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 -- 172.21.15.59:0/2517871669 learned_addr learned my addr 172.21.15.59:0/2517871669 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71ac004d40 0x7f71c015b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71c52e7640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71c015c0a0 0x7f71c015c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 -- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71c015c0a0 msgr2=0x7f71c015c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71c015c0a0 0x7f71c015c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 -- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71ac004d40 msgr2=0x7f71c015b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71ac004d40 0x7f71c015b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 -- 172.21.15.59:0/2517871669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c015c9f0 con 0x7f71c0165aa0 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.864+0000 7f71bffff640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71c0165aa0 0x7f71c0165eb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f71a8003ea0 tx=0x7f71a8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.865+0000 7f71bd7fa640 1 -- 172.21.15.59:0/2517871669 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8015070 con 0x7f71c0165aa0 2024-06-29T14:22:24.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.865+0000 7f71bd7fa640 1 -- 172.21.15.59:0/2517871669 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71a8011840 con 0x7f71c0165aa0 2024-06-29T14:22:24.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.865+0000 7f71c6d4a640 1 -- 172.21.15.59:0/2517871669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c01667d0 con 0x7f71c0165aa0 2024-06-29T14:22:24.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.865+0000 7f71bd7fa640 1 -- 172.21.15.59:0/2517871669 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8011c70 con 0x7f71c0165aa0 2024-06-29T14:22:24.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.865+0000 7f71c6d4a640 1 -- 172.21.15.59:0/2517871669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c0166d90 con 0x7f71c0165aa0 2024-06-29T14:22:24.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.866+0000 7f71bd7fa640 1 -- 172.21.15.59:0/2517871669 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f71a8018070 con 0x7f71c0165aa0 2024-06-29T14:22:24.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.867+0000 7f71bd7fa640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71a0076300 0x7f71a0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.867+0000 7f71bd7fa640 1 -- 172.21.15.59:0/2517871669 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f71a80985d0 con 0x7f71c0165aa0 2024-06-29T14:22:24.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.867+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71a0076300 0x7f71a0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.868+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71a0076300 0x7f71a0078810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f71b8001d00 tx=0x7f71b8006720 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:24.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.868+0000 7f71cdc09740 1 Processor -- start 2024-06-29T14:22:24.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.868+0000 7f71cdc09740 1 -- start start 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc0e96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcd43b0 0x55d6dc0cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71cdc09740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d6dc065bc0 con 0x55d6dbcdc590 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71cdc09740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d6dc014630 con 0x55d6dbcfd270 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71cdc09740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d6dc003bd0 con 0x55d6dbcd43b0 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71c52e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71c52e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35140/0 (socket says 172.21.15.59:35140) 2024-06-29T14:22:24.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.869+0000 7f71c52e7640 1 -- 172.21.15.59:0/231480698 learned_addr learned my addr 172.21.15.59:0/231480698 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc0e96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71bffff640 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcd43b0 0x55d6dc0cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71c52e7640 1 -- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcd43b0 msgr2=0x55d6dc0cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71c52e7640 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcd43b0 0x55d6dc0cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71c52e7640 1 -- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 msgr2=0x55d6dc0e96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71c52e7640 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc0e96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71c52e7640 1 -- 172.21.15.59:0/231480698 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6dc000ca0 con 0x55d6dbcdc590 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71bffff640 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcd43b0 0x55d6dc0cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71c52e7640 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cd8e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f71b0002230 tx=0x7f71b0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55093a84cd9e5745 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71b67fc640 1 -- 172.21.15.59:0/231480698 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0038040 con 0x55d6dbcdc590 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71b67fc640 1 -- 172.21.15.59:0/231480698 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71b0042070 con 0x55d6dbcdc590 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.870+0000 7f71b67fc640 1 -- 172.21.15.59:0/231480698 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b003d6d0 con 0x55d6dbcdc590 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 -- 172.21.15.59:0/231480698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 msgr2=0x55d6dc0cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cd8e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f71b0002230 tx=0x7f71b0004ab0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 -- 172.21.15.59:0/231480698 shutdown_connections 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cd8e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcd43b0 0x55d6dc0cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 --2- 172.21.15.59:0/231480698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc0e96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 -- 172.21.15.59:0/231480698 >> 172.21.15.59:0/231480698 conn(0x55d6dbcf13c0 msgr2=0x55d6dc0a0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 -- 172.21.15.59:0/231480698 shutdown_connections 2024-06-29T14:22:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.871+0000 7f71cdc09740 1 -- 172.21.15.59:0/231480698 wait complete. 2024-06-29T14:22:24.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.872+0000 7f71cdc09740 1 -- start start 2024-06-29T14:22:24.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.872+0000 7f71cdc09740 1 Processor -- start 2024-06-29T14:22:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.873+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcd43b0 0x55d6dc0e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.873+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.873+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcfd270 0x55d6dc0f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.873+0000 7f71cdc09740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d6dbfa1ae0 con 0x55d6dbcdc590 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.873+0000 7f71cdc09740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d6dc007c10 con 0x55d6dbcd43b0 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.873+0000 7f71cdc09740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d6dbd77620 con 0x55d6dbcfd270 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.873+0000 7f71cdc09740 10 client.0 _ll_register_callbacks cb 0x55d6dbfae480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35146/0 (socket says 172.21.15.59:35146) 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 -- 172.21.15.59:0/1323267782 learned_addr learned my addr 172.21.15.59:0/1323267782 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcd43b0 0x55d6dc0e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71c52e7640 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcfd270 0x55d6dc0f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 -- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcfd270 msgr2=0x55d6dc0f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcfd270 0x55d6dc0f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 -- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcd43b0 msgr2=0x55d6dc0e8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcd43b0 0x55d6dc0e8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 -- 172.21.15.59:0/1323267782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6dbd68850 con 0x55d6dbcdc590 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71c52e7640 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcfd270 0x55d6dc0f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71bffff640 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cc0f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f71a80612a0 tx=0x7f71a8060dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6243fdb1351fd7 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f719f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f719f7fe640 1 -- 172.21.15.59:0/1323267782 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a801f040 con 0x55d6dbcdc590 2024-06-29T14:22:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f719f7fe640 1 -- 172.21.15.59:0/1323267782 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f71a8069e30 con 0x55d6dbcdc590 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71cdc09740 1 -- 172.21.15.59:0/1323267782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b0042070 con 0x55d6dbcdc590 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f719f7fe640 1 -- 172.21.15.59:0/1323267782 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f71a80658a0 con 0x55d6dbcdc590 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71cdc09740 10 client.4935 fetch_fsmap learned FSMap version 9 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f71cdc09740 1 -- 172.21.15.59:0/1323267782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x55d6dbddf6b0 con 0x55d6dbcdc590 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.874+0000 7f719f7fe640 1 -- 172.21.15.59:0/1323267782 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71b0042070 con 0x55d6dbcdc590 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f719f7fe640 1 -- 172.21.15.59:0/1323267782 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 9) ==== 3510+0+0 (secure 0 0 0) 0x7f71a8096300 con 0x55d6dbcdc590 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f71cdc09740 10 client.4935 fetch_fsmap finished waiting for FSMap version 9 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f71cdc09740 10 client.4935 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f71cdc09740 20 client.4935 populate_metadata read hostname 'smithi059' 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f71cdc09740 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x55d6dbcbbe40 0x55d6dc0b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f71cdc09740 4 client.4935 mds_command: new command op to 4375 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f71cdc09740 1 -- 172.21.15.59:0/1323267782 --> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6dc016580 con 0x55d6dbcbbe40 2024-06-29T14:22:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x55d6dbcbbe40 0x55d6dc0b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x55d6dbcbbe40 0x55d6dc0b4c70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d552384ebc9a2c6 server_cookie=21bd6e1eb01c1639 in_seq=0 out_seq=0 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.875+0000 7f719f7fe640 10 client.4935 ms_handle_connect on v2:172.21.15.153:6834/2240732059 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.876+0000 7f719f7fe640 1 -- 172.21.15.59:0/1323267782 <== mds.0 v2:172.21.15.153:6834/2240732059 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d6dc016580 con 0x55d6dbcbbe40 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.876+0000 7f719f7fe640 10 client.4935 handle_command_reply: tid=0 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.876+0000 7f71cdc09740 1 client.4935 shutdown 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.876+0000 7f71cdc09740 1 -- 172.21.15.59:0/1323267782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 msgr2=0x55d6dc0cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.876+0000 7f71cdc09740 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cc0f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f71a80612a0 tx=0x7f71a8060dd0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 1 -- 172.21.15.59:0/1323267782 shutdown_connections 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x55d6dbcbbe40 0x55d6dc0b4c70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcfd270 0x55d6dc0f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcdc590 0x55d6dc0cc0f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f719f7fe640 0 client.4935 ms_handle_reset on v2:172.21.15.153:6834/2240732059 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 1 --2- 172.21.15.59:0/1323267782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcd43b0 0x55d6dc0e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 1 -- 172.21.15.59:0/1323267782 >> 172.21.15.59:0/1323267782 conn(0x55d6dbcf13c0 msgr2=0x55d6dc09cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 1 -- 172.21.15.59:0/1323267782 shutdown_connections 2024-06-29T14:22:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 1 -- 172.21.15.59:0/1323267782 wait complete. 2024-06-29T14:22:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.877+0000 7f71cdc09740 20 client.4935 trim_cache size 0 max 16384 2024-06-29T14:22:24.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.896+0000 7f71cdc09740 1 Processor -- start 2024-06-29T14:22:24.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.896+0000 7f71cdc09740 1 -- start start 2024-06-29T14:22:24.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc10bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 0x55d6dc0a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc0a1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71cdc09740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0201f50 con 0x55d6dbcd43b0 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71cdc09740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71c02012b0 con 0x55d6dbcfd270 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71cdc09740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f71c0201900 con 0x55d6dbcdc590 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71c4ae6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc10bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71c4ae6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc10bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35156/0 (socket says 172.21.15.59:35156) 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.897+0000 7f71c4ae6640 1 -- 172.21.15.59:0/2555382018 learned_addr learned my addr 172.21.15.59:0/2555382018 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71bffff640 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 0x55d6dc0a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71c52e7640 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc0a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71c4ae6640 1 -- 172.21.15.59:0/2555382018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 msgr2=0x55d6dc0a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 0x55d6dc0a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71c4ae6640 1 -- 172.21.15.59:0/2555382018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 msgr2=0x55d6dc0a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:24.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc0a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71c4ae6640 1 -- 172.21.15.59:0/2555382018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6dc0ca440 con 0x55d6dbcd43b0 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc10bb60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55d6dc0fb4d0 tx=0x7f71b8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dccfc5d1a64d17b1 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71b5ffb640 1 -- 172.21.15.59:0/2555382018 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b802e070 con 0x55d6dbcd43b0 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71b5ffb640 1 -- 172.21.15.59:0/2555382018 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71b8004e00 con 0x55d6dbcd43b0 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.898+0000 7f71b5ffb640 1 -- 172.21.15.59:0/2555382018 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8012e60 con 0x55d6dbcd43b0 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 -- 172.21.15.59:0/2555382018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 msgr2=0x55d6dc10bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc10bb60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55d6dc0fb4d0 tx=0x7f71b8015000 comp rx=0 tx=0).stop 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 -- 172.21.15.59:0/2555382018 shutdown_connections 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc0a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 0x55d6dc0a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 --2- 172.21.15.59:0/2555382018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc10bb60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 -- 172.21.15.59:0/2555382018 >> 172.21.15.59:0/2555382018 conn(0x55d6dbcf13c0 msgr2=0x55d6dc09cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:24.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 -- 172.21.15.59:0/2555382018 shutdown_connections 2024-06-29T14:22:24.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.899+0000 7f71cdc09740 1 -- 172.21.15.59:0/2555382018 wait complete. 2024-06-29T14:22:24.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.900+0000 7f71cdc09740 1 -- start start 2024-06-29T14:22:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.900+0000 7f71cdc09740 1 Processor -- start 2024-06-29T14:22:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc09d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 0x55d6dc164dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71cdc09740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc09dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71cdc09740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d6dc0b5d70 con 0x55d6dbcd43b0 2024-06-29T14:22:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71cdc09740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d6dc0cd320 con 0x55d6dbcfd270 2024-06-29T14:22:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71cdc09740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d6dc0fb070 con 0x55d6dbcdc590 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71cdc09740 10 client.0 _ll_register_callbacks cb 0x7f71b0036070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71c4ae6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc09d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71c4ae6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc09d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35158/0 (socket says 172.21.15.59:35158) 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71c4ae6640 1 -- 172.21.15.59:0/3812788999 learned_addr learned my addr 172.21.15.59:0/3812788999 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71bffff640 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 0x55d6dc164dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.901+0000 7f71c52e7640 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc09dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71c4ae6640 1 -- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 msgr2=0x55d6dc164dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 0x55d6dc164dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71c4ae6640 1 -- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 msgr2=0x55d6dc09dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc09dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71c4ae6640 1 -- 172.21.15.59:0/3812788999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6dc0ee3f0 con 0x55d6dbcd43b0 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71c4ae6640 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc09d210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f71c00b7620 tx=0x7f71b8001830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482eec64e08954d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:24.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f719f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f719f7fe640 1 -- 172.21.15.59:0/3812788999 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8012e60 con 0x55d6dbcd43b0 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f719f7fe640 1 -- 172.21.15.59:0/3812788999 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f71b8003cc0 con 0x55d6dbcd43b0 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71cdc09740 1 -- 172.21.15.59:0/3812788999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b8004e00 con 0x55d6dbcd43b0 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f719f7fe640 1 -- 172.21.15.59:0/3812788999 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f71b8040050 con 0x55d6dbcd43b0 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71cdc09740 10 client.4941 fetch_fsmap learned FSMap version 9 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f71cdc09740 1 -- 172.21.15.59:0/3812788999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x55d6dc09e550 con 0x55d6dbcd43b0 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f719f7fe640 1 -- 172.21.15.59:0/3812788999 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71b8004e00 con 0x55d6dbcd43b0 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.902+0000 7f719f7fe640 1 -- 172.21.15.59:0/3812788999 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 9) ==== 3510+0+0 (secure 0 0 0) 0x7f71b8043020 con 0x55d6dbcd43b0 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f71cdc09740 10 client.4941 fetch_fsmap finished waiting for FSMap version 9 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f71cdc09740 10 client.4941 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f71cdc09740 20 client.4941 populate_metadata read hostname 'smithi059' 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f71cdc09740 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x55d6dc0a1320 0x55d6dc10a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:24.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f71cdc09740 4 client.4941 mds_command: new command op to 4375 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f71cdc09740 1 -- 172.21.15.59:0/3812788999 --> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] -- command(tid 0: {"prefix": "status"}) -- 0x55d6dc012020 con 0x55d6dc0a1320 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f71bffff640 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x55d6dc0a1320 0x55d6dc10a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f71bffff640 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x55d6dc0a1320 0x55d6dc10a4a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e313496a022831b3 server_cookie=4f9d7acd995e002a in_seq=0 out_seq=0 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f719f7fe640 10 client.4941 ms_handle_connect on v2:172.21.15.153:6834/2240732059 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f719f7fe640 1 -- 172.21.15.59:0/3812788999 <== mds.0 v2:172.21.15.153:6834/2240732059 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55d6dc012020 con 0x55d6dc0a1320 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.903+0000 7f719f7fe640 10 client.4941 handle_command_reply: tid=0 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 client.4941 shutdown 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 -- 172.21.15.59:0/3812788999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 msgr2=0x55d6dc09d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc09d210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f71c00b7620 tx=0x7f71b8001830 comp rx=0 tx=0).stop 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 -- 172.21.15.59:0/3812788999 shutdown_connections 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:6834/2240732059,v1:172.21.15.153:6836/2240732059] conn(0x55d6dc0a1320 0x55d6dc10a4a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d6dbcfd270 0x55d6dc09dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d6dbcdc590 0x55d6dc164dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 --2- 172.21.15.59:0/3812788999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6dbcd43b0 0x55d6dc09d210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f719f7fe640 0 client.4941 ms_handle_reset on v2:172.21.15.153:6834/2240732059 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 -- 172.21.15.59:0/3812788999 >> 172.21.15.59:0/3812788999 conn(0x55d6dbcf13c0 msgr2=0x55d6dc0e9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 -- 172.21.15.59:0/3812788999 shutdown_connections 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 1 -- 172.21.15.59:0/3812788999 wait complete. 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71cdc09740 20 client.4941 trim_cache size 0 max 16384 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:22:24.908 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 4375, 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 29.642147734000002, 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 9, 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 26, 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 26, 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 36.677054003999999 2024-06-29T14:22:24.909 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:22:24.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71c6d4a640 1 -- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71a0076300 msgr2=0x7f71a0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71c6d4a640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71a0076300 0x7f71a0078810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f71b8001d00 tx=0x7f71b8006720 comp rx=0 tx=0).stop 2024-06-29T14:22:24.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71c6d4a640 1 -- 172.21.15.59:0/2517871669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71c0165aa0 msgr2=0x7f71c0165eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:24.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.904+0000 7f71c6d4a640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71c0165aa0 0x7f71c0165eb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f71a8003ea0 tx=0x7f71a8003f90 comp rx=0 tx=0).stop 2024-06-29T14:22:24.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.905+0000 7f71c6d4a640 1 -- 172.21.15.59:0/2517871669 shutdown_connections 2024-06-29T14:22:24.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.905+0000 7f71c6d4a640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71a0076300 0x7f71a0078810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.905+0000 7f71c6d4a640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71c015c0a0 0x7f71c015c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.905+0000 7f71c6d4a640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71c0165aa0 0x7f71c0165eb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.905+0000 7f71c6d4a640 1 --2- 172.21.15.59:0/2517871669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71ac004d40 0x7f71c015b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:24.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.905+0000 7f71c6d4a640 1 -- 172.21.15.59:0/2517871669 >> 172.21.15.59:0/2517871669 conn(0x7f71c00b5610 msgr2=0x7f71c00c71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:24.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.905+0000 7f71c6d4a640 1 -- 172.21.15.59:0/2517871669 shutdown_connections 2024-06-29T14:22:24.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:24.905+0000 7f71c6d4a640 1 -- 172.21.15.59:0/2517871669 wait complete. 2024-06-29T14:22:24.914 INFO:teuthology.run_tasks:Running task sequential... 2024-06-29T14:22:24.924 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-06-29T14:22:24.924 INFO:teuthology.task.sequential:In sequential, running task print... 2024-06-29T14:22:24.924 INFO:teuthology.task.print:Enabling mgr modules 2024-06-29T14:22:24.924 INFO:teuthology.run_tasks:Running task kclient... 2024-06-29T14:22:24.963 INFO:tasks.kclient:Mounting kernel clients... 2024-06-29T14:22:24.963 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-06-29T14:22:24.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:24.963 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:22:24.979 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-06-29T14:22:24.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:24.980 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-29T14:22:25.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:25.045 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2024-06-29T14:22:25.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:25.156 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:22:25.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:25.172 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2024-06-29T14:22:25.242 INFO:teuthology.orchestra.run.smithi059.stderr:Cannot find device "ceph-brx" 2024-06-29T14:22:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:22:25.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:25.244 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-06-29T14:22:25.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:25.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-06-29T14:22:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:Cannot find device "ceph-brx" 2024-06-29T14:22:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:22:25.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:25.338 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-06-29T14:22:25.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:25.397 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-06-29T14:22:25.465 INFO:teuthology.orchestra.run.smithi153.stderr:Cannot find device "ceph-brx" 2024-06-29T14:22:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:22:25.467 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-06-29T14:22:25.467 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:22:25.467 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:22:25.467 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:22:25.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.588+0000 7f6146081640 1 -- 172.21.15.59:0/4088885013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6128004800 msgr2=0x7f6128004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:25.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.588+0000 7f6146081640 1 --2- 172.21.15.59:0/4088885013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6128004800 0x7f6128004c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6134009b50 tx=0x7f6128005830 comp rx=0 tx=0).stop 2024-06-29T14:22:25.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.588+0000 7f6146081640 1 -- 172.21.15.59:0/4088885013 shutdown_connections 2024-06-29T14:22:25.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.588+0000 7f6146081640 1 --2- 172.21.15.59:0/4088885013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6128004800 0x7f6128004c10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:25.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.588+0000 7f6146081640 1 -- 172.21.15.59:0/4088885013 >> 172.21.15.59:0/4088885013 conn(0x7f61400b24c0 msgr2=0x7f61400cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:25.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.588+0000 7f6146081640 1 -- 172.21.15.59:0/4088885013 shutdown_connections 2024-06-29T14:22:25.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.588+0000 7f6146081640 1 -- 172.21.15.59:0/4088885013 wait complete. 2024-06-29T14:22:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.590+0000 7f6146081640 1 Processor -- start 2024-06-29T14:22:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.590+0000 7f6146081640 1 -- start start 2024-06-29T14:22:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.591+0000 7f6146081640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128004800 0x7f6140143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.591+0000 7f6146081640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61401f7cc0 0x7f61401f80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.591+0000 7f6146081640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61400b36f0 0x7f61400b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.591+0000 7f6146081640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f614015f5d0 con 0x7f61400b36f0 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.591+0000 7f6146081640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f614015f450 con 0x7f61401f7cc0 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.591+0000 7f6146081640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f614015f750 con 0x7f6128004800 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.591+0000 7f613ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61400b36f0 0x7f61400b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.591+0000 7f613ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61400b36f0 0x7f61400b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35176/0 (socket says 172.21.15.59:35176) 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613ffff640 1 -- 172.21.15.59:0/3385770979 learned_addr learned my addr 172.21.15.59:0/3385770979 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613effd640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61401f7cc0 0x7f61401f80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613f7fe640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128004800 0x7f6140143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613ffff640 1 -- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128004800 msgr2=0x7f6140143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613ffff640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128004800 0x7f6140143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613ffff640 1 -- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61401f7cc0 msgr2=0x7f61401f80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613ffff640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61401f7cc0 0x7f61401f80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613ffff640 1 -- 172.21.15.59:0/3385770979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61401422b0 con 0x7f61400b36f0 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613effd640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61401f7cc0 0x7f61401f80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613f7fe640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128004800 0x7f6140143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f613ffff640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61400b36f0 0x7f61400b3b00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f61400744f0 tx=0x7f61380064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f6127fff640 1 -- 172.21.15.59:0/3385770979 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138011900 con 0x7f61400b36f0 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f6127fff640 1 -- 172.21.15.59:0/3385770979 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6138018070 con 0x7f61400b36f0 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61401f8610 con 0x7f61400b36f0 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f6127fff640 1 -- 172.21.15.59:0/3385770979 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138011d60 con 0x7f61400b36f0 2024-06-29T14:22:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.592+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6140141910 con 0x7f61400b36f0 2024-06-29T14:22:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.593+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614015b620 con 0x7f61400b36f0 2024-06-29T14:22:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.594+0000 7f6127fff640 1 -- 172.21.15.59:0/3385770979 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f613801d460 con 0x7f61400b36f0 2024-06-29T14:22:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.594+0000 7f6127fff640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f612c076300 0x7f612c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.595+0000 7f6127fff640 1 -- 172.21.15.59:0/3385770979 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6138098210 con 0x7f61400b36f0 2024-06-29T14:22:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.595+0000 7f613f7fe640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f612c076300 0x7f612c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.595+0000 7f613f7fe640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f612c076300 0x7f612c078810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6130000c00 tx=0x7f6130006170 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:25.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.599+0000 7f6127fff640 1 -- 172.21.15.59:0/3385770979 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f61401f8610 con 0x7f61400b36f0 2024-06-29T14:22:25.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.829+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f61400735f0 con 0x7f61400b36f0 2024-06-29T14:22:25.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.829+0000 7f6127fff640 1 -- 172.21.15.59:0/3385770979 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f61401422b0 con 0x7f61400b36f0 2024-06-29T14:22:25.828 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-29T14:22:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f612c076300 msgr2=0x7f612c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f612c076300 0x7f612c078810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6130000c00 tx=0x7f6130006170 comp rx=0 tx=0).stop 2024-06-29T14:22:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61400b36f0 msgr2=0x7f61400b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61400b36f0 0x7f61400b3b00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f61400744f0 tx=0x7f61380064f0 comp rx=0 tx=0).stop 2024-06-29T14:22:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 shutdown_connections 2024-06-29T14:22:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f612c076300 0x7f612c078810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61400b36f0 0x7f61400b3b00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61401f7cc0 0x7f61401f80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 --2- 172.21.15.59:0/3385770979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6128004800 0x7f6140143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 >> 172.21.15.59:0/3385770979 conn(0x7f61400b24c0 msgr2=0x7f614015e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 shutdown_connections 2024-06-29T14:22:25.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25.831+0000 7f6146081640 1 -- 172.21.15.59:0/3385770979 wait complete. 2024-06-29T14:22:25.841 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-29T14:22:25.841 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:22:25.841 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:22:25.841 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:22:25.842 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:22:25.842 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-29T14:22:25.842 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:22:25.842 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:22:25.842 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:22:25.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:25.842 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:22:25.858 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:25.858 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:25.859 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:25.859 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:25.859 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:25.859 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:25.859 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:25.859 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:25.859 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42705sec preferred_lft 42705sec 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:25.860 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:22:25.861 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-29T14:22:25.861 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:25.861 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:25.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add name ceph-brx type bridge 2024-06-29T14:22:25.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr flush dev ceph-brx 2024-06-29T14:22:25.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set ceph-brx up 2024-06-29T14:22:25.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-29T14:22:25.861 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:25.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:26.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:26.001 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:22:26.026 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:22:26.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:26.028 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:22:26.085 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:22:26.085 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:22:26.085 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:22:26.085 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:22:26.086 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:22:26.086 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:26.086 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:26.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:22:26.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:22:26.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:22:26.086 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:26.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:26.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:26.216 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:22:26.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:26.232 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list-id 2024-06-29T14:22:26.288 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:26.288 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:26.288 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:22:26.288 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-06-29T14:22:26.288 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:26.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:26.408 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-06-29T14:22:26.408 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:26.408 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:26.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-06-29T14:22:26.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-29T14:22:26.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-06-29T14:22:26.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-06-29T14:22:26.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-06-29T14:22:26.409 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:26.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:26.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:26.526 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:26.526 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:26.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:22:26.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set dev brx.0 master ceph-brx 2024-06-29T14:22:26.526 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:26.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:26.607 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:26.607 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:22:26.623 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:22:26.623 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:26.623 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:22:26.679 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:22:26.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:26.680 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:22:26.775 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:22:26.776 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:22:26.776 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:22:26.897 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:22:26.897 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:22:26.897 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:22:26.897 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:22:26.897 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:22:26.903 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:22:26.903 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:22:26.903 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:22:26.903 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:22:26.903 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:22:26.903 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:22:26.903 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:22:26.903 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:22:26.904 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:22:26.904 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:22:26.904 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:22:26.904 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:22:26.904 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:22:26.905 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:22:26.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:26.906 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:22:26.931 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:27.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:27.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:27.030 INFO:teuthology.orchestra.run.smithi059.stdout:client4670 2024-06-29T14:22:27.031 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:27.031 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:27.047 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:27.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:27.144 INFO:teuthology.orchestra.run.smithi059.stdout:client4670 2024-06-29T14:22:27.145 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:27.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:27.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:27.203 INFO:teuthology.orchestra.run.smithi059.stdout:client4670 2024-06-29T14:22:27.203 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/150b8fd8-1bf5-44f0-b8d0-0083a48c0071.client4670/status') 2024-06-29T14:22:27.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:27.264 INFO:teuthology.orchestra.run.smithi059.stdout:instance: client.4670 (3)172.21.15.59:0/2233389723 2024-06-29T14:22:27.264 INFO:teuthology.orchestra.run.smithi059.stdout:blocklisted: false 2024-06-29T14:22:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records in 2024-06-29T14:22:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records out 2024-06-29T14:22:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:70 bytes copied, 2.3571e-05 s, 3.0 MB/s 2024-06-29T14:22:27.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:27.271 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-06-29T14:22:27.272 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:22:27.272 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-06-29T14:22:27.272 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:22:27.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.392+0000 7f2a3a119640 1 -- 172.21.15.59:0/4159963874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a3415c2f0 msgr2=0x7f2a3415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:27.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.392+0000 7f2a3a119640 1 --2- 172.21.15.59:0/4159963874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a3415c2f0 0x7f2a3415c700 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c005fd0 tx=0x7f2a2c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.392+0000 7f2a3a119640 1 -- 172.21.15.59:0/4159963874 shutdown_connections 2024-06-29T14:22:27.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.392+0000 7f2a3a119640 1 --2- 172.21.15.59:0/4159963874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a3415c2f0 0x7f2a3415c700 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.392+0000 7f2a3a119640 1 --2- 172.21.15.59:0/4159963874 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a340b3740 0x7f2a340b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.392+0000 7f2a3a119640 1 -- 172.21.15.59:0/4159963874 >> 172.21.15.59:0/4159963874 conn(0x7f2a3406c950 msgr2=0x7f2a34073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:27.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.392+0000 7f2a3a119640 1 -- 172.21.15.59:0/4159963874 shutdown_connections 2024-06-29T14:22:27.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.392+0000 7f2a3a119640 1 -- 172.21.15.59:0/4159963874 wait complete. 2024-06-29T14:22:27.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.394+0000 7f2a3a119640 1 Processor -- start 2024-06-29T14:22:27.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.394+0000 7f2a3a119640 1 -- start start 2024-06-29T14:22:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a3a119640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a340b3740 0x7f2a340ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a3a119640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a3415c2f0 0x7f2a341f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a3a119640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a34137c20 0x7f2a341f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a3a119640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2a340b5fd0 con 0x7f2a340b3740 2024-06-29T14:22:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a3a119640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2a340b5e50 con 0x7f2a3415c2f0 2024-06-29T14:22:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a3a119640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2a340b6150 con 0x7f2a34137c20 2024-06-29T14:22:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a337fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a340b3740 0x7f2a340ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a337fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a340b3740 0x7f2a340ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35206/0 (socket says 172.21.15.59:35206) 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.395+0000 7f2a337fe640 1 -- 172.21.15.59:0/571670329 learned_addr learned my addr 172.21.15.59:0/571670329 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a33fff640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a34137c20 0x7f2a341f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a32ffd640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a3415c2f0 0x7f2a341f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a337fe640 1 -- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a34137c20 msgr2=0x7f2a341f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a337fe640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a34137c20 0x7f2a341f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a337fe640 1 -- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a3415c2f0 msgr2=0x7f2a341f8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a337fe640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a3415c2f0 0x7f2a341f8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a337fe640 1 -- 172.21.15.59:0/571670329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3413afa0 con 0x7f2a340b3740 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a337fe640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a340b3740 0x7f2a340ce4a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400c3a0 tx=0x7f2a2400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a1bfff640 1 -- 172.21.15.59:0/571670329 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24011790 con 0x7f2a340b3740 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a1bfff640 1 -- 172.21.15.59:0/571670329 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a24017070 con 0x7f2a340b3740 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a1bfff640 1 -- 172.21.15.59:0/571670329 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24011cc0 con 0x7f2a340b3740 2024-06-29T14:22:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3413a9e0 con 0x7f2a340b3740 2024-06-29T14:22:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.396+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a34138db0 con 0x7f2a340b3740 2024-06-29T14:22:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.397+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a340b1f10 con 0x7f2a340b3740 2024-06-29T14:22:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.398+0000 7f2a1bfff640 1 -- 172.21.15.59:0/571670329 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2a2401c460 con 0x7f2a340b3740 2024-06-29T14:22:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.398+0000 7f2a1bfff640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2a200762d0 0x7f2a200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.399+0000 7f2a1bfff640 1 -- 172.21.15.59:0/571670329 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2a240978c0 con 0x7f2a340b3740 2024-06-29T14:22:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.399+0000 7f2a32ffd640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2a200762d0 0x7f2a200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.399+0000 7f2a32ffd640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2a200762d0 0x7f2a200787e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2a28005fd0 tx=0x7f2a28005850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:27.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.403+0000 7f2a1bfff640 1 -- 172.21.15.59:0/571670329 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2a3413afa0 con 0x7f2a340b3740 2024-06-29T14:22:27.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.632+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2a340d2690 con 0x7f2a340b3740 2024-06-29T14:22:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.633+0000 7f2a1bfff640 1 -- 172.21.15.59:0/571670329 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f2a24060ee0 con 0x7f2a340b3740 2024-06-29T14:22:27.632 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2a200762d0 msgr2=0x7f2a200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2a200762d0 0x7f2a200787e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2a28005fd0 tx=0x7f2a28005850 comp rx=0 tx=0).stop 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a340b3740 msgr2=0x7f2a340ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a340b3740 0x7f2a340ce4a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400c3a0 tx=0x7f2a2400c860 comp rx=0 tx=0).stop 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 shutdown_connections 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2a200762d0 0x7f2a200787e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a34137c20 0x7f2a341f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a3415c2f0 0x7f2a341f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 --2- 172.21.15.59:0/571670329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a340b3740 0x7f2a340ce4a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:27.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 >> 172.21.15.59:0/571670329 conn(0x7f2a3406c950 msgr2=0x7f2a340d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:27.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 shutdown_connections 2024-06-29T14:22:27.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27.635+0000 7f2a3a119640 1 -- 172.21.15.59:0/571670329 wait complete. 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:22:27.645 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-06-29T14:22:27.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:27.646 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:22:27.661 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:27.661 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:22:27.662 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42703sec preferred_lft 42703sec 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:27.663 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link tentative 2024-06-29T14:22:27.664 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:27.664 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:22:27.664 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:22:27.664 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link tentative 2024-06-29T14:22:27.664 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:27.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:27.664 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:22:27.726 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:22:27.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:27.727 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:22:27.784 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:22:27.785 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:22:27.785 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:22:27.785 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:22:27.785 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:22:27.785 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:27.785 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:27.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:22:27.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:22:27.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:22:27.786 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:27.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:27.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:27.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:27.915 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:22:27.931 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:22:27.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:27.932 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list-id 2024-06-29T14:22:27.988 INFO:teuthology.orchestra.run.smithi059.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-06-29T14:22:27.988 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:27.988 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:27.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:22:27.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-06-29T14:22:27.988 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:28.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:28.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:28.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:28.107 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-06-29T14:22:28.132 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 32:fa:6b:e1:ab:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-06-29T14:22:28.133 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:28.134 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::30fa:6bff:fee1:ab2e/64 scope link 2024-06-29T14:22:28.134 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:28.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:28.135 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-06-29T14:22:28.202 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:28.202 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:28.202 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:28.202 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:28.203 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:28.203 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:28.203 INFO:teuthology.orchestra.run.smithi059.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:22:28.203 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 32:fa:6b:e1:ab:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-06-29T14:22:28.203 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-06-29T14:22:28.203 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:28.203 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::30fa:6bff:fee1:ab2e/64 scope link 2024-06-29T14:22:28.203 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:28.204 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-06-29T14:22:28.204 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:28.204 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:28.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-06-29T14:22:28.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-06-29T14:22:28.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-06-29T14:22:28.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-06-29T14:22:28.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-06-29T14:22:28.204 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:28.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:28.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:28.362 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:28.362 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:22:28.362 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.1 up 2024-06-29T14:22:28.362 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set dev brx.1 master ceph-brx 2024-06-29T14:22:28.362 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:22:28.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:28.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:28.444 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:28.444 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-06-29T14:22:28.460 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-06-29T14:22:28.460 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:28.460 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-06-29T14:22:28.515 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:22:28.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:28.516 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-06-29T14:22:28.611 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-06-29T14:22:28.611 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-06-29T14:22:28.611 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:22:28.622 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-06-29T14:22:28.622 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-06-29T14:22:28.622 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:22:28.622 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:22:28.622 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:22:28.628 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-06-29T14:22:28.628 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-06-29T14:22:28.629 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:22:28.629 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:22:28.629 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:22:28.629 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:22:28.629 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:22:28.629 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:22:28.630 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-06-29T14:22:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:22:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:22:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:22:28.630 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:22:28.631 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:22:28.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:28.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-06-29T14:22:28.658 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:28.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:28.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:28.756 INFO:teuthology.orchestra.run.smithi059.stdout:client4959 2024-06-29T14:22:28.757 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:28.757 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:28.773 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:28.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:28.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:28.873 INFO:teuthology.orchestra.run.smithi059.stdout:client4959 2024-06-29T14:22:28.873 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:28.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:28.931 INFO:teuthology.orchestra.run.smithi059.stdout:client4959 2024-06-29T14:22:28.932 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/150b8fd8-1bf5-44f0-b8d0-0083a48c0071.client4959/status') 2024-06-29T14:22:28.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:28.992 INFO:teuthology.orchestra.run.smithi059.stdout:instance: client.4959 (3)192.168.0.2:0/3118640377 2024-06-29T14:22:28.992 INFO:teuthology.orchestra.run.smithi059.stdout:blocklisted: false 2024-06-29T14:22:28.992 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records in 2024-06-29T14:22:28.992 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records out 2024-06-29T14:22:28.993 INFO:teuthology.orchestra.run.smithi059.stderr:69 bytes copied, 2.2893e-05 s, 3.0 MB/s 2024-06-29T14:22:28.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:28.999 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-06-29T14:22:28.999 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:22:28.999 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-06-29T14:22:29.000 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:22:29.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.119+0000 7f0788065640 1 -- 172.21.15.59:0/3048356293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0768004800 msgr2=0x7f0768004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:29.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.119+0000 7f0788065640 1 --2- 172.21.15.59:0/3048356293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0768004800 0x7f0768004c10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0774006fc0 tx=0x7f0768005830 comp rx=0 tx=0).stop 2024-06-29T14:22:29.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.119+0000 7f0788065640 1 -- 172.21.15.59:0/3048356293 shutdown_connections 2024-06-29T14:22:29.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.119+0000 7f0788065640 1 --2- 172.21.15.59:0/3048356293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0768004800 0x7f0768004c10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.119+0000 7f0788065640 1 -- 172.21.15.59:0/3048356293 >> 172.21.15.59:0/3048356293 conn(0x7f078006c950 msgr2=0x7f0780073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:29.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.119+0000 7f0788065640 1 -- 172.21.15.59:0/3048356293 shutdown_connections 2024-06-29T14:22:29.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.119+0000 7f0788065640 1 -- 172.21.15.59:0/3048356293 wait complete. 2024-06-29T14:22:29.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.121+0000 7f0788065640 1 Processor -- start 2024-06-29T14:22:29.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.121+0000 7f0788065640 1 -- start start 2024-06-29T14:22:29.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0788065640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0768004800 0x7f078006dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0788065640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f078013e440 0x7f078013b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0788065640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0780159870 0x7f0780139c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0788065640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f07800cf9f0 con 0x7f0780159870 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0788065640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f07800cf870 con 0x7f0768004800 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0788065640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f07800cfb70 con 0x7f078013e440 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0780159870 0x7f0780139c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0780159870 0x7f0780139c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35226/0 (socket says 172.21.15.59:35226) 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 -- 172.21.15.59:0/1295882434 learned_addr learned my addr 172.21.15.59:0/1295882434 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0785e01640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0768004800 0x7f078006dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0785600640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f078013e440 0x7f078013b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 -- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f078013e440 msgr2=0x7f078013b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f078013e440 0x7f078013b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 -- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0768004800 msgr2=0x7f078006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0768004800 0x7f078006dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 -- 172.21.15.59:0/1295882434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0780138e70 con 0x7f0780159870 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0785600640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f078013e440 0x7f078013b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0786602640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0780159870 0x7f0780139c30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f07800b55a0 tx=0x7f0778001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f077e7fc640 1 -- 172.21.15.59:0/1295882434 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770009ad0 con 0x7f0780159870 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f077e7fc640 1 -- 172.21.15.59:0/1295882434 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0778016070 con 0x7f0780159870 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0780139150 con 0x7f0780159870 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f077e7fc640 1 -- 172.21.15.59:0/1295882434 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778010b60 con 0x7f0780159870 2024-06-29T14:22:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.123+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078006e140 con 0x7f0780159870 2024-06-29T14:22:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.124+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07800cfd50 con 0x7f0780159870 2024-06-29T14:22:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.125+0000 7f077e7fc640 1 -- 172.21.15.59:0/1295882434 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0778010d00 con 0x7f0780159870 2024-06-29T14:22:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.126+0000 7f077e7fc640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f076c0762d0 0x7f076c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.126+0000 7f077e7fc640 1 -- 172.21.15.59:0/1295882434 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0778097fa0 con 0x7f0780159870 2024-06-29T14:22:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.126+0000 7f0785e01640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f076c0762d0 0x7f076c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.127+0000 7f0785e01640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f076c0762d0 0x7f076c0787e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0770009aa0 tx=0x7f0770009500 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:29.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.130+0000 7f077e7fc640 1 -- 172.21.15.59:0/1295882434 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0778061650 con 0x7f0780159870 2024-06-29T14:22:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.359+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f07800b1f10 con 0x7f0780159870 2024-06-29T14:22:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.360+0000 7f077e7fc640 1 -- 172.21.15.59:0/1295882434 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f0778060f50 con 0x7f0780159870 2024-06-29T14:22:29.359 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-29T14:22:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f076c0762d0 msgr2=0x7f076c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f076c0762d0 0x7f076c0787e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0770009aa0 tx=0x7f0770009500 comp rx=0 tx=0).stop 2024-06-29T14:22:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0780159870 msgr2=0x7f0780139c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0780159870 0x7f0780139c30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f07800b55a0 tx=0x7f0778001630 comp rx=0 tx=0).stop 2024-06-29T14:22:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 shutdown_connections 2024-06-29T14:22:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f076c0762d0 0x7f076c0787e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0780159870 0x7f0780139c30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f078013e440 0x7f078013b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 --2- 172.21.15.59:0/1295882434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0768004800 0x7f078006dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 >> 172.21.15.59:0/1295882434 conn(0x7f078006c950 msgr2=0x7f0780143020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 shutdown_connections 2024-06-29T14:22:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:29.362+0000 7f0788065640 1 -- 172.21.15.59:0/1295882434 wait complete. 2024-06-29T14:22:29.372 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-29T14:22:29.372 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:22:29.372 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi153.front.sepia.ceph.com 2024-06-29T14:22:29.372 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-06-29T14:22:29.372 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-06-29T14:22:29.372 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-29T14:22:29.373 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:22:29.373 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:22:29.373 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-06-29T14:22:29.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:29.373 DEBUG:teuthology.orchestra.run.smithi153:> ip addr 2024-06-29T14:22:29.392 INFO:teuthology.orchestra.run.smithi153.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:29.392 INFO:teuthology.orchestra.run.smithi153.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:29.392 INFO:teuthology.orchestra.run.smithi153.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b2 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f0 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b3 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f1 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:ea brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:29.393 INFO:teuthology.orchestra.run.smithi153.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:22:29.394 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:eb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:29.394 INFO:teuthology.orchestra.run.smithi153.stdout: inet 172.21.15.153/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:22:29.394 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft 42708sec preferred_lft 42708sec 2024-06-29T14:22:29.394 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::ec4:7aff:fe88:6deb/64 scope link 2024-06-29T14:22:29.394 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:29.394 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-29T14:22:29.394 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:29.394 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:29.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add name ceph-brx type bridge 2024-06-29T14:22:29.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr flush dev ceph-brx 2024-06-29T14:22:29.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set ceph-brx up 2024-06-29T14:22:29.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-29T14:22:29.395 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:29.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:29.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:29.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:29.929 DEBUG:teuthology.orchestra.run.smithi153:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:22:29.955 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-06-29T14:22:29.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:29.957 DEBUG:teuthology.orchestra.run.smithi153:> route 2024-06-29T14:22:30.027 INFO:teuthology.orchestra.run.smithi153.stdout:Kernel IP routing table 2024-06-29T14:22:30.027 INFO:teuthology.orchestra.run.smithi153.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:22:30.027 INFO:teuthology.orchestra.run.smithi153.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:22:30.027 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:22:30.027 INFO:teuthology.orchestra.run.smithi153.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:22:30.027 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:30.027 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:30.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:22:30.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:22:30.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:22:30.028 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:30.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:30.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:30.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:30.314 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-06-29T14:22:30.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:30.329 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list-id 2024-06-29T14:22:30.385 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:30.385 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:30.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-06-29T14:22:30.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-06-29T14:22:30.385 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:30.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:30.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:30.503 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-06-29T14:22:30.503 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:30.503 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:30.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-06-29T14:22:30.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-29T14:22:30.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-06-29T14:22:30.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-06-29T14:22:30.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-06-29T14:22:30.504 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:30.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:30.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:30.635 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:30.635 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:30.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set brx.0 up 2024-06-29T14:22:30.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set dev brx.0 master ceph-brx 2024-06-29T14:22:30.636 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:30.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:30.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:30.713 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:30.713 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-06-29T14:22:30.728 INFO:teuthology.orchestra.run.smithi153.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-06-29T14:22:30.728 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:30.728 DEBUG:teuthology.orchestra.run.smithi153:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-06-29T14:22:30.783 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:22:30.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:30.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-06-29T14:22:30.898 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-06-29T14:22:30.898 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-06-29T14:22:30.898 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-06-29T14:22:31.022 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-06-29T14:22:31.022 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-06-29T14:22:31.022 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-06-29T14:22:31.022 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:22:31.022 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:22:31.028 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-06-29T14:22:31.028 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-06-29T14:22:31.028 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-06-29T14:22:31.028 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:22:31.028 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:22:31.028 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:22:31.029 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:22:31.029 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:22:31.029 INFO:teuthology.orchestra.run.smithi153.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-06-29T14:22:31.029 INFO:teuthology.orchestra.run.smithi153.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:22:31.029 INFO:teuthology.orchestra.run.smithi153.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:22:31.029 INFO:teuthology.orchestra.run.smithi153.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:22:31.030 INFO:teuthology.orchestra.run.smithi153.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:22:31.031 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:22:31.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:31.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-06-29T14:22:31.056 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:31.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:31.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:31.153 INFO:teuthology.orchestra.run.smithi153.stdout:client4968 2024-06-29T14:22:31.153 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:22:31.153 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:31.169 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:31.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:31.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:31.265 INFO:teuthology.orchestra.run.smithi153.stdout:client4968 2024-06-29T14:22:31.266 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:31.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:31.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:31.323 INFO:teuthology.orchestra.run.smithi153.stdout:client4968 2024-06-29T14:22:31.324 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/150b8fd8-1bf5-44f0-b8d0-0083a48c0071.client4968/status') 2024-06-29T14:22:31.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:31.383 INFO:teuthology.orchestra.run.smithi153.stdout:instance: client.4968 (3)172.21.15.153:0/3240643842 2024-06-29T14:22:31.383 INFO:teuthology.orchestra.run.smithi153.stdout:blocklisted: false 2024-06-29T14:22:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:0+1 records in 2024-06-29T14:22:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:0+1 records out 2024-06-29T14:22:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:71 bytes copied, 2.2707e-05 s, 3.1 MB/s 2024-06-29T14:22:31.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:31.390 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-06-29T14:22:31.390 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:22:31.390 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-06-29T14:22:31.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:22:31.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.524+0000 7f588d3e0640 1 -- 172.21.15.59:0/3949240999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880b2fa0 msgr2=0x7f58880d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:31.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.524+0000 7f588d3e0640 1 --2- 172.21.15.59:0/3949240999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880b2fa0 0x7f58880d1520 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f58880cf310 tx=0x7f587c02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.524+0000 7f588d3e0640 1 -- 172.21.15.59:0/3949240999 shutdown_connections 2024-06-29T14:22:31.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.524+0000 7f588d3e0640 1 --2- 172.21.15.59:0/3949240999 >> v2:172.21.15.153:3300/0 conn(0x7f58880d1a60 0x7f58880b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.524+0000 7f588d3e0640 1 --2- 172.21.15.59:0/3949240999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880b2fa0 0x7f58880d1520 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.524+0000 7f588d3e0640 1 -- 172.21.15.59:0/3949240999 >> 172.21.15.59:0/3949240999 conn(0x7f5888073e10 msgr2=0x7f5888074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:31.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.524+0000 7f588d3e0640 1 -- 172.21.15.59:0/3949240999 shutdown_connections 2024-06-29T14:22:31.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.524+0000 7f588d3e0640 1 -- 172.21.15.59:0/3949240999 wait complete. 2024-06-29T14:22:31.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.526+0000 7f588d3e0640 1 Processor -- start 2024-06-29T14:22:31.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.526+0000 7f588d3e0640 1 -- start start 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f588d3e0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58880b2270 0x7f5888144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f588d3e0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58880b2fa0 0x7f5888146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f588d3e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880d1a60 0x7f5888146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f588d3e0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f58880b6400 con 0x7f58880d1a60 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f588d3e0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58880b6280 con 0x7f58880b2270 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f588d3e0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f58880b6580 con 0x7f58880b2fa0 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880d1a60 0x7f5888146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880d1a60 0x7f5888146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35246/0 (socket says 172.21.15.59:35246) 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 -- 172.21.15.59:0/2595049160 learned_addr learned my addr 172.21.15.59:0/2595049160 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:31.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f5886ffd640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58880b2270 0x7f5888144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58867fc640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58880b2fa0 0x7f5888146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 -- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58880b2fa0 msgr2=0x7f5888146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58880b2fa0 0x7f5888146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 -- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58880b2270 msgr2=0x7f5888144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58880b2270 0x7f5888144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 -- 172.21.15.59:0/2595049160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58881453b0 con 0x7f58880d1a60 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f58877fe640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880d1a60 0x7f5888146990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f588000ea80 tx=0x7f588000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.528+0000 7f586f7fe640 1 -- 172.21.15.59:0/2595049160 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880004b20 con 0x7f58880d1a60 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.529+0000 7f586f7fe640 1 -- 172.21.15.59:0/2595049160 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5880004ce0 con 0x7f58880d1a60 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.529+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5888145690 con 0x7f58880d1a60 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.529+0000 7f586f7fe640 1 -- 172.21.15.59:0/2595049160 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880005280 con 0x7f58880d1a60 2024-06-29T14:22:31.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.529+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5888145c50 con 0x7f58880d1a60 2024-06-29T14:22:31.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.530+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5888145120 con 0x7f58880d1a60 2024-06-29T14:22:31.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.531+0000 7f586f7fe640 1 -- 172.21.15.59:0/2595049160 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5880005420 con 0x7f58880d1a60 2024-06-29T14:22:31.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.531+0000 7f586f7fe640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5874076300 0x7f5874078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:31.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.531+0000 7f586f7fe640 1 -- 172.21.15.59:0/2595049160 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f5880098600 con 0x7f58880d1a60 2024-06-29T14:22:31.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.531+0000 7f5886ffd640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5874076300 0x7f5874078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:31.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.532+0000 7f5886ffd640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5874076300 0x7f5874078810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5878001720 tx=0x7f5878009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:31.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.536+0000 7f586f7fe640 1 -- 172.21.15.59:0/2595049160 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5880061540 con 0x7f58880d1a60 2024-06-29T14:22:31.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.769+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f58880b6aa0 con 0x7f58880d1a60 2024-06-29T14:22:31.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.770+0000 7f586f7fe640 1 -- 172.21.15.59:0/2595049160 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f5880065840 con 0x7f58880d1a60 2024-06-29T14:22:31.769 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-29T14:22:31.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5874076300 msgr2=0x7f5874078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:31.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5874076300 0x7f5874078810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5878001720 tx=0x7f5878009040 comp rx=0 tx=0).stop 2024-06-29T14:22:31.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880d1a60 msgr2=0x7f5888146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:31.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880d1a60 0x7f5888146990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f588000ea80 tx=0x7f588000ef40 comp rx=0 tx=0).stop 2024-06-29T14:22:31.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 shutdown_connections 2024-06-29T14:22:31.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5874076300 0x7f5874078810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58880d1a60 0x7f5888146990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f58880b2fa0 0x7f5888146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 --2- 172.21.15.59:0/2595049160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58880b2270 0x7f5888144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 >> 172.21.15.59:0/2595049160 conn(0x7f5888073e10 msgr2=0x7f588815b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 shutdown_connections 2024-06-29T14:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:31.772+0000 7f588d3e0640 1 -- 172.21.15.59:0/2595049160 wait complete. 2024-06-29T14:22:31.782 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-29T14:22:31.782 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:22:31.782 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi153.front.sepia.ceph.com 2024-06-29T14:22:31.782 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-06-29T14:22:31.782 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-06-29T14:22:31.782 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-29T14:22:31.783 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:22:31.783 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:22:31.783 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-06-29T14:22:31.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:31.783 DEBUG:teuthology.orchestra.run.smithi153:> ip addr 2024-06-29T14:22:31.799 INFO:teuthology.orchestra.run.smithi153.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:31.799 INFO:teuthology.orchestra.run.smithi153.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:31.799 INFO:teuthology.orchestra.run.smithi153.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:31.799 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b2 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f0 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b3 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f1 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:ea brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:eb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: inet 172.21.15.153/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:22:31.800 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft 42705sec preferred_lft 42705sec 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::ec4:7aff:fe88:6deb/64 scope link 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether ba:f2:f1:29:61:98 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::6c04:13ff:feda:52e8/64 scope link 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether ba:f2:f1:29:61:98 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::b8f2:f1ff:fe29:6198/64 scope link tentative 2024-06-29T14:22:31.801 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:31.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:31.802 DEBUG:teuthology.orchestra.run.smithi153:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:22:31.863 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-06-29T14:22:31.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:31.865 DEBUG:teuthology.orchestra.run.smithi153:> route 2024-06-29T14:22:31.921 INFO:teuthology.orchestra.run.smithi153.stdout:Kernel IP routing table 2024-06-29T14:22:31.921 INFO:teuthology.orchestra.run.smithi153.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:22:31.921 INFO:teuthology.orchestra.run.smithi153.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:22:31.921 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:22:31.921 INFO:teuthology.orchestra.run.smithi153.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:22:31.921 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:31.922 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:31.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:22:31.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:22:31.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:22:31.922 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:32.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:32.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:32.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:32.047 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-06-29T14:22:32.063 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-06-29T14:22:32.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:32.063 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list-id 2024-06-29T14:22:32.118 INFO:teuthology.orchestra.run.smithi153.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-06-29T14:22:32.118 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:32.119 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:32.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-06-29T14:22:32.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-06-29T14:22:32.119 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:32.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:32.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:32.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:32.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-06-29T14:22:32.261 INFO:teuthology.orchestra.run.smithi153.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:32.261 INFO:teuthology.orchestra.run.smithi153.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:32.261 INFO:teuthology.orchestra.run.smithi153.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:32.261 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:32.262 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:32.262 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:32.262 INFO:teuthology.orchestra.run.smithi153.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:22:32.262 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 66:5b:ac:9a:d1:97 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-06-29T14:22:32.262 INFO:teuthology.orchestra.run.smithi153.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-06-29T14:22:32.262 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:32.262 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::645b:acff:fe9a:d197/64 scope link tentative 2024-06-29T14:22:32.262 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:32.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:32.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-06-29T14:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 66:5b:ac:9a:d1:97 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-06-29T14:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-06-29T14:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::645b:acff:fe9a:d197/64 scope link tentative 2024-06-29T14:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:32.330 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-06-29T14:22:32.330 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:32.330 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:32.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-06-29T14:22:32.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-06-29T14:22:32.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-06-29T14:22:32.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-06-29T14:22:32.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-06-29T14:22:32.331 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:32.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:32.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:32.483 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:32.483 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-06-29T14:22:32.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set brx.1 up 2024-06-29T14:22:32.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set dev brx.1 master ceph-brx 2024-06-29T14:22:32.483 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-06-29T14:22:32.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:32.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:32.561 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:32.561 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-06-29T14:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-06-29T14:22:32.577 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:22:32.577 DEBUG:teuthology.orchestra.run.smithi153:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-06-29T14:22:32.632 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:22:32.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:32.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-06-29T14:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-06-29T14:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-06-29T14:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-06-29T14:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-06-29T14:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-06-29T14:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-06-29T14:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-06-29T14:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-06-29T14:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-06-29T14:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-06-29T14:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:22:32.745 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:22:32.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:32.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-06-29T14:22:32.770 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:32.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:32.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout:client4885 2024-06-29T14:22:32.867 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:22:32.867 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:32.884 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:32.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout:client4885 2024-06-29T14:22:32.980 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:22:33.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:33 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:22:33.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout:client4885 2024-06-29T14:22:33.036 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/150b8fd8-1bf5-44f0-b8d0-0083a48c0071.client4885/status') 2024-06-29T14:22:33.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout:instance: client.4885 (3)192.168.0.2:0/2734067298 2024-06-29T14:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout:blocklisted: false 2024-06-29T14:22:33.096 INFO:teuthology.orchestra.run.smithi153.stderr:0+1 records in 2024-06-29T14:22:33.096 INFO:teuthology.orchestra.run.smithi153.stderr:0+1 records out 2024-06-29T14:22:33.096 INFO:teuthology.orchestra.run.smithi153.stderr:69 bytes copied, 3.7704e-05 s, 1.8 MB/s 2024-06-29T14:22:33.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-29T14:22:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:22:33.103 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-06-29T14:22:33.123 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:33.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.257+0000 7fc3a326d640 1 -- 172.21.15.59:0/2098328040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cccc0 msgr2=0x7fc39c0cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.257+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2098328040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cccc0 0x7fc39c0cd0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc394005fd0 tx=0x7fc39402a940 comp rx=0 tx=0).stop 2024-06-29T14:22:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.257+0000 7fc3a326d640 1 -- 172.21.15.59:0/2098328040 shutdown_connections 2024-06-29T14:22:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.257+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2098328040 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cccc0 0x7fc39c0cd0d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.257+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2098328040 >> v2:172.21.15.153:3300/0 conn(0x7fc39c0b3800 0x7fc39c0b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.257+0000 7fc3a326d640 1 -- 172.21.15.59:0/2098328040 >> 172.21.15.59:0/2098328040 conn(0x7fc39c0b24c0 msgr2=0x7fc39c0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.257+0000 7fc3a326d640 1 -- 172.21.15.59:0/2098328040 shutdown_connections 2024-06-29T14:22:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.257+0000 7fc3a326d640 1 -- 172.21.15.59:0/2098328040 wait complete. 2024-06-29T14:22:33.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.259+0000 7fc3a326d640 1 Processor -- start 2024-06-29T14:22:33.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.259+0000 7fc3a326d640 1 -- start start 2024-06-29T14:22:33.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.260+0000 7fc3a326d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c0b3800 0x7fc39c150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.260+0000 7fc3a326d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc39c0cccc0 0x7fc39c1f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.260+0000 7fc3a326d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cf380 0x7fc39c1f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.260+0000 7fc3a326d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c070020 con 0x7fc39c0b3800 2024-06-29T14:22:33.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.260+0000 7fc3a326d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c06fea0 con 0x7fc39c0cccc0 2024-06-29T14:22:33.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.260+0000 7fc3a326d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc39c15dee0 con 0x7fc39c0cf380 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.260+0000 7fc3a1009640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c0b3800 0x7fc39c150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a180a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cf380 0x7fc39c1f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a1009640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c0b3800 0x7fc39c150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38016/0 (socket says 172.21.15.59:38016) 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a0808640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc39c0cccc0 0x7fc39c1f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a1009640 1 -- 172.21.15.59:0/2483775423 learned_addr learned my addr 172.21.15.59:0/2483775423 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a180a640 1 -- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc39c0cccc0 msgr2=0x7fc39c1f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a180a640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc39c0cccc0 0x7fc39c1f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a180a640 1 -- 172.21.15.59:0/2483775423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c0b3800 msgr2=0x7fc39c150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a180a640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c0b3800 0x7fc39c150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a180a640 1 -- 172.21.15.59:0/2483775423 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c1f6df0 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a180a640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cf380 0x7fc39c1f64d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc394009000 tx=0x7fc394031ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc391ffb640 1 -- 172.21.15.59:0/2483775423 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39403e070 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc391ffb640 1 -- 172.21.15.59:0/2483775423 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc394004540 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc391ffb640 1 -- 172.21.15.59:0/2483775423 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394030430 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.261+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39c1512e0 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.262+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc39c1518a0 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.263+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc39c151050 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.263+0000 7fc391ffb640 1 -- 172.21.15.59:0/2483775423 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc394043460 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.264+0000 7fc391ffb640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc3800762d0 0x7fc3800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.264+0000 7fc391ffb640 1 -- 172.21.15.59:0/2483775423 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fc3940b8ff0 con 0x7fc39c0cf380 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.264+0000 7fc3a1009640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc3800762d0 0x7fc3800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.265+0000 7fc3a1009640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc3800762d0 0x7fc3800787e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc3980018a0 tx=0x7fc398007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:33.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.269+0000 7fc391ffb640 1 -- 172.21.15.59:0/2483775423 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc39404a030 con 0x7fc39c0cf380 2024-06-29T14:22:33.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.461+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc39c0b3c10 con 0x7fc39c0cf380 2024-06-29T14:22:33.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.462+0000 7fc391ffb640 1 -- 172.21.15.59:0/2483775423 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18526 (secure 0 0 0) 0x7fc39c1f6df0 con 0x7fc39c0cf380 2024-06-29T14:22:33.461 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:33.461 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":26,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:21:59.698621+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc3800762d0 msgr2=0x7fc3800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc3800762d0 0x7fc3800787e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc3980018a0 tx=0x7fc398007be0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cf380 msgr2=0x7fc39c1f64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cf380 0x7fc39c1f64d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc394009000 tx=0x7fc394031ea0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 shutdown_connections 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc3800762d0 0x7fc3800787e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc39c0cf380 0x7fc39c1f64d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc39c0cccc0 0x7fc39c1f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 --2- 172.21.15.59:0/2483775423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c0b3800 0x7fc39c150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 >> 172.21.15.59:0/2483775423 conn(0x7fc39c0b24c0 msgr2=0x7fc39c157150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.464+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 shutdown_connections 2024-06-29T14:22:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.465+0000 7fc3a326d640 1 -- 172.21.15.59:0/2483775423 wait complete. 2024-06-29T14:22:33.475 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:33.475 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:33.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.596+0000 7f9627524640 1 -- 172.21.15.59:0/3958195321 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 msgr2=0x7f962015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.596+0000 7f9627524640 1 --2- 172.21.15.59:0/3958195321 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 0x7f962015f9b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9618005fd0 tx=0x7f961802a940 comp rx=0 tx=0).stop 2024-06-29T14:22:33.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.596+0000 7f9627524640 1 -- 172.21.15.59:0/3958195321 shutdown_connections 2024-06-29T14:22:33.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.596+0000 7f9627524640 1 --2- 172.21.15.59:0/3958195321 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 0x7f962015f9b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.596+0000 7f9627524640 1 -- 172.21.15.59:0/3958195321 >> 172.21.15.59:0/3958195321 conn(0x7f96200d3ee0 msgr2=0x7f96200d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:33.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.597+0000 7f9627524640 1 -- 172.21.15.59:0/3958195321 shutdown_connections 2024-06-29T14:22:33.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.597+0000 7f9627524640 1 -- 172.21.15.59:0/3958195321 wait complete. 2024-06-29T14:22:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.598+0000 7f9627524640 1 Processor -- start 2024-06-29T14:22:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.598+0000 7f9627524640 1 -- start start 2024-06-29T14:22:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9627524640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96200d23c0 0x7f962014c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9627524640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 0x7f96201540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9627524640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96201577e0 0x7f9620157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9627524640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9620160420 con 0x7f962015efe0 2024-06-29T14:22:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9627524640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96201602a0 con 0x7f96200d23c0 2024-06-29T14:22:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9627524640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f96201605a0 con 0x7f96201577e0 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 0x7f96201540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 0x7f96201540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38046/0 (socket says 172.21.15.59:38046) 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 -- 172.21.15.59:0/1172184022 learned_addr learned my addr 172.21.15.59:0/1172184022 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f96252c0640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96200d23c0 0x7f962014c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9625ac1640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96201577e0 0x7f9620157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 -- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96201577e0 msgr2=0x7f9620157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96201577e0 0x7f9620157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 -- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96200d23c0 msgr2=0x7f962014c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96200d23c0 0x7f962014c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 -- 172.21.15.59:0/1172184022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9620154630 con 0x7f962015efe0 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f96252c0640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96200d23c0 0x7f962014c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9625ac1640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96201577e0 0x7f9620157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:33.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9624abf640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 0x7f96201540f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9610001eb0 tx=0x7f961000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:33.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9615ffb640 1 -- 172.21.15.59:0/1172184022 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610011790 con 0x7f962015efe0 2024-06-29T14:22:33.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.600+0000 7f9615ffb640 1 -- 172.21.15.59:0/1172184022 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9610017070 con 0x7f962015efe0 2024-06-29T14:22:33.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.601+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962014cce0 con 0x7f962015efe0 2024-06-29T14:22:33.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.601+0000 7f9615ffb640 1 -- 172.21.15.59:0/1172184022 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610011ca0 con 0x7f962015efe0 2024-06-29T14:22:33.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.601+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f962014d2a0 con 0x7f962015efe0 2024-06-29T14:22:33.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.602+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f962014ca30 con 0x7f962015efe0 2024-06-29T14:22:33.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.603+0000 7f9615ffb640 1 -- 172.21.15.59:0/1172184022 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f961001c460 con 0x7f962015efe0 2024-06-29T14:22:33.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.603+0000 7f9615ffb640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f96040762d0 0x7f96040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.603+0000 7f9615ffb640 1 -- 172.21.15.59:0/1172184022 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f9610097860 con 0x7f962015efe0 2024-06-29T14:22:33.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.603+0000 7f96252c0640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f96040762d0 0x7f96040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.604+0000 7f96252c0640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f96040762d0 0x7f96040787e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f961c007fd0 tx=0x7f961c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:33.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.607+0000 7f9615ffb640 1 -- 172.21.15.59:0/1172184022 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9620154630 con 0x7f962015efe0 2024-06-29T14:22:33.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.799+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f96200b7a50 con 0x7f962015efe0 2024-06-29T14:22:33.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.799+0000 7f9615ffb640 1 -- 172.21.15.59:0/1172184022 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f9610067e20 con 0x7f962015efe0 2024-06-29T14:22:33.798 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:33.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f96040762d0 msgr2=0x7f96040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f96040762d0 0x7f96040787e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f961c007fd0 tx=0x7f961c009040 comp rx=0 tx=0).stop 2024-06-29T14:22:33.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 msgr2=0x7f96201540f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 0x7f96201540f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9610001eb0 tx=0x7f961000c890 comp rx=0 tx=0).stop 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 shutdown_connections 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f96040762d0 0x7f96040787e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96201577e0 0x7f9620157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962015efe0 0x7f96201540f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 --2- 172.21.15.59:0/1172184022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96200d23c0 0x7f962014c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 >> 172.21.15.59:0/1172184022 conn(0x7f96200d3ee0 msgr2=0x7f9620160900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 shutdown_connections 2024-06-29T14:22:33.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.801+0000 7f9627524640 1 -- 172.21.15.59:0/1172184022 wait complete. 2024-06-29T14:22:33.812 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:33.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.937+0000 7fd4ced9f640 1 -- 172.21.15.59:0/307755130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c8073f30 msgr2=0x7fd4c815a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.937+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/307755130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c8073f30 0x7fd4c815a400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80b4720 tx=0x7fd4b802a530 comp rx=0 tx=0).stop 2024-06-29T14:22:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.937+0000 7fd4ced9f640 1 -- 172.21.15.59:0/307755130 shutdown_connections 2024-06-29T14:22:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.937+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/307755130 >> v2:172.21.15.153:3300/0 conn(0x7fd4c80cf120 0x7fd4c8073500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.937+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/307755130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c8073f30 0x7fd4c815a400 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.937+0000 7fd4ced9f640 1 -- 172.21.15.59:0/307755130 >> 172.21.15.59:0/307755130 conn(0x7fd4c80b24c0 msgr2=0x7fd4c80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.937+0000 7fd4ced9f640 1 -- 172.21.15.59:0/307755130 shutdown_connections 2024-06-29T14:22:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.937+0000 7fd4ced9f640 1 -- 172.21.15.59:0/307755130 wait complete. 2024-06-29T14:22:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.939+0000 7fd4ced9f640 1 Processor -- start 2024-06-29T14:22:33.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.939+0000 7fd4ced9f640 1 -- start start 2024-06-29T14:22:33.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.940+0000 7fd4ced9f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4c8073a40 0x7fd4c8149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.940+0000 7fd4ced9f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4c8073f30 0x7fd4c8147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.940+0000 7fd4ced9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c80cf120 0x7fd4c814ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.940+0000 7fd4ced9f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c80cce40 con 0x7fd4c80cf120 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.940+0000 7fd4ced9f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c80cccc0 con 0x7fd4c8073a40 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.940+0000 7fd4ced9f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd4c80ccfc0 con 0x7fd4c8073f30 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.940+0000 7fd4cd33c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c80cf120 0x7fd4c814ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.940+0000 7fd4cd33c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c80cf120 0x7fd4c814ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38058/0 (socket says 172.21.15.59:38058) 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4cd33c640 1 -- 172.21.15.59:0/4189271757 learned_addr learned my addr 172.21.15.59:0/4189271757 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4ccb3b640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4c8073a40 0x7fd4c8149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4c7fff640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4c8073f30 0x7fd4c8147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4cd33c640 1 -- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4c8073f30 msgr2=0x7fd4c8147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4cd33c640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4c8073f30 0x7fd4c8147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4cd33c640 1 -- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4c8073a40 msgr2=0x7fd4c8149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4cd33c640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4c8073a40 0x7fd4c8149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:33.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4cd33c640 1 -- 172.21.15.59:0/4189271757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c814bae0 con 0x7fd4c80cf120 2024-06-29T14:22:33.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4cd33c640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c80cf120 0x7fd4c814ddb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000c3c0 tx=0x7fd4c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:33.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4c57fa640 1 -- 172.21.15.59:0/4189271757 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0017070 con 0x7fd4c80cf120 2024-06-29T14:22:33.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4c57fa640 1 -- 172.21.15.59:0/4189271757 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4c00035f0 con 0x7fd4c80cf120 2024-06-29T14:22:33.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4c57fa640 1 -- 172.21.15.59:0/4189271757 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0003a40 con 0x7fd4c80cf120 2024-06-29T14:22:33.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c814bfe0 con 0x7fd4c80cf120 2024-06-29T14:22:33.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.941+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c8149140 con 0x7fd4c80cf120 2024-06-29T14:22:33.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.942+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c814bd50 con 0x7fd4c80cf120 2024-06-29T14:22:33.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.943+0000 7fd4c57fa640 1 -- 172.21.15.59:0/4189271757 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd4c001a070 con 0x7fd4c80cf120 2024-06-29T14:22:33.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.943+0000 7fd4c57fa640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4a80762d0 0x7fd4a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:33.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.943+0000 7fd4c57fa640 1 -- 172.21.15.59:0/4189271757 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd4c0096da0 con 0x7fd4c80cf120 2024-06-29T14:22:33.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.944+0000 7fd4ccb3b640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4a80762d0 0x7fd4a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:33.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.944+0000 7fd4ccb3b640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4a80762d0 0x7fd4a80787e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8034000 tx=0x7fd4b8004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:33.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:33.948+0000 7fd4c57fa640 1 -- 172.21.15.59:0/4189271757 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd4c005fce0 con 0x7fd4c80cf120 2024-06-29T14:22:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.142+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd4c815a400 con 0x7fd4c80cf120 2024-06-29T14:22:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.142+0000 7fd4c57fa640 1 -- 172.21.15.59:0/4189271757 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fd4c0063fe0 con 0x7fd4c80cf120 2024-06-29T14:22:34.141 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4a80762d0 msgr2=0x7fd4a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4a80762d0 0x7fd4a80787e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8034000 tx=0x7fd4b8004730 comp rx=0 tx=0).stop 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c80cf120 msgr2=0x7fd4c814ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c80cf120 0x7fd4c814ddb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000c3c0 tx=0x7fd4c000c880 comp rx=0 tx=0).stop 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 shutdown_connections 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4a80762d0 0x7fd4a80787e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4c80cf120 0x7fd4c814ddb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4c8073f30 0x7fd4c8147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 --2- 172.21.15.59:0/4189271757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4c8073a40 0x7fd4c8149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.144+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 >> 172.21.15.59:0/4189271757 conn(0x7fd4c80b24c0 msgr2=0x7fd4c80afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:34.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.145+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 shutdown_connections 2024-06-29T14:22:34.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.145+0000 7fd4ced9f640 1 -- 172.21.15.59:0/4189271757 wait complete. 2024-06-29T14:22:34.155 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:34.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.278+0000 7f48e6343640 1 -- 172.21.15.59:0/4082204113 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48c8004800 msgr2=0x7f48c8004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.278+0000 7f48e6343640 1 --2- 172.21.15.59:0/4082204113 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48c8004800 0x7f48c8004c70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f48d4001800 tx=0x7f48c8005830 comp rx=0 tx=0).stop 2024-06-29T14:22:34.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.278+0000 7f48e48e0640 1 -- 172.21.15.59:0/4082204113 reap_dead start 2024-06-29T14:22:34.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.278+0000 7f48e6343640 1 -- 172.21.15.59:0/4082204113 shutdown_connections 2024-06-29T14:22:34.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.278+0000 7f48e6343640 1 -- 172.21.15.59:0/4082204113 >> 172.21.15.59:0/4082204113 conn(0x7f48e00b2ab0 msgr2=0x7f48e00b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:34.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.278+0000 7f48e6343640 1 -- 172.21.15.59:0/4082204113 shutdown_connections 2024-06-29T14:22:34.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.278+0000 7f48e6343640 1 -- 172.21.15.59:0/4082204113 wait complete. 2024-06-29T14:22:34.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.280+0000 7f48e6343640 1 Processor -- start 2024-06-29T14:22:34.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.280+0000 7f48e6343640 1 -- start start 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.281+0000 7f48e6343640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c8004800 0x7f48e0137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.281+0000 7f48e6343640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48e0154970 0x7f48e013fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.281+0000 7f48e6343640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0140000 0x7f48e0157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.281+0000 7f48e6343640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f48e015cfc0 con 0x7f48e0154970 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.281+0000 7f48e6343640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48e015ce40 con 0x7f48e0140000 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.281+0000 7f48e6343640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f48e015d140 con 0x7f48c8004800 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48e0154970 0x7f48e013fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48e0154970 0x7f48e013fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38084/0 (socket says 172.21.15.59:38084) 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 -- 172.21.15.59:0/1675074996 learned_addr learned my addr 172.21.15.59:0/1675074996 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:34.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48dffff640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c8004800 0x7f48e0137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48e48e0640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0140000 0x7f48e0157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 -- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c8004800 msgr2=0x7f48e0137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c8004800 0x7f48e0137130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 -- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0140000 msgr2=0x7f48e0157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0140000 0x7f48e0157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 -- 172.21.15.59:0/1675074996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e0140410 con 0x7f48e0154970 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48df7fe640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48e0154970 0x7f48e013fac0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f48d4000c00 tx=0x7f48d40069d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48dcff9640 1 -- 172.21.15.59:0/1675074996 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4043070 con 0x7f48e0154970 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48dcff9640 1 -- 172.21.15.59:0/1675074996 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48d4003900 con 0x7f48e0154970 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e01f4ad0 con 0x7f48e0154970 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48dcff9640 1 -- 172.21.15.59:0/1675074996 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4003d50 con 0x7f48e0154970 2024-06-29T14:22:34.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.282+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e01571f0 con 0x7f48e0154970 2024-06-29T14:22:34.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.284+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48e0156f40 con 0x7f48e0154970 2024-06-29T14:22:34.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.284+0000 7f48dcff9640 1 -- 172.21.15.59:0/1675074996 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f48d4033480 con 0x7f48e0154970 2024-06-29T14:22:34.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.285+0000 7f48dcff9640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f48cc076300 0x7f48cc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.285+0000 7f48dcff9640 1 -- 172.21.15.59:0/1675074996 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f48d40b7a90 con 0x7f48e0154970 2024-06-29T14:22:34.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.285+0000 7f48dffff640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f48cc076300 0x7f48cc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.286+0000 7f48dffff640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f48cc076300 0x7f48cc078810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f48e015c420 tx=0x7f48d0001020 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:34.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.289+0000 7f48dcff9640 1 -- 172.21.15.59:0/1675074996 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f48d404c080 con 0x7f48e0154970 2024-06-29T14:22:34.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.479+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f48e015d6d0 con 0x7f48e0154970 2024-06-29T14:22:34.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.479+0000 7f48dcff9640 1 -- 172.21.15.59:0/1675074996 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f48e0140410 con 0x7f48e0154970 2024-06-29T14:22:34.478 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f48cc076300 msgr2=0x7f48cc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f48cc076300 0x7f48cc078810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f48e015c420 tx=0x7f48d0001020 comp rx=0 tx=0).stop 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48e0154970 msgr2=0x7f48e013fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48e0154970 0x7f48e013fac0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f48d4000c00 tx=0x7f48d40069d0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 shutdown_connections 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f48cc076300 0x7f48cc078810 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48e0140000 0x7f48e0157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48e0154970 0x7f48e013fac0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 --2- 172.21.15.59:0/1675074996 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48c8004800 0x7f48e0137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 >> 172.21.15.59:0/1675074996 conn(0x7f48e00b2ab0 msgr2=0x7f48e00b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 shutdown_connections 2024-06-29T14:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.481+0000 7f48e6343640 1 -- 172.21.15.59:0/1675074996 wait complete. 2024-06-29T14:22:34.492 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:34.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.613+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/3719262387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a40d23c0 msgr2=0x7fe9a40b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.613+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/3719262387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a40d23c0 0x7fe9a40b3990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009830 tx=0x7fe9a002a510 comp rx=0 tx=0).stop 2024-06-29T14:22:34.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.613+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/3719262387 shutdown_connections 2024-06-29T14:22:34.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.613+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/3719262387 >> v2:172.21.15.153:3300/0 conn(0x7fe9a415efe0 0x7fe9a415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.613+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/3719262387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a40d23c0 0x7fe9a40b3990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.613+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/3719262387 >> 172.21.15.59:0/3719262387 conn(0x7fe9a40d3ee0 msgr2=0x7fe9a40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:34.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.614+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/3719262387 shutdown_connections 2024-06-29T14:22:34.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.614+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/3719262387 wait complete. 2024-06-29T14:22:34.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.615+0000 7fe9ac7d2640 1 Processor -- start 2024-06-29T14:22:34.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.615+0000 7fe9ac7d2640 1 -- start start 2024-06-29T14:22:34.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9ac7d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9a40b3ed0 0x7fe9a41f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9ac7d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9a40d23c0 0x7fe9a4153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9ac7d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a415efe0 0x7fe9a41e9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9ac7d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a4160280 con 0x7fe9a40d23c0 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9ac7d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a4160100 con 0x7fe9a40b3ed0 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9ac7d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a4160400 con 0x7fe9a415efe0 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9a40d23c0 0x7fe9a4153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9a40d23c0 0x7fe9a4153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38100/0 (socket says 172.21.15.59:38100) 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 -- 172.21.15.59:0/2651244824 learned_addr learned my addr 172.21.15.59:0/2651244824 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9aa56e640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9a40b3ed0 0x7fe9a41f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9aad6f640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a415efe0 0x7fe9a41e9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 -- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a415efe0 msgr2=0x7fe9a41e9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a415efe0 0x7fe9a41e9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 -- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9a40b3ed0 msgr2=0x7fe9a41f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9a40b3ed0 0x7fe9a41f3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 -- 172.21.15.59:0/2651244824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a4154130 con 0x7fe9a40d23c0 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9aad6f640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a415efe0 0x7fe9a41e9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9a9d6d640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9a40d23c0 0x7fe9a4153690 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe99400c760 tx=0x7fe99400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe99affd640 1 -- 172.21.15.59:0/2651244824 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe994016020 con 0x7fe9a40d23c0 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a41e6f80 con 0x7fe9a40d23c0 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe99affd640 1 -- 172.21.15.59:0/2651244824 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe99400a950 con 0x7fe9a40d23c0 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.617+0000 7fe99affd640 1 -- 172.21.15.59:0/2651244824 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99400ada0 con 0x7fe9a40d23c0 2024-06-29T14:22:34.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.618+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a41e7540 con 0x7fe9a40d23c0 2024-06-29T14:22:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.618+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a40b3990 con 0x7fe9a40d23c0 2024-06-29T14:22:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.619+0000 7fe99affd640 1 -- 172.21.15.59:0/2651244824 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe994019020 con 0x7fe9a40d23c0 2024-06-29T14:22:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.620+0000 7fe99affd640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe9880762d0 0x7fe9880787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.620+0000 7fe99affd640 1 -- 172.21.15.59:0/2651244824 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe994097fd0 con 0x7fe9a40d23c0 2024-06-29T14:22:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.620+0000 7fe9aa56e640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe9880762d0 0x7fe9880787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.621+0000 7fe9aa56e640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe9880762d0 0x7fe9880787e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0034000 tx=0x7fe9a0004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:34.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.624+0000 7fe99affd640 1 -- 172.21.15.59:0/2651244824 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe994020020 con 0x7fe9a40d23c0 2024-06-29T14:22:34.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.814+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe9a40d2950 con 0x7fe9a40d23c0 2024-06-29T14:22:34.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.815+0000 7fe99affd640 1 -- 172.21.15.59:0/2651244824 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fe9a40b42e0 con 0x7fe9a40d23c0 2024-06-29T14:22:34.814 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe9880762d0 msgr2=0x7fe9880787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe9880762d0 0x7fe9880787e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0034000 tx=0x7fe9a0004730 comp rx=0 tx=0).stop 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9a40d23c0 msgr2=0x7fe9a4153690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9a40d23c0 0x7fe9a4153690 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe99400c760 tx=0x7fe99400cc20 comp rx=0 tx=0).stop 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 shutdown_connections 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe9880762d0 0x7fe9880787e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9a415efe0 0x7fe9a41e9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9a40d23c0 0x7fe9a4153690 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 --2- 172.21.15.59:0/2651244824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9a40b3ed0 0x7fe9a41f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 >> 172.21.15.59:0/2651244824 conn(0x7fe9a40d3ee0 msgr2=0x7fe9a406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:34.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 shutdown_connections 2024-06-29T14:22:34.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.817+0000 7fe9ac7d2640 1 -- 172.21.15.59:0/2651244824 wait complete. 2024-06-29T14:22:34.830 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:34.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.951+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/2716633686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 msgr2=0x7fe0f80d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.951+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/2716633686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 0x7fe0f80d1520 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009830 tx=0x7fe0ec02a510 comp rx=0 tx=0).stop 2024-06-29T14:22:34.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.952+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/2716633686 shutdown_connections 2024-06-29T14:22:34.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.952+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/2716633686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0f80d1a60 0x7fe0f80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.952+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/2716633686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 0x7fe0f80d1520 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.952+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/2716633686 >> 172.21.15.59:0/2716633686 conn(0x7fe0f8073e10 msgr2=0x7fe0f8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:34.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.952+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/2716633686 shutdown_connections 2024-06-29T14:22:34.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.952+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/2716633686 wait complete. 2024-06-29T14:22:34.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.953+0000 7fe0ff2f6640 1 Processor -- start 2024-06-29T14:22:34.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.953+0000 7fe0ff2f6640 1 -- start start 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0ff2f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0f80b2270 0x7fe0f815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0ff2f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 0x7fe0f81451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0ff2f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f80d1a60 0x7fe0f8153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0ff2f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f80b6400 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0ff2f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f80b6280 con 0x7fe0f80d1a60 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0ff2f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe0f80b6580 con 0x7fe0f80b2270 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 0x7fe0f81451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 0x7fe0f81451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38116/0 (socket says 172.21.15.59:38116) 2024-06-29T14:22:34.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 -- 172.21.15.59:0/610426924 learned_addr learned my addr 172.21.15.59:0/610426924 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fd092640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0f80b2270 0x7fe0f815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fd893640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f80d1a60 0x7fe0f8153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 -- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0f80b2270 msgr2=0x7fe0f815ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0f80b2270 0x7fe0f815ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 -- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f80d1a60 msgr2=0x7fe0f8153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f80d1a60 0x7fe0f8153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 -- 172.21.15.59:0/610426924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f815d960 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fd893640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f80d1a60 0x7fe0f8153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0fc891640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 0x7fe0f81451c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009950 tx=0x7fe0ec033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.955+0000 7fe0f5ffb640 1 -- 172.21.15.59:0/610426924 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec031d60 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.956+0000 7fe0f5ffb640 1 -- 172.21.15.59:0/610426924 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0ec033e10 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.956+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f8145700 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.956+0000 7fe0f5ffb640 1 -- 172.21.15.59:0/610426924 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec005260 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.956+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f815dee0 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.957+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0f80b6ac0 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.958+0000 7fe0f5ffb640 1 -- 172.21.15.59:0/610426924 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe0ec005400 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.958+0000 7fe0f5ffb640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe0e4076300 0x7fe0e4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:34.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.958+0000 7fe0f5ffb640 1 -- 172.21.15.59:0/610426924 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fe0ec0b9190 con 0x7fe0f80b2fa0 2024-06-29T14:22:34.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.959+0000 7fe0fd092640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe0e4076300 0x7fe0e4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:34.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.959+0000 7fe0fd092640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe0e4076300 0x7fe0e4078810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00046b0 tx=0x7fe0f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:34.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:34.962+0000 7fe0f5ffb640 1 -- 172.21.15.59:0/610426924 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe0ec03f030 con 0x7fe0f80b2fa0 2024-06-29T14:22:35.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.152+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0f815d360 con 0x7fe0f80b2fa0 2024-06-29T14:22:35.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.153+0000 7fe0f5ffb640 1 -- 172.21.15.59:0/610426924 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7fe0ec0827d0 con 0x7fe0f80b2fa0 2024-06-29T14:22:35.152 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:35.153 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe0e4076300 msgr2=0x7fe0e4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe0e4076300 0x7fe0e4078810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00046b0 tx=0x7fe0f0009040 comp rx=0 tx=0).stop 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 msgr2=0x7fe0f81451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 0x7fe0f81451c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009950 tx=0x7fe0ec033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 shutdown_connections 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe0e4076300 0x7fe0e4078810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f80d1a60 0x7fe0f8153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0f80b2fa0 0x7fe0f81451c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 --2- 172.21.15.59:0/610426924 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0f80b2270 0x7fe0f815ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.155+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 >> 172.21.15.59:0/610426924 conn(0x7fe0f8073e10 msgr2=0x7fe0f8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.156+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 shutdown_connections 2024-06-29T14:22:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.156+0000 7fe0ff2f6640 1 -- 172.21.15.59:0/610426924 wait complete. 2024-06-29T14:22:35.166 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:35.167 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 -- 172.21.15.59:0/1091394220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e880b3ed0 msgr2=0x7f9e8815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/1091394220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e880b3ed0 0x7f9e8815e8d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9e880d15d0 tx=0x7f9e7c02a560 comp rx=0 tx=0).stop 2024-06-29T14:22:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 -- 172.21.15.59:0/1091394220 shutdown_connections 2024-06-29T14:22:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/1091394220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e8815ee10 0x7f9e8815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/1091394220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e880b3ed0 0x7f9e8815e8d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/1091394220 >> v2:172.21.15.153:3300/0 conn(0x7f9e880d23c0 0x7f9e880b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 -- 172.21.15.59:0/1091394220 >> 172.21.15.59:0/1091394220 conn(0x7f9e880d3ee0 msgr2=0x7f9e880d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:35.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 -- 172.21.15.59:0/1091394220 shutdown_connections 2024-06-29T14:22:35.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.290+0000 7f9e8e79f640 1 -- 172.21.15.59:0/1091394220 wait complete. 2024-06-29T14:22:35.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.292+0000 7f9e8e79f640 1 Processor -- start 2024-06-29T14:22:35.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.292+0000 7f9e8e79f640 1 -- start start 2024-06-29T14:22:35.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8e79f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e880b3ed0 0x7f9e88154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8e79f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880d23c0 0x7f9e881eb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8e79f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e8815ee10 0x7f9e880c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8e79f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9e88160070 con 0x7f9e8815ee10 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8e79f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8815fef0 con 0x7f9e880b3ed0 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8e79f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9e881601f0 con 0x7f9e880d23c0 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8cd3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e8815ee10 0x7f9e880c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8cd3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e8815ee10 0x7f9e880c8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38140/0 (socket says 172.21.15.59:38140) 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e8cd3c640 1 -- 172.21.15.59:0/688066266 learned_addr learned my addr 172.21.15.59:0/688066266 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e87fff640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e880b3ed0 0x7f9e88154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.293+0000 7f9e877fe640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880d23c0 0x7f9e881eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e8cd3c640 1 -- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880d23c0 msgr2=0x7f9e881eb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e8cd3c640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880d23c0 0x7f9e881eb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e8cd3c640 1 -- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e880b3ed0 msgr2=0x7f9e88154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e8cd3c640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e880b3ed0 0x7f9e88154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e8cd3c640 1 -- 172.21.15.59:0/688066266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e88154910 con 0x7f9e8815ee10 2024-06-29T14:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e8cd3c640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e8815ee10 0x7f9e880c8bb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000ea70 tx=0x7f9e8000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e84ff9640 1 -- 172.21.15.59:0/688066266 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8000ce40 con 0x7f9e8815ee10 2024-06-29T14:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e880cd200 con 0x7f9e8815ee10 2024-06-29T14:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e84ff9640 1 -- 172.21.15.59:0/688066266 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e80004c10 con 0x7f9e8815ee10 2024-06-29T14:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e84ff9640 1 -- 172.21.15.59:0/688066266 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80005020 con 0x7f9e8815ee10 2024-06-29T14:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.294+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e881e8bb0 con 0x7f9e8815ee10 2024-06-29T14:22:35.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.295+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e880b3990 con 0x7f9e8815ee10 2024-06-29T14:22:35.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.296+0000 7f9e84ff9640 1 -- 172.21.15.59:0/688066266 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9e800051c0 con 0x7f9e8815ee10 2024-06-29T14:22:35.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.297+0000 7f9e84ff9640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e740762d0 0x7f9e740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.297+0000 7f9e84ff9640 1 -- 172.21.15.59:0/688066266 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f9e80098840 con 0x7f9e8815ee10 2024-06-29T14:22:35.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.297+0000 7f9e87fff640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e740762d0 0x7f9e740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.298+0000 7f9e87fff640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e740762d0 0x7f9e740787e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9e880d2e40 tx=0x7f9e78005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:35.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.301+0000 7f9e84ff9640 1 -- 172.21.15.59:0/688066266 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9e800616d0 con 0x7f9e8815ee10 2024-06-29T14:22:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.494+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9e880d2950 con 0x7f9e8815ee10 2024-06-29T14:22:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.495+0000 7f9e84ff9640 1 -- 172.21.15.59:0/688066266 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f9e800659d0 con 0x7f9e8815ee10 2024-06-29T14:22:35.494 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:35.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.496+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e740762d0 msgr2=0x7f9e740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.496+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e740762d0 0x7f9e740787e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9e880d2e40 tx=0x7f9e78005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.496+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e8815ee10 msgr2=0x7f9e880c8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.496+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e8815ee10 0x7f9e880c8bb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000ea70 tx=0x7f9e8000ef30 comp rx=0 tx=0).stop 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.496+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 shutdown_connections 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.497+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9e740762d0 0x7f9e740787e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.497+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e8815ee10 0x7f9e880c8bb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.497+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e880d23c0 0x7f9e881eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.497+0000 7f9e8e79f640 1 --2- 172.21.15.59:0/688066266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e880b3ed0 0x7f9e88154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.497+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 >> 172.21.15.59:0/688066266 conn(0x7f9e880d3ee0 msgr2=0x7f9e88160730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.497+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 shutdown_connections 2024-06-29T14:22:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.497+0000 7f9e8e79f640 1 -- 172.21.15.59:0/688066266 wait complete. 2024-06-29T14:22:35.507 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.629+0000 7f86b483c640 1 -- 172.21.15.59:0/3269779075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b0162800 msgr2=0x7f86b0162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.629+0000 7f86b483c640 1 --2- 172.21.15.59:0/3269779075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b0162800 0x7f86b0162c10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f86a0005fd0 tx=0x7f86a002a9c0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.629+0000 7f86b483c640 1 -- 172.21.15.59:0/3269779075 shutdown_connections 2024-06-29T14:22:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.629+0000 7f86b483c640 1 --2- 172.21.15.59:0/3269779075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b0162800 0x7f86b0162c10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.629+0000 7f86b483c640 1 --2- 172.21.15.59:0/3269779075 >> v2:172.21.15.153:3300/0 conn(0x7f86b00ce950 0x7f86b01622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.629+0000 7f86b483c640 1 -- 172.21.15.59:0/3269779075 >> 172.21.15.59:0/3269779075 conn(0x7f86b00b5610 msgr2=0x7f86b00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.629+0000 7f86b483c640 1 -- 172.21.15.59:0/3269779075 shutdown_connections 2024-06-29T14:22:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.629+0000 7f86b483c640 1 -- 172.21.15.59:0/3269779075 wait complete. 2024-06-29T14:22:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.631+0000 7f86b483c640 1 Processor -- start 2024-06-29T14:22:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.631+0000 7f86b483c640 1 -- start start 2024-06-29T14:22:35.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86b483c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b008c190 0x7f86b01f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86b483c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b00ce950 0x7f86b00ab3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86b483c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86b0162800 0x7f86b00c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86b483c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f86b00b7f60 con 0x7f86b008c190 2024-06-29T14:22:35.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86b483c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86b00b7de0 con 0x7f86b00ce950 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86b483c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f86b00b80e0 con 0x7f86b0162800 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86ae59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b008c190 0x7f86b01f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86ae59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b008c190 0x7f86b01f4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38154/0 (socket says 172.21.15.59:38154) 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86ae59c640 1 -- 172.21.15.59:0/4210661548 learned_addr learned my addr 172.21.15.59:0/4210661548 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86add9b640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b00ce950 0x7f86b00ab3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86aed9d640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86b0162800 0x7f86b00c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86ae59c640 1 -- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86b0162800 msgr2=0x7f86b00c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86ae59c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86b0162800 0x7f86b00c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86ae59c640 1 -- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b00ce950 msgr2=0x7f86b00ab3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86ae59c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b00ce950 0x7f86b00ab3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86ae59c640 1 -- 172.21.15.59:0/4210661548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b01614f0 con 0x7f86b008c190 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.632+0000 7f86aed9d640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86b0162800 0x7f86b00c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.633+0000 7f86ae59c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b008c190 0x7f86b01f4990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f86a800b410 tx=0x7f86a800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:35.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.633+0000 7f86a6ffd640 1 -- 172.21.15.59:0/4210661548 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8015020 con 0x7f86b008c190 2024-06-29T14:22:35.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.633+0000 7f86a6ffd640 1 -- 172.21.15.59:0/4210661548 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86a8011920 con 0x7f86b008c190 2024-06-29T14:22:35.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.633+0000 7f86a6ffd640 1 -- 172.21.15.59:0/4210661548 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8011d70 con 0x7f86b008c190 2024-06-29T14:22:35.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.633+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b01617d0 con 0x7f86b008c190 2024-06-29T14:22:35.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.633+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b0161d90 con 0x7f86b008c190 2024-06-29T14:22:35.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.634+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86b01622c0 con 0x7f86b008c190 2024-06-29T14:22:35.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.635+0000 7f86a6ffd640 1 -- 172.21.15.59:0/4210661548 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f86a8018020 con 0x7f86b008c190 2024-06-29T14:22:35.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.635+0000 7f86a6ffd640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86900762d0 0x7f86900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.635+0000 7f86a6ffd640 1 -- 172.21.15.59:0/4210661548 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f86a8097b90 con 0x7f86b008c190 2024-06-29T14:22:35.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.635+0000 7f86add9b640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86900762d0 0x7f86900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.636+0000 7f86add9b640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86900762d0 0x7f86900787e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f869800a490 tx=0x7f869800a420 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:35.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.640+0000 7f86a6ffd640 1 -- 172.21.15.59:0/4210661548 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f86a801f020 con 0x7f86b008c190 2024-06-29T14:22:35.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.832+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f86b0161f60 con 0x7f86b008c190 2024-06-29T14:22:35.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.832+0000 7f86a6ffd640 1 -- 172.21.15.59:0/4210661548 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f86b0161f60 con 0x7f86b008c190 2024-06-29T14:22:35.831 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:35.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86900762d0 msgr2=0x7f86900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86900762d0 0x7f86900787e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f869800a490 tx=0x7f869800a420 comp rx=0 tx=0).stop 2024-06-29T14:22:35.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b008c190 msgr2=0x7f86b01f4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b008c190 0x7f86b01f4990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f86a800b410 tx=0x7f86a800b8d0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 shutdown_connections 2024-06-29T14:22:35.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86900762d0 0x7f86900787e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86b0162800 0x7f86b00c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b00ce950 0x7f86b00ab3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 --2- 172.21.15.59:0/4210661548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b008c190 0x7f86b01f4990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 >> 172.21.15.59:0/4210661548 conn(0x7f86b00b5610 msgr2=0x7f86b00c80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:35.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 shutdown_connections 2024-06-29T14:22:35.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.834+0000 7f86b483c640 1 -- 172.21.15.59:0/4210661548 wait complete. 2024-06-29T14:22:35.844 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.966+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3240304595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec1458a0 msgr2=0x7fcdec0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.966+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3240304595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec1458a0 0x7fcdec0cf870 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0b1c40 tx=0x7fcddc02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.966+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3240304595 shutdown_connections 2024-06-29T14:22:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.966+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3240304595 >> v2:172.21.15.153:3300/0 conn(0x7fcdec0b3740 0x7fcdec0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.966+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3240304595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec1458a0 0x7fcdec0cf870 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.966+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3240304595 >> 172.21.15.59:0/3240304595 conn(0x7fcdec06c950 msgr2=0x7fcdec073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.966+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3240304595 shutdown_connections 2024-06-29T14:22:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.966+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3240304595 wait complete. 2024-06-29T14:22:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.968+0000 7fcdf3b73640 1 Processor -- start 2024-06-29T14:22:35.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.968+0000 7fcdf3b73640 1 -- start start 2024-06-29T14:22:35.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf3b73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec0b3740 0x7fcdec06f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf3b73640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdec15c2f0 0x7fcdec1e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf3b73640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdec1e6a70 0x7fcdec15e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf3b73640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec160020 con 0x7fcdec0b3740 2024-06-29T14:22:35.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf3b73640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec15cbb0 con 0x7fcdec1e6a70 2024-06-29T14:22:35.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf3b73640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcdec1601a0 con 0x7fcdec15c2f0 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf190f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec0b3740 0x7fcdec06f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf190f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec0b3740 0x7fcdec06f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38170/0 (socket says 172.21.15.59:38170) 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf190f640 1 -- 172.21.15.59:0/3981018336 learned_addr learned my addr 172.21.15.59:0/3981018336 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf2110640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdec1e6a70 0x7fcdec15e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf110e640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdec15c2f0 0x7fcdec1e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf190f640 1 -- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdec15c2f0 msgr2=0x7fcdec1e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf190f640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdec15c2f0 0x7fcdec1e78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf190f640 1 -- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdec1e6a70 msgr2=0x7fcdec15e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf190f640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdec1e6a70 0x7fcdec15e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf190f640 1 -- 172.21.15.59:0/3981018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec1e7e30 con 0x7fcdec0b3740 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.969+0000 7fcdf2110640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdec1e6a70 0x7fcdec15e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:35.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.970+0000 7fcdf190f640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec0b3740 0x7fcdec06f620 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcddc005fd0 tx=0x7fcddc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:35.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.970+0000 7fcdea7fc640 1 -- 172.21.15.59:0/3981018336 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc031b90 con 0x7fcdec0b3740 2024-06-29T14:22:35.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.970+0000 7fcdea7fc640 1 -- 172.21.15.59:0/3981018336 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcddc031e20 con 0x7fcdec0b3740 2024-06-29T14:22:35.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.970+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdec06def0 con 0x7fcdec0b3740 2024-06-29T14:22:35.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.970+0000 7fcdea7fc640 1 -- 172.21.15.59:0/3981018336 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc0332d0 con 0x7fcdec0b3740 2024-06-29T14:22:35.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.970+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdec06e4b0 con 0x7fcdec0b3740 2024-06-29T14:22:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.971+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdec1606c0 con 0x7fcdec0b3740 2024-06-29T14:22:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.972+0000 7fcdea7fc640 1 -- 172.21.15.59:0/3981018336 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcddc033470 con 0x7fcdec0b3740 2024-06-29T14:22:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.972+0000 7fcdea7fc640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcdd80762d0 0x7fcdd80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.972+0000 7fcdea7fc640 1 -- 172.21.15.59:0/3981018336 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fcddc0b95b0 con 0x7fcdec0b3740 2024-06-29T14:22:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.973+0000 7fcdf110e640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcdd80762d0 0x7fcdd80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:35.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.973+0000 7fcdf110e640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcdd80762d0 0x7fcdd80787e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcdec06fe00 tx=0x7fcde0005c70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:35.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:35.976+0000 7fcdea7fc640 1 -- 172.21.15.59:0/3981018336 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcddc052080 con 0x7fcdec0b3740 2024-06-29T14:22:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.166+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcdec1f7cd0 con 0x7fcdec0b3740 2024-06-29T14:22:36.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.167+0000 7fcdea7fc640 1 -- 172.21.15.59:0/3981018336 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fcddc03f030 con 0x7fcdec0b3740 2024-06-29T14:22:36.166 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcdd80762d0 msgr2=0x7fcdd80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcdd80762d0 0x7fcdd80787e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcdec06fe00 tx=0x7fcde0005c70 comp rx=0 tx=0).stop 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec0b3740 msgr2=0x7fcdec06f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec0b3740 0x7fcdec06f620 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcddc005fd0 tx=0x7fcddc003970 comp rx=0 tx=0).stop 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 shutdown_connections 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcdd80762d0 0x7fcdd80787e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdec1e6a70 0x7fcdec15e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdec15c2f0 0x7fcdec1e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 --2- 172.21.15.59:0/3981018336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdec0b3740 0x7fcdec06f620 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 >> 172.21.15.59:0/3981018336 conn(0x7fcdec06c950 msgr2=0x7fcdec141df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:36.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 shutdown_connections 2024-06-29T14:22:36.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.169+0000 7fcdf3b73640 1 -- 172.21.15.59:0/3981018336 wait complete. 2024-06-29T14:22:36.179 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:36.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.302+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2644398042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe10c003a70 msgr2=0x7fe10c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.302+0000 7fe12c1fb640 1 --2- 172.21.15.59:0/2644398042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe10c003a70 0x7fe10c003ee0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00c290 tx=0x7fe10c005830 comp rx=0 tx=0).stop 2024-06-29T14:22:36.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.302+0000 7fe12a798640 1 -- 172.21.15.59:0/2644398042 reap_dead start 2024-06-29T14:22:36.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.302+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2644398042 shutdown_connections 2024-06-29T14:22:36.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.302+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2644398042 >> 172.21.15.59:0/2644398042 conn(0x7fe1240b24c0 msgr2=0x7fe1240cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:36.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.302+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2644398042 shutdown_connections 2024-06-29T14:22:36.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.302+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2644398042 wait complete. 2024-06-29T14:22:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.304+0000 7fe12c1fb640 1 Processor -- start 2024-06-29T14:22:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.304+0000 7fe12c1fb640 1 -- start start 2024-06-29T14:22:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe12c1fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10c003a70 0x7fe1240aea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe12c1fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe12413b820 0x7fe12415a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe12c1fb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe12406c1e0 0x7fe12406c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe12c1fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe12415f240 con 0x7fe12413b820 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe12c1fb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe12406dc10 con 0x7fe10c003a70 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe12c1fb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe12415f3c0 con 0x7fe12406c1e0 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129796640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe12413b820 0x7fe12415a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129796640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe12413b820 0x7fe12415a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38194/0 (socket says 172.21.15.59:38194) 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129796640 1 -- 172.21.15.59:0/2059115773 learned_addr learned my addr 172.21.15.59:0/2059115773 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe12a798640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe12406c1e0 0x7fe12406c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129f97640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10c003a70 0x7fe1240aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129796640 1 -- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe12406c1e0 msgr2=0x7fe12406c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129796640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe12406c1e0 0x7fe12406c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129796640 1 -- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10c003a70 msgr2=0x7fe1240aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129796640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10c003a70 0x7fe1240aea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129796640 1 -- 172.21.15.59:0/2059115773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12406cb30 con 0x7fe12413b820 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe12a798640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe12406c1e0 0x7fe12406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.305+0000 7fe129f97640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10c003a70 0x7fe1240aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.306+0000 7fe129796640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe12413b820 0x7fe12415a160 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe118011890 tx=0x7fe118011d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.306+0000 7fe1227fc640 1 -- 172.21.15.59:0/2059115773 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11801e070 con 0x7fe12413b820 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.306+0000 7fe1227fc640 1 -- 172.21.15.59:0/2059115773 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe11800fbb0 con 0x7fe12413b820 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.306+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1240ad8b0 con 0x7fe12413b820 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.306+0000 7fe1227fc640 1 -- 172.21.15.59:0/2059115773 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe118019410 con 0x7fe12413b820 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.306+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1240adb50 con 0x7fe12413b820 2024-06-29T14:22:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.307+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe12406d2f0 con 0x7fe12413b820 2024-06-29T14:22:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.307+0000 7fe1227fc640 1 -- 172.21.15.59:0/2059115773 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe118021070 con 0x7fe12413b820 2024-06-29T14:22:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.308+0000 7fe1227fc640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe110076300 0x7fe110078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.308+0000 7fe1227fc640 1 -- 172.21.15.59:0/2059115773 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fe118098280 con 0x7fe12413b820 2024-06-29T14:22:36.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.308+0000 7fe129f97640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe110076300 0x7fe110078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.309+0000 7fe129f97640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe110076300 0x7fe110078810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe11c033000 tx=0x7fe1240ce1a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.312+0000 7fe1227fc640 1 -- 172.21.15.59:0/2059115773 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe12406cb30 con 0x7fe12413b820 2024-06-29T14:22:36.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.502+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe1240b06f0 con 0x7fe12413b820 2024-06-29T14:22:36.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.502+0000 7fe1227fc640 1 -- 172.21.15.59:0/2059115773 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fe118061190 con 0x7fe12413b820 2024-06-29T14:22:36.501 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:36.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe110076300 msgr2=0x7fe110078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe110076300 0x7fe110078810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe11c033000 tx=0x7fe1240ce1a0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe12413b820 msgr2=0x7fe12415a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe12413b820 0x7fe12415a160 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe118011890 tx=0x7fe118011d50 comp rx=0 tx=0).stop 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 shutdown_connections 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe110076300 0x7fe110078810 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe12406c1e0 0x7fe12406c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe12413b820 0x7fe12415a160 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 --2- 172.21.15.59:0/2059115773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe10c003a70 0x7fe1240aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 >> 172.21.15.59:0/2059115773 conn(0x7fe1240b24c0 msgr2=0x7fe12415d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 shutdown_connections 2024-06-29T14:22:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.504+0000 7fe12c1fb640 1 -- 172.21.15.59:0/2059115773 wait complete. 2024-06-29T14:22:36.515 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:36.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.633+0000 7fd2c8e38640 1 -- 172.21.15.59:0/2617274457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c4162800 msgr2=0x7fd2c4162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.633+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/2617274457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c4162800 0x7fd2c4162c10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4005fd0 tx=0x7fd2b402a870 comp rx=0 tx=0).stop 2024-06-29T14:22:36.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.633+0000 7fd2c8e38640 1 -- 172.21.15.59:0/2617274457 shutdown_connections 2024-06-29T14:22:36.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.633+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/2617274457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c4162800 0x7fd2c4162c10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.633+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/2617274457 >> v2:172.21.15.153:3300/0 conn(0x7fd2c40ce950 0x7fd2c41622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.633+0000 7fd2c8e38640 1 -- 172.21.15.59:0/2617274457 >> 172.21.15.59:0/2617274457 conn(0x7fd2c40b5610 msgr2=0x7fd2c40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:36.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.633+0000 7fd2c8e38640 1 -- 172.21.15.59:0/2617274457 shutdown_connections 2024-06-29T14:22:36.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.633+0000 7fd2c8e38640 1 -- 172.21.15.59:0/2617274457 wait complete. 2024-06-29T14:22:36.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.635+0000 7fd2c8e38640 1 Processor -- start 2024-06-29T14:22:36.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.635+0000 7fd2c8e38640 1 -- start start 2024-06-29T14:22:36.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.636+0000 7fd2c8e38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c408c190 0x7fd2c4155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.636+0000 7fd2c8e38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c40ce950 0x7fd2c40cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.636+0000 7fd2c8e38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c4162800 0x7fd2c40c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c8e38640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c40b7f60 con 0x7fd2c40ce950 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c8e38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c40b7de0 con 0x7fd2c408c190 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c8e38640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c40b80e0 con 0x7fd2c4162800 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c40ce950 0x7fd2c40cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c40ce950 0x7fd2c40cbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38202/0 (socket says 172.21.15.59:38202) 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 -- 172.21.15.59:0/664588350 learned_addr learned my addr 172.21.15.59:0/664588350 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c2d9d640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c4162800 0x7fd2c40c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c259c640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c408c190 0x7fd2c4155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 -- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c4162800 msgr2=0x7fd2c40c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c4162800 0x7fd2c40c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 -- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c408c190 msgr2=0x7fd2c4155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c408c190 0x7fd2c4155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 -- 172.21.15.59:0/664588350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c40cc030 con 0x7fd2c40ce950 2024-06-29T14:22:36.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c1d9b640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c40ce950 0x7fd2c40cbaf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00c9a0 tx=0x7fd2ac00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:36.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2baffd640 1 -- 172.21.15.59:0/664588350 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac007bb0 con 0x7fd2c40ce950 2024-06-29T14:22:36.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2baffd640 1 -- 172.21.15.59:0/664588350 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2ac007e40 con 0x7fd2c40ce950 2024-06-29T14:22:36.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c40cad70 con 0x7fd2c40ce950 2024-06-29T14:22:36.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2baffd640 1 -- 172.21.15.59:0/664588350 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac00f6b0 con 0x7fd2c40ce950 2024-06-29T14:22:36.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.637+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c40cb010 con 0x7fd2c40ce950 2024-06-29T14:22:36.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.638+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c40cff20 con 0x7fd2c40ce950 2024-06-29T14:22:36.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.639+0000 7fd2baffd640 1 -- 172.21.15.59:0/664588350 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd2ac018400 con 0x7fd2c40ce950 2024-06-29T14:22:36.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.640+0000 7fd2baffd640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd2a4076300 0x7fd2a4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.640+0000 7fd2baffd640 1 -- 172.21.15.59:0/664588350 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd2ac0982f0 con 0x7fd2c40ce950 2024-06-29T14:22:36.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.640+0000 7fd2c259c640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd2a4076300 0x7fd2a4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.640+0000 7fd2c259c640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd2a4076300 0x7fd2a4078810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc007fd0 tx=0x7fd2bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.644+0000 7fd2baffd640 1 -- 172.21.15.59:0/664588350 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd2ac014030 con 0x7fd2c40ce950 2024-06-29T14:22:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.838+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2c41622c0 con 0x7fd2c40ce950 2024-06-29T14:22:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.839+0000 7fd2baffd640 1 -- 172.21.15.59:0/664588350 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7fd2ac0617b0 con 0x7fd2c40ce950 2024-06-29T14:22:36.838 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:36.838 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd2a4076300 msgr2=0x7fd2a4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd2a4076300 0x7fd2a4078810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc007fd0 tx=0x7fd2bc009040 comp rx=0 tx=0).stop 2024-06-29T14:22:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c40ce950 msgr2=0x7fd2c40cbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c40ce950 0x7fd2c40cbaf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00c9a0 tx=0x7fd2ac00ce60 comp rx=0 tx=0).stop 2024-06-29T14:22:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 shutdown_connections 2024-06-29T14:22:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd2a4076300 0x7fd2a4078810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c4162800 0x7fd2c40c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c40ce950 0x7fd2c40cbaf0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 --2- 172.21.15.59:0/664588350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c408c190 0x7fd2c4155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 >> 172.21.15.59:0/664588350 conn(0x7fd2c40b5610 msgr2=0x7fd2c40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 shutdown_connections 2024-06-29T14:22:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.841+0000 7fd2c8e38640 1 -- 172.21.15.59:0/664588350 wait complete. 2024-06-29T14:22:36.852 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:36.852 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:36.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.974+0000 7ff66ff4a640 1 -- 172.21.15.59:0/2669929804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff654004d40 msgr2=0x7ff654005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.974+0000 7ff66ff4a640 1 --2- 172.21.15.59:0/2669929804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff654004d40 0x7ff654005210 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff6680b7620 tx=0x7ff654005830 comp rx=0 tx=0).stop 2024-06-29T14:22:36.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.974+0000 7ff66e4e7640 1 -- 172.21.15.59:0/2669929804 reap_dead start 2024-06-29T14:22:36.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.974+0000 7ff66ff4a640 1 -- 172.21.15.59:0/2669929804 shutdown_connections 2024-06-29T14:22:36.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.975+0000 7ff66ff4a640 1 -- 172.21.15.59:0/2669929804 >> 172.21.15.59:0/2669929804 conn(0x7ff6680b5610 msgr2=0x7ff6680b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:36.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.975+0000 7ff66ff4a640 1 -- 172.21.15.59:0/2669929804 shutdown_connections 2024-06-29T14:22:36.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.975+0000 7ff66ff4a640 1 -- 172.21.15.59:0/2669929804 wait complete. 2024-06-29T14:22:36.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.976+0000 7ff66ff4a640 1 Processor -- start 2024-06-29T14:22:36.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.976+0000 7ff66ff4a640 1 -- start start 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66ff4a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff66814bd10 0x7ff66815a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66ff4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff66815a7d0 0x7ff6681566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66ff4a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff66814c120 0x7ff668156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66ff4a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff6680cf410 con 0x7ff66815a7d0 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66ff4a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6680cf290 con 0x7ff66814c120 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66ff4a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff6680b7bd0 con 0x7ff66814bd10 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff66815a7d0 0x7ff6681566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff66815a7d0 0x7ff6681566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38208/0 (socket says 172.21.15.59:38208) 2024-06-29T14:22:36.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 -- 172.21.15.59:0/3221879727 learned_addr learned my addr 172.21.15.59:0/3221879727 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66dce6640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff66814bd10 0x7ff66815a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66e4e7640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff66814c120 0x7ff668156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 -- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff66814bd10 msgr2=0x7ff66815a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff66814bd10 0x7ff66815a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 -- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff66814c120 msgr2=0x7ff668156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff66814c120 0x7ff668156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 -- 172.21.15.59:0/3221879727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff668159270 con 0x7ff66815a7d0 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66e4e7640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff66814c120 0x7ff668156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.978+0000 7ff66d4e5640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff66815a7d0 0x7ff6681566d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff65800f9d0 tx=0x7ff65800fe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.979+0000 7ff65e7fc640 1 -- 172.21.15.59:0/3221879727 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658015070 con 0x7ff66815a7d0 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.979+0000 7ff65e7fc640 1 -- 172.21.15.59:0/3221879727 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff65800b960 con 0x7ff66815a7d0 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.979+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff668159550 con 0x7ff66815a7d0 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.979+0000 7ff65e7fc640 1 -- 172.21.15.59:0/3221879727 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65800bd70 con 0x7ff66815a7d0 2024-06-29T14:22:36.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.979+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff668159b10 con 0x7ff66815a7d0 2024-06-29T14:22:36.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.980+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff668159ef0 con 0x7ff66815a7d0 2024-06-29T14:22:36.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.980+0000 7ff65e7fc640 1 -- 172.21.15.59:0/3221879727 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff6580035f0 con 0x7ff66815a7d0 2024-06-29T14:22:36.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.981+0000 7ff65e7fc640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff64c0762d0 0x7ff64c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:36.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.981+0000 7ff65e7fc640 1 -- 172.21.15.59:0/3221879727 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7ff658098760 con 0x7ff66815a7d0 2024-06-29T14:22:36.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.981+0000 7ff66dce6640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff64c0762d0 0x7ff64c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:36.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.982+0000 7ff66dce6640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff64c0762d0 0x7ff64c0787e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff664003700 tx=0x7ff66400b260 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:36.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:36.986+0000 7ff65e7fc640 1 -- 172.21.15.59:0/3221879727 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff668159ef0 con 0x7ff66815a7d0 2024-06-29T14:22:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.176+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff6680d1a20 con 0x7ff66815a7d0 2024-06-29T14:22:37.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.177+0000 7ff65e7fc640 1 -- 172.21.15.59:0/3221879727 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7ff658061cc0 con 0x7ff66815a7d0 2024-06-29T14:22:37.176 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:37.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff64c0762d0 msgr2=0x7ff64c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff64c0762d0 0x7ff64c0787e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff664003700 tx=0x7ff66400b260 comp rx=0 tx=0).stop 2024-06-29T14:22:37.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff66815a7d0 msgr2=0x7ff6681566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff66815a7d0 0x7ff6681566d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff65800f9d0 tx=0x7ff65800fe90 comp rx=0 tx=0).stop 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 shutdown_connections 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff64c0762d0 0x7ff64c0787e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff66814c120 0x7ff668156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff66815a7d0 0x7ff6681566d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 --2- 172.21.15.59:0/3221879727 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff66814bd10 0x7ff66815a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 >> 172.21.15.59:0/3221879727 conn(0x7ff6680b5610 msgr2=0x7ff6681582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 shutdown_connections 2024-06-29T14:22:37.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.179+0000 7ff66ff4a640 1 -- 172.21.15.59:0/3221879727 wait complete. 2024-06-29T14:22:37.189 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:37.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.311+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/3622626653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc681458a0 msgr2=0x7fcc680cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.311+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/3622626653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc681458a0 0x7fcc680cf870 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcc60005fd0 tx=0x7fcc6002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.311+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/3622626653 shutdown_connections 2024-06-29T14:22:37.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.311+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/3622626653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc680b3740 0x7fcc680b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.311+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/3622626653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc681458a0 0x7fcc680cf870 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.311+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/3622626653 >> 172.21.15.59:0/3622626653 conn(0x7fcc6806c950 msgr2=0x7fcc68073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.311+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/3622626653 shutdown_connections 2024-06-29T14:22:37.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.311+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/3622626653 wait complete. 2024-06-29T14:22:37.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.313+0000 7fcc6cc8f640 1 Processor -- start 2024-06-29T14:22:37.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.313+0000 7fcc6cc8f640 1 -- start start 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc6cc8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc680b3740 0x7fcc68134720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc6cc8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6815c2f0 0x7fcc680b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc6cc8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc6815b2e0 0x7fcc6815e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc6cc8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcc68160020 con 0x7fcc6815c2f0 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc6cc8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6815cbb0 con 0x7fcc680b3740 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc6cc8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcc681601a0 con 0x7fcc6815b2e0 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc65d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6815c2f0 0x7fcc680b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc65d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6815c2f0 0x7fcc680b0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38228/0 (socket says 172.21.15.59:38228) 2024-06-29T14:22:37.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.314+0000 7fcc65d9b640 1 -- 172.21.15.59:0/569332208 learned_addr learned my addr 172.21.15.59:0/569332208 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc6659c640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc680b3740 0x7fcc68134720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc66d9d640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc6815b2e0 0x7fcc6815e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc65d9b640 1 -- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc6815b2e0 msgr2=0x7fcc6815e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc65d9b640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc6815b2e0 0x7fcc6815e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc65d9b640 1 -- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc680b3740 msgr2=0x7fcc68134720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc65d9b640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc680b3740 0x7fcc68134720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc65d9b640 1 -- 172.21.15.59:0/569332208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc681313d0 con 0x7fcc6815c2f0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc65d9b640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6815c2f0 0x7fcc680b0c90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000d770 tx=0x7fcc5000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc5effd640 1 -- 172.21.15.59:0/569332208 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50017070 con 0x7fcc6815c2f0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc5effd640 1 -- 172.21.15.59:0/569332208 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc50009d20 con 0x7fcc6815c2f0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc681316b0 con 0x7fcc6815c2f0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc5effd640 1 -- 172.21.15.59:0/569332208 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc500106c0 con 0x7fcc6815c2f0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.315+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc68131c70 con 0x7fcc6815c2f0 2024-06-29T14:22:37.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.316+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc681606c0 con 0x7fcc6815c2f0 2024-06-29T14:22:37.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.317+0000 7fcc5effd640 1 -- 172.21.15.59:0/569332208 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcc5001a070 con 0x7fcc6815c2f0 2024-06-29T14:22:37.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.317+0000 7fcc5effd640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcc480762d0 0x7fcc480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.317+0000 7fcc5effd640 1 -- 172.21.15.59:0/569332208 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fcc50097c50 con 0x7fcc6815c2f0 2024-06-29T14:22:37.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.318+0000 7fcc6659c640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcc480762d0 0x7fcc480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.318+0000 7fcc6659c640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcc480762d0 0x7fcc480787e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcc60034000 tx=0x7fcc60004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.322+0000 7fcc5effd640 1 -- 172.21.15.59:0/569332208 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc5009d020 con 0x7fcc6815c2f0 2024-06-29T14:22:37.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.511+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcc680b1f10 con 0x7fcc6815c2f0 2024-06-29T14:22:37.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.512+0000 7fcc5effd640 1 -- 172.21.15.59:0/569332208 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7fcc500611b0 con 0x7fcc6815c2f0 2024-06-29T14:22:37.511 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcc480762d0 msgr2=0x7fcc480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcc480762d0 0x7fcc480787e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcc60034000 tx=0x7fcc60004ab0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6815c2f0 msgr2=0x7fcc680b0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6815c2f0 0x7fcc680b0c90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000d770 tx=0x7fcc5000dc30 comp rx=0 tx=0).stop 2024-06-29T14:22:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 shutdown_connections 2024-06-29T14:22:37.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcc480762d0 0x7fcc480787e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc6815b2e0 0x7fcc6815e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6815c2f0 0x7fcc680b0c90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 --2- 172.21.15.59:0/569332208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc680b3740 0x7fcc68134720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 >> 172.21.15.59:0/569332208 conn(0x7fcc6806c950 msgr2=0x7fcc68144f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 shutdown_connections 2024-06-29T14:22:37.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.514+0000 7fcc6cc8f640 1 -- 172.21.15.59:0/569332208 wait complete. 2024-06-29T14:22:37.524 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.646+0000 7f1cc08e4640 1 -- 172.21.15.59:0/3908239074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cbc144760 msgr2=0x7f1cbc0734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.646+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/3908239074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cbc144760 0x7f1cbc0734c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4009830 tx=0x7f1ca402a510 comp rx=0 tx=0).stop 2024-06-29T14:22:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.646+0000 7f1cc08e4640 1 -- 172.21.15.59:0/3908239074 shutdown_connections 2024-06-29T14:22:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.646+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/3908239074 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cbc073a00 0x7f1cbc06ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.646+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/3908239074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cbc144760 0x7f1cbc0734c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.646+0000 7f1cc08e4640 1 -- 172.21.15.59:0/3908239074 >> 172.21.15.59:0/3908239074 conn(0x7f1cbc06cef0 msgr2=0x7f1cbc06d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.646+0000 7f1cc08e4640 1 -- 172.21.15.59:0/3908239074 shutdown_connections 2024-06-29T14:22:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.646+0000 7f1cc08e4640 1 -- 172.21.15.59:0/3908239074 wait complete. 2024-06-29T14:22:37.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.648+0000 7f1cc08e4640 1 Processor -- start 2024-06-29T14:22:37.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.648+0000 7f1cc08e4640 1 -- start start 2024-06-29T14:22:37.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cc08e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cbc073a00 0x7f1cbc0a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cc08e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cbc0ce6c0 0x7f1cbc1f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cc08e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cbc144760 0x7f1cbc1f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cc08e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc06f920 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cc08e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc06f7a0 con 0x7f1cbc073a00 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cc08e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1cbc06faa0 con 0x7f1cbc144760 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cb1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cbc0ce6c0 0x7f1cbc1f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cb1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cbc0ce6c0 0x7f1cbc1f34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38244/0 (socket says 172.21.15.59:38244) 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cb1d9b640 1 -- 172.21.15.59:0/983917615 learned_addr learned my addr 172.21.15.59:0/983917615 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cba59c640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cbc073a00 0x7f1cbc0a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.649+0000 7f1cbad9d640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cbc144760 0x7f1cbc1f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb1d9b640 1 -- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cbc144760 msgr2=0x7f1cbc1f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb1d9b640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cbc144760 0x7f1cbc1f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb1d9b640 1 -- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cbc073a00 msgr2=0x7f1cbc0a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb1d9b640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cbc073a00 0x7f1cbc0a9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb1d9b640 1 -- 172.21.15.59:0/983917615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cbc1333c0 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb1d9b640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cbc0ce6c0 0x7f1cbc1f34e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1ca402ab00 tx=0x7f1ca4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb37fe640 1 -- 172.21.15.59:0/983917615 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4031d60 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb37fe640 1 -- 172.21.15.59:0/983917615 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ca4033e10 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cb37fe640 1 -- 172.21.15.59:0/983917615 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4005260 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cbc1336a0 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.650+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cbc133c60 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.651+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cbc0cdb00 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.652+0000 7f1cb37fe640 1 -- 172.21.15.59:0/983917615 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ca4005400 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.653+0000 7f1cb37fe640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1c9c0762d0 0x7f1c9c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.653+0000 7f1cb37fe640 1 -- 172.21.15.59:0/983917615 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f1ca40b8be0 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.653+0000 7f1cba59c640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1c9c0762d0 0x7f1c9c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.653+0000 7f1cba59c640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1c9c0762d0 0x7f1c9c0787e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0ce430 tx=0x7f1cac007eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.657+0000 7f1cb37fe640 1 -- 172.21.15.59:0/983917615 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1cbc0cdb00 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.848+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1cbc133150 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.849+0000 7f1cb37fe640 1 -- 172.21.15.59:0/983917615 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f1ca403f030 con 0x7f1cbc0ce6c0 2024-06-29T14:22:37.848 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:37.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1c9c0762d0 msgr2=0x7f1c9c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1c9c0762d0 0x7f1c9c0787e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0ce430 tx=0x7f1cac007eb0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cbc0ce6c0 msgr2=0x7f1cbc1f34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cbc0ce6c0 0x7f1cbc1f34e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1ca402ab00 tx=0x7f1ca4033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:37.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 shutdown_connections 2024-06-29T14:22:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1c9c0762d0 0x7f1c9c0787e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cbc144760 0x7f1cbc1f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cbc0ce6c0 0x7f1cbc1f34e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 --2- 172.21.15.59:0/983917615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cbc073a00 0x7f1cbc0a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 >> 172.21.15.59:0/983917615 conn(0x7f1cbc06cef0 msgr2=0x7f1cbc142430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 shutdown_connections 2024-06-29T14:22:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.851+0000 7f1cc08e4640 1 -- 172.21.15.59:0/983917615 wait complete. 2024-06-29T14:22:37.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 -- 172.21.15.59:0/106467324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 msgr2=0x7fcb6c0cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 --2- 172.21.15.59:0/106467324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 0x7fcb6c0cfd00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c074220 tx=0x7fcb6402a560 comp rx=0 tx=0).stop 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 -- 172.21.15.59:0/106467324 shutdown_connections 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 --2- 172.21.15.59:0/106467324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 0x7fcb6c0cfd00 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 --2- 172.21.15.59:0/106467324 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6c161960 0x7fcb6c161d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 --2- 172.21.15.59:0/106467324 >> v2:172.21.15.153:3300/0 conn(0x7fcb6c0afda0 0x7fcb6c0ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 -- 172.21.15.59:0/106467324 >> 172.21.15.59:0/106467324 conn(0x7fcb6c0b2ab0 msgr2=0x7fcb6c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 -- 172.21.15.59:0/106467324 shutdown_connections 2024-06-29T14:22:37.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.982+0000 7fcb737ee640 1 -- 172.21.15.59:0/106467324 wait complete. 2024-06-29T14:22:37.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.984+0000 7fcb737ee640 1 Processor -- start 2024-06-29T14:22:37.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.984+0000 7fcb737ee640 1 -- start start 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.985+0000 7fcb737ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c0afda0 0x7fcb6c06fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.985+0000 7fcb737ee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 0x7fcb6c159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb737ee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6c161960 0x7fcb6c15a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb737ee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c0b4c00 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb737ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c0b4a80 con 0x7fcb6c0afda0 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb737ee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6c0b4d80 con 0x7fcb6c161960 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 0x7fcb6c159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 0x7fcb6c159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38274/0 (socket says 172.21.15.59:38274) 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 -- 172.21.15.59:0/1929852661 learned_addr learned my addr 172.21.15.59:0/1929852661 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:37.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb71d8b640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6c161960 0x7fcb6c15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb7158a640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c0afda0 0x7fcb6c06fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 -- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6c161960 msgr2=0x7fcb6c15a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6c161960 0x7fcb6c15a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 -- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c0afda0 msgr2=0x7fcb6c06fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c0afda0 0x7fcb6c06fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 -- 172.21.15.59:0/1929852661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c0c5900 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb70d89640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 0x7fcb6c159b20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000c740 tx=0x7fcb6000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb69ffb640 1 -- 172.21.15.59:0/1929852661 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60004220 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb69ffb640 1 -- 172.21.15.59:0/1929852661 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb6000ce50 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c06d3e0 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.986+0000 7fcb69ffb640 1 -- 172.21.15.59:0/1929852661 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60007c10 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.987+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c06d980 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.988+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb6c0c5ad0 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.988+0000 7fcb69ffb640 1 -- 172.21.15.59:0/1929852661 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcb600035f0 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.989+0000 7fcb69ffb640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb580762d0 0x7fcb580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.989+0000 7fcb69ffb640 1 -- 172.21.15.59:0/1929852661 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fcb60097fd0 con 0x7fcb6c0cf8f0 2024-06-29T14:22:37.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.990+0000 7fcb7158a640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb580762d0 0x7fcb580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.990+0000 7fcb7158a640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb580762d0 0x7fcb580787e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c005fd0 tx=0x7fcb5c00a4d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:37.994+0000 7fcb69ffb640 1 -- 172.21.15.59:0/1929852661 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcb6c0c5ad0 con 0x7fcb6c0cf8f0 2024-06-29T14:22:38.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.185+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcb6c0b52a0 con 0x7fcb6c0cf8f0 2024-06-29T14:22:38.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.185+0000 7fcb69ffb640 1 -- 172.21.15.59:0/1929852661 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fcb60014030 con 0x7fcb6c0cf8f0 2024-06-29T14:22:38.184 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:38.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb580762d0 msgr2=0x7fcb580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb580762d0 0x7fcb580787e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c005fd0 tx=0x7fcb5c00a4d0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 msgr2=0x7fcb6c159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 0x7fcb6c159b20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000c740 tx=0x7fcb6000cc00 comp rx=0 tx=0).stop 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 shutdown_connections 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb580762d0 0x7fcb580787e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb6c161960 0x7fcb6c15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb6c0cf8f0 0x7fcb6c159b20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 --2- 172.21.15.59:0/1929852661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6c0afda0 0x7fcb6c06fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 >> 172.21.15.59:0/1929852661 conn(0x7fcb6c0b2ab0 msgr2=0x7fcb6c1581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 shutdown_connections 2024-06-29T14:22:38.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.187+0000 7fcb737ee640 1 -- 172.21.15.59:0/1929852661 wait complete. 2024-06-29T14:22:38.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:38.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.317+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/4056406490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc0b3ed0 msgr2=0x7fa3cc15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.317+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/4056406490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc0b3ed0 0x7fa3cc15e8d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0d15d0 tx=0x7fa3bc02a530 comp rx=0 tx=0).stop 2024-06-29T14:22:38.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.317+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/4056406490 shutdown_connections 2024-06-29T14:22:38.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.317+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/4056406490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc0b3ed0 0x7fa3cc15e8d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.317+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/4056406490 >> v2:172.21.15.153:3300/0 conn(0x7fa3cc0d23c0 0x7fa3cc0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.317+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/4056406490 >> 172.21.15.59:0/4056406490 conn(0x7fa3cc0d3ee0 msgr2=0x7fa3cc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:38.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.317+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/4056406490 shutdown_connections 2024-06-29T14:22:38.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.317+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/4056406490 wait complete. 2024-06-29T14:22:38.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.319+0000 7fa3d3b4a640 1 Processor -- start 2024-06-29T14:22:38.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.319+0000 7fa3d3b4a640 1 -- start start 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.320+0000 7fa3d3b4a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3cc0b3ed0 0x7fa3cc0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.320+0000 7fa3d3b4a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3cc0d23c0 0x7fa3cc1559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.320+0000 7fa3d3b4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc15ee10 0x7fa3cc0c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.320+0000 7fa3d3b4a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc160240 con 0x7fa3cc15ee10 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.320+0000 7fa3d3b4a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc1600c0 con 0x7fa3cc0d23c0 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.320+0000 7fa3d3b4a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa3cc1603c0 con 0x7fa3cc0b3ed0 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc15ee10 0x7fa3cc0c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc15ee10 0x7fa3cc0c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38294/0 (socket says 172.21.15.59:38294) 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 -- 172.21.15.59:0/3834613922 learned_addr learned my addr 172.21.15.59:0/3834613922 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d10e5640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3cc0d23c0 0x7fa3cc1559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d18e6640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3cc0b3ed0 0x7fa3cc0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 -- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3cc0b3ed0 msgr2=0x7fa3cc0c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3cc0b3ed0 0x7fa3cc0c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 -- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3cc0d23c0 msgr2=0x7fa3cc1559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3cc0d23c0 0x7fa3cc1559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 -- 172.21.15.59:0/3834613922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc0c7ef0 con 0x7fa3cc15ee10 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d20e7640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc15ee10 0x7fa3cc0c9d70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400ca70 tx=0x7fa3c400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3c27fc640 1 -- 172.21.15.59:0/3834613922 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4016020 con 0x7fa3cc15ee10 2024-06-29T14:22:38.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3c27fc640 1 -- 172.21.15.59:0/3834613922 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3c400acc0 con 0x7fa3cc15ee10 2024-06-29T14:22:38.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3c27fc640 1 -- 172.21.15.59:0/3834613922 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4012680 con 0x7fa3cc15ee10 2024-06-29T14:22:38.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3cc0c81d0 con 0x7fa3cc15ee10 2024-06-29T14:22:38.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.321+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3cc0c8790 con 0x7fa3cc15ee10 2024-06-29T14:22:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.323+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3cc0c7c60 con 0x7fa3cc15ee10 2024-06-29T14:22:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.323+0000 7fa3c27fc640 1 -- 172.21.15.59:0/3834613922 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa3c4019020 con 0x7fa3cc15ee10 2024-06-29T14:22:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.324+0000 7fa3c27fc640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa3b00762d0 0x7fa3b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.324+0000 7fa3c27fc640 1 -- 172.21.15.59:0/3834613922 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fa3c4097d30 con 0x7fa3cc15ee10 2024-06-29T14:22:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.324+0000 7fa3d18e6640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa3b00762d0 0x7fa3b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.324+0000 7fa3d18e6640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa3b00762d0 0x7fa3b00787e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0d2e40 tx=0x7fa3c8005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:38.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.328+0000 7fa3c27fc640 1 -- 172.21.15.59:0/3834613922 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa3c4020030 con 0x7fa3cc15ee10 2024-06-29T14:22:38.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.518+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3cc0c7370 con 0x7fa3cc15ee10 2024-06-29T14:22:38.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.519+0000 7fa3c27fc640 1 -- 172.21.15.59:0/3834613922 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7fa3cc0c7370 con 0x7fa3cc15ee10 2024-06-29T14:22:38.518 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:38.518 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:38.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa3b00762d0 msgr2=0x7fa3b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa3b00762d0 0x7fa3b00787e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0d2e40 tx=0x7fa3c8005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:38.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc15ee10 msgr2=0x7fa3cc0c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc15ee10 0x7fa3cc0c9d70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400ca70 tx=0x7fa3c400cf30 comp rx=0 tx=0).stop 2024-06-29T14:22:38.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 shutdown_connections 2024-06-29T14:22:38.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa3b00762d0 0x7fa3b00787e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3cc15ee10 0x7fa3cc0c9d70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3cc0d23c0 0x7fa3cc1559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 --2- 172.21.15.59:0/3834613922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3cc0b3ed0 0x7fa3cc0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 >> 172.21.15.59:0/3834613922 conn(0x7fa3cc0d3ee0 msgr2=0x7fa3cc157600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:38.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 shutdown_connections 2024-06-29T14:22:38.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.521+0000 7fa3d3b4a640 1 -- 172.21.15.59:0/3834613922 wait complete. 2024-06-29T14:22:38.531 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:38.532 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.653+0000 7f043f33c640 1 -- 172.21.15.59:0/3068799594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043808c190 msgr2=0x7f04380cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.653+0000 7f043f33c640 1 --2- 172.21.15.59:0/3068799594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043808c190 0x7f04380cf080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0434009860 tx=0x7f043402a4b0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.653+0000 7f043f33c640 1 -- 172.21.15.59:0/3068799594 shutdown_connections 2024-06-29T14:22:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.653+0000 7f043f33c640 1 --2- 172.21.15.59:0/3068799594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043808c190 0x7f04380cf080 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.653+0000 7f043f33c640 1 -- 172.21.15.59:0/3068799594 >> 172.21.15.59:0/3068799594 conn(0x7f04380b5610 msgr2=0x7f04380b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:38.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.654+0000 7f043f33c640 1 -- 172.21.15.59:0/3068799594 shutdown_connections 2024-06-29T14:22:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.654+0000 7f043f33c640 1 -- 172.21.15.59:0/3068799594 wait complete. 2024-06-29T14:22:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.655+0000 7f043f33c640 1 Processor -- start 2024-06-29T14:22:38.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.655+0000 7f043f33c640 1 -- start start 2024-06-29T14:22:38.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043f33c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f043808c190 0x7f04380c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043f33c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04380ce950 0x7f043815d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043f33c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04380cf5c0 0x7f043815dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043f33c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0438169690 con 0x7f04380ce950 2024-06-29T14:22:38.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043f33c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0438169510 con 0x7f04380cf5c0 2024-06-29T14:22:38.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043f33c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0438169810 con 0x7f043808c190 2024-06-29T14:22:38.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04380ce950 0x7f043815d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04380ce950 0x7f043815d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38302/0 (socket says 172.21.15.59:38302) 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 -- 172.21.15.59:0/2161500068 learned_addr learned my addr 172.21.15.59:0/2161500068 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043d0d8640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f043808c190 0x7f04380c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043d8d9640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04380cf5c0 0x7f043815dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 -- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f043808c190 msgr2=0x7f04380c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f043808c190 0x7f04380c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 -- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04380cf5c0 msgr2=0x7f043815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04380cf5c0 0x7f043815dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 -- 172.21.15.59:0/2161500068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04380c8e70 con 0x7f04380ce950 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043d8d9640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04380cf5c0 0x7f043815dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:38.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043c8d7640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04380ce950 0x7f043815d730 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f042c00b3c0 tx=0x7f042c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:38.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f0429ffb640 1 -- 172.21.15.59:0/2161500068 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c015020 con 0x7f04380ce950 2024-06-29T14:22:38.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f0429ffb640 1 -- 172.21.15.59:0/2161500068 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f042c011920 con 0x7f04380ce950 2024-06-29T14:22:38.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043815bb20 con 0x7f04380ce950 2024-06-29T14:22:38.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.657+0000 7f0429ffb640 1 -- 172.21.15.59:0/2161500068 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c011db0 con 0x7f04380ce950 2024-06-29T14:22:38.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.658+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043815e300 con 0x7f04380ce950 2024-06-29T14:22:38.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.659+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04380cf080 con 0x7f04380ce950 2024-06-29T14:22:38.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.659+0000 7f0429ffb640 1 -- 172.21.15.59:0/2161500068 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f042c018020 con 0x7f04380ce950 2024-06-29T14:22:38.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.660+0000 7f0429ffb640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f041c076300 0x7f041c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.660+0000 7f0429ffb640 1 -- 172.21.15.59:0/2161500068 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f042c097920 con 0x7f04380ce950 2024-06-29T14:22:38.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.660+0000 7f043d0d8640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f041c076300 0x7f041c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.661+0000 7f043d0d8640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f041c076300 0x7f041c078810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0434034000 tx=0x7f0434004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:38.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.664+0000 7f0429ffb640 1 -- 172.21.15.59:0/2161500068 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f042c01f030 con 0x7f04380ce950 2024-06-29T14:22:38.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.858+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f04380ca160 con 0x7f04380ce950 2024-06-29T14:22:38.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.858+0000 7f0429ffb640 1 -- 172.21.15.59:0/2161500068 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f04380c8e70 con 0x7f04380ce950 2024-06-29T14:22:38.857 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:38.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f041c076300 msgr2=0x7f041c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f041c076300 0x7f041c078810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0434034000 tx=0x7f0434004730 comp rx=0 tx=0).stop 2024-06-29T14:22:38.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04380ce950 msgr2=0x7f043815d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04380ce950 0x7f043815d730 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f042c00b3c0 tx=0x7f042c00b880 comp rx=0 tx=0).stop 2024-06-29T14:22:38.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 shutdown_connections 2024-06-29T14:22:38.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f041c076300 0x7f041c078810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04380cf5c0 0x7f043815dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04380ce950 0x7f043815d730 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 --2- 172.21.15.59:0/2161500068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f043808c190 0x7f04380c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 >> 172.21.15.59:0/2161500068 conn(0x7f04380b5610 msgr2=0x7f04380d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:38.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 shutdown_connections 2024-06-29T14:22:38.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.861+0000 7f043f33c640 1 -- 172.21.15.59:0/2161500068 wait complete. 2024-06-29T14:22:38.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 -- 172.21.15.59:0/565282943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873408c190 msgr2=0x7f87340cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 --2- 172.21.15.59:0/565282943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873408c190 0x7f87340cf080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f87340cfbc0 tx=0x7f872402aa30 comp rx=0 tx=0).stop 2024-06-29T14:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 -- 172.21.15.59:0/565282943 shutdown_connections 2024-06-29T14:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 --2- 172.21.15.59:0/565282943 >> v2:172.21.15.153:3300/0 conn(0x7f87340ce950 0x7f87340d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 --2- 172.21.15.59:0/565282943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87340cf5c0 0x7f87340d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 --2- 172.21.15.59:0/565282943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f873408c190 0x7f87340cf080 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 -- 172.21.15.59:0/565282943 >> 172.21.15.59:0/565282943 conn(0x7f87340b5610 msgr2=0x7f87340b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 -- 172.21.15.59:0/565282943 shutdown_connections 2024-06-29T14:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.992+0000 7f873bbc7640 1 -- 172.21.15.59:0/565282943 wait complete. 2024-06-29T14:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.994+0000 7f873bbc7640 1 Processor -- start 2024-06-29T14:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.994+0000 7f873bbc7640 1 -- start start 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f873bbc7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f873408c190 0x7f87341e9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f873bbc7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87340ce950 0x7f87340ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f873bbc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87340cf5c0 0x7f87340cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f873bbc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8734162440 con 0x7f87340cf5c0 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f873bbc7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87341622c0 con 0x7f873408c190 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f873bbc7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f87341625c0 con 0x7f87340ce950 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f873a164640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87340cf5c0 0x7f87340cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f8739963640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f873408c190 0x7f87341e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f8739963640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f873408c190 0x7f87341e9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:54358/0 (socket says 172.21.15.59:54358) 2024-06-29T14:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f873a164640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87340cf5c0 0x7f87340cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38322/0 (socket says 172.21.15.59:38322) 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f8739162640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87340ce950 0x7f87340ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.995+0000 7f8739963640 1 -- 172.21.15.59:0/3372963127 learned_addr learned my addr 172.21.15.59:0/3372963127 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f873a164640 1 -- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87340ce950 msgr2=0x7f87340ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f873a164640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87340ce950 0x7f87340ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f873a164640 1 -- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f873408c190 msgr2=0x7f87341e9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f873a164640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f873408c190 0x7f87341e9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f873a164640 1 -- 172.21.15.59:0/3372963127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87341e7a50 con 0x7f87340cf5c0 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f873a164640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87340cf5c0 0x7f87340cd2e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f872c00b390 tx=0x7f872c00b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f87327fc640 1 -- 172.21.15.59:0/3372963127 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c015020 con 0x7f87340cf5c0 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f87327fc640 1 -- 172.21.15.59:0/3372963127 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f872c011920 con 0x7f87340cf5c0 2024-06-29T14:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87341e7d30 con 0x7f87340cf5c0 2024-06-29T14:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f87327fc640 1 -- 172.21.15.59:0/3372963127 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c011d70 con 0x7f87340cf5c0 2024-06-29T14:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.996+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87341e82f0 con 0x7f87340cf5c0 2024-06-29T14:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.997+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87340cf080 con 0x7f87340cf5c0 2024-06-29T14:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.998+0000 7f87327fc640 1 -- 172.21.15.59:0/3372963127 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f872c018020 con 0x7f87340cf5c0 2024-06-29T14:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.998+0000 7f87327fc640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f87200762d0 0x7f87200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.998+0000 7f87327fc640 1 -- 172.21.15.59:0/3372963127 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f872c097970 con 0x7f87340cf5c0 2024-06-29T14:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.999+0000 7f8739963640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f87200762d0 0x7f87200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:38.999+0000 7f8739963640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f87200762d0 0x7f87200787e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8724000f80 tx=0x7f872403b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:39.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.003+0000 7f87327fc640 1 -- 172.21.15.59:0/3372963127 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f872c01f020 con 0x7f87340cf5c0 2024-06-29T14:22:39.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.190+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f87340b4d80 con 0x7f87340cf5c0 2024-06-29T14:22:39.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.191+0000 7f87327fc640 1 -- 172.21.15.59:0/3372963127 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f872c060880 con 0x7f87340cf5c0 2024-06-29T14:22:39.190 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:39.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f87200762d0 msgr2=0x7f87200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f87200762d0 0x7f87200787e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8724000f80 tx=0x7f872403b040 comp rx=0 tx=0).stop 2024-06-29T14:22:39.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87340cf5c0 msgr2=0x7f87340cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87340cf5c0 0x7f87340cd2e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f872c00b390 tx=0x7f872c00b850 comp rx=0 tx=0).stop 2024-06-29T14:22:39.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 shutdown_connections 2024-06-29T14:22:39.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f87200762d0 0x7f87200787e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87340cf5c0 0x7f87340cd2e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87340ce950 0x7f87340ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 --2- 172.21.15.59:0/3372963127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f873408c190 0x7f87341e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 >> 172.21.15.59:0/3372963127 conn(0x7f87340b5610 msgr2=0x7f87340c8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:39.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 shutdown_connections 2024-06-29T14:22:39.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.193+0000 7f873bbc7640 1 -- 172.21.15.59:0/3372963127 wait complete. 2024-06-29T14:22:39.203 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.326+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3005938094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 msgr2=0x7fb24815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.326+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3005938094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 0x7fb24815bc20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb2480cf310 tx=0x7fb23802a530 comp rx=0 tx=0).stop 2024-06-29T14:22:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.326+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3005938094 shutdown_connections 2024-06-29T14:22:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.326+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3005938094 >> v2:172.21.15.153:3300/0 conn(0x7fb2480d1c30 0x7fb2480b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.326+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3005938094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 0x7fb24815bc20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.326+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3005938094 >> 172.21.15.59:0/3005938094 conn(0x7fb248073e10 msgr2=0x7fb248074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.326+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3005938094 shutdown_connections 2024-06-29T14:22:39.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.326+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3005938094 wait complete. 2024-06-29T14:22:39.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.328+0000 7fb24e4f6640 1 Processor -- start 2024-06-29T14:22:39.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.328+0000 7fb24e4f6640 1 -- start start 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb24e4f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 0x7fb2481f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb24e4f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2480b2fa0 0x7fb24815d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb24e4f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2480d1c30 0x7fb2480ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb24e4f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb2480b6400 con 0x7fb2480b2270 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb24e4f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb2480b6280 con 0x7fb2480b2fa0 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb24e4f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb2480b6580 con 0x7fb2480d1c30 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 0x7fb2481f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 0x7fb2481f32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38344/0 (socket says 172.21.15.59:38344) 2024-06-29T14:22:39.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 -- 172.21.15.59:0/3229779835 learned_addr learned my addr 172.21.15.59:0/3229779835 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb2477fe640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2480b2fa0 0x7fb24815d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb24ca93640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2480d1c30 0x7fb2480ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 -- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2480d1c30 msgr2=0x7fb2480ac5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2480d1c30 0x7fb2480ac5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 -- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2480b2fa0 msgr2=0x7fb24815d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2480b2fa0 0x7fb24815d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 -- 172.21.15.59:0/3229779835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24815dfc0 con 0x7fb2480b2270 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb2477fe640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2480b2fa0 0x7fb24815d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb247fff640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 0x7fb2481f32c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb2380098f0 tx=0x7fb238033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb244ff9640 1 -- 172.21.15.59:0/3229779835 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb238005c90 con 0x7fb2480b2270 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb244ff9640 1 -- 172.21.15.59:0/3229779835 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb238031e50 con 0x7fb2480b2270 2024-06-29T14:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb244ff9640 1 -- 172.21.15.59:0/3229779835 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb238005260 con 0x7fb2480b2270 2024-06-29T14:22:39.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.330+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24815e2a0 con 0x7fb2480b2270 2024-06-29T14:22:39.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.331+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb24815e860 con 0x7fb2480b2270 2024-06-29T14:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.332+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2480b6aa0 con 0x7fb2480b2270 2024-06-29T14:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.332+0000 7fb244ff9640 1 -- 172.21.15.59:0/3229779835 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb238005400 con 0x7fb2480b2270 2024-06-29T14:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.333+0000 7fb244ff9640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb234076300 0x7fb234078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.333+0000 7fb244ff9640 1 -- 172.21.15.59:0/3229779835 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fb2380b8b40 con 0x7fb2480b2270 2024-06-29T14:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.333+0000 7fb2477fe640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb234076300 0x7fb234078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.334+0000 7fb2477fe640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb234076300 0x7fb234078810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb248157840 tx=0x7fb23c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:39.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.339+0000 7fb244ff9640 1 -- 172.21.15.59:0/3229779835 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb23803f030 con 0x7fb2480b2270 2024-06-29T14:22:39.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.528+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb2481f2be0 con 0x7fb2480b2270 2024-06-29T14:22:39.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.528+0000 7fb244ff9640 1 -- 172.21.15.59:0/3229779835 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fb238082130 con 0x7fb2480b2270 2024-06-29T14:22:39.527 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb234076300 msgr2=0x7fb234078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb234076300 0x7fb234078810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb248157840 tx=0x7fb23c009040 comp rx=0 tx=0).stop 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 msgr2=0x7fb2481f32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 0x7fb2481f32c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb2380098f0 tx=0x7fb238033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 shutdown_connections 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb234076300 0x7fb234078810 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2480d1c30 0x7fb2480ac5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2480b2fa0 0x7fb24815d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 --2- 172.21.15.59:0/3229779835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2480b2270 0x7fb2481f32c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 >> 172.21.15.59:0/3229779835 conn(0x7fb248073e10 msgr2=0x7fb248074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 shutdown_connections 2024-06-29T14:22:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.530+0000 7fb24e4f6640 1 -- 172.21.15.59:0/3229779835 wait complete. 2024-06-29T14:22:39.540 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:39.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.662+0000 7efce0f12640 1 -- 172.21.15.59:0/2432585793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcdc06d2f0 msgr2=0x7efcdc06d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.662+0000 7efce0f12640 1 --2- 172.21.15.59:0/2432585793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcdc06d2f0 0x7efcdc06d700 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009830 tx=0x7efcc402a510 comp rx=0 tx=0).stop 2024-06-29T14:22:39.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.662+0000 7efce0f12640 1 -- 172.21.15.59:0/2432585793 shutdown_connections 2024-06-29T14:22:39.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.662+0000 7efce0f12640 1 --2- 172.21.15.59:0/2432585793 >> v2:172.21.15.153:3300/0 conn(0x7efcdc162810 0x7efcdc164c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.662+0000 7efce0f12640 1 --2- 172.21.15.59:0/2432585793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcdc06d2f0 0x7efcdc06d700 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.662+0000 7efce0f12640 1 -- 172.21.15.59:0/2432585793 >> 172.21.15.59:0/2432585793 conn(0x7efcdc0b24c0 msgr2=0x7efcdc0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:39.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.662+0000 7efce0f12640 1 -- 172.21.15.59:0/2432585793 shutdown_connections 2024-06-29T14:22:39.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.662+0000 7efce0f12640 1 -- 172.21.15.59:0/2432585793 wait complete. 2024-06-29T14:22:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.664+0000 7efce0f12640 1 Processor -- start 2024-06-29T14:22:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.664+0000 7efce0f12640 1 -- start start 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.665+0000 7efce0f12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcdc06d2f0 0x7efcdc15d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efce0f12640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcdc0cd9b0 0x7efcdc157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efce0f12640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcdc162810 0x7efcdc13dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efce0f12640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc1448f0 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efce0f12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc144770 con 0x7efcdc06d2f0 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efce0f12640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efcdc165170 con 0x7efcdc162810 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcdc0cd9b0 0x7efcdc157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcdc0cd9b0 0x7efcdc157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38356/0 (socket says 172.21.15.59:38356) 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 -- 172.21.15.59:0/2857811513 learned_addr learned my addr 172.21.15.59:0/2857811513 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcda59c640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcdc06d2f0 0x7efcdc15d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:39.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcdad9d640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcdc162810 0x7efcdc13dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 -- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcdc162810 msgr2=0x7efcdc13dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcdc162810 0x7efcdc13dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 -- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcdc06d2f0 msgr2=0x7efcdc15d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcdc06d2f0 0x7efcdc15d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 -- 172.21.15.59:0/2857811513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcdc15e0d0 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd9d9b640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcdc0cd9b0 0x7efcdc157e30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009950 tx=0x7efcc40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.666+0000 7efcd2ffd640 1 -- 172.21.15.59:0/2857811513 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4005220 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.667+0000 7efcd2ffd640 1 -- 172.21.15.59:0/2857811513 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcc40054b0 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.667+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcdc15db10 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.667+0000 7efcd2ffd640 1 -- 172.21.15.59:0/2857811513 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4004a00 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.667+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcdc162cc0 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.668+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcdc06d700 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.668+0000 7efcd2ffd640 1 -- 172.21.15.59:0/2857811513 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efcc4004ba0 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.669+0000 7efcd2ffd640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efcbc076300 0x7efcbc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:39.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.669+0000 7efcd2ffd640 1 -- 172.21.15.59:0/2857811513 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7efcc40b8d80 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.669+0000 7efcda59c640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efcbc076300 0x7efcbc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:39.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.670+0000 7efcda59c640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efcbc076300 0x7efcbc078810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efcd40013f0 tx=0x7efcd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.673+0000 7efcd2ffd640 1 -- 172.21.15.59:0/2857811513 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efcc403f030 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.866+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efcdc13f090 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.867+0000 7efcd2ffd640 1 -- 172.21.15.59:0/2857811513 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7efcc40822f0 con 0x7efcdc0cd9b0 2024-06-29T14:22:39.866 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efcbc076300 msgr2=0x7efcbc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efcbc076300 0x7efcbc078810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efcd40013f0 tx=0x7efcd4009040 comp rx=0 tx=0).stop 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcdc0cd9b0 msgr2=0x7efcdc157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcdc0cd9b0 0x7efcdc157e30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009950 tx=0x7efcc40040c0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 shutdown_connections 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efcbc076300 0x7efcbc078810 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcdc162810 0x7efcdc13dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcdc0cd9b0 0x7efcdc157e30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 --2- 172.21.15.59:0/2857811513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efcdc06d2f0 0x7efcdc15d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:39.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 >> 172.21.15.59:0/2857811513 conn(0x7efcdc0b24c0 msgr2=0x7efcdc0b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:39.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 shutdown_connections 2024-06-29T14:22:39.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:39.869+0000 7efce0f12640 1 -- 172.21.15.59:0/2857811513 wait complete. 2024-06-29T14:22:39.896 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-06-29T14:22:39.896 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:22:39.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:40.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.016+0000 7f34dfb23640 1 -- 172.21.15.59:0/670132485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c40051f0 msgr2=0x7f34c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.016+0000 7f34dfb23640 1 --2- 172.21.15.59:0/670132485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c40051f0 0x7f34c400b9c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f34d80d3730 tx=0x7f34c4005830 comp rx=0 tx=0).stop 2024-06-29T14:22:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.016+0000 7f34de0c0640 1 -- 172.21.15.59:0/670132485 reap_dead start 2024-06-29T14:22:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.016+0000 7f34dfb23640 1 -- 172.21.15.59:0/670132485 shutdown_connections 2024-06-29T14:22:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.016+0000 7f34dfb23640 1 -- 172.21.15.59:0/670132485 >> 172.21.15.59:0/670132485 conn(0x7f34d80d45b0 msgr2=0x7f34d80d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.016+0000 7f34dfb23640 1 -- 172.21.15.59:0/670132485 shutdown_connections 2024-06-29T14:22:40.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.016+0000 7f34dfb23640 1 -- 172.21.15.59:0/670132485 wait complete. 2024-06-29T14:22:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.018+0000 7f34dfb23640 1 Processor -- start 2024-06-29T14:22:40.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.018+0000 7f34dfb23640 1 -- start start 2024-06-29T14:22:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.019+0000 7f34dfb23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c40051f0 0x7f34d8157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.019+0000 7f34dfb23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34d8154600 0x7f34d8152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.019+0000 7f34dfb23640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8154a10 0x7f34d80cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.019+0000 7f34dfb23640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f34d80ce6d0 con 0x7f34d8154600 2024-06-29T14:22:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.019+0000 7f34dfb23640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34d80ce550 con 0x7f34c40051f0 2024-06-29T14:22:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.019+0000 7f34dfb23640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f34d80ce850 con 0x7f34d8154a10 2024-06-29T14:22:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.019+0000 7f34dd0be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34d8154600 0x7f34d8152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd0be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34d8154600 0x7f34d8152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38376/0 (socket says 172.21.15.59:38376) 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd0be640 1 -- 172.21.15.59:0/2737742317 learned_addr learned my addr 172.21.15.59:0/2737742317 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd8bf640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c40051f0 0x7f34d8157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34de0c0640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8154a10 0x7f34d80cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd0be640 1 -- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8154a10 msgr2=0x7f34d80cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd0be640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8154a10 0x7f34d80cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd0be640 1 -- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c40051f0 msgr2=0x7f34d8157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd0be640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c40051f0 0x7f34d8157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd0be640 1 -- 172.21.15.59:0/2737742317 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d8157740 con 0x7f34d8154600 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34de0c0640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8154a10 0x7f34d80cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dd0be640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34d8154600 0x7f34d8152e80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f34c8011b10 tx=0x7f34c8009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34ce7fc640 1 -- 172.21.15.59:0/2737742317 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c801e070 con 0x7f34d8154600 2024-06-29T14:22:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34ce7fc640 1 -- 172.21.15.59:0/2737742317 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34c800fe40 con 0x7f34d8154600 2024-06-29T14:22:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34ce7fc640 1 -- 172.21.15.59:0/2737742317 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c80042b0 con 0x7f34d8154600 2024-06-29T14:22:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d80cc500 con 0x7f34d8154600 2024-06-29T14:22:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.020+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d80ccac0 con 0x7f34d8154600 2024-06-29T14:22:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.021+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d80b38e0 con 0x7f34d8154600 2024-06-29T14:22:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.022+0000 7f34ce7fc640 1 -- 172.21.15.59:0/2737742317 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f34c8021070 con 0x7f34d8154600 2024-06-29T14:22:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.022+0000 7f34ce7fc640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f34bc076300 0x7f34bc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.023+0000 7f34ce7fc640 1 -- 172.21.15.59:0/2737742317 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f34c80985b0 con 0x7f34d8154600 2024-06-29T14:22:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.023+0000 7f34dd8bf640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f34bc076300 0x7f34bc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.023+0000 7f34dd8bf640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f34bc076300 0x7f34bc078810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f34d4001260 tx=0x7f34d4006720 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.027+0000 7f34ce7fc640 1 -- 172.21.15.59:0/2737742317 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f34c80613b0 con 0x7f34d8154600 2024-06-29T14:22:40.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.219+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34d806c920 con 0x7f34d8154600 2024-06-29T14:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.220+0000 7f34ce7fc640 1 -- 172.21.15.59:0/2737742317 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7f34c80656b0 con 0x7f34d8154600 2024-06-29T14:22:40.220 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:40.220 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:40.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f34bc076300 msgr2=0x7f34bc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f34bc076300 0x7f34bc078810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f34d4001260 tx=0x7f34d4006720 comp rx=0 tx=0).stop 2024-06-29T14:22:40.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34d8154600 msgr2=0x7f34d8152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34d8154600 0x7f34d8152e80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f34c8011b10 tx=0x7f34c8009e00 comp rx=0 tx=0).stop 2024-06-29T14:22:40.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 shutdown_connections 2024-06-29T14:22:40.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f34bc076300 0x7f34bc078810 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34d8154a10 0x7f34d80cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34d8154600 0x7f34d8152e80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 --2- 172.21.15.59:0/2737742317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34c40051f0 0x7f34d8157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 >> 172.21.15.59:0/2737742317 conn(0x7f34d80d45b0 msgr2=0x7f34d80b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 shutdown_connections 2024-06-29T14:22:40.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.223+0000 7f34dfb23640 1 -- 172.21.15.59:0/2737742317 wait complete. 2024-06-29T14:22:40.233 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:40.234 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:40.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.356+0000 7feb2572a640 1 -- 172.21.15.59:0/500343752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb20143840 msgr2=0x7feb2006e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.356+0000 7feb2572a640 1 --2- 172.21.15.59:0/500343752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb20143840 0x7feb2006e4d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feb08005fd0 tx=0x7feb0802a540 comp rx=0 tx=0).stop 2024-06-29T14:22:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.356+0000 7feb2572a640 1 -- 172.21.15.59:0/500343752 shutdown_connections 2024-06-29T14:22:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.356+0000 7feb2572a640 1 --2- 172.21.15.59:0/500343752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb2006ea10 0x7feb200cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.356+0000 7feb2572a640 1 --2- 172.21.15.59:0/500343752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb20143840 0x7feb2006e4d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.356+0000 7feb2572a640 1 -- 172.21.15.59:0/500343752 >> 172.21.15.59:0/500343752 conn(0x7feb200b2ab0 msgr2=0x7feb200b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.356+0000 7feb2572a640 1 -- 172.21.15.59:0/500343752 shutdown_connections 2024-06-29T14:22:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.356+0000 7feb2572a640 1 -- 172.21.15.59:0/500343752 wait complete. 2024-06-29T14:22:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.358+0000 7feb2572a640 1 Processor -- start 2024-06-29T14:22:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.358+0000 7feb2572a640 1 -- start start 2024-06-29T14:22:40.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.359+0000 7feb2572a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb2006ea10 0x7feb201f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.359+0000 7feb2572a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb200d0130 0x7feb2013ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.359+0000 7feb2572a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb20143840 0x7feb201f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.359+0000 7feb2572a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7feb200cf650 con 0x7feb2006ea10 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.359+0000 7feb2572a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb200cf4d0 con 0x7feb20143840 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.359+0000 7feb2572a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7feb200cf7d0 con 0x7feb200d0130 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb2006ea10 0x7feb201f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb2006ea10 0x7feb201f25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38390/0 (socket says 172.21.15.59:38390) 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 -- 172.21.15.59:0/1393705178 learned_addr learned my addr 172.21.15.59:0/1393705178 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1f7fe640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb20143840 0x7feb201f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1e7fc640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb200d0130 0x7feb2013ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 -- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb200d0130 msgr2=0x7feb2013ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb200d0130 0x7feb2013ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 -- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb20143840 msgr2=0x7feb201f5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb20143840 0x7feb201f5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 -- 172.21.15.59:0/1393705178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2015b8d0 con 0x7feb2006ea10 2024-06-29T14:22:40.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb1effd640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb2006ea10 0x7feb201f25a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7feb1800c730 tx=0x7feb1800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb177fe640 1 -- 172.21.15.59:0/1393705178 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18003b70 con 0x7feb2006ea10 2024-06-29T14:22:40.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb177fe640 1 -- 172.21.15.59:0/1393705178 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb18003d30 con 0x7feb2006ea10 2024-06-29T14:22:40.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2015bbb0 con 0x7feb2006ea10 2024-06-29T14:22:40.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb177fe640 1 -- 172.21.15.59:0/1393705178 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1800aab0 con 0x7feb2006ea10 2024-06-29T14:22:40.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.360+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb201f1dc0 con 0x7feb2006ea10 2024-06-29T14:22:40.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.361+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb200cfc20 con 0x7feb2006ea10 2024-06-29T14:22:40.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.362+0000 7feb177fe640 1 -- 172.21.15.59:0/1393705178 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feb1800ac50 con 0x7feb2006ea10 2024-06-29T14:22:40.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.363+0000 7feb177fe640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7feb00076300 0x7feb00078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.363+0000 7feb177fe640 1 -- 172.21.15.59:0/1393705178 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7feb18097a30 con 0x7feb2006ea10 2024-06-29T14:22:40.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.363+0000 7feb1e7fc640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7feb00076300 0x7feb00078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.364+0000 7feb1e7fc640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7feb00076300 0x7feb00078810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feb08033000 tx=0x7feb08004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.367+0000 7feb177fe640 1 -- 172.21.15.59:0/1393705178 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feb18060810 con 0x7feb2006ea10 2024-06-29T14:22:40.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.557+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feb2013d700 con 0x7feb2006ea10 2024-06-29T14:22:40.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.557+0000 7feb177fe640 1 -- 172.21.15.59:0/1393705178 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7feb18064b10 con 0x7feb2006ea10 2024-06-29T14:22:40.556 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7feb00076300 msgr2=0x7feb00078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7feb00076300 0x7feb00078810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feb08033000 tx=0x7feb08004730 comp rx=0 tx=0).stop 2024-06-29T14:22:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb2006ea10 msgr2=0x7feb201f25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb2006ea10 0x7feb201f25a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7feb1800c730 tx=0x7feb1800cbf0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 shutdown_connections 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7feb00076300 0x7feb00078810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb20143840 0x7feb201f5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb200d0130 0x7feb2013ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 --2- 172.21.15.59:0/1393705178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb2006ea10 0x7feb201f25a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 >> 172.21.15.59:0/1393705178 conn(0x7feb200b2ab0 msgr2=0x7feb2013e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 shutdown_connections 2024-06-29T14:22:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.559+0000 7feb2572a640 1 -- 172.21.15.59:0/1393705178 wait complete. 2024-06-29T14:22:40.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.691+0000 7f76c497a640 1 -- 172.21.15.59:0/3307938423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76ac005350 msgr2=0x7f76ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.691+0000 7f76c497a640 1 --2- 172.21.15.59:0/3307938423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76ac005350 0x7f76ac00b9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f76c00cfa80 tx=0x7f76ac005830 comp rx=0 tx=0).stop 2024-06-29T14:22:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.692+0000 7f76c497a640 1 -- 172.21.15.59:0/3307938423 shutdown_connections 2024-06-29T14:22:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.692+0000 7f76c497a640 1 --2- 172.21.15.59:0/3307938423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76ac005350 0x7f76ac00b9c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.692+0000 7f76c497a640 1 -- 172.21.15.59:0/3307938423 >> 172.21.15.59:0/3307938423 conn(0x7f76c00b24c0 msgr2=0x7f76c00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.692+0000 7f76c497a640 1 -- 172.21.15.59:0/3307938423 shutdown_connections 2024-06-29T14:22:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.692+0000 7f76c497a640 1 -- 172.21.15.59:0/3307938423 wait complete. 2024-06-29T14:22:40.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.694+0000 7f76c497a640 1 Processor -- start 2024-06-29T14:22:40.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.694+0000 7f76c497a640 1 -- start start 2024-06-29T14:22:40.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76c497a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac005350 0x7f76c0143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76c497a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76c01f7cf0 0x7f76c015e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76c497a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76c0136d40 0x7f76c015ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76c497a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f76c015f5d0 con 0x7f76c0136d40 2024-06-29T14:22:40.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76c497a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76c015f450 con 0x7f76ac005350 2024-06-29T14:22:40.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76c497a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f76c015f750 con 0x7f76c01f7cf0 2024-06-29T14:22:40.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76c0136d40 0x7f76c015ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76c0136d40 0x7f76c015ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38404/0 (socket says 172.21.15.59:38404) 2024-06-29T14:22:40.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bed9d640 1 -- 172.21.15.59:0/906038364 learned_addr learned my addr 172.21.15.59:0/906038364 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:40.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76be59c640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac005350 0x7f76c0143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bdd9b640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76c01f7cf0 0x7f76c015e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bed9d640 1 -- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76c01f7cf0 msgr2=0x7f76c015e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bed9d640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76c01f7cf0 0x7f76c015e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bed9d640 1 -- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac005350 msgr2=0x7f76c0143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bed9d640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac005350 0x7f76c0143930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.695+0000 7f76bed9d640 1 -- 172.21.15.59:0/906038364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c0141f10 con 0x7f76c0136d40 2024-06-29T14:22:40.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.696+0000 7f76bed9d640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76c0136d40 0x7f76c015ea80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f76b0001780 tx=0x7f76b0006300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.696+0000 7f76b6ffd640 1 -- 172.21.15.59:0/906038364 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0003a00 con 0x7f76c0136d40 2024-06-29T14:22:40.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.696+0000 7f76b6ffd640 1 -- 172.21.15.59:0/906038364 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76b00064f0 con 0x7f76c0136d40 2024-06-29T14:22:40.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.696+0000 7f76b6ffd640 1 -- 172.21.15.59:0/906038364 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0003e40 con 0x7f76c0136d40 2024-06-29T14:22:40.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.696+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c01421f0 con 0x7f76c0136d40 2024-06-29T14:22:40.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.696+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c01427b0 con 0x7f76c0136d40 2024-06-29T14:22:40.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.697+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c0141c80 con 0x7f76c0136d40 2024-06-29T14:22:40.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.698+0000 7f76b6ffd640 1 -- 172.21.15.59:0/906038364 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f76b0030020 con 0x7f76c0136d40 2024-06-29T14:22:40.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.698+0000 7f76b6ffd640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f76a00762d0 0x7f76a00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.698+0000 7f76b6ffd640 1 -- 172.21.15.59:0/906038364 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f76b0038020 con 0x7f76c0136d40 2024-06-29T14:22:40.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.699+0000 7f76be59c640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f76a00762d0 0x7f76a00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.699+0000 7f76be59c640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f76a00762d0 0x7f76a00787e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f76b8002230 tx=0x7f76b8006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.703+0000 7f76b6ffd640 1 -- 172.21.15.59:0/906038364 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76b007feb0 con 0x7f76c0136d40 2024-06-29T14:22:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.898+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f76c006d670 con 0x7f76c0136d40 2024-06-29T14:22:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.899+0000 7f76b6ffd640 1 -- 172.21.15.59:0/906038364 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f76b00841b0 con 0x7f76c0136d40 2024-06-29T14:22:40.898 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:40.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f76a00762d0 msgr2=0x7f76a00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f76a00762d0 0x7f76a00787e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f76b8002230 tx=0x7f76b8006040 comp rx=0 tx=0).stop 2024-06-29T14:22:40.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76c0136d40 msgr2=0x7f76c015ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76c0136d40 0x7f76c015ea80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f76b0001780 tx=0x7f76b0006300 comp rx=0 tx=0).stop 2024-06-29T14:22:40.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 shutdown_connections 2024-06-29T14:22:40.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f76a00762d0 0x7f76a00787e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76c0136d40 0x7f76c015ea80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76c01f7cf0 0x7f76c015e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 --2- 172.21.15.59:0/906038364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76ac005350 0x7f76c0143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 >> 172.21.15.59:0/906038364 conn(0x7f76c00b24c0 msgr2=0x7f76c015b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 shutdown_connections 2024-06-29T14:22:40.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:40.901+0000 7f76c497a640 1 -- 172.21.15.59:0/906038364 wait complete. 2024-06-29T14:22:40.911 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.033+0000 7f8f42352640 1 -- 172.21.15.59:0/4249312171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 msgr2=0x7f8f3c0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.033+0000 7f8f42352640 1 --2- 172.21.15.59:0/4249312171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 0x7f8f3c0b38e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c0d3aa0 tx=0x7f8f3402a920 comp rx=0 tx=0).stop 2024-06-29T14:22:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.033+0000 7f8f42352640 1 -- 172.21.15.59:0/4249312171 shutdown_connections 2024-06-29T14:22:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.033+0000 7f8f42352640 1 --2- 172.21.15.59:0/4249312171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 0x7f8f3c0b38e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.033+0000 7f8f42352640 1 --2- 172.21.15.59:0/4249312171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f3c162190 0x7f8f3c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.033+0000 7f8f42352640 1 -- 172.21.15.59:0/4249312171 >> 172.21.15.59:0/4249312171 conn(0x7f8f3c0d45b0 msgr2=0x7f8f3c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.033+0000 7f8f42352640 1 -- 172.21.15.59:0/4249312171 shutdown_connections 2024-06-29T14:22:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.033+0000 7f8f42352640 1 -- 172.21.15.59:0/4249312171 wait complete. 2024-06-29T14:22:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.035+0000 7f8f42352640 1 Processor -- start 2024-06-29T14:22:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.035+0000 7f8f42352640 1 -- start start 2024-06-29T14:22:41.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f42352640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 0x7f8f3c161c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f42352640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f3c162770 0x7f8f3c15f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f42352640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f3c0b28a0 0x7f8f3c0cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f42352640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c0b43a0 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f42352640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c0b4220 con 0x7f8f3c162770 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f42352640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8f3c0b4520 con 0x7f8f3c0b28a0 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 0x7f8f3c161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 0x7f8f3c161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38430/0 (socket says 172.21.15.59:38430) 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f3bfff640 1 -- 172.21.15.59:0/3740167686 learned_addr learned my addr 172.21.15.59:0/3740167686 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f3b7fe640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f3c162770 0x7f8f3c15f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.036+0000 7f8f408ef640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f3c0b28a0 0x7f8f3c0cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f3bfff640 1 -- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f3c0b28a0 msgr2=0x7f8f3c0cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f3bfff640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f3c0b28a0 0x7f8f3c0cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f3bfff640 1 -- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f3c162770 msgr2=0x7f8f3c15f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f3bfff640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f3c162770 0x7f8f3c15f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f3bfff640 1 -- 172.21.15.59:0/3740167686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3c15e6e0 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f3bfff640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 0x7f8f3c161c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00b3f0 tx=0x7f8f2c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f38ff9640 1 -- 172.21.15.59:0/3740167686 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c015020 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f38ff9640 1 -- 172.21.15.59:0/3740167686 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f2c011920 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f3c15e9c0 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f38ff9640 1 -- 172.21.15.59:0/3740167686 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c011d70 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.037+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f3c0cb3d0 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.038+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f3c0b4a30 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.039+0000 7f8f38ff9640 1 -- 172.21.15.59:0/3740167686 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8f2c018020 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.039+0000 7f8f38ff9640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8f1c0762d0 0x7f8f1c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.039+0000 7f8f38ff9640 1 -- 172.21.15.59:0/3740167686 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f8f2c097ab0 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.039+0000 7f8f3b7fe640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8f1c0762d0 0x7f8f1c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.040+0000 7f8f3b7fe640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8f1c0762d0 0x7f8f1c0787e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c06cea0 tx=0x7f8f2400b2e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.044+0000 7f8f38ff9640 1 -- 172.21.15.59:0/3740167686 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8f2c01f020 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.234+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8f3c06ced0 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.234+0000 7f8f38ff9640 1 -- 172.21.15.59:0/3740167686 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f8f2c060930 con 0x7f8f3c15e0a0 2024-06-29T14:22:41.233 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8f1c0762d0 msgr2=0x7f8f1c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8f1c0762d0 0x7f8f1c0787e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c06cea0 tx=0x7f8f2400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 msgr2=0x7f8f3c161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 0x7f8f3c161c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00b3f0 tx=0x7f8f2c00b8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 shutdown_connections 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8f1c0762d0 0x7f8f1c0787e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f3c0b28a0 0x7f8f3c0cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f3c162770 0x7f8f3c15f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 --2- 172.21.15.59:0/3740167686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f3c15e0a0 0x7f8f3c161c60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 >> 172.21.15.59:0/3740167686 conn(0x7f8f3c0d45b0 msgr2=0x7f8f3c15c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 shutdown_connections 2024-06-29T14:22:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.236+0000 7f8f42352640 1 -- 172.21.15.59:0/3740167686 wait complete. 2024-06-29T14:22:41.247 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:41.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.368+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2667034716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f015ceb0 msgr2=0x7fc1f00b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.368+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2667034716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f015ceb0 0x7fc1f00b56d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009830 tx=0x7fc1e002a480 comp rx=0 tx=0).stop 2024-06-29T14:22:41.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.368+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2667034716 shutdown_connections 2024-06-29T14:22:41.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.368+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2667034716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1f0141bd0 0x7fc1f013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.368+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2667034716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f015ceb0 0x7fc1f00b56d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.368+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2667034716 >> 172.21.15.59:0/2667034716 conn(0x7fc1f00b6fb0 msgr2=0x7fc1f00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.368+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2667034716 shutdown_connections 2024-06-29T14:22:41.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.368+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2667034716 wait complete. 2024-06-29T14:22:41.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.370+0000 7fc1f6a74640 1 Processor -- start 2024-06-29T14:22:41.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.370+0000 7fc1f6a74640 1 -- start start 2024-06-29T14:22:41.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.371+0000 7fc1f6a74640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1f013e9b0 0x7fc1f0138780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.371+0000 7fc1f6a74640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f015ceb0 0x7fc1f0150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.371+0000 7fc1f6a74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f01580f0 0x7fc1f0158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.371+0000 7fc1f6a74640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f00b2a00 con 0x7fc1f01580f0 2024-06-29T14:22:41.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.371+0000 7fc1f6a74640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f00b2880 con 0x7fc1f015ceb0 2024-06-29T14:22:41.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.371+0000 7fc1f6a74640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc1f00b2b80 con 0x7fc1f013e9b0 2024-06-29T14:22:41.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f01580f0 0x7fc1f0158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f01580f0 0x7fc1f0158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38460/0 (socket says 172.21.15.59:38460) 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 -- 172.21.15.59:0/2421003495 learned_addr learned my addr 172.21.15.59:0/2421003495 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1effff640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f015ceb0 0x7fc1f0150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f4810640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1f013e9b0 0x7fc1f0138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 -- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1f013e9b0 msgr2=0x7fc1f0138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1f013e9b0 0x7fc1f0138780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 -- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f015ceb0 msgr2=0x7fc1f0150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f015ceb0 0x7fc1f0150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 -- 172.21.15.59:0/2421003495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f0158a40 con 0x7fc1f01580f0 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f4810640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1f013e9b0 0x7fc1f0138780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:41.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f5011640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f01580f0 0x7fc1f0158500 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800ea70 tx=0x7fc1e800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1ed7fa640 1 -- 172.21.15.59:0/2421003495 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800ce40 con 0x7fc1f01580f0 2024-06-29T14:22:41.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1ed7fa640 1 -- 172.21.15.59:0/2421003495 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1e8004c10 con 0x7fc1f01580f0 2024-06-29T14:22:41.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1ed7fa640 1 -- 172.21.15.59:0/2421003495 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8005020 con 0x7fc1f01580f0 2024-06-29T14:22:41.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f0151690 con 0x7fc1f01580f0 2024-06-29T14:22:41.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.372+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f0151c50 con 0x7fc1f01580f0 2024-06-29T14:22:41.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.374+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1f00d16e0 con 0x7fc1f01580f0 2024-06-29T14:22:41.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.374+0000 7fc1ed7fa640 1 -- 172.21.15.59:0/2421003495 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc1e80051c0 con 0x7fc1f01580f0 2024-06-29T14:22:41.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.374+0000 7fc1ed7fa640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1dc076300 0x7fc1dc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.375+0000 7fc1ed7fa640 1 -- 172.21.15.59:0/2421003495 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fc1e80987f0 con 0x7fc1f01580f0 2024-06-29T14:22:41.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.375+0000 7fc1f4810640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1dc076300 0x7fc1dc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.375+0000 7fc1f4810640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1dc076300 0x7fc1dc078810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0034000 tx=0x7fc1e0004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.379+0000 7fc1ed7fa640 1 -- 172.21.15.59:0/2421003495 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc1f00d16e0 con 0x7fc1f01580f0 2024-06-29T14:22:41.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.567+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc1f0151420 con 0x7fc1f01580f0 2024-06-29T14:22:41.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.568+0000 7fc1ed7fa640 1 -- 172.21.15.59:0/2421003495 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fc1f0158a40 con 0x7fc1f01580f0 2024-06-29T14:22:41.567 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1dc076300 msgr2=0x7fc1dc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1dc076300 0x7fc1dc078810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0034000 tx=0x7fc1e0004730 comp rx=0 tx=0).stop 2024-06-29T14:22:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f01580f0 msgr2=0x7fc1f0158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f01580f0 0x7fc1f0158500 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800ea70 tx=0x7fc1e800ef30 comp rx=0 tx=0).stop 2024-06-29T14:22:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 shutdown_connections 2024-06-29T14:22:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc1dc076300 0x7fc1dc078810 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1f01580f0 0x7fc1f0158500 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f015ceb0 0x7fc1f0150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 --2- 172.21.15.59:0/2421003495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1f013e9b0 0x7fc1f0138780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 >> 172.21.15.59:0/2421003495 conn(0x7fc1f00b6fb0 msgr2=0x7fc1f015de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 shutdown_connections 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.570+0000 7fc1f6a74640 1 -- 172.21.15.59:0/2421003495 wait complete. 2024-06-29T14:22:41.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-06-29T14:22:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.698+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/2701108662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 msgr2=0x7fc8cc1622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.699+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/2701108662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 0x7fc8cc1622c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4005fd0 tx=0x7fc8b402a5a0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.699+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/2701108662 shutdown_connections 2024-06-29T14:22:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.699+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/2701108662 >> v2:172.21.15.153:3300/0 conn(0x7fc8cc162800 0x7fc8cc162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.699+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/2701108662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 0x7fc8cc1622c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.699+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/2701108662 >> 172.21.15.59:0/2701108662 conn(0x7fc8cc0b5610 msgr2=0x7fc8cc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.699+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/2701108662 shutdown_connections 2024-06-29T14:22:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.699+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/2701108662 wait complete. 2024-06-29T14:22:41.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.701+0000 7fc8d0d7f640 1 Processor -- start 2024-06-29T14:22:41.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.701+0000 7fc8d0d7f640 1 -- start start 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8d0d7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8cc08c190 0x7fc8cc160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8d0d7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 0x7fc8cc1f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8d0d7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8cc162800 0x7fc8cc15d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8d0d7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc0b7f60 con 0x7fc8cc0ce950 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8d0d7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc0b7de0 con 0x7fc8cc08c190 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8d0d7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc8cc0b80e0 con 0x7fc8cc162800 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8c9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 0x7fc8cc1f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8c9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 0x7fc8cc1f5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38490/0 (socket says 172.21.15.59:38490) 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8c9d9b640 1 -- 172.21.15.59:0/3221709250 learned_addr learned my addr 172.21.15.59:0/3221709250 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8ca59c640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8cc08c190 0x7fc8cc160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8cad9d640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8cc162800 0x7fc8cc15d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8c9d9b640 1 -- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8cc162800 msgr2=0x7fc8cc15d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8c9d9b640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8cc162800 0x7fc8cc15d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8c9d9b640 1 -- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8cc08c190 msgr2=0x7fc8cc160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8c9d9b640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8cc08c190 0x7fc8cc160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8c9d9b640 1 -- 172.21.15.59:0/3221709250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc1f8aa0 con 0x7fc8cc0ce950 2024-06-29T14:22:41.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.702+0000 7fc8cad9d640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8cc162800 0x7fc8cc15d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:41.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.703+0000 7fc8c9d9b640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 0x7fc8cc1f5b50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4034040 tx=0x7fc8b40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.703+0000 7fc8c2ffd640 1 -- 172.21.15.59:0/3221709250 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b403e030 con 0x7fc8cc0ce950 2024-06-29T14:22:41.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.703+0000 7fc8c2ffd640 1 -- 172.21.15.59:0/3221709250 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8b4003d90 con 0x7fc8cc0ce950 2024-06-29T14:22:41.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.703+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8cc1f8d80 con 0x7fc8cc0ce950 2024-06-29T14:22:41.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.703+0000 7fc8c2ffd640 1 -- 172.21.15.59:0/3221709250 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4035040 con 0x7fc8cc0ce950 2024-06-29T14:22:41.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.703+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8cc1f9340 con 0x7fc8cc0ce950 2024-06-29T14:22:41.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.704+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8cc161680 con 0x7fc8cc0ce950 2024-06-29T14:22:41.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.705+0000 7fc8c2ffd640 1 -- 172.21.15.59:0/3221709250 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc8b4041030 con 0x7fc8cc0ce950 2024-06-29T14:22:41.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.705+0000 7fc8c2ffd640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc8ac0762d0 0x7fc8ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.705+0000 7fc8c2ffd640 1 -- 172.21.15.59:0/3221709250 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fc8b40b81d0 con 0x7fc8cc0ce950 2024-06-29T14:22:41.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.705+0000 7fc8ca59c640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc8ac0762d0 0x7fc8ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.706+0000 7fc8ca59c640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc8ac0762d0 0x7fc8ac0787e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4001720 tx=0x7fc8c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.710+0000 7fc8c2ffd640 1 -- 172.21.15.59:0/3221709250 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc8b40810e0 con 0x7fc8cc0ce950 2024-06-29T14:22:41.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:41.982+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) -- 0x7fc8cc1622c0 con 0x7fc8cc0ce950 2024-06-29T14:22:42.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.852+0000 7fc8c2ffd640 1 -- 172.21.15.59:0/3221709250 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v71) ==== 161+0+0 (secure 0 0 0) 0x7fc8b40853e0 con 0x7fc8cc0ce950 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc8ac0762d0 msgr2=0x7fc8ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc8ac0762d0 0x7fc8ac0787e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4001720 tx=0x7fc8c4009040 comp rx=0 tx=0).stop 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 msgr2=0x7fc8cc1f5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 0x7fc8cc1f5b50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4034040 tx=0x7fc8b40038a0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 shutdown_connections 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc8ac0762d0 0x7fc8ac0787e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8cc162800 0x7fc8cc15d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8cc0ce950 0x7fc8cc1f5b50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 --2- 172.21.15.59:0/3221709250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8cc08c190 0x7fc8cc160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 >> 172.21.15.59:0/3221709250 conn(0x7fc8cc0b5610 msgr2=0x7fc8cc06d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:42.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 shutdown_connections 2024-06-29T14:22:42.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.855+0000 7fc8d0d7f640 1 -- 172.21.15.59:0/3221709250 wait complete. 2024-06-29T14:22:42.868 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:22:42.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.994+0000 7fb8a0f73640 1 -- 172.21.15.59:0/3918813044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c0b2210 msgr2=0x7fb89c06e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.994+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/3918813044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c0b2210 0x7fb89c06e300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb89c072d60 tx=0x7fb88402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.994+0000 7fb8a0f73640 1 -- 172.21.15.59:0/3918813044 shutdown_connections 2024-06-29T14:22:42.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.994+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/3918813044 >> v2:172.21.15.153:3300/0 conn(0x7fb89c06e840 0x7fb89c0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.994+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/3918813044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c0b2210 0x7fb89c06e300 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.994+0000 7fb8a0f73640 1 -- 172.21.15.59:0/3918813044 >> 172.21.15.59:0/3918813044 conn(0x7fb89c0b06e0 msgr2=0x7fb89c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:42.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.994+0000 7fb8a0f73640 1 -- 172.21.15.59:0/3918813044 shutdown_connections 2024-06-29T14:22:42.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.994+0000 7fb8a0f73640 1 -- 172.21.15.59:0/3918813044 wait complete. 2024-06-29T14:22:42.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.996+0000 7fb8a0f73640 1 Processor -- start 2024-06-29T14:22:42.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.996+0000 7fb8a0f73640 1 -- start start 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb8a0f73640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb89c06e840 0x7fb89c13d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb8a0f73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c144470 0x7fb89c13cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb8a0f73640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89c13be50 0x7fb89c1409c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb8a0f73640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c0cfae0 con 0x7fb89c144470 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb8a0f73640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c0cf960 con 0x7fb89c13be50 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb8a0f73640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb89c0cfc60 con 0x7fb89c06e840 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c144470 0x7fb89c13cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c144470 0x7fb89c13cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42736/0 (socket says 172.21.15.59:42736) 2024-06-29T14:22:42.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 -- 172.21.15.59:0/858218411 learned_addr learned my addr 172.21.15.59:0/858218411 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb89ad9d640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89c13be50 0x7fb89c1409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb89a59c640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb89c06e840 0x7fb89c13d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 -- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb89c06e840 msgr2=0x7fb89c13d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb89c06e840 0x7fb89c13d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 -- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89c13be50 msgr2=0x7fb89c1409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89c13be50 0x7fb89c1409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 -- 172.21.15.59:0/858218411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89c13f170 con 0x7fb89c144470 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb89ad9d640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89c13be50 0x7fb89c1409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb89a59c640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb89c06e840 0x7fb89c13d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb899d9b640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c144470 0x7fb89c13cb90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb884034040 tx=0x7fb884003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:42.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.998+0000 7fb892ffd640 1 -- 172.21.15.59:0/858218411 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88403e030 con 0x7fb89c144470 2024-06-29T14:22:43.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.999+0000 7fb892ffd640 1 -- 172.21.15.59:0/858218411 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb884003e00 con 0x7fb89c144470 2024-06-29T14:22:43.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.999+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89c13e0d0 con 0x7fb89c144470 2024-06-29T14:22:43.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.999+0000 7fb892ffd640 1 -- 172.21.15.59:0/858218411 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884035040 con 0x7fb89c144470 2024-06-29T14:22:43.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:42.999+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89c13e690 con 0x7fb89c144470 2024-06-29T14:22:43.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.000+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb89c0d0110 con 0x7fb89c144470 2024-06-29T14:22:43.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.001+0000 7fb892ffd640 1 -- 172.21.15.59:0/858218411 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb884041030 con 0x7fb89c144470 2024-06-29T14:22:43.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.001+0000 7fb892ffd640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb87c0762d0 0x7fb87c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.001+0000 7fb892ffd640 1 -- 172.21.15.59:0/858218411 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fb8840b8400 con 0x7fb89c144470 2024-06-29T14:22:43.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.001+0000 7fb89a59c640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb87c0762d0 0x7fb87c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.002+0000 7fb89a59c640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb87c0762d0 0x7fb87c0787e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb88c001d10 tx=0x7fb88c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:43.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.006+0000 7fb892ffd640 1 -- 172.21.15.59:0/858218411 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb8840818d0 con 0x7fb89c144470 2024-06-29T14:22:43.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.197+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fb89c15f680 con 0x7fb89c144470 2024-06-29T14:22:43.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.198+0000 7fb892ffd640 1 -- 172.21.15.59:0/858218411 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v27) ==== 59+0+1998 (secure 0 0 0) 0x7fb8840811d0 con 0x7fb89c144470 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T14:22:43.197 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T14:22:43.198 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:22:43.199 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb87c0762d0 msgr2=0x7fb87c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb87c0762d0 0x7fb87c0787e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb88c001d10 tx=0x7fb88c009040 comp rx=0 tx=0).stop 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c144470 msgr2=0x7fb89c13cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c144470 0x7fb89c13cb90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb884034040 tx=0x7fb884003910 comp rx=0 tx=0).stop 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 shutdown_connections 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb87c0762d0 0x7fb87c0787e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb89c13be50 0x7fb89c1409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb89c144470 0x7fb89c13cb90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 --2- 172.21.15.59:0/858218411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb89c06e840 0x7fb89c13d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 >> 172.21.15.59:0/858218411 conn(0x7fb89c0b06e0 msgr2=0x7fb89c0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 shutdown_connections 2024-06-29T14:22:43.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.200+0000 7fb8a0f73640 1 -- 172.21.15.59:0/858218411 wait complete. 2024-06-29T14:22:43.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:22:43.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 -- 172.21.15.59:0/2398629877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300b3ed0 msgr2=0x7f673015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 --2- 172.21.15.59:0/2398629877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300b3ed0 0x7f673015e8d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f67300d15d0 tx=0x7f672402a560 comp rx=0 tx=0).stop 2024-06-29T14:22:43.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 -- 172.21.15.59:0/2398629877 shutdown_connections 2024-06-29T14:22:43.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 --2- 172.21.15.59:0/2398629877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f673015ee10 0x7f673015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 --2- 172.21.15.59:0/2398629877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300b3ed0 0x7f673015e8d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 --2- 172.21.15.59:0/2398629877 >> v2:172.21.15.153:3300/0 conn(0x7f67300d23c0 0x7f67300b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 -- 172.21.15.59:0/2398629877 >> 172.21.15.59:0/2398629877 conn(0x7f67300d3ee0 msgr2=0x7f67300d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:43.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 -- 172.21.15.59:0/2398629877 shutdown_connections 2024-06-29T14:22:43.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.333+0000 7f67385c7640 1 -- 172.21.15.59:0/2398629877 wait complete. 2024-06-29T14:22:43.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.335+0000 7f67385c7640 1 Processor -- start 2024-06-29T14:22:43.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.335+0000 7f67385c7640 1 -- start start 2024-06-29T14:22:43.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f67385c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67300b3ed0 0x7f67301f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f67385c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300d23c0 0x7f67301f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f67385c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f673015ee10 0x7f67301f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f67385c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6730160070 con 0x7f67300d23c0 2024-06-29T14:22:43.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f67385c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f673015fef0 con 0x7f673015ee10 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f67385c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f67301601f0 con 0x7f67300b3ed0 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300d23c0 0x7f67301f90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300d23c0 0x7f67301f90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42746/0 (socket says 172.21.15.59:42746) 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 -- 172.21.15.59:0/3277075672 learned_addr learned my addr 172.21.15.59:0/3277075672 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6736363640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67300b3ed0 0x7f67301f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6736b64640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f673015ee10 0x7f67301f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 -- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67300b3ed0 msgr2=0x7f67301f8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67300b3ed0 0x7f67301f8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 -- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f673015ee10 msgr2=0x7f67301f95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f673015ee10 0x7f67301f95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 -- 172.21.15.59:0/3277075672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67300c8f20 con 0x7f67300d23c0 2024-06-29T14:22:43.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6736b64640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f673015ee10 0x7f67301f95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:43.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6736363640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67300b3ed0 0x7f67301f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:43.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f6735b62640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300d23c0 0x7f67301f90a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6724034040 tx=0x7f67240038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:43.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f672effd640 1 -- 172.21.15.59:0/3277075672 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672403e030 con 0x7f67300d23c0 2024-06-29T14:22:43.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.337+0000 7f672effd640 1 -- 172.21.15.59:0/3277075672 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6724003ca0 con 0x7f67300d23c0 2024-06-29T14:22:43.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.338+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67300c81a0 con 0x7f67300d23c0 2024-06-29T14:22:43.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.338+0000 7f672effd640 1 -- 172.21.15.59:0/3277075672 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6724035040 con 0x7f67300d23c0 2024-06-29T14:22:43.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.338+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67300c8720 con 0x7f67300d23c0 2024-06-29T14:22:43.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.338+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67300ca380 con 0x7f67300d23c0 2024-06-29T14:22:43.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.339+0000 7f672effd640 1 -- 172.21.15.59:0/3277075672 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6724041030 con 0x7f67300d23c0 2024-06-29T14:22:43.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.340+0000 7f672effd640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f671c076300 0x7f671c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.340+0000 7f672effd640 1 -- 172.21.15.59:0/3277075672 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f67240b81d0 con 0x7f67300d23c0 2024-06-29T14:22:43.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.340+0000 7f6736363640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f671c076300 0x7f671c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.341+0000 7f6736363640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f671c076300 0x7f671c078810 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f67300d2e40 tx=0x7f6720009500 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:43.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.344+0000 7f672effd640 1 -- 172.21.15.59:0/3277075672 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6724081730 con 0x7f67300d23c0 2024-06-29T14:22:43.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.536+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f67300b3990 con 0x7f67300d23c0 2024-06-29T14:22:43.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.540+0000 7f672effd640 1 -- 172.21.15.59:0/3277075672 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f6724081030 con 0x7f67300d23c0 2024-06-29T14:22:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f671c076300 msgr2=0x7f671c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f671c076300 0x7f671c078810 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f67300d2e40 tx=0x7f6720009500 comp rx=0 tx=0).stop 2024-06-29T14:22:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300d23c0 msgr2=0x7f67301f90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300d23c0 0x7f67301f90a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6724034040 tx=0x7f67240038a0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 shutdown_connections 2024-06-29T14:22:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f671c076300 0x7f671c078810 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f673015ee10 0x7f67301f95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67300d23c0 0x7f67301f90a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 --2- 172.21.15.59:0/3277075672 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67300b3ed0 0x7f67301f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 >> 172.21.15.59:0/3277075672 conn(0x7f67300d3ee0 msgr2=0x7f67300b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:43.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 shutdown_connections 2024-06-29T14:22:43.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.542+0000 7f67385c7640 1 -- 172.21.15.59:0/3277075672 wait complete. 2024-06-29T14:22:43.552 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:22:43.552 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:22:43.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:22:43.641 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:22:43.641 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:43.641 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:43.658 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:43.658 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:43.715 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:43.716 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:43.773 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:43.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.934+0000 7f73f3438640 1 -- 172.21.15.59:0/3174103924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 msgr2=0x7f73d4003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.934+0000 7f73f3438640 1 --2- 172.21.15.59:0/3174103924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 0x7f73d4003ee0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009b30 tx=0x7f73d4005830 comp rx=0 tx=0).stop 2024-06-29T14:22:43.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.934+0000 7f73f19d5640 1 -- 172.21.15.59:0/3174103924 reap_dead start 2024-06-29T14:22:43.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.934+0000 7f73f3438640 1 -- 172.21.15.59:0/3174103924 shutdown_connections 2024-06-29T14:22:43.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.934+0000 7f73f3438640 1 -- 172.21.15.59:0/3174103924 >> 172.21.15.59:0/3174103924 conn(0x7f73ec0b24c0 msgr2=0x7f73ec0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:43.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.934+0000 7f73f3438640 1 -- 172.21.15.59:0/3174103924 shutdown_connections 2024-06-29T14:22:43.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.934+0000 7f73f3438640 1 -- 172.21.15.59:0/3174103924 wait complete. 2024-06-29T14:22:43.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.936+0000 7f73f3438640 1 Processor -- start 2024-06-29T14:22:43.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.936+0000 7f73f3438640 1 -- start start 2024-06-29T14:22:43.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f3438640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 0x7f73ec15ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f3438640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec15b370 0x7f73ec15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f3438640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73ec15d1a0 0x7f73ec13a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f3438640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec0ce540 con 0x7f73d4003a70 2024-06-29T14:22:43.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f3438640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec0741a0 con 0x7f73ec15b370 2024-06-29T14:22:43.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f3438640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f73ec0ce6c0 con 0x7f73ec15d1a0 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 0x7f73ec15ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 0x7f73ec15ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42760/0 (socket says 172.21.15.59:42760) 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 -- 172.21.15.59:0/3952356529 learned_addr learned my addr 172.21.15.59:0/3952356529 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f19d5640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73ec15d1a0 0x7f73ec13a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f09d3640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec15b370 0x7f73ec15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 -- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73ec15d1a0 msgr2=0x7f73ec13a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73ec15d1a0 0x7f73ec13a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 -- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec15b370 msgr2=0x7f73ec15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec15b370 0x7f73ec15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 -- 172.21.15.59:0/3952356529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ec15d5b0 con 0x7f73d4003a70 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f09d3640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec15b370 0x7f73ec15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.938+0000 7f73f11d4640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 0x7f73ec15ace0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009a50 tx=0x7f73e402ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:43.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.939+0000 7f73e9ffb640 1 -- 172.21.15.59:0/3952356529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4041020 con 0x7f73d4003a70 2024-06-29T14:22:43.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.939+0000 7f73e9ffb640 1 -- 172.21.15.59:0/3952356529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73e4004ca0 con 0x7f73d4003a70 2024-06-29T14:22:43.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.939+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ec141430 con 0x7f73d4003a70 2024-06-29T14:22:43.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.939+0000 7f73e9ffb640 1 -- 172.21.15.59:0/3952356529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e403d6c0 con 0x7f73d4003a70 2024-06-29T14:22:43.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.939+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ec1420d0 con 0x7f73d4003a70 2024-06-29T14:22:43.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.940+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ec06c1e0 con 0x7f73d4003a70 2024-06-29T14:22:43.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.941+0000 7f73e9ffb640 1 -- 172.21.15.59:0/3952356529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f73e4044020 con 0x7f73d4003a70 2024-06-29T14:22:43.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.942+0000 7f73e9ffb640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f73d80762d0 0x7f73d80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.942+0000 7f73e9ffb640 1 -- 172.21.15.59:0/3952356529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f73e40b9110 con 0x7f73d4003a70 2024-06-29T14:22:43.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.942+0000 7f73f09d3640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f73d80762d0 0x7f73d80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.943+0000 7f73f09d3640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f73d80762d0 0x7f73d80787e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f73e0005eb0 tx=0x7f73e0005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:43.946+0000 7f73e9ffb640 1 -- 172.21.15.59:0/3952356529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73e404b030 con 0x7f73d4003a70 2024-06-29T14:22:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.139+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73ec06da90 con 0x7f73d4003a70 2024-06-29T14:22:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.140+0000 7f73e9ffb640 1 -- 172.21.15.59:0/3952356529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7f73ec15d5b0 con 0x7f73d4003a70 2024-06-29T14:22:44.139 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:44.139 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:44.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f73d80762d0 msgr2=0x7f73d80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f73d80762d0 0x7f73d80787e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f73e0005eb0 tx=0x7f73e0005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:44.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 msgr2=0x7f73ec15ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 0x7f73ec15ace0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009a50 tx=0x7f73e402ba60 comp rx=0 tx=0).stop 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 shutdown_connections 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f73d80762d0 0x7f73d80787e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73ec15d1a0 0x7f73ec13a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec15b370 0x7f73ec15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 --2- 172.21.15.59:0/3952356529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73d4003a70 0x7f73ec15ace0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 >> 172.21.15.59:0/3952356529 conn(0x7f73ec0b24c0 msgr2=0x7f73ec0af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 shutdown_connections 2024-06-29T14:22:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.142+0000 7f73f3438640 1 -- 172.21.15.59:0/3952356529 wait complete. 2024-06-29T14:22:44.153 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:44.153 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.276+0000 7f8aa058c640 1 -- 172.21.15.59:0/1156400264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980b4ee0 msgr2=0x7f8a9806e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.276+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1156400264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980b4ee0 0x7f8a9806e4d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009830 tx=0x7f8a8c02a510 comp rx=0 tx=0).stop 2024-06-29T14:22:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.276+0000 7f8aa058c640 1 -- 172.21.15.59:0/1156400264 shutdown_connections 2024-06-29T14:22:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.276+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1156400264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a980b4ee0 0x7f8a9806e4d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.276+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1156400264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a980d0130 0x7f8a98143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.276+0000 7f8aa058c640 1 -- 172.21.15.59:0/1156400264 >> 172.21.15.59:0/1156400264 conn(0x7f8a980b2ab0 msgr2=0x7f8a980b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:44.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.277+0000 7f8aa058c640 1 -- 172.21.15.59:0/1156400264 shutdown_connections 2024-06-29T14:22:44.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.277+0000 7f8aa058c640 1 -- 172.21.15.59:0/1156400264 wait complete. 2024-06-29T14:22:44.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.278+0000 7f8aa058c640 1 Processor -- start 2024-06-29T14:22:44.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.278+0000 7f8aa058c640 1 -- start start 2024-06-29T14:22:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8aa058c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9806ea10 0x7f8a9813d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8aa058c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a980b4ee0 0x7f8a9815d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8aa058c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a980d0130 0x7f8a980ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8aa058c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9815ed80 con 0x7f8a980b4ee0 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8aa058c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9806f030 con 0x7f8a9806ea10 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8aa058c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8a9815ef00 con 0x7f8a980d0130 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a980b4ee0 0x7f8a9815d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a980b4ee0 0x7f8a9815d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42776/0 (socket says 172.21.15.59:42776) 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 -- 172.21.15.59:0/1095406206 learned_addr learned my addr 172.21.15.59:0/1095406206 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9e328640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9806ea10 0x7f8a9813d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9eb29640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a980d0130 0x7f8a980ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 -- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a980d0130 msgr2=0x7f8a980ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a980d0130 0x7f8a980ac610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 -- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9806ea10 msgr2=0x7f8a9813d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9806ea10 0x7f8a9813d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 -- 172.21.15.59:0/1095406206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9815db00 con 0x7f8a980b4ee0 2024-06-29T14:22:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9e328640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9806ea10 0x7f8a9813d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.280+0000 7f8a9db27640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a980b4ee0 0x7f8a9815d5c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009950 tx=0x7f8a8c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.281+0000 7f8a96ffd640 1 -- 172.21.15.59:0/1095406206 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c005c90 con 0x7f8a980b4ee0 2024-06-29T14:22:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.281+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a980ab850 con 0x7f8a980b4ee0 2024-06-29T14:22:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.281+0000 7f8a96ffd640 1 -- 172.21.15.59:0/1095406206 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a8c031e50 con 0x7f8a980b4ee0 2024-06-29T14:22:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.281+0000 7f8a96ffd640 1 -- 172.21.15.59:0/1095406206 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c005260 con 0x7f8a980b4ee0 2024-06-29T14:22:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.281+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a980abe10 con 0x7f8a980b4ee0 2024-06-29T14:22:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.282+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a9815f3a0 con 0x7f8a980b4ee0 2024-06-29T14:22:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.283+0000 7f8a96ffd640 1 -- 172.21.15.59:0/1095406206 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8a8c005400 con 0x7f8a980b4ee0 2024-06-29T14:22:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.283+0000 7f8a96ffd640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8a84076300 0x7f8a84078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.283+0000 7f8a96ffd640 1 -- 172.21.15.59:0/1095406206 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f8a8c0b8be0 con 0x7f8a980b4ee0 2024-06-29T14:22:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.284+0000 7f8a9e328640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8a84076300 0x7f8a84078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.284+0000 7f8a9e328640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8a84076300 0x7f8a84078810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009b80 tx=0x7f8a90005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.288+0000 7f8a96ffd640 1 -- 172.21.15.59:0/1095406206 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8a8c03f030 con 0x7f8a980b4ee0 2024-06-29T14:22:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.477+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8a9815dcd0 con 0x7f8a980b4ee0 2024-06-29T14:22:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.477+0000 7f8a96ffd640 1 -- 172.21.15.59:0/1095406206 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f8a8c082030 con 0x7f8a980b4ee0 2024-06-29T14:22:44.476 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:44.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8a84076300 msgr2=0x7f8a84078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8a84076300 0x7f8a84078810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009b80 tx=0x7f8a90005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a980b4ee0 msgr2=0x7f8a9815d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a980b4ee0 0x7f8a9815d5c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009950 tx=0x7f8a8c033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 shutdown_connections 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8a84076300 0x7f8a84078810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a980d0130 0x7f8a980ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a980b4ee0 0x7f8a9815d5c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 --2- 172.21.15.59:0/1095406206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9806ea10 0x7f8a9813d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 >> 172.21.15.59:0/1095406206 conn(0x7f8a980b2ab0 msgr2=0x7f8a98072c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 shutdown_connections 2024-06-29T14:22:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.479+0000 7f8aa058c640 1 -- 172.21.15.59:0/1095406206 wait complete. 2024-06-29T14:22:44.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.612+0000 7f4351d32640 1 -- 172.21.15.59:0/2336298497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c0b3ed0 msgr2=0x7f434c0d1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.612+0000 7f4351d32640 1 --2- 172.21.15.59:0/2336298497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c0b3ed0 0x7f434c0d1690 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f434c0b3960 tx=0x7f434402aa30 comp rx=0 tx=0).stop 2024-06-29T14:22:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.613+0000 7f4351d32640 1 -- 172.21.15.59:0/2336298497 shutdown_connections 2024-06-29T14:22:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.613+0000 7f4351d32640 1 --2- 172.21.15.59:0/2336298497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c0b3ed0 0x7f434c0d1690 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.613+0000 7f4351d32640 1 --2- 172.21.15.59:0/2336298497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f434c0ce650 0x7f434c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.613+0000 7f4351d32640 1 --2- 172.21.15.59:0/2336298497 >> v2:172.21.15.153:3300/0 conn(0x7f434c15e290 0x7f434c0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.613+0000 7f4351d32640 1 -- 172.21.15.59:0/2336298497 >> 172.21.15.59:0/2336298497 conn(0x7f434c0d3ee0 msgr2=0x7f434c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.613+0000 7f4351d32640 1 -- 172.21.15.59:0/2336298497 shutdown_connections 2024-06-29T14:22:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.613+0000 7f4351d32640 1 -- 172.21.15.59:0/2336298497 wait complete. 2024-06-29T14:22:44.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.614+0000 7f4351d32640 1 Processor -- start 2024-06-29T14:22:44.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.615+0000 7f4351d32640 1 -- start start 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f4351d32640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f434c0b3ed0 0x7f434c14adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f4351d32640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f434c0ce650 0x7f434c158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f4351d32640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c15e290 0x7f434c1f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f4351d32640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f434c0d1fa0 con 0x7f434c15e290 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f4351d32640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f434c0d1e20 con 0x7f434c0b3ed0 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f4351d32640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f434c0d2120 con 0x7f434c0ce650 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c15e290 0x7f434c1f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c15e290 0x7f434c1f4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42798/0 (socket says 172.21.15.59:42798) 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434bfff640 1 -- 172.21.15.59:0/649916395 learned_addr learned my addr 172.21.15.59:0/649916395 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:44.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434b7fe640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f434c0b3ed0 0x7f434c14adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434affd640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f434c0ce650 0x7f434c158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434bfff640 1 -- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f434c0ce650 msgr2=0x7f434c158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434bfff640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f434c0ce650 0x7f434c158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434bfff640 1 -- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f434c0b3ed0 msgr2=0x7f434c14adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434bfff640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f434c0b3ed0 0x7f434c14adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.616+0000 7f434bfff640 1 -- 172.21.15.59:0/649916395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c157160 con 0x7f434c15e290 2024-06-29T14:22:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.617+0000 7f434bfff640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c15e290 0x7f434c1f4df0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4344009d60 tx=0x7f4344003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.617+0000 7f4333fff640 1 -- 172.21.15.59:0/649916395 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344033050 con 0x7f434c15e290 2024-06-29T14:22:44.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.617+0000 7f4333fff640 1 -- 172.21.15.59:0/649916395 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4344004450 con 0x7f434c15e290 2024-06-29T14:22:44.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.617+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434c157440 con 0x7f434c15e290 2024-06-29T14:22:44.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.617+0000 7f4333fff640 1 -- 172.21.15.59:0/649916395 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344038450 con 0x7f434c15e290 2024-06-29T14:22:44.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.617+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f434c157a00 con 0x7f434c15e290 2024-06-29T14:22:44.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.618+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f434c0b3990 con 0x7f434c15e290 2024-06-29T14:22:44.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.619+0000 7f4333fff640 1 -- 172.21.15.59:0/649916395 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4344036070 con 0x7f434c15e290 2024-06-29T14:22:44.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.619+0000 7f4333fff640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43380762d0 0x7f43380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.619+0000 7f4333fff640 1 -- 172.21.15.59:0/649916395 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f43440b8f80 con 0x7f434c15e290 2024-06-29T14:22:44.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.619+0000 7f434b7fe640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43380762d0 0x7f43380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.620+0000 7f434b7fe640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43380762d0 0x7f43380787e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f433c005fd0 tx=0x7f433c00b2e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:44.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.624+0000 7f4333fff640 1 -- 172.21.15.59:0/649916395 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4344081e00 con 0x7f434c15e290 2024-06-29T14:22:44.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.822+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f434c0b8140 con 0x7f434c15e290 2024-06-29T14:22:44.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.823+0000 7f4333fff640 1 -- 172.21.15.59:0/649916395 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f4344086100 con 0x7f434c15e290 2024-06-29T14:22:44.822 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43380762d0 msgr2=0x7f43380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43380762d0 0x7f43380787e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f433c005fd0 tx=0x7f433c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c15e290 msgr2=0x7f434c1f4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c15e290 0x7f434c1f4df0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4344009d60 tx=0x7f4344003f10 comp rx=0 tx=0).stop 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 shutdown_connections 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f43380762d0 0x7f43380787e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f434c15e290 0x7f434c1f4df0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f434c0ce650 0x7f434c158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 --2- 172.21.15.59:0/649916395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f434c0b3ed0 0x7f434c14adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 >> 172.21.15.59:0/649916395 conn(0x7f434c0d3ee0 msgr2=0x7f434c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:44.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 shutdown_connections 2024-06-29T14:22:44.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.825+0000 7f4351d32640 1 -- 172.21.15.59:0/649916395 wait complete. 2024-06-29T14:22:44.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:44.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 -- 172.21.15.59:0/1339680459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 msgr2=0x7fc3680ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/1339680459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 0x7fc3680ce550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc3680b35d0 tx=0x7fc36402a970 comp rx=0 tx=0).stop 2024-06-29T14:22:44.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 -- 172.21.15.59:0/1339680459 shutdown_connections 2024-06-29T14:22:44.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/1339680459 >> v2:172.21.15.153:3300/0 conn(0x7fc368161fc0 0x7fc3680d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/1339680459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc36815e0a0 0x7fc368161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/1339680459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 0x7fc3680ce550 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 -- 172.21.15.59:0/1339680459 >> 172.21.15.59:0/1339680459 conn(0x7fc3680d45b0 msgr2=0x7fc3680d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 -- 172.21.15.59:0/1339680459 shutdown_connections 2024-06-29T14:22:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.957+0000 7fc36f7c1640 1 -- 172.21.15.59:0/1339680459 wait complete. 2024-06-29T14:22:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.958+0000 7fc36f7c1640 1 Processor -- start 2024-06-29T14:22:44.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.959+0000 7fc36f7c1640 1 -- start start 2024-06-29T14:22:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36f7c1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc36815e0a0 0x7fc3681f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36f7c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc368161fc0 0x7fc368155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36f7c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 0x7fc3680c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36f7c1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc3680d2f40 con 0x7fc368162770 2024-06-29T14:22:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36f7c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc3680d2dc0 con 0x7fc368161fc0 2024-06-29T14:22:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36f7c1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc3680d30c0 con 0x7fc36815e0a0 2024-06-29T14:22:44.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 0x7fc3680c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 0x7fc3680c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42810/0 (socket says 172.21.15.59:42810) 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 -- 172.21.15.59:0/2898833114 learned_addr learned my addr 172.21.15.59:0/2898833114 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36d55d640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc36815e0a0 0x7fc3681f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36cd5c640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc368161fc0 0x7fc368155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 -- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc36815e0a0 msgr2=0x7fc3681f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc36815e0a0 0x7fc3681f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 -- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc368161fc0 msgr2=0x7fc368155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc368161fc0 0x7fc368155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 -- 172.21.15.59:0/2898833114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc368155550 con 0x7fc368162770 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.960+0000 7fc36dd5e640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 0x7fc3680c9d50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc36000d620 tx=0x7fc36000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.961+0000 7fc35dffb640 1 -- 172.21.15.59:0/2898833114 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360017070 con 0x7fc368162770 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.961+0000 7fc35dffb640 1 -- 172.21.15.59:0/2898833114 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc360005310 con 0x7fc368162770 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.961+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc368153f90 con 0x7fc368162770 2024-06-29T14:22:44.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.961+0000 7fc35dffb640 1 -- 172.21.15.59:0/2898833114 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3600047a0 con 0x7fc368162770 2024-06-29T14:22:44.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.961+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc368154550 con 0x7fc368162770 2024-06-29T14:22:44.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.962+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc36806cf30 con 0x7fc368162770 2024-06-29T14:22:44.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.962+0000 7fc35dffb640 1 -- 172.21.15.59:0/2898833114 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc360004940 con 0x7fc368162770 2024-06-29T14:22:44.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.963+0000 7fc35dffb640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc34c076300 0x7fc34c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.963+0000 7fc35dffb640 1 -- 172.21.15.59:0/2898833114 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fc360097bc0 con 0x7fc368162770 2024-06-29T14:22:44.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.963+0000 7fc36d55d640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc34c076300 0x7fc34c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.964+0000 7fc36d55d640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc34c076300 0x7fc34c078810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc364035000 tx=0x7fc364004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:44.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:44.967+0000 7fc35dffb640 1 -- 172.21.15.59:0/2898833114 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc360061080 con 0x7fc368162770 2024-06-29T14:22:45.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.162+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc368075b40 con 0x7fc368162770 2024-06-29T14:22:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.163+0000 7fc35dffb640 1 -- 172.21.15.59:0/2898833114 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fc360069070 con 0x7fc368162770 2024-06-29T14:22:45.162 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc34c076300 msgr2=0x7fc34c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc34c076300 0x7fc34c078810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc364035000 tx=0x7fc364004ab0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 msgr2=0x7fc3680c9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 0x7fc3680c9d50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc36000d620 tx=0x7fc36000dae0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 shutdown_connections 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc34c076300 0x7fc34c078810 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc368162770 0x7fc3680c9d50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc368161fc0 0x7fc368155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 --2- 172.21.15.59:0/2898833114 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc36815e0a0 0x7fc3681f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 >> 172.21.15.59:0/2898833114 conn(0x7fc3680d45b0 msgr2=0x7fc3680b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 shutdown_connections 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.165+0000 7fc36f7c1640 1 -- 172.21.15.59:0/2898833114 wait complete. 2024-06-29T14:22:45.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.297+0000 7f0556626640 1 -- 172.21.15.59:0/965564633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0538004120 msgr2=0x7f0538004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.297+0000 7f0556626640 1 --2- 172.21.15.59:0/965564633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0538004120 0x7f0538004800 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f054400bb20 tx=0x7f0538005830 comp rx=0 tx=0).stop 2024-06-29T14:22:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.297+0000 7f0554bc3640 1 -- 172.21.15.59:0/965564633 reap_dead start 2024-06-29T14:22:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.297+0000 7f0556626640 1 -- 172.21.15.59:0/965564633 shutdown_connections 2024-06-29T14:22:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.297+0000 7f0556626640 1 -- 172.21.15.59:0/965564633 >> 172.21.15.59:0/965564633 conn(0x7f05500b5610 msgr2=0x7f05500b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.297+0000 7f0556626640 1 -- 172.21.15.59:0/965564633 shutdown_connections 2024-06-29T14:22:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.297+0000 7f0556626640 1 -- 172.21.15.59:0/965564633 wait complete. 2024-06-29T14:22:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.299+0000 7f0556626640 1 Processor -- start 2024-06-29T14:22:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.299+0000 7f0556626640 1 -- start start 2024-06-29T14:22:45.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f0556626640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0538004120 0x7f05501572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f0556626640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05501577e0 0x7f05501f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f0556626640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05501f5980 0x7f055015b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f0556626640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0550162c10 con 0x7f05501577e0 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f0556626640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05501625c0 con 0x7f05501f5980 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f0556626640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0550162d90 con 0x7f0538004120 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05501577e0 0x7f05501f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05501577e0 0x7f05501f5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42840/0 (socket says 172.21.15.59:42840) 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 -- 172.21.15.59:0/2901847518 learned_addr learned my addr 172.21.15.59:0/2901847518 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f0554bc3640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05501f5980 0x7f055015b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054ffff640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0538004120 0x7f05501572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 -- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0538004120 msgr2=0x7f05501572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0538004120 0x7f05501572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 -- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05501f5980 msgr2=0x7f055015b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05501f5980 0x7f055015b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 -- 172.21.15.59:0/2901847518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0550157bf0 con 0x7f05501577e0 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f0554bc3640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05501f5980 0x7f055015b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054f7fe640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05501577e0 0x7f05501f5440 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f054400bc40 tx=0x7f0544031f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.301+0000 7f054cff9640 1 -- 172.21.15.59:0/2901847518 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544040070 con 0x7f05501577e0 2024-06-29T14:22:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.302+0000 7f054cff9640 1 -- 172.21.15.59:0/2901847518 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0544004120 con 0x7f05501577e0 2024-06-29T14:22:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.302+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0550159670 con 0x7f05501577e0 2024-06-29T14:22:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.302+0000 7f054cff9640 1 -- 172.21.15.59:0/2901847518 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544035040 con 0x7f05501577e0 2024-06-29T14:22:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.302+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0550159c30 con 0x7f05501577e0 2024-06-29T14:22:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.302+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05500cf080 con 0x7f05501577e0 2024-06-29T14:22:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.303+0000 7f054cff9640 1 -- 172.21.15.59:0/2901847518 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0544038020 con 0x7f05501577e0 2024-06-29T14:22:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.304+0000 7f054cff9640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f053c0762d0 0x7f053c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.304+0000 7f054cff9640 1 -- 172.21.15.59:0/2901847518 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f05440b85a0 con 0x7f05501577e0 2024-06-29T14:22:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.304+0000 7f054ffff640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f053c0762d0 0x7f053c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.305+0000 7f054ffff640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f053c0762d0 0x7f053c0787e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f05500b7620 tx=0x7f05400067a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:45.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.308+0000 7f054cff9640 1 -- 172.21.15.59:0/2901847518 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f05440819f0 con 0x7f05501577e0 2024-06-29T14:22:45.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.496+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f055006c8d0 con 0x7f05501577e0 2024-06-29T14:22:45.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.497+0000 7f054cff9640 1 -- 172.21.15.59:0/2901847518 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f05440812f0 con 0x7f05501577e0 2024-06-29T14:22:45.496 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:45.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f053c0762d0 msgr2=0x7f053c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f053c0762d0 0x7f053c0787e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f05500b7620 tx=0x7f05400067a0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05501577e0 msgr2=0x7f05501f5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05501577e0 0x7f05501f5440 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f054400bc40 tx=0x7f0544031f80 comp rx=0 tx=0).stop 2024-06-29T14:22:45.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 shutdown_connections 2024-06-29T14:22:45.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f053c0762d0 0x7f053c0787e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05501f5980 0x7f055015b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05501577e0 0x7f05501f5440 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 --2- 172.21.15.59:0/2901847518 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0538004120 0x7f05501572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 >> 172.21.15.59:0/2901847518 conn(0x7f05500b5610 msgr2=0x7f055015f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 shutdown_connections 2024-06-29T14:22:45.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.499+0000 7f0556626640 1 -- 172.21.15.59:0/2901847518 wait complete. 2024-06-29T14:22:45.509 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:22:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.628+0000 7fb3d2b55640 1 -- 172.21.15.59:0/4285580633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162190 msgr2=0x7fb3cc0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.628+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/4285580633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162190 0x7fb3cc0b38e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4005fd0 tx=0x7fb3c402aa10 comp rx=0 tx=0).stop 2024-06-29T14:22:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.629+0000 7fb3d2b55640 1 -- 172.21.15.59:0/4285580633 shutdown_connections 2024-06-29T14:22:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.629+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/4285580633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162190 0x7fb3cc0b38e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.629+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/4285580633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3cc162770 0x7fb3cc0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.629+0000 7fb3d2b55640 1 -- 172.21.15.59:0/4285580633 >> 172.21.15.59:0/4285580633 conn(0x7fb3cc0d45b0 msgr2=0x7fb3cc0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.629+0000 7fb3d2b55640 1 -- 172.21.15.59:0/4285580633 shutdown_connections 2024-06-29T14:22:45.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.629+0000 7fb3d2b55640 1 -- 172.21.15.59:0/4285580633 wait complete. 2024-06-29T14:22:45.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.630+0000 7fb3d2b55640 1 Processor -- start 2024-06-29T14:22:45.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.630+0000 7fb3d2b55640 1 -- start start 2024-06-29T14:22:45.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3d2b55640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3cc15e0a0 0x7fb3cc15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3d2b55640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162770 0x7fb3cc0b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3d2b55640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3cc0ca390 0x7fb3cc0b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3d2b55640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc0b43a0 con 0x7fb3cc162770 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3d2b55640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc0b4220 con 0x7fb3cc15e0a0 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3d2b55640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb3cc0b4520 con 0x7fb3cc0ca390 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3cbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162770 0x7fb3cc0b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3cbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162770 0x7fb3cc0b83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42844/0 (socket says 172.21.15.59:42844) 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3cbfff640 1 -- 172.21.15.59:0/3653991090 learned_addr learned my addr 172.21.15.59:0/3653991090 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3d08f1640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3cc15e0a0 0x7fb3cc15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3d10f2640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3cc0ca390 0x7fb3cc0b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3cbfff640 1 -- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3cc0ca390 msgr2=0x7fb3cc0b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3cbfff640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3cc0ca390 0x7fb3cc0b8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3cbfff640 1 -- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3cc15e0a0 msgr2=0x7fb3cc15b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3cbfff640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3cc15e0a0 0x7fb3cc15b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.632+0000 7fb3cbfff640 1 -- 172.21.15.59:0/3653991090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3cc15e540 con 0x7fb3cc162770 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.633+0000 7fb3cbfff640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162770 0x7fb3cc0b83f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb3b800c3c0 tx=0x7fb3b800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:45.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.633+0000 7fb3c97fa640 1 -- 172.21.15.59:0/3653991090 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b80052d0 con 0x7fb3cc162770 2024-06-29T14:22:45.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.633+0000 7fb3c97fa640 1 -- 172.21.15.59:0/3653991090 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3b8007020 con 0x7fb3cc162770 2024-06-29T14:22:45.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.633+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3cc15e820 con 0x7fb3cc162770 2024-06-29T14:22:45.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.633+0000 7fb3c97fa640 1 -- 172.21.15.59:0/3653991090 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8007470 con 0x7fb3cc162770 2024-06-29T14:22:45.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.633+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3cc15ede0 con 0x7fb3cc162770 2024-06-29T14:22:45.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.634+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3cc0ca850 con 0x7fb3cc162770 2024-06-29T14:22:45.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.635+0000 7fb3c97fa640 1 -- 172.21.15.59:0/3653991090 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb3b80067d0 con 0x7fb3cc162770 2024-06-29T14:22:45.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.635+0000 7fb3c97fa640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb3b0076300 0x7fb3b0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.635+0000 7fb3c97fa640 1 -- 172.21.15.59:0/3653991090 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fb3b8097030 con 0x7fb3cc162770 2024-06-29T14:22:45.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.635+0000 7fb3d08f1640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb3b0076300 0x7fb3b0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.636+0000 7fb3d08f1640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb3b0076300 0x7fb3b0078810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009b30 tx=0x7fb3c0005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:45.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.640+0000 7fb3c97fa640 1 -- 172.21.15.59:0/3653991090 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb3b805feb0 con 0x7fb3cc162770 2024-06-29T14:22:45.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.843+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb3cc0b49d0 con 0x7fb3cc162770 2024-06-29T14:22:45.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.844+0000 7fb3c97fa640 1 -- 172.21.15.59:0/3653991090 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7fb3b80641b0 con 0x7fb3cc162770 2024-06-29T14:22:45.843 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-06-29T14:22:45.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb3b0076300 msgr2=0x7fb3b0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb3b0076300 0x7fb3b0078810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009b30 tx=0x7fb3c0005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:45.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162770 msgr2=0x7fb3cc0b83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162770 0x7fb3cc0b83f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb3b800c3c0 tx=0x7fb3b800c880 comp rx=0 tx=0).stop 2024-06-29T14:22:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 shutdown_connections 2024-06-29T14:22:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb3b0076300 0x7fb3b0078810 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3cc0ca390 0x7fb3cc0b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3cc162770 0x7fb3cc0b83f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 --2- 172.21.15.59:0/3653991090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3cc15e0a0 0x7fb3cc15b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 >> 172.21.15.59:0/3653991090 conn(0x7fb3cc0d45b0 msgr2=0x7fb3cc15bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 shutdown_connections 2024-06-29T14:22:45.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:45.846+0000 7fb3d2b55640 1 -- 172.21.15.59:0/3653991090 wait complete. 2024-06-29T14:22:45.856 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:22:45.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:45.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-06-29T14:22:45.884 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:22:45.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:45.884 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:22:45.940 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:22:45.940 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:45.959 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:22:45.959 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:46.017 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:22:46.017 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:46.076 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.197+0000 7f27683d2640 1 -- 172.21.15.59:0/1134465180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2748003a70 msgr2=0x7f2748003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.197+0000 7f27683d2640 1 --2- 172.21.15.59:0/1134465180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2748003a70 0x7f2748003ee0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f275800c310 tx=0x7f2748005830 comp rx=0 tx=0).stop 2024-06-29T14:22:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.197+0000 7f276696f640 1 -- 172.21.15.59:0/1134465180 reap_dead start 2024-06-29T14:22:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.197+0000 7f27683d2640 1 -- 172.21.15.59:0/1134465180 shutdown_connections 2024-06-29T14:22:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.197+0000 7f27683d2640 1 -- 172.21.15.59:0/1134465180 >> 172.21.15.59:0/1134465180 conn(0x7f2760073e10 msgr2=0x7f2760074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.197+0000 7f27683d2640 1 -- 172.21.15.59:0/1134465180 shutdown_connections 2024-06-29T14:22:46.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.197+0000 7f27683d2640 1 -- 172.21.15.59:0/1134465180 wait complete. 2024-06-29T14:22:46.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.199+0000 7f27683d2640 1 Processor -- start 2024-06-29T14:22:46.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.199+0000 7f27683d2640 1 -- start start 2024-06-29T14:22:46.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f27683d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748003a70 0x7f27600aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f27683d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27600a94e0 0x7f2760154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f27683d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27600ab9c0 0x7f27600abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f27683d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f27600d18b0 con 0x7f27600a94e0 2024-06-29T14:22:46.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f27683d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27600d1730 con 0x7f27600ab9c0 2024-06-29T14:22:46.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f27683d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f276015fa40 con 0x7f2748003a70 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f276596d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27600a94e0 0x7f2760154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f276596d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27600a94e0 0x7f2760154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42848/0 (socket says 172.21.15.59:42848) 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f276596d640 1 -- 172.21.15.59:0/1486694054 learned_addr learned my addr 172.21.15.59:0/1486694054 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f276616e640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748003a70 0x7f27600aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.200+0000 7f276696f640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27600ab9c0 0x7f27600abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f276596d640 1 -- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748003a70 msgr2=0x7f27600aa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f276596d640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748003a70 0x7f27600aa500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f276596d640 1 -- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27600ab9c0 msgr2=0x7f27600abdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f276596d640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27600ab9c0 0x7f27600abdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f276596d640 1 -- 172.21.15.59:0/1486694054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27600ac310 con 0x7f27600a94e0 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f276616e640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748003a70 0x7f27600aa500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:46.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f276696f640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27600ab9c0 0x7f27600abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:46.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f276596d640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27600a94e0 0x7f2760154180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2754011ab0 tx=0x7f2754011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:46.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f275effd640 1 -- 172.21.15.59:0/1486694054 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275401e070 con 0x7f27600a94e0 2024-06-29T14:22:46.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f275effd640 1 -- 172.21.15.59:0/1486694054 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2754004460 con 0x7f27600a94e0 2024-06-29T14:22:46.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27601f3ea0 con 0x7f27600a94e0 2024-06-29T14:22:46.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f275effd640 1 -- 172.21.15.59:0/1486694054 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275400fba0 con 0x7f27600a94e0 2024-06-29T14:22:46.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.201+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f276015cd40 con 0x7f27600a94e0 2024-06-29T14:22:46.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.202+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2760159060 con 0x7f27600a94e0 2024-06-29T14:22:46.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.203+0000 7f275effd640 1 -- 172.21.15.59:0/1486694054 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2754021070 con 0x7f27600a94e0 2024-06-29T14:22:46.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.203+0000 7f275effd640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f274c0762d0 0x7f274c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.203+0000 7f275effd640 1 -- 172.21.15.59:0/1486694054 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f27540984c0 con 0x7f27600a94e0 2024-06-29T14:22:46.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.204+0000 7f276616e640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f274c0762d0 0x7f274c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.204+0000 7f276616e640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f274c0762d0 0x7f274c0787e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2758033000 tx=0x7f275802d000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:46.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.208+0000 7f275effd640 1 -- 172.21.15.59:0/1486694054 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2760159060 con 0x7f27600a94e0 2024-06-29T14:22:46.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.399+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27600aaad0 con 0x7f27600a94e0 2024-06-29T14:22:46.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.400+0000 7f275effd640 1 -- 172.21.15.59:0/1486694054 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7f27600ac310 con 0x7f27600a94e0 2024-06-29T14:22:46.399 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:46.399 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f274c0762d0 msgr2=0x7f274c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f274c0762d0 0x7f274c0787e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2758033000 tx=0x7f275802d000 comp rx=0 tx=0).stop 2024-06-29T14:22:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27600a94e0 msgr2=0x7f2760154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27600a94e0 0x7f2760154180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2754011ab0 tx=0x7f2754011f70 comp rx=0 tx=0).stop 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 shutdown_connections 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f274c0762d0 0x7f274c0787e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27600ab9c0 0x7f27600abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27600a94e0 0x7f2760154180 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 --2- 172.21.15.59:0/1486694054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2748003a70 0x7f27600aa500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 >> 172.21.15.59:0/1486694054 conn(0x7f2760073e10 msgr2=0x7f276015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 shutdown_connections 2024-06-29T14:22:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.402+0000 7f27683d2640 1 -- 172.21.15.59:0/1486694054 wait complete. 2024-06-29T14:22:46.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:46.412 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:46.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.534+0000 7f45cb03c640 1 -- 172.21.15.59:0/1728295858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 msgr2=0x7f45c406e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.534+0000 7f45cb03c640 1 --2- 172.21.15.59:0/1728295858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 0x7f45c406e4d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009830 tx=0x7f45ac02a530 comp rx=0 tx=0).stop 2024-06-29T14:22:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.534+0000 7f45cb03c640 1 -- 172.21.15.59:0/1728295858 shutdown_connections 2024-06-29T14:22:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.534+0000 7f45cb03c640 1 --2- 172.21.15.59:0/1728295858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 0x7f45c406e4d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.534+0000 7f45cb03c640 1 -- 172.21.15.59:0/1728295858 >> 172.21.15.59:0/1728295858 conn(0x7f45c40b2ab0 msgr2=0x7f45c40b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.534+0000 7f45cb03c640 1 -- 172.21.15.59:0/1728295858 shutdown_connections 2024-06-29T14:22:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.534+0000 7f45cb03c640 1 -- 172.21.15.59:0/1728295858 wait complete. 2024-06-29T14:22:46.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.536+0000 7f45cb03c640 1 Processor -- start 2024-06-29T14:22:46.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.536+0000 7f45cb03c640 1 -- start start 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45cb03c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45c406ea10 0x7f45c413ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45cb03c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c40d0130 0x7f45c415da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45cb03c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 0x7f45c413ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45cb03c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f45c40cf820 con 0x7f45c4143840 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45cb03c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45c40cf6a0 con 0x7f45c40d0130 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45cb03c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45c40cf9a0 con 0x7f45c406ea10 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 0x7f45c413ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 0x7f45c413ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42864/0 (socket says 172.21.15.59:42864) 2024-06-29T14:22:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 -- 172.21.15.59:0/1882969655 learned_addr learned my addr 172.21.15.59:0/1882969655 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c3fff640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c40d0130 0x7f45c415da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c8dd8640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45c406ea10 0x7f45c413ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 -- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45c406ea10 msgr2=0x7f45c413ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45c406ea10 0x7f45c413ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 -- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c40d0130 msgr2=0x7f45c415da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c40d0130 0x7f45c415da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 -- 172.21.15.59:0/1882969655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c41f1cf0 con 0x7f45c4143840 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c95d9640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 0x7f45c413ef30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f45b4002170 tx=0x7f45b400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.538+0000 7f45c17fa640 1 -- 172.21.15.59:0/1882969655 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4015070 con 0x7f45c4143840 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.539+0000 7f45c17fa640 1 -- 172.21.15.59:0/1882969655 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45b4003c80 con 0x7f45c4143840 2024-06-29T14:22:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.539+0000 7f45c17fa640 1 -- 172.21.15.59:0/1882969655 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b40052d0 con 0x7f45c4143840 2024-06-29T14:22:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.539+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c415b460 con 0x7f45c4143840 2024-06-29T14:22:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.539+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c415ba20 con 0x7f45c4143840 2024-06-29T14:22:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.540+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c40cfe10 con 0x7f45c4143840 2024-06-29T14:22:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.540+0000 7f45c17fa640 1 -- 172.21.15.59:0/1882969655 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f45b4018070 con 0x7f45c4143840 2024-06-29T14:22:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.541+0000 7f45c17fa640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f45a40762d0 0x7f45a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.541+0000 7f45c17fa640 1 -- 172.21.15.59:0/1882969655 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f45b4099730 con 0x7f45c4143840 2024-06-29T14:22:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.541+0000 7f45c8dd8640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f45a40762d0 0x7f45a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.542+0000 7f45c8dd8640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f45a40762d0 0x7f45a40787e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000f60 tx=0x7f45bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:46.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.545+0000 7f45c17fa640 1 -- 172.21.15.59:0/1882969655 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f45b40626d0 con 0x7f45c4143840 2024-06-29T14:22:46.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.737+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f45c406e7f0 con 0x7f45c4143840 2024-06-29T14:22:46.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.738+0000 7f45c17fa640 1 -- 172.21.15.59:0/1882969655 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f45b40669d0 con 0x7f45c4143840 2024-06-29T14:22:46.737 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f45a40762d0 msgr2=0x7f45a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f45a40762d0 0x7f45a40787e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000f60 tx=0x7f45bc009040 comp rx=0 tx=0).stop 2024-06-29T14:22:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 msgr2=0x7f45c413ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 0x7f45c413ef30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f45b4002170 tx=0x7f45b400cb30 comp rx=0 tx=0).stop 2024-06-29T14:22:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 shutdown_connections 2024-06-29T14:22:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f45a40762d0 0x7f45a40787e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c4143840 0x7f45c413ef30 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45c40d0130 0x7f45c415da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 --2- 172.21.15.59:0/1882969655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45c406ea10 0x7f45c413ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 >> 172.21.15.59:0/1882969655 conn(0x7f45c40b2ab0 msgr2=0x7f45c415a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:46.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 shutdown_connections 2024-06-29T14:22:46.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.740+0000 7f45cb03c640 1 -- 172.21.15.59:0/1882969655 wait complete. 2024-06-29T14:22:46.750 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:46.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.872+0000 7fa4a4b43640 1 -- 172.21.15.59:0/39478515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a0072cb0 msgr2=0x7fa4a01439b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.872+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/39478515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a0072cb0 0x7fa4a01439b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0144450 tx=0x7fa49002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.873+0000 7fa4a4b43640 1 -- 172.21.15.59:0/39478515 shutdown_connections 2024-06-29T14:22:46.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.873+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/39478515 >> v2:172.21.15.153:3300/0 conn(0x7fa4a00cf860 0x7fa4a00cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.873+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/39478515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a0072cb0 0x7fa4a01439b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.873+0000 7fa4a4b43640 1 -- 172.21.15.59:0/39478515 >> 172.21.15.59:0/39478515 conn(0x7fa4a00b30c0 msgr2=0x7fa4a00cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:46.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.873+0000 7fa4a4b43640 1 -- 172.21.15.59:0/39478515 shutdown_connections 2024-06-29T14:22:46.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.873+0000 7fa4a4b43640 1 -- 172.21.15.59:0/39478515 wait complete. 2024-06-29T14:22:46.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.874+0000 7fa4a4b43640 1 Processor -- start 2024-06-29T14:22:46.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.875+0000 7fa4a4b43640 1 -- start start 2024-06-29T14:22:46.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa4a4b43640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a006efa0 0x7fa4a01f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa4a4b43640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4a0072cb0 0x7fa4a0139850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa4a4b43640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a00cf860 0x7fa4a0139d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa4a4b43640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a015bb50 con 0x7fa4a006efa0 2024-06-29T14:22:46.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa4a4b43640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a015b9d0 con 0x7fa4a00cf860 2024-06-29T14:22:46.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa4a4b43640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a015bcd0 con 0x7fa4a0072cb0 2024-06-29T14:22:46.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a006efa0 0x7fa4a01f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a006efa0 0x7fa4a01f57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42886/0 (socket says 172.21.15.59:42886) 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49e59c640 1 -- 172.21.15.59:0/3296700521 learned_addr learned my addr 172.21.15.59:0/3296700521 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49ed9d640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a00cf860 0x7fa4a0139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49dd9b640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4a0072cb0 0x7fa4a0139850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49e59c640 1 -- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4a0072cb0 msgr2=0x7fa4a0139850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49e59c640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4a0072cb0 0x7fa4a0139850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49e59c640 1 -- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a00cf860 msgr2=0x7fa4a0139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49e59c640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a00cf860 0x7fa4a0139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.876+0000 7fa49e59c640 1 -- 172.21.15.59:0/3296700521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a013bd10 con 0x7fa4a006efa0 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.877+0000 7fa49e59c640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a006efa0 0x7fa4a01f57d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa490000c00 tx=0x7fa490003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.877+0000 7fa48effd640 1 -- 172.21.15.59:0/3296700521 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49003e030 con 0x7fa4a006efa0 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.877+0000 7fa48effd640 1 -- 172.21.15.59:0/3296700521 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa490003e00 con 0x7fa4a006efa0 2024-06-29T14:22:46.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.877+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a013bff0 con 0x7fa4a006efa0 2024-06-29T14:22:46.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.877+0000 7fa48effd640 1 -- 172.21.15.59:0/3296700521 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490035040 con 0x7fa4a006efa0 2024-06-29T14:22:46.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.877+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a00aac00 con 0x7fa4a006efa0 2024-06-29T14:22:46.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.878+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a015c1f0 con 0x7fa4a006efa0 2024-06-29T14:22:46.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.879+0000 7fa48effd640 1 -- 172.21.15.59:0/3296700521 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa490041030 con 0x7fa4a006efa0 2024-06-29T14:22:46.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.879+0000 7fa48effd640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa480076300 0x7fa480078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.879+0000 7fa48effd640 1 -- 172.21.15.59:0/3296700521 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fa4900b82b0 con 0x7fa4a006efa0 2024-06-29T14:22:46.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.879+0000 7fa49dd9b640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa480076300 0x7fa480078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.880+0000 7fa49dd9b640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa480076300 0x7fa480078810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa494000f60 tx=0x7fa494009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:46.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:46.883+0000 7fa48effd640 1 -- 172.21.15.59:0/3296700521 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa490081130 con 0x7fa4a006efa0 2024-06-29T14:22:47.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.076+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa4a00aae30 con 0x7fa4a006efa0 2024-06-29T14:22:47.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.076+0000 7fa48effd640 1 -- 172.21.15.59:0/3296700521 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7fa490085430 con 0x7fa4a006efa0 2024-06-29T14:22:47.075 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa480076300 msgr2=0x7fa480078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa480076300 0x7fa480078810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa494000f60 tx=0x7fa494009040 comp rx=0 tx=0).stop 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a006efa0 msgr2=0x7fa4a01f57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a006efa0 0x7fa4a01f57d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa490000c00 tx=0x7fa490003910 comp rx=0 tx=0).stop 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 shutdown_connections 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa480076300 0x7fa480078810 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a00cf860 0x7fa4a0139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4a0072cb0 0x7fa4a0139850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 --2- 172.21.15.59:0/3296700521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4a006efa0 0x7fa4a01f57d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.078+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 >> 172.21.15.59:0/3296700521 conn(0x7fa4a00b30c0 msgr2=0x7fa4a00b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.079+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 shutdown_connections 2024-06-29T14:22:47.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.079+0000 7fa4a4b43640 1 -- 172.21.15.59:0/3296700521 wait complete. 2024-06-29T14:22:47.089 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.211+0000 7f714e355640 1 -- 172.21.15.59:0/4054365439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7130004800 msgr2=0x7f7130004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.211+0000 7f714e355640 1 --2- 172.21.15.59:0/4054365439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7130004800 0x7f7130004c10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f713c00c290 tx=0x7f7130005830 comp rx=0 tx=0).stop 2024-06-29T14:22:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.211+0000 7f714c8f2640 1 -- 172.21.15.59:0/4054365439 reap_dead start 2024-06-29T14:22:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.211+0000 7f714e355640 1 -- 172.21.15.59:0/4054365439 shutdown_connections 2024-06-29T14:22:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.211+0000 7f714e355640 1 -- 172.21.15.59:0/4054365439 >> 172.21.15.59:0/4054365439 conn(0x7f714806c950 msgr2=0x7f7148073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.211+0000 7f714e355640 1 -- 172.21.15.59:0/4054365439 shutdown_connections 2024-06-29T14:22:47.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.212+0000 7f714e355640 1 -- 172.21.15.59:0/4054365439 wait complete. 2024-06-29T14:22:47.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.213+0000 7f714e355640 1 Processor -- start 2024-06-29T14:22:47.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.213+0000 7f714e355640 1 -- start start 2024-06-29T14:22:47.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f714e355640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7130004800 0x7f714815e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f714e355640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f714815b680 0x7f71480c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f714e355640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71480c92d0 0x7f71481f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f714e355640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f71480b5c10 con 0x7f714815b680 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f714e355640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71480b5a90 con 0x7f7130004800 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f714e355640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f71480b5d90 con 0x7f71480c92d0 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f714815b680 0x7f71480c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f714815b680 0x7f71480c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42906/0 (socket says 172.21.15.59:42906) 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 -- 172.21.15.59:0/3639286068 learned_addr learned my addr 172.21.15.59:0/3639286068 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f7147fff640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7130004800 0x7f714815e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f714c8f2640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71480c92d0 0x7f71481f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 -- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71480c92d0 msgr2=0x7f71481f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71480c92d0 0x7f71481f65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 -- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7130004800 msgr2=0x7f714815e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7130004800 0x7f714815e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 -- 172.21.15.59:0/3639286068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71480c96e0 con 0x7f714815b680 2024-06-29T14:22:47.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f714c8f2640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71480c92d0 0x7f71481f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.215+0000 7f71477fe640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f714815b680 0x7f71480c8d90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f713c00c3b0 tx=0x7f713c006ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.216+0000 7f7144ff9640 1 -- 172.21.15.59:0/3639286068 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c031040 con 0x7f714815b680 2024-06-29T14:22:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.216+0000 7f7144ff9640 1 -- 172.21.15.59:0/3639286068 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f713c0035f0 con 0x7f714815b680 2024-06-29T14:22:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.216+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f714815eb60 con 0x7f714815b680 2024-06-29T14:22:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.216+0000 7f7144ff9640 1 -- 172.21.15.59:0/3639286068 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c003a00 con 0x7f714815b680 2024-06-29T14:22:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.216+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f714815f350 con 0x7f714815b680 2024-06-29T14:22:47.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.217+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71481458a0 con 0x7f714815b680 2024-06-29T14:22:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.217+0000 7f7144ff9640 1 -- 172.21.15.59:0/3639286068 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f713c034070 con 0x7f714815b680 2024-06-29T14:22:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.218+0000 7f7144ff9640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71340762d0 0x7f71340787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.218+0000 7f7144ff9640 1 -- 172.21.15.59:0/3639286068 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f713c0b7c00 con 0x7f714815b680 2024-06-29T14:22:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.218+0000 7f7147fff640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71340762d0 0x7f71340787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.219+0000 7f7147fff640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71340762d0 0x7f71340787e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7138009c50 tx=0x7f7138005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:47.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.222+0000 7f7144ff9640 1 -- 172.21.15.59:0/3639286068 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f713c0bd020 con 0x7f714815b680 2024-06-29T14:22:47.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.416+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f71480d0e40 con 0x7f714815b680 2024-06-29T14:22:47.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.417+0000 7f7144ff9640 1 -- 172.21.15.59:0/3639286068 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f71480c96e0 con 0x7f714815b680 2024-06-29T14:22:47.416 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71340762d0 msgr2=0x7f71340787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71340762d0 0x7f71340787e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7138009c50 tx=0x7f7138005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f714815b680 msgr2=0x7f71480c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f714815b680 0x7f71480c8d90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f713c00c3b0 tx=0x7f713c006ce0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 shutdown_connections 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f71340762d0 0x7f71340787e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71480c92d0 0x7f71481f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f714815b680 0x7f71480c8d90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 --2- 172.21.15.59:0/3639286068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7130004800 0x7f714815e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.418+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 >> 172.21.15.59:0/3639286068 conn(0x7f714806c950 msgr2=0x7f71481e76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:47.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.419+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 shutdown_connections 2024-06-29T14:22:47.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.419+0000 7f714e355640 1 -- 172.21.15.59:0/3639286068 wait complete. 2024-06-29T14:22:47.429 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:47.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.551+0000 7f2d21f18640 1 -- 172.21.15.59:0/3465456703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c162770 msgr2=0x7f2d1c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.551+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3465456703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c162770 0x7f2d1c0ce550 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c005fd0 tx=0x7f2d0c02a540 comp rx=0 tx=0).stop 2024-06-29T14:22:47.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.551+0000 7f2d21f18640 1 -- 172.21.15.59:0/3465456703 shutdown_connections 2024-06-29T14:22:47.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.551+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3465456703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d1c162190 0x7f2d1c0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.551+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3465456703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c162770 0x7f2d1c0ce550 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.551+0000 7f2d21f18640 1 -- 172.21.15.59:0/3465456703 >> 172.21.15.59:0/3465456703 conn(0x7f2d1c0d45b0 msgr2=0x7f2d1c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:47.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.551+0000 7f2d21f18640 1 -- 172.21.15.59:0/3465456703 shutdown_connections 2024-06-29T14:22:47.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.551+0000 7f2d21f18640 1 -- 172.21.15.59:0/3465456703 wait complete. 2024-06-29T14:22:47.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.553+0000 7f2d21f18640 1 Processor -- start 2024-06-29T14:22:47.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.553+0000 7f2d21f18640 1 -- start start 2024-06-29T14:22:47.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d21f18640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c15e0a0 0x7f2d1c1f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d21f18640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1c162770 0x7f2d1c14bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d21f18640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d1c0c9a50 0x7f2d1c14c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d21f18640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c0b42c0 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d21f18640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c0b4140 con 0x7f2d1c162770 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d21f18640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2d1c0b4440 con 0x7f2d1c0c9a50 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c15e0a0 0x7f2d1c1f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c15e0a0 0x7f2d1c1f54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42920/0 (socket says 172.21.15.59:42920) 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1b7fe640 1 -- 172.21.15.59:0/3293310957 learned_addr learned my addr 172.21.15.59:0/3293310957 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1bfff640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d1c0c9a50 0x7f2d1c14c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1affd640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1c162770 0x7f2d1c14bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1b7fe640 1 -- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d1c0c9a50 msgr2=0x7f2d1c14c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1b7fe640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d1c0c9a50 0x7f2d1c14c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1b7fe640 1 -- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1c162770 msgr2=0x7f2d1c14bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.554+0000 7f2d1b7fe640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1c162770 0x7f2d1c14bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.555+0000 7f2d1b7fe640 1 -- 172.21.15.59:0/3293310957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c14bbe0 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.555+0000 7f2d1b7fe640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c15e0a0 0x7f2d1c1f54c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c034040 tx=0x7f2d0c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:47.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.555+0000 7f2d13fff640 1 -- 172.21.15.59:0/3293310957 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c03e030 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.555+0000 7f2d13fff640 1 -- 172.21.15.59:0/3293310957 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d0c003d90 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.555+0000 7f2d13fff640 1 -- 172.21.15.59:0/3293310957 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c035040 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.555+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1c1f5a00 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.555+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1c0c8c90 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.556+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d1c0b49d0 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.557+0000 7f2d13fff640 1 -- 172.21.15.59:0/3293310957 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d0c041030 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.557+0000 7f2d13fff640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.557+0000 7f2d13fff640 1 -- 172.21.15.59:0/3293310957 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f2d0c0b8300 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.558+0000 7f2d1affd640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.558+0000 7f2d1affd640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c1625c0 tx=0x7f2d04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:47.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.562+0000 7f2d13fff640 1 -- 172.21.15.59:0/3293310957 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d1c1f5a00 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.754+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2d1c06cf00 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.755+0000 7f2d13fff640 1 -- 172.21.15.59:0/3293310957 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f2d1c14bbe0 con 0x7f2d1c15e0a0 2024-06-29T14:22:47.754 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 msgr2=0x7f2cfc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c1625c0 tx=0x7f2d04009040 comp rx=0 tx=0).stop 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c15e0a0 msgr2=0x7f2d1c1f54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c15e0a0 0x7f2d1c1f54c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c034040 tx=0x7f2d0c0038a0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 shutdown_connections 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2cfc0762d0 0x7f2cfc0787e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d1c0c9a50 0x7f2d1c14c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1c162770 0x7f2d1c14bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 --2- 172.21.15.59:0/3293310957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d1c15e0a0 0x7f2d1c1f54c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 >> 172.21.15.59:0/3293310957 conn(0x7f2d1c0d45b0 msgr2=0x7f2d1c0b79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 shutdown_connections 2024-06-29T14:22:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.757+0000 7f2d21f18640 1 -- 172.21.15.59:0/3293310957 wait complete. 2024-06-29T14:22:47.767 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:22:47.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.888+0000 7f4e36dfc640 1 -- 172.21.15.59:0/1127509130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e30144470 msgr2=0x7f4e300b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.888+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/1127509130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e30144470 0x7f4e300b1b00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4e30072d60 tx=0x7f4e2002a560 comp rx=0 tx=0).stop 2024-06-29T14:22:47.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.888+0000 7f4e36dfc640 1 -- 172.21.15.59:0/1127509130 shutdown_connections 2024-06-29T14:22:47.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.888+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/1127509130 >> v2:172.21.15.153:3300/0 conn(0x7f4e3006ea10 0x7f4e300cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.888+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/1127509130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e30144470 0x7f4e300b1b00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.888+0000 7f4e36dfc640 1 -- 172.21.15.59:0/1127509130 >> 172.21.15.59:0/1127509130 conn(0x7f4e300b06e0 msgr2=0x7f4e300b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:47.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.888+0000 7f4e36dfc640 1 -- 172.21.15.59:0/1127509130 shutdown_connections 2024-06-29T14:22:47.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.888+0000 7f4e36dfc640 1 -- 172.21.15.59:0/1127509130 wait complete. 2024-06-29T14:22:47.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.890+0000 7f4e36dfc640 1 Processor -- start 2024-06-29T14:22:47.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.890+0000 7f4e36dfc640 1 -- start start 2024-06-29T14:22:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e36dfc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e3006ea10 0x7f4e3013fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e36dfc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e300b2040 0x7f4e301f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e36dfc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e30144470 0x7f4e301590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e36dfc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4e300cfaa0 con 0x7f4e300b2040 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e36dfc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e300cf920 con 0x7f4e30144470 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e36dfc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4e300cfc20 con 0x7f4e3006ea10 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e2ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e300b2040 0x7f4e301f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e2ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e300b2040 0x7f4e301f28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42946/0 (socket says 172.21.15.59:42946) 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e2ffff640 1 -- 172.21.15.59:0/2399623521 learned_addr learned my addr 172.21.15.59:0/2399623521 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e34b98640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e3006ea10 0x7f4e3013fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.891+0000 7f4e35399640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e30144470 0x7f4e301590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2ffff640 1 -- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e3006ea10 msgr2=0x7f4e3013fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2ffff640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e3006ea10 0x7f4e3013fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2ffff640 1 -- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e30144470 msgr2=0x7f4e301590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2ffff640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e30144470 0x7f4e301590c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2ffff640 1 -- 172.21.15.59:0/2399623521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e30155a90 con 0x7f4e300b2040 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2ffff640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e300b2040 0x7f4e301f28c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800b9c0 tx=0x7f4e1800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2d7fa640 1 -- 172.21.15.59:0/2399623521 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e18003b70 con 0x7f4e300b2040 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2d7fa640 1 -- 172.21.15.59:0/2399623521 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e18003e00 con 0x7f4e300b2040 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e30155d70 con 0x7f4e300b2040 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e2d7fa640 1 -- 172.21.15.59:0/2399623521 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1800cba0 con 0x7f4e300b2040 2024-06-29T14:22:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.892+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e301f2fb0 con 0x7f4e300b2040 2024-06-29T14:22:47.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.893+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e30155820 con 0x7f4e300b2040 2024-06-29T14:22:47.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.894+0000 7f4e2d7fa640 1 -- 172.21.15.59:0/2399623521 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4e18004000 con 0x7f4e300b2040 2024-06-29T14:22:47.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.895+0000 7f4e2d7fa640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e10076300 0x7f4e10078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.895+0000 7f4e2d7fa640 1 -- 172.21.15.59:0/2399623521 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f4e18097a10 con 0x7f4e300b2040 2024-06-29T14:22:47.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.895+0000 7f4e34b98640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e10076300 0x7f4e10078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.896+0000 7f4e34b98640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e10076300 0x7f4e10078810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4e20033000 tx=0x7f4e20004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:47.899+0000 7f4e2d7fa640 1 -- 172.21.15.59:0/2399623521 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4e18060890 con 0x7f4e300b2040 2024-06-29T14:22:48.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.100+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4e300d0180 con 0x7f4e300b2040 2024-06-29T14:22:48.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.101+0000 7f4e2d7fa640 1 -- 172.21.15.59:0/2399623521 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7f4e18064b90 con 0x7f4e300b2040 2024-06-29T14:22:48.100 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-06-29T14:22:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e10076300 msgr2=0x7f4e10078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e10076300 0x7f4e10078810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4e20033000 tx=0x7f4e20004730 comp rx=0 tx=0).stop 2024-06-29T14:22:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e300b2040 msgr2=0x7f4e301f28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e300b2040 0x7f4e301f28c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800b9c0 tx=0x7f4e1800be80 comp rx=0 tx=0).stop 2024-06-29T14:22:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 shutdown_connections 2024-06-29T14:22:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e10076300 0x7f4e10078810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e30144470 0x7f4e301590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e300b2040 0x7f4e301f28c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 --2- 172.21.15.59:0/2399623521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e3006ea10 0x7f4e3013fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 >> 172.21.15.59:0/2399623521 conn(0x7f4e300b06e0 msgr2=0x7f4e300b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 shutdown_connections 2024-06-29T14:22:48.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.103+0000 7f4e36dfc640 1 -- 172.21.15.59:0/2399623521 wait complete. 2024-06-29T14:22:48.113 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-06-29T14:22:48.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:48.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-06-29T14:22:48.145 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T14:22:48.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:48.145 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-06-29T14:22:48.283 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:48.284 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:48.301 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:48.301 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:48.357 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:22:48.357 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:48.414 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:48.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.573+0000 7efc14dee640 1 -- 172.21.15.59:0/231736322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efbfc005150 msgr2=0x7efbfc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.573+0000 7efc14dee640 1 --2- 172.21.15.59:0/231736322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efbfc005150 0x7efbfc00b9c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efc08009b20 tx=0x7efbfc005830 comp rx=0 tx=0).stop 2024-06-29T14:22:48.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.573+0000 7efc0ed9d640 1 -- 172.21.15.59:0/231736322 reap_dead start 2024-06-29T14:22:48.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.573+0000 7efc14dee640 1 -- 172.21.15.59:0/231736322 shutdown_connections 2024-06-29T14:22:48.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.573+0000 7efc14dee640 1 -- 172.21.15.59:0/231736322 >> 172.21.15.59:0/231736322 conn(0x7efc1006c950 msgr2=0x7efc10073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.573+0000 7efc14dee640 1 -- 172.21.15.59:0/231736322 shutdown_connections 2024-06-29T14:22:48.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.573+0000 7efc14dee640 1 -- 172.21.15.59:0/231736322 wait complete. 2024-06-29T14:22:48.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.575+0000 7efc14dee640 1 Processor -- start 2024-06-29T14:22:48.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.575+0000 7efc14dee640 1 -- start start 2024-06-29T14:22:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc14dee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc005150 0x7efc1006db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc14dee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc10142470 0x7efc1006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc14dee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc101f4ac0 0x7efc1006e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc14dee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efc100cf870 con 0x7efc10142470 2024-06-29T14:22:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc14dee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc10145ab0 con 0x7efbfc005150 2024-06-29T14:22:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc14dee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efc100cf9f0 con 0x7efc101f4ac0 2024-06-29T14:22:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc0dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc10142470 0x7efc1006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc0dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc10142470 0x7efc1006e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42956/0 (socket says 172.21.15.59:42956) 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc0dd9b640 1 -- 172.21.15.59:0/2253094499 learned_addr learned my addr 172.21.15.59:0/2253094499 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc0e59c640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc005150 0x7efc1006db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.576+0000 7efc0ed9d640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc101f4ac0 0x7efc1006e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc0dd9b640 1 -- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc101f4ac0 msgr2=0x7efc1006e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc0dd9b640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc101f4ac0 0x7efc1006e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc0dd9b640 1 -- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc005150 msgr2=0x7efc1006db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc0dd9b640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc005150 0x7efc1006db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc0dd9b640 1 -- 172.21.15.59:0/2253094499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1015bcc0 con 0x7efc10142470 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc0dd9b640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc10142470 0x7efc1006e0c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efc100b1c40 tx=0x7efbf80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc06ffd640 1 -- 172.21.15.59:0/2253094499 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8015070 con 0x7efc10142470 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc06ffd640 1 -- 172.21.15.59:0/2253094499 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efbf8011920 con 0x7efc10142470 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc101399d0 con 0x7efc10142470 2024-06-29T14:22:48.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc06ffd640 1 -- 172.21.15.59:0/2253094499 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8011d70 con 0x7efc10142470 2024-06-29T14:22:48.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.577+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc10139f90 con 0x7efc10142470 2024-06-29T14:22:48.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.578+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc100cfd50 con 0x7efc10142470 2024-06-29T14:22:48.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.579+0000 7efc06ffd640 1 -- 172.21.15.59:0/2253094499 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efbf8018070 con 0x7efc10142470 2024-06-29T14:22:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.580+0000 7efc06ffd640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efbf0076300 0x7efbf0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.580+0000 7efc06ffd640 1 -- 172.21.15.59:0/2253094499 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7efbf8098870 con 0x7efc10142470 2024-06-29T14:22:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.580+0000 7efc0e59c640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efbf0076300 0x7efbf0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.580+0000 7efc0e59c640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efbf0076300 0x7efbf0078810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efc00005fd0 tx=0x7efc00009500 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.584+0000 7efc06ffd640 1 -- 172.21.15.59:0/2253094499 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efbf8061790 con 0x7efc10142470 2024-06-29T14:22:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.775+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc100b1f10 con 0x7efc10142470 2024-06-29T14:22:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.776+0000 7efc06ffd640 1 -- 172.21.15.59:0/2253094499 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7efbf8065a90 con 0x7efc10142470 2024-06-29T14:22:48.776 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:48.776 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efbf0076300 msgr2=0x7efbf0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efbf0076300 0x7efbf0078810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efc00005fd0 tx=0x7efc00009500 comp rx=0 tx=0).stop 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc10142470 msgr2=0x7efc1006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc10142470 0x7efc1006e0c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efc100b1c40 tx=0x7efbf80039c0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 shutdown_connections 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efbf0076300 0x7efbf0078810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc101f4ac0 0x7efc1006e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc10142470 0x7efc1006e0c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 --2- 172.21.15.59:0/2253094499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc005150 0x7efc1006db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.778+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 >> 172.21.15.59:0/2253094499 conn(0x7efc1006c950 msgr2=0x7efc100d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.779+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 shutdown_connections 2024-06-29T14:22:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.779+0000 7efc14dee640 1 -- 172.21.15.59:0/2253094499 wait complete. 2024-06-29T14:22:48.789 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:48.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:48.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.913+0000 7fd1f6efe640 1 -- 172.21.15.59:0/2289574665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 msgr2=0x7fd1f00cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.913+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/2289574665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 0x7fd1f00cf080 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009900 tx=0x7fd1e002a610 comp rx=0 tx=0).stop 2024-06-29T14:22:48.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.913+0000 7fd1f6efe640 1 -- 172.21.15.59:0/2289574665 shutdown_connections 2024-06-29T14:22:48.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.913+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/2289574665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f00ce950 0x7fd1f00d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.913+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/2289574665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 0x7fd1f00cf080 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.913+0000 7fd1f6efe640 1 -- 172.21.15.59:0/2289574665 >> 172.21.15.59:0/2289574665 conn(0x7fd1f00b5610 msgr2=0x7fd1f00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.913+0000 7fd1f6efe640 1 -- 172.21.15.59:0/2289574665 shutdown_connections 2024-06-29T14:22:48.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.913+0000 7fd1f6efe640 1 -- 172.21.15.59:0/2289574665 wait complete. 2024-06-29T14:22:48.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.915+0000 7fd1f6efe640 1 Processor -- start 2024-06-29T14:22:48.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.915+0000 7fd1f6efe640 1 -- start start 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.916+0000 7fd1f6efe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 0x7fd1f01f8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.916+0000 7fd1f6efe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f00ce950 0x7fd1f0159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.916+0000 7fd1f6efe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1f00cf5c0 0x7fd1f01604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.916+0000 7fd1f6efe640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f00b7f60 con 0x7fd1f008c190 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.916+0000 7fd1f6efe640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f00b7de0 con 0x7fd1f00cf5c0 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.916+0000 7fd1f6efe640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f00b80e0 con 0x7fd1f00ce950 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 0x7fd1f01f8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 0x7fd1f01f8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42982/0 (socket says 172.21.15.59:42982) 2024-06-29T14:22:48.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 -- 172.21.15.59:0/1939775454 learned_addr learned my addr 172.21.15.59:0/1939775454 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1effff640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f00ce950 0x7fd1f0159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f549b640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1f00cf5c0 0x7fd1f01604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 -- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f00ce950 msgr2=0x7fd1f0159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f00ce950 0x7fd1f0159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 -- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1f00cf5c0 msgr2=0x7fd1f01604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1f00cf5c0 0x7fd1f01604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 -- 172.21.15.59:0/1939775454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f01f94c0 con 0x7fd1f008c190 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1effff640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f00ce950 0x7fd1f0159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f4c9a640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 0x7fd1f01f8f80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009a20 tx=0x7fd1e0003690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1ed7fa640 1 -- 172.21.15.59:0/1939775454 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e003e030 con 0x7fd1f008c190 2024-06-29T14:22:48.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1ed7fa640 1 -- 172.21.15.59:0/1939775454 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1e0003b80 con 0x7fd1f008c190 2024-06-29T14:22:48.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1ed7fa640 1 -- 172.21.15.59:0/1939775454 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0035040 con 0x7fd1f008c190 2024-06-29T14:22:48.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.917+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f01e9af0 con 0x7fd1f008c190 2024-06-29T14:22:48.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.918+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f01e7000 con 0x7fd1f008c190 2024-06-29T14:22:48.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.919+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f00cf080 con 0x7fd1f008c190 2024-06-29T14:22:48.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.919+0000 7fd1ed7fa640 1 -- 172.21.15.59:0/1939775454 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd1e0041030 con 0x7fd1f008c190 2024-06-29T14:22:48.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.920+0000 7fd1ed7fa640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd1dc076300 0x7fd1dc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.920+0000 7fd1ed7fa640 1 -- 172.21.15.59:0/1939775454 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd1e00b80c0 con 0x7fd1f008c190 2024-06-29T14:22:48.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.920+0000 7fd1effff640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd1dc076300 0x7fd1dc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.921+0000 7fd1effff640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd1dc076300 0x7fd1dc078810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4007fd0 tx=0x7fd1e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:48.925+0000 7fd1ed7fa640 1 -- 172.21.15.59:0/1939775454 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1e0081590 con 0x7fd1f008c190 2024-06-29T14:22:49.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.114+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1f00cbc00 con 0x7fd1f008c190 2024-06-29T14:22:49.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.115+0000 7fd1ed7fa640 1 -- 172.21.15.59:0/1939775454 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fd1e0080e90 con 0x7fd1f008c190 2024-06-29T14:22:49.114 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.116+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd1dc076300 msgr2=0x7fd1dc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd1dc076300 0x7fd1dc078810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4007fd0 tx=0x7fd1e4009040 comp rx=0 tx=0).stop 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 msgr2=0x7fd1f01f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 0x7fd1f01f8f80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009a20 tx=0x7fd1e0003690 comp rx=0 tx=0).stop 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 shutdown_connections 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd1dc076300 0x7fd1dc078810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1f00cf5c0 0x7fd1f01604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f00ce950 0x7fd1f0159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 --2- 172.21.15.59:0/1939775454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f008c190 0x7fd1f01f8f80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 >> 172.21.15.59:0/1939775454 conn(0x7fd1f00b5610 msgr2=0x7fd1f015e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 shutdown_connections 2024-06-29T14:22:49.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.117+0000 7fd1f6efe640 1 -- 172.21.15.59:0/1939775454 wait complete. 2024-06-29T14:22:49.127 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:49.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.248+0000 7f3ff003f640 1 -- 172.21.15.59:0/2722921761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8147370 msgr2=0x7f3fe80b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.248+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2722921761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8147370 0x7f3fe80b1d10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8074a60 tx=0x7f3fe002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.248+0000 7f3ff003f640 1 -- 172.21.15.59:0/2722921761 shutdown_connections 2024-06-29T14:22:49.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.248+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2722921761 >> v2:172.21.15.153:3300/0 conn(0x7f3fe8161b00 0x7f3fe8161f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.248+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2722921761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8147370 0x7f3fe80b1d10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.248+0000 7f3ff003f640 1 -- 172.21.15.59:0/2722921761 >> 172.21.15.59:0/2722921761 conn(0x7f3fe806c950 msgr2=0x7f3fe8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.248+0000 7f3ff003f640 1 -- 172.21.15.59:0/2722921761 shutdown_connections 2024-06-29T14:22:49.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.248+0000 7f3ff003f640 1 -- 172.21.15.59:0/2722921761 wait complete. 2024-06-29T14:22:49.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.250+0000 7f3ff003f640 1 Processor -- start 2024-06-29T14:22:49.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.250+0000 7f3ff003f640 1 -- start start 2024-06-29T14:22:49.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3ff003f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe806f8a0 0x7f3fe813de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3ff003f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe8147370 0x7f3fe81e7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3ff003f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8161b00 0x7f3fe8141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3ff003f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe815ea90 con 0x7f3fe8161b00 2024-06-29T14:22:49.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3ff003f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe815e910 con 0x7f3fe806f8a0 2024-06-29T14:22:49.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3ff003f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3fe815ec10 con 0x7f3fe8147370 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3fee5dc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8161b00 0x7f3fe8141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3fee5dc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8161b00 0x7f3fe8141830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43008/0 (socket says 172.21.15.59:43008) 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3fedddb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe806f8a0 0x7f3fe813de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3fee5dc640 1 -- 172.21.15.59:0/2717546703 learned_addr learned my addr 172.21.15.59:0/2717546703 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.251+0000 7f3fed5da640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe8147370 0x7f3fe81e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fee5dc640 1 -- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe8147370 msgr2=0x7f3fe81e7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fee5dc640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe8147370 0x7f3fe81e7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fee5dc640 1 -- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe806f8a0 msgr2=0x7f3fe813de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fee5dc640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe806f8a0 0x7f3fe813de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fee5dc640 1 -- 172.21.15.59:0/2717546703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe8140790 con 0x7f3fe8161b00 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fee5dc640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8161b00 0x7f3fe8141830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800d9e0 tx=0x7f3fd800dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fe67fc640 1 -- 172.21.15.59:0/2717546703 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8017030 con 0x7f3fe8161b00 2024-06-29T14:22:49.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fe67fc640 1 -- 172.21.15.59:0/2717546703 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fd800bc30 con 0x7f3fe8161b00 2024-06-29T14:22:49.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3fe67fc640 1 -- 172.21.15.59:0/2717546703 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8003970 con 0x7f3fe8161b00 2024-06-29T14:22:49.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe8140a70 con 0x7f3fe8161b00 2024-06-29T14:22:49.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.252+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe8141030 con 0x7f3fe8161b00 2024-06-29T14:22:49.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.253+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fe815f130 con 0x7f3fe8161b00 2024-06-29T14:22:49.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.254+0000 7f3fe67fc640 1 -- 172.21.15.59:0/2717546703 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3fd801a070 con 0x7f3fe8161b00 2024-06-29T14:22:49.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.254+0000 7f3fe67fc640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3fd40762d0 0x7f3fd40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.254+0000 7f3fe67fc640 1 -- 172.21.15.59:0/2717546703 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f3fd8097e80 con 0x7f3fe8161b00 2024-06-29T14:22:49.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.255+0000 7f3fedddb640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3fd40762d0 0x7f3fd40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.255+0000 7f3fedddb640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3fd40762d0 0x7f3fd40787e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0033000 tx=0x7f3fe0004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.259+0000 7f3fe67fc640 1 -- 172.21.15.59:0/2717546703 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3fd8060d90 con 0x7f3fe8161b00 2024-06-29T14:22:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.448+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3fe80742a0 con 0x7f3fe8161b00 2024-06-29T14:22:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.449+0000 7f3fe67fc640 1 -- 172.21.15.59:0/2717546703 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f3fd8065090 con 0x7f3fe8161b00 2024-06-29T14:22:49.448 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3fd40762d0 msgr2=0x7f3fd40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3fd40762d0 0x7f3fd40787e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0033000 tx=0x7f3fe0004730 comp rx=0 tx=0).stop 2024-06-29T14:22:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8161b00 msgr2=0x7f3fe8141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8161b00 0x7f3fe8141830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800d9e0 tx=0x7f3fd800dea0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 shutdown_connections 2024-06-29T14:22:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3fd40762d0 0x7f3fd40787e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fe8161b00 0x7f3fe8141830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3fe8147370 0x7f3fe81e7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 --2- 172.21.15.59:0/2717546703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fe806f8a0 0x7f3fe813de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 >> 172.21.15.59:0/2717546703 conn(0x7f3fe806c950 msgr2=0x7f3fe80b2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 shutdown_connections 2024-06-29T14:22:49.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.451+0000 7f3ff003f640 1 -- 172.21.15.59:0/2717546703 wait complete. 2024-06-29T14:22:49.461 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:49.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.584+0000 7f886e94b640 1 -- 172.21.15.59:0/2185427879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 msgr2=0x7f8854003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.584+0000 7f886e94b640 1 --2- 172.21.15.59:0/2185427879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 0x7f8854003ee0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8858009b20 tx=0x7f8854005830 comp rx=0 tx=0).stop 2024-06-29T14:22:49.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.584+0000 7f886cee8640 1 -- 172.21.15.59:0/2185427879 reap_dead start 2024-06-29T14:22:49.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.584+0000 7f886e94b640 1 -- 172.21.15.59:0/2185427879 shutdown_connections 2024-06-29T14:22:49.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.584+0000 7f886e94b640 1 -- 172.21.15.59:0/2185427879 >> 172.21.15.59:0/2185427879 conn(0x7f88680b24c0 msgr2=0x7f88680cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.584+0000 7f886e94b640 1 -- 172.21.15.59:0/2185427879 shutdown_connections 2024-06-29T14:22:49.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.584+0000 7f886e94b640 1 -- 172.21.15.59:0/2185427879 wait complete. 2024-06-29T14:22:49.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.586+0000 7f886e94b640 1 Processor -- start 2024-06-29T14:22:49.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.586+0000 7f886e94b640 1 -- start start 2024-06-29T14:22:49.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f886e94b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 0x7f8868159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f886e94b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88681f8540 0x7f88681f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f886e94b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8868147b30 0x7f8868147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f886e94b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f886806cfb0 con 0x7f8854003a70 2024-06-29T14:22:49.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f886e94b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f886806ce30 con 0x7f88681f8540 2024-06-29T14:22:49.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f886e94b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f88680ae770 con 0x7f8868147b30 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f8867fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 0x7f8868159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f8867fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 0x7f8868159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43024/0 (socket says 172.21.15.59:43024) 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f8867fff640 1 -- 172.21.15.59:0/144587025 learned_addr learned my addr 172.21.15.59:0/144587025 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f886cee8640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8868147b30 0x7f8868147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.587+0000 7f885ffff640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88681f8540 0x7f88681f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f8867fff640 1 -- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8868147b30 msgr2=0x7f8868147f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f8867fff640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8868147b30 0x7f8868147f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f8867fff640 1 -- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88681f8540 msgr2=0x7f88681f8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f8867fff640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88681f8540 0x7f88681f8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f8867fff640 1 -- 172.21.15.59:0/144587025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88681f8e90 con 0x7f8854003a70 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f8867fff640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 0x7f8868159130 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8858036040 tx=0x7f8858033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f88657fa640 1 -- 172.21.15.59:0/144587025 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858038040 con 0x7f8854003a70 2024-06-29T14:22:49.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f88657fa640 1 -- 172.21.15.59:0/144587025 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f885802b7b0 con 0x7f8854003a70 2024-06-29T14:22:49.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88681ea610 con 0x7f8854003a70 2024-06-29T14:22:49.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f88657fa640 1 -- 172.21.15.59:0/144587025 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858033ac0 con 0x7f8854003a70 2024-06-29T14:22:49.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.588+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88681eabd0 con 0x7f8854003a70 2024-06-29T14:22:49.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.589+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88680ccda0 con 0x7f8854003a70 2024-06-29T14:22:49.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.590+0000 7f88657fa640 1 -- 172.21.15.59:0/144587025 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8858045020 con 0x7f8854003a70 2024-06-29T14:22:49.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.590+0000 7f88657fa640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88480762d0 0x7f88480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.590+0000 7f88657fa640 1 -- 172.21.15.59:0/144587025 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f88580b81e0 con 0x7f8854003a70 2024-06-29T14:22:49.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.591+0000 7f885ffff640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88480762d0 0x7f88480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.591+0000 7f885ffff640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88480762d0 0x7f88480787e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f886815cf10 tx=0x7f88500012d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.595+0000 7f88657fa640 1 -- 172.21.15.59:0/144587025 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f885804c020 con 0x7f8854003a70 2024-06-29T14:22:49.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.785+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f88680ce1a0 con 0x7f8854003a70 2024-06-29T14:22:49.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.785+0000 7f88657fa640 1 -- 172.21.15.59:0/144587025 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f88681f8e90 con 0x7f8854003a70 2024-06-29T14:22:49.784 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:49.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88480762d0 msgr2=0x7f88480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88480762d0 0x7f88480787e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f886815cf10 tx=0x7f88500012d0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 msgr2=0x7f8868159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 0x7f8868159130 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8858036040 tx=0x7f8858033670 comp rx=0 tx=0).stop 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 shutdown_connections 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f88480762d0 0x7f88480787e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8868147b30 0x7f8868147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88681f8540 0x7f88681f8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 --2- 172.21.15.59:0/144587025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8854003a70 0x7f8868159130 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 >> 172.21.15.59:0/144587025 conn(0x7f88680b24c0 msgr2=0x7f886815da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 shutdown_connections 2024-06-29T14:22:49.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.787+0000 7f886e94b640 1 -- 172.21.15.59:0/144587025 wait complete. 2024-06-29T14:22:49.797 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.919+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/673750467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0d1a60 msgr2=0x7fc09c0b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.919+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/673750467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0d1a60 0x7fc09c0b5b30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0b2680 tx=0x7fc09402a560 comp rx=0 tx=0).stop 2024-06-29T14:22:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.919+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/673750467 shutdown_connections 2024-06-29T14:22:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.919+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/673750467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0d1a60 0x7fc09c0b5b30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.919+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/673750467 >> v2:172.21.15.153:3300/0 conn(0x7fc09c0b2fa0 0x7fc09c0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.919+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/673750467 >> 172.21.15.59:0/673750467 conn(0x7fc09c073e10 msgr2=0x7fc09c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.919+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/673750467 shutdown_connections 2024-06-29T14:22:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.919+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/673750467 wait complete. 2024-06-29T14:22:49.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.921+0000 7fc0a1bf5640 1 Processor -- start 2024-06-29T14:22:49.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.921+0000 7fc0a1bf5640 1 -- start start 2024-06-29T14:22:49.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc0a1bf5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc09c0b2270 0x7fc09c148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc0a1bf5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0b2fa0 0x7fc09c1f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc0a1bf5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc09c0d1a60 0x7fc09c15ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc0a1bf5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc09c0b6400 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc0a1bf5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc09c0b6280 con 0x7fc09c0d1a60 2024-06-29T14:22:49.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc0a1bf5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc09c0b6580 con 0x7fc09c0b2270 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0b2fa0 0x7fc09c1f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0b2fa0 0x7fc09c1f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43044/0 (socket says 172.21.15.59:43044) 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 -- 172.21.15.59:0/4071932227 learned_addr learned my addr 172.21.15.59:0/4071932227 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09b7fe640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc09c0b2270 0x7fc09c148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09bfff640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc09c0d1a60 0x7fc09c15ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 -- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc09c0b2270 msgr2=0x7fc09c148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc09c0b2270 0x7fc09c148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 -- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc09c0d1a60 msgr2=0x7fc09c15ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc09c0d1a60 0x7fc09c15ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 -- 172.21.15.59:0/4071932227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09c1f2d80 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.923+0000 7fc09affd640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0b2fa0 0x7fc09c1f2840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc09000d6d0 tx=0x7fc09000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.924+0000 7fc083fff640 1 -- 172.21.15.59:0/4071932227 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090017070 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.924+0000 7fc083fff640 1 -- 172.21.15.59:0/4071932227 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc090005330 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.924+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc09c074470 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.924+0000 7fc083fff640 1 -- 172.21.15.59:0/4071932227 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090009820 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.924+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc09c15e230 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.925+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc09c0b6ac0 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.925+0000 7fc083fff640 1 -- 172.21.15.59:0/4071932227 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc090009bd0 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.926+0000 7fc083fff640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0880762d0 0x7fc0880787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.926+0000 7fc083fff640 1 -- 172.21.15.59:0/4071932227 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fc090097dc0 con 0x7fc09c0b2fa0 2024-06-29T14:22:49.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.926+0000 7fc09b7fe640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0880762d0 0x7fc0880787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.927+0000 7fc09b7fe640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0880762d0 0x7fc0880787e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00b450 tx=0x7fc08c00d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:49.930+0000 7fc083fff640 1 -- 172.21.15.59:0/4071932227 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc090060c40 con 0x7fc09c0b2fa0 2024-06-29T14:22:50.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.121+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc09c0b4cc0 con 0x7fc09c0b2fa0 2024-06-29T14:22:50.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.121+0000 7fc083fff640 1 -- 172.21.15.59:0/4071932227 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fc090069310 con 0x7fc09c0b2fa0 2024-06-29T14:22:50.120 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:50.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0880762d0 msgr2=0x7fc0880787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0880762d0 0x7fc0880787e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00b450 tx=0x7fc08c00d040 comp rx=0 tx=0).stop 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0b2fa0 msgr2=0x7fc09c1f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0b2fa0 0x7fc09c1f2840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc09000d6d0 tx=0x7fc09000db90 comp rx=0 tx=0).stop 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 shutdown_connections 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc0880762d0 0x7fc0880787e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc09c0d1a60 0x7fc09c15ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc09c0b2fa0 0x7fc09c1f2840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 --2- 172.21.15.59:0/4071932227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc09c0b2270 0x7fc09c148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 >> 172.21.15.59:0/4071932227 conn(0x7fc09c073e10 msgr2=0x7fc09c15bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 shutdown_connections 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.123+0000 7fc0a1bf5640 1 -- 172.21.15.59:0/4071932227 wait complete. 2024-06-29T14:22:50.134 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:22:50.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.254+0000 7fe40e881640 1 -- 172.21.15.59:0/185105219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4080d1c30 msgr2=0x7fe408166a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.254+0000 7fe40e881640 1 --2- 172.21.15.59:0/185105219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4080d1c30 0x7fe408166a20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe400005fd0 tx=0x7fe40002a940 comp rx=0 tx=0).stop 2024-06-29T14:22:50.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.254+0000 7fe40e881640 1 -- 172.21.15.59:0/185105219 shutdown_connections 2024-06-29T14:22:50.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.254+0000 7fe40e881640 1 --2- 172.21.15.59:0/185105219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4080d1c30 0x7fe408166a20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.254+0000 7fe40e881640 1 -- 172.21.15.59:0/185105219 >> 172.21.15.59:0/185105219 conn(0x7fe408073e10 msgr2=0x7fe408074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.254+0000 7fe40e881640 1 -- 172.21.15.59:0/185105219 shutdown_connections 2024-06-29T14:22:50.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.255+0000 7fe40e881640 1 -- 172.21.15.59:0/185105219 wait complete. 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.256+0000 7fe40e881640 1 Processor -- start 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.256+0000 7fe40e881640 1 -- start start 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe40e881640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4080b2270 0x7fe40806dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe40e881640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4080b2fa0 0x7fe40814b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe40e881640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4080d1c30 0x7fe40814bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe40e881640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe4081670e0 con 0x7fe4080b2fa0 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe40e881640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe408166f60 con 0x7fe4080d1c30 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe40e881640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe408167260 con 0x7fe4080b2270 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe4077fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4080b2fa0 0x7fe40814b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe4077fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4080b2fa0 0x7fe40814b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43054/0 (socket says 172.21.15.59:43054) 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe4077fe640 1 -- 172.21.15.59:0/281892765 learned_addr learned my addr 172.21.15.59:0/281892765 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe40ce1e640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4080d1c30 0x7fe40814bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe407fff640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4080b2270 0x7fe40806dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe4077fe640 1 -- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4080b2270 msgr2=0x7fe40806dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe4077fe640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4080b2270 0x7fe40806dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe4077fe640 1 -- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4080d1c30 msgr2=0x7fe40814bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe4077fe640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4080d1c30 0x7fe40814bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.258+0000 7fe4077fe640 1 -- 172.21.15.59:0/281892765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40806e6f0 con 0x7fe4080b2fa0 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.259+0000 7fe4077fe640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4080b2fa0 0x7fe40814b240 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00c390 tx=0x7fe3fc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.259+0000 7fe404ff9640 1 -- 172.21.15.59:0/281892765 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc011790 con 0x7fe4080b2fa0 2024-06-29T14:22:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.259+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40806e9d0 con 0x7fe4080b2fa0 2024-06-29T14:22:50.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.259+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40806ef90 con 0x7fe4080b2fa0 2024-06-29T14:22:50.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.259+0000 7fe404ff9640 1 -- 172.21.15.59:0/281892765 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3fc017070 con 0x7fe4080b2fa0 2024-06-29T14:22:50.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.259+0000 7fe404ff9640 1 -- 172.21.15.59:0/281892765 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc011c60 con 0x7fe4080b2fa0 2024-06-29T14:22:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.260+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe40806e460 con 0x7fe4080b2fa0 2024-06-29T14:22:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.261+0000 7fe404ff9640 1 -- 172.21.15.59:0/281892765 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe3fc012420 con 0x7fe4080b2fa0 2024-06-29T14:22:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.261+0000 7fe404ff9640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3f4076300 0x7fe3f4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.262+0000 7fe404ff9640 1 -- 172.21.15.59:0/281892765 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fe3fc097700 con 0x7fe4080b2fa0 2024-06-29T14:22:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.262+0000 7fe407fff640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3f4076300 0x7fe3f4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.262+0000 7fe407fff640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3f4076300 0x7fe3f4078810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8001120 tx=0x7fe3f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.266+0000 7fe404ff9640 1 -- 172.21.15.59:0/281892765 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe3fc021020 con 0x7fe4080b2fa0 2024-06-29T14:22:50.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.473+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe408158d00 con 0x7fe4080b2fa0 2024-06-29T14:22:50.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.473+0000 7fe404ff9640 1 -- 172.21.15.59:0/281892765 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7fe3fc060580 con 0x7fe4080b2fa0 2024-06-29T14:22:50.472 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-06-29T14:22:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3f4076300 msgr2=0x7fe3f4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3f4076300 0x7fe3f4078810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8001120 tx=0x7fe3f8009040 comp rx=0 tx=0).stop 2024-06-29T14:22:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4080b2fa0 msgr2=0x7fe40814b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4080b2fa0 0x7fe40814b240 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc00c390 tx=0x7fe3fc00c850 comp rx=0 tx=0).stop 2024-06-29T14:22:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 shutdown_connections 2024-06-29T14:22:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe3f4076300 0x7fe3f4078810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4080d1c30 0x7fe40814bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4080b2fa0 0x7fe40814b240 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 --2- 172.21.15.59:0/281892765 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4080b2270 0x7fe40806dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 >> 172.21.15.59:0/281892765 conn(0x7fe408073e10 msgr2=0x7fe4080d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 shutdown_connections 2024-06-29T14:22:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.476+0000 7fe40e881640 1 -- 172.21.15.59:0/281892765 wait complete. 2024-06-29T14:22:50.486 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-06-29T14:22:50.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:50.487 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-06-29T14:22:50.514 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:22:50.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:50.515 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-06-29T14:22:50.571 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:22:50.571 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:50.592 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:22:50.592 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:50.653 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:22:50.653 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:22:50.712 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:50.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.831+0000 7f22d68ac640 1 -- 172.21.15.59:0/3311478846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 msgr2=0x7f22d00b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.831+0000 7f22d68ac640 1 --2- 172.21.15.59:0/3311478846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 0x7f22d00b3990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f22c0009830 tx=0x7f22c002a510 comp rx=0 tx=0).stop 2024-06-29T14:22:50.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.831+0000 7f22d68ac640 1 -- 172.21.15.59:0/3311478846 shutdown_connections 2024-06-29T14:22:50.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.831+0000 7f22d68ac640 1 --2- 172.21.15.59:0/3311478846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22d00b3ed0 0x7f22d015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.831+0000 7f22d68ac640 1 --2- 172.21.15.59:0/3311478846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 0x7f22d00b3990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.831+0000 7f22d68ac640 1 -- 172.21.15.59:0/3311478846 >> 172.21.15.59:0/3311478846 conn(0x7f22d00d3ee0 msgr2=0x7f22d00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.831+0000 7f22d68ac640 1 -- 172.21.15.59:0/3311478846 shutdown_connections 2024-06-29T14:22:50.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.831+0000 7f22d68ac640 1 -- 172.21.15.59:0/3311478846 wait complete. 2024-06-29T14:22:50.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.833+0000 7f22d68ac640 1 Processor -- start 2024-06-29T14:22:50.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.833+0000 7f22d68ac640 1 -- start start 2024-06-29T14:22:50.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22d68ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22d00b3ed0 0x7f22d00c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22d68ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 0x7f22d0153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22d68ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22d015ee10 0x7f22d01f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22d68ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f22d0160240 con 0x7f22d00d23c0 2024-06-29T14:22:50.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22d68ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22d01600c0 con 0x7f22d00b3ed0 2024-06-29T14:22:50.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22d68ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f22d01603c0 con 0x7f22d015ee10 2024-06-29T14:22:50.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 0x7f22d0153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 0x7f22d0153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43078/0 (socket says 172.21.15.59:43078) 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 -- 172.21.15.59:0/706574189 learned_addr learned my addr 172.21.15.59:0/706574189 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cffff640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22d00b3ed0 0x7f22d00c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22d4e49640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22d015ee10 0x7f22d01f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 -- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22d015ee10 msgr2=0x7f22d01f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22d015ee10 0x7f22d01f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 -- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22d00b3ed0 msgr2=0x7f22d00c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22d00b3ed0 0x7f22d00c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 -- 172.21.15.59:0/706574189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d01e7310 con 0x7f22d00d23c0 2024-06-29T14:22:50.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22cf7fe640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 0x7f22d0153a60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f22c400d740 tx=0x7f22c400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.835+0000 7f22ccff9640 1 -- 172.21.15.59:0/706574189 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4017070 con 0x7f22d00d23c0 2024-06-29T14:22:50.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.836+0000 7f22ccff9640 1 -- 172.21.15.59:0/706574189 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22c4009cc0 con 0x7f22d00d23c0 2024-06-29T14:22:50.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.836+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d01e75f0 con 0x7f22d00d23c0 2024-06-29T14:22:50.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.836+0000 7f22ccff9640 1 -- 172.21.15.59:0/706574189 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c40106c0 con 0x7f22d00d23c0 2024-06-29T14:22:50.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.836+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d01e7bb0 con 0x7f22d00d23c0 2024-06-29T14:22:50.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.837+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d006d520 con 0x7f22d00d23c0 2024-06-29T14:22:50.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.837+0000 7f22ccff9640 1 -- 172.21.15.59:0/706574189 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f22c401a070 con 0x7f22d00d23c0 2024-06-29T14:22:50.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.838+0000 7f22ccff9640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22bc0762d0 0x7f22bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.838+0000 7f22ccff9640 1 -- 172.21.15.59:0/706574189 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f22c4097b90 con 0x7f22d00d23c0 2024-06-29T14:22:50.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.838+0000 7f22cffff640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22bc0762d0 0x7f22bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.839+0000 7f22cffff640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22bc0762d0 0x7f22bc0787e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f22c0034000 tx=0x7f22c0004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:50.842+0000 7f22ccff9640 1 -- 172.21.15.59:0/706574189 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f22d006d520 con 0x7f22d00d23c0 2024-06-29T14:22:51.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.038+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22d015e8d0 con 0x7f22d00d23c0 2024-06-29T14:22:51.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.039+0000 7f22ccff9640 1 -- 172.21.15.59:0/706574189 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7f22c4060a10 con 0x7f22d00d23c0 2024-06-29T14:22:51.038 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22bc0762d0 msgr2=0x7f22bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22bc0762d0 0x7f22bc0787e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f22c0034000 tx=0x7f22c0004730 comp rx=0 tx=0).stop 2024-06-29T14:22:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 msgr2=0x7f22d0153a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 0x7f22d0153a60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f22c400d740 tx=0x7f22c400dc00 comp rx=0 tx=0).stop 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 shutdown_connections 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22bc0762d0 0x7f22bc0787e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22d015ee10 0x7f22d01f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22d00d23c0 0x7f22d0153a60 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 --2- 172.21.15.59:0/706574189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22d00b3ed0 0x7f22d00c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 >> 172.21.15.59:0/706574189 conn(0x7f22d00d3ee0 msgr2=0x7f22d00ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 shutdown_connections 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.041+0000 7f22d68ac640 1 -- 172.21.15.59:0/706574189 wait complete. 2024-06-29T14:22:51.052 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:51.053 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:51.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.175+0000 7f64d2b32640 1 -- 172.21.15.59:0/1252462467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64bc003ac0 msgr2=0x7f64bc003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.175+0000 7f64d2b32640 1 --2- 172.21.15.59:0/1252462467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64bc003ac0 0x7f64bc003f30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f64cc0d2e40 tx=0x7f64bc005830 comp rx=0 tx=0).stop 2024-06-29T14:22:51.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.175+0000 7f64d2b32640 1 -- 172.21.15.59:0/1252462467 shutdown_connections 2024-06-29T14:22:51.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.175+0000 7f64d2b32640 1 --2- 172.21.15.59:0/1252462467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64bc003ac0 0x7f64bc003f30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.175+0000 7f64d2b32640 1 -- 172.21.15.59:0/1252462467 >> 172.21.15.59:0/1252462467 conn(0x7f64cc0d3ee0 msgr2=0x7f64cc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.175+0000 7f64d2b32640 1 -- 172.21.15.59:0/1252462467 shutdown_connections 2024-06-29T14:22:51.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.175+0000 7f64d2b32640 1 -- 172.21.15.59:0/1252462467 wait complete. 2024-06-29T14:22:51.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.177+0000 7f64d2b32640 1 Processor -- start 2024-06-29T14:22:51.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.177+0000 7f64d2b32640 1 -- start start 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.178+0000 7f64d2b32640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc003ac0 0x7f64cc0ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.178+0000 7f64d2b32640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64cc1f82f0 0x7f64cc1f8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.178+0000 7f64d2b32640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc151e00 0x7f64cc152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.178+0000 7f64d2b32640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc15ec60 con 0x7f64cc1f82f0 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.178+0000 7f64d2b32640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc15eae0 con 0x7f64cc151e00 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.178+0000 7f64d2b32640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f64cc0b3990 con 0x7f64bc003ac0 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64cc1f82f0 0x7f64cc1f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64cc1f82f0 0x7f64cc1f8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43094/0 (socket says 172.21.15.59:43094) 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 -- 172.21.15.59:0/1343216483 learned_addr learned my addr 172.21.15.59:0/1343216483 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64d10cf640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc151e00 0x7f64cc152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64d08ce640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc003ac0 0x7f64cc0ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 -- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc003ac0 msgr2=0x7f64cc0ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc003ac0 0x7f64cc0ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 -- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc151e00 msgr2=0x7f64cc152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc151e00 0x7f64cc152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 -- 172.21.15.59:0/1343216483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64cc0ccf50 con 0x7f64cc1f82f0 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64d10cf640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc151e00 0x7f64cc152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64cbfff640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64cc1f82f0 0x7f64cc1f8700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f64b8011780 tx=0x7f64b8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64c97fa640 1 -- 172.21.15.59:0/1343216483 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b800fd50 con 0x7f64cc1f82f0 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64c97fa640 1 -- 172.21.15.59:0/1343216483 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64b8011e90 con 0x7f64cc1f82f0 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64cc0cd230 con 0x7f64cc1f82f0 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64c97fa640 1 -- 172.21.15.59:0/1343216483 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b8005280 con 0x7f64cc1f82f0 2024-06-29T14:22:51.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.179+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64cc0cd7f0 con 0x7f64cc1f82f0 2024-06-29T14:22:51.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.180+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64cc0cccc0 con 0x7f64cc1f82f0 2024-06-29T14:22:51.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.181+0000 7f64c97fa640 1 -- 172.21.15.59:0/1343216483 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f64b8005420 con 0x7f64cc1f82f0 2024-06-29T14:22:51.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.182+0000 7f64c97fa640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f64b00762d0 0x7f64b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.182+0000 7f64c97fa640 1 -- 172.21.15.59:0/1343216483 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f64b8098770 con 0x7f64cc1f82f0 2024-06-29T14:22:51.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.182+0000 7f64d08ce640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f64b00762d0 0x7f64b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.183+0000 7f64d08ce640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f64b00762d0 0x7f64b00787e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f64c0033000 tx=0x7f64c0000fe0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.186+0000 7f64c97fa640 1 -- 172.21.15.59:0/1343216483 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f64b8005710 con 0x7f64cc1f82f0 2024-06-29T14:22:51.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.378+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f64cc0b3ef0 con 0x7f64cc1f82f0 2024-06-29T14:22:51.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.378+0000 7f64c97fa640 1 -- 172.21.15.59:0/1343216483 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f64b8061c30 con 0x7f64cc1f82f0 2024-06-29T14:22:51.377 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:51.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f64b00762d0 msgr2=0x7f64b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f64b00762d0 0x7f64b00787e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f64c0033000 tx=0x7f64c0000fe0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64cc1f82f0 msgr2=0x7f64cc1f8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64cc1f82f0 0x7f64cc1f8700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f64b8011780 tx=0x7f64b8011c40 comp rx=0 tx=0).stop 2024-06-29T14:22:51.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 shutdown_connections 2024-06-29T14:22:51.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f64b00762d0 0x7f64b00787e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64cc151e00 0x7f64cc152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64cc1f82f0 0x7f64cc1f8700 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 --2- 172.21.15.59:0/1343216483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64bc003ac0 0x7f64cc0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 >> 172.21.15.59:0/1343216483 conn(0x7f64cc0d3ee0 msgr2=0x7f64cc158360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 shutdown_connections 2024-06-29T14:22:51.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.380+0000 7f64d2b32640 1 -- 172.21.15.59:0/1343216483 wait complete. 2024-06-29T14:22:51.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.513+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3760437597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 msgr2=0x7fd4f0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.513+0000 7fd50f9bd640 1 --2- 172.21.15.59:0/3760437597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 0x7fd4f0005150 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd508070240 tx=0x7fd4f0005830 comp rx=0 tx=0).stop 2024-06-29T14:22:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.513+0000 7fd50df5a640 1 -- 172.21.15.59:0/3760437597 reap_dead start 2024-06-29T14:22:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.513+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3760437597 shutdown_connections 2024-06-29T14:22:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.513+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3760437597 >> 172.21.15.59:0/3760437597 conn(0x7fd5080b24c0 msgr2=0x7fd5080cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.513+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3760437597 shutdown_connections 2024-06-29T14:22:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.513+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3760437597 wait complete. 2024-06-29T14:22:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.515+0000 7fd50f9bd640 1 Processor -- start 2024-06-29T14:22:51.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.515+0000 7fd50f9bd640 1 -- start start 2024-06-29T14:22:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50f9bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 0x7fd50813c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50f9bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5080b4060 0x7fd50813b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50f9bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd50813bac0 0x7fd5081f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50f9bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd50815f5d0 con 0x7fd4f0004d40 2024-06-29T14:22:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50f9bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd50815f450 con 0x7fd50813bac0 2024-06-29T14:22:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50f9bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd50815f750 con 0x7fd5080b4060 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50d759640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 0x7fd50813c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50d759640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 0x7fd50813c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43118/0 (socket says 172.21.15.59:43118) 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50d759640 1 -- 172.21.15.59:0/3481799583 learned_addr learned my addr 172.21.15.59:0/3481799583 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50cf58640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5080b4060 0x7fd50813b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.516+0000 7fd50df5a640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd50813bac0 0x7fd5081f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd50d759640 1 -- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5080b4060 msgr2=0x7fd50813b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd50d759640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5080b4060 0x7fd50813b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd50d759640 1 -- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd50813bac0 msgr2=0x7fd5081f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd50d759640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd50813bac0 0x7fd5081f18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd50d759640 1 -- 172.21.15.59:0/3481799583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5080b2fc0 con 0x7fd4f0004d40 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd50d759640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 0x7fd50813c690 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd500011ab0 tx=0x7fd500011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd505ffb640 1 -- 172.21.15.59:0/3481799583 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50001e070 con 0x7fd4f0004d40 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd505ffb640 1 -- 172.21.15.59:0/3481799583 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd500004460 con 0x7fd4f0004d40 2024-06-29T14:22:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5080b32a0 con 0x7fd4f0004d40 2024-06-29T14:22:51.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd505ffb640 1 -- 172.21.15.59:0/3481799583 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50000fc00 con 0x7fd4f0004d40 2024-06-29T14:22:51.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.517+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5080b3860 con 0x7fd4f0004d40 2024-06-29T14:22:51.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.518+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd508073270 con 0x7fd4f0004d40 2024-06-29T14:22:51.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.519+0000 7fd505ffb640 1 -- 172.21.15.59:0/3481799583 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd500021070 con 0x7fd4f0004d40 2024-06-29T14:22:51.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.519+0000 7fd505ffb640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4f40762d0 0x7fd4f40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.519+0000 7fd505ffb640 1 -- 172.21.15.59:0/3481799583 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd5000984a0 con 0x7fd4f0004d40 2024-06-29T14:22:51.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.520+0000 7fd50cf58640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4f40762d0 0x7fd4f40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.520+0000 7fd50cf58640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4f40762d0 0x7fd4f40787e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd5080cbc80 tx=0x7fd4fc0067a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.524+0000 7fd505ffb640 1 -- 172.21.15.59:0/3481799583 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd5000613b0 con 0x7fd4f0004d40 2024-06-29T14:22:51.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.712+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5081440a0 con 0x7fd4f0004d40 2024-06-29T14:22:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.713+0000 7fd505ffb640 1 -- 172.21.15.59:0/3481799583 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7fd5000656b0 con 0x7fd4f0004d40 2024-06-29T14:22:51.712 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:51.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4f40762d0 msgr2=0x7fd4f40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4f40762d0 0x7fd4f40787e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd5080cbc80 tx=0x7fd4fc0067a0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 msgr2=0x7fd50813c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 0x7fd50813c690 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd500011ab0 tx=0x7fd500011eb0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 shutdown_connections 2024-06-29T14:22:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd4f40762d0 0x7fd4f40787e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd50813bac0 0x7fd5081f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5080b4060 0x7fd50813b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 --2- 172.21.15.59:0/3481799583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4f0004d40 0x7fd50813c690 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 >> 172.21.15.59:0/3481799583 conn(0x7fd5080b24c0 msgr2=0x7fd50815dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 shutdown_connections 2024-06-29T14:22:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.715+0000 7fd50f9bd640 1 -- 172.21.15.59:0/3481799583 wait complete. 2024-06-29T14:22:51.725 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:51.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.848+0000 7fd6530ac640 1 -- 172.21.15.59:0/2913456050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd638004040 msgr2=0x7fd638004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.848+0000 7fd6530ac640 1 --2- 172.21.15.59:0/2913456050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd638004040 0x7fd638004800 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0b4610 tx=0x7fd638005830 comp rx=0 tx=0).stop 2024-06-29T14:22:51.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.848+0000 7fd651649640 1 -- 172.21.15.59:0/2913456050 reap_dead start 2024-06-29T14:22:51.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.848+0000 7fd6530ac640 1 -- 172.21.15.59:0/2913456050 shutdown_connections 2024-06-29T14:22:51.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.848+0000 7fd6530ac640 1 -- 172.21.15.59:0/2913456050 >> 172.21.15.59:0/2913456050 conn(0x7fd64c0b2ab0 msgr2=0x7fd64c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.848+0000 7fd6530ac640 1 -- 172.21.15.59:0/2913456050 shutdown_connections 2024-06-29T14:22:51.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.848+0000 7fd6530ac640 1 -- 172.21.15.59:0/2913456050 wait complete. 2024-06-29T14:22:51.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.850+0000 7fd6530ac640 1 Processor -- start 2024-06-29T14:22:51.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.850+0000 7fd6530ac640 1 -- start start 2024-06-29T14:22:51.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd6530ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd638004040 0x7fd64c1f14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd6530ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c15a630 0x7fd64c15b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd6530ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15ba60 0x7fd64c1f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd6530ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c0cf3a0 con 0x7fd64c15a630 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd6530ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c0cf220 con 0x7fd638004040 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd6530ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd64c0cf520 con 0x7fd64c15ba60 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd64bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c15a630 0x7fd64c15b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd64bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c15a630 0x7fd64c15b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43130/0 (socket says 172.21.15.59:43130) 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.851+0000 7fd64bfff640 1 -- 172.21.15.59:0/58495323 learned_addr learned my addr 172.21.15.59:0/58495323 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd650e48640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd638004040 0x7fd64c1f14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd651649640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15ba60 0x7fd64c1f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd64bfff640 1 -- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15ba60 msgr2=0x7fd64c1f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd64bfff640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15ba60 0x7fd64c1f3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd64bfff640 1 -- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd638004040 msgr2=0x7fd64c1f14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd64bfff640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd638004040 0x7fd64c1f14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd64bfff640 1 -- 172.21.15.59:0/58495323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c15be70 con 0x7fd64c15a630 2024-06-29T14:22:51.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd64bfff640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c15a630 0x7fd64c15b520 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd63400c700 tx=0x7fd634004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd6497fa640 1 -- 172.21.15.59:0/58495323 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63402b7c0 con 0x7fd64c15a630 2024-06-29T14:22:51.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd6497fa640 1 -- 172.21.15.59:0/58495323 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd63402b960 con 0x7fd64c15a630 2024-06-29T14:22:51.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64c0a9780 con 0x7fd64c15a630 2024-06-29T14:22:51.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd6497fa640 1 -- 172.21.15.59:0/58495323 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63402bd70 con 0x7fd64c15a630 2024-06-29T14:22:51.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.852+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64c0a9d40 con 0x7fd64c15a630 2024-06-29T14:22:51.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.853+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64c143840 con 0x7fd64c15a630 2024-06-29T14:22:51.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.854+0000 7fd6497fa640 1 -- 172.21.15.59:0/58495323 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd6340060c0 con 0x7fd64c15a630 2024-06-29T14:22:51.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.854+0000 7fd6497fa640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd62c0762d0 0x7fd62c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.854+0000 7fd6497fa640 1 -- 172.21.15.59:0/58495323 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd6340b7b70 con 0x7fd64c15a630 2024-06-29T14:22:51.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.855+0000 7fd650e48640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd62c0762d0 0x7fd62c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.855+0000 7fd650e48640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd62c0762d0 0x7fd62c0787e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009cd0 tx=0x7fd63c005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:51.859+0000 7fd6497fa640 1 -- 172.21.15.59:0/58495323 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd64c15be70 con 0x7fd64c15a630 2024-06-29T14:22:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.051+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd64c0cfcf0 con 0x7fd64c15a630 2024-06-29T14:22:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.052+0000 7fd6497fa640 1 -- 172.21.15.59:0/58495323 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fd634080a30 con 0x7fd64c15a630 2024-06-29T14:22:52.051 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd62c0762d0 msgr2=0x7fd62c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd62c0762d0 0x7fd62c0787e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009cd0 tx=0x7fd63c005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c15a630 msgr2=0x7fd64c15b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c15a630 0x7fd64c15b520 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd63400c700 tx=0x7fd634004bd0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 shutdown_connections 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd62c0762d0 0x7fd62c0787e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15ba60 0x7fd64c1f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c15a630 0x7fd64c15b520 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 --2- 172.21.15.59:0/58495323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd638004040 0x7fd64c1f14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 >> 172.21.15.59:0/58495323 conn(0x7fd64c0b2ab0 msgr2=0x7fd64c0b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 shutdown_connections 2024-06-29T14:22:52.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.054+0000 7fd6530ac640 1 -- 172.21.15.59:0/58495323 wait complete. 2024-06-29T14:22:52.064 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 -- 172.21.15.59:0/3184401743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 msgr2=0x7fc62c15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 --2- 172.21.15.59:0/3184401743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 0x7fc62c15f9b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0b4a80 tx=0x7fc62402a560 comp rx=0 tx=0).stop 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 -- 172.21.15.59:0/3184401743 shutdown_connections 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 --2- 172.21.15.59:0/3184401743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 0x7fc62c15f9b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 --2- 172.21.15.59:0/3184401743 >> v2:172.21.15.153:3300/0 conn(0x7fc62c0b3ed0 0x7fc62c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 --2- 172.21.15.59:0/3184401743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc62c0d23c0 0x7fc62c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 -- 172.21.15.59:0/3184401743 >> 172.21.15.59:0/3184401743 conn(0x7fc62c0d3ee0 msgr2=0x7fc62c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 -- 172.21.15.59:0/3184401743 shutdown_connections 2024-06-29T14:22:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.185+0000 7fc632e09640 1 -- 172.21.15.59:0/3184401743 wait complete. 2024-06-29T14:22:52.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.187+0000 7fc632e09640 1 Processor -- start 2024-06-29T14:22:52.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.187+0000 7fc632e09640 1 -- start start 2024-06-29T14:22:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc632e09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc62c0b3ed0 0x7fc62c1e8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc632e09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc62c0d23c0 0x7fc62c159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc632e09640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 0x7fc62c1f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc632e09640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c160070 con 0x7fc62c15ee10 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc632e09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c15fef0 con 0x7fc62c0d23c0 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc632e09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc62c1601f0 con 0x7fc62c0b3ed0 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc6313a6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 0x7fc62c1f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc6313a6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 0x7fc62c1f8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43150/0 (socket says 172.21.15.59:43150) 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc6313a6640 1 -- 172.21.15.59:0/3187528150 learned_addr learned my addr 172.21.15.59:0/3187528150 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc62bfff640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc62c0d23c0 0x7fc62c159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc630ba5640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc62c0b3ed0 0x7fc62c1e8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc6313a6640 1 -- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc62c0b3ed0 msgr2=0x7fc62c1e8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc6313a6640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc62c0b3ed0 0x7fc62c1e8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc6313a6640 1 -- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc62c0d23c0 msgr2=0x7fc62c159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc6313a6640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc62c0d23c0 0x7fc62c159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.189+0000 7fc6313a6640 1 -- 172.21.15.59:0/3187528150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62c1e7f40 con 0x7fc62c15ee10 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.190+0000 7fc6313a6640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 0x7fc62c1f8b00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc624034040 tx=0x7fc624003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.190+0000 7fc6297fa640 1 -- 172.21.15.59:0/3187528150 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62403e030 con 0x7fc62c15ee10 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.190+0000 7fc6297fa640 1 -- 172.21.15.59:0/3187528150 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc624003e40 con 0x7fc62c15ee10 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.190+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62c1e8220 con 0x7fc62c15ee10 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.190+0000 7fc6297fa640 1 -- 172.21.15.59:0/3187528150 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc624035040 con 0x7fc62c15ee10 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.190+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62c1e87e0 con 0x7fc62c15ee10 2024-06-29T14:22:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.191+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc62c0b3990 con 0x7fc62c15ee10 2024-06-29T14:22:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.192+0000 7fc6297fa640 1 -- 172.21.15.59:0/3187528150 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc624041030 con 0x7fc62c15ee10 2024-06-29T14:22:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.192+0000 7fc6297fa640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc60c0762d0 0x7fc60c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.192+0000 7fc6297fa640 1 -- 172.21.15.59:0/3187528150 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fc6240b8240 con 0x7fc62c15ee10 2024-06-29T14:22:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.192+0000 7fc630ba5640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc60c0762d0 0x7fc60c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.193+0000 7fc630ba5640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc60c0762d0 0x7fc60c0787e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0d2e40 tx=0x7fc61400b2e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.197+0000 7fc6297fa640 1 -- 172.21.15.59:0/3187528150 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc624081150 con 0x7fc62c15ee10 2024-06-29T14:22:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.385+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc62c0d2950 con 0x7fc62c15ee10 2024-06-29T14:22:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.385+0000 7fc6297fa640 1 -- 172.21.15.59:0/3187528150 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fc624085450 con 0x7fc62c15ee10 2024-06-29T14:22:52.384 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:52.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc60c0762d0 msgr2=0x7fc60c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc60c0762d0 0x7fc60c0787e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0d2e40 tx=0x7fc61400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 msgr2=0x7fc62c1f8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 0x7fc62c1f8b00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc624034040 tx=0x7fc624003910 comp rx=0 tx=0).stop 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 shutdown_connections 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc60c0762d0 0x7fc60c0787e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc62c15ee10 0x7fc62c1f8b00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc62c0d23c0 0x7fc62c159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 --2- 172.21.15.59:0/3187528150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc62c0b3ed0 0x7fc62c1e8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 >> 172.21.15.59:0/3187528150 conn(0x7fc62c0d3ee0 msgr2=0x7fc62c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.387+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 shutdown_connections 2024-06-29T14:22:52.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.388+0000 7fc632e09640 1 -- 172.21.15.59:0/3187528150 wait complete. 2024-06-29T14:22:52.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:22:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.517+0000 7f5df3311640 1 -- 172.21.15.59:0/3113772741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec162770 msgr2=0x7f5dec0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.517+0000 7f5df3311640 1 --2- 172.21.15.59:0/3113772741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec162770 0x7f5dec0ce550 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0d3710 tx=0x7f5de802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.517+0000 7f5df3311640 1 -- 172.21.15.59:0/3113772741 shutdown_connections 2024-06-29T14:22:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.517+0000 7f5df3311640 1 --2- 172.21.15.59:0/3113772741 >> v2:172.21.15.153:3300/0 conn(0x7f5dec162190 0x7f5dec0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.517+0000 7f5df3311640 1 --2- 172.21.15.59:0/3113772741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec162770 0x7f5dec0ce550 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.517+0000 7f5df3311640 1 -- 172.21.15.59:0/3113772741 >> 172.21.15.59:0/3113772741 conn(0x7f5dec0d45b0 msgr2=0x7f5dec0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.517+0000 7f5df3311640 1 -- 172.21.15.59:0/3113772741 shutdown_connections 2024-06-29T14:22:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.517+0000 7f5df3311640 1 -- 172.21.15.59:0/3113772741 wait complete. 2024-06-29T14:22:52.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.519+0000 7f5df3311640 1 Processor -- start 2024-06-29T14:22:52.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.519+0000 7f5df3311640 1 -- start start 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df3311640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dec15e0a0 0x7f5dec1f5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df3311640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dec162770 0x7f5dec154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df3311640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec0cab30 0x7f5dec0cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df3311640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec0b4270 con 0x7f5dec0cab30 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df3311640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec0b40f0 con 0x7f5dec162770 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df3311640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5dec0b43f0 con 0x7f5dec15e0a0 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df18ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec0cab30 0x7f5dec0cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df18ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec0cab30 0x7f5dec0cc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43172/0 (socket says 172.21.15.59:43172) 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.520+0000 7f5df18ae640 1 -- 172.21.15.59:0/1871181787 learned_addr learned my addr 172.21.15.59:0/1871181787 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df08ac640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dec162770 0x7f5dec154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df10ad640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dec15e0a0 0x7f5dec1f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df18ae640 1 -- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dec15e0a0 msgr2=0x7f5dec1f5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df18ae640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dec15e0a0 0x7f5dec1f5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df18ae640 1 -- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dec162770 msgr2=0x7f5dec154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df18ae640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dec162770 0x7f5dec154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df18ae640 1 -- 172.21.15.59:0/1871181787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec0cb340 con 0x7f5dec0cab30 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df08ac640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dec162770 0x7f5dec154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df10ad640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dec15e0a0 0x7f5dec1f5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df18ae640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec0cab30 0x7f5dec0cc290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5de400ccb0 tx=0x7f5de400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5de1ffb640 1 -- 172.21.15.59:0/1871181787 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de400f450 con 0x7f5dec0cab30 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5de1ffb640 1 -- 172.21.15.59:0/1871181787 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5de400f700 con 0x7f5dec0cab30 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5de1ffb640 1 -- 172.21.15.59:0/1871181787 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de400fb50 con 0x7f5dec0cab30 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dec0cb620 con 0x7f5dec0cab30 2024-06-29T14:22:52.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.521+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dec0cbbe0 con 0x7f5dec0cab30 2024-06-29T14:22:52.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.522+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dec0cb0b0 con 0x7f5dec0cab30 2024-06-29T14:22:52.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.523+0000 7f5de1ffb640 1 -- 172.21.15.59:0/1871181787 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5de400fcf0 con 0x7f5dec0cab30 2024-06-29T14:22:52.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.523+0000 7f5de1ffb640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5dd00762d0 0x7f5dd00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.524+0000 7f5de1ffb640 1 -- 172.21.15.59:0/1871181787 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f5de40982b0 con 0x7f5dec0cab30 2024-06-29T14:22:52.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.524+0000 7f5df10ad640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5dd00762d0 0x7f5dd00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.525+0000 7f5df10ad640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5dd00762d0 0x7f5dd00787e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5de8034000 tx=0x7f5de8004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.528+0000 7f5de1ffb640 1 -- 172.21.15.59:0/1871181787 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5de4014030 con 0x7f5dec0cab30 2024-06-29T14:22:52.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.731+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5dec0b4910 con 0x7f5dec0cab30 2024-06-29T14:22:52.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.731+0000 7f5de1ffb640 1 -- 172.21.15.59:0/1871181787 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7f5de4061770 con 0x7f5dec0cab30 2024-06-29T14:22:52.730 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-06-29T14:22:52.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5dd00762d0 msgr2=0x7f5dd00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5dd00762d0 0x7f5dd00787e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5de8034000 tx=0x7f5de8004730 comp rx=0 tx=0).stop 2024-06-29T14:22:52.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec0cab30 msgr2=0x7f5dec0cc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec0cab30 0x7f5dec0cc290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5de400ccb0 tx=0x7f5de400a620 comp rx=0 tx=0).stop 2024-06-29T14:22:52.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 shutdown_connections 2024-06-29T14:22:52.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5dd00762d0 0x7f5dd00787e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dec0cab30 0x7f5dec0cc290 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dec162770 0x7f5dec154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 --2- 172.21.15.59:0/1871181787 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dec15e0a0 0x7f5dec1f5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 >> 172.21.15.59:0/1871181787 conn(0x7f5dec0d45b0 msgr2=0x7f5dec0b77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 shutdown_connections 2024-06-29T14:22:52.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.733+0000 7f5df3311640 1 -- 172.21.15.59:0/1871181787 wait complete. 2024-06-29T14:22:52.743 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-06-29T14:22:52.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:52.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-06-29T14:22:52.775 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T14:22:52.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:52.776 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-06-29T14:22:52.834 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:22:52.834 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:52.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:22:52.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:22:52.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.954+0000 7fcb5a68f640 1 -- 172.21.15.59:0/923700568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb5406ebe0 msgr2=0x7fcb540cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.954+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/923700568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb5406ebe0 0x7fcb540cef90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcb54072c40 tx=0x7fcb4c02a9e0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.954+0000 7fcb5a68f640 1 -- 172.21.15.59:0/923700568 shutdown_connections 2024-06-29T14:22:52.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.954+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/923700568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb5406ebe0 0x7fcb540cef90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.954+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/923700568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb540d0130 0x7fcb54143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.954+0000 7fcb5a68f640 1 -- 172.21.15.59:0/923700568 >> 172.21.15.59:0/923700568 conn(0x7fcb540b2ab0 msgr2=0x7fcb540b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.955+0000 7fcb5a68f640 1 -- 172.21.15.59:0/923700568 shutdown_connections 2024-06-29T14:22:52.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.955+0000 7fcb5a68f640 1 -- 172.21.15.59:0/923700568 wait complete. 2024-06-29T14:22:52.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.956+0000 7fcb5a68f640 1 Processor -- start 2024-06-29T14:22:52.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.957+0000 7fcb5a68f640 1 -- start start 2024-06-29T14:22:52.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb5a68f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb5406ebe0 0x7fcb541f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb5a68f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb540b4ee0 0x7fcb54157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb5a68f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb540d0130 0x7fcb5415d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb5a68f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcb540cf860 con 0x7fcb540d0130 2024-06-29T14:22:52.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb5a68f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb540cf6e0 con 0x7fcb5406ebe0 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb5a68f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcb540cf9e0 con 0x7fcb540b4ee0 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb58c2c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb540d0130 0x7fcb5415d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb58c2c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb540d0130 0x7fcb5415d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39860/0 (socket says 172.21.15.59:39860) 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb58c2c640 1 -- 172.21.15.59:0/3162182584 learned_addr learned my addr 172.21.15.59:0/3162182584 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb53fff640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb5406ebe0 0x7fcb541f7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb537fe640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb540b4ee0 0x7fcb54157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb58c2c640 1 -- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb540b4ee0 msgr2=0x7fcb54157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb58c2c640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb540b4ee0 0x7fcb54157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb58c2c640 1 -- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb5406ebe0 msgr2=0x7fcb541f7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb58c2c640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb5406ebe0 0x7fcb541f7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.958+0000 7fcb58c2c640 1 -- 172.21.15.59:0/3162182584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5415c4d0 con 0x7fcb540d0130 2024-06-29T14:22:52.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.959+0000 7fcb58c2c640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb540d0130 0x7fcb5415d570 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009d60 tx=0x7fcb4c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.959+0000 7fcb50ff9640 1 -- 172.21.15.59:0/3162182584 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c033050 con 0x7fcb540d0130 2024-06-29T14:22:52.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.959+0000 7fcb50ff9640 1 -- 172.21.15.59:0/3162182584 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb4c004450 con 0x7fcb540d0130 2024-06-29T14:22:52.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.959+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5415c7b0 con 0x7fcb540d0130 2024-06-29T14:22:52.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.959+0000 7fcb50ff9640 1 -- 172.21.15.59:0/3162182584 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c038450 con 0x7fcb540d0130 2024-06-29T14:22:52.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.959+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5415cd70 con 0x7fcb540d0130 2024-06-29T14:22:52.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.960+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb5415c230 con 0x7fcb540d0130 2024-06-29T14:22:52.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.960+0000 7fcb50ff9640 1 -- 172.21.15.59:0/3162182584 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcb4c036070 con 0x7fcb540d0130 2024-06-29T14:22:52.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.961+0000 7fcb50ff9640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb340762d0 0x7fcb340787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.961+0000 7fcb50ff9640 1 -- 172.21.15.59:0/3162182584 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fcb4c0b8d90 con 0x7fcb540d0130 2024-06-29T14:22:52.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.961+0000 7fcb53fff640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb340762d0 0x7fcb340787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.962+0000 7fcb53fff640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb340762d0 0x7fcb340787e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcb44004630 tx=0x7fcb4400b2e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:52.965+0000 7fcb50ff9640 1 -- 172.21.15.59:0/3162182584 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcb5415c230 con 0x7fcb540d0130 2024-06-29T14:22:53.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.193+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb540ae9e0 con 0x7fcb540d0130 2024-06-29T14:22:53.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.194+0000 7fcb50ff9640 1 -- 172.21.15.59:0/3162182584 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5222 (secure 0 0 0) 0x7fcb4c081c10 con 0x7fcb540d0130 2024-06-29T14:22:53.193 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:53.193 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":9,"btime":"2024-06-29T14:22:12:518854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:22:11.909450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:53.193 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 9 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb340762d0 msgr2=0x7fcb340787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb340762d0 0x7fcb340787e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcb44004630 tx=0x7fcb4400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb540d0130 msgr2=0x7fcb5415d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb540d0130 0x7fcb5415d570 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009d60 tx=0x7fcb4c003f10 comp rx=0 tx=0).stop 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 shutdown_connections 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcb340762d0 0x7fcb340787e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb540d0130 0x7fcb5415d570 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb540b4ee0 0x7fcb54157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 --2- 172.21.15.59:0/3162182584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb5406ebe0 0x7fcb541f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 >> 172.21.15.59:0/3162182584 conn(0x7fcb540b2ab0 msgr2=0x7fcb5406ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 shutdown_connections 2024-06-29T14:22:53.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.196+0000 7fcb5a68f640 1 -- 172.21.15.59:0/3162182584 wait complete. 2024-06-29T14:22:53.206 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:53.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.365+0000 7f86dbd2e640 1 -- 172.21.15.59:0/3944622200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d4162210 msgr2=0x7f86d40b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.365+0000 7f86dbd2e640 1 --2- 172.21.15.59:0/3944622200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d4162210 0x7f86d40b1d10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f86cc009830 tx=0x7f86cc02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.365+0000 7f86dbd2e640 1 -- 172.21.15.59:0/3944622200 shutdown_connections 2024-06-29T14:22:53.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.365+0000 7f86dbd2e640 1 --2- 172.21.15.59:0/3944622200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d4162210 0x7f86d40b1d10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.365+0000 7f86dbd2e640 1 -- 172.21.15.59:0/3944622200 >> 172.21.15.59:0/3944622200 conn(0x7f86d406c950 msgr2=0x7f86d4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.365+0000 7f86dbd2e640 1 -- 172.21.15.59:0/3944622200 shutdown_connections 2024-06-29T14:22:53.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.365+0000 7f86dbd2e640 1 -- 172.21.15.59:0/3944622200 wait complete. 2024-06-29T14:22:53.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.367+0000 7f86dbd2e640 1 Processor -- start 2024-06-29T14:22:53.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.367+0000 7f86dbd2e640 1 -- start start 2024-06-29T14:22:53.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86dbd2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d406f8a0 0x7f86d41f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86dbd2e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86d4147370 0x7f86d415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86dbd2e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86d40b6340 0x7f86d40b6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86dbd2e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f86d4169340 con 0x7f86d406f8a0 2024-06-29T14:22:53.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86dbd2e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86d41691c0 con 0x7f86d40b6340 2024-06-29T14:22:53.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86dbd2e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f86d41694c0 con 0x7f86d4147370 2024-06-29T14:22:53.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86d9aca640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d406f8a0 0x7f86d41f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86d9aca640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d406f8a0 0x7f86d41f4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39878/0 (socket says 172.21.15.59:39878) 2024-06-29T14:22:53.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86d9aca640 1 -- 172.21.15.59:0/218753069 learned_addr learned my addr 172.21.15.59:0/218753069 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:53.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86da2cb640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86d40b6340 0x7f86d40b6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.368+0000 7f86d92c9640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86d4147370 0x7f86d415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d9aca640 1 -- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86d4147370 msgr2=0x7f86d415d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d9aca640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86d4147370 0x7f86d415d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d9aca640 1 -- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86d40b6340 msgr2=0x7f86d40b6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d9aca640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86d40b6340 0x7f86d40b6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d9aca640 1 -- 172.21.15.59:0/218753069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d40b6c90 con 0x7f86d406f8a0 2024-06-29T14:22:53.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d9aca640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d406f8a0 0x7f86d41f4c50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f86c400c7d0 tx=0x7f86c400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:53.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d27fc640 1 -- 172.21.15.59:0/218753069 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4003b70 con 0x7f86d406f8a0 2024-06-29T14:22:53.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d27fc640 1 -- 172.21.15.59:0/218753069 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86c4003d10 con 0x7f86d406f8a0 2024-06-29T14:22:53.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d41404d0 con 0x7f86d406f8a0 2024-06-29T14:22:53.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86d27fc640 1 -- 172.21.15.59:0/218753069 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c400aa50 con 0x7f86d406f8a0 2024-06-29T14:22:53.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.369+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d4140a90 con 0x7f86d406f8a0 2024-06-29T14:22:53.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.370+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d40b1d10 con 0x7f86d406f8a0 2024-06-29T14:22:53.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.371+0000 7f86d27fc640 1 -- 172.21.15.59:0/218753069 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f86c400abf0 con 0x7f86d406f8a0 2024-06-29T14:22:53.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.371+0000 7f86d27fc640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86c00762d0 0x7f86c00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.371+0000 7f86d27fc640 1 -- 172.21.15.59:0/218753069 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f86c4097880 con 0x7f86d406f8a0 2024-06-29T14:22:53.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.371+0000 7f86d92c9640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86c00762d0 0x7f86c00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.372+0000 7f86d92c9640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86c00762d0 0x7f86c00787e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f86d40b6f30 tx=0x7f86c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:53.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.376+0000 7f86d27fc640 1 -- 172.21.15.59:0/218753069 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f86d40b6c90 con 0x7f86d406f8a0 2024-06-29T14:22:53.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.566+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86d40cf110 con 0x7f86d406f8a0 2024-06-29T14:22:53.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.567+0000 7f86d27fc640 1 -- 172.21.15.59:0/218753069 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7f86c4060790 con 0x7f86d406f8a0 2024-06-29T14:22:53.567 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:53.567 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:53.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86c00762d0 msgr2=0x7f86c00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86c00762d0 0x7f86c00787e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f86d40b6f30 tx=0x7f86c8009040 comp rx=0 tx=0).stop 2024-06-29T14:22:53.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d406f8a0 msgr2=0x7f86d41f4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d406f8a0 0x7f86d41f4c50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f86c400c7d0 tx=0x7f86c400cc90 comp rx=0 tx=0).stop 2024-06-29T14:22:53.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 shutdown_connections 2024-06-29T14:22:53.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f86c00762d0 0x7f86c00787e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86d40b6340 0x7f86d40b6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86d4147370 0x7f86d415d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 --2- 172.21.15.59:0/218753069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86d406f8a0 0x7f86d41f4c50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 >> 172.21.15.59:0/218753069 conn(0x7f86d406c950 msgr2=0x7f86d406fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 shutdown_connections 2024-06-29T14:22:53.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.569+0000 7f86dbd2e640 1 -- 172.21.15.59:0/218753069 wait complete. 2024-06-29T14:22:53.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:21:52.912626+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:21:53.621207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-29T14:21:57.292065+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-29T14:22:53.581 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.702+0000 7f245a363640 1 -- 172.21.15.59:0/3352488051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 msgr2=0x7f24540b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.702+0000 7f245a363640 1 --2- 172.21.15.59:0/3352488051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 0x7f24540b5b30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f244c005fd0 tx=0x7f244c02aac0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.702+0000 7f245a363640 1 -- 172.21.15.59:0/3352488051 shutdown_connections 2024-06-29T14:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.702+0000 7f245a363640 1 --2- 172.21.15.59:0/3352488051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 0x7f24540b5b30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.702+0000 7f245a363640 1 --2- 172.21.15.59:0/3352488051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24540b2270 0x7f245415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.702+0000 7f245a363640 1 -- 172.21.15.59:0/3352488051 >> 172.21.15.59:0/3352488051 conn(0x7f2454073e10 msgr2=0x7f2454074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.702+0000 7f245a363640 1 -- 172.21.15.59:0/3352488051 shutdown_connections 2024-06-29T14:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.702+0000 7f245a363640 1 -- 172.21.15.59:0/3352488051 wait complete. 2024-06-29T14:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.704+0000 7f245a363640 1 Processor -- start 2024-06-29T14:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.704+0000 7f245a363640 1 -- start start 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f245a363640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24540b2270 0x7f245414d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f245a363640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24540b2fa0 0x7f24540ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f245a363640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 0x7f24540b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f245a363640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f24540b6400 con 0x7f24540d1c30 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f245a363640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f24540b6280 con 0x7f24540b2270 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f245a363640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f24540b6580 con 0x7f24540b2fa0 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 0x7f24540b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 0x7f24540b0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39908/0 (socket says 172.21.15.59:39908) 2024-06-29T14:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 -- 172.21.15.59:0/1950844364 learned_addr learned my addr 172.21.15.59:0/1950844364 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f24537fe640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24540b2fa0 0x7f24540ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2453fff640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24540b2270 0x7f245414d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 -- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24540b2fa0 msgr2=0x7f24540ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24540b2fa0 0x7f24540ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 -- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24540b2270 msgr2=0x7f245414d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24540b2270 0x7f245414d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 -- 172.21.15.59:0/1950844364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24541f4f70 con 0x7f24540d1c30 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.706+0000 7f2458900640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 0x7f24540b0070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f244c009bf0 tx=0x7f244c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.707+0000 7f2450ff9640 1 -- 172.21.15.59:0/1950844364 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c044070 con 0x7f24540d1c30 2024-06-29T14:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.707+0000 7f2450ff9640 1 -- 172.21.15.59:0/1950844364 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f244c003e00 con 0x7f24540d1c30 2024-06-29T14:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.707+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24541f5250 con 0x7f24540d1c30 2024-06-29T14:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.707+0000 7f2450ff9640 1 -- 172.21.15.59:0/1950844364 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c0044b0 con 0x7f24540d1c30 2024-06-29T14:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.707+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f245415d6d0 con 0x7f24540d1c30 2024-06-29T14:22:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.708+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24541f5770 con 0x7f24540d1c30 2024-06-29T14:22:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.708+0000 7f2450ff9640 1 -- 172.21.15.59:0/1950844364 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f244c004650 con 0x7f24540d1c30 2024-06-29T14:22:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.709+0000 7f2450ff9640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2440076300 0x7f2440078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.709+0000 7f2450ff9640 1 -- 172.21.15.59:0/1950844364 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f244c0b81a0 con 0x7f24540d1c30 2024-06-29T14:22:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.709+0000 7f2453fff640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2440076300 0x7f2440078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.710+0000 7f2453fff640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2440076300 0x7f2440078810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2444009b30 tx=0x7f2444005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:53.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.713+0000 7f2450ff9640 1 -- 172.21.15.59:0/1950844364 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f244c032030 con 0x7f24540d1c30 2024-06-29T14:22:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.906+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f24540b6ac0 con 0x7f24540d1c30 2024-06-29T14:22:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.907+0000 7f2450ff9640 1 -- 172.21.15.59:0/1950844364 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f244c0098b0 con 0x7f24540d1c30 2024-06-29T14:22:53.906 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:22:53.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2440076300 msgr2=0x7f2440078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2440076300 0x7f2440078810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2444009b30 tx=0x7f2444005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:53.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 msgr2=0x7f24540b0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 0x7f24540b0070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f244c009bf0 tx=0x7f244c003c10 comp rx=0 tx=0).stop 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 shutdown_connections 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2440076300 0x7f2440078810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24540d1c30 0x7f24540b0070 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24540b2fa0 0x7f24540ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 --2- 172.21.15.59:0/1950844364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24540b2270 0x7f245414d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 >> 172.21.15.59:0/1950844364 conn(0x7f2454073e10 msgr2=0x7f24540b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 shutdown_connections 2024-06-29T14:22:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:53.909+0000 7f245a363640 1 -- 172.21.15.59:0/1950844364 wait complete. 2024-06-29T14:22:53.919 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:54.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.041+0000 7f4e5905e640 1 -- 172.21.15.59:0/1986429175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e540b2fa0 msgr2=0x7f4e540d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.041+0000 7f4e5905e640 1 --2- 172.21.15.59:0/1986429175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e540b2fa0 0x7f4e540d1520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009830 tx=0x7f4e3c02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.041+0000 7f4e5905e640 1 -- 172.21.15.59:0/1986429175 shutdown_connections 2024-06-29T14:22:54.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.041+0000 7f4e5905e640 1 --2- 172.21.15.59:0/1986429175 >> v2:172.21.15.153:3300/0 conn(0x7f4e540d1a60 0x7f4e540b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.041+0000 7f4e5905e640 1 --2- 172.21.15.59:0/1986429175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e540b2fa0 0x7f4e540d1520 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.041+0000 7f4e5905e640 1 -- 172.21.15.59:0/1986429175 >> 172.21.15.59:0/1986429175 conn(0x7f4e54073e10 msgr2=0x7f4e54074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.041+0000 7f4e5905e640 1 -- 172.21.15.59:0/1986429175 shutdown_connections 2024-06-29T14:22:54.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.041+0000 7f4e5905e640 1 -- 172.21.15.59:0/1986429175 wait complete. 2024-06-29T14:22:54.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.043+0000 7f4e5905e640 1 Processor -- start 2024-06-29T14:22:54.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.043+0000 7f4e5905e640 1 -- start start 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5905e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e540b2270 0x7f4e5415d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5905e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e540b2fa0 0x7f4e54152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5905e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e540d1a60 0x7f4e54154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5905e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4e540b6400 con 0x7f4e540d1a60 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5905e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e540b6280 con 0x7f4e540b2fa0 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5905e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4e540b6580 con 0x7f4e540b2270 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e540d1a60 0x7f4e54154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e540d1a60 0x7f4e54154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39920/0 (socket says 172.21.15.59:39920) 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 -- 172.21.15.59:0/2753671898 learned_addr learned my addr 172.21.15.59:0/2753671898 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5259c640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e540b2fa0 0x7f4e54152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e52d9d640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e540b2270 0x7f4e5415d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 -- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e540b2270 msgr2=0x7f4e5415d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e540b2270 0x7f4e5415d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 -- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e540b2fa0 msgr2=0x7f4e54152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e540b2fa0 0x7f4e54152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 -- 172.21.15.59:0/2753671898 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e541e6c80 con 0x7f4e540d1a60 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.045+0000 7f4e5359e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e540d1a60 0x7f4e54154150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00e970 tx=0x7f4e4c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.046+0000 7f4e4b7fe640 1 -- 172.21.15.59:0/2753671898 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c00ccc0 con 0x7f4e540d1a60 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.046+0000 7f4e4b7fe640 1 -- 172.21.15.59:0/2753671898 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e4c010430 con 0x7f4e540d1a60 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.046+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e541e81e0 con 0x7f4e540d1a60 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.046+0000 7f4e4b7fe640 1 -- 172.21.15.59:0/2753671898 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c010880 con 0x7f4e540d1a60 2024-06-29T14:22:54.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.046+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e5415dea0 con 0x7f4e540d1a60 2024-06-29T14:22:54.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.047+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e540b6ae0 con 0x7f4e540d1a60 2024-06-29T14:22:54.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.047+0000 7f4e4b7fe640 1 -- 172.21.15.59:0/2753671898 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4e4c01f020 con 0x7f4e540d1a60 2024-06-29T14:22:54.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.048+0000 7f4e4b7fe640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e340762d0 0x7f4e340787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.048+0000 7f4e4b7fe640 1 -- 172.21.15.59:0/2753671898 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f4e4c098160 con 0x7f4e540d1a60 2024-06-29T14:22:54.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.048+0000 7f4e52d9d640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e340762d0 0x7f4e340787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.049+0000 7f4e52d9d640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e340762d0 0x7f4e340787e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4e540b1fc0 tx=0x7f4e44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.052+0000 7f4e4b7fe640 1 -- 172.21.15.59:0/2753671898 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4e541e6c80 con 0x7f4e540d1a60 2024-06-29T14:22:54.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.242+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4e540b10d0 con 0x7f4e540d1a60 2024-06-29T14:22:54.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.242+0000 7f4e4b7fe640 1 -- 172.21.15.59:0/2753671898 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f4e4c061070 con 0x7f4e540d1a60 2024-06-29T14:22:54.242 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e340762d0 msgr2=0x7f4e340787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e340762d0 0x7f4e340787e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4e540b1fc0 tx=0x7f4e44009040 comp rx=0 tx=0).stop 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e540d1a60 msgr2=0x7f4e54154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e540d1a60 0x7f4e54154150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00e970 tx=0x7f4e4c00ee30 comp rx=0 tx=0).stop 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 shutdown_connections 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4e340762d0 0x7f4e340787e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e540d1a60 0x7f4e54154150 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e540b2fa0 0x7f4e54152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 --2- 172.21.15.59:0/2753671898 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e540b2270 0x7f4e5415d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 >> 172.21.15.59:0/2753671898 conn(0x7f4e54073e10 msgr2=0x7f4e540d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 shutdown_connections 2024-06-29T14:22:54.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.245+0000 7f4e5905e640 1 -- 172.21.15.59:0/2753671898 wait complete. 2024-06-29T14:22:54.258 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:54.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.382+0000 7facb5a47640 1 -- 172.21.15.59:0/2448721773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 msgr2=0x7facb0073500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.382+0000 7facb5a47640 1 --2- 172.21.15.59:0/2448721773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 0x7facb0073500 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7faca4005fd0 tx=0x7faca402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.382+0000 7facb5a47640 1 -- 172.21.15.59:0/2448721773 shutdown_connections 2024-06-29T14:22:54.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.382+0000 7facb5a47640 1 --2- 172.21.15.59:0/2448721773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 0x7facb0073500 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.382+0000 7facb5a47640 1 --2- 172.21.15.59:0/2448721773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facb0073f30 0x7facb015a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.382+0000 7facb5a47640 1 -- 172.21.15.59:0/2448721773 >> 172.21.15.59:0/2448721773 conn(0x7facb00b24c0 msgr2=0x7facb00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.382+0000 7facb5a47640 1 -- 172.21.15.59:0/2448721773 shutdown_connections 2024-06-29T14:22:54.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.382+0000 7facb5a47640 1 -- 172.21.15.59:0/2448721773 wait complete. 2024-06-29T14:22:54.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.384+0000 7facb5a47640 1 Processor -- start 2024-06-29T14:22:54.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.384+0000 7facb5a47640 1 -- start start 2024-06-29T14:22:54.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facb5a47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facb0073a40 0x7facb01e65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facb5a47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facb0073f30 0x7facb01f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facb5a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 0x7facb00af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facb5a47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7facb00b3a80 con 0x7facb00cf120 2024-06-29T14:22:54.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facb5a47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7facb006cb00 con 0x7facb0073f30 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facb5a47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7facb00b3c00 con 0x7facb0073a40 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facaf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 0x7facb00af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facaf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 0x7facb00af790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39932/0 (socket says 172.21.15.59:39932) 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facaf7fe640 1 -- 172.21.15.59:0/4170412116 learned_addr learned my addr 172.21.15.59:0/4170412116 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facae7fc640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facb0073f30 0x7facb01f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.385+0000 7facaeffd640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facb0073a40 0x7facb01e65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facaf7fe640 1 -- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facb0073a40 msgr2=0x7facb01e65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facaf7fe640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facb0073a40 0x7facb01e65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facaf7fe640 1 -- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facb0073f30 msgr2=0x7facb01f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facaf7fe640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facb0073f30 0x7facb01f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facaf7fe640 1 -- 172.21.15.59:0/4170412116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb01e6ae0 con 0x7facb00cf120 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facae7fc640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facb0073f30 0x7facb01f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facaf7fe640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 0x7facb00af790 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faca800c420 tx=0x7faca800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7fac97fff640 1 -- 172.21.15.59:0/4170412116 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8017070 con 0x7facb00cf120 2024-06-29T14:22:54.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7fac97fff640 1 -- 172.21.15.59:0/4170412116 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faca80053e0 con 0x7facb00cf120 2024-06-29T14:22:54.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7fac97fff640 1 -- 172.21.15.59:0/4170412116 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8003870 con 0x7facb00cf120 2024-06-29T14:22:54.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb01e2ef0 con 0x7facb00cf120 2024-06-29T14:22:54.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.386+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb01e34b0 con 0x7facb00cf120 2024-06-29T14:22:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.387+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb00b41c0 con 0x7facb00cf120 2024-06-29T14:22:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.388+0000 7fac97fff640 1 -- 172.21.15.59:0/4170412116 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faca801a070 con 0x7facb00cf120 2024-06-29T14:22:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.389+0000 7fac97fff640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fac9c076300 0x7fac9c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.389+0000 7fac97fff640 1 -- 172.21.15.59:0/4170412116 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7faca8096d60 con 0x7facb00cf120 2024-06-29T14:22:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.389+0000 7facaeffd640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fac9c076300 0x7fac9c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.389+0000 7facaeffd640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fac9c076300 0x7fac9c078810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7facb006f700 tx=0x7faca00095d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.393+0000 7fac97fff640 1 -- 172.21.15.59:0/4170412116 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faca805fc70 con 0x7facb00cf120 2024-06-29T14:22:54.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.583+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7facb01566c0 con 0x7facb00cf120 2024-06-29T14:22:54.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.583+0000 7fac97fff640 1 -- 172.21.15.59:0/4170412116 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7faca80682b0 con 0x7facb00cf120 2024-06-29T14:22:54.582 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:22:54.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fac9c076300 msgr2=0x7fac9c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fac9c076300 0x7fac9c078810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7facb006f700 tx=0x7faca00095d0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 msgr2=0x7facb00af790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 0x7facb00af790 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faca800c420 tx=0x7faca800c8e0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 shutdown_connections 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fac9c076300 0x7fac9c078810 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facb00cf120 0x7facb00af790 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facb0073f30 0x7facb01f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 --2- 172.21.15.59:0/4170412116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facb0073a40 0x7facb01e65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 >> 172.21.15.59:0/4170412116 conn(0x7facb00b24c0 msgr2=0x7facb0068420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 shutdown_connections 2024-06-29T14:22:54.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.585+0000 7facb5a47640 1 -- 172.21.15.59:0/4170412116 wait complete. 2024-06-29T14:22:54.596 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-29T14:22:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.718+0000 7f84ad78c640 1 -- 172.21.15.59:0/419192993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 msgr2=0x7f84a8162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.718+0000 7f84ad78c640 1 --2- 172.21.15.59:0/419192993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 0x7f84a8162c10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f84a0005fd0 tx=0x7f84a002a870 comp rx=0 tx=0).stop 2024-06-29T14:22:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.718+0000 7f84ad78c640 1 -- 172.21.15.59:0/419192993 shutdown_connections 2024-06-29T14:22:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.718+0000 7f84ad78c640 1 --2- 172.21.15.59:0/419192993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 0x7f84a8162c10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.718+0000 7f84ad78c640 1 --2- 172.21.15.59:0/419192993 >> v2:172.21.15.153:3300/0 conn(0x7f84a80ce950 0x7f84a81622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.718+0000 7f84ad78c640 1 -- 172.21.15.59:0/419192993 >> 172.21.15.59:0/419192993 conn(0x7f84a80b5610 msgr2=0x7f84a80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.718+0000 7f84ad78c640 1 -- 172.21.15.59:0/419192993 shutdown_connections 2024-06-29T14:22:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.718+0000 7f84ad78c640 1 -- 172.21.15.59:0/419192993 wait complete. 2024-06-29T14:22:54.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.720+0000 7f84ad78c640 1 Processor -- start 2024-06-29T14:22:54.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.720+0000 7f84ad78c640 1 -- start start 2024-06-29T14:22:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84ad78c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84a808c190 0x7f84a815bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84ad78c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84a80ce950 0x7f84a80b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84ad78c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 0x7f84a80b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84ad78c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f84a80b7f60 con 0x7f84a8162800 2024-06-29T14:22:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84ad78c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84a80b7de0 con 0x7f84a80ce950 2024-06-29T14:22:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84ad78c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f84a80b80e0 con 0x7f84a808c190 2024-06-29T14:22:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 0x7f84a80b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 0x7f84a80b8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39940/0 (socket says 172.21.15.59:39940) 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a77fe640 1 -- 172.21.15.59:0/2204294607 learned_addr learned my addr 172.21.15.59:0/2204294607 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a6ffd640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84a808c190 0x7f84a815bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a67fc640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84a80ce950 0x7f84a80b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a77fe640 1 -- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84a808c190 msgr2=0x7f84a815bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a77fe640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84a808c190 0x7f84a815bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a77fe640 1 -- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84a80ce950 msgr2=0x7f84a80b85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a77fe640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84a80ce950 0x7f84a80b85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.721+0000 7f84a77fe640 1 -- 172.21.15.59:0/2204294607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a80c9a70 con 0x7f84a8162800 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.722+0000 7f84a77fe640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 0x7f84a80b8af0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f84a0005fd0 tx=0x7f84a0033ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.722+0000 7f848f7fe640 1 -- 172.21.15.59:0/2204294607 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a0042020 con 0x7f84a8162800 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.722+0000 7f848f7fe640 1 -- 172.21.15.59:0/2204294607 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84a0033cc0 con 0x7f84a8162800 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.722+0000 7f848f7fe640 1 -- 172.21.15.59:0/2204294607 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a0037040 con 0x7f84a8162800 2024-06-29T14:22:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.722+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a81f54c0 con 0x7f84a8162800 2024-06-29T14:22:54.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.722+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a81f5a80 con 0x7f84a8162800 2024-06-29T14:22:54.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.723+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84a80d2c40 con 0x7f84a8162800 2024-06-29T14:22:54.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.724+0000 7f848f7fe640 1 -- 172.21.15.59:0/2204294607 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f84a0045020 con 0x7f84a8162800 2024-06-29T14:22:54.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.724+0000 7f848f7fe640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8494076300 0x7f8494078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.724+0000 7f848f7fe640 1 -- 172.21.15.59:0/2204294607 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f84a00b8900 con 0x7f84a8162800 2024-06-29T14:22:54.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.724+0000 7f84a6ffd640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8494076300 0x7f8494078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.725+0000 7f84a6ffd640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8494076300 0x7f8494078810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8498001610 tx=0x7f8498009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.729+0000 7f848f7fe640 1 -- 172.21.15.59:0/2204294607 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f84a80d2c40 con 0x7f84a8162800 2024-06-29T14:22:54.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.919+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f84a81f5c80 con 0x7f84a8162800 2024-06-29T14:22:54.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.919+0000 7f848f7fe640 1 -- 172.21.15.59:0/2204294607 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f84a004c030 con 0x7f84a8162800 2024-06-29T14:22:54.918 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:22:54.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8494076300 msgr2=0x7f8494078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8494076300 0x7f8494078810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8498001610 tx=0x7f8498009040 comp rx=0 tx=0).stop 2024-06-29T14:22:54.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 msgr2=0x7f84a80b8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 0x7f84a80b8af0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f84a0005fd0 tx=0x7f84a0033ad0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 shutdown_connections 2024-06-29T14:22:54.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f8494076300 0x7f8494078810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84a8162800 0x7f84a80b8af0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84a80ce950 0x7f84a80b85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 --2- 172.21.15.59:0/2204294607 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f84a808c190 0x7f84a815bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 >> 172.21.15.59:0/2204294607 conn(0x7f84a80b5610 msgr2=0x7f84a806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 shutdown_connections 2024-06-29T14:22:54.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:54.921+0000 7f84ad78c640 1 -- 172.21.15.59:0/2204294607 wait complete. 2024-06-29T14:22:54.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.051+0000 7fd9c3e09640 1 -- 172.21.15.59:0/56314859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 msgr2=0x7fd9a4003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.051+0000 7fd9c3e09640 1 --2- 172.21.15.59:0/56314859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 0x7fd9a4003f10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40016c0 tx=0x7fd9a4005830 comp rx=0 tx=0).stop 2024-06-29T14:22:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.051+0000 7fd9c23a6640 1 -- 172.21.15.59:0/56314859 reap_dead start 2024-06-29T14:22:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.051+0000 7fd9c3e09640 1 -- 172.21.15.59:0/56314859 shutdown_connections 2024-06-29T14:22:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.051+0000 7fd9c3e09640 1 -- 172.21.15.59:0/56314859 >> 172.21.15.59:0/56314859 conn(0x7fd9bc06c950 msgr2=0x7fd9bc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.051+0000 7fd9c3e09640 1 -- 172.21.15.59:0/56314859 shutdown_connections 2024-06-29T14:22:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.051+0000 7fd9c3e09640 1 -- 172.21.15.59:0/56314859 wait complete. 2024-06-29T14:22:55.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.053+0000 7fd9c3e09640 1 Processor -- start 2024-06-29T14:22:55.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.053+0000 7fd9c3e09640 1 -- start start 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.054+0000 7fd9c3e09640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 0x7fd9bc15b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.054+0000 7fd9c3e09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc1428a0 0x7fd9bc13b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.054+0000 7fd9c3e09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9bc1f5fb0 0x7fd9bc1f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.054+0000 7fd9c3e09640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc15cb10 con 0x7fd9a4003aa0 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.054+0000 7fd9c3e09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc15c990 con 0x7fd9bc1428a0 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.054+0000 7fd9c3e09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd9bc0b5f90 con 0x7fd9bc1f5fb0 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 0x7fd9bc15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 0x7fd9bc15b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39964/0 (socket says 172.21.15.59:39964) 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 -- 172.21.15.59:0/1289753680 learned_addr learned my addr 172.21.15.59:0/1289753680 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 -- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9bc1f5fb0 msgr2=0x7fd9bc1f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c23a6640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9bc1f5fb0 0x7fd9bc1f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c13a4640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc1428a0 0x7fd9bc13b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9bc1f5fb0 0x7fd9bc1f63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 -- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc1428a0 msgr2=0x7fd9bc13b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc1428a0 0x7fd9bc13b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 -- 172.21.15.59:0/1289753680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc1f6900 con 0x7fd9a4003aa0 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c23a6640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9bc1f5fb0 0x7fd9bc1f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c13a4640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc1428a0 0x7fd9bc13b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c1ba5640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 0x7fd9bc15b680 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40017b0 tx=0x7fd9b4006310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9ba7fc640 1 -- 172.21.15.59:0/1289753680 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4041070 con 0x7fd9a4003aa0 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9ba7fc640 1 -- 172.21.15.59:0/1289753680 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9b4006530 con 0x7fd9a4003aa0 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9ba7fc640 1 -- 172.21.15.59:0/1289753680 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4038070 con 0x7fd9a4003aa0 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9bc144e10 con 0x7fd9a4003aa0 2024-06-29T14:22:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.055+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9bc13c970 con 0x7fd9a4003aa0 2024-06-29T14:22:55.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.056+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9bc1458a0 con 0x7fd9a4003aa0 2024-06-29T14:22:55.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.057+0000 7fd9ba7fc640 1 -- 172.21.15.59:0/1289753680 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd9b4044070 con 0x7fd9a4003aa0 2024-06-29T14:22:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.058+0000 7fd9ba7fc640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd9a8076300 0x7fd9a8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.058+0000 7fd9ba7fc640 1 -- 172.21.15.59:0/1289753680 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd9b40b7330 con 0x7fd9a4003aa0 2024-06-29T14:22:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.058+0000 7fd9c13a4640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd9a8076300 0x7fd9a8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.059+0000 7fd9c13a4640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd9a8076300 0x7fd9a8078810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc15f5c0 tx=0x7fd9b0005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.062+0000 7fd9ba7fc640 1 -- 172.21.15.59:0/1289753680 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd9bc1f6900 con 0x7fd9a4003aa0 2024-06-29T14:22:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.293+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9bc0b1f10 con 0x7fd9a4003aa0 2024-06-29T14:22:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.293+0000 7fd9ba7fc640 1 -- 172.21.15.59:0/1289753680 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5222 (secure 0 0 0) 0x7fd9b4080890 con 0x7fd9a4003aa0 2024-06-29T14:22:55.293 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:55.293 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":9,"btime":"2024-06-29T14:22:12:518854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:22:11.909450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 9 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.295+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd9a8076300 msgr2=0x7fd9a8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.295+0000 7fd9c3e09640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd9a8076300 0x7fd9a8078810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc15f5c0 tx=0x7fd9b0005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.295+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 msgr2=0x7fd9bc15b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.295+0000 7fd9c3e09640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 0x7fd9bc15b680 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40017b0 tx=0x7fd9b4006310 comp rx=0 tx=0).stop 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.296+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 shutdown_connections 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.296+0000 7fd9c3e09640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd9a8076300 0x7fd9a8078810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.296+0000 7fd9c3e09640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9bc1f5fb0 0x7fd9bc1f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.296+0000 7fd9c3e09640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9bc1428a0 0x7fd9bc13b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.296+0000 7fd9c3e09640 1 --2- 172.21.15.59:0/1289753680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9a4003aa0 0x7fd9bc15b680 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.296+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 >> 172.21.15.59:0/1289753680 conn(0x7fd9bc06c950 msgr2=0x7fd9bc0d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.296+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 shutdown_connections 2024-06-29T14:22:55.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.296+0000 7fd9c3e09640 1 -- 172.21.15.59:0/1289753680 wait complete. 2024-06-29T14:22:55.305 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.425+0000 7f93dcb8c640 1 -- 172.21.15.59:0/867289471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 msgr2=0x7f93d80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.425+0000 7f93dcb8c640 1 --2- 172.21.15.59:0/867289471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 0x7f93d80b3990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009830 tx=0x7f93c802a5c0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.425+0000 7f93dcb8c640 1 -- 172.21.15.59:0/867289471 shutdown_connections 2024-06-29T14:22:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.425+0000 7f93dcb8c640 1 --2- 172.21.15.59:0/867289471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 0x7f93d80b3990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.425+0000 7f93dcb8c640 1 -- 172.21.15.59:0/867289471 >> 172.21.15.59:0/867289471 conn(0x7f93d80d3ee0 msgr2=0x7f93d80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.425+0000 7f93dcb8c640 1 -- 172.21.15.59:0/867289471 shutdown_connections 2024-06-29T14:22:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.425+0000 7f93dcb8c640 1 -- 172.21.15.59:0/867289471 wait complete. 2024-06-29T14:22:55.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.427+0000 7f93dcb8c640 1 Processor -- start 2024-06-29T14:22:55.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.427+0000 7f93dcb8c640 1 -- start start 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93dcb8c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80b3ed0 0x7f93d814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93dcb8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 0x7f93d8150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93dcb8c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d815efe0 0x7f93d806ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93dcb8c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f93d8160450 con 0x7f93d80d23c0 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93dcb8c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93d81602d0 con 0x7f93d80b3ed0 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93dcb8c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f93d81605d0 con 0x7f93d815efe0 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 0x7f93d8150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 0x7f93d8150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39978/0 (socket says 172.21.15.59:39978) 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 -- 172.21.15.59:0/3805257253 learned_addr learned my addr 172.21.15.59:0/3805257253 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d659c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80b3ed0 0x7f93d814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d6d9d640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d815efe0 0x7f93d806ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 -- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d815efe0 msgr2=0x7f93d806ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d815efe0 0x7f93d806ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 -- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80b3ed0 msgr2=0x7f93d814e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80b3ed0 0x7f93d814e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 -- 172.21.15.59:0/3805257253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d80c8670 con 0x7f93d80d23c0 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93d5d9b640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 0x7f93d8150550 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00c370 tx=0x7f93cc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.429+0000 7f93c6ffd640 1 -- 172.21.15.59:0/3805257253 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc017070 con 0x7f93d80d23c0 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.430+0000 7f93c6ffd640 1 -- 172.21.15.59:0/3805257253 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93cc0053c0 con 0x7f93d80d23c0 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.430+0000 7f93c6ffd640 1 -- 172.21.15.59:0/3805257253 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc003850 con 0x7f93d80d23c0 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.430+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d814eee0 con 0x7f93d80d23c0 2024-06-29T14:22:55.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.430+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d814f4a0 con 0x7f93d80d23c0 2024-06-29T14:22:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.431+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d814ec50 con 0x7f93d80d23c0 2024-06-29T14:22:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.431+0000 7f93c6ffd640 1 -- 172.21.15.59:0/3805257253 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f93cc01a070 con 0x7f93d80d23c0 2024-06-29T14:22:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.432+0000 7f93c6ffd640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f93b80762d0 0x7f93b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.432+0000 7f93c6ffd640 1 -- 172.21.15.59:0/3805257253 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f93cc096f40 con 0x7f93d80d23c0 2024-06-29T14:22:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.432+0000 7f93d659c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f93b80762d0 0x7f93b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.433+0000 7f93d659c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f93b80762d0 0x7f93b80787e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f93c8034000 tx=0x7f93c8004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.436+0000 7f93c6ffd640 1 -- 172.21.15.59:0/3805257253 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f93cc05fdc0 con 0x7f93d80d23c0 2024-06-29T14:22:55.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.631+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93d8155440 con 0x7f93d80d23c0 2024-06-29T14:22:55.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.632+0000 7f93c6ffd640 1 -- 172.21.15.59:0/3805257253 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7f93cc0640c0 con 0x7f93d80d23c0 2024-06-29T14:22:55.631 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:55.631 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f93b80762d0 msgr2=0x7f93b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f93b80762d0 0x7f93b80787e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f93c8034000 tx=0x7f93c8004730 comp rx=0 tx=0).stop 2024-06-29T14:22:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 msgr2=0x7f93d8150550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 0x7f93d8150550 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00c370 tx=0x7f93cc00c830 comp rx=0 tx=0).stop 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 shutdown_connections 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f93b80762d0 0x7f93b80787e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f93d815efe0 0x7f93d806ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d80d23c0 0x7f93d8150550 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 --2- 172.21.15.59:0/3805257253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93d80b3ed0 0x7f93d814e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 >> 172.21.15.59:0/3805257253 conn(0x7f93d80d3ee0 msgr2=0x7f93d8153a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 shutdown_connections 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.634+0000 7f93dcb8c640 1 -- 172.21.15.59:0/3805257253 wait complete. 2024-06-29T14:22:55.645 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:22:55.645 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.764+0000 7f54a5931640 1 -- 172.21.15.59:0/3676540747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a013e9b0 msgr2=0x7f54a013edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.764+0000 7f54a5931640 1 --2- 172.21.15.59:0/3676540747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a013e9b0 0x7f54a013edc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5488009830 tx=0x7f548802a480 comp rx=0 tx=0).stop 2024-06-29T14:22:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.764+0000 7f54a5931640 1 -- 172.21.15.59:0/3676540747 shutdown_connections 2024-06-29T14:22:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.764+0000 7f54a5931640 1 --2- 172.21.15.59:0/3676540747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a013e9b0 0x7f54a013edc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.764+0000 7f54a5931640 1 --2- 172.21.15.59:0/3676540747 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54a015ceb0 0x7f54a00b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.764+0000 7f54a5931640 1 -- 172.21.15.59:0/3676540747 >> 172.21.15.59:0/3676540747 conn(0x7f54a00b6fb0 msgr2=0x7f54a00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.764+0000 7f54a5931640 1 -- 172.21.15.59:0/3676540747 shutdown_connections 2024-06-29T14:22:55.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.764+0000 7f54a5931640 1 -- 172.21.15.59:0/3676540747 wait complete. 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.766+0000 7f54a5931640 1 Processor -- start 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.766+0000 7f54a5931640 1 -- start start 2024-06-29T14:22:55.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f54a5931640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a013e9b0 0x7f54a013b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f54a5931640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a01414c0 0x7f54a013f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f54a5931640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54a015ceb0 0x7f54a013fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f54a5931640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f54a00b2ab0 con 0x7f54a013e9b0 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f54a5931640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54a00b2930 con 0x7f54a01414c0 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f54a5931640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f54a00b2c30 con 0x7f54a015ceb0 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f549effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a013e9b0 0x7f54a013b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f549effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a013e9b0 0x7f54a013b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39986/0 (socket says 172.21.15.59:39986) 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f549effd640 1 -- 172.21.15.59:0/1407161350 learned_addr learned my addr 172.21.15.59:0/1407161350 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.767+0000 7f5497fff640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a01414c0 0x7f54a013f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549f7fe640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54a015ceb0 0x7f54a013fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549effd640 1 -- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54a015ceb0 msgr2=0x7f54a013fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549effd640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54a015ceb0 0x7f54a013fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549effd640 1 -- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a01414c0 msgr2=0x7f54a013f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549effd640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a01414c0 0x7f54a013f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549effd640 1 -- 172.21.15.59:0/1407161350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a013c890 con 0x7f54a013e9b0 2024-06-29T14:22:55.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549effd640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a013e9b0 0x7f54a013b920 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f549000ea20 tx=0x7f549000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549cff9640 1 -- 172.21.15.59:0/1407161350 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549000ce40 con 0x7f54a013e9b0 2024-06-29T14:22:55.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549cff9640 1 -- 172.21.15.59:0/1407161350 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5490004020 con 0x7f54a013e9b0 2024-06-29T14:22:55.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a00ccec0 con 0x7f54a013e9b0 2024-06-29T14:22:55.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f549cff9640 1 -- 172.21.15.59:0/1407161350 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490004470 con 0x7f54a013e9b0 2024-06-29T14:22:55.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.768+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a00cd480 con 0x7f54a013e9b0 2024-06-29T14:22:55.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.769+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a013edc0 con 0x7f54a013e9b0 2024-06-29T14:22:55.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.770+0000 7f549cff9640 1 -- 172.21.15.59:0/1407161350 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f54900035f0 con 0x7f54a013e9b0 2024-06-29T14:22:55.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.770+0000 7f549cff9640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5480076300 0x7f5480078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.770+0000 7f549cff9640 1 -- 172.21.15.59:0/1407161350 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f54900982a0 con 0x7f54a013e9b0 2024-06-29T14:22:55.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.771+0000 7f5497fff640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5480076300 0x7f5480078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.771+0000 7f5497fff640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5480076300 0x7f5480078810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5488034000 tx=0x7f5488004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:55.775+0000 7f549cff9640 1 -- 172.21.15.59:0/1407161350 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f54a013c890 con 0x7f54a013e9b0 2024-06-29T14:22:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.006+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54a00c7fc0 con 0x7f54a013e9b0 2024-06-29T14:22:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.006+0000 7f549cff9640 1 -- 172.21.15.59:0/1407161350 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5222 (secure 0 0 0) 0x7f54900611b0 con 0x7f54a013e9b0 2024-06-29T14:22:56.005 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:56.005 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":9,"btime":"2024-06-29T14:22:12:518854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:22:11.909450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:56.006 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 9 2024-06-29T14:22:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5480076300 msgr2=0x7f5480078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5480076300 0x7f5480078810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5488034000 tx=0x7f5488004730 comp rx=0 tx=0).stop 2024-06-29T14:22:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a013e9b0 msgr2=0x7f54a013b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a013e9b0 0x7f54a013b920 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f549000ea20 tx=0x7f549000eee0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 shutdown_connections 2024-06-29T14:22:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f5480076300 0x7f5480078810 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54a015ceb0 0x7f54a013fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a01414c0 0x7f54a013f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 --2- 172.21.15.59:0/1407161350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a013e9b0 0x7f54a013b920 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 >> 172.21.15.59:0/1407161350 conn(0x7f54a00b6fb0 msgr2=0x7f54a015d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 shutdown_connections 2024-06-29T14:22:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.008+0000 7f54a5931640 1 -- 172.21.15.59:0/1407161350 wait complete. 2024-06-29T14:22:56.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:56.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.138+0000 7f5842d7f640 1 -- 172.21.15.59:0/2283904487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 msgr2=0x7f583c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.138+0000 7f5842d7f640 1 --2- 172.21.15.59:0/2283904487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 0x7f583c0cf080 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f582c009900 tx=0x7f582c02a580 comp rx=0 tx=0).stop 2024-06-29T14:22:56.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.138+0000 7f5842d7f640 1 -- 172.21.15.59:0/2283904487 shutdown_connections 2024-06-29T14:22:56.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.138+0000 7f5842d7f640 1 --2- 172.21.15.59:0/2283904487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 0x7f583c0cf080 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.138+0000 7f5842d7f640 1 -- 172.21.15.59:0/2283904487 >> 172.21.15.59:0/2283904487 conn(0x7f583c0b5610 msgr2=0x7f583c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.138+0000 7f5842d7f640 1 -- 172.21.15.59:0/2283904487 shutdown_connections 2024-06-29T14:22:56.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.138+0000 7f5842d7f640 1 -- 172.21.15.59:0/2283904487 wait complete. 2024-06-29T14:22:56.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.140+0000 7f5842d7f640 1 Processor -- start 2024-06-29T14:22:56.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.140+0000 7f5842d7f640 1 -- start start 2024-06-29T14:22:56.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5842d7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 0x7f583c162250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5842d7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f583c0ce950 0x7f583c0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5842d7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f583c0cf5c0 0x7f583c159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5842d7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f583c169690 con 0x7f583c08c190 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5842d7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f583c169510 con 0x7f583c0cf5c0 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5842d7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f583c169810 con 0x7f583c0ce950 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5840b1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 0x7f583c162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5840b1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 0x7f583c162250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40000/0 (socket says 172.21.15.59:40000) 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f5840b1b640 1 -- 172.21.15.59:0/394798907 learned_addr learned my addr 172.21.15.59:0/394798907 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f584131c640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f583c0cf5c0 0x7f583c159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.141+0000 7f583bfff640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f583c0ce950 0x7f583c0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f5840b1b640 1 -- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f583c0ce950 msgr2=0x7f583c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f5840b1b640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f583c0ce950 0x7f583c0cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f5840b1b640 1 -- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f583c0cf5c0 msgr2=0x7f583c159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f5840b1b640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f583c0cf5c0 0x7f583c159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f5840b1b640 1 -- 172.21.15.59:0/394798907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583c0cd9f0 con 0x7f583c08c190 2024-06-29T14:22:56.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f5840b1b640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 0x7f583c162250 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f582c006010 tx=0x7f582c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f58397fa640 1 -- 172.21.15.59:0/394798907 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c03e030 con 0x7f583c08c190 2024-06-29T14:22:56.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f58397fa640 1 -- 172.21.15.59:0/394798907 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f582c003e60 con 0x7f583c08c190 2024-06-29T14:22:56.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583c0cdcd0 con 0x7f583c08c190 2024-06-29T14:22:56.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f58397fa640 1 -- 172.21.15.59:0/394798907 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c035040 con 0x7f583c08c190 2024-06-29T14:22:56.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.142+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f583c15a0b0 con 0x7f583c08c190 2024-06-29T14:22:56.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.143+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f583c0cf080 con 0x7f583c08c190 2024-06-29T14:22:56.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.144+0000 7f58397fa640 1 -- 172.21.15.59:0/394798907 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f582c041030 con 0x7f583c08c190 2024-06-29T14:22:56.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.145+0000 7f58397fa640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58280762d0 0x7f58280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.145+0000 7f58397fa640 1 -- 172.21.15.59:0/394798907 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f582c0b8330 con 0x7f583c08c190 2024-06-29T14:22:56.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.145+0000 7f583bfff640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58280762d0 0x7f58280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.145+0000 7f583bfff640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58280762d0 0x7f58280787e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f583c08c720 tx=0x7f5830009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.149+0000 7f58397fa640 1 -- 172.21.15.59:0/394798907 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f582c0811b0 con 0x7f583c08c190 2024-06-29T14:22:56.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.378+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f583c0cdea0 con 0x7f583c08c190 2024-06-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.379+0000 7f58397fa640 1 -- 172.21.15.59:0/394798907 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5222 (secure 0 0 0) 0x7f582c0854b0 con 0x7f583c08c190 2024-06-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:56.378 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":9,"btime":"2024-06-29T14:22:12:518854+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/911874535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":911874535},{"type":"v1","addr":"172.21.15.59:6834","nonce":911874535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2387017032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2387017032},{"type":"v1","addr":"172.21.15.59:6835","nonce":2387017032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2028766371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2028766371},{"type":"v1","addr":"172.21.15.153:6837","nonce":2028766371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:21:54.635958+0000","modified":"2024-06-29T14:22:11.909450+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4375":{"gid":4375,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2240732059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2240732059},{"type":"v1","addr":"172.21.15.153:6836","nonce":2240732059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4375,"qdb_cluster":[4375]},"id":1}]} 2024-06-29T14:22:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 9 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.381+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58280762d0 msgr2=0x7f58280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.381+0000 7f5842d7f640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58280762d0 0x7f58280787e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f583c08c720 tx=0x7f5830009060 comp rx=0 tx=0).stop 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.381+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 msgr2=0x7f583c162250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.381+0000 7f5842d7f640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 0x7f583c162250 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f582c006010 tx=0x7f582c003970 comp rx=0 tx=0).stop 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.382+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 shutdown_connections 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.382+0000 7f5842d7f640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f58280762d0 0x7f58280787e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.382+0000 7f5842d7f640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f583c0cf5c0 0x7f583c159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.382+0000 7f5842d7f640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f583c0ce950 0x7f583c0cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.382+0000 7f5842d7f640 1 --2- 172.21.15.59:0/394798907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583c08c190 0x7f583c162250 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.382+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 >> 172.21.15.59:0/394798907 conn(0x7f583c0b5610 msgr2=0x7f583c157060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.382+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 shutdown_connections 2024-06-29T14:22:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.382+0000 7f5842d7f640 1 -- 172.21.15.59:0/394798907 wait complete. 2024-06-29T14:22:56.392 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:56.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 -- 172.21.15.59:0/3125578888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e40cf5c0 msgr2=0x7fa9e40d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 --2- 172.21.15.59:0/3125578888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e40cf5c0 0x7fa9e40d1a20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40cfbc0 tx=0x7fa9d802a560 comp rx=0 tx=0).stop 2024-06-29T14:22:56.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 -- 172.21.15.59:0/3125578888 shutdown_connections 2024-06-29T14:22:56.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 --2- 172.21.15.59:0/3125578888 >> v2:172.21.15.153:3300/0 conn(0x7fa9e40ce950 0x7fa9e40d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 --2- 172.21.15.59:0/3125578888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e40cf5c0 0x7fa9e40d1a20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 --2- 172.21.15.59:0/3125578888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9e408c190 0x7fa9e40cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 -- 172.21.15.59:0/3125578888 >> 172.21.15.59:0/3125578888 conn(0x7fa9e40b5610 msgr2=0x7fa9e40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 -- 172.21.15.59:0/3125578888 shutdown_connections 2024-06-29T14:22:56.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.510+0000 7fa9ea490640 1 -- 172.21.15.59:0/3125578888 wait complete. 2024-06-29T14:22:56.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.512+0000 7fa9ea490640 1 Processor -- start 2024-06-29T14:22:56.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.512+0000 7fa9ea490640 1 -- start start 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9ea490640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e408c190 0x7fa9e41532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9ea490640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9e40ce950 0x7fa9e41613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9ea490640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e40cf5c0 0x7fa9e4161910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9ea490640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e4162440 con 0x7fa9e408c190 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9ea490640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e41622c0 con 0x7fa9e40cf5c0 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9ea490640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e41625c0 con 0x7fa9e40ce950 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e408c190 0x7fa9e41532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e408c190 0x7fa9e41532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40008/0 (socket says 172.21.15.59:40008) 2024-06-29T14:22:56.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 -- 172.21.15.59:0/594759440 learned_addr learned my addr 172.21.15.59:0/594759440 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e37fe640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9e40ce950 0x7fa9e41613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e8a2d640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e40cf5c0 0x7fa9e4161910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 -- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9e40ce950 msgr2=0x7fa9e41613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9e40ce950 0x7fa9e41613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 -- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e40cf5c0 msgr2=0x7fa9e4161910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e40cf5c0 0x7fa9e4161910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 -- 172.21.15.59:0/594759440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e40cd590 con 0x7fa9e408c190 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.514+0000 7fa9e3fff640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e408c190 0x7fa9e41532b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400d780 tx=0x7fa9d400dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.515+0000 7fa9e0ff9640 1 -- 172.21.15.59:0/594759440 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4017070 con 0x7fa9e408c190 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.515+0000 7fa9e0ff9640 1 -- 172.21.15.59:0/594759440 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9d400de90 con 0x7fa9e408c190 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.515+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e40cc980 con 0x7fa9e408c190 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.515+0000 7fa9e0ff9640 1 -- 172.21.15.59:0/594759440 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4010460 con 0x7fa9e408c190 2024-06-29T14:22:56.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.515+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e40ccf40 con 0x7fa9e408c190 2024-06-29T14:22:56.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.516+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e40cf080 con 0x7fa9e408c190 2024-06-29T14:22:56.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.517+0000 7fa9e0ff9640 1 -- 172.21.15.59:0/594759440 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa9d401a070 con 0x7fa9e408c190 2024-06-29T14:22:56.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.517+0000 7fa9e0ff9640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa9d0076300 0x7fa9d0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.517+0000 7fa9e0ff9640 1 -- 172.21.15.59:0/594759440 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fa9d4097c60 con 0x7fa9e408c190 2024-06-29T14:22:56.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.517+0000 7fa9e37fe640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa9d0076300 0x7fa9d0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.518+0000 7fa9e37fe640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa9d0076300 0x7fa9d0078810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8033000 tx=0x7fa9d8004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.521+0000 7fa9e0ff9640 1 -- 172.21.15.59:0/594759440 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa9d40610b0 con 0x7fa9e408c190 2024-06-29T14:22:56.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.716+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9e40b5cd0 con 0x7fa9e408c190 2024-06-29T14:22:56.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.717+0000 7fa9e0ff9640 1 -- 172.21.15.59:0/594759440 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18915 (secure 0 0 0) 0x7fa9d4097090 con 0x7fa9e408c190 2024-06-29T14:22:56.716 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:22:56.716 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:22:34.818829+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:21:52.912626+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:21:53.621207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-29T14:21:57.292065+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.13","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:56.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa9d0076300 msgr2=0x7fa9d0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa9d0076300 0x7fa9d0078810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8033000 tx=0x7fa9d8004730 comp rx=0 tx=0).stop 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e408c190 msgr2=0x7fa9e41532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e408c190 0x7fa9e41532b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400d780 tx=0x7fa9d400dc40 comp rx=0 tx=0).stop 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 shutdown_connections 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa9d0076300 0x7fa9d0078810 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9e40cf5c0 0x7fa9e4161910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9e40ce950 0x7fa9e41613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 --2- 172.21.15.59:0/594759440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e408c190 0x7fa9e41532b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 >> 172.21.15.59:0/594759440 conn(0x7fa9e40b5610 msgr2=0x7fa9e40c8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 shutdown_connections 2024-06-29T14:22:56.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.719+0000 7fa9ea490640 1 -- 172.21.15.59:0/594759440 wait complete. 2024-06-29T14:22:56.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:22:56.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.850+0000 7fae745b2640 1 -- 172.21.15.59:0/1952319433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae6c08c190 msgr2=0x7fae6c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.850+0000 7fae745b2640 1 --2- 172.21.15.59:0/1952319433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae6c08c190 0x7fae6c0cf080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fae68009be0 tx=0x7fae6802a8d0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.850+0000 7fae745b2640 1 -- 172.21.15.59:0/1952319433 shutdown_connections 2024-06-29T14:22:56.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.850+0000 7fae745b2640 1 --2- 172.21.15.59:0/1952319433 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae6c0ce950 0x7fae6c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.850+0000 7fae745b2640 1 --2- 172.21.15.59:0/1952319433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae6c08c190 0x7fae6c0cf080 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.850+0000 7fae745b2640 1 -- 172.21.15.59:0/1952319433 >> 172.21.15.59:0/1952319433 conn(0x7fae6c0b5610 msgr2=0x7fae6c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.851+0000 7fae745b2640 1 -- 172.21.15.59:0/1952319433 shutdown_connections 2024-06-29T14:22:56.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.851+0000 7fae745b2640 1 -- 172.21.15.59:0/1952319433 wait complete. 2024-06-29T14:22:56.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.852+0000 7fae745b2640 1 Processor -- start 2024-06-29T14:22:56.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.852+0000 7fae745b2640 1 -- start start 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae745b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae6c08c190 0x7fae6c1529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae745b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae6c0ce950 0x7fae6c160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae745b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae6c0cf5c0 0x7fae6c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae745b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c0b7f60 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae745b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae6c0b7de0 con 0x7fae6c0ce950 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae745b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fae6c0b80e0 con 0x7fae6c08c190 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae6c0cf5c0 0x7fae6c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae6c0cf5c0 0x7fae6c0ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40020/0 (socket says 172.21.15.59:40020) 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 -- 172.21.15.59:0/1333834473 learned_addr learned my addr 172.21.15.59:0/1333834473 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:56.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae7234e640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae6c08c190 0x7fae6c1529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae71b4d640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae6c0ce950 0x7fae6c160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 -- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae6c08c190 msgr2=0x7fae6c1529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae6c08c190 0x7fae6c1529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 -- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae6c0ce950 msgr2=0x7fae6c160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae6c0ce950 0x7fae6c160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 -- 172.21.15.59:0/1333834473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6c15ff90 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.854+0000 7fae72b4f640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae6c0cf5c0 0x7fae6c0ccbe0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fae640058c0 tx=0x7fae6400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.855+0000 7fae62ffd640 1 -- 172.21.15.59:0/1333834473 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6400ccc0 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.855+0000 7fae62ffd640 1 -- 172.21.15.59:0/1333834473 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae64004030 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.855+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6c15eef0 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.855+0000 7fae62ffd640 1 -- 172.21.15.59:0/1333834473 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6400ce60 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.855+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6c15f4b0 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.856+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae6c0cf080 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.856+0000 7fae62ffd640 1 -- 172.21.15.59:0/1333834473 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fae640035f0 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.857+0000 7fae62ffd640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fae50076300 0x7fae50078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.857+0000 7fae62ffd640 1 -- 172.21.15.59:0/1333834473 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fae640983c0 con 0x7fae6c0cf5c0 2024-06-29T14:22:56.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.857+0000 7fae7234e640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fae50076300 0x7fae50078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.858+0000 7fae7234e640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fae50076300 0x7fae50078810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fae68034000 tx=0x7fae68004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:56.862+0000 7fae62ffd640 1 -- 172.21.15.59:0/1333834473 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fae64061240 con 0x7fae6c0cf5c0 2024-06-29T14:22:57.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.093+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fae6c15db00 con 0x7fae6c0cf5c0 2024-06-29T14:22:57.093 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T14:22:57.094+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:22:57.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.095+0000 7fae62ffd640 1 -- 172.21.15.59:0/1333834473 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7fae64065540 con 0x7fae6c0cf5c0 2024-06-29T14:22:57.094 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:22:57.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fae50076300 msgr2=0x7fae50078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fae50076300 0x7fae50078810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fae68034000 tx=0x7fae68004ab0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae6c0cf5c0 msgr2=0x7fae6c0ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae6c0cf5c0 0x7fae6c0ccbe0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fae640058c0 tx=0x7fae6400ed20 comp rx=0 tx=0).stop 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 shutdown_connections 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fae50076300 0x7fae50078810 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae6c0cf5c0 0x7fae6c0ccbe0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae6c0ce950 0x7fae6c160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 --2- 172.21.15.59:0/1333834473 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae6c08c190 0x7fae6c1529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 >> 172.21.15.59:0/1333834473 conn(0x7fae6c0b5610 msgr2=0x7fae6c0b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 shutdown_connections 2024-06-29T14:22:57.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.098+0000 7fae745b2640 1 -- 172.21.15.59:0/1333834473 wait complete. 2024-06-29T14:22:57.102 INFO:tasks.ceph.mds.b.smithi153.stderr: -5565> 2024-06-29T14:21:59.225+0000 7f05276ca640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:22:57.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:22:57.160 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:22:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.230+0000 7fef8b90a640 1 -- 172.21.15.59:0/3917257734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef840b2a50 msgr2=0x7fef84161380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.230+0000 7fef8b90a640 1 --2- 172.21.15.59:0/3917257734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef840b2a50 0x7fef84161380 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fef840d2c60 tx=0x7fef8002a510 comp rx=0 tx=0).stop 2024-06-29T14:22:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.230+0000 7fef8b90a640 1 -- 172.21.15.59:0/3917257734 shutdown_connections 2024-06-29T14:22:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.230+0000 7fef8b90a640 1 --2- 172.21.15.59:0/3917257734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef840b2a50 0x7fef84161380 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.230+0000 7fef8b90a640 1 -- 172.21.15.59:0/3917257734 >> 172.21.15.59:0/3917257734 conn(0x7fef840b5590 msgr2=0x7fef840b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.230+0000 7fef8b90a640 1 -- 172.21.15.59:0/3917257734 shutdown_connections 2024-06-29T14:22:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.230+0000 7fef8b90a640 1 -- 172.21.15.59:0/3917257734 wait complete. 2024-06-29T14:22:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.232+0000 7fef8b90a640 1 Processor -- start 2024-06-29T14:22:57.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.232+0000 7fef8b90a640 1 -- start start 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef8b90a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef840b2a50 0x7fef84160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef8b90a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef840b7150 0x7fef8415b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef8b90a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef841f5430 0x7fef841599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef8b90a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fef840b4da0 con 0x7fef841f5430 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef8b90a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fef840b4c20 con 0x7fef840b2a50 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef8b90a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fef840b7560 con 0x7fef840b7150 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef89ea7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef841f5430 0x7fef841599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef89ea7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef841f5430 0x7fef841599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40048/0 (socket says 172.21.15.59:40048) 2024-06-29T14:22:57.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef89ea7640 1 -- 172.21.15.59:0/2241038949 learned_addr learned my addr 172.21.15.59:0/2241038949 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef896a6640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef840b2a50 0x7fef84160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef88ea5640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef840b7150 0x7fef8415b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef89ea7640 1 -- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef840b7150 msgr2=0x7fef8415b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef89ea7640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef840b7150 0x7fef8415b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef89ea7640 1 -- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef840b2a50 msgr2=0x7fef84160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef89ea7640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef840b2a50 0x7fef84160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.233+0000 7fef89ea7640 1 -- 172.21.15.59:0/2241038949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef84160c80 con 0x7fef841f5430 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.234+0000 7fef89ea7640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef841f5430 0x7fef841599f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00c760 tx=0x7fef7c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.234+0000 7fef79ffb640 1 -- 172.21.15.59:0/2241038949 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c003da0 con 0x7fef841f5430 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.234+0000 7fef79ffb640 1 -- 172.21.15.59:0/2241038949 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef7c00ce10 con 0x7fef841f5430 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.234+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef841609a0 con 0x7fef841f5430 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.234+0000 7fef79ffb640 1 -- 172.21.15.59:0/2241038949 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0052d0 con 0x7fef841f5430 2024-06-29T14:22:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.234+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef841591f0 con 0x7fef841f5430 2024-06-29T14:22:57.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.235+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef84161380 con 0x7fef841f5430 2024-06-29T14:22:57.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.236+0000 7fef79ffb640 1 -- 172.21.15.59:0/2241038949 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fef7c005470 con 0x7fef841f5430 2024-06-29T14:22:57.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.236+0000 7fef79ffb640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fef68076300 0x7fef68078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.236+0000 7fef79ffb640 1 -- 172.21.15.59:0/2241038949 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6821+0+0 (secure 0 0 0) 0x7fef7c0977c0 con 0x7fef841f5430 2024-06-29T14:22:57.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.236+0000 7fef896a6640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fef68076300 0x7fef68078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.237+0000 7fef896a6640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fef68076300 0x7fef68078810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fef80034000 tx=0x7fef80004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:57.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.241+0000 7fef79ffb640 1 -- 172.21.15.59:0/2241038949 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fef7c060700 con 0x7fef841f5430 2024-06-29T14:22:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:57.472+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fef8408ccd0 con 0x7fef841f5430 2024-06-29T14:22:58.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.101+0000 7fef79ffb640 1 -- 172.21.15.59:0/2241038949 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fef7c064a00 con 0x7fef841f5430 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fef68076300 msgr2=0x7fef68078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fef68076300 0x7fef68078810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fef80034000 tx=0x7fef80004730 comp rx=0 tx=0).stop 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef841f5430 msgr2=0x7fef841599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef841f5430 0x7fef841599f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00c760 tx=0x7fef7c00cc20 comp rx=0 tx=0).stop 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 shutdown_connections 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fef68076300 0x7fef68078810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef841f5430 0x7fef841599f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef840b7150 0x7fef8415b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 --2- 172.21.15.59:0/2241038949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef840b2a50 0x7fef84160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 >> 172.21.15.59:0/2241038949 conn(0x7fef840b5590 msgr2=0x7fef840b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 shutdown_connections 2024-06-29T14:22:58.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.105+0000 7fef8b90a640 1 -- 172.21.15.59:0/2241038949 wait complete. 2024-06-29T14:22:58.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:22:58.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.250+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/2510258160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d74004d40 msgr2=0x7f7d74005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.250+0000 7f7d8fbcb640 1 --2- 172.21.15.59:0/2510258160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d74004d40 0x7f7d74005150 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7d80005fd0 tx=0x7f7d74005830 comp rx=0 tx=0).stop 2024-06-29T14:22:58.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.250+0000 7f7d8e168640 1 -- 172.21.15.59:0/2510258160 reap_dead start 2024-06-29T14:22:58.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.250+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/2510258160 shutdown_connections 2024-06-29T14:22:58.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.250+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/2510258160 >> 172.21.15.59:0/2510258160 conn(0x7f7d880d3e60 msgr2=0x7f7d880d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.250+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/2510258160 shutdown_connections 2024-06-29T14:22:58.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.250+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/2510258160 wait complete. 2024-06-29T14:22:58.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.252+0000 7f7d8fbcb640 1 Processor -- start 2024-06-29T14:22:58.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.252+0000 7f7d8fbcb640 1 -- start start 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8fbcb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d74004d40 0x7f7d881f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8fbcb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d88150f30 0x7f7d881507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8fbcb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d88160b10 0x7f7d88160f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8fbcb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7d880d2b10 con 0x7f7d88160b10 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8fbcb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d880d2990 con 0x7f7d74004d40 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8fbcb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7d880b4410 con 0x7f7d88150f30 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8e168640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d88160b10 0x7f7d88160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8e168640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d88160b10 0x7f7d88160f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40066/0 (socket says 172.21.15.59:40066) 2024-06-29T14:22:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8e168640 1 -- 172.21.15.59:0/4075820136 learned_addr learned my addr 172.21.15.59:0/4075820136 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8d967640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d74004d40 0x7f7d881f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.253+0000 7f7d8d166640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d88150f30 0x7f7d881507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d8e168640 1 -- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d88150f30 msgr2=0x7f7d881507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d8e168640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d88150f30 0x7f7d881507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d8e168640 1 -- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d74004d40 msgr2=0x7f7d881f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d8e168640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d74004d40 0x7f7d881f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d8e168640 1 -- 172.21.15.59:0/4075820136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d88150ce0 con 0x7f7d88160b10 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d8e168640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d88160b10 0x7f7d88160f20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7d80036040 tx=0x7f7d80034c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d7e7fc640 1 -- 172.21.15.59:0/4075820136 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80038040 con 0x7f7d88160b10 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d7e7fc640 1 -- 172.21.15.59:0/4075820136 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d8002bc30 con 0x7f7d88160b10 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d88161460 con 0x7f7d88160b10 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d7e7fc640 1 -- 172.21.15.59:0/4075820136 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80009200 con 0x7f7d88160b10 2024-06-29T14:22:58.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.254+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d880cad40 con 0x7f7d88160b10 2024-06-29T14:22:58.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.255+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d880ce690 con 0x7f7d88160b10 2024-06-29T14:22:58.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.256+0000 7f7d7e7fc640 1 -- 172.21.15.59:0/4075820136 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7d80044020 con 0x7f7d88160b10 2024-06-29T14:22:58.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.256+0000 7f7d7e7fc640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d6c0762d0 0x7f7d6c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.257+0000 7f7d7e7fc640 1 -- 172.21.15.59:0/4075820136 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6821+0+0 (secure 0 0 0) 0x7f7d800b80b0 con 0x7f7d88160b10 2024-06-29T14:22:58.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.257+0000 7f7d8d967640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d6c0762d0 0x7f7d6c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.257+0000 7f7d8d967640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d6c0762d0 0x7f7d6c0787e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009bd0 tx=0x7f7d84005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.261+0000 7f7d7e7fc640 1 -- 172.21.15.59:0/4075820136 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7d8004b020 con 0x7f7d88160b10 2024-06-29T14:22:58.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:58.453+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7d880b3ed0 con 0x7f7d88160b10 2024-06-29T14:22:59.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.104+0000 7f7d7e7fc640 1 -- 172.21.15.59:0/4075820136 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v30) ==== 181+0+0 (secure 0 0 0) 0x7f7d800814c0 con 0x7f7d88160b10 2024-06-29T14:22:59.104 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:22:59.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d6c0762d0 msgr2=0x7f7d6c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d6c0762d0 0x7f7d6c0787e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009bd0 tx=0x7f7d84005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:59.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d88160b10 msgr2=0x7f7d88160f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d88160b10 0x7f7d88160f20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7d80036040 tx=0x7f7d80034c20 comp rx=0 tx=0).stop 2024-06-29T14:22:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 shutdown_connections 2024-06-29T14:22:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7d6c0762d0 0x7f7d6c0787e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d88160b10 0x7f7d88160f20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d88150f30 0x7f7d881507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 --2- 172.21.15.59:0/4075820136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d74004d40 0x7f7d881f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 >> 172.21.15.59:0/4075820136 conn(0x7f7d880d3e60 msgr2=0x7f7d880b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:59.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 shutdown_connections 2024-06-29T14:22:59.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.108+0000 7f7d8fbcb640 1 -- 172.21.15.59:0/4075820136 wait complete. 2024-06-29T14:22:59.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:22:59.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.254+0000 7f39efb9e640 1 -- 172.21.15.59:0/3813107350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e8145650 msgr2=0x7f39e80b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.254+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3813107350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e8145650 0x7f39e80b5cb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009830 tx=0x7f39e002a530 comp rx=0 tx=0).stop 2024-06-29T14:22:59.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.254+0000 7f39efb9e640 1 -- 172.21.15.59:0/3813107350 shutdown_connections 2024-06-29T14:22:59.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.254+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3813107350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39e80d1130 0x7f39e815f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.254+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3813107350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e8145650 0x7f39e80b5cb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.254+0000 7f39efb9e640 1 -- 172.21.15.59:0/3813107350 >> 172.21.15.59:0/3813107350 conn(0x7f39e80b6ed0 msgr2=0x7f39e80b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.254+0000 7f39efb9e640 1 -- 172.21.15.59:0/3813107350 shutdown_connections 2024-06-29T14:22:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.254+0000 7f39efb9e640 1 -- 172.21.15.59:0/3813107350 wait complete. 2024-06-29T14:22:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.256+0000 7f39efb9e640 1 Processor -- start 2024-06-29T14:22:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.256+0000 7f39efb9e640 1 -- start start 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39efb9e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39e80d1130 0x7f39e815b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39efb9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e8145650 0x7f39e80cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39efb9e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e815fd30 0x7f39e80cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39efb9e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f39e815c910 con 0x7f39e815fd30 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39efb9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39e815eba0 con 0x7f39e8145650 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39efb9e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f39e815ca90 con 0x7f39e80d1130 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39ee13b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e815fd30 0x7f39e80cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39ee13b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e815fd30 0x7f39e80cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40082/0 (socket says 172.21.15.59:40082) 2024-06-29T14:22:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39ee13b640 1 -- 172.21.15.59:0/3208734349 learned_addr learned my addr 172.21.15.59:0/3208734349 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39ed139640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e8145650 0x7f39e80cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.257+0000 7f39ed93a640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39e80d1130 0x7f39e815b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39ee13b640 1 -- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39e80d1130 msgr2=0x7f39e815b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39ee13b640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39e80d1130 0x7f39e815b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39ee13b640 1 -- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e8145650 msgr2=0x7f39e80cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39ee13b640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e8145650 0x7f39e80cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39ee13b640 1 -- 172.21.15.59:0/3208734349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e80ca200 con 0x7f39e815fd30 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39ed93a640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39e80d1130 0x7f39e815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39ed139640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e8145650 0x7f39e80cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39ee13b640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e815fd30 0x7f39e80cc8b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f39d800c730 tx=0x7f39d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39e67fc640 1 -- 172.21.15.59:0/3208734349 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d8003b70 con 0x7f39e815fd30 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39e67fc640 1 -- 172.21.15.59:0/3208734349 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39d8003d30 con 0x7f39e815fd30 2024-06-29T14:22:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e80cb890 con 0x7f39e815fd30 2024-06-29T14:22:59.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39e67fc640 1 -- 172.21.15.59:0/3208734349 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d80052d0 con 0x7f39e815fd30 2024-06-29T14:22:59.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.258+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e80cd3f0 con 0x7f39e815fd30 2024-06-29T14:22:59.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.259+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39e80c9f70 con 0x7f39e815fd30 2024-06-29T14:22:59.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.260+0000 7f39e67fc640 1 -- 172.21.15.59:0/3208734349 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f39d8005470 con 0x7f39e815fd30 2024-06-29T14:22:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.260+0000 7f39e67fc640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f39d40762d0 0x7f39d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.261+0000 7f39e67fc640 1 -- 172.21.15.59:0/3208734349 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6383+0+0 (secure 0 0 0) 0x7f39d8097870 con 0x7f39e815fd30 2024-06-29T14:22:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.261+0000 7f39ed93a640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f39d40762d0 0x7f39d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.261+0000 7f39ed93a640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f39d40762d0 0x7f39d40787e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f39e0034000 tx=0x7f39e0004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:59.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.265+0000 7f39e67fc640 1 -- 172.21.15.59:0/3208734349 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f39e8160140 con 0x7f39e815fd30 2024-06-29T14:22:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:22:59.458+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f39e815d010 con 0x7f39e815fd30 2024-06-29T14:23:00.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.109+0000 7f39e67fc640 1 -- 172.21.15.59:0/3208734349 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v31) ==== 169+0+0 (secure 0 0 0) 0x7f39d8060ea0 con 0x7f39e815fd30 2024-06-29T14:23:00.108 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T14:23:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.117+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f39d40762d0 msgr2=0x7f39d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.117+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f39d40762d0 0x7f39d40787e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f39e0034000 tx=0x7f39e0004730 comp rx=0 tx=0).stop 2024-06-29T14:23:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.117+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e815fd30 msgr2=0x7f39e80cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.117+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e815fd30 0x7f39e80cc8b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f39d800c730 tx=0x7f39d800cbf0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.117+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 shutdown_connections 2024-06-29T14:23:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.118+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f39d40762d0 0x7f39d40787e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.118+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39e815fd30 0x7f39e80cc8b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.118+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39e8145650 0x7f39e80cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.118+0000 7f39efb9e640 1 --2- 172.21.15.59:0/3208734349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39e80d1130 0x7f39e815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.118+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 >> 172.21.15.59:0/3208734349 conn(0x7f39e80b6ed0 msgr2=0x7f39e815f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.118+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 shutdown_connections 2024-06-29T14:23:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.118+0000 7f39efb9e640 1 -- 172.21.15.59:0/3208734349 wait complete. 2024-06-29T14:23:00.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-06-29T14:23:00.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.261+0000 7f898084e640 1 -- 172.21.15.59:0/3498543560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b7150 msgr2=0x7f897c0b7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.261+0000 7f898084e640 1 --2- 172.21.15.59:0/3498543560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b7150 0x7f897c0b7560 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8970009830 tx=0x7f897002a510 comp rx=0 tx=0).stop 2024-06-29T14:23:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.261+0000 7f898084e640 1 -- 172.21.15.59:0/3498543560 shutdown_connections 2024-06-29T14:23:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.261+0000 7f898084e640 1 --2- 172.21.15.59:0/3498543560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b7150 0x7f897c0b7560 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.261+0000 7f898084e640 1 --2- 172.21.15.59:0/3498543560 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f897c0b2a50 0x7f897c161380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.261+0000 7f898084e640 1 -- 172.21.15.59:0/3498543560 >> 172.21.15.59:0/3498543560 conn(0x7f897c0b5590 msgr2=0x7f897c0b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.261+0000 7f898084e640 1 -- 172.21.15.59:0/3498543560 shutdown_connections 2024-06-29T14:23:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.261+0000 7f898084e640 1 -- 172.21.15.59:0/3498543560 wait complete. 2024-06-29T14:23:00.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.263+0000 7f898084e640 1 Processor -- start 2024-06-29T14:23:00.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.263+0000 7f898084e640 1 -- start start 2024-06-29T14:23:00.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f898084e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b2a50 0x7f897c150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:00.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f898084e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f897c0b7150 0x7f897c0cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:00.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f898084e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f897c0d2f70 0x7f897c0cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:00.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f898084e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f897c0b4510 con 0x7f897c0b2a50 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f898084e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f897c0b53e0 con 0x7f897c0d2f70 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f898084e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f897c0b4690 con 0x7f897c0b7150 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f897a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b2a50 0x7f897c150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f897a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b2a50 0x7f897c150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40096/0 (socket says 172.21.15.59:40096) 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f897a59c640 1 -- 172.21.15.59:0/3388560202 learned_addr learned my addr 172.21.15.59:0/3388560202 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f8979d9b640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f897c0b7150 0x7f897c0cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.265+0000 7f897ad9d640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f897c0d2f70 0x7f897c0cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.266+0000 7f897a59c640 1 -- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f897c0b7150 msgr2=0x7f897c0cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.266+0000 7f897a59c640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f897c0b7150 0x7f897c0cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.266+0000 7f897a59c640 1 -- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f897c0d2f70 msgr2=0x7f897c0cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:00.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.266+0000 7f897a59c640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f897c0d2f70 0x7f897c0cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:00.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.266+0000 7f897a59c640 1 -- 172.21.15.59:0/3388560202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c15a750 con 0x7f897c0b2a50 2024-06-29T14:23:00.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.266+0000 7f897ad9d640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f897c0d2f70 0x7f897c0cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:00.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.266+0000 7f897a59c640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b2a50 0x7f897c150ca0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f896c00cce0 tx=0x7f896c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:00.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.266+0000 7f896affd640 1 -- 172.21.15.59:0/3388560202 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c00f450 con 0x7f897c0b2a50 2024-06-29T14:23:00.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.267+0000 7f896affd640 1 -- 172.21.15.59:0/3388560202 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f896c00f6e0 con 0x7f897c0b2a50 2024-06-29T14:23:00.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.267+0000 7f896affd640 1 -- 172.21.15.59:0/3388560202 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c00fb10 con 0x7f897c0b2a50 2024-06-29T14:23:00.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.270+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897c15aa30 con 0x7f897c0b2a50 2024-06-29T14:23:00.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.270+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897c0c8f10 con 0x7f897c0b2a50 2024-06-29T14:23:00.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.272+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f897c0b4b80 con 0x7f897c0b2a50 2024-06-29T14:23:00.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.273+0000 7f896affd640 1 -- 172.21.15.59:0/3388560202 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f896c00fcb0 con 0x7f897c0b2a50 2024-06-29T14:23:00.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.274+0000 7f896affd640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f895c0762d0 0x7f895c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:00.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.274+0000 7f896affd640 1 -- 172.21.15.59:0/3388560202 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5853+0+0 (secure 0 0 0) 0x7f896c098300 con 0x7f897c0b2a50 2024-06-29T14:23:00.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.274+0000 7f8979d9b640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f895c0762d0 0x7f895c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:00.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.276+0000 7f8979d9b640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f895c0762d0 0x7f895c0787e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8970034000 tx=0x7f8970004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:00.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.279+0000 7f896affd640 1 -- 172.21.15.59:0/3388560202 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f896c014030 con 0x7f897c0b2a50 2024-06-29T14:23:00.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:00.469+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f897c08ccd0 con 0x7f897c0b2a50 2024-06-29T14:23:01.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.111+0000 7f896affd640 1 -- 172.21.15.59:0/3388560202 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v32) ==== 178+0+0 (secure 0 0 0) 0x7f896c061b40 con 0x7f897c0b2a50 2024-06-29T14:23:01.110 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data_ec' removed 2024-06-29T14:23:01.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f895c0762d0 msgr2=0x7f895c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:01.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f895c0762d0 0x7f895c0787e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8970034000 tx=0x7f8970004730 comp rx=0 tx=0).stop 2024-06-29T14:23:01.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b2a50 msgr2=0x7f897c150ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b2a50 0x7f897c150ca0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f896c00cce0 tx=0x7f896c00a620 comp rx=0 tx=0).stop 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 shutdown_connections 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f895c0762d0 0x7f895c0787e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f897c0d2f70 0x7f897c0cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f897c0b7150 0x7f897c0cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 --2- 172.21.15.59:0/3388560202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c0b2a50 0x7f897c150ca0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 >> 172.21.15.59:0/3388560202 conn(0x7f897c0b5590 msgr2=0x7f897c0d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.116+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 shutdown_connections 2024-06-29T14:23:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:01.117+0000 7f898084e640 1 -- 172.21.15.59:0/3388560202 wait complete. 2024-06-29T14:23:01.132 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:23:01.132 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:23:01.132 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:23:01.133 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:23:01.133 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:23:01.133 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:23:01.133 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:23:01.133 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:23:01.133 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:23:01.133 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:23:01.134 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:23:01.134 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:23:01.134 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:23:01.134 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:23:01.134 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:23:01.134 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:23:01.144 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:23:01.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:23:01.146 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:23:01.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:23:01.147 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:23:01.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:23:01.188 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:23:01.188 DEBUG:teuthology.parallel:result is None 2024-06-29T14:23:01.190 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:23:01.190 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:23:01.191 DEBUG:teuthology.parallel:result is None 2024-06-29T14:23:01.191 DEBUG:teuthology.parallel:result is None 2024-06-29T14:23:01.281 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:23:01.283 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:23:01.283 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:23:02.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-06-29T14:23:07.145 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:23:07.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:23:07.148 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:23:07.149 DEBUG:teuthology.parallel:result is None 2024-06-29T14:23:07.149 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:23:07.251 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:23:07.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.284+0000 7f660cd0a640 1 -- 172.21.15.59:0/1002125911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66080b3990 msgr2=0x7f66080b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.284+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1002125911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66080b3990 0x7f66080b3da0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009830 tx=0x7f65f002a510 comp rx=0 tx=0).stop 2024-06-29T14:23:07.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.284+0000 7f660cd0a640 1 -- 172.21.15.59:0/1002125911 shutdown_connections 2024-06-29T14:23:07.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.284+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1002125911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66080b73f0 0x7f66080b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.284+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1002125911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66080b3990 0x7f66080b3da0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.284+0000 7f660cd0a640 1 -- 172.21.15.59:0/1002125911 >> 172.21.15.59:0/1002125911 conn(0x7f66080d3ee0 msgr2=0x7f66080d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:07.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.284+0000 7f660cd0a640 1 -- 172.21.15.59:0/1002125911 shutdown_connections 2024-06-29T14:23:07.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.284+0000 7f660cd0a640 1 -- 172.21.15.59:0/1002125911 wait complete. 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.286+0000 7f660cd0a640 1 Processor -- start 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.287+0000 7f660cd0a640 1 -- start start 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f660cd0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66080b3990 0x7f66080cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f660cd0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080b73f0 0x7f66080aa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f660cd0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f660815e290 0x7f66080b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f660cd0a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f66080b7d40 con 0x7f660815e290 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f660cd0a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66080cea30 con 0x7f66080b73f0 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f660cd0a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f66080b7ec0 con 0x7f66080b3990 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f660815e290 0x7f66080b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f660815e290 0x7f66080b8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55982/0 (socket says 172.21.15.59:55982) 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 -- 172.21.15.59:0/1131866401 learned_addr learned my addr 172.21.15.59:0/1131866401 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f660659c640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66080b3990 0x7f66080cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6605d9b640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080b73f0 0x7f66080aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 -- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66080b3990 msgr2=0x7f66080cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66080b3990 0x7f66080cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 -- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080b73f0 msgr2=0x7f66080aa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080b73f0 0x7f66080aa470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 -- 172.21.15.59:0/1131866401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66080ab1a0 con 0x7f660815e290 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.288+0000 7f6606d9d640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f660815e290 0x7f66080b8710 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f660000e990 tx=0x7f660000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.289+0000 7f65feffd640 1 -- 172.21.15.59:0/1131866401 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6600004d20 con 0x7f660815e290 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.289+0000 7f65feffd640 1 -- 172.21.15.59:0/1131866401 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f660000cb00 con 0x7f660815e290 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.289+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66080ab480 con 0x7f660815e290 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.289+0000 7f65feffd640 1 -- 172.21.15.59:0/1131866401 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6600005020 con 0x7f660815e290 2024-06-29T14:23:07.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.289+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66080aba40 con 0x7f660815e290 2024-06-29T14:23:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.290+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66080b8420 con 0x7f660815e290 2024-06-29T14:23:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.290+0000 7f65feffd640 1 -- 172.21.15.59:0/1131866401 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f66000051c0 con 0x7f660815e290 2024-06-29T14:23:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.291+0000 7f65feffd640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f65e80762d0 0x7f65e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.291+0000 7f65feffd640 1 -- 172.21.15.59:0/1131866401 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5335+0+0 (secure 0 0 0) 0x7f6600097f80 con 0x7f660815e290 2024-06-29T14:23:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.291+0000 7f660659c640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f65e80762d0 0x7f65e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.292+0000 7f660659c640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f65e80762d0 0x7f65e80787e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f65f8007fd0 tx=0x7f65f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:07.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.296+0000 7f65feffd640 1 -- 172.21.15.59:0/1131866401 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6600016030 con 0x7f660815e290 2024-06-29T14:23:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:07.498+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6608075ad0 con 0x7f660815e290 2024-06-29T14:23:09.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.144+0000 7f65feffd640 1 -- 172.21.15.59:0/1131866401 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v37) ==== 97+0+0 (secure 0 0 0) 0x7f66000619c0 con 0x7f660815e290 2024-06-29T14:23:09.143 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.147+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f65e80762d0 msgr2=0x7f65e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.147+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f65e80762d0 0x7f65e80787e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f65f8007fd0 tx=0x7f65f8009040 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.147+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f660815e290 msgr2=0x7f66080b8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.147+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f660815e290 0x7f66080b8710 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f660000e990 tx=0x7f660000ee50 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.147+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 shutdown_connections 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.147+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f65e80762d0 0x7f65e80787e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.148+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f660815e290 0x7f66080b8710 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.148+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080b73f0 0x7f66080aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.148+0000 7f660cd0a640 1 --2- 172.21.15.59:0/1131866401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66080b3990 0x7f66080cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.148+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 >> 172.21.15.59:0/1131866401 conn(0x7f66080d3ee0 msgr2=0x7f66080ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.148+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 shutdown_connections 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.148+0000 7f660cd0a640 1 -- 172.21.15.59:0/1131866401 wait complete. 2024-06-29T14:23:09.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:23:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.287+0000 7fd09987f640 1 -- 172.21.15.59:0/1166471932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 msgr2=0x7fd08000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.287+0000 7fd09987f640 1 --2- 172.21.15.59:0/1166471932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 0x7fd08000b9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00c260 tx=0x7fd080005830 comp rx=0 tx=0).stop 2024-06-29T14:23:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.287+0000 7fd0937fe640 1 -- 172.21.15.59:0/1166471932 reap_dead start 2024-06-29T14:23:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.287+0000 7fd09987f640 1 -- 172.21.15.59:0/1166471932 shutdown_connections 2024-06-29T14:23:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.287+0000 7fd09987f640 1 -- 172.21.15.59:0/1166471932 >> 172.21.15.59:0/1166471932 conn(0x7fd09406c950 msgr2=0x7fd094073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.287+0000 7fd09987f640 1 -- 172.21.15.59:0/1166471932 shutdown_connections 2024-06-29T14:23:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.288+0000 7fd09987f640 1 -- 172.21.15.59:0/1166471932 wait complete. 2024-06-29T14:23:09.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.289+0000 7fd09987f640 1 Processor -- start 2024-06-29T14:23:09.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.290+0000 7fd09987f640 1 -- start start 2024-06-29T14:23:09.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd09987f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 0x7fd09415ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd09987f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd094139a40 0x7fd09406ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd09987f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09406f120 0x7fd0941f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd09987f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd0940cf870 con 0x7fd080005250 2024-06-29T14:23:09.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd09987f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd094145ab0 con 0x7fd094139a40 2024-06-29T14:23:09.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd09987f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd0940cf9f0 con 0x7fd09406f120 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd092ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 0x7fd09415ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd092ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 0x7fd09415ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55998/0 (socket says 172.21.15.59:55998) 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd092ffd640 1 -- 172.21.15.59:0/3744152616 learned_addr learned my addr 172.21.15.59:0/3744152616 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd0927fc640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd094139a40 0x7fd09406ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd0937fe640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09406f120 0x7fd0941f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd092ffd640 1 -- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09406f120 msgr2=0x7fd0941f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd092ffd640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09406f120 0x7fd0941f7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd092ffd640 1 -- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd094139a40 msgr2=0x7fd09406ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd092ffd640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd094139a40 0x7fd09406ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.291+0000 7fd092ffd640 1 -- 172.21.15.59:0/3744152616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09406db80 con 0x7fd080005250 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.292+0000 7fd092ffd640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 0x7fd09415ba80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd084010bc0 tx=0x7fd08400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.292+0000 7fd08bfff640 1 -- 172.21.15.59:0/3744152616 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084003b70 con 0x7fd080005250 2024-06-29T14:23:09.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.292+0000 7fd08bfff640 1 -- 172.21.15.59:0/3744152616 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd08400ee90 con 0x7fd080005250 2024-06-29T14:23:09.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.292+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09406de20 con 0x7fd080005250 2024-06-29T14:23:09.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.292+0000 7fd08bfff640 1 -- 172.21.15.59:0/3744152616 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084004020 con 0x7fd080005250 2024-06-29T14:23:09.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.292+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09406e3e0 con 0x7fd080005250 2024-06-29T14:23:09.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.293+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd09415c2f0 con 0x7fd080005250 2024-06-29T14:23:09.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.294+0000 7fd08bfff640 1 -- 172.21.15.59:0/3744152616 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd08401a030 con 0x7fd080005250 2024-06-29T14:23:09.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.294+0000 7fd08bfff640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd074076300 0x7fd074078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.294+0000 7fd08bfff640 1 -- 172.21.15.59:0/3744152616 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7fd084098180 con 0x7fd080005250 2024-06-29T14:23:09.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.295+0000 7fd0927fc640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd074076300 0x7fd074078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.295+0000 7fd0927fc640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd074076300 0x7fd074078810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd09406f6e0 tx=0x7fd07c006b70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.299+0000 7fd08bfff640 1 -- 172.21.15.59:0/3744152616 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd084061650 con 0x7fd080005250 2024-06-29T14:23:09.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-06-29T14:23:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.565+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fd0940b1f10 con 0x7fd080005250 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.566+0000 7fd08bfff640 1 -- 172.21.15.59:0/3744152616 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7fd084065950 con 0x7fd080005250 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.565 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:23:09.566 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T14:23:09.571 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T14:23:09.573 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:09.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.575 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:09.576 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T14:23:09.578 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:23:09.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-mds", 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGIBmo4rmDhAAvvUopMTC6gz7b703LI2dBg==", 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-mds" 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-mgr", 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGIBm2pPmDhAAxHp+/K9JjBNNH6rilYwdDA==", 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-mgr" 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-osd", 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGIBmN5zmDhAAIA1Zi+Km/pnDSR7u/3JL3w==", 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-osd" 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-rbd", 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGIBmPaTmDhAARS0PH9HrVSRmUJ7AjYEHkQ==", 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-rbd" 2024-06-29T14:23:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGIBmM6zmDhAAIXV1qHjy0wcZWarUxkDTog==", 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-rgw", 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGIBmVbTmDhAApoGZjHiR9EQNgCLL0DWrnw==", 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-rgw" 2024-06-29T14:23:09.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T14:23:09.584 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:23:09.585 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:23:09.585 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:23:09.585 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:23:09.585 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.585 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:09.585 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T14:23:09.585 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:23:09.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd074076300 msgr2=0x7fd074078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd074076300 0x7fd074078810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd09406f6e0 tx=0x7fd07c006b70 comp rx=0 tx=0).stop 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 msgr2=0x7fd09415ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 0x7fd09415ba80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd084010bc0 tx=0x7fd08400e6c0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 shutdown_connections 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fd074076300 0x7fd074078810 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09406f120 0x7fd0941f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd094139a40 0x7fd09406ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 --2- 172.21.15.59:0/3744152616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080005250 0x7fd09415ba80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 >> 172.21.15.59:0/3744152616 conn(0x7fd09406c950 msgr2=0x7fd09413fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 shutdown_connections 2024-06-29T14:23:09.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.568+0000 7fd09987f640 1 -- 172.21.15.59:0/3744152616 wait complete. 2024-06-29T14:23:09.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-06-29T14:23:09.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.709+0000 7f74cb3a1640 1 -- 172.21.15.59:0/4115829051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74ac004d90 msgr2=0x7f74ac005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.709+0000 7f74cb3a1640 1 --2- 172.21.15.59:0/4115829051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74ac004d90 0x7f74ac005260 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009b00 tx=0x7f74ac005830 comp rx=0 tx=0).stop 2024-06-29T14:23:09.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.709+0000 7f74c993e640 1 -- 172.21.15.59:0/4115829051 reap_dead start 2024-06-29T14:23:09.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.709+0000 7f74cb3a1640 1 -- 172.21.15.59:0/4115829051 shutdown_connections 2024-06-29T14:23:09.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.709+0000 7f74cb3a1640 1 -- 172.21.15.59:0/4115829051 >> 172.21.15.59:0/4115829051 conn(0x7f74c40d3ee0 msgr2=0x7f74c40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.709+0000 7f74cb3a1640 1 -- 172.21.15.59:0/4115829051 shutdown_connections 2024-06-29T14:23:09.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.709+0000 7f74cb3a1640 1 -- 172.21.15.59:0/4115829051 wait complete. 2024-06-29T14:23:09.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.711+0000 7f74cb3a1640 1 Processor -- start 2024-06-29T14:23:09.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.711+0000 7f74cb3a1640 1 -- start start 2024-06-29T14:23:09.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74cb3a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c41f5360 0x7f74c4153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74cb3a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74c4155250 0x7f74c40c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74cb3a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c40cc1f0 0x7f74c40cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74cb3a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f74c415ff00 con 0x7f74c40cc1f0 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74cb3a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74c415fd80 con 0x7f74c41f5360 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74cb3a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f74c4160080 con 0x7f74c4155250 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74c993e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c40cc1f0 0x7f74c40cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74c993e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c40cc1f0 0x7f74c40cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56024/0 (socket says 172.21.15.59:56024) 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.712+0000 7f74c993e640 1 -- 172.21.15.59:0/1962192063 learned_addr learned my addr 172.21.15.59:0/1962192063 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c913d640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c41f5360 0x7f74c4153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c893c640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74c4155250 0x7f74c40c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c993e640 1 -- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74c4155250 msgr2=0x7f74c40c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c993e640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74c4155250 0x7f74c40c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c993e640 1 -- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c41f5360 msgr2=0x7f74c4153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c993e640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c41f5360 0x7f74c4153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c993e640 1 -- 172.21.15.59:0/1962192063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c40ccb40 con 0x7f74c40cc1f0 2024-06-29T14:23:09.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c913d640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c41f5360 0x7f74c4153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:09.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c993e640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c40cc1f0 0x7f74c40cc600 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f74b402b460 tx=0x7f74b4004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c1ffb640 1 -- 172.21.15.59:0/1962192063 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4009700 con 0x7f74c40cc1f0 2024-06-29T14:23:09.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c1ffb640 1 -- 172.21.15.59:0/1962192063 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74b4043070 con 0x7f74c40cc1f0 2024-06-29T14:23:09.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74c1ffb640 1 -- 172.21.15.59:0/1962192063 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b402bd20 con 0x7f74c40cc1f0 2024-06-29T14:23:09.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c4150e70 con 0x7f74c40cc1f0 2024-06-29T14:23:09.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.713+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c4151430 con 0x7f74c40cc1f0 2024-06-29T14:23:09.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.715+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c4160610 con 0x7f74c40cc1f0 2024-06-29T14:23:09.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.715+0000 7f74c1ffb640 1 -- 172.21.15.59:0/1962192063 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f74b4004fe0 con 0x7f74c40cc1f0 2024-06-29T14:23:09.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.716+0000 7f74c1ffb640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f74b00762d0 0x7f74b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.716+0000 7f74c1ffb640 1 -- 172.21.15.59:0/1962192063 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7f74b40b8300 con 0x7f74c40cc1f0 2024-06-29T14:23:09.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.716+0000 7f74c913d640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f74b00762d0 0x7f74b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.716+0000 7f74c913d640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f74b00762d0 0x7f74b00787e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006d10 tx=0x7f74bc006e60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.720+0000 7f74c1ffb640 1 -- 172.21.15.59:0/1962192063 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f74c40ccb40 con 0x7f74c40cc1f0 2024-06-29T14:23:09.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.992+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f74c41571a0 con 0x7f74c40cc1f0 2024-06-29T14:23:09.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.994+0000 7f74c1ffb640 1 -- 172.21.15.59:0/1962192063 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f74b4081eb0 con 0x7f74c40cc1f0 2024-06-29T14:23:09.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f74b00762d0 msgr2=0x7f74b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f74b00762d0 0x7f74b00787e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006d10 tx=0x7f74bc006e60 comp rx=0 tx=0).stop 2024-06-29T14:23:09.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c40cc1f0 msgr2=0x7f74c40cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c40cc1f0 0x7f74c40cc600 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f74b402b460 tx=0x7f74b4004f70 comp rx=0 tx=0).stop 2024-06-29T14:23:09.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 shutdown_connections 2024-06-29T14:23:09.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f74b00762d0 0x7f74b00787e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c40cc1f0 0x7f74c40cc600 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f74c4155250 0x7f74c40c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 --2- 172.21.15.59:0/1962192063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74c41f5360 0x7f74c4153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 >> 172.21.15.59:0/1962192063 conn(0x7f74c40d3ee0 msgr2=0x7f74c40d3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 shutdown_connections 2024-06-29T14:23:09.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:09.997+0000 7f74cb3a1640 1 -- 172.21.15.59:0/1962192063 wait complete. 2024-06-29T14:23:10.007 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-06-29T14:23:10.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.126+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/1665297309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 msgr2=0x7f4d780b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.126+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/1665297309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 0x7f4d780b1b00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4d78072d60 tx=0x7f4d6802a560 comp rx=0 tx=0).stop 2024-06-29T14:23:10.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.126+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/1665297309 shutdown_connections 2024-06-29T14:23:10.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.126+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/1665297309 >> v2:172.21.15.153:3300/0 conn(0x7f4d7806ea10 0x7f4d780cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.126+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/1665297309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 0x7f4d780b1b00 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.126+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/1665297309 >> 172.21.15.59:0/1665297309 conn(0x7f4d780b06e0 msgr2=0x7f4d780b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.126+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/1665297309 shutdown_connections 2024-06-29T14:23:10.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.126+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/1665297309 wait complete. 2024-06-29T14:23:10.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.128+0000 7f4d7e5ef640 1 Processor -- start 2024-06-29T14:23:10.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.128+0000 7f4d7e5ef640 1 -- start start 2024-06-29T14:23:10.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7e5ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7806ea10 0x7f4d78140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7e5ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d780b2040 0x7f4d78141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7e5ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 0x7f4d78155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7e5ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4d780cfaa0 con 0x7f4d78144470 2024-06-29T14:23:10.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7e5ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4d780cf920 con 0x7f4d7806ea10 2024-06-29T14:23:10.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7e5ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4d780cfc20 con 0x7f4d780b2040 2024-06-29T14:23:10.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 0x7f4d78155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 0x7f4d78155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56040/0 (socket says 172.21.15.59:56040) 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 -- 172.21.15.59:0/3412880031 learned_addr learned my addr 172.21.15.59:0/3412880031 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d77fff640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7806ea10 0x7f4d78140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d777fe640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d780b2040 0x7f4d78141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 -- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d780b2040 msgr2=0x7f4d78141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d780b2040 0x7f4d78141300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 -- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7806ea10 msgr2=0x7f4d78140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7806ea10 0x7f4d78140dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 -- 172.21.15.59:0/3412880031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7815a840 con 0x7f4d78144470 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d777fe640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d780b2040 0x7f4d78141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7cb8c640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 0x7f4d78155940 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000b9e0 tx=0x7f4d7000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d74ff9640 1 -- 172.21.15.59:0/3412880031 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70019070 con 0x7f4d78144470 2024-06-29T14:23:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d74ff9640 1 -- 172.21.15.59:0/3412880031 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d7000cc30 con 0x7f4d78144470 2024-06-29T14:23:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7813f670 con 0x7f4d78144470 2024-06-29T14:23:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.130+0000 7f4d74ff9640 1 -- 172.21.15.59:0/3412880031 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d700146a0 con 0x7f4d78144470 2024-06-29T14:23:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.131+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7813fc30 con 0x7f4d78144470 2024-06-29T14:23:10.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.131+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d780d01c0 con 0x7f4d78144470 2024-06-29T14:23:10.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.132+0000 7f4d74ff9640 1 -- 172.21.15.59:0/3412880031 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4d7001c070 con 0x7f4d78144470 2024-06-29T14:23:10.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.133+0000 7f4d74ff9640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4d640762d0 0x7f4d640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.133+0000 7f4d74ff9640 1 -- 172.21.15.59:0/3412880031 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7f4d70097260 con 0x7f4d78144470 2024-06-29T14:23:10.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.133+0000 7f4d77fff640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4d640762d0 0x7f4d640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.134+0000 7f4d77fff640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4d640762d0 0x7f4d640787e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4d68033000 tx=0x7f4d68004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.137+0000 7f4d74ff9640 1 -- 172.21.15.59:0/3412880031 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d7000ce20 con 0x7f4d78144470 2024-06-29T14:23:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.408+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f4d7815ac50 con 0x7f4d78144470 2024-06-29T14:23:10.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.411+0000 7f4d74ff9640 1 -- 172.21.15.59:0/3412880031 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f4d70060730 con 0x7f4d78144470 2024-06-29T14:23:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4d640762d0 msgr2=0x7f4d640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4d640762d0 0x7f4d640787e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4d68033000 tx=0x7f4d68004730 comp rx=0 tx=0).stop 2024-06-29T14:23:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 msgr2=0x7f4d78155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 0x7f4d78155940 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000b9e0 tx=0x7f4d7000bea0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 shutdown_connections 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f4d640762d0 0x7f4d640787e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d78144470 0x7f4d78155940 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d780b2040 0x7f4d78141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 --2- 172.21.15.59:0/3412880031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d7806ea10 0x7f4d78140dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 >> 172.21.15.59:0/3412880031 conn(0x7f4d780b06e0 msgr2=0x7f4d7815ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 shutdown_connections 2024-06-29T14:23:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.413+0000 7f4d7e5ef640 1 -- 172.21.15.59:0/3412880031 wait complete. 2024-06-29T14:23:10.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-06-29T14:23:10.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.543+0000 7efe55a40640 1 -- 172.21.15.59:0/1668612660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c004800 msgr2=0x7efe3c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.543+0000 7efe55a40640 1 --2- 172.21.15.59:0/1668612660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c004800 0x7efe3c004c10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7efe38006fd0 tx=0x7efe3c005830 comp rx=0 tx=0).stop 2024-06-29T14:23:10.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.543+0000 7efe55a40640 1 -- 172.21.15.59:0/1668612660 shutdown_connections 2024-06-29T14:23:10.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.543+0000 7efe55a40640 1 --2- 172.21.15.59:0/1668612660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c004800 0x7efe3c004c10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.543+0000 7efe55a40640 1 -- 172.21.15.59:0/1668612660 >> 172.21.15.59:0/1668612660 conn(0x7efe500b24c0 msgr2=0x7efe500cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.543+0000 7efe55a40640 1 -- 172.21.15.59:0/1668612660 shutdown_connections 2024-06-29T14:23:10.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.543+0000 7efe55a40640 1 -- 172.21.15.59:0/1668612660 wait complete. 2024-06-29T14:23:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.545+0000 7efe55a40640 1 Processor -- start 2024-06-29T14:23:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.545+0000 7efe55a40640 1 -- start start 2024-06-29T14:23:10.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.546+0000 7efe55a40640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c004800 0x7efe50137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.546+0000 7efe55a40640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5015a620 0x7efe501f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe55a40640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501f3050 0x7efe5013d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe55a40640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efe5015f5d0 con 0x7efe5015a620 2024-06-29T14:23:10.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe55a40640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe5015f450 con 0x7efe501f3050 2024-06-29T14:23:10.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe55a40640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efe5015f750 con 0x7efe3c004800 2024-06-29T14:23:10.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5015a620 0x7efe501f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5015a620 0x7efe501f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56056/0 (socket says 172.21.15.59:56056) 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 -- 172.21.15.59:0/2194638547 learned_addr learned my addr 172.21.15.59:0/2194638547 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe4effd640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c004800 0x7efe50137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe4f7fe640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501f3050 0x7efe5013d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 -- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c004800 msgr2=0x7efe50137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c004800 0x7efe50137230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 -- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501f3050 msgr2=0x7efe5013d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501f3050 0x7efe5013d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 -- 172.21.15.59:0/2194638547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5013aea0 con 0x7efe5015a620 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe47fff640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5015a620 0x7efe501f6680 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efe3800c350 tx=0x7efe38006c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe4cff9640 1 -- 172.21.15.59:0/2194638547 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38031040 con 0x7efe5015a620 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe4cff9640 1 -- 172.21.15.59:0/2194638547 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe380035f0 con 0x7efe5015a620 2024-06-29T14:23:10.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe501f6bc0 con 0x7efe5015a620 2024-06-29T14:23:10.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe4cff9640 1 -- 172.21.15.59:0/2194638547 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38003a00 con 0x7efe5015a620 2024-06-29T14:23:10.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.547+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe5013b140 con 0x7efe5015a620 2024-06-29T14:23:10.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.549+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe5006d6d0 con 0x7efe5015a620 2024-06-29T14:23:10.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.549+0000 7efe4cff9640 1 -- 172.21.15.59:0/2194638547 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efe38034070 con 0x7efe5015a620 2024-06-29T14:23:10.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.550+0000 7efe4cff9640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efe300762d0 0x7efe300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.550+0000 7efe4cff9640 1 -- 172.21.15.59:0/2194638547 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7efe380b7610 con 0x7efe5015a620 2024-06-29T14:23:10.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.550+0000 7efe4effd640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efe300762d0 0x7efe300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.550+0000 7efe4effd640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efe300762d0 0x7efe300787e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efe40004630 tx=0x7efe4000b260 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.554+0000 7efe4cff9640 1 -- 172.21.15.59:0/2194638547 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efe501f6bc0 con 0x7efe5015a620 2024-06-29T14:23:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.822+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7efe5013b380 con 0x7efe5015a620 2024-06-29T14:23:10.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.824+0000 7efe4cff9640 1 -- 172.21.15.59:0/2194638547 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7efe5013aea0 con 0x7efe5015a620 2024-06-29T14:23:10.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efe300762d0 msgr2=0x7efe300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efe300762d0 0x7efe300787e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efe40004630 tx=0x7efe4000b260 comp rx=0 tx=0).stop 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5015a620 msgr2=0x7efe501f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5015a620 0x7efe501f6680 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efe3800c350 tx=0x7efe38006c80 comp rx=0 tx=0).stop 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 shutdown_connections 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efe300762d0 0x7efe300787e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe501f3050 0x7efe5013d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5015a620 0x7efe501f6680 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 --2- 172.21.15.59:0/2194638547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c004800 0x7efe50137230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 >> 172.21.15.59:0/2194638547 conn(0x7efe500b24c0 msgr2=0x7efe5015e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 shutdown_connections 2024-06-29T14:23:10.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.826+0000 7efe55a40640 1 -- 172.21.15.59:0/2194638547 wait complete. 2024-06-29T14:23:10.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-06-29T14:23:10.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.957+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/932117446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 msgr2=0x7f7eec003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.957+0000 7f7f0a4e4640 1 --2- 172.21.15.59:0/932117446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 0x7f7eec003f10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7f040d1600 tx=0x7f7eec005830 comp rx=0 tx=0).stop 2024-06-29T14:23:10.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.957+0000 7f7f08a81640 1 -- 172.21.15.59:0/932117446 reap_dead start 2024-06-29T14:23:10.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.957+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/932117446 shutdown_connections 2024-06-29T14:23:10.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.957+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/932117446 >> 172.21.15.59:0/932117446 conn(0x7f7f040d3ee0 msgr2=0x7f7f040d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.957+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/932117446 shutdown_connections 2024-06-29T14:23:10.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.957+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/932117446 wait complete. 2024-06-29T14:23:10.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.959+0000 7f7f0a4e4640 1 Processor -- start 2024-06-29T14:23:10.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.959+0000 7f7f0a4e4640 1 -- start start 2024-06-29T14:23:10.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f0a4e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 0x7f7f04158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f0a4e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f041ea150 0x7f7f041ea560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f0a4e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0415d090 0x7f7f0415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f0a4e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7f040b3d20 con 0x7f7eec003aa0 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f0a4e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f040b3ba0 con 0x7f7f0415d090 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f0a4e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7f041600c0 con 0x7f7f041ea150 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f03fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 0x7f7f04158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f03fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 0x7f7f04158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56070/0 (socket says 172.21.15.59:56070) 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f03fff640 1 -- 172.21.15.59:0/971743650 learned_addr learned my addr 172.21.15.59:0/971743650 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f08a81640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0415d090 0x7f7f0415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.960+0000 7f7f037fe640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f041ea150 0x7f7f041ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f03fff640 1 -- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f041ea150 msgr2=0x7f7f041ea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f03fff640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f041ea150 0x7f7f041ea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f03fff640 1 -- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0415d090 msgr2=0x7f7f0415d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f03fff640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0415d090 0x7f7f0415d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f03fff640 1 -- 172.21.15.59:0/971743650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0415d9e0 con 0x7f7eec003aa0 2024-06-29T14:23:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f03fff640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 0x7f7f04158130 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4036040 tx=0x7f7ef4005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f00ff9640 1 -- 172.21.15.59:0/971743650 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4004230 con 0x7f7eec003aa0 2024-06-29T14:23:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f00ff9640 1 -- 172.21.15.59:0/971743650 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ef4004a70 con 0x7f7eec003aa0 2024-06-29T14:23:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f041eaf20 con 0x7f7eec003aa0 2024-06-29T14:23:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f00ff9640 1 -- 172.21.15.59:0/971743650 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4004ec0 con 0x7f7eec003aa0 2024-06-29T14:23:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.961+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f041e8410 con 0x7f7eec003aa0 2024-06-29T14:23:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.962+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f04160620 con 0x7f7eec003aa0 2024-06-29T14:23:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.963+0000 7f7f00ff9640 1 -- 172.21.15.59:0/971743650 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7ef4005060 con 0x7f7eec003aa0 2024-06-29T14:23:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.964+0000 7f7f00ff9640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7ef0076300 0x7f7ef0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.964+0000 7f7f00ff9640 1 -- 172.21.15.59:0/971743650 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7f7ef40b7e30 con 0x7f7eec003aa0 2024-06-29T14:23:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.964+0000 7f7f037fe640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7ef0076300 0x7f7ef0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.965+0000 7f7f037fe640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7ef0076300 0x7f7ef0078810 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7f040b9300 tx=0x7f7ef8006b70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:10.968+0000 7f7f00ff9640 1 -- 172.21.15.59:0/971743650 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7f0415d9e0 con 0x7f7eec003aa0 2024-06-29T14:23:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.236+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f7f040d2950 con 0x7f7eec003aa0 2024-06-29T14:23:11.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.239+0000 7f7f00ff9640 1 -- 172.21.15.59:0/971743650 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f7ef402f070 con 0x7f7eec003aa0 2024-06-29T14:23:11.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7ef0076300 msgr2=0x7f7ef0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7ef0076300 0x7f7ef0078810 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7f040b9300 tx=0x7f7ef8006b70 comp rx=0 tx=0).stop 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 msgr2=0x7f7f04158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 0x7f7f04158130 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4036040 tx=0x7f7ef4005590 comp rx=0 tx=0).stop 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 shutdown_connections 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7ef0076300 0x7f7ef0078810 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0415d090 0x7f7f0415d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f041ea150 0x7f7f041ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 --2- 172.21.15.59:0/971743650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec003aa0 0x7f7f04158130 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 >> 172.21.15.59:0/971743650 conn(0x7f7f040d3ee0 msgr2=0x7f7f0406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 shutdown_connections 2024-06-29T14:23:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.242+0000 7f7f0a4e4640 1 -- 172.21.15.59:0/971743650 wait complete. 2024-06-29T14:23:11.252 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-06-29T14:23:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.371+0000 7ff660c5e640 1 -- 172.21.15.59:0/2851374902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 msgr2=0x7ff65c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.371+0000 7ff660c5e640 1 --2- 172.21.15.59:0/2851374902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 0x7ff65c15e8d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0d15d0 tx=0x7ff65002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.371+0000 7ff660c5e640 1 -- 172.21.15.59:0/2851374902 shutdown_connections 2024-06-29T14:23:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.371+0000 7ff660c5e640 1 --2- 172.21.15.59:0/2851374902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 0x7ff65c15e8d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.371+0000 7ff660c5e640 1 --2- 172.21.15.59:0/2851374902 >> v2:172.21.15.153:3300/0 conn(0x7ff65c0d23c0 0x7ff65c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.371+0000 7ff660c5e640 1 -- 172.21.15.59:0/2851374902 >> 172.21.15.59:0/2851374902 conn(0x7ff65c0d3ee0 msgr2=0x7ff65c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.372+0000 7ff660c5e640 1 -- 172.21.15.59:0/2851374902 shutdown_connections 2024-06-29T14:23:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.372+0000 7ff660c5e640 1 -- 172.21.15.59:0/2851374902 wait complete. 2024-06-29T14:23:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.373+0000 7ff660c5e640 1 Processor -- start 2024-06-29T14:23:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.374+0000 7ff660c5e640 1 -- start start 2024-06-29T14:23:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff660c5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 0x7ff65c15de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff660c5e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff65c0d23c0 0x7ff65c1e85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff660c5e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff65c15ee10 0x7ff65c1e8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff660c5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c160240 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff660c5e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c1600c0 con 0x7ff65c15ee10 2024-06-29T14:23:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff660c5e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff65c1603c0 con 0x7ff65c0d23c0 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 0x7ff65c15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 0x7ff65c15de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56084/0 (socket says 172.21.15.59:56084) 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65a59c640 1 -- 172.21.15.59:0/28627364 learned_addr learned my addr 172.21.15.59:0/28627364 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65ad9d640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff65c15ee10 0x7ff65c1e8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff659d9b640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff65c0d23c0 0x7ff65c1e85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65a59c640 1 -- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff65c0d23c0 msgr2=0x7ff65c1e85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65a59c640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff65c0d23c0 0x7ff65c1e85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65a59c640 1 -- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff65c15ee10 msgr2=0x7ff65c1e8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65a59c640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff65c15ee10 0x7ff65c1e8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.375+0000 7ff65a59c640 1 -- 172.21.15.59:0/28627364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c0cb960 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.376+0000 7ff65a59c640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 0x7ff65c15de70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009fd0 tx=0x7ff64c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.376+0000 7ff64affd640 1 -- 172.21.15.59:0/28627364 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c00ccc0 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.376+0000 7ff64affd640 1 -- 172.21.15.59:0/28627364 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff64c00ee70 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.376+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65c0cbc40 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.376+0000 7ff64affd640 1 -- 172.21.15.59:0/28627364 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c00ce60 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.376+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65c0cc200 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.377+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65c15e8d0 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.378+0000 7ff64affd640 1 -- 172.21.15.59:0/28627364 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff64c004280 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.378+0000 7ff64affd640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff63c0762d0 0x7ff63c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.378+0000 7ff64affd640 1 -- 172.21.15.59:0/28627364 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7ff64c097f30 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.378+0000 7ff659d9b640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff63c0762d0 0x7ff63c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.379+0000 7ff659d9b640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff63c0762d0 0x7ff63c0787e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff650033000 tx=0x7ff650004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.383+0000 7ff64affd640 1 -- 172.21.15.59:0/28627364 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff64c0619c0 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.653+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7ff65c0d2950 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.655+0000 7ff64affd640 1 -- 172.21.15.59:0/28627364 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7ff64c0612c0 con 0x7ff65c0b3ed0 2024-06-29T14:23:11.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff63c0762d0 msgr2=0x7ff63c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff63c0762d0 0x7ff63c0787e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff650033000 tx=0x7ff650004730 comp rx=0 tx=0).stop 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 msgr2=0x7ff65c15de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 0x7ff65c15de70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009fd0 tx=0x7ff64c00ec20 comp rx=0 tx=0).stop 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 shutdown_connections 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7ff63c0762d0 0x7ff63c0787e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff65c15ee10 0x7ff65c1e8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff65c0d23c0 0x7ff65c1e85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 --2- 172.21.15.59:0/28627364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff65c0b3ed0 0x7ff65c15de70 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 >> 172.21.15.59:0/28627364 conn(0x7ff65c0d3ee0 msgr2=0x7ff65c06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.657+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 shutdown_connections 2024-06-29T14:23:11.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.658+0000 7ff660c5e640 1 -- 172.21.15.59:0/28627364 wait complete. 2024-06-29T14:23:11.668 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-06-29T14:23:11.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.787+0000 7f3e906b3640 1 -- 172.21.15.59:0/676875876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 msgr2=0x7f3e880ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.787+0000 7f3e906b3640 1 --2- 172.21.15.59:0/676875876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 0x7f3e880ce110 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3e84005fd0 tx=0x7f3e8402a580 comp rx=0 tx=0).stop 2024-06-29T14:23:11.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.787+0000 7f3e906b3640 1 -- 172.21.15.59:0/676875876 shutdown_connections 2024-06-29T14:23:11.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.787+0000 7f3e906b3640 1 --2- 172.21.15.59:0/676875876 >> v2:172.21.15.153:3300/0 conn(0x7f3e880ce650 0x7f3e880b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.787+0000 7f3e906b3640 1 --2- 172.21.15.59:0/676875876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 0x7f3e880ce110 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.787+0000 7f3e906b3640 1 -- 172.21.15.59:0/676875876 >> 172.21.15.59:0/676875876 conn(0x7f3e880d3ee0 msgr2=0x7f3e880d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.787+0000 7f3e906b3640 1 -- 172.21.15.59:0/676875876 shutdown_connections 2024-06-29T14:23:11.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.788+0000 7f3e906b3640 1 -- 172.21.15.59:0/676875876 wait complete. 2024-06-29T14:23:11.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.789+0000 7f3e906b3640 1 Processor -- start 2024-06-29T14:23:11.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.789+0000 7f3e906b3640 1 -- start start 2024-06-29T14:23:11.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e906b3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e880b3ed0 0x7f3e881486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e906b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e880ce650 0x7f3e881e86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e906b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 0x7f3e88147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e906b3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3e880d1f20 con 0x7f3e8815e290 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e906b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e880d1da0 con 0x7f3e880ce650 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e906b3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3e880d20a0 con 0x7f3e880b3ed0 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 0x7f3e88147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 0x7f3e88147330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56104/0 (socket says 172.21.15.59:56104) 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 -- 172.21.15.59:0/1273455119 learned_addr learned my addr 172.21.15.59:0/1273455119 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8dc4e640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e880ce650 0x7f3e881e86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8e44f640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e880b3ed0 0x7f3e881486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 -- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e880b3ed0 msgr2=0x7f3e881486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e880b3ed0 0x7f3e881486b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 -- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e880ce650 msgr2=0x7f3e881e86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e880ce650 0x7f3e881e86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 -- 172.21.15.59:0/1273455119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e88148bf0 con 0x7f3e8815e290 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e8ec50640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 0x7f3e88147330 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000ba10 tx=0x7f3e8000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.791+0000 7f3e7effd640 1 -- 172.21.15.59:0/1273455119 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80019070 con 0x7f3e8815e290 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.792+0000 7f3e7effd640 1 -- 172.21.15.59:0/1273455119 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e8000cc30 con 0x7f3e8815e290 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.792+0000 7f3e7effd640 1 -- 172.21.15.59:0/1273455119 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000ce60 con 0x7f3e8815e290 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.792+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e88146290 con 0x7f3e8815e290 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.792+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e88146850 con 0x7f3e8815e290 2024-06-29T14:23:11.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.793+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e88146000 con 0x7f3e8815e290 2024-06-29T14:23:11.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.793+0000 7f3e7effd640 1 -- 172.21.15.59:0/1273455119 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3e8001c070 con 0x7f3e8815e290 2024-06-29T14:23:11.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.794+0000 7f3e7effd640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e6c076300 0x7f3e6c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.794+0000 7f3e7effd640 1 -- 172.21.15.59:0/1273455119 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7f3e80097220 con 0x7f3e8815e290 2024-06-29T14:23:11.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.794+0000 7f3e8e44f640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e6c076300 0x7f3e6c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.795+0000 7f3e8e44f640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e6c076300 0x7f3e6c078810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3e84033000 tx=0x7f3e84004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:11.798+0000 7f3e7effd640 1 -- 172.21.15.59:0/1273455119 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3e80060cb0 con 0x7f3e8815e290 2024-06-29T14:23:12.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.062+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f3e880b9280 con 0x7f3e8815e290 2024-06-29T14:23:12.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.065+0000 7f3e7effd640 1 -- 172.21.15.59:0/1273455119 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f3e880b9280 con 0x7f3e8815e290 2024-06-29T14:23:12.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e6c076300 msgr2=0x7f3e6c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e6c076300 0x7f3e6c078810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3e84033000 tx=0x7f3e84004730 comp rx=0 tx=0).stop 2024-06-29T14:23:12.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 msgr2=0x7f3e88147330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 0x7f3e88147330 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000ba10 tx=0x7f3e8000bed0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 shutdown_connections 2024-06-29T14:23:12.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e6c076300 0x7f3e6c078810 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e8815e290 0x7f3e88147330 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e880ce650 0x7f3e881e86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 --2- 172.21.15.59:0/1273455119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e880b3ed0 0x7f3e881486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 >> 172.21.15.59:0/1273455119 conn(0x7f3e880d3ee0 msgr2=0x7f3e880d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 shutdown_connections 2024-06-29T14:23:12.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.067+0000 7f3e906b3640 1 -- 172.21.15.59:0/1273455119 wait complete. 2024-06-29T14:23:12.077 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.196+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2000824349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf50004800 msgr2=0x7fcf50004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.196+0000 7fcf6ce1f640 1 --2- 172.21.15.59:0/2000824349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf50004800 0x7fcf50004c10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009b50 tx=0x7fcf50005830 comp rx=0 tx=0).stop 2024-06-29T14:23:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.196+0000 7fcf66d9d640 1 -- 172.21.15.59:0/2000824349 reap_dead start 2024-06-29T14:23:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.197+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2000824349 shutdown_connections 2024-06-29T14:23:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.197+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2000824349 >> 172.21.15.59:0/2000824349 conn(0x7fcf680b1ea0 msgr2=0x7fcf680b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.197+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2000824349 shutdown_connections 2024-06-29T14:23:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.197+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2000824349 wait complete. 2024-06-29T14:23:12.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.199+0000 7fcf6ce1f640 1 Processor -- start 2024-06-29T14:23:12.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.199+0000 7fcf6ce1f640 1 -- start start 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf6ce1f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf50004800 0x7fcf6814ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf6ce1f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6806e2c0 0x7fcf681f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf6ce1f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf6806e6d0 0x7fcf681f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf6ce1f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6815be90 con 0x7fcf6806e2c0 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf6ce1f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf680cbc70 con 0x7fcf6806e6d0 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf6ce1f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcf6815c010 con 0x7fcf50004800 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6806e2c0 0x7fcf681f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6806e2c0 0x7fcf681f61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56120/0 (socket says 172.21.15.59:56120) 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 -- 172.21.15.59:0/2847046611 learned_addr learned my addr 172.21.15.59:0/2847046611 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf66d9d640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf6806e6d0 0x7fcf681f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 -- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf50004800 msgr2=0x7fcf6814ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf6659c640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf50004800 0x7fcf6814ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf50004800 0x7fcf6814ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 -- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf6806e6d0 msgr2=0x7fcf681f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf6806e6d0 0x7fcf681f66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 -- 172.21.15.59:0/2847046611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6814fd50 con 0x7fcf6806e2c0 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf66d9d640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf6806e6d0 0x7fcf681f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf6659c640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf50004800 0x7fcf6814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.200+0000 7fcf65d9b640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6806e2c0 0x7fcf681f61a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009c70 tx=0x7fcf5c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.201+0000 7fcf56ffd640 1 -- 172.21.15.59:0/2847046611 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c038040 con 0x7fcf6806e2c0 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.201+0000 7fcf56ffd640 1 -- 172.21.15.59:0/2847046611 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf5c004020 con 0x7fcf6806e2c0 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.201+0000 7fcf56ffd640 1 -- 172.21.15.59:0/2847046611 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c004470 con 0x7fcf6806e2c0 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.201+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6809cd80 con 0x7fcf6806e2c0 2024-06-29T14:23:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.201+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf6809d340 con 0x7fcf6806e2c0 2024-06-29T14:23:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.202+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf681f7020 con 0x7fcf6806e2c0 2024-06-29T14:23:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.202+0000 7fcf56ffd640 1 -- 172.21.15.59:0/2847046611 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcf5c045020 con 0x7fcf6806e2c0 2024-06-29T14:23:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.203+0000 7fcf56ffd640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcf480762d0 0x7fcf480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.203+0000 7fcf56ffd640 1 -- 172.21.15.59:0/2847046611 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7fcf5c0b81b0 con 0x7fcf6806e2c0 2024-06-29T14:23:12.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.203+0000 7fcf6659c640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcf480762d0 0x7fcf480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.204+0000 7fcf6659c640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcf480762d0 0x7fcf480787e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcf680ceab0 tx=0x7fcf580067d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.207+0000 7fcf56ffd640 1 -- 172.21.15.59:0/2847046611 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcf5c081bc0 con 0x7fcf6806e2c0 2024-06-29T14:23:12.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.400+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf680caf40 con 0x7fcf6806e2c0 2024-06-29T14:23:12.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.401+0000 7fcf56ffd640 1 -- 172.21.15.59:0/2847046611 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+13176 (secure 0 0 0) 0x7fcf5c0814c0 con 0x7fcf6806e2c0 2024-06-29T14:23:12.400 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:12.400 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":37,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:23:09.143987+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcf480762d0 msgr2=0x7fcf480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcf480762d0 0x7fcf480787e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcf680ceab0 tx=0x7fcf580067d0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6806e2c0 msgr2=0x7fcf681f61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6806e2c0 0x7fcf681f61a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009c70 tx=0x7fcf5c033670 comp rx=0 tx=0).stop 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 shutdown_connections 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fcf480762d0 0x7fcf480787e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf6806e6d0 0x7fcf681f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6806e2c0 0x7fcf681f61a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 --2- 172.21.15.59:0/2847046611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf50004800 0x7fcf6814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.403+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 >> 172.21.15.59:0/2847046611 conn(0x7fcf680b1ea0 msgr2=0x7fcf680b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.404+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 shutdown_connections 2024-06-29T14:23:12.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.404+0000 7fcf6ce1f640 1 -- 172.21.15.59:0/2847046611 wait complete. 2024-06-29T14:23:12.414 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:12.414 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:12.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.535+0000 7f8830aef640 1 -- 172.21.15.59:0/627217969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c0b8be0 msgr2=0x7f882c1629d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.535+0000 7f8830aef640 1 --2- 172.21.15.59:0/627217969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c0b8be0 0x7f882c1629d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f882c0b78c0 tx=0x7f882402aac0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.536+0000 7f8830aef640 1 -- 172.21.15.59:0/627217969 shutdown_connections 2024-06-29T14:23:12.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.536+0000 7f8830aef640 1 --2- 172.21.15.59:0/627217969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c0b8be0 0x7f882c1629d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.536+0000 7f8830aef640 1 -- 172.21.15.59:0/627217969 >> 172.21.15.59:0/627217969 conn(0x7f882c0b5610 msgr2=0x7f882c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.536+0000 7f8830aef640 1 -- 172.21.15.59:0/627217969 shutdown_connections 2024-06-29T14:23:12.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.536+0000 7f8830aef640 1 -- 172.21.15.59:0/627217969 wait complete. 2024-06-29T14:23:12.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.538+0000 7f8830aef640 1 Processor -- start 2024-06-29T14:23:12.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.538+0000 7f8830aef640 1 -- start start 2024-06-29T14:23:12.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f8830aef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c08c190 0x7f882c154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f8830aef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f882c0b8be0 0x7f882c1550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f8830aef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882c0ce950 0x7f882c155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f8830aef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f882c169690 con 0x7f882c08c190 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f8830aef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f882c169510 con 0x7f882c0ce950 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f8830aef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f882c169810 con 0x7f882c0b8be0 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c08c190 0x7f882c154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c08c190 0x7f882c154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56142/0 (socket says 172.21.15.59:56142) 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 -- 172.21.15.59:0/1663725921 learned_addr learned my addr 172.21.15.59:0/1663725921 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f8829d9b640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f882c0b8be0 0x7f882c1550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882ad9d640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882c0ce950 0x7f882c155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 -- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f882c0b8be0 msgr2=0x7f882c1550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f882c0b8be0 0x7f882c1550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 -- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882c0ce950 msgr2=0x7f882c155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882c0ce950 0x7f882c155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 -- 172.21.15.59:0/1663725921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c0cdb70 con 0x7f882c08c190 2024-06-29T14:23:12.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.539+0000 7f882a59c640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c08c190 0x7f882c154ba0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f881c00c370 tx=0x7f881c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.540+0000 7f881affd640 1 -- 172.21.15.59:0/1663725921 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881c017070 con 0x7f882c08c190 2024-06-29T14:23:12.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.540+0000 7f881affd640 1 -- 172.21.15.59:0/1663725921 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f881c0035f0 con 0x7f882c08c190 2024-06-29T14:23:12.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.540+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f882c1527c0 con 0x7f882c08c190 2024-06-29T14:23:12.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.540+0000 7f881affd640 1 -- 172.21.15.59:0/1663725921 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881c003a40 con 0x7f882c08c190 2024-06-29T14:23:12.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.540+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f882c152d80 con 0x7f882c08c190 2024-06-29T14:23:12.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.541+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f882c0b9960 con 0x7f882c08c190 2024-06-29T14:23:12.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.542+0000 7f881affd640 1 -- 172.21.15.59:0/1663725921 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f881c01a070 con 0x7f882c08c190 2024-06-29T14:23:12.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.542+0000 7f881affd640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f880c0762d0 0x7f880c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.542+0000 7f881affd640 1 -- 172.21.15.59:0/1663725921 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7f881c096910 con 0x7f882c08c190 2024-06-29T14:23:12.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.543+0000 7f8829d9b640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f880c0762d0 0x7f880c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.543+0000 7f8829d9b640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f880c0762d0 0x7f880c0787e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f882c161900 tx=0x7f8820009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.547+0000 7f881affd640 1 -- 172.21.15.59:0/1663725921 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f882c0b9960 con 0x7f882c08c190 2024-06-29T14:23:12.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.737+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f882c0cf080 con 0x7f882c08c190 2024-06-29T14:23:12.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.737+0000 7f881affd640 1 -- 172.21.15.59:0/1663725921 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) ==== 93+0+10 (secure 0 0 0) 0x7f881c0603a0 con 0x7f882c08c190 2024-06-29T14:23:12.736 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:23:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f880c0762d0 msgr2=0x7f880c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f880c0762d0 0x7f880c0787e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f882c161900 tx=0x7f8820009040 comp rx=0 tx=0).stop 2024-06-29T14:23:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c08c190 msgr2=0x7f882c154ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c08c190 0x7f882c154ba0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f881c00c370 tx=0x7f881c00c830 comp rx=0 tx=0).stop 2024-06-29T14:23:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 shutdown_connections 2024-06-29T14:23:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f880c0762d0 0x7f880c0787e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f882c0ce950 0x7f882c155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f882c0b8be0 0x7f882c1550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 --2- 172.21.15.59:0/1663725921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f882c08c190 0x7f882c154ba0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 >> 172.21.15.59:0/1663725921 conn(0x7f882c0b5610 msgr2=0x7f882c06d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 shutdown_connections 2024-06-29T14:23:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.739+0000 7f8830aef640 1 -- 172.21.15.59:0/1663725921 wait complete. 2024-06-29T14:23:12.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:23:12.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.868+0000 7f94c529d640 1 -- 172.21.15.59:0/426037972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c00b3740 msgr2=0x7f94c00b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.868+0000 7f94c529d640 1 --2- 172.21.15.59:0/426037972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c00b3740 0x7f94c00b3b50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f94b4005fd0 tx=0x7f94b402a670 comp rx=0 tx=0).stop 2024-06-29T14:23:12.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.868+0000 7f94c529d640 1 -- 172.21.15.59:0/426037972 shutdown_connections 2024-06-29T14:23:12.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.868+0000 7f94c529d640 1 --2- 172.21.15.59:0/426037972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c00b3740 0x7f94c00b3b50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.868+0000 7f94c529d640 1 -- 172.21.15.59:0/426037972 >> 172.21.15.59:0/426037972 conn(0x7f94c006c950 msgr2=0x7f94c0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.868+0000 7f94c529d640 1 -- 172.21.15.59:0/426037972 shutdown_connections 2024-06-29T14:23:12.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.868+0000 7f94c529d640 1 -- 172.21.15.59:0/426037972 wait complete. 2024-06-29T14:23:12.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.870+0000 7f94c529d640 1 Processor -- start 2024-06-29T14:23:12.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.870+0000 7f94c529d640 1 -- start start 2024-06-29T14:23:12.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94c529d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c00b3740 0x7f94c013fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94c529d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c015c2f0 0x7f94c00ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94c529d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94c0140440 0x7f94c0157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94c529d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f94c00b6320 con 0x7f94c015c2f0 2024-06-29T14:23:12.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94c529d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94c00b61a0 con 0x7f94c00b3740 2024-06-29T14:23:12.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94c529d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f94c00b64a0 con 0x7f94c0140440 2024-06-29T14:23:12.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c015c2f0 0x7f94c00ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c015c2f0 0x7f94c00ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47990/0 (socket says 172.21.15.59:47990) 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 -- 172.21.15.59:0/4115398547 learned_addr learned my addr 172.21.15.59:0/4115398547 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94beffd640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c00b3740 0x7f94c013fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94bf7fe640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94c0140440 0x7f94c0157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 -- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94c0140440 msgr2=0x7f94c0157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94c0140440 0x7f94c0157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 -- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c00b3740 msgr2=0x7f94c013fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c00b3740 0x7f94c013fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 -- 172.21.15.59:0/4115398547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c00aec50 con 0x7f94c015c2f0 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.871+0000 7f94be7fc640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c015c2f0 0x7f94c00ae710 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f94b4034040 tx=0x7f94b4003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.872+0000 7f94a77fe640 1 -- 172.21.15.59:0/4115398547 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b403e030 con 0x7f94c015c2f0 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.872+0000 7f94a77fe640 1 -- 172.21.15.59:0/4115398547 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94b40047a0 con 0x7f94c015c2f0 2024-06-29T14:23:12.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.872+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c0156e60 con 0x7f94c015c2f0 2024-06-29T14:23:12.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.872+0000 7f94a77fe640 1 -- 172.21.15.59:0/4115398547 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4035040 con 0x7f94c015c2f0 2024-06-29T14:23:12.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.872+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c0159820 con 0x7f94c015c2f0 2024-06-29T14:23:12.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.873+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94c00b2350 con 0x7f94c015c2f0 2024-06-29T14:23:12.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.874+0000 7f94a77fe640 1 -- 172.21.15.59:0/4115398547 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f94b4041030 con 0x7f94c015c2f0 2024-06-29T14:23:12.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.874+0000 7f94a77fe640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f94ac076300 0x7f94ac078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.874+0000 7f94a77fe640 1 -- 172.21.15.59:0/4115398547 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7f94b40b7e70 con 0x7f94c015c2f0 2024-06-29T14:23:12.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.874+0000 7f94beffd640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f94ac076300 0x7f94ac078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.875+0000 7f94beffd640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f94ac076300 0x7f94ac078810 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f94b0000f60 tx=0x7f94b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:12.879+0000 7f94a77fe640 1 -- 172.21.15.59:0/4115398547 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f94c00b2350 con 0x7f94c015c2f0 2024-06-29T14:23:13.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.071+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f94c01458a0 con 0x7f94c015c2f0 2024-06-29T14:23:13.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.072+0000 7f94a77fe640 1 -- 172.21.15.59:0/4115398547 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v37) ==== 84+0+65 (secure 0 0 0) 0x7f94b4081900 con 0x7f94c015c2f0 2024-06-29T14:23:13.071 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:13.071 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:23:13.071 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:13.071 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:23:13.071 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:23:13.072 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:13.072 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:23:13.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f94ac076300 msgr2=0x7f94ac078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f94ac076300 0x7f94ac078810 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f94b0000f60 tx=0x7f94b0009040 comp rx=0 tx=0).stop 2024-06-29T14:23:13.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c015c2f0 msgr2=0x7f94c00ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c015c2f0 0x7f94c00ae710 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f94b4034040 tx=0x7f94b4003970 comp rx=0 tx=0).stop 2024-06-29T14:23:13.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 shutdown_connections 2024-06-29T14:23:13.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f94ac076300 0x7f94ac078810 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94c0140440 0x7f94c0157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94c015c2f0 0x7f94c00ae710 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 --2- 172.21.15.59:0/4115398547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94c00b3740 0x7f94c013fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 >> 172.21.15.59:0/4115398547 conn(0x7f94c006c950 msgr2=0x7f94c0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:13.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 shutdown_connections 2024-06-29T14:23:13.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.074+0000 7f94c529d640 1 -- 172.21.15.59:0/4115398547 wait complete. 2024-06-29T14:23:13.084 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:23:13.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:23:13.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.208+0000 7efc7c82e640 1 -- 172.21.15.59:0/2176604260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc781423c0 msgr2=0x7efc780b56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.208+0000 7efc7c82e640 1 --2- 172.21.15.59:0/2176604260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc781423c0 0x7efc780b56e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efc60005fd0 tx=0x7efc6002a5a0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.208+0000 7efc7c82e640 1 -- 172.21.15.59:0/2176604260 shutdown_connections 2024-06-29T14:23:13.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.208+0000 7efc7c82e640 1 --2- 172.21.15.59:0/2176604260 >> v2:172.21.15.153:3300/0 conn(0x7efc780cec20 0x7efc780d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.208+0000 7efc7c82e640 1 --2- 172.21.15.59:0/2176604260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc781423c0 0x7efc780b56e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.208+0000 7efc7c82e640 1 -- 172.21.15.59:0/2176604260 >> 172.21.15.59:0/2176604260 conn(0x7efc78073e10 msgr2=0x7efc78074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:13.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.208+0000 7efc7c82e640 1 -- 172.21.15.59:0/2176604260 shutdown_connections 2024-06-29T14:23:13.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.208+0000 7efc7c82e640 1 -- 172.21.15.59:0/2176604260 wait complete. 2024-06-29T14:23:13.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.210+0000 7efc7c82e640 1 Processor -- start 2024-06-29T14:23:13.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.210+0000 7efc7c82e640 1 -- start start 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc7c82e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc780cec20 0x7efc7815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc7c82e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc780d1cd0 0x7efc7813fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc7c82e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc781423c0 0x7efc780c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc7c82e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efc780d0b10 con 0x7efc780d1cd0 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc7c82e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc7815c4d0 con 0x7efc781423c0 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc7c82e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efc780b2bf0 con 0x7efc780cec20 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc780d1cd0 0x7efc7813fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc780d1cd0 0x7efc7813fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48002/0 (socket says 172.21.15.59:48002) 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc75d9b640 1 -- 172.21.15.59:0/4083581695 learned_addr learned my addr 172.21.15.59:0/4083581695 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc7659c640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc780cec20 0x7efc7815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc76d9d640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc781423c0 0x7efc780c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc75d9b640 1 -- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc780cec20 msgr2=0x7efc7815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc75d9b640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc780cec20 0x7efc7815aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc75d9b640 1 -- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc781423c0 msgr2=0x7efc780c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc75d9b640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc781423c0 0x7efc780c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc75d9b640 1 -- 172.21.15.59:0/4083581695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc781f2ef0 con 0x7efc780d1cd0 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.212+0000 7efc7659c640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc780cec20 0x7efc7815aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.213+0000 7efc75d9b640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc780d1cd0 0x7efc7813fb00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e7c0 tx=0x7efc6800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.213+0000 7efc6effd640 1 -- 172.21.15.59:0/4083581695 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6800ccc0 con 0x7efc780d1cd0 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.213+0000 7efc6effd640 1 -- 172.21.15.59:0/4083581695 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc68010430 con 0x7efc780d1cd0 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.213+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc78159e80 con 0x7efc780d1cd0 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.213+0000 7efc6effd640 1 -- 172.21.15.59:0/4083581695 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc68010860 con 0x7efc780d1cd0 2024-06-29T14:23:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.213+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc7815a440 con 0x7efc780d1cd0 2024-06-29T14:23:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.214+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc7815b020 con 0x7efc780d1cd0 2024-06-29T14:23:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.214+0000 7efc6effd640 1 -- 172.21.15.59:0/4083581695 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efc68010a00 con 0x7efc780d1cd0 2024-06-29T14:23:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.215+0000 7efc6effd640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efc580762d0 0x7efc580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.215+0000 7efc6effd640 1 -- 172.21.15.59:0/4083581695 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5249+0+0 (secure 0 0 0) 0x7efc68097fe0 con 0x7efc780d1cd0 2024-06-29T14:23:13.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.216+0000 7efc7659c640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efc580762d0 0x7efc580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.216+0000 7efc7659c640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efc580762d0 0x7efc580787e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efc60033000 tx=0x7efc60004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:13.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.220+0000 7efc6effd640 1 -- 172.21.15.59:0/4083581695 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efc68061a70 con 0x7efc780d1cd0 2024-06-29T14:23:13.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:13.416+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7efc7815a640 con 0x7efc780d1cd0 2024-06-29T14:23:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.168+0000 7efc6effd640 1 -- 172.21.15.59:0/4083581695 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v38) ==== 138+0+0 (secure 0 0 0) 0x7efc7815a640 con 0x7efc780d1cd0 2024-06-29T14:23:14.168 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:23:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efc580762d0 msgr2=0x7efc580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efc580762d0 0x7efc580787e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efc60033000 tx=0x7efc60004730 comp rx=0 tx=0).stop 2024-06-29T14:23:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc780d1cd0 msgr2=0x7efc7813fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc780d1cd0 0x7efc7813fb00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e7c0 tx=0x7efc6800ec80 comp rx=0 tx=0).stop 2024-06-29T14:23:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 shutdown_connections 2024-06-29T14:23:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7efc580762d0 0x7efc580787e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc781423c0 0x7efc780c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc780d1cd0 0x7efc7813fb00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 --2- 172.21.15.59:0/4083581695 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc780cec20 0x7efc7815aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 >> 172.21.15.59:0/4083581695 conn(0x7efc78073e10 msgr2=0x7efc780b6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 shutdown_connections 2024-06-29T14:23:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.171+0000 7efc7c82e640 1 -- 172.21.15.59:0/4083581695 wait complete. 2024-06-29T14:23:14.186 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.320+0000 7fecb2344640 1 -- 172.21.15.59:0/1720160248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecac0b3e10 msgr2=0x7fecac167040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.320+0000 7fecb2344640 1 --2- 172.21.15.59:0/1720160248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecac0b3e10 0x7fecac167040 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7feca4005fd0 tx=0x7feca402a8d0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.320+0000 7fecb2344640 1 -- 172.21.15.59:0/1720160248 shutdown_connections 2024-06-29T14:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.320+0000 7fecb2344640 1 --2- 172.21.15.59:0/1720160248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecac0b3e10 0x7fecac167040 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.320+0000 7fecb2344640 1 -- 172.21.15.59:0/1720160248 >> 172.21.15.59:0/1720160248 conn(0x7fecac0b0670 msgr2=0x7fecac0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.321+0000 7fecb2344640 1 -- 172.21.15.59:0/1720160248 shutdown_connections 2024-06-29T14:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.321+0000 7fecb2344640 1 -- 172.21.15.59:0/1720160248 wait complete. 2024-06-29T14:23:14.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.322+0000 7fecb2344640 1 Processor -- start 2024-06-29T14:23:14.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.323+0000 7fecb2344640 1 -- start start 2024-06-29T14:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb2344640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecac0b3a00 0x7fecac13d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb2344640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecac0b3e10 0x7fecac073800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb2344640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecac13e190 0x7fecac073d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb2344640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fecac0cfb90 con 0x7fecac13e190 2024-06-29T14:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb2344640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fecac0cfa10 con 0x7fecac0b3e10 2024-06-29T14:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb2344640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fecac0cfd10 con 0x7fecac0b3a00 2024-06-29T14:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb08e1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecac13e190 0x7fecac073d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb08e1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecac13e190 0x7fecac073d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48010/0 (socket says 172.21.15.59:48010) 2024-06-29T14:23:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb08e1640 1 -- 172.21.15.59:0/2470276854 learned_addr learned my addr 172.21.15.59:0/2470276854 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecab7fe640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecac0b3e10 0x7fecac073800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecabfff640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecac0b3a00 0x7fecac13d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb08e1640 1 -- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecac0b3a00 msgr2=0x7fecac13d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb08e1640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecac0b3a00 0x7fecac13d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb08e1640 1 -- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecac0b3e10 msgr2=0x7fecac073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb08e1640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecac0b3e10 0x7fecac073800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.324+0000 7fecb08e1640 1 -- 172.21.15.59:0/2470276854 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecac144750 con 0x7fecac13e190 2024-06-29T14:23:14.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.325+0000 7fecb08e1640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecac13e190 0x7fecac073d40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7feca4005fd0 tx=0x7feca4005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:14.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.325+0000 7feca8ff9640 1 -- 172.21.15.59:0/2470276854 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4037040 con 0x7fecac13e190 2024-06-29T14:23:14.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.325+0000 7feca8ff9640 1 -- 172.21.15.59:0/2470276854 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feca4003c60 con 0x7fecac13e190 2024-06-29T14:23:14.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.325+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecac144a30 con 0x7fecac13e190 2024-06-29T14:23:14.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.325+0000 7feca8ff9640 1 -- 172.21.15.59:0/2470276854 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca403c990 con 0x7fecac13e190 2024-06-29T14:23:14.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.325+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecac144ff0 con 0x7fecac13e190 2024-06-29T14:23:14.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.326+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecac1444c0 con 0x7fecac13e190 2024-06-29T14:23:14.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.327+0000 7feca8ff9640 1 -- 172.21.15.59:0/2470276854 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feca403a070 con 0x7fecac13e190 2024-06-29T14:23:14.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.327+0000 7feca8ff9640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fec980762d0 0x7fec980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:14.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.327+0000 7feca8ff9640 1 -- 172.21.15.59:0/2470276854 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5619+0+0 (secure 0 0 0) 0x7feca40b8470 con 0x7fecac13e190 2024-06-29T14:23:14.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.327+0000 7fecabfff640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fec980762d0 0x7fec980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:14.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.328+0000 7fecabfff640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fec980762d0 0x7fec980787e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec9c000f60 tx=0x7fec9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:14.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.332+0000 7feca8ff9640 1 -- 172.21.15.59:0/2470276854 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feca4041020 con 0x7fecac13e190 2024-06-29T14:23:14.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:14.522+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fecac15d540 con 0x7fecac13e190 2024-06-29T14:23:15.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.175+0000 7feca8ff9640 1 -- 172.21.15.59:0/2470276854 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v39) ==== 170+0+0 (secure 0 0 0) 0x7feca40817d0 con 0x7fecac13e190 2024-06-29T14:23:15.174 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:23:15.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.178+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fec980762d0 msgr2=0x7fec980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.178+0000 7fecb2344640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fec980762d0 0x7fec980787e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec9c000f60 tx=0x7fec9c009040 comp rx=0 tx=0).stop 2024-06-29T14:23:15.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.178+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecac13e190 msgr2=0x7fecac073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.178+0000 7fecb2344640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecac13e190 0x7fecac073d40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7feca4005fd0 tx=0x7feca4005c30 comp rx=0 tx=0).stop 2024-06-29T14:23:15.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.179+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 shutdown_connections 2024-06-29T14:23:15.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.179+0000 7fecb2344640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fec980762d0 0x7fec980787e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.179+0000 7fecb2344640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecac13e190 0x7fecac073d40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.179+0000 7fecb2344640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecac0b3e10 0x7fecac073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.179+0000 7fecb2344640 1 --2- 172.21.15.59:0/2470276854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecac0b3a00 0x7fecac13d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.179+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 >> 172.21.15.59:0/2470276854 conn(0x7fecac0b0670 msgr2=0x7fecac15ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:15.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.179+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 shutdown_connections 2024-06-29T14:23:15.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.180+0000 7fecb2344640 1 -- 172.21.15.59:0/2470276854 wait complete. 2024-06-29T14:23:15.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:23:15.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.332+0000 7fc8a31d9640 1 -- 172.21.15.59:0/1473016250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 msgr2=0x7fc88c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.332+0000 7fc8a31d9640 1 --2- 172.21.15.59:0/1473016250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 0x7fc88c00b9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0c75c0 tx=0x7fc88c005830 comp rx=0 tx=0).stop 2024-06-29T14:23:15.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.332+0000 7fc8a31d9640 1 -- 172.21.15.59:0/1473016250 shutdown_connections 2024-06-29T14:23:15.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.332+0000 7fc8a31d9640 1 --2- 172.21.15.59:0/1473016250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 0x7fc88c00b9c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.332+0000 7fc8a31d9640 1 -- 172.21.15.59:0/1473016250 >> 172.21.15.59:0/1473016250 conn(0x7fc89c0d3ee0 msgr2=0x7fc89c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:15.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.332+0000 7fc8a31d9640 1 -- 172.21.15.59:0/1473016250 shutdown_connections 2024-06-29T14:23:15.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.332+0000 7fc8a31d9640 1 -- 172.21.15.59:0/1473016250 wait complete. 2024-06-29T14:23:15.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.334+0000 7fc8a31d9640 1 Processor -- start 2024-06-29T14:23:15.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.334+0000 7fc8a31d9640 1 -- start start 2024-06-29T14:23:15.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.335+0000 7fc8a31d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 0x7fc89c143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.335+0000 7fc8a31d9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc89c150830 0x7fc89c150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.335+0000 7fc8a31d9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc89c1f53a0 0x7fc89c1f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.335+0000 7fc8a31d9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc89c15ed10 con 0x7fc88c005300 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.335+0000 7fc8a31d9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc89c15eb90 con 0x7fc89c1f53a0 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.335+0000 7fc8a31d9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc89c0b3990 con 0x7fc89c150830 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 0x7fc89c143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 0x7fc89c143dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48022/0 (socket says 172.21.15.59:48022) 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 -- 172.21.15.59:0/3010220176 learned_addr learned my addr 172.21.15.59:0/3010220176 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a1776640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc89c1f53a0 0x7fc89c1f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc89bfff640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc89c150830 0x7fc89c150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 -- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc89c150830 msgr2=0x7fc89c150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc89c150830 0x7fc89c150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 -- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc89c1f53a0 msgr2=0x7fc89c1f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc89c1f53a0 0x7fc89c1f57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 -- 172.21.15.59:0/3010220176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89c152670 con 0x7fc88c005300 2024-06-29T14:23:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a0f75640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 0x7fc89c143dd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc890003d80 tx=0x7fc890001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8997fa640 1 -- 172.21.15.59:0/3010220176 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890011700 con 0x7fc88c005300 2024-06-29T14:23:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8997fa640 1 -- 172.21.15.59:0/3010220176 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc890016020 con 0x7fc88c005300 2024-06-29T14:23:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89c1515d0 con 0x7fc88c005300 2024-06-29T14:23:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8997fa640 1 -- 172.21.15.59:0/3010220176 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890011b60 con 0x7fc88c005300 2024-06-29T14:23:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.336+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc89c151b90 con 0x7fc88c005300 2024-06-29T14:23:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.338+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc89c1605f0 con 0x7fc88c005300 2024-06-29T14:23:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.338+0000 7fc8997fa640 1 -- 172.21.15.59:0/3010220176 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc890011d00 con 0x7fc88c005300 2024-06-29T14:23:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.339+0000 7fc8997fa640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc880076300 0x7fc880078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.339+0000 7fc8997fa640 1 -- 172.21.15.59:0/3010220176 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6001+0+0 (secure 0 0 0) 0x7fc890098560 con 0x7fc88c005300 2024-06-29T14:23:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.339+0000 7fc89bfff640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc880076300 0x7fc880078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.340+0000 7fc89bfff640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc880076300 0x7fc880078810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc888000c00 tx=0x7fc888000f70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:15.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.343+0000 7fc8997fa640 1 -- 172.21.15.59:0/3010220176 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc89c1605f0 con 0x7fc88c005300 2024-06-29T14:23:15.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:15.577+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc89c0b3ef0 con 0x7fc88c005300 2024-06-29T14:23:16.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.194+0000 7fc8997fa640 1 -- 172.21.15.59:0/3010220176 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v15) ==== 172+0+0 (secure 0 0 0) 0x7fc890020020 con 0x7fc88c005300 2024-06-29T14:23:16.193 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 5 and data pool 6 2024-06-29T14:23:16.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.202+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc880076300 msgr2=0x7fc880078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.202+0000 7fc8a31d9640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc880076300 0x7fc880078810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc888000c00 tx=0x7fc888000f70 comp rx=0 tx=0).stop 2024-06-29T14:23:16.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.202+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 msgr2=0x7fc89c143dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.202+0000 7fc8a31d9640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 0x7fc89c143dd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc890003d80 tx=0x7fc890001800 comp rx=0 tx=0).stop 2024-06-29T14:23:16.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.203+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 shutdown_connections 2024-06-29T14:23:16.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.203+0000 7fc8a31d9640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fc880076300 0x7fc880078810 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.203+0000 7fc8a31d9640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc89c1f53a0 0x7fc89c1f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.203+0000 7fc8a31d9640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc89c150830 0x7fc89c150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.203+0000 7fc8a31d9640 1 --2- 172.21.15.59:0/3010220176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc88c005300 0x7fc89c143dd0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.203+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 >> 172.21.15.59:0/3010220176 conn(0x7fc89c0d3ee0 msgr2=0x7fc89c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.206+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 shutdown_connections 2024-06-29T14:23:16.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.206+0000 7fc8a31d9640 1 -- 172.21.15.59:0/3010220176 wait complete. 2024-06-29T14:23:16.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:16.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.349+0000 7f08d5a47640 1 -- 172.21.15.59:0/754753708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d01444c0 msgr2=0x7f08d006d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.349+0000 7f08d5a47640 1 --2- 172.21.15.59:0/754753708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d01444c0 0x7f08d006d2f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f08b8005fd0 tx=0x7f08b802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.349+0000 7f08d5a47640 1 -- 172.21.15.59:0/754753708 shutdown_connections 2024-06-29T14:23:16.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.349+0000 7f08d5a47640 1 --2- 172.21.15.59:0/754753708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d01444c0 0x7f08d006d2f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.349+0000 7f08d5a47640 1 --2- 172.21.15.59:0/754753708 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08d00cd9b0 0x7f08d015f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.349+0000 7f08d5a47640 1 -- 172.21.15.59:0/754753708 >> 172.21.15.59:0/754753708 conn(0x7f08d00b24c0 msgr2=0x7f08d00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.349+0000 7f08d5a47640 1 -- 172.21.15.59:0/754753708 shutdown_connections 2024-06-29T14:23:16.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.349+0000 7f08d5a47640 1 -- 172.21.15.59:0/754753708 wait complete. 2024-06-29T14:23:16.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.351+0000 7f08d5a47640 1 Processor -- start 2024-06-29T14:23:16.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.351+0000 7f08d5a47640 1 -- start start 2024-06-29T14:23:16.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08d5a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d006d830 0x7f08d0138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08d5a47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08d00cd9b0 0x7f08d0138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08d5a47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08d01444c0 0x7f08d013a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08d5a47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f08d015b9b0 con 0x7f08d006d830 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08d5a47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08d0074300 con 0x7f08d01444c0 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08d5a47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f08d015bb30 con 0x7f08d00cd9b0 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d006d830 0x7f08d0138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d006d830 0x7f08d0138840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48040/0 (socket says 172.21.15.59:48040) 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08ceffd640 1 -- 172.21.15.59:0/4103386260 learned_addr learned my addr 172.21.15.59:0/4103386260 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08cf7fe640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08d01444c0 0x7f08d013a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08c7fff640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08d00cd9b0 0x7f08d0138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08ceffd640 1 -- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08d00cd9b0 msgr2=0x7f08d0138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08ceffd640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08d00cd9b0 0x7f08d0138d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08ceffd640 1 -- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08d01444c0 msgr2=0x7f08d013a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08ceffd640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08d01444c0 0x7f08d013a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08ceffd640 1 -- 172.21.15.59:0/4103386260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d01f6070 con 0x7f08d006d830 2024-06-29T14:23:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.352+0000 7f08c7fff640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08d00cd9b0 0x7f08d0138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.353+0000 7f08ceffd640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d006d830 0x7f08d0138840 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f08c000c970 tx=0x7f08c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.353+0000 7f08ccff9640 1 -- 172.21.15.59:0/4103386260 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0007b50 con 0x7f08d006d830 2024-06-29T14:23:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.353+0000 7f08ccff9640 1 -- 172.21.15.59:0/4103386260 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08c0007de0 con 0x7f08d006d830 2024-06-29T14:23:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.353+0000 7f08ccff9640 1 -- 172.21.15.59:0/4103386260 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0005280 con 0x7f08d006d830 2024-06-29T14:23:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.353+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d01f54f0 con 0x7f08d006d830 2024-06-29T14:23:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.353+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d01f5ab0 con 0x7f08d006d830 2024-06-29T14:23:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.354+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d015bf20 con 0x7f08d006d830 2024-06-29T14:23:16.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.354+0000 7f08ccff9640 1 -- 172.21.15.59:0/4103386260 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f08c0005420 con 0x7f08d006d830 2024-06-29T14:23:16.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.355+0000 7f08ccff9640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08b00762d0 0x7f08b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.355+0000 7f08ccff9640 1 -- 172.21.15.59:0/4103386260 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f08c0097c30 con 0x7f08d006d830 2024-06-29T14:23:16.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.356+0000 7f08c7fff640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08b00762d0 0x7f08b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.356+0000 7f08c7fff640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08b00762d0 0x7f08b00787e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f08b8033000 tx=0x7f08b8004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.360+0000 7f08ccff9640 1 -- 172.21.15.59:0/4103386260 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f08c0014030 con 0x7f08d006d830 2024-06-29T14:23:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.550+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08d006d610 con 0x7f08d006d830 2024-06-29T14:23:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.551+0000 7f08ccff9640 1 -- 172.21.15.59:0/4103386260 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+16967 (secure 0 0 0) 0x7f08c0060de0 con 0x7f08d006d830 2024-06-29T14:23:16.551 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:16.551 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":40,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:23:15.579355+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:13.422166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:14.527412+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.553+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08b00762d0 msgr2=0x7f08b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.553+0000 7f08d5a47640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08b00762d0 0x7f08b00787e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f08b8033000 tx=0x7f08b8004730 comp rx=0 tx=0).stop 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.553+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d006d830 msgr2=0x7f08d0138840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.553+0000 7f08d5a47640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d006d830 0x7f08d0138840 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f08c000c970 tx=0x7f08c000ce30 comp rx=0 tx=0).stop 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.554+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 shutdown_connections 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.554+0000 7f08d5a47640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08b00762d0 0x7f08b00787e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.554+0000 7f08d5a47640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08d01444c0 0x7f08d013a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.554+0000 7f08d5a47640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08d00cd9b0 0x7f08d0138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.554+0000 7f08d5a47640 1 --2- 172.21.15.59:0/4103386260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08d006d830 0x7f08d0138840 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.554+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 >> 172.21.15.59:0/4103386260 conn(0x7f08d00b24c0 msgr2=0x7f08d0142c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.554+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 shutdown_connections 2024-06-29T14:23:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.554+0000 7f08d5a47640 1 -- 172.21.15.59:0/4103386260 wait complete. 2024-06-29T14:23:16.564 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.683+0000 7fb015d7b640 1 -- 172.21.15.59:0/2715010886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 msgr2=0x7fb0100b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.683+0000 7fb015d7b640 1 --2- 172.21.15.59:0/2715010886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 0x7fb0100b3b50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb010074b10 tx=0x7fb00402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.683+0000 7fb015d7b640 1 -- 172.21.15.59:0/2715010886 shutdown_connections 2024-06-29T14:23:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.683+0000 7fb015d7b640 1 --2- 172.21.15.59:0/2715010886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 0x7fb0100b3b50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.683+0000 7fb015d7b640 1 --2- 172.21.15.59:0/2715010886 >> v2:172.21.15.153:3300/0 conn(0x7fb0101458a0 0x7fb0100cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.683+0000 7fb015d7b640 1 -- 172.21.15.59:0/2715010886 >> 172.21.15.59:0/2715010886 conn(0x7fb01006c950 msgr2=0x7fb010073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.683+0000 7fb015d7b640 1 -- 172.21.15.59:0/2715010886 shutdown_connections 2024-06-29T14:23:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.683+0000 7fb015d7b640 1 -- 172.21.15.59:0/2715010886 wait complete. 2024-06-29T14:23:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.685+0000 7fb015d7b640 1 Processor -- start 2024-06-29T14:23:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.685+0000 7fb015d7b640 1 -- start start 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb015d7b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 0x7fb010131630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb015d7b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb01015c2f0 0x7fb010140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb015d7b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb010133120 0x7fb010134b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb015d7b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb010160020 con 0x7fb0100b3740 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb015d7b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb01015cbb0 con 0x7fb010133120 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb015d7b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb0101601a0 con 0x7fb01015c2f0 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb00f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 0x7fb010131630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb00f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 0x7fb010131630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48070/0 (socket says 172.21.15.59:48070) 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb00f7fe640 1 -- 172.21.15.59:0/3796411759 learned_addr learned my addr 172.21.15.59:0/3796411759 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb00ffff640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb010133120 0x7fb010134b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.686+0000 7fb00effd640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb01015c2f0 0x7fb010140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7fb00f7fe640 1 -- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb01015c2f0 msgr2=0x7fb010140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7fb00f7fe640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb01015c2f0 0x7fb010140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7fb00f7fe640 1 -- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb010133120 msgr2=0x7fb010134b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7fb00f7fe640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb010133120 0x7fb010134b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7fb00f7fe640 1 -- 172.21.15.59:0/3796411759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb010131b70 con 0x7fb0100b3740 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7fb00f7fe640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 0x7fb010131630 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c970 tx=0x7fb00000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7faff7fff640 1 -- 172.21.15.59:0/3796411759 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000007bb0 con 0x7fb0100b3740 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7faff7fff640 1 -- 172.21.15.59:0/3796411759 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb000007e40 con 0x7fb0100b3740 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb010140c70 con 0x7fb0100b3740 2024-06-29T14:23:16.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7faff7fff640 1 -- 172.21.15.59:0/3796411759 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000005280 con 0x7fb0100b3740 2024-06-29T14:23:16.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.687+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb010132360 con 0x7fb0100b3740 2024-06-29T14:23:16.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.688+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0101606c0 con 0x7fb0100b3740 2024-06-29T14:23:16.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.689+0000 7faff7fff640 1 -- 172.21.15.59:0/3796411759 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb000005420 con 0x7fb0100b3740 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.689+0000 7faff7fff640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7faffc076300 0x7faffc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.689+0000 7faff7fff640 1 -- 172.21.15.59:0/3796411759 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fb000097d90 con 0x7fb0100b3740 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.690+0000 7fb00effd640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7faffc076300 0x7faffc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.690+0000 7fb00effd640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7faffc076300 0x7faffc078810 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb004033000 tx=0x7fb004004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.694+0000 7faff7fff640 1 -- 172.21.15.59:0/3796411759 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb000014030 con 0x7fb0100b3740 2024-06-29T14:23:16.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.885+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb010070e10 con 0x7fb0100b3740 2024-06-29T14:23:16.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.886+0000 7faff7fff640 1 -- 172.21.15.59:0/3796411759 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+16967 (secure 0 0 0) 0x7fb000061620 con 0x7fb0100b3740 2024-06-29T14:23:16.885 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:16.885 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":40,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:23:15.579355+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:13.422166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:14.527412+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:16.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7faffc076300 msgr2=0x7faffc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7faffc076300 0x7faffc078810 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb004033000 tx=0x7fb004004730 comp rx=0 tx=0).stop 2024-06-29T14:23:16.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 msgr2=0x7fb010131630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 0x7fb010131630 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c970 tx=0x7fb00000ce30 comp rx=0 tx=0).stop 2024-06-29T14:23:16.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 shutdown_connections 2024-06-29T14:23:16.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7faffc076300 0x7faffc078810 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb010133120 0x7fb010134b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb01015c2f0 0x7fb010140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 --2- 172.21.15.59:0/3796411759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0100b3740 0x7fb010131630 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 >> 172.21.15.59:0/3796411759 conn(0x7fb01006c950 msgr2=0x7fb010070700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 shutdown_connections 2024-06-29T14:23:16.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:16.888+0000 7fb015d7b640 1 -- 172.21.15.59:0/3796411759 wait complete. 2024-06-29T14:23:16.898 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:23:17.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.021+0000 7fb5d0240640 1 -- 172.21.15.59:0/499911222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5c80b52b0 msgr2=0x7fb5c80b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.021+0000 7fb5d0240640 1 --2- 172.21.15.59:0/499911222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5c80b52b0 0x7fb5c80b49d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009830 tx=0x7fb5c402a510 comp rx=0 tx=0).stop 2024-06-29T14:23:17.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.021+0000 7fb5d0240640 1 -- 172.21.15.59:0/499911222 shutdown_connections 2024-06-29T14:23:17.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.021+0000 7fb5d0240640 1 --2- 172.21.15.59:0/499911222 >> v2:172.21.15.153:3300/0 conn(0x7fb5c815be90 0x7fb5c80caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.021+0000 7fb5d0240640 1 --2- 172.21.15.59:0/499911222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5c80b52b0 0x7fb5c80b49d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.021+0000 7fb5d0240640 1 -- 172.21.15.59:0/499911222 >> 172.21.15.59:0/499911222 conn(0x7fb5c80b1ea0 msgr2=0x7fb5c80b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:17.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.021+0000 7fb5d0240640 1 -- 172.21.15.59:0/499911222 shutdown_connections 2024-06-29T14:23:17.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.021+0000 7fb5d0240640 1 -- 172.21.15.59:0/499911222 wait complete. 2024-06-29T14:23:17.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.023+0000 7fb5d0240640 1 Processor -- start 2024-06-29T14:23:17.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.023+0000 7fb5d0240640 1 -- start start 2024-06-29T14:23:17.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5d0240640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c80b52b0 0x7fb5c8154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5d0240640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5c80cb480 0x7fb5c809bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5d0240640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c815be90 0x7fb5c806d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5d0240640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c80cbc20 con 0x7fb5c815be90 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5d0240640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c80cbaa0 con 0x7fb5c80b52b0 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5d0240640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb5c81601a0 con 0x7fb5c80cb480 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5cdfdc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c80b52b0 0x7fb5c8154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5ce7dd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c815be90 0x7fb5c806d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5cdfdc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c80b52b0 0x7fb5c8154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:59370/0 (socket says 172.21.15.59:59370) 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5cdfdc640 1 -- 172.21.15.59:0/885127393 learned_addr learned my addr 172.21.15.59:0/885127393 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5cd7db640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5c80cb480 0x7fb5c809bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5ce7dd640 1 -- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5c80cb480 msgr2=0x7fb5c809bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5ce7dd640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5c80cb480 0x7fb5c809bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5ce7dd640 1 -- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c80b52b0 msgr2=0x7fb5c8154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5ce7dd640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c80b52b0 0x7fb5c8154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5ce7dd640 1 -- 172.21.15.59:0/885127393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c809aae0 con 0x7fb5c815be90 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5cdfdc640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c80b52b0 0x7fb5c8154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5ce7dd640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c815be90 0x7fb5c806d040 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80724c0 tx=0x7fb5c000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.025+0000 7fb5be7fc640 1 -- 172.21.15.59:0/885127393 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0003b90 con 0x7fb5c815be90 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.026+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c809ad80 con 0x7fb5c815be90 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.026+0000 7fb5be7fc640 1 -- 172.21.15.59:0/885127393 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5c0003d50 con 0x7fb5c815be90 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.026+0000 7fb5be7fc640 1 -- 172.21.15.59:0/885127393 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0005330 con 0x7fb5c815be90 2024-06-29T14:23:17.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.026+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c809b340 con 0x7fb5c815be90 2024-06-29T14:23:17.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.026+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c80cfad0 con 0x7fb5c815be90 2024-06-29T14:23:17.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.027+0000 7fb5be7fc640 1 -- 172.21.15.59:0/885127393 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb5c00054d0 con 0x7fb5c815be90 2024-06-29T14:23:17.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.028+0000 7fb5be7fc640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5ac0762d0 0x7fb5ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.028+0000 7fb5be7fc640 1 -- 172.21.15.59:0/885127393 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fb5c0097580 con 0x7fb5c815be90 2024-06-29T14:23:17.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.028+0000 7fb5cdfdc640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5ac0762d0 0x7fb5ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.029+0000 7fb5cdfdc640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5ac0762d0 0x7fb5ac0787e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb5c40095f0 tx=0x7fb5c4004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.032+0000 7fb5be7fc640 1 -- 172.21.15.59:0/885127393 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb5c80cfad0 con 0x7fb5c815be90 2024-06-29T14:23:17.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:17.263+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb5c81605f0 con 0x7fb5c815be90 2024-06-29T14:23:18.153 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T14:23:18.155+0000 7f05461e8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:23:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.214+0000 7fb5be7fc640 1 -- 172.21.15.59:0/885127393 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fb5c809aae0 con 0x7fb5c815be90 2024-06-29T14:23:18.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5ac0762d0 msgr2=0x7fb5ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5ac0762d0 0x7fb5ac0787e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb5c40095f0 tx=0x7fb5c4004730 comp rx=0 tx=0).stop 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c815be90 msgr2=0x7fb5c806d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c815be90 0x7fb5c806d040 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80724c0 tx=0x7fb5c000cb90 comp rx=0 tx=0).stop 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 shutdown_connections 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fb5ac0762d0 0x7fb5ac0787e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5c815be90 0x7fb5c806d040 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5c80cb480 0x7fb5c809bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 --2- 172.21.15.59:0/885127393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5c80b52b0 0x7fb5c8154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 >> 172.21.15.59:0/885127393 conn(0x7fb5c80b1ea0 msgr2=0x7fb5c80afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 shutdown_connections 2024-06-29T14:23:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.217+0000 7fb5d0240640 1 -- 172.21.15.59:0/885127393 wait complete. 2024-06-29T14:23:18.229 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:23:18.230 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:23:18.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.359+0000 7fe1af7a1640 1 -- 172.21.15.59:0/3169629301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80b2270 msgr2=0x7fe1a815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.359+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/3169629301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80b2270 0x7fe1a815bc20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe1a80b2680 tx=0x7fe1a402aac0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.359+0000 7fe1af7a1640 1 -- 172.21.15.59:0/3169629301 shutdown_connections 2024-06-29T14:23:18.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.359+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/3169629301 >> v2:172.21.15.153:3300/0 conn(0x7fe1a80b2fa0 0x7fe1a80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.359+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/3169629301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80b2270 0x7fe1a815bc20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.359+0000 7fe1af7a1640 1 -- 172.21.15.59:0/3169629301 >> 172.21.15.59:0/3169629301 conn(0x7fe1a8073e10 msgr2=0x7fe1a8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.359+0000 7fe1af7a1640 1 -- 172.21.15.59:0/3169629301 shutdown_connections 2024-06-29T14:23:18.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.360+0000 7fe1af7a1640 1 -- 172.21.15.59:0/3169629301 wait complete. 2024-06-29T14:23:18.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.361+0000 7fe1af7a1640 1 Processor -- start 2024-06-29T14:23:18.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.361+0000 7fe1af7a1640 1 -- start start 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1af7a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a80b2270 0x7fe1a81545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1af7a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a80b2fa0 0x7fe1a80ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1af7a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80d1a60 0x7fe1a80ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1af7a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a80b6cc0 con 0x7fe1a80d1a60 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1af7a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a80b6b40 con 0x7fe1a80b2270 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1af7a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe1a80cfe90 con 0x7fe1a80b2fa0 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80d1a60 0x7fe1a80ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80d1a60 0x7fe1a80ac930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48098/0 (socket says 172.21.15.59:48098) 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 -- 172.21.15.59:0/2525613385 learned_addr learned my addr 172.21.15.59:0/2525613385 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1ad53d640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a80b2270 0x7fe1a81545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1acd3c640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a80b2fa0 0x7fe1a80ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 -- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a80b2fa0 msgr2=0x7fe1a80ac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a80b2fa0 0x7fe1a80ac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 -- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a80b2270 msgr2=0x7fe1a81545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a80b2270 0x7fe1a81545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 -- 172.21.15.59:0/2525613385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a80a9ba0 con 0x7fe1a80d1a60 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1add3e640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80d1a60 0x7fe1a80ac930 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000c3f0 tx=0x7fe1a000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe19dffb640 1 -- 172.21.15.59:0/2525613385 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0017070 con 0x7fe1a80d1a60 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe19dffb640 1 -- 172.21.15.59:0/2525613385 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1a00053c0 con 0x7fe1a80d1a60 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a80a9e80 con 0x7fe1a80d1a60 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe19dffb640 1 -- 172.21.15.59:0/2525613385 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0003850 con 0x7fe1a80d1a60 2024-06-29T14:23:18.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.364+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1a80aa440 con 0x7fe1a80d1a60 2024-06-29T14:23:18.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.365+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a80d0390 con 0x7fe1a80d1a60 2024-06-29T14:23:18.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.366+0000 7fe19dffb640 1 -- 172.21.15.59:0/2525613385 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe1a001a070 con 0x7fe1a80d1a60 2024-06-29T14:23:18.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.366+0000 7fe19dffb640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe18c076300 0x7fe18c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.366+0000 7fe19dffb640 1 -- 172.21.15.59:0/2525613385 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7fe1a0096bb0 con 0x7fe1a80d1a60 2024-06-29T14:23:18.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.366+0000 7fe1ad53d640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe18c076300 0x7fe18c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.367+0000 7fe1ad53d640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe18c076300 0x7fe18c078810 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4035000 tx=0x7fe1a4004ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.370+0000 7fe19dffb640 1 -- 172.21.15.59:0/2525613385 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe1a0060410 con 0x7fe1a80d1a60 2024-06-29T14:23:18.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:18.605+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe1a80a98c0 con 0x7fe1a80d1a60 2024-06-29T14:23:19.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.215+0000 7fe19dffb640 1 -- 172.21.15.59:0/2525613385 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v19) ==== 122+0+0 (secure 0 0 0) 0x7fe1a0068400 con 0x7fe1a80d1a60 2024-06-29T14:23:19.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe18c076300 msgr2=0x7fe18c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe18c076300 0x7fe18c078810 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4035000 tx=0x7fe1a4004ab0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80d1a60 msgr2=0x7fe1a80ac930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80d1a60 0x7fe1a80ac930 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000c3f0 tx=0x7fe1a000c8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 shutdown_connections 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fe18c076300 0x7fe18c078810 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a80d1a60 0x7fe1a80ac930 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a80b2fa0 0x7fe1a80ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 --2- 172.21.15.59:0/2525613385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a80b2270 0x7fe1a81545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 >> 172.21.15.59:0/2525613385 conn(0x7fe1a8073e10 msgr2=0x7fe1a80d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 shutdown_connections 2024-06-29T14:23:19.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.219+0000 7fe1af7a1640 1 -- 172.21.15.59:0/2525613385 wait complete. 2024-06-29T14:23:19.232 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:23:19.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:23:19.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.359+0000 7f9b9023f640 1 -- 172.21.15.59:0/2379129403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8815e0a0 msgr2=0x7f9b88161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.359+0000 7f9b9023f640 1 --2- 172.21.15.59:0/2379129403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8815e0a0 0x7f9b88161a80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9b880b29a0 tx=0x7f9b7c02a4b0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.359+0000 7f9b9023f640 1 -- 172.21.15.59:0/2379129403 shutdown_connections 2024-06-29T14:23:19.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.359+0000 7f9b9023f640 1 --2- 172.21.15.59:0/2379129403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b8815e0a0 0x7f9b88161a80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.359+0000 7f9b9023f640 1 --2- 172.21.15.59:0/2379129403 >> v2:172.21.15.153:3300/0 conn(0x7f9b88162770 0x7f9b880ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.359+0000 7f9b9023f640 1 -- 172.21.15.59:0/2379129403 >> 172.21.15.59:0/2379129403 conn(0x7f9b880d45b0 msgr2=0x7f9b880d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.359+0000 7f9b9023f640 1 -- 172.21.15.59:0/2379129403 shutdown_connections 2024-06-29T14:23:19.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.359+0000 7f9b9023f640 1 -- 172.21.15.59:0/2379129403 wait complete. 2024-06-29T14:23:19.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.361+0000 7f9b9023f640 1 Processor -- start 2024-06-29T14:23:19.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.361+0000 7f9b9023f640 1 -- start start 2024-06-29T14:23:19.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.362+0000 7f9b9023f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b8815e0a0 0x7f9b880ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.362+0000 7f9b9023f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b88161fc0 0x7f9b8815f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.362+0000 7f9b9023f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b88162770 0x7f9b8815fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.362+0000 7f9b9023f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9b880d2d80 con 0x7f9b88162770 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.362+0000 7f9b9023f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b880b4b00 con 0x7f9b8815e0a0 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.362+0000 7f9b9023f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9b880d2f00 con 0x7f9b88161fc0 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8e7dc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b88162770 0x7f9b8815fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8dfdb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b8815e0a0 0x7f9b880ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8dfdb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b8815e0a0 0x7f9b880ca0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:59390/0 (socket says 172.21.15.59:59390) 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8e7dc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b88162770 0x7f9b8815fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48112/0 (socket says 172.21.15.59:48112) 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8dfdb640 1 -- 172.21.15.59:0/4046013191 learned_addr learned my addr 172.21.15.59:0/4046013191 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8d7da640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b88161fc0 0x7f9b8815f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8e7dc640 1 -- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b88161fc0 msgr2=0x7f9b8815f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8e7dc640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b88161fc0 0x7f9b8815f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8e7dc640 1 -- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b8815e0a0 msgr2=0x7f9b880ca0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8e7dc640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b8815e0a0 0x7f9b880ca0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8e7dc640 1 -- 172.21.15.59:0/4046013191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b88160010 con 0x7f9b88162770 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b8e7dc640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b88162770 0x7f9b8815fad0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000b3e0 tx=0x7f9b8000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b867fc640 1 -- 172.21.15.59:0/4046013191 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80015020 con 0x7f9b88162770 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b867fc640 1 -- 172.21.15.59:0/4046013191 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b80011920 con 0x7f9b88162770 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b867fc640 1 -- 172.21.15.59:0/4046013191 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80011d70 con 0x7f9b88162770 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b880c7b50 con 0x7f9b88162770 2024-06-29T14:23:19.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.363+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b880c8110 con 0x7f9b88162770 2024-06-29T14:23:19.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.364+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b880d3420 con 0x7f9b88162770 2024-06-29T14:23:19.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.365+0000 7f9b867fc640 1 -- 172.21.15.59:0/4046013191 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9b80018020 con 0x7f9b88162770 2024-06-29T14:23:19.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.367+0000 7f9b867fc640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9b74076300 0x7f9b74078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.367+0000 7f9b867fc640 1 -- 172.21.15.59:0/4046013191 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f9b800975f0 con 0x7f9b88162770 2024-06-29T14:23:19.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.367+0000 7f9b8dfdb640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9b74076300 0x7f9b74078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.367+0000 7f9b8dfdb640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9b74076300 0x7f9b74078810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9b78007ee0 tx=0x7f9b780075d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:19.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.371+0000 7f9b867fc640 1 -- 172.21.15.59:0/4046013191 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9b8001f020 con 0x7f9b88162770 2024-06-29T14:23:19.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.599+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b880cdd90 con 0x7f9b88162770 2024-06-29T14:23:19.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.600+0000 7f9b867fc640 1 -- 172.21.15.59:0/4046013191 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5223 (secure 0 0 0) 0x7f9b800606e0 con 0x7f9b88162770 2024-06-29T14:23:19.599 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:19.599 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"btime":"2024-06-29T14:23:19:215604+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/634859216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":634859216},{"type":"v1","addr":"172.21.15.153:6837","nonce":634859216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3661382082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3661382082},{"type":"v1","addr":"172.21.15.153:6836","nonce":3661382082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4012387069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4012387069},{"type":"v1","addr":"172.21.15.59:6833","nonce":4012387069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:15.579309+0000","modified":"2024-06-29T14:23:18.607053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5469},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5469":{"gid":5469,"name":"a","rank":0,"incarnation":15,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/812277010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":812277010},{"type":"v1","addr":"172.21.15.59:6835","nonce":812277010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5469,"qdb_cluster":[5469]},"id":2}]} 2024-06-29T14:23:19.599 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 19 2024-06-29T14:23:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9b74076300 msgr2=0x7f9b74078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9b74076300 0x7f9b74078810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9b78007ee0 tx=0x7f9b780075d0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b88162770 msgr2=0x7f9b8815fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b88162770 0x7f9b8815fad0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000b3e0 tx=0x7f9b8000b8a0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 shutdown_connections 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f9b74076300 0x7f9b74078810 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b88162770 0x7f9b8815fad0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b88161fc0 0x7f9b8815f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 --2- 172.21.15.59:0/4046013191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b8815e0a0 0x7f9b880ca0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.602+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 >> 172.21.15.59:0/4046013191 conn(0x7f9b880d45b0 msgr2=0x7f9b880ccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.603+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 shutdown_connections 2024-06-29T14:23:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.603+0000 7f9b9023f640 1 -- 172.21.15.59:0/4046013191 wait complete. 2024-06-29T14:23:19.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.732+0000 7fbb1b438640 1 -- 172.21.15.59:0/21195495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 msgr2=0x7fbb140b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.732+0000 7fbb1b438640 1 --2- 172.21.15.59:0/21195495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 0x7fbb140b1b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009830 tx=0x7fbb1002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.733+0000 7fbb1b438640 1 -- 172.21.15.59:0/21195495 shutdown_connections 2024-06-29T14:23:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.733+0000 7fbb1b438640 1 --2- 172.21.15.59:0/21195495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 0x7fbb140b1b00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.733+0000 7fbb1b438640 1 -- 172.21.15.59:0/21195495 >> 172.21.15.59:0/21195495 conn(0x7fbb140b06e0 msgr2=0x7fbb140b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.733+0000 7fbb1b438640 1 -- 172.21.15.59:0/21195495 shutdown_connections 2024-06-29T14:23:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.733+0000 7fbb1b438640 1 -- 172.21.15.59:0/21195495 wait complete. 2024-06-29T14:23:19.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.734+0000 7fbb1b438640 1 Processor -- start 2024-06-29T14:23:19.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.734+0000 7fbb1b438640 1 -- start start 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb1b438640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1406ea10 0x7fbb14130230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb1b438640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb140b2040 0x7fbb1413f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb1b438640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 0x7fbb14159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb1b438640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbb140cfc70 con 0x7fbb1406ea10 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb1b438640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbb140cfaf0 con 0x7fbb14144470 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb1b438640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbb140cfdf0 con 0x7fbb140b2040 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1406ea10 0x7fbb14130230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1406ea10 0x7fbb14130230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48126/0 (socket says 172.21.15.59:48126) 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 -- 172.21.15.59:0/2686929345 learned_addr learned my addr 172.21.15.59:0/2686929345 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:19.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb199d5640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 0x7fbb14159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb189d3640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb140b2040 0x7fbb1413f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 -- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb140b2040 msgr2=0x7fbb1413f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb140b2040 0x7fbb1413f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 -- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 msgr2=0x7fbb14159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 0x7fbb14159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 -- 172.21.15.59:0/2686929345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb141f4eb0 con 0x7fbb1406ea10 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb199d5640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 0x7fbb14159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb191d4640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1406ea10 0x7fbb14130230 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbb1002abb0 tx=0x7fbb10033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.736+0000 7fbb09ffb640 1 -- 172.21.15.59:0/2686929345 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10031b50 con 0x7fbb1406ea10 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.737+0000 7fbb09ffb640 1 -- 172.21.15.59:0/2686929345 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb10031de0 con 0x7fbb1406ea10 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.737+0000 7fbb09ffb640 1 -- 172.21.15.59:0/2686929345 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb10005280 con 0x7fbb1406ea10 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.737+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb14130c10 con 0x7fbb1406ea10 2024-06-29T14:23:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.737+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1412e340 con 0x7fbb1406ea10 2024-06-29T14:23:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.738+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb140d0310 con 0x7fbb1406ea10 2024-06-29T14:23:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.738+0000 7fbb09ffb640 1 -- 172.21.15.59:0/2686929345 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbb10005420 con 0x7fbb1406ea10 2024-06-29T14:23:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.739+0000 7fbb09ffb640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fbaf8076300 0x7fbaf8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.739+0000 7fbb09ffb640 1 -- 172.21.15.59:0/2686929345 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7fbb100b89a0 con 0x7fbb1406ea10 2024-06-29T14:23:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.739+0000 7fbb189d3640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fbaf8076300 0x7fbaf8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.740+0000 7fbb189d3640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fbaf8076300 0x7fbaf8078810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbb1406df60 tx=0x7fbb04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:19.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.743+0000 7fbb09ffb640 1 -- 172.21.15.59:0/2686929345 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb1003f030 con 0x7fbb1406ea10 2024-06-29T14:23:19.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.936+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb140cd230 con 0x7fbb1406ea10 2024-06-29T14:23:19.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.938+0000 7fbb09ffb640 1 -- 172.21.15.59:0/2686929345 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16954 (secure 0 0 0) 0x7fbb100820e0 con 0x7fbb1406ea10 2024-06-29T14:23:19.936 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:19.936 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:23:17.178396+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:13.422166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:14.527412+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:19.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fbaf8076300 msgr2=0x7fbaf8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fbaf8076300 0x7fbaf8078810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbb1406df60 tx=0x7fbb04009040 comp rx=0 tx=0).stop 2024-06-29T14:23:19.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1406ea10 msgr2=0x7fbb14130230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1406ea10 0x7fbb14130230 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbb1002abb0 tx=0x7fbb10033ba0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 shutdown_connections 2024-06-29T14:23:19.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fbaf8076300 0x7fbaf8078810 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb14144470 0x7fbb14159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb140b2040 0x7fbb1413f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 --2- 172.21.15.59:0/2686929345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1406ea10 0x7fbb14130230 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 >> 172.21.15.59:0/2686929345 conn(0x7fbb140b06e0 msgr2=0x7fbb1415cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 shutdown_connections 2024-06-29T14:23:19.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:19.940+0000 7fbb1b438640 1 -- 172.21.15.59:0/2686929345 wait complete. 2024-06-29T14:23:19.949 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:23:19.950 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:23:20.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.069+0000 7f08c4f9e640 1 -- 172.21.15.59:0/3574022304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c006da00 msgr2=0x7f08c015f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.069+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/3574022304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c006da00 0x7f08c015f240 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f08c0070220 tx=0x7f08a802a530 comp rx=0 tx=0).stop 2024-06-29T14:23:20.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.069+0000 7f08c4f9e640 1 -- 172.21.15.59:0/3574022304 shutdown_connections 2024-06-29T14:23:20.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.069+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/3574022304 >> v2:172.21.15.153:3300/0 conn(0x7f08c01444c0 0x7f08c0073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.069+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/3574022304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c006da00 0x7f08c015f240 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.069+0000 7f08c4f9e640 1 -- 172.21.15.59:0/3574022304 >> 172.21.15.59:0/3574022304 conn(0x7f08c00b24c0 msgr2=0x7f08c00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.069+0000 7f08c4f9e640 1 -- 172.21.15.59:0/3574022304 shutdown_connections 2024-06-29T14:23:20.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.069+0000 7f08c4f9e640 1 -- 172.21.15.59:0/3574022304 wait complete. 2024-06-29T14:23:20.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.071+0000 7f08c4f9e640 1 Processor -- start 2024-06-29T14:23:20.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.071+0000 7f08c4f9e640 1 -- start start 2024-06-29T14:23:20.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08c4f9e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c00cd9b0 0x7f08c00b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08c4f9e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c01444c0 0x7f08c01f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08c4f9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c00cb260 0x7f08c006c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08c4f9e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f08c0073b40 con 0x7f08c01444c0 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08c4f9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08c00739c0 con 0x7f08c00cb260 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08c4f9e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f08c0073cc0 con 0x7f08c00cd9b0 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08bdd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c01444c0 0x7f08c01f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08bdd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c01444c0 0x7f08c01f2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48156/0 (socket says 172.21.15.59:48156) 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08bdd9b640 1 -- 172.21.15.59:0/192023995 learned_addr learned my addr 172.21.15.59:0/192023995 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.072+0000 7f08bed9d640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c00cb260 0x7f08c006c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08be59c640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c00cd9b0 0x7f08c00b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08bdd9b640 1 -- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c00cd9b0 msgr2=0x7f08c00b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08bdd9b640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c00cd9b0 0x7f08c00b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08bdd9b640 1 -- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c00cb260 msgr2=0x7f08c006c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08bdd9b640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c00cb260 0x7f08c006c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08bdd9b640 1 -- 172.21.15.59:0/192023995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c00b3d00 con 0x7f08c01444c0 2024-06-29T14:23:20.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08be59c640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c00cd9b0 0x7f08c00b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:20.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08bdd9b640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c01444c0 0x7f08c01f2970 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f08a802ab20 tx=0x7f08a8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08b6ffd640 1 -- 172.21.15.59:0/192023995 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8031d60 con 0x7f08c01444c0 2024-06-29T14:23:20.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08b6ffd640 1 -- 172.21.15.59:0/192023995 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08a8033e10 con 0x7f08c01444c0 2024-06-29T14:23:20.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c01f33f0 con 0x7f08c01444c0 2024-06-29T14:23:20.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08b6ffd640 1 -- 172.21.15.59:0/192023995 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8005260 con 0x7f08c01444c0 2024-06-29T14:23:20.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.073+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c00b2fc0 con 0x7f08c01444c0 2024-06-29T14:23:20.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.074+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c0072460 con 0x7f08c01444c0 2024-06-29T14:23:20.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.075+0000 7f08b6ffd640 1 -- 172.21.15.59:0/192023995 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f08a8005400 con 0x7f08c01444c0 2024-06-29T14:23:20.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.075+0000 7f08b6ffd640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08a0076300 0x7f08a0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.075+0000 7f08b6ffd640 1 -- 172.21.15.59:0/192023995 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f08a80b89e0 con 0x7f08c01444c0 2024-06-29T14:23:20.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.076+0000 7f08be59c640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08a0076300 0x7f08a0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.076+0000 7f08be59c640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08a0076300 0x7f08a0078810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f08b0007fd0 tx=0x7f08b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.080+0000 7f08b6ffd640 1 -- 172.21.15.59:0/192023995 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f08c0072460 con 0x7f08c01444c0 2024-06-29T14:23:20.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.267+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08c00b2db0 con 0x7f08a0076300 2024-06-29T14:23:20.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.272+0000 7f08b6ffd640 1 -- 172.21.15.59:0/192023995 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249106 (secure 0 0 0) 0x7f08c00b2db0 con 0x7f08a0076300 2024-06-29T14:23:20.276 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:20.278 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:23:20.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08a0076300 msgr2=0x7f08a0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08a0076300 0x7f08a0078810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f08b0007fd0 tx=0x7f08b0009040 comp rx=0 tx=0).stop 2024-06-29T14:23:20.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c01444c0 msgr2=0x7f08c01f2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c01444c0 0x7f08c01f2970 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f08a802ab20 tx=0x7f08a8033c20 comp rx=0 tx=0).stop 2024-06-29T14:23:20.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 shutdown_connections 2024-06-29T14:23:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f08a0076300 0x7f08a0078810 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c00cb260 0x7f08c006c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c01444c0 0x7f08c01f2970 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 --2- 172.21.15.59:0/192023995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c00cd9b0 0x7f08c00b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.275+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 >> 172.21.15.59:0/192023995 conn(0x7f08c00b24c0 msgr2=0x7f08c00b0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.276+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 shutdown_connections 2024-06-29T14:23:20.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.276+0000 7f08c4f9e640 1 -- 172.21.15.59:0/192023995 wait complete. 2024-06-29T14:23:20.285 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-06-29T14:23:19.752591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592084,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":116,"num_write_kb":1639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220812,"kb_used_data":4652,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753908,"statfs":{"total":773094113280,"available":772868001792,"internally_reserved":0,"allocated":4763648,"data_stored":2703342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1716,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.579212"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.211606+0000","last_change":"2024-06-29T14:23:16.201160+0000","last_active":"2024-06-29T14:23:16.211606+0000","last_peered":"2024-06-29T14:23:16.211606+0000","last_clean":"2024-06-29T14:23:16.211606+0000","last_became_active":"2024-06-29T14:23:16.200155+0000","last_became_peered":"2024-06-29T14:23:16.200155+0000","last_unstale":"2024-06-29T14:23:16.211606+0000","last_undegraded":"2024-06-29T14:23:16.211606+0000","last_fullsized":"2024-06-29T14:23:16.211606+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:23.684550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.199308+0000","last_change":"2024-06-29T14:23:16.189930+0000","last_active":"2024-06-29T14:23:16.199308+0000","last_peered":"2024-06-29T14:23:16.199308+0000","last_clean":"2024-06-29T14:23:16.199308+0000","last_became_active":"2024-06-29T14:23:16.189571+0000","last_became_peered":"2024-06-29T14:23:16.189571+0000","last_unstale":"2024-06-29T14:23:16.199308+0000","last_undegraded":"2024-06-29T14:23:16.199308+0000","last_fullsized":"2024-06-29T14:23:16.199308+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:12.662244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207289+0000","last_change":"2024-06-29T14:23:16.203182+0000","last_active":"2024-06-29T14:23:16.207289+0000","last_peered":"2024-06-29T14:23:16.207289+0000","last_clean":"2024-06-29T14:23:16.207289+0000","last_became_active":"2024-06-29T14:23:16.202851+0000","last_became_peered":"2024-06-29T14:23:16.202851+0000","last_unstale":"2024-06-29T14:23:16.207289+0000","last_undegraded":"2024-06-29T14:23:16.207289+0000","last_fullsized":"2024-06-29T14:23:16.207289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:13.507757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972956+0000","last_change":"2024-06-29T14:23:16.202834+0000","last_active":"2024-06-29T14:23:17.972956+0000","last_peered":"2024-06-29T14:23:17.972956+0000","last_clean":"2024-06-29T14:23:17.972956+0000","last_became_active":"2024-06-29T14:23:16.202357+0000","last_became_peered":"2024-06-29T14:23:16.202357+0000","last_unstale":"2024-06-29T14:23:17.972956+0000","last_undegraded":"2024-06-29T14:23:17.972956+0000","last_fullsized":"2024-06-29T14:23:17.972956+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:35.097600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972601+0000","last_change":"2024-06-29T14:23:16.194759+0000","last_active":"2024-06-29T14:23:17.972601+0000","last_peered":"2024-06-29T14:23:17.972601+0000","last_clean":"2024-06-29T14:23:17.972601+0000","last_became_active":"2024-06-29T14:23:16.194351+0000","last_became_peered":"2024-06-29T14:23:16.194351+0000","last_unstale":"2024-06-29T14:23:17.972601+0000","last_undegraded":"2024-06-29T14:23:17.972601+0000","last_fullsized":"2024-06-29T14:23:17.972601+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:30.838175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207114+0000","last_change":"2024-06-29T14:23:16.195834+0000","last_active":"2024-06-29T14:23:16.207114+0000","last_peered":"2024-06-29T14:23:16.207114+0000","last_clean":"2024-06-29T14:23:16.207114+0000","last_became_active":"2024-06-29T14:23:16.195476+0000","last_became_peered":"2024-06-29T14:23:16.195476+0000","last_unstale":"2024-06-29T14:23:16.207114+0000","last_undegraded":"2024-06-29T14:23:16.207114+0000","last_fullsized":"2024-06-29T14:23:16.207114+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:30.001157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192292+0000","last_change":"2024-06-29T14:23:15.192292+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192292+0000","last_undegraded":"2024-06-29T14:23:15.192292+0000","last_fullsized":"2024-06-29T14:23:15.192292+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.206121+0000","last_change":"2024-06-29T14:23:16.199825+0000","last_active":"2024-06-29T14:23:16.206121+0000","last_peered":"2024-06-29T14:23:16.206121+0000","last_clean":"2024-06-29T14:23:16.206121+0000","last_became_active":"2024-06-29T14:23:16.199499+0000","last_became_peered":"2024-06-29T14:23:16.199499+0000","last_unstale":"2024-06-29T14:23:16.206121+0000","last_undegraded":"2024-06-29T14:23:16.206121+0000","last_fullsized":"2024-06-29T14:23:16.206121+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:10.882976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214762+0000","last_change":"2024-06-29T14:23:16.209692+0000","last_active":"2024-06-29T14:23:16.214762+0000","last_peered":"2024-06-29T14:23:16.214762+0000","last_clean":"2024-06-29T14:23:16.214762+0000","last_became_active":"2024-06-29T14:23:16.202968+0000","last_became_peered":"2024-06-29T14:23:16.202968+0000","last_unstale":"2024-06-29T14:23:16.214762+0000","last_undegraded":"2024-06-29T14:23:16.214762+0000","last_fullsized":"2024-06-29T14:23:16.214762+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:10.402585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.194916+0000","last_change":"2024-06-29T14:23:15.194916+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.194916+0000","last_undegraded":"2024-06-29T14:23:15.194916+0000","last_fullsized":"2024-06-29T14:23:15.194916+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193674+0000","last_change":"2024-06-29T14:23:15.193674+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193674+0000","last_undegraded":"2024-06-29T14:23:15.193674+0000","last_fullsized":"2024-06-29T14:23:15.193674+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214775+0000","last_change":"2024-06-29T14:23:16.209122+0000","last_active":"2024-06-29T14:23:16.214775+0000","last_peered":"2024-06-29T14:23:16.214775+0000","last_clean":"2024-06-29T14:23:16.214775+0000","last_became_active":"2024-06-29T14:23:16.208783+0000","last_became_peered":"2024-06-29T14:23:16.208783+0000","last_unstale":"2024-06-29T14:23:16.214775+0000","last_undegraded":"2024-06-29T14:23:16.214775+0000","last_fullsized":"2024-06-29T14:23:16.214775+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:30.001157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.196595+0000","last_change":"2024-06-29T14:23:15.196595+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.196595+0000","last_undegraded":"2024-06-29T14:23:15.196595+0000","last_fullsized":"2024-06-29T14:23:15.196595+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.194142+0000","last_change":"2024-06-29T14:23:15.194142+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.194142+0000","last_undegraded":"2024-06-29T14:23:15.194142+0000","last_fullsized":"2024-06-29T14:23:15.194142+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191760+0000","last_change":"2024-06-29T14:23:15.191760+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191760+0000","last_undegraded":"2024-06-29T14:23:15.191760+0000","last_fullsized":"2024-06-29T14:23:15.191760+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189097+0000","last_change":"2024-06-29T14:23:15.189097+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189097+0000","last_undegraded":"2024-06-29T14:23:15.189097+0000","last_fullsized":"2024-06-29T14:23:15.189097+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188330+0000","last_change":"2024-06-29T14:23:15.188330+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188330+0000","last_undegraded":"2024-06-29T14:23:15.188330+0000","last_fullsized":"2024-06-29T14:23:15.188330+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.184396+0000","last_change":"2024-06-29T14:23:15.184396+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.184396+0000","last_undegraded":"2024-06-29T14:23:15.184396+0000","last_fullsized":"2024-06-29T14:23:15.184396+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972388+0000","last_change":"2024-06-29T14:23:16.190111+0000","last_active":"2024-06-29T14:23:17.972388+0000","last_peered":"2024-06-29T14:23:17.972388+0000","last_clean":"2024-06-29T14:23:17.972388+0000","last_became_active":"2024-06-29T14:23:16.189786+0000","last_became_peered":"2024-06-29T14:23:16.189786+0000","last_unstale":"2024-06-29T14:23:17.972388+0000","last_undegraded":"2024-06-29T14:23:17.972388+0000","last_fullsized":"2024-06-29T14:23:17.972388+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:12.500255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.184382+0000","last_change":"2024-06-29T14:23:15.184382+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.184382+0000","last_undegraded":"2024-06-29T14:23:15.184382+0000","last_fullsized":"2024-06-29T14:23:15.184382+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.190853+0000","last_change":"2024-06-29T14:23:15.190853+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.190853+0000","last_undegraded":"2024-06-29T14:23:15.190853+0000","last_fullsized":"2024-06-29T14:23:15.190853+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193127+0000","last_change":"2024-06-29T14:23:15.193127+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193127+0000","last_undegraded":"2024-06-29T14:23:15.193127+0000","last_fullsized":"2024-06-29T14:23:15.193127+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188344+0000","last_change":"2024-06-29T14:23:15.188344+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188344+0000","last_undegraded":"2024-06-29T14:23:15.188344+0000","last_fullsized":"2024-06-29T14:23:15.188344+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972485+0000","last_change":"2024-06-29T14:23:16.210617+0000","last_active":"2024-06-29T14:23:17.972485+0000","last_peered":"2024-06-29T14:23:17.972485+0000","last_clean":"2024-06-29T14:23:17.972485+0000","last_became_active":"2024-06-29T14:23:16.210242+0000","last_became_peered":"2024-06-29T14:23:16.210242+0000","last_unstale":"2024-06-29T14:23:17.972485+0000","last_undegraded":"2024-06-29T14:23:17.972485+0000","last_fullsized":"2024-06-29T14:23:17.972485+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:21.067086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185688+0000","last_change":"2024-06-29T14:23:15.185688+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185688+0000","last_undegraded":"2024-06-29T14:23:15.185688+0000","last_fullsized":"2024-06-29T14:23:15.185688+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207195+0000","last_change":"2024-06-29T14:23:16.205741+0000","last_active":"2024-06-29T14:23:16.207195+0000","last_peered":"2024-06-29T14:23:16.207195+0000","last_clean":"2024-06-29T14:23:16.207195+0000","last_became_active":"2024-06-29T14:23:16.205351+0000","last_became_peered":"2024-06-29T14:23:16.205351+0000","last_unstale":"2024-06-29T14:23:16.207195+0000","last_undegraded":"2024-06-29T14:23:16.207195+0000","last_fullsized":"2024-06-29T14:23:16.207195+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:10.402585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185734+0000","last_change":"2024-06-29T14:23:15.185734+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185734+0000","last_undegraded":"2024-06-29T14:23:15.185734+0000","last_fullsized":"2024-06-29T14:23:15.185734+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192802+0000","last_change":"2024-06-29T14:23:15.192802+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192802+0000","last_undegraded":"2024-06-29T14:23:15.192802+0000","last_fullsized":"2024-06-29T14:23:15.192802+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.186290+0000","last_change":"2024-06-29T14:23:15.186290+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.186290+0000","last_undegraded":"2024-06-29T14:23:15.186290+0000","last_fullsized":"2024-06-29T14:23:15.186290+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185611+0000","last_change":"2024-06-29T14:23:15.185611+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185611+0000","last_undegraded":"2024-06-29T14:23:15.185611+0000","last_fullsized":"2024-06-29T14:23:15.185611+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187368+0000","last_change":"2024-06-29T14:23:15.187368+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187368+0000","last_undegraded":"2024-06-29T14:23:15.187368+0000","last_fullsized":"2024-06-29T14:23:15.187368+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207193+0000","last_change":"2024-06-29T14:23:16.204276+0000","last_active":"2024-06-29T14:23:16.207193+0000","last_peered":"2024-06-29T14:23:16.207193+0000","last_clean":"2024-06-29T14:23:16.207193+0000","last_became_active":"2024-06-29T14:23:16.203937+0000","last_became_peered":"2024-06-29T14:23:16.203937+0000","last_unstale":"2024-06-29T14:23:16.207193+0000","last_undegraded":"2024-06-29T14:23:16.207193+0000","last_fullsized":"2024-06-29T14:23:16.207193+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:40.723736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188473+0000","last_change":"2024-06-29T14:23:15.188473+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188473+0000","last_undegraded":"2024-06-29T14:23:15.188473+0000","last_fullsized":"2024-06-29T14:23:15.188473+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.199746+0000","last_change":"2024-06-29T14:23:15.197029+0000","last_active":"2024-06-29T14:23:15.199746+0000","last_peered":"2024-06-29T14:23:15.199746+0000","last_clean":"2024-06-29T14:23:15.199746+0000","last_became_active":"2024-06-29T14:23:15.196636+0000","last_became_peered":"2024-06-29T14:23:15.196636+0000","last_unstale":"2024-06-29T14:23:15.199746+0000","last_undegraded":"2024-06-29T14:23:15.199746+0000","last_fullsized":"2024-06-29T14:23:15.199746+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:29.832584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.208350+0000","last_change":"2024-06-29T14:23:16.206697+0000","last_active":"2024-06-29T14:23:16.208350+0000","last_peered":"2024-06-29T14:23:16.208350+0000","last_clean":"2024-06-29T14:23:16.208350+0000","last_became_active":"2024-06-29T14:23:16.206325+0000","last_became_peered":"2024-06-29T14:23:16.206325+0000","last_unstale":"2024-06-29T14:23:16.208350+0000","last_undegraded":"2024-06-29T14:23:16.208350+0000","last_fullsized":"2024-06-29T14:23:16.208350+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"40'1","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217808+0000","last_change":"2024-06-29T14:23:15.204215+0000","last_active":"2024-06-29T14:23:16.217808+0000","last_peered":"2024-06-29T14:23:16.217808+0000","last_clean":"2024-06-29T14:23:16.217808+0000","last_became_active":"2024-06-29T14:23:15.203861+0000","last_became_peered":"2024-06-29T14:23:15.203861+0000","last_unstale":"2024-06-29T14:23:16.217808+0000","last_undegraded":"2024-06-29T14:23:16.217808+0000","last_fullsized":"2024-06-29T14:23:16.217808+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:15.271848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189311+0000","last_change":"2024-06-29T14:23:15.189311+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189311+0000","last_undegraded":"2024-06-29T14:23:15.189311+0000","last_fullsized":"2024-06-29T14:23:15.189311+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"40'2","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.219346+0000","last_change":"2024-06-29T14:23:15.205028+0000","last_active":"2024-06-29T14:23:16.219346+0000","last_peered":"2024-06-29T14:23:16.219346+0000","last_clean":"2024-06-29T14:23:16.219346+0000","last_became_active":"2024-06-29T14:23:15.204734+0000","last_became_peered":"2024-06-29T14:23:15.204734+0000","last_unstale":"2024-06-29T14:23:16.219346+0000","last_undegraded":"2024-06-29T14:23:16.219346+0000","last_fullsized":"2024-06-29T14:23:16.219346+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:12.502166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.198933+0000","last_change":"2024-06-29T14:23:16.190648+0000","last_active":"2024-06-29T14:23:16.198933+0000","last_peered":"2024-06-29T14:23:16.198933+0000","last_clean":"2024-06-29T14:23:16.198933+0000","last_became_active":"2024-06-29T14:23:16.190331+0000","last_became_peered":"2024-06-29T14:23:16.190331+0000","last_unstale":"2024-06-29T14:23:16.198933+0000","last_undegraded":"2024-06-29T14:23:16.198933+0000","last_fullsized":"2024-06-29T14:23:16.198933+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:52.309099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201353+0000","last_change":"2024-06-29T14:23:15.195155+0000","last_active":"2024-06-29T14:23:15.201353+0000","last_peered":"2024-06-29T14:23:15.201353+0000","last_clean":"2024-06-29T14:23:15.201353+0000","last_became_active":"2024-06-29T14:23:15.194809+0000","last_became_peered":"2024-06-29T14:23:15.194809+0000","last_unstale":"2024-06-29T14:23:15.201353+0000","last_undegraded":"2024-06-29T14:23:15.201353+0000","last_fullsized":"2024-06-29T14:23:15.201353+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:29.832584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214639+0000","last_change":"2024-06-29T14:23:16.209386+0000","last_active":"2024-06-29T14:23:16.214639+0000","last_peered":"2024-06-29T14:23:16.214639+0000","last_clean":"2024-06-29T14:23:16.214639+0000","last_became_active":"2024-06-29T14:23:16.209011+0000","last_became_peered":"2024-06-29T14:23:16.209011+0000","last_unstale":"2024-06-29T14:23:16.214639+0000","last_undegraded":"2024-06-29T14:23:16.214639+0000","last_fullsized":"2024-06-29T14:23:16.214639+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:40.723736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203843+0000","last_change":"2024-06-29T14:23:15.201776+0000","last_active":"2024-06-29T14:23:15.203843+0000","last_peered":"2024-06-29T14:23:15.203843+0000","last_clean":"2024-06-29T14:23:15.203843+0000","last_became_active":"2024-06-29T14:23:15.201376+0000","last_became_peered":"2024-06-29T14:23:15.201376+0000","last_unstale":"2024-06-29T14:23:15.203843+0000","last_undegraded":"2024-06-29T14:23:15.203843+0000","last_fullsized":"2024-06-29T14:23:15.203843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191269+0000","last_change":"2024-06-29T14:23:15.191269+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191269+0000","last_undegraded":"2024-06-29T14:23:15.191269+0000","last_fullsized":"2024-06-29T14:23:15.191269+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.204276+0000","last_change":"2024-06-29T14:23:15.203169+0000","last_active":"2024-06-29T14:23:15.204276+0000","last_peered":"2024-06-29T14:23:15.204276+0000","last_clean":"2024-06-29T14:23:15.204276+0000","last_became_active":"2024-06-29T14:23:15.202809+0000","last_became_peered":"2024-06-29T14:23:15.202809+0000","last_unstale":"2024-06-29T14:23:15.204276+0000","last_undegraded":"2024-06-29T14:23:15.204276+0000","last_fullsized":"2024-06-29T14:23:15.204276+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:29.639570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191458+0000","last_change":"2024-06-29T14:23:15.191458+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191458+0000","last_undegraded":"2024-06-29T14:23:15.191458+0000","last_fullsized":"2024-06-29T14:23:15.191458+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185188+0000","last_change":"2024-06-29T14:23:15.200889+0000","last_active":"2024-06-29T14:23:16.185188+0000","last_peered":"2024-06-29T14:23:16.185188+0000","last_clean":"2024-06-29T14:23:16.185188+0000","last_became_active":"2024-06-29T14:23:15.200521+0000","last_became_peered":"2024-06-29T14:23:15.200521+0000","last_unstale":"2024-06-29T14:23:16.185188+0000","last_undegraded":"2024-06-29T14:23:16.185188+0000","last_fullsized":"2024-06-29T14:23:16.185188+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217265+0000","last_change":"2024-06-29T14:23:16.207437+0000","last_active":"2024-06-29T14:23:16.217265+0000","last_peered":"2024-06-29T14:23:16.217265+0000","last_clean":"2024-06-29T14:23:16.217265+0000","last_became_active":"2024-06-29T14:23:16.207056+0000","last_became_peered":"2024-06-29T14:23:16.207056+0000","last_unstale":"2024-06-29T14:23:16.217265+0000","last_undegraded":"2024-06-29T14:23:16.217265+0000","last_fullsized":"2024-06-29T14:23:16.217265+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:52.297506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203620+0000","last_change":"2024-06-29T14:23:15.199990+0000","last_active":"2024-06-29T14:23:15.203620+0000","last_peered":"2024-06-29T14:23:15.203620+0000","last_clean":"2024-06-29T14:23:15.203620+0000","last_became_active":"2024-06-29T14:23:15.199697+0000","last_became_peered":"2024-06-29T14:23:15.199697+0000","last_unstale":"2024-06-29T14:23:15.203620+0000","last_undegraded":"2024-06-29T14:23:15.203620+0000","last_fullsized":"2024-06-29T14:23:15.203620+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:54.406021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.973018+0000","last_change":"2024-06-29T14:23:16.197089+0000","last_active":"2024-06-29T14:23:17.973018+0000","last_peered":"2024-06-29T14:23:17.973018+0000","last_clean":"2024-06-29T14:23:17.973018+0000","last_became_active":"2024-06-29T14:23:16.196730+0000","last_became_peered":"2024-06-29T14:23:16.196730+0000","last_unstale":"2024-06-29T14:23:17.973018+0000","last_undegraded":"2024-06-29T14:23:17.973018+0000","last_fullsized":"2024-06-29T14:23:17.973018+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:55.411612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.182345+0000","last_change":"2024-06-29T14:23:15.189674+0000","last_active":"2024-06-29T14:23:16.182345+0000","last_peered":"2024-06-29T14:23:16.182345+0000","last_clean":"2024-06-29T14:23:16.182345+0000","last_became_active":"2024-06-29T14:23:15.189326+0000","last_became_peered":"2024-06-29T14:23:15.189326+0000","last_unstale":"2024-06-29T14:23:16.182345+0000","last_undegraded":"2024-06-29T14:23:16.182345+0000","last_fullsized":"2024-06-29T14:23:16.182345+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:09.396994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.190068+0000","last_change":"2024-06-29T14:23:15.190068+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.190068+0000","last_undegraded":"2024-06-29T14:23:15.190068+0000","last_fullsized":"2024-06-29T14:23:15.190068+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.207378+0000","last_change":"2024-06-29T14:23:15.205426+0000","last_active":"2024-06-29T14:23:15.207378+0000","last_peered":"2024-06-29T14:23:15.207378+0000","last_clean":"2024-06-29T14:23:15.207378+0000","last_became_active":"2024-06-29T14:23:15.204970+0000","last_became_peered":"2024-06-29T14:23:15.204970+0000","last_unstale":"2024-06-29T14:23:15.207378+0000","last_undegraded":"2024-06-29T14:23:15.207378+0000","last_fullsized":"2024-06-29T14:23:15.207378+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:12.502166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.213758+0000","last_change":"2024-06-29T14:23:16.208578+0000","last_active":"2024-06-29T14:23:16.213758+0000","last_peered":"2024-06-29T14:23:16.213758+0000","last_clean":"2024-06-29T14:23:16.213758+0000","last_became_active":"2024-06-29T14:23:16.208189+0000","last_became_peered":"2024-06-29T14:23:16.208189+0000","last_unstale":"2024-06-29T14:23:16.213758+0000","last_undegraded":"2024-06-29T14:23:16.213758+0000","last_fullsized":"2024-06-29T14:23:16.213758+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:08.341184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203812+0000","last_change":"2024-06-29T14:23:15.202041+0000","last_active":"2024-06-29T14:23:15.203812+0000","last_peered":"2024-06-29T14:23:15.203812+0000","last_clean":"2024-06-29T14:23:15.203812+0000","last_became_active":"2024-06-29T14:23:15.201722+0000","last_became_peered":"2024-06-29T14:23:15.201722+0000","last_unstale":"2024-06-29T14:23:15.203812+0000","last_undegraded":"2024-06-29T14:23:15.203812+0000","last_fullsized":"2024-06-29T14:23:15.203812+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:18.072309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191167+0000","last_change":"2024-06-29T14:23:15.191167+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191167+0000","last_undegraded":"2024-06-29T14:23:15.191167+0000","last_fullsized":"2024-06-29T14:23:15.191167+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"40'6","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.220943+0000","last_change":"2024-06-29T14:23:15.199353+0000","last_active":"2024-06-29T14:23:16.220943+0000","last_peered":"2024-06-29T14:23:16.220943+0000","last_clean":"2024-06-29T14:23:16.220943+0000","last_became_active":"2024-06-29T14:23:15.198339+0000","last_became_peered":"2024-06-29T14:23:15.198339+0000","last_unstale":"2024-06-29T14:23:16.220943+0000","last_undegraded":"2024-06-29T14:23:16.220943+0000","last_fullsized":"2024-06-29T14:23:16.220943+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:39.718146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214670+0000","last_change":"2024-06-29T14:23:16.209778+0000","last_active":"2024-06-29T14:23:16.214670+0000","last_peered":"2024-06-29T14:23:16.214670+0000","last_clean":"2024-06-29T14:23:16.214670+0000","last_became_active":"2024-06-29T14:23:16.209443+0000","last_became_peered":"2024-06-29T14:23:16.209443+0000","last_unstale":"2024-06-29T14:23:16.214670+0000","last_undegraded":"2024-06-29T14:23:16.214670+0000","last_fullsized":"2024-06-29T14:23:16.214670+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:55.671459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"40'2","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.219006+0000","last_change":"2024-06-29T14:23:15.200701+0000","last_active":"2024-06-29T14:23:16.219006+0000","last_peered":"2024-06-29T14:23:16.219006+0000","last_clean":"2024-06-29T14:23:16.219006+0000","last_became_active":"2024-06-29T14:23:15.200338+0000","last_became_peered":"2024-06-29T14:23:15.200338+0000","last_unstale":"2024-06-29T14:23:16.219006+0000","last_undegraded":"2024-06-29T14:23:16.219006+0000","last_fullsized":"2024-06-29T14:23:16.219006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:54.665868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.201400+0000","last_change":"2024-06-29T14:23:16.196448+0000","last_active":"2024-06-29T14:23:16.201400+0000","last_peered":"2024-06-29T14:23:16.201400+0000","last_clean":"2024-06-29T14:23:16.201400+0000","last_became_active":"2024-06-29T14:23:16.195006+0000","last_became_peered":"2024-06-29T14:23:16.195006+0000","last_unstale":"2024-06-29T14:23:16.201400+0000","last_undegraded":"2024-06-29T14:23:16.201400+0000","last_fullsized":"2024-06-29T14:23:16.201400+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:05.074083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185776+0000","last_change":"2024-06-29T14:23:15.199955+0000","last_active":"2024-06-29T14:23:16.185776+0000","last_peered":"2024-06-29T14:23:16.185776+0000","last_clean":"2024-06-29T14:23:16.185776+0000","last_became_active":"2024-06-29T14:23:15.199619+0000","last_became_peered":"2024-06-29T14:23:15.199619+0000","last_unstale":"2024-06-29T14:23:16.185776+0000","last_undegraded":"2024-06-29T14:23:16.185776+0000","last_fullsized":"2024-06-29T14:23:16.185776+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:04.589609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187426+0000","last_change":"2024-06-29T14:23:15.187426+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187426+0000","last_undegraded":"2024-06-29T14:23:15.187426+0000","last_fullsized":"2024-06-29T14:23:15.187426+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.198643+0000","last_change":"2024-06-29T14:23:15.196439+0000","last_active":"2024-06-29T14:23:15.198643+0000","last_peered":"2024-06-29T14:23:15.198643+0000","last_clean":"2024-06-29T14:23:15.198643+0000","last_became_active":"2024-06-29T14:23:15.196116+0000","last_became_peered":"2024-06-29T14:23:15.196116+0000","last_unstale":"2024-06-29T14:23:15.198643+0000","last_undegraded":"2024-06-29T14:23:15.198643+0000","last_fullsized":"2024-06-29T14:23:15.198643+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:20.061496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.213976+0000","last_change":"2024-06-29T14:23:16.200847+0000","last_active":"2024-06-29T14:23:16.213976+0000","last_peered":"2024-06-29T14:23:16.213976+0000","last_clean":"2024-06-29T14:23:16.213976+0000","last_became_active":"2024-06-29T14:23:16.200264+0000","last_became_peered":"2024-06-29T14:23:16.200264+0000","last_unstale":"2024-06-29T14:23:16.213976+0000","last_undegraded":"2024-06-29T14:23:16.213976+0000","last_fullsized":"2024-06-29T14:23:16.213976+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:46.554739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202335+0000","last_change":"2024-06-29T14:23:15.200739+0000","last_active":"2024-06-29T14:23:15.202335+0000","last_peered":"2024-06-29T14:23:15.202335+0000","last_clean":"2024-06-29T14:23:15.202335+0000","last_became_active":"2024-06-29T14:23:15.200401+0000","last_became_peered":"2024-06-29T14:23:15.200401+0000","last_unstale":"2024-06-29T14:23:15.202335+0000","last_undegraded":"2024-06-29T14:23:15.202335+0000","last_fullsized":"2024-06-29T14:23:15.202335+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:49.969939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.194644+0000","last_change":"2024-06-29T14:23:16.188970+0000","last_active":"2024-06-29T14:23:16.194644+0000","last_peered":"2024-06-29T14:23:16.194644+0000","last_clean":"2024-06-29T14:23:16.194644+0000","last_became_active":"2024-06-29T14:23:16.188588+0000","last_became_peered":"2024-06-29T14:23:16.188588+0000","last_unstale":"2024-06-29T14:23:16.194644+0000","last_undegraded":"2024-06-29T14:23:16.194644+0000","last_fullsized":"2024-06-29T14:23:16.194644+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201969+0000","last_change":"2024-06-29T14:23:15.199255+0000","last_active":"2024-06-29T14:23:15.201969+0000","last_peered":"2024-06-29T14:23:15.201969+0000","last_clean":"2024-06-29T14:23:15.201969+0000","last_became_active":"2024-06-29T14:23:15.198806+0000","last_became_peered":"2024-06-29T14:23:15.198806+0000","last_unstale":"2024-06-29T14:23:15.201969+0000","last_undegraded":"2024-06-29T14:23:15.201969+0000","last_fullsized":"2024-06-29T14:23:15.201969+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:41.669608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193179+0000","last_change":"2024-06-29T14:23:15.193179+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193179+0000","last_undegraded":"2024-06-29T14:23:15.193179+0000","last_fullsized":"2024-06-29T14:23:15.193179+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203612+0000","last_change":"2024-06-29T14:23:15.200790+0000","last_active":"2024-06-29T14:23:15.203612+0000","last_peered":"2024-06-29T14:23:15.203612+0000","last_clean":"2024-06-29T14:23:15.203612+0000","last_became_active":"2024-06-29T14:23:15.200452+0000","last_became_peered":"2024-06-29T14:23:15.200452+0000","last_unstale":"2024-06-29T14:23:15.203612+0000","last_undegraded":"2024-06-29T14:23:15.203612+0000","last_fullsized":"2024-06-29T14:23:15.203612+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:27.766774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192686+0000","last_change":"2024-06-29T14:23:15.192686+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192686+0000","last_undegraded":"2024-06-29T14:23:15.192686+0000","last_fullsized":"2024-06-29T14:23:15.192686+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972729+0000","last_change":"2024-06-29T14:23:15.195356+0000","last_active":"2024-06-29T14:23:17.972729+0000","last_peered":"2024-06-29T14:23:17.972729+0000","last_clean":"2024-06-29T14:23:17.972729+0000","last_became_active":"2024-06-29T14:23:15.195037+0000","last_became_peered":"2024-06-29T14:23:15.195037+0000","last_unstale":"2024-06-29T14:23:17.972729+0000","last_undegraded":"2024-06-29T14:23:17.972729+0000","last_fullsized":"2024-06-29T14:23:17.972729+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:32.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971916+0000","last_change":"2024-06-29T14:23:16.197431+0000","last_active":"2024-06-29T14:23:17.971916+0000","last_peered":"2024-06-29T14:23:17.971916+0000","last_clean":"2024-06-29T14:23:17.971916+0000","last_became_active":"2024-06-29T14:23:16.197091+0000","last_became_peered":"2024-06-29T14:23:16.197091+0000","last_unstale":"2024-06-29T14:23:17.971916+0000","last_undegraded":"2024-06-29T14:23:17.971916+0000","last_fullsized":"2024-06-29T14:23:17.971916+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:46.040585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185494+0000","last_change":"2024-06-29T14:23:15.203782+0000","last_active":"2024-06-29T14:23:16.185494+0000","last_peered":"2024-06-29T14:23:16.185494+0000","last_clean":"2024-06-29T14:23:16.185494+0000","last_became_active":"2024-06-29T14:23:15.203451+0000","last_became_peered":"2024-06-29T14:23:15.203451+0000","last_unstale":"2024-06-29T14:23:16.185494+0000","last_undegraded":"2024-06-29T14:23:16.185494+0000","last_fullsized":"2024-06-29T14:23:16.185494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:18.072309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187906+0000","last_change":"2024-06-29T14:23:15.187906+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187906+0000","last_undegraded":"2024-06-29T14:23:15.187906+0000","last_fullsized":"2024-06-29T14:23:15.187906+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972722+0000","last_change":"2024-06-29T14:23:15.201493+0000","last_active":"2024-06-29T14:23:17.972722+0000","last_peered":"2024-06-29T14:23:17.972722+0000","last_clean":"2024-06-29T14:23:17.972722+0000","last_became_active":"2024-06-29T14:23:15.201150+0000","last_became_peered":"2024-06-29T14:23:15.201150+0000","last_unstale":"2024-06-29T14:23:17.972722+0000","last_undegraded":"2024-06-29T14:23:17.972722+0000","last_fullsized":"2024-06-29T14:23:17.972722+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:00.323663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.206186+0000","last_change":"2024-06-29T14:23:16.202653+0000","last_active":"2024-06-29T14:23:16.206186+0000","last_peered":"2024-06-29T14:23:16.206186+0000","last_clean":"2024-06-29T14:23:16.206186+0000","last_became_active":"2024-06-29T14:23:16.202279+0000","last_became_peered":"2024-06-29T14:23:16.202279+0000","last_unstale":"2024-06-29T14:23:16.206186+0000","last_undegraded":"2024-06-29T14:23:16.206186+0000","last_fullsized":"2024-06-29T14:23:16.206186+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:05.595200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971710+0000","last_change":"2024-06-29T14:23:15.196760+0000","last_active":"2024-06-29T14:23:17.971710+0000","last_peered":"2024-06-29T14:23:17.971710+0000","last_clean":"2024-06-29T14:23:17.971710+0000","last_became_active":"2024-06-29T14:23:15.196278+0000","last_became_peered":"2024-06-29T14:23:15.196278+0000","last_unstale":"2024-06-29T14:23:17.971710+0000","last_undegraded":"2024-06-29T14:23:17.971710+0000","last_fullsized":"2024-06-29T14:23:17.971710+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:32:01.571661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193709+0000","last_change":"2024-06-29T14:23:15.193709+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193709+0000","last_undegraded":"2024-06-29T14:23:15.193709+0000","last_fullsized":"2024-06-29T14:23:15.193709+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202585+0000","last_change":"2024-06-29T14:23:15.199837+0000","last_active":"2024-06-29T14:23:15.202585+0000","last_peered":"2024-06-29T14:23:15.202585+0000","last_clean":"2024-06-29T14:23:15.202585+0000","last_became_active":"2024-06-29T14:23:15.199539+0000","last_became_peered":"2024-06-29T14:23:15.199539+0000","last_unstale":"2024-06-29T14:23:15.202585+0000","last_undegraded":"2024-06-29T14:23:15.202585+0000","last_fullsized":"2024-06-29T14:23:15.202585+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:12.157450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217291+0000","last_change":"2024-06-29T14:23:16.211956+0000","last_active":"2024-06-29T14:23:16.217291+0000","last_peered":"2024-06-29T14:23:16.217291+0000","last_clean":"2024-06-29T14:23:16.217291+0000","last_became_active":"2024-06-29T14:23:16.211649+0000","last_became_peered":"2024-06-29T14:23:16.211649+0000","last_unstale":"2024-06-29T14:23:16.217291+0000","last_undegraded":"2024-06-29T14:23:16.217291+0000","last_fullsized":"2024-06-29T14:23:16.217291+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:37.213779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972618+0000","last_change":"2024-06-29T14:23:15.201805+0000","last_active":"2024-06-29T14:23:17.972618+0000","last_peered":"2024-06-29T14:23:17.972618+0000","last_clean":"2024-06-29T14:23:17.972618+0000","last_became_active":"2024-06-29T14:23:15.200924+0000","last_became_peered":"2024-06-29T14:23:15.200924+0000","last_unstale":"2024-06-29T14:23:17.972618+0000","last_undegraded":"2024-06-29T14:23:17.972618+0000","last_fullsized":"2024-06-29T14:23:17.972618+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:18.651137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207150+0000","last_change":"2024-06-29T14:23:16.205768+0000","last_active":"2024-06-29T14:23:16.207150+0000","last_peered":"2024-06-29T14:23:16.207150+0000","last_clean":"2024-06-29T14:23:16.207150+0000","last_became_active":"2024-06-29T14:23:16.205382+0000","last_became_peered":"2024-06-29T14:23:16.205382+0000","last_unstale":"2024-06-29T14:23:16.207150+0000","last_undegraded":"2024-06-29T14:23:16.207150+0000","last_fullsized":"2024-06-29T14:23:16.207150+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:55.671459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.205384+0000","last_change":"2024-06-29T14:23:15.203975+0000","last_active":"2024-06-29T14:23:15.205384+0000","last_peered":"2024-06-29T14:23:15.205384+0000","last_clean":"2024-06-29T14:23:15.205384+0000","last_became_active":"2024-06-29T14:23:15.203652+0000","last_became_peered":"2024-06-29T14:23:15.203652+0000","last_unstale":"2024-06-29T14:23:15.205384+0000","last_undegraded":"2024-06-29T14:23:15.205384+0000","last_fullsized":"2024-06-29T14:23:15.205384+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:04.589609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189204+0000","last_change":"2024-06-29T14:23:15.189204+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189204+0000","last_undegraded":"2024-06-29T14:23:15.189204+0000","last_fullsized":"2024-06-29T14:23:15.189204+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"41'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:18.201843+0000","last_change":"2024-06-29T14:23:15.199068+0000","last_active":"2024-06-29T14:23:18.201843+0000","last_peered":"2024-06-29T14:23:18.201843+0000","last_clean":"2024-06-29T14:23:18.201843+0000","last_became_active":"2024-06-29T14:23:15.198714+0000","last_became_peered":"2024-06-29T14:23:15.198714+0000","last_unstale":"2024-06-29T14:23:18.201843+0000","last_undegraded":"2024-06-29T14:23:18.201843+0000","last_fullsized":"2024-06-29T14:23:18.201843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:11.722673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.199343+0000","last_change":"2024-06-29T14:23:16.194459+0000","last_active":"2024-06-29T14:23:16.199343+0000","last_peered":"2024-06-29T14:23:16.199343+0000","last_clean":"2024-06-29T14:23:16.199343+0000","last_became_active":"2024-06-29T14:23:16.194084+0000","last_became_peered":"2024-06-29T14:23:16.194084+0000","last_unstale":"2024-06-29T14:23:16.199343+0000","last_undegraded":"2024-06-29T14:23:16.199343+0000","last_fullsized":"2024-06-29T14:23:16.199343+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:46.325166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.192313+0000","last_change":"2024-06-29T14:23:15.199199+0000","last_active":"2024-06-29T14:23:16.192313+0000","last_peered":"2024-06-29T14:23:16.192313+0000","last_clean":"2024-06-29T14:23:16.192313+0000","last_became_active":"2024-06-29T14:23:15.198802+0000","last_became_peered":"2024-06-29T14:23:15.198802+0000","last_unstale":"2024-06-29T14:23:16.192313+0000","last_undegraded":"2024-06-29T14:23:16.192313+0000","last_fullsized":"2024-06-29T14:23:16.192313+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:41.669608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207171+0000","last_change":"2024-06-29T14:23:16.200965+0000","last_active":"2024-06-29T14:23:16.207171+0000","last_peered":"2024-06-29T14:23:16.207171+0000","last_clean":"2024-06-29T14:23:16.207171+0000","last_became_active":"2024-06-29T14:23:16.200584+0000","last_became_peered":"2024-06-29T14:23:16.200584+0000","last_unstale":"2024-06-29T14:23:16.207171+0000","last_undegraded":"2024-06-29T14:23:16.207171+0000","last_fullsized":"2024-06-29T14:23:16.207171+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:19.077899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.194138+0000","last_change":"2024-06-29T14:23:15.203329+0000","last_active":"2024-06-29T14:23:16.194138+0000","last_peered":"2024-06-29T14:23:16.194138+0000","last_clean":"2024-06-29T14:23:16.194138+0000","last_became_active":"2024-06-29T14:23:15.202429+0000","last_became_peered":"2024-06-29T14:23:15.202429+0000","last_unstale":"2024-06-29T14:23:16.194138+0000","last_undegraded":"2024-06-29T14:23:16.194138+0000","last_fullsized":"2024-06-29T14:23:16.194138+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.198488+0000","last_change":"2024-06-29T14:23:16.190685+0000","last_active":"2024-06-29T14:23:16.198488+0000","last_peered":"2024-06-29T14:23:16.198488+0000","last_clean":"2024-06-29T14:23:16.198488+0000","last_became_active":"2024-06-29T14:23:16.190310+0000","last_became_peered":"2024-06-29T14:23:16.190310+0000","last_unstale":"2024-06-29T14:23:16.198488+0000","last_undegraded":"2024-06-29T14:23:16.198488+0000","last_fullsized":"2024-06-29T14:23:16.198488+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:08.341184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202801+0000","last_change":"2024-06-29T14:23:15.200968+0000","last_active":"2024-06-29T14:23:15.202801+0000","last_peered":"2024-06-29T14:23:15.202801+0000","last_clean":"2024-06-29T14:23:15.202801+0000","last_became_active":"2024-06-29T14:23:15.200573+0000","last_became_peered":"2024-06-29T14:23:15.200573+0000","last_unstale":"2024-06-29T14:23:15.202801+0000","last_undegraded":"2024-06-29T14:23:15.202801+0000","last_fullsized":"2024-06-29T14:23:15.202801+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:34.092009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193645+0000","last_change":"2024-06-29T14:23:15.193645+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193645+0000","last_undegraded":"2024-06-29T14:23:15.193645+0000","last_fullsized":"2024-06-29T14:23:15.193645+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201435+0000","last_change":"2024-06-29T14:23:15.197575+0000","last_active":"2024-06-29T14:23:15.201435+0000","last_peered":"2024-06-29T14:23:15.201435+0000","last_clean":"2024-06-29T14:23:15.201435+0000","last_became_active":"2024-06-29T14:23:15.197267+0000","last_became_peered":"2024-06-29T14:23:15.197267+0000","last_unstale":"2024-06-29T14:23:15.201435+0000","last_undegraded":"2024-06-29T14:23:15.201435+0000","last_fullsized":"2024-06-29T14:23:15.201435+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:34.092009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187502+0000","last_change":"2024-06-29T14:23:15.187502+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187502+0000","last_undegraded":"2024-06-29T14:23:15.187502+0000","last_fullsized":"2024-06-29T14:23:15.187502+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.181400+0000","last_change":"2024-06-29T14:23:15.192161+0000","last_active":"2024-06-29T14:23:16.181400+0000","last_peered":"2024-06-29T14:23:16.181400+0000","last_clean":"2024-06-29T14:23:16.181400+0000","last_became_active":"2024-06-29T14:23:15.191810+0000","last_became_peered":"2024-06-29T14:23:15.191810+0000","last_unstale":"2024-06-29T14:23:16.181400+0000","last_undegraded":"2024-06-29T14:23:16.181400+0000","last_fullsized":"2024-06-29T14:23:16.181400+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:28.995567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.197944+0000","last_change":"2024-06-29T14:23:15.197944+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.197944+0000","last_undegraded":"2024-06-29T14:23:15.197944+0000","last_fullsized":"2024-06-29T14:23:15.197944+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202122+0000","last_change":"2024-06-29T14:23:15.198041+0000","last_active":"2024-06-29T14:23:15.202122+0000","last_peered":"2024-06-29T14:23:15.202122+0000","last_clean":"2024-06-29T14:23:15.202122+0000","last_became_active":"2024-06-29T14:23:15.197662+0000","last_became_peered":"2024-06-29T14:23:15.197662+0000","last_unstale":"2024-06-29T14:23:15.202122+0000","last_undegraded":"2024-06-29T14:23:15.202122+0000","last_fullsized":"2024-06-29T14:23:15.202122+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:48.024521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":142,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.190774+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:23:15.190774+0000","last_peered":"2024-06-29T14:23:15.190774+0000","last_clean":"2024-06-29T14:23:15.190774+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:23:15.190774+0000","last_undegraded":"2024-06-29T14:23:15.190774+0000","last_fullsized":"2024-06-29T14:23:15.190774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1716,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":197889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509462,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":196333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":779444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":779444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":203005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:23:20.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:23:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.446+0000 7f22cdfe0640 1 -- 172.21.15.59:0/212811607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22b0005150 msgr2=0x7f22b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.446+0000 7f22cdfe0640 1 --2- 172.21.15.59:0/212811607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22b0005150 0x7f22b000b9c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f22c0005fd0 tx=0x7f22b0005830 comp rx=0 tx=0).stop 2024-06-29T14:23:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.446+0000 7f22c7fff640 1 -- 172.21.15.59:0/212811607 reap_dead start 2024-06-29T14:23:20.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.446+0000 7f22cdfe0640 1 -- 172.21.15.59:0/212811607 shutdown_connections 2024-06-29T14:23:20.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.446+0000 7f22cdfe0640 1 -- 172.21.15.59:0/212811607 >> 172.21.15.59:0/212811607 conn(0x7f22c80b6fb0 msgr2=0x7f22c80b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.446+0000 7f22cdfe0640 1 -- 172.21.15.59:0/212811607 shutdown_connections 2024-06-29T14:23:20.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.446+0000 7f22cdfe0640 1 -- 172.21.15.59:0/212811607 wait complete. 2024-06-29T14:23:20.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.448+0000 7f22cdfe0640 1 Processor -- start 2024-06-29T14:23:20.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.448+0000 7f22cdfe0640 1 -- start start 2024-06-29T14:23:20.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22cdfe0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b0005150 0x7f22c815a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22cdfe0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c81576b0 0x7f22c81f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22cdfe0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22c81f2650 0x7f22c813dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22cdfe0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f22c813ef40 con 0x7f22c81576b0 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22cdfe0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22c813edc0 con 0x7f22b0005150 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22cdfe0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f22c813f0c0 con 0x7f22c81f2650 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22c6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c81576b0 0x7f22c81f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22c6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c81576b0 0x7f22c81f2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48178/0 (socket says 172.21.15.59:48178) 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22c6ffd640 1 -- 172.21.15.59:0/2926734916 learned_addr learned my addr 172.21.15.59:0/2926734916 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22c77fe640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b0005150 0x7f22c815a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.449+0000 7f22c7fff640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22c81f2650 0x7f22c813dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22c6ffd640 1 -- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22c81f2650 msgr2=0x7f22c813dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22c6ffd640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22c81f2650 0x7f22c813dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22c6ffd640 1 -- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b0005150 msgr2=0x7f22c815a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22c6ffd640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b0005150 0x7f22c815a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22c6ffd640 1 -- 172.21.15.59:0/2926734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c815ad40 con 0x7f22c81576b0 2024-06-29T14:23:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22c6ffd640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c81576b0 0x7f22c81f2110 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00e630 tx=0x7f22bc00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22affff640 1 -- 172.21.15.59:0/2926734916 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc015070 con 0x7f22c81576b0 2024-06-29T14:23:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22affff640 1 -- 172.21.15.59:0/2926734916 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22bc00ca50 con 0x7f22c81576b0 2024-06-29T14:23:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c8157d90 con 0x7f22c81576b0 2024-06-29T14:23:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22affff640 1 -- 172.21.15.59:0/2926734916 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc00cea0 con 0x7f22c81576b0 2024-06-29T14:23:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.450+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c8158350 con 0x7f22c81576b0 2024-06-29T14:23:20.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.451+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c80b56d0 con 0x7f22c81576b0 2024-06-29T14:23:20.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.452+0000 7f22affff640 1 -- 172.21.15.59:0/2926734916 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f22bc018070 con 0x7f22c81576b0 2024-06-29T14:23:20.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.452+0000 7f22affff640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22b40762d0 0x7f22b40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.453+0000 7f22affff640 1 -- 172.21.15.59:0/2926734916 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f22bc097d50 con 0x7f22c81576b0 2024-06-29T14:23:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.453+0000 7f22c77fe640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22b40762d0 0x7f22b40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.453+0000 7f22c77fe640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22b40762d0 0x7f22b40787e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009b20 tx=0x7f22b8005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.457+0000 7f22affff640 1 -- 172.21.15.59:0/2926734916 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f22bc09b030 con 0x7f22c81576b0 2024-06-29T14:23:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.644+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f22c81f2a60 con 0x7f22b40762d0 2024-06-29T14:23:20.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.648+0000 7f22affff640 1 -- 172.21.15.59:0/2926734916 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249106 (secure 0 0 0) 0x7f22b8004030 con 0x7f22b40762d0 2024-06-29T14:23:20.650 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:23:20.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22b40762d0 msgr2=0x7f22b40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22b40762d0 0x7f22b40787e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009b20 tx=0x7f22b8005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:20.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c81576b0 msgr2=0x7f22c81f2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c81576b0 0x7f22c81f2110 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00e630 tx=0x7f22bc00eaf0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 shutdown_connections 2024-06-29T14:23:20.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f22b40762d0 0x7f22b40787e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22c81f2650 0x7f22c813dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c81576b0 0x7f22c81f2110 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 --2- 172.21.15.59:0/2926734916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22b0005150 0x7f22c815a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 >> 172.21.15.59:0/2926734916 conn(0x7f22c80b6fb0 msgr2=0x7f22c8071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 shutdown_connections 2024-06-29T14:23:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.651+0000 7f22cdfe0640 1 -- 172.21.15.59:0/2926734916 wait complete. 2024-06-29T14:23:20.660 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-06-29T14:23:19.752591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592084,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":116,"num_write_kb":1639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220812,"kb_used_data":4652,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753908,"statfs":{"total":773094113280,"available":772868001792,"internally_reserved":0,"allocated":4763648,"data_stored":2703342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1716,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.579212"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.211606+0000","last_change":"2024-06-29T14:23:16.201160+0000","last_active":"2024-06-29T14:23:16.211606+0000","last_peered":"2024-06-29T14:23:16.211606+0000","last_clean":"2024-06-29T14:23:16.211606+0000","last_became_active":"2024-06-29T14:23:16.200155+0000","last_became_peered":"2024-06-29T14:23:16.200155+0000","last_unstale":"2024-06-29T14:23:16.211606+0000","last_undegraded":"2024-06-29T14:23:16.211606+0000","last_fullsized":"2024-06-29T14:23:16.211606+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:23.684550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.199308+0000","last_change":"2024-06-29T14:23:16.189930+0000","last_active":"2024-06-29T14:23:16.199308+0000","last_peered":"2024-06-29T14:23:16.199308+0000","last_clean":"2024-06-29T14:23:16.199308+0000","last_became_active":"2024-06-29T14:23:16.189571+0000","last_became_peered":"2024-06-29T14:23:16.189571+0000","last_unstale":"2024-06-29T14:23:16.199308+0000","last_undegraded":"2024-06-29T14:23:16.199308+0000","last_fullsized":"2024-06-29T14:23:16.199308+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:12.662244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207289+0000","last_change":"2024-06-29T14:23:16.203182+0000","last_active":"2024-06-29T14:23:16.207289+0000","last_peered":"2024-06-29T14:23:16.207289+0000","last_clean":"2024-06-29T14:23:16.207289+0000","last_became_active":"2024-06-29T14:23:16.202851+0000","last_became_peered":"2024-06-29T14:23:16.202851+0000","last_unstale":"2024-06-29T14:23:16.207289+0000","last_undegraded":"2024-06-29T14:23:16.207289+0000","last_fullsized":"2024-06-29T14:23:16.207289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:13.507757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972956+0000","last_change":"2024-06-29T14:23:16.202834+0000","last_active":"2024-06-29T14:23:17.972956+0000","last_peered":"2024-06-29T14:23:17.972956+0000","last_clean":"2024-06-29T14:23:17.972956+0000","last_became_active":"2024-06-29T14:23:16.202357+0000","last_became_peered":"2024-06-29T14:23:16.202357+0000","last_unstale":"2024-06-29T14:23:17.972956+0000","last_undegraded":"2024-06-29T14:23:17.972956+0000","last_fullsized":"2024-06-29T14:23:17.972956+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:35.097600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972601+0000","last_change":"2024-06-29T14:23:16.194759+0000","last_active":"2024-06-29T14:23:17.972601+0000","last_peered":"2024-06-29T14:23:17.972601+0000","last_clean":"2024-06-29T14:23:17.972601+0000","last_became_active":"2024-06-29T14:23:16.194351+0000","last_became_peered":"2024-06-29T14:23:16.194351+0000","last_unstale":"2024-06-29T14:23:17.972601+0000","last_undegraded":"2024-06-29T14:23:17.972601+0000","last_fullsized":"2024-06-29T14:23:17.972601+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:30.838175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207114+0000","last_change":"2024-06-29T14:23:16.195834+0000","last_active":"2024-06-29T14:23:16.207114+0000","last_peered":"2024-06-29T14:23:16.207114+0000","last_clean":"2024-06-29T14:23:16.207114+0000","last_became_active":"2024-06-29T14:23:16.195476+0000","last_became_peered":"2024-06-29T14:23:16.195476+0000","last_unstale":"2024-06-29T14:23:16.207114+0000","last_undegraded":"2024-06-29T14:23:16.207114+0000","last_fullsized":"2024-06-29T14:23:16.207114+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:30.001157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192292+0000","last_change":"2024-06-29T14:23:15.192292+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192292+0000","last_undegraded":"2024-06-29T14:23:15.192292+0000","last_fullsized":"2024-06-29T14:23:15.192292+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.206121+0000","last_change":"2024-06-29T14:23:16.199825+0000","last_active":"2024-06-29T14:23:16.206121+0000","last_peered":"2024-06-29T14:23:16.206121+0000","last_clean":"2024-06-29T14:23:16.206121+0000","last_became_active":"2024-06-29T14:23:16.199499+0000","last_became_peered":"2024-06-29T14:23:16.199499+0000","last_unstale":"2024-06-29T14:23:16.206121+0000","last_undegraded":"2024-06-29T14:23:16.206121+0000","last_fullsized":"2024-06-29T14:23:16.206121+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:10.882976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214762+0000","last_change":"2024-06-29T14:23:16.209692+0000","last_active":"2024-06-29T14:23:16.214762+0000","last_peered":"2024-06-29T14:23:16.214762+0000","last_clean":"2024-06-29T14:23:16.214762+0000","last_became_active":"2024-06-29T14:23:16.202968+0000","last_became_peered":"2024-06-29T14:23:16.202968+0000","last_unstale":"2024-06-29T14:23:16.214762+0000","last_undegraded":"2024-06-29T14:23:16.214762+0000","last_fullsized":"2024-06-29T14:23:16.214762+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:10.402585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.194916+0000","last_change":"2024-06-29T14:23:15.194916+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.194916+0000","last_undegraded":"2024-06-29T14:23:15.194916+0000","last_fullsized":"2024-06-29T14:23:15.194916+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193674+0000","last_change":"2024-06-29T14:23:15.193674+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193674+0000","last_undegraded":"2024-06-29T14:23:15.193674+0000","last_fullsized":"2024-06-29T14:23:15.193674+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214775+0000","last_change":"2024-06-29T14:23:16.209122+0000","last_active":"2024-06-29T14:23:16.214775+0000","last_peered":"2024-06-29T14:23:16.214775+0000","last_clean":"2024-06-29T14:23:16.214775+0000","last_became_active":"2024-06-29T14:23:16.208783+0000","last_became_peered":"2024-06-29T14:23:16.208783+0000","last_unstale":"2024-06-29T14:23:16.214775+0000","last_undegraded":"2024-06-29T14:23:16.214775+0000","last_fullsized":"2024-06-29T14:23:16.214775+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:30.001157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.196595+0000","last_change":"2024-06-29T14:23:15.196595+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.196595+0000","last_undegraded":"2024-06-29T14:23:15.196595+0000","last_fullsized":"2024-06-29T14:23:15.196595+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.194142+0000","last_change":"2024-06-29T14:23:15.194142+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.194142+0000","last_undegraded":"2024-06-29T14:23:15.194142+0000","last_fullsized":"2024-06-29T14:23:15.194142+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191760+0000","last_change":"2024-06-29T14:23:15.191760+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191760+0000","last_undegraded":"2024-06-29T14:23:15.191760+0000","last_fullsized":"2024-06-29T14:23:15.191760+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189097+0000","last_change":"2024-06-29T14:23:15.189097+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189097+0000","last_undegraded":"2024-06-29T14:23:15.189097+0000","last_fullsized":"2024-06-29T14:23:15.189097+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188330+0000","last_change":"2024-06-29T14:23:15.188330+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188330+0000","last_undegraded":"2024-06-29T14:23:15.188330+0000","last_fullsized":"2024-06-29T14:23:15.188330+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.184396+0000","last_change":"2024-06-29T14:23:15.184396+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.184396+0000","last_undegraded":"2024-06-29T14:23:15.184396+0000","last_fullsized":"2024-06-29T14:23:15.184396+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972388+0000","last_change":"2024-06-29T14:23:16.190111+0000","last_active":"2024-06-29T14:23:17.972388+0000","last_peered":"2024-06-29T14:23:17.972388+0000","last_clean":"2024-06-29T14:23:17.972388+0000","last_became_active":"2024-06-29T14:23:16.189786+0000","last_became_peered":"2024-06-29T14:23:16.189786+0000","last_unstale":"2024-06-29T14:23:17.972388+0000","last_undegraded":"2024-06-29T14:23:17.972388+0000","last_fullsized":"2024-06-29T14:23:17.972388+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:12.500255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.184382+0000","last_change":"2024-06-29T14:23:15.184382+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.184382+0000","last_undegraded":"2024-06-29T14:23:15.184382+0000","last_fullsized":"2024-06-29T14:23:15.184382+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.190853+0000","last_change":"2024-06-29T14:23:15.190853+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.190853+0000","last_undegraded":"2024-06-29T14:23:15.190853+0000","last_fullsized":"2024-06-29T14:23:15.190853+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193127+0000","last_change":"2024-06-29T14:23:15.193127+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193127+0000","last_undegraded":"2024-06-29T14:23:15.193127+0000","last_fullsized":"2024-06-29T14:23:15.193127+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188344+0000","last_change":"2024-06-29T14:23:15.188344+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188344+0000","last_undegraded":"2024-06-29T14:23:15.188344+0000","last_fullsized":"2024-06-29T14:23:15.188344+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972485+0000","last_change":"2024-06-29T14:23:16.210617+0000","last_active":"2024-06-29T14:23:17.972485+0000","last_peered":"2024-06-29T14:23:17.972485+0000","last_clean":"2024-06-29T14:23:17.972485+0000","last_became_active":"2024-06-29T14:23:16.210242+0000","last_became_peered":"2024-06-29T14:23:16.210242+0000","last_unstale":"2024-06-29T14:23:17.972485+0000","last_undegraded":"2024-06-29T14:23:17.972485+0000","last_fullsized":"2024-06-29T14:23:17.972485+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:21.067086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185688+0000","last_change":"2024-06-29T14:23:15.185688+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185688+0000","last_undegraded":"2024-06-29T14:23:15.185688+0000","last_fullsized":"2024-06-29T14:23:15.185688+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207195+0000","last_change":"2024-06-29T14:23:16.205741+0000","last_active":"2024-06-29T14:23:16.207195+0000","last_peered":"2024-06-29T14:23:16.207195+0000","last_clean":"2024-06-29T14:23:16.207195+0000","last_became_active":"2024-06-29T14:23:16.205351+0000","last_became_peered":"2024-06-29T14:23:16.205351+0000","last_unstale":"2024-06-29T14:23:16.207195+0000","last_undegraded":"2024-06-29T14:23:16.207195+0000","last_fullsized":"2024-06-29T14:23:16.207195+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:10.402585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185734+0000","last_change":"2024-06-29T14:23:15.185734+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185734+0000","last_undegraded":"2024-06-29T14:23:15.185734+0000","last_fullsized":"2024-06-29T14:23:15.185734+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192802+0000","last_change":"2024-06-29T14:23:15.192802+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192802+0000","last_undegraded":"2024-06-29T14:23:15.192802+0000","last_fullsized":"2024-06-29T14:23:15.192802+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.186290+0000","last_change":"2024-06-29T14:23:15.186290+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.186290+0000","last_undegraded":"2024-06-29T14:23:15.186290+0000","last_fullsized":"2024-06-29T14:23:15.186290+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185611+0000","last_change":"2024-06-29T14:23:15.185611+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185611+0000","last_undegraded":"2024-06-29T14:23:15.185611+0000","last_fullsized":"2024-06-29T14:23:15.185611+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187368+0000","last_change":"2024-06-29T14:23:15.187368+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187368+0000","last_undegraded":"2024-06-29T14:23:15.187368+0000","last_fullsized":"2024-06-29T14:23:15.187368+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207193+0000","last_change":"2024-06-29T14:23:16.204276+0000","last_active":"2024-06-29T14:23:16.207193+0000","last_peered":"2024-06-29T14:23:16.207193+0000","last_clean":"2024-06-29T14:23:16.207193+0000","last_became_active":"2024-06-29T14:23:16.203937+0000","last_became_peered":"2024-06-29T14:23:16.203937+0000","last_unstale":"2024-06-29T14:23:16.207193+0000","last_undegraded":"2024-06-29T14:23:16.207193+0000","last_fullsized":"2024-06-29T14:23:16.207193+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:40.723736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188473+0000","last_change":"2024-06-29T14:23:15.188473+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188473+0000","last_undegraded":"2024-06-29T14:23:15.188473+0000","last_fullsized":"2024-06-29T14:23:15.188473+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.199746+0000","last_change":"2024-06-29T14:23:15.197029+0000","last_active":"2024-06-29T14:23:15.199746+0000","last_peered":"2024-06-29T14:23:15.199746+0000","last_clean":"2024-06-29T14:23:15.199746+0000","last_became_active":"2024-06-29T14:23:15.196636+0000","last_became_peered":"2024-06-29T14:23:15.196636+0000","last_unstale":"2024-06-29T14:23:15.199746+0000","last_undegraded":"2024-06-29T14:23:15.199746+0000","last_fullsized":"2024-06-29T14:23:15.199746+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:29.832584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.208350+0000","last_change":"2024-06-29T14:23:16.206697+0000","last_active":"2024-06-29T14:23:16.208350+0000","last_peered":"2024-06-29T14:23:16.208350+0000","last_clean":"2024-06-29T14:23:16.208350+0000","last_became_active":"2024-06-29T14:23:16.206325+0000","last_became_peered":"2024-06-29T14:23:16.206325+0000","last_unstale":"2024-06-29T14:23:16.208350+0000","last_undegraded":"2024-06-29T14:23:16.208350+0000","last_fullsized":"2024-06-29T14:23:16.208350+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"40'1","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217808+0000","last_change":"2024-06-29T14:23:15.204215+0000","last_active":"2024-06-29T14:23:16.217808+0000","last_peered":"2024-06-29T14:23:16.217808+0000","last_clean":"2024-06-29T14:23:16.217808+0000","last_became_active":"2024-06-29T14:23:15.203861+0000","last_became_peered":"2024-06-29T14:23:15.203861+0000","last_unstale":"2024-06-29T14:23:16.217808+0000","last_undegraded":"2024-06-29T14:23:16.217808+0000","last_fullsized":"2024-06-29T14:23:16.217808+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:15.271848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189311+0000","last_change":"2024-06-29T14:23:15.189311+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189311+0000","last_undegraded":"2024-06-29T14:23:15.189311+0000","last_fullsized":"2024-06-29T14:23:15.189311+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"40'2","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.219346+0000","last_change":"2024-06-29T14:23:15.205028+0000","last_active":"2024-06-29T14:23:16.219346+0000","last_peered":"2024-06-29T14:23:16.219346+0000","last_clean":"2024-06-29T14:23:16.219346+0000","last_became_active":"2024-06-29T14:23:15.204734+0000","last_became_peered":"2024-06-29T14:23:15.204734+0000","last_unstale":"2024-06-29T14:23:16.219346+0000","last_undegraded":"2024-06-29T14:23:16.219346+0000","last_fullsized":"2024-06-29T14:23:16.219346+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:12.502166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.198933+0000","last_change":"2024-06-29T14:23:16.190648+0000","last_active":"2024-06-29T14:23:16.198933+0000","last_peered":"2024-06-29T14:23:16.198933+0000","last_clean":"2024-06-29T14:23:16.198933+0000","last_became_active":"2024-06-29T14:23:16.190331+0000","last_became_peered":"2024-06-29T14:23:16.190331+0000","last_unstale":"2024-06-29T14:23:16.198933+0000","last_undegraded":"2024-06-29T14:23:16.198933+0000","last_fullsized":"2024-06-29T14:23:16.198933+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:52.309099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201353+0000","last_change":"2024-06-29T14:23:15.195155+0000","last_active":"2024-06-29T14:23:15.201353+0000","last_peered":"2024-06-29T14:23:15.201353+0000","last_clean":"2024-06-29T14:23:15.201353+0000","last_became_active":"2024-06-29T14:23:15.194809+0000","last_became_peered":"2024-06-29T14:23:15.194809+0000","last_unstale":"2024-06-29T14:23:15.201353+0000","last_undegraded":"2024-06-29T14:23:15.201353+0000","last_fullsized":"2024-06-29T14:23:15.201353+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:29.832584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214639+0000","last_change":"2024-06-29T14:23:16.209386+0000","last_active":"2024-06-29T14:23:16.214639+0000","last_peered":"2024-06-29T14:23:16.214639+0000","last_clean":"2024-06-29T14:23:16.214639+0000","last_became_active":"2024-06-29T14:23:16.209011+0000","last_became_peered":"2024-06-29T14:23:16.209011+0000","last_unstale":"2024-06-29T14:23:16.214639+0000","last_undegraded":"2024-06-29T14:23:16.214639+0000","last_fullsized":"2024-06-29T14:23:16.214639+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:40.723736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203843+0000","last_change":"2024-06-29T14:23:15.201776+0000","last_active":"2024-06-29T14:23:15.203843+0000","last_peered":"2024-06-29T14:23:15.203843+0000","last_clean":"2024-06-29T14:23:15.203843+0000","last_became_active":"2024-06-29T14:23:15.201376+0000","last_became_peered":"2024-06-29T14:23:15.201376+0000","last_unstale":"2024-06-29T14:23:15.203843+0000","last_undegraded":"2024-06-29T14:23:15.203843+0000","last_fullsized":"2024-06-29T14:23:15.203843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191269+0000","last_change":"2024-06-29T14:23:15.191269+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191269+0000","last_undegraded":"2024-06-29T14:23:15.191269+0000","last_fullsized":"2024-06-29T14:23:15.191269+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.204276+0000","last_change":"2024-06-29T14:23:15.203169+0000","last_active":"2024-06-29T14:23:15.204276+0000","last_peered":"2024-06-29T14:23:15.204276+0000","last_clean":"2024-06-29T14:23:15.204276+0000","last_became_active":"2024-06-29T14:23:15.202809+0000","last_became_peered":"2024-06-29T14:23:15.202809+0000","last_unstale":"2024-06-29T14:23:15.204276+0000","last_undegraded":"2024-06-29T14:23:15.204276+0000","last_fullsized":"2024-06-29T14:23:15.204276+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:29.639570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191458+0000","last_change":"2024-06-29T14:23:15.191458+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191458+0000","last_undegraded":"2024-06-29T14:23:15.191458+0000","last_fullsized":"2024-06-29T14:23:15.191458+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185188+0000","last_change":"2024-06-29T14:23:15.200889+0000","last_active":"2024-06-29T14:23:16.185188+0000","last_peered":"2024-06-29T14:23:16.185188+0000","last_clean":"2024-06-29T14:23:16.185188+0000","last_became_active":"2024-06-29T14:23:15.200521+0000","last_became_peered":"2024-06-29T14:23:15.200521+0000","last_unstale":"2024-06-29T14:23:16.185188+0000","last_undegraded":"2024-06-29T14:23:16.185188+0000","last_fullsized":"2024-06-29T14:23:16.185188+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217265+0000","last_change":"2024-06-29T14:23:16.207437+0000","last_active":"2024-06-29T14:23:16.217265+0000","last_peered":"2024-06-29T14:23:16.217265+0000","last_clean":"2024-06-29T14:23:16.217265+0000","last_became_active":"2024-06-29T14:23:16.207056+0000","last_became_peered":"2024-06-29T14:23:16.207056+0000","last_unstale":"2024-06-29T14:23:16.217265+0000","last_undegraded":"2024-06-29T14:23:16.217265+0000","last_fullsized":"2024-06-29T14:23:16.217265+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:52.297506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203620+0000","last_change":"2024-06-29T14:23:15.199990+0000","last_active":"2024-06-29T14:23:15.203620+0000","last_peered":"2024-06-29T14:23:15.203620+0000","last_clean":"2024-06-29T14:23:15.203620+0000","last_became_active":"2024-06-29T14:23:15.199697+0000","last_became_peered":"2024-06-29T14:23:15.199697+0000","last_unstale":"2024-06-29T14:23:15.203620+0000","last_undegraded":"2024-06-29T14:23:15.203620+0000","last_fullsized":"2024-06-29T14:23:15.203620+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:54.406021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.973018+0000","last_change":"2024-06-29T14:23:16.197089+0000","last_active":"2024-06-29T14:23:17.973018+0000","last_peered":"2024-06-29T14:23:17.973018+0000","last_clean":"2024-06-29T14:23:17.973018+0000","last_became_active":"2024-06-29T14:23:16.196730+0000","last_became_peered":"2024-06-29T14:23:16.196730+0000","last_unstale":"2024-06-29T14:23:17.973018+0000","last_undegraded":"2024-06-29T14:23:17.973018+0000","last_fullsized":"2024-06-29T14:23:17.973018+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:55.411612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.182345+0000","last_change":"2024-06-29T14:23:15.189674+0000","last_active":"2024-06-29T14:23:16.182345+0000","last_peered":"2024-06-29T14:23:16.182345+0000","last_clean":"2024-06-29T14:23:16.182345+0000","last_became_active":"2024-06-29T14:23:15.189326+0000","last_became_peered":"2024-06-29T14:23:15.189326+0000","last_unstale":"2024-06-29T14:23:16.182345+0000","last_undegraded":"2024-06-29T14:23:16.182345+0000","last_fullsized":"2024-06-29T14:23:16.182345+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:09.396994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.190068+0000","last_change":"2024-06-29T14:23:15.190068+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.190068+0000","last_undegraded":"2024-06-29T14:23:15.190068+0000","last_fullsized":"2024-06-29T14:23:15.190068+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.207378+0000","last_change":"2024-06-29T14:23:15.205426+0000","last_active":"2024-06-29T14:23:15.207378+0000","last_peered":"2024-06-29T14:23:15.207378+0000","last_clean":"2024-06-29T14:23:15.207378+0000","last_became_active":"2024-06-29T14:23:15.204970+0000","last_became_peered":"2024-06-29T14:23:15.204970+0000","last_unstale":"2024-06-29T14:23:15.207378+0000","last_undegraded":"2024-06-29T14:23:15.207378+0000","last_fullsized":"2024-06-29T14:23:15.207378+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:12.502166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.213758+0000","last_change":"2024-06-29T14:23:16.208578+0000","last_active":"2024-06-29T14:23:16.213758+0000","last_peered":"2024-06-29T14:23:16.213758+0000","last_clean":"2024-06-29T14:23:16.213758+0000","last_became_active":"2024-06-29T14:23:16.208189+0000","last_became_peered":"2024-06-29T14:23:16.208189+0000","last_unstale":"2024-06-29T14:23:16.213758+0000","last_undegraded":"2024-06-29T14:23:16.213758+0000","last_fullsized":"2024-06-29T14:23:16.213758+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:08.341184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203812+0000","last_change":"2024-06-29T14:23:15.202041+0000","last_active":"2024-06-29T14:23:15.203812+0000","last_peered":"2024-06-29T14:23:15.203812+0000","last_clean":"2024-06-29T14:23:15.203812+0000","last_became_active":"2024-06-29T14:23:15.201722+0000","last_became_peered":"2024-06-29T14:23:15.201722+0000","last_unstale":"2024-06-29T14:23:15.203812+0000","last_undegraded":"2024-06-29T14:23:15.203812+0000","last_fullsized":"2024-06-29T14:23:15.203812+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:18.072309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191167+0000","last_change":"2024-06-29T14:23:15.191167+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191167+0000","last_undegraded":"2024-06-29T14:23:15.191167+0000","last_fullsized":"2024-06-29T14:23:15.191167+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"40'6","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.220943+0000","last_change":"2024-06-29T14:23:15.199353+0000","last_active":"2024-06-29T14:23:16.220943+0000","last_peered":"2024-06-29T14:23:16.220943+0000","last_clean":"2024-06-29T14:23:16.220943+0000","last_became_active":"2024-06-29T14:23:15.198339+0000","last_became_peered":"2024-06-29T14:23:15.198339+0000","last_unstale":"2024-06-29T14:23:16.220943+0000","last_undegraded":"2024-06-29T14:23:16.220943+0000","last_fullsized":"2024-06-29T14:23:16.220943+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:39.718146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214670+0000","last_change":"2024-06-29T14:23:16.209778+0000","last_active":"2024-06-29T14:23:16.214670+0000","last_peered":"2024-06-29T14:23:16.214670+0000","last_clean":"2024-06-29T14:23:16.214670+0000","last_became_active":"2024-06-29T14:23:16.209443+0000","last_became_peered":"2024-06-29T14:23:16.209443+0000","last_unstale":"2024-06-29T14:23:16.214670+0000","last_undegraded":"2024-06-29T14:23:16.214670+0000","last_fullsized":"2024-06-29T14:23:16.214670+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:55.671459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"40'2","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.219006+0000","last_change":"2024-06-29T14:23:15.200701+0000","last_active":"2024-06-29T14:23:16.219006+0000","last_peered":"2024-06-29T14:23:16.219006+0000","last_clean":"2024-06-29T14:23:16.219006+0000","last_became_active":"2024-06-29T14:23:15.200338+0000","last_became_peered":"2024-06-29T14:23:15.200338+0000","last_unstale":"2024-06-29T14:23:16.219006+0000","last_undegraded":"2024-06-29T14:23:16.219006+0000","last_fullsized":"2024-06-29T14:23:16.219006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:54.665868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.201400+0000","last_change":"2024-06-29T14:23:16.196448+0000","last_active":"2024-06-29T14:23:16.201400+0000","last_peered":"2024-06-29T14:23:16.201400+0000","last_clean":"2024-06-29T14:23:16.201400+0000","last_became_active":"2024-06-29T14:23:16.195006+0000","last_became_peered":"2024-06-29T14:23:16.195006+0000","last_unstale":"2024-06-29T14:23:16.201400+0000","last_undegraded":"2024-06-29T14:23:16.201400+0000","last_fullsized":"2024-06-29T14:23:16.201400+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:05.074083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185776+0000","last_change":"2024-06-29T14:23:15.199955+0000","last_active":"2024-06-29T14:23:16.185776+0000","last_peered":"2024-06-29T14:23:16.185776+0000","last_clean":"2024-06-29T14:23:16.185776+0000","last_became_active":"2024-06-29T14:23:15.199619+0000","last_became_peered":"2024-06-29T14:23:15.199619+0000","last_unstale":"2024-06-29T14:23:16.185776+0000","last_undegraded":"2024-06-29T14:23:16.185776+0000","last_fullsized":"2024-06-29T14:23:16.185776+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:04.589609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187426+0000","last_change":"2024-06-29T14:23:15.187426+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187426+0000","last_undegraded":"2024-06-29T14:23:15.187426+0000","last_fullsized":"2024-06-29T14:23:15.187426+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.198643+0000","last_change":"2024-06-29T14:23:15.196439+0000","last_active":"2024-06-29T14:23:15.198643+0000","last_peered":"2024-06-29T14:23:15.198643+0000","last_clean":"2024-06-29T14:23:15.198643+0000","last_became_active":"2024-06-29T14:23:15.196116+0000","last_became_peered":"2024-06-29T14:23:15.196116+0000","last_unstale":"2024-06-29T14:23:15.198643+0000","last_undegraded":"2024-06-29T14:23:15.198643+0000","last_fullsized":"2024-06-29T14:23:15.198643+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:20.061496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.213976+0000","last_change":"2024-06-29T14:23:16.200847+0000","last_active":"2024-06-29T14:23:16.213976+0000","last_peered":"2024-06-29T14:23:16.213976+0000","last_clean":"2024-06-29T14:23:16.213976+0000","last_became_active":"2024-06-29T14:23:16.200264+0000","last_became_peered":"2024-06-29T14:23:16.200264+0000","last_unstale":"2024-06-29T14:23:16.213976+0000","last_undegraded":"2024-06-29T14:23:16.213976+0000","last_fullsized":"2024-06-29T14:23:16.213976+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:46.554739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202335+0000","last_change":"2024-06-29T14:23:15.200739+0000","last_active":"2024-06-29T14:23:15.202335+0000","last_peered":"2024-06-29T14:23:15.202335+0000","last_clean":"2024-06-29T14:23:15.202335+0000","last_became_active":"2024-06-29T14:23:15.200401+0000","last_became_peered":"2024-06-29T14:23:15.200401+0000","last_unstale":"2024-06-29T14:23:15.202335+0000","last_undegraded":"2024-06-29T14:23:15.202335+0000","last_fullsized":"2024-06-29T14:23:15.202335+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:49.969939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.194644+0000","last_change":"2024-06-29T14:23:16.188970+0000","last_active":"2024-06-29T14:23:16.194644+0000","last_peered":"2024-06-29T14:23:16.194644+0000","last_clean":"2024-06-29T14:23:16.194644+0000","last_became_active":"2024-06-29T14:23:16.188588+0000","last_became_peered":"2024-06-29T14:23:16.188588+0000","last_unstale":"2024-06-29T14:23:16.194644+0000","last_undegraded":"2024-06-29T14:23:16.194644+0000","last_fullsized":"2024-06-29T14:23:16.194644+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201969+0000","last_change":"2024-06-29T14:23:15.199255+0000","last_active":"2024-06-29T14:23:15.201969+0000","last_peered":"2024-06-29T14:23:15.201969+0000","last_clean":"2024-06-29T14:23:15.201969+0000","last_became_active":"2024-06-29T14:23:15.198806+0000","last_became_peered":"2024-06-29T14:23:15.198806+0000","last_unstale":"2024-06-29T14:23:15.201969+0000","last_undegraded":"2024-06-29T14:23:15.201969+0000","last_fullsized":"2024-06-29T14:23:15.201969+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:41.669608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193179+0000","last_change":"2024-06-29T14:23:15.193179+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193179+0000","last_undegraded":"2024-06-29T14:23:15.193179+0000","last_fullsized":"2024-06-29T14:23:15.193179+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203612+0000","last_change":"2024-06-29T14:23:15.200790+0000","last_active":"2024-06-29T14:23:15.203612+0000","last_peered":"2024-06-29T14:23:15.203612+0000","last_clean":"2024-06-29T14:23:15.203612+0000","last_became_active":"2024-06-29T14:23:15.200452+0000","last_became_peered":"2024-06-29T14:23:15.200452+0000","last_unstale":"2024-06-29T14:23:15.203612+0000","last_undegraded":"2024-06-29T14:23:15.203612+0000","last_fullsized":"2024-06-29T14:23:15.203612+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:27.766774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192686+0000","last_change":"2024-06-29T14:23:15.192686+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192686+0000","last_undegraded":"2024-06-29T14:23:15.192686+0000","last_fullsized":"2024-06-29T14:23:15.192686+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972729+0000","last_change":"2024-06-29T14:23:15.195356+0000","last_active":"2024-06-29T14:23:17.972729+0000","last_peered":"2024-06-29T14:23:17.972729+0000","last_clean":"2024-06-29T14:23:17.972729+0000","last_became_active":"2024-06-29T14:23:15.195037+0000","last_became_peered":"2024-06-29T14:23:15.195037+0000","last_unstale":"2024-06-29T14:23:17.972729+0000","last_undegraded":"2024-06-29T14:23:17.972729+0000","last_fullsized":"2024-06-29T14:23:17.972729+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:32.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971916+0000","last_change":"2024-06-29T14:23:16.197431+0000","last_active":"2024-06-29T14:23:17.971916+0000","last_peered":"2024-06-29T14:23:17.971916+0000","last_clean":"2024-06-29T14:23:17.971916+0000","last_became_active":"2024-06-29T14:23:16.197091+0000","last_became_peered":"2024-06-29T14:23:16.197091+0000","last_unstale":"2024-06-29T14:23:17.971916+0000","last_undegraded":"2024-06-29T14:23:17.971916+0000","last_fullsized":"2024-06-29T14:23:17.971916+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:46.040585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185494+0000","last_change":"2024-06-29T14:23:15.203782+0000","last_active":"2024-06-29T14:23:16.185494+0000","last_peered":"2024-06-29T14:23:16.185494+0000","last_clean":"2024-06-29T14:23:16.185494+0000","last_became_active":"2024-06-29T14:23:15.203451+0000","last_became_peered":"2024-06-29T14:23:15.203451+0000","last_unstale":"2024-06-29T14:23:16.185494+0000","last_undegraded":"2024-06-29T14:23:16.185494+0000","last_fullsized":"2024-06-29T14:23:16.185494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:18.072309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187906+0000","last_change":"2024-06-29T14:23:15.187906+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187906+0000","last_undegraded":"2024-06-29T14:23:15.187906+0000","last_fullsized":"2024-06-29T14:23:15.187906+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972722+0000","last_change":"2024-06-29T14:23:15.201493+0000","last_active":"2024-06-29T14:23:17.972722+0000","last_peered":"2024-06-29T14:23:17.972722+0000","last_clean":"2024-06-29T14:23:17.972722+0000","last_became_active":"2024-06-29T14:23:15.201150+0000","last_became_peered":"2024-06-29T14:23:15.201150+0000","last_unstale":"2024-06-29T14:23:17.972722+0000","last_undegraded":"2024-06-29T14:23:17.972722+0000","last_fullsized":"2024-06-29T14:23:17.972722+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:00.323663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.206186+0000","last_change":"2024-06-29T14:23:16.202653+0000","last_active":"2024-06-29T14:23:16.206186+0000","last_peered":"2024-06-29T14:23:16.206186+0000","last_clean":"2024-06-29T14:23:16.206186+0000","last_became_active":"2024-06-29T14:23:16.202279+0000","last_became_peered":"2024-06-29T14:23:16.202279+0000","last_unstale":"2024-06-29T14:23:16.206186+0000","last_undegraded":"2024-06-29T14:23:16.206186+0000","last_fullsized":"2024-06-29T14:23:16.206186+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:05.595200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971710+0000","last_change":"2024-06-29T14:23:15.196760+0000","last_active":"2024-06-29T14:23:17.971710+0000","last_peered":"2024-06-29T14:23:17.971710+0000","last_clean":"2024-06-29T14:23:17.971710+0000","last_became_active":"2024-06-29T14:23:15.196278+0000","last_became_peered":"2024-06-29T14:23:15.196278+0000","last_unstale":"2024-06-29T14:23:17.971710+0000","last_undegraded":"2024-06-29T14:23:17.971710+0000","last_fullsized":"2024-06-29T14:23:17.971710+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:32:01.571661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193709+0000","last_change":"2024-06-29T14:23:15.193709+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193709+0000","last_undegraded":"2024-06-29T14:23:15.193709+0000","last_fullsized":"2024-06-29T14:23:15.193709+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202585+0000","last_change":"2024-06-29T14:23:15.199837+0000","last_active":"2024-06-29T14:23:15.202585+0000","last_peered":"2024-06-29T14:23:15.202585+0000","last_clean":"2024-06-29T14:23:15.202585+0000","last_became_active":"2024-06-29T14:23:15.199539+0000","last_became_peered":"2024-06-29T14:23:15.199539+0000","last_unstale":"2024-06-29T14:23:15.202585+0000","last_undegraded":"2024-06-29T14:23:15.202585+0000","last_fullsized":"2024-06-29T14:23:15.202585+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:12.157450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217291+0000","last_change":"2024-06-29T14:23:16.211956+0000","last_active":"2024-06-29T14:23:16.217291+0000","last_peered":"2024-06-29T14:23:16.217291+0000","last_clean":"2024-06-29T14:23:16.217291+0000","last_became_active":"2024-06-29T14:23:16.211649+0000","last_became_peered":"2024-06-29T14:23:16.211649+0000","last_unstale":"2024-06-29T14:23:16.217291+0000","last_undegraded":"2024-06-29T14:23:16.217291+0000","last_fullsized":"2024-06-29T14:23:16.217291+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:37.213779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972618+0000","last_change":"2024-06-29T14:23:15.201805+0000","last_active":"2024-06-29T14:23:17.972618+0000","last_peered":"2024-06-29T14:23:17.972618+0000","last_clean":"2024-06-29T14:23:17.972618+0000","last_became_active":"2024-06-29T14:23:15.200924+0000","last_became_peered":"2024-06-29T14:23:15.200924+0000","last_unstale":"2024-06-29T14:23:17.972618+0000","last_undegraded":"2024-06-29T14:23:17.972618+0000","last_fullsized":"2024-06-29T14:23:17.972618+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:18.651137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207150+0000","last_change":"2024-06-29T14:23:16.205768+0000","last_active":"2024-06-29T14:23:16.207150+0000","last_peered":"2024-06-29T14:23:16.207150+0000","last_clean":"2024-06-29T14:23:16.207150+0000","last_became_active":"2024-06-29T14:23:16.205382+0000","last_became_peered":"2024-06-29T14:23:16.205382+0000","last_unstale":"2024-06-29T14:23:16.207150+0000","last_undegraded":"2024-06-29T14:23:16.207150+0000","last_fullsized":"2024-06-29T14:23:16.207150+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:55.671459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.205384+0000","last_change":"2024-06-29T14:23:15.203975+0000","last_active":"2024-06-29T14:23:15.205384+0000","last_peered":"2024-06-29T14:23:15.205384+0000","last_clean":"2024-06-29T14:23:15.205384+0000","last_became_active":"2024-06-29T14:23:15.203652+0000","last_became_peered":"2024-06-29T14:23:15.203652+0000","last_unstale":"2024-06-29T14:23:15.205384+0000","last_undegraded":"2024-06-29T14:23:15.205384+0000","last_fullsized":"2024-06-29T14:23:15.205384+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:04.589609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189204+0000","last_change":"2024-06-29T14:23:15.189204+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189204+0000","last_undegraded":"2024-06-29T14:23:15.189204+0000","last_fullsized":"2024-06-29T14:23:15.189204+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"41'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:18.201843+0000","last_change":"2024-06-29T14:23:15.199068+0000","last_active":"2024-06-29T14:23:18.201843+0000","last_peered":"2024-06-29T14:23:18.201843+0000","last_clean":"2024-06-29T14:23:18.201843+0000","last_became_active":"2024-06-29T14:23:15.198714+0000","last_became_peered":"2024-06-29T14:23:15.198714+0000","last_unstale":"2024-06-29T14:23:18.201843+0000","last_undegraded":"2024-06-29T14:23:18.201843+0000","last_fullsized":"2024-06-29T14:23:18.201843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:11.722673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.199343+0000","last_change":"2024-06-29T14:23:16.194459+0000","last_active":"2024-06-29T14:23:16.199343+0000","last_peered":"2024-06-29T14:23:16.199343+0000","last_clean":"2024-06-29T14:23:16.199343+0000","last_became_active":"2024-06-29T14:23:16.194084+0000","last_became_peered":"2024-06-29T14:23:16.194084+0000","last_unstale":"2024-06-29T14:23:16.199343+0000","last_undegraded":"2024-06-29T14:23:16.199343+0000","last_fullsized":"2024-06-29T14:23:16.199343+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:46.325166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.192313+0000","last_change":"2024-06-29T14:23:15.199199+0000","last_active":"2024-06-29T14:23:16.192313+0000","last_peered":"2024-06-29T14:23:16.192313+0000","last_clean":"2024-06-29T14:23:16.192313+0000","last_became_active":"2024-06-29T14:23:15.198802+0000","last_became_peered":"2024-06-29T14:23:15.198802+0000","last_unstale":"2024-06-29T14:23:16.192313+0000","last_undegraded":"2024-06-29T14:23:16.192313+0000","last_fullsized":"2024-06-29T14:23:16.192313+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:41.669608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207171+0000","last_change":"2024-06-29T14:23:16.200965+0000","last_active":"2024-06-29T14:23:16.207171+0000","last_peered":"2024-06-29T14:23:16.207171+0000","last_clean":"2024-06-29T14:23:16.207171+0000","last_became_active":"2024-06-29T14:23:16.200584+0000","last_became_peered":"2024-06-29T14:23:16.200584+0000","last_unstale":"2024-06-29T14:23:16.207171+0000","last_undegraded":"2024-06-29T14:23:16.207171+0000","last_fullsized":"2024-06-29T14:23:16.207171+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:19.077899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.194138+0000","last_change":"2024-06-29T14:23:15.203329+0000","last_active":"2024-06-29T14:23:16.194138+0000","last_peered":"2024-06-29T14:23:16.194138+0000","last_clean":"2024-06-29T14:23:16.194138+0000","last_became_active":"2024-06-29T14:23:15.202429+0000","last_became_peered":"2024-06-29T14:23:15.202429+0000","last_unstale":"2024-06-29T14:23:16.194138+0000","last_undegraded":"2024-06-29T14:23:16.194138+0000","last_fullsized":"2024-06-29T14:23:16.194138+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.198488+0000","last_change":"2024-06-29T14:23:16.190685+0000","last_active":"2024-06-29T14:23:16.198488+0000","last_peered":"2024-06-29T14:23:16.198488+0000","last_clean":"2024-06-29T14:23:16.198488+0000","last_became_active":"2024-06-29T14:23:16.190310+0000","last_became_peered":"2024-06-29T14:23:16.190310+0000","last_unstale":"2024-06-29T14:23:16.198488+0000","last_undegraded":"2024-06-29T14:23:16.198488+0000","last_fullsized":"2024-06-29T14:23:16.198488+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:08.341184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202801+0000","last_change":"2024-06-29T14:23:15.200968+0000","last_active":"2024-06-29T14:23:15.202801+0000","last_peered":"2024-06-29T14:23:15.202801+0000","last_clean":"2024-06-29T14:23:15.202801+0000","last_became_active":"2024-06-29T14:23:15.200573+0000","last_became_peered":"2024-06-29T14:23:15.200573+0000","last_unstale":"2024-06-29T14:23:15.202801+0000","last_undegraded":"2024-06-29T14:23:15.202801+0000","last_fullsized":"2024-06-29T14:23:15.202801+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:34.092009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193645+0000","last_change":"2024-06-29T14:23:15.193645+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193645+0000","last_undegraded":"2024-06-29T14:23:15.193645+0000","last_fullsized":"2024-06-29T14:23:15.193645+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201435+0000","last_change":"2024-06-29T14:23:15.197575+0000","last_active":"2024-06-29T14:23:15.201435+0000","last_peered":"2024-06-29T14:23:15.201435+0000","last_clean":"2024-06-29T14:23:15.201435+0000","last_became_active":"2024-06-29T14:23:15.197267+0000","last_became_peered":"2024-06-29T14:23:15.197267+0000","last_unstale":"2024-06-29T14:23:15.201435+0000","last_undegraded":"2024-06-29T14:23:15.201435+0000","last_fullsized":"2024-06-29T14:23:15.201435+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:34.092009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187502+0000","last_change":"2024-06-29T14:23:15.187502+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187502+0000","last_undegraded":"2024-06-29T14:23:15.187502+0000","last_fullsized":"2024-06-29T14:23:15.187502+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.181400+0000","last_change":"2024-06-29T14:23:15.192161+0000","last_active":"2024-06-29T14:23:16.181400+0000","last_peered":"2024-06-29T14:23:16.181400+0000","last_clean":"2024-06-29T14:23:16.181400+0000","last_became_active":"2024-06-29T14:23:15.191810+0000","last_became_peered":"2024-06-29T14:23:15.191810+0000","last_unstale":"2024-06-29T14:23:16.181400+0000","last_undegraded":"2024-06-29T14:23:16.181400+0000","last_fullsized":"2024-06-29T14:23:16.181400+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:28.995567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.197944+0000","last_change":"2024-06-29T14:23:15.197944+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.197944+0000","last_undegraded":"2024-06-29T14:23:15.197944+0000","last_fullsized":"2024-06-29T14:23:15.197944+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202122+0000","last_change":"2024-06-29T14:23:15.198041+0000","last_active":"2024-06-29T14:23:15.202122+0000","last_peered":"2024-06-29T14:23:15.202122+0000","last_clean":"2024-06-29T14:23:15.202122+0000","last_became_active":"2024-06-29T14:23:15.197662+0000","last_became_peered":"2024-06-29T14:23:15.197662+0000","last_unstale":"2024-06-29T14:23:15.202122+0000","last_undegraded":"2024-06-29T14:23:15.202122+0000","last_fullsized":"2024-06-29T14:23:15.202122+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:48.024521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":142,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.190774+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:23:15.190774+0000","last_peered":"2024-06-29T14:23:15.190774+0000","last_clean":"2024-06-29T14:23:15.190774+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:23:15.190774+0000","last_undegraded":"2024-06-29T14:23:15.190774+0000","last_fullsized":"2024-06-29T14:23:15.190774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1716,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":197889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509462,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":196333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":779444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":779444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":203005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:23:20.664 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:23:20.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 -- 172.21.15.59:0/3620627594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 msgr2=0x7f3e780cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 --2- 172.21.15.59:0/3620627594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 0x7f3e780cf080 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3e780cfbc0 tx=0x7f3e6802aa30 comp rx=0 tx=0).stop 2024-06-29T14:23:20.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 -- 172.21.15.59:0/3620627594 shutdown_connections 2024-06-29T14:23:20.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 --2- 172.21.15.59:0/3620627594 >> v2:172.21.15.153:3300/0 conn(0x7f3e780ce950 0x7f3e780d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 --2- 172.21.15.59:0/3620627594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e780cf5c0 0x7f3e780d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 --2- 172.21.15.59:0/3620627594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 0x7f3e780cf080 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 -- 172.21.15.59:0/3620627594 >> 172.21.15.59:0/3620627594 conn(0x7f3e780b5610 msgr2=0x7f3e780b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 -- 172.21.15.59:0/3620627594 shutdown_connections 2024-06-29T14:23:20.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.820+0000 7f3e806df640 1 -- 172.21.15.59:0/3620627594 wait complete. 2024-06-29T14:23:20.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.822+0000 7f3e806df640 1 Processor -- start 2024-06-29T14:23:20.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.822+0000 7f3e806df640 1 -- start start 2024-06-29T14:23:20.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e806df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 0x7f3e780aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e806df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e780ce950 0x7f3e780ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e806df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e780cf5c0 0x7f3e780acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e806df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3e78162440 con 0x7f3e7808c190 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e806df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e781622c0 con 0x7f3e780ce950 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e806df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3e781625c0 con 0x7f3e780cf5c0 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e7e47b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 0x7f3e780aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e7e47b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 0x7f3e780aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48196/0 (socket says 172.21.15.59:48196) 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.823+0000 7f3e7e47b640 1 -- 172.21.15.59:0/3056643826 learned_addr learned my addr 172.21.15.59:0/3056643826 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7dc7a640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e780ce950 0x7f3e780ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7ec7c640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e780cf5c0 0x7f3e780acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7e47b640 1 -- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e780cf5c0 msgr2=0x7f3e780acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7e47b640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e780cf5c0 0x7f3e780acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7e47b640 1 -- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e780ce950 msgr2=0x7f3e780ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7e47b640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e780ce950 0x7f3e780ac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7e47b640 1 -- 172.21.15.59:0/3056643826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e780aaa30 con 0x7f3e7808c190 2024-06-29T14:23:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7ec7c640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e780cf5c0 0x7f3e780acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e7e47b640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 0x7f3e780aa4f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3e6802b020 tx=0x7f3e68003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e76ffd640 1 -- 172.21.15.59:0/3056643826 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68033050 con 0x7f3e7808c190 2024-06-29T14:23:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e76ffd640 1 -- 172.21.15.59:0/3056643826 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e680044f0 con 0x7f3e7808c190 2024-06-29T14:23:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e780aaf40 con 0x7f3e7808c190 2024-06-29T14:23:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e76ffd640 1 -- 172.21.15.59:0/3056643826 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68038450 con 0x7f3e7808c190 2024-06-29T14:23:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.824+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e780ab500 con 0x7f3e7808c190 2024-06-29T14:23:20.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.825+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e780c9ef0 con 0x7f3e7808c190 2024-06-29T14:23:20.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.826+0000 7f3e76ffd640 1 -- 172.21.15.59:0/3056643826 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3e68036070 con 0x7f3e7808c190 2024-06-29T14:23:20.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.827+0000 7f3e76ffd640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e640762d0 0x7f3e640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.827+0000 7f3e76ffd640 1 -- 172.21.15.59:0/3056643826 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f3e680b8e30 con 0x7f3e7808c190 2024-06-29T14:23:20.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.827+0000 7f3e7dc7a640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e640762d0 0x7f3e640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.828+0000 7f3e7dc7a640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e640762d0 0x7f3e640787e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3e780abe20 tx=0x7f3e6c00b2e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:20.831+0000 7f3e76ffd640 1 -- 172.21.15.59:0/3056643826 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3e780c9ef0 con 0x7f3e7808c190 2024-06-29T14:23:21.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.017+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3e780cf080 con 0x7f3e640762d0 2024-06-29T14:23:21.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.021+0000 7f3e76ffd640 1 -- 172.21.15.59:0/3056643826 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249106 (secure 0 0 0) 0x7f3e780cf080 con 0x7f3e640762d0 2024-06-29T14:23:21.022 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:21.023 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e640762d0 msgr2=0x7f3e640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e640762d0 0x7f3e640787e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3e780abe20 tx=0x7f3e6c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 msgr2=0x7f3e780aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 0x7f3e780aa4f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3e6802b020 tx=0x7f3e68003f10 comp rx=0 tx=0).stop 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 shutdown_connections 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f3e640762d0 0x7f3e640787e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e780cf5c0 0x7f3e780acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e780ce950 0x7f3e780ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 --2- 172.21.15.59:0/3056643826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7808c190 0x7f3e780aa4f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.024+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 >> 172.21.15.59:0/3056643826 conn(0x7f3e780b5610 msgr2=0x7f3e780c8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.025+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 shutdown_connections 2024-06-29T14:23:21.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.025+0000 7f3e806df640 1 -- 172.21.15.59:0/3056643826 wait complete. 2024-06-29T14:23:21.033 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-06-29T14:23:19.752591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592084,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":116,"num_write_kb":1639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220812,"kb_used_data":4652,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753908,"statfs":{"total":773094113280,"available":772868001792,"internally_reserved":0,"allocated":4763648,"data_stored":2703342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1716,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.579212"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.211606+0000","last_change":"2024-06-29T14:23:16.201160+0000","last_active":"2024-06-29T14:23:16.211606+0000","last_peered":"2024-06-29T14:23:16.211606+0000","last_clean":"2024-06-29T14:23:16.211606+0000","last_became_active":"2024-06-29T14:23:16.200155+0000","last_became_peered":"2024-06-29T14:23:16.200155+0000","last_unstale":"2024-06-29T14:23:16.211606+0000","last_undegraded":"2024-06-29T14:23:16.211606+0000","last_fullsized":"2024-06-29T14:23:16.211606+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:23.684550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.199308+0000","last_change":"2024-06-29T14:23:16.189930+0000","last_active":"2024-06-29T14:23:16.199308+0000","last_peered":"2024-06-29T14:23:16.199308+0000","last_clean":"2024-06-29T14:23:16.199308+0000","last_became_active":"2024-06-29T14:23:16.189571+0000","last_became_peered":"2024-06-29T14:23:16.189571+0000","last_unstale":"2024-06-29T14:23:16.199308+0000","last_undegraded":"2024-06-29T14:23:16.199308+0000","last_fullsized":"2024-06-29T14:23:16.199308+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:12.662244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207289+0000","last_change":"2024-06-29T14:23:16.203182+0000","last_active":"2024-06-29T14:23:16.207289+0000","last_peered":"2024-06-29T14:23:16.207289+0000","last_clean":"2024-06-29T14:23:16.207289+0000","last_became_active":"2024-06-29T14:23:16.202851+0000","last_became_peered":"2024-06-29T14:23:16.202851+0000","last_unstale":"2024-06-29T14:23:16.207289+0000","last_undegraded":"2024-06-29T14:23:16.207289+0000","last_fullsized":"2024-06-29T14:23:16.207289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:13.507757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972956+0000","last_change":"2024-06-29T14:23:16.202834+0000","last_active":"2024-06-29T14:23:17.972956+0000","last_peered":"2024-06-29T14:23:17.972956+0000","last_clean":"2024-06-29T14:23:17.972956+0000","last_became_active":"2024-06-29T14:23:16.202357+0000","last_became_peered":"2024-06-29T14:23:16.202357+0000","last_unstale":"2024-06-29T14:23:17.972956+0000","last_undegraded":"2024-06-29T14:23:17.972956+0000","last_fullsized":"2024-06-29T14:23:17.972956+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:35.097600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972601+0000","last_change":"2024-06-29T14:23:16.194759+0000","last_active":"2024-06-29T14:23:17.972601+0000","last_peered":"2024-06-29T14:23:17.972601+0000","last_clean":"2024-06-29T14:23:17.972601+0000","last_became_active":"2024-06-29T14:23:16.194351+0000","last_became_peered":"2024-06-29T14:23:16.194351+0000","last_unstale":"2024-06-29T14:23:17.972601+0000","last_undegraded":"2024-06-29T14:23:17.972601+0000","last_fullsized":"2024-06-29T14:23:17.972601+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:30.838175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207114+0000","last_change":"2024-06-29T14:23:16.195834+0000","last_active":"2024-06-29T14:23:16.207114+0000","last_peered":"2024-06-29T14:23:16.207114+0000","last_clean":"2024-06-29T14:23:16.207114+0000","last_became_active":"2024-06-29T14:23:16.195476+0000","last_became_peered":"2024-06-29T14:23:16.195476+0000","last_unstale":"2024-06-29T14:23:16.207114+0000","last_undegraded":"2024-06-29T14:23:16.207114+0000","last_fullsized":"2024-06-29T14:23:16.207114+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:30.001157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192292+0000","last_change":"2024-06-29T14:23:15.192292+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192292+0000","last_undegraded":"2024-06-29T14:23:15.192292+0000","last_fullsized":"2024-06-29T14:23:15.192292+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.206121+0000","last_change":"2024-06-29T14:23:16.199825+0000","last_active":"2024-06-29T14:23:16.206121+0000","last_peered":"2024-06-29T14:23:16.206121+0000","last_clean":"2024-06-29T14:23:16.206121+0000","last_became_active":"2024-06-29T14:23:16.199499+0000","last_became_peered":"2024-06-29T14:23:16.199499+0000","last_unstale":"2024-06-29T14:23:16.206121+0000","last_undegraded":"2024-06-29T14:23:16.206121+0000","last_fullsized":"2024-06-29T14:23:16.206121+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:10.882976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214762+0000","last_change":"2024-06-29T14:23:16.209692+0000","last_active":"2024-06-29T14:23:16.214762+0000","last_peered":"2024-06-29T14:23:16.214762+0000","last_clean":"2024-06-29T14:23:16.214762+0000","last_became_active":"2024-06-29T14:23:16.202968+0000","last_became_peered":"2024-06-29T14:23:16.202968+0000","last_unstale":"2024-06-29T14:23:16.214762+0000","last_undegraded":"2024-06-29T14:23:16.214762+0000","last_fullsized":"2024-06-29T14:23:16.214762+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:10.402585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.194916+0000","last_change":"2024-06-29T14:23:15.194916+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.194916+0000","last_undegraded":"2024-06-29T14:23:15.194916+0000","last_fullsized":"2024-06-29T14:23:15.194916+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193674+0000","last_change":"2024-06-29T14:23:15.193674+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193674+0000","last_undegraded":"2024-06-29T14:23:15.193674+0000","last_fullsized":"2024-06-29T14:23:15.193674+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214775+0000","last_change":"2024-06-29T14:23:16.209122+0000","last_active":"2024-06-29T14:23:16.214775+0000","last_peered":"2024-06-29T14:23:16.214775+0000","last_clean":"2024-06-29T14:23:16.214775+0000","last_became_active":"2024-06-29T14:23:16.208783+0000","last_became_peered":"2024-06-29T14:23:16.208783+0000","last_unstale":"2024-06-29T14:23:16.214775+0000","last_undegraded":"2024-06-29T14:23:16.214775+0000","last_fullsized":"2024-06-29T14:23:16.214775+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:30.001157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.196595+0000","last_change":"2024-06-29T14:23:15.196595+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.196595+0000","last_undegraded":"2024-06-29T14:23:15.196595+0000","last_fullsized":"2024-06-29T14:23:15.196595+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.194142+0000","last_change":"2024-06-29T14:23:15.194142+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.194142+0000","last_undegraded":"2024-06-29T14:23:15.194142+0000","last_fullsized":"2024-06-29T14:23:15.194142+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191760+0000","last_change":"2024-06-29T14:23:15.191760+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191760+0000","last_undegraded":"2024-06-29T14:23:15.191760+0000","last_fullsized":"2024-06-29T14:23:15.191760+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189097+0000","last_change":"2024-06-29T14:23:15.189097+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189097+0000","last_undegraded":"2024-06-29T14:23:15.189097+0000","last_fullsized":"2024-06-29T14:23:15.189097+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188330+0000","last_change":"2024-06-29T14:23:15.188330+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188330+0000","last_undegraded":"2024-06-29T14:23:15.188330+0000","last_fullsized":"2024-06-29T14:23:15.188330+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.184396+0000","last_change":"2024-06-29T14:23:15.184396+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.184396+0000","last_undegraded":"2024-06-29T14:23:15.184396+0000","last_fullsized":"2024-06-29T14:23:15.184396+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972388+0000","last_change":"2024-06-29T14:23:16.190111+0000","last_active":"2024-06-29T14:23:17.972388+0000","last_peered":"2024-06-29T14:23:17.972388+0000","last_clean":"2024-06-29T14:23:17.972388+0000","last_became_active":"2024-06-29T14:23:16.189786+0000","last_became_peered":"2024-06-29T14:23:16.189786+0000","last_unstale":"2024-06-29T14:23:17.972388+0000","last_undegraded":"2024-06-29T14:23:17.972388+0000","last_fullsized":"2024-06-29T14:23:17.972388+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:12.500255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.184382+0000","last_change":"2024-06-29T14:23:15.184382+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.184382+0000","last_undegraded":"2024-06-29T14:23:15.184382+0000","last_fullsized":"2024-06-29T14:23:15.184382+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.190853+0000","last_change":"2024-06-29T14:23:15.190853+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.190853+0000","last_undegraded":"2024-06-29T14:23:15.190853+0000","last_fullsized":"2024-06-29T14:23:15.190853+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193127+0000","last_change":"2024-06-29T14:23:15.193127+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193127+0000","last_undegraded":"2024-06-29T14:23:15.193127+0000","last_fullsized":"2024-06-29T14:23:15.193127+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188344+0000","last_change":"2024-06-29T14:23:15.188344+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188344+0000","last_undegraded":"2024-06-29T14:23:15.188344+0000","last_fullsized":"2024-06-29T14:23:15.188344+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972485+0000","last_change":"2024-06-29T14:23:16.210617+0000","last_active":"2024-06-29T14:23:17.972485+0000","last_peered":"2024-06-29T14:23:17.972485+0000","last_clean":"2024-06-29T14:23:17.972485+0000","last_became_active":"2024-06-29T14:23:16.210242+0000","last_became_peered":"2024-06-29T14:23:16.210242+0000","last_unstale":"2024-06-29T14:23:17.972485+0000","last_undegraded":"2024-06-29T14:23:17.972485+0000","last_fullsized":"2024-06-29T14:23:17.972485+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:21.067086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185688+0000","last_change":"2024-06-29T14:23:15.185688+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185688+0000","last_undegraded":"2024-06-29T14:23:15.185688+0000","last_fullsized":"2024-06-29T14:23:15.185688+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207195+0000","last_change":"2024-06-29T14:23:16.205741+0000","last_active":"2024-06-29T14:23:16.207195+0000","last_peered":"2024-06-29T14:23:16.207195+0000","last_clean":"2024-06-29T14:23:16.207195+0000","last_became_active":"2024-06-29T14:23:16.205351+0000","last_became_peered":"2024-06-29T14:23:16.205351+0000","last_unstale":"2024-06-29T14:23:16.207195+0000","last_undegraded":"2024-06-29T14:23:16.207195+0000","last_fullsized":"2024-06-29T14:23:16.207195+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:10.402585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185734+0000","last_change":"2024-06-29T14:23:15.185734+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185734+0000","last_undegraded":"2024-06-29T14:23:15.185734+0000","last_fullsized":"2024-06-29T14:23:15.185734+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192802+0000","last_change":"2024-06-29T14:23:15.192802+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192802+0000","last_undegraded":"2024-06-29T14:23:15.192802+0000","last_fullsized":"2024-06-29T14:23:15.192802+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.186290+0000","last_change":"2024-06-29T14:23:15.186290+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.186290+0000","last_undegraded":"2024-06-29T14:23:15.186290+0000","last_fullsized":"2024-06-29T14:23:15.186290+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.185611+0000","last_change":"2024-06-29T14:23:15.185611+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.185611+0000","last_undegraded":"2024-06-29T14:23:15.185611+0000","last_fullsized":"2024-06-29T14:23:15.185611+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187368+0000","last_change":"2024-06-29T14:23:15.187368+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187368+0000","last_undegraded":"2024-06-29T14:23:15.187368+0000","last_fullsized":"2024-06-29T14:23:15.187368+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207193+0000","last_change":"2024-06-29T14:23:16.204276+0000","last_active":"2024-06-29T14:23:16.207193+0000","last_peered":"2024-06-29T14:23:16.207193+0000","last_clean":"2024-06-29T14:23:16.207193+0000","last_became_active":"2024-06-29T14:23:16.203937+0000","last_became_peered":"2024-06-29T14:23:16.203937+0000","last_unstale":"2024-06-29T14:23:16.207193+0000","last_undegraded":"2024-06-29T14:23:16.207193+0000","last_fullsized":"2024-06-29T14:23:16.207193+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:40.723736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.188473+0000","last_change":"2024-06-29T14:23:15.188473+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.188473+0000","last_undegraded":"2024-06-29T14:23:15.188473+0000","last_fullsized":"2024-06-29T14:23:15.188473+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.199746+0000","last_change":"2024-06-29T14:23:15.197029+0000","last_active":"2024-06-29T14:23:15.199746+0000","last_peered":"2024-06-29T14:23:15.199746+0000","last_clean":"2024-06-29T14:23:15.199746+0000","last_became_active":"2024-06-29T14:23:15.196636+0000","last_became_peered":"2024-06-29T14:23:15.196636+0000","last_unstale":"2024-06-29T14:23:15.199746+0000","last_undegraded":"2024-06-29T14:23:15.199746+0000","last_fullsized":"2024-06-29T14:23:15.199746+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:29.832584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.208350+0000","last_change":"2024-06-29T14:23:16.206697+0000","last_active":"2024-06-29T14:23:16.208350+0000","last_peered":"2024-06-29T14:23:16.208350+0000","last_clean":"2024-06-29T14:23:16.208350+0000","last_became_active":"2024-06-29T14:23:16.206325+0000","last_became_peered":"2024-06-29T14:23:16.206325+0000","last_unstale":"2024-06-29T14:23:16.208350+0000","last_undegraded":"2024-06-29T14:23:16.208350+0000","last_fullsized":"2024-06-29T14:23:16.208350+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"40'1","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217808+0000","last_change":"2024-06-29T14:23:15.204215+0000","last_active":"2024-06-29T14:23:16.217808+0000","last_peered":"2024-06-29T14:23:16.217808+0000","last_clean":"2024-06-29T14:23:16.217808+0000","last_became_active":"2024-06-29T14:23:15.203861+0000","last_became_peered":"2024-06-29T14:23:15.203861+0000","last_unstale":"2024-06-29T14:23:16.217808+0000","last_undegraded":"2024-06-29T14:23:16.217808+0000","last_fullsized":"2024-06-29T14:23:16.217808+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:15.271848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189311+0000","last_change":"2024-06-29T14:23:15.189311+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189311+0000","last_undegraded":"2024-06-29T14:23:15.189311+0000","last_fullsized":"2024-06-29T14:23:15.189311+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"40'2","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.219346+0000","last_change":"2024-06-29T14:23:15.205028+0000","last_active":"2024-06-29T14:23:16.219346+0000","last_peered":"2024-06-29T14:23:16.219346+0000","last_clean":"2024-06-29T14:23:16.219346+0000","last_became_active":"2024-06-29T14:23:15.204734+0000","last_became_peered":"2024-06-29T14:23:15.204734+0000","last_unstale":"2024-06-29T14:23:16.219346+0000","last_undegraded":"2024-06-29T14:23:16.219346+0000","last_fullsized":"2024-06-29T14:23:16.219346+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:12.502166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.198933+0000","last_change":"2024-06-29T14:23:16.190648+0000","last_active":"2024-06-29T14:23:16.198933+0000","last_peered":"2024-06-29T14:23:16.198933+0000","last_clean":"2024-06-29T14:23:16.198933+0000","last_became_active":"2024-06-29T14:23:16.190331+0000","last_became_peered":"2024-06-29T14:23:16.190331+0000","last_unstale":"2024-06-29T14:23:16.198933+0000","last_undegraded":"2024-06-29T14:23:16.198933+0000","last_fullsized":"2024-06-29T14:23:16.198933+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:52.309099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201353+0000","last_change":"2024-06-29T14:23:15.195155+0000","last_active":"2024-06-29T14:23:15.201353+0000","last_peered":"2024-06-29T14:23:15.201353+0000","last_clean":"2024-06-29T14:23:15.201353+0000","last_became_active":"2024-06-29T14:23:15.194809+0000","last_became_peered":"2024-06-29T14:23:15.194809+0000","last_unstale":"2024-06-29T14:23:15.201353+0000","last_undegraded":"2024-06-29T14:23:15.201353+0000","last_fullsized":"2024-06-29T14:23:15.201353+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:29.832584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214639+0000","last_change":"2024-06-29T14:23:16.209386+0000","last_active":"2024-06-29T14:23:16.214639+0000","last_peered":"2024-06-29T14:23:16.214639+0000","last_clean":"2024-06-29T14:23:16.214639+0000","last_became_active":"2024-06-29T14:23:16.209011+0000","last_became_peered":"2024-06-29T14:23:16.209011+0000","last_unstale":"2024-06-29T14:23:16.214639+0000","last_undegraded":"2024-06-29T14:23:16.214639+0000","last_fullsized":"2024-06-29T14:23:16.214639+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:40.723736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203843+0000","last_change":"2024-06-29T14:23:15.201776+0000","last_active":"2024-06-29T14:23:15.203843+0000","last_peered":"2024-06-29T14:23:15.203843+0000","last_clean":"2024-06-29T14:23:15.203843+0000","last_became_active":"2024-06-29T14:23:15.201376+0000","last_became_peered":"2024-06-29T14:23:15.201376+0000","last_unstale":"2024-06-29T14:23:15.203843+0000","last_undegraded":"2024-06-29T14:23:15.203843+0000","last_fullsized":"2024-06-29T14:23:15.203843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191269+0000","last_change":"2024-06-29T14:23:15.191269+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191269+0000","last_undegraded":"2024-06-29T14:23:15.191269+0000","last_fullsized":"2024-06-29T14:23:15.191269+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.204276+0000","last_change":"2024-06-29T14:23:15.203169+0000","last_active":"2024-06-29T14:23:15.204276+0000","last_peered":"2024-06-29T14:23:15.204276+0000","last_clean":"2024-06-29T14:23:15.204276+0000","last_became_active":"2024-06-29T14:23:15.202809+0000","last_became_peered":"2024-06-29T14:23:15.202809+0000","last_unstale":"2024-06-29T14:23:15.204276+0000","last_undegraded":"2024-06-29T14:23:15.204276+0000","last_fullsized":"2024-06-29T14:23:15.204276+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:29.639570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191458+0000","last_change":"2024-06-29T14:23:15.191458+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191458+0000","last_undegraded":"2024-06-29T14:23:15.191458+0000","last_fullsized":"2024-06-29T14:23:15.191458+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185188+0000","last_change":"2024-06-29T14:23:15.200889+0000","last_active":"2024-06-29T14:23:16.185188+0000","last_peered":"2024-06-29T14:23:16.185188+0000","last_clean":"2024-06-29T14:23:16.185188+0000","last_became_active":"2024-06-29T14:23:15.200521+0000","last_became_peered":"2024-06-29T14:23:15.200521+0000","last_unstale":"2024-06-29T14:23:16.185188+0000","last_undegraded":"2024-06-29T14:23:16.185188+0000","last_fullsized":"2024-06-29T14:23:16.185188+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217265+0000","last_change":"2024-06-29T14:23:16.207437+0000","last_active":"2024-06-29T14:23:16.217265+0000","last_peered":"2024-06-29T14:23:16.217265+0000","last_clean":"2024-06-29T14:23:16.217265+0000","last_became_active":"2024-06-29T14:23:16.207056+0000","last_became_peered":"2024-06-29T14:23:16.207056+0000","last_unstale":"2024-06-29T14:23:16.217265+0000","last_undegraded":"2024-06-29T14:23:16.217265+0000","last_fullsized":"2024-06-29T14:23:16.217265+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:52.297506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203620+0000","last_change":"2024-06-29T14:23:15.199990+0000","last_active":"2024-06-29T14:23:15.203620+0000","last_peered":"2024-06-29T14:23:15.203620+0000","last_clean":"2024-06-29T14:23:15.203620+0000","last_became_active":"2024-06-29T14:23:15.199697+0000","last_became_peered":"2024-06-29T14:23:15.199697+0000","last_unstale":"2024-06-29T14:23:15.203620+0000","last_undegraded":"2024-06-29T14:23:15.203620+0000","last_fullsized":"2024-06-29T14:23:15.203620+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:54.406021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.973018+0000","last_change":"2024-06-29T14:23:16.197089+0000","last_active":"2024-06-29T14:23:17.973018+0000","last_peered":"2024-06-29T14:23:17.973018+0000","last_clean":"2024-06-29T14:23:17.973018+0000","last_became_active":"2024-06-29T14:23:16.196730+0000","last_became_peered":"2024-06-29T14:23:16.196730+0000","last_unstale":"2024-06-29T14:23:17.973018+0000","last_undegraded":"2024-06-29T14:23:17.973018+0000","last_fullsized":"2024-06-29T14:23:17.973018+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:55.411612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.182345+0000","last_change":"2024-06-29T14:23:15.189674+0000","last_active":"2024-06-29T14:23:16.182345+0000","last_peered":"2024-06-29T14:23:16.182345+0000","last_clean":"2024-06-29T14:23:16.182345+0000","last_became_active":"2024-06-29T14:23:15.189326+0000","last_became_peered":"2024-06-29T14:23:15.189326+0000","last_unstale":"2024-06-29T14:23:16.182345+0000","last_undegraded":"2024-06-29T14:23:16.182345+0000","last_fullsized":"2024-06-29T14:23:16.182345+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:09.396994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.190068+0000","last_change":"2024-06-29T14:23:15.190068+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.190068+0000","last_undegraded":"2024-06-29T14:23:15.190068+0000","last_fullsized":"2024-06-29T14:23:15.190068+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.207378+0000","last_change":"2024-06-29T14:23:15.205426+0000","last_active":"2024-06-29T14:23:15.207378+0000","last_peered":"2024-06-29T14:23:15.207378+0000","last_clean":"2024-06-29T14:23:15.207378+0000","last_became_active":"2024-06-29T14:23:15.204970+0000","last_became_peered":"2024-06-29T14:23:15.204970+0000","last_unstale":"2024-06-29T14:23:15.207378+0000","last_undegraded":"2024-06-29T14:23:15.207378+0000","last_fullsized":"2024-06-29T14:23:15.207378+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:12.502166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.213758+0000","last_change":"2024-06-29T14:23:16.208578+0000","last_active":"2024-06-29T14:23:16.213758+0000","last_peered":"2024-06-29T14:23:16.213758+0000","last_clean":"2024-06-29T14:23:16.213758+0000","last_became_active":"2024-06-29T14:23:16.208189+0000","last_became_peered":"2024-06-29T14:23:16.208189+0000","last_unstale":"2024-06-29T14:23:16.213758+0000","last_undegraded":"2024-06-29T14:23:16.213758+0000","last_fullsized":"2024-06-29T14:23:16.213758+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:08.341184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203812+0000","last_change":"2024-06-29T14:23:15.202041+0000","last_active":"2024-06-29T14:23:15.203812+0000","last_peered":"2024-06-29T14:23:15.203812+0000","last_clean":"2024-06-29T14:23:15.203812+0000","last_became_active":"2024-06-29T14:23:15.201722+0000","last_became_peered":"2024-06-29T14:23:15.201722+0000","last_unstale":"2024-06-29T14:23:15.203812+0000","last_undegraded":"2024-06-29T14:23:15.203812+0000","last_fullsized":"2024-06-29T14:23:15.203812+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:18.072309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.191167+0000","last_change":"2024-06-29T14:23:15.191167+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.191167+0000","last_undegraded":"2024-06-29T14:23:15.191167+0000","last_fullsized":"2024-06-29T14:23:15.191167+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"40'6","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.220943+0000","last_change":"2024-06-29T14:23:15.199353+0000","last_active":"2024-06-29T14:23:16.220943+0000","last_peered":"2024-06-29T14:23:16.220943+0000","last_clean":"2024-06-29T14:23:16.220943+0000","last_became_active":"2024-06-29T14:23:15.198339+0000","last_became_peered":"2024-06-29T14:23:15.198339+0000","last_unstale":"2024-06-29T14:23:16.220943+0000","last_undegraded":"2024-06-29T14:23:16.220943+0000","last_fullsized":"2024-06-29T14:23:16.220943+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:39.718146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214670+0000","last_change":"2024-06-29T14:23:16.209778+0000","last_active":"2024-06-29T14:23:16.214670+0000","last_peered":"2024-06-29T14:23:16.214670+0000","last_clean":"2024-06-29T14:23:16.214670+0000","last_became_active":"2024-06-29T14:23:16.209443+0000","last_became_peered":"2024-06-29T14:23:16.209443+0000","last_unstale":"2024-06-29T14:23:16.214670+0000","last_undegraded":"2024-06-29T14:23:16.214670+0000","last_fullsized":"2024-06-29T14:23:16.214670+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:55.671459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"40'2","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.219006+0000","last_change":"2024-06-29T14:23:15.200701+0000","last_active":"2024-06-29T14:23:16.219006+0000","last_peered":"2024-06-29T14:23:16.219006+0000","last_clean":"2024-06-29T14:23:16.219006+0000","last_became_active":"2024-06-29T14:23:15.200338+0000","last_became_peered":"2024-06-29T14:23:15.200338+0000","last_unstale":"2024-06-29T14:23:16.219006+0000","last_undegraded":"2024-06-29T14:23:16.219006+0000","last_fullsized":"2024-06-29T14:23:16.219006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:54.665868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.201400+0000","last_change":"2024-06-29T14:23:16.196448+0000","last_active":"2024-06-29T14:23:16.201400+0000","last_peered":"2024-06-29T14:23:16.201400+0000","last_clean":"2024-06-29T14:23:16.201400+0000","last_became_active":"2024-06-29T14:23:16.195006+0000","last_became_peered":"2024-06-29T14:23:16.195006+0000","last_unstale":"2024-06-29T14:23:16.201400+0000","last_undegraded":"2024-06-29T14:23:16.201400+0000","last_fullsized":"2024-06-29T14:23:16.201400+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:05.074083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185776+0000","last_change":"2024-06-29T14:23:15.199955+0000","last_active":"2024-06-29T14:23:16.185776+0000","last_peered":"2024-06-29T14:23:16.185776+0000","last_clean":"2024-06-29T14:23:16.185776+0000","last_became_active":"2024-06-29T14:23:15.199619+0000","last_became_peered":"2024-06-29T14:23:15.199619+0000","last_unstale":"2024-06-29T14:23:16.185776+0000","last_undegraded":"2024-06-29T14:23:16.185776+0000","last_fullsized":"2024-06-29T14:23:16.185776+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:04.589609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187426+0000","last_change":"2024-06-29T14:23:15.187426+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187426+0000","last_undegraded":"2024-06-29T14:23:15.187426+0000","last_fullsized":"2024-06-29T14:23:15.187426+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.198643+0000","last_change":"2024-06-29T14:23:15.196439+0000","last_active":"2024-06-29T14:23:15.198643+0000","last_peered":"2024-06-29T14:23:15.198643+0000","last_clean":"2024-06-29T14:23:15.198643+0000","last_became_active":"2024-06-29T14:23:15.196116+0000","last_became_peered":"2024-06-29T14:23:15.196116+0000","last_unstale":"2024-06-29T14:23:15.198643+0000","last_undegraded":"2024-06-29T14:23:15.198643+0000","last_fullsized":"2024-06-29T14:23:15.198643+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:20.061496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.213976+0000","last_change":"2024-06-29T14:23:16.200847+0000","last_active":"2024-06-29T14:23:16.213976+0000","last_peered":"2024-06-29T14:23:16.213976+0000","last_clean":"2024-06-29T14:23:16.213976+0000","last_became_active":"2024-06-29T14:23:16.200264+0000","last_became_peered":"2024-06-29T14:23:16.200264+0000","last_unstale":"2024-06-29T14:23:16.213976+0000","last_undegraded":"2024-06-29T14:23:16.213976+0000","last_fullsized":"2024-06-29T14:23:16.213976+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:46.554739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202335+0000","last_change":"2024-06-29T14:23:15.200739+0000","last_active":"2024-06-29T14:23:15.202335+0000","last_peered":"2024-06-29T14:23:15.202335+0000","last_clean":"2024-06-29T14:23:15.202335+0000","last_became_active":"2024-06-29T14:23:15.200401+0000","last_became_peered":"2024-06-29T14:23:15.200401+0000","last_unstale":"2024-06-29T14:23:15.202335+0000","last_undegraded":"2024-06-29T14:23:15.202335+0000","last_fullsized":"2024-06-29T14:23:15.202335+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:49.969939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.194644+0000","last_change":"2024-06-29T14:23:16.188970+0000","last_active":"2024-06-29T14:23:16.194644+0000","last_peered":"2024-06-29T14:23:16.194644+0000","last_clean":"2024-06-29T14:23:16.194644+0000","last_became_active":"2024-06-29T14:23:16.188588+0000","last_became_peered":"2024-06-29T14:23:16.188588+0000","last_unstale":"2024-06-29T14:23:16.194644+0000","last_undegraded":"2024-06-29T14:23:16.194644+0000","last_fullsized":"2024-06-29T14:23:16.194644+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201969+0000","last_change":"2024-06-29T14:23:15.199255+0000","last_active":"2024-06-29T14:23:15.201969+0000","last_peered":"2024-06-29T14:23:15.201969+0000","last_clean":"2024-06-29T14:23:15.201969+0000","last_became_active":"2024-06-29T14:23:15.198806+0000","last_became_peered":"2024-06-29T14:23:15.198806+0000","last_unstale":"2024-06-29T14:23:15.201969+0000","last_undegraded":"2024-06-29T14:23:15.201969+0000","last_fullsized":"2024-06-29T14:23:15.201969+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:41.669608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193179+0000","last_change":"2024-06-29T14:23:15.193179+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193179+0000","last_undegraded":"2024-06-29T14:23:15.193179+0000","last_fullsized":"2024-06-29T14:23:15.193179+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.203612+0000","last_change":"2024-06-29T14:23:15.200790+0000","last_active":"2024-06-29T14:23:15.203612+0000","last_peered":"2024-06-29T14:23:15.203612+0000","last_clean":"2024-06-29T14:23:15.203612+0000","last_became_active":"2024-06-29T14:23:15.200452+0000","last_became_peered":"2024-06-29T14:23:15.200452+0000","last_unstale":"2024-06-29T14:23:15.203612+0000","last_undegraded":"2024-06-29T14:23:15.203612+0000","last_fullsized":"2024-06-29T14:23:15.203612+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:27.766774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.192686+0000","last_change":"2024-06-29T14:23:15.192686+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.192686+0000","last_undegraded":"2024-06-29T14:23:15.192686+0000","last_fullsized":"2024-06-29T14:23:15.192686+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972729+0000","last_change":"2024-06-29T14:23:15.195356+0000","last_active":"2024-06-29T14:23:17.972729+0000","last_peered":"2024-06-29T14:23:17.972729+0000","last_clean":"2024-06-29T14:23:17.972729+0000","last_became_active":"2024-06-29T14:23:15.195037+0000","last_became_peered":"2024-06-29T14:23:15.195037+0000","last_unstale":"2024-06-29T14:23:17.972729+0000","last_undegraded":"2024-06-29T14:23:17.972729+0000","last_fullsized":"2024-06-29T14:23:17.972729+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:32.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971916+0000","last_change":"2024-06-29T14:23:16.197431+0000","last_active":"2024-06-29T14:23:17.971916+0000","last_peered":"2024-06-29T14:23:17.971916+0000","last_clean":"2024-06-29T14:23:17.971916+0000","last_became_active":"2024-06-29T14:23:16.197091+0000","last_became_peered":"2024-06-29T14:23:16.197091+0000","last_unstale":"2024-06-29T14:23:17.971916+0000","last_undegraded":"2024-06-29T14:23:17.971916+0000","last_fullsized":"2024-06-29T14:23:17.971916+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:46.040585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185494+0000","last_change":"2024-06-29T14:23:15.203782+0000","last_active":"2024-06-29T14:23:16.185494+0000","last_peered":"2024-06-29T14:23:16.185494+0000","last_clean":"2024-06-29T14:23:16.185494+0000","last_became_active":"2024-06-29T14:23:15.203451+0000","last_became_peered":"2024-06-29T14:23:15.203451+0000","last_unstale":"2024-06-29T14:23:16.185494+0000","last_undegraded":"2024-06-29T14:23:16.185494+0000","last_fullsized":"2024-06-29T14:23:16.185494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:18.072309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187906+0000","last_change":"2024-06-29T14:23:15.187906+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187906+0000","last_undegraded":"2024-06-29T14:23:15.187906+0000","last_fullsized":"2024-06-29T14:23:15.187906+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972722+0000","last_change":"2024-06-29T14:23:15.201493+0000","last_active":"2024-06-29T14:23:17.972722+0000","last_peered":"2024-06-29T14:23:17.972722+0000","last_clean":"2024-06-29T14:23:17.972722+0000","last_became_active":"2024-06-29T14:23:15.201150+0000","last_became_peered":"2024-06-29T14:23:15.201150+0000","last_unstale":"2024-06-29T14:23:17.972722+0000","last_undegraded":"2024-06-29T14:23:17.972722+0000","last_fullsized":"2024-06-29T14:23:17.972722+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:00.323663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.206186+0000","last_change":"2024-06-29T14:23:16.202653+0000","last_active":"2024-06-29T14:23:16.206186+0000","last_peered":"2024-06-29T14:23:16.206186+0000","last_clean":"2024-06-29T14:23:16.206186+0000","last_became_active":"2024-06-29T14:23:16.202279+0000","last_became_peered":"2024-06-29T14:23:16.202279+0000","last_unstale":"2024-06-29T14:23:16.206186+0000","last_undegraded":"2024-06-29T14:23:16.206186+0000","last_fullsized":"2024-06-29T14:23:16.206186+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:05.595200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971710+0000","last_change":"2024-06-29T14:23:15.196760+0000","last_active":"2024-06-29T14:23:17.971710+0000","last_peered":"2024-06-29T14:23:17.971710+0000","last_clean":"2024-06-29T14:23:17.971710+0000","last_became_active":"2024-06-29T14:23:15.196278+0000","last_became_peered":"2024-06-29T14:23:15.196278+0000","last_unstale":"2024-06-29T14:23:17.971710+0000","last_undegraded":"2024-06-29T14:23:17.971710+0000","last_fullsized":"2024-06-29T14:23:17.971710+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:32:01.571661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193709+0000","last_change":"2024-06-29T14:23:15.193709+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193709+0000","last_undegraded":"2024-06-29T14:23:15.193709+0000","last_fullsized":"2024-06-29T14:23:15.193709+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202585+0000","last_change":"2024-06-29T14:23:15.199837+0000","last_active":"2024-06-29T14:23:15.202585+0000","last_peered":"2024-06-29T14:23:15.202585+0000","last_clean":"2024-06-29T14:23:15.202585+0000","last_became_active":"2024-06-29T14:23:15.199539+0000","last_became_peered":"2024-06-29T14:23:15.199539+0000","last_unstale":"2024-06-29T14:23:15.202585+0000","last_undegraded":"2024-06-29T14:23:15.202585+0000","last_fullsized":"2024-06-29T14:23:15.202585+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:12.157450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217291+0000","last_change":"2024-06-29T14:23:16.211956+0000","last_active":"2024-06-29T14:23:16.217291+0000","last_peered":"2024-06-29T14:23:16.217291+0000","last_clean":"2024-06-29T14:23:16.217291+0000","last_became_active":"2024-06-29T14:23:16.211649+0000","last_became_peered":"2024-06-29T14:23:16.211649+0000","last_unstale":"2024-06-29T14:23:16.217291+0000","last_undegraded":"2024-06-29T14:23:16.217291+0000","last_fullsized":"2024-06-29T14:23:16.217291+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:37.213779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972618+0000","last_change":"2024-06-29T14:23:15.201805+0000","last_active":"2024-06-29T14:23:17.972618+0000","last_peered":"2024-06-29T14:23:17.972618+0000","last_clean":"2024-06-29T14:23:17.972618+0000","last_became_active":"2024-06-29T14:23:15.200924+0000","last_became_peered":"2024-06-29T14:23:15.200924+0000","last_unstale":"2024-06-29T14:23:17.972618+0000","last_undegraded":"2024-06-29T14:23:17.972618+0000","last_fullsized":"2024-06-29T14:23:17.972618+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:18.651137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207150+0000","last_change":"2024-06-29T14:23:16.205768+0000","last_active":"2024-06-29T14:23:16.207150+0000","last_peered":"2024-06-29T14:23:16.207150+0000","last_clean":"2024-06-29T14:23:16.207150+0000","last_became_active":"2024-06-29T14:23:16.205382+0000","last_became_peered":"2024-06-29T14:23:16.205382+0000","last_unstale":"2024-06-29T14:23:16.207150+0000","last_undegraded":"2024-06-29T14:23:16.207150+0000","last_fullsized":"2024-06-29T14:23:16.207150+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:55.671459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.205384+0000","last_change":"2024-06-29T14:23:15.203975+0000","last_active":"2024-06-29T14:23:15.205384+0000","last_peered":"2024-06-29T14:23:15.205384+0000","last_clean":"2024-06-29T14:23:15.205384+0000","last_became_active":"2024-06-29T14:23:15.203652+0000","last_became_peered":"2024-06-29T14:23:15.203652+0000","last_unstale":"2024-06-29T14:23:15.205384+0000","last_undegraded":"2024-06-29T14:23:15.205384+0000","last_fullsized":"2024-06-29T14:23:15.205384+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:04.589609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.189204+0000","last_change":"2024-06-29T14:23:15.189204+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.189204+0000","last_undegraded":"2024-06-29T14:23:15.189204+0000","last_fullsized":"2024-06-29T14:23:15.189204+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"41'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:18.201843+0000","last_change":"2024-06-29T14:23:15.199068+0000","last_active":"2024-06-29T14:23:18.201843+0000","last_peered":"2024-06-29T14:23:18.201843+0000","last_clean":"2024-06-29T14:23:18.201843+0000","last_became_active":"2024-06-29T14:23:15.198714+0000","last_became_peered":"2024-06-29T14:23:15.198714+0000","last_unstale":"2024-06-29T14:23:18.201843+0000","last_undegraded":"2024-06-29T14:23:18.201843+0000","last_fullsized":"2024-06-29T14:23:18.201843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:11.722673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.199343+0000","last_change":"2024-06-29T14:23:16.194459+0000","last_active":"2024-06-29T14:23:16.199343+0000","last_peered":"2024-06-29T14:23:16.199343+0000","last_clean":"2024-06-29T14:23:16.199343+0000","last_became_active":"2024-06-29T14:23:16.194084+0000","last_became_peered":"2024-06-29T14:23:16.194084+0000","last_unstale":"2024-06-29T14:23:16.199343+0000","last_undegraded":"2024-06-29T14:23:16.199343+0000","last_fullsized":"2024-06-29T14:23:16.199343+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:46.325166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.192313+0000","last_change":"2024-06-29T14:23:15.199199+0000","last_active":"2024-06-29T14:23:16.192313+0000","last_peered":"2024-06-29T14:23:16.192313+0000","last_clean":"2024-06-29T14:23:16.192313+0000","last_became_active":"2024-06-29T14:23:15.198802+0000","last_became_peered":"2024-06-29T14:23:15.198802+0000","last_unstale":"2024-06-29T14:23:16.192313+0000","last_undegraded":"2024-06-29T14:23:16.192313+0000","last_fullsized":"2024-06-29T14:23:16.192313+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:41.669608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207171+0000","last_change":"2024-06-29T14:23:16.200965+0000","last_active":"2024-06-29T14:23:16.207171+0000","last_peered":"2024-06-29T14:23:16.207171+0000","last_clean":"2024-06-29T14:23:16.207171+0000","last_became_active":"2024-06-29T14:23:16.200584+0000","last_became_peered":"2024-06-29T14:23:16.200584+0000","last_unstale":"2024-06-29T14:23:16.207171+0000","last_undegraded":"2024-06-29T14:23:16.207171+0000","last_fullsized":"2024-06-29T14:23:16.207171+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:19.077899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.194138+0000","last_change":"2024-06-29T14:23:15.203329+0000","last_active":"2024-06-29T14:23:16.194138+0000","last_peered":"2024-06-29T14:23:16.194138+0000","last_clean":"2024-06-29T14:23:16.194138+0000","last_became_active":"2024-06-29T14:23:15.202429+0000","last_became_peered":"2024-06-29T14:23:15.202429+0000","last_unstale":"2024-06-29T14:23:16.194138+0000","last_undegraded":"2024-06-29T14:23:16.194138+0000","last_fullsized":"2024-06-29T14:23:16.194138+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.198488+0000","last_change":"2024-06-29T14:23:16.190685+0000","last_active":"2024-06-29T14:23:16.198488+0000","last_peered":"2024-06-29T14:23:16.198488+0000","last_clean":"2024-06-29T14:23:16.198488+0000","last_became_active":"2024-06-29T14:23:16.190310+0000","last_became_peered":"2024-06-29T14:23:16.190310+0000","last_unstale":"2024-06-29T14:23:16.198488+0000","last_undegraded":"2024-06-29T14:23:16.198488+0000","last_fullsized":"2024-06-29T14:23:16.198488+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:08.341184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202801+0000","last_change":"2024-06-29T14:23:15.200968+0000","last_active":"2024-06-29T14:23:15.202801+0000","last_peered":"2024-06-29T14:23:15.202801+0000","last_clean":"2024-06-29T14:23:15.202801+0000","last_became_active":"2024-06-29T14:23:15.200573+0000","last_became_peered":"2024-06-29T14:23:15.200573+0000","last_unstale":"2024-06-29T14:23:15.202801+0000","last_undegraded":"2024-06-29T14:23:15.202801+0000","last_fullsized":"2024-06-29T14:23:15.202801+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:34.092009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.193645+0000","last_change":"2024-06-29T14:23:15.193645+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.193645+0000","last_undegraded":"2024-06-29T14:23:15.193645+0000","last_fullsized":"2024-06-29T14:23:15.193645+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.201435+0000","last_change":"2024-06-29T14:23:15.197575+0000","last_active":"2024-06-29T14:23:15.201435+0000","last_peered":"2024-06-29T14:23:15.201435+0000","last_clean":"2024-06-29T14:23:15.201435+0000","last_became_active":"2024-06-29T14:23:15.197267+0000","last_became_peered":"2024-06-29T14:23:15.197267+0000","last_unstale":"2024-06-29T14:23:15.201435+0000","last_undegraded":"2024-06-29T14:23:15.201435+0000","last_fullsized":"2024-06-29T14:23:15.201435+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:34.092009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.187502+0000","last_change":"2024-06-29T14:23:15.187502+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.187502+0000","last_undegraded":"2024-06-29T14:23:15.187502+0000","last_fullsized":"2024-06-29T14:23:15.187502+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.181400+0000","last_change":"2024-06-29T14:23:15.192161+0000","last_active":"2024-06-29T14:23:16.181400+0000","last_peered":"2024-06-29T14:23:16.181400+0000","last_clean":"2024-06-29T14:23:16.181400+0000","last_became_active":"2024-06-29T14:23:15.191810+0000","last_became_peered":"2024-06-29T14:23:15.191810+0000","last_unstale":"2024-06-29T14:23:16.181400+0000","last_undegraded":"2024-06-29T14:23:16.181400+0000","last_fullsized":"2024-06-29T14:23:16.181400+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:28.995567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":39,"state":"creating+peering","last_fresh":"2024-06-29T14:23:15.197944+0000","last_change":"2024-06-29T14:23:15.197944+0000","last_active":"2024-06-29T14:23:15.170617+0000","last_peered":"2024-06-29T14:23:15.170617+0000","last_clean":"2024-06-29T14:23:15.170617+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:23:15.197944+0000","last_undegraded":"2024-06-29T14:23:15.197944+0000","last_fullsized":"2024-06-29T14:23:15.197944+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.202122+0000","last_change":"2024-06-29T14:23:15.198041+0000","last_active":"2024-06-29T14:23:15.202122+0000","last_peered":"2024-06-29T14:23:15.202122+0000","last_clean":"2024-06-29T14:23:15.202122+0000","last_became_active":"2024-06-29T14:23:15.197662+0000","last_became_peered":"2024-06-29T14:23:15.197662+0000","last_unstale":"2024-06-29T14:23:15.202122+0000","last_undegraded":"2024-06-29T14:23:15.202122+0000","last_fullsized":"2024-06-29T14:23:15.202122+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:48.024521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":142,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-29T14:23:15.190774+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:23:15.190774+0000","last_peered":"2024-06-29T14:23:15.190774+0000","last_clean":"2024-06-29T14:23:15.190774+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:23:15.190774+0000","last_undegraded":"2024-06-29T14:23:15.190774+0000","last_fullsized":"2024-06-29T14:23:15.190774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1716,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":197889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509462,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":196333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":779444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542166,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":779444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":182409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":203005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:23:21.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2024-06-29T14:23:21.037 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.192292+0000', 'last_change': '2024-06-29T14:23:15.192292+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.192292+0000', 'last_undegraded': '2024-06-29T14:23:15.192292+0000', 'last_fullsized': '2024-06-29T14:23:15.192292+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.037 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.194916+0000', 'last_change': '2024-06-29T14:23:15.194916+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.194916+0000', 'last_undegraded': '2024-06-29T14:23:15.194916+0000', 'last_fullsized': '2024-06-29T14:23:15.194916+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.193674+0000', 'last_change': '2024-06-29T14:23:15.193674+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.193674+0000', 'last_undegraded': '2024-06-29T14:23:15.193674+0000', 'last_fullsized': '2024-06-29T14:23:15.193674+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.196595+0000', 'last_change': '2024-06-29T14:23:15.196595+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.196595+0000', 'last_undegraded': '2024-06-29T14:23:15.196595+0000', 'last_fullsized': '2024-06-29T14:23:15.196595+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.194142+0000', 'last_change': '2024-06-29T14:23:15.194142+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.194142+0000', 'last_undegraded': '2024-06-29T14:23:15.194142+0000', 'last_fullsized': '2024-06-29T14:23:15.194142+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.191760+0000', 'last_change': '2024-06-29T14:23:15.191760+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.191760+0000', 'last_undegraded': '2024-06-29T14:23:15.191760+0000', 'last_fullsized': '2024-06-29T14:23:15.191760+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-06-29T14:23:21.038 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.189097+0000', 'last_change': '2024-06-29T14:23:15.189097+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.189097+0000', 'last_undegraded': '2024-06-29T14:23:15.189097+0000', 'last_fullsized': '2024-06-29T14:23:15.189097+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.188330+0000', 'last_change': '2024-06-29T14:23:15.188330+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.188330+0000', 'last_undegraded': '2024-06-29T14:23:15.188330+0000', 'last_fullsized': '2024-06-29T14:23:15.188330+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.184396+0000', 'last_change': '2024-06-29T14:23:15.184396+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.184396+0000', 'last_undegraded': '2024-06-29T14:23:15.184396+0000', 'last_fullsized': '2024-06-29T14:23:15.184396+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.184382+0000', 'last_change': '2024-06-29T14:23:15.184382+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.184382+0000', 'last_undegraded': '2024-06-29T14:23:15.184382+0000', 'last_fullsized': '2024-06-29T14:23:15.184382+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.190853+0000', 'last_change': '2024-06-29T14:23:15.190853+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.190853+0000', 'last_undegraded': '2024-06-29T14:23:15.190853+0000', 'last_fullsized': '2024-06-29T14:23:15.190853+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2024-06-29T14:23:21.039 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.193127+0000', 'last_change': '2024-06-29T14:23:15.193127+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.193127+0000', 'last_undegraded': '2024-06-29T14:23:15.193127+0000', 'last_fullsized': '2024-06-29T14:23:15.193127+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.188344+0000', 'last_change': '2024-06-29T14:23:15.188344+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.188344+0000', 'last_undegraded': '2024-06-29T14:23:15.188344+0000', 'last_fullsized': '2024-06-29T14:23:15.188344+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.185688+0000', 'last_change': '2024-06-29T14:23:15.185688+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.185688+0000', 'last_undegraded': '2024-06-29T14:23:15.185688+0000', 'last_fullsized': '2024-06-29T14:23:15.185688+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.185734+0000', 'last_change': '2024-06-29T14:23:15.185734+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.185734+0000', 'last_undegraded': '2024-06-29T14:23:15.185734+0000', 'last_fullsized': '2024-06-29T14:23:15.185734+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.192802+0000', 'last_change': '2024-06-29T14:23:15.192802+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.192802+0000', 'last_undegraded': '2024-06-29T14:23:15.192802+0000', 'last_fullsized': '2024-06-29T14:23:15.192802+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.186290+0000', 'last_change': '2024-06-29T14:23:15.186290+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.186290+0000', 'last_undegraded': '2024-06-29T14:23:15.186290+0000', 'last_fullsized': '2024-06-29T14:23:15.186290+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:23:21.040 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.185611+0000', 'last_change': '2024-06-29T14:23:15.185611+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.185611+0000', 'last_undegraded': '2024-06-29T14:23:15.185611+0000', 'last_fullsized': '2024-06-29T14:23:15.185611+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.187368+0000', 'last_change': '2024-06-29T14:23:15.187368+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.187368+0000', 'last_undegraded': '2024-06-29T14:23:15.187368+0000', 'last_fullsized': '2024-06-29T14:23:15.187368+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.188473+0000', 'last_change': '2024-06-29T14:23:15.188473+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.188473+0000', 'last_undegraded': '2024-06-29T14:23:15.188473+0000', 'last_fullsized': '2024-06-29T14:23:15.188473+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.189311+0000', 'last_change': '2024-06-29T14:23:15.189311+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.189311+0000', 'last_undegraded': '2024-06-29T14:23:15.189311+0000', 'last_fullsized': '2024-06-29T14:23:15.189311+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.191269+0000', 'last_change': '2024-06-29T14:23:15.191269+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.191269+0000', 'last_undegraded': '2024-06-29T14:23:15.191269+0000', 'last_fullsized': '2024-06-29T14:23:15.191269+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-06-29T14:23:21.041 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.191458+0000', 'last_change': '2024-06-29T14:23:15.191458+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.191458+0000', 'last_undegraded': '2024-06-29T14:23:15.191458+0000', 'last_fullsized': '2024-06-29T14:23:15.191458+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.190068+0000', 'last_change': '2024-06-29T14:23:15.190068+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.190068+0000', 'last_undegraded': '2024-06-29T14:23:15.190068+0000', 'last_fullsized': '2024-06-29T14:23:15.190068+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.191167+0000', 'last_change': '2024-06-29T14:23:15.191167+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.191167+0000', 'last_undegraded': '2024-06-29T14:23:15.191167+0000', 'last_fullsized': '2024-06-29T14:23:15.191167+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.187426+0000', 'last_change': '2024-06-29T14:23:15.187426+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.187426+0000', 'last_undegraded': '2024-06-29T14:23:15.187426+0000', 'last_fullsized': '2024-06-29T14:23:15.187426+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.193179+0000', 'last_change': '2024-06-29T14:23:15.193179+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.193179+0000', 'last_undegraded': '2024-06-29T14:23:15.193179+0000', 'last_fullsized': '2024-06-29T14:23:15.193179+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.192686+0000', 'last_change': '2024-06-29T14:23:15.192686+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.192686+0000', 'last_undegraded': '2024-06-29T14:23:15.192686+0000', 'last_fullsized': '2024-06-29T14:23:15.192686+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:23:21.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.187906+0000', 'last_change': '2024-06-29T14:23:15.187906+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.187906+0000', 'last_undegraded': '2024-06-29T14:23:15.187906+0000', 'last_fullsized': '2024-06-29T14:23:15.187906+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.193709+0000', 'last_change': '2024-06-29T14:23:15.193709+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.193709+0000', 'last_undegraded': '2024-06-29T14:23:15.193709+0000', 'last_fullsized': '2024-06-29T14:23:15.193709+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.189204+0000', 'last_change': '2024-06-29T14:23:15.189204+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.189204+0000', 'last_undegraded': '2024-06-29T14:23:15.189204+0000', 'last_fullsized': '2024-06-29T14:23:15.189204+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.193645+0000', 'last_change': '2024-06-29T14:23:15.193645+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.193645+0000', 'last_undegraded': '2024-06-29T14:23:15.193645+0000', 'last_fullsized': '2024-06-29T14:23:15.193645+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.187502+0000', 'last_change': '2024-06-29T14:23:15.187502+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.187502+0000', 'last_undegraded': '2024-06-29T14:23:15.187502+0000', 'last_fullsized': '2024-06-29T14:23:15.187502+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-06-29T14:23:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 39, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:23:15.197944+0000', 'last_change': '2024-06-29T14:23:15.197944+0000', 'last_active': '2024-06-29T14:23:15.170617+0000', 'last_peered': '2024-06-29T14:23:15.170617+0000', 'last_clean': '2024-06-29T14:23:15.170617+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:23:15.197944+0000', 'last_undegraded': '2024-06-29T14:23:15.197944+0000', 'last_fullsized': '2024-06-29T14:23:15.197944+0000', 'mapping_epoch': 39, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 39, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'last_clean_scrub_stamp': '2024-06-29T14:23:15.170617+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:23:21.044 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-29T14:23:21.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.193+0000 7f32b45ee640 1 -- 172.21.15.59:0/1687565765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac15ceb0 msgr2=0x7f32ac0b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.193+0000 7f32b45ee640 1 --2- 172.21.15.59:0/1687565765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac15ceb0 0x7f32ac0b56d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0b5600 tx=0x7f32a802a560 comp rx=0 tx=0).stop 2024-06-29T14:23:21.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.193+0000 7f32b45ee640 1 -- 172.21.15.59:0/1687565765 shutdown_connections 2024-06-29T14:23:21.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.193+0000 7f32b45ee640 1 --2- 172.21.15.59:0/1687565765 >> v2:172.21.15.153:3300/0 conn(0x7f32ac141bd0 0x7f32ac13edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.193+0000 7f32b45ee640 1 --2- 172.21.15.59:0/1687565765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac15ceb0 0x7f32ac0b56d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.193+0000 7f32b45ee640 1 -- 172.21.15.59:0/1687565765 >> 172.21.15.59:0/1687565765 conn(0x7f32ac0b6fb0 msgr2=0x7f32ac0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.193+0000 7f32b45ee640 1 -- 172.21.15.59:0/1687565765 shutdown_connections 2024-06-29T14:23:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.193+0000 7f32b45ee640 1 -- 172.21.15.59:0/1687565765 wait complete. 2024-06-29T14:23:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.195+0000 7f32b45ee640 1 Processor -- start 2024-06-29T14:23:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.195+0000 7f32b45ee640 1 -- start start 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b45ee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac13e9b0 0x7f32ac13acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b45ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac15ceb0 0x7f32ac151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b45ee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32ac155960 0x7f32ac157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b45ee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac0b29d0 con 0x7f32ac13e9b0 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b45ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac0b2850 con 0x7f32ac15ceb0 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b45ee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f32ac0b2b50 con 0x7f32ac155960 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b238a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac13e9b0 0x7f32ac13acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b238a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac13e9b0 0x7f32ac13acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48214/0 (socket says 172.21.15.59:48214) 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b238a640 1 -- 172.21.15.59:0/2741207445 learned_addr learned my addr 172.21.15.59:0/2741207445 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b1b89640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac15ceb0 0x7f32ac151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b2b8b640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32ac155960 0x7f32ac157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b238a640 1 -- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32ac155960 msgr2=0x7f32ac157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b238a640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32ac155960 0x7f32ac157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b238a640 1 -- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac15ceb0 msgr2=0x7f32ac151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b238a640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac15ceb0 0x7f32ac151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b238a640 1 -- 172.21.15.59:0/2741207445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ac151570 con 0x7f32ac13e9b0 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b2b8b640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32ac155960 0x7f32ac157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.196+0000 7f32b1b89640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac15ceb0 0x7f32ac151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.197+0000 7f32b238a640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac13e9b0 0x7f32ac13acd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f32a80099c0 tx=0x7f32a80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.197+0000 7f32a2ffd640 1 -- 172.21.15.59:0/2741207445 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a803e030 con 0x7f32ac13e9b0 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.197+0000 7f32a2ffd640 1 -- 172.21.15.59:0/2741207445 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a8003db0 con 0x7f32ac13e9b0 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.197+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ac13b520 con 0x7f32ac13e9b0 2024-06-29T14:23:21.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.197+0000 7f32a2ffd640 1 -- 172.21.15.59:0/2741207445 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8035040 con 0x7f32ac13e9b0 2024-06-29T14:23:21.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.197+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ac13bae0 con 0x7f32ac13e9b0 2024-06-29T14:23:21.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.198+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ac0b3000 con 0x7f32ac13e9b0 2024-06-29T14:23:21.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.199+0000 7f32a2ffd640 1 -- 172.21.15.59:0/2741207445 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f32a8041030 con 0x7f32ac13e9b0 2024-06-29T14:23:21.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.199+0000 7f32a2ffd640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32900762d0 0x7f32900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.199+0000 7f32a2ffd640 1 -- 172.21.15.59:0/2741207445 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f32a80b8330 con 0x7f32ac13e9b0 2024-06-29T14:23:21.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.200+0000 7f32b1b89640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32900762d0 0x7f32900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.200+0000 7f32b1b89640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32900762d0 0x7f32900787e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f32ac155ff0 tx=0x7f329c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:21.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.204+0000 7f32a2ffd640 1 -- 172.21.15.59:0/2741207445 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f32a8081a70 con 0x7f32ac13e9b0 2024-06-29T14:23:21.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.470+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f32ac0b5940 con 0x7f32ac13e9b0 2024-06-29T14:23:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.471+0000 7f32a2ffd640 1 -- 172.21.15.59:0/2741207445 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1140 (secure 0 0 0) 0x7f32a8081370 con 0x7f32ac13e9b0 2024-06-29T14:23:21.469 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:21.469 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":112,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":8,"num_up_osds":8,"osd_up_since":1719670901,"num_in_osds":8,"osd_in_since":1719670892,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":63},{"state_name":"creating+peering","count":34}],"num_pgs":97,"num_pools":3,"num_objects":12,"data_bytes":592084,"bytes_used":226111488,"bytes_avail":772868001792,"bytes_total":773094113280,"inactive_pgs_ratio":0.3505154550075531,"write_bytes_sec":2018,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":19,"btime":"2024-06-29T14:23:19:215604+0000","id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5469}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-06-29T14:22:57.748294+0000","services":{}},"progress_events":{}} 2024-06-29T14:23:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32900762d0 msgr2=0x7f32900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32900762d0 0x7f32900787e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f32ac155ff0 tx=0x7f329c009040 comp rx=0 tx=0).stop 2024-06-29T14:23:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac13e9b0 msgr2=0x7f32ac13acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac13e9b0 0x7f32ac13acd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f32a80099c0 tx=0x7f32a80038a0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 shutdown_connections 2024-06-29T14:23:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f32900762d0 0x7f32900787e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32ac155960 0x7f32ac157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32ac15ceb0 0x7f32ac151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 --2- 172.21.15.59:0/2741207445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ac13e9b0 0x7f32ac13acd0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 >> 172.21.15.59:0/2741207445 conn(0x7f32ac0b6fb0 msgr2=0x7f32ac0cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 shutdown_connections 2024-06-29T14:23:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.473+0000 7f32b45ee640 1 -- 172.21.15.59:0/2741207445 wait complete. 2024-06-29T14:23:21.482 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-06-29T14:23:21.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:23:21.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.601+0000 7f31eef54640 1 -- 172.21.15.59:0/939754251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80cf5c0 msgr2=0x7f31e80d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.601+0000 7f31eef54640 1 --2- 172.21.15.59:0/939754251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80cf5c0 0x7f31e80d1a20 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009830 tx=0x7f31d802a4d0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.601+0000 7f31eef54640 1 -- 172.21.15.59:0/939754251 shutdown_connections 2024-06-29T14:23:21.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.601+0000 7f31eef54640 1 --2- 172.21.15.59:0/939754251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80cf5c0 0x7f31e80d1a20 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.601+0000 7f31eef54640 1 --2- 172.21.15.59:0/939754251 >> v2:172.21.15.153:3300/0 conn(0x7f31e808c190 0x7f31e80cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.601+0000 7f31eef54640 1 -- 172.21.15.59:0/939754251 >> 172.21.15.59:0/939754251 conn(0x7f31e80b5610 msgr2=0x7f31e80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.601+0000 7f31eef54640 1 -- 172.21.15.59:0/939754251 shutdown_connections 2024-06-29T14:23:21.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.601+0000 7f31eef54640 1 -- 172.21.15.59:0/939754251 wait complete. 2024-06-29T14:23:21.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.603+0000 7f31eef54640 1 Processor -- start 2024-06-29T14:23:21.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.603+0000 7f31eef54640 1 -- start start 2024-06-29T14:23:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.604+0000 7f31eef54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31e808c190 0x7f31e80ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.604+0000 7f31eef54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80ce950 0x7f31e8151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.604+0000 7f31eef54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31e80cf5c0 0x7f31e81f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.604+0000 7f31eef54640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f31e80b7f60 con 0x7f31e80ce950 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.604+0000 7f31eef54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31e80b7de0 con 0x7f31e808c190 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.604+0000 7f31eef54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f31e80b80e0 con 0x7f31e80cf5c0 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80ce950 0x7f31e8151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80ce950 0x7f31e8151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48226/0 (socket says 172.21.15.59:48226) 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 -- 172.21.15.59:0/3839996995 learned_addr learned my addr 172.21.15.59:0/3839996995 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31eccf0640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31e808c190 0x7f31e80ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31ed4f1640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31e80cf5c0 0x7f31e81f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 -- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31e80cf5c0 msgr2=0x7f31e81f8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31e80cf5c0 0x7f31e81f8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 -- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31e808c190 msgr2=0x7f31e80ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31e808c190 0x7f31e80ab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 -- 172.21.15.59:0/3839996995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e80aa590 con 0x7f31e80ce950 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31ed4f1640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31e80cf5c0 0x7f31e81f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e7fff640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80ce950 0x7f31e8151f80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f31d802aac0 tx=0x7f31d8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e57fa640 1 -- 172.21.15.59:0/3839996995 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8031d60 con 0x7f31e80ce950 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.605+0000 7f31e57fa640 1 -- 172.21.15.59:0/3839996995 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31d8033e10 con 0x7f31e80ce950 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.606+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e80aa800 con 0x7f31e80ce950 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.606+0000 7f31e57fa640 1 -- 172.21.15.59:0/3839996995 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8005260 con 0x7f31e80ce950 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.606+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e80aadc0 con 0x7f31e80ce950 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.607+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e81f9860 con 0x7f31e80ce950 2024-06-29T14:23:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.607+0000 7f31e57fa640 1 -- 172.21.15.59:0/3839996995 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f31d8005400 con 0x7f31e80ce950 2024-06-29T14:23:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.608+0000 7f31e57fa640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f31c80762d0 0x7f31c80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.608+0000 7f31e57fa640 1 -- 172.21.15.59:0/3839996995 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f31d80b89a0 con 0x7f31e80ce950 2024-06-29T14:23:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.608+0000 7f31eccf0640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f31c80762d0 0x7f31c80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.609+0000 7f31eccf0640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f31c80762d0 0x7f31c80787e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f31d0005fd0 tx=0x7f31d000b2e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:21.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.613+0000 7f31e57fa640 1 -- 172.21.15.59:0/3839996995 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f31d803f030 con 0x7f31e80ce950 2024-06-29T14:23:21.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.804+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f31e80b4be0 con 0x7f31c80762d0 2024-06-29T14:23:21.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.809+0000 7f31e57fa640 1 -- 172.21.15.59:0/3839996995 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251922 (secure 0 0 0) 0x7f31e80b4be0 con 0x7f31c80762d0 2024-06-29T14:23:21.809 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:21.810 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:23:21.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f31c80762d0 msgr2=0x7f31c80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f31c80762d0 0x7f31c80787e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f31d0005fd0 tx=0x7f31d000b2e0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80ce950 msgr2=0x7f31e8151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80ce950 0x7f31e8151f80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f31d802aac0 tx=0x7f31d8033c20 comp rx=0 tx=0).stop 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 shutdown_connections 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f31c80762d0 0x7f31c80787e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31e80cf5c0 0x7f31e81f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31e80ce950 0x7f31e8151f80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 --2- 172.21.15.59:0/3839996995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31e808c190 0x7f31e80ab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 >> 172.21.15.59:0/3839996995 conn(0x7f31e80b5610 msgr2=0x7f31e815e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 shutdown_connections 2024-06-29T14:23:21.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:21.812+0000 7f31eef54640 1 -- 172.21.15.59:0/3839996995 wait complete. 2024-06-29T14:23:21.821 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-06-29T14:23:21.753014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221000,"kb_used_data":4820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753720,"statfs":{"total":773094113280,"available":772867809280,"internally_reserved":0,"allocated":4935680,"data_stored":2781759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.570392"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.211606+0000","last_change":"2024-06-29T14:23:16.201160+0000","last_active":"2024-06-29T14:23:16.211606+0000","last_peered":"2024-06-29T14:23:16.211606+0000","last_clean":"2024-06-29T14:23:16.211606+0000","last_became_active":"2024-06-29T14:23:16.200155+0000","last_became_peered":"2024-06-29T14:23:16.200155+0000","last_unstale":"2024-06-29T14:23:16.211606+0000","last_undegraded":"2024-06-29T14:23:16.211606+0000","last_fullsized":"2024-06-29T14:23:16.211606+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:23.684550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.199308+0000","last_change":"2024-06-29T14:23:16.189930+0000","last_active":"2024-06-29T14:23:16.199308+0000","last_peered":"2024-06-29T14:23:16.199308+0000","last_clean":"2024-06-29T14:23:16.199308+0000","last_became_active":"2024-06-29T14:23:16.189571+0000","last_became_peered":"2024-06-29T14:23:16.189571+0000","last_unstale":"2024-06-29T14:23:16.199308+0000","last_undegraded":"2024-06-29T14:23:16.199308+0000","last_fullsized":"2024-06-29T14:23:16.199308+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:12.662244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207289+0000","last_change":"2024-06-29T14:23:16.203182+0000","last_active":"2024-06-29T14:23:16.207289+0000","last_peered":"2024-06-29T14:23:16.207289+0000","last_clean":"2024-06-29T14:23:16.207289+0000","last_became_active":"2024-06-29T14:23:16.202851+0000","last_became_peered":"2024-06-29T14:23:16.202851+0000","last_unstale":"2024-06-29T14:23:16.207289+0000","last_undegraded":"2024-06-29T14:23:16.207289+0000","last_fullsized":"2024-06-29T14:23:16.207289+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:13.507757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972956+0000","last_change":"2024-06-29T14:23:16.202834+0000","last_active":"2024-06-29T14:23:17.972956+0000","last_peered":"2024-06-29T14:23:17.972956+0000","last_clean":"2024-06-29T14:23:17.972956+0000","last_became_active":"2024-06-29T14:23:16.202357+0000","last_became_peered":"2024-06-29T14:23:16.202357+0000","last_unstale":"2024-06-29T14:23:17.972956+0000","last_undegraded":"2024-06-29T14:23:17.972956+0000","last_fullsized":"2024-06-29T14:23:17.972956+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:35.097600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972601+0000","last_change":"2024-06-29T14:23:16.194759+0000","last_active":"2024-06-29T14:23:17.972601+0000","last_peered":"2024-06-29T14:23:17.972601+0000","last_clean":"2024-06-29T14:23:17.972601+0000","last_became_active":"2024-06-29T14:23:16.194351+0000","last_became_peered":"2024-06-29T14:23:16.194351+0000","last_unstale":"2024-06-29T14:23:17.972601+0000","last_undegraded":"2024-06-29T14:23:17.972601+0000","last_fullsized":"2024-06-29T14:23:17.972601+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:30.838175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207114+0000","last_change":"2024-06-29T14:23:16.195834+0000","last_active":"2024-06-29T14:23:16.207114+0000","last_peered":"2024-06-29T14:23:16.207114+0000","last_clean":"2024-06-29T14:23:16.207114+0000","last_became_active":"2024-06-29T14:23:16.195476+0000","last_became_peered":"2024-06-29T14:23:16.195476+0000","last_unstale":"2024-06-29T14:23:16.207114+0000","last_undegraded":"2024-06-29T14:23:16.207114+0000","last_fullsized":"2024-06-29T14:23:16.207114+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:30.001157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971518+0000","last_change":"2024-06-29T14:23:16.205371+0000","last_active":"2024-06-29T14:23:17.971518+0000","last_peered":"2024-06-29T14:23:17.971518+0000","last_clean":"2024-06-29T14:23:17.971518+0000","last_became_active":"2024-06-29T14:23:16.204429+0000","last_became_peered":"2024-06-29T14:23:16.204429+0000","last_unstale":"2024-06-29T14:23:17.971518+0000","last_undegraded":"2024-06-29T14:23:17.971518+0000","last_fullsized":"2024-06-29T14:23:17.971518+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:14:42.425008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.206121+0000","last_change":"2024-06-29T14:23:16.199825+0000","last_active":"2024-06-29T14:23:16.206121+0000","last_peered":"2024-06-29T14:23:16.206121+0000","last_clean":"2024-06-29T14:23:16.206121+0000","last_became_active":"2024-06-29T14:23:16.199499+0000","last_became_peered":"2024-06-29T14:23:16.199499+0000","last_unstale":"2024-06-29T14:23:16.206121+0000","last_undegraded":"2024-06-29T14:23:16.206121+0000","last_fullsized":"2024-06-29T14:23:16.206121+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:10.882976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214762+0000","last_change":"2024-06-29T14:23:16.209692+0000","last_active":"2024-06-29T14:23:16.214762+0000","last_peered":"2024-06-29T14:23:16.214762+0000","last_clean":"2024-06-29T14:23:16.214762+0000","last_became_active":"2024-06-29T14:23:16.202968+0000","last_became_peered":"2024-06-29T14:23:16.202968+0000","last_unstale":"2024-06-29T14:23:16.214762+0000","last_undegraded":"2024-06-29T14:23:16.214762+0000","last_fullsized":"2024-06-29T14:23:16.214762+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:10.402585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972219+0000","last_change":"2024-06-29T14:23:16.202448+0000","last_active":"2024-06-29T14:23:17.972219+0000","last_peered":"2024-06-29T14:23:17.972219+0000","last_clean":"2024-06-29T14:23:17.972219+0000","last_became_active":"2024-06-29T14:23:16.201952+0000","last_became_peered":"2024-06-29T14:23:16.201952+0000","last_unstale":"2024-06-29T14:23:17.972219+0000","last_undegraded":"2024-06-29T14:23:17.972219+0000","last_fullsized":"2024-06-29T14:23:17.972219+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:51.801025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972537+0000","last_change":"2024-06-29T14:23:16.204313+0000","last_active":"2024-06-29T14:23:17.972537+0000","last_peered":"2024-06-29T14:23:17.972537+0000","last_clean":"2024-06-29T14:23:17.972537+0000","last_became_active":"2024-06-29T14:23:16.203981+0000","last_became_peered":"2024-06-29T14:23:16.203981+0000","last_unstale":"2024-06-29T14:23:17.972537+0000","last_undegraded":"2024-06-29T14:23:17.972537+0000","last_fullsized":"2024-06-29T14:23:17.972537+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:46.690692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214775+0000","last_change":"2024-06-29T14:23:16.209122+0000","last_active":"2024-06-29T14:23:16.214775+0000","last_peered":"2024-06-29T14:23:16.214775+0000","last_clean":"2024-06-29T14:23:16.214775+0000","last_became_active":"2024-06-29T14:23:16.208783+0000","last_became_peered":"2024-06-29T14:23:16.208783+0000","last_unstale":"2024-06-29T14:23:16.214775+0000","last_undegraded":"2024-06-29T14:23:16.214775+0000","last_fullsized":"2024-06-29T14:23:16.214775+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:30.001157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971838+0000","last_change":"2024-06-29T14:23:16.206491+0000","last_active":"2024-06-29T14:23:17.971838+0000","last_peered":"2024-06-29T14:23:17.971838+0000","last_clean":"2024-06-29T14:23:17.971838+0000","last_became_active":"2024-06-29T14:23:16.205423+0000","last_became_peered":"2024-06-29T14:23:16.205423+0000","last_unstale":"2024-06-29T14:23:17.971838+0000","last_undegraded":"2024-06-29T14:23:17.971838+0000","last_fullsized":"2024-06-29T14:23:17.971838+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:28.205354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971455+0000","last_change":"2024-06-29T14:23:16.199501+0000","last_active":"2024-06-29T14:23:17.971455+0000","last_peered":"2024-06-29T14:23:17.971455+0000","last_clean":"2024-06-29T14:23:17.971455+0000","last_became_active":"2024-06-29T14:23:16.199071+0000","last_became_peered":"2024-06-29T14:23:16.199071+0000","last_unstale":"2024-06-29T14:23:17.971455+0000","last_undegraded":"2024-06-29T14:23:17.971455+0000","last_fullsized":"2024-06-29T14:23:17.971455+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:39.380846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971720+0000","last_change":"2024-06-29T14:23:16.205418+0000","last_active":"2024-06-29T14:23:17.971720+0000","last_peered":"2024-06-29T14:23:17.971720+0000","last_clean":"2024-06-29T14:23:17.971720+0000","last_became_active":"2024-06-29T14:23:16.205110+0000","last_became_peered":"2024-06-29T14:23:16.205110+0000","last_unstale":"2024-06-29T14:23:17.971720+0000","last_undegraded":"2024-06-29T14:23:17.971720+0000","last_fullsized":"2024-06-29T14:23:17.971720+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:21.735913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192094+0000","last_change":"2024-06-29T14:23:16.206412+0000","last_active":"2024-06-29T14:23:17.192094+0000","last_peered":"2024-06-29T14:23:17.192094+0000","last_clean":"2024-06-29T14:23:17.192094+0000","last_became_active":"2024-06-29T14:23:16.205447+0000","last_became_peered":"2024-06-29T14:23:16.205447+0000","last_unstale":"2024-06-29T14:23:17.192094+0000","last_undegraded":"2024-06-29T14:23:17.192094+0000","last_fullsized":"2024-06-29T14:23:17.192094+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:38:40.833526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192670+0000","last_change":"2024-06-29T14:23:16.205425+0000","last_active":"2024-06-29T14:23:17.192670+0000","last_peered":"2024-06-29T14:23:17.192670+0000","last_clean":"2024-06-29T14:23:17.192670+0000","last_became_active":"2024-06-29T14:23:16.205061+0000","last_became_peered":"2024-06-29T14:23:16.205061+0000","last_unstale":"2024-06-29T14:23:17.192670+0000","last_undegraded":"2024-06-29T14:23:17.192670+0000","last_fullsized":"2024-06-29T14:23:17.192670+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:18.957684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192585+0000","last_change":"2024-06-29T14:23:16.197724+0000","last_active":"2024-06-29T14:23:17.192585+0000","last_peered":"2024-06-29T14:23:17.192585+0000","last_clean":"2024-06-29T14:23:17.192585+0000","last_became_active":"2024-06-29T14:23:16.197347+0000","last_became_peered":"2024-06-29T14:23:16.197347+0000","last_unstale":"2024-06-29T14:23:17.192585+0000","last_undegraded":"2024-06-29T14:23:17.192585+0000","last_fullsized":"2024-06-29T14:23:17.192585+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:35.902597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972388+0000","last_change":"2024-06-29T14:23:16.190111+0000","last_active":"2024-06-29T14:23:17.972388+0000","last_peered":"2024-06-29T14:23:17.972388+0000","last_clean":"2024-06-29T14:23:17.972388+0000","last_became_active":"2024-06-29T14:23:16.189786+0000","last_became_peered":"2024-06-29T14:23:16.189786+0000","last_unstale":"2024-06-29T14:23:17.972388+0000","last_undegraded":"2024-06-29T14:23:17.972388+0000","last_fullsized":"2024-06-29T14:23:17.972388+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:12.500255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192271+0000","last_change":"2024-06-29T14:23:16.196160+0000","last_active":"2024-06-29T14:23:17.192271+0000","last_peered":"2024-06-29T14:23:17.192271+0000","last_clean":"2024-06-29T14:23:17.192271+0000","last_became_active":"2024-06-29T14:23:16.195798+0000","last_became_peered":"2024-06-29T14:23:16.195798+0000","last_unstale":"2024-06-29T14:23:17.192271+0000","last_undegraded":"2024-06-29T14:23:17.192271+0000","last_fullsized":"2024-06-29T14:23:17.192271+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:33.154800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971957+0000","last_change":"2024-06-29T14:23:16.209041+0000","last_active":"2024-06-29T14:23:17.971957+0000","last_peered":"2024-06-29T14:23:17.971957+0000","last_clean":"2024-06-29T14:23:17.971957+0000","last_became_active":"2024-06-29T14:23:16.208679+0000","last_became_peered":"2024-06-29T14:23:16.208679+0000","last_unstale":"2024-06-29T14:23:17.971957+0000","last_undegraded":"2024-06-29T14:23:17.971957+0000","last_fullsized":"2024-06-29T14:23:17.971957+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:05:44.443149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971753+0000","last_change":"2024-06-29T14:23:16.203100+0000","last_active":"2024-06-29T14:23:17.971753+0000","last_peered":"2024-06-29T14:23:17.971753+0000","last_clean":"2024-06-29T14:23:17.971753+0000","last_became_active":"2024-06-29T14:23:16.201831+0000","last_became_peered":"2024-06-29T14:23:16.201831+0000","last_unstale":"2024-06-29T14:23:17.971753+0000","last_undegraded":"2024-06-29T14:23:17.971753+0000","last_fullsized":"2024-06-29T14:23:17.971753+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:47:33.460264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192279+0000","last_change":"2024-06-29T14:23:16.211154+0000","last_active":"2024-06-29T14:23:17.192279+0000","last_peered":"2024-06-29T14:23:17.192279+0000","last_clean":"2024-06-29T14:23:17.192279+0000","last_became_active":"2024-06-29T14:23:16.210853+0000","last_became_peered":"2024-06-29T14:23:16.210853+0000","last_unstale":"2024-06-29T14:23:17.192279+0000","last_undegraded":"2024-06-29T14:23:17.192279+0000","last_fullsized":"2024-06-29T14:23:17.192279+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:38.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972485+0000","last_change":"2024-06-29T14:23:16.210617+0000","last_active":"2024-06-29T14:23:17.972485+0000","last_peered":"2024-06-29T14:23:17.972485+0000","last_clean":"2024-06-29T14:23:17.972485+0000","last_became_active":"2024-06-29T14:23:16.210242+0000","last_became_peered":"2024-06-29T14:23:16.210242+0000","last_unstale":"2024-06-29T14:23:17.972485+0000","last_undegraded":"2024-06-29T14:23:17.972485+0000","last_fullsized":"2024-06-29T14:23:17.972485+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:21.067086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192727+0000","last_change":"2024-06-29T14:23:16.195148+0000","last_active":"2024-06-29T14:23:17.192727+0000","last_peered":"2024-06-29T14:23:17.192727+0000","last_clean":"2024-06-29T14:23:17.192727+0000","last_became_active":"2024-06-29T14:23:16.194809+0000","last_became_peered":"2024-06-29T14:23:16.194809+0000","last_unstale":"2024-06-29T14:23:17.192727+0000","last_undegraded":"2024-06-29T14:23:17.192727+0000","last_fullsized":"2024-06-29T14:23:17.192727+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:39.380846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207195+0000","last_change":"2024-06-29T14:23:16.205741+0000","last_active":"2024-06-29T14:23:16.207195+0000","last_peered":"2024-06-29T14:23:16.207195+0000","last_clean":"2024-06-29T14:23:16.207195+0000","last_became_active":"2024-06-29T14:23:16.205351+0000","last_became_peered":"2024-06-29T14:23:16.205351+0000","last_unstale":"2024-06-29T14:23:16.207195+0000","last_undegraded":"2024-06-29T14:23:16.207195+0000","last_fullsized":"2024-06-29T14:23:16.207195+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:10.402585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192298+0000","last_change":"2024-06-29T14:23:16.211247+0000","last_active":"2024-06-29T14:23:17.192298+0000","last_peered":"2024-06-29T14:23:17.192298+0000","last_clean":"2024-06-29T14:23:17.192298+0000","last_became_active":"2024-06-29T14:23:16.210937+0000","last_became_peered":"2024-06-29T14:23:16.210937+0000","last_unstale":"2024-06-29T14:23:17.192298+0000","last_undegraded":"2024-06-29T14:23:17.192298+0000","last_fullsized":"2024-06-29T14:23:17.192298+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:02.929994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972492+0000","last_change":"2024-06-29T14:23:16.200185+0000","last_active":"2024-06-29T14:23:17.972492+0000","last_peered":"2024-06-29T14:23:17.972492+0000","last_clean":"2024-06-29T14:23:17.972492+0000","last_became_active":"2024-06-29T14:23:16.199847+0000","last_became_peered":"2024-06-29T14:23:16.199847+0000","last_unstale":"2024-06-29T14:23:17.972492+0000","last_undegraded":"2024-06-29T14:23:17.972492+0000","last_fullsized":"2024-06-29T14:23:17.972492+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:44:43.833435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191983+0000","last_change":"2024-06-29T14:23:16.189510+0000","last_active":"2024-06-29T14:23:17.191983+0000","last_peered":"2024-06-29T14:23:17.191983+0000","last_clean":"2024-06-29T14:23:17.191983+0000","last_became_active":"2024-06-29T14:23:16.189171+0000","last_became_peered":"2024-06-29T14:23:16.189171+0000","last_unstale":"2024-06-29T14:23:17.191983+0000","last_undegraded":"2024-06-29T14:23:17.191983+0000","last_fullsized":"2024-06-29T14:23:17.191983+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:25.573671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191939+0000","last_change":"2024-06-29T14:23:16.191577+0000","last_active":"2024-06-29T14:23:17.191939+0000","last_peered":"2024-06-29T14:23:17.191939+0000","last_clean":"2024-06-29T14:23:17.191939+0000","last_became_active":"2024-06-29T14:23:16.191245+0000","last_became_peered":"2024-06-29T14:23:16.191245+0000","last_unstale":"2024-06-29T14:23:17.191939+0000","last_undegraded":"2024-06-29T14:23:17.191939+0000","last_fullsized":"2024-06-29T14:23:17.191939+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:29.260203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192483+0000","last_change":"2024-06-29T14:23:16.210335+0000","last_active":"2024-06-29T14:23:17.192483+0000","last_peered":"2024-06-29T14:23:17.192483+0000","last_clean":"2024-06-29T14:23:17.192483+0000","last_became_active":"2024-06-29T14:23:16.210029+0000","last_became_peered":"2024-06-29T14:23:16.210029+0000","last_unstale":"2024-06-29T14:23:17.192483+0000","last_undegraded":"2024-06-29T14:23:17.192483+0000","last_fullsized":"2024-06-29T14:23:17.192483+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:56.733163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207193+0000","last_change":"2024-06-29T14:23:16.204276+0000","last_active":"2024-06-29T14:23:16.207193+0000","last_peered":"2024-06-29T14:23:16.207193+0000","last_clean":"2024-06-29T14:23:16.207193+0000","last_became_active":"2024-06-29T14:23:16.203937+0000","last_became_peered":"2024-06-29T14:23:16.203937+0000","last_unstale":"2024-06-29T14:23:16.207193+0000","last_undegraded":"2024-06-29T14:23:16.207193+0000","last_fullsized":"2024-06-29T14:23:16.207193+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:40.723736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192893+0000","last_change":"2024-06-29T14:23:16.205642+0000","last_active":"2024-06-29T14:23:17.192893+0000","last_peered":"2024-06-29T14:23:17.192893+0000","last_clean":"2024-06-29T14:23:17.192893+0000","last_became_active":"2024-06-29T14:23:16.205040+0000","last_became_peered":"2024-06-29T14:23:16.205040+0000","last_unstale":"2024-06-29T14:23:17.192893+0000","last_undegraded":"2024-06-29T14:23:17.192893+0000","last_fullsized":"2024-06-29T14:23:17.192893+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:43:10.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"40'4","reported_seq":18,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192842+0000","last_change":"2024-06-29T14:23:15.197029+0000","last_active":"2024-06-29T14:23:17.192842+0000","last_peered":"2024-06-29T14:23:17.192842+0000","last_clean":"2024-06-29T14:23:17.192842+0000","last_became_active":"2024-06-29T14:23:15.196636+0000","last_became_peered":"2024-06-29T14:23:15.196636+0000","last_unstale":"2024-06-29T14:23:17.192842+0000","last_undegraded":"2024-06-29T14:23:17.192842+0000","last_fullsized":"2024-06-29T14:23:17.192842+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:15:02.928561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.208350+0000","last_change":"2024-06-29T14:23:16.206697+0000","last_active":"2024-06-29T14:23:16.208350+0000","last_peered":"2024-06-29T14:23:16.208350+0000","last_clean":"2024-06-29T14:23:16.208350+0000","last_became_active":"2024-06-29T14:23:16.206325+0000","last_became_peered":"2024-06-29T14:23:16.206325+0000","last_unstale":"2024-06-29T14:23:16.208350+0000","last_undegraded":"2024-06-29T14:23:16.208350+0000","last_fullsized":"2024-06-29T14:23:16.208350+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"40'1","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217808+0000","last_change":"2024-06-29T14:23:15.204215+0000","last_active":"2024-06-29T14:23:16.217808+0000","last_peered":"2024-06-29T14:23:16.217808+0000","last_clean":"2024-06-29T14:23:16.217808+0000","last_became_active":"2024-06-29T14:23:15.203861+0000","last_became_peered":"2024-06-29T14:23:15.203861+0000","last_unstale":"2024-06-29T14:23:16.217808+0000","last_undegraded":"2024-06-29T14:23:16.217808+0000","last_fullsized":"2024-06-29T14:23:16.217808+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:15.271848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972355+0000","last_change":"2024-06-29T14:23:16.193555+0000","last_active":"2024-06-29T14:23:17.972355+0000","last_peered":"2024-06-29T14:23:17.972355+0000","last_clean":"2024-06-29T14:23:17.972355+0000","last_became_active":"2024-06-29T14:23:16.193197+0000","last_became_peered":"2024-06-29T14:23:16.193197+0000","last_unstale":"2024-06-29T14:23:17.972355+0000","last_undegraded":"2024-06-29T14:23:17.972355+0000","last_fullsized":"2024-06-29T14:23:17.972355+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:09:52.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"40'2","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.219346+0000","last_change":"2024-06-29T14:23:15.205028+0000","last_active":"2024-06-29T14:23:16.219346+0000","last_peered":"2024-06-29T14:23:16.219346+0000","last_clean":"2024-06-29T14:23:16.219346+0000","last_became_active":"2024-06-29T14:23:15.204734+0000","last_became_peered":"2024-06-29T14:23:15.204734+0000","last_unstale":"2024-06-29T14:23:16.219346+0000","last_undegraded":"2024-06-29T14:23:16.219346+0000","last_fullsized":"2024-06-29T14:23:16.219346+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:12.502166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.198933+0000","last_change":"2024-06-29T14:23:16.190648+0000","last_active":"2024-06-29T14:23:16.198933+0000","last_peered":"2024-06-29T14:23:16.198933+0000","last_clean":"2024-06-29T14:23:16.198933+0000","last_became_active":"2024-06-29T14:23:16.190331+0000","last_became_peered":"2024-06-29T14:23:16.190331+0000","last_unstale":"2024-06-29T14:23:16.198933+0000","last_undegraded":"2024-06-29T14:23:16.198933+0000","last_fullsized":"2024-06-29T14:23:16.198933+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:52.309099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972248+0000","last_change":"2024-06-29T14:23:15.195155+0000","last_active":"2024-06-29T14:23:17.972248+0000","last_peered":"2024-06-29T14:23:17.972248+0000","last_clean":"2024-06-29T14:23:17.972248+0000","last_became_active":"2024-06-29T14:23:15.194809+0000","last_became_peered":"2024-06-29T14:23:15.194809+0000","last_unstale":"2024-06-29T14:23:17.972248+0000","last_undegraded":"2024-06-29T14:23:17.972248+0000","last_fullsized":"2024-06-29T14:23:17.972248+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:48.024521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214639+0000","last_change":"2024-06-29T14:23:16.209386+0000","last_active":"2024-06-29T14:23:16.214639+0000","last_peered":"2024-06-29T14:23:16.214639+0000","last_clean":"2024-06-29T14:23:16.214639+0000","last_became_active":"2024-06-29T14:23:16.209011+0000","last_became_peered":"2024-06-29T14:23:16.209011+0000","last_unstale":"2024-06-29T14:23:16.214639+0000","last_undegraded":"2024-06-29T14:23:16.214639+0000","last_fullsized":"2024-06-29T14:23:16.214639+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:40.723736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971977+0000","last_change":"2024-06-29T14:23:15.201776+0000","last_active":"2024-06-29T14:23:17.971977+0000","last_peered":"2024-06-29T14:23:17.971977+0000","last_clean":"2024-06-29T14:23:17.971977+0000","last_became_active":"2024-06-29T14:23:15.201376+0000","last_became_peered":"2024-06-29T14:23:15.201376+0000","last_unstale":"2024-06-29T14:23:17.971977+0000","last_undegraded":"2024-06-29T14:23:17.971977+0000","last_fullsized":"2024-06-29T14:23:17.971977+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:07.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971988+0000","last_change":"2024-06-29T14:23:16.212639+0000","last_active":"2024-06-29T14:23:17.971988+0000","last_peered":"2024-06-29T14:23:17.971988+0000","last_clean":"2024-06-29T14:23:17.971988+0000","last_became_active":"2024-06-29T14:23:16.211460+0000","last_became_peered":"2024-06-29T14:23:16.211460+0000","last_unstale":"2024-06-29T14:23:17.971988+0000","last_undegraded":"2024-06-29T14:23:17.971988+0000","last_fullsized":"2024-06-29T14:23:17.971988+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:06.893440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191665+0000","last_change":"2024-06-29T14:23:15.203169+0000","last_active":"2024-06-29T14:23:17.191665+0000","last_peered":"2024-06-29T14:23:17.191665+0000","last_clean":"2024-06-29T14:23:17.191665+0000","last_became_active":"2024-06-29T14:23:15.202809+0000","last_became_peered":"2024-06-29T14:23:15.202809+0000","last_unstale":"2024-06-29T14:23:17.191665+0000","last_undegraded":"2024-06-29T14:23:17.191665+0000","last_fullsized":"2024-06-29T14:23:17.191665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:24.568080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971836+0000","last_change":"2024-06-29T14:23:16.201511+0000","last_active":"2024-06-29T14:23:17.971836+0000","last_peered":"2024-06-29T14:23:17.971836+0000","last_clean":"2024-06-29T14:23:17.971836+0000","last_became_active":"2024-06-29T14:23:16.200919+0000","last_became_peered":"2024-06-29T14:23:16.200919+0000","last_unstale":"2024-06-29T14:23:17.971836+0000","last_undegraded":"2024-06-29T14:23:17.971836+0000","last_fullsized":"2024-06-29T14:23:17.971836+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:25.539503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185188+0000","last_change":"2024-06-29T14:23:15.200889+0000","last_active":"2024-06-29T14:23:16.185188+0000","last_peered":"2024-06-29T14:23:16.185188+0000","last_clean":"2024-06-29T14:23:16.185188+0000","last_became_active":"2024-06-29T14:23:15.200521+0000","last_became_peered":"2024-06-29T14:23:15.200521+0000","last_unstale":"2024-06-29T14:23:16.185188+0000","last_undegraded":"2024-06-29T14:23:16.185188+0000","last_fullsized":"2024-06-29T14:23:16.185188+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217265+0000","last_change":"2024-06-29T14:23:16.207437+0000","last_active":"2024-06-29T14:23:16.217265+0000","last_peered":"2024-06-29T14:23:16.217265+0000","last_clean":"2024-06-29T14:23:16.217265+0000","last_became_active":"2024-06-29T14:23:16.207056+0000","last_became_peered":"2024-06-29T14:23:16.207056+0000","last_unstale":"2024-06-29T14:23:16.217265+0000","last_undegraded":"2024-06-29T14:23:16.217265+0000","last_fullsized":"2024-06-29T14:23:16.217265+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:52.297506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971336+0000","last_change":"2024-06-29T14:23:15.199990+0000","last_active":"2024-06-29T14:23:17.971336+0000","last_peered":"2024-06-29T14:23:17.971336+0000","last_clean":"2024-06-29T14:23:17.971336+0000","last_became_active":"2024-06-29T14:23:15.199697+0000","last_became_peered":"2024-06-29T14:23:15.199697+0000","last_unstale":"2024-06-29T14:23:17.971336+0000","last_undegraded":"2024-06-29T14:23:17.971336+0000","last_fullsized":"2024-06-29T14:23:17.971336+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:03.756929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.973018+0000","last_change":"2024-06-29T14:23:16.197089+0000","last_active":"2024-06-29T14:23:17.973018+0000","last_peered":"2024-06-29T14:23:17.973018+0000","last_clean":"2024-06-29T14:23:17.973018+0000","last_became_active":"2024-06-29T14:23:16.196730+0000","last_became_peered":"2024-06-29T14:23:16.196730+0000","last_unstale":"2024-06-29T14:23:17.973018+0000","last_undegraded":"2024-06-29T14:23:17.973018+0000","last_fullsized":"2024-06-29T14:23:17.973018+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:55.411612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.182345+0000","last_change":"2024-06-29T14:23:15.189674+0000","last_active":"2024-06-29T14:23:16.182345+0000","last_peered":"2024-06-29T14:23:16.182345+0000","last_clean":"2024-06-29T14:23:16.182345+0000","last_became_active":"2024-06-29T14:23:15.189326+0000","last_became_peered":"2024-06-29T14:23:15.189326+0000","last_unstale":"2024-06-29T14:23:16.182345+0000","last_undegraded":"2024-06-29T14:23:16.182345+0000","last_fullsized":"2024-06-29T14:23:16.182345+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:09.396994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191913+0000","last_change":"2024-06-29T14:23:16.209462+0000","last_active":"2024-06-29T14:23:17.191913+0000","last_peered":"2024-06-29T14:23:17.191913+0000","last_clean":"2024-06-29T14:23:17.191913+0000","last_became_active":"2024-06-29T14:23:16.209117+0000","last_became_peered":"2024-06-29T14:23:16.209117+0000","last_unstale":"2024-06-29T14:23:17.191913+0000","last_undegraded":"2024-06-29T14:23:17.191913+0000","last_fullsized":"2024-06-29T14:23:17.191913+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:13:09.963558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972810+0000","last_change":"2024-06-29T14:23:15.205426+0000","last_active":"2024-06-29T14:23:17.972810+0000","last_peered":"2024-06-29T14:23:17.972810+0000","last_clean":"2024-06-29T14:23:17.972810+0000","last_became_active":"2024-06-29T14:23:15.204970+0000","last_became_peered":"2024-06-29T14:23:15.204970+0000","last_unstale":"2024-06-29T14:23:17.972810+0000","last_undegraded":"2024-06-29T14:23:17.972810+0000","last_fullsized":"2024-06-29T14:23:17.972810+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:54.665868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.213758+0000","last_change":"2024-06-29T14:23:16.208578+0000","last_active":"2024-06-29T14:23:16.213758+0000","last_peered":"2024-06-29T14:23:16.213758+0000","last_clean":"2024-06-29T14:23:16.213758+0000","last_became_active":"2024-06-29T14:23:16.208189+0000","last_became_peered":"2024-06-29T14:23:16.208189+0000","last_unstale":"2024-06-29T14:23:16.213758+0000","last_undegraded":"2024-06-29T14:23:16.213758+0000","last_fullsized":"2024-06-29T14:23:16.213758+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:08.341184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"40'3","reported_seq":17,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972679+0000","last_change":"2024-06-29T14:23:15.202041+0000","last_active":"2024-06-29T14:23:17.972679+0000","last_peered":"2024-06-29T14:23:17.972679+0000","last_clean":"2024-06-29T14:23:17.972679+0000","last_became_active":"2024-06-29T14:23:15.201722+0000","last_became_peered":"2024-06-29T14:23:15.201722+0000","last_unstale":"2024-06-29T14:23:17.972679+0000","last_undegraded":"2024-06-29T14:23:17.972679+0000","last_fullsized":"2024-06-29T14:23:17.972679+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:28.995567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972654+0000","last_change":"2024-06-29T14:23:16.202659+0000","last_active":"2024-06-29T14:23:17.972654+0000","last_peered":"2024-06-29T14:23:17.972654+0000","last_clean":"2024-06-29T14:23:17.972654+0000","last_became_active":"2024-06-29T14:23:16.202249+0000","last_became_peered":"2024-06-29T14:23:16.202249+0000","last_unstale":"2024-06-29T14:23:17.972654+0000","last_undegraded":"2024-06-29T14:23:17.972654+0000","last_fullsized":"2024-06-29T14:23:17.972654+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:52:54.351905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"40'6","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.220943+0000","last_change":"2024-06-29T14:23:15.199353+0000","last_active":"2024-06-29T14:23:16.220943+0000","last_peered":"2024-06-29T14:23:16.220943+0000","last_clean":"2024-06-29T14:23:16.220943+0000","last_became_active":"2024-06-29T14:23:15.198339+0000","last_became_peered":"2024-06-29T14:23:15.198339+0000","last_unstale":"2024-06-29T14:23:16.220943+0000","last_undegraded":"2024-06-29T14:23:16.220943+0000","last_fullsized":"2024-06-29T14:23:16.220943+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:39.718146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.214670+0000","last_change":"2024-06-29T14:23:16.209778+0000","last_active":"2024-06-29T14:23:16.214670+0000","last_peered":"2024-06-29T14:23:16.214670+0000","last_clean":"2024-06-29T14:23:16.214670+0000","last_became_active":"2024-06-29T14:23:16.209443+0000","last_became_peered":"2024-06-29T14:23:16.209443+0000","last_unstale":"2024-06-29T14:23:16.214670+0000","last_undegraded":"2024-06-29T14:23:16.214670+0000","last_fullsized":"2024-06-29T14:23:16.214670+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:55.671459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"40'2","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.219006+0000","last_change":"2024-06-29T14:23:15.200701+0000","last_active":"2024-06-29T14:23:16.219006+0000","last_peered":"2024-06-29T14:23:16.219006+0000","last_clean":"2024-06-29T14:23:16.219006+0000","last_became_active":"2024-06-29T14:23:15.200338+0000","last_became_peered":"2024-06-29T14:23:15.200338+0000","last_unstale":"2024-06-29T14:23:16.219006+0000","last_undegraded":"2024-06-29T14:23:16.219006+0000","last_fullsized":"2024-06-29T14:23:16.219006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:54.665868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.201400+0000","last_change":"2024-06-29T14:23:16.196448+0000","last_active":"2024-06-29T14:23:16.201400+0000","last_peered":"2024-06-29T14:23:16.201400+0000","last_clean":"2024-06-29T14:23:16.201400+0000","last_became_active":"2024-06-29T14:23:16.195006+0000","last_became_peered":"2024-06-29T14:23:16.195006+0000","last_unstale":"2024-06-29T14:23:16.201400+0000","last_undegraded":"2024-06-29T14:23:16.201400+0000","last_fullsized":"2024-06-29T14:23:16.201400+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:05.074083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185776+0000","last_change":"2024-06-29T14:23:15.199955+0000","last_active":"2024-06-29T14:23:16.185776+0000","last_peered":"2024-06-29T14:23:16.185776+0000","last_clean":"2024-06-29T14:23:16.185776+0000","last_became_active":"2024-06-29T14:23:15.199619+0000","last_became_peered":"2024-06-29T14:23:15.199619+0000","last_unstale":"2024-06-29T14:23:16.185776+0000","last_undegraded":"2024-06-29T14:23:16.185776+0000","last_fullsized":"2024-06-29T14:23:16.185776+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:04.589609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191906+0000","last_change":"2024-06-29T14:23:16.211883+0000","last_active":"2024-06-29T14:23:17.191906+0000","last_peered":"2024-06-29T14:23:17.191906+0000","last_clean":"2024-06-29T14:23:17.191906+0000","last_became_active":"2024-06-29T14:23:16.211577+0000","last_became_peered":"2024-06-29T14:23:16.211577+0000","last_unstale":"2024-06-29T14:23:17.191906+0000","last_undegraded":"2024-06-29T14:23:17.191906+0000","last_fullsized":"2024-06-29T14:23:17.191906+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:39.476522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191955+0000","last_change":"2024-06-29T14:23:15.196439+0000","last_active":"2024-06-29T14:23:17.191955+0000","last_peered":"2024-06-29T14:23:17.191955+0000","last_clean":"2024-06-29T14:23:17.191955+0000","last_became_active":"2024-06-29T14:23:15.196116+0000","last_became_peered":"2024-06-29T14:23:15.196116+0000","last_unstale":"2024-06-29T14:23:17.191955+0000","last_undegraded":"2024-06-29T14:23:17.191955+0000","last_fullsized":"2024-06-29T14:23:17.191955+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:27.766774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.213976+0000","last_change":"2024-06-29T14:23:16.200847+0000","last_active":"2024-06-29T14:23:16.213976+0000","last_peered":"2024-06-29T14:23:16.213976+0000","last_clean":"2024-06-29T14:23:16.213976+0000","last_became_active":"2024-06-29T14:23:16.200264+0000","last_became_peered":"2024-06-29T14:23:16.200264+0000","last_unstale":"2024-06-29T14:23:16.213976+0000","last_undegraded":"2024-06-29T14:23:16.213976+0000","last_fullsized":"2024-06-29T14:23:16.213976+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:46.554739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"40'8","reported_seq":22,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192728+0000","last_change":"2024-06-29T14:23:15.200739+0000","last_active":"2024-06-29T14:23:17.192728+0000","last_peered":"2024-06-29T14:23:17.192728+0000","last_clean":"2024-06-29T14:23:17.192728+0000","last_became_active":"2024-06-29T14:23:15.200401+0000","last_became_peered":"2024-06-29T14:23:15.200401+0000","last_unstale":"2024-06-29T14:23:17.192728+0000","last_undegraded":"2024-06-29T14:23:17.192728+0000","last_fullsized":"2024-06-29T14:23:17.192728+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:28.254612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.194644+0000","last_change":"2024-06-29T14:23:16.188970+0000","last_active":"2024-06-29T14:23:16.194644+0000","last_peered":"2024-06-29T14:23:16.194644+0000","last_clean":"2024-06-29T14:23:16.194644+0000","last_became_active":"2024-06-29T14:23:16.188588+0000","last_became_peered":"2024-06-29T14:23:16.188588+0000","last_unstale":"2024-06-29T14:23:16.194644+0000","last_undegraded":"2024-06-29T14:23:16.194644+0000","last_fullsized":"2024-06-29T14:23:16.194644+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972725+0000","last_change":"2024-06-29T14:23:15.199255+0000","last_active":"2024-06-29T14:23:17.972725+0000","last_peered":"2024-06-29T14:23:17.972725+0000","last_clean":"2024-06-29T14:23:17.972725+0000","last_became_active":"2024-06-29T14:23:15.198806+0000","last_became_peered":"2024-06-29T14:23:15.198806+0000","last_unstale":"2024-06-29T14:23:17.972725+0000","last_undegraded":"2024-06-29T14:23:17.972725+0000","last_fullsized":"2024-06-29T14:23:17.972725+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:39.718146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972659+0000","last_change":"2024-06-29T14:23:16.210090+0000","last_active":"2024-06-29T14:23:17.972659+0000","last_peered":"2024-06-29T14:23:17.972659+0000","last_clean":"2024-06-29T14:23:17.972659+0000","last_became_active":"2024-06-29T14:23:16.209664+0000","last_became_peered":"2024-06-29T14:23:16.209664+0000","last_unstale":"2024-06-29T14:23:17.972659+0000","last_undegraded":"2024-06-29T14:23:17.972659+0000","last_fullsized":"2024-06-29T14:23:17.972659+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:11.338016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972475+0000","last_change":"2024-06-29T14:23:15.200790+0000","last_active":"2024-06-29T14:23:17.972475+0000","last_peered":"2024-06-29T14:23:17.972475+0000","last_clean":"2024-06-29T14:23:17.972475+0000","last_became_active":"2024-06-29T14:23:15.200452+0000","last_became_peered":"2024-06-29T14:23:15.200452+0000","last_unstale":"2024-06-29T14:23:17.972475+0000","last_undegraded":"2024-06-29T14:23:17.972475+0000","last_fullsized":"2024-06-29T14:23:17.972475+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:12.157450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972305+0000","last_change":"2024-06-29T14:23:16.204457+0000","last_active":"2024-06-29T14:23:17.972305+0000","last_peered":"2024-06-29T14:23:17.972305+0000","last_clean":"2024-06-29T14:23:17.972305+0000","last_became_active":"2024-06-29T14:23:16.204087+0000","last_became_peered":"2024-06-29T14:23:16.204087+0000","last_unstale":"2024-06-29T14:23:17.972305+0000","last_undegraded":"2024-06-29T14:23:17.972305+0000","last_fullsized":"2024-06-29T14:23:17.972305+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:54.822759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972729+0000","last_change":"2024-06-29T14:23:15.195356+0000","last_active":"2024-06-29T14:23:17.972729+0000","last_peered":"2024-06-29T14:23:17.972729+0000","last_clean":"2024-06-29T14:23:17.972729+0000","last_became_active":"2024-06-29T14:23:15.195037+0000","last_became_peered":"2024-06-29T14:23:15.195037+0000","last_unstale":"2024-06-29T14:23:17.972729+0000","last_undegraded":"2024-06-29T14:23:17.972729+0000","last_fullsized":"2024-06-29T14:23:17.972729+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:32.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971916+0000","last_change":"2024-06-29T14:23:16.197431+0000","last_active":"2024-06-29T14:23:17.971916+0000","last_peered":"2024-06-29T14:23:17.971916+0000","last_clean":"2024-06-29T14:23:17.971916+0000","last_became_active":"2024-06-29T14:23:16.197091+0000","last_became_peered":"2024-06-29T14:23:16.197091+0000","last_unstale":"2024-06-29T14:23:17.971916+0000","last_undegraded":"2024-06-29T14:23:17.971916+0000","last_fullsized":"2024-06-29T14:23:17.971916+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:46.040585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.185494+0000","last_change":"2024-06-29T14:23:15.203782+0000","last_active":"2024-06-29T14:23:16.185494+0000","last_peered":"2024-06-29T14:23:16.185494+0000","last_clean":"2024-06-29T14:23:16.185494+0000","last_became_active":"2024-06-29T14:23:15.203451+0000","last_became_peered":"2024-06-29T14:23:15.203451+0000","last_unstale":"2024-06-29T14:23:16.185494+0000","last_undegraded":"2024-06-29T14:23:16.185494+0000","last_fullsized":"2024-06-29T14:23:16.185494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:18.072309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192298+0000","last_change":"2024-06-29T14:23:16.212117+0000","last_active":"2024-06-29T14:23:17.192298+0000","last_peered":"2024-06-29T14:23:17.192298+0000","last_clean":"2024-06-29T14:23:17.192298+0000","last_became_active":"2024-06-29T14:23:16.211660+0000","last_became_peered":"2024-06-29T14:23:16.211660+0000","last_unstale":"2024-06-29T14:23:17.192298+0000","last_undegraded":"2024-06-29T14:23:17.192298+0000","last_fullsized":"2024-06-29T14:23:17.192298+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:56.501064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972722+0000","last_change":"2024-06-29T14:23:15.201493+0000","last_active":"2024-06-29T14:23:17.972722+0000","last_peered":"2024-06-29T14:23:17.972722+0000","last_clean":"2024-06-29T14:23:17.972722+0000","last_became_active":"2024-06-29T14:23:15.201150+0000","last_became_peered":"2024-06-29T14:23:15.201150+0000","last_unstale":"2024-06-29T14:23:17.972722+0000","last_undegraded":"2024-06-29T14:23:17.972722+0000","last_fullsized":"2024-06-29T14:23:17.972722+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:00.323663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.206186+0000","last_change":"2024-06-29T14:23:16.202653+0000","last_active":"2024-06-29T14:23:16.206186+0000","last_peered":"2024-06-29T14:23:16.206186+0000","last_clean":"2024-06-29T14:23:16.206186+0000","last_became_active":"2024-06-29T14:23:16.202279+0000","last_became_peered":"2024-06-29T14:23:16.202279+0000","last_unstale":"2024-06-29T14:23:16.206186+0000","last_undegraded":"2024-06-29T14:23:16.206186+0000","last_fullsized":"2024-06-29T14:23:16.206186+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:05.595200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971710+0000","last_change":"2024-06-29T14:23:15.196760+0000","last_active":"2024-06-29T14:23:17.971710+0000","last_peered":"2024-06-29T14:23:17.971710+0000","last_clean":"2024-06-29T14:23:17.971710+0000","last_became_active":"2024-06-29T14:23:15.196278+0000","last_became_peered":"2024-06-29T14:23:15.196278+0000","last_unstale":"2024-06-29T14:23:17.971710+0000","last_undegraded":"2024-06-29T14:23:17.971710+0000","last_fullsized":"2024-06-29T14:23:17.971710+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:32:01.571661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972354+0000","last_change":"2024-06-29T14:23:16.200152+0000","last_active":"2024-06-29T14:23:17.972354+0000","last_peered":"2024-06-29T14:23:17.972354+0000","last_clean":"2024-06-29T14:23:17.972354+0000","last_became_active":"2024-06-29T14:23:16.199828+0000","last_became_peered":"2024-06-29T14:23:16.199828+0000","last_unstale":"2024-06-29T14:23:17.972354+0000","last_undegraded":"2024-06-29T14:23:17.972354+0000","last_fullsized":"2024-06-29T14:23:17.972354+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:43.638548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192608+0000","last_change":"2024-06-29T14:23:15.199837+0000","last_active":"2024-06-29T14:23:17.192608+0000","last_peered":"2024-06-29T14:23:17.192608+0000","last_clean":"2024-06-29T14:23:17.192608+0000","last_became_active":"2024-06-29T14:23:15.199539+0000","last_became_peered":"2024-06-29T14:23:15.199539+0000","last_unstale":"2024-06-29T14:23:17.192608+0000","last_undegraded":"2024-06-29T14:23:17.192608+0000","last_fullsized":"2024-06-29T14:23:17.192608+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:32.149209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.217291+0000","last_change":"2024-06-29T14:23:16.211956+0000","last_active":"2024-06-29T14:23:16.217291+0000","last_peered":"2024-06-29T14:23:16.217291+0000","last_clean":"2024-06-29T14:23:16.217291+0000","last_became_active":"2024-06-29T14:23:16.211649+0000","last_became_peered":"2024-06-29T14:23:16.211649+0000","last_unstale":"2024-06-29T14:23:16.217291+0000","last_undegraded":"2024-06-29T14:23:16.217291+0000","last_fullsized":"2024-06-29T14:23:16.217291+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:37.213779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972618+0000","last_change":"2024-06-29T14:23:15.201805+0000","last_active":"2024-06-29T14:23:17.972618+0000","last_peered":"2024-06-29T14:23:17.972618+0000","last_clean":"2024-06-29T14:23:17.972618+0000","last_became_active":"2024-06-29T14:23:15.200924+0000","last_became_peered":"2024-06-29T14:23:15.200924+0000","last_unstale":"2024-06-29T14:23:17.972618+0000","last_undegraded":"2024-06-29T14:23:17.972618+0000","last_fullsized":"2024-06-29T14:23:17.972618+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:18.651137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207150+0000","last_change":"2024-06-29T14:23:16.205768+0000","last_active":"2024-06-29T14:23:16.207150+0000","last_peered":"2024-06-29T14:23:16.207150+0000","last_clean":"2024-06-29T14:23:16.207150+0000","last_became_active":"2024-06-29T14:23:16.205382+0000","last_became_peered":"2024-06-29T14:23:16.205382+0000","last_unstale":"2024-06-29T14:23:16.207150+0000","last_undegraded":"2024-06-29T14:23:16.207150+0000","last_fullsized":"2024-06-29T14:23:16.207150+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:55.671459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972843+0000","last_change":"2024-06-29T14:23:15.203975+0000","last_active":"2024-06-29T14:23:17.972843+0000","last_peered":"2024-06-29T14:23:17.972843+0000","last_clean":"2024-06-29T14:23:17.972843+0000","last_became_active":"2024-06-29T14:23:15.203652+0000","last_became_peered":"2024-06-29T14:23:15.203652+0000","last_unstale":"2024-06-29T14:23:17.972843+0000","last_undegraded":"2024-06-29T14:23:17.972843+0000","last_fullsized":"2024-06-29T14:23:17.972843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:09.396994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192108+0000","last_change":"2024-06-29T14:23:16.210252+0000","last_active":"2024-06-29T14:23:17.192108+0000","last_peered":"2024-06-29T14:23:17.192108+0000","last_clean":"2024-06-29T14:23:17.192108+0000","last_became_active":"2024-06-29T14:23:16.209936+0000","last_became_peered":"2024-06-29T14:23:16.209936+0000","last_unstale":"2024-06-29T14:23:17.192108+0000","last_undegraded":"2024-06-29T14:23:17.192108+0000","last_fullsized":"2024-06-29T14:23:17.192108+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:36.806180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"41'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:18.201843+0000","last_change":"2024-06-29T14:23:15.199068+0000","last_active":"2024-06-29T14:23:18.201843+0000","last_peered":"2024-06-29T14:23:18.201843+0000","last_clean":"2024-06-29T14:23:18.201843+0000","last_became_active":"2024-06-29T14:23:15.198714+0000","last_became_peered":"2024-06-29T14:23:15.198714+0000","last_unstale":"2024-06-29T14:23:18.201843+0000","last_undegraded":"2024-06-29T14:23:18.201843+0000","last_fullsized":"2024-06-29T14:23:18.201843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:11.722673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.199343+0000","last_change":"2024-06-29T14:23:16.194459+0000","last_active":"2024-06-29T14:23:16.199343+0000","last_peered":"2024-06-29T14:23:16.199343+0000","last_clean":"2024-06-29T14:23:16.199343+0000","last_became_active":"2024-06-29T14:23:16.194084+0000","last_became_peered":"2024-06-29T14:23:16.194084+0000","last_unstale":"2024-06-29T14:23:16.199343+0000","last_undegraded":"2024-06-29T14:23:16.199343+0000","last_fullsized":"2024-06-29T14:23:16.199343+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:46.325166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.192313+0000","last_change":"2024-06-29T14:23:15.199199+0000","last_active":"2024-06-29T14:23:16.192313+0000","last_peered":"2024-06-29T14:23:16.192313+0000","last_clean":"2024-06-29T14:23:16.192313+0000","last_became_active":"2024-06-29T14:23:15.198802+0000","last_became_peered":"2024-06-29T14:23:15.198802+0000","last_unstale":"2024-06-29T14:23:16.192313+0000","last_undegraded":"2024-06-29T14:23:16.192313+0000","last_fullsized":"2024-06-29T14:23:16.192313+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:41.669608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.207171+0000","last_change":"2024-06-29T14:23:16.200965+0000","last_active":"2024-06-29T14:23:16.207171+0000","last_peered":"2024-06-29T14:23:16.207171+0000","last_clean":"2024-06-29T14:23:16.207171+0000","last_became_active":"2024-06-29T14:23:16.200584+0000","last_became_peered":"2024-06-29T14:23:16.200584+0000","last_unstale":"2024-06-29T14:23:16.207171+0000","last_undegraded":"2024-06-29T14:23:16.207171+0000","last_fullsized":"2024-06-29T14:23:16.207171+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:19.077899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.194138+0000","last_change":"2024-06-29T14:23:15.203329+0000","last_active":"2024-06-29T14:23:16.194138+0000","last_peered":"2024-06-29T14:23:16.194138+0000","last_clean":"2024-06-29T14:23:16.194138+0000","last_became_active":"2024-06-29T14:23:15.202429+0000","last_became_peered":"2024-06-29T14:23:15.202429+0000","last_unstale":"2024-06-29T14:23:16.194138+0000","last_undegraded":"2024-06-29T14:23:16.194138+0000","last_fullsized":"2024-06-29T14:23:16.194138+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.198488+0000","last_change":"2024-06-29T14:23:16.190685+0000","last_active":"2024-06-29T14:23:16.198488+0000","last_peered":"2024-06-29T14:23:16.198488+0000","last_clean":"2024-06-29T14:23:16.198488+0000","last_became_active":"2024-06-29T14:23:16.190310+0000","last_became_peered":"2024-06-29T14:23:16.190310+0000","last_unstale":"2024-06-29T14:23:16.198488+0000","last_undegraded":"2024-06-29T14:23:16.198488+0000","last_fullsized":"2024-06-29T14:23:16.198488+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:08.341184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192125+0000","last_change":"2024-06-29T14:23:15.200968+0000","last_active":"2024-06-29T14:23:17.192125+0000","last_peered":"2024-06-29T14:23:17.192125+0000","last_clean":"2024-06-29T14:23:17.192125+0000","last_became_active":"2024-06-29T14:23:15.200573+0000","last_became_peered":"2024-06-29T14:23:15.200573+0000","last_unstale":"2024-06-29T14:23:17.192125+0000","last_undegraded":"2024-06-29T14:23:17.192125+0000","last_fullsized":"2024-06-29T14:23:17.192125+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:54.406021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971610+0000","last_change":"2024-06-29T14:23:16.200262+0000","last_active":"2024-06-29T14:23:17.971610+0000","last_peered":"2024-06-29T14:23:17.971610+0000","last_clean":"2024-06-29T14:23:17.971610+0000","last_became_active":"2024-06-29T14:23:16.199984+0000","last_became_peered":"2024-06-29T14:23:16.199984+0000","last_unstale":"2024-06-29T14:23:17.971610+0000","last_undegraded":"2024-06-29T14:23:17.971610+0000","last_fullsized":"2024-06-29T14:23:17.971610+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:52:50.336628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971381+0000","last_change":"2024-06-29T14:23:15.197575+0000","last_active":"2024-06-29T14:23:17.971381+0000","last_peered":"2024-06-29T14:23:17.971381+0000","last_clean":"2024-06-29T14:23:17.971381+0000","last_became_active":"2024-06-29T14:23:15.197267+0000","last_became_peered":"2024-06-29T14:23:15.197267+0000","last_unstale":"2024-06-29T14:23:17.971381+0000","last_undegraded":"2024-06-29T14:23:17.971381+0000","last_fullsized":"2024-06-29T14:23:17.971381+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:35.221447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191930+0000","last_change":"2024-06-29T14:23:16.207769+0000","last_active":"2024-06-29T14:23:17.191930+0000","last_peered":"2024-06-29T14:23:17.191930+0000","last_clean":"2024-06-29T14:23:17.191930+0000","last_became_active":"2024-06-29T14:23:16.207431+0000","last_became_peered":"2024-06-29T14:23:16.207431+0000","last_unstale":"2024-06-29T14:23:17.191930+0000","last_undegraded":"2024-06-29T14:23:17.191930+0000","last_fullsized":"2024-06-29T14:23:17.191930+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:32.820240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-29T14:23:16.181400+0000","last_change":"2024-06-29T14:23:15.192161+0000","last_active":"2024-06-29T14:23:16.181400+0000","last_peered":"2024-06-29T14:23:16.181400+0000","last_clean":"2024-06-29T14:23:16.181400+0000","last_became_active":"2024-06-29T14:23:15.191810+0000","last_became_peered":"2024-06-29T14:23:15.191810+0000","last_unstale":"2024-06-29T14:23:16.181400+0000","last_undegraded":"2024-06-29T14:23:16.181400+0000","last_fullsized":"2024-06-29T14:23:16.181400+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:28.995567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972582+0000","last_change":"2024-06-29T14:23:16.202651+0000","last_active":"2024-06-29T14:23:17.972582+0000","last_peered":"2024-06-29T14:23:17.972582+0000","last_clean":"2024-06-29T14:23:17.972582+0000","last_became_active":"2024-06-29T14:23:16.202214+0000","last_became_peered":"2024-06-29T14:23:16.202214+0000","last_unstale":"2024-06-29T14:23:17.972582+0000","last_undegraded":"2024-06-29T14:23:17.972582+0000","last_fullsized":"2024-06-29T14:23:17.972582+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:44.745370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192983+0000","last_change":"2024-06-29T14:23:15.198041+0000","last_active":"2024-06-29T14:23:17.192983+0000","last_peered":"2024-06-29T14:23:17.192983+0000","last_clean":"2024-06-29T14:23:17.192983+0000","last_became_active":"2024-06-29T14:23:15.197662+0000","last_became_peered":"2024-06-29T14:23:15.197662+0000","last_unstale":"2024-06-29T14:23:17.192983+0000","last_undegraded":"2024-06-29T14:23:17.192983+0000","last_fullsized":"2024-06-29T14:23:17.192983+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:01.924403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":146,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192855+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:23:17.192855+0000","last_peered":"2024-06-29T14:23:17.192855+0000","last_clean":"2024-06-29T14:23:17.192855+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:23:17.192855+0000","last_undegraded":"2024-06-29T14:23:17.192855+0000","last_fullsized":"2024-06-29T14:23:17.192855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":197889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509462,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":196333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":779444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1077248,"data_stored":794985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":471040,"data_stored":203517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":475136,"data_stored":203047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":203539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":203005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:23:23.009 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:23:23.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:23:24.825 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:23:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.952+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/1187356015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e0141bd0 msgr2=0x7fa4e013edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.952+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/1187356015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e0141bd0 0x7fa4e013edc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005fd0 tx=0x7fa4d802aa10 comp rx=0 tx=0).stop 2024-06-29T14:23:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.952+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/1187356015 shutdown_connections 2024-06-29T14:23:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.952+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/1187356015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e0141bd0 0x7fa4e013edc0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.952+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/1187356015 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4e015ceb0 0x7fa4e00b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.952+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/1187356015 >> 172.21.15.59:0/1187356015 conn(0x7fa4e00b6fb0 msgr2=0x7fa4e00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.952+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/1187356015 shutdown_connections 2024-06-29T14:23:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.952+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/1187356015 wait complete. 2024-06-29T14:23:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.954+0000 7fa4e5b1c640 1 Processor -- start 2024-06-29T14:23:24.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.954+0000 7fa4e5b1c640 1 -- start start 2024-06-29T14:23:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.955+0000 7fa4e5b1c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4e013e9b0 0x7fa4e01f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.955+0000 7fa4e5b1c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4e015ceb0 0x7fa4e0150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.955+0000 7fa4e5b1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e01561a0 0x7fa4e01510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.955+0000 7fa4e5b1c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e00b2a80 con 0x7fa4e01561a0 2024-06-29T14:23:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.955+0000 7fa4e5b1c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e00b2900 con 0x7fa4e015ceb0 2024-06-29T14:23:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.955+0000 7fa4e5b1c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e00b2c00 con 0x7fa4e013e9b0 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.955+0000 7fa4dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e01561a0 0x7fa4e01510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e01561a0 0x7fa4e01510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59550/0 (socket says 172.21.15.59:59550) 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4dffff640 1 -- 172.21.15.59:0/4176348034 learned_addr learned my addr 172.21.15.59:0/4176348034 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4df7fe640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4e013e9b0 0x7fa4e01f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4deffd640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4e015ceb0 0x7fa4e0150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4dffff640 1 -- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4e013e9b0 msgr2=0x7fa4e01f4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4dffff640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4e013e9b0 0x7fa4e01f4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4dffff640 1 -- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4e015ceb0 msgr2=0x7fa4e0150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4dffff640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4e015ceb0 0x7fa4e0150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4dffff640 1 -- 172.21.15.59:0/4176348034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e01f4590 con 0x7fa4e01561a0 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4dffff640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e01561a0 0x7fa4e01510a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8006010 tx=0x7fa4d8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4c7fff640 1 -- 172.21.15.59:0/4176348034 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8033050 con 0x7fa4e01561a0 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4c7fff640 1 -- 172.21.15.59:0/4176348034 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4d8004250 con 0x7fa4e01561a0 2024-06-29T14:23:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4c7fff640 1 -- 172.21.15.59:0/4176348034 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8004680 con 0x7fa4e01561a0 2024-06-29T14:23:24.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e006e630 con 0x7fa4e01561a0 2024-06-29T14:23:24.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.956+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e006ebf0 con 0x7fa4e01561a0 2024-06-29T14:23:24.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.957+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e013edc0 con 0x7fa4e01561a0 2024-06-29T14:23:24.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.958+0000 7fa4c7fff640 1 -- 172.21.15.59:0/4176348034 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa4d8036070 con 0x7fa4e01561a0 2024-06-29T14:23:24.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.959+0000 7fa4c7fff640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa4cc076300 0x7fa4cc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.959+0000 7fa4c7fff640 1 -- 172.21.15.59:0/4176348034 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7fa4d80b8960 con 0x7fa4e01561a0 2024-06-29T14:23:24.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.959+0000 7fa4df7fe640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa4cc076300 0x7fa4cc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.959+0000 7fa4df7fe640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa4cc076300 0x7fa4cc078810 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0009b30 tx=0x7fa4d0005f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:24.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:24.963+0000 7fa4c7fff640 1 -- 172.21.15.59:0/4176348034 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa4d8004820 con 0x7fa4e01561a0 2024-06-29T14:23:25.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.152+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa4e00708d0 con 0x7fa4cc076300 2024-06-29T14:23:25.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.157+0000 7fa4c7fff640 1 -- 172.21.15.59:0/4176348034 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251936 (secure 0 0 0) 0x7fa4e00708d0 con 0x7fa4cc076300 2024-06-29T14:23:25.157 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:25.159 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.159+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa4cc076300 msgr2=0x7fa4cc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.159+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa4cc076300 0x7fa4cc078810 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0009b30 tx=0x7fa4d0005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.159+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e01561a0 msgr2=0x7fa4e01510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e01561a0 0x7fa4e01510a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8006010 tx=0x7fa4d8003f10 comp rx=0 tx=0).stop 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 shutdown_connections 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa4cc076300 0x7fa4cc078810 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e01561a0 0x7fa4e01510a0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4e015ceb0 0x7fa4e0150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 --2- 172.21.15.59:0/4176348034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4e013e9b0 0x7fa4e01f4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 >> 172.21.15.59:0/4176348034 conn(0x7fa4e00b6fb0 msgr2=0x7fa4e0070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 shutdown_connections 2024-06-29T14:23:25.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.160+0000 7fa4e5b1c640 1 -- 172.21.15.59:0/4176348034 wait complete. 2024-06-29T14:23:25.171 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-06-29T14:23:23.753533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221060,"kb_used_data":4872,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753660,"statfs":{"total":773094113280,"available":772867747840,"internally_reserved":0,"allocated":4988928,"data_stored":2809036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001820"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189726+0000","last_change":"2024-06-29T14:23:16.201160+0000","last_active":"2024-06-29T14:23:17.189726+0000","last_peered":"2024-06-29T14:23:17.189726+0000","last_clean":"2024-06-29T14:23:17.189726+0000","last_became_active":"2024-06-29T14:23:16.200155+0000","last_became_peered":"2024-06-29T14:23:16.200155+0000","last_unstale":"2024-06-29T14:23:17.189726+0000","last_undegraded":"2024-06-29T14:23:17.189726+0000","last_fullsized":"2024-06-29T14:23:17.189726+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:46.690692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189303+0000","last_change":"2024-06-29T14:23:16.189930+0000","last_active":"2024-06-29T14:23:17.189303+0000","last_peered":"2024-06-29T14:23:17.189303+0000","last_clean":"2024-06-29T14:23:17.189303+0000","last_became_active":"2024-06-29T14:23:16.189571+0000","last_became_peered":"2024-06-29T14:23:16.189571+0000","last_unstale":"2024-06-29T14:23:17.189303+0000","last_undegraded":"2024-06-29T14:23:17.189303+0000","last_fullsized":"2024-06-29T14:23:17.189303+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:21.735913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971590+0000","last_change":"2024-06-29T14:23:16.203182+0000","last_active":"2024-06-29T14:23:17.971590+0000","last_peered":"2024-06-29T14:23:17.971590+0000","last_clean":"2024-06-29T14:23:17.971590+0000","last_became_active":"2024-06-29T14:23:16.202851+0000","last_became_peered":"2024-06-29T14:23:16.202851+0000","last_unstale":"2024-06-29T14:23:17.971590+0000","last_undegraded":"2024-06-29T14:23:17.971590+0000","last_fullsized":"2024-06-29T14:23:17.971590+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:10.882976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972956+0000","last_change":"2024-06-29T14:23:16.202834+0000","last_active":"2024-06-29T14:23:17.972956+0000","last_peered":"2024-06-29T14:23:17.972956+0000","last_clean":"2024-06-29T14:23:17.972956+0000","last_became_active":"2024-06-29T14:23:16.202357+0000","last_became_peered":"2024-06-29T14:23:16.202357+0000","last_unstale":"2024-06-29T14:23:17.972956+0000","last_undegraded":"2024-06-29T14:23:17.972956+0000","last_fullsized":"2024-06-29T14:23:17.972956+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:35.097600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972601+0000","last_change":"2024-06-29T14:23:16.194759+0000","last_active":"2024-06-29T14:23:17.972601+0000","last_peered":"2024-06-29T14:23:17.972601+0000","last_clean":"2024-06-29T14:23:17.972601+0000","last_became_active":"2024-06-29T14:23:16.194351+0000","last_became_peered":"2024-06-29T14:23:16.194351+0000","last_unstale":"2024-06-29T14:23:17.972601+0000","last_undegraded":"2024-06-29T14:23:17.972601+0000","last_fullsized":"2024-06-29T14:23:17.972601+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:30.838175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971538+0000","last_change":"2024-06-29T14:23:16.195834+0000","last_active":"2024-06-29T14:23:17.971538+0000","last_peered":"2024-06-29T14:23:17.971538+0000","last_clean":"2024-06-29T14:23:17.971538+0000","last_became_active":"2024-06-29T14:23:16.195476+0000","last_became_peered":"2024-06-29T14:23:16.195476+0000","last_unstale":"2024-06-29T14:23:17.971538+0000","last_undegraded":"2024-06-29T14:23:17.971538+0000","last_fullsized":"2024-06-29T14:23:17.971538+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:12.662244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971518+0000","last_change":"2024-06-29T14:23:16.205371+0000","last_active":"2024-06-29T14:23:17.971518+0000","last_peered":"2024-06-29T14:23:17.971518+0000","last_clean":"2024-06-29T14:23:17.971518+0000","last_became_active":"2024-06-29T14:23:16.204429+0000","last_became_peered":"2024-06-29T14:23:16.204429+0000","last_unstale":"2024-06-29T14:23:17.971518+0000","last_undegraded":"2024-06-29T14:23:17.971518+0000","last_fullsized":"2024-06-29T14:23:17.971518+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:14:42.425008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189076+0000","last_change":"2024-06-29T14:23:16.199825+0000","last_active":"2024-06-29T14:23:17.189076+0000","last_peered":"2024-06-29T14:23:17.189076+0000","last_clean":"2024-06-29T14:23:17.189076+0000","last_became_active":"2024-06-29T14:23:16.199499+0000","last_became_peered":"2024-06-29T14:23:16.199499+0000","last_unstale":"2024-06-29T14:23:17.189076+0000","last_undegraded":"2024-06-29T14:23:17.189076+0000","last_fullsized":"2024-06-29T14:23:17.189076+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:11:53.623336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972146+0000","last_change":"2024-06-29T14:23:16.209692+0000","last_active":"2024-06-29T14:23:17.972146+0000","last_peered":"2024-06-29T14:23:17.972146+0000","last_clean":"2024-06-29T14:23:17.972146+0000","last_became_active":"2024-06-29T14:23:16.202968+0000","last_became_peered":"2024-06-29T14:23:16.202968+0000","last_unstale":"2024-06-29T14:23:17.972146+0000","last_undegraded":"2024-06-29T14:23:17.972146+0000","last_fullsized":"2024-06-29T14:23:17.972146+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:23.684550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972219+0000","last_change":"2024-06-29T14:23:16.202448+0000","last_active":"2024-06-29T14:23:17.972219+0000","last_peered":"2024-06-29T14:23:17.972219+0000","last_clean":"2024-06-29T14:23:17.972219+0000","last_became_active":"2024-06-29T14:23:16.201952+0000","last_became_peered":"2024-06-29T14:23:16.201952+0000","last_unstale":"2024-06-29T14:23:17.972219+0000","last_undegraded":"2024-06-29T14:23:17.972219+0000","last_fullsized":"2024-06-29T14:23:17.972219+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:51.801025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972537+0000","last_change":"2024-06-29T14:23:16.204313+0000","last_active":"2024-06-29T14:23:17.972537+0000","last_peered":"2024-06-29T14:23:17.972537+0000","last_clean":"2024-06-29T14:23:17.972537+0000","last_became_active":"2024-06-29T14:23:16.203981+0000","last_became_peered":"2024-06-29T14:23:16.203981+0000","last_unstale":"2024-06-29T14:23:17.972537+0000","last_undegraded":"2024-06-29T14:23:17.972537+0000","last_fullsized":"2024-06-29T14:23:17.972537+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:46.690692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971365+0000","last_change":"2024-06-29T14:23:16.209122+0000","last_active":"2024-06-29T14:23:17.971365+0000","last_peered":"2024-06-29T14:23:17.971365+0000","last_clean":"2024-06-29T14:23:17.971365+0000","last_became_active":"2024-06-29T14:23:16.208783+0000","last_became_peered":"2024-06-29T14:23:16.208783+0000","last_unstale":"2024-06-29T14:23:17.971365+0000","last_undegraded":"2024-06-29T14:23:17.971365+0000","last_fullsized":"2024-06-29T14:23:17.971365+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:46.325166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971838+0000","last_change":"2024-06-29T14:23:16.206491+0000","last_active":"2024-06-29T14:23:17.971838+0000","last_peered":"2024-06-29T14:23:17.971838+0000","last_clean":"2024-06-29T14:23:17.971838+0000","last_became_active":"2024-06-29T14:23:16.205423+0000","last_became_peered":"2024-06-29T14:23:16.205423+0000","last_unstale":"2024-06-29T14:23:17.971838+0000","last_undegraded":"2024-06-29T14:23:17.971838+0000","last_fullsized":"2024-06-29T14:23:17.971838+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:28.205354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971455+0000","last_change":"2024-06-29T14:23:16.199501+0000","last_active":"2024-06-29T14:23:17.971455+0000","last_peered":"2024-06-29T14:23:17.971455+0000","last_clean":"2024-06-29T14:23:17.971455+0000","last_became_active":"2024-06-29T14:23:16.199071+0000","last_became_peered":"2024-06-29T14:23:16.199071+0000","last_unstale":"2024-06-29T14:23:17.971455+0000","last_undegraded":"2024-06-29T14:23:17.971455+0000","last_fullsized":"2024-06-29T14:23:17.971455+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:39.380846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971720+0000","last_change":"2024-06-29T14:23:16.205418+0000","last_active":"2024-06-29T14:23:17.971720+0000","last_peered":"2024-06-29T14:23:17.971720+0000","last_clean":"2024-06-29T14:23:17.971720+0000","last_became_active":"2024-06-29T14:23:16.205110+0000","last_became_peered":"2024-06-29T14:23:16.205110+0000","last_unstale":"2024-06-29T14:23:17.971720+0000","last_undegraded":"2024-06-29T14:23:17.971720+0000","last_fullsized":"2024-06-29T14:23:17.971720+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:21.735913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192094+0000","last_change":"2024-06-29T14:23:16.206412+0000","last_active":"2024-06-29T14:23:17.192094+0000","last_peered":"2024-06-29T14:23:17.192094+0000","last_clean":"2024-06-29T14:23:17.192094+0000","last_became_active":"2024-06-29T14:23:16.205447+0000","last_became_peered":"2024-06-29T14:23:16.205447+0000","last_unstale":"2024-06-29T14:23:17.192094+0000","last_undegraded":"2024-06-29T14:23:17.192094+0000","last_fullsized":"2024-06-29T14:23:17.192094+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:38:40.833526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192670+0000","last_change":"2024-06-29T14:23:16.205425+0000","last_active":"2024-06-29T14:23:17.192670+0000","last_peered":"2024-06-29T14:23:17.192670+0000","last_clean":"2024-06-29T14:23:17.192670+0000","last_became_active":"2024-06-29T14:23:16.205061+0000","last_became_peered":"2024-06-29T14:23:16.205061+0000","last_unstale":"2024-06-29T14:23:17.192670+0000","last_undegraded":"2024-06-29T14:23:17.192670+0000","last_fullsized":"2024-06-29T14:23:17.192670+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:18.957684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192585+0000","last_change":"2024-06-29T14:23:16.197724+0000","last_active":"2024-06-29T14:23:17.192585+0000","last_peered":"2024-06-29T14:23:17.192585+0000","last_clean":"2024-06-29T14:23:17.192585+0000","last_became_active":"2024-06-29T14:23:16.197347+0000","last_became_peered":"2024-06-29T14:23:16.197347+0000","last_unstale":"2024-06-29T14:23:17.192585+0000","last_undegraded":"2024-06-29T14:23:17.192585+0000","last_fullsized":"2024-06-29T14:23:17.192585+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:35.902597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972388+0000","last_change":"2024-06-29T14:23:16.190111+0000","last_active":"2024-06-29T14:23:17.972388+0000","last_peered":"2024-06-29T14:23:17.972388+0000","last_clean":"2024-06-29T14:23:17.972388+0000","last_became_active":"2024-06-29T14:23:16.189786+0000","last_became_peered":"2024-06-29T14:23:16.189786+0000","last_unstale":"2024-06-29T14:23:17.972388+0000","last_undegraded":"2024-06-29T14:23:17.972388+0000","last_fullsized":"2024-06-29T14:23:17.972388+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:31:12.500255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192271+0000","last_change":"2024-06-29T14:23:16.196160+0000","last_active":"2024-06-29T14:23:17.192271+0000","last_peered":"2024-06-29T14:23:17.192271+0000","last_clean":"2024-06-29T14:23:17.192271+0000","last_became_active":"2024-06-29T14:23:16.195798+0000","last_became_peered":"2024-06-29T14:23:16.195798+0000","last_unstale":"2024-06-29T14:23:17.192271+0000","last_undegraded":"2024-06-29T14:23:17.192271+0000","last_fullsized":"2024-06-29T14:23:17.192271+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:33.154800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971957+0000","last_change":"2024-06-29T14:23:16.209041+0000","last_active":"2024-06-29T14:23:17.971957+0000","last_peered":"2024-06-29T14:23:17.971957+0000","last_clean":"2024-06-29T14:23:17.971957+0000","last_became_active":"2024-06-29T14:23:16.208679+0000","last_became_peered":"2024-06-29T14:23:16.208679+0000","last_unstale":"2024-06-29T14:23:17.971957+0000","last_undegraded":"2024-06-29T14:23:17.971957+0000","last_fullsized":"2024-06-29T14:23:17.971957+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:05:44.443149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971753+0000","last_change":"2024-06-29T14:23:16.203100+0000","last_active":"2024-06-29T14:23:17.971753+0000","last_peered":"2024-06-29T14:23:17.971753+0000","last_clean":"2024-06-29T14:23:17.971753+0000","last_became_active":"2024-06-29T14:23:16.201831+0000","last_became_peered":"2024-06-29T14:23:16.201831+0000","last_unstale":"2024-06-29T14:23:17.971753+0000","last_undegraded":"2024-06-29T14:23:17.971753+0000","last_fullsized":"2024-06-29T14:23:17.971753+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:47:33.460264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192279+0000","last_change":"2024-06-29T14:23:16.211154+0000","last_active":"2024-06-29T14:23:17.192279+0000","last_peered":"2024-06-29T14:23:17.192279+0000","last_clean":"2024-06-29T14:23:17.192279+0000","last_became_active":"2024-06-29T14:23:16.210853+0000","last_became_peered":"2024-06-29T14:23:16.210853+0000","last_unstale":"2024-06-29T14:23:17.192279+0000","last_undegraded":"2024-06-29T14:23:17.192279+0000","last_fullsized":"2024-06-29T14:23:17.192279+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:38.716709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972485+0000","last_change":"2024-06-29T14:23:16.210617+0000","last_active":"2024-06-29T14:23:17.972485+0000","last_peered":"2024-06-29T14:23:17.972485+0000","last_clean":"2024-06-29T14:23:17.972485+0000","last_became_active":"2024-06-29T14:23:16.210242+0000","last_became_peered":"2024-06-29T14:23:16.210242+0000","last_unstale":"2024-06-29T14:23:17.972485+0000","last_undegraded":"2024-06-29T14:23:17.972485+0000","last_fullsized":"2024-06-29T14:23:17.972485+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:21.067086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192727+0000","last_change":"2024-06-29T14:23:16.195148+0000","last_active":"2024-06-29T14:23:17.192727+0000","last_peered":"2024-06-29T14:23:17.192727+0000","last_clean":"2024-06-29T14:23:17.192727+0000","last_became_active":"2024-06-29T14:23:16.194809+0000","last_became_peered":"2024-06-29T14:23:16.194809+0000","last_unstale":"2024-06-29T14:23:17.192727+0000","last_undegraded":"2024-06-29T14:23:17.192727+0000","last_fullsized":"2024-06-29T14:23:17.192727+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:39.380846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972355+0000","last_change":"2024-06-29T14:23:16.205741+0000","last_active":"2024-06-29T14:23:17.972355+0000","last_peered":"2024-06-29T14:23:17.972355+0000","last_clean":"2024-06-29T14:23:17.972355+0000","last_became_active":"2024-06-29T14:23:16.205351+0000","last_became_peered":"2024-06-29T14:23:16.205351+0000","last_unstale":"2024-06-29T14:23:17.972355+0000","last_undegraded":"2024-06-29T14:23:17.972355+0000","last_fullsized":"2024-06-29T14:23:17.972355+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:05.074083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192298+0000","last_change":"2024-06-29T14:23:16.211247+0000","last_active":"2024-06-29T14:23:17.192298+0000","last_peered":"2024-06-29T14:23:17.192298+0000","last_clean":"2024-06-29T14:23:17.192298+0000","last_became_active":"2024-06-29T14:23:16.210937+0000","last_became_peered":"2024-06-29T14:23:16.210937+0000","last_unstale":"2024-06-29T14:23:17.192298+0000","last_undegraded":"2024-06-29T14:23:17.192298+0000","last_fullsized":"2024-06-29T14:23:17.192298+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:02.929994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972492+0000","last_change":"2024-06-29T14:23:16.200185+0000","last_active":"2024-06-29T14:23:17.972492+0000","last_peered":"2024-06-29T14:23:17.972492+0000","last_clean":"2024-06-29T14:23:17.972492+0000","last_became_active":"2024-06-29T14:23:16.199847+0000","last_became_peered":"2024-06-29T14:23:16.199847+0000","last_unstale":"2024-06-29T14:23:17.972492+0000","last_undegraded":"2024-06-29T14:23:17.972492+0000","last_fullsized":"2024-06-29T14:23:17.972492+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:44:43.833435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191983+0000","last_change":"2024-06-29T14:23:16.189510+0000","last_active":"2024-06-29T14:23:17.191983+0000","last_peered":"2024-06-29T14:23:17.191983+0000","last_clean":"2024-06-29T14:23:17.191983+0000","last_became_active":"2024-06-29T14:23:16.189171+0000","last_became_peered":"2024-06-29T14:23:16.189171+0000","last_unstale":"2024-06-29T14:23:17.191983+0000","last_undegraded":"2024-06-29T14:23:17.191983+0000","last_fullsized":"2024-06-29T14:23:17.191983+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:25.573671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191939+0000","last_change":"2024-06-29T14:23:16.191577+0000","last_active":"2024-06-29T14:23:17.191939+0000","last_peered":"2024-06-29T14:23:17.191939+0000","last_clean":"2024-06-29T14:23:17.191939+0000","last_became_active":"2024-06-29T14:23:16.191245+0000","last_became_peered":"2024-06-29T14:23:16.191245+0000","last_unstale":"2024-06-29T14:23:17.191939+0000","last_undegraded":"2024-06-29T14:23:17.191939+0000","last_fullsized":"2024-06-29T14:23:17.191939+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:29.260203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192483+0000","last_change":"2024-06-29T14:23:16.210335+0000","last_active":"2024-06-29T14:23:17.192483+0000","last_peered":"2024-06-29T14:23:17.192483+0000","last_clean":"2024-06-29T14:23:17.192483+0000","last_became_active":"2024-06-29T14:23:16.210029+0000","last_became_peered":"2024-06-29T14:23:16.210029+0000","last_unstale":"2024-06-29T14:23:17.192483+0000","last_undegraded":"2024-06-29T14:23:17.192483+0000","last_fullsized":"2024-06-29T14:23:17.192483+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:56.733163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972869+0000","last_change":"2024-06-29T14:23:16.204276+0000","last_active":"2024-06-29T14:23:17.972869+0000","last_peered":"2024-06-29T14:23:17.972869+0000","last_clean":"2024-06-29T14:23:17.972869+0000","last_became_active":"2024-06-29T14:23:16.203937+0000","last_became_peered":"2024-06-29T14:23:16.203937+0000","last_unstale":"2024-06-29T14:23:17.972869+0000","last_undegraded":"2024-06-29T14:23:17.972869+0000","last_fullsized":"2024-06-29T14:23:17.972869+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:10.882976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192893+0000","last_change":"2024-06-29T14:23:16.205642+0000","last_active":"2024-06-29T14:23:17.192893+0000","last_peered":"2024-06-29T14:23:17.192893+0000","last_clean":"2024-06-29T14:23:17.192893+0000","last_became_active":"2024-06-29T14:23:16.205040+0000","last_became_peered":"2024-06-29T14:23:16.205040+0000","last_unstale":"2024-06-29T14:23:17.192893+0000","last_undegraded":"2024-06-29T14:23:17.192893+0000","last_fullsized":"2024-06-29T14:23:17.192893+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:43:10.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"40'4","reported_seq":18,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192842+0000","last_change":"2024-06-29T14:23:15.197029+0000","last_active":"2024-06-29T14:23:17.192842+0000","last_peered":"2024-06-29T14:23:17.192842+0000","last_clean":"2024-06-29T14:23:17.192842+0000","last_became_active":"2024-06-29T14:23:15.196636+0000","last_became_peered":"2024-06-29T14:23:15.196636+0000","last_unstale":"2024-06-29T14:23:17.192842+0000","last_undegraded":"2024-06-29T14:23:17.192842+0000","last_fullsized":"2024-06-29T14:23:17.192842+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:15:02.928561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.973072+0000","last_change":"2024-06-29T14:23:16.206697+0000","last_active":"2024-06-29T14:23:17.973072+0000","last_peered":"2024-06-29T14:23:17.973072+0000","last_clean":"2024-06-29T14:23:17.973072+0000","last_became_active":"2024-06-29T14:23:16.206325+0000","last_became_peered":"2024-06-29T14:23:16.206325+0000","last_unstale":"2024-06-29T14:23:17.973072+0000","last_undegraded":"2024-06-29T14:23:17.973072+0000","last_fullsized":"2024-06-29T14:23:17.973072+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:46.554739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971733+0000","last_change":"2024-06-29T14:23:15.204215+0000","last_active":"2024-06-29T14:23:17.971733+0000","last_peered":"2024-06-29T14:23:17.971733+0000","last_clean":"2024-06-29T14:23:17.971733+0000","last_became_active":"2024-06-29T14:23:15.203861+0000","last_became_peered":"2024-06-29T14:23:15.203861+0000","last_unstale":"2024-06-29T14:23:17.971733+0000","last_undegraded":"2024-06-29T14:23:17.971733+0000","last_fullsized":"2024-06-29T14:23:17.971733+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:15.271848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972355+0000","last_change":"2024-06-29T14:23:16.193555+0000","last_active":"2024-06-29T14:23:17.972355+0000","last_peered":"2024-06-29T14:23:17.972355+0000","last_clean":"2024-06-29T14:23:17.972355+0000","last_became_active":"2024-06-29T14:23:16.193197+0000","last_became_peered":"2024-06-29T14:23:16.193197+0000","last_unstale":"2024-06-29T14:23:17.972355+0000","last_undegraded":"2024-06-29T14:23:17.972355+0000","last_fullsized":"2024-06-29T14:23:17.972355+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:09:52.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972632+0000","last_change":"2024-06-29T14:23:15.205028+0000","last_active":"2024-06-29T14:23:17.972632+0000","last_peered":"2024-06-29T14:23:17.972632+0000","last_clean":"2024-06-29T14:23:17.972632+0000","last_became_active":"2024-06-29T14:23:15.204734+0000","last_became_peered":"2024-06-29T14:23:15.204734+0000","last_unstale":"2024-06-29T14:23:17.972632+0000","last_undegraded":"2024-06-29T14:23:17.972632+0000","last_fullsized":"2024-06-29T14:23:17.972632+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:12.502166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189499+0000","last_change":"2024-06-29T14:23:16.190648+0000","last_active":"2024-06-29T14:23:17.189499+0000","last_peered":"2024-06-29T14:23:17.189499+0000","last_clean":"2024-06-29T14:23:17.189499+0000","last_became_active":"2024-06-29T14:23:16.190331+0000","last_became_peered":"2024-06-29T14:23:16.190331+0000","last_unstale":"2024-06-29T14:23:17.189499+0000","last_undegraded":"2024-06-29T14:23:17.189499+0000","last_fullsized":"2024-06-29T14:23:17.189499+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:43.638548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972248+0000","last_change":"2024-06-29T14:23:15.195155+0000","last_active":"2024-06-29T14:23:17.972248+0000","last_peered":"2024-06-29T14:23:17.972248+0000","last_clean":"2024-06-29T14:23:17.972248+0000","last_became_active":"2024-06-29T14:23:15.194809+0000","last_became_peered":"2024-06-29T14:23:15.194809+0000","last_unstale":"2024-06-29T14:23:17.972248+0000","last_undegraded":"2024-06-29T14:23:17.972248+0000","last_fullsized":"2024-06-29T14:23:17.972248+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:48.024521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971120+0000","last_change":"2024-06-29T14:23:16.209386+0000","last_active":"2024-06-29T14:23:17.971120+0000","last_peered":"2024-06-29T14:23:17.971120+0000","last_clean":"2024-06-29T14:23:17.971120+0000","last_became_active":"2024-06-29T14:23:16.209011+0000","last_became_peered":"2024-06-29T14:23:16.209011+0000","last_unstale":"2024-06-29T14:23:17.971120+0000","last_undegraded":"2024-06-29T14:23:17.971120+0000","last_fullsized":"2024-06-29T14:23:17.971120+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:57.720170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971977+0000","last_change":"2024-06-29T14:23:15.201776+0000","last_active":"2024-06-29T14:23:17.971977+0000","last_peered":"2024-06-29T14:23:17.971977+0000","last_clean":"2024-06-29T14:23:17.971977+0000","last_became_active":"2024-06-29T14:23:15.201376+0000","last_became_peered":"2024-06-29T14:23:15.201376+0000","last_unstale":"2024-06-29T14:23:17.971977+0000","last_undegraded":"2024-06-29T14:23:17.971977+0000","last_fullsized":"2024-06-29T14:23:17.971977+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:07.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971988+0000","last_change":"2024-06-29T14:23:16.212639+0000","last_active":"2024-06-29T14:23:17.971988+0000","last_peered":"2024-06-29T14:23:17.971988+0000","last_clean":"2024-06-29T14:23:17.971988+0000","last_became_active":"2024-06-29T14:23:16.211460+0000","last_became_peered":"2024-06-29T14:23:16.211460+0000","last_unstale":"2024-06-29T14:23:17.971988+0000","last_undegraded":"2024-06-29T14:23:17.971988+0000","last_fullsized":"2024-06-29T14:23:17.971988+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:06.893440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191665+0000","last_change":"2024-06-29T14:23:15.203169+0000","last_active":"2024-06-29T14:23:17.191665+0000","last_peered":"2024-06-29T14:23:17.191665+0000","last_clean":"2024-06-29T14:23:17.191665+0000","last_became_active":"2024-06-29T14:23:15.202809+0000","last_became_peered":"2024-06-29T14:23:15.202809+0000","last_unstale":"2024-06-29T14:23:17.191665+0000","last_undegraded":"2024-06-29T14:23:17.191665+0000","last_fullsized":"2024-06-29T14:23:17.191665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:24.568080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971836+0000","last_change":"2024-06-29T14:23:16.201511+0000","last_active":"2024-06-29T14:23:17.971836+0000","last_peered":"2024-06-29T14:23:17.971836+0000","last_clean":"2024-06-29T14:23:17.971836+0000","last_became_active":"2024-06-29T14:23:16.200919+0000","last_became_peered":"2024-06-29T14:23:16.200919+0000","last_unstale":"2024-06-29T14:23:17.971836+0000","last_undegraded":"2024-06-29T14:23:17.971836+0000","last_fullsized":"2024-06-29T14:23:17.971836+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:25.539503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972074+0000","last_change":"2024-06-29T14:23:15.200889+0000","last_active":"2024-06-29T14:23:17.972074+0000","last_peered":"2024-06-29T14:23:17.972074+0000","last_clean":"2024-06-29T14:23:17.972074+0000","last_became_active":"2024-06-29T14:23:15.200521+0000","last_became_peered":"2024-06-29T14:23:15.200521+0000","last_unstale":"2024-06-29T14:23:17.972074+0000","last_undegraded":"2024-06-29T14:23:17.972074+0000","last_fullsized":"2024-06-29T14:23:17.972074+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189403+0000","last_change":"2024-06-29T14:23:16.207437+0000","last_active":"2024-06-29T14:23:17.189403+0000","last_peered":"2024-06-29T14:23:17.189403+0000","last_clean":"2024-06-29T14:23:17.189403+0000","last_became_active":"2024-06-29T14:23:16.207056+0000","last_became_peered":"2024-06-29T14:23:16.207056+0000","last_unstale":"2024-06-29T14:23:17.189403+0000","last_undegraded":"2024-06-29T14:23:17.189403+0000","last_fullsized":"2024-06-29T14:23:17.189403+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:06.893440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971336+0000","last_change":"2024-06-29T14:23:15.199990+0000","last_active":"2024-06-29T14:23:17.971336+0000","last_peered":"2024-06-29T14:23:17.971336+0000","last_clean":"2024-06-29T14:23:17.971336+0000","last_became_active":"2024-06-29T14:23:15.199697+0000","last_became_peered":"2024-06-29T14:23:15.199697+0000","last_unstale":"2024-06-29T14:23:17.971336+0000","last_undegraded":"2024-06-29T14:23:17.971336+0000","last_fullsized":"2024-06-29T14:23:17.971336+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:03.756929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.973018+0000","last_change":"2024-06-29T14:23:16.197089+0000","last_active":"2024-06-29T14:23:17.973018+0000","last_peered":"2024-06-29T14:23:17.973018+0000","last_clean":"2024-06-29T14:23:17.973018+0000","last_became_active":"2024-06-29T14:23:16.196730+0000","last_became_peered":"2024-06-29T14:23:16.196730+0000","last_unstale":"2024-06-29T14:23:17.973018+0000","last_undegraded":"2024-06-29T14:23:17.973018+0000","last_fullsized":"2024-06-29T14:23:17.973018+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:55.411612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189657+0000","last_change":"2024-06-29T14:23:15.189674+0000","last_active":"2024-06-29T14:23:17.189657+0000","last_peered":"2024-06-29T14:23:17.189657+0000","last_clean":"2024-06-29T14:23:17.189657+0000","last_became_active":"2024-06-29T14:23:15.189326+0000","last_became_peered":"2024-06-29T14:23:15.189326+0000","last_unstale":"2024-06-29T14:23:17.189657+0000","last_undegraded":"2024-06-29T14:23:17.189657+0000","last_fullsized":"2024-06-29T14:23:17.189657+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:09.396994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191913+0000","last_change":"2024-06-29T14:23:16.209462+0000","last_active":"2024-06-29T14:23:17.191913+0000","last_peered":"2024-06-29T14:23:17.191913+0000","last_clean":"2024-06-29T14:23:17.191913+0000","last_became_active":"2024-06-29T14:23:16.209117+0000","last_became_peered":"2024-06-29T14:23:16.209117+0000","last_unstale":"2024-06-29T14:23:17.191913+0000","last_undegraded":"2024-06-29T14:23:17.191913+0000","last_fullsized":"2024-06-29T14:23:17.191913+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:13:09.963558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972810+0000","last_change":"2024-06-29T14:23:15.205426+0000","last_active":"2024-06-29T14:23:17.972810+0000","last_peered":"2024-06-29T14:23:17.972810+0000","last_clean":"2024-06-29T14:23:17.972810+0000","last_became_active":"2024-06-29T14:23:15.204970+0000","last_became_peered":"2024-06-29T14:23:15.204970+0000","last_unstale":"2024-06-29T14:23:17.972810+0000","last_undegraded":"2024-06-29T14:23:17.972810+0000","last_fullsized":"2024-06-29T14:23:17.972810+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:54.665868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971608+0000","last_change":"2024-06-29T14:23:16.208578+0000","last_active":"2024-06-29T14:23:17.971608+0000","last_peered":"2024-06-29T14:23:17.971608+0000","last_clean":"2024-06-29T14:23:17.971608+0000","last_became_active":"2024-06-29T14:23:16.208189+0000","last_became_peered":"2024-06-29T14:23:16.208189+0000","last_unstale":"2024-06-29T14:23:17.971608+0000","last_undegraded":"2024-06-29T14:23:17.971608+0000","last_fullsized":"2024-06-29T14:23:17.971608+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:46.554739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"40'3","reported_seq":17,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972679+0000","last_change":"2024-06-29T14:23:15.202041+0000","last_active":"2024-06-29T14:23:17.972679+0000","last_peered":"2024-06-29T14:23:17.972679+0000","last_clean":"2024-06-29T14:23:17.972679+0000","last_became_active":"2024-06-29T14:23:15.201722+0000","last_became_peered":"2024-06-29T14:23:15.201722+0000","last_unstale":"2024-06-29T14:23:17.972679+0000","last_undegraded":"2024-06-29T14:23:17.972679+0000","last_fullsized":"2024-06-29T14:23:17.972679+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:28.995567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972654+0000","last_change":"2024-06-29T14:23:16.202659+0000","last_active":"2024-06-29T14:23:17.972654+0000","last_peered":"2024-06-29T14:23:17.972654+0000","last_clean":"2024-06-29T14:23:17.972654+0000","last_became_active":"2024-06-29T14:23:16.202249+0000","last_became_peered":"2024-06-29T14:23:16.202249+0000","last_unstale":"2024-06-29T14:23:17.972654+0000","last_undegraded":"2024-06-29T14:23:17.972654+0000","last_fullsized":"2024-06-29T14:23:17.972654+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:52:54.351905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"40'6","reported_seq":20,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189598+0000","last_change":"2024-06-29T14:23:15.199353+0000","last_active":"2024-06-29T14:23:17.189598+0000","last_peered":"2024-06-29T14:23:17.189598+0000","last_clean":"2024-06-29T14:23:17.189598+0000","last_became_active":"2024-06-29T14:23:15.198339+0000","last_became_peered":"2024-06-29T14:23:15.198339+0000","last_unstale":"2024-06-29T14:23:17.189598+0000","last_undegraded":"2024-06-29T14:23:17.189598+0000","last_fullsized":"2024-06-29T14:23:17.189598+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:39.718146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971167+0000","last_change":"2024-06-29T14:23:16.209778+0000","last_active":"2024-06-29T14:23:17.971167+0000","last_peered":"2024-06-29T14:23:17.971167+0000","last_clean":"2024-06-29T14:23:17.971167+0000","last_became_active":"2024-06-29T14:23:16.209443+0000","last_became_peered":"2024-06-29T14:23:16.209443+0000","last_unstale":"2024-06-29T14:23:17.971167+0000","last_undegraded":"2024-06-29T14:23:17.971167+0000","last_fullsized":"2024-06-29T14:23:17.971167+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:12.662244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189562+0000","last_change":"2024-06-29T14:23:15.200701+0000","last_active":"2024-06-29T14:23:17.189562+0000","last_peered":"2024-06-29T14:23:17.189562+0000","last_clean":"2024-06-29T14:23:17.189562+0000","last_became_active":"2024-06-29T14:23:15.200338+0000","last_became_peered":"2024-06-29T14:23:15.200338+0000","last_unstale":"2024-06-29T14:23:17.189562+0000","last_undegraded":"2024-06-29T14:23:17.189562+0000","last_fullsized":"2024-06-29T14:23:17.189562+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:54.665868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189197+0000","last_change":"2024-06-29T14:23:16.196448+0000","last_active":"2024-06-29T14:23:17.189197+0000","last_peered":"2024-06-29T14:23:17.189197+0000","last_clean":"2024-06-29T14:23:17.189197+0000","last_became_active":"2024-06-29T14:23:16.195006+0000","last_became_peered":"2024-06-29T14:23:16.195006+0000","last_unstale":"2024-06-29T14:23:17.189197+0000","last_undegraded":"2024-06-29T14:23:17.189197+0000","last_fullsized":"2024-06-29T14:23:17.189197+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:14:42.425008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971669+0000","last_change":"2024-06-29T14:23:15.199955+0000","last_active":"2024-06-29T14:23:17.971669+0000","last_peered":"2024-06-29T14:23:17.971669+0000","last_clean":"2024-06-29T14:23:17.971669+0000","last_became_active":"2024-06-29T14:23:15.199619+0000","last_became_peered":"2024-06-29T14:23:15.199619+0000","last_unstale":"2024-06-29T14:23:17.971669+0000","last_undegraded":"2024-06-29T14:23:17.971669+0000","last_fullsized":"2024-06-29T14:23:17.971669+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:04.589609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191906+0000","last_change":"2024-06-29T14:23:16.211883+0000","last_active":"2024-06-29T14:23:17.191906+0000","last_peered":"2024-06-29T14:23:17.191906+0000","last_clean":"2024-06-29T14:23:17.191906+0000","last_became_active":"2024-06-29T14:23:16.211577+0000","last_became_peered":"2024-06-29T14:23:16.211577+0000","last_unstale":"2024-06-29T14:23:17.191906+0000","last_undegraded":"2024-06-29T14:23:17.191906+0000","last_fullsized":"2024-06-29T14:23:17.191906+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:39.476522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191955+0000","last_change":"2024-06-29T14:23:15.196439+0000","last_active":"2024-06-29T14:23:17.191955+0000","last_peered":"2024-06-29T14:23:17.191955+0000","last_clean":"2024-06-29T14:23:17.191955+0000","last_became_active":"2024-06-29T14:23:15.196116+0000","last_became_peered":"2024-06-29T14:23:15.196116+0000","last_unstale":"2024-06-29T14:23:17.191955+0000","last_undegraded":"2024-06-29T14:23:17.191955+0000","last_fullsized":"2024-06-29T14:23:17.191955+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:27.766774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189405+0000","last_change":"2024-06-29T14:23:16.200847+0000","last_active":"2024-06-29T14:23:17.189405+0000","last_peered":"2024-06-29T14:23:17.189405+0000","last_clean":"2024-06-29T14:23:17.189405+0000","last_became_active":"2024-06-29T14:23:16.200264+0000","last_became_peered":"2024-06-29T14:23:16.200264+0000","last_unstale":"2024-06-29T14:23:17.189405+0000","last_undegraded":"2024-06-29T14:23:17.189405+0000","last_fullsized":"2024-06-29T14:23:17.189405+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:28.205354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"40'8","reported_seq":22,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192728+0000","last_change":"2024-06-29T14:23:15.200739+0000","last_active":"2024-06-29T14:23:17.192728+0000","last_peered":"2024-06-29T14:23:17.192728+0000","last_clean":"2024-06-29T14:23:17.192728+0000","last_became_active":"2024-06-29T14:23:15.200401+0000","last_became_peered":"2024-06-29T14:23:15.200401+0000","last_unstale":"2024-06-29T14:23:17.192728+0000","last_undegraded":"2024-06-29T14:23:17.192728+0000","last_fullsized":"2024-06-29T14:23:17.192728+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:28.254612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189474+0000","last_change":"2024-06-29T14:23:16.188970+0000","last_active":"2024-06-29T14:23:17.189474+0000","last_peered":"2024-06-29T14:23:17.189474+0000","last_clean":"2024-06-29T14:23:17.189474+0000","last_became_active":"2024-06-29T14:23:16.188588+0000","last_became_peered":"2024-06-29T14:23:16.188588+0000","last_unstale":"2024-06-29T14:23:17.189474+0000","last_undegraded":"2024-06-29T14:23:17.189474+0000","last_fullsized":"2024-06-29T14:23:17.189474+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:05:44.443149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972725+0000","last_change":"2024-06-29T14:23:15.199255+0000","last_active":"2024-06-29T14:23:17.972725+0000","last_peered":"2024-06-29T14:23:17.972725+0000","last_clean":"2024-06-29T14:23:17.972725+0000","last_became_active":"2024-06-29T14:23:15.198806+0000","last_became_peered":"2024-06-29T14:23:15.198806+0000","last_unstale":"2024-06-29T14:23:17.972725+0000","last_undegraded":"2024-06-29T14:23:17.972725+0000","last_fullsized":"2024-06-29T14:23:17.972725+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:39.718146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972659+0000","last_change":"2024-06-29T14:23:16.210090+0000","last_active":"2024-06-29T14:23:17.972659+0000","last_peered":"2024-06-29T14:23:17.972659+0000","last_clean":"2024-06-29T14:23:17.972659+0000","last_became_active":"2024-06-29T14:23:16.209664+0000","last_became_peered":"2024-06-29T14:23:16.209664+0000","last_unstale":"2024-06-29T14:23:17.972659+0000","last_undegraded":"2024-06-29T14:23:17.972659+0000","last_fullsized":"2024-06-29T14:23:17.972659+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:11.338016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972475+0000","last_change":"2024-06-29T14:23:15.200790+0000","last_active":"2024-06-29T14:23:17.972475+0000","last_peered":"2024-06-29T14:23:17.972475+0000","last_clean":"2024-06-29T14:23:17.972475+0000","last_became_active":"2024-06-29T14:23:15.200452+0000","last_became_peered":"2024-06-29T14:23:15.200452+0000","last_unstale":"2024-06-29T14:23:17.972475+0000","last_undegraded":"2024-06-29T14:23:17.972475+0000","last_fullsized":"2024-06-29T14:23:17.972475+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:12.157450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972305+0000","last_change":"2024-06-29T14:23:16.204457+0000","last_active":"2024-06-29T14:23:17.972305+0000","last_peered":"2024-06-29T14:23:17.972305+0000","last_clean":"2024-06-29T14:23:17.972305+0000","last_became_active":"2024-06-29T14:23:16.204087+0000","last_became_peered":"2024-06-29T14:23:16.204087+0000","last_unstale":"2024-06-29T14:23:17.972305+0000","last_undegraded":"2024-06-29T14:23:17.972305+0000","last_fullsized":"2024-06-29T14:23:17.972305+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:54.822759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"40'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972729+0000","last_change":"2024-06-29T14:23:15.195356+0000","last_active":"2024-06-29T14:23:17.972729+0000","last_peered":"2024-06-29T14:23:17.972729+0000","last_clean":"2024-06-29T14:23:17.972729+0000","last_became_active":"2024-06-29T14:23:15.195037+0000","last_became_peered":"2024-06-29T14:23:15.195037+0000","last_unstale":"2024-06-29T14:23:17.972729+0000","last_undegraded":"2024-06-29T14:23:17.972729+0000","last_fullsized":"2024-06-29T14:23:17.972729+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:32.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971916+0000","last_change":"2024-06-29T14:23:16.197431+0000","last_active":"2024-06-29T14:23:17.971916+0000","last_peered":"2024-06-29T14:23:17.971916+0000","last_clean":"2024-06-29T14:23:17.971916+0000","last_became_active":"2024-06-29T14:23:16.197091+0000","last_became_peered":"2024-06-29T14:23:16.197091+0000","last_unstale":"2024-06-29T14:23:17.971916+0000","last_undegraded":"2024-06-29T14:23:17.971916+0000","last_fullsized":"2024-06-29T14:23:17.971916+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:46.040585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972009+0000","last_change":"2024-06-29T14:23:15.203782+0000","last_active":"2024-06-29T14:23:17.972009+0000","last_peered":"2024-06-29T14:23:17.972009+0000","last_clean":"2024-06-29T14:23:17.972009+0000","last_became_active":"2024-06-29T14:23:15.203451+0000","last_became_peered":"2024-06-29T14:23:15.203451+0000","last_unstale":"2024-06-29T14:23:17.972009+0000","last_undegraded":"2024-06-29T14:23:17.972009+0000","last_fullsized":"2024-06-29T14:23:17.972009+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:18.072309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192298+0000","last_change":"2024-06-29T14:23:16.212117+0000","last_active":"2024-06-29T14:23:17.192298+0000","last_peered":"2024-06-29T14:23:17.192298+0000","last_clean":"2024-06-29T14:23:17.192298+0000","last_became_active":"2024-06-29T14:23:16.211660+0000","last_became_peered":"2024-06-29T14:23:16.211660+0000","last_unstale":"2024-06-29T14:23:17.192298+0000","last_undegraded":"2024-06-29T14:23:17.192298+0000","last_fullsized":"2024-06-29T14:23:17.192298+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:56.501064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972722+0000","last_change":"2024-06-29T14:23:15.201493+0000","last_active":"2024-06-29T14:23:17.972722+0000","last_peered":"2024-06-29T14:23:17.972722+0000","last_clean":"2024-06-29T14:23:17.972722+0000","last_became_active":"2024-06-29T14:23:15.201150+0000","last_became_peered":"2024-06-29T14:23:15.201150+0000","last_unstale":"2024-06-29T14:23:17.972722+0000","last_undegraded":"2024-06-29T14:23:17.972722+0000","last_fullsized":"2024-06-29T14:23:17.972722+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:00.323663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971299+0000","last_change":"2024-06-29T14:23:16.202653+0000","last_active":"2024-06-29T14:23:17.971299+0000","last_peered":"2024-06-29T14:23:17.971299+0000","last_clean":"2024-06-29T14:23:17.971299+0000","last_became_active":"2024-06-29T14:23:16.202279+0000","last_became_peered":"2024-06-29T14:23:16.202279+0000","last_unstale":"2024-06-29T14:23:17.971299+0000","last_undegraded":"2024-06-29T14:23:17.971299+0000","last_fullsized":"2024-06-29T14:23:17.971299+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:52.309099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971710+0000","last_change":"2024-06-29T14:23:15.196760+0000","last_active":"2024-06-29T14:23:17.971710+0000","last_peered":"2024-06-29T14:23:17.971710+0000","last_clean":"2024-06-29T14:23:17.971710+0000","last_became_active":"2024-06-29T14:23:15.196278+0000","last_became_peered":"2024-06-29T14:23:15.196278+0000","last_unstale":"2024-06-29T14:23:17.971710+0000","last_undegraded":"2024-06-29T14:23:17.971710+0000","last_fullsized":"2024-06-29T14:23:17.971710+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:32:01.571661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972354+0000","last_change":"2024-06-29T14:23:16.200152+0000","last_active":"2024-06-29T14:23:17.972354+0000","last_peered":"2024-06-29T14:23:17.972354+0000","last_clean":"2024-06-29T14:23:17.972354+0000","last_became_active":"2024-06-29T14:23:16.199828+0000","last_became_peered":"2024-06-29T14:23:16.199828+0000","last_unstale":"2024-06-29T14:23:17.972354+0000","last_undegraded":"2024-06-29T14:23:17.972354+0000","last_fullsized":"2024-06-29T14:23:17.972354+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:43.638548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192608+0000","last_change":"2024-06-29T14:23:15.199837+0000","last_active":"2024-06-29T14:23:17.192608+0000","last_peered":"2024-06-29T14:23:17.192608+0000","last_clean":"2024-06-29T14:23:17.192608+0000","last_became_active":"2024-06-29T14:23:15.199539+0000","last_became_peered":"2024-06-29T14:23:15.199539+0000","last_unstale":"2024-06-29T14:23:17.192608+0000","last_undegraded":"2024-06-29T14:23:17.192608+0000","last_fullsized":"2024-06-29T14:23:17.192608+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:32.149209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189570+0000","last_change":"2024-06-29T14:23:16.211956+0000","last_active":"2024-06-29T14:23:17.189570+0000","last_peered":"2024-06-29T14:23:17.189570+0000","last_clean":"2024-06-29T14:23:17.189570+0000","last_became_active":"2024-06-29T14:23:16.211649+0000","last_became_peered":"2024-06-29T14:23:16.211649+0000","last_unstale":"2024-06-29T14:23:17.189570+0000","last_undegraded":"2024-06-29T14:23:17.189570+0000","last_fullsized":"2024-06-29T14:23:17.189570+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:44:43.833435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972618+0000","last_change":"2024-06-29T14:23:15.201805+0000","last_active":"2024-06-29T14:23:17.972618+0000","last_peered":"2024-06-29T14:23:17.972618+0000","last_clean":"2024-06-29T14:23:17.972618+0000","last_became_active":"2024-06-29T14:23:15.200924+0000","last_became_peered":"2024-06-29T14:23:15.200924+0000","last_unstale":"2024-06-29T14:23:17.972618+0000","last_undegraded":"2024-06-29T14:23:17.972618+0000","last_fullsized":"2024-06-29T14:23:17.972618+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:18.651137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971892+0000","last_change":"2024-06-29T14:23:16.205768+0000","last_active":"2024-06-29T14:23:17.971892+0000","last_peered":"2024-06-29T14:23:17.971892+0000","last_clean":"2024-06-29T14:23:17.971892+0000","last_became_active":"2024-06-29T14:23:16.205382+0000","last_became_peered":"2024-06-29T14:23:16.205382+0000","last_unstale":"2024-06-29T14:23:17.971892+0000","last_undegraded":"2024-06-29T14:23:17.971892+0000","last_fullsized":"2024-06-29T14:23:17.971892+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:52.309099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"40'2","reported_seq":16,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972843+0000","last_change":"2024-06-29T14:23:15.203975+0000","last_active":"2024-06-29T14:23:17.972843+0000","last_peered":"2024-06-29T14:23:17.972843+0000","last_clean":"2024-06-29T14:23:17.972843+0000","last_became_active":"2024-06-29T14:23:15.203652+0000","last_became_peered":"2024-06-29T14:23:15.203652+0000","last_unstale":"2024-06-29T14:23:17.972843+0000","last_undegraded":"2024-06-29T14:23:17.972843+0000","last_fullsized":"2024-06-29T14:23:17.972843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:09.396994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192108+0000","last_change":"2024-06-29T14:23:16.210252+0000","last_active":"2024-06-29T14:23:17.192108+0000","last_peered":"2024-06-29T14:23:17.192108+0000","last_clean":"2024-06-29T14:23:17.192108+0000","last_became_active":"2024-06-29T14:23:16.209936+0000","last_became_peered":"2024-06-29T14:23:16.209936+0000","last_unstale":"2024-06-29T14:23:17.192108+0000","last_undegraded":"2024-06-29T14:23:17.192108+0000","last_fullsized":"2024-06-29T14:23:17.192108+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:36.806180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"41'1","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:18.201843+0000","last_change":"2024-06-29T14:23:15.199068+0000","last_active":"2024-06-29T14:23:18.201843+0000","last_peered":"2024-06-29T14:23:18.201843+0000","last_clean":"2024-06-29T14:23:18.201843+0000","last_became_active":"2024-06-29T14:23:15.198714+0000","last_became_peered":"2024-06-29T14:23:15.198714+0000","last_unstale":"2024-06-29T14:23:18.201843+0000","last_undegraded":"2024-06-29T14:23:18.201843+0000","last_fullsized":"2024-06-29T14:23:18.201843+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:11.722673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189505+0000","last_change":"2024-06-29T14:23:16.194459+0000","last_active":"2024-06-29T14:23:17.189505+0000","last_peered":"2024-06-29T14:23:17.189505+0000","last_clean":"2024-06-29T14:23:17.189505+0000","last_became_active":"2024-06-29T14:23:16.194084+0000","last_became_peered":"2024-06-29T14:23:16.194084+0000","last_unstale":"2024-06-29T14:23:17.189505+0000","last_undegraded":"2024-06-29T14:23:17.189505+0000","last_fullsized":"2024-06-29T14:23:17.189505+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:51.801025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972288+0000","last_change":"2024-06-29T14:23:15.199199+0000","last_active":"2024-06-29T14:23:17.972288+0000","last_peered":"2024-06-29T14:23:17.972288+0000","last_clean":"2024-06-29T14:23:17.972288+0000","last_became_active":"2024-06-29T14:23:15.198802+0000","last_became_peered":"2024-06-29T14:23:15.198802+0000","last_unstale":"2024-06-29T14:23:17.972288+0000","last_undegraded":"2024-06-29T14:23:17.972288+0000","last_fullsized":"2024-06-29T14:23:17.972288+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:41.669608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971366+0000","last_change":"2024-06-29T14:23:16.200965+0000","last_active":"2024-06-29T14:23:17.971366+0000","last_peered":"2024-06-29T14:23:17.971366+0000","last_clean":"2024-06-29T14:23:17.971366+0000","last_became_active":"2024-06-29T14:23:16.200584+0000","last_became_peered":"2024-06-29T14:23:16.200584+0000","last_unstale":"2024-06-29T14:23:17.971366+0000","last_undegraded":"2024-06-29T14:23:17.971366+0000","last_fullsized":"2024-06-29T14:23:17.971366+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:05.074083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972973+0000","last_change":"2024-06-29T14:23:15.203329+0000","last_active":"2024-06-29T14:23:17.972973+0000","last_peered":"2024-06-29T14:23:17.972973+0000","last_clean":"2024-06-29T14:23:17.972973+0000","last_became_active":"2024-06-29T14:23:15.202429+0000","last_became_peered":"2024-06-29T14:23:15.202429+0000","last_unstale":"2024-06-29T14:23:17.972973+0000","last_undegraded":"2024-06-29T14:23:17.972973+0000","last_fullsized":"2024-06-29T14:23:17.972973+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:26.898686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971946+0000","last_change":"2024-06-29T14:23:16.190685+0000","last_active":"2024-06-29T14:23:17.971946+0000","last_peered":"2024-06-29T14:23:17.971946+0000","last_clean":"2024-06-29T14:23:17.971946+0000","last_became_active":"2024-06-29T14:23:16.190310+0000","last_became_peered":"2024-06-29T14:23:16.190310+0000","last_unstale":"2024-06-29T14:23:17.971946+0000","last_undegraded":"2024-06-29T14:23:17.971946+0000","last_fullsized":"2024-06-29T14:23:17.971946+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:46.325166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192125+0000","last_change":"2024-06-29T14:23:15.200968+0000","last_active":"2024-06-29T14:23:17.192125+0000","last_peered":"2024-06-29T14:23:17.192125+0000","last_clean":"2024-06-29T14:23:17.192125+0000","last_became_active":"2024-06-29T14:23:15.200573+0000","last_became_peered":"2024-06-29T14:23:15.200573+0000","last_unstale":"2024-06-29T14:23:17.192125+0000","last_undegraded":"2024-06-29T14:23:17.192125+0000","last_fullsized":"2024-06-29T14:23:17.192125+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:54.406021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971610+0000","last_change":"2024-06-29T14:23:16.200262+0000","last_active":"2024-06-29T14:23:17.971610+0000","last_peered":"2024-06-29T14:23:17.971610+0000","last_clean":"2024-06-29T14:23:17.971610+0000","last_became_active":"2024-06-29T14:23:16.199984+0000","last_became_peered":"2024-06-29T14:23:16.199984+0000","last_unstale":"2024-06-29T14:23:17.971610+0000","last_undegraded":"2024-06-29T14:23:17.971610+0000","last_fullsized":"2024-06-29T14:23:17.971610+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:52:50.336628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.971381+0000","last_change":"2024-06-29T14:23:15.197575+0000","last_active":"2024-06-29T14:23:17.971381+0000","last_peered":"2024-06-29T14:23:17.971381+0000","last_clean":"2024-06-29T14:23:17.971381+0000","last_became_active":"2024-06-29T14:23:15.197267+0000","last_became_peered":"2024-06-29T14:23:15.197267+0000","last_unstale":"2024-06-29T14:23:17.971381+0000","last_undegraded":"2024-06-29T14:23:17.971381+0000","last_fullsized":"2024-06-29T14:23:17.971381+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:35.221447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.191930+0000","last_change":"2024-06-29T14:23:16.207769+0000","last_active":"2024-06-29T14:23:17.191930+0000","last_peered":"2024-06-29T14:23:17.191930+0000","last_clean":"2024-06-29T14:23:17.191930+0000","last_became_active":"2024-06-29T14:23:16.207431+0000","last_became_peered":"2024-06-29T14:23:16.207431+0000","last_unstale":"2024-06-29T14:23:17.191930+0000","last_undegraded":"2024-06-29T14:23:17.191930+0000","last_fullsized":"2024-06-29T14:23:17.191930+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:32.820240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.189093+0000","last_change":"2024-06-29T14:23:15.192161+0000","last_active":"2024-06-29T14:23:17.189093+0000","last_peered":"2024-06-29T14:23:17.189093+0000","last_clean":"2024-06-29T14:23:17.189093+0000","last_became_active":"2024-06-29T14:23:15.191810+0000","last_became_peered":"2024-06-29T14:23:15.191810+0000","last_unstale":"2024-06-29T14:23:17.189093+0000","last_undegraded":"2024-06-29T14:23:17.189093+0000","last_fullsized":"2024-06-29T14:23:17.189093+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:28.995567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.972582+0000","last_change":"2024-06-29T14:23:16.202651+0000","last_active":"2024-06-29T14:23:17.972582+0000","last_peered":"2024-06-29T14:23:17.972582+0000","last_clean":"2024-06-29T14:23:17.972582+0000","last_became_active":"2024-06-29T14:23:16.202214+0000","last_became_peered":"2024-06-29T14:23:16.202214+0000","last_unstale":"2024-06-29T14:23:17.972582+0000","last_undegraded":"2024-06-29T14:23:17.972582+0000","last_fullsized":"2024-06-29T14:23:17.972582+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:15.170617+0000","last_clean_scrub_stamp":"2024-06-29T14:23:15.170617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:44.745370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192983+0000","last_change":"2024-06-29T14:23:15.198041+0000","last_active":"2024-06-29T14:23:17.192983+0000","last_peered":"2024-06-29T14:23:17.192983+0000","last_clean":"2024-06-29T14:23:17.192983+0000","last_became_active":"2024-06-29T14:23:15.197662+0000","last_became_peered":"2024-06-29T14:23:15.197662+0000","last_unstale":"2024-06-29T14:23:17.192983+0000","last_undegraded":"2024-06-29T14:23:17.192983+0000","last_fullsized":"2024-06-29T14:23:17.192983+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:14.165027+0000","last_clean_scrub_stamp":"2024-06-29T14:23:14.165027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:01.924403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":146,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-06-29T14:23:17.192855+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:23:17.192855+0000","last_peered":"2024-06-29T14:23:17.192855+0000","last_clean":"2024-06-29T14:23:17.192855+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:23:17.192855+0000","last_undegraded":"2024-06-29T14:23:17.192855+0000","last_fullsized":"2024-06-29T14:23:17.192855+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":483328,"data_stored":204581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509463,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":471040,"data_stored":203025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509465,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1064960,"data_stored":793337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1077248,"data_stored":794985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":471040,"data_stored":203517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":475136,"data_stored":203047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":203539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":203005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:23:25.178 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:23:25.179 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:23:25.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.339+0000 7f1f615be640 1 -- 172.21.15.59:0/179868120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0ce090 msgr2=0x7f1f5c0b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.339+0000 7f1f615be640 1 --2- 172.21.15.59:0/179868120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0ce090 0x7f1f5c0b2e60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c005fd0 tx=0x7f1f4c02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.340+0000 7f1f615be640 1 -- 172.21.15.59:0/179868120 shutdown_connections 2024-06-29T14:23:25.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.340+0000 7f1f615be640 1 --2- 172.21.15.59:0/179868120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f5c0b4c20 0x7f1f5c0b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.340+0000 7f1f615be640 1 --2- 172.21.15.59:0/179868120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0ce090 0x7f1f5c0b2e60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.340+0000 7f1f615be640 1 -- 172.21.15.59:0/179868120 >> 172.21.15.59:0/179868120 conn(0x7f1f5c0d4070 msgr2=0x7f1f5c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.340+0000 7f1f615be640 1 -- 172.21.15.59:0/179868120 shutdown_connections 2024-06-29T14:23:25.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.340+0000 7f1f615be640 1 -- 172.21.15.59:0/179868120 wait complete. 2024-06-29T14:23:25.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.341+0000 7f1f615be640 1 Processor -- start 2024-06-29T14:23:25.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.342+0000 7f1f615be640 1 -- start start 2024-06-29T14:23:25.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f615be640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f5c0ce090 0x7f1f5c14d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f615be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0cefb0 0x7f1f5c0c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f615be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c0cc0e0 0x7f1f5c155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f615be640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c0b8880 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f615be640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5c0b8700 con 0x7f1f5c0cc0e0 2024-06-29T14:23:25.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f615be640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5c0b8a00 con 0x7f1f5c0ce090 2024-06-29T14:23:25.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0cefb0 0x7f1f5c0c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0cefb0 0x7f1f5c0c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59564/0 (socket says 172.21.15.59:59564) 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5a7fc640 1 -- 172.21.15.59:0/2168120061 learned_addr learned my addr 172.21.15.59:0/2168120061 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5affd640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f5c0ce090 0x7f1f5c14d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5b7fe640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c0cc0e0 0x7f1f5c155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5a7fc640 1 -- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f5c0ce090 msgr2=0x7f1f5c14d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5a7fc640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f5c0ce090 0x7f1f5c14d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5a7fc640 1 -- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c0cc0e0 msgr2=0x7f1f5c155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5a7fc640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c0cc0e0 0x7f1f5c155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.343+0000 7f1f5a7fc640 1 -- 172.21.15.59:0/2168120061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5c14d5d0 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.344+0000 7f1f5a7fc640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0cefb0 0x7f1f5c0c8420 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000b3f0 tx=0x7f1f5000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.344+0000 7f1f437fe640 1 -- 172.21.15.59:0/2168120061 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50015020 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.344+0000 7f1f437fe640 1 -- 172.21.15.59:0/2168120061 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f50011920 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.344+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5c158810 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.344+0000 7f1f437fe640 1 -- 172.21.15.59:0/2168120061 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50011d70 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.344+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f5c0c8de0 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.345+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f5c0b8f90 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.346+0000 7f1f437fe640 1 -- 172.21.15.59:0/2168120061 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1f50018020 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.346+0000 7f1f437fe640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1f48076300 0x7f1f48078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.346+0000 7f1f437fe640 1 -- 172.21.15.59:0/2168120061 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f1f500974b0 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.347+0000 7f1f5affd640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1f48076300 0x7f1f48078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.347+0000 7f1f5affd640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1f48076300 0x7f1f48078810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c033000 tx=0x7f1f4c004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.351+0000 7f1f437fe640 1 -- 172.21.15.59:0/2168120061 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1f5001f030 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.617+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f1f5c15d0c0 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.620+0000 7f1f437fe640 1 -- 172.21.15.59:0/2168120061 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f1f5c14d5d0 con 0x7f1f5c0cefb0 2024-06-29T14:23:25.618 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:23:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:23:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T14:23:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T14:23:25.618 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:23:25.618 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T14:23:25.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1f48076300 msgr2=0x7f1f48078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1f48076300 0x7f1f48078810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c033000 tx=0x7f1f4c004730 comp rx=0 tx=0).stop 2024-06-29T14:23:25.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0cefb0 msgr2=0x7f1f5c0c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0cefb0 0x7f1f5c0c8420 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000b3f0 tx=0x7f1f5000b8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 shutdown_connections 2024-06-29T14:23:25.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f1f48076300 0x7f1f48078810 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f5c0cc0e0 0x7f1f5c155830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f5c0cefb0 0x7f1f5c0c8420 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 --2- 172.21.15.59:0/2168120061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f5c0ce090 0x7f1f5c14d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 >> 172.21.15.59:0/2168120061 conn(0x7f1f5c0d4070 msgr2=0x7f1f5c161120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 shutdown_connections 2024-06-29T14:23:25.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.622+0000 7f1f615be640 1 -- 172.21.15.59:0/2168120061 wait complete. 2024-06-29T14:23:25.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:23:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.752+0000 7fa84c288640 1 -- 172.21.15.59:0/2326966690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa84415ceb0 msgr2=0x7fa8440b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.752+0000 7fa84c288640 1 --2- 172.21.15.59:0/2326966690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa84415ceb0 0x7fa8440b56d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa840009830 tx=0x7fa84002a480 comp rx=0 tx=0).stop 2024-06-29T14:23:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.752+0000 7fa84c288640 1 -- 172.21.15.59:0/2326966690 shutdown_connections 2024-06-29T14:23:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.752+0000 7fa84c288640 1 --2- 172.21.15.59:0/2326966690 >> v2:172.21.15.153:3300/0 conn(0x7fa844141bd0 0x7fa84413edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.752+0000 7fa84c288640 1 --2- 172.21.15.59:0/2326966690 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa84415ceb0 0x7fa8440b56d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.752+0000 7fa84c288640 1 -- 172.21.15.59:0/2326966690 >> 172.21.15.59:0/2326966690 conn(0x7fa8440b6fb0 msgr2=0x7fa8440b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.752+0000 7fa84c288640 1 -- 172.21.15.59:0/2326966690 shutdown_connections 2024-06-29T14:23:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.752+0000 7fa84c288640 1 -- 172.21.15.59:0/2326966690 wait complete. 2024-06-29T14:23:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.754+0000 7fa84c288640 1 Processor -- start 2024-06-29T14:23:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.754+0000 7fa84c288640 1 -- start start 2024-06-29T14:23:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.755+0000 7fa84c288640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa84413e9b0 0x7fa844156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.755+0000 7fa84c288640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84415ceb0 0x7fa8441f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.755+0000 7fa84c288640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8441f2c20 0x7fa8441f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.755+0000 7fa84c288640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa8440b2a80 con 0x7fa84415ceb0 2024-06-29T14:23:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.755+0000 7fa84c288640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa8440b2900 con 0x7fa8441f2c20 2024-06-29T14:23:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.755+0000 7fa84c288640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa8440b2c00 con 0x7fa84413e9b0 2024-06-29T14:23:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84415ceb0 0x7fa8441f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84415ceb0 0x7fa8441f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59576/0 (socket says 172.21.15.59:59576) 2024-06-29T14:23:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 -- 172.21.15.59:0/2197626023 learned_addr learned my addr 172.21.15.59:0/2197626023 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa84a825640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8441f2c20 0x7fa8441f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa84a024640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa84413e9b0 0x7fa844156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 -- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa84413e9b0 msgr2=0x7fa844156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa84413e9b0 0x7fa844156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 -- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8441f2c20 msgr2=0x7fa8441f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8441f2c20 0x7fa8441f1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 -- 172.21.15.59:0/2197626023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa844139740 con 0x7fa84415ceb0 2024-06-29T14:23:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa849823640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84415ceb0 0x7fa8441f26e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c400 tx=0x7fa83400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa83affd640 1 -- 172.21.15.59:0/2197626023 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834011790 con 0x7fa84415ceb0 2024-06-29T14:23:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa83affd640 1 -- 172.21.15.59:0/2197626023 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa834017070 con 0x7fa84415ceb0 2024-06-29T14:23:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa844139a20 con 0x7fa84415ceb0 2024-06-29T14:23:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa83affd640 1 -- 172.21.15.59:0/2197626023 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834011cc0 con 0x7fa84415ceb0 2024-06-29T14:23:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.756+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa844139fe0 con 0x7fa84415ceb0 2024-06-29T14:23:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.757+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa84413edc0 con 0x7fa84415ceb0 2024-06-29T14:23:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.758+0000 7fa83affd640 1 -- 172.21.15.59:0/2197626023 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa83401c460 con 0x7fa84415ceb0 2024-06-29T14:23:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.759+0000 7fa83affd640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa8280762d0 0x7fa8280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.759+0000 7fa83affd640 1 -- 172.21.15.59:0/2197626023 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7fa834097820 con 0x7fa84415ceb0 2024-06-29T14:23:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.759+0000 7fa84a024640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa8280762d0 0x7fa8280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.760+0000 7fa84a024640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa8280762d0 0x7fa8280787e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa840034000 tx=0x7fa840004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.763+0000 7fa83affd640 1 -- 172.21.15.59:0/2197626023 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa834061010 con 0x7fa84415ceb0 2024-06-29T14:23:25.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.992+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa844139490 con 0x7fa84415ceb0 2024-06-29T14:23:25.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.992+0000 7fa83affd640 1 -- 172.21.15.59:0/2197626023 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5223 (secure 0 0 0) 0x7fa834060910 con 0x7fa84415ceb0 2024-06-29T14:23:25.992 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:23:25.992 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"btime":"2024-06-29T14:23:19:215604+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/634859216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":634859216},{"type":"v1","addr":"172.21.15.153:6837","nonce":634859216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3661382082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3661382082},{"type":"v1","addr":"172.21.15.153:6836","nonce":3661382082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4012387069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4012387069},{"type":"v1","addr":"172.21.15.59:6833","nonce":4012387069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:15.579309+0000","modified":"2024-06-29T14:23:18.607053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5469},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5469":{"gid":5469,"name":"a","rank":0,"incarnation":15,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/812277010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":812277010},{"type":"v1","addr":"172.21.15.59:6835","nonce":812277010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5469,"qdb_cluster":[5469]},"id":2}]} 2024-06-29T14:23:25.993 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 19 2024-06-29T14:23:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa8280762d0 msgr2=0x7fa8280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa8280762d0 0x7fa8280787e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa840034000 tx=0x7fa840004730 comp rx=0 tx=0).stop 2024-06-29T14:23:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84415ceb0 msgr2=0x7fa8441f26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84415ceb0 0x7fa8441f26e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c400 tx=0x7fa83400c8c0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 shutdown_connections 2024-06-29T14:23:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7fa8280762d0 0x7fa8280787e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8441f2c20 0x7fa8441f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84415ceb0 0x7fa8441f26e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 --2- 172.21.15.59:0/2197626023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa84413e9b0 0x7fa844156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.994+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 >> 172.21.15.59:0/2197626023 conn(0x7fa8440b6fb0 msgr2=0x7fa844136e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.995+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 shutdown_connections 2024-06-29T14:23:25.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:25.995+0000 7fa84c288640 1 -- 172.21.15.59:0/2197626023 wait complete. 2024-06-29T14:23:26.408 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:23:15.579309+0000', 'modified': '2024-06-29T14:23:18.607053+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5469}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5469': {'gid': 5469, 'name': 'a', 'rank': 0, 'incarnation': 15, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.59:6835/812277010', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 812277010}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 812277010}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5469, 'qdb_cluster': [5469]} 2024-06-29T14:23:26.408 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:23:26.408 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:23:26.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.535+0000 7f2558593640 1 -- 172.21.15.59:0/2621384671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253c004800 msgr2=0x7f253c004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.535+0000 7f2558593640 1 --2- 172.21.15.59:0/2621384671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253c004800 0x7f253c004c70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f254c009b50 tx=0x7f253c005830 comp rx=0 tx=0).stop 2024-06-29T14:23:26.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.535+0000 7f2556b30640 1 -- 172.21.15.59:0/2621384671 reap_dead start 2024-06-29T14:23:26.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.535+0000 7f2558593640 1 -- 172.21.15.59:0/2621384671 shutdown_connections 2024-06-29T14:23:26.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.535+0000 7f2558593640 1 -- 172.21.15.59:0/2621384671 >> 172.21.15.59:0/2621384671 conn(0x7f25500d3ee0 msgr2=0x7f25500d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.535+0000 7f2558593640 1 -- 172.21.15.59:0/2621384671 shutdown_connections 2024-06-29T14:23:26.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.535+0000 7f2558593640 1 -- 172.21.15.59:0/2621384671 wait complete. 2024-06-29T14:23:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.537+0000 7f2558593640 1 Processor -- start 2024-06-29T14:23:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.537+0000 7f2558593640 1 -- start start 2024-06-29T14:23:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2558593640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c004800 0x7f255015c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2558593640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25500b8780 0x7f25500b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2558593640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2550146eb0 0x7f25501472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2558593640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f255015e8d0 con 0x7f2550146eb0 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2558593640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f25500b42b0 con 0x7f253c004800 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2558593640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f255015ea50 con 0x7f25500b8780 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2556b30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2550146eb0 0x7f25501472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2556b30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2550146eb0 0x7f25501472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59592/0 (socket says 172.21.15.59:59592) 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.538+0000 7f2556b30640 1 -- 172.21.15.59:0/3042695570 learned_addr learned my addr 172.21.15.59:0/3042695570 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2555b2e640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25500b8780 0x7f25500b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f255632f640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c004800 0x7f255015c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2556b30640 1 -- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25500b8780 msgr2=0x7f25500b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2556b30640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25500b8780 0x7f25500b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2556b30640 1 -- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c004800 msgr2=0x7f255015c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2556b30640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c004800 0x7f255015c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2556b30640 1 -- 172.21.15.59:0/3042695570 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2550144c00 con 0x7f2550146eb0 2024-06-29T14:23:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2556b30640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2550146eb0 0x7f25501472c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f25500b5540 tx=0x7f2548006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2542ffd640 1 -- 172.21.15.59:0/3042695570 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548011900 con 0x7f2550146eb0 2024-06-29T14:23:26.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2542ffd640 1 -- 172.21.15.59:0/3042695570 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2548018070 con 0x7f2550146eb0 2024-06-29T14:23:26.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2558593640 1 -- 172.21.15.59:0/3042695570 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2550144ee0 con 0x7f2550146eb0 2024-06-29T14:23:26.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2542ffd640 1 -- 172.21.15.59:0/3042695570 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548011d20 con 0x7f2550146eb0 2024-06-29T14:23:26.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.539+0000 7f2558593640 1 -- 172.21.15.59:0/3042695570 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25501454a0 con 0x7f2550146eb0 2024-06-29T14:23:26.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.541+0000 7f2542ffd640 1 -- 172.21.15.59:0/3042695570 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f254801d460 con 0x7f2550146eb0 2024-06-29T14:23:26.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.542+0000 7f2542ffd640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2534076300 0x7f2534078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.542+0000 7f2542ffd640 1 -- 172.21.15.59:0/3042695570 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f2548098190 con 0x7f2550146eb0 2024-06-29T14:23:26.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.542+0000 7f255632f640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2534076300 0x7f2534078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.542+0000 7f255632f640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2534076300 0x7f2534078810 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2544001220 tx=0x7f2544001110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.543+0000 7f255f44d740 1 Processor -- start 2024-06-29T14:23:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.543+0000 7f255f44d740 1 -- start start 2024-06-29T14:23:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f255f44d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e5a4270 0x55f88e9906e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f255f44d740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 0x55f88e9743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f255f44d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e583590 0x55f88e9748e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f255f44d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55f88e90cbc0 con 0x55f88e583590 2024-06-29T14:23:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f255f44d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f88e8bb630 con 0x55f88e5a4270 2024-06-29T14:23:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f255f44d740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55f88e8aabd0 con 0x55f88e57b3b0 2024-06-29T14:23:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f2556b30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e583590 0x55f88e9748e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f2556b30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e583590 0x55f88e9748e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59604/0 (socket says 172.21.15.59:59604) 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f2556b30640 1 -- 172.21.15.59:0/3079600101 learned_addr learned my addr 172.21.15.59:0/3079600101 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f2555b2e640 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 0x55f88e9743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.544+0000 7f255632f640 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e5a4270 0x55f88e9906e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2556b30640 1 -- 172.21.15.59:0/3079600101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 msgr2=0x55f88e9743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2556b30640 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 0x55f88e9743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2556b30640 1 -- 172.21.15.59:0/3079600101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e5a4270 msgr2=0x55f88e9906e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2556b30640 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e5a4270 0x55f88e9906e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2556b30640 1 -- 172.21.15.59:0/3079600101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f88e8a7ca0 con 0x55f88e583590 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2556b30640 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e583590 0x55f88e9748e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f25480963d0 tx=0x7f2548060fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b73aa329f89b6be server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2533fff640 1 -- 172.21.15.59:0/3079600101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548060c40 con 0x55f88e583590 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2533fff640 1 -- 172.21.15.59:0/3079600101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2548061830 con 0x55f88e583590 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.545+0000 7f2533fff640 1 -- 172.21.15.59:0/3079600101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548061130 con 0x55f88e583590 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 -- 172.21.15.59:0/3079600101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e583590 msgr2=0x55f88e9748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e583590 0x55f88e9748e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f25480963d0 tx=0x7f2548060fb0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 -- 172.21.15.59:0/3079600101 shutdown_connections 2024-06-29T14:23:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e583590 0x55f88e9748e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 0x55f88e9743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 --2- 172.21.15.59:0/3079600101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e5a4270 0x55f88e9906e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 -- 172.21.15.59:0/3079600101 >> 172.21.15.59:0/3079600101 conn(0x55f88e5983c0 msgr2=0x55f88e947f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 -- 172.21.15.59:0/3079600101 shutdown_connections 2024-06-29T14:23:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.546+0000 7f255f44d740 1 -- 172.21.15.59:0/3079600101 wait complete. 2024-06-29T14:23:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.547+0000 7f255f44d740 1 -- start start 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.547+0000 7f255f44d740 1 Processor -- start 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.548+0000 7f255f44d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e57b3b0 0x55f88e96d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.548+0000 7f255f44d740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e583590 0x55f88e99df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.548+0000 7f255f44d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e96ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.548+0000 7f255f44d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55f88e848ae0 con 0x55f88e5a4270 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.548+0000 7f255f44d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f88e8aec10 con 0x55f88e57b3b0 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.548+0000 7f255f44d740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55f88e61e620 con 0x55f88e583590 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.548+0000 7f255f44d740 10 client.0 _ll_register_callbacks cb 0x55f88e855480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.548+0000 7f2556b30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e96ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2556b30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e96ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59614/0 (socket says 172.21.15.59:59614) 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2556b30640 1 -- 172.21.15.59:0/2226406301 learned_addr learned my addr 172.21.15.59:0/2226406301 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f255632f640 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e57b3b0 0x55f88e96d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2555b2e640 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e583590 0x55f88e99df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2556b30640 1 -- 172.21.15.59:0/2226406301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e583590 msgr2=0x55f88e99df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2556b30640 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e583590 0x55f88e99df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2556b30640 1 -- 172.21.15.59:0/2226406301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e57b3b0 msgr2=0x55f88e96d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2556b30640 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e57b3b0 0x55f88e96d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2556b30640 1 -- 172.21.15.59:0/2226406301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f88e60f850 con 0x55f88e5a4270 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f2556b30640 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e96ec20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f25480619e0 tx=0x7f2548060640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a003efcc44fa2d server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f25317fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f25317fa640 1 -- 172.21.15.59:0/2226406301 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548094060 con 0x55f88e5a4270 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f25317fa640 1 -- 172.21.15.59:0/2226406301 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f254805cca0 con 0x55f88e5a4270 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f255f44d740 1 -- 172.21.15.59:0/2226406301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2548061830 con 0x55f88e5a4270 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f25317fa640 1 -- 172.21.15.59:0/2226406301 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2548060c40 con 0x55f88e5a4270 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f255f44d740 10 client.5667 fetch_fsmap learned FSMap version 19 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f255f44d740 1 -- 172.21.15.59:0/2226406301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55f88e6866b0 con 0x55f88e5a4270 2024-06-29T14:23:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f25317fa640 1 -- 172.21.15.59:0/2226406301 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2548061830 con 0x55f88e5a4270 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.549+0000 7f25317fa640 1 -- 172.21.15.59:0/2226406301 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7f254805fe00 con 0x55f88e5a4270 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f255f44d740 10 client.5667 fetch_fsmap finished waiting for FSMap version 19 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f255f44d740 10 client.5667 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f255f44d740 20 client.5667 populate_metadata read hostname 'smithi059' 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f255f44d740 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] conn(0x55f88e562e40 0x55f88e972c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f255f44d740 4 client.5667 mds_command: new command op to 5469 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f255f44d740 1 -- 172.21.15.59:0/2226406301 --> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f88e8bd580 con 0x55f88e562e40 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f255632f640 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] conn(0x55f88e562e40 0x55f88e972c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f25317fa640 10 client.5667 ms_handle_connect on v2:172.21.15.59:6834/812277010 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.550+0000 7f255632f640 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] conn(0x55f88e562e40 0x55f88e972c10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=497c68b6678cfa3b server_cookie=54014a4b71657917 in_seq=0 out_seq=0 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f25317fa640 1 -- 172.21.15.59:0/2226406301 <== mds.0 v2:172.21.15.59:6834/812277010 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55f88e8bd580 con 0x55f88e562e40 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f25317fa640 10 client.5667 handle_command_reply: tid=0 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 client.5667 shutdown 2024-06-29T14:23:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 -- 172.21.15.59:0/2226406301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 msgr2=0x55f88e96ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e96ec20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f25480619e0 tx=0x7f2548060640 comp rx=0 tx=0).stop 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 -- 172.21.15.59:0/2226406301 shutdown_connections 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] conn(0x55f88e562e40 0x55f88e972c10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e96ec20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e583590 0x55f88e99df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 --2- 172.21.15.59:0/2226406301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e57b3b0 0x55f88e96d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f25317fa640 0 client.5667 ms_handle_reset on v2:172.21.15.59:6834/812277010 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 -- 172.21.15.59:0/2226406301 >> 172.21.15.59:0/2226406301 conn(0x55f88e5983c0 msgr2=0x55f88e943b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 -- 172.21.15.59:0/2226406301 shutdown_connections 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 1 -- 172.21.15.59:0/2226406301 wait complete. 2024-06-29T14:23:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.551+0000 7f255f44d740 20 client.5667 trim_cache size 0 max 16384 2024-06-29T14:23:26.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.570+0000 7f255f44d740 1 Processor -- start 2024-06-29T14:23:26.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.571+0000 7f255f44d740 1 -- start start 2024-06-29T14:23:26.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f255f44d740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 0x55f88e9465a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f255f44d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 0x55f88e9b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f255f44d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e9b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f255f44d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f25501f4320 con 0x55f88e5a4270 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f255f44d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f25501f8be0 con 0x55f88e583590 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f255f44d740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f25501f3cd0 con 0x55f88e57b3b0 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2556b30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e9b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2556b30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e9b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59618/0 (socket says 172.21.15.59:59618) 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2556b30640 1 -- 172.21.15.59:0/1917587985 learned_addr learned my addr 172.21.15.59:0/1917587985 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f255632f640 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 0x55f88e9465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2555b2e640 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 0x55f88e9b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2556b30640 1 -- 172.21.15.59:0/1917587985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 msgr2=0x55f88e9465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2556b30640 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 0x55f88e9465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2556b30640 1 -- 172.21.15.59:0/1917587985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 msgr2=0x55f88e9b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2556b30640 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 0x55f88e9b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.572+0000 7f2556b30640 1 -- 172.21.15.59:0/1917587985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f88e6866b0 con 0x55f88e5a4270 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.573+0000 7f2556b30640 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e9b28e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f25480610c0 tx=0x7f2548060f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970cf2ef4a13bec server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.573+0000 7f25337fe640 1 -- 172.21.15.59:0/1917587985 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548061830 con 0x55f88e5a4270 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.573+0000 7f25337fe640 1 -- 172.21.15.59:0/1917587985 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25480606b0 con 0x55f88e5a4270 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.573+0000 7f25337fe640 1 -- 172.21.15.59:0/1917587985 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254805cce0 con 0x55f88e5a4270 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.573+0000 7f255f44d740 1 -- 172.21.15.59:0/1917587985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 msgr2=0x55f88e9b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.573+0000 7f255f44d740 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e9b28e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f25480610c0 tx=0x7f2548060f40 comp rx=0 tx=0).stop 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 -- 172.21.15.59:0/1917587985 shutdown_connections 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e5a4270 0x55f88e9b28e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 0x55f88e9b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 --2- 172.21.15.59:0/1917587985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e57b3b0 0x55f88e9465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 -- 172.21.15.59:0/1917587985 >> 172.21.15.59:0/1917587985 conn(0x55f88e5983c0 msgr2=0x55f88e972c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 -- 172.21.15.59:0/1917587985 shutdown_connections 2024-06-29T14:23:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 -- 172.21.15.59:0/1917587985 wait complete. 2024-06-29T14:23:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 -- start start 2024-06-29T14:23:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.574+0000 7f255f44d740 1 Processor -- start 2024-06-29T14:23:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255f44d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e57b3b0 0x55f88ea0ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255f44d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 0x55f88e99c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255f44d740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e5a4270 0x55f88e99cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255f44d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55f88e971f90 con 0x55f88e57b3b0 2024-06-29T14:23:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255f44d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f88e848ae0 con 0x55f88e583590 2024-06-29T14:23:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255f44d740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55f88e96e050 con 0x55f88e5a4270 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255f44d740 10 client.0 _ll_register_callbacks cb 0x7f253c0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255632f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e57b3b0 0x55f88ea0ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255632f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e57b3b0 0x55f88ea0ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59634/0 (socket says 172.21.15.59:59634) 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255632f640 1 -- 172.21.15.59:0/1427593525 learned_addr learned my addr 172.21.15.59:0/1427593525 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f2555b2e640 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 0x55f88e99c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f2556b30640 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e5a4270 0x55f88e99cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255632f640 1 -- 172.21.15.59:0/1427593525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e5a4270 msgr2=0x55f88e99cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255632f640 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e5a4270 0x55f88e99cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255632f640 1 -- 172.21.15.59:0/1427593525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 msgr2=0x55f88e99c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255632f640 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 0x55f88e99c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.576+0000 7f255632f640 1 -- 172.21.15.59:0/1427593525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f88e971440 con 0x55f88e57b3b0 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255632f640 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e57b3b0 0x55f88ea0ac30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x55f88e974e20 tx=0x7f255015f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30bf836417c160b server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f25317fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f25317fa640 1 -- 172.21.15.59:0/1427593525 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544012d60 con 0x55f88e57b3b0 2024-06-29T14:23:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f25317fa640 1 -- 172.21.15.59:0/1427593525 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f254400a030 con 0x55f88e57b3b0 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 1 -- 172.21.15.59:0/1427593525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25480606b0 con 0x55f88e57b3b0 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f25317fa640 1 -- 172.21.15.59:0/1427593525 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f2544039050 con 0x55f88e57b3b0 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 10 client.5673 fetch_fsmap learned FSMap version 19 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 1 -- 172.21.15.59:0/1427593525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55f88ea0b170 con 0x55f88e57b3b0 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f25317fa640 1 -- 172.21.15.59:0/1427593525 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25480606b0 con 0x55f88e57b3b0 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f25317fa640 1 -- 172.21.15.59:0/1427593525 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7f254403c020 con 0x55f88e57b3b0 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 10 client.5673 fetch_fsmap finished waiting for FSMap version 19 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 10 client.5673 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 20 client.5673 populate_metadata read hostname 'smithi059' 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] conn(0x55f88e96e9b0 0x55f88e9644a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 4 client.5673 mds_command: new command op to 5469 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f255f44d740 1 -- 172.21.15.59:0/1427593525 --> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] -- command(tid 0: {"prefix": "status"}) -- 0x55f88e8b9020 con 0x55f88e96e9b0 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.577+0000 7f2555b2e640 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] conn(0x55f88e96e9b0 0x55f88e9644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f25317fa640 10 client.5673 ms_handle_connect on v2:172.21.15.59:6834/812277010 2024-06-29T14:23:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f2555b2e640 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] conn(0x55f88e96e9b0 0x55f88e9644a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f78ed4e9a64119b6 server_cookie=bfe410aeac79319c in_seq=0 out_seq=0 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f25317fa640 1 -- 172.21.15.59:0/1427593525 <== mds.0 v2:172.21.15.59:6834/812277010 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55f88e8b9020 con 0x55f88e96e9b0 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f25317fa640 10 client.5673 handle_command_reply: tid=0 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 client.5673 shutdown 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 -- 172.21.15.59:0/1427593525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e57b3b0 msgr2=0x55f88ea0ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e57b3b0 0x55f88ea0ac30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x55f88e974e20 tx=0x7f255015f810 comp rx=0 tx=0).stop 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 -- 172.21.15.59:0/1427593525 shutdown_connections 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.59:6834/812277010,v1:172.21.15.59:6835/812277010] conn(0x55f88e96e9b0 0x55f88e9644a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f88e5a4270 0x55f88e99cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f88e583590 0x55f88e99c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 --2- 172.21.15.59:0/1427593525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f88e57b3b0 0x55f88ea0ac30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f25317fa640 0 client.5673 ms_handle_reset on v2:172.21.15.59:6834/812277010 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 -- 172.21.15.59:0/1427593525 >> 172.21.15.59:0/1427593525 conn(0x55f88e5983c0 msgr2=0x55f88e972bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 -- 172.21.15.59:0/1427593525 shutdown_connections 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.578+0000 7f255f44d740 1 -- 172.21.15.59:0/1427593525 wait complete. 2024-06-29T14:23:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f255f44d740 20 client.5673 trim_cache size 0 max 16384 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 5469, 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 10.383226928999999, 2024-06-29T14:23:26.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 19, 2024-06-29T14:23:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 41, 2024-06-29T14:23:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 41, 2024-06-29T14:23:26.583 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 19.323420274 2024-06-29T14:23:26.583 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:23:26.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 -- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2534076300 msgr2=0x7f2534078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2534076300 0x7f2534078810 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2544001220 tx=0x7f2544001110 comp rx=0 tx=0).stop 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 -- 172.21.15.59:0/3042695570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2550146eb0 msgr2=0x7f25501472c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2550146eb0 0x7f25501472c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f25500b5540 tx=0x7f2548006430 comp rx=0 tx=0).stop 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 -- 172.21.15.59:0/3042695570 shutdown_connections 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2534076300 0x7f2534078810 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2550146eb0 0x7f25501472c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25500b8780 0x7f25500b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 --2- 172.21.15.59:0/3042695570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c004800 0x7f255015c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 -- 172.21.15.59:0/3042695570 >> 172.21.15.59:0/3042695570 conn(0x7f25500d3ee0 msgr2=0x7f255006c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 -- 172.21.15.59:0/3042695570 shutdown_connections 2024-06-29T14:23:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.579+0000 7f2558593640 1 -- 172.21.15.59:0/3042695570 wait complete. 2024-06-29T14:23:26.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:23:26.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.708+0000 7f2e56ec1640 1 -- 172.21.15.59:0/2701587504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 msgr2=0x7f2e500b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.708+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/2701587504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 0x7f2e500b5880 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2e40005fd0 tx=0x7f2e4002a530 comp rx=0 tx=0).stop 2024-06-29T14:23:26.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.708+0000 7f2e56ec1640 1 -- 172.21.15.59:0/2701587504 shutdown_connections 2024-06-29T14:23:26.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.708+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/2701587504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 0x7f2e500b5880 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.708+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/2701587504 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e501458a0 0x7f2e500cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.708+0000 7f2e56ec1640 1 -- 172.21.15.59:0/2701587504 >> 172.21.15.59:0/2701587504 conn(0x7f2e5006c950 msgr2=0x7f2e50073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.708+0000 7f2e56ec1640 1 -- 172.21.15.59:0/2701587504 shutdown_connections 2024-06-29T14:23:26.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.708+0000 7f2e56ec1640 1 -- 172.21.15.59:0/2701587504 wait complete. 2024-06-29T14:23:26.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.709+0000 7f2e56ec1640 1 Processor -- start 2024-06-29T14:23:26.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.709+0000 7f2e56ec1640 1 -- start start 2024-06-29T14:23:26.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e56ec1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e500b3740 0x7f2e5015f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e56ec1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 0x7f2e5015fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e56ec1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e501f2e10 0x7f2e501f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e56ec1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2e500b6150 con 0x7f2e500b3b50 2024-06-29T14:23:26.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e56ec1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e500b5fd0 con 0x7f2e501f2e10 2024-06-29T14:23:26.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e56ec1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2e500b62d0 con 0x7f2e500b3740 2024-06-29T14:23:26.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 0x7f2e5015fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 0x7f2e5015fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59664/0 (socket says 172.21.15.59:59664) 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e4ffff640 1 -- 172.21.15.59:0/1856206879 learned_addr learned my addr 172.21.15.59:0/1856206879 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e5545e640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e501f2e10 0x7f2e501f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e54c5d640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e500b3740 0x7f2e5015f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e4ffff640 1 -- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e500b3740 msgr2=0x7f2e5015f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e4ffff640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e500b3740 0x7f2e5015f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e4ffff640 1 -- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e501f2e10 msgr2=0x7f2e501f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e4ffff640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e501f2e10 0x7f2e501f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e4ffff640 1 -- 172.21.15.59:0/1856206879 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5014e890 con 0x7f2e500b3b50 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.711+0000 7f2e54c5d640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e500b3740 0x7f2e5015f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.712+0000 7f2e4ffff640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 0x7f2e5015fde0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400b390 tx=0x7f2e4400b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.712+0000 7f2e4d7fa640 1 -- 172.21.15.59:0/1856206879 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44015030 con 0x7f2e500b3b50 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.712+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5014eb70 con 0x7f2e500b3b50 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.712+0000 7f2e4d7fa640 1 -- 172.21.15.59:0/1856206879 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e44004020 con 0x7f2e500b3b50 2024-06-29T14:23:26.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.712+0000 7f2e4d7fa640 1 -- 172.21.15.59:0/1856206879 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44004470 con 0x7f2e500b3b50 2024-06-29T14:23:26.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.712+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5014f130 con 0x7f2e500b3b50 2024-06-29T14:23:26.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.713+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e501458a0 con 0x7f2e500b3b50 2024-06-29T14:23:26.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.714+0000 7f2e4d7fa640 1 -- 172.21.15.59:0/1856206879 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2e44018070 con 0x7f2e500b3b50 2024-06-29T14:23:26.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.714+0000 7f2e4d7fa640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2e3c0762d0 0x7f2e3c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.714+0000 7f2e4d7fa640 1 -- 172.21.15.59:0/1856206879 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f2e44097480 con 0x7f2e500b3b50 2024-06-29T14:23:26.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.714+0000 7f2e54c5d640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2e3c0762d0 0x7f2e3c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.715+0000 7f2e54c5d640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2e3c0762d0 0x7f2e3c0787e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e4800ae00 tx=0x7f2e4800a7e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.719+0000 7f2e4d7fa640 1 -- 172.21.15.59:0/1856206879 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2e44060b40 con 0x7f2e500b3b50 2024-06-29T14:23:26.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.950+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2e500b1f10 con 0x7f2e500b3b50 2024-06-29T14:23:26.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.951+0000 7f2e4d7fa640 1 -- 172.21.15.59:0/1856206879 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) ==== 53+0+73 (secure 0 0 0) 0x7f2e44060440 con 0x7f2e500b3b50 2024-06-29T14:23:26.949 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:23:26.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2e3c0762d0 msgr2=0x7f2e3c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2e3c0762d0 0x7f2e3c0787e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e4800ae00 tx=0x7f2e4800a7e0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 msgr2=0x7f2e5015fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 0x7f2e5015fde0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400b390 tx=0x7f2e4400b850 comp rx=0 tx=0).stop 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 shutdown_connections 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f2e3c0762d0 0x7f2e3c0787e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e501f2e10 0x7f2e501f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b3b50 0x7f2e5015fde0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 --2- 172.21.15.59:0/1856206879 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e500b3740 0x7f2e5015f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 >> 172.21.15.59:0/1856206879 conn(0x7f2e5006c950 msgr2=0x7f2e500d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 shutdown_connections 2024-06-29T14:23:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:26.953+0000 7f2e56ec1640 1 -- 172.21.15.59:0/1856206879 wait complete. 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:23:26.963 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:23:26.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:26.964 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:23:26.980 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:26.980 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:26.980 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:23:26.981 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42644sec preferred_lft 42644sec 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:23:26.982 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:23:26.983 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T14:23:26.983 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:26.983 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:23:26.983 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:23:26.983 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T14:23:26.983 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:26.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:26.983 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:23:27.045 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:23:27.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:27.047 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:23:27.103 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:23:27.103 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:23:27.104 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:23:27.104 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:23:27.104 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:23:27.104 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:27.104 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:23:27.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:23:27.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:23:27.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:23:27.105 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:23:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:27.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:27.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:27.233 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:23:27.249 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:23:27.249 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:23:27.249 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:23:27.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:27.249 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:23:27.315 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:27.316 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:27.371 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:23:27.372 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:27.372 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:27.427 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:23:27.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:27.428 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:23:27.523 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:23:27.523 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:23:27.523 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:23:27.534 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:23:27.534 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:23:27.534 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:23:27.534 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:23:27.534 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:23:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:23:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:23:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:23:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:23:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:23:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:23:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:23:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:23:27.545 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:23:27.545 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:23:27.545 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:23:27.545 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:23:27.545 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:23:27.547 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:23:27.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:27.547 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:27.573 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:27.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:27.677 INFO:teuthology.orchestra.run.smithi059.stdout:client5682 2024-06-29T14:23:27.677 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:23:27.677 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:27.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:23:27.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.856+0000 7f14dc59a640 1 -- 172.21.15.59:0/981644675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14bc004d40 msgr2=0x7f14bc005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:27.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.856+0000 7f14dc59a640 1 --2- 172.21.15.59:0/981644675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14bc004d40 0x7f14bc005150 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f14c4005fd0 tx=0x7f14bc005830 comp rx=0 tx=0).stop 2024-06-29T14:23:27.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.856+0000 7f14dab37640 1 -- 172.21.15.59:0/981644675 reap_dead start 2024-06-29T14:23:27.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.856+0000 7f14dc59a640 1 -- 172.21.15.59:0/981644675 shutdown_connections 2024-06-29T14:23:27.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.856+0000 7f14dc59a640 1 -- 172.21.15.59:0/981644675 >> 172.21.15.59:0/981644675 conn(0x7f14d40b06e0 msgr2=0x7f14d40b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:27.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.856+0000 7f14dc59a640 1 -- 172.21.15.59:0/981644675 shutdown_connections 2024-06-29T14:23:27.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.856+0000 7f14dc59a640 1 -- 172.21.15.59:0/981644675 wait complete. 2024-06-29T14:23:27.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.858+0000 7f14dc59a640 1 Processor -- start 2024-06-29T14:23:27.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.858+0000 7f14dc59a640 1 -- start start 2024-06-29T14:23:27.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14dc59a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14bc004d40 0x7f14d413f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:27.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14dc59a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14d41eff90 0x7f14d41f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:27.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14dc59a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14d40b0af0 0x7f14d40b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:27.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14dc59a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f14d406ce00 con 0x7f14d41eff90 2024-06-29T14:23:27.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14dc59a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14d406cc80 con 0x7f14d40b0af0 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14dc59a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f14d406cf80 con 0x7f14bc004d40 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14d9b35640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14d41eff90 0x7f14d41f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14d9b35640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14d41eff90 0x7f14d41f03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59668/0 (socket says 172.21.15.59:59668) 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14d9b35640 1 -- 172.21.15.59:0/239539983 learned_addr learned my addr 172.21.15.59:0/239539983 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14da336640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14bc004d40 0x7f14d413f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14dab37640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14d40b0af0 0x7f14d40b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14d9b35640 1 -- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14bc004d40 msgr2=0x7f14d413f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14d9b35640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14bc004d40 0x7f14d413f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.859+0000 7f14d9b35640 1 -- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14d40b0af0 msgr2=0x7f14d40b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14d9b35640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14d40b0af0 0x7f14d40b0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14d9b35640 1 -- 172.21.15.59:0/239539983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d40b1440 con 0x7f14d41eff90 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14da336640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14bc004d40 0x7f14d413f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:27.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14dab37640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14d40b0af0 0x7f14d40b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:27.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14d9b35640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14d41eff90 0x7f14d41f03a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f14c8011ae0 tx=0x7f14c8009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:27.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14d2ffd640 1 -- 172.21.15.59:0/239539983 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c801e070 con 0x7f14d41eff90 2024-06-29T14:23:27.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14d2ffd640 1 -- 172.21.15.59:0/239539983 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14c800fc70 con 0x7f14d41eff90 2024-06-29T14:23:27.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d413f610 con 0x7f14d41eff90 2024-06-29T14:23:27.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14d2ffd640 1 -- 172.21.15.59:0/239539983 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8019410 con 0x7f14d41eff90 2024-06-29T14:23:27.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.860+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d4157170 con 0x7f14d41eff90 2024-06-29T14:23:27.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.861+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d406e300 con 0x7f14d41eff90 2024-06-29T14:23:27.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.862+0000 7f14d2ffd640 1 -- 172.21.15.59:0/239539983 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f14c8021070 con 0x7f14d41eff90 2024-06-29T14:23:27.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.862+0000 7f14d2ffd640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14c0076300 0x7f14c0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:27.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.862+0000 7f14d2ffd640 1 -- 172.21.15.59:0/239539983 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f14c8098150 con 0x7f14d41eff90 2024-06-29T14:23:27.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.863+0000 7f14da336640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14c0076300 0x7f14c0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:27.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.863+0000 7f14da336640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14c0076300 0x7f14c0078810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f14cc006d30 tx=0x7f14cc001590 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:27.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:27.867+0000 7f14d2ffd640 1 -- 172.21.15.59:0/239539983 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f14d40b1440 con 0x7f14d41eff90 2024-06-29T14:23:28.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.081+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f14d41f0ac0 con 0x7f14c0076300 2024-06-29T14:23:28.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.084+0000 7f14d2ffd640 1 -- 172.21.15.59:0/239539983 <== mgr.4102 v2:172.21.15.153:6832/654263470 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f14d41f0ac0 con 0x7f14c0076300 2024-06-29T14:23:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:23:28.082 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:23:28.082 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T14:23:28.082 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:23:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:23:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14c0076300 msgr2=0x7f14c0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14c0076300 0x7f14c0078810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f14cc006d30 tx=0x7f14cc001590 comp rx=0 tx=0).stop 2024-06-29T14:23:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14d41eff90 msgr2=0x7f14d41f03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14d41eff90 0x7f14d41f03a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f14c8011ae0 tx=0x7f14c8009ed0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 shutdown_connections 2024-06-29T14:23:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f14c0076300 0x7f14c0078810 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14d40b0af0 0x7f14d40b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14d41eff90 0x7f14d41f03a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 --2- 172.21.15.59:0/239539983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14bc004d40 0x7f14d413f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 >> 172.21.15.59:0/239539983 conn(0x7f14d40b06e0 msgr2=0x7f14d40b1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:28.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 shutdown_connections 2024-06-29T14:23:28.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.086+0000 7f14dc59a640 1 -- 172.21.15.59:0/239539983 wait complete. 2024-06-29T14:23:28.095 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:23:28.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.217+0000 7f125a8ce640 1 -- 172.21.15.59:0/3380340855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 msgr2=0x7f123c0052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:28.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.217+0000 7f125a8ce640 1 --2- 172.21.15.59:0/3380340855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 0x7f123c0052a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f124c001f30 tx=0x7f123c005830 comp rx=0 tx=0).stop 2024-06-29T14:23:28.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.217+0000 7f1258e6b640 1 -- 172.21.15.59:0/3380340855 reap_dead start 2024-06-29T14:23:28.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.217+0000 7f125a8ce640 1 -- 172.21.15.59:0/3380340855 shutdown_connections 2024-06-29T14:23:28.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.217+0000 7f125a8ce640 1 -- 172.21.15.59:0/3380340855 >> 172.21.15.59:0/3380340855 conn(0x7f1254073e10 msgr2=0x7f1254074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:28.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.217+0000 7f125a8ce640 1 -- 172.21.15.59:0/3380340855 shutdown_connections 2024-06-29T14:23:28.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.217+0000 7f125a8ce640 1 -- 172.21.15.59:0/3380340855 wait complete. 2024-06-29T14:23:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.219+0000 7f125a8ce640 1 Processor -- start 2024-06-29T14:23:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.219+0000 7f125a8ce640 1 -- start start 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.220+0000 7f125a8ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 0x7f125414e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.220+0000 7f125a8ce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f125406e590 0x7f1254070de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.220+0000 7f125a8ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125406eff0 0x7f1254071320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.220+0000 7f125a8ce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f125415c120 con 0x7f123c004dd0 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.220+0000 7f125a8ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f125415bfa0 con 0x7f125406eff0 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.220+0000 7f125a8ce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f12540b6240 con 0x7f125406e590 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.220+0000 7f1253fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 0x7f125414e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1253fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 0x7f125414e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59692/0 (socket says 172.21.15.59:59692) 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1253fff640 1 -- 172.21.15.59:0/3922306990 learned_addr learned my addr 172.21.15.59:0/3922306990 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1258e6b640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125406eff0 0x7f1254071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:28.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f12537fe640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f125406e590 0x7f1254070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1253fff640 1 -- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f125406e590 msgr2=0x7f1254070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1253fff640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f125406e590 0x7f1254070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1253fff640 1 -- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125406eff0 msgr2=0x7f1254071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1253fff640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125406eff0 0x7f1254071320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1253fff640 1 -- 172.21.15.59:0/3922306990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125406eda0 con 0x7f123c004dd0 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1253fff640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 0x7f125414e190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1244011ae0 tx=0x7f1244009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1250ff9640 1 -- 172.21.15.59:0/3922306990 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124401e070 con 0x7f123c004dd0 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1250ff9640 1 -- 172.21.15.59:0/3922306990 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f124400fe70 con 0x7f123c004dd0 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12540ad950 con 0x7f123c004dd0 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f1250ff9640 1 -- 172.21.15.59:0/3922306990 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1244004290 con 0x7f123c004dd0 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.221+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12540adf10 con 0x7f123c004dd0 2024-06-29T14:23:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.222+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12540d1520 con 0x7f123c004dd0 2024-06-29T14:23:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.223+0000 7f1250ff9640 1 -- 172.21.15.59:0/3922306990 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1244021070 con 0x7f123c004dd0 2024-06-29T14:23:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.224+0000 7f1250ff9640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f12400762d0 0x7f12400787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.224+0000 7f1250ff9640 1 -- 172.21.15.59:0/3922306990 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f1244098430 con 0x7f123c004dd0 2024-06-29T14:23:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.224+0000 7f12537fe640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f12400762d0 0x7f12400787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.224+0000 7f12537fe640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f12400762d0 0x7f12400787e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f12541580b0 tx=0x7f1248006720 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:28.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.228+0000 7f1250ff9640 1 -- 172.21.15.59:0/3922306990 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1244061560 con 0x7f123c004dd0 2024-06-29T14:23:28.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:28.468+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f12540b5cb0 con 0x7f123c004dd0 2024-06-29T14:23:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.240+0000 7f1250ff9640 1 -- 172.21.15.59:0/3922306990 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) ==== 92+0+0 (secure 0 0 0) 0x7f1244065860 con 0x7f123c004dd0 2024-06-29T14:23:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.243+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f12400762d0 msgr2=0x7f12400787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.243+0000 7f125a8ce640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f12400762d0 0x7f12400787e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f12541580b0 tx=0x7f1248006720 comp rx=0 tx=0).stop 2024-06-29T14:23:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.244+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 msgr2=0x7f125414e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.244+0000 7f125a8ce640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 0x7f125414e190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1244011ae0 tx=0x7f1244009e90 comp rx=0 tx=0).stop 2024-06-29T14:23:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.244+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 shutdown_connections 2024-06-29T14:23:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.244+0000 7f125a8ce640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f12400762d0 0x7f12400787e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.244+0000 7f125a8ce640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f125406eff0 0x7f1254071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.244+0000 7f125a8ce640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f125406e590 0x7f1254070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.244+0000 7f125a8ce640 1 --2- 172.21.15.59:0/3922306990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f123c004dd0 0x7f125414e190 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.245+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 >> 172.21.15.59:0/3922306990 conn(0x7f1254073e10 msgr2=0x7f12540d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.245+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 shutdown_connections 2024-06-29T14:23:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.245+0000 7f125a8ce640 1 -- 172.21.15.59:0/3922306990 wait complete. 2024-06-29T14:23:29.260 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:23:29.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.393+0000 7f7febefd640 1 -- 172.21.15.59:0/3753300960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fe40b2270 msgr2=0x7f7fe415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.393+0000 7f7febefd640 1 --2- 172.21.15.59:0/3753300960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fe40b2270 0x7f7fe415bc20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009be0 tx=0x7f7fdc02a920 comp rx=0 tx=0).stop 2024-06-29T14:23:29.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.393+0000 7f7febefd640 1 -- 172.21.15.59:0/3753300960 shutdown_connections 2024-06-29T14:23:29.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.393+0000 7f7febefd640 1 --2- 172.21.15.59:0/3753300960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fe40b2270 0x7f7fe415bc20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.393+0000 7f7febefd640 1 -- 172.21.15.59:0/3753300960 >> 172.21.15.59:0/3753300960 conn(0x7f7fe4073e10 msgr2=0x7f7fe4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:29.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.393+0000 7f7febefd640 1 -- 172.21.15.59:0/3753300960 shutdown_connections 2024-06-29T14:23:29.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.393+0000 7f7febefd640 1 -- 172.21.15.59:0/3753300960 wait complete. 2024-06-29T14:23:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.395+0000 7f7febefd640 1 Processor -- start 2024-06-29T14:23:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.395+0000 7f7febefd640 1 -- start start 2024-06-29T14:23:29.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.396+0000 7f7febefd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fe40b2270 0x7f7fe4153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.396+0000 7f7febefd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40b2fa0 0x7f7fe41f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.396+0000 7f7febefd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe40d1c30 0x7f7fe415ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.396+0000 7f7febefd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe40b65d0 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.396+0000 7f7febefd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe40b6450 con 0x7f7fe40d1c30 2024-06-29T14:23:29.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.396+0000 7f7febefd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7fe40b6750 con 0x7f7fe40b2270 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40b2fa0 0x7f7fe41f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40b2fa0 0x7f7fe41f54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59716/0 (socket says 172.21.15.59:59716) 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 -- 172.21.15.59:0/1951872705 learned_addr learned my addr 172.21.15.59:0/1951872705 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9c99640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fe40b2270 0x7f7fe4153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fea49a640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe40d1c30 0x7f7fe415ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 -- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fe40b2270 msgr2=0x7f7fe4153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fe40b2270 0x7f7fe4153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 -- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe40d1c30 msgr2=0x7f7fe415ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe40d1c30 0x7f7fe415ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 -- 172.21.15.59:0/1951872705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe41f5a00 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe9498640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40b2fa0 0x7f7fe41f54c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c3c0 tx=0x7f7fd800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe27fc640 1 -- 172.21.15.59:0/1951872705 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8017070 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe27fc640 1 -- 172.21.15.59:0/1951872705 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fd8005400 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe415d550 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7fe27fc640 1 -- 172.21.15.59:0/1951872705 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80118b0 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.397+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe415c950 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.398+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe40b6c40 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.399+0000 7f7fe27fc640 1 -- 172.21.15.59:0/1951872705 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 6) ==== 98218+0+0 (secure 0 0 0) 0x7f7fd801a070 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.400+0000 7f7fe27fc640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7fd0076320 0x7f7fd0078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.400+0000 7f7fe9c99640 1 -- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7fd0076320 msgr2=0x7f7fd0078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/654263470 2024-06-29T14:23:29.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.400+0000 7f7fe27fc640 1 -- 172.21.15.59:0/1951872705 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f7fd8096ca0 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.400+0000 7f7fe9c99640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7fd0076320 0x7f7fd0078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:23:29.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.404+0000 7f7fe27fc640 1 -- 172.21.15.59:0/1951872705 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7fe41f5a00 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.598+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f7fe415d270 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.601+0000 7f7fe9c99640 1 -- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7fd0076320 msgr2=0x7f7fd0078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/654263470 2024-06-29T14:23:29.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.601+0000 7f7fe9c99640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7fd0076320 0x7f7fd0078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:23:29.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.603+0000 7f7fe27fc640 1 -- 172.21.15.59:0/1951872705 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f7fd805fe20 con 0x7f7fe40b2fa0 2024-06-29T14:23:29.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7fd0076320 msgr2=0x7f7fd0078830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:23:29.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7fd0076320 0x7f7fd0078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40b2fa0 msgr2=0x7f7fe41f54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40b2fa0 0x7f7fe41f54c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c3c0 tx=0x7f7fd800c880 comp rx=0 tx=0).stop 2024-06-29T14:23:29.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 shutdown_connections 2024-06-29T14:23:29.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f7fd0076320 0x7f7fd0078830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe40d1c30 0x7f7fe415ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40b2fa0 0x7f7fe41f54c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 --2- 172.21.15.59:0/1951872705 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fe40b2270 0x7f7fe4153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 >> 172.21.15.59:0/1951872705 conn(0x7f7fe4073e10 msgr2=0x7f7fe4070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:29.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 shutdown_connections 2024-06-29T14:23:29.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.606+0000 7f7febefd640 1 -- 172.21.15.59:0/1951872705 wait complete. 2024-06-29T14:23:29.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:23:29.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.740+0000 7f838fe3f640 1 -- 172.21.15.59:0/4216286115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 msgr2=0x7f838815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.740+0000 7f838fe3f640 1 --2- 172.21.15.59:0/4216286115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 0x7f838815e8d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f83880d15d0 tx=0x7f838402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.740+0000 7f838fe3f640 1 -- 172.21.15.59:0/4216286115 shutdown_connections 2024-06-29T14:23:29.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.740+0000 7f838fe3f640 1 --2- 172.21.15.59:0/4216286115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 0x7f838815e8d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.740+0000 7f838fe3f640 1 --2- 172.21.15.59:0/4216286115 >> v2:172.21.15.153:3300/0 conn(0x7f83880d23c0 0x7f83880b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.740+0000 7f838fe3f640 1 -- 172.21.15.59:0/4216286115 >> 172.21.15.59:0/4216286115 conn(0x7f83880d3ee0 msgr2=0x7f83880d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:29.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.741+0000 7f838fe3f640 1 -- 172.21.15.59:0/4216286115 shutdown_connections 2024-06-29T14:23:29.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.741+0000 7f838fe3f640 1 -- 172.21.15.59:0/4216286115 wait complete. 2024-06-29T14:23:29.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.742+0000 7f838fe3f640 1 Processor -- start 2024-06-29T14:23:29.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.742+0000 7f838fe3f640 1 -- start start 2024-06-29T14:23:29.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838fe3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 0x7f83881e8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838fe3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83880d23c0 0x7f8388155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838fe3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838815ee10 0x7f83881f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838fe3f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8388160240 con 0x7f83880b3ed0 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838fe3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83881600c0 con 0x7f838815ee10 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838fe3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f83881603c0 con 0x7f83880d23c0 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 0x7f83881e8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 0x7f83881e8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59742/0 (socket says 172.21.15.59:59742) 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 -- 172.21.15.59:0/2414209604 learned_addr learned my addr 172.21.15.59:0/2414209604 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838e3dc640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838815ee10 0x7f83881f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838d3da640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83880d23c0 0x7f8388155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 -- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83880d23c0 msgr2=0x7f8388155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83880d23c0 0x7f8388155430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 -- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838815ee10 msgr2=0x7f83881f5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838815ee10 0x7f83881f5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 -- 172.21.15.59:0/2414209604 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83881e88c0 con 0x7f83880b3ed0 2024-06-29T14:23:29.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.744+0000 7f838dbdb640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 0x7f83881e8380 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f837800e790 tx=0x7f837800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:29.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.745+0000 7f837e7fc640 1 -- 172.21.15.59:0/2414209604 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837800eea0 con 0x7f83880b3ed0 2024-06-29T14:23:29.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.745+0000 7f837e7fc640 1 -- 172.21.15.59:0/2414209604 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f837800ccc0 con 0x7f83880b3ed0 2024-06-29T14:23:29.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.745+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83881e75c0 con 0x7f83880b3ed0 2024-06-29T14:23:29.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.745+0000 7f837e7fc640 1 -- 172.21.15.59:0/2414209604 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378004280 con 0x7f83880b3ed0 2024-06-29T14:23:29.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.745+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83881e7b80 con 0x7f83880b3ed0 2024-06-29T14:23:29.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.746+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8388069de0 con 0x7f83880b3ed0 2024-06-29T14:23:29.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.747+0000 7f837e7fc640 1 -- 172.21.15.59:0/2414209604 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 6) ==== 98218+0+0 (secure 0 0 0) 0x7f83780035f0 con 0x7f83880b3ed0 2024-06-29T14:23:29.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.747+0000 7f837e7fc640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f836c076350 0x7f836c078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.748+0000 7f838d3da640 1 -- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f836c076350 msgr2=0x7f836c078860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/654263470 2024-06-29T14:23:29.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.748+0000 7f837e7fc640 1 -- 172.21.15.59:0/2414209604 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f8378098100 con 0x7f83880b3ed0 2024-06-29T14:23:29.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.748+0000 7f838d3da640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f836c076350 0x7f836c078860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:23:29.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.752+0000 7f837e7fc640 1 -- 172.21.15.59:0/2414209604 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8388069de0 con 0x7f83880b3ed0 2024-06-29T14:23:29.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.945+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f838815e8d0 con 0x7f83880b3ed0 2024-06-29T14:23:29.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.949+0000 7f838d3da640 1 -- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f836c076350 msgr2=0x7f836c078860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/654263470 2024-06-29T14:23:29.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.949+0000 7f838d3da640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f836c076350 0x7f836c078860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:23:29.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.950+0000 7f837e7fc640 1 -- 172.21.15.59:0/2414209604 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f8378061200 con 0x7f83880b3ed0 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f836c076350 msgr2=0x7f836c078860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f836c076350 0x7f836c078860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 msgr2=0x7f83881e8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 0x7f83881e8380 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f837800e790 tx=0x7f837800ec50 comp rx=0 tx=0).stop 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 shutdown_connections 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f836c076350 0x7f836c078860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f838815ee10 0x7f83881f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83880d23c0 0x7f8388155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 --2- 172.21.15.59:0/2414209604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83880b3ed0 0x7f83881e8380 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 >> 172.21.15.59:0/2414209604 conn(0x7f83880d3ee0 msgr2=0x7f8388160900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:29.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.952+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 shutdown_connections 2024-06-29T14:23:29.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:29.953+0000 7f838fe3f640 1 -- 172.21.15.59:0/2414209604 wait complete. 2024-06-29T14:23:29.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:29.962 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:23:29.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:29.985 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-06-29T14:23:30.042 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-06-29T14:23:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.210+0000 7f1465d3c640 1 -- 172.21.15.59:0/952548080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14600b2270 msgr2=0x7f146015bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.210+0000 7f1465d3c640 1 --2- 172.21.15.59:0/952548080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14600b2270 0x7f146015bc20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1450009830 tx=0x7f145002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.210+0000 7f1465d3c640 1 -- 172.21.15.59:0/952548080 shutdown_connections 2024-06-29T14:23:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.210+0000 7f1465d3c640 1 --2- 172.21.15.59:0/952548080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14600b2270 0x7f146015bc20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.210+0000 7f1465d3c640 1 -- 172.21.15.59:0/952548080 >> 172.21.15.59:0/952548080 conn(0x7f1460073e10 msgr2=0x7f1460074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.210+0000 7f1465d3c640 1 -- 172.21.15.59:0/952548080 shutdown_connections 2024-06-29T14:23:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.210+0000 7f1465d3c640 1 -- 172.21.15.59:0/952548080 wait complete. 2024-06-29T14:23:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.212+0000 7f1465d3c640 1 Processor -- start 2024-06-29T14:23:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.212+0000 7f1465d3c640 1 -- start start 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f1465d3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14600b2270 0x7f146014f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f1465d3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14600b2fa0 0x7f1460156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f1465d3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14600d1c30 0x7f1460156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f1465d3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f14600b65d0 con 0x7f14600b2fa0 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f1465d3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14600b6450 con 0x7f14600d1c30 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f1465d3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f14600b6750 con 0x7f14600b2270 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f145effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14600b2fa0 0x7f1460156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f145f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14600b2270 0x7f146014f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f145effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14600b2fa0 0x7f1460156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59758/0 (socket says 172.21.15.59:59758) 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f145effd640 1 -- 172.21.15.59:0/3399470361 learned_addr learned my addr 172.21.15.59:0/3399470361 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.213+0000 7f145ffff640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14600d1c30 0x7f1460156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f145effd640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14600b2270 msgr2=0x7f146014f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f145effd640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14600b2270 0x7f146014f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f145effd640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14600d1c30 msgr2=0x7f1460156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f145effd640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14600d1c30 0x7f1460156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f145effd640 1 -- 172.21.15.59:0/3399470361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14601f80c0 con 0x7f14600b2fa0 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f145effd640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14600b2fa0 0x7f1460156690 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1454001220 tx=0x7f145400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454011790 con 0x7f14600b2fa0 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1454017070 con 0x7f14600b2fa0 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14601f8530 con 0x7f14600b2fa0 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454011cc0 con 0x7f14600b2fa0 2024-06-29T14:23:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.214+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14601f8af0 con 0x7f14600b2fa0 2024-06-29T14:23:30.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.215+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14600b6c70 con 0x7f14600b2fa0 2024-06-29T14:23:30.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.216+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 6) ==== 98218+0+0 (secure 0 0 0) 0x7f145401c460 con 0x7f14600b2fa0 2024-06-29T14:23:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.217+0000 7f1447fff640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 0x7f144c078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.217+0000 7f145f7fe640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 msgr2=0x7f144c078860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/654263470 2024-06-29T14:23:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.217+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6101+0+0 (secure 0 0 0) 0x7f1454097680 con 0x7f14600b2fa0 2024-06-29T14:23:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.217+0000 7f145f7fe640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 0x7f144c078860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:23:30.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.221+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1454060760 con 0x7f14600b2fa0 2024-06-29T14:23:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.418+0000 7f145f7fe640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 msgr2=0x7f144c078860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/654263470 2024-06-29T14:23:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.418+0000 7f145f7fe640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 0x7f144c078860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:23:30.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.454+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 --> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f14601f8290 con 0x7f144c076350 2024-06-29T14:23:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.819+0000 7f145f7fe640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 msgr2=0x7f144c078860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/654263470 2024-06-29T14:23:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:30.819+0000 7f145f7fe640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 0x7f144c078860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:23:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:31.620+0000 7f145f7fe640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 msgr2=0x7f144c078860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/654263470 2024-06-29T14:23:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:31.620+0000 7f145f7fe640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 0x7f144c078860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:23:32.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:32.835+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 7) ==== 97929+0+0 (secure 0 0 0) 0x7f145405c9f0 con 0x7f14600b2fa0 2024-06-29T14:23:32.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:32.836+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 msgr2=0x7f144c078860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:23:32.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:32.836+0000 7f1447fff640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/654263470,v1:172.21.15.153:6833/654263470] conn(0x7f144c076350 0x7f144c078860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b5fb31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b5fb31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b5fb31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b5fb31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b5fb31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b62b37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b62b37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b62b37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b62b37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.019 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:23:33.020+0000 7f4b62b37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:33.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.840+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 8) ==== 98056+0+0 (secure 0 0 0) 0x7f145406a140 con 0x7f14600b2fa0 2024-06-29T14:23:33.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.840+0000 7f1447fff640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f144c07e8a0 0x7f144c080cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.840+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1454097470 con 0x7f144c07e8a0 2024-06-29T14:23:33.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.842+0000 7f145f7fe640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f144c07e8a0 0x7f144c080cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:33.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.847+0000 7f145f7fe640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f144c07e8a0 0x7f144c080cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1450034000 tx=0x7f1450004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:33.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.938+0000 7f1447fff640 1 -- 172.21.15.59:0/3399470361 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f1454097470 con 0x7f144c07e8a0 2024-06-29T14:23:33.936 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir0 2024-06-29T14:23:33.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.940+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f144c07e8a0 msgr2=0x7f144c080cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.940+0000 7f1465d3c640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f144c07e8a0 0x7f144c080cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1450034000 tx=0x7f1450004730 comp rx=0 tx=0).stop 2024-06-29T14:23:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.940+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14600b2fa0 msgr2=0x7f1460156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.940+0000 7f1465d3c640 1 --2- 172.21.15.59:0/3399470361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14600b2fa0 0x7f1460156690 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1454001220 tx=0x7f145400c890 comp rx=0 tx=0).stop 2024-06-29T14:23:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.940+0000 7f145ffff640 1 -- 172.21.15.59:0/3399470361 reap_dead start 2024-06-29T14:23:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.940+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 shutdown_connections 2024-06-29T14:23:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.940+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 >> 172.21.15.59:0/3399470361 conn(0x7f1460073e10 msgr2=0x7f1460158170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.941+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 shutdown_connections 2024-06-29T14:23:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:33.941+0000 7f1465d3c640 1 -- 172.21.15.59:0/3399470361 wait complete. 2024-06-29T14:23:33.950 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-06-29T14:23:33.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:33.951 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-06-29T14:23:33.968 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-06-29T14:23:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.133+0000 7f26a789c640 1 -- 172.21.15.59:0/466991006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 msgr2=0x7f268c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.133+0000 7f26a789c640 1 --2- 172.21.15.59:0/466991006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 0x7f268c004800 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f269000c3c0 tx=0x7f268c005830 comp rx=0 tx=0).stop 2024-06-29T14:23:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.133+0000 7f26a5e39640 1 -- 172.21.15.59:0/466991006 reap_dead start 2024-06-29T14:23:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.133+0000 7f26a789c640 1 -- 172.21.15.59:0/466991006 shutdown_connections 2024-06-29T14:23:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.133+0000 7f26a789c640 1 -- 172.21.15.59:0/466991006 >> 172.21.15.59:0/466991006 conn(0x7f26a00d3ee0 msgr2=0x7f26a00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.133+0000 7f26a789c640 1 -- 172.21.15.59:0/466991006 shutdown_connections 2024-06-29T14:23:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.133+0000 7f26a789c640 1 -- 172.21.15.59:0/466991006 wait complete. 2024-06-29T14:23:34.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.135+0000 7f26a789c640 1 Processor -- start 2024-06-29T14:23:34.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.135+0000 7f26a789c640 1 -- start start 2024-06-29T14:23:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.136+0000 7f26a789c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 0x7f26a015b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.136+0000 7f26a789c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26a0145b00 0x7f26a0160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.136+0000 7f26a789c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a0160d70 0x7f26a0143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.136+0000 7f26a789c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f26a00b3d20 con 0x7f268c004000 2024-06-29T14:23:34.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.136+0000 7f26a789c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26a00b3ba0 con 0x7f26a0160d70 2024-06-29T14:23:34.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.136+0000 7f26a789c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f26a01600c0 con 0x7f26a0145b00 2024-06-29T14:23:34.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.136+0000 7f26a5638640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 0x7f26a015b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5638640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 0x7f26a015b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44108/0 (socket says 172.21.15.59:44108) 2024-06-29T14:23:34.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5638640 1 -- 172.21.15.59:0/3807029136 learned_addr learned my addr 172.21.15.59:0/3807029136 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:34.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5e39640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a0160d70 0x7f26a0143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a4e37640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26a0145b00 0x7f26a0160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5638640 1 -- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26a0145b00 msgr2=0x7f26a0160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5638640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26a0145b00 0x7f26a0160830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5638640 1 -- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a0160d70 msgr2=0x7f26a0143090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5638640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a0160d70 0x7f26a0143090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5638640 1 -- 172.21.15.59:0/3807029136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a015bb90 con 0x7f268c004000 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a5638640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 0x7f26a015b650 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f269c011ae0 tx=0x7f269c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f2695ffb640 1 -- 172.21.15.59:0/3807029136 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269c00fda0 con 0x7f268c004000 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f2695ffb640 1 -- 172.21.15.59:0/3807029136 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f269c005020 con 0x7f268c004000 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a0161180 con 0x7f268c004000 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f2695ffb640 1 -- 172.21.15.59:0/3807029136 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269c005450 con 0x7f268c004000 2024-06-29T14:23:34.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.137+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a015d7d0 con 0x7f268c004000 2024-06-29T14:23:34.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.139+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a00cf980 con 0x7f268c004000 2024-06-29T14:23:34.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.139+0000 7f2695ffb640 1 -- 172.21.15.59:0/3807029136 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) ==== 98056+0+0 (secure 0 0 0) 0x7f269c0055f0 con 0x7f268c004000 2024-06-29T14:23:34.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.140+0000 7f2695ffb640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2684076170 0x7f2684078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.140+0000 7f2695ffb640 1 -- 172.21.15.59:0/3807029136 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f269c098240 con 0x7f268c004000 2024-06-29T14:23:34.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.140+0000 7f26a4e37640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2684076170 0x7f2684078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.141+0000 7f26a4e37640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2684076170 0x7f2684078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2690033000 tx=0x7f269002d040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.144+0000 7f2695ffb640 1 -- 172.21.15.59:0/3807029136 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f26a00cf980 con 0x7f268c004000 2024-06-29T14:23:34.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.357+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f26a0160620 con 0x7f2684076170 2024-06-29T14:23:34.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.439+0000 7f2695ffb640 1 -- 172.21.15.59:0/3807029136 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f26a0160620 con 0x7f2684076170 2024-06-29T14:23:34.437 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir1 2024-06-29T14:23:34.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2684076170 msgr2=0x7f2684078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2684076170 0x7f2684078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2690033000 tx=0x7f269002d040 comp rx=0 tx=0).stop 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 msgr2=0x7f26a015b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 0x7f26a015b650 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f269c011ae0 tx=0x7f269c00f3c0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 shutdown_connections 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2684076170 0x7f2684078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26a0160d70 0x7f26a0143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26a0145b00 0x7f26a0160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 --2- 172.21.15.59:0/3807029136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f268c004000 0x7f26a015b650 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 >> 172.21.15.59:0/3807029136 conn(0x7f26a00d3ee0 msgr2=0x7f26a0156930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 shutdown_connections 2024-06-29T14:23:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.441+0000 7f26a789c640 1 -- 172.21.15.59:0/3807029136 wait complete. 2024-06-29T14:23:34.450 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-06-29T14:23:34.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:34.451 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-06-29T14:23:34.468 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-06-29T14:23:34.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.630+0000 7f66170a5640 1 -- 172.21.15.59:0/1757346105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661006da00 msgr2=0x7f661015f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.630+0000 7f66170a5640 1 --2- 172.21.15.59:0/1757346105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661006da00 0x7f661015f240 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6610070220 tx=0x7f660002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.631+0000 7f66170a5640 1 -- 172.21.15.59:0/1757346105 shutdown_connections 2024-06-29T14:23:34.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.631+0000 7f66170a5640 1 --2- 172.21.15.59:0/1757346105 >> v2:172.21.15.153:3300/0 conn(0x7f66101444c0 0x7f6610073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.631+0000 7f66170a5640 1 --2- 172.21.15.59:0/1757346105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661006da00 0x7f661015f240 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.631+0000 7f66170a5640 1 -- 172.21.15.59:0/1757346105 >> 172.21.15.59:0/1757346105 conn(0x7f66100b24c0 msgr2=0x7f66100cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.631+0000 7f66170a5640 1 -- 172.21.15.59:0/1757346105 shutdown_connections 2024-06-29T14:23:34.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.631+0000 7f66170a5640 1 -- 172.21.15.59:0/1757346105 wait complete. 2024-06-29T14:23:34.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.632+0000 7f66170a5640 1 Processor -- start 2024-06-29T14:23:34.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.632+0000 7f66170a5640 1 -- start start 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f66170a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66100cd9b0 0x7f66101f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f66170a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66101444c0 0x7f66100ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f66170a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66100ae390 0x7f661006c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f66170a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6610073b40 con 0x7f66101444c0 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f66170a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66100739c0 con 0x7f66100cd9b0 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f66170a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6610073cc0 con 0x7f66100ae390 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6607fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66101444c0 0x7f66100ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6607fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66101444c0 0x7f66100ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44128/0 (socket says 172.21.15.59:44128) 2024-06-29T14:23:34.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6607fff640 1 -- 172.21.15.59:0/2263109555 learned_addr learned my addr 172.21.15.59:0/2263109555 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6615642640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66100ae390 0x7f661006c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6614e41640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66100cd9b0 0x7f66101f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6607fff640 1 -- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66100ae390 msgr2=0x7f661006c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6607fff640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66100ae390 0x7f661006c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6607fff640 1 -- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66100cd9b0 msgr2=0x7f66101f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6607fff640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66100cd9b0 0x7f66101f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.634+0000 7f6607fff640 1 -- 172.21.15.59:0/2263109555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66101f49a0 con 0x7f66101444c0 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.635+0000 7f6607fff640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66101444c0 0x7f66100ade50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6600034040 tx=0x7f6600003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.635+0000 7f66057fa640 1 -- 172.21.15.59:0/2263109555 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660003e030 con 0x7f66101444c0 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.635+0000 7f66057fa640 1 -- 172.21.15.59:0/2263109555 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6600003e00 con 0x7f66101444c0 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.635+0000 7f66057fa640 1 -- 172.21.15.59:0/2263109555 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6600035040 con 0x7f66101444c0 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.635+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66100aebd0 con 0x7f66101444c0 2024-06-29T14:23:34.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.635+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66100af190 con 0x7f66101444c0 2024-06-29T14:23:34.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.636+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66100af3d0 con 0x7f66101444c0 2024-06-29T14:23:34.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.637+0000 7f66057fa640 1 -- 172.21.15.59:0/2263109555 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) ==== 98056+0+0 (secure 0 0 0) 0x7f6600041030 con 0x7f66101444c0 2024-06-29T14:23:34.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.637+0000 7f66057fa640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f65fc076170 0x7f65fc078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.637+0000 7f66057fa640 1 -- 172.21.15.59:0/2263109555 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f66000b7f00 con 0x7f66101444c0 2024-06-29T14:23:34.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.638+0000 7f6614e41640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f65fc076170 0x7f65fc078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.638+0000 7f6614e41640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f65fc076170 0x7f65fc078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f660c001720 tx=0x7f660c009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.642+0000 7f66057fa640 1 -- 172.21.15.59:0/2263109555 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f66100af3d0 con 0x7f66101444c0 2024-06-29T14:23:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.854+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6610074170 con 0x7f65fc076170 2024-06-29T14:23:34.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.888+0000 7f66057fa640 1 -- 172.21.15.59:0/2263109555 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7f660007d3f0 con 0x7f66101444c0 2024-06-29T14:23:34.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.938+0000 7f66057fa640 1 -- 172.21.15.59:0/2263109555 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f6610074170 con 0x7f65fc076170 2024-06-29T14:23:34.936 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir2 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.940+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f65fc076170 msgr2=0x7f65fc078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.940+0000 7f66170a5640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f65fc076170 0x7f65fc078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f660c001720 tx=0x7f660c009040 comp rx=0 tx=0).stop 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.940+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66101444c0 msgr2=0x7f66100ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.940+0000 7f66170a5640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66101444c0 0x7f66100ade50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6600034040 tx=0x7f6600003910 comp rx=0 tx=0).stop 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.941+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 shutdown_connections 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.941+0000 7f66170a5640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f65fc076170 0x7f65fc078680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.941+0000 7f66170a5640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f66100ae390 0x7f661006c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.941+0000 7f66170a5640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66101444c0 0x7f66100ade50 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.941+0000 7f66170a5640 1 --2- 172.21.15.59:0/2263109555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66100cd9b0 0x7f66101f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.941+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 >> 172.21.15.59:0/2263109555 conn(0x7f66100b24c0 msgr2=0x7f66100b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.941+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 shutdown_connections 2024-06-29T14:23:34.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:34.941+0000 7f66170a5640 1 -- 172.21.15.59:0/2263109555 wait complete. 2024-06-29T14:23:34.950 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-06-29T14:23:34.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:34.950 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-06-29T14:23:34.967 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-06-29T14:23:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.130+0000 7fdcfc363640 1 -- 172.21.15.59:0/2203133196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf415e0a0 msgr2=0x7fdcf40b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.130+0000 7fdcfc363640 1 --2- 172.21.15.59:0/2203133196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf415e0a0 0x7fdcf40b38e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdcec009830 tx=0x7fdcec02a470 comp rx=0 tx=0).stop 2024-06-29T14:23:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.130+0000 7fdcfc363640 1 -- 172.21.15.59:0/2203133196 shutdown_connections 2024-06-29T14:23:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.130+0000 7fdcfc363640 1 --2- 172.21.15.59:0/2203133196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf415e0a0 0x7fdcf40b38e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.130+0000 7fdcfc363640 1 --2- 172.21.15.59:0/2203133196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcf4162190 0x7fdcf40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.130+0000 7fdcfc363640 1 -- 172.21.15.59:0/2203133196 >> 172.21.15.59:0/2203133196 conn(0x7fdcf40d45b0 msgr2=0x7fdcf40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.130+0000 7fdcfc363640 1 -- 172.21.15.59:0/2203133196 shutdown_connections 2024-06-29T14:23:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.130+0000 7fdcfc363640 1 -- 172.21.15.59:0/2203133196 wait complete. 2024-06-29T14:23:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.132+0000 7fdcfc363640 1 Processor -- start 2024-06-29T14:23:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.132+0000 7fdcfc363640 1 -- start start 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.133+0000 7fdcfc363640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf415e0a0 0x7fdcf40cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcfc363640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcf4162770 0x7fdcf4152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcfc363640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcf40c8470 0x7fdcf40cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcfc363640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf40b43a0 con 0x7fdcf4162770 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcfc363640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf40b4220 con 0x7fdcf415e0a0 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcfc363640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf40b4520 con 0x7fdcf40c8470 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcf4162770 0x7fdcf4152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcf4162770 0x7fdcf4152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44130/0 (socket says 172.21.15.59:44130) 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 -- 172.21.15.59:0/4012619063 learned_addr learned my addr 172.21.15.59:0/4012619063 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:35.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcfa900640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcf40c8470 0x7fdcf40cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcfa0ff640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf415e0a0 0x7fdcf40cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 -- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcf40c8470 msgr2=0x7fdcf40cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcf40c8470 0x7fdcf40cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 -- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf415e0a0 msgr2=0x7fdcf40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf415e0a0 0x7fdcf40cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 -- 172.21.15.59:0/4012619063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf4155280 con 0x7fdcf4162770 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf98fe640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcf4162770 0x7fdcf4152550 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdce800e9f0 tx=0x7fdce800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf2ffd640 1 -- 172.21.15.59:0/4012619063 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce800cd20 con 0x7fdcf4162770 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf2ffd640 1 -- 172.21.15.59:0/4012619063 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdce8004020 con 0x7fdcf4162770 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf4155560 con 0x7fdcf4162770 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.134+0000 7fdcf2ffd640 1 -- 172.21.15.59:0/4012619063 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8004450 con 0x7fdcf4162770 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.135+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf40c88b0 con 0x7fdcf4162770 2024-06-29T14:23:35.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.135+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf40b49d0 con 0x7fdcf4162770 2024-06-29T14:23:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.136+0000 7fdcf2ffd640 1 -- 172.21.15.59:0/4012619063 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7fdce80035f0 con 0x7fdcf4162770 2024-06-29T14:23:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.137+0000 7fdcf2ffd640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fdce0076440 0x7fdce0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.137+0000 7fdcf2ffd640 1 -- 172.21.15.59:0/4012619063 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7fdce80983e0 con 0x7fdcf4162770 2024-06-29T14:23:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.137+0000 7fdcfa0ff640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fdce0076440 0x7fdce0078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:35.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.137+0000 7fdcfa0ff640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fdce0076440 0x7fdce0078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdce40017c0 tx=0x7fdce4001750 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:35.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.141+0000 7fdcf2ffd640 1 -- 172.21.15.59:0/4012619063 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdce8061240 con 0x7fdcf4162770 2024-06-29T14:23:35.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.354+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fdcf4155030 con 0x7fdce0076440 2024-06-29T14:23:35.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.432+0000 7fdcf2ffd640 1 -- 172.21.15.59:0/4012619063 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fdcf4155030 con 0x7fdce0076440 2024-06-29T14:23:35.430 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir3 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fdce0076440 msgr2=0x7fdce0078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fdce0076440 0x7fdce0078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdce40017c0 tx=0x7fdce4001750 comp rx=0 tx=0).stop 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcf4162770 msgr2=0x7fdcf4152550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcf4162770 0x7fdcf4152550 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdce800e9f0 tx=0x7fdce800eeb0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 shutdown_connections 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fdce0076440 0x7fdce0078950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdcf40c8470 0x7fdcf40cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcf4162770 0x7fdcf4152550 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 --2- 172.21.15.59:0/4012619063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdcf415e0a0 0x7fdcf40cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 >> 172.21.15.59:0/4012619063 conn(0x7fdcf40d45b0 msgr2=0x7fdcf406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:35.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 shutdown_connections 2024-06-29T14:23:35.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.435+0000 7fdcfc363640 1 -- 172.21.15.59:0/4012619063 wait complete. 2024-06-29T14:23:35.444 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-06-29T14:23:35.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:35.444 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-06-29T14:23:35.461 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-06-29T14:23:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.625+0000 7f398f012640 1 -- 172.21.15.59:0/4245254445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3988162190 msgr2=0x7f39880ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.625+0000 7f398f012640 1 --2- 172.21.15.59:0/4245254445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3988162190 0x7f39880ce550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f397c009860 tx=0x7f397c02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.625+0000 7f398f012640 1 -- 172.21.15.59:0/4245254445 shutdown_connections 2024-06-29T14:23:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.625+0000 7f398f012640 1 --2- 172.21.15.59:0/4245254445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3988162190 0x7f39880ce550 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.625+0000 7f398f012640 1 -- 172.21.15.59:0/4245254445 >> 172.21.15.59:0/4245254445 conn(0x7f39880d45b0 msgr2=0x7f39880d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:35.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.625+0000 7f398f012640 1 -- 172.21.15.59:0/4245254445 shutdown_connections 2024-06-29T14:23:35.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.625+0000 7f398f012640 1 -- 172.21.15.59:0/4245254445 wait complete. 2024-06-29T14:23:35.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.627+0000 7f398f012640 1 Processor -- start 2024-06-29T14:23:35.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.627+0000 7f398f012640 1 -- start start 2024-06-29T14:23:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398f012640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f398815e0a0 0x7f3988160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398f012640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3988162770 0x7f39881f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398f012640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39880cb040 0x7f39880cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398f012640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f39880b45d0 con 0x7f39880cb040 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398f012640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39880b4450 con 0x7f398815e0a0 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398f012640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f39880b4750 con 0x7f3988162770 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398d5af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39880cb040 0x7f39880cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398d5af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39880cb040 0x7f39880cb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44142/0 (socket says 172.21.15.59:44142) 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398d5af640 1 -- 172.21.15.59:0/1389117202 learned_addr learned my addr 172.21.15.59:0/1389117202 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.628+0000 7f398cdae640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f398815e0a0 0x7f3988160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f3987fff640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3988162770 0x7f39881f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f398d5af640 1 -- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3988162770 msgr2=0x7f39881f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f398d5af640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3988162770 0x7f39881f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f398d5af640 1 -- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f398815e0a0 msgr2=0x7f3988160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f398d5af640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f398815e0a0 0x7f3988160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f398d5af640 1 -- 172.21.15.59:0/1389117202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3988161250 con 0x7f39880cb040 2024-06-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f398d5af640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39880cb040 0x7f39880cb450 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f398000c700 tx=0x7f398000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f39857fa640 1 -- 172.21.15.59:0/1389117202 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980003b70 con 0x7f39880cb040 2024-06-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3988161530 con 0x7f39880cb040 2024-06-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f39857fa640 1 -- 172.21.15.59:0/1389117202 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3980003d10 con 0x7f39880cb040 2024-06-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f39857fa640 1 -- 172.21.15.59:0/1389117202 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980005330 con 0x7f39880cb040 2024-06-29T14:23:35.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.629+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3988161af0 con 0x7f39880cb040 2024-06-29T14:23:35.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.630+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3988050590 con 0x7f39880cb040 2024-06-29T14:23:35.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.631+0000 7f39857fa640 1 -- 172.21.15.59:0/1389117202 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7f39800054d0 con 0x7f39880cb040 2024-06-29T14:23:35.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.631+0000 7f39857fa640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3974076470 0x7f3974078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:35.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.632+0000 7f39857fa640 1 -- 172.21.15.59:0/1389117202 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f3980097900 con 0x7f39880cb040 2024-06-29T14:23:35.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.632+0000 7f398cdae640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3974076470 0x7f3974078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:35.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.632+0000 7f398cdae640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3974076470 0x7f3974078980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3978000c00 tx=0x7f3978009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:35.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.636+0000 7f39857fa640 1 -- 172.21.15.59:0/1389117202 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3988050590 con 0x7f39880cb040 2024-06-29T14:23:35.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.849+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f39880ce550 con 0x7f3974076470 2024-06-29T14:23:35.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.931+0000 7f39857fa640 1 -- 172.21.15.59:0/1389117202 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f39880ce550 con 0x7f3974076470 2024-06-29T14:23:35.929 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir4 2024-06-29T14:23:35.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3974076470 msgr2=0x7f3974078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3974076470 0x7f3974078980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3978000c00 tx=0x7f3978009040 comp rx=0 tx=0).stop 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39880cb040 msgr2=0x7f39880cb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39880cb040 0x7f39880cb450 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f398000c700 tx=0x7f398000cbc0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 shutdown_connections 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3974076470 0x7f3974078980 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39880cb040 0x7f39880cb450 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3988162770 0x7f39881f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 --2- 172.21.15.59:0/1389117202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f398815e0a0 0x7f3988160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.933+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 >> 172.21.15.59:0/1389117202 conn(0x7f39880d45b0 msgr2=0x7f39880b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.934+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 shutdown_connections 2024-06-29T14:23:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:35.934+0000 7f398f012640 1 -- 172.21.15.59:0/1389117202 wait complete. 2024-06-29T14:23:35.942 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-06-29T14:23:35.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:35.943 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-06-29T14:23:35.960 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-06-29T14:23:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.122+0000 7f6ce5265640 1 -- 172.21.15.59:0/4154852907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce00b3740 msgr2=0x7f6ce00b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.122+0000 7f6ce5265640 1 --2- 172.21.15.59:0/4154852907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce00b3740 0x7f6ce00b3b50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8009830 tx=0x7f6cc802a510 comp rx=0 tx=0).stop 2024-06-29T14:23:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.122+0000 7f6ce5265640 1 -- 172.21.15.59:0/4154852907 shutdown_connections 2024-06-29T14:23:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.122+0000 7f6ce5265640 1 --2- 172.21.15.59:0/4154852907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ce015c2f0 0x7f6ce015c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.122+0000 7f6ce5265640 1 --2- 172.21.15.59:0/4154852907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce00b3740 0x7f6ce00b3b50 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.122+0000 7f6ce5265640 1 -- 172.21.15.59:0/4154852907 >> 172.21.15.59:0/4154852907 conn(0x7f6ce006c950 msgr2=0x7f6ce0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.122+0000 7f6ce5265640 1 -- 172.21.15.59:0/4154852907 shutdown_connections 2024-06-29T14:23:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.122+0000 7f6ce5265640 1 -- 172.21.15.59:0/4154852907 wait complete. 2024-06-29T14:23:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.124+0000 7f6ce5265640 1 Processor -- start 2024-06-29T14:23:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.124+0000 7f6ce5265640 1 -- start start 2024-06-29T14:23:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6ce5265640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ce00b3740 0x7f6ce013c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6ce5265640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce015c2f0 0x7f6ce013adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6ce5265640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ce013b690 0x7f6ce013d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6ce5265640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce00b5fd0 con 0x7f6ce015c2f0 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6ce5265640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce00b5e50 con 0x7f6ce013b690 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6ce5265640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce00b6150 con 0x7f6ce00b3740 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6cde7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce015c2f0 0x7f6ce013adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6cde7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce015c2f0 0x7f6ce013adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44154/0 (socket says 172.21.15.59:44154) 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6cde7fc640 1 -- 172.21.15.59:0/3053427216 learned_addr learned my addr 172.21.15.59:0/3053427216 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6cdeffd640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ce00b3740 0x7f6ce013c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.125+0000 7f6cdf7fe640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ce013b690 0x7f6ce013d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cde7fc640 1 -- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ce00b3740 msgr2=0x7f6ce013c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cde7fc640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ce00b3740 0x7f6ce013c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cde7fc640 1 -- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ce013b690 msgr2=0x7f6ce013d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cde7fc640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ce013b690 0x7f6ce013d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cde7fc640 1 -- 172.21.15.59:0/3053427216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce01390b0 con 0x7f6ce015c2f0 2024-06-29T14:23:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cde7fc640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce015c2f0 0x7f6ce013adc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8009950 tx=0x7f6cc8033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cd77fe640 1 -- 172.21.15.59:0/3053427216 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8005c90 con 0x7f6ce015c2f0 2024-06-29T14:23:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cd77fe640 1 -- 172.21.15.59:0/3053427216 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cc8031e50 con 0x7f6ce015c2f0 2024-06-29T14:23:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce006de60 con 0x7f6ce015c2f0 2024-06-29T14:23:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6cd77fe640 1 -- 172.21.15.59:0/3053427216 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc80332f0 con 0x7f6ce015c2f0 2024-06-29T14:23:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.126+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce006e420 con 0x7f6ce015c2f0 2024-06-29T14:23:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.127+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce00b6670 con 0x7f6ce015c2f0 2024-06-29T14:23:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.128+0000 7f6cd77fe640 1 -- 172.21.15.59:0/3053427216 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7f6cc8033490 con 0x7f6ce015c2f0 2024-06-29T14:23:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.128+0000 7f6cd77fe640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6cc0076440 0x7f6cc0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.129+0000 7f6cd77fe640 1 -- 172.21.15.59:0/3053427216 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f6cc80b8ea0 con 0x7f6ce015c2f0 2024-06-29T14:23:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.129+0000 7f6cdeffd640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6cc0076440 0x7f6cc0078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.129+0000 7f6cdeffd640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6cc0076440 0x7f6cc0078950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8001720 tx=0x7f6cd8007e90 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.133+0000 7f6cd77fe640 1 -- 172.21.15.59:0/3053427216 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6cc804a030 con 0x7f6ce015c2f0 2024-06-29T14:23:36.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.346+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6ce006dbf0 con 0x7f6cc0076440 2024-06-29T14:23:36.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.424+0000 7f6cd77fe640 1 -- 172.21.15.59:0/3053427216 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f6ce006dbf0 con 0x7f6cc0076440 2024-06-29T14:23:36.423 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir5 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6cc0076440 msgr2=0x7f6cc0078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6cc0076440 0x7f6cc0078950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8001720 tx=0x7f6cd8007e90 comp rx=0 tx=0).stop 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce015c2f0 msgr2=0x7f6ce013adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce015c2f0 0x7f6ce013adc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8009950 tx=0x7f6cc8033c90 comp rx=0 tx=0).stop 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 shutdown_connections 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6cc0076440 0x7f6cc0078950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ce013b690 0x7f6ce013d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ce015c2f0 0x7f6ce013adc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 --2- 172.21.15.59:0/3053427216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ce00b3740 0x7f6ce013c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 >> 172.21.15.59:0/3053427216 conn(0x7f6ce006c950 msgr2=0x7f6ce015fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:36.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 shutdown_connections 2024-06-29T14:23:36.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.427+0000 7f6ce5265640 1 -- 172.21.15.59:0/3053427216 wait complete. 2024-06-29T14:23:36.436 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-06-29T14:23:36.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:36.436 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-06-29T14:23:36.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-06-29T14:23:36.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.616+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/112252802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080ce950 msgr2=0x7f9b080d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.616+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/112252802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080ce950 0x7f9b080d2130 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9b00005fd0 tx=0x7f9b0002aa90 comp rx=0 tx=0).stop 2024-06-29T14:23:36.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.616+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/112252802 shutdown_connections 2024-06-29T14:23:36.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.616+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/112252802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080ce950 0x7f9b080d2130 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.617+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/112252802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b0808c190 0x7f9b080cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.617+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/112252802 >> 172.21.15.59:0/112252802 conn(0x7f9b080b5610 msgr2=0x7f9b080b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:36.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.617+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/112252802 shutdown_connections 2024-06-29T14:23:36.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.617+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/112252802 wait complete. 2024-06-29T14:23:36.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.619+0000 7f9b0d1c7640 1 Processor -- start 2024-06-29T14:23:36.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.619+0000 7f9b0d1c7640 1 -- start start 2024-06-29T14:23:36.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0d1c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b0808c190 0x7f9b081f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0d1c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b080ce950 0x7f9b0814d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0d1c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080cf5c0 0x7f9b080c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0d1c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9b080b7fa0 con 0x7f9b080cf5c0 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0d1c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b080b7e20 con 0x7f9b080ce950 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0d1c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9b080b8120 con 0x7f9b0808c190 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080cf5c0 0x7f9b080c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080cf5c0 0x7f9b080c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44170/0 (socket says 172.21.15.59:44170) 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0759e640 1 -- 172.21.15.59:0/2857154139 learned_addr learned my addr 172.21.15.59:0/2857154139 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b06d9d640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b0808c190 0x7f9b081f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0659c640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b080ce950 0x7f9b0814d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0759e640 1 -- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b0808c190 msgr2=0x7f9b081f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0759e640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b0808c190 0x7f9b081f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0759e640 1 -- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b080ce950 msgr2=0x7f9b0814d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0759e640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b080ce950 0x7f9b0814d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.620+0000 7f9b0759e640 1 -- 172.21.15.59:0/2857154139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b081f8fa0 con 0x7f9b080cf5c0 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.621+0000 7f9b0759e640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080cf5c0 0x7f9b080c87d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9b00006010 tx=0x7f9b00003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.621+0000 7f9aef7fe640 1 -- 172.21.15.59:0/2857154139 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00044070 con 0x7f9b080cf5c0 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.621+0000 7f9aef7fe640 1 -- 172.21.15.59:0/2857154139 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b0002e7f0 con 0x7f9b080cf5c0 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.621+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0815f160 con 0x7f9b080cf5c0 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.621+0000 7f9aef7fe640 1 -- 172.21.15.59:0/2857154139 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b000044b0 con 0x7f9b080cf5c0 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.621+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b0815f720 con 0x7f9b080cf5c0 2024-06-29T14:23:36.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.622+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b080cf080 con 0x7f9b080cf5c0 2024-06-29T14:23:36.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.623+0000 7f9aef7fe640 1 -- 172.21.15.59:0/2857154139 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7f9b00047070 con 0x7f9b080cf5c0 2024-06-29T14:23:36.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.623+0000 7f9aef7fe640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9af4076440 0x7f9af4078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.623+0000 7f9aef7fe640 1 -- 172.21.15.59:0/2857154139 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f9b000b8b60 con 0x7f9b080cf5c0 2024-06-29T14:23:36.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.623+0000 7f9b06d9d640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9af4076440 0x7f9af4078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.624+0000 7f9b06d9d640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9af4076440 0x7f9af4078950 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9af8005fd0 tx=0x7f9af800b2e0 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:36.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.628+0000 7f9aef7fe640 1 -- 172.21.15.59:0/2857154139 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9b081f8fa0 con 0x7f9b080cf5c0 2024-06-29T14:23:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.841+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9b080ca040 con 0x7f9af4076440 2024-06-29T14:23:36.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.909+0000 7f9aef7fe640 1 -- 172.21.15.59:0/2857154139 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f9b0007dbb0 con 0x7f9b080cf5c0 2024-06-29T14:23:36.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.939+0000 7f9aef7fe640 1 -- 172.21.15.59:0/2857154139 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f9b080ca040 con 0x7f9af4076440 2024-06-29T14:23:36.937 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir6 2024-06-29T14:23:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9af4076440 msgr2=0x7f9af4078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9af4076440 0x7f9af4078950 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9af8005fd0 tx=0x7f9af800b2e0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080cf5c0 msgr2=0x7f9b080c87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080cf5c0 0x7f9b080c87d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9b00006010 tx=0x7f9b00003f10 comp rx=0 tx=0).stop 2024-06-29T14:23:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 shutdown_connections 2024-06-29T14:23:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9af4076440 0x7f9af4078950 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b080cf5c0 0x7f9b080c87d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b080ce950 0x7f9b0814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 --2- 172.21.15.59:0/2857154139 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b0808c190 0x7f9b081f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 >> 172.21.15.59:0/2857154139 conn(0x7f9b080b5610 msgr2=0x7f9b081e6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 shutdown_connections 2024-06-29T14:23:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:36.941+0000 7f9b0d1c7640 1 -- 172.21.15.59:0/2857154139 wait complete. 2024-06-29T14:23:36.950 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-06-29T14:23:36.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:36.951 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-06-29T14:23:36.968 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-06-29T14:23:37.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.132+0000 7f4d51e3f640 1 -- 172.21.15.59:0/629200417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c141360 msgr2=0x7f4d4c141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.132+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/629200417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c141360 0x7f4d4c141770 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4d44005fd0 tx=0x7f4d4402aa00 comp rx=0 tx=0).stop 2024-06-29T14:23:37.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.132+0000 7f4d51e3f640 1 -- 172.21.15.59:0/629200417 shutdown_connections 2024-06-29T14:23:37.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.132+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/629200417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c141360 0x7f4d4c141770 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.132+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/629200417 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d4c1604d0 0x7f4d4c13ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.132+0000 7f4d51e3f640 1 -- 172.21.15.59:0/629200417 >> 172.21.15.59:0/629200417 conn(0x7f4d4c0b06e0 msgr2=0x7f4d4c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:37.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.132+0000 7f4d51e3f640 1 -- 172.21.15.59:0/629200417 shutdown_connections 2024-06-29T14:23:37.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.132+0000 7f4d51e3f640 1 -- 172.21.15.59:0/629200417 wait complete. 2024-06-29T14:23:37.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.134+0000 7f4d51e3f640 1 Processor -- start 2024-06-29T14:23:37.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.134+0000 7f4d51e3f640 1 -- start start 2024-06-29T14:23:37.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.135+0000 7f4d51e3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c0b1b00 0x7f4d4c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.135+0000 7f4d51e3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d4c141360 0x7f4d4c15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.135+0000 7f4d51e3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c1604d0 0x7f4d4c1f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.135+0000 7f4d51e3f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c0afa30 con 0x7f4d4c1604d0 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.135+0000 7f4d51e3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c0af8b0 con 0x7f4d4c0b1b00 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.135+0000 7f4d51e3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4c0afbb0 con 0x7f4d4c141360 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c1604d0 0x7f4d4c1f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c1604d0 0x7f4d4c1f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44186/0 (socket says 172.21.15.59:44186) 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 -- 172.21.15.59:0/2221588278 learned_addr learned my addr 172.21.15.59:0/2221588278 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4b7fe640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c0b1b00 0x7f4d4c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4affd640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d4c141360 0x7f4d4c15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 -- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d4c141360 msgr2=0x7f4d4c15cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d4c141360 0x7f4d4c15cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 -- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c0b1b00 msgr2=0x7f4d4c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c0b1b00 0x7f4d4c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 -- 172.21.15.59:0/2221588278 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4c1583f0 con 0x7f4d4c1604d0 2024-06-29T14:23:37.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d4bfff640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c1604d0 0x7f4d4c1f6310 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009d90 tx=0x7f4d44003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:37.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d33fff640 1 -- 172.21.15.59:0/2221588278 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44046070 con 0x7f4d4c1604d0 2024-06-29T14:23:37.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d33fff640 1 -- 172.21.15.59:0/2221588278 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d44004540 con 0x7f4d4c1604d0 2024-06-29T14:23:37.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4c1586d0 con 0x7f4d4c1604d0 2024-06-29T14:23:37.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d33fff640 1 -- 172.21.15.59:0/2221588278 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44033070 con 0x7f4d4c1604d0 2024-06-29T14:23:37.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.136+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d4c158c90 con 0x7f4d4c1604d0 2024-06-29T14:23:37.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.138+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d4c141770 con 0x7f4d4c1604d0 2024-06-29T14:23:37.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.138+0000 7f4d33fff640 1 -- 172.21.15.59:0/2221588278 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f4d44049070 con 0x7f4d4c1604d0 2024-06-29T14:23:37.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.138+0000 7f4d33fff640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f4d380764d0 0x7f4d380789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.139+0000 7f4d33fff640 1 -- 172.21.15.59:0/2221588278 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f4d440b8e70 con 0x7f4d4c1604d0 2024-06-29T14:23:37.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.139+0000 7f4d4b7fe640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f4d380764d0 0x7f4d380789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.139+0000 7f4d4b7fe640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f4d380764d0 0x7f4d380789e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00bb50 tx=0x7f4d3c00b2e0 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:37.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.143+0000 7f4d33fff640 1 -- 172.21.15.59:0/2221588278 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d4403e070 con 0x7f4d4c1604d0 2024-06-29T14:23:37.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.357+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4d4c1581a0 con 0x7f4d380764d0 2024-06-29T14:23:37.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.431+0000 7f4d33fff640 1 -- 172.21.15.59:0/2221588278 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f4d4c1581a0 con 0x7f4d380764d0 2024-06-29T14:23:37.429 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir7 2024-06-29T14:23:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f4d380764d0 msgr2=0x7f4d380789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f4d380764d0 0x7f4d380789e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00bb50 tx=0x7f4d3c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c1604d0 msgr2=0x7f4d4c1f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c1604d0 0x7f4d4c1f6310 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009d90 tx=0x7f4d44003f20 comp rx=0 tx=0).stop 2024-06-29T14:23:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 shutdown_connections 2024-06-29T14:23:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f4d380764d0 0x7f4d380789e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d4c1604d0 0x7f4d4c1f6310 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4d4c141360 0x7f4d4c15cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 --2- 172.21.15.59:0/2221588278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c0b1b00 0x7f4d4c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 >> 172.21.15.59:0/2221588278 conn(0x7f4d4c0b06e0 msgr2=0x7f4d4c15e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 shutdown_connections 2024-06-29T14:23:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.433+0000 7f4d51e3f640 1 -- 172.21.15.59:0/2221588278 wait complete. 2024-06-29T14:23:37.442 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-06-29T14:23:37.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:37.442 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-06-29T14:23:37.459 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-06-29T14:23:37.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.622+0000 7f521832a640 1 -- 172.21.15.59:0/1293165479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51f8003a70 msgr2=0x7f51f8003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.622+0000 7f521832a640 1 --2- 172.21.15.59:0/1293165479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51f8003a70 0x7f51f8003ee0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f520800bb20 tx=0x7f51f8005830 comp rx=0 tx=0).stop 2024-06-29T14:23:37.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.622+0000 7f52168c7640 1 -- 172.21.15.59:0/1293165479 reap_dead start 2024-06-29T14:23:37.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.622+0000 7f521832a640 1 -- 172.21.15.59:0/1293165479 shutdown_connections 2024-06-29T14:23:37.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.622+0000 7f521832a640 1 -- 172.21.15.59:0/1293165479 >> 172.21.15.59:0/1293165479 conn(0x7f52100b5610 msgr2=0x7f52100b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:37.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.622+0000 7f521832a640 1 -- 172.21.15.59:0/1293165479 shutdown_connections 2024-06-29T14:23:37.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.622+0000 7f521832a640 1 -- 172.21.15.59:0/1293165479 wait complete. 2024-06-29T14:23:37.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.624+0000 7f521832a640 1 Processor -- start 2024-06-29T14:23:37.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.624+0000 7f521832a640 1 -- start start 2024-06-29T14:23:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f521832a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8003a70 0x7f52101f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f521832a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210151400 0x7f5210151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f521832a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5210151d50 0x7f52101f8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f521832a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f52100b7bd0 con 0x7f5210151400 2024-06-29T14:23:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f521832a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52100cf2d0 con 0x7f51f8003a70 2024-06-29T14:23:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f521832a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f52100b7d50 con 0x7f5210151d50 2024-06-29T14:23:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52158c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210151400 0x7f5210151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52158c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210151400 0x7f5210151810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44210/0 (socket says 172.21.15.59:44210) 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52158c5640 1 -- 172.21.15.59:0/3984089525 learned_addr learned my addr 172.21.15.59:0/3984089525 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52160c6640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8003a70 0x7f52101f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52168c7640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5210151d50 0x7f52101f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52158c5640 1 -- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5210151d50 msgr2=0x7f52101f8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52158c5640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5210151d50 0x7f52101f8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52158c5640 1 -- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8003a70 msgr2=0x7f52101f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52158c5640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8003a70 0x7f52101f4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.625+0000 7f52158c5640 1 -- 172.21.15.59:0/3984089525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5210153910 con 0x7f5210151400 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.626+0000 7f52158c5640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210151400 0x7f5210151810 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5204001240 tx=0x7f52040039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.626+0000 7f520effd640 1 -- 172.21.15.59:0/3984089525 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204015070 con 0x7f5210151400 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.626+0000 7f520effd640 1 -- 172.21.15.59:0/3984089525 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5204011840 con 0x7f5210151400 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.626+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52100abae0 con 0x7f5210151400 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.626+0000 7f520effd640 1 -- 172.21.15.59:0/3984089525 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204011c90 con 0x7f5210151400 2024-06-29T14:23:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.626+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5210150c00 con 0x7f5210151400 2024-06-29T14:23:37.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.627+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52100ab850 con 0x7f5210151400 2024-06-29T14:23:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.628+0000 7f520effd640 1 -- 172.21.15.59:0/3984089525 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f5204018070 con 0x7f5210151400 2024-06-29T14:23:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.628+0000 7f520effd640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51fc0764d0 0x7f51fc0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.628+0000 7f520effd640 1 -- 172.21.15.59:0/3984089525 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f5204098970 con 0x7f5210151400 2024-06-29T14:23:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.628+0000 7f52160c6640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51fc0764d0 0x7f51fc0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.629+0000 7f52160c6640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51fc0764d0 0x7f51fc0789e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5208036000 tx=0x7f520803c040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:37.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.633+0000 7f520effd640 1 -- 172.21.15.59:0/3984089525 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5204061710 con 0x7f5210151400 2024-06-29T14:23:37.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.846+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f52100ac000 con 0x7f51fc0764d0 2024-06-29T14:23:37.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.924+0000 7f520effd640 1 -- 172.21.15.59:0/3984089525 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f52100ac000 con 0x7f51fc0764d0 2024-06-29T14:23:37.922 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir8 2024-06-29T14:23:37.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51fc0764d0 msgr2=0x7f51fc0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51fc0764d0 0x7f51fc0789e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5208036000 tx=0x7f520803c040 comp rx=0 tx=0).stop 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210151400 msgr2=0x7f5210151810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210151400 0x7f5210151810 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5204001240 tx=0x7f52040039c0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 shutdown_connections 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51fc0764d0 0x7f51fc0789e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5210151d50 0x7f52101f8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5210151400 0x7f5210151810 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 --2- 172.21.15.59:0/3984089525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51f8003a70 0x7f52101f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.926+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 >> 172.21.15.59:0/3984089525 conn(0x7f52100b5610 msgr2=0x7f521015dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.927+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 shutdown_connections 2024-06-29T14:23:37.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:37.927+0000 7f521832a640 1 -- 172.21.15.59:0/3984089525 wait complete. 2024-06-29T14:23:37.935 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-06-29T14:23:37.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:37.936 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-06-29T14:23:37.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-06-29T14:23:38.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.115+0000 7efeb994a640 1 -- 172.21.15.59:0/1511933756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40b2980 msgr2=0x7efeb415dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.115+0000 7efeb994a640 1 --2- 172.21.15.59:0/1511933756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40b2980 0x7efeb415dd10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7efeb40b6be0 tx=0x7efea402a410 comp rx=0 tx=0).stop 2024-06-29T14:23:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.115+0000 7efeb994a640 1 -- 172.21.15.59:0/1511933756 shutdown_connections 2024-06-29T14:23:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.115+0000 7efeb994a640 1 --2- 172.21.15.59:0/1511933756 >> v2:172.21.15.153:3300/0 conn(0x7efeb40cefb0 0x7efeb40d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.115+0000 7efeb994a640 1 --2- 172.21.15.59:0/1511933756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40b2980 0x7efeb415dd10 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.115+0000 7efeb994a640 1 -- 172.21.15.59:0/1511933756 >> 172.21.15.59:0/1511933756 conn(0x7efeb40d3b20 msgr2=0x7efeb40d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.115+0000 7efeb994a640 1 -- 172.21.15.59:0/1511933756 shutdown_connections 2024-06-29T14:23:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.115+0000 7efeb994a640 1 -- 172.21.15.59:0/1511933756 wait complete. 2024-06-29T14:23:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.117+0000 7efeb994a640 1 Processor -- start 2024-06-29T14:23:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.117+0000 7efeb994a640 1 -- start start 2024-06-29T14:23:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb994a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efeb40b2980 0x7efeb41f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb994a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40cefb0 0x7efeb4149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb994a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb40d2be0 0x7efeb4157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb994a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41620a0 con 0x7efeb40cefb0 2024-06-29T14:23:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb994a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efeb40b4db0 con 0x7efeb40d2be0 2024-06-29T14:23:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb994a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efeb4162220 con 0x7efeb40b2980 2024-06-29T14:23:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40cefb0 0x7efeb4149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40cefb0 0x7efeb4149c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44226/0 (socket says 172.21.15.59:44226) 2024-06-29T14:23:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 -- 172.21.15.59:0/705912478 learned_addr learned my addr 172.21.15.59:0/705912478 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb2ffd640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efeb40b2980 0x7efeb41f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb37fe640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb40d2be0 0x7efeb4157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 -- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efeb40b2980 msgr2=0x7efeb41f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efeb40b2980 0x7efeb41f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 -- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb40d2be0 msgr2=0x7efeb4157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb40d2be0 0x7efeb4157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 -- 172.21.15.59:0/705912478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb41553c0 con 0x7efeb40cefb0 2024-06-29T14:23:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efeb27fc640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40cefb0 0x7efeb4149c60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efea800d700 tx=0x7efea800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efe9bfff640 1 -- 172.21.15.59:0/705912478 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8017070 con 0x7efeb40cefb0 2024-06-29T14:23:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.119+0000 7efe9bfff640 1 -- 172.21.15.59:0/705912478 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efea8009b10 con 0x7efeb40cefb0 2024-06-29T14:23:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.120+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb41585e0 con 0x7efeb40cefb0 2024-06-29T14:23:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.120+0000 7efe9bfff640 1 -- 172.21.15.59:0/705912478 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8010460 con 0x7efeb40cefb0 2024-06-29T14:23:38.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.120+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb4155980 con 0x7efeb40cefb0 2024-06-29T14:23:38.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.121+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb4162740 con 0x7efeb40cefb0 2024-06-29T14:23:38.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.121+0000 7efe9bfff640 1 -- 172.21.15.59:0/705912478 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7efea801a070 con 0x7efeb40cefb0 2024-06-29T14:23:38.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.122+0000 7efe9bfff640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7efea00764d0 0x7efea00789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.122+0000 7efe9bfff640 1 -- 172.21.15.59:0/705912478 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7efea8097f20 con 0x7efeb40cefb0 2024-06-29T14:23:38.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.122+0000 7efeb2ffd640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7efea00764d0 0x7efea00789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.123+0000 7efeb2ffd640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7efea00764d0 0x7efea00789e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efea4034000 tx=0x7efea4004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.126+0000 7efe9bfff640 1 -- 172.21.15.59:0/705912478 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efea8060c30 con 0x7efeb40cefb0 2024-06-29T14:23:38.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.342+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7efeb40d1270 con 0x7efea00764d0 2024-06-29T14:23:38.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.407+0000 7efe9bfff640 1 -- 172.21.15.59:0/705912478 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7efeb40d1270 con 0x7efea00764d0 2024-06-29T14:23:38.406 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/dir9 2024-06-29T14:23:38.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7efea00764d0 msgr2=0x7efea00789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7efea00764d0 0x7efea00789e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efea4034000 tx=0x7efea4004730 comp rx=0 tx=0).stop 2024-06-29T14:23:38.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40cefb0 msgr2=0x7efeb4149c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40cefb0 0x7efeb4149c60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efea800d700 tx=0x7efea800dbc0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 shutdown_connections 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7efea00764d0 0x7efea00789e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb40d2be0 0x7efeb4157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efeb40cefb0 0x7efeb4149c60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 --2- 172.21.15.59:0/705912478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efeb40b2980 0x7efeb41f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 >> 172.21.15.59:0/705912478 conn(0x7efeb40d3b20 msgr2=0x7efeb40d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 shutdown_connections 2024-06-29T14:23:38.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.409+0000 7efeb994a640 1 -- 172.21.15.59:0/705912478 wait complete. 2024-06-29T14:23:38.419 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-06-29T14:23:38.419 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-06-29T14:23:38.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.541+0000 7f44e9688640 1 -- 172.21.15.59:0/1310073031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e415c2f0 msgr2=0x7f44e415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.541+0000 7f44e9688640 1 --2- 172.21.15.59:0/1310073031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e415c2f0 0x7f44e415c700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005fd0 tx=0x7f44d402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.542+0000 7f44e9688640 1 -- 172.21.15.59:0/1310073031 shutdown_connections 2024-06-29T14:23:38.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.542+0000 7f44e9688640 1 --2- 172.21.15.59:0/1310073031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e415c2f0 0x7f44e415c700 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.542+0000 7f44e9688640 1 --2- 172.21.15.59:0/1310073031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44e40b3740 0x7f44e40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.542+0000 7f44e9688640 1 -- 172.21.15.59:0/1310073031 >> 172.21.15.59:0/1310073031 conn(0x7f44e406c950 msgr2=0x7f44e4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.542+0000 7f44e9688640 1 -- 172.21.15.59:0/1310073031 shutdown_connections 2024-06-29T14:23:38.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.542+0000 7f44e9688640 1 -- 172.21.15.59:0/1310073031 wait complete. 2024-06-29T14:23:38.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.543+0000 7f44e9688640 1 Processor -- start 2024-06-29T14:23:38.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.544+0000 7f44e9688640 1 -- start start 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e9688640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e40b3740 0x7f44e41f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e9688640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44e415c2f0 0x7f44e40aa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e9688640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e40aaac0 0x7f44e40aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e9688640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f44e40b5fd0 con 0x7f44e40aaac0 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e9688640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44e40b5e50 con 0x7f44e40b3740 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e9688640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f44e40b6150 con 0x7f44e415c2f0 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e40aaac0 0x7f44e40aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e40aaac0 0x7f44e40aaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44236/0 (socket says 172.21.15.59:44236) 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e37fe640 1 -- 172.21.15.59:0/3291967968 learned_addr learned my addr 172.21.15.59:0/3291967968 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e27fc640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44e415c2f0 0x7f44e40aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e2ffd640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e40b3740 0x7f44e41f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e37fe640 1 -- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44e415c2f0 msgr2=0x7f44e40aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e37fe640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44e415c2f0 0x7f44e40aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e37fe640 1 -- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e40b3740 msgr2=0x7f44e41f5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e37fe640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e40b3740 0x7f44e41f5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.545+0000 7f44e37fe640 1 -- 172.21.15.59:0/3291967968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e41e6ee0 con 0x7f44e40aaac0 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.546+0000 7f44e37fe640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e40aaac0 0x7f44e40aaed0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f44d4006010 tx=0x7f44d4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.546+0000 7f44db7fe640 1 -- 172.21.15.59:0/3291967968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4031b90 con 0x7f44e40aaac0 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.546+0000 7f44db7fe640 1 -- 172.21.15.59:0/3291967968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44d4031e20 con 0x7f44e40aaac0 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.546+0000 7f44db7fe640 1 -- 172.21.15.59:0/3291967968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d40332f0 con 0x7f44e40aaac0 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.546+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e40a9aa0 con 0x7f44e40aaac0 2024-06-29T14:23:38.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.546+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e40aa060 con 0x7f44e40aaac0 2024-06-29T14:23:38.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.547+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44e40a9890 con 0x7f44e40aaac0 2024-06-29T14:23:38.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.548+0000 7f44db7fe640 1 -- 172.21.15.59:0/3291967968 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f44d4031660 con 0x7f44e40aaac0 2024-06-29T14:23:38.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.548+0000 7f44db7fe640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f44c40764d0 0x7f44c40789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.548+0000 7f44db7fe640 1 -- 172.21.15.59:0/3291967968 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f44d40b9400 con 0x7f44e40aaac0 2024-06-29T14:23:38.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.548+0000 7f44e2ffd640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f44c40764d0 0x7f44c40789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.549+0000 7f44e2ffd640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f44c40764d0 0x7f44c40789e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f44dc000f60 tx=0x7f44dc009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.553+0000 7f44db7fe640 1 -- 172.21.15.59:0/3291967968 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f44d403f030 con 0x7f44e40aaac0 2024-06-29T14:23:38.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.769+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f44e40ab410 con 0x7f44c40764d0 2024-06-29T14:23:38.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.819+0000 7f44db7fe640 1 -- 172.21.15.59:0/3291967968 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f44e40ab410 con 0x7f44c40764d0 2024-06-29T14:23:38.817 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-06-29T14:23:38.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f44c40764d0 msgr2=0x7f44c40789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f44c40764d0 0x7f44c40789e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f44dc000f60 tx=0x7f44dc009040 comp rx=0 tx=0).stop 2024-06-29T14:23:38.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e40aaac0 msgr2=0x7f44e40aaed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e40aaac0 0x7f44e40aaed0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f44d4006010 tx=0x7f44d4003ba0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 shutdown_connections 2024-06-29T14:23:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f44c40764d0 0x7f44c40789e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e40aaac0 0x7f44e40aaed0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44e415c2f0 0x7f44e40aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 --2- 172.21.15.59:0/3291967968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44e40b3740 0x7f44e41f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 >> 172.21.15.59:0/3291967968 conn(0x7f44e406c950 msgr2=0x7f44e4071330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 shutdown_connections 2024-06-29T14:23:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.821+0000 7f44e9688640 1 -- 172.21.15.59:0/3291967968 wait complete. 2024-06-29T14:23:38.830 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-06-29T14:23:38.830 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-06-29T14:23:38.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.954+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2518824739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0ce950 msgr2=0x7f9fdc1622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.954+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2518824739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0ce950 0x7f9fdc1622c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009830 tx=0x7f9fd002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.954+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2518824739 shutdown_connections 2024-06-29T14:23:38.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.954+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2518824739 >> v2:172.21.15.153:3300/0 conn(0x7f9fdc162800 0x7f9fdc162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.954+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2518824739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0ce950 0x7f9fdc1622c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.954+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2518824739 >> 172.21.15.59:0/2518824739 conn(0x7f9fdc0b5610 msgr2=0x7f9fdc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.954+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2518824739 shutdown_connections 2024-06-29T14:23:38.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.954+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2518824739 wait complete. 2024-06-29T14:23:38.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.956+0000 7f9fe15b2640 1 Processor -- start 2024-06-29T14:23:38.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.956+0000 7f9fe15b2640 1 -- start start 2024-06-29T14:23:38.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fe15b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc08c190 0x7f9fdc15be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fe15b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc0ce950 0x7f9fdc1f8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fe15b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc162800 0x7f9fdc0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fe15b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc0b7f60 con 0x7f9fdc08c190 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fe15b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc0b7de0 con 0x7f9fdc0ce950 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fe15b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9fdc0b80e0 con 0x7f9fdc162800 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fdaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc08c190 0x7f9fdc15be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fdaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc08c190 0x7f9fdc15be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44240/0 (socket says 172.21.15.59:44240) 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fdaffd640 1 -- 172.21.15.59:0/2953284630 learned_addr learned my addr 172.21.15.59:0/2953284630 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fdb7fe640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc162800 0x7f9fdc0ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.957+0000 7f9fda7fc640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc0ce950 0x7f9fdc1f8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fdaffd640 1 -- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc162800 msgr2=0x7f9fdc0ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fdaffd640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc162800 0x7f9fdc0ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fdaffd640 1 -- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc0ce950 msgr2=0x7f9fdc1f8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fdaffd640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc0ce950 0x7f9fdc1f8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fdaffd640 1 -- 172.21.15.59:0/2953284630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc1f9480 con 0x7f9fdc08c190 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fda7fc640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc0ce950 0x7f9fdc1f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fdb7fe640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc162800 0x7f9fdc0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fdaffd640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc08c190 0x7f9fdc15be80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00cb40 tx=0x7f9fcc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fc37fe640 1 -- 172.21.15.59:0/2953284630 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc016020 con 0x7f9fdc08c190 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fc37fe640 1 -- 172.21.15.59:0/2953284630 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fcc00adb0 con 0x7f9fdc08c190 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fc37fe640 1 -- 172.21.15.59:0/2953284630 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc012680 con 0x7f9fdc08c190 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fdc0cd3c0 con 0x7f9fdc08c190 2024-06-29T14:23:38.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.958+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fdc0cad70 con 0x7f9fdc08c190 2024-06-29T14:23:38.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.959+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fdc1622c0 con 0x7f9fdc08c190 2024-06-29T14:23:38.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.960+0000 7f9fc37fe640 1 -- 172.21.15.59:0/2953284630 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f9fcc019020 con 0x7f9fdc08c190 2024-06-29T14:23:38.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.960+0000 7f9fc37fe640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9fc8076500 0x7f9fc8078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.960+0000 7f9fc37fe640 1 -- 172.21.15.59:0/2953284630 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f9fcc098570 con 0x7f9fdc08c190 2024-06-29T14:23:38.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.961+0000 7f9fda7fc640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9fc8076500 0x7f9fc8078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.961+0000 7f9fda7fc640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9fc8076500 0x7f9fc8078a10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0034000 tx=0x7f9fd0004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:38.965+0000 7f9fc37fe640 1 -- 172.21.15.59:0/2953284630 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9fcc020020 con 0x7f9fdc08c190 2024-06-29T14:23:39.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.185+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9fdc15c3c0 con 0x7f9fc8076500 2024-06-29T14:23:39.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.249+0000 7f9fc37fe640 1 -- 172.21.15.59:0/2953284630 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f9fd0004290 con 0x7f9fc8076500 2024-06-29T14:23:39.248 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.251+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9fc8076500 msgr2=0x7f9fc8078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.251+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9fc8076500 0x7f9fc8078a10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0034000 tx=0x7f9fd0004730 comp rx=0 tx=0).stop 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.251+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc08c190 msgr2=0x7f9fdc15be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.251+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc08c190 0x7f9fdc15be80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00cb40 tx=0x7f9fcc00a060 comp rx=0 tx=0).stop 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.252+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 shutdown_connections 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.252+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f9fc8076500 0x7f9fc8078a10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.252+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc162800 0x7f9fdc0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.252+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc0ce950 0x7f9fdc1f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.252+0000 7f9fe15b2640 1 --2- 172.21.15.59:0/2953284630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc08c190 0x7f9fdc15be80 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.252+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 >> 172.21.15.59:0/2953284630 conn(0x7f9fdc0b5610 msgr2=0x7f9fdc0c7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.252+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 shutdown_connections 2024-06-29T14:23:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.252+0000 7f9fe15b2640 1 -- 172.21.15.59:0/2953284630 wait complete. 2024-06-29T14:23:39.261 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-06-29T14:23:39.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-06-29T14:23:39.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.383+0000 7fea2ad7f640 1 -- 172.21.15.59:0/249938909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240d23c0 msgr2=0x7fea240b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.383+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/249938909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240d23c0 0x7fea240b3990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fea24161d40 tx=0x7fea1802a530 comp rx=0 tx=0).stop 2024-06-29T14:23:39.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.383+0000 7fea2ad7f640 1 -- 172.21.15.59:0/249938909 shutdown_connections 2024-06-29T14:23:39.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.383+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/249938909 >> v2:172.21.15.153:3300/0 conn(0x7fea2415efe0 0x7fea2415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.383+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/249938909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240d23c0 0x7fea240b3990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.383+0000 7fea2ad7f640 1 -- 172.21.15.59:0/249938909 >> 172.21.15.59:0/249938909 conn(0x7fea240d3ee0 msgr2=0x7fea240d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:39.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.383+0000 7fea2ad7f640 1 -- 172.21.15.59:0/249938909 shutdown_connections 2024-06-29T14:23:39.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.383+0000 7fea2ad7f640 1 -- 172.21.15.59:0/249938909 wait complete. 2024-06-29T14:23:39.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.385+0000 7fea2ad7f640 1 Processor -- start 2024-06-29T14:23:39.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.385+0000 7fea2ad7f640 1 -- start start 2024-06-29T14:23:39.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea2ad7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240b3ed0 0x7fea24148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:39.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea2ad7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea240d23c0 0x7fea24146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:39.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea2ad7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea2415efe0 0x7fea241ea690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:39.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea2ad7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fea24160280 con 0x7fea240b3ed0 2024-06-29T14:23:39.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea2ad7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea24160100 con 0x7fea240d23c0 2024-06-29T14:23:39.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea2ad7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fea24160400 con 0x7fea2415efe0 2024-06-29T14:23:39.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240b3ed0 0x7fea24148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:39.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240b3ed0 0x7fea24148150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44256/0 (socket says 172.21.15.59:44256) 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 -- 172.21.15.59:0/2856986817 learned_addr learned my addr 172.21.15.59:0/2856986817 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea23fff640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea240d23c0 0x7fea24146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea2931c640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea2415efe0 0x7fea241ea690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 -- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea2415efe0 msgr2=0x7fea241ea690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea2415efe0 0x7fea241ea690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 -- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea240d23c0 msgr2=0x7fea24146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea240d23c0 0x7fea24146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 -- 172.21.15.59:0/2856986817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea24145390 con 0x7fea240b3ed0 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea28b1b640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240b3ed0 0x7fea24148150 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fea18009780 tx=0x7fea18033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.387+0000 7fea217fa640 1 -- 172.21.15.59:0/2856986817 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea18031d60 con 0x7fea240b3ed0 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.388+0000 7fea217fa640 1 -- 172.21.15.59:0/2856986817 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea18033e10 con 0x7fea240b3ed0 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.388+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea24145670 con 0x7fea240b3ed0 2024-06-29T14:23:39.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.388+0000 7fea217fa640 1 -- 172.21.15.59:0/2856986817 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea18005260 con 0x7fea240b3ed0 2024-06-29T14:23:39.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.388+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea24145c30 con 0x7fea240b3ed0 2024-06-29T14:23:39.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.389+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea240b3990 con 0x7fea240b3ed0 2024-06-29T14:23:39.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.389+0000 7fea217fa640 1 -- 172.21.15.59:0/2856986817 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7fea18005400 con 0x7fea240b3ed0 2024-06-29T14:23:39.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.390+0000 7fea217fa640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fea080764d0 0x7fea080789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:39.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.390+0000 7fea217fa640 1 -- 172.21.15.59:0/2856986817 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7fea180b8f80 con 0x7fea240b3ed0 2024-06-29T14:23:39.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.390+0000 7fea23fff640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fea080764d0 0x7fea080789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:39.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.391+0000 7fea23fff640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fea080764d0 0x7fea080789e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fea241585e0 tx=0x7fea10007e90 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:39.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.394+0000 7fea217fa640 1 -- 172.21.15.59:0/2856986817 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fea1803f030 con 0x7fea240b3ed0 2024-06-29T14:23:39.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.611+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fea240d2950 con 0x7fea080764d0 2024-06-29T14:23:39.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.663+0000 7fea217fa640 1 -- 172.21.15.59:0/2856986817 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fea240d2950 con 0x7fea080764d0 2024-06-29T14:23:39.661 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-06-29T14:23:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fea080764d0 msgr2=0x7fea080789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fea080764d0 0x7fea080789e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fea241585e0 tx=0x7fea10007e90 comp rx=0 tx=0).stop 2024-06-29T14:23:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240b3ed0 msgr2=0x7fea24148150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240b3ed0 0x7fea24148150 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fea18009780 tx=0x7fea18033c20 comp rx=0 tx=0).stop 2024-06-29T14:23:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 shutdown_connections 2024-06-29T14:23:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fea080764d0 0x7fea080789e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea2415efe0 0x7fea241ea690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea240d23c0 0x7fea24146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 --2- 172.21.15.59:0/2856986817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea240b3ed0 0x7fea24148150 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 >> 172.21.15.59:0/2856986817 conn(0x7fea240d3ee0 msgr2=0x7fea240ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:39.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 shutdown_connections 2024-06-29T14:23:39.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.665+0000 7fea2ad7f640 1 -- 172.21.15.59:0/2856986817 wait complete. 2024-06-29T14:23:39.674 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-06-29T14:23:39.674 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-06-29T14:23:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.798+0000 7f8003faa640 1 -- 172.21.15.59:0/1385630664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ffc0b2980 msgr2=0x7f7ffc15dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.798+0000 7f8003faa640 1 --2- 172.21.15.59:0/1385630664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ffc0b2980 0x7f7ffc15dd10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8005fd0 tx=0x7f7ff802a820 comp rx=0 tx=0).stop 2024-06-29T14:23:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.798+0000 7f8003faa640 1 -- 172.21.15.59:0/1385630664 shutdown_connections 2024-06-29T14:23:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.798+0000 7f8003faa640 1 --2- 172.21.15.59:0/1385630664 >> v2:172.21.15.153:3300/0 conn(0x7f7ffc0cefb0 0x7f7ffc0d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.798+0000 7f8003faa640 1 --2- 172.21.15.59:0/1385630664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ffc0b2980 0x7f7ffc15dd10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.798+0000 7f8003faa640 1 -- 172.21.15.59:0/1385630664 >> 172.21.15.59:0/1385630664 conn(0x7f7ffc0d3b20 msgr2=0x7f7ffc0d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:39.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.798+0000 7f8003faa640 1 -- 172.21.15.59:0/1385630664 shutdown_connections 2024-06-29T14:23:39.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.798+0000 7f8003faa640 1 -- 172.21.15.59:0/1385630664 wait complete. 2024-06-29T14:23:39.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.800+0000 7f8003faa640 1 Processor -- start 2024-06-29T14:23:39.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.800+0000 7f8003faa640 1 -- start start 2024-06-29T14:23:39.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8003faa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ffc0b2980 0x7f7ffc15a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:39.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8003faa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0cefb0 0x7f7ffc149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:39.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8003faa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ffc0d2be0 0x7f7ffc1571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:39.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8003faa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7ffc1620a0 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8003faa640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7ffc0b4db0 con 0x7f7ffc0d2be0 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8003faa640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7ffc162220 con 0x7f7ffc0b2980 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001545640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0cefb0 0x7f7ffc149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001545640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0cefb0 0x7f7ffc149d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44260/0 (socket says 172.21.15.59:44260) 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001545640 1 -- 172.21.15.59:0/4254074915 learned_addr learned my addr 172.21.15.59:0/4254074915 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001d46640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ffc0b2980 0x7f7ffc15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8002547640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ffc0d2be0 0x7f7ffc1571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001545640 1 -- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ffc0b2980 msgr2=0x7f7ffc15a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001545640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ffc0b2980 0x7f7ffc15a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001545640 1 -- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ffc0d2be0 msgr2=0x7f7ffc1571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001545640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ffc0d2be0 0x7f7ffc1571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.801+0000 7f8001545640 1 -- 172.21.15.59:0/4254074915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ffc155b90 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.802+0000 7f8001545640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0cefb0 0x7f7ffc149d60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00e9f0 tx=0x7f7fec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:39.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.802+0000 7f7ff27fc640 1 -- 172.21.15.59:0/4254074915 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fec00cd80 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.802+0000 7f7ff27fc640 1 -- 172.21.15.59:0/4254074915 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fec004020 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.802+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ffc155e70 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.802+0000 7f7ff27fc640 1 -- 172.21.15.59:0/4254074915 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fec004470 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.802+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ffc156430 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.803+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ffc162740 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.804+0000 7f7ff27fc640 1 -- 172.21.15.59:0/4254074915 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f7fec0035f0 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.804+0000 7f7ff27fc640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7fe00764d0 0x7f7fe00789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:39.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.804+0000 7f7ff27fc640 1 -- 172.21.15.59:0/4254074915 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f7fec0988a0 con 0x7f7ffc0cefb0 2024-06-29T14:23:39.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.804+0000 7f8001d46640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7fe00764d0 0x7f7fe00789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:39.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.805+0000 7f8001d46640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7fe00764d0 0x7f7fe00789e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8034000 tx=0x7f7ff8004ab0 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:39.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:39.808+0000 7f7ff27fc640 1 -- 172.21.15.59:0/4254074915 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7fec061480 con 0x7f7ffc0cefb0 2024-06-29T14:23:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.023+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f7ffc15d380 con 0x7f7fe00764d0 2024-06-29T14:23:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.070+0000 7f7ff27fc640 1 -- 172.21.15.59:0/4254074915 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f7ffc15d380 con 0x7f7fe00764d0 2024-06-29T14:23:40.068 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-06-29T14:23:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7fe00764d0 msgr2=0x7f7fe00789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7fe00764d0 0x7f7fe00789e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8034000 tx=0x7f7ff8004ab0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0cefb0 msgr2=0x7f7ffc149d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0cefb0 0x7f7ffc149d60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00e9f0 tx=0x7f7fec00eeb0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 shutdown_connections 2024-06-29T14:23:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7fe00764d0 0x7f7fe00789e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ffc0d2be0 0x7f7ffc1571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc0cefb0 0x7f7ffc149d60 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 --2- 172.21.15.59:0/4254074915 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ffc0b2980 0x7f7ffc15a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 >> 172.21.15.59:0/4254074915 conn(0x7f7ffc0d3b20 msgr2=0x7f7ffc06d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 shutdown_connections 2024-06-29T14:23:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.072+0000 7f8003faa640 1 -- 172.21.15.59:0/4254074915 wait complete. 2024-06-29T14:23:40.081 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-06-29T14:23:40.081 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-06-29T14:23:40.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.205+0000 7f24f6135640 1 -- 172.21.15.59:0/1202896752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 msgr2=0x7f24f0140d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.205+0000 7f24f6135640 1 --2- 172.21.15.59:0/1202896752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 0x7f24f0140d10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009830 tx=0x7f24e002a510 comp rx=0 tx=0).stop 2024-06-29T14:23:40.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.206+0000 7f24f6135640 1 -- 172.21.15.59:0/1202896752 shutdown_connections 2024-06-29T14:23:40.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.206+0000 7f24f6135640 1 --2- 172.21.15.59:0/1202896752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f0074450 0x7f24f015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.206+0000 7f24f6135640 1 --2- 172.21.15.59:0/1202896752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 0x7f24f0140d10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.206+0000 7f24f6135640 1 -- 172.21.15.59:0/1202896752 >> 172.21.15.59:0/1202896752 conn(0x7f24f00cd4c0 msgr2=0x7f24f00cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:40.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.206+0000 7f24f6135640 1 -- 172.21.15.59:0/1202896752 shutdown_connections 2024-06-29T14:23:40.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.206+0000 7f24f6135640 1 -- 172.21.15.59:0/1202896752 wait complete. 2024-06-29T14:23:40.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.207+0000 7f24f6135640 1 Processor -- start 2024-06-29T14:23:40.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.208+0000 7f24f6135640 1 -- start start 2024-06-29T14:23:40.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24f6135640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f013d990 0x7f24f013d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24f6135640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 0x7f24f0156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24f6135640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24f01587b0 0x7f24f0159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24f6135640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f24f015f540 con 0x7f24f01587b0 2024-06-29T14:23:40.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24f6135640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f24f015f3c0 con 0x7f24f015b2c0 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24f6135640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f24f015f6c0 con 0x7f24f013d990 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24ef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f013d990 0x7f24f013d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24effff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24f01587b0 0x7f24f0159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24e6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 0x7f24f0156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24ef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f013d990 0x7f24f013d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:35612/0 (socket says 172.21.15.59:35612) 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.209+0000 7f24ef7fe640 1 -- 172.21.15.59:0/3565265471 learned_addr learned my addr 172.21.15.59:0/3565265471 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24effff640 1 -- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f013d990 msgr2=0x7f24f013d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24effff640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f013d990 0x7f24f013d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24effff640 1 -- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 msgr2=0x7f24f0156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24effff640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 0x7f24f0156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24effff640 1 -- 172.21.15.59:0/3565265471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f015a980 con 0x7f24f01587b0 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24e6ffd640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 0x7f24f0156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24effff640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24f01587b0 0x7f24f0159fa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f24e8005fd0 tx=0x7f24e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:40.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24ecff9640 1 -- 172.21.15.59:0/3565265471 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e800ce50 con 0x7f24f01587b0 2024-06-29T14:23:40.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24ecff9640 1 -- 172.21.15.59:0/3565265471 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24e8003970 con 0x7f24f01587b0 2024-06-29T14:23:40.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24ecff9640 1 -- 172.21.15.59:0/3565265471 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8003dc0 con 0x7f24f01587b0 2024-06-29T14:23:40.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f0155ec0 con 0x7f24f01587b0 2024-06-29T14:23:40.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.210+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f0156480 con 0x7f24f01587b0 2024-06-29T14:23:40.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.211+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f015a6d0 con 0x7f24f01587b0 2024-06-29T14:23:40.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.212+0000 7f24ecff9640 1 -- 172.21.15.59:0/3565265471 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f24e8003f60 con 0x7f24f01587b0 2024-06-29T14:23:40.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.213+0000 7f24ecff9640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f24d00766a0 0x7f24d0078bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.213+0000 7f24ecff9640 1 -- 172.21.15.59:0/3565265471 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f24e80987b0 con 0x7f24f01587b0 2024-06-29T14:23:40.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.213+0000 7f24ef7fe640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f24d00766a0 0x7f24d0078bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.213+0000 7f24ef7fe640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f24d00766a0 0x7f24d0078bb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f24e00095f0 tx=0x7f24e0004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:40.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.217+0000 7f24ecff9640 1 -- 172.21.15.59:0/3565265471 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f24e80614a0 con 0x7f24f01587b0 2024-06-29T14:23:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.435+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f24f015ee80 con 0x7f24d00766a0 2024-06-29T14:23:40.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.483+0000 7f24ecff9640 1 -- 172.21.15.59:0/3565265471 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f24f015ee80 con 0x7f24d00766a0 2024-06-29T14:23:40.481 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-06-29T14:23:40.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f24d00766a0 msgr2=0x7f24d0078bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f24d00766a0 0x7f24d0078bb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f24e00095f0 tx=0x7f24e0004730 comp rx=0 tx=0).stop 2024-06-29T14:23:40.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24f01587b0 msgr2=0x7f24f0159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24f01587b0 0x7f24f0159fa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f24e8005fd0 tx=0x7f24e800c2d0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 shutdown_connections 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f24d00766a0 0x7f24d0078bb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24f01587b0 0x7f24f0159fa0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24f015b2c0 0x7f24f0156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 --2- 172.21.15.59:0/3565265471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24f013d990 0x7f24f013d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.485+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 >> 172.21.15.59:0/3565265471 conn(0x7f24f00cd4c0 msgr2=0x7f24f01402a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.486+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 shutdown_connections 2024-06-29T14:23:40.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.486+0000 7f24f6135640 1 -- 172.21.15.59:0/3565265471 wait complete. 2024-06-29T14:23:40.494 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-06-29T14:23:40.495 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-06-29T14:23:40.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.617+0000 7fbc482ac640 1 -- 172.21.15.59:0/2112462662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4015ee10 msgr2=0x7fbc4015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.617+0000 7fbc482ac640 1 --2- 172.21.15.59:0/2112462662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4015ee10 0x7fbc4015f9b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009830 tx=0x7fbc3802a510 comp rx=0 tx=0).stop 2024-06-29T14:23:40.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.617+0000 7fbc482ac640 1 -- 172.21.15.59:0/2112462662 shutdown_connections 2024-06-29T14:23:40.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.617+0000 7fbc482ac640 1 --2- 172.21.15.59:0/2112462662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4015ee10 0x7fbc4015f9b0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.617+0000 7fbc482ac640 1 --2- 172.21.15.59:0/2112462662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc400b40a0 0x7fbc4015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.617+0000 7fbc482ac640 1 -- 172.21.15.59:0/2112462662 >> 172.21.15.59:0/2112462662 conn(0x7fbc400d3ee0 msgr2=0x7fbc400d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:40.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.618+0000 7fbc482ac640 1 -- 172.21.15.59:0/2112462662 shutdown_connections 2024-06-29T14:23:40.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.618+0000 7fbc482ac640 1 -- 172.21.15.59:0/2112462662 wait complete. 2024-06-29T14:23:40.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.619+0000 7fbc482ac640 1 Processor -- start 2024-06-29T14:23:40.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.619+0000 7fbc482ac640 1 -- start start 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc482ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc400d23c0 0x7fbc4015c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc482ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4015ee10 0x7fbc4014eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc482ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc400ca130 0x7fbc400ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc482ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbc40160280 con 0x7fbc400d23c0 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc482ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc40160100 con 0x7fbc4015ee10 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc482ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbc40160400 con 0x7fbc400ca130 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc400d23c0 0x7fbc4015c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc400d23c0 0x7fbc4015c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44282/0 (socket says 172.21.15.59:44282) 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 -- 172.21.15.59:0/24497939 learned_addr learned my addr 172.21.15.59:0/24497939 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46849640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc400ca130 0x7fbc400ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc45847640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4015ee10 0x7fbc4014eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 -- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc400ca130 msgr2=0x7fbc400ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc400ca130 0x7fbc400ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 -- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4015ee10 msgr2=0x7fbc4014eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4015ee10 0x7fbc4014eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 -- 172.21.15.59:0/24497939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc400caa80 con 0x7fbc400d23c0 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46849640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc400ca130 0x7fbc400ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc45847640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4015ee10 0x7fbc4014eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc46048640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc400d23c0 0x7fbc4015c900 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00c710 tx=0x7fbc3c00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.621+0000 7fbc36ffd640 1 -- 172.21.15.59:0/24497939 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c017070 con 0x7fbc400d23c0 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.622+0000 7fbc36ffd640 1 -- 172.21.15.59:0/24497939 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc3c00ce20 con 0x7fbc400d23c0 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.622+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4014f430 con 0x7fbc400d23c0 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.622+0000 7fbc36ffd640 1 -- 172.21.15.59:0/24497939 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c0052d0 con 0x7fbc400d23c0 2024-06-29T14:23:40.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.622+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4015d410 con 0x7fbc400d23c0 2024-06-29T14:23:40.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.623+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc400b8d00 con 0x7fbc400d23c0 2024-06-29T14:23:40.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.623+0000 7fbc36ffd640 1 -- 172.21.15.59:0/24497939 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7fbc3c01a070 con 0x7fbc400d23c0 2024-06-29T14:23:40.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.624+0000 7fbc36ffd640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbc240764d0 0x7fbc240789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.624+0000 7fbc36ffd640 1 -- 172.21.15.59:0/24497939 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7fbc3c0974a0 con 0x7fbc400d23c0 2024-06-29T14:23:40.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.624+0000 7fbc45847640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbc240764d0 0x7fbc240789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.625+0000 7fbc45847640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbc240764d0 0x7fbc240789e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc30005fd0 tx=0x7fbc30005ea0 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:40.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.628+0000 7fbc36ffd640 1 -- 172.21.15.59:0/24497939 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbc400b8d00 con 0x7fbc400d23c0 2024-06-29T14:23:40.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.843+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fbc400c79f0 con 0x7fbc240764d0 2024-06-29T14:23:40.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.889+0000 7fbc36ffd640 1 -- 172.21.15.59:0/24497939 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fbc400c79f0 con 0x7fbc240764d0 2024-06-29T14:23:40.887 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-06-29T14:23:40.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbc240764d0 msgr2=0x7fbc240789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbc240764d0 0x7fbc240789e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc30005fd0 tx=0x7fbc30005ea0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc400d23c0 msgr2=0x7fbc4015c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc400d23c0 0x7fbc4015c900 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00c710 tx=0x7fbc3c00cbd0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 shutdown_connections 2024-06-29T14:23:40.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbc240764d0 0x7fbc240789e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc400ca130 0x7fbc400ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc4015ee10 0x7fbc4014eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 --2- 172.21.15.59:0/24497939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc400d23c0 0x7fbc4015c900 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 >> 172.21.15.59:0/24497939 conn(0x7fbc400d3ee0 msgr2=0x7fbc400b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:40.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 shutdown_connections 2024-06-29T14:23:40.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:40.891+0000 7fbc482ac640 1 -- 172.21.15.59:0/24497939 wait complete. 2024-06-29T14:23:40.900 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-06-29T14:23:40.900 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-06-29T14:23:41.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.024+0000 7fb31b28f640 1 -- 172.21.15.59:0/592280847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 msgr2=0x7fb314166ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.024+0000 7fb31b28f640 1 --2- 172.21.15.59:0/592280847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 0x7fb314166ca0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb30c009830 tx=0x7fb30c02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.024+0000 7fb31b28f640 1 -- 172.21.15.59:0/592280847 shutdown_connections 2024-06-29T14:23:41.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.024+0000 7fb31b28f640 1 --2- 172.21.15.59:0/592280847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 0x7fb314166ca0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.024+0000 7fb31b28f640 1 -- 172.21.15.59:0/592280847 >> 172.21.15.59:0/592280847 conn(0x7fb31406c950 msgr2=0x7fb314073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.024+0000 7fb31b28f640 1 -- 172.21.15.59:0/592280847 shutdown_connections 2024-06-29T14:23:41.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.024+0000 7fb31b28f640 1 -- 172.21.15.59:0/592280847 wait complete. 2024-06-29T14:23:41.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.026+0000 7fb31b28f640 1 Processor -- start 2024-06-29T14:23:41.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.026+0000 7fb31b28f640 1 -- start start 2024-06-29T14:23:41.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31b28f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3140b3740 0x7fb31415b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31b28f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 0x7fb314140570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31b28f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3141f2d90 0x7fb31415cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31b28f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb314167570 con 0x7fb3140b3b50 2024-06-29T14:23:41.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31b28f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141673f0 con 0x7fb3141f2d90 2024-06-29T14:23:41.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31b28f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb3141676f0 con 0x7fb3140b3740 2024-06-29T14:23:41.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31882a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 0x7fb314140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31882a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 0x7fb314140570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44298/0 (socket says 172.21.15.59:44298) 2024-06-29T14:23:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31882a640 1 -- 172.21.15.59:0/1760937344 learned_addr learned my addr 172.21.15.59:0/1760937344 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31982c640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3141f2d90 0x7fb31415cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.027+0000 7fb31902b640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3140b3740 0x7fb31415b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb31882a640 1 -- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3140b3740 msgr2=0x7fb31415b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb31882a640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3140b3740 0x7fb31415b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb31882a640 1 -- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3141f2d90 msgr2=0x7fb31415cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb31882a640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3141f2d90 0x7fb31415cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb31882a640 1 -- 172.21.15.59:0/1760937344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb314140ab0 con 0x7fb3140b3b50 2024-06-29T14:23:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb31882a640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 0x7fb314140570 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb30800c3d0 tx=0x7fb30800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb311ffb640 1 -- 172.21.15.59:0/1760937344 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308017070 con 0x7fb3140b3b50 2024-06-29T14:23:41.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb311ffb640 1 -- 172.21.15.59:0/1760937344 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3080053c0 con 0x7fb3140b3b50 2024-06-29T14:23:41.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb314141510 con 0x7fb3140b3b50 2024-06-29T14:23:41.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb311ffb640 1 -- 172.21.15.59:0/1760937344 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308003850 con 0x7fb3140b3b50 2024-06-29T14:23:41.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.028+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3141417d0 con 0x7fb3140b3b50 2024-06-29T14:23:41.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.029+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb31413ae50 con 0x7fb3140b3b50 2024-06-29T14:23:41.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.030+0000 7fb311ffb640 1 -- 172.21.15.59:0/1760937344 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7fb30801a070 con 0x7fb3140b3b50 2024-06-29T14:23:41.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.031+0000 7fb311ffb640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb300076500 0x7fb300078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.031+0000 7fb311ffb640 1 -- 172.21.15.59:0/1760937344 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7fb308096fe0 con 0x7fb3140b3b50 2024-06-29T14:23:41.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.031+0000 7fb31902b640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb300076500 0x7fb300078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.032+0000 7fb31902b640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb300076500 0x7fb300078a10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb304004a30 tx=0x7fb304001a50 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.035+0000 7fb311ffb640 1 -- 172.21.15.59:0/1760937344 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb30805fee0 con 0x7fb3140b3b50 2024-06-29T14:23:41.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.250+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb3141458a0 con 0x7fb300076500 2024-06-29T14:23:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.295+0000 7fb311ffb640 1 -- 172.21.15.59:0/1760937344 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb3141458a0 con 0x7fb300076500 2024-06-29T14:23:41.293 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb300076500 msgr2=0x7fb300078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb300076500 0x7fb300078a10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb304004a30 tx=0x7fb304001a50 comp rx=0 tx=0).stop 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 msgr2=0x7fb314140570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 0x7fb314140570 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb30800c3d0 tx=0x7fb30800c890 comp rx=0 tx=0).stop 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 shutdown_connections 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb300076500 0x7fb300078a10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3141f2d90 0x7fb31415cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3140b3b50 0x7fb314140570 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 --2- 172.21.15.59:0/1760937344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3140b3740 0x7fb31415b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 >> 172.21.15.59:0/1760937344 conn(0x7fb31406c950 msgr2=0x7fb3140b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 shutdown_connections 2024-06-29T14:23:41.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.298+0000 7fb31b28f640 1 -- 172.21.15.59:0/1760937344 wait complete. 2024-06-29T14:23:41.307 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-06-29T14:23:41.307 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-06-29T14:23:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.429+0000 7f27e629d640 1 -- 172.21.15.59:0/226393467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 msgr2=0x7f27e00cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.429+0000 7f27e629d640 1 --2- 172.21.15.59:0/226393467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 0x7f27e00cf1d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f27d8005fd0 tx=0x7f27d802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.429+0000 7f27e629d640 1 -- 172.21.15.59:0/226393467 shutdown_connections 2024-06-29T14:23:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.429+0000 7f27e629d640 1 --2- 172.21.15.59:0/226393467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 0x7f27e00cf1d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.429+0000 7f27e629d640 1 --2- 172.21.15.59:0/226393467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27e0144470 0x7f27e00b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.429+0000 7f27e629d640 1 -- 172.21.15.59:0/226393467 >> 172.21.15.59:0/226393467 conn(0x7f27e00b06e0 msgr2=0x7f27e00b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.429+0000 7f27e629d640 1 -- 172.21.15.59:0/226393467 shutdown_connections 2024-06-29T14:23:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.429+0000 7f27e629d640 1 -- 172.21.15.59:0/226393467 wait complete. 2024-06-29T14:23:41.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.431+0000 7f27e629d640 1 Processor -- start 2024-06-29T14:23:41.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.431+0000 7f27e629d640 1 -- start start 2024-06-29T14:23:41.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27e629d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 0x7f27e0158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27e629d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e00b2040 0x7f27e01f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27e629d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27e0144470 0x7f27e006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27e629d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f27e00cfaa0 con 0x7f27e006ea10 2024-06-29T14:23:41.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27e629d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27e00cf920 con 0x7f27e00b2040 2024-06-29T14:23:41.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27e629d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f27e00cfc20 con 0x7f27e0144470 2024-06-29T14:23:41.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 0x7f27e0158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 0x7f27e0158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44326/0 (socket says 172.21.15.59:44326) 2024-06-29T14:23:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 -- 172.21.15.59:0/1616016738 learned_addr learned my addr 172.21.15.59:0/1616016738 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27e483a640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27e0144470 0x7f27e006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27df7fe640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e00b2040 0x7f27e01f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 -- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27e0144470 msgr2=0x7f27e006ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27e0144470 0x7f27e006ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 -- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e00b2040 msgr2=0x7f27e01f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e00b2040 0x7f27e01f5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 -- 172.21.15.59:0/1616016738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e01591c0 con 0x7f27e006ea10 2024-06-29T14:23:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dffff640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 0x7f27e0158940 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f27d000c9d0 tx=0x7f27d000ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dcff9640 1 -- 172.21.15.59:0/1616016738 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d0007b50 con 0x7f27e006ea10 2024-06-29T14:23:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dcff9640 1 -- 172.21.15.59:0/1616016738 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27d0007de0 con 0x7f27e006ea10 2024-06-29T14:23:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.433+0000 7f27dcff9640 1 -- 172.21.15.59:0/1616016738 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d0005280 con 0x7f27e006ea10 2024-06-29T14:23:41.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.434+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e01594a0 con 0x7f27e006ea10 2024-06-29T14:23:41.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.434+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e0159a60 con 0x7f27e006ea10 2024-06-29T14:23:41.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.435+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e00d0140 con 0x7f27e006ea10 2024-06-29T14:23:41.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.435+0000 7f27dcff9640 1 -- 172.21.15.59:0/1616016738 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f27d0005420 con 0x7f27e006ea10 2024-06-29T14:23:41.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.436+0000 7f27dcff9640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f27cc076500 0x7f27cc078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.436+0000 7f27dcff9640 1 -- 172.21.15.59:0/1616016738 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f27d00981c0 con 0x7f27e006ea10 2024-06-29T14:23:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.436+0000 7f27df7fe640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f27cc076500 0x7f27cc078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.437+0000 7f27df7fe640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f27cc076500 0x7f27cc078a10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f27d4007fd0 tx=0x7f27d4009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.440+0000 7f27dcff9640 1 -- 172.21.15.59:0/1616016738 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f27d0014030 con 0x7f27e006ea10 2024-06-29T14:23:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.658+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f27e0071770 con 0x7f27cc076500 2024-06-29T14:23:41.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.705+0000 7f27dcff9640 1 -- 172.21.15.59:0/1616016738 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f27e0071770 con 0x7f27cc076500 2024-06-29T14:23:41.704 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-06-29T14:23:41.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f27cc076500 msgr2=0x7f27cc078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f27cc076500 0x7f27cc078a10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f27d4007fd0 tx=0x7f27d4009040 comp rx=0 tx=0).stop 2024-06-29T14:23:41.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 msgr2=0x7f27e0158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 0x7f27e0158940 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f27d000c9d0 tx=0x7f27d000ce90 comp rx=0 tx=0).stop 2024-06-29T14:23:41.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 shutdown_connections 2024-06-29T14:23:41.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f27cc076500 0x7f27cc078a10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27e0144470 0x7f27e006ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27e00b2040 0x7f27e01f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 --2- 172.21.15.59:0/1616016738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e006ea10 0x7f27e0158940 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 >> 172.21.15.59:0/1616016738 conn(0x7f27e00b06e0 msgr2=0x7f27e0141230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 shutdown_connections 2024-06-29T14:23:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.708+0000 7f27e629d640 1 -- 172.21.15.59:0/1616016738 wait complete. 2024-06-29T14:23:41.717 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-06-29T14:23:41.717 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-06-29T14:23:41.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.840+0000 7f15e2b6f640 1 -- 172.21.15.59:0/2661843313 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15dc162800 msgr2=0x7f15dc162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.840+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/2661843313 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15dc162800 0x7f15dc162c10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009860 tx=0x7f15d402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.840+0000 7f15e2b6f640 1 -- 172.21.15.59:0/2661843313 shutdown_connections 2024-06-29T14:23:41.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.840+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/2661843313 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15dc162800 0x7f15dc162c10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.840+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/2661843313 >> v2:172.21.15.153:3300/0 conn(0x7f15dc0ce950 0x7f15dc1622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.840+0000 7f15e2b6f640 1 -- 172.21.15.59:0/2661843313 >> 172.21.15.59:0/2661843313 conn(0x7f15dc0b5610 msgr2=0x7f15dc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.840+0000 7f15e2b6f640 1 -- 172.21.15.59:0/2661843313 shutdown_connections 2024-06-29T14:23:41.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.840+0000 7f15e2b6f640 1 -- 172.21.15.59:0/2661843313 wait complete. 2024-06-29T14:23:41.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.842+0000 7f15e2b6f640 1 Processor -- start 2024-06-29T14:23:41.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.842+0000 7f15e2b6f640 1 -- start start 2024-06-29T14:23:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e2b6f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15dc08c190 0x7f15dc15b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e2b6f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0ce950 0x7f15dc0c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e2b6f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15dc162800 0x7f15dc0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e2b6f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc0b7f60 con 0x7f15dc0ce950 2024-06-29T14:23:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e2b6f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc0b7de0 con 0x7f15dc08c190 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e2b6f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f15dc0b80e0 con 0x7f15dc162800 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0ce950 0x7f15dc0c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0ce950 0x7f15dc0c7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44328/0 (socket says 172.21.15.59:44328) 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 -- 172.21.15.59:0/1901046403 learned_addr learned my addr 172.21.15.59:0/1901046403 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e090b640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15dc08c190 0x7f15dc15b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e110c640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15dc162800 0x7f15dc0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 -- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15dc162800 msgr2=0x7f15dc0c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15dc162800 0x7f15dc0c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 -- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15dc08c190 msgr2=0x7f15dc15b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15dc08c190 0x7f15dc15b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 -- 172.21.15.59:0/1901046403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15dc0aaf00 con 0x7f15dc0ce950 2024-06-29T14:23:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15dbfff640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0ce950 0x7f15dc0c7610 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009fd0 tx=0x7f15c400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15d97fa640 1 -- 172.21.15.59:0/1901046403 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c400ccc0 con 0x7f15dc0ce950 2024-06-29T14:23:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15dc0ab1e0 con 0x7f15dc0ce950 2024-06-29T14:23:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15d97fa640 1 -- 172.21.15.59:0/1901046403 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15c400ee70 con 0x7f15dc0ce950 2024-06-29T14:23:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.844+0000 7f15d97fa640 1 -- 172.21.15.59:0/1901046403 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c400ce60 con 0x7f15dc0ce950 2024-06-29T14:23:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.845+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15dc0ab7a0 con 0x7f15dc0ce950 2024-06-29T14:23:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.845+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15dc1622c0 con 0x7f15dc0ce950 2024-06-29T14:23:41.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.846+0000 7f15d97fa640 1 -- 172.21.15.59:0/1901046403 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f15c4004280 con 0x7f15dc0ce950 2024-06-29T14:23:41.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.847+0000 7f15d97fa640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f15bc0764d0 0x7f15bc0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.847+0000 7f15d97fa640 1 -- 172.21.15.59:0/1901046403 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f15c4098660 con 0x7f15dc0ce950 2024-06-29T14:23:41.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.847+0000 7f15e090b640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f15bc0764d0 0x7f15bc0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.847+0000 7f15e090b640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f15bc0764d0 0x7f15bc0789e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0b7620 tx=0x7f15cc009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:41.851+0000 7f15d97fa640 1 -- 172.21.15.59:0/1901046403 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f15c40612d0 con 0x7f15dc0ce950 2024-06-29T14:23:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.069+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f15dc162d40 con 0x7f15bc0764d0 2024-06-29T14:23:42.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.118+0000 7f15d97fa640 1 -- 172.21.15.59:0/1901046403 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f15dc162d40 con 0x7f15bc0764d0 2024-06-29T14:23:42.116 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.120+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f15bc0764d0 msgr2=0x7f15bc0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.120+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f15bc0764d0 0x7f15bc0789e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0b7620 tx=0x7f15cc009040 comp rx=0 tx=0).stop 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.120+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0ce950 msgr2=0x7f15dc0c7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.120+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0ce950 0x7f15dc0c7610 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009fd0 tx=0x7f15c400ec20 comp rx=0 tx=0).stop 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.121+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 shutdown_connections 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.121+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f15bc0764d0 0x7f15bc0789e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.121+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15dc162800 0x7f15dc0c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.121+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0ce950 0x7f15dc0c7610 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.121+0000 7f15e2b6f640 1 --2- 172.21.15.59:0/1901046403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15dc08c190 0x7f15dc15b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.121+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 >> 172.21.15.59:0/1901046403 conn(0x7f15dc0b5610 msgr2=0x7f15dc06d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:42.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.121+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 shutdown_connections 2024-06-29T14:23:42.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.121+0000 7f15e2b6f640 1 -- 172.21.15.59:0/1901046403 wait complete. 2024-06-29T14:23:42.130 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-06-29T14:23:42.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-06-29T14:23:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.253+0000 7f6f22535640 1 -- 172.21.15.59:0/737475550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 msgr2=0x7f6f1c0d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.253+0000 7f6f22535640 1 --2- 172.21.15.59:0/737475550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 0x7f6f1c0d1a20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6f10005fd0 tx=0x7f6f1002a5a0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.253+0000 7f6f22535640 1 -- 172.21.15.59:0/737475550 shutdown_connections 2024-06-29T14:23:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.253+0000 7f6f22535640 1 --2- 172.21.15.59:0/737475550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 0x7f6f1c0d1a20 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.253+0000 7f6f22535640 1 --2- 172.21.15.59:0/737475550 >> v2:172.21.15.153:3300/0 conn(0x7f6f1c08c190 0x7f6f1c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.253+0000 7f6f22535640 1 -- 172.21.15.59:0/737475550 >> 172.21.15.59:0/737475550 conn(0x7f6f1c0b5610 msgr2=0x7f6f1c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:42.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.253+0000 7f6f22535640 1 -- 172.21.15.59:0/737475550 shutdown_connections 2024-06-29T14:23:42.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.253+0000 7f6f22535640 1 -- 172.21.15.59:0/737475550 wait complete. 2024-06-29T14:23:42.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.255+0000 7f6f22535640 1 Processor -- start 2024-06-29T14:23:42.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.255+0000 7f6f22535640 1 -- start start 2024-06-29T14:23:42.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f22535640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f1c08c190 0x7f6f1c14d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f22535640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f1c0ce950 0x7f6f1c1e8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f22535640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 0x7f6f1c1e9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f22535640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c0b7f60 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f22535640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c0b7de0 con 0x7f6f1c08c190 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f22535640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1c0b80e0 con 0x7f6f1c0ce950 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f20ad2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 0x7f6f1c1e9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f1bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f1c08c190 0x7f6f1c14d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f20ad2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 0x7f6f1c1e9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44358/0 (socket says 172.21.15.59:44358) 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f20ad2640 1 -- 172.21.15.59:0/123477503 learned_addr learned my addr 172.21.15.59:0/123477503 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.256+0000 7f6f1b7fe640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f1c0ce950 0x7f6f1c1e8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f20ad2640 1 -- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f1c0ce950 msgr2=0x7f6f1c1e8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f20ad2640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f1c0ce950 0x7f6f1c1e8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f20ad2640 1 -- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f1c08c190 msgr2=0x7f6f1c14d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f20ad2640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f1c08c190 0x7f6f1c14d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f20ad2640 1 -- 172.21.15.59:0/123477503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c14d590 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f1bfff640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f1c08c190 0x7f6f1c14d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f20ad2640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 0x7f6f1c1e9500 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c730 tx=0x7f6f1400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f18ff9640 1 -- 172.21.15.59:0/123477503 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14003b70 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f18ff9640 1 -- 172.21.15.59:0/123477503 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f14003d10 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1c159b40 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f18ff9640 1 -- 172.21.15.59:0/123477503 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14005330 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.257+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1c1e47c0 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.259+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f1c0cf080 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.259+0000 7f6f18ff9640 1 -- 172.21.15.59:0/123477503 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f6f140054d0 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.260+0000 7f6f18ff9640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6f080764d0 0x7f6f080789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.260+0000 7f6f18ff9640 1 -- 172.21.15.59:0/123477503 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6359+0+0 (secure 0 0 0) 0x7f6f14097a70 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.260+0000 7f6f1bfff640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6f080764d0 0x7f6f080789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.261+0000 7f6f1bfff640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6f080764d0 0x7f6f080789e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009f20 tx=0x7f6f0c005f90 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:42.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.264+0000 7f6f18ff9640 1 -- 172.21.15.59:0/123477503 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6f140607e0 con 0x7f6f1c0cf5c0 2024-06-29T14:23:42.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.481+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6f1c15d5d0 con 0x7f6f080764d0 2024-06-29T14:23:42.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.528+0000 7f6f18ff9640 1 -- 172.21.15.59:0/123477503 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f6f1c15d5d0 con 0x7f6f080764d0 2024-06-29T14:23:42.526 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.530+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6f080764d0 msgr2=0x7f6f080789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.530+0000 7f6f22535640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6f080764d0 0x7f6f080789e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009f20 tx=0x7f6f0c005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.530+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 msgr2=0x7f6f1c1e9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.530+0000 7f6f22535640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 0x7f6f1c1e9500 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c730 tx=0x7f6f1400cb30 comp rx=0 tx=0).stop 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.531+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 shutdown_connections 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.531+0000 7f6f22535640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f6f080764d0 0x7f6f080789e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.531+0000 7f6f22535640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f1c0cf5c0 0x7f6f1c1e9500 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.531+0000 7f6f22535640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f1c0ce950 0x7f6f1c1e8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.531+0000 7f6f22535640 1 --2- 172.21.15.59:0/123477503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f1c08c190 0x7f6f1c14d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.531+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 >> 172.21.15.59:0/123477503 conn(0x7f6f1c0b5610 msgr2=0x7f6f1c0c8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:42.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.531+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 shutdown_connections 2024-06-29T14:23:42.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:23:42.531+0000 7f6f22535640 1 -- 172.21.15.59:0/123477503 wait complete. 2024-06-29T14:23:42.540 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-06-29T14:23:53.165 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:23:53.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:24:00.079 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:24:00.081+0000 7f4b4d24c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:00.080 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:24:00.081+0000 7f4b4d24c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:00.080 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:24:00.081+0000 7f4b4d24c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:00.080 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:24:00.081+0000 7f4b4d24c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:00.080 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:24:00.081+0000 7f4b4d24c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:23.236 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:24:23.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:24:53.301 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:24:53.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:25:23.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:25:23.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:25:53.429 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:25:53.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:26:23.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:26:23.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:26:53.575 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:26:53.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:27:23.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:27:23.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:27:53.702 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:27:53.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:28:23.776 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:28:23.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:28:54.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:28:54.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:29:00.287 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:29:00.289+0000 7f4b51b15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:29:00.287 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:29:00.289+0000 7f4b51b15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:29:00.288 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:29:00.289+0000 7f4b51b15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:29:00.288 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:29:00.289+0000 7f4b51b15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:29:00.288 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:29:00.289+0000 7f4b51b15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:29:24.285 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:29:24.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:29:54.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:29:54.446 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:30:24.604 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:30:24.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:30:54.710 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:30:54.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:31:24.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:31:24.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:31:54.978 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:31:54.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:32:25.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:32:25.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:32:55.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:32:55.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:33:25.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:33:25.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:33:55.472 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:33:55.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:19.541 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-06-29T14:34:19.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.680+0000 7fec0c8b3640 1 -- 172.21.15.59:0/1890104029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febf0004070 msgr2=0x7febf0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.680+0000 7fec0c8b3640 1 --2- 172.21.15.59:0/1890104029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febf0004070 0x7febf0004800 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7febfc009b50 tx=0x7febf0005830 comp rx=0 tx=0).stop 2024-06-29T14:34:19.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.680+0000 7fec06d9d640 1 -- 172.21.15.59:0/1890104029 reap_dead start 2024-06-29T14:34:19.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.680+0000 7fec0c8b3640 1 -- 172.21.15.59:0/1890104029 shutdown_connections 2024-06-29T14:34:19.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.680+0000 7fec0c8b3640 1 -- 172.21.15.59:0/1890104029 >> 172.21.15.59:0/1890104029 conn(0x7fec080b24c0 msgr2=0x7fec080cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:19.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.680+0000 7fec0c8b3640 1 -- 172.21.15.59:0/1890104029 shutdown_connections 2024-06-29T14:34:19.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.680+0000 7fec0c8b3640 1 -- 172.21.15.59:0/1890104029 wait complete. 2024-06-29T14:34:19.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.682+0000 7fec0c8b3640 1 Processor -- start 2024-06-29T14:34:19.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.682+0000 7fec0c8b3640 1 -- start start 2024-06-29T14:34:19.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec0c8b3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febf0004070 0x7fec08135a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec0c8b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec08158570 0x7fec0813cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec0c8b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec0813d100 0x7fec081375b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec0c8b3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fec081448f0 con 0x7fec0813d100 2024-06-29T14:34:19.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec0c8b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec08144770 con 0x7fec08158570 2024-06-29T14:34:19.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec0c8b3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fec0806d700 con 0x7febf0004070 2024-06-29T14:34:19.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec0813d100 0x7fec081375b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec0813d100 0x7fec081375b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36170/0 (socket says 172.21.15.59:36170) 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 -- 172.21.15.59:0/507237606 learned_addr learned my addr 172.21.15.59:0/507237606 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec0659c640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febf0004070 0x7fec08135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec05d9b640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec08158570 0x7fec0813cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 -- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febf0004070 msgr2=0x7fec08135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febf0004070 0x7fec08135a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 -- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec08158570 msgr2=0x7fec0813cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec08158570 0x7fec0813cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 -- 172.21.15.59:0/507237606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0813d510 con 0x7fec0813d100 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7fec06d9d640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec0813d100 0x7fec081375b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fec08158df0 tx=0x7fec00006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.684+0000 7febf6ffd640 1 -- 172.21.15.59:0/507237606 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00011900 con 0x7fec0813d100 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.685+0000 7febf6ffd640 1 -- 172.21.15.59:0/507237606 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec00018070 con 0x7fec0813d100 2024-06-29T14:34:19.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.685+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec08136510 con 0x7fec0813d100 2024-06-29T14:34:19.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.685+0000 7febf6ffd640 1 -- 172.21.15.59:0/507237606 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00011d40 con 0x7fec0813d100 2024-06-29T14:34:19.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.685+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec08136ad0 con 0x7fec0813d100 2024-06-29T14:34:19.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.686+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec0815bc80 con 0x7fec0813d100 2024-06-29T14:34:19.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.686+0000 7febf6ffd640 1 -- 172.21.15.59:0/507237606 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fec0001d460 con 0x7fec0813d100 2024-06-29T14:34:19.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.687+0000 7febf6ffd640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7febe80764d0 0x7febe80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.687+0000 7febf6ffd640 1 -- 172.21.15.59:0/507237606 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fec000986d0 con 0x7fec0813d100 2024-06-29T14:34:19.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.687+0000 7fec0659c640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7febe80764d0 0x7febe80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.688+0000 7fec0659c640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7febe80764d0 0x7febe80789e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7febf8000fb0 tx=0x7febf8006910 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:19.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.692+0000 7febf6ffd640 1 -- 172.21.15.59:0/507237606 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fec0813d510 con 0x7fec0813d100 2024-06-29T14:34:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:19.909+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fec0813e5b0 con 0x7febe80764d0 2024-06-29T14:34:20.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.002+0000 7febf6ffd640 1 -- 172.21.15.59:0/507237606 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fec0813e5b0 con 0x7febe80764d0 2024-06-29T14:34:20.001 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-06-29T14:34:20.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7febe80764d0 msgr2=0x7febe80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7febe80764d0 0x7febe80789e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7febf8000fb0 tx=0x7febf8006910 comp rx=0 tx=0).stop 2024-06-29T14:34:20.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec0813d100 msgr2=0x7fec081375b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec0813d100 0x7fec081375b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fec08158df0 tx=0x7fec00006430 comp rx=0 tx=0).stop 2024-06-29T14:34:20.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 shutdown_connections 2024-06-29T14:34:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7febe80764d0 0x7febe80789e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec0813d100 0x7fec081375b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec08158570 0x7fec0813cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 --2- 172.21.15.59:0/507237606 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febf0004070 0x7fec08135a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 >> 172.21.15.59:0/507237606 conn(0x7fec080b24c0 msgr2=0x7fec080afdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 shutdown_connections 2024-06-29T14:34:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.004+0000 7fec0c8b3640 1 -- 172.21.15.59:0/507237606 wait complete. 2024-06-29T14:34:20.015 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-06-29T14:34:20.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-06-29T14:34:20.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.138+0000 7ff202368640 1 -- 172.21.15.59:0/3295534475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0b2270 msgr2=0x7ff1fc15bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.138+0000 7ff202368640 1 --2- 172.21.15.59:0/3295534475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0b2270 0x7ff1fc15bc20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec009830 tx=0x7ff1ec02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.138+0000 7ff202368640 1 -- 172.21.15.59:0/3295534475 shutdown_connections 2024-06-29T14:34:20.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.138+0000 7ff202368640 1 --2- 172.21.15.59:0/3295534475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0b2270 0x7ff1fc15bc20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.138+0000 7ff202368640 1 -- 172.21.15.59:0/3295534475 >> 172.21.15.59:0/3295534475 conn(0x7ff1fc073e10 msgr2=0x7ff1fc074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.138+0000 7ff202368640 1 -- 172.21.15.59:0/3295534475 shutdown_connections 2024-06-29T14:34:20.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.138+0000 7ff202368640 1 -- 172.21.15.59:0/3295534475 wait complete. 2024-06-29T14:34:20.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.140+0000 7ff202368640 1 Processor -- start 2024-06-29T14:34:20.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.140+0000 7ff202368640 1 -- start start 2024-06-29T14:34:20.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff202368640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1fc0b2270 0x7ff1fc1f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff202368640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1fc0b2fa0 0x7ff1fc15eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff202368640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0d1c30 0x7ff1fc154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff202368640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc0b65d0 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff202368640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc0b6450 con 0x7ff1fc0b2270 2024-06-29T14:34:20.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff202368640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff1fc0b6750 con 0x7ff1fc0b2fa0 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff200905640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0d1c30 0x7ff1fc154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff200905640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0d1c30 0x7ff1fc154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36182/0 (socket says 172.21.15.59:36182) 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff200905640 1 -- 172.21.15.59:0/2097616889 learned_addr learned my addr 172.21.15.59:0/2097616889 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff1fbfff640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1fc0b2270 0x7ff1fc1f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff1fb7fe640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1fc0b2fa0 0x7ff1fc15eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff200905640 1 -- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1fc0b2fa0 msgr2=0x7ff1fc15eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff200905640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1fc0b2fa0 0x7ff1fc15eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff200905640 1 -- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1fc0b2270 msgr2=0x7ff1fc1f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff200905640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1fc0b2270 0x7ff1fc1f5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.142+0000 7ff200905640 1 -- 172.21.15.59:0/2097616889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc0a9eb0 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.143+0000 7ff200905640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0d1c30 0x7ff1fc154d60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400c730 tx=0x7ff1f400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:20.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.143+0000 7ff1f8ff9640 1 -- 172.21.15.59:0/2097616889 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4003b70 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.143+0000 7ff1f8ff9640 1 -- 172.21.15.59:0/2097616889 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1f4003d10 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.143+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1fc1f60b0 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.143+0000 7ff1f8ff9640 1 -- 172.21.15.59:0/2097616889 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40052d0 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.143+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1fc0aa690 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.144+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1fc0ab2a0 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.145+0000 7ff1f8ff9640 1 -- 172.21.15.59:0/2097616889 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7ff1f4005470 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.145+0000 7ff1f8ff9640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7ff1e80764d0 0x7ff1e80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.146+0000 7ff1f8ff9640 1 -- 172.21.15.59:0/2097616889 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7ff1f40979b0 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.146+0000 7ff1fbfff640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7ff1e80764d0 0x7ff1e80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.146+0000 7ff1fbfff640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7ff1e80764d0 0x7ff1e80789e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec034000 tx=0x7ff1ec004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:20.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.150+0000 7ff1f8ff9640 1 -- 172.21.15.59:0/2097616889 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff1f4060820 con 0x7ff1fc0d1c30 2024-06-29T14:34:20.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.370+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff1fc0aa140 con 0x7ff1e80764d0 2024-06-29T14:34:20.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.462+0000 7ff1f8ff9640 1 -- 172.21.15.59:0/2097616889 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ff1fc0aa140 con 0x7ff1e80764d0 2024-06-29T14:34:20.461 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-06-29T14:34:20.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.464+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7ff1e80764d0 msgr2=0x7ff1e80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.464+0000 7ff202368640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7ff1e80764d0 0x7ff1e80789e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec034000 tx=0x7ff1ec004730 comp rx=0 tx=0).stop 2024-06-29T14:34:20.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.464+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0d1c30 msgr2=0x7ff1fc154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.464+0000 7ff202368640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0d1c30 0x7ff1fc154d60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400c730 tx=0x7ff1f400cb30 comp rx=0 tx=0).stop 2024-06-29T14:34:20.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.465+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 shutdown_connections 2024-06-29T14:34:20.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.465+0000 7ff202368640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7ff1e80764d0 0x7ff1e80789e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.465+0000 7ff202368640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1fc0d1c30 0x7ff1fc154d60 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.465+0000 7ff202368640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1fc0b2fa0 0x7ff1fc15eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.465+0000 7ff202368640 1 --2- 172.21.15.59:0/2097616889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1fc0b2270 0x7ff1fc1f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.465+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 >> 172.21.15.59:0/2097616889 conn(0x7ff1fc073e10 msgr2=0x7ff1fc070ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.465+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 shutdown_connections 2024-06-29T14:34:20.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.465+0000 7ff202368640 1 -- 172.21.15.59:0/2097616889 wait complete. 2024-06-29T14:34:20.476 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-06-29T14:34:20.476 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-06-29T14:34:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.598+0000 7f13e0288640 1 -- 172.21.15.59:0/3482644822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80d23c0 msgr2=0x7f13d80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.598+0000 7f13e0288640 1 --2- 172.21.15.59:0/3482644822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80d23c0 0x7f13d80b3990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f13d8161d40 tx=0x7f13d402a580 comp rx=0 tx=0).stop 2024-06-29T14:34:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.598+0000 7f13e0288640 1 -- 172.21.15.59:0/3482644822 shutdown_connections 2024-06-29T14:34:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.598+0000 7f13e0288640 1 --2- 172.21.15.59:0/3482644822 >> v2:172.21.15.153:3300/0 conn(0x7f13d815efe0 0x7f13d815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.598+0000 7f13e0288640 1 --2- 172.21.15.59:0/3482644822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80d23c0 0x7f13d80b3990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.598+0000 7f13e0288640 1 -- 172.21.15.59:0/3482644822 >> 172.21.15.59:0/3482644822 conn(0x7f13d80d3ee0 msgr2=0x7f13d80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.599+0000 7f13e0288640 1 -- 172.21.15.59:0/3482644822 shutdown_connections 2024-06-29T14:34:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.599+0000 7f13e0288640 1 -- 172.21.15.59:0/3482644822 wait complete. 2024-06-29T14:34:20.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.600+0000 7f13e0288640 1 Processor -- start 2024-06-29T14:34:20.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.600+0000 7f13e0288640 1 -- start start 2024-06-29T14:34:20.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13e0288640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13d80b3ed0 0x7f13d80c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13e0288640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d80d23c0 0x7f13d815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13e0288640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d815efe0 0x7f13d8147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13e0288640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8160280 con 0x7f13d815efe0 2024-06-29T14:34:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13e0288640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8160100 con 0x7f13d80d23c0 2024-06-29T14:34:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13e0288640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f13d8160400 con 0x7f13d80b3ed0 2024-06-29T14:34:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d815efe0 0x7f13d8147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d815efe0 0x7f13d8147620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36208/0 (socket says 172.21.15.59:36208) 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 -- 172.21.15.59:0/1087733967 learned_addr learned my addr 172.21.15.59:0/1087733967 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13dd823640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d80d23c0 0x7f13d815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de024640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13d80b3ed0 0x7f13d80c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 -- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13d80b3ed0 msgr2=0x7f13d80c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13d80b3ed0 0x7f13d80c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 -- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d80d23c0 msgr2=0x7f13d815dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d80d23c0 0x7f13d815dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 -- 172.21.15.59:0/1087733967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d815b5c0 con 0x7f13d815efe0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de024640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13d80b3ed0 0x7f13d80c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13de825640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d815efe0 0x7f13d8147620 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f13d000b930 tx=0x7f13d000bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13ceffd640 1 -- 172.21.15.59:0/1087733967 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0019070 con 0x7f13d815efe0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13ceffd640 1 -- 172.21.15.59:0/1087733967 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13d000cc50 con 0x7f13d815efe0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13ceffd640 1 -- 172.21.15.59:0/1087733967 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00146a0 con 0x7f13d815efe0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.602+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d815b8a0 con 0x7f13d815efe0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.603+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d815be60 con 0x7f13d815efe0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.604+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d815b310 con 0x7f13d815efe0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.604+0000 7f13ceffd640 1 -- 172.21.15.59:0/1087733967 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f13d001c070 con 0x7f13d815efe0 2024-06-29T14:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.605+0000 7f13ceffd640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f13bc0764d0 0x7f13bc0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.605+0000 7f13ceffd640 1 -- 172.21.15.59:0/1087733967 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f13d0097a60 con 0x7f13d815efe0 2024-06-29T14:34:20.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.605+0000 7f13de024640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f13bc0764d0 0x7f13bc0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.606+0000 7f13de024640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f13bc0764d0 0x7f13bc0789e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f13d4033000 tx=0x7f13d4004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:20.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.609+0000 7f13ceffd640 1 -- 172.21.15.59:0/1087733967 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f13d00607b0 con 0x7f13d815efe0 2024-06-29T14:34:20.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.826+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f13d80b3990 con 0x7f13bc0764d0 2024-06-29T14:34:20.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.916+0000 7f13ceffd640 1 -- 172.21.15.59:0/1087733967 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f13d80b3990 con 0x7f13bc0764d0 2024-06-29T14:34:20.915 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-06-29T14:34:20.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f13bc0764d0 msgr2=0x7f13bc0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f13bc0764d0 0x7f13bc0789e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f13d4033000 tx=0x7f13d4004730 comp rx=0 tx=0).stop 2024-06-29T14:34:20.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d815efe0 msgr2=0x7f13d8147620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d815efe0 0x7f13d8147620 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f13d000b930 tx=0x7f13d000bdf0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 shutdown_connections 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f13bc0764d0 0x7f13bc0789e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d815efe0 0x7f13d8147620 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d80d23c0 0x7f13d815dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 --2- 172.21.15.59:0/1087733967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13d80b3ed0 0x7f13d80c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 >> 172.21.15.59:0/1087733967 conn(0x7f13d80d3ee0 msgr2=0x7f13d80b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 shutdown_connections 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:20.918+0000 7f13e0288640 1 -- 172.21.15.59:0/1087733967 wait complete. 2024-06-29T14:34:20.927 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-06-29T14:34:20.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-06-29T14:34:21.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.050+0000 7fd222455640 1 -- 172.21.15.59:0/3646306198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c08c190 msgr2=0x7fd21c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.050+0000 7fd222455640 1 --2- 172.21.15.59:0/3646306198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c08c190 0x7fd21c0cf080 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009830 tx=0x7fd20c02a4d0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.050+0000 7fd222455640 1 -- 172.21.15.59:0/3646306198 shutdown_connections 2024-06-29T14:34:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.050+0000 7fd222455640 1 --2- 172.21.15.59:0/3646306198 >> v2:172.21.15.153:3300/0 conn(0x7fd21c0ce950 0x7fd21c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.050+0000 7fd222455640 1 --2- 172.21.15.59:0/3646306198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c08c190 0x7fd21c0cf080 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.050+0000 7fd222455640 1 -- 172.21.15.59:0/3646306198 >> 172.21.15.59:0/3646306198 conn(0x7fd21c0b5610 msgr2=0x7fd21c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.050+0000 7fd222455640 1 -- 172.21.15.59:0/3646306198 shutdown_connections 2024-06-29T14:34:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.050+0000 7fd222455640 1 -- 172.21.15.59:0/3646306198 wait complete. 2024-06-29T14:34:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.052+0000 7fd222455640 1 Processor -- start 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.052+0000 7fd222455640 1 -- start start 2024-06-29T14:34:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd222455640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd21c08c190 0x7fd21c1f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd222455640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ce950 0x7fd21c15f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd222455640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd21c0cf5c0 0x7fd21c1f8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd222455640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c0b7f60 con 0x7fd21c0ce950 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd222455640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c0b7de0 con 0x7fd21c08c190 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd222455640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd21c0b80e0 con 0x7fd21c0cf5c0 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd21b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ce950 0x7fd21c15f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd21b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ce950 0x7fd21c15f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36226/0 (socket says 172.21.15.59:36226) 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd21b7fe640 1 -- 172.21.15.59:0/3160776969 learned_addr learned my addr 172.21.15.59:0/3160776969 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd21bfff640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd21c08c190 0x7fd21c1f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.053+0000 7fd2209f2640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd21c0cf5c0 0x7fd21c1f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd21b7fe640 1 -- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd21c0cf5c0 msgr2=0x7fd21c1f8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd21b7fe640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd21c0cf5c0 0x7fd21c1f8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd21b7fe640 1 -- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd21c08c190 msgr2=0x7fd21c1f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd21b7fe640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd21c08c190 0x7fd21c1f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd21b7fe640 1 -- 172.21.15.59:0/3160776969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21c1f2af0 con 0x7fd21c0ce950 2024-06-29T14:34:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd21b7fe640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ce950 0x7fd21c15f230 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd21000c3c0 tx=0x7fd21000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd218ff9640 1 -- 172.21.15.59:0/3160776969 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210017070 con 0x7fd21c0ce950 2024-06-29T14:34:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd218ff9640 1 -- 172.21.15.59:0/3160776969 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2100035f0 con 0x7fd21c0ce950 2024-06-29T14:34:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd218ff9640 1 -- 172.21.15.59:0/3160776969 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210003a20 con 0x7fd21c0ce950 2024-06-29T14:34:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd21c15fcb0 con 0x7fd21c0ce950 2024-06-29T14:34:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.054+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd21c160270 con 0x7fd21c0ce950 2024-06-29T14:34:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.055+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21c15fa20 con 0x7fd21c0ce950 2024-06-29T14:34:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.056+0000 7fd218ff9640 1 -- 172.21.15.59:0/3160776969 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fd21001a070 con 0x7fd21c0ce950 2024-06-29T14:34:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.057+0000 7fd218ff9640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fd2080764d0 0x7fd2080789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.057+0000 7fd218ff9640 1 -- 172.21.15.59:0/3160776969 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fd210096f40 con 0x7fd21c0ce950 2024-06-29T14:34:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.057+0000 7fd21bfff640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fd2080764d0 0x7fd2080789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.058+0000 7fd21bfff640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fd2080764d0 0x7fd2080789e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd20c034000 tx=0x7fd20c004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.061+0000 7fd218ff9640 1 -- 172.21.15.59:0/3160776969 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd21005fdc0 con 0x7fd21c0ce950 2024-06-29T14:34:21.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.278+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd21c0c9a40 con 0x7fd2080764d0 2024-06-29T14:34:21.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.367+0000 7fd218ff9640 1 -- 172.21.15.59:0/3160776969 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd21c0c9a40 con 0x7fd2080764d0 2024-06-29T14:34:21.366 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-06-29T14:34:21.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fd2080764d0 msgr2=0x7fd2080789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fd2080764d0 0x7fd2080789e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd20c034000 tx=0x7fd20c004730 comp rx=0 tx=0).stop 2024-06-29T14:34:21.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ce950 msgr2=0x7fd21c15f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ce950 0x7fd21c15f230 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd21000c3c0 tx=0x7fd21000c880 comp rx=0 tx=0).stop 2024-06-29T14:34:21.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 shutdown_connections 2024-06-29T14:34:21.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fd2080764d0 0x7fd2080789e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd21c0cf5c0 0x7fd21c1f8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd21c0ce950 0x7fd21c15f230 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 --2- 172.21.15.59:0/3160776969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd21c08c190 0x7fd21c1f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 >> 172.21.15.59:0/3160776969 conn(0x7fd21c0b5610 msgr2=0x7fd21c0cabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 shutdown_connections 2024-06-29T14:34:21.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.370+0000 7fd222455640 1 -- 172.21.15.59:0/3160776969 wait complete. 2024-06-29T14:34:21.379 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-06-29T14:34:21.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-06-29T14:34:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 -- 172.21.15.59:0/3669488331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c0b3740 msgr2=0x7fb60c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 --2- 172.21.15.59:0/3669488331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c0b3740 0x7fb60c0b3b50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb60c074b10 tx=0x7fb5f402a560 comp rx=0 tx=0).stop 2024-06-29T14:34:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 -- 172.21.15.59:0/3669488331 shutdown_connections 2024-06-29T14:34:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 --2- 172.21.15.59:0/3669488331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb60c15c2f0 0x7fb60c15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 --2- 172.21.15.59:0/3669488331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c0b3740 0x7fb60c0b3b50 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 --2- 172.21.15.59:0/3669488331 >> v2:172.21.15.153:3300/0 conn(0x7fb60c1458a0 0x7fb60c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 -- 172.21.15.59:0/3669488331 >> 172.21.15.59:0/3669488331 conn(0x7fb60c06c950 msgr2=0x7fb60c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 -- 172.21.15.59:0/3669488331 shutdown_connections 2024-06-29T14:34:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.502+0000 7fb6126c1640 1 -- 172.21.15.59:0/3669488331 wait complete. 2024-06-29T14:34:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.504+0000 7fb6126c1640 1 Processor -- start 2024-06-29T14:34:21.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.504+0000 7fb6126c1640 1 -- start start 2024-06-29T14:34:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb6126c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0b3740 0x7fb60c0b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb6126c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c15c2f0 0x7fb60c071160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb6126c1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb60c0716a0 0x7fb60c0ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb6126c1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c15fe40 con 0x7fb60c15c2f0 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb6126c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c15fcc0 con 0x7fb60c0b3740 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb6126c1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb60c15ffc0 con 0x7fb60c0716a0 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb603fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c15c2f0 0x7fb60c071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb603fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c15c2f0 0x7fb60c071160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36246/0 (socket says 172.21.15.59:36246) 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb603fff640 1 -- 172.21.15.59:0/2844366129 learned_addr learned my addr 172.21.15.59:0/2844366129 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb610c5e640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb60c0716a0 0x7fb60c0ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb60bfff640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0b3740 0x7fb60c0b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb603fff640 1 -- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb60c0716a0 msgr2=0x7fb60c0ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb603fff640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb60c0716a0 0x7fb60c0ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb603fff640 1 -- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0b3740 msgr2=0x7fb60c0b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb603fff640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0b3740 0x7fb60c0b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.505+0000 7fb603fff640 1 -- 172.21.15.59:0/2844366129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c071ab0 con 0x7fb60c15c2f0 2024-06-29T14:34:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.506+0000 7fb603fff640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c15c2f0 0x7fb60c071160 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4034040 tx=0x7fb5f4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.506+0000 7fb6097fa640 1 -- 172.21.15.59:0/2844366129 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f403e030 con 0x7fb60c15c2f0 2024-06-29T14:34:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.506+0000 7fb6097fa640 1 -- 172.21.15.59:0/2844366129 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5f4003e00 con 0x7fb60c15c2f0 2024-06-29T14:34:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.506+0000 7fb6097fa640 1 -- 172.21.15.59:0/2844366129 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4035040 con 0x7fb60c15c2f0 2024-06-29T14:34:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.506+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60c15b7e0 con 0x7fb60c15c2f0 2024-06-29T14:34:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.506+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb60c14e770 con 0x7fb60c15c2f0 2024-06-29T14:34:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.507+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb60c0b0fd0 con 0x7fb60c15c2f0 2024-06-29T14:34:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.508+0000 7fb6097fa640 1 -- 172.21.15.59:0/2844366129 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fb5f4041030 con 0x7fb60c15c2f0 2024-06-29T14:34:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.508+0000 7fb6097fa640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb5ec0764d0 0x7fb5ec0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.508+0000 7fb6097fa640 1 -- 172.21.15.59:0/2844366129 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fb5f40b86f0 con 0x7fb60c15c2f0 2024-06-29T14:34:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.509+0000 7fb60bfff640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb5ec0764d0 0x7fb5ec0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.509+0000 7fb60bfff640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb5ec0764d0 0x7fb5ec0789e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00daa0 tx=0x7fb5fc005d70 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.513+0000 7fb6097fa640 1 -- 172.21.15.59:0/2844366129 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb60c071ab0 con 0x7fb60c15c2f0 2024-06-29T14:34:21.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.727+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb60c1458a0 con 0x7fb5ec0764d0 2024-06-29T14:34:21.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.817+0000 7fb6097fa640 1 -- 172.21.15.59:0/2844366129 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fb60c1458a0 con 0x7fb5ec0764d0 2024-06-29T14:34:21.816 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-06-29T14:34:21.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb5ec0764d0 msgr2=0x7fb5ec0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb5ec0764d0 0x7fb5ec0789e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00daa0 tx=0x7fb5fc005d70 comp rx=0 tx=0).stop 2024-06-29T14:34:21.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c15c2f0 msgr2=0x7fb60c071160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c15c2f0 0x7fb60c071160 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4034040 tx=0x7fb5f4003910 comp rx=0 tx=0).stop 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 shutdown_connections 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb5ec0764d0 0x7fb5ec0789e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb60c0716a0 0x7fb60c0ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb60c15c2f0 0x7fb60c071160 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 --2- 172.21.15.59:0/2844366129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0b3740 0x7fb60c0b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 >> 172.21.15.59:0/2844366129 conn(0x7fb60c06c950 msgr2=0x7fb60c1415a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 shutdown_connections 2024-06-29T14:34:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.820+0000 7fb6126c1640 1 -- 172.21.15.59:0/2844366129 wait complete. 2024-06-29T14:34:21.830 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-06-29T14:34:21.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-06-29T14:34:21.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.958+0000 7f88f9ce7640 1 -- 172.21.15.59:0/2757455214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88dc004d40 msgr2=0x7f88dc005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.958+0000 7f88f9ce7640 1 --2- 172.21.15.59:0/2757455214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88dc004d40 0x7f88dc005150 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c290 tx=0x7f88dc005830 comp rx=0 tx=0).stop 2024-06-29T14:34:21.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.958+0000 7f88f3fff640 1 -- 172.21.15.59:0/2757455214 reap_dead start 2024-06-29T14:34:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.958+0000 7f88f9ce7640 1 -- 172.21.15.59:0/2757455214 shutdown_connections 2024-06-29T14:34:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.958+0000 7f88f9ce7640 1 -- 172.21.15.59:0/2757455214 >> 172.21.15.59:0/2757455214 conn(0x7f88f40d3ee0 msgr2=0x7f88f40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.958+0000 7f88f9ce7640 1 -- 172.21.15.59:0/2757455214 shutdown_connections 2024-06-29T14:34:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.958+0000 7f88f9ce7640 1 -- 172.21.15.59:0/2757455214 wait complete. 2024-06-29T14:34:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.960+0000 7f88f9ce7640 1 Processor -- start 2024-06-29T14:34:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.960+0000 7f88f9ce7640 1 -- start start 2024-06-29T14:34:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f9ce7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88dc004d40 0x7f88f40ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f9ce7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f40aad60 0x7f88f40ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f9ce7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f40ae5d0 0x7f88f41606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f9ce7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f88f40b3990 con 0x7f88f40aad60 2024-06-29T14:34:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f9ce7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88f40b84e0 con 0x7f88f40ae5d0 2024-06-29T14:34:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f9ce7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f88f40b3b10 con 0x7f88dc004d40 2024-06-29T14:34:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f40aad60 0x7f88f40ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f40aad60 0x7f88f40ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36262/0 (socket says 172.21.15.59:36262) 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f2ffd640 1 -- 172.21.15.59:0/313593458 learned_addr learned my addr 172.21.15.59:0/313593458 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f37fe640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88dc004d40 0x7f88f40ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f2ffd640 1 -- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88dc004d40 msgr2=0x7f88f40ab800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f2ffd640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88dc004d40 0x7f88f40ab800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f2ffd640 1 -- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f40ae5d0 msgr2=0x7f88f41606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f2ffd640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f40ae5d0 0x7f88f41606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.961+0000 7f88f2ffd640 1 -- 172.21.15.59:0/313593458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f414a2c0 con 0x7f88f40aad60 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.962+0000 7f88f2ffd640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f40aad60 0x7f88f40ae090 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f88e8011b10 tx=0x7f88e8009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.962+0000 7f88dbfff640 1 -- 172.21.15.59:0/313593458 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e801e070 con 0x7f88f40aad60 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.962+0000 7f88dbfff640 1 -- 172.21.15.59:0/313593458 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88e8004460 con 0x7f88f40aad60 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.962+0000 7f88dbfff640 1 -- 172.21.15.59:0/313593458 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e800fbb0 con 0x7f88f40aad60 2024-06-29T14:34:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.962+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f414a5a0 con 0x7f88f40aad60 2024-06-29T14:34:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.962+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f414ab60 con 0x7f88f40aad60 2024-06-29T14:34:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.963+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f40b6fd0 con 0x7f88f40aad60 2024-06-29T14:34:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.964+0000 7f88dbfff640 1 -- 172.21.15.59:0/313593458 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f88e8021070 con 0x7f88f40aad60 2024-06-29T14:34:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.964+0000 7f88dbfff640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88e0076500 0x7f88e0078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.964+0000 7f88dbfff640 1 -- 172.21.15.59:0/313593458 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f88e8098850 con 0x7f88f40aad60 2024-06-29T14:34:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.964+0000 7f88f37fe640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88e0076500 0x7f88e0078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.965+0000 7f88f37fe640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88e0076500 0x7f88e0078a10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f88e4005fd0 tx=0x7f88e4009500 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:21.968+0000 7f88dbfff640 1 -- 172.21.15.59:0/313593458 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f88f40b6fd0 con 0x7f88f40aad60 2024-06-29T14:34:22.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.182+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f88f40d1bc0 con 0x7f88e0076500 2024-06-29T14:34:22.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.267+0000 7f88dbfff640 1 -- 172.21.15.59:0/313593458 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f88f40d1bc0 con 0x7f88e0076500 2024-06-29T14:34:22.265 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-06-29T14:34:22.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88e0076500 msgr2=0x7f88e0078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88e0076500 0x7f88e0078a10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f88e4005fd0 tx=0x7f88e4009500 comp rx=0 tx=0).stop 2024-06-29T14:34:22.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f40aad60 msgr2=0x7f88f40ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f40aad60 0x7f88f40ae090 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f88e8011b10 tx=0x7f88e8009e00 comp rx=0 tx=0).stop 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 shutdown_connections 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88e0076500 0x7f88e0078a10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f40ae5d0 0x7f88f41606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f40aad60 0x7f88f40ae090 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 --2- 172.21.15.59:0/313593458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88dc004d40 0x7f88f40ab800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 >> 172.21.15.59:0/313593458 conn(0x7f88f40d3ee0 msgr2=0x7f88f415ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 shutdown_connections 2024-06-29T14:34:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.269+0000 7f88f9ce7640 1 -- 172.21.15.59:0/313593458 wait complete. 2024-06-29T14:34:22.278 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-06-29T14:34:22.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-06-29T14:34:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.401+0000 7fe973826640 1 -- 172.21.15.59:0/780314154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c162190 msgr2=0x7fe96c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.401+0000 7fe973826640 1 --2- 172.21.15.59:0/780314154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c162190 0x7fe96c0ce550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009830 tx=0x7fe95c02a480 comp rx=0 tx=0).stop 2024-06-29T14:34:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.401+0000 7fe973826640 1 -- 172.21.15.59:0/780314154 shutdown_connections 2024-06-29T14:34:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.401+0000 7fe973826640 1 --2- 172.21.15.59:0/780314154 >> v2:172.21.15.153:3300/0 conn(0x7fe96c15e0a0 0x7fe96c0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.401+0000 7fe973826640 1 --2- 172.21.15.59:0/780314154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c162190 0x7fe96c0ce550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.401+0000 7fe973826640 1 -- 172.21.15.59:0/780314154 >> 172.21.15.59:0/780314154 conn(0x7fe96c0d45b0 msgr2=0x7fe96c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.401+0000 7fe973826640 1 -- 172.21.15.59:0/780314154 shutdown_connections 2024-06-29T14:34:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.401+0000 7fe973826640 1 -- 172.21.15.59:0/780314154 wait complete. 2024-06-29T14:34:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.403+0000 7fe973826640 1 Processor -- start 2024-06-29T14:34:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.403+0000 7fe973826640 1 -- start start 2024-06-29T14:34:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe973826640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c15e0a0 0x7fe96c1564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe973826640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe96c162770 0x7fe96c1586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe973826640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe96c156dd0 0x7fe96c158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe973826640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe96c0b43a0 con 0x7fe96c156dd0 2024-06-29T14:34:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe973826640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe96c0b4220 con 0x7fe96c162770 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe973826640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe96c0b4520 con 0x7fe96c15e0a0 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe971dc3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe96c156dd0 0x7fe96c158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe971dc3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe96c156dd0 0x7fe96c158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36278/0 (socket says 172.21.15.59:36278) 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe971dc3640 1 -- 172.21.15.59:0/2031663147 learned_addr learned my addr 172.21.15.59:0/2031663147 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe970dc1640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe96c162770 0x7fe96c1586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.404+0000 7fe9715c2640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c15e0a0 0x7fe96c1564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe971dc3640 1 -- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c15e0a0 msgr2=0x7fe96c1564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe971dc3640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c15e0a0 0x7fe96c1564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe971dc3640 1 -- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe96c162770 msgr2=0x7fe96c1586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe971dc3640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe96c162770 0x7fe96c1586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe971dc3640 1 -- 172.21.15.59:0/2031663147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96c154050 con 0x7fe96c156dd0 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe9715c2640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c15e0a0 0x7fe96c1564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe971dc3640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe96c156dd0 0x7fe96c158c30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe96400ebf0 tx=0x7fe96400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:22.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe961ffb640 1 -- 172.21.15.59:0/2031663147 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964003b70 con 0x7fe96c156dd0 2024-06-29T14:34:22.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe961ffb640 1 -- 172.21.15.59:0/2031663147 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe964003d10 con 0x7fe96c156dd0 2024-06-29T14:34:22.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe96c0ccf30 con 0x7fe96c156dd0 2024-06-29T14:34:22.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe961ffb640 1 -- 172.21.15.59:0/2031663147 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe964004280 con 0x7fe96c156dd0 2024-06-29T14:34:22.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.405+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe96c1545d0 con 0x7fe96c156dd0 2024-06-29T14:34:22.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.406+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe96c0d0aa0 con 0x7fe96c156dd0 2024-06-29T14:34:22.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.407+0000 7fe961ffb640 1 -- 172.21.15.59:0/2031663147 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fe964004420 con 0x7fe96c156dd0 2024-06-29T14:34:22.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.407+0000 7fe961ffb640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe9500764d0 0x7fe9500789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.408+0000 7fe961ffb640 1 -- 172.21.15.59:0/2031663147 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fe964098a70 con 0x7fe96c156dd0 2024-06-29T14:34:22.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.408+0000 7fe9715c2640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe9500764d0 0x7fe9500789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.408+0000 7fe9715c2640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe9500764d0 0x7fe9500789e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe968007a50 tx=0x7fe968009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:22.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.412+0000 7fe961ffb640 1 -- 172.21.15.59:0/2031663147 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe96c0d0aa0 con 0x7fe96c156dd0 2024-06-29T14:34:22.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.628+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fe96c0cd210 con 0x7fe9500764d0 2024-06-29T14:34:22.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.716+0000 7fe961ffb640 1 -- 172.21.15.59:0/2031663147 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fe96c0cd210 con 0x7fe9500764d0 2024-06-29T14:34:22.715 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-06-29T14:34:22.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe9500764d0 msgr2=0x7fe9500789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe9500764d0 0x7fe9500789e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe968007a50 tx=0x7fe968009040 comp rx=0 tx=0).stop 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe96c156dd0 msgr2=0x7fe96c158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe96c156dd0 0x7fe96c158c30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe96400ebf0 tx=0x7fe96400c6a0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 shutdown_connections 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe9500764d0 0x7fe9500789e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe96c156dd0 0x7fe96c158c30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe96c162770 0x7fe96c1586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 --2- 172.21.15.59:0/2031663147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe96c15e0a0 0x7fe96c1564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 >> 172.21.15.59:0/2031663147 conn(0x7fe96c0d45b0 msgr2=0x7fe96c06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 shutdown_connections 2024-06-29T14:34:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.719+0000 7fe973826640 1 -- 172.21.15.59:0/2031663147 wait complete. 2024-06-29T14:34:22.728 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-06-29T14:34:22.728 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-06-29T14:34:22.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.850+0000 7f3dec912640 1 -- 172.21.15.59:0/2686081290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de80b3b50 msgr2=0x7f3de8166ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.850+0000 7f3dec912640 1 --2- 172.21.15.59:0/2686081290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de80b3b50 0x7f3de8166ca0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8005fd0 tx=0x7f3dd802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.850+0000 7f3dec912640 1 -- 172.21.15.59:0/2686081290 shutdown_connections 2024-06-29T14:34:22.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.850+0000 7f3dec912640 1 --2- 172.21.15.59:0/2686081290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de80b3b50 0x7f3de8166ca0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.850+0000 7f3dec912640 1 -- 172.21.15.59:0/2686081290 >> 172.21.15.59:0/2686081290 conn(0x7f3de806c950 msgr2=0x7f3de8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:22.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.850+0000 7f3dec912640 1 -- 172.21.15.59:0/2686081290 shutdown_connections 2024-06-29T14:34:22.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.850+0000 7f3dec912640 1 -- 172.21.15.59:0/2686081290 wait complete. 2024-06-29T14:34:22.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.852+0000 7f3dec912640 1 Processor -- start 2024-06-29T14:34:22.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.852+0000 7f3dec912640 1 -- start start 2024-06-29T14:34:22.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.853+0000 7f3dec912640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de80b3740 0x7f3de815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.853+0000 7f3dec912640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de80b3b50 0x7f3de806db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3dec912640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de8141440 0x7f3de806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3dec912640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3de8167570 con 0x7f3de8141440 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3dec912640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3de81673f0 con 0x7f3de80b3b50 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3dec912640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3de81676f0 con 0x7f3de80b3740 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de8141440 0x7f3de806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de8141440 0x7f3de806e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50736/0 (socket says 172.21.15.59:50736) 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 -- 172.21.15.59:0/560153804 learned_addr learned my addr 172.21.15.59:0/560153804 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de659c640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de80b3740 0x7f3de815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de5d9b640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de80b3b50 0x7f3de806db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 -- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de80b3740 msgr2=0x7f3de815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de80b3740 0x7f3de815b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 -- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de80b3b50 msgr2=0x7f3de806db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de80b3b50 0x7f3de806db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 -- 172.21.15.59:0/560153804 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de8141bb0 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3de6d9d640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de8141440 0x7f3de806e0c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3dd802aea0 tx=0x7f3dd80039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3ddeffd640 1 -- 172.21.15.59:0/560153804 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8031990 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3ddeffd640 1 -- 172.21.15.59:0/560153804 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd8031c20 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3ddeffd640 1 -- 172.21.15.59:0/560153804 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80330d0 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.854+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de815cec0 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.855+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de815d480 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.855+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de80b5f50 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.856+0000 7f3ddeffd640 1 -- 172.21.15.59:0/560153804 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f3dd8033290 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.857+0000 7f3ddeffd640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3dc80764d0 0x7f3dc80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.857+0000 7f3ddeffd640 1 -- 172.21.15.59:0/560153804 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f3dd80b9480 con 0x7f3de8141440 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.857+0000 7f3de659c640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3dc80764d0 0x7f3dc80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.858+0000 7f3de659c640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3dc80764d0 0x7f3dc80789e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3de0007fd0 tx=0x7f3de0009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:22.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:22.861+0000 7f3ddeffd640 1 -- 172.21.15.59:0/560153804 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3de80b5f50 con 0x7f3de8141440 2024-06-29T14:34:23.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.076+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3de8141940 con 0x7f3dc80764d0 2024-06-29T14:34:23.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.163+0000 7f3ddeffd640 1 -- 172.21.15.59:0/560153804 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f3de8141940 con 0x7f3dc80764d0 2024-06-29T14:34:23.162 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-06-29T14:34:23.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.165+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3dc80764d0 msgr2=0x7f3dc80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.165+0000 7f3dec912640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3dc80764d0 0x7f3dc80789e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3de0007fd0 tx=0x7f3de0009040 comp rx=0 tx=0).stop 2024-06-29T14:34:23.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de8141440 msgr2=0x7f3de806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de8141440 0x7f3de806e0c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3dd802aea0 tx=0x7f3dd80039f0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 shutdown_connections 2024-06-29T14:34:23.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3dc80764d0 0x7f3dc80789e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de8141440 0x7f3de806e0c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de80b3b50 0x7f3de806db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 --2- 172.21.15.59:0/560153804 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de80b3740 0x7f3de815b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 >> 172.21.15.59:0/560153804 conn(0x7f3de806c950 msgr2=0x7f3de80b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 shutdown_connections 2024-06-29T14:34:23.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.166+0000 7f3dec912640 1 -- 172.21.15.59:0/560153804 wait complete. 2024-06-29T14:34:23.175 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-06-29T14:34:23.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-06-29T14:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.300+0000 7f2d5b770640 1 -- 172.21.15.59:0/1802539332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 msgr2=0x7f2d40004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.300+0000 7f2d5b770640 1 --2- 172.21.15.59:0/1802539332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 0x7f2d40004c10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2d440016c0 tx=0x7f2d40005830 comp rx=0 tx=0).stop 2024-06-29T14:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.300+0000 7f2d59d0d640 1 -- 172.21.15.59:0/1802539332 reap_dead start 2024-06-29T14:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.300+0000 7f2d5b770640 1 -- 172.21.15.59:0/1802539332 shutdown_connections 2024-06-29T14:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.300+0000 7f2d5b770640 1 -- 172.21.15.59:0/1802539332 >> 172.21.15.59:0/1802539332 conn(0x7f2d540b1ea0 msgr2=0x7f2d540b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.300+0000 7f2d5b770640 1 -- 172.21.15.59:0/1802539332 shutdown_connections 2024-06-29T14:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.300+0000 7f2d5b770640 1 -- 172.21.15.59:0/1802539332 wait complete. 2024-06-29T14:34:23.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.302+0000 7f2d5b770640 1 Processor -- start 2024-06-29T14:34:23.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.302+0000 7f2d5b770640 1 -- start start 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5b770640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 0x7f2d5414a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5b770640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5415d1e0 0x7f2d541530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5b770640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5414a950 0x7f2d541535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5b770640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2d54163ab0 con 0x7f2d40004800 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5b770640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d540ad8f0 con 0x7f2d5414a950 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5b770640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2d54163c30 con 0x7f2d5415d1e0 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 0x7f2d5414a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 0x7f2d5414a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50750/0 (socket says 172.21.15.59:50750) 2024-06-29T14:34:23.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 -- 172.21.15.59:0/459280878 learned_addr learned my addr 172.21.15.59:0/459280878 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d58d0b640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5415d1e0 0x7f2d541530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d59d0d640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5414a950 0x7f2d541535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 -- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5415d1e0 msgr2=0x7f2d541530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5415d1e0 0x7f2d541530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 -- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5414a950 msgr2=0x7f2d541535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5414a950 0x7f2d541535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 -- 172.21.15.59:0/459280878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d540b4fc0 con 0x7f2d40004800 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d59d0d640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5414a950 0x7f2d541535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5950c640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 0x7f2d5414a150 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2d50001bc0 tx=0x7f2d500039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d49ffb640 1 -- 172.21.15.59:0/459280878 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50015070 con 0x7f2d40004800 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d49ffb640 1 -- 172.21.15.59:0/459280878 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d50011840 con 0x7f2d40004800 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5415c420 con 0x7f2d40004800 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.304+0000 7f2d49ffb640 1 -- 172.21.15.59:0/459280878 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50011c70 con 0x7f2d40004800 2024-06-29T14:34:23.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.305+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d5415c9e0 con 0x7f2d40004800 2024-06-29T14:34:23.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.306+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d54073be0 con 0x7f2d40004800 2024-06-29T14:34:23.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.306+0000 7f2d49ffb640 1 -- 172.21.15.59:0/459280878 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f2d50018070 con 0x7f2d40004800 2024-06-29T14:34:23.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.307+0000 7f2d49ffb640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2d380764d0 0x7f2d380789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.307+0000 7f2d49ffb640 1 -- 172.21.15.59:0/459280878 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f2d50098ad0 con 0x7f2d40004800 2024-06-29T14:34:23.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.307+0000 7f2d58d0b640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2d380764d0 0x7f2d380789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.308+0000 7f2d58d0b640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2d380764d0 0x7f2d380789e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d44032000 tx=0x7f2d4402c040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:23.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.311+0000 7f2d49ffb640 1 -- 172.21.15.59:0/459280878 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d50061720 con 0x7f2d40004800 2024-06-29T14:34:23.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.526+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f2d540cc780 con 0x7f2d380764d0 2024-06-29T14:34:23.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.609+0000 7f2d49ffb640 1 -- 172.21.15.59:0/459280878 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f2d540cc780 con 0x7f2d380764d0 2024-06-29T14:34:23.607 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-06-29T14:34:23.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2d380764d0 msgr2=0x7f2d380789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2d380764d0 0x7f2d380789e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d44032000 tx=0x7f2d4402c040 comp rx=0 tx=0).stop 2024-06-29T14:34:23.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 msgr2=0x7f2d5414a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 0x7f2d5414a150 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2d50001bc0 tx=0x7f2d500039c0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 shutdown_connections 2024-06-29T14:34:23.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2d380764d0 0x7f2d380789e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5414a950 0x7f2d541535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d5415d1e0 0x7f2d541530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 --2- 172.21.15.59:0/459280878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d40004800 0x7f2d5414a150 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 >> 172.21.15.59:0/459280878 conn(0x7f2d540b1ea0 msgr2=0x7f2d540acfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 shutdown_connections 2024-06-29T14:34:23.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.611+0000 7f2d5b770640 1 -- 172.21.15.59:0/459280878 wait complete. 2024-06-29T14:34:23.620 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-06-29T14:34:23.621 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-06-29T14:34:23.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.743+0000 7fec28e4e640 1 -- 172.21.15.59:0/2572384326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 msgr2=0x7fec2415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.743+0000 7fec28e4e640 1 --2- 172.21.15.59:0/2572384326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 0x7fec2415f9b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fec1c005fd0 tx=0x7fec1c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.743+0000 7fec28e4e640 1 -- 172.21.15.59:0/2572384326 shutdown_connections 2024-06-29T14:34:23.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.743+0000 7fec28e4e640 1 --2- 172.21.15.59:0/2572384326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 0x7fec2415f9b0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.743+0000 7fec28e4e640 1 --2- 172.21.15.59:0/2572384326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec240b40a0 0x7fec2415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.743+0000 7fec28e4e640 1 -- 172.21.15.59:0/2572384326 >> 172.21.15.59:0/2572384326 conn(0x7fec240d3ee0 msgr2=0x7fec240d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.743+0000 7fec28e4e640 1 -- 172.21.15.59:0/2572384326 shutdown_connections 2024-06-29T14:34:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.743+0000 7fec28e4e640 1 -- 172.21.15.59:0/2572384326 wait complete. 2024-06-29T14:34:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.745+0000 7fec28e4e640 1 Processor -- start 2024-06-29T14:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.746+0000 7fec28e4e640 1 -- start start 2024-06-29T14:34:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec28e4e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec240d23c0 0x7fec241f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec28e4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 0x7fec241555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec28e4e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec24153220 0x7fec24153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec28e4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fec24160280 con 0x7fec2415ee10 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec28e4e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec24160100 con 0x7fec24153220 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec28e4e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fec24160400 con 0x7fec240d23c0 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec21d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 0x7fec241555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec21d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 0x7fec241555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50778/0 (socket says 172.21.15.59:50778) 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec21d9b640 1 -- 172.21.15.59:0/4033422858 learned_addr learned my addr 172.21.15.59:0/4033422858 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec2259c640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec240d23c0 0x7fec241f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec22d9d640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec24153220 0x7fec24153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec21d9b640 1 -- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec240d23c0 msgr2=0x7fec241f7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec21d9b640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec240d23c0 0x7fec241f7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec21d9b640 1 -- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec24153220 msgr2=0x7fec24153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec21d9b640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec24153220 0x7fec24153630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.747+0000 7fec21d9b640 1 -- 172.21.15.59:0/4033422858 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec241f6cd0 con 0x7fec2415ee10 2024-06-29T14:34:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.748+0000 7fec21d9b640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 0x7fec241555a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00d780 tx=0x7fec0c00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.748+0000 7fec1affd640 1 -- 172.21.15.59:0/4033422858 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c017070 con 0x7fec2415ee10 2024-06-29T14:34:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.748+0000 7fec1affd640 1 -- 172.21.15.59:0/4033422858 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec0c00de90 con 0x7fec2415ee10 2024-06-29T14:34:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.748+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec241f6fb0 con 0x7fec2415ee10 2024-06-29T14:34:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.748+0000 7fec1affd640 1 -- 172.21.15.59:0/4033422858 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c0099d0 con 0x7fec2415ee10 2024-06-29T14:34:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.748+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec241f7570 con 0x7fec2415ee10 2024-06-29T14:34:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.749+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec2415e8d0 con 0x7fec2415ee10 2024-06-29T14:34:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.750+0000 7fec1affd640 1 -- 172.21.15.59:0/4033422858 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fec0c009b70 con 0x7fec2415ee10 2024-06-29T14:34:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.750+0000 7fec1affd640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fec040764d0 0x7fec040789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.750+0000 7fec1affd640 1 -- 172.21.15.59:0/4033422858 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fec0c098140 con 0x7fec2415ee10 2024-06-29T14:34:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.750+0000 7fec2259c640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fec040764d0 0x7fec040789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.751+0000 7fec2259c640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fec040764d0 0x7fec040789e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fec240d2e40 tx=0x7fec14009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:23.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.755+0000 7fec1affd640 1 -- 172.21.15.59:0/4033422858 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fec0c020020 con 0x7fec2415ee10 2024-06-29T14:34:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:23.971+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fec241f82b0 con 0x7fec040764d0 2024-06-29T14:34:24.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.056+0000 7fec1affd640 1 -- 172.21.15.59:0/4033422858 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fec14001ac0 con 0x7fec040764d0 2024-06-29T14:34:24.054 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-06-29T14:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fec040764d0 msgr2=0x7fec040789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fec040764d0 0x7fec040789e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fec240d2e40 tx=0x7fec14009040 comp rx=0 tx=0).stop 2024-06-29T14:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 msgr2=0x7fec241555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 0x7fec241555a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00d780 tx=0x7fec0c00dc40 comp rx=0 tx=0).stop 2024-06-29T14:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 shutdown_connections 2024-06-29T14:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fec040764d0 0x7fec040789e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec24153220 0x7fec24153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec2415ee10 0x7fec241555a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 --2- 172.21.15.59:0/4033422858 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec240d23c0 0x7fec241f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 >> 172.21.15.59:0/4033422858 conn(0x7fec240d3ee0 msgr2=0x7fec2406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 shutdown_connections 2024-06-29T14:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.058+0000 7fec28e4e640 1 -- 172.21.15.59:0/4033422858 wait complete. 2024-06-29T14:34:24.067 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-06-29T14:34:24.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:24.067 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-06-29T14:34:24.085 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:24.085 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:24.085 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:24.085 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:24.085 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:24.085 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:24.085 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:24.086 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:24.086 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:24.086 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:24.086 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:24.086 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:24.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-06-29T14:34:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.248+0000 7f52010fe640 1 -- 172.21.15.59:0/295802670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51e4004e80 msgr2=0x7f51e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.248+0000 7f52010fe640 1 --2- 172.21.15.59:0/295802670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51e4004e80 0x7f51e400b9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f51f4001830 tx=0x7f51e4005830 comp rx=0 tx=0).stop 2024-06-29T14:34:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.248+0000 7f51fb59e640 1 -- 172.21.15.59:0/295802670 reap_dead start 2024-06-29T14:34:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.248+0000 7f52010fe640 1 -- 172.21.15.59:0/295802670 shutdown_connections 2024-06-29T14:34:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.248+0000 7f52010fe640 1 -- 172.21.15.59:0/295802670 >> 172.21.15.59:0/295802670 conn(0x7f51fc0b5610 msgr2=0x7f51fc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.248+0000 7f52010fe640 1 -- 172.21.15.59:0/295802670 shutdown_connections 2024-06-29T14:34:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.248+0000 7f52010fe640 1 -- 172.21.15.59:0/295802670 wait complete. 2024-06-29T14:34:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.250+0000 7f52010fe640 1 Processor -- start 2024-06-29T14:34:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.250+0000 7f52010fe640 1 -- start start 2024-06-29T14:34:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f52010fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51fc1572c0 0x7f51fc156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f52010fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51fc15edc0 0x7f51fc0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f52010fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51fc0cd6c0 0x7f51fc1f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f52010fe640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc0cf080 con 0x7f51fc1572c0 2024-06-29T14:34:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f52010fe640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc0b8f20 con 0x7f51fc15edc0 2024-06-29T14:34:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f52010fe640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f51fc0cf200 con 0x7f51fc0cd6c0 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51fc1572c0 0x7f51fc156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51fc1572c0 0x7f51fc156b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50802/0 (socket says 172.21.15.59:50802) 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 -- 172.21.15.59:0/2037171509 learned_addr learned my addr 172.21.15.59:0/2037171509 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fa59c640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51fc15edc0 0x7f51fc0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fb59e640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51fc0cd6c0 0x7f51fc1f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 -- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51fc0cd6c0 msgr2=0x7f51fc1f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51fc0cd6c0 0x7f51fc1f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 -- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51fc15edc0 msgr2=0x7f51fc0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51fc15edc0 0x7f51fc0cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 -- 172.21.15.59:0/2037171509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc0cdad0 con 0x7f51fc1572c0 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.252+0000 7f51fad9d640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51fc1572c0 0x7f51fc156b10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f51ec010b50 tx=0x7f51ec010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.253+0000 7f51e37fe640 1 -- 172.21.15.59:0/2037171509 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec00ee60 con 0x7f51fc1572c0 2024-06-29T14:34:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.253+0000 7f51e37fe640 1 -- 172.21.15.59:0/2037171509 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51ec004c10 con 0x7f51fc1572c0 2024-06-29T14:34:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.253+0000 7f51e37fe640 1 -- 172.21.15.59:0/2037171509 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec005020 con 0x7f51fc1572c0 2024-06-29T14:34:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.253+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51fc15afc0 con 0x7f51fc1572c0 2024-06-29T14:34:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.253+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51fc161b30 con 0x7f51fc1572c0 2024-06-29T14:34:24.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.254+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51fc0d1a20 con 0x7f51fc1572c0 2024-06-29T14:34:24.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.254+0000 7f51e37fe640 1 -- 172.21.15.59:0/2037171509 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f51ec01a030 con 0x7f51fc1572c0 2024-06-29T14:34:24.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.255+0000 7f51e37fe640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51e80764d0 0x7f51e80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.255+0000 7f51e37fe640 1 -- 172.21.15.59:0/2037171509 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f51ec098860 con 0x7f51fc1572c0 2024-06-29T14:34:24.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.255+0000 7f51fa59c640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51e80764d0 0x7f51e80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.255+0000 7f51fa59c640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51e80764d0 0x7f51e80789e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0b5c60 tx=0x7f51f0006040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.259+0000 7f51e37fe640 1 -- 172.21.15.59:0/2037171509 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f51fc0cdad0 con 0x7f51fc1572c0 2024-06-29T14:34:24.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.472+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f51fc1632c0 con 0x7f51e80764d0 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.483+0000 7f51e37fe640 1 -- 172.21.15.59:0/2037171509 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f51fc1632c0 con 0x7f51e80764d0 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:33", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51e80764d0 msgr2=0x7f51e80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51e80764d0 0x7f51e80789e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0b5c60 tx=0x7f51f0006040 comp rx=0 tx=0).stop 2024-06-29T14:34:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51fc1572c0 msgr2=0x7f51fc156b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51fc1572c0 0x7f51fc156b10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f51ec010b50 tx=0x7f51ec010f50 comp rx=0 tx=0).stop 2024-06-29T14:34:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 shutdown_connections 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f51e80764d0 0x7f51e80789e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51fc0cd6c0 0x7f51fc1f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51fc15edc0 0x7f51fc0cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 --2- 172.21.15.59:0/2037171509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51fc1572c0 0x7f51fc156b10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 >> 172.21.15.59:0/2037171509 conn(0x7f51fc0b5610 msgr2=0x7f51fc15cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 shutdown_connections 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.485+0000 7f52010fe640 1 -- 172.21.15.59:0/2037171509 wait complete. 2024-06-29T14:34:24.497 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:33", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:24.498 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:33', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:24.498 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:24.498 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:24.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:24.498 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-06-29T14:34:24.515 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:24.515 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:24.515 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:24.515 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:24.516 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:24.516 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-06-29T14:34:24.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.678+0000 7f378c2cc640 1 -- 172.21.15.59:0/2017838208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f378415a400 msgr2=0x7f37840cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.678+0000 7f378c2cc640 1 --2- 172.21.15.59:0/2017838208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f378415a400 0x7f37840cf120 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3778009830 tx=0x7f377802a5c0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.678+0000 7f378c2cc640 1 -- 172.21.15.59:0/2017838208 shutdown_connections 2024-06-29T14:34:24.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.678+0000 7f378c2cc640 1 --2- 172.21.15.59:0/2017838208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f378415a400 0x7f37840cf120 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.678+0000 7f378c2cc640 1 -- 172.21.15.59:0/2017838208 >> 172.21.15.59:0/2017838208 conn(0x7f37840b24c0 msgr2=0x7f37840cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.678+0000 7f378c2cc640 1 -- 172.21.15.59:0/2017838208 shutdown_connections 2024-06-29T14:34:24.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.678+0000 7f378c2cc640 1 -- 172.21.15.59:0/2017838208 wait complete. 2024-06-29T14:34:24.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.680+0000 7f378c2cc640 1 Processor -- start 2024-06-29T14:34:24.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.680+0000 7f378c2cc640 1 -- start start 2024-06-29T14:34:24.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f378c2cc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f378406c1e0 0x7f378414e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f378c2cc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3784073f30 0x7f378414f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f378c2cc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f378415a400 0x7f378415e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f378c2cc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3784073d20 con 0x7f3784073f30 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f378c2cc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37840b43a0 con 0x7f378415a400 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f378c2cc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f378406c5f0 con 0x7f378406c1e0 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f3789867640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3784073f30 0x7f378414f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f3789867640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3784073f30 0x7f378414f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50820/0 (socket says 172.21.15.59:50820) 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f3789867640 1 -- 172.21.15.59:0/1583200654 learned_addr learned my addr 172.21.15.59:0/1583200654 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f378a869640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f378415a400 0x7f378415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.681+0000 7f378a068640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f378406c1e0 0x7f378414e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3789867640 1 -- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f378406c1e0 msgr2=0x7f378414e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3789867640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f378406c1e0 0x7f378414e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3789867640 1 -- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f378415a400 msgr2=0x7f378415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3789867640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f378415a400 0x7f378415e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3789867640 1 -- 172.21.15.59:0/1583200654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37841f6850 con 0x7f3784073f30 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3789867640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3784073f30 0x7f378414f3b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f377802abb0 tx=0x7f3778033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3782ffd640 1 -- 172.21.15.59:0/1583200654 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778031b60 con 0x7f3784073f30 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3782ffd640 1 -- 172.21.15.59:0/1583200654 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3778031df0 con 0x7f3784073f30 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f3782ffd640 1 -- 172.21.15.59:0/1583200654 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778005280 con 0x7f3784073f30 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37841f6b30 con 0x7f3784073f30 2024-06-29T14:34:24.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.682+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37841f70f0 con 0x7f3784073f30 2024-06-29T14:34:24.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.683+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37840cf120 con 0x7f3784073f30 2024-06-29T14:34:24.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.684+0000 7f3782ffd640 1 -- 172.21.15.59:0/1583200654 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f3778005420 con 0x7f3784073f30 2024-06-29T14:34:24.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.684+0000 7f3782ffd640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3770076500 0x7f3770078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.684+0000 7f3782ffd640 1 -- 172.21.15.59:0/1583200654 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f37780b8f50 con 0x7f3784073f30 2024-06-29T14:34:24.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.685+0000 7f378a068640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3770076500 0x7f3770078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.685+0000 7f378a068640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3770076500 0x7f3770078a10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37740012e0 tx=0x7f3774009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.689+0000 7f3782ffd640 1 -- 172.21.15.59:0/1583200654 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f377803f030 con 0x7f3784073f30 2024-06-29T14:34:24.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.902+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f37841556b0 con 0x7f3770076500 2024-06-29T14:34:24.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.912+0000 7f3782ffd640 1 -- 172.21.15.59:0/1583200654 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f37841556b0 con 0x7f3770076500 2024-06-29T14:34:24.910 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:24.910 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:34", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3770076500 msgr2=0x7f3770078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3770076500 0x7f3770078a10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37740012e0 tx=0x7f3774009040 comp rx=0 tx=0).stop 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3784073f30 msgr2=0x7f378414f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3784073f30 0x7f378414f3b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f377802abb0 tx=0x7f3778033c10 comp rx=0 tx=0).stop 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 shutdown_connections 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3770076500 0x7f3770078a10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f378415a400 0x7f378415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3784073f30 0x7f378414f3b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 --2- 172.21.15.59:0/1583200654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f378406c1e0 0x7f378414e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 >> 172.21.15.59:0/1583200654 conn(0x7f37840b24c0 msgr2=0x7f3784068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 shutdown_connections 2024-06-29T14:34:24.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:24.914+0000 7f378c2cc640 1 -- 172.21.15.59:0/1583200654 wait complete. 2024-06-29T14:34:24.923 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:34", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:24.923 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:34', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:24.923 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:24.923 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:24.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:24.924 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-06-29T14:34:24.940 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:24.940 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:24.940 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:24.940 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:24.940 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:24.941 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_01_UTC 2024-06-29T14:34:24.941 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:24.941 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:24.941 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:24.941 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:24.941 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:24.941 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_01_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:24.941 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-06-29T14:34:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.103+0000 7f3971cd6640 1 -- 172.21.15.59:0/4209474854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c0ce950 msgr2=0x7f396c0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.103+0000 7f3971cd6640 1 --2- 172.21.15.59:0/4209474854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c0ce950 0x7f396c0d2130 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3964005fd0 tx=0x7f396402aa90 comp rx=0 tx=0).stop 2024-06-29T14:34:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.103+0000 7f3971cd6640 1 -- 172.21.15.59:0/4209474854 shutdown_connections 2024-06-29T14:34:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.103+0000 7f3971cd6640 1 --2- 172.21.15.59:0/4209474854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c0ce950 0x7f396c0d2130 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.103+0000 7f3971cd6640 1 --2- 172.21.15.59:0/4209474854 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f396c08c190 0x7f396c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.103+0000 7f3971cd6640 1 -- 172.21.15.59:0/4209474854 >> 172.21.15.59:0/4209474854 conn(0x7f396c0b5610 msgr2=0x7f396c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.103+0000 7f3971cd6640 1 -- 172.21.15.59:0/4209474854 shutdown_connections 2024-06-29T14:34:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.103+0000 7f3971cd6640 1 -- 172.21.15.59:0/4209474854 wait complete. 2024-06-29T14:34:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.105+0000 7f3971cd6640 1 Processor -- start 2024-06-29T14:34:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.105+0000 7f3971cd6640 1 -- start start 2024-06-29T14:34:25.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.106+0000 7f3971cd6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c08c190 0x7f396c1f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.106+0000 7f3971cd6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f396c0ce950 0x7f396c14cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f3971cd6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0cf5c0 0x7f396c15af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f3971cd6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f396c0b7fa0 con 0x7f396c0cf5c0 2024-06-29T14:34:25.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f3971cd6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f396c0b7e20 con 0x7f396c08c190 2024-06-29T14:34:25.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f3971cd6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f396c0b8120 con 0x7f396c0ce950 2024-06-29T14:34:25.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0cf5c0 0x7f396c15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0cf5c0 0x7f396c15af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50822/0 (socket says 172.21.15.59:50822) 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 -- 172.21.15.59:0/1697597468 learned_addr learned my addr 172.21.15.59:0/1697597468 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396b7fe640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c08c190 0x7f396c1f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396affd640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f396c0ce950 0x7f396c14cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 -- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f396c0ce950 msgr2=0x7f396c14cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f396c0ce950 0x7f396c14cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 -- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c08c190 msgr2=0x7f396c1f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c08c190 0x7f396c1f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 -- 172.21.15.59:0/1697597468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c1f6080 con 0x7f396c0cf5c0 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f396bfff640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0cf5c0 0x7f396c15af10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3964009d90 tx=0x7f3964003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.107+0000 7f3953fff640 1 -- 172.21.15.59:0/1697597468 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964044070 con 0x7f396c0cf5c0 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.108+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c1593c0 con 0x7f396c0cf5c0 2024-06-29T14:34:25.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.108+0000 7f3953fff640 1 -- 172.21.15.59:0/1697597468 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3964003e00 con 0x7f396c0cf5c0 2024-06-29T14:34:25.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.108+0000 7f3953fff640 1 -- 172.21.15.59:0/1697597468 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39640044b0 con 0x7f396c0cf5c0 2024-06-29T14:34:25.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.108+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c159980 con 0x7f396c0cf5c0 2024-06-29T14:34:25.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.109+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c0cf080 con 0x7f396c0cf5c0 2024-06-29T14:34:25.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.110+0000 7f3953fff640 1 -- 172.21.15.59:0/1697597468 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f3964047070 con 0x7f396c0cf5c0 2024-06-29T14:34:25.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.110+0000 7f3953fff640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3958076500 0x7f3958078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.110+0000 7f3953fff640 1 -- 172.21.15.59:0/1697597468 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f39640b8c10 con 0x7f396c0cf5c0 2024-06-29T14:34:25.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.110+0000 7f396b7fe640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3958076500 0x7f3958078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.111+0000 7f396b7fe640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3958076500 0x7f3958078a10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f395c0095f0 tx=0x7f395c005f90 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.115+0000 7f3953fff640 1 -- 172.21.15.59:0/1697597468 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3964032030 con 0x7f396c0cf5c0 2024-06-29T14:34:25.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.326+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f396c159150 con 0x7f3958076500 2024-06-29T14:34:25.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.338+0000 7f3953fff640 1 -- 172.21.15.59:0/1697597468 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f396c159150 con 0x7f3958076500 2024-06-29T14:34:25.337 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:25.337 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:34", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3958076500 msgr2=0x7f3958078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3958076500 0x7f3958078a10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f395c0095f0 tx=0x7f395c005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:25.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0cf5c0 msgr2=0x7f396c15af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0cf5c0 0x7f396c15af10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3964009d90 tx=0x7f3964003c10 comp rx=0 tx=0).stop 2024-06-29T14:34:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 shutdown_connections 2024-06-29T14:34:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3958076500 0x7f3958078a10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f396c0cf5c0 0x7f396c15af10 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f396c0ce950 0x7f396c14cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 --2- 172.21.15.59:0/1697597468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f396c08c190 0x7f396c1f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 >> 172.21.15.59:0/1697597468 conn(0x7f396c0b5610 msgr2=0x7f396c0c9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 shutdown_connections 2024-06-29T14:34:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.341+0000 7f3971cd6640 1 -- 172.21.15.59:0/1697597468 wait complete. 2024-06-29T14:34:25.351 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:34", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.351 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:34', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:25.351 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:25.351 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:25.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:25.352 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-06-29T14:34:25.369 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:25.369 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:25.369 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:25.370 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:25.371 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:25.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-06-29T14:34:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.531+0000 7fbebaa1f640 1 -- 172.21.15.59:0/967221007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb415ceb0 msgr2=0x7fbeb40b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.531+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/967221007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb415ceb0 0x7fbeb40b56d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbea4005fd0 tx=0x7fbea402a540 comp rx=0 tx=0).stop 2024-06-29T14:34:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.531+0000 7fbebaa1f640 1 -- 172.21.15.59:0/967221007 shutdown_connections 2024-06-29T14:34:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.531+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/967221007 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeb4141bd0 0x7fbeb413edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.531+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/967221007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb415ceb0 0x7fbeb40b56d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.531+0000 7fbebaa1f640 1 -- 172.21.15.59:0/967221007 >> 172.21.15.59:0/967221007 conn(0x7fbeb40b6fb0 msgr2=0x7fbeb40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.531+0000 7fbebaa1f640 1 -- 172.21.15.59:0/967221007 shutdown_connections 2024-06-29T14:34:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.531+0000 7fbebaa1f640 1 -- 172.21.15.59:0/967221007 wait complete. 2024-06-29T14:34:25.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.533+0000 7fbebaa1f640 1 Processor -- start 2024-06-29T14:34:25.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.533+0000 7fbebaa1f640 1 -- start start 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbebaa1f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb413e9b0 0x7fbeb4138d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbebaa1f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeb415ceb0 0x7fbeb4151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbebaa1f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb4155da0 0x7fbeb413d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbebaa1f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb40b2a00 con 0x7fbeb413e9b0 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbebaa1f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb40b2880 con 0x7fbeb4155da0 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbebaa1f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb40b2b80 con 0x7fbeb415ceb0 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbeb3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb413e9b0 0x7fbeb4138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbeb3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb413e9b0 0x7fbeb4138d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50838/0 (socket says 172.21.15.59:50838) 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.534+0000 7fbeb3fff640 1 -- 172.21.15.59:0/4021021767 learned_addr learned my addr 172.21.15.59:0/4021021767 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb8fbc640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb4155da0 0x7fbeb413d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb3fff640 1 -- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeb415ceb0 msgr2=0x7fbeb4151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb3fff640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeb415ceb0 0x7fbeb4151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb3fff640 1 -- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb4155da0 msgr2=0x7fbeb413d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb3fff640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb4155da0 0x7fbeb413d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb3fff640 1 -- 172.21.15.59:0/4021021767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb4139580 con 0x7fbeb413e9b0 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb8fbc640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb4155da0 0x7fbeb413d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb3fff640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb413e9b0 0x7fbeb4138d30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbea4034040 tx=0x7fbea40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb0ff9640 1 -- 172.21.15.59:0/4021021767 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea403e030 con 0x7fbeb413e9b0 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb0ff9640 1 -- 172.21.15.59:0/4021021767 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbea4003d90 con 0x7fbeb413e9b0 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb4139860 con 0x7fbeb413e9b0 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbeb0ff9640 1 -- 172.21.15.59:0/4021021767 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4035040 con 0x7fbeb413e9b0 2024-06-29T14:34:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.535+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb4139e20 con 0x7fbeb413e9b0 2024-06-29T14:34:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.536+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb413a060 con 0x7fbeb413e9b0 2024-06-29T14:34:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.537+0000 7fbeb0ff9640 1 -- 172.21.15.59:0/4021021767 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fbea4041030 con 0x7fbeb413e9b0 2024-06-29T14:34:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.538+0000 7fbeb0ff9640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbea00764d0 0x7fbea00789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.538+0000 7fbeb0ff9640 1 -- 172.21.15.59:0/4021021767 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fbea40b8620 con 0x7fbeb413e9b0 2024-06-29T14:34:25.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.538+0000 7fbeb37fe640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbea00764d0 0x7fbea00789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.539+0000 7fbeb37fe640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbea00764d0 0x7fbea00789e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbeb415dd00 tx=0x7fbea8007970 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.542+0000 7fbeb0ff9640 1 -- 172.21.15.59:0/4021021767 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbeb413a060 con 0x7fbeb413e9b0 2024-06-29T14:34:25.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:25.630 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.757+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbeb413edc0 con 0x7fbea00764d0 2024-06-29T14:34:25.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.768+0000 7fbeb0ff9640 1 -- 172.21.15.59:0/4021021767 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7fbeb413edc0 con 0x7fbea00764d0 2024-06-29T14:34:25.766 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:25.767 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:35", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbea00764d0 msgr2=0x7fbea00789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbea00764d0 0x7fbea00789e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbeb415dd00 tx=0x7fbea8007970 comp rx=0 tx=0).stop 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb413e9b0 msgr2=0x7fbeb4138d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb413e9b0 0x7fbeb4138d30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbea4034040 tx=0x7fbea40038a0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 shutdown_connections 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fbea00764d0 0x7fbea00789e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbeb4155da0 0x7fbeb413d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeb415ceb0 0x7fbeb4151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 --2- 172.21.15.59:0/4021021767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeb413e9b0 0x7fbeb4138d30 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 >> 172.21.15.59:0/4021021767 conn(0x7fbeb40b6fb0 msgr2=0x7fbeb4152ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 shutdown_connections 2024-06-29T14:34:25.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.770+0000 7fbebaa1f640 1 -- 172.21.15.59:0/4021021767 wait complete. 2024-06-29T14:34:25.779 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:35", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.780 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:35', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:25.780 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:25.780 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:25.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:25.780 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-06-29T14:34:25.797 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:25.797 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:25.797 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:25.797 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:25.798 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:25.798 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:25.798 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:25.798 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:25.798 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:25.798 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:25.798 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:25.798 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:25.798 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-06-29T14:34:25.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.959+0000 7f83b193c640 1 -- 172.21.15.59:0/2678782008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac06ea10 msgr2=0x7f83ac0cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.959+0000 7f83b193c640 1 --2- 172.21.15.59:0/2678782008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac06ea10 0x7f83ac0cf1d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009830 tx=0x7f83a402a510 comp rx=0 tx=0).stop 2024-06-29T14:34:25.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.959+0000 7f83b193c640 1 -- 172.21.15.59:0/2678782008 shutdown_connections 2024-06-29T14:34:25.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.959+0000 7f83b193c640 1 --2- 172.21.15.59:0/2678782008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac06ea10 0x7f83ac0cf1d0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.959+0000 7f83b193c640 1 --2- 172.21.15.59:0/2678782008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac144470 0x7f83ac0b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.959+0000 7f83b193c640 1 -- 172.21.15.59:0/2678782008 >> 172.21.15.59:0/2678782008 conn(0x7f83ac0b06e0 msgr2=0x7f83ac0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.959+0000 7f83b193c640 1 -- 172.21.15.59:0/2678782008 shutdown_connections 2024-06-29T14:34:25.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.959+0000 7f83b193c640 1 -- 172.21.15.59:0/2678782008 wait complete. 2024-06-29T14:34:25.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.960+0000 7f83b193c640 1 Processor -- start 2024-06-29T14:34:25.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.961+0000 7f83b193c640 1 -- start start 2024-06-29T14:34:25.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83b193c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac06ea10 0x7f83ac13dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83b193c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac0b2040 0x7f83ac158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83b193c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac144470 0x7f83ac158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83b193c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac0cfaa0 con 0x7f83ac144470 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83b193c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac0cf920 con 0x7f83ac06ea10 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83b193c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f83ac0cfc20 con 0x7f83ac0b2040 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83ab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac144470 0x7f83ac158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83ab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac144470 0x7f83ac158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50860/0 (socket says 172.21.15.59:50860) 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83ab7fe640 1 -- 172.21.15.59:0/3723494920 learned_addr learned my addr 172.21.15.59:0/3723494920 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83aa7fc640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac0b2040 0x7f83ac158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83aaffd640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac06ea10 0x7f83ac13dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83ab7fe640 1 -- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac0b2040 msgr2=0x7f83ac158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83ab7fe640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac0b2040 0x7f83ac158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83ab7fe640 1 -- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac06ea10 msgr2=0x7f83ac13dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83ab7fe640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac06ea10 0x7f83ac13dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83ab7fe640 1 -- 172.21.15.59:0/3723494920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ac0b4ea0 con 0x7f83ac144470 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83aa7fc640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac0b2040 0x7f83ac158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.962+0000 7f83aaffd640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac06ea10 0x7f83ac13dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.963+0000 7f83ab7fe640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac144470 0x7f83ac158ff0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009920 tx=0x7f83a4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.963+0000 7f83a3fff640 1 -- 172.21.15.59:0/3723494920 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4031d60 con 0x7f83ac144470 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.963+0000 7f83a3fff640 1 -- 172.21.15.59:0/3723494920 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83a4033e40 con 0x7f83ac144470 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.963+0000 7f83a3fff640 1 -- 172.21.15.59:0/3723494920 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4005240 con 0x7f83ac144470 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.963+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ac0b5180 con 0x7f83ac144470 2024-06-29T14:34:25.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.963+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ac1569f0 con 0x7f83ac144470 2024-06-29T14:34:25.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.964+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83ac0b4c10 con 0x7f83ac144470 2024-06-29T14:34:25.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.965+0000 7f83a3fff640 1 -- 172.21.15.59:0/3723494920 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f83a40053e0 con 0x7f83ac144470 2024-06-29T14:34:25.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.965+0000 7f83a3fff640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f838c076500 0x7f838c078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.965+0000 7f83a3fff640 1 -- 172.21.15.59:0/3723494920 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f83a40b9060 con 0x7f83ac144470 2024-06-29T14:34:25.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.966+0000 7f83aaffd640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f838c076500 0x7f838c078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.966+0000 7f83aaffd640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f838c076500 0x7f838c078a10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f839c005c40 tx=0x7f839c00a410 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:25.970+0000 7f83a3fff640 1 -- 172.21.15.59:0/3723494920 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f83a403f030 con 0x7f83ac144470 2024-06-29T14:34:26.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.185+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f83ac0b56a0 con 0x7f838c076500 2024-06-29T14:34:26.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.196+0000 7f83a3fff640 1 -- 172.21.15.59:0/3723494920 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f83ac0b56a0 con 0x7f838c076500 2024-06-29T14:34:26.194 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:26.194 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:35", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:26.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f838c076500 msgr2=0x7f838c078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f838c076500 0x7f838c078a10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f839c005c40 tx=0x7f839c00a410 comp rx=0 tx=0).stop 2024-06-29T14:34:26.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac144470 msgr2=0x7f83ac158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac144470 0x7f83ac158ff0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009920 tx=0x7f83a4033c20 comp rx=0 tx=0).stop 2024-06-29T14:34:26.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 shutdown_connections 2024-06-29T14:34:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f838c076500 0x7f838c078a10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac144470 0x7f83ac158ff0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83ac0b2040 0x7f83ac158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 --2- 172.21.15.59:0/3723494920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83ac06ea10 0x7f83ac13dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 >> 172.21.15.59:0/3723494920 conn(0x7f83ac0b06e0 msgr2=0x7f83ac15c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 shutdown_connections 2024-06-29T14:34:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.198+0000 7f83b193c640 1 -- 172.21.15.59:0/3723494920 wait complete. 2024-06-29T14:34:26.208 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:35", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:26.208 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:35', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:26.208 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:26.208 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:26.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:26.209 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-06-29T14:34:26.227 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:26.227 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:26.227 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:26.227 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:26.227 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:26.228 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:26.228 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:26.228 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:26.228 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:26.228 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:26.228 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:26.228 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:26.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-06-29T14:34:26.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.388+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1896617522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f81458a0 msgr2=0x7fe1f80cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.388+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1896617522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f81458a0 0x7fe1f80cf870 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80b1c40 tx=0x7fe1f002a560 comp rx=0 tx=0).stop 2024-06-29T14:34:26.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.388+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1896617522 shutdown_connections 2024-06-29T14:34:26.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.388+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1896617522 >> v2:172.21.15.153:3300/0 conn(0x7fe1f80b3740 0x7fe1f80b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.388+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1896617522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f81458a0 0x7fe1f80cf870 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.388+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1896617522 >> 172.21.15.59:0/1896617522 conn(0x7fe1f806c950 msgr2=0x7fe1f8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.388+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1896617522 shutdown_connections 2024-06-29T14:34:26.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.388+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1896617522 wait complete. 2024-06-29T14:34:26.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.390+0000 7fe1fcb23640 1 Processor -- start 2024-06-29T14:34:26.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.390+0000 7fe1fcb23640 1 -- start start 2024-06-29T14:34:26.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1fcb23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f80b3740 0x7fe1f80abb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1fcb23640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1f815c2f0 0x7fe1f81f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1fcb23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f80aa4f0 0x7fe1f80ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1fcb23640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f8160020 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1fcb23640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f815cbb0 con 0x7fe1f80b3740 2024-06-29T14:34:26.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1fcb23640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f81601a0 con 0x7fe1f815c2f0 2024-06-29T14:34:26.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f80aa4f0 0x7fe1f80ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f80aa4f0 0x7fe1f80ade60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50878/0 (socket says 172.21.15.59:50878) 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 -- 172.21.15.59:0/1126869612 learned_addr learned my addr 172.21.15.59:0/1126869612 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f659c640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f80b3740 0x7fe1f80abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f5d9b640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1f815c2f0 0x7fe1f81f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 -- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1f815c2f0 msgr2=0x7fe1f81f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1f815c2f0 0x7fe1f81f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 -- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f80b3740 msgr2=0x7fe1f80abb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f80b3740 0x7fe1f80abb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 -- 172.21.15.59:0/1126869612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f81f3800 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1f6d9d640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f80aa4f0 0x7fe1f80ade60 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800b9e0 tx=0x7fe1e800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.392+0000 7fe1e6ffd640 1 -- 172.21.15.59:0/1126869612 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8019070 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.393+0000 7fe1e6ffd640 1 -- 172.21.15.59:0/1126869612 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1e800cc10 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.393+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f80aaad0 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.393+0000 7fe1e6ffd640 1 -- 172.21.15.59:0/1126869612 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e80146a0 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.393+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f80ab090 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.394+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f81606c0 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.395+0000 7fe1e6ffd640 1 -- 172.21.15.59:0/1126869612 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fe1e801c070 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.395+0000 7fe1e6ffd640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe1d8076500 0x7fe1d8078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.395+0000 7fe1e6ffd640 1 -- 172.21.15.59:0/1126869612 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fe1e8097b50 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.395+0000 7fe1f659c640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe1d8076500 0x7fe1d8078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.396+0000 7fe1f659c640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe1d8076500 0x7fe1d8078a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0033000 tx=0x7fe1f0004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.400+0000 7fe1e6ffd640 1 -- 172.21.15.59:0/1126869612 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe1f81f3800 con 0x7fe1f80aa4f0 2024-06-29T14:34:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.611+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1f80b1ef0 con 0x7fe1d8076500 2024-06-29T14:34:26.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.622+0000 7fe1e6ffd640 1 -- 172.21.15.59:0/1126869612 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7fe1f80b1ef0 con 0x7fe1d8076500 2024-06-29T14:34:26.621 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:26.621 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:36", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe1d8076500 msgr2=0x7fe1d8078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe1d8076500 0x7fe1d8078a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0033000 tx=0x7fe1f0004730 comp rx=0 tx=0).stop 2024-06-29T14:34:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f80aa4f0 msgr2=0x7fe1f80ade60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f80aa4f0 0x7fe1f80ade60 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800b9e0 tx=0x7fe1e800bea0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 shutdown_connections 2024-06-29T14:34:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fe1d8076500 0x7fe1d8078a10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1f80aa4f0 0x7fe1f80ade60 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1f815c2f0 0x7fe1f81f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 --2- 172.21.15.59:0/1126869612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1f80b3740 0x7fe1f80abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 >> 172.21.15.59:0/1126869612 conn(0x7fe1f806c950 msgr2=0x7fe1f806fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 shutdown_connections 2024-06-29T14:34:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.625+0000 7fe1fcb23640 1 -- 172.21.15.59:0/1126869612 wait complete. 2024-06-29T14:34:26.634 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:36", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:26.634 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:36', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:26.634 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:26.634 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:26.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:26.635 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-06-29T14:34:26.652 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:26.652 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:26.652 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:26.652 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:26.652 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:26.652 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:26.652 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:26.653 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:26.653 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:26.653 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:26.653 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:26.653 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:26.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-06-29T14:34:26.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.814+0000 7f23105e7640 1 -- 172.21.15.59:0/4167113670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 msgr2=0x7f2308162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.814+0000 7f23105e7640 1 --2- 172.21.15.59:0/4167113670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 0x7f2308162c10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2300005fd0 tx=0x7f230002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.814+0000 7f23105e7640 1 -- 172.21.15.59:0/4167113670 shutdown_connections 2024-06-29T14:34:26.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.814+0000 7f23105e7640 1 --2- 172.21.15.59:0/4167113670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 0x7f2308162c10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.814+0000 7f23105e7640 1 --2- 172.21.15.59:0/4167113670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23080ce950 0x7f23081622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.814+0000 7f23105e7640 1 -- 172.21.15.59:0/4167113670 >> 172.21.15.59:0/4167113670 conn(0x7f23080b5610 msgr2=0x7f23080b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.814+0000 7f23105e7640 1 -- 172.21.15.59:0/4167113670 shutdown_connections 2024-06-29T14:34:26.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.814+0000 7f23105e7640 1 -- 172.21.15.59:0/4167113670 wait complete. 2024-06-29T14:34:26.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.815+0000 7f23105e7640 1 Processor -- start 2024-06-29T14:34:26.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.816+0000 7f23105e7640 1 -- start start 2024-06-29T14:34:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f23105e7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230808c190 0x7f23080aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f23105e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23080ce950 0x7f2308160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f23105e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 0x7f2308161310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f23105e7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f23080b7fa0 con 0x7f2308162800 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f23105e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23080b7e20 con 0x7f23080ce950 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f23105e7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f23080b8120 con 0x7f230808c190 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 0x7f2308161310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 0x7f2308161310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50888/0 (socket says 172.21.15.59:50888) 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 -- 172.21.15.59:0/2478179418 learned_addr learned my addr 172.21.15.59:0/2478179418 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230e383640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230808c190 0x7f23080aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230db82640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23080ce950 0x7f2308160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 -- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230808c190 msgr2=0x7f23080aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230808c190 0x7f23080aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 -- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23080ce950 msgr2=0x7f2308160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23080ce950 0x7f2308160dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 -- 172.21.15.59:0/2478179418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23080ab440 con 0x7f2308162800 2024-06-29T14:34:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230e383640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230808c190 0x7f23080aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:26.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.817+0000 7f230eb84640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 0x7f2308161310 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2300009cd0 tx=0x7f2300003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.818+0000 7f22feffd640 1 -- 172.21.15.59:0/2478179418 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300031b90 con 0x7f2308162800 2024-06-29T14:34:26.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.818+0000 7f22feffd640 1 -- 172.21.15.59:0/2478179418 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2300031e20 con 0x7f2308162800 2024-06-29T14:34:26.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.818+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23080caf00 con 0x7f2308162800 2024-06-29T14:34:26.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.818+0000 7f22feffd640 1 -- 172.21.15.59:0/2478179418 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300033290 con 0x7f2308162800 2024-06-29T14:34:26.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.818+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23080cb4c0 con 0x7f2308162800 2024-06-29T14:34:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.819+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23080cf9b0 con 0x7f2308162800 2024-06-29T14:34:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.820+0000 7f22feffd640 1 -- 172.21.15.59:0/2478179418 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f2300031660 con 0x7f2308162800 2024-06-29T14:34:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.820+0000 7f22feffd640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f22ec0764d0 0x7f22ec0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.820+0000 7f22feffd640 1 -- 172.21.15.59:0/2478179418 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f23000b9580 con 0x7f2308162800 2024-06-29T14:34:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.820+0000 7f230e383640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f22ec0764d0 0x7f22ec0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.821+0000 7f230e383640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f22ec0764d0 0x7f22ec0789e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f23080b7620 tx=0x7f2304009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:26.825+0000 7f22feffd640 1 -- 172.21.15.59:0/2478179418 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f23080cf9b0 con 0x7f2308162800 2024-06-29T14:34:27.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.036+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f23081622c0 con 0x7f22ec0764d0 2024-06-29T14:34:27.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.045+0000 7f22feffd640 1 -- 172.21.15.59:0/2478179418 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f23081622c0 con 0x7f22ec0764d0 2024-06-29T14:34:27.043 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:27.044 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:36", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f22ec0764d0 msgr2=0x7f22ec0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f22ec0764d0 0x7f22ec0789e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f23080b7620 tx=0x7f2304009040 comp rx=0 tx=0).stop 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 msgr2=0x7f2308161310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 0x7f2308161310 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2300009cd0 tx=0x7f2300003ba0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 shutdown_connections 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f22ec0764d0 0x7f22ec0789e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2308162800 0x7f2308161310 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23080ce950 0x7f2308160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 --2- 172.21.15.59:0/2478179418 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230808c190 0x7f23080aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 >> 172.21.15.59:0/2478179418 conn(0x7f23080b5610 msgr2=0x7f23080cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 shutdown_connections 2024-06-29T14:34:27.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.047+0000 7f23105e7640 1 -- 172.21.15.59:0/2478179418 wait complete. 2024-06-29T14:34:27.057 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:36", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.057 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:36', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:27.057 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:27.057 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:27.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:27.057 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-06-29T14:34:27.074 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:27.074 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_01_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:27.075 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:27.076 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_01_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:27.076 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-06-29T14:34:27.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.237+0000 7f586b718640 1 -- 172.21.15.59:0/183742890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864162770 msgr2=0x7f58640ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.237+0000 7f586b718640 1 --2- 172.21.15.59:0/183742890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864162770 0x7f58640ce550 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f58640b29a0 tx=0x7f586002a4a0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.237+0000 7f586b718640 1 -- 172.21.15.59:0/183742890 shutdown_connections 2024-06-29T14:34:27.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.237+0000 7f586b718640 1 --2- 172.21.15.59:0/183742890 >> v2:172.21.15.153:3300/0 conn(0x7f586415e0a0 0x7f5864161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.237+0000 7f586b718640 1 --2- 172.21.15.59:0/183742890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864162770 0x7f58640ce550 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.237+0000 7f586b718640 1 -- 172.21.15.59:0/183742890 >> 172.21.15.59:0/183742890 conn(0x7f58640d45b0 msgr2=0x7f58640d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.237+0000 7f586b718640 1 -- 172.21.15.59:0/183742890 shutdown_connections 2024-06-29T14:34:27.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.237+0000 7f586b718640 1 -- 172.21.15.59:0/183742890 wait complete. 2024-06-29T14:34:27.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.239+0000 7f586b718640 1 Processor -- start 2024-06-29T14:34:27.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.239+0000 7f586b718640 1 -- start start 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.240+0000 7f586b718640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f586415e0a0 0x7f58640ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.240+0000 7f586b718640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864161fc0 0x7f58641f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.240+0000 7f586b718640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5864162770 0x7f5864152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.240+0000 7f586b718640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f58640d2d80 con 0x7f5864161fc0 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.240+0000 7f586b718640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58640b4b00 con 0x7f5864162770 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.240+0000 7f586b718640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f58640d2f00 con 0x7f586415e0a0 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864161fc0 0x7f58641f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864161fc0 0x7f58641f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50900/0 (socket says 172.21.15.59:50900) 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 -- 172.21.15.59:0/2192628133 learned_addr learned my addr 172.21.15.59:0/2192628133 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f58694b4640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f586415e0a0 0x7f58640ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5869cb5640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5864162770 0x7f5864152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 -- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f586415e0a0 msgr2=0x7f58640ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f586415e0a0 0x7f58640ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 -- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5864162770 msgr2=0x7f5864152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5864162770 0x7f5864152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 -- 172.21.15.59:0/2192628133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5864153100 con 0x7f5864161fc0 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5869cb5640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5864162770 0x7f5864152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5868cb3640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864161fc0 0x7f58641f51b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f585800d770 tx=0x7f585800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5855ffb640 1 -- 172.21.15.59:0/2192628133 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858017070 con 0x7f5864161fc0 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5855ffb640 1 -- 172.21.15.59:0/2192628133 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5858009d20 con 0x7f5864161fc0 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58640c8900 con 0x7f5864161fc0 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f5855ffb640 1 -- 172.21.15.59:0/2192628133 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58580106c0 con 0x7f5864161fc0 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.241+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586415ff50 con 0x7f5864161fc0 2024-06-29T14:34:27.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.242+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58640d3420 con 0x7f5864161fc0 2024-06-29T14:34:27.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.243+0000 7f5855ffb640 1 -- 172.21.15.59:0/2192628133 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f585801a070 con 0x7f5864161fc0 2024-06-29T14:34:27.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.244+0000 7f5855ffb640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f58480764d0 0x7f58480789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.244+0000 7f5855ffb640 1 -- 172.21.15.59:0/2192628133 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f5858097fb0 con 0x7f5864161fc0 2024-06-29T14:34:27.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.244+0000 7f58694b4640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f58480764d0 0x7f58480789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.244+0000 7f58694b4640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f58480764d0 0x7f58480789e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5860034000 tx=0x7f5860004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:27.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.248+0000 7f5855ffb640 1 -- 172.21.15.59:0/2192628133 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f58580612d0 con 0x7f5864161fc0 2024-06-29T14:34:27.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.463+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f58640cdd90 con 0x7f58480764d0 2024-06-29T14:34:27.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.475+0000 7f5855ffb640 1 -- 172.21.15.59:0/2192628133 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f58640cdd90 con 0x7f58480764d0 2024-06-29T14:34:27.473 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:27.473 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:37", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f58480764d0 msgr2=0x7f58480789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f58480764d0 0x7f58480789e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5860034000 tx=0x7f5860004730 comp rx=0 tx=0).stop 2024-06-29T14:34:27.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864161fc0 msgr2=0x7f58641f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864161fc0 0x7f58641f51b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f585800d770 tx=0x7f585800dc30 comp rx=0 tx=0).stop 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 shutdown_connections 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f58480764d0 0x7f58480789e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5864162770 0x7f5864152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5864161fc0 0x7f58641f51b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 --2- 172.21.15.59:0/2192628133 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f586415e0a0 0x7f58640ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 >> 172.21.15.59:0/2192628133 conn(0x7f58640d45b0 msgr2=0x7f5864161790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 shutdown_connections 2024-06-29T14:34:27.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.477+0000 7f586b718640 1 -- 172.21.15.59:0/2192628133 wait complete. 2024-06-29T14:34:27.486 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:37", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.486 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:37', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:27.486 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:27.486 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:27.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:27.487 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-06-29T14:34:27.503 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:27.504 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:27.505 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:27.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-06-29T14:34:27.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.665+0000 7f63cd943640 1 -- 172.21.15.59:0/3872893876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 msgr2=0x7f63c80b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.665+0000 7f63cd943640 1 --2- 172.21.15.59:0/3872893876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 0x7f63c80b38e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f63c80d3710 tx=0x7f63c002a950 comp rx=0 tx=0).stop 2024-06-29T14:34:27.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.665+0000 7f63cd943640 1 -- 172.21.15.59:0/3872893876 shutdown_connections 2024-06-29T14:34:27.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.665+0000 7f63cd943640 1 --2- 172.21.15.59:0/3872893876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 0x7f63c80b38e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.665+0000 7f63cd943640 1 --2- 172.21.15.59:0/3872893876 >> v2:172.21.15.153:3300/0 conn(0x7f63c8162190 0x7f63c80ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.665+0000 7f63cd943640 1 -- 172.21.15.59:0/3872893876 >> 172.21.15.59:0/3872893876 conn(0x7f63c80d45b0 msgr2=0x7f63c80d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.665+0000 7f63cd943640 1 -- 172.21.15.59:0/3872893876 shutdown_connections 2024-06-29T14:34:27.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.665+0000 7f63cd943640 1 -- 172.21.15.59:0/3872893876 wait complete. 2024-06-29T14:34:27.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.667+0000 7f63cd943640 1 Processor -- start 2024-06-29T14:34:27.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.667+0000 7f63cd943640 1 -- start start 2024-06-29T14:34:27.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63cd943640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 0x7f63c80cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63cd943640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63c8162770 0x7f63c81e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63cd943640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c8150e30 0x7f63c81e8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63cd943640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80b4270 con 0x7f63c815e0a0 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63cd943640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80b40f0 con 0x7f63c8150e30 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63cd943640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80b43f0 con 0x7f63c8162770 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63c6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 0x7f63c80cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63c6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 0x7f63c80cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50926/0 (socket says 172.21.15.59:50926) 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63c6ffd640 1 -- 172.21.15.59:0/3105471477 learned_addr learned my addr 172.21.15.59:0/3105471477 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.668+0000 7f63c77fe640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c8150e30 0x7f63c81e8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63c67fc640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63c8162770 0x7f63c81e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63c6ffd640 1 -- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63c8162770 msgr2=0x7f63c81e88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63c6ffd640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63c8162770 0x7f63c81e88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63c6ffd640 1 -- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c8150e30 msgr2=0x7f63c81e8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63c6ffd640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c8150e30 0x7f63c81e8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63c6ffd640 1 -- 172.21.15.59:0/3105471477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c80c73e0 con 0x7f63c815e0a0 2024-06-29T14:34:27.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63c6ffd640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 0x7f63c80cb120 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f63b800b420 tx=0x7f63b800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63bffff640 1 -- 172.21.15.59:0/3105471477 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8011830 con 0x7f63c815e0a0 2024-06-29T14:34:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63bffff640 1 -- 172.21.15.59:0/3105471477 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63b8016070 con 0x7f63c815e0a0 2024-06-29T14:34:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c8151320 con 0x7f63c815e0a0 2024-06-29T14:34:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63bffff640 1 -- 172.21.15.59:0/3105471477 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8011d60 con 0x7f63c815e0a0 2024-06-29T14:34:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.669+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c80c7680 con 0x7f63c815e0a0 2024-06-29T14:34:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.670+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c80b4910 con 0x7f63c815e0a0 2024-06-29T14:34:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.671+0000 7f63bffff640 1 -- 172.21.15.59:0/3105471477 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f63b801b460 con 0x7f63c815e0a0 2024-06-29T14:34:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.671+0000 7f63bffff640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f63a8076500 0x7f63a8078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.672+0000 7f63bffff640 1 -- 172.21.15.59:0/3105471477 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f63b8097970 con 0x7f63c815e0a0 2024-06-29T14:34:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.672+0000 7f63c67fc640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f63a8076500 0x7f63a8078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.672+0000 7f63c67fc640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f63a8076500 0x7f63a8078a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f63c80cdf80 tx=0x7f63b000a250 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:27.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.676+0000 7f63bffff640 1 -- 172.21.15.59:0/3105471477 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f63b8060750 con 0x7f63c815e0a0 2024-06-29T14:34:27.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.888+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63c806c660 con 0x7f63a8076500 2024-06-29T14:34:27.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.897+0000 7f63bffff640 1 -- 172.21.15.59:0/3105471477 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f63c806c660 con 0x7f63a8076500 2024-06-29T14:34:27.896 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:27.896 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:37", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f63a8076500 msgr2=0x7f63a8078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f63a8076500 0x7f63a8078a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f63c80cdf80 tx=0x7f63b000a250 comp rx=0 tx=0).stop 2024-06-29T14:34:27.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 msgr2=0x7f63c80cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 0x7f63c80cb120 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f63b800b420 tx=0x7f63b800b8e0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 shutdown_connections 2024-06-29T14:34:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f63a8076500 0x7f63a8078a10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63c8150e30 0x7f63c81e8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63c8162770 0x7f63c81e88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 --2- 172.21.15.59:0/3105471477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c815e0a0 0x7f63c80cb120 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 >> 172.21.15.59:0/3105471477 conn(0x7f63c80d45b0 msgr2=0x7f63c8152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 shutdown_connections 2024-06-29T14:34:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:27.900+0000 7f63cd943640 1 -- 172.21.15.59:0/3105471477 wait complete. 2024-06-29T14:34:27.909 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:37", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.909 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:37', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:27.909 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:27.909 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:27.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:27.910 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:27.927 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:27.928 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:27.928 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:27.928 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-06-29T14:34:28.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.088+0000 7f2134df0640 1 -- 172.21.15.59:0/2829271524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21300cf5c0 msgr2=0x7f21300d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.088+0000 7f2134df0640 1 --2- 172.21.15.59:0/2829271524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21300cf5c0 0x7f21300d1a20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2124009830 tx=0x7f212402a510 comp rx=0 tx=0).stop 2024-06-29T14:34:28.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.088+0000 7f2134df0640 1 -- 172.21.15.59:0/2829271524 shutdown_connections 2024-06-29T14:34:28.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.088+0000 7f2134df0640 1 --2- 172.21.15.59:0/2829271524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21300cf5c0 0x7f21300d1a20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.088+0000 7f2134df0640 1 --2- 172.21.15.59:0/2829271524 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213008c190 0x7f21300cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.088+0000 7f2134df0640 1 -- 172.21.15.59:0/2829271524 >> 172.21.15.59:0/2829271524 conn(0x7f21300b5610 msgr2=0x7f21300b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:28.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.088+0000 7f2134df0640 1 -- 172.21.15.59:0/2829271524 shutdown_connections 2024-06-29T14:34:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.088+0000 7f2134df0640 1 -- 172.21.15.59:0/2829271524 wait complete. 2024-06-29T14:34:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.090+0000 7f2134df0640 1 Processor -- start 2024-06-29T14:34:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.090+0000 7f2134df0640 1 -- start start 2024-06-29T14:34:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.091+0000 7f2134df0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213008c190 0x7f213015d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.091+0000 7f2134df0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21300ce950 0x7f21300aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.091+0000 7f2134df0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21300cf5c0 0x7f21300ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.091+0000 7f2134df0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f21300b7fa0 con 0x7f21300ce950 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.091+0000 7f2134df0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f21300b7e20 con 0x7f21300cf5c0 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.091+0000 7f2134df0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f21300b8120 con 0x7f213008c190 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21300ce950 0x7f21300aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21300ce950 0x7f21300aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50940/0 (socket says 172.21.15.59:50940) 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 -- 172.21.15.59:0/2590127343 learned_addr learned my addr 172.21.15.59:0/2590127343 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212e59c640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213008c190 0x7f213015d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212ed9d640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21300cf5c0 0x7f21300ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 -- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213008c190 msgr2=0x7f213015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213008c190 0x7f213015d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 -- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21300cf5c0 msgr2=0x7f21300ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21300cf5c0 0x7f21300ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 -- 172.21.15.59:0/2590127343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21300ab1e0 con 0x7f21300ce950 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f212dd9b640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21300ce950 0x7f21300aa4f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2124009950 tx=0x7f2124033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f211effd640 1 -- 172.21.15.59:0/2590127343 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2124031d60 con 0x7f21300ce950 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f211effd640 1 -- 172.21.15.59:0/2590127343 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2124033e10 con 0x7f21300ce950 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f211effd640 1 -- 172.21.15.59:0/2590127343 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2124005260 con 0x7f21300ce950 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.092+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21300ab4c0 con 0x7f21300ce950 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.093+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21300aba80 con 0x7f21300ce950 2024-06-29T14:34:28.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.094+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21300aaf50 con 0x7f21300ce950 2024-06-29T14:34:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.094+0000 7f211effd640 1 -- 172.21.15.59:0/2590127343 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f2124005400 con 0x7f21300ce950 2024-06-29T14:34:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.095+0000 7f211effd640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f21100764d0 0x7f21100789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.095+0000 7f211effd640 1 -- 172.21.15.59:0/2590127343 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f21240b8f00 con 0x7f21300ce950 2024-06-29T14:34:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.095+0000 7f212e59c640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f21100764d0 0x7f21100789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.096+0000 7f212e59c640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f21100764d0 0x7f21100789e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2120005fd0 tx=0x7f2120009260 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:28.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.099+0000 7f211effd640 1 -- 172.21.15.59:0/2590127343 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f212403f030 con 0x7f21300ce950 2024-06-29T14:34:28.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.312+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21300c9b10 con 0x7f21100764d0 2024-06-29T14:34:28.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.323+0000 7f211effd640 1 -- 172.21.15.59:0/2590127343 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f21300c9b10 con 0x7f21100764d0 2024-06-29T14:34:28.321 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:28.321 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:38", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:28.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f21100764d0 msgr2=0x7f21100789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f21100764d0 0x7f21100789e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2120005fd0 tx=0x7f2120009260 comp rx=0 tx=0).stop 2024-06-29T14:34:28.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21300ce950 msgr2=0x7f21300aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21300ce950 0x7f21300aa4f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2124009950 tx=0x7f2124033c20 comp rx=0 tx=0).stop 2024-06-29T14:34:28.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 shutdown_connections 2024-06-29T14:34:28.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f21100764d0 0x7f21100789e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21300cf5c0 0x7f21300ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21300ce950 0x7f21300aa4f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 --2- 172.21.15.59:0/2590127343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f213008c190 0x7f213015d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 >> 172.21.15.59:0/2590127343 conn(0x7f21300b5610 msgr2=0x7f21300c8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:28.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 shutdown_connections 2024-06-29T14:34:28.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.325+0000 7f2134df0640 1 -- 172.21.15.59:0/2590127343 wait complete. 2024-06-29T14:34:28.334 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:38", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:28.334 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:38', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:28.334 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:28.334 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:28.334 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-06-29T14:34:28.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.457+0000 7fa75c37a640 1 -- 172.21.15.59:0/4176761985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c0053a0 msgr2=0x7fa73c00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.457+0000 7fa75c37a640 1 --2- 172.21.15.59:0/4176761985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c0053a0 0x7fa73c00ba20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa744001d90 tx=0x7fa73c005830 comp rx=0 tx=0).stop 2024-06-29T14:34:28.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.457+0000 7fa75c37a640 1 -- 172.21.15.59:0/4176761985 shutdown_connections 2024-06-29T14:34:28.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.457+0000 7fa75c37a640 1 --2- 172.21.15.59:0/4176761985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c0053a0 0x7fa73c00ba20 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.457+0000 7fa75c37a640 1 -- 172.21.15.59:0/4176761985 >> 172.21.15.59:0/4176761985 conn(0x7fa7540b2ab0 msgr2=0x7fa7540b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:28.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.457+0000 7fa75c37a640 1 -- 172.21.15.59:0/4176761985 shutdown_connections 2024-06-29T14:34:28.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.457+0000 7fa75c37a640 1 -- 172.21.15.59:0/4176761985 wait complete. 2024-06-29T14:34:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.459+0000 7fa75c37a640 1 Processor -- start 2024-06-29T14:34:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.459+0000 7fa75c37a640 1 -- start start 2024-06-29T14:34:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75c37a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c0053a0 0x7fa75415b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75c37a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa754142170 0x7fa7541e7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75c37a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7541e7c70 0x7fa75413c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75c37a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa75406dc90 con 0x7fa7541e7c70 2024-06-29T14:34:28.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75c37a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa75415faa0 con 0x7fa73c0053a0 2024-06-29T14:34:28.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75c37a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa75406de10 con 0x7fa754142170 2024-06-29T14:34:28.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7541e7c70 0x7fa75413c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7541e7c70 0x7fa75413c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50964/0 (socket says 172.21.15.59:50964) 2024-06-29T14:34:28.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 -- 172.21.15.59:0/2450441146 learned_addr learned my addr 172.21.15.59:0/2450441146 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:28.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa759915640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa754142170 0x7fa7541e7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a116640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c0053a0 0x7fa75415b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 -- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa754142170 msgr2=0x7fa7541e7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa754142170 0x7fa7541e7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 -- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c0053a0 msgr2=0x7fa75415b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c0053a0 0x7fa75415b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 -- 172.21.15.59:0/2450441146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7541e8080 con 0x7fa7541e7c70 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75a917640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7541e7c70 0x7fa75413c230 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa744001eb0 tx=0x7fa744006d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa752ffd640 1 -- 172.21.15.59:0/2450441146 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744003d80 con 0x7fa7541e7c70 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa752ffd640 1 -- 172.21.15.59:0/2450441146 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa744005ba0 con 0x7fa7541e7c70 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75413a890 con 0x7fa7541e7c70 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.461+0000 7fa752ffd640 1 -- 172.21.15.59:0/2450441146 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744005ff0 con 0x7fa7541e7c70 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.462+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75413ae50 con 0x7fa7541e7c70 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.462+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa75406e320 con 0x7fa7541e7c70 2024-06-29T14:34:28.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.463+0000 7fa752ffd640 1 -- 172.21.15.59:0/2450441146 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fa744005030 con 0x7fa7541e7c70 2024-06-29T14:34:28.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.464+0000 7fa752ffd640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fa7400764d0 0x7fa7400789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.464+0000 7fa752ffd640 1 -- 172.21.15.59:0/2450441146 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fa7440b7f90 con 0x7fa7541e7c70 2024-06-29T14:34:28.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.464+0000 7fa75a116640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fa7400764d0 0x7fa7400789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.465+0000 7fa75a116640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fa7400764d0 0x7fa7400789e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009b10 tx=0x7fa74c005f90 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:28.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.468+0000 7fa752ffd640 1 -- 172.21.15.59:0/2450441146 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa744043060 con 0x7fa7541e7c70 2024-06-29T14:34:28.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.679+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fa7540cf7e0 con 0x7fa7400764d0 2024-06-29T14:34:28.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.775+0000 7fa752ffd640 1 -- 172.21.15.59:0/2450441146 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7fa7540cf7e0 con 0x7fa7400764d0 2024-06-29T14:34:28.774 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-06-29T14:34:28.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.777+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fa7400764d0 msgr2=0x7fa7400789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.777+0000 7fa75c37a640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fa7400764d0 0x7fa7400789e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009b10 tx=0x7fa74c005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:28.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.777+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7541e7c70 msgr2=0x7fa75413c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.777+0000 7fa75c37a640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7541e7c70 0x7fa75413c230 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa744001eb0 tx=0x7fa744006d00 comp rx=0 tx=0).stop 2024-06-29T14:34:28.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.778+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 shutdown_connections 2024-06-29T14:34:28.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.778+0000 7fa75c37a640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fa7400764d0 0x7fa7400789e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.778+0000 7fa75c37a640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7541e7c70 0x7fa75413c230 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.778+0000 7fa75c37a640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa754142170 0x7fa7541e7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.778+0000 7fa75c37a640 1 --2- 172.21.15.59:0/2450441146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c0053a0 0x7fa75415b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.778+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 >> 172.21.15.59:0/2450441146 conn(0x7fa7540b2ab0 msgr2=0x7fa75415e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:28.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.778+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 shutdown_connections 2024-06-29T14:34:28.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:28.778+0000 7fa75c37a640 1 -- 172.21.15.59:0/2450441146 wait complete. 2024-06-29T14:34:28.787 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-06-29T14:34:28.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.787 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-06-29T14:34:28.805 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:28.805 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:28.805 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:28.805 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:28.806 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:28.806 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:28.806 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:28.806 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:28.806 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:28.806 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:28.807 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:28.807 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:28.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.807 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:28.867 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:28.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.867 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:28.929 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:28.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.929 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:28.991 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:28.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.991 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:29.053 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:29.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.054 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:29.122 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:29.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.122 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:29.185 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:29.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.186 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:29.251 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:29.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.252 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:29.313 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:29.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.314 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:29.376 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:29.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.376 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:29.438 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:29.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.438 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:29.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.503 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-06-29T14:34:29.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-06-29T14:34:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.724+0000 7f1a53754640 1 -- 172.21.15.59:0/2414577950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 msgr2=0x7f1a4c0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.724+0000 7f1a53754640 1 --2- 172.21.15.59:0/2414577950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 0x7f1a4c0b3990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1a48005fd0 tx=0x7f1a4802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.724+0000 7f1a53754640 1 -- 172.21.15.59:0/2414577950 shutdown_connections 2024-06-29T14:34:29.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.724+0000 7f1a53754640 1 --2- 172.21.15.59:0/2414577950 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a4c0b3ed0 0x7f1a4c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.724+0000 7f1a53754640 1 --2- 172.21.15.59:0/2414577950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 0x7f1a4c0b3990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.724+0000 7f1a53754640 1 -- 172.21.15.59:0/2414577950 >> 172.21.15.59:0/2414577950 conn(0x7f1a4c0d3ee0 msgr2=0x7f1a4c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:29.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.725+0000 7f1a53754640 1 -- 172.21.15.59:0/2414577950 shutdown_connections 2024-06-29T14:34:29.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.725+0000 7f1a53754640 1 -- 172.21.15.59:0/2414577950 wait complete. 2024-06-29T14:34:29.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.726+0000 7f1a53754640 1 Processor -- start 2024-06-29T14:34:29.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.726+0000 7f1a53754640 1 -- start start 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a53754640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a4c0b3ed0 0x7f1a4c1f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a53754640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 0x7f1a4c0cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a53754640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a4c15ee10 0x7f1a4c1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a53754640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c160240 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a53754640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c1600c0 con 0x7f1a4c0b3ed0 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a53754640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4c1603c0 con 0x7f1a4c15ee10 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 0x7f1a4c0cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 0x7f1a4c0cc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50978/0 (socket says 172.21.15.59:50978) 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 -- 172.21.15.59:0/2783779449 learned_addr learned my addr 172.21.15.59:0/2783779449 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a51cf1640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a4c15ee10 0x7f1a4c1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a514f0640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a4c0b3ed0 0x7f1a4c1f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:29.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 -- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a4c15ee10 msgr2=0x7f1a4c1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a4c15ee10 0x7f1a4c1521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 -- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a4c0b3ed0 msgr2=0x7f1a4c1f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a4c0b3ed0 0x7f1a4c1f53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 -- 172.21.15.59:0/2783779449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c0cc8e0 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a51cf1640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a4c15ee10 0x7f1a4c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a50cef640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 0x7f1a4c0cc3a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00d6d0 tx=0x7f1a3c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a41ffb640 1 -- 172.21.15.59:0/2783779449 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c017070 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a41ffb640 1 -- 172.21.15.59:0/2783779449 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a3c005330 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4c151110 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.728+0000 7f1a41ffb640 1 -- 172.21.15.59:0/2783779449 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c009820 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.729+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a4c1516d0 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.730+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a4c15e8d0 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.731+0000 7f1a41ffb640 1 -- 172.21.15.59:0/2783779449 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f1a3c0099c0 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.731+0000 7f1a41ffb640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f1a300764d0 0x7f1a300789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.731+0000 7f1a41ffb640 1 -- 172.21.15.59:0/2783779449 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f1a3c097f70 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.731+0000 7f1a514f0640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f1a300764d0 0x7f1a300789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:29.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.732+0000 7f1a514f0640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f1a300764d0 0x7f1a300789e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1a48033000 tx=0x7f1a48004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:29.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.736+0000 7f1a41ffb640 1 -- 172.21.15.59:0/2783779449 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a3c061200 con 0x7f1a4c0d23c0 2024-06-29T14:34:29.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:29.950+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1a4c14f070 con 0x7f1a300764d0 2024-06-29T14:34:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.040+0000 7f1a41ffb640 1 -- 172.21.15.59:0/2783779449 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f1a4c14f070 con 0x7f1a300764d0 2024-06-29T14:34:30.038 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-06-29T14:34:30.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f1a300764d0 msgr2=0x7f1a300789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f1a300764d0 0x7f1a300789e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1a48033000 tx=0x7f1a48004730 comp rx=0 tx=0).stop 2024-06-29T14:34:30.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 msgr2=0x7f1a4c0cc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 0x7f1a4c0cc3a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00d6d0 tx=0x7f1a3c00db90 comp rx=0 tx=0).stop 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 shutdown_connections 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f1a300764d0 0x7f1a300789e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a4c15ee10 0x7f1a4c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a4c0d23c0 0x7f1a4c0cc3a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 --2- 172.21.15.59:0/2783779449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a4c0b3ed0 0x7f1a4c1f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 >> 172.21.15.59:0/2783779449 conn(0x7f1a4c0d3ee0 msgr2=0x7f1a4c06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 shutdown_connections 2024-06-29T14:34:30.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.042+0000 7f1a53754640 1 -- 172.21.15.59:0/2783779449 wait complete. 2024-06-29T14:34:30.051 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-06-29T14:34:30.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.052 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-06-29T14:34:30.069 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:30.069 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:30.069 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:30.070 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:30.070 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:30.070 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:30.070 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:30.070 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:30.070 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:30.071 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:30.071 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:30.071 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:30.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.072 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:30.133 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:30.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.134 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:30.195 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:30.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.196 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:30.259 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:30.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.259 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:30.322 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:30.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.322 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:30.384 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:30.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.384 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:30.447 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:30.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.447 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:30.509 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:30.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.509 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:30.571 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:30.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.571 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:30.633 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:30.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.633 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:30.696 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:30.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.696 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:30.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.758 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-06-29T14:34:30.818 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-06-29T14:34:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.980+0000 7f313a697640 1 -- 172.21.15.59:0/579381936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340d23c0 msgr2=0x7f31340b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.980+0000 7f313a697640 1 --2- 172.21.15.59:0/579381936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340d23c0 0x7f31340b3990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3124005fd0 tx=0x7f312402a530 comp rx=0 tx=0).stop 2024-06-29T14:34:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.980+0000 7f313a697640 1 -- 172.21.15.59:0/579381936 shutdown_connections 2024-06-29T14:34:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.980+0000 7f313a697640 1 --2- 172.21.15.59:0/579381936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f313415efe0 0x7f313415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.980+0000 7f313a697640 1 --2- 172.21.15.59:0/579381936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340d23c0 0x7f31340b3990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.980+0000 7f313a697640 1 -- 172.21.15.59:0/579381936 >> 172.21.15.59:0/579381936 conn(0x7f31340d3ee0 msgr2=0x7f31340d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:30.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.980+0000 7f313a697640 1 -- 172.21.15.59:0/579381936 shutdown_connections 2024-06-29T14:34:30.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.980+0000 7f313a697640 1 -- 172.21.15.59:0/579381936 wait complete. 2024-06-29T14:34:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.982+0000 7f313a697640 1 Processor -- start 2024-06-29T14:34:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.982+0000 7f313a697640 1 -- start start 2024-06-29T14:34:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f313a697640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340b3ed0 0x7f3134146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f313a697640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31340d23c0 0x7f3134143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f313a697640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f313415efe0 0x7f3134143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f313a697640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3134160280 con 0x7f31340b3ed0 2024-06-29T14:34:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f313a697640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3134160100 con 0x7f313415efe0 2024-06-29T14:34:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f313a697640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3134160400 con 0x7f31340d23c0 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340b3ed0 0x7f3134146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340b3ed0 0x7f3134146560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50992/0 (socket says 172.21.15.59:50992) 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 -- 172.21.15.59:0/1438077031 learned_addr learned my addr 172.21.15.59:0/1438077031 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 -- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31340d23c0 msgr2=0x7f3134143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31340d23c0 0x7f3134143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 -- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f313415efe0 msgr2=0x7f3134143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f313415efe0 0x7f3134143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 -- 172.21.15.59:0/1438077031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3134143ff0 con 0x7f31340b3ed0 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3133fff640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340b3ed0 0x7f3134146560 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3124006010 tx=0x7f3124003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3130ff9640 1 -- 172.21.15.59:0/1438077031 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312403e030 con 0x7f31340b3ed0 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3130ff9640 1 -- 172.21.15.59:0/1438077031 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3124003e00 con 0x7f31340b3ed0 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3134144540 con 0x7f31340b3ed0 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.984+0000 7f3130ff9640 1 -- 172.21.15.59:0/1438077031 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124035040 con 0x7f31340b3ed0 2024-06-29T14:34:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.985+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3134144b00 con 0x7f31340b3ed0 2024-06-29T14:34:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.986+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f313406d620 con 0x7f31340b3ed0 2024-06-29T14:34:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.986+0000 7f3130ff9640 1 -- 172.21.15.59:0/1438077031 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f3124041030 con 0x7f31340b3ed0 2024-06-29T14:34:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.987+0000 7f3130ff9640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3120076500 0x7f3120078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.987+0000 7f3130ff9640 1 -- 172.21.15.59:0/1438077031 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7f31240b8750 con 0x7f31340b3ed0 2024-06-29T14:34:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.987+0000 7f31337fe640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3120076500 0x7f3120078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.988+0000 7f31337fe640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3120076500 0x7f3120078a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3128006fd0 tx=0x7f3128006bc0 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:30.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:30.992+0000 7f3130ff9640 1 -- 172.21.15.59:0/1438077031 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f313406d620 con 0x7f31340b3ed0 2024-06-29T14:34:31.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.209+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f31341442f0 con 0x7f3120076500 2024-06-29T14:34:31.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.310+0000 7f3130ff9640 1 -- 172.21.15.59:0/1438077031 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f31341442f0 con 0x7f3120076500 2024-06-29T14:34:31.308 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3120076500 msgr2=0x7f3120078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3120076500 0x7f3120078a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3128006fd0 tx=0x7f3128006bc0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340b3ed0 msgr2=0x7f3134146560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340b3ed0 0x7f3134146560 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3124006010 tx=0x7f3124003910 comp rx=0 tx=0).stop 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 shutdown_connections 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f3120076500 0x7f3120078a10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f313415efe0 0x7f3134143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31340d23c0 0x7f3134143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 --2- 172.21.15.59:0/1438077031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31340b3ed0 0x7f3134146560 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 >> 172.21.15.59:0/1438077031 conn(0x7f31340d3ee0 msgr2=0x7f313406c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:31.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 shutdown_connections 2024-06-29T14:34:31.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:31.312+0000 7f313a697640 1 -- 172.21.15.59:0/1438077031 wait complete. 2024-06-29T14:34:31.322 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-06-29T14:34:31.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.322 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-06-29T14:34:31.340 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:31.340 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:31.340 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:31.340 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:31.340 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:31.340 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_01_UTC 2024-06-29T14:34:31.340 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:31.340 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:31.341 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:31.341 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:31.341 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:31.341 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:31.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.341 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:31.404 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:31.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.404 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:31.467 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:31.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.467 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:31.531 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:31.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.532 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:31.595 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:31.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.595 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:31.660 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_29_01_UTC 2024-06-29T14:34:31.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.660 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_29_01_UTC) 2024-06-29T14:34:31.722 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:31.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.722 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:31.787 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:31.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.787 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:31.849 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:31.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.850 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:31.916 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:31.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.916 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:31.981 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:31.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.982 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:32.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.045 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-06-29T14:34:32.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-06-29T14:34:32.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.269+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/1180836048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3b015b150 msgr2=0x7fb3b006da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:32.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.269+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/1180836048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3b015b150 0x7fb3b006da80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009830 tx=0x7fb3a802a480 comp rx=0 tx=0).stop 2024-06-29T14:34:32.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.269+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/1180836048 shutdown_connections 2024-06-29T14:34:32.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.269+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/1180836048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3b015f2c0 0x7fb3b00b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.269+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/1180836048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3b015b150 0x7fb3b006da80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.269+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/1180836048 >> 172.21.15.59:0/1180836048 conn(0x7fb3b0070100 msgr2=0x7fb3b0070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:32.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.269+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/1180836048 shutdown_connections 2024-06-29T14:34:32.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.269+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/1180836048 wait complete. 2024-06-29T14:34:32.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.271+0000 7fb3b4fbd640 1 Processor -- start 2024-06-29T14:34:32.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.271+0000 7fb3b4fbd640 1 -- start start 2024-06-29T14:34:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3b4fbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3b006dfc0 0x7fb3b01522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3b4fbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3b015b150 0x7fb3b0140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3b4fbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b015f2c0 0x7fb3b0157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3b4fbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b00b5af0 con 0x7fb3b015f2c0 2024-06-29T14:34:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3b4fbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b015b560 con 0x7fb3b006dfc0 2024-06-29T14:34:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3b4fbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b00b5c70 con 0x7fb3b015b150 2024-06-29T14:34:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3aed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b015f2c0 0x7fb3b0157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3aed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b015f2c0 0x7fb3b0157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51004/0 (socket says 172.21.15.59:51004) 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3aed9d640 1 -- 172.21.15.59:0/137716356 learned_addr learned my addr 172.21.15.59:0/137716356 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3ae59c640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3b006dfc0 0x7fb3b01522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.272+0000 7fb3add9b640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3b015b150 0x7fb3b0140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.273+0000 7fb3aed9d640 1 -- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3b015b150 msgr2=0x7fb3b0140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.273+0000 7fb3aed9d640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3b015b150 0x7fb3b0140af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.273+0000 7fb3aed9d640 1 -- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3b006dfc0 msgr2=0x7fb3b01522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.273+0000 7fb3aed9d640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3b006dfc0 0x7fb3b01522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.273+0000 7fb3aed9d640 1 -- 172.21.15.59:0/137716356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b0141030 con 0x7fb3b015f2c0 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.273+0000 7fb3aed9d640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b015f2c0 0x7fb3b0157990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009fd0 tx=0x7fb3a000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.273+0000 7fb3a6ffd640 1 -- 172.21.15.59:0/137716356 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0003970 con 0x7fb3b015f2c0 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.274+0000 7fb3a6ffd640 1 -- 172.21.15.59:0/137716356 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3a0003c20 con 0x7fb3b015f2c0 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.274+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b0157ed0 con 0x7fb3b015f2c0 2024-06-29T14:34:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.274+0000 7fb3a6ffd640 1 -- 172.21.15.59:0/137716356 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0004280 con 0x7fb3b015f2c0 2024-06-29T14:34:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.274+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b01f3890 con 0x7fb3b015f2c0 2024-06-29T14:34:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.275+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b00ce390 con 0x7fb3b015f2c0 2024-06-29T14:34:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.276+0000 7fb3a6ffd640 1 -- 172.21.15.59:0/137716356 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fb3a0004420 con 0x7fb3b015f2c0 2024-06-29T14:34:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.276+0000 7fb3a6ffd640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb390076500 0x7fb390078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.276+0000 7fb3a6ffd640 1 -- 172.21.15.59:0/137716356 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6388+0+0 (secure 0 0 0) 0x7fb3a00985c0 con 0x7fb3b015f2c0 2024-06-29T14:34:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.277+0000 7fb3ae59c640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb390076500 0x7fb390078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.277+0000 7fb3ae59c640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb390076500 0x7fb390078a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8034000 tx=0x7fb3a8004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:32.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.281+0000 7fb3a6ffd640 1 -- 172.21.15.59:0/137716356 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb3b00ce390 con 0x7fb3b015f2c0 2024-06-29T14:34:32.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.673+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb3b00b3b70 con 0x7fb390076500 2024-06-29T14:34:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.766+0000 7fb3a6ffd640 1 -- 172.21.15.59:0/137716356 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7fb3b00b3b70 con 0x7fb390076500 2024-06-29T14:34:32.765 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-06-29T14:34:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.768+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb390076500 msgr2=0x7fb390078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.768+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb390076500 0x7fb390078a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8034000 tx=0x7fb3a8004730 comp rx=0 tx=0).stop 2024-06-29T14:34:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b015f2c0 msgr2=0x7fb3b0157990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b015f2c0 0x7fb3b0157990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009fd0 tx=0x7fb3a000c740 comp rx=0 tx=0).stop 2024-06-29T14:34:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 shutdown_connections 2024-06-29T14:34:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb390076500 0x7fb390078a10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b015f2c0 0x7fb3b0157990 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3b015b150 0x7fb3b0140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 --2- 172.21.15.59:0/137716356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3b006dfc0 0x7fb3b01522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 >> 172.21.15.59:0/137716356 conn(0x7fb3b0070100 msgr2=0x7fb3b006ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 shutdown_connections 2024-06-29T14:34:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:32.769+0000 7fb3b4fbd640 1 -- 172.21.15.59:0/137716356 wait complete. 2024-06-29T14:34:32.778 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-06-29T14:34:32.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.778 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-06-29T14:34:32.795 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:32.797 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:32.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.797 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:32.858 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:32.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.859 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:32.921 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:32.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.921 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:32.983 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:32.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.983 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:33.045 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:33.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.045 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:33.108 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:33.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.109 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:33.171 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:33.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.171 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:33.234 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:33.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.234 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:33.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:33.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.307 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:33.370 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:33.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.370 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:33.432 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:33.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.432 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:33.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.494 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-06-29T14:34:33.554 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-06-29T14:34:33.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.717+0000 7f86d243c640 1 -- 172.21.15.59:0/448106275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b4005300 msgr2=0x7f86b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:33.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.717+0000 7f86d243c640 1 --2- 172.21.15.59:0/448106275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b4005300 0x7f86b400b9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0c75c0 tx=0x7f86b4005830 comp rx=0 tx=0).stop 2024-06-29T14:34:33.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.717+0000 7f86d243c640 1 -- 172.21.15.59:0/448106275 shutdown_connections 2024-06-29T14:34:33.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.717+0000 7f86d243c640 1 --2- 172.21.15.59:0/448106275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86b4005300 0x7f86b400b9c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:33.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.717+0000 7f86d243c640 1 -- 172.21.15.59:0/448106275 >> 172.21.15.59:0/448106275 conn(0x7f86cc0d3ee0 msgr2=0x7f86cc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:33.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.717+0000 7f86d243c640 1 -- 172.21.15.59:0/448106275 shutdown_connections 2024-06-29T14:34:33.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.717+0000 7f86d243c640 1 -- 172.21.15.59:0/448106275 wait complete. 2024-06-29T14:34:33.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.719+0000 7f86d243c640 1 Processor -- start 2024-06-29T14:34:33.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.719+0000 7f86d243c640 1 -- start start 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.720+0000 7f86d243c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b4005300 0x7f86cc1f8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.720+0000 7f86d243c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86cc0c6340 0x7f86cc1e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.720+0000 7f86d243c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86cc1e6f90 0x7f86cc154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.720+0000 7f86d243c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc0b3d20 con 0x7f86cc0c6340 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.720+0000 7f86d243c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc0b3ba0 con 0x7f86b4005300 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.720+0000 7f86d243c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f86cc15e8d0 con 0x7f86cc1e6f90 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.720+0000 7f86cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86cc0c6340 0x7f86cc1e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86cc0c6340 0x7f86cc1e6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55390/0 (socket says 172.21.15.59:55390) 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cb7fe640 1 -- 172.21.15.59:0/2168430014 learned_addr learned my addr 172.21.15.59:0/2168430014 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:33.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cbfff640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b4005300 0x7f86cc1f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86d09d9640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86cc1e6f90 0x7f86cc154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cb7fe640 1 -- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86cc1e6f90 msgr2=0x7f86cc154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cb7fe640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86cc1e6f90 0x7f86cc154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cb7fe640 1 -- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b4005300 msgr2=0x7f86cc1f8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cb7fe640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b4005300 0x7f86cc1f8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cb7fe640 1 -- 172.21.15.59:0/2168430014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86cc1f6f80 con 0x7f86cc0c6340 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86cb7fe640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86cc0c6340 0x7f86cc1e6a50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f86c0003d80 tx=0x7f86c0001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86c8ff9640 1 -- 172.21.15.59:0/2168430014 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c0011640 con 0x7f86cc0c6340 2024-06-29T14:34:33.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86c8ff9640 1 -- 172.21.15.59:0/2168430014 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86c0016020 con 0x7f86cc0c6340 2024-06-29T14:34:33.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86c8ff9640 1 -- 172.21.15.59:0/2168430014 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c0011a60 con 0x7f86cc0c6340 2024-06-29T14:34:33.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86cc1f7260 con 0x7f86cc0c6340 2024-06-29T14:34:33.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.721+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86cc1f7820 con 0x7f86cc0c6340 2024-06-29T14:34:33.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.722+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86cc160170 con 0x7f86cc0c6340 2024-06-29T14:34:33.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.723+0000 7f86c8ff9640 1 -- 172.21.15.59:0/2168430014 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f86c0011c00 con 0x7f86cc0c6340 2024-06-29T14:34:33.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.723+0000 7f86c8ff9640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f86b80764d0 0x7f86b80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:33.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.724+0000 7f86c8ff9640 1 -- 172.21.15.59:0/2168430014 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6940+0+0 (secure 0 0 0) 0x7f86c0098c90 con 0x7f86cc0c6340 2024-06-29T14:34:33.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.724+0000 7f86cbfff640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f86b80764d0 0x7f86b80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:33.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.724+0000 7f86cbfff640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f86b80764d0 0x7f86b80789e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0d2e40 tx=0x7f86bc005f90 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:33.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.728+0000 7f86c8ff9640 1 -- 172.21.15.59:0/2168430014 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f86c0020020 con 0x7f86cc0c6340 2024-06-29T14:34:33.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:33.946+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f86cc0d2950 con 0x7f86b80764d0 2024-06-29T14:34:34.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.030+0000 7f86c8ff9640 1 -- 172.21.15.59:0/2168430014 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f86cc0d2950 con 0x7f86b80764d0 2024-06-29T14:34:34.029 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-06-29T14:34:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f86b80764d0 msgr2=0x7f86b80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f86b80764d0 0x7f86b80789e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0d2e40 tx=0x7f86bc005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86cc0c6340 msgr2=0x7f86cc1e6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86cc0c6340 0x7f86cc1e6a50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f86c0003d80 tx=0x7f86c0001800 comp rx=0 tx=0).stop 2024-06-29T14:34:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 shutdown_connections 2024-06-29T14:34:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f86b80764d0 0x7f86b80789e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f86cc1e6f90 0x7f86cc154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86cc0c6340 0x7f86cc1e6a50 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 --2- 172.21.15.59:0/2168430014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86b4005300 0x7f86cc1f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 >> 172.21.15.59:0/2168430014 conn(0x7f86cc0d3ee0 msgr2=0x7f86cc0d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:34.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 shutdown_connections 2024-06-29T14:34:34.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:34.033+0000 7f86d243c640 1 -- 172.21.15.59:0/2168430014 wait complete. 2024-06-29T14:34:34.042 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-06-29T14:34:34.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.042 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-06-29T14:34:34.059 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:34.059 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:34.059 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:34.059 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:34.059 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:34.059 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:34.060 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:34.060 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:34.060 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:34.060 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:34.060 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:34.060 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:34.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.060 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:34.125 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:34.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.125 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:34.188 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:34.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.188 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:34.250 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:34.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.251 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:34.336 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:34.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.336 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:34.399 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:34.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.400 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:34.464 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:34.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.465 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:34.527 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:34.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.527 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:34.593 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:34.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.593 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:34.658 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:34.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.658 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:34.723 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:34.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.723 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:34.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.787 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-06-29T14:34:34.846 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-06-29T14:34:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 -- 172.21.15.59:0/1160145034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f804815c2f0 msgr2=0x7f804815c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 --2- 172.21.15.59:0/1160145034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f804815c2f0 0x7f804815c700 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8048074b10 tx=0x7f804002aa30 comp rx=0 tx=0).stop 2024-06-29T14:34:35.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 -- 172.21.15.59:0/1160145034 shutdown_connections 2024-06-29T14:34:35.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 --2- 172.21.15.59:0/1160145034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f804815c2f0 0x7f804815c700 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 --2- 172.21.15.59:0/1160145034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f80480b3740 0x7f80480b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 --2- 172.21.15.59:0/1160145034 >> v2:172.21.15.153:3300/0 conn(0x7f80481458a0 0x7f80480cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 -- 172.21.15.59:0/1160145034 >> 172.21.15.59:0/1160145034 conn(0x7f804806c950 msgr2=0x7f8048073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:35.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 -- 172.21.15.59:0/1160145034 shutdown_connections 2024-06-29T14:34:35.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.008+0000 7f804e7d9640 1 -- 172.21.15.59:0/1160145034 wait complete. 2024-06-29T14:34:35.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.010+0000 7f804e7d9640 1 Processor -- start 2024-06-29T14:34:35.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.010+0000 7f804e7d9640 1 -- start start 2024-06-29T14:34:35.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f804e7d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80480b3740 0x7f8048152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:35.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f804e7d9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804815c2f0 0x7f8048136b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:35.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f804e7d9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8048143f70 0x7f8048144380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:35.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f804e7d9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f804815fe40 con 0x7f80480b3740 2024-06-29T14:34:35.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f804e7d9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f804815fcc0 con 0x7f804815c2f0 2024-06-29T14:34:35.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f804e7d9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f804815ffc0 con 0x7f8048143f70 2024-06-29T14:34:35.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f8047fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80480b3740 0x7f8048152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:35.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f8047fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80480b3740 0x7f8048152d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55406/0 (socket says 172.21.15.59:55406) 2024-06-29T14:34:35.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.011+0000 7f8047fff640 1 -- 172.21.15.59:0/2498222322 learned_addr learned my addr 172.21.15.59:0/2498222322 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:35.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f80477fe640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804815c2f0 0x7f8048136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:35.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f804cd76640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8048143f70 0x7f8048144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:35.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8047fff640 1 -- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8048143f70 msgr2=0x7f8048144380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8047fff640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8048143f70 0x7f8048144380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8047fff640 1 -- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804815c2f0 msgr2=0x7f8048136b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8047fff640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804815c2f0 0x7f8048136b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8047fff640 1 -- 172.21.15.59:0/2498222322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80481448c0 con 0x7f80480b3740 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8047fff640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80480b3740 0x7f8048152d60 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f803800c9a0 tx=0x7f803800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8044ff9640 1 -- 172.21.15.59:0/2498222322 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038007bb0 con 0x7f80480b3740 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8044ff9640 1 -- 172.21.15.59:0/2498222322 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8038007e40 con 0x7f80480b3740 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8048155900 con 0x7f80480b3740 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f8044ff9640 1 -- 172.21.15.59:0/2498222322 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803800f690 con 0x7f80480b3740 2024-06-29T14:34:35.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.012+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8048155ec0 con 0x7f80480b3740 2024-06-29T14:34:35.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.013+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80481458a0 con 0x7f80480b3740 2024-06-29T14:34:35.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.014+0000 7f8044ff9640 1 -- 172.21.15.59:0/2498222322 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f803800f830 con 0x7f80480b3740 2024-06-29T14:34:35.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.015+0000 7f8044ff9640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f80340764d0 0x7f80340789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:35.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.015+0000 7f8044ff9640 1 -- 172.21.15.59:0/2498222322 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 7116+0+0 (secure 0 0 0) 0x7f8038098820 con 0x7f80480b3740 2024-06-29T14:34:35.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.015+0000 7f80477fe640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f80340764d0 0x7f80340789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:35.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.016+0000 7f80477fe640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f80340764d0 0x7f80340789e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8048133ec0 tx=0x7f803c005ef0 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:35.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.019+0000 7f8044ff9640 1 -- 172.21.15.59:0/2498222322 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f80481448c0 con 0x7f80480b3740 2024-06-29T14:34:35.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.239+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f80480b1f10 con 0x7f80340764d0 2024-06-29T14:34:35.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.380+0000 7f8044ff9640 1 -- 172.21.15.59:0/2498222322 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f80480b1f10 con 0x7f80340764d0 2024-06-29T14:34:35.379 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-06-29T14:34:35.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f80340764d0 msgr2=0x7f80340789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f80340764d0 0x7f80340789e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8048133ec0 tx=0x7f803c005ef0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80480b3740 msgr2=0x7f8048152d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80480b3740 0x7f8048152d60 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f803800c9a0 tx=0x7f803800ce60 comp rx=0 tx=0).stop 2024-06-29T14:34:35.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 shutdown_connections 2024-06-29T14:34:35.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f80340764d0 0x7f80340789e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8048143f70 0x7f8048144380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f804815c2f0 0x7f8048136b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 --2- 172.21.15.59:0/2498222322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80480b3740 0x7f8048152d60 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 >> 172.21.15.59:0/2498222322 conn(0x7f804806c950 msgr2=0x7f8048071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:35.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 shutdown_connections 2024-06-29T14:34:35.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:35.383+0000 7f804e7d9640 1 -- 172.21.15.59:0/2498222322 wait complete. 2024-06-29T14:34:35.392 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-06-29T14:34:35.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.392 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-06-29T14:34:35.409 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:35.409 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:35.409 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:35.409 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:35.409 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:35.410 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:35.410 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:35.410 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:35.410 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:35.410 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:35.410 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:35.410 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:35.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.410 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:35.478 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:35.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.478 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:35.540 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:35.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.540 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:35.604 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:35.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.605 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:35.668 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:35.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.668 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:35.730 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:35.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.730 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:35.792 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:35.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.792 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:35.854 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:35.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.854 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:35.916 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:35.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.917 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:35.979 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:35.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.980 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:36.045 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:36.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.045 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:36.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.108 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-06-29T14:34:36.168 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-06-29T14:34:36.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.358+0000 7f47cfbb2640 1 -- 172.21.15.59:0/3316605789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80b2fa0 msgr2=0x7f47c80d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.358+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/3316605789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80b2fa0 0x7f47c80d1520 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f47c80b2680 tx=0x7f47bc02a510 comp rx=0 tx=0).stop 2024-06-29T14:34:36.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.358+0000 7f47cfbb2640 1 -- 172.21.15.59:0/3316605789 shutdown_connections 2024-06-29T14:34:36.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.358+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/3316605789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c80d1a60 0x7f47c80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.358+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/3316605789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80b2fa0 0x7f47c80d1520 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.358+0000 7f47cfbb2640 1 -- 172.21.15.59:0/3316605789 >> 172.21.15.59:0/3316605789 conn(0x7f47c8073e10 msgr2=0x7f47c8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:36.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.359+0000 7f47cfbb2640 1 -- 172.21.15.59:0/3316605789 shutdown_connections 2024-06-29T14:34:36.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.359+0000 7f47cfbb2640 1 -- 172.21.15.59:0/3316605789 wait complete. 2024-06-29T14:34:36.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.360+0000 7f47cfbb2640 1 Processor -- start 2024-06-29T14:34:36.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.360+0000 7f47cfbb2640 1 -- start start 2024-06-29T14:34:36.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47cfbb2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c80b2270 0x7f47c815ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:36.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47cfbb2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c80b2fa0 0x7f47c80a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:36.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47cfbb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80d1a60 0x7f47c815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:36.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47cfbb2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f47c80b6400 con 0x7f47c80d1a60 2024-06-29T14:34:36.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47cfbb2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f47c80b6280 con 0x7f47c80b2fa0 2024-06-29T14:34:36.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47cfbb2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f47c80b6580 con 0x7f47c80b2270 2024-06-29T14:34:36.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80d1a60 0x7f47c815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80d1a60 0x7f47c815a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55430/0 (socket says 172.21.15.59:55430) 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 -- 172.21.15.59:0/1716331761 learned_addr learned my addr 172.21.15.59:0/1716331761 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47cd14d640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c80b2fa0 0x7f47c80a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47cd94e640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c80b2270 0x7f47c815ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 -- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c80b2270 msgr2=0x7f47c815ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c80b2270 0x7f47c815ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 -- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c80b2fa0 msgr2=0x7f47c80a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c80b2fa0 0x7f47c80a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 -- 172.21.15.59:0/1716331761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c81f3820 con 0x7f47c80d1a60 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.362+0000 7f47ce14f640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80d1a60 0x7f47c815a2f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f47c000d5f0 tx=0x7f47c000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.363+0000 7f47ba7fc640 1 -- 172.21.15.59:0/1716331761 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0017070 con 0x7f47c80d1a60 2024-06-29T14:34:36.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.363+0000 7f47ba7fc640 1 -- 172.21.15.59:0/1716331761 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47c0005330 con 0x7f47c80d1a60 2024-06-29T14:34:36.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.363+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c81f3b00 con 0x7f47c80d1a60 2024-06-29T14:34:36.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.363+0000 7f47ba7fc640 1 -- 172.21.15.59:0/1716331761 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c00047a0 con 0x7f47c80d1a60 2024-06-29T14:34:36.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.363+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c81f40c0 con 0x7f47c80d1a60 2024-06-29T14:34:36.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.364+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c80a9e40 con 0x7f47c80d1a60 2024-06-29T14:34:36.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.364+0000 7f47ba7fc640 1 -- 172.21.15.59:0/1716331761 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f47c0004940 con 0x7f47c80d1a60 2024-06-29T14:34:36.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.365+0000 7f47ba7fc640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f47ac076500 0x7f47ac078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:36.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.365+0000 7f47ba7fc640 1 -- 172.21.15.59:0/1716331761 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 7356+0+0 (secure 0 0 0) 0x7f47c0098150 con 0x7f47c80d1a60 2024-06-29T14:34:36.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.365+0000 7f47cd94e640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f47ac076500 0x7f47ac078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:36.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.366+0000 7f47cd94e640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f47ac076500 0x7f47ac078a10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f47c400b450 tx=0x7f47c400d040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:36.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.369+0000 7f47ba7fc640 1 -- 172.21.15.59:0/1716331761 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47c0060a80 con 0x7f47c80d1a60 2024-06-29T14:34:36.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.593+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f47c80b6b20 con 0x7f47ac076500 2024-06-29T14:34:36.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.677+0000 7f47ba7fc640 1 -- 172.21.15.59:0/1716331761 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f47c80b6b20 con 0x7f47ac076500 2024-06-29T14:34:36.675 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-06-29T14:34:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f47ac076500 msgr2=0x7f47ac078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f47ac076500 0x7f47ac078a10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f47c400b450 tx=0x7f47c400d040 comp rx=0 tx=0).stop 2024-06-29T14:34:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80d1a60 msgr2=0x7f47c815a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80d1a60 0x7f47c815a2f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f47c000d5f0 tx=0x7f47c000dab0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 shutdown_connections 2024-06-29T14:34:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f47ac076500 0x7f47ac078a10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c80d1a60 0x7f47c815a2f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c80b2fa0 0x7f47c80a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 --2- 172.21.15.59:0/1716331761 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c80b2270 0x7f47c815ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 >> 172.21.15.59:0/1716331761 conn(0x7f47c8073e10 msgr2=0x7f47c8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 shutdown_connections 2024-06-29T14:34:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:36.679+0000 7f47cfbb2640 1 -- 172.21.15.59:0/1716331761 wait complete. 2024-06-29T14:34:36.688 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-06-29T14:34:36.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.688 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-06-29T14:34:36.705 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:36.705 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:36.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:36.706 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:36.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.707 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:36.768 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:36.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.769 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:36.831 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:36.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.831 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:36.895 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:36.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.895 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:36.957 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:36.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.958 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:37.021 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:37.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.022 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:37.103 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:37.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.104 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:37.168 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:37.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.169 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:37.231 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:37.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.231 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:37.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:37.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.307 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:37.394 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:37.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.394 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:37.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.428 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-06-29T14:34:37.494 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-06-29T14:34:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.656+0000 7f68f6e47640 1 -- 172.21.15.59:0/3745927251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f013e9b0 msgr2=0x7f68f013edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.656+0000 7f68f6e47640 1 --2- 172.21.15.59:0/3745927251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f013e9b0 0x7f68f013edc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f68d8005fd0 tx=0x7f68d802a7a0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.656+0000 7f68f6e47640 1 -- 172.21.15.59:0/3745927251 shutdown_connections 2024-06-29T14:34:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.656+0000 7f68f6e47640 1 --2- 172.21.15.59:0/3745927251 >> v2:172.21.15.153:3300/0 conn(0x7f68f01414c0 0x7f68f01418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.656+0000 7f68f6e47640 1 --2- 172.21.15.59:0/3745927251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f013e9b0 0x7f68f013edc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.656+0000 7f68f6e47640 1 -- 172.21.15.59:0/3745927251 >> 172.21.15.59:0/3745927251 conn(0x7f68f00b6fb0 msgr2=0x7f68f00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:37.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.656+0000 7f68f6e47640 1 -- 172.21.15.59:0/3745927251 shutdown_connections 2024-06-29T14:34:37.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.656+0000 7f68f6e47640 1 -- 172.21.15.59:0/3745927251 wait complete. 2024-06-29T14:34:37.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.658+0000 7f68f6e47640 1 Processor -- start 2024-06-29T14:34:37.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.658+0000 7f68f6e47640 1 -- start start 2024-06-29T14:34:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f6e47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f013e9b0 0x7f68f0070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f6e47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f01414c0 0x7f68f014a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f6e47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f015ceb0 0x7f68f014a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f6e47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f68f00b2a30 con 0x7f68f01414c0 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f6e47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f68f00b28b0 con 0x7f68f015ceb0 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f6e47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f68f00b2bb0 con 0x7f68f013e9b0 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f01414c0 0x7f68f014a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f01414c0 0x7f68f014a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55438/0 (socket says 172.21.15.59:55438) 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 -- 172.21.15.59:0/690040079 learned_addr learned my addr 172.21.15.59:0/690040079 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f4be3640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f013e9b0 0x7f68f0070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f53e4640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f015ceb0 0x7f68f014a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 -- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f013e9b0 msgr2=0x7f68f0070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f013e9b0 0x7f68f0070bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 -- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f015ceb0 msgr2=0x7f68f014a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:37.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f015ceb0 0x7f68f014a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 -- 172.21.15.59:0/690040079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f01f36b0 con 0x7f68f01414c0 2024-06-29T14:34:37.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68effff640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f01414c0 0x7f68f014a2a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f68d8005fd0 tx=0x7f68d8032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:37.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68ed7fa640 1 -- 172.21.15.59:0/690040079 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d8041020 con 0x7f68f01414c0 2024-06-29T14:34:37.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68ed7fa640 1 -- 172.21.15.59:0/690040079 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68d8032e50 con 0x7f68f01414c0 2024-06-29T14:34:37.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.660+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f0152f50 con 0x7f68f01414c0 2024-06-29T14:34:37.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.661+0000 7f68ed7fa640 1 -- 172.21.15.59:0/690040079 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d8036040 con 0x7f68f01414c0 2024-06-29T14:34:37.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.661+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f01f3920 con 0x7f68f01414c0 2024-06-29T14:34:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.662+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f01f2a30 con 0x7f68f01414c0 2024-06-29T14:34:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.663+0000 7f68ed7fa640 1 -- 172.21.15.59:0/690040079 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f68d8044020 con 0x7f68f01414c0 2024-06-29T14:34:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.663+0000 7f68ed7fa640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f68d00764d0 0x7f68d00789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.663+0000 7f68ed7fa640 1 -- 172.21.15.59:0/690040079 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 7276+0+0 (secure 0 0 0) 0x7f68d80b94d0 con 0x7f68f01414c0 2024-06-29T14:34:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.663+0000 7f68f4be3640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f68d00764d0 0x7f68d00789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:37.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.664+0000 7f68f4be3640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f68d00764d0 0x7f68d00789e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f68e0007960 tx=0x7f68e0001290 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:37.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.667+0000 7f68ed7fa640 1 -- 172.21.15.59:0/690040079 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f68d804b070 con 0x7f68f01414c0 2024-06-29T14:34:37.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.896+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f68f013edc0 con 0x7f68d00764d0 2024-06-29T14:34:37.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.974+0000 7f68ed7fa640 1 -- 172.21.15.59:0/690040079 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f68f013edc0 con 0x7f68d00764d0 2024-06-29T14:34:37.973 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-06-29T14:34:37.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.976+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f68d00764d0 msgr2=0x7f68d00789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:37.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.976+0000 7f68f6e47640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f68d00764d0 0x7f68d00789e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f68e0007960 tx=0x7f68e0001290 comp rx=0 tx=0).stop 2024-06-29T14:34:37.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.976+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f01414c0 msgr2=0x7f68f014a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:37.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.976+0000 7f68f6e47640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f01414c0 0x7f68f014a2a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f68d8005fd0 tx=0x7f68d8032c60 comp rx=0 tx=0).stop 2024-06-29T14:34:37.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.977+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 shutdown_connections 2024-06-29T14:34:37.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.977+0000 7f68f6e47640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f68d00764d0 0x7f68d00789e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.977+0000 7f68f6e47640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f015ceb0 0x7f68f014a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.977+0000 7f68f6e47640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f01414c0 0x7f68f014a2a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.977+0000 7f68f6e47640 1 --2- 172.21.15.59:0/690040079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f013e9b0 0x7f68f0070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.977+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 >> 172.21.15.59:0/690040079 conn(0x7f68f00b6fb0 msgr2=0x7f68f013d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:37.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.977+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 shutdown_connections 2024-06-29T14:34:37.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:37.977+0000 7f68f6e47640 1 -- 172.21.15.59:0/690040079 wait complete. 2024-06-29T14:34:37.986 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-06-29T14:34:37.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.986 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_01_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:38.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:38.004 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:38.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.004 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:38.066 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:38.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.067 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:38.129 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:38.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.129 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:38.191 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:38.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.191 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:38.252 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_28_01_UTC 2024-06-29T14:34:38.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.253 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_28_01_UTC) 2024-06-29T14:34:38.322 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:38.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.322 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:38.384 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:38.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.384 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:38.447 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:38.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.447 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:38.509 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:38.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.510 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:38.570 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:38.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.571 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:38.635 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:38.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.635 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:38.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.697 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-06-29T14:34:38.756 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-06-29T14:34:38.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.918+0000 7fda76874640 1 -- 172.21.15.59:0/2700866960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda701458a0 msgr2=0x7fda700cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:38.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.918+0000 7fda76874640 1 --2- 172.21.15.59:0/2700866960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda701458a0 0x7fda700cf870 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fda60009830 tx=0x7fda6002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:34:38.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.918+0000 7fda76874640 1 -- 172.21.15.59:0/2700866960 shutdown_connections 2024-06-29T14:34:38.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.918+0000 7fda76874640 1 --2- 172.21.15.59:0/2700866960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda701458a0 0x7fda700cf870 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:38.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.918+0000 7fda76874640 1 -- 172.21.15.59:0/2700866960 >> 172.21.15.59:0/2700866960 conn(0x7fda7006c950 msgr2=0x7fda70073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:38.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.918+0000 7fda76874640 1 -- 172.21.15.59:0/2700866960 shutdown_connections 2024-06-29T14:34:38.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.918+0000 7fda76874640 1 -- 172.21.15.59:0/2700866960 wait complete. 2024-06-29T14:34:38.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.920+0000 7fda76874640 1 Processor -- start 2024-06-29T14:34:38.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.920+0000 7fda76874640 1 -- start start 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda76874640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda700b3740 0x7fda70071650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda76874640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda700b3b50 0x7fda700ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda76874640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda70142f20 0x7fda70143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda76874640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fda700b6320 con 0x7fda700b3740 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda76874640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda700b61a0 con 0x7fda700b3b50 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda76874640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda700b64a0 con 0x7fda70142f20 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda700b3740 0x7fda70071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda700b3740 0x7fda70071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55448/0 (socket says 172.21.15.59:55448) 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 -- 172.21.15.59:0/2539700253 learned_addr learned my addr 172.21.15.59:0/2539700253 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6f7fe640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda700b3b50 0x7fda700ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda74e11640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda70142f20 0x7fda70143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 -- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda70142f20 msgr2=0x7fda70143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda70142f20 0x7fda70143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 -- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda700b3b50 msgr2=0x7fda700ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda700b3b50 0x7fda700ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 -- 172.21.15.59:0/2539700253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda70143870 con 0x7fda700b3740 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.922+0000 7fda6ffff640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda700b3740 0x7fda70071650 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fda60009950 tx=0x7fda60033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.923+0000 7fda6cff9640 1 -- 172.21.15.59:0/2539700253 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60031b50 con 0x7fda700b3740 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.923+0000 7fda6cff9640 1 -- 172.21.15.59:0/2539700253 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda60031de0 con 0x7fda700b3740 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.923+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda70136590 con 0x7fda700b3740 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.923+0000 7fda6cff9640 1 -- 172.21.15.59:0/2539700253 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60005280 con 0x7fda700b3740 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.923+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda70071b90 con 0x7fda700b3740 2024-06-29T14:34:38.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.924+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda701458a0 con 0x7fda700b3740 2024-06-29T14:34:38.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.925+0000 7fda6cff9640 1 -- 172.21.15.59:0/2539700253 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fda60005420 con 0x7fda700b3740 2024-06-29T14:34:38.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.925+0000 7fda6cff9640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fda5c076500 0x7fda5c078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:38.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.925+0000 7fda6cff9640 1 -- 172.21.15.59:0/2539700253 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6988+0+0 (secure 0 0 0) 0x7fda600b9060 con 0x7fda700b3740 2024-06-29T14:34:38.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.925+0000 7fda6f7fe640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fda5c076500 0x7fda5c078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.926+0000 7fda6f7fe640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fda5c076500 0x7fda5c078a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda70140cb0 tx=0x7fda64009040 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:38.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:38.930+0000 7fda6cff9640 1 -- 172.21.15.59:0/2539700253 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda70143870 con 0x7fda700b3740 2024-06-29T14:34:39.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.148+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fda701409b0 con 0x7fda5c076500 2024-06-29T14:34:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.282+0000 7fda6cff9640 1 -- 172.21.15.59:0/2539700253 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7fda6400f7d0 con 0x7fda5c076500 2024-06-29T14:34:39.280 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-06-29T14:34:39.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fda5c076500 msgr2=0x7fda5c078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:39.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fda5c076500 0x7fda5c078a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda70140cb0 tx=0x7fda64009040 comp rx=0 tx=0).stop 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda700b3740 msgr2=0x7fda70071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda700b3740 0x7fda70071650 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fda60009950 tx=0x7fda60033ba0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 shutdown_connections 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fda5c076500 0x7fda5c078a10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda70142f20 0x7fda70143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda700b3b50 0x7fda700ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 --2- 172.21.15.59:0/2539700253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda700b3740 0x7fda70071650 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 >> 172.21.15.59:0/2539700253 conn(0x7fda7006c950 msgr2=0x7fda700d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 shutdown_connections 2024-06-29T14:34:39.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:39.284+0000 7fda76874640 1 -- 172.21.15.59:0/2539700253 wait complete. 2024-06-29T14:34:39.293 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-06-29T14:34:39.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.294 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:39.312 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:39.313 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:39.313 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:39.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.313 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:39.380 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:39.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.380 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:39.443 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:39.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.443 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:39.506 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:39.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.506 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:39.569 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:39.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.569 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:39.631 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:39.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.632 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:39.695 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:39.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.695 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:39.757 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:39.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.758 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:39.821 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:39.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.822 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:39.888 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:39.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.888 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:39.951 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:39.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.952 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:40.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.015 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-06-29T14:34:40.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-06-29T14:34:40.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.247+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/3972598724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c141448e0 msgr2=0x7f5c140cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:40.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.247+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/3972598724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c141448e0 0x7f5c140cfbe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5c08005fd0 tx=0x7f5c0802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.247+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/3972598724 shutdown_connections 2024-06-29T14:34:40.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.247+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/3972598724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c140d0120 0x7f5c140cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.247+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/3972598724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c141448e0 0x7f5c140cfbe0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.247+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/3972598724 >> 172.21.15.59:0/3972598724 conn(0x7f5c140b2ab0 msgr2=0x7f5c140b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:40.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.247+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/3972598724 shutdown_connections 2024-06-29T14:34:40.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.247+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/3972598724 wait complete. 2024-06-29T14:34:40.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.249+0000 7f5c1b8fd640 1 Processor -- start 2024-06-29T14:34:40.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.249+0000 7f5c1b8fd640 1 -- start start 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.250+0000 7f5c1b8fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c140cd7c0 0x7f5c140b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.250+0000 7f5c1b8fd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c140d0120 0x7f5c141596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.250+0000 7f5c1b8fd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c141448e0 0x7f5c14072c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.250+0000 7f5c1b8fd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1415fe30 con 0x7f5c140d0120 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.250+0000 7f5c1b8fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1406dc90 con 0x7f5c140cd7c0 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.250+0000 7f5c1b8fd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5c1415ffb0 con 0x7f5c141448e0 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c140d0120 0x7f5c141596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c140d0120 0x7f5c141596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55456/0 (socket says 172.21.15.59:55456) 2024-06-29T14:34:40.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 -- 172.21.15.59:0/2306069278 learned_addr learned my addr 172.21.15.59:0/2306069278 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c19699640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c140cd7c0 0x7f5c140b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c19e9a640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c141448e0 0x7f5c14072c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 -- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c141448e0 msgr2=0x7f5c14072c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c141448e0 0x7f5c14072c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 -- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c140cd7c0 msgr2=0x7f5c140b55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c140cd7c0 0x7f5c140b55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 -- 172.21.15.59:0/2306069278 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1415e500 con 0x7f5c140d0120 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c18e98640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c140d0120 0x7f5c141596a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000e790 tx=0x7f5c1000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c05ffb640 1 -- 172.21.15.59:0/2306069278 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1000eea0 con 0x7f5c140d0120 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c05ffb640 1 -- 172.21.15.59:0/2306069278 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c1000cdb0 con 0x7f5c140d0120 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c1415e7e0 con 0x7f5c140d0120 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.251+0000 7f5c05ffb640 1 -- 172.21.15.59:0/2306069278 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c10010690 con 0x7f5c140d0120 2024-06-29T14:34:40.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.252+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c140ad950 con 0x7f5c140d0120 2024-06-29T14:34:40.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.253+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c1415e270 con 0x7f5c140d0120 2024-06-29T14:34:40.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.253+0000 7f5c05ffb640 1 -- 172.21.15.59:0/2306069278 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f5c10010830 con 0x7f5c140d0120 2024-06-29T14:34:40.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.254+0000 7f5c05ffb640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5bf80764d0 0x7f5bf80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:40.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.254+0000 7f5c05ffb640 1 -- 172.21.15.59:0/2306069278 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 7020+0+0 (secure 0 0 0) 0x7f5c10098930 con 0x7f5c140d0120 2024-06-29T14:34:40.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.254+0000 7f5c19699640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5bf80764d0 0x7f5bf80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:40.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.255+0000 7f5c19699640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5bf80764d0 0x7f5bf80789e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c08033000 tx=0x7f5c08004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:40.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.259+0000 7f5c05ffb640 1 -- 172.21.15.59:0/2306069278 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5c10061440 con 0x7f5c140d0120 2024-06-29T14:34:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.506+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 --> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5c140b5ae0 con 0x7f5bf80764d0 2024-06-29T14:34:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.595+0000 7f5c05ffb640 1 -- 172.21.15.59:0/2306069278 <== mgr.5458 v2:172.21.15.153:6832/3430128047 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f5c080055c0 con 0x7f5bf80764d0 2024-06-29T14:34:40.594 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5bf80764d0 msgr2=0x7f5bf80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5bf80764d0 0x7f5bf80789e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c08033000 tx=0x7f5c08004730 comp rx=0 tx=0).stop 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c140d0120 msgr2=0x7f5c141596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c140d0120 0x7f5c141596a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000e790 tx=0x7f5c1000ec50 comp rx=0 tx=0).stop 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 shutdown_connections 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5bf80764d0 0x7f5bf80789e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c141448e0 0x7f5c14072c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c140d0120 0x7f5c141596a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 --2- 172.21.15.59:0/2306069278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c140cd7c0 0x7f5c140b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.597+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 >> 172.21.15.59:0/2306069278 conn(0x7f5c140b2ab0 msgr2=0x7f5c140b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:40.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.598+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 shutdown_connections 2024-06-29T14:34:40.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:40.598+0000 7f5c1b8fd640 1 -- 172.21.15.59:0/2306069278 wait complete. 2024-06-29T14:34:40.606 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-06-29T14:34:40.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.607 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-06-29T14:34:40.623 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:40.623 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:40.624 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:40.624 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:40.624 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:40.624 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:40.624 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:40.624 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:40.624 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:40.624 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:40.625 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:40.625 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:40.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.625 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:40.687 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:40.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.687 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:40.751 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:40.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.751 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:40.815 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:40.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.815 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:40.877 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:40.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.877 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:40.938 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:40.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.939 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:41.000 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:41.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:41.001 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:41.063 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:41.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:41.063 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:41.127 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:41.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:41.128 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:41.198 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:41.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:41.198 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:41.261 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:41.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:41.261 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:41.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:41.343 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-06-29T14:34:41.403 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:34:41.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.564+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/2700673888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e64144470 msgr2=0x7f7e640b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.564+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/2700673888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e64144470 0x7f7e640b1b00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7e64072450 tx=0x7f7e6002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.565+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/2700673888 shutdown_connections 2024-06-29T14:34:41.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.565+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/2700673888 >> v2:172.21.15.153:3300/0 conn(0x7f7e640b2040 0x7f7e6406e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.565+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/2700673888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e64144470 0x7f7e640b1b00 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.565+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/2700673888 >> 172.21.15.59:0/2700673888 conn(0x7f7e640b06e0 msgr2=0x7f7e640b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:41.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.565+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/2700673888 shutdown_connections 2024-06-29T14:34:41.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.565+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/2700673888 wait complete. 2024-06-29T14:34:41.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.566+0000 7f7e6c3a5640 1 Processor -- start 2024-06-29T14:34:41.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.566+0000 7f7e6c3a5640 1 -- start start 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6c3a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e6406e840 0x7f7e640b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6c3a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e640b2040 0x7f7e6415b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6c3a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e64144470 0x7f7e6413fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6c3a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7e640d0360 con 0x7f7e640b2040 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6c3a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e640d01e0 con 0x7f7e6406e840 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6c3a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6415ee80 con 0x7f7e64144470 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e69940640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e640b2040 0x7f7e6415b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6a141640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e6406e840 0x7f7e640b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e69940640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e640b2040 0x7f7e6415b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55482/0 (socket says 172.21.15.59:55482) 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6a141640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e6406e840 0x7f7e640b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:58178/0 (socket says 172.21.15.59:58178) 2024-06-29T14:34:41.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e69940640 1 -- 172.21.15.59:0/1567448014 learned_addr learned my addr 172.21.15.59:0/1567448014 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.568+0000 7f7e6a942640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e64144470 0x7f7e6413fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e69940640 1 -- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e64144470 msgr2=0x7f7e6413fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e69940640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e64144470 0x7f7e6413fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e69940640 1 -- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e6406e840 msgr2=0x7f7e640b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e69940640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e6406e840 0x7f7e640b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e69940640 1 -- 172.21.15.59:0/1567448014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e640b4670 con 0x7f7e640b2040 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e69940640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e640b2040 0x7f7e6415b3b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400d750 tx=0x7f7e5400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e5affd640 1 -- 172.21.15.59:0/1567448014 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e54017070 con 0x7f7e640b2040 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e5affd640 1 -- 172.21.15.59:0/1567448014 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e54009b20 con 0x7f7e640b2040 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e640b4950 con 0x7f7e640b2040 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e5affd640 1 -- 172.21.15.59:0/1567448014 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e54010460 con 0x7f7e640b2040 2024-06-29T14:34:41.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.569+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e640b4f10 con 0x7f7e640b2040 2024-06-29T14:34:41.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.570+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e6415f340 con 0x7f7e640b2040 2024-06-29T14:34:41.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.571+0000 7f7e5affd640 1 -- 172.21.15.59:0/1567448014 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f7e5401a070 con 0x7f7e640b2040 2024-06-29T14:34:41.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.572+0000 7f7e5affd640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7e480764d0 0x7f7e480789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.572+0000 7f7e5affd640 1 -- 172.21.15.59:0/1567448014 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7116+0+0 (secure 0 0 0) 0x7f7e54098080 con 0x7f7e640b2040 2024-06-29T14:34:41.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.572+0000 7f7e6a141640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7e480764d0 0x7f7e480789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.572+0000 7f7e6a141640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7e480764d0 0x7f7e480789e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e60033000 tx=0x7f7e60004730 comp rx=0 tx=0).ready entity=mgr.5458 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:41.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.576+0000 7f7e5affd640 1 -- 172.21.15.59:0/1567448014 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7e54060bc0 con 0x7f7e640b2040 2024-06-29T14:34:41.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:41.825+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f7e640b1e20 con 0x7f7e640b2040 2024-06-29T14:34:42.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.323+0000 7f7e5affd640 1 -- 172.21.15.59:0/1567448014 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) ==== 93+0+0 (secure 0 0 0) 0x7f7e54069030 con 0x7f7e640b2040 2024-06-29T14:34:42.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.329+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7e480764d0 msgr2=0x7f7e480789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.329+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7e480764d0 0x7f7e480789e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e60033000 tx=0x7f7e60004730 comp rx=0 tx=0).stop 2024-06-29T14:34:42.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.329+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e640b2040 msgr2=0x7f7e6415b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.329+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e640b2040 0x7f7e6415b3b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400d750 tx=0x7f7e5400dc10 comp rx=0 tx=0).stop 2024-06-29T14:34:42.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.330+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 shutdown_connections 2024-06-29T14:34:42.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.330+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f7e480764d0 0x7f7e480789e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.330+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e64144470 0x7f7e6413fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.330+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e640b2040 0x7f7e6415b3b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.330+0000 7f7e6c3a5640 1 --2- 172.21.15.59:0/1567448014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e6406e840 0x7f7e640b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.330+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 >> 172.21.15.59:0/1567448014 conn(0x7f7e640b06e0 msgr2=0x7f7e64138ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.330+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 shutdown_connections 2024-06-29T14:34:42.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.330+0000 7f7e6c3a5640 1 -- 172.21.15.59:0/1567448014 wait complete. 2024-06-29T14:34:42.343 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:34:42.343 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:34:42.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:34:42.431 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:34:42.431 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:34:42.431 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:34:42.431 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:42.448 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:34:42.448 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:42.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 -- 172.21.15.59:0/2031232542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 msgr2=0x7f60d815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/2031232542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 0x7f60d815e8d0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f60d80d15d0 tx=0x7f60c002a560 comp rx=0 tx=0).stop 2024-06-29T14:34:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 -- 172.21.15.59:0/2031232542 shutdown_connections 2024-06-29T14:34:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/2031232542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d815ee10 0x7f60d815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/2031232542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 0x7f60d815e8d0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/2031232542 >> v2:172.21.15.153:3300/0 conn(0x7f60d80d23c0 0x7f60d80b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 -- 172.21.15.59:0/2031232542 >> 172.21.15.59:0/2031232542 conn(0x7f60d80d3ee0 msgr2=0x7f60d80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 -- 172.21.15.59:0/2031232542 shutdown_connections 2024-06-29T14:34:42.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.681+0000 7f60dd6cc640 1 -- 172.21.15.59:0/2031232542 wait complete. 2024-06-29T14:34:42.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.683+0000 7f60dd6cc640 1 Processor -- start 2024-06-29T14:34:42.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.683+0000 7f60dd6cc640 1 -- start start 2024-06-29T14:34:42.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.684+0000 7f60dd6cc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 0x7f60d80c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.684+0000 7f60dd6cc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d23c0 0x7f60d81f5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.684+0000 7f60dd6cc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d815ee10 0x7f60d81557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60dd6cc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f60d8160070 con 0x7f60d80b3ed0 2024-06-29T14:34:42.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60dd6cc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60d815fef0 con 0x7f60d80d23c0 2024-06-29T14:34:42.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60dd6cc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f60d81601f0 con 0x7f60d815ee10 2024-06-29T14:34:42.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 0x7f60d80c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 0x7f60d80c99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55520/0 (socket says 172.21.15.59:55520) 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 -- 172.21.15.59:0/3950802384 learned_addr learned my addr 172.21.15.59:0/3950802384 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60cffff640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d23c0 0x7f60d81f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d77fe640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d815ee10 0x7f60d81557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 -- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d815ee10 msgr2=0x7f60d81557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d815ee10 0x7f60d81557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 -- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d23c0 msgr2=0x7f60d81f5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d23c0 0x7f60d81f5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 -- 172.21.15.59:0/3950802384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d8157a80 con 0x7f60d80b3ed0 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60d6ffd640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 0x7f60d80c99f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f60c800e710 tx=0x7f60c800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60cf7fe640 1 -- 172.21.15.59:0/3950802384 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8004220 con 0x7f60d80b3ed0 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60cf7fe640 1 -- 172.21.15.59:0/3950802384 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60c80043c0 con 0x7f60d80b3ed0 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d81e80f0 con 0x7f60d80b3ed0 2024-06-29T14:34:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60cf7fe640 1 -- 172.21.15.59:0/3950802384 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c800cb10 con 0x7f60d80b3ed0 2024-06-29T14:34:42.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.685+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d81e86b0 con 0x7f60d80b3ed0 2024-06-29T14:34:42.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.687+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d80b3990 con 0x7f60d80b3ed0 2024-06-29T14:34:42.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.687+0000 7f60cf7fe640 1 -- 172.21.15.59:0/3950802384 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f60c80035f0 con 0x7f60d80b3ed0 2024-06-29T14:34:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.688+0000 7f60cf7fe640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f60b8076480 0x7f60b8078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.688+0000 7f60cffff640 1 -- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f60b8076480 msgr2=0x7f60b8078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.688+0000 7f60cf7fe640 1 -- 172.21.15.59:0/3950802384 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7068+0+0 (secure 0 0 0) 0x7f60c8098ab0 con 0x7f60d80b3ed0 2024-06-29T14:34:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.688+0000 7f60cffff640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f60b8076480 0x7f60b8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:42.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.693+0000 7f60cf7fe640 1 -- 172.21.15.59:0/3950802384 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f60c8061450 con 0x7f60d80b3ed0 2024-06-29T14:34:42.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.887+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60d80d2950 con 0x7f60d80b3ed0 2024-06-29T14:34:42.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.888+0000 7f60cf7fe640 1 -- 172.21.15.59:0/3950802384 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+18435 (secure 0 0 0) 0x7f60c8065750 con 0x7f60d80b3ed0 2024-06-29T14:34:42.886 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:42.886 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:34:42.189630+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:13.422166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:14.527412+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":218,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1180005807":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6832/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6833/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/2990390183":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/3207465637":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/2185789752":"2024-06-30T14:23:32.829085+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":2},{"begin":5,"length":3},{"begin":9,"length":1},{"begin":11,"length":3},{"begin":15,"length":3},{"begin":19,"length":1},{"begin":21,"length":4},{"begin":26,"length":3},{"begin":30,"length":1},{"begin":33,"length":3},{"begin":37,"length":3},{"begin":41,"length":1},{"begin":43,"length":3},{"begin":47,"length":3},{"begin":51,"length":2},{"begin":54,"length":3},{"begin":58,"length":3},{"begin":63,"length":1},{"begin":65,"length":3},{"begin":69,"length":4},{"begin":74,"length":1},{"begin":76,"length":3},{"begin":80,"length":3},{"begin":85,"length":1},{"begin":87,"length":3},{"begin":91,"length":3},{"begin":96,"length":1},{"begin":98,"length":3},{"begin":103,"length":2},{"begin":107,"length":1},{"begin":109,"length":3},{"begin":145,"length":74}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":7,"length":1},{"begin":17,"length":1},{"begin":28,"length":1},{"begin":39,"length":1},{"begin":49,"length":1},{"begin":60,"length":1},{"begin":71,"length":2},{"begin":111,"length":1},{"begin":210,"length":9}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:42.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60cffff640 1 -- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f60b8076480 msgr2=0x7f60b8078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:42.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60cffff640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f60b8076480 0x7f60b8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:42.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f60b8076480 msgr2=0x7f60b8078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:42.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f60b8076480 0x7f60b8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 msgr2=0x7f60d80c99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 0x7f60d80c99f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f60c800e710 tx=0x7f60c800ebd0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 shutdown_connections 2024-06-29T14:34:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f60b8076480 0x7f60b8078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d815ee10 0x7f60d81557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d23c0 0x7f60d81f5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 --2- 172.21.15.59:0/3950802384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d80b3ed0 0x7f60d80c99f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 >> 172.21.15.59:0/3950802384 conn(0x7f60d80d3ee0 msgr2=0x7f60d8160730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 shutdown_connections 2024-06-29T14:34:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:42.890+0000 7f60dd6cc640 1 -- 172.21.15.59:0/3950802384 wait complete. 2024-06-29T14:34:42.899 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:23:13.422166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:23:14.527412+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 218, 'snap_epoch': 55, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:34:42.900 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:34:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.024+0000 7f2757b9a640 1 -- 172.21.15.59:0/1526444969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500d1a60 msgr2=0x7f27500b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.024+0000 7f2757b9a640 1 --2- 172.21.15.59:0/1526444969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500d1a60 0x7f27500b5b30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f27500b2680 tx=0x7f274802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.024+0000 7f2757b9a640 1 -- 172.21.15.59:0/1526444969 shutdown_connections 2024-06-29T14:34:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.024+0000 7f2757b9a640 1 --2- 172.21.15.59:0/1526444969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500d1a60 0x7f27500b5b30 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.024+0000 7f2757b9a640 1 --2- 172.21.15.59:0/1526444969 >> v2:172.21.15.153:3300/0 conn(0x7f27500b2fa0 0x7f27500d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.024+0000 7f2757b9a640 1 -- 172.21.15.59:0/1526444969 >> 172.21.15.59:0/1526444969 conn(0x7f2750073e10 msgr2=0x7f2750074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.024+0000 7f2757b9a640 1 -- 172.21.15.59:0/1526444969 shutdown_connections 2024-06-29T14:34:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.024+0000 7f2757b9a640 1 -- 172.21.15.59:0/1526444969 wait complete. 2024-06-29T14:34:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.026+0000 7f2757b9a640 1 Processor -- start 2024-06-29T14:34:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.026+0000 7f2757b9a640 1 -- start start 2024-06-29T14:34:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.027+0000 7f2757b9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27500b2270 0x7f275015d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.027+0000 7f2757b9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500b2fa0 0x7f27500ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.027+0000 7f2757b9a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27500d1a60 0x7f27501592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.027+0000 7f2757b9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f27500b6400 con 0x7f27500b2fa0 2024-06-29T14:34:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.027+0000 7f2757b9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27500b6280 con 0x7f27500b2270 2024-06-29T14:34:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.027+0000 7f2757b9a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f27500b6580 con 0x7f27500d1a60 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500b2fa0 0x7f27500ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500b2fa0 0x7f27500ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49732/0 (socket says 172.21.15.59:49732) 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 -- 172.21.15.59:0/2320572070 learned_addr learned my addr 172.21.15.59:0/2320572070 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755936640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27500b2270 0x7f275015d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2756137640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27500d1a60 0x7f27501592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 -- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27500d1a60 msgr2=0x7f27501592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27500d1a60 0x7f27501592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 -- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27500b2270 msgr2=0x7f275015d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27500b2270 0x7f275015d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 -- 172.21.15.59:0/2320572070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275015ded0 con 0x7f27500b2fa0 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2755135640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500b2fa0 0x7f27500ae770 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f274400e790 tx=0x7f274400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f27427fc640 1 -- 172.21.15.59:0/2320572070 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274400eea0 con 0x7f27500b2fa0 2024-06-29T14:34:43.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f27427fc640 1 -- 172.21.15.59:0/2320572070 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f274400cd50 con 0x7f27500b2fa0 2024-06-29T14:34:43.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f27427fc640 1 -- 172.21.15.59:0/2320572070 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744010690 con 0x7f27500b2fa0 2024-06-29T14:34:43.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275015e1b0 con 0x7f27500b2fa0 2024-06-29T14:34:43.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.028+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275015d990 con 0x7f27500b2fa0 2024-06-29T14:34:43.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.030+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27500b6aa0 con 0x7f27500b2fa0 2024-06-29T14:34:43.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.030+0000 7f27427fc640 1 -- 172.21.15.59:0/2320572070 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f2744010830 con 0x7f27500b2fa0 2024-06-29T14:34:43.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.031+0000 7f27427fc640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2734076480 0x7f2734078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.031+0000 7f2755936640 1 -- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2734076480 msgr2=0x7f2734078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:43.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.031+0000 7f27427fc640 1 -- 172.21.15.59:0/2320572070 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 7068+0+0 (secure 0 0 0) 0x7f2744098910 con 0x7f27500b2fa0 2024-06-29T14:34:43.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.031+0000 7f2755936640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2734076480 0x7f2734078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.036+0000 7f27427fc640 1 -- 172.21.15.59:0/2320572070 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f27440613f0 con 0x7f27500b2fa0 2024-06-29T14:34:43.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.233+0000 7f2755936640 1 -- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2734076480 msgr2=0x7f2734078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:43.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.233+0000 7f2755936640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2734076480 0x7f2734078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.236+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2750157ee0 con 0x7f27500b2fa0 2024-06-29T14:34:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.236+0000 7f27427fc640 1 -- 172.21.15.59:0/2320572070 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f27440656f0 con 0x7f27500b2fa0 2024-06-29T14:34:43.233 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:34:43.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2734076480 msgr2=0x7f2734078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2734076480 0x7f2734078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500b2fa0 msgr2=0x7f27500ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500b2fa0 0x7f27500ae770 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f274400e790 tx=0x7f274400ec50 comp rx=0 tx=0).stop 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 shutdown_connections 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2734076480 0x7f2734078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27500d1a60 0x7f27501592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27500b2fa0 0x7f27500ae770 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 --2- 172.21.15.59:0/2320572070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27500b2270 0x7f275015d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 >> 172.21.15.59:0/2320572070 conn(0x7f2750073e10 msgr2=0x7f275015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 shutdown_connections 2024-06-29T14:34:43.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.238+0000 7f2757b9a640 1 -- 172.21.15.59:0/2320572070 wait complete. 2024-06-29T14:34:43.247 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:34:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.373+0000 7f55add73640 1 -- 172.21.15.59:0/3301904343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a8140f50 msgr2=0x7f55a80b6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.373+0000 7f55add73640 1 --2- 172.21.15.59:0/3301904343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a8140f50 0x7f55a80b6700 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5598005fd0 tx=0x7f559802a4b0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.373+0000 7f55add73640 1 -- 172.21.15.59:0/3301904343 shutdown_connections 2024-06-29T14:34:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.373+0000 7f55add73640 1 --2- 172.21.15.59:0/3301904343 >> v2:172.21.15.153:3300/0 conn(0x7f55a80d0090 0x7f55a8140300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.373+0000 7f55add73640 1 --2- 172.21.15.59:0/3301904343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a8140f50 0x7f55a80b6700 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.373+0000 7f55add73640 1 -- 172.21.15.59:0/3301904343 >> 172.21.15.59:0/3301904343 conn(0x7f55a8073270 msgr2=0x7f55a8073680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.373+0000 7f55add73640 1 -- 172.21.15.59:0/3301904343 shutdown_connections 2024-06-29T14:34:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.373+0000 7f55add73640 1 -- 172.21.15.59:0/3301904343 wait complete. 2024-06-29T14:34:43.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.375+0000 7f55add73640 1 Processor -- start 2024-06-29T14:34:43.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.375+0000 7f55add73640 1 -- start start 2024-06-29T14:34:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.376+0000 7f55add73640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a80cfc80 0x7f55a81e46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55add73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a80d0090 0x7f55a81578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55add73640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a8140f50 0x7f55a81541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55add73640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f55a806fc00 con 0x7f55a80d0090 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55add73640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55a806fa80 con 0x7f55a8140f50 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55add73640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f55a806fd80 con 0x7f55a80cfc80 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a80d0090 0x7f55a81578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a80cfc80 0x7f55a81e46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a8140f50 0x7f55a81541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a80cfc80 0x7f55a81e46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:34910/0 (socket says 172.21.15.59:34910) 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a80d0090 0x7f55a81578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49742/0 (socket says 172.21.15.59:49742) 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 -- 172.21.15.59:0/3469506092 learned_addr learned my addr 172.21.15.59:0/3469506092 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 -- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a8140f50 msgr2=0x7f55a81541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a8140f50 0x7f55a81541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 -- 172.21.15.59:0/3469506092 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a80d0090 msgr2=0x7f55a81578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a80d0090 0x7f55a81578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 -- 172.21.15.59:0/3469506092 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a81e3640 con 0x7f55a80cfc80 2024-06-29T14:34:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a6ffd640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a80d0090 0x7f55a81578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:43.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.377+0000 7f55a77fe640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a80cfc80 0x7f55a81e46e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5598000c00 tx=0x7f55980038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:43.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.378+0000 7f559ffff640 1 -- 172.21.15.59:0/3469506092 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559803e030 con 0x7f55a80cfc80 2024-06-29T14:34:43.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.378+0000 7f559ffff640 1 -- 172.21.15.59:0/3469506092 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5598003ca0 con 0x7f55a80cfc80 2024-06-29T14:34:43.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.378+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a81e3920 con 0x7f55a80cfc80 2024-06-29T14:34:43.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.378+0000 7f559ffff640 1 -- 172.21.15.59:0/3469506092 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5598035040 con 0x7f55a80cfc80 2024-06-29T14:34:43.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.378+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a81e3ee0 con 0x7f55a80cfc80 2024-06-29T14:34:43.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.379+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a80702a0 con 0x7f55a80cfc80 2024-06-29T14:34:43.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.379+0000 7f559ffff640 1 -- 172.21.15.59:0/3469506092 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f5598041030 con 0x7f55a80cfc80 2024-06-29T14:34:43.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.380+0000 7f559ffff640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5588076480 0x7f5588078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.380+0000 7f559ffff640 1 -- 172.21.15.59:0/3469506092 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 6972+0+0 (secure 0 0 0) 0x7f55980b8680 con 0x7f55a80cfc80 2024-06-29T14:34:43.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.380+0000 7f55a6ffd640 1 -- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5588076480 msgr2=0x7f5588078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:43.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.380+0000 7f55a6ffd640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5588076480 0x7f5588078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:43.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.385+0000 7f559ffff640 1 -- 172.21.15.59:0/3469506092 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5598081080 con 0x7f55a80cfc80 2024-06-29T14:34:43.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.581+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f55a815df20 con 0x7f55a80cfc80 2024-06-29T14:34:43.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.582+0000 7f55a6ffd640 1 -- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5588076480 msgr2=0x7f5588078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:43.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.582+0000 7f55a6ffd640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5588076480 0x7f5588078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:43.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.582+0000 7f559ffff640 1 -- 172.21.15.59:0/3469506092 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f5598085380 con 0x7f55a80cfc80 2024-06-29T14:34:43.579 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:34:43.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5588076480 msgr2=0x7f5588078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:43.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5588076480 0x7f5588078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a80cfc80 msgr2=0x7f55a81e46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a80cfc80 0x7f55a81e46e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5598000c00 tx=0x7f55980038a0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 shutdown_connections 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f5588076480 0x7f5588078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a8140f50 0x7f55a81541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55a80d0090 0x7f55a81578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 --2- 172.21.15.59:0/3469506092 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55a80cfc80 0x7f55a81e46e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.584+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 >> 172.21.15.59:0/3469506092 conn(0x7f55a8073270 msgr2=0x7f55a81415d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.585+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 shutdown_connections 2024-06-29T14:34:43.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.585+0000 7f55add73640 1 -- 172.21.15.59:0/3469506092 wait complete. 2024-06-29T14:34:43.593 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:34:43.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.718+0000 7fb306081640 1 -- 172.21.15.59:0/909452809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2fa0 msgr2=0x7fb3000d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.718+0000 7fb306081640 1 --2- 172.21.15.59:0/909452809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2fa0 0x7fb3000d1520 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009830 tx=0x7fb2e802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.718+0000 7fb306081640 1 -- 172.21.15.59:0/909452809 shutdown_connections 2024-06-29T14:34:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.718+0000 7fb306081640 1 --2- 172.21.15.59:0/909452809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2fa0 0x7fb3000d1520 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.718+0000 7fb306081640 1 --2- 172.21.15.59:0/909452809 >> v2:172.21.15.153:3300/0 conn(0x7fb3000b2270 0x7fb30015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.718+0000 7fb306081640 1 -- 172.21.15.59:0/909452809 >> 172.21.15.59:0/909452809 conn(0x7fb300073e10 msgr2=0x7fb300074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.718+0000 7fb306081640 1 -- 172.21.15.59:0/909452809 shutdown_connections 2024-06-29T14:34:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.718+0000 7fb306081640 1 -- 172.21.15.59:0/909452809 wait complete. 2024-06-29T14:34:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.720+0000 7fb306081640 1 Processor -- start 2024-06-29T14:34:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.720+0000 7fb306081640 1 -- start start 2024-06-29T14:34:43.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb306081640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2270 0x7fb30014d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb306081640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3000b2fa0 0x7fb3000adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb306081640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3000d1a60 0x7fb3000ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb306081640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb3000b6cc0 con 0x7fb3000d1a60 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb306081640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3000b6b40 con 0x7fb3000b2fa0 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb306081640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb3000cfe90 con 0x7fb3000b2270 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb2fffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3000d1a60 0x7fb3000ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb2ff7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2270 0x7fb30014d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb2f6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3000b2fa0 0x7fb3000adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb2fffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3000d1a60 0x7fb3000ae1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49746/0 (socket says 172.21.15.59:49746) 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb2ff7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2270 0x7fb30014d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:34930/0 (socket says 172.21.15.59:34930) 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb2f6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3000b2fa0 0x7fb3000adc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:40714/0 (socket says 172.21.15.59:40714) 2024-06-29T14:34:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.721+0000 7fb2fffff640 1 -- 172.21.15.59:0/2549382000 learned_addr learned my addr 172.21.15.59:0/2549382000 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2ff7fe640 1 -- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3000b2fa0 msgr2=0x7fb3000adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2ff7fe640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3000b2fa0 0x7fb3000adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2ff7fe640 1 -- 172.21.15.59:0/2549382000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3000d1a60 msgr2=0x7fb3000ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2ff7fe640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3000d1a60 0x7fb3000ae1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2ff7fe640 1 -- 172.21.15.59:0/2549382000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3000afe10 con 0x7fb3000b2270 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2fffff640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3000d1a60 0x7fb3000ae1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2ff7fe640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2270 0x7fb30014d6f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000e9f0 tx=0x7fb2f000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2fcff9640 1 -- 172.21.15.59:0/2549382000 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f000cc60 con 0x7fb3000b2270 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2fcff9640 1 -- 172.21.15.59:0/2549382000 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2f000ce00 con 0x7fb3000b2270 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3000b00f0 con 0x7fb3000b2270 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb2fcff9640 1 -- 172.21.15.59:0/2549382000 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0004280 con 0x7fb3000b2270 2024-06-29T14:34:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.722+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3000b06b0 con 0x7fb3000b2270 2024-06-29T14:34:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.723+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3000d0400 con 0x7fb3000b2270 2024-06-29T14:34:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.724+0000 7fb2fcff9640 1 -- 172.21.15.59:0/2549382000 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7fb2f00035f0 con 0x7fb3000b2270 2024-06-29T14:34:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.724+0000 7fb2fcff9640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb2e0076480 0x7fb2e0078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.725+0000 7fb2f6ffd640 1 -- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb2e0076480 msgr2=0x7fb2e0078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.725+0000 7fb2fcff9640 1 -- 172.21.15.59:0/2549382000 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 6972+0+0 (secure 0 0 0) 0x7fb2f00988c0 con 0x7fb3000b2270 2024-06-29T14:34:43.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.725+0000 7fb2f6ffd640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb2e0076480 0x7fb2e0078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:43.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.729+0000 7fb2fcff9640 1 -- 172.21.15.59:0/2549382000 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb2f0061240 con 0x7fb3000b2270 2024-06-29T14:34:43.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.920+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb300158ef0 con 0x7fb3000b2270 2024-06-29T14:34:43.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.921+0000 7fb2fcff9640 1 -- 172.21.15.59:0/2549382000 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7fb2f0065540 con 0x7fb3000b2270 2024-06-29T14:34:43.918 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:34:43.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb2e0076480 msgr2=0x7fb2e0078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:43.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb2e0076480 0x7fb2e0078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2270 msgr2=0x7fb30014d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2270 0x7fb30014d6f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000e9f0 tx=0x7fb2f000eeb0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 shutdown_connections 2024-06-29T14:34:43.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fb2e0076480 0x7fb2e0078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3000d1a60 0x7fb3000ae1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3000b2fa0 0x7fb3000adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 --2- 172.21.15.59:0/2549382000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3000b2270 0x7fb30014d6f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 >> 172.21.15.59:0/2549382000 conn(0x7fb300073e10 msgr2=0x7fb300074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 shutdown_connections 2024-06-29T14:34:43.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:43.923+0000 7fb306081640 1 -- 172.21.15.59:0/2549382000 wait complete. 2024-06-29T14:34:43.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:34:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.057+0000 7f05a8a95640 1 -- 172.21.15.59:0/897302834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40b2270 msgr2=0x7f05a415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.057+0000 7f05a8a95640 1 --2- 172.21.15.59:0/897302834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40b2270 0x7f05a415bc20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f05a40b2d10 tx=0x7f059402a560 comp rx=0 tx=0).stop 2024-06-29T14:34:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.058+0000 7f05a8a95640 1 -- 172.21.15.59:0/897302834 shutdown_connections 2024-06-29T14:34:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.058+0000 7f05a8a95640 1 --2- 172.21.15.59:0/897302834 >> v2:172.21.15.153:3300/0 conn(0x7f05a40d1a60 0x7f05a40cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.058+0000 7f05a8a95640 1 --2- 172.21.15.59:0/897302834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a40b2fa0 0x7f05a40d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.058+0000 7f05a8a95640 1 --2- 172.21.15.59:0/897302834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40b2270 0x7f05a415bc20 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.058+0000 7f05a8a95640 1 -- 172.21.15.59:0/897302834 >> 172.21.15.59:0/897302834 conn(0x7f05a4073e10 msgr2=0x7f05a4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.058+0000 7f05a8a95640 1 -- 172.21.15.59:0/897302834 shutdown_connections 2024-06-29T14:34:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.058+0000 7f05a8a95640 1 -- 172.21.15.59:0/897302834 wait complete. 2024-06-29T14:34:44.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.060+0000 7f05a8a95640 1 Processor -- start 2024-06-29T14:34:44.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.060+0000 7f05a8a95640 1 -- start start 2024-06-29T14:34:44.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a8a95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a40b2270 0x7f05a41f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a8a95640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a40b2fa0 0x7f05a40ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a8a95640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40d1a60 0x7f05a40a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a8a95640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f05a40cfe40 con 0x7f05a40d1a60 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a8a95640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05a40cfcc0 con 0x7f05a40b2270 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a8a95640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f05a40cffc0 con 0x7f05a40b2fa0 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40d1a60 0x7f05a40a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40d1a60 0x7f05a40a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49752/0 (socket says 172.21.15.59:49752) 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a2d9d640 1 -- 172.21.15.59:0/3081710330 learned_addr learned my addr 172.21.15.59:0/3081710330 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a259c640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a40b2270 0x7f05a41f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a1d9b640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a40b2fa0 0x7f05a40ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.061+0000 7f05a2d9d640 1 -- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a40b2fa0 msgr2=0x7f05a40ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f05a2d9d640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a40b2fa0 0x7f05a40ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f05a2d9d640 1 -- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a40b2270 msgr2=0x7f05a41f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f05a2d9d640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a40b2270 0x7f05a41f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f05a2d9d640 1 -- 172.21.15.59:0/3081710330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a40af570 con 0x7f05a40d1a60 2024-06-29T14:34:44.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f05a1d9b640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a40b2fa0 0x7f05a40ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:44.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f05a2d9d640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40d1a60 0x7f05a40a94a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f059c00da10 tx=0x7f059c00ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:44.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f0592ffd640 1 -- 172.21.15.59:0/3081710330 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c017070 con 0x7f05a40d1a60 2024-06-29T14:34:44.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f0592ffd640 1 -- 172.21.15.59:0/3081710330 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f059c010460 con 0x7f05a40d1a60 2024-06-29T14:34:44.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f0592ffd640 1 -- 172.21.15.59:0/3081710330 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0108b0 con 0x7f05a40d1a60 2024-06-29T14:34:44.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a40af850 con 0x7f05a40d1a60 2024-06-29T14:34:44.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.062+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a40afe10 con 0x7f05a40d1a60 2024-06-29T14:34:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.063+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a40af2e0 con 0x7f05a40d1a60 2024-06-29T14:34:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.064+0000 7f0592ffd640 1 -- 172.21.15.59:0/3081710330 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f059c01a070 con 0x7f05a40d1a60 2024-06-29T14:34:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.065+0000 7f0592ffd640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f05840764b0 0x7f05840789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.065+0000 7f05a259c640 1 -- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f05840764b0 msgr2=0x7f05840789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.065+0000 7f0592ffd640 1 -- 172.21.15.59:0/3081710330 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6972+0+0 (secure 0 0 0) 0x7f059c098160 con 0x7f05a40d1a60 2024-06-29T14:34:44.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.065+0000 7f05a259c640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f05840764b0 0x7f05840789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:44.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.069+0000 7f0592ffd640 1 -- 172.21.15.59:0/3081710330 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f059c060d30 con 0x7f05a40d1a60 2024-06-29T14:34:44.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.266+0000 7f05a259c640 1 -- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f05840764b0 msgr2=0x7f05840789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:44.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.266+0000 7f05a259c640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f05840764b0 0x7f05840789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.284+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f05a40d0450 con 0x7f05a40d1a60 2024-06-29T14:34:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.285+0000 7f0592ffd640 1 -- 172.21.15.59:0/3081710330 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v57) ==== 80+0+358 (secure 0 0 0) 0x7f059c065030 con 0x7f05a40d1a60 2024-06-29T14:34:44.282 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1180005807 2024-06-30T14:23:32.829085+0000 2024-06-29T14:34:44.282 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/654263470 2024-06-30T14:23:32.829085+0000 2024-06-29T14:34:44.282 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/654263470 2024-06-30T14:23:32.829085+0000 2024-06-29T14:34:44.282 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2990390183 2024-06-30T14:23:32.829085+0000 2024-06-29T14:34:44.282 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3207465637 2024-06-30T14:23:32.829085+0000 2024-06-29T14:34:44.282 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2185789752 2024-06-30T14:23:32.829085+0000 2024-06-29T14:34:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:listed 6 entries 2024-06-29T14:34:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f05840764b0 msgr2=0x7f05840789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f05840764b0 0x7f05840789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40d1a60 msgr2=0x7f05a40a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40d1a60 0x7f05a40a94a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f059c00da10 tx=0x7f059c00ded0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 shutdown_connections 2024-06-29T14:34:44.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f05840764b0 0x7f05840789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05a40d1a60 0x7f05a40a94a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05a40b2fa0 0x7f05a40ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 --2- 172.21.15.59:0/3081710330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a40b2270 0x7f05a41f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 >> 172.21.15.59:0/3081710330 conn(0x7f05a4073e10 msgr2=0x7f05a41578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 shutdown_connections 2024-06-29T14:34:44.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.287+0000 7f05a8a95640 1 -- 172.21.15.59:0/3081710330 wait complete. 2024-06-29T14:34:44.295 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:34:44.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:44.296 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:34:44.329 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:34:44.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:44.330 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:34:44.386 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:34:44.386 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:34:44.386 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:44.402 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:34:44.403 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:34:44.403 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:34:44.403 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:44.444 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:34:44.445 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:34:44.445 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:34:44.445 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:44.460 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:34:44.460 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.606+0000 7f70df391640 1 -- 172.21.15.59:0/2208886753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2270 msgr2=0x7f70d815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.606+0000 7f70df391640 1 --2- 172.21.15.59:0/2208886753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2270 0x7f70d815bc20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f70d80b2680 tx=0x7f70d402a530 comp rx=0 tx=0).stop 2024-06-29T14:34:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.606+0000 7f70df391640 1 -- 172.21.15.59:0/2208886753 shutdown_connections 2024-06-29T14:34:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.606+0000 7f70df391640 1 --2- 172.21.15.59:0/2208886753 >> v2:172.21.15.153:3300/0 conn(0x7f70d80b2fa0 0x7f70d80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.606+0000 7f70df391640 1 --2- 172.21.15.59:0/2208886753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2270 0x7f70d815bc20 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.606+0000 7f70df391640 1 -- 172.21.15.59:0/2208886753 >> 172.21.15.59:0/2208886753 conn(0x7f70d8073e10 msgr2=0x7f70d8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.606+0000 7f70df391640 1 -- 172.21.15.59:0/2208886753 shutdown_connections 2024-06-29T14:34:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.607+0000 7f70df391640 1 -- 172.21.15.59:0/2208886753 wait complete. 2024-06-29T14:34:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.608+0000 7f70df391640 1 Processor -- start 2024-06-29T14:34:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.608+0000 7f70df391640 1 -- start start 2024-06-29T14:34:44.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70df391640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70d80b2270 0x7f70d80ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70df391640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2fa0 0x7f70d80b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70df391640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d80d1a60 0x7f70d814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70df391640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f70d80b6cc0 con 0x7f70d80b2fa0 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70df391640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f70d80b6b40 con 0x7f70d80d1a60 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70df391640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f70d80cfe90 con 0x7f70d80b2270 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2fa0 0x7f70d80b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2fa0 0x7f70d80b3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49774/0 (socket says 172.21.15.59:49774) 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 -- 172.21.15.59:0/818722209 learned_addr learned my addr 172.21.15.59:0/818722209 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dd12d640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70d80b2270 0x7f70d80ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dd92e640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d80d1a60 0x7f70d814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 -- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70d80b2270 msgr2=0x7f70d80ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70d80b2270 0x7f70d80ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 -- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d80d1a60 msgr2=0x7f70d814cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d80d1a60 0x7f70d814cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 -- 172.21.15.59:0/818722209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d80ae100 con 0x7f70d80b2fa0 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.610+0000 7f70dc92c640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2fa0 0x7f70d80b3690 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00e790 tx=0x7f70cc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.611+0000 7f70c9ffb640 1 -- 172.21.15.59:0/818722209 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc00eea0 con 0x7f70d80b2fa0 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.611+0000 7f70c9ffb640 1 -- 172.21.15.59:0/818722209 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70cc00cdb0 con 0x7f70d80b2fa0 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.611+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d80ae3e0 con 0x7f70d80b2fa0 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.611+0000 7f70c9ffb640 1 -- 172.21.15.59:0/818722209 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc010690 con 0x7f70d80b2fa0 2024-06-29T14:34:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.611+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d80ae9a0 con 0x7f70d80b2fa0 2024-06-29T14:34:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.612+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d80ade70 con 0x7f70d80b2fa0 2024-06-29T14:34:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.613+0000 7f70c9ffb640 1 -- 172.21.15.59:0/818722209 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f70cc010830 con 0x7f70d80b2fa0 2024-06-29T14:34:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.613+0000 7f70c9ffb640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f70bc076480 0x7f70bc078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.613+0000 7f70dd12d640 1 -- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f70bc076480 msgr2=0x7f70bc078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.613+0000 7f70c9ffb640 1 -- 172.21.15.59:0/818722209 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6984+0+0 (secure 0 0 0) 0x7f70cc098920 con 0x7f70d80b2fa0 2024-06-29T14:34:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.614+0000 7f70dd12d640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f70bc076480 0x7f70bc078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:44.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.618+0000 7f70c9ffb640 1 -- 172.21.15.59:0/818722209 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f70cc0613c0 con 0x7f70d80b2fa0 2024-06-29T14:34:44.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.815+0000 7f70dd12d640 1 -- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f70bc076480 msgr2=0x7f70bc078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:44.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.815+0000 7f70dd12d640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f70bc076480 0x7f70bc078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:44.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.852+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70d80d03b0 con 0x7f70d80b2fa0 2024-06-29T14:34:44.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.852+0000 7f70c9ffb640 1 -- 172.21.15.59:0/818722209 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5223 (secure 0 0 0) 0x7f70cc0656c0 con 0x7f70d80b2fa0 2024-06-29T14:34:44.850 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:44.850 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"btime":"2024-06-29T14:23:19:215604+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/634859216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":634859216},{"type":"v1","addr":"172.21.15.153:6837","nonce":634859216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3661382082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3661382082},{"type":"v1","addr":"172.21.15.153:6836","nonce":3661382082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4012387069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4012387069},{"type":"v1","addr":"172.21.15.59:6833","nonce":4012387069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:15.579309+0000","modified":"2024-06-29T14:23:18.607053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5469},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5469":{"gid":5469,"name":"a","rank":0,"incarnation":15,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/812277010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":812277010},{"type":"v1","addr":"172.21.15.59:6835","nonce":812277010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5469,"qdb_cluster":[5469]},"id":2}]} 2024-06-29T14:34:44.850 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 19 2024-06-29T14:34:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f70bc076480 msgr2=0x7f70bc078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f70bc076480 0x7f70bc078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2fa0 msgr2=0x7f70d80b3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2fa0 0x7f70d80b3690 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00e790 tx=0x7f70cc00ec50 comp rx=0 tx=0).stop 2024-06-29T14:34:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 shutdown_connections 2024-06-29T14:34:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f70bc076480 0x7f70bc078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d80d1a60 0x7f70d814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70d80b2fa0 0x7f70d80b3690 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 --2- 172.21.15.59:0/818722209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f70d80b2270 0x7f70d80ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 >> 172.21.15.59:0/818722209 conn(0x7f70d8073e10 msgr2=0x7f70d80d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 shutdown_connections 2024-06-29T14:34:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.855+0000 7f70df391640 1 -- 172.21.15.59:0/818722209 wait complete. 2024-06-29T14:34:44.863 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:44.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 -- 172.21.15.59:0/3930961131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 msgr2=0x7f254415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 --2- 172.21.15.59:0/3930961131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 0x7f254415bc20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f25440b2680 tx=0x7f253402a560 comp rx=0 tx=0).stop 2024-06-29T14:34:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 -- 172.21.15.59:0/3930961131 shutdown_connections 2024-06-29T14:34:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 --2- 172.21.15.59:0/3930961131 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25440d1a60 0x7f25440cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 --2- 172.21.15.59:0/3930961131 >> v2:172.21.15.153:3300/0 conn(0x7f25440b2fa0 0x7f25440d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 --2- 172.21.15.59:0/3930961131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 0x7f254415bc20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 -- 172.21.15.59:0/3930961131 >> 172.21.15.59:0/3930961131 conn(0x7f2544073e10 msgr2=0x7f2544074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 -- 172.21.15.59:0/3930961131 shutdown_connections 2024-06-29T14:34:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.985+0000 7f2549de1640 1 -- 172.21.15.59:0/3930961131 wait complete. 2024-06-29T14:34:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.986+0000 7f2549de1640 1 Processor -- start 2024-06-29T14:34:44.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.987+0000 7f2549de1640 1 -- start start 2024-06-29T14:34:44.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2549de1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 0x7f254414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2549de1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25440b2fa0 0x7f254415e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2549de1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25440d1a60 0x7f25440ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2549de1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f25440cfe40 con 0x7f25440b2270 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2549de1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f25440cfcc0 con 0x7f25440d1a60 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2549de1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f25440cffc0 con 0x7f25440b2fa0 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f25437fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 0x7f254414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f25437fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 0x7f254414b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49800/0 (socket says 172.21.15.59:49800) 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2542ffd640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25440b2fa0 0x7f254415e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f25437fe640 1 -- 172.21.15.59:0/876421523 learned_addr learned my addr 172.21.15.59:0/876421523 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2543fff640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25440d1a60 0x7f25440ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2542ffd640 1 -- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25440d1a60 msgr2=0x7f25440ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2542ffd640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25440d1a60 0x7f25440ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2542ffd640 1 -- 172.21.15.59:0/876421523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 msgr2=0x7f254414b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.988+0000 7f2542ffd640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 0x7f254414b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.989+0000 7f2542ffd640 1 -- 172.21.15.59:0/876421523 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25440aaa90 con 0x7f25440b2fa0 2024-06-29T14:34:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.989+0000 7f2542ffd640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25440b2fa0 0x7f254415e6e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f253800e6d0 tx=0x7f253800eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.989+0000 7f252bfff640 1 -- 172.21.15.59:0/876421523 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253800cab0 con 0x7f25440b2fa0 2024-06-29T14:34:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.989+0000 7f252bfff640 1 -- 172.21.15.59:0/876421523 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f253800cd40 con 0x7f25440b2fa0 2024-06-29T14:34:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.989+0000 7f252bfff640 1 -- 172.21.15.59:0/876421523 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538010690 con 0x7f25440b2fa0 2024-06-29T14:34:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.989+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25440aad70 con 0x7f25440b2fa0 2024-06-29T14:34:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.989+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25440ab330 con 0x7f25440b2fa0 2024-06-29T14:34:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.990+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25440aa800 con 0x7f25440b2fa0 2024-06-29T14:34:44.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.991+0000 7f252bfff640 1 -- 172.21.15.59:0/876421523 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f2538010830 con 0x7f25440b2fa0 2024-06-29T14:34:44.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.991+0000 7f252bfff640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2530076480 0x7f2530078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.991+0000 7f25437fe640 1 -- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2530076480 msgr2=0x7f2530078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:44.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.991+0000 7f252bfff640 1 -- 172.21.15.59:0/876421523 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 6984+0+0 (secure 0 0 0) 0x7f25380988b0 con 0x7f25440b2fa0 2024-06-29T14:34:44.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.991+0000 7f25437fe640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2530076480 0x7f2530078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:44.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:44.996+0000 7f252bfff640 1 -- 172.21.15.59:0/876421523 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2538061360 con 0x7f25440b2fa0 2024-06-29T14:34:45.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.189+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25440d0450 con 0x7f25440b2fa0 2024-06-29T14:34:45.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.191+0000 7f252bfff640 1 -- 172.21.15.59:0/876421523 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+18311 (secure 0 0 0) 0x7f2538065660 con 0x7f25440b2fa0 2024-06-29T14:34:45.188 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:45.189 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":57,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:34:44.186507+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:13.422166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:14.527412+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1180005807":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6832/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6833/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/2990390183":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/3207465637":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/2185789752":"2024-06-30T14:23:32.829085+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":5,"length":3},{"begin":9,"length":1},{"begin":11,"length":3},{"begin":15,"length":3},{"begin":19,"length":1},{"begin":21,"length":4},{"begin":26,"length":3},{"begin":30,"length":1},{"begin":33,"length":3},{"begin":37,"length":3},{"begin":41,"length":1},{"begin":43,"length":3},{"begin":47,"length":3},{"begin":51,"length":2},{"begin":54,"length":3},{"begin":58,"length":3},{"begin":63,"length":1},{"begin":65,"length":3},{"begin":69,"length":4},{"begin":74,"length":1},{"begin":76,"length":3},{"begin":80,"length":4},{"begin":85,"length":1},{"begin":87,"length":3},{"begin":91,"length":4},{"begin":96,"length":1},{"begin":98,"length":3},{"begin":103,"length":3},{"begin":107,"length":1},{"begin":109,"length":3},{"begin":145,"length":77}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":105,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:45.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.192+0000 7f25437fe640 1 -- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2530076480 msgr2=0x7f2530078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:45.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.192+0000 7f25437fe640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2530076480 0x7f2530078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2530076480 msgr2=0x7f2530078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2530076480 0x7f2530078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25440b2fa0 msgr2=0x7f254415e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25440b2fa0 0x7f254415e6e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f253800e6d0 tx=0x7f253800eb90 comp rx=0 tx=0).stop 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 shutdown_connections 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f2530076480 0x7f2530078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25440d1a60 0x7f25440ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25440b2fa0 0x7f254415e6e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 --2- 172.21.15.59:0/876421523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25440b2270 0x7f254414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 >> 172.21.15.59:0/876421523 conn(0x7f2544073e10 msgr2=0x7f2544074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 shutdown_connections 2024-06-29T14:34:45.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.193+0000 7f2549de1640 1 -- 172.21.15.59:0/876421523 wait complete. 2024-06-29T14:34:45.202 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:23:13.422166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:23:14.527412+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 221, 'snap_epoch': 57, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:34:45.202 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:34:45.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.327+0000 7fee417c7640 1 -- 172.21.15.59:0/2509239952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c0b3ed0 msgr2=0x7fee3c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.327+0000 7fee417c7640 1 --2- 172.21.15.59:0/2509239952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c0b3ed0 0x7fee3c15e8d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fee30009830 tx=0x7fee3002a510 comp rx=0 tx=0).stop 2024-06-29T14:34:45.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.327+0000 7fee417c7640 1 -- 172.21.15.59:0/2509239952 shutdown_connections 2024-06-29T14:34:45.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.327+0000 7fee417c7640 1 --2- 172.21.15.59:0/2509239952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c0b3ed0 0x7fee3c15e8d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.327+0000 7fee417c7640 1 --2- 172.21.15.59:0/2509239952 >> v2:172.21.15.153:3300/0 conn(0x7fee3c0d23c0 0x7fee3c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.327+0000 7fee417c7640 1 -- 172.21.15.59:0/2509239952 >> 172.21.15.59:0/2509239952 conn(0x7fee3c0d3ee0 msgr2=0x7fee3c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.327+0000 7fee417c7640 1 -- 172.21.15.59:0/2509239952 shutdown_connections 2024-06-29T14:34:45.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.327+0000 7fee417c7640 1 -- 172.21.15.59:0/2509239952 wait complete. 2024-06-29T14:34:45.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.329+0000 7fee417c7640 1 Processor -- start 2024-06-29T14:34:45.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.329+0000 7fee417c7640 1 -- start start 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.330+0000 7fee417c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3c0b3ed0 0x7fee3c150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.330+0000 7fee417c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee3c0d23c0 0x7fee3c14f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.330+0000 7fee417c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c15ee10 0x7fee3c154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.330+0000 7fee417c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c160240 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.330+0000 7fee417c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c1600c0 con 0x7fee3c0d23c0 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.330+0000 7fee417c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fee3c1603c0 con 0x7fee3c15ee10 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.330+0000 7fee3affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3c0b3ed0 0x7fee3c150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3c0b3ed0 0x7fee3c150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49816/0 (socket says 172.21.15.59:49816) 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3affd640 1 -- 172.21.15.59:0/276582470 learned_addr learned my addr 172.21.15.59:0/276582470 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3a7fc640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee3c0d23c0 0x7fee3c14f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3b7fe640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c15ee10 0x7fee3c154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3affd640 1 -- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c15ee10 msgr2=0x7fee3c154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3affd640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c15ee10 0x7fee3c154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3affd640 1 -- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee3c0d23c0 msgr2=0x7fee3c14f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3affd640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee3c0d23c0 0x7fee3c14f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3affd640 1 -- 172.21.15.59:0/276582470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3c1517e0 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3b7fe640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c15ee10 0x7fee3c154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3a7fc640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee3c0d23c0 0x7fee3c14f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee3affd640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3c0b3ed0 0x7fee3c150ee0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009fd0 tx=0x7fee2c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee237fe640 1 -- 172.21.15.59:0/276582470 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00ccc0 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee237fe640 1 -- 172.21.15.59:0/276582470 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee2c00ee70 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3c151ac0 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.331+0000 7fee237fe640 1 -- 172.21.15.59:0/276582470 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00ce60 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.332+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3c152080 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.333+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3c069de0 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.333+0000 7fee237fe640 1 -- 172.21.15.59:0/276582470 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7fee2c004280 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.334+0000 7fee237fe640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fee28076480 0x7fee28078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.334+0000 7fee237fe640 1 -- 172.21.15.59:0/276582470 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6984+0+0 (secure 0 0 0) 0x7fee2c0988e0 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.334+0000 7fee3a7fc640 1 -- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fee28076480 msgr2=0x7fee28078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.334+0000 7fee3a7fc640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fee28076480 0x7fee28078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:45.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.343+0000 7fee237fe640 1 -- 172.21.15.59:0/276582470 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fee3c069de0 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.535+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee3c0b5320 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.536+0000 7fee237fe640 1 -- 172.21.15.59:0/276582470 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7fee2c061250 con 0x7fee3c0b3ed0 2024-06-29T14:34:45.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.536+0000 7fee3a7fc640 1 -- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fee28076480 msgr2=0x7fee28078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:45.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.536+0000 7fee3a7fc640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fee28076480 0x7fee28078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:45.533 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:34:45.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fee28076480 msgr2=0x7fee28078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:45.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fee28076480 0x7fee28078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3c0b3ed0 msgr2=0x7fee3c150ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3c0b3ed0 0x7fee3c150ee0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009fd0 tx=0x7fee2c00ec20 comp rx=0 tx=0).stop 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 shutdown_connections 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7fee28076480 0x7fee28078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee3c15ee10 0x7fee3c154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee3c0d23c0 0x7fee3c14f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 --2- 172.21.15.59:0/276582470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3c0b3ed0 0x7fee3c150ee0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 >> 172.21.15.59:0/276582470 conn(0x7fee3c0d3ee0 msgr2=0x7fee3c156b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 shutdown_connections 2024-06-29T14:34:45.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.538+0000 7fee417c7640 1 -- 172.21.15.59:0/276582470 wait complete. 2024-06-29T14:34:45.546 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:34:45.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.671+0000 7f88a9743640 1 -- 172.21.15.59:0/2615986685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 msgr2=0x7f888c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.671+0000 7f88a9743640 1 --2- 172.21.15.59:0/2615986685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 0x7f888c00b9c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f889c00e3d0 tx=0x7f888c005830 comp rx=0 tx=0).stop 2024-06-29T14:34:45.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.671+0000 7f88a37fe640 1 -- 172.21.15.59:0/2615986685 reap_dead start 2024-06-29T14:34:45.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.671+0000 7f88a9743640 1 -- 172.21.15.59:0/2615986685 shutdown_connections 2024-06-29T14:34:45.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.671+0000 7f88a9743640 1 -- 172.21.15.59:0/2615986685 >> 172.21.15.59:0/2615986685 conn(0x7f88a40b6fb0 msgr2=0x7f88a40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.671+0000 7f88a9743640 1 -- 172.21.15.59:0/2615986685 shutdown_connections 2024-06-29T14:34:45.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.671+0000 7f88a9743640 1 -- 172.21.15.59:0/2615986685 wait complete. 2024-06-29T14:34:45.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.673+0000 7f88a9743640 1 Processor -- start 2024-06-29T14:34:45.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.673+0000 7f88a9743640 1 -- start start 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.674+0000 7f88a9743640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 0x7f88a4139580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.674+0000 7f88a9743640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88a40b6670 0x7f88a41f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.674+0000 7f88a9743640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88a41f6480 0x7f88a41f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.674+0000 7f88a9743640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f88a40d0400 con 0x7f888c005150 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.674+0000 7f88a9743640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88a40d22c0 con 0x7f88a41f6480 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.674+0000 7f88a9743640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f88a40d0580 con 0x7f88a40b6670 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.674+0000 7f88a2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 0x7f88a4139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 0x7f88a4139580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49840/0 (socket says 172.21.15.59:49840) 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a2ffd640 1 -- 172.21.15.59:0/4101861347 learned_addr learned my addr 172.21.15.59:0/4101861347 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:45.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a27fc640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88a40b6670 0x7f88a41f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a37fe640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88a41f6480 0x7f88a41f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a2ffd640 1 -- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88a40b6670 msgr2=0x7f88a41f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a2ffd640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88a40b6670 0x7f88a41f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a2ffd640 1 -- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88a41f6480 msgr2=0x7f88a41f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a2ffd640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88a41f6480 0x7f88a41f4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a2ffd640 1 -- 172.21.15.59:0/4101861347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a415f1a0 con 0x7f888c005150 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a2ffd640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 0x7f88a4139580 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8894006680 tx=0x7f88940039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f888b7fe640 1 -- 172.21.15.59:0/4101861347 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894015070 con 0x7f888c005150 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f888b7fe640 1 -- 172.21.15.59:0/4101861347 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8894011860 con 0x7f888c005150 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a415f480 con 0x7f888c005150 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f888b7fe640 1 -- 172.21.15.59:0/4101861347 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894011cb0 con 0x7f888c005150 2024-06-29T14:34:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.675+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a41f5580 con 0x7f888c005150 2024-06-29T14:34:45.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.676+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a415f6c0 con 0x7f888c005150 2024-06-29T14:34:45.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.678+0000 7f888b7fe640 1 -- 172.21.15.59:0/4101861347 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f8894018070 con 0x7f888c005150 2024-06-29T14:34:45.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.678+0000 7f888b7fe640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88900764b0 0x7f88900789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.678+0000 7f888b7fe640 1 -- 172.21.15.59:0/4101861347 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6984+0+0 (secure 0 0 0) 0x7f8894098d20 con 0x7f888c005150 2024-06-29T14:34:45.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.678+0000 7f88a27fc640 1 -- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88900764b0 msgr2=0x7f88900789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:45.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.678+0000 7f88a27fc640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88900764b0 0x7f88900789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:45.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.683+0000 7f888b7fe640 1 -- 172.21.15.59:0/4101861347 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f88a415f6c0 con 0x7f888c005150 2024-06-29T14:34:45.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.877+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f88a413fda0 con 0x7f888c005150 2024-06-29T14:34:45.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.877+0000 7f888b7fe640 1 -- 172.21.15.59:0/4101861347 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7f8894061720 con 0x7f888c005150 2024-06-29T14:34:45.875 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:34:45.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.879+0000 7f88a27fc640 1 -- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88900764b0 msgr2=0x7f88900789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3430128047 2024-06-29T14:34:45.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.879+0000 7f88a27fc640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88900764b0 0x7f88900789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:45.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88900764b0 msgr2=0x7f88900789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:45.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88900764b0 0x7f88900789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 msgr2=0x7f88a4139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 0x7f88a4139580 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8894006680 tx=0x7f88940039c0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 shutdown_connections 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:6832/3430128047,v1:172.21.15.153:6833/3430128047] conn(0x7f88900764b0 0x7f88900789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88a41f6480 0x7f88a41f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88a40b6670 0x7f88a41f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 --2- 172.21.15.59:0/4101861347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f888c005150 0x7f88a4139580 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 >> 172.21.15.59:0/4101861347 conn(0x7f88a40b6fb0 msgr2=0x7f88a41415e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 shutdown_connections 2024-06-29T14:34:45.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:45.880+0000 7f88a9743640 1 -- 172.21.15.59:0/4101861347 wait complete. 2024-06-29T14:34:45.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 -- 172.21.15.59:0/2454602939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f194006eb40 msgr2=0x7f19401443e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 --2- 172.21.15.59:0/2454602939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f194006eb40 0x7f19401443e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f19400b3b90 tx=0x7f193402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 -- 172.21.15.59:0/2454602939 shutdown_connections 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 --2- 172.21.15.59:0/2454602939 >> v2:172.21.15.153:3300/0 conn(0x7f19400b5320 0x7f19400b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 --2- 172.21.15.59:0/2454602939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f194006eb40 0x7f19401443e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 --2- 172.21.15.59:0/2454602939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1940071620 0x7f194015bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 -- 172.21.15.59:0/2454602939 >> 172.21.15.59:0/2454602939 conn(0x7f1940073e50 msgr2=0x7f19400b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 -- 172.21.15.59:0/2454602939 shutdown_connections 2024-06-29T14:34:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.013+0000 7f194763d640 1 -- 172.21.15.59:0/2454602939 wait complete. 2024-06-29T14:34:46.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.015+0000 7f194763d640 1 Processor -- start 2024-06-29T14:34:46.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.015+0000 7f194763d640 1 -- start start 2024-06-29T14:34:46.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.017+0000 7f194763d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f194006eb40 0x7f1940163610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.018+0000 7f194763d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1940071620 0x7f1940159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.018+0000 7f194763d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19400b5320 0x7f1940142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.018+0000 7f194763d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f194015ebd0 con 0x7f19400b5320 2024-06-29T14:34:46.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.018+0000 7f194763d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f194015ea50 con 0x7f1940071620 2024-06-29T14:34:46.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.018+0000 7f194763d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f194015ed50 con 0x7f194006eb40 2024-06-29T14:34:46.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.020+0000 7f1945bda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19400b5320 0x7f1940142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.020+0000 7f1945bda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19400b5320 0x7f1940142700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49866/0 (socket says 172.21.15.59:49866) 2024-06-29T14:34:46.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.020+0000 7f1945bda640 1 -- 172.21.15.59:0/11802529 learned_addr learned my addr 172.21.15.59:0/11802529 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:46.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.020+0000 7f1944bd8640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1940071620 0x7f1940159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.021+0000 7f19453d9640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f194006eb40 0x7f1940163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.021+0000 7f1945bda640 1 -- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f194006eb40 msgr2=0x7f1940163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.021+0000 7f1945bda640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f194006eb40 0x7f1940163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.021+0000 7f1945bda640 1 -- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1940071620 msgr2=0x7f1940159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.021+0000 7f1945bda640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1940071620 0x7f1940159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.021+0000 7f1945bda640 1 -- 172.21.15.59:0/11802529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1940163b50 con 0x7f19400b5320 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.021+0000 7f19453d9640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f194006eb40 0x7f1940163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.021+0000 7f1945bda640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19400b5320 0x7f1940142700 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f193800ec20 tx=0x7f193800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.022+0000 7f1931ffb640 1 -- 172.21.15.59:0/11802529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1938003b70 con 0x7f19400b5320 2024-06-29T14:34:46.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.022+0000 7f1931ffb640 1 -- 172.21.15.59:0/11802529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1938003e00 con 0x7f19400b5320 2024-06-29T14:34:46.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.022+0000 7f1931ffb640 1 -- 172.21.15.59:0/11802529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1938004280 con 0x7f19400b5320 2024-06-29T14:34:46.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.022+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1940141660 con 0x7f19400b5320 2024-06-29T14:34:46.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.022+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1940141c20 con 0x7f19400b5320 2024-06-29T14:34:46.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.023+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194015bcd0 con 0x7f19400b5320 2024-06-29T14:34:46.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.024+0000 7f1931ffb640 1 -- 172.21.15.59:0/11802529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) ==== 97912+0+0 (secure 0 0 0) 0x7f1938004420 con 0x7f19400b5320 2024-06-29T14:34:46.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.024+0000 7f1931ffb640 1 -- 172.21.15.59:0/11802529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7299+0+0 (secure 0 0 0) 0x7f1938098540 con 0x7f19400b5320 2024-06-29T14:34:46.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.029+0000 7f1931ffb640 1 -- 172.21.15.59:0/11802529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1938061cb0 con 0x7f19400b5320 2024-06-29T14:34:46.067 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.068+0000 7f6575e88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.067 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.068+0000 7f6575e88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.067 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.068+0000 7f6575e88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.067 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.068+0000 7f6575e88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.068 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.068+0000 7f6575e88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.068 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.069+0000 7f657167f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.068 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.069+0000 7f657167f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.068 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.069+0000 7f657167f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.068 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.069+0000 7f657167f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.068 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:34:46.069+0000 7f657167f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:46.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.223+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f19400cec90 con 0x7f19400b5320 2024-06-29T14:34:46.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.223+0000 7f1931ffb640 1 -- 172.21.15.59:0/11802529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) ==== 100+0+11 (secure 0 0 0) 0x7f19380615b0 con 0x7f19400b5320 2024-06-29T14:34:46.221 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:34:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.225+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19400b5320 msgr2=0x7f1940142700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.226+0000 7f194763d640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19400b5320 0x7f1940142700 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f193800ec20 tx=0x7f193800c6a0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.226+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 shutdown_connections 2024-06-29T14:34:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.226+0000 7f194763d640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19400b5320 0x7f1940142700 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.226+0000 7f194763d640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1940071620 0x7f1940159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.226+0000 7f194763d640 1 --2- 172.21.15.59:0/11802529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f194006eb40 0x7f1940163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.226+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 >> 172.21.15.59:0/11802529 conn(0x7f1940073e50 msgr2=0x7f19400cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.226+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 shutdown_connections 2024-06-29T14:34:46.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.226+0000 7f194763d640 1 -- 172.21.15.59:0/11802529 wait complete. 2024-06-29T14:34:46.234 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:46.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.356+0000 7fdb02c26640 1 -- 172.21.15.59:0/4063781783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae4004070 msgr2=0x7fdae4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.356+0000 7fdb02c26640 1 --2- 172.21.15.59:0/4063781783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae4004070 0x7fdae4004800 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00016c0 tx=0x7fdae4005830 comp rx=0 tx=0).stop 2024-06-29T14:34:46.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.356+0000 7fdb011c3640 1 -- 172.21.15.59:0/4063781783 reap_dead start 2024-06-29T14:34:46.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.356+0000 7fdb02c26640 1 -- 172.21.15.59:0/4063781783 shutdown_connections 2024-06-29T14:34:46.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.356+0000 7fdb02c26640 1 -- 172.21.15.59:0/4063781783 >> 172.21.15.59:0/4063781783 conn(0x7fdafc0b5610 msgr2=0x7fdafc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.356+0000 7fdb02c26640 1 -- 172.21.15.59:0/4063781783 shutdown_connections 2024-06-29T14:34:46.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.356+0000 7fdb02c26640 1 -- 172.21.15.59:0/4063781783 wait complete. 2024-06-29T14:34:46.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.357+0000 7fdb02c26640 1 Processor -- start 2024-06-29T14:34:46.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.357+0000 7fdb02c26640 1 -- start start 2024-06-29T14:34:46.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb02c26640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae4004070 0x7fdafc1595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb02c26640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdafc159af0 0x7fdafc1f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb02c26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdafc1f5520 0x7fdafc1f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb02c26640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc0cf080 con 0x7fdafc159af0 2024-06-29T14:34:46.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb02c26640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc0b8f00 con 0x7fdafc1f5520 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb02c26640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdafc0cf200 con 0x7fdae4004070 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdafbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdafc159af0 0x7fdafc1f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdafbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdafc159af0 0x7fdafc1f4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49892/0 (socket says 172.21.15.59:49892) 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdafbfff640 1 -- 172.21.15.59:0/921654950 learned_addr learned my addr 172.21.15.59:0/921654950 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb009c2640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae4004070 0x7fdafc1595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb011c3640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdafc1f5520 0x7fdafc1f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdafbfff640 1 -- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae4004070 msgr2=0x7fdafc1595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdafbfff640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae4004070 0x7fdafc1595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdafbfff640 1 -- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdafc1f5520 msgr2=0x7fdafc1f85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdafbfff640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdafc1f5520 0x7fdafc1f85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdafbfff640 1 -- 172.21.15.59:0/921654950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdafc158510 con 0x7fdafc159af0 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.359+0000 7fdb011c3640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdafc1f5520 0x7fdafc1f85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:46.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.360+0000 7fdafbfff640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdafc159af0 0x7fdafc1f4fe0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00017e0 tx=0x7fdaf0006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.360+0000 7fdaf97fa640 1 -- 172.21.15.59:0/921654950 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0041070 con 0x7fdafc159af0 2024-06-29T14:34:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.360+0000 7fdaf97fa640 1 -- 172.21.15.59:0/921654950 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdaf0003af0 con 0x7fdafc159af0 2024-06-29T14:34:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.360+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdafc1587f0 con 0x7fdafc159af0 2024-06-29T14:34:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.360+0000 7fdaf97fa640 1 -- 172.21.15.59:0/921654950 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0038070 con 0x7fdafc159af0 2024-06-29T14:34:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.360+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdafc158db0 con 0x7fdafc159af0 2024-06-29T14:34:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.361+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdafc0d1a20 con 0x7fdafc159af0 2024-06-29T14:34:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.361+0000 7fdaf97fa640 1 -- 172.21.15.59:0/921654950 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) ==== 97912+0+0 (secure 0 0 0) 0x7fdaf0030070 con 0x7fdafc159af0 2024-06-29T14:34:46.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.362+0000 7fdaf97fa640 1 -- 172.21.15.59:0/921654950 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7299+0+0 (secure 0 0 0) 0x7fdaf00b6c10 con 0x7fdafc159af0 2024-06-29T14:34:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.366+0000 7fdaf97fa640 1 -- 172.21.15.59:0/921654950 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdaf00803b0 con 0x7fdafc159af0 2024-06-29T14:34:46.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.594+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdafc15e970 con 0x7fdafc159af0 2024-06-29T14:34:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.595+0000 7fdaf97fa640 1 -- 172.21.15.59:0/921654950 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5223 (secure 0 0 0) 0x7fdaf007fcb0 con 0x7fdafc159af0 2024-06-29T14:34:46.592 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:46.592 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"btime":"2024-06-29T14:23:19:215604+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/634859216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":634859216},{"type":"v1","addr":"172.21.15.153:6837","nonce":634859216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3661382082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3661382082},{"type":"v1","addr":"172.21.15.153:6836","nonce":3661382082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4012387069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4012387069},{"type":"v1","addr":"172.21.15.59:6833","nonce":4012387069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:15.579309+0000","modified":"2024-06-29T14:23:18.607053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5469},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5469":{"gid":5469,"name":"a","rank":0,"incarnation":15,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/812277010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":812277010},{"type":"v1","addr":"172.21.15.59:6835","nonce":812277010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5469,"qdb_cluster":[5469]},"id":2}]} 2024-06-29T14:34:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 19 2024-06-29T14:34:46.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdafc159af0 msgr2=0x7fdafc1f4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdafc159af0 0x7fdafc1f4fe0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00017e0 tx=0x7fdaf0006360 comp rx=0 tx=0).stop 2024-06-29T14:34:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 shutdown_connections 2024-06-29T14:34:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdafc1f5520 0x7fdafc1f85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdafc159af0 0x7fdafc1f4fe0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 --2- 172.21.15.59:0/921654950 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae4004070 0x7fdafc1595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 >> 172.21.15.59:0/921654950 conn(0x7fdafc0b5610 msgr2=0x7fdafc0b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 shutdown_connections 2024-06-29T14:34:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.597+0000 7fdb02c26640 1 -- 172.21.15.59:0/921654950 wait complete. 2024-06-29T14:34:46.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.726+0000 7f91bb1d4640 1 -- 172.21.15.59:0/4266224889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a4003a70 msgr2=0x7f91a4003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.726+0000 7f91bb1d4640 1 --2- 172.21.15.59:0/4266224889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a4003a70 0x7f91a4003ee0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009b20 tx=0x7f91a4005830 comp rx=0 tx=0).stop 2024-06-29T14:34:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.726+0000 7f91b9771640 1 -- 172.21.15.59:0/4266224889 reap_dead start 2024-06-29T14:34:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.726+0000 7f91bb1d4640 1 -- 172.21.15.59:0/4266224889 shutdown_connections 2024-06-29T14:34:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.726+0000 7f91bb1d4640 1 -- 172.21.15.59:0/4266224889 >> 172.21.15.59:0/4266224889 conn(0x7f91b40d3ee0 msgr2=0x7f91b40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.726+0000 7f91bb1d4640 1 -- 172.21.15.59:0/4266224889 shutdown_connections 2024-06-29T14:34:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.726+0000 7f91bb1d4640 1 -- 172.21.15.59:0/4266224889 wait complete. 2024-06-29T14:34:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.728+0000 7f91bb1d4640 1 Processor -- start 2024-06-29T14:34:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.728+0000 7f91bb1d4640 1 -- start start 2024-06-29T14:34:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91bb1d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a4003a70 0x7f91b415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91bb1d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91b41f4d90 0x7f91b41f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91bb1d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91b41466a0 0x7f91b4146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91bb1d4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f91b40b3d20 con 0x7f91b41f4d90 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91bb1d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f91b40b3ba0 con 0x7f91a4003a70 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91bb1d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f91b415fef0 con 0x7f91b41466a0 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91b41f4d90 0x7f91b41f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91b41f4d90 0x7f91b41f51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49912/0 (socket says 172.21.15.59:49912) 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 -- 172.21.15.59:0/1593542394 learned_addr learned my addr 172.21.15.59:0/1593542394 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b9771640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91b41466a0 0x7f91b4146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b8f70640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a4003a70 0x7f91b415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 -- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91b41466a0 msgr2=0x7f91b4146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91b41466a0 0x7f91b4146ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 -- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a4003a70 msgr2=0x7f91b415ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a4003a70 0x7f91b415ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 -- 172.21.15.59:0/1593542394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b415d390 con 0x7f91b41f4d90 2024-06-29T14:34:46.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b9771640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91b41466a0 0x7f91b4146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b8f70640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a4003a70 0x7f91b415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b3fff640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91b41f4d90 0x7f91b41f51a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f91a0006630 tx=0x7f91a00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b17fa640 1 -- 172.21.15.59:0/1593542394 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0015070 con 0x7f91b41f4d90 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.730+0000 7f91b17fa640 1 -- 172.21.15.59:0/1593542394 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91a0011920 con 0x7f91b41f4d90 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.731+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b415d670 con 0x7f91b41f4d90 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.731+0000 7f91b17fa640 1 -- 172.21.15.59:0/1593542394 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0011d70 con 0x7f91b41f4d90 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.731+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b415dc30 con 0x7f91b41f4d90 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.731+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b4050560 con 0x7f91b41f4d90 2024-06-29T14:34:46.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.732+0000 7f91b17fa640 1 -- 172.21.15.59:0/1593542394 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) ==== 97912+0+0 (secure 0 0 0) 0x7f91a0018070 con 0x7f91b41f4d90 2024-06-29T14:34:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.733+0000 7f91b17fa640 1 -- 172.21.15.59:0/1593542394 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7299+0+0 (secure 0 0 0) 0x7f91a0098540 con 0x7f91b41f4d90 2024-06-29T14:34:46.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.737+0000 7f91b17fa640 1 -- 172.21.15.59:0/1593542394 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f91b4050560 con 0x7f91b41f4d90 2024-06-29T14:34:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.929+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91b4160c10 con 0x7f91b41f4d90 2024-06-29T14:34:46.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.930+0000 7f91b17fa640 1 -- 172.21.15.59:0/1593542394 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+18770 (secure 0 0 0) 0x7f91b40b42e0 con 0x7f91b41f4d90 2024-06-29T14:34:46.927 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:46.928 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":58,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:34:46.010680+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:13.422166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:14.527412+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1169887706":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/2575712432":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3207465637":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/2990390183":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6833/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/1180005807":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/779704172":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/1179750521":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3939656574":"2024-06-30T14:34:46.010641+0000","172.21.15.153:6832/3430128047":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/2185789752":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6832/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/4049145115":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3509841566":"2024-06-30T14:34:46.010641+0000","172.21.15.153:6833/3430128047":"2024-06-30T14:34:46.010641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":5,"length":3},{"begin":9,"length":1},{"begin":11,"length":3},{"begin":15,"length":3},{"begin":19,"length":1},{"begin":21,"length":4},{"begin":26,"length":3},{"begin":30,"length":1},{"begin":33,"length":3},{"begin":37,"length":3},{"begin":41,"length":1},{"begin":43,"length":3},{"begin":47,"length":3},{"begin":51,"length":2},{"begin":54,"length":3},{"begin":58,"length":3},{"begin":63,"length":1},{"begin":65,"length":3},{"begin":69,"length":4},{"begin":74,"length":1},{"begin":76,"length":3},{"begin":80,"length":4},{"begin":85,"length":1},{"begin":87,"length":3},{"begin":91,"length":4},{"begin":96,"length":1},{"begin":98,"length":3},{"begin":103,"length":3},{"begin":107,"length":1},{"begin":109,"length":3},{"begin":145,"length":77}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91b41f4d90 msgr2=0x7f91b41f51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91b41f4d90 0x7f91b41f51a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f91a0006630 tx=0x7f91a00039c0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 shutdown_connections 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91b41466a0 0x7f91b4146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91b41f4d90 0x7f91b41f51a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 --2- 172.21.15.59:0/1593542394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91a4003a70 0x7f91b415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 >> 172.21.15.59:0/1593542394 conn(0x7f91b40d3ee0 msgr2=0x7f91b40b7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 shutdown_connections 2024-06-29T14:34:46.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:46.932+0000 7f91bb1d4640 1 -- 172.21.15.59:0/1593542394 wait complete. 2024-06-29T14:34:46.941 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:34:46.941 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:47.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.064+0000 7f5661ffb640 1 -- 172.21.15.59:0/746791172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 msgr2=0x7f565c161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.064+0000 7f5661ffb640 1 --2- 172.21.15.59:0/746791172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 0x7f565c161a80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5644005fd0 tx=0x7f564402a540 comp rx=0 tx=0).stop 2024-06-29T14:34:47.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.064+0000 7f5661ffb640 1 -- 172.21.15.59:0/746791172 shutdown_connections 2024-06-29T14:34:47.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.064+0000 7f5661ffb640 1 --2- 172.21.15.59:0/746791172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 0x7f565c161a80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.064+0000 7f5661ffb640 1 --2- 172.21.15.59:0/746791172 >> v2:172.21.15.153:3300/0 conn(0x7f565c162770 0x7f565c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.064+0000 7f5661ffb640 1 -- 172.21.15.59:0/746791172 >> 172.21.15.59:0/746791172 conn(0x7f565c0d45b0 msgr2=0x7f565c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:47.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.064+0000 7f5661ffb640 1 -- 172.21.15.59:0/746791172 shutdown_connections 2024-06-29T14:34:47.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.064+0000 7f5661ffb640 1 -- 172.21.15.59:0/746791172 wait complete. 2024-06-29T14:34:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.066+0000 7f5661ffb640 1 Processor -- start 2024-06-29T14:34:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.066+0000 7f5661ffb640 1 -- start start 2024-06-29T14:34:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f5661ffb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 0x7f565c0cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f5661ffb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f565c161fc0 0x7f565c0cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f5661ffb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f565c162770 0x7f565c0c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f5661ffb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f565c0d2d80 con 0x7f565c162770 2024-06-29T14:34:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f5661ffb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f565c0b4b00 con 0x7f565c161fc0 2024-06-29T14:34:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f5661ffb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f565c0d2f00 con 0x7f565c15e0a0 2024-06-29T14:34:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f565bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f565c162770 0x7f565c0c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f565bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f565c162770 0x7f565c0c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49928/0 (socket says 172.21.15.59:49928) 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f5652ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f565c161fc0 0x7f565c0cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.067+0000 7f565bfff640 1 -- 172.21.15.59:0/1235690564 learned_addr learned my addr 172.21.15.59:0/1235690564 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f565bfff640 1 -- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 msgr2=0x7f565c0cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f565b7fe640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 0x7f565c0cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f565bfff640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 0x7f565c0cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f565bfff640 1 -- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f565c161fc0 msgr2=0x7f565c0cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f565bfff640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f565c161fc0 0x7f565c0cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f565bfff640 1 -- 172.21.15.59:0/1235690564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565c0c8e30 con 0x7f565c162770 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f5652ffd640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f565c161fc0 0x7f565c0cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f565b7fe640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 0x7f565c0cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f565bfff640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f565c162770 0x7f565c0c7510 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f565400c730 tx=0x7f565400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:47.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f5658ff9640 1 -- 172.21.15.59:0/1235690564 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654003b90 con 0x7f565c162770 2024-06-29T14:34:47.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f5658ff9640 1 -- 172.21.15.59:0/1235690564 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5654003d50 con 0x7f565c162770 2024-06-29T14:34:47.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f5658ff9640 1 -- 172.21.15.59:0/1235690564 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654005330 con 0x7f565c162770 2024-06-29T14:34:47.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f565c15e9a0 con 0x7f565c162770 2024-06-29T14:34:47.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.068+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f565c161120 con 0x7f565c162770 2024-06-29T14:34:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.069+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f565c15eec0 con 0x7f565c162770 2024-06-29T14:34:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.070+0000 7f5658ff9640 1 -- 172.21.15.59:0/1235690564 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7f56540054d0 con 0x7f565c162770 2024-06-29T14:34:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.071+0000 7f5658ff9640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f563c076120 0x7f563c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.071+0000 7f5658ff9640 1 -- 172.21.15.59:0/1235690564 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7299+0+0 (secure 0 0 0) 0x7f56540977b0 con 0x7f565c162770 2024-06-29T14:34:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.071+0000 7f565b7fe640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f563c076120 0x7f563c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.072+0000 7f565b7fe640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f563c076120 0x7f563c078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f564c009ab0 tx=0x7f564c005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:47.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.075+0000 7f5658ff9640 1 -- 172.21.15.59:0/1235690564 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f565c15eec0 con 0x7f565c162770 2024-06-29T14:34:47.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.308+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f565c0d33b0 con 0x7f565c162770 2024-06-29T14:34:47.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.308+0000 7f5658ff9640 1 -- 172.21.15.59:0/1235690564 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5223 (secure 0 0 0) 0x7f565c1623d0 con 0x7f565c162770 2024-06-29T14:34:47.306 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:47.306 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"btime":"2024-06-29T14:23:19:215604+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/634859216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":634859216},{"type":"v1","addr":"172.21.15.153:6837","nonce":634859216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3661382082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3661382082},{"type":"v1","addr":"172.21.15.153:6836","nonce":3661382082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4012387069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4012387069},{"type":"v1","addr":"172.21.15.59:6833","nonce":4012387069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:15.579309+0000","modified":"2024-06-29T14:23:18.607053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5469},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5469":{"gid":5469,"name":"a","rank":0,"incarnation":15,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/812277010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":812277010},{"type":"v1","addr":"172.21.15.59:6835","nonce":812277010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5469,"qdb_cluster":[5469]},"id":2}]} 2024-06-29T14:34:47.306 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 19 2024-06-29T14:34:47.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f563c076120 msgr2=0x7f563c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f563c076120 0x7f563c078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f564c009ab0 tx=0x7f564c005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:47.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f565c162770 msgr2=0x7f565c0c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f565c162770 0x7f565c0c7510 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f565400c730 tx=0x7f565400cbf0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 shutdown_connections 2024-06-29T14:34:47.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f563c076120 0x7f563c078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f565c162770 0x7f565c0c7510 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f565c161fc0 0x7f565c0cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 --2- 172.21.15.59:0/1235690564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f565c15e0a0 0x7f565c0cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 >> 172.21.15.59:0/1235690564 conn(0x7f565c0d45b0 msgr2=0x7f565c0d2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:47.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 shutdown_connections 2024-06-29T14:34:47.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.311+0000 7f5661ffb640 1 -- 172.21.15.59:0/1235690564 wait complete. 2024-06-29T14:34:47.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:47.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.441+0000 7f06fe591640 1 -- 172.21.15.59:0/2461487401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 msgr2=0x7f06e0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.441+0000 7f06fe591640 1 --2- 172.21.15.59:0/2461487401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 0x7f06e0004800 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00c4a0 tx=0x7f06e0005830 comp rx=0 tx=0).stop 2024-06-29T14:34:47.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.441+0000 7f06fcb2e640 1 -- 172.21.15.59:0/2461487401 reap_dead start 2024-06-29T14:34:47.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.441+0000 7f06fe591640 1 -- 172.21.15.59:0/2461487401 shutdown_connections 2024-06-29T14:34:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.441+0000 7f06fe591640 1 -- 172.21.15.59:0/2461487401 >> 172.21.15.59:0/2461487401 conn(0x7f06f80b06e0 msgr2=0x7f06f80b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.441+0000 7f06fe591640 1 -- 172.21.15.59:0/2461487401 shutdown_connections 2024-06-29T14:34:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.441+0000 7f06fe591640 1 -- 172.21.15.59:0/2461487401 wait complete. 2024-06-29T14:34:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.443+0000 7f06fe591640 1 Processor -- start 2024-06-29T14:34:47.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.443+0000 7f06fe591640 1 -- start start 2024-06-29T14:34:47.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.444+0000 7f06fe591640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 0x7f06f8140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.444+0000 7f06fe591640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06f815a780 0x7f06f815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06fe591640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06f80b0010 0x7f06f815b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06fe591640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f06f80b1e90 con 0x7f06e0004230 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06fe591640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06f80b1d10 con 0x7f06f815a780 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06fe591640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f06f80cf8e0 con 0x7f06f80b0010 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 0x7f06f8140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 0x7f06f8140860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49934/0 (socket says 172.21.15.59:49934) 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 -- 172.21.15.59:0/382226368 learned_addr learned my addr 172.21.15.59:0/382226368 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f77fe640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06f815a780 0x7f06f815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06fcb2e640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06f80b0010 0x7f06f815b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 -- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06f80b0010 msgr2=0x7f06f815b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06f80b0010 0x7f06f815b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 -- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06f815a780 msgr2=0x7f06f815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06f815a780 0x7f06f815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 -- 172.21.15.59:0/382226368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f81596e0 con 0x7f06e0004230 2024-06-29T14:34:47.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06fcb2e640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06f80b0010 0x7f06f815b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:47.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f77fe640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06f815a780 0x7f06f815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:47.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f7fff640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 0x7f06f8140860 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f06e8011bf0 tx=0x7f06e800e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:47.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f4ff9640 1 -- 172.21.15.59:0/382226368 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e8004260 con 0x7f06e0004230 2024-06-29T14:34:47.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f4ff9640 1 -- 172.21.15.59:0/382226368 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06e8004400 con 0x7f06e0004230 2024-06-29T14:34:47.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f81599c0 con 0x7f06e0004230 2024-06-29T14:34:47.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.445+0000 7f06f4ff9640 1 -- 172.21.15.59:0/382226368 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e8003850 con 0x7f06e0004230 2024-06-29T14:34:47.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.446+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f8159f80 con 0x7f06e0004230 2024-06-29T14:34:47.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.447+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f80ce8d0 con 0x7f06e0004230 2024-06-29T14:34:47.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.447+0000 7f06f4ff9640 1 -- 172.21.15.59:0/382226368 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7f06e80039f0 con 0x7f06e0004230 2024-06-29T14:34:47.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.448+0000 7f06f4ff9640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f06e4076120 0x7f06e4078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.448+0000 7f06f4ff9640 1 -- 172.21.15.59:0/382226368 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7299+0+0 (secure 0 0 0) 0x7f06e809a910 con 0x7f06e0004230 2024-06-29T14:34:47.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.448+0000 7f06f77fe640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f06e4076120 0x7f06e4078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.449+0000 7f06f77fe640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f06e4076120 0x7f06e4078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f06ec033000 tx=0x7f06ec02d070 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:47.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.452+0000 7f06f4ff9640 1 -- 172.21.15.59:0/382226368 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f06e8064030 con 0x7f06e0004230 2024-06-29T14:34:47.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.682+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06f806e500 con 0x7f06e0004230 2024-06-29T14:34:47.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.683+0000 7f06f4ff9640 1 -- 172.21.15.59:0/382226368 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5223 (secure 0 0 0) 0x7f06e8063930 con 0x7f06e0004230 2024-06-29T14:34:47.680 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:47.680 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"btime":"2024-06-29T14:23:19:215604+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/634859216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":634859216},{"type":"v1","addr":"172.21.15.153:6837","nonce":634859216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3661382082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3661382082},{"type":"v1","addr":"172.21.15.153:6836","nonce":3661382082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4012387069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4012387069},{"type":"v1","addr":"172.21.15.59:6833","nonce":4012387069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:15.579309+0000","modified":"2024-06-29T14:23:18.607053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5469},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5469":{"gid":5469,"name":"a","rank":0,"incarnation":15,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/812277010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":812277010},{"type":"v1","addr":"172.21.15.59:6835","nonce":812277010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5469,"qdb_cluster":[5469]},"id":2}]} 2024-06-29T14:34:47.680 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 19 2024-06-29T14:34:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f06e4076120 msgr2=0x7f06e4078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f06e4076120 0x7f06e4078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f06ec033000 tx=0x7f06ec02d070 comp rx=0 tx=0).stop 2024-06-29T14:34:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 msgr2=0x7f06f8140860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 0x7f06f8140860 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f06e8011bf0 tx=0x7f06e800e780 comp rx=0 tx=0).stop 2024-06-29T14:34:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 shutdown_connections 2024-06-29T14:34:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f06e4076120 0x7f06e4078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06f80b0010 0x7f06f815b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06f815a780 0x7f06f815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 --2- 172.21.15.59:0/382226368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06e0004230 0x7f06f8140860 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 >> 172.21.15.59:0/382226368 conn(0x7f06f80b06e0 msgr2=0x7f06f815cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 shutdown_connections 2024-06-29T14:34:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.685+0000 7f06fe591640 1 -- 172.21.15.59:0/382226368 wait complete. 2024-06-29T14:34:47.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:47.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.814+0000 7f6b9b518640 1 -- 172.21.15.59:0/2926310931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c004800 msgr2=0x7f6b7c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.814+0000 7f6b9b518640 1 --2- 172.21.15.59:0/2926310931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c004800 0x7f6b7c004c10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6b940736f0 tx=0x7f6b7c005830 comp rx=0 tx=0).stop 2024-06-29T14:34:47.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.814+0000 7f6b99ab5640 1 -- 172.21.15.59:0/2926310931 reap_dead start 2024-06-29T14:34:47.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.814+0000 7f6b9b518640 1 -- 172.21.15.59:0/2926310931 shutdown_connections 2024-06-29T14:34:47.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.814+0000 7f6b9b518640 1 -- 172.21.15.59:0/2926310931 >> 172.21.15.59:0/2926310931 conn(0x7f6b9406c950 msgr2=0x7f6b94073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:47.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.814+0000 7f6b9b518640 1 -- 172.21.15.59:0/2926310931 shutdown_connections 2024-06-29T14:34:47.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.814+0000 7f6b9b518640 1 -- 172.21.15.59:0/2926310931 wait complete. 2024-06-29T14:34:47.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.816+0000 7f6b9b518640 1 Processor -- start 2024-06-29T14:34:47.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.816+0000 7f6b9b518640 1 -- start start 2024-06-29T14:34:47.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b9b518640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c004800 0x7f6b940ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b9b518640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b941430d0 0x7f6b941f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b9b518640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b941f43b0 0x7f6b941f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b9b518640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6b940cf870 con 0x7f6b941f43b0 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b9b518640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6b94145ab0 con 0x7f6b7c004800 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b9b518640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6b940cf9f0 con 0x7f6b941430d0 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b99ab5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b941f43b0 0x7f6b941f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b99ab5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b941f43b0 0x7f6b941f5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49956/0 (socket says 172.21.15.59:49956) 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b99ab5640 1 -- 172.21.15.59:0/2850721652 learned_addr learned my addr 172.21.15.59:0/2850721652 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b98ab3640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b941430d0 0x7f6b941f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.817+0000 7f6b992b4640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c004800 0x7f6b940ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b99ab5640 1 -- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b941430d0 msgr2=0x7f6b941f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b99ab5640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b941430d0 0x7f6b941f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b99ab5640 1 -- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c004800 msgr2=0x7f6b940ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b99ab5640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c004800 0x7f6b940ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b99ab5640 1 -- 172.21.15.59:0/2850721652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b941438b0 con 0x7f6b941f43b0 2024-06-29T14:34:47.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b992b4640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c004800 0x7f6b940ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:47.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b99ab5640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b941f43b0 0x7f6b941f5f70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6b940b55a0 tx=0x7f6b84006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:47.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b91ffb640 1 -- 172.21.15.59:0/2850721652 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8400fc60 con 0x7f6b941f43b0 2024-06-29T14:34:47.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b91ffb640 1 -- 172.21.15.59:0/2850721652 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b8400fe00 con 0x7f6b941f43b0 2024-06-29T14:34:47.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b91ffb640 1 -- 172.21.15.59:0/2850721652 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84011690 con 0x7f6b941f43b0 2024-06-29T14:34:47.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b94143b90 con 0x7f6b941f43b0 2024-06-29T14:34:47.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.818+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b94144150 con 0x7f6b941f43b0 2024-06-29T14:34:47.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.819+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9415c2f0 con 0x7f6b941f43b0 2024-06-29T14:34:47.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.820+0000 7f6b91ffb640 1 -- 172.21.15.59:0/2850721652 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7f6b84011830 con 0x7f6b941f43b0 2024-06-29T14:34:47.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.820+0000 7f6b91ffb640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6b80076120 0x7f6b80078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.821+0000 7f6b91ffb640 1 -- 172.21.15.59:0/2850721652 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7299+0+0 (secure 0 0 0) 0x7f6b84098480 con 0x7f6b941f43b0 2024-06-29T14:34:47.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.821+0000 7f6b992b4640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6b80076120 0x7f6b80078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.821+0000 7f6b992b4640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6b80076120 0x7f6b80078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c005fd0 tx=0x7f6b8c00d550 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:47.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:47.825+0000 7f6b91ffb640 1 -- 172.21.15.59:0/2850721652 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b84061a50 con 0x7f6b941f43b0 2024-06-29T14:34:48.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.016+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b94143640 con 0x7f6b941f43b0 2024-06-29T14:34:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.018+0000 7f6b91ffb640 1 -- 172.21.15.59:0/2850721652 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+18770 (secure 0 0 0) 0x7f6b84061350 con 0x7f6b941f43b0 2024-06-29T14:34:48.015 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:48.015 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":58,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:34:46.010680+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:13.422166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:14.527412+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1169887706":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/2575712432":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3207465637":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/2990390183":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6833/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/1180005807":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/779704172":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/1179750521":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3939656574":"2024-06-30T14:34:46.010641+0000","172.21.15.153:6832/3430128047":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/2185789752":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6832/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/4049145115":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3509841566":"2024-06-30T14:34:46.010641+0000","172.21.15.153:6833/3430128047":"2024-06-30T14:34:46.010641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":1},{"begin":5,"length":3},{"begin":9,"length":1},{"begin":11,"length":3},{"begin":15,"length":3},{"begin":19,"length":1},{"begin":21,"length":4},{"begin":26,"length":3},{"begin":30,"length":1},{"begin":33,"length":3},{"begin":37,"length":3},{"begin":41,"length":1},{"begin":43,"length":3},{"begin":47,"length":3},{"begin":51,"length":2},{"begin":54,"length":3},{"begin":58,"length":3},{"begin":63,"length":1},{"begin":65,"length":3},{"begin":69,"length":4},{"begin":74,"length":1},{"begin":76,"length":3},{"begin":80,"length":4},{"begin":85,"length":1},{"begin":87,"length":3},{"begin":91,"length":4},{"begin":96,"length":1},{"begin":98,"length":3},{"begin":103,"length":3},{"begin":107,"length":1},{"begin":109,"length":3},{"begin":145,"length":77}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6b80076120 msgr2=0x7f6b80078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6b80076120 0x7f6b80078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c005fd0 tx=0x7f6b8c00d550 comp rx=0 tx=0).stop 2024-06-29T14:34:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b941f43b0 msgr2=0x7f6b941f5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b941f43b0 0x7f6b941f5f70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6b940b55a0 tx=0x7f6b84006720 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 shutdown_connections 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6b80076120 0x7f6b80078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b941f43b0 0x7f6b941f5f70 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b941430d0 0x7f6b941f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 --2- 172.21.15.59:0/2850721652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c004800 0x7f6b940ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 >> 172.21.15.59:0/2850721652 conn(0x7f6b9406c950 msgr2=0x7f6b940d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 shutdown_connections 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.020+0000 7f6b9b518640 1 -- 172.21.15.59:0/2850721652 wait complete. 2024-06-29T14:34:48.029 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:34:48.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.152+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/2881671963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 msgr2=0x7f3a080d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.152+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/2881671963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 0x7f3a080d2130 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3a00005fd0 tx=0x7f3a0002a4f0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.152+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/2881671963 shutdown_connections 2024-06-29T14:34:48.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.152+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/2881671963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 0x7f3a080d2130 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.152+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/2881671963 >> v2:172.21.15.153:3300/0 conn(0x7f3a0808c190 0x7f3a080cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.152+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/2881671963 >> 172.21.15.59:0/2881671963 conn(0x7f3a080b5610 msgr2=0x7f3a080b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:48.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.152+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/2881671963 shutdown_connections 2024-06-29T14:34:48.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.152+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/2881671963 wait complete. 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.154+0000 7f3a0f5f0640 1 Processor -- start 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.154+0000 7f3a0f5f0640 1 -- start start 2024-06-29T14:34:48.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0f5f0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0808c190 0x7f3a08155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0f5f0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 0x7f3a08154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0f5f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a080cf5c0 0x7f3a080ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0f5f0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3a080b7f60 con 0x7f3a080ce950 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0f5f0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a080b7de0 con 0x7f3a0808c190 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0f5f0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3a080b80e0 con 0x7f3a080cf5c0 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0cb8b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 0x7f3a08154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0cb8b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 0x7f3a08154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49964/0 (socket says 172.21.15.59:49964) 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0cb8b640 1 -- 172.21.15.59:0/1136828327 learned_addr learned my addr 172.21.15.59:0/1136828327 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0d38c640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0808c190 0x7f3a08155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0db8d640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a080cf5c0 0x7f3a080ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0cb8b640 1 -- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a080cf5c0 msgr2=0x7f3a080ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0cb8b640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a080cf5c0 0x7f3a080ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0cb8b640 1 -- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0808c190 msgr2=0x7f3a08155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0cb8b640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0808c190 0x7f3a08155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0cb8b640 1 -- 172.21.15.59:0/1136828327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a08158c60 con 0x7f3a080ce950 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.155+0000 7f3a0db8d640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a080cf5c0 0x7f3a080ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.156+0000 7f3a0cb8b640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 0x7f3a08154350 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00b3c0 tx=0x7f39fc00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.156+0000 7f3a05ffb640 1 -- 172.21.15.59:0/1136828327 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc011830 con 0x7f3a080ce950 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.156+0000 7f3a05ffb640 1 -- 172.21.15.59:0/1136828327 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39fc016070 con 0x7f3a080ce950 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.156+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0815e3c0 con 0x7f3a080ce950 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.156+0000 7f3a05ffb640 1 -- 172.21.15.59:0/1136828327 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc011d80 con 0x7f3a080ce950 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.156+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a081e84b0 con 0x7f3a080ce950 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.157+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a08158980 con 0x7f3a080ce950 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.158+0000 7f3a05ffb640 1 -- 172.21.15.59:0/1136828327 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7f39fc01b460 con 0x7f3a080ce950 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.158+0000 7f3a05ffb640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f39f4076150 0x7f39f4078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.158+0000 7f3a05ffb640 1 -- 172.21.15.59:0/1136828327 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7299+0+0 (secure 0 0 0) 0x7f39fc097770 con 0x7f3a080ce950 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.158+0000 7f3a0d38c640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f39f4076150 0x7f39f4078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.159+0000 7f3a0d38c640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f39f4076150 0x7f39f4078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f39f8009f20 tx=0x7f39f80094f0 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:48.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.162+0000 7f3a05ffb640 1 -- 172.21.15.59:0/1136828327 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f39fc060dd0 con 0x7f3a080ce950 2024-06-29T14:34:48.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.223+0000 7f3a05ffb640 1 -- 172.21.15.59:0/1136828327 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 17) ==== 98247+0+0 (secure 0 0 0) 0x7f39fc099d00 con 0x7f3a080ce950 2024-06-29T14:34:48.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.401+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3a0815d580 con 0x7f3a080ce950 2024-06-29T14:34:48.398 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T14:34:48.401+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:34:48.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.403+0000 7f3a05ffb640 1 -- 172.21.15.59:0/1136828327 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) ==== 161+0+0 (secure 0 0 0) 0x7f39fc0750f0 con 0x7f3a080ce950 2024-06-29T14:34:48.400 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:34:48.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.405+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f39f4076150 msgr2=0x7f39f4078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.405+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f39f4076150 0x7f39f4078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f39f8009f20 tx=0x7f39f80094f0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.405+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 msgr2=0x7f3a08154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.405+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 0x7f3a08154350 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00b3c0 tx=0x7f39fc00b880 comp rx=0 tx=0).stop 2024-06-29T14:34:48.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.406+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 shutdown_connections 2024-06-29T14:34:48.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.406+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f39f4076150 0x7f39f4078660 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.406+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a080cf5c0 0x7f3a080ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.406+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a080ce950 0x7f3a08154350 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.406+0000 7f3a0f5f0640 1 --2- 172.21.15.59:0/1136828327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0808c190 0x7f3a08155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.406+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 >> 172.21.15.59:0/1136828327 conn(0x7f3a080b5610 msgr2=0x7f3a08160dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:48.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.406+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 shutdown_connections 2024-06-29T14:34:48.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.406+0000 7f3a0f5f0640 1 -- 172.21.15.59:0/1136828327 wait complete. 2024-06-29T14:34:48.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:34:48.466 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:34:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.539+0000 7fab80903640 1 -- 172.21.15.59:0/642048109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c0b4080 msgr2=0x7fab7c0b24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.539+0000 7fab80903640 1 --2- 172.21.15.59:0/642048109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c0b4080 0x7fab7c0b24d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009bb0 tx=0x7fab6c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.539+0000 7fab80903640 1 -- 172.21.15.59:0/642048109 shutdown_connections 2024-06-29T14:34:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.539+0000 7fab80903640 1 --2- 172.21.15.59:0/642048109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab7c0d2880 0x7fab7c0b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.539+0000 7fab80903640 1 --2- 172.21.15.59:0/642048109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c0b4080 0x7fab7c0b24d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.539+0000 7fab80903640 1 -- 172.21.15.59:0/642048109 >> 172.21.15.59:0/642048109 conn(0x7fab7c0d3e80 msgr2=0x7fab7c0d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.539+0000 7fab80903640 1 -- 172.21.15.59:0/642048109 shutdown_connections 2024-06-29T14:34:48.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.539+0000 7fab80903640 1 -- 172.21.15.59:0/642048109 wait complete. 2024-06-29T14:34:48.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.541+0000 7fab80903640 1 Processor -- start 2024-06-29T14:34:48.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.541+0000 7fab80903640 1 -- start start 2024-06-29T14:34:48.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab80903640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab7c0d2880 0x7fab7c0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab80903640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab7c162340 0x7fab7c0c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab80903640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c155c90 0x7fab7c0c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab80903640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c15dc10 con 0x7fab7c155c90 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab80903640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c0b3160 con 0x7fab7c0d2880 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab80903640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fab7c15dd90 con 0x7fab7c162340 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c155c90 0x7fab7c0c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c155c90 0x7fab7c0c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49998/0 (socket says 172.21.15.59:49998) 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 -- 172.21.15.59:0/3623196419 learned_addr learned my addr 172.21.15.59:0/3623196419 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7a59c640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab7c0d2880 0x7fab7c0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab79d9b640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab7c162340 0x7fab7c0c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 -- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab7c162340 msgr2=0x7fab7c0c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab7c162340 0x7fab7c0c8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 -- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab7c0d2880 msgr2=0x7fab7c0ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab7c0d2880 0x7fab7c0ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 -- 172.21.15.59:0/3623196419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c156230 con 0x7fab7c155c90 2024-06-29T14:34:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7a59c640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab7c0d2880 0x7fab7c0ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab7ad9d640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c155c90 0x7fab7c0c90c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fab7400d660 tx=0x7fab7400db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab72ffd640 1 -- 172.21.15.59:0/3623196419 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74017070 con 0x7fab7c155c90 2024-06-29T14:34:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab72ffd640 1 -- 172.21.15.59:0/3623196419 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab74004310 con 0x7fab7c155c90 2024-06-29T14:34:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab72ffd640 1 -- 172.21.15.59:0/3623196419 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74005280 con 0x7fab7c155c90 2024-06-29T14:34:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.543+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7c156510 con 0x7fab7c155c90 2024-06-29T14:34:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.544+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7c156ad0 con 0x7fab7c155c90 2024-06-29T14:34:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.544+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7c0b24d0 con 0x7fab7c155c90 2024-06-29T14:34:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.545+0000 7fab72ffd640 1 -- 172.21.15.59:0/3623196419 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) ==== 98247+0+0 (secure 0 0 0) 0x7fab74005420 con 0x7fab7c155c90 2024-06-29T14:34:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.546+0000 7fab72ffd640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fab5c076390 0x7fab5c0788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.546+0000 7fab72ffd640 1 -- 172.21.15.59:0/3623196419 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 7385+0+0 (secure 0 0 0) 0x7fab74098220 con 0x7fab7c155c90 2024-06-29T14:34:48.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.546+0000 7fab7a59c640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fab5c076390 0x7fab5c0788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.547+0000 7fab7a59c640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fab5c076390 0x7fab5c0788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab64007fd0 tx=0x7fab64009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:48.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.550+0000 7fab72ffd640 1 -- 172.21.15.59:0/3623196419 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fab74020020 con 0x7fab7c155c90 2024-06-29T14:34:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:48.780+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fab7c08c820 con 0x7fab7c155c90 2024-06-29T14:34:49.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.404+0000 7fab72ffd640 1 -- 172.21.15.59:0/3623196419 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7fab740613f0 con 0x7fab7c155c90 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fab5c076390 msgr2=0x7fab5c0788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fab5c076390 0x7fab5c0788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fab64007fd0 tx=0x7fab64009040 comp rx=0 tx=0).stop 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c155c90 msgr2=0x7fab7c0c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c155c90 0x7fab7c0c90c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fab7400d660 tx=0x7fab7400db20 comp rx=0 tx=0).stop 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 shutdown_connections 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fab5c076390 0x7fab5c0788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab7c155c90 0x7fab7c0c90c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab7c162340 0x7fab7c0c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 --2- 172.21.15.59:0/3623196419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab7c0d2880 0x7fab7c0ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 >> 172.21.15.59:0/3623196419 conn(0x7fab7c0d3e80 msgr2=0x7fab7c0b8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 shutdown_connections 2024-06-29T14:34:49.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.407+0000 7fab80903640 1 -- 172.21.15.59:0/3623196419 wait complete. 2024-06-29T14:34:49.418 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:34:49.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.547+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/321113819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc0d1ea0 msgr2=0x7ff8dc0cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.547+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/321113819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc0d1ea0 0x7ff8dc0cf700 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009be0 tx=0x7ff8cc02a870 comp rx=0 tx=0).stop 2024-06-29T14:34:49.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.547+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/321113819 shutdown_connections 2024-06-29T14:34:49.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.547+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/321113819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc0d1ea0 0x7ff8dc0cf700 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.547+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/321113819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8dc1423c0 0x7ff8dc0b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.547+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/321113819 >> 172.21.15.59:0/321113819 conn(0x7ff8dc073e10 msgr2=0x7ff8dc074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:49.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.547+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/321113819 shutdown_connections 2024-06-29T14:34:49.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.547+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/321113819 wait complete. 2024-06-29T14:34:49.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.549+0000 7ff8e3a3f640 1 Processor -- start 2024-06-29T14:34:49.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.549+0000 7ff8e3a3f640 1 -- start start 2024-06-29T14:34:49.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e3a3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8dc0cec20 0x7ff8dc0c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e3a3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc1423c0 0x7ff8dc1f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e3a3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc143330 0x7ff8dc1f60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e3a3f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc0cfdc0 con 0x7ff8dc1423c0 2024-06-29T14:34:49.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e3a3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc0cfc40 con 0x7ff8dc143330 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e3a3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc0cff40 con 0x7ff8dc0cec20 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e0fda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc1423c0 0x7ff8dc1f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e0fda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc1423c0 0x7ff8dc1f5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50010/0 (socket says 172.21.15.59:50010) 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e0fda640 1 -- 172.21.15.59:0/382180925 learned_addr learned my addr 172.21.15.59:0/382180925 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e1fdc640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc143330 0x7ff8dc1f60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e17db640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8dc0cec20 0x7ff8dc0c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.550+0000 7ff8e0fda640 1 -- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8dc0cec20 msgr2=0x7ff8dc0c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8e0fda640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8dc0cec20 0x7ff8dc0c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8e0fda640 1 -- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc143330 msgr2=0x7ff8dc1f60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8e0fda640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc143330 0x7ff8dc1f60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8e0fda640 1 -- 172.21.15.59:0/382180925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc1e6e20 con 0x7ff8dc1423c0 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8e1fdc640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc143330 0x7ff8dc1f60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:49.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8e0fda640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc1423c0 0x7ff8dc1f5b80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000c3c0 tx=0x7ff8d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:49.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8d9ffb640 1 -- 172.21.15.59:0/382180925 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0011790 con 0x7ff8dc1423c0 2024-06-29T14:34:49.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8d9ffb640 1 -- 172.21.15.59:0/382180925 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8d0017070 con 0x7ff8dc1423c0 2024-06-29T14:34:49.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc1e6080 con 0x7ff8dc1423c0 2024-06-29T14:34:49.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8d9ffb640 1 -- 172.21.15.59:0/382180925 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0011ce0 con 0x7ff8dc1423c0 2024-06-29T14:34:49.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.551+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc1e6620 con 0x7ff8dc1423c0 2024-06-29T14:34:49.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.552+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc0cdc90 con 0x7ff8dc1423c0 2024-06-29T14:34:49.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.553+0000 7ff8d9ffb640 1 -- 172.21.15.59:0/382180925 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) ==== 98247+0+0 (secure 0 0 0) 0x7ff8d001c460 con 0x7ff8dc1423c0 2024-06-29T14:34:49.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.553+0000 7ff8d9ffb640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff8c8076360 0x7ff8c8078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.554+0000 7ff8d9ffb640 1 -- 172.21.15.59:0/382180925 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 7385+0+0 (secure 0 0 0) 0x7ff8d0097d20 con 0x7ff8dc1423c0 2024-06-29T14:34:49.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.554+0000 7ff8e17db640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff8c8076360 0x7ff8c8078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:49.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.554+0000 7ff8e17db640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff8c8076360 0x7ff8c8078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009ac0 tx=0x7ff8d4005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:49.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.558+0000 7ff8d9ffb640 1 -- 172.21.15.59:0/382180925 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff8dc0cdc90 con 0x7ff8dc1423c0 2024-06-29T14:34:49.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:49.753+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff8dc0d0430 con 0x7ff8dc1423c0 2024-06-29T14:34:50.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.414+0000 7ff8d9ffb640 1 -- 172.21.15.59:0/382180925 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) ==== 181+0+0 (secure 0 0 0) 0x7ff8d0060e70 con 0x7ff8dc1423c0 2024-06-29T14:34:50.411 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:34:50.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.417+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff8c8076360 msgr2=0x7ff8c8078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.417+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff8c8076360 0x7ff8c8078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009ac0 tx=0x7ff8d4005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:50.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.417+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc1423c0 msgr2=0x7ff8dc1f5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.417+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc1423c0 0x7ff8dc1f5b80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000c3c0 tx=0x7ff8d000c880 comp rx=0 tx=0).stop 2024-06-29T14:34:50.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.419+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 shutdown_connections 2024-06-29T14:34:50.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.419+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff8c8076360 0x7ff8c8078870 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009ac0 tx=0x7ff8d4005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:50.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.419+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8dc143330 0x7ff8dc1f60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.419+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc1423c0 0x7ff8dc1f5b80 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.419+0000 7ff8e3a3f640 1 --2- 172.21.15.59:0/382180925 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8dc0cec20 0x7ff8dc0c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.419+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 >> 172.21.15.59:0/382180925 conn(0x7ff8dc073e10 msgr2=0x7ff8dc15b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:50.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.422+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 shutdown_connections 2024-06-29T14:34:50.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.422+0000 7ff8e3a3f640 1 -- 172.21.15.59:0/382180925 wait complete. 2024-06-29T14:34:50.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:34:50.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.565+0000 7ff250355640 1 -- 172.21.15.59:0/2417603654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2480b2a50 msgr2=0x7ff248161380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.565+0000 7ff250355640 1 --2- 172.21.15.59:0/2417603654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2480b2a50 0x7ff248161380 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff244009830 tx=0x7ff24402a510 comp rx=0 tx=0).stop 2024-06-29T14:34:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.566+0000 7ff250355640 1 -- 172.21.15.59:0/2417603654 shutdown_connections 2024-06-29T14:34:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.566+0000 7ff250355640 1 --2- 172.21.15.59:0/2417603654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2480d3680 0x7ff2480b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.566+0000 7ff250355640 1 --2- 172.21.15.59:0/2417603654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2480b2a50 0x7ff248161380 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.566+0000 7ff250355640 1 -- 172.21.15.59:0/2417603654 >> 172.21.15.59:0/2417603654 conn(0x7ff2480b5590 msgr2=0x7ff2480b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.566+0000 7ff250355640 1 -- 172.21.15.59:0/2417603654 shutdown_connections 2024-06-29T14:34:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.566+0000 7ff250355640 1 -- 172.21.15.59:0/2417603654 wait complete. 2024-06-29T14:34:50.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.567+0000 7ff250355640 1 Processor -- start 2024-06-29T14:34:50.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.567+0000 7ff250355640 1 -- start start 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff250355640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2480b2a50 0x7ff248154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff250355640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2480b7150 0x7ff2480cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff250355640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff248155e80 0x7ff248156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff250355640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff2480b4510 con 0x7ff248155e80 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff250355640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2480b53e0 con 0x7ff2480b2a50 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff250355640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff2480b4690 con 0x7ff2480b7150 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff248155e80 0x7ff248156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff248155e80 0x7ff248156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50024/0 (socket says 172.21.15.59:50024) 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 -- 172.21.15.59:0/3551387490 learned_addr learned my addr 172.21.15.59:0/3551387490 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e0f1640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2480b2a50 0x7ff248154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24d8f0640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2480b7150 0x7ff2480cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 -- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2480b7150 msgr2=0x7ff2480cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2480b7150 0x7ff2480cd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 -- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2480b2a50 msgr2=0x7ff248154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2480b2a50 0x7ff248154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 -- 172.21.15.59:0/3551387490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2481567d0 con 0x7ff248155e80 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e0f1640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2480b2a50 0x7ff248154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.569+0000 7ff24e8f2640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff248155e80 0x7ff248156290 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff24000ca80 tx=0x7ff24000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.570+0000 7ff23affd640 1 -- 172.21.15.59:0/3551387490 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24000ae20 con 0x7ff248155e80 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.570+0000 7ff23affd640 1 -- 172.21.15.59:0/3551387490 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff240003a80 con 0x7ff248155e80 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.570+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff248160ce0 con 0x7ff248155e80 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.570+0000 7ff23affd640 1 -- 172.21.15.59:0/3551387490 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240004030 con 0x7ff248155e80 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.570+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2481f87e0 con 0x7ff248155e80 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.571+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2480cdb70 con 0x7ff248155e80 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.571+0000 7ff23affd640 1 -- 172.21.15.59:0/3551387490 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7ff2400041d0 con 0x7ff248155e80 2024-06-29T14:34:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.572+0000 7ff23affd640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff22c076420 0x7ff22c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.572+0000 7ff23affd640 1 -- 172.21.15.59:0/3551387490 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6423+0+0 (secure 0 0 0) 0x7ff240098450 con 0x7ff248155e80 2024-06-29T14:34:50.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.572+0000 7ff24e0f1640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff22c076420 0x7ff22c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.573+0000 7ff24e0f1640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff22c076420 0x7ff22c078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff244034000 tx=0x7ff244004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:50.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.576+0000 7ff23affd640 1 -- 172.21.15.59:0/3551387490 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff2481567d0 con 0x7ff248155e80 2024-06-29T14:34:50.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:50.769+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff248161380 con 0x7ff248155e80 2024-06-29T14:34:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.416+0000 7ff23affd640 1 -- 172.21.15.59:0/3551387490 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) ==== 169+0+0 (secure 0 0 0) 0x7ff240014070 con 0x7ff248155e80 2024-06-29T14:34:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff22c076420 msgr2=0x7ff22c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff22c076420 0x7ff22c078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff244034000 tx=0x7ff244004730 comp rx=0 tx=0).stop 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff248155e80 msgr2=0x7ff248156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff248155e80 0x7ff248156290 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff24000ca80 tx=0x7ff24000cf40 comp rx=0 tx=0).stop 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 shutdown_connections 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff22c076420 0x7ff22c078930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff248155e80 0x7ff248156290 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2480b7150 0x7ff2480cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 --2- 172.21.15.59:0/3551387490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2480b2a50 0x7ff248154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 >> 172.21.15.59:0/3551387490 conn(0x7ff2480b5590 msgr2=0x7ff24806c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 shutdown_connections 2024-06-29T14:34:51.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.418+0000 7ff250355640 1 -- 172.21.15.59:0/3551387490 wait complete. 2024-06-29T14:34:51.432 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:34:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.563+0000 7efe4437f640 1 -- 172.21.15.59:0/287104650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe3c0ce950 msgr2=0x7efe3c0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.563+0000 7efe4437f640 1 --2- 172.21.15.59:0/287104650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe3c0ce950 0x7efe3c0d2130 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efe34009830 tx=0x7efe3402a510 comp rx=0 tx=0).stop 2024-06-29T14:34:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.564+0000 7efe4437f640 1 -- 172.21.15.59:0/287104650 shutdown_connections 2024-06-29T14:34:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.564+0000 7efe4437f640 1 --2- 172.21.15.59:0/287104650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe3c0ce950 0x7efe3c0d2130 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.564+0000 7efe4437f640 1 --2- 172.21.15.59:0/287104650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c08c190 0x7efe3c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.564+0000 7efe4437f640 1 -- 172.21.15.59:0/287104650 >> 172.21.15.59:0/287104650 conn(0x7efe3c0b5610 msgr2=0x7efe3c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.564+0000 7efe4437f640 1 -- 172.21.15.59:0/287104650 shutdown_connections 2024-06-29T14:34:51.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.564+0000 7efe4437f640 1 -- 172.21.15.59:0/287104650 wait complete. 2024-06-29T14:34:51.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.565+0000 7efe4437f640 1 Processor -- start 2024-06-29T14:34:51.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.565+0000 7efe4437f640 1 -- start start 2024-06-29T14:34:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4437f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe3c08c190 0x7efe3c15cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4437f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c0ce950 0x7efe3c0ab180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4437f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c0cf5c0 0x7efe3c0ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4437f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c0b7fa0 con 0x7efe3c0ce950 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4437f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c0b7e20 con 0x7efe3c08c190 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4437f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efe3c0b8120 con 0x7efe3c0cf5c0 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c0ce950 0x7efe3c0ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c0ce950 0x7efe3c0ab180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50030/0 (socket says 172.21.15.59:50030) 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 -- 172.21.15.59:0/1934915795 learned_addr learned my addr 172.21.15.59:0/1934915795 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4211b640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe3c08c190 0x7efe3c15cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4291c640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c0cf5c0 0x7efe3c0ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 -- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c0cf5c0 msgr2=0x7efe3c0ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c0cf5c0 0x7efe3c0ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 -- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe3c08c190 msgr2=0x7efe3c15cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe3c08c190 0x7efe3c15cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 -- 172.21.15.59:0/1934915795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3c0abe70 con 0x7efe3c0ce950 2024-06-29T14:34:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.567+0000 7efe4191a640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c0ce950 0x7efe3c0ab180 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c3c0 tx=0x7efe3000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.568+0000 7efe3affd640 1 -- 172.21.15.59:0/1934915795 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30017070 con 0x7efe3c0ce950 2024-06-29T14:34:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.568+0000 7efe3affd640 1 -- 172.21.15.59:0/1934915795 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe300053e0 con 0x7efe3c0ce950 2024-06-29T14:34:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.568+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3c0ac150 con 0x7efe3c0ce950 2024-06-29T14:34:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.568+0000 7efe3affd640 1 -- 172.21.15.59:0/1934915795 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30003870 con 0x7efe3c0ce950 2024-06-29T14:34:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.568+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe3c0ac710 con 0x7efe3c0ce950 2024-06-29T14:34:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.569+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe3c0abbe0 con 0x7efe3c0ce950 2024-06-29T14:34:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.569+0000 7efe3affd640 1 -- 172.21.15.59:0/1934915795 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7efe3001a070 con 0x7efe3c0ce950 2024-06-29T14:34:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.570+0000 7efe3affd640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7efe280763f0 0x7efe28078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.570+0000 7efe3affd640 1 -- 172.21.15.59:0/1934915795 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5980+0+0 (secure 0 0 0) 0x7efe30096d60 con 0x7efe3c0ce950 2024-06-29T14:34:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.570+0000 7efe4211b640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7efe280763f0 0x7efe28078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.571+0000 7efe4211b640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7efe280763f0 0x7efe28078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009a60 tx=0x7efe2c010040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.575+0000 7efe3affd640 1 -- 172.21.15.59:0/1934915795 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efe3005fdc0 con 0x7efe3c0ce950 2024-06-29T14:34:51.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.766+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7efe3c0cf080 con 0x7efe3c0ce950 2024-06-29T14:34:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.770+0000 7efe3affd640 1 -- 172.21.15.59:0/1934915795 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v31) ==== 71+0+0 (secure 0 0 0) 0x7efe300640c0 con 0x7efe3c0ce950 2024-06-29T14:34:51.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7efe280763f0 msgr2=0x7efe28078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7efe280763f0 0x7efe28078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009a60 tx=0x7efe2c010040 comp rx=0 tx=0).stop 2024-06-29T14:34:51.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c0ce950 msgr2=0x7efe3c0ab180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c0ce950 0x7efe3c0ab180 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c3c0 tx=0x7efe3000c880 comp rx=0 tx=0).stop 2024-06-29T14:34:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 shutdown_connections 2024-06-29T14:34:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7efe280763f0 0x7efe28078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe3c0cf5c0 0x7efe3c0ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe3c0ce950 0x7efe3c0ab180 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 --2- 172.21.15.59:0/1934915795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe3c08c190 0x7efe3c15cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 >> 172.21.15.59:0/1934915795 conn(0x7efe3c0b5610 msgr2=0x7efe3c0c7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 shutdown_connections 2024-06-29T14:34:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.772+0000 7efe4437f640 1 -- 172.21.15.59:0/1934915795 wait complete. 2024-06-29T14:34:51.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-06-29T14:34:51.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.900+0000 7f026f0d6640 1 -- 172.21.15.59:0/806419811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680cf5c0 msgr2=0x7f02680d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.900+0000 7f026f0d6640 1 --2- 172.21.15.59:0/806419811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680cf5c0 0x7f02680d1a20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0250005fd0 tx=0x7f025002aa90 comp rx=0 tx=0).stop 2024-06-29T14:34:51.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.900+0000 7f026f0d6640 1 -- 172.21.15.59:0/806419811 shutdown_connections 2024-06-29T14:34:51.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.900+0000 7f026f0d6640 1 --2- 172.21.15.59:0/806419811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680cf5c0 0x7f02680d1a20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.900+0000 7f026f0d6640 1 --2- 172.21.15.59:0/806419811 >> v2:172.21.15.153:3300/0 conn(0x7f026808c190 0x7f02680cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.900+0000 7f026f0d6640 1 -- 172.21.15.59:0/806419811 >> 172.21.15.59:0/806419811 conn(0x7f02680b5610 msgr2=0x7f02680b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.900+0000 7f026f0d6640 1 -- 172.21.15.59:0/806419811 shutdown_connections 2024-06-29T14:34:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.900+0000 7f026f0d6640 1 -- 172.21.15.59:0/806419811 wait complete. 2024-06-29T14:34:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.902+0000 7f026f0d6640 1 Processor -- start 2024-06-29T14:34:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.902+0000 7f026f0d6640 1 -- start start 2024-06-29T14:34:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026f0d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f026808c190 0x7f026815a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026f0d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680ce950 0x7f0268157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026f0d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680cf5c0 0x7f026815d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026f0d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f02680b7fa0 con 0x7f026808c190 2024-06-29T14:34:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026f0d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02680b7e20 con 0x7f02680ce950 2024-06-29T14:34:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026f0d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f02680b8120 con 0x7f02680cf5c0 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f026808c190 0x7f026815a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f026808c190 0x7f026815a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50044/0 (socket says 172.21.15.59:50044) 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 -- 172.21.15.59:0/3444353654 learned_addr learned my addr 172.21.15.59:0/3444353654 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f0267fff640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680ce950 0x7f0268157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026d673640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680cf5c0 0x7f026815d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 -- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680cf5c0 msgr2=0x7f026815d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680cf5c0 0x7f026815d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 -- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680ce950 msgr2=0x7f0268157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680ce950 0x7f0268157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 -- 172.21.15.59:0/3444353654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02681f5810 con 0x7f026808c190 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.904+0000 7f026ce72640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f026808c190 0x7f026815a070 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f025800e9f0 tx=0x7f025800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.905+0000 7f02657fa640 1 -- 172.21.15.59:0/3444353654 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025800cd80 con 0x7f026808c190 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.905+0000 7f02657fa640 1 -- 172.21.15.59:0/3444353654 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0258004020 con 0x7f026808c190 2024-06-29T14:34:51.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.905+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02681f5af0 con 0x7f026808c190 2024-06-29T14:34:51.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.905+0000 7f02657fa640 1 -- 172.21.15.59:0/3444353654 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0258004470 con 0x7f026808c190 2024-06-29T14:34:51.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.905+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f026815ffb0 con 0x7f026808c190 2024-06-29T14:34:51.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.906+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02680cf080 con 0x7f026808c190 2024-06-29T14:34:51.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.907+0000 7f02657fa640 1 -- 172.21.15.59:0/3444353654 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f02580035f0 con 0x7f026808c190 2024-06-29T14:34:51.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.907+0000 7f02657fa640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0248076420 0x7f0248078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.907+0000 7f02657fa640 1 -- 172.21.15.59:0/3444353654 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5980+0+0 (secure 0 0 0) 0x7f02580982e0 con 0x7f026808c190 2024-06-29T14:34:51.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.907+0000 7f0267fff640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0248076420 0x7f0248078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.908+0000 7f0267fff640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0248076420 0x7f0248078930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0250035000 tx=0x7f0250038040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:51.911+0000 7f02657fa640 1 -- 172.21.15.59:0/3444353654 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f02580612c0 con 0x7f026808c190 2024-06-29T14:34:52.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.176+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) -- 0x7f02680c7910 con 0x7f026808c190 2024-06-29T14:34:52.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.425+0000 7f02657fa640 1 -- 172.21.15.59:0/3444353654 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v476) ==== 158+0+0 (secure 0 0 0) 0x7f02580655c0 con 0x7f026808c190 2024-06-29T14:34:52.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.427+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0248076420 msgr2=0x7f0248078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.427+0000 7f026f0d6640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0248076420 0x7f0248078930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0250035000 tx=0x7f0250038040 comp rx=0 tx=0).stop 2024-06-29T14:34:52.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.427+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f026808c190 msgr2=0x7f026815a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.427+0000 7f026f0d6640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f026808c190 0x7f026815a070 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f025800e9f0 tx=0x7f025800eeb0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.428+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 shutdown_connections 2024-06-29T14:34:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.428+0000 7f026f0d6640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0248076420 0x7f0248078930 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.428+0000 7f026f0d6640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02680cf5c0 0x7f026815d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.428+0000 7f026f0d6640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02680ce950 0x7f0268157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.428+0000 7f026f0d6640 1 --2- 172.21.15.59:0/3444353654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f026808c190 0x7f026815a070 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.428+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 >> 172.21.15.59:0/3444353654 conn(0x7f02680b5610 msgr2=0x7f026815e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.428+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 shutdown_connections 2024-06-29T14:34:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.428+0000 7f026f0d6640 1 -- 172.21.15.59:0/3444353654 wait complete. 2024-06-29T14:34:52.436 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-06-29T14:34:52.436 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-06-29T14:34:52.437 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:34:52.437 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:52.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.558+0000 7f076d8e4640 1 -- 172.21.15.59:0/2392706735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07681458a0 msgr2=0x7f07680cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.558+0000 7f076d8e4640 1 --2- 172.21.15.59:0/2392706735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07681458a0 0x7f07680cf870 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0760005fd0 tx=0x7f076002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.558+0000 7f076d8e4640 1 -- 172.21.15.59:0/2392706735 shutdown_connections 2024-06-29T14:34:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.558+0000 7f076d8e4640 1 --2- 172.21.15.59:0/2392706735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07680b3740 0x7f07680b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.558+0000 7f076d8e4640 1 --2- 172.21.15.59:0/2392706735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07681458a0 0x7f07680cf870 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.558+0000 7f076d8e4640 1 -- 172.21.15.59:0/2392706735 >> 172.21.15.59:0/2392706735 conn(0x7f076806c950 msgr2=0x7f0768073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.558+0000 7f076d8e4640 1 -- 172.21.15.59:0/2392706735 shutdown_connections 2024-06-29T14:34:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.558+0000 7f076d8e4640 1 -- 172.21.15.59:0/2392706735 wait complete. 2024-06-29T14:34:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.560+0000 7f076d8e4640 1 Processor -- start 2024-06-29T14:34:52.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.560+0000 7f076d8e4640 1 -- start start 2024-06-29T14:34:52.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f076d8e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07680b3740 0x7f076806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:52.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f076d8e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f076815c2f0 0x7f0768156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:52.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f076d8e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07680ce4a0 0x7f076806e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f076d8e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0768160020 con 0x7f07680ce4a0 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f076d8e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f076815cb90 con 0x7f076815c2f0 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f076d8e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f07681601a0 con 0x7f07680b3740 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f07677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07680ce4a0 0x7f076806e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f07677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07680ce4a0 0x7f076806e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50056/0 (socket says 172.21.15.59:50056) 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f07677fe640 1 -- 172.21.15.59:0/1850803983 learned_addr learned my addr 172.21.15.59:0/1850803983 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f07667fc640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f076815c2f0 0x7f0768156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.561+0000 7f0766ffd640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07680b3740 0x7f076806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f07677fe640 1 -- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07680b3740 msgr2=0x7f076806ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f07677fe640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07680b3740 0x7f076806ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f07677fe640 1 -- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f076815c2f0 msgr2=0x7f0768156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f07677fe640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f076815c2f0 0x7f0768156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f07677fe640 1 -- 172.21.15.59:0/1850803983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076806fb70 con 0x7f07680ce4a0 2024-06-29T14:34:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f07677fe640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07680ce4a0 0x7f076806e490 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f075800c700 tx=0x7f075800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:52.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f075ffff640 1 -- 172.21.15.59:0/1850803983 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758003b70 con 0x7f07680ce4a0 2024-06-29T14:34:52.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f075ffff640 1 -- 172.21.15.59:0/1850803983 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0758003d10 con 0x7f07680ce4a0 2024-06-29T14:34:52.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076806f5b0 con 0x7f07680ce4a0 2024-06-29T14:34:52.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f075ffff640 1 -- 172.21.15.59:0/1850803983 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075800aab0 con 0x7f07680ce4a0 2024-06-29T14:34:52.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.562+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f076806dc90 con 0x7f07680ce4a0 2024-06-29T14:34:52.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.563+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07681606c0 con 0x7f07680ce4a0 2024-06-29T14:34:52.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.564+0000 7f075ffff640 1 -- 172.21.15.59:0/1850803983 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f075800ac50 con 0x7f07680ce4a0 2024-06-29T14:34:52.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.564+0000 7f075ffff640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0748076420 0x7f0748078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:52.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.564+0000 7f075ffff640 1 -- 172.21.15.59:0/1850803983 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5980+0+0 (secure 0 0 0) 0x7f0758097740 con 0x7f07680ce4a0 2024-06-29T14:34:52.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.565+0000 7f0766ffd640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0748076420 0x7f0748078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:52.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.565+0000 7f0766ffd640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0748076420 0x7f0748078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0760033000 tx=0x7f0760004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:52.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.569+0000 7f075ffff640 1 -- 172.21.15.59:0/1850803983 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f07580607a0 con 0x7f07680ce4a0 2024-06-29T14:34:52.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.766+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f076806f340 con 0x7f07680ce4a0 2024-06-29T14:34:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.767+0000 7f075ffff640 1 -- 172.21.15.59:0/1850803983 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+14257 (secure 0 0 0) 0x7f0758067e20 con 0x7f07680ce4a0 2024-06-29T14:34:52.764 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:52.764 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":62,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:34:51.414519+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1169887706":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/2575712432":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3207465637":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/2990390183":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6833/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.59:6834/812277010":"2024-06-30T14:34:48.401803+0000","172.21.15.153:0/1180005807":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/779704172":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/1179750521":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3939656574":"2024-06-30T14:34:46.010641+0000","172.21.15.59:6835/812277010":"2024-06-30T14:34:48.401803+0000","172.21.15.153:6832/3430128047":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/2185789752":"2024-06-30T14:23:32.829085+0000","172.21.15.153:6832/654263470":"2024-06-30T14:23:32.829085+0000","172.21.15.153:0/4049145115":"2024-06-30T14:34:46.010641+0000","172.21.15.153:0/3509841566":"2024-06-30T14:34:46.010641+0000","172.21.15.153:6833/3430128047":"2024-06-30T14:34:46.010641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0748076420 msgr2=0x7f0748078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0748076420 0x7f0748078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0760033000 tx=0x7f0760004730 comp rx=0 tx=0).stop 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07680ce4a0 msgr2=0x7f076806e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07680ce4a0 0x7f076806e490 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f075800c700 tx=0x7f075800cbc0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 shutdown_connections 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0748076420 0x7f0748078930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07680ce4a0 0x7f076806e490 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f076815c2f0 0x7f0768156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 --2- 172.21.15.59:0/1850803983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07680b3740 0x7f076806ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 >> 172.21.15.59:0/1850803983 conn(0x7f076806c950 msgr2=0x7f076815f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 shutdown_connections 2024-06-29T14:34:52.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.769+0000 7f076d8e4640 1 -- 172.21.15.59:0/1850803983 wait complete. 2024-06-29T14:34:52.778 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:34:52.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:34:52.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 -- 172.21.15.59:0/3735273461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c162770 msgr2=0x7f517c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 --2- 172.21.15.59:0/3735273461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c162770 0x7f517c0ce550 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f517c0b29a0 tx=0x7f516c02a4d0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 -- 172.21.15.59:0/3735273461 shutdown_connections 2024-06-29T14:34:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 --2- 172.21.15.59:0/3735273461 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517c161fc0 0x7f517c0d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 --2- 172.21.15.59:0/3735273461 >> v2:172.21.15.153:3300/0 conn(0x7f517c15e0a0 0x7f517c161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 --2- 172.21.15.59:0/3735273461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c162770 0x7f517c0ce550 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 -- 172.21.15.59:0/3735273461 >> 172.21.15.59:0/3735273461 conn(0x7f517c0d45b0 msgr2=0x7f517c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 -- 172.21.15.59:0/3735273461 shutdown_connections 2024-06-29T14:34:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.901+0000 7f5182cb9640 1 -- 172.21.15.59:0/3735273461 wait complete. 2024-06-29T14:34:52.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.903+0000 7f5182cb9640 1 Processor -- start 2024-06-29T14:34:52.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.903+0000 7f5182cb9640 1 -- start start 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.904+0000 7f5182cb9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f517c15e0a0 0x7f517c153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.904+0000 7f5182cb9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c161fc0 0x7f517c154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.904+0000 7f5182cb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517c162770 0x7f517c159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.904+0000 7f5182cb9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f517c0d2dc0 con 0x7f517c161fc0 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.904+0000 7f5182cb9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f517c0b31a0 con 0x7f517c15e0a0 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.904+0000 7f5182cb9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f517c0d2f40 con 0x7f517c162770 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c161fc0 0x7f517c154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c161fc0 0x7f517c154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60466/0 (socket says 172.21.15.59:60466) 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 -- 172.21.15.59:0/1495131010 learned_addr learned my addr 172.21.15.59:0/1495131010 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f5181256640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517c162770 0x7f517c159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f5180a55640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f517c15e0a0 0x7f517c153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 -- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517c162770 msgr2=0x7f517c159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517c162770 0x7f517c159cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 -- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f517c15e0a0 msgr2=0x7f517c153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f517c15e0a0 0x7f517c153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 -- 172.21.15.59:0/1495131010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517c156630 con 0x7f517c161fc0 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f517bfff640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c161fc0 0x7f517c154520 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f516400e9c0 tx=0x7f516400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f51797fa640 1 -- 172.21.15.59:0/1495131010 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516400cd80 con 0x7f517c161fc0 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f51797fa640 1 -- 172.21.15.59:0/1495131010 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5164011470 con 0x7f517c161fc0 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f51797fa640 1 -- 172.21.15.59:0/1495131010 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51640118c0 con 0x7f517c161fc0 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f517c156910 con 0x7f517c161fc0 2024-06-29T14:34:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.905+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f517c156ed0 con 0x7f517c161fc0 2024-06-29T14:34:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.906+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f517c0ca8d0 con 0x7f517c161fc0 2024-06-29T14:34:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.907+0000 7f51797fa640 1 -- 172.21.15.59:0/1495131010 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f5164011a60 con 0x7f517c161fc0 2024-06-29T14:34:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.908+0000 7f51797fa640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f515c0763f0 0x7f515c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.908+0000 7f51797fa640 1 -- 172.21.15.59:0/1495131010 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5980+0+0 (secure 0 0 0) 0x7f51640986d0 con 0x7f517c161fc0 2024-06-29T14:34:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.908+0000 7f5180a55640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f515c0763f0 0x7f515c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:52.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.909+0000 7f5180a55640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f515c0763f0 0x7f515c078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f516c033000 tx=0x7f516c004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:52.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:52.912+0000 7f51797fa640 1 -- 172.21.15.59:0/1495131010 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5164016030 con 0x7f517c161fc0 2024-06-29T14:34:53.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.102+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f517c0d05c0 con 0x7f517c161fc0 2024-06-29T14:34:53.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.102+0000 7f51797fa640 1 -- 172.21.15.59:0/1495131010 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) ==== 93+0+10 (secure 0 0 0) 0x7f5164061bf0 con 0x7f517c161fc0 2024-06-29T14:34:53.099 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:34:53.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f515c0763f0 msgr2=0x7f515c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f515c0763f0 0x7f515c078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f516c033000 tx=0x7f516c004730 comp rx=0 tx=0).stop 2024-06-29T14:34:53.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c161fc0 msgr2=0x7f517c154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c161fc0 0x7f517c154520 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f516400e9c0 tx=0x7f516400ee80 comp rx=0 tx=0).stop 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 shutdown_connections 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f515c0763f0 0x7f515c078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517c162770 0x7f517c159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f517c161fc0 0x7f517c154520 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 --2- 172.21.15.59:0/1495131010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f517c15e0a0 0x7f517c153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 >> 172.21.15.59:0/1495131010 conn(0x7f517c0d45b0 msgr2=0x7f517c068330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 shutdown_connections 2024-06-29T14:34:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.104+0000 7f5182cb9640 1 -- 172.21.15.59:0/1495131010 wait complete. 2024-06-29T14:34:53.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-06-29T14:34:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.232+0000 7f8ef2b03640 1 -- 172.21.15.59:0/3735226132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec15e0a0 msgr2=0x7f8eec161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.232+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/3735226132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec15e0a0 0x7f8eec161a80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8eec0b35d0 tx=0x7f8ed802a4b0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.233+0000 7f8ef2b03640 1 -- 172.21.15.59:0/3735226132 shutdown_connections 2024-06-29T14:34:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.233+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/3735226132 >> v2:172.21.15.153:3300/0 conn(0x7f8eec161fc0 0x7f8eec0d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.233+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/3735226132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec15e0a0 0x7f8eec161a80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.233+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/3735226132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eec162770 0x7f8eec0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.233+0000 7f8ef2b03640 1 -- 172.21.15.59:0/3735226132 >> 172.21.15.59:0/3735226132 conn(0x7f8eec0d45b0 msgr2=0x7f8eec0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:53.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.233+0000 7f8ef2b03640 1 -- 172.21.15.59:0/3735226132 shutdown_connections 2024-06-29T14:34:53.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.233+0000 7f8ef2b03640 1 -- 172.21.15.59:0/3735226132 wait complete. 2024-06-29T14:34:53.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.234+0000 7f8ef2b03640 1 Processor -- start 2024-06-29T14:34:53.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.234+0000 7f8ef2b03640 1 -- start start 2024-06-29T14:34:53.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef2b03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec15e0a0 0x7f8eec15a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef2b03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eec161fc0 0x7f8eec0c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef2b03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec162770 0x7f8eec0cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef2b03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8eec0d2f40 con 0x7f8eec162770 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef2b03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8eec0d2dc0 con 0x7f8eec15e0a0 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef2b03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8eec0d30c0 con 0x7f8eec161fc0 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef10a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec162770 0x7f8eec0cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef10a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec162770 0x7f8eec0cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60478/0 (socket says 172.21.15.59:60478) 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef10a0640 1 -- 172.21.15.59:0/1277630197 learned_addr learned my addr 172.21.15.59:0/1277630197 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef089f640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec15e0a0 0x7f8eec15a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8eebfff640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eec161fc0 0x7f8eec0c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef10a0640 1 -- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eec161fc0 msgr2=0x7f8eec0c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef10a0640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eec161fc0 0x7f8eec0c8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef10a0640 1 -- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec15e0a0 msgr2=0x7f8eec15a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef10a0640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec15e0a0 0x7f8eec15a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.236+0000 7f8ef10a0640 1 -- 172.21.15.59:0/1277630197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eec0c76c0 con 0x7f8eec162770 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.237+0000 7f8ef10a0640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec162770 0x7f8eec0cd040 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4005f40 tx=0x7f8ee400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.237+0000 7f8ee97fa640 1 -- 172.21.15.59:0/1277630197 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee400cd80 con 0x7f8eec162770 2024-06-29T14:34:53.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.237+0000 7f8ee97fa640 1 -- 172.21.15.59:0/1277630197 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ee4004b20 con 0x7f8eec162770 2024-06-29T14:34:53.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.237+0000 7f8ee97fa640 1 -- 172.21.15.59:0/1277630197 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4005020 con 0x7f8eec162770 2024-06-29T14:34:53.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.237+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eec0c79a0 con 0x7f8eec162770 2024-06-29T14:34:53.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.237+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eec0c7f60 con 0x7f8eec162770 2024-06-29T14:34:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.238+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eec0b8ca0 con 0x7f8eec162770 2024-06-29T14:34:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.239+0000 7f8ee97fa640 1 -- 172.21.15.59:0/1277630197 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f8ee40051c0 con 0x7f8eec162770 2024-06-29T14:34:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.239+0000 7f8ee97fa640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f8ed00763f0 0x7f8ed0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.239+0000 7f8ee97fa640 1 -- 172.21.15.59:0/1277630197 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5980+0+0 (secure 0 0 0) 0x7f8ee4098500 con 0x7f8eec162770 2024-06-29T14:34:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.239+0000 7f8ef089f640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f8ed00763f0 0x7f8ed0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.240+0000 7f8ef089f640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f8ed00763f0 0x7f8ed0078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009b30 tx=0x7f8ee0005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.243+0000 7f8ee97fa640 1 -- 172.21.15.59:0/1277630197 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8ee4061450 con 0x7f8eec162770 2024-06-29T14:34:53.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:53.511+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) -- 0x7f8eec0c7450 con 0x7f8eec162770 2024-06-29T14:34:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.432+0000 7f8ee97fa640 1 -- 172.21.15.59:0/1277630197 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v478) ==== 157+0+0 (secure 0 0 0) 0x7f8ee4065750 con 0x7f8eec162770 2024-06-29T14:34:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f8ed00763f0 msgr2=0x7f8ed0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f8ed00763f0 0x7f8ed0078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009b30 tx=0x7f8ee0005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec162770 msgr2=0x7f8eec0cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec162770 0x7f8eec0cd040 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4005f40 tx=0x7f8ee400ee60 comp rx=0 tx=0).stop 2024-06-29T14:34:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 shutdown_connections 2024-06-29T14:34:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f8ed00763f0 0x7f8ed0078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eec162770 0x7f8eec0cd040 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eec161fc0 0x7f8eec0c8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 --2- 172.21.15.59:0/1277630197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec15e0a0 0x7f8eec15a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 >> 172.21.15.59:0/1277630197 conn(0x7f8eec0d45b0 msgr2=0x7f8eec0b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:54.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 shutdown_connections 2024-06-29T14:34:54.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.435+0000 7f8ef2b03640 1 -- 172.21.15.59:0/1277630197 wait complete. 2024-06-29T14:34:54.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:34:54.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.573+0000 7fb208568640 1 -- 172.21.15.59:0/3133292060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000b40a0 msgr2=0x7fb20015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.573+0000 7fb208568640 1 --2- 172.21.15.59:0/3133292060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000b40a0 0x7fb20015e8d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009830 tx=0x7fb1f002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.574+0000 7fb208568640 1 -- 172.21.15.59:0/3133292060 shutdown_connections 2024-06-29T14:34:54.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.574+0000 7fb208568640 1 --2- 172.21.15.59:0/3133292060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000b40a0 0x7fb20015e8d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.574+0000 7fb208568640 1 -- 172.21.15.59:0/3133292060 >> 172.21.15.59:0/3133292060 conn(0x7fb2000d3ee0 msgr2=0x7fb2000d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:54.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.574+0000 7fb208568640 1 -- 172.21.15.59:0/3133292060 shutdown_connections 2024-06-29T14:34:54.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.574+0000 7fb208568640 1 -- 172.21.15.59:0/3133292060 wait complete. 2024-06-29T14:34:54.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.575+0000 7fb208568640 1 Processor -- start 2024-06-29T14:34:54.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.575+0000 7fb208568640 1 -- start start 2024-06-29T14:34:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb208568640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000d23c0 0x7fb200150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb208568640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20015ee10 0x7fb200157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb208568640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20014b140 0x7fb20014ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb208568640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb200160450 con 0x7fb2000d23c0 2024-06-29T14:34:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb208568640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb2001602d0 con 0x7fb20014b140 2024-06-29T14:34:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb208568640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb2001605d0 con 0x7fb20015ee10 2024-06-29T14:34:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000d23c0 0x7fb200150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000d23c0 0x7fb200150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60508/0 (socket says 172.21.15.59:60508) 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 -- 172.21.15.59:0/4281668903 learned_addr learned my addr 172.21.15.59:0/4281668903 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206b05640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20014b140 0x7fb20014ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb205b03640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20015ee10 0x7fb200157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 -- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20015ee10 msgr2=0x7fb200157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20015ee10 0x7fb200157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 -- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20014b140 msgr2=0x7fb20014ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20014b140 0x7fb20014ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 -- 172.21.15.59:0/4281668903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb200152ee0 con 0x7fb2000d23c0 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb206304640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000d23c0 0x7fb200150270 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00c400 tx=0x7fb1fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb1f6ffd640 1 -- 172.21.15.59:0/4281668903 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc017070 con 0x7fb2000d23c0 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.577+0000 7fb1f6ffd640 1 -- 172.21.15.59:0/4281668903 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1fc0053c0 con 0x7fb2000d23c0 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.578+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb200156b90 con 0x7fb2000d23c0 2024-06-29T14:34:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.578+0000 7fb1f6ffd640 1 -- 172.21.15.59:0/4281668903 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc003830 con 0x7fb2000d23c0 2024-06-29T14:34:54.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.578+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb200157150 con 0x7fb2000d23c0 2024-06-29T14:34:54.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.579+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb20015e8d0 con 0x7fb2000d23c0 2024-06-29T14:34:54.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.579+0000 7fb1f6ffd640 1 -- 172.21.15.59:0/4281668903 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fb1fc01a070 con 0x7fb2000d23c0 2024-06-29T14:34:54.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.580+0000 7fb1f6ffd640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fb1e4076420 0x7fb1e4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.580+0000 7fb1f6ffd640 1 -- 172.21.15.59:0/4281668903 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5980+0+0 (secure 0 0 0) 0x7fb1fc096fb0 con 0x7fb2000d23c0 2024-06-29T14:34:54.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.580+0000 7fb205b03640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fb1e4076420 0x7fb1e4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.581+0000 7fb205b03640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fb1e4076420 0x7fb1e4078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0034000 tx=0x7fb1f0004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:54.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.584+0000 7fb1f6ffd640 1 -- 172.21.15.59:0/4281668903 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb1fc0605c0 con 0x7fb2000d23c0 2024-06-29T14:34:54.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.775+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fb2000d2950 con 0x7fb2000d23c0 2024-06-29T14:34:54.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.775+0000 7fb1f6ffd640 1 -- 172.21.15.59:0/4281668903 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v31) ==== 59+0+1998 (secure 0 0 0) 0x7fb1fc05fec0 con 0x7fb2000d23c0 2024-06-29T14:34:54.772 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T14:34:54.773 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:34:54.774 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:34:54.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fb1e4076420 msgr2=0x7fb1e4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fb1e4076420 0x7fb1e4078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0034000 tx=0x7fb1f0004730 comp rx=0 tx=0).stop 2024-06-29T14:34:54.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000d23c0 msgr2=0x7fb200150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000d23c0 0x7fb200150270 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00c400 tx=0x7fb1fc00c8c0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 shutdown_connections 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fb1e4076420 0x7fb1e4078930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb20014b140 0x7fb20014ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb20015ee10 0x7fb200157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.777+0000 7fb208568640 1 --2- 172.21.15.59:0/4281668903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2000d23c0 0x7fb200150270 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.778+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 >> 172.21.15.59:0/4281668903 conn(0x7fb2000d3ee0 msgr2=0x7fb200151260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.778+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 shutdown_connections 2024-06-29T14:34:54.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.778+0000 7fb208568640 1 -- 172.21.15.59:0/4281668903 wait complete. 2024-06-29T14:34:54.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:34:54.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.910+0000 7f58b5a47640 1 -- 172.21.15.59:0/33430134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5898004040 msgr2=0x7f5898004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.910+0000 7f58b5a47640 1 --2- 172.21.15.59:0/33430134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5898004040 0x7f5898004800 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009b20 tx=0x7f5898005830 comp rx=0 tx=0).stop 2024-06-29T14:34:54.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.910+0000 7f58af7fe640 1 -- 172.21.15.59:0/33430134 reap_dead start 2024-06-29T14:34:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.910+0000 7f58b5a47640 1 -- 172.21.15.59:0/33430134 shutdown_connections 2024-06-29T14:34:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.910+0000 7f58b5a47640 1 -- 172.21.15.59:0/33430134 >> 172.21.15.59:0/33430134 conn(0x7f58b00b5610 msgr2=0x7f58b00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.910+0000 7f58b5a47640 1 -- 172.21.15.59:0/33430134 shutdown_connections 2024-06-29T14:34:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.910+0000 7f58b5a47640 1 -- 172.21.15.59:0/33430134 wait complete. 2024-06-29T14:34:54.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.912+0000 7f58b5a47640 1 Processor -- start 2024-06-29T14:34:54.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.912+0000 7f58b5a47640 1 -- start start 2024-06-29T14:34:54.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58b5a47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5898004040 0x7f58b01619b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58b5a47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b00aa4f0 0x7f58b0160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58b5a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b015c2d0 0x7f58b01f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58b5a47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f58b0162fa0 con 0x7f58b015c2d0 2024-06-29T14:34:54.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58b5a47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58b0162e20 con 0x7f58b00aa4f0 2024-06-29T14:34:54.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58b5a47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f58b00b7bd0 con 0x7f5898004040 2024-06-29T14:34:54.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58af7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b015c2d0 0x7f58b01f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58af7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b015c2d0 0x7f58b01f85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60516/0 (socket says 172.21.15.59:60516) 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58af7fe640 1 -- 172.21.15.59:0/3566718003 learned_addr learned my addr 172.21.15.59:0/3566718003 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58ae7fc640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b00aa4f0 0x7f58b0160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.913+0000 7f58aeffd640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5898004040 0x7f58b01619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f58af7fe640 1 -- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5898004040 msgr2=0x7f58b01619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f58af7fe640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5898004040 0x7f58b01619b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f58af7fe640 1 -- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b00aa4f0 msgr2=0x7f58b0160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f58af7fe640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b00aa4f0 0x7f58b0160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f58af7fe640 1 -- 172.21.15.59:0/3566718003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b015efd0 con 0x7f58b015c2d0 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f58af7fe640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b015c2d0 0x7f58b01f85f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f58b008ccc0 tx=0x7f58a8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f5897fff640 1 -- 172.21.15.59:0/3566718003 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a801f070 con 0x7f58b015c2d0 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f5897fff640 1 -- 172.21.15.59:0/3566718003 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58a800fae0 con 0x7f58b015c2d0 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b015f2b0 con 0x7f58b015c2d0 2024-06-29T14:34:54.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f5897fff640 1 -- 172.21.15.59:0/3566718003 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a8012450 con 0x7f58b015c2d0 2024-06-29T14:34:54.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.914+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b015f870 con 0x7f58b015c2d0 2024-06-29T14:34:54.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.915+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b01622c0 con 0x7f58b015c2d0 2024-06-29T14:34:54.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.916+0000 7f5897fff640 1 -- 172.21.15.59:0/3566718003 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f58a8022070 con 0x7f58b015c2d0 2024-06-29T14:34:54.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.916+0000 7f5897fff640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f589c0763f0 0x7f589c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.916+0000 7f5897fff640 1 -- 172.21.15.59:0/3566718003 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5980+0+0 (secure 0 0 0) 0x7f58a8098170 con 0x7f58b015c2d0 2024-06-29T14:34:54.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.916+0000 7f58aeffd640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f589c0763f0 0x7f589c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.917+0000 7f58aeffd640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f589c0763f0 0x7f589c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f58a0001370 tx=0x7f58a0001300 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:54.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:54.921+0000 7f5897fff640 1 -- 172.21.15.59:0/3566718003 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f58a80611e0 con 0x7f58b015c2d0 2024-06-29T14:34:55.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.118+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f58b0161ef0 con 0x7f58b015c2d0 2024-06-29T14:34:55.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.121+0000 7f5897fff640 1 -- 172.21.15.59:0/3566718003 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f58a80654e0 con 0x7f58b015c2d0 2024-06-29T14:34:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f589c0763f0 msgr2=0x7f589c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f589c0763f0 0x7f589c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f58a0001370 tx=0x7f58a0001300 comp rx=0 tx=0).stop 2024-06-29T14:34:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b015c2d0 msgr2=0x7f58b01f85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b015c2d0 0x7f58b01f85f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f58b008ccc0 tx=0x7f58a8011aa0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 shutdown_connections 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f589c0763f0 0x7f589c078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b015c2d0 0x7f58b01f85f0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58b00aa4f0 0x7f58b0160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 --2- 172.21.15.59:0/3566718003 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5898004040 0x7f58b01619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 >> 172.21.15.59:0/3566718003 conn(0x7f58b00b5610 msgr2=0x7f58b0068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 shutdown_connections 2024-06-29T14:34:55.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.124+0000 7f58b5a47640 1 -- 172.21.15.59:0/3566718003 wait complete. 2024-06-29T14:34:55.133 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:34:55.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:34:55.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:34:55.226 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:34:55.227 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:34:55.227 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:55.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:34:55.243 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:34:55.243 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:55.261 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:34:55.261 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:34:55.261 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:55.299 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:34:55.299 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:34:55.299 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:55.319 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:34:55.319 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:34:55.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 -- 172.21.15.59:0/3819718539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc006d2f0 msgr2=0x7f5fc006d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 --2- 172.21.15.59:0/3819718539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc006d2f0 0x7f5fc006d700 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5fc006ddf0 tx=0x7f5fa802a540 comp rx=0 tx=0).stop 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 -- 172.21.15.59:0/3819718539 shutdown_connections 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 --2- 172.21.15.59:0/3819718539 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fc00732b0 0x7f5fc00736c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 --2- 172.21.15.59:0/3819718539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc006d2f0 0x7f5fc006d700 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 --2- 172.21.15.59:0/3819718539 >> v2:172.21.15.153:3300/0 conn(0x7f5fc00cd9b0 0x7f5fc015bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 -- 172.21.15.59:0/3819718539 >> 172.21.15.59:0/3819718539 conn(0x7f5fc00b24c0 msgr2=0x7f5fc00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 -- 172.21.15.59:0/3819718539 shutdown_connections 2024-06-29T14:34:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.459+0000 7f5fc530a640 1 -- 172.21.15.59:0/3819718539 wait complete. 2024-06-29T14:34:55.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.461+0000 7f5fc530a640 1 Processor -- start 2024-06-29T14:34:55.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.461+0000 7f5fc530a640 1 -- start start 2024-06-29T14:34:55.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.462+0000 7f5fc530a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fc006d2f0 0x7f5fc015a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.462+0000 7f5fc530a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc00732b0 0x7f5fc01e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.462+0000 7f5fc530a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fc00cd9b0 0x7f5fc0132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.462+0000 7f5fc530a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc01448f0 con 0x7f5fc00732b0 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.462+0000 7f5fc530a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc0144770 con 0x7f5fc00cd9b0 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.462+0000 7f5fc530a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5fc0073c00 con 0x7f5fc006d2f0 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.462+0000 7f5fbe7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc00732b0 0x7f5fc01e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbe7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc00732b0 0x7f5fc01e38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60530/0 (socket says 172.21.15.59:60530) 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbe7fc640 1 -- 172.21.15.59:0/45863296 learned_addr learned my addr 172.21.15.59:0/45863296 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbeffd640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fc006d2f0 0x7f5fc015a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbf7fe640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fc00cd9b0 0x7f5fc0132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbe7fc640 1 -- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fc006d2f0 msgr2=0x7f5fc015a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbe7fc640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fc006d2f0 0x7f5fc015a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbe7fc640 1 -- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fc00cd9b0 msgr2=0x7f5fc0132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbe7fc640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fc00cd9b0 0x7f5fc0132d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbe7fc640 1 -- 172.21.15.59:0/45863296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc0133250 con 0x7f5fc00732b0 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fbe7fc640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc00732b0 0x7f5fc01e38f0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8034040 tx=0x7f5fa80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fb77fe640 1 -- 172.21.15.59:0/45863296 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa803e030 con 0x7f5fc00732b0 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fb77fe640 1 -- 172.21.15.59:0/45863296 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fa8003ca0 con 0x7f5fc00732b0 2024-06-29T14:34:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fb77fe640 1 -- 172.21.15.59:0/45863296 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8035040 con 0x7f5fc00732b0 2024-06-29T14:34:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc01e42d0 con 0x7f5fc00732b0 2024-06-29T14:34:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.463+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc01e4890 con 0x7f5fc00732b0 2024-06-29T14:34:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.465+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc01e4020 con 0x7f5fc00732b0 2024-06-29T14:34:55.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.465+0000 7f5fb77fe640 1 -- 172.21.15.59:0/45863296 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f5fa8041030 con 0x7f5fc00732b0 2024-06-29T14:34:55.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.466+0000 7f5fb77fe640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f5fa00763f0 0x7f5fa0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.466+0000 7f5fb77fe640 1 -- 172.21.15.59:0/45863296 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5980+0+0 (secure 0 0 0) 0x7f5fa80b8130 con 0x7f5fc00732b0 2024-06-29T14:34:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.466+0000 7f5fbeffd640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f5fa00763f0 0x7f5fa0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.467+0000 7f5fbeffd640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f5fa00763f0 0x7f5fa0078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0005fd0 tx=0x7f5fb0009500 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:55.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.470+0000 7f5fb77fe640 1 -- 172.21.15.59:0/45863296 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5fa8081080 con 0x7f5fc00732b0 2024-06-29T14:34:55.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.698+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fc0074040 con 0x7f5fc00732b0 2024-06-29T14:34:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.699+0000 7f5fb77fe640 1 -- 172.21.15.59:0/45863296 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+3914 (secure 0 0 0) 0x7f5fa8085380 con 0x7f5fc00732b0 2024-06-29T14:34:55.696 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:34:55.696 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":21,"btime":"2024-06-29T14:34:49:403615+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/634859216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":634859216},{"type":"v1","addr":"172.21.15.153:6837","nonce":634859216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3661382082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3661382082},{"type":"v1","addr":"172.21.15.153:6836","nonce":3661382082}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4012387069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4012387069},{"type":"v1","addr":"172.21.15.59:6833","nonce":4012387069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1024462192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1024462192},{"type":"v1","addr":"172.21.15.59:6835","nonce":1024462192}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[]} 2024-06-29T14:34:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 21 2024-06-29T14:34:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f5fa00763f0 msgr2=0x7f5fa0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f5fa00763f0 0x7f5fa0078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0005fd0 tx=0x7f5fb0009500 comp rx=0 tx=0).stop 2024-06-29T14:34:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc00732b0 msgr2=0x7f5fc01e38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc00732b0 0x7f5fc01e38f0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8034040 tx=0x7f5fa80038a0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 shutdown_connections 2024-06-29T14:34:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f5fa00763f0 0x7f5fa0078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fc00cd9b0 0x7f5fc0132d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc00732b0 0x7f5fc01e38f0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 --2- 172.21.15.59:0/45863296 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5fc006d2f0 0x7f5fc015a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 >> 172.21.15.59:0/45863296 conn(0x7f5fc00b24c0 msgr2=0x7f5fc015c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 shutdown_connections 2024-06-29T14:34:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:34:55.701+0000 7f5fc530a640 1 -- 172.21.15.59:0/45863296 wait complete. 2024-06-29T14:34:55.699 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:55.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:55.710 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:34:55.710 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:34:55.710 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:34:55.710 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:34:55.711 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:34:55.711 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:34:55.711 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:34:55.711 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:34:55.711 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:34:55.711 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:34:55.711 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:34:55.711 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:34:55.712 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:34:55.712 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:34:55.712 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:34:55.712 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:34:55.759 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:34:55.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:34:55.760 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:34:55.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:34:55.762 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:34:55.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:34:55.803 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:34:55.804 DEBUG:teuthology.parallel:result is None 2024-06-29T14:34:55.804 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:34:55.805 DEBUG:teuthology.parallel:result is None 2024-06-29T14:34:55.806 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:34:55.806 DEBUG:teuthology.parallel:result is None 2024-06-29T14:34:55.892 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:34:55.892 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:34:55.895 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:35:00.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-06-29T14:35:01.759 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:35:01.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:35:01.763 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:35:01.763 DEBUG:teuthology.parallel:result is None 2024-06-29T14:35:01.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:35:01.860 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 -- 172.21.15.59:0/2166494342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c15e0a0 msgr2=0x7f2e8c161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 --2- 172.21.15.59:0/2166494342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c15e0a0 0x7f2e8c161a80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0b35d0 tx=0x7f2e7402a4b0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 -- 172.21.15.59:0/2166494342 shutdown_connections 2024-06-29T14:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 --2- 172.21.15.59:0/2166494342 >> v2:172.21.15.153:3300/0 conn(0x7f2e8c161fc0 0x7f2e8c0d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 --2- 172.21.15.59:0/2166494342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c15e0a0 0x7f2e8c161a80 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 --2- 172.21.15.59:0/2166494342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8c162770 0x7f2e8c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 -- 172.21.15.59:0/2166494342 >> 172.21.15.59:0/2166494342 conn(0x7f2e8c0d45b0 msgr2=0x7f2e8c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 -- 172.21.15.59:0/2166494342 shutdown_connections 2024-06-29T14:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.900+0000 7f2e918c1640 1 -- 172.21.15.59:0/2166494342 wait complete. 2024-06-29T14:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.902+0000 7f2e918c1640 1 Processor -- start 2024-06-29T14:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.902+0000 7f2e918c1640 1 -- start start 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e918c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c15e0a0 0x7f2e8c151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e918c1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8c161fc0 0x7f2e8c0c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e918c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c162770 0x7f2e8c0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e918c1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c0d2f40 con 0x7f2e8c162770 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e918c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c0d2dc0 con 0x7f2e8c15e0a0 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e918c1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8c0d30c0 con 0x7f2e8c161fc0 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e8b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c162770 0x7f2e8c0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e8b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c162770 0x7f2e8c0ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60572/0 (socket says 172.21.15.59:60572) 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e8affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c15e0a0 0x7f2e8c151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e8b7fe640 1 -- 172.21.15.59:0/1017668280 learned_addr learned my addr 172.21.15.59:0/1017668280 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.903+0000 7f2e83fff640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8c161fc0 0x7f2e8c0c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e8b7fe640 1 -- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8c161fc0 msgr2=0x7f2e8c0c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e8b7fe640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8c161fc0 0x7f2e8c0c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e8b7fe640 1 -- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c15e0a0 msgr2=0x7f2e8c151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e8b7fe640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c15e0a0 0x7f2e8c151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e8b7fe640 1 -- 172.21.15.59:0/1017668280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c1f5450 con 0x7f2e8c162770 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e8b7fe640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c162770 0x7f2e8c0ccd00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2e84005f40 tx=0x7f2e8400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e88ff9640 1 -- 172.21.15.59:0/1017668280 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8400cd80 con 0x7f2e8c162770 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e88ff9640 1 -- 172.21.15.59:0/1017668280 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e84004b20 con 0x7f2e8c162770 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e8c1f5730 con 0x7f2e8c162770 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e88ff9640 1 -- 172.21.15.59:0/1017668280 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e84005020 con 0x7f2e8c162770 2024-06-29T14:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.904+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e8c1f5cf0 con 0x7f2e8c162770 2024-06-29T14:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.905+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e8c0cf210 con 0x7f2e8c162770 2024-06-29T14:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.906+0000 7f2e88ff9640 1 -- 172.21.15.59:0/1017668280 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f2e840051c0 con 0x7f2e8c162770 2024-06-29T14:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.906+0000 7f2e88ff9640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2e6c0763f0 0x7f2e6c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.906+0000 7f2e88ff9640 1 -- 172.21.15.59:0/1017668280 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5980+0+0 (secure 0 0 0) 0x7f2e84098520 con 0x7f2e8c162770 2024-06-29T14:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.907+0000 7f2e8affd640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2e6c0763f0 0x7f2e6c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.907+0000 7f2e8affd640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2e6c0763f0 0x7f2e6c078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009b30 tx=0x7f2e7c005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:01.911+0000 7f2e88ff9640 1 -- 172.21.15.59:0/1017668280 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2e84061500 con 0x7f2e8c162770 2024-06-29T14:35:02.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:02.116+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f2e8c15d600 con 0x7f2e8c162770 2024-06-29T14:35:03.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.104+0000 7f2e88ff9640 1 -- 172.21.15.59:0/1017668280 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v67) ==== 97+0+0 (secure 0 0 0) 0x7f2e84065800 con 0x7f2e8c162770 2024-06-29T14:35:03.101 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T14:35:03.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.106+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2e6c0763f0 msgr2=0x7f2e6c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.106+0000 7f2e918c1640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2e6c0763f0 0x7f2e6c078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009b30 tx=0x7f2e7c005f90 comp rx=0 tx=0).stop 2024-06-29T14:35:03.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.106+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c162770 msgr2=0x7f2e8c0ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.106+0000 7f2e918c1640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c162770 0x7f2e8c0ccd00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2e84005f40 tx=0x7f2e8400ee60 comp rx=0 tx=0).stop 2024-06-29T14:35:03.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.107+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 shutdown_connections 2024-06-29T14:35:03.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.107+0000 7f2e918c1640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2e6c0763f0 0x7f2e6c078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.107+0000 7f2e918c1640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e8c162770 0x7f2e8c0ccd00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.107+0000 7f2e918c1640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e8c161fc0 0x7f2e8c0c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.107+0000 7f2e918c1640 1 --2- 172.21.15.59:0/1017668280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c15e0a0 0x7f2e8c151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.107+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 >> 172.21.15.59:0/1017668280 conn(0x7f2e8c0d45b0 msgr2=0x7f2e8c0b7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.107+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 shutdown_connections 2024-06-29T14:35:03.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.107+0000 7f2e918c1640 1 -- 172.21.15.59:0/1017668280 wait complete. 2024-06-29T14:35:03.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:35:03.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.246+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1481776662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 msgr2=0x7f1f28004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.246+0000 7f1f42f9a640 1 --2- 172.21.15.59:0/1481776662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 0x7f1f28004c10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c005fd0 tx=0x7f1f28005830 comp rx=0 tx=0).stop 2024-06-29T14:35:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.246+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1481776662 shutdown_connections 2024-06-29T14:35:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.246+0000 7f1f42f9a640 1 --2- 172.21.15.59:0/1481776662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 0x7f1f28004c10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.246+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1481776662 >> 172.21.15.59:0/1481776662 conn(0x7f1f3c0d3ee0 msgr2=0x7f1f3c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.246+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1481776662 shutdown_connections 2024-06-29T14:35:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.246+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1481776662 wait complete. 2024-06-29T14:35:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.248+0000 7f1f42f9a640 1 Processor -- start 2024-06-29T14:35:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.248+0000 7f1f42f9a640 1 -- start start 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.249+0000 7f1f42f9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 0x7f1f3c1f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.249+0000 7f1f42f9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f3c1f54a0 0x7f1f3c0c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f42f9a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3c152f80 0x7f1f3c0c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f42f9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c15ec60 con 0x7f1f28004800 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f42f9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c15eae0 con 0x7f1f3c1f54a0 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f42f9a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1f3c0b3990 con 0x7f1f3c152f80 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 0x7f1f3c1f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 0x7f1f3c1f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37682/0 (socket says 172.21.15.59:37682) 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 -- 172.21.15.59:0/1814140251 learned_addr learned my addr 172.21.15.59:0/1814140251 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f3bfff640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f3c1f54a0 0x7f1f3c0c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f41537640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3c152f80 0x7f1f3c0c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 -- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3c152f80 msgr2=0x7f1f3c0c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3c152f80 0x7f1f3c0c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 -- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f3c1f54a0 msgr2=0x7f1f3c0c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f3c1f54a0 0x7f1f3c0c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 -- 172.21.15.59:0/1814140251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3c0c86a0 con 0x7f1f28004800 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f3bfff640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f3c1f54a0 0x7f1f3c0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f41537640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3c152f80 0x7f1f3c0c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f40d36640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 0x7f1f3c1f4f60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1f34001be0 tx=0x7f1f340039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f397fa640 1 -- 172.21.15.59:0/1814140251 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f34015070 con 0x7f1f28004800 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f397fa640 1 -- 172.21.15.59:0/1814140251 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f34011920 con 0x7f1f28004800 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f3c0c8980 con 0x7f1f28004800 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f397fa640 1 -- 172.21.15.59:0/1814140251 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f34011d70 con 0x7f1f28004800 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.250+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f3c0c8f40 con 0x7f1f28004800 2024-06-29T14:35:03.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.252+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f3c0b3ef0 con 0x7f1f28004800 2024-06-29T14:35:03.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.252+0000 7f1f397fa640 1 -- 172.21.15.59:0/1814140251 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f1f34018070 con 0x7f1f28004800 2024-06-29T14:35:03.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.253+0000 7f1f397fa640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f1f1c0763f0 0x7f1f1c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.253+0000 7f1f397fa640 1 -- 172.21.15.59:0/1814140251 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5249+0+0 (secure 0 0 0) 0x7f1f34098690 con 0x7f1f28004800 2024-06-29T14:35:03.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.253+0000 7f1f3bfff640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f1f1c0763f0 0x7f1f1c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.254+0000 7f1f3bfff640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f1f1c0763f0 0x7f1f1c078900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c035000 tx=0x7f1f2c02d040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.258+0000 7f1f397fa640 1 -- 172.21.15.59:0/1814140251 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1f34061f10 con 0x7f1f28004800 2024-06-29T14:35:03.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.521+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f1f3c0d2950 con 0x7f1f28004800 2024-06-29T14:35:03.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.522+0000 7f1f397fa640 1 -- 172.21.15.59:0/1814140251 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7f1f34061810 con 0x7f1f28004800 2024-06-29T14:35:03.519 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:03.519 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:35:03.519 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T14:35:03.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T14:35:03.520 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:35:03.521 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:35:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:35:03.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:35:03.525 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.525 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.525 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T14:35:03.525 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.526 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T14:35:03.530 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:35:03.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:35:03.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:35:03.534 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:35:03.534 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.534 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:03.534 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T14:35:03.534 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:35:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f1f1c0763f0 msgr2=0x7f1f1c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f1f1c0763f0 0x7f1f1c078900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c035000 tx=0x7f1f2c02d040 comp rx=0 tx=0).stop 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 msgr2=0x7f1f3c1f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 0x7f1f3c1f4f60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1f34001be0 tx=0x7f1f340039c0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 shutdown_connections 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f1f1c0763f0 0x7f1f1c078900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f3c152f80 0x7f1f3c0c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f3c1f54a0 0x7f1f3c0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 --2- 172.21.15.59:0/1814140251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f28004800 0x7f1f3c1f4f60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 >> 172.21.15.59:0/1814140251 conn(0x7f1f3c0d3ee0 msgr2=0x7f1f3c1605d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 shutdown_connections 2024-06-29T14:35:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.524+0000 7f1f42f9a640 1 -- 172.21.15.59:0/1814140251 wait complete. 2024-06-29T14:35:03.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:35:03.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.657+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/3380733423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80ad6b0 msgr2=0x7fe1e815b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.657+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/3380733423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80ad6b0 0x7fe1e815b140 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8005fd0 tx=0x7fe1d802aa90 comp rx=0 tx=0).stop 2024-06-29T14:35:03.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.657+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/3380733423 shutdown_connections 2024-06-29T14:35:03.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.657+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/3380733423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80ad6b0 0x7fe1e815b140 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.657+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/3380733423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1e80cb490 0x7fe1e8073be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.657+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/3380733423 >> 172.21.15.59:0/3380733423 conn(0x7fe1e80b1ea0 msgr2=0x7fe1e80b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.657+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/3380733423 shutdown_connections 2024-06-29T14:35:03.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.657+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/3380733423 wait complete. 2024-06-29T14:35:03.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.659+0000 7fe1ecf5b640 1 Processor -- start 2024-06-29T14:35:03.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.659+0000 7fe1ecf5b640 1 -- start start 2024-06-29T14:35:03.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.660+0000 7fe1ecf5b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80cb490 0x7fe1e806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.660+0000 7fe1ecf5b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e80cc060 0x7fe1e80b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.660+0000 7fe1ecf5b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1e8151130 0x7fe1e80b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1ecf5b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e815bb20 con 0x7fe1e80cb490 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1ecf5b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e815b9a0 con 0x7fe1e80cc060 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1ecf5b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe1e815bca0 con 0x7fe1e8151130 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80cb490 0x7fe1e806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80cb490 0x7fe1e806ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37706/0 (socket says 172.21.15.59:37706) 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 -- 172.21.15.59:0/2688107089 learned_addr learned my addr 172.21.15.59:0/2688107089 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e6d9d640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1e8151130 0x7fe1e80b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e5d9b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e80cc060 0x7fe1e80b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 -- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1e8151130 msgr2=0x7fe1e80b5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1e8151130 0x7fe1e80b5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 -- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e80cc060 msgr2=0x7fe1e80b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e80cc060 0x7fe1e80b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 -- 172.21.15.59:0/2688107089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e806f1d0 con 0x7fe1e80cb490 2024-06-29T14:35:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e5d9b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e80cc060 0x7fe1e80b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1e659c640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80cb490 0x7fe1e806ec90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000d740 tx=0x7fe1e000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1deffd640 1 -- 172.21.15.59:0/2688107089 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0017070 con 0x7fe1e80cb490 2024-06-29T14:35:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1deffd640 1 -- 172.21.15.59:0/2688107089 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1e0009cc0 con 0x7fe1e80cb490 2024-06-29T14:35:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e806f6e0 con 0x7fe1e80cb490 2024-06-29T14:35:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1deffd640 1 -- 172.21.15.59:0/2688107089 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e00106c0 con 0x7fe1e80cb490 2024-06-29T14:35:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.661+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e806fca0 con 0x7fe1e80cb490 2024-06-29T14:35:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.662+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e815b140 con 0x7fe1e80cb490 2024-06-29T14:35:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.663+0000 7fe1deffd640 1 -- 172.21.15.59:0/2688107089 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fe1e001a070 con 0x7fe1e80cb490 2024-06-29T14:35:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.664+0000 7fe1deffd640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe1c80763f0 0x7fe1c8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.664+0000 7fe1deffd640 1 -- 172.21.15.59:0/2688107089 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5249+0+0 (secure 0 0 0) 0x7fe1e00979d0 con 0x7fe1e80cb490 2024-06-29T14:35:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.664+0000 7fe1e5d9b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe1c80763f0 0x7fe1c8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.665+0000 7fe1e5d9b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe1c80763f0 0x7fe1c8078900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8159e60 tx=0x7fe1d0009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.669+0000 7fe1deffd640 1 -- 172.21.15.59:0/2688107089 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe1e0061220 con 0x7fe1e80cb490 2024-06-29T14:35:03.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.860+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1e80cc780 con 0x7fe1e80cb490 2024-06-29T14:35:03.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.861+0000 7fe1deffd640 1 -- 172.21.15.59:0/2688107089 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+13176 (secure 0 0 0) 0x7fe1e0060b20 con 0x7fe1e80cb490 2024-06-29T14:35:03.859 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:03.859 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":67,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:35:03.093892+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:35:03.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe1c80763f0 msgr2=0x7fe1c8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe1c80763f0 0x7fe1c8078900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8159e60 tx=0x7fe1d0009040 comp rx=0 tx=0).stop 2024-06-29T14:35:03.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80cb490 msgr2=0x7fe1e806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80cb490 0x7fe1e806ec90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000d740 tx=0x7fe1e000dc00 comp rx=0 tx=0).stop 2024-06-29T14:35:03.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 shutdown_connections 2024-06-29T14:35:03.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe1c80763f0 0x7fe1c8078900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1e8151130 0x7fe1e80b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1e80cc060 0x7fe1e80b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 --2- 172.21.15.59:0/2688107089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1e80cb490 0x7fe1e806ec90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 >> 172.21.15.59:0/2688107089 conn(0x7fe1e80b1ea0 msgr2=0x7fe1e80b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 shutdown_connections 2024-06-29T14:35:03.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.863+0000 7fe1ecf5b640 1 -- 172.21.15.59:0/2688107089 wait complete. 2024-06-29T14:35:03.872 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:35:03.872 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:35:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.996+0000 7f0bcad3c640 1 -- 172.21.15.59:0/2233257248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bc4162190 msgr2=0x7f0bc40ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.996+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/2233257248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bc4162190 0x7f0bc40ce550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009830 tx=0x7f0bb802a480 comp rx=0 tx=0).stop 2024-06-29T14:35:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.996+0000 7f0bcad3c640 1 -- 172.21.15.59:0/2233257248 shutdown_connections 2024-06-29T14:35:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.996+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/2233257248 >> v2:172.21.15.153:3300/0 conn(0x7f0bc415e0a0 0x7f0bc40b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.996+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/2233257248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bc4162190 0x7f0bc40ce550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.996+0000 7f0bcad3c640 1 -- 172.21.15.59:0/2233257248 >> 172.21.15.59:0/2233257248 conn(0x7f0bc40d45b0 msgr2=0x7f0bc40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.996+0000 7f0bcad3c640 1 -- 172.21.15.59:0/2233257248 shutdown_connections 2024-06-29T14:35:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.996+0000 7f0bcad3c640 1 -- 172.21.15.59:0/2233257248 wait complete. 2024-06-29T14:35:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.998+0000 7f0bcad3c640 1 Processor -- start 2024-06-29T14:35:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.998+0000 7f0bcad3c640 1 -- start start 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.999+0000 7f0bcad3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bc415e0a0 0x7f0bc4155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.999+0000 7f0bcad3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bc4162770 0x7f0bc41f7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.999+0000 7f0bcad3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bc41561f0 0x7f0bc41f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.999+0000 7f0bcad3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc40b43b0 con 0x7f0bc415e0a0 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.999+0000 7f0bcad3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc40b4230 con 0x7f0bc4162770 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:03.999+0000 7f0bcad3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc40b4530 con 0x7f0bc41561f0 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bc415e0a0 0x7f0bc4155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bc415e0a0 0x7f0bc4155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37712/0 (socket says 172.21.15.59:37712) 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 -- 172.21.15.59:0/183295022 learned_addr learned my addr 172.21.15.59:0/183295022 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc3fff640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bc4162770 0x7f0bc41f7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc92d9640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bc41561f0 0x7f0bc41f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 -- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bc41561f0 msgr2=0x7f0bc41f82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bc41561f0 0x7f0bc41f82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 -- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bc4162770 msgr2=0x7f0bc41f7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bc4162770 0x7f0bc41f7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 -- 172.21.15.59:0/183295022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc4157240 con 0x7f0bc415e0a0 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc8ad8640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bc415e0a0 0x7f0bc4155cb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400c3a0 tx=0x7f0bb400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc17fa640 1 -- 172.21.15.59:0/183295022 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb40052d0 con 0x7f0bc415e0a0 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc17fa640 1 -- 172.21.15.59:0/183295022 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0bb4007020 con 0x7f0bc415e0a0 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bc17fa640 1 -- 172.21.15.59:0/183295022 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4007470 con 0x7f0bc415e0a0 2024-06-29T14:35:03.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.000+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bc4157520 con 0x7f0bc415e0a0 2024-06-29T14:35:04.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.001+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bc4157ae0 con 0x7f0bc415e0a0 2024-06-29T14:35:04.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.002+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bc40b49e0 con 0x7f0bc415e0a0 2024-06-29T14:35:04.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.002+0000 7f0bc17fa640 1 -- 172.21.15.59:0/183295022 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f0bb40067d0 con 0x7f0bc415e0a0 2024-06-29T14:35:04.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.003+0000 7f0bc17fa640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0bb0076420 0x7f0bb0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.003+0000 7f0bc17fa640 1 -- 172.21.15.59:0/183295022 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5249+0+0 (secure 0 0 0) 0x7f0bb4096bf0 con 0x7f0bc415e0a0 2024-06-29T14:35:04.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.003+0000 7f0bc3fff640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0bb0076420 0x7f0bb0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.004+0000 7f0bc3fff640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0bb0076420 0x7f0bb0078930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8034000 tx=0x7f0bb8004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.007+0000 7f0bc17fa640 1 -- 172.21.15.59:0/183295022 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0bb405ff10 con 0x7f0bc415e0a0 2024-06-29T14:35:04.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.198+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0bc406d2e0 con 0x7f0bc415e0a0 2024-06-29T14:35:04.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.198+0000 7f0bc17fa640 1 -- 172.21.15.59:0/183295022 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7f0bb4064210 con 0x7f0bc415e0a0 2024-06-29T14:35:04.195 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0bb0076420 msgr2=0x7f0bb0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0bb0076420 0x7f0bb0078930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8034000 tx=0x7f0bb8004730 comp rx=0 tx=0).stop 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bc415e0a0 msgr2=0x7f0bc4155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bc415e0a0 0x7f0bc4155cb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400c3a0 tx=0x7f0bb400c860 comp rx=0 tx=0).stop 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 shutdown_connections 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0bb0076420 0x7f0bb0078930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0bc41561f0 0x7f0bc41f82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bc4162770 0x7f0bc41f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 --2- 172.21.15.59:0/183295022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0bc415e0a0 0x7f0bc4155cb0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 >> 172.21.15.59:0/183295022 conn(0x7f0bc40d45b0 msgr2=0x7f0bc406dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 shutdown_connections 2024-06-29T14:35:04.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.200+0000 7f0bcad3c640 1 -- 172.21.15.59:0/183295022 wait complete. 2024-06-29T14:35:04.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:35:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.331+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2233418437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 msgr2=0x7ff69c0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.331+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2233418437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 0x7ff69c0b3990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff68c005fd0 tx=0x7ff68c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.332+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2233418437 shutdown_connections 2024-06-29T14:35:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.332+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2233418437 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff69c15efe0 0x7ff69c15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.332+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2233418437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 0x7ff69c0b3990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.332+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2233418437 >> 172.21.15.59:0/2233418437 conn(0x7ff69c0d3ee0 msgr2=0x7ff69c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.332+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2233418437 shutdown_connections 2024-06-29T14:35:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.332+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2233418437 wait complete. 2024-06-29T14:35:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.334+0000 7ff6a0ac5640 1 Processor -- start 2024-06-29T14:35:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.334+0000 7ff6a0ac5640 1 -- start start 2024-06-29T14:35:04.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff6a0ac5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff69c0b3ed0 0x7ff69c0ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff6a0ac5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 0x7ff69c1e7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff6a0ac5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff69c15efe0 0x7ff69c1e8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff6a0ac5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff69c160280 con 0x7ff69c15efe0 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff6a0ac5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff69c160100 con 0x7ff69c0d23c0 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff6a0ac5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff69c160400 con 0x7ff69c0b3ed0 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff69ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff69c15efe0 0x7ff69c1e8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff69ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff69c15efe0 0x7ff69c1e8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37722/0 (socket says 172.21.15.59:37722) 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff69ad9d640 1 -- 172.21.15.59:0/2998907808 learned_addr learned my addr 172.21.15.59:0/2998907808 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff69a59c640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff69c0b3ed0 0x7ff69c0ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff699d9b640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 0x7ff69c1e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff69ad9d640 1 -- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff69c0b3ed0 msgr2=0x7ff69c0ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff69ad9d640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff69c0b3ed0 0x7ff69c0ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff69ad9d640 1 -- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 msgr2=0x7ff69c1e7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.335+0000 7ff69ad9d640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 0x7ff69c1e7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.336+0000 7ff69ad9d640 1 -- 172.21.15.59:0/2998907808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69c1477c0 con 0x7ff69c15efe0 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.336+0000 7ff699d9b640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 0x7ff69c1e7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.336+0000 7ff69ad9d640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff69c15efe0 0x7ff69c1e8fd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff69400e970 tx=0x7ff69400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.336+0000 7ff68affd640 1 -- 172.21.15.59:0/2998907808 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69400cd80 con 0x7ff69c15efe0 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.336+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69c1e7f30 con 0x7ff69c15efe0 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.336+0000 7ff68affd640 1 -- 172.21.15.59:0/2998907808 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff694004020 con 0x7ff69c15efe0 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.336+0000 7ff68affd640 1 -- 172.21.15.59:0/2998907808 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694004470 con 0x7ff69c15efe0 2024-06-29T14:35:04.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.336+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff69c1e84f0 con 0x7ff69c15efe0 2024-06-29T14:35:04.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.337+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff69c0b3990 con 0x7ff69c15efe0 2024-06-29T14:35:04.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.338+0000 7ff68affd640 1 -- 172.21.15.59:0/2998907808 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7ff6940035f0 con 0x7ff69c15efe0 2024-06-29T14:35:04.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.338+0000 7ff68affd640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff67c0763f0 0x7ff67c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.338+0000 7ff68affd640 1 -- 172.21.15.59:0/2998907808 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5249+0+0 (secure 0 0 0) 0x7ff6940983b0 con 0x7ff69c15efe0 2024-06-29T14:35:04.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.339+0000 7ff69a59c640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff67c0763f0 0x7ff67c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.339+0000 7ff69a59c640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff67c0763f0 0x7ff67c078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff68c034000 tx=0x7ff68c004ab0 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.343+0000 7ff68affd640 1 -- 172.21.15.59:0/2998907808 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff694061b00 con 0x7ff69c15efe0 2024-06-29T14:35:04.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.539+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff69c0d2950 con 0x7ff69c15efe0 2024-06-29T14:35:04.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.539+0000 7ff68affd640 1 -- 172.21.15.59:0/2998907808 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v67) ==== 84+0+65 (secure 0 0 0) 0x7ff694096e20 con 0x7ff69c15efe0 2024-06-29T14:35:04.537 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:04.537 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:35:04.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:04.537 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:35:04.537 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:35:04.538 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:04.538 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:35:04.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff67c0763f0 msgr2=0x7ff67c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff67c0763f0 0x7ff67c078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff68c034000 tx=0x7ff68c004ab0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff69c15efe0 msgr2=0x7ff69c1e8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff69c15efe0 0x7ff69c1e8fd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff69400e970 tx=0x7ff69400ee30 comp rx=0 tx=0).stop 2024-06-29T14:35:04.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 shutdown_connections 2024-06-29T14:35:04.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff67c0763f0 0x7ff67c078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff69c15efe0 0x7ff69c1e8fd0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff69c0d23c0 0x7ff69c1e7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 --2- 172.21.15.59:0/2998907808 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff69c0b3ed0 0x7ff69c0ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 >> 172.21.15.59:0/2998907808 conn(0x7ff69c0d3ee0 msgr2=0x7ff69c0d2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 shutdown_connections 2024-06-29T14:35:04.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.542+0000 7ff6a0ac5640 1 -- 172.21.15.59:0/2998907808 wait complete. 2024-06-29T14:35:04.551 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:35:04.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:35:04.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.675+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/4132319558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 msgr2=0x7f4cd80b7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.675+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/4132319558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 0x7f4cd80b7560 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8050ac0 tx=0x7f4cc002a560 comp rx=0 tx=0).stop 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.675+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/4132319558 shutdown_connections 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.675+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/4132319558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 0x7f4cd80b7560 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.675+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/4132319558 >> v2:172.21.15.153:3300/0 conn(0x7f4cd80b2a50 0x7f4cd8161380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.675+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/4132319558 >> 172.21.15.59:0/4132319558 conn(0x7f4cd80b5590 msgr2=0x7f4cd80b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.675+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/4132319558 shutdown_connections 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.675+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/4132319558 wait complete. 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.677+0000 7f4cdcf9a640 1 Processor -- start 2024-06-29T14:35:04.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.677+0000 7f4cdcf9a640 1 -- start start 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.678+0000 7f4cdcf9a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cd80b2a50 0x7f4cd80cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.678+0000 7f4cdcf9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 0x7f4cd80cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.678+0000 7f4cdcf9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cd80d2f70 0x7f4cd80cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.678+0000 7f4cdcf9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd80b4510 con 0x7f4cd80b7150 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.678+0000 7f4cdcf9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd80b53e0 con 0x7f4cd80d2f70 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.678+0000 7f4cdcf9a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd80b4690 con 0x7f4cd80b2a50 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 0x7f4cd80cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 0x7f4cd80cab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37750/0 (socket says 172.21.15.59:37750) 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 -- 172.21.15.59:0/2079115142 learned_addr learned my addr 172.21.15.59:0/2079115142 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd659c640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cd80b2a50 0x7f4cd80cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd6d9d640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cd80d2f70 0x7f4cd80cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 -- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cd80b2a50 msgr2=0x7f4cd80cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cd80b2a50 0x7f4cd80cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 -- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cd80d2f70 msgr2=0x7f4cd80cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cd80d2f70 0x7f4cd80cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 -- 172.21.15.59:0/2079115142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd80aa590 con 0x7f4cd80b7150 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd6d9d640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cd80d2f70 0x7f4cd80cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cd5d9b640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 0x7f4cd80cab50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0000c00 tx=0x7f4cc0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cceffd640 1 -- 172.21.15.59:0/2079115142 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc003e030 con 0x7f4cd80b7150 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cceffd640 1 -- 172.21.15.59:0/2079115142 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cc0003e00 con 0x7f4cd80b7150 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd80aa830 con 0x7f4cd80b7150 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cceffd640 1 -- 172.21.15.59:0/2079115142 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0035040 con 0x7f4cd80b7150 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.679+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd80aadf0 con 0x7f4cd80b7150 2024-06-29T14:35:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.680+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd80b4b80 con 0x7f4cd80b7150 2024-06-29T14:35:04.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.681+0000 7f4cceffd640 1 -- 172.21.15.59:0/2079115142 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f4cc0041030 con 0x7f4cd80b7150 2024-06-29T14:35:04.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.682+0000 7f4cceffd640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f4cb80763f0 0x7f4cb8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.682+0000 7f4cceffd640 1 -- 172.21.15.59:0/2079115142 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5249+0+0 (secure 0 0 0) 0x7f4cc00b7f10 con 0x7f4cd80b7150 2024-06-29T14:35:04.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.682+0000 7f4cd659c640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f4cb80763f0 0x7f4cb8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.683+0000 7f4cd659c640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f4cb80763f0 0x7f4cb8078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8005fd0 tx=0x7f4cc80095e0 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.686+0000 7f4cceffd640 1 -- 172.21.15.59:0/2079115142 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4cd80aa590 con 0x7f4cd80b7150 2024-06-29T14:35:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:04.877+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4cd808ccd0 con 0x7f4cd80b7150 2024-06-29T14:35:05.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.119+0000 7f4cceffd640 1 -- 172.21.15.59:0/2079115142 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v68) ==== 138+0+0 (secure 0 0 0) 0x7f4cc00811c0 con 0x7f4cd80b7150 2024-06-29T14:35:05.117 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:35:05.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f4cb80763f0 msgr2=0x7f4cb8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f4cb80763f0 0x7f4cb8078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8005fd0 tx=0x7f4cc80095e0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 msgr2=0x7f4cd80cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 0x7f4cd80cab50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0000c00 tx=0x7f4cc0003910 comp rx=0 tx=0).stop 2024-06-29T14:35:05.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 shutdown_connections 2024-06-29T14:35:05.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f4cb80763f0 0x7f4cb8078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cd80d2f70 0x7f4cd80cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cd80b7150 0x7f4cd80cab50 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 --2- 172.21.15.59:0/2079115142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cd80b2a50 0x7f4cd80cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 >> 172.21.15.59:0/2079115142 conn(0x7f4cd80b5590 msgr2=0x7f4cd80b80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:05.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 shutdown_connections 2024-06-29T14:35:05.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.122+0000 7f4cdcf9a640 1 -- 172.21.15.59:0/2079115142 wait complete. 2024-06-29T14:35:05.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:35:05.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.262+0000 7f2ac60ce640 1 -- 172.21.15.59:0/4003380659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ac00b4c80 msgr2=0x7f2ac0168d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.262+0000 7f2ac60ce640 1 --2- 172.21.15.59:0/4003380659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ac00b4c80 0x7f2ac0168d70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00cf2d0 tx=0x7f2ab802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.262+0000 7f2ac60ce640 1 -- 172.21.15.59:0/4003380659 shutdown_connections 2024-06-29T14:35:05.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.262+0000 7f2ac60ce640 1 --2- 172.21.15.59:0/4003380659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ac00b4c80 0x7f2ac0168d70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.262+0000 7f2ac60ce640 1 -- 172.21.15.59:0/4003380659 >> 172.21.15.59:0/4003380659 conn(0x7f2ac00d04e0 msgr2=0x7f2ac00d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:05.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.262+0000 7f2ac60ce640 1 -- 172.21.15.59:0/4003380659 shutdown_connections 2024-06-29T14:35:05.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.262+0000 7f2ac60ce640 1 -- 172.21.15.59:0/4003380659 wait complete. 2024-06-29T14:35:05.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.263+0000 7f2ac60ce640 1 Processor -- start 2024-06-29T14:35:05.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.264+0000 7f2ac60ce640 1 -- start start 2024-06-29T14:35:05.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2ac60ce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ac00b2c40 0x7f2ac015c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:05.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2ac60ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ac00d3260 0x7f2ac015a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:05.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2ac60ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac015acd0 0x7f2ac00c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:05.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2ac60ce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac01692b0 con 0x7f2ac015acd0 2024-06-29T14:35:05.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2ac60ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac00b8280 con 0x7f2ac00d3260 2024-06-29T14:35:05.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2ac60ce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac0169430 con 0x7f2ac00b2c40 2024-06-29T14:35:05.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac015acd0 0x7f2ac00c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:05.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac015acd0 0x7f2ac00c8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37756/0 (socket says 172.21.15.59:37756) 2024-06-29T14:35:05.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abffff640 1 -- 172.21.15.59:0/3020302085 learned_addr learned my addr 172.21.15.59:0/3020302085 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:05.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abf7fe640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ac00b2c40 0x7f2ac015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:05.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abeffd640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ac00d3260 0x7f2ac015a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:05.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abffff640 1 -- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ac00b2c40 msgr2=0x7f2ac015c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abffff640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ac00b2c40 0x7f2ac015c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abffff640 1 -- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ac00d3260 msgr2=0x7f2ac015a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abffff640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ac00d3260 0x7f2ac015a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.265+0000 7f2abffff640 1 -- 172.21.15.59:0/3020302085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac00cab10 con 0x7f2ac015acd0 2024-06-29T14:35:05.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.266+0000 7f2abffff640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac015acd0 0x7f2ac00c8610 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009ca0 tx=0x7f2ab8003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:05.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.266+0000 7f2aa7fff640 1 -- 172.21.15.59:0/3020302085 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8042020 con 0x7f2ac015acd0 2024-06-29T14:35:05.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.266+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac0154280 con 0x7f2ac015acd0 2024-06-29T14:35:05.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.266+0000 7f2aa7fff640 1 -- 172.21.15.59:0/3020302085 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ab8033c00 con 0x7f2ac015acd0 2024-06-29T14:35:05.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.266+0000 7f2aa7fff640 1 -- 172.21.15.59:0/3020302085 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8037040 con 0x7f2ac015acd0 2024-06-29T14:35:05.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.266+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac015b610 con 0x7f2ac015acd0 2024-06-29T14:35:05.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.267+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac015e0a0 con 0x7f2ac015acd0 2024-06-29T14:35:05.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.268+0000 7f2aa7fff640 1 -- 172.21.15.59:0/3020302085 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f2ab8045020 con 0x7f2ac015acd0 2024-06-29T14:35:05.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.268+0000 7f2aa7fff640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2aac076420 0x7f2aac078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:05.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.268+0000 7f2aa7fff640 1 -- 172.21.15.59:0/3020302085 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5619+0+0 (secure 0 0 0) 0x7f2ab80b86c0 con 0x7f2ac015acd0 2024-06-29T14:35:05.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.269+0000 7f2abf7fe640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2aac076420 0x7f2aac078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:05.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.269+0000 7f2abf7fe640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2aac076420 0x7f2aac078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0000f60 tx=0x7f2ab0009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:05.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.273+0000 7f2aa7fff640 1 -- 172.21.15.59:0/3020302085 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ab804c030 con 0x7f2ac015acd0 2024-06-29T14:35:05.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:05.463+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f2ac00c8230 con 0x7f2ac015acd0 2024-06-29T14:35:06.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.128+0000 7f2aa7fff640 1 -- 172.21.15.59:0/3020302085 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v69) ==== 170+0+0 (secure 0 0 0) 0x7f2ab8081d30 con 0x7f2ac015acd0 2024-06-29T14:35:06.125 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:35:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2aac076420 msgr2=0x7f2aac078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2aac076420 0x7f2aac078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0000f60 tx=0x7f2ab0009040 comp rx=0 tx=0).stop 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac015acd0 msgr2=0x7f2ac00c8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac015acd0 0x7f2ac00c8610 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009ca0 tx=0x7f2ab8003f50 comp rx=0 tx=0).stop 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 shutdown_connections 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f2aac076420 0x7f2aac078930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac015acd0 0x7f2ac00c8610 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ac00d3260 0x7f2ac015a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 --2- 172.21.15.59:0/3020302085 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ac00b2c40 0x7f2ac015c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 >> 172.21.15.59:0/3020302085 conn(0x7f2ac00d04e0 msgr2=0x7f2ac00b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 shutdown_connections 2024-06-29T14:35:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.131+0000 7f2ac60ce640 1 -- 172.21.15.59:0/3020302085 wait complete. 2024-06-29T14:35:06.155 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:35:06.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.285+0000 7f97cba8f640 1 -- 172.21.15.59:0/562798680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 msgr2=0x7f97c40b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.285+0000 7f97cba8f640 1 --2- 172.21.15.59:0/562798680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 0x7f97c40b1d10 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005fd0 tx=0x7f97c002a670 comp rx=0 tx=0).stop 2024-06-29T14:35:06.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.285+0000 7f97cba8f640 1 -- 172.21.15.59:0/562798680 shutdown_connections 2024-06-29T14:35:06.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.285+0000 7f97cba8f640 1 --2- 172.21.15.59:0/562798680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 0x7f97c40b1d10 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.285+0000 7f97cba8f640 1 -- 172.21.15.59:0/562798680 >> 172.21.15.59:0/562798680 conn(0x7f97c406c950 msgr2=0x7f97c4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:06.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.286+0000 7f97cba8f640 1 -- 172.21.15.59:0/562798680 shutdown_connections 2024-06-29T14:35:06.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.286+0000 7f97cba8f640 1 -- 172.21.15.59:0/562798680 wait complete. 2024-06-29T14:35:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.287+0000 7f97cba8f640 1 Processor -- start 2024-06-29T14:35:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.287+0000 7f97cba8f640 1 -- start start 2024-06-29T14:35:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97cba8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97c406f8a0 0x7f97c40cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97cba8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 0x7f97c41430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97cba8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97c4161b00 0x7f97c40ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97cba8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4169340 con 0x7f97c4147370 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97cba8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41691c0 con 0x7f97c406f8a0 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97cba8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f97c41694c0 con 0x7f97c4161b00 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 0x7f97c41430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 0x7f97c41430d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37776/0 (socket says 172.21.15.59:37776) 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 -- 172.21.15.59:0/2430718500 learned_addr learned my addr 172.21.15.59:0/2430718500 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c982b640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97c406f8a0 0x7f97c40cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97ca02c640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97c4161b00 0x7f97c40ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 -- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97c4161b00 msgr2=0x7f97c40ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97c4161b00 0x7f97c40ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 -- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97c406f8a0 msgr2=0x7f97c40cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97c406f8a0 0x7f97c40cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 -- 172.21.15.59:0/2430718500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c40ce420 con 0x7f97c4147370 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97c902a640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 0x7f97c41430d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c370 tx=0x7f97b400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.289+0000 7f97ba7fc640 1 -- 172.21.15.59:0/2430718500 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4017070 con 0x7f97c4147370 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.290+0000 7f97ba7fc640 1 -- 172.21.15.59:0/2430718500 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97b40035f0 con 0x7f97c4147370 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.290+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c415d850 con 0x7f97c4147370 2024-06-29T14:35:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.290+0000 7f97ba7fc640 1 -- 172.21.15.59:0/2430718500 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4003a40 con 0x7f97c4147370 2024-06-29T14:35:06.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.290+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c415de10 con 0x7f97c4147370 2024-06-29T14:35:06.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.290+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c40cada0 con 0x7f97c4147370 2024-06-29T14:35:06.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.292+0000 7f97ba7fc640 1 -- 172.21.15.59:0/2430718500 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f97b401a070 con 0x7f97c4147370 2024-06-29T14:35:06.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.292+0000 7f97ba7fc640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f97a80763f0 0x7f97a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.292+0000 7f97ba7fc640 1 -- 172.21.15.59:0/2430718500 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6001+0+0 (secure 0 0 0) 0x7f97b4097020 con 0x7f97c4147370 2024-06-29T14:35:06.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.292+0000 7f97c982b640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f97a80763f0 0x7f97a8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.293+0000 7f97c982b640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f97a80763f0 0x7f97a8078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f97c0033000 tx=0x7f97c0004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:06.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.296+0000 7f97ba7fc640 1 -- 172.21.15.59:0/2430718500 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f97b4021020 con 0x7f97c4147370 2024-06-29T14:35:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:06.529+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f97c40ce1b0 con 0x7f97c4147370 2024-06-29T14:35:07.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.129+0000 7f97ba7fc640 1 -- 172.21.15.59:0/2430718500 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v25) ==== 172+0+0 (secure 0 0 0) 0x7f97b40601d0 con 0x7f97c4147370 2024-06-29T14:35:07.127 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 7 and data pool 8 2024-06-29T14:35:07.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.134+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f97a80763f0 msgr2=0x7f97a8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.134+0000 7f97cba8f640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f97a80763f0 0x7f97a8078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f97c0033000 tx=0x7f97c0004730 comp rx=0 tx=0).stop 2024-06-29T14:35:07.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.134+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 msgr2=0x7f97c41430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.134+0000 7f97cba8f640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 0x7f97c41430d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c370 tx=0x7f97b400c830 comp rx=0 tx=0).stop 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.135+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 shutdown_connections 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.135+0000 7f97cba8f640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f97a80763f0 0x7f97a8078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.135+0000 7f97cba8f640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97c4161b00 0x7f97c40ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.135+0000 7f97cba8f640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c4147370 0x7f97c41430d0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.135+0000 7f97cba8f640 1 --2- 172.21.15.59:0/2430718500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97c406f8a0 0x7f97c40cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.135+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 >> 172.21.15.59:0/2430718500 conn(0x7f97c406c950 msgr2=0x7f97c40d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.135+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 shutdown_connections 2024-06-29T14:35:07.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.135+0000 7f97cba8f640 1 -- 172.21.15.59:0/2430718500 wait complete. 2024-06-29T14:35:07.163 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:35:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.292+0000 7f058e226640 1 -- 172.21.15.59:0/1008147954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0570004d40 msgr2=0x7f05700051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.292+0000 7f058e226640 1 --2- 172.21.15.59:0/1008147954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0570004d40 0x7f05700051f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f058000c740 tx=0x7f0570005830 comp rx=0 tx=0).stop 2024-06-29T14:35:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.292+0000 7f0587fff640 1 -- 172.21.15.59:0/1008147954 reap_dead start 2024-06-29T14:35:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.292+0000 7f058e226640 1 -- 172.21.15.59:0/1008147954 shutdown_connections 2024-06-29T14:35:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.292+0000 7f058e226640 1 -- 172.21.15.59:0/1008147954 >> 172.21.15.59:0/1008147954 conn(0x7f05880d3ee0 msgr2=0x7f05880d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.292+0000 7f058e226640 1 -- 172.21.15.59:0/1008147954 shutdown_connections 2024-06-29T14:35:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.292+0000 7f058e226640 1 -- 172.21.15.59:0/1008147954 wait complete. 2024-06-29T14:35:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.294+0000 7f058e226640 1 Processor -- start 2024-06-29T14:35:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.294+0000 7f058e226640 1 -- start start 2024-06-29T14:35:07.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.295+0000 7f058e226640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0570004d40 0x7f058815dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.295+0000 7f058e226640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0588151030 0x7f05881575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.295+0000 7f058e226640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0588157af0 0x7f0588152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.295+0000 7f058e226640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f05880b3d20 con 0x7f0588151030 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.295+0000 7f058e226640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05880b3ba0 con 0x7f0570004d40 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.295+0000 7f058e226640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f05881600c0 con 0x7f0588157af0 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0588151030 0x7f05881575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0588151030 0x7f05881575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37788/0 (socket says 172.21.15.59:37788) 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 -- 172.21.15.59:0/2343383323 learned_addr learned my addr 172.21.15.59:0/2343383323 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f05877fe640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0570004d40 0x7f058815dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0587fff640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0588157af0 0x7f0588152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 -- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0588157af0 msgr2=0x7f0588152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0588157af0 0x7f0588152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 -- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0570004d40 msgr2=0x7f058815dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0570004d40 0x7f058815dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 -- 172.21.15.59:0/2343383323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0588157f00 con 0x7f0588151030 2024-06-29T14:35:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f0586ffd640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0588151030 0x7f05881575b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f057c003f20 tx=0x7f057c001730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f056ffff640 1 -- 172.21.15.59:0/2343383323 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c015070 con 0x7f0588151030 2024-06-29T14:35:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f056ffff640 1 -- 172.21.15.59:0/2343383323 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f057c011860 con 0x7f0588151030 2024-06-29T14:35:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058815af90 con 0x7f0588151030 2024-06-29T14:35:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f056ffff640 1 -- 172.21.15.59:0/2343383323 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c011cb0 con 0x7f0588151030 2024-06-29T14:35:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.296+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058815b550 con 0x7f0588151030 2024-06-29T14:35:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.298+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05881617a0 con 0x7f0588151030 2024-06-29T14:35:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.298+0000 7f056ffff640 1 -- 172.21.15.59:0/2343383323 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f057c018070 con 0x7f0588151030 2024-06-29T14:35:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.299+0000 7f056ffff640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f05740763f0 0x7f0574078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.299+0000 7f056ffff640 1 -- 172.21.15.59:0/2343383323 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6101+0+0 (secure 0 0 0) 0x7f057c098700 con 0x7f0588151030 2024-06-29T14:35:07.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.299+0000 7f05877fe640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f05740763f0 0x7f0574078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.300+0000 7f05877fe640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f05740763f0 0x7f0574078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05880d2e40 tx=0x7f0578005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.303+0000 7f056ffff640 1 -- 172.21.15.59:0/2343383323 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0588157f00 con 0x7f0588151030 2024-06-29T14:35:07.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.496+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f058815ad20 con 0x7f0588151030 2024-06-29T14:35:07.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.497+0000 7f056ffff640 1 -- 172.21.15.59:0/2343383323 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16967 (secure 0 0 0) 0x7f057c0615d0 con 0x7f0588151030 2024-06-29T14:35:07.495 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:07.495 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":70,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:35:06.530024+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:35:04.881993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-06-29T14:35:05.467776+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f05740763f0 msgr2=0x7f0574078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f05740763f0 0x7f0574078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05880d2e40 tx=0x7f0578005f90 comp rx=0 tx=0).stop 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0588151030 msgr2=0x7f05881575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0588151030 0x7f05881575b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f057c003f20 tx=0x7f057c001730 comp rx=0 tx=0).stop 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 shutdown_connections 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f05740763f0 0x7f0574078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0588157af0 0x7f0588152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0588151030 0x7f05881575b0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 --2- 172.21.15.59:0/2343383323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0570004d40 0x7f058815dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.499+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 >> 172.21.15.59:0/2343383323 conn(0x7f05880d3ee0 msgr2=0x7f058806c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.500+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 shutdown_connections 2024-06-29T14:35:07.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.500+0000 7f058e226640 1 -- 172.21.15.59:0/2343383323 wait complete. 2024-06-29T14:35:07.508 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:35:07.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-06-29T14:35:07.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.629+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/1884715625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac13e9b0 msgr2=0x7fc0ac13edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.629+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/1884715625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac13e9b0 0x7fc0ac13edc0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac073da0 tx=0x7fc0a002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.629+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/1884715625 shutdown_connections 2024-06-29T14:35:07.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.629+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/1884715625 >> v2:172.21.15.153:3300/0 conn(0x7fc0ac1414c0 0x7fc0ac1418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.629+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/1884715625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac13e9b0 0x7fc0ac13edc0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.629+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/1884715625 >> 172.21.15.59:0/1884715625 conn(0x7fc0ac0b6fb0 msgr2=0x7fc0ac0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.629+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/1884715625 shutdown_connections 2024-06-29T14:35:07.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.629+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/1884715625 wait complete. 2024-06-29T14:35:07.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.631+0000 7fc0b1d7a640 1 Processor -- start 2024-06-29T14:35:07.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.631+0000 7fc0b1d7a640 1 -- start start 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0b1d7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac13e9b0 0x7fc0ac0c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0b1d7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac1414c0 0x7fc0ac1520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0b1d7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0ac15ceb0 0x7fc0ac1376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0b1d7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac0b29d0 con 0x7fc0ac1414c0 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0b1d7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac0b2850 con 0x7fc0ac13e9b0 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0b1d7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ac0b2b50 con 0x7fc0ac15ceb0 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0aaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac1414c0 0x7fc0ac1520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0aaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac1414c0 0x7fc0ac1520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37812/0 (socket says 172.21.15.59:37812) 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0aaffd640 1 -- 172.21.15.59:0/2663949566 learned_addr learned my addr 172.21.15.59:0/2663949566 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0ab7fe640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac13e9b0 0x7fc0ac0c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.632+0000 7fc0abfff640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0ac15ceb0 0x7fc0ac1376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc0aaffd640 1 -- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0ac15ceb0 msgr2=0x7fc0ac1376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc0aaffd640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0ac15ceb0 0x7fc0ac1376b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc0aaffd640 1 -- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac13e9b0 msgr2=0x7fc0ac0c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc0aaffd640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac13e9b0 0x7fc0ac0c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc0aaffd640 1 -- 172.21.15.59:0/2663949566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ac0c73d0 con 0x7fc0ac1414c0 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc0aaffd640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac1414c0 0x7fc0ac1520f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0034040 tx=0x7fc0a0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc093fff640 1 -- 172.21.15.59:0/2663949566 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a003e030 con 0x7fc0ac1414c0 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc093fff640 1 -- 172.21.15.59:0/2663949566 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0a0003e00 con 0x7fc0ac1414c0 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ac0c76b0 con 0x7fc0ac1414c0 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc093fff640 1 -- 172.21.15.59:0/2663949566 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0035040 con 0x7fc0ac1414c0 2024-06-29T14:35:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.633+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ac1f2ea0 con 0x7fc0ac1414c0 2024-06-29T14:35:07.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.634+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0ac0b3000 con 0x7fc0ac1414c0 2024-06-29T14:35:07.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.635+0000 7fc093fff640 1 -- 172.21.15.59:0/2663949566 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fc0a0041030 con 0x7fc0ac1414c0 2024-06-29T14:35:07.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.635+0000 7fc093fff640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc0980763f0 0x7fc098078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.635+0000 7fc093fff640 1 -- 172.21.15.59:0/2663949566 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6101+0+0 (secure 0 0 0) 0x7fc0a00b81d0 con 0x7fc0ac1414c0 2024-06-29T14:35:07.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.636+0000 7fc0ab7fe640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc0980763f0 0x7fc098078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.636+0000 7fc0ab7fe640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc0980763f0 0x7fc098078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc09c006fd0 tx=0x7fc09c009060 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:07.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.640+0000 7fc093fff640 1 -- 172.21.15.59:0/2663949566 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0ac141e10 con 0x7fc0ac1414c0 2024-06-29T14:35:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.831+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0ac13e4e0 con 0x7fc0ac1414c0 2024-06-29T14:35:07.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.832+0000 7fc093fff640 1 -- 172.21.15.59:0/2663949566 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16967 (secure 0 0 0) 0x7fc0a00811c0 con 0x7fc0ac1414c0 2024-06-29T14:35:07.830 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:07.830 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":70,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:35:06.530024+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:35:04.881993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-06-29T14:35:05.467776+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:35:07.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.834+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc0980763f0 msgr2=0x7fc098078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.834+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc0980763f0 0x7fc098078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc09c006fd0 tx=0x7fc09c009060 comp rx=0 tx=0).stop 2024-06-29T14:35:07.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.834+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac1414c0 msgr2=0x7fc0ac1520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.834+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac1414c0 0x7fc0ac1520f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0034040 tx=0x7fc0a0003910 comp rx=0 tx=0).stop 2024-06-29T14:35:07.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.835+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 shutdown_connections 2024-06-29T14:35:07.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.835+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc0980763f0 0x7fc098078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.835+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0ac15ceb0 0x7fc0ac1376b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.835+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0ac1414c0 0x7fc0ac1520f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.835+0000 7fc0b1d7a640 1 --2- 172.21.15.59:0/2663949566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ac13e9b0 0x7fc0ac0c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.835+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 >> 172.21.15.59:0/2663949566 conn(0x7fc0ac0b6fb0 msgr2=0x7fc0ac0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.835+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 shutdown_connections 2024-06-29T14:35:07.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.835+0000 7fc0b1d7a640 1 -- 172.21.15.59:0/2663949566 wait complete. 2024-06-29T14:35:07.843 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:35:07.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.967+0000 7fd21443d640 1 -- 172.21.15.59:0/1084045453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f4004800 msgr2=0x7fd1f4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.967+0000 7fd21443d640 1 --2- 172.21.15.59:0/1084045453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1f4004800 0x7fd1f4004c10 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd20c13f300 tx=0x7fd1f4005830 comp rx=0 tx=0).stop 2024-06-29T14:35:07.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.967+0000 7fd2129da640 1 -- 172.21.15.59:0/1084045453 reap_dead start 2024-06-29T14:35:07.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.967+0000 7fd21443d640 1 -- 172.21.15.59:0/1084045453 shutdown_connections 2024-06-29T14:35:07.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.968+0000 7fd21443d640 1 -- 172.21.15.59:0/1084045453 >> 172.21.15.59:0/1084045453 conn(0x7fd20c0b6fb0 msgr2=0x7fd20c0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.968+0000 7fd21443d640 1 -- 172.21.15.59:0/1084045453 shutdown_connections 2024-06-29T14:35:07.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.968+0000 7fd21443d640 1 -- 172.21.15.59:0/1084045453 wait complete. 2024-06-29T14:35:07.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.969+0000 7fd21443d640 1 Processor -- start 2024-06-29T14:35:07.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.969+0000 7fd21443d640 1 -- start start 2024-06-29T14:35:07.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd21443d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f4004800 0x7fd20c1560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd21443d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd20c1f4500 0x7fd20c06f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd21443d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd20c1f04b0 0x7fd20c1f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd21443d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c0b26b0 con 0x7fd20c1f4500 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd21443d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c0b2530 con 0x7fd20c1f04b0 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd21443d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd20c0b2830 con 0x7fd1f4004800 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd20c1f4500 0x7fd20c06f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd20c1f4500 0x7fd20c06f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37822/0 (socket says 172.21.15.59:37822) 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 -- 172.21.15.59:0/498371968 learned_addr learned my addr 172.21.15.59:0/498371968 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2129da640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd20c1f04b0 0x7fd20c1f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2121d9640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f4004800 0x7fd20c1560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 -- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f4004800 msgr2=0x7fd20c1560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f4004800 0x7fd20c1560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 -- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd20c1f04b0 msgr2=0x7fd20c1f08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd20c1f04b0 0x7fd20c1f08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 -- 172.21.15.59:0/498371968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c1386f0 con 0x7fd20c1f4500 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2129da640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd20c1f04b0 0x7fd20c1f08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.971+0000 7fd2119d8640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd20c1f4500 0x7fd20c06f130 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd200009a70 tx=0x7fd200033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.972+0000 7fd20affd640 1 -- 172.21.15.59:0/498371968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd200038040 con 0x7fd20c1f4500 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.972+0000 7fd20affd640 1 -- 172.21.15.59:0/498371968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd200033860 con 0x7fd20c1f4500 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.972+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd20c138bf0 con 0x7fd20c1f4500 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.972+0000 7fd20affd640 1 -- 172.21.15.59:0/498371968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd200033c70 con 0x7fd20c1f4500 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.972+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20c1f3d00 con 0x7fd20c1f4500 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.973+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20c1f3af0 con 0x7fd20c1f4500 2024-06-29T14:35:07.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.973+0000 7fd20affd640 1 -- 172.21.15.59:0/498371968 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fd200045020 con 0x7fd20c1f4500 2024-06-29T14:35:07.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.974+0000 7fd20affd640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd1f8076420 0x7fd1f8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.974+0000 7fd20affd640 1 -- 172.21.15.59:0/498371968 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6101+0+0 (secure 0 0 0) 0x7fd2000b8140 con 0x7fd20c1f4500 2024-06-29T14:35:07.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.974+0000 7fd2121d9640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd1f8076420 0x7fd1f8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.975+0000 7fd2121d9640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd1f8076420 0x7fd1f8078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc005fd0 tx=0x7fd1fc009500 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:07.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:07.978+0000 7fd20affd640 1 -- 172.21.15.59:0/498371968 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd20004c020 con 0x7fd20c1f4500 2024-06-29T14:35:08.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:08.214+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd20c0b2dc0 con 0x7fd20c1f4500 2024-06-29T14:35:09.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.154+0000 7fd20affd640 1 -- 172.21.15.59:0/498371968 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7fd200081800 con 0x7fd20c1f4500 2024-06-29T14:35:09.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd1f8076420 msgr2=0x7fd1f8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd1f8076420 0x7fd1f8078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc005fd0 tx=0x7fd1fc009500 comp rx=0 tx=0).stop 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd20c1f4500 msgr2=0x7fd20c06f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd20c1f4500 0x7fd20c06f130 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd200009a70 tx=0x7fd200033670 comp rx=0 tx=0).stop 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 shutdown_connections 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd1f8076420 0x7fd1f8078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd20c1f04b0 0x7fd20c1f08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd20c1f4500 0x7fd20c06f130 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 --2- 172.21.15.59:0/498371968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1f4004800 0x7fd20c1560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 >> 172.21.15.59:0/498371968 conn(0x7fd20c0b6fb0 msgr2=0x7fd20c140a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 shutdown_connections 2024-06-29T14:35:09.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.157+0000 7fd21443d640 1 -- 172.21.15.59:0/498371968 wait complete. 2024-06-29T14:35:09.169 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:35:09.169 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:35:09.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.300+0000 7f0fa552a640 1 -- 172.21.15.59:0/4111483119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa00d1c30 msgr2=0x7f0fa00b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:09.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.300+0000 7f0fa552a640 1 --2- 172.21.15.59:0/4111483119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa00d1c30 0x7f0fa00b5b30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0f90009830 tx=0x7f0f9002a510 comp rx=0 tx=0).stop 2024-06-29T14:35:09.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.301+0000 7f0fa552a640 1 -- 172.21.15.59:0/4111483119 shutdown_connections 2024-06-29T14:35:09.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.301+0000 7f0fa552a640 1 --2- 172.21.15.59:0/4111483119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa00d1c30 0x7f0fa00b5b30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:09.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.301+0000 7f0fa552a640 1 --2- 172.21.15.59:0/4111483119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa00b2270 0x7f0fa015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:09.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.301+0000 7f0fa552a640 1 -- 172.21.15.59:0/4111483119 >> 172.21.15.59:0/4111483119 conn(0x7f0fa0073e10 msgr2=0x7f0fa0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:09.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.301+0000 7f0fa552a640 1 -- 172.21.15.59:0/4111483119 shutdown_connections 2024-06-29T14:35:09.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.301+0000 7f0fa552a640 1 -- 172.21.15.59:0/4111483119 wait complete. 2024-06-29T14:35:09.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.302+0000 7f0fa552a640 1 Processor -- start 2024-06-29T14:35:09.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.303+0000 7f0fa552a640 1 -- start start 2024-06-29T14:35:09.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0fa552a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa00b2270 0x7f0fa0152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:09.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0fa552a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa00b2fa0 0x7f0fa006e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:09.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0fa552a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa00d1c30 0x7f0fa015c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:09.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0fa552a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa00b6400 con 0x7f0fa00b2270 2024-06-29T14:35:09.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0fa552a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa00b6280 con 0x7f0fa00b2fa0 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0fa552a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa00b6580 con 0x7f0fa00d1c30 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa00b2270 0x7f0fa0152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa00b2270 0x7f0fa0152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37826/0 (socket says 172.21.15.59:37826) 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9effd640 1 -- 172.21.15.59:0/1641045326 learned_addr learned my addr 172.21.15.59:0/1641045326 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9f7fe640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa00d1c30 0x7f0fa015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9e7fc640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa00b2fa0 0x7f0fa006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9effd640 1 -- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa00d1c30 msgr2=0x7f0fa015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9effd640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa00d1c30 0x7f0fa015c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9effd640 1 -- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa00b2fa0 msgr2=0x7f0fa006e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9effd640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa00b2fa0 0x7f0fa006e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:09.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.304+0000 7f0f9effd640 1 -- 172.21.15.59:0/1641045326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa01f5e20 con 0x7f0fa00b2270 2024-06-29T14:35:09.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.305+0000 7f0f9effd640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa00b2270 0x7f0fa0152060 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0f9800ecb0 tx=0x7f0f9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:09.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.305+0000 7f0f977fe640 1 -- 172.21.15.59:0/1641045326 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f98003b70 con 0x7f0fa00b2270 2024-06-29T14:35:09.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.305+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa006dd20 con 0x7f0fa00b2270 2024-06-29T14:35:09.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.305+0000 7f0f977fe640 1 -- 172.21.15.59:0/1641045326 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f98003d10 con 0x7f0fa00b2270 2024-06-29T14:35:09.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.305+0000 7f0f977fe640 1 -- 172.21.15.59:0/1641045326 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f98004280 con 0x7f0fa00b2270 2024-06-29T14:35:09.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.305+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa015cce0 con 0x7f0fa00b2270 2024-06-29T14:35:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.306+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fa00d0b60 con 0x7f0fa00b2270 2024-06-29T14:35:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.306+0000 7f0f977fe640 1 -- 172.21.15.59:0/1641045326 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f0f98004420 con 0x7f0fa00b2270 2024-06-29T14:35:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.307+0000 7f0f977fe640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f80076420 0x7f0f80078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.307+0000 7f0f977fe640 1 -- 172.21.15.59:0/1641045326 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f0f98098670 con 0x7f0fa00b2270 2024-06-29T14:35:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.307+0000 7f0f9e7fc640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f80076420 0x7f0f80078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.308+0000 7f0f9e7fc640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f80076420 0x7f0f80078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fa006fd60 tx=0x7f0f88007e90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:09.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.311+0000 7f0f977fe640 1 -- 172.21.15.59:0/1641045326 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f98061550 con 0x7f0fa00b2270 2024-06-29T14:35:09.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:09.543+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0fa00cebc0 con 0x7f0fa00b2270 2024-06-29T14:35:10.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.160+0000 7f0f977fe640 1 -- 172.21.15.59:0/1641045326 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v29) ==== 122+0+0 (secure 0 0 0) 0x7f0f98065850 con 0x7f0fa00b2270 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f80076420 msgr2=0x7f0f80078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f80076420 0x7f0f80078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fa006fd60 tx=0x7f0f88007e90 comp rx=0 tx=0).stop 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa00b2270 msgr2=0x7f0fa0152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa00b2270 0x7f0fa0152060 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f0f9800ecb0 tx=0x7f0f9800c6a0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 shutdown_connections 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f80076420 0x7f0f80078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa00d1c30 0x7f0fa015c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa00b2fa0 0x7f0fa006e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 --2- 172.21.15.59:0/1641045326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa00b2270 0x7f0fa0152060 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.164+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 >> 172.21.15.59:0/1641045326 conn(0x7f0fa0073e10 msgr2=0x7f0fa0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:10.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.165+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 shutdown_connections 2024-06-29T14:35:10.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.165+0000 7f0fa552a640 1 -- 172.21.15.59:0/1641045326 wait complete. 2024-06-29T14:35:10.176 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:35:10.176 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:35:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.303+0000 7fed3cd0a640 1 -- 172.21.15.59:0/1266552825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed380b40a0 msgr2=0x7fed3815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.303+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/1266552825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed380b40a0 0x7fed3815e8d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fed2c005fd0 tx=0x7fed2c02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.303+0000 7fed3cd0a640 1 -- 172.21.15.59:0/1266552825 shutdown_connections 2024-06-29T14:35:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.303+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/1266552825 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed3815ee10 0x7fed3815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.303+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/1266552825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed380b40a0 0x7fed3815e8d0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.303+0000 7fed3cd0a640 1 -- 172.21.15.59:0/1266552825 >> 172.21.15.59:0/1266552825 conn(0x7fed380d3ee0 msgr2=0x7fed380d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.303+0000 7fed3cd0a640 1 -- 172.21.15.59:0/1266552825 shutdown_connections 2024-06-29T14:35:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.303+0000 7fed3cd0a640 1 -- 172.21.15.59:0/1266552825 wait complete. 2024-06-29T14:35:10.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.305+0000 7fed3cd0a640 1 Processor -- start 2024-06-29T14:35:10.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.305+0000 7fed3cd0a640 1 -- start start 2024-06-29T14:35:10.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.306+0000 7fed3cd0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed380d23c0 0x7fed381e6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:10.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.306+0000 7fed3cd0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3815ee10 0x7fed3813fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed3cd0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed381e72f0 0x7fed38156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed3cd0a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fed38160280 con 0x7fed381e72f0 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed3cd0a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed38160100 con 0x7fed3815ee10 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed3cd0a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fed38160400 con 0x7fed380d23c0 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed381e72f0 0x7fed38156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed381e72f0 0x7fed38156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37838/0 (socket says 172.21.15.59:37838) 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 -- 172.21.15.59:0/2457448376 learned_addr learned my addr 172.21.15.59:0/2457448376 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed35d9b640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3815ee10 0x7fed3813fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed3659c640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed380d23c0 0x7fed381e6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 -- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed380d23c0 msgr2=0x7fed381e6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed380d23c0 0x7fed381e6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 -- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3815ee10 msgr2=0x7fed3813fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3815ee10 0x7fed3813fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 -- 172.21.15.59:0/2457448376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3813b4f0 con 0x7fed381e72f0 2024-06-29T14:35:10.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.307+0000 7fed36d9d640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed381e72f0 0x7fed38156d60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fed3000d650 tx=0x7fed3000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:10.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.308+0000 7fed26ffd640 1 -- 172.21.15.59:0/2457448376 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed30017070 con 0x7fed381e72f0 2024-06-29T14:35:10.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.308+0000 7fed26ffd640 1 -- 172.21.15.59:0/2457448376 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed30005310 con 0x7fed381e72f0 2024-06-29T14:35:10.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.308+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3813dc20 con 0x7fed381e72f0 2024-06-29T14:35:10.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.308+0000 7fed26ffd640 1 -- 172.21.15.59:0/2457448376 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed30009820 con 0x7fed381e72f0 2024-06-29T14:35:10.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.308+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3813e1e0 con 0x7fed381e72f0 2024-06-29T14:35:10.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.309+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3813b6c0 con 0x7fed381e72f0 2024-06-29T14:35:10.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.310+0000 7fed26ffd640 1 -- 172.21.15.59:0/2457448376 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fed300099c0 con 0x7fed381e72f0 2024-06-29T14:35:10.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.310+0000 7fed26ffd640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fed180763f0 0x7fed18078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:10.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.310+0000 7fed26ffd640 1 -- 172.21.15.59:0/2457448376 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7fed30097a70 con 0x7fed381e72f0 2024-06-29T14:35:10.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.310+0000 7fed3659c640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fed180763f0 0x7fed18078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:10.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.311+0000 7fed3659c640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fed180763f0 0x7fed18078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed380d2e40 tx=0x7fed28009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:10.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.315+0000 7fed26ffd640 1 -- 172.21.15.59:0/2457448376 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fed300609d0 con 0x7fed381e72f0 2024-06-29T14:35:10.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.544+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed380689f0 con 0x7fed381e72f0 2024-06-29T14:35:10.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.544+0000 7fed26ffd640 1 -- 172.21.15.59:0/2457448376 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5223 (secure 0 0 0) 0x7fed380689f0 con 0x7fed381e72f0 2024-06-29T14:35:10.542 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:10.542 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":29,"btime":"2024-06-29T14:35:10:154758+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/484297155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":484297155},{"type":"v1","addr":"172.21.15.153:6835","nonce":484297155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2309835731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":2309835731},{"type":"v1","addr":"172.21.15.153:6837","nonce":2309835731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":6330,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/413487166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":413487166},{"type":"v1","addr":"172.21.15.59:6834","nonce":413487166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:35:06.529979+0000","modified":"2024-06-29T14:35:09.544199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6333":{"gid":6333,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3627888249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3627888249},{"type":"v1","addr":"172.21.15.59:6835","nonce":3627888249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6333,"qdb_cluster":[6333]},"id":3}]} 2024-06-29T14:35:10.542 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 29 2024-06-29T14:35:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.546+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fed180763f0 msgr2=0x7fed18078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.546+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fed180763f0 0x7fed18078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed380d2e40 tx=0x7fed28009040 comp rx=0 tx=0).stop 2024-06-29T14:35:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.546+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed381e72f0 msgr2=0x7fed38156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.546+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed381e72f0 0x7fed38156d60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fed3000d650 tx=0x7fed3000db10 comp rx=0 tx=0).stop 2024-06-29T14:35:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.547+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 shutdown_connections 2024-06-29T14:35:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.547+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fed180763f0 0x7fed18078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.547+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed381e72f0 0x7fed38156d60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.547+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed3815ee10 0x7fed3813fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.547+0000 7fed3cd0a640 1 --2- 172.21.15.59:0/2457448376 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed380d23c0 0x7fed381e6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.547+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 >> 172.21.15.59:0/2457448376 conn(0x7fed380d3ee0 msgr2=0x7fed380d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:10.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.547+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 shutdown_connections 2024-06-29T14:35:10.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.547+0000 7fed3cd0a640 1 -- 172.21.15.59:0/2457448376 wait complete. 2024-06-29T14:35:10.555 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:35:10.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.676+0000 7fc59637f640 1 -- 172.21.15.59:0/443047138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 msgr2=0x7fc5901629d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.676+0000 7fc59637f640 1 --2- 172.21.15.59:0/443047138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 0x7fc5901629d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc5900b78c0 tx=0x7fc58802a8d0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.676+0000 7fc59637f640 1 -- 172.21.15.59:0/443047138 shutdown_connections 2024-06-29T14:35:10.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.676+0000 7fc59637f640 1 --2- 172.21.15.59:0/443047138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 0x7fc5901629d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.676+0000 7fc59637f640 1 -- 172.21.15.59:0/443047138 >> 172.21.15.59:0/443047138 conn(0x7fc5900b5610 msgr2=0x7fc5900b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:10.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.676+0000 7fc59637f640 1 -- 172.21.15.59:0/443047138 shutdown_connections 2024-06-29T14:35:10.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.676+0000 7fc59637f640 1 -- 172.21.15.59:0/443047138 wait complete. 2024-06-29T14:35:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.678+0000 7fc59637f640 1 Processor -- start 2024-06-29T14:35:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.678+0000 7fc59637f640 1 -- start start 2024-06-29T14:35:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc59637f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc59008c190 0x7fc5901528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc59637f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 0x7fc59014ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc59637f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5900ce950 0x7fc59014f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc59637f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc590169690 con 0x7fc5900b8be0 2024-06-29T14:35:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc59637f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc590169510 con 0x7fc5900ce950 2024-06-29T14:35:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc59637f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc590169810 con 0x7fc59008c190 2024-06-29T14:35:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc587fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 0x7fc59014ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc587fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 0x7fc59014ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37856/0 (socket says 172.21.15.59:37856) 2024-06-29T14:35:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc587fff640 1 -- 172.21.15.59:0/3555437179 learned_addr learned my addr 172.21.15.59:0/3555437179 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc58ffff640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc59008c190 0x7fc5901528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc59491c640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5900ce950 0x7fc59014f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc587fff640 1 -- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc59008c190 msgr2=0x7fc5901528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc587fff640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc59008c190 0x7fc5901528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc587fff640 1 -- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5900ce950 msgr2=0x7fc59014f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc587fff640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5900ce950 0x7fc59014f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc587fff640 1 -- 172.21.15.59:0/3555437179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5901541d0 con 0x7fc5900b8be0 2024-06-29T14:35:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.679+0000 7fc58ffff640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc59008c190 0x7fc5901528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.680+0000 7fc587fff640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 0x7fc59014ec60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c400 tx=0x7fc57800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.680+0000 7fc58d7fa640 1 -- 172.21.15.59:0/3555437179 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578017070 con 0x7fc5900b8be0 2024-06-29T14:35:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.680+0000 7fc58d7fa640 1 -- 172.21.15.59:0/3555437179 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5780053c0 con 0x7fc5900b8be0 2024-06-29T14:35:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.680+0000 7fc58d7fa640 1 -- 172.21.15.59:0/3555437179 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578003830 con 0x7fc5900b8be0 2024-06-29T14:35:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.680+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc590153130 con 0x7fc5900b8be0 2024-06-29T14:35:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.680+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5901536f0 con 0x7fc5900b8be0 2024-06-29T14:35:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.681+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc590152ea0 con 0x7fc5900b8be0 2024-06-29T14:35:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.682+0000 7fc58d7fa640 1 -- 172.21.15.59:0/3555437179 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fc57801a070 con 0x7fc5900b8be0 2024-06-29T14:35:10.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.682+0000 7fc58d7fa640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc5700763f0 0x7fc570078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:10.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.682+0000 7fc58d7fa640 1 -- 172.21.15.59:0/3555437179 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7fc578096eb0 con 0x7fc5900b8be0 2024-06-29T14:35:10.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.683+0000 7fc58ffff640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc5700763f0 0x7fc570078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:10.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.683+0000 7fc58ffff640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc5700763f0 0x7fc570078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc580000f60 tx=0x7fc580009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:10.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.687+0000 7fc58d7fa640 1 -- 172.21.15.59:0/3555437179 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc578060570 con 0x7fc5900b8be0 2024-06-29T14:35:10.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.880+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5900cf080 con 0x7fc5900b8be0 2024-06-29T14:35:10.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.882+0000 7fc58d7fa640 1 -- 172.21.15.59:0/3555437179 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16954 (secure 0 0 0) 0x7fc57805fe70 con 0x7fc5900b8be0 2024-06-29T14:35:10.879 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:10.879 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":71,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:35:08.139668+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:35:04.881993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-06-29T14:35:05.467776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:35:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.883+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc5700763f0 msgr2=0x7fc570078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.883+0000 7fc59637f640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc5700763f0 0x7fc570078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc580000f60 tx=0x7fc580009040 comp rx=0 tx=0).stop 2024-06-29T14:35:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.883+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 msgr2=0x7fc59014ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.883+0000 7fc59637f640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 0x7fc59014ec60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c400 tx=0x7fc57800c8c0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.884+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 shutdown_connections 2024-06-29T14:35:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.884+0000 7fc59637f640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fc5700763f0 0x7fc570078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.884+0000 7fc59637f640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5900ce950 0x7fc59014f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.884+0000 7fc59637f640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5900b8be0 0x7fc59014ec60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.884+0000 7fc59637f640 1 --2- 172.21.15.59:0/3555437179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc59008c190 0x7fc5901528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:10.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.884+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 >> 172.21.15.59:0/3555437179 conn(0x7fc5900b5610 msgr2=0x7fc5900b7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:10.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.884+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 shutdown_connections 2024-06-29T14:35:10.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:10.884+0000 7fc59637f640 1 -- 172.21.15.59:0/3555437179 wait complete. 2024-06-29T14:35:10.892 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:35:10.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:35:11.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.013+0000 7fba23d7a640 1 -- 172.21.15.59:0/2649133372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 msgr2=0x7fba1c144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.013+0000 7fba23d7a640 1 --2- 172.21.15.59:0/2649133372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 0x7fba1c144350 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0b3b00 tx=0x7fba1002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.013+0000 7fba23d7a640 1 -- 172.21.15.59:0/2649133372 shutdown_connections 2024-06-29T14:35:11.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.013+0000 7fba23d7a640 1 --2- 172.21.15.59:0/2649133372 >> v2:172.21.15.153:3300/0 conn(0x7fba1c0b1b00 0x7fba1c0b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.013+0000 7fba23d7a640 1 --2- 172.21.15.59:0/2649133372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 0x7fba1c144350 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.013+0000 7fba23d7a640 1 -- 172.21.15.59:0/2649133372 >> 172.21.15.59:0/2649133372 conn(0x7fba1c0b06e0 msgr2=0x7fba1c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:11.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.013+0000 7fba23d7a640 1 -- 172.21.15.59:0/2649133372 shutdown_connections 2024-06-29T14:35:11.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.013+0000 7fba23d7a640 1 -- 172.21.15.59:0/2649133372 wait complete. 2024-06-29T14:35:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.015+0000 7fba23d7a640 1 Processor -- start 2024-06-29T14:35:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.015+0000 7fba23d7a640 1 -- start start 2024-06-29T14:35:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba23d7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1c0b1b00 0x7fba1c0c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba23d7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 0x7fba1c0cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba23d7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1c15f5a0 0x7fba1c131410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba23d7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c160e50 con 0x7fba1c15eed0 2024-06-29T14:35:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba23d7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c160cd0 con 0x7fba1c0b1b00 2024-06-29T14:35:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba23d7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fba1c160fd0 con 0x7fba1c15f5a0 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21315640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 0x7fba1c0cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21315640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 0x7fba1c0cf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37876/0 (socket says 172.21.15.59:37876) 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21315640 1 -- 172.21.15.59:0/875000256 learned_addr learned my addr 172.21.15.59:0/875000256 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21b16640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1c0b1b00 0x7fba1c0c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba22317640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1c15f5a0 0x7fba1c131410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21315640 1 -- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1c15f5a0 msgr2=0x7fba1c131410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21315640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1c15f5a0 0x7fba1c131410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21315640 1 -- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1c0b1b00 msgr2=0x7fba1c0c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21315640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1c0b1b00 0x7fba1c0c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.016+0000 7fba21315640 1 -- 172.21.15.59:0/875000256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c132150 con 0x7fba1c15eed0 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.017+0000 7fba21315640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 0x7fba1c0cf5d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fba10000c00 tx=0x7fba10003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.017+0000 7fba1a7fc640 1 -- 172.21.15.59:0/875000256 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1003e030 con 0x7fba1c15eed0 2024-06-29T14:35:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.017+0000 7fba1a7fc640 1 -- 172.21.15.59:0/875000256 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba10003e00 con 0x7fba1c15eed0 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.017+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1c1329f0 con 0x7fba1c15eed0 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.017+0000 7fba1a7fc640 1 -- 172.21.15.59:0/875000256 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10035040 con 0x7fba1c15eed0 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.017+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba1c134270 con 0x7fba1c15eed0 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.018+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba1c161570 con 0x7fba1c15eed0 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.019+0000 7fba1a7fc640 1 -- 172.21.15.59:0/875000256 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fba10041030 con 0x7fba1c15eed0 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.019+0000 7fba1a7fc640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fba08076420 0x7fba08078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.019+0000 7fba1a7fc640 1 -- 172.21.15.59:0/875000256 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7fba100b8250 con 0x7fba1c15eed0 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.019+0000 7fba21b16640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fba08076420 0x7fba08078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.020+0000 7fba21b16640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fba08076420 0x7fba08078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba14006fd0 tx=0x7fba14009060 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.024+0000 7fba1a7fc640 1 -- 172.21.15.59:0/875000256 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fba1c0b2450 con 0x7fba1c15eed0 2024-06-29T14:35:11.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.212+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 --> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fba1c132320 con 0x7fba08076420 2024-06-29T14:35:11.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.218+0000 7fba1a7fc640 1 -- 172.21.15.59:0/875000256 <== mgr.5809 v2:172.21.15.153:6832/2957341575 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246630 (secure 0 0 0) 0x7fba1c132320 con 0x7fba08076420 2024-06-29T14:35:11.218 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:11.220 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:35:11.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.220+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fba08076420 msgr2=0x7fba08078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.220+0000 7fba23d7a640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fba08076420 0x7fba08078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba14006fd0 tx=0x7fba14009060 comp rx=0 tx=0).stop 2024-06-29T14:35:11.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.220+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 msgr2=0x7fba1c0cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.220+0000 7fba23d7a640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 0x7fba1c0cf5d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fba10000c00 tx=0x7fba10003910 comp rx=0 tx=0).stop 2024-06-29T14:35:11.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.221+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 shutdown_connections 2024-06-29T14:35:11.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.221+0000 7fba23d7a640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fba08076420 0x7fba08078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.221+0000 7fba23d7a640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba1c15f5a0 0x7fba1c131410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.221+0000 7fba23d7a640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba1c15eed0 0x7fba1c0cf5d0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.221+0000 7fba23d7a640 1 --2- 172.21.15.59:0/875000256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba1c0b1b00 0x7fba1c0c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.221+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 >> 172.21.15.59:0/875000256 conn(0x7fba1c0b06e0 msgr2=0x7fba1c143630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:11.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.221+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 shutdown_connections 2024-06-29T14:35:11.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.221+0000 7fba23d7a640 1 -- 172.21.15.59:0/875000256 wait complete. 2024-06-29T14:35:11.229 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-06-29T14:35:10.033416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323648,"kb_used_data":7560,"kb_used_omap":12,"kb_used_meta":315955,"kb_avail":754651072,"statfs":{"total":773094113280,"available":772762697728,"internally_reserved":0,"allocated":7741440,"data_stored":4437072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323538482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.908141"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151105+0000","last_change":"2024-06-29T14:35:06.151105+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151105+0000","last_undegraded":"2024-06-29T14:35:06.151105+0000","last_fullsized":"2024-06-29T14:35:06.151105+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138965+0000","last_change":"2024-06-29T14:35:06.138965+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138965+0000","last_undegraded":"2024-06-29T14:35:06.138965+0000","last_fullsized":"2024-06-29T14:35:06.138965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138100+0000","last_change":"2024-06-29T14:35:06.138100+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138100+0000","last_undegraded":"2024-06-29T14:35:06.138100+0000","last_fullsized":"2024-06-29T14:35:06.138100+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138146+0000","last_change":"2024-06-29T14:35:06.138146+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138146+0000","last_undegraded":"2024-06-29T14:35:06.138146+0000","last_fullsized":"2024-06-29T14:35:06.138146+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132969+0000","last_change":"2024-06-29T14:35:06.132969+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132969+0000","last_undegraded":"2024-06-29T14:35:06.132969+0000","last_fullsized":"2024-06-29T14:35:06.132969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149626+0000","last_change":"2024-06-29T14:35:06.149626+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149626+0000","last_undegraded":"2024-06-29T14:35:06.149626+0000","last_fullsized":"2024-06-29T14:35:06.149626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135540+0000","last_change":"2024-06-29T14:35:06.135540+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135540+0000","last_undegraded":"2024-06-29T14:35:06.135540+0000","last_fullsized":"2024-06-29T14:35:06.135540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.148127+0000","last_change":"2024-06-29T14:35:06.148127+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.148127+0000","last_undegraded":"2024-06-29T14:35:06.148127+0000","last_fullsized":"2024-06-29T14:35:06.148127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.147564+0000","last_change":"2024-06-29T14:35:06.147564+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.147564+0000","last_undegraded":"2024-06-29T14:35:06.147564+0000","last_fullsized":"2024-06-29T14:35:06.147564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151644+0000","last_change":"2024-06-29T14:35:06.151644+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151644+0000","last_undegraded":"2024-06-29T14:35:06.151644+0000","last_fullsized":"2024-06-29T14:35:06.151644+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.144328+0000","last_change":"2024-06-29T14:35:06.144328+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.144328+0000","last_undegraded":"2024-06-29T14:35:06.144328+0000","last_fullsized":"2024-06-29T14:35:06.144328+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146980+0000","last_change":"2024-06-29T14:35:06.146980+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146980+0000","last_undegraded":"2024-06-29T14:35:06.146980+0000","last_fullsized":"2024-06-29T14:35:06.146980+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151643+0000","last_change":"2024-06-29T14:35:06.151643+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151643+0000","last_undegraded":"2024-06-29T14:35:06.151643+0000","last_fullsized":"2024-06-29T14:35:06.151643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136411+0000","last_change":"2024-06-29T14:35:06.136411+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136411+0000","last_undegraded":"2024-06-29T14:35:06.136411+0000","last_fullsized":"2024-06-29T14:35:06.136411+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137811+0000","last_change":"2024-06-29T14:35:06.137811+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137811+0000","last_undegraded":"2024-06-29T14:35:06.137811+0000","last_fullsized":"2024-06-29T14:35:06.137811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146620+0000","last_change":"2024-06-29T14:35:06.146620+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146620+0000","last_undegraded":"2024-06-29T14:35:06.146620+0000","last_fullsized":"2024-06-29T14:35:06.146620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137727+0000","last_change":"2024-06-29T14:35:06.137727+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137727+0000","last_undegraded":"2024-06-29T14:35:06.137727+0000","last_fullsized":"2024-06-29T14:35:06.137727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138015+0000","last_change":"2024-06-29T14:35:06.138015+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138015+0000","last_undegraded":"2024-06-29T14:35:06.138015+0000","last_fullsized":"2024-06-29T14:35:06.138015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137633+0000","last_change":"2024-06-29T14:35:06.137633+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137633+0000","last_undegraded":"2024-06-29T14:35:06.137633+0000","last_fullsized":"2024-06-29T14:35:06.137633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135000+0000","last_change":"2024-06-29T14:35:06.135000+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135000+0000","last_undegraded":"2024-06-29T14:35:06.135000+0000","last_fullsized":"2024-06-29T14:35:06.135000+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.133885+0000","last_change":"2024-06-29T14:35:06.133885+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.133885+0000","last_undegraded":"2024-06-29T14:35:06.133885+0000","last_fullsized":"2024-06-29T14:35:06.133885+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146360+0000","last_change":"2024-06-29T14:35:06.146360+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146360+0000","last_undegraded":"2024-06-29T14:35:06.146360+0000","last_fullsized":"2024-06-29T14:35:06.146360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141017+0000","last_change":"2024-06-29T14:35:06.141017+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141017+0000","last_undegraded":"2024-06-29T14:35:06.141017+0000","last_fullsized":"2024-06-29T14:35:06.141017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138890+0000","last_change":"2024-06-29T14:35:06.138890+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138890+0000","last_undegraded":"2024-06-29T14:35:06.138890+0000","last_fullsized":"2024-06-29T14:35:06.138890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137237+0000","last_change":"2024-06-29T14:35:06.137237+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137237+0000","last_undegraded":"2024-06-29T14:35:06.137237+0000","last_fullsized":"2024-06-29T14:35:06.137237+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.145779+0000","last_change":"2024-06-29T14:35:06.145779+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.145779+0000","last_undegraded":"2024-06-29T14:35:06.145779+0000","last_fullsized":"2024-06-29T14:35:06.145779+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.155904+0000","last_change":"2024-06-29T14:35:06.155904+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.155904+0000","last_undegraded":"2024-06-29T14:35:06.155904+0000","last_fullsized":"2024-06-29T14:35:06.155904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134043+0000","last_change":"2024-06-29T14:35:06.134043+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134043+0000","last_undegraded":"2024-06-29T14:35:06.134043+0000","last_fullsized":"2024-06-29T14:35:06.134043+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137017+0000","last_change":"2024-06-29T14:35:06.137017+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137017+0000","last_undegraded":"2024-06-29T14:35:06.137017+0000","last_fullsized":"2024-06-29T14:35:06.137017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136330+0000","last_change":"2024-06-29T14:35:06.136330+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136330+0000","last_undegraded":"2024-06-29T14:35:06.136330+0000","last_fullsized":"2024-06-29T14:35:06.136330+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134973+0000","last_change":"2024-06-29T14:35:06.134973+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134973+0000","last_undegraded":"2024-06-29T14:35:06.134973+0000","last_fullsized":"2024-06-29T14:35:06.134973+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139697+0000","last_change":"2024-06-29T14:35:06.139697+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139697+0000","last_undegraded":"2024-06-29T14:35:06.139697+0000","last_fullsized":"2024-06-29T14:35:06.139697+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137035+0000","last_change":"2024-06-29T14:35:06.137035+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137035+0000","last_undegraded":"2024-06-29T14:35:06.137035+0000","last_fullsized":"2024-06-29T14:35:06.137035+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.141964+0000","last_change":"2024-06-29T14:35:06.139267+0000","last_active":"2024-06-29T14:35:06.141964+0000","last_peered":"2024-06-29T14:35:06.141964+0000","last_clean":"2024-06-29T14:35:06.141964+0000","last_became_active":"2024-06-29T14:35:06.138932+0000","last_became_peered":"2024-06-29T14:35:06.138932+0000","last_unstale":"2024-06-29T14:35:06.141964+0000","last_undegraded":"2024-06-29T14:35:06.141964+0000","last_fullsized":"2024-06-29T14:35:06.141964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:47.855244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136457+0000","last_change":"2024-06-29T14:35:06.136457+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136457+0000","last_undegraded":"2024-06-29T14:35:06.136457+0000","last_fullsized":"2024-06-29T14:35:06.136457+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.158642+0000","last_change":"2024-06-29T14:35:06.157899+0000","last_active":"2024-06-29T14:35:06.158642+0000","last_peered":"2024-06-29T14:35:06.158642+0000","last_clean":"2024-06-29T14:35:06.158642+0000","last_became_active":"2024-06-29T14:35:06.157591+0000","last_became_peered":"2024-06-29T14:35:06.157591+0000","last_unstale":"2024-06-29T14:35:06.158642+0000","last_undegraded":"2024-06-29T14:35:06.158642+0000","last_fullsized":"2024-06-29T14:35:06.158642+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:04:01.035476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134883+0000","last_change":"2024-06-29T14:35:06.134883+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134883+0000","last_undegraded":"2024-06-29T14:35:06.134883+0000","last_fullsized":"2024-06-29T14:35:06.134883+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.162625+0000","last_change":"2024-06-29T14:35:06.160847+0000","last_active":"2024-06-29T14:35:06.162625+0000","last_peered":"2024-06-29T14:35:06.162625+0000","last_clean":"2024-06-29T14:35:06.162625+0000","last_became_active":"2024-06-29T14:35:06.160558+0000","last_became_peered":"2024-06-29T14:35:06.160558+0000","last_unstale":"2024-06-29T14:35:06.162625+0000","last_undegraded":"2024-06-29T14:35:06.162625+0000","last_fullsized":"2024-06-29T14:35:06.162625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:29.140867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.142843+0000","last_change":"2024-06-29T14:35:06.142843+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.142843+0000","last_undegraded":"2024-06-29T14:35:06.142843+0000","last_fullsized":"2024-06-29T14:35:06.142843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.147719+0000","last_change":"2024-06-29T14:35:06.147172+0000","last_active":"2024-06-29T14:35:06.147719+0000","last_peered":"2024-06-29T14:35:06.147719+0000","last_clean":"2024-06-29T14:35:06.147719+0000","last_became_active":"2024-06-29T14:35:06.146877+0000","last_became_peered":"2024-06-29T14:35:06.146877+0000","last_unstale":"2024-06-29T14:35:06.147719+0000","last_undegraded":"2024-06-29T14:35:06.147719+0000","last_fullsized":"2024-06-29T14:35:06.147719+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:11:45.335558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132273+0000","last_change":"2024-06-29T14:35:06.132273+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132273+0000","last_undegraded":"2024-06-29T14:35:06.132273+0000","last_fullsized":"2024-06-29T14:35:06.132273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.155462+0000","last_change":"2024-06-29T14:35:06.154311+0000","last_active":"2024-06-29T14:35:06.155462+0000","last_peered":"2024-06-29T14:35:06.155462+0000","last_clean":"2024-06-29T14:35:06.155462+0000","last_became_active":"2024-06-29T14:35:06.153161+0000","last_became_peered":"2024-06-29T14:35:06.153161+0000","last_unstale":"2024-06-29T14:35:06.155462+0000","last_undegraded":"2024-06-29T14:35:06.155462+0000","last_fullsized":"2024-06-29T14:35:06.155462+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:03.504751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138048+0000","last_change":"2024-06-29T14:35:06.138048+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138048+0000","last_undegraded":"2024-06-29T14:35:06.138048+0000","last_fullsized":"2024-06-29T14:35:06.138048+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144211+0000","last_change":"2024-06-29T14:35:06.142746+0000","last_active":"2024-06-29T14:35:06.144211+0000","last_peered":"2024-06-29T14:35:06.144211+0000","last_clean":"2024-06-29T14:35:06.144211+0000","last_became_active":"2024-06-29T14:35:06.142398+0000","last_became_peered":"2024-06-29T14:35:06.142398+0000","last_unstale":"2024-06-29T14:35:06.144211+0000","last_undegraded":"2024-06-29T14:35:06.144211+0000","last_fullsized":"2024-06-29T14:35:06.144211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:52:09.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.143419+0000","last_change":"2024-06-29T14:35:06.143419+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.143419+0000","last_undegraded":"2024-06-29T14:35:06.143419+0000","last_fullsized":"2024-06-29T14:35:06.143419+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.142739+0000","last_change":"2024-06-29T14:35:06.141432+0000","last_active":"2024-06-29T14:35:06.142739+0000","last_peered":"2024-06-29T14:35:06.142739+0000","last_clean":"2024-06-29T14:35:06.142739+0000","last_became_active":"2024-06-29T14:35:06.141079+0000","last_became_peered":"2024-06-29T14:35:06.141079+0000","last_unstale":"2024-06-29T14:35:06.142739+0000","last_undegraded":"2024-06-29T14:35:06.142739+0000","last_fullsized":"2024-06-29T14:35:06.142739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:28.307716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136427+0000","last_change":"2024-06-29T14:35:06.136427+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136427+0000","last_undegraded":"2024-06-29T14:35:06.136427+0000","last_fullsized":"2024-06-29T14:35:06.136427+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154777+0000","last_change":"2024-06-29T14:35:06.153920+0000","last_active":"2024-06-29T14:35:06.154777+0000","last_peered":"2024-06-29T14:35:06.154777+0000","last_clean":"2024-06-29T14:35:06.154777+0000","last_became_active":"2024-06-29T14:35:06.153047+0000","last_became_peered":"2024-06-29T14:35:06.153047+0000","last_unstale":"2024-06-29T14:35:06.154777+0000","last_undegraded":"2024-06-29T14:35:06.154777+0000","last_fullsized":"2024-06-29T14:35:06.154777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:42.614890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132166+0000","last_change":"2024-06-29T14:35:06.132166+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132166+0000","last_undegraded":"2024-06-29T14:35:06.132166+0000","last_fullsized":"2024-06-29T14:35:06.132166+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.146782+0000","last_change":"2024-06-29T14:35:06.145945+0000","last_active":"2024-06-29T14:35:06.146782+0000","last_peered":"2024-06-29T14:35:06.146782+0000","last_clean":"2024-06-29T14:35:06.146782+0000","last_became_active":"2024-06-29T14:35:06.145555+0000","last_became_peered":"2024-06-29T14:35:06.145555+0000","last_unstale":"2024-06-29T14:35:06.146782+0000","last_undegraded":"2024-06-29T14:35:06.146782+0000","last_fullsized":"2024-06-29T14:35:06.146782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:53.987132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136980+0000","last_change":"2024-06-29T14:35:06.136980+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136980+0000","last_undegraded":"2024-06-29T14:35:06.136980+0000","last_fullsized":"2024-06-29T14:35:06.136980+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156774+0000","last_change":"2024-06-29T14:35:06.156160+0000","last_active":"2024-06-29T14:35:06.156774+0000","last_peered":"2024-06-29T14:35:06.156774+0000","last_clean":"2024-06-29T14:35:06.156774+0000","last_became_active":"2024-06-29T14:35:06.155823+0000","last_became_peered":"2024-06-29T14:35:06.155823+0000","last_unstale":"2024-06-29T14:35:06.156774+0000","last_undegraded":"2024-06-29T14:35:06.156774+0000","last_fullsized":"2024-06-29T14:35:06.156774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:00.015532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135625+0000","last_change":"2024-06-29T14:35:06.135625+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135625+0000","last_undegraded":"2024-06-29T14:35:06.135625+0000","last_fullsized":"2024-06-29T14:35:06.135625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.159311+0000","last_change":"2024-06-29T14:35:06.158866+0000","last_active":"2024-06-29T14:35:06.159311+0000","last_peered":"2024-06-29T14:35:06.159311+0000","last_clean":"2024-06-29T14:35:06.159311+0000","last_became_active":"2024-06-29T14:35:06.158615+0000","last_became_peered":"2024-06-29T14:35:06.158615+0000","last_unstale":"2024-06-29T14:35:06.159311+0000","last_undegraded":"2024-06-29T14:35:06.159311+0000","last_fullsized":"2024-06-29T14:35:06.159311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:34.626783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132361+0000","last_change":"2024-06-29T14:35:06.132361+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132361+0000","last_undegraded":"2024-06-29T14:35:06.132361+0000","last_fullsized":"2024-06-29T14:35:06.132361+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153751+0000","last_change":"2024-06-29T14:35:06.148671+0000","last_active":"2024-06-29T14:35:06.153751+0000","last_peered":"2024-06-29T14:35:06.153751+0000","last_clean":"2024-06-29T14:35:06.153751+0000","last_became_active":"2024-06-29T14:35:06.148322+0000","last_became_peered":"2024-06-29T14:35:06.148322+0000","last_unstale":"2024-06-29T14:35:06.153751+0000","last_undegraded":"2024-06-29T14:35:06.153751+0000","last_fullsized":"2024-06-29T14:35:06.153751+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:52.995413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139128+0000","last_change":"2024-06-29T14:35:06.139128+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139128+0000","last_undegraded":"2024-06-29T14:35:06.139128+0000","last_fullsized":"2024-06-29T14:35:06.139128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153807+0000","last_change":"2024-06-29T14:35:06.148685+0000","last_active":"2024-06-29T14:35:06.153807+0000","last_peered":"2024-06-29T14:35:06.153807+0000","last_clean":"2024-06-29T14:35:06.153807+0000","last_became_active":"2024-06-29T14:35:06.148343+0000","last_became_peered":"2024-06-29T14:35:06.148343+0000","last_unstale":"2024-06-29T14:35:06.153807+0000","last_undegraded":"2024-06-29T14:35:06.153807+0000","last_fullsized":"2024-06-29T14:35:06.153807+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:30.741869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137093+0000","last_change":"2024-06-29T14:35:06.137093+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137093+0000","last_undegraded":"2024-06-29T14:35:06.137093+0000","last_fullsized":"2024-06-29T14:35:06.137093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.146783+0000","last_change":"2024-06-29T14:35:06.146043+0000","last_active":"2024-06-29T14:35:06.146783+0000","last_peered":"2024-06-29T14:35:06.146783+0000","last_clean":"2024-06-29T14:35:06.146783+0000","last_became_active":"2024-06-29T14:35:06.145648+0000","last_became_peered":"2024-06-29T14:35:06.145648+0000","last_unstale":"2024-06-29T14:35:06.146783+0000","last_undegraded":"2024-06-29T14:35:06.146783+0000","last_fullsized":"2024-06-29T14:35:06.146783+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:23.312796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.143970+0000","last_change":"2024-06-29T14:35:06.143970+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.143970+0000","last_undegraded":"2024-06-29T14:35:06.143970+0000","last_fullsized":"2024-06-29T14:35:06.143970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144837+0000","last_change":"2024-06-29T14:35:06.144003+0000","last_active":"2024-06-29T14:35:06.144837+0000","last_peered":"2024-06-29T14:35:06.144837+0000","last_clean":"2024-06-29T14:35:06.144837+0000","last_became_active":"2024-06-29T14:35:06.143542+0000","last_became_peered":"2024-06-29T14:35:06.143542+0000","last_unstale":"2024-06-29T14:35:06.144837+0000","last_undegraded":"2024-06-29T14:35:06.144837+0000","last_fullsized":"2024-06-29T14:35:06.144837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:20.980556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138217+0000","last_change":"2024-06-29T14:35:06.138217+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138217+0000","last_undegraded":"2024-06-29T14:35:06.138217+0000","last_fullsized":"2024-06-29T14:35:06.138217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.145401+0000","last_change":"2024-06-29T14:35:06.143641+0000","last_active":"2024-06-29T14:35:06.145401+0000","last_peered":"2024-06-29T14:35:06.145401+0000","last_clean":"2024-06-29T14:35:06.145401+0000","last_became_active":"2024-06-29T14:35:06.143030+0000","last_became_peered":"2024-06-29T14:35:06.143030+0000","last_unstale":"2024-06-29T14:35:06.145401+0000","last_undegraded":"2024-06-29T14:35:06.145401+0000","last_fullsized":"2024-06-29T14:35:06.145401+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:21:59.463680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141803+0000","last_change":"2024-06-29T14:35:06.141803+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141803+0000","last_undegraded":"2024-06-29T14:35:06.141803+0000","last_fullsized":"2024-06-29T14:35:06.141803+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156969+0000","last_change":"2024-06-29T14:35:06.155992+0000","last_active":"2024-06-29T14:35:06.156969+0000","last_peered":"2024-06-29T14:35:06.156969+0000","last_clean":"2024-06-29T14:35:06.156969+0000","last_became_active":"2024-06-29T14:35:06.155609+0000","last_became_peered":"2024-06-29T14:35:06.155609+0000","last_unstale":"2024-06-29T14:35:06.156969+0000","last_undegraded":"2024-06-29T14:35:06.156969+0000","last_fullsized":"2024-06-29T14:35:06.156969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:59.794303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.147888+0000","last_change":"2024-06-29T14:35:06.147888+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.147888+0000","last_undegraded":"2024-06-29T14:35:06.147888+0000","last_fullsized":"2024-06-29T14:35:06.147888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.150684+0000","last_change":"2024-06-29T14:35:06.145742+0000","last_active":"2024-06-29T14:35:06.150684+0000","last_peered":"2024-06-29T14:35:06.150684+0000","last_clean":"2024-06-29T14:35:06.150684+0000","last_became_active":"2024-06-29T14:35:06.145327+0000","last_became_peered":"2024-06-29T14:35:06.145327+0000","last_unstale":"2024-06-29T14:35:06.150684+0000","last_undegraded":"2024-06-29T14:35:06.150684+0000","last_fullsized":"2024-06-29T14:35:06.150684+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:40:50.966470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132769+0000","last_change":"2024-06-29T14:35:06.132769+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132769+0000","last_undegraded":"2024-06-29T14:35:06.132769+0000","last_fullsized":"2024-06-29T14:35:06.132769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.160532+0000","last_change":"2024-06-29T14:35:06.159992+0000","last_active":"2024-06-29T14:35:06.160532+0000","last_peered":"2024-06-29T14:35:06.160532+0000","last_clean":"2024-06-29T14:35:06.160532+0000","last_became_active":"2024-06-29T14:35:06.159710+0000","last_became_peered":"2024-06-29T14:35:06.159710+0000","last_unstale":"2024-06-29T14:35:06.160532+0000","last_undegraded":"2024-06-29T14:35:06.160532+0000","last_fullsized":"2024-06-29T14:35:06.160532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:47.483637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149852+0000","last_change":"2024-06-29T14:35:06.149852+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149852+0000","last_undegraded":"2024-06-29T14:35:06.149852+0000","last_fullsized":"2024-06-29T14:35:06.149852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.149922+0000","last_change":"2024-06-29T14:35:06.149423+0000","last_active":"2024-06-29T14:35:06.149922+0000","last_peered":"2024-06-29T14:35:06.149922+0000","last_clean":"2024-06-29T14:35:06.149922+0000","last_became_active":"2024-06-29T14:35:06.149111+0000","last_became_peered":"2024-06-29T14:35:06.149111+0000","last_unstale":"2024-06-29T14:35:06.149922+0000","last_undegraded":"2024-06-29T14:35:06.149922+0000","last_fullsized":"2024-06-29T14:35:06.149922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:29.397336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138649+0000","last_change":"2024-06-29T14:35:06.138649+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138649+0000","last_undegraded":"2024-06-29T14:35:06.138649+0000","last_fullsized":"2024-06-29T14:35:06.138649+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.159867+0000","last_change":"2024-06-29T14:35:06.159208+0000","last_active":"2024-06-29T14:35:06.159867+0000","last_peered":"2024-06-29T14:35:06.159867+0000","last_clean":"2024-06-29T14:35:06.159867+0000","last_became_active":"2024-06-29T14:35:06.158901+0000","last_became_peered":"2024-06-29T14:35:06.158901+0000","last_unstale":"2024-06-29T14:35:06.159867+0000","last_undegraded":"2024-06-29T14:35:06.159867+0000","last_fullsized":"2024-06-29T14:35:06.159867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:41:37.874410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.133545+0000","last_change":"2024-06-29T14:35:06.133545+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.133545+0000","last_undegraded":"2024-06-29T14:35:06.133545+0000","last_fullsized":"2024-06-29T14:35:06.133545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.145350+0000","last_change":"2024-06-29T14:35:06.142795+0000","last_active":"2024-06-29T14:35:06.145350+0000","last_peered":"2024-06-29T14:35:06.145350+0000","last_clean":"2024-06-29T14:35:06.145350+0000","last_became_active":"2024-06-29T14:35:06.142463+0000","last_became_peered":"2024-06-29T14:35:06.142463+0000","last_unstale":"2024-06-29T14:35:06.145350+0000","last_undegraded":"2024-06-29T14:35:06.145350+0000","last_fullsized":"2024-06-29T14:35:06.145350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:13.219465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136269+0000","last_change":"2024-06-29T14:35:06.136269+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136269+0000","last_undegraded":"2024-06-29T14:35:06.136269+0000","last_fullsized":"2024-06-29T14:35:06.136269+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156310+0000","last_change":"2024-06-29T14:35:06.154839+0000","last_active":"2024-06-29T14:35:06.156310+0000","last_peered":"2024-06-29T14:35:06.156310+0000","last_clean":"2024-06-29T14:35:06.156310+0000","last_became_active":"2024-06-29T14:35:06.154513+0000","last_became_peered":"2024-06-29T14:35:06.154513+0000","last_unstale":"2024-06-29T14:35:06.156310+0000","last_undegraded":"2024-06-29T14:35:06.156310+0000","last_fullsized":"2024-06-29T14:35:06.156310+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:13.939011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.150280+0000","last_change":"2024-06-29T14:35:06.150280+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.150280+0000","last_undegraded":"2024-06-29T14:35:06.150280+0000","last_fullsized":"2024-06-29T14:35:06.150280+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144749+0000","last_change":"2024-06-29T14:35:06.143667+0000","last_active":"2024-06-29T14:35:06.144749+0000","last_peered":"2024-06-29T14:35:06.144749+0000","last_clean":"2024-06-29T14:35:06.144749+0000","last_became_active":"2024-06-29T14:35:06.143312+0000","last_became_peered":"2024-06-29T14:35:06.143312+0000","last_unstale":"2024-06-29T14:35:06.144749+0000","last_undegraded":"2024-06-29T14:35:06.144749+0000","last_fullsized":"2024-06-29T14:35:06.144749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:00.265866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139175+0000","last_change":"2024-06-29T14:35:06.139175+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139175+0000","last_undegraded":"2024-06-29T14:35:06.139175+0000","last_fullsized":"2024-06-29T14:35:06.139175+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153279+0000","last_change":"2024-06-29T14:35:06.152663+0000","last_active":"2024-06-29T14:35:06.153279+0000","last_peered":"2024-06-29T14:35:06.153279+0000","last_clean":"2024-06-29T14:35:06.153279+0000","last_became_active":"2024-06-29T14:35:06.152267+0000","last_became_peered":"2024-06-29T14:35:06.152267+0000","last_unstale":"2024-06-29T14:35:06.153279+0000","last_undegraded":"2024-06-29T14:35:06.153279+0000","last_fullsized":"2024-06-29T14:35:06.153279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:15.826985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149080+0000","last_change":"2024-06-29T14:35:06.149080+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149080+0000","last_undegraded":"2024-06-29T14:35:06.149080+0000","last_fullsized":"2024-06-29T14:35:06.149080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.155693+0000","last_change":"2024-06-29T14:35:06.155153+0000","last_active":"2024-06-29T14:35:06.155693+0000","last_peered":"2024-06-29T14:35:06.155693+0000","last_clean":"2024-06-29T14:35:06.155693+0000","last_became_active":"2024-06-29T14:35:06.154919+0000","last_became_peered":"2024-06-29T14:35:06.154919+0000","last_unstale":"2024-06-29T14:35:06.155693+0000","last_undegraded":"2024-06-29T14:35:06.155693+0000","last_fullsized":"2024-06-29T14:35:06.155693+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:21.593203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141904+0000","last_change":"2024-06-29T14:35:06.141904+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141904+0000","last_undegraded":"2024-06-29T14:35:06.141904+0000","last_fullsized":"2024-06-29T14:35:06.141904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154161+0000","last_change":"2024-06-29T14:35:06.149483+0000","last_active":"2024-06-29T14:35:06.154161+0000","last_peered":"2024-06-29T14:35:06.154161+0000","last_clean":"2024-06-29T14:35:06.154161+0000","last_became_active":"2024-06-29T14:35:06.149163+0000","last_became_peered":"2024-06-29T14:35:06.149163+0000","last_unstale":"2024-06-29T14:35:06.154161+0000","last_undegraded":"2024-06-29T14:35:06.154161+0000","last_fullsized":"2024-06-29T14:35:06.154161+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:37:49.909672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135183+0000","last_change":"2024-06-29T14:35:06.135183+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135183+0000","last_undegraded":"2024-06-29T14:35:06.135183+0000","last_fullsized":"2024-06-29T14:35:06.135183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153790+0000","last_change":"2024-06-29T14:35:06.148884+0000","last_active":"2024-06-29T14:35:06.153790+0000","last_peered":"2024-06-29T14:35:06.153790+0000","last_clean":"2024-06-29T14:35:06.153790+0000","last_became_active":"2024-06-29T14:35:06.148389+0000","last_became_peered":"2024-06-29T14:35:06.148389+0000","last_unstale":"2024-06-29T14:35:06.153790+0000","last_undegraded":"2024-06-29T14:35:06.153790+0000","last_fullsized":"2024-06-29T14:35:06.153790+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:08.291163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139396+0000","last_change":"2024-06-29T14:35:06.139396+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139396+0000","last_undegraded":"2024-06-29T14:35:06.139396+0000","last_fullsized":"2024-06-29T14:35:06.139396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154578+0000","last_change":"2024-06-29T14:35:06.152507+0000","last_active":"2024-06-29T14:35:06.154578+0000","last_peered":"2024-06-29T14:35:06.154578+0000","last_clean":"2024-06-29T14:35:06.154578+0000","last_became_active":"2024-06-29T14:35:06.152204+0000","last_became_peered":"2024-06-29T14:35:06.152204+0000","last_unstale":"2024-06-29T14:35:06.154578+0000","last_undegraded":"2024-06-29T14:35:06.154578+0000","last_fullsized":"2024-06-29T14:35:06.154578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:07.902240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136887+0000","last_change":"2024-06-29T14:35:06.136887+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136887+0000","last_undegraded":"2024-06-29T14:35:06.136887+0000","last_fullsized":"2024-06-29T14:35:06.136887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144393+0000","last_change":"2024-06-29T14:35:06.141470+0000","last_active":"2024-06-29T14:35:06.144393+0000","last_peered":"2024-06-29T14:35:06.144393+0000","last_clean":"2024-06-29T14:35:06.144393+0000","last_became_active":"2024-06-29T14:35:06.141175+0000","last_became_peered":"2024-06-29T14:35:06.141175+0000","last_unstale":"2024-06-29T14:35:06.144393+0000","last_undegraded":"2024-06-29T14:35:06.144393+0000","last_fullsized":"2024-06-29T14:35:06.144393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:54.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.142459+0000","last_change":"2024-06-29T14:35:06.142459+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.142459+0000","last_undegraded":"2024-06-29T14:35:06.142459+0000","last_fullsized":"2024-06-29T14:35:06.142459+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153326+0000","last_change":"2024-06-29T14:35:06.152825+0000","last_active":"2024-06-29T14:35:06.153326+0000","last_peered":"2024-06-29T14:35:06.153326+0000","last_clean":"2024-06-29T14:35:06.153326+0000","last_became_active":"2024-06-29T14:35:06.152469+0000","last_became_peered":"2024-06-29T14:35:06.152469+0000","last_unstale":"2024-06-29T14:35:06.153326+0000","last_undegraded":"2024-06-29T14:35:06.153326+0000","last_fullsized":"2024-06-29T14:35:06.153326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:39.872319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134310+0000","last_change":"2024-06-29T14:35:06.134310+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134310+0000","last_undegraded":"2024-06-29T14:35:06.134310+0000","last_fullsized":"2024-06-29T14:35:06.134310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153817+0000","last_change":"2024-06-29T14:35:06.148661+0000","last_active":"2024-06-29T14:35:06.153817+0000","last_peered":"2024-06-29T14:35:06.153817+0000","last_clean":"2024-06-29T14:35:06.153817+0000","last_became_active":"2024-06-29T14:35:06.148307+0000","last_became_peered":"2024-06-29T14:35:06.148307+0000","last_unstale":"2024-06-29T14:35:06.153817+0000","last_undegraded":"2024-06-29T14:35:06.153817+0000","last_fullsized":"2024-06-29T14:35:06.153817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:46.690921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'86","reported_seq":257,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.140303+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:35:06.140303+0000","last_peered":"2024-06-29T14:35:06.140303+0000","last_clean":"2024-06-29T14:35:06.140303+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:35:06.140303+0000","last_undegraded":"2024-06-29T14:35:06.140303+0000","last_fullsized":"2024-06-29T14:35:06.140303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54768,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317072,"statfs":{"total":96636764160,"available":96580681728,"internally_reserved":0,"allocated":1413120,"data_stored":997410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542308,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50028,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321812,"statfs":{"total":96636764160,"available":96585535488,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542306,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509604,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575012,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58348,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313492,"statfs":{"total":96636764160,"available":96577015808,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509603,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1413120,"data_stored":997410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:35:11.233 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:35:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.390+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2374147444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 msgr2=0x7f0f8c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.390+0000 7f0fa93e0640 1 --2- 172.21.15.59:0/2374147444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 0x7f0f8c004800 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0f98001830 tx=0x7f0f8c005830 comp rx=0 tx=0).stop 2024-06-29T14:35:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.390+0000 7f0fa37fe640 1 -- 172.21.15.59:0/2374147444 reap_dead start 2024-06-29T14:35:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.390+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2374147444 shutdown_connections 2024-06-29T14:35:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.390+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2374147444 >> 172.21.15.59:0/2374147444 conn(0x7f0fa40b5610 msgr2=0x7f0fa40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.390+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2374147444 shutdown_connections 2024-06-29T14:35:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.390+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2374147444 wait complete. 2024-06-29T14:35:11.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.392+0000 7f0fa93e0640 1 Processor -- start 2024-06-29T14:35:11.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.392+0000 7f0fa93e0640 1 -- start start 2024-06-29T14:35:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa93e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 0x7f0fa4149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa93e0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa41f8550 0x7f0fa41f8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa93e0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa406cec0 0x7f0fa406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa93e0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa40cf080 con 0x7f0f8c004230 2024-06-29T14:35:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa93e0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa40b8f00 con 0x7f0fa41f8550 2024-06-29T14:35:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa93e0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa40cf200 con 0x7f0fa406cec0 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 0x7f0fa4149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 0x7f0fa4149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37898/0 (socket says 172.21.15.59:37898) 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 -- 172.21.15.59:0/2732564390 learned_addr learned my addr 172.21.15.59:0/2732564390 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa37fe640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa406cec0 0x7f0fa406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa27fc640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa41f8550 0x7f0fa41f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 -- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa406cec0 msgr2=0x7f0fa406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa406cec0 0x7f0fa406d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 -- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa41f8550 msgr2=0x7f0fa41f8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa41f8550 0x7f0fa41f8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 -- 172.21.15.59:0/2732564390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa406d810 con 0x7f0f8c004230 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa27fc640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa41f8550 0x7f0fa41f8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa2ffd640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 0x7f0fa4149c40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0f94011ae0 tx=0x7f0f94009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0f8b7fe640 1 -- 172.21.15.59:0/2732564390 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94004460 con 0x7f0f8c004230 2024-06-29T14:35:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0f8b7fe640 1 -- 172.21.15.59:0/2732564390 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f9400fa00 con 0x7f0f8c004230 2024-06-29T14:35:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa41f92c0 con 0x7f0f8c004230 2024-06-29T14:35:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.394+0000 7f0f8b7fe640 1 -- 172.21.15.59:0/2732564390 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9400fe50 con 0x7f0f8c004230 2024-06-29T14:35:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.395+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa41f5500 con 0x7f0f8c004230 2024-06-29T14:35:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.396+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fa40d1a20 con 0x7f0f8c004230 2024-06-29T14:35:11.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.396+0000 7f0f8b7fe640 1 -- 172.21.15.59:0/2732564390 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f0f940035f0 con 0x7f0f8c004230 2024-06-29T14:35:11.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.397+0000 7f0f8b7fe640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f90076420 0x7f0f90078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.397+0000 7f0f8b7fe640 1 -- 172.21.15.59:0/2732564390 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f0f94098b80 con 0x7f0f8c004230 2024-06-29T14:35:11.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.397+0000 7f0fa27fc640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f90076420 0x7f0f90078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.398+0000 7f0fa27fc640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f90076420 0x7f0f90078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f98032000 tx=0x7f0f98001360 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:11.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.401+0000 7f0f8b7fe640 1 -- 172.21.15.59:0/2732564390 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0fa406d810 con 0x7f0f8c004230 2024-06-29T14:35:11.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.588+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 --> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0fa406c710 con 0x7f0f90076420 2024-06-29T14:35:11.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.593+0000 7f0f8b7fe640 1 -- 172.21.15.59:0/2732564390 <== mgr.5809 v2:172.21.15.153:6832/2957341575 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246630 (secure 0 0 0) 0x7f0fa406c710 con 0x7f0f90076420 2024-06-29T14:35:11.593 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:11.594 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:35:11.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f90076420 msgr2=0x7f0f90078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f90076420 0x7f0f90078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f98032000 tx=0x7f0f98001360 comp rx=0 tx=0).stop 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 msgr2=0x7f0fa4149c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 0x7f0fa4149c40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0f94011ae0 tx=0x7f0f94009e90 comp rx=0 tx=0).stop 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 shutdown_connections 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f0f90076420 0x7f0f90078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fa406cec0 0x7f0fa406d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fa41f8550 0x7f0fa41f8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 --2- 172.21.15.59:0/2732564390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f8c004230 0x7f0fa4149c40 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 >> 172.21.15.59:0/2732564390 conn(0x7f0fa40b5610 msgr2=0x7f0fa40b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 shutdown_connections 2024-06-29T14:35:11.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.596+0000 7f0fa93e0640 1 -- 172.21.15.59:0/2732564390 wait complete. 2024-06-29T14:35:11.604 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-06-29T14:35:10.033416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323648,"kb_used_data":7560,"kb_used_omap":12,"kb_used_meta":315955,"kb_avail":754651072,"statfs":{"total":773094113280,"available":772762697728,"internally_reserved":0,"allocated":7741440,"data_stored":4437072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323538482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.908141"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151105+0000","last_change":"2024-06-29T14:35:06.151105+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151105+0000","last_undegraded":"2024-06-29T14:35:06.151105+0000","last_fullsized":"2024-06-29T14:35:06.151105+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138965+0000","last_change":"2024-06-29T14:35:06.138965+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138965+0000","last_undegraded":"2024-06-29T14:35:06.138965+0000","last_fullsized":"2024-06-29T14:35:06.138965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138100+0000","last_change":"2024-06-29T14:35:06.138100+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138100+0000","last_undegraded":"2024-06-29T14:35:06.138100+0000","last_fullsized":"2024-06-29T14:35:06.138100+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138146+0000","last_change":"2024-06-29T14:35:06.138146+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138146+0000","last_undegraded":"2024-06-29T14:35:06.138146+0000","last_fullsized":"2024-06-29T14:35:06.138146+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132969+0000","last_change":"2024-06-29T14:35:06.132969+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132969+0000","last_undegraded":"2024-06-29T14:35:06.132969+0000","last_fullsized":"2024-06-29T14:35:06.132969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149626+0000","last_change":"2024-06-29T14:35:06.149626+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149626+0000","last_undegraded":"2024-06-29T14:35:06.149626+0000","last_fullsized":"2024-06-29T14:35:06.149626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135540+0000","last_change":"2024-06-29T14:35:06.135540+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135540+0000","last_undegraded":"2024-06-29T14:35:06.135540+0000","last_fullsized":"2024-06-29T14:35:06.135540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.148127+0000","last_change":"2024-06-29T14:35:06.148127+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.148127+0000","last_undegraded":"2024-06-29T14:35:06.148127+0000","last_fullsized":"2024-06-29T14:35:06.148127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.147564+0000","last_change":"2024-06-29T14:35:06.147564+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.147564+0000","last_undegraded":"2024-06-29T14:35:06.147564+0000","last_fullsized":"2024-06-29T14:35:06.147564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151644+0000","last_change":"2024-06-29T14:35:06.151644+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151644+0000","last_undegraded":"2024-06-29T14:35:06.151644+0000","last_fullsized":"2024-06-29T14:35:06.151644+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.144328+0000","last_change":"2024-06-29T14:35:06.144328+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.144328+0000","last_undegraded":"2024-06-29T14:35:06.144328+0000","last_fullsized":"2024-06-29T14:35:06.144328+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146980+0000","last_change":"2024-06-29T14:35:06.146980+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146980+0000","last_undegraded":"2024-06-29T14:35:06.146980+0000","last_fullsized":"2024-06-29T14:35:06.146980+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151643+0000","last_change":"2024-06-29T14:35:06.151643+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151643+0000","last_undegraded":"2024-06-29T14:35:06.151643+0000","last_fullsized":"2024-06-29T14:35:06.151643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136411+0000","last_change":"2024-06-29T14:35:06.136411+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136411+0000","last_undegraded":"2024-06-29T14:35:06.136411+0000","last_fullsized":"2024-06-29T14:35:06.136411+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137811+0000","last_change":"2024-06-29T14:35:06.137811+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137811+0000","last_undegraded":"2024-06-29T14:35:06.137811+0000","last_fullsized":"2024-06-29T14:35:06.137811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146620+0000","last_change":"2024-06-29T14:35:06.146620+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146620+0000","last_undegraded":"2024-06-29T14:35:06.146620+0000","last_fullsized":"2024-06-29T14:35:06.146620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137727+0000","last_change":"2024-06-29T14:35:06.137727+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137727+0000","last_undegraded":"2024-06-29T14:35:06.137727+0000","last_fullsized":"2024-06-29T14:35:06.137727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138015+0000","last_change":"2024-06-29T14:35:06.138015+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138015+0000","last_undegraded":"2024-06-29T14:35:06.138015+0000","last_fullsized":"2024-06-29T14:35:06.138015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137633+0000","last_change":"2024-06-29T14:35:06.137633+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137633+0000","last_undegraded":"2024-06-29T14:35:06.137633+0000","last_fullsized":"2024-06-29T14:35:06.137633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135000+0000","last_change":"2024-06-29T14:35:06.135000+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135000+0000","last_undegraded":"2024-06-29T14:35:06.135000+0000","last_fullsized":"2024-06-29T14:35:06.135000+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.133885+0000","last_change":"2024-06-29T14:35:06.133885+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.133885+0000","last_undegraded":"2024-06-29T14:35:06.133885+0000","last_fullsized":"2024-06-29T14:35:06.133885+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146360+0000","last_change":"2024-06-29T14:35:06.146360+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146360+0000","last_undegraded":"2024-06-29T14:35:06.146360+0000","last_fullsized":"2024-06-29T14:35:06.146360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141017+0000","last_change":"2024-06-29T14:35:06.141017+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141017+0000","last_undegraded":"2024-06-29T14:35:06.141017+0000","last_fullsized":"2024-06-29T14:35:06.141017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138890+0000","last_change":"2024-06-29T14:35:06.138890+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138890+0000","last_undegraded":"2024-06-29T14:35:06.138890+0000","last_fullsized":"2024-06-29T14:35:06.138890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137237+0000","last_change":"2024-06-29T14:35:06.137237+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137237+0000","last_undegraded":"2024-06-29T14:35:06.137237+0000","last_fullsized":"2024-06-29T14:35:06.137237+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.145779+0000","last_change":"2024-06-29T14:35:06.145779+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.145779+0000","last_undegraded":"2024-06-29T14:35:06.145779+0000","last_fullsized":"2024-06-29T14:35:06.145779+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.155904+0000","last_change":"2024-06-29T14:35:06.155904+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.155904+0000","last_undegraded":"2024-06-29T14:35:06.155904+0000","last_fullsized":"2024-06-29T14:35:06.155904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134043+0000","last_change":"2024-06-29T14:35:06.134043+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134043+0000","last_undegraded":"2024-06-29T14:35:06.134043+0000","last_fullsized":"2024-06-29T14:35:06.134043+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137017+0000","last_change":"2024-06-29T14:35:06.137017+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137017+0000","last_undegraded":"2024-06-29T14:35:06.137017+0000","last_fullsized":"2024-06-29T14:35:06.137017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136330+0000","last_change":"2024-06-29T14:35:06.136330+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136330+0000","last_undegraded":"2024-06-29T14:35:06.136330+0000","last_fullsized":"2024-06-29T14:35:06.136330+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134973+0000","last_change":"2024-06-29T14:35:06.134973+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134973+0000","last_undegraded":"2024-06-29T14:35:06.134973+0000","last_fullsized":"2024-06-29T14:35:06.134973+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139697+0000","last_change":"2024-06-29T14:35:06.139697+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139697+0000","last_undegraded":"2024-06-29T14:35:06.139697+0000","last_fullsized":"2024-06-29T14:35:06.139697+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137035+0000","last_change":"2024-06-29T14:35:06.137035+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137035+0000","last_undegraded":"2024-06-29T14:35:06.137035+0000","last_fullsized":"2024-06-29T14:35:06.137035+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.141964+0000","last_change":"2024-06-29T14:35:06.139267+0000","last_active":"2024-06-29T14:35:06.141964+0000","last_peered":"2024-06-29T14:35:06.141964+0000","last_clean":"2024-06-29T14:35:06.141964+0000","last_became_active":"2024-06-29T14:35:06.138932+0000","last_became_peered":"2024-06-29T14:35:06.138932+0000","last_unstale":"2024-06-29T14:35:06.141964+0000","last_undegraded":"2024-06-29T14:35:06.141964+0000","last_fullsized":"2024-06-29T14:35:06.141964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:47.855244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136457+0000","last_change":"2024-06-29T14:35:06.136457+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136457+0000","last_undegraded":"2024-06-29T14:35:06.136457+0000","last_fullsized":"2024-06-29T14:35:06.136457+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.158642+0000","last_change":"2024-06-29T14:35:06.157899+0000","last_active":"2024-06-29T14:35:06.158642+0000","last_peered":"2024-06-29T14:35:06.158642+0000","last_clean":"2024-06-29T14:35:06.158642+0000","last_became_active":"2024-06-29T14:35:06.157591+0000","last_became_peered":"2024-06-29T14:35:06.157591+0000","last_unstale":"2024-06-29T14:35:06.158642+0000","last_undegraded":"2024-06-29T14:35:06.158642+0000","last_fullsized":"2024-06-29T14:35:06.158642+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:04:01.035476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134883+0000","last_change":"2024-06-29T14:35:06.134883+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134883+0000","last_undegraded":"2024-06-29T14:35:06.134883+0000","last_fullsized":"2024-06-29T14:35:06.134883+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.162625+0000","last_change":"2024-06-29T14:35:06.160847+0000","last_active":"2024-06-29T14:35:06.162625+0000","last_peered":"2024-06-29T14:35:06.162625+0000","last_clean":"2024-06-29T14:35:06.162625+0000","last_became_active":"2024-06-29T14:35:06.160558+0000","last_became_peered":"2024-06-29T14:35:06.160558+0000","last_unstale":"2024-06-29T14:35:06.162625+0000","last_undegraded":"2024-06-29T14:35:06.162625+0000","last_fullsized":"2024-06-29T14:35:06.162625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:29.140867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.142843+0000","last_change":"2024-06-29T14:35:06.142843+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.142843+0000","last_undegraded":"2024-06-29T14:35:06.142843+0000","last_fullsized":"2024-06-29T14:35:06.142843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.147719+0000","last_change":"2024-06-29T14:35:06.147172+0000","last_active":"2024-06-29T14:35:06.147719+0000","last_peered":"2024-06-29T14:35:06.147719+0000","last_clean":"2024-06-29T14:35:06.147719+0000","last_became_active":"2024-06-29T14:35:06.146877+0000","last_became_peered":"2024-06-29T14:35:06.146877+0000","last_unstale":"2024-06-29T14:35:06.147719+0000","last_undegraded":"2024-06-29T14:35:06.147719+0000","last_fullsized":"2024-06-29T14:35:06.147719+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:11:45.335558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132273+0000","last_change":"2024-06-29T14:35:06.132273+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132273+0000","last_undegraded":"2024-06-29T14:35:06.132273+0000","last_fullsized":"2024-06-29T14:35:06.132273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.155462+0000","last_change":"2024-06-29T14:35:06.154311+0000","last_active":"2024-06-29T14:35:06.155462+0000","last_peered":"2024-06-29T14:35:06.155462+0000","last_clean":"2024-06-29T14:35:06.155462+0000","last_became_active":"2024-06-29T14:35:06.153161+0000","last_became_peered":"2024-06-29T14:35:06.153161+0000","last_unstale":"2024-06-29T14:35:06.155462+0000","last_undegraded":"2024-06-29T14:35:06.155462+0000","last_fullsized":"2024-06-29T14:35:06.155462+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:03.504751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138048+0000","last_change":"2024-06-29T14:35:06.138048+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138048+0000","last_undegraded":"2024-06-29T14:35:06.138048+0000","last_fullsized":"2024-06-29T14:35:06.138048+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144211+0000","last_change":"2024-06-29T14:35:06.142746+0000","last_active":"2024-06-29T14:35:06.144211+0000","last_peered":"2024-06-29T14:35:06.144211+0000","last_clean":"2024-06-29T14:35:06.144211+0000","last_became_active":"2024-06-29T14:35:06.142398+0000","last_became_peered":"2024-06-29T14:35:06.142398+0000","last_unstale":"2024-06-29T14:35:06.144211+0000","last_undegraded":"2024-06-29T14:35:06.144211+0000","last_fullsized":"2024-06-29T14:35:06.144211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:52:09.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.143419+0000","last_change":"2024-06-29T14:35:06.143419+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.143419+0000","last_undegraded":"2024-06-29T14:35:06.143419+0000","last_fullsized":"2024-06-29T14:35:06.143419+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.142739+0000","last_change":"2024-06-29T14:35:06.141432+0000","last_active":"2024-06-29T14:35:06.142739+0000","last_peered":"2024-06-29T14:35:06.142739+0000","last_clean":"2024-06-29T14:35:06.142739+0000","last_became_active":"2024-06-29T14:35:06.141079+0000","last_became_peered":"2024-06-29T14:35:06.141079+0000","last_unstale":"2024-06-29T14:35:06.142739+0000","last_undegraded":"2024-06-29T14:35:06.142739+0000","last_fullsized":"2024-06-29T14:35:06.142739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:28.307716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136427+0000","last_change":"2024-06-29T14:35:06.136427+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136427+0000","last_undegraded":"2024-06-29T14:35:06.136427+0000","last_fullsized":"2024-06-29T14:35:06.136427+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154777+0000","last_change":"2024-06-29T14:35:06.153920+0000","last_active":"2024-06-29T14:35:06.154777+0000","last_peered":"2024-06-29T14:35:06.154777+0000","last_clean":"2024-06-29T14:35:06.154777+0000","last_became_active":"2024-06-29T14:35:06.153047+0000","last_became_peered":"2024-06-29T14:35:06.153047+0000","last_unstale":"2024-06-29T14:35:06.154777+0000","last_undegraded":"2024-06-29T14:35:06.154777+0000","last_fullsized":"2024-06-29T14:35:06.154777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:42.614890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132166+0000","last_change":"2024-06-29T14:35:06.132166+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132166+0000","last_undegraded":"2024-06-29T14:35:06.132166+0000","last_fullsized":"2024-06-29T14:35:06.132166+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.146782+0000","last_change":"2024-06-29T14:35:06.145945+0000","last_active":"2024-06-29T14:35:06.146782+0000","last_peered":"2024-06-29T14:35:06.146782+0000","last_clean":"2024-06-29T14:35:06.146782+0000","last_became_active":"2024-06-29T14:35:06.145555+0000","last_became_peered":"2024-06-29T14:35:06.145555+0000","last_unstale":"2024-06-29T14:35:06.146782+0000","last_undegraded":"2024-06-29T14:35:06.146782+0000","last_fullsized":"2024-06-29T14:35:06.146782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:53.987132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136980+0000","last_change":"2024-06-29T14:35:06.136980+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136980+0000","last_undegraded":"2024-06-29T14:35:06.136980+0000","last_fullsized":"2024-06-29T14:35:06.136980+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156774+0000","last_change":"2024-06-29T14:35:06.156160+0000","last_active":"2024-06-29T14:35:06.156774+0000","last_peered":"2024-06-29T14:35:06.156774+0000","last_clean":"2024-06-29T14:35:06.156774+0000","last_became_active":"2024-06-29T14:35:06.155823+0000","last_became_peered":"2024-06-29T14:35:06.155823+0000","last_unstale":"2024-06-29T14:35:06.156774+0000","last_undegraded":"2024-06-29T14:35:06.156774+0000","last_fullsized":"2024-06-29T14:35:06.156774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:00.015532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135625+0000","last_change":"2024-06-29T14:35:06.135625+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135625+0000","last_undegraded":"2024-06-29T14:35:06.135625+0000","last_fullsized":"2024-06-29T14:35:06.135625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.159311+0000","last_change":"2024-06-29T14:35:06.158866+0000","last_active":"2024-06-29T14:35:06.159311+0000","last_peered":"2024-06-29T14:35:06.159311+0000","last_clean":"2024-06-29T14:35:06.159311+0000","last_became_active":"2024-06-29T14:35:06.158615+0000","last_became_peered":"2024-06-29T14:35:06.158615+0000","last_unstale":"2024-06-29T14:35:06.159311+0000","last_undegraded":"2024-06-29T14:35:06.159311+0000","last_fullsized":"2024-06-29T14:35:06.159311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:34.626783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132361+0000","last_change":"2024-06-29T14:35:06.132361+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132361+0000","last_undegraded":"2024-06-29T14:35:06.132361+0000","last_fullsized":"2024-06-29T14:35:06.132361+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153751+0000","last_change":"2024-06-29T14:35:06.148671+0000","last_active":"2024-06-29T14:35:06.153751+0000","last_peered":"2024-06-29T14:35:06.153751+0000","last_clean":"2024-06-29T14:35:06.153751+0000","last_became_active":"2024-06-29T14:35:06.148322+0000","last_became_peered":"2024-06-29T14:35:06.148322+0000","last_unstale":"2024-06-29T14:35:06.153751+0000","last_undegraded":"2024-06-29T14:35:06.153751+0000","last_fullsized":"2024-06-29T14:35:06.153751+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:52.995413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139128+0000","last_change":"2024-06-29T14:35:06.139128+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139128+0000","last_undegraded":"2024-06-29T14:35:06.139128+0000","last_fullsized":"2024-06-29T14:35:06.139128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153807+0000","last_change":"2024-06-29T14:35:06.148685+0000","last_active":"2024-06-29T14:35:06.153807+0000","last_peered":"2024-06-29T14:35:06.153807+0000","last_clean":"2024-06-29T14:35:06.153807+0000","last_became_active":"2024-06-29T14:35:06.148343+0000","last_became_peered":"2024-06-29T14:35:06.148343+0000","last_unstale":"2024-06-29T14:35:06.153807+0000","last_undegraded":"2024-06-29T14:35:06.153807+0000","last_fullsized":"2024-06-29T14:35:06.153807+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:30.741869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137093+0000","last_change":"2024-06-29T14:35:06.137093+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137093+0000","last_undegraded":"2024-06-29T14:35:06.137093+0000","last_fullsized":"2024-06-29T14:35:06.137093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.146783+0000","last_change":"2024-06-29T14:35:06.146043+0000","last_active":"2024-06-29T14:35:06.146783+0000","last_peered":"2024-06-29T14:35:06.146783+0000","last_clean":"2024-06-29T14:35:06.146783+0000","last_became_active":"2024-06-29T14:35:06.145648+0000","last_became_peered":"2024-06-29T14:35:06.145648+0000","last_unstale":"2024-06-29T14:35:06.146783+0000","last_undegraded":"2024-06-29T14:35:06.146783+0000","last_fullsized":"2024-06-29T14:35:06.146783+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:23.312796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.143970+0000","last_change":"2024-06-29T14:35:06.143970+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.143970+0000","last_undegraded":"2024-06-29T14:35:06.143970+0000","last_fullsized":"2024-06-29T14:35:06.143970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144837+0000","last_change":"2024-06-29T14:35:06.144003+0000","last_active":"2024-06-29T14:35:06.144837+0000","last_peered":"2024-06-29T14:35:06.144837+0000","last_clean":"2024-06-29T14:35:06.144837+0000","last_became_active":"2024-06-29T14:35:06.143542+0000","last_became_peered":"2024-06-29T14:35:06.143542+0000","last_unstale":"2024-06-29T14:35:06.144837+0000","last_undegraded":"2024-06-29T14:35:06.144837+0000","last_fullsized":"2024-06-29T14:35:06.144837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:20.980556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138217+0000","last_change":"2024-06-29T14:35:06.138217+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138217+0000","last_undegraded":"2024-06-29T14:35:06.138217+0000","last_fullsized":"2024-06-29T14:35:06.138217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.145401+0000","last_change":"2024-06-29T14:35:06.143641+0000","last_active":"2024-06-29T14:35:06.145401+0000","last_peered":"2024-06-29T14:35:06.145401+0000","last_clean":"2024-06-29T14:35:06.145401+0000","last_became_active":"2024-06-29T14:35:06.143030+0000","last_became_peered":"2024-06-29T14:35:06.143030+0000","last_unstale":"2024-06-29T14:35:06.145401+0000","last_undegraded":"2024-06-29T14:35:06.145401+0000","last_fullsized":"2024-06-29T14:35:06.145401+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:21:59.463680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141803+0000","last_change":"2024-06-29T14:35:06.141803+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141803+0000","last_undegraded":"2024-06-29T14:35:06.141803+0000","last_fullsized":"2024-06-29T14:35:06.141803+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156969+0000","last_change":"2024-06-29T14:35:06.155992+0000","last_active":"2024-06-29T14:35:06.156969+0000","last_peered":"2024-06-29T14:35:06.156969+0000","last_clean":"2024-06-29T14:35:06.156969+0000","last_became_active":"2024-06-29T14:35:06.155609+0000","last_became_peered":"2024-06-29T14:35:06.155609+0000","last_unstale":"2024-06-29T14:35:06.156969+0000","last_undegraded":"2024-06-29T14:35:06.156969+0000","last_fullsized":"2024-06-29T14:35:06.156969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:59.794303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.147888+0000","last_change":"2024-06-29T14:35:06.147888+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.147888+0000","last_undegraded":"2024-06-29T14:35:06.147888+0000","last_fullsized":"2024-06-29T14:35:06.147888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.150684+0000","last_change":"2024-06-29T14:35:06.145742+0000","last_active":"2024-06-29T14:35:06.150684+0000","last_peered":"2024-06-29T14:35:06.150684+0000","last_clean":"2024-06-29T14:35:06.150684+0000","last_became_active":"2024-06-29T14:35:06.145327+0000","last_became_peered":"2024-06-29T14:35:06.145327+0000","last_unstale":"2024-06-29T14:35:06.150684+0000","last_undegraded":"2024-06-29T14:35:06.150684+0000","last_fullsized":"2024-06-29T14:35:06.150684+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:40:50.966470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132769+0000","last_change":"2024-06-29T14:35:06.132769+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132769+0000","last_undegraded":"2024-06-29T14:35:06.132769+0000","last_fullsized":"2024-06-29T14:35:06.132769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.160532+0000","last_change":"2024-06-29T14:35:06.159992+0000","last_active":"2024-06-29T14:35:06.160532+0000","last_peered":"2024-06-29T14:35:06.160532+0000","last_clean":"2024-06-29T14:35:06.160532+0000","last_became_active":"2024-06-29T14:35:06.159710+0000","last_became_peered":"2024-06-29T14:35:06.159710+0000","last_unstale":"2024-06-29T14:35:06.160532+0000","last_undegraded":"2024-06-29T14:35:06.160532+0000","last_fullsized":"2024-06-29T14:35:06.160532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:47.483637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149852+0000","last_change":"2024-06-29T14:35:06.149852+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149852+0000","last_undegraded":"2024-06-29T14:35:06.149852+0000","last_fullsized":"2024-06-29T14:35:06.149852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.149922+0000","last_change":"2024-06-29T14:35:06.149423+0000","last_active":"2024-06-29T14:35:06.149922+0000","last_peered":"2024-06-29T14:35:06.149922+0000","last_clean":"2024-06-29T14:35:06.149922+0000","last_became_active":"2024-06-29T14:35:06.149111+0000","last_became_peered":"2024-06-29T14:35:06.149111+0000","last_unstale":"2024-06-29T14:35:06.149922+0000","last_undegraded":"2024-06-29T14:35:06.149922+0000","last_fullsized":"2024-06-29T14:35:06.149922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:29.397336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138649+0000","last_change":"2024-06-29T14:35:06.138649+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138649+0000","last_undegraded":"2024-06-29T14:35:06.138649+0000","last_fullsized":"2024-06-29T14:35:06.138649+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.159867+0000","last_change":"2024-06-29T14:35:06.159208+0000","last_active":"2024-06-29T14:35:06.159867+0000","last_peered":"2024-06-29T14:35:06.159867+0000","last_clean":"2024-06-29T14:35:06.159867+0000","last_became_active":"2024-06-29T14:35:06.158901+0000","last_became_peered":"2024-06-29T14:35:06.158901+0000","last_unstale":"2024-06-29T14:35:06.159867+0000","last_undegraded":"2024-06-29T14:35:06.159867+0000","last_fullsized":"2024-06-29T14:35:06.159867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:41:37.874410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.133545+0000","last_change":"2024-06-29T14:35:06.133545+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.133545+0000","last_undegraded":"2024-06-29T14:35:06.133545+0000","last_fullsized":"2024-06-29T14:35:06.133545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.145350+0000","last_change":"2024-06-29T14:35:06.142795+0000","last_active":"2024-06-29T14:35:06.145350+0000","last_peered":"2024-06-29T14:35:06.145350+0000","last_clean":"2024-06-29T14:35:06.145350+0000","last_became_active":"2024-06-29T14:35:06.142463+0000","last_became_peered":"2024-06-29T14:35:06.142463+0000","last_unstale":"2024-06-29T14:35:06.145350+0000","last_undegraded":"2024-06-29T14:35:06.145350+0000","last_fullsized":"2024-06-29T14:35:06.145350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:13.219465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136269+0000","last_change":"2024-06-29T14:35:06.136269+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136269+0000","last_undegraded":"2024-06-29T14:35:06.136269+0000","last_fullsized":"2024-06-29T14:35:06.136269+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156310+0000","last_change":"2024-06-29T14:35:06.154839+0000","last_active":"2024-06-29T14:35:06.156310+0000","last_peered":"2024-06-29T14:35:06.156310+0000","last_clean":"2024-06-29T14:35:06.156310+0000","last_became_active":"2024-06-29T14:35:06.154513+0000","last_became_peered":"2024-06-29T14:35:06.154513+0000","last_unstale":"2024-06-29T14:35:06.156310+0000","last_undegraded":"2024-06-29T14:35:06.156310+0000","last_fullsized":"2024-06-29T14:35:06.156310+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:13.939011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.150280+0000","last_change":"2024-06-29T14:35:06.150280+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.150280+0000","last_undegraded":"2024-06-29T14:35:06.150280+0000","last_fullsized":"2024-06-29T14:35:06.150280+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144749+0000","last_change":"2024-06-29T14:35:06.143667+0000","last_active":"2024-06-29T14:35:06.144749+0000","last_peered":"2024-06-29T14:35:06.144749+0000","last_clean":"2024-06-29T14:35:06.144749+0000","last_became_active":"2024-06-29T14:35:06.143312+0000","last_became_peered":"2024-06-29T14:35:06.143312+0000","last_unstale":"2024-06-29T14:35:06.144749+0000","last_undegraded":"2024-06-29T14:35:06.144749+0000","last_fullsized":"2024-06-29T14:35:06.144749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:00.265866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139175+0000","last_change":"2024-06-29T14:35:06.139175+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139175+0000","last_undegraded":"2024-06-29T14:35:06.139175+0000","last_fullsized":"2024-06-29T14:35:06.139175+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153279+0000","last_change":"2024-06-29T14:35:06.152663+0000","last_active":"2024-06-29T14:35:06.153279+0000","last_peered":"2024-06-29T14:35:06.153279+0000","last_clean":"2024-06-29T14:35:06.153279+0000","last_became_active":"2024-06-29T14:35:06.152267+0000","last_became_peered":"2024-06-29T14:35:06.152267+0000","last_unstale":"2024-06-29T14:35:06.153279+0000","last_undegraded":"2024-06-29T14:35:06.153279+0000","last_fullsized":"2024-06-29T14:35:06.153279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:15.826985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149080+0000","last_change":"2024-06-29T14:35:06.149080+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149080+0000","last_undegraded":"2024-06-29T14:35:06.149080+0000","last_fullsized":"2024-06-29T14:35:06.149080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.155693+0000","last_change":"2024-06-29T14:35:06.155153+0000","last_active":"2024-06-29T14:35:06.155693+0000","last_peered":"2024-06-29T14:35:06.155693+0000","last_clean":"2024-06-29T14:35:06.155693+0000","last_became_active":"2024-06-29T14:35:06.154919+0000","last_became_peered":"2024-06-29T14:35:06.154919+0000","last_unstale":"2024-06-29T14:35:06.155693+0000","last_undegraded":"2024-06-29T14:35:06.155693+0000","last_fullsized":"2024-06-29T14:35:06.155693+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:21.593203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141904+0000","last_change":"2024-06-29T14:35:06.141904+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141904+0000","last_undegraded":"2024-06-29T14:35:06.141904+0000","last_fullsized":"2024-06-29T14:35:06.141904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154161+0000","last_change":"2024-06-29T14:35:06.149483+0000","last_active":"2024-06-29T14:35:06.154161+0000","last_peered":"2024-06-29T14:35:06.154161+0000","last_clean":"2024-06-29T14:35:06.154161+0000","last_became_active":"2024-06-29T14:35:06.149163+0000","last_became_peered":"2024-06-29T14:35:06.149163+0000","last_unstale":"2024-06-29T14:35:06.154161+0000","last_undegraded":"2024-06-29T14:35:06.154161+0000","last_fullsized":"2024-06-29T14:35:06.154161+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:37:49.909672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135183+0000","last_change":"2024-06-29T14:35:06.135183+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135183+0000","last_undegraded":"2024-06-29T14:35:06.135183+0000","last_fullsized":"2024-06-29T14:35:06.135183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153790+0000","last_change":"2024-06-29T14:35:06.148884+0000","last_active":"2024-06-29T14:35:06.153790+0000","last_peered":"2024-06-29T14:35:06.153790+0000","last_clean":"2024-06-29T14:35:06.153790+0000","last_became_active":"2024-06-29T14:35:06.148389+0000","last_became_peered":"2024-06-29T14:35:06.148389+0000","last_unstale":"2024-06-29T14:35:06.153790+0000","last_undegraded":"2024-06-29T14:35:06.153790+0000","last_fullsized":"2024-06-29T14:35:06.153790+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:08.291163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139396+0000","last_change":"2024-06-29T14:35:06.139396+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139396+0000","last_undegraded":"2024-06-29T14:35:06.139396+0000","last_fullsized":"2024-06-29T14:35:06.139396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154578+0000","last_change":"2024-06-29T14:35:06.152507+0000","last_active":"2024-06-29T14:35:06.154578+0000","last_peered":"2024-06-29T14:35:06.154578+0000","last_clean":"2024-06-29T14:35:06.154578+0000","last_became_active":"2024-06-29T14:35:06.152204+0000","last_became_peered":"2024-06-29T14:35:06.152204+0000","last_unstale":"2024-06-29T14:35:06.154578+0000","last_undegraded":"2024-06-29T14:35:06.154578+0000","last_fullsized":"2024-06-29T14:35:06.154578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:07.902240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136887+0000","last_change":"2024-06-29T14:35:06.136887+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136887+0000","last_undegraded":"2024-06-29T14:35:06.136887+0000","last_fullsized":"2024-06-29T14:35:06.136887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144393+0000","last_change":"2024-06-29T14:35:06.141470+0000","last_active":"2024-06-29T14:35:06.144393+0000","last_peered":"2024-06-29T14:35:06.144393+0000","last_clean":"2024-06-29T14:35:06.144393+0000","last_became_active":"2024-06-29T14:35:06.141175+0000","last_became_peered":"2024-06-29T14:35:06.141175+0000","last_unstale":"2024-06-29T14:35:06.144393+0000","last_undegraded":"2024-06-29T14:35:06.144393+0000","last_fullsized":"2024-06-29T14:35:06.144393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:54.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.142459+0000","last_change":"2024-06-29T14:35:06.142459+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.142459+0000","last_undegraded":"2024-06-29T14:35:06.142459+0000","last_fullsized":"2024-06-29T14:35:06.142459+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153326+0000","last_change":"2024-06-29T14:35:06.152825+0000","last_active":"2024-06-29T14:35:06.153326+0000","last_peered":"2024-06-29T14:35:06.153326+0000","last_clean":"2024-06-29T14:35:06.153326+0000","last_became_active":"2024-06-29T14:35:06.152469+0000","last_became_peered":"2024-06-29T14:35:06.152469+0000","last_unstale":"2024-06-29T14:35:06.153326+0000","last_undegraded":"2024-06-29T14:35:06.153326+0000","last_fullsized":"2024-06-29T14:35:06.153326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:39.872319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134310+0000","last_change":"2024-06-29T14:35:06.134310+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134310+0000","last_undegraded":"2024-06-29T14:35:06.134310+0000","last_fullsized":"2024-06-29T14:35:06.134310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153817+0000","last_change":"2024-06-29T14:35:06.148661+0000","last_active":"2024-06-29T14:35:06.153817+0000","last_peered":"2024-06-29T14:35:06.153817+0000","last_clean":"2024-06-29T14:35:06.153817+0000","last_became_active":"2024-06-29T14:35:06.148307+0000","last_became_peered":"2024-06-29T14:35:06.148307+0000","last_unstale":"2024-06-29T14:35:06.153817+0000","last_undegraded":"2024-06-29T14:35:06.153817+0000","last_fullsized":"2024-06-29T14:35:06.153817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:46.690921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'86","reported_seq":257,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.140303+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:35:06.140303+0000","last_peered":"2024-06-29T14:35:06.140303+0000","last_clean":"2024-06-29T14:35:06.140303+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:35:06.140303+0000","last_undegraded":"2024-06-29T14:35:06.140303+0000","last_fullsized":"2024-06-29T14:35:06.140303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54768,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317072,"statfs":{"total":96636764160,"available":96580681728,"internally_reserved":0,"allocated":1413120,"data_stored":997410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542308,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50028,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321812,"statfs":{"total":96636764160,"available":96585535488,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542306,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509604,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575012,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58348,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313492,"statfs":{"total":96636764160,"available":96577015808,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509603,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1413120,"data_stored":997410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:35:11.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:35:11.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.766+0000 7fecd9b4a640 1 -- 172.21.15.59:0/1468790061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd40b40a0 msgr2=0x7fecd415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.766+0000 7fecd9b4a640 1 --2- 172.21.15.59:0/1468790061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd40b40a0 0x7fecd415e8d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fecbc005fd0 tx=0x7fecbc02a670 comp rx=0 tx=0).stop 2024-06-29T14:35:11.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.766+0000 7fecd9b4a640 1 -- 172.21.15.59:0/1468790061 shutdown_connections 2024-06-29T14:35:11.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.766+0000 7fecd9b4a640 1 --2- 172.21.15.59:0/1468790061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd40b40a0 0x7fecd415e8d0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.766+0000 7fecd9b4a640 1 -- 172.21.15.59:0/1468790061 >> 172.21.15.59:0/1468790061 conn(0x7fecd40d3ee0 msgr2=0x7fecd40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:11.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.766+0000 7fecd9b4a640 1 -- 172.21.15.59:0/1468790061 shutdown_connections 2024-06-29T14:35:11.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.766+0000 7fecd9b4a640 1 -- 172.21.15.59:0/1468790061 wait complete. 2024-06-29T14:35:11.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.768+0000 7fecd9b4a640 1 Processor -- start 2024-06-29T14:35:11.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.768+0000 7fecd9b4a640 1 -- start start 2024-06-29T14:35:11.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.769+0000 7fecd9b4a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecd40d23c0 0x7fecd4150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd9b4a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd415ee10 0x7fecd4157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd9b4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd414b140 0x7fecd414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd9b4a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fecd4160450 con 0x7fecd414b140 2024-06-29T14:35:11.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd9b4a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fecd41602d0 con 0x7fecd415ee10 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd9b4a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fecd41605d0 con 0x7fecd40d23c0 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd414b140 0x7fecd414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd414b140 0x7fecd414ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37916/0 (socket says 172.21.15.59:37916) 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 -- 172.21.15.59:0/4224334335 learned_addr learned my addr 172.21.15.59:0/4224334335 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd2ffd640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd415ee10 0x7fecd4157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd37fe640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecd40d23c0 0x7fecd4150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 -- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecd40d23c0 msgr2=0x7fecd4150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecd40d23c0 0x7fecd4150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 -- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd415ee10 msgr2=0x7fecd4157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd415ee10 0x7fecd4157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 -- 172.21.15.59:0/4224334335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd4152ee0 con 0x7fecd414b140 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd2ffd640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd415ee10 0x7fecd4157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:11.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd3fff640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd414b140 0x7fecd414ba80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7feccc00c760 tx=0x7feccc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:11.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7feccbfff640 1 -- 172.21.15.59:0/4224334335 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc003b70 con 0x7fecd414b140 2024-06-29T14:35:11.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7feccbfff640 1 -- 172.21.15.59:0/4224334335 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feccc003d30 con 0x7fecd414b140 2024-06-29T14:35:11.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd4156b90 con 0x7fecd414b140 2024-06-29T14:35:11.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7feccbfff640 1 -- 172.21.15.59:0/4224334335 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc0052d0 con 0x7fecd414b140 2024-06-29T14:35:11.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.770+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd4157150 con 0x7fecd414b140 2024-06-29T14:35:11.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.772+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd4152c50 con 0x7fecd414b140 2024-06-29T14:35:11.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.772+0000 7feccbfff640 1 -- 172.21.15.59:0/4224334335 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7feccc005470 con 0x7fecd414b140 2024-06-29T14:35:11.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.773+0000 7feccbfff640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fecb4076420 0x7fecb4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.773+0000 7feccbfff640 1 -- 172.21.15.59:0/4224334335 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7feccc097900 con 0x7fecd414b140 2024-06-29T14:35:11.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.773+0000 7fecd37fe640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fecb4076420 0x7fecb4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.773+0000 7fecd37fe640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fecb4076420 0x7fecb4078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fecc4000f60 tx=0x7fecc4009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:11.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.777+0000 7feccbfff640 1 -- 172.21.15.59:0/4224334335 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feccc060f30 con 0x7fecd414b140 2024-06-29T14:35:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.963+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 --> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fecd415e8d0 con 0x7fecb4076420 2024-06-29T14:35:11.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.968+0000 7feccbfff640 1 -- 172.21.15.59:0/4224334335 <== mgr.5809 v2:172.21.15.153:6832/2957341575 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246630 (secure 0 0 0) 0x7fecd415e8d0 con 0x7fecb4076420 2024-06-29T14:35:11.968 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:11.969 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:35:11.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fecb4076420 msgr2=0x7fecb4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fecb4076420 0x7fecb4078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fecc4000f60 tx=0x7fecc4009040 comp rx=0 tx=0).stop 2024-06-29T14:35:11.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd414b140 msgr2=0x7fecd414ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:11.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd414b140 0x7fecd414ba80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7feccc00c760 tx=0x7feccc00cc20 comp rx=0 tx=0).stop 2024-06-29T14:35:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 shutdown_connections 2024-06-29T14:35:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fecb4076420 0x7fecb4078930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd414b140 0x7fecd414ba80 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fecd415ee10 0x7fecd4157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 --2- 172.21.15.59:0/4224334335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fecd40d23c0 0x7fecd4150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 >> 172.21.15.59:0/4224334335 conn(0x7fecd40d3ee0 msgr2=0x7fecd4151260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 shutdown_connections 2024-06-29T14:35:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:11.971+0000 7fecd9b4a640 1 -- 172.21.15.59:0/4224334335 wait complete. 2024-06-29T14:35:11.979 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-06-29T14:35:10.033416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323648,"kb_used_data":7560,"kb_used_omap":12,"kb_used_meta":315955,"kb_avail":754651072,"statfs":{"total":773094113280,"available":772762697728,"internally_reserved":0,"allocated":7741440,"data_stored":4437072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323538482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.908141"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151105+0000","last_change":"2024-06-29T14:35:06.151105+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151105+0000","last_undegraded":"2024-06-29T14:35:06.151105+0000","last_fullsized":"2024-06-29T14:35:06.151105+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138965+0000","last_change":"2024-06-29T14:35:06.138965+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138965+0000","last_undegraded":"2024-06-29T14:35:06.138965+0000","last_fullsized":"2024-06-29T14:35:06.138965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138100+0000","last_change":"2024-06-29T14:35:06.138100+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138100+0000","last_undegraded":"2024-06-29T14:35:06.138100+0000","last_fullsized":"2024-06-29T14:35:06.138100+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138146+0000","last_change":"2024-06-29T14:35:06.138146+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138146+0000","last_undegraded":"2024-06-29T14:35:06.138146+0000","last_fullsized":"2024-06-29T14:35:06.138146+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132969+0000","last_change":"2024-06-29T14:35:06.132969+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132969+0000","last_undegraded":"2024-06-29T14:35:06.132969+0000","last_fullsized":"2024-06-29T14:35:06.132969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149626+0000","last_change":"2024-06-29T14:35:06.149626+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149626+0000","last_undegraded":"2024-06-29T14:35:06.149626+0000","last_fullsized":"2024-06-29T14:35:06.149626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135540+0000","last_change":"2024-06-29T14:35:06.135540+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135540+0000","last_undegraded":"2024-06-29T14:35:06.135540+0000","last_fullsized":"2024-06-29T14:35:06.135540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.148127+0000","last_change":"2024-06-29T14:35:06.148127+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.148127+0000","last_undegraded":"2024-06-29T14:35:06.148127+0000","last_fullsized":"2024-06-29T14:35:06.148127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.147564+0000","last_change":"2024-06-29T14:35:06.147564+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.147564+0000","last_undegraded":"2024-06-29T14:35:06.147564+0000","last_fullsized":"2024-06-29T14:35:06.147564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151644+0000","last_change":"2024-06-29T14:35:06.151644+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151644+0000","last_undegraded":"2024-06-29T14:35:06.151644+0000","last_fullsized":"2024-06-29T14:35:06.151644+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.144328+0000","last_change":"2024-06-29T14:35:06.144328+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.144328+0000","last_undegraded":"2024-06-29T14:35:06.144328+0000","last_fullsized":"2024-06-29T14:35:06.144328+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146980+0000","last_change":"2024-06-29T14:35:06.146980+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146980+0000","last_undegraded":"2024-06-29T14:35:06.146980+0000","last_fullsized":"2024-06-29T14:35:06.146980+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.151643+0000","last_change":"2024-06-29T14:35:06.151643+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.151643+0000","last_undegraded":"2024-06-29T14:35:06.151643+0000","last_fullsized":"2024-06-29T14:35:06.151643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136411+0000","last_change":"2024-06-29T14:35:06.136411+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136411+0000","last_undegraded":"2024-06-29T14:35:06.136411+0000","last_fullsized":"2024-06-29T14:35:06.136411+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137811+0000","last_change":"2024-06-29T14:35:06.137811+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137811+0000","last_undegraded":"2024-06-29T14:35:06.137811+0000","last_fullsized":"2024-06-29T14:35:06.137811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146620+0000","last_change":"2024-06-29T14:35:06.146620+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146620+0000","last_undegraded":"2024-06-29T14:35:06.146620+0000","last_fullsized":"2024-06-29T14:35:06.146620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137727+0000","last_change":"2024-06-29T14:35:06.137727+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137727+0000","last_undegraded":"2024-06-29T14:35:06.137727+0000","last_fullsized":"2024-06-29T14:35:06.137727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138015+0000","last_change":"2024-06-29T14:35:06.138015+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138015+0000","last_undegraded":"2024-06-29T14:35:06.138015+0000","last_fullsized":"2024-06-29T14:35:06.138015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137633+0000","last_change":"2024-06-29T14:35:06.137633+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137633+0000","last_undegraded":"2024-06-29T14:35:06.137633+0000","last_fullsized":"2024-06-29T14:35:06.137633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135000+0000","last_change":"2024-06-29T14:35:06.135000+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135000+0000","last_undegraded":"2024-06-29T14:35:06.135000+0000","last_fullsized":"2024-06-29T14:35:06.135000+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.133885+0000","last_change":"2024-06-29T14:35:06.133885+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.133885+0000","last_undegraded":"2024-06-29T14:35:06.133885+0000","last_fullsized":"2024-06-29T14:35:06.133885+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.146360+0000","last_change":"2024-06-29T14:35:06.146360+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.146360+0000","last_undegraded":"2024-06-29T14:35:06.146360+0000","last_fullsized":"2024-06-29T14:35:06.146360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141017+0000","last_change":"2024-06-29T14:35:06.141017+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141017+0000","last_undegraded":"2024-06-29T14:35:06.141017+0000","last_fullsized":"2024-06-29T14:35:06.141017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138890+0000","last_change":"2024-06-29T14:35:06.138890+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138890+0000","last_undegraded":"2024-06-29T14:35:06.138890+0000","last_fullsized":"2024-06-29T14:35:06.138890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137237+0000","last_change":"2024-06-29T14:35:06.137237+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137237+0000","last_undegraded":"2024-06-29T14:35:06.137237+0000","last_fullsized":"2024-06-29T14:35:06.137237+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.145779+0000","last_change":"2024-06-29T14:35:06.145779+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.145779+0000","last_undegraded":"2024-06-29T14:35:06.145779+0000","last_fullsized":"2024-06-29T14:35:06.145779+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.155904+0000","last_change":"2024-06-29T14:35:06.155904+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.155904+0000","last_undegraded":"2024-06-29T14:35:06.155904+0000","last_fullsized":"2024-06-29T14:35:06.155904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134043+0000","last_change":"2024-06-29T14:35:06.134043+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134043+0000","last_undegraded":"2024-06-29T14:35:06.134043+0000","last_fullsized":"2024-06-29T14:35:06.134043+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137017+0000","last_change":"2024-06-29T14:35:06.137017+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137017+0000","last_undegraded":"2024-06-29T14:35:06.137017+0000","last_fullsized":"2024-06-29T14:35:06.137017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136330+0000","last_change":"2024-06-29T14:35:06.136330+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136330+0000","last_undegraded":"2024-06-29T14:35:06.136330+0000","last_fullsized":"2024-06-29T14:35:06.136330+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134973+0000","last_change":"2024-06-29T14:35:06.134973+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134973+0000","last_undegraded":"2024-06-29T14:35:06.134973+0000","last_fullsized":"2024-06-29T14:35:06.134973+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139697+0000","last_change":"2024-06-29T14:35:06.139697+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139697+0000","last_undegraded":"2024-06-29T14:35:06.139697+0000","last_fullsized":"2024-06-29T14:35:06.139697+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137035+0000","last_change":"2024-06-29T14:35:06.137035+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137035+0000","last_undegraded":"2024-06-29T14:35:06.137035+0000","last_fullsized":"2024-06-29T14:35:06.137035+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.141964+0000","last_change":"2024-06-29T14:35:06.139267+0000","last_active":"2024-06-29T14:35:06.141964+0000","last_peered":"2024-06-29T14:35:06.141964+0000","last_clean":"2024-06-29T14:35:06.141964+0000","last_became_active":"2024-06-29T14:35:06.138932+0000","last_became_peered":"2024-06-29T14:35:06.138932+0000","last_unstale":"2024-06-29T14:35:06.141964+0000","last_undegraded":"2024-06-29T14:35:06.141964+0000","last_fullsized":"2024-06-29T14:35:06.141964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:47.855244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136457+0000","last_change":"2024-06-29T14:35:06.136457+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136457+0000","last_undegraded":"2024-06-29T14:35:06.136457+0000","last_fullsized":"2024-06-29T14:35:06.136457+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.158642+0000","last_change":"2024-06-29T14:35:06.157899+0000","last_active":"2024-06-29T14:35:06.158642+0000","last_peered":"2024-06-29T14:35:06.158642+0000","last_clean":"2024-06-29T14:35:06.158642+0000","last_became_active":"2024-06-29T14:35:06.157591+0000","last_became_peered":"2024-06-29T14:35:06.157591+0000","last_unstale":"2024-06-29T14:35:06.158642+0000","last_undegraded":"2024-06-29T14:35:06.158642+0000","last_fullsized":"2024-06-29T14:35:06.158642+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:04:01.035476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134883+0000","last_change":"2024-06-29T14:35:06.134883+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134883+0000","last_undegraded":"2024-06-29T14:35:06.134883+0000","last_fullsized":"2024-06-29T14:35:06.134883+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.162625+0000","last_change":"2024-06-29T14:35:06.160847+0000","last_active":"2024-06-29T14:35:06.162625+0000","last_peered":"2024-06-29T14:35:06.162625+0000","last_clean":"2024-06-29T14:35:06.162625+0000","last_became_active":"2024-06-29T14:35:06.160558+0000","last_became_peered":"2024-06-29T14:35:06.160558+0000","last_unstale":"2024-06-29T14:35:06.162625+0000","last_undegraded":"2024-06-29T14:35:06.162625+0000","last_fullsized":"2024-06-29T14:35:06.162625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:29.140867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.142843+0000","last_change":"2024-06-29T14:35:06.142843+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.142843+0000","last_undegraded":"2024-06-29T14:35:06.142843+0000","last_fullsized":"2024-06-29T14:35:06.142843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.147719+0000","last_change":"2024-06-29T14:35:06.147172+0000","last_active":"2024-06-29T14:35:06.147719+0000","last_peered":"2024-06-29T14:35:06.147719+0000","last_clean":"2024-06-29T14:35:06.147719+0000","last_became_active":"2024-06-29T14:35:06.146877+0000","last_became_peered":"2024-06-29T14:35:06.146877+0000","last_unstale":"2024-06-29T14:35:06.147719+0000","last_undegraded":"2024-06-29T14:35:06.147719+0000","last_fullsized":"2024-06-29T14:35:06.147719+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:11:45.335558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132273+0000","last_change":"2024-06-29T14:35:06.132273+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132273+0000","last_undegraded":"2024-06-29T14:35:06.132273+0000","last_fullsized":"2024-06-29T14:35:06.132273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.155462+0000","last_change":"2024-06-29T14:35:06.154311+0000","last_active":"2024-06-29T14:35:06.155462+0000","last_peered":"2024-06-29T14:35:06.155462+0000","last_clean":"2024-06-29T14:35:06.155462+0000","last_became_active":"2024-06-29T14:35:06.153161+0000","last_became_peered":"2024-06-29T14:35:06.153161+0000","last_unstale":"2024-06-29T14:35:06.155462+0000","last_undegraded":"2024-06-29T14:35:06.155462+0000","last_fullsized":"2024-06-29T14:35:06.155462+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:03.504751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138048+0000","last_change":"2024-06-29T14:35:06.138048+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138048+0000","last_undegraded":"2024-06-29T14:35:06.138048+0000","last_fullsized":"2024-06-29T14:35:06.138048+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144211+0000","last_change":"2024-06-29T14:35:06.142746+0000","last_active":"2024-06-29T14:35:06.144211+0000","last_peered":"2024-06-29T14:35:06.144211+0000","last_clean":"2024-06-29T14:35:06.144211+0000","last_became_active":"2024-06-29T14:35:06.142398+0000","last_became_peered":"2024-06-29T14:35:06.142398+0000","last_unstale":"2024-06-29T14:35:06.144211+0000","last_undegraded":"2024-06-29T14:35:06.144211+0000","last_fullsized":"2024-06-29T14:35:06.144211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:52:09.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.143419+0000","last_change":"2024-06-29T14:35:06.143419+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.143419+0000","last_undegraded":"2024-06-29T14:35:06.143419+0000","last_fullsized":"2024-06-29T14:35:06.143419+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.142739+0000","last_change":"2024-06-29T14:35:06.141432+0000","last_active":"2024-06-29T14:35:06.142739+0000","last_peered":"2024-06-29T14:35:06.142739+0000","last_clean":"2024-06-29T14:35:06.142739+0000","last_became_active":"2024-06-29T14:35:06.141079+0000","last_became_peered":"2024-06-29T14:35:06.141079+0000","last_unstale":"2024-06-29T14:35:06.142739+0000","last_undegraded":"2024-06-29T14:35:06.142739+0000","last_fullsized":"2024-06-29T14:35:06.142739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:28.307716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136427+0000","last_change":"2024-06-29T14:35:06.136427+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136427+0000","last_undegraded":"2024-06-29T14:35:06.136427+0000","last_fullsized":"2024-06-29T14:35:06.136427+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154777+0000","last_change":"2024-06-29T14:35:06.153920+0000","last_active":"2024-06-29T14:35:06.154777+0000","last_peered":"2024-06-29T14:35:06.154777+0000","last_clean":"2024-06-29T14:35:06.154777+0000","last_became_active":"2024-06-29T14:35:06.153047+0000","last_became_peered":"2024-06-29T14:35:06.153047+0000","last_unstale":"2024-06-29T14:35:06.154777+0000","last_undegraded":"2024-06-29T14:35:06.154777+0000","last_fullsized":"2024-06-29T14:35:06.154777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:42.614890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132166+0000","last_change":"2024-06-29T14:35:06.132166+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132166+0000","last_undegraded":"2024-06-29T14:35:06.132166+0000","last_fullsized":"2024-06-29T14:35:06.132166+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.146782+0000","last_change":"2024-06-29T14:35:06.145945+0000","last_active":"2024-06-29T14:35:06.146782+0000","last_peered":"2024-06-29T14:35:06.146782+0000","last_clean":"2024-06-29T14:35:06.146782+0000","last_became_active":"2024-06-29T14:35:06.145555+0000","last_became_peered":"2024-06-29T14:35:06.145555+0000","last_unstale":"2024-06-29T14:35:06.146782+0000","last_undegraded":"2024-06-29T14:35:06.146782+0000","last_fullsized":"2024-06-29T14:35:06.146782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:53.987132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136980+0000","last_change":"2024-06-29T14:35:06.136980+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136980+0000","last_undegraded":"2024-06-29T14:35:06.136980+0000","last_fullsized":"2024-06-29T14:35:06.136980+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156774+0000","last_change":"2024-06-29T14:35:06.156160+0000","last_active":"2024-06-29T14:35:06.156774+0000","last_peered":"2024-06-29T14:35:06.156774+0000","last_clean":"2024-06-29T14:35:06.156774+0000","last_became_active":"2024-06-29T14:35:06.155823+0000","last_became_peered":"2024-06-29T14:35:06.155823+0000","last_unstale":"2024-06-29T14:35:06.156774+0000","last_undegraded":"2024-06-29T14:35:06.156774+0000","last_fullsized":"2024-06-29T14:35:06.156774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:00.015532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135625+0000","last_change":"2024-06-29T14:35:06.135625+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135625+0000","last_undegraded":"2024-06-29T14:35:06.135625+0000","last_fullsized":"2024-06-29T14:35:06.135625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.159311+0000","last_change":"2024-06-29T14:35:06.158866+0000","last_active":"2024-06-29T14:35:06.159311+0000","last_peered":"2024-06-29T14:35:06.159311+0000","last_clean":"2024-06-29T14:35:06.159311+0000","last_became_active":"2024-06-29T14:35:06.158615+0000","last_became_peered":"2024-06-29T14:35:06.158615+0000","last_unstale":"2024-06-29T14:35:06.159311+0000","last_undegraded":"2024-06-29T14:35:06.159311+0000","last_fullsized":"2024-06-29T14:35:06.159311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:34.626783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132361+0000","last_change":"2024-06-29T14:35:06.132361+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132361+0000","last_undegraded":"2024-06-29T14:35:06.132361+0000","last_fullsized":"2024-06-29T14:35:06.132361+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153751+0000","last_change":"2024-06-29T14:35:06.148671+0000","last_active":"2024-06-29T14:35:06.153751+0000","last_peered":"2024-06-29T14:35:06.153751+0000","last_clean":"2024-06-29T14:35:06.153751+0000","last_became_active":"2024-06-29T14:35:06.148322+0000","last_became_peered":"2024-06-29T14:35:06.148322+0000","last_unstale":"2024-06-29T14:35:06.153751+0000","last_undegraded":"2024-06-29T14:35:06.153751+0000","last_fullsized":"2024-06-29T14:35:06.153751+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:52.995413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139128+0000","last_change":"2024-06-29T14:35:06.139128+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139128+0000","last_undegraded":"2024-06-29T14:35:06.139128+0000","last_fullsized":"2024-06-29T14:35:06.139128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153807+0000","last_change":"2024-06-29T14:35:06.148685+0000","last_active":"2024-06-29T14:35:06.153807+0000","last_peered":"2024-06-29T14:35:06.153807+0000","last_clean":"2024-06-29T14:35:06.153807+0000","last_became_active":"2024-06-29T14:35:06.148343+0000","last_became_peered":"2024-06-29T14:35:06.148343+0000","last_unstale":"2024-06-29T14:35:06.153807+0000","last_undegraded":"2024-06-29T14:35:06.153807+0000","last_fullsized":"2024-06-29T14:35:06.153807+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:30.741869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.137093+0000","last_change":"2024-06-29T14:35:06.137093+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.137093+0000","last_undegraded":"2024-06-29T14:35:06.137093+0000","last_fullsized":"2024-06-29T14:35:06.137093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.146783+0000","last_change":"2024-06-29T14:35:06.146043+0000","last_active":"2024-06-29T14:35:06.146783+0000","last_peered":"2024-06-29T14:35:06.146783+0000","last_clean":"2024-06-29T14:35:06.146783+0000","last_became_active":"2024-06-29T14:35:06.145648+0000","last_became_peered":"2024-06-29T14:35:06.145648+0000","last_unstale":"2024-06-29T14:35:06.146783+0000","last_undegraded":"2024-06-29T14:35:06.146783+0000","last_fullsized":"2024-06-29T14:35:06.146783+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:23.312796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.143970+0000","last_change":"2024-06-29T14:35:06.143970+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.143970+0000","last_undegraded":"2024-06-29T14:35:06.143970+0000","last_fullsized":"2024-06-29T14:35:06.143970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144837+0000","last_change":"2024-06-29T14:35:06.144003+0000","last_active":"2024-06-29T14:35:06.144837+0000","last_peered":"2024-06-29T14:35:06.144837+0000","last_clean":"2024-06-29T14:35:06.144837+0000","last_became_active":"2024-06-29T14:35:06.143542+0000","last_became_peered":"2024-06-29T14:35:06.143542+0000","last_unstale":"2024-06-29T14:35:06.144837+0000","last_undegraded":"2024-06-29T14:35:06.144837+0000","last_fullsized":"2024-06-29T14:35:06.144837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:20.980556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138217+0000","last_change":"2024-06-29T14:35:06.138217+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138217+0000","last_undegraded":"2024-06-29T14:35:06.138217+0000","last_fullsized":"2024-06-29T14:35:06.138217+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.145401+0000","last_change":"2024-06-29T14:35:06.143641+0000","last_active":"2024-06-29T14:35:06.145401+0000","last_peered":"2024-06-29T14:35:06.145401+0000","last_clean":"2024-06-29T14:35:06.145401+0000","last_became_active":"2024-06-29T14:35:06.143030+0000","last_became_peered":"2024-06-29T14:35:06.143030+0000","last_unstale":"2024-06-29T14:35:06.145401+0000","last_undegraded":"2024-06-29T14:35:06.145401+0000","last_fullsized":"2024-06-29T14:35:06.145401+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:21:59.463680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141803+0000","last_change":"2024-06-29T14:35:06.141803+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141803+0000","last_undegraded":"2024-06-29T14:35:06.141803+0000","last_fullsized":"2024-06-29T14:35:06.141803+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156969+0000","last_change":"2024-06-29T14:35:06.155992+0000","last_active":"2024-06-29T14:35:06.156969+0000","last_peered":"2024-06-29T14:35:06.156969+0000","last_clean":"2024-06-29T14:35:06.156969+0000","last_became_active":"2024-06-29T14:35:06.155609+0000","last_became_peered":"2024-06-29T14:35:06.155609+0000","last_unstale":"2024-06-29T14:35:06.156969+0000","last_undegraded":"2024-06-29T14:35:06.156969+0000","last_fullsized":"2024-06-29T14:35:06.156969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:59.794303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.147888+0000","last_change":"2024-06-29T14:35:06.147888+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.147888+0000","last_undegraded":"2024-06-29T14:35:06.147888+0000","last_fullsized":"2024-06-29T14:35:06.147888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.150684+0000","last_change":"2024-06-29T14:35:06.145742+0000","last_active":"2024-06-29T14:35:06.150684+0000","last_peered":"2024-06-29T14:35:06.150684+0000","last_clean":"2024-06-29T14:35:06.150684+0000","last_became_active":"2024-06-29T14:35:06.145327+0000","last_became_peered":"2024-06-29T14:35:06.145327+0000","last_unstale":"2024-06-29T14:35:06.150684+0000","last_undegraded":"2024-06-29T14:35:06.150684+0000","last_fullsized":"2024-06-29T14:35:06.150684+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:40:50.966470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132769+0000","last_change":"2024-06-29T14:35:06.132769+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132769+0000","last_undegraded":"2024-06-29T14:35:06.132769+0000","last_fullsized":"2024-06-29T14:35:06.132769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.160532+0000","last_change":"2024-06-29T14:35:06.159992+0000","last_active":"2024-06-29T14:35:06.160532+0000","last_peered":"2024-06-29T14:35:06.160532+0000","last_clean":"2024-06-29T14:35:06.160532+0000","last_became_active":"2024-06-29T14:35:06.159710+0000","last_became_peered":"2024-06-29T14:35:06.159710+0000","last_unstale":"2024-06-29T14:35:06.160532+0000","last_undegraded":"2024-06-29T14:35:06.160532+0000","last_fullsized":"2024-06-29T14:35:06.160532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:47.483637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149852+0000","last_change":"2024-06-29T14:35:06.149852+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149852+0000","last_undegraded":"2024-06-29T14:35:06.149852+0000","last_fullsized":"2024-06-29T14:35:06.149852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.149922+0000","last_change":"2024-06-29T14:35:06.149423+0000","last_active":"2024-06-29T14:35:06.149922+0000","last_peered":"2024-06-29T14:35:06.149922+0000","last_clean":"2024-06-29T14:35:06.149922+0000","last_became_active":"2024-06-29T14:35:06.149111+0000","last_became_peered":"2024-06-29T14:35:06.149111+0000","last_unstale":"2024-06-29T14:35:06.149922+0000","last_undegraded":"2024-06-29T14:35:06.149922+0000","last_fullsized":"2024-06-29T14:35:06.149922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:29.397336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.138649+0000","last_change":"2024-06-29T14:35:06.138649+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.138649+0000","last_undegraded":"2024-06-29T14:35:06.138649+0000","last_fullsized":"2024-06-29T14:35:06.138649+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.159867+0000","last_change":"2024-06-29T14:35:06.159208+0000","last_active":"2024-06-29T14:35:06.159867+0000","last_peered":"2024-06-29T14:35:06.159867+0000","last_clean":"2024-06-29T14:35:06.159867+0000","last_became_active":"2024-06-29T14:35:06.158901+0000","last_became_peered":"2024-06-29T14:35:06.158901+0000","last_unstale":"2024-06-29T14:35:06.159867+0000","last_undegraded":"2024-06-29T14:35:06.159867+0000","last_fullsized":"2024-06-29T14:35:06.159867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:41:37.874410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.133545+0000","last_change":"2024-06-29T14:35:06.133545+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.133545+0000","last_undegraded":"2024-06-29T14:35:06.133545+0000","last_fullsized":"2024-06-29T14:35:06.133545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.145350+0000","last_change":"2024-06-29T14:35:06.142795+0000","last_active":"2024-06-29T14:35:06.145350+0000","last_peered":"2024-06-29T14:35:06.145350+0000","last_clean":"2024-06-29T14:35:06.145350+0000","last_became_active":"2024-06-29T14:35:06.142463+0000","last_became_peered":"2024-06-29T14:35:06.142463+0000","last_unstale":"2024-06-29T14:35:06.145350+0000","last_undegraded":"2024-06-29T14:35:06.145350+0000","last_fullsized":"2024-06-29T14:35:06.145350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:13.219465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136269+0000","last_change":"2024-06-29T14:35:06.136269+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136269+0000","last_undegraded":"2024-06-29T14:35:06.136269+0000","last_fullsized":"2024-06-29T14:35:06.136269+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156310+0000","last_change":"2024-06-29T14:35:06.154839+0000","last_active":"2024-06-29T14:35:06.156310+0000","last_peered":"2024-06-29T14:35:06.156310+0000","last_clean":"2024-06-29T14:35:06.156310+0000","last_became_active":"2024-06-29T14:35:06.154513+0000","last_became_peered":"2024-06-29T14:35:06.154513+0000","last_unstale":"2024-06-29T14:35:06.156310+0000","last_undegraded":"2024-06-29T14:35:06.156310+0000","last_fullsized":"2024-06-29T14:35:06.156310+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:13.939011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.150280+0000","last_change":"2024-06-29T14:35:06.150280+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.150280+0000","last_undegraded":"2024-06-29T14:35:06.150280+0000","last_fullsized":"2024-06-29T14:35:06.150280+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144749+0000","last_change":"2024-06-29T14:35:06.143667+0000","last_active":"2024-06-29T14:35:06.144749+0000","last_peered":"2024-06-29T14:35:06.144749+0000","last_clean":"2024-06-29T14:35:06.144749+0000","last_became_active":"2024-06-29T14:35:06.143312+0000","last_became_peered":"2024-06-29T14:35:06.143312+0000","last_unstale":"2024-06-29T14:35:06.144749+0000","last_undegraded":"2024-06-29T14:35:06.144749+0000","last_fullsized":"2024-06-29T14:35:06.144749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:00.265866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139175+0000","last_change":"2024-06-29T14:35:06.139175+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139175+0000","last_undegraded":"2024-06-29T14:35:06.139175+0000","last_fullsized":"2024-06-29T14:35:06.139175+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153279+0000","last_change":"2024-06-29T14:35:06.152663+0000","last_active":"2024-06-29T14:35:06.153279+0000","last_peered":"2024-06-29T14:35:06.153279+0000","last_clean":"2024-06-29T14:35:06.153279+0000","last_became_active":"2024-06-29T14:35:06.152267+0000","last_became_peered":"2024-06-29T14:35:06.152267+0000","last_unstale":"2024-06-29T14:35:06.153279+0000","last_undegraded":"2024-06-29T14:35:06.153279+0000","last_fullsized":"2024-06-29T14:35:06.153279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:15.826985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.149080+0000","last_change":"2024-06-29T14:35:06.149080+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.149080+0000","last_undegraded":"2024-06-29T14:35:06.149080+0000","last_fullsized":"2024-06-29T14:35:06.149080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.155693+0000","last_change":"2024-06-29T14:35:06.155153+0000","last_active":"2024-06-29T14:35:06.155693+0000","last_peered":"2024-06-29T14:35:06.155693+0000","last_clean":"2024-06-29T14:35:06.155693+0000","last_became_active":"2024-06-29T14:35:06.154919+0000","last_became_peered":"2024-06-29T14:35:06.154919+0000","last_unstale":"2024-06-29T14:35:06.155693+0000","last_undegraded":"2024-06-29T14:35:06.155693+0000","last_fullsized":"2024-06-29T14:35:06.155693+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:21.593203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.141904+0000","last_change":"2024-06-29T14:35:06.141904+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.141904+0000","last_undegraded":"2024-06-29T14:35:06.141904+0000","last_fullsized":"2024-06-29T14:35:06.141904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154161+0000","last_change":"2024-06-29T14:35:06.149483+0000","last_active":"2024-06-29T14:35:06.154161+0000","last_peered":"2024-06-29T14:35:06.154161+0000","last_clean":"2024-06-29T14:35:06.154161+0000","last_became_active":"2024-06-29T14:35:06.149163+0000","last_became_peered":"2024-06-29T14:35:06.149163+0000","last_unstale":"2024-06-29T14:35:06.154161+0000","last_undegraded":"2024-06-29T14:35:06.154161+0000","last_fullsized":"2024-06-29T14:35:06.154161+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:37:49.909672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135183+0000","last_change":"2024-06-29T14:35:06.135183+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135183+0000","last_undegraded":"2024-06-29T14:35:06.135183+0000","last_fullsized":"2024-06-29T14:35:06.135183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153790+0000","last_change":"2024-06-29T14:35:06.148884+0000","last_active":"2024-06-29T14:35:06.153790+0000","last_peered":"2024-06-29T14:35:06.153790+0000","last_clean":"2024-06-29T14:35:06.153790+0000","last_became_active":"2024-06-29T14:35:06.148389+0000","last_became_peered":"2024-06-29T14:35:06.148389+0000","last_unstale":"2024-06-29T14:35:06.153790+0000","last_undegraded":"2024-06-29T14:35:06.153790+0000","last_fullsized":"2024-06-29T14:35:06.153790+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:08.291163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.139396+0000","last_change":"2024-06-29T14:35:06.139396+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.139396+0000","last_undegraded":"2024-06-29T14:35:06.139396+0000","last_fullsized":"2024-06-29T14:35:06.139396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.154578+0000","last_change":"2024-06-29T14:35:06.152507+0000","last_active":"2024-06-29T14:35:06.154578+0000","last_peered":"2024-06-29T14:35:06.154578+0000","last_clean":"2024-06-29T14:35:06.154578+0000","last_became_active":"2024-06-29T14:35:06.152204+0000","last_became_peered":"2024-06-29T14:35:06.152204+0000","last_unstale":"2024-06-29T14:35:06.154578+0000","last_undegraded":"2024-06-29T14:35:06.154578+0000","last_fullsized":"2024-06-29T14:35:06.154578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:07.902240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.136887+0000","last_change":"2024-06-29T14:35:06.136887+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.136887+0000","last_undegraded":"2024-06-29T14:35:06.136887+0000","last_fullsized":"2024-06-29T14:35:06.136887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144393+0000","last_change":"2024-06-29T14:35:06.141470+0000","last_active":"2024-06-29T14:35:06.144393+0000","last_peered":"2024-06-29T14:35:06.144393+0000","last_clean":"2024-06-29T14:35:06.144393+0000","last_became_active":"2024-06-29T14:35:06.141175+0000","last_became_peered":"2024-06-29T14:35:06.141175+0000","last_unstale":"2024-06-29T14:35:06.144393+0000","last_undegraded":"2024-06-29T14:35:06.144393+0000","last_fullsized":"2024-06-29T14:35:06.144393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:54.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.142459+0000","last_change":"2024-06-29T14:35:06.142459+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.142459+0000","last_undegraded":"2024-06-29T14:35:06.142459+0000","last_fullsized":"2024-06-29T14:35:06.142459+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153326+0000","last_change":"2024-06-29T14:35:06.152825+0000","last_active":"2024-06-29T14:35:06.153326+0000","last_peered":"2024-06-29T14:35:06.153326+0000","last_clean":"2024-06-29T14:35:06.153326+0000","last_became_active":"2024-06-29T14:35:06.152469+0000","last_became_peered":"2024-06-29T14:35:06.152469+0000","last_unstale":"2024-06-29T14:35:06.153326+0000","last_undegraded":"2024-06-29T14:35:06.153326+0000","last_fullsized":"2024-06-29T14:35:06.153326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:39.872319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134310+0000","last_change":"2024-06-29T14:35:06.134310+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134310+0000","last_undegraded":"2024-06-29T14:35:06.134310+0000","last_fullsized":"2024-06-29T14:35:06.134310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153817+0000","last_change":"2024-06-29T14:35:06.148661+0000","last_active":"2024-06-29T14:35:06.153817+0000","last_peered":"2024-06-29T14:35:06.153817+0000","last_clean":"2024-06-29T14:35:06.153817+0000","last_became_active":"2024-06-29T14:35:06.148307+0000","last_became_peered":"2024-06-29T14:35:06.148307+0000","last_unstale":"2024-06-29T14:35:06.153817+0000","last_undegraded":"2024-06-29T14:35:06.153817+0000","last_fullsized":"2024-06-29T14:35:06.153817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:46.690921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'86","reported_seq":257,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.140303+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:35:06.140303+0000","last_peered":"2024-06-29T14:35:06.140303+0000","last_clean":"2024-06-29T14:35:06.140303+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:35:06.140303+0000","last_undegraded":"2024-06-29T14:35:06.140303+0000","last_fullsized":"2024-06-29T14:35:06.140303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54768,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317072,"statfs":{"total":96636764160,"available":96580681728,"internally_reserved":0,"allocated":1413120,"data_stored":997410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542308,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542307,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50028,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321812,"statfs":{"total":96636764160,"available":96585535488,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542306,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509604,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575012,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58348,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313492,"statfs":{"total":96636764160,"available":96577015808,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509603,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1413120,"data_stored":997410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:35:11.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-06-29T14:35:11.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.151105+0000', 'last_change': '2024-06-29T14:35:06.151105+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.151105+0000', 'last_undegraded': '2024-06-29T14:35:06.151105+0000', 'last_fullsized': '2024-06-29T14:35:06.151105+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-06-29T14:35:11.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.138965+0000', 'last_change': '2024-06-29T14:35:06.138965+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.138965+0000', 'last_undegraded': '2024-06-29T14:35:06.138965+0000', 'last_fullsized': '2024-06-29T14:35:06.138965+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.138100+0000', 'last_change': '2024-06-29T14:35:06.138100+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.138100+0000', 'last_undegraded': '2024-06-29T14:35:06.138100+0000', 'last_fullsized': '2024-06-29T14:35:06.138100+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.138146+0000', 'last_change': '2024-06-29T14:35:06.138146+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.138146+0000', 'last_undegraded': '2024-06-29T14:35:06.138146+0000', 'last_fullsized': '2024-06-29T14:35:06.138146+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.132969+0000', 'last_change': '2024-06-29T14:35:06.132969+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.132969+0000', 'last_undegraded': '2024-06-29T14:35:06.132969+0000', 'last_fullsized': '2024-06-29T14:35:06.132969+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.149626+0000', 'last_change': '2024-06-29T14:35:06.149626+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.149626+0000', 'last_undegraded': '2024-06-29T14:35:06.149626+0000', 'last_fullsized': '2024-06-29T14:35:06.149626+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.135540+0000', 'last_change': '2024-06-29T14:35:06.135540+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.135540+0000', 'last_undegraded': '2024-06-29T14:35:06.135540+0000', 'last_fullsized': '2024-06-29T14:35:06.135540+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.148127+0000', 'last_change': '2024-06-29T14:35:06.148127+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.148127+0000', 'last_undegraded': '2024-06-29T14:35:06.148127+0000', 'last_fullsized': '2024-06-29T14:35:06.148127+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.147564+0000', 'last_change': '2024-06-29T14:35:06.147564+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.147564+0000', 'last_undegraded': '2024-06-29T14:35:06.147564+0000', 'last_fullsized': '2024-06-29T14:35:06.147564+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.151644+0000', 'last_change': '2024-06-29T14:35:06.151644+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.151644+0000', 'last_undegraded': '2024-06-29T14:35:06.151644+0000', 'last_fullsized': '2024-06-29T14:35:06.151644+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.144328+0000', 'last_change': '2024-06-29T14:35:06.144328+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.144328+0000', 'last_undegraded': '2024-06-29T14:35:06.144328+0000', 'last_fullsized': '2024-06-29T14:35:06.144328+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.146980+0000', 'last_change': '2024-06-29T14:35:06.146980+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.146980+0000', 'last_undegraded': '2024-06-29T14:35:06.146980+0000', 'last_fullsized': '2024-06-29T14:35:06.146980+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-06-29T14:35:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.151643+0000', 'last_change': '2024-06-29T14:35:06.151643+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.151643+0000', 'last_undegraded': '2024-06-29T14:35:06.151643+0000', 'last_fullsized': '2024-06-29T14:35:06.151643+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.136411+0000', 'last_change': '2024-06-29T14:35:06.136411+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.136411+0000', 'last_undegraded': '2024-06-29T14:35:06.136411+0000', 'last_fullsized': '2024-06-29T14:35:06.136411+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.137811+0000', 'last_change': '2024-06-29T14:35:06.137811+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.137811+0000', 'last_undegraded': '2024-06-29T14:35:06.137811+0000', 'last_fullsized': '2024-06-29T14:35:06.137811+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.146620+0000', 'last_change': '2024-06-29T14:35:06.146620+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.146620+0000', 'last_undegraded': '2024-06-29T14:35:06.146620+0000', 'last_fullsized': '2024-06-29T14:35:06.146620+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.137727+0000', 'last_change': '2024-06-29T14:35:06.137727+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.137727+0000', 'last_undegraded': '2024-06-29T14:35:06.137727+0000', 'last_fullsized': '2024-06-29T14:35:06.137727+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.138015+0000', 'last_change': '2024-06-29T14:35:06.138015+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.138015+0000', 'last_undegraded': '2024-06-29T14:35:06.138015+0000', 'last_fullsized': '2024-06-29T14:35:06.138015+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-06-29T14:35:11.986 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.137633+0000', 'last_change': '2024-06-29T14:35:06.137633+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.137633+0000', 'last_undegraded': '2024-06-29T14:35:06.137633+0000', 'last_fullsized': '2024-06-29T14:35:06.137633+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.135000+0000', 'last_change': '2024-06-29T14:35:06.135000+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.135000+0000', 'last_undegraded': '2024-06-29T14:35:06.135000+0000', 'last_fullsized': '2024-06-29T14:35:06.135000+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.133885+0000', 'last_change': '2024-06-29T14:35:06.133885+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.133885+0000', 'last_undegraded': '2024-06-29T14:35:06.133885+0000', 'last_fullsized': '2024-06-29T14:35:06.133885+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.146360+0000', 'last_change': '2024-06-29T14:35:06.146360+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.146360+0000', 'last_undegraded': '2024-06-29T14:35:06.146360+0000', 'last_fullsized': '2024-06-29T14:35:06.146360+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.141017+0000', 'last_change': '2024-06-29T14:35:06.141017+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.141017+0000', 'last_undegraded': '2024-06-29T14:35:06.141017+0000', 'last_fullsized': '2024-06-29T14:35:06.141017+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.138890+0000', 'last_change': '2024-06-29T14:35:06.138890+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.138890+0000', 'last_undegraded': '2024-06-29T14:35:06.138890+0000', 'last_fullsized': '2024-06-29T14:35:06.138890+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.987 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.137237+0000', 'last_change': '2024-06-29T14:35:06.137237+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.137237+0000', 'last_undegraded': '2024-06-29T14:35:06.137237+0000', 'last_fullsized': '2024-06-29T14:35:06.137237+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.145779+0000', 'last_change': '2024-06-29T14:35:06.145779+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.145779+0000', 'last_undegraded': '2024-06-29T14:35:06.145779+0000', 'last_fullsized': '2024-06-29T14:35:06.145779+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.155904+0000', 'last_change': '2024-06-29T14:35:06.155904+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.155904+0000', 'last_undegraded': '2024-06-29T14:35:06.155904+0000', 'last_fullsized': '2024-06-29T14:35:06.155904+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.134043+0000', 'last_change': '2024-06-29T14:35:06.134043+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.134043+0000', 'last_undegraded': '2024-06-29T14:35:06.134043+0000', 'last_fullsized': '2024-06-29T14:35:06.134043+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.137017+0000', 'last_change': '2024-06-29T14:35:06.137017+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.137017+0000', 'last_undegraded': '2024-06-29T14:35:06.137017+0000', 'last_fullsized': '2024-06-29T14:35:06.137017+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-06-29T14:35:11.988 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.136330+0000', 'last_change': '2024-06-29T14:35:06.136330+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.136330+0000', 'last_undegraded': '2024-06-29T14:35:06.136330+0000', 'last_fullsized': '2024-06-29T14:35:06.136330+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.134973+0000', 'last_change': '2024-06-29T14:35:06.134973+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.134973+0000', 'last_undegraded': '2024-06-29T14:35:06.134973+0000', 'last_fullsized': '2024-06-29T14:35:06.134973+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.139697+0000', 'last_change': '2024-06-29T14:35:06.139697+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.139697+0000', 'last_undegraded': '2024-06-29T14:35:06.139697+0000', 'last_fullsized': '2024-06-29T14:35:06.139697+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.137035+0000', 'last_change': '2024-06-29T14:35:06.137035+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.137035+0000', 'last_undegraded': '2024-06-29T14:35:06.137035+0000', 'last_fullsized': '2024-06-29T14:35:06.137035+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.136457+0000', 'last_change': '2024-06-29T14:35:06.136457+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.136457+0000', 'last_undegraded': '2024-06-29T14:35:06.136457+0000', 'last_fullsized': '2024-06-29T14:35:06.136457+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.134883+0000', 'last_change': '2024-06-29T14:35:06.134883+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.134883+0000', 'last_undegraded': '2024-06-29T14:35:06.134883+0000', 'last_fullsized': '2024-06-29T14:35:06.134883+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-06-29T14:35:11.989 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.142843+0000', 'last_change': '2024-06-29T14:35:06.142843+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.142843+0000', 'last_undegraded': '2024-06-29T14:35:06.142843+0000', 'last_fullsized': '2024-06-29T14:35:06.142843+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.132273+0000', 'last_change': '2024-06-29T14:35:06.132273+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.132273+0000', 'last_undegraded': '2024-06-29T14:35:06.132273+0000', 'last_fullsized': '2024-06-29T14:35:06.132273+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.138048+0000', 'last_change': '2024-06-29T14:35:06.138048+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.138048+0000', 'last_undegraded': '2024-06-29T14:35:06.138048+0000', 'last_fullsized': '2024-06-29T14:35:06.138048+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.143419+0000', 'last_change': '2024-06-29T14:35:06.143419+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.143419+0000', 'last_undegraded': '2024-06-29T14:35:06.143419+0000', 'last_fullsized': '2024-06-29T14:35:06.143419+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.136427+0000', 'last_change': '2024-06-29T14:35:06.136427+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.136427+0000', 'last_undegraded': '2024-06-29T14:35:06.136427+0000', 'last_fullsized': '2024-06-29T14:35:06.136427+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.132166+0000', 'last_change': '2024-06-29T14:35:06.132166+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.132166+0000', 'last_undegraded': '2024-06-29T14:35:06.132166+0000', 'last_fullsized': '2024-06-29T14:35:06.132166+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:35:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.136980+0000', 'last_change': '2024-06-29T14:35:06.136980+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.136980+0000', 'last_undegraded': '2024-06-29T14:35:06.136980+0000', 'last_fullsized': '2024-06-29T14:35:06.136980+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.135625+0000', 'last_change': '2024-06-29T14:35:06.135625+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.135625+0000', 'last_undegraded': '2024-06-29T14:35:06.135625+0000', 'last_fullsized': '2024-06-29T14:35:06.135625+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.132361+0000', 'last_change': '2024-06-29T14:35:06.132361+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.132361+0000', 'last_undegraded': '2024-06-29T14:35:06.132361+0000', 'last_fullsized': '2024-06-29T14:35:06.132361+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.139128+0000', 'last_change': '2024-06-29T14:35:06.139128+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.139128+0000', 'last_undegraded': '2024-06-29T14:35:06.139128+0000', 'last_fullsized': '2024-06-29T14:35:06.139128+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.137093+0000', 'last_change': '2024-06-29T14:35:06.137093+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.137093+0000', 'last_undegraded': '2024-06-29T14:35:06.137093+0000', 'last_fullsized': '2024-06-29T14:35:06.137093+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-06-29T14:35:11.991 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.143970+0000', 'last_change': '2024-06-29T14:35:06.143970+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.143970+0000', 'last_undegraded': '2024-06-29T14:35:06.143970+0000', 'last_fullsized': '2024-06-29T14:35:06.143970+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.138217+0000', 'last_change': '2024-06-29T14:35:06.138217+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.138217+0000', 'last_undegraded': '2024-06-29T14:35:06.138217+0000', 'last_fullsized': '2024-06-29T14:35:06.138217+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.141803+0000', 'last_change': '2024-06-29T14:35:06.141803+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.141803+0000', 'last_undegraded': '2024-06-29T14:35:06.141803+0000', 'last_fullsized': '2024-06-29T14:35:06.141803+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.147888+0000', 'last_change': '2024-06-29T14:35:06.147888+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.147888+0000', 'last_undegraded': '2024-06-29T14:35:06.147888+0000', 'last_fullsized': '2024-06-29T14:35:06.147888+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.132769+0000', 'last_change': '2024-06-29T14:35:06.132769+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.132769+0000', 'last_undegraded': '2024-06-29T14:35:06.132769+0000', 'last_fullsized': '2024-06-29T14:35:06.132769+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.149852+0000', 'last_change': '2024-06-29T14:35:06.149852+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.149852+0000', 'last_undegraded': '2024-06-29T14:35:06.149852+0000', 'last_fullsized': '2024-06-29T14:35:06.149852+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-06-29T14:35:11.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.138649+0000', 'last_change': '2024-06-29T14:35:06.138649+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.138649+0000', 'last_undegraded': '2024-06-29T14:35:06.138649+0000', 'last_fullsized': '2024-06-29T14:35:06.138649+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.133545+0000', 'last_change': '2024-06-29T14:35:06.133545+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.133545+0000', 'last_undegraded': '2024-06-29T14:35:06.133545+0000', 'last_fullsized': '2024-06-29T14:35:06.133545+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.136269+0000', 'last_change': '2024-06-29T14:35:06.136269+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.136269+0000', 'last_undegraded': '2024-06-29T14:35:06.136269+0000', 'last_fullsized': '2024-06-29T14:35:06.136269+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.150280+0000', 'last_change': '2024-06-29T14:35:06.150280+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.150280+0000', 'last_undegraded': '2024-06-29T14:35:06.150280+0000', 'last_fullsized': '2024-06-29T14:35:06.150280+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.139175+0000', 'last_change': '2024-06-29T14:35:06.139175+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.139175+0000', 'last_undegraded': '2024-06-29T14:35:06.139175+0000', 'last_fullsized': '2024-06-29T14:35:06.139175+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.149080+0000', 'last_change': '2024-06-29T14:35:06.149080+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.149080+0000', 'last_undegraded': '2024-06-29T14:35:06.149080+0000', 'last_fullsized': '2024-06-29T14:35:06.149080+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T14:35:11.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.141904+0000', 'last_change': '2024-06-29T14:35:06.141904+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.141904+0000', 'last_undegraded': '2024-06-29T14:35:06.141904+0000', 'last_fullsized': '2024-06-29T14:35:06.141904+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.135183+0000', 'last_change': '2024-06-29T14:35:06.135183+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.135183+0000', 'last_undegraded': '2024-06-29T14:35:06.135183+0000', 'last_fullsized': '2024-06-29T14:35:06.135183+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.139396+0000', 'last_change': '2024-06-29T14:35:06.139396+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.139396+0000', 'last_undegraded': '2024-06-29T14:35:06.139396+0000', 'last_fullsized': '2024-06-29T14:35:06.139396+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.136887+0000', 'last_change': '2024-06-29T14:35:06.136887+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.136887+0000', 'last_undegraded': '2024-06-29T14:35:06.136887+0000', 'last_fullsized': '2024-06-29T14:35:06.136887+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.142459+0000', 'last_change': '2024-06-29T14:35:06.142459+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.142459+0000', 'last_undegraded': '2024-06-29T14:35:06.142459+0000', 'last_fullsized': '2024-06-29T14:35:06.142459+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-06-29T14:35:11.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-06-29T14:35:06.134310+0000', 'last_change': '2024-06-29T14:35:06.134310+0000', 'last_active': '2024-06-29T14:35:06.120894+0000', 'last_peered': '2024-06-29T14:35:06.120894+0000', 'last_clean': '2024-06-29T14:35:06.120894+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T14:35:06.134310+0000', 'last_undegraded': '2024-06-29T14:35:06.134310+0000', 'last_fullsized': '2024-06-29T14:35:06.134310+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'last_clean_scrub_stamp': '2024-06-29T14:35:06.120894+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T14:35:11.995 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-29T14:35:12.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.139+0000 7f6a334a5640 1 -- 172.21.15.59:0/1344394603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a18004040 msgr2=0x7f6a18004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.139+0000 7f6a334a5640 1 --2- 172.21.15.59:0/1344394603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a18004040 0x7f6a18004800 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009b50 tx=0x7f6a18005830 comp rx=0 tx=0).stop 2024-06-29T14:35:12.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.139+0000 7f6a31a42640 1 -- 172.21.15.59:0/1344394603 reap_dead start 2024-06-29T14:35:12.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.139+0000 7f6a334a5640 1 -- 172.21.15.59:0/1344394603 shutdown_connections 2024-06-29T14:35:12.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.139+0000 7f6a334a5640 1 -- 172.21.15.59:0/1344394603 >> 172.21.15.59:0/1344394603 conn(0x7f6a2c0b5610 msgr2=0x7f6a2c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.139+0000 7f6a334a5640 1 -- 172.21.15.59:0/1344394603 shutdown_connections 2024-06-29T14:35:12.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.139+0000 7f6a334a5640 1 -- 172.21.15.59:0/1344394603 wait complete. 2024-06-29T14:35:12.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.141+0000 7f6a334a5640 1 Processor -- start 2024-06-29T14:35:12.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.141+0000 7f6a334a5640 1 -- start start 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.142+0000 7f6a334a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a18004040 0x7f6a2c154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.142+0000 7f6a334a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a2c0ccdd0 0x7f6a2c15ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.142+0000 7f6a334a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a2c15cfb0 0x7f6a2c155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.142+0000 7f6a334a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c162650 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.142+0000 7f6a334a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c1624d0 con 0x7f6a18004040 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.142+0000 7f6a334a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6a2c0b7bd0 con 0x7f6a2c0ccdd0 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a2c15cfb0 0x7f6a2c155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a2c15cfb0 0x7f6a2c155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37942/0 (socket says 172.21.15.59:37942) 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 -- 172.21.15.59:0/2865193302 learned_addr learned my addr 172.21.15.59:0/2865193302 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:12.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a30a40640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a2c0ccdd0 0x7f6a2c15ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31241640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a18004040 0x7f6a2c154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 -- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a2c0ccdd0 msgr2=0x7f6a2c15ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a2c0ccdd0 0x7f6a2c15ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 -- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a18004040 msgr2=0x7f6a2c154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a18004040 0x7f6a2c154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 -- 172.21.15.59:0/2865193302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2c15d3c0 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31241640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a18004040 0x7f6a2c154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a30a40640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a2c0ccdd0 0x7f6a2c15ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a31a42640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a2c15cfb0 0x7f6a2c155920 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c08ccc0 tx=0x7f6a24010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a21ffb640 1 -- 172.21.15.59:0/2865193302 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2400edf0 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a21ffb640 1 -- 172.21.15.59:0/2865193302 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a24015070 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a2c157df0 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.143+0000 7f6a21ffb640 1 -- 172.21.15.59:0/2865193302 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24003d00 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.144+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a2c154e40 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.144+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a2c162c10 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.145+0000 7f6a21ffb640 1 -- 172.21.15.59:0/2865193302 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f6a24004030 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.146+0000 7f6a21ffb640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6a100763f0 0x7f6a10078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.146+0000 7f6a21ffb640 1 -- 172.21.15.59:0/2865193302 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f6a24098650 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.146+0000 7f6a31241640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6a100763f0 0x7f6a10078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.147+0000 7f6a31241640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6a100763f0 0x7f6a10078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6a28000ff0 tx=0x7f6a28006c10 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:12.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.150+0000 7f6a21ffb640 1 -- 172.21.15.59:0/2865193302 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6a2c15d3c0 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.417+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f6a2c0cf570 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.418+0000 7f6a21ffb640 1 -- 172.21.15.59:0/2865193302 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1141 (secure 0 0 0) 0x7f6a24061c70 con 0x7f6a2c15cfb0 2024-06-29T14:35:12.416 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:12.416 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":823,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":71,"num_osds":8,"num_up_osds":8,"osd_up_since":1719670901,"num_in_osds":8,"osd_in_since":1719670892,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":86},{"state_name":"creating+peering","count":11}],"num_pgs":97,"num_pools":3,"num_objects":20,"data_bytes":592126,"bytes_used":331616256,"bytes_avail":772762497024,"bytes_total":773094113280,"inactive_pgs_ratio":0.11340206116437912,"write_bytes_sec":2047,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":29,"btime":"2024-06-29T14:35:10:154758+0000","id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":6333}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-06-29T14:24:34.861407+0000","services":{}},"progress_events":{}} 2024-06-29T14:35:12.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6a100763f0 msgr2=0x7f6a10078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6a100763f0 0x7f6a10078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6a28000ff0 tx=0x7f6a28006c10 comp rx=0 tx=0).stop 2024-06-29T14:35:12.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a2c15cfb0 msgr2=0x7f6a2c155920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a2c15cfb0 0x7f6a2c155920 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c08ccc0 tx=0x7f6a24010f40 comp rx=0 tx=0).stop 2024-06-29T14:35:12.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 shutdown_connections 2024-06-29T14:35:12.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6a100763f0 0x7f6a10078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a2c15cfb0 0x7f6a2c155920 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a2c0ccdd0 0x7f6a2c15ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 --2- 172.21.15.59:0/2865193302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a18004040 0x7f6a2c154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 >> 172.21.15.59:0/2865193302 conn(0x7f6a2c0b5610 msgr2=0x7f6a2c15faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 shutdown_connections 2024-06-29T14:35:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.420+0000 7f6a334a5640 1 -- 172.21.15.59:0/2865193302 wait complete. 2024-06-29T14:35:12.429 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-06-29T14:35:12.429 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:35:12.453 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T14:35:12.455+0000 7fda977d9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:35:12.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.549+0000 7f6ccb810640 1 -- 172.21.15.59:0/2171620899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 msgr2=0x7f6cb000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.549+0000 7f6ccb810640 1 --2- 172.21.15.59:0/2171620899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 0x7f6cb000ba20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00c640 tx=0x7f6cb0005830 comp rx=0 tx=0).stop 2024-06-29T14:35:12.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.549+0000 7f6ccb810640 1 -- 172.21.15.59:0/2171620899 shutdown_connections 2024-06-29T14:35:12.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.549+0000 7f6ccb810640 1 --2- 172.21.15.59:0/2171620899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 0x7f6cb000ba20 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.549+0000 7f6ccb810640 1 -- 172.21.15.59:0/2171620899 >> 172.21.15.59:0/2171620899 conn(0x7f6cc40b6fb0 msgr2=0x7f6cc40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.549+0000 7f6ccb810640 1 -- 172.21.15.59:0/2171620899 shutdown_connections 2024-06-29T14:35:12.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.549+0000 7f6ccb810640 1 -- 172.21.15.59:0/2171620899 wait complete. 2024-06-29T14:35:12.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.551+0000 7f6ccb810640 1 Processor -- start 2024-06-29T14:35:12.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.551+0000 7f6ccb810640 1 -- start start 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.552+0000 7f6ccb810640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 0x7f6cc41f8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.552+0000 7f6ccb810640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc4159d20 0x7f6cc4155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.552+0000 7f6ccb810640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cc40cd1d0 0x7f6cc40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.552+0000 7f6ccb810640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc40b5850 con 0x7f6cb00053a0 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.552+0000 7f6ccb810640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc40b56d0 con 0x7f6cc4159d20 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.552+0000 7f6ccb810640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc40b59d0 con 0x7f6cc40cd1d0 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.552+0000 7f6cc95ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 0x7f6cc41f8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc95ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 0x7f6cc41f8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37960/0 (socket says 172.21.15.59:37960) 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc95ac640 1 -- 172.21.15.59:0/3308694150 learned_addr learned my addr 172.21.15.59:0/3308694150 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc9dad640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cc40cd1d0 0x7f6cc40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc8dab640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc4159d20 0x7f6cc4155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc95ac640 1 -- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cc40cd1d0 msgr2=0x7f6cc40cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc95ac640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cc40cd1d0 0x7f6cc40cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc95ac640 1 -- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc4159d20 msgr2=0x7f6cc4155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc95ac640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc4159d20 0x7f6cc4155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc95ac640 1 -- 172.21.15.59:0/3308694150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc40cb060 con 0x7f6cb00053a0 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cc95ac640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 0x7f6cc41f8f00 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0004e50 tx=0x7f6cc0001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cb9ffb640 1 -- 172.21.15.59:0/3308694150 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0001d40 con 0x7f6cb00053a0 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cb9ffb640 1 -- 172.21.15.59:0/3308694150 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cc0016070 con 0x7f6cb00053a0 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc40cb7c0 con 0x7f6cb00053a0 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6cb9ffb640 1 -- 172.21.15.59:0/3308694150 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0010b20 con 0x7f6cb00053a0 2024-06-29T14:35:12.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.553+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc406efe0 con 0x7f6cb00053a0 2024-06-29T14:35:12.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.554+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc41419b0 con 0x7f6cb00053a0 2024-06-29T14:35:12.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.555+0000 7f6cb9ffb640 1 -- 172.21.15.59:0/3308694150 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f6cc0010cc0 con 0x7f6cb00053a0 2024-06-29T14:35:12.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.556+0000 7f6cb9ffb640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6ca8076420 0x7f6ca8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.556+0000 7f6cb9ffb640 1 -- 172.21.15.59:0/3308694150 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f6cc0098150 con 0x7f6cb00053a0 2024-06-29T14:35:12.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.556+0000 7f6cc8dab640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6ca8076420 0x7f6ca8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.556+0000 7f6cc8dab640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6ca8076420 0x7f6ca8078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6cc406f930 tx=0x7f6cb4005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:12.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.560+0000 7f6cb9ffb640 1 -- 172.21.15.59:0/3308694150 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6cc00616f0 con 0x7f6cb00053a0 2024-06-29T14:35:12.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.746+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 --> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6cc40cb340 con 0x7f6ca8076420 2024-06-29T14:35:12.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.751+0000 7f6cb9ffb640 1 -- 172.21.15.59:0/3308694150 <== mgr.5809 v2:172.21.15.153:6832/2957341575 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251034 (secure 0 0 0) 0x7f6cb405e4b0 con 0x7f6ca8076420 2024-06-29T14:35:12.751 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:12.752 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:35:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6ca8076420 msgr2=0x7f6ca8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6ca8076420 0x7f6ca8078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6cc406f930 tx=0x7f6cb4005f90 comp rx=0 tx=0).stop 2024-06-29T14:35:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 msgr2=0x7f6cc41f8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 0x7f6cc41f8f00 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0004e50 tx=0x7f6cc0001630 comp rx=0 tx=0).stop 2024-06-29T14:35:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 shutdown_connections 2024-06-29T14:35:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f6ca8076420 0x7f6ca8078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cc40cd1d0 0x7f6cc40cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cc4159d20 0x7f6cc4155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 --2- 172.21.15.59:0/3308694150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cb00053a0 0x7f6cc41f8f00 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 >> 172.21.15.59:0/3308694150 conn(0x7f6cc40b6fb0 msgr2=0x7f6cc4140c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 shutdown_connections 2024-06-29T14:35:12.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:12.754+0000 7f6ccb810640 1 -- 172.21.15.59:0/3308694150 wait complete. 2024-06-29T14:35:12.763 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-06-29T14:35:12.033815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592126,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":162,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323844,"kb_used_data":7756,"kb_used_omap":12,"kb_used_meta":315955,"kb_avail":754650876,"statfs":{"total":773094113280,"available":772762497024,"internally_reserved":0,"allocated":7942144,"data_stored":4523812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323538482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1758,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001443"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149385+0000","last_change":"2024-06-29T14:35:07.148232+0000","last_active":"2024-06-29T14:35:08.149385+0000","last_peered":"2024-06-29T14:35:08.149385+0000","last_clean":"2024-06-29T14:35:08.149385+0000","last_became_active":"2024-06-29T14:35:07.147891+0000","last_became_peered":"2024-06-29T14:35:07.147891+0000","last_unstale":"2024-06-29T14:35:08.149385+0000","last_undegraded":"2024-06-29T14:35:08.149385+0000","last_fullsized":"2024-06-29T14:35:08.149385+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:05.408389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311755+0000","last_change":"2024-06-29T14:35:07.151865+0000","last_active":"2024-06-29T14:35:08.311755+0000","last_peered":"2024-06-29T14:35:08.311755+0000","last_clean":"2024-06-29T14:35:08.311755+0000","last_became_active":"2024-06-29T14:35:07.150798+0000","last_became_peered":"2024-06-29T14:35:07.150798+0000","last_unstale":"2024-06-29T14:35:08.311755+0000","last_undegraded":"2024-06-29T14:35:08.311755+0000","last_fullsized":"2024-06-29T14:35:08.311755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:08.314133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310912+0000","last_change":"2024-06-29T14:35:07.163408+0000","last_active":"2024-06-29T14:35:08.310912+0000","last_peered":"2024-06-29T14:35:08.310912+0000","last_clean":"2024-06-29T14:35:08.310912+0000","last_became_active":"2024-06-29T14:35:07.163088+0000","last_became_peered":"2024-06-29T14:35:07.163088+0000","last_unstale":"2024-06-29T14:35:08.310912+0000","last_undegraded":"2024-06-29T14:35:08.310912+0000","last_fullsized":"2024-06-29T14:35:08.310912+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:09.300549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311366+0000","last_change":"2024-06-29T14:35:07.151617+0000","last_active":"2024-06-29T14:35:08.311366+0000","last_peered":"2024-06-29T14:35:08.311366+0000","last_clean":"2024-06-29T14:35:08.311366+0000","last_became_active":"2024-06-29T14:35:07.151303+0000","last_became_peered":"2024-06-29T14:35:07.151303+0000","last_unstale":"2024-06-29T14:35:08.311366+0000","last_undegraded":"2024-06-29T14:35:08.311366+0000","last_fullsized":"2024-06-29T14:35:08.311366+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:09.711945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132969+0000","last_change":"2024-06-29T14:35:06.132969+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132969+0000","last_undegraded":"2024-06-29T14:35:06.132969+0000","last_fullsized":"2024-06-29T14:35:06.132969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.148946+0000","last_change":"2024-06-29T14:35:07.146210+0000","last_active":"2024-06-29T14:35:08.148946+0000","last_peered":"2024-06-29T14:35:08.148946+0000","last_clean":"2024-06-29T14:35:08.148946+0000","last_became_active":"2024-06-29T14:35:07.145876+0000","last_became_peered":"2024-06-29T14:35:07.145876+0000","last_unstale":"2024-06-29T14:35:08.148946+0000","last_undegraded":"2024-06-29T14:35:08.148946+0000","last_fullsized":"2024-06-29T14:35:08.148946+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:55.203613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310346+0000","last_change":"2024-06-29T14:35:07.166641+0000","last_active":"2024-06-29T14:35:08.310346+0000","last_peered":"2024-06-29T14:35:08.310346+0000","last_clean":"2024-06-29T14:35:08.310346+0000","last_became_active":"2024-06-29T14:35:07.164908+0000","last_became_peered":"2024-06-29T14:35:07.164908+0000","last_unstale":"2024-06-29T14:35:08.310346+0000","last_undegraded":"2024-06-29T14:35:08.310346+0000","last_fullsized":"2024-06-29T14:35:08.310346+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:29.981193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310804+0000","last_change":"2024-06-29T14:35:07.175938+0000","last_active":"2024-06-29T14:35:08.310804+0000","last_peered":"2024-06-29T14:35:08.310804+0000","last_clean":"2024-06-29T14:35:08.310804+0000","last_became_active":"2024-06-29T14:35:07.175603+0000","last_became_peered":"2024-06-29T14:35:07.175603+0000","last_unstale":"2024-06-29T14:35:08.310804+0000","last_undegraded":"2024-06-29T14:35:08.310804+0000","last_fullsized":"2024-06-29T14:35:08.310804+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:53:55.448605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311635+0000","last_change":"2024-06-29T14:35:07.175303+0000","last_active":"2024-06-29T14:35:08.311635+0000","last_peered":"2024-06-29T14:35:08.311635+0000","last_clean":"2024-06-29T14:35:08.311635+0000","last_became_active":"2024-06-29T14:35:07.174953+0000","last_became_peered":"2024-06-29T14:35:07.174953+0000","last_unstale":"2024-06-29T14:35:08.311635+0000","last_undegraded":"2024-06-29T14:35:08.311635+0000","last_fullsized":"2024-06-29T14:35:08.311635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:35:38.877477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149358+0000","last_change":"2024-06-29T14:35:07.147352+0000","last_active":"2024-06-29T14:35:08.149358+0000","last_peered":"2024-06-29T14:35:08.149358+0000","last_clean":"2024-06-29T14:35:08.149358+0000","last_became_active":"2024-06-29T14:35:07.147022+0000","last_became_peered":"2024-06-29T14:35:07.147022+0000","last_unstale":"2024-06-29T14:35:08.149358+0000","last_undegraded":"2024-06-29T14:35:08.149358+0000","last_fullsized":"2024-06-29T14:35:08.149358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:20.093276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310504+0000","last_change":"2024-06-29T14:35:07.154352+0000","last_active":"2024-06-29T14:35:08.310504+0000","last_peered":"2024-06-29T14:35:08.310504+0000","last_clean":"2024-06-29T14:35:08.310504+0000","last_became_active":"2024-06-29T14:35:07.153984+0000","last_became_peered":"2024-06-29T14:35:07.153984+0000","last_unstale":"2024-06-29T14:35:08.310504+0000","last_undegraded":"2024-06-29T14:35:08.310504+0000","last_fullsized":"2024-06-29T14:35:08.310504+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:29.317102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311524+0000","last_change":"2024-06-29T14:35:07.174086+0000","last_active":"2024-06-29T14:35:08.311524+0000","last_peered":"2024-06-29T14:35:08.311524+0000","last_clean":"2024-06-29T14:35:08.311524+0000","last_became_active":"2024-06-29T14:35:07.173614+0000","last_became_peered":"2024-06-29T14:35:07.173614+0000","last_unstale":"2024-06-29T14:35:08.311524+0000","last_undegraded":"2024-06-29T14:35:08.311524+0000","last_fullsized":"2024-06-29T14:35:08.311524+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:15.078360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.148925+0000","last_change":"2024-06-29T14:35:07.139620+0000","last_active":"2024-06-29T14:35:08.148925+0000","last_peered":"2024-06-29T14:35:08.148925+0000","last_clean":"2024-06-29T14:35:08.148925+0000","last_became_active":"2024-06-29T14:35:07.139272+0000","last_became_peered":"2024-06-29T14:35:07.139272+0000","last_unstale":"2024-06-29T14:35:08.148925+0000","last_undegraded":"2024-06-29T14:35:08.148925+0000","last_fullsized":"2024-06-29T14:35:08.148925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:41.070905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310884+0000","last_change":"2024-06-29T14:35:07.163462+0000","last_active":"2024-06-29T14:35:08.310884+0000","last_peered":"2024-06-29T14:35:08.310884+0000","last_clean":"2024-06-29T14:35:08.310884+0000","last_became_active":"2024-06-29T14:35:07.163096+0000","last_became_peered":"2024-06-29T14:35:07.163096+0000","last_unstale":"2024-06-29T14:35:08.310884+0000","last_undegraded":"2024-06-29T14:35:08.310884+0000","last_fullsized":"2024-06-29T14:35:08.310884+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:14.228851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311383+0000","last_change":"2024-06-29T14:35:07.152446+0000","last_active":"2024-06-29T14:35:08.311383+0000","last_peered":"2024-06-29T14:35:08.311383+0000","last_clean":"2024-06-29T14:35:08.311383+0000","last_became_active":"2024-06-29T14:35:07.152148+0000","last_became_peered":"2024-06-29T14:35:07.152148+0000","last_unstale":"2024-06-29T14:35:08.311383+0000","last_undegraded":"2024-06-29T14:35:08.311383+0000","last_fullsized":"2024-06-29T14:35:08.311383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:14:13.781939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.148809+0000","last_change":"2024-06-29T14:35:07.147287+0000","last_active":"2024-06-29T14:35:08.148809+0000","last_peered":"2024-06-29T14:35:08.148809+0000","last_clean":"2024-06-29T14:35:08.148809+0000","last_became_active":"2024-06-29T14:35:07.146953+0000","last_became_peered":"2024-06-29T14:35:07.146953+0000","last_unstale":"2024-06-29T14:35:08.148809+0000","last_undegraded":"2024-06-29T14:35:08.148809+0000","last_fullsized":"2024-06-29T14:35:08.148809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:14:41.680505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311510+0000","last_change":"2024-06-29T14:35:07.166454+0000","last_active":"2024-06-29T14:35:08.311510+0000","last_peered":"2024-06-29T14:35:08.311510+0000","last_clean":"2024-06-29T14:35:08.311510+0000","last_became_active":"2024-06-29T14:35:07.166122+0000","last_became_peered":"2024-06-29T14:35:07.166122+0000","last_unstale":"2024-06-29T14:35:08.311510+0000","last_undegraded":"2024-06-29T14:35:08.311510+0000","last_fullsized":"2024-06-29T14:35:08.311510+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:26.977880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311797+0000","last_change":"2024-06-29T14:35:07.153548+0000","last_active":"2024-06-29T14:35:08.311797+0000","last_peered":"2024-06-29T14:35:08.311797+0000","last_clean":"2024-06-29T14:35:08.311797+0000","last_became_active":"2024-06-29T14:35:07.150930+0000","last_became_peered":"2024-06-29T14:35:07.150930+0000","last_unstale":"2024-06-29T14:35:08.311797+0000","last_undegraded":"2024-06-29T14:35:08.311797+0000","last_fullsized":"2024-06-29T14:35:08.311797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:30.544211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311185+0000","last_change":"2024-06-29T14:35:07.162909+0000","last_active":"2024-06-29T14:35:08.311185+0000","last_peered":"2024-06-29T14:35:08.311185+0000","last_clean":"2024-06-29T14:35:08.311185+0000","last_became_active":"2024-06-29T14:35:07.162351+0000","last_became_peered":"2024-06-29T14:35:07.162351+0000","last_unstale":"2024-06-29T14:35:08.311185+0000","last_undegraded":"2024-06-29T14:35:08.311185+0000","last_fullsized":"2024-06-29T14:35:08.311185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:11:42.575135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.135000+0000","last_change":"2024-06-29T14:35:06.135000+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.135000+0000","last_undegraded":"2024-06-29T14:35:06.135000+0000","last_fullsized":"2024-06-29T14:35:06.135000+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.133885+0000","last_change":"2024-06-29T14:35:06.133885+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.133885+0000","last_undegraded":"2024-06-29T14:35:06.133885+0000","last_fullsized":"2024-06-29T14:35:06.133885+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311102+0000","last_change":"2024-06-29T14:35:07.175079+0000","last_active":"2024-06-29T14:35:08.311102+0000","last_peered":"2024-06-29T14:35:08.311102+0000","last_clean":"2024-06-29T14:35:08.311102+0000","last_became_active":"2024-06-29T14:35:07.174741+0000","last_became_peered":"2024-06-29T14:35:07.174741+0000","last_unstale":"2024-06-29T14:35:08.311102+0000","last_undegraded":"2024-06-29T14:35:08.311102+0000","last_fullsized":"2024-06-29T14:35:08.311102+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:44.473786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310128+0000","last_change":"2024-06-29T14:35:07.153237+0000","last_active":"2024-06-29T14:35:08.310128+0000","last_peered":"2024-06-29T14:35:08.310128+0000","last_clean":"2024-06-29T14:35:08.310128+0000","last_became_active":"2024-06-29T14:35:07.152912+0000","last_became_peered":"2024-06-29T14:35:07.152912+0000","last_unstale":"2024-06-29T14:35:08.310128+0000","last_undegraded":"2024-06-29T14:35:08.310128+0000","last_fullsized":"2024-06-29T14:35:08.310128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:48.864630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311490+0000","last_change":"2024-06-29T14:35:07.166071+0000","last_active":"2024-06-29T14:35:08.311490+0000","last_peered":"2024-06-29T14:35:08.311490+0000","last_clean":"2024-06-29T14:35:08.311490+0000","last_became_active":"2024-06-29T14:35:07.165348+0000","last_became_peered":"2024-06-29T14:35:07.165348+0000","last_unstale":"2024-06-29T14:35:08.311490+0000","last_undegraded":"2024-06-29T14:35:08.311490+0000","last_fullsized":"2024-06-29T14:35:08.311490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:37.777093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311643+0000","last_change":"2024-06-29T14:35:07.173325+0000","last_active":"2024-06-29T14:35:08.311643+0000","last_peered":"2024-06-29T14:35:08.311643+0000","last_clean":"2024-06-29T14:35:08.311643+0000","last_became_active":"2024-06-29T14:35:07.172988+0000","last_became_peered":"2024-06-29T14:35:07.172988+0000","last_unstale":"2024-06-29T14:35:08.311643+0000","last_undegraded":"2024-06-29T14:35:08.311643+0000","last_fullsized":"2024-06-29T14:35:08.311643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:53.421947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311323+0000","last_change":"2024-06-29T14:35:07.169265+0000","last_active":"2024-06-29T14:35:08.311323+0000","last_peered":"2024-06-29T14:35:08.311323+0000","last_clean":"2024-06-29T14:35:08.311323+0000","last_became_active":"2024-06-29T14:35:07.157925+0000","last_became_peered":"2024-06-29T14:35:07.157925+0000","last_unstale":"2024-06-29T14:35:08.311323+0000","last_undegraded":"2024-06-29T14:35:08.311323+0000","last_fullsized":"2024-06-29T14:35:08.311323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:49.386835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149306+0000","last_change":"2024-06-29T14:35:07.168806+0000","last_active":"2024-06-29T14:35:08.149306+0000","last_peered":"2024-06-29T14:35:08.149306+0000","last_clean":"2024-06-29T14:35:08.149306+0000","last_became_active":"2024-06-29T14:35:07.168306+0000","last_became_peered":"2024-06-29T14:35:07.168306+0000","last_unstale":"2024-06-29T14:35:08.149306+0000","last_undegraded":"2024-06-29T14:35:08.149306+0000","last_fullsized":"2024-06-29T14:35:08.149306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:03:03.192407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134043+0000","last_change":"2024-06-29T14:35:06.134043+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134043+0000","last_undegraded":"2024-06-29T14:35:06.134043+0000","last_fullsized":"2024-06-29T14:35:06.134043+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310764+0000","last_change":"2024-06-29T14:35:07.165187+0000","last_active":"2024-06-29T14:35:08.310764+0000","last_peered":"2024-06-29T14:35:08.310764+0000","last_clean":"2024-06-29T14:35:08.310764+0000","last_became_active":"2024-06-29T14:35:07.164841+0000","last_became_peered":"2024-06-29T14:35:07.164841+0000","last_unstale":"2024-06-29T14:35:08.310764+0000","last_undegraded":"2024-06-29T14:35:08.310764+0000","last_fullsized":"2024-06-29T14:35:08.310764+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:22:00.473066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311492+0000","last_change":"2024-06-29T14:35:07.153200+0000","last_active":"2024-06-29T14:35:08.311492+0000","last_peered":"2024-06-29T14:35:08.311492+0000","last_clean":"2024-06-29T14:35:08.311492+0000","last_became_active":"2024-06-29T14:35:07.152658+0000","last_became_peered":"2024-06-29T14:35:07.152658+0000","last_unstale":"2024-06-29T14:35:08.311492+0000","last_undegraded":"2024-06-29T14:35:08.311492+0000","last_fullsized":"2024-06-29T14:35:08.311492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:55:30.896310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310532+0000","last_change":"2024-06-29T14:35:07.165606+0000","last_active":"2024-06-29T14:35:08.310532+0000","last_peered":"2024-06-29T14:35:08.310532+0000","last_clean":"2024-06-29T14:35:08.310532+0000","last_became_active":"2024-06-29T14:35:07.165243+0000","last_became_peered":"2024-06-29T14:35:07.165243+0000","last_unstale":"2024-06-29T14:35:08.310532+0000","last_undegraded":"2024-06-29T14:35:08.310532+0000","last_fullsized":"2024-06-29T14:35:08.310532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:19:15.674922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310512+0000","last_change":"2024-06-29T14:35:07.153804+0000","last_active":"2024-06-29T14:35:08.310512+0000","last_peered":"2024-06-29T14:35:08.310512+0000","last_clean":"2024-06-29T14:35:08.310512+0000","last_became_active":"2024-06-29T14:35:07.152882+0000","last_became_peered":"2024-06-29T14:35:07.152882+0000","last_unstale":"2024-06-29T14:35:08.310512+0000","last_undegraded":"2024-06-29T14:35:08.310512+0000","last_fullsized":"2024-06-29T14:35:08.310512+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:11.443951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310453+0000","last_change":"2024-06-29T14:35:07.149792+0000","last_active":"2024-06-29T14:35:08.310453+0000","last_peered":"2024-06-29T14:35:08.310453+0000","last_clean":"2024-06-29T14:35:08.310453+0000","last_became_active":"2024-06-29T14:35:07.149465+0000","last_became_peered":"2024-06-29T14:35:07.149465+0000","last_unstale":"2024-06-29T14:35:08.310453+0000","last_undegraded":"2024-06-29T14:35:08.310453+0000","last_fullsized":"2024-06-29T14:35:08.310453+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:02.662232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.141964+0000","last_change":"2024-06-29T14:35:06.139267+0000","last_active":"2024-06-29T14:35:06.141964+0000","last_peered":"2024-06-29T14:35:06.141964+0000","last_clean":"2024-06-29T14:35:06.141964+0000","last_became_active":"2024-06-29T14:35:06.138932+0000","last_became_peered":"2024-06-29T14:35:06.138932+0000","last_unstale":"2024-06-29T14:35:06.141964+0000","last_undegraded":"2024-06-29T14:35:06.141964+0000","last_fullsized":"2024-06-29T14:35:06.141964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:47.855244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310757+0000","last_change":"2024-06-29T14:35:07.156215+0000","last_active":"2024-06-29T14:35:08.310757+0000","last_peered":"2024-06-29T14:35:08.310757+0000","last_clean":"2024-06-29T14:35:08.310757+0000","last_became_active":"2024-06-29T14:35:07.155486+0000","last_became_peered":"2024-06-29T14:35:07.155486+0000","last_unstale":"2024-06-29T14:35:08.310757+0000","last_undegraded":"2024-06-29T14:35:08.310757+0000","last_fullsized":"2024-06-29T14:35:08.310757+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:26.761362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149359+0000","last_change":"2024-06-29T14:35:06.157899+0000","last_active":"2024-06-29T14:35:08.149359+0000","last_peered":"2024-06-29T14:35:08.149359+0000","last_clean":"2024-06-29T14:35:08.149359+0000","last_became_active":"2024-06-29T14:35:06.157591+0000","last_became_peered":"2024-06-29T14:35:06.157591+0000","last_unstale":"2024-06-29T14:35:08.149359+0000","last_undegraded":"2024-06-29T14:35:08.149359+0000","last_fullsized":"2024-06-29T14:35:08.149359+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:59:55.288599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134883+0000","last_change":"2024-06-29T14:35:06.134883+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134883+0000","last_undegraded":"2024-06-29T14:35:06.134883+0000","last_fullsized":"2024-06-29T14:35:06.134883+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149465+0000","last_change":"2024-06-29T14:35:06.160847+0000","last_active":"2024-06-29T14:35:08.149465+0000","last_peered":"2024-06-29T14:35:08.149465+0000","last_clean":"2024-06-29T14:35:08.149465+0000","last_became_active":"2024-06-29T14:35:06.160558+0000","last_became_peered":"2024-06-29T14:35:06.160558+0000","last_unstale":"2024-06-29T14:35:08.149465+0000","last_undegraded":"2024-06-29T14:35:08.149465+0000","last_fullsized":"2024-06-29T14:35:08.149465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:11.743130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310796+0000","last_change":"2024-06-29T14:35:07.160448+0000","last_active":"2024-06-29T14:35:08.310796+0000","last_peered":"2024-06-29T14:35:08.310796+0000","last_clean":"2024-06-29T14:35:08.310796+0000","last_became_active":"2024-06-29T14:35:07.157426+0000","last_became_peered":"2024-06-29T14:35:07.157426+0000","last_unstale":"2024-06-29T14:35:08.310796+0000","last_undegraded":"2024-06-29T14:35:08.310796+0000","last_fullsized":"2024-06-29T14:35:08.310796+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:57.297610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311059+0000","last_change":"2024-06-29T14:35:06.147172+0000","last_active":"2024-06-29T14:35:08.311059+0000","last_peered":"2024-06-29T14:35:08.311059+0000","last_clean":"2024-06-29T14:35:08.311059+0000","last_became_active":"2024-06-29T14:35:06.146877+0000","last_became_peered":"2024-06-29T14:35:06.146877+0000","last_unstale":"2024-06-29T14:35:08.311059+0000","last_undegraded":"2024-06-29T14:35:08.311059+0000","last_fullsized":"2024-06-29T14:35:08.311059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:56.288224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132273+0000","last_change":"2024-06-29T14:35:06.132273+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132273+0000","last_undegraded":"2024-06-29T14:35:06.132273+0000","last_fullsized":"2024-06-29T14:35:06.132273+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310894+0000","last_change":"2024-06-29T14:35:06.154311+0000","last_active":"2024-06-29T14:35:08.310894+0000","last_peered":"2024-06-29T14:35:08.310894+0000","last_clean":"2024-06-29T14:35:08.310894+0000","last_became_active":"2024-06-29T14:35:06.153161+0000","last_became_peered":"2024-06-29T14:35:06.153161+0000","last_unstale":"2024-06-29T14:35:08.310894+0000","last_undegraded":"2024-06-29T14:35:08.310894+0000","last_fullsized":"2024-06-29T14:35:08.310894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:33:09.220270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311433+0000","last_change":"2024-06-29T14:35:07.158401+0000","last_active":"2024-06-29T14:35:08.311433+0000","last_peered":"2024-06-29T14:35:08.311433+0000","last_clean":"2024-06-29T14:35:08.311433+0000","last_became_active":"2024-06-29T14:35:07.158065+0000","last_became_peered":"2024-06-29T14:35:07.158065+0000","last_unstale":"2024-06-29T14:35:08.311433+0000","last_undegraded":"2024-06-29T14:35:08.311433+0000","last_fullsized":"2024-06-29T14:35:08.311433+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:33:58.907465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144211+0000","last_change":"2024-06-29T14:35:06.142746+0000","last_active":"2024-06-29T14:35:06.144211+0000","last_peered":"2024-06-29T14:35:06.144211+0000","last_clean":"2024-06-29T14:35:06.144211+0000","last_became_active":"2024-06-29T14:35:06.142398+0000","last_became_peered":"2024-06-29T14:35:06.142398+0000","last_unstale":"2024-06-29T14:35:06.144211+0000","last_undegraded":"2024-06-29T14:35:06.144211+0000","last_fullsized":"2024-06-29T14:35:06.144211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:52:09.442403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311770+0000","last_change":"2024-06-29T14:35:07.160762+0000","last_active":"2024-06-29T14:35:08.311770+0000","last_peered":"2024-06-29T14:35:08.311770+0000","last_clean":"2024-06-29T14:35:08.311770+0000","last_became_active":"2024-06-29T14:35:07.157499+0000","last_became_peered":"2024-06-29T14:35:07.157499+0000","last_unstale":"2024-06-29T14:35:08.311770+0000","last_undegraded":"2024-06-29T14:35:08.311770+0000","last_fullsized":"2024-06-29T14:35:08.311770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:23.186337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.142739+0000","last_change":"2024-06-29T14:35:06.141432+0000","last_active":"2024-06-29T14:35:06.142739+0000","last_peered":"2024-06-29T14:35:06.142739+0000","last_clean":"2024-06-29T14:35:06.142739+0000","last_became_active":"2024-06-29T14:35:06.141079+0000","last_became_peered":"2024-06-29T14:35:06.141079+0000","last_unstale":"2024-06-29T14:35:06.142739+0000","last_undegraded":"2024-06-29T14:35:06.142739+0000","last_fullsized":"2024-06-29T14:35:06.142739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:28.307716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311712+0000","last_change":"2024-06-29T14:35:07.166345+0000","last_active":"2024-06-29T14:35:08.311712+0000","last_peered":"2024-06-29T14:35:08.311712+0000","last_clean":"2024-06-29T14:35:08.311712+0000","last_became_active":"2024-06-29T14:35:07.166020+0000","last_became_peered":"2024-06-29T14:35:07.166020+0000","last_unstale":"2024-06-29T14:35:08.311712+0000","last_undegraded":"2024-06-29T14:35:08.311712+0000","last_fullsized":"2024-06-29T14:35:08.311712+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:07.951025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310073+0000","last_change":"2024-06-29T14:35:06.153920+0000","last_active":"2024-06-29T14:35:08.310073+0000","last_peered":"2024-06-29T14:35:08.310073+0000","last_clean":"2024-06-29T14:35:08.310073+0000","last_became_active":"2024-06-29T14:35:06.153047+0000","last_became_peered":"2024-06-29T14:35:06.153047+0000","last_unstale":"2024-06-29T14:35:08.310073+0000","last_undegraded":"2024-06-29T14:35:08.310073+0000","last_fullsized":"2024-06-29T14:35:08.310073+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:13:00.313486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132166+0000","last_change":"2024-06-29T14:35:06.132166+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132166+0000","last_undegraded":"2024-06-29T14:35:06.132166+0000","last_fullsized":"2024-06-29T14:35:06.132166+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311658+0000","last_change":"2024-06-29T14:35:06.145945+0000","last_active":"2024-06-29T14:35:08.311658+0000","last_peered":"2024-06-29T14:35:08.311658+0000","last_clean":"2024-06-29T14:35:08.311658+0000","last_became_active":"2024-06-29T14:35:06.145555+0000","last_became_peered":"2024-06-29T14:35:06.145555+0000","last_unstale":"2024-06-29T14:35:08.311658+0000","last_undegraded":"2024-06-29T14:35:08.311658+0000","last_fullsized":"2024-06-29T14:35:08.311658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:03.780434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310506+0000","last_change":"2024-06-29T14:35:07.149203+0000","last_active":"2024-06-29T14:35:08.310506+0000","last_peered":"2024-06-29T14:35:08.310506+0000","last_clean":"2024-06-29T14:35:08.310506+0000","last_became_active":"2024-06-29T14:35:07.148882+0000","last_became_peered":"2024-06-29T14:35:07.148882+0000","last_unstale":"2024-06-29T14:35:08.310506+0000","last_undegraded":"2024-06-29T14:35:08.310506+0000","last_fullsized":"2024-06-29T14:35:08.310506+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:35:38.877477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.156774+0000","last_change":"2024-06-29T14:35:06.156160+0000","last_active":"2024-06-29T14:35:06.156774+0000","last_peered":"2024-06-29T14:35:06.156774+0000","last_clean":"2024-06-29T14:35:06.156774+0000","last_became_active":"2024-06-29T14:35:06.155823+0000","last_became_peered":"2024-06-29T14:35:06.155823+0000","last_unstale":"2024-06-29T14:35:06.156774+0000","last_undegraded":"2024-06-29T14:35:06.156774+0000","last_fullsized":"2024-06-29T14:35:06.156774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:00.015532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311056+0000","last_change":"2024-06-29T14:35:07.162868+0000","last_active":"2024-06-29T14:35:08.311056+0000","last_peered":"2024-06-29T14:35:08.311056+0000","last_clean":"2024-06-29T14:35:08.311056+0000","last_became_active":"2024-06-29T14:35:07.161415+0000","last_became_peered":"2024-06-29T14:35:07.161415+0000","last_unstale":"2024-06-29T14:35:08.311056+0000","last_undegraded":"2024-06-29T14:35:08.311056+0000","last_fullsized":"2024-06-29T14:35:08.311056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:08.911626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"70'3","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311509+0000","last_change":"2024-06-29T14:35:06.158866+0000","last_active":"2024-06-29T14:35:08.311509+0000","last_peered":"2024-06-29T14:35:08.311509+0000","last_clean":"2024-06-29T14:35:08.311509+0000","last_became_active":"2024-06-29T14:35:06.158615+0000","last_became_peered":"2024-06-29T14:35:06.158615+0000","last_unstale":"2024-06-29T14:35:08.311509+0000","last_undegraded":"2024-06-29T14:35:08.311509+0000","last_fullsized":"2024-06-29T14:35:08.311509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:46.507397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132361+0000","last_change":"2024-06-29T14:35:06.132361+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132361+0000","last_undegraded":"2024-06-29T14:35:06.132361+0000","last_fullsized":"2024-06-29T14:35:06.132361+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"70'6","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311627+0000","last_change":"2024-06-29T14:35:06.148671+0000","last_active":"2024-06-29T14:35:08.311627+0000","last_peered":"2024-06-29T14:35:08.311627+0000","last_clean":"2024-06-29T14:35:08.311627+0000","last_became_active":"2024-06-29T14:35:06.148322+0000","last_became_peered":"2024-06-29T14:35:06.148322+0000","last_unstale":"2024-06-29T14:35:08.311627+0000","last_undegraded":"2024-06-29T14:35:08.311627+0000","last_fullsized":"2024-06-29T14:35:08.311627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:02:47.756994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310958+0000","last_change":"2024-06-29T14:35:07.157004+0000","last_active":"2024-06-29T14:35:08.310958+0000","last_peered":"2024-06-29T14:35:08.310958+0000","last_clean":"2024-06-29T14:35:08.310958+0000","last_became_active":"2024-06-29T14:35:07.156577+0000","last_became_peered":"2024-06-29T14:35:07.156577+0000","last_unstale":"2024-06-29T14:35:08.310958+0000","last_undegraded":"2024-06-29T14:35:08.310958+0000","last_fullsized":"2024-06-29T14:35:08.310958+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:14.623606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311031+0000","last_change":"2024-06-29T14:35:06.148685+0000","last_active":"2024-06-29T14:35:08.311031+0000","last_peered":"2024-06-29T14:35:08.311031+0000","last_clean":"2024-06-29T14:35:08.311031+0000","last_became_active":"2024-06-29T14:35:06.148343+0000","last_became_peered":"2024-06-29T14:35:06.148343+0000","last_unstale":"2024-06-29T14:35:08.311031+0000","last_undegraded":"2024-06-29T14:35:08.311031+0000","last_fullsized":"2024-06-29T14:35:08.311031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:27.880059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311326+0000","last_change":"2024-06-29T14:35:07.161296+0000","last_active":"2024-06-29T14:35:08.311326+0000","last_peered":"2024-06-29T14:35:08.311326+0000","last_clean":"2024-06-29T14:35:08.311326+0000","last_became_active":"2024-06-29T14:35:07.159183+0000","last_became_peered":"2024-06-29T14:35:07.159183+0000","last_unstale":"2024-06-29T14:35:08.311326+0000","last_undegraded":"2024-06-29T14:35:08.311326+0000","last_fullsized":"2024-06-29T14:35:08.311326+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:31.680178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.146783+0000","last_change":"2024-06-29T14:35:06.146043+0000","last_active":"2024-06-29T14:35:06.146783+0000","last_peered":"2024-06-29T14:35:06.146783+0000","last_clean":"2024-06-29T14:35:06.146783+0000","last_became_active":"2024-06-29T14:35:06.145648+0000","last_became_peered":"2024-06-29T14:35:06.145648+0000","last_unstale":"2024-06-29T14:35:06.146783+0000","last_undegraded":"2024-06-29T14:35:06.146783+0000","last_fullsized":"2024-06-29T14:35:06.146783+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:23.312796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311395+0000","last_change":"2024-06-29T14:35:07.170146+0000","last_active":"2024-06-29T14:35:08.311395+0000","last_peered":"2024-06-29T14:35:08.311395+0000","last_clean":"2024-06-29T14:35:08.311395+0000","last_became_active":"2024-06-29T14:35:07.169829+0000","last_became_peered":"2024-06-29T14:35:07.169829+0000","last_unstale":"2024-06-29T14:35:08.311395+0000","last_undegraded":"2024-06-29T14:35:08.311395+0000","last_fullsized":"2024-06-29T14:35:08.311395+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:17.091758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311228+0000","last_change":"2024-06-29T14:35:06.144003+0000","last_active":"2024-06-29T14:35:08.311228+0000","last_peered":"2024-06-29T14:35:08.311228+0000","last_clean":"2024-06-29T14:35:08.311228+0000","last_became_active":"2024-06-29T14:35:06.143542+0000","last_became_peered":"2024-06-29T14:35:06.143542+0000","last_unstale":"2024-06-29T14:35:08.311228+0000","last_undegraded":"2024-06-29T14:35:08.311228+0000","last_fullsized":"2024-06-29T14:35:08.311228+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:59.804285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311246+0000","last_change":"2024-06-29T14:35:07.165765+0000","last_active":"2024-06-29T14:35:08.311246+0000","last_peered":"2024-06-29T14:35:08.311246+0000","last_clean":"2024-06-29T14:35:08.311246+0000","last_became_active":"2024-06-29T14:35:07.165324+0000","last_became_peered":"2024-06-29T14:35:07.165324+0000","last_unstale":"2024-06-29T14:35:08.311246+0000","last_undegraded":"2024-06-29T14:35:08.311246+0000","last_fullsized":"2024-06-29T14:35:08.311246+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:04:13.409311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"70'8","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311158+0000","last_change":"2024-06-29T14:35:06.143641+0000","last_active":"2024-06-29T14:35:08.311158+0000","last_peered":"2024-06-29T14:35:08.311158+0000","last_clean":"2024-06-29T14:35:08.311158+0000","last_became_active":"2024-06-29T14:35:06.143030+0000","last_became_peered":"2024-06-29T14:35:06.143030+0000","last_unstale":"2024-06-29T14:35:08.311158+0000","last_undegraded":"2024-06-29T14:35:08.311158+0000","last_fullsized":"2024-06-29T14:35:08.311158+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:52.412561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311051+0000","last_change":"2024-06-29T14:35:07.152115+0000","last_active":"2024-06-29T14:35:08.311051+0000","last_peered":"2024-06-29T14:35:08.311051+0000","last_clean":"2024-06-29T14:35:08.311051+0000","last_became_active":"2024-06-29T14:35:07.151780+0000","last_became_peered":"2024-06-29T14:35:07.151780+0000","last_unstale":"2024-06-29T14:35:08.311051+0000","last_undegraded":"2024-06-29T14:35:08.311051+0000","last_fullsized":"2024-06-29T14:35:08.311051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:07.537750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311624+0000","last_change":"2024-06-29T14:35:06.155992+0000","last_active":"2024-06-29T14:35:08.311624+0000","last_peered":"2024-06-29T14:35:08.311624+0000","last_clean":"2024-06-29T14:35:08.311624+0000","last_became_active":"2024-06-29T14:35:06.155609+0000","last_became_peered":"2024-06-29T14:35:06.155609+0000","last_unstale":"2024-06-29T14:35:08.311624+0000","last_undegraded":"2024-06-29T14:35:08.311624+0000","last_fullsized":"2024-06-29T14:35:08.311624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:53.681843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.148935+0000","last_change":"2024-06-29T14:35:07.167767+0000","last_active":"2024-06-29T14:35:08.148935+0000","last_peered":"2024-06-29T14:35:08.148935+0000","last_clean":"2024-06-29T14:35:08.148935+0000","last_became_active":"2024-06-29T14:35:07.167452+0000","last_became_peered":"2024-06-29T14:35:07.167452+0000","last_unstale":"2024-06-29T14:35:08.148935+0000","last_undegraded":"2024-06-29T14:35:08.148935+0000","last_fullsized":"2024-06-29T14:35:08.148935+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.921967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310446+0000","last_change":"2024-06-29T14:35:06.145742+0000","last_active":"2024-06-29T14:35:08.310446+0000","last_peered":"2024-06-29T14:35:08.310446+0000","last_clean":"2024-06-29T14:35:08.310446+0000","last_became_active":"2024-06-29T14:35:06.145327+0000","last_became_peered":"2024-06-29T14:35:06.145327+0000","last_unstale":"2024-06-29T14:35:08.310446+0000","last_undegraded":"2024-06-29T14:35:08.310446+0000","last_fullsized":"2024-06-29T14:35:08.310446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:54.528772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.132769+0000","last_change":"2024-06-29T14:35:06.132769+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.132769+0000","last_undegraded":"2024-06-29T14:35:06.132769+0000","last_fullsized":"2024-06-29T14:35:06.132769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.160532+0000","last_change":"2024-06-29T14:35:06.159992+0000","last_active":"2024-06-29T14:35:06.160532+0000","last_peered":"2024-06-29T14:35:06.160532+0000","last_clean":"2024-06-29T14:35:06.160532+0000","last_became_active":"2024-06-29T14:35:06.159710+0000","last_became_peered":"2024-06-29T14:35:06.159710+0000","last_unstale":"2024-06-29T14:35:06.160532+0000","last_undegraded":"2024-06-29T14:35:06.160532+0000","last_fullsized":"2024-06-29T14:35:06.160532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:47.483637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149099+0000","last_change":"2024-06-29T14:35:07.149007+0000","last_active":"2024-06-29T14:35:08.149099+0000","last_peered":"2024-06-29T14:35:08.149099+0000","last_clean":"2024-06-29T14:35:08.149099+0000","last_became_active":"2024-06-29T14:35:07.148691+0000","last_became_peered":"2024-06-29T14:35:07.148691+0000","last_unstale":"2024-06-29T14:35:08.149099+0000","last_undegraded":"2024-06-29T14:35:08.149099+0000","last_fullsized":"2024-06-29T14:35:08.149099+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:43.787786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.149922+0000","last_change":"2024-06-29T14:35:06.149423+0000","last_active":"2024-06-29T14:35:06.149922+0000","last_peered":"2024-06-29T14:35:06.149922+0000","last_clean":"2024-06-29T14:35:06.149922+0000","last_became_active":"2024-06-29T14:35:06.149111+0000","last_became_peered":"2024-06-29T14:35:06.149111+0000","last_unstale":"2024-06-29T14:35:06.149922+0000","last_undegraded":"2024-06-29T14:35:06.149922+0000","last_fullsized":"2024-06-29T14:35:06.149922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:29.397336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311276+0000","last_change":"2024-06-29T14:35:07.149524+0000","last_active":"2024-06-29T14:35:08.311276+0000","last_peered":"2024-06-29T14:35:08.311276+0000","last_clean":"2024-06-29T14:35:08.311276+0000","last_became_active":"2024-06-29T14:35:07.148978+0000","last_became_peered":"2024-06-29T14:35:07.148978+0000","last_unstale":"2024-06-29T14:35:08.311276+0000","last_undegraded":"2024-06-29T14:35:08.311276+0000","last_fullsized":"2024-06-29T14:35:08.311276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:03.215623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149063+0000","last_change":"2024-06-29T14:35:06.159208+0000","last_active":"2024-06-29T14:35:08.149063+0000","last_peered":"2024-06-29T14:35:08.149063+0000","last_clean":"2024-06-29T14:35:08.149063+0000","last_became_active":"2024-06-29T14:35:06.158901+0000","last_became_peered":"2024-06-29T14:35:06.158901+0000","last_unstale":"2024-06-29T14:35:08.149063+0000","last_undegraded":"2024-06-29T14:35:08.149063+0000","last_fullsized":"2024-06-29T14:35:08.149063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:22.290921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.133545+0000","last_change":"2024-06-29T14:35:06.133545+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.133545+0000","last_undegraded":"2024-06-29T14:35:06.133545+0000","last_fullsized":"2024-06-29T14:35:06.133545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311224+0000","last_change":"2024-06-29T14:35:06.142795+0000","last_active":"2024-06-29T14:35:08.311224+0000","last_peered":"2024-06-29T14:35:08.311224+0000","last_clean":"2024-06-29T14:35:08.311224+0000","last_became_active":"2024-06-29T14:35:06.142463+0000","last_became_peered":"2024-06-29T14:35:06.142463+0000","last_unstale":"2024-06-29T14:35:08.311224+0000","last_undegraded":"2024-06-29T14:35:08.311224+0000","last_fullsized":"2024-06-29T14:35:08.311224+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:36.767707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310881+0000","last_change":"2024-06-29T14:35:07.165455+0000","last_active":"2024-06-29T14:35:08.310881+0000","last_peered":"2024-06-29T14:35:08.310881+0000","last_clean":"2024-06-29T14:35:08.310881+0000","last_became_active":"2024-06-29T14:35:07.165053+0000","last_became_peered":"2024-06-29T14:35:07.165053+0000","last_unstale":"2024-06-29T14:35:08.310881+0000","last_undegraded":"2024-06-29T14:35:08.310881+0000","last_fullsized":"2024-06-29T14:35:08.310881+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:54.004799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310571+0000","last_change":"2024-06-29T14:35:06.154839+0000","last_active":"2024-06-29T14:35:08.310571+0000","last_peered":"2024-06-29T14:35:08.310571+0000","last_clean":"2024-06-29T14:35:08.310571+0000","last_became_active":"2024-06-29T14:35:06.154513+0000","last_became_peered":"2024-06-29T14:35:06.154513+0000","last_unstale":"2024-06-29T14:35:08.310571+0000","last_undegraded":"2024-06-29T14:35:08.310571+0000","last_fullsized":"2024-06-29T14:35:08.310571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:26.853277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149536+0000","last_change":"2024-06-29T14:35:07.162073+0000","last_active":"2024-06-29T14:35:08.149536+0000","last_peered":"2024-06-29T14:35:08.149536+0000","last_clean":"2024-06-29T14:35:08.149536+0000","last_became_active":"2024-06-29T14:35:07.161748+0000","last_became_peered":"2024-06-29T14:35:07.161748+0000","last_unstale":"2024-06-29T14:35:08.149536+0000","last_undegraded":"2024-06-29T14:35:08.149536+0000","last_fullsized":"2024-06-29T14:35:08.149536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:26:40.579154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.309898+0000","last_change":"2024-06-29T14:35:06.143667+0000","last_active":"2024-06-29T14:35:08.309898+0000","last_peered":"2024-06-29T14:35:08.309898+0000","last_clean":"2024-06-29T14:35:08.309898+0000","last_became_active":"2024-06-29T14:35:06.143312+0000","last_became_peered":"2024-06-29T14:35:06.143312+0000","last_unstale":"2024-06-29T14:35:08.309898+0000","last_undegraded":"2024-06-29T14:35:08.309898+0000","last_fullsized":"2024-06-29T14:35:08.309898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:20:07.071600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310552+0000","last_change":"2024-06-29T14:35:07.153363+0000","last_active":"2024-06-29T14:35:08.310552+0000","last_peered":"2024-06-29T14:35:08.310552+0000","last_clean":"2024-06-29T14:35:08.310552+0000","last_became_active":"2024-06-29T14:35:07.152829+0000","last_became_peered":"2024-06-29T14:35:07.152829+0000","last_unstale":"2024-06-29T14:35:08.310552+0000","last_undegraded":"2024-06-29T14:35:08.310552+0000","last_fullsized":"2024-06-29T14:35:08.310552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:23.186337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.153279+0000","last_change":"2024-06-29T14:35:06.152663+0000","last_active":"2024-06-29T14:35:06.153279+0000","last_peered":"2024-06-29T14:35:06.153279+0000","last_clean":"2024-06-29T14:35:06.153279+0000","last_became_active":"2024-06-29T14:35:06.152267+0000","last_became_peered":"2024-06-29T14:35:06.152267+0000","last_unstale":"2024-06-29T14:35:06.153279+0000","last_undegraded":"2024-06-29T14:35:06.153279+0000","last_fullsized":"2024-06-29T14:35:06.153279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:15.826985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149100+0000","last_change":"2024-06-29T14:35:07.141322+0000","last_active":"2024-06-29T14:35:08.149100+0000","last_peered":"2024-06-29T14:35:08.149100+0000","last_clean":"2024-06-29T14:35:08.149100+0000","last_became_active":"2024-06-29T14:35:07.140970+0000","last_became_peered":"2024-06-29T14:35:07.140970+0000","last_unstale":"2024-06-29T14:35:08.149100+0000","last_undegraded":"2024-06-29T14:35:08.149100+0000","last_fullsized":"2024-06-29T14:35:08.149100+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:07.467630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311608+0000","last_change":"2024-06-29T14:35:06.155153+0000","last_active":"2024-06-29T14:35:08.311608+0000","last_peered":"2024-06-29T14:35:08.311608+0000","last_clean":"2024-06-29T14:35:08.311608+0000","last_became_active":"2024-06-29T14:35:06.154919+0000","last_became_peered":"2024-06-29T14:35:06.154919+0000","last_unstale":"2024-06-29T14:35:08.311608+0000","last_undegraded":"2024-06-29T14:35:08.311608+0000","last_fullsized":"2024-06-29T14:35:08.311608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:05:28.872856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310970+0000","last_change":"2024-06-29T14:35:07.152311+0000","last_active":"2024-06-29T14:35:08.310970+0000","last_peered":"2024-06-29T14:35:08.310970+0000","last_clean":"2024-06-29T14:35:08.310970+0000","last_became_active":"2024-06-29T14:35:07.151939+0000","last_became_peered":"2024-06-29T14:35:07.151939+0000","last_unstale":"2024-06-29T14:35:08.310970+0000","last_undegraded":"2024-06-29T14:35:08.310970+0000","last_fullsized":"2024-06-29T14:35:08.310970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:25.526535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310688+0000","last_change":"2024-06-29T14:35:06.149483+0000","last_active":"2024-06-29T14:35:08.310688+0000","last_peered":"2024-06-29T14:35:08.310688+0000","last_clean":"2024-06-29T14:35:08.310688+0000","last_became_active":"2024-06-29T14:35:06.149163+0000","last_became_peered":"2024-06-29T14:35:06.149163+0000","last_unstale":"2024-06-29T14:35:08.310688+0000","last_undegraded":"2024-06-29T14:35:08.310688+0000","last_fullsized":"2024-06-29T14:35:08.310688+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:57:29.454929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310576+0000","last_change":"2024-06-29T14:35:07.171505+0000","last_active":"2024-06-29T14:35:08.310576+0000","last_peered":"2024-06-29T14:35:08.310576+0000","last_clean":"2024-06-29T14:35:08.310576+0000","last_became_active":"2024-06-29T14:35:07.171166+0000","last_became_peered":"2024-06-29T14:35:07.171166+0000","last_unstale":"2024-06-29T14:35:08.310576+0000","last_undegraded":"2024-06-29T14:35:08.310576+0000","last_fullsized":"2024-06-29T14:35:08.310576+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:04:55.611535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311072+0000","last_change":"2024-06-29T14:35:06.148884+0000","last_active":"2024-06-29T14:35:08.311072+0000","last_peered":"2024-06-29T14:35:08.311072+0000","last_clean":"2024-06-29T14:35:08.311072+0000","last_became_active":"2024-06-29T14:35:06.148389+0000","last_became_peered":"2024-06-29T14:35:06.148389+0000","last_unstale":"2024-06-29T14:35:08.311072+0000","last_undegraded":"2024-06-29T14:35:08.311072+0000","last_fullsized":"2024-06-29T14:35:08.311072+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:25.968494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310814+0000","last_change":"2024-06-29T14:35:07.157970+0000","last_active":"2024-06-29T14:35:08.310814+0000","last_peered":"2024-06-29T14:35:08.310814+0000","last_clean":"2024-06-29T14:35:08.310814+0000","last_became_active":"2024-06-29T14:35:07.157652+0000","last_became_peered":"2024-06-29T14:35:07.157652+0000","last_unstale":"2024-06-29T14:35:08.310814+0000","last_undegraded":"2024-06-29T14:35:08.310814+0000","last_fullsized":"2024-06-29T14:35:08.310814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:25.977050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311237+0000","last_change":"2024-06-29T14:35:06.152507+0000","last_active":"2024-06-29T14:35:08.311237+0000","last_peered":"2024-06-29T14:35:08.311237+0000","last_clean":"2024-06-29T14:35:08.311237+0000","last_became_active":"2024-06-29T14:35:06.152204+0000","last_became_peered":"2024-06-29T14:35:06.152204+0000","last_unstale":"2024-06-29T14:35:08.311237+0000","last_undegraded":"2024-06-29T14:35:08.311237+0000","last_fullsized":"2024-06-29T14:35:08.311237+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:01.111148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310536+0000","last_change":"2024-06-29T14:35:07.166439+0000","last_active":"2024-06-29T14:35:08.310536+0000","last_peered":"2024-06-29T14:35:08.310536+0000","last_clean":"2024-06-29T14:35:08.310536+0000","last_became_active":"2024-06-29T14:35:07.166106+0000","last_became_peered":"2024-06-29T14:35:07.166106+0000","last_unstale":"2024-06-29T14:35:08.310536+0000","last_undegraded":"2024-06-29T14:35:08.310536+0000","last_fullsized":"2024-06-29T14:35:08.310536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:02:36.725468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-29T14:35:06.144393+0000","last_change":"2024-06-29T14:35:06.141470+0000","last_active":"2024-06-29T14:35:06.144393+0000","last_peered":"2024-06-29T14:35:06.144393+0000","last_clean":"2024-06-29T14:35:06.144393+0000","last_became_active":"2024-06-29T14:35:06.141175+0000","last_became_peered":"2024-06-29T14:35:06.141175+0000","last_unstale":"2024-06-29T14:35:06.144393+0000","last_undegraded":"2024-06-29T14:35:06.144393+0000","last_fullsized":"2024-06-29T14:35:06.144393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:54.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311657+0000","last_change":"2024-06-29T14:35:07.160252+0000","last_active":"2024-06-29T14:35:08.311657+0000","last_peered":"2024-06-29T14:35:08.311657+0000","last_clean":"2024-06-29T14:35:08.311657+0000","last_became_active":"2024-06-29T14:35:07.158059+0000","last_became_peered":"2024-06-29T14:35:07.158059+0000","last_unstale":"2024-06-29T14:35:08.311657+0000","last_undegraded":"2024-06-29T14:35:08.311657+0000","last_fullsized":"2024-06-29T14:35:08.311657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:11.443951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310767+0000","last_change":"2024-06-29T14:35:06.152825+0000","last_active":"2024-06-29T14:35:08.310767+0000","last_peered":"2024-06-29T14:35:08.310767+0000","last_clean":"2024-06-29T14:35:08.310767+0000","last_became_active":"2024-06-29T14:35:06.152469+0000","last_became_peered":"2024-06-29T14:35:06.152469+0000","last_unstale":"2024-06-29T14:35:08.310767+0000","last_undegraded":"2024-06-29T14:35:08.310767+0000","last_fullsized":"2024-06-29T14:35:08.310767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:27:10.436963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-06-29T14:35:06.134310+0000","last_change":"2024-06-29T14:35:06.134310+0000","last_active":"2024-06-29T14:35:06.120894+0000","last_peered":"2024-06-29T14:35:06.120894+0000","last_clean":"2024-06-29T14:35:06.120894+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T14:35:06.134310+0000","last_undegraded":"2024-06-29T14:35:06.134310+0000","last_fullsized":"2024-06-29T14:35:06.134310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"71'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310878+0000","last_change":"2024-06-29T14:35:06.148661+0000","last_active":"2024-06-29T14:35:08.310878+0000","last_peered":"2024-06-29T14:35:08.310878+0000","last_clean":"2024-06-29T14:35:08.310878+0000","last_became_active":"2024-06-29T14:35:06.148307+0000","last_became_peered":"2024-06-29T14:35:06.148307+0000","last_unstale":"2024-06-29T14:35:08.310878+0000","last_undegraded":"2024-06-29T14:35:08.310878+0000","last_fullsized":"2024-06-29T14:35:08.310878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:07:18.680056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'86","reported_seq":261,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310437+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:35:08.310437+0000","last_peered":"2024-06-29T14:35:08.310437+0000","last_clean":"2024-06-29T14:35:08.310437+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:35:08.310437+0000","last_undegraded":"2024-06-29T14:35:08.310437+0000","last_fullsized":"2024-06-29T14:35:08.310437+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":1758,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":3586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542308,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54808,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317032,"statfs":{"total":96636764160,"available":96580640768,"internally_reserved":0,"allocated":1445888,"data_stored":1011393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542308,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542308,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50052,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321788,"statfs":{"total":96636764160,"available":96585510912,"internally_reserved":0,"allocated":843776,"data_stored":420901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542306,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":407042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509605,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":422535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58384,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313456,"statfs":{"total":96636764160,"available":96576978944,"internally_reserved":0,"allocated":856064,"data_stored":422513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509604,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":851968,"data_stored":421061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509606,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1445888,"data_stored":1011325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:35:15.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:35:15.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.903+0000 7fe83cba1640 1 -- 172.21.15.59:0/1821188240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe838144760 msgr2=0x7fe8380734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:15.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.903+0000 7fe83cba1640 1 --2- 172.21.15.59:0/1821188240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe838144760 0x7fe8380734c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe820009830 tx=0x7fe82002a510 comp rx=0 tx=0).stop 2024-06-29T14:35:15.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.903+0000 7fe83cba1640 1 -- 172.21.15.59:0/1821188240 shutdown_connections 2024-06-29T14:35:15.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.903+0000 7fe83cba1640 1 --2- 172.21.15.59:0/1821188240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe838144760 0x7fe8380734c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:15.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.903+0000 7fe83cba1640 1 --2- 172.21.15.59:0/1821188240 >> v2:172.21.15.153:3300/0 conn(0x7fe8380ce6c0 0x7fe83815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:15.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.903+0000 7fe83cba1640 1 -- 172.21.15.59:0/1821188240 >> 172.21.15.59:0/1821188240 conn(0x7fe83806cef0 msgr2=0x7fe83806d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:15.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.903+0000 7fe83cba1640 1 -- 172.21.15.59:0/1821188240 shutdown_connections 2024-06-29T14:35:15.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.903+0000 7fe83cba1640 1 -- 172.21.15.59:0/1821188240 wait complete. 2024-06-29T14:35:15.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.905+0000 7fe83cba1640 1 Processor -- start 2024-06-29T14:35:15.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.905+0000 7fe83cba1640 1 -- start start 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe83cba1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe838073a00 0x7fe838136680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe83cba1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8380ce6c0 0x7fe8381e35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe83cba1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe838144760 0x7fe8381e3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe83cba1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe8380b4880 con 0x7fe838144760 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe83cba1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe8380b4700 con 0x7fe838073a00 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe83cba1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe8380b4a00 con 0x7fe8380ce6c0 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe836d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe838144760 0x7fe8381e3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe836d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe838144760 0x7fe8381e3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34864/0 (socket says 172.21.15.59:34864) 2024-06-29T14:35:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe836d9d640 1 -- 172.21.15.59:0/2806658650 learned_addr learned my addr 172.21.15.59:0/2806658650 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe83659c640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe838073a00 0x7fe838136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.906+0000 7fe835d9b640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8380ce6c0 0x7fe8381e35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe836d9d640 1 -- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8380ce6c0 msgr2=0x7fe8381e35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe836d9d640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8380ce6c0 0x7fe8381e35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe836d9d640 1 -- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe838073a00 msgr2=0x7fe838136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe836d9d640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe838073a00 0x7fe838136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe836d9d640 1 -- 172.21.15.59:0/2806658650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8381e42f0 con 0x7fe838144760 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe835d9b640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8380ce6c0 0x7fe8381e35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe836d9d640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe838144760 0x7fe8381e3ae0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe83000c730 tx=0x7fe83000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe82effd640 1 -- 172.21.15.59:0/2806658650 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830003b70 con 0x7fe838144760 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe82effd640 1 -- 172.21.15.59:0/2806658650 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe830003d10 con 0x7fe838144760 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe82effd640 1 -- 172.21.15.59:0/2806658650 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830005330 con 0x7fe838144760 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8381e4b10 con 0x7fe838144760 2024-06-29T14:35:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.907+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8381e50d0 con 0x7fe838144760 2024-06-29T14:35:15.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.908+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe838073ff0 con 0x7fe838144760 2024-06-29T14:35:15.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.909+0000 7fe82effd640 1 -- 172.21.15.59:0/2806658650 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fe8300056e0 con 0x7fe838144760 2024-06-29T14:35:15.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.910+0000 7fe82effd640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe818076420 0x7fe818078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:15.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.910+0000 7fe82effd640 1 -- 172.21.15.59:0/2806658650 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7fe830097880 con 0x7fe838144760 2024-06-29T14:35:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.910+0000 7fe83659c640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe818076420 0x7fe818078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.911+0000 7fe83659c640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe818076420 0x7fe818078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe828009530 tx=0x7fe828009260 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:15.914+0000 7fe82effd640 1 -- 172.21.15.59:0/2806658650 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe838073ff0 con 0x7fe838144760 2024-06-29T14:35:16.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.107+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 --> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8381f2fb0 con 0x7fe818076420 2024-06-29T14:35:16.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.111+0000 7fe82effd640 1 -- 172.21.15.59:0/2806658650 <== mgr.5809 v2:172.21.15.153:6832/2957341575 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251953 (secure 0 0 0) 0x7fe8381f2fb0 con 0x7fe818076420 2024-06-29T14:35:16.112 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:16.113 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe818076420 msgr2=0x7fe818078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe818076420 0x7fe818078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe828009530 tx=0x7fe828009260 comp rx=0 tx=0).stop 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe838144760 msgr2=0x7fe8381e3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe838144760 0x7fe8381e3ae0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe83000c730 tx=0x7fe83000cbf0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 shutdown_connections 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe818076420 0x7fe818078930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe838144760 0x7fe8381e3ae0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8380ce6c0 0x7fe8381e35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 --2- 172.21.15.59:0/2806658650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe838073a00 0x7fe838136680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 >> 172.21.15.59:0/2806658650 conn(0x7fe83806cef0 msgr2=0x7fe83815d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 shutdown_connections 2024-06-29T14:35:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.115+0000 7fe83cba1640 1 -- 172.21.15.59:0/2806658650 wait complete. 2024-06-29T14:35:16.124 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-06-29T14:35:16.034989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":170,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323904,"kb_used_data":7816,"kb_used_omap":12,"kb_used_meta":315955,"kb_avail":754650816,"statfs":{"total":773094113280,"available":772762435584,"internally_reserved":0,"allocated":8003584,"data_stored":4552764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":323538482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.903435"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149385+0000","last_change":"2024-06-29T14:35:07.148232+0000","last_active":"2024-06-29T14:35:08.149385+0000","last_peered":"2024-06-29T14:35:08.149385+0000","last_clean":"2024-06-29T14:35:08.149385+0000","last_became_active":"2024-06-29T14:35:07.147891+0000","last_became_peered":"2024-06-29T14:35:07.147891+0000","last_unstale":"2024-06-29T14:35:08.149385+0000","last_undegraded":"2024-06-29T14:35:08.149385+0000","last_fullsized":"2024-06-29T14:35:08.149385+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:05.408389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311755+0000","last_change":"2024-06-29T14:35:07.151865+0000","last_active":"2024-06-29T14:35:08.311755+0000","last_peered":"2024-06-29T14:35:08.311755+0000","last_clean":"2024-06-29T14:35:08.311755+0000","last_became_active":"2024-06-29T14:35:07.150798+0000","last_became_peered":"2024-06-29T14:35:07.150798+0000","last_unstale":"2024-06-29T14:35:08.311755+0000","last_undegraded":"2024-06-29T14:35:08.311755+0000","last_fullsized":"2024-06-29T14:35:08.311755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:08.314133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310912+0000","last_change":"2024-06-29T14:35:07.163408+0000","last_active":"2024-06-29T14:35:08.310912+0000","last_peered":"2024-06-29T14:35:08.310912+0000","last_clean":"2024-06-29T14:35:08.310912+0000","last_became_active":"2024-06-29T14:35:07.163088+0000","last_became_peered":"2024-06-29T14:35:07.163088+0000","last_unstale":"2024-06-29T14:35:08.310912+0000","last_undegraded":"2024-06-29T14:35:08.310912+0000","last_fullsized":"2024-06-29T14:35:08.310912+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:09.300549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311366+0000","last_change":"2024-06-29T14:35:07.151617+0000","last_active":"2024-06-29T14:35:08.311366+0000","last_peered":"2024-06-29T14:35:08.311366+0000","last_clean":"2024-06-29T14:35:08.311366+0000","last_became_active":"2024-06-29T14:35:07.151303+0000","last_became_peered":"2024-06-29T14:35:07.151303+0000","last_unstale":"2024-06-29T14:35:08.311366+0000","last_undegraded":"2024-06-29T14:35:08.311366+0000","last_fullsized":"2024-06-29T14:35:08.311366+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:09.711945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150279+0000","last_change":"2024-06-29T14:35:07.145786+0000","last_active":"2024-06-29T14:35:08.150279+0000","last_peered":"2024-06-29T14:35:08.150279+0000","last_clean":"2024-06-29T14:35:08.150279+0000","last_became_active":"2024-06-29T14:35:07.145454+0000","last_became_peered":"2024-06-29T14:35:07.145454+0000","last_unstale":"2024-06-29T14:35:08.150279+0000","last_undegraded":"2024-06-29T14:35:08.150279+0000","last_fullsized":"2024-06-29T14:35:08.150279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:43.568137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.148946+0000","last_change":"2024-06-29T14:35:07.146210+0000","last_active":"2024-06-29T14:35:08.148946+0000","last_peered":"2024-06-29T14:35:08.148946+0000","last_clean":"2024-06-29T14:35:08.148946+0000","last_became_active":"2024-06-29T14:35:07.145876+0000","last_became_peered":"2024-06-29T14:35:07.145876+0000","last_unstale":"2024-06-29T14:35:08.148946+0000","last_undegraded":"2024-06-29T14:35:08.148946+0000","last_fullsized":"2024-06-29T14:35:08.148946+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:55.203613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310346+0000","last_change":"2024-06-29T14:35:07.166641+0000","last_active":"2024-06-29T14:35:08.310346+0000","last_peered":"2024-06-29T14:35:08.310346+0000","last_clean":"2024-06-29T14:35:08.310346+0000","last_became_active":"2024-06-29T14:35:07.164908+0000","last_became_peered":"2024-06-29T14:35:07.164908+0000","last_unstale":"2024-06-29T14:35:08.310346+0000","last_undegraded":"2024-06-29T14:35:08.310346+0000","last_fullsized":"2024-06-29T14:35:08.310346+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:29.981193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310804+0000","last_change":"2024-06-29T14:35:07.175938+0000","last_active":"2024-06-29T14:35:08.310804+0000","last_peered":"2024-06-29T14:35:08.310804+0000","last_clean":"2024-06-29T14:35:08.310804+0000","last_became_active":"2024-06-29T14:35:07.175603+0000","last_became_peered":"2024-06-29T14:35:07.175603+0000","last_unstale":"2024-06-29T14:35:08.310804+0000","last_undegraded":"2024-06-29T14:35:08.310804+0000","last_fullsized":"2024-06-29T14:35:08.310804+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:53:55.448605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311635+0000","last_change":"2024-06-29T14:35:07.175303+0000","last_active":"2024-06-29T14:35:08.311635+0000","last_peered":"2024-06-29T14:35:08.311635+0000","last_clean":"2024-06-29T14:35:08.311635+0000","last_became_active":"2024-06-29T14:35:07.174953+0000","last_became_peered":"2024-06-29T14:35:07.174953+0000","last_unstale":"2024-06-29T14:35:08.311635+0000","last_undegraded":"2024-06-29T14:35:08.311635+0000","last_fullsized":"2024-06-29T14:35:08.311635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:35:38.877477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149358+0000","last_change":"2024-06-29T14:35:07.147352+0000","last_active":"2024-06-29T14:35:08.149358+0000","last_peered":"2024-06-29T14:35:08.149358+0000","last_clean":"2024-06-29T14:35:08.149358+0000","last_became_active":"2024-06-29T14:35:07.147022+0000","last_became_peered":"2024-06-29T14:35:07.147022+0000","last_unstale":"2024-06-29T14:35:08.149358+0000","last_undegraded":"2024-06-29T14:35:08.149358+0000","last_fullsized":"2024-06-29T14:35:08.149358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:20.093276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310504+0000","last_change":"2024-06-29T14:35:07.154352+0000","last_active":"2024-06-29T14:35:08.310504+0000","last_peered":"2024-06-29T14:35:08.310504+0000","last_clean":"2024-06-29T14:35:08.310504+0000","last_became_active":"2024-06-29T14:35:07.153984+0000","last_became_peered":"2024-06-29T14:35:07.153984+0000","last_unstale":"2024-06-29T14:35:08.310504+0000","last_undegraded":"2024-06-29T14:35:08.310504+0000","last_fullsized":"2024-06-29T14:35:08.310504+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:29.317102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311524+0000","last_change":"2024-06-29T14:35:07.174086+0000","last_active":"2024-06-29T14:35:08.311524+0000","last_peered":"2024-06-29T14:35:08.311524+0000","last_clean":"2024-06-29T14:35:08.311524+0000","last_became_active":"2024-06-29T14:35:07.173614+0000","last_became_peered":"2024-06-29T14:35:07.173614+0000","last_unstale":"2024-06-29T14:35:08.311524+0000","last_undegraded":"2024-06-29T14:35:08.311524+0000","last_fullsized":"2024-06-29T14:35:08.311524+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:15.078360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.148925+0000","last_change":"2024-06-29T14:35:07.139620+0000","last_active":"2024-06-29T14:35:08.148925+0000","last_peered":"2024-06-29T14:35:08.148925+0000","last_clean":"2024-06-29T14:35:08.148925+0000","last_became_active":"2024-06-29T14:35:07.139272+0000","last_became_peered":"2024-06-29T14:35:07.139272+0000","last_unstale":"2024-06-29T14:35:08.148925+0000","last_undegraded":"2024-06-29T14:35:08.148925+0000","last_fullsized":"2024-06-29T14:35:08.148925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:41.070905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310884+0000","last_change":"2024-06-29T14:35:07.163462+0000","last_active":"2024-06-29T14:35:08.310884+0000","last_peered":"2024-06-29T14:35:08.310884+0000","last_clean":"2024-06-29T14:35:08.310884+0000","last_became_active":"2024-06-29T14:35:07.163096+0000","last_became_peered":"2024-06-29T14:35:07.163096+0000","last_unstale":"2024-06-29T14:35:08.310884+0000","last_undegraded":"2024-06-29T14:35:08.310884+0000","last_fullsized":"2024-06-29T14:35:08.310884+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:14.228851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311383+0000","last_change":"2024-06-29T14:35:07.152446+0000","last_active":"2024-06-29T14:35:08.311383+0000","last_peered":"2024-06-29T14:35:08.311383+0000","last_clean":"2024-06-29T14:35:08.311383+0000","last_became_active":"2024-06-29T14:35:07.152148+0000","last_became_peered":"2024-06-29T14:35:07.152148+0000","last_unstale":"2024-06-29T14:35:08.311383+0000","last_undegraded":"2024-06-29T14:35:08.311383+0000","last_fullsized":"2024-06-29T14:35:08.311383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:14:13.781939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.148809+0000","last_change":"2024-06-29T14:35:07.147287+0000","last_active":"2024-06-29T14:35:08.148809+0000","last_peered":"2024-06-29T14:35:08.148809+0000","last_clean":"2024-06-29T14:35:08.148809+0000","last_became_active":"2024-06-29T14:35:07.146953+0000","last_became_peered":"2024-06-29T14:35:07.146953+0000","last_unstale":"2024-06-29T14:35:08.148809+0000","last_undegraded":"2024-06-29T14:35:08.148809+0000","last_fullsized":"2024-06-29T14:35:08.148809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:14:41.680505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311510+0000","last_change":"2024-06-29T14:35:07.166454+0000","last_active":"2024-06-29T14:35:08.311510+0000","last_peered":"2024-06-29T14:35:08.311510+0000","last_clean":"2024-06-29T14:35:08.311510+0000","last_became_active":"2024-06-29T14:35:07.166122+0000","last_became_peered":"2024-06-29T14:35:07.166122+0000","last_unstale":"2024-06-29T14:35:08.311510+0000","last_undegraded":"2024-06-29T14:35:08.311510+0000","last_fullsized":"2024-06-29T14:35:08.311510+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:26.977880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311797+0000","last_change":"2024-06-29T14:35:07.153548+0000","last_active":"2024-06-29T14:35:08.311797+0000","last_peered":"2024-06-29T14:35:08.311797+0000","last_clean":"2024-06-29T14:35:08.311797+0000","last_became_active":"2024-06-29T14:35:07.150930+0000","last_became_peered":"2024-06-29T14:35:07.150930+0000","last_unstale":"2024-06-29T14:35:08.311797+0000","last_undegraded":"2024-06-29T14:35:08.311797+0000","last_fullsized":"2024-06-29T14:35:08.311797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:30.544211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311185+0000","last_change":"2024-06-29T14:35:07.162909+0000","last_active":"2024-06-29T14:35:08.311185+0000","last_peered":"2024-06-29T14:35:08.311185+0000","last_clean":"2024-06-29T14:35:08.311185+0000","last_became_active":"2024-06-29T14:35:07.162351+0000","last_became_peered":"2024-06-29T14:35:07.162351+0000","last_unstale":"2024-06-29T14:35:08.311185+0000","last_undegraded":"2024-06-29T14:35:08.311185+0000","last_fullsized":"2024-06-29T14:35:08.311185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:11:42.575135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150197+0000","last_change":"2024-06-29T14:35:07.138253+0000","last_active":"2024-06-29T14:35:08.150197+0000","last_peered":"2024-06-29T14:35:08.150197+0000","last_clean":"2024-06-29T14:35:08.150197+0000","last_became_active":"2024-06-29T14:35:07.137886+0000","last_became_peered":"2024-06-29T14:35:07.137886+0000","last_unstale":"2024-06-29T14:35:08.150197+0000","last_undegraded":"2024-06-29T14:35:08.150197+0000","last_fullsized":"2024-06-29T14:35:08.150197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:55.243000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150835+0000","last_change":"2024-06-29T14:35:07.145656+0000","last_active":"2024-06-29T14:35:08.150835+0000","last_peered":"2024-06-29T14:35:08.150835+0000","last_clean":"2024-06-29T14:35:08.150835+0000","last_became_active":"2024-06-29T14:35:07.145362+0000","last_became_peered":"2024-06-29T14:35:07.145362+0000","last_unstale":"2024-06-29T14:35:08.150835+0000","last_undegraded":"2024-06-29T14:35:08.150835+0000","last_fullsized":"2024-06-29T14:35:08.150835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:52:10.451789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311102+0000","last_change":"2024-06-29T14:35:07.175079+0000","last_active":"2024-06-29T14:35:08.311102+0000","last_peered":"2024-06-29T14:35:08.311102+0000","last_clean":"2024-06-29T14:35:08.311102+0000","last_became_active":"2024-06-29T14:35:07.174741+0000","last_became_peered":"2024-06-29T14:35:07.174741+0000","last_unstale":"2024-06-29T14:35:08.311102+0000","last_undegraded":"2024-06-29T14:35:08.311102+0000","last_fullsized":"2024-06-29T14:35:08.311102+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:44.473786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310128+0000","last_change":"2024-06-29T14:35:07.153237+0000","last_active":"2024-06-29T14:35:08.310128+0000","last_peered":"2024-06-29T14:35:08.310128+0000","last_clean":"2024-06-29T14:35:08.310128+0000","last_became_active":"2024-06-29T14:35:07.152912+0000","last_became_peered":"2024-06-29T14:35:07.152912+0000","last_unstale":"2024-06-29T14:35:08.310128+0000","last_undegraded":"2024-06-29T14:35:08.310128+0000","last_fullsized":"2024-06-29T14:35:08.310128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:48.864630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311490+0000","last_change":"2024-06-29T14:35:07.166071+0000","last_active":"2024-06-29T14:35:08.311490+0000","last_peered":"2024-06-29T14:35:08.311490+0000","last_clean":"2024-06-29T14:35:08.311490+0000","last_became_active":"2024-06-29T14:35:07.165348+0000","last_became_peered":"2024-06-29T14:35:07.165348+0000","last_unstale":"2024-06-29T14:35:08.311490+0000","last_undegraded":"2024-06-29T14:35:08.311490+0000","last_fullsized":"2024-06-29T14:35:08.311490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:37.777093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311643+0000","last_change":"2024-06-29T14:35:07.173325+0000","last_active":"2024-06-29T14:35:08.311643+0000","last_peered":"2024-06-29T14:35:08.311643+0000","last_clean":"2024-06-29T14:35:08.311643+0000","last_became_active":"2024-06-29T14:35:07.172988+0000","last_became_peered":"2024-06-29T14:35:07.172988+0000","last_unstale":"2024-06-29T14:35:08.311643+0000","last_undegraded":"2024-06-29T14:35:08.311643+0000","last_fullsized":"2024-06-29T14:35:08.311643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:53.421947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311323+0000","last_change":"2024-06-29T14:35:07.169265+0000","last_active":"2024-06-29T14:35:08.311323+0000","last_peered":"2024-06-29T14:35:08.311323+0000","last_clean":"2024-06-29T14:35:08.311323+0000","last_became_active":"2024-06-29T14:35:07.157925+0000","last_became_peered":"2024-06-29T14:35:07.157925+0000","last_unstale":"2024-06-29T14:35:08.311323+0000","last_undegraded":"2024-06-29T14:35:08.311323+0000","last_fullsized":"2024-06-29T14:35:08.311323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:49.386835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149306+0000","last_change":"2024-06-29T14:35:07.168806+0000","last_active":"2024-06-29T14:35:08.149306+0000","last_peered":"2024-06-29T14:35:08.149306+0000","last_clean":"2024-06-29T14:35:08.149306+0000","last_became_active":"2024-06-29T14:35:07.168306+0000","last_became_peered":"2024-06-29T14:35:07.168306+0000","last_unstale":"2024-06-29T14:35:08.149306+0000","last_undegraded":"2024-06-29T14:35:08.149306+0000","last_fullsized":"2024-06-29T14:35:08.149306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:03:03.192407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149919+0000","last_change":"2024-06-29T14:35:07.143564+0000","last_active":"2024-06-29T14:35:08.149919+0000","last_peered":"2024-06-29T14:35:08.149919+0000","last_clean":"2024-06-29T14:35:08.149919+0000","last_became_active":"2024-06-29T14:35:07.143227+0000","last_became_peered":"2024-06-29T14:35:07.143227+0000","last_unstale":"2024-06-29T14:35:08.149919+0000","last_undegraded":"2024-06-29T14:35:08.149919+0000","last_fullsized":"2024-06-29T14:35:08.149919+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:08.363103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310764+0000","last_change":"2024-06-29T14:35:07.165187+0000","last_active":"2024-06-29T14:35:08.310764+0000","last_peered":"2024-06-29T14:35:08.310764+0000","last_clean":"2024-06-29T14:35:08.310764+0000","last_became_active":"2024-06-29T14:35:07.164841+0000","last_became_peered":"2024-06-29T14:35:07.164841+0000","last_unstale":"2024-06-29T14:35:08.310764+0000","last_undegraded":"2024-06-29T14:35:08.310764+0000","last_fullsized":"2024-06-29T14:35:08.310764+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:22:00.473066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311492+0000","last_change":"2024-06-29T14:35:07.153200+0000","last_active":"2024-06-29T14:35:08.311492+0000","last_peered":"2024-06-29T14:35:08.311492+0000","last_clean":"2024-06-29T14:35:08.311492+0000","last_became_active":"2024-06-29T14:35:07.152658+0000","last_became_peered":"2024-06-29T14:35:07.152658+0000","last_unstale":"2024-06-29T14:35:08.311492+0000","last_undegraded":"2024-06-29T14:35:08.311492+0000","last_fullsized":"2024-06-29T14:35:08.311492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:55:30.896310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310532+0000","last_change":"2024-06-29T14:35:07.165606+0000","last_active":"2024-06-29T14:35:08.310532+0000","last_peered":"2024-06-29T14:35:08.310532+0000","last_clean":"2024-06-29T14:35:08.310532+0000","last_became_active":"2024-06-29T14:35:07.165243+0000","last_became_peered":"2024-06-29T14:35:07.165243+0000","last_unstale":"2024-06-29T14:35:08.310532+0000","last_undegraded":"2024-06-29T14:35:08.310532+0000","last_fullsized":"2024-06-29T14:35:08.310532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:19:15.674922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310512+0000","last_change":"2024-06-29T14:35:07.153804+0000","last_active":"2024-06-29T14:35:08.310512+0000","last_peered":"2024-06-29T14:35:08.310512+0000","last_clean":"2024-06-29T14:35:08.310512+0000","last_became_active":"2024-06-29T14:35:07.152882+0000","last_became_peered":"2024-06-29T14:35:07.152882+0000","last_unstale":"2024-06-29T14:35:08.310512+0000","last_undegraded":"2024-06-29T14:35:08.310512+0000","last_fullsized":"2024-06-29T14:35:08.310512+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:11.443951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310453+0000","last_change":"2024-06-29T14:35:07.149792+0000","last_active":"2024-06-29T14:35:08.310453+0000","last_peered":"2024-06-29T14:35:08.310453+0000","last_clean":"2024-06-29T14:35:08.310453+0000","last_became_active":"2024-06-29T14:35:07.149465+0000","last_became_peered":"2024-06-29T14:35:07.149465+0000","last_unstale":"2024-06-29T14:35:08.310453+0000","last_undegraded":"2024-06-29T14:35:08.310453+0000","last_fullsized":"2024-06-29T14:35:08.310453+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:02.662232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"70'4","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150352+0000","last_change":"2024-06-29T14:35:06.139267+0000","last_active":"2024-06-29T14:35:08.150352+0000","last_peered":"2024-06-29T14:35:08.150352+0000","last_clean":"2024-06-29T14:35:08.150352+0000","last_became_active":"2024-06-29T14:35:06.138932+0000","last_became_peered":"2024-06-29T14:35:06.138932+0000","last_unstale":"2024-06-29T14:35:08.150352+0000","last_undegraded":"2024-06-29T14:35:08.150352+0000","last_fullsized":"2024-06-29T14:35:08.150352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:17:15.758595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310757+0000","last_change":"2024-06-29T14:35:07.156215+0000","last_active":"2024-06-29T14:35:08.310757+0000","last_peered":"2024-06-29T14:35:08.310757+0000","last_clean":"2024-06-29T14:35:08.310757+0000","last_became_active":"2024-06-29T14:35:07.155486+0000","last_became_peered":"2024-06-29T14:35:07.155486+0000","last_unstale":"2024-06-29T14:35:08.310757+0000","last_undegraded":"2024-06-29T14:35:08.310757+0000","last_fullsized":"2024-06-29T14:35:08.310757+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:26.761362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149359+0000","last_change":"2024-06-29T14:35:06.157899+0000","last_active":"2024-06-29T14:35:08.149359+0000","last_peered":"2024-06-29T14:35:08.149359+0000","last_clean":"2024-06-29T14:35:08.149359+0000","last_became_active":"2024-06-29T14:35:06.157591+0000","last_became_peered":"2024-06-29T14:35:06.157591+0000","last_unstale":"2024-06-29T14:35:08.149359+0000","last_undegraded":"2024-06-29T14:35:08.149359+0000","last_fullsized":"2024-06-29T14:35:08.149359+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:59:55.288599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150965+0000","last_change":"2024-06-29T14:35:07.146536+0000","last_active":"2024-06-29T14:35:08.150965+0000","last_peered":"2024-06-29T14:35:08.150965+0000","last_clean":"2024-06-29T14:35:08.150965+0000","last_became_active":"2024-06-29T14:35:07.146224+0000","last_became_peered":"2024-06-29T14:35:07.146224+0000","last_unstale":"2024-06-29T14:35:08.150965+0000","last_undegraded":"2024-06-29T14:35:08.150965+0000","last_fullsized":"2024-06-29T14:35:08.150965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:01.024918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149465+0000","last_change":"2024-06-29T14:35:06.160847+0000","last_active":"2024-06-29T14:35:08.149465+0000","last_peered":"2024-06-29T14:35:08.149465+0000","last_clean":"2024-06-29T14:35:08.149465+0000","last_became_active":"2024-06-29T14:35:06.160558+0000","last_became_peered":"2024-06-29T14:35:06.160558+0000","last_unstale":"2024-06-29T14:35:08.149465+0000","last_undegraded":"2024-06-29T14:35:08.149465+0000","last_fullsized":"2024-06-29T14:35:08.149465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:11.743130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310796+0000","last_change":"2024-06-29T14:35:07.160448+0000","last_active":"2024-06-29T14:35:08.310796+0000","last_peered":"2024-06-29T14:35:08.310796+0000","last_clean":"2024-06-29T14:35:08.310796+0000","last_became_active":"2024-06-29T14:35:07.157426+0000","last_became_peered":"2024-06-29T14:35:07.157426+0000","last_unstale":"2024-06-29T14:35:08.310796+0000","last_undegraded":"2024-06-29T14:35:08.310796+0000","last_fullsized":"2024-06-29T14:35:08.310796+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:57.297610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311059+0000","last_change":"2024-06-29T14:35:06.147172+0000","last_active":"2024-06-29T14:35:08.311059+0000","last_peered":"2024-06-29T14:35:08.311059+0000","last_clean":"2024-06-29T14:35:08.311059+0000","last_became_active":"2024-06-29T14:35:06.146877+0000","last_became_peered":"2024-06-29T14:35:06.146877+0000","last_unstale":"2024-06-29T14:35:08.311059+0000","last_undegraded":"2024-06-29T14:35:08.311059+0000","last_fullsized":"2024-06-29T14:35:08.311059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:56.288224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150754+0000","last_change":"2024-06-29T14:35:07.166421+0000","last_active":"2024-06-29T14:35:08.150754+0000","last_peered":"2024-06-29T14:35:08.150754+0000","last_clean":"2024-06-29T14:35:08.150754+0000","last_became_active":"2024-06-29T14:35:07.165990+0000","last_became_peered":"2024-06-29T14:35:07.165990+0000","last_unstale":"2024-06-29T14:35:08.150754+0000","last_undegraded":"2024-06-29T14:35:08.150754+0000","last_fullsized":"2024-06-29T14:35:08.150754+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:15.864035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310894+0000","last_change":"2024-06-29T14:35:06.154311+0000","last_active":"2024-06-29T14:35:08.310894+0000","last_peered":"2024-06-29T14:35:08.310894+0000","last_clean":"2024-06-29T14:35:08.310894+0000","last_became_active":"2024-06-29T14:35:06.153161+0000","last_became_peered":"2024-06-29T14:35:06.153161+0000","last_unstale":"2024-06-29T14:35:08.310894+0000","last_undegraded":"2024-06-29T14:35:08.310894+0000","last_fullsized":"2024-06-29T14:35:08.310894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:33:09.220270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311433+0000","last_change":"2024-06-29T14:35:07.158401+0000","last_active":"2024-06-29T14:35:08.311433+0000","last_peered":"2024-06-29T14:35:08.311433+0000","last_clean":"2024-06-29T14:35:08.311433+0000","last_became_active":"2024-06-29T14:35:07.158065+0000","last_became_peered":"2024-06-29T14:35:07.158065+0000","last_unstale":"2024-06-29T14:35:08.311433+0000","last_undegraded":"2024-06-29T14:35:08.311433+0000","last_fullsized":"2024-06-29T14:35:08.311433+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:33:58.907465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150605+0000","last_change":"2024-06-29T14:35:06.142746+0000","last_active":"2024-06-29T14:35:08.150605+0000","last_peered":"2024-06-29T14:35:08.150605+0000","last_clean":"2024-06-29T14:35:08.150605+0000","last_became_active":"2024-06-29T14:35:06.142398+0000","last_became_peered":"2024-06-29T14:35:06.142398+0000","last_unstale":"2024-06-29T14:35:08.150605+0000","last_undegraded":"2024-06-29T14:35:08.150605+0000","last_fullsized":"2024-06-29T14:35:08.150605+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:01.975624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311770+0000","last_change":"2024-06-29T14:35:07.160762+0000","last_active":"2024-06-29T14:35:08.311770+0000","last_peered":"2024-06-29T14:35:08.311770+0000","last_clean":"2024-06-29T14:35:08.311770+0000","last_became_active":"2024-06-29T14:35:07.157499+0000","last_became_peered":"2024-06-29T14:35:07.157499+0000","last_unstale":"2024-06-29T14:35:08.311770+0000","last_undegraded":"2024-06-29T14:35:08.311770+0000","last_fullsized":"2024-06-29T14:35:08.311770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:23.186337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150775+0000","last_change":"2024-06-29T14:35:06.141432+0000","last_active":"2024-06-29T14:35:08.150775+0000","last_peered":"2024-06-29T14:35:08.150775+0000","last_clean":"2024-06-29T14:35:08.150775+0000","last_became_active":"2024-06-29T14:35:06.141079+0000","last_became_peered":"2024-06-29T14:35:06.141079+0000","last_unstale":"2024-06-29T14:35:08.150775+0000","last_undegraded":"2024-06-29T14:35:08.150775+0000","last_fullsized":"2024-06-29T14:35:08.150775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:14.640383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311712+0000","last_change":"2024-06-29T14:35:07.166345+0000","last_active":"2024-06-29T14:35:08.311712+0000","last_peered":"2024-06-29T14:35:08.311712+0000","last_clean":"2024-06-29T14:35:08.311712+0000","last_became_active":"2024-06-29T14:35:07.166020+0000","last_became_peered":"2024-06-29T14:35:07.166020+0000","last_unstale":"2024-06-29T14:35:08.311712+0000","last_undegraded":"2024-06-29T14:35:08.311712+0000","last_fullsized":"2024-06-29T14:35:08.311712+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:07.951025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310073+0000","last_change":"2024-06-29T14:35:06.153920+0000","last_active":"2024-06-29T14:35:08.310073+0000","last_peered":"2024-06-29T14:35:08.310073+0000","last_clean":"2024-06-29T14:35:08.310073+0000","last_became_active":"2024-06-29T14:35:06.153047+0000","last_became_peered":"2024-06-29T14:35:06.153047+0000","last_unstale":"2024-06-29T14:35:08.310073+0000","last_undegraded":"2024-06-29T14:35:08.310073+0000","last_fullsized":"2024-06-29T14:35:08.310073+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:13:00.313486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149794+0000","last_change":"2024-06-29T14:35:07.154389+0000","last_active":"2024-06-29T14:35:08.149794+0000","last_peered":"2024-06-29T14:35:08.149794+0000","last_clean":"2024-06-29T14:35:08.149794+0000","last_became_active":"2024-06-29T14:35:07.154067+0000","last_became_peered":"2024-06-29T14:35:07.154067+0000","last_unstale":"2024-06-29T14:35:08.149794+0000","last_undegraded":"2024-06-29T14:35:08.149794+0000","last_fullsized":"2024-06-29T14:35:08.149794+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:42.235918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311658+0000","last_change":"2024-06-29T14:35:06.145945+0000","last_active":"2024-06-29T14:35:08.311658+0000","last_peered":"2024-06-29T14:35:08.311658+0000","last_clean":"2024-06-29T14:35:08.311658+0000","last_became_active":"2024-06-29T14:35:06.145555+0000","last_became_peered":"2024-06-29T14:35:06.145555+0000","last_unstale":"2024-06-29T14:35:08.311658+0000","last_undegraded":"2024-06-29T14:35:08.311658+0000","last_fullsized":"2024-06-29T14:35:08.311658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:03.780434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310506+0000","last_change":"2024-06-29T14:35:07.149203+0000","last_active":"2024-06-29T14:35:08.310506+0000","last_peered":"2024-06-29T14:35:08.310506+0000","last_clean":"2024-06-29T14:35:08.310506+0000","last_became_active":"2024-06-29T14:35:07.148882+0000","last_became_peered":"2024-06-29T14:35:07.148882+0000","last_unstale":"2024-06-29T14:35:08.310506+0000","last_undegraded":"2024-06-29T14:35:08.310506+0000","last_fullsized":"2024-06-29T14:35:08.310506+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:35:38.877477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150181+0000","last_change":"2024-06-29T14:35:06.156160+0000","last_active":"2024-06-29T14:35:08.150181+0000","last_peered":"2024-06-29T14:35:08.150181+0000","last_clean":"2024-06-29T14:35:08.150181+0000","last_became_active":"2024-06-29T14:35:06.155823+0000","last_became_peered":"2024-06-29T14:35:06.155823+0000","last_unstale":"2024-06-29T14:35:08.150181+0000","last_undegraded":"2024-06-29T14:35:08.150181+0000","last_fullsized":"2024-06-29T14:35:08.150181+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:55.297366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311056+0000","last_change":"2024-06-29T14:35:07.162868+0000","last_active":"2024-06-29T14:35:08.311056+0000","last_peered":"2024-06-29T14:35:08.311056+0000","last_clean":"2024-06-29T14:35:08.311056+0000","last_became_active":"2024-06-29T14:35:07.161415+0000","last_became_peered":"2024-06-29T14:35:07.161415+0000","last_unstale":"2024-06-29T14:35:08.311056+0000","last_undegraded":"2024-06-29T14:35:08.311056+0000","last_fullsized":"2024-06-29T14:35:08.311056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:08.911626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"70'3","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311509+0000","last_change":"2024-06-29T14:35:06.158866+0000","last_active":"2024-06-29T14:35:08.311509+0000","last_peered":"2024-06-29T14:35:08.311509+0000","last_clean":"2024-06-29T14:35:08.311509+0000","last_became_active":"2024-06-29T14:35:06.158615+0000","last_became_peered":"2024-06-29T14:35:06.158615+0000","last_unstale":"2024-06-29T14:35:08.311509+0000","last_undegraded":"2024-06-29T14:35:08.311509+0000","last_fullsized":"2024-06-29T14:35:08.311509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:46.507397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150496+0000","last_change":"2024-06-29T14:35:07.142640+0000","last_active":"2024-06-29T14:35:08.150496+0000","last_peered":"2024-06-29T14:35:08.150496+0000","last_clean":"2024-06-29T14:35:08.150496+0000","last_became_active":"2024-06-29T14:35:07.142297+0000","last_became_peered":"2024-06-29T14:35:07.142297+0000","last_unstale":"2024-06-29T14:35:08.150496+0000","last_undegraded":"2024-06-29T14:35:08.150496+0000","last_fullsized":"2024-06-29T14:35:08.150496+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:31:55.299907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"70'6","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311627+0000","last_change":"2024-06-29T14:35:06.148671+0000","last_active":"2024-06-29T14:35:08.311627+0000","last_peered":"2024-06-29T14:35:08.311627+0000","last_clean":"2024-06-29T14:35:08.311627+0000","last_became_active":"2024-06-29T14:35:06.148322+0000","last_became_peered":"2024-06-29T14:35:06.148322+0000","last_unstale":"2024-06-29T14:35:08.311627+0000","last_undegraded":"2024-06-29T14:35:08.311627+0000","last_fullsized":"2024-06-29T14:35:08.311627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:02:47.756994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310958+0000","last_change":"2024-06-29T14:35:07.157004+0000","last_active":"2024-06-29T14:35:08.310958+0000","last_peered":"2024-06-29T14:35:08.310958+0000","last_clean":"2024-06-29T14:35:08.310958+0000","last_became_active":"2024-06-29T14:35:07.156577+0000","last_became_peered":"2024-06-29T14:35:07.156577+0000","last_unstale":"2024-06-29T14:35:08.310958+0000","last_undegraded":"2024-06-29T14:35:08.310958+0000","last_fullsized":"2024-06-29T14:35:08.310958+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:14.623606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311031+0000","last_change":"2024-06-29T14:35:06.148685+0000","last_active":"2024-06-29T14:35:08.311031+0000","last_peered":"2024-06-29T14:35:08.311031+0000","last_clean":"2024-06-29T14:35:08.311031+0000","last_became_active":"2024-06-29T14:35:06.148343+0000","last_became_peered":"2024-06-29T14:35:06.148343+0000","last_unstale":"2024-06-29T14:35:08.311031+0000","last_undegraded":"2024-06-29T14:35:08.311031+0000","last_fullsized":"2024-06-29T14:35:08.311031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:27.880059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311326+0000","last_change":"2024-06-29T14:35:07.161296+0000","last_active":"2024-06-29T14:35:08.311326+0000","last_peered":"2024-06-29T14:35:08.311326+0000","last_clean":"2024-06-29T14:35:08.311326+0000","last_became_active":"2024-06-29T14:35:07.159183+0000","last_became_peered":"2024-06-29T14:35:07.159183+0000","last_unstale":"2024-06-29T14:35:08.311326+0000","last_undegraded":"2024-06-29T14:35:08.311326+0000","last_fullsized":"2024-06-29T14:35:08.311326+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:31.680178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.151007+0000","last_change":"2024-06-29T14:35:06.146043+0000","last_active":"2024-06-29T14:35:08.151007+0000","last_peered":"2024-06-29T14:35:08.151007+0000","last_clean":"2024-06-29T14:35:08.151007+0000","last_became_active":"2024-06-29T14:35:06.145648+0000","last_became_peered":"2024-06-29T14:35:06.145648+0000","last_unstale":"2024-06-29T14:35:08.151007+0000","last_undegraded":"2024-06-29T14:35:08.151007+0000","last_fullsized":"2024-06-29T14:35:08.151007+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:37.407405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311395+0000","last_change":"2024-06-29T14:35:07.170146+0000","last_active":"2024-06-29T14:35:08.311395+0000","last_peered":"2024-06-29T14:35:08.311395+0000","last_clean":"2024-06-29T14:35:08.311395+0000","last_became_active":"2024-06-29T14:35:07.169829+0000","last_became_peered":"2024-06-29T14:35:07.169829+0000","last_unstale":"2024-06-29T14:35:08.311395+0000","last_undegraded":"2024-06-29T14:35:08.311395+0000","last_fullsized":"2024-06-29T14:35:08.311395+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:17.091758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311228+0000","last_change":"2024-06-29T14:35:06.144003+0000","last_active":"2024-06-29T14:35:08.311228+0000","last_peered":"2024-06-29T14:35:08.311228+0000","last_clean":"2024-06-29T14:35:08.311228+0000","last_became_active":"2024-06-29T14:35:06.143542+0000","last_became_peered":"2024-06-29T14:35:06.143542+0000","last_unstale":"2024-06-29T14:35:08.311228+0000","last_undegraded":"2024-06-29T14:35:08.311228+0000","last_fullsized":"2024-06-29T14:35:08.311228+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:59.804285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311246+0000","last_change":"2024-06-29T14:35:07.165765+0000","last_active":"2024-06-29T14:35:08.311246+0000","last_peered":"2024-06-29T14:35:08.311246+0000","last_clean":"2024-06-29T14:35:08.311246+0000","last_became_active":"2024-06-29T14:35:07.165324+0000","last_became_peered":"2024-06-29T14:35:07.165324+0000","last_unstale":"2024-06-29T14:35:08.311246+0000","last_undegraded":"2024-06-29T14:35:08.311246+0000","last_fullsized":"2024-06-29T14:35:08.311246+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:04:13.409311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"70'8","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311158+0000","last_change":"2024-06-29T14:35:06.143641+0000","last_active":"2024-06-29T14:35:08.311158+0000","last_peered":"2024-06-29T14:35:08.311158+0000","last_clean":"2024-06-29T14:35:08.311158+0000","last_became_active":"2024-06-29T14:35:06.143030+0000","last_became_peered":"2024-06-29T14:35:06.143030+0000","last_unstale":"2024-06-29T14:35:08.311158+0000","last_undegraded":"2024-06-29T14:35:08.311158+0000","last_fullsized":"2024-06-29T14:35:08.311158+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:52.412561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311051+0000","last_change":"2024-06-29T14:35:07.152115+0000","last_active":"2024-06-29T14:35:08.311051+0000","last_peered":"2024-06-29T14:35:08.311051+0000","last_clean":"2024-06-29T14:35:08.311051+0000","last_became_active":"2024-06-29T14:35:07.151780+0000","last_became_peered":"2024-06-29T14:35:07.151780+0000","last_unstale":"2024-06-29T14:35:08.311051+0000","last_undegraded":"2024-06-29T14:35:08.311051+0000","last_fullsized":"2024-06-29T14:35:08.311051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:07.537750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311624+0000","last_change":"2024-06-29T14:35:06.155992+0000","last_active":"2024-06-29T14:35:08.311624+0000","last_peered":"2024-06-29T14:35:08.311624+0000","last_clean":"2024-06-29T14:35:08.311624+0000","last_became_active":"2024-06-29T14:35:06.155609+0000","last_became_peered":"2024-06-29T14:35:06.155609+0000","last_unstale":"2024-06-29T14:35:08.311624+0000","last_undegraded":"2024-06-29T14:35:08.311624+0000","last_fullsized":"2024-06-29T14:35:08.311624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:53.681843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.148935+0000","last_change":"2024-06-29T14:35:07.167767+0000","last_active":"2024-06-29T14:35:08.148935+0000","last_peered":"2024-06-29T14:35:08.148935+0000","last_clean":"2024-06-29T14:35:08.148935+0000","last_became_active":"2024-06-29T14:35:07.167452+0000","last_became_peered":"2024-06-29T14:35:07.167452+0000","last_unstale":"2024-06-29T14:35:08.148935+0000","last_undegraded":"2024-06-29T14:35:08.148935+0000","last_fullsized":"2024-06-29T14:35:08.148935+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.921967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310446+0000","last_change":"2024-06-29T14:35:06.145742+0000","last_active":"2024-06-29T14:35:08.310446+0000","last_peered":"2024-06-29T14:35:08.310446+0000","last_clean":"2024-06-29T14:35:08.310446+0000","last_became_active":"2024-06-29T14:35:06.145327+0000","last_became_peered":"2024-06-29T14:35:06.145327+0000","last_unstale":"2024-06-29T14:35:08.310446+0000","last_undegraded":"2024-06-29T14:35:08.310446+0000","last_fullsized":"2024-06-29T14:35:08.310446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:54.528772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150735+0000","last_change":"2024-06-29T14:35:07.167886+0000","last_active":"2024-06-29T14:35:08.150735+0000","last_peered":"2024-06-29T14:35:08.150735+0000","last_clean":"2024-06-29T14:35:08.150735+0000","last_became_active":"2024-06-29T14:35:07.167543+0000","last_became_peered":"2024-06-29T14:35:07.167543+0000","last_unstale":"2024-06-29T14:35:08.150735+0000","last_undegraded":"2024-06-29T14:35:08.150735+0000","last_fullsized":"2024-06-29T14:35:08.150735+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:19:24.608068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150203+0000","last_change":"2024-06-29T14:35:06.159992+0000","last_active":"2024-06-29T14:35:08.150203+0000","last_peered":"2024-06-29T14:35:08.150203+0000","last_clean":"2024-06-29T14:35:08.150203+0000","last_became_active":"2024-06-29T14:35:06.159710+0000","last_became_peered":"2024-06-29T14:35:06.159710+0000","last_unstale":"2024-06-29T14:35:08.150203+0000","last_undegraded":"2024-06-29T14:35:08.150203+0000","last_fullsized":"2024-06-29T14:35:08.150203+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:26:50.197704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149099+0000","last_change":"2024-06-29T14:35:07.149007+0000","last_active":"2024-06-29T14:35:08.149099+0000","last_peered":"2024-06-29T14:35:08.149099+0000","last_clean":"2024-06-29T14:35:08.149099+0000","last_became_active":"2024-06-29T14:35:07.148691+0000","last_became_peered":"2024-06-29T14:35:07.148691+0000","last_unstale":"2024-06-29T14:35:08.149099+0000","last_undegraded":"2024-06-29T14:35:08.149099+0000","last_fullsized":"2024-06-29T14:35:08.149099+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:43.787786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150052+0000","last_change":"2024-06-29T14:35:06.149423+0000","last_active":"2024-06-29T14:35:08.150052+0000","last_peered":"2024-06-29T14:35:08.150052+0000","last_clean":"2024-06-29T14:35:08.150052+0000","last_became_active":"2024-06-29T14:35:06.149111+0000","last_became_peered":"2024-06-29T14:35:06.149111+0000","last_unstale":"2024-06-29T14:35:08.150052+0000","last_undegraded":"2024-06-29T14:35:08.150052+0000","last_fullsized":"2024-06-29T14:35:08.150052+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:38:05.508055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311276+0000","last_change":"2024-06-29T14:35:07.149524+0000","last_active":"2024-06-29T14:35:08.311276+0000","last_peered":"2024-06-29T14:35:08.311276+0000","last_clean":"2024-06-29T14:35:08.311276+0000","last_became_active":"2024-06-29T14:35:07.148978+0000","last_became_peered":"2024-06-29T14:35:07.148978+0000","last_unstale":"2024-06-29T14:35:08.311276+0000","last_undegraded":"2024-06-29T14:35:08.311276+0000","last_fullsized":"2024-06-29T14:35:08.311276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:03.215623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149063+0000","last_change":"2024-06-29T14:35:06.159208+0000","last_active":"2024-06-29T14:35:08.149063+0000","last_peered":"2024-06-29T14:35:08.149063+0000","last_clean":"2024-06-29T14:35:08.149063+0000","last_became_active":"2024-06-29T14:35:06.158901+0000","last_became_peered":"2024-06-29T14:35:06.158901+0000","last_unstale":"2024-06-29T14:35:08.149063+0000","last_undegraded":"2024-06-29T14:35:08.149063+0000","last_fullsized":"2024-06-29T14:35:08.149063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:22.290921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150083+0000","last_change":"2024-06-29T14:35:07.138249+0000","last_active":"2024-06-29T14:35:08.150083+0000","last_peered":"2024-06-29T14:35:08.150083+0000","last_clean":"2024-06-29T14:35:08.150083+0000","last_became_active":"2024-06-29T14:35:07.137889+0000","last_became_peered":"2024-06-29T14:35:07.137889+0000","last_unstale":"2024-06-29T14:35:08.150083+0000","last_undegraded":"2024-06-29T14:35:08.150083+0000","last_fullsized":"2024-06-29T14:35:08.150083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:44.466559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311224+0000","last_change":"2024-06-29T14:35:06.142795+0000","last_active":"2024-06-29T14:35:08.311224+0000","last_peered":"2024-06-29T14:35:08.311224+0000","last_clean":"2024-06-29T14:35:08.311224+0000","last_became_active":"2024-06-29T14:35:06.142463+0000","last_became_peered":"2024-06-29T14:35:06.142463+0000","last_unstale":"2024-06-29T14:35:08.311224+0000","last_undegraded":"2024-06-29T14:35:08.311224+0000","last_fullsized":"2024-06-29T14:35:08.311224+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:36.767707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310881+0000","last_change":"2024-06-29T14:35:07.165455+0000","last_active":"2024-06-29T14:35:08.310881+0000","last_peered":"2024-06-29T14:35:08.310881+0000","last_clean":"2024-06-29T14:35:08.310881+0000","last_became_active":"2024-06-29T14:35:07.165053+0000","last_became_peered":"2024-06-29T14:35:07.165053+0000","last_unstale":"2024-06-29T14:35:08.310881+0000","last_undegraded":"2024-06-29T14:35:08.310881+0000","last_fullsized":"2024-06-29T14:35:08.310881+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:54.004799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310571+0000","last_change":"2024-06-29T14:35:06.154839+0000","last_active":"2024-06-29T14:35:08.310571+0000","last_peered":"2024-06-29T14:35:08.310571+0000","last_clean":"2024-06-29T14:35:08.310571+0000","last_became_active":"2024-06-29T14:35:06.154513+0000","last_became_peered":"2024-06-29T14:35:06.154513+0000","last_unstale":"2024-06-29T14:35:08.310571+0000","last_undegraded":"2024-06-29T14:35:08.310571+0000","last_fullsized":"2024-06-29T14:35:08.310571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:26.853277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149536+0000","last_change":"2024-06-29T14:35:07.162073+0000","last_active":"2024-06-29T14:35:08.149536+0000","last_peered":"2024-06-29T14:35:08.149536+0000","last_clean":"2024-06-29T14:35:08.149536+0000","last_became_active":"2024-06-29T14:35:07.161748+0000","last_became_peered":"2024-06-29T14:35:07.161748+0000","last_unstale":"2024-06-29T14:35:08.149536+0000","last_undegraded":"2024-06-29T14:35:08.149536+0000","last_fullsized":"2024-06-29T14:35:08.149536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:26:40.579154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.309898+0000","last_change":"2024-06-29T14:35:06.143667+0000","last_active":"2024-06-29T14:35:08.309898+0000","last_peered":"2024-06-29T14:35:08.309898+0000","last_clean":"2024-06-29T14:35:08.309898+0000","last_became_active":"2024-06-29T14:35:06.143312+0000","last_became_peered":"2024-06-29T14:35:06.143312+0000","last_unstale":"2024-06-29T14:35:08.309898+0000","last_undegraded":"2024-06-29T14:35:08.309898+0000","last_fullsized":"2024-06-29T14:35:08.309898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:20:07.071600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310552+0000","last_change":"2024-06-29T14:35:07.153363+0000","last_active":"2024-06-29T14:35:08.310552+0000","last_peered":"2024-06-29T14:35:08.310552+0000","last_clean":"2024-06-29T14:35:08.310552+0000","last_became_active":"2024-06-29T14:35:07.152829+0000","last_became_peered":"2024-06-29T14:35:07.152829+0000","last_unstale":"2024-06-29T14:35:08.310552+0000","last_undegraded":"2024-06-29T14:35:08.310552+0000","last_fullsized":"2024-06-29T14:35:08.310552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:23.186337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.150362+0000","last_change":"2024-06-29T14:35:06.152663+0000","last_active":"2024-06-29T14:35:08.150362+0000","last_peered":"2024-06-29T14:35:08.150362+0000","last_clean":"2024-06-29T14:35:08.150362+0000","last_became_active":"2024-06-29T14:35:06.152267+0000","last_became_peered":"2024-06-29T14:35:06.152267+0000","last_unstale":"2024-06-29T14:35:08.150362+0000","last_undegraded":"2024-06-29T14:35:08.150362+0000","last_fullsized":"2024-06-29T14:35:08.150362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:55:25.675236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149100+0000","last_change":"2024-06-29T14:35:07.141322+0000","last_active":"2024-06-29T14:35:08.149100+0000","last_peered":"2024-06-29T14:35:08.149100+0000","last_clean":"2024-06-29T14:35:08.149100+0000","last_became_active":"2024-06-29T14:35:07.140970+0000","last_became_peered":"2024-06-29T14:35:07.140970+0000","last_unstale":"2024-06-29T14:35:08.149100+0000","last_undegraded":"2024-06-29T14:35:08.149100+0000","last_fullsized":"2024-06-29T14:35:08.149100+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:07.467630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311608+0000","last_change":"2024-06-29T14:35:06.155153+0000","last_active":"2024-06-29T14:35:08.311608+0000","last_peered":"2024-06-29T14:35:08.311608+0000","last_clean":"2024-06-29T14:35:08.311608+0000","last_became_active":"2024-06-29T14:35:06.154919+0000","last_became_peered":"2024-06-29T14:35:06.154919+0000","last_unstale":"2024-06-29T14:35:08.311608+0000","last_undegraded":"2024-06-29T14:35:08.311608+0000","last_fullsized":"2024-06-29T14:35:08.311608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:05:28.872856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310970+0000","last_change":"2024-06-29T14:35:07.152311+0000","last_active":"2024-06-29T14:35:08.310970+0000","last_peered":"2024-06-29T14:35:08.310970+0000","last_clean":"2024-06-29T14:35:08.310970+0000","last_became_active":"2024-06-29T14:35:07.151939+0000","last_became_peered":"2024-06-29T14:35:07.151939+0000","last_unstale":"2024-06-29T14:35:08.310970+0000","last_undegraded":"2024-06-29T14:35:08.310970+0000","last_fullsized":"2024-06-29T14:35:08.310970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:25.526535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310688+0000","last_change":"2024-06-29T14:35:06.149483+0000","last_active":"2024-06-29T14:35:08.310688+0000","last_peered":"2024-06-29T14:35:08.310688+0000","last_clean":"2024-06-29T14:35:08.310688+0000","last_became_active":"2024-06-29T14:35:06.149163+0000","last_became_peered":"2024-06-29T14:35:06.149163+0000","last_unstale":"2024-06-29T14:35:08.310688+0000","last_undegraded":"2024-06-29T14:35:08.310688+0000","last_fullsized":"2024-06-29T14:35:08.310688+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:57:29.454929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310576+0000","last_change":"2024-06-29T14:35:07.171505+0000","last_active":"2024-06-29T14:35:08.310576+0000","last_peered":"2024-06-29T14:35:08.310576+0000","last_clean":"2024-06-29T14:35:08.310576+0000","last_became_active":"2024-06-29T14:35:07.171166+0000","last_became_peered":"2024-06-29T14:35:07.171166+0000","last_unstale":"2024-06-29T14:35:08.310576+0000","last_undegraded":"2024-06-29T14:35:08.310576+0000","last_fullsized":"2024-06-29T14:35:08.310576+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:04:55.611535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311072+0000","last_change":"2024-06-29T14:35:06.148884+0000","last_active":"2024-06-29T14:35:08.311072+0000","last_peered":"2024-06-29T14:35:08.311072+0000","last_clean":"2024-06-29T14:35:08.311072+0000","last_became_active":"2024-06-29T14:35:06.148389+0000","last_became_peered":"2024-06-29T14:35:06.148389+0000","last_unstale":"2024-06-29T14:35:08.311072+0000","last_undegraded":"2024-06-29T14:35:08.311072+0000","last_fullsized":"2024-06-29T14:35:08.311072+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:25.968494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310814+0000","last_change":"2024-06-29T14:35:07.157970+0000","last_active":"2024-06-29T14:35:08.310814+0000","last_peered":"2024-06-29T14:35:08.310814+0000","last_clean":"2024-06-29T14:35:08.310814+0000","last_became_active":"2024-06-29T14:35:07.157652+0000","last_became_peered":"2024-06-29T14:35:07.157652+0000","last_unstale":"2024-06-29T14:35:08.310814+0000","last_undegraded":"2024-06-29T14:35:08.310814+0000","last_fullsized":"2024-06-29T14:35:08.310814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:25.977050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311237+0000","last_change":"2024-06-29T14:35:06.152507+0000","last_active":"2024-06-29T14:35:08.311237+0000","last_peered":"2024-06-29T14:35:08.311237+0000","last_clean":"2024-06-29T14:35:08.311237+0000","last_became_active":"2024-06-29T14:35:06.152204+0000","last_became_peered":"2024-06-29T14:35:06.152204+0000","last_unstale":"2024-06-29T14:35:08.311237+0000","last_undegraded":"2024-06-29T14:35:08.311237+0000","last_fullsized":"2024-06-29T14:35:08.311237+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:01.111148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310536+0000","last_change":"2024-06-29T14:35:07.166439+0000","last_active":"2024-06-29T14:35:08.310536+0000","last_peered":"2024-06-29T14:35:08.310536+0000","last_clean":"2024-06-29T14:35:08.310536+0000","last_became_active":"2024-06-29T14:35:07.166106+0000","last_became_peered":"2024-06-29T14:35:07.166106+0000","last_unstale":"2024-06-29T14:35:08.310536+0000","last_undegraded":"2024-06-29T14:35:08.310536+0000","last_fullsized":"2024-06-29T14:35:08.310536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:02:36.725468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149927+0000","last_change":"2024-06-29T14:35:06.141470+0000","last_active":"2024-06-29T14:35:08.149927+0000","last_peered":"2024-06-29T14:35:08.149927+0000","last_clean":"2024-06-29T14:35:08.149927+0000","last_became_active":"2024-06-29T14:35:06.141175+0000","last_became_peered":"2024-06-29T14:35:06.141175+0000","last_unstale":"2024-06-29T14:35:08.149927+0000","last_undegraded":"2024-06-29T14:35:08.149927+0000","last_fullsized":"2024-06-29T14:35:08.149927+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:06:49.387973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.311657+0000","last_change":"2024-06-29T14:35:07.160252+0000","last_active":"2024-06-29T14:35:08.311657+0000","last_peered":"2024-06-29T14:35:08.311657+0000","last_clean":"2024-06-29T14:35:08.311657+0000","last_became_active":"2024-06-29T14:35:07.158059+0000","last_became_peered":"2024-06-29T14:35:07.158059+0000","last_unstale":"2024-06-29T14:35:08.311657+0000","last_undegraded":"2024-06-29T14:35:08.311657+0000","last_fullsized":"2024-06-29T14:35:08.311657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:11.443951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310767+0000","last_change":"2024-06-29T14:35:06.152825+0000","last_active":"2024-06-29T14:35:08.310767+0000","last_peered":"2024-06-29T14:35:08.310767+0000","last_clean":"2024-06-29T14:35:08.310767+0000","last_became_active":"2024-06-29T14:35:06.152469+0000","last_became_peered":"2024-06-29T14:35:06.152469+0000","last_unstale":"2024-06-29T14:35:08.310767+0000","last_undegraded":"2024-06-29T14:35:08.310767+0000","last_fullsized":"2024-06-29T14:35:08.310767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:27:10.436963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.149880+0000","last_change":"2024-06-29T14:35:07.144638+0000","last_active":"2024-06-29T14:35:08.149880+0000","last_peered":"2024-06-29T14:35:08.149880+0000","last_clean":"2024-06-29T14:35:08.149880+0000","last_became_active":"2024-06-29T14:35:07.144342+0000","last_became_peered":"2024-06-29T14:35:07.144342+0000","last_unstale":"2024-06-29T14:35:08.149880+0000","last_undegraded":"2024-06-29T14:35:08.149880+0000","last_fullsized":"2024-06-29T14:35:08.149880+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:06.120894+0000","last_clean_scrub_stamp":"2024-06-29T14:35:06.120894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:10:23.403328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"71'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310878+0000","last_change":"2024-06-29T14:35:06.148661+0000","last_active":"2024-06-29T14:35:08.310878+0000","last_peered":"2024-06-29T14:35:08.310878+0000","last_clean":"2024-06-29T14:35:08.310878+0000","last_became_active":"2024-06-29T14:35:06.148307+0000","last_became_peered":"2024-06-29T14:35:06.148307+0000","last_unstale":"2024-06-29T14:35:08.310878+0000","last_undegraded":"2024-06-29T14:35:08.310878+0000","last_fullsized":"2024-06-29T14:35:08.310878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:35:05.111508+0000","last_clean_scrub_stamp":"2024-06-29T14:35:05.111508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:07:18.680056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'86","reported_seq":261,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-06-29T14:35:08.310437+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:35:08.310437+0000","last_peered":"2024-06-29T14:35:08.310437+0000","last_clean":"2024-06-29T14:35:08.310437+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:35:08.310437+0000","last_undegraded":"2024-06-29T14:35:08.310437+0000","last_fullsized":"2024-06-29T14:35:08.310437+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542308,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54808,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317032,"statfs":{"total":96636764160,"available":96580640768,"internally_reserved":0,"allocated":1445888,"data_stored":1011393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542309,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":851968,"data_stored":421553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542308,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50052,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321788,"statfs":{"total":96636764160,"available":96585510912,"internally_reserved":0,"allocated":843776,"data_stored":420901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542307,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":421483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509605,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":422535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58384,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313456,"statfs":{"total":96636764160,"available":96576978944,"internally_reserved":0,"allocated":856064,"data_stored":422513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509604,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":851968,"data_stored":421061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509606,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1445888,"data_stored":1011325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:35:16.128 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:35:16.236 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:35:16.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.363+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1543972918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff424144500 msgr2=0x7ff4240cd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.363+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1543972918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff424144500 0x7ff4240cd350 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ff4240afc70 tx=0x7ff41c02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.363+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1543972918 shutdown_connections 2024-06-29T14:35:16.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.363+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1543972918 >> v2:172.21.15.153:3300/0 conn(0x7ff42415e620 0x7ff42415ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.363+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1543972918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff424144500 0x7ff4240cd350 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.363+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1543972918 >> 172.21.15.59:0/1543972918 conn(0x7ff424072cb0 msgr2=0x7ff4240730c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:16.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.364+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1543972918 shutdown_connections 2024-06-29T14:35:16.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.364+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1543972918 wait complete. 2024-06-29T14:35:16.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.365+0000 7ff42bfa5640 1 Processor -- start 2024-06-29T14:35:16.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.365+0000 7ff42bfa5640 1 -- start start 2024-06-29T14:35:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff42bfa5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff42415b030 0x7ff4241f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff42bfa5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff42415e620 0x7ff424135c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff42bfa5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff42406ca70 0x7ff42406ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff42bfa5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff4240702e0 con 0x7ff42415b030 2024-06-29T14:35:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff42bfa5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff424070160 con 0x7ff42406ca70 2024-06-29T14:35:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff42bfa5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff424070460 con 0x7ff42415e620 2024-06-29T14:35:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff42415b030 0x7ff4241f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff42415b030 0x7ff4241f1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34882/0 (socket says 172.21.15.59:34882) 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 -- 172.21.15.59:0/1786096048 learned_addr learned my addr 172.21.15.59:0/1786096048 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff42a542640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff42406ca70 0x7ff42406ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429540640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff42415e620 0x7ff424135c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 -- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff42415e620 msgr2=0x7ff424135c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff42415e620 0x7ff424135c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 -- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff42406ca70 msgr2=0x7ff42406ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff42406ca70 0x7ff42406ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 -- 172.21.15.59:0/1786096048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42406d3c0 con 0x7ff42415b030 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff429d41640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff42415b030 0x7ff4241f1c00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff41c034040 tx=0x7ff41c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:16.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff4227fc640 1 -- 172.21.15.59:0/1786096048 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c03e030 con 0x7ff42415b030 2024-06-29T14:35:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff4227fc640 1 -- 172.21.15.59:0/1786096048 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff41c0047a0 con 0x7ff42415b030 2024-06-29T14:35:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.367+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4241361c0 con 0x7ff42415b030 2024-06-29T14:35:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.368+0000 7ff4227fc640 1 -- 172.21.15.59:0/1786096048 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c035040 con 0x7ff42415b030 2024-06-29T14:35:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.368+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4241f4e60 con 0x7ff42415b030 2024-06-29T14:35:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.369+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff42415be60 con 0x7ff42415b030 2024-06-29T14:35:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.369+0000 7ff4227fc640 1 -- 172.21.15.59:0/1786096048 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7ff41c041030 con 0x7ff42415b030 2024-06-29T14:35:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.370+0000 7ff4227fc640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff410076420 0x7ff410078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.370+0000 7ff4227fc640 1 -- 172.21.15.59:0/1786096048 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7ff41c0b82b0 con 0x7ff42415b030 2024-06-29T14:35:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.370+0000 7ff429540640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff410076420 0x7ff410078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.371+0000 7ff429540640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff410076420 0x7ff410078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff42406db50 tx=0x7ff418005e30 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:16.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.374+0000 7ff4227fc640 1 -- 172.21.15.59:0/1786096048 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff42406d3c0 con 0x7ff42415b030 2024-06-29T14:35:16.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.640+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff4240b42c0 con 0x7ff42415b030 2024-06-29T14:35:16.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.645+0000 7ff4227fc640 1 -- 172.21.15.59:0/1786096048 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7ff41c081860 con 0x7ff42415b030 2024-06-29T14:35:16.643 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:35:16.643 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:35:16.643 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T14:35:16.643 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T14:35:16.643 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:35:16.644 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T14:35:16.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.647+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff410076420 msgr2=0x7ff410078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff410076420 0x7ff410078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff42406db50 tx=0x7ff418005e30 comp rx=0 tx=0).stop 2024-06-29T14:35:16.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff42415b030 msgr2=0x7ff4241f1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff42415b030 0x7ff4241f1c00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff41c034040 tx=0x7ff41c0039e0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 shutdown_connections 2024-06-29T14:35:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7ff410076420 0x7ff410078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff42406ca70 0x7ff42406ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff42415e620 0x7ff424135c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 --2- 172.21.15.59:0/1786096048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff42415b030 0x7ff4241f1c00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 >> 172.21.15.59:0/1786096048 conn(0x7ff424072cb0 msgr2=0x7ff4240730c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 shutdown_connections 2024-06-29T14:35:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.648+0000 7ff42bfa5640 1 -- 172.21.15.59:0/1786096048 wait complete. 2024-06-29T14:35:16.657 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:35:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.776+0000 7fecaf19a640 1 -- 172.21.15.59:0/3374643366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca81458a0 msgr2=0x7feca80cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.777+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3374643366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca81458a0 0x7feca80cf870 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7feca4005fd0 tx=0x7feca402a530 comp rx=0 tx=0).stop 2024-06-29T14:35:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.777+0000 7fecaf19a640 1 -- 172.21.15.59:0/3374643366 shutdown_connections 2024-06-29T14:35:16.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.777+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3374643366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca80b3740 0x7feca80b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.777+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3374643366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca81458a0 0x7feca80cf870 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.777+0000 7fecaf19a640 1 -- 172.21.15.59:0/3374643366 >> 172.21.15.59:0/3374643366 conn(0x7feca806c950 msgr2=0x7feca8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.777+0000 7fecaf19a640 1 -- 172.21.15.59:0/3374643366 shutdown_connections 2024-06-29T14:35:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.777+0000 7fecaf19a640 1 -- 172.21.15.59:0/3374643366 wait complete. 2024-06-29T14:35:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.778+0000 7fecaf19a640 1 Processor -- start 2024-06-29T14:35:16.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.778+0000 7fecaf19a640 1 -- start start 2024-06-29T14:35:16.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecaf19a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca80b3740 0x7feca813ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:16.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecaf19a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca815c2f0 0x7feca813b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:16.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecaf19a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca80a94a0 0x7feca80a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:16.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecaf19a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7feca8160020 con 0x7feca80b3740 2024-06-29T14:35:16.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecaf19a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feca815cbb0 con 0x7feca80a94a0 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecaf19a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7feca81601a0 con 0x7feca815c2f0 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecacf36640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca80b3740 0x7feca813ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecacf36640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca80b3740 0x7feca813ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34906/0 (socket says 172.21.15.59:34906) 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecacf36640 1 -- 172.21.15.59:0/3921419335 learned_addr learned my addr 172.21.15.59:0/3921419335 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecad737640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca80a94a0 0x7feca80a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fec9ffff640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca815c2f0 0x7feca813b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecacf36640 1 -- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca815c2f0 msgr2=0x7feca813b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecacf36640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca815c2f0 0x7feca813b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecacf36640 1 -- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca80a94a0 msgr2=0x7feca80a98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecacf36640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca80a94a0 0x7feca80a98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fecacf36640 1 -- 172.21.15.59:0/3921419335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca813dba0 con 0x7feca80b3740 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.780+0000 7fec9ffff640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca815c2f0 0x7feca813b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.781+0000 7fecacf36640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca80b3740 0x7feca813ec40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feca4000c00 tx=0x7feca4003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.781+0000 7fec9d7fa640 1 -- 172.21.15.59:0/3921419335 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca403e030 con 0x7feca80b3740 2024-06-29T14:35:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.781+0000 7fec9d7fa640 1 -- 172.21.15.59:0/3921419335 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feca4003e60 con 0x7feca80b3740 2024-06-29T14:35:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.781+0000 7fec9d7fa640 1 -- 172.21.15.59:0/3921419335 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4035040 con 0x7feca80b3740 2024-06-29T14:35:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.781+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca813de80 con 0x7feca80b3740 2024-06-29T14:35:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.781+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca813e440 con 0x7feca80b3740 2024-06-29T14:35:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.782+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca81606c0 con 0x7feca80b3740 2024-06-29T14:35:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.783+0000 7fec9d7fa640 1 -- 172.21.15.59:0/3921419335 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7feca4041030 con 0x7feca80b3740 2024-06-29T14:35:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.783+0000 7fec9d7fa640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fec94076420 0x7fec94078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.783+0000 7fec9d7fa640 1 -- 172.21.15.59:0/3921419335 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7feca40b8250 con 0x7feca80b3740 2024-06-29T14:35:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.783+0000 7fec9ffff640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fec94076420 0x7fec94078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.784+0000 7fec9ffff640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fec94076420 0x7fec94078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fec98005fd0 tx=0x7fec98005e00 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:16.788+0000 7fec9d7fa640 1 -- 172.21.15.59:0/3921419335 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feca40811b0 con 0x7feca80b3740 2024-06-29T14:35:17.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.016+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feca80b6ab0 con 0x7feca80b3740 2024-06-29T14:35:17.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.017+0000 7fec9d7fa640 1 -- 172.21.15.59:0/3921419335 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5223 (secure 0 0 0) 0x7feca40854b0 con 0x7feca80b3740 2024-06-29T14:35:17.016 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:17.016 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":29,"btime":"2024-06-29T14:35:10:154758+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/484297155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":484297155},{"type":"v1","addr":"172.21.15.153:6835","nonce":484297155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2309835731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":2309835731},{"type":"v1","addr":"172.21.15.153:6837","nonce":2309835731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":6330,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/413487166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":413487166},{"type":"v1","addr":"172.21.15.59:6834","nonce":413487166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:35:06.529979+0000","modified":"2024-06-29T14:35:09.544199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6333":{"gid":6333,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3627888249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3627888249},{"type":"v1","addr":"172.21.15.59:6835","nonce":3627888249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6333,"qdb_cluster":[6333]},"id":3}]} 2024-06-29T14:35:17.016 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 29 2024-06-29T14:35:17.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fec94076420 msgr2=0x7fec94078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fec94076420 0x7fec94078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fec98005fd0 tx=0x7fec98005e00 comp rx=0 tx=0).stop 2024-06-29T14:35:17.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca80b3740 msgr2=0x7feca813ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca80b3740 0x7feca813ec40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feca4000c00 tx=0x7feca4003970 comp rx=0 tx=0).stop 2024-06-29T14:35:17.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 shutdown_connections 2024-06-29T14:35:17.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fec94076420 0x7fec94078930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feca80a94a0 0x7feca80a98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feca815c2f0 0x7feca813b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 --2- 172.21.15.59:0/3921419335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca80b3740 0x7feca813ec40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 >> 172.21.15.59:0/3921419335 conn(0x7feca806c950 msgr2=0x7feca806fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 shutdown_connections 2024-06-29T14:35:17.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.019+0000 7fecaf19a640 1 -- 172.21.15.59:0/3921419335 wait complete. 2024-06-29T14:35:17.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:35:06.529979+0000', 'modified': '2024-06-29T14:35:09.544199+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6333}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6333': {'gid': 6333, 'name': 'a', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.59:6835/3627888249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 3627888249}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 3627888249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6333, 'qdb_cluster': [6333]} 2024-06-29T14:35:17.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:35:17.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1066579237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 msgr2=0x7f46f80d1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1066579237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 0x7f46f80d1f60 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f46f80b64f0 tx=0x7f46f002a560 comp rx=0 tx=0).stop 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1066579237 shutdown_connections 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1066579237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 0x7f46f80d1f60 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1066579237 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46f80cf5c0 0x7f46f80d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1066579237 >> v2:172.21.15.153:3300/0 conn(0x7f46f808c190 0x7f46f80cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1066579237 >> 172.21.15.59:0/1066579237 conn(0x7f46f80b5610 msgr2=0x7f46f80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1066579237 shutdown_connections 2024-06-29T14:35:17.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.556+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1066579237 wait complete. 2024-06-29T14:35:17.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.558+0000 7f46fd8f6640 1 Processor -- start 2024-06-29T14:35:17.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.558+0000 7f46fd8f6640 1 -- start start 2024-06-29T14:35:17.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46fd8f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f808c190 0x7f46f815aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46fd8f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 0x7f46f80ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46fd8f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46f80cf5c0 0x7f46f8158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46fd8f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8162440 con 0x7f46f80ce950 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46fd8f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46f81622c0 con 0x7f46f808c190 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46fd8f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46f81625c0 con 0x7f46f80cf5c0 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46f67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 0x7f46f80ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46f67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 0x7f46f80ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34926/0 (socket says 172.21.15.59:34926) 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.559+0000 7f46f67fc640 1 -- 172.21.15.59:0/1092440098 learned_addr learned my addr 172.21.15.59:0/1092440098 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f77fe640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46f80cf5c0 0x7f46f8158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f808c190 0x7f46f815aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f67fc640 1 -- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46f80cf5c0 msgr2=0x7f46f8158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f67fc640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46f80cf5c0 0x7f46f8158d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f67fc640 1 -- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f808c190 msgr2=0x7f46f815aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f67fc640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f808c190 0x7f46f815aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f67fc640 1 -- 172.21.15.59:0/1092440098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f80cd6e0 con 0x7f46f80ce950 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f77fe640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46f80cf5c0 0x7f46f8158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f808c190 0x7f46f815aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46f67fc640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 0x7f46f80ccc20 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c9a0 tx=0x7f46e000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46effff640 1 -- 172.21.15.59:0/1092440098 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0007bb0 con 0x7f46f80ce950 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46effff640 1 -- 172.21.15.59:0/1092440098 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46e0007e40 con 0x7f46f80ce950 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1092440098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f80cd9c0 con 0x7f46f80ce950 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46effff640 1 -- 172.21.15.59:0/1092440098 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e000f690 con 0x7f46f80ce950 2024-06-29T14:35:17.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.560+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1092440098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f8158850 con 0x7f46f80ce950 2024-06-29T14:35:17.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.562+0000 7f46effff640 1 -- 172.21.15.59:0/1092440098 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f46e000f830 con 0x7f46f80ce950 2024-06-29T14:35:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.563+0000 7f46effff640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f46d80763f0 0x7f46d8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.563+0000 7f46effff640 1 -- 172.21.15.59:0/1092440098 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f46e0098360 con 0x7f46f80ce950 2024-06-29T14:35:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.563+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f46d80763f0 0x7f46d8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.563+0000 7f47047b2740 1 Processor -- start 2024-06-29T14:35:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.563+0000 7f47047b2740 1 -- start start 2024-06-29T14:35:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.564+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f46d80763f0 0x7f46d8078900 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46e8005cc0 tx=0x7f46e800a430 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:17.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f47047b2740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 0x55d92bef16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f47047b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92bed53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f47047b2740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92bed58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f47047b2740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d92be6dbc0 con 0x55d92bae4590 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f47047b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d92be1c630 con 0x55d92badc3b0 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f47047b2740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d92be0bbd0 con 0x55d92bb05270 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92bed58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92bed58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34938/0 (socket says 172.21.15.59:34938) 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f77fe640 1 -- 172.21.15.59:0/4216900012 learned_addr learned my addr 172.21.15.59:0/4216900012 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 0x55d92bef16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f67fc640 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92bed53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f77fe640 1 -- 172.21.15.59:0/4216900012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 msgr2=0x55d92bef16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f77fe640 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 0x55d92bef16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f77fe640 1 -- 172.21.15.59:0/4216900012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 msgr2=0x55d92bed53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f77fe640 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92bed53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.565+0000 7f46f77fe640 1 -- 172.21.15.59:0/4216900012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d92be08ca0 con 0x55d92bae4590 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.566+0000 7f46f77fe640 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92bed58e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f46f0034040 tx=0x7f46f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56adcf137f3635a5 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.566+0000 7f46ed7fa640 1 -- 172.21.15.59:0/4216900012 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0040070 con 0x55d92bae4590 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.566+0000 7f46ed7fa640 1 -- 172.21.15.59:0/4216900012 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46f0004330 con 0x55d92bae4590 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.566+0000 7f46ed7fa640 1 -- 172.21.15.59:0/4216900012 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0035040 con 0x55d92bae4590 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 -- 172.21.15.59:0/4216900012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 msgr2=0x55d92bed58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92bed58e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f46f0034040 tx=0x7f46f00046c0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 -- 172.21.15.59:0/4216900012 shutdown_connections 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92bed58e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92bed53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 --2- 172.21.15.59:0/4216900012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 0x55d92bef16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 -- 172.21.15.59:0/4216900012 >> 172.21.15.59:0/4216900012 conn(0x55d92baf93c0 msgr2=0x55d92bea8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 -- 172.21.15.59:0/4216900012 shutdown_connections 2024-06-29T14:35:17.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.567+0000 7f47047b2740 1 -- 172.21.15.59:0/4216900012 wait complete. 2024-06-29T14:35:17.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.568+0000 7f47047b2740 1 -- start start 2024-06-29T14:35:17.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.568+0000 7f47047b2740 1 Processor -- start 2024-06-29T14:35:17.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f47047b2740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92badc3b0 0x55d92beccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f47047b2740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bae4590 0x55d92beea7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f47047b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bb05270 0x55d92bf03730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f47047b2740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d92bda9ae0 con 0x55d92badc3b0 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f47047b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d92be0fc10 con 0x55d92bb05270 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f47047b2740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d92bb7f620 con 0x55d92bae4590 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f47047b2740 10 client.0 _ll_register_callbacks cb 0x55d92bdb6480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f46f6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92badc3b0 0x55d92beccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f46f6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92badc3b0 0x55d92beccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34954/0 (socket says 172.21.15.59:34954) 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f46f6ffd640 1 -- 172.21.15.59:0/4170976381 learned_addr learned my addr 172.21.15.59:0/4170976381 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f46f67fc640 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bae4590 0x55d92beea7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.569+0000 7f46f77fe640 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bb05270 0x55d92bf03730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46f6ffd640 1 -- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bae4590 msgr2=0x55d92beea7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bae4590 0x55d92beea7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46f6ffd640 1 -- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bb05270 msgr2=0x55d92bf03730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bb05270 0x55d92bf03730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46f6ffd640 1 -- 172.21.15.59:0/4170976381 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d92bb70850 con 0x55d92badc3b0 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46f67fc640 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bae4590 0x55d92beea7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92badc3b0 0x55d92beccdd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f46e8010e00 tx=0x7f46e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5351e9f300991bc server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46d67fc640 1 -- 172.21.15.59:0/4170976381 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e801e630 con 0x55d92badc3b0 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46d67fc640 1 -- 172.21.15.59:0/4170976381 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f46e801e7d0 con 0x55d92badc3b0 2024-06-29T14:35:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f47047b2740 1 -- 172.21.15.59:0/4170976381 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46f0004330 con 0x55d92badc3b0 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46d67fc640 1 -- 172.21.15.59:0/4170976381 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f46e8030050 con 0x55d92badc3b0 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f47047b2740 10 client.6483 fetch_fsmap learned FSMap version 29 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f47047b2740 1 -- 172.21.15.59:0/4170976381 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) -- 0x55d92bbe76b0 con 0x55d92badc3b0 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46d67fc640 1 -- 172.21.15.59:0/4170976381 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46f0004330 con 0x55d92badc3b0 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f46d67fc640 1 -- 172.21.15.59:0/4170976381 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 29) ==== 3502+0+0 (secure 0 0 0) 0x7f46e8033020 con 0x55d92badc3b0 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f47047b2740 10 client.6483 fetch_fsmap finished waiting for FSMap version 29 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.570+0000 7f47047b2740 10 client.6483 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.571+0000 7f47047b2740 20 client.6483 populate_metadata read hostname 'smithi059' 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.571+0000 7f47047b2740 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] conn(0x55d92bac3e40 0x55d92bf00190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.571+0000 7f47047b2740 4 client.6483 mds_command: new command op to 6333 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.571+0000 7f47047b2740 1 -- 172.21.15.59:0/4170976381 --> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d92be1e580 con 0x55d92bac3e40 2024-06-29T14:35:17.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.571+0000 7f46f67fc640 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] conn(0x55d92bac3e40 0x55d92bf00190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.571+0000 7f46d67fc640 10 client.6483 ms_handle_connect on v2:172.21.15.59:6833/3627888249 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.571+0000 7f46f67fc640 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] conn(0x55d92bac3e40 0x55d92bf00190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fa4ecabcaca38e5 server_cookie=e332f74e414c21bb in_seq=0 out_seq=0 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f46d67fc640 1 -- 172.21.15.59:0/4170976381 <== mds.0 v2:172.21.15.59:6833/3627888249 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d92be1e580 con 0x55d92bac3e40 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f46d67fc640 10 client.6483 handle_command_reply: tid=0 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 client.6483 shutdown 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 -- 172.21.15.59:0/4170976381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92badc3b0 msgr2=0x55d92beccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92badc3b0 0x55d92beccdd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f46e8010e00 tx=0x7f46e8004a40 comp rx=0 tx=0).stop 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 -- 172.21.15.59:0/4170976381 shutdown_connections 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] conn(0x55d92bac3e40 0x55d92bf00190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bb05270 0x55d92bf03730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bae4590 0x55d92beea7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 --2- 172.21.15.59:0/4170976381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92badc3b0 0x55d92beccdd0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f46d67fc640 0 client.6483 ms_handle_reset on v2:172.21.15.59:6833/3627888249 2024-06-29T14:35:17.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 -- 172.21.15.59:0/4170976381 >> 172.21.15.59:0/4170976381 conn(0x55d92baf93c0 msgr2=0x55d92bed1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 -- 172.21.15.59:0/4170976381 shutdown_connections 2024-06-29T14:35:17.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 1 -- 172.21.15.59:0/4170976381 wait complete. 2024-06-29T14:35:17.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.572+0000 7f47047b2740 20 client.6483 trim_cache size 0 max 16384 2024-06-29T14:35:17.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.591+0000 7f47047b2740 1 Processor -- start 2024-06-29T14:35:17.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.591+0000 7f47047b2740 1 -- start start 2024-06-29T14:35:17.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f47047b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92beaaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f47047b2740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92beab460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f47047b2740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 0x55d92bf13c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f47047b2740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8200ca0 con 0x55d92bae4590 2024-06-29T14:35:17.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f47047b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8160e80 con 0x55d92badc3b0 2024-06-29T14:35:17.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f47047b2740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46f81614d0 con 0x55d92bb05270 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92beab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92beab460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34968/0 (socket says 172.21.15.59:34968) 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 -- 172.21.15.59:0/4126862544 learned_addr learned my addr 172.21.15.59:0/4126862544 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 -- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 msgr2=0x55d92bf13c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f77fe640 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 0x55d92bf13c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92beaaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 0x55d92bf13c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 -- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 msgr2=0x55d92beaaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92beaaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 -- 172.21.15.59:0/4126862544 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d92bed2440 con 0x55d92bae4590 2024-06-29T14:35:17.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92beaaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:17.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46f67fc640 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92beab460 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f46e0060e90 tx=0x7f46e005d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86f6afcb3f6dc73 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:17.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46ecff9640 1 -- 172.21.15.59:0/4126862544 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e001d040 con 0x55d92bae4590 2024-06-29T14:35:17.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46ecff9640 1 -- 172.21.15.59:0/4126862544 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46e00967e0 con 0x55d92bae4590 2024-06-29T14:35:17.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.593+0000 7f46ecff9640 1 -- 172.21.15.59:0/4126862544 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0076120 con 0x55d92bae4590 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 -- 172.21.15.59:0/4126862544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 msgr2=0x55d92beab460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92beab460 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f46e0060e90 tx=0x7f46e005d910 comp rx=0 tx=0).stop 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 -- 172.21.15.59:0/4126862544 shutdown_connections 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92bb05270 0x55d92bf13c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bae4590 0x55d92beab460 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 --2- 172.21.15.59:0/4126862544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92badc3b0 0x55d92beaaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 -- 172.21.15.59:0/4126862544 >> 172.21.15.59:0/4126862544 conn(0x55d92baf93c0 msgr2=0x55d92bea1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 -- 172.21.15.59:0/4126862544 shutdown_connections 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.594+0000 7f47047b2740 1 -- 172.21.15.59:0/4126862544 wait complete. 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.595+0000 7f47047b2740 1 -- start start 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.595+0000 7f47047b2740 1 Processor -- start 2024-06-29T14:35:17.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f47047b2740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92badc3b0 0x55d92bf6d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f47047b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bae4590 0x55d92bf142a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f47047b2740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bb05270 0x55d92bee67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f47047b2740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d92becf600 con 0x55d92bb05270 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f47047b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d92bee8aa0 con 0x55d92bae4590 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f47047b2740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d92becf1c0 con 0x55d92badc3b0 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f47047b2740 10 client.0 _ll_register_callbacks cb 0x7f46e8010ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bb05270 0x55d92bee67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bb05270 0x55d92bee67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34970/0 (socket says 172.21.15.59:34970) 2024-06-29T14:35:17.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 -- 172.21.15.59:0/2510176265 learned_addr learned my addr 172.21.15.59:0/2510176265 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92badc3b0 0x55d92bf6d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f67fc640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bae4590 0x55d92bf142a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 -- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92badc3b0 msgr2=0x55d92bf6d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92badc3b0 0x55d92bf6d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 -- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bae4590 msgr2=0x55d92bf142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bae4590 0x55d92bf142a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 -- 172.21.15.59:0/2510176265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d92bee99a0 con 0x55d92bb05270 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92badc3b0 0x55d92bf6d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f67fc640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bae4590 0x55d92bf142a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46f77fe640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bb05270 0x55d92bee67d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f46f002ab50 tx=0x7f46f0004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ece99ac005dcdbf server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:17.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46d67fc640 1 -- 172.21.15.59:0/2510176265 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f004e060 con 0x55d92bb05270 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f46d67fc640 1 -- 172.21.15.59:0/2510176265 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f46f0004bb0 con 0x55d92bb05270 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.597+0000 7f47047b2740 1 -- 172.21.15.59:0/2510176265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46e00967e0 con 0x55d92bb05270 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f46d67fc640 1 -- 172.21.15.59:0/2510176265 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f46f0035040 con 0x55d92bb05270 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f47047b2740 10 client.6489 fetch_fsmap learned FSMap version 29 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f47047b2740 1 -- 172.21.15.59:0/2510176265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) -- 0x55d92bee6d10 con 0x55d92bb05270 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f46d67fc640 1 -- 172.21.15.59:0/2510176265 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46e00967e0 con 0x55d92bb05270 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f46d67fc640 1 -- 172.21.15.59:0/2510176265 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 29) ==== 3502+0+0 (secure 0 0 0) 0x7f46f003a040 con 0x55d92bb05270 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f47047b2740 10 client.6489 fetch_fsmap finished waiting for FSMap version 29 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f47047b2740 10 client.6489 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f47047b2740 20 client.6489 populate_metadata read hostname 'smithi059' 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f47047b2740 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] conn(0x55d92bf02160 0x55d92bf793e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f47047b2740 4 client.6489 mds_command: new command op to 6333 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f47047b2740 1 -- 172.21.15.59:0/2510176265 --> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] -- command(tid 0: {"prefix": "status"}) -- 0x55d92be1a020 con 0x55d92bf02160 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] conn(0x55d92bf02160 0x55d92bf793e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.598+0000 7f46d67fc640 10 client.6489 ms_handle_connect on v2:172.21.15.59:6833/3627888249 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f46f6ffd640 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] conn(0x55d92bf02160 0x55d92bf793e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6dbfc38adab805c server_cookie=1cbc7ef1c372e715 in_seq=0 out_seq=0 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f46d67fc640 1 -- 172.21.15.59:0/2510176265 <== mds.0 v2:172.21.15.59:6833/3627888249 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55d92be1a020 con 0x55d92bf02160 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f46d67fc640 10 client.6489 handle_command_reply: tid=0 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 client.6489 shutdown 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 -- 172.21.15.59:0/2510176265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bb05270 msgr2=0x55d92bee67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bb05270 0x55d92bee67d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f46f002ab50 tx=0x7f46f0004560 comp rx=0 tx=0).stop 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 -- 172.21.15.59:0/2510176265 shutdown_connections 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.59:6833/3627888249,v1:172.21.15.59:6835/3627888249] conn(0x55d92bf02160 0x55d92bf793e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d92bb05270 0x55d92bee67d0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d92bae4590 0x55d92bf142a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f46d67fc640 0 client.6489 ms_handle_reset on v2:172.21.15.59:6833/3627888249 2024-06-29T14:35:17.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 --2- 172.21.15.59:0/2510176265 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d92badc3b0 0x55d92bf6d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 -- 172.21.15.59:0/2510176265 >> 172.21.15.59:0/2510176265 conn(0x55d92baf93c0 msgr2=0x55d92bed1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 -- 172.21.15.59:0/2510176265 shutdown_connections 2024-06-29T14:35:17.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 1 -- 172.21.15.59:0/2510176265 wait complete. 2024-06-29T14:35:17.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.599+0000 7f47047b2740 20 client.6489 trim_cache size 0 max 16384 2024-06-29T14:35:17.604 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 6333, 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 10.469461768, 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 29, 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 71, 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 71, 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 21.703369740999999 2024-06-29T14:35:17.605 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:35:17.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f46d80763f0 msgr2=0x7f46d8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f46d80763f0 0x7f46d8078900 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46e8005cc0 tx=0x7f46e800a430 comp rx=0 tx=0).stop 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1092440098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 msgr2=0x7f46f80ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 0x7f46f80ccc20 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c9a0 tx=0x7f46e000ce60 comp rx=0 tx=0).stop 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1092440098 shutdown_connections 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f46d80763f0 0x7f46d8078900 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46f80cf5c0 0x7f46f8158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46f80ce950 0x7f46f80ccc20 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 --2- 172.21.15.59:0/1092440098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f808c190 0x7f46f815aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1092440098 >> 172.21.15.59:0/1092440098 conn(0x7f46f80b5610 msgr2=0x7f46f80c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1092440098 shutdown_connections 2024-06-29T14:35:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.600+0000 7f46fd8f6640 1 -- 172.21.15.59:0/1092440098 wait complete. 2024-06-29T14:35:17.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:35:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.728+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1887994477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec0b3ed0 msgr2=0x7fa4ec15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.728+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1887994477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec0b3ed0 0x7fa4ec15e8d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0d15d0 tx=0x7fa4e002a560 comp rx=0 tx=0).stop 2024-06-29T14:35:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.728+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1887994477 shutdown_connections 2024-06-29T14:35:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.728+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1887994477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec0b3ed0 0x7fa4ec15e8d0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.728+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1887994477 >> v2:172.21.15.153:3300/0 conn(0x7fa4ec0d23c0 0x7fa4ec0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.728+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1887994477 >> 172.21.15.59:0/1887994477 conn(0x7fa4ec0d3ee0 msgr2=0x7fa4ec0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.728+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1887994477 shutdown_connections 2024-06-29T14:35:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.728+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1887994477 wait complete. 2024-06-29T14:35:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.730+0000 7fa4f3ee4640 1 Processor -- start 2024-06-29T14:35:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.730+0000 7fa4f3ee4640 1 -- start start 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f3ee4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec0b3ed0 0x7fa4ec0ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f3ee4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0d23c0 0x7fa4ec155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f3ee4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec15ee10 0x7fa4ec1f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f3ee4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec160240 con 0x7fa4ec15ee10 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f3ee4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1600c0 con 0x7fa4ec0b3ed0 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f3ee4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ec1603c0 con 0x7fa4ec0d23c0 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec15ee10 0x7fa4ec1f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f1c80640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec0b3ed0 0x7fa4ec0ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec15ee10 0x7fa4ec1f8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34990/0 (socket says 172.21.15.59:34990) 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 -- 172.21.15.59:0/1207736906 learned_addr learned my addr 172.21.15.59:0/1207736906 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f147f640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0d23c0 0x7fa4ec155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 -- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0d23c0 msgr2=0x7fa4ec155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0d23c0 0x7fa4ec155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 -- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec0b3ed0 msgr2=0x7fa4ec0ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec0b3ed0 0x7fa4ec0ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 -- 172.21.15.59:0/1207736906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ec0cb790 con 0x7fa4ec15ee10 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f2481640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec15ee10 0x7fa4ec1f8be0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400b3e0 tx=0x7fa4e400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4ea7fc640 1 -- 172.21.15.59:0/1207736906 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4015030 con 0x7fa4ec15ee10 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4ea7fc640 1 -- 172.21.15.59:0/1207736906 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4e4004110 con 0x7fa4ec15ee10 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ec0cba70 con 0x7fa4ec15ee10 2024-06-29T14:35:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.732+0000 7fa4ea7fc640 1 -- 172.21.15.59:0/1207736906 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4011630 con 0x7fa4ec15ee10 2024-06-29T14:35:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.733+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ec1e91f0 con 0x7fa4ec15ee10 2024-06-29T14:35:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.733+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ec15e8d0 con 0x7fa4ec15ee10 2024-06-29T14:35:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.734+0000 7fa4ea7fc640 1 -- 172.21.15.59:0/1207736906 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fa4e4018070 con 0x7fa4ec15ee10 2024-06-29T14:35:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.735+0000 7fa4ea7fc640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fa4d80763f0 0x7fa4d8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.735+0000 7fa4ea7fc640 1 -- 172.21.15.59:0/1207736906 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7fa4e4097370 con 0x7fa4ec15ee10 2024-06-29T14:35:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.735+0000 7fa4f1c80640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fa4d80763f0 0x7fa4d8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.735+0000 7fa4f1c80640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fa4d80763f0 0x7fa4d8078900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0d2e40 tx=0x7fa4dc0095d0 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.739+0000 7fa4ea7fc640 1 -- 172.21.15.59:0/1207736906 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa4e4060470 con 0x7fa4ec15ee10 2024-06-29T14:35:17.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.971+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa4ec0d2950 con 0x7fa4ec15ee10 2024-06-29T14:35:17.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.971+0000 7fa4ea7fc640 1 -- 172.21.15.59:0/1207736906 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v29) ==== 53+0+73 (secure 0 0 0) 0x7fa4e4064770 con 0x7fa4ec15ee10 2024-06-29T14:35:17.969 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fa4d80763f0 msgr2=0x7fa4d8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fa4d80763f0 0x7fa4d8078900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0d2e40 tx=0x7fa4dc0095d0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec15ee10 msgr2=0x7fa4ec1f8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec15ee10 0x7fa4ec1f8be0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400b3e0 tx=0x7fa4e400b8a0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 shutdown_connections 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fa4d80763f0 0x7fa4d8078900 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4ec15ee10 0x7fa4ec1f8be0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4ec0d23c0 0x7fa4ec155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 --2- 172.21.15.59:0/1207736906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4ec0b3ed0 0x7fa4ec0ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 >> 172.21.15.59:0/1207736906 conn(0x7fa4ec0d3ee0 msgr2=0x7fa4ec157590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 shutdown_connections 2024-06-29T14:35:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:17.973+0000 7fa4f3ee4640 1 -- 172.21.15.59:0/1207736906 wait complete. 2024-06-29T14:35:17.982 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:35:17.982 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:35:17.983 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:35:17.983 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:35:17.983 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:35:17.983 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:35:17.983 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:35:17.983 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:35:17.983 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:35:17.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:17.983 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:35:18.000 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:35:18.000 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:35:18.000 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:35:18.000 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:18.000 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:35:18.000 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:18.000 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 41933sec preferred_lft 41933sec 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:18.001 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:35:18.002 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T14:35:18.003 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:18.003 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:35:18.003 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:35:18.003 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T14:35:18.003 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:18.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:18.003 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:35:18.064 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:35:18.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:18.066 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:35:18.123 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:35:18.123 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:35:18.123 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:35:18.123 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:35:18.124 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:35:18.124 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:35:18.124 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:35:18.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:35:18.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:35:18.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:35:18.124 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:35:18.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:35:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:35:18.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:18.255 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:35:18.271 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:35:18.271 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:35:18.271 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:35:18.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:18.272 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:35:18.337 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:35:18.337 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:35:18.393 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:35:18.393 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:35:18.393 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:35:18.449 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:35:18.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:18.449 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:35:18.544 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:35:18.544 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:35:18.544 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:35:18.555 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:35:18.555 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:35:18.555 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:35:18.555 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:35:18.555 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:35:18.562 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:35:18.562 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:35:18.562 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:35:18.562 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:35:18.562 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:35:18.562 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:35:18.562 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:35:18.562 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:35:18.563 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:35:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:35:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:35:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:35:18.563 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:35:18.564 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:35:18.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:18.565 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:35:18.590 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:35:18.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:35:18.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:35:18.693 INFO:teuthology.orchestra.run.smithi059.stdout:client6193 2024-06-29T14:35:18.694 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:35:18.694 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:35:18.710 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:35:18.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.870+0000 7f3c876f7640 1 -- 172.21.15.59:0/2789972831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c80141bd0 msgr2=0x7f3c8013edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:18.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.870+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2789972831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c80141bd0 0x7f3c8013edc0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3c80069670 tx=0x7f3c7002a980 comp rx=0 tx=0).stop 2024-06-29T14:35:18.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.870+0000 7f3c876f7640 1 -- 172.21.15.59:0/2789972831 shutdown_connections 2024-06-29T14:35:18.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.870+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2789972831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c80141bd0 0x7f3c8013edc0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:18.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.870+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2789972831 >> v2:172.21.15.153:3300/0 conn(0x7f3c8015ceb0 0x7f3c800b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:18.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.870+0000 7f3c876f7640 1 -- 172.21.15.59:0/2789972831 >> 172.21.15.59:0/2789972831 conn(0x7f3c800b6fb0 msgr2=0x7f3c800b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:18.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.870+0000 7f3c876f7640 1 -- 172.21.15.59:0/2789972831 shutdown_connections 2024-06-29T14:35:18.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.870+0000 7f3c876f7640 1 -- 172.21.15.59:0/2789972831 wait complete. 2024-06-29T14:35:18.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.872+0000 7f3c876f7640 1 Processor -- start 2024-06-29T14:35:18.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.872+0000 7f3c876f7640 1 -- start start 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c876f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8013e9b0 0x7f3c80156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c876f7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8015ceb0 0x7f3c8015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c876f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c800c6d50 0x7f3c801414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c876f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3c800b29d0 con 0x7f3c8013e9b0 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c876f7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c800b2850 con 0x7f3c800c6d50 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c876f7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3c800b2b50 con 0x7f3c8015ceb0 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c85493640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8013e9b0 0x7f3c80156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c85493640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8013e9b0 0x7f3c80156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35012/0 (socket says 172.21.15.59:35012) 2024-06-29T14:35:18.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.873+0000 7f3c85493640 1 -- 172.21.15.59:0/2790024728 learned_addr learned my addr 172.21.15.59:0/2790024728 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c85c94640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c800c6d50 0x7f3c801414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c84c92640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8015ceb0 0x7f3c8015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c85493640 1 -- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8015ceb0 msgr2=0x7f3c8015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c85493640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8015ceb0 0x7f3c8015b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c85493640 1 -- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c800c6d50 msgr2=0x7f3c801414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c85493640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c800c6d50 0x7f3c801414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c85493640 1 -- 172.21.15.59:0/2790024728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c80157360 con 0x7f3c8013e9b0 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c85c94640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c800c6d50 0x7f3c801414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c85493640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8013e9b0 0x7f3c80156a80 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800b470 tx=0x7f3c7800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c7dffb640 1 -- 172.21.15.59:0/2790024728 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78015020 con 0x7f3c8013e9b0 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c7dffb640 1 -- 172.21.15.59:0/2790024728 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c78011940 con 0x7f3c8013e9b0 2024-06-29T14:35:18.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c80157640 con 0x7f3c8013e9b0 2024-06-29T14:35:18.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c7dffb640 1 -- 172.21.15.59:0/2790024728 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78011d90 con 0x7f3c8013e9b0 2024-06-29T14:35:18.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.874+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c80157c00 con 0x7f3c8013e9b0 2024-06-29T14:35:18.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.875+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c8013edc0 con 0x7f3c8013e9b0 2024-06-29T14:35:18.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.876+0000 7f3c7dffb640 1 -- 172.21.15.59:0/2790024728 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f3c78018020 con 0x7f3c8013e9b0 2024-06-29T14:35:18.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.877+0000 7f3c7dffb640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f3c6c076420 0x7f3c6c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:18.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.877+0000 7f3c7dffb640 1 -- 172.21.15.59:0/2790024728 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f3c78097bc0 con 0x7f3c8013e9b0 2024-06-29T14:35:18.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.877+0000 7f3c84c92640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f3c6c076420 0x7f3c6c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:18.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.878+0000 7f3c84c92640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f3c6c076420 0x7f3c6c078930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c74006fd0 tx=0x7f3c74008040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:18.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:18.881+0000 7f3c7dffb640 1 -- 172.21.15.59:0/2790024728 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3c7801f020 con 0x7f3c8013e9b0 2024-06-29T14:35:19.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.095+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 --> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f3c801540c0 con 0x7f3c6c076420 2024-06-29T14:35:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.098+0000 7f3c7dffb640 1 -- 172.21.15.59:0/2790024728 <== mgr.5809 v2:172.21.15.153:6832/2957341575 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f3c801540c0 con 0x7f3c6c076420 2024-06-29T14:35:19.095 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:35:19.096 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:35:19.096 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T14:35:19.096 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:35:19.096 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f3c6c076420 msgr2=0x7f3c6c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f3c6c076420 0x7f3c6c078930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c74006fd0 tx=0x7f3c74008040 comp rx=0 tx=0).stop 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8013e9b0 msgr2=0x7f3c80156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8013e9b0 0x7f3c80156a80 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800b470 tx=0x7f3c7800b930 comp rx=0 tx=0).stop 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 shutdown_connections 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f3c6c076420 0x7f3c6c078930 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c800c6d50 0x7f3c801414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8015ceb0 0x7f3c8015b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 --2- 172.21.15.59:0/2790024728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8013e9b0 0x7f3c80156a80 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 >> 172.21.15.59:0/2790024728 conn(0x7f3c800b6fb0 msgr2=0x7f3c8015f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 shutdown_connections 2024-06-29T14:35:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.099+0000 7f3c876f7640 1 -- 172.21.15.59:0/2790024728 wait complete. 2024-06-29T14:35:19.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:35:19.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.232+0000 7f11fae7f640 1 -- 172.21.15.59:0/2359587255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11e00053a0 msgr2=0x7f11e000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:19.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.232+0000 7f11fae7f640 1 --2- 172.21.15.59:0/2359587255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11e00053a0 0x7f11e000ba20 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009b20 tx=0x7f11e0005830 comp rx=0 tx=0).stop 2024-06-29T14:35:19.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.232+0000 7f11fae7f640 1 -- 172.21.15.59:0/2359587255 shutdown_connections 2024-06-29T14:35:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.232+0000 7f11fae7f640 1 --2- 172.21.15.59:0/2359587255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11e00053a0 0x7f11e000ba20 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.232+0000 7f11fae7f640 1 -- 172.21.15.59:0/2359587255 >> 172.21.15.59:0/2359587255 conn(0x7f11f40d45b0 msgr2=0x7f11f40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.232+0000 7f11fae7f640 1 -- 172.21.15.59:0/2359587255 shutdown_connections 2024-06-29T14:35:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.232+0000 7f11fae7f640 1 -- 172.21.15.59:0/2359587255 wait complete. 2024-06-29T14:35:19.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.234+0000 7f11fae7f640 1 Processor -- start 2024-06-29T14:35:19.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.234+0000 7f11fae7f640 1 -- start start 2024-06-29T14:35:19.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11fae7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e00053a0 0x7f11f4159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11fae7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11f40c9ed0 0x7f11f40ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11fae7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11f40b3850 0x7f11f40b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11fae7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f11f4161a80 con 0x7f11f40c9ed0 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11fae7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11f40ce7b0 con 0x7f11f40b3850 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11fae7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11f4161c00 con 0x7f11e00053a0 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11f3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11f40c9ed0 0x7f11f40ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11f3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11f40c9ed0 0x7f11f40ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35034/0 (socket says 172.21.15.59:35034) 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11f3fff640 1 -- 172.21.15.59:0/3047008859 learned_addr learned my addr 172.21.15.59:0/3047008859 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11f941c640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11f40b3850 0x7f11f40b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.235+0000 7f11f8c1b640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e00053a0 0x7f11f4159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f3fff640 1 -- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e00053a0 msgr2=0x7f11f4159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f3fff640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e00053a0 0x7f11f4159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:19.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f3fff640 1 -- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11f40b3850 msgr2=0x7f11f40b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f3fff640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11f40b3850 0x7f11f40b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f3fff640 1 -- 172.21.15.59:0/3047008859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f40b41a0 con 0x7f11f40c9ed0 2024-06-29T14:35:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f3fff640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11f40c9ed0 0x7f11f40ca2e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f11dc006060 tx=0x7f11dc006520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f17fa640 1 -- 172.21.15.59:0/3047008859 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc00f890 con 0x7f11f40c9ed0 2024-06-29T14:35:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f17fa640 1 -- 172.21.15.59:0/3047008859 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11dc016020 con 0x7f11f40c9ed0 2024-06-29T14:35:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f415e8b0 con 0x7f11f40c9ed0 2024-06-29T14:35:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11f17fa640 1 -- 172.21.15.59:0/3047008859 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc00fcd0 con 0x7f11f40c9ed0 2024-06-29T14:35:19.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.236+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f415ee70 con 0x7f11f40c9ed0 2024-06-29T14:35:19.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.237+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f4162160 con 0x7f11f40c9ed0 2024-06-29T14:35:19.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.238+0000 7f11f17fa640 1 -- 172.21.15.59:0/3047008859 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f11dc011430 con 0x7f11f40c9ed0 2024-06-29T14:35:19.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.238+0000 7f11f17fa640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f11d4076420 0x7f11d4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:19.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.238+0000 7f11f17fa640 1 -- 172.21.15.59:0/3047008859 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7f11dc0986c0 con 0x7f11f40c9ed0 2024-06-29T14:35:19.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.238+0000 7f11f8c1b640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f11d4076420 0x7f11d4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:19.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.239+0000 7f11f8c1b640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f11d4076420 0x7f11d4078930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f11e40018c0 tx=0x7f11e4006040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:19.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.243+0000 7f11f17fa640 1 -- 172.21.15.59:0/3047008859 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11dc020020 con 0x7f11f40c9ed0 2024-06-29T14:35:19.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:19.488+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f11f406c4c0 con 0x7f11f40c9ed0 2024-06-29T14:35:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.196+0000 7f11f17fa640 1 -- 172.21.15.59:0/3047008859 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f11f40b41a0 con 0x7f11f40c9ed0 2024-06-29T14:35:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.199+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f11d4076420 msgr2=0x7f11d4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.199+0000 7f11fae7f640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f11d4076420 0x7f11d4078930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f11e40018c0 tx=0x7f11e4006040 comp rx=0 tx=0).stop 2024-06-29T14:35:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.199+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11f40c9ed0 msgr2=0x7f11f40ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.199+0000 7f11fae7f640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11f40c9ed0 0x7f11f40ca2e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f11dc006060 tx=0x7f11dc006520 comp rx=0 tx=0).stop 2024-06-29T14:35:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.201+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 shutdown_connections 2024-06-29T14:35:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.201+0000 7f11fae7f640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7f11d4076420 0x7f11d4078930 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.201+0000 7f11fae7f640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11f40b3850 0x7f11f40b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.201+0000 7f11fae7f640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11f40c9ed0 0x7f11f40ca2e0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.201+0000 7f11fae7f640 1 --2- 172.21.15.59:0/3047008859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e00053a0 0x7f11f4159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.201+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 >> 172.21.15.59:0/3047008859 conn(0x7f11f40d45b0 msgr2=0x7f11f415be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.201+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 shutdown_connections 2024-06-29T14:35:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.201+0000 7f11fae7f640 1 -- 172.21.15.59:0/3047008859 wait complete. 2024-06-29T14:35:20.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:35:20.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.366+0000 7fe29113c640 1 -- 172.21.15.59:0/2220110671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c166da0 msgr2=0x7fe28c1691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.366+0000 7fe29113c640 1 --2- 172.21.15.59:0/2220110671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c166da0 0x7fe28c1691c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe284009be0 tx=0x7fe28402a920 comp rx=0 tx=0).stop 2024-06-29T14:35:20.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.366+0000 7fe29113c640 1 -- 172.21.15.59:0/2220110671 shutdown_connections 2024-06-29T14:35:20.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.366+0000 7fe29113c640 1 --2- 172.21.15.59:0/2220110671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c166da0 0x7fe28c1691c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.366+0000 7fe29113c640 1 -- 172.21.15.59:0/2220110671 >> 172.21.15.59:0/2220110671 conn(0x7fe28c0d3ee0 msgr2=0x7fe28c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:20.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.366+0000 7fe29113c640 1 -- 172.21.15.59:0/2220110671 shutdown_connections 2024-06-29T14:35:20.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.366+0000 7fe29113c640 1 -- 172.21.15.59:0/2220110671 wait complete. 2024-06-29T14:35:20.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.368+0000 7fe29113c640 1 Processor -- start 2024-06-29T14:35:20.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.368+0000 7fe29113c640 1 -- start start 2024-06-29T14:35:20.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe29113c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c0b3990 0x7fe28c150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:20.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe29113c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28c15e290 0x7fe28c15ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:20.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe29113c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28c166da0 0x7fe28c15b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:20.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe29113c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c169700 con 0x7fe28c166da0 2024-06-29T14:35:20.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe29113c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0cea30 con 0x7fe28c15e290 2024-06-29T14:35:20.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe29113c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe28c169880 con 0x7fe28c0b3990 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28c166da0 0x7fe28c15b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28c166da0 0x7fe28c15b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35052/0 (socket says 172.21.15.59:35052) 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28b59e640 1 -- 172.21.15.59:0/2868940401 learned_addr learned my addr 172.21.15.59:0/2868940401 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28ad9d640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c0b3990 0x7fe28c150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28a59c640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28c15e290 0x7fe28c15ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28b59e640 1 -- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c0b3990 msgr2=0x7fe28c150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28b59e640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c0b3990 0x7fe28c150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28b59e640 1 -- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28c15e290 msgr2=0x7fe28c15ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28b59e640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28c15e290 0x7fe28c15ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28b59e640 1 -- 172.21.15.59:0/2868940401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c159550 con 0x7fe28c166da0 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28ad9d640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c0b3990 0x7fe28c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.370+0000 7fe28a59c640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28c15e290 0x7fe28c15ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:35:20.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.371+0000 7fe28b59e640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28c166da0 0x7fe28c15b0d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe284009be0 tx=0x7fe284003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:20.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.371+0000 7fe2737fe640 1 -- 172.21.15.59:0/2868940401 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284030c50 con 0x7fe28c166da0 2024-06-29T14:35:20.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.371+0000 7fe2737fe640 1 -- 172.21.15.59:0/2868940401 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe28403e070 con 0x7fe28c166da0 2024-06-29T14:35:20.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.371+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c159830 con 0x7fe28c166da0 2024-06-29T14:35:20.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.371+0000 7fe2737fe640 1 -- 172.21.15.59:0/2868940401 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284032330 con 0x7fe28c166da0 2024-06-29T14:35:20.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.371+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c159df0 con 0x7fe28c166da0 2024-06-29T14:35:20.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.372+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe28c0ce110 con 0x7fe28c166da0 2024-06-29T14:35:20.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.373+0000 7fe2737fe640 1 -- 172.21.15.59:0/2868940401 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) ==== 98320+0+0 (secure 0 0 0) 0x7fe2840324d0 con 0x7fe28c166da0 2024-06-29T14:35:20.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.373+0000 7fe2737fe640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe278076440 0x7fe278078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:20.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.373+0000 7fe28ad9d640 1 -- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe278076440 msgr2=0x7fe278078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2957341575 2024-06-29T14:35:20.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.373+0000 7fe2737fe640 1 -- 172.21.15.59:0/2868940401 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7fe2840b91f0 con 0x7fe28c166da0 2024-06-29T14:35:20.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.373+0000 7fe28ad9d640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe278076440 0x7fe278078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:35:20.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.382+0000 7fe2737fe640 1 -- 172.21.15.59:0/2868940401 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe28402d070 con 0x7fe28c166da0 2024-06-29T14:35:20.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.574+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fe28c15db20 con 0x7fe28c166da0 2024-06-29T14:35:20.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.575+0000 7fe28ad9d640 1 -- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe278076440 msgr2=0x7fe278078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2957341575 2024-06-29T14:35:20.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.575+0000 7fe28ad9d640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe278076440 0x7fe278078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:35:20.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.579+0000 7fe2737fe640 1 -- 172.21.15.59:0/2868940401 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v33)=0 v33) ==== 138+0+0 (secure 0 0 0) 0x7fe2840821e0 con 0x7fe28c166da0 2024-06-29T14:35:20.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe278076440 msgr2=0x7fe278078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:35:20.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe278076440 0x7fe278078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28c166da0 msgr2=0x7fe28c15b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28c166da0 0x7fe28c15b0d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe284009be0 tx=0x7fe284003ba0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 shutdown_connections 2024-06-29T14:35:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fe278076440 0x7fe278078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28c166da0 0x7fe28c15b0d0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe28c15e290 0x7fe28c15ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 --2- 172.21.15.59:0/2868940401 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28c0b3990 0x7fe28c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 >> 172.21.15.59:0/2868940401 conn(0x7fe28c0d3ee0 msgr2=0x7fe28c0b8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 shutdown_connections 2024-06-29T14:35:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.582+0000 7fe29113c640 1 -- 172.21.15.59:0/2868940401 wait complete. 2024-06-29T14:35:20.592 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:35:20.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.715+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1487378993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815f360 msgr2=0x7fd3f80ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.715+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1487378993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815f360 0x7fd3f80ce8c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0005fd0 tx=0x7fd3f002aa90 comp rx=0 tx=0).stop 2024-06-29T14:35:20.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.715+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1487378993 shutdown_connections 2024-06-29T14:35:20.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.715+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1487378993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815f360 0x7fd3f80ce8c0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.715+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1487378993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3f8140350 0x7fd3f8140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.715+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1487378993 >> 172.21.15.59:0/1487378993 conn(0x7fd3f8072d80 msgr2=0x7fd3f8073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:20.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.715+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1487378993 shutdown_connections 2024-06-29T14:35:20.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.715+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1487378993 wait complete. 2024-06-29T14:35:20.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.717+0000 7fd3ffbd4640 1 Processor -- start 2024-06-29T14:35:20.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.717+0000 7fd3ffbd4640 1 -- start start 2024-06-29T14:35:20.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3ffbd4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f8140350 0x7fd3f813a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3ffbd4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815e840 0x7fd3f8141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3ffbd4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3f815f360 0x7fd3f80c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3ffbd4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f8141150 con 0x7fd3f815e840 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3ffbd4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f8140fd0 con 0x7fd3f8140350 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3ffbd4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd3f80cee00 con 0x7fd3f815f360 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3fd16f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815e840 0x7fd3f8141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3fd970640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f8140350 0x7fd3f813a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3fd16f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815e840 0x7fd3f8141360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35060/0 (socket says 172.21.15.59:35060) 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3fe171640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3f815f360 0x7fd3f80c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.718+0000 7fd3fd16f640 1 -- 172.21.15.59:0/1547645142 learned_addr learned my addr 172.21.15.59:0/1547645142 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3fd16f640 1 -- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3f815f360 msgr2=0x7fd3f80c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3fd16f640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3f815f360 0x7fd3f80c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3fd16f640 1 -- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f8140350 msgr2=0x7fd3f813a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3fd16f640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f8140350 0x7fd3f813a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3fd16f640 1 -- 172.21.15.59:0/1547645142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f81418a0 con 0x7fd3f815e840 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3fd16f640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815e840 0x7fd3f8141360 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400c370 tx=0x7fd3f400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3ea7fc640 1 -- 172.21.15.59:0/1547645142 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4017070 con 0x7fd3f815e840 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3ea7fc640 1 -- 172.21.15.59:0/1547645142 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3f40053c0 con 0x7fd3f815e840 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f80c67b0 con 0x7fd3f815e840 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3ea7fc640 1 -- 172.21.15.59:0/1547645142 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4003850 con 0x7fd3f815e840 2024-06-29T14:35:20.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.719+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f81f5ff0 con 0x7fd3f815e840 2024-06-29T14:35:20.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.720+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3f8140760 con 0x7fd3f815e840 2024-06-29T14:35:20.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.722+0000 7fd3ea7fc640 1 -- 172.21.15.59:0/1547645142 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) ==== 98320+0+0 (secure 0 0 0) 0x7fd3f401a070 con 0x7fd3f815e840 2024-06-29T14:35:20.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.722+0000 7fd3ea7fc640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd3dc076470 0x7fd3dc078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:20.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.722+0000 7fd3fd970640 1 -- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd3dc076470 msgr2=0x7fd3dc078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2957341575 2024-06-29T14:35:20.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.722+0000 7fd3ea7fc640 1 -- 172.21.15.59:0/1547645142 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7fd3f4096dc0 con 0x7fd3f815e840 2024-06-29T14:35:20.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.722+0000 7fd3fd970640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd3dc076470 0x7fd3dc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:35:20.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.727+0000 7fd3ea7fc640 1 -- 172.21.15.59:0/1547645142 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd3f8140ca0 con 0x7fd3f815e840 2024-06-29T14:35:20.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.921+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fd3f813a7b0 con 0x7fd3f815e840 2024-06-29T14:35:20.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.923+0000 7fd3fd970640 1 -- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd3dc076470 msgr2=0x7fd3dc078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2957341575 2024-06-29T14:35:20.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.923+0000 7fd3fd970640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd3dc076470 0x7fd3dc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:35:20.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.926+0000 7fd3ea7fc640 1 -- 172.21.15.59:0/1547645142 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v34)=0 v34) ==== 133+0+0 (secure 0 0 0) 0x7fd3f405fda0 con 0x7fd3f815e840 2024-06-29T14:35:20.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd3dc076470 msgr2=0x7fd3dc078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:35:20.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd3dc076470 0x7fd3dc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815e840 msgr2=0x7fd3f8141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:20.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815e840 0x7fd3f8141360 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400c370 tx=0x7fd3f400c830 comp rx=0 tx=0).stop 2024-06-29T14:35:20.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 shutdown_connections 2024-06-29T14:35:20.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fd3dc076470 0x7fd3dc078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3f815f360 0x7fd3f80c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3f815e840 0x7fd3f8141360 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 --2- 172.21.15.59:0/1547645142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3f8140350 0x7fd3f813a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:20.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 >> 172.21.15.59:0/1547645142 conn(0x7fd3f8072d80 msgr2=0x7fd3f813e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:20.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 shutdown_connections 2024-06-29T14:35:20.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:20.929+0000 7fd3ffbd4640 1 -- 172.21.15.59:0/1547645142 wait complete. 2024-06-29T14:35:20.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:20.938 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:35:20.957 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-06-29T14:35:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.122+0000 7fea47b12640 1 -- 172.21.15.59:0/2995862853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea400b2fa0 msgr2=0x7fea400d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.122+0000 7fea47b12640 1 --2- 172.21.15.59:0/2995862853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea400b2fa0 0x7fea400d1520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea30009860 tx=0x7fea3002a5a0 comp rx=0 tx=0).stop 2024-06-29T14:35:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.122+0000 7fea47b12640 1 -- 172.21.15.59:0/2995862853 shutdown_connections 2024-06-29T14:35:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.122+0000 7fea47b12640 1 --2- 172.21.15.59:0/2995862853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea400b2fa0 0x7fea400d1520 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.122+0000 7fea47b12640 1 -- 172.21.15.59:0/2995862853 >> 172.21.15.59:0/2995862853 conn(0x7fea40073e10 msgr2=0x7fea40074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.122+0000 7fea47b12640 1 -- 172.21.15.59:0/2995862853 shutdown_connections 2024-06-29T14:35:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.122+0000 7fea47b12640 1 -- 172.21.15.59:0/2995862853 wait complete. 2024-06-29T14:35:21.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.124+0000 7fea47b12640 1 Processor -- start 2024-06-29T14:35:21.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.124+0000 7fea47b12640 1 -- start start 2024-06-29T14:35:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea47b12640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea400b2270 0x7fea40070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea47b12640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea400b2fa0 0x7fea400710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea47b12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea400d1a60 0x7fea40071600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea47b12640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fea400b65d0 con 0x7fea400b2270 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea47b12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea400b6450 con 0x7fea400d1a60 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea47b12640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fea400b6750 con 0x7fea400b2fa0 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea458ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea400b2270 0x7fea40070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea458ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea400b2270 0x7fea40070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35066/0 (socket says 172.21.15.59:35066) 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea458ae640 1 -- 172.21.15.59:0/437037219 learned_addr learned my addr 172.21.15.59:0/437037219 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea450ad640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea400b2fa0 0x7fea400710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.125+0000 7fea460af640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea400d1a60 0x7fea40071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea458ae640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea400b2fa0 msgr2=0x7fea400710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea458ae640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea400b2fa0 0x7fea400710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea458ae640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea400d1a60 msgr2=0x7fea40071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea458ae640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea400d1a60 0x7fea40071600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea458ae640 1 -- 172.21.15.59:0/437037219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4006db80 con 0x7fea400b2270 2024-06-29T14:35:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea458ae640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea400b2270 0x7fea40070b80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00c3e0 tx=0x7fea3c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c011790 con 0x7fea400b2270 2024-06-29T14:35:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea3c017070 con 0x7fea400b2270 2024-06-29T14:35:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4015f020 con 0x7fea400b2270 2024-06-29T14:35:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c011ca0 con 0x7fea400b2270 2024-06-29T14:35:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.126+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4015c650 con 0x7fea400b2270 2024-06-29T14:35:21.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.128+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea4006dd50 con 0x7fea400b2270 2024-06-29T14:35:21.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.128+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) ==== 98320+0+0 (secure 0 0 0) 0x7fea3c01c460 con 0x7fea400b2270 2024-06-29T14:35:21.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.129+0000 7fea367fc640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 0x7fea24078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:21.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.129+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6101+0+0 (secure 0 0 0) 0x7fea3c097880 con 0x7fea400b2270 2024-06-29T14:35:21.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.129+0000 7fea450ad640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 msgr2=0x7fea24078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2957341575 2024-06-29T14:35:21.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.129+0000 7fea450ad640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 0x7fea24078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:35:21.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.133+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fea3c0608a0 con 0x7fea400b2270 2024-06-29T14:35:21.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.330+0000 7fea450ad640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 msgr2=0x7fea24078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2957341575 2024-06-29T14:35:21.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.330+0000 7fea450ad640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 0x7fea24078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:35:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.348+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 --> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fea40153d60 con 0x7fea24076440 2024-06-29T14:35:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.731+0000 7fea450ad640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 msgr2=0x7fea24078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2957341575 2024-06-29T14:35:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:21.731+0000 7fea450ad640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 0x7fea24078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:35:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:22.532+0000 7fea450ad640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 msgr2=0x7fea24078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2957341575 2024-06-29T14:35:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:22.532+0000 7fea450ad640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 0x7fea24078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:35:23.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:23.836+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 20) ==== 97929+0+0 (secure 0 0 0) 0x7fea3c095c10 con 0x7fea400b2270 2024-06-29T14:35:23.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:23.837+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 msgr2=0x7fea24078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:35:23.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:23.837+0000 7fea367fc640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2957341575,v1:172.21.15.153:6833/2957341575] conn(0x7fea24076440 0x7fea24078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:23.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa81f43b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa81f43b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa81f43b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa81f43b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa81f43b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.983 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa822441640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.983 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa822441640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.983 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa822441640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.983 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa822441640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:23.983 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:35:23.984+0000 7fa822441640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:24.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.839+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 21) ==== 98056+0+0 (secure 0 0 0) 0x7fea3c007570 con 0x7fea400b2270 2024-06-29T14:35:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.840+0000 7fea367fc640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fea2407e990 0x7fea24080db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.840+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 --> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fea3c064990 con 0x7fea2407e990 2024-06-29T14:35:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.840+0000 7fea450ad640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fea2407e990 0x7fea24080db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:24.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.842+0000 7fea450ad640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fea2407e990 0x7fea24080db0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fea30034000 tx=0x7fea30004730 comp rx=0 tx=0).ready entity=mgr.6214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:24.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.906+0000 7fea367fc640 1 -- 172.21.15.59:0/437037219 <== mgr.6214 v2:172.21.15.153:6832/2118890682 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fea3c064990 con 0x7fea2407e990 2024-06-29T14:35:24.904 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fea2407e990 msgr2=0x7fea24080db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea47b12640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fea2407e990 0x7fea24080db0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fea30034000 tx=0x7fea30004730 comp rx=0 tx=0).stop 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea400b2270 msgr2=0x7fea40070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea47b12640 1 --2- 172.21.15.59:0/437037219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea400b2270 0x7fea40070b80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00c3e0 tx=0x7fea3c00c8a0 comp rx=0 tx=0).stop 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea460af640 1 -- 172.21.15.59:0/437037219 reap_dead start 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 shutdown_connections 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 >> 172.21.15.59:0/437037219 conn(0x7fea40073e10 msgr2=0x7fea4015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 shutdown_connections 2024-06-29T14:35:24.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:24.908+0000 7fea47b12640 1 -- 172.21.15.59:0/437037219 wait complete. 2024-06-29T14:35:24.918 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T14:35:24.919 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-06-29T14:35:25.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.043+0000 7f5309568640 1 -- 172.21.15.59:0/4110471699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53040cb650 msgr2=0x7f53040acfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.043+0000 7f5309568640 1 --2- 172.21.15.59:0/4110471699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53040cb650 0x7f53040acfa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f52fc005fd0 tx=0x7f52fc02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.044+0000 7f5309568640 1 -- 172.21.15.59:0/4110471699 shutdown_connections 2024-06-29T14:35:25.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.044+0000 7f5309568640 1 --2- 172.21.15.59:0/4110471699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53040cb650 0x7f53040acfa0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.044+0000 7f5309568640 1 --2- 172.21.15.59:0/4110471699 >> v2:172.21.15.153:3300/0 conn(0x7f53040b52b0 0x7f53040b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.044+0000 7f5309568640 1 -- 172.21.15.59:0/4110471699 >> 172.21.15.59:0/4110471699 conn(0x7f53040b1ea0 msgr2=0x7f53040b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:25.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.044+0000 7f5309568640 1 -- 172.21.15.59:0/4110471699 shutdown_connections 2024-06-29T14:35:25.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.044+0000 7f5309568640 1 -- 172.21.15.59:0/4110471699 wait complete. 2024-06-29T14:35:25.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.045+0000 7f5309568640 1 Processor -- start 2024-06-29T14:35:25.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.046+0000 7f5309568640 1 -- start start 2024-06-29T14:35:25.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f5309568640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53040b52b0 0x7f530409cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:25.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f5309568640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53040cb650 0x7f5304073430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:25.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f5309568640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f530415be90 0x7f53040cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:25.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f5309568640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f53040ad4e0 con 0x7f530415be90 2024-06-29T14:35:25.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f5309568640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53040cbc70 con 0x7f53040cb650 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f5309568640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f53040ad660 con 0x7f53040b52b0 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f530415be90 0x7f53040cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f530415be90 0x7f53040cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60026/0 (socket says 172.21.15.59:60026) 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 -- 172.21.15.59:0/3117326051 learned_addr learned my addr 172.21.15.59:0/3117326051 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f5302ffd640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53040b52b0 0x7f530409cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53027fc640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53040cb650 0x7f5304073430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 -- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53040b52b0 msgr2=0x7f530409cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53040b52b0 0x7f530409cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 -- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53040cb650 msgr2=0x7f5304073430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53040cb650 0x7f5304073430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 -- 172.21.15.59:0/3117326051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530409d000 con 0x7f530415be90 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.047+0000 7f53037fe640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f530415be90 0x7f53040cbdf0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f52fc02aea0 tx=0x7f52fc0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:25.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.048+0000 7f52fb7fe640 1 -- 172.21.15.59:0/3117326051 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc031da0 con 0x7f530415be90 2024-06-29T14:35:25.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.048+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530409d510 con 0x7f530415be90 2024-06-29T14:35:25.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.048+0000 7f52fb7fe640 1 -- 172.21.15.59:0/3117326051 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52fc003a60 con 0x7f530415be90 2024-06-29T14:35:25.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.048+0000 7f52fb7fe640 1 -- 172.21.15.59:0/3117326051 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc0330d0 con 0x7f530415be90 2024-06-29T14:35:25.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.048+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5304072c30 con 0x7f530415be90 2024-06-29T14:35:25.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.049+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53040adbb0 con 0x7f530415be90 2024-06-29T14:35:25.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.050+0000 7f52fb7fe640 1 -- 172.21.15.59:0/3117326051 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 21) ==== 98056+0+0 (secure 0 0 0) 0x7f52fc033270 con 0x7f530415be90 2024-06-29T14:35:25.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.050+0000 7f52fb7fe640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f52e40761a0 0x7f52e40786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:25.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.050+0000 7f52fb7fe640 1 -- 172.21.15.59:0/3117326051 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6445+0+0 (secure 0 0 0) 0x7f52fc0b86a0 con 0x7f530415be90 2024-06-29T14:35:25.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.050+0000 7f5302ffd640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f52e40761a0 0x7f52e40786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:25.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.051+0000 7f5302ffd640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f52e40761a0 0x7f52e40786b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0095f0 tx=0x7f52ec005f90 comp rx=0 tx=0).ready entity=mgr.6214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:25.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.055+0000 7f52fb7fe640 1 -- 172.21.15.59:0/3117326051 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f52fc04a030 con 0x7f530415be90 2024-06-29T14:35:25.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.266+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 --> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) -- 0x7f530409d230 con 0x7f52e40761a0 2024-06-29T14:35:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.319+0000 7f52fb7fe640 1 -- 172.21.15.59:0/3117326051 <== mgr.6214 v2:172.21.15.153:6832/2118890682 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f530409d230 con 0x7f52e40761a0 2024-06-29T14:35:25.318 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1 2024-06-29T14:35:25.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f52e40761a0 msgr2=0x7f52e40786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f52e40761a0 0x7f52e40786b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0095f0 tx=0x7f52ec005f90 comp rx=0 tx=0).stop 2024-06-29T14:35:25.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f530415be90 msgr2=0x7f53040cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f530415be90 0x7f53040cbdf0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f52fc02aea0 tx=0x7f52fc0316f0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 shutdown_connections 2024-06-29T14:35:25.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f52e40761a0 0x7f52e40786b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f530415be90 0x7f53040cbdf0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53040cb650 0x7f5304073430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 --2- 172.21.15.59:0/3117326051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53040b52b0 0x7f530409cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 >> 172.21.15.59:0/3117326051 conn(0x7f53040b1ea0 msgr2=0x7f53040b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:25.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 shutdown_connections 2024-06-29T14:35:25.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.322+0000 7f5309568640 1 -- 172.21.15.59:0/3117326051 wait complete. 2024-06-29T14:35:25.331 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T14:35:25.331 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-14_36_00 in ~35s... 2024-06-29T14:35:25.332 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-14_37_00 in ~95s... 2024-06-29T14:35:25.332 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-06-29T14:35:25.332 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-06-29T14:35:25.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.453+0000 7f286134a640 1 -- 172.21.15.59:0/3802677705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0ce950 msgr2=0x7f285c0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.453+0000 7f286134a640 1 --2- 172.21.15.59:0/3802677705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0ce950 0x7f285c0d2130 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2854005fd0 tx=0x7f285402a4f0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.453+0000 7f286134a640 1 -- 172.21.15.59:0/3802677705 shutdown_connections 2024-06-29T14:35:25.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.453+0000 7f286134a640 1 --2- 172.21.15.59:0/3802677705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0ce950 0x7f285c0d2130 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.453+0000 7f286134a640 1 --2- 172.21.15.59:0/3802677705 >> v2:172.21.15.153:3300/0 conn(0x7f285c08c190 0x7f285c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.453+0000 7f286134a640 1 -- 172.21.15.59:0/3802677705 >> 172.21.15.59:0/3802677705 conn(0x7f285c0b5610 msgr2=0x7f285c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:25.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.453+0000 7f286134a640 1 -- 172.21.15.59:0/3802677705 shutdown_connections 2024-06-29T14:35:25.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.453+0000 7f286134a640 1 -- 172.21.15.59:0/3802677705 wait complete. 2024-06-29T14:35:25.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.455+0000 7f286134a640 1 Processor -- start 2024-06-29T14:35:25.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.455+0000 7f286134a640 1 -- start start 2024-06-29T14:35:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.456+0000 7f286134a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f285c08c190 0x7f285c0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.456+0000 7f286134a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f285c0ce950 0x7f285c1f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.456+0000 7f286134a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0cf5c0 0x7f285c0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.456+0000 7f286134a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f285c0b7f60 con 0x7f285c0cf5c0 2024-06-29T14:35:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.456+0000 7f286134a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f285c0b7de0 con 0x7f285c0ce950 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.456+0000 7f286134a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f285c0b80e0 con 0x7f285c08c190 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.456+0000 7f285b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0cf5c0 0x7f285c0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f285c08c190 0x7f285c0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f285c08c190 0x7f285c0cb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:43996/0 (socket says 172.21.15.59:43996) 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0cf5c0 0x7f285c0cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60032/0 (socket says 172.21.15.59:60032) 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285affd640 1 -- 172.21.15.59:0/908666151 learned_addr learned my addr 172.21.15.59:0/908666151 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285a7fc640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f285c0ce950 0x7f285c1f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285b7fe640 1 -- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f285c08c190 msgr2=0x7f285c0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285b7fe640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f285c08c190 0x7f285c0cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285b7fe640 1 -- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f285c0ce950 msgr2=0x7f285c1f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285b7fe640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f285c0ce950 0x7f285c1f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285b7fe640 1 -- 172.21.15.59:0/908666151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c160f20 con 0x7f285c0cf5c0 2024-06-29T14:35:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f285b7fe640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0cf5c0 0x7f285c0cd020 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2854031000 tx=0x7f28540039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f28537fe640 1 -- 172.21.15.59:0/908666151 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285403e030 con 0x7f285c0cf5c0 2024-06-29T14:35:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f28537fe640 1 -- 172.21.15.59:0/908666151 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28540047a0 con 0x7f285c0cf5c0 2024-06-29T14:35:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285c161200 con 0x7f285c0cf5c0 2024-06-29T14:35:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f28537fe640 1 -- 172.21.15.59:0/908666151 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854035040 con 0x7f285c0cf5c0 2024-06-29T14:35:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.457+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285c1617c0 con 0x7f285c0cf5c0 2024-06-29T14:35:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.458+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f285c0cf080 con 0x7f285c0cf5c0 2024-06-29T14:35:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.459+0000 7f28537fe640 1 -- 172.21.15.59:0/908666151 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 21) ==== 98056+0+0 (secure 0 0 0) 0x7f2854041030 con 0x7f285c0cf5c0 2024-06-29T14:35:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.459+0000 7f28537fe640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f283c076170 0x7f283c078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.460+0000 7f28537fe640 1 -- 172.21.15.59:0/908666151 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6445+0+0 (secure 0 0 0) 0x7f28540b7e10 con 0x7f285c0cf5c0 2024-06-29T14:35:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.460+0000 7f285affd640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f283c076170 0x7f283c078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.460+0000 7f285affd640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f283c076170 0x7f283c078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f284c007ee0 tx=0x7f284c0075d0 comp rx=0 tx=0).ready entity=mgr.6214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:25.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.464+0000 7f28537fe640 1 -- 172.21.15.59:0/908666151 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f28540bb030 con 0x7f285c0cf5c0 2024-06-29T14:35:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.677+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 --> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f285c15d300 con 0x7f283c076170 2024-06-29T14:35:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.684+0000 7f28537fe640 1 -- 172.21.15.59:0/908666151 <== mgr.6214 v2:172.21.15.153:6832/2118890682 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+76 (secure 0 0 0) 0x7f285c15d300 con 0x7f283c076170 2024-06-29T14:35:25.682 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:35:25.682 INFO:teuthology.orchestra.run.smithi059.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-06-29T14:35:25.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f283c076170 msgr2=0x7f283c078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f283c076170 0x7f283c078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f284c007ee0 tx=0x7f284c0075d0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0cf5c0 msgr2=0x7f285c0cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:25.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0cf5c0 0x7f285c0cd020 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2854031000 tx=0x7f28540039e0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 shutdown_connections 2024-06-29T14:35:25.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f283c076170 0x7f283c078680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f285c0cf5c0 0x7f285c0cd020 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f285c0ce950 0x7f285c1f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 --2- 172.21.15.59:0/908666151 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f285c08c190 0x7f285c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:25.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 >> 172.21.15.59:0/908666151 conn(0x7f285c0b5610 msgr2=0x7f285c0d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:25.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 shutdown_connections 2024-06-29T14:35:25.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:35:25.686+0000 7f286134a640 1 -- 172.21.15.59:0/908666151 wait complete. 2024-06-29T14:35:25.695 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-06-29T14:35:25.695 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-06-29T14:35:25.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:25.696 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:25.712 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:25.712 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:25.712 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:25.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:35:25.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:35:26.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:26.714 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:26.734 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:26.734 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:26.734 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:27.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:27.736 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:27.757 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:27.757 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:27.757 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:28.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:28.758 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:28.779 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:28.779 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:28.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:29.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:29.781 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:29.802 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:29.802 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:29.802 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:30.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:30.803 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:30.824 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:30.824 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:30.825 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:31.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:31.825 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:31.846 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:31.846 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:31.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:32.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:32.848 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:32.869 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:32.869 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:32.869 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:33.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:33.870 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:33.891 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:33.891 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:33.891 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:34.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:34.892 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:34.913 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:34.913 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:34.913 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:35.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:35.914 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:35.935 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:35.935 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:35.935 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:36.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:36.936 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:36.957 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:36.957 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:36.957 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:37.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:37.958 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:37.979 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:37.979 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:37.979 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:38.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:38.980 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:39.001 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:39.001 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:39.001 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:40.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:40.002 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:40.023 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:40.023 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:40.023 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:41.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:41.024 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:41.045 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:41.045 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:41.045 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:42.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:42.046 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:42.067 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:42.067 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:42.067 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:43.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:43.068 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:43.089 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:43.089 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:43.089 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:44.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:44.090 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:44.111 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:44.111 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:44.111 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:45.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:45.112 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:45.133 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:45.133 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:45.133 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:46.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:46.134 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:46.156 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:46.156 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:46.156 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:47.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:47.157 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:47.177 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:47.177 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:47.178 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:48.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:48.179 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:48.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:48.200 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:48.200 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:49.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:49.201 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:49.222 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:49.222 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:49.222 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:50.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:50.223 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:50.244 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:50.244 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:50.244 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:51.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:51.245 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:51.267 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:51.267 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:51.267 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:52.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:52.268 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:52.289 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:52.289 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:52.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:53.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:53.290 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:53.311 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:53.312 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:53.312 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:54.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:54.313 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:54.334 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:54.334 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:54.334 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:55.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:55.335 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:55.356 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:55.356 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:55.356 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:55.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:35:55.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:35:56.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:56.357 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:56.378 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:56.378 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:56.378 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:57.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:57.379 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:57.400 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:57.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:57.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:58.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:58.401 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:58.422 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:58.422 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:58.422 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:59.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:59.423 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:59.444 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:59.444 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:59.444 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:00.329 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:36:00.332+0000 7fa81141f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.330 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:36:00.332+0000 7fa81141f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.330 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:36:00.332+0000 7fa81141f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.330 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:36:00.332+0000 7fa81141f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.330 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:36:00.332+0000 7fa81141f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:00.445 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:00.463 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:00.464 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:00.464 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:00.464 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:00.464 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-06-29-14_36_00_UTC'] 2024-06-29T14:36:00.464 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719671725.331854, now = 1719671760.464576, timo = 35 2024-06-29T14:36:01.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:01.465 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:01.486 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:01.487 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:01.487 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:01.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:02.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:02.488 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:02.509 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:02.509 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:02.509 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:02.509 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:03.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:03.510 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:03.531 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:03.531 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:03.532 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:03.532 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:04.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:04.533 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:04.553 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:04.554 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:04.554 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:04.554 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:05.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:05.555 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:05.575 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:05.576 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:05.576 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:05.576 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:06.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:06.577 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:06.596 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:06.597 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:06.598 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:06.598 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:07.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:07.599 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:07.619 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:07.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:07.620 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:07.620 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:08.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:08.621 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:08.642 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:08.643 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:08.643 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:08.643 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:09.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:09.645 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:09.665 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:09.666 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:09.666 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:09.666 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:10.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:10.667 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:10.687 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:10.688 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:10.688 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:10.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:11.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:11.689 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:11.710 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:11.712 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:11.712 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:11.712 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:12.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:12.713 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:12.735 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:12.735 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:12.735 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:12.735 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:13.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:13.736 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:13.756 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:13.757 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:13.757 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:13.757 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:14.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:14.758 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:14.778 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:14.779 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:14.779 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:14.779 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:15.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:15.780 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:15.800 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:15.801 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:15.801 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:15.801 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:16.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:16.803 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:16.823 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:16.824 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:16.824 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:16.824 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:17.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:17.825 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:17.846 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:17.846 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:17.846 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:17.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:18.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:18.848 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:18.868 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:18.869 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:18.869 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:18.869 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:19.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:19.870 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:19.891 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:19.892 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:19.892 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:19.892 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:20.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:20.893 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:20.913 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:20.914 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:20.914 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:20.914 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:21.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:21.915 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:21.936 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:21.936 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:21.936 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:21.936 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:22.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:22.937 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:22.958 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:22.958 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:22.958 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:22.958 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:23.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:23.959 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:23.980 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:23.980 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:23.980 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:23.980 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:24.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:24.982 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:25.003 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:25.004 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:25.004 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:25.004 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:25.883 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:36:25.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:36:26.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:26.004 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:26.024 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:26.025 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:26.025 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:26.025 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:27.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:27.026 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:27.046 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:27.047 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:27.048 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:27.048 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:28.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:28.049 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:28.070 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:28.070 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:28.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:28.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:29.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:29.071 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:29.093 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:29.093 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:29.093 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:30.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:30.094 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:30.114 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:30.115 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:30.115 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:30.115 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:31.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:31.117 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:31.138 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:31.138 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:31.138 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:31.138 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:32.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:32.139 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:32.161 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:32.161 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:32.161 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:32.161 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:33.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:33.163 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:33.183 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:33.184 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:33.184 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:33.184 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:34.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:34.185 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:34.205 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:34.205 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:34.206 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:34.206 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:35.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:35.207 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:35.226 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:35.227 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:35.227 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:35.227 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:36.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:36.228 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:36.249 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:36.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:36.249 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:36.249 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:37.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:37.250 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:37.270 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:37.271 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:37.271 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:37.271 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:38.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:38.272 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:38.293 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:38.293 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:38.293 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:38.293 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:39.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:39.294 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:39.314 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:39.315 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:39.315 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:39.315 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:40.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:40.316 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:40.336 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:40.337 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:40.337 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:40.337 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:41.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:41.338 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:41.358 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:41.359 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:41.359 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:41.359 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:42.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:42.360 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:42.379 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:42.380 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:42.380 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:42.380 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:43.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:43.382 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:43.402 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:43.403 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:43.403 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:43.403 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:44.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:44.404 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:44.425 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:44.425 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:44.425 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:44.425 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:45.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:45.426 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:45.447 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:45.447 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:45.447 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:45.447 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:46.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:46.448 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:46.469 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:46.469 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:46.470 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:46.470 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:47.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:47.471 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:47.491 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:47.491 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:47.491 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:47.492 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:48.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:48.493 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:48.514 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:48.515 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:48.515 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:48.515 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:49.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:49.516 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:49.536 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:49.537 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:49.537 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:49.537 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:50.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:50.538 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:50.558 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:50.559 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:50.559 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:50.559 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:51.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:51.560 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:51.580 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:51.581 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:51.581 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:51.581 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:52.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:52.582 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:52.603 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:52.603 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:52.603 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:52.603 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:53.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:53.605 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:53.625 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:53.625 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:53.626 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:53.626 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:54.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:54.627 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:54.646 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:54.646 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:54.647 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:54.647 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:55.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:55.648 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:55.667 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:55.668 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:55.668 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:55.669 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:55.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:36:55.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:36:56.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:56.670 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:56.691 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:56.691 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:56.691 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:56.691 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:57.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:57.692 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:57.713 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:57.713 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:57.713 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:57.713 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:58.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:58.715 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:58.735 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:58.736 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:58.737 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:58.737 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:59.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:59.738 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:59.758 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:59.758 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:59.758 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:59.758 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:37:00.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:00.760 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:37:00.782 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:37:00.783 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_37_00_UTC 2024-06-29T14:37:00.783 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_37_00_UTC', 'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:37:00.783 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-14_37_00_UTC'} 2024-06-29T14:37:00.783 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:37:00.783 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-06-29-14_37_00_UTC'] 2024-06-29T14:37:00.783 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719671725.331854, now = 1719671820.783674, timo = 95 2024-06-29T14:37:01.784 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T14:37:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.921+0000 7f87d0794640 1 -- 172.21.15.59:0/2257216033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c806c1e0 msgr2=0x7f87c806c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.921+0000 7f87d0794640 1 --2- 172.21.15.59:0/2257216033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c806c1e0 0x7f87c806c5f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f87b8005fd0 tx=0x7f87b802aa90 comp rx=0 tx=0).stop 2024-06-29T14:37:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.921+0000 7f87d0794640 1 -- 172.21.15.59:0/2257216033 shutdown_connections 2024-06-29T14:37:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.921+0000 7f87d0794640 1 --2- 172.21.15.59:0/2257216033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c806c1e0 0x7f87c806c5f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.921+0000 7f87d0794640 1 --2- 172.21.15.59:0/2257216033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87c8073f30 0x7f87c815a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.921+0000 7f87d0794640 1 -- 172.21.15.59:0/2257216033 >> 172.21.15.59:0/2257216033 conn(0x7f87c80b24c0 msgr2=0x7f87c80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.921+0000 7f87d0794640 1 -- 172.21.15.59:0/2257216033 shutdown_connections 2024-06-29T14:37:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.921+0000 7f87d0794640 1 -- 172.21.15.59:0/2257216033 wait complete. 2024-06-29T14:37:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.923+0000 7f87d0794640 1 Processor -- start 2024-06-29T14:37:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.923+0000 7f87d0794640 1 -- start start 2024-06-29T14:37:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.924+0000 7f87d0794640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c806c1e0 0x7f87c80ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.924+0000 7f87d0794640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87c8073f30 0x7f87c8149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.924+0000 7f87d0794640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c80cf120 0x7f87c8155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.924+0000 7f87d0794640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f87c80cce40 con 0x7f87c80cf120 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.924+0000 7f87d0794640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87c80cccc0 con 0x7f87c806c1e0 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.924+0000 7f87d0794640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f87c80ccfc0 con 0x7f87c8073f30 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c80cf120 0x7f87c8155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c80cf120 0x7f87c8155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47968/0 (socket says 172.21.15.59:47968) 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 -- 172.21.15.59:0/2840490661 learned_addr learned my addr 172.21.15.59:0/2840490661 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ce530640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c806c1e0 0x7f87c80ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87cdd2f640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87c8073f30 0x7f87c8149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 -- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87c8073f30 msgr2=0x7f87c8149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87c8073f30 0x7f87c8149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 -- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c806c1e0 msgr2=0x7f87c80ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c806c1e0 0x7f87c80ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 -- 172.21.15.59:0/2840490661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c8151890 con 0x7f87c80cf120 2024-06-29T14:37:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87cdd2f640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87c8073f30 0x7f87c8149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87ced31640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c80cf120 0x7f87c8155380 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f87b8006010 tx=0x7f87b8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87c6ffd640 1 -- 172.21.15.59:0/2840490661 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b8044070 con 0x7f87c80cf120 2024-06-29T14:37:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87c6ffd640 1 -- 172.21.15.59:0/2840490661 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87b802e7f0 con 0x7f87c80cf120 2024-06-29T14:37:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c8152ff0 con 0x7f87c80cf120 2024-06-29T14:37:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87c6ffd640 1 -- 172.21.15.59:0/2840490661 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b80044b0 con 0x7f87c80cf120 2024-06-29T14:37:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.925+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c81535b0 con 0x7f87c80cf120 2024-06-29T14:37:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.926+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c806c5f0 con 0x7f87c80cf120 2024-06-29T14:37:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.927+0000 7f87c6ffd640 1 -- 172.21.15.59:0/2840490661 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 25) ==== 98332+0+0 (secure 0 0 0) 0x7f87b8047070 con 0x7f87c80cf120 2024-06-29T14:37:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.928+0000 7f87c6ffd640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f87b4076440 0x7f87b4078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.928+0000 7f87c6ffd640 1 -- 172.21.15.59:0/2840490661 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7f87b80b8b10 con 0x7f87c80cf120 2024-06-29T14:37:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.928+0000 7f87ce530640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f87b4076440 0x7f87b4078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.929+0000 7f87ce530640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f87b4076440 0x7f87b4078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f87c806f700 tx=0x7f87c000b3e0 comp rx=0 tx=0).ready entity=mgr.6214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:01.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:01.932+0000 7f87c6ffd640 1 -- 172.21.15.59:0/2840490661 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f87b8032030 con 0x7f87c80cf120 2024-06-29T14:37:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.149+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 --> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f87c80b2b60 con 0x7f87b4076440 2024-06-29T14:37:02.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.224+0000 7f87c6ffd640 1 -- 172.21.15.59:0/2840490661 <== mgr.6214 v2:172.21.15.153:6832/2118890682 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f87c80b2b60 con 0x7f87b4076440 2024-06-29T14:37:02.222 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path snap_test_dir1 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f87b4076440 msgr2=0x7f87b4078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f87b4076440 0x7f87b4078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f87c806f700 tx=0x7f87c000b3e0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c80cf120 msgr2=0x7f87c8155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c80cf120 0x7f87c8155380 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f87b8006010 tx=0x7f87b8003f10 comp rx=0 tx=0).stop 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 shutdown_connections 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f87b4076440 0x7f87b4078950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87c80cf120 0x7f87c8155380 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87c8073f30 0x7f87c8149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 --2- 172.21.15.59:0/2840490661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c806c1e0 0x7f87c80ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 >> 172.21.15.59:0/2840490661 conn(0x7f87c80b24c0 msgr2=0x7f87c8156950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:02.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 shutdown_connections 2024-06-29T14:37:02.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.226+0000 7f87d0794640 1 -- 172.21.15.59:0/2840490661 wait complete. 2024-06-29T14:37:02.235 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T14:37:02.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:02.236 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:37:02.252 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:37:02.252 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_37_00_UTC 2024-06-29T14:37:02.252 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:37:02.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:02.252 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_36_00_UTC) 2024-06-29T14:37:02.312 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_37_00_UTC 2024-06-29T14:37:02.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:02.312 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_37_00_UTC) 2024-06-29T14:37:02.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:02.372 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T14:37:02.433 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:37:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.594+0000 7f1ec985e640 1 -- 172.21.15.59:0/4290596631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40cf5c0 msgr2=0x7f1ec40d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.594+0000 7f1ec985e640 1 --2- 172.21.15.59:0/4290596631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40cf5c0 0x7f1ec40d1a20 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8005fd0 tx=0x7f1eb802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.594+0000 7f1ec985e640 1 -- 172.21.15.59:0/4290596631 shutdown_connections 2024-06-29T14:37:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.594+0000 7f1ec985e640 1 --2- 172.21.15.59:0/4290596631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec40cf5c0 0x7f1ec40d1a20 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.594+0000 7f1ec985e640 1 --2- 172.21.15.59:0/4290596631 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec408c190 0x7f1ec40cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.594+0000 7f1ec985e640 1 -- 172.21.15.59:0/4290596631 >> 172.21.15.59:0/4290596631 conn(0x7f1ec40b5610 msgr2=0x7f1ec40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.594+0000 7f1ec985e640 1 -- 172.21.15.59:0/4290596631 shutdown_connections 2024-06-29T14:37:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.594+0000 7f1ec985e640 1 -- 172.21.15.59:0/4290596631 wait complete. 2024-06-29T14:37:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.595+0000 7f1ec985e640 1 Processor -- start 2024-06-29T14:37:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.596+0000 7f1ec985e640 1 -- start start 2024-06-29T14:37:02.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec985e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec408c190 0x7f1ec4153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec985e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40ce950 0x7f1ec40cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec985e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec40cf5c0 0x7f1ec415edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec985e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec40b7fa0 con 0x7f1ec408c190 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec985e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec40b7e20 con 0x7f1ec40cf5c0 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec985e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec40b8120 con 0x7f1ec40ce950 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec408c190 0x7f1ec4153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec408c190 0x7f1ec4153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47990/0 (socket says 172.21.15.59:47990) 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec2ffd640 1 -- 172.21.15.59:0/1044625897 learned_addr learned my addr 172.21.15.59:0/1044625897 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec27fc640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40ce950 0x7f1ec40cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec37fe640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec40cf5c0 0x7f1ec415edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec2ffd640 1 -- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40ce950 msgr2=0x7f1ec40cbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec2ffd640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40ce950 0x7f1ec40cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec2ffd640 1 -- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec40cf5c0 msgr2=0x7f1ec415edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec2ffd640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec40cf5c0 0x7f1ec415edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec2ffd640 1 -- 172.21.15.59:0/1044625897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec40cc9d0 con 0x7f1ec408c190 2024-06-29T14:37:02.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.597+0000 7f1ec37fe640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec40cf5c0 0x7f1ec415edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:02.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.598+0000 7f1ec2ffd640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec408c190 0x7f1ec4153020 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400d770 tx=0x7f1eb400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:02.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.598+0000 7f1eabfff640 1 -- 172.21.15.59:0/1044625897 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4017070 con 0x7f1ec408c190 2024-06-29T14:37:02.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.598+0000 7f1eabfff640 1 -- 172.21.15.59:0/1044625897 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1eb4009d20 con 0x7f1ec408c190 2024-06-29T14:37:02.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.598+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec40cccb0 con 0x7f1ec408c190 2024-06-29T14:37:02.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.598+0000 7f1eabfff640 1 -- 172.21.15.59:0/1044625897 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40106c0 con 0x7f1ec408c190 2024-06-29T14:37:02.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.598+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec40cd270 con 0x7f1ec408c190 2024-06-29T14:37:02.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.599+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec40cf080 con 0x7f1ec408c190 2024-06-29T14:37:02.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.599+0000 7f1eabfff640 1 -- 172.21.15.59:0/1044625897 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 25) ==== 98332+0+0 (secure 0 0 0) 0x7f1eb401a070 con 0x7f1ec408c190 2024-06-29T14:37:02.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.600+0000 7f1eabfff640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1eb0076440 0x7f1eb0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.600+0000 7f1eabfff640 1 -- 172.21.15.59:0/1044625897 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7f1eb4097e80 con 0x7f1ec408c190 2024-06-29T14:37:02.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.600+0000 7f1ec27fc640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1eb0076440 0x7f1eb0078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.601+0000 7f1ec27fc640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1eb0076440 0x7f1eb0078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8033000 tx=0x7f1eb8004730 comp rx=0 tx=0).ready entity=mgr.6214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:02.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.605+0000 7f1eabfff640 1 -- 172.21.15.59:0/1044625897 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1eb4061210 con 0x7f1ec408c190 2024-06-29T14:37:02.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:02.847+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f1ec4050470 con 0x7f1ec408c190 2024-06-29T14:37:03.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.135+0000 7f1eabfff640 1 -- 172.21.15.59:0/1044625897 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v26) ==== 93+0+0 (secure 0 0 0) 0x7f1eb4060b10 con 0x7f1ec408c190 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1eb0076440 msgr2=0x7f1eb0078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1eb0076440 0x7f1eb0078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8033000 tx=0x7f1eb8004730 comp rx=0 tx=0).stop 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec408c190 msgr2=0x7f1ec4153020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec408c190 0x7f1ec4153020 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400d770 tx=0x7f1eb400dc30 comp rx=0 tx=0).stop 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 shutdown_connections 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1eb0076440 0x7f1eb0078950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec40cf5c0 0x7f1ec415edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1ec40ce950 0x7f1ec40cbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 --2- 172.21.15.59:0/1044625897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec408c190 0x7f1ec4153020 secure :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400d770 tx=0x7f1eb400dc30 comp rx=0 tx=0).stop 2024-06-29T14:37:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 >> 172.21.15.59:0/1044625897 conn(0x7f1ec40b5610 msgr2=0x7f1ec415dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:03.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.138+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 shutdown_connections 2024-06-29T14:37:03.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.139+0000 7f1ec985e640 1 -- 172.21.15.59:0/1044625897 wait complete. 2024-06-29T14:37:03.150 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:37:03.151 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:03.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:03.240 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:37:03.241 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:03.241 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:03.241 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:03.258 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:03.258 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:03.315 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:03.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.476+0000 7fb811705640 1 -- 172.21.15.59:0/3314992511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c15efe0 msgr2=0x7fb80c15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.476+0000 7fb811705640 1 --2- 172.21.15.59:0/3314992511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c15efe0 0x7fb80c15f9b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0d15d0 tx=0x7fb80402aac0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.476+0000 7fb811705640 1 -- 172.21.15.59:0/3314992511 shutdown_connections 2024-06-29T14:37:03.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.476+0000 7fb811705640 1 --2- 172.21.15.59:0/3314992511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c15efe0 0x7fb80c15f9b0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.476+0000 7fb811705640 1 --2- 172.21.15.59:0/3314992511 >> v2:172.21.15.153:3300/0 conn(0x7fb80c0d23c0 0x7fb80c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.476+0000 7fb811705640 1 -- 172.21.15.59:0/3314992511 >> 172.21.15.59:0/3314992511 conn(0x7fb80c0d3ee0 msgr2=0x7fb80c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:03.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.476+0000 7fb811705640 1 -- 172.21.15.59:0/3314992511 shutdown_connections 2024-06-29T14:37:03.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.476+0000 7fb811705640 1 -- 172.21.15.59:0/3314992511 wait complete. 2024-06-29T14:37:03.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.478+0000 7fb811705640 1 Processor -- start 2024-06-29T14:37:03.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.478+0000 7fb811705640 1 -- start start 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.479+0000 7fb811705640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb80c0b3ed0 0x7fb80c158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.479+0000 7fb811705640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0d23c0 0x7fb80c1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.479+0000 7fb811705640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb80c15efe0 0x7fb80c1eab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.479+0000 7fb811705640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c160280 con 0x7fb80c0d23c0 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.479+0000 7fb811705640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c160100 con 0x7fb80c0b3ed0 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.479+0000 7fb811705640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb80c160400 con 0x7fb80c15efe0 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0d23c0 0x7fb80c1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0d23c0 0x7fb80c1573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57862/0 (socket says 172.21.15.59:57862) 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 -- 172.21.15.59:0/699770315 learned_addr learned my addr 172.21.15.59:0/699770315 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb80affd640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb80c0b3ed0 0x7fb80c158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb80b7fe640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb80c15efe0 0x7fb80c1eab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 -- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb80c15efe0 msgr2=0x7fb80c1eab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb80c15efe0 0x7fb80c1eab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 -- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb80c0b3ed0 msgr2=0x7fb80c158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb80c0b3ed0 0x7fb80c158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 -- 172.21.15.59:0/699770315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c15c040 con 0x7fb80c0d23c0 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb803fff640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0d23c0 0x7fb80c1573e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400c370 tx=0x7fb7f400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb8037fe640 1 -- 172.21.15.59:0/699770315 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4017070 con 0x7fb80c0d23c0 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb8037fe640 1 -- 172.21.15.59:0/699770315 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7f40053c0 con 0x7fb80c0d23c0 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80c1597b0 con 0x7fb80c0d23c0 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb8037fe640 1 -- 172.21.15.59:0/699770315 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4003850 con 0x7fb80c0d23c0 2024-06-29T14:37:03.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.480+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80c159d70 con 0x7fb80c0d23c0 2024-06-29T14:37:03.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.482+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb80c0b3990 con 0x7fb80c0d23c0 2024-06-29T14:37:03.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.482+0000 7fb8037fe640 1 -- 172.21.15.59:0/699770315 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7fb7f401a070 con 0x7fb80c0d23c0 2024-06-29T14:37:03.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.483+0000 7fb8037fe640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fb7ec0763f0 0x7fb7ec078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.483+0000 7fb80affd640 1 -- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fb7ec0763f0 msgr2=0x7fb7ec078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:03.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.483+0000 7fb8037fe640 1 -- 172.21.15.59:0/699770315 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7fb7f4096fe0 con 0x7fb80c0d23c0 2024-06-29T14:37:03.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.483+0000 7fb80affd640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fb7ec0763f0 0x7fb7ec078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:03.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.488+0000 7fb8037fe640 1 -- 172.21.15.59:0/699770315 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb80c0b42e0 con 0x7fb80c0d23c0 2024-06-29T14:37:03.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.685+0000 7fb80affd640 1 -- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fb7ec0763f0 msgr2=0x7fb7ec078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:03.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.685+0000 7fb80affd640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fb7ec0763f0 0x7fb7ec078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:03.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.706+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb80c0d2950 con 0x7fb80c0d23c0 2024-06-29T14:37:03.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.708+0000 7fb8037fe640 1 -- 172.21.15.59:0/699770315 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+17556 (secure 0 0 0) 0x7fb7f405fe30 con 0x7fb80c0d23c0 2024-06-29T14:37:03.707 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:03.707 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":74,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:36:24.999861+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:35:04.881993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-06-29T14:35:05.467776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2f","mappings":[{"from":0,"to":6}]},{"pgid":"8.39","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1176138421":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/106596755":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6833/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6832/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/1025227607":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/387060511":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/297750041":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/2704927245":"2024-06-30T14:35:23.828270+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.710+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fb7ec0763f0 msgr2=0x7fb7ec078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fb7ec0763f0 0x7fb7ec078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0d23c0 msgr2=0x7fb80c1573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0d23c0 0x7fb80c1573e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400c370 tx=0x7fb7f400c830 comp rx=0 tx=0).stop 2024-06-29T14:37:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 shutdown_connections 2024-06-29T14:37:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fb7ec0763f0 0x7fb7ec078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb80c15efe0 0x7fb80c1eab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0d23c0 0x7fb80c1573e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 --2- 172.21.15.59:0/699770315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb80c0b3ed0 0x7fb80c158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 >> 172.21.15.59:0/699770315 conn(0x7fb80c0d3ee0 msgr2=0x7fb80c0b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 shutdown_connections 2024-06-29T14:37:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.711+0000 7fb811705640 1 -- 172.21.15.59:0/699770315 wait complete. 2024-06-29T14:37:03.721 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:35:04.881993+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:35:05.467776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:03.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.845+0000 7f624aeba640 1 -- 172.21.15.59:0/213948225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440d0130 msgr2=0x7f6244143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.845+0000 7f624aeba640 1 --2- 172.21.15.59:0/213948225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440d0130 0x7f6244143840 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6244144290 tx=0x7f623802a5f0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.845+0000 7f624aeba640 1 -- 172.21.15.59:0/213948225 shutdown_connections 2024-06-29T14:37:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.845+0000 7f624aeba640 1 --2- 172.21.15.59:0/213948225 >> v2:172.21.15.153:3300/0 conn(0x7f624406ebe0 0x7f62440cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.845+0000 7f624aeba640 1 --2- 172.21.15.59:0/213948225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440d0130 0x7f6244143840 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.845+0000 7f624aeba640 1 -- 172.21.15.59:0/213948225 >> 172.21.15.59:0/213948225 conn(0x7f62440b2ab0 msgr2=0x7f62440b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.845+0000 7f624aeba640 1 -- 172.21.15.59:0/213948225 shutdown_connections 2024-06-29T14:37:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.845+0000 7f624aeba640 1 -- 172.21.15.59:0/213948225 wait complete. 2024-06-29T14:37:03.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.847+0000 7f624aeba640 1 Processor -- start 2024-06-29T14:37:03.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.847+0000 7f624aeba640 1 -- start start 2024-06-29T14:37:03.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f624aeba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f624406ebe0 0x7f624413f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f624aeba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440b4ee0 0x7f62441f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f624aeba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62440d0130 0x7f62441f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f624aeba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f62440cf650 con 0x7f62440b4ee0 2024-06-29T14:37:03.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f624aeba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62440cf4d0 con 0x7f624406ebe0 2024-06-29T14:37:03.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f624aeba640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f62440cf7d0 con 0x7f62440d0130 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440b4ee0 0x7f62441f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440b4ee0 0x7f62441f5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57884/0 (socket says 172.21.15.59:57884) 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 -- 172.21.15.59:0/3712086862 learned_addr learned my addr 172.21.15.59:0/3712086862 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6249457640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62440d0130 0x7f62441f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6248c56640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f624406ebe0 0x7f624413f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 -- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62440d0130 msgr2=0x7f62441f61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62440d0130 0x7f62441f61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 -- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f624406ebe0 msgr2=0x7f624413f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f624406ebe0 0x7f624413f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 -- 172.21.15.59:0/3712086862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62440ab0a0 con 0x7f62440b4ee0 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.849+0000 7f6243fff640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440b4ee0 0x7f62441f5cb0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f623000c3a0 tx=0x7f623000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.850+0000 7f62417fa640 1 -- 172.21.15.59:0/3712086862 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62300052d0 con 0x7f62440b4ee0 2024-06-29T14:37:03.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.850+0000 7f62417fa640 1 -- 172.21.15.59:0/3712086862 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6230007020 con 0x7f62440b4ee0 2024-06-29T14:37:03.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.850+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62440ab380 con 0x7f62440b4ee0 2024-06-29T14:37:03.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.850+0000 7f62417fa640 1 -- 172.21.15.59:0/3712086862 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6230007470 con 0x7f62440b4ee0 2024-06-29T14:37:03.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.850+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62440ab940 con 0x7f62440b4ee0 2024-06-29T14:37:03.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.851+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62440c5e50 con 0x7f62440b4ee0 2024-06-29T14:37:03.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.852+0000 7f62417fa640 1 -- 172.21.15.59:0/3712086862 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f62300067d0 con 0x7f62440b4ee0 2024-06-29T14:37:03.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.852+0000 7f62417fa640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f62280763f0 0x7f6228078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.852+0000 7f6248c56640 1 -- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f62280763f0 msgr2=0x7f6228078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:03.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.852+0000 7f62417fa640 1 -- 172.21.15.59:0/3712086862 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7f6230097070 con 0x7f62440b4ee0 2024-06-29T14:37:03.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.852+0000 7f6248c56640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f62280763f0 0x7f6228078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:03.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:03.857+0000 7f62417fa640 1 -- 172.21.15.59:0/3712086862 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f62440c5e50 con 0x7f62440b4ee0 2024-06-29T14:37:04.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.049+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f62440cfcf0 con 0x7f62440b4ee0 2024-06-29T14:37:04.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.050+0000 7f62417fa640 1 -- 172.21.15.59:0/3712086862 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f623005ff50 con 0x7f62440b4ee0 2024-06-29T14:37:04.048 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:37:04.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f62280763f0 msgr2=0x7f6228078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:04.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f62280763f0 0x7f6228078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440b4ee0 msgr2=0x7f62441f5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440b4ee0 0x7f62441f5cb0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f623000c3a0 tx=0x7f623000c860 comp rx=0 tx=0).stop 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 shutdown_connections 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f62280763f0 0x7f6228078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62440d0130 0x7f62441f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62440b4ee0 0x7f62441f5cb0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 --2- 172.21.15.59:0/3712086862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f624406ebe0 0x7f624413f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 >> 172.21.15.59:0/3712086862 conn(0x7f62440b2ab0 msgr2=0x7f624406e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 shutdown_connections 2024-06-29T14:37:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.052+0000 7f624aeba640 1 -- 172.21.15.59:0/3712086862 wait complete. 2024-06-29T14:37:04.063 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:37:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.188+0000 7f1c9aed2640 1 -- 172.21.15.59:0/2452305251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2fa0 msgr2=0x7f1c940d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.188+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/2452305251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2fa0 0x7f1c940d1520 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1c80005fd0 tx=0x7f1c8002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.188+0000 7f1c9aed2640 1 -- 172.21.15.59:0/2452305251 shutdown_connections 2024-06-29T14:37:04.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.188+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/2452305251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2fa0 0x7f1c940d1520 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.188+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/2452305251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c940b2270 0x7f1c9415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.188+0000 7f1c9aed2640 1 -- 172.21.15.59:0/2452305251 >> 172.21.15.59:0/2452305251 conn(0x7f1c94073e10 msgr2=0x7f1c94074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.188+0000 7f1c9aed2640 1 -- 172.21.15.59:0/2452305251 shutdown_connections 2024-06-29T14:37:04.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.188+0000 7f1c9aed2640 1 -- 172.21.15.59:0/2452305251 wait complete. 2024-06-29T14:37:04.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.189+0000 7f1c9aed2640 1 Processor -- start 2024-06-29T14:37:04.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.190+0000 7f1c9aed2640 1 -- start start 2024-06-29T14:37:04.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c9aed2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2270 0x7f1c941452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c9aed2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c940b2fa0 0x7f1c9415a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c9aed2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c940d1a60 0x7f1c9415ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c9aed2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1c940b6cc0 con 0x7f1c940b2270 2024-06-29T14:37:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c9aed2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c940b6b40 con 0x7f1c940d1a60 2024-06-29T14:37:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c9aed2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1c940cfe90 con 0x7f1c940b2fa0 2024-06-29T14:37:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c98c6e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2270 0x7f1c941452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c98c6e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2270 0x7f1c941452c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57902/0 (socket says 172.21.15.59:57902) 2024-06-29T14:37:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c98c6e640 1 -- 172.21.15.59:0/1554675768 learned_addr learned my addr 172.21.15.59:0/1554675768 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:04.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c93fff640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c940b2fa0 0x7f1c9415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c9946f640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c940d1a60 0x7f1c9415ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c98c6e640 1 -- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c940b2fa0 msgr2=0x7f1c9415a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c98c6e640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c940b2fa0 0x7f1c9415a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c98c6e640 1 -- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c940d1a60 msgr2=0x7f1c9415ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c98c6e640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c940d1a60 0x7f1c9415ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.191+0000 7f1c98c6e640 1 -- 172.21.15.59:0/1554675768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9414ff20 con 0x7f1c940b2270 2024-06-29T14:37:04.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.192+0000 7f1c98c6e640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2270 0x7f1c941452c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800e790 tx=0x7f1c8800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:04.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.192+0000 7f1c917fa640 1 -- 172.21.15.59:0/1554675768 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800eea0 con 0x7f1c940b2270 2024-06-29T14:37:04.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.192+0000 7f1c917fa640 1 -- 172.21.15.59:0/1554675768 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c8800cdb0 con 0x7f1c940b2270 2024-06-29T14:37:04.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.192+0000 7f1c917fa640 1 -- 172.21.15.59:0/1554675768 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c88010690 con 0x7f1c940b2270 2024-06-29T14:37:04.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.192+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c94150200 con 0x7f1c940b2270 2024-06-29T14:37:04.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.192+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c941507c0 con 0x7f1c940b2270 2024-06-29T14:37:04.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.193+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c940d0340 con 0x7f1c940b2270 2024-06-29T14:37:04.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.194+0000 7f1c917fa640 1 -- 172.21.15.59:0/1554675768 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f1c88010830 con 0x7f1c940b2270 2024-06-29T14:37:04.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.194+0000 7f1c917fa640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1c78076420 0x7f1c78078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.194+0000 7f1c93fff640 1 -- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1c78076420 msgr2=0x7f1c78078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:04.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.194+0000 7f1c917fa640 1 -- 172.21.15.59:0/1554675768 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7f1c880985f0 con 0x7f1c940b2270 2024-06-29T14:37:04.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.194+0000 7f1c93fff640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1c78076420 0x7f1c78078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:04.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.198+0000 7f1c917fa640 1 -- 172.21.15.59:0/1554675768 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c88061340 con 0x7f1c940b2270 2024-06-29T14:37:04.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.391+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1c94159060 con 0x7f1c940b2270 2024-06-29T14:37:04.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.392+0000 7f1c917fa640 1 -- 172.21.15.59:0/1554675768 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v74) ==== 104+0+11 (secure 0 0 0) 0x7f1c88065640 con 0x7f1c940b2270 2024-06-29T14:37:04.390 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:37:04.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1c78076420 msgr2=0x7f1c78078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:04.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1c78076420 0x7f1c78078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2270 msgr2=0x7f1c941452c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2270 0x7f1c941452c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800e790 tx=0x7f1c8800ec50 comp rx=0 tx=0).stop 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 shutdown_connections 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f1c78076420 0x7f1c78078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c940d1a60 0x7f1c9415ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c940b2fa0 0x7f1c9415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 --2- 172.21.15.59:0/1554675768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c940b2270 0x7f1c941452c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 >> 172.21.15.59:0/1554675768 conn(0x7f1c94073e10 msgr2=0x7f1c940d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 shutdown_connections 2024-06-29T14:37:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.394+0000 7f1c9aed2640 1 -- 172.21.15.59:0/1554675768 wait complete. 2024-06-29T14:37:04.404 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:37:04.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 -- 172.21.15.59:0/1073356966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade00b3ed0 msgr2=0x7fade015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 --2- 172.21.15.59:0/1073356966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade00b3ed0 0x7fade015e8d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fade00d04f0 tx=0x7fadd002a560 comp rx=0 tx=0).stop 2024-06-29T14:37:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 -- 172.21.15.59:0/1073356966 shutdown_connections 2024-06-29T14:37:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 --2- 172.21.15.59:0/1073356966 >> v2:172.21.15.153:3300/0 conn(0x7fade015ee10 0x7fade015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 --2- 172.21.15.59:0/1073356966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade00b3ed0 0x7fade015e8d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 --2- 172.21.15.59:0/1073356966 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fade00d23c0 0x7fade00b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 -- 172.21.15.59:0/1073356966 >> 172.21.15.59:0/1073356966 conn(0x7fade00d3ee0 msgr2=0x7fade00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 -- 172.21.15.59:0/1073356966 shutdown_connections 2024-06-29T14:37:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.527+0000 7fade7de0640 1 -- 172.21.15.59:0/1073356966 wait complete. 2024-06-29T14:37:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.529+0000 7fade7de0640 1 Processor -- start 2024-06-29T14:37:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.529+0000 7fade7de0640 1 -- start start 2024-06-29T14:37:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade7de0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fade00b3ed0 0x7fade00c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade7de0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fade00d23c0 0x7fade015dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade7de0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade015ee10 0x7fade015d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade7de0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fade0160070 con 0x7fade015ee10 2024-06-29T14:37:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade7de0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fade015fef0 con 0x7fade00b3ed0 2024-06-29T14:37:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade7de0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fade01601f0 con 0x7fade00d23c0 2024-06-29T14:37:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade5b7c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fade00b3ed0 0x7fade00c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade637d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade015ee10 0x7fade015d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.530+0000 7fade5b7c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fade00b3ed0 0x7fade00c6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:51218/0 (socket says 172.21.15.59:51218) 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade5b7c640 1 -- 172.21.15.59:0/3464970624 learned_addr learned my addr 172.21.15.59:0/3464970624 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade537b640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fade00d23c0 0x7fade015dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade637d640 1 -- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fade00d23c0 msgr2=0x7fade015dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade637d640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fade00d23c0 0x7fade015dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade637d640 1 -- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fade00b3ed0 msgr2=0x7fade00c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade637d640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fade00b3ed0 0x7fade00c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade637d640 1 -- 172.21.15.59:0/3464970624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade01e9e20 con 0x7fade015ee10 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade537b640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fade00d23c0 0x7fade015dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade5b7c640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fade00b3ed0 0x7fade00c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade637d640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade015ee10 0x7fade015d0f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fadd8005fd0 tx=0x7fadd800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fadd67fc640 1 -- 172.21.15.59:0/3464970624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8016070 con 0x7fade015ee10 2024-06-29T14:37:04.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fadd67fc640 1 -- 172.21.15.59:0/3464970624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadd800bdd0 con 0x7fade015ee10 2024-06-29T14:37:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fade01ea100 con 0x7fade015ee10 2024-06-29T14:37:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fadd67fc640 1 -- 172.21.15.59:0/3464970624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8003bd0 con 0x7fade015ee10 2024-06-29T14:37:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.531+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fade00c7080 con 0x7fade015ee10 2024-06-29T14:37:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.533+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fade01ea620 con 0x7fade015ee10 2024-06-29T14:37:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.534+0000 7fadd67fc640 1 -- 172.21.15.59:0/3464970624 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7fadd8004b20 con 0x7fade015ee10 2024-06-29T14:37:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.534+0000 7fadd67fc640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fadc40763f0 0x7fadc4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.535+0000 7fadd67fc640 1 -- 172.21.15.59:0/3464970624 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7fadd8098670 con 0x7fade015ee10 2024-06-29T14:37:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.535+0000 7fade5b7c640 1 -- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fadc40763f0 msgr2=0x7fadc4078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.535+0000 7fade5b7c640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fadc40763f0 0x7fadc4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:04.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.538+0000 7fadd67fc640 1 -- 172.21.15.59:0/3464970624 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fade01ea620 con 0x7fade015ee10 2024-06-29T14:37:04.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.730+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fade00d2950 con 0x7fade015ee10 2024-06-29T14:37:04.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.731+0000 7fadd67fc640 1 -- 172.21.15.59:0/3464970624 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v74) ==== 100+0+11 (secure 0 0 0) 0x7fadd80613a0 con 0x7fade015ee10 2024-06-29T14:37:04.729 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:37:04.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fadc40763f0 msgr2=0x7fadc4078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:04.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fadc40763f0 0x7fadc4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade015ee10 msgr2=0x7fade015d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade015ee10 0x7fade015d0f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fadd8005fd0 tx=0x7fadd800dde0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 shutdown_connections 2024-06-29T14:37:04.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fadc40763f0 0x7fadc4078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fade015ee10 0x7fade015d0f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fade00d23c0 0x7fade015dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 --2- 172.21.15.59:0/3464970624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fade00b3ed0 0x7fade00c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 >> 172.21.15.59:0/3464970624 conn(0x7fade00d3ee0 msgr2=0x7fade006ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 shutdown_connections 2024-06-29T14:37:04.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.733+0000 7fade7de0640 1 -- 172.21.15.59:0/3464970624 wait complete. 2024-06-29T14:37:04.743 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:37:04.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.864+0000 7f3ed5e38640 1 -- 172.21.15.59:0/2941662643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed015e290 msgr2=0x7f3ed00ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.864+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/2941662643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed015e290 0x7f3ed00ce110 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009830 tx=0x7f3ec002a510 comp rx=0 tx=0).stop 2024-06-29T14:37:04.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.864+0000 7f3ed5e38640 1 -- 172.21.15.59:0/2941662643 shutdown_connections 2024-06-29T14:37:04.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.864+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/2941662643 >> v2:172.21.15.153:3300/0 conn(0x7f3ed00b40a0 0x7f3ed00b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.864+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/2941662643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed015e290 0x7f3ed00ce110 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.864+0000 7f3ed5e38640 1 -- 172.21.15.59:0/2941662643 >> 172.21.15.59:0/2941662643 conn(0x7f3ed00d3ee0 msgr2=0x7f3ed00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.864+0000 7f3ed5e38640 1 -- 172.21.15.59:0/2941662643 shutdown_connections 2024-06-29T14:37:04.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.864+0000 7f3ed5e38640 1 -- 172.21.15.59:0/2941662643 wait complete. 2024-06-29T14:37:04.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.866+0000 7f3ed5e38640 1 Processor -- start 2024-06-29T14:37:04.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.866+0000 7f3ed5e38640 1 -- start start 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ed5e38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ed00ce650 0x7f3ed01f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ed5e38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed015e290 0x7f3ed0158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ed5e38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed0155cc0 0x7f3ed015f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ed5e38640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed00b7cc0 con 0x7f3ed00ce650 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ed5e38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed00b7b40 con 0x7f3ed015e290 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ed5e38640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed00b7e40 con 0x7f3ed0155cc0 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ecf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ed00ce650 0x7f3ed01f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ecf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ed00ce650 0x7f3ed01f5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57928/0 (socket says 172.21.15.59:57928) 2024-06-29T14:37:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.867+0000 7f3ecf7fe640 1 -- 172.21.15.59:0/3525947739 learned_addr learned my addr 172.21.15.59:0/3525947739 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3eceffd640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed015e290 0x7f3ed0158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ecffff640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed0155cc0 0x7f3ed015f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ecf7fe640 1 -- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed0155cc0 msgr2=0x7f3ed015f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ecf7fe640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed0155cc0 0x7f3ed015f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ecf7fe640 1 -- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed015e290 msgr2=0x7f3ed0158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ecf7fe640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed015e290 0x7f3ed0158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ecf7fe640 1 -- 172.21.15.59:0/3525947739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed0157a30 con 0x7f3ed00ce650 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3eceffd640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed015e290 0x7f3ed0158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ecffff640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed0155cc0 0x7f3ed015f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ecf7fe640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ed00ce650 0x7f3ed01f5640 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0000c00 tx=0x7f3ec0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3eb7fff640 1 -- 172.21.15.59:0/3525947739 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0005c90 con 0x7f3ed00ce650 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3eb7fff640 1 -- 172.21.15.59:0/3525947739 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ec0031e50 con 0x7f3ed00ce650 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed0157d10 con 0x7f3ed00ce650 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.868+0000 7f3eb7fff640 1 -- 172.21.15.59:0/3525947739 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0033290 con 0x7f3ed00ce650 2024-06-29T14:37:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.869+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed01582d0 con 0x7f3ed00ce650 2024-06-29T14:37:04.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.869+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ed0157790 con 0x7f3ed00ce650 2024-06-29T14:37:04.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.870+0000 7f3eb7fff640 1 -- 172.21.15.59:0/3525947739 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f3ec0033430 con 0x7f3ed00ce650 2024-06-29T14:37:04.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.871+0000 7f3eb7fff640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f3ebc076420 0x7f3ebc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.871+0000 7f3eceffd640 1 -- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f3ebc076420 msgr2=0x7f3ebc078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:04.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.871+0000 7f3eb7fff640 1 -- 172.21.15.59:0/3525947739 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7f3ec00b8c50 con 0x7f3ed00ce650 2024-06-29T14:37:04.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.871+0000 7f3eceffd640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f3ebc076420 0x7f3ebc078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:04.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:04.879+0000 7f3eb7fff640 1 -- 172.21.15.59:0/3525947739 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3ed0157790 con 0x7f3ed00ce650 2024-06-29T14:37:05.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.072+0000 7f3eceffd640 1 -- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f3ebc076420 msgr2=0x7f3ebc078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:05.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.072+0000 7f3eceffd640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f3ebc076420 0x7f3ebc078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:05.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.085+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3ed00d4500 con 0x7f3ed00ce650 2024-06-29T14:37:05.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.085+0000 7f3eb7fff640 1 -- 172.21.15.59:0/3525947739 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v74) ==== 80+0+475 (secure 0 0 0) 0x7f3ec004a030 con 0x7f3ed00ce650 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1176138421 2024-06-30T14:35:23.828270+0000 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/106596755 2024-06-30T14:35:23.828270+0000 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/2957341575 2024-06-30T14:35:23.828270+0000 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/2957341575 2024-06-30T14:35:23.828270+0000 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1025227607 2024-06-30T14:35:23.828270+0000 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/387060511 2024-06-30T14:35:23.828270+0000 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/297750041 2024-06-30T14:35:23.828270+0000 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2704927245 2024-06-30T14:35:23.828270+0000 2024-06-29T14:37:05.084 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T14:37:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.087+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f3ebc076420 msgr2=0x7f3ebc078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.087+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f3ebc076420 0x7f3ebc078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.087+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ed00ce650 msgr2=0x7f3ed01f5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.087+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ed00ce650 0x7f3ed01f5640 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0000c00 tx=0x7f3ec0033c20 comp rx=0 tx=0).stop 2024-06-29T14:37:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.088+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 shutdown_connections 2024-06-29T14:37:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.088+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f3ebc076420 0x7f3ebc078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.088+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ed0155cc0 0x7f3ed015f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.088+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed015e290 0x7f3ed0158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.088+0000 7f3ed5e38640 1 --2- 172.21.15.59:0/3525947739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ed00ce650 0x7f3ed01f5640 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.088+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 >> 172.21.15.59:0/3525947739 conn(0x7f3ed00d3ee0 msgr2=0x7f3ed00b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.088+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 shutdown_connections 2024-06-29T14:37:05.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.088+0000 7f3ed5e38640 1 -- 172.21.15.59:0/3525947739 wait complete. 2024-06-29T14:37:05.098 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:05.098 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:05.131 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:37:05.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:05.131 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:37:05.187 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:05.187 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:37:05.187 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:05.203 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:05.203 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:05.203 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:05.203 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:05.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:05.243 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:05.243 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:37:05.243 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:05.259 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:05.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:05.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.403+0000 7f08cd4f6640 1 -- 172.21.15.59:0/3563902613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c80d23c0 msgr2=0x7f08c80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.403+0000 7f08cd4f6640 1 --2- 172.21.15.59:0/3563902613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c80d23c0 0x7f08c80b3990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009830 tx=0x7f08b802a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.403+0000 7f08cd4f6640 1 -- 172.21.15.59:0/3563902613 shutdown_connections 2024-06-29T14:37:05.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.403+0000 7f08cd4f6640 1 --2- 172.21.15.59:0/3563902613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c80d23c0 0x7f08c80b3990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.403+0000 7f08cd4f6640 1 -- 172.21.15.59:0/3563902613 >> 172.21.15.59:0/3563902613 conn(0x7f08c80d3ee0 msgr2=0x7f08c80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.403+0000 7f08cd4f6640 1 -- 172.21.15.59:0/3563902613 shutdown_connections 2024-06-29T14:37:05.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.403+0000 7f08cd4f6640 1 -- 172.21.15.59:0/3563902613 wait complete. 2024-06-29T14:37:05.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.405+0000 7f08cd4f6640 1 Processor -- start 2024-06-29T14:37:05.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.405+0000 7f08cd4f6640 1 -- start start 2024-06-29T14:37:05.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.406+0000 7f08cd4f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c80b3ed0 0x7f08c814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.406+0000 7f08cd4f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c80d23c0 0x7f08c8150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.406+0000 7f08cd4f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c815efe0 0x7f08c806ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.406+0000 7f08cd4f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f08c8160450 con 0x7f08c815efe0 2024-06-29T14:37:05.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.406+0000 7f08cd4f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08c81602d0 con 0x7f08c80b3ed0 2024-06-29T14:37:05.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.406+0000 7f08cd4f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f08c81605d0 con 0x7f08c80d23c0 2024-06-29T14:37:05.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c815efe0 0x7f08c806ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c815efe0 0x7f08c806ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57932/0 (socket says 172.21.15.59:57932) 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 -- 172.21.15.59:0/1934388532 learned_addr learned my addr 172.21.15.59:0/1934388532 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c6ffd640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c80b3ed0 0x7f08c814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c67fc640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c80d23c0 0x7f08c8150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 -- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c80d23c0 msgr2=0x7f08c8150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c80d23c0 0x7f08c8150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 -- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c80b3ed0 msgr2=0x7f08c814e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c80b3ed0 0x7f08c814e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 -- 172.21.15.59:0/1934388532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c80c8670 con 0x7f08c815efe0 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08c77fe640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c815efe0 0x7f08c806ccf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f08c000ca80 tx=0x7f08c000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08bf7fe640 1 -- 172.21.15.59:0/1934388532 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0016020 con 0x7f08c815efe0 2024-06-29T14:37:05.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08bf7fe640 1 -- 172.21.15.59:0/1934388532 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08c000ad20 con 0x7f08c815efe0 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c814eee0 con 0x7f08c815efe0 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.407+0000 7f08bf7fe640 1 -- 172.21.15.59:0/1934388532 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0012680 con 0x7f08c815efe0 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.408+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c814f4a0 con 0x7f08c815efe0 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.408+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c814ec50 con 0x7f08c815efe0 2024-06-29T14:37:05.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.409+0000 7f08bf7fe640 1 -- 172.21.15.59:0/1934388532 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f08c0019020 con 0x7f08c815efe0 2024-06-29T14:37:05.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.410+0000 7f08bf7fe640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f08a80763f0 0x7f08a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.410+0000 7f08c6ffd640 1 -- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f08a80763f0 msgr2=0x7f08a8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:05.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.410+0000 7f08bf7fe640 1 -- 172.21.15.59:0/1934388532 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7f08c0097e50 con 0x7f08c815efe0 2024-06-29T14:37:05.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.410+0000 7f08c6ffd640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f08a80763f0 0x7f08a8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:05.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.419+0000 7f08bf7fe640 1 -- 172.21.15.59:0/1934388532 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f08c0020030 con 0x7f08c815efe0 2024-06-29T14:37:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.612+0000 7f08c6ffd640 1 -- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f08a80763f0 msgr2=0x7f08a8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:05.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.612+0000 7f08c6ffd640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f08a80763f0 0x7f08a8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:05.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.649+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08c80c8840 con 0x7f08c815efe0 2024-06-29T14:37:05.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.650+0000 7f08bf7fe640 1 -- 172.21.15.59:0/1934388532 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5223 (secure 0 0 0) 0x7f08c80b42e0 con 0x7f08c815efe0 2024-06-29T14:37:05.648 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:05.649 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":29,"btime":"2024-06-29T14:35:10:154758+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/484297155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":484297155},{"type":"v1","addr":"172.21.15.153:6835","nonce":484297155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2309835731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":2309835731},{"type":"v1","addr":"172.21.15.153:6837","nonce":2309835731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":6330,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/413487166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":413487166},{"type":"v1","addr":"172.21.15.59:6834","nonce":413487166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:35:06.529979+0000","modified":"2024-06-29T14:35:09.544199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6333":{"gid":6333,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3627888249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3627888249},{"type":"v1","addr":"172.21.15.59:6835","nonce":3627888249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6333,"qdb_cluster":[6333]},"id":3}]} 2024-06-29T14:37:05.649 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 29 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f08a80763f0 msgr2=0x7f08a8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f08a80763f0 0x7f08a8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c815efe0 msgr2=0x7f08c806ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c815efe0 0x7f08c806ccf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f08c000ca80 tx=0x7f08c000cf40 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 shutdown_connections 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f08a80763f0 0x7f08a8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08c815efe0 0x7f08c806ccf0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08c80d23c0 0x7f08c8150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 --2- 172.21.15.59:0/1934388532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c80b3ed0 0x7f08c814e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 >> 172.21.15.59:0/1934388532 conn(0x7f08c80d3ee0 msgr2=0x7f08c8153a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 shutdown_connections 2024-06-29T14:37:05.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.652+0000 7f08cd4f6640 1 -- 172.21.15.59:0/1934388532 wait complete. 2024-06-29T14:37:05.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.782+0000 7f391f43f640 1 -- 172.21.15.59:0/3506940024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39180d23c0 msgr2=0x7f39180b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.782+0000 7f391f43f640 1 --2- 172.21.15.59:0/3506940024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39180d23c0 0x7f39180b3990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f390c009be0 tx=0x7f390c02a920 comp rx=0 tx=0).stop 2024-06-29T14:37:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.782+0000 7f391f43f640 1 -- 172.21.15.59:0/3506940024 shutdown_connections 2024-06-29T14:37:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.782+0000 7f391f43f640 1 --2- 172.21.15.59:0/3506940024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39180d23c0 0x7f39180b3990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.782+0000 7f391f43f640 1 -- 172.21.15.59:0/3506940024 >> 172.21.15.59:0/3506940024 conn(0x7f39180d3ee0 msgr2=0x7f39180d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.782+0000 7f391f43f640 1 -- 172.21.15.59:0/3506940024 shutdown_connections 2024-06-29T14:37:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.782+0000 7f391f43f640 1 -- 172.21.15.59:0/3506940024 wait complete. 2024-06-29T14:37:05.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.784+0000 7f391f43f640 1 Processor -- start 2024-06-29T14:37:05.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.784+0000 7f391f43f640 1 -- start start 2024-06-29T14:37:05.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.785+0000 7f391f43f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180b3ed0 0x7f391814d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.785+0000 7f391f43f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39180d23c0 0x7f3918150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.785+0000 7f391f43f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391815efe0 0x7f3918151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.785+0000 7f391f43f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3918160450 con 0x7f39180d23c0 2024-06-29T14:37:05.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.785+0000 7f391f43f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39181602d0 con 0x7f39180b3ed0 2024-06-29T14:37:05.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.785+0000 7f391f43f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f39181605d0 con 0x7f391815efe0 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391c9da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39180d23c0 0x7f3918150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391c9da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39180d23c0 0x7f3918150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57942/0 (socket says 172.21.15.59:57942) 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d1db640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180b3ed0 0x7f391814d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391c9da640 1 -- 172.21.15.59:0/2618954294 learned_addr learned my addr 172.21.15.59:0/2618954294 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d9dc640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391815efe0 0x7f3918151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d1db640 1 -- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391815efe0 msgr2=0x7f3918151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d1db640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391815efe0 0x7f3918151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d1db640 1 -- 172.21.15.59:0/2618954294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39180d23c0 msgr2=0x7f3918150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d1db640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39180d23c0 0x7f3918150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d1db640 1 -- 172.21.15.59:0/2618954294 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391814dab0 con 0x7f39180b3ed0 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d9dc640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391815efe0 0x7f3918151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391c9da640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39180d23c0 0x7f3918150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f391d1db640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180b3ed0 0x7f391814d060 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f390c009000 tx=0x7f390c003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:05.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.786+0000 7f3909ffb640 1 -- 172.21.15.59:0/2618954294 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c030e60 con 0x7f39180b3ed0 2024-06-29T14:37:05.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.787+0000 7f3909ffb640 1 -- 172.21.15.59:0/2618954294 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f390c03e070 con 0x7f39180b3ed0 2024-06-29T14:37:05.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.787+0000 7f3909ffb640 1 -- 172.21.15.59:0/2618954294 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c032330 con 0x7f39180b3ed0 2024-06-29T14:37:05.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.787+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3918151680 con 0x7f39180b3ed0 2024-06-29T14:37:05.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.787+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3918157370 con 0x7f39180b3ed0 2024-06-29T14:37:05.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.788+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39180b3990 con 0x7f39180b3ed0 2024-06-29T14:37:05.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.789+0000 7f3909ffb640 1 -- 172.21.15.59:0/2618954294 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f390c0324d0 con 0x7f39180b3ed0 2024-06-29T14:37:05.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.789+0000 7f3909ffb640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f38fc0763f0 0x7f38fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.789+0000 7f3909ffb640 1 -- 172.21.15.59:0/2618954294 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7f390c0b9350 con 0x7f39180b3ed0 2024-06-29T14:37:05.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.789+0000 7f391c9da640 1 -- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f38fc0763f0 msgr2=0x7f38fc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:05.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.789+0000 7f391c9da640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f38fc0763f0 0x7f38fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:05.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.794+0000 7f3909ffb640 1 -- 172.21.15.59:0/2618954294 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f390c02d070 con 0x7f39180b3ed0 2024-06-29T14:37:05.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.986+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39180d2950 con 0x7f39180b3ed0 2024-06-29T14:37:05.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.988+0000 7f3909ffb640 1 -- 172.21.15.59:0/2618954294 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+17556 (secure 0 0 0) 0x7f390c0821a0 con 0x7f39180b3ed0 2024-06-29T14:37:05.986 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:05.986 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":74,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:36:24.999861+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:35:04.881993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-06-29T14:35:05.467776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2f","mappings":[{"from":0,"to":6}]},{"pgid":"8.39","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1176138421":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/106596755":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6833/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6832/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/1025227607":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/387060511":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/297750041":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/2704927245":"2024-06-30T14:35:23.828270+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f38fc0763f0 msgr2=0x7f38fc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f38fc0763f0 0x7f38fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180b3ed0 msgr2=0x7f391814d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180b3ed0 0x7f391814d060 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f390c009000 tx=0x7f390c003c20 comp rx=0 tx=0).stop 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 shutdown_connections 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f38fc0763f0 0x7f38fc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391815efe0 0x7f3918151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39180d23c0 0x7f3918150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 --2- 172.21.15.59:0/2618954294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180b3ed0 0x7f391814d060 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 >> 172.21.15.59:0/2618954294 conn(0x7f39180d3ee0 msgr2=0x7f39180d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 shutdown_connections 2024-06-29T14:37:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:05.990+0000 7f391f43f640 1 -- 172.21.15.59:0/2618954294 wait complete. 2024-06-29T14:37:06.000 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:35:04.881993+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:35:05.467776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:06.000 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:06.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.123+0000 7f00c67fc640 1 -- 172.21.15.59:0/13606571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00d0130 msgr2=0x7f00c0143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.123+0000 7f00c67fc640 1 --2- 172.21.15.59:0/13606571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00d0130 0x7f00c0143840 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f00b0009830 tx=0x7f00b002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.123+0000 7f00c67fc640 1 -- 172.21.15.59:0/13606571 shutdown_connections 2024-06-29T14:37:06.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.123+0000 7f00c67fc640 1 --2- 172.21.15.59:0/13606571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00d0130 0x7f00c0143840 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.123+0000 7f00c67fc640 1 -- 172.21.15.59:0/13606571 >> 172.21.15.59:0/13606571 conn(0x7f00c00b2ab0 msgr2=0x7f00c00b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.123+0000 7f00c67fc640 1 -- 172.21.15.59:0/13606571 shutdown_connections 2024-06-29T14:37:06.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.123+0000 7f00c67fc640 1 -- 172.21.15.59:0/13606571 wait complete. 2024-06-29T14:37:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.125+0000 7f00c67fc640 1 Processor -- start 2024-06-29T14:37:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.125+0000 7f00c67fc640 1 -- start start 2024-06-29T14:37:06.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.126+0000 7f00c67fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c006ebe0 0x7f00c0159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.126+0000 7f00c67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00b4ee0 0x7f00c01e4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.126+0000 7f00c67fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00c00d0130 0x7f00c015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.126+0000 7f00c67fc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f00c00cfa30 con 0x7f00c00b4ee0 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.126+0000 7f00c67fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00c00cf8b0 con 0x7f00c006ebe0 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.126+0000 7f00c67fc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f00c00cfbb0 con 0x7f00c00d0130 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00b4ee0 0x7f00c01e4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00b4ee0 0x7f00c01e4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57956/0 (socket says 172.21.15.59:57956) 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bf7fe640 1 -- 172.21.15.59:0/1964196281 learned_addr learned my addr 172.21.15.59:0/1964196281 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bffff640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c006ebe0 0x7f00c0159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00c4d99640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00c00d0130 0x7f00c015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bffff640 1 -- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00c00d0130 msgr2=0x7f00c015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bffff640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00c00d0130 0x7f00c015d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bffff640 1 -- 172.21.15.59:0/1964196281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00b4ee0 msgr2=0x7f00c01e4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bffff640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00b4ee0 0x7f00c01e4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bffff640 1 -- 172.21.15.59:0/1964196281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c01e5a70 con 0x7f00c006ebe0 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bf7fe640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00b4ee0 0x7f00c01e4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00c4d99640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00c00d0130 0x7f00c015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bffff640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c006ebe0 0x7f00c0159a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f00b0031000 tx=0x7f00b0033c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.127+0000 7f00bcff9640 1 -- 172.21.15.59:0/1964196281 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0031d60 con 0x7f00c006ebe0 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.128+0000 7f00bcff9640 1 -- 172.21.15.59:0/1964196281 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00b0005020 con 0x7f00c006ebe0 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.128+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c01e53e0 con 0x7f00c006ebe0 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.128+0000 7f00bcff9640 1 -- 172.21.15.59:0/1964196281 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0005470 con 0x7f00c006ebe0 2024-06-29T14:37:06.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.128+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c01e6250 con 0x7f00c006ebe0 2024-06-29T14:37:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.129+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c006f290 con 0x7f00c006ebe0 2024-06-29T14:37:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.129+0000 7f00bcff9640 1 -- 172.21.15.59:0/1964196281 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f00b00047a0 con 0x7f00c006ebe0 2024-06-29T14:37:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.130+0000 7f00bcff9640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f00ac0763f0 0x7f00ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.130+0000 7f00bf7fe640 1 -- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f00ac0763f0 msgr2=0x7f00ac078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.130+0000 7f00bcff9640 1 -- 172.21.15.59:0/1964196281 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6503+0+0 (secure 0 0 0) 0x7f00b00b8e70 con 0x7f00c006ebe0 2024-06-29T14:37:06.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.130+0000 7f00bf7fe640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f00ac0763f0 0x7f00ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:06.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.134+0000 7f00bcff9640 1 -- 172.21.15.59:0/1964196281 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f00c006f290 con 0x7f00c006ebe0 2024-06-29T14:37:06.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.331+0000 7f00bf7fe640 1 -- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f00ac0763f0 msgr2=0x7f00ac078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:06.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.332+0000 7f00bf7fe640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f00ac0763f0 0x7f00ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:06.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.339+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f00c0144000 con 0x7f00c006ebe0 2024-06-29T14:37:06.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.340+0000 7f00bcff9640 1 -- 172.21.15.59:0/1964196281 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) ==== 93+0+10 (secure 0 0 0) 0x7f00b003f030 con 0x7f00c006ebe0 2024-06-29T14:37:06.338 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f00ac0763f0 msgr2=0x7f00ac078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f00ac0763f0 0x7f00ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c006ebe0 msgr2=0x7f00c0159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c006ebe0 0x7f00c0159a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f00b0031000 tx=0x7f00b0033c10 comp rx=0 tx=0).stop 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 shutdown_connections 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7f00ac0763f0 0x7f00ac078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00c00d0130 0x7f00c015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c00b4ee0 0x7f00c01e4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 --2- 172.21.15.59:0/1964196281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c006ebe0 0x7f00c0159a40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 >> 172.21.15.59:0/1964196281 conn(0x7f00c00b2ab0 msgr2=0x7f00c006e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 shutdown_connections 2024-06-29T14:37:06.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.342+0000 7f00c67fc640 1 -- 172.21.15.59:0/1964196281 wait complete. 2024-06-29T14:37:06.352 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:37:06.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.476+0000 7fdbdb436640 1 -- 172.21.15.59:0/3201461960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd40d1c30 msgr2=0x7fdbd40b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.476+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3201461960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd40d1c30 0x7fdbd40b5b30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc005fd0 tx=0x7fdbcc02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.476+0000 7fdbdb436640 1 -- 172.21.15.59:0/3201461960 shutdown_connections 2024-06-29T14:37:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.476+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3201461960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd40d1c30 0x7fdbd40b5b30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.476+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3201461960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdbd40b2270 0x7fdbd415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.476+0000 7fdbdb436640 1 -- 172.21.15.59:0/3201461960 >> 172.21.15.59:0/3201461960 conn(0x7fdbd4073e10 msgr2=0x7fdbd4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.476+0000 7fdbdb436640 1 -- 172.21.15.59:0/3201461960 shutdown_connections 2024-06-29T14:37:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.476+0000 7fdbdb436640 1 -- 172.21.15.59:0/3201461960 wait complete. 2024-06-29T14:37:06.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.478+0000 7fdbdb436640 1 Processor -- start 2024-06-29T14:37:06.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.478+0000 7fdbdb436640 1 -- start start 2024-06-29T14:37:06.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbdb436640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd40b2270 0x7fdbd415e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbdb436640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdbd40b2fa0 0x7fdbd41450d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbdb436640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbd40d1c30 0x7fdbd40ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbdb436640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd40b6400 con 0x7fdbd40d1c30 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbdb436640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd40b6280 con 0x7fdbd40b2270 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbdb436640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdbd40b6580 con 0x7fdbd40b2fa0 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbd99d3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbd40d1c30 0x7fdbd40ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbd99d3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbd40d1c30 0x7fdbd40ac510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57958/0 (socket says 172.21.15.59:57958) 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbd99d3640 1 -- 172.21.15.59:0/3776395718 learned_addr learned my addr 172.21.15.59:0/3776395718 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbd91d2640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd40b2270 0x7fdbd415e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.479+0000 7fdbd89d1640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdbd40b2fa0 0x7fdbd41450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbd99d3640 1 -- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdbd40b2fa0 msgr2=0x7fdbd41450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbd99d3640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdbd40b2fa0 0x7fdbd41450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbd99d3640 1 -- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd40b2270 msgr2=0x7fdbd415e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbd99d3640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd40b2270 0x7fdbd415e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbd99d3640 1 -- 172.21.15.59:0/3776395718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd415d900 con 0x7fdbd40d1c30 2024-06-29T14:37:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbd99d3640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbd40d1c30 0x7fdbd40ac510 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009cd0 tx=0x7fdbcc031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbc9ffb640 1 -- 172.21.15.59:0/3776395718 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc031990 con 0x7fdbd40d1c30 2024-06-29T14:37:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd415dbe0 con 0x7fdbd40d1c30 2024-06-29T14:37:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbc9ffb640 1 -- 172.21.15.59:0/3776395718 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbcc031c20 con 0x7fdbd40d1c30 2024-06-29T14:37:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbc9ffb640 1 -- 172.21.15.59:0/3776395718 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0330d0 con 0x7fdbd40d1c30 2024-06-29T14:37:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.480+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd415e1a0 con 0x7fdbd40d1c30 2024-06-29T14:37:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.481+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbd40b6ac0 con 0x7fdbd40d1c30 2024-06-29T14:37:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.482+0000 7fdbc9ffb640 1 -- 172.21.15.59:0/3776395718 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7fdbcc033270 con 0x7fdbd40d1c30 2024-06-29T14:37:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.483+0000 7fdbc9ffb640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fdbb80763f0 0x7fdbb8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.483+0000 7fdbc9ffb640 1 -- 172.21.15.59:0/3776395718 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6559+0+0 (secure 0 0 0) 0x7fdbcc0b90a0 con 0x7fdbd40d1c30 2024-06-29T14:37:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.483+0000 7fdbd91d2640 1 -- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fdbb80763f0 msgr2=0x7fdbb8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.483+0000 7fdbd91d2640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fdbb80763f0 0x7fdbb8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.487+0000 7fdbc9ffb640 1 -- 172.21.15.59:0/3776395718 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdbcc03f030 con 0x7fdbd40d1c30 2024-06-29T14:37:06.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.684+0000 7fdbd91d2640 1 -- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fdbb80763f0 msgr2=0x7fdbb8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2118890682 2024-06-29T14:37:06.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.684+0000 7fdbd91d2640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fdbb80763f0 0x7fdbb8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:06.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.688+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdbd4155520 con 0x7fdbd40d1c30 2024-06-29T14:37:06.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.688+0000 7fdbc9ffb640 1 -- 172.21.15.59:0/3776395718 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v75) ==== 104+0+11 (secure 0 0 0) 0x7fdbcc081dc0 con 0x7fdbd40d1c30 2024-06-29T14:37:06.687 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:37:06.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fdbb80763f0 msgr2=0x7fdbb8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:06.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fdbb80763f0 0x7fdbb8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbd40d1c30 msgr2=0x7fdbd40ac510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbd40d1c30 0x7fdbd40ac510 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009cd0 tx=0x7fdbcc031700 comp rx=0 tx=0).stop 2024-06-29T14:37:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 shutdown_connections 2024-06-29T14:37:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:6832/2118890682,v1:172.21.15.153:6833/2118890682] conn(0x7fdbb80763f0 0x7fdbb8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbd40d1c30 0x7fdbd40ac510 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdbd40b2fa0 0x7fdbd41450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 --2- 172.21.15.59:0/3776395718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd40b2270 0x7fdbd415e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 >> 172.21.15.59:0/3776395718 conn(0x7fdbd4073e10 msgr2=0x7fdbd406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 shutdown_connections 2024-06-29T14:37:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.691+0000 7fdbdb436640 1 -- 172.21.15.59:0/3776395718 wait complete. 2024-06-29T14:37:06.700 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:37:06.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.829+0000 7fc7fe131640 1 -- 172.21.15.59:0/1437476079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f8143840 msgr2=0x7fc7f806e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.829+0000 7fc7fe131640 1 --2- 172.21.15.59:0/1437476079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f8143840 0x7fc7f806e4d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec005fd0 tx=0x7fc7ec02a540 comp rx=0 tx=0).stop 2024-06-29T14:37:06.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.829+0000 7fc7fe131640 1 -- 172.21.15.59:0/1437476079 shutdown_connections 2024-06-29T14:37:06.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.829+0000 7fc7fe131640 1 --2- 172.21.15.59:0/1437476079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7f806ea10 0x7fc7f80cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.829+0000 7fc7fe131640 1 --2- 172.21.15.59:0/1437476079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f8143840 0x7fc7f806e4d0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.829+0000 7fc7fe131640 1 -- 172.21.15.59:0/1437476079 >> 172.21.15.59:0/1437476079 conn(0x7fc7f80b2ab0 msgr2=0x7fc7f80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.829+0000 7fc7fe131640 1 -- 172.21.15.59:0/1437476079 shutdown_connections 2024-06-29T14:37:06.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.829+0000 7fc7fe131640 1 -- 172.21.15.59:0/1437476079 wait complete. 2024-06-29T14:37:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.831+0000 7fc7fe131640 1 Processor -- start 2024-06-29T14:37:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.832+0000 7fc7fe131640 1 -- start start 2024-06-29T14:37:06.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7fe131640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7f806ea10 0x7fc7f815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7fe131640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f80d0130 0x7fc7f813d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7fe131640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7f8143840 0x7fc7f80b1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7fe131640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f80cf650 con 0x7fc7f80d0130 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7fe131640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f80cf4d0 con 0x7fc7f8143840 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7fe131640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f80cf7d0 con 0x7fc7f806ea10 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7f6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f80d0130 0x7fc7f813d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7f6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f80d0130 0x7fc7f813d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57962/0 (socket says 172.21.15.59:57962) 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7f6ffd640 1 -- 172.21.15.59:0/3336399480 learned_addr learned my addr 172.21.15.59:0/3336399480 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7f77fe640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7f806ea10 0x7fc7f815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.833+0000 7fc7f7fff640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7f8143840 0x7fc7f80b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7f6ffd640 1 -- 172.21.15.59:0/3336399480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7f806ea10 msgr2=0x7fc7f815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7f6ffd640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7f806ea10 0x7fc7f815c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7f6ffd640 1 -- 172.21.15.59:0/3336399480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7f8143840 msgr2=0x7fc7f80b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7f6ffd640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7f8143840 0x7fc7f80b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7f6ffd640 1 -- 172.21.15.59:0/3336399480 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f8151e40 con 0x7fc7f80d0130 2024-06-29T14:37:06.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7f6ffd640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f80d0130 0x7fc7f813d200 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec000c00 tx=0x7fc7ec003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:06.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7dffff640 1 -- 172.21.15.59:0/3336399480 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec03e030 con 0x7fc7f80d0130 2024-06-29T14:37:06.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7dffff640 1 -- 172.21.15.59:0/3336399480 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7ec003e60 con 0x7fc7f80d0130 2024-06-29T14:37:06.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f815c070 con 0x7fc7f80d0130 2024-06-29T14:37:06.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7dffff640 1 -- 172.21.15.59:0/3336399480 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec035040 con 0x7fc7f80d0130 2024-06-29T14:37:06.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.834+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f8155740 con 0x7fc7f80d0130 2024-06-29T14:37:06.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.835+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f80cfc80 con 0x7fc7f80d0130 2024-06-29T14:37:06.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.836+0000 7fc7dffff640 1 -- 172.21.15.59:0/3336399480 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) ==== 97912+0+0 (secure 0 0 0) 0x7fc7ec041030 con 0x7fc7f80d0130 2024-06-29T14:37:06.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.837+0000 7fc7dffff640 1 -- 172.21.15.59:0/3336399480 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6875+0+0 (secure 0 0 0) 0x7fc7ec0b7fa0 con 0x7fc7f80d0130 2024-06-29T14:37:06.839 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec8c19b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.839 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec8c19b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.839 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec8c19b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.839 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec8c19b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.839 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec8c19b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.839 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec89195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.839 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec89195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.840 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec89195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.840 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec89195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.840 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:06.841+0000 7fec89195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:06.841+0000 7fc7dffff640 1 -- 172.21.15.59:0/3336399480 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc7ec081100 con 0x7fc7f80d0130 2024-06-29T14:37:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.034+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc7f815bd90 con 0x7fc7f80d0130 2024-06-29T14:37:07.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.034+0000 7fc7dffff640 1 -- 172.21.15.59:0/3336399480 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) ==== 100+0+11 (secure 0 0 0) 0x7fc7ec085400 con 0x7fc7f80d0130 2024-06-29T14:37:07.033 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:37:07.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.036+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f80d0130 msgr2=0x7fc7f813d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.036+0000 7fc7fe131640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f80d0130 0x7fc7f813d200 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec000c00 tx=0x7fc7ec003970 comp rx=0 tx=0).stop 2024-06-29T14:37:07.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.037+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 shutdown_connections 2024-06-29T14:37:07.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.037+0000 7fc7fe131640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7f8143840 0x7fc7f80b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.037+0000 7fc7fe131640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7f80d0130 0x7fc7f813d200 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.037+0000 7fc7fe131640 1 --2- 172.21.15.59:0/3336399480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7f806ea10 0x7fc7f815c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.037+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 >> 172.21.15.59:0/3336399480 conn(0x7fc7f80b2ab0 msgr2=0x7fc7f81515e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.037+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 shutdown_connections 2024-06-29T14:37:07.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.037+0000 7fc7fe131640 1 -- 172.21.15.59:0/3336399480 wait complete. 2024-06-29T14:37:07.046 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:07.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.170+0000 7f88bff03640 1 -- 172.21.15.59:0/219238359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 msgr2=0x7f88b815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.170+0000 7f88bff03640 1 --2- 172.21.15.59:0/219238359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 0x7f88b815e8d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f88ac005fd0 tx=0x7f88ac02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.170+0000 7f88bff03640 1 -- 172.21.15.59:0/219238359 shutdown_connections 2024-06-29T14:37:07.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.170+0000 7f88bff03640 1 --2- 172.21.15.59:0/219238359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 0x7f88b815e8d0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.170+0000 7f88bff03640 1 --2- 172.21.15.59:0/219238359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88b80d23c0 0x7f88b80b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.170+0000 7f88bff03640 1 -- 172.21.15.59:0/219238359 >> 172.21.15.59:0/219238359 conn(0x7f88b80d3ee0 msgr2=0x7f88b80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.170+0000 7f88bff03640 1 -- 172.21.15.59:0/219238359 shutdown_connections 2024-06-29T14:37:07.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.170+0000 7f88bff03640 1 -- 172.21.15.59:0/219238359 wait complete. 2024-06-29T14:37:07.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.172+0000 7f88bff03640 1 Processor -- start 2024-06-29T14:37:07.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.172+0000 7f88bff03640 1 -- start start 2024-06-29T14:37:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bff03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 0x7f88b81e8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bff03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88b80d23c0 0x7f88b815cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bff03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88b815ee10 0x7f88b815d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bff03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8160240 con 0x7f88b80b3ed0 2024-06-29T14:37:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bff03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88b81600c0 con 0x7f88b80d23c0 2024-06-29T14:37:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bff03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f88b81603c0 con 0x7f88b815ee10 2024-06-29T14:37:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bdc9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 0x7f88b81e8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bdc9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 0x7f88b81e8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57970/0 (socket says 172.21.15.59:57970) 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bdc9f640 1 -- 172.21.15.59:0/859994334 learned_addr learned my addr 172.21.15.59:0/859994334 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88be4a0640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88b815ee10 0x7f88b815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bd49e640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88b80d23c0 0x7f88b815cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bdc9f640 1 -- 172.21.15.59:0/859994334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88b815ee10 msgr2=0x7f88b815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bdc9f640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88b815ee10 0x7f88b815d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bdc9f640 1 -- 172.21.15.59:0/859994334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88b80d23c0 msgr2=0x7f88b815cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bdc9f640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88b80d23c0 0x7f88b815cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.173+0000 7f88bdc9f640 1 -- 172.21.15.59:0/859994334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b81e9710 con 0x7f88b80b3ed0 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.174+0000 7f88bdc9f640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 0x7f88b81e8c50 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e790 tx=0x7f88a800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.174+0000 7f88b67fc640 1 -- 172.21.15.59:0/859994334 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a800eea0 con 0x7f88b80b3ed0 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.174+0000 7f88b67fc640 1 -- 172.21.15.59:0/859994334 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88a800ccc0 con 0x7f88b80b3ed0 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.174+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b81e99f0 con 0x7f88b80b3ed0 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.174+0000 7f88b67fc640 1 -- 172.21.15.59:0/859994334 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8004280 con 0x7f88b80b3ed0 2024-06-29T14:37:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.174+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b81e9fb0 con 0x7f88b80b3ed0 2024-06-29T14:37:07.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.175+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b815e8d0 con 0x7f88b80b3ed0 2024-06-29T14:37:07.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.176+0000 7f88b67fc640 1 -- 172.21.15.59:0/859994334 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) ==== 97912+0+0 (secure 0 0 0) 0x7f88a80035f0 con 0x7f88b80b3ed0 2024-06-29T14:37:07.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.177+0000 7f88b67fc640 1 -- 172.21.15.59:0/859994334 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6875+0+0 (secure 0 0 0) 0x7f88a8097e40 con 0x7f88b80b3ed0 2024-06-29T14:37:07.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.181+0000 7f88b67fc640 1 -- 172.21.15.59:0/859994334 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f88a8061080 con 0x7f88b80b3ed0 2024-06-29T14:37:07.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.416+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88b80d2950 con 0x7f88b80b3ed0 2024-06-29T14:37:07.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.416+0000 7f88b67fc640 1 -- 172.21.15.59:0/859994334 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5223 (secure 0 0 0) 0x7f88a8065380 con 0x7f88b80b3ed0 2024-06-29T14:37:07.415 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:07.415 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":29,"btime":"2024-06-29T14:35:10:154758+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/484297155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":484297155},{"type":"v1","addr":"172.21.15.153:6835","nonce":484297155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2309835731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":2309835731},{"type":"v1","addr":"172.21.15.153:6837","nonce":2309835731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":6330,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/413487166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":413487166},{"type":"v1","addr":"172.21.15.59:6834","nonce":413487166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:35:06.529979+0000","modified":"2024-06-29T14:35:09.544199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6333":{"gid":6333,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3627888249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3627888249},{"type":"v1","addr":"172.21.15.59:6835","nonce":3627888249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6333,"qdb_cluster":[6333]},"id":3}]} 2024-06-29T14:37:07.415 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 29 2024-06-29T14:37:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 msgr2=0x7f88b81e8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 0x7f88b81e8c50 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e790 tx=0x7f88a800ec50 comp rx=0 tx=0).stop 2024-06-29T14:37:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 shutdown_connections 2024-06-29T14:37:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88b815ee10 0x7f88b815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88b80d23c0 0x7f88b815cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 --2- 172.21.15.59:0/859994334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b80b3ed0 0x7f88b81e8c50 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 >> 172.21.15.59:0/859994334 conn(0x7f88b80d3ee0 msgr2=0x7f88b81577e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 shutdown_connections 2024-06-29T14:37:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.419+0000 7f88bff03640 1 -- 172.21.15.59:0/859994334 wait complete. 2024-06-29T14:37:07.428 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:07.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.547+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/1812575099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0b52b0 msgr2=0x7f8a9c0b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.547+0000 7f8aa1d2a640 1 --2- 172.21.15.59:0/1812575099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0b52b0 0x7f8a9c0b49d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009830 tx=0x7f8a8c02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.547+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/1812575099 shutdown_connections 2024-06-29T14:37:07.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.547+0000 7f8aa1d2a640 1 --2- 172.21.15.59:0/1812575099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0b52b0 0x7f8a9c0b49d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.547+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/1812575099 >> 172.21.15.59:0/1812575099 conn(0x7f8a9c0b1ea0 msgr2=0x7f8a9c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.547+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/1812575099 shutdown_connections 2024-06-29T14:37:07.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.547+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/1812575099 wait complete. 2024-06-29T14:37:07.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.549+0000 7f8aa1d2a640 1 Processor -- start 2024-06-29T14:37:07.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.549+0000 7f8aa1d2a640 1 -- start start 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.550+0000 7f8aa1d2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0b52b0 0x7f8a9c154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8aa1d2a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0cb650 0x7f8a9c0c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8aa1d2a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a9c15be90 0x7f8a9c073830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8aa1d2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c0ad6b0 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8aa1d2a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c0cbc70 con 0x7f8a9c0cb650 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8aa1d2a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8a9c0ad830 con 0x7f8a9c15be90 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0b52b0 0x7f8a9c154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0b52b0 0x7f8a9c154890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57984/0 (socket says 172.21.15.59:57984) 2024-06-29T14:37:07.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 -- 172.21.15.59:0/4079608421 learned_addr learned my addr 172.21.15.59:0/4079608421 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9affd640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0cb650 0x7f8a9c0c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9bfff640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a9c15be90 0x7f8a9c073830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 -- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a9c15be90 msgr2=0x7f8a9c073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a9c15be90 0x7f8a9c073830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 -- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0cb650 msgr2=0x7f8a9c0c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0cb650 0x7f8a9c0c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 -- 172.21.15.59:0/4079608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c06c310 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9bfff640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a9c15be90 0x7f8a9c073830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a9b7fe640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0b52b0 0x7f8a9c154890 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009950 tx=0x7f8a8c033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a83fff640 1 -- 172.21.15.59:0/4079608421 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c031b50 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a83fff640 1 -- 172.21.15.59:0/4079608421 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a8c031de0 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8a83fff640 1 -- 172.21.15.59:0/4079608421 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c005280 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.551+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9c06c5f0 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.552+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9c06cbb0 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.553+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a9c0adc80 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.553+0000 7f8a83fff640 1 -- 172.21.15.59:0/4079608421 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 27) ==== 97912+0+0 (secure 0 0 0) 0x7f8a8c005420 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.554+0000 7f8a83fff640 1 -- 172.21.15.59:0/4079608421 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6875+0+0 (secure 0 0 0) 0x7f8a8c0b86b0 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.558+0000 7f8a83fff640 1 -- 172.21.15.59:0/4079608421 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8a8c03f030 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.748+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a9c0b2450 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.750+0000 7f8a83fff640 1 -- 172.21.15.59:0/4079608421 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+18130 (secure 0 0 0) 0x7f8a8c081fd0 con 0x7f8a9c0b52b0 2024-06-29T14:37:07.748 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:07.748 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":76,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:06.779564+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:35:04.881993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-06-29T14:35:05.467776+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":75,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2f","mappings":[{"from":0,"to":6}]},{"pgid":"8.39","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2661755582":"2024-06-30T14:37:06.779521+0000","172.21.15.153:6832/2118890682":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2930251413":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/1279472232":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/3733140807":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/1176138421":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/106596755":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6833/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6832/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/1025227607":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/719687282":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/387060511":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/297750041":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6833/2118890682":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2149747529":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2704927245":"2024-06-30T14:35:23.828270+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.751+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0b52b0 msgr2=0x7f8a9c154890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.751+0000 7f8aa1d2a640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0b52b0 0x7f8a9c154890 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009950 tx=0x7f8a8c033ba0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.752+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 shutdown_connections 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.752+0000 7f8aa1d2a640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a9c15be90 0x7f8a9c073830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.752+0000 7f8aa1d2a640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0cb650 0x7f8a9c0c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.752+0000 7f8aa1d2a640 1 --2- 172.21.15.59:0/4079608421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0b52b0 0x7f8a9c154890 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.752+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 >> 172.21.15.59:0/4079608421 conn(0x7f8a9c0b1ea0 msgr2=0x7f8a9c1583e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.752+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 shutdown_connections 2024-06-29T14:37:07.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.752+0000 7f8aa1d2a640 1 -- 172.21.15.59:0/4079608421 wait complete. 2024-06-29T14:37:07.761 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:37:07.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.884+0000 7fb3da9e0640 1 -- 172.21.15.59:0/2094108072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40cf5c0 msgr2=0x7fb3d40d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.884+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/2094108072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40cf5c0 0x7fb3d40d1a20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8005fd0 tx=0x7fb3c802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.884+0000 7fb3da9e0640 1 -- 172.21.15.59:0/2094108072 shutdown_connections 2024-06-29T14:37:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.884+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/2094108072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40cf5c0 0x7fb3d40d1a20 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.884+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/2094108072 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3d408c190 0x7fb3d40cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.884+0000 7fb3da9e0640 1 -- 172.21.15.59:0/2094108072 >> 172.21.15.59:0/2094108072 conn(0x7fb3d40b5610 msgr2=0x7fb3d40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.884+0000 7fb3da9e0640 1 -- 172.21.15.59:0/2094108072 shutdown_connections 2024-06-29T14:37:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.884+0000 7fb3da9e0640 1 -- 172.21.15.59:0/2094108072 wait complete. 2024-06-29T14:37:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.886+0000 7fb3da9e0640 1 Processor -- start 2024-06-29T14:37:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.886+0000 7fb3da9e0640 1 -- start start 2024-06-29T14:37:07.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.887+0000 7fb3da9e0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d408c190 0x7fb3d41f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.887+0000 7fb3da9e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ce950 0x7fb3d41f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3da9e0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3d40cf5c0 0x7fb3d41581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3da9e0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d40b7fa0 con 0x7fb3d40ce950 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3da9e0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d40b7e20 con 0x7fb3d408c190 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3da9e0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d40b8120 con 0x7fb3d40cf5c0 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ce950 0x7fb3d41f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ce950 0x7fb3d41f5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57998/0 (socket says 172.21.15.59:57998) 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 -- 172.21.15.59:0/307350651 learned_addr learned my addr 172.21.15.59:0/307350651 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d3fff640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d408c190 0x7fb3d41f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d8f7d640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3d40cf5c0 0x7fb3d41581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 -- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3d40cf5c0 msgr2=0x7fb3d41581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3d40cf5c0 0x7fb3d41581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 -- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d408c190 msgr2=0x7fb3d41f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d408c190 0x7fb3d41f5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 -- 172.21.15.59:0/307350651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d415c200 con 0x7fb3d40ce950 2024-06-29T14:37:07.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d8f7d640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3d40cf5c0 0x7fb3d41581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:07.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d37fe640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ce950 0x7fb3d41f5870 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006010 tx=0x7fb3c80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:07.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d0ff9640 1 -- 172.21.15.59:0/307350651 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c803e030 con 0x7fb3d40ce950 2024-06-29T14:37:07.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d0ff9640 1 -- 172.21.15.59:0/307350651 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3c8003ae0 con 0x7fb3d40ce950 2024-06-29T14:37:07.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d4159cb0 con 0x7fb3d40ce950 2024-06-29T14:37:07.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3d0ff9640 1 -- 172.21.15.59:0/307350651 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8035040 con 0x7fb3d40ce950 2024-06-29T14:37:07.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.888+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d415bf20 con 0x7fb3d40ce950 2024-06-29T14:37:07.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.889+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d40cf080 con 0x7fb3d40ce950 2024-06-29T14:37:07.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.890+0000 7fb3d0ff9640 1 -- 172.21.15.59:0/307350651 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7fb3c8041030 con 0x7fb3d40ce950 2024-06-29T14:37:07.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.891+0000 7fb3d0ff9640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fb3c0076120 0x7fb3c0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.891+0000 7fb3d0ff9640 1 -- 172.21.15.59:0/307350651 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6875+0+0 (secure 0 0 0) 0x7fb3c80b7c00 con 0x7fb3d40ce950 2024-06-29T14:37:07.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.891+0000 7fb3d3fff640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fb3c0076120 0x7fb3c0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.892+0000 7fb3d3fff640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fb3c0076120 0x7fb3c0078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009bb0 tx=0x7fb3c4005f90 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:07.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:07.896+0000 7fb3d0ff9640 1 -- 172.21.15.59:0/307350651 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb3c8080dc0 con 0x7fb3d40ce950 2024-06-29T14:37:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.130+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3d406d530 con 0x7fb3d40ce950 2024-06-29T14:37:08.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.131+0000 7fb3d0ff9640 1 -- 172.21.15.59:0/307350651 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5223 (secure 0 0 0) 0x7fb3c8089400 con 0x7fb3d40ce950 2024-06-29T14:37:08.129 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:08.129 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":29,"btime":"2024-06-29T14:35:10:154758+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/484297155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":484297155},{"type":"v1","addr":"172.21.15.153:6835","nonce":484297155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2309835731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":2309835731},{"type":"v1","addr":"172.21.15.153:6837","nonce":2309835731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":6330,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/413487166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":413487166},{"type":"v1","addr":"172.21.15.59:6834","nonce":413487166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:35:06.529979+0000","modified":"2024-06-29T14:35:09.544199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6333":{"gid":6333,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3627888249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3627888249},{"type":"v1","addr":"172.21.15.59:6835","nonce":3627888249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6333,"qdb_cluster":[6333]},"id":3}]} 2024-06-29T14:37:08.130 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 29 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fb3c0076120 msgr2=0x7fb3c0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fb3c0076120 0x7fb3c0078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009bb0 tx=0x7fb3c4005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ce950 msgr2=0x7fb3d41f5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ce950 0x7fb3d41f5870 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006010 tx=0x7fb3c80035f0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 shutdown_connections 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fb3c0076120 0x7fb3c0078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb3d40cf5c0 0x7fb3d41581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3d40ce950 0x7fb3d41f5870 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 --2- 172.21.15.59:0/307350651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3d408c190 0x7fb3d41f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 >> 172.21.15.59:0/307350651 conn(0x7fb3d40b5610 msgr2=0x7fb3d4155180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 shutdown_connections 2024-06-29T14:37:08.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.133+0000 7fb3da9e0640 1 -- 172.21.15.59:0/307350651 wait complete. 2024-06-29T14:37:08.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.263+0000 7fa321bd2640 1 -- 172.21.15.59:0/476581275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0cf5c0 msgr2=0x7fa31c0d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.263+0000 7fa321bd2640 1 --2- 172.21.15.59:0/476581275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0cf5c0 0x7fa31c0d1a20 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa310009830 tx=0x7fa31002a510 comp rx=0 tx=0).stop 2024-06-29T14:37:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.263+0000 7fa321bd2640 1 -- 172.21.15.59:0/476581275 shutdown_connections 2024-06-29T14:37:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.263+0000 7fa321bd2640 1 --2- 172.21.15.59:0/476581275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0cf5c0 0x7fa31c0d1a20 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.263+0000 7fa321bd2640 1 --2- 172.21.15.59:0/476581275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31c08c190 0x7fa31c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.263+0000 7fa321bd2640 1 -- 172.21.15.59:0/476581275 >> 172.21.15.59:0/476581275 conn(0x7fa31c0b5610 msgr2=0x7fa31c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.263+0000 7fa321bd2640 1 -- 172.21.15.59:0/476581275 shutdown_connections 2024-06-29T14:37:08.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.263+0000 7fa321bd2640 1 -- 172.21.15.59:0/476581275 wait complete. 2024-06-29T14:37:08.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.265+0000 7fa321bd2640 1 Processor -- start 2024-06-29T14:37:08.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.265+0000 7fa321bd2640 1 -- start start 2024-06-29T14:37:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa321bd2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa31c08c190 0x7fa31c1e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa321bd2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0ce950 0x7fa31c1f87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa321bd2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31c0cf5c0 0x7fa31c1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa321bd2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c0b7fa0 con 0x7fa31c0ce950 2024-06-29T14:37:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa321bd2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c0b7e20 con 0x7fa31c08c190 2024-06-29T14:37:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa321bd2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa31c0b8120 con 0x7fa31c0cf5c0 2024-06-29T14:37:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa31affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0ce950 0x7fa31c1f87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa31affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0ce950 0x7fa31c1f87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58014/0 (socket says 172.21.15.59:58014) 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa31affd640 1 -- 172.21.15.59:0/199478498 learned_addr learned my addr 172.21.15.59:0/199478498 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa31bfff640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31c0cf5c0 0x7fa31c1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.266+0000 7fa31b7fe640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa31c08c190 0x7fa31c1e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa31affd640 1 -- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31c0cf5c0 msgr2=0x7fa31c1f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa31affd640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31c0cf5c0 0x7fa31c1f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa31affd640 1 -- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa31c08c190 msgr2=0x7fa31c1e71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa31affd640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa31c08c190 0x7fa31c1e71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa31affd640 1 -- 172.21.15.59:0/199478498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c1e7c60 con 0x7fa31c0ce950 2024-06-29T14:37:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa31affd640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0ce950 0x7fa31c1f87b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa310000c00 tx=0x7fa310033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa303fff640 1 -- 172.21.15.59:0/199478498 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310005c90 con 0x7fa31c0ce950 2024-06-29T14:37:08.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa303fff640 1 -- 172.21.15.59:0/199478498 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa310031e50 con 0x7fa31c0ce950 2024-06-29T14:37:08.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31c1e7f40 con 0x7fa31c0ce950 2024-06-29T14:37:08.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa303fff640 1 -- 172.21.15.59:0/199478498 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3100332b0 con 0x7fa31c0ce950 2024-06-29T14:37:08.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.267+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31c1e8500 con 0x7fa31c0ce950 2024-06-29T14:37:08.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.268+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31c0cf080 con 0x7fa31c0ce950 2024-06-29T14:37:08.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.269+0000 7fa303fff640 1 -- 172.21.15.59:0/199478498 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7fa310033450 con 0x7fa31c0ce950 2024-06-29T14:37:08.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.269+0000 7fa303fff640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa308076150 0x7fa308078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.269+0000 7fa303fff640 1 -- 172.21.15.59:0/199478498 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6875+0+0 (secure 0 0 0) 0x7fa3100b8a50 con 0x7fa31c0ce950 2024-06-29T14:37:08.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.270+0000 7fa31b7fe640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa308076150 0x7fa308078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.270+0000 7fa31b7fe640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa308076150 0x7fa308078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00bb50 tx=0x7fa30c00b2e0 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.274+0000 7fa303fff640 1 -- 172.21.15.59:0/199478498 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa31003f030 con 0x7fa31c0ce950 2024-06-29T14:37:08.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.501+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa31c06d950 con 0x7fa31c0ce950 2024-06-29T14:37:08.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.502+0000 7fa303fff640 1 -- 172.21.15.59:0/199478498 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5223 (secure 0 0 0) 0x7fa31008a250 con 0x7fa31c0ce950 2024-06-29T14:37:08.500 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:08.500 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":29,"btime":"2024-06-29T14:35:10:154758+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/484297155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":484297155},{"type":"v1","addr":"172.21.15.153:6835","nonce":484297155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2309835731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":2309835731},{"type":"v1","addr":"172.21.15.153:6837","nonce":2309835731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":6330,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/413487166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":413487166},{"type":"v1","addr":"172.21.15.59:6834","nonce":413487166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:35:06.529979+0000","modified":"2024-06-29T14:35:09.544199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6333},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6333":{"gid":6333,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3627888249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3627888249},{"type":"v1","addr":"172.21.15.59:6835","nonce":3627888249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6333,"qdb_cluster":[6333]},"id":3}]} 2024-06-29T14:37:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 29 2024-06-29T14:37:08.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa308076150 msgr2=0x7fa308078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa308076150 0x7fa308078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00bb50 tx=0x7fa30c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0ce950 msgr2=0x7fa31c1f87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0ce950 0x7fa31c1f87b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa310000c00 tx=0x7fa310033c90 comp rx=0 tx=0).stop 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 shutdown_connections 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa308076150 0x7fa308078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa31c0cf5c0 0x7fa31c1f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa31c0ce950 0x7fa31c1f87b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 --2- 172.21.15.59:0/199478498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa31c08c190 0x7fa31c1e71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 >> 172.21.15.59:0/199478498 conn(0x7fa31c0b5610 msgr2=0x7fa31c0c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 shutdown_connections 2024-06-29T14:37:08.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.504+0000 7fa321bd2640 1 -- 172.21.15.59:0/199478498 wait complete. 2024-06-29T14:37:08.513 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.632+0000 7f86917da640 1 -- 172.21.15.59:0/3956972222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0b2fa0 msgr2=0x7f868c0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.632+0000 7f86917da640 1 --2- 172.21.15.59:0/3956972222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0b2fa0 0x7f868c0d1520 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8680005fd0 tx=0x7f868002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.632+0000 7f86917da640 1 -- 172.21.15.59:0/3956972222 shutdown_connections 2024-06-29T14:37:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.632+0000 7f86917da640 1 --2- 172.21.15.59:0/3956972222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f868c0d1a60 0x7f868c0b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.632+0000 7f86917da640 1 --2- 172.21.15.59:0/3956972222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0b2fa0 0x7f868c0d1520 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.632+0000 7f86917da640 1 -- 172.21.15.59:0/3956972222 >> 172.21.15.59:0/3956972222 conn(0x7f868c073e10 msgr2=0x7f868c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.632+0000 7f86917da640 1 -- 172.21.15.59:0/3956972222 shutdown_connections 2024-06-29T14:37:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.632+0000 7f86917da640 1 -- 172.21.15.59:0/3956972222 wait complete. 2024-06-29T14:37:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.634+0000 7f86917da640 1 Processor -- start 2024-06-29T14:37:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.634+0000 7f86917da640 1 -- start start 2024-06-29T14:37:08.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.635+0000 7f86917da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f868c0b2270 0x7f868c14ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.635+0000 7f86917da640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f868c0b2fa0 0x7f868c06f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.635+0000 7f86917da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0d1a60 0x7f868c14a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.635+0000 7f86917da640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f868c0b6400 con 0x7f868c0d1a60 2024-06-29T14:37:08.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.635+0000 7f86917da640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f868c0b6280 con 0x7f868c0b2270 2024-06-29T14:37:08.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.635+0000 7f86917da640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f868c0b6580 con 0x7f868c0b2fa0 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0d1a60 0x7f868c14a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0d1a60 0x7f868c14a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58038/0 (socket says 172.21.15.59:58038) 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 -- 172.21.15.59:0/975581593 learned_addr learned my addr 172.21.15.59:0/975581593 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868a7fc640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f868c0b2fa0 0x7f868c06f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868affd640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f868c0b2270 0x7f868c14ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 -- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f868c0b2fa0 msgr2=0x7f868c06f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f868c0b2fa0 0x7f868c06f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 -- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f868c0b2270 msgr2=0x7f868c14ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f868c0b2270 0x7f868c14ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 -- 172.21.15.59:0/975581593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868c0b42b0 con 0x7f868c0d1a60 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868affd640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f868c0b2270 0x7f868c14ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f868b7fe640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0d1a60 0x7f868c14a170 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f868400ea10 tx=0x7f868400eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f86737fe640 1 -- 172.21.15.59:0/975581593 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868400cde0 con 0x7f868c0d1a60 2024-06-29T14:37:08.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f86737fe640 1 -- 172.21.15.59:0/975581593 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8684004c10 con 0x7f868c0d1a60 2024-06-29T14:37:08.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868c0b3cf0 con 0x7f868c0d1a60 2024-06-29T14:37:08.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f86737fe640 1 -- 172.21.15.59:0/975581593 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684005020 con 0x7f868c0d1a60 2024-06-29T14:37:08.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.636+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f868c06eca0 con 0x7f868c0d1a60 2024-06-29T14:37:08.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.637+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f868c0b3a60 con 0x7f868c0d1a60 2024-06-29T14:37:08.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.638+0000 7f86737fe640 1 -- 172.21.15.59:0/975581593 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7f86840051c0 con 0x7f868c0d1a60 2024-06-29T14:37:08.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.639+0000 7f86737fe640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8678076150 0x7f8678078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.639+0000 7f86737fe640 1 -- 172.21.15.59:0/975581593 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6875+0+0 (secure 0 0 0) 0x7f8684098560 con 0x7f868c0d1a60 2024-06-29T14:37:08.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.639+0000 7f868affd640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8678076150 0x7f8678078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.640+0000 7f868affd640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8678076150 0x7f8678078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f868c0b1fc0 tx=0x7f867c009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.643+0000 7f86737fe640 1 -- 172.21.15.59:0/975581593 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8684061cf0 con 0x7f868c0d1a60 2024-06-29T14:37:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.836+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f868c0b6aa0 con 0x7f868c0d1a60 2024-06-29T14:37:08.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.837+0000 7f86737fe640 1 -- 172.21.15.59:0/975581593 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+18130 (secure 0 0 0) 0x7f86840615f0 con 0x7f868c0d1a60 2024-06-29T14:37:08.835 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:08.836 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":76,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:06.779564+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:35:04.881993+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-06-29T14:35:05.467776+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":75,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2f","mappings":[{"from":0,"to":6}]},{"pgid":"8.39","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2661755582":"2024-06-30T14:37:06.779521+0000","172.21.15.153:6832/2118890682":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2930251413":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/1279472232":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/3733140807":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/1176138421":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/106596755":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6833/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6832/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/1025227607":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/719687282":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/387060511":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/297750041":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6833/2118890682":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2149747529":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2704927245":"2024-06-30T14:35:23.828270+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:08.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8678076150 msgr2=0x7f8678078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8678076150 0x7f8678078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f868c0b1fc0 tx=0x7f867c009040 comp rx=0 tx=0).stop 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0d1a60 msgr2=0x7f868c14a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0d1a60 0x7f868c14a170 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f868400ea10 tx=0x7f868400eed0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 shutdown_connections 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8678076150 0x7f8678078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868c0d1a60 0x7f868c14a170 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f868c0b2fa0 0x7f868c06f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 --2- 172.21.15.59:0/975581593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f868c0b2270 0x7f868c14ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 >> 172.21.15.59:0/975581593 conn(0x7f868c073e10 msgr2=0x7f868c15bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 shutdown_connections 2024-06-29T14:37:08.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.839+0000 7f86917da640 1 -- 172.21.15.59:0/975581593 wait complete. 2024-06-29T14:37:08.849 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:37:08.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.970+0000 7ff093a47640 1 -- 172.21.15.59:0/3143562975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 msgr2=0x7ff074005020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.970+0000 7ff093a47640 1 --2- 172.21.15.59:0/3143562975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 0x7ff074005020 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff07c001d90 tx=0x7ff074006940 comp rx=0 tx=0).stop 2024-06-29T14:37:08.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.970+0000 7ff091fe4640 1 -- 172.21.15.59:0/3143562975 reap_dead start 2024-06-29T14:37:08.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.970+0000 7ff093a47640 1 -- 172.21.15.59:0/3143562975 shutdown_connections 2024-06-29T14:37:08.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.970+0000 7ff093a47640 1 -- 172.21.15.59:0/3143562975 >> 172.21.15.59:0/3143562975 conn(0x7ff08c0b24c0 msgr2=0x7ff08c0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.970+0000 7ff093a47640 1 -- 172.21.15.59:0/3143562975 shutdown_connections 2024-06-29T14:37:08.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.970+0000 7ff093a47640 1 -- 172.21.15.59:0/3143562975 wait complete. 2024-06-29T14:37:08.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.972+0000 7ff093a47640 1 Processor -- start 2024-06-29T14:37:08.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.972+0000 7ff093a47640 1 -- start start 2024-06-29T14:37:08.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff093a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 0x7ff08c151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff093a47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08c13d3e0 0x7ff08c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff093a47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff08c159420 0x7ff08c1f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff093a47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c06ca70 con 0x7ff074004bb0 2024-06-29T14:37:08.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff093a47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c06c8f0 con 0x7ff08c13d3e0 2024-06-29T14:37:08.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff093a47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff08c0cccc0 con 0x7ff08c159420 2024-06-29T14:37:08.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff0917e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 0x7ff08c151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff0917e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 0x7ff08c151990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58060/0 (socket says 172.21.15.59:58060) 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff0917e3640 1 -- 172.21.15.59:0/2788370554 learned_addr learned my addr 172.21.15.59:0/2788370554 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff090fe2640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08c13d3e0 0x7ff08c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.973+0000 7ff091fe4640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff08c159420 0x7ff08c1f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff0917e3640 1 -- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff08c159420 msgr2=0x7ff08c1f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff0917e3640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff08c159420 0x7ff08c1f60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff0917e3640 1 -- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08c13d3e0 msgr2=0x7ff08c158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff0917e3640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08c13d3e0 0x7ff08c158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff0917e3640 1 -- 172.21.15.59:0/2788370554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08c159830 con 0x7ff074004bb0 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff090fe2640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08c13d3e0 0x7ff08c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff0917e3640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 0x7ff08c151990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff07c02ac80 tx=0x7ff07c003a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff089ffb640 1 -- 172.21.15.59:0/2788370554 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07c006440 con 0x7ff074004bb0 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff089ffb640 1 -- 172.21.15.59:0/2788370554 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff07c0065e0 con 0x7ff074004bb0 2024-06-29T14:37:08.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08c151ed0 con 0x7ff074004bb0 2024-06-29T14:37:08.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff089ffb640 1 -- 172.21.15.59:0/2788370554 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07c003e40 con 0x7ff074004bb0 2024-06-29T14:37:08.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.974+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff08c1f4090 con 0x7ff074004bb0 2024-06-29T14:37:08.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.975+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff08c0cd200 con 0x7ff074004bb0 2024-06-29T14:37:08.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.976+0000 7ff089ffb640 1 -- 172.21.15.59:0/2788370554 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7ff07c033480 con 0x7ff074004bb0 2024-06-29T14:37:08.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.977+0000 7ff089ffb640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff078076120 0x7ff078078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.977+0000 7ff089ffb640 1 -- 172.21.15.59:0/2788370554 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6875+0+0 (secure 0 0 0) 0x7ff07c0b7e30 con 0x7ff074004bb0 2024-06-29T14:37:08.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.977+0000 7ff090fe2640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff078076120 0x7ff078078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.977+0000 7ff090fe2640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff078076120 0x7ff078078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff080001410 tx=0x7ff080000ed0 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:08.981+0000 7ff089ffb640 1 -- 172.21.15.59:0/2788370554 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff08c159830 con 0x7ff074004bb0 2024-06-29T14:37:09.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.169+0000 7ff089ffb640 1 -- 172.21.15.59:0/2788370554 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 29) ==== 98247+0+0 (secure 0 0 0) 0x7ff07c07d070 con 0x7ff074004bb0 2024-06-29T14:37:09.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.223+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff08c1e84b0 con 0x7ff074004bb0 2024-06-29T14:37:09.222 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T14:37:09.224+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:37:09.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.227+0000 7ff089ffb640 1 -- 172.21.15.59:0/2788370554 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) ==== 161+0+0 (secure 0 0 0) 0x7ff07c0814e0 con 0x7ff074004bb0 2024-06-29T14:37:09.225 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:37:09.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff078076120 msgr2=0x7ff078078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff078076120 0x7ff078078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff080001410 tx=0x7ff080000ed0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 msgr2=0x7ff08c151990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 0x7ff08c151990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff07c02ac80 tx=0x7ff07c003a60 comp rx=0 tx=0).stop 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 shutdown_connections 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff078076120 0x7ff078078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff08c159420 0x7ff08c1f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08c13d3e0 0x7ff08c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 --2- 172.21.15.59:0/2788370554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff074004bb0 0x7ff08c151990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 >> 172.21.15.59:0/2788370554 conn(0x7ff08c0b24c0 msgr2=0x7ff08c0ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 shutdown_connections 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.230+0000 7ff093a47640 1 -- 172.21.15.59:0/2788370554 wait complete. 2024-06-29T14:37:09.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:37:09.292 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:37:09.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.365+0000 7f7d7499a640 1 -- 172.21.15.59:0/2293655729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 msgr2=0x7f7d700b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.365+0000 7f7d7499a640 1 --2- 172.21.15.59:0/2293655729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 0x7f7d700b5cb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7d60009830 tx=0x7f7d6002a510 comp rx=0 tx=0).stop 2024-06-29T14:37:09.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.365+0000 7f7d7499a640 1 -- 172.21.15.59:0/2293655729 shutdown_connections 2024-06-29T14:37:09.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.365+0000 7f7d7499a640 1 --2- 172.21.15.59:0/2293655729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d700d1130 0x7f7d7015f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.365+0000 7f7d7499a640 1 --2- 172.21.15.59:0/2293655729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 0x7f7d700b5cb0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.365+0000 7f7d7499a640 1 -- 172.21.15.59:0/2293655729 >> 172.21.15.59:0/2293655729 conn(0x7f7d700b6ed0 msgr2=0x7f7d700b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:09.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.365+0000 7f7d7499a640 1 -- 172.21.15.59:0/2293655729 shutdown_connections 2024-06-29T14:37:09.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.366+0000 7f7d7499a640 1 -- 172.21.15.59:0/2293655729 wait complete. 2024-06-29T14:37:09.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.367+0000 7f7d7499a640 1 Processor -- start 2024-06-29T14:37:09.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.368+0000 7f7d7499a640 1 -- start start 2024-06-29T14:37:09.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d7499a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d700d1130 0x7f7d7013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:09.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d7499a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 0x7f7d700c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:09.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d7499a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d7015fd30 0x7f7d700c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:09.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d7499a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7015c910 con 0x7f7d70145650 2024-06-29T14:37:09.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d7499a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7015eba0 con 0x7f7d700d1130 2024-06-29T14:37:09.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d7499a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7015ca90 con 0x7f7d7015fd30 2024-06-29T14:37:09.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 0x7f7d700c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:09.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 0x7f7d700c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58090/0 (socket says 172.21.15.59:58090) 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6dd9b640 1 -- 172.21.15.59:0/1249614049 learned_addr learned my addr 172.21.15.59:0/1249614049 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6e59c640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d700d1130 0x7f7d7013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6ed9d640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d7015fd30 0x7f7d700c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6dd9b640 1 -- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d7015fd30 msgr2=0x7f7d700c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6dd9b640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d7015fd30 0x7f7d700c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6dd9b640 1 -- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d700d1130 msgr2=0x7f7d7013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6dd9b640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d700d1130 0x7f7d7013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.369+0000 7f7d6dd9b640 1 -- 172.21.15.59:0/1249614049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d701f44d0 con 0x7f7d70145650 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.370+0000 7f7d6dd9b640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 0x7f7d700c96c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800d680 tx=0x7f7d5800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.370+0000 7f7d66ffd640 1 -- 172.21.15.59:0/1249614049 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58017070 con 0x7f7d70145650 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.370+0000 7f7d66ffd640 1 -- 172.21.15.59:0/1249614049 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d58009ab0 con 0x7f7d70145650 2024-06-29T14:37:09.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.370+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d701f47b0 con 0x7f7d70145650 2024-06-29T14:37:09.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.370+0000 7f7d66ffd640 1 -- 172.21.15.59:0/1249614049 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58010460 con 0x7f7d70145650 2024-06-29T14:37:09.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.370+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d701f4d70 con 0x7f7d70145650 2024-06-29T14:37:09.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.371+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7015cfc0 con 0x7f7d70145650 2024-06-29T14:37:09.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.372+0000 7f7d66ffd640 1 -- 172.21.15.59:0/1249614049 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 29) ==== 98247+0+0 (secure 0 0 0) 0x7f7d5801a070 con 0x7f7d70145650 2024-06-29T14:37:09.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.372+0000 7f7d66ffd640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7d50076390 0x7f7d500788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:09.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.372+0000 7f7d66ffd640 1 -- 172.21.15.59:0/1249614049 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6933+0+0 (secure 0 0 0) 0x7f7d58097cb0 con 0x7f7d70145650 2024-06-29T14:37:09.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.372+0000 7f7d6e59c640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7d50076390 0x7f7d500788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:09.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.373+0000 7f7d6e59c640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7d50076390 0x7f7d500788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d60034000 tx=0x7f7d60004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:09.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.377+0000 7f7d66ffd640 1 -- 172.21.15.59:0/1249614049 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7d58060960 con 0x7f7d70145650 2024-06-29T14:37:09.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:09.610+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7d700b77d0 con 0x7f7d70145650 2024-06-29T14:37:10.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.228+0000 7f7d66ffd640 1 -- 172.21.15.59:0/1249614049 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) ==== 104+0+0 (secure 0 0 0) 0x7f7d58064c60 con 0x7f7d70145650 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7d50076390 msgr2=0x7f7d500788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7d50076390 0x7f7d500788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d60034000 tx=0x7f7d60004730 comp rx=0 tx=0).stop 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 msgr2=0x7f7d700c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 0x7f7d700c96c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7d5800d680 tx=0x7f7d5800db40 comp rx=0 tx=0).stop 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 shutdown_connections 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7d50076390 0x7f7d500788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d7015fd30 0x7f7d700c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d70145650 0x7f7d700c96c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 --2- 172.21.15.59:0/1249614049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d700d1130 0x7f7d7013dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.230+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 >> 172.21.15.59:0/1249614049 conn(0x7f7d700b6ed0 msgr2=0x7f7d700b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:10.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.231+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 shutdown_connections 2024-06-29T14:37:10.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.231+0000 7f7d7499a640 1 -- 172.21.15.59:0/1249614049 wait complete. 2024-06-29T14:37:10.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:37:10.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.370+0000 7fe5269cb640 1 -- 172.21.15.59:0/165233802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200d2f70 msgr2=0x7fe5200d3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.370+0000 7fe5269cb640 1 --2- 172.21.15.59:0/165233802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200d2f70 0x7fe5200d3380 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe5200b34d0 tx=0x7fe51802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.370+0000 7fe5269cb640 1 -- 172.21.15.59:0/165233802 shutdown_connections 2024-06-29T14:37:10.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.370+0000 7fe5269cb640 1 --2- 172.21.15.59:0/165233802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200d2f70 0x7fe5200d3380 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.370+0000 7fe5269cb640 1 --2- 172.21.15.59:0/165233802 >> v2:172.21.15.153:3300/0 conn(0x7fe5200b7150 0x7fe5200b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.370+0000 7fe5269cb640 1 -- 172.21.15.59:0/165233802 >> 172.21.15.59:0/165233802 conn(0x7fe5200b5590 msgr2=0x7fe5200b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:10.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.370+0000 7fe5269cb640 1 -- 172.21.15.59:0/165233802 shutdown_connections 2024-06-29T14:37:10.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.370+0000 7fe5269cb640 1 -- 172.21.15.59:0/165233802 wait complete. 2024-06-29T14:37:10.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.372+0000 7fe5269cb640 1 Processor -- start 2024-06-29T14:37:10.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.372+0000 7fe5269cb640 1 -- start start 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.373+0000 7fe5269cb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200b2a50 0x7fe5200cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.373+0000 7fe5269cb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5200b7150 0x7fe520150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.373+0000 7fe5269cb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5200d2f70 0x7fe5200cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.373+0000 7fe5269cb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe5200b4510 con 0x7fe5200b2a50 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.373+0000 7fe5269cb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe5200b53e0 con 0x7fe5200d2f70 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.373+0000 7fe5269cb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe5200b4690 con 0x7fe5200b7150 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.373+0000 7fe51ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200b2a50 0x7fe5200cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.373+0000 7fe51ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200b2a50 0x7fe5200cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58098/0 (socket says 172.21.15.59:58098) 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51ffff640 1 -- 172.21.15.59:0/3000619692 learned_addr learned my addr 172.21.15.59:0/3000619692 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe524f68640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5200d2f70 0x7fe5200cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51f7fe640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5200b7150 0x7fe520150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:10.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51ffff640 1 -- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5200b7150 msgr2=0x7fe520150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51ffff640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5200b7150 0x7fe520150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51ffff640 1 -- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5200d2f70 msgr2=0x7fe5200cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51ffff640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5200d2f70 0x7fe5200cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51ffff640 1 -- 172.21.15.59:0/3000619692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5200cf200 con 0x7fe5200b2a50 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51ffff640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200b2a50 0x7fe5200cecc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe51000c3a0 tx=0x7fe51000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51cff9640 1 -- 172.21.15.59:0/3000619692 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510011790 con 0x7fe5200b2a50 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51cff9640 1 -- 172.21.15.59:0/3000619692 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe510017070 con 0x7fe5200b2a50 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe51cff9640 1 -- 172.21.15.59:0/3000619692 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510011cc0 con 0x7fe5200b2a50 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe520151410 con 0x7fe5200b2a50 2024-06-29T14:37:10.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.374+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5200cb530 con 0x7fe5200b2a50 2024-06-29T14:37:10.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.375+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5200b4b80 con 0x7fe5200b2a50 2024-06-29T14:37:10.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.376+0000 7fe51cff9640 1 -- 172.21.15.59:0/3000619692 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 29) ==== 98247+0+0 (secure 0 0 0) 0x7fe51001c460 con 0x7fe5200b2a50 2024-06-29T14:37:10.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.377+0000 7fe51cff9640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe50c076390 0x7fe50c0788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:10.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.377+0000 7fe51cff9640 1 -- 172.21.15.59:0/3000619692 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6933+0+0 (secure 0 0 0) 0x7fe510097c60 con 0x7fe5200b2a50 2024-06-29T14:37:10.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.377+0000 7fe51f7fe640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe50c076390 0x7fe50c0788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:10.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.377+0000 7fe51f7fe640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe50c076390 0x7fe50c0788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe514005d70 tx=0x7fe514005d00 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.381+0000 7fe51cff9640 1 -- 172.21.15.59:0/3000619692 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe5200cf200 con 0x7fe5200b2a50 2024-06-29T14:37:10.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:10.570+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe52008ccd0 con 0x7fe5200b2a50 2024-06-29T14:37:11.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.235+0000 7fe51cff9640 1 -- 172.21.15.59:0/3000619692 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) ==== 181+0+0 (secure 0 0 0) 0x7fe5200d38c0 con 0x7fe5200b2a50 2024-06-29T14:37:11.233 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:37:11.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.239+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe50c076390 msgr2=0x7fe50c0788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.239+0000 7fe5269cb640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe50c076390 0x7fe50c0788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe514005d70 tx=0x7fe514005d00 comp rx=0 tx=0).stop 2024-06-29T14:37:11.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.239+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200b2a50 msgr2=0x7fe5200cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.239+0000 7fe5269cb640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200b2a50 0x7fe5200cecc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe51000c3a0 tx=0x7fe51000c860 comp rx=0 tx=0).stop 2024-06-29T14:37:11.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.240+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 shutdown_connections 2024-06-29T14:37:11.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.240+0000 7fe5269cb640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe50c076390 0x7fe50c0788a0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe514005d70 tx=0x7fe514005d00 comp rx=0 tx=0).stop 2024-06-29T14:37:11.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.240+0000 7fe5269cb640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5200d2f70 0x7fe5200cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.240+0000 7fe5269cb640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5200b7150 0x7fe520150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.240+0000 7fe5269cb640 1 --2- 172.21.15.59:0/3000619692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5200b2a50 0x7fe5200cecc0 secure :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe51000c3a0 tx=0x7fe51000c860 comp rx=0 tx=0).stop 2024-06-29T14:37:11.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.240+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 >> 172.21.15.59:0/3000619692 conn(0x7fe5200b5590 msgr2=0x7fe52006c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:11.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.241+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 shutdown_connections 2024-06-29T14:37:11.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.241+0000 7fe5269cb640 1 -- 172.21.15.59:0/3000619692 wait complete. 2024-06-29T14:37:11.255 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:37:11.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.388+0000 7f8ee085b640 1 -- 172.21.15.59:0/2525314007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0d3680 msgr2=0x7f8edc0b7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.388+0000 7f8ee085b640 1 --2- 172.21.15.59:0/2525314007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0d3680 0x7f8edc0b7560 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0d2c60 tx=0x7f8ed402a920 comp rx=0 tx=0).stop 2024-06-29T14:37:11.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.388+0000 7f8ee085b640 1 -- 172.21.15.59:0/2525314007 shutdown_connections 2024-06-29T14:37:11.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.388+0000 7f8ee085b640 1 --2- 172.21.15.59:0/2525314007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0d3680 0x7f8edc0b7560 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.388+0000 7f8ee085b640 1 -- 172.21.15.59:0/2525314007 >> 172.21.15.59:0/2525314007 conn(0x7f8edc0b5590 msgr2=0x7f8edc0b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:11.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.388+0000 7f8ee085b640 1 -- 172.21.15.59:0/2525314007 shutdown_connections 2024-06-29T14:37:11.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.388+0000 7f8ee085b640 1 -- 172.21.15.59:0/2525314007 wait complete. 2024-06-29T14:37:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.390+0000 7f8ee085b640 1 Processor -- start 2024-06-29T14:37:11.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.390+0000 7f8ee085b640 1 -- start start 2024-06-29T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8ee085b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0b2a50 0x7f8edc157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8ee085b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc0b7150 0x7f8edc1539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8ee085b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc0c9400 0x7f8edc154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8ee085b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0b4690 con 0x7f8edc0b2a50 2024-06-29T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8ee085b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0b4510 con 0x7f8edc0c9400 2024-06-29T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8ee085b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8edc0b4810 con 0x7f8edc0b7150 2024-06-29T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8eda59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0b2a50 0x7f8edc157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8eda59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0b2a50 0x7f8edc157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58116/0 (socket says 172.21.15.59:58116) 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8eda59c640 1 -- 172.21.15.59:0/1317756396 learned_addr learned my addr 172.21.15.59:0/1317756396 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8edad9d640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc0c9400 0x7f8edc154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8ed9d9b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc0b7150 0x7f8edc1539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8eda59c640 1 -- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc0b7150 msgr2=0x7f8edc1539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8eda59c640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc0b7150 0x7f8edc1539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8eda59c640 1 -- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc0c9400 msgr2=0x7f8edc154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8eda59c640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc0c9400 0x7f8edc154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.391+0000 7f8eda59c640 1 -- 172.21.15.59:0/1317756396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc0c9c30 con 0x7f8edc0b2a50 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.392+0000 7f8edad9d640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc0c9400 0x7f8edc154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.392+0000 7f8eda59c640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0b2a50 0x7f8edc157750 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00c3a0 tx=0x7f8ecc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.392+0000 7f8ecaffd640 1 -- 172.21.15.59:0/1317756396 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc017070 con 0x7f8edc0b2a50 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.392+0000 7f8ecaffd640 1 -- 172.21.15.59:0/1317756396 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ecc0035f0 con 0x7f8edc0b2a50 2024-06-29T14:37:11.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.392+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8edc0c9f10 con 0x7f8edc0b2a50 2024-06-29T14:37:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.392+0000 7f8ecaffd640 1 -- 172.21.15.59:0/1317756396 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc003a40 con 0x7f8edc0b2a50 2024-06-29T14:37:11.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.392+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8edc0ca4d0 con 0x7f8edc0b2a50 2024-06-29T14:37:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.393+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8edc0b4c90 con 0x7f8edc0b2a50 2024-06-29T14:37:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.394+0000 7f8ecaffd640 1 -- 172.21.15.59:0/1317756396 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f8ecc01a070 con 0x7f8edc0b2a50 2024-06-29T14:37:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.394+0000 7f8ecaffd640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8ebc076420 0x7f8ebc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.394+0000 7f8ecaffd640 1 -- 172.21.15.59:0/1317756396 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6495+0+0 (secure 0 0 0) 0x7f8ecc096fa0 con 0x7f8edc0b2a50 2024-06-29T14:37:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.395+0000 7f8ed9d9b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8ebc076420 0x7f8ebc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:11.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.395+0000 7f8ed9d9b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8ebc076420 0x7f8ebc078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000f60 tx=0x7f8ed0009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:11.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.399+0000 7f8ecaffd640 1 -- 172.21.15.59:0/1317756396 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8ecc060450 con 0x7f8edc0b2a50 2024-06-29T14:37:11.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:11.596+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8edc0cf640 con 0x7f8edc0b2a50 2024-06-29T14:37:12.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.236+0000 7f8ecaffd640 1 -- 172.21.15.59:0/1317756396 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) ==== 169+0+0 (secure 0 0 0) 0x7f8ecc05fd50 con 0x7f8edc0b2a50 2024-06-29T14:37:12.235 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T14:37:12.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8ebc076420 msgr2=0x7f8ebc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8ebc076420 0x7f8ebc078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000f60 tx=0x7f8ed0009040 comp rx=0 tx=0).stop 2024-06-29T14:37:12.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0b2a50 msgr2=0x7f8edc157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0b2a50 0x7f8edc157750 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00c3a0 tx=0x7f8ecc00c860 comp rx=0 tx=0).stop 2024-06-29T14:37:12.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 shutdown_connections 2024-06-29T14:37:12.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8ebc076420 0x7f8ebc078930 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000f60 tx=0x7f8ed0009040 comp rx=0 tx=0).stop 2024-06-29T14:37:12.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc0c9400 0x7f8edc154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8edc0b7150 0x7f8edc1539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 --2- 172.21.15.59:0/1317756396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0b2a50 0x7f8edc157750 secure :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00c3a0 tx=0x7f8ecc00c860 comp rx=0 tx=0).stop 2024-06-29T14:37:12.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.240+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 >> 172.21.15.59:0/1317756396 conn(0x7f8edc0b5590 msgr2=0x7f8edc15b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:12.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.241+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 shutdown_connections 2024-06-29T14:37:12.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.241+0000 7f8ee085b640 1 -- 172.21.15.59:0/1317756396 wait complete. 2024-06-29T14:37:12.253 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:37:12.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.379+0000 7f1ce35f5640 1 -- 172.21.15.59:0/3513521408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cc4005300 msgr2=0x7f1cc400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.379+0000 7f1ce35f5640 1 --2- 172.21.15.59:0/3513521408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cc4005300 0x7f1cc400b9c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4005fd0 tx=0x7f1cc4005830 comp rx=0 tx=0).stop 2024-06-29T14:37:12.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.379+0000 7f1ce35f5640 1 -- 172.21.15.59:0/3513521408 shutdown_connections 2024-06-29T14:37:12.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.379+0000 7f1ce35f5640 1 --2- 172.21.15.59:0/3513521408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cc4005300 0x7f1cc400b9c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.379+0000 7f1ce35f5640 1 -- 172.21.15.59:0/3513521408 >> 172.21.15.59:0/3513521408 conn(0x7f1cdc06c950 msgr2=0x7f1cdc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:12.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.379+0000 7f1ce35f5640 1 -- 172.21.15.59:0/3513521408 shutdown_connections 2024-06-29T14:37:12.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.379+0000 7f1ce35f5640 1 -- 172.21.15.59:0/3513521408 wait complete. 2024-06-29T14:37:12.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.380+0000 7f1ce35f5640 1 Processor -- start 2024-06-29T14:37:12.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.381+0000 7f1ce35f5640 1 -- start start 2024-06-29T14:37:12.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce35f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc4005300 0x7f1cdc13c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce35f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc13d2c0 0x7f1cdc13d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce35f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cdc14e510 0x7f1cdc14e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce35f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc0cfc00 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce35f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc0cfa80 con 0x7f1cc4005300 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce35f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1cdc0cfd80 con 0x7f1cdc14e510 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc13d2c0 0x7f1cdc13d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc13d2c0 0x7f1cdc13d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58136/0 (socket says 172.21.15.59:58136) 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 -- 172.21.15.59:0/1092136110 learned_addr learned my addr 172.21.15.59:0/1092136110 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce1391640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc4005300 0x7f1cdc13c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce1b92640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cdc14e510 0x7f1cdc14e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 -- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cdc14e510 msgr2=0x7f1cdc14e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cdc14e510 0x7f1cdc14e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 -- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc4005300 msgr2=0x7f1cdc13c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc4005300 0x7f1cdc13c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 -- 172.21.15.59:0/1092136110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cdc144fe0 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce1b92640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cdc14e510 0x7f1cdc14e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:12.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.382+0000 7f1ce0b90640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc13d2c0 0x7f1cdc13d6d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0006660 tx=0x7f1cd00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:12.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.383+0000 7f1cd9ffb640 1 -- 172.21.15.59:0/1092136110 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0011700 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.383+0000 7f1cd9ffb640 1 -- 172.21.15.59:0/1092136110 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1cd0016020 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.383+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cdc142000 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.383+0000 7f1cd9ffb640 1 -- 172.21.15.59:0/1092136110 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0011ac0 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.383+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cdc15bd20 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.384+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cdc144d50 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.385+0000 7f1cd9ffb640 1 -- 172.21.15.59:0/1092136110 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f1cd0011c60 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.385+0000 7f1cd9ffb640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f1cc8076420 0x7f1cc8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.386+0000 7f1cd9ffb640 1 -- 172.21.15.59:0/1092136110 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7f1cd00983d0 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.386+0000 7f1ce1391640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f1cc8076420 0x7f1cc8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.386+0000 7f1ce1391640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f1cc8076420 0x7f1cc8078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00d9b0 tx=0x7f1ccc006b70 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:12.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.390+0000 7f1cd9ffb640 1 -- 172.21.15.59:0/1092136110 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1cd0020020 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.586+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f1cdc141df0 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.590+0000 7f1cd9ffb640 1 -- 172.21.15.59:0/1092136110 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v35) ==== 71+0+0 (secure 0 0 0) 0x7f1cd00619e0 con 0x7f1cdc13d2c0 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f1cc8076420 msgr2=0x7f1cc8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f1cc8076420 0x7f1cc8078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00d9b0 tx=0x7f1ccc006b70 comp rx=0 tx=0).stop 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc13d2c0 msgr2=0x7f1cdc13d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc13d2c0 0x7f1cdc13d6d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0006660 tx=0x7f1cd00039c0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 shutdown_connections 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f1cc8076420 0x7f1cc8078930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cdc14e510 0x7f1cdc14e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc13d2c0 0x7f1cdc13d6d0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 --2- 172.21.15.59:0/1092136110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc4005300 0x7f1cdc13c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 >> 172.21.15.59:0/1092136110 conn(0x7f1cdc06c950 msgr2=0x7f1cdc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 shutdown_connections 2024-06-29T14:37:12.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.592+0000 7f1ce35f5640 1 -- 172.21.15.59:0/1092136110 wait complete. 2024-06-29T14:37:12.601 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-06-29T14:37:12.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.721+0000 7fa050d19640 1 -- 172.21.15.59:0/3852154160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 msgr2=0x7fa04c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.721+0000 7fa050d19640 1 --2- 172.21.15.59:0/3852154160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 0x7fa04c0b3b50 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa034009830 tx=0x7fa03402a510 comp rx=0 tx=0).stop 2024-06-29T14:37:12.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.721+0000 7fa050d19640 1 -- 172.21.15.59:0/3852154160 shutdown_connections 2024-06-29T14:37:12.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.721+0000 7fa050d19640 1 --2- 172.21.15.59:0/3852154160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 0x7fa04c0b3b50 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.721+0000 7fa050d19640 1 --2- 172.21.15.59:0/3852154160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa04c1458a0 0x7fa04c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.721+0000 7fa050d19640 1 -- 172.21.15.59:0/3852154160 >> 172.21.15.59:0/3852154160 conn(0x7fa04c06c950 msgr2=0x7fa04c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:12.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.721+0000 7fa050d19640 1 -- 172.21.15.59:0/3852154160 shutdown_connections 2024-06-29T14:37:12.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.721+0000 7fa050d19640 1 -- 172.21.15.59:0/3852154160 wait complete. 2024-06-29T14:37:12.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.723+0000 7fa050d19640 1 Processor -- start 2024-06-29T14:37:12.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.723+0000 7fa050d19640 1 -- start start 2024-06-29T14:37:12.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa050d19640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 0x7fa04c0ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa050d19640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa04c15c2f0 0x7fa04c1f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa050d19640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa04c1f3420 0x7fa04c0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa050d19640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c1601a0 con 0x7fa04c0b3740 2024-06-29T14:37:12.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa050d19640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c160020 con 0x7fa04c15c2f0 2024-06-29T14:37:12.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa050d19640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa04c160320 con 0x7fa04c1f3420 2024-06-29T14:37:12.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa04a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 0x7fa04c0ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa04a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 0x7fa04c0ad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36782/0 (socket says 172.21.15.59:36782) 2024-06-29T14:37:12.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa04a59c640 1 -- 172.21.15.59:0/310254941 learned_addr learned my addr 172.21.15.59:0/310254941 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:12.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa04ad9d640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa04c1f3420 0x7fa04c0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.724+0000 7fa049d9b640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa04c15c2f0 0x7fa04c1f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa04a59c640 1 -- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa04c1f3420 msgr2=0x7fa04c0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa04a59c640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa04c1f3420 0x7fa04c0d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa04a59c640 1 -- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa04c15c2f0 msgr2=0x7fa04c1f2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa04a59c640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa04c15c2f0 0x7fa04c1f2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa04a59c640 1 -- 172.21.15.59:0/310254941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c0ab4c0 con 0x7fa04c0b3740 2024-06-29T14:37:12.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa04a59c640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 0x7fa04c0ad080 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00ea50 tx=0x7fa03c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:12.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa042ffd640 1 -- 172.21.15.59:0/310254941 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c00cdc0 con 0x7fa04c0b3740 2024-06-29T14:37:12.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa042ffd640 1 -- 172.21.15.59:0/310254941 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa03c011560 con 0x7fa04c0b3740 2024-06-29T14:37:12.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04c0aa700 con 0x7fa04c0b3740 2024-06-29T14:37:12.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa042ffd640 1 -- 172.21.15.59:0/310254941 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c011990 con 0x7fa04c0b3740 2024-06-29T14:37:12.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.725+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa04c0aacc0 con 0x7fa04c0b3740 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.726+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04c15ee30 con 0x7fa04c0b3740 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.727+0000 7fa042ffd640 1 -- 172.21.15.59:0/310254941 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fa03c011b30 con 0x7fa04c0b3740 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.727+0000 7fa042ffd640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa02c0763f0 0x7fa02c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.727+0000 7fa042ffd640 1 -- 172.21.15.59:0/310254941 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7fa03c098730 con 0x7fa04c0b3740 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.728+0000 7fa049d9b640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa02c0763f0 0x7fa02c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.728+0000 7fa049d9b640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa02c0763f0 0x7fa02c078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa034034000 tx=0x7fa034004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:12.732+0000 7fa042ffd640 1 -- 172.21.15.59:0/310254941 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa04c15ee30 con 0x7fa04c0b3740 2024-06-29T14:37:12.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.000+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) -- 0x7fa04c071bd0 con 0x7fa04c0b3740 2024-06-29T14:37:13.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.238+0000 7fa042ffd640 1 -- 172.21.15.59:0/310254941 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v567) ==== 154+0+0 (secure 0 0 0) 0x7fa04c071bd0 con 0x7fa04c0b3740 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa02c0763f0 msgr2=0x7fa02c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa02c0763f0 0x7fa02c078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa034034000 tx=0x7fa034004730 comp rx=0 tx=0).stop 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 msgr2=0x7fa04c0ad080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 0x7fa04c0ad080 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00ea50 tx=0x7fa03c00ef10 comp rx=0 tx=0).stop 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 shutdown_connections 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa02c0763f0 0x7fa02c078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa04c1f3420 0x7fa04c0d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa04c15c2f0 0x7fa04c1f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 --2- 172.21.15.59:0/310254941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0b3740 0x7fa04c0ad080 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 >> 172.21.15.59:0/310254941 conn(0x7fa04c06c950 msgr2=0x7fa04c141580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 shutdown_connections 2024-06-29T14:37:13.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.240+0000 7fa050d19640 1 -- 172.21.15.59:0/310254941 wait complete. 2024-06-29T14:37:13.250 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-06-29T14:37:13.250 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-06-29T14:37:13.250 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:37:13.251 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.371+0000 7efef8523640 1 -- 172.21.15.59:0/1123475184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00cf0f0 msgr2=0x7efef00b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.371+0000 7efef8523640 1 --2- 172.21.15.59:0/1123475184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00cf0f0 0x7efef00b2bd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efee0005fd0 tx=0x7efee002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.371+0000 7efef8523640 1 -- 172.21.15.59:0/1123475184 shutdown_connections 2024-06-29T14:37:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.371+0000 7efef8523640 1 --2- 172.21.15.59:0/1123475184 >> v2:172.21.15.153:3300/0 conn(0x7efef00d32b0 0x7efef0167c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.371+0000 7efef8523640 1 --2- 172.21.15.59:0/1123475184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00cf0f0 0x7efef00b2bd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.371+0000 7efef8523640 1 -- 172.21.15.59:0/1123475184 >> 172.21.15.59:0/1123475184 conn(0x7efef00d3ec0 msgr2=0x7efef00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.371+0000 7efef8523640 1 -- 172.21.15.59:0/1123475184 shutdown_connections 2024-06-29T14:37:13.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.371+0000 7efef8523640 1 -- 172.21.15.59:0/1123475184 wait complete. 2024-06-29T14:37:13.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.373+0000 7efef8523640 1 Processor -- start 2024-06-29T14:37:13.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.373+0000 7efef8523640 1 -- start start 2024-06-29T14:37:13.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.374+0000 7efef8523640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00b27c0 0x7efef0154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.374+0000 7efef8523640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00d32b0 0x7efef0155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.374+0000 7efef8523640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efef0159850 0x7efef0159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.374+0000 7efef8523640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efef0168320 con 0x7efef00d32b0 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.374+0000 7efef8523640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efef01681a0 con 0x7efef0159850 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.374+0000 7efef8523640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efef01684a0 con 0x7efef00b27c0 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.374+0000 7efef5abe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00d32b0 0x7efef0155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef5abe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00d32b0 0x7efef0155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36794/0 (socket says 172.21.15.59:36794) 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef5abe640 1 -- 172.21.15.59:0/3497904953 learned_addr learned my addr 172.21.15.59:0/3497904953 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef6ac0640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efef0159850 0x7efef0159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef62bf640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00b27c0 0x7efef0154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef5abe640 1 -- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00b27c0 msgr2=0x7efef0154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef5abe640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00b27c0 0x7efef0154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef5abe640 1 -- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efef0159850 msgr2=0x7efef0159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef5abe640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efef0159850 0x7efef0159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef5abe640 1 -- 172.21.15.59:0/3497904953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef015a1a0 con 0x7efef00d32b0 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef6ac0640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efef0159850 0x7efef0159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef62bf640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00b27c0 0x7efef0154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef5abe640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00d32b0 0x7efef0155720 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efee0034040 tx=0x7efee00035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efee6ffd640 1 -- 172.21.15.59:0/3497904953 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee003e030 con 0x7efef00d32b0 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efee6ffd640 1 -- 172.21.15.59:0/3497904953 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efee0003ae0 con 0x7efef00d32b0 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef0161720 con 0x7efef00d32b0 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efee6ffd640 1 -- 172.21.15.59:0/3497904953 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0035040 con 0x7efef00d32b0 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.375+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef0161160 con 0x7efef00d32b0 2024-06-29T14:37:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.376+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef00b2bd0 con 0x7efef00d32b0 2024-06-29T14:37:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.377+0000 7efee6ffd640 1 -- 172.21.15.59:0/3497904953 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7efee0041030 con 0x7efef00d32b0 2024-06-29T14:37:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.377+0000 7efee6ffd640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efed40763f0 0x7efed4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.378+0000 7efee6ffd640 1 -- 172.21.15.59:0/3497904953 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7efee00b7fc0 con 0x7efef00d32b0 2024-06-29T14:37:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.378+0000 7efef62bf640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efed40763f0 0x7efed4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.378+0000 7efef62bf640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efed40763f0 0x7efed4078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efeec007fd0 tx=0x7efeec009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.382+0000 7efee6ffd640 1 -- 172.21.15.59:0/3497904953 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efef015a1a0 con 0x7efef00d32b0 2024-06-29T14:37:13.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.573+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efef00cf360 con 0x7efef00d32b0 2024-06-29T14:37:13.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.574+0000 7efee6ffd640 1 -- 172.21.15.59:0/3497904953 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+14321 (secure 0 0 0) 0x7efee0081530 con 0x7efef00d32b0 2024-06-29T14:37:13.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:13.573 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:12.236535+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2661755582":"2024-06-30T14:37:06.779521+0000","172.21.15.153:6832/2118890682":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2930251413":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/1279472232":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/3733140807":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/1176138421":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/106596755":"2024-06-30T14:35:23.828270+0000","172.21.15.153:6833/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.59:6833/3627888249":"2024-06-30T14:37:09.225650+0000","172.21.15.153:6832/2957341575":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/1025227607":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/719687282":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/387060511":"2024-06-30T14:35:23.828270+0000","172.21.15.153:0/297750041":"2024-06-30T14:35:23.828270+0000","172.21.15.59:6835/3627888249":"2024-06-30T14:37:09.225650+0000","172.21.15.153:6833/2118890682":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2149747529":"2024-06-30T14:37:06.779521+0000","172.21.15.153:0/2704927245":"2024-06-30T14:35:23.828270+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.576+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efed40763f0 msgr2=0x7efed4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.576+0000 7efef8523640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efed40763f0 0x7efed4078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efeec007fd0 tx=0x7efeec009040 comp rx=0 tx=0).stop 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.576+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00d32b0 msgr2=0x7efef0155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.576+0000 7efef8523640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00d32b0 0x7efef0155720 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efee0034040 tx=0x7efee00035f0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.576+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 shutdown_connections 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.576+0000 7efef8523640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efed40763f0 0x7efed4078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.577+0000 7efef8523640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efef0159850 0x7efef0159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.577+0000 7efef8523640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00d32b0 0x7efef0155720 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.577+0000 7efef8523640 1 --2- 172.21.15.59:0/3497904953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efef00b27c0 0x7efef0154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.577+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 >> 172.21.15.59:0/3497904953 conn(0x7efef00d3ec0 msgr2=0x7efef00d29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.577+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 shutdown_connections 2024-06-29T14:37:13.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.577+0000 7efef8523640 1 -- 172.21.15.59:0/3497904953 wait complete. 2024-06-29T14:37:13.586 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:13.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:13.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.709+0000 7f4ae5bee640 1 -- 172.21.15.59:0/613403375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00cb650 msgr2=0x7f4ae00acfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.709+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/613403375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00cb650 0x7f4ae00acfa0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0071440 tx=0x7f4ad802aa30 comp rx=0 tx=0).stop 2024-06-29T14:37:13.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.709+0000 7f4ae5bee640 1 -- 172.21.15.59:0/613403375 shutdown_connections 2024-06-29T14:37:13.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.709+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/613403375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00cb650 0x7f4ae00acfa0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.709+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/613403375 >> v2:172.21.15.153:3300/0 conn(0x7f4ae00b52b0 0x7f4ae00b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.709+0000 7f4ae5bee640 1 -- 172.21.15.59:0/613403375 >> 172.21.15.59:0/613403375 conn(0x7f4ae00b1ea0 msgr2=0x7f4ae00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.709+0000 7f4ae5bee640 1 -- 172.21.15.59:0/613403375 shutdown_connections 2024-06-29T14:37:13.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.709+0000 7f4ae5bee640 1 -- 172.21.15.59:0/613403375 wait complete. 2024-06-29T14:37:13.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.711+0000 7f4ae5bee640 1 Processor -- start 2024-06-29T14:37:13.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.711+0000 7f4ae5bee640 1 -- start start 2024-06-29T14:37:13.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4ae5bee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00b52b0 0x7f4ae01583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4ae5bee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae00cb650 0x7f4ae014c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4ae5bee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ae015be90 0x7f4ae00c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4ae5bee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae00ad4e0 con 0x7f4ae00b52b0 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4ae5bee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae00cbc70 con 0x7f4ae00cb650 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4ae5bee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae00ad660 con 0x7f4ae015be90 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4adf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00b52b0 0x7f4ae01583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4adf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00b52b0 0x7f4ae01583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36804/0 (socket says 172.21.15.59:36804) 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4adf7fe640 1 -- 172.21.15.59:0/4183071 learned_addr learned my addr 172.21.15.59:0/4183071 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4adeffd640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae00cb650 0x7f4ae014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4adffff640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ae015be90 0x7f4ae00c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4adf7fe640 1 -- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ae015be90 msgr2=0x7f4ae00c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.712+0000 7f4adf7fe640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ae015be90 0x7f4ae00c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4adf7fe640 1 -- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae00cb650 msgr2=0x7f4ae014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4adf7fe640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae00cb650 0x7f4ae014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4adf7fe640 1 -- 172.21.15.59:0/4183071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae0072bd0 con 0x7f4ae00b52b0 2024-06-29T14:37:13.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4adffff640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ae015be90 0x7f4ae00c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:13.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4adf7fe640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00b52b0 0x7f4ae01583e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000c9a0 tx=0x7f4ad000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:13.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4ac7fff640 1 -- 172.21.15.59:0/4183071 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0007bb0 con 0x7f4ae00b52b0 2024-06-29T14:37:13.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4ac7fff640 1 -- 172.21.15.59:0/4183071 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ad0007e40 con 0x7f4ae00b52b0 2024-06-29T14:37:13.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae015b740 con 0x7f4ae00b52b0 2024-06-29T14:37:13.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4ac7fff640 1 -- 172.21.15.59:0/4183071 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad000f690 con 0x7f4ae00b52b0 2024-06-29T14:37:13.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.713+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae015b460 con 0x7f4ae00b52b0 2024-06-29T14:37:13.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.714+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae00adb10 con 0x7f4ae00b52b0 2024-06-29T14:37:13.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.715+0000 7f4ac7fff640 1 -- 172.21.15.59:0/4183071 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f4ad000f830 con 0x7f4ae00b52b0 2024-06-29T14:37:13.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.715+0000 7f4ac7fff640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f4acc0763f0 0x7f4acc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.715+0000 7f4ac7fff640 1 -- 172.21.15.59:0/4183071 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7f4ad0098260 con 0x7f4ae00b52b0 2024-06-29T14:37:13.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.715+0000 7f4adeffd640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f4acc0763f0 0x7f4acc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.716+0000 7f4adeffd640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f4acc0763f0 0x7f4acc078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0072a30 tx=0x7f4ad4009060 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:13.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.720+0000 7f4ac7fff640 1 -- 172.21.15.59:0/4183071 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ad0014030 con 0x7f4ae00b52b0 2024-06-29T14:37:13.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.911+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4ae0158920 con 0x7f4ae00b52b0 2024-06-29T14:37:13.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.911+0000 7f4ac7fff640 1 -- 172.21.15.59:0/4183071 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7f4ad0061860 con 0x7f4ae00b52b0 2024-06-29T14:37:13.909 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:37:13.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f4acc0763f0 msgr2=0x7f4acc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f4acc0763f0 0x7f4acc078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0072a30 tx=0x7f4ad4009060 comp rx=0 tx=0).stop 2024-06-29T14:37:13.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00b52b0 msgr2=0x7f4ae01583e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00b52b0 0x7f4ae01583e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ad000c9a0 tx=0x7f4ad000ce60 comp rx=0 tx=0).stop 2024-06-29T14:37:13.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 shutdown_connections 2024-06-29T14:37:13.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f4acc0763f0 0x7f4acc078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ae015be90 0x7f4ae00c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ae00cb650 0x7f4ae014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 --2- 172.21.15.59:0/4183071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ae00b52b0 0x7f4ae01583e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 >> 172.21.15.59:0/4183071 conn(0x7f4ae00b1ea0 msgr2=0x7f4ae00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 shutdown_connections 2024-06-29T14:37:13.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:13.913+0000 7f4ae5bee640 1 -- 172.21.15.59:0/4183071 wait complete. 2024-06-29T14:37:13.922 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-06-29T14:37:14.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.041+0000 7fa355646640 1 -- 172.21.15.59:0/3110873741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 msgr2=0x7fa350162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:14.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.041+0000 7fa355646640 1 --2- 172.21.15.59:0/3110873741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 0x7fa350162c10 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa348005fd0 tx=0x7fa34802aa90 comp rx=0 tx=0).stop 2024-06-29T14:37:14.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.041+0000 7fa355646640 1 -- 172.21.15.59:0/3110873741 shutdown_connections 2024-06-29T14:37:14.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.041+0000 7fa355646640 1 --2- 172.21.15.59:0/3110873741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 0x7fa350162c10 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:14.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.041+0000 7fa355646640 1 --2- 172.21.15.59:0/3110873741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3500ce950 0x7fa3501622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:14.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.041+0000 7fa355646640 1 -- 172.21.15.59:0/3110873741 >> 172.21.15.59:0/3110873741 conn(0x7fa3500b5610 msgr2=0x7fa3500b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:14.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.041+0000 7fa355646640 1 -- 172.21.15.59:0/3110873741 shutdown_connections 2024-06-29T14:37:14.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.042+0000 7fa355646640 1 -- 172.21.15.59:0/3110873741 wait complete. 2024-06-29T14:37:14.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.043+0000 7fa355646640 1 Processor -- start 2024-06-29T14:37:14.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.043+0000 7fa355646640 1 -- start start 2024-06-29T14:37:14.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa355646640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35008c190 0x7fa350152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:14.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa355646640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3500ce950 0x7fa350155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:14.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa355646640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 0x7fa35015c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa355646640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa3500b7fa0 con 0x7fa350162800 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa355646640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa3500b7e20 con 0x7fa3500ce950 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa355646640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa3500b8120 con 0x7fa35008c190 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 0x7fa35015c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 0x7fa35015c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36830/0 (socket says 172.21.15.59:36830) 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34f7fe640 1 -- 172.21.15.59:0/187123338 learned_addr learned my addr 172.21.15.59:0/187123338 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34e7fc640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3500ce950 0x7fa350155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34effd640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35008c190 0x7fa350152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34f7fe640 1 -- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35008c190 msgr2=0x7fa350152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34f7fe640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35008c190 0x7fa350152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34f7fe640 1 -- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3500ce950 msgr2=0x7fa350155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34f7fe640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3500ce950 0x7fa350155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34f7fe640 1 -- 172.21.15.59:0/187123338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3501560b0 con 0x7fa350162800 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.045+0000 7fa34effd640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35008c190 0x7fa350152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:14.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.046+0000 7fa34f7fe640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 0x7fa35015c4c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa348009d90 tx=0x7fa348003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:14.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.046+0000 7fa3377fe640 1 -- 172.21.15.59:0/187123338 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348044070 con 0x7fa350162800 2024-06-29T14:37:14.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.046+0000 7fa3377fe640 1 -- 172.21.15.59:0/187123338 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa348004250 con 0x7fa350162800 2024-06-29T14:37:14.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.046+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa350157cc0 con 0x7fa350162800 2024-06-29T14:37:14.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.046+0000 7fa3377fe640 1 -- 172.21.15.59:0/187123338 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348004660 con 0x7fa350162800 2024-06-29T14:37:14.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.046+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa35015f1d0 con 0x7fa350162800 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.047+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3501622c0 con 0x7fa350162800 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.047+0000 7fa3377fe640 1 -- 172.21.15.59:0/187123338 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fa348047070 con 0x7fa350162800 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.048+0000 7fa3377fe640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa33c0763f0 0x7fa33c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.048+0000 7fa3377fe640 1 -- 172.21.15.59:0/187123338 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7fa3480b8900 con 0x7fa350162800 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.048+0000 7fa34effd640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa33c0763f0 0x7fa33c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.049+0000 7fa34effd640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa33c0763f0 0x7fa33c078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa340007fd0 tx=0x7fa340009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:14.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.052+0000 7fa3377fe640 1 -- 172.21.15.59:0/187123338 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa350157cc0 con 0x7fa350162800 2024-06-29T14:37:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:14.318+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) -- 0x7fa3500ca1c0 con 0x7fa350162800 2024-06-29T14:37:15.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.246+0000 7fa3377fe640 1 -- 172.21.15.59:0/187123338 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v569) ==== 159+0+0 (secure 0 0 0) 0x7fa348032030 con 0x7fa350162800 2024-06-29T14:37:15.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa33c0763f0 msgr2=0x7fa33c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa33c0763f0 0x7fa33c078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa340007fd0 tx=0x7fa340009040 comp rx=0 tx=0).stop 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 msgr2=0x7fa35015c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 0x7fa35015c4c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa348009d90 tx=0x7fa348003ce0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 shutdown_connections 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fa33c0763f0 0x7fa33c078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa350162800 0x7fa35015c4c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3500ce950 0x7fa350155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 --2- 172.21.15.59:0/187123338 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35008c190 0x7fa350152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 >> 172.21.15.59:0/187123338 conn(0x7fa3500b5610 msgr2=0x7fa3500c8010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 shutdown_connections 2024-06-29T14:37:15.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.249+0000 7fa355646640 1 -- 172.21.15.59:0/187123338 wait complete. 2024-06-29T14:37:15.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:37:15.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.388+0000 7f47906df640 1 -- 172.21.15.59:0/2489506109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 msgr2=0x7f47880b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.388+0000 7f47906df640 1 --2- 172.21.15.59:0/2489506109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 0x7f47880b3b50 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4788074b10 tx=0x7f477c02a530 comp rx=0 tx=0).stop 2024-06-29T14:37:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.388+0000 7f47906df640 1 -- 172.21.15.59:0/2489506109 shutdown_connections 2024-06-29T14:37:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.388+0000 7f47906df640 1 --2- 172.21.15.59:0/2489506109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 0x7f47880b3b50 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.388+0000 7f47906df640 1 --2- 172.21.15.59:0/2489506109 >> v2:172.21.15.153:3300/0 conn(0x7f47881458a0 0x7f47880cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.388+0000 7f47906df640 1 -- 172.21.15.59:0/2489506109 >> 172.21.15.59:0/2489506109 conn(0x7f478806c950 msgr2=0x7f4788073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.388+0000 7f47906df640 1 -- 172.21.15.59:0/2489506109 shutdown_connections 2024-06-29T14:37:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.388+0000 7f47906df640 1 -- 172.21.15.59:0/2489506109 wait complete. 2024-06-29T14:37:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.390+0000 7f47906df640 1 Processor -- start 2024-06-29T14:37:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.390+0000 7f47906df640 1 -- start start 2024-06-29T14:37:15.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f47906df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 0x7f47880aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f47906df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f478815c2f0 0x7f4788136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f47906df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4788133510 0x7f4788133920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f47906df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4788160020 con 0x7f47880b3740 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f47906df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f478815cbb0 con 0x7f4788133510 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f47906df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f47881601a0 con 0x7f478815c2f0 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f478e47b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 0x7f47880aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f478e47b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 0x7f47880aaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36850/0 (socket says 172.21.15.59:36850) 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f478e47b640 1 -- 172.21.15.59:0/40746359 learned_addr learned my addr 172.21.15.59:0/40746359 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f478dc7a640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f478815c2f0 0x7f4788136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.391+0000 7f478ec7c640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4788133510 0x7f4788133920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f478e47b640 1 -- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f478815c2f0 msgr2=0x7f4788136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f478e47b640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f478815c2f0 0x7f4788136d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f478e47b640 1 -- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4788133510 msgr2=0x7f4788133920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f478e47b640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4788133510 0x7f4788133920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f478e47b640 1 -- 172.21.15.59:0/40746359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4788133e60 con 0x7f47880b3740 2024-06-29T14:37:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f478e47b640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 0x7f47880aaf00 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f477800ea50 tx=0x7f477800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f4786ffd640 1 -- 172.21.15.59:0/40746359 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477800cdc0 con 0x7f47880b3740 2024-06-29T14:37:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f4786ffd640 1 -- 172.21.15.59:0/40746359 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4778011560 con 0x7f47880b3740 2024-06-29T14:37:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47880ab8e0 con 0x7f47880b3740 2024-06-29T14:37:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f4786ffd640 1 -- 172.21.15.59:0/40746359 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4778011990 con 0x7f47880b3740 2024-06-29T14:37:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.392+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47880abea0 con 0x7f47880b3740 2024-06-29T14:37:15.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.394+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47881606c0 con 0x7f47880b3740 2024-06-29T14:37:15.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.394+0000 7f4786ffd640 1 -- 172.21.15.59:0/40746359 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f477801a400 con 0x7f47880b3740 2024-06-29T14:37:15.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.395+0000 7f4786ffd640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f47740763f0 0x7f4774078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.395+0000 7f4786ffd640 1 -- 172.21.15.59:0/40746359 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7f47780985c0 con 0x7f47880b3740 2024-06-29T14:37:15.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.395+0000 7f478dc7a640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f47740763f0 0x7f4774078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.396+0000 7f478dc7a640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f47740763f0 0x7f4774078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f477c034000 tx=0x7f477c004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:15.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.400+0000 7f4786ffd640 1 -- 172.21.15.59:0/40746359 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4788133e60 con 0x7f47880b3740 2024-06-29T14:37:15.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.593+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f478815fe00 con 0x7f47880b3740 2024-06-29T14:37:15.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.593+0000 7f4786ffd640 1 -- 172.21.15.59:0/40746359 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v35) ==== 59+0+1998 (secure 0 0 0) 0x7f4778016030 con 0x7f47880b3740 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T14:37:15.592 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:37:15.593 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T14:37:15.594 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T14:37:15.594 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T14:37:15.594 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:37:15.594 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:37:15.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f47740763f0 msgr2=0x7f4774078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f47740763f0 0x7f4774078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f477c034000 tx=0x7f477c004730 comp rx=0 tx=0).stop 2024-06-29T14:37:15.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 msgr2=0x7f47880aaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 0x7f47880aaf00 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f477800ea50 tx=0x7f477800ef10 comp rx=0 tx=0).stop 2024-06-29T14:37:15.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 shutdown_connections 2024-06-29T14:37:15.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f47740763f0 0x7f4774078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4788133510 0x7f4788133920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f478815c2f0 0x7f4788136d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 --2- 172.21.15.59:0/40746359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880b3740 0x7f47880aaf00 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 >> 172.21.15.59:0/40746359 conn(0x7f478806c950 msgr2=0x7f47880d2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 shutdown_connections 2024-06-29T14:37:15.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.595+0000 7f47906df640 1 -- 172.21.15.59:0/40746359 wait complete. 2024-06-29T14:37:15.604 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:37:15.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.726+0000 7fd450eb9640 1 -- 172.21.15.59:0/2202605612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c162190 msgr2=0x7fd44c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.726+0000 7fd450eb9640 1 --2- 172.21.15.59:0/2202605612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c162190 0x7fd44c0ce550 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd440009830 tx=0x7fd44002a470 comp rx=0 tx=0).stop 2024-06-29T14:37:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.726+0000 7fd450eb9640 1 -- 172.21.15.59:0/2202605612 shutdown_connections 2024-06-29T14:37:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.726+0000 7fd450eb9640 1 --2- 172.21.15.59:0/2202605612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd44c15e0a0 0x7fd44c0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.726+0000 7fd450eb9640 1 --2- 172.21.15.59:0/2202605612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c162190 0x7fd44c0ce550 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.726+0000 7fd450eb9640 1 -- 172.21.15.59:0/2202605612 >> 172.21.15.59:0/2202605612 conn(0x7fd44c0d45b0 msgr2=0x7fd44c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.726+0000 7fd450eb9640 1 -- 172.21.15.59:0/2202605612 shutdown_connections 2024-06-29T14:37:15.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.726+0000 7fd450eb9640 1 -- 172.21.15.59:0/2202605612 wait complete. 2024-06-29T14:37:15.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.728+0000 7fd450eb9640 1 Processor -- start 2024-06-29T14:37:15.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.728+0000 7fd450eb9640 1 -- start start 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.729+0000 7fd450eb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd44c15e0a0 0x7fd44c15e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.729+0000 7fd450eb9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c162770 0x7fd44c151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.729+0000 7fd450eb9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c0b54b0 0x7fd44c0b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.729+0000 7fd450eb9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c0b42c0 con 0x7fd44c0b54b0 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.729+0000 7fd450eb9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c0b4140 con 0x7fd44c162770 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.729+0000 7fd450eb9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd44c0b4440 con 0x7fd44c15e0a0 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c0b54b0 0x7fd44c0b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c0b54b0 0x7fd44c0b58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36870/0 (socket says 172.21.15.59:36870) 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 -- 172.21.15.59:0/1219330755 learned_addr learned my addr 172.21.15.59:0/1219330755 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44a59c640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd44c15e0a0 0x7fd44c15e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd449d9b640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c162770 0x7fd44c151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 -- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd44c15e0a0 msgr2=0x7fd44c15e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd44c15e0a0 0x7fd44c15e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 -- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c162770 msgr2=0x7fd44c151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c162770 0x7fd44c151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 -- 172.21.15.59:0/1219330755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c0b5e00 con 0x7fd44c0b54b0 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd44ad9d640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c0b54b0 0x7fd44c0b58c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd44400d5f0 tx=0x7fd44400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd43affd640 1 -- 172.21.15.59:0/1219330755 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd444017070 con 0x7fd44c0b54b0 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd43affd640 1 -- 172.21.15.59:0/1219330755 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd444005330 con 0x7fd44c0b54b0 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44c15ea80 con 0x7fd44c0b54b0 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd43affd640 1 -- 172.21.15.59:0/1219330755 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4440047a0 con 0x7fd44c0b54b0 2024-06-29T14:37:15.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.730+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44c15ef90 con 0x7fd44c0b54b0 2024-06-29T14:37:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.731+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd44c0b49d0 con 0x7fd44c0b54b0 2024-06-29T14:37:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.732+0000 7fd43affd640 1 -- 172.21.15.59:0/1219330755 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fd444004940 con 0x7fd44c0b54b0 2024-06-29T14:37:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.733+0000 7fd43affd640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd42c0763f0 0x7fd42c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.733+0000 7fd43affd640 1 -- 172.21.15.59:0/1219330755 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7fd444097b90 con 0x7fd44c0b54b0 2024-06-29T14:37:15.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.733+0000 7fd44a59c640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd42c0763f0 0x7fd42c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.734+0000 7fd44a59c640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd42c0763f0 0x7fd42c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd43c002020 tx=0x7fd43c009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:15.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.737+0000 7fd43affd640 1 -- 172.21.15.59:0/1219330755 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd44c0b5e00 con 0x7fd44c0b54b0 2024-06-29T14:37:15.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.928+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd44c15ed20 con 0x7fd44c0b54b0 2024-06-29T14:37:15.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.931+0000 7fd43affd640 1 -- 172.21.15.59:0/1219330755 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7fd4440610f0 con 0x7fd44c0b54b0 2024-06-29T14:37:15.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.933+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd42c0763f0 msgr2=0x7fd42c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.933+0000 7fd450eb9640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd42c0763f0 0x7fd42c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd43c002020 tx=0x7fd43c009040 comp rx=0 tx=0).stop 2024-06-29T14:37:15.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.933+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c0b54b0 msgr2=0x7fd44c0b58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.933+0000 7fd450eb9640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c0b54b0 0x7fd44c0b58c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd44400d5f0 tx=0x7fd44400dab0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.934+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 shutdown_connections 2024-06-29T14:37:15.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.934+0000 7fd450eb9640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd42c0763f0 0x7fd42c078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.934+0000 7fd450eb9640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c0b54b0 0x7fd44c0b58c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.934+0000 7fd450eb9640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd44c162770 0x7fd44c151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.934+0000 7fd450eb9640 1 --2- 172.21.15.59:0/1219330755 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd44c15e0a0 0x7fd44c15e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.934+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 >> 172.21.15.59:0/1219330755 conn(0x7fd44c0d45b0 msgr2=0x7fd44c0b7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.934+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 shutdown_connections 2024-06-29T14:37:15.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:15.934+0000 7fd450eb9640 1 -- 172.21.15.59:0/1219330755 wait complete. 2024-06-29T14:37:15.943 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:37:15.943 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:15.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:16.030 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:37:16.030 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:16.030 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:16.046 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:16.046 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:37:16.046 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:16.064 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:16.064 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:16.064 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:16.103 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:16.103 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:37:16.103 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:16.121 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:16.121 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:37:16.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.263+0000 7f269a7a1640 1 -- 172.21.15.59:0/1577592736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2694143840 msgr2=0x7f269406e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.263+0000 7f269a7a1640 1 --2- 172.21.15.59:0/1577592736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2694143840 0x7f269406e4d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f267c005fd0 tx=0x7f267c02a540 comp rx=0 tx=0).stop 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.263+0000 7f269a7a1640 1 -- 172.21.15.59:0/1577592736 shutdown_connections 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.263+0000 7f269a7a1640 1 --2- 172.21.15.59:0/1577592736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f269406ea10 0x7f26940cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.263+0000 7f269a7a1640 1 --2- 172.21.15.59:0/1577592736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2694143840 0x7f269406e4d0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.263+0000 7f269a7a1640 1 -- 172.21.15.59:0/1577592736 >> 172.21.15.59:0/1577592736 conn(0x7f26940b2ab0 msgr2=0x7f26940b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.263+0000 7f269a7a1640 1 -- 172.21.15.59:0/1577592736 shutdown_connections 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.263+0000 7f269a7a1640 1 -- 172.21.15.59:0/1577592736 wait complete. 2024-06-29T14:37:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.265+0000 7f269a7a1640 1 Processor -- start 2024-06-29T14:37:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.265+0000 7f269a7a1640 1 -- start start 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.266+0000 7f269a7a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f269406ea10 0x7f26940af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.266+0000 7f269a7a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26940d0130 0x7f269413d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.266+0000 7f269a7a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2694143840 0x7f26940ac760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.266+0000 7f269a7a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f26940cf650 con 0x7f269406ea10 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.266+0000 7f269a7a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26940cf4d0 con 0x7f2694143840 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.266+0000 7f269a7a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f26940cf7d0 con 0x7f26940d0130 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f269406ea10 0x7f26940af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f269406ea10 0x7f26940af7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36894/0 (socket says 172.21.15.59:36894) 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 -- 172.21.15.59:0/2767218957 learned_addr learned my addr 172.21.15.59:0/2767218957 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2698d3e640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2694143840 0x7f26940ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f268bfff640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26940d0130 0x7f269413d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 -- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26940d0130 msgr2=0x7f269413d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26940d0130 0x7f269413d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 -- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2694143840 msgr2=0x7f26940ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2694143840 0x7f26940ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 -- 172.21.15.59:0/2767218957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26940ab480 con 0x7f269406ea10 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f2693fff640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f269406ea10 0x7f26940af7a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f268400c3d0 tx=0x7f268400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f26917fa640 1 -- 172.21.15.59:0/2767218957 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684017070 con 0x7f269406ea10 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f26917fa640 1 -- 172.21.15.59:0/2767218957 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26840053c0 con 0x7f269406ea10 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26940ab760 con 0x7f269406ea10 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f26917fa640 1 -- 172.21.15.59:0/2767218957 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684003850 con 0x7f269406ea10 2024-06-29T14:37:16.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.267+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26941420f0 con 0x7f269406ea10 2024-06-29T14:37:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.268+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26940cfc20 con 0x7f269406ea10 2024-06-29T14:37:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.269+0000 7f26917fa640 1 -- 172.21.15.59:0/2767218957 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f268401a070 con 0x7f269406ea10 2024-06-29T14:37:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.270+0000 7f26917fa640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f26740763f0 0x7f2674078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.270+0000 7f26917fa640 1 -- 172.21.15.59:0/2767218957 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7f2684096de0 con 0x7f269406ea10 2024-06-29T14:37:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.270+0000 7f268bfff640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f26740763f0 0x7f2674078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.270+0000 7f268bfff640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f26740763f0 0x7f2674078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f267c033000 tx=0x7f267c004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:16.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.274+0000 7f26917fa640 1 -- 172.21.15.59:0/2767218957 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f268405fe20 con 0x7f269406ea10 2024-06-29T14:37:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.504+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2694070090 con 0x7f269406ea10 2024-06-29T14:37:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.504+0000 7f26917fa640 1 -- 172.21.15.59:0/2767218957 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+3911 (secure 0 0 0) 0x7f2684064120 con 0x7f269406ea10 2024-06-29T14:37:16.503 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:16.503 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":31,"btime":"2024-06-29T14:37:10:228347+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/484297155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":484297155},{"type":"v1","addr":"172.21.15.153:6835","nonce":484297155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2309835731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":2309835731},{"type":"v1","addr":"172.21.15.153:6837","nonce":2309835731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":6330,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/413487166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":413487166},{"type":"v1","addr":"172.21.15.59:6834","nonce":413487166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":6762,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1023207399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1023207399},{"type":"v1","addr":"172.21.15.59:6835","nonce":1023207399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[]} 2024-06-29T14:37:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 31 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f26740763f0 msgr2=0x7f2674078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f26740763f0 0x7f2674078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f267c033000 tx=0x7f267c004730 comp rx=0 tx=0).stop 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f269406ea10 msgr2=0x7f26940af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f269406ea10 0x7f26940af7a0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f268400c3d0 tx=0x7f268400c890 comp rx=0 tx=0).stop 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 shutdown_connections 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f26740763f0 0x7f2674078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2694143840 0x7f26940ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26940d0130 0x7f269413d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 --2- 172.21.15.59:0/2767218957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f269406ea10 0x7f26940af7a0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 >> 172.21.15.59:0/2767218957 conn(0x7f26940b2ab0 msgr2=0x7f26940b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 shutdown_connections 2024-06-29T14:37:16.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.506+0000 7f269a7a1640 1 -- 172.21.15.59:0/2767218957 wait complete. 2024-06-29T14:37:16.516 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:37:16.516 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:37:16.516 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:37:16.516 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:16.516 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:37:16.517 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:37:16.517 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:37:16.517 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:16.517 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:37:16.517 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:37:16.517 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:37:16.517 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:16.518 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:37:16.518 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:37:16.518 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:37:16.518 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:16.528 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:37:16.529 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:37:16.530 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:37:16.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:37:16.531 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:37:16.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:37:16.532 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:37:16.573 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:37:16.573 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:16.574 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:37:16.574 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:37:16.575 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:16.575 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:16.575 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:37:16.575 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:16.576 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:37:16.662 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:37:16.663 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:37:16.665 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:37:16.666 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:37:16.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.702+0000 7f5ac877f640 1 -- 172.21.15.59:0/3112846093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac00b40a0 msgr2=0x7f5ac015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.702+0000 7f5ac877f640 1 --2- 172.21.15.59:0/3112846093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac00b40a0 0x7f5ac015e8d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5abc005fd0 tx=0x7f5abc02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.703+0000 7f5ac877f640 1 -- 172.21.15.59:0/3112846093 shutdown_connections 2024-06-29T14:37:16.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.703+0000 7f5ac877f640 1 --2- 172.21.15.59:0/3112846093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ac015ee10 0x7f5ac015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.703+0000 7f5ac877f640 1 --2- 172.21.15.59:0/3112846093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac00b40a0 0x7f5ac015e8d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.703+0000 7f5ac877f640 1 -- 172.21.15.59:0/3112846093 >> 172.21.15.59:0/3112846093 conn(0x7f5ac00d3ee0 msgr2=0x7f5ac00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:16.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.703+0000 7f5ac877f640 1 -- 172.21.15.59:0/3112846093 shutdown_connections 2024-06-29T14:37:16.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.703+0000 7f5ac877f640 1 -- 172.21.15.59:0/3112846093 wait complete. 2024-06-29T14:37:16.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.704+0000 7f5ac877f640 1 Processor -- start 2024-06-29T14:37:16.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.704+0000 7f5ac877f640 1 -- start start 2024-06-29T14:37:16.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac877f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ac00d23c0 0x7f5ac01f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac877f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ac015ee10 0x7f5ac01f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac877f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac01599e0 0x7f5ac0151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac877f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac0160280 con 0x7f5ac01599e0 2024-06-29T14:37:16.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac877f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac0160100 con 0x7f5ac015ee10 2024-06-29T14:37:16.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac877f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac0160400 con 0x7f5ac00d23c0 2024-06-29T14:37:16.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac01599e0 0x7f5ac0151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac01599e0 0x7f5ac0151490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36936/0 (socket says 172.21.15.59:36936) 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 -- 172.21.15.59:0/831700935 learned_addr learned my addr 172.21.15.59:0/831700935 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 -- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ac00d23c0 msgr2=0x7f5ac01f7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac5d1a640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ac015ee10 0x7f5ac01f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac651b640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ac00d23c0 0x7f5ac01f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ac00d23c0 0x7f5ac01f7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 -- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ac015ee10 msgr2=0x7f5ac01f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ac015ee10 0x7f5ac01f5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 -- 172.21.15.59:0/831700935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac01f5a90 con 0x7f5ac01599e0 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.706+0000 7f5ac6d1c640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac01599e0 0x7f5ac0151490 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ec20 tx=0x7f5ab800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.707+0000 7f5ab6ffd640 1 -- 172.21.15.59:0/831700935 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8003b70 con 0x7f5ac01599e0 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.707+0000 7f5ab6ffd640 1 -- 172.21.15.59:0/831700935 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ab8003d10 con 0x7f5ac01599e0 2024-06-29T14:37:16.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.707+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac0154410 con 0x7f5ac01599e0 2024-06-29T14:37:16.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.707+0000 7f5ab6ffd640 1 -- 172.21.15.59:0/831700935 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8004280 con 0x7f5ac01599e0 2024-06-29T14:37:16.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.707+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac01549d0 con 0x7f5ac01599e0 2024-06-29T14:37:16.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.708+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac015e8d0 con 0x7f5ac01599e0 2024-06-29T14:37:16.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.708+0000 7f5ab6ffd640 1 -- 172.21.15.59:0/831700935 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f5ab8004420 con 0x7f5ac01599e0 2024-06-29T14:37:16.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.709+0000 7f5ab6ffd640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5aa40763f0 0x7f5aa4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.709+0000 7f5ab6ffd640 1 -- 172.21.15.59:0/831700935 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6023+0+0 (secure 0 0 0) 0x7f5ab8098760 con 0x7f5ac01599e0 2024-06-29T14:37:16.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.709+0000 7f5ac651b640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5aa40763f0 0x7f5aa4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.710+0000 7f5ac651b640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5aa40763f0 0x7f5aa4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00d2e40 tx=0x7f5ab0009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:16.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.713+0000 7f5ab6ffd640 1 -- 172.21.15.59:0/831700935 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ab8061d60 con 0x7f5ac01599e0 2024-06-29T14:37:16.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:16.915+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5ac00d2950 con 0x7f5ac01599e0 2024-06-29T14:37:17.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.262+0000 7f5ab6ffd640 1 -- 172.21.15.59:0/831700935 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v82) ==== 97+0+0 (secure 0 0 0) 0x7f5ab8061660 con 0x7f5ac01599e0 2024-06-29T14:37:17.261 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T14:37:17.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5aa40763f0 msgr2=0x7f5aa4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5aa40763f0 0x7f5aa4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00d2e40 tx=0x7f5ab0009040 comp rx=0 tx=0).stop 2024-06-29T14:37:17.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac01599e0 msgr2=0x7f5ac0151490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac01599e0 0x7f5ac0151490 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ec20 tx=0x7f5ab800c6a0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 shutdown_connections 2024-06-29T14:37:17.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5aa40763f0 0x7f5aa4078900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ac01599e0 0x7f5ac0151490 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ac015ee10 0x7f5ac01f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 --2- 172.21.15.59:0/831700935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ac00d23c0 0x7f5ac01f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 >> 172.21.15.59:0/831700935 conn(0x7f5ac00d3ee0 msgr2=0x7f5ac006cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:17.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 shutdown_connections 2024-06-29T14:37:17.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.266+0000 7f5ac877f640 1 -- 172.21.15.59:0/831700935 wait complete. 2024-06-29T14:37:17.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:37:17.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.398+0000 7efd75503640 1 -- 172.21.15.59:0/1485758247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 msgr2=0x7efd700b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.398+0000 7efd75503640 1 --2- 172.21.15.59:0/1485758247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 0x7efd700b3990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7efd60009830 tx=0x7efd6002a510 comp rx=0 tx=0).stop 2024-06-29T14:37:17.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.398+0000 7efd75503640 1 -- 172.21.15.59:0/1485758247 shutdown_connections 2024-06-29T14:37:17.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.398+0000 7efd75503640 1 --2- 172.21.15.59:0/1485758247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd700b3ed0 0x7efd7015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.398+0000 7efd75503640 1 --2- 172.21.15.59:0/1485758247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 0x7efd700b3990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.398+0000 7efd75503640 1 -- 172.21.15.59:0/1485758247 >> 172.21.15.59:0/1485758247 conn(0x7efd700d3ee0 msgr2=0x7efd700d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:17.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.398+0000 7efd75503640 1 -- 172.21.15.59:0/1485758247 shutdown_connections 2024-06-29T14:37:17.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.398+0000 7efd75503640 1 -- 172.21.15.59:0/1485758247 wait complete. 2024-06-29T14:37:17.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.399+0000 7efd75503640 1 Processor -- start 2024-06-29T14:37:17.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.399+0000 7efd75503640 1 -- start start 2024-06-29T14:37:17.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd75503640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd700b3ed0 0x7efd700c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd75503640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 0x7efd7015c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd75503640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7015ee10 0x7efd7015cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd75503640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efd70160240 con 0x7efd700d23c0 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd75503640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd701600c0 con 0x7efd700b3ed0 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd75503640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efd701603c0 con 0x7efd7015ee10 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 0x7efd7015c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 0x7efd7015c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36964/0 (socket says 172.21.15.59:36964) 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 -- 172.21.15.59:0/3057014816 learned_addr learned my addr 172.21.15.59:0/3057014816 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd6effd640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd700b3ed0 0x7efd700c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd6f7fe640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7015ee10 0x7efd7015cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 -- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7015ee10 msgr2=0x7efd7015cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7015ee10 0x7efd7015cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 -- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd700b3ed0 msgr2=0x7efd700c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd700b3ed0 0x7efd700c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 -- 172.21.15.59:0/3057014816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd70146740 con 0x7efd700d23c0 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd6effd640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd700b3ed0 0x7efd700c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.401+0000 7efd67fff640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 0x7efd7015c880 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7efd5800e790 tx=0x7efd5800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:17.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.402+0000 7efd677fe640 1 -- 172.21.15.59:0/3057014816 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5800eea0 con 0x7efd700d23c0 2024-06-29T14:37:17.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.402+0000 7efd677fe640 1 -- 172.21.15.59:0/3057014816 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd5800cdb0 con 0x7efd700d23c0 2024-06-29T14:37:17.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.402+0000 7efd677fe640 1 -- 172.21.15.59:0/3057014816 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58010690 con 0x7efd700d23c0 2024-06-29T14:37:17.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.402+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd7015a960 con 0x7efd700d23c0 2024-06-29T14:37:17.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.402+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd7015af00 con 0x7efd700d23c0 2024-06-29T14:37:17.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.403+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd7015a210 con 0x7efd700d23c0 2024-06-29T14:37:17.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.403+0000 7efd677fe640 1 -- 172.21.15.59:0/3057014816 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7efd58010830 con 0x7efd700d23c0 2024-06-29T14:37:17.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.404+0000 7efd677fe640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efd500763f0 0x7efd50078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.404+0000 7efd677fe640 1 -- 172.21.15.59:0/3057014816 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5249+0+0 (secure 0 0 0) 0x7efd580981e0 con 0x7efd700d23c0 2024-06-29T14:37:17.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.404+0000 7efd6effd640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efd500763f0 0x7efd50078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.405+0000 7efd6effd640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efd500763f0 0x7efd50078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efd60034000 tx=0x7efd60004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:17.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.408+0000 7efd677fe640 1 -- 172.21.15.59:0/3057014816 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd58061ae0 con 0x7efd700d23c0 2024-06-29T14:37:17.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.671+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7efd7015e8d0 con 0x7efd700d23c0 2024-06-29T14:37:17.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.672+0000 7efd677fe640 1 -- 172.21.15.59:0/3057014816 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5304 (secure 0 0 0) 0x7efd580613e0 con 0x7efd700d23c0 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:17.671 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:17.672 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:17.672 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.672 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.672 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.672 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.672 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T14:37:17.672 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T14:37:17.672 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.673 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:17.673 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:17.673 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:17.673 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.673 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.673 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.673 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.673 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.674 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T14:37:17.675 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.676 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.677 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.678 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.679 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.680 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.681 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:37:17.682 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efd500763f0 msgr2=0x7efd50078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efd500763f0 0x7efd50078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efd60034000 tx=0x7efd60004730 comp rx=0 tx=0).stop 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 msgr2=0x7efd7015c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 0x7efd7015c880 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7efd5800e790 tx=0x7efd5800ec50 comp rx=0 tx=0).stop 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 shutdown_connections 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7efd500763f0 0x7efd50078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7015ee10 0x7efd7015cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd700d23c0 0x7efd7015c880 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 --2- 172.21.15.59:0/3057014816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd700b3ed0 0x7efd700c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 >> 172.21.15.59:0/3057014816 conn(0x7efd700d3ee0 msgr2=0x7efd701571f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 shutdown_connections 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.674+0000 7efd75503640 1 -- 172.21.15.59:0/3057014816 wait complete. 2024-06-29T14:37:17.690 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.810+0000 7f8b695d2640 1 -- 172.21.15.59:0/3887402996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b640ce950 msgr2=0x7f8b641622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.810+0000 7f8b695d2640 1 --2- 172.21.15.59:0/3887402996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b640ce950 0x7f8b641622c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8b58005fd0 tx=0x7f8b5802a4f0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.810+0000 7f8b695d2640 1 -- 172.21.15.59:0/3887402996 shutdown_connections 2024-06-29T14:37:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.810+0000 7f8b695d2640 1 --2- 172.21.15.59:0/3887402996 >> v2:172.21.15.153:3300/0 conn(0x7f8b64162800 0x7f8b64162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.810+0000 7f8b695d2640 1 --2- 172.21.15.59:0/3887402996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b640ce950 0x7f8b641622c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.810+0000 7f8b695d2640 1 -- 172.21.15.59:0/3887402996 >> 172.21.15.59:0/3887402996 conn(0x7f8b640b5610 msgr2=0x7f8b640b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.810+0000 7f8b695d2640 1 -- 172.21.15.59:0/3887402996 shutdown_connections 2024-06-29T14:37:17.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.810+0000 7f8b695d2640 1 -- 172.21.15.59:0/3887402996 wait complete. 2024-06-29T14:37:17.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.812+0000 7f8b695d2640 1 Processor -- start 2024-06-29T14:37:17.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.812+0000 7f8b695d2640 1 -- start start 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b695d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6408c190 0x7f8b6415c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b695d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b640ce950 0x7f8b641f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b695d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b64162800 0x7f8b640cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b695d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8b640b7f60 con 0x7f8b6408c190 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b695d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b640b7de0 con 0x7f8b640ce950 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b695d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8b640b80e0 con 0x7f8b64162800 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6408c190 0x7f8b6415c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6408c190 0x7f8b6415c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36984/0 (socket says 172.21.15.59:36984) 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 -- 172.21.15.59:0/4258157120 learned_addr learned my addr 172.21.15.59:0/4258157120 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b627fc640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b640ce950 0x7f8b641f4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b637fe640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b64162800 0x7f8b640cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 -- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b64162800 msgr2=0x7f8b640cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b64162800 0x7f8b640cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 -- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b640ce950 msgr2=0x7f8b641f4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b640ce950 0x7f8b641f4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 -- 172.21.15.59:0/4258157120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b640cac70 con 0x7f8b6408c190 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b62ffd640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6408c190 0x7f8b6415c040 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400b410 tx=0x7f8b5400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.814+0000 7f8b4b7fe640 1 -- 172.21.15.59:0/4258157120 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b54015020 con 0x7f8b6408c190 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.815+0000 7f8b4b7fe640 1 -- 172.21.15.59:0/4258157120 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b54011920 con 0x7f8b6408c190 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.815+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b640caf50 con 0x7f8b6408c190 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.815+0000 7f8b4b7fe640 1 -- 172.21.15.59:0/4258157120 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b54011d70 con 0x7f8b6408c190 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.815+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b640cb510 con 0x7f8b6408c190 2024-06-29T14:37:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.816+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b641622c0 con 0x7f8b6408c190 2024-06-29T14:37:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.817+0000 7f8b4b7fe640 1 -- 172.21.15.59:0/4258157120 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f8b54018020 con 0x7f8b6408c190 2024-06-29T14:37:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.817+0000 7f8b4b7fe640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8b500763f0 0x7f8b50078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.817+0000 7f8b4b7fe640 1 -- 172.21.15.59:0/4258157120 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5249+0+0 (secure 0 0 0) 0x7f8b54097b90 con 0x7f8b6408c190 2024-06-29T14:37:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.817+0000 7f8b627fc640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8b500763f0 0x7f8b50078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.818+0000 7f8b627fc640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8b500763f0 0x7f8b50078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b58033000 tx=0x7f8b58004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:17.821+0000 7f8b4b7fe640 1 -- 172.21.15.59:0/4258157120 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8b5401f020 con 0x7f8b6408c190 2024-06-29T14:37:18.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.013+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b640ce030 con 0x7f8b6408c190 2024-06-29T14:37:18.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.014+0000 7f8b4b7fe640 1 -- 172.21.15.59:0/4258157120 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+13176 (secure 0 0 0) 0x7f8b540615f0 con 0x7f8b6408c190 2024-06-29T14:37:18.012 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:18.012 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":84,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:17.271629+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:18.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8b500763f0 msgr2=0x7f8b50078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8b500763f0 0x7f8b50078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b58033000 tx=0x7f8b58004730 comp rx=0 tx=0).stop 2024-06-29T14:37:18.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6408c190 msgr2=0x7f8b6415c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6408c190 0x7f8b6415c040 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400b410 tx=0x7f8b5400b8d0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 shutdown_connections 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8b500763f0 0x7f8b50078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b64162800 0x7f8b640cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b640ce950 0x7f8b641f4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 --2- 172.21.15.59:0/4258157120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6408c190 0x7f8b6415c040 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 >> 172.21.15.59:0/4258157120 conn(0x7f8b640b5610 msgr2=0x7f8b6406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 shutdown_connections 2024-06-29T14:37:18.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.016+0000 7f8b695d2640 1 -- 172.21.15.59:0/4258157120 wait complete. 2024-06-29T14:37:18.026 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:18.026 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.149+0000 7ff41a4ac640 1 -- 172.21.15.59:0/964988412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4140ce950 msgr2=0x7ff4141622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.149+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/964988412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4140ce950 0x7ff4141622c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009900 tx=0x7ff3fc02a610 comp rx=0 tx=0).stop 2024-06-29T14:37:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.149+0000 7ff41a4ac640 1 -- 172.21.15.59:0/964988412 shutdown_connections 2024-06-29T14:37:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.149+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/964988412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff414162800 0x7ff414162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.149+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/964988412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4140ce950 0x7ff4141622c0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.149+0000 7ff41a4ac640 1 -- 172.21.15.59:0/964988412 >> 172.21.15.59:0/964988412 conn(0x7ff4140b5610 msgr2=0x7ff4140b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.149+0000 7ff41a4ac640 1 -- 172.21.15.59:0/964988412 shutdown_connections 2024-06-29T14:37:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.149+0000 7ff41a4ac640 1 -- 172.21.15.59:0/964988412 wait complete. 2024-06-29T14:37:18.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.150+0000 7ff41a4ac640 1 Processor -- start 2024-06-29T14:37:18.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.151+0000 7ff41a4ac640 1 -- start start 2024-06-29T14:37:18.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff41a4ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff41408c190 0x7ff4141f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff41a4ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4140ce950 0x7ff414160e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff41a4ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff414162800 0x7ff41415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff41a4ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff4140b7f60 con 0x7ff41408c190 2024-06-29T14:37:18.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff41a4ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4140b7de0 con 0x7ff4140ce950 2024-06-29T14:37:18.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff41a4ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff4140b80e0 con 0x7ff414162800 2024-06-29T14:37:18.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff41408c190 0x7ff4141f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff41408c190 0x7ff4141f48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37000/0 (socket says 172.21.15.59:37000) 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 -- 172.21.15.59:0/302503752 learned_addr learned my addr 172.21.15.59:0/302503752 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff40bfff640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4140ce950 0x7ff414160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 -- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff414162800 msgr2=0x7ff41415ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff418a49640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff414162800 0x7ff41415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff414162800 0x7ff41415ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 -- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4140ce950 msgr2=0x7ff414160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4140ce950 0x7ff414160e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 -- 172.21.15.59:0/302503752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff414154250 con 0x7ff41408c190 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff40bfff640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4140ce950 0x7ff414160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff418a49640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff414162800 0x7ff41415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.152+0000 7ff413fff640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff41408c190 0x7ff4141f48f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff40400c3a0 tx=0x7ff40400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.153+0000 7ff4117fa640 1 -- 172.21.15.59:0/302503752 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4040052d0 con 0x7ff41408c190 2024-06-29T14:37:18.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.153+0000 7ff4117fa640 1 -- 172.21.15.59:0/302503752 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff404007020 con 0x7ff41408c190 2024-06-29T14:37:18.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.153+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff414154530 con 0x7ff41408c190 2024-06-29T14:37:18.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.153+0000 7ff4117fa640 1 -- 172.21.15.59:0/302503752 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff404007470 con 0x7ff41408c190 2024-06-29T14:37:18.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.153+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff414158370 con 0x7ff41408c190 2024-06-29T14:37:18.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.154+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff414154770 con 0x7ff41408c190 2024-06-29T14:37:18.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.154+0000 7ff4117fa640 1 -- 172.21.15.59:0/302503752 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7ff4040067d0 con 0x7ff41408c190 2024-06-29T14:37:18.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.155+0000 7ff4117fa640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff3f40763f0 0x7ff3f4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.155+0000 7ff4117fa640 1 -- 172.21.15.59:0/302503752 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5249+0+0 (secure 0 0 0) 0x7ff404096cc0 con 0x7ff41408c190 2024-06-29T14:37:18.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.155+0000 7ff40bfff640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff3f40763f0 0x7ff3f4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.156+0000 7ff40bfff640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff3f40763f0 0x7ff3f4078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc033000 tx=0x7ff3fc004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.160+0000 7ff4117fa640 1 -- 172.21.15.59:0/302503752 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff414154770 con 0x7ff41408c190 2024-06-29T14:37:18.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.351+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4141622c0 con 0x7ff41408c190 2024-06-29T14:37:18.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.351+0000 7ff4117fa640 1 -- 172.21.15.59:0/302503752 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7ff404060640 con 0x7ff41408c190 2024-06-29T14:37:18.350 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:37:18.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff3f40763f0 msgr2=0x7ff3f4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff3f40763f0 0x7ff3f4078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc033000 tx=0x7ff3fc004730 comp rx=0 tx=0).stop 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff41408c190 msgr2=0x7ff4141f48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff41408c190 0x7ff4141f48f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff40400c3a0 tx=0x7ff40400c860 comp rx=0 tx=0).stop 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 shutdown_connections 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff3f40763f0 0x7ff3f4078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff414162800 0x7ff41415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4140ce950 0x7ff414160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 --2- 172.21.15.59:0/302503752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff41408c190 0x7ff4141f48f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 >> 172.21.15.59:0/302503752 conn(0x7ff4140b5610 msgr2=0x7ff4140c9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 shutdown_connections 2024-06-29T14:37:18.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.353+0000 7ff41a4ac640 1 -- 172.21.15.59:0/302503752 wait complete. 2024-06-29T14:37:18.363 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:37:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.482+0000 7f74001f5640 1 -- 172.21.15.59:0/1866156057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 msgr2=0x7f73f80b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.482+0000 7f74001f5640 1 --2- 172.21.15.59:0/1866156057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 0x7f73f80b1d10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009830 tx=0x7f73f402a510 comp rx=0 tx=0).stop 2024-06-29T14:37:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.482+0000 7f74001f5640 1 -- 172.21.15.59:0/1866156057 shutdown_connections 2024-06-29T14:37:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.482+0000 7f74001f5640 1 --2- 172.21.15.59:0/1866156057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 0x7f73f80b1d10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.482+0000 7f74001f5640 1 -- 172.21.15.59:0/1866156057 >> 172.21.15.59:0/1866156057 conn(0x7f73f806c950 msgr2=0x7f73f8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.482+0000 7f74001f5640 1 -- 172.21.15.59:0/1866156057 shutdown_connections 2024-06-29T14:37:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.482+0000 7f74001f5640 1 -- 172.21.15.59:0/1866156057 wait complete. 2024-06-29T14:37:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.484+0000 7f74001f5640 1 Processor -- start 2024-06-29T14:37:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.484+0000 7f74001f5640 1 -- start start 2024-06-29T14:37:18.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f74001f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73f806f8a0 0x7f73f80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f74001f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 0x7f73f8143680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f74001f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73f8161b00 0x7f73f8167b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f74001f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f73f8169340 con 0x7f73f8147370 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f74001f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73f81691c0 con 0x7f73f806f8a0 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f74001f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f73f81694c0 con 0x7f73f8161b00 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f73fd790640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 0x7f73f8143680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f73fd790640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 0x7f73f8143680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37014/0 (socket says 172.21.15.59:37014) 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f73fd790640 1 -- 172.21.15.59:0/2490317754 learned_addr learned my addr 172.21.15.59:0/2490317754 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f73fdf91640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73f806f8a0 0x7f73f80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.485+0000 7f73fe792640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73f8161b00 0x7f73f8167b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73fd790640 1 -- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73f8161b00 msgr2=0x7f73f8167b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73fd790640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73f8161b00 0x7f73f8167b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73fd790640 1 -- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73f806f8a0 msgr2=0x7f73f80ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73fd790640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73f806f8a0 0x7f73f80ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73fd790640 1 -- 172.21.15.59:0/2490317754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f81425e0 con 0x7f73f8147370 2024-06-29T14:37:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73fd790640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 0x7f73f8143680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f73e800c3a0 tx=0x7f73e800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73ee7fc640 1 -- 172.21.15.59:0/2490317754 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8011790 con 0x7f73f8147370 2024-06-29T14:37:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73ee7fc640 1 -- 172.21.15.59:0/2490317754 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73e8017070 con 0x7f73f8147370 2024-06-29T14:37:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f81428c0 con 0x7f73f8147370 2024-06-29T14:37:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f73ee7fc640 1 -- 172.21.15.59:0/2490317754 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8011cc0 con 0x7f73f8147370 2024-06-29T14:37:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.486+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f8142e80 con 0x7f73f8147370 2024-06-29T14:37:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.487+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f80709a0 con 0x7f73f8147370 2024-06-29T14:37:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.488+0000 7f73ee7fc640 1 -- 172.21.15.59:0/2490317754 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f73e801c460 con 0x7f73f8147370 2024-06-29T14:37:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.488+0000 7f73ee7fc640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f73dc0763f0 0x7f73dc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.488+0000 7f73ee7fc640 1 -- 172.21.15.59:0/2490317754 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5249+0+0 (secure 0 0 0) 0x7f73e8097460 con 0x7f73f8147370 2024-06-29T14:37:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.489+0000 7f73fdf91640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f73dc0763f0 0x7f73dc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.489+0000 7f73fdf91640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f73dc0763f0 0x7f73dc078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f73f4034000 tx=0x7f73f4004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.493+0000 7f73ee7fc640 1 -- 172.21.15.59:0/2490317754 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73f80709a0 con 0x7f73f8147370 2024-06-29T14:37:18.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.685+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f73f806db80 con 0x7f73f8147370 2024-06-29T14:37:18.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.686+0000 7f73ee7fc640 1 -- 172.21.15.59:0/2490317754 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) ==== 84+0+65 (secure 0 0 0) 0x7f73f806db80 con 0x7f73f8147370 2024-06-29T14:37:18.685 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:18.685 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:37:18.685 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:18.685 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:37:18.686 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:37:18.686 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:18.686 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:37:18.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f73dc0763f0 msgr2=0x7f73dc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f73dc0763f0 0x7f73dc078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f73f4034000 tx=0x7f73f4004730 comp rx=0 tx=0).stop 2024-06-29T14:37:18.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 msgr2=0x7f73f8143680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 0x7f73f8143680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f73e800c3a0 tx=0x7f73e800c860 comp rx=0 tx=0).stop 2024-06-29T14:37:18.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 shutdown_connections 2024-06-29T14:37:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f73dc0763f0 0x7f73dc078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73f8161b00 0x7f73f8167b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73f8147370 0x7f73f8143680 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 --2- 172.21.15.59:0/2490317754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73f806f8a0 0x7f73f80ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 >> 172.21.15.59:0/2490317754 conn(0x7f73f806c950 msgr2=0x7f73f80b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 shutdown_connections 2024-06-29T14:37:18.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.688+0000 7f74001f5640 1 -- 172.21.15.59:0/2490317754 wait complete. 2024-06-29T14:37:18.698 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:37:18.698 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:37:18.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.822+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/1672983639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b6f70 msgr2=0x7f5c240b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.822+0000 7f5c2bdf5640 1 --2- 172.21.15.59:0/1672983639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b6f70 0x7f5c240b4070 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c005fd0 tx=0x7f5c1c02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.822+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/1672983639 shutdown_connections 2024-06-29T14:37:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.822+0000 7f5c2bdf5640 1 --2- 172.21.15.59:0/1672983639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b6f70 0x7f5c240b4070 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.822+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/1672983639 >> 172.21.15.59:0/1672983639 conn(0x7f5c240d0f90 msgr2=0x7f5c240d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.822+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/1672983639 shutdown_connections 2024-06-29T14:37:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.822+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/1672983639 wait complete. 2024-06-29T14:37:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.824+0000 7f5c2bdf5640 1 Processor -- start 2024-06-29T14:37:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.824+0000 7f5c2bdf5640 1 -- start start 2024-06-29T14:37:18.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c2bdf5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b2980 0x7f5c24154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c2bdf5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c240b6f70 0x7f5c2415c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c2bdf5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c240d1c30 0x7f5c2415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c2bdf5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5c240b4940 con 0x7f5c240b2980 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c2bdf5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c240b47c0 con 0x7f5c240d1c30 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c2bdf5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5c240b4ac0 con 0x7f5c240b6f70 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c29b91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b2980 0x7f5c24154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c29b91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b2980 0x7f5c24154410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37030/0 (socket says 172.21.15.59:37030) 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c29b91640 1 -- 172.21.15.59:0/562545402 learned_addr learned my addr 172.21.15.59:0/562545402 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c29390640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c240b6f70 0x7f5c2415c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c2a392640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c240d1c30 0x7f5c2415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c29b91640 1 -- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c240b6f70 msgr2=0x7f5c2415c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c29b91640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c240b6f70 0x7f5c2415c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c29b91640 1 -- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c240d1c30 msgr2=0x7f5c2415c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.825+0000 7f5c29b91640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c240d1c30 0x7f5c2415c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c29b91640 1 -- 172.21.15.59:0/562545402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c241e4160 con 0x7f5c240b2980 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c29390640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c240b6f70 0x7f5c2415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c2a392640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c240d1c30 0x7f5c2415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c29b91640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b2980 0x7f5c24154410 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000c780 tx=0x7f5c2000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c1a7fc640 1 -- 172.21.15.59:0/562545402 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20003b70 con 0x7f5c240b2980 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c1a7fc640 1 -- 172.21.15.59:0/562545402 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c20003d10 con 0x7f5c240b2980 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c241e4440 con 0x7f5c240b2980 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c1a7fc640 1 -- 172.21.15.59:0/562545402 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2000aab0 con 0x7f5c240b2980 2024-06-29T14:37:18.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.826+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c241e4a00 con 0x7f5c240b2980 2024-06-29T14:37:18.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.827+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c240b4070 con 0x7f5c240b2980 2024-06-29T14:37:18.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.828+0000 7f5c1a7fc640 1 -- 172.21.15.59:0/562545402 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f5c2000ac50 con 0x7f5c240b2980 2024-06-29T14:37:18.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.828+0000 7f5c1a7fc640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5c080763f0 0x7f5c08078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.828+0000 7f5c1a7fc640 1 -- 172.21.15.59:0/562545402 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5249+0+0 (secure 0 0 0) 0x7f5c20097750 con 0x7f5c240b2980 2024-06-29T14:37:18.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.829+0000 7f5c29390640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5c080763f0 0x7f5c08078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.829+0000 7f5c29390640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5c080763f0 0x7f5c08078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c241f8730 tx=0x7f5c14009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:18.833+0000 7f5c1a7fc640 1 -- 172.21.15.59:0/562545402 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5c20061050 con 0x7f5c240b2980 2024-06-29T14:37:19.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.022+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5c2415dae0 con 0x7f5c240b2980 2024-06-29T14:37:19.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.272+0000 7f5c1a7fc640 1 -- 172.21.15.59:0/562545402 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) ==== 138+0+0 (secure 0 0 0) 0x7f5c200690d0 con 0x7f5c240b2980 2024-06-29T14:37:19.271 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:37:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5c080763f0 msgr2=0x7f5c08078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5c080763f0 0x7f5c08078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c241f8730 tx=0x7f5c14009040 comp rx=0 tx=0).stop 2024-06-29T14:37:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b2980 msgr2=0x7f5c24154410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b2980 0x7f5c24154410 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000c780 tx=0x7f5c2000cc40 comp rx=0 tx=0).stop 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 shutdown_connections 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f5c080763f0 0x7f5c08078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c240d1c30 0x7f5c2415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5c240b6f70 0x7f5c2415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 --2- 172.21.15.59:0/562545402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c240b2980 0x7f5c24154410 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 >> 172.21.15.59:0/562545402 conn(0x7f5c240d0f90 msgr2=0x7f5c2406d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 shutdown_connections 2024-06-29T14:37:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.275+0000 7f5c2bdf5640 1 -- 172.21.15.59:0/562545402 wait complete. 2024-06-29T14:37:19.289 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:37:19.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.418+0000 7fc6cf69c640 1 -- 172.21.15.59:0/3609834516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 msgr2=0x7fc6c80b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.418+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/3609834516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 0x7fc6c80b4570 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80698a0 tx=0x7fc6b802a530 comp rx=0 tx=0).stop 2024-06-29T14:37:19.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.418+0000 7fc6cf69c640 1 -- 172.21.15.59:0/3609834516 shutdown_connections 2024-06-29T14:37:19.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.418+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/3609834516 >> v2:172.21.15.153:3300/0 conn(0x7fc6c80b4ab0 0x7fc6c80b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.418+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/3609834516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 0x7fc6c80b4570 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.418+0000 7fc6cf69c640 1 -- 172.21.15.59:0/3609834516 >> 172.21.15.59:0/3609834516 conn(0x7fc6c80d04e0 msgr2=0x7fc6c80d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.418+0000 7fc6cf69c640 1 -- 172.21.15.59:0/3609834516 shutdown_connections 2024-06-29T14:37:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.419+0000 7fc6cf69c640 1 -- 172.21.15.59:0/3609834516 wait complete. 2024-06-29T14:37:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.420+0000 7fc6cf69c640 1 Processor -- start 2024-06-29T14:37:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.420+0000 7fc6cf69c640 1 -- start start 2024-06-29T14:37:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cf69c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 0x7fc6c80b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cf69c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6c80b4ab0 0x7fc6c80b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cf69c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c80d3260 0x7fc6c8148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cf69c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c80b77b0 con 0x7fc6c80b2c40 2024-06-29T14:37:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cf69c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c808c380 con 0x7fc6c80d3260 2024-06-29T14:37:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cf69c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c80b7930 con 0x7fc6c80b4ab0 2024-06-29T14:37:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 0x7fc6c80b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 0x7fc6c80b1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37046/0 (socket says 172.21.15.59:37046) 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 -- 172.21.15.59:0/2670813757 learned_addr learned my addr 172.21.15.59:0/2670813757 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cdc39640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c80d3260 0x7fc6c8148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6ccc37640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6c80b4ab0 0x7fc6c80b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 -- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6c80b4ab0 msgr2=0x7fc6c80b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6c80b4ab0 0x7fc6c80b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 -- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c80d3260 msgr2=0x7fc6c8148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c80d3260 0x7fc6c8148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 -- 172.21.15.59:0/2670813757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c814a380 con 0x7fc6c80b2c40 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cd438640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 0x7fc6c80b1990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c3c0 tx=0x7fc6c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6bdffb640 1 -- 172.21.15.59:0/2670813757 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4017070 con 0x7fc6c80b2c40 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6bdffb640 1 -- 172.21.15.59:0/2670813757 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6c40035f0 con 0x7fc6c80b2c40 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c814a660 con 0x7fc6c80b2c40 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.422+0000 7fc6bdffb640 1 -- 172.21.15.59:0/2670813757 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4003a80 con 0x7fc6c80b2c40 2024-06-29T14:37:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.423+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c814ac20 con 0x7fc6c80b2c40 2024-06-29T14:37:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.424+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c80b4570 con 0x7fc6c80b2c40 2024-06-29T14:37:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.425+0000 7fc6bdffb640 1 -- 172.21.15.59:0/2670813757 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fc6c401a070 con 0x7fc6c80b2c40 2024-06-29T14:37:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.425+0000 7fc6bdffb640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc6ac0763f0 0x7fc6ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.425+0000 7fc6bdffb640 1 -- 172.21.15.59:0/2670813757 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5619+0+0 (secure 0 0 0) 0x7fc6c40970e0 con 0x7fc6c80b2c40 2024-06-29T14:37:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.426+0000 7fc6ccc37640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc6ac0763f0 0x7fc6ac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.426+0000 7fc6ccc37640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc6ac0763f0 0x7fc6ac078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8034000 tx=0x7fc6b8004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:19.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.429+0000 7fc6bdffb640 1 -- 172.21.15.59:0/2670813757 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc6c4021020 con 0x7fc6c80b2c40 2024-06-29T14:37:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:19.625+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc6c80b3ee0 con 0x7fc6c80b2c40 2024-06-29T14:37:20.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.275+0000 7fc6bdffb640 1 -- 172.21.15.59:0/2670813757 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) ==== 170+0+0 (secure 0 0 0) 0x7fc6c80b4ec0 con 0x7fc6c80b2c40 2024-06-29T14:37:20.274 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:37:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.277+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc6ac0763f0 msgr2=0x7fc6ac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.277+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc6ac0763f0 0x7fc6ac078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8034000 tx=0x7fc6b8004730 comp rx=0 tx=0).stop 2024-06-29T14:37:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.277+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 msgr2=0x7fc6c80b1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.277+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 0x7fc6c80b1990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c3c0 tx=0x7fc6c400c880 comp rx=0 tx=0).stop 2024-06-29T14:37:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.278+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 shutdown_connections 2024-06-29T14:37:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.278+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc6ac0763f0 0x7fc6ac078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.278+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6c80d3260 0x7fc6c8148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.278+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc6c80b4ab0 0x7fc6c80b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.278+0000 7fc6cf69c640 1 --2- 172.21.15.59:0/2670813757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6c80b2c40 0x7fc6c80b1990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.278+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 >> 172.21.15.59:0/2670813757 conn(0x7fc6c80d04e0 msgr2=0x7fc6c80b8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.278+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 shutdown_connections 2024-06-29T14:37:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.278+0000 7fc6cf69c640 1 -- 172.21.15.59:0/2670813757 wait complete. 2024-06-29T14:37:20.301 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:37:20.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.429+0000 7f0a657c7640 1 -- 172.21.15.59:0/2605869298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 msgr2=0x7f0a4c005280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.429+0000 7f0a657c7640 1 --2- 172.21.15.59:0/2605869298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 0x7f0a4c005280 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0a58001690 tx=0x7f0a4c005830 comp rx=0 tx=0).stop 2024-06-29T14:37:20.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.429+0000 7f0a5f7fe640 1 -- 172.21.15.59:0/2605869298 reap_dead start 2024-06-29T14:37:20.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.429+0000 7f0a657c7640 1 -- 172.21.15.59:0/2605869298 shutdown_connections 2024-06-29T14:37:20.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.429+0000 7f0a657c7640 1 -- 172.21.15.59:0/2605869298 >> 172.21.15.59:0/2605869298 conn(0x7f0a6006c950 msgr2=0x7f0a60073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:20.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.429+0000 7f0a657c7640 1 -- 172.21.15.59:0/2605869298 shutdown_connections 2024-06-29T14:37:20.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.429+0000 7f0a657c7640 1 -- 172.21.15.59:0/2605869298 wait complete. 2024-06-29T14:37:20.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.431+0000 7f0a657c7640 1 Processor -- start 2024-06-29T14:37:20.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.431+0000 7f0a657c7640 1 -- start start 2024-06-29T14:37:20.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a657c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 0x7f0a601f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:20.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a657c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6013cf40 0x7f0a600d1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a657c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a600d1e40 0x7f0a60139490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a657c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6015ca90 con 0x7f0a4c004db0 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a657c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6015c910 con 0x7f0a6013cf40 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a657c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0a6015cc10 con 0x7f0a600d1e40 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 0x7f0a601f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 0x7f0a601f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37074/0 (socket says 172.21.15.59:37074) 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 -- 172.21.15.59:0/2055961845 learned_addr learned my addr 172.21.15.59:0/2055961845 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5e7fc640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6013cf40 0x7f0a600d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:20.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5f7fe640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a600d1e40 0x7f0a60139490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 -- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a600d1e40 msgr2=0x7f0a60139490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a600d1e40 0x7f0a60139490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 -- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6013cf40 msgr2=0x7f0a600d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6013cf40 0x7f0a600d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 -- 172.21.15.59:0/2055961845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a600d2250 con 0x7f0a4c004db0 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a5effd640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 0x7f0a601f3090 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0a50006340 tx=0x7f0a50006800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a577fe640 1 -- 172.21.15.59:0/2055961845 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50015070 con 0x7f0a4c004db0 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a577fe640 1 -- 172.21.15.59:0/2055961845 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a5000fd40 con 0x7f0a4c004db0 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6013b0b0 con 0x7f0a4c004db0 2024-06-29T14:37:20.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.433+0000 7f0a577fe640 1 -- 172.21.15.59:0/2055961845 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50011690 con 0x7f0a4c004db0 2024-06-29T14:37:20.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.434+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a60138c90 con 0x7f0a4c004db0 2024-06-29T14:37:20.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.434+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a601458a0 con 0x7f0a4c004db0 2024-06-29T14:37:20.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.435+0000 7f0a577fe640 1 -- 172.21.15.59:0/2055961845 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f0a50011830 con 0x7f0a4c004db0 2024-06-29T14:37:20.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.436+0000 7f0a577fe640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f0a40076420 0x7f0a40078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:20.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.436+0000 7f0a577fe640 1 -- 172.21.15.59:0/2055961845 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6001+0+0 (secure 0 0 0) 0x7f0a50098c10 con 0x7f0a4c004db0 2024-06-29T14:37:20.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.436+0000 7f0a5e7fc640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f0a40076420 0x7f0a40078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:20.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.437+0000 7f0a5e7fc640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f0a40076420 0x7f0a40078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0a60071b00 tx=0x7f0a48005f90 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:20.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.441+0000 7f0a577fe640 1 -- 172.21.15.59:0/2055961845 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0a600d2250 con 0x7f0a4c004db0 2024-06-29T14:37:20.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:20.670+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0a6013e980 con 0x7f0a4c004db0 2024-06-29T14:37:21.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.305+0000 7f0a577fe640 1 -- 172.21.15.59:0/2055961845 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v34) ==== 173+0+0 (secure 0 0 0) 0x7f0a50062190 con 0x7f0a4c004db0 2024-06-29T14:37:21.304 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 9 and data pool 10 2024-06-29T14:37:21.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.313+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f0a40076420 msgr2=0x7f0a40078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.313+0000 7f0a657c7640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f0a40076420 0x7f0a40078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0a60071b00 tx=0x7f0a48005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:21.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.313+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 msgr2=0x7f0a601f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.313+0000 7f0a657c7640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 0x7f0a601f3090 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0a50006340 tx=0x7f0a50006800 comp rx=0 tx=0).stop 2024-06-29T14:37:21.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.314+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 shutdown_connections 2024-06-29T14:37:21.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.314+0000 7f0a657c7640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f0a40076420 0x7f0a40078930 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.314+0000 7f0a657c7640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a600d1e40 0x7f0a60139490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.314+0000 7f0a657c7640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a6013cf40 0x7f0a600d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.314+0000 7f0a657c7640 1 --2- 172.21.15.59:0/2055961845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a4c004db0 0x7f0a601f3090 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.314+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 >> 172.21.15.59:0/2055961845 conn(0x7f0a6006c950 msgr2=0x7f0a60070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.316+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 shutdown_connections 2024-06-29T14:37:21.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.316+0000 7f0a657c7640 1 -- 172.21.15.59:0/2055961845 wait complete. 2024-06-29T14:37:21.331 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.452+0000 7fd687acc640 1 -- 172.21.15.59:0/828302913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6800ce950 msgr2=0x7fd6801622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.452+0000 7fd687acc640 1 --2- 172.21.15.59:0/828302913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6800ce950 0x7fd6801622c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd674005fd0 tx=0x7fd67402a5a0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.452+0000 7fd687acc640 1 -- 172.21.15.59:0/828302913 shutdown_connections 2024-06-29T14:37:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.452+0000 7fd687acc640 1 --2- 172.21.15.59:0/828302913 >> v2:172.21.15.153:3300/0 conn(0x7fd680162800 0x7fd680162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.452+0000 7fd687acc640 1 --2- 172.21.15.59:0/828302913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6800ce950 0x7fd6801622c0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.452+0000 7fd687acc640 1 -- 172.21.15.59:0/828302913 >> 172.21.15.59:0/828302913 conn(0x7fd6800b5610 msgr2=0x7fd6800b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.452+0000 7fd687acc640 1 -- 172.21.15.59:0/828302913 shutdown_connections 2024-06-29T14:37:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.452+0000 7fd687acc640 1 -- 172.21.15.59:0/828302913 wait complete. 2024-06-29T14:37:21.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.454+0000 7fd687acc640 1 Processor -- start 2024-06-29T14:37:21.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.454+0000 7fd687acc640 1 -- start start 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.455+0000 7fd687acc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd68008c190 0x7fd6801547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.455+0000 7fd687acc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6800ce950 0x7fd680157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.455+0000 7fd687acc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd680162800 0x7fd68014df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.455+0000 7fd687acc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd6800b7f60 con 0x7fd680162800 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.455+0000 7fd687acc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6800b7de0 con 0x7fd6800ce950 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.455+0000 7fd687acc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd6800b80e0 con 0x7fd68008c190 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.455+0000 7fd686069640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd680162800 0x7fd68014df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd686069640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd680162800 0x7fd68014df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37094/0 (socket says 172.21.15.59:37094) 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd686069640 1 -- 172.21.15.59:0/1352013597 learned_addr learned my addr 172.21.15.59:0/1352013597 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd685067640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6800ce950 0x7fd680157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd685868640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd68008c190 0x7fd6801547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd686069640 1 -- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd68008c190 msgr2=0x7fd6801547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd686069640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd68008c190 0x7fd6801547e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd686069640 1 -- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6800ce950 msgr2=0x7fd680157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd686069640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6800ce950 0x7fd680157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd686069640 1 -- 172.21.15.59:0/1352013597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6801616c0 con 0x7fd680162800 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd685067640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6800ce950 0x7fd680157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd686069640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd680162800 0x7fd68014df60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd678005d00 tx=0x7fd67800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd67e7fc640 1 -- 172.21.15.59:0/1352013597 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678017070 con 0x7fd680162800 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd67e7fc640 1 -- 172.21.15.59:0/1352013597 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd678005330 con 0x7fd680162800 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd67e7fc640 1 -- 172.21.15.59:0/1352013597 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6780047a0 con 0x7fd680162800 2024-06-29T14:37:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6801619a0 con 0x7fd680162800 2024-06-29T14:37:21.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.456+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd680157200 con 0x7fd680162800 2024-06-29T14:37:21.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.457+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6801622c0 con 0x7fd680162800 2024-06-29T14:37:21.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.458+0000 7fd67e7fc640 1 -- 172.21.15.59:0/1352013597 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fd678004940 con 0x7fd680162800 2024-06-29T14:37:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.458+0000 7fd67e7fc640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd66c0763f0 0x7fd66c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.459+0000 7fd67e7fc640 1 -- 172.21.15.59:0/1352013597 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6101+0+0 (secure 0 0 0) 0x7fd678097b90 con 0x7fd680162800 2024-06-29T14:37:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.459+0000 7fd685868640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd66c0763f0 0x7fd66c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.459+0000 7fd685868640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd66c0763f0 0x7fd66c078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd670001ce0 tx=0x7fd670009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.463+0000 7fd67e7fc640 1 -- 172.21.15.59:0/1352013597 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd678061140 con 0x7fd680162800 2024-06-29T14:37:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.654+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd680162eb0 con 0x7fd680162800 2024-06-29T14:37:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.655+0000 7fd67e7fc640 1 -- 172.21.15.59:0/1352013597 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+16909 (secure 0 0 0) 0x7fd678069130 con 0x7fd680162800 2024-06-29T14:37:21.653 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:21.654 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":87,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:20.672434+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:19.028073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.631284+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd66c0763f0 msgr2=0x7fd66c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd66c0763f0 0x7fd66c078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd670001ce0 tx=0x7fd670009040 comp rx=0 tx=0).stop 2024-06-29T14:37:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd680162800 msgr2=0x7fd68014df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd680162800 0x7fd68014df60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd678005d00 tx=0x7fd67800dae0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 shutdown_connections 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fd66c0763f0 0x7fd66c078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd680162800 0x7fd68014df60 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6800ce950 0x7fd680157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 --2- 172.21.15.59:0/1352013597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd68008c190 0x7fd6801547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 >> 172.21.15.59:0/1352013597 conn(0x7fd6800b5610 msgr2=0x7fd68006da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 shutdown_connections 2024-06-29T14:37:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.657+0000 7fd687acc640 1 -- 172.21.15.59:0/1352013597 wait complete. 2024-06-29T14:37:21.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.786+0000 7ff688e74640 1 -- 172.21.15.59:0/2720401175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff68415efe0 msgr2=0x7ff68415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.786+0000 7ff688e74640 1 --2- 172.21.15.59:0/2720401175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff68415efe0 0x7ff68415f9b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff6840d15d0 tx=0x7ff67c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.786+0000 7ff688e74640 1 -- 172.21.15.59:0/2720401175 shutdown_connections 2024-06-29T14:37:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.786+0000 7ff688e74640 1 --2- 172.21.15.59:0/2720401175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff68415efe0 0x7ff68415f9b0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.786+0000 7ff688e74640 1 --2- 172.21.15.59:0/2720401175 >> v2:172.21.15.153:3300/0 conn(0x7ff6840d23c0 0x7ff6840b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.786+0000 7ff688e74640 1 -- 172.21.15.59:0/2720401175 >> 172.21.15.59:0/2720401175 conn(0x7ff6840d3ee0 msgr2=0x7ff6840d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.786+0000 7ff688e74640 1 -- 172.21.15.59:0/2720401175 shutdown_connections 2024-06-29T14:37:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.786+0000 7ff688e74640 1 -- 172.21.15.59:0/2720401175 wait complete. 2024-06-29T14:37:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.788+0000 7ff688e74640 1 Processor -- start 2024-06-29T14:37:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.788+0000 7ff688e74640 1 -- start start 2024-06-29T14:37:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff688e74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6840b3ed0 0x7ff68415a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff688e74640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6840d23c0 0x7ff684155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff688e74640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff68415efe0 0x7ff68415b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff688e74640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff684160280 con 0x7ff6840b3ed0 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff688e74640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff684160100 con 0x7ff6840d23c0 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff688e74640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff684160400 con 0x7ff68415efe0 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6840b3ed0 0x7ff68415a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6840b3ed0 0x7ff68415a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37104/0 (socket says 172.21.15.59:37104) 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 -- 172.21.15.59:0/989916279 learned_addr learned my addr 172.21.15.59:0/989916279 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff681d9b640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6840d23c0 0x7ff684155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff682d9d640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff68415efe0 0x7ff68415b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 -- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff68415efe0 msgr2=0x7ff68415b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff68415efe0 0x7ff68415b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 -- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6840d23c0 msgr2=0x7ff684155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6840d23c0 0x7ff684155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 -- 172.21.15.59:0/989916279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68415a870 con 0x7ff6840b3ed0 2024-06-29T14:37:21.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff681d9b640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6840d23c0 0x7ff684155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:21.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.790+0000 7ff68259c640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6840b3ed0 0x7ff68415a330 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ff67400d6d0 tx=0x7ff67400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:21.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.791+0000 7ff672ffd640 1 -- 172.21.15.59:0/989916279 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674017070 con 0x7ff6840b3ed0 2024-06-29T14:37:21.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.791+0000 7ff672ffd640 1 -- 172.21.15.59:0/989916279 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff674004310 con 0x7ff6840b3ed0 2024-06-29T14:37:21.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.791+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff684156070 con 0x7ff6840b3ed0 2024-06-29T14:37:21.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.791+0000 7ff672ffd640 1 -- 172.21.15.59:0/989916279 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674009a80 con 0x7ff6840b3ed0 2024-06-29T14:37:21.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.791+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff684159570 con 0x7ff6840b3ed0 2024-06-29T14:37:21.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.792+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff684069de0 con 0x7ff6840b3ed0 2024-06-29T14:37:21.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.793+0000 7ff672ffd640 1 -- 172.21.15.59:0/989916279 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7ff6740035f0 con 0x7ff6840b3ed0 2024-06-29T14:37:21.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.793+0000 7ff672ffd640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff6640763f0 0x7ff664078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.793+0000 7ff672ffd640 1 -- 172.21.15.59:0/989916279 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6101+0+0 (secure 0 0 0) 0x7ff674098340 con 0x7ff6840b3ed0 2024-06-29T14:37:21.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.794+0000 7ff681d9b640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff6640763f0 0x7ff664078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.794+0000 7ff681d9b640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff6640763f0 0x7ff664078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6841e85f0 tx=0x7ff678009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:21.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.797+0000 7ff672ffd640 1 -- 172.21.15.59:0/989916279 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff684069de0 con 0x7ff6840b3ed0 2024-06-29T14:37:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.990+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6841581c0 con 0x7ff6840b3ed0 2024-06-29T14:37:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.991+0000 7ff672ffd640 1 -- 172.21.15.59:0/989916279 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+16909 (secure 0 0 0) 0x7ff6841581c0 con 0x7ff6840b3ed0 2024-06-29T14:37:21.990 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:21.991 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":87,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:20.672434+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:19.028073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.631284+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.993+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff6640763f0 msgr2=0x7ff664078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.993+0000 7ff688e74640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff6640763f0 0x7ff664078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6841e85f0 tx=0x7ff678009040 comp rx=0 tx=0).stop 2024-06-29T14:37:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.993+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6840b3ed0 msgr2=0x7ff68415a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.993+0000 7ff688e74640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6840b3ed0 0x7ff68415a330 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ff67400d6d0 tx=0x7ff67400db90 comp rx=0 tx=0).stop 2024-06-29T14:37:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.994+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 shutdown_connections 2024-06-29T14:37:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.994+0000 7ff688e74640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7ff6640763f0 0x7ff664078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.994+0000 7ff688e74640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff68415efe0 0x7ff68415b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.994+0000 7ff688e74640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6840d23c0 0x7ff684155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.994+0000 7ff688e74640 1 --2- 172.21.15.59:0/989916279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6840b3ed0 0x7ff68415a330 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.994+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 >> 172.21.15.59:0/989916279 conn(0x7ff6840d3ee0 msgr2=0x7ff684161300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.994+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 shutdown_connections 2024-06-29T14:37:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:21.994+0000 7ff688e74640 1 -- 172.21.15.59:0/989916279 wait complete. 2024-06-29T14:37:22.003 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:37:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.126+0000 7f8f249ef640 1 -- 172.21.15.59:0/4013330427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200d1a60 msgr2=0x7f8f200b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.126+0000 7f8f249ef640 1 --2- 172.21.15.59:0/4013330427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200d1a60 0x7f8f200b5b30 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8f10005fd0 tx=0x7f8f1002aa90 comp rx=0 tx=0).stop 2024-06-29T14:37:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.126+0000 7f8f249ef640 1 -- 172.21.15.59:0/4013330427 shutdown_connections 2024-06-29T14:37:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.126+0000 7f8f249ef640 1 --2- 172.21.15.59:0/4013330427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200d1a60 0x7f8f200b5b30 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.126+0000 7f8f249ef640 1 --2- 172.21.15.59:0/4013330427 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f200b2fa0 0x7f8f200d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.126+0000 7f8f249ef640 1 -- 172.21.15.59:0/4013330427 >> 172.21.15.59:0/4013330427 conn(0x7f8f20073e10 msgr2=0x7f8f20074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.126+0000 7f8f249ef640 1 -- 172.21.15.59:0/4013330427 shutdown_connections 2024-06-29T14:37:22.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.126+0000 7f8f249ef640 1 -- 172.21.15.59:0/4013330427 wait complete. 2024-06-29T14:37:22.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.128+0000 7f8f249ef640 1 Processor -- start 2024-06-29T14:37:22.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.128+0000 7f8f249ef640 1 -- start start 2024-06-29T14:37:22.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f249ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f200b2270 0x7f8f2006db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:22.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f249ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200b2fa0 0x7f8f200ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:22.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f249ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f200d1a60 0x7f8f201f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:22.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f249ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8f200b6400 con 0x7f8f200b2fa0 2024-06-29T14:37:22.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f249ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f200b6280 con 0x7f8f200d1a60 2024-06-29T14:37:22.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f249ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8f200b6580 con 0x7f8f200b2270 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200b2fa0 0x7f8f200ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200b2fa0 0x7f8f200ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37120/0 (socket says 172.21.15.59:37120) 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 -- 172.21.15.59:0/2890607459 learned_addr learned my addr 172.21.15.59:0/2890607459 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1e59c640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f200b2270 0x7f8f2006db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1ed9d640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f200d1a60 0x7f8f201f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 -- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f200b2270 msgr2=0x7f8f2006db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f200b2270 0x7f8f2006db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 -- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f200d1a60 msgr2=0x7f8f201f52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f200d1a60 0x7f8f201f52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 -- 172.21.15.59:0/2890607459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2006ea40 con 0x7f8f200b2fa0 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1e59c640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f200b2270 0x7f8f2006db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1ed9d640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f200d1a60 0x7f8f201f52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.130+0000 7f8f1dd9b640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200b2fa0 0x7f8f200ad8b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800d740 tx=0x7f8f0800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.131+0000 7f8f16ffd640 1 -- 172.21.15.59:0/2890607459 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08017070 con 0x7f8f200b2fa0 2024-06-29T14:37:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.131+0000 7f8f16ffd640 1 -- 172.21.15.59:0/2890607459 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f08009d20 con 0x7f8f200b2fa0 2024-06-29T14:37:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.131+0000 7f8f16ffd640 1 -- 172.21.15.59:0/2890607459 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f080106c0 con 0x7f8f200b2fa0 2024-06-29T14:37:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.131+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2006e480 con 0x7f8f200b2fa0 2024-06-29T14:37:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.131+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f201f4af0 con 0x7f8f200b2fa0 2024-06-29T14:37:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.132+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f2006e1f0 con 0x7f8f200b2fa0 2024-06-29T14:37:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.132+0000 7f8f16ffd640 1 -- 172.21.15.59:0/2890607459 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f8f0801a070 con 0x7f8f200b2fa0 2024-06-29T14:37:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.133+0000 7f8f16ffd640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8f000763f0 0x7f8f00078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.133+0000 7f8f16ffd640 1 -- 172.21.15.59:0/2890607459 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6101+0+0 (secure 0 0 0) 0x7f8f08097cd0 con 0x7f8f200b2fa0 2024-06-29T14:37:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.133+0000 7f8f1e59c640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8f000763f0 0x7f8f00078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.134+0000 7f8f1e59c640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8f000763f0 0x7f8f00078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f200b1fc0 tx=0x7f8f18009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.137+0000 7f8f16ffd640 1 -- 172.21.15.59:0/2890607459 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8f080611f0 con 0x7f8f200b2fa0 2024-06-29T14:37:22.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:22.373+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f8f200b6aa0 con 0x7f8f200b2fa0 2024-06-29T14:37:23.269 INFO:tasks.ceph.mds.d.smithi153.stderr:2024-06-29T14:37:23.269+0000 7faa379c0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:37:23.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.315+0000 7f8f16ffd640 1 -- 172.21.15.59:0/2890607459 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v37) ==== 118+0+0 (secure 0 0 0) 0x7f8f08060af0 con 0x7f8f200b2fa0 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.318+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8f000763f0 msgr2=0x7f8f00078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.318+0000 7f8f249ef640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8f000763f0 0x7f8f00078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f200b1fc0 tx=0x7f8f18009040 comp rx=0 tx=0).stop 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.318+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200b2fa0 msgr2=0x7f8f200ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.318+0000 7f8f249ef640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200b2fa0 0x7f8f200ad8b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800d740 tx=0x7f8f0800dc00 comp rx=0 tx=0).stop 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.318+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 shutdown_connections 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.318+0000 7f8f249ef640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f8f000763f0 0x7f8f00078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.318+0000 7f8f249ef640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f200d1a60 0x7f8f201f52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.319+0000 7f8f249ef640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f200b2fa0 0x7f8f200ad8b0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.319+0000 7f8f249ef640 1 --2- 172.21.15.59:0/2890607459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8f200b2270 0x7f8f2006db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.319+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 >> 172.21.15.59:0/2890607459 conn(0x7f8f20073e10 msgr2=0x7f8f20155590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:23.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.319+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 shutdown_connections 2024-06-29T14:37:23.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.319+0000 7f8f249ef640 1 -- 172.21.15.59:0/2890607459 wait complete. 2024-06-29T14:37:23.331 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:37:23.331 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:37:23.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.461+0000 7f2c8a70a640 1 -- 172.21.15.59:0/727176267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c6c0040f0 msgr2=0x7f2c6c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.461+0000 7f2c8a70a640 1 --2- 172.21.15.59:0/727176267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c6c0040f0 0x7f2c6c004800 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800bb20 tx=0x7f2c6c005830 comp rx=0 tx=0).stop 2024-06-29T14:37:23.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.461+0000 7f2c88ca7640 1 -- 172.21.15.59:0/727176267 reap_dead start 2024-06-29T14:37:23.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.462+0000 7f2c8a70a640 1 -- 172.21.15.59:0/727176267 shutdown_connections 2024-06-29T14:37:23.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.462+0000 7f2c8a70a640 1 -- 172.21.15.59:0/727176267 >> 172.21.15.59:0/727176267 conn(0x7f2c8406c950 msgr2=0x7f2c84073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:23.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.462+0000 7f2c8a70a640 1 -- 172.21.15.59:0/727176267 shutdown_connections 2024-06-29T14:37:23.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.462+0000 7f2c8a70a640 1 -- 172.21.15.59:0/727176267 wait complete. 2024-06-29T14:37:23.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.463+0000 7f2c8a70a640 1 Processor -- start 2024-06-29T14:37:23.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.464+0000 7f2c8a70a640 1 -- start start 2024-06-29T14:37:23.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c8a70a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c6c0040f0 0x7f2c840c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:23.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c8a70a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c841402d0 0x7f2c841f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:23.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c8a70a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c841406e0 0x7f2c841f6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:23.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c8a70a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2c840b5a00 con 0x7f2c841402d0 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c8a70a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c840b5880 con 0x7f2c841406e0 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c8a70a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2c840b5b80 con 0x7f2c6c0040f0 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c837fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c841402d0 0x7f2c841f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c837fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c841402d0 0x7f2c841f5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58692/0 (socket says 172.21.15.59:58692) 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c837fe640 1 -- 172.21.15.59:0/2051512333 learned_addr learned my addr 172.21.15.59:0/2051512333 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c88ca7640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c841406e0 0x7f2c841f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c83fff640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c6c0040f0 0x7f2c840c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c837fe640 1 -- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c6c0040f0 msgr2=0x7f2c840c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c837fe640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c6c0040f0 0x7f2c840c6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c837fe640 1 -- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c841406e0 msgr2=0x7f2c841f6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c837fe640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c841406e0 0x7f2c841f6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.465+0000 7f2c837fe640 1 -- 172.21.15.59:0/2051512333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8413c1e0 con 0x7f2c841402d0 2024-06-29T14:37:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.466+0000 7f2c837fe640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c841402d0 0x7f2c841f5f30 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800bb20 tx=0x7f2c7800b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:23.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.466+0000 7f2c80ff9640 1 -- 172.21.15.59:0/2051512333 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78040070 con 0x7f2c841402d0 2024-06-29T14:37:23.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.466+0000 7f2c80ff9640 1 -- 172.21.15.59:0/2051512333 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c7803a920 con 0x7f2c841402d0 2024-06-29T14:37:23.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.466+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8413c4c0 con 0x7f2c841402d0 2024-06-29T14:37:23.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.466+0000 7f2c80ff9640 1 -- 172.21.15.59:0/2051512333 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78035040 con 0x7f2c841402d0 2024-06-29T14:37:23.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.466+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c8413ca80 con 0x7f2c841402d0 2024-06-29T14:37:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.467+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c841458a0 con 0x7f2c841402d0 2024-06-29T14:37:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.468+0000 7f2c80ff9640 1 -- 172.21.15.59:0/2051512333 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f2c78038020 con 0x7f2c841402d0 2024-06-29T14:37:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.468+0000 7f2c80ff9640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2c700763f0 0x7f2c70078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.468+0000 7f2c80ff9640 1 -- 172.21.15.59:0/2051512333 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f2c780b8a30 con 0x7f2c841402d0 2024-06-29T14:37:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.468+0000 7f2c83fff640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2c700763f0 0x7f2c70078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.469+0000 7f2c83fff640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2c700763f0 0x7f2c70078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c74006bf0 tx=0x7f2c740011c0 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:23.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.473+0000 7f2c80ff9640 1 -- 172.21.15.59:0/2051512333 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2c78081a10 con 0x7f2c841402d0 2024-06-29T14:37:23.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:23.704+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f2c840b1f10 con 0x7f2c841402d0 2024-06-29T14:37:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.318+0000 7f2c80ff9640 1 -- 172.21.15.59:0/2051512333 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7f2c78085d10 con 0x7f2c841402d0 2024-06-29T14:37:24.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2c700763f0 msgr2=0x7f2c70078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2c700763f0 0x7f2c70078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2c74006bf0 tx=0x7f2c740011c0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c841402d0 msgr2=0x7f2c841f5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c841402d0 0x7f2c841f5f30 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800bb20 tx=0x7f2c7800b770 comp rx=0 tx=0).stop 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 shutdown_connections 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2c700763f0 0x7f2c70078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c841406e0 0x7f2c841f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c841402d0 0x7f2c841f5f30 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 --2- 172.21.15.59:0/2051512333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c6c0040f0 0x7f2c840c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 >> 172.21.15.59:0/2051512333 conn(0x7f2c8406c950 msgr2=0x7f2c8406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 shutdown_connections 2024-06-29T14:37:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.321+0000 7f2c8a70a640 1 -- 172.21.15.59:0/2051512333 wait complete. 2024-06-29T14:37:24.333 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:37:24.334 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.460+0000 7fe1d857a640 1 -- 172.21.15.59:0/512917368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc004d90 msgr2=0x7fe1bc005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.460+0000 7fe1d857a640 1 --2- 172.21.15.59:0/512917368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1bc004d90 0x7fe1bc005260 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe1c800c630 tx=0x7fe1bc005830 comp rx=0 tx=0).stop 2024-06-29T14:37:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.460+0000 7fe1d6b17640 1 -- 172.21.15.59:0/512917368 reap_dead start 2024-06-29T14:37:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.460+0000 7fe1d857a640 1 -- 172.21.15.59:0/512917368 shutdown_connections 2024-06-29T14:37:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.461+0000 7fe1d857a640 1 -- 172.21.15.59:0/512917368 >> 172.21.15.59:0/512917368 conn(0x7fe1d00b5610 msgr2=0x7fe1d00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.461+0000 7fe1d857a640 1 -- 172.21.15.59:0/512917368 shutdown_connections 2024-06-29T14:37:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.461+0000 7fe1d857a640 1 -- 172.21.15.59:0/512917368 wait complete. 2024-06-29T14:37:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.463+0000 7fe1d857a640 1 Processor -- start 2024-06-29T14:37:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.463+0000 7fe1d857a640 1 -- start start 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d857a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc004d90 0x7fe1d0159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d857a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d015a260 0x7fe1d00cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d857a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1d015a670 0x7fe1d00ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d857a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d0162c10 con 0x7fe1d015a260 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d857a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d01625c0 con 0x7fe1bc004d90 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d857a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d0162d90 con 0x7fe1d015a670 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d015a260 0x7fe1d00cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d015a260 0x7fe1d00cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58706/0 (socket says 172.21.15.59:58706) 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 -- 172.21.15.59:0/3427078500 learned_addr learned my addr 172.21.15.59:0/3427078500 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d6316640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc004d90 0x7fe1d0159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d6b17640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1d015a670 0x7fe1d00ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 -- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1d015a670 msgr2=0x7fe1d00ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1d015a670 0x7fe1d00ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 -- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc004d90 msgr2=0x7fe1d0159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc004d90 0x7fe1d0159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 -- 172.21.15.59:0/3427078500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d015b5a0 con 0x7fe1d015a260 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d6b17640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1d015a670 0x7fe1d00ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d5b15640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d015a260 0x7fe1d00cc6a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0011a50 tx=0x7fe1c0011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1c6ffd640 1 -- 172.21.15.59:0/3427078500 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c001f070 con 0x7fe1d015a260 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1c6ffd640 1 -- 172.21.15.59:0/3427078500 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1c000fd60 con 0x7fe1d015a260 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d015ad00 con 0x7fe1d015a260 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1c6ffd640 1 -- 172.21.15.59:0/3427078500 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00126b0 con 0x7fe1d015a260 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.465+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d015b2c0 con 0x7fe1d015a260 2024-06-29T14:37:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.466+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d00cf080 con 0x7fe1d015a260 2024-06-29T14:37:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.466+0000 7fe1c6ffd640 1 -- 172.21.15.59:0/3427078500 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fe1c0022070 con 0x7fe1d015a260 2024-06-29T14:37:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.467+0000 7fe1c6ffd640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe1b40763f0 0x7fe1b4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.467+0000 7fe1c6ffd640 1 -- 172.21.15.59:0/3427078500 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7fe1c0098510 con 0x7fe1d015a260 2024-06-29T14:37:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.467+0000 7fe1d6316640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe1b40763f0 0x7fe1b4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.468+0000 7fe1d6316640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe1b40763f0 0x7fe1b4078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc000c00 tx=0x7fe1cc006d60 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.472+0000 7fe1c6ffd640 1 -- 172.21.15.59:0/3427078500 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe1c00614f0 con 0x7fe1d015a260 2024-06-29T14:37:24.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.701+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1d015aa90 con 0x7fe1d015a260 2024-06-29T14:37:24.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.702+0000 7fe1c6ffd640 1 -- 172.21.15.59:0/3427078500 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5230 (secure 0 0 0) 0x7fe1c00657f0 con 0x7fe1d015a260 2024-06-29T14:37:24.700 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:24.700 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":38,"btime":"2024-06-29T14:37:24:318813+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6529,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3719851385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3719851385},{"type":"v1","addr":"172.21.15.153:6836","nonce":3719851385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1280649067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1280649067},{"type":"v1","addr":"172.21.15.59:6833","nonce":1280649067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2847930906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2847930906},{"type":"v1","addr":"172.21.15.59:6835","nonce":2847930906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.672386+0000","modified":"2024-06-29T14:37:23.706036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"d","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/3891057790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3891057790},{"type":"v1","addr":"172.21.15.153:6837","nonce":3891057790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":4}]} 2024-06-29T14:37:24.700 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 38 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe1b40763f0 msgr2=0x7fe1b4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe1b40763f0 0x7fe1b4078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc000c00 tx=0x7fe1cc006d60 comp rx=0 tx=0).stop 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d015a260 msgr2=0x7fe1d00cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d015a260 0x7fe1d00cc6a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0011a50 tx=0x7fe1c0011f10 comp rx=0 tx=0).stop 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 shutdown_connections 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fe1b40763f0 0x7fe1b4078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1d015a670 0x7fe1d00ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d015a260 0x7fe1d00cc6a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 --2- 172.21.15.59:0/3427078500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1bc004d90 0x7fe1d0159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 >> 172.21.15.59:0/3427078500 conn(0x7fe1d00b5610 msgr2=0x7fe1d00b7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 shutdown_connections 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.704+0000 7fe1d857a640 1 -- 172.21.15.59:0/3427078500 wait complete. 2024-06-29T14:37:24.714 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 -- 172.21.15.59:0/3406072898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 msgr2=0x7f039013edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 --2- 172.21.15.59:0/3406072898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 0x7f039013edc0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0390073f30 tx=0x7f038402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 -- 172.21.15.59:0/3406072898 shutdown_connections 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 --2- 172.21.15.59:0/3406072898 >> v2:172.21.15.153:3300/0 conn(0x7f03901414c0 0x7f03901418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 --2- 172.21.15.59:0/3406072898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 0x7f039013edc0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 --2- 172.21.15.59:0/3406072898 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f039015ceb0 0x7f03900b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 -- 172.21.15.59:0/3406072898 >> 172.21.15.59:0/3406072898 conn(0x7f03900b6fb0 msgr2=0x7f03900b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:24.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 -- 172.21.15.59:0/3406072898 shutdown_connections 2024-06-29T14:37:24.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.834+0000 7f039623d640 1 -- 172.21.15.59:0/3406072898 wait complete. 2024-06-29T14:37:24.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.836+0000 7f039623d640 1 Processor -- start 2024-06-29T14:37:24.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.836+0000 7f039623d640 1 -- start start 2024-06-29T14:37:24.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f039623d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 0x7f039013c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f039623d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03901414c0 0x7f039006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f039623d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f039015ceb0 0x7f03900ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f039623d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f03900b28c0 con 0x7f039013e9b0 2024-06-29T14:37:24.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f039623d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03900b2740 con 0x7f03901414c0 2024-06-29T14:37:24.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f039623d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f03900b2a40 con 0x7f039015ceb0 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 0x7f039013c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 0x7f039013c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58724/0 (socket says 172.21.15.59:58724) 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 -- 172.21.15.59:0/528806209 learned_addr learned my addr 172.21.15.59:0/528806209 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038ffff640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f039015ceb0 0x7f03900ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038effd640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03901414c0 0x7f039006ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 -- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f039015ceb0 msgr2=0x7f03900ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f039015ceb0 0x7f03900ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 -- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03901414c0 msgr2=0x7f039006ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03901414c0 0x7f039006ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 -- 172.21.15.59:0/528806209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03901538a0 con 0x7f039013e9b0 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f038f7fe640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 0x7f039013c5a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f038000d770 tx=0x7f038000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f0377fff640 1 -- 172.21.15.59:0/528806209 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0380017070 con 0x7f039013e9b0 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f0377fff640 1 -- 172.21.15.59:0/528806209 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0380009d20 con 0x7f039013e9b0 2024-06-29T14:37:24.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0390153b80 con 0x7f039013e9b0 2024-06-29T14:37:24.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f0377fff640 1 -- 172.21.15.59:0/528806209 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03800106c0 con 0x7f039013e9b0 2024-06-29T14:37:24.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.839+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0390154140 con 0x7f039013e9b0 2024-06-29T14:37:24.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.840+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03900b2f60 con 0x7f039013e9b0 2024-06-29T14:37:24.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.840+0000 7f0377fff640 1 -- 172.21.15.59:0/528806209 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f038001a070 con 0x7f039013e9b0 2024-06-29T14:37:24.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.841+0000 7f0377fff640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f037c0763f0 0x7f037c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.841+0000 7f0377fff640 1 -- 172.21.15.59:0/528806209 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f0380097cd0 con 0x7f039013e9b0 2024-06-29T14:37:24.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.841+0000 7f038effd640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f037c0763f0 0x7f037c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.842+0000 7f038effd640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f037c0763f0 0x7f037c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0384033000 tx=0x7f0384004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:24.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:24.845+0000 7f0377fff640 1 -- 172.21.15.59:0/528806209 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0380060b20 con 0x7f039013e9b0 2024-06-29T14:37:25.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.037+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f039013e470 con 0x7f039013e9b0 2024-06-29T14:37:25.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.039+0000 7f0377fff640 1 -- 172.21.15.59:0/528806209 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+16896 (secure 0 0 0) 0x7f0380064e20 con 0x7f039013e9b0 2024-06-29T14:37:25.036 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:25.036 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:22.293574+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:19.028073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.631284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:25.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f037c0763f0 msgr2=0x7f037c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f037c0763f0 0x7f037c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0384033000 tx=0x7f0384004730 comp rx=0 tx=0).stop 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 msgr2=0x7f039013c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 0x7f039013c5a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f038000d770 tx=0x7f038000dc30 comp rx=0 tx=0).stop 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 shutdown_connections 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f037c0763f0 0x7f037c078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f039015ceb0 0x7f03900ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03901414c0 0x7f039006ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 --2- 172.21.15.59:0/528806209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f039013e9b0 0x7f039013c5a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 >> 172.21.15.59:0/528806209 conn(0x7f03900b6fb0 msgr2=0x7f039015d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 shutdown_connections 2024-06-29T14:37:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.041+0000 7f039623d640 1 -- 172.21.15.59:0/528806209 wait complete. 2024-06-29T14:37:25.050 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:37:25.050 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:37:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.170+0000 7fdae91bd640 1 -- 172.21.15.59:0/2677213220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40b40a0 msgr2=0x7fdae415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.170+0000 7fdae91bd640 1 --2- 172.21.15.59:0/2677213220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40b40a0 0x7fdae415e8d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fdad8005fd0 tx=0x7fdad802a520 comp rx=0 tx=0).stop 2024-06-29T14:37:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.170+0000 7fdae91bd640 1 -- 172.21.15.59:0/2677213220 shutdown_connections 2024-06-29T14:37:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.170+0000 7fdae91bd640 1 --2- 172.21.15.59:0/2677213220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae415ee10 0x7fdae415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.170+0000 7fdae91bd640 1 --2- 172.21.15.59:0/2677213220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40b40a0 0x7fdae415e8d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.170+0000 7fdae91bd640 1 -- 172.21.15.59:0/2677213220 >> 172.21.15.59:0/2677213220 conn(0x7fdae40d3ee0 msgr2=0x7fdae40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.170+0000 7fdae91bd640 1 -- 172.21.15.59:0/2677213220 shutdown_connections 2024-06-29T14:37:25.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.170+0000 7fdae91bd640 1 -- 172.21.15.59:0/2677213220 wait complete. 2024-06-29T14:37:25.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.172+0000 7fdae91bd640 1 Processor -- start 2024-06-29T14:37:25.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.172+0000 7fdae91bd640 1 -- start start 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae91bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40d23c0 0x7fdae40ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae91bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae415ee10 0x7fdae40c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae91bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdae40c8d10 0x7fdae40c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae91bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdae4160280 con 0x7fdae40d23c0 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae91bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdae4160100 con 0x7fdae40c8d10 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae91bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdae4160400 con 0x7fdae415ee10 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40d23c0 0x7fdae40ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40d23c0 0x7fdae40ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58750/0 (socket says 172.21.15.59:58750) 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.173+0000 7fdae2d9d640 1 -- 172.21.15.59:0/3121082664 learned_addr learned my addr 172.21.15.59:0/3121082664 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae259c640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae415ee10 0x7fdae40c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae359e640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdae40c8d10 0x7fdae40c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae2d9d640 1 -- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae415ee10 msgr2=0x7fdae40c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae2d9d640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae415ee10 0x7fdae40c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae2d9d640 1 -- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdae40c8d10 msgr2=0x7fdae40c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae2d9d640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdae40c8d10 0x7fdae40c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae2d9d640 1 -- 172.21.15.59:0/3121082664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae40c9660 con 0x7fdae40d23c0 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae359e640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdae40c8d10 0x7fdae40c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae259c640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae415ee10 0x7fdae40c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae2d9d640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40d23c0 0x7fdae40ca0e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fdad400ba10 tx=0x7fdad400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdacb7fe640 1 -- 172.21.15.59:0/3121082664 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad4019070 con 0x7fdae40d23c0 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdacb7fe640 1 -- 172.21.15.59:0/3121082664 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdad400ccc0 con 0x7fdae40d23c0 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae40cb3d0 con 0x7fdae40d23c0 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdacb7fe640 1 -- 172.21.15.59:0/3121082664 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad40146a0 con 0x7fdae40d23c0 2024-06-29T14:37:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.174+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae40cab30 con 0x7fdae40d23c0 2024-06-29T14:37:25.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.175+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae415e8d0 con 0x7fdae40d23c0 2024-06-29T14:37:25.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.176+0000 7fdacb7fe640 1 -- 172.21.15.59:0/3121082664 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fdad401c070 con 0x7fdae40d23c0 2024-06-29T14:37:25.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.176+0000 7fdacb7fe640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fdad00763f0 0x7fdad0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.176+0000 7fdacb7fe640 1 -- 172.21.15.59:0/3121082664 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7fdad4097840 con 0x7fdae40d23c0 2024-06-29T14:37:25.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.177+0000 7fdae259c640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fdad00763f0 0x7fdad0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.177+0000 7fdae259c640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fdad00763f0 0x7fdad0078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdad8033000 tx=0x7fdad8004730 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.181+0000 7fdacb7fe640 1 -- 172.21.15.59:0/3121082664 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdae40cb3d0 con 0x7fdae40d23c0 2024-06-29T14:37:25.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.372+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 --> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdae40cd9a0 con 0x7fdad00763f0 2024-06-29T14:37:25.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.377+0000 7fdacb7fe640 1 -- 172.21.15.59:0/3121082664 <== mgr.6316 v2:172.21.15.153:6832/4269929333 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252032 (secure 0 0 0) 0x7fdae40cd9a0 con 0x7fdad00763f0 2024-06-29T14:37:25.377 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:25.379 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fdad00763f0 msgr2=0x7fdad0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fdad00763f0 0x7fdad0078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdad8033000 tx=0x7fdad8004730 comp rx=0 tx=0).stop 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40d23c0 msgr2=0x7fdae40ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40d23c0 0x7fdae40ca0e0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fdad400ba10 tx=0x7fdad400bed0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 shutdown_connections 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fdad00763f0 0x7fdad0078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdae40c8d10 0x7fdae40c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdae415ee10 0x7fdae40c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 --2- 172.21.15.59:0/3121082664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdae40d23c0 0x7fdae40ca0e0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 >> 172.21.15.59:0/3121082664 conn(0x7fdae40d3ee0 msgr2=0x7fdae41608f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 shutdown_connections 2024-06-29T14:37:25.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.381+0000 7fdae91bd640 1 -- 172.21.15.59:0/3121082664 wait complete. 2024-06-29T14:37:25.389 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-06-29T14:37:24.799111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":194,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329752,"kb_used_data":9500,"kb_used_omap":12,"kb_used_meta":320115,"kb_avail":754644968,"statfs":{"total":773094113280,"available":772756447232,"internally_reserved":0,"allocated":9728000,"data_stored":5451088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":327798322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.524256"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332197+0000","last_change":"2024-06-29T14:37:21.325479+0000","last_active":"2024-06-29T14:37:22.332197+0000","last_peered":"2024-06-29T14:37:22.332197+0000","last_clean":"2024-06-29T14:37:22.332197+0000","last_became_active":"2024-06-29T14:37:21.325172+0000","last_became_peered":"2024-06-29T14:37:21.325172+0000","last_unstale":"2024-06-29T14:37:22.332197+0000","last_undegraded":"2024-06-29T14:37:22.332197+0000","last_fullsized":"2024-06-29T14:37:22.332197+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:36.065136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331851+0000","last_change":"2024-06-29T14:37:21.314109+0000","last_active":"2024-06-29T14:37:22.331851+0000","last_peered":"2024-06-29T14:37:22.331851+0000","last_clean":"2024-06-29T14:37:22.331851+0000","last_became_active":"2024-06-29T14:37:21.313798+0000","last_became_peered":"2024-06-29T14:37:21.313798+0000","last_unstale":"2024-06-29T14:37:22.331851+0000","last_undegraded":"2024-06-29T14:37:22.331851+0000","last_fullsized":"2024-06-29T14:37:22.331851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:21:40.130162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330797+0000","last_change":"2024-06-29T14:37:21.314458+0000","last_active":"2024-06-29T14:37:22.330797+0000","last_peered":"2024-06-29T14:37:22.330797+0000","last_clean":"2024-06-29T14:37:22.330797+0000","last_became_active":"2024-06-29T14:37:21.314145+0000","last_became_peered":"2024-06-29T14:37:21.314145+0000","last_unstale":"2024-06-29T14:37:22.330797+0000","last_undegraded":"2024-06-29T14:37:22.330797+0000","last_fullsized":"2024-06-29T14:37:22.330797+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:06.315908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299576+0000","last_change":"2024-06-29T14:37:21.318960+0000","last_active":"2024-06-29T14:37:22.299576+0000","last_peered":"2024-06-29T14:37:22.299576+0000","last_clean":"2024-06-29T14:37:22.299576+0000","last_became_active":"2024-06-29T14:37:21.316698+0000","last_became_peered":"2024-06-29T14:37:21.316698+0000","last_unstale":"2024-06-29T14:37:22.299576+0000","last_undegraded":"2024-06-29T14:37:22.299576+0000","last_fullsized":"2024-06-29T14:37:22.299576+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:32.698381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298991+0000","last_change":"2024-06-29T14:37:21.314305+0000","last_active":"2024-06-29T14:37:22.298991+0000","last_peered":"2024-06-29T14:37:22.298991+0000","last_clean":"2024-06-29T14:37:22.298991+0000","last_became_active":"2024-06-29T14:37:21.314014+0000","last_became_peered":"2024-06-29T14:37:21.314014+0000","last_unstale":"2024-06-29T14:37:22.298991+0000","last_undegraded":"2024-06-29T14:37:22.298991+0000","last_fullsized":"2024-06-29T14:37:22.298991+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:21.787216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331089+0000","last_change":"2024-06-29T14:37:21.329594+0000","last_active":"2024-06-29T14:37:22.331089+0000","last_peered":"2024-06-29T14:37:22.331089+0000","last_clean":"2024-06-29T14:37:22.331089+0000","last_became_active":"2024-06-29T14:37:21.329289+0000","last_became_peered":"2024-06-29T14:37:21.329289+0000","last_unstale":"2024-06-29T14:37:22.331089+0000","last_undegraded":"2024-06-29T14:37:22.331089+0000","last_fullsized":"2024-06-29T14:37:22.331089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:37:58.714617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297089+0000","last_change":"2024-06-29T14:37:21.316962+0000","last_active":"2024-06-29T14:37:22.297089+0000","last_peered":"2024-06-29T14:37:22.297089+0000","last_clean":"2024-06-29T14:37:22.297089+0000","last_became_active":"2024-06-29T14:37:21.316670+0000","last_became_peered":"2024-06-29T14:37:21.316670+0000","last_unstale":"2024-06-29T14:37:22.297089+0000","last_undegraded":"2024-06-29T14:37:22.297089+0000","last_fullsized":"2024-06-29T14:37:22.297089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:34.422922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315653+0000","last_change":"2024-06-29T14:37:21.318221+0000","last_active":"2024-06-29T14:37:22.315653+0000","last_peered":"2024-06-29T14:37:22.315653+0000","last_clean":"2024-06-29T14:37:22.315653+0000","last_became_active":"2024-06-29T14:37:21.317883+0000","last_became_peered":"2024-06-29T14:37:21.317883+0000","last_unstale":"2024-06-29T14:37:22.315653+0000","last_undegraded":"2024-06-29T14:37:22.315653+0000","last_fullsized":"2024-06-29T14:37:22.315653+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:23.160262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331229+0000","last_change":"2024-06-29T14:37:21.312937+0000","last_active":"2024-06-29T14:37:22.331229+0000","last_peered":"2024-06-29T14:37:22.331229+0000","last_clean":"2024-06-29T14:37:22.331229+0000","last_became_active":"2024-06-29T14:37:21.312610+0000","last_became_peered":"2024-06-29T14:37:21.312610+0000","last_unstale":"2024-06-29T14:37:22.331229+0000","last_undegraded":"2024-06-29T14:37:22.331229+0000","last_fullsized":"2024-06-29T14:37:22.331229+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:17:23.261516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331153+0000","last_change":"2024-06-29T14:37:21.321802+0000","last_active":"2024-06-29T14:37:22.331153+0000","last_peered":"2024-06-29T14:37:22.331153+0000","last_clean":"2024-06-29T14:37:22.331153+0000","last_became_active":"2024-06-29T14:37:21.321487+0000","last_became_peered":"2024-06-29T14:37:21.321487+0000","last_unstale":"2024-06-29T14:37:22.331153+0000","last_undegraded":"2024-06-29T14:37:22.331153+0000","last_fullsized":"2024-06-29T14:37:22.331153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:29.050996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298410+0000","last_change":"2024-06-29T14:37:21.311941+0000","last_active":"2024-06-29T14:37:22.298410+0000","last_peered":"2024-06-29T14:37:22.298410+0000","last_clean":"2024-06-29T14:37:22.298410+0000","last_became_active":"2024-06-29T14:37:21.311617+0000","last_became_peered":"2024-06-29T14:37:21.311617+0000","last_unstale":"2024-06-29T14:37:22.298410+0000","last_undegraded":"2024-06-29T14:37:22.298410+0000","last_fullsized":"2024-06-29T14:37:22.298410+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:21.787216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315916+0000","last_change":"2024-06-29T14:37:21.328427+0000","last_active":"2024-06-29T14:37:22.315916+0000","last_peered":"2024-06-29T14:37:22.315916+0000","last_clean":"2024-06-29T14:37:22.315916+0000","last_became_active":"2024-06-29T14:37:21.328087+0000","last_became_peered":"2024-06-29T14:37:21.328087+0000","last_unstale":"2024-06-29T14:37:22.315916+0000","last_undegraded":"2024-06-29T14:37:22.315916+0000","last_fullsized":"2024-06-29T14:37:22.315916+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:24.144680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315753+0000","last_change":"2024-06-29T14:37:21.316159+0000","last_active":"2024-06-29T14:37:22.315753+0000","last_peered":"2024-06-29T14:37:22.315753+0000","last_clean":"2024-06-29T14:37:22.315753+0000","last_became_active":"2024-06-29T14:37:21.315836+0000","last_became_peered":"2024-06-29T14:37:21.315836+0000","last_unstale":"2024-06-29T14:37:22.315753+0000","last_undegraded":"2024-06-29T14:37:22.315753+0000","last_fullsized":"2024-06-29T14:37:22.315753+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:58.778234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330865+0000","last_change":"2024-06-29T14:37:21.315261+0000","last_active":"2024-06-29T14:37:22.330865+0000","last_peered":"2024-06-29T14:37:22.330865+0000","last_clean":"2024-06-29T14:37:22.330865+0000","last_became_active":"2024-06-29T14:37:21.314899+0000","last_became_peered":"2024-06-29T14:37:21.314899+0000","last_unstale":"2024-06-29T14:37:22.330865+0000","last_undegraded":"2024-06-29T14:37:22.330865+0000","last_fullsized":"2024-06-29T14:37:22.330865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:29:57.954983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330714+0000","last_change":"2024-06-29T14:37:21.322853+0000","last_active":"2024-06-29T14:37:22.330714+0000","last_peered":"2024-06-29T14:37:22.330714+0000","last_clean":"2024-06-29T14:37:22.330714+0000","last_became_active":"2024-06-29T14:37:21.314707+0000","last_became_peered":"2024-06-29T14:37:21.314707+0000","last_unstale":"2024-06-29T14:37:22.330714+0000","last_undegraded":"2024-06-29T14:37:22.330714+0000","last_fullsized":"2024-06-29T14:37:22.330714+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:13.135049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297065+0000","last_change":"2024-06-29T14:37:21.323649+0000","last_active":"2024-06-29T14:37:22.297065+0000","last_peered":"2024-06-29T14:37:22.297065+0000","last_clean":"2024-06-29T14:37:22.297065+0000","last_became_active":"2024-06-29T14:37:21.323301+0000","last_became_peered":"2024-06-29T14:37:21.323301+0000","last_unstale":"2024-06-29T14:37:22.297065+0000","last_undegraded":"2024-06-29T14:37:22.297065+0000","last_fullsized":"2024-06-29T14:37:22.297065+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:30:51.584785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332391+0000","last_change":"2024-06-29T14:37:21.314115+0000","last_active":"2024-06-29T14:37:22.332391+0000","last_peered":"2024-06-29T14:37:22.332391+0000","last_clean":"2024-06-29T14:37:22.332391+0000","last_became_active":"2024-06-29T14:37:21.313380+0000","last_became_peered":"2024-06-29T14:37:21.313380+0000","last_unstale":"2024-06-29T14:37:22.332391+0000","last_undegraded":"2024-06-29T14:37:22.332391+0000","last_fullsized":"2024-06-29T14:37:22.332391+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:05.689467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299099+0000","last_change":"2024-06-29T14:37:21.317315+0000","last_active":"2024-06-29T14:37:22.299099+0000","last_peered":"2024-06-29T14:37:22.299099+0000","last_clean":"2024-06-29T14:37:22.299099+0000","last_became_active":"2024-06-29T14:37:21.316741+0000","last_became_peered":"2024-06-29T14:37:21.316741+0000","last_unstale":"2024-06-29T14:37:22.299099+0000","last_undegraded":"2024-06-29T14:37:22.299099+0000","last_fullsized":"2024-06-29T14:37:22.299099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:05.745617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298566+0000","last_change":"2024-06-29T14:37:21.305898+0000","last_active":"2024-06-29T14:37:22.298566+0000","last_peered":"2024-06-29T14:37:22.298566+0000","last_clean":"2024-06-29T14:37:22.298566+0000","last_became_active":"2024-06-29T14:37:21.305566+0000","last_became_peered":"2024-06-29T14:37:21.305566+0000","last_unstale":"2024-06-29T14:37:22.298566+0000","last_undegraded":"2024-06-29T14:37:22.298566+0000","last_fullsized":"2024-06-29T14:37:22.298566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:41.159474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331902+0000","last_change":"2024-06-29T14:37:21.303624+0000","last_active":"2024-06-29T14:37:22.331902+0000","last_peered":"2024-06-29T14:37:22.331902+0000","last_clean":"2024-06-29T14:37:22.331902+0000","last_became_active":"2024-06-29T14:37:21.303305+0000","last_became_peered":"2024-06-29T14:37:21.303305+0000","last_unstale":"2024-06-29T14:37:22.331902+0000","last_undegraded":"2024-06-29T14:37:22.331902+0000","last_fullsized":"2024-06-29T14:37:22.331902+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:59.455244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315875+0000","last_change":"2024-06-29T14:37:21.317932+0000","last_active":"2024-06-29T14:37:22.315875+0000","last_peered":"2024-06-29T14:37:22.315875+0000","last_clean":"2024-06-29T14:37:22.315875+0000","last_became_active":"2024-06-29T14:37:21.317596+0000","last_became_peered":"2024-06-29T14:37:21.317596+0000","last_unstale":"2024-06-29T14:37:22.315875+0000","last_undegraded":"2024-06-29T14:37:22.315875+0000","last_fullsized":"2024-06-29T14:37:22.315875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:51.620680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331541+0000","last_change":"2024-06-29T14:37:21.304728+0000","last_active":"2024-06-29T14:37:22.331541+0000","last_peered":"2024-06-29T14:37:22.331541+0000","last_clean":"2024-06-29T14:37:22.331541+0000","last_became_active":"2024-06-29T14:37:21.304385+0000","last_became_peered":"2024-06-29T14:37:21.304385+0000","last_unstale":"2024-06-29T14:37:22.331541+0000","last_undegraded":"2024-06-29T14:37:22.331541+0000","last_fullsized":"2024-06-29T14:37:22.331541+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:23.160262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331397+0000","last_change":"2024-06-29T14:37:21.311815+0000","last_active":"2024-06-29T14:37:22.331397+0000","last_peered":"2024-06-29T14:37:22.331397+0000","last_clean":"2024-06-29T14:37:22.331397+0000","last_became_active":"2024-06-29T14:37:21.311514+0000","last_became_peered":"2024-06-29T14:37:21.311514+0000","last_unstale":"2024-06-29T14:37:22.331397+0000","last_undegraded":"2024-06-29T14:37:22.331397+0000","last_fullsized":"2024-06-29T14:37:22.331397+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:20.337643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330984+0000","last_change":"2024-06-29T14:37:21.315395+0000","last_active":"2024-06-29T14:37:22.330984+0000","last_peered":"2024-06-29T14:37:22.330984+0000","last_clean":"2024-06-29T14:37:22.330984+0000","last_became_active":"2024-06-29T14:37:21.315072+0000","last_became_peered":"2024-06-29T14:37:21.315072+0000","last_unstale":"2024-06-29T14:37:22.330984+0000","last_undegraded":"2024-06-29T14:37:22.330984+0000","last_fullsized":"2024-06-29T14:37:22.330984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:00:24.182053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315880+0000","last_change":"2024-06-29T14:37:21.317693+0000","last_active":"2024-06-29T14:37:22.315880+0000","last_peered":"2024-06-29T14:37:22.315880+0000","last_clean":"2024-06-29T14:37:22.315880+0000","last_became_active":"2024-06-29T14:37:21.317336+0000","last_became_peered":"2024-06-29T14:37:21.317336+0000","last_unstale":"2024-06-29T14:37:22.315880+0000","last_undegraded":"2024-06-29T14:37:22.315880+0000","last_fullsized":"2024-06-29T14:37:22.315880+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:31.176809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315927+0000","last_change":"2024-06-29T14:37:21.322410+0000","last_active":"2024-06-29T14:37:22.315927+0000","last_peered":"2024-06-29T14:37:22.315927+0000","last_clean":"2024-06-29T14:37:22.315927+0000","last_became_active":"2024-06-29T14:37:21.322044+0000","last_became_peered":"2024-06-29T14:37:21.322044+0000","last_unstale":"2024-06-29T14:37:22.315927+0000","last_undegraded":"2024-06-29T14:37:22.315927+0000","last_fullsized":"2024-06-29T14:37:22.315927+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:12.503380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298573+0000","last_change":"2024-06-29T14:37:21.311920+0000","last_active":"2024-06-29T14:37:22.298573+0000","last_peered":"2024-06-29T14:37:22.298573+0000","last_clean":"2024-06-29T14:37:22.298573+0000","last_became_active":"2024-06-29T14:37:21.311586+0000","last_became_peered":"2024-06-29T14:37:21.311586+0000","last_unstale":"2024-06-29T14:37:22.298573+0000","last_undegraded":"2024-06-29T14:37:22.298573+0000","last_fullsized":"2024-06-29T14:37:22.298573+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:11.052839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331774+0000","last_change":"2024-06-29T14:37:21.317292+0000","last_active":"2024-06-29T14:37:22.331774+0000","last_peered":"2024-06-29T14:37:22.331774+0000","last_clean":"2024-06-29T14:37:22.331774+0000","last_became_active":"2024-06-29T14:37:21.316975+0000","last_became_peered":"2024-06-29T14:37:21.316975+0000","last_unstale":"2024-06-29T14:37:22.331774+0000","last_undegraded":"2024-06-29T14:37:22.331774+0000","last_fullsized":"2024-06-29T14:37:22.331774+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:28.776718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330917+0000","last_change":"2024-06-29T14:37:21.321547+0000","last_active":"2024-06-29T14:37:22.330917+0000","last_peered":"2024-06-29T14:37:22.330917+0000","last_clean":"2024-06-29T14:37:22.330917+0000","last_became_active":"2024-06-29T14:37:21.312371+0000","last_became_peered":"2024-06-29T14:37:21.312371+0000","last_unstale":"2024-06-29T14:37:22.330917+0000","last_undegraded":"2024-06-29T14:37:22.330917+0000","last_fullsized":"2024-06-29T14:37:22.330917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:46.476553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315681+0000","last_change":"2024-06-29T14:37:21.328902+0000","last_active":"2024-06-29T14:37:22.315681+0000","last_peered":"2024-06-29T14:37:22.315681+0000","last_clean":"2024-06-29T14:37:22.315681+0000","last_became_active":"2024-06-29T14:37:21.328651+0000","last_became_peered":"2024-06-29T14:37:21.328651+0000","last_unstale":"2024-06-29T14:37:22.315681+0000","last_undegraded":"2024-06-29T14:37:22.315681+0000","last_fullsized":"2024-06-29T14:37:22.315681+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:59.455244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298101+0000","last_change":"2024-06-29T14:37:21.310485+0000","last_active":"2024-06-29T14:37:22.298101+0000","last_peered":"2024-06-29T14:37:22.298101+0000","last_clean":"2024-06-29T14:37:22.298101+0000","last_became_active":"2024-06-29T14:37:21.310133+0000","last_became_peered":"2024-06-29T14:37:21.310133+0000","last_unstale":"2024-06-29T14:37:22.298101+0000","last_undegraded":"2024-06-29T14:37:22.298101+0000","last_fullsized":"2024-06-29T14:37:22.298101+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:22.177194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330826+0000","last_change":"2024-06-29T14:37:21.314043+0000","last_active":"2024-06-29T14:37:22.330826+0000","last_peered":"2024-06-29T14:37:22.330826+0000","last_clean":"2024-06-29T14:37:22.330826+0000","last_became_active":"2024-06-29T14:37:21.313707+0000","last_became_peered":"2024-06-29T14:37:21.313707+0000","last_unstale":"2024-06-29T14:37:22.330826+0000","last_undegraded":"2024-06-29T14:37:22.330826+0000","last_fullsized":"2024-06-29T14:37:22.330826+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:33.333380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298525+0000","last_change":"2024-06-29T14:37:21.310224+0000","last_active":"2024-06-29T14:37:22.298525+0000","last_peered":"2024-06-29T14:37:22.298525+0000","last_clean":"2024-06-29T14:37:22.298525+0000","last_became_active":"2024-06-29T14:37:21.309932+0000","last_became_peered":"2024-06-29T14:37:21.309932+0000","last_unstale":"2024-06-29T14:37:22.298525+0000","last_undegraded":"2024-06-29T14:37:22.298525+0000","last_fullsized":"2024-06-29T14:37:22.298525+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:33.524218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"87'4","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332344+0000","last_change":"2024-06-29T14:37:20.303082+0000","last_active":"2024-06-29T14:37:22.332344+0000","last_peered":"2024-06-29T14:37:22.332344+0000","last_clean":"2024-06-29T14:37:22.332344+0000","last_became_active":"2024-06-29T14:37:20.302767+0000","last_became_peered":"2024-06-29T14:37:20.302767+0000","last_unstale":"2024-06-29T14:37:22.332344+0000","last_undegraded":"2024-06-29T14:37:22.332344+0000","last_fullsized":"2024-06-29T14:37:22.332344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:10.445714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298727+0000","last_change":"2024-06-29T14:37:21.317749+0000","last_active":"2024-06-29T14:37:22.298727+0000","last_peered":"2024-06-29T14:37:22.298727+0000","last_clean":"2024-06-29T14:37:22.298727+0000","last_became_active":"2024-06-29T14:37:21.317418+0000","last_became_peered":"2024-06-29T14:37:21.317418+0000","last_unstale":"2024-06-29T14:37:22.298727+0000","last_undegraded":"2024-06-29T14:37:22.298727+0000","last_fullsized":"2024-06-29T14:37:22.298727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:09:49.985745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331887+0000","last_change":"2024-06-29T14:37:20.301311+0000","last_active":"2024-06-29T14:37:22.331887+0000","last_peered":"2024-06-29T14:37:22.331887+0000","last_clean":"2024-06-29T14:37:22.331887+0000","last_became_active":"2024-06-29T14:37:20.300988+0000","last_became_peered":"2024-06-29T14:37:20.300988+0000","last_unstale":"2024-06-29T14:37:22.331887+0000","last_undegraded":"2024-06-29T14:37:22.331887+0000","last_fullsized":"2024-06-29T14:37:22.331887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:55.830301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331674+0000","last_change":"2024-06-29T14:37:21.313558+0000","last_active":"2024-06-29T14:37:22.331674+0000","last_peered":"2024-06-29T14:37:22.331674+0000","last_clean":"2024-06-29T14:37:22.331674+0000","last_became_active":"2024-06-29T14:37:21.313230+0000","last_became_peered":"2024-06-29T14:37:21.313230+0000","last_unstale":"2024-06-29T14:37:22.331674+0000","last_undegraded":"2024-06-29T14:37:22.331674+0000","last_fullsized":"2024-06-29T14:37:22.331674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:28.290552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331612+0000","last_change":"2024-06-29T14:37:20.300262+0000","last_active":"2024-06-29T14:37:22.331612+0000","last_peered":"2024-06-29T14:37:22.331612+0000","last_clean":"2024-06-29T14:37:22.331612+0000","last_became_active":"2024-06-29T14:37:20.299596+0000","last_became_peered":"2024-06-29T14:37:20.299596+0000","last_unstale":"2024-06-29T14:37:22.331612+0000","last_undegraded":"2024-06-29T14:37:22.331612+0000","last_fullsized":"2024-06-29T14:37:22.331612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:42:42.724298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331078+0000","last_change":"2024-06-29T14:37:21.321287+0000","last_active":"2024-06-29T14:37:22.331078+0000","last_peered":"2024-06-29T14:37:22.331078+0000","last_clean":"2024-06-29T14:37:22.331078+0000","last_became_active":"2024-06-29T14:37:21.315884+0000","last_became_peered":"2024-06-29T14:37:21.315884+0000","last_unstale":"2024-06-29T14:37:22.331078+0000","last_undegraded":"2024-06-29T14:37:22.331078+0000","last_fullsized":"2024-06-29T14:37:22.331078+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:52:44.152310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331980+0000","last_change":"2024-06-29T14:37:20.300752+0000","last_active":"2024-06-29T14:37:22.331980+0000","last_peered":"2024-06-29T14:37:22.331980+0000","last_clean":"2024-06-29T14:37:22.331980+0000","last_became_active":"2024-06-29T14:37:20.300440+0000","last_became_peered":"2024-06-29T14:37:20.300440+0000","last_unstale":"2024-06-29T14:37:22.331980+0000","last_undegraded":"2024-06-29T14:37:22.331980+0000","last_fullsized":"2024-06-29T14:37:22.331980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:42.227035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297229+0000","last_change":"2024-06-29T14:37:21.309654+0000","last_active":"2024-06-29T14:37:22.297229+0000","last_peered":"2024-06-29T14:37:22.297229+0000","last_clean":"2024-06-29T14:37:22.297229+0000","last_became_active":"2024-06-29T14:37:21.309282+0000","last_became_peered":"2024-06-29T14:37:21.309282+0000","last_unstale":"2024-06-29T14:37:22.297229+0000","last_undegraded":"2024-06-29T14:37:22.297229+0000","last_fullsized":"2024-06-29T14:37:22.297229+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:45:38.559161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331462+0000","last_change":"2024-06-29T14:37:20.295528+0000","last_active":"2024-06-29T14:37:22.331462+0000","last_peered":"2024-06-29T14:37:22.331462+0000","last_clean":"2024-06-29T14:37:22.331462+0000","last_became_active":"2024-06-29T14:37:20.294656+0000","last_became_peered":"2024-06-29T14:37:20.294656+0000","last_unstale":"2024-06-29T14:37:22.331462+0000","last_undegraded":"2024-06-29T14:37:22.331462+0000","last_fullsized":"2024-06-29T14:37:22.331462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:02.146628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331075+0000","last_change":"2024-06-29T14:37:21.312980+0000","last_active":"2024-06-29T14:37:22.331075+0000","last_peered":"2024-06-29T14:37:22.331075+0000","last_clean":"2024-06-29T14:37:22.331075+0000","last_became_active":"2024-06-29T14:37:21.312654+0000","last_became_peered":"2024-06-29T14:37:21.312654+0000","last_unstale":"2024-06-29T14:37:22.331075+0000","last_undegraded":"2024-06-29T14:37:22.331075+0000","last_fullsized":"2024-06-29T14:37:22.331075+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:25.597063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331646+0000","last_change":"2024-06-29T14:37:20.294982+0000","last_active":"2024-06-29T14:37:22.331646+0000","last_peered":"2024-06-29T14:37:22.331646+0000","last_clean":"2024-06-29T14:37:22.331646+0000","last_became_active":"2024-06-29T14:37:20.294351+0000","last_became_peered":"2024-06-29T14:37:20.294351+0000","last_unstale":"2024-06-29T14:37:22.331646+0000","last_undegraded":"2024-06-29T14:37:22.331646+0000","last_fullsized":"2024-06-29T14:37:22.331646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:24.988008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298872+0000","last_change":"2024-06-29T14:37:21.312743+0000","last_active":"2024-06-29T14:37:22.298872+0000","last_peered":"2024-06-29T14:37:22.298872+0000","last_clean":"2024-06-29T14:37:22.298872+0000","last_became_active":"2024-06-29T14:37:21.312201+0000","last_became_peered":"2024-06-29T14:37:21.312201+0000","last_unstale":"2024-06-29T14:37:22.298872+0000","last_undegraded":"2024-06-29T14:37:22.298872+0000","last_fullsized":"2024-06-29T14:37:22.298872+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:33.504070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331362+0000","last_change":"2024-06-29T14:37:20.293349+0000","last_active":"2024-06-29T14:37:22.331362+0000","last_peered":"2024-06-29T14:37:22.331362+0000","last_clean":"2024-06-29T14:37:22.331362+0000","last_became_active":"2024-06-29T14:37:20.292982+0000","last_became_peered":"2024-06-29T14:37:20.292982+0000","last_unstale":"2024-06-29T14:37:22.331362+0000","last_undegraded":"2024-06-29T14:37:22.331362+0000","last_fullsized":"2024-06-29T14:37:22.331362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:29.149704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331327+0000","last_change":"2024-06-29T14:37:21.316255+0000","last_active":"2024-06-29T14:37:22.331327+0000","last_peered":"2024-06-29T14:37:22.331327+0000","last_clean":"2024-06-29T14:37:22.331327+0000","last_became_active":"2024-06-29T14:37:21.315928+0000","last_became_peered":"2024-06-29T14:37:21.315928+0000","last_unstale":"2024-06-29T14:37:22.331327+0000","last_undegraded":"2024-06-29T14:37:22.331327+0000","last_fullsized":"2024-06-29T14:37:22.331327+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:54:02.988704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298242+0000","last_change":"2024-06-29T14:37:20.296335+0000","last_active":"2024-06-29T14:37:22.298242+0000","last_peered":"2024-06-29T14:37:22.298242+0000","last_clean":"2024-06-29T14:37:22.298242+0000","last_became_active":"2024-06-29T14:37:20.296046+0000","last_became_peered":"2024-06-29T14:37:20.296046+0000","last_unstale":"2024-06-29T14:37:22.298242+0000","last_undegraded":"2024-06-29T14:37:22.298242+0000","last_fullsized":"2024-06-29T14:37:22.298242+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:14.341107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331944+0000","last_change":"2024-06-29T14:37:21.308791+0000","last_active":"2024-06-29T14:37:22.331944+0000","last_peered":"2024-06-29T14:37:22.331944+0000","last_clean":"2024-06-29T14:37:22.331944+0000","last_became_active":"2024-06-29T14:37:21.308479+0000","last_became_peered":"2024-06-29T14:37:21.308479+0000","last_unstale":"2024-06-29T14:37:22.331944+0000","last_undegraded":"2024-06-29T14:37:22.331944+0000","last_fullsized":"2024-06-29T14:37:22.331944+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:17.368735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331181+0000","last_change":"2024-06-29T14:37:20.304108+0000","last_active":"2024-06-29T14:37:22.331181+0000","last_peered":"2024-06-29T14:37:22.331181+0000","last_clean":"2024-06-29T14:37:22.331181+0000","last_became_active":"2024-06-29T14:37:20.303755+0000","last_became_peered":"2024-06-29T14:37:20.303755+0000","last_unstale":"2024-06-29T14:37:22.331181+0000","last_undegraded":"2024-06-29T14:37:22.331181+0000","last_fullsized":"2024-06-29T14:37:22.331181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:47.128695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331056+0000","last_change":"2024-06-29T14:37:21.320528+0000","last_active":"2024-06-29T14:37:22.331056+0000","last_peered":"2024-06-29T14:37:22.331056+0000","last_clean":"2024-06-29T14:37:22.331056+0000","last_became_active":"2024-06-29T14:37:21.320193+0000","last_became_peered":"2024-06-29T14:37:21.320193+0000","last_unstale":"2024-06-29T14:37:22.331056+0000","last_undegraded":"2024-06-29T14:37:22.331056+0000","last_fullsized":"2024-06-29T14:37:22.331056+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:11:10.100908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299235+0000","last_change":"2024-06-29T14:37:20.298814+0000","last_active":"2024-06-29T14:37:22.299235+0000","last_peered":"2024-06-29T14:37:22.299235+0000","last_clean":"2024-06-29T14:37:22.299235+0000","last_became_active":"2024-06-29T14:37:20.298533+0000","last_became_peered":"2024-06-29T14:37:20.298533+0000","last_unstale":"2024-06-29T14:37:22.299235+0000","last_undegraded":"2024-06-29T14:37:22.299235+0000","last_fullsized":"2024-06-29T14:37:22.299235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:04:22.860050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298262+0000","last_change":"2024-06-29T14:37:21.323782+0000","last_active":"2024-06-29T14:37:22.298262+0000","last_peered":"2024-06-29T14:37:22.298262+0000","last_clean":"2024-06-29T14:37:22.298262+0000","last_became_active":"2024-06-29T14:37:21.323446+0000","last_became_peered":"2024-06-29T14:37:21.323446+0000","last_unstale":"2024-06-29T14:37:22.298262+0000","last_undegraded":"2024-06-29T14:37:22.298262+0000","last_fullsized":"2024-06-29T14:37:22.298262+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:25:52.388229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"87'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330758+0000","last_change":"2024-06-29T14:37:20.303709+0000","last_active":"2024-06-29T14:37:22.330758+0000","last_peered":"2024-06-29T14:37:22.330758+0000","last_clean":"2024-06-29T14:37:22.330758+0000","last_became_active":"2024-06-29T14:37:20.303413+0000","last_became_peered":"2024-06-29T14:37:20.303413+0000","last_unstale":"2024-06-29T14:37:22.330758+0000","last_undegraded":"2024-06-29T14:37:22.330758+0000","last_fullsized":"2024-06-29T14:37:22.330758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:37.074235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299084+0000","last_change":"2024-06-29T14:37:21.308810+0000","last_active":"2024-06-29T14:37:22.299084+0000","last_peered":"2024-06-29T14:37:22.299084+0000","last_clean":"2024-06-29T14:37:22.299084+0000","last_became_active":"2024-06-29T14:37:21.308487+0000","last_became_peered":"2024-06-29T14:37:21.308487+0000","last_unstale":"2024-06-29T14:37:22.299084+0000","last_undegraded":"2024-06-29T14:37:22.299084+0000","last_fullsized":"2024-06-29T14:37:22.299084+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:51.302471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"87'6","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332141+0000","last_change":"2024-06-29T14:37:20.298014+0000","last_active":"2024-06-29T14:37:22.332141+0000","last_peered":"2024-06-29T14:37:22.332141+0000","last_clean":"2024-06-29T14:37:22.332141+0000","last_became_active":"2024-06-29T14:37:20.297505+0000","last_became_peered":"2024-06-29T14:37:20.297505+0000","last_unstale":"2024-06-29T14:37:22.332141+0000","last_undegraded":"2024-06-29T14:37:22.332141+0000","last_fullsized":"2024-06-29T14:37:22.332141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:54.029809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297985+0000","last_change":"2024-06-29T14:37:21.314656+0000","last_active":"2024-06-29T14:37:22.297985+0000","last_peered":"2024-06-29T14:37:22.297985+0000","last_clean":"2024-06-29T14:37:22.297985+0000","last_became_active":"2024-06-29T14:37:21.314116+0000","last_became_peered":"2024-06-29T14:37:21.314116+0000","last_unstale":"2024-06-29T14:37:22.297985+0000","last_undegraded":"2024-06-29T14:37:22.297985+0000","last_fullsized":"2024-06-29T14:37:22.297985+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:05.918168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330959+0000","last_change":"2024-06-29T14:37:20.296074+0000","last_active":"2024-06-29T14:37:22.330959+0000","last_peered":"2024-06-29T14:37:22.330959+0000","last_clean":"2024-06-29T14:37:22.330959+0000","last_became_active":"2024-06-29T14:37:20.294396+0000","last_became_peered":"2024-06-29T14:37:20.294396+0000","last_unstale":"2024-06-29T14:37:22.330959+0000","last_undegraded":"2024-06-29T14:37:22.330959+0000","last_fullsized":"2024-06-29T14:37:22.330959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:12.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315615+0000","last_change":"2024-06-29T14:37:21.322914+0000","last_active":"2024-06-29T14:37:22.315615+0000","last_peered":"2024-06-29T14:37:22.315615+0000","last_clean":"2024-06-29T14:37:22.315615+0000","last_became_active":"2024-06-29T14:37:21.322582+0000","last_became_peered":"2024-06-29T14:37:21.322582+0000","last_unstale":"2024-06-29T14:37:22.315615+0000","last_undegraded":"2024-06-29T14:37:22.315615+0000","last_fullsized":"2024-06-29T14:37:22.315615+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:20.337643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315656+0000","last_change":"2024-06-29T14:37:20.302109+0000","last_active":"2024-06-29T14:37:22.315656+0000","last_peered":"2024-06-29T14:37:22.315656+0000","last_clean":"2024-06-29T14:37:22.315656+0000","last_became_active":"2024-06-29T14:37:20.301765+0000","last_became_peered":"2024-06-29T14:37:20.301765+0000","last_unstale":"2024-06-29T14:37:22.315656+0000","last_undegraded":"2024-06-29T14:37:22.315656+0000","last_fullsized":"2024-06-29T14:37:22.315656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:34.482542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299100+0000","last_change":"2024-06-29T14:37:21.304641+0000","last_active":"2024-06-29T14:37:22.299100+0000","last_peered":"2024-06-29T14:37:22.299100+0000","last_clean":"2024-06-29T14:37:22.299100+0000","last_became_active":"2024-06-29T14:37:21.304328+0000","last_became_peered":"2024-06-29T14:37:21.304328+0000","last_unstale":"2024-06-29T14:37:22.299100+0000","last_undegraded":"2024-06-29T14:37:22.299100+0000","last_fullsized":"2024-06-29T14:37:22.299100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:09:49.985745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330944+0000","last_change":"2024-06-29T14:37:20.298474+0000","last_active":"2024-06-29T14:37:22.330944+0000","last_peered":"2024-06-29T14:37:22.330944+0000","last_clean":"2024-06-29T14:37:22.330944+0000","last_became_active":"2024-06-29T14:37:20.298155+0000","last_became_peered":"2024-06-29T14:37:20.298155+0000","last_unstale":"2024-06-29T14:37:22.330944+0000","last_undegraded":"2024-06-29T14:37:22.330944+0000","last_fullsized":"2024-06-29T14:37:22.330944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:32.529040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331334+0000","last_change":"2024-06-29T14:37:21.316244+0000","last_active":"2024-06-29T14:37:22.331334+0000","last_peered":"2024-06-29T14:37:22.331334+0000","last_clean":"2024-06-29T14:37:22.331334+0000","last_became_active":"2024-06-29T14:37:21.315803+0000","last_became_peered":"2024-06-29T14:37:21.315803+0000","last_unstale":"2024-06-29T14:37:22.331334+0000","last_undegraded":"2024-06-29T14:37:22.331334+0000","last_fullsized":"2024-06-29T14:37:22.331334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:37.339449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"87'8","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298132+0000","last_change":"2024-06-29T14:37:20.297924+0000","last_active":"2024-06-29T14:37:22.298132+0000","last_peered":"2024-06-29T14:37:22.298132+0000","last_clean":"2024-06-29T14:37:22.298132+0000","last_became_active":"2024-06-29T14:37:20.297604+0000","last_became_peered":"2024-06-29T14:37:20.297604+0000","last_unstale":"2024-06-29T14:37:22.298132+0000","last_undegraded":"2024-06-29T14:37:22.298132+0000","last_fullsized":"2024-06-29T14:37:22.298132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:45:15.810337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330972+0000","last_change":"2024-06-29T14:37:21.312532+0000","last_active":"2024-06-29T14:37:22.330972+0000","last_peered":"2024-06-29T14:37:22.330972+0000","last_clean":"2024-06-29T14:37:22.330972+0000","last_became_active":"2024-06-29T14:37:21.312192+0000","last_became_peered":"2024-06-29T14:37:21.312192+0000","last_unstale":"2024-06-29T14:37:22.330972+0000","last_undegraded":"2024-06-29T14:37:22.330972+0000","last_fullsized":"2024-06-29T14:37:22.330972+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:43.362310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315808+0000","last_change":"2024-06-29T14:37:20.304834+0000","last_active":"2024-06-29T14:37:22.315808+0000","last_peered":"2024-06-29T14:37:22.315808+0000","last_clean":"2024-06-29T14:37:22.315808+0000","last_became_active":"2024-06-29T14:37:20.304472+0000","last_became_peered":"2024-06-29T14:37:20.304472+0000","last_unstale":"2024-06-29T14:37:22.315808+0000","last_undegraded":"2024-06-29T14:37:22.315808+0000","last_fullsized":"2024-06-29T14:37:22.315808+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:29.149704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298753+0000","last_change":"2024-06-29T14:37:21.314575+0000","last_active":"2024-06-29T14:37:22.298753+0000","last_peered":"2024-06-29T14:37:22.298753+0000","last_clean":"2024-06-29T14:37:22.298753+0000","last_became_active":"2024-06-29T14:37:21.314257+0000","last_became_peered":"2024-06-29T14:37:21.314257+0000","last_unstale":"2024-06-29T14:37:22.298753+0000","last_undegraded":"2024-06-29T14:37:22.298753+0000","last_fullsized":"2024-06-29T14:37:22.298753+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:22.177194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331145+0000","last_change":"2024-06-29T14:37:20.297122+0000","last_active":"2024-06-29T14:37:22.331145+0000","last_peered":"2024-06-29T14:37:22.331145+0000","last_clean":"2024-06-29T14:37:22.331145+0000","last_became_active":"2024-06-29T14:37:20.296799+0000","last_became_peered":"2024-06-29T14:37:20.296799+0000","last_unstale":"2024-06-29T14:37:22.331145+0000","last_undegraded":"2024-06-29T14:37:22.331145+0000","last_fullsized":"2024-06-29T14:37:22.331145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:14.404396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299292+0000","last_change":"2024-06-29T14:37:21.325669+0000","last_active":"2024-06-29T14:37:22.299292+0000","last_peered":"2024-06-29T14:37:22.299292+0000","last_clean":"2024-06-29T14:37:22.299292+0000","last_became_active":"2024-06-29T14:37:21.325363+0000","last_became_peered":"2024-06-29T14:37:21.325363+0000","last_unstale":"2024-06-29T14:37:22.299292+0000","last_undegraded":"2024-06-29T14:37:22.299292+0000","last_fullsized":"2024-06-29T14:37:22.299292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:48.654672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330917+0000","last_change":"2024-06-29T14:37:20.308208+0000","last_active":"2024-06-29T14:37:22.330917+0000","last_peered":"2024-06-29T14:37:22.330917+0000","last_clean":"2024-06-29T14:37:22.330917+0000","last_became_active":"2024-06-29T14:37:20.307910+0000","last_became_peered":"2024-06-29T14:37:20.307910+0000","last_unstale":"2024-06-29T14:37:22.330917+0000","last_undegraded":"2024-06-29T14:37:22.330917+0000","last_fullsized":"2024-06-29T14:37:22.330917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:50.043421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298488+0000","last_change":"2024-06-29T14:37:21.310399+0000","last_active":"2024-06-29T14:37:22.298488+0000","last_peered":"2024-06-29T14:37:22.298488+0000","last_clean":"2024-06-29T14:37:22.298488+0000","last_became_active":"2024-06-29T14:37:21.306777+0000","last_became_peered":"2024-06-29T14:37:21.306777+0000","last_unstale":"2024-06-29T14:37:22.298488+0000","last_undegraded":"2024-06-29T14:37:22.298488+0000","last_fullsized":"2024-06-29T14:37:22.298488+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:40:09.321136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332009+0000","last_change":"2024-06-29T14:37:20.296918+0000","last_active":"2024-06-29T14:37:22.332009+0000","last_peered":"2024-06-29T14:37:22.332009+0000","last_clean":"2024-06-29T14:37:22.332009+0000","last_became_active":"2024-06-29T14:37:20.296570+0000","last_became_peered":"2024-06-29T14:37:20.296570+0000","last_unstale":"2024-06-29T14:37:22.332009+0000","last_undegraded":"2024-06-29T14:37:22.332009+0000","last_fullsized":"2024-06-29T14:37:22.332009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:25.104665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330991+0000","last_change":"2024-06-29T14:37:21.313296+0000","last_active":"2024-06-29T14:37:22.330991+0000","last_peered":"2024-06-29T14:37:22.330991+0000","last_clean":"2024-06-29T14:37:22.330991+0000","last_became_active":"2024-06-29T14:37:21.312985+0000","last_became_peered":"2024-06-29T14:37:21.312985+0000","last_unstale":"2024-06-29T14:37:22.330991+0000","last_undegraded":"2024-06-29T14:37:22.330991+0000","last_fullsized":"2024-06-29T14:37:22.330991+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:16.740545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330871+0000","last_change":"2024-06-29T14:37:20.297386+0000","last_active":"2024-06-29T14:37:22.330871+0000","last_peered":"2024-06-29T14:37:22.330871+0000","last_clean":"2024-06-29T14:37:22.330871+0000","last_became_active":"2024-06-29T14:37:20.292152+0000","last_became_peered":"2024-06-29T14:37:20.292152+0000","last_unstale":"2024-06-29T14:37:22.330871+0000","last_undegraded":"2024-06-29T14:37:22.330871+0000","last_fullsized":"2024-06-29T14:37:22.330871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:26:56.590026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315627+0000","last_change":"2024-06-29T14:37:21.319010+0000","last_active":"2024-06-29T14:37:22.315627+0000","last_peered":"2024-06-29T14:37:22.315627+0000","last_clean":"2024-06-29T14:37:22.315627+0000","last_became_active":"2024-06-29T14:37:21.318578+0000","last_became_peered":"2024-06-29T14:37:21.318578+0000","last_unstale":"2024-06-29T14:37:22.315627+0000","last_undegraded":"2024-06-29T14:37:22.315627+0000","last_fullsized":"2024-06-29T14:37:22.315627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:54:02.988704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315866+0000","last_change":"2024-06-29T14:37:20.301660+0000","last_active":"2024-06-29T14:37:22.315866+0000","last_peered":"2024-06-29T14:37:22.315866+0000","last_clean":"2024-06-29T14:37:22.315866+0000","last_became_active":"2024-06-29T14:37:20.301346+0000","last_became_peered":"2024-06-29T14:37:20.301346+0000","last_unstale":"2024-06-29T14:37:22.315866+0000","last_undegraded":"2024-06-29T14:37:22.315866+0000","last_fullsized":"2024-06-29T14:37:22.315866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:57.474408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299087+0000","last_change":"2024-06-29T14:37:21.312489+0000","last_active":"2024-06-29T14:37:22.299087+0000","last_peered":"2024-06-29T14:37:22.299087+0000","last_clean":"2024-06-29T14:37:22.299087+0000","last_became_active":"2024-06-29T14:37:21.311809+0000","last_became_peered":"2024-06-29T14:37:21.311809+0000","last_unstale":"2024-06-29T14:37:22.299087+0000","last_undegraded":"2024-06-29T14:37:22.299087+0000","last_fullsized":"2024-06-29T14:37:22.299087+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:11.052839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331186+0000","last_change":"2024-06-29T14:37:20.300564+0000","last_active":"2024-06-29T14:37:22.331186+0000","last_peered":"2024-06-29T14:37:22.331186+0000","last_clean":"2024-06-29T14:37:22.331186+0000","last_became_active":"2024-06-29T14:37:20.300232+0000","last_became_peered":"2024-06-29T14:37:20.300232+0000","last_unstale":"2024-06-29T14:37:22.331186+0000","last_undegraded":"2024-06-29T14:37:22.331186+0000","last_fullsized":"2024-06-29T14:37:22.331186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:23.430715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331913+0000","last_change":"2024-06-29T14:37:21.316305+0000","last_active":"2024-06-29T14:37:22.331913+0000","last_peered":"2024-06-29T14:37:22.331913+0000","last_clean":"2024-06-29T14:37:22.331913+0000","last_became_active":"2024-06-29T14:37:21.315989+0000","last_became_peered":"2024-06-29T14:37:21.315989+0000","last_unstale":"2024-06-29T14:37:22.331913+0000","last_undegraded":"2024-06-29T14:37:22.331913+0000","last_fullsized":"2024-06-29T14:37:22.331913+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:07:12.439421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.316004+0000","last_change":"2024-06-29T14:37:20.297706+0000","last_active":"2024-06-29T14:37:22.316004+0000","last_peered":"2024-06-29T14:37:22.316004+0000","last_clean":"2024-06-29T14:37:22.316004+0000","last_became_active":"2024-06-29T14:37:20.297369+0000","last_became_peered":"2024-06-29T14:37:20.297369+0000","last_unstale":"2024-06-29T14:37:22.316004+0000","last_undegraded":"2024-06-29T14:37:22.316004+0000","last_fullsized":"2024-06-29T14:37:22.316004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:25.104665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297348+0000","last_change":"2024-06-29T14:37:21.315791+0000","last_active":"2024-06-29T14:37:22.297348+0000","last_peered":"2024-06-29T14:37:22.297348+0000","last_clean":"2024-06-29T14:37:22.297348+0000","last_became_active":"2024-06-29T14:37:21.315481+0000","last_became_peered":"2024-06-29T14:37:21.315481+0000","last_unstale":"2024-06-29T14:37:22.297348+0000","last_undegraded":"2024-06-29T14:37:22.297348+0000","last_fullsized":"2024-06-29T14:37:22.297348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:47:55.915681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"88'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.317351+0000","last_change":"2024-06-29T14:37:20.295958+0000","last_active":"2024-06-29T14:37:22.317351+0000","last_peered":"2024-06-29T14:37:22.317351+0000","last_clean":"2024-06-29T14:37:22.317351+0000","last_became_active":"2024-06-29T14:37:20.295598+0000","last_became_peered":"2024-06-29T14:37:20.295598+0000","last_unstale":"2024-06-29T14:37:22.317351+0000","last_undegraded":"2024-06-29T14:37:22.317351+0000","last_fullsized":"2024-06-29T14:37:22.317351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:02.146628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315970+0000","last_change":"2024-06-29T14:37:21.322411+0000","last_active":"2024-06-29T14:37:22.315970+0000","last_peered":"2024-06-29T14:37:22.315970+0000","last_clean":"2024-06-29T14:37:22.315970+0000","last_became_active":"2024-06-29T14:37:21.322097+0000","last_became_peered":"2024-06-29T14:37:21.322097+0000","last_unstale":"2024-06-29T14:37:22.315970+0000","last_undegraded":"2024-06-29T14:37:22.315970+0000","last_fullsized":"2024-06-29T14:37:22.315970+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:22.750933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331728+0000","last_change":"2024-06-29T14:37:20.298730+0000","last_active":"2024-06-29T14:37:22.331728+0000","last_peered":"2024-06-29T14:37:22.331728+0000","last_clean":"2024-06-29T14:37:22.331728+0000","last_became_active":"2024-06-29T14:37:20.298362+0000","last_became_peered":"2024-06-29T14:37:20.298362+0000","last_unstale":"2024-06-29T14:37:22.331728+0000","last_undegraded":"2024-06-29T14:37:22.331728+0000","last_fullsized":"2024-06-29T14:37:22.331728+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:03.760040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298664+0000","last_change":"2024-06-29T14:37:21.302425+0000","last_active":"2024-06-29T14:37:22.298664+0000","last_peered":"2024-06-29T14:37:22.298664+0000","last_clean":"2024-06-29T14:37:22.298664+0000","last_became_active":"2024-06-29T14:37:21.302091+0000","last_became_peered":"2024-06-29T14:37:21.302091+0000","last_unstale":"2024-06-29T14:37:22.298664+0000","last_undegraded":"2024-06-29T14:37:22.298664+0000","last_fullsized":"2024-06-29T14:37:22.298664+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:05.918168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332176+0000","last_change":"2024-06-29T14:37:20.298015+0000","last_active":"2024-06-29T14:37:22.332176+0000","last_peered":"2024-06-29T14:37:22.332176+0000","last_clean":"2024-06-29T14:37:22.332176+0000","last_became_active":"2024-06-29T14:37:20.297677+0000","last_became_peered":"2024-06-29T14:37:20.297677+0000","last_unstale":"2024-06-29T14:37:22.332176+0000","last_undegraded":"2024-06-29T14:37:22.332176+0000","last_fullsized":"2024-06-29T14:37:22.332176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:29:24.594454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298104+0000","last_change":"2024-06-29T14:37:21.309531+0000","last_active":"2024-06-29T14:37:22.298104+0000","last_peered":"2024-06-29T14:37:22.298104+0000","last_clean":"2024-06-29T14:37:22.298104+0000","last_became_active":"2024-06-29T14:37:21.309209+0000","last_became_peered":"2024-06-29T14:37:21.309209+0000","last_unstale":"2024-06-29T14:37:22.298104+0000","last_undegraded":"2024-06-29T14:37:22.298104+0000","last_fullsized":"2024-06-29T14:37:22.298104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:44.697323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331089+0000","last_change":"2024-06-29T14:37:20.300925+0000","last_active":"2024-06-29T14:37:22.331089+0000","last_peered":"2024-06-29T14:37:22.331089+0000","last_clean":"2024-06-29T14:37:22.331089+0000","last_became_active":"2024-06-29T14:37:20.300594+0000","last_became_peered":"2024-06-29T14:37:20.300594+0000","last_unstale":"2024-06-29T14:37:22.331089+0000","last_undegraded":"2024-06-29T14:37:22.331089+0000","last_fullsized":"2024-06-29T14:37:22.331089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:43.510209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298292+0000","last_change":"2024-06-29T14:37:21.314337+0000","last_active":"2024-06-29T14:37:22.298292+0000","last_peered":"2024-06-29T14:37:22.298292+0000","last_clean":"2024-06-29T14:37:22.298292+0000","last_became_active":"2024-06-29T14:37:21.313936+0000","last_became_peered":"2024-06-29T14:37:21.313936+0000","last_unstale":"2024-06-29T14:37:22.298292+0000","last_undegraded":"2024-06-29T14:37:22.298292+0000","last_fullsized":"2024-06-29T14:37:22.298292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:33.504070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331656+0000","last_change":"2024-06-29T14:37:20.301959+0000","last_active":"2024-06-29T14:37:22.331656+0000","last_peered":"2024-06-29T14:37:22.331656+0000","last_clean":"2024-06-29T14:37:22.331656+0000","last_became_active":"2024-06-29T14:37:20.301503+0000","last_became_peered":"2024-06-29T14:37:20.301503+0000","last_unstale":"2024-06-29T14:37:22.331656+0000","last_undegraded":"2024-06-29T14:37:22.331656+0000","last_fullsized":"2024-06-29T14:37:22.331656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:05.989680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331480+0000","last_change":"2024-06-29T14:37:21.312540+0000","last_active":"2024-06-29T14:37:22.331480+0000","last_peered":"2024-06-29T14:37:22.331480+0000","last_clean":"2024-06-29T14:37:22.331480+0000","last_became_active":"2024-06-29T14:37:21.312184+0000","last_became_peered":"2024-06-29T14:37:21.312184+0000","last_unstale":"2024-06-29T14:37:22.331480+0000","last_undegraded":"2024-06-29T14:37:22.331480+0000","last_fullsized":"2024-06-29T14:37:22.331480+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:40.914474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331482+0000","last_change":"2024-06-29T14:37:20.292457+0000","last_active":"2024-06-29T14:37:22.331482+0000","last_peered":"2024-06-29T14:37:22.331482+0000","last_clean":"2024-06-29T14:37:22.331482+0000","last_became_active":"2024-06-29T14:37:20.292042+0000","last_became_peered":"2024-06-29T14:37:20.292042+0000","last_unstale":"2024-06-29T14:37:22.331482+0000","last_undegraded":"2024-06-29T14:37:22.331482+0000","last_fullsized":"2024-06-29T14:37:22.331482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:43.310708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297425+0000","last_change":"2024-06-29T14:37:21.318844+0000","last_active":"2024-06-29T14:37:22.297425+0000","last_peered":"2024-06-29T14:37:22.297425+0000","last_clean":"2024-06-29T14:37:22.297425+0000","last_became_active":"2024-06-29T14:37:21.318508+0000","last_became_peered":"2024-06-29T14:37:21.318508+0000","last_unstale":"2024-06-29T14:37:22.297425+0000","last_undegraded":"2024-06-29T14:37:22.297425+0000","last_fullsized":"2024-06-29T14:37:22.297425+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:29.326926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330878+0000","last_change":"2024-06-29T14:37:20.296547+0000","last_active":"2024-06-29T14:37:22.330878+0000","last_peered":"2024-06-29T14:37:22.330878+0000","last_clean":"2024-06-29T14:37:22.330878+0000","last_became_active":"2024-06-29T14:37:20.296259+0000","last_became_peered":"2024-06-29T14:37:20.296259+0000","last_unstale":"2024-06-29T14:37:22.330878+0000","last_undegraded":"2024-06-29T14:37:22.330878+0000","last_fullsized":"2024-06-29T14:37:22.330878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:13:59.493049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299189+0000","last_change":"2024-06-29T14:37:21.310452+0000","last_active":"2024-06-29T14:37:22.299189+0000","last_peered":"2024-06-29T14:37:22.299189+0000","last_clean":"2024-06-29T14:37:22.299189+0000","last_became_active":"2024-06-29T14:37:21.308946+0000","last_became_peered":"2024-06-29T14:37:21.308946+0000","last_unstale":"2024-06-29T14:37:22.299189+0000","last_undegraded":"2024-06-29T14:37:22.299189+0000","last_fullsized":"2024-06-29T14:37:22.299189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:51.302471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330864+0000","last_change":"2024-06-29T14:37:20.294167+0000","last_active":"2024-06-29T14:37:22.330864+0000","last_peered":"2024-06-29T14:37:22.330864+0000","last_clean":"2024-06-29T14:37:22.330864+0000","last_became_active":"2024-06-29T14:37:20.293843+0000","last_became_peered":"2024-06-29T14:37:20.293843+0000","last_unstale":"2024-06-29T14:37:22.330864+0000","last_undegraded":"2024-06-29T14:37:22.330864+0000","last_fullsized":"2024-06-29T14:37:22.330864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:14:03.939141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":343,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330733+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:37:22.330733+0000","last_peered":"2024-06-29T14:37:22.330733+0000","last_clean":"2024-06-29T14:37:22.330733+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:37:22.330733+0000","last_undegraded":"2024-06-29T14:37:22.330733+0000","last_fullsized":"2024-06-29T14:37:22.330733+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542333,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59180,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312660,"statfs":{"total":96636764160,"available":96576163840,"internally_reserved":0,"allocated":1671168,"data_stored":1125172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542334,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":1064960,"data_stored":534062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542333,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50272,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321568,"statfs":{"total":96636764160,"available":96585285632,"internally_reserved":0,"allocated":1064960,"data_stored":534050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339620,"statfs":{"total":96636764160,"available":96603770880,"internally_reserved":0,"allocated":1060864,"data_stored":528948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509630,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":1060864,"data_stored":534028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575038,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58588,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313252,"statfs":{"total":96636764160,"available":96576770048,"internally_reserved":0,"allocated":1060864,"data_stored":534028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509629,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":536404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509631,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1654784,"data_stored":1124396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:37:25.394 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:37:25.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.551+0000 7f8648a3f640 1 -- 172.21.15.59:0/2225489049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8630004de0 msgr2=0x7f86300052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.551+0000 7f8648a3f640 1 --2- 172.21.15.59:0/2225489049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8630004de0 0x7f86300052b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f863c001830 tx=0x7f8630005830 comp rx=0 tx=0).stop 2024-06-29T14:37:25.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.551+0000 7f8642d9d640 1 -- 172.21.15.59:0/2225489049 reap_dead start 2024-06-29T14:37:25.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.551+0000 7f8648a3f640 1 -- 172.21.15.59:0/2225489049 shutdown_connections 2024-06-29T14:37:25.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.551+0000 7f8648a3f640 1 -- 172.21.15.59:0/2225489049 >> 172.21.15.59:0/2225489049 conn(0x7f86440b6fb0 msgr2=0x7f86440b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.551+0000 7f8648a3f640 1 -- 172.21.15.59:0/2225489049 shutdown_connections 2024-06-29T14:37:25.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.551+0000 7f8648a3f640 1 -- 172.21.15.59:0/2225489049 wait complete. 2024-06-29T14:37:25.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.553+0000 7f8648a3f640 1 Processor -- start 2024-06-29T14:37:25.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.553+0000 7f8648a3f640 1 -- start start 2024-06-29T14:37:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8648a3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8630004de0 0x7f864415e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8648a3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f864413d610 0x7f8644155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8648a3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f864413da20 0x7f86441563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8648a3f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f864413f150 con 0x7f864413d610 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8648a3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f864413efd0 con 0x7f864413da20 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8648a3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f86440b2530 con 0x7f8630004de0 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8641d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f864413d610 0x7f8644155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8641d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f864413d610 0x7f8644155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58766/0 (socket says 172.21.15.59:58766) 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8641d9b640 1 -- 172.21.15.59:0/3296939602 learned_addr learned my addr 172.21.15.59:0/3296939602 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f8642d9d640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f864413da20 0x7f86441563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.554+0000 7f864259c640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8630004de0 0x7f864415e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f8641d9b640 1 -- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8630004de0 msgr2=0x7f864415e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f8641d9b640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8630004de0 0x7f864415e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f8641d9b640 1 -- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f864413da20 msgr2=0x7f86441563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f8641d9b640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f864413da20 0x7f86441563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f8641d9b640 1 -- 172.21.15.59:0/3296939602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8644152790 con 0x7f864413d610 2024-06-29T14:37:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f8641d9b640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f864413d610 0x7f8644155eb0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f862c011ae0 tx=0x7f862c009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f863affd640 1 -- 172.21.15.59:0/3296939602 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c01e070 con 0x7f864413d610 2024-06-29T14:37:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f863affd640 1 -- 172.21.15.59:0/3296939602 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f862c004460 con 0x7f864413d610 2024-06-29T14:37:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8644152a70 con 0x7f864413d610 2024-06-29T14:37:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f863affd640 1 -- 172.21.15.59:0/3296939602 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c00fc60 con 0x7f864413d610 2024-06-29T14:37:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.555+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8644153030 con 0x7f864413d610 2024-06-29T14:37:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.556+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86440b56d0 con 0x7f864413d610 2024-06-29T14:37:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.557+0000 7f863affd640 1 -- 172.21.15.59:0/3296939602 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f862c021070 con 0x7f864413d610 2024-06-29T14:37:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.557+0000 7f863affd640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f86240763f0 0x7f8624078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.557+0000 7f863affd640 1 -- 172.21.15.59:0/3296939602 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f862c098580 con 0x7f864413d610 2024-06-29T14:37:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.557+0000 7f864259c640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f86240763f0 0x7f8624078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.558+0000 7f864259c640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f86240763f0 0x7f8624078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86340015c0 tx=0x7f86340012f0 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.562+0000 7f863affd640 1 -- 172.21.15.59:0/3296939602 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f862c061450 con 0x7f864413d610 2024-06-29T14:37:25.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.750+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 --> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f86440b5b10 con 0x7f86240763f0 2024-06-29T14:37:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.755+0000 7f863affd640 1 -- 172.21.15.59:0/3296939602 <== mgr.6316 v2:172.21.15.153:6832/4269929333 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252032 (secure 0 0 0) 0x7f86440b5b10 con 0x7f86240763f0 2024-06-29T14:37:25.755 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f86240763f0 msgr2=0x7f8624078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f86240763f0 0x7f8624078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86340015c0 tx=0x7f86340012f0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f864413d610 msgr2=0x7f8644155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f864413d610 0x7f8644155eb0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f862c011ae0 tx=0x7f862c009e00 comp rx=0 tx=0).stop 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 shutdown_connections 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f86240763f0 0x7f8624078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f864413da20 0x7f86441563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f864413d610 0x7f8644155eb0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 --2- 172.21.15.59:0/3296939602 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8630004de0 0x7f864415e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 >> 172.21.15.59:0/3296939602 conn(0x7f86440b6fb0 msgr2=0x7f86440ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.758+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 shutdown_connections 2024-06-29T14:37:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.759+0000 7f8648a3f640 1 -- 172.21.15.59:0/3296939602 wait complete. 2024-06-29T14:37:25.767 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-06-29T14:37:24.799111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":194,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329752,"kb_used_data":9500,"kb_used_omap":12,"kb_used_meta":320115,"kb_avail":754644968,"statfs":{"total":773094113280,"available":772756447232,"internally_reserved":0,"allocated":9728000,"data_stored":5451088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":327798322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.524256"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332197+0000","last_change":"2024-06-29T14:37:21.325479+0000","last_active":"2024-06-29T14:37:22.332197+0000","last_peered":"2024-06-29T14:37:22.332197+0000","last_clean":"2024-06-29T14:37:22.332197+0000","last_became_active":"2024-06-29T14:37:21.325172+0000","last_became_peered":"2024-06-29T14:37:21.325172+0000","last_unstale":"2024-06-29T14:37:22.332197+0000","last_undegraded":"2024-06-29T14:37:22.332197+0000","last_fullsized":"2024-06-29T14:37:22.332197+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:36.065136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331851+0000","last_change":"2024-06-29T14:37:21.314109+0000","last_active":"2024-06-29T14:37:22.331851+0000","last_peered":"2024-06-29T14:37:22.331851+0000","last_clean":"2024-06-29T14:37:22.331851+0000","last_became_active":"2024-06-29T14:37:21.313798+0000","last_became_peered":"2024-06-29T14:37:21.313798+0000","last_unstale":"2024-06-29T14:37:22.331851+0000","last_undegraded":"2024-06-29T14:37:22.331851+0000","last_fullsized":"2024-06-29T14:37:22.331851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:21:40.130162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330797+0000","last_change":"2024-06-29T14:37:21.314458+0000","last_active":"2024-06-29T14:37:22.330797+0000","last_peered":"2024-06-29T14:37:22.330797+0000","last_clean":"2024-06-29T14:37:22.330797+0000","last_became_active":"2024-06-29T14:37:21.314145+0000","last_became_peered":"2024-06-29T14:37:21.314145+0000","last_unstale":"2024-06-29T14:37:22.330797+0000","last_undegraded":"2024-06-29T14:37:22.330797+0000","last_fullsized":"2024-06-29T14:37:22.330797+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:06.315908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299576+0000","last_change":"2024-06-29T14:37:21.318960+0000","last_active":"2024-06-29T14:37:22.299576+0000","last_peered":"2024-06-29T14:37:22.299576+0000","last_clean":"2024-06-29T14:37:22.299576+0000","last_became_active":"2024-06-29T14:37:21.316698+0000","last_became_peered":"2024-06-29T14:37:21.316698+0000","last_unstale":"2024-06-29T14:37:22.299576+0000","last_undegraded":"2024-06-29T14:37:22.299576+0000","last_fullsized":"2024-06-29T14:37:22.299576+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:32.698381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298991+0000","last_change":"2024-06-29T14:37:21.314305+0000","last_active":"2024-06-29T14:37:22.298991+0000","last_peered":"2024-06-29T14:37:22.298991+0000","last_clean":"2024-06-29T14:37:22.298991+0000","last_became_active":"2024-06-29T14:37:21.314014+0000","last_became_peered":"2024-06-29T14:37:21.314014+0000","last_unstale":"2024-06-29T14:37:22.298991+0000","last_undegraded":"2024-06-29T14:37:22.298991+0000","last_fullsized":"2024-06-29T14:37:22.298991+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:21.787216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331089+0000","last_change":"2024-06-29T14:37:21.329594+0000","last_active":"2024-06-29T14:37:22.331089+0000","last_peered":"2024-06-29T14:37:22.331089+0000","last_clean":"2024-06-29T14:37:22.331089+0000","last_became_active":"2024-06-29T14:37:21.329289+0000","last_became_peered":"2024-06-29T14:37:21.329289+0000","last_unstale":"2024-06-29T14:37:22.331089+0000","last_undegraded":"2024-06-29T14:37:22.331089+0000","last_fullsized":"2024-06-29T14:37:22.331089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:37:58.714617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297089+0000","last_change":"2024-06-29T14:37:21.316962+0000","last_active":"2024-06-29T14:37:22.297089+0000","last_peered":"2024-06-29T14:37:22.297089+0000","last_clean":"2024-06-29T14:37:22.297089+0000","last_became_active":"2024-06-29T14:37:21.316670+0000","last_became_peered":"2024-06-29T14:37:21.316670+0000","last_unstale":"2024-06-29T14:37:22.297089+0000","last_undegraded":"2024-06-29T14:37:22.297089+0000","last_fullsized":"2024-06-29T14:37:22.297089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:34.422922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315653+0000","last_change":"2024-06-29T14:37:21.318221+0000","last_active":"2024-06-29T14:37:22.315653+0000","last_peered":"2024-06-29T14:37:22.315653+0000","last_clean":"2024-06-29T14:37:22.315653+0000","last_became_active":"2024-06-29T14:37:21.317883+0000","last_became_peered":"2024-06-29T14:37:21.317883+0000","last_unstale":"2024-06-29T14:37:22.315653+0000","last_undegraded":"2024-06-29T14:37:22.315653+0000","last_fullsized":"2024-06-29T14:37:22.315653+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:23.160262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331229+0000","last_change":"2024-06-29T14:37:21.312937+0000","last_active":"2024-06-29T14:37:22.331229+0000","last_peered":"2024-06-29T14:37:22.331229+0000","last_clean":"2024-06-29T14:37:22.331229+0000","last_became_active":"2024-06-29T14:37:21.312610+0000","last_became_peered":"2024-06-29T14:37:21.312610+0000","last_unstale":"2024-06-29T14:37:22.331229+0000","last_undegraded":"2024-06-29T14:37:22.331229+0000","last_fullsized":"2024-06-29T14:37:22.331229+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:17:23.261516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331153+0000","last_change":"2024-06-29T14:37:21.321802+0000","last_active":"2024-06-29T14:37:22.331153+0000","last_peered":"2024-06-29T14:37:22.331153+0000","last_clean":"2024-06-29T14:37:22.331153+0000","last_became_active":"2024-06-29T14:37:21.321487+0000","last_became_peered":"2024-06-29T14:37:21.321487+0000","last_unstale":"2024-06-29T14:37:22.331153+0000","last_undegraded":"2024-06-29T14:37:22.331153+0000","last_fullsized":"2024-06-29T14:37:22.331153+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:45:29.050996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298410+0000","last_change":"2024-06-29T14:37:21.311941+0000","last_active":"2024-06-29T14:37:22.298410+0000","last_peered":"2024-06-29T14:37:22.298410+0000","last_clean":"2024-06-29T14:37:22.298410+0000","last_became_active":"2024-06-29T14:37:21.311617+0000","last_became_peered":"2024-06-29T14:37:21.311617+0000","last_unstale":"2024-06-29T14:37:22.298410+0000","last_undegraded":"2024-06-29T14:37:22.298410+0000","last_fullsized":"2024-06-29T14:37:22.298410+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:21.787216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315916+0000","last_change":"2024-06-29T14:37:21.328427+0000","last_active":"2024-06-29T14:37:22.315916+0000","last_peered":"2024-06-29T14:37:22.315916+0000","last_clean":"2024-06-29T14:37:22.315916+0000","last_became_active":"2024-06-29T14:37:21.328087+0000","last_became_peered":"2024-06-29T14:37:21.328087+0000","last_unstale":"2024-06-29T14:37:22.315916+0000","last_undegraded":"2024-06-29T14:37:22.315916+0000","last_fullsized":"2024-06-29T14:37:22.315916+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:24.144680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315753+0000","last_change":"2024-06-29T14:37:21.316159+0000","last_active":"2024-06-29T14:37:22.315753+0000","last_peered":"2024-06-29T14:37:22.315753+0000","last_clean":"2024-06-29T14:37:22.315753+0000","last_became_active":"2024-06-29T14:37:21.315836+0000","last_became_peered":"2024-06-29T14:37:21.315836+0000","last_unstale":"2024-06-29T14:37:22.315753+0000","last_undegraded":"2024-06-29T14:37:22.315753+0000","last_fullsized":"2024-06-29T14:37:22.315753+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:58.778234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330865+0000","last_change":"2024-06-29T14:37:21.315261+0000","last_active":"2024-06-29T14:37:22.330865+0000","last_peered":"2024-06-29T14:37:22.330865+0000","last_clean":"2024-06-29T14:37:22.330865+0000","last_became_active":"2024-06-29T14:37:21.314899+0000","last_became_peered":"2024-06-29T14:37:21.314899+0000","last_unstale":"2024-06-29T14:37:22.330865+0000","last_undegraded":"2024-06-29T14:37:22.330865+0000","last_fullsized":"2024-06-29T14:37:22.330865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:29:57.954983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330714+0000","last_change":"2024-06-29T14:37:21.322853+0000","last_active":"2024-06-29T14:37:22.330714+0000","last_peered":"2024-06-29T14:37:22.330714+0000","last_clean":"2024-06-29T14:37:22.330714+0000","last_became_active":"2024-06-29T14:37:21.314707+0000","last_became_peered":"2024-06-29T14:37:21.314707+0000","last_unstale":"2024-06-29T14:37:22.330714+0000","last_undegraded":"2024-06-29T14:37:22.330714+0000","last_fullsized":"2024-06-29T14:37:22.330714+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:13.135049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297065+0000","last_change":"2024-06-29T14:37:21.323649+0000","last_active":"2024-06-29T14:37:22.297065+0000","last_peered":"2024-06-29T14:37:22.297065+0000","last_clean":"2024-06-29T14:37:22.297065+0000","last_became_active":"2024-06-29T14:37:21.323301+0000","last_became_peered":"2024-06-29T14:37:21.323301+0000","last_unstale":"2024-06-29T14:37:22.297065+0000","last_undegraded":"2024-06-29T14:37:22.297065+0000","last_fullsized":"2024-06-29T14:37:22.297065+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:30:51.584785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332391+0000","last_change":"2024-06-29T14:37:21.314115+0000","last_active":"2024-06-29T14:37:22.332391+0000","last_peered":"2024-06-29T14:37:22.332391+0000","last_clean":"2024-06-29T14:37:22.332391+0000","last_became_active":"2024-06-29T14:37:21.313380+0000","last_became_peered":"2024-06-29T14:37:21.313380+0000","last_unstale":"2024-06-29T14:37:22.332391+0000","last_undegraded":"2024-06-29T14:37:22.332391+0000","last_fullsized":"2024-06-29T14:37:22.332391+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:05.689467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299099+0000","last_change":"2024-06-29T14:37:21.317315+0000","last_active":"2024-06-29T14:37:22.299099+0000","last_peered":"2024-06-29T14:37:22.299099+0000","last_clean":"2024-06-29T14:37:22.299099+0000","last_became_active":"2024-06-29T14:37:21.316741+0000","last_became_peered":"2024-06-29T14:37:21.316741+0000","last_unstale":"2024-06-29T14:37:22.299099+0000","last_undegraded":"2024-06-29T14:37:22.299099+0000","last_fullsized":"2024-06-29T14:37:22.299099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:05.745617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298566+0000","last_change":"2024-06-29T14:37:21.305898+0000","last_active":"2024-06-29T14:37:22.298566+0000","last_peered":"2024-06-29T14:37:22.298566+0000","last_clean":"2024-06-29T14:37:22.298566+0000","last_became_active":"2024-06-29T14:37:21.305566+0000","last_became_peered":"2024-06-29T14:37:21.305566+0000","last_unstale":"2024-06-29T14:37:22.298566+0000","last_undegraded":"2024-06-29T14:37:22.298566+0000","last_fullsized":"2024-06-29T14:37:22.298566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:41.159474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331902+0000","last_change":"2024-06-29T14:37:21.303624+0000","last_active":"2024-06-29T14:37:22.331902+0000","last_peered":"2024-06-29T14:37:22.331902+0000","last_clean":"2024-06-29T14:37:22.331902+0000","last_became_active":"2024-06-29T14:37:21.303305+0000","last_became_peered":"2024-06-29T14:37:21.303305+0000","last_unstale":"2024-06-29T14:37:22.331902+0000","last_undegraded":"2024-06-29T14:37:22.331902+0000","last_fullsized":"2024-06-29T14:37:22.331902+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:59.455244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315875+0000","last_change":"2024-06-29T14:37:21.317932+0000","last_active":"2024-06-29T14:37:22.315875+0000","last_peered":"2024-06-29T14:37:22.315875+0000","last_clean":"2024-06-29T14:37:22.315875+0000","last_became_active":"2024-06-29T14:37:21.317596+0000","last_became_peered":"2024-06-29T14:37:21.317596+0000","last_unstale":"2024-06-29T14:37:22.315875+0000","last_undegraded":"2024-06-29T14:37:22.315875+0000","last_fullsized":"2024-06-29T14:37:22.315875+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:51.620680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331541+0000","last_change":"2024-06-29T14:37:21.304728+0000","last_active":"2024-06-29T14:37:22.331541+0000","last_peered":"2024-06-29T14:37:22.331541+0000","last_clean":"2024-06-29T14:37:22.331541+0000","last_became_active":"2024-06-29T14:37:21.304385+0000","last_became_peered":"2024-06-29T14:37:21.304385+0000","last_unstale":"2024-06-29T14:37:22.331541+0000","last_undegraded":"2024-06-29T14:37:22.331541+0000","last_fullsized":"2024-06-29T14:37:22.331541+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:23.160262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331397+0000","last_change":"2024-06-29T14:37:21.311815+0000","last_active":"2024-06-29T14:37:22.331397+0000","last_peered":"2024-06-29T14:37:22.331397+0000","last_clean":"2024-06-29T14:37:22.331397+0000","last_became_active":"2024-06-29T14:37:21.311514+0000","last_became_peered":"2024-06-29T14:37:21.311514+0000","last_unstale":"2024-06-29T14:37:22.331397+0000","last_undegraded":"2024-06-29T14:37:22.331397+0000","last_fullsized":"2024-06-29T14:37:22.331397+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:20.337643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330984+0000","last_change":"2024-06-29T14:37:21.315395+0000","last_active":"2024-06-29T14:37:22.330984+0000","last_peered":"2024-06-29T14:37:22.330984+0000","last_clean":"2024-06-29T14:37:22.330984+0000","last_became_active":"2024-06-29T14:37:21.315072+0000","last_became_peered":"2024-06-29T14:37:21.315072+0000","last_unstale":"2024-06-29T14:37:22.330984+0000","last_undegraded":"2024-06-29T14:37:22.330984+0000","last_fullsized":"2024-06-29T14:37:22.330984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:00:24.182053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315880+0000","last_change":"2024-06-29T14:37:21.317693+0000","last_active":"2024-06-29T14:37:22.315880+0000","last_peered":"2024-06-29T14:37:22.315880+0000","last_clean":"2024-06-29T14:37:22.315880+0000","last_became_active":"2024-06-29T14:37:21.317336+0000","last_became_peered":"2024-06-29T14:37:21.317336+0000","last_unstale":"2024-06-29T14:37:22.315880+0000","last_undegraded":"2024-06-29T14:37:22.315880+0000","last_fullsized":"2024-06-29T14:37:22.315880+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:31.176809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315927+0000","last_change":"2024-06-29T14:37:21.322410+0000","last_active":"2024-06-29T14:37:22.315927+0000","last_peered":"2024-06-29T14:37:22.315927+0000","last_clean":"2024-06-29T14:37:22.315927+0000","last_became_active":"2024-06-29T14:37:21.322044+0000","last_became_peered":"2024-06-29T14:37:21.322044+0000","last_unstale":"2024-06-29T14:37:22.315927+0000","last_undegraded":"2024-06-29T14:37:22.315927+0000","last_fullsized":"2024-06-29T14:37:22.315927+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:12.503380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298573+0000","last_change":"2024-06-29T14:37:21.311920+0000","last_active":"2024-06-29T14:37:22.298573+0000","last_peered":"2024-06-29T14:37:22.298573+0000","last_clean":"2024-06-29T14:37:22.298573+0000","last_became_active":"2024-06-29T14:37:21.311586+0000","last_became_peered":"2024-06-29T14:37:21.311586+0000","last_unstale":"2024-06-29T14:37:22.298573+0000","last_undegraded":"2024-06-29T14:37:22.298573+0000","last_fullsized":"2024-06-29T14:37:22.298573+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:11.052839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331774+0000","last_change":"2024-06-29T14:37:21.317292+0000","last_active":"2024-06-29T14:37:22.331774+0000","last_peered":"2024-06-29T14:37:22.331774+0000","last_clean":"2024-06-29T14:37:22.331774+0000","last_became_active":"2024-06-29T14:37:21.316975+0000","last_became_peered":"2024-06-29T14:37:21.316975+0000","last_unstale":"2024-06-29T14:37:22.331774+0000","last_undegraded":"2024-06-29T14:37:22.331774+0000","last_fullsized":"2024-06-29T14:37:22.331774+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:28.776718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330917+0000","last_change":"2024-06-29T14:37:21.321547+0000","last_active":"2024-06-29T14:37:22.330917+0000","last_peered":"2024-06-29T14:37:22.330917+0000","last_clean":"2024-06-29T14:37:22.330917+0000","last_became_active":"2024-06-29T14:37:21.312371+0000","last_became_peered":"2024-06-29T14:37:21.312371+0000","last_unstale":"2024-06-29T14:37:22.330917+0000","last_undegraded":"2024-06-29T14:37:22.330917+0000","last_fullsized":"2024-06-29T14:37:22.330917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:46.476553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315681+0000","last_change":"2024-06-29T14:37:21.328902+0000","last_active":"2024-06-29T14:37:22.315681+0000","last_peered":"2024-06-29T14:37:22.315681+0000","last_clean":"2024-06-29T14:37:22.315681+0000","last_became_active":"2024-06-29T14:37:21.328651+0000","last_became_peered":"2024-06-29T14:37:21.328651+0000","last_unstale":"2024-06-29T14:37:22.315681+0000","last_undegraded":"2024-06-29T14:37:22.315681+0000","last_fullsized":"2024-06-29T14:37:22.315681+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:59.455244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298101+0000","last_change":"2024-06-29T14:37:21.310485+0000","last_active":"2024-06-29T14:37:22.298101+0000","last_peered":"2024-06-29T14:37:22.298101+0000","last_clean":"2024-06-29T14:37:22.298101+0000","last_became_active":"2024-06-29T14:37:21.310133+0000","last_became_peered":"2024-06-29T14:37:21.310133+0000","last_unstale":"2024-06-29T14:37:22.298101+0000","last_undegraded":"2024-06-29T14:37:22.298101+0000","last_fullsized":"2024-06-29T14:37:22.298101+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:22.177194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330826+0000","last_change":"2024-06-29T14:37:21.314043+0000","last_active":"2024-06-29T14:37:22.330826+0000","last_peered":"2024-06-29T14:37:22.330826+0000","last_clean":"2024-06-29T14:37:22.330826+0000","last_became_active":"2024-06-29T14:37:21.313707+0000","last_became_peered":"2024-06-29T14:37:21.313707+0000","last_unstale":"2024-06-29T14:37:22.330826+0000","last_undegraded":"2024-06-29T14:37:22.330826+0000","last_fullsized":"2024-06-29T14:37:22.330826+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:33.333380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298525+0000","last_change":"2024-06-29T14:37:21.310224+0000","last_active":"2024-06-29T14:37:22.298525+0000","last_peered":"2024-06-29T14:37:22.298525+0000","last_clean":"2024-06-29T14:37:22.298525+0000","last_became_active":"2024-06-29T14:37:21.309932+0000","last_became_peered":"2024-06-29T14:37:21.309932+0000","last_unstale":"2024-06-29T14:37:22.298525+0000","last_undegraded":"2024-06-29T14:37:22.298525+0000","last_fullsized":"2024-06-29T14:37:22.298525+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:33.524218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"87'4","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332344+0000","last_change":"2024-06-29T14:37:20.303082+0000","last_active":"2024-06-29T14:37:22.332344+0000","last_peered":"2024-06-29T14:37:22.332344+0000","last_clean":"2024-06-29T14:37:22.332344+0000","last_became_active":"2024-06-29T14:37:20.302767+0000","last_became_peered":"2024-06-29T14:37:20.302767+0000","last_unstale":"2024-06-29T14:37:22.332344+0000","last_undegraded":"2024-06-29T14:37:22.332344+0000","last_fullsized":"2024-06-29T14:37:22.332344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:10.445714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298727+0000","last_change":"2024-06-29T14:37:21.317749+0000","last_active":"2024-06-29T14:37:22.298727+0000","last_peered":"2024-06-29T14:37:22.298727+0000","last_clean":"2024-06-29T14:37:22.298727+0000","last_became_active":"2024-06-29T14:37:21.317418+0000","last_became_peered":"2024-06-29T14:37:21.317418+0000","last_unstale":"2024-06-29T14:37:22.298727+0000","last_undegraded":"2024-06-29T14:37:22.298727+0000","last_fullsized":"2024-06-29T14:37:22.298727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:09:49.985745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331887+0000","last_change":"2024-06-29T14:37:20.301311+0000","last_active":"2024-06-29T14:37:22.331887+0000","last_peered":"2024-06-29T14:37:22.331887+0000","last_clean":"2024-06-29T14:37:22.331887+0000","last_became_active":"2024-06-29T14:37:20.300988+0000","last_became_peered":"2024-06-29T14:37:20.300988+0000","last_unstale":"2024-06-29T14:37:22.331887+0000","last_undegraded":"2024-06-29T14:37:22.331887+0000","last_fullsized":"2024-06-29T14:37:22.331887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:55.830301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331674+0000","last_change":"2024-06-29T14:37:21.313558+0000","last_active":"2024-06-29T14:37:22.331674+0000","last_peered":"2024-06-29T14:37:22.331674+0000","last_clean":"2024-06-29T14:37:22.331674+0000","last_became_active":"2024-06-29T14:37:21.313230+0000","last_became_peered":"2024-06-29T14:37:21.313230+0000","last_unstale":"2024-06-29T14:37:22.331674+0000","last_undegraded":"2024-06-29T14:37:22.331674+0000","last_fullsized":"2024-06-29T14:37:22.331674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:28.290552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331612+0000","last_change":"2024-06-29T14:37:20.300262+0000","last_active":"2024-06-29T14:37:22.331612+0000","last_peered":"2024-06-29T14:37:22.331612+0000","last_clean":"2024-06-29T14:37:22.331612+0000","last_became_active":"2024-06-29T14:37:20.299596+0000","last_became_peered":"2024-06-29T14:37:20.299596+0000","last_unstale":"2024-06-29T14:37:22.331612+0000","last_undegraded":"2024-06-29T14:37:22.331612+0000","last_fullsized":"2024-06-29T14:37:22.331612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:42:42.724298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331078+0000","last_change":"2024-06-29T14:37:21.321287+0000","last_active":"2024-06-29T14:37:22.331078+0000","last_peered":"2024-06-29T14:37:22.331078+0000","last_clean":"2024-06-29T14:37:22.331078+0000","last_became_active":"2024-06-29T14:37:21.315884+0000","last_became_peered":"2024-06-29T14:37:21.315884+0000","last_unstale":"2024-06-29T14:37:22.331078+0000","last_undegraded":"2024-06-29T14:37:22.331078+0000","last_fullsized":"2024-06-29T14:37:22.331078+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:52:44.152310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331980+0000","last_change":"2024-06-29T14:37:20.300752+0000","last_active":"2024-06-29T14:37:22.331980+0000","last_peered":"2024-06-29T14:37:22.331980+0000","last_clean":"2024-06-29T14:37:22.331980+0000","last_became_active":"2024-06-29T14:37:20.300440+0000","last_became_peered":"2024-06-29T14:37:20.300440+0000","last_unstale":"2024-06-29T14:37:22.331980+0000","last_undegraded":"2024-06-29T14:37:22.331980+0000","last_fullsized":"2024-06-29T14:37:22.331980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:42.227035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297229+0000","last_change":"2024-06-29T14:37:21.309654+0000","last_active":"2024-06-29T14:37:22.297229+0000","last_peered":"2024-06-29T14:37:22.297229+0000","last_clean":"2024-06-29T14:37:22.297229+0000","last_became_active":"2024-06-29T14:37:21.309282+0000","last_became_peered":"2024-06-29T14:37:21.309282+0000","last_unstale":"2024-06-29T14:37:22.297229+0000","last_undegraded":"2024-06-29T14:37:22.297229+0000","last_fullsized":"2024-06-29T14:37:22.297229+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:45:38.559161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331462+0000","last_change":"2024-06-29T14:37:20.295528+0000","last_active":"2024-06-29T14:37:22.331462+0000","last_peered":"2024-06-29T14:37:22.331462+0000","last_clean":"2024-06-29T14:37:22.331462+0000","last_became_active":"2024-06-29T14:37:20.294656+0000","last_became_peered":"2024-06-29T14:37:20.294656+0000","last_unstale":"2024-06-29T14:37:22.331462+0000","last_undegraded":"2024-06-29T14:37:22.331462+0000","last_fullsized":"2024-06-29T14:37:22.331462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:02.146628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331075+0000","last_change":"2024-06-29T14:37:21.312980+0000","last_active":"2024-06-29T14:37:22.331075+0000","last_peered":"2024-06-29T14:37:22.331075+0000","last_clean":"2024-06-29T14:37:22.331075+0000","last_became_active":"2024-06-29T14:37:21.312654+0000","last_became_peered":"2024-06-29T14:37:21.312654+0000","last_unstale":"2024-06-29T14:37:22.331075+0000","last_undegraded":"2024-06-29T14:37:22.331075+0000","last_fullsized":"2024-06-29T14:37:22.331075+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:25.597063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331646+0000","last_change":"2024-06-29T14:37:20.294982+0000","last_active":"2024-06-29T14:37:22.331646+0000","last_peered":"2024-06-29T14:37:22.331646+0000","last_clean":"2024-06-29T14:37:22.331646+0000","last_became_active":"2024-06-29T14:37:20.294351+0000","last_became_peered":"2024-06-29T14:37:20.294351+0000","last_unstale":"2024-06-29T14:37:22.331646+0000","last_undegraded":"2024-06-29T14:37:22.331646+0000","last_fullsized":"2024-06-29T14:37:22.331646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:24.988008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298872+0000","last_change":"2024-06-29T14:37:21.312743+0000","last_active":"2024-06-29T14:37:22.298872+0000","last_peered":"2024-06-29T14:37:22.298872+0000","last_clean":"2024-06-29T14:37:22.298872+0000","last_became_active":"2024-06-29T14:37:21.312201+0000","last_became_peered":"2024-06-29T14:37:21.312201+0000","last_unstale":"2024-06-29T14:37:22.298872+0000","last_undegraded":"2024-06-29T14:37:22.298872+0000","last_fullsized":"2024-06-29T14:37:22.298872+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:33.504070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331362+0000","last_change":"2024-06-29T14:37:20.293349+0000","last_active":"2024-06-29T14:37:22.331362+0000","last_peered":"2024-06-29T14:37:22.331362+0000","last_clean":"2024-06-29T14:37:22.331362+0000","last_became_active":"2024-06-29T14:37:20.292982+0000","last_became_peered":"2024-06-29T14:37:20.292982+0000","last_unstale":"2024-06-29T14:37:22.331362+0000","last_undegraded":"2024-06-29T14:37:22.331362+0000","last_fullsized":"2024-06-29T14:37:22.331362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:29.149704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331327+0000","last_change":"2024-06-29T14:37:21.316255+0000","last_active":"2024-06-29T14:37:22.331327+0000","last_peered":"2024-06-29T14:37:22.331327+0000","last_clean":"2024-06-29T14:37:22.331327+0000","last_became_active":"2024-06-29T14:37:21.315928+0000","last_became_peered":"2024-06-29T14:37:21.315928+0000","last_unstale":"2024-06-29T14:37:22.331327+0000","last_undegraded":"2024-06-29T14:37:22.331327+0000","last_fullsized":"2024-06-29T14:37:22.331327+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:54:02.988704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298242+0000","last_change":"2024-06-29T14:37:20.296335+0000","last_active":"2024-06-29T14:37:22.298242+0000","last_peered":"2024-06-29T14:37:22.298242+0000","last_clean":"2024-06-29T14:37:22.298242+0000","last_became_active":"2024-06-29T14:37:20.296046+0000","last_became_peered":"2024-06-29T14:37:20.296046+0000","last_unstale":"2024-06-29T14:37:22.298242+0000","last_undegraded":"2024-06-29T14:37:22.298242+0000","last_fullsized":"2024-06-29T14:37:22.298242+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:14.341107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331944+0000","last_change":"2024-06-29T14:37:21.308791+0000","last_active":"2024-06-29T14:37:22.331944+0000","last_peered":"2024-06-29T14:37:22.331944+0000","last_clean":"2024-06-29T14:37:22.331944+0000","last_became_active":"2024-06-29T14:37:21.308479+0000","last_became_peered":"2024-06-29T14:37:21.308479+0000","last_unstale":"2024-06-29T14:37:22.331944+0000","last_undegraded":"2024-06-29T14:37:22.331944+0000","last_fullsized":"2024-06-29T14:37:22.331944+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:17.368735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331181+0000","last_change":"2024-06-29T14:37:20.304108+0000","last_active":"2024-06-29T14:37:22.331181+0000","last_peered":"2024-06-29T14:37:22.331181+0000","last_clean":"2024-06-29T14:37:22.331181+0000","last_became_active":"2024-06-29T14:37:20.303755+0000","last_became_peered":"2024-06-29T14:37:20.303755+0000","last_unstale":"2024-06-29T14:37:22.331181+0000","last_undegraded":"2024-06-29T14:37:22.331181+0000","last_fullsized":"2024-06-29T14:37:22.331181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:47.128695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331056+0000","last_change":"2024-06-29T14:37:21.320528+0000","last_active":"2024-06-29T14:37:22.331056+0000","last_peered":"2024-06-29T14:37:22.331056+0000","last_clean":"2024-06-29T14:37:22.331056+0000","last_became_active":"2024-06-29T14:37:21.320193+0000","last_became_peered":"2024-06-29T14:37:21.320193+0000","last_unstale":"2024-06-29T14:37:22.331056+0000","last_undegraded":"2024-06-29T14:37:22.331056+0000","last_fullsized":"2024-06-29T14:37:22.331056+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:11:10.100908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299235+0000","last_change":"2024-06-29T14:37:20.298814+0000","last_active":"2024-06-29T14:37:22.299235+0000","last_peered":"2024-06-29T14:37:22.299235+0000","last_clean":"2024-06-29T14:37:22.299235+0000","last_became_active":"2024-06-29T14:37:20.298533+0000","last_became_peered":"2024-06-29T14:37:20.298533+0000","last_unstale":"2024-06-29T14:37:22.299235+0000","last_undegraded":"2024-06-29T14:37:22.299235+0000","last_fullsized":"2024-06-29T14:37:22.299235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:04:22.860050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298262+0000","last_change":"2024-06-29T14:37:21.323782+0000","last_active":"2024-06-29T14:37:22.298262+0000","last_peered":"2024-06-29T14:37:22.298262+0000","last_clean":"2024-06-29T14:37:22.298262+0000","last_became_active":"2024-06-29T14:37:21.323446+0000","last_became_peered":"2024-06-29T14:37:21.323446+0000","last_unstale":"2024-06-29T14:37:22.298262+0000","last_undegraded":"2024-06-29T14:37:22.298262+0000","last_fullsized":"2024-06-29T14:37:22.298262+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:25:52.388229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"87'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330758+0000","last_change":"2024-06-29T14:37:20.303709+0000","last_active":"2024-06-29T14:37:22.330758+0000","last_peered":"2024-06-29T14:37:22.330758+0000","last_clean":"2024-06-29T14:37:22.330758+0000","last_became_active":"2024-06-29T14:37:20.303413+0000","last_became_peered":"2024-06-29T14:37:20.303413+0000","last_unstale":"2024-06-29T14:37:22.330758+0000","last_undegraded":"2024-06-29T14:37:22.330758+0000","last_fullsized":"2024-06-29T14:37:22.330758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:37.074235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299084+0000","last_change":"2024-06-29T14:37:21.308810+0000","last_active":"2024-06-29T14:37:22.299084+0000","last_peered":"2024-06-29T14:37:22.299084+0000","last_clean":"2024-06-29T14:37:22.299084+0000","last_became_active":"2024-06-29T14:37:21.308487+0000","last_became_peered":"2024-06-29T14:37:21.308487+0000","last_unstale":"2024-06-29T14:37:22.299084+0000","last_undegraded":"2024-06-29T14:37:22.299084+0000","last_fullsized":"2024-06-29T14:37:22.299084+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:51.302471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"87'6","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332141+0000","last_change":"2024-06-29T14:37:20.298014+0000","last_active":"2024-06-29T14:37:22.332141+0000","last_peered":"2024-06-29T14:37:22.332141+0000","last_clean":"2024-06-29T14:37:22.332141+0000","last_became_active":"2024-06-29T14:37:20.297505+0000","last_became_peered":"2024-06-29T14:37:20.297505+0000","last_unstale":"2024-06-29T14:37:22.332141+0000","last_undegraded":"2024-06-29T14:37:22.332141+0000","last_fullsized":"2024-06-29T14:37:22.332141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:54.029809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297985+0000","last_change":"2024-06-29T14:37:21.314656+0000","last_active":"2024-06-29T14:37:22.297985+0000","last_peered":"2024-06-29T14:37:22.297985+0000","last_clean":"2024-06-29T14:37:22.297985+0000","last_became_active":"2024-06-29T14:37:21.314116+0000","last_became_peered":"2024-06-29T14:37:21.314116+0000","last_unstale":"2024-06-29T14:37:22.297985+0000","last_undegraded":"2024-06-29T14:37:22.297985+0000","last_fullsized":"2024-06-29T14:37:22.297985+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:05.918168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330959+0000","last_change":"2024-06-29T14:37:20.296074+0000","last_active":"2024-06-29T14:37:22.330959+0000","last_peered":"2024-06-29T14:37:22.330959+0000","last_clean":"2024-06-29T14:37:22.330959+0000","last_became_active":"2024-06-29T14:37:20.294396+0000","last_became_peered":"2024-06-29T14:37:20.294396+0000","last_unstale":"2024-06-29T14:37:22.330959+0000","last_undegraded":"2024-06-29T14:37:22.330959+0000","last_fullsized":"2024-06-29T14:37:22.330959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:12.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315615+0000","last_change":"2024-06-29T14:37:21.322914+0000","last_active":"2024-06-29T14:37:22.315615+0000","last_peered":"2024-06-29T14:37:22.315615+0000","last_clean":"2024-06-29T14:37:22.315615+0000","last_became_active":"2024-06-29T14:37:21.322582+0000","last_became_peered":"2024-06-29T14:37:21.322582+0000","last_unstale":"2024-06-29T14:37:22.315615+0000","last_undegraded":"2024-06-29T14:37:22.315615+0000","last_fullsized":"2024-06-29T14:37:22.315615+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:20.337643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315656+0000","last_change":"2024-06-29T14:37:20.302109+0000","last_active":"2024-06-29T14:37:22.315656+0000","last_peered":"2024-06-29T14:37:22.315656+0000","last_clean":"2024-06-29T14:37:22.315656+0000","last_became_active":"2024-06-29T14:37:20.301765+0000","last_became_peered":"2024-06-29T14:37:20.301765+0000","last_unstale":"2024-06-29T14:37:22.315656+0000","last_undegraded":"2024-06-29T14:37:22.315656+0000","last_fullsized":"2024-06-29T14:37:22.315656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:34.482542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299100+0000","last_change":"2024-06-29T14:37:21.304641+0000","last_active":"2024-06-29T14:37:22.299100+0000","last_peered":"2024-06-29T14:37:22.299100+0000","last_clean":"2024-06-29T14:37:22.299100+0000","last_became_active":"2024-06-29T14:37:21.304328+0000","last_became_peered":"2024-06-29T14:37:21.304328+0000","last_unstale":"2024-06-29T14:37:22.299100+0000","last_undegraded":"2024-06-29T14:37:22.299100+0000","last_fullsized":"2024-06-29T14:37:22.299100+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:09:49.985745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330944+0000","last_change":"2024-06-29T14:37:20.298474+0000","last_active":"2024-06-29T14:37:22.330944+0000","last_peered":"2024-06-29T14:37:22.330944+0000","last_clean":"2024-06-29T14:37:22.330944+0000","last_became_active":"2024-06-29T14:37:20.298155+0000","last_became_peered":"2024-06-29T14:37:20.298155+0000","last_unstale":"2024-06-29T14:37:22.330944+0000","last_undegraded":"2024-06-29T14:37:22.330944+0000","last_fullsized":"2024-06-29T14:37:22.330944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:32.529040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331334+0000","last_change":"2024-06-29T14:37:21.316244+0000","last_active":"2024-06-29T14:37:22.331334+0000","last_peered":"2024-06-29T14:37:22.331334+0000","last_clean":"2024-06-29T14:37:22.331334+0000","last_became_active":"2024-06-29T14:37:21.315803+0000","last_became_peered":"2024-06-29T14:37:21.315803+0000","last_unstale":"2024-06-29T14:37:22.331334+0000","last_undegraded":"2024-06-29T14:37:22.331334+0000","last_fullsized":"2024-06-29T14:37:22.331334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:37.339449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"87'8","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298132+0000","last_change":"2024-06-29T14:37:20.297924+0000","last_active":"2024-06-29T14:37:22.298132+0000","last_peered":"2024-06-29T14:37:22.298132+0000","last_clean":"2024-06-29T14:37:22.298132+0000","last_became_active":"2024-06-29T14:37:20.297604+0000","last_became_peered":"2024-06-29T14:37:20.297604+0000","last_unstale":"2024-06-29T14:37:22.298132+0000","last_undegraded":"2024-06-29T14:37:22.298132+0000","last_fullsized":"2024-06-29T14:37:22.298132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:45:15.810337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330972+0000","last_change":"2024-06-29T14:37:21.312532+0000","last_active":"2024-06-29T14:37:22.330972+0000","last_peered":"2024-06-29T14:37:22.330972+0000","last_clean":"2024-06-29T14:37:22.330972+0000","last_became_active":"2024-06-29T14:37:21.312192+0000","last_became_peered":"2024-06-29T14:37:21.312192+0000","last_unstale":"2024-06-29T14:37:22.330972+0000","last_undegraded":"2024-06-29T14:37:22.330972+0000","last_fullsized":"2024-06-29T14:37:22.330972+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:43.362310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315808+0000","last_change":"2024-06-29T14:37:20.304834+0000","last_active":"2024-06-29T14:37:22.315808+0000","last_peered":"2024-06-29T14:37:22.315808+0000","last_clean":"2024-06-29T14:37:22.315808+0000","last_became_active":"2024-06-29T14:37:20.304472+0000","last_became_peered":"2024-06-29T14:37:20.304472+0000","last_unstale":"2024-06-29T14:37:22.315808+0000","last_undegraded":"2024-06-29T14:37:22.315808+0000","last_fullsized":"2024-06-29T14:37:22.315808+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:29.149704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298753+0000","last_change":"2024-06-29T14:37:21.314575+0000","last_active":"2024-06-29T14:37:22.298753+0000","last_peered":"2024-06-29T14:37:22.298753+0000","last_clean":"2024-06-29T14:37:22.298753+0000","last_became_active":"2024-06-29T14:37:21.314257+0000","last_became_peered":"2024-06-29T14:37:21.314257+0000","last_unstale":"2024-06-29T14:37:22.298753+0000","last_undegraded":"2024-06-29T14:37:22.298753+0000","last_fullsized":"2024-06-29T14:37:22.298753+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:22.177194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331145+0000","last_change":"2024-06-29T14:37:20.297122+0000","last_active":"2024-06-29T14:37:22.331145+0000","last_peered":"2024-06-29T14:37:22.331145+0000","last_clean":"2024-06-29T14:37:22.331145+0000","last_became_active":"2024-06-29T14:37:20.296799+0000","last_became_peered":"2024-06-29T14:37:20.296799+0000","last_unstale":"2024-06-29T14:37:22.331145+0000","last_undegraded":"2024-06-29T14:37:22.331145+0000","last_fullsized":"2024-06-29T14:37:22.331145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:48:14.404396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299292+0000","last_change":"2024-06-29T14:37:21.325669+0000","last_active":"2024-06-29T14:37:22.299292+0000","last_peered":"2024-06-29T14:37:22.299292+0000","last_clean":"2024-06-29T14:37:22.299292+0000","last_became_active":"2024-06-29T14:37:21.325363+0000","last_became_peered":"2024-06-29T14:37:21.325363+0000","last_unstale":"2024-06-29T14:37:22.299292+0000","last_undegraded":"2024-06-29T14:37:22.299292+0000","last_fullsized":"2024-06-29T14:37:22.299292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:48.654672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330917+0000","last_change":"2024-06-29T14:37:20.308208+0000","last_active":"2024-06-29T14:37:22.330917+0000","last_peered":"2024-06-29T14:37:22.330917+0000","last_clean":"2024-06-29T14:37:22.330917+0000","last_became_active":"2024-06-29T14:37:20.307910+0000","last_became_peered":"2024-06-29T14:37:20.307910+0000","last_unstale":"2024-06-29T14:37:22.330917+0000","last_undegraded":"2024-06-29T14:37:22.330917+0000","last_fullsized":"2024-06-29T14:37:22.330917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:50.043421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298488+0000","last_change":"2024-06-29T14:37:21.310399+0000","last_active":"2024-06-29T14:37:22.298488+0000","last_peered":"2024-06-29T14:37:22.298488+0000","last_clean":"2024-06-29T14:37:22.298488+0000","last_became_active":"2024-06-29T14:37:21.306777+0000","last_became_peered":"2024-06-29T14:37:21.306777+0000","last_unstale":"2024-06-29T14:37:22.298488+0000","last_undegraded":"2024-06-29T14:37:22.298488+0000","last_fullsized":"2024-06-29T14:37:22.298488+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:40:09.321136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332009+0000","last_change":"2024-06-29T14:37:20.296918+0000","last_active":"2024-06-29T14:37:22.332009+0000","last_peered":"2024-06-29T14:37:22.332009+0000","last_clean":"2024-06-29T14:37:22.332009+0000","last_became_active":"2024-06-29T14:37:20.296570+0000","last_became_peered":"2024-06-29T14:37:20.296570+0000","last_unstale":"2024-06-29T14:37:22.332009+0000","last_undegraded":"2024-06-29T14:37:22.332009+0000","last_fullsized":"2024-06-29T14:37:22.332009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:25.104665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330991+0000","last_change":"2024-06-29T14:37:21.313296+0000","last_active":"2024-06-29T14:37:22.330991+0000","last_peered":"2024-06-29T14:37:22.330991+0000","last_clean":"2024-06-29T14:37:22.330991+0000","last_became_active":"2024-06-29T14:37:21.312985+0000","last_became_peered":"2024-06-29T14:37:21.312985+0000","last_unstale":"2024-06-29T14:37:22.330991+0000","last_undegraded":"2024-06-29T14:37:22.330991+0000","last_fullsized":"2024-06-29T14:37:22.330991+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:16.740545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330871+0000","last_change":"2024-06-29T14:37:20.297386+0000","last_active":"2024-06-29T14:37:22.330871+0000","last_peered":"2024-06-29T14:37:22.330871+0000","last_clean":"2024-06-29T14:37:22.330871+0000","last_became_active":"2024-06-29T14:37:20.292152+0000","last_became_peered":"2024-06-29T14:37:20.292152+0000","last_unstale":"2024-06-29T14:37:22.330871+0000","last_undegraded":"2024-06-29T14:37:22.330871+0000","last_fullsized":"2024-06-29T14:37:22.330871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:26:56.590026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315627+0000","last_change":"2024-06-29T14:37:21.319010+0000","last_active":"2024-06-29T14:37:22.315627+0000","last_peered":"2024-06-29T14:37:22.315627+0000","last_clean":"2024-06-29T14:37:22.315627+0000","last_became_active":"2024-06-29T14:37:21.318578+0000","last_became_peered":"2024-06-29T14:37:21.318578+0000","last_unstale":"2024-06-29T14:37:22.315627+0000","last_undegraded":"2024-06-29T14:37:22.315627+0000","last_fullsized":"2024-06-29T14:37:22.315627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:54:02.988704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315866+0000","last_change":"2024-06-29T14:37:20.301660+0000","last_active":"2024-06-29T14:37:22.315866+0000","last_peered":"2024-06-29T14:37:22.315866+0000","last_clean":"2024-06-29T14:37:22.315866+0000","last_became_active":"2024-06-29T14:37:20.301346+0000","last_became_peered":"2024-06-29T14:37:20.301346+0000","last_unstale":"2024-06-29T14:37:22.315866+0000","last_undegraded":"2024-06-29T14:37:22.315866+0000","last_fullsized":"2024-06-29T14:37:22.315866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:57.474408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299087+0000","last_change":"2024-06-29T14:37:21.312489+0000","last_active":"2024-06-29T14:37:22.299087+0000","last_peered":"2024-06-29T14:37:22.299087+0000","last_clean":"2024-06-29T14:37:22.299087+0000","last_became_active":"2024-06-29T14:37:21.311809+0000","last_became_peered":"2024-06-29T14:37:21.311809+0000","last_unstale":"2024-06-29T14:37:22.299087+0000","last_undegraded":"2024-06-29T14:37:22.299087+0000","last_fullsized":"2024-06-29T14:37:22.299087+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:11.052839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331186+0000","last_change":"2024-06-29T14:37:20.300564+0000","last_active":"2024-06-29T14:37:22.331186+0000","last_peered":"2024-06-29T14:37:22.331186+0000","last_clean":"2024-06-29T14:37:22.331186+0000","last_became_active":"2024-06-29T14:37:20.300232+0000","last_became_peered":"2024-06-29T14:37:20.300232+0000","last_unstale":"2024-06-29T14:37:22.331186+0000","last_undegraded":"2024-06-29T14:37:22.331186+0000","last_fullsized":"2024-06-29T14:37:22.331186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:23.430715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331913+0000","last_change":"2024-06-29T14:37:21.316305+0000","last_active":"2024-06-29T14:37:22.331913+0000","last_peered":"2024-06-29T14:37:22.331913+0000","last_clean":"2024-06-29T14:37:22.331913+0000","last_became_active":"2024-06-29T14:37:21.315989+0000","last_became_peered":"2024-06-29T14:37:21.315989+0000","last_unstale":"2024-06-29T14:37:22.331913+0000","last_undegraded":"2024-06-29T14:37:22.331913+0000","last_fullsized":"2024-06-29T14:37:22.331913+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:07:12.439421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"87'2","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.316004+0000","last_change":"2024-06-29T14:37:20.297706+0000","last_active":"2024-06-29T14:37:22.316004+0000","last_peered":"2024-06-29T14:37:22.316004+0000","last_clean":"2024-06-29T14:37:22.316004+0000","last_became_active":"2024-06-29T14:37:20.297369+0000","last_became_peered":"2024-06-29T14:37:20.297369+0000","last_unstale":"2024-06-29T14:37:22.316004+0000","last_undegraded":"2024-06-29T14:37:22.316004+0000","last_fullsized":"2024-06-29T14:37:22.316004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:25.104665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297348+0000","last_change":"2024-06-29T14:37:21.315791+0000","last_active":"2024-06-29T14:37:22.297348+0000","last_peered":"2024-06-29T14:37:22.297348+0000","last_clean":"2024-06-29T14:37:22.297348+0000","last_became_active":"2024-06-29T14:37:21.315481+0000","last_became_peered":"2024-06-29T14:37:21.315481+0000","last_unstale":"2024-06-29T14:37:22.297348+0000","last_undegraded":"2024-06-29T14:37:22.297348+0000","last_fullsized":"2024-06-29T14:37:22.297348+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:47:55.915681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"88'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.317351+0000","last_change":"2024-06-29T14:37:20.295958+0000","last_active":"2024-06-29T14:37:22.317351+0000","last_peered":"2024-06-29T14:37:22.317351+0000","last_clean":"2024-06-29T14:37:22.317351+0000","last_became_active":"2024-06-29T14:37:20.295598+0000","last_became_peered":"2024-06-29T14:37:20.295598+0000","last_unstale":"2024-06-29T14:37:22.317351+0000","last_undegraded":"2024-06-29T14:37:22.317351+0000","last_fullsized":"2024-06-29T14:37:22.317351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:02.146628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.315970+0000","last_change":"2024-06-29T14:37:21.322411+0000","last_active":"2024-06-29T14:37:22.315970+0000","last_peered":"2024-06-29T14:37:22.315970+0000","last_clean":"2024-06-29T14:37:22.315970+0000","last_became_active":"2024-06-29T14:37:21.322097+0000","last_became_peered":"2024-06-29T14:37:21.322097+0000","last_unstale":"2024-06-29T14:37:22.315970+0000","last_undegraded":"2024-06-29T14:37:22.315970+0000","last_fullsized":"2024-06-29T14:37:22.315970+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:22.750933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331728+0000","last_change":"2024-06-29T14:37:20.298730+0000","last_active":"2024-06-29T14:37:22.331728+0000","last_peered":"2024-06-29T14:37:22.331728+0000","last_clean":"2024-06-29T14:37:22.331728+0000","last_became_active":"2024-06-29T14:37:20.298362+0000","last_became_peered":"2024-06-29T14:37:20.298362+0000","last_unstale":"2024-06-29T14:37:22.331728+0000","last_undegraded":"2024-06-29T14:37:22.331728+0000","last_fullsized":"2024-06-29T14:37:22.331728+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:03.760040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298664+0000","last_change":"2024-06-29T14:37:21.302425+0000","last_active":"2024-06-29T14:37:22.298664+0000","last_peered":"2024-06-29T14:37:22.298664+0000","last_clean":"2024-06-29T14:37:22.298664+0000","last_became_active":"2024-06-29T14:37:21.302091+0000","last_became_peered":"2024-06-29T14:37:21.302091+0000","last_unstale":"2024-06-29T14:37:22.298664+0000","last_undegraded":"2024-06-29T14:37:22.298664+0000","last_fullsized":"2024-06-29T14:37:22.298664+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:05.918168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.332176+0000","last_change":"2024-06-29T14:37:20.298015+0000","last_active":"2024-06-29T14:37:22.332176+0000","last_peered":"2024-06-29T14:37:22.332176+0000","last_clean":"2024-06-29T14:37:22.332176+0000","last_became_active":"2024-06-29T14:37:20.297677+0000","last_became_peered":"2024-06-29T14:37:20.297677+0000","last_unstale":"2024-06-29T14:37:22.332176+0000","last_undegraded":"2024-06-29T14:37:22.332176+0000","last_fullsized":"2024-06-29T14:37:22.332176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:29:24.594454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298104+0000","last_change":"2024-06-29T14:37:21.309531+0000","last_active":"2024-06-29T14:37:22.298104+0000","last_peered":"2024-06-29T14:37:22.298104+0000","last_clean":"2024-06-29T14:37:22.298104+0000","last_became_active":"2024-06-29T14:37:21.309209+0000","last_became_peered":"2024-06-29T14:37:21.309209+0000","last_unstale":"2024-06-29T14:37:22.298104+0000","last_undegraded":"2024-06-29T14:37:22.298104+0000","last_fullsized":"2024-06-29T14:37:22.298104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:44.697323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331089+0000","last_change":"2024-06-29T14:37:20.300925+0000","last_active":"2024-06-29T14:37:22.331089+0000","last_peered":"2024-06-29T14:37:22.331089+0000","last_clean":"2024-06-29T14:37:22.331089+0000","last_became_active":"2024-06-29T14:37:20.300594+0000","last_became_peered":"2024-06-29T14:37:20.300594+0000","last_unstale":"2024-06-29T14:37:22.331089+0000","last_undegraded":"2024-06-29T14:37:22.331089+0000","last_fullsized":"2024-06-29T14:37:22.331089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:43.510209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.298292+0000","last_change":"2024-06-29T14:37:21.314337+0000","last_active":"2024-06-29T14:37:22.298292+0000","last_peered":"2024-06-29T14:37:22.298292+0000","last_clean":"2024-06-29T14:37:22.298292+0000","last_became_active":"2024-06-29T14:37:21.313936+0000","last_became_peered":"2024-06-29T14:37:21.313936+0000","last_unstale":"2024-06-29T14:37:22.298292+0000","last_undegraded":"2024-06-29T14:37:22.298292+0000","last_fullsized":"2024-06-29T14:37:22.298292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:33.504070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331656+0000","last_change":"2024-06-29T14:37:20.301959+0000","last_active":"2024-06-29T14:37:22.331656+0000","last_peered":"2024-06-29T14:37:22.331656+0000","last_clean":"2024-06-29T14:37:22.331656+0000","last_became_active":"2024-06-29T14:37:20.301503+0000","last_became_peered":"2024-06-29T14:37:20.301503+0000","last_unstale":"2024-06-29T14:37:22.331656+0000","last_undegraded":"2024-06-29T14:37:22.331656+0000","last_fullsized":"2024-06-29T14:37:22.331656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:05.989680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331480+0000","last_change":"2024-06-29T14:37:21.312540+0000","last_active":"2024-06-29T14:37:22.331480+0000","last_peered":"2024-06-29T14:37:22.331480+0000","last_clean":"2024-06-29T14:37:22.331480+0000","last_became_active":"2024-06-29T14:37:21.312184+0000","last_became_peered":"2024-06-29T14:37:21.312184+0000","last_unstale":"2024-06-29T14:37:22.331480+0000","last_undegraded":"2024-06-29T14:37:22.331480+0000","last_fullsized":"2024-06-29T14:37:22.331480+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:40.914474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.331482+0000","last_change":"2024-06-29T14:37:20.292457+0000","last_active":"2024-06-29T14:37:22.331482+0000","last_peered":"2024-06-29T14:37:22.331482+0000","last_clean":"2024-06-29T14:37:22.331482+0000","last_became_active":"2024-06-29T14:37:20.292042+0000","last_became_peered":"2024-06-29T14:37:20.292042+0000","last_unstale":"2024-06-29T14:37:22.331482+0000","last_undegraded":"2024-06-29T14:37:22.331482+0000","last_fullsized":"2024-06-29T14:37:22.331482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:43.310708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.297425+0000","last_change":"2024-06-29T14:37:21.318844+0000","last_active":"2024-06-29T14:37:22.297425+0000","last_peered":"2024-06-29T14:37:22.297425+0000","last_clean":"2024-06-29T14:37:22.297425+0000","last_became_active":"2024-06-29T14:37:21.318508+0000","last_became_peered":"2024-06-29T14:37:21.318508+0000","last_unstale":"2024-06-29T14:37:22.297425+0000","last_undegraded":"2024-06-29T14:37:22.297425+0000","last_fullsized":"2024-06-29T14:37:22.297425+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:29.326926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330878+0000","last_change":"2024-06-29T14:37:20.296547+0000","last_active":"2024-06-29T14:37:22.330878+0000","last_peered":"2024-06-29T14:37:22.330878+0000","last_clean":"2024-06-29T14:37:22.330878+0000","last_became_active":"2024-06-29T14:37:20.296259+0000","last_became_peered":"2024-06-29T14:37:20.296259+0000","last_unstale":"2024-06-29T14:37:22.330878+0000","last_undegraded":"2024-06-29T14:37:22.330878+0000","last_fullsized":"2024-06-29T14:37:22.330878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:13:59.493049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.299189+0000","last_change":"2024-06-29T14:37:21.310452+0000","last_active":"2024-06-29T14:37:22.299189+0000","last_peered":"2024-06-29T14:37:22.299189+0000","last_clean":"2024-06-29T14:37:22.299189+0000","last_became_active":"2024-06-29T14:37:21.308946+0000","last_became_peered":"2024-06-29T14:37:21.308946+0000","last_unstale":"2024-06-29T14:37:22.299189+0000","last_undegraded":"2024-06-29T14:37:22.299189+0000","last_fullsized":"2024-06-29T14:37:22.299189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.274005+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.274005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:51.302471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330864+0000","last_change":"2024-06-29T14:37:20.294167+0000","last_active":"2024-06-29T14:37:22.330864+0000","last_peered":"2024-06-29T14:37:22.330864+0000","last_clean":"2024-06-29T14:37:22.330864+0000","last_became_active":"2024-06-29T14:37:20.293843+0000","last_became_peered":"2024-06-29T14:37:20.293843+0000","last_unstale":"2024-06-29T14:37:22.330864+0000","last_undegraded":"2024-06-29T14:37:22.330864+0000","last_fullsized":"2024-06-29T14:37:22.330864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.268998+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.268998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:14:03.939141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":343,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.330733+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:37:22.330733+0000","last_peered":"2024-06-29T14:37:22.330733+0000","last_clean":"2024-06-29T14:37:22.330733+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:37:22.330733+0000","last_undegraded":"2024-06-29T14:37:22.330733+0000","last_fullsized":"2024-06-29T14:37:22.330733+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542333,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59180,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312660,"statfs":{"total":96636764160,"available":96576163840,"internally_reserved":0,"allocated":1671168,"data_stored":1125172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542334,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":1064960,"data_stored":534062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542333,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50272,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321568,"statfs":{"total":96636764160,"available":96585285632,"internally_reserved":0,"allocated":1064960,"data_stored":534050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542332,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339620,"statfs":{"total":96636764160,"available":96603770880,"internally_reserved":0,"allocated":1060864,"data_stored":528948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509630,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":1060864,"data_stored":534028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575038,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58588,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313252,"statfs":{"total":96636764160,"available":96576770048,"internally_reserved":0,"allocated":1060864,"data_stored":534028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509629,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":536404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509631,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1654784,"data_stored":1124396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:37:25.771 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:37:25.771 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:37:25.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.935+0000 7f032f28f640 1 -- 172.21.15.59:0/422274675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f032815c440 msgr2=0x7f03281564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.935+0000 7f032f28f640 1 --2- 172.21.15.59:0/422274675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f032815c440 0x7f03281564f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0320009830 tx=0x7f032002a510 comp rx=0 tx=0).stop 2024-06-29T14:37:25.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.935+0000 7f032f28f640 1 -- 172.21.15.59:0/422274675 shutdown_connections 2024-06-29T14:37:25.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.935+0000 7f032f28f640 1 --2- 172.21.15.59:0/422274675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03280d1c70 0x7f03280d0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.935+0000 7f032f28f640 1 --2- 172.21.15.59:0/422274675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f032815c440 0x7f03281564f0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.935+0000 7f032f28f640 1 -- 172.21.15.59:0/422274675 >> 172.21.15.59:0/422274675 conn(0x7f0328073e10 msgr2=0x7f0328074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.935+0000 7f032f28f640 1 -- 172.21.15.59:0/422274675 shutdown_connections 2024-06-29T14:37:25.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.935+0000 7f032f28f640 1 -- 172.21.15.59:0/422274675 wait complete. 2024-06-29T14:37:25.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.937+0000 7f032f28f640 1 Processor -- start 2024-06-29T14:37:25.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.937+0000 7f032f28f640 1 -- start start 2024-06-29T14:37:25.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032f28f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280d0980 0x7f0328144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032f28f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03280d1c70 0x7f0328146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032f28f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f032815c440 0x7f0328145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032f28f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f03280b6430 con 0x7f03280d0980 2024-06-29T14:37:25.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032f28f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03280b62b0 con 0x7f032815c440 2024-06-29T14:37:25.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032f28f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f03280b65b0 con 0x7f03280d1c70 2024-06-29T14:37:25.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280d0980 0x7f0328144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280d0980 0x7f0328144e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58780/0 (socket says 172.21.15.59:58780) 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 -- 172.21.15.59:0/3653297519 learned_addr learned my addr 172.21.15.59:0/3653297519 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032c82a640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03280d1c70 0x7f0328146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d82c640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f032815c440 0x7f0328145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 -- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03280d1c70 msgr2=0x7f0328146e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03280d1c70 0x7f0328146e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 -- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f032815c440 msgr2=0x7f0328145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f032815c440 0x7f0328145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 -- 172.21.15.59:0/3653297519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0328145db0 con 0x7f03280d0980 2024-06-29T14:37:25.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d82c640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f032815c440 0x7f0328145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:25.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f032d02b640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280d0980 0x7f0328144e00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f032002ab00 tx=0x7f0320033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f0325ffb640 1 -- 172.21.15.59:0/3653297519 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320031d60 con 0x7f03280d0980 2024-06-29T14:37:25.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.939+0000 7f0325ffb640 1 -- 172.21.15.59:0/3653297519 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0320005020 con 0x7f03280d0980 2024-06-29T14:37:25.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.940+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0328146090 con 0x7f03280d0980 2024-06-29T14:37:25.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.940+0000 7f0325ffb640 1 -- 172.21.15.59:0/3653297519 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320005470 con 0x7f03280d0980 2024-06-29T14:37:25.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.940+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0328146650 con 0x7f03280d0980 2024-06-29T14:37:25.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.941+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03280b6a60 con 0x7f03280d0980 2024-06-29T14:37:25.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.942+0000 7f0325ffb640 1 -- 172.21.15.59:0/3653297519 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f03200047a0 con 0x7f03280d0980 2024-06-29T14:37:25.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.942+0000 7f0325ffb640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f03140763f0 0x7f0314078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.942+0000 7f0325ffb640 1 -- 172.21.15.59:0/3653297519 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f03200b8d30 con 0x7f03280d0980 2024-06-29T14:37:25.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.942+0000 7f032c82a640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f03140763f0 0x7f0314078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.943+0000 7f032c82a640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f03140763f0 0x7f0314078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f03280b3f90 tx=0x7f031c001a70 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:25.946+0000 7f0325ffb640 1 -- 172.21.15.59:0/3653297519 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f032003f030 con 0x7f03280d0980 2024-06-29T14:37:26.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:37:26.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:37:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.214+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f0328153da0 con 0x7f03280d0980 2024-06-29T14:37:26.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.217+0000 7f0325ffb640 1 -- 172.21.15.59:0/3653297519 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f032008a2d0 con 0x7f03280d0980 2024-06-29T14:37:26.215 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:37:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:37:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T14:37:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T14:37:26.215 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:37:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T14:37:26.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f03140763f0 msgr2=0x7f0314078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f03140763f0 0x7f0314078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f03280b3f90 tx=0x7f031c001a70 comp rx=0 tx=0).stop 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280d0980 msgr2=0x7f0328144e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280d0980 0x7f0328144e00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f032002ab00 tx=0x7f0320033ce0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 shutdown_connections 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f03140763f0 0x7f0314078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f032815c440 0x7f0328145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03280d1c70 0x7f0328146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 --2- 172.21.15.59:0/3653297519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03280d0980 0x7f0328144e00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 >> 172.21.15.59:0/3653297519 conn(0x7f0328073e10 msgr2=0x7f032815dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 shutdown_connections 2024-06-29T14:37:26.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.220+0000 7f032f28f640 1 -- 172.21.15.59:0/3653297519 wait complete. 2024-06-29T14:37:26.229 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 -- 172.21.15.59:0/2666657997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0540b2040 msgr2=0x7fc05406e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 --2- 172.21.15.59:0/2666657997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0540b2040 0x7fc05406e300 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc0540b6a80 tx=0x7fc03c02a560 comp rx=0 tx=0).stop 2024-06-29T14:37:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 -- 172.21.15.59:0/2666657997 shutdown_connections 2024-06-29T14:37:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 --2- 172.21.15.59:0/2666657997 >> v2:172.21.15.153:3300/0 conn(0x7fc05406e840 0x7fc05415e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 --2- 172.21.15.59:0/2666657997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0540b2040 0x7fc05406e300 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 --2- 172.21.15.59:0/2666657997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc054144470 0x7fc0540b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 -- 172.21.15.59:0/2666657997 >> 172.21.15.59:0/2666657997 conn(0x7fc0540b06e0 msgr2=0x7fc0540b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 -- 172.21.15.59:0/2666657997 shutdown_connections 2024-06-29T14:37:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.350+0000 7fc05a43d640 1 -- 172.21.15.59:0/2666657997 wait complete. 2024-06-29T14:37:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.351+0000 7fc05a43d640 1 Processor -- start 2024-06-29T14:37:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.352+0000 7fc05a43d640 1 -- start start 2024-06-29T14:37:26.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc05a43d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc05406e840 0x7fc054154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc05a43d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0540b2040 0x7fc054138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc05a43d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc054144470 0x7fc05406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc05a43d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc05415ee30 con 0x7fc054144470 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc05a43d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc05415ecb0 con 0x7fc0540b2040 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc05a43d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc05415efb0 con 0x7fc05406e840 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc054144470 0x7fc05406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc054144470 0x7fc05406ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58800/0 (socket says 172.21.15.59:58800) 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 -- 172.21.15.59:0/1259406667 learned_addr learned my addr 172.21.15.59:0/1259406667 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc053fff640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc05406e840 0x7fc054154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc04bfff640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0540b2040 0x7fc054138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 -- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc05406e840 msgr2=0x7fc054154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc05406e840 0x7fc054154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 -- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0540b2040 msgr2=0x7fc054138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0540b2040 0x7fc054138940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 -- 172.21.15.59:0/1259406667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05414eff0 con 0x7fc054144470 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.353+0000 7fc0589da640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc054144470 0x7fc05406ca50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00cab0 tx=0x7fc04c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.354+0000 7fc0517fa640 1 -- 172.21.15.59:0/1259406667 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c007d30 con 0x7fc054144470 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.354+0000 7fc0517fa640 1 -- 172.21.15.59:0/1259406667 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc04c00f450 con 0x7fc054144470 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.354+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc054151860 con 0x7fc054144470 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.354+0000 7fc0517fa640 1 -- 172.21.15.59:0/1259406667 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c00f8a0 con 0x7fc054144470 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.354+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc054151e20 con 0x7fc054144470 2024-06-29T14:37:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.355+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0541515d0 con 0x7fc054144470 2024-06-29T14:37:26.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.356+0000 7fc0517fa640 1 -- 172.21.15.59:0/1259406667 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fc04c004b20 con 0x7fc054144470 2024-06-29T14:37:26.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.356+0000 7fc0517fa640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc034076420 0x7fc034078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:26.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.356+0000 7fc0517fa640 1 -- 172.21.15.59:0/1259406667 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7fc04c098670 con 0x7fc054144470 2024-06-29T14:37:26.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.356+0000 7fc053fff640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc034076420 0x7fc034078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:26.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.357+0000 7fc053fff640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc034076420 0x7fc034078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc04400a490 tx=0x7fc04400a420 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.360+0000 7fc0517fa640 1 -- 172.21.15.59:0/1259406667 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc04c014030 con 0x7fc054144470 2024-06-29T14:37:26.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.589+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc05415f470 con 0x7fc054144470 2024-06-29T14:37:26.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.590+0000 7fc0517fa640 1 -- 172.21.15.59:0/1259406667 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5230 (secure 0 0 0) 0x7fc04c061660 con 0x7fc054144470 2024-06-29T14:37:26.587 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:26.588 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":38,"btime":"2024-06-29T14:37:24:318813+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6529,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3719851385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3719851385},{"type":"v1","addr":"172.21.15.153:6836","nonce":3719851385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1280649067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1280649067},{"type":"v1","addr":"172.21.15.59:6833","nonce":1280649067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2847930906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2847930906},{"type":"v1","addr":"172.21.15.59:6835","nonce":2847930906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.672386+0000","modified":"2024-06-29T14:37:23.706036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"d","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/3891057790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3891057790},{"type":"v1","addr":"172.21.15.153:6837","nonce":3891057790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":4}]} 2024-06-29T14:37:26.588 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 38 2024-06-29T14:37:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc034076420 msgr2=0x7fc034078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc034076420 0x7fc034078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc04400a490 tx=0x7fc04400a420 comp rx=0 tx=0).stop 2024-06-29T14:37:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc054144470 msgr2=0x7fc05406ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc054144470 0x7fc05406ca50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00cab0 tx=0x7fc04c00cf70 comp rx=0 tx=0).stop 2024-06-29T14:37:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 shutdown_connections 2024-06-29T14:37:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc034076420 0x7fc034078930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc054144470 0x7fc05406ca50 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0540b2040 0x7fc054138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 --2- 172.21.15.59:0/1259406667 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc05406e840 0x7fc054154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.592+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 >> 172.21.15.59:0/1259406667 conn(0x7fc0540b06e0 msgr2=0x7fc054070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.593+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 shutdown_connections 2024-06-29T14:37:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:26.593+0000 7fc05a43d640 1 -- 172.21.15.59:0/1259406667 wait complete. 2024-06-29T14:37:27.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 38, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:37:20.672386+0000', 'modified': '2024-06-29T14:37:23.706036+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6864}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6864': {'gid': 6864, 'name': 'd', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6837/3891057790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6835', 'nonce': 3891057790}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 3891057790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6864, 'qdb_cluster': [6864]} 2024-06-29T14:37:27.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:37:27.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-06-29T14:37:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.132+0000 7f2df56e4640 1 -- 172.21.15.59:0/3634994292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ddc003aa0 msgr2=0x7f2ddc003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.132+0000 7f2df56e4640 1 --2- 172.21.15.59:0/3634994292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ddc003aa0 0x7f2ddc003f10 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2de0001d90 tx=0x7f2ddc005830 comp rx=0 tx=0).stop 2024-06-29T14:37:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.132+0000 7f2def7fe640 1 -- 172.21.15.59:0/3634994292 reap_dead start 2024-06-29T14:37:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.132+0000 7f2df56e4640 1 -- 172.21.15.59:0/3634994292 shutdown_connections 2024-06-29T14:37:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.132+0000 7f2df56e4640 1 -- 172.21.15.59:0/3634994292 >> 172.21.15.59:0/3634994292 conn(0x7f2df006c950 msgr2=0x7f2df0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.133+0000 7f2df56e4640 1 -- 172.21.15.59:0/3634994292 shutdown_connections 2024-06-29T14:37:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.133+0000 7f2df56e4640 1 -- 172.21.15.59:0/3634994292 wait complete. 2024-06-29T14:37:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.134+0000 7f2df56e4640 1 Processor -- start 2024-06-29T14:37:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.134+0000 7f2df56e4640 1 -- start start 2024-06-29T14:37:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2df56e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ddc003aa0 0x7f2df015a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2df56e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2df00aebc0 0x7f2df00aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2df56e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2df00a9a10 0x7f2df00a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2df56e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2df015cb10 con 0x7f2df00aebc0 2024-06-29T14:37:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2df56e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2df015c990 con 0x7f2df00a9a10 2024-06-29T14:37:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2df56e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2df00b5f90 con 0x7f2ddc003aa0 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2df00aebc0 0x7f2df00aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2df00aebc0 0x7f2df00aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58822/0 (socket says 172.21.15.59:58822) 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 -- 172.21.15.59:0/3972172289 learned_addr learned my addr 172.21.15.59:0/3972172289 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2def7fe640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2df00a9a10 0x7f2df00a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2deeffd640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ddc003aa0 0x7f2df015a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 -- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ddc003aa0 msgr2=0x7f2df015a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ddc003aa0 0x7f2df015a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 -- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2df00a9a10 msgr2=0x7f2df00a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2df00a9a10 0x7f2df00a9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 -- 172.21.15.59:0/3972172289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df00af510 con 0x7f2df00aebc0 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2deeffd640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ddc003aa0 0x7f2df015a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2df00aebc0 0x7f2df00aefd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8011b10 tx=0x7f2dd8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2de77fe640 1 -- 172.21.15.59:0/3972172289 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd801e070 con 0x7f2df00aebc0 2024-06-29T14:37:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.136+0000 7f2de77fe640 1 -- 172.21.15.59:0/3972172289 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2dd8004050 con 0x7f2df00aebc0 2024-06-29T14:37:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.137+0000 7f2de77fe640 1 -- 172.21.15.59:0/3972172289 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd800fea0 con 0x7f2df00aebc0 2024-06-29T14:37:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.137+0000 7f2df56e4640 1 -- 172.21.15.59:0/3972172289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df01f33d0 con 0x7f2df00aebc0 2024-06-29T14:37:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.137+0000 7f2df56e4640 1 -- 172.21.15.59:0/3972172289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df01f3990 con 0x7f2df00aebc0 2024-06-29T14:37:27.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.138+0000 7f2de77fe640 1 -- 172.21.15.59:0/3972172289 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f2dd8021070 con 0x7f2df00aebc0 2024-06-29T14:37:27.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.139+0000 7f2de77fe640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2dd00763f0 0x7f2dd0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.139+0000 7f2de77fe640 1 -- 172.21.15.59:0/3972172289 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f2dd8098640 con 0x7f2df00aebc0 2024-06-29T14:37:27.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.139+0000 7f2deeffd640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2dd00763f0 0x7f2dd0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.140+0000 7f2deeffd640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2dd00763f0 0x7f2dd0078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2de0032000 tx=0x7f2de0001510 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.140+0000 7f2dfc5a5740 1 Processor -- start 2024-06-29T14:37:27.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.140+0000 7f2dfc5a5740 1 -- start start 2024-06-29T14:37:27.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.141+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab9b5270 0x5645abd598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.141+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab98c3b0 0x5645abd571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.141+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd81a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.141+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5645abd1dbc0 con 0x5645ab994590 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.141+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5645abccc630 con 0x5645ab9b5270 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.141+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5645abcbbbd0 con 0x5645ab98c3b0 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd81a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd81a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58832/0 (socket says 172.21.15.59:58832) 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 -- 172.21.15.59:0/2132992086 learned_addr learned my addr 172.21.15.59:0/2132992086 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2deeffd640 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab9b5270 0x5645abd598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab98c3b0 0x5645abd571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 -- 172.21.15.59:0/2132992086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab98c3b0 msgr2=0x5645abd571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab98c3b0 0x5645abd571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 -- 172.21.15.59:0/2132992086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab9b5270 msgr2=0x5645abd598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab9b5270 0x5645abd598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 -- 172.21.15.59:0/2132992086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645abcb8ca0 con 0x5645ab994590 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2def7fe640 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd81a20 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2de8001050 tx=0x7f2de80018c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e495ff4b5502e949 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2de4ff9640 1 -- 172.21.15.59:0/2132992086 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de8003970 con 0x5645ab994590 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2de4ff9640 1 -- 172.21.15.59:0/2132992086 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2de8003b10 con 0x5645ab994590 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.142+0000 7f2de4ff9640 1 -- 172.21.15.59:0/2132992086 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de8011430 con 0x5645ab994590 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2132992086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 msgr2=0x5645abd81a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd81a20 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2de8001050 tx=0x7f2de80018c0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2132992086 shutdown_connections 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd81a20 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab98c3b0 0x5645abd571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/2132992086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab9b5270 0x5645abd598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2132992086 >> 172.21.15.59:0/2132992086 conn(0x5645ab9a93c0 msgr2=0x5645abd58f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2132992086 shutdown_connections 2024-06-29T14:37:27.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.143+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2132992086 wait complete. 2024-06-29T14:37:27.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.144+0000 7f2dfc5a5740 1 -- start start 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.144+0000 7f2dfc5a5740 1 Processor -- start 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab98c3b0 0x5645abd747a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab994590 0x5645abd761d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab9b5270 0x5645abd76710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5645abc59ae0 con 0x5645ab98c3b0 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5645abcbfc10 con 0x5645ab994590 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5645aba2f620 con 0x5645ab9b5270 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dfc5a5740 10 client.0 _ll_register_callbacks cb 0x5645abc66480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab98c3b0 0x5645abd747a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab98c3b0 0x5645abd747a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58834/0 (socket says 172.21.15.59:58834) 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 -- 172.21.15.59:0/3871937739 learned_addr learned my addr 172.21.15.59:0/3871937739 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2def7fe640 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab9b5270 0x5645abd76710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab994590 0x5645abd761d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 -- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab9b5270 msgr2=0x5645abd76710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab9b5270 0x5645abd76710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 -- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab994590 msgr2=0x5645abd761d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab994590 0x5645abd761d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 -- 172.21.15.59:0/3871937739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645aba20850 con 0x5645ab98c3b0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2deeffd640 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab98c3b0 0x5645abd747a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2de0001e80 tx=0x7f2ddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d20055eee01dc0c server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dcdffb640 1 -- 172.21.15.59:0/3871937739 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de0053060 con 0x5645ab98c3b0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dcdffb640 1 -- 172.21.15.59:0/3871937739 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f2de00411d0 con 0x5645ab98c3b0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.146+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/3871937739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2de8003b10 con 0x5645ab98c3b0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dcdffb640 1 -- 172.21.15.59:0/3871937739 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2de0041540 con 0x5645ab98c3b0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dfc5a5740 10 client.7008 fetch_fsmap learned FSMap version 38 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/3871937739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=2+}) -- 0x5645aba976b0 con 0x5645ab98c3b0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dcdffb640 1 -- 172.21.15.59:0/3871937739 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2de8003b10 con 0x5645ab98c3b0 2024-06-29T14:37:27.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dcdffb640 1 -- 172.21.15.59:0/3871937739 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 38) ==== 3502+0+0 (secure 0 0 0) 0x7f2de00417d0 con 0x5645ab98c3b0 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dfc5a5740 10 client.7008 fetch_fsmap finished waiting for FSMap version 38 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dfc5a5740 10 client.7008 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dfc5a5740 20 client.7008 populate_metadata read hostname 'smithi059' 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] conn(0x5645ab973e40 0x5645abd77040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dfc5a5740 4 client.7008 mds_command: new command op to 6864 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/3871937739 --> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645abcce580 con 0x5645ab973e40 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.147+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] conn(0x5645ab973e40 0x5645abd77040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.148+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] conn(0x5645ab973e40 0x5645abd77040 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9a0c249ebe6d2fd server_cookie=288af4a7d35d1bcd in_seq=0 out_seq=0 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.148+0000 7f2dcdffb640 10 client.7008 ms_handle_connect on v2:172.21.15.153:6835/3891057790 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.148+0000 7f2dcdffb640 1 -- 172.21.15.59:0/3871937739 <== mds.0 v2:172.21.15.153:6835/3891057790 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5645abcce580 con 0x5645ab973e40 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.148+0000 7f2dcdffb640 10 client.7008 handle_command_reply: tid=0 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.148+0000 7f2dfc5a5740 1 client.7008 shutdown 2024-06-29T14:37:27.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/3871937739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab98c3b0 msgr2=0x5645abd747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab98c3b0 0x5645abd747a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2de0001e80 tx=0x7f2ddc005830 comp rx=0 tx=0).stop 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/3871937739 shutdown_connections 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] conn(0x5645ab973e40 0x5645abd77040 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab9b5270 0x5645abd76710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab994590 0x5645abd761d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dcdffb640 0 client.7008 ms_handle_reset on v2:172.21.15.153:6835/3891057790 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/3871937739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab98c3b0 0x5645abd747a0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/3871937739 >> 172.21.15.59:0/3871937739 conn(0x5645ab9a93c0 msgr2=0x5645abd567b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/3871937739 shutdown_connections 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/3871937739 wait complete. 2024-06-29T14:37:27.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.149+0000 7f2dfc5a5740 20 client.7008 trim_cache size 0 max 16384 2024-06-29T14:37:27.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.168+0000 7f2dfc5a5740 1 Processor -- start 2024-06-29T14:37:27.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.168+0000 7f2dfc5a5740 1 -- start start 2024-06-29T14:37:27.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.169+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 0x5645abd81780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.169+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abd9e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.169+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd9eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.169+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2df006f3e0 con 0x5645ab994590 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.169+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2df00ad5c0 con 0x5645ab98c3b0 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.169+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2df006ed90 con 0x5645ab973e40 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd9eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd9eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58838/0 (socket says 172.21.15.59:58838) 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 -- 172.21.15.59:0/2234776700 learned_addr learned my addr 172.21.15.59:0/2234776700 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2deeffd640 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 0x5645abd81780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abd9e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 -- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 msgr2=0x5645abd81780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 0x5645abd81780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 -- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 msgr2=0x5645abd9e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abd9e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 -- 172.21.15.59:0/2234776700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645abd76dd0 con 0x5645ab994590 2024-06-29T14:37:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abd9e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2def7fe640 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd9eb10 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2de8006770 tx=0x7f2de8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd22a41606227d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2dcffff640 1 -- 172.21.15.59:0/2234776700 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de8003a70 con 0x5645ab994590 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2dcffff640 1 -- 172.21.15.59:0/2234776700 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2de8003c10 con 0x5645ab994590 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.170+0000 7f2dcffff640 1 -- 172.21.15.59:0/2234776700 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de801b430 con 0x5645ab994590 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2234776700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 msgr2=0x5645abd9eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd9eb10 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2de8006770 tx=0x7f2de8000fe0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2234776700 shutdown_connections 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd9eb10 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abd9e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/2234776700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 0x5645abd81780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2234776700 >> 172.21.15.59:0/2234776700 conn(0x5645ab9a93c0 msgr2=0x5645abd56f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2234776700 shutdown_connections 2024-06-29T14:37:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.171+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/2234776700 wait complete. 2024-06-29T14:37:27.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.172+0000 7f2dfc5a5740 1 -- start start 2024-06-29T14:37:27.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.172+0000 7f2dfc5a5740 1 Processor -- start 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 0x5645abe21840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abe255d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2dfc5a5740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd55290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5645abd76c50 con 0x5645ab994590 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5645abd8e0e0 con 0x5645ab98c3b0 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2dfc5a5740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5645abd833f0 con 0x5645ab973e40 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2dfc5a5740 10 client.0 _ll_register_callbacks cb 0x7f2ddc0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2def7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd55290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2def7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd55290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58840/0 (socket says 172.21.15.59:58840) 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2def7fe640 1 -- 172.21.15.59:0/4040275500 learned_addr learned my addr 172.21.15.59:0/4040275500 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2deeffd640 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 0x5645abe21840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.173+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abe255d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2def7fe640 1 -- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 msgr2=0x5645abe21840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2def7fe640 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 0x5645abe21840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2def7fe640 1 -- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 msgr2=0x5645abe255d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2def7fe640 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abe255d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2def7fe640 1 -- 172.21.15.59:0/4040275500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645abd75c50 con 0x5645ab994590 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2def7fe640 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd55290 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2de80064d0 tx=0x7f2de8004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4312e15a0cf0853 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dcdffb640 1 -- 172.21.15.59:0/4040275500 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de8011630 con 0x5645ab994590 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dcdffb640 1 -- 172.21.15.59:0/4040275500 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f2de80117d0 con 0x5645ab994590 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/4040275500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2de8003c10 con 0x5645ab994590 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dcdffb640 1 -- 172.21.15.59:0/4040275500 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2de8038050 con 0x5645ab994590 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dfc5a5740 10 client.7014 fetch_fsmap learned FSMap version 38 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/4040275500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=2+}) -- 0x5645abe21d80 con 0x5645ab994590 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dcdffb640 1 -- 172.21.15.59:0/4040275500 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2de8003c10 con 0x5645ab994590 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.174+0000 7f2dcdffb640 1 -- 172.21.15.59:0/4040275500 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 38) ==== 3502+0+0 (secure 0 0 0) 0x7f2de803b020 con 0x5645ab994590 2024-06-29T14:37:27.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2dfc5a5740 10 client.7014 fetch_fsmap finished waiting for FSMap version 38 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2dfc5a5740 10 client.7014 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2dfc5a5740 20 client.7014 populate_metadata read hostname 'smithi059' 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] conn(0x5645ab949280 0x5645abdc7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2dfc5a5740 4 client.7014 mds_command: new command op to 6864 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/4040275500 --> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] -- command(tid 0: {"prefix": "status"}) -- 0x5645abd920e0 con 0x5645ab949280 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2deeffd640 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] conn(0x5645ab949280 0x5645abdc7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2dcdffb640 10 client.7014 ms_handle_connect on v2:172.21.15.153:6835/3891057790 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.175+0000 7f2deeffd640 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] conn(0x5645ab949280 0x5645abdc7770 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6925472cf8344e1b server_cookie=974ec2578a395a8a in_seq=0 out_seq=0 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dcdffb640 1 -- 172.21.15.59:0/4040275500 <== mds.0 v2:172.21.15.153:6835/3891057790 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5645abd920e0 con 0x5645ab949280 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dcdffb640 10 client.7014 handle_command_reply: tid=0 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 client.7014 shutdown 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/4040275500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 msgr2=0x5645abd55290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd55290 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2de80064d0 tx=0x7f2de8004dd0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/4040275500 shutdown_connections 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:6835/3891057790,v1:172.21.15.153:6837/3891057790] conn(0x5645ab949280 0x5645abdc7770 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5645ab994590 0x5645abd55290 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5645ab98c3b0 0x5645abe255d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dcdffb640 0 client.7014 ms_handle_reset on v2:172.21.15.153:6835/3891057790 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 --2- 172.21.15.59:0/4040275500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5645ab973e40 0x5645abe21840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/4040275500 >> 172.21.15.59:0/4040275500 conn(0x5645ab9a93c0 msgr2=0x5645abda1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/4040275500 shutdown_connections 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 1 -- 172.21.15.59:0/4040275500 wait complete. 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2dfc5a5740 20 client.7014 trim_cache size 0 max 16384 2024-06-29T14:37:27.178 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 6864, 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8686524279999999, 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 38, 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 88, 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 88, 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.505018594999999 2024-06-29T14:37:27.179 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:37:27.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 -- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2dd00763f0 msgr2=0x7f2dd0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2dd00763f0 0x7f2dd0078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2de0032000 tx=0x7f2de0001510 comp rx=0 tx=0).stop 2024-06-29T14:37:27.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 -- 172.21.15.59:0/3972172289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2df00aebc0 msgr2=0x7f2df00aefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2df00aebc0 0x7f2df00aefd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8011b10 tx=0x7f2dd8009e90 comp rx=0 tx=0).stop 2024-06-29T14:37:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 -- 172.21.15.59:0/3972172289 shutdown_connections 2024-06-29T14:37:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f2dd00763f0 0x7f2dd0078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2df00a9a10 0x7f2df00a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2df00aebc0 0x7f2df00aefd0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 --2- 172.21.15.59:0/3972172289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ddc003aa0 0x7f2df015a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 -- 172.21.15.59:0/3972172289 >> 172.21.15.59:0/3972172289 conn(0x7f2df006c950 msgr2=0x7f2df00d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.176+0000 7f2df56e4640 1 -- 172.21.15.59:0/3972172289 shutdown_connections 2024-06-29T14:37:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.177+0000 7f2df56e4640 1 -- 172.21.15.59:0/3972172289 wait complete. 2024-06-29T14:37:27.185 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:37:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.305+0000 7f07bed2e640 1 -- 172.21.15.59:0/1851442802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07a00051c0 msgr2=0x7f07a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.305+0000 7f07bed2e640 1 --2- 172.21.15.59:0/1851442802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07a00051c0 0x7f07a000b9c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f07b8072d60 tx=0x7f07a0005830 comp rx=0 tx=0).stop 2024-06-29T14:37:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.305+0000 7f07bd2cb640 1 -- 172.21.15.59:0/1851442802 reap_dead start 2024-06-29T14:37:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.306+0000 7f07bed2e640 1 -- 172.21.15.59:0/1851442802 shutdown_connections 2024-06-29T14:37:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.306+0000 7f07bed2e640 1 -- 172.21.15.59:0/1851442802 >> 172.21.15.59:0/1851442802 conn(0x7f07b80b06e0 msgr2=0x7f07b80b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.306+0000 7f07bed2e640 1 -- 172.21.15.59:0/1851442802 shutdown_connections 2024-06-29T14:37:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.306+0000 7f07bed2e640 1 -- 172.21.15.59:0/1851442802 wait complete. 2024-06-29T14:37:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.307+0000 7f07bed2e640 1 Processor -- start 2024-06-29T14:37:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.307+0000 7f07bed2e640 1 -- start start 2024-06-29T14:37:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bed2e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a00051c0 0x7f07b81f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bed2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b8138b20 0x7f07b8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bed2e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b8138f30 0x7f07b815a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bed2e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f07b806e300 con 0x7f07b8138b20 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bed2e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f07b80b2460 con 0x7f07b8138f30 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bed2e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f07b806e480 con 0x7f07a00051c0 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07b7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b8138b20 0x7f07b8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07b7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b8138b20 0x7f07b8159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58866/0 (socket says 172.21.15.59:58866) 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07b7fff640 1 -- 172.21.15.59:0/804945357 learned_addr learned my addr 172.21.15.59:0/804945357 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bcaca640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a00051c0 0x7f07b81f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bd2cb640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b8138f30 0x7f07b815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07b7fff640 1 -- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a00051c0 msgr2=0x7f07b81f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07b7fff640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a00051c0 0x7f07b81f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07b7fff640 1 -- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b8138f30 msgr2=0x7f07b815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07b7fff640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b8138f30 0x7f07b815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07b7fff640 1 -- 172.21.15.59:0/804945357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b81f7c10 con 0x7f07b8138b20 2024-06-29T14:37:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.309+0000 7f07bd2cb640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b8138f30 0x7f07b815a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.310+0000 7f07b7fff640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b8138b20 0x7f07b8159c30 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f07ac011ae0 tx=0x7f07ac009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.310+0000 7f07b57fa640 1 -- 172.21.15.59:0/804945357 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac01e070 con 0x7f07b8138b20 2024-06-29T14:37:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.310+0000 7f07b57fa640 1 -- 172.21.15.59:0/804945357 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07ac00fe70 con 0x7f07b8138b20 2024-06-29T14:37:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.310+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b81f7ef0 con 0x7f07b8138b20 2024-06-29T14:37:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.310+0000 7f07b57fa640 1 -- 172.21.15.59:0/804945357 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac004290 con 0x7f07b8138b20 2024-06-29T14:37:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.310+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b81e57a0 con 0x7f07b8138b20 2024-06-29T14:37:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.311+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b80b67f0 con 0x7f07b8138b20 2024-06-29T14:37:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.312+0000 7f07b57fa640 1 -- 172.21.15.59:0/804945357 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f07ac021070 con 0x7f07b8138b20 2024-06-29T14:37:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.312+0000 7f07b57fa640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f07a40763f0 0x7f07a4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.312+0000 7f07b57fa640 1 -- 172.21.15.59:0/804945357 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f07ac098650 con 0x7f07b8138b20 2024-06-29T14:37:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.312+0000 7f07bcaca640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f07a40763f0 0x7f07a4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.313+0000 7f07bcaca640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f07a40763f0 0x7f07a4078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f07a8000c00 tx=0x7f07a8006b70 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.317+0000 7f07b57fa640 1 -- 172.21.15.59:0/804945357 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f07ac0615e0 con 0x7f07b8138b20 2024-06-29T14:37:27.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.546+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f07b815e450 con 0x7f07b8138b20 2024-06-29T14:37:27.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.547+0000 7f07b57fa640 1 -- 172.21.15.59:0/804945357 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v38) ==== 53+0+73 (secure 0 0 0) 0x7f07ac0658e0 con 0x7f07b8138b20 2024-06-29T14:37:27.544 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:37:27.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.548+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f07a40763f0 msgr2=0x7f07a4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.548+0000 7f07bed2e640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f07a40763f0 0x7f07a4078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f07a8000c00 tx=0x7f07a8006b70 comp rx=0 tx=0).stop 2024-06-29T14:37:27.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.548+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b8138b20 msgr2=0x7f07b8159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.548+0000 7f07bed2e640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b8138b20 0x7f07b8159c30 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f07ac011ae0 tx=0x7f07ac009e90 comp rx=0 tx=0).stop 2024-06-29T14:37:27.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.549+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 shutdown_connections 2024-06-29T14:37:27.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.549+0000 7f07bed2e640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f07a40763f0 0x7f07a4078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.549+0000 7f07bed2e640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b8138f30 0x7f07b815a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.549+0000 7f07bed2e640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07b8138b20 0x7f07b8159c30 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.549+0000 7f07bed2e640 1 --2- 172.21.15.59:0/804945357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07a00051c0 0x7f07b81f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.549+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 >> 172.21.15.59:0/804945357 conn(0x7f07b80b06e0 msgr2=0x7f07b8070910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.549+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 shutdown_connections 2024-06-29T14:37:27.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27.549+0000 7f07bed2e640 1 -- 172.21.15.59:0/804945357 wait complete. 2024-06-29T14:37:27.557 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:37:27.557 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:37:27.557 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:37:27.558 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:37:27.558 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:27.558 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:37:27.558 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:37:27.558 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:37:27.558 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:27.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.558 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:37:27.574 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:37:27.574 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:37:27.574 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:37:27.574 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 41803sec preferred_lft 41803sec 2024-06-29T14:37:27.575 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:37:27.576 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T14:37:27.577 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.577 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:37:27.577 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:37:27.577 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T14:37:27.577 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.577 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:37:27.640 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:37:27.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.641 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:37:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:37:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:37:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:37:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:37:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:37:27.699 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:37:27.699 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:37:27.699 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:37:27.699 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:37:27.699 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:37:27.700 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:37:27.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:37:27.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:37:27.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.829 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:37:27.846 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:37:27.846 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:37:27.846 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:27.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.846 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:37:27.912 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:37:27.912 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:27.968 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:27.968 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:37:27.968 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:28.024 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:37:28.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:28.024 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:37:28.119 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:37:28.119 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:37:28.119 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:37:28.130 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:37:28.130 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:37:28.130 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:37:28.130 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:37:28.130 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:37:28.136 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:37:28.136 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:37:28.136 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:37:28.136 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:37:28.137 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:37:28.139 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:37:28.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:28.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:28.165 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:37:28.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:37:28.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:37:28.270 INFO:teuthology.orchestra.run.smithi059.stdout:client7023 2024-06-29T14:37:28.270 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:28.271 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:28.287 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:37:28.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.450+0000 7fc46223f640 1 -- 172.21.15.59:0/2118535860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0b2fa0 msgr2=0x7fc45c0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.450+0000 7fc46223f640 1 --2- 172.21.15.59:0/2118535860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0b2fa0 0x7fc45c0d1520 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc450005fd0 tx=0x7fc45002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.450+0000 7fc46223f640 1 -- 172.21.15.59:0/2118535860 shutdown_connections 2024-06-29T14:37:28.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.450+0000 7fc46223f640 1 --2- 172.21.15.59:0/2118535860 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc45c0d1a60 0x7fc45c0b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.450+0000 7fc46223f640 1 --2- 172.21.15.59:0/2118535860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0b2fa0 0x7fc45c0d1520 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.450+0000 7fc46223f640 1 -- 172.21.15.59:0/2118535860 >> 172.21.15.59:0/2118535860 conn(0x7fc45c073e10 msgr2=0x7fc45c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:28.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.450+0000 7fc46223f640 1 -- 172.21.15.59:0/2118535860 shutdown_connections 2024-06-29T14:37:28.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.450+0000 7fc46223f640 1 -- 172.21.15.59:0/2118535860 wait complete. 2024-06-29T14:37:28.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.452+0000 7fc46223f640 1 Processor -- start 2024-06-29T14:37:28.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.452+0000 7fc46223f640 1 -- start start 2024-06-29T14:37:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc46223f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc45c0b2270 0x7fc45c1475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc46223f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc45c0b2fa0 0x7fc45c147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc46223f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0d1a60 0x7fc45c146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc46223f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c0b6400 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc46223f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c0b6280 con 0x7fc45c0d1a60 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc46223f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc45c0b6580 con 0x7fc45c0b2270 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc45affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc45c0b2fa0 0x7fc45c147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc45affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc45c0b2fa0 0x7fc45c147b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58880/0 (socket says 172.21.15.59:58880) 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc45affd640 1 -- 172.21.15.59:0/1739925271 learned_addr learned my addr 172.21.15.59:0/1739925271 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc45b7fe640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc45c0b2270 0x7fc45c1475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.453+0000 7fc45bfff640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0d1a60 0x7fc45c146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc45affd640 1 -- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc45c0b2270 msgr2=0x7fc45c1475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc45affd640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc45c0b2270 0x7fc45c1475d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc45affd640 1 -- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0d1a60 msgr2=0x7fc45c146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc45affd640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0d1a60 0x7fc45c146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc45affd640 1 -- 172.21.15.59:0/1739925271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45c0a98e0 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc45affd640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc45c0b2fa0 0x7fc45c147b10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc450000c00 tx=0x7fc4500035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc443fff640 1 -- 172.21.15.59:0/1739925271 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45003e030 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc443fff640 1 -- 172.21.15.59:0/1739925271 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc450003b00 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc45c145670 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc443fff640 1 -- 172.21.15.59:0/1739925271 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc450035040 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.454+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc45c145c30 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.455+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc45c15eb00 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.456+0000 7fc443fff640 1 -- 172.21.15.59:0/1739925271 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fc450041030 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.456+0000 7fc443fff640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc4480763f0 0x7fc448078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.456+0000 7fc443fff640 1 -- 172.21.15.59:0/1739925271 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7fc4500b7e50 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.457+0000 7fc45b7fe640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc4480763f0 0x7fc448078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.457+0000 7fc45b7fe640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc4480763f0 0x7fc448078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0b1fc0 tx=0x7fc44c009040 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.461+0000 7fc443fff640 1 -- 172.21.15.59:0/1739925271 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc450080db0 con 0x7fc45c0b2fa0 2024-06-29T14:37:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.678+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 --> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fc45c15ed90 con 0x7fc4480763f0 2024-06-29T14:37:28.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.680+0000 7fc443fff640 1 -- 172.21.15.59:0/1739925271 <== mgr.6316 v2:172.21.15.153:6832/4269929333 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fc45c15ed90 con 0x7fc4480763f0 2024-06-29T14:37:28.678 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:37:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T14:37:28.678 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:28.678 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:37:28.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc4480763f0 msgr2=0x7fc448078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc4480763f0 0x7fc448078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0b1fc0 tx=0x7fc44c009040 comp rx=0 tx=0).stop 2024-06-29T14:37:28.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc45c0b2fa0 msgr2=0x7fc45c147b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc45c0b2fa0 0x7fc45c147b10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc450000c00 tx=0x7fc4500035f0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 shutdown_connections 2024-06-29T14:37:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fc4480763f0 0x7fc448078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0d1a60 0x7fc45c146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc45c0b2fa0 0x7fc45c147b10 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 --2- 172.21.15.59:0/1739925271 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc45c0b2270 0x7fc45c1475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 >> 172.21.15.59:0/1739925271 conn(0x7fc45c073e10 msgr2=0x7fc45c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 shutdown_connections 2024-06-29T14:37:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.683+0000 7fc46223f640 1 -- 172.21.15.59:0/1739925271 wait complete. 2024-06-29T14:37:28.691 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:37:28.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.815+0000 7f7a34f31640 1 -- 172.21.15.59:0/4056030124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a1c003ac0 msgr2=0x7f7a1c003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.815+0000 7f7a34f31640 1 --2- 172.21.15.59:0/4056030124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a1c003ac0 0x7f7a1c003f30 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009ed0 tx=0x7f7a1c005830 comp rx=0 tx=0).stop 2024-06-29T14:37:28.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.815+0000 7f7a34f31640 1 -- 172.21.15.59:0/4056030124 shutdown_connections 2024-06-29T14:37:28.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.815+0000 7f7a34f31640 1 --2- 172.21.15.59:0/4056030124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a1c003ac0 0x7f7a1c003f30 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.815+0000 7f7a34f31640 1 -- 172.21.15.59:0/4056030124 >> 172.21.15.59:0/4056030124 conn(0x7f7a300d45b0 msgr2=0x7f7a300d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:28.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.815+0000 7f7a34f31640 1 -- 172.21.15.59:0/4056030124 shutdown_connections 2024-06-29T14:37:28.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.815+0000 7f7a34f31640 1 -- 172.21.15.59:0/4056030124 wait complete. 2024-06-29T14:37:28.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.817+0000 7f7a34f31640 1 Processor -- start 2024-06-29T14:37:28.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.817+0000 7f7a34f31640 1 -- start start 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a34f31640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a1c003ac0 0x7f7a300cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a34f31640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a301614d0 0x7f7a300b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a34f31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a300b90e0 0x7f7a30158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a34f31640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7a30161a80 con 0x7f7a300b90e0 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a34f31640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300ce7c0 con 0x7f7a1c003ac0 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a34f31640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7a30161c00 con 0x7f7a301614d0 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a2ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a300b90e0 0x7f7a30158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a2ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a300b90e0 0x7f7a30158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58898/0 (socket says 172.21.15.59:58898) 2024-06-29T14:37:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a2ed9d640 1 -- 172.21.15.59:0/1495391205 learned_addr learned my addr 172.21.15.59:0/1495391205 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a2e59c640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a1c003ac0 0x7f7a300cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.818+0000 7f7a2dd9b640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a301614d0 0x7f7a300b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a2ed9d640 1 -- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a301614d0 msgr2=0x7f7a300b8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a2ed9d640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a301614d0 0x7f7a300b8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a2ed9d640 1 -- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a1c003ac0 msgr2=0x7f7a300cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a2ed9d640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a1c003ac0 0x7f7a300cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a2ed9d640 1 -- 172.21.15.59:0/1495391205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a300b94f0 con 0x7f7a300b90e0 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a2ed9d640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a300b90e0 0x7f7a30158330 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7a300ce310 tx=0x7f7a2800e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a26ffd640 1 -- 172.21.15.59:0/1495391205 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28011660 con 0x7f7a300b90e0 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a26ffd640 1 -- 172.21.15.59:0/1495391205 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a28016070 con 0x7f7a300b90e0 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a26ffd640 1 -- 172.21.15.59:0/1495391205 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28011b90 con 0x7f7a300b90e0 2024-06-29T14:37:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a300cb0a0 con 0x7f7a300b90e0 2024-06-29T14:37:28.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.819+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a300cb660 con 0x7f7a300b90e0 2024-06-29T14:37:28.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.820+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a300cae10 con 0x7f7a300b90e0 2024-06-29T14:37:28.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.821+0000 7f7a26ffd640 1 -- 172.21.15.59:0/1495391205 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f7a2801b460 con 0x7f7a300b90e0 2024-06-29T14:37:28.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.822+0000 7f7a26ffd640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7a10076420 0x7f7a10078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.822+0000 7f7a26ffd640 1 -- 172.21.15.59:0/1495391205 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f7a28097ad0 con 0x7f7a300b90e0 2024-06-29T14:37:28.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.822+0000 7f7a2e59c640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7a10076420 0x7f7a10078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.822+0000 7f7a2e59c640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7a10076420 0x7f7a10078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a20035000 tx=0x7f7a20004a40 comp rx=0 tx=0).ready entity=mgr.6316 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:28.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:28.826+0000 7f7a26ffd640 1 -- 172.21.15.59:0/1495391205 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7a300b94f0 con 0x7f7a300b90e0 2024-06-29T14:37:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.070+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f7a30162160 con 0x7f7a300b90e0 2024-06-29T14:37:29.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.340+0000 7f7a26ffd640 1 -- 172.21.15.59:0/1495391205 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v31) ==== 92+0+0 (secure 0 0 0) 0x7f7a28069070 con 0x7f7a300b90e0 2024-06-29T14:37:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7a10076420 msgr2=0x7f7a10078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7a10076420 0x7f7a10078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a20035000 tx=0x7f7a20004a40 comp rx=0 tx=0).stop 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a300b90e0 msgr2=0x7f7a30158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a300b90e0 0x7f7a30158330 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7a300ce310 tx=0x7f7a2800e770 comp rx=0 tx=0).stop 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 shutdown_connections 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f7a10076420 0x7f7a10078930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a300b90e0 0x7f7a30158330 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a301614d0 0x7f7a300b8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 --2- 172.21.15.59:0/1495391205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a1c003ac0 0x7f7a300cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 >> 172.21.15.59:0/1495391205 conn(0x7f7a300d45b0 msgr2=0x7f7a3015c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 shutdown_connections 2024-06-29T14:37:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.344+0000 7f7a34f31640 1 -- 172.21.15.59:0/1495391205 wait complete. 2024-06-29T14:37:29.355 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:37:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.500+0000 7f77df24e640 1 -- 172.21.15.59:0/3950567744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 msgr2=0x7f77d806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.500+0000 7f77df24e640 1 --2- 172.21.15.59:0/3950567744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 0x7f77d806ca70 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f77d80ccee0 tx=0x7f77d002aac0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.500+0000 7f77df24e640 1 -- 172.21.15.59:0/3950567744 shutdown_connections 2024-06-29T14:37:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.500+0000 7f77df24e640 1 --2- 172.21.15.59:0/3950567744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 0x7f77d806ca70 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.500+0000 7f77df24e640 1 --2- 172.21.15.59:0/3950567744 >> v2:172.21.15.153:3300/0 conn(0x7f77d80d02f0 0x7f77d80cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.500+0000 7f77df24e640 1 -- 172.21.15.59:0/3950567744 >> 172.21.15.59:0/3950567744 conn(0x7f77d80b2ab0 msgr2=0x7f77d80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.500+0000 7f77df24e640 1 -- 172.21.15.59:0/3950567744 shutdown_connections 2024-06-29T14:37:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.500+0000 7f77df24e640 1 -- 172.21.15.59:0/3950567744 wait complete. 2024-06-29T14:37:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.501+0000 7f77df24e640 1 Processor -- start 2024-06-29T14:37:29.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.502+0000 7f77df24e640 1 -- start start 2024-06-29T14:37:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77df24e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 0x7f77d815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77df24e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d81448e0 0x7f77d813c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77df24e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d815dcd0 0x7f77d815e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77df24e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f77d806d380 con 0x7f77d80cd7c0 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77df24e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f77d806d200 con 0x7f77d81448e0 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77df24e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f77d806d500 con 0x7f77d815dcd0 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77dcfea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 0x7f77d815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77dcfea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 0x7f77d815c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58942/0 (socket says 172.21.15.59:58942) 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77dcfea640 1 -- 172.21.15.59:0/2224802205 learned_addr learned my addr 172.21.15.59:0/2224802205 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77d7fff640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d81448e0 0x7f77d813c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77dd7eb640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d815dcd0 0x7f77d815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.503+0000 7f77dcfea640 1 -- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d815dcd0 msgr2=0x7f77d815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77dcfea640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d815dcd0 0x7f77d815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77dcfea640 1 -- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d81448e0 msgr2=0x7f77d813c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77dcfea640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d81448e0 0x7f77d813c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77dcfea640 1 -- 172.21.15.59:0/2224802205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d815c720 con 0x7f77d80cd7c0 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77d7fff640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d81448e0 0x7f77d813c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77dcfea640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 0x7f77d815c1e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00c3a0 tx=0x7f77cc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77d57fa640 1 -- 172.21.15.59:0/2224802205 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc017070 con 0x7f77d80cd7c0 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77d57fa640 1 -- 172.21.15.59:0/2224802205 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77cc0053c0 con 0x7f77d80cd7c0 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d815cc30 con 0x7f77d80cd7c0 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77d57fa640 1 -- 172.21.15.59:0/2224802205 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc003850 con 0x7f77d80cd7c0 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.504+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d815d1f0 con 0x7f77d80cd7c0 2024-06-29T14:37:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.505+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d8069ca0 con 0x7f77d80cd7c0 2024-06-29T14:37:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.506+0000 7f77d57fa640 1 -- 172.21.15.59:0/2224802205 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 31) ==== 98320+0+0 (secure 0 0 0) 0x7f77cc01a070 con 0x7f77d80cd7c0 2024-06-29T14:37:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.506+0000 7f77d57fa640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f77bc076470 0x7f77bc078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.507+0000 7f77d7fff640 1 -- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f77bc076470 msgr2=0x7f77bc078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/4269929333 2024-06-29T14:37:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.507+0000 7f77d57fa640 1 -- 172.21.15.59:0/2224802205 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f77cc096e40 con 0x7f77d80cd7c0 2024-06-29T14:37:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.507+0000 7f77d7fff640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f77bc076470 0x7f77bc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:29.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.511+0000 7f77d57fa640 1 -- 172.21.15.59:0/2224802205 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f77d8069ca0 con 0x7f77d80cd7c0 2024-06-29T14:37:29.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.704+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f77d806f750 con 0x7f77d80cd7c0 2024-06-29T14:37:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.708+0000 7f77d57fa640 1 -- 172.21.15.59:0/2224802205 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v37)=0 v37) ==== 138+0+0 (secure 0 0 0) 0x7f77d806f750 con 0x7f77d80cd7c0 2024-06-29T14:37:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.708+0000 7f77d7fff640 1 -- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f77bc076470 msgr2=0x7f77bc078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/4269929333 2024-06-29T14:37:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.708+0000 7f77d7fff640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f77bc076470 0x7f77bc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f77bc076470 msgr2=0x7f77bc078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f77bc076470 0x7f77bc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 msgr2=0x7f77d815c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 0x7f77d815c1e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00c3a0 tx=0x7f77cc00c860 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 shutdown_connections 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f77bc076470 0x7f77bc078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77d815dcd0 0x7f77d815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d81448e0 0x7f77d813c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 --2- 172.21.15.59:0/2224802205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77d80cd7c0 0x7f77d815c1e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 >> 172.21.15.59:0/2224802205 conn(0x7f77d80b2ab0 msgr2=0x7f77d813e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 shutdown_connections 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.710+0000 7f77df24e640 1 -- 172.21.15.59:0/2224802205 wait complete. 2024-06-29T14:37:29.719 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 -- 172.21.15.59:0/724247350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2270 msgr2=0x7fcca815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 --2- 172.21.15.59:0/724247350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2270 0x7fcca815bc20 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fcca80b2d10 tx=0x7fcc9802a560 comp rx=0 tx=0).stop 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 -- 172.21.15.59:0/724247350 shutdown_connections 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 --2- 172.21.15.59:0/724247350 >> v2:172.21.15.153:3300/0 conn(0x7fcca80d1a60 0x7fcca80cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 --2- 172.21.15.59:0/724247350 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca80b2fa0 0x7fcca80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 --2- 172.21.15.59:0/724247350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2270 0x7fcca815bc20 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 -- 172.21.15.59:0/724247350 >> 172.21.15.59:0/724247350 conn(0x7fcca8073e10 msgr2=0x7fcca8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 -- 172.21.15.59:0/724247350 shutdown_connections 2024-06-29T14:37:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.845+0000 7fccadfeb640 1 -- 172.21.15.59:0/724247350 wait complete. 2024-06-29T14:37:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.846+0000 7fccadfeb640 1 Processor -- start 2024-06-29T14:37:29.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.847+0000 7fccadfeb640 1 -- start start 2024-06-29T14:37:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fccadfeb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca80b2270 0x7fcca81f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fccadfeb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2fa0 0x7fcca8070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fccadfeb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80d1a60 0x7fcca80707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fccadfeb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcca80cfe40 con 0x7fcca80b2fa0 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fccadfeb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcca80cfcc0 con 0x7fcca80d1a60 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fccadfeb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcca80cffc0 con 0x7fcca80b2270 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fcca6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2fa0 0x7fcca8070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fcca6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2fa0 0x7fcca8070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58960/0 (socket says 172.21.15.59:58960) 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fcca77fe640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca80b2270 0x7fcca81f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fcca6ffd640 1 -- 172.21.15.59:0/1577194369 learned_addr learned my addr 172.21.15.59:0/1577194369 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.848+0000 7fcca7fff640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80d1a60 0x7fcca80707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcca77fe640 1 -- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80d1a60 msgr2=0x7fcca80707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcca77fe640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80d1a60 0x7fcca80707b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcca77fe640 1 -- 172.21.15.59:0/1577194369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2fa0 msgr2=0x7fcca8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcca77fe640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2fa0 0x7fcca8070270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcca77fe640 1 -- 172.21.15.59:0/1577194369 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca81f44f0 con 0x7fcca80b2270 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcca7fff640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80d1a60 0x7fcca80707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcca6ffd640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2fa0 0x7fcca8070270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcca77fe640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca80b2270 0x7fcca81f3fb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcca80d0800 tx=0x7fcc980038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcc8ffff640 1 -- 172.21.15.59:0/1577194369 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9803e030 con 0x7fcca80b2270 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcc8ffff640 1 -- 172.21.15.59:0/1577194369 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc98003ca0 con 0x7fcca80b2270 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca80aeb20 con 0x7fcca80b2270 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fcc8ffff640 1 -- 172.21.15.59:0/1577194369 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc98035040 con 0x7fcca80b2270 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.849+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca80af0e0 con 0x7fcca80b2270 2024-06-29T14:37:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.850+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca80d0450 con 0x7fcca80b2270 2024-06-29T14:37:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.851+0000 7fcc8ffff640 1 -- 172.21.15.59:0/1577194369 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 31) ==== 98320+0+0 (secure 0 0 0) 0x7fcc98041030 con 0x7fcca80b2270 2024-06-29T14:37:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.852+0000 7fcc8ffff640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fcc94076440 0x7fcc94078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.852+0000 7fcca6ffd640 1 -- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fcc94076440 msgr2=0x7fcc94078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/4269929333 2024-06-29T14:37:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.852+0000 7fcc8ffff640 1 -- 172.21.15.59:0/1577194369 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7fcc980b84e0 con 0x7fcca80b2270 2024-06-29T14:37:29.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.852+0000 7fcca6ffd640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fcc94076440 0x7fcc94078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:29.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:29.861+0000 7fcc8ffff640 1 -- 172.21.15.59:0/1577194369 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc98081330 con 0x7fcca80b2270 2024-06-29T14:37:30.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.053+0000 7fcca6ffd640 1 -- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fcc94076440 msgr2=0x7fcc94078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/4269929333 2024-06-29T14:37:30.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.053+0000 7fcca6ffd640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fcc94076440 0x7fcc94078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:30.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.053+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fcca8075b40 con 0x7fcca80b2270 2024-06-29T14:37:30.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.060+0000 7fcc8ffff640 1 -- 172.21.15.59:0/1577194369 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v38)=0 v38) ==== 133+0+0 (secure 0 0 0) 0x7fcc98085630 con 0x7fcca80b2270 2024-06-29T14:37:30.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.062+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fcc94076440 msgr2=0x7fcc94078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:30.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.062+0000 7fccadfeb640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fcc94076440 0x7fcc94078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.062+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca80b2270 msgr2=0x7fcca81f3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.062+0000 7fccadfeb640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca80b2270 0x7fcca81f3fb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcca80d0800 tx=0x7fcc980038a0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.063+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 shutdown_connections 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.063+0000 7fccadfeb640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7fcc94076440 0x7fcc94078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.063+0000 7fccadfeb640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80d1a60 0x7fcca80707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.063+0000 7fccadfeb640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcca80b2fa0 0x7fcca8070270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.063+0000 7fccadfeb640 1 --2- 172.21.15.59:0/1577194369 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcca80b2270 0x7fcca81f3fb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.063+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 >> 172.21.15.59:0/1577194369 conn(0x7fcca8073e10 msgr2=0x7fcca80d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.063+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 shutdown_connections 2024-06-29T14:37:30.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.063+0000 7fccadfeb640 1 -- 172.21.15.59:0/1577194369 wait complete. 2024-06-29T14:37:30.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:30.072 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:37:30.094 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-06-29T14:37:30.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.260+0000 7f6f8752a640 1 -- 172.21.15.59:0/3443164933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8015eca0 msgr2=0x7f6f80168c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:30.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.260+0000 7f6f8752a640 1 --2- 172.21.15.59:0/3443164933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8015eca0 0x7f6f80168c20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009be0 tx=0x7f6f7802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.260+0000 7f6f8752a640 1 -- 172.21.15.59:0/3443164933 shutdown_connections 2024-06-29T14:37:30.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.260+0000 7f6f8752a640 1 --2- 172.21.15.59:0/3443164933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8015eca0 0x7f6f80168c20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.260+0000 7f6f8752a640 1 -- 172.21.15.59:0/3443164933 >> 172.21.15.59:0/3443164933 conn(0x7f6f80073e10 msgr2=0x7f6f80074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:30.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.260+0000 7f6f8752a640 1 -- 172.21.15.59:0/3443164933 shutdown_connections 2024-06-29T14:37:30.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.260+0000 7f6f8752a640 1 -- 172.21.15.59:0/3443164933 wait complete. 2024-06-29T14:37:30.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.262+0000 7f6f8752a640 1 Processor -- start 2024-06-29T14:37:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.262+0000 7f6f8752a640 1 -- start start 2024-06-29T14:37:30.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f8752a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f800d0dc0 0x7f6f800b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:30.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f8752a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f800d1ce0 0x7f6f80150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:30.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f8752a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8015eca0 0x7f6f80151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:30.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f8752a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801694f0 con 0x7f6f800d0dc0 2024-06-29T14:37:30.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f8752a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f80169370 con 0x7f6f800d1ce0 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f8752a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6f80169670 con 0x7f6f8015eca0 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f852c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f800d0dc0 0x7f6f800b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f852c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f800d0dc0 0x7f6f800b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58976/0 (socket says 172.21.15.59:58976) 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f852c6640 1 -- 172.21.15.59:0/2798791885 learned_addr learned my addr 172.21.15.59:0/2798791885 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f84ac5640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f800d1ce0 0x7f6f80150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.263+0000 7f6f85ac7640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8015eca0 0x7f6f80151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f852c6640 1 -- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8015eca0 msgr2=0x7f6f80151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f852c6640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6f8015eca0 0x7f6f80151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f852c6640 1 -- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f800d1ce0 msgr2=0x7f6f80150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f852c6640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f800d1ce0 0x7f6f80150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f852c6640 1 -- 172.21.15.59:0/2798791885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f801571b0 con 0x7f6f800d0dc0 2024-06-29T14:37:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f852c6640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f800d0dc0 0x7f6f800b5480 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c430 tx=0x7f6f7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:30.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c017070 con 0x7f6f800d0dc0 2024-06-29T14:37:30.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f7c0053e0 con 0x7f6f800d0dc0 2024-06-29T14:37:30.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f8752a640 1 -- 172.21.15.59:0/2798791885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8015f4e0 con 0x7f6f800d0dc0 2024-06-29T14:37:30.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c003850 con 0x7f6f800d0dc0 2024-06-29T14:37:30.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.264+0000 7f6f8752a640 1 -- 172.21.15.59:0/2798791885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8015faa0 con 0x7f6f800d0dc0 2024-06-29T14:37:30.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.265+0000 7f6f8752a640 1 -- 172.21.15.59:0/2798791885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f800d15d0 con 0x7f6f800d0dc0 2024-06-29T14:37:30.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.266+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 31) ==== 98320+0+0 (secure 0 0 0) 0x7f6f7c01a070 con 0x7f6f800d0dc0 2024-06-29T14:37:30.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.267+0000 7f6f75ffb640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 0x7f6f64078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:30.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.267+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6101+0+0 (secure 0 0 0) 0x7f6f7c096fe0 con 0x7f6f800d0dc0 2024-06-29T14:37:30.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.267+0000 7f6f84ac5640 1 -- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 msgr2=0x7f6f64078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/4269929333 2024-06-29T14:37:30.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.267+0000 7f6f84ac5640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 0x7f6f64078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:30.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.271+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6f7c05fee0 con 0x7f6f800d0dc0 2024-06-29T14:37:30.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.468+0000 7f6f84ac5640 1 -- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 msgr2=0x7f6f64078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/4269929333 2024-06-29T14:37:30.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.468+0000 7f6f84ac5640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 0x7f6f64078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.485+0000 7f6f8752a640 1 -- 172.21.15.59:0/2798791885 --> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f6f800b2d40 con 0x7f6f64076440 2024-06-29T14:37:30.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.869+0000 7f6f84ac5640 1 -- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 msgr2=0x7f6f64078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/4269929333 2024-06-29T14:37:30.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:30.869+0000 7f6f84ac5640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 0x7f6f64078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:37:31.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:31.670+0000 7f6f84ac5640 1 -- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 msgr2=0x7f6f64078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/4269929333 2024-06-29T14:37:31.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:31.670+0000 7f6f84ac5640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 0x7f6f64078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:37:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:33.008+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 32) ==== 97929+0+0 (secure 0 0 0) 0x7f6f7c05c170 con 0x7f6f800d0dc0 2024-06-29T14:37:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:33.008+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 msgr2=0x7f6f64078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:33.008+0000 7f6f75ffb640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/4269929333,v1:172.21.15.153:6833/4269929333] conn(0x7f6f64076440 0x7f6f64078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:33.165 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbad0c4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.165 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbad0c4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.165 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbad0c4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.165 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbad0c4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.165 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbad0c4c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.165 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbacdc46640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.166 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbacdc46640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.166 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbacdc46640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.166 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbacdc46640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.166 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:33.166+0000 7fbacdc46640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.013+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 33) ==== 98056+0+0 (secure 0 0 0) 0x7f6f7c094f20 con 0x7f6f800d0dc0 2024-06-29T14:37:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.013+0000 7f6f75ffb640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f6f6403b070 0x7f6f640809a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.013+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 --> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f6f7c0991b0 con 0x7f6f6403b070 2024-06-29T14:37:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.017+0000 7f6f84ac5640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f6f6403b070 0x7f6f640809a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.019+0000 7f6f84ac5640 1 --2- 172.21.15.59:0/2798791885 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f6f6403b070 0x7f6f640809a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6f8015e150 tx=0x7f6f700016d0 comp rx=0 tx=0).ready entity=mgr.6110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:34.025 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:34.026+0000 7fbad7c5a640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-06-29T14:37:34.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.028+0000 7f6f75ffb640 1 -- 172.21.15.59:0/2798791885 <== mgr.6110 v2:172.21.15.153:6832/3110229799 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f6f7c0991b0 con 0x7f6f6403b070 2024-06-29T14:37:34.025 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-06-29T14:37:34.031 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:37:34.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:34.032 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T14:37:34.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:37:34.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.221+0000 7f52f7cdf640 1 -- 172.21.15.59:0/2907836116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52f00cccc0 msgr2=0x7f52f00cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:34.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.221+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/2907836116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52f00cccc0 0x7f52f00cd0d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009830 tx=0x7f52e002a500 comp rx=0 tx=0).stop 2024-06-29T14:37:34.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.221+0000 7f52f7cdf640 1 -- 172.21.15.59:0/2907836116 shutdown_connections 2024-06-29T14:37:34.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.221+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/2907836116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52f00cccc0 0x7f52f00cd0d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:34.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.221+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/2907836116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52f01425e0 0x7f52f0142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:34.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.221+0000 7f52f7cdf640 1 -- 172.21.15.59:0/2907836116 >> 172.21.15.59:0/2907836116 conn(0x7f52f00b24c0 msgr2=0x7f52f00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:34.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.221+0000 7f52f7cdf640 1 -- 172.21.15.59:0/2907836116 shutdown_connections 2024-06-29T14:37:34.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.221+0000 7f52f7cdf640 1 -- 172.21.15.59:0/2907836116 wait complete. 2024-06-29T14:37:34.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.223+0000 7f52f7cdf640 1 Processor -- start 2024-06-29T14:37:34.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.223+0000 7f52f7cdf640 1 -- start start 2024-06-29T14:37:34.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f7cdf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f006c270 0x7f52f0156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f7cdf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52f00cccc0 0x7f52f0159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f7cdf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52f01425e0 0x7f52f0159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f7cdf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f52f00cea70 con 0x7f52f006c270 2024-06-29T14:37:34.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f7cdf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52f00ce8f0 con 0x7f52f01425e0 2024-06-29T14:37:34.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f7cdf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f52f00cebf0 con 0x7f52f00cccc0 2024-06-29T14:37:34.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f006c270 0x7f52f0156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f006c270 0x7f52f0156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50444/0 (socket says 172.21.15.59:50444) 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 -- 172.21.15.59:0/1091377888 learned_addr learned my addr 172.21.15.59:0/1091377888 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f527a640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52f00cccc0 0x7f52f0159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f627c640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52f01425e0 0x7f52f0159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 -- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52f00cccc0 msgr2=0x7f52f0159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52f00cccc0 0x7f52f0159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 -- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52f01425e0 msgr2=0x7f52f0159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52f01425e0 0x7f52f0159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 -- 172.21.15.59:0/1091377888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f0157410 con 0x7f52f006c270 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52f5a7b640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f006c270 0x7f52f0156800 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00ea50 tx=0x7f52ec00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52e67fc640 1 -- 172.21.15.59:0/1091377888 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec00cd60 con 0x7f52f006c270 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.225+0000 7f52e67fc640 1 -- 172.21.15.59:0/1091377888 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52ec004020 con 0x7f52f006c270 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.226+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f01576f0 con 0x7f52f006c270 2024-06-29T14:37:34.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.226+0000 7f52e67fc640 1 -- 172.21.15.59:0/1091377888 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ec004470 con 0x7f52f006c270 2024-06-29T14:37:34.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.226+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f0157cb0 con 0x7f52f006c270 2024-06-29T14:37:34.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.226+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f00cf110 con 0x7f52f006c270 2024-06-29T14:37:34.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.227+0000 7f52e67fc640 1 -- 172.21.15.59:0/1091377888 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 33) ==== 98056+0+0 (secure 0 0 0) 0x7f52ec0035f0 con 0x7f52f006c270 2024-06-29T14:37:34.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.228+0000 7f52e67fc640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f52d40761a0 0x7f52d40786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.228+0000 7f52e67fc640 1 -- 172.21.15.59:0/1091377888 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7f52ec097d40 con 0x7f52f006c270 2024-06-29T14:37:34.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.228+0000 7f52f527a640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f52d40761a0 0x7f52d40786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.229+0000 7f52f527a640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f52d40761a0 0x7f52d40786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f52e0034000 tx=0x7f52e0004730 comp rx=0 tx=0).ready entity=mgr.6110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:34.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.232+0000 7f52e67fc640 1 -- 172.21.15.59:0/1091377888 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f52ec09b030 con 0x7f52f006c270 2024-06-29T14:37:34.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:34.472+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f52f015f120 con 0x7f52f006c270 2024-06-29T14:37:35.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.231+0000 7f52e67fc640 1 -- 172.21.15.59:0/1091377888 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v34) ==== 93+0+0 (secure 0 0 0) 0x7f52ec0610b0 con 0x7f52f006c270 2024-06-29T14:37:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f52d40761a0 msgr2=0x7f52d40786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f52d40761a0 0x7f52d40786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f52e0034000 tx=0x7f52e0004730 comp rx=0 tx=0).stop 2024-06-29T14:37:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f006c270 msgr2=0x7f52f0156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f006c270 0x7f52f0156800 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00ea50 tx=0x7f52ec00ef10 comp rx=0 tx=0).stop 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 shutdown_connections 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f52d40761a0 0x7f52d40786b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52f01425e0 0x7f52f0159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52f00cccc0 0x7f52f0159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 --2- 172.21.15.59:0/1091377888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f006c270 0x7f52f0156800 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 >> 172.21.15.59:0/1091377888 conn(0x7f52f00b24c0 msgr2=0x7f52f00af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.235+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 shutdown_connections 2024-06-29T14:37:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.236+0000 7f52f7cdf640 1 -- 172.21.15.59:0/1091377888 wait complete. 2024-06-29T14:37:35.251 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:37:35.251 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:35.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:35.331 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:37:35.331 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:35.331 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:35.331 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:35.350 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:35.350 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:35.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:35.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.567+0000 7f672aa10640 1 -- 172.21.15.59:0/3841621719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f672415e290 msgr2=0x7f67240ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.567+0000 7f672aa10640 1 --2- 172.21.15.59:0/3841621719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f672415e290 0x7f67240ce110 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f6714005fd0 tx=0x7f671402a620 comp rx=0 tx=0).stop 2024-06-29T14:37:35.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.567+0000 7f672aa10640 1 -- 172.21.15.59:0/3841621719 shutdown_connections 2024-06-29T14:37:35.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.567+0000 7f672aa10640 1 --2- 172.21.15.59:0/3841621719 >> v2:172.21.15.153:3300/0 conn(0x7f67240b40a0 0x7f67240b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.567+0000 7f672aa10640 1 --2- 172.21.15.59:0/3841621719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f672415e290 0x7f67240ce110 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.567+0000 7f672aa10640 1 -- 172.21.15.59:0/3841621719 >> 172.21.15.59:0/3841621719 conn(0x7f67240d3ee0 msgr2=0x7f67240d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.567+0000 7f672aa10640 1 -- 172.21.15.59:0/3841621719 shutdown_connections 2024-06-29T14:37:35.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.567+0000 7f672aa10640 1 -- 172.21.15.59:0/3841621719 wait complete. 2024-06-29T14:37:35.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.569+0000 7f672aa10640 1 Processor -- start 2024-06-29T14:37:35.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.569+0000 7f672aa10640 1 -- start start 2024-06-29T14:37:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f672aa10640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67240ce650 0x7f67240cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f672aa10640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672415e290 0x7f6724160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f672aa10640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6724160c40 0x7f6724155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f672aa10640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f67240b7cc0 con 0x7f6724160c40 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f672aa10640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67240b7b40 con 0x7f67240ce650 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f672aa10640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f67240b7e40 con 0x7f672415e290 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6724160c40 0x7f6724155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6723fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67240ce650 0x7f67240cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6724160c40 0x7f6724155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50482/0 (socket says 172.21.15.59:50482) 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 -- 172.21.15.59:0/912194917 learned_addr learned my addr 172.21.15.59:0/912194917 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6723fff640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67240ce650 0x7f67240cd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:38142/0 (socket says 172.21.15.59:38142) 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f67237fe640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672415e290 0x7f6724160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 -- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672415e290 msgr2=0x7f6724160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672415e290 0x7f6724160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 -- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67240ce650 msgr2=0x7f67240cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67240ce650 0x7f67240cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 -- 172.21.15.59:0/912194917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6724161440 con 0x7f6724160c40 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6723fff640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67240ce650 0x7f67240cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.571+0000 7f6728fad640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6724160c40 0x7f6724155e70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f671c00ea40 tx=0x7f671c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.572+0000 7f6720ff9640 1 -- 172.21.15.59:0/912194917 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c00ce40 con 0x7f6724160c40 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.572+0000 7f6720ff9640 1 -- 172.21.15.59:0/912194917 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f671c004c10 con 0x7f6724160c40 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.572+0000 7f6720ff9640 1 -- 172.21.15.59:0/912194917 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c005020 con 0x7f6724160c40 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.572+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6724161720 con 0x7f6724160c40 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.572+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6724153940 con 0x7f6724160c40 2024-06-29T14:37:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.573+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67240b8380 con 0x7f6724160c40 2024-06-29T14:37:35.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.574+0000 7f6720ff9640 1 -- 172.21.15.59:0/912194917 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f671c0051c0 con 0x7f6724160c40 2024-06-29T14:37:35.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.574+0000 7f6720ff9640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f67100763f0 0x7f6710078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.574+0000 7f6723fff640 1 -- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f67100763f0 msgr2=0x7f6710078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:35.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.574+0000 7f6720ff9640 1 -- 172.21.15.59:0/912194917 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7f671c098e40 con 0x7f6724160c40 2024-06-29T14:37:35.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.574+0000 7f6723fff640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f67100763f0 0x7f6710078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:35.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.582+0000 7f6720ff9640 1 -- 172.21.15.59:0/912194917 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f671c026020 con 0x7f6724160c40 2024-06-29T14:37:35.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.774+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f672415b140 con 0x7f6724160c40 2024-06-29T14:37:35.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.776+0000 7f6723fff640 1 -- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f67100763f0 msgr2=0x7f6710078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:35.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.776+0000 7f6723fff640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f67100763f0 0x7f6710078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:35.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.776+0000 7f6720ff9640 1 -- 172.21.15.59:0/912194917 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+17402 (secure 0 0 0) 0x7f671c06a370 con 0x7f6724160c40 2024-06-29T14:37:35.774 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:35.774 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":89,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:32.999905+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:19.028073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.631284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1132863181":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/901558684":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6832/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6833/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2816930410":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2052586834":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/62694730":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2198109495":"2024-06-30T14:37:32.999861+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f67100763f0 msgr2=0x7f6710078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f67100763f0 0x7f6710078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6724160c40 msgr2=0x7f6724155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6724160c40 0x7f6724155e70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f671c00ea40 tx=0x7f671c00ef00 comp rx=0 tx=0).stop 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 shutdown_connections 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f67100763f0 0x7f6710078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6724160c40 0x7f6724155e70 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672415e290 0x7f6724160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 --2- 172.21.15.59:0/912194917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67240ce650 0x7f67240cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 >> 172.21.15.59:0/912194917 conn(0x7f67240d3ee0 msgr2=0x7f672415bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 shutdown_connections 2024-06-29T14:37:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.778+0000 7f672aa10640 1 -- 172.21.15.59:0/912194917 wait complete. 2024-06-29T14:37:35.787 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:37:19.028073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:37:19.631284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:35.788 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:35.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.913+0000 7ff6201ac640 1 -- 172.21.15.59:0/1711110848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2270 msgr2=0x7ff61815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.913+0000 7ff6201ac640 1 --2- 172.21.15.59:0/1711110848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2270 0x7ff61815bc20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff614009830 tx=0x7ff61402a510 comp rx=0 tx=0).stop 2024-06-29T14:37:35.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.913+0000 7ff6201ac640 1 -- 172.21.15.59:0/1711110848 shutdown_connections 2024-06-29T14:37:35.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.913+0000 7ff6201ac640 1 --2- 172.21.15.59:0/1711110848 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6180b2fa0 0x7ff6180d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.913+0000 7ff6201ac640 1 --2- 172.21.15.59:0/1711110848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2270 0x7ff61815bc20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.913+0000 7ff6201ac640 1 -- 172.21.15.59:0/1711110848 >> 172.21.15.59:0/1711110848 conn(0x7ff618073e10 msgr2=0x7ff618074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.914+0000 7ff6201ac640 1 -- 172.21.15.59:0/1711110848 shutdown_connections 2024-06-29T14:37:35.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.914+0000 7ff6201ac640 1 -- 172.21.15.59:0/1711110848 wait complete. 2024-06-29T14:37:35.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.915+0000 7ff6201ac640 1 Processor -- start 2024-06-29T14:37:35.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.915+0000 7ff6201ac640 1 -- start start 2024-06-29T14:37:35.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff6201ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6180b2270 0x7ff61814d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff6201ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2fa0 0x7ff618158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff6201ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6180d1a60 0x7ff6181586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff6201ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff6180b6cc0 con 0x7ff6180d1a60 2024-06-29T14:37:35.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff6201ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6180b6b40 con 0x7ff6180b2fa0 2024-06-29T14:37:35.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff6201ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff6180cfe90 con 0x7ff6180b2270 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61e749640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6180d1a60 0x7ff6181586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61e749640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6180d1a60 0x7ff6181586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50486/0 (socket says 172.21.15.59:50486) 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61df48640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6180b2270 0x7ff61814d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61e749640 1 -- 172.21.15.59:0/698403076 learned_addr learned my addr 172.21.15.59:0/698403076 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61d747640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2fa0 0x7ff618158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61e749640 1 -- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6180b2270 msgr2=0x7ff61814d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61e749640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6180b2270 0x7ff61814d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61e749640 1 -- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2fa0 msgr2=0x7ff618158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61e749640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2fa0 0x7ff618158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61e749640 1 -- 172.21.15.59:0/698403076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6180ae000 con 0x7ff6180d1a60 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.917+0000 7ff61d747640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2fa0 0x7ff618158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.918+0000 7ff61e749640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6180d1a60 0x7ff6181586b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ff61000ca80 tx=0x7ff61000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.918+0000 7ff60a7fc640 1 -- 172.21.15.59:0/698403076 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610016020 con 0x7ff6180d1a60 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.918+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6180ae2e0 con 0x7ff6180d1a60 2024-06-29T14:37:35.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.918+0000 7ff60a7fc640 1 -- 172.21.15.59:0/698403076 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff61000ad40 con 0x7ff6180d1a60 2024-06-29T14:37:35.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.918+0000 7ff60a7fc640 1 -- 172.21.15.59:0/698403076 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610012680 con 0x7ff6180d1a60 2024-06-29T14:37:35.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.918+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6180ae8a0 con 0x7ff6180d1a60 2024-06-29T14:37:35.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.919+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6180d03e0 con 0x7ff6180d1a60 2024-06-29T14:37:35.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.921+0000 7ff60a7fc640 1 -- 172.21.15.59:0/698403076 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7ff610019020 con 0x7ff6180d1a60 2024-06-29T14:37:35.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.921+0000 7ff60a7fc640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7ff5fc0763f0 0x7ff5fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.921+0000 7ff61df48640 1 -- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7ff5fc0763f0 msgr2=0x7ff5fc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:35.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.921+0000 7ff60a7fc640 1 -- 172.21.15.59:0/698403076 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7ff610098150 con 0x7ff6180d1a60 2024-06-29T14:37:35.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.921+0000 7ff61df48640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7ff5fc0763f0 0x7ff5fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:35.926+0000 7ff60a7fc640 1 -- 172.21.15.59:0/698403076 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff610020020 con 0x7ff6180d1a60 2024-06-29T14:37:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.119+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff618070280 con 0x7ff6180d1a60 2024-06-29T14:37:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.120+0000 7ff60a7fc640 1 -- 172.21.15.59:0/698403076 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7ff610060ff0 con 0x7ff6180d1a60 2024-06-29T14:37:36.117 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:37:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7ff5fc0763f0 msgr2=0x7ff5fc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7ff5fc0763f0 0x7ff5fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6180d1a60 msgr2=0x7ff6181586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6180d1a60 0x7ff6181586b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ff61000ca80 tx=0x7ff61000cf40 comp rx=0 tx=0).stop 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 shutdown_connections 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7ff5fc0763f0 0x7ff5fc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6180d1a60 0x7ff6181586b0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6180b2fa0 0x7ff618158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 --2- 172.21.15.59:0/698403076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6180b2270 0x7ff61814d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 >> 172.21.15.59:0/698403076 conn(0x7ff618073e10 msgr2=0x7ff6180cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 shutdown_connections 2024-06-29T14:37:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.122+0000 7ff6201ac640 1 -- 172.21.15.59:0/698403076 wait complete. 2024-06-29T14:37:36.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:37:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.258+0000 7fc83c826640 1 -- 172.21.15.59:0/1204210023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380b2270 msgr2=0x7fc83815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.258+0000 7fc83c826640 1 --2- 172.21.15.59:0/1204210023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380b2270 0x7fc83815bc20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc828005fd0 tx=0x7fc82802a670 comp rx=0 tx=0).stop 2024-06-29T14:37:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.258+0000 7fc83c826640 1 -- 172.21.15.59:0/1204210023 shutdown_connections 2024-06-29T14:37:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.258+0000 7fc83c826640 1 --2- 172.21.15.59:0/1204210023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380b2270 0x7fc83815bc20 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.258+0000 7fc83c826640 1 -- 172.21.15.59:0/1204210023 >> 172.21.15.59:0/1204210023 conn(0x7fc838073e10 msgr2=0x7fc838074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.258+0000 7fc83c826640 1 -- 172.21.15.59:0/1204210023 shutdown_connections 2024-06-29T14:37:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.258+0000 7fc83c826640 1 -- 172.21.15.59:0/1204210023 wait complete. 2024-06-29T14:37:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.260+0000 7fc83c826640 1 Processor -- start 2024-06-29T14:37:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.260+0000 7fc83c826640 1 -- start start 2024-06-29T14:37:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc83c826640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8380b2270 0x7fc838146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc83c826640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b2fa0 0x7fc8380ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc83c826640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380d1c30 0x7fc8380addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc83c826640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc8380b65d0 con 0x7fc8380d1c30 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc83c826640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc8380b6450 con 0x7fc8380b2270 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc83c826640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc8380b6750 con 0x7fc8380b2fa0 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc836d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380d1c30 0x7fc8380addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc836d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380d1c30 0x7fc8380addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50502/0 (socket says 172.21.15.59:50502) 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc836d9d640 1 -- 172.21.15.59:0/231380272 learned_addr learned my addr 172.21.15.59:0/231380272 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc83659c640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8380b2270 0x7fc838146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.261+0000 7fc835d9b640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b2fa0 0x7fc8380ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc836d9d640 1 -- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b2fa0 msgr2=0x7fc8380ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc836d9d640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b2fa0 0x7fc8380ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc836d9d640 1 -- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8380b2270 msgr2=0x7fc838146eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc836d9d640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8380b2270 0x7fc838146eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc836d9d640 1 -- 172.21.15.59:0/231380272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83806f4d0 con 0x7fc8380d1c30 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc836d9d640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380d1c30 0x7fc8380addf0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c730 tx=0x7fc83000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc826ffd640 1 -- 172.21.15.59:0/231380272 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830003b70 con 0x7fc8380d1c30 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc826ffd640 1 -- 172.21.15.59:0/231380272 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc830003d10 con 0x7fc8380d1c30 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83806e710 con 0x7fc8380d1c30 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc826ffd640 1 -- 172.21.15.59:0/231380272 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83000aab0 con 0x7fc8380d1c30 2024-06-29T14:37:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.262+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83806ecd0 con 0x7fc8380d1c30 2024-06-29T14:37:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.263+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8380b6c70 con 0x7fc8380d1c30 2024-06-29T14:37:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.264+0000 7fc826ffd640 1 -- 172.21.15.59:0/231380272 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7fc83000ac50 con 0x7fc8380d1c30 2024-06-29T14:37:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.265+0000 7fc826ffd640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fc8180763f0 0x7fc818078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.265+0000 7fc83659c640 1 -- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fc8180763f0 msgr2=0x7fc818078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.265+0000 7fc826ffd640 1 -- 172.21.15.59:0/231380272 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7fc830097990 con 0x7fc8380d1c30 2024-06-29T14:37:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.265+0000 7fc83659c640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fc8180763f0 0x7fc818078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:36.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.269+0000 7fc826ffd640 1 -- 172.21.15.59:0/231380272 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc830060820 con 0x7fc8380d1c30 2024-06-29T14:37:36.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.461+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc83815bda0 con 0x7fc8380d1c30 2024-06-29T14:37:36.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.461+0000 7fc826ffd640 1 -- 172.21.15.59:0/231380272 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fc830064b20 con 0x7fc8380d1c30 2024-06-29T14:37:36.459 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:37:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fc8180763f0 msgr2=0x7fc818078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fc8180763f0 0x7fc818078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380d1c30 msgr2=0x7fc8380addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380d1c30 0x7fc8380addf0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c730 tx=0x7fc83000cbf0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 shutdown_connections 2024-06-29T14:37:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fc8180763f0 0x7fc818078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380d1c30 0x7fc8380addf0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b2fa0 0x7fc8380ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 --2- 172.21.15.59:0/231380272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8380b2270 0x7fc838146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 >> 172.21.15.59:0/231380272 conn(0x7fc838073e10 msgr2=0x7fc8380b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 shutdown_connections 2024-06-29T14:37:36.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.463+0000 7fc83c826640 1 -- 172.21.15.59:0/231380272 wait complete. 2024-06-29T14:37:36.472 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:37:36.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.597+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3865757453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2d4004800 msgr2=0x7fa2d4004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.597+0000 7fa2ef61f640 1 --2- 172.21.15.59:0/3865757453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2d4004800 0x7fa2d4004c70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80b2f50 tx=0x7fa2d4005830 comp rx=0 tx=0).stop 2024-06-29T14:37:36.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.597+0000 7fa2edbbc640 1 -- 172.21.15.59:0/3865757453 reap_dead start 2024-06-29T14:37:36.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.598+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3865757453 shutdown_connections 2024-06-29T14:37:36.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.598+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3865757453 >> 172.21.15.59:0/3865757453 conn(0x7fa2e8073e10 msgr2=0x7fa2e8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.598+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3865757453 shutdown_connections 2024-06-29T14:37:36.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.598+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3865757453 wait complete. 2024-06-29T14:37:36.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.599+0000 7fa2ef61f640 1 Processor -- start 2024-06-29T14:37:36.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.600+0000 7fa2ef61f640 1 -- start start 2024-06-29T14:37:36.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ef61f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2d4004800 0x7fa2e814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ef61f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2e80b0020 0x7fa2e815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ef61f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e80b0430 0x7fa2e815f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ef61f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e815c170 con 0x7fa2d4004800 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ef61f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e815bff0 con 0x7fa2e80b0430 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ef61f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e80b6240 con 0x7fa2e80b0020 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ed3bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2d4004800 0x7fa2e814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ed3bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2d4004800 0x7fa2e814f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50524/0 (socket says 172.21.15.59:50524) 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ed3bb640 1 -- 172.21.15.59:0/3296602270 learned_addr learned my addr 172.21.15.59:0/3296602270 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ecbba640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2e80b0020 0x7fa2e815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2edbbc640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e80b0430 0x7fa2e815f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ed3bb640 1 -- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2e80b0020 msgr2=0x7fa2e815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ed3bb640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2e80b0020 0x7fa2e815eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ed3bb640 1 -- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e80b0430 msgr2=0x7fa2e815f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ed3bb640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e80b0430 0x7fa2e815f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.601+0000 7fa2ed3bb640 1 -- 172.21.15.59:0/3296602270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e814f9b0 con 0x7fa2d4004800 2024-06-29T14:37:36.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.602+0000 7fa2ed3bb640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2d4004800 0x7fa2e814f470 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4011a10 tx=0x7fa2e4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:36.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.602+0000 7fa2ddffb640 1 -- 172.21.15.59:0/3296602270 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e401e070 con 0x7fa2d4004800 2024-06-29T14:37:36.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.602+0000 7fa2ddffb640 1 -- 172.21.15.59:0/3296602270 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2e400fd30 con 0x7fa2d4004800 2024-06-29T14:37:36.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.602+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e80ae750 con 0x7fa2d4004800 2024-06-29T14:37:36.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.602+0000 7fa2ddffb640 1 -- 172.21.15.59:0/3296602270 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4004290 con 0x7fa2d4004800 2024-06-29T14:37:36.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.602+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e80aed10 con 0x7fa2d4004800 2024-06-29T14:37:36.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.603+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e80d1520 con 0x7fa2d4004800 2024-06-29T14:37:36.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.604+0000 7fa2ddffb640 1 -- 172.21.15.59:0/3296602270 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7fa2e4021070 con 0x7fa2d4004800 2024-06-29T14:37:36.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.604+0000 7fa2ddffb640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fa2cc076420 0x7fa2cc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.604+0000 7fa2ddffb640 1 -- 172.21.15.59:0/3296602270 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7fa2e4098500 con 0x7fa2d4004800 2024-06-29T14:37:36.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.604+0000 7fa2ecbba640 1 -- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fa2cc076420 msgr2=0x7fa2cc078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:36.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.604+0000 7fa2ecbba640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fa2cc076420 0x7fa2cc078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:36.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.608+0000 7fa2ddffb640 1 -- 172.21.15.59:0/3296602270 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa2e4061280 con 0x7fa2d4004800 2024-06-29T14:37:36.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.805+0000 7fa2ecbba640 1 -- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fa2cc076420 msgr2=0x7fa2cc078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:36.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.805+0000 7fa2ecbba640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fa2cc076420 0x7fa2cc078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:36.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.824+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa2e815b390 con 0x7fa2d4004800 2024-06-29T14:37:36.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.824+0000 7fa2ddffb640 1 -- 172.21.15.59:0/3296602270 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fa2e4065580 con 0x7fa2d4004800 2024-06-29T14:37:36.821 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fa2cc076420 msgr2=0x7fa2cc078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fa2cc076420 0x7fa2cc078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2d4004800 msgr2=0x7fa2e814f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2d4004800 0x7fa2e814f470 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4011a10 tx=0x7fa2e4011ed0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 shutdown_connections 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7fa2cc076420 0x7fa2cc078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2e80b0430 0x7fa2e815f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa2e80b0020 0x7fa2e815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 --2- 172.21.15.59:0/3296602270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2d4004800 0x7fa2e814f470 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.826+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 >> 172.21.15.59:0/3296602270 conn(0x7fa2e8073e10 msgr2=0x7fa2e8068460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.827+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 shutdown_connections 2024-06-29T14:37:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.827+0000 7fa2ef61f640 1 -- 172.21.15.59:0/3296602270 wait complete. 2024-06-29T14:37:36.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:37:36.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.958+0000 7f4413f12640 1 -- 172.21.15.59:0/314150806 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 msgr2=0x7f440c143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.958+0000 7f4413f12640 1 --2- 172.21.15.59:0/314150806 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 0x7f440c143840 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4408005fd0 tx=0x7f440802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.958+0000 7f4413f12640 1 -- 172.21.15.59:0/314150806 shutdown_connections 2024-06-29T14:37:36.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.958+0000 7f4413f12640 1 --2- 172.21.15.59:0/314150806 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f440c0b4ee0 0x7f440c06e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.958+0000 7f4413f12640 1 --2- 172.21.15.59:0/314150806 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 0x7f440c143840 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.958+0000 7f4413f12640 1 -- 172.21.15.59:0/314150806 >> 172.21.15.59:0/314150806 conn(0x7f440c0b2ab0 msgr2=0x7f440c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.958+0000 7f4413f12640 1 -- 172.21.15.59:0/314150806 shutdown_connections 2024-06-29T14:37:36.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.958+0000 7f4413f12640 1 -- 172.21.15.59:0/314150806 wait complete. 2024-06-29T14:37:36.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.960+0000 7f4413f12640 1 Processor -- start 2024-06-29T14:37:36.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.960+0000 7f4413f12640 1 -- start start 2024-06-29T14:37:36.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4413f12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f440c06ea10 0x7f440c159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4413f12640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f440c0b4ee0 0x7f440c0b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4413f12640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 0x7f440c0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4413f12640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f440c15ed80 con 0x7f440c0d0130 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4413f12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f440c06f030 con 0x7f440c06ea10 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4413f12640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f440c15ef00 con 0x7f440c0b4ee0 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f44124af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 0x7f440c0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4411cae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f440c06ea10 0x7f440c159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4411cae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f440c06ea10 0x7f440c159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:38192/0 (socket says 172.21.15.59:38192) 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f44114ad640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f440c0b4ee0 0x7f440c0b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.961+0000 7f4411cae640 1 -- 172.21.15.59:0/349159779 learned_addr learned my addr 172.21.15.59:0/349159779 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f4411cae640 1 -- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f440c0b4ee0 msgr2=0x7f440c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f4411cae640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f440c0b4ee0 0x7f440c0b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f4411cae640 1 -- 172.21.15.59:0/349159779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 msgr2=0x7f440c0b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f4411cae640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 0x7f440c0b15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f4411cae640 1 -- 172.21.15.59:0/349159779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c1f4e00 con 0x7f440c06ea10 2024-06-29T14:37:36.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f44124af640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 0x7f440c0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:36.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f4411cae640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f440c06ea10 0x7f440c159f20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4408000c00 tx=0x7f4408005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:36.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f44027fc640 1 -- 172.21.15.59:0/349159779 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408035040 con 0x7f440c06ea10 2024-06-29T14:37:36.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f44027fc640 1 -- 172.21.15.59:0/349159779 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4408003970 con 0x7f440c06ea10 2024-06-29T14:37:36.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c1f50e0 con 0x7f440c06ea10 2024-06-29T14:37:36.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f44027fc640 1 -- 172.21.15.59:0/349159779 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408003dc0 con 0x7f440c06ea10 2024-06-29T14:37:36.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.962+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c1f56a0 con 0x7f440c06ea10 2024-06-29T14:37:36.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.963+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c15f3a0 con 0x7f440c06ea10 2024-06-29T14:37:36.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.964+0000 7f44027fc640 1 -- 172.21.15.59:0/349159779 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f44080047a0 con 0x7f440c06ea10 2024-06-29T14:37:36.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.964+0000 7f44027fc640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f43f00763f0 0x7f43f0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.965+0000 7f44114ad640 1 -- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f43f00763f0 msgr2=0x7f43f0078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:36.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.965+0000 7f44027fc640 1 -- 172.21.15.59:0/349159779 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7f44080ba2c0 con 0x7f440c06ea10 2024-06-29T14:37:36.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.965+0000 7f44114ad640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f43f00763f0 0x7f43f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:36.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:36.969+0000 7f44027fc640 1 -- 172.21.15.59:0/349159779 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4408083150 con 0x7f440c06ea10 2024-06-29T14:37:37.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.166+0000 7f44114ad640 1 -- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f43f00763f0 msgr2=0x7f43f0078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:37.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.166+0000 7f44114ad640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f43f00763f0 0x7f43f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:37.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.174+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f440c0afdd0 con 0x7f440c06ea10 2024-06-29T14:37:37.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.175+0000 7f44027fc640 1 -- 172.21.15.59:0/349159779 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v89) ==== 80+0+475 (secure 0 0 0) 0x7f4408087450 con 0x7f440c06ea10 2024-06-29T14:37:37.172 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1132863181 2024-06-30T14:37:32.999861+0000 2024-06-29T14:37:37.172 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/901558684 2024-06-30T14:37:32.999861+0000 2024-06-29T14:37:37.172 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/4269929333 2024-06-30T14:37:32.999861+0000 2024-06-29T14:37:37.172 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/4269929333 2024-06-30T14:37:32.999861+0000 2024-06-29T14:37:37.172 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2816930410 2024-06-30T14:37:32.999861+0000 2024-06-29T14:37:37.172 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2052586834 2024-06-30T14:37:32.999861+0000 2024-06-29T14:37:37.172 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/62694730 2024-06-30T14:37:32.999861+0000 2024-06-29T14:37:37.173 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2198109495 2024-06-30T14:37:32.999861+0000 2024-06-29T14:37:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T14:37:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f43f00763f0 msgr2=0x7f43f0078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f43f00763f0 0x7f43f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f440c06ea10 msgr2=0x7f440c159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f440c06ea10 0x7f440c159f20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4408000c00 tx=0x7f4408005770 comp rx=0 tx=0).stop 2024-06-29T14:37:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 shutdown_connections 2024-06-29T14:37:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f43f00763f0 0x7f43f0078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f440c0d0130 0x7f440c0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f440c0b4ee0 0x7f440c0b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 --2- 172.21.15.59:0/349159779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f440c06ea10 0x7f440c159f20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 >> 172.21.15.59:0/349159779 conn(0x7f440c0b2ab0 msgr2=0x7f440c072c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 shutdown_connections 2024-06-29T14:37:37.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.177+0000 7f4413f12640 1 -- 172.21.15.59:0/349159779 wait complete. 2024-06-29T14:37:37.186 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:37:37.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:37.186 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:37.216 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:37:37.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:37.216 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:37:37.272 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:37.272 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:37:37.272 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:37.287 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:37.288 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:37.288 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:37.288 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:37.330 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:37.330 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:37.330 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:37:37.331 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:37.346 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:37.347 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:37.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.491+0000 7f4449d7f640 1 -- 172.21.15.59:0/546470081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 msgr2=0x7f444415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.491+0000 7f4449d7f640 1 --2- 172.21.15.59:0/546470081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 0x7f444415e8d0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4438005fd0 tx=0x7f443802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.491+0000 7f4449d7f640 1 -- 172.21.15.59:0/546470081 shutdown_connections 2024-06-29T14:37:37.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.491+0000 7f4449d7f640 1 --2- 172.21.15.59:0/546470081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 0x7f444415e8d0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.491+0000 7f4449d7f640 1 --2- 172.21.15.59:0/546470081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44440d23c0 0x7f44440b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.491+0000 7f4449d7f640 1 -- 172.21.15.59:0/546470081 >> 172.21.15.59:0/546470081 conn(0x7f44440d3ee0 msgr2=0x7f44440d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.491+0000 7f4449d7f640 1 -- 172.21.15.59:0/546470081 shutdown_connections 2024-06-29T14:37:37.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.491+0000 7f4449d7f640 1 -- 172.21.15.59:0/546470081 wait complete. 2024-06-29T14:37:37.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.493+0000 7f4449d7f640 1 Processor -- start 2024-06-29T14:37:37.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.493+0000 7f4449d7f640 1 -- start start 2024-06-29T14:37:37.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f4449d7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 0x7f44441f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f4449d7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44440d23c0 0x7f44440c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f4449d7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f444415ee10 0x7f444414db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f4449d7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4444160240 con 0x7f44440b3ed0 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f4449d7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44441600c0 con 0x7f444415ee10 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f4449d7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f44441603c0 con 0x7f44440d23c0 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f44437fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 0x7f44441f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f44437fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 0x7f44441f5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50574/0 (socket says 172.21.15.59:50574) 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f44437fe640 1 -- 172.21.15.59:0/193958522 learned_addr learned my addr 172.21.15.59:0/193958522 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f4442ffd640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44440d23c0 0x7f44440c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f4443fff640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f444415ee10 0x7f444414db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f44437fe640 1 -- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44440d23c0 msgr2=0x7f44440c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f44437fe640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44440d23c0 0x7f44440c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f44437fe640 1 -- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f444415ee10 msgr2=0x7f444414db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f44437fe640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f444415ee10 0x7f444414db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.495+0000 7f44437fe640 1 -- 172.21.15.59:0/193958522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444415e410 con 0x7f44440b3ed0 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.496+0000 7f44437fe640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 0x7f44441f5bc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f4434009fd0 tx=0x7f443400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.496+0000 7f442bfff640 1 -- 172.21.15.59:0/193958522 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443400ccc0 con 0x7f44440b3ed0 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.496+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444415db70 con 0x7f44440b3ed0 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.496+0000 7f442bfff640 1 -- 172.21.15.59:0/193958522 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f443400ee70 con 0x7f44440b3ed0 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.496+0000 7f442bfff640 1 -- 172.21.15.59:0/193958522 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443400ce60 con 0x7f44440b3ed0 2024-06-29T14:37:37.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.496+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f444415e130 con 0x7f44440b3ed0 2024-06-29T14:37:37.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.497+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f444415e8d0 con 0x7f44440b3ed0 2024-06-29T14:37:37.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.498+0000 7f442bfff640 1 -- 172.21.15.59:0/193958522 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f4434004280 con 0x7f44440b3ed0 2024-06-29T14:37:37.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.498+0000 7f442bfff640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f4430076420 0x7f4430078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.498+0000 7f442bfff640 1 -- 172.21.15.59:0/193958522 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7f4434098580 con 0x7f44440b3ed0 2024-06-29T14:37:37.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.498+0000 7f4442ffd640 1 -- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f4430076420 msgr2=0x7f4430078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:37.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.498+0000 7f4442ffd640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f4430076420 0x7f4430078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:37.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.503+0000 7f442bfff640 1 -- 172.21.15.59:0/193958522 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f44340611e0 con 0x7f44440b3ed0 2024-06-29T14:37:37.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.699+0000 7f4442ffd640 1 -- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f4430076420 msgr2=0x7f4430078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:37.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.699+0000 7f4442ffd640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f4430076420 0x7f4430078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:37.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.734+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44440d2950 con 0x7f44440b3ed0 2024-06-29T14:37:37.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.734+0000 7f442bfff640 1 -- 172.21.15.59:0/193958522 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5230 (secure 0 0 0) 0x7f44340654e0 con 0x7f44440b3ed0 2024-06-29T14:37:37.732 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:37.732 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":38,"btime":"2024-06-29T14:37:24:318813+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6529,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3719851385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3719851385},{"type":"v1","addr":"172.21.15.153:6836","nonce":3719851385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1280649067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1280649067},{"type":"v1","addr":"172.21.15.59:6833","nonce":1280649067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2847930906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2847930906},{"type":"v1","addr":"172.21.15.59:6835","nonce":2847930906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.672386+0000","modified":"2024-06-29T14:37:23.706036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"d","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/3891057790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3891057790},{"type":"v1","addr":"172.21.15.153:6837","nonce":3891057790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":4}]} 2024-06-29T14:37:37.733 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 38 2024-06-29T14:37:37.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f4430076420 msgr2=0x7f4430078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:37.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f4430076420 0x7f4430078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 msgr2=0x7f44441f5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 0x7f44441f5bc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f4434009fd0 tx=0x7f443400ec20 comp rx=0 tx=0).stop 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 shutdown_connections 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f4430076420 0x7f4430078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f444415ee10 0x7f444414db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44440d23c0 0x7f44440c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 --2- 172.21.15.59:0/193958522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44440b3ed0 0x7f44441f5bc0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 >> 172.21.15.59:0/193958522 conn(0x7f44440d3ee0 msgr2=0x7f444406cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 shutdown_connections 2024-06-29T14:37:37.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.737+0000 7f4449d7f640 1 -- 172.21.15.59:0/193958522 wait complete. 2024-06-29T14:37:37.746 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.867+0000 7f9b97731640 1 -- 172.21.15.59:0/3053395418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b9015ee10 msgr2=0x7f9b9015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.867+0000 7f9b97731640 1 --2- 172.21.15.59:0/3053395418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b9015ee10 0x7f9b9015f9b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f9b9008c4f0 tx=0x7f9b8802aac0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.867+0000 7f9b97731640 1 -- 172.21.15.59:0/3053395418 shutdown_connections 2024-06-29T14:37:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.867+0000 7f9b97731640 1 --2- 172.21.15.59:0/3053395418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b9015ee10 0x7f9b9015f9b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.867+0000 7f9b97731640 1 --2- 172.21.15.59:0/3053395418 >> v2:172.21.15.153:3300/0 conn(0x7f9b900b40a0 0x7f9b9015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.867+0000 7f9b97731640 1 -- 172.21.15.59:0/3053395418 >> 172.21.15.59:0/3053395418 conn(0x7f9b900d3ee0 msgr2=0x7f9b900d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.868+0000 7f9b97731640 1 -- 172.21.15.59:0/3053395418 shutdown_connections 2024-06-29T14:37:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.868+0000 7f9b97731640 1 -- 172.21.15.59:0/3053395418 wait complete. 2024-06-29T14:37:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.869+0000 7f9b97731640 1 Processor -- start 2024-06-29T14:37:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.869+0000 7f9b97731640 1 -- start start 2024-06-29T14:37:37.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b97731640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b900d23c0 0x7f9b9014f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b97731640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b9015ee10 0x7f9b901e6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b97731640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b901e7350 0x7f9b90151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b97731640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90160280 con 0x7f9b900d23c0 2024-06-29T14:37:37.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b97731640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90160100 con 0x7f9b9015ee10 2024-06-29T14:37:37.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b97731640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9b90160400 con 0x7f9b901e7350 2024-06-29T14:37:37.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b900d23c0 0x7f9b9014f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b900d23c0 0x7f9b9014f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50600/0 (socket says 172.21.15.59:50600) 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 -- 172.21.15.59:0/530779153 learned_addr learned my addr 172.21.15.59:0/530779153 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b94ccc640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b9015ee10 0x7f9b901e6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b95cce640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b901e7350 0x7f9b90151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 -- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b901e7350 msgr2=0x7f9b90151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b901e7350 0x7f9b90151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 -- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b9015ee10 msgr2=0x7f9b901e6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b9015ee10 0x7f9b901e6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 -- 172.21.15.59:0/530779153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b90156350 con 0x7f9b900d23c0 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b954cd640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b900d23c0 0x7f9b9014f230 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00c370 tx=0x7f9b8c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b81ffb640 1 -- 172.21.15.59:0/530779153 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c017070 con 0x7f9b900d23c0 2024-06-29T14:37:37.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.871+0000 7f9b81ffb640 1 -- 172.21.15.59:0/530779153 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b8c0053c0 con 0x7f9b900d23c0 2024-06-29T14:37:37.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.872+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b901e7b40 con 0x7f9b900d23c0 2024-06-29T14:37:37.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.872+0000 7f9b81ffb640 1 -- 172.21.15.59:0/530779153 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c003850 con 0x7f9b900d23c0 2024-06-29T14:37:37.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.872+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b90154ad0 con 0x7f9b900d23c0 2024-06-29T14:37:37.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.873+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b9015ec30 con 0x7f9b900d23c0 2024-06-29T14:37:37.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.873+0000 7f9b81ffb640 1 -- 172.21.15.59:0/530779153 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f9b8c01a070 con 0x7f9b900d23c0 2024-06-29T14:37:37.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.874+0000 7f9b81ffb640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f9b740763f0 0x7f9b74078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.874+0000 7f9b94ccc640 1 -- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f9b740763f0 msgr2=0x7f9b74078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:37.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.874+0000 7f9b81ffb640 1 -- 172.21.15.59:0/530779153 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7f9b8c096f50 con 0x7f9b900d23c0 2024-06-29T14:37:37.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.874+0000 7f9b94ccc640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f9b740763f0 0x7f9b74078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:37.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:37.878+0000 7f9b81ffb640 1 -- 172.21.15.59:0/530779153 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9b9015ec30 con 0x7f9b900d23c0 2024-06-29T14:37:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.076+0000 7f9b94ccc640 1 -- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f9b740763f0 msgr2=0x7f9b74078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.076+0000 7f9b94ccc640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f9b740763f0 0x7f9b74078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:38.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.093+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b9015e8d0 con 0x7f9b900d23c0 2024-06-29T14:37:38.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.094+0000 7f9b81ffb640 1 -- 172.21.15.59:0/530779153 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+17402 (secure 0 0 0) 0x7f9b8c05fde0 con 0x7f9b900d23c0 2024-06-29T14:37:38.091 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:38.091 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":89,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:32.999905+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:19.028073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.631284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1132863181":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/901558684":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6832/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6833/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2816930410":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2052586834":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/62694730":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2198109495":"2024-06-30T14:37:32.999861+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f9b740763f0 msgr2=0x7f9b74078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f9b740763f0 0x7f9b74078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b900d23c0 msgr2=0x7f9b9014f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b900d23c0 0x7f9b9014f230 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00c370 tx=0x7f9b8c00c830 comp rx=0 tx=0).stop 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 shutdown_connections 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f9b740763f0 0x7f9b74078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b901e7350 0x7f9b90151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b9015ee10 0x7f9b901e6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 --2- 172.21.15.59:0/530779153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b900d23c0 0x7f9b9014f230 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 >> 172.21.15.59:0/530779153 conn(0x7f9b900d3ee0 msgr2=0x7f9b900b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 shutdown_connections 2024-06-29T14:37:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.096+0000 7f9b97731640 1 -- 172.21.15.59:0/530779153 wait complete. 2024-06-29T14:37:38.105 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:37:19.028073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:37:19.631284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:38.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:38.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.230+0000 7f99d65f5640 1 -- 172.21.15.59:0/2343467015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d00b2270 msgr2=0x7f99d015bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.230+0000 7f99d65f5640 1 --2- 172.21.15.59:0/2343467015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d00b2270 0x7f99d015bc20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009830 tx=0x7f99c002a510 comp rx=0 tx=0).stop 2024-06-29T14:37:38.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.230+0000 7f99d65f5640 1 -- 172.21.15.59:0/2343467015 shutdown_connections 2024-06-29T14:37:38.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.230+0000 7f99d65f5640 1 --2- 172.21.15.59:0/2343467015 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d00b2fa0 0x7f99d00d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.230+0000 7f99d65f5640 1 --2- 172.21.15.59:0/2343467015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d00b2270 0x7f99d015bc20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.230+0000 7f99d65f5640 1 -- 172.21.15.59:0/2343467015 >> 172.21.15.59:0/2343467015 conn(0x7f99d0073e10 msgr2=0x7f99d0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.230+0000 7f99d65f5640 1 -- 172.21.15.59:0/2343467015 shutdown_connections 2024-06-29T14:37:38.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.230+0000 7f99d65f5640 1 -- 172.21.15.59:0/2343467015 wait complete. 2024-06-29T14:37:38.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.231+0000 7f99d65f5640 1 Processor -- start 2024-06-29T14:37:38.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.232+0000 7f99d65f5640 1 -- start start 2024-06-29T14:37:38.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99d65f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99d00b2270 0x7f99d01f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99d65f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d00b2fa0 0x7f99d0154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99d65f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d00d1a60 0x7f99d0070630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99d65f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f99d00b6cc0 con 0x7f99d00b2270 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99d65f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99d00b6b40 con 0x7f99d00d1a60 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99d65f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f99d00cfe90 con 0x7f99d00b2fa0 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99cffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99d00b2270 0x7f99d01f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99cffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99d00b2270 0x7f99d01f5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50604/0 (socket says 172.21.15.59:50604) 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99cffff640 1 -- 172.21.15.59:0/3459272370 learned_addr learned my addr 172.21.15.59:0/3459272370 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99cf7fe640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d00b2fa0 0x7f99d0154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.233+0000 7f99d4b92640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d00d1a60 0x7f99d0070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99cf7fe640 1 -- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d00d1a60 msgr2=0x7f99d0070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99cf7fe640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d00d1a60 0x7f99d0070630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99cf7fe640 1 -- 172.21.15.59:0/3459272370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99d00b2270 msgr2=0x7f99d01f5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99cf7fe640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99d00b2270 0x7f99d01f5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99cf7fe640 1 -- 172.21.15.59:0/3459272370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d006dc20 con 0x7f99d00b2fa0 2024-06-29T14:37:38.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99cffff640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99d00b2270 0x7f99d01f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:38.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99cf7fe640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d00b2fa0 0x7f99d0154420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f99c400e9c0 tx=0x7f99c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:38.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99ccff9640 1 -- 172.21.15.59:0/3459272370 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c400ccc0 con 0x7f99d00b2fa0 2024-06-29T14:37:38.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99ccff9640 1 -- 172.21.15.59:0/3459272370 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99c400ce60 con 0x7f99d00b2fa0 2024-06-29T14:37:38.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99ccff9640 1 -- 172.21.15.59:0/3459272370 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4004260 con 0x7f99d00b2fa0 2024-06-29T14:37:38.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d015ee20 con 0x7f99d00b2fa0 2024-06-29T14:37:38.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.234+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d015f3c0 con 0x7f99d00b2fa0 2024-06-29T14:37:38.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.235+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d006f490 con 0x7f99d00b2fa0 2024-06-29T14:37:38.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.236+0000 7f99ccff9640 1 -- 172.21.15.59:0/3459272370 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f99c4005030 con 0x7f99d00b2fa0 2024-06-29T14:37:38.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.237+0000 7f99ccff9640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f99bc0763f0 0x7f99bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.237+0000 7f99cffff640 1 -- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f99bc0763f0 msgr2=0x7f99bc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:38.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.237+0000 7f99cffff640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f99bc0763f0 0x7f99bc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:38.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.239+0000 7f99ccff9640 1 -- 172.21.15.59:0/3459272370 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7f99c4004400 con 0x7f99d00b2fa0 2024-06-29T14:37:38.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.248+0000 7f99ccff9640 1 -- 172.21.15.59:0/3459272370 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99c40614c0 con 0x7f99d00b2fa0 2024-06-29T14:37:38.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.439+0000 7f99cffff640 1 -- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f99bc0763f0 msgr2=0x7f99bc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:38.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.439+0000 7f99cffff640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f99bc0763f0 0x7f99bc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:38.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.442+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f99d00b2680 con 0x7f99d00b2fa0 2024-06-29T14:37:38.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.443+0000 7f99ccff9640 1 -- 172.21.15.59:0/3459272370 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f99d00b2680 con 0x7f99d00b2fa0 2024-06-29T14:37:38.440 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:37:38.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f99bc0763f0 msgr2=0x7f99bc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:38.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f99bc0763f0 0x7f99bc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d00b2fa0 msgr2=0x7f99d0154420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d00b2fa0 0x7f99d0154420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f99c400e9c0 tx=0x7f99c400ee80 comp rx=0 tx=0).stop 2024-06-29T14:37:38.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 shutdown_connections 2024-06-29T14:37:38.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f99bc0763f0 0x7f99bc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99d00d1a60 0x7f99d0070630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99d00b2fa0 0x7f99d0154420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 --2- 172.21.15.59:0/3459272370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99d00b2270 0x7f99d01f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 >> 172.21.15.59:0/3459272370 conn(0x7f99d0073e10 msgr2=0x7f99d00d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 shutdown_connections 2024-06-29T14:37:38.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.445+0000 7f99d65f5640 1 -- 172.21.15.59:0/3459272370 wait complete. 2024-06-29T14:37:38.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:37:38.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.578+0000 7f574bcac640 1 -- 172.21.15.59:0/729034463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2fa0 msgr2=0x7f57440d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.578+0000 7f574bcac640 1 --2- 172.21.15.59:0/729034463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2fa0 0x7f57440d1520 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5734005fd0 tx=0x7f573402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.578+0000 7f574bcac640 1 -- 172.21.15.59:0/729034463 shutdown_connections 2024-06-29T14:37:38.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.578+0000 7f574bcac640 1 --2- 172.21.15.59:0/729034463 >> v2:172.21.15.153:3300/0 conn(0x7f57440d1a60 0x7f57440b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.578+0000 7f574bcac640 1 --2- 172.21.15.59:0/729034463 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2fa0 0x7f57440d1520 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.578+0000 7f574bcac640 1 -- 172.21.15.59:0/729034463 >> 172.21.15.59:0/729034463 conn(0x7f5744073e10 msgr2=0x7f5744074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.578+0000 7f574bcac640 1 -- 172.21.15.59:0/729034463 shutdown_connections 2024-06-29T14:37:38.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.578+0000 7f574bcac640 1 -- 172.21.15.59:0/729034463 wait complete. 2024-06-29T14:37:38.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.580+0000 7f574bcac640 1 Processor -- start 2024-06-29T14:37:38.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.580+0000 7f574bcac640 1 -- start start 2024-06-29T14:37:38.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.581+0000 7f574bcac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2270 0x7f574415e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.581+0000 7f574bcac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57440b2fa0 0x7f5744153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.581+0000 7f574bcac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57440d1a60 0x7f5744153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.581+0000 7f574bcac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f57440b6400 con 0x7f57440b2fa0 2024-06-29T14:37:38.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.581+0000 7f574bcac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57440b6280 con 0x7f57440d1a60 2024-06-29T14:37:38.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.581+0000 7f574bcac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f57440b6580 con 0x7f57440b2270 2024-06-29T14:37:38.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57440b2fa0 0x7f5744153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749a48640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2270 0x7f574415e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57440b2fa0 0x7f5744153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50616/0 (socket says 172.21.15.59:50616) 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749247640 1 -- 172.21.15.59:0/1182818348 learned_addr learned my addr 172.21.15.59:0/1182818348 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f574a249640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57440d1a60 0x7f5744153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749a48640 1 -- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57440d1a60 msgr2=0x7f5744153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749a48640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57440d1a60 0x7f5744153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749a48640 1 -- 172.21.15.59:0/1182818348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57440b2fa0 msgr2=0x7f5744153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749a48640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57440b2fa0 0x7f5744153630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749a48640 1 -- 172.21.15.59:0/1182818348 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57440d1690 con 0x7f57440b2270 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749247640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57440b2fa0 0x7f5744153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f5749a48640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2270 0x7f574415e7b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f574000c3c0 tx=0x7f574000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f573a7fc640 1 -- 172.21.15.59:0/1182818348 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57400052d0 con 0x7f57440b2270 2024-06-29T14:37:38.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f573a7fc640 1 -- 172.21.15.59:0/1182818348 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5740007020 con 0x7f57440b2270 2024-06-29T14:37:38.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57440a96f0 con 0x7f57440b2270 2024-06-29T14:37:38.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.582+0000 7f573a7fc640 1 -- 172.21.15.59:0/1182818348 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740007470 con 0x7f57440b2270 2024-06-29T14:37:38.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.583+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f57440a9cb0 con 0x7f57440b2270 2024-06-29T14:37:38.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.583+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57440b6aa0 con 0x7f57440b2270 2024-06-29T14:37:38.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.584+0000 7f573a7fc640 1 -- 172.21.15.59:0/1182818348 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f57400067d0 con 0x7f57440b2270 2024-06-29T14:37:38.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.585+0000 7f573a7fc640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f57280763f0 0x7f5728078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.585+0000 7f5749247640 1 -- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f57280763f0 msgr2=0x7f5728078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3110229799 2024-06-29T14:37:38.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.585+0000 7f573a7fc640 1 -- 172.21.15.59:0/1182818348 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(89..89 src has 1..89) ==== 6445+0+0 (secure 0 0 0) 0x7f5740097880 con 0x7f57440b2270 2024-06-29T14:37:38.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.585+0000 7f5749247640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f57280763f0 0x7f5728078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:38.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.593+0000 7f573a7fc640 1 -- 172.21.15.59:0/1182818348 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5740060710 con 0x7f57440b2270 2024-06-29T14:37:38.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.783+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5744070ad0 con 0x7f57440b2270 2024-06-29T14:37:38.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.784+0000 7f573a7fc640 1 -- 172.21.15.59:0/1182818348 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f5740064a10 con 0x7f57440b2270 2024-06-29T14:37:38.781 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:37:38.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f57280763f0 msgr2=0x7f5728078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:38.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f57280763f0 0x7f5728078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2270 msgr2=0x7f574415e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2270 0x7f574415e7b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f574000c3c0 tx=0x7f574000c880 comp rx=0 tx=0).stop 2024-06-29T14:37:38.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 shutdown_connections 2024-06-29T14:37:38.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:6832/3110229799,v1:172.21.15.153:6833/3110229799] conn(0x7f57280763f0 0x7f5728078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57440d1a60 0x7f5744153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57440b2fa0 0x7f5744153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 --2- 172.21.15.59:0/1182818348 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57440b2270 0x7f574415e7b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 >> 172.21.15.59:0/1182818348 conn(0x7f5744073e10 msgr2=0x7f574415bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 shutdown_connections 2024-06-29T14:37:38.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.786+0000 7f574bcac640 1 -- 172.21.15.59:0/1182818348 wait complete. 2024-06-29T14:37:38.794 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.923+0000 7f4c810a5640 1 -- 172.21.15.59:0/2959183080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c7c0d1c30 msgr2=0x7f4c7c0b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.923+0000 7f4c810a5640 1 --2- 172.21.15.59:0/2959183080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c7c0d1c30 0x7f4c7c0b5b30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4c74005fd0 tx=0x7f4c7402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.923+0000 7f4c810a5640 1 -- 172.21.15.59:0/2959183080 shutdown_connections 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.923+0000 7f4c810a5640 1 --2- 172.21.15.59:0/2959183080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c7c0d1c30 0x7f4c7c0b5b30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.923+0000 7f4c810a5640 1 --2- 172.21.15.59:0/2959183080 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7c0b2270 0x7f4c7c15bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.923+0000 7f4c810a5640 1 -- 172.21.15.59:0/2959183080 >> 172.21.15.59:0/2959183080 conn(0x7f4c7c073e10 msgr2=0x7f4c7c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.923+0000 7f4c810a5640 1 -- 172.21.15.59:0/2959183080 shutdown_connections 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.923+0000 7f4c810a5640 1 -- 172.21.15.59:0/2959183080 wait complete. 2024-06-29T14:37:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.925+0000 7f4c810a5640 1 Processor -- start 2024-06-29T14:37:38.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.925+0000 7f4c810a5640 1 -- start start 2024-06-29T14:37:38.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.926+0000 7f4c810a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7c0b2270 0x7f4c7c0a9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.926+0000 7f4c810a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c0b2fa0 0x7f4c7c153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.926+0000 7f4c810a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c7c0d1c30 0x7f4c7c0ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.926+0000 7f4c810a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c0b6400 con 0x7f4c7c0b2fa0 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.926+0000 7f4c810a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c0b6280 con 0x7f4c7c0d1c30 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.926+0000 7f4c810a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7c0b6580 con 0x7f4c7c0b2270 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c0b2fa0 0x7f4c7c153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7c0b2270 0x7f4c7c0a9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7c0b2270 0x7f4c7c0a9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:37850/0 (socket says 172.21.15.59:37850) 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7b59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c7c0d1c30 0x7f4c7c0ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c0b2fa0 0x7f4c7c153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50622/0 (socket says 172.21.15.59:50622) 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 -- 172.21.15.59:0/1153038922 learned_addr learned my addr 172.21.15.59:0/1153038922 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 -- 172.21.15.59:0/1153038922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c7c0d1c30 msgr2=0x7f4c7c0ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 --2- 172.21.15.59:0/1153038922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c7c0d1c30 0x7f4c7c0ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 -- 172.21.15.59:0/1153038922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c0b2fa0 msgr2=0x7f4c7c153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 --2- 172.21.15.59:0/1153038922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c0b2fa0 0x7f4c7c153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 -- 172.21.15.59:0/1153038922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c0aa3d0 con 0x7f4c7c0b2270 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c7ad9d640 1 --2- 172.21.15.59:0/1153038922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7c0b2270 0x7f4c7c0a9b80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00e9f0 tx=0x7f4c6c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c637fe640 1 -- 172.21.15.59:0/1153038922 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c00cd80 con 0x7f4c7c0b2270 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c637fe640 1 -- 172.21.15.59:0/1153038922 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c6c004020 con 0x7f4c7c0b2270 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7c0aa6b0 con 0x7f4c7c0b2270 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.927+0000 7f4c637fe640 1 -- 172.21.15.59:0/1153038922 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c004450 con 0x7f4c7c0b2270 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.928+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7c0aac70 con 0x7f4c7c0b2270 2024-06-29T14:37:38.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.928+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c7c0b6ac0 con 0x7f4c7c0b2270 2024-06-29T14:37:38.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.929+0000 7f4c637fe640 1 -- 172.21.15.59:0/1153038922 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 35) ==== 97912+0+0 (secure 0 0 0) 0x7f4c6c005030 con 0x7f4c7c0b2270 2024-06-29T14:37:38.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.930+0000 7f4c637fe640 1 -- 172.21.15.59:0/1153038922 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(90..90 src has 1..90) ==== 6789+0+0 (secure 0 0 0) 0x7f4c6c097f30 con 0x7f4c7c0b2270 2024-06-29T14:37:38.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:38.934+0000 7f4c637fe640 1 -- 172.21.15.59:0/1153038922 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4c6c0c7af0 con 0x7f4c7c0b2270 2024-06-29T14:37:38.954 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.956+0000 7fad1176b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.954 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.956+0000 7fad1176b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.954 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.956+0000 7fad1176b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.954 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.956+0000 7fad1176b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.954 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.956+0000 7fad1176b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.955 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.957+0000 7fad14771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.955 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.957+0000 7fad14771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.955 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.957+0000 7fad14771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.955 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.957+0000 7fad14771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.955 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:37:38.957+0000 7fad14771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:39.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.125+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4c7c155870 con 0x7f4c7c0b2270 2024-06-29T14:37:39.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.126+0000 7f4c637fe640 1 -- 172.21.15.59:0/1153038922 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f4c6c061210 con 0x7f4c7c0b2270 2024-06-29T14:37:39.123 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:37:39.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7c0b2270 msgr2=0x7f4c7c0a9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 --2- 172.21.15.59:0/1153038922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7c0b2270 0x7f4c7c0a9b80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00e9f0 tx=0x7f4c6c00eeb0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 shutdown_connections 2024-06-29T14:37:39.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 --2- 172.21.15.59:0/1153038922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c7c0d1c30 0x7f4c7c0ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 --2- 172.21.15.59:0/1153038922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c7c0b2fa0 0x7f4c7c153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 --2- 172.21.15.59:0/1153038922 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c7c0b2270 0x7f4c7c0a9b80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 >> 172.21.15.59:0/1153038922 conn(0x7f4c7c073e10 msgr2=0x7f4c7c070770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 shutdown_connections 2024-06-29T14:37:39.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.128+0000 7f4c810a5640 1 -- 172.21.15.59:0/1153038922 wait complete. 2024-06-29T14:37:39.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:39.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 -- 172.21.15.59:0/916610558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf808c190 msgr2=0x7fbdf80cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 --2- 172.21.15.59:0/916610558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf808c190 0x7fbdf80cf080 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fbdf80cfbc0 tx=0x7fbde802a560 comp rx=0 tx=0).stop 2024-06-29T14:37:39.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 -- 172.21.15.59:0/916610558 shutdown_connections 2024-06-29T14:37:39.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 --2- 172.21.15.59:0/916610558 >> v2:172.21.15.153:3300/0 conn(0x7fbdf80ce950 0x7fbdf80d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 --2- 172.21.15.59:0/916610558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdf80cf5c0 0x7fbdf80d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 --2- 172.21.15.59:0/916610558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf808c190 0x7fbdf80cf080 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 -- 172.21.15.59:0/916610558 >> 172.21.15.59:0/916610558 conn(0x7fbdf80b5610 msgr2=0x7fbdf80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 -- 172.21.15.59:0/916610558 shutdown_connections 2024-06-29T14:37:39.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.259+0000 7fbdfed2a640 1 -- 172.21.15.59:0/916610558 wait complete. 2024-06-29T14:37:39.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.260+0000 7fbdfed2a640 1 Processor -- start 2024-06-29T14:37:39.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.260+0000 7fbdfed2a640 1 -- start start 2024-06-29T14:37:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfed2a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdf808c190 0x7fbdf8152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfed2a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdf80ce950 0x7fbdf80acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfed2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf80cf5c0 0x7fbdf80ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfed2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf8162440 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfed2a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf81622c0 con 0x7fbdf80ce950 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfed2a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf81625c0 con 0x7fbdf808c190 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfd2c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf80cf5c0 0x7fbdf80ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfd2c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf80cf5c0 0x7fbdf80ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50626/0 (socket says 172.21.15.59:50626) 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfd2c7640 1 -- 172.21.15.59:0/1228110438 learned_addr learned my addr 172.21.15.59:0/1228110438 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfcac6640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdf808c190 0x7fbdf8152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdf7fff640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdf80ce950 0x7fbdf80acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfd2c7640 1 -- 172.21.15.59:0/1228110438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdf808c190 msgr2=0x7fbdf8152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfd2c7640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdf808c190 0x7fbdf8152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfd2c7640 1 -- 172.21.15.59:0/1228110438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdf80ce950 msgr2=0x7fbdf80acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfd2c7640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdf80ce950 0x7fbdf80acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.262+0000 7fbdfd2c7640 1 -- 172.21.15.59:0/1228110438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf81534f0 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.263+0000 7fbdfd2c7640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf80cf5c0 0x7fbdf80ccda0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000eab0 tx=0x7fbdf0009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.263+0000 7fbdf57fa640 1 -- 172.21.15.59:0/1228110438 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf000ce40 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.263+0000 7fbdf57fa640 1 -- 172.21.15.59:0/1228110438 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbdf0004c10 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.263+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf80ad7f0 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.263+0000 7fbdf57fa640 1 -- 172.21.15.59:0/1228110438 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0005020 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.263+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf80addb0 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.264+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf80cf080 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.265+0000 7fbdf57fa640 1 -- 172.21.15.59:0/1228110438 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 35) ==== 97912+0+0 (secure 0 0 0) 0x7fbdf00051c0 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.265+0000 7fbdf57fa640 1 -- 172.21.15.59:0/1228110438 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6789+0+0 (secure 0 0 0) 0x7fbdf0098400 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.270+0000 7fbdf57fa640 1 -- 172.21.15.59:0/1228110438 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbdf81534f0 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.501+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdf80c73f0 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.501+0000 7fbdf57fa640 1 -- 172.21.15.59:0/1228110438 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5230 (secure 0 0 0) 0x7fbdf00615a0 con 0x7fbdf80cf5c0 2024-06-29T14:37:39.499 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:39.499 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":38,"btime":"2024-06-29T14:37:24:318813+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6529,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3719851385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3719851385},{"type":"v1","addr":"172.21.15.153:6836","nonce":3719851385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1280649067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1280649067},{"type":"v1","addr":"172.21.15.59:6833","nonce":1280649067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2847930906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2847930906},{"type":"v1","addr":"172.21.15.59:6835","nonce":2847930906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.672386+0000","modified":"2024-06-29T14:37:23.706036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"d","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/3891057790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3891057790},{"type":"v1","addr":"172.21.15.153:6837","nonce":3891057790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":4}]} 2024-06-29T14:37:39.499 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 38 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf80cf5c0 msgr2=0x7fbdf80ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf80cf5c0 0x7fbdf80ccda0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000eab0 tx=0x7fbdf0009f70 comp rx=0 tx=0).stop 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 shutdown_connections 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbdf80cf5c0 0x7fbdf80ccda0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdf80ce950 0x7fbdf80acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 --2- 172.21.15.59:0/1228110438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbdf808c190 0x7fbdf8152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 >> 172.21.15.59:0/1228110438 conn(0x7fbdf80b5610 msgr2=0x7fbdf80c8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 shutdown_connections 2024-06-29T14:37:39.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.503+0000 7fbdfed2a640 1 -- 172.21.15.59:0/1228110438 wait complete. 2024-06-29T14:37:39.512 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:39.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.633+0000 7f377da38640 1 -- 172.21.15.59:0/3920027475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f37780cb490 msgr2=0x7f3778073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.633+0000 7f377da38640 1 --2- 172.21.15.59:0/3920027475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f37780cb490 0x7f3778073be0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3768009830 tx=0x7f376802a510 comp rx=0 tx=0).stop 2024-06-29T14:37:39.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.633+0000 7f377da38640 1 -- 172.21.15.59:0/3920027475 shutdown_connections 2024-06-29T14:37:39.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.633+0000 7f377da38640 1 --2- 172.21.15.59:0/3920027475 >> v2:172.21.15.153:3300/0 conn(0x7f37780ad6b0 0x7f377815b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.633+0000 7f377da38640 1 --2- 172.21.15.59:0/3920027475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f37780cb490 0x7f3778073be0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.633+0000 7f377da38640 1 -- 172.21.15.59:0/3920027475 >> 172.21.15.59:0/3920027475 conn(0x7f37780b1ea0 msgr2=0x7f37780b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.633+0000 7f377da38640 1 -- 172.21.15.59:0/3920027475 shutdown_connections 2024-06-29T14:37:39.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.633+0000 7f377da38640 1 -- 172.21.15.59:0/3920027475 wait complete. 2024-06-29T14:37:39.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.635+0000 7f377da38640 1 Processor -- start 2024-06-29T14:37:39.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.635+0000 7f377da38640 1 -- start start 2024-06-29T14:37:39.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f377da38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f37780cb490 0x7f377806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f377da38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37780cc060 0x7f37781f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f377da38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37781f7440 0x7f37781f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f377da38640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f377815bbb0 con 0x7f37780cc060 2024-06-29T14:37:39.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f377da38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f377815ba30 con 0x7f37781f7440 2024-06-29T14:37:39.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f377da38640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f377815bd30 con 0x7f37780cb490 2024-06-29T14:37:39.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f37767fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37780cc060 0x7f37781f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f37767fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37780cc060 0x7f37781f2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50628/0 (socket says 172.21.15.59:50628) 2024-06-29T14:37:39.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f37767fc640 1 -- 172.21.15.59:0/2514858852 learned_addr learned my addr 172.21.15.59:0/2514858852 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:39.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f3776ffd640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f37780cb490 0x7f377806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.636+0000 7f37777fe640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37781f7440 0x7f37781f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f37767fc640 1 -- 172.21.15.59:0/2514858852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f37780cb490 msgr2=0x7f377806c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f37767fc640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f37780cb490 0x7f377806c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f37767fc640 1 -- 172.21.15.59:0/2514858852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37781f7440 msgr2=0x7f37781f3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f37767fc640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37781f7440 0x7f37781f3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f37767fc640 1 -- 172.21.15.59:0/2514858852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3778153e50 con 0x7f37780cc060 2024-06-29T14:37:39.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f37767fc640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37780cc060 0x7f37781f2d10 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f376c00c340 tx=0x7f376c00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f375ffff640 1 -- 172.21.15.59:0/2514858852 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c011790 con 0x7f37780cc060 2024-06-29T14:37:39.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f375ffff640 1 -- 172.21.15.59:0/2514858852 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f376c017070 con 0x7f37780cc060 2024-06-29T14:37:39.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3778154410 con 0x7f37780cc060 2024-06-29T14:37:39.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f375ffff640 1 -- 172.21.15.59:0/2514858852 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c011cc0 con 0x7f37780cc060 2024-06-29T14:37:39.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.637+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37781f35a0 con 0x7f37780cc060 2024-06-29T14:37:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.638+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3778050470 con 0x7f37780cc060 2024-06-29T14:37:39.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.639+0000 7f375ffff640 1 -- 172.21.15.59:0/2514858852 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 35) ==== 97912+0+0 (secure 0 0 0) 0x7f376c01c460 con 0x7f37780cc060 2024-06-29T14:37:39.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.640+0000 7f375ffff640 1 -- 172.21.15.59:0/2514858852 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6789+0+0 (secure 0 0 0) 0x7f376c0974f0 con 0x7f37780cc060 2024-06-29T14:37:39.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.644+0000 7f375ffff640 1 -- 172.21.15.59:0/2514858852 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3778050470 con 0x7f37780cc060 2024-06-29T14:37:39.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.840+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37780b00b0 con 0x7f37780cc060 2024-06-29T14:37:39.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.842+0000 7f375ffff640 1 -- 172.21.15.59:0/2514858852 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+17910 (secure 0 0 0) 0x7f37780b00b0 con 0x7f37780cc060 2024-06-29T14:37:39.839 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:39.839 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":90,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:38.892528+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:19.028073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.631284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/131993077":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/3045910470":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/2009989783":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/277169477":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/3061329079":"2024-06-30T14:37:38.892471+0000","172.21.15.153:6832/3110229799":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/1132863181":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/901558684":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6832/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6833/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2816930410":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/1030990875":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/2052586834":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6833/3110229799":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/62694730":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2198109495":"2024-06-30T14:37:32.999861+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37780cc060 msgr2=0x7f37781f2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37780cc060 0x7f37781f2d10 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f376c00c340 tx=0x7f376c00c740 comp rx=0 tx=0).stop 2024-06-29T14:37:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 shutdown_connections 2024-06-29T14:37:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37781f7440 0x7f37781f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37780cc060 0x7f37781f2d10 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 --2- 172.21.15.59:0/2514858852 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f37780cb490 0x7f377806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 >> 172.21.15.59:0/2514858852 conn(0x7f37780b1ea0 msgr2=0x7f37780b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 shutdown_connections 2024-06-29T14:37:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.844+0000 7f377da38640 1 -- 172.21.15.59:0/2514858852 wait complete. 2024-06-29T14:37:39.853 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:37:39.853 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:39.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.977+0000 7f117bbc7640 1 -- 172.21.15.59:0/3554521406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 msgr2=0x7f11740ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.977+0000 7f117bbc7640 1 --2- 172.21.15.59:0/3554521406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 0x7f11740ce530 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1164005fd0 tx=0x7f116402bc80 comp rx=0 tx=0).stop 2024-06-29T14:37:39.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.977+0000 7f117bbc7640 1 -- 172.21.15.59:0/3554521406 shutdown_connections 2024-06-29T14:37:39.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.977+0000 7f117bbc7640 1 --2- 172.21.15.59:0/3554521406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 0x7f11740ce530 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.977+0000 7f117bbc7640 1 -- 172.21.15.59:0/3554521406 >> 172.21.15.59:0/3554521406 conn(0x7f11740b24c0 msgr2=0x7f11740cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.977+0000 7f117bbc7640 1 -- 172.21.15.59:0/3554521406 shutdown_connections 2024-06-29T14:37:39.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.977+0000 7f117bbc7640 1 -- 172.21.15.59:0/3554521406 wait complete. 2024-06-29T14:37:39.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.979+0000 7f117bbc7640 1 Processor -- start 2024-06-29T14:37:39.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.979+0000 7f117bbc7640 1 -- start start 2024-06-29T14:37:39.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f117bbc7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11740b3800 0x7f117413a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f117bbc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 0x7f11741e3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f117bbc7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11741e4090 0x7f1174158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f117bbc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f117415e5f0 con 0x7f11740cf380 2024-06-29T14:37:39.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f117bbc7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11740700b0 con 0x7f11741e4090 2024-06-29T14:37:39.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f117bbc7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f117415e770 con 0x7f11740b3800 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f1179162640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 0x7f11741e3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f1179162640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 0x7f11741e3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50648/0 (socket says 172.21.15.59:50648) 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.980+0000 7f1179162640 1 -- 172.21.15.59:0/3155441962 learned_addr learned my addr 172.21.15.59:0/3155441962 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f117a164640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11741e4090 0x7f1174158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f1179162640 1 -- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11740b3800 msgr2=0x7f117413a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f1179162640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11740b3800 0x7f117413a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f1179162640 1 -- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11741e4090 msgr2=0x7f1174158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f1179162640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11741e4090 0x7f1174158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f1179162640 1 -- 172.21.15.59:0/3155441962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117413acd0 con 0x7f11740cf380 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f1179162640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 0x7f11741e3b50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f116800c400 tx=0x7f116800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f11727fc640 1 -- 172.21.15.59:0/3155441962 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168017070 con 0x7f11740cf380 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f11727fc640 1 -- 172.21.15.59:0/3155441962 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11680053c0 con 0x7f11740cf380 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f11727fc640 1 -- 172.21.15.59:0/3155441962 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168003850 con 0x7f11740cf380 2024-06-29T14:37:39.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11741e48f0 con 0x7f11740cf380 2024-06-29T14:37:39.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.981+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f117414e680 con 0x7f11740cf380 2024-06-29T14:37:39.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.982+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11741e4610 con 0x7f11740cf380 2024-06-29T14:37:39.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.983+0000 7f11727fc640 1 -- 172.21.15.59:0/3155441962 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7f116801a070 con 0x7f11740cf380 2024-06-29T14:37:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.983+0000 7f11727fc640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1160076120 0x7f1160078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.983+0000 7f11727fc640 1 -- 172.21.15.59:0/3155441962 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6789+0+0 (secure 0 0 0) 0x7f1168096c00 con 0x7f11740cf380 2024-06-29T14:37:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.984+0000 7f1179963640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1160076120 0x7f1160078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.984+0000 7f1179963640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1160076120 0x7f1160078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1164035000 tx=0x7f1164004a40 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:39.988+0000 7f11727fc640 1 -- 172.21.15.59:0/3155441962 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1168060460 con 0x7f11740cf380 2024-06-29T14:37:40.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.214+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f117415b900 con 0x7f11740cf380 2024-06-29T14:37:40.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.215+0000 7f11727fc640 1 -- 172.21.15.59:0/3155441962 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5230 (secure 0 0 0) 0x7f116805fd60 con 0x7f11740cf380 2024-06-29T14:37:40.212 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:40.212 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":38,"btime":"2024-06-29T14:37:24:318813+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6529,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3719851385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3719851385},{"type":"v1","addr":"172.21.15.153:6836","nonce":3719851385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1280649067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1280649067},{"type":"v1","addr":"172.21.15.59:6833","nonce":1280649067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2847930906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2847930906},{"type":"v1","addr":"172.21.15.59:6835","nonce":2847930906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.672386+0000","modified":"2024-06-29T14:37:23.706036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"d","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/3891057790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3891057790},{"type":"v1","addr":"172.21.15.153:6837","nonce":3891057790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":4}]} 2024-06-29T14:37:40.212 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 38 2024-06-29T14:37:40.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1160076120 msgr2=0x7f1160078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1160076120 0x7f1160078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1164035000 tx=0x7f1164004a40 comp rx=0 tx=0).stop 2024-06-29T14:37:40.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 msgr2=0x7f11741e3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 0x7f11741e3b50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f116800c400 tx=0x7f116800c8c0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 shutdown_connections 2024-06-29T14:37:40.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11741e4090 0x7f1174158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11740cf380 0x7f11741e3b50 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1160076120 0x7f1160078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 --2- 172.21.15.59:0/3155441962 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11740b3800 0x7f117413a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 >> 172.21.15.59:0/3155441962 conn(0x7f11740b24c0 msgr2=0x7f11740b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 shutdown_connections 2024-06-29T14:37:40.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.217+0000 7f117bbc7640 1 -- 172.21.15.59:0/3155441962 wait complete. 2024-06-29T14:37:40.225 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.346+0000 7faf8d83d640 1 -- 172.21.15.59:0/2547406408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf70003a70 msgr2=0x7faf70003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.346+0000 7faf8d83d640 1 --2- 172.21.15.59:0/2547406408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf70003a70 0x7faf70003ee0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7faf78009b50 tx=0x7faf70005830 comp rx=0 tx=0).stop 2024-06-29T14:37:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.346+0000 7faf877fe640 1 -- 172.21.15.59:0/2547406408 reap_dead start 2024-06-29T14:37:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.346+0000 7faf8d83d640 1 -- 172.21.15.59:0/2547406408 shutdown_connections 2024-06-29T14:37:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.346+0000 7faf8d83d640 1 -- 172.21.15.59:0/2547406408 >> 172.21.15.59:0/2547406408 conn(0x7faf880b5610 msgr2=0x7faf880b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.346+0000 7faf8d83d640 1 -- 172.21.15.59:0/2547406408 shutdown_connections 2024-06-29T14:37:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.346+0000 7faf8d83d640 1 -- 172.21.15.59:0/2547406408 wait complete. 2024-06-29T14:37:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.348+0000 7faf8d83d640 1 Processor -- start 2024-06-29T14:37:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.348+0000 7faf8d83d640 1 -- start start 2024-06-29T14:37:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.349+0000 7faf8d83d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf70003a70 0x7faf880adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.349+0000 7faf8d83d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf881e9310 0x7faf881e9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.349+0000 7faf8d83d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf881511e0 0x7faf881515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.349+0000 7faf8d83d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7faf880b7bd0 con 0x7faf881511e0 2024-06-29T14:37:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.349+0000 7faf8d83d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf880d1c70 con 0x7faf70003a70 2024-06-29T14:37:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.349+0000 7faf8d83d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7faf880b7d50 con 0x7faf881e9310 2024-06-29T14:37:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf881511e0 0x7faf881515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf881511e0 0x7faf881515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50658/0 (socket says 172.21.15.59:50658) 2024-06-29T14:37:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 -- 172.21.15.59:0/884154037 learned_addr learned my addr 172.21.15.59:0/884154037 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf86ffd640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf70003a70 0x7faf880adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf867fc640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf881e9310 0x7faf881e9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 -- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf881e9310 msgr2=0x7faf881e9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf881e9310 0x7faf881e9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 -- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf70003a70 msgr2=0x7faf880adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf70003a70 0x7faf880adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 -- 172.21.15.59:0/884154037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8814ead0 con 0x7faf881511e0 2024-06-29T14:37:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf877fe640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf881511e0 0x7faf881515f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faf880b7620 tx=0x7faf80006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf6ffff640 1 -- 172.21.15.59:0/884154037 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8000fc90 con 0x7faf881511e0 2024-06-29T14:37:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf6ffff640 1 -- 172.21.15.59:0/884154037 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf8000fe30 con 0x7faf881511e0 2024-06-29T14:37:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf88151b30 con 0x7faf881511e0 2024-06-29T14:37:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf6ffff640 1 -- 172.21.15.59:0/884154037 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf80011670 con 0x7faf881511e0 2024-06-29T14:37:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.350+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf880ab870 con 0x7faf881511e0 2024-06-29T14:37:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.352+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf880ab300 con 0x7faf881511e0 2024-06-29T14:37:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.352+0000 7faf6ffff640 1 -- 172.21.15.59:0/884154037 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7faf80011810 con 0x7faf881511e0 2024-06-29T14:37:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.353+0000 7faf6ffff640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf74076150 0x7faf74078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.353+0000 7faf6ffff640 1 -- 172.21.15.59:0/884154037 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6789+0+0 (secure 0 0 0) 0x7faf80098340 con 0x7faf881511e0 2024-06-29T14:37:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.353+0000 7faf86ffd640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf74076150 0x7faf74078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.354+0000 7faf86ffd640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf74076150 0x7faf74078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faf78035000 tx=0x7faf7802d040 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.357+0000 7faf6ffff640 1 -- 172.21.15.59:0/884154037 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf800619c0 con 0x7faf881511e0 2024-06-29T14:37:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.586+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf8814eca0 con 0x7faf881511e0 2024-06-29T14:37:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.586+0000 7faf6ffff640 1 -- 172.21.15.59:0/884154037 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+5230 (secure 0 0 0) 0x7faf800612c0 con 0x7faf881511e0 2024-06-29T14:37:40.584 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:40.584 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":38,"btime":"2024-06-29T14:37:24:318813+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6529,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3719851385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3719851385},{"type":"v1","addr":"172.21.15.153:6836","nonce":3719851385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1280649067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1280649067},{"type":"v1","addr":"172.21.15.59:6833","nonce":1280649067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2847930906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2847930906},{"type":"v1","addr":"172.21.15.59:6835","nonce":2847930906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.672386+0000","modified":"2024-06-29T14:37:23.706036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6864},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6864":{"gid":6864,"name":"d","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/3891057790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3891057790},{"type":"v1","addr":"172.21.15.153:6837","nonce":3891057790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6864,"qdb_cluster":[6864]},"id":4}]} 2024-06-29T14:37:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 38 2024-06-29T14:37:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf74076150 msgr2=0x7faf74078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf74076150 0x7faf74078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faf78035000 tx=0x7faf7802d040 comp rx=0 tx=0).stop 2024-06-29T14:37:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf881511e0 msgr2=0x7faf881515f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf881511e0 0x7faf881515f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faf880b7620 tx=0x7faf80006660 comp rx=0 tx=0).stop 2024-06-29T14:37:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 shutdown_connections 2024-06-29T14:37:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf881511e0 0x7faf881515f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf74076150 0x7faf74078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf881e9310 0x7faf881e9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 --2- 172.21.15.59:0/884154037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf70003a70 0x7faf880adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 >> 172.21.15.59:0/884154037 conn(0x7faf880b5610 msgr2=0x7faf8815dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 shutdown_connections 2024-06-29T14:37:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.589+0000 7faf8d83d640 1 -- 172.21.15.59:0/884154037 wait complete. 2024-06-29T14:37:40.597 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:40.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.718+0000 7f138ec95640 1 -- 172.21.15.59:0/70471881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13880b3ed0 msgr2=0x7f138815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.718+0000 7f138ec95640 1 --2- 172.21.15.59:0/70471881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13880b3ed0 0x7f138815e8d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1378009830 tx=0x7f137802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.718+0000 7f138ec95640 1 -- 172.21.15.59:0/70471881 shutdown_connections 2024-06-29T14:37:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.718+0000 7f138ec95640 1 --2- 172.21.15.59:0/70471881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13880b3ed0 0x7f138815e8d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.718+0000 7f138ec95640 1 --2- 172.21.15.59:0/70471881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13880d23c0 0x7f13880b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.718+0000 7f138ec95640 1 -- 172.21.15.59:0/70471881 >> 172.21.15.59:0/70471881 conn(0x7f13880d3ee0 msgr2=0x7f13880d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.718+0000 7f138ec95640 1 -- 172.21.15.59:0/70471881 shutdown_connections 2024-06-29T14:37:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.718+0000 7f138ec95640 1 -- 172.21.15.59:0/70471881 wait complete. 2024-06-29T14:37:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.720+0000 7f138ec95640 1 Processor -- start 2024-06-29T14:37:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.720+0000 7f138ec95640 1 -- start start 2024-06-29T14:37:40.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ec95640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13880b3ed0 0x7f1388156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ec95640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13880d23c0 0x7f138815d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ec95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f138815ee10 0x7f138815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ec95640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1388160240 con 0x7f13880b3ed0 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ec95640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13881600c0 con 0x7f138815ee10 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ec95640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f13881603c0 con 0x7f13880d23c0 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ca31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13880b3ed0 0x7f1388156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ca31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13880b3ed0 0x7f1388156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50674/0 (socket says 172.21.15.59:50674) 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138ca31640 1 -- 172.21.15.59:0/1177351771 learned_addr learned my addr 172.21.15.59:0/1177351771 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f137ffff640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13880d23c0 0x7f138815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.721+0000 7f138d232640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f138815ee10 0x7f138815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f138ca31640 1 -- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13880d23c0 msgr2=0x7f138815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f138ca31640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13880d23c0 0x7f138815d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f138ca31640 1 -- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f138815ee10 msgr2=0x7f138815dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f138ca31640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f138815ee10 0x7f138815dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f138ca31640 1 -- 172.21.15.59:0/1177351771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13881f6160 con 0x7f13880b3ed0 2024-06-29T14:37:40.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f138ca31640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13880b3ed0 0x7f1388156110 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f138400eae0 tx=0x7f138400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f137d7fa640 1 -- 172.21.15.59:0/1177351771 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384003970 con 0x7f13880b3ed0 2024-06-29T14:37:40.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f137d7fa640 1 -- 172.21.15.59:0/1177351771 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1384003c00 con 0x7f13880b3ed0 2024-06-29T14:37:40.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f137d7fa640 1 -- 172.21.15.59:0/1177351771 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384004030 con 0x7f13880b3ed0 2024-06-29T14:37:40.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13881f6440 con 0x7f13880b3ed0 2024-06-29T14:37:40.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.722+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1388159100 con 0x7f13880b3ed0 2024-06-29T14:37:40.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.723+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138815e8d0 con 0x7f13880b3ed0 2024-06-29T14:37:40.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.724+0000 7f137d7fa640 1 -- 172.21.15.59:0/1177351771 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7f13840041d0 con 0x7f13880b3ed0 2024-06-29T14:37:40.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.724+0000 7f137d7fa640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1374076120 0x7f1374078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.724+0000 7f137d7fa640 1 -- 172.21.15.59:0/1177351771 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6789+0+0 (secure 0 0 0) 0x7f13840985a0 con 0x7f13880b3ed0 2024-06-29T14:37:40.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.725+0000 7f137ffff640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1374076120 0x7f1374078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.725+0000 7f137ffff640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1374076120 0x7f1374078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1378034000 tx=0x7f1378004730 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.728+0000 7f137d7fa640 1 -- 172.21.15.59:0/1177351771 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f13840617b0 con 0x7f13880b3ed0 2024-06-29T14:37:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.924+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13880d2950 con 0x7f13880b3ed0 2024-06-29T14:37:40.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.925+0000 7f137d7fa640 1 -- 172.21.15.59:0/1177351771 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+17910 (secure 0 0 0) 0x7f1384065ab0 con 0x7f13880b3ed0 2024-06-29T14:37:40.922 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:40.922 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":90,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:38.892528+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:19.028073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.631284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/131993077":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/3045910470":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/2009989783":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/277169477":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/3061329079":"2024-06-30T14:37:38.892471+0000","172.21.15.153:6832/3110229799":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/1132863181":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/901558684":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6832/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6833/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2816930410":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/1030990875":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/2052586834":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6833/3110229799":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/62694730":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2198109495":"2024-06-30T14:37:32.999861+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1374076120 msgr2=0x7f1374078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1374076120 0x7f1374078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1378034000 tx=0x7f1378004730 comp rx=0 tx=0).stop 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13880b3ed0 msgr2=0x7f1388156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13880b3ed0 0x7f1388156110 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f138400eae0 tx=0x7f138400c2d0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 shutdown_connections 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f138815ee10 0x7f138815dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1374076120 0x7f1374078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13880d23c0 0x7f138815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 --2- 172.21.15.59:0/1177351771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13880b3ed0 0x7f1388156110 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 >> 172.21.15.59:0/1177351771 conn(0x7f13880d3ee0 msgr2=0x7f13880ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 shutdown_connections 2024-06-29T14:37:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:40.927+0000 7f138ec95640 1 -- 172.21.15.59:0/1177351771 wait complete. 2024-06-29T14:37:40.936 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:37:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.058+0000 7f976298c640 1 -- 172.21.15.59:0/1303793650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9744004800 msgr2=0x7f9744004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.058+0000 7f976298c640 1 --2- 172.21.15.59:0/1303793650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9744004800 0x7f9744004c10 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f9750009b50 tx=0x7f9744005830 comp rx=0 tx=0).stop 2024-06-29T14:37:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.059+0000 7f976298c640 1 -- 172.21.15.59:0/1303793650 shutdown_connections 2024-06-29T14:37:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.059+0000 7f976298c640 1 --2- 172.21.15.59:0/1303793650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9744004800 0x7f9744004c10 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.059+0000 7f976298c640 1 -- 172.21.15.59:0/1303793650 >> 172.21.15.59:0/1303793650 conn(0x7f975c0d45b0 msgr2=0x7f975c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.059+0000 7f976298c640 1 -- 172.21.15.59:0/1303793650 shutdown_connections 2024-06-29T14:37:41.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.059+0000 7f976298c640 1 -- 172.21.15.59:0/1303793650 wait complete. 2024-06-29T14:37:41.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.060+0000 7f976298c640 1 Processor -- start 2024-06-29T14:37:41.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.061+0000 7f976298c640 1 -- start start 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f976298c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9744004800 0x7f975c161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f976298c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f975c15ae20 0x7f975c0b8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f976298c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f975c15b230 0x7f975c0b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f976298c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f975c161a80 con 0x7f975c15b230 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f976298c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f975c0ce7c0 con 0x7f9744004800 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f976298c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f975c161c00 con 0x7f975c15ae20 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f9760f29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f975c15b230 0x7f975c0b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f9760f29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f975c15b230 0x7f975c0b91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50686/0 (socket says 172.21.15.59:50686) 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f9760f29640 1 -- 172.21.15.59:0/3850534430 learned_addr learned my addr 172.21.15.59:0/3850534430 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f975bfff640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9744004800 0x7f975c161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f975b7fe640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f975c15ae20 0x7f975c0b8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f9760f29640 1 -- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f975c15ae20 msgr2=0x7f975c0b8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f9760f29640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f975c15ae20 0x7f975c0b8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f9760f29640 1 -- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9744004800 msgr2=0x7f975c161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f9760f29640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9744004800 0x7f975c161000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f9760f29640 1 -- 172.21.15.59:0/3850534430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c1f7740 con 0x7f975c15b230 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f975bfff640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9744004800 0x7f975c161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.062+0000 7f975b7fe640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f975c15ae20 0x7f975c0b8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.063+0000 7f9760f29640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f975c15b230 0x7f975c0b91a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f975c0ce310 tx=0x7f9754006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.063+0000 7f9758ff9640 1 -- 172.21.15.59:0/3850534430 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975400fa20 con 0x7f975c15b230 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.063+0000 7f9758ff9640 1 -- 172.21.15.59:0/3850534430 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9754018070 con 0x7f975c15b230 2024-06-29T14:37:41.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.063+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975c1f7a20 con 0x7f975c15b230 2024-06-29T14:37:41.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.063+0000 7f9758ff9640 1 -- 172.21.15.59:0/3850534430 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975400fe60 con 0x7f975c15b230 2024-06-29T14:37:41.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.063+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f975c1f7fe0 con 0x7f975c15b230 2024-06-29T14:37:41.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.064+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f975c0b25c0 con 0x7f975c15b230 2024-06-29T14:37:41.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.064+0000 7f9758ff9640 1 -- 172.21.15.59:0/3850534430 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7f975401d460 con 0x7f975c15b230 2024-06-29T14:37:41.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.065+0000 7f9758ff9640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f9748076120 0x7f9748078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.065+0000 7f9758ff9640 1 -- 172.21.15.59:0/3850534430 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6789+0+0 (secure 0 0 0) 0x7f9754097f20 con 0x7f975c15b230 2024-06-29T14:37:41.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.065+0000 7f975bfff640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f9748076120 0x7f9748078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.066+0000 7f975bfff640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f9748076120 0x7f9748078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f974c000c00 tx=0x7f974c006b70 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:41.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.070+0000 7f9758ff9640 1 -- 172.21.15.59:0/3850534430 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9754061840 con 0x7f975c15b230 2024-06-29T14:37:41.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.247+0000 7f9758ff9640 1 -- 172.21.15.59:0/3850534430 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 37) ==== 98247+0+0 (secure 0 0 0) 0x7f975405d3d0 con 0x7f975c15b230 2024-06-29T14:37:41.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.301+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f975c162160 con 0x7f975c15b230 2024-06-29T14:37:41.299 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T14:37:41.302+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:37:41.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.303+0000 7f9758ff9640 1 -- 172.21.15.59:0/3850534430 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v39) ==== 161+0+0 (secure 0 0 0) 0x7f9754087850 con 0x7f975c15b230 2024-06-29T14:37:41.300 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:37:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f9748076120 msgr2=0x7f9748078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f9748076120 0x7f9748078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f974c000c00 tx=0x7f974c006b70 comp rx=0 tx=0).stop 2024-06-29T14:37:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f975c15b230 msgr2=0x7f975c0b91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f975c15b230 0x7f975c0b91a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f975c0ce310 tx=0x7f9754006720 comp rx=0 tx=0).stop 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 shutdown_connections 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f975c15b230 0x7f975c0b91a0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f9748076120 0x7f9748078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f975c15ae20 0x7f975c0b8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 --2- 172.21.15.59:0/3850534430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9744004800 0x7f975c161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 >> 172.21.15.59:0/3850534430 conn(0x7f975c0d45b0 msgr2=0x7f975c0b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 shutdown_connections 2024-06-29T14:37:41.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.306+0000 7f976298c640 1 -- 172.21.15.59:0/3850534430 wait complete. 2024-06-29T14:37:41.305 INFO:tasks.ceph.mds.d.smithi153.stderr: -4147> 2024-06-29T14:37:23.269+0000 7faa379c0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:37:41.315 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:37:41.361 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:37:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.437+0000 7f0a11ea7640 1 -- 172.21.15.59:0/4243849279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0c0d1130 msgr2=0x7f0a0c15f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.437+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/4243849279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0c0d1130 0x7f0a0c15f7f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009830 tx=0x7f0a0002a510 comp rx=0 tx=0).stop 2024-06-29T14:37:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.437+0000 7f0a11ea7640 1 -- 172.21.15.59:0/4243849279 shutdown_connections 2024-06-29T14:37:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.437+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/4243849279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0c0d1130 0x7f0a0c15f7f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.437+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/4243849279 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0c145650 0x7f0a0c0b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.437+0000 7f0a11ea7640 1 -- 172.21.15.59:0/4243849279 >> 172.21.15.59:0/4243849279 conn(0x7f0a0c0b6ed0 msgr2=0x7f0a0c0b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.437+0000 7f0a11ea7640 1 -- 172.21.15.59:0/4243849279 shutdown_connections 2024-06-29T14:37:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.437+0000 7f0a11ea7640 1 -- 172.21.15.59:0/4243849279 wait complete. 2024-06-29T14:37:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.439+0000 7f0a11ea7640 1 Processor -- start 2024-06-29T14:37:41.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.439+0000 7f0a11ea7640 1 -- start start 2024-06-29T14:37:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a11ea7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0c0d1130 0x7f0a0c1f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a11ea7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0c145650 0x7f0a0c1f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a11ea7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c15fd30 0x7f0a0c1f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a11ea7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c15c910 con 0x7f0a0c15fd30 2024-06-29T14:37:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a11ea7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c15eba0 con 0x7f0a0c0d1130 2024-06-29T14:37:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a11ea7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0a0c15ca90 con 0x7f0a0c145650 2024-06-29T14:37:41.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c15fd30 0x7f0a0c1f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c15fd30 0x7f0a0c1f3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50694/0 (socket says 172.21.15.59:50694) 2024-06-29T14:37:41.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 -- 172.21.15.59:0/3685660274 learned_addr learned my addr 172.21.15.59:0/3685660274 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:41.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0b7fe640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0c0d1130 0x7f0a0c1f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0affd640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0c145650 0x7f0a0c1f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 -- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0c145650 msgr2=0x7f0a0c1f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0c145650 0x7f0a0c1f2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 -- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0c0d1130 msgr2=0x7f0a0c1f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0c0d1130 0x7f0a0c1f5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 -- 172.21.15.59:0/3685660274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0c1f3bf0 con 0x7f0a0c15fd30 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f0a0bfff640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c15fd30 0x7f0a0c1f3390 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f0a0400caa0 tx=0x7f0a0400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.441+0000 7f09f3fff640 1 -- 172.21.15.59:0/3685660274 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04016020 con 0x7f0a0c15fd30 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.442+0000 7f09f3fff640 1 -- 172.21.15.59:0/3685660274 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a0400ac90 con 0x7f0a0c15fd30 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.442+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0c1f3ed0 con 0x7f0a0c15fd30 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.442+0000 7f09f3fff640 1 -- 172.21.15.59:0/3685660274 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04012680 con 0x7f0a0c15fd30 2024-06-29T14:37:41.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.442+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0c141e80 con 0x7f0a0c15fd30 2024-06-29T14:37:41.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.443+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a0c15cf80 con 0x7f0a0c15fd30 2024-06-29T14:37:41.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.443+0000 7f09f3fff640 1 -- 172.21.15.59:0/3685660274 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 37) ==== 98247+0+0 (secure 0 0 0) 0x7f0a04019020 con 0x7f0a0c15fd30 2024-06-29T14:37:41.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.444+0000 7f09f3fff640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f09f8076360 0x7f09f8078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.444+0000 7f09f3fff640 1 -- 172.21.15.59:0/3685660274 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6875+0+0 (secure 0 0 0) 0x7f0a04097e60 con 0x7f0a0c15fd30 2024-06-29T14:37:41.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.444+0000 7f0a0b7fe640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f09f8076360 0x7f09f8078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.445+0000 7f0a0b7fe640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f09f8076360 0x7f09f8078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009b30 tx=0x7f09fc005f90 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:41.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.448+0000 7f09f3fff640 1 -- 172.21.15.59:0/3685660274 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0a04020030 con 0x7f0a0c15fd30 2024-06-29T14:37:41.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:41.680+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f0a0c15bcf0 con 0x7f0a0c15fd30 2024-06-29T14:37:42.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.305+0000 7f09f3fff640 1 -- 172.21.15.59:0/3685660274 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v40) ==== 104+0+0 (secure 0 0 0) 0x7f0a0c160140 con 0x7f0a0c15fd30 2024-06-29T14:37:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f09f8076360 msgr2=0x7f09f8078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f09f8076360 0x7f09f8078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009b30 tx=0x7f09fc005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c15fd30 msgr2=0x7f0a0c1f3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c15fd30 0x7f0a0c1f3390 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f0a0400caa0 tx=0x7f0a0400cf60 comp rx=0 tx=0).stop 2024-06-29T14:37:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 shutdown_connections 2024-06-29T14:37:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c15fd30 0x7f0a0c1f3390 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f09f8076360 0x7f09f8078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a0c145650 0x7f0a0c1f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 --2- 172.21.15.59:0/3685660274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a0c0d1130 0x7f0a0c1f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.307+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 >> 172.21.15.59:0/3685660274 conn(0x7f0a0c0b6ed0 msgr2=0x7f0a0c068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.308+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 shutdown_connections 2024-06-29T14:37:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.308+0000 7f0a11ea7640 1 -- 172.21.15.59:0/3685660274 wait complete. 2024-06-29T14:37:42.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:37:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.449+0000 7ff48f2df640 1 -- 172.21.15.59:0/1251554288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 msgr2=0x7ff470003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.449+0000 7ff48f2df640 1 --2- 172.21.15.59:0/1251554288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 0x7ff470003ec0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7ff48000c290 tx=0x7ff470005830 comp rx=0 tx=0).stop 2024-06-29T14:37:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.449+0000 7ff48f2df640 1 -- 172.21.15.59:0/1251554288 shutdown_connections 2024-06-29T14:37:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.449+0000 7ff48f2df640 1 --2- 172.21.15.59:0/1251554288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 0x7ff470003ec0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.449+0000 7ff48f2df640 1 -- 172.21.15.59:0/1251554288 >> 172.21.15.59:0/1251554288 conn(0x7ff4880b7690 msgr2=0x7ff488073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.449+0000 7ff48f2df640 1 -- 172.21.15.59:0/1251554288 shutdown_connections 2024-06-29T14:37:42.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.449+0000 7ff48f2df640 1 -- 172.21.15.59:0/1251554288 wait complete. 2024-06-29T14:37:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.451+0000 7ff48f2df640 1 Processor -- start 2024-06-29T14:37:42.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.451+0000 7ff48f2df640 1 -- start start 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48f2df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 0x7ff4880cff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48f2df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff488155c80 0x7ff488152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48f2df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4880b4940 0x7ff4880b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48f2df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff4880b08a0 con 0x7ff470003a50 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48f2df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4880b0720 con 0x7ff4880b4940 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48f2df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff4880b0a20 con 0x7ff488155c80 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 0x7ff4880cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 0x7ff4880cff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50702/0 (socket says 172.21.15.59:50702) 2024-06-29T14:37:42.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 -- 172.21.15.59:0/3639416935 learned_addr learned my addr 172.21.15.59:0/3639416935 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 -- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff488155c80 msgr2=0x7ff488152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff488155c80 0x7ff488152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 -- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4880b4940 msgr2=0x7ff4880b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4880b4940 0x7ff4880b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 -- 172.21.15.59:0/3639416935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4880b5290 con 0x7ff470003a50 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48d07b640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 0x7ff4880cff50 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff48000c3b0 tx=0x7ff480006d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff485ffb640 1 -- 172.21.15.59:0/3639416935 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480031040 con 0x7ff470003a50 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff485ffb640 1 -- 172.21.15.59:0/3639416935 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4800035f0 con 0x7ff470003a50 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff485ffb640 1 -- 172.21.15.59:0/3639416935 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4800039e0 con 0x7ff470003a50 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.453+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4880d0850 con 0x7ff470003a50 2024-06-29T14:37:42.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.454+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4880d0e10 con 0x7ff470003a50 2024-06-29T14:37:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.454+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4880b0f80 con 0x7ff470003a50 2024-06-29T14:37:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.455+0000 7ff485ffb640 1 -- 172.21.15.59:0/3639416935 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 37) ==== 98247+0+0 (secure 0 0 0) 0x7ff48003e030 con 0x7ff470003a50 2024-06-29T14:37:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.456+0000 7ff485ffb640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7ff474076360 0x7ff474078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.456+0000 7ff485ffb640 1 -- 172.21.15.59:0/3639416935 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6875+0+0 (secure 0 0 0) 0x7ff4800b7930 con 0x7ff470003a50 2024-06-29T14:37:42.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.456+0000 7ff48c87a640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7ff474076360 0x7ff474078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:42.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.457+0000 7ff48c87a640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7ff474076360 0x7ff474078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff47c006850 tx=0x7ff47c0067e0 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:42.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.460+0000 7ff485ffb640 1 -- 172.21.15.59:0/3639416935 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff4880b5290 con 0x7ff470003a50 2024-06-29T14:37:42.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:42.650+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff4880d1da0 con 0x7ff470003a50 2024-06-29T14:37:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.312+0000 7ff485ffb640 1 -- 172.21.15.59:0/3639416935 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) ==== 181+0+0 (secure 0 0 0) 0x7ff480080d00 con 0x7ff470003a50 2024-06-29T14:37:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:37:43.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.319+0000 7ff485ffb640 1 -- 172.21.15.59:0/3639416935 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7ff480034070 con 0x7ff470003a50 2024-06-29T14:37:43.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7ff474076360 msgr2=0x7ff474078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7ff474076360 0x7ff474078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff47c006850 tx=0x7ff47c0067e0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 msgr2=0x7ff4880cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 0x7ff4880cff50 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff48000c3b0 tx=0x7ff480006d00 comp rx=0 tx=0).stop 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 shutdown_connections 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4880b4940 0x7ff4880b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7ff474076360 0x7ff474078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff488155c80 0x7ff488152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 --2- 172.21.15.59:0/3639416935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff470003a50 0x7ff4880cff50 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 >> 172.21.15.59:0/3639416935 conn(0x7ff4880b7690 msgr2=0x7ff488155250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 shutdown_connections 2024-06-29T14:37:43.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.320+0000 7ff48f2df640 1 -- 172.21.15.59:0/3639416935 wait complete. 2024-06-29T14:37:43.331 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:37:43.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.460+0000 7f41b8bd2640 1 -- 172.21.15.59:0/216381119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b40b4ee0 msgr2=0x7f41b4168c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.460+0000 7f41b8bd2640 1 --2- 172.21.15.59:0/216381119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b40b4ee0 0x7f41b4168c60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f41ac005fd0 tx=0x7f41ac02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.460+0000 7f41b8bd2640 1 -- 172.21.15.59:0/216381119 shutdown_connections 2024-06-29T14:37:43.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.460+0000 7f41b8bd2640 1 --2- 172.21.15.59:0/216381119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b40b4ee0 0x7f41b4168c60 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.460+0000 7f41b8bd2640 1 -- 172.21.15.59:0/216381119 >> 172.21.15.59:0/216381119 conn(0x7f41b40d3e60 msgr2=0x7f41b40d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:43.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.461+0000 7f41b8bd2640 1 -- 172.21.15.59:0/216381119 shutdown_connections 2024-06-29T14:37:43.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.461+0000 7f41b8bd2640 1 -- 172.21.15.59:0/216381119 wait complete. 2024-06-29T14:37:43.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.462+0000 7f41b8bd2640 1 Processor -- start 2024-06-29T14:37:43.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.462+0000 7f41b8bd2640 1 -- start start 2024-06-29T14:37:43.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b8bd2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41b40b4ee0 0x7f41b415f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:43.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b8bd2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f41b40d2780 0x7f41b40cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:43.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b8bd2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b415dff0 0x7f41b4162c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:43.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b8bd2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f41b4169320 con 0x7f41b415dff0 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b8bd2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41b41691a0 con 0x7f41b40b4ee0 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b8bd2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f41b41694a0 con 0x7f41b40d2780 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b415dff0 0x7f41b4162c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b415dff0 0x7f41b4162c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51336/0 (socket says 172.21.15.59:51336) 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 -- 172.21.15.59:0/823155196 learned_addr learned my addr 172.21.15.59:0/823155196 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b259c640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41b40b4ee0 0x7f41b415f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b1d9b640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f41b40d2780 0x7f41b40cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 -- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f41b40d2780 msgr2=0x7f41b40cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f41b40d2780 0x7f41b40cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 -- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41b40b4ee0 msgr2=0x7f41b415f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41b40b4ee0 0x7f41b415f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 -- 172.21.15.59:0/823155196 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b4163150 con 0x7f41b415dff0 2024-06-29T14:37:43.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b1d9b640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f41b40d2780 0x7f41b40cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:43.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b259c640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41b40b4ee0 0x7f41b415f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:43.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.464+0000 7f41b2d9d640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b415dff0 0x7f41b4162c10 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009cd0 tx=0x7f41ac003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:43.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.465+0000 7f41a2ffd640 1 -- 172.21.15.59:0/823155196 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac031b90 con 0x7f41b415dff0 2024-06-29T14:37:43.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.465+0000 7f41a2ffd640 1 -- 172.21.15.59:0/823155196 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41ac031e20 con 0x7f41b415dff0 2024-06-29T14:37:43.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.465+0000 7f41a2ffd640 1 -- 172.21.15.59:0/823155196 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac033310 con 0x7f41b415dff0 2024-06-29T14:37:43.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.465+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b4141c50 con 0x7f41b415dff0 2024-06-29T14:37:43.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.465+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b4142430 con 0x7f41b415dff0 2024-06-29T14:37:43.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.466+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b4161ae0 con 0x7f41b415dff0 2024-06-29T14:37:43.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.466+0000 7f41a2ffd640 1 -- 172.21.15.59:0/823155196 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f41ac031660 con 0x7f41b415dff0 2024-06-29T14:37:43.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.467+0000 7f41a2ffd640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f41940763f0 0x7f4194078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:43.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.467+0000 7f41a2ffd640 1 -- 172.21.15.59:0/823155196 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6437+0+0 (secure 0 0 0) 0x7f41ac0b9390 con 0x7f41b415dff0 2024-06-29T14:37:43.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.467+0000 7f41b259c640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f41940763f0 0x7f4194078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:43.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.468+0000 7f41b259c640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f41940763f0 0x7f4194078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f41a4006fd0 tx=0x7f41a4008040 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:43.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.471+0000 7f41a2ffd640 1 -- 172.21.15.59:0/823155196 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f41ac03f030 con 0x7f41b415dff0 2024-06-29T14:37:43.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:43.663+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f41b4141ee0 con 0x7f41b415dff0 2024-06-29T14:37:44.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.317+0000 7f41a2ffd640 1 -- 172.21.15.59:0/823155196 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) ==== 169+0+0 (secure 0 0 0) 0x7f41ac082760 con 0x7f41b415dff0 2024-06-29T14:37:44.315 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T14:37:44.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.324+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f41940763f0 msgr2=0x7f4194078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.324+0000 7f41b8bd2640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f41940763f0 0x7f4194078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f41a4006fd0 tx=0x7f41a4008040 comp rx=0 tx=0).stop 2024-06-29T14:37:44.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.324+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b415dff0 msgr2=0x7f41b4162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.324+0000 7f41b8bd2640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b415dff0 0x7f41b4162c10 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009cd0 tx=0x7f41ac003ba0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.325+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 shutdown_connections 2024-06-29T14:37:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.325+0000 7f41b8bd2640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41b415dff0 0x7f41b4162c10 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.325+0000 7f41b8bd2640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f41940763f0 0x7f4194078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.325+0000 7f41b8bd2640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f41b40d2780 0x7f41b40cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.325+0000 7f41b8bd2640 1 --2- 172.21.15.59:0/823155196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41b40b4ee0 0x7f41b415f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.325+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 >> 172.21.15.59:0/823155196 conn(0x7f41b40d3e60 msgr2=0x7f41b40d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.326+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 shutdown_connections 2024-06-29T14:37:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.326+0000 7f41b8bd2640 1 -- 172.21.15.59:0/823155196 wait complete. 2024-06-29T14:37:44.340 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:37:44.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.468+0000 7f0df06d6640 1 -- 172.21.15.59:0/3434805701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de81458a0 msgr2=0x7f0de80cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.468+0000 7f0df06d6640 1 --2- 172.21.15.59:0/3434805701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de81458a0 0x7f0de80cf870 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f0de80b1c40 tx=0x7f0dd802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.468+0000 7f0df06d6640 1 -- 172.21.15.59:0/3434805701 shutdown_connections 2024-06-29T14:37:44.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.468+0000 7f0df06d6640 1 --2- 172.21.15.59:0/3434805701 >> v2:172.21.15.153:3300/0 conn(0x7f0de80b3740 0x7f0de80b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.468+0000 7f0df06d6640 1 --2- 172.21.15.59:0/3434805701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de81458a0 0x7f0de80cf870 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.468+0000 7f0df06d6640 1 -- 172.21.15.59:0/3434805701 >> 172.21.15.59:0/3434805701 conn(0x7f0de806c950 msgr2=0x7f0de8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:44.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.468+0000 7f0df06d6640 1 -- 172.21.15.59:0/3434805701 shutdown_connections 2024-06-29T14:37:44.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.468+0000 7f0df06d6640 1 -- 172.21.15.59:0/3434805701 wait complete. 2024-06-29T14:37:44.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.470+0000 7f0df06d6640 1 Processor -- start 2024-06-29T14:37:44.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.470+0000 7f0df06d6640 1 -- start start 2024-06-29T14:37:44.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0df06d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de80b3740 0x7f0de813abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0df06d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0de815c2f0 0x7f0de815b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0df06d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de81389b0 0x7f0de8138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0df06d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0de8160020 con 0x7f0de80b3740 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0df06d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0de815cbb0 con 0x7f0de81389b0 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0df06d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0de81601a0 con 0x7f0de815c2f0 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0dee472640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de80b3740 0x7f0de813abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0dee472640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de80b3740 0x7f0de813abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51350/0 (socket says 172.21.15.59:51350) 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0dee472640 1 -- 172.21.15.59:0/511681742 learned_addr learned my addr 172.21.15.59:0/511681742 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0deec73640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de81389b0 0x7f0de8138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0dee472640 1 -- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0de815c2f0 msgr2=0x7f0de815b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0dee472640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0de815c2f0 0x7f0de815b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0dee472640 1 -- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de81389b0 msgr2=0x7f0de8138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0dee472640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de81389b0 0x7f0de8138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.471+0000 7f0dee472640 1 -- 172.21.15.59:0/511681742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de8139300 con 0x7f0de80b3740 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.472+0000 7f0dee472640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de80b3740 0x7f0de813abc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8005fd0 tx=0x7f0dd8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.472+0000 7f0de6ffd640 1 -- 172.21.15.59:0/511681742 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8031b90 con 0x7f0de80b3740 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.472+0000 7f0de6ffd640 1 -- 172.21.15.59:0/511681742 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dd8031e20 con 0x7f0de80b3740 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.472+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de8140a80 con 0x7f0de80b3740 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.472+0000 7f0de6ffd640 1 -- 172.21.15.59:0/511681742 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80332f0 con 0x7f0de80b3740 2024-06-29T14:37:44.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.472+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de81e5880 con 0x7f0de80b3740 2024-06-29T14:37:44.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.473+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de81606c0 con 0x7f0de80b3740 2024-06-29T14:37:44.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.474+0000 7f0de6ffd640 1 -- 172.21.15.59:0/511681742 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f0dd8033490 con 0x7f0de80b3740 2024-06-29T14:37:44.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.474+0000 7f0de6ffd640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f0dd40763f0 0x7f0dd4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.474+0000 7f0de6ffd640 1 -- 172.21.15.59:0/511681742 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7f0dd80b9010 con 0x7f0de80b3740 2024-06-29T14:37:44.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.474+0000 7f0dedc71640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f0dd40763f0 0x7f0dd4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.475+0000 7f0dedc71640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f0dd40763f0 0x7f0dd4078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0058c0 tx=0x7f0ddc005850 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.479+0000 7f0de6ffd640 1 -- 172.21.15.59:0/511681742 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0de8139300 con 0x7f0de80b3740 2024-06-29T14:37:44.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.671+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f0de80b1f10 con 0x7f0de80b3740 2024-06-29T14:37:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.675+0000 7f0de6ffd640 1 -- 172.21.15.59:0/511681742 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v39) ==== 71+0+0 (secure 0 0 0) 0x7f0dd803f030 con 0x7f0de80b3740 2024-06-29T14:37:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f0dd40763f0 msgr2=0x7f0dd4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f0dd40763f0 0x7f0dd4078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0058c0 tx=0x7f0ddc005850 comp rx=0 tx=0).stop 2024-06-29T14:37:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de80b3740 msgr2=0x7f0de813abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de80b3740 0x7f0de813abc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8005fd0 tx=0x7f0dd8003970 comp rx=0 tx=0).stop 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 shutdown_connections 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de81389b0 0x7f0de8138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f0dd40763f0 0x7f0dd4078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0de815c2f0 0x7f0de815b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 --2- 172.21.15.59:0/511681742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0de80b3740 0x7f0de813abc0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 >> 172.21.15.59:0/511681742 conn(0x7f0de806c950 msgr2=0x7f0de80cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 shutdown_connections 2024-06-29T14:37:44.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.678+0000 7f0df06d6640 1 -- 172.21.15.59:0/511681742 wait complete. 2024-06-29T14:37:44.686 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-06-29T14:37:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.807+0000 7fc8679c7640 1 -- 172.21.15.59:0/1357738546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 msgr2=0x7fc8600d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.807+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1357738546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 0x7fc8600d1520 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc854009830 tx=0x7fc85402a510 comp rx=0 tx=0).stop 2024-06-29T14:37:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.807+0000 7fc8679c7640 1 -- 172.21.15.59:0/1357738546 shutdown_connections 2024-06-29T14:37:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.807+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1357738546 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600d1a60 0x7fc8600b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.807+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1357738546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 0x7fc8600d1520 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.807+0000 7fc8679c7640 1 -- 172.21.15.59:0/1357738546 >> 172.21.15.59:0/1357738546 conn(0x7fc860073e10 msgr2=0x7fc860074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.807+0000 7fc8679c7640 1 -- 172.21.15.59:0/1357738546 shutdown_connections 2024-06-29T14:37:44.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.808+0000 7fc8679c7640 1 -- 172.21.15.59:0/1357738546 wait complete. 2024-06-29T14:37:44.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.809+0000 7fc8679c7640 1 Processor -- start 2024-06-29T14:37:44.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.809+0000 7fc8679c7640 1 -- start start 2024-06-29T14:37:44.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc8679c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2270 0x7fc8600ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc8679c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 0x7fc86014a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc8679c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600d1a60 0x7fc86006e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc8679c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600b6400 con 0x7fc8600b2fa0 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc8679c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600b6280 con 0x7fc8600d1a60 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc8679c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc8600b6580 con 0x7fc8600b2270 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 0x7fc86014a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 0x7fc86014a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51372/0 (socket says 172.21.15.59:51372) 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 -- 172.21.15.59:0/1993877226 learned_addr learned my addr 172.21.15.59:0/1993877226 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc865f64640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600d1a60 0x7fc86006e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc865763640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2270 0x7fc8600ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 -- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2270 msgr2=0x7fc8600ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2270 0x7fc8600ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 -- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600d1a60 msgr2=0x7fc86006e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600d1a60 0x7fc86006e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 -- 172.21.15.59:0/1993877226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86014ac60 con 0x7fc8600b2fa0 2024-06-29T14:37:44.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.811+0000 7fc864f62640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 0x7fc86014a230 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc854000c00 tx=0x7fc854033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:44.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.812+0000 7fc85dffb640 1 -- 172.21.15.59:0/1993877226 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc854005c90 con 0x7fc8600b2fa0 2024-06-29T14:37:44.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.812+0000 7fc85dffb640 1 -- 172.21.15.59:0/1993877226 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc854031e50 con 0x7fc8600b2fa0 2024-06-29T14:37:44.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.812+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8600b0730 con 0x7fc8600b2fa0 2024-06-29T14:37:44.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.812+0000 7fc85dffb640 1 -- 172.21.15.59:0/1993877226 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc854033310 con 0x7fc8600b2fa0 2024-06-29T14:37:44.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.812+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc86006df00 con 0x7fc8600b2fa0 2024-06-29T14:37:44.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.813+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8600b6aa0 con 0x7fc8600b2fa0 2024-06-29T14:37:44.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.813+0000 7fc85dffb640 1 -- 172.21.15.59:0/1993877226 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fc8540334b0 con 0x7fc8600b2fa0 2024-06-29T14:37:44.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.814+0000 7fc85dffb640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fc84c0763f0 0x7fc84c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.814+0000 7fc85dffb640 1 -- 172.21.15.59:0/1993877226 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7fc8540b8fb0 con 0x7fc8600b2fa0 2024-06-29T14:37:44.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.814+0000 7fc865763640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fc84c0763f0 0x7fc84c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.815+0000 7fc865763640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fc84c0763f0 0x7fc84c078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc8600b1fc0 tx=0x7fc858009040 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:44.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:44.818+0000 7fc85dffb640 1 -- 172.21.15.59:0/1993877226 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc85403f030 con 0x7fc8600b2fa0 2024-06-29T14:37:45.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.091+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) -- 0x7fc86015bd50 con 0x7fc8600b2fa0 2024-06-29T14:37:45.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.318+0000 7fc85dffb640 1 -- 172.21.15.59:0/1993877226 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v597) ==== 156+0+0 (secure 0 0 0) 0x7fc8540825b0 con 0x7fc8600b2fa0 2024-06-29T14:37:45.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fc84c0763f0 msgr2=0x7fc84c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fc84c0763f0 0x7fc84c078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc8600b1fc0 tx=0x7fc858009040 comp rx=0 tx=0).stop 2024-06-29T14:37:45.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 msgr2=0x7fc86014a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 0x7fc86014a230 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc854000c00 tx=0x7fc854033c90 comp rx=0 tx=0).stop 2024-06-29T14:37:45.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 shutdown_connections 2024-06-29T14:37:45.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600d1a60 0x7fc86006e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8600b2fa0 0x7fc86014a230 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fc84c0763f0 0x7fc84c078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 --2- 172.21.15.59:0/1993877226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2270 0x7fc8600ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 >> 172.21.15.59:0/1993877226 conn(0x7fc860073e10 msgr2=0x7fc8600d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 shutdown_connections 2024-06-29T14:37:45.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.321+0000 7fc8679c7640 1 -- 172.21.15.59:0/1993877226 wait complete. 2024-06-29T14:37:45.330 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-06-29T14:37:45.330 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-06-29T14:37:45.330 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:37:45.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:45.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.452+0000 7f2c2d81f640 1 -- 172.21.15.59:0/963054051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c14003aa0 msgr2=0x7f2c14003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.452+0000 7f2c2d81f640 1 --2- 172.21.15.59:0/963054051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c14003aa0 0x7f2c14003f10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f2c20001660 tx=0x7f2c14005830 comp rx=0 tx=0).stop 2024-06-29T14:37:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.452+0000 7f2c277fe640 1 -- 172.21.15.59:0/963054051 reap_dead start 2024-06-29T14:37:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.452+0000 7f2c2d81f640 1 -- 172.21.15.59:0/963054051 shutdown_connections 2024-06-29T14:37:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.452+0000 7f2c2d81f640 1 -- 172.21.15.59:0/963054051 >> 172.21.15.59:0/963054051 conn(0x7f2c280b24c0 msgr2=0x7f2c280cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.452+0000 7f2c2d81f640 1 -- 172.21.15.59:0/963054051 shutdown_connections 2024-06-29T14:37:45.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.452+0000 7f2c2d81f640 1 -- 172.21.15.59:0/963054051 wait complete. 2024-06-29T14:37:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.454+0000 7f2c2d81f640 1 Processor -- start 2024-06-29T14:37:45.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.454+0000 7f2c2d81f640 1 -- start start 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c2d81f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c14003aa0 0x7f2c2813e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c2d81f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c2813f6c0 0x7f2c2813fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c2d81f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c28154440 0x7f2c28154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c2d81f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2c280ce6e0 con 0x7f2c28154440 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c2d81f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c280cd450 con 0x7f2c2813f6c0 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c2d81f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2c280ce860 con 0x7f2c14003aa0 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c28154440 0x7f2c28154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c28154440 0x7f2c28154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51390/0 (socket says 172.21.15.59:51390) 2024-06-29T14:37:45.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.455+0000 7f2c277fe640 1 -- 172.21.15.59:0/2117153284 learned_addr learned my addr 172.21.15.59:0/2117153284 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c26ffd640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c14003aa0 0x7f2c2813e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c267fc640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c2813f6c0 0x7f2c2813fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c277fe640 1 -- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c14003aa0 msgr2=0x7f2c2813e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c277fe640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c14003aa0 0x7f2c2813e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c277fe640 1 -- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c2813f6c0 msgr2=0x7f2c2813fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c277fe640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c2813f6c0 0x7f2c2813fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c277fe640 1 -- 172.21.15.59:0/2117153284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c28154d90 con 0x7f2c28154440 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c277fe640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c28154440 0x7f2c28154850 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2c280b51d0 tx=0x7f2c18006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c1ffff640 1 -- 172.21.15.59:0/2117153284 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1800fa20 con 0x7f2c28154440 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c1ffff640 1 -- 172.21.15.59:0/2117153284 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c18018070 con 0x7f2c28154440 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c1ffff640 1 -- 172.21.15.59:0/2117153284 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1800fe60 con 0x7f2c28154440 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c2813e8a0 con 0x7f2c28154440 2024-06-29T14:37:45.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.456+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c281563f0 con 0x7f2c28154440 2024-06-29T14:37:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.457+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c28155e60 con 0x7f2c28154440 2024-06-29T14:37:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.458+0000 7f2c1ffff640 1 -- 172.21.15.59:0/2117153284 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f2c1801d460 con 0x7f2c28154440 2024-06-29T14:37:45.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.459+0000 7f2c1ffff640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2c080763f0 0x7f2c08078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.459+0000 7f2c1ffff640 1 -- 172.21.15.59:0/2117153284 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7f2c18098220 con 0x7f2c28154440 2024-06-29T14:37:45.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.459+0000 7f2c26ffd640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2c080763f0 0x7f2c08078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.460+0000 7f2c26ffd640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2c080763f0 0x7f2c08078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2c20032000 tx=0x7f2c2002c040 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.463+0000 7f2c1ffff640 1 -- 172.21.15.59:0/2117153284 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2c28154d90 con 0x7f2c28154440 2024-06-29T14:37:45.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.655+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c28156d80 con 0x7f2c28154440 2024-06-29T14:37:45.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.656+0000 7f2c1ffff640 1 -- 172.21.15.59:0/2117153284 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+14323 (secure 0 0 0) 0x7f2c280cd610 con 0x7f2c28154440 2024-06-29T14:37:45.654 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:45.654 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":93,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:44.312588+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6837/3891057790":"2024-06-30T14:37:41.302437+0000","172.21.15.153:6835/3891057790":"2024-06-30T14:37:41.302437+0000","172.21.15.153:0/131993077":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/3045910470":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/2009989783":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/277169477":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/3061329079":"2024-06-30T14:37:38.892471+0000","172.21.15.153:6832/3110229799":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/1132863181":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/901558684":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6832/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6833/4269929333":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2816930410":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/1030990875":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/2052586834":"2024-06-30T14:37:32.999861+0000","172.21.15.153:6833/3110229799":"2024-06-30T14:37:38.892471+0000","172.21.15.153:0/62694730":"2024-06-30T14:37:32.999861+0000","172.21.15.153:0/2198109495":"2024-06-30T14:37:32.999861+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2c080763f0 msgr2=0x7f2c08078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2c080763f0 0x7f2c08078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2c20032000 tx=0x7f2c2002c040 comp rx=0 tx=0).stop 2024-06-29T14:37:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c28154440 msgr2=0x7f2c28154850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c28154440 0x7f2c28154850 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2c280b51d0 tx=0x7f2c18006660 comp rx=0 tx=0).stop 2024-06-29T14:37:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 shutdown_connections 2024-06-29T14:37:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c28154440 0x7f2c28154850 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c2813f6c0 0x7f2c2813fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2c080763f0 0x7f2c08078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 --2- 172.21.15.59:0/2117153284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c14003aa0 0x7f2c2813e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.658+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 >> 172.21.15.59:0/2117153284 conn(0x7f2c280b24c0 msgr2=0x7f2c2815fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.659+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 shutdown_connections 2024-06-29T14:37:45.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.659+0000 7f2c2d81f640 1 -- 172.21.15.59:0/2117153284 wait complete. 2024-06-29T14:37:45.667 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:45.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:45.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.792+0000 7f1b6e822640 1 -- 172.21.15.59:0/2089507002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 msgr2=0x7f1b681622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.792+0000 7f1b6e822640 1 --2- 172.21.15.59:0/2089507002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 0x7f1b681622c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c005fd0 tx=0x7f1b5c02a4f0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.793+0000 7f1b6e822640 1 -- 172.21.15.59:0/2089507002 shutdown_connections 2024-06-29T14:37:45.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.793+0000 7f1b6e822640 1 --2- 172.21.15.59:0/2089507002 >> v2:172.21.15.153:3300/0 conn(0x7f1b68162800 0x7f1b68162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.793+0000 7f1b6e822640 1 --2- 172.21.15.59:0/2089507002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 0x7f1b681622c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.793+0000 7f1b6e822640 1 -- 172.21.15.59:0/2089507002 >> 172.21.15.59:0/2089507002 conn(0x7f1b680b5610 msgr2=0x7f1b680b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.793+0000 7f1b6e822640 1 -- 172.21.15.59:0/2089507002 shutdown_connections 2024-06-29T14:37:45.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.793+0000 7f1b6e822640 1 -- 172.21.15.59:0/2089507002 wait complete. 2024-06-29T14:37:45.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.794+0000 7f1b6e822640 1 Processor -- start 2024-06-29T14:37:45.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.795+0000 7f1b6e822640 1 -- start start 2024-06-29T14:37:45.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b6e822640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b6808c190 0x7f1b680b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b6e822640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 0x7f1b6814d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b6e822640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b68162800 0x7f1b681f8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b6e822640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1b680b7f60 con 0x7f1b680ce950 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b6e822640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1b680b7de0 con 0x7f1b68162800 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b6e822640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1b680b80e0 con 0x7f1b6808c190 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 0x7f1b6814d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 0x7f1b6814d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51420/0 (socket says 172.21.15.59:51420) 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b677fe640 1 -- 172.21.15.59:0/447045907 learned_addr learned my addr 172.21.15.59:0/447045907 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b67fff640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b6808c190 0x7f1b680b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b6cdbf640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b68162800 0x7f1b681f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b677fe640 1 -- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b6808c190 msgr2=0x7f1b680b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b677fe640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b6808c190 0x7f1b680b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b677fe640 1 -- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b68162800 msgr2=0x7f1b681f8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b677fe640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b68162800 0x7f1b681f8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.796+0000 7f1b677fe640 1 -- 172.21.15.59:0/447045907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b680cc1c0 con 0x7f1b680ce950 2024-06-29T14:37:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.797+0000 7f1b677fe640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 0x7f1b6814d480 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c034040 tx=0x7f1b5c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.797+0000 7f1b64ff9640 1 -- 172.21.15.59:0/447045907 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c03e030 con 0x7f1b680ce950 2024-06-29T14:37:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.797+0000 7f1b64ff9640 1 -- 172.21.15.59:0/447045907 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b5c0047a0 con 0x7f1b680ce950 2024-06-29T14:37:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.797+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b680cc4a0 con 0x7f1b680ce950 2024-06-29T14:37:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.797+0000 7f1b64ff9640 1 -- 172.21.15.59:0/447045907 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c035040 con 0x7f1b680ce950 2024-06-29T14:37:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.797+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b680cca60 con 0x7f1b680ce950 2024-06-29T14:37:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.798+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b681622c0 con 0x7f1b680ce950 2024-06-29T14:37:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.798+0000 7f1b64ff9640 1 -- 172.21.15.59:0/447045907 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f1b5c041030 con 0x7f1b680ce950 2024-06-29T14:37:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.799+0000 7f1b64ff9640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1b540763f0 0x7f1b54078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.799+0000 7f1b64ff9640 1 -- 172.21.15.59:0/447045907 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7f1b5c0b81b0 con 0x7f1b680ce950 2024-06-29T14:37:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.800+0000 7f1b67fff640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1b540763f0 0x7f1b54078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.800+0000 7f1b67fff640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1b540763f0 0x7f1b54078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1b58006fd0 tx=0x7f1b58009060 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.804+0000 7f1b64ff9640 1 -- 172.21.15.59:0/447045907 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1b5c0811f0 con 0x7f1b680ce950 2024-06-29T14:37:45.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.992+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b680cf220 con 0x7f1b680ce950 2024-06-29T14:37:45.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.993+0000 7f1b64ff9640 1 -- 172.21.15.59:0/447045907 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f1b5c0854f0 con 0x7f1b680ce950 2024-06-29T14:37:45.990 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:37:45.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1b540763f0 msgr2=0x7f1b54078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1b540763f0 0x7f1b54078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1b58006fd0 tx=0x7f1b58009060 comp rx=0 tx=0).stop 2024-06-29T14:37:45.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 msgr2=0x7f1b6814d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 0x7f1b6814d480 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c034040 tx=0x7f1b5c003970 comp rx=0 tx=0).stop 2024-06-29T14:37:45.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 shutdown_connections 2024-06-29T14:37:45.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b68162800 0x7f1b681f8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b680ce950 0x7f1b6814d480 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f1b540763f0 0x7f1b54078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 --2- 172.21.15.59:0/447045907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b6808c190 0x7f1b680b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 >> 172.21.15.59:0/447045907 conn(0x7f1b680b5610 msgr2=0x7f1b680b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 shutdown_connections 2024-06-29T14:37:45.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:45.995+0000 7f1b6e822640 1 -- 172.21.15.59:0/447045907 wait complete. 2024-06-29T14:37:46.003 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-06-29T14:37:46.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.125+0000 7fe41b76f640 1 -- 172.21.15.59:0/2401557901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4140b3740 msgr2=0x7fe4140b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:46.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.125+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2401557901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4140b3740 0x7fe4140b3b50 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe414074b10 tx=0x7fe40402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:37:46.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.125+0000 7fe41b76f640 1 -- 172.21.15.59:0/2401557901 shutdown_connections 2024-06-29T14:37:46.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.125+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2401557901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4140b3740 0x7fe4140b3b50 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:46.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.125+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2401557901 >> v2:172.21.15.153:3300/0 conn(0x7fe4141458a0 0x7fe4140cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:46.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.125+0000 7fe41b76f640 1 -- 172.21.15.59:0/2401557901 >> 172.21.15.59:0/2401557901 conn(0x7fe41406c950 msgr2=0x7fe414073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:46.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.125+0000 7fe41b76f640 1 -- 172.21.15.59:0/2401557901 shutdown_connections 2024-06-29T14:37:46.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.125+0000 7fe41b76f640 1 -- 172.21.15.59:0/2401557901 wait complete. 2024-06-29T14:37:46.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.126+0000 7fe41b76f640 1 Processor -- start 2024-06-29T14:37:46.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.127+0000 7fe41b76f640 1 -- start start 2024-06-29T14:37:46.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe41b76f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4140b3740 0x7fe4140afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:46.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe41b76f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe41415c2f0 0x7fe41406db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:46.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe41b76f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4140c6990 0x7fe4140ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:46.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe41b76f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe414160020 con 0x7fe41415c2f0 2024-06-29T14:37:46.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe41b76f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe41415cbb0 con 0x7fe4140c6990 2024-06-29T14:37:46.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe41b76f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe4141601a0 con 0x7fe4140b3740 2024-06-29T14:37:46.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe41415c2f0 0x7fe41406db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:46.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe41415c2f0 0x7fe41406db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51438/0 (socket says 172.21.15.59:51438) 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 -- 172.21.15.59:0/2241805571 learned_addr learned my addr 172.21.15.59:0/2241805571 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe419d0c640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4140c6990 0x7fe4140ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe41950b640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4140b3740 0x7fe4140afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 -- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4140b3740 msgr2=0x7fe4140afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4140b3740 0x7fe4140afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 -- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4140c6990 msgr2=0x7fe4140ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4140c6990 0x7fe4140ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 -- 172.21.15.59:0/2241805571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4140b1520 con 0x7fe41415c2f0 2024-06-29T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.128+0000 7fe418d0a640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe41415c2f0 0x7fe41406db80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe404034040 tx=0x7fe4040039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:46.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.129+0000 7fe409ffb640 1 -- 172.21.15.59:0/2241805571 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40403e030 con 0x7fe41415c2f0 2024-06-29T14:37:46.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.129+0000 7fe409ffb640 1 -- 172.21.15.59:0/2241805571 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4040047a0 con 0x7fe41415c2f0 2024-06-29T14:37:46.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.129+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4140aea60 con 0x7fe41415c2f0 2024-06-29T14:37:46.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.129+0000 7fe409ffb640 1 -- 172.21.15.59:0/2241805571 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe404035040 con 0x7fe41415c2f0 2024-06-29T14:37:46.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.129+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4140af000 con 0x7fe41415c2f0 2024-06-29T14:37:46.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.130+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4141606c0 con 0x7fe41415c2f0 2024-06-29T14:37:46.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.131+0000 7fe409ffb640 1 -- 172.21.15.59:0/2241805571 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fe404041030 con 0x7fe41415c2f0 2024-06-29T14:37:46.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.131+0000 7fe409ffb640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe3f80763f0 0x7fe3f8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:46.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.131+0000 7fe409ffb640 1 -- 172.21.15.59:0/2241805571 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7fe4040b84d0 con 0x7fe41415c2f0 2024-06-29T14:37:46.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.131+0000 7fe41950b640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe3f80763f0 0x7fe3f8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:46.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.132+0000 7fe41950b640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe3f80763f0 0x7fe3f8078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4100058c0 tx=0x7fe41000c440 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:46.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.136+0000 7fe409ffb640 1 -- 172.21.15.59:0/2241805571 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe404081a40 con 0x7fe41415c2f0 2024-06-29T14:37:46.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:46.402+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) -- 0x7fe41406fec0 con 0x7fe41415c2f0 2024-06-29T14:37:47.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.325+0000 7fe409ffb640 1 -- 172.21.15.59:0/2241805571 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v599) ==== 169+0+0 (secure 0 0 0) 0x7fe404081340 con 0x7fe41415c2f0 2024-06-29T14:37:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe3f80763f0 msgr2=0x7fe3f8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe3f80763f0 0x7fe3f8078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4100058c0 tx=0x7fe41000c440 comp rx=0 tx=0).stop 2024-06-29T14:37:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe41415c2f0 msgr2=0x7fe41406db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe41415c2f0 0x7fe41406db80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe404034040 tx=0x7fe4040039e0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 shutdown_connections 2024-06-29T14:37:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4140c6990 0x7fe4140ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe41415c2f0 0x7fe41406db80 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe3f80763f0 0x7fe3f8078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 --2- 172.21.15.59:0/2241805571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4140b3740 0x7fe4140afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.329+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 >> 172.21.15.59:0/2241805571 conn(0x7fe41406c950 msgr2=0x7fe414070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.330+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 shutdown_connections 2024-06-29T14:37:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.330+0000 7fe41b76f640 1 -- 172.21.15.59:0/2241805571 wait complete. 2024-06-29T14:37:47.340 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:37:47.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.468+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2005841235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea4004800 msgr2=0x7f2ea4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.468+0000 7f2ebac8f640 1 --2- 172.21.15.59:0/2005841235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea4004800 0x7f2ea4004c10 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0001710 tx=0x7f2ea4005830 comp rx=0 tx=0).stop 2024-06-29T14:37:47.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.468+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2005841235 shutdown_connections 2024-06-29T14:37:47.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.468+0000 7f2ebac8f640 1 --2- 172.21.15.59:0/2005841235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ea4004800 0x7f2ea4004c10 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.468+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2005841235 >> 172.21.15.59:0/2005841235 conn(0x7f2eb40b24c0 msgr2=0x7f2eb40cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.468+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2005841235 shutdown_connections 2024-06-29T14:37:47.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.468+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2005841235 wait complete. 2024-06-29T14:37:47.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.470+0000 7f2ebac8f640 1 Processor -- start 2024-06-29T14:37:47.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.470+0000 7f2ebac8f640 1 -- start start 2024-06-29T14:37:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.471+0000 7f2ebac8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ea4004800 0x7f2eb415b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2ebac8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eb413f730 0x7f2eb41f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2ebac8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2eb41f6490 0x7f2eb413c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2ebac8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb406c360 con 0x7f2eb41f6490 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2ebac8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb406c1e0 con 0x7f2eb413f730 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2ebac8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb406c4e0 con 0x7f2ea4004800 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2eb41f6490 0x7f2eb413c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2eb41f6490 0x7f2eb413c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51460/0 (socket says 172.21.15.59:51460) 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 -- 172.21.15.59:0/2813586966 learned_addr learned my addr 172.21.15.59:0/2813586966 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb3fff640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eb413f730 0x7f2eb41f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb8a2b640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ea4004800 0x7f2eb415b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 -- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ea4004800 msgr2=0x7f2eb415b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ea4004800 0x7f2eb415b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 -- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eb413f730 msgr2=0x7f2eb41f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eb413f730 0x7f2eb41f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 -- 172.21.15.59:0/2813586966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb41f68a0 con 0x7f2eb41f6490 2024-06-29T14:37:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb3fff640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eb413f730 0x7f2eb41f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb922c640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2eb41f6490 0x7f2eb413c910 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4160980 tx=0x7f2eac0064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb17fa640 1 -- 172.21.15.59:0/2813586966 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac011900 con 0x7f2eb41f6490 2024-06-29T14:37:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb17fa640 1 -- 172.21.15.59:0/2813586966 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2eac018070 con 0x7f2eb41f6490 2024-06-29T14:37:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb41401f0 con 0x7f2eb41f6490 2024-06-29T14:37:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2eb17fa640 1 -- 172.21.15.59:0/2813586966 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac011d80 con 0x7f2eb41f6490 2024-06-29T14:37:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.472+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb41407b0 con 0x7f2eb41f6490 2024-06-29T14:37:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.474+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb40cd120 con 0x7f2eb41f6490 2024-06-29T14:37:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.474+0000 7f2eb17fa640 1 -- 172.21.15.59:0/2813586966 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f2eac01d460 con 0x7f2eb41f6490 2024-06-29T14:37:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.475+0000 7f2eb17fa640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2e98076420 0x7f2e98078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.475+0000 7f2eb17fa640 1 -- 172.21.15.59:0/2813586966 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7f2eac098260 con 0x7f2eb41f6490 2024-06-29T14:37:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.475+0000 7f2eb8a2b640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2e98076420 0x7f2e98078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.476+0000 7f2eb8a2b640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2e98076420 0x7f2e98078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4051e20 tx=0x7f2ea8005f90 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:47.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.479+0000 7f2eb17fa640 1 -- 172.21.15.59:0/2813586966 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2eb41f68a0 con 0x7f2eb41f6490 2024-06-29T14:37:47.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.670+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f2eb40afa50 con 0x7f2eb41f6490 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.670+0000 7f2eb17fa640 1 -- 172.21.15.59:0/2813586966 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v39) ==== 59+0+1998 (secure 0 0 0) 0x7f2eac0617c0 con 0x7f2eb41f6490 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T14:37:47.668 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:37:47.669 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T14:37:47.670 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T14:37:47.670 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T14:37:47.670 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:37:47.670 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2e98076420 msgr2=0x7f2e98078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2e98076420 0x7f2e98078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4051e20 tx=0x7f2ea8005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2eb41f6490 msgr2=0x7f2eb413c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2eb41f6490 0x7f2eb413c910 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4160980 tx=0x7f2eac0064f0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 shutdown_connections 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2eb41f6490 0x7f2eb413c910 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eb413f730 0x7f2eb41f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f2e98076420 0x7f2e98078930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 --2- 172.21.15.59:0/2813586966 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ea4004800 0x7f2eb415b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 >> 172.21.15.59:0/2813586966 conn(0x7f2eb40b24c0 msgr2=0x7f2eb415d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 shutdown_connections 2024-06-29T14:37:47.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.673+0000 7f2ebac8f640 1 -- 172.21.15.59:0/2813586966 wait complete. 2024-06-29T14:37:47.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:37:47.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.805+0000 7f5150d7b640 1 -- 172.21.15.59:0/4163108937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f514c162770 msgr2=0x7f514c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.805+0000 7f5150d7b640 1 --2- 172.21.15.59:0/4163108937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f514c162770 0x7f514c0ce550 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f513c005fd0 tx=0x7f513c02aa10 comp rx=0 tx=0).stop 2024-06-29T14:37:47.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.805+0000 7f5150d7b640 1 -- 172.21.15.59:0/4163108937 shutdown_connections 2024-06-29T14:37:47.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.805+0000 7f5150d7b640 1 --2- 172.21.15.59:0/4163108937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f514c162190 0x7f514c0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.805+0000 7f5150d7b640 1 --2- 172.21.15.59:0/4163108937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f514c162770 0x7f514c0ce550 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.805+0000 7f5150d7b640 1 -- 172.21.15.59:0/4163108937 >> 172.21.15.59:0/4163108937 conn(0x7f514c0d45b0 msgr2=0x7f514c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.805+0000 7f5150d7b640 1 -- 172.21.15.59:0/4163108937 shutdown_connections 2024-06-29T14:37:47.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.805+0000 7f5150d7b640 1 -- 172.21.15.59:0/4163108937 wait complete. 2024-06-29T14:37:47.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.807+0000 7f5150d7b640 1 Processor -- start 2024-06-29T14:37:47.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.807+0000 7f5150d7b640 1 -- start start 2024-06-29T14:37:47.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5150d7b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f514c15e0a0 0x7f514c0cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5150d7b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f514c162770 0x7f514c1f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5150d7b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f514c0c7db0 0x7f514c1f8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5150d7b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f514c0b42c0 con 0x7f514c162770 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5150d7b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f514c0b4140 con 0x7f514c15e0a0 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5150d7b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f514c0b4440 con 0x7f514c0c7db0 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5149d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f514c162770 0x7f514c1f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5149d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f514c162770 0x7f514c1f8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51470/0 (socket says 172.21.15.59:51470) 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.808+0000 7f5149d9b640 1 -- 172.21.15.59:0/1401203010 learned_addr learned my addr 172.21.15.59:0/1401203010 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f514a59c640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f514c15e0a0 0x7f514c0cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f514ad9d640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f514c0c7db0 0x7f514c1f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5149d9b640 1 -- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f514c0c7db0 msgr2=0x7f514c1f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5149d9b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f514c0c7db0 0x7f514c1f8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5149d9b640 1 -- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f514c15e0a0 msgr2=0x7f514c0cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5149d9b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f514c15e0a0 0x7f514c0cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5149d9b640 1 -- 172.21.15.59:0/1401203010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514c0c8560 con 0x7f514c162770 2024-06-29T14:37:47.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5149d9b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f514c162770 0x7f514c1f8120 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f513400c3a0 tx=0x7f513400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:47.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5142ffd640 1 -- 172.21.15.59:0/1401203010 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134017070 con 0x7f514c162770 2024-06-29T14:37:47.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5142ffd640 1 -- 172.21.15.59:0/1401203010 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51340053c0 con 0x7f514c162770 2024-06-29T14:37:47.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514c0c8840 con 0x7f514c162770 2024-06-29T14:37:47.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5142ffd640 1 -- 172.21.15.59:0/1401203010 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134003850 con 0x7f514c162770 2024-06-29T14:37:47.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.809+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f514c1f5e90 con 0x7f514c162770 2024-06-29T14:37:47.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.810+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f514c0c8d60 con 0x7f514c162770 2024-06-29T14:37:47.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.811+0000 7f5142ffd640 1 -- 172.21.15.59:0/1401203010 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f513401a070 con 0x7f514c162770 2024-06-29T14:37:47.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.812+0000 7f5142ffd640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f512c0763f0 0x7f512c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.812+0000 7f5142ffd640 1 -- 172.21.15.59:0/1401203010 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7f5134096da0 con 0x7f514c162770 2024-06-29T14:37:47.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.812+0000 7f514a59c640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f512c0763f0 0x7f512c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.812+0000 7f514a59c640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f512c0763f0 0x7f512c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f513c035000 tx=0x7f513c004ab0 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:47.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:47.816+0000 7f5142ffd640 1 -- 172.21.15.59:0/1401203010 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f514c0c8d60 con 0x7f514c162770 2024-06-29T14:37:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.006+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f514c0b3a00 con 0x7f514c162770 2024-06-29T14:37:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.010+0000 7f5142ffd640 1 -- 172.21.15.59:0/1401203010 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7f514c0b3a00 con 0x7f514c162770 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f512c0763f0 msgr2=0x7f512c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f512c0763f0 0x7f512c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f513c035000 tx=0x7f513c004ab0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f514c162770 msgr2=0x7f514c1f8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f514c162770 0x7f514c1f8120 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f513400c3a0 tx=0x7f513400c860 comp rx=0 tx=0).stop 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 shutdown_connections 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f512c0763f0 0x7f512c078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f514c0c7db0 0x7f514c1f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f514c162770 0x7f514c1f8120 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 --2- 172.21.15.59:0/1401203010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f514c15e0a0 0x7f514c0cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 >> 172.21.15.59:0/1401203010 conn(0x7f514c0d45b0 msgr2=0x7f514c15be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 shutdown_connections 2024-06-29T14:37:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.012+0000 7f5150d7b640 1 -- 172.21.15.59:0/1401203010 wait complete. 2024-06-29T14:37:48.020 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:37:48.021 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:48.057 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:48.102 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:37:48.102 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:48.102 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:48.119 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:48.120 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:37:48.120 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:48.138 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:48.138 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:37:48.139 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:48.175 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:48.176 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:37:48.176 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:48.196 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:48.196 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:37:48.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 -- 172.21.15.59:0/1433076774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640d1a60 msgr2=0x7fb7640cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 --2- 172.21.15.59:0/1433076774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640d1a60 0x7fb7640cf780 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb764074c50 tx=0x7fb75c02aa30 comp rx=0 tx=0).stop 2024-06-29T14:37:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 -- 172.21.15.59:0/1433076774 shutdown_connections 2024-06-29T14:37:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 --2- 172.21.15.59:0/1433076774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640d1a60 0x7fb7640cf780 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 --2- 172.21.15.59:0/1433076774 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7640b2fa0 0x7fb7640d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 --2- 172.21.15.59:0/1433076774 >> v2:172.21.15.153:3300/0 conn(0x7fb7640b2270 0x7fb76415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 -- 172.21.15.59:0/1433076774 >> 172.21.15.59:0/1433076774 conn(0x7fb764073e10 msgr2=0x7fb764074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 -- 172.21.15.59:0/1433076774 shutdown_connections 2024-06-29T14:37:48.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.336+0000 7fb76a03c640 1 -- 172.21.15.59:0/1433076774 wait complete. 2024-06-29T14:37:48.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.338+0000 7fb76a03c640 1 Processor -- start 2024-06-29T14:37:48.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.338+0000 7fb76a03c640 1 -- start start 2024-06-29T14:37:48.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.339+0000 7fb76a03c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640b2270 0x7fb76414a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.339+0000 7fb76a03c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7640b2fa0 0x7fb76415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.339+0000 7fb76a03c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7640d1a60 0x7fb76415e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.339+0000 7fb76a03c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb7640cfe40 con 0x7fb7640b2270 2024-06-29T14:37:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.339+0000 7fb76a03c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7640cfcc0 con 0x7fb7640d1a60 2024-06-29T14:37:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.339+0000 7fb76a03c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb7640cffc0 con 0x7fb7640b2fa0 2024-06-29T14:37:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640b2270 0x7fb76414a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640b2270 0x7fb76414a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51490/0 (socket says 172.21.15.59:51490) 2024-06-29T14:37:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 -- 172.21.15.59:0/556738494 learned_addr learned my addr 172.21.15.59:0/556738494 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:48.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb762ffd640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7640b2fa0 0x7fb76415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb763fff640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7640d1a60 0x7fb76415e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 -- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7640b2fa0 msgr2=0x7fb76415d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7640b2fa0 0x7fb76415d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 -- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7640d1a60 msgr2=0x7fb76415e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7640d1a60 0x7fb76415e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 -- 172.21.15.59:0/556738494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7641f28d0 con 0x7fb7640b2270 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb7637fe640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640b2270 0x7fb76414a100 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb75400d7a0 tx=0x7fb75400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb74bfff640 1 -- 172.21.15.59:0/556738494 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb754017070 con 0x7fb7640b2270 2024-06-29T14:37:48.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb74bfff640 1 -- 172.21.15.59:0/556738494 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb754009d20 con 0x7fb7640b2270 2024-06-29T14:37:48.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7640a9540 con 0x7fb7640b2270 2024-06-29T14:37:48.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.340+0000 7fb74bfff640 1 -- 172.21.15.59:0/556738494 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb754004280 con 0x7fb7640b2270 2024-06-29T14:37:48.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.341+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7640a9b00 con 0x7fb7640b2270 2024-06-29T14:37:48.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.342+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7640d0450 con 0x7fb7640b2270 2024-06-29T14:37:48.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.342+0000 7fb74bfff640 1 -- 172.21.15.59:0/556738494 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fb754005020 con 0x7fb7640b2270 2024-06-29T14:37:48.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.343+0000 7fb74bfff640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb750076420 0x7fb750078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.343+0000 7fb74bfff640 1 -- 172.21.15.59:0/556738494 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7fb754097d20 con 0x7fb7640b2270 2024-06-29T14:37:48.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.343+0000 7fb762ffd640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb750076420 0x7fb750078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.344+0000 7fb762ffd640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb750076420 0x7fb750078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb764157b20 tx=0x7fb75800b390 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:48.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.347+0000 7fb74bfff640 1 -- 172.21.15.59:0/556738494 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb754020020 con 0x7fb7640b2270 2024-06-29T14:37:48.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.575+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb764070ed0 con 0x7fb7640b2270 2024-06-29T14:37:48.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.576+0000 7fb74bfff640 1 -- 172.21.15.59:0/556738494 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+3917 (secure 0 0 0) 0x7fb754069310 con 0x7fb7640b2270 2024-06-29T14:37:48.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:48.573 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":40,"btime":"2024-06-29T14:37:42:303929+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6529,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3719851385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3719851385},{"type":"v1","addr":"172.21.15.153:6836","nonce":3719851385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6855,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1280649067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1280649067},{"type":"v1","addr":"172.21.15.59:6833","nonce":1280649067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":6858,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2847930906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2847930906},{"type":"v1","addr":"172.21.15.59:6835","nonce":2847930906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6871,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2478716140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2478716140},{"type":"v1","addr":"172.21.15.153:6837","nonce":2478716140}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[]} 2024-06-29T14:37:48.573 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 40 2024-06-29T14:37:48.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb750076420 msgr2=0x7fb750078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb750076420 0x7fb750078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb764157b20 tx=0x7fb75800b390 comp rx=0 tx=0).stop 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640b2270 msgr2=0x7fb76414a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640b2270 0x7fb76414a100 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb75400d7a0 tx=0x7fb75400dc60 comp rx=0 tx=0).stop 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 shutdown_connections 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7640d1a60 0x7fb76415e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb750076420 0x7fb750078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7640b2fa0 0x7fb76415d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 --2- 172.21.15.59:0/556738494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7640b2270 0x7fb76414a100 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 >> 172.21.15.59:0/556738494 conn(0x7fb764073e10 msgr2=0x7fb76406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 shutdown_connections 2024-06-29T14:37:48.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.578+0000 7fb76a03c640 1 -- 172.21.15.59:0/556738494 wait complete. 2024-06-29T14:37:48.587 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:37:48.587 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:37:48.587 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:37:48.587 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:48.588 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:37:48.588 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:37:48.588 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:37:48.588 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:48.588 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:37:48.588 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:37:48.588 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:37:48.589 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:48.589 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:37:48.589 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:37:48.589 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:37:48.589 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:48.597 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:37:48.597 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:37:48.598 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:37:48.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:37:48.601 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:37:48.601 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:37:48.603 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:37:48.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:37:48.639 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:37:48.640 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:48.640 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:37:48.640 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:48.645 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:37:48.645 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:48.647 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:37:48.647 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:48.647 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:37:48.730 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:37:48.730 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:37:48.741 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:37:48.741 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:37:48.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.775+0000 7fa602ab3640 1 -- 172.21.15.59:0/260005095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5fc0ce650 msgr2=0x7fa5fc0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.775+0000 7fa602ab3640 1 --2- 172.21.15.59:0/260005095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5fc0ce650 0x7fa5fc0b3990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009830 tx=0x7fa5e802a500 comp rx=0 tx=0).stop 2024-06-29T14:37:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.775+0000 7fa602ab3640 1 -- 172.21.15.59:0/260005095 shutdown_connections 2024-06-29T14:37:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.775+0000 7fa602ab3640 1 --2- 172.21.15.59:0/260005095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5fc0ce650 0x7fa5fc0b3990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.775+0000 7fa602ab3640 1 --2- 172.21.15.59:0/260005095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5fc15e290 0x7fa5fc0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.775+0000 7fa602ab3640 1 -- 172.21.15.59:0/260005095 >> 172.21.15.59:0/260005095 conn(0x7fa5fc0d3ee0 msgr2=0x7fa5fc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.775+0000 7fa602ab3640 1 -- 172.21.15.59:0/260005095 shutdown_connections 2024-06-29T14:37:48.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.775+0000 7fa602ab3640 1 -- 172.21.15.59:0/260005095 wait complete. 2024-06-29T14:37:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.777+0000 7fa602ab3640 1 Processor -- start 2024-06-29T14:37:48.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.777+0000 7fa602ab3640 1 -- start start 2024-06-29T14:37:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.778+0000 7fa602ab3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0b3ed0 0x7fa5fc149d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.778+0000 7fa602ab3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5fc0ce650 0x7fa5fc160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.778+0000 7fa602ab3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5fc15e290 0x7fa5fc159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.778+0000 7fa602ab3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc0d1f20 con 0x7fa5fc0b3ed0 2024-06-29T14:37:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.778+0000 7fa602ab3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc0d1da0 con 0x7fa5fc15e290 2024-06-29T14:37:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.778+0000 7fa602ab3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa5fc0d20a0 con 0x7fa5fc0ce650 2024-06-29T14:37:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.778+0000 7fa60084f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0b3ed0 0x7fa5fc149d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa60084f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0b3ed0 0x7fa5fc149d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51546/0 (socket says 172.21.15.59:51546) 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa60084f640 1 -- 172.21.15.59:0/237735327 learned_addr learned my addr 172.21.15.59:0/237735327 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5fbfff640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5fc0ce650 0x7fa5fc160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa601050640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5fc15e290 0x7fa5fc159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5fbfff640 1 -- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5fc15e290 msgr2=0x7fa5fc159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5fbfff640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5fc15e290 0x7fa5fc159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5fbfff640 1 -- 172.21.15.59:0/237735327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0b3ed0 msgr2=0x7fa5fc149d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5fbfff640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0b3ed0 0x7fa5fc149d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5fbfff640 1 -- 172.21.15.59:0/237735327 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5fc154e80 con 0x7fa5fc0ce650 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa60084f640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0b3ed0 0x7fa5fc149d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5fbfff640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5fc0ce650 0x7fa5fc160530 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0d1620 tx=0x7fa5e8033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5f97fa640 1 -- 172.21.15.59:0/237735327 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8005c90 con 0x7fa5fc0ce650 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5f97fa640 1 -- 172.21.15.59:0/237735327 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5e8031e50 con 0x7fa5fc0ce650 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa5f97fa640 1 -- 172.21.15.59:0/237735327 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8005280 con 0x7fa5fc0ce650 2024-06-29T14:37:48.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5fc155160 con 0x7fa5fc0ce650 2024-06-29T14:37:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.779+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5fc155720 con 0x7fa5fc0ce650 2024-06-29T14:37:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.780+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5fc155b00 con 0x7fa5fc0ce650 2024-06-29T14:37:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.781+0000 7fa5f97fa640 1 -- 172.21.15.59:0/237735327 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fa5e8005420 con 0x7fa5fc0ce650 2024-06-29T14:37:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.782+0000 7fa5f97fa640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fa5e00763f0 0x7fa5e0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.782+0000 7fa5f97fa640 1 -- 172.21.15.59:0/237735327 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(93..93 src has 1..93) ==== 6023+0+0 (secure 0 0 0) 0x7fa5e80b8ae0 con 0x7fa5fc0ce650 2024-06-29T14:37:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.782+0000 7fa60084f640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fa5e00763f0 0x7fa5e0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.782+0000 7fa60084f640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fa5e00763f0 0x7fa5e0078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00096c0 tx=0x7fa5f0009260 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:48.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.786+0000 7fa5f97fa640 1 -- 172.21.15.59:0/237735327 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa5fc155b00 con 0x7fa5fc0ce650 2024-06-29T14:37:48.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:48.992+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa5fc069ca0 con 0x7fa5fc0ce650 2024-06-29T14:37:49.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.347+0000 7fa5f97fa640 1 -- 172.21.15.59:0/237735327 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v95) ==== 97+0+0 (secure 0 0 0) 0x7fa5fc069ca0 con 0x7fa5fc0ce650 2024-06-29T14:37:49.345 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T14:37:49.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fa5e00763f0 msgr2=0x7fa5e0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fa5e00763f0 0x7fa5e0078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00096c0 tx=0x7fa5f0009260 comp rx=0 tx=0).stop 2024-06-29T14:37:49.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5fc0ce650 msgr2=0x7fa5fc160530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5fc0ce650 0x7fa5fc160530 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0d1620 tx=0x7fa5e8033c20 comp rx=0 tx=0).stop 2024-06-29T14:37:49.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 shutdown_connections 2024-06-29T14:37:49.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5fc15e290 0x7fa5fc159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fa5e00763f0 0x7fa5e0078900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa5fc0ce650 0x7fa5fc160530 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 --2- 172.21.15.59:0/237735327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0b3ed0 0x7fa5fc149d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 >> 172.21.15.59:0/237735327 conn(0x7fa5fc0d3ee0 msgr2=0x7fa5fc159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 shutdown_connections 2024-06-29T14:37:49.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.350+0000 7fa602ab3640 1 -- 172.21.15.59:0/237735327 wait complete. 2024-06-29T14:37:49.361 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:37:49.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.482+0000 7fb52e85e640 1 -- 172.21.15.59:0/2848808428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280d23c0 msgr2=0x7fb5280b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.482+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2848808428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280d23c0 0x7fb5280b3990 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb518005fd0 tx=0x7fb51802a530 comp rx=0 tx=0).stop 2024-06-29T14:37:49.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.482+0000 7fb52e85e640 1 -- 172.21.15.59:0/2848808428 shutdown_connections 2024-06-29T14:37:49.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.482+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2848808428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5280b3ed0 0x7fb52815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.482+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2848808428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280d23c0 0x7fb5280b3990 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.482+0000 7fb52e85e640 1 -- 172.21.15.59:0/2848808428 >> 172.21.15.59:0/2848808428 conn(0x7fb5280d3ee0 msgr2=0x7fb5280d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.482+0000 7fb52e85e640 1 -- 172.21.15.59:0/2848808428 shutdown_connections 2024-06-29T14:37:49.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.482+0000 7fb52e85e640 1 -- 172.21.15.59:0/2848808428 wait complete. 2024-06-29T14:37:49.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.484+0000 7fb52e85e640 1 Processor -- start 2024-06-29T14:37:49.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.485+0000 7fb52e85e640 1 -- start start 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb52e85e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280b3ed0 0x7fb5280c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb52e85e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5280d23c0 0x7fb52815d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb52e85e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52815ee10 0x7fb528157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb52e85e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb528160240 con 0x7fb5280b3ed0 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb52e85e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5281600c0 con 0x7fb52815ee10 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb52e85e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb5281603c0 con 0x7fb5280d23c0 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb527fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280b3ed0 0x7fb5280c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb527fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280b3ed0 0x7fb5280c7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51564/0 (socket says 172.21.15.59:51564) 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb527fff640 1 -- 172.21.15.59:0/2241118771 learned_addr learned my addr 172.21.15.59:0/2241118771 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:49.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb5277fe640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5280d23c0 0x7fb52815d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb52cdfb640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52815ee10 0x7fb528157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb527fff640 1 -- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5280d23c0 msgr2=0x7fb52815d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb527fff640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5280d23c0 0x7fb52815d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb527fff640 1 -- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52815ee10 msgr2=0x7fb528157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb527fff640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52815ee10 0x7fb528157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb527fff640 1 -- 172.21.15.59:0/2241118771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5280c8140 con 0x7fb5280b3ed0 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.486+0000 7fb5277fe640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5280d23c0 0x7fb52815d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.487+0000 7fb527fff640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280b3ed0 0x7fb5280c7c00 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fb518034040 tx=0x7fb5180038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.487+0000 7fb524ff9640 1 -- 172.21.15.59:0/2241118771 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51803e030 con 0x7fb5280b3ed0 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.487+0000 7fb524ff9640 1 -- 172.21.15.59:0/2241118771 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb518003d90 con 0x7fb5280b3ed0 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.487+0000 7fb524ff9640 1 -- 172.21.15.59:0/2241118771 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518035040 con 0x7fb5280b3ed0 2024-06-29T14:37:49.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.487+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52815c650 con 0x7fb5280b3ed0 2024-06-29T14:37:49.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.487+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52815cbd0 con 0x7fb5280b3ed0 2024-06-29T14:37:49.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.488+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb52815e8d0 con 0x7fb5280b3ed0 2024-06-29T14:37:49.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.488+0000 7fb524ff9640 1 -- 172.21.15.59:0/2241118771 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fb518041030 con 0x7fb5280b3ed0 2024-06-29T14:37:49.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.489+0000 7fb524ff9640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb5140763f0 0x7fb514078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.489+0000 7fb524ff9640 1 -- 172.21.15.59:0/2241118771 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5249+0+0 (secure 0 0 0) 0x7fb5180b8070 con 0x7fb5280b3ed0 2024-06-29T14:37:49.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.489+0000 7fb5277fe640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb5140763f0 0x7fb514078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.490+0000 7fb5277fe640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb5140763f0 0x7fb514078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5280c7140 tx=0x7fb51c005dd0 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.494+0000 7fb524ff9640 1 -- 172.21.15.59:0/2241118771 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb5280b42e0 con 0x7fb5280b3ed0 2024-06-29T14:37:49.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.756+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb5280d2950 con 0x7fb5280b3ed0 2024-06-29T14:37:49.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.757+0000 7fb524ff9640 1 -- 172.21.15.59:0/2241118771 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7fb5180818e0 con 0x7fb5280b3ed0 2024-06-29T14:37:49.754 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:49.755 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T14:37:49.757 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.758 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T14:37:49.759 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.760 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T14:37:49.761 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.762 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.762 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.762 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.762 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.762 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.762 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.762 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.763 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:37:49.764 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T14:37:49.765 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:37:49.766 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T14:37:49.767 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:37:49.768 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:37:49.769 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.769 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:49.769 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T14:37:49.769 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:37:49.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb5140763f0 msgr2=0x7fb514078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb5140763f0 0x7fb514078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb5280c7140 tx=0x7fb51c005dd0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280b3ed0 msgr2=0x7fb5280c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280b3ed0 0x7fb5280c7c00 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fb518034040 tx=0x7fb5180038a0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 shutdown_connections 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52815ee10 0x7fb528157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb5140763f0 0x7fb514078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5280d23c0 0x7fb52815d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 --2- 172.21.15.59:0/2241118771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5280b3ed0 0x7fb5280c7c00 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 >> 172.21.15.59:0/2241118771 conn(0x7fb5280d3ee0 msgr2=0x7fb52806c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 shutdown_connections 2024-06-29T14:37:49.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.759+0000 7fb52e85e640 1 -- 172.21.15.59:0/2241118771 wait complete. 2024-06-29T14:37:49.772 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:49.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 -- 172.21.15.59:0/1679647103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc13e9b0 msgr2=0x7f26cc13edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 --2- 172.21.15.59:0/1679647103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc13e9b0 0x7f26cc13edc0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f26cc073f30 tx=0x7f26b402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 -- 172.21.15.59:0/1679647103 shutdown_connections 2024-06-29T14:37:49.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 --2- 172.21.15.59:0/1679647103 >> v2:172.21.15.153:3300/0 conn(0x7f26cc1414c0 0x7f26cc1418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 --2- 172.21.15.59:0/1679647103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc13e9b0 0x7f26cc13edc0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 --2- 172.21.15.59:0/1679647103 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26cc15ceb0 0x7f26cc0b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 -- 172.21.15.59:0/1679647103 >> 172.21.15.59:0/1679647103 conn(0x7f26cc0b6fb0 msgr2=0x7f26cc0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 -- 172.21.15.59:0/1679647103 shutdown_connections 2024-06-29T14:37:49.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.893+0000 7f26d18f6640 1 -- 172.21.15.59:0/1679647103 wait complete. 2024-06-29T14:37:49.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.895+0000 7f26d18f6640 1 Processor -- start 2024-06-29T14:37:49.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.895+0000 7f26d18f6640 1 -- start start 2024-06-29T14:37:49.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26d18f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26cc13e9b0 0x7f26cc070750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26d18f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26cc1414c0 0x7f26cc070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26d18f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc15ceb0 0x7f26cc0c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26d18f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc0b28c0 con 0x7f26cc15ceb0 2024-06-29T14:37:49.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26d18f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26cc0b2740 con 0x7f26cc13e9b0 2024-06-29T14:37:49.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26d18f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f26cc0b2a40 con 0x7f26cc1414c0 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc15ceb0 0x7f26cc0c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc15ceb0 0x7f26cc0c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51578/0 (socket says 172.21.15.59:51578) 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26cb7fe640 1 -- 172.21.15.59:0/4134031696 learned_addr learned my addr 172.21.15.59:0/4134031696 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26ca7fc640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26cc1414c0 0x7f26cc070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.896+0000 7f26caffd640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26cc13e9b0 0x7f26cc070750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26cb7fe640 1 -- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26cc1414c0 msgr2=0x7f26cc070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26cb7fe640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26cc1414c0 0x7f26cc070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26cb7fe640 1 -- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26cc13e9b0 msgr2=0x7f26cc070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26cb7fe640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26cc13e9b0 0x7f26cc070750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26cb7fe640 1 -- 172.21.15.59:0/4134031696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26cc13c4e0 con 0x7f26cc15ceb0 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26cb7fe640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc15ceb0 0x7f26cc0c9e10 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f26c400a6f0 tx=0x7f26c400abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26c3fff640 1 -- 172.21.15.59:0/4134031696 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c40079b0 con 0x7f26cc15ceb0 2024-06-29T14:37:49.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26c3fff640 1 -- 172.21.15.59:0/4134031696 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26c4007c40 con 0x7f26cc15ceb0 2024-06-29T14:37:49.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26cc0cdba0 con 0x7f26cc15ceb0 2024-06-29T14:37:49.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26c3fff640 1 -- 172.21.15.59:0/4134031696 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c4005280 con 0x7f26cc15ceb0 2024-06-29T14:37:49.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.897+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26cc12e6c0 con 0x7f26cc15ceb0 2024-06-29T14:37:49.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.898+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26cc0cd910 con 0x7f26cc15ceb0 2024-06-29T14:37:49.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.899+0000 7f26c3fff640 1 -- 172.21.15.59:0/4134031696 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f26c400c050 con 0x7f26cc15ceb0 2024-06-29T14:37:49.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.899+0000 7f26c3fff640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f26ac0763f0 0x7f26ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.899+0000 7f26c3fff640 1 -- 172.21.15.59:0/4134031696 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5249+0+0 (secure 0 0 0) 0x7f26c4097ab0 con 0x7f26cc15ceb0 2024-06-29T14:37:49.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.900+0000 7f26caffd640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f26ac0763f0 0x7f26ac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.900+0000 7f26caffd640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f26ac0763f0 0x7f26ac078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f26bc005c40 tx=0x7f26bc00a350 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:49.904+0000 7f26c3fff640 1 -- 172.21.15.59:0/4134031696 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f26c4013030 con 0x7f26cc15ceb0 2024-06-29T14:37:50.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.099+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26cc0b2f60 con 0x7f26cc15ceb0 2024-06-29T14:37:50.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.099+0000 7f26c3fff640 1 -- 172.21.15.59:0/4134031696 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+13177 (secure 0 0 0) 0x7f26c40613b0 con 0x7f26cc15ceb0 2024-06-29T14:37:50.097 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:50.097 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:49.354201+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:50.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f26ac0763f0 msgr2=0x7f26ac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f26ac0763f0 0x7f26ac078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f26bc005c40 tx=0x7f26bc00a350 comp rx=0 tx=0).stop 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc15ceb0 msgr2=0x7f26cc0c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc15ceb0 0x7f26cc0c9e10 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f26c400a6f0 tx=0x7f26c400abb0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 shutdown_connections 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26cc15ceb0 0x7f26cc0c9e10 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f26ac0763f0 0x7f26ac078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26cc1414c0 0x7f26cc070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 --2- 172.21.15.59:0/4134031696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26cc13e9b0 0x7f26cc070750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.101+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 >> 172.21.15.59:0/4134031696 conn(0x7f26cc0b6fb0 msgr2=0x7f26cc13e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.102+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 shutdown_connections 2024-06-29T14:37:50.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.102+0000 7f26d18f6640 1 -- 172.21.15.59:0/4134031696 wait complete. 2024-06-29T14:37:50.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:50.111 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:50.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.235+0000 7fb90785b640 1 -- 172.21.15.59:0/3409102049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8e8003ac0 msgr2=0x7fb8e8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.235+0000 7fb90785b640 1 --2- 172.21.15.59:0/3409102049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8e8003ac0 0x7fb8e8003f30 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0006fd0 tx=0x7fb8e8005830 comp rx=0 tx=0).stop 2024-06-29T14:37:50.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.235+0000 7fb905df8640 1 -- 172.21.15.59:0/3409102049 reap_dead start 2024-06-29T14:37:50.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.235+0000 7fb90785b640 1 -- 172.21.15.59:0/3409102049 shutdown_connections 2024-06-29T14:37:50.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.235+0000 7fb90785b640 1 -- 172.21.15.59:0/3409102049 >> 172.21.15.59:0/3409102049 conn(0x7fb9000d3ee0 msgr2=0x7fb9000d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.235+0000 7fb90785b640 1 -- 172.21.15.59:0/3409102049 shutdown_connections 2024-06-29T14:37:50.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.235+0000 7fb90785b640 1 -- 172.21.15.59:0/3409102049 wait complete. 2024-06-29T14:37:50.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.237+0000 7fb90785b640 1 Processor -- start 2024-06-29T14:37:50.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.237+0000 7fb90785b640 1 -- start start 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb90785b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8e8003ac0 0x7fb9001541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb90785b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9001f7d40 0x7fb900159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb90785b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9001f5f40 0x7fb900156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb90785b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb90015fd40 con 0x7fb9001f5f40 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb90785b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb90015fbc0 con 0x7fb9001f7d40 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb90785b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb90015fec0 con 0x7fb8e8003ac0 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb905df8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9001f5f40 0x7fb900156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb905df8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9001f5f40 0x7fb900156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51590/0 (socket says 172.21.15.59:51590) 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb905df8640 1 -- 172.21.15.59:0/1758218763 learned_addr learned my addr 172.21.15.59:0/1758218763 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb9055f7640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8e8003ac0 0x7fb9001541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.238+0000 7fb904df6640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9001f7d40 0x7fb900159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb905df8640 1 -- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8e8003ac0 msgr2=0x7fb9001541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb905df8640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8e8003ac0 0x7fb9001541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb905df8640 1 -- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9001f7d40 msgr2=0x7fb900159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb905df8640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9001f7d40 0x7fb900159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb905df8640 1 -- 172.21.15.59:0/1758218763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb900155180 con 0x7fb9001f5f40 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb905df8640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9001f5f40 0x7fb900156220 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fb9000b5540 tx=0x7fb8f8011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb8fdffb640 1 -- 172.21.15.59:0/1758218763 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8004230 con 0x7fb9001f5f40 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb8fdffb640 1 -- 172.21.15.59:0/1758218763 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8f8016070 con 0x7fb9001f5f40 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb900155460 con 0x7fb9001f5f40 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb8fdffb640 1 -- 172.21.15.59:0/1758218763 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f800fa90 con 0x7fb9001f5f40 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.239+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb900155a20 con 0x7fb9001f5f40 2024-06-29T14:37:50.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.241+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb900160450 con 0x7fb9001f5f40 2024-06-29T14:37:50.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.241+0000 7fb8fdffb640 1 -- 172.21.15.59:0/1758218763 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fb8f800fc30 con 0x7fb9001f5f40 2024-06-29T14:37:50.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.241+0000 7fb8fdffb640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb8ec076420 0x7fb8ec078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.242+0000 7fb8fdffb640 1 -- 172.21.15.59:0/1758218763 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5249+0+0 (secure 0 0 0) 0x7fb8f8097fd0 con 0x7fb9001f5f40 2024-06-29T14:37:50.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.242+0000 7fb9055f7640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb8ec076420 0x7fb8ec078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.242+0000 7fb9055f7640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb8ec076420 0x7fb8ec078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0033000 tx=0x7fb8f002d070 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.246+0000 7fb8fdffb640 1 -- 172.21.15.59:0/1758218763 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb8f8061300 con 0x7fb9001f5f40 2024-06-29T14:37:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.439+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb9000ceca0 con 0x7fb9001f5f40 2024-06-29T14:37:50.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.439+0000 7fb8fdffb640 1 -- 172.21.15.59:0/1758218763 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7fb8f8065600 con 0x7fb9001f5f40 2024-06-29T14:37:50.437 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:37:50.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb8ec076420 msgr2=0x7fb8ec078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb8ec076420 0x7fb8ec078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0033000 tx=0x7fb8f002d070 comp rx=0 tx=0).stop 2024-06-29T14:37:50.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9001f5f40 msgr2=0x7fb900156220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9001f5f40 0x7fb900156220 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fb9000b5540 tx=0x7fb8f8011b10 comp rx=0 tx=0).stop 2024-06-29T14:37:50.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 shutdown_connections 2024-06-29T14:37:50.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9001f5f40 0x7fb900156220 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9001f7d40 0x7fb900159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fb8ec076420 0x7fb8ec078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 --2- 172.21.15.59:0/1758218763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8e8003ac0 0x7fb9001541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 >> 172.21.15.59:0/1758218763 conn(0x7fb9000d3ee0 msgr2=0x7fb90006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 shutdown_connections 2024-06-29T14:37:50.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.441+0000 7fb90785b640 1 -- 172.21.15.59:0/1758218763 wait complete. 2024-06-29T14:37:50.450 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:37:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.571+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/3438275000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc80b40a0 msgr2=0x7f4cc80b73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.571+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/3438275000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc80b40a0 0x7f4cc80b73f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0005fd0 tx=0x7f4cc002aa00 comp rx=0 tx=0).stop 2024-06-29T14:37:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.571+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/3438275000 shutdown_connections 2024-06-29T14:37:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.571+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/3438275000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc80b40a0 0x7f4cc80b73f0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.571+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/3438275000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cc815e290 0x7f4cc80ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.571+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/3438275000 >> 172.21.15.59:0/3438275000 conn(0x7f4cc80d3ee0 msgr2=0x7f4cc80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.571+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/3438275000 shutdown_connections 2024-06-29T14:37:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.571+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/3438275000 wait complete. 2024-06-29T14:37:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.573+0000 7f4ccd2fd640 1 Processor -- start 2024-06-29T14:37:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.573+0000 7f4ccd2fd640 1 -- start start 2024-06-29T14:37:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.574+0000 7f4ccd2fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc80ce650 0x7f4cc81f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4ccd2fd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc815e290 0x7f4cc8156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4ccd2fd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cc8158280 0x7f4cc8158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4ccd2fd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc80b7cc0 con 0x7f4cc815e290 2024-06-29T14:37:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4ccd2fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc80b7b40 con 0x7f4cc80ce650 2024-06-29T14:37:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4ccd2fd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc80b7e40 con 0x7f4cc8158280 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc815e290 0x7f4cc8156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc815e290 0x7f4cc8156820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51596/0 (socket says 172.21.15.59:51596) 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 -- 172.21.15.59:0/2464681125 learned_addr learned my addr 172.21.15.59:0/2464681125 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cc77fe640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cc8158280 0x7f4cc8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cc6ffd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc80ce650 0x7f4cc81f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 -- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cc8158280 msgr2=0x7f4cc8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cc8158280 0x7f4cc8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 -- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc80ce650 msgr2=0x7f4cc81f7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc80ce650 0x7f4cc81f7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 -- 172.21.15.59:0/2464681125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc8153c80 con 0x7f4cc815e290 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cc6ffd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc80ce650 0x7f4cc81f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:50.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cc77fe640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cc8158280 0x7f4cc8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbffff640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc815e290 0x7f4cc8156820 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000b3e0 tx=0x7f4cb000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbf7fe640 1 -- 172.21.15.59:0/2464681125 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0015020 con 0x7f4cc815e290 2024-06-29T14:37:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbf7fe640 1 -- 172.21.15.59:0/2464681125 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cb0011920 con 0x7f4cc815e290 2024-06-29T14:37:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc8153f60 con 0x7f4cc815e290 2024-06-29T14:37:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.575+0000 7f4cbf7fe640 1 -- 172.21.15.59:0/2464681125 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0011d70 con 0x7f4cc815e290 2024-06-29T14:37:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.576+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc8155ac0 con 0x7f4cc815e290 2024-06-29T14:37:50.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.577+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc80b8410 con 0x7f4cc815e290 2024-06-29T14:37:50.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.577+0000 7f4cbf7fe640 1 -- 172.21.15.59:0/2464681125 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f4cb0018020 con 0x7f4cc815e290 2024-06-29T14:37:50.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.578+0000 7f4cbf7fe640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f4ca80763f0 0x7f4ca8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.578+0000 7f4cbf7fe640 1 -- 172.21.15.59:0/2464681125 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5249+0+0 (secure 0 0 0) 0x7f4cb0097650 con 0x7f4cc815e290 2024-06-29T14:37:50.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.578+0000 7f4cc6ffd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f4ca80763f0 0x7f4ca8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.579+0000 7f4cc6ffd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f4ca80763f0 0x7f4ca8078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80058c0 tx=0x7f4cb8005c90 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.582+0000 7f4cbf7fe640 1 -- 172.21.15.59:0/2464681125 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4cb001f020 con 0x7f4cc815e290 2024-06-29T14:37:50.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.774+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f4cc815c450 con 0x7f4cc815e290 2024-06-29T14:37:50.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.775+0000 7f4cbf7fe640 1 -- 172.21.15.59:0/2464681125 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v97) ==== 84+0+65 (secure 0 0 0) 0x7f4cb00610f0 con 0x7f4cc815e290 2024-06-29T14:37:50.772 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:50.772 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:37:50.772 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:37:50.772 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:37:50.772 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:37:50.772 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:37:50.773 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:37:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f4ca80763f0 msgr2=0x7f4ca8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f4ca80763f0 0x7f4ca8078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80058c0 tx=0x7f4cb8005c90 comp rx=0 tx=0).stop 2024-06-29T14:37:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc815e290 msgr2=0x7f4cc8156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc815e290 0x7f4cc8156820 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000b3e0 tx=0x7f4cb000b8a0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 shutdown_connections 2024-06-29T14:37:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f4ca80763f0 0x7f4ca8078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4cc8158280 0x7f4cc8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cc815e290 0x7f4cc8156820 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 --2- 172.21.15.59:0/2464681125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc80ce650 0x7f4cc81f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 >> 172.21.15.59:0/2464681125 conn(0x7f4cc80d3ee0 msgr2=0x7f4cc80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 shutdown_connections 2024-06-29T14:37:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.777+0000 7f4ccd2fd640 1 -- 172.21.15.59:0/2464681125 wait complete. 2024-06-29T14:37:50.785 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:37:50.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:37:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.909+0000 7f96b7ece640 1 -- 172.21.15.59:0/469409836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 msgr2=0x7f96b01621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.909+0000 7f96b7ece640 1 --2- 172.21.15.59:0/469409836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 0x7f96b01621f0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f96b00b3bc0 tx=0x7f96a002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.909+0000 7f96b7ece640 1 -- 172.21.15.59:0/469409836 shutdown_connections 2024-06-29T14:37:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.909+0000 7f96b7ece640 1 --2- 172.21.15.59:0/469409836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 0x7f96b01621f0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.909+0000 7f96b7ece640 1 -- 172.21.15.59:0/469409836 >> 172.21.15.59:0/469409836 conn(0x7f96b00d3e60 msgr2=0x7f96b00d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.910+0000 7f96b7ece640 1 -- 172.21.15.59:0/469409836 shutdown_connections 2024-06-29T14:37:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.910+0000 7f96b7ece640 1 -- 172.21.15.59:0/469409836 wait complete. 2024-06-29T14:37:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.911+0000 7f96b7ece640 1 Processor -- start 2024-06-29T14:37:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.911+0000 7f96b7ece640 1 -- start start 2024-06-29T14:37:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b7ece640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b00b3ed0 0x7f96b01f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b7ece640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 0x7f96b00c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b7ece640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96b015dff0 0x7f96b014c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b7ece640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f96b00b4b70 con 0x7f96b00d2780 2024-06-29T14:37:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b7ece640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96b00b49f0 con 0x7f96b00b3ed0 2024-06-29T14:37:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b7ece640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f96b00b4cf0 con 0x7f96b015dff0 2024-06-29T14:37:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 0x7f96b00c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 0x7f96b00c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51618/0 (socket says 172.21.15.59:51618) 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 -- 172.21.15.59:0/3852205992 learned_addr learned my addr 172.21.15.59:0/3852205992 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5c6a640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b00b3ed0 0x7f96b01f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b646b640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96b015dff0 0x7f96b014c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 -- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96b015dff0 msgr2=0x7f96b014c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96b015dff0 0x7f96b014c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 -- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b00b3ed0 msgr2=0x7f96b01f54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b00b3ed0 0x7f96b01f54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 -- 172.21.15.59:0/3852205992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b015e640 con 0x7f96b00d2780 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96b5469640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 0x7f96b00c9650 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009920 tx=0x7f96a0033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.913+0000 7f96a67fc640 1 -- 172.21.15.59:0/3852205992 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0031e20 con 0x7f96b00d2780 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.914+0000 7f96a67fc640 1 -- 172.21.15.59:0/3852205992 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96a0005110 con 0x7f96b00d2780 2024-06-29T14:37:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.914+0000 7f96a67fc640 1 -- 172.21.15.59:0/3852205992 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0005560 con 0x7f96b00d2780 2024-06-29T14:37:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.914+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b015e900 con 0x7f96b00d2780 2024-06-29T14:37:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.914+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b015eec0 con 0x7f96b00d2780 2024-06-29T14:37:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.915+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b01f5ae0 con 0x7f96b00d2780 2024-06-29T14:37:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.915+0000 7f96a67fc640 1 -- 172.21.15.59:0/3852205992 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f96a00047a0 con 0x7f96b00d2780 2024-06-29T14:37:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.916+0000 7f96a67fc640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f96940763f0 0x7f9694078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.916+0000 7f96a67fc640 1 -- 172.21.15.59:0/3852205992 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5249+0+0 (secure 0 0 0) 0x7f96a00b89e0 con 0x7f96b00d2780 2024-06-29T14:37:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.916+0000 7f96b5c6a640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f96940763f0 0x7f9694078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.917+0000 7f96b5c6a640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f96940763f0 0x7f9694078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f96ac001720 tx=0x7f96ac009040 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:50.920+0000 7f96a67fc640 1 -- 172.21.15.59:0/3852205992 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f96a003f030 con 0x7f96b00d2780 2024-06-29T14:37:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.115+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f96b00b42e0 con 0x7f96b00d2780 2024-06-29T14:37:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.348+0000 7f96a67fc640 1 -- 172.21.15.59:0/3852205992 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v98) ==== 138+0+0 (secure 0 0 0) 0x7f96a008a2d0 con 0x7f96b00d2780 2024-06-29T14:37:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:37:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f96940763f0 msgr2=0x7f9694078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f96940763f0 0x7f9694078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f96ac001720 tx=0x7f96ac009040 comp rx=0 tx=0).stop 2024-06-29T14:37:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 msgr2=0x7f96b00c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 0x7f96b00c9650 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009920 tx=0x7f96a0033d60 comp rx=0 tx=0).stop 2024-06-29T14:37:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 shutdown_connections 2024-06-29T14:37:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f96940763f0 0x7f9694078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96b015dff0 0x7f96b014c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96b00d2780 0x7f96b00c9650 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 --2- 172.21.15.59:0/3852205992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b00b3ed0 0x7f96b01f54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 >> 172.21.15.59:0/3852205992 conn(0x7f96b00d3e60 msgr2=0x7f96b00b8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 shutdown_connections 2024-06-29T14:37:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.350+0000 7f96b7ece640 1 -- 172.21.15.59:0/3852205992 wait complete. 2024-06-29T14:37:51.362 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:37:51.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.489+0000 7f971dbee640 1 -- 172.21.15.59:0/1985509533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180b2c40 msgr2=0x7f97180b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.489+0000 7f971dbee640 1 --2- 172.21.15.59:0/1985509533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180b2c40 0x7f97180b4570 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f971808c7a0 tx=0x7f970c02a530 comp rx=0 tx=0).stop 2024-06-29T14:37:51.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.489+0000 7f971dbee640 1 -- 172.21.15.59:0/1985509533 shutdown_connections 2024-06-29T14:37:51.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.489+0000 7f971dbee640 1 --2- 172.21.15.59:0/1985509533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180b2c40 0x7f97180b4570 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.489+0000 7f971dbee640 1 --2- 172.21.15.59:0/1985509533 >> v2:172.21.15.153:3300/0 conn(0x7f97180d3260 0x7f971815e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.489+0000 7f971dbee640 1 -- 172.21.15.59:0/1985509533 >> 172.21.15.59:0/1985509533 conn(0x7f97180d04e0 msgr2=0x7f97180d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:51.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.489+0000 7f971dbee640 1 -- 172.21.15.59:0/1985509533 shutdown_connections 2024-06-29T14:37:51.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.489+0000 7f971dbee640 1 -- 172.21.15.59:0/1985509533 wait complete. 2024-06-29T14:37:51.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.491+0000 7f971dbee640 1 Processor -- start 2024-06-29T14:37:51.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.491+0000 7f971dbee640 1 -- start start 2024-06-29T14:37:51.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f971dbee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97180b2c40 0x7f9718146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:51.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f971dbee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97180b4ab0 0x7f97180cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f971dbee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180d3260 0x7f97181597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f971dbee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97180b81f0 con 0x7f97180d3260 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f971dbee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97180b8070 con 0x7f97180b2c40 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f971dbee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f97180d1d40 con 0x7f97180b4ab0 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180d3260 0x7f97181597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180d3260 0x7f97181597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51642/0 (socket says 172.21.15.59:51642) 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 -- 172.21.15.59:0/294117207 learned_addr learned my addr 172.21.15.59:0/294117207 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f97177fe640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97180b2c40 0x7f9718146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9716ffd640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97180b4ab0 0x7f97180cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 -- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97180b4ab0 msgr2=0x7f97180cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97180b4ab0 0x7f97180cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 -- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97180b2c40 msgr2=0x7f9718146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97180b2c40 0x7f9718146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 -- 172.21.15.59:0/294117207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97181e6cd0 con 0x7f97180d3260 2024-06-29T14:37:51.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.493+0000 7f9717fff640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180d3260 0x7f97181597b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f971000c760 tx=0x7f971000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:51.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.494+0000 7f96fffff640 1 -- 172.21.15.59:0/294117207 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710003da0 con 0x7f97180d3260 2024-06-29T14:37:51.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.494+0000 7f96fffff640 1 -- 172.21.15.59:0/294117207 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f971000ce10 con 0x7f97180d3260 2024-06-29T14:37:51.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.494+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97181e6fb0 con 0x7f97180d3260 2024-06-29T14:37:51.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.494+0000 7f96fffff640 1 -- 172.21.15.59:0/294117207 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97100052d0 con 0x7f97180d3260 2024-06-29T14:37:51.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.494+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97181e7570 con 0x7f97180d3260 2024-06-29T14:37:51.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.495+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97180d2260 con 0x7f97180d3260 2024-06-29T14:37:51.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.496+0000 7f96fffff640 1 -- 172.21.15.59:0/294117207 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f9710005470 con 0x7f97180d3260 2024-06-29T14:37:51.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.496+0000 7f96fffff640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f97040763f0 0x7f9704078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:51.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.496+0000 7f96fffff640 1 -- 172.21.15.59:0/294117207 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5619+0+0 (secure 0 0 0) 0x7f9710098040 con 0x7f97180d3260 2024-06-29T14:37:51.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.496+0000 7f97177fe640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f97040763f0 0x7f9704078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:51.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.497+0000 7f97177fe640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f97040763f0 0x7f9704078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f970800bb50 tx=0x7f970800b2e0 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:51.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.500+0000 7f96fffff640 1 -- 172.21.15.59:0/294117207 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f97180b4ec0 con 0x7f97180d3260 2024-06-29T14:37:51.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:51.693+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f97180cdf00 con 0x7f97180d3260 2024-06-29T14:37:52.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.350+0000 7f96fffff640 1 -- 172.21.15.59:0/294117207 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v99) ==== 170+0+0 (secure 0 0 0) 0x7f97100612e0 con 0x7f97180d3260 2024-06-29T14:37:52.348 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:37:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.353+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f97040763f0 msgr2=0x7f9704078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.353+0000 7f971dbee640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f97040763f0 0x7f9704078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f970800bb50 tx=0x7f970800b2e0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.353+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180d3260 msgr2=0x7f97181597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.353+0000 7f971dbee640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180d3260 0x7f97181597b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f971000c760 tx=0x7f971000cc20 comp rx=0 tx=0).stop 2024-06-29T14:37:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.354+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 shutdown_connections 2024-06-29T14:37:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.354+0000 7f971dbee640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97180d3260 0x7f97181597b0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.354+0000 7f971dbee640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f97040763f0 0x7f9704078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.354+0000 7f971dbee640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97180b4ab0 0x7f97180cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.354+0000 7f971dbee640 1 --2- 172.21.15.59:0/294117207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97180b2c40 0x7f9718146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.354+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 >> 172.21.15.59:0/294117207 conn(0x7f97180d04e0 msgr2=0x7f971815ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:52.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.354+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 shutdown_connections 2024-06-29T14:37:52.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.354+0000 7f971dbee640 1 -- 172.21.15.59:0/294117207 wait complete. 2024-06-29T14:37:52.380 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:37:52.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.510+0000 7fd65192a640 1 -- 172.21.15.59:0/1635954290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c162770 msgr2=0x7fd64c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.510+0000 7fd65192a640 1 --2- 172.21.15.59:0/1635954290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c162770 0x7fd64c0ce550 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd63c005fd0 tx=0x7fd63c02aa10 comp rx=0 tx=0).stop 2024-06-29T14:37:52.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.510+0000 7fd65192a640 1 -- 172.21.15.59:0/1635954290 shutdown_connections 2024-06-29T14:37:52.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.510+0000 7fd65192a640 1 --2- 172.21.15.59:0/1635954290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15e0a0 0x7fd64c161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.510+0000 7fd65192a640 1 --2- 172.21.15.59:0/1635954290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c162770 0x7fd64c0ce550 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.510+0000 7fd65192a640 1 -- 172.21.15.59:0/1635954290 >> 172.21.15.59:0/1635954290 conn(0x7fd64c0d45b0 msgr2=0x7fd64c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:52.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.510+0000 7fd65192a640 1 -- 172.21.15.59:0/1635954290 shutdown_connections 2024-06-29T14:37:52.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.510+0000 7fd65192a640 1 -- 172.21.15.59:0/1635954290 wait complete. 2024-06-29T14:37:52.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.512+0000 7fd65192a640 1 Processor -- start 2024-06-29T14:37:52.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.512+0000 7fd65192a640 1 -- start start 2024-06-29T14:37:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.513+0000 7fd65192a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15e0a0 0x7fd64c15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.513+0000 7fd65192a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c161fc0 0x7fd64c1f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.513+0000 7fd65192a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd64c162770 0x7fd64c1f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.513+0000 7fd65192a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c0d2d80 con 0x7fd64c161fc0 2024-06-29T14:37:52.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.513+0000 7fd65192a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c0b4b00 con 0x7fd64c162770 2024-06-29T14:37:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.513+0000 7fd65192a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd64c0d2f00 con 0x7fd64c15e0a0 2024-06-29T14:37:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c161fc0 0x7fd64c1f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c161fc0 0x7fd64c1f8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51648/0 (socket says 172.21.15.59:51648) 2024-06-29T14:37:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 -- 172.21.15.59:0/2271429946 learned_addr learned my addr 172.21.15.59:0/2271429946 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64affd640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15e0a0 0x7fd64c15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:52.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64b7fe640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd64c162770 0x7fd64c1f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:52.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 -- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15e0a0 msgr2=0x7fd64c15ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15e0a0 0x7fd64c15ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 -- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd64c162770 msgr2=0x7fd64c1f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd64c162770 0x7fd64c1f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 -- 172.21.15.59:0/2271429946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c0c9570 con 0x7fd64c161fc0 2024-06-29T14:37:52.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64b7fe640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd64c162770 0x7fd64c1f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:52.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd64a7fc640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c161fc0 0x7fd64c1f8280 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd63400d6a0 tx=0x7fd63400db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:52.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd643fff640 1 -- 172.21.15.59:0/2271429946 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634017070 con 0x7fd64c161fc0 2024-06-29T14:37:52.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd643fff640 1 -- 172.21.15.59:0/2271429946 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd634009b10 con 0x7fd64c161fc0 2024-06-29T14:37:52.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64c0c9850 con 0x7fd64c161fc0 2024-06-29T14:37:52.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd643fff640 1 -- 172.21.15.59:0/2271429946 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634010460 con 0x7fd64c161fc0 2024-06-29T14:37:52.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.514+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64c0c9e10 con 0x7fd64c161fc0 2024-06-29T14:37:52.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.516+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64c0d33d0 con 0x7fd64c161fc0 2024-06-29T14:37:52.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.516+0000 7fd643fff640 1 -- 172.21.15.59:0/2271429946 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fd63401a070 con 0x7fd64c161fc0 2024-06-29T14:37:52.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.517+0000 7fd643fff640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fd62c0763f0 0x7fd62c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:52.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.517+0000 7fd643fff640 1 -- 172.21.15.59:0/2271429946 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6001+0+0 (secure 0 0 0) 0x7fd634097bc0 con 0x7fd64c161fc0 2024-06-29T14:37:52.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.517+0000 7fd64affd640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fd62c0763f0 0x7fd62c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.518+0000 7fd64affd640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fd62c0763f0 0x7fd62c078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd63c035000 tx=0x7fd63c03b040 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:52.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.521+0000 7fd643fff640 1 -- 172.21.15.59:0/2271429946 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd634060ae0 con 0x7fd64c161fc0 2024-06-29T14:37:52.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:52.752+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd64c15db30 con 0x7fd64c161fc0 2024-06-29T14:37:53.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.384+0000 7fd643fff640 1 -- 172.21.15.59:0/2271429946 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v43) ==== 174+0+0 (secure 0 0 0) 0x7fd634064de0 con 0x7fd64c161fc0 2024-06-29T14:37:53.383 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 11 and data pool 12 2024-06-29T14:37:53.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.401+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fd62c0763f0 msgr2=0x7fd62c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.401+0000 7fd65192a640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fd62c0763f0 0x7fd62c078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd63c035000 tx=0x7fd63c03b040 comp rx=0 tx=0).stop 2024-06-29T14:37:53.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.401+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c161fc0 msgr2=0x7fd64c1f8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.401+0000 7fd65192a640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c161fc0 0x7fd64c1f8280 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd63400d6a0 tx=0x7fd63400db60 comp rx=0 tx=0).stop 2024-06-29T14:37:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.404+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 shutdown_connections 2024-06-29T14:37:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.404+0000 7fd65192a640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd64c162770 0x7fd64c1f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.404+0000 7fd65192a640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd64c161fc0 0x7fd64c1f8280 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.404+0000 7fd65192a640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fd62c0763f0 0x7fd62c078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.404+0000 7fd65192a640 1 --2- 172.21.15.59:0/2271429946 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd64c15e0a0 0x7fd64c15ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.404+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 >> 172.21.15.59:0/2271429946 conn(0x7fd64c0d45b0 msgr2=0x7fd64c0b73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.405+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 shutdown_connections 2024-06-29T14:37:53.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.406+0000 7fd65192a640 1 -- 172.21.15.59:0/2271429946 wait complete. 2024-06-29T14:37:53.420 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:53.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.546+0000 7fe9758d1640 1 -- 172.21.15.59:0/1740749756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97015ceb0 msgr2=0x7fe9700b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.546+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1740749756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97015ceb0 0x7fe9700b56d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe960005fd0 tx=0x7fe96002aa10 comp rx=0 tx=0).stop 2024-06-29T14:37:53.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.546+0000 7fe9758d1640 1 -- 172.21.15.59:0/1740749756 shutdown_connections 2024-06-29T14:37:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.546+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1740749756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe97013e9b0 0x7fe97013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.546+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1740749756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97015ceb0 0x7fe9700b56d0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.546+0000 7fe9758d1640 1 -- 172.21.15.59:0/1740749756 >> 172.21.15.59:0/1740749756 conn(0x7fe9700b6fb0 msgr2=0x7fe9700b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.546+0000 7fe9758d1640 1 -- 172.21.15.59:0/1740749756 shutdown_connections 2024-06-29T14:37:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.546+0000 7fe9758d1640 1 -- 172.21.15.59:0/1740749756 wait complete. 2024-06-29T14:37:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.548+0000 7fe9758d1640 1 Processor -- start 2024-06-29T14:37:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.548+0000 7fe9758d1640 1 -- start start 2024-06-29T14:37:53.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe9758d1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97013e9b0 0x7fe97013d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe9758d1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9701414c0 0x7fe97006f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe9758d1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97015ceb0 0x7fe970070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe9758d1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe9700b2ab0 con 0x7fe97013e9b0 2024-06-29T14:37:53.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe9758d1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9700b2930 con 0x7fe97015ceb0 2024-06-29T14:37:53.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe9758d1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe9700b2c30 con 0x7fe9701414c0 2024-06-29T14:37:53.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe96effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97013e9b0 0x7fe97013d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe96effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97013e9b0 0x7fe97013d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33562/0 (socket says 172.21.15.59:33562) 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe96effd640 1 -- 172.21.15.59:0/1427713624 learned_addr learned my addr 172.21.15.59:0/1427713624 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe96e7fc640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9701414c0 0x7fe97006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.549+0000 7fe96f7fe640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97015ceb0 0x7fe970070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe96effd640 1 -- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9701414c0 msgr2=0x7fe97006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe96effd640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9701414c0 0x7fe97006f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe96effd640 1 -- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97015ceb0 msgr2=0x7fe970070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe96effd640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97015ceb0 0x7fe970070f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe96effd640 1 -- 172.21.15.59:0/1427713624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9701f48b0 con 0x7fe97013e9b0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe96effd640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97013e9b0 0x7fe97013d480 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe960009d90 tx=0x7fe960003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe967fff640 1 -- 172.21.15.59:0/1427713624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960033050 con 0x7fe97013e9b0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe967fff640 1 -- 172.21.15.59:0/1427713624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe960004510 con 0x7fe97013e9b0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe967fff640 1 -- 172.21.15.59:0/1427713624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960038450 con 0x7fe97013e9b0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97006df60 con 0x7fe97013e9b0 2024-06-29T14:37:53.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.550+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97006e520 con 0x7fe97013e9b0 2024-06-29T14:37:53.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.551+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9701f4a80 con 0x7fe97013e9b0 2024-06-29T14:37:53.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.552+0000 7fe967fff640 1 -- 172.21.15.59:0/1427713624 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fe960036070 con 0x7fe97013e9b0 2024-06-29T14:37:53.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.552+0000 7fe967fff640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe9500763f0 0x7fe950078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.552+0000 7fe967fff640 1 -- 172.21.15.59:0/1427713624 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6101+0+0 (secure 0 0 0) 0x7fe9600b9070 con 0x7fe97013e9b0 2024-06-29T14:37:53.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.553+0000 7fe96e7fc640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe9500763f0 0x7fe950078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.553+0000 7fe96e7fc640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe9500763f0 0x7fe950078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe9700cff30 tx=0x7fe95800b2e0 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.556+0000 7fe967fff640 1 -- 172.21.15.59:0/1427713624 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe960081fd0 con 0x7fe97013e9b0 2024-06-29T14:37:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.751+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe97013edc0 con 0x7fe97013e9b0 2024-06-29T14:37:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.753+0000 7fe967fff640 1 -- 172.21.15.59:0/1427713624 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+16973 (secure 0 0 0) 0x7fe9600862d0 con 0x7fe97013e9b0 2024-06-29T14:37:53.750 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:53.750 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":100,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:52.753233+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:51.120910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.698419+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:53.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe9500763f0 msgr2=0x7fe950078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe9500763f0 0x7fe950078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe9700cff30 tx=0x7fe95800b2e0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97013e9b0 msgr2=0x7fe97013d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97013e9b0 0x7fe97013d480 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe960009d90 tx=0x7fe960003f10 comp rx=0 tx=0).stop 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 shutdown_connections 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe97015ceb0 0x7fe970070f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fe9500763f0 0x7fe950078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9701414c0 0x7fe97006f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 --2- 172.21.15.59:0/1427713624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe97013e9b0 0x7fe97013d480 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 >> 172.21.15.59:0/1427713624 conn(0x7fe9700b6fb0 msgr2=0x7fe97015e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 shutdown_connections 2024-06-29T14:37:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.755+0000 7fe9758d1640 1 -- 172.21.15.59:0/1427713624 wait complete. 2024-06-29T14:37:53.764 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:53.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.884+0000 7f046f23d640 1 -- 172.21.15.59:0/2052543614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 msgr2=0x7f0454003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.884+0000 7f046f23d640 1 --2- 172.21.15.59:0/2052543614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 0x7f0454003ee0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f045800c290 tx=0x7f0454005830 comp rx=0 tx=0).stop 2024-06-29T14:37:53.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.884+0000 7f046d7da640 1 -- 172.21.15.59:0/2052543614 reap_dead start 2024-06-29T14:37:53.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.885+0000 7f046f23d640 1 -- 172.21.15.59:0/2052543614 shutdown_connections 2024-06-29T14:37:53.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.885+0000 7f046f23d640 1 -- 172.21.15.59:0/2052543614 >> 172.21.15.59:0/2052543614 conn(0x7f046806c950 msgr2=0x7f0468073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.885+0000 7f046f23d640 1 -- 172.21.15.59:0/2052543614 shutdown_connections 2024-06-29T14:37:53.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.885+0000 7f046f23d640 1 -- 172.21.15.59:0/2052543614 wait complete. 2024-06-29T14:37:53.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.886+0000 7f046f23d640 1 Processor -- start 2024-06-29T14:37:53.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.886+0000 7f046f23d640 1 -- start start 2024-06-29T14:37:53.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046f23d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 0x7f046813b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046f23d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f046815dc10 0x7f046815b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046f23d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046813a7f0 0x7f046813ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046f23d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f04680b5c10 con 0x7f0454003a70 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046f23d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f04680b5a90 con 0x7f046813a7f0 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046f23d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f04680b5d90 con 0x7f046815dc10 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 0x7f046813b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 0x7f046813b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33570/0 (socket says 172.21.15.59:33570) 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 -- 172.21.15.59:0/2792734006 learned_addr learned my addr 172.21.15.59:0/2792734006 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f0467fff640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f046815dc10 0x7f046815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046d7da640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046813a7f0 0x7f046813ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 -- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f046815dc10 msgr2=0x7f046815b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f046815dc10 0x7f046815b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 -- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046813a7f0 msgr2=0x7f046813ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046813a7f0 0x7f046813ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 -- 172.21.15.59:0/2792734006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0468144c60 con 0x7f0454003a70 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046d7da640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046813a7f0 0x7f046813ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.888+0000 7f046cfd9640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 0x7f046813b410 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f045800c3b0 tx=0x7f0458006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.889+0000 7f04657fa640 1 -- 172.21.15.59:0/2792734006 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458031040 con 0x7f0454003a70 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.889+0000 7f04657fa640 1 -- 172.21.15.59:0/2792734006 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04580035f0 con 0x7f0454003a70 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.889+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0468139750 con 0x7f0454003a70 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.889+0000 7f04657fa640 1 -- 172.21.15.59:0/2792734006 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458003a00 con 0x7f0454003a70 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.889+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0468139d10 con 0x7f0454003a70 2024-06-29T14:37:53.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.890+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04681394b0 con 0x7f0454003a70 2024-06-29T14:37:53.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.891+0000 7f04657fa640 1 -- 172.21.15.59:0/2792734006 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f0458034070 con 0x7f0454003a70 2024-06-29T14:37:53.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.891+0000 7f04657fa640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f04480763f0 0x7f0448078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.891+0000 7f04657fa640 1 -- 172.21.15.59:0/2792734006 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6101+0+0 (secure 0 0 0) 0x7f04580b7bc0 con 0x7f0454003a70 2024-06-29T14:37:53.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.891+0000 7f0467fff640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f04480763f0 0x7f0448078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.892+0000 7f0467fff640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f04480763f0 0x7f0448078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0468074380 tx=0x7f0450009500 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:53.895+0000 7f04657fa640 1 -- 172.21.15.59:0/2792734006 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f04681394b0 con 0x7f0454003a70 2024-06-29T14:37:54.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.086+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04681458a0 con 0x7f0454003a70 2024-06-29T14:37:54.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.087+0000 7f04657fa640 1 -- 172.21.15.59:0/2792734006 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+16973 (secure 0 0 0) 0x7f0458081170 con 0x7f0454003a70 2024-06-29T14:37:54.085 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:54.085 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":100,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:52.753233+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:51.120910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.698419+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f04480763f0 msgr2=0x7f0448078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f04480763f0 0x7f0448078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0468074380 tx=0x7f0450009500 comp rx=0 tx=0).stop 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 msgr2=0x7f046813b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 0x7f046813b410 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f045800c3b0 tx=0x7f0458006d70 comp rx=0 tx=0).stop 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 shutdown_connections 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f046813a7f0 0x7f046813ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f04480763f0 0x7f0448078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f046815dc10 0x7f046815b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 --2- 172.21.15.59:0/2792734006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0454003a70 0x7f046813b410 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.089+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 >> 172.21.15.59:0/2792734006 conn(0x7f046806c950 msgr2=0x7f046815ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:54.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.090+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 shutdown_connections 2024-06-29T14:37:54.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.090+0000 7f046f23d640 1 -- 172.21.15.59:0/2792734006 wait complete. 2024-06-29T14:37:54.098 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:37:54.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.222+0000 7f577aec3640 1 -- 172.21.15.59:0/2178685186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f577406d2f0 msgr2=0x7f577406d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:54.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.222+0000 7f577aec3640 1 --2- 172.21.15.59:0/2178685186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f577406d2f0 0x7f577406d700 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f5764005fd0 tx=0x7f576402a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.222+0000 7f577aec3640 1 -- 172.21.15.59:0/2178685186 shutdown_connections 2024-06-29T14:37:54.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.222+0000 7f577aec3640 1 --2- 172.21.15.59:0/2178685186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f577406d2f0 0x7f577406d700 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.222+0000 7f577aec3640 1 --2- 172.21.15.59:0/2178685186 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f57740cd9b0 0x7f577415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.222+0000 7f577aec3640 1 -- 172.21.15.59:0/2178685186 >> 172.21.15.59:0/2178685186 conn(0x7f57740b24c0 msgr2=0x7f57740cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:54.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.222+0000 7f577aec3640 1 -- 172.21.15.59:0/2178685186 shutdown_connections 2024-06-29T14:37:54.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.222+0000 7f577aec3640 1 -- 172.21.15.59:0/2178685186 wait complete. 2024-06-29T14:37:54.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.223+0000 7f577aec3640 1 Processor -- start 2024-06-29T14:37:54.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.223+0000 7f577aec3640 1 -- start start 2024-06-29T14:37:54.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f577aec3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f577406d2f0 0x7f57741eed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:54.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f577aec3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740732b0 0x7f57740ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:54.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f577aec3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740cd9b0 0x7f57740b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:54.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f577aec3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f57741448f0 con 0x7f57740cd9b0 2024-06-29T14:37:54.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f577aec3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5774144770 con 0x7f57740732b0 2024-06-29T14:37:54.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f577aec3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5774073dd0 con 0x7f577406d2f0 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f5779460640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740cd9b0 0x7f57740b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f5779460640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740cd9b0 0x7f57740b3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33594/0 (socket says 172.21.15.59:33594) 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f5778c5f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f577406d2f0 0x7f57741eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f5779460640 1 -- 172.21.15.59:0/4027816993 learned_addr learned my addr 172.21.15.59:0/4027816993 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.225+0000 7f576bfff640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740732b0 0x7f57740ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.226+0000 7f5779460640 1 -- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f577406d2f0 msgr2=0x7f57741eed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.226+0000 7f5779460640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f577406d2f0 0x7f57741eed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.226+0000 7f5779460640 1 -- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740732b0 msgr2=0x7f57740ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.226+0000 7f5779460640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740732b0 0x7f57740ae980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.226+0000 7f5779460640 1 -- 172.21.15.59:0/4027816993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57741ef270 con 0x7f57740cd9b0 2024-06-29T14:37:54.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.226+0000 7f5779460640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740cd9b0 0x7f57740b3e50 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c760 tx=0x7f576c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:54.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.228+0000 7f57697fa640 1 -- 172.21.15.59:0/4027816993 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c003da0 con 0x7f57740cd9b0 2024-06-29T14:37:54.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.228+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57741f4eb0 con 0x7f57740cd9b0 2024-06-29T14:37:54.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.228+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57741f5470 con 0x7f57740cd9b0 2024-06-29T14:37:54.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.228+0000 7f57697fa640 1 -- 172.21.15.59:0/4027816993 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f576c00ce10 con 0x7f57740cd9b0 2024-06-29T14:37:54.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.228+0000 7f57697fa640 1 -- 172.21.15.59:0/4027816993 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c005290 con 0x7f57740cd9b0 2024-06-29T14:37:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.229+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5754005290 con 0x7f57740cd9b0 2024-06-29T14:37:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.230+0000 7f57697fa640 1 -- 172.21.15.59:0/4027816993 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f576c005430 con 0x7f57740cd9b0 2024-06-29T14:37:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.230+0000 7f57697fa640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5760076420 0x7f5760078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.230+0000 7f57697fa640 1 -- 172.21.15.59:0/4027816993 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6101+0+0 (secure 0 0 0) 0x7f576c0976a0 con 0x7f57740cd9b0 2024-06-29T14:37:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.231+0000 7f5778c5f640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5760076420 0x7f5760078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.231+0000 7f5778c5f640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5760076420 0x7f5760078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5770009b30 tx=0x7f5770005f90 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:54.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.235+0000 7f57697fa640 1 -- 172.21.15.59:0/4027816993 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f577415c1c0 con 0x7f57740cd9b0 2024-06-29T14:37:54.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:54.469+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5754005850 con 0x7f57740cd9b0 2024-06-29T14:37:55.341 INFO:tasks.ceph.mds.b.smithi153.stderr:2024-06-29T14:37:55.343+0000 7f5572dea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:37:55.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.391+0000 7f57697fa640 1 -- 172.21.15.59:0/4027816993 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) ==== 118+0+0 (secure 0 0 0) 0x7f576c060600 con 0x7f57740cd9b0 2024-06-29T14:37:55.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5760076420 msgr2=0x7f5760078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5760076420 0x7f5760078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5770009b30 tx=0x7f5770005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740cd9b0 msgr2=0x7f57740b3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740cd9b0 0x7f57740b3e50 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c760 tx=0x7f576c00cc20 comp rx=0 tx=0).stop 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 shutdown_connections 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740cd9b0 0x7f57740b3e50 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57740732b0 0x7f57740ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5760076420 0x7f5760078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 --2- 172.21.15.59:0/4027816993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f577406d2f0 0x7f57741eed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 >> 172.21.15.59:0/4027816993 conn(0x7f57740b24c0 msgr2=0x7f57740736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 shutdown_connections 2024-06-29T14:37:55.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.394+0000 7f577aec3640 1 -- 172.21.15.59:0/4027816993 wait complete. 2024-06-29T14:37:55.406 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:37:55.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:37:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.537+0000 7f71bde9e640 1 -- 172.21.15.59:0/2688185087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 msgr2=0x7f71b815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.537+0000 7f71bde9e640 1 --2- 172.21.15.59:0/2688185087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 0x7f71b815bc20 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f71a8005fd0 tx=0x7f71a802aa90 comp rx=0 tx=0).stop 2024-06-29T14:37:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.537+0000 7f71bde9e640 1 -- 172.21.15.59:0/2688185087 shutdown_connections 2024-06-29T14:37:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.537+0000 7f71bde9e640 1 --2- 172.21.15.59:0/2688185087 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b80b2fa0 0x7f71b80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.537+0000 7f71bde9e640 1 --2- 172.21.15.59:0/2688185087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 0x7f71b815bc20 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.537+0000 7f71bde9e640 1 -- 172.21.15.59:0/2688185087 >> 172.21.15.59:0/2688185087 conn(0x7f71b8073e10 msgr2=0x7f71b8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.537+0000 7f71bde9e640 1 -- 172.21.15.59:0/2688185087 shutdown_connections 2024-06-29T14:37:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.537+0000 7f71bde9e640 1 -- 172.21.15.59:0/2688185087 wait complete. 2024-06-29T14:37:55.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.539+0000 7f71bde9e640 1 Processor -- start 2024-06-29T14:37:55.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.539+0000 7f71bde9e640 1 -- start start 2024-06-29T14:37:55.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71bde9e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 0x7f71b80af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:55.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71bde9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b80b2fa0 0x7f71b80afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71bde9e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b80d1a60 0x7f71b80b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71bde9e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f71b80b6cc0 con 0x7f71b80b2270 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71bde9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71b80b6b40 con 0x7f71b80b2fa0 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71bde9e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f71b80cfe90 con 0x7f71b80d1a60 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 0x7f71b80af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 0x7f71b80af540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33614/0 (socket says 172.21.15.59:33614) 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 -- 172.21.15.59:0/438180983 learned_addr learned my addr 172.21.15.59:0/438180983 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b6ffd640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b80b2fa0 0x7f71b80afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b7fff640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b80d1a60 0x7f71b80b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 -- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b80d1a60 msgr2=0x7f71b80b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b80d1a60 0x7f71b80b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 -- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b80b2fa0 msgr2=0x7f71b80afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b80b2fa0 0x7f71b80afe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 -- 172.21.15.59:0/438180983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b80afa80 con 0x7f71b80b2270 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b7fff640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b80d1a60 0x7f71b80b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71b77fe640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 0x7f71b80af540 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f71a8000f80 tx=0x7f71a8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f719ffff640 1 -- 172.21.15.59:0/438180983 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8044070 con 0x7f71b80b2270 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f719ffff640 1 -- 172.21.15.59:0/438180983 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71a8004250 con 0x7f71b80b2270 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.541+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b80ae4a0 con 0x7f71b80b2270 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.542+0000 7f719ffff640 1 -- 172.21.15.59:0/438180983 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8004660 con 0x7f71b80b2270 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.542+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b80aea60 con 0x7f71b80b2270 2024-06-29T14:37:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.543+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b80d0390 con 0x7f71b80b2270 2024-06-29T14:37:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.543+0000 7f719ffff640 1 -- 172.21.15.59:0/438180983 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f71a8047070 con 0x7f71b80b2270 2024-06-29T14:37:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.544+0000 7f719ffff640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f71a40763f0 0x7f71a4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.544+0000 7f719ffff640 1 -- 172.21.15.59:0/438180983 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f71a80b89e0 con 0x7f71b80b2270 2024-06-29T14:37:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.544+0000 7f71b6ffd640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f71a40763f0 0x7f71a4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:55.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.545+0000 7f71b6ffd640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f71a40763f0 0x7f71a4078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f71b81558f0 tx=0x7f71ac00b2e0 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.548+0000 7f719ffff640 1 -- 172.21.15.59:0/438180983 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f71a8032030 con 0x7f71b80b2270 2024-06-29T14:37:55.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:55.780+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f71b8075b40 con 0x7f71b80b2270 2024-06-29T14:37:56.119 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:37:56.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:37:56.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.391+0000 7f719ffff640 1 -- 172.21.15.59:0/438180983 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7f71a8081940 con 0x7f71b80b2270 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f71a40763f0 msgr2=0x7f71a4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f71a40763f0 0x7f71a4078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f71b81558f0 tx=0x7f71ac00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 msgr2=0x7f71b80af540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 0x7f71b80af540 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f71a8000f80 tx=0x7f71a8003ce0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 shutdown_connections 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f71a40763f0 0x7f71a4078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71b80d1a60 0x7f71b80b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71b80b2fa0 0x7f71b80afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 --2- 172.21.15.59:0/438180983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71b80b2270 0x7f71b80af540 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 >> 172.21.15.59:0/438180983 conn(0x7f71b8073e10 msgr2=0x7f71b80d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 shutdown_connections 2024-06-29T14:37:56.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.394+0000 7f71bde9e640 1 -- 172.21.15.59:0/438180983 wait complete. 2024-06-29T14:37:56.405 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:37:56.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:56.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 -- 172.21.15.59:0/1076112619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04162770 msgr2=0x7f8c040ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 --2- 172.21.15.59:0/1076112619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04162770 0x7f8c040ce550 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f8c040b35d0 tx=0x7f8bf402a4b0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 -- 172.21.15.59:0/1076112619 shutdown_connections 2024-06-29T14:37:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 --2- 172.21.15.59:0/1076112619 >> v2:172.21.15.153:3300/0 conn(0x7f8c04161fc0 0x7f8c040d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 --2- 172.21.15.59:0/1076112619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c0415e0a0 0x7f8c04161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 --2- 172.21.15.59:0/1076112619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04162770 0x7f8c040ce550 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 -- 172.21.15.59:0/1076112619 >> 172.21.15.59:0/1076112619 conn(0x7f8c040d45b0 msgr2=0x7f8c040d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 -- 172.21.15.59:0/1076112619 shutdown_connections 2024-06-29T14:37:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.531+0000 7f8c09794640 1 -- 172.21.15.59:0/1076112619 wait complete. 2024-06-29T14:37:56.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.533+0000 7f8c09794640 1 Processor -- start 2024-06-29T14:37:56.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.533+0000 7f8c09794640 1 -- start start 2024-06-29T14:37:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.534+0000 7f8c09794640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c0415e0a0 0x7f8c041526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.534+0000 7f8c09794640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04161fc0 0x7f8c04156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c09794640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04162770 0x7f8c0415b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c09794640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8c040d2f40 con 0x7f8c04161fc0 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c09794640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c040d2dc0 con 0x7f8c04162770 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c09794640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8c040d30c0 con 0x7f8c0415e0a0 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04161fc0 0x7f8c04156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04161fc0 0x7f8c04156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33628/0 (socket says 172.21.15.59:33628) 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 -- 172.21.15.59:0/1902622175 learned_addr learned my addr 172.21.15.59:0/1902622175 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c037fe640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04162770 0x7f8c0415b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c02ffd640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c0415e0a0 0x7f8c041526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 -- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c0415e0a0 msgr2=0x7f8c041526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c0415e0a0 0x7f8c041526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 -- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04162770 msgr2=0x7f8c0415b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04162770 0x7f8c0415b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 -- 172.21.15.59:0/1902622175 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c041548b0 con 0x7f8c04161fc0 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c037fe640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04162770 0x7f8c0415b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c02ffd640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c0415e0a0 0x7f8c041526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c027fc640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04161fc0 0x7f8c04156c30 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00c770 tx=0x7f8bec00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8bfb7fe640 1 -- 172.21.15.59:0/1902622175 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bec007b50 con 0x7f8c04161fc0 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8bfb7fe640 1 -- 172.21.15.59:0/1902622175 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bec007de0 con 0x7f8c04161fc0 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c041555c0 con 0x7f8c04161fc0 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.535+0000 7f8bfb7fe640 1 -- 172.21.15.59:0/1902622175 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bec005280 con 0x7f8c04161fc0 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.536+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c04154d20 con 0x7f8c04161fc0 2024-06-29T14:37:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.537+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c04157170 con 0x7f8c04161fc0 2024-06-29T14:37:56.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.537+0000 7f8bfb7fe640 1 -- 172.21.15.59:0/1902622175 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f8bec005420 con 0x7f8c04161fc0 2024-06-29T14:37:56.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.538+0000 7f8bfb7fe640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f8be40763f0 0x7f8be4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.538+0000 7f8bfb7fe640 1 -- 172.21.15.59:0/1902622175 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f8bec097f60 con 0x7f8c04161fc0 2024-06-29T14:37:56.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.538+0000 7f8c02ffd640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f8be40763f0 0x7f8be4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.539+0000 7f8c02ffd640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f8be40763f0 0x7f8be4078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4033000 tx=0x7f8bf4004730 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.542+0000 7f8bfb7fe640 1 -- 172.21.15.59:0/1902622175 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8bec014030 con 0x7f8c04161fc0 2024-06-29T14:37:56.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.776+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c04154a80 con 0x7f8c04161fc0 2024-06-29T14:37:56.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.777+0000 7f8bfb7fe640 1 -- 172.21.15.59:0/1902622175 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5228 (secure 0 0 0) 0x7f8bec061480 con 0x7f8c04161fc0 2024-06-29T14:37:56.775 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:56.776 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":46,"btime":"2024-06-29T14:37:56:391508+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2174310652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2174310652},{"type":"v1","addr":"172.21.15.59:6835","nonce":2174310652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3217643445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3217643445},{"type":"v1","addr":"172.21.15.153:6836","nonce":3217643445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/832666341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":832666341},{"type":"v1","addr":"172.21.15.59:6833","nonce":832666341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.753189+0000","modified":"2024-06-29T14:37:55.781006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7365},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7365":{"gid":7365,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/1942687234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1942687234},{"type":"v1","addr":"172.21.15.153:6837","nonce":1942687234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7365,"qdb_cluster":[7365]},"id":5}]} 2024-06-29T14:37:56.776 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 46 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f8be40763f0 msgr2=0x7f8be4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f8be40763f0 0x7f8be4078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4033000 tx=0x7f8bf4004730 comp rx=0 tx=0).stop 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04161fc0 msgr2=0x7f8c04156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04161fc0 0x7f8c04156c30 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00c770 tx=0x7f8bec00cc30 comp rx=0 tx=0).stop 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 shutdown_connections 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04162770 0x7f8c0415b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c04161fc0 0x7f8c04156c30 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f8be40763f0 0x7f8be4078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 --2- 172.21.15.59:0/1902622175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c0415e0a0 0x7f8c041526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 >> 172.21.15.59:0/1902622175 conn(0x7f8c040d45b0 msgr2=0x7f8c0406d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 shutdown_connections 2024-06-29T14:37:56.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.779+0000 7f8c09794640 1 -- 172.21.15.59:0/1902622175 wait complete. 2024-06-29T14:37:56.789 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.908+0000 7f7b242e4640 1 -- 172.21.15.59:0/3437735441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 msgr2=0x7f7b1c073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.908+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3437735441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 0x7f7b1c073be0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0705a0 tx=0x7f7b1802a530 comp rx=0 tx=0).stop 2024-06-29T14:37:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.909+0000 7f7b242e4640 1 -- 172.21.15.59:0/3437735441 shutdown_connections 2024-06-29T14:37:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.909+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3437735441 >> v2:172.21.15.153:3300/0 conn(0x7f7b1c0cc060 0x7f7b1c0acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.909+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3437735441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 0x7f7b1c073be0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.909+0000 7f7b242e4640 1 -- 172.21.15.59:0/3437735441 >> 172.21.15.59:0/3437735441 conn(0x7f7b1c0b1ea0 msgr2=0x7f7b1c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.909+0000 7f7b242e4640 1 -- 172.21.15.59:0/3437735441 shutdown_connections 2024-06-29T14:37:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.909+0000 7f7b242e4640 1 -- 172.21.15.59:0/3437735441 wait complete. 2024-06-29T14:37:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.910+0000 7f7b242e4640 1 Processor -- start 2024-06-29T14:37:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.910+0000 7f7b242e4640 1 -- start start 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b242e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b1c0ad4e0 0x7f7b1c146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b242e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 0x7f7b1c09c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b242e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b1c0cc060 0x7f7b1c1574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b242e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c0ce750 con 0x7f7b1c0cb490 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b242e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c074230 con 0x7f7b1c0ad4e0 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b242e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1c0ce8d0 con 0x7f7b1c0cc060 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 0x7f7b1c09c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 0x7f7b1c09c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33638/0 (socket says 172.21.15.59:33638) 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 -- 172.21.15.59:0/3067475285 learned_addr learned my addr 172.21.15.59:0/3067475285 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b22881640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b1c0cc060 0x7f7b1c1574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 -- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b1c0cc060 msgr2=0x7f7b1c1574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b22080640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b1c0ad4e0 0x7f7b1c146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b1c0cc060 0x7f7b1c1574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 -- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b1c0ad4e0 msgr2=0x7f7b1c146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b1c0ad4e0 0x7f7b1c146ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 -- 172.21.15.59:0/3067475285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c09c3d0 con 0x7f7b1c0cb490 2024-06-29T14:37:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b22881640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b1c0cc060 0x7f7b1c1574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b22080640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b1c0ad4e0 0x7f7b1c146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.912+0000 7f7b2187f640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 0x7f7b1c09c630 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00c9a0 tx=0x7f7b0c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.913+0000 7f7b12ffd640 1 -- 172.21.15.59:0/3067475285 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c007bb0 con 0x7f7b1c0cb490 2024-06-29T14:37:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.913+0000 7f7b12ffd640 1 -- 172.21.15.59:0/3067475285 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b0c007e40 con 0x7f7b1c0cb490 2024-06-29T14:37:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.913+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1c09aed0 con 0x7f7b1c0cb490 2024-06-29T14:37:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.913+0000 7f7b12ffd640 1 -- 172.21.15.59:0/3067475285 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c005280 con 0x7f7b1c0cb490 2024-06-29T14:37:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.913+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1c09b490 con 0x7f7b1c0cb490 2024-06-29T14:37:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.914+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b1c0b1470 con 0x7f7b1c0cb490 2024-06-29T14:37:56.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.915+0000 7f7b12ffd640 1 -- 172.21.15.59:0/3067475285 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f7b0c005420 con 0x7f7b1c0cb490 2024-06-29T14:37:56.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.915+0000 7f7b12ffd640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7b000763f0 0x7f7b00078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.915+0000 7f7b12ffd640 1 -- 172.21.15.59:0/3067475285 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f7b0c097fb0 con 0x7f7b1c0cb490 2024-06-29T14:37:56.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.915+0000 7f7b22080640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7b000763f0 0x7f7b00078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.916+0000 7f7b22080640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7b000763f0 0x7f7b00078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7b18034000 tx=0x7f7b18004730 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:56.919+0000 7f7b12ffd640 1 -- 172.21.15.59:0/3067475285 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7b1c0b1470 con 0x7f7b1c0cb490 2024-06-29T14:37:57.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.113+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b1c09ac60 con 0x7f7b1c0cb490 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.114+0000 7f7b12ffd640 1 -- 172.21.15.59:0/3067475285 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+16960 (secure 0 0 0) 0x7f7b0c014030 con 0x7f7b1c0cb490 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:57.113 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":101,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:37:54.369835+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:51.120910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.698419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7b000763f0 msgr2=0x7f7b00078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7b000763f0 0x7f7b00078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7b18034000 tx=0x7f7b18004730 comp rx=0 tx=0).stop 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 msgr2=0x7f7b1c09c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 0x7f7b1c09c630 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00c9a0 tx=0x7f7b0c00ce60 comp rx=0 tx=0).stop 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 shutdown_connections 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7b000763f0 0x7f7b00078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b1c0cc060 0x7f7b1c1574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b1c0cb490 0x7f7b1c09c630 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 --2- 172.21.15.59:0/3067475285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b1c0ad4e0 0x7f7b1c146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 >> 172.21.15.59:0/3067475285 conn(0x7f7b1c0b1ea0 msgr2=0x7f7b1c0cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 shutdown_connections 2024-06-29T14:37:57.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.116+0000 7f7b242e4640 1 -- 172.21.15.59:0/3067475285 wait complete. 2024-06-29T14:37:57.126 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:37:57.126 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:37:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.246+0000 7f14c5b03640 1 -- 172.21.15.59:0/2359084021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01458a0 msgr2=0x7f14c00cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.246+0000 7f14c5b03640 1 --2- 172.21.15.59:0/2359084021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01458a0 0x7f14c00cf870 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f14c00b1c40 tx=0x7f14b002a530 comp rx=0 tx=0).stop 2024-06-29T14:37:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.246+0000 7f14c5b03640 1 -- 172.21.15.59:0/2359084021 shutdown_connections 2024-06-29T14:37:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.246+0000 7f14c5b03640 1 --2- 172.21.15.59:0/2359084021 >> v2:172.21.15.153:3300/0 conn(0x7f14c00b3740 0x7f14c00b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.246+0000 7f14c5b03640 1 --2- 172.21.15.59:0/2359084021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01458a0 0x7f14c00cf870 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.246+0000 7f14c5b03640 1 -- 172.21.15.59:0/2359084021 >> 172.21.15.59:0/2359084021 conn(0x7f14c006c950 msgr2=0x7f14c0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.246+0000 7f14c5b03640 1 -- 172.21.15.59:0/2359084021 shutdown_connections 2024-06-29T14:37:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.246+0000 7f14c5b03640 1 -- 172.21.15.59:0/2359084021 wait complete. 2024-06-29T14:37:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.247+0000 7f14c5b03640 1 Processor -- start 2024-06-29T14:37:57.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.248+0000 7f14c5b03640 1 -- start start 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14c5b03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14c00b3740 0x7f14c013afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14c5b03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c015c2f0 0x7f14c0139560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14c5b03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01f4a20 0x7f14c01380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14c5b03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f14c0160020 con 0x7f14c01f4a20 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14c5b03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14c015cbb0 con 0x7f14c015c2f0 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14c5b03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f14c01601a0 con 0x7f14c00b3740 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01f4a20 0x7f14c01380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01f4a20 0x7f14c01380a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33646/0 (socket says 172.21.15.59:33646) 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bffff640 1 -- 172.21.15.59:0/465588018 learned_addr learned my addr 172.21.15.59:0/465588018 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bf7fe640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14c00b3740 0x7f14c013afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14beffd640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c015c2f0 0x7f14c0139560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bffff640 1 -- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14c00b3740 msgr2=0x7f14c013afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bffff640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14c00b3740 0x7f14c013afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bffff640 1 -- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c015c2f0 msgr2=0x7f14c0139560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bffff640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c015c2f0 0x7f14c0139560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bffff640 1 -- 172.21.15.59:0/465588018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c0144fc0 con 0x7f14c01f4a20 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.249+0000 7f14bf7fe640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14c00b3740 0x7f14c013afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.250+0000 7f14bffff640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01f4a20 0x7f14c01380a0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f14b800cb40 tx=0x7f14b800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.250+0000 7f14a7fff640 1 -- 172.21.15.59:0/465588018 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b8016020 con 0x7f14c01f4a20 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.250+0000 7f14a7fff640 1 -- 172.21.15.59:0/465588018 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14b800ad30 con 0x7f14c01f4a20 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.250+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c01f5170 con 0x7f14c01f4a20 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.250+0000 7f14a7fff640 1 -- 172.21.15.59:0/465588018 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b8012680 con 0x7f14c01f4a20 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.250+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c01f5730 con 0x7f14c01f4a20 2024-06-29T14:37:57.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.251+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c01f4ee0 con 0x7f14c01f4a20 2024-06-29T14:37:57.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.252+0000 7f14a7fff640 1 -- 172.21.15.59:0/465588018 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f14b8019020 con 0x7f14c01f4a20 2024-06-29T14:37:57.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.252+0000 7f14a7fff640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f14ac0763f0 0x7f14ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.252+0000 7f14a7fff640 1 -- 172.21.15.59:0/465588018 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f14b8097ce0 con 0x7f14c01f4a20 2024-06-29T14:37:57.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.253+0000 7f14bf7fe640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f14ac0763f0 0x7f14ac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.253+0000 7f14bf7fe640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f14ac0763f0 0x7f14ac078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14b0034000 tx=0x7f14b0004730 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:57.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.257+0000 7f14a7fff640 1 -- 172.21.15.59:0/465588018 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f14b8020030 con 0x7f14c01f4a20 2024-06-29T14:37:57.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.449+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 --> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14c01606c0 con 0x7f14ac0763f0 2024-06-29T14:37:57.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.455+0000 7f14a7fff640 1 -- 172.21.15.59:0/465588018 <== mgr.6179 v2:172.21.15.153:6832/1607211985 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252265 (secure 0 0 0) 0x7f14c01606c0 con 0x7f14ac0763f0 2024-06-29T14:37:57.456 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:57.457 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f14ac0763f0 msgr2=0x7f14ac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f14ac0763f0 0x7f14ac078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14b0034000 tx=0x7f14b0004730 comp rx=0 tx=0).stop 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01f4a20 msgr2=0x7f14c01380a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01f4a20 0x7f14c01380a0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f14b800cb40 tx=0x7f14b800a060 comp rx=0 tx=0).stop 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 shutdown_connections 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c01f4a20 0x7f14c01380a0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14c015c2f0 0x7f14c0139560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f14ac0763f0 0x7f14ac078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 --2- 172.21.15.59:0/465588018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14c00b3740 0x7f14c013afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 >> 172.21.15.59:0/465588018 conn(0x7f14c006c950 msgr2=0x7f14c00ced30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 shutdown_connections 2024-06-29T14:37:57.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.458+0000 7f14c5b03640 1 -- 172.21.15.59:0/465588018 wait complete. 2024-06-29T14:37:57.467 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:37:56.914877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":218,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331048,"kb_used_data":10800,"kb_used_omap":12,"kb_used_meta":320115,"kb_avail":754643672,"statfs":{"total":773094113280,"available":772755120128,"internally_reserved":0,"allocated":11059200,"data_stored":6125564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":327798322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.565570"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377063+0000","last_change":"2024-06-29T14:37:53.394513+0000","last_active":"2024-06-29T14:37:54.377063+0000","last_peered":"2024-06-29T14:37:54.377063+0000","last_clean":"2024-06-29T14:37:54.377063+0000","last_became_active":"2024-06-29T14:37:53.394173+0000","last_became_peered":"2024-06-29T14:37:53.394173+0000","last_unstale":"2024-06-29T14:37:54.377063+0000","last_undegraded":"2024-06-29T14:37:54.377063+0000","last_fullsized":"2024-06-29T14:37:54.377063+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:08.626526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397379+0000","last_change":"2024-06-29T14:37:53.411060+0000","last_active":"2024-06-29T14:37:54.397379+0000","last_peered":"2024-06-29T14:37:54.397379+0000","last_clean":"2024-06-29T14:37:54.397379+0000","last_became_active":"2024-06-29T14:37:53.405510+0000","last_became_peered":"2024-06-29T14:37:53.405510+0000","last_unstale":"2024-06-29T14:37:54.397379+0000","last_undegraded":"2024-06-29T14:37:54.397379+0000","last_fullsized":"2024-06-29T14:37:54.397379+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:08.171838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500680+0000","last_change":"2024-06-29T14:37:53.402011+0000","last_active":"2024-06-29T14:37:54.500680+0000","last_peered":"2024-06-29T14:37:54.500680+0000","last_clean":"2024-06-29T14:37:54.500680+0000","last_became_active":"2024-06-29T14:37:53.401641+0000","last_became_peered":"2024-06-29T14:37:53.401641+0000","last_unstale":"2024-06-29T14:37:54.500680+0000","last_undegraded":"2024-06-29T14:37:54.500680+0000","last_fullsized":"2024-06-29T14:37:54.500680+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:13:24.710230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501264+0000","last_change":"2024-06-29T14:37:53.388955+0000","last_active":"2024-06-29T14:37:54.501264+0000","last_peered":"2024-06-29T14:37:54.501264+0000","last_clean":"2024-06-29T14:37:54.501264+0000","last_became_active":"2024-06-29T14:37:53.388227+0000","last_became_peered":"2024-06-29T14:37:53.388227+0000","last_unstale":"2024-06-29T14:37:54.501264+0000","last_undegraded":"2024-06-29T14:37:54.501264+0000","last_fullsized":"2024-06-29T14:37:54.501264+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:01.383994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399472+0000","last_change":"2024-06-29T14:37:53.388122+0000","last_active":"2024-06-29T14:37:54.399472+0000","last_peered":"2024-06-29T14:37:54.399472+0000","last_clean":"2024-06-29T14:37:54.399472+0000","last_became_active":"2024-06-29T14:37:53.387220+0000","last_became_peered":"2024-06-29T14:37:53.387220+0000","last_unstale":"2024-06-29T14:37:54.399472+0000","last_undegraded":"2024-06-29T14:37:54.399472+0000","last_fullsized":"2024-06-29T14:37:54.399472+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:23.754075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373708+0000","last_change":"2024-06-29T14:37:53.396720+0000","last_active":"2024-06-29T14:37:54.373708+0000","last_peered":"2024-06-29T14:37:54.373708+0000","last_clean":"2024-06-29T14:37:54.373708+0000","last_became_active":"2024-06-29T14:37:53.396035+0000","last_became_peered":"2024-06-29T14:37:53.396035+0000","last_unstale":"2024-06-29T14:37:54.373708+0000","last_undegraded":"2024-06-29T14:37:54.373708+0000","last_fullsized":"2024-06-29T14:37:54.373708+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:39.642171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377171+0000","last_change":"2024-06-29T14:37:53.378369+0000","last_active":"2024-06-29T14:37:54.377171+0000","last_peered":"2024-06-29T14:37:54.377171+0000","last_clean":"2024-06-29T14:37:54.377171+0000","last_became_active":"2024-06-29T14:37:53.377909+0000","last_became_peered":"2024-06-29T14:37:53.377909+0000","last_unstale":"2024-06-29T14:37:54.377171+0000","last_undegraded":"2024-06-29T14:37:54.377171+0000","last_fullsized":"2024-06-29T14:37:54.377171+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:37:40.218294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399269+0000","last_change":"2024-06-29T14:37:53.397911+0000","last_active":"2024-06-29T14:37:54.399269+0000","last_peered":"2024-06-29T14:37:54.399269+0000","last_clean":"2024-06-29T14:37:54.399269+0000","last_became_active":"2024-06-29T14:37:53.397535+0000","last_became_peered":"2024-06-29T14:37:53.397535+0000","last_unstale":"2024-06-29T14:37:54.399269+0000","last_undegraded":"2024-06-29T14:37:54.399269+0000","last_fullsized":"2024-06-29T14:37:54.399269+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:23.062165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502139+0000","last_change":"2024-06-29T14:37:53.388967+0000","last_active":"2024-06-29T14:37:54.502139+0000","last_peered":"2024-06-29T14:37:54.502139+0000","last_clean":"2024-06-29T14:37:54.502139+0000","last_became_active":"2024-06-29T14:37:53.388657+0000","last_became_peered":"2024-06-29T14:37:53.388657+0000","last_unstale":"2024-06-29T14:37:54.502139+0000","last_undegraded":"2024-06-29T14:37:54.502139+0000","last_fullsized":"2024-06-29T14:37:54.502139+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:26.121311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501904+0000","last_change":"2024-06-29T14:37:53.387077+0000","last_active":"2024-06-29T14:37:54.501904+0000","last_peered":"2024-06-29T14:37:54.501904+0000","last_clean":"2024-06-29T14:37:54.501904+0000","last_became_active":"2024-06-29T14:37:53.386679+0000","last_became_peered":"2024-06-29T14:37:53.386679+0000","last_unstale":"2024-06-29T14:37:54.501904+0000","last_undegraded":"2024-06-29T14:37:54.501904+0000","last_fullsized":"2024-06-29T14:37:54.501904+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:28.890236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397746+0000","last_change":"2024-06-29T14:37:53.410583+0000","last_active":"2024-06-29T14:37:54.397746+0000","last_peered":"2024-06-29T14:37:54.397746+0000","last_clean":"2024-06-29T14:37:54.397746+0000","last_became_active":"2024-06-29T14:37:53.407764+0000","last_became_peered":"2024-06-29T14:37:53.407764+0000","last_unstale":"2024-06-29T14:37:54.397746+0000","last_undegraded":"2024-06-29T14:37:54.397746+0000","last_fullsized":"2024-06-29T14:37:54.397746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:13.215024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501459+0000","last_change":"2024-06-29T14:37:53.402577+0000","last_active":"2024-06-29T14:37:54.501459+0000","last_peered":"2024-06-29T14:37:54.501459+0000","last_clean":"2024-06-29T14:37:54.501459+0000","last_became_active":"2024-06-29T14:37:53.402232+0000","last_became_peered":"2024-06-29T14:37:53.402232+0000","last_unstale":"2024-06-29T14:37:54.501459+0000","last_undegraded":"2024-06-29T14:37:54.501459+0000","last_fullsized":"2024-06-29T14:37:54.501459+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:57.543562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399576+0000","last_change":"2024-06-29T14:37:53.388501+0000","last_active":"2024-06-29T14:37:54.399576+0000","last_peered":"2024-06-29T14:37:54.399576+0000","last_clean":"2024-06-29T14:37:54.399576+0000","last_became_active":"2024-06-29T14:37:53.387644+0000","last_became_peered":"2024-06-29T14:37:53.387644+0000","last_unstale":"2024-06-29T14:37:54.399576+0000","last_undegraded":"2024-06-29T14:37:54.399576+0000","last_fullsized":"2024-06-29T14:37:54.399576+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:45:48.889699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373728+0000","last_change":"2024-06-29T14:37:53.396397+0000","last_active":"2024-06-29T14:37:54.373728+0000","last_peered":"2024-06-29T14:37:54.373728+0000","last_clean":"2024-06-29T14:37:54.373728+0000","last_became_active":"2024-06-29T14:37:53.395459+0000","last_became_peered":"2024-06-29T14:37:53.395459+0000","last_unstale":"2024-06-29T14:37:54.373728+0000","last_undegraded":"2024-06-29T14:37:54.373728+0000","last_fullsized":"2024-06-29T14:37:54.373728+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:58.497717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397040+0000","last_change":"2024-06-29T14:37:53.396528+0000","last_active":"2024-06-29T14:37:54.397040+0000","last_peered":"2024-06-29T14:37:54.397040+0000","last_clean":"2024-06-29T14:37:54.397040+0000","last_became_active":"2024-06-29T14:37:53.395999+0000","last_became_peered":"2024-06-29T14:37:53.395999+0000","last_unstale":"2024-06-29T14:37:54.397040+0000","last_undegraded":"2024-06-29T14:37:54.397040+0000","last_fullsized":"2024-06-29T14:37:54.397040+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:56.219035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377296+0000","last_change":"2024-06-29T14:37:53.387998+0000","last_active":"2024-06-29T14:37:54.377296+0000","last_peered":"2024-06-29T14:37:54.377296+0000","last_clean":"2024-06-29T14:37:54.377296+0000","last_became_active":"2024-06-29T14:37:53.387673+0000","last_became_peered":"2024-06-29T14:37:53.387673+0000","last_unstale":"2024-06-29T14:37:54.377296+0000","last_undegraded":"2024-06-29T14:37:54.377296+0000","last_fullsized":"2024-06-29T14:37:54.377296+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:37.919672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399693+0000","last_change":"2024-06-29T14:37:53.386517+0000","last_active":"2024-06-29T14:37:54.399693+0000","last_peered":"2024-06-29T14:37:54.399693+0000","last_clean":"2024-06-29T14:37:54.399693+0000","last_became_active":"2024-06-29T14:37:53.385987+0000","last_became_peered":"2024-06-29T14:37:53.385987+0000","last_unstale":"2024-06-29T14:37:54.399693+0000","last_undegraded":"2024-06-29T14:37:54.399693+0000","last_fullsized":"2024-06-29T14:37:54.399693+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:00.009406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500722+0000","last_change":"2024-06-29T14:37:53.396666+0000","last_active":"2024-06-29T14:37:54.500722+0000","last_peered":"2024-06-29T14:37:54.500722+0000","last_clean":"2024-06-29T14:37:54.500722+0000","last_became_active":"2024-06-29T14:37:53.396236+0000","last_became_peered":"2024-06-29T14:37:53.396236+0000","last_unstale":"2024-06-29T14:37:54.500722+0000","last_undegraded":"2024-06-29T14:37:54.500722+0000","last_fullsized":"2024-06-29T14:37:54.500722+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:04.070850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373655+0000","last_change":"2024-06-29T14:37:53.383889+0000","last_active":"2024-06-29T14:37:54.373655+0000","last_peered":"2024-06-29T14:37:54.373655+0000","last_clean":"2024-06-29T14:37:54.373655+0000","last_became_active":"2024-06-29T14:37:53.383550+0000","last_became_peered":"2024-06-29T14:37:53.383550+0000","last_unstale":"2024-06-29T14:37:54.373655+0000","last_undegraded":"2024-06-29T14:37:54.373655+0000","last_fullsized":"2024-06-29T14:37:54.373655+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:35.647625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399718+0000","last_change":"2024-06-29T14:37:53.386428+0000","last_active":"2024-06-29T14:37:54.399718+0000","last_peered":"2024-06-29T14:37:54.399718+0000","last_clean":"2024-06-29T14:37:54.399718+0000","last_became_active":"2024-06-29T14:37:53.386068+0000","last_became_peered":"2024-06-29T14:37:53.386068+0000","last_unstale":"2024-06-29T14:37:54.399718+0000","last_undegraded":"2024-06-29T14:37:54.399718+0000","last_fullsized":"2024-06-29T14:37:54.399718+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:45.134932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501658+0000","last_change":"2024-06-29T14:37:53.386340+0000","last_active":"2024-06-29T14:37:54.501658+0000","last_peered":"2024-06-29T14:37:54.501658+0000","last_clean":"2024-06-29T14:37:54.501658+0000","last_became_active":"2024-06-29T14:37:53.385945+0000","last_became_peered":"2024-06-29T14:37:53.385945+0000","last_unstale":"2024-06-29T14:37:54.501658+0000","last_undegraded":"2024-06-29T14:37:54.501658+0000","last_fullsized":"2024-06-29T14:37:54.501658+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:31.302377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399650+0000","last_change":"2024-06-29T14:37:53.389244+0000","last_active":"2024-06-29T14:37:54.399650+0000","last_peered":"2024-06-29T14:37:54.399650+0000","last_clean":"2024-06-29T14:37:54.399650+0000","last_became_active":"2024-06-29T14:37:53.388900+0000","last_became_peered":"2024-06-29T14:37:53.388900+0000","last_unstale":"2024-06-29T14:37:54.399650+0000","last_undegraded":"2024-06-29T14:37:54.399650+0000","last_fullsized":"2024-06-29T14:37:54.399650+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:04:55.939413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501341+0000","last_change":"2024-06-29T14:37:53.381911+0000","last_active":"2024-06-29T14:37:54.501341+0000","last_peered":"2024-06-29T14:37:54.501341+0000","last_clean":"2024-06-29T14:37:54.501341+0000","last_became_active":"2024-06-29T14:37:53.380888+0000","last_became_peered":"2024-06-29T14:37:53.380888+0000","last_unstale":"2024-06-29T14:37:54.501341+0000","last_undegraded":"2024-06-29T14:37:54.501341+0000","last_fullsized":"2024-06-29T14:37:54.501341+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:53.579615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397023+0000","last_change":"2024-06-29T14:37:53.399470+0000","last_active":"2024-06-29T14:37:54.397023+0000","last_peered":"2024-06-29T14:37:54.397023+0000","last_clean":"2024-06-29T14:37:54.397023+0000","last_became_active":"2024-06-29T14:37:53.399121+0000","last_became_peered":"2024-06-29T14:37:53.399121+0000","last_unstale":"2024-06-29T14:37:54.397023+0000","last_undegraded":"2024-06-29T14:37:54.397023+0000","last_fullsized":"2024-06-29T14:37:54.397023+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:07:03.251164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397499+0000","last_change":"2024-06-29T14:37:53.404979+0000","last_active":"2024-06-29T14:37:54.397499+0000","last_peered":"2024-06-29T14:37:54.397499+0000","last_clean":"2024-06-29T14:37:54.397499+0000","last_became_active":"2024-06-29T14:37:53.404637+0000","last_became_peered":"2024-06-29T14:37:53.404637+0000","last_unstale":"2024-06-29T14:37:54.397499+0000","last_undegraded":"2024-06-29T14:37:54.397499+0000","last_fullsized":"2024-06-29T14:37:54.397499+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:45:31.098895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501815+0000","last_change":"2024-06-29T14:37:53.403833+0000","last_active":"2024-06-29T14:37:54.501815+0000","last_peered":"2024-06-29T14:37:54.501815+0000","last_clean":"2024-06-29T14:37:54.501815+0000","last_became_active":"2024-06-29T14:37:53.403521+0000","last_became_peered":"2024-06-29T14:37:53.403521+0000","last_unstale":"2024-06-29T14:37:54.501815+0000","last_undegraded":"2024-06-29T14:37:54.501815+0000","last_fullsized":"2024-06-29T14:37:54.501815+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:12.286797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377237+0000","last_change":"2024-06-29T14:37:53.388693+0000","last_active":"2024-06-29T14:37:54.377237+0000","last_peered":"2024-06-29T14:37:54.377237+0000","last_clean":"2024-06-29T14:37:54.377237+0000","last_became_active":"2024-06-29T14:37:53.388365+0000","last_became_peered":"2024-06-29T14:37:53.388365+0000","last_unstale":"2024-06-29T14:37:54.377237+0000","last_undegraded":"2024-06-29T14:37:54.377237+0000","last_fullsized":"2024-06-29T14:37:54.377237+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:19.201197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399903+0000","last_change":"2024-06-29T14:37:53.384862+0000","last_active":"2024-06-29T14:37:54.399903+0000","last_peered":"2024-06-29T14:37:54.399903+0000","last_clean":"2024-06-29T14:37:54.399903+0000","last_became_active":"2024-06-29T14:37:53.384501+0000","last_became_peered":"2024-06-29T14:37:53.384501+0000","last_unstale":"2024-06-29T14:37:54.399903+0000","last_undegraded":"2024-06-29T14:37:54.399903+0000","last_fullsized":"2024-06-29T14:37:54.399903+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:18:54.178493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397587+0000","last_change":"2024-06-29T14:37:53.401791+0000","last_active":"2024-06-29T14:37:54.397587+0000","last_peered":"2024-06-29T14:37:54.397587+0000","last_clean":"2024-06-29T14:37:54.397587+0000","last_became_active":"2024-06-29T14:37:53.401016+0000","last_became_peered":"2024-06-29T14:37:53.401016+0000","last_unstale":"2024-06-29T14:37:54.397587+0000","last_undegraded":"2024-06-29T14:37:54.397587+0000","last_fullsized":"2024-06-29T14:37:54.397587+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:52.890466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376682+0000","last_change":"2024-06-29T14:37:53.400832+0000","last_active":"2024-06-29T14:37:54.376682+0000","last_peered":"2024-06-29T14:37:54.376682+0000","last_clean":"2024-06-29T14:37:54.376682+0000","last_became_active":"2024-06-29T14:37:53.400455+0000","last_became_peered":"2024-06-29T14:37:53.400455+0000","last_unstale":"2024-06-29T14:37:54.376682+0000","last_undegraded":"2024-06-29T14:37:54.376682+0000","last_fullsized":"2024-06-29T14:37:54.376682+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:45.134205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500478+0000","last_change":"2024-06-29T14:37:53.388185+0000","last_active":"2024-06-29T14:37:54.500478+0000","last_peered":"2024-06-29T14:37:54.500478+0000","last_clean":"2024-06-29T14:37:54.500478+0000","last_became_active":"2024-06-29T14:37:53.387840+0000","last_became_peered":"2024-06-29T14:37:53.387840+0000","last_unstale":"2024-06-29T14:37:54.500478+0000","last_undegraded":"2024-06-29T14:37:54.500478+0000","last_fullsized":"2024-06-29T14:37:54.500478+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:55.568117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376855+0000","last_change":"2024-06-29T14:37:53.394441+0000","last_active":"2024-06-29T14:37:54.376855+0000","last_peered":"2024-06-29T14:37:54.376855+0000","last_clean":"2024-06-29T14:37:54.376855+0000","last_became_active":"2024-06-29T14:37:53.394112+0000","last_became_peered":"2024-06-29T14:37:53.394112+0000","last_unstale":"2024-06-29T14:37:54.376855+0000","last_undegraded":"2024-06-29T14:37:54.376855+0000","last_fullsized":"2024-06-29T14:37:54.376855+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:20.364846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500444+0000","last_change":"2024-06-29T14:37:53.398460+0000","last_active":"2024-06-29T14:37:54.500444+0000","last_peered":"2024-06-29T14:37:54.500444+0000","last_clean":"2024-06-29T14:37:54.500444+0000","last_became_active":"2024-06-29T14:37:53.398123+0000","last_became_peered":"2024-06-29T14:37:53.398123+0000","last_unstale":"2024-06-29T14:37:54.500444+0000","last_undegraded":"2024-06-29T14:37:54.500444+0000","last_fullsized":"2024-06-29T14:37:54.500444+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:04.069782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"100'4","reported_seq":18,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399724+0000","last_change":"2024-06-29T14:37:52.371901+0000","last_active":"2024-06-29T14:37:54.399724+0000","last_peered":"2024-06-29T14:37:54.399724+0000","last_clean":"2024-06-29T14:37:54.399724+0000","last_became_active":"2024-06-29T14:37:52.371603+0000","last_became_peered":"2024-06-29T14:37:52.371603+0000","last_unstale":"2024-06-29T14:37:54.399724+0000","last_undegraded":"2024-06-29T14:37:54.399724+0000","last_fullsized":"2024-06-29T14:37:54.399724+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:40.218022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397115+0000","last_change":"2024-06-29T14:37:53.400195+0000","last_active":"2024-06-29T14:37:54.397115+0000","last_peered":"2024-06-29T14:37:54.397115+0000","last_clean":"2024-06-29T14:37:54.397115+0000","last_became_active":"2024-06-29T14:37:53.399863+0000","last_became_peered":"2024-06-29T14:37:53.399863+0000","last_unstale":"2024-06-29T14:37:54.397115+0000","last_undegraded":"2024-06-29T14:37:54.397115+0000","last_fullsized":"2024-06-29T14:37:54.397115+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:44.577736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377805+0000","last_change":"2024-06-29T14:37:52.374061+0000","last_active":"2024-06-29T14:37:54.377805+0000","last_peered":"2024-06-29T14:37:54.377805+0000","last_clean":"2024-06-29T14:37:54.377805+0000","last_became_active":"2024-06-29T14:37:52.373637+0000","last_became_peered":"2024-06-29T14:37:52.373637+0000","last_unstale":"2024-06-29T14:37:54.377805+0000","last_undegraded":"2024-06-29T14:37:54.377805+0000","last_fullsized":"2024-06-29T14:37:54.377805+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:04:55.861797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501537+0000","last_change":"2024-06-29T14:37:53.391600+0000","last_active":"2024-06-29T14:37:54.501537+0000","last_peered":"2024-06-29T14:37:54.501537+0000","last_clean":"2024-06-29T14:37:54.501537+0000","last_became_active":"2024-06-29T14:37:53.391265+0000","last_became_peered":"2024-06-29T14:37:53.391265+0000","last_unstale":"2024-06-29T14:37:54.501537+0000","last_undegraded":"2024-06-29T14:37:54.501537+0000","last_fullsized":"2024-06-29T14:37:54.501537+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:18:49.439460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377370+0000","last_change":"2024-06-29T14:37:52.379761+0000","last_active":"2024-06-29T14:37:54.377370+0000","last_peered":"2024-06-29T14:37:54.377370+0000","last_clean":"2024-06-29T14:37:54.377370+0000","last_became_active":"2024-06-29T14:37:52.379541+0000","last_became_peered":"2024-06-29T14:37:52.379541+0000","last_unstale":"2024-06-29T14:37:54.377370+0000","last_undegraded":"2024-06-29T14:37:54.377370+0000","last_fullsized":"2024-06-29T14:37:54.377370+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:21.142171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502051+0000","last_change":"2024-06-29T14:37:53.387813+0000","last_active":"2024-06-29T14:37:54.502051+0000","last_peered":"2024-06-29T14:37:54.502051+0000","last_clean":"2024-06-29T14:37:54.502051+0000","last_became_active":"2024-06-29T14:37:53.385790+0000","last_became_peered":"2024-06-29T14:37:53.385790+0000","last_unstale":"2024-06-29T14:37:54.502051+0000","last_undegraded":"2024-06-29T14:37:54.502051+0000","last_fullsized":"2024-06-29T14:37:54.502051+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:55.092845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378087+0000","last_change":"2024-06-29T14:37:52.370965+0000","last_active":"2024-06-29T14:37:54.378087+0000","last_peered":"2024-06-29T14:37:54.378087+0000","last_clean":"2024-06-29T14:37:54.378087+0000","last_became_active":"2024-06-29T14:37:52.370661+0000","last_became_peered":"2024-06-29T14:37:52.370661+0000","last_unstale":"2024-06-29T14:37:54.378087+0000","last_undegraded":"2024-06-29T14:37:54.378087+0000","last_fullsized":"2024-06-29T14:37:54.378087+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:00:15.689139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397348+0000","last_change":"2024-06-29T14:37:53.410454+0000","last_active":"2024-06-29T14:37:54.397348+0000","last_peered":"2024-06-29T14:37:54.397348+0000","last_clean":"2024-06-29T14:37:54.397348+0000","last_became_active":"2024-06-29T14:37:53.406280+0000","last_became_peered":"2024-06-29T14:37:53.406280+0000","last_unstale":"2024-06-29T14:37:54.397348+0000","last_undegraded":"2024-06-29T14:37:54.397348+0000","last_fullsized":"2024-06-29T14:37:54.397348+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.825289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501802+0000","last_change":"2024-06-29T14:37:52.369426+0000","last_active":"2024-06-29T14:37:54.501802+0000","last_peered":"2024-06-29T14:37:54.501802+0000","last_clean":"2024-06-29T14:37:54.501802+0000","last_became_active":"2024-06-29T14:37:52.369068+0000","last_became_peered":"2024-06-29T14:37:52.369068+0000","last_unstale":"2024-06-29T14:37:54.501802+0000","last_undegraded":"2024-06-29T14:37:54.501802+0000","last_fullsized":"2024-06-29T14:37:54.501802+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:00:15.689139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397708+0000","last_change":"2024-06-29T14:37:53.407498+0000","last_active":"2024-06-29T14:37:54.397708+0000","last_peered":"2024-06-29T14:37:54.397708+0000","last_clean":"2024-06-29T14:37:54.397708+0000","last_became_active":"2024-06-29T14:37:53.406972+0000","last_became_peered":"2024-06-29T14:37:53.406972+0000","last_unstale":"2024-06-29T14:37:54.397708+0000","last_undegraded":"2024-06-29T14:37:54.397708+0000","last_fullsized":"2024-06-29T14:37:54.397708+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:24.170491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500934+0000","last_change":"2024-06-29T14:37:52.371987+0000","last_active":"2024-06-29T14:37:54.500934+0000","last_peered":"2024-06-29T14:37:54.500934+0000","last_clean":"2024-06-29T14:37:54.500934+0000","last_became_active":"2024-06-29T14:37:52.371575+0000","last_became_peered":"2024-06-29T14:37:52.371575+0000","last_unstale":"2024-06-29T14:37:54.500934+0000","last_undegraded":"2024-06-29T14:37:54.500934+0000","last_fullsized":"2024-06-29T14:37:54.500934+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:25.267941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502152+0000","last_change":"2024-06-29T14:37:53.388309+0000","last_active":"2024-06-29T14:37:54.502152+0000","last_peered":"2024-06-29T14:37:54.502152+0000","last_clean":"2024-06-29T14:37:54.502152+0000","last_became_active":"2024-06-29T14:37:53.388004+0000","last_became_peered":"2024-06-29T14:37:53.388004+0000","last_unstale":"2024-06-29T14:37:54.502152+0000","last_undegraded":"2024-06-29T14:37:54.502152+0000","last_fullsized":"2024-06-29T14:37:54.502152+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:25.689885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397160+0000","last_change":"2024-06-29T14:37:52.380053+0000","last_active":"2024-06-29T14:37:54.397160+0000","last_peered":"2024-06-29T14:37:54.397160+0000","last_clean":"2024-06-29T14:37:54.397160+0000","last_became_active":"2024-06-29T14:37:52.379670+0000","last_became_peered":"2024-06-29T14:37:52.379670+0000","last_unstale":"2024-06-29T14:37:54.397160+0000","last_undegraded":"2024-06-29T14:37:54.397160+0000","last_fullsized":"2024-06-29T14:37:54.397160+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:38.063079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377833+0000","last_change":"2024-06-29T14:37:53.381393+0000","last_active":"2024-06-29T14:37:54.377833+0000","last_peered":"2024-06-29T14:37:54.377833+0000","last_clean":"2024-06-29T14:37:54.377833+0000","last_became_active":"2024-06-29T14:37:53.381010+0000","last_became_peered":"2024-06-29T14:37:53.381010+0000","last_unstale":"2024-06-29T14:37:54.377833+0000","last_undegraded":"2024-06-29T14:37:54.377833+0000","last_fullsized":"2024-06-29T14:37:54.377833+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:29.394763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399337+0000","last_change":"2024-06-29T14:37:52.367630+0000","last_active":"2024-06-29T14:37:54.399337+0000","last_peered":"2024-06-29T14:37:54.399337+0000","last_clean":"2024-06-29T14:37:54.399337+0000","last_became_active":"2024-06-29T14:37:52.366598+0000","last_became_peered":"2024-06-29T14:37:52.366598+0000","last_unstale":"2024-06-29T14:37:54.399337+0000","last_undegraded":"2024-06-29T14:37:54.399337+0000","last_fullsized":"2024-06-29T14:37:54.399337+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:43.687013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397026+0000","last_change":"2024-06-29T14:37:53.398609+0000","last_active":"2024-06-29T14:37:54.397026+0000","last_peered":"2024-06-29T14:37:54.397026+0000","last_clean":"2024-06-29T14:37:54.397026+0000","last_became_active":"2024-06-29T14:37:53.398271+0000","last_became_peered":"2024-06-29T14:37:53.398271+0000","last_unstale":"2024-06-29T14:37:54.397026+0000","last_undegraded":"2024-06-29T14:37:54.397026+0000","last_fullsized":"2024-06-29T14:37:54.397026+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:23.695035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377247+0000","last_change":"2024-06-29T14:37:52.374251+0000","last_active":"2024-06-29T14:37:54.377247+0000","last_peered":"2024-06-29T14:37:54.377247+0000","last_clean":"2024-06-29T14:37:54.377247+0000","last_became_active":"2024-06-29T14:37:52.373551+0000","last_became_peered":"2024-06-29T14:37:52.373551+0000","last_unstale":"2024-06-29T14:37:54.377247+0000","last_undegraded":"2024-06-29T14:37:54.377247+0000","last_fullsized":"2024-06-29T14:37:54.377247+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:09.510075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501126+0000","last_change":"2024-06-29T14:37:53.383525+0000","last_active":"2024-06-29T14:37:54.501126+0000","last_peered":"2024-06-29T14:37:54.501126+0000","last_clean":"2024-06-29T14:37:54.501126+0000","last_became_active":"2024-06-29T14:37:53.383206+0000","last_became_peered":"2024-06-29T14:37:53.383206+0000","last_unstale":"2024-06-29T14:37:54.501126+0000","last_undegraded":"2024-06-29T14:37:54.501126+0000","last_fullsized":"2024-06-29T14:37:54.501126+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:39.705828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501966+0000","last_change":"2024-06-29T14:37:52.368842+0000","last_active":"2024-06-29T14:37:54.501966+0000","last_peered":"2024-06-29T14:37:54.501966+0000","last_clean":"2024-06-29T14:37:54.501966+0000","last_became_active":"2024-06-29T14:37:52.368528+0000","last_became_peered":"2024-06-29T14:37:52.368528+0000","last_unstale":"2024-06-29T14:37:54.501966+0000","last_undegraded":"2024-06-29T14:37:54.501966+0000","last_fullsized":"2024-06-29T14:37:54.501966+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:26:22.971432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373889+0000","last_change":"2024-06-29T14:37:53.387662+0000","last_active":"2024-06-29T14:37:54.373889+0000","last_peered":"2024-06-29T14:37:54.373889+0000","last_clean":"2024-06-29T14:37:54.373889+0000","last_became_active":"2024-06-29T14:37:53.387030+0000","last_became_peered":"2024-06-29T14:37:53.387030+0000","last_unstale":"2024-06-29T14:37:54.373889+0000","last_undegraded":"2024-06-29T14:37:54.373889+0000","last_fullsized":"2024-06-29T14:37:54.373889+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:30:18.362323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"100'3","reported_seq":17,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399852+0000","last_change":"2024-06-29T14:37:52.368996+0000","last_active":"2024-06-29T14:37:54.399852+0000","last_peered":"2024-06-29T14:37:54.399852+0000","last_clean":"2024-06-29T14:37:54.399852+0000","last_became_active":"2024-06-29T14:37:52.368587+0000","last_became_peered":"2024-06-29T14:37:52.368587+0000","last_unstale":"2024-06-29T14:37:54.399852+0000","last_undegraded":"2024-06-29T14:37:54.399852+0000","last_fullsized":"2024-06-29T14:37:54.399852+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:20.857673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500483+0000","last_change":"2024-06-29T14:37:53.405485+0000","last_active":"2024-06-29T14:37:54.500483+0000","last_peered":"2024-06-29T14:37:54.500483+0000","last_clean":"2024-06-29T14:37:54.500483+0000","last_became_active":"2024-06-29T14:37:53.404742+0000","last_became_peered":"2024-06-29T14:37:53.404742+0000","last_unstale":"2024-06-29T14:37:54.500483+0000","last_undegraded":"2024-06-29T14:37:54.500483+0000","last_fullsized":"2024-06-29T14:37:54.500483+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:51.681296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"100'6","reported_seq":20,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378182+0000","last_change":"2024-06-29T14:37:52.369312+0000","last_active":"2024-06-29T14:37:54.378182+0000","last_peered":"2024-06-29T14:37:54.378182+0000","last_clean":"2024-06-29T14:37:54.378182+0000","last_became_active":"2024-06-29T14:37:52.368394+0000","last_became_peered":"2024-06-29T14:37:52.368394+0000","last_unstale":"2024-06-29T14:37:54.378182+0000","last_undegraded":"2024-06-29T14:37:54.378182+0000","last_fullsized":"2024-06-29T14:37:54.378182+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:08.856778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501768+0000","last_change":"2024-06-29T14:37:53.389221+0000","last_active":"2024-06-29T14:37:54.501768+0000","last_peered":"2024-06-29T14:37:54.501768+0000","last_clean":"2024-06-29T14:37:54.501768+0000","last_became_active":"2024-06-29T14:37:53.388659+0000","last_became_peered":"2024-06-29T14:37:53.388659+0000","last_unstale":"2024-06-29T14:37:54.501768+0000","last_undegraded":"2024-06-29T14:37:54.501768+0000","last_fullsized":"2024-06-29T14:37:54.501768+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:18.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500948+0000","last_change":"2024-06-29T14:37:52.368116+0000","last_active":"2024-06-29T14:37:54.500948+0000","last_peered":"2024-06-29T14:37:54.500948+0000","last_clean":"2024-06-29T14:37:54.500948+0000","last_became_active":"2024-06-29T14:37:52.367664+0000","last_became_peered":"2024-06-29T14:37:52.367664+0000","last_unstale":"2024-06-29T14:37:54.500948+0000","last_undegraded":"2024-06-29T14:37:54.500948+0000","last_fullsized":"2024-06-29T14:37:54.500948+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:38.764223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376993+0000","last_change":"2024-06-29T14:37:53.385089+0000","last_active":"2024-06-29T14:37:54.376993+0000","last_peered":"2024-06-29T14:37:54.376993+0000","last_clean":"2024-06-29T14:37:54.376993+0000","last_became_active":"2024-06-29T14:37:53.384776+0000","last_became_peered":"2024-06-29T14:37:53.384776+0000","last_unstale":"2024-06-29T14:37:54.376993+0000","last_undegraded":"2024-06-29T14:37:54.376993+0000","last_fullsized":"2024-06-29T14:37:54.376993+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:55.092845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376825+0000","last_change":"2024-06-29T14:37:52.377689+0000","last_active":"2024-06-29T14:37:54.376825+0000","last_peered":"2024-06-29T14:37:54.376825+0000","last_clean":"2024-06-29T14:37:54.376825+0000","last_became_active":"2024-06-29T14:37:52.377384+0000","last_became_peered":"2024-06-29T14:37:52.377384+0000","last_unstale":"2024-06-29T14:37:54.376825+0000","last_undegraded":"2024-06-29T14:37:54.376825+0000","last_fullsized":"2024-06-29T14:37:54.376825+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:05.602047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377549+0000","last_change":"2024-06-29T14:37:53.378494+0000","last_active":"2024-06-29T14:37:54.377549+0000","last_peered":"2024-06-29T14:37:54.377549+0000","last_clean":"2024-06-29T14:37:54.377549+0000","last_became_active":"2024-06-29T14:37:53.377998+0000","last_became_peered":"2024-06-29T14:37:53.377998+0000","last_unstale":"2024-06-29T14:37:54.377549+0000","last_undegraded":"2024-06-29T14:37:54.377549+0000","last_fullsized":"2024-06-29T14:37:54.377549+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:25.689885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399368+0000","last_change":"2024-06-29T14:37:52.371238+0000","last_active":"2024-06-29T14:37:54.399368+0000","last_peered":"2024-06-29T14:37:54.399368+0000","last_clean":"2024-06-29T14:37:54.399368+0000","last_became_active":"2024-06-29T14:37:52.370917+0000","last_became_peered":"2024-06-29T14:37:52.370917+0000","last_unstale":"2024-06-29T14:37:54.399368+0000","last_undegraded":"2024-06-29T14:37:54.399368+0000","last_fullsized":"2024-06-29T14:37:54.399368+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:15:39.995393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378052+0000","last_change":"2024-06-29T14:37:53.392895+0000","last_active":"2024-06-29T14:37:54.378052+0000","last_peered":"2024-06-29T14:37:54.378052+0000","last_clean":"2024-06-29T14:37:54.378052+0000","last_became_active":"2024-06-29T14:37:53.392585+0000","last_became_peered":"2024-06-29T14:37:53.392585+0000","last_unstale":"2024-06-29T14:37:54.378052+0000","last_undegraded":"2024-06-29T14:37:54.378052+0000","last_fullsized":"2024-06-29T14:37:54.378052+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:38.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397340+0000","last_change":"2024-06-29T14:37:52.377275+0000","last_active":"2024-06-29T14:37:54.397340+0000","last_peered":"2024-06-29T14:37:54.397340+0000","last_clean":"2024-06-29T14:37:54.397340+0000","last_became_active":"2024-06-29T14:37:52.376947+0000","last_became_peered":"2024-06-29T14:37:52.376947+0000","last_unstale":"2024-06-29T14:37:54.397340+0000","last_undegraded":"2024-06-29T14:37:54.397340+0000","last_fullsized":"2024-06-29T14:37:54.397340+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:34.220027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373314+0000","last_change":"2024-06-29T14:37:53.383101+0000","last_active":"2024-06-29T14:37:54.373314+0000","last_peered":"2024-06-29T14:37:54.373314+0000","last_clean":"2024-06-29T14:37:54.373314+0000","last_became_active":"2024-06-29T14:37:53.382775+0000","last_became_peered":"2024-06-29T14:37:53.382775+0000","last_unstale":"2024-06-29T14:37:54.373314+0000","last_undegraded":"2024-06-29T14:37:54.373314+0000","last_fullsized":"2024-06-29T14:37:54.373314+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:32.510147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377338+0000","last_change":"2024-06-29T14:37:52.372210+0000","last_active":"2024-06-29T14:37:54.377338+0000","last_peered":"2024-06-29T14:37:54.377338+0000","last_clean":"2024-06-29T14:37:54.377338+0000","last_became_active":"2024-06-29T14:37:52.371896+0000","last_became_peered":"2024-06-29T14:37:52.371896+0000","last_unstale":"2024-06-29T14:37:54.377338+0000","last_undegraded":"2024-06-29T14:37:54.377338+0000","last_fullsized":"2024-06-29T14:37:54.377338+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:07.184093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399385+0000","last_change":"2024-06-29T14:37:53.386600+0000","last_active":"2024-06-29T14:37:54.399385+0000","last_peered":"2024-06-29T14:37:54.399385+0000","last_clean":"2024-06-29T14:37:54.399385+0000","last_became_active":"2024-06-29T14:37:53.386248+0000","last_became_peered":"2024-06-29T14:37:53.386248+0000","last_unstale":"2024-06-29T14:37:54.399385+0000","last_undegraded":"2024-06-29T14:37:54.399385+0000","last_fullsized":"2024-06-29T14:37:54.399385+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:50.022396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501756+0000","last_change":"2024-06-29T14:37:52.364386+0000","last_active":"2024-06-29T14:37:54.501756+0000","last_peered":"2024-06-29T14:37:54.501756+0000","last_clean":"2024-06-29T14:37:54.501756+0000","last_became_active":"2024-06-29T14:37:52.363965+0000","last_became_peered":"2024-06-29T14:37:52.363965+0000","last_unstale":"2024-06-29T14:37:54.501756+0000","last_undegraded":"2024-06-29T14:37:54.501756+0000","last_fullsized":"2024-06-29T14:37:54.501756+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:58:22.236585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399856+0000","last_change":"2024-06-29T14:37:53.402834+0000","last_active":"2024-06-29T14:37:54.399856+0000","last_peered":"2024-06-29T14:37:54.399856+0000","last_clean":"2024-06-29T14:37:54.399856+0000","last_became_active":"2024-06-29T14:37:53.402513+0000","last_became_peered":"2024-06-29T14:37:53.402513+0000","last_unstale":"2024-06-29T14:37:54.399856+0000","last_undegraded":"2024-06-29T14:37:54.399856+0000","last_fullsized":"2024-06-29T14:37:54.399856+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:17.123778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399964+0000","last_change":"2024-06-29T14:37:52.365260+0000","last_active":"2024-06-29T14:37:54.399964+0000","last_peered":"2024-06-29T14:37:54.399964+0000","last_clean":"2024-06-29T14:37:54.399964+0000","last_became_active":"2024-06-29T14:37:52.364944+0000","last_became_peered":"2024-06-29T14:37:52.364944+0000","last_unstale":"2024-06-29T14:37:54.399964+0000","last_undegraded":"2024-06-29T14:37:54.399964+0000","last_fullsized":"2024-06-29T14:37:54.399964+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:07.824092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373601+0000","last_change":"2024-06-29T14:37:53.387309+0000","last_active":"2024-06-29T14:37:54.373601+0000","last_peered":"2024-06-29T14:37:54.373601+0000","last_clean":"2024-06-29T14:37:54.373601+0000","last_became_active":"2024-06-29T14:37:53.386992+0000","last_became_peered":"2024-06-29T14:37:53.386992+0000","last_unstale":"2024-06-29T14:37:54.373601+0000","last_undegraded":"2024-06-29T14:37:54.373601+0000","last_fullsized":"2024-06-29T14:37:54.373601+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:33:08.670834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500589+0000","last_change":"2024-06-29T14:37:52.377838+0000","last_active":"2024-06-29T14:37:54.500589+0000","last_peered":"2024-06-29T14:37:54.500589+0000","last_clean":"2024-06-29T14:37:54.500589+0000","last_became_active":"2024-06-29T14:37:52.377501+0000","last_became_peered":"2024-06-29T14:37:52.377501+0000","last_unstale":"2024-06-29T14:37:54.500589+0000","last_undegraded":"2024-06-29T14:37:54.500589+0000","last_fullsized":"2024-06-29T14:37:54.500589+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:36.851395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397379+0000","last_change":"2024-06-29T14:37:53.408012+0000","last_active":"2024-06-29T14:37:54.397379+0000","last_peered":"2024-06-29T14:37:54.397379+0000","last_clean":"2024-06-29T14:37:54.397379+0000","last_became_active":"2024-06-29T14:37:53.407646+0000","last_became_peered":"2024-06-29T14:37:53.407646+0000","last_unstale":"2024-06-29T14:37:54.397379+0000","last_undegraded":"2024-06-29T14:37:54.397379+0000","last_fullsized":"2024-06-29T14:37:54.397379+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:41:19.482180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373570+0000","last_change":"2024-06-29T14:37:52.368873+0000","last_active":"2024-06-29T14:37:54.373570+0000","last_peered":"2024-06-29T14:37:54.373570+0000","last_clean":"2024-06-29T14:37:54.373570+0000","last_became_active":"2024-06-29T14:37:52.368563+0000","last_became_peered":"2024-06-29T14:37:52.368563+0000","last_unstale":"2024-06-29T14:37:54.373570+0000","last_undegraded":"2024-06-29T14:37:54.373570+0000","last_fullsized":"2024-06-29T14:37:54.373570+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:54:28.883517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397423+0000","last_change":"2024-06-29T14:37:53.407244+0000","last_active":"2024-06-29T14:37:54.397423+0000","last_peered":"2024-06-29T14:37:54.397423+0000","last_clean":"2024-06-29T14:37:54.397423+0000","last_became_active":"2024-06-29T14:37:53.406909+0000","last_became_peered":"2024-06-29T14:37:53.406909+0000","last_unstale":"2024-06-29T14:37:54.397423+0000","last_undegraded":"2024-06-29T14:37:54.397423+0000","last_fullsized":"2024-06-29T14:37:54.397423+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:52:37.676943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373415+0000","last_change":"2024-06-29T14:37:52.374678+0000","last_active":"2024-06-29T14:37:54.373415+0000","last_peered":"2024-06-29T14:37:54.373415+0000","last_clean":"2024-06-29T14:37:54.373415+0000","last_became_active":"2024-06-29T14:37:52.374278+0000","last_became_peered":"2024-06-29T14:37:52.374278+0000","last_unstale":"2024-06-29T14:37:54.373415+0000","last_undegraded":"2024-06-29T14:37:54.373415+0000","last_fullsized":"2024-06-29T14:37:54.373415+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:58.929743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378079+0000","last_change":"2024-06-29T14:37:53.383274+0000","last_active":"2024-06-29T14:37:54.378079+0000","last_peered":"2024-06-29T14:37:54.378079+0000","last_clean":"2024-06-29T14:37:54.378079+0000","last_became_active":"2024-06-29T14:37:53.382901+0000","last_became_peered":"2024-06-29T14:37:53.382901+0000","last_unstale":"2024-06-29T14:37:54.378079+0000","last_undegraded":"2024-06-29T14:37:54.378079+0000","last_fullsized":"2024-06-29T14:37:54.378079+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:32.247587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373509+0000","last_change":"2024-06-29T14:37:52.363357+0000","last_active":"2024-06-29T14:37:54.373509+0000","last_peered":"2024-06-29T14:37:54.373509+0000","last_clean":"2024-06-29T14:37:54.373509+0000","last_became_active":"2024-06-29T14:37:52.363018+0000","last_became_peered":"2024-06-29T14:37:52.363018+0000","last_unstale":"2024-06-29T14:37:54.373509+0000","last_undegraded":"2024-06-29T14:37:54.373509+0000","last_fullsized":"2024-06-29T14:37:54.373509+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:02:36.554625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502068+0000","last_change":"2024-06-29T14:37:53.381661+0000","last_active":"2024-06-29T14:37:54.502068+0000","last_peered":"2024-06-29T14:37:54.502068+0000","last_clean":"2024-06-29T14:37:54.502068+0000","last_became_active":"2024-06-29T14:37:53.381367+0000","last_became_peered":"2024-06-29T14:37:53.381367+0000","last_unstale":"2024-06-29T14:37:54.502068+0000","last_undegraded":"2024-06-29T14:37:54.502068+0000","last_fullsized":"2024-06-29T14:37:54.502068+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:37:40.218294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501918+0000","last_change":"2024-06-29T14:37:52.365709+0000","last_active":"2024-06-29T14:37:54.501918+0000","last_peered":"2024-06-29T14:37:54.501918+0000","last_clean":"2024-06-29T14:37:54.501918+0000","last_became_active":"2024-06-29T14:37:52.365353+0000","last_became_peered":"2024-06-29T14:37:52.365353+0000","last_unstale":"2024-06-29T14:37:54.501918+0000","last_undegraded":"2024-06-29T14:37:54.501918+0000","last_fullsized":"2024-06-29T14:37:54.501918+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:19.432254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377337+0000","last_change":"2024-06-29T14:37:53.388153+0000","last_active":"2024-06-29T14:37:54.377337+0000","last_peered":"2024-06-29T14:37:54.377337+0000","last_clean":"2024-06-29T14:37:54.377337+0000","last_became_active":"2024-06-29T14:37:53.387848+0000","last_became_peered":"2024-06-29T14:37:53.387848+0000","last_unstale":"2024-06-29T14:37:54.377337+0000","last_undegraded":"2024-06-29T14:37:54.377337+0000","last_fullsized":"2024-06-29T14:37:54.377337+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:26.121311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"101'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.398690+0000","last_change":"2024-06-29T14:37:52.380126+0000","last_active":"2024-06-29T14:37:54.398690+0000","last_peered":"2024-06-29T14:37:54.398690+0000","last_clean":"2024-06-29T14:37:54.398690+0000","last_became_active":"2024-06-29T14:37:52.379705+0000","last_became_peered":"2024-06-29T14:37:52.379705+0000","last_unstale":"2024-06-29T14:37:54.398690+0000","last_undegraded":"2024-06-29T14:37:54.398690+0000","last_fullsized":"2024-06-29T14:37:54.398690+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:15.384107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502122+0000","last_change":"2024-06-29T14:37:53.386061+0000","last_active":"2024-06-29T14:37:54.502122+0000","last_peered":"2024-06-29T14:37:54.502122+0000","last_clean":"2024-06-29T14:37:54.502122+0000","last_became_active":"2024-06-29T14:37:53.385707+0000","last_became_peered":"2024-06-29T14:37:53.385707+0000","last_unstale":"2024-06-29T14:37:54.502122+0000","last_undegraded":"2024-06-29T14:37:54.502122+0000","last_fullsized":"2024-06-29T14:37:54.502122+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:01.014097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377655+0000","last_change":"2024-06-29T14:37:52.370103+0000","last_active":"2024-06-29T14:37:54.377655+0000","last_peered":"2024-06-29T14:37:54.377655+0000","last_clean":"2024-06-29T14:37:54.377655+0000","last_became_active":"2024-06-29T14:37:52.369784+0000","last_became_peered":"2024-06-29T14:37:52.369784+0000","last_unstale":"2024-06-29T14:37:54.377655+0000","last_undegraded":"2024-06-29T14:37:54.377655+0000","last_fullsized":"2024-06-29T14:37:54.377655+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:22:48.722841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397450+0000","last_change":"2024-06-29T14:37:53.405338+0000","last_active":"2024-06-29T14:37:54.397450+0000","last_peered":"2024-06-29T14:37:54.397450+0000","last_clean":"2024-06-29T14:37:54.397450+0000","last_became_active":"2024-06-29T14:37:53.405000+0000","last_became_peered":"2024-06-29T14:37:53.405000+0000","last_unstale":"2024-06-29T14:37:54.397450+0000","last_undegraded":"2024-06-29T14:37:54.397450+0000","last_fullsized":"2024-06-29T14:37:54.397450+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:45.965101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376932+0000","last_change":"2024-06-29T14:37:52.372455+0000","last_active":"2024-06-29T14:37:54.376932+0000","last_peered":"2024-06-29T14:37:54.376932+0000","last_clean":"2024-06-29T14:37:54.376932+0000","last_became_active":"2024-06-29T14:37:52.372119+0000","last_became_peered":"2024-06-29T14:37:52.372119+0000","last_unstale":"2024-06-29T14:37:54.376932+0000","last_undegraded":"2024-06-29T14:37:54.376932+0000","last_fullsized":"2024-06-29T14:37:54.376932+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:43.766024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378080+0000","last_change":"2024-06-29T14:37:53.384508+0000","last_active":"2024-06-29T14:37:54.378080+0000","last_peered":"2024-06-29T14:37:54.378080+0000","last_clean":"2024-06-29T14:37:54.378080+0000","last_became_active":"2024-06-29T14:37:53.384154+0000","last_became_peered":"2024-06-29T14:37:53.384154+0000","last_unstale":"2024-06-29T14:37:54.378080+0000","last_undegraded":"2024-06-29T14:37:54.378080+0000","last_fullsized":"2024-06-29T14:37:54.378080+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:34.922824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377614+0000","last_change":"2024-06-29T14:37:52.368381+0000","last_active":"2024-06-29T14:37:54.377614+0000","last_peered":"2024-06-29T14:37:54.377614+0000","last_clean":"2024-06-29T14:37:54.377614+0000","last_became_active":"2024-06-29T14:37:52.368071+0000","last_became_peered":"2024-06-29T14:37:52.368071+0000","last_unstale":"2024-06-29T14:37:54.377614+0000","last_undegraded":"2024-06-29T14:37:54.377614+0000","last_fullsized":"2024-06-29T14:37:54.377614+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:15:44.081382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377174+0000","last_change":"2024-06-29T14:37:53.380753+0000","last_active":"2024-06-29T14:37:54.377174+0000","last_peered":"2024-06-29T14:37:54.377174+0000","last_clean":"2024-06-29T14:37:54.377174+0000","last_became_active":"2024-06-29T14:37:53.380387+0000","last_became_peered":"2024-06-29T14:37:53.380387+0000","last_unstale":"2024-06-29T14:37:54.377174+0000","last_undegraded":"2024-06-29T14:37:54.377174+0000","last_fullsized":"2024-06-29T14:37:54.377174+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:01.014097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399877+0000","last_change":"2024-06-29T14:37:52.370239+0000","last_active":"2024-06-29T14:37:54.399877+0000","last_peered":"2024-06-29T14:37:54.399877+0000","last_clean":"2024-06-29T14:37:54.399877+0000","last_became_active":"2024-06-29T14:37:52.369936+0000","last_became_peered":"2024-06-29T14:37:52.369936+0000","last_unstale":"2024-06-29T14:37:54.399877+0000","last_undegraded":"2024-06-29T14:37:54.399877+0000","last_fullsized":"2024-06-29T14:37:54.399877+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:07.211445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500732+0000","last_change":"2024-06-29T14:37:53.406486+0000","last_active":"2024-06-29T14:37:54.500732+0000","last_peered":"2024-06-29T14:37:54.500732+0000","last_clean":"2024-06-29T14:37:54.500732+0000","last_became_active":"2024-06-29T14:37:53.406114+0000","last_became_peered":"2024-06-29T14:37:53.406114+0000","last_unstale":"2024-06-29T14:37:54.500732+0000","last_undegraded":"2024-06-29T14:37:54.500732+0000","last_fullsized":"2024-06-29T14:37:54.500732+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:11.537473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377447+0000","last_change":"2024-06-29T14:37:52.365139+0000","last_active":"2024-06-29T14:37:54.377447+0000","last_peered":"2024-06-29T14:37:54.377447+0000","last_clean":"2024-06-29T14:37:54.377447+0000","last_became_active":"2024-06-29T14:37:52.364711+0000","last_became_peered":"2024-06-29T14:37:52.364711+0000","last_unstale":"2024-06-29T14:37:54.377447+0000","last_undegraded":"2024-06-29T14:37:54.377447+0000","last_fullsized":"2024-06-29T14:37:54.377447+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:58:22.236585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500826+0000","last_change":"2024-06-29T14:37:53.392368+0000","last_active":"2024-06-29T14:37:54.500826+0000","last_peered":"2024-06-29T14:37:54.500826+0000","last_clean":"2024-06-29T14:37:54.500826+0000","last_became_active":"2024-06-29T14:37:53.392022+0000","last_became_peered":"2024-06-29T14:37:53.392022+0000","last_unstale":"2024-06-29T14:37:54.500826+0000","last_undegraded":"2024-06-29T14:37:54.500826+0000","last_fullsized":"2024-06-29T14:37:54.500826+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:41:21.804940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500365+0000","last_change":"2024-06-29T14:37:52.373923+0000","last_active":"2024-06-29T14:37:54.500365+0000","last_peered":"2024-06-29T14:37:54.500365+0000","last_clean":"2024-06-29T14:37:54.500365+0000","last_became_active":"2024-06-29T14:37:52.373594+0000","last_became_peered":"2024-06-29T14:37:52.373594+0000","last_unstale":"2024-06-29T14:37:54.500365+0000","last_undegraded":"2024-06-29T14:37:54.500365+0000","last_fullsized":"2024-06-29T14:37:54.500365+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:27.635671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377538+0000","last_change":"2024-06-29T14:37:53.391960+0000","last_active":"2024-06-29T14:37:54.377538+0000","last_peered":"2024-06-29T14:37:54.377538+0000","last_clean":"2024-06-29T14:37:54.377538+0000","last_became_active":"2024-06-29T14:37:53.391629+0000","last_became_peered":"2024-06-29T14:37:53.391629+0000","last_unstale":"2024-06-29T14:37:54.377538+0000","last_undegraded":"2024-06-29T14:37:54.377538+0000","last_fullsized":"2024-06-29T14:37:54.377538+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:03.508237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.374057+0000","last_change":"2024-06-29T14:37:52.374339+0000","last_active":"2024-06-29T14:37:54.374057+0000","last_peered":"2024-06-29T14:37:54.374057+0000","last_clean":"2024-06-29T14:37:54.374057+0000","last_became_active":"2024-06-29T14:37:52.374029+0000","last_became_peered":"2024-06-29T14:37:52.374029+0000","last_unstale":"2024-06-29T14:37:54.374057+0000","last_undegraded":"2024-06-29T14:37:54.374057+0000","last_fullsized":"2024-06-29T14:37:54.374057+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:19.432254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'114","reported_seq":418,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376935+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:37:54.376935+0000","last_peered":"2024-06-29T14:37:54.376935+0000","last_clean":"2024-06-29T14:37:54.376935+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:37:54.376935+0000","last_undegraded":"2024-06-29T14:37:54.376935+0000","last_fullsized":"2024-06-29T14:37:54.376935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":173,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":173,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542339,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59324,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312516,"statfs":{"total":96636764160,"available":96576016384,"internally_reserved":0,"allocated":1814528,"data_stored":1202872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542340,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1241088,"data_stored":620786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542339,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50432,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321408,"statfs":{"total":96636764160,"available":96585121792,"internally_reserved":0,"allocated":1232896,"data_stored":619208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1236992,"data_stored":614198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509636,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1232896,"data_stored":619756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575044,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58760,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313080,"statfs":{"total":96636764160,"available":96576593920,"internally_reserved":0,"allocated":1241088,"data_stored":619916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509635,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1236992,"data_stored":619286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32968,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338872,"statfs":{"total":96636764160,"available":96603004928,"internally_reserved":0,"allocated":1822720,"data_stored":1209542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:37:57.472 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:37:57.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.627+0000 7f75b157a640 1 -- 172.21.15.59:0/2460300834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0b40a0 msgr2=0x7f75ac15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.627+0000 7f75b157a640 1 --2- 172.21.15.59:0/2460300834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0b40a0 0x7f75ac15e8d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009830 tx=0x7f75a002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.627+0000 7f75b157a640 1 -- 172.21.15.59:0/2460300834 shutdown_connections 2024-06-29T14:37:57.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.627+0000 7f75b157a640 1 --2- 172.21.15.59:0/2460300834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0b40a0 0x7f75ac15e8d0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.627+0000 7f75b157a640 1 -- 172.21.15.59:0/2460300834 >> 172.21.15.59:0/2460300834 conn(0x7f75ac0d3ee0 msgr2=0x7f75ac0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.627+0000 7f75b157a640 1 -- 172.21.15.59:0/2460300834 shutdown_connections 2024-06-29T14:37:57.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.627+0000 7f75b157a640 1 -- 172.21.15.59:0/2460300834 wait complete. 2024-06-29T14:37:57.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.629+0000 7f75b157a640 1 Processor -- start 2024-06-29T14:37:57.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.629+0000 7f75b157a640 1 -- start start 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75b157a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0d23c0 0x7f75ac150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75b157a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac15ee10 0x7f75ac157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75b157a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac14b140 0x7f75ac14ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75b157a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac160450 con 0x7f75ac0d23c0 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75b157a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1602d0 con 0x7f75ac15ee10 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75b157a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f75ac1605d0 con 0x7f75ac14b140 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75aaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0d23c0 0x7f75ac150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75aaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0d23c0 0x7f75ac150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33666/0 (socket says 172.21.15.59:33666) 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75aaffd640 1 -- 172.21.15.59:0/3850018247 learned_addr learned my addr 172.21.15.59:0/3850018247 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:57.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.630+0000 7f75aa7fc640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac15ee10 0x7f75ac157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75ab7fe640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac14b140 0x7f75ac14ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75aaffd640 1 -- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac14b140 msgr2=0x7f75ac14ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75aaffd640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac14b140 0x7f75ac14ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75aaffd640 1 -- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac15ee10 msgr2=0x7f75ac157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75aaffd640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac15ee10 0x7f75ac157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75aaffd640 1 -- 172.21.15.59:0/3850018247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac152ee0 con 0x7f75ac0d23c0 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75aaffd640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0d23c0 0x7f75ac150270 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f759c00c780 tx=0x7f759c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75937fe640 1 -- 172.21.15.59:0/3850018247 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c003b70 con 0x7f75ac0d23c0 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75937fe640 1 -- 172.21.15.59:0/3850018247 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f759c003d10 con 0x7f75ac0d23c0 2024-06-29T14:37:57.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ac156b90 con 0x7f75ac0d23c0 2024-06-29T14:37:57.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75937fe640 1 -- 172.21.15.59:0/3850018247 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c00aab0 con 0x7f75ac0d23c0 2024-06-29T14:37:57.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.631+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ac157150 con 0x7f75ac0d23c0 2024-06-29T14:37:57.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.632+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ac15e8d0 con 0x7f75ac0d23c0 2024-06-29T14:37:57.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.633+0000 7f75937fe640 1 -- 172.21.15.59:0/3850018247 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f759c00ac50 con 0x7f75ac0d23c0 2024-06-29T14:37:57.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.634+0000 7f75937fe640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7598076420 0x7f7598078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.634+0000 7f75937fe640 1 -- 172.21.15.59:0/3850018247 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f759c097960 con 0x7f75ac0d23c0 2024-06-29T14:37:57.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.634+0000 7f75aa7fc640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7598076420 0x7f7598078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.634+0000 7f75aa7fc640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7598076420 0x7f7598078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f75a0034000 tx=0x7f75a0004730 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:57.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.638+0000 7f75937fe640 1 -- 172.21.15.59:0/3850018247 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f759c0608a0 con 0x7f75ac0d23c0 2024-06-29T14:37:57.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.830+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 --> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f75ac0d2950 con 0x7f7598076420 2024-06-29T14:37:57.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.835+0000 7f75937fe640 1 -- 172.21.15.59:0/3850018247 <== mgr.6179 v2:172.21.15.153:6832/1607211985 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252265 (secure 0 0 0) 0x7f75ac0d2950 con 0x7f7598076420 2024-06-29T14:37:57.835 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:37:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.838+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7598076420 msgr2=0x7f7598078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7598076420 0x7f7598078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f75a0034000 tx=0x7f75a0004730 comp rx=0 tx=0).stop 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0d23c0 msgr2=0x7f75ac150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0d23c0 0x7f75ac150270 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f759c00c780 tx=0x7f759c00cc40 comp rx=0 tx=0).stop 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 shutdown_connections 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f7598076420 0x7f7598078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75ac14b140 0x7f75ac14ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75ac15ee10 0x7f75ac157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 --2- 172.21.15.59:0/3850018247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75ac0d23c0 0x7f75ac150270 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 >> 172.21.15.59:0/3850018247 conn(0x7f75ac0d3ee0 msgr2=0x7f75ac151260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 shutdown_connections 2024-06-29T14:37:57.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:57.839+0000 7f75b157a640 1 -- 172.21.15.59:0/3850018247 wait complete. 2024-06-29T14:37:57.848 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:37:56.914877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":218,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331048,"kb_used_data":10800,"kb_used_omap":12,"kb_used_meta":320115,"kb_avail":754643672,"statfs":{"total":773094113280,"available":772755120128,"internally_reserved":0,"allocated":11059200,"data_stored":6125564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":327798322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.565570"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377063+0000","last_change":"2024-06-29T14:37:53.394513+0000","last_active":"2024-06-29T14:37:54.377063+0000","last_peered":"2024-06-29T14:37:54.377063+0000","last_clean":"2024-06-29T14:37:54.377063+0000","last_became_active":"2024-06-29T14:37:53.394173+0000","last_became_peered":"2024-06-29T14:37:53.394173+0000","last_unstale":"2024-06-29T14:37:54.377063+0000","last_undegraded":"2024-06-29T14:37:54.377063+0000","last_fullsized":"2024-06-29T14:37:54.377063+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:08.626526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397379+0000","last_change":"2024-06-29T14:37:53.411060+0000","last_active":"2024-06-29T14:37:54.397379+0000","last_peered":"2024-06-29T14:37:54.397379+0000","last_clean":"2024-06-29T14:37:54.397379+0000","last_became_active":"2024-06-29T14:37:53.405510+0000","last_became_peered":"2024-06-29T14:37:53.405510+0000","last_unstale":"2024-06-29T14:37:54.397379+0000","last_undegraded":"2024-06-29T14:37:54.397379+0000","last_fullsized":"2024-06-29T14:37:54.397379+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:08.171838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500680+0000","last_change":"2024-06-29T14:37:53.402011+0000","last_active":"2024-06-29T14:37:54.500680+0000","last_peered":"2024-06-29T14:37:54.500680+0000","last_clean":"2024-06-29T14:37:54.500680+0000","last_became_active":"2024-06-29T14:37:53.401641+0000","last_became_peered":"2024-06-29T14:37:53.401641+0000","last_unstale":"2024-06-29T14:37:54.500680+0000","last_undegraded":"2024-06-29T14:37:54.500680+0000","last_fullsized":"2024-06-29T14:37:54.500680+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:13:24.710230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501264+0000","last_change":"2024-06-29T14:37:53.388955+0000","last_active":"2024-06-29T14:37:54.501264+0000","last_peered":"2024-06-29T14:37:54.501264+0000","last_clean":"2024-06-29T14:37:54.501264+0000","last_became_active":"2024-06-29T14:37:53.388227+0000","last_became_peered":"2024-06-29T14:37:53.388227+0000","last_unstale":"2024-06-29T14:37:54.501264+0000","last_undegraded":"2024-06-29T14:37:54.501264+0000","last_fullsized":"2024-06-29T14:37:54.501264+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:01.383994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399472+0000","last_change":"2024-06-29T14:37:53.388122+0000","last_active":"2024-06-29T14:37:54.399472+0000","last_peered":"2024-06-29T14:37:54.399472+0000","last_clean":"2024-06-29T14:37:54.399472+0000","last_became_active":"2024-06-29T14:37:53.387220+0000","last_became_peered":"2024-06-29T14:37:53.387220+0000","last_unstale":"2024-06-29T14:37:54.399472+0000","last_undegraded":"2024-06-29T14:37:54.399472+0000","last_fullsized":"2024-06-29T14:37:54.399472+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:23.754075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373708+0000","last_change":"2024-06-29T14:37:53.396720+0000","last_active":"2024-06-29T14:37:54.373708+0000","last_peered":"2024-06-29T14:37:54.373708+0000","last_clean":"2024-06-29T14:37:54.373708+0000","last_became_active":"2024-06-29T14:37:53.396035+0000","last_became_peered":"2024-06-29T14:37:53.396035+0000","last_unstale":"2024-06-29T14:37:54.373708+0000","last_undegraded":"2024-06-29T14:37:54.373708+0000","last_fullsized":"2024-06-29T14:37:54.373708+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:39.642171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377171+0000","last_change":"2024-06-29T14:37:53.378369+0000","last_active":"2024-06-29T14:37:54.377171+0000","last_peered":"2024-06-29T14:37:54.377171+0000","last_clean":"2024-06-29T14:37:54.377171+0000","last_became_active":"2024-06-29T14:37:53.377909+0000","last_became_peered":"2024-06-29T14:37:53.377909+0000","last_unstale":"2024-06-29T14:37:54.377171+0000","last_undegraded":"2024-06-29T14:37:54.377171+0000","last_fullsized":"2024-06-29T14:37:54.377171+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:37:40.218294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399269+0000","last_change":"2024-06-29T14:37:53.397911+0000","last_active":"2024-06-29T14:37:54.399269+0000","last_peered":"2024-06-29T14:37:54.399269+0000","last_clean":"2024-06-29T14:37:54.399269+0000","last_became_active":"2024-06-29T14:37:53.397535+0000","last_became_peered":"2024-06-29T14:37:53.397535+0000","last_unstale":"2024-06-29T14:37:54.399269+0000","last_undegraded":"2024-06-29T14:37:54.399269+0000","last_fullsized":"2024-06-29T14:37:54.399269+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:23.062165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502139+0000","last_change":"2024-06-29T14:37:53.388967+0000","last_active":"2024-06-29T14:37:54.502139+0000","last_peered":"2024-06-29T14:37:54.502139+0000","last_clean":"2024-06-29T14:37:54.502139+0000","last_became_active":"2024-06-29T14:37:53.388657+0000","last_became_peered":"2024-06-29T14:37:53.388657+0000","last_unstale":"2024-06-29T14:37:54.502139+0000","last_undegraded":"2024-06-29T14:37:54.502139+0000","last_fullsized":"2024-06-29T14:37:54.502139+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:26.121311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501904+0000","last_change":"2024-06-29T14:37:53.387077+0000","last_active":"2024-06-29T14:37:54.501904+0000","last_peered":"2024-06-29T14:37:54.501904+0000","last_clean":"2024-06-29T14:37:54.501904+0000","last_became_active":"2024-06-29T14:37:53.386679+0000","last_became_peered":"2024-06-29T14:37:53.386679+0000","last_unstale":"2024-06-29T14:37:54.501904+0000","last_undegraded":"2024-06-29T14:37:54.501904+0000","last_fullsized":"2024-06-29T14:37:54.501904+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:28.890236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397746+0000","last_change":"2024-06-29T14:37:53.410583+0000","last_active":"2024-06-29T14:37:54.397746+0000","last_peered":"2024-06-29T14:37:54.397746+0000","last_clean":"2024-06-29T14:37:54.397746+0000","last_became_active":"2024-06-29T14:37:53.407764+0000","last_became_peered":"2024-06-29T14:37:53.407764+0000","last_unstale":"2024-06-29T14:37:54.397746+0000","last_undegraded":"2024-06-29T14:37:54.397746+0000","last_fullsized":"2024-06-29T14:37:54.397746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:13.215024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501459+0000","last_change":"2024-06-29T14:37:53.402577+0000","last_active":"2024-06-29T14:37:54.501459+0000","last_peered":"2024-06-29T14:37:54.501459+0000","last_clean":"2024-06-29T14:37:54.501459+0000","last_became_active":"2024-06-29T14:37:53.402232+0000","last_became_peered":"2024-06-29T14:37:53.402232+0000","last_unstale":"2024-06-29T14:37:54.501459+0000","last_undegraded":"2024-06-29T14:37:54.501459+0000","last_fullsized":"2024-06-29T14:37:54.501459+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:57.543562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399576+0000","last_change":"2024-06-29T14:37:53.388501+0000","last_active":"2024-06-29T14:37:54.399576+0000","last_peered":"2024-06-29T14:37:54.399576+0000","last_clean":"2024-06-29T14:37:54.399576+0000","last_became_active":"2024-06-29T14:37:53.387644+0000","last_became_peered":"2024-06-29T14:37:53.387644+0000","last_unstale":"2024-06-29T14:37:54.399576+0000","last_undegraded":"2024-06-29T14:37:54.399576+0000","last_fullsized":"2024-06-29T14:37:54.399576+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:45:48.889699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373728+0000","last_change":"2024-06-29T14:37:53.396397+0000","last_active":"2024-06-29T14:37:54.373728+0000","last_peered":"2024-06-29T14:37:54.373728+0000","last_clean":"2024-06-29T14:37:54.373728+0000","last_became_active":"2024-06-29T14:37:53.395459+0000","last_became_peered":"2024-06-29T14:37:53.395459+0000","last_unstale":"2024-06-29T14:37:54.373728+0000","last_undegraded":"2024-06-29T14:37:54.373728+0000","last_fullsized":"2024-06-29T14:37:54.373728+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:58.497717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397040+0000","last_change":"2024-06-29T14:37:53.396528+0000","last_active":"2024-06-29T14:37:54.397040+0000","last_peered":"2024-06-29T14:37:54.397040+0000","last_clean":"2024-06-29T14:37:54.397040+0000","last_became_active":"2024-06-29T14:37:53.395999+0000","last_became_peered":"2024-06-29T14:37:53.395999+0000","last_unstale":"2024-06-29T14:37:54.397040+0000","last_undegraded":"2024-06-29T14:37:54.397040+0000","last_fullsized":"2024-06-29T14:37:54.397040+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:56.219035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377296+0000","last_change":"2024-06-29T14:37:53.387998+0000","last_active":"2024-06-29T14:37:54.377296+0000","last_peered":"2024-06-29T14:37:54.377296+0000","last_clean":"2024-06-29T14:37:54.377296+0000","last_became_active":"2024-06-29T14:37:53.387673+0000","last_became_peered":"2024-06-29T14:37:53.387673+0000","last_unstale":"2024-06-29T14:37:54.377296+0000","last_undegraded":"2024-06-29T14:37:54.377296+0000","last_fullsized":"2024-06-29T14:37:54.377296+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:37.919672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399693+0000","last_change":"2024-06-29T14:37:53.386517+0000","last_active":"2024-06-29T14:37:54.399693+0000","last_peered":"2024-06-29T14:37:54.399693+0000","last_clean":"2024-06-29T14:37:54.399693+0000","last_became_active":"2024-06-29T14:37:53.385987+0000","last_became_peered":"2024-06-29T14:37:53.385987+0000","last_unstale":"2024-06-29T14:37:54.399693+0000","last_undegraded":"2024-06-29T14:37:54.399693+0000","last_fullsized":"2024-06-29T14:37:54.399693+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:00.009406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500722+0000","last_change":"2024-06-29T14:37:53.396666+0000","last_active":"2024-06-29T14:37:54.500722+0000","last_peered":"2024-06-29T14:37:54.500722+0000","last_clean":"2024-06-29T14:37:54.500722+0000","last_became_active":"2024-06-29T14:37:53.396236+0000","last_became_peered":"2024-06-29T14:37:53.396236+0000","last_unstale":"2024-06-29T14:37:54.500722+0000","last_undegraded":"2024-06-29T14:37:54.500722+0000","last_fullsized":"2024-06-29T14:37:54.500722+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:04.070850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373655+0000","last_change":"2024-06-29T14:37:53.383889+0000","last_active":"2024-06-29T14:37:54.373655+0000","last_peered":"2024-06-29T14:37:54.373655+0000","last_clean":"2024-06-29T14:37:54.373655+0000","last_became_active":"2024-06-29T14:37:53.383550+0000","last_became_peered":"2024-06-29T14:37:53.383550+0000","last_unstale":"2024-06-29T14:37:54.373655+0000","last_undegraded":"2024-06-29T14:37:54.373655+0000","last_fullsized":"2024-06-29T14:37:54.373655+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:35.647625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399718+0000","last_change":"2024-06-29T14:37:53.386428+0000","last_active":"2024-06-29T14:37:54.399718+0000","last_peered":"2024-06-29T14:37:54.399718+0000","last_clean":"2024-06-29T14:37:54.399718+0000","last_became_active":"2024-06-29T14:37:53.386068+0000","last_became_peered":"2024-06-29T14:37:53.386068+0000","last_unstale":"2024-06-29T14:37:54.399718+0000","last_undegraded":"2024-06-29T14:37:54.399718+0000","last_fullsized":"2024-06-29T14:37:54.399718+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:45.134932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501658+0000","last_change":"2024-06-29T14:37:53.386340+0000","last_active":"2024-06-29T14:37:54.501658+0000","last_peered":"2024-06-29T14:37:54.501658+0000","last_clean":"2024-06-29T14:37:54.501658+0000","last_became_active":"2024-06-29T14:37:53.385945+0000","last_became_peered":"2024-06-29T14:37:53.385945+0000","last_unstale":"2024-06-29T14:37:54.501658+0000","last_undegraded":"2024-06-29T14:37:54.501658+0000","last_fullsized":"2024-06-29T14:37:54.501658+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:31.302377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399650+0000","last_change":"2024-06-29T14:37:53.389244+0000","last_active":"2024-06-29T14:37:54.399650+0000","last_peered":"2024-06-29T14:37:54.399650+0000","last_clean":"2024-06-29T14:37:54.399650+0000","last_became_active":"2024-06-29T14:37:53.388900+0000","last_became_peered":"2024-06-29T14:37:53.388900+0000","last_unstale":"2024-06-29T14:37:54.399650+0000","last_undegraded":"2024-06-29T14:37:54.399650+0000","last_fullsized":"2024-06-29T14:37:54.399650+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:04:55.939413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501341+0000","last_change":"2024-06-29T14:37:53.381911+0000","last_active":"2024-06-29T14:37:54.501341+0000","last_peered":"2024-06-29T14:37:54.501341+0000","last_clean":"2024-06-29T14:37:54.501341+0000","last_became_active":"2024-06-29T14:37:53.380888+0000","last_became_peered":"2024-06-29T14:37:53.380888+0000","last_unstale":"2024-06-29T14:37:54.501341+0000","last_undegraded":"2024-06-29T14:37:54.501341+0000","last_fullsized":"2024-06-29T14:37:54.501341+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:53.579615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397023+0000","last_change":"2024-06-29T14:37:53.399470+0000","last_active":"2024-06-29T14:37:54.397023+0000","last_peered":"2024-06-29T14:37:54.397023+0000","last_clean":"2024-06-29T14:37:54.397023+0000","last_became_active":"2024-06-29T14:37:53.399121+0000","last_became_peered":"2024-06-29T14:37:53.399121+0000","last_unstale":"2024-06-29T14:37:54.397023+0000","last_undegraded":"2024-06-29T14:37:54.397023+0000","last_fullsized":"2024-06-29T14:37:54.397023+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:07:03.251164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397499+0000","last_change":"2024-06-29T14:37:53.404979+0000","last_active":"2024-06-29T14:37:54.397499+0000","last_peered":"2024-06-29T14:37:54.397499+0000","last_clean":"2024-06-29T14:37:54.397499+0000","last_became_active":"2024-06-29T14:37:53.404637+0000","last_became_peered":"2024-06-29T14:37:53.404637+0000","last_unstale":"2024-06-29T14:37:54.397499+0000","last_undegraded":"2024-06-29T14:37:54.397499+0000","last_fullsized":"2024-06-29T14:37:54.397499+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:45:31.098895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501815+0000","last_change":"2024-06-29T14:37:53.403833+0000","last_active":"2024-06-29T14:37:54.501815+0000","last_peered":"2024-06-29T14:37:54.501815+0000","last_clean":"2024-06-29T14:37:54.501815+0000","last_became_active":"2024-06-29T14:37:53.403521+0000","last_became_peered":"2024-06-29T14:37:53.403521+0000","last_unstale":"2024-06-29T14:37:54.501815+0000","last_undegraded":"2024-06-29T14:37:54.501815+0000","last_fullsized":"2024-06-29T14:37:54.501815+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:12.286797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377237+0000","last_change":"2024-06-29T14:37:53.388693+0000","last_active":"2024-06-29T14:37:54.377237+0000","last_peered":"2024-06-29T14:37:54.377237+0000","last_clean":"2024-06-29T14:37:54.377237+0000","last_became_active":"2024-06-29T14:37:53.388365+0000","last_became_peered":"2024-06-29T14:37:53.388365+0000","last_unstale":"2024-06-29T14:37:54.377237+0000","last_undegraded":"2024-06-29T14:37:54.377237+0000","last_fullsized":"2024-06-29T14:37:54.377237+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:19.201197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399903+0000","last_change":"2024-06-29T14:37:53.384862+0000","last_active":"2024-06-29T14:37:54.399903+0000","last_peered":"2024-06-29T14:37:54.399903+0000","last_clean":"2024-06-29T14:37:54.399903+0000","last_became_active":"2024-06-29T14:37:53.384501+0000","last_became_peered":"2024-06-29T14:37:53.384501+0000","last_unstale":"2024-06-29T14:37:54.399903+0000","last_undegraded":"2024-06-29T14:37:54.399903+0000","last_fullsized":"2024-06-29T14:37:54.399903+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:18:54.178493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397587+0000","last_change":"2024-06-29T14:37:53.401791+0000","last_active":"2024-06-29T14:37:54.397587+0000","last_peered":"2024-06-29T14:37:54.397587+0000","last_clean":"2024-06-29T14:37:54.397587+0000","last_became_active":"2024-06-29T14:37:53.401016+0000","last_became_peered":"2024-06-29T14:37:53.401016+0000","last_unstale":"2024-06-29T14:37:54.397587+0000","last_undegraded":"2024-06-29T14:37:54.397587+0000","last_fullsized":"2024-06-29T14:37:54.397587+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:52.890466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376682+0000","last_change":"2024-06-29T14:37:53.400832+0000","last_active":"2024-06-29T14:37:54.376682+0000","last_peered":"2024-06-29T14:37:54.376682+0000","last_clean":"2024-06-29T14:37:54.376682+0000","last_became_active":"2024-06-29T14:37:53.400455+0000","last_became_peered":"2024-06-29T14:37:53.400455+0000","last_unstale":"2024-06-29T14:37:54.376682+0000","last_undegraded":"2024-06-29T14:37:54.376682+0000","last_fullsized":"2024-06-29T14:37:54.376682+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:45.134205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500478+0000","last_change":"2024-06-29T14:37:53.388185+0000","last_active":"2024-06-29T14:37:54.500478+0000","last_peered":"2024-06-29T14:37:54.500478+0000","last_clean":"2024-06-29T14:37:54.500478+0000","last_became_active":"2024-06-29T14:37:53.387840+0000","last_became_peered":"2024-06-29T14:37:53.387840+0000","last_unstale":"2024-06-29T14:37:54.500478+0000","last_undegraded":"2024-06-29T14:37:54.500478+0000","last_fullsized":"2024-06-29T14:37:54.500478+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:55.568117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376855+0000","last_change":"2024-06-29T14:37:53.394441+0000","last_active":"2024-06-29T14:37:54.376855+0000","last_peered":"2024-06-29T14:37:54.376855+0000","last_clean":"2024-06-29T14:37:54.376855+0000","last_became_active":"2024-06-29T14:37:53.394112+0000","last_became_peered":"2024-06-29T14:37:53.394112+0000","last_unstale":"2024-06-29T14:37:54.376855+0000","last_undegraded":"2024-06-29T14:37:54.376855+0000","last_fullsized":"2024-06-29T14:37:54.376855+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:20.364846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500444+0000","last_change":"2024-06-29T14:37:53.398460+0000","last_active":"2024-06-29T14:37:54.500444+0000","last_peered":"2024-06-29T14:37:54.500444+0000","last_clean":"2024-06-29T14:37:54.500444+0000","last_became_active":"2024-06-29T14:37:53.398123+0000","last_became_peered":"2024-06-29T14:37:53.398123+0000","last_unstale":"2024-06-29T14:37:54.500444+0000","last_undegraded":"2024-06-29T14:37:54.500444+0000","last_fullsized":"2024-06-29T14:37:54.500444+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:04.069782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"100'4","reported_seq":18,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399724+0000","last_change":"2024-06-29T14:37:52.371901+0000","last_active":"2024-06-29T14:37:54.399724+0000","last_peered":"2024-06-29T14:37:54.399724+0000","last_clean":"2024-06-29T14:37:54.399724+0000","last_became_active":"2024-06-29T14:37:52.371603+0000","last_became_peered":"2024-06-29T14:37:52.371603+0000","last_unstale":"2024-06-29T14:37:54.399724+0000","last_undegraded":"2024-06-29T14:37:54.399724+0000","last_fullsized":"2024-06-29T14:37:54.399724+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:40.218022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397115+0000","last_change":"2024-06-29T14:37:53.400195+0000","last_active":"2024-06-29T14:37:54.397115+0000","last_peered":"2024-06-29T14:37:54.397115+0000","last_clean":"2024-06-29T14:37:54.397115+0000","last_became_active":"2024-06-29T14:37:53.399863+0000","last_became_peered":"2024-06-29T14:37:53.399863+0000","last_unstale":"2024-06-29T14:37:54.397115+0000","last_undegraded":"2024-06-29T14:37:54.397115+0000","last_fullsized":"2024-06-29T14:37:54.397115+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:44.577736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377805+0000","last_change":"2024-06-29T14:37:52.374061+0000","last_active":"2024-06-29T14:37:54.377805+0000","last_peered":"2024-06-29T14:37:54.377805+0000","last_clean":"2024-06-29T14:37:54.377805+0000","last_became_active":"2024-06-29T14:37:52.373637+0000","last_became_peered":"2024-06-29T14:37:52.373637+0000","last_unstale":"2024-06-29T14:37:54.377805+0000","last_undegraded":"2024-06-29T14:37:54.377805+0000","last_fullsized":"2024-06-29T14:37:54.377805+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:04:55.861797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501537+0000","last_change":"2024-06-29T14:37:53.391600+0000","last_active":"2024-06-29T14:37:54.501537+0000","last_peered":"2024-06-29T14:37:54.501537+0000","last_clean":"2024-06-29T14:37:54.501537+0000","last_became_active":"2024-06-29T14:37:53.391265+0000","last_became_peered":"2024-06-29T14:37:53.391265+0000","last_unstale":"2024-06-29T14:37:54.501537+0000","last_undegraded":"2024-06-29T14:37:54.501537+0000","last_fullsized":"2024-06-29T14:37:54.501537+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:18:49.439460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377370+0000","last_change":"2024-06-29T14:37:52.379761+0000","last_active":"2024-06-29T14:37:54.377370+0000","last_peered":"2024-06-29T14:37:54.377370+0000","last_clean":"2024-06-29T14:37:54.377370+0000","last_became_active":"2024-06-29T14:37:52.379541+0000","last_became_peered":"2024-06-29T14:37:52.379541+0000","last_unstale":"2024-06-29T14:37:54.377370+0000","last_undegraded":"2024-06-29T14:37:54.377370+0000","last_fullsized":"2024-06-29T14:37:54.377370+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:21.142171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502051+0000","last_change":"2024-06-29T14:37:53.387813+0000","last_active":"2024-06-29T14:37:54.502051+0000","last_peered":"2024-06-29T14:37:54.502051+0000","last_clean":"2024-06-29T14:37:54.502051+0000","last_became_active":"2024-06-29T14:37:53.385790+0000","last_became_peered":"2024-06-29T14:37:53.385790+0000","last_unstale":"2024-06-29T14:37:54.502051+0000","last_undegraded":"2024-06-29T14:37:54.502051+0000","last_fullsized":"2024-06-29T14:37:54.502051+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:55.092845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378087+0000","last_change":"2024-06-29T14:37:52.370965+0000","last_active":"2024-06-29T14:37:54.378087+0000","last_peered":"2024-06-29T14:37:54.378087+0000","last_clean":"2024-06-29T14:37:54.378087+0000","last_became_active":"2024-06-29T14:37:52.370661+0000","last_became_peered":"2024-06-29T14:37:52.370661+0000","last_unstale":"2024-06-29T14:37:54.378087+0000","last_undegraded":"2024-06-29T14:37:54.378087+0000","last_fullsized":"2024-06-29T14:37:54.378087+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:00:15.689139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397348+0000","last_change":"2024-06-29T14:37:53.410454+0000","last_active":"2024-06-29T14:37:54.397348+0000","last_peered":"2024-06-29T14:37:54.397348+0000","last_clean":"2024-06-29T14:37:54.397348+0000","last_became_active":"2024-06-29T14:37:53.406280+0000","last_became_peered":"2024-06-29T14:37:53.406280+0000","last_unstale":"2024-06-29T14:37:54.397348+0000","last_undegraded":"2024-06-29T14:37:54.397348+0000","last_fullsized":"2024-06-29T14:37:54.397348+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.825289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501802+0000","last_change":"2024-06-29T14:37:52.369426+0000","last_active":"2024-06-29T14:37:54.501802+0000","last_peered":"2024-06-29T14:37:54.501802+0000","last_clean":"2024-06-29T14:37:54.501802+0000","last_became_active":"2024-06-29T14:37:52.369068+0000","last_became_peered":"2024-06-29T14:37:52.369068+0000","last_unstale":"2024-06-29T14:37:54.501802+0000","last_undegraded":"2024-06-29T14:37:54.501802+0000","last_fullsized":"2024-06-29T14:37:54.501802+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:00:15.689139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397708+0000","last_change":"2024-06-29T14:37:53.407498+0000","last_active":"2024-06-29T14:37:54.397708+0000","last_peered":"2024-06-29T14:37:54.397708+0000","last_clean":"2024-06-29T14:37:54.397708+0000","last_became_active":"2024-06-29T14:37:53.406972+0000","last_became_peered":"2024-06-29T14:37:53.406972+0000","last_unstale":"2024-06-29T14:37:54.397708+0000","last_undegraded":"2024-06-29T14:37:54.397708+0000","last_fullsized":"2024-06-29T14:37:54.397708+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:24.170491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500934+0000","last_change":"2024-06-29T14:37:52.371987+0000","last_active":"2024-06-29T14:37:54.500934+0000","last_peered":"2024-06-29T14:37:54.500934+0000","last_clean":"2024-06-29T14:37:54.500934+0000","last_became_active":"2024-06-29T14:37:52.371575+0000","last_became_peered":"2024-06-29T14:37:52.371575+0000","last_unstale":"2024-06-29T14:37:54.500934+0000","last_undegraded":"2024-06-29T14:37:54.500934+0000","last_fullsized":"2024-06-29T14:37:54.500934+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:25.267941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502152+0000","last_change":"2024-06-29T14:37:53.388309+0000","last_active":"2024-06-29T14:37:54.502152+0000","last_peered":"2024-06-29T14:37:54.502152+0000","last_clean":"2024-06-29T14:37:54.502152+0000","last_became_active":"2024-06-29T14:37:53.388004+0000","last_became_peered":"2024-06-29T14:37:53.388004+0000","last_unstale":"2024-06-29T14:37:54.502152+0000","last_undegraded":"2024-06-29T14:37:54.502152+0000","last_fullsized":"2024-06-29T14:37:54.502152+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:25.689885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397160+0000","last_change":"2024-06-29T14:37:52.380053+0000","last_active":"2024-06-29T14:37:54.397160+0000","last_peered":"2024-06-29T14:37:54.397160+0000","last_clean":"2024-06-29T14:37:54.397160+0000","last_became_active":"2024-06-29T14:37:52.379670+0000","last_became_peered":"2024-06-29T14:37:52.379670+0000","last_unstale":"2024-06-29T14:37:54.397160+0000","last_undegraded":"2024-06-29T14:37:54.397160+0000","last_fullsized":"2024-06-29T14:37:54.397160+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:38.063079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377833+0000","last_change":"2024-06-29T14:37:53.381393+0000","last_active":"2024-06-29T14:37:54.377833+0000","last_peered":"2024-06-29T14:37:54.377833+0000","last_clean":"2024-06-29T14:37:54.377833+0000","last_became_active":"2024-06-29T14:37:53.381010+0000","last_became_peered":"2024-06-29T14:37:53.381010+0000","last_unstale":"2024-06-29T14:37:54.377833+0000","last_undegraded":"2024-06-29T14:37:54.377833+0000","last_fullsized":"2024-06-29T14:37:54.377833+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:29.394763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399337+0000","last_change":"2024-06-29T14:37:52.367630+0000","last_active":"2024-06-29T14:37:54.399337+0000","last_peered":"2024-06-29T14:37:54.399337+0000","last_clean":"2024-06-29T14:37:54.399337+0000","last_became_active":"2024-06-29T14:37:52.366598+0000","last_became_peered":"2024-06-29T14:37:52.366598+0000","last_unstale":"2024-06-29T14:37:54.399337+0000","last_undegraded":"2024-06-29T14:37:54.399337+0000","last_fullsized":"2024-06-29T14:37:54.399337+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:43.687013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397026+0000","last_change":"2024-06-29T14:37:53.398609+0000","last_active":"2024-06-29T14:37:54.397026+0000","last_peered":"2024-06-29T14:37:54.397026+0000","last_clean":"2024-06-29T14:37:54.397026+0000","last_became_active":"2024-06-29T14:37:53.398271+0000","last_became_peered":"2024-06-29T14:37:53.398271+0000","last_unstale":"2024-06-29T14:37:54.397026+0000","last_undegraded":"2024-06-29T14:37:54.397026+0000","last_fullsized":"2024-06-29T14:37:54.397026+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:23.695035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377247+0000","last_change":"2024-06-29T14:37:52.374251+0000","last_active":"2024-06-29T14:37:54.377247+0000","last_peered":"2024-06-29T14:37:54.377247+0000","last_clean":"2024-06-29T14:37:54.377247+0000","last_became_active":"2024-06-29T14:37:52.373551+0000","last_became_peered":"2024-06-29T14:37:52.373551+0000","last_unstale":"2024-06-29T14:37:54.377247+0000","last_undegraded":"2024-06-29T14:37:54.377247+0000","last_fullsized":"2024-06-29T14:37:54.377247+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:09.510075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501126+0000","last_change":"2024-06-29T14:37:53.383525+0000","last_active":"2024-06-29T14:37:54.501126+0000","last_peered":"2024-06-29T14:37:54.501126+0000","last_clean":"2024-06-29T14:37:54.501126+0000","last_became_active":"2024-06-29T14:37:53.383206+0000","last_became_peered":"2024-06-29T14:37:53.383206+0000","last_unstale":"2024-06-29T14:37:54.501126+0000","last_undegraded":"2024-06-29T14:37:54.501126+0000","last_fullsized":"2024-06-29T14:37:54.501126+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:39.705828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501966+0000","last_change":"2024-06-29T14:37:52.368842+0000","last_active":"2024-06-29T14:37:54.501966+0000","last_peered":"2024-06-29T14:37:54.501966+0000","last_clean":"2024-06-29T14:37:54.501966+0000","last_became_active":"2024-06-29T14:37:52.368528+0000","last_became_peered":"2024-06-29T14:37:52.368528+0000","last_unstale":"2024-06-29T14:37:54.501966+0000","last_undegraded":"2024-06-29T14:37:54.501966+0000","last_fullsized":"2024-06-29T14:37:54.501966+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:26:22.971432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373889+0000","last_change":"2024-06-29T14:37:53.387662+0000","last_active":"2024-06-29T14:37:54.373889+0000","last_peered":"2024-06-29T14:37:54.373889+0000","last_clean":"2024-06-29T14:37:54.373889+0000","last_became_active":"2024-06-29T14:37:53.387030+0000","last_became_peered":"2024-06-29T14:37:53.387030+0000","last_unstale":"2024-06-29T14:37:54.373889+0000","last_undegraded":"2024-06-29T14:37:54.373889+0000","last_fullsized":"2024-06-29T14:37:54.373889+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:30:18.362323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"100'3","reported_seq":17,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399852+0000","last_change":"2024-06-29T14:37:52.368996+0000","last_active":"2024-06-29T14:37:54.399852+0000","last_peered":"2024-06-29T14:37:54.399852+0000","last_clean":"2024-06-29T14:37:54.399852+0000","last_became_active":"2024-06-29T14:37:52.368587+0000","last_became_peered":"2024-06-29T14:37:52.368587+0000","last_unstale":"2024-06-29T14:37:54.399852+0000","last_undegraded":"2024-06-29T14:37:54.399852+0000","last_fullsized":"2024-06-29T14:37:54.399852+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:20.857673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500483+0000","last_change":"2024-06-29T14:37:53.405485+0000","last_active":"2024-06-29T14:37:54.500483+0000","last_peered":"2024-06-29T14:37:54.500483+0000","last_clean":"2024-06-29T14:37:54.500483+0000","last_became_active":"2024-06-29T14:37:53.404742+0000","last_became_peered":"2024-06-29T14:37:53.404742+0000","last_unstale":"2024-06-29T14:37:54.500483+0000","last_undegraded":"2024-06-29T14:37:54.500483+0000","last_fullsized":"2024-06-29T14:37:54.500483+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:51.681296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"100'6","reported_seq":20,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378182+0000","last_change":"2024-06-29T14:37:52.369312+0000","last_active":"2024-06-29T14:37:54.378182+0000","last_peered":"2024-06-29T14:37:54.378182+0000","last_clean":"2024-06-29T14:37:54.378182+0000","last_became_active":"2024-06-29T14:37:52.368394+0000","last_became_peered":"2024-06-29T14:37:52.368394+0000","last_unstale":"2024-06-29T14:37:54.378182+0000","last_undegraded":"2024-06-29T14:37:54.378182+0000","last_fullsized":"2024-06-29T14:37:54.378182+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:08.856778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501768+0000","last_change":"2024-06-29T14:37:53.389221+0000","last_active":"2024-06-29T14:37:54.501768+0000","last_peered":"2024-06-29T14:37:54.501768+0000","last_clean":"2024-06-29T14:37:54.501768+0000","last_became_active":"2024-06-29T14:37:53.388659+0000","last_became_peered":"2024-06-29T14:37:53.388659+0000","last_unstale":"2024-06-29T14:37:54.501768+0000","last_undegraded":"2024-06-29T14:37:54.501768+0000","last_fullsized":"2024-06-29T14:37:54.501768+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:18.210057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500948+0000","last_change":"2024-06-29T14:37:52.368116+0000","last_active":"2024-06-29T14:37:54.500948+0000","last_peered":"2024-06-29T14:37:54.500948+0000","last_clean":"2024-06-29T14:37:54.500948+0000","last_became_active":"2024-06-29T14:37:52.367664+0000","last_became_peered":"2024-06-29T14:37:52.367664+0000","last_unstale":"2024-06-29T14:37:54.500948+0000","last_undegraded":"2024-06-29T14:37:54.500948+0000","last_fullsized":"2024-06-29T14:37:54.500948+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:38.764223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376993+0000","last_change":"2024-06-29T14:37:53.385089+0000","last_active":"2024-06-29T14:37:54.376993+0000","last_peered":"2024-06-29T14:37:54.376993+0000","last_clean":"2024-06-29T14:37:54.376993+0000","last_became_active":"2024-06-29T14:37:53.384776+0000","last_became_peered":"2024-06-29T14:37:53.384776+0000","last_unstale":"2024-06-29T14:37:54.376993+0000","last_undegraded":"2024-06-29T14:37:54.376993+0000","last_fullsized":"2024-06-29T14:37:54.376993+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:55.092845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376825+0000","last_change":"2024-06-29T14:37:52.377689+0000","last_active":"2024-06-29T14:37:54.376825+0000","last_peered":"2024-06-29T14:37:54.376825+0000","last_clean":"2024-06-29T14:37:54.376825+0000","last_became_active":"2024-06-29T14:37:52.377384+0000","last_became_peered":"2024-06-29T14:37:52.377384+0000","last_unstale":"2024-06-29T14:37:54.376825+0000","last_undegraded":"2024-06-29T14:37:54.376825+0000","last_fullsized":"2024-06-29T14:37:54.376825+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:05.602047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377549+0000","last_change":"2024-06-29T14:37:53.378494+0000","last_active":"2024-06-29T14:37:54.377549+0000","last_peered":"2024-06-29T14:37:54.377549+0000","last_clean":"2024-06-29T14:37:54.377549+0000","last_became_active":"2024-06-29T14:37:53.377998+0000","last_became_peered":"2024-06-29T14:37:53.377998+0000","last_unstale":"2024-06-29T14:37:54.377549+0000","last_undegraded":"2024-06-29T14:37:54.377549+0000","last_fullsized":"2024-06-29T14:37:54.377549+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:25.689885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399368+0000","last_change":"2024-06-29T14:37:52.371238+0000","last_active":"2024-06-29T14:37:54.399368+0000","last_peered":"2024-06-29T14:37:54.399368+0000","last_clean":"2024-06-29T14:37:54.399368+0000","last_became_active":"2024-06-29T14:37:52.370917+0000","last_became_peered":"2024-06-29T14:37:52.370917+0000","last_unstale":"2024-06-29T14:37:54.399368+0000","last_undegraded":"2024-06-29T14:37:54.399368+0000","last_fullsized":"2024-06-29T14:37:54.399368+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:15:39.995393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378052+0000","last_change":"2024-06-29T14:37:53.392895+0000","last_active":"2024-06-29T14:37:54.378052+0000","last_peered":"2024-06-29T14:37:54.378052+0000","last_clean":"2024-06-29T14:37:54.378052+0000","last_became_active":"2024-06-29T14:37:53.392585+0000","last_became_peered":"2024-06-29T14:37:53.392585+0000","last_unstale":"2024-06-29T14:37:54.378052+0000","last_undegraded":"2024-06-29T14:37:54.378052+0000","last_fullsized":"2024-06-29T14:37:54.378052+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:38.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397340+0000","last_change":"2024-06-29T14:37:52.377275+0000","last_active":"2024-06-29T14:37:54.397340+0000","last_peered":"2024-06-29T14:37:54.397340+0000","last_clean":"2024-06-29T14:37:54.397340+0000","last_became_active":"2024-06-29T14:37:52.376947+0000","last_became_peered":"2024-06-29T14:37:52.376947+0000","last_unstale":"2024-06-29T14:37:54.397340+0000","last_undegraded":"2024-06-29T14:37:54.397340+0000","last_fullsized":"2024-06-29T14:37:54.397340+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:34.220027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373314+0000","last_change":"2024-06-29T14:37:53.383101+0000","last_active":"2024-06-29T14:37:54.373314+0000","last_peered":"2024-06-29T14:37:54.373314+0000","last_clean":"2024-06-29T14:37:54.373314+0000","last_became_active":"2024-06-29T14:37:53.382775+0000","last_became_peered":"2024-06-29T14:37:53.382775+0000","last_unstale":"2024-06-29T14:37:54.373314+0000","last_undegraded":"2024-06-29T14:37:54.373314+0000","last_fullsized":"2024-06-29T14:37:54.373314+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:32.510147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377338+0000","last_change":"2024-06-29T14:37:52.372210+0000","last_active":"2024-06-29T14:37:54.377338+0000","last_peered":"2024-06-29T14:37:54.377338+0000","last_clean":"2024-06-29T14:37:54.377338+0000","last_became_active":"2024-06-29T14:37:52.371896+0000","last_became_peered":"2024-06-29T14:37:52.371896+0000","last_unstale":"2024-06-29T14:37:54.377338+0000","last_undegraded":"2024-06-29T14:37:54.377338+0000","last_fullsized":"2024-06-29T14:37:54.377338+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:07.184093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399385+0000","last_change":"2024-06-29T14:37:53.386600+0000","last_active":"2024-06-29T14:37:54.399385+0000","last_peered":"2024-06-29T14:37:54.399385+0000","last_clean":"2024-06-29T14:37:54.399385+0000","last_became_active":"2024-06-29T14:37:53.386248+0000","last_became_peered":"2024-06-29T14:37:53.386248+0000","last_unstale":"2024-06-29T14:37:54.399385+0000","last_undegraded":"2024-06-29T14:37:54.399385+0000","last_fullsized":"2024-06-29T14:37:54.399385+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:50.022396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501756+0000","last_change":"2024-06-29T14:37:52.364386+0000","last_active":"2024-06-29T14:37:54.501756+0000","last_peered":"2024-06-29T14:37:54.501756+0000","last_clean":"2024-06-29T14:37:54.501756+0000","last_became_active":"2024-06-29T14:37:52.363965+0000","last_became_peered":"2024-06-29T14:37:52.363965+0000","last_unstale":"2024-06-29T14:37:54.501756+0000","last_undegraded":"2024-06-29T14:37:54.501756+0000","last_fullsized":"2024-06-29T14:37:54.501756+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:58:22.236585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399856+0000","last_change":"2024-06-29T14:37:53.402834+0000","last_active":"2024-06-29T14:37:54.399856+0000","last_peered":"2024-06-29T14:37:54.399856+0000","last_clean":"2024-06-29T14:37:54.399856+0000","last_became_active":"2024-06-29T14:37:53.402513+0000","last_became_peered":"2024-06-29T14:37:53.402513+0000","last_unstale":"2024-06-29T14:37:54.399856+0000","last_undegraded":"2024-06-29T14:37:54.399856+0000","last_fullsized":"2024-06-29T14:37:54.399856+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:17.123778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"100'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399964+0000","last_change":"2024-06-29T14:37:52.365260+0000","last_active":"2024-06-29T14:37:54.399964+0000","last_peered":"2024-06-29T14:37:54.399964+0000","last_clean":"2024-06-29T14:37:54.399964+0000","last_became_active":"2024-06-29T14:37:52.364944+0000","last_became_peered":"2024-06-29T14:37:52.364944+0000","last_unstale":"2024-06-29T14:37:54.399964+0000","last_undegraded":"2024-06-29T14:37:54.399964+0000","last_fullsized":"2024-06-29T14:37:54.399964+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:07.824092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373601+0000","last_change":"2024-06-29T14:37:53.387309+0000","last_active":"2024-06-29T14:37:54.373601+0000","last_peered":"2024-06-29T14:37:54.373601+0000","last_clean":"2024-06-29T14:37:54.373601+0000","last_became_active":"2024-06-29T14:37:53.386992+0000","last_became_peered":"2024-06-29T14:37:53.386992+0000","last_unstale":"2024-06-29T14:37:54.373601+0000","last_undegraded":"2024-06-29T14:37:54.373601+0000","last_fullsized":"2024-06-29T14:37:54.373601+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:33:08.670834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500589+0000","last_change":"2024-06-29T14:37:52.377838+0000","last_active":"2024-06-29T14:37:54.500589+0000","last_peered":"2024-06-29T14:37:54.500589+0000","last_clean":"2024-06-29T14:37:54.500589+0000","last_became_active":"2024-06-29T14:37:52.377501+0000","last_became_peered":"2024-06-29T14:37:52.377501+0000","last_unstale":"2024-06-29T14:37:54.500589+0000","last_undegraded":"2024-06-29T14:37:54.500589+0000","last_fullsized":"2024-06-29T14:37:54.500589+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:36.851395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397379+0000","last_change":"2024-06-29T14:37:53.408012+0000","last_active":"2024-06-29T14:37:54.397379+0000","last_peered":"2024-06-29T14:37:54.397379+0000","last_clean":"2024-06-29T14:37:54.397379+0000","last_became_active":"2024-06-29T14:37:53.407646+0000","last_became_peered":"2024-06-29T14:37:53.407646+0000","last_unstale":"2024-06-29T14:37:54.397379+0000","last_undegraded":"2024-06-29T14:37:54.397379+0000","last_fullsized":"2024-06-29T14:37:54.397379+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:41:19.482180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373570+0000","last_change":"2024-06-29T14:37:52.368873+0000","last_active":"2024-06-29T14:37:54.373570+0000","last_peered":"2024-06-29T14:37:54.373570+0000","last_clean":"2024-06-29T14:37:54.373570+0000","last_became_active":"2024-06-29T14:37:52.368563+0000","last_became_peered":"2024-06-29T14:37:52.368563+0000","last_unstale":"2024-06-29T14:37:54.373570+0000","last_undegraded":"2024-06-29T14:37:54.373570+0000","last_fullsized":"2024-06-29T14:37:54.373570+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:54:28.883517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397423+0000","last_change":"2024-06-29T14:37:53.407244+0000","last_active":"2024-06-29T14:37:54.397423+0000","last_peered":"2024-06-29T14:37:54.397423+0000","last_clean":"2024-06-29T14:37:54.397423+0000","last_became_active":"2024-06-29T14:37:53.406909+0000","last_became_peered":"2024-06-29T14:37:53.406909+0000","last_unstale":"2024-06-29T14:37:54.397423+0000","last_undegraded":"2024-06-29T14:37:54.397423+0000","last_fullsized":"2024-06-29T14:37:54.397423+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:52:37.676943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373415+0000","last_change":"2024-06-29T14:37:52.374678+0000","last_active":"2024-06-29T14:37:54.373415+0000","last_peered":"2024-06-29T14:37:54.373415+0000","last_clean":"2024-06-29T14:37:54.373415+0000","last_became_active":"2024-06-29T14:37:52.374278+0000","last_became_peered":"2024-06-29T14:37:52.374278+0000","last_unstale":"2024-06-29T14:37:54.373415+0000","last_undegraded":"2024-06-29T14:37:54.373415+0000","last_fullsized":"2024-06-29T14:37:54.373415+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:58.929743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378079+0000","last_change":"2024-06-29T14:37:53.383274+0000","last_active":"2024-06-29T14:37:54.378079+0000","last_peered":"2024-06-29T14:37:54.378079+0000","last_clean":"2024-06-29T14:37:54.378079+0000","last_became_active":"2024-06-29T14:37:53.382901+0000","last_became_peered":"2024-06-29T14:37:53.382901+0000","last_unstale":"2024-06-29T14:37:54.378079+0000","last_undegraded":"2024-06-29T14:37:54.378079+0000","last_fullsized":"2024-06-29T14:37:54.378079+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:32.247587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.373509+0000","last_change":"2024-06-29T14:37:52.363357+0000","last_active":"2024-06-29T14:37:54.373509+0000","last_peered":"2024-06-29T14:37:54.373509+0000","last_clean":"2024-06-29T14:37:54.373509+0000","last_became_active":"2024-06-29T14:37:52.363018+0000","last_became_peered":"2024-06-29T14:37:52.363018+0000","last_unstale":"2024-06-29T14:37:54.373509+0000","last_undegraded":"2024-06-29T14:37:54.373509+0000","last_fullsized":"2024-06-29T14:37:54.373509+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:02:36.554625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502068+0000","last_change":"2024-06-29T14:37:53.381661+0000","last_active":"2024-06-29T14:37:54.502068+0000","last_peered":"2024-06-29T14:37:54.502068+0000","last_clean":"2024-06-29T14:37:54.502068+0000","last_became_active":"2024-06-29T14:37:53.381367+0000","last_became_peered":"2024-06-29T14:37:53.381367+0000","last_unstale":"2024-06-29T14:37:54.502068+0000","last_undegraded":"2024-06-29T14:37:54.502068+0000","last_fullsized":"2024-06-29T14:37:54.502068+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:37:40.218294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"100'2","reported_seq":16,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.501918+0000","last_change":"2024-06-29T14:37:52.365709+0000","last_active":"2024-06-29T14:37:54.501918+0000","last_peered":"2024-06-29T14:37:54.501918+0000","last_clean":"2024-06-29T14:37:54.501918+0000","last_became_active":"2024-06-29T14:37:52.365353+0000","last_became_peered":"2024-06-29T14:37:52.365353+0000","last_unstale":"2024-06-29T14:37:54.501918+0000","last_undegraded":"2024-06-29T14:37:54.501918+0000","last_fullsized":"2024-06-29T14:37:54.501918+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:19.432254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377337+0000","last_change":"2024-06-29T14:37:53.388153+0000","last_active":"2024-06-29T14:37:54.377337+0000","last_peered":"2024-06-29T14:37:54.377337+0000","last_clean":"2024-06-29T14:37:54.377337+0000","last_became_active":"2024-06-29T14:37:53.387848+0000","last_became_peered":"2024-06-29T14:37:53.387848+0000","last_unstale":"2024-06-29T14:37:54.377337+0000","last_undegraded":"2024-06-29T14:37:54.377337+0000","last_fullsized":"2024-06-29T14:37:54.377337+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:26.121311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"101'1","reported_seq":15,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.398690+0000","last_change":"2024-06-29T14:37:52.380126+0000","last_active":"2024-06-29T14:37:54.398690+0000","last_peered":"2024-06-29T14:37:54.398690+0000","last_clean":"2024-06-29T14:37:54.398690+0000","last_became_active":"2024-06-29T14:37:52.379705+0000","last_became_peered":"2024-06-29T14:37:52.379705+0000","last_unstale":"2024-06-29T14:37:54.398690+0000","last_undegraded":"2024-06-29T14:37:54.398690+0000","last_fullsized":"2024-06-29T14:37:54.398690+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:15.384107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.502122+0000","last_change":"2024-06-29T14:37:53.386061+0000","last_active":"2024-06-29T14:37:54.502122+0000","last_peered":"2024-06-29T14:37:54.502122+0000","last_clean":"2024-06-29T14:37:54.502122+0000","last_became_active":"2024-06-29T14:37:53.385707+0000","last_became_peered":"2024-06-29T14:37:53.385707+0000","last_unstale":"2024-06-29T14:37:54.502122+0000","last_undegraded":"2024-06-29T14:37:54.502122+0000","last_fullsized":"2024-06-29T14:37:54.502122+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:01.014097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377655+0000","last_change":"2024-06-29T14:37:52.370103+0000","last_active":"2024-06-29T14:37:54.377655+0000","last_peered":"2024-06-29T14:37:54.377655+0000","last_clean":"2024-06-29T14:37:54.377655+0000","last_became_active":"2024-06-29T14:37:52.369784+0000","last_became_peered":"2024-06-29T14:37:52.369784+0000","last_unstale":"2024-06-29T14:37:54.377655+0000","last_undegraded":"2024-06-29T14:37:54.377655+0000","last_fullsized":"2024-06-29T14:37:54.377655+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:22:48.722841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.397450+0000","last_change":"2024-06-29T14:37:53.405338+0000","last_active":"2024-06-29T14:37:54.397450+0000","last_peered":"2024-06-29T14:37:54.397450+0000","last_clean":"2024-06-29T14:37:54.397450+0000","last_became_active":"2024-06-29T14:37:53.405000+0000","last_became_peered":"2024-06-29T14:37:53.405000+0000","last_unstale":"2024-06-29T14:37:54.397450+0000","last_undegraded":"2024-06-29T14:37:54.397450+0000","last_fullsized":"2024-06-29T14:37:54.397450+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:45.965101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376932+0000","last_change":"2024-06-29T14:37:52.372455+0000","last_active":"2024-06-29T14:37:54.376932+0000","last_peered":"2024-06-29T14:37:54.376932+0000","last_clean":"2024-06-29T14:37:54.376932+0000","last_became_active":"2024-06-29T14:37:52.372119+0000","last_became_peered":"2024-06-29T14:37:52.372119+0000","last_unstale":"2024-06-29T14:37:54.376932+0000","last_undegraded":"2024-06-29T14:37:54.376932+0000","last_fullsized":"2024-06-29T14:37:54.376932+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:43.766024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.378080+0000","last_change":"2024-06-29T14:37:53.384508+0000","last_active":"2024-06-29T14:37:54.378080+0000","last_peered":"2024-06-29T14:37:54.378080+0000","last_clean":"2024-06-29T14:37:54.378080+0000","last_became_active":"2024-06-29T14:37:53.384154+0000","last_became_peered":"2024-06-29T14:37:53.384154+0000","last_unstale":"2024-06-29T14:37:54.378080+0000","last_undegraded":"2024-06-29T14:37:54.378080+0000","last_fullsized":"2024-06-29T14:37:54.378080+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:34.922824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377614+0000","last_change":"2024-06-29T14:37:52.368381+0000","last_active":"2024-06-29T14:37:54.377614+0000","last_peered":"2024-06-29T14:37:54.377614+0000","last_clean":"2024-06-29T14:37:54.377614+0000","last_became_active":"2024-06-29T14:37:52.368071+0000","last_became_peered":"2024-06-29T14:37:52.368071+0000","last_unstale":"2024-06-29T14:37:54.377614+0000","last_undegraded":"2024-06-29T14:37:54.377614+0000","last_fullsized":"2024-06-29T14:37:54.377614+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:15:44.081382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377174+0000","last_change":"2024-06-29T14:37:53.380753+0000","last_active":"2024-06-29T14:37:54.377174+0000","last_peered":"2024-06-29T14:37:54.377174+0000","last_clean":"2024-06-29T14:37:54.377174+0000","last_became_active":"2024-06-29T14:37:53.380387+0000","last_became_peered":"2024-06-29T14:37:53.380387+0000","last_unstale":"2024-06-29T14:37:54.377174+0000","last_undegraded":"2024-06-29T14:37:54.377174+0000","last_fullsized":"2024-06-29T14:37:54.377174+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:01.014097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.399877+0000","last_change":"2024-06-29T14:37:52.370239+0000","last_active":"2024-06-29T14:37:54.399877+0000","last_peered":"2024-06-29T14:37:54.399877+0000","last_clean":"2024-06-29T14:37:54.399877+0000","last_became_active":"2024-06-29T14:37:52.369936+0000","last_became_peered":"2024-06-29T14:37:52.369936+0000","last_unstale":"2024-06-29T14:37:54.399877+0000","last_undegraded":"2024-06-29T14:37:54.399877+0000","last_fullsized":"2024-06-29T14:37:54.399877+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:07.211445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500732+0000","last_change":"2024-06-29T14:37:53.406486+0000","last_active":"2024-06-29T14:37:54.500732+0000","last_peered":"2024-06-29T14:37:54.500732+0000","last_clean":"2024-06-29T14:37:54.500732+0000","last_became_active":"2024-06-29T14:37:53.406114+0000","last_became_peered":"2024-06-29T14:37:53.406114+0000","last_unstale":"2024-06-29T14:37:54.500732+0000","last_undegraded":"2024-06-29T14:37:54.500732+0000","last_fullsized":"2024-06-29T14:37:54.500732+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:11.537473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377447+0000","last_change":"2024-06-29T14:37:52.365139+0000","last_active":"2024-06-29T14:37:54.377447+0000","last_peered":"2024-06-29T14:37:54.377447+0000","last_clean":"2024-06-29T14:37:54.377447+0000","last_became_active":"2024-06-29T14:37:52.364711+0000","last_became_peered":"2024-06-29T14:37:52.364711+0000","last_unstale":"2024-06-29T14:37:54.377447+0000","last_undegraded":"2024-06-29T14:37:54.377447+0000","last_fullsized":"2024-06-29T14:37:54.377447+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:58:22.236585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500826+0000","last_change":"2024-06-29T14:37:53.392368+0000","last_active":"2024-06-29T14:37:54.500826+0000","last_peered":"2024-06-29T14:37:54.500826+0000","last_clean":"2024-06-29T14:37:54.500826+0000","last_became_active":"2024-06-29T14:37:53.392022+0000","last_became_peered":"2024-06-29T14:37:53.392022+0000","last_unstale":"2024-06-29T14:37:54.500826+0000","last_undegraded":"2024-06-29T14:37:54.500826+0000","last_fullsized":"2024-06-29T14:37:54.500826+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:41:21.804940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.500365+0000","last_change":"2024-06-29T14:37:52.373923+0000","last_active":"2024-06-29T14:37:54.500365+0000","last_peered":"2024-06-29T14:37:54.500365+0000","last_clean":"2024-06-29T14:37:54.500365+0000","last_became_active":"2024-06-29T14:37:52.373594+0000","last_became_peered":"2024-06-29T14:37:52.373594+0000","last_unstale":"2024-06-29T14:37:54.500365+0000","last_undegraded":"2024-06-29T14:37:54.500365+0000","last_fullsized":"2024-06-29T14:37:54.500365+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:27.635671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.377538+0000","last_change":"2024-06-29T14:37:53.391960+0000","last_active":"2024-06-29T14:37:54.377538+0000","last_peered":"2024-06-29T14:37:54.377538+0000","last_clean":"2024-06-29T14:37:54.377538+0000","last_became_active":"2024-06-29T14:37:53.391629+0000","last_became_peered":"2024-06-29T14:37:53.391629+0000","last_unstale":"2024-06-29T14:37:54.377538+0000","last_undegraded":"2024-06-29T14:37:54.377538+0000","last_fullsized":"2024-06-29T14:37:54.377538+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.348361+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.348361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:03.508237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.374057+0000","last_change":"2024-06-29T14:37:52.374339+0000","last_active":"2024-06-29T14:37:54.374057+0000","last_peered":"2024-06-29T14:37:54.374057+0000","last_clean":"2024-06-29T14:37:54.374057+0000","last_became_active":"2024-06-29T14:37:52.374029+0000","last_became_peered":"2024-06-29T14:37:52.374029+0000","last_unstale":"2024-06-29T14:37:54.374057+0000","last_undegraded":"2024-06-29T14:37:54.374057+0000","last_fullsized":"2024-06-29T14:37:54.374057+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.342397+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.342397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:19.432254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'114","reported_seq":418,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.376935+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:37:54.376935+0000","last_peered":"2024-06-29T14:37:54.376935+0000","last_clean":"2024-06-29T14:37:54.376935+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:37:54.376935+0000","last_undegraded":"2024-06-29T14:37:54.376935+0000","last_fullsized":"2024-06-29T14:37:54.376935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":173,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":173,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542339,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59324,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312516,"statfs":{"total":96636764160,"available":96576016384,"internally_reserved":0,"allocated":1814528,"data_stored":1202872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542340,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1241088,"data_stored":620786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542339,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50432,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321408,"statfs":{"total":96636764160,"available":96585121792,"internally_reserved":0,"allocated":1232896,"data_stored":619208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1236992,"data_stored":614198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509636,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1232896,"data_stored":619756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575044,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58760,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313080,"statfs":{"total":96636764160,"available":96576593920,"internally_reserved":0,"allocated":1241088,"data_stored":619916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509635,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1236992,"data_stored":619286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509637,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32968,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338872,"statfs":{"total":96636764160,"available":96603004928,"internally_reserved":0,"allocated":1822720,"data_stored":1209542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:37:57.852 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:37:57.852 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:37:58.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.016+0000 7f62253b2640 1 -- 172.21.15.59:0/2433859882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c003a30 msgr2=0x7f620c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.016+0000 7f62253b2640 1 --2- 172.21.15.59:0/2433859882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c003a30 0x7f620c003ea0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f62100016c0 tx=0x7f620c005830 comp rx=0 tx=0).stop 2024-06-29T14:37:58.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.016+0000 7f621f7fe640 1 -- 172.21.15.59:0/2433859882 reap_dead start 2024-06-29T14:37:58.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.017+0000 7f62253b2640 1 -- 172.21.15.59:0/2433859882 shutdown_connections 2024-06-29T14:37:58.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.017+0000 7f62253b2640 1 -- 172.21.15.59:0/2433859882 >> 172.21.15.59:0/2433859882 conn(0x7f6220073ef0 msgr2=0x7f6220074300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.017+0000 7f62253b2640 1 -- 172.21.15.59:0/2433859882 shutdown_connections 2024-06-29T14:37:58.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.017+0000 7f62253b2640 1 -- 172.21.15.59:0/2433859882 wait complete. 2024-06-29T14:37:58.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.019+0000 7f62253b2640 1 Processor -- start 2024-06-29T14:37:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.019+0000 7f62253b2640 1 -- start start 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f62253b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c003a30 0x7f62201e8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f62253b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f622013f1e0 0x7f622013f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f62253b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6220141a30 0x7f6220141e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f62253b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f62200d0cf0 con 0x7f622013f1e0 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f62253b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62200d0b70 con 0x7f620c003a30 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f62253b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f62200b5050 con 0x7f6220141a30 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f622013f1e0 0x7f622013f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f622013f1e0 0x7f622013f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33674/0 (socket says 172.21.15.59:33674) 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621e7fc640 1 -- 172.21.15.59:0/2109034510 learned_addr learned my addr 172.21.15.59:0/2109034510 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621effd640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c003a30 0x7f62201e8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621f7fe640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6220141a30 0x7f6220141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621e7fc640 1 -- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6220141a30 msgr2=0x7f6220141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621e7fc640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6220141a30 0x7f6220141e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621e7fc640 1 -- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c003a30 msgr2=0x7f62201e8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621e7fc640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c003a30 0x7f62201e8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621e7fc640 1 -- 172.21.15.59:0/2109034510 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6220140c70 con 0x7f622013f1e0 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.020+0000 7f621f7fe640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6220141a30 0x7f6220141e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.021+0000 7f621e7fc640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f622013f1e0 0x7f622013f5f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f6208006680 tx=0x7f62080066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.021+0000 7f62177fe640 1 -- 172.21.15.59:0/2109034510 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208015070 con 0x7f622013f1e0 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.021+0000 7f62177fe640 1 -- 172.21.15.59:0/2109034510 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6208011940 con 0x7f622013f1e0 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.021+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6220140f50 con 0x7f622013f1e0 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.021+0000 7f62177fe640 1 -- 172.21.15.59:0/2109034510 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208011d90 con 0x7f622013f1e0 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.021+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62201e8ae0 con 0x7f622013f1e0 2024-06-29T14:37:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.022+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62200b55e0 con 0x7f622013f1e0 2024-06-29T14:37:58.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.023+0000 7f62177fe640 1 -- 172.21.15.59:0/2109034510 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f6208018070 con 0x7f622013f1e0 2024-06-29T14:37:58.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.023+0000 7f62177fe640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f62000763f0 0x7f6200078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.023+0000 7f62177fe640 1 -- 172.21.15.59:0/2109034510 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f6208098860 con 0x7f622013f1e0 2024-06-29T14:37:58.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.024+0000 7f621effd640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f62000763f0 0x7f6200078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.024+0000 7f621effd640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f62000763f0 0x7f6200078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6210032000 tx=0x7f621002c040 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.028+0000 7f62177fe640 1 -- 172.21.15.59:0/2109034510 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f62080617c0 con 0x7f622013f1e0 2024-06-29T14:37:58.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.294+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f622015bbe0 con 0x7f622013f1e0 2024-06-29T14:37:58.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.296+0000 7f62177fe640 1 -- 172.21.15.59:0/2109034510 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f6208065ac0 con 0x7f622013f1e0 2024-06-29T14:37:58.295 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:37:58.295 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:37:58.295 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T14:37:58.295 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T14:37:58.295 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:37:58.295 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T14:37:58.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f62000763f0 msgr2=0x7f6200078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f62000763f0 0x7f6200078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6210032000 tx=0x7f621002c040 comp rx=0 tx=0).stop 2024-06-29T14:37:58.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f622013f1e0 msgr2=0x7f622013f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f622013f1e0 0x7f622013f5f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f6208006680 tx=0x7f62080066b0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 shutdown_connections 2024-06-29T14:37:58.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f62000763f0 0x7f6200078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6220141a30 0x7f6220141e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f622013f1e0 0x7f622013f5f0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 --2- 172.21.15.59:0/2109034510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c003a30 0x7f62201e8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 >> 172.21.15.59:0/2109034510 conn(0x7f6220073ef0 msgr2=0x7f62200b57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 shutdown_connections 2024-06-29T14:37:58.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.299+0000 7f62253b2640 1 -- 172.21.15.59:0/2109034510 wait complete. 2024-06-29T14:37:58.308 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:58.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.428+0000 7f15766ef640 1 -- 172.21.15.59:0/3958766549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157013e9b0 msgr2=0x7f157013edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.428+0000 7f15766ef640 1 --2- 172.21.15.59:0/3958766549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157013e9b0 0x7f157013edc0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1570073da0 tx=0x7f156402a4a0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.428+0000 7f15766ef640 1 -- 172.21.15.59:0/3958766549 shutdown_connections 2024-06-29T14:37:58.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.428+0000 7f15766ef640 1 --2- 172.21.15.59:0/3958766549 >> v2:172.21.15.153:3300/0 conn(0x7f15701414c0 0x7f15701418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.428+0000 7f15766ef640 1 --2- 172.21.15.59:0/3958766549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157013e9b0 0x7f157013edc0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.428+0000 7f15766ef640 1 -- 172.21.15.59:0/3958766549 >> 172.21.15.59:0/3958766549 conn(0x7f15700b6fb0 msgr2=0x7f15700b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.428+0000 7f15766ef640 1 -- 172.21.15.59:0/3958766549 shutdown_connections 2024-06-29T14:37:58.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.428+0000 7f15766ef640 1 -- 172.21.15.59:0/3958766549 wait complete. 2024-06-29T14:37:58.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.430+0000 7f15766ef640 1 Processor -- start 2024-06-29T14:37:58.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.430+0000 7f15766ef640 1 -- start start 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.431+0000 7f15766ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157013e9b0 0x7f157013bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.431+0000 7f15766ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15701414c0 0x7f15700c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.431+0000 7f15766ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157015ceb0 0x7f15701f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.431+0000 7f15766ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f15700b29d0 con 0x7f157015ceb0 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.431+0000 7f15766ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15700b2850 con 0x7f157013e9b0 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.431+0000 7f15766ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f15700b2b50 con 0x7f15701414c0 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.431+0000 7f1574c8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157015ceb0 0x7f15701f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.431+0000 7f1574c8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157015ceb0 0x7f15701f2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33686/0 (socket says 172.21.15.59:33686) 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f1574c8c640 1 -- 172.21.15.59:0/897615878 learned_addr learned my addr 172.21.15.59:0/897615878 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f156ffff640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157013e9b0 0x7f157013bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f156f7fe640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15701414c0 0x7f15700c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f1574c8c640 1 -- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15701414c0 msgr2=0x7f15700c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f1574c8c640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15701414c0 0x7f15700c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f1574c8c640 1 -- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157013e9b0 msgr2=0x7f157013bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f1574c8c640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157013e9b0 0x7f157013bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f1574c8c640 1 -- 172.21.15.59:0/897615878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15700c6f40 con 0x7f157015ceb0 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f156f7fe640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15701414c0 0x7f15700c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f1574c8c640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157015ceb0 0x7f15701f2c70 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f156800ea80 tx=0x7f156800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f156cff9640 1 -- 172.21.15.59:0/897615878 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156800ce40 con 0x7f157015ceb0 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f156cff9640 1 -- 172.21.15.59:0/897615878 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1568004b20 con 0x7f157015ceb0 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15700c75d0 con 0x7f157015ceb0 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f156cff9640 1 -- 172.21.15.59:0/897615878 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1568005020 con 0x7f157015ceb0 2024-06-29T14:37:58.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.432+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f157013b770 con 0x7f157015ceb0 2024-06-29T14:37:58.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.433+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15700714f0 con 0x7f157015ceb0 2024-06-29T14:37:58.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.434+0000 7f156cff9640 1 -- 172.21.15.59:0/897615878 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f15680051c0 con 0x7f157015ceb0 2024-06-29T14:37:58.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.435+0000 7f156cff9640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f155c0763f0 0x7f155c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.435+0000 7f156cff9640 1 -- 172.21.15.59:0/897615878 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f1568098730 con 0x7f157015ceb0 2024-06-29T14:37:58.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.435+0000 7f156ffff640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f155c0763f0 0x7f155c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.436+0000 7f156ffff640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f155c0763f0 0x7f155c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1560007d60 tx=0x7f1560007cf0 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.439+0000 7f156cff9640 1 -- 172.21.15.59:0/897615878 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f15700714f0 con 0x7f157015ceb0 2024-06-29T14:37:58.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.673+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15700b3000 con 0x7f157015ceb0 2024-06-29T14:37:58.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.674+0000 7f156cff9640 1 -- 172.21.15.59:0/897615878 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5228 (secure 0 0 0) 0x7f1568061c50 con 0x7f157015ceb0 2024-06-29T14:37:58.672 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:37:58.672 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":46,"btime":"2024-06-29T14:37:56:391508+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2174310652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2174310652},{"type":"v1","addr":"172.21.15.59:6835","nonce":2174310652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3217643445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3217643445},{"type":"v1","addr":"172.21.15.153:6836","nonce":3217643445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/832666341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":832666341},{"type":"v1","addr":"172.21.15.59:6833","nonce":832666341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.753189+0000","modified":"2024-06-29T14:37:55.781006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7365},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7365":{"gid":7365,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/1942687234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1942687234},{"type":"v1","addr":"172.21.15.153:6837","nonce":1942687234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7365,"qdb_cluster":[7365]},"id":5}]} 2024-06-29T14:37:58.673 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 46 2024-06-29T14:37:58.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f155c0763f0 msgr2=0x7f155c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f155c0763f0 0x7f155c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1560007d60 tx=0x7f1560007cf0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157015ceb0 msgr2=0x7f15701f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157015ceb0 0x7f15701f2c70 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f156800ea80 tx=0x7f156800ef40 comp rx=0 tx=0).stop 2024-06-29T14:37:58.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 shutdown_connections 2024-06-29T14:37:58.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f157015ceb0 0x7f15701f2c70 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f155c0763f0 0x7f155c078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f15701414c0 0x7f15700c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 --2- 172.21.15.59:0/897615878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f157013e9b0 0x7f157013bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 >> 172.21.15.59:0/897615878 conn(0x7f15700b6fb0 msgr2=0x7f15700b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 shutdown_connections 2024-06-29T14:37:58.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:58.676+0000 7f15766ef640 1 -- 172.21.15.59:0/897615878 wait complete. 2024-06-29T14:37:59.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:37:52.753189+0000', 'modified': '2024-06-29T14:37:55.781006+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7365}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7365': {'gid': 7365, 'name': 'b', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6837/1942687234', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6835', 'nonce': 1942687234}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 1942687234}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7365, 'qdb_cluster': [7365]} 2024-06-29T14:37:59.088 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:37:59.088 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T14:37:59.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.212+0000 7f23134d6640 1 -- 172.21.15.59:0/4192620847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c162800 msgr2=0x7f230c162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.212+0000 7f23134d6640 1 --2- 172.21.15.59:0/4192620847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c162800 0x7f230c162c10 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f230c0cfbc0 tx=0x7f230402a9e0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.212+0000 7f23134d6640 1 -- 172.21.15.59:0/4192620847 shutdown_connections 2024-06-29T14:37:59.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.212+0000 7f23134d6640 1 --2- 172.21.15.59:0/4192620847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c162800 0x7f230c162c10 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.212+0000 7f23134d6640 1 --2- 172.21.15.59:0/4192620847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230c0ce950 0x7f230c1622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.212+0000 7f23134d6640 1 -- 172.21.15.59:0/4192620847 >> 172.21.15.59:0/4192620847 conn(0x7f230c0b5610 msgr2=0x7f230c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.212+0000 7f23134d6640 1 -- 172.21.15.59:0/4192620847 shutdown_connections 2024-06-29T14:37:59.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.212+0000 7f23134d6640 1 -- 172.21.15.59:0/4192620847 wait complete. 2024-06-29T14:37:59.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.214+0000 7f23134d6640 1 Processor -- start 2024-06-29T14:37:59.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.214+0000 7f23134d6640 1 -- start start 2024-06-29T14:37:59.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f23134d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f230c08c190 0x7f230c0aafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f23134d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c0ce950 0x7f230c0ac240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f23134d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230c162800 0x7f230c0ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f23134d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f230c0b7fa0 con 0x7f230c0ce950 2024-06-29T14:37:59.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f23134d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f230c0b7e20 con 0x7f230c08c190 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f23134d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f230c0b8120 con 0x7f230c162800 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2310a71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c0ce950 0x7f230c0ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2310a71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c0ce950 0x7f230c0ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33690/0 (socket says 172.21.15.59:33690) 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2310a71640 1 -- 172.21.15.59:0/1299159489 learned_addr learned my addr 172.21.15.59:0/1299159489 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2311a73640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230c162800 0x7f230c0ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2311272640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f230c08c190 0x7f230c0aafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2310a71640 1 -- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230c162800 msgr2=0x7f230c0ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2310a71640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230c162800 0x7f230c0ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2310a71640 1 -- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f230c08c190 msgr2=0x7f230c0aafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2310a71640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f230c08c190 0x7f230c0aafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.215+0000 7f2310a71640 1 -- 172.21.15.59:0/1299159489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230c0cd390 con 0x7f230c0ce950 2024-06-29T14:37:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.216+0000 7f2310a71640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c0ce950 0x7f230c0ac240 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00d740 tx=0x7f22fc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.216+0000 7f2301ffb640 1 -- 172.21.15.59:0/1299159489 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc017070 con 0x7f230c0ce950 2024-06-29T14:37:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.216+0000 7f2301ffb640 1 -- 172.21.15.59:0/1299159489 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22fc009d20 con 0x7f230c0ce950 2024-06-29T14:37:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.216+0000 7f23134d6640 1 -- 172.21.15.59:0/1299159489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230c0cd890 con 0x7f230c0ce950 2024-06-29T14:37:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.216+0000 7f2301ffb640 1 -- 172.21.15.59:0/1299159489 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0106c0 con 0x7f230c0ce950 2024-06-29T14:37:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.216+0000 7f23134d6640 1 -- 172.21.15.59:0/1299159489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f230c0cabd0 con 0x7f230c0ce950 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.218+0000 7f2301ffb640 1 -- 172.21.15.59:0/1299159489 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f22fc01a070 con 0x7f230c0ce950 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.218+0000 7f2301ffb640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f22f00763f0 0x7f22f0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.219+0000 7f2301ffb640 1 -- 172.21.15.59:0/1299159489 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f22fc097d80 con 0x7f230c0ce950 2024-06-29T14:37:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.219+0000 7f2311272640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f22f00763f0 0x7f22f0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.219+0000 7f231a390740 1 Processor -- start 2024-06-29T14:37:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.219+0000 7f231a390740 1 -- start start 2024-06-29T14:37:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.219+0000 7f2311272640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f22f00763f0 0x7f22f0078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2308006fd0 tx=0x7f2308009060 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.220+0000 7f231a390740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 0x560a4e9ac6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.220+0000 7f231a390740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9903a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.220+0000 7f231a390740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9908e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.220+0000 7f231a390740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x560a4e928bc0 con 0x560a4e59f590 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.220+0000 7f231a390740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560a4e8d7630 con 0x560a4e5973b0 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.220+0000 7f231a390740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x560a4e8c6bd0 con 0x560a4e5c0270 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9908e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9908e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33704/0 (socket says 172.21.15.59:33704) 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 -- 172.21.15.59:0/1396396915 learned_addr learned my addr 172.21.15.59:0/1396396915 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311272640 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 0x560a4e9ac6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2310a71640 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9903a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 -- 172.21.15.59:0/1396396915 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 msgr2=0x560a4e9ac6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 0x560a4e9ac6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 -- 172.21.15.59:0/1396396915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 msgr2=0x560a4e9903a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9903a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 -- 172.21.15.59:0/1396396915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a4e8c3ca0 con 0x560a4e59f590 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f2311a73640 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9908e0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f2304005fd0 tx=0x7f2304004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2070500ad96f760 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f22f6ffd640 1 -- 172.21.15.59:0/1396396915 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2304033030 con 0x560a4e59f590 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f22f6ffd640 1 -- 172.21.15.59:0/1396396915 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2304003d30 con 0x560a4e59f590 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.221+0000 7f22f6ffd640 1 -- 172.21.15.59:0/1396396915 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230402fb70 con 0x560a4e59f590 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 -- 172.21.15.59:0/1396396915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 msgr2=0x560a4e9908e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9908e0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f2304005fd0 tx=0x7f2304004ab0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 -- 172.21.15.59:0/1396396915 shutdown_connections 2024-06-29T14:37:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9908e0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9903a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 --2- 172.21.15.59:0/1396396915 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 0x560a4e9ac6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 -- 172.21.15.59:0/1396396915 >> 172.21.15.59:0/1396396915 conn(0x560a4e5b43c0 msgr2=0x560a4e963f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 -- 172.21.15.59:0/1396396915 shutdown_connections 2024-06-29T14:37:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.222+0000 7f231a390740 1 -- 172.21.15.59:0/1396396915 wait complete. 2024-06-29T14:37:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.223+0000 7f231a390740 1 -- start start 2024-06-29T14:37:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.223+0000 7f231a390740 1 Processor -- start 2024-06-29T14:37:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f231a390740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9a5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f231a390740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f231a390740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 0x560a4e9be760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f231a390740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x560a4e864ae0 con 0x560a4e59f590 2024-06-29T14:37:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f231a390740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560a4e8cac10 con 0x560a4e5973b0 2024-06-29T14:37:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f231a390740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x560a4e63a620 con 0x560a4e5c0270 2024-06-29T14:37:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f231a390740 10 client.0 _ll_register_callbacks cb 0x560a4e871480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:37:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33710/0 (socket says 172.21.15.59:33710) 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 -- 172.21.15.59:0/1296885704 learned_addr learned my addr 172.21.15.59:0/1296885704 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2311a73640 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 0x560a4e9be760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2311272640 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 -- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 msgr2=0x560a4e9be760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 0x560a4e9be760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 -- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 msgr2=0x560a4e9a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 -- 172.21.15.59:0/1296885704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a4e62b850 con 0x560a4e59f590 2024-06-29T14:37:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2311272640 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f2310a71640 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987dd0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f22fc05deb0 tx=0x7f22fc0606c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c8774beb00220e server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f22ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f22ebfff640 1 -- 172.21.15.59:0/1296885704 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc079070 con 0x560a4e59f590 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f22ebfff640 1 -- 172.21.15.59:0/1296885704 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f22fc05c5b0 con 0x560a4e59f590 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.225+0000 7f231a390740 1 -- 172.21.15.59:0/1296885704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2304003d30 con 0x560a4e59f590 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f22ebfff640 1 -- 172.21.15.59:0/1296885704 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f22fc021040 con 0x560a4e59f590 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f231a390740 10 client.7509 fetch_fsmap learned FSMap version 46 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f231a390740 1 -- 172.21.15.59:0/1296885704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x560a4e6a26b0 con 0x560a4e59f590 2024-06-29T14:37:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f22ebfff640 1 -- 172.21.15.59:0/1296885704 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2304003d30 con 0x560a4e59f590 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f22ebfff640 1 -- 172.21.15.59:0/1296885704 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 46) ==== 3502+0+0 (secure 0 0 0) 0x7f22fc07b020 con 0x560a4e59f590 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f231a390740 10 client.7509 fetch_fsmap finished waiting for FSMap version 46 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f231a390740 10 client.7509 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f231a390740 20 client.7509 populate_metadata read hostname 'smithi059' 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f231a390740 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] conn(0x560a4e57ee40 0x560a4e9bb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f231a390740 4 client.7509 mds_command: new command op to 7365 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f231a390740 1 -- 172.21.15.59:0/1296885704 --> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a4e8d9580 con 0x560a4e57ee40 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.226+0000 7f2311272640 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] conn(0x560a4e57ee40 0x560a4e9bb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.227+0000 7f22ebfff640 10 client.7509 ms_handle_connect on v2:172.21.15.153:6835/1942687234 2024-06-29T14:37:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.227+0000 7f2311272640 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] conn(0x560a4e57ee40 0x560a4e9bb170 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3454f9e69c722ad server_cookie=2d5f965c07933c6f in_seq=0 out_seq=0 2024-06-29T14:37:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f22ebfff640 1 -- 172.21.15.59:0/1296885704 <== mds.0 v2:172.21.15.153:6835/1942687234 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x560a4e8d9580 con 0x560a4e57ee40 2024-06-29T14:37:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f22ebfff640 10 client.7509 handle_command_reply: tid=0 2024-06-29T14:37:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 client.7509 shutdown 2024-06-29T14:37:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 -- 172.21.15.59:0/1296885704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 msgr2=0x560a4e987dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987dd0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f22fc05deb0 tx=0x7f22fc0606c0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 -- 172.21.15.59:0/1296885704 shutdown_connections 2024-06-29T14:37:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] conn(0x560a4e57ee40 0x560a4e9bb170 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5c0270 0x560a4e9be760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987dd0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 --2- 172.21.15.59:0/1296885704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e9a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f22ebfff640 0 client.7509 ms_handle_reset on v2:172.21.15.153:6835/1942687234 2024-06-29T14:37:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 -- 172.21.15.59:0/1296885704 >> 172.21.15.59:0/1296885704 conn(0x560a4e5b43c0 msgr2=0x560a4e98cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 -- 172.21.15.59:0/1296885704 shutdown_connections 2024-06-29T14:37:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 1 -- 172.21.15.59:0/1296885704 wait complete. 2024-06-29T14:37:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.228+0000 7f231a390740 20 client.7509 trim_cache size 0 max 16384 2024-06-29T14:37:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.247+0000 7f231a390740 1 Processor -- start 2024-06-29T14:37:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.247+0000 7f231a390740 1 -- start start 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f231a390740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5b43c0 0x560a4e963e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f231a390740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5973b0 0x560a4e9643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f231a390740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f231a390740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f230c201d00 con 0x560a4e59f590 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f231a390740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f230c200d00 con 0x560a4e5b43c0 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f231a390740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f230c1f9450 con 0x560a4e5973b0 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33720/0 (socket says 172.21.15.59:33720) 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 -- 172.21.15.59:0/1379925721 learned_addr learned my addr 172.21.15.59:0/1379925721 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2310a71640 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5973b0 0x560a4e9643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311272640 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5b43c0 0x560a4e963e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 -- 172.21.15.59:0/1379925721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5973b0 msgr2=0x560a4e9643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5973b0 0x560a4e9643b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 -- 172.21.15.59:0/1379925721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5b43c0 msgr2=0x560a4e963e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5b43c0 0x560a4e963e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 -- 172.21.15.59:0/1379925721 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a4e9634f0 con 0x560a4e59f590 2024-06-29T14:37:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f2311a73640 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987830 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x560a4e9ae420 tx=0x7f23040095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b9a4fce71aa443 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f22f67fc640 1 -- 172.21.15.59:0/1379925721 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2304051070 con 0x560a4e59f590 2024-06-29T14:37:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f22f67fc640 1 -- 172.21.15.59:0/1379925721 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f230402fb70 con 0x560a4e59f590 2024-06-29T14:37:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.249+0000 7f22f67fc640 1 -- 172.21.15.59:0/1379925721 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2304033070 con 0x560a4e59f590 2024-06-29T14:37:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 -- 172.21.15.59:0/1379925721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 msgr2=0x560a4e987830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987830 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x560a4e9ae420 tx=0x7f23040095c0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 -- 172.21.15.59:0/1379925721 shutdown_connections 2024-06-29T14:37:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e987830 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5973b0 0x560a4e9643b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 --2- 172.21.15.59:0/1379925721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5b43c0 0x560a4e963e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 -- 172.21.15.59:0/1379925721 >> 172.21.15.59:0/1379925721 conn(0x560a4e962ad0 msgr2=0x560a4e9aca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 -- 172.21.15.59:0/1379925721 shutdown_connections 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.250+0000 7f231a390740 1 -- 172.21.15.59:0/1379925721 wait complete. 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.251+0000 7f231a390740 1 -- start start 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.251+0000 7f231a390740 1 Processor -- start 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e990090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5b43c0 0x560a4e9ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x560a4e9bf3e0 con 0x560a4e59f590 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f230c0cc3f0 con 0x560a4e5973b0 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x560a4e9bef10 con 0x560a4e5b43c0 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 10 client.0 _ll_register_callbacks cb 0x7f22fc0962a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33732/0 (socket says 172.21.15.59:33732) 2024-06-29T14:37:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 -- 172.21.15.59:0/3098698697 learned_addr learned my addr 172.21.15.59:0/3098698697 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2311272640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e990090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2311a73640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5b43c0 0x560a4e9ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 -- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5b43c0 msgr2=0x560a4e9ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5b43c0 0x560a4e9ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 -- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 msgr2=0x560a4e990090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e990090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 -- 172.21.15.59:0/3098698697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a4e98d440 con 0x560a4e59f590 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2311272640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e990090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2311a73640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5b43c0 0x560a4e9ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f2310a71640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9cdc10 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f22fc07a260 tx=0x7f22fc05fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0f44d640a5730e server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f22ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:37:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f22ebfff640 1 -- 172.21.15.59:0/3098698697 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc082060 con 0x560a4e59f590 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f22ebfff640 1 -- 172.21.15.59:0/3098698697 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f22fc082200 con 0x560a4e59f590 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 1 -- 172.21.15.59:0/3098698697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f230402fb70 con 0x560a4e59f590 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f22ebfff640 1 -- 172.21.15.59:0/3098698697 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f22fc070050 con 0x560a4e59f590 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.253+0000 7f231a390740 10 client.7515 fetch_fsmap learned FSMap version 46 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f231a390740 1 -- 172.21.15.59:0/3098698697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x560a4e9884e0 con 0x560a4e59f590 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f22ebfff640 1 -- 172.21.15.59:0/3098698697 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f230402fb70 con 0x560a4e59f590 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f22ebfff640 1 -- 172.21.15.59:0/3098698697 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 46) ==== 3502+0+0 (secure 0 0 0) 0x7f22fc07b040 con 0x560a4e59f590 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f231a390740 10 client.7515 fetch_fsmap finished waiting for FSMap version 46 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f231a390740 10 client.7515 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f231a390740 20 client.7515 populate_metadata read hostname 'smithi059' 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f231a390740 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] conn(0x560a4e554280 0x560a4ea2ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f231a390740 4 client.7515 mds_command: new command op to 7365 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:37:59.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f231a390740 1 -- 172.21.15.59:0/3098698697 --> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] -- command(tid 0: {"prefix": "status"}) -- 0x560a4e9a7020 con 0x560a4e554280 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.254+0000 7f2311272640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] conn(0x560a4e554280 0x560a4ea2ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f22ebfff640 10 client.7515 ms_handle_connect on v2:172.21.15.153:6835/1942687234 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f2311272640 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] conn(0x560a4e554280 0x560a4ea2ce00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39e4fcd1699107be server_cookie=edd0bd820f5413a7 in_seq=0 out_seq=0 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f22ebfff640 1 -- 172.21.15.59:0/3098698697 <== mds.0 v2:172.21.15.153:6835/1942687234 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x560a4e9a7020 con 0x560a4e554280 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f22ebfff640 10 client.7515 handle_command_reply: tid=0 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 client.7515 shutdown 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 -- 172.21.15.59:0/3098698697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 msgr2=0x560a4e9cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9cdc10 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f22fc07a260 tx=0x7f22fc05fb40 comp rx=0 tx=0).stop 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 -- 172.21.15.59:0/3098698697 shutdown_connections 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:6835/1942687234,v1:172.21.15.153:6837/1942687234] conn(0x560a4e554280 0x560a4ea2ce00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x560a4e5b43c0 0x560a4e9ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x560a4e59f590 0x560a4e9cdc10 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 --2- 172.21.15.59:0/3098698697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560a4e5973b0 0x560a4e990090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f22ebfff640 0 client.7515 ms_handle_reset on v2:172.21.15.153:6835/1942687234 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 -- 172.21.15.59:0/3098698697 >> 172.21.15.59:0/3098698697 conn(0x560a4e962ad0 msgr2=0x560a4e963e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 -- 172.21.15.59:0/3098698697 shutdown_connections 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 1 -- 172.21.15.59:0/3098698697 wait complete. 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.255+0000 7f231a390740 20 client.7515 trim_cache size 0 max 16384 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 7365, 2024-06-29T14:37:59.259 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:37:59.260 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:37:59.260 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:37:59.260 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8708952050000001, 2024-06-29T14:37:59.260 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 46, 2024-06-29T14:37:59.260 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 101, 2024-06-29T14:37:59.260 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 101, 2024-06-29T14:37:59.260 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.510501705999999 2024-06-29T14:37:59.260 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:37:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 -- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f22f00763f0 msgr2=0x7f22f0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f22f00763f0 0x7f22f0078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2308006fd0 tx=0x7f2308009060 comp rx=0 tx=0).stop 2024-06-29T14:37:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 -- 172.21.15.59:0/1299159489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c0ce950 msgr2=0x7f230c0ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c0ce950 0x7f230c0ac240 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00d740 tx=0x7f22fc00dc00 comp rx=0 tx=0).stop 2024-06-29T14:37:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 -- 172.21.15.59:0/1299159489 shutdown_connections 2024-06-29T14:37:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f22f00763f0 0x7f22f0078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f230c162800 0x7f230c0ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f230c0ce950 0x7f230c0ac240 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 --2- 172.21.15.59:0/1299159489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f230c08c190 0x7f230c0aafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 -- 172.21.15.59:0/1299159489 >> 172.21.15.59:0/1299159489 conn(0x7f230c0b5610 msgr2=0x7f230c15e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 -- 172.21.15.59:0/1299159489 shutdown_connections 2024-06-29T14:37:59.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.256+0000 7f23134d6640 1 -- 172.21.15.59:0/1299159489 wait complete. 2024-06-29T14:37:59.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:37:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.384+0000 7fafa2817640 1 -- 172.21.15.59:0/912057339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 msgr2=0x7faf9c06ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.384+0000 7fafa2817640 1 --2- 172.21.15.59:0/912057339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 0x7faf9c06ee50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7faf9c144d50 tx=0x7faf9402aa30 comp rx=0 tx=0).stop 2024-06-29T14:37:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.384+0000 7fafa2817640 1 -- 172.21.15.59:0/912057339 shutdown_connections 2024-06-29T14:37:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.384+0000 7fafa2817640 1 --2- 172.21.15.59:0/912057339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 0x7faf9c06ee50 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.384+0000 7fafa2817640 1 --2- 172.21.15.59:0/912057339 >> v2:172.21.15.153:3300/0 conn(0x7faf9c144760 0x7faf9c0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.384+0000 7fafa2817640 1 -- 172.21.15.59:0/912057339 >> 172.21.15.59:0/912057339 conn(0x7faf9c06cef0 msgr2=0x7faf9c06d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.384+0000 7fafa2817640 1 -- 172.21.15.59:0/912057339 shutdown_connections 2024-06-29T14:37:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.384+0000 7fafa2817640 1 -- 172.21.15.59:0/912057339 wait complete. 2024-06-29T14:37:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.386+0000 7fafa2817640 1 Processor -- start 2024-06-29T14:37:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.386+0000 7fafa2817640 1 -- start start 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.387+0000 7fafa2817640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 0x7faf9c1f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.387+0000 7fafa2817640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf9c0ce6c0 0x7faf9c0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7fafa2817640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c144760 0x7faf9c0c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7fafa2817640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c06f910 con 0x7faf9c073a00 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7fafa2817640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c06f790 con 0x7faf9c144760 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7fafa2817640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7faf9c06fa90 con 0x7faf9c0ce6c0 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 0x7faf9c1f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 0x7faf9c1f2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33754/0 (socket says 172.21.15.59:33754) 2024-06-29T14:37:59.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 -- 172.21.15.59:0/523207320 learned_addr learned my addr 172.21.15.59:0/523207320 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7fafa0db4640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c144760 0x7faf9c0c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9b7fe640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf9c0ce6c0 0x7faf9c0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 -- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf9c0ce6c0 msgr2=0x7faf9c0cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf9c0ce6c0 0x7faf9c0cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 -- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c144760 msgr2=0x7faf9c0c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c144760 0x7faf9c0c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 -- 172.21.15.59:0/523207320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c1f2580 con 0x7faf9c073a00 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf9bfff640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 0x7faf9c1f2040 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00b3c0 tx=0x7faf8c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf98ff9640 1 -- 172.21.15.59:0/523207320 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c015020 con 0x7faf9c073a00 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf98ff9640 1 -- 172.21.15.59:0/523207320 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf8c011920 con 0x7faf9c073a00 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9c0ac720 con 0x7faf9c073a00 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.388+0000 7faf98ff9640 1 -- 172.21.15.59:0/523207320 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c011db0 con 0x7faf9c073a00 2024-06-29T14:37:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.389+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9c0acce0 con 0x7faf9c073a00 2024-06-29T14:37:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.389+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9c1f2750 con 0x7faf9c073a00 2024-06-29T14:37:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.390+0000 7faf98ff9640 1 -- 172.21.15.59:0/523207320 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7faf8c018020 con 0x7faf9c073a00 2024-06-29T14:37:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.391+0000 7faf98ff9640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf88076420 0x7faf88078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.391+0000 7faf98ff9640 1 -- 172.21.15.59:0/523207320 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7faf8c097890 con 0x7faf9c073a00 2024-06-29T14:37:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.391+0000 7faf9b7fe640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf88076420 0x7faf88078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.392+0000 7faf9b7fe640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf88076420 0x7faf88078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0c6b70 tx=0x7faf9000b2e0 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:59.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.395+0000 7faf98ff9640 1 -- 172.21.15.59:0/523207320 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf8c01f020 con 0x7faf9c073a00 2024-06-29T14:37:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.626+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7faf9c06ffd0 con 0x7faf9c073a00 2024-06-29T14:37:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.627+0000 7faf98ff9640 1 -- 172.21.15.59:0/523207320 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) ==== 53+0+73 (secure 0 0 0) 0x7faf8c0608f0 con 0x7faf9c073a00 2024-06-29T14:37:59.625 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:37:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf88076420 msgr2=0x7faf88078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf88076420 0x7faf88078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0c6b70 tx=0x7faf9000b2e0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 msgr2=0x7faf9c1f2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 0x7faf9c1f2040 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00b3c0 tx=0x7faf8c00b880 comp rx=0 tx=0).stop 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 shutdown_connections 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf9c144760 0x7faf9c0c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7faf88076420 0x7faf88078930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faf9c0ce6c0 0x7faf9c0cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 --2- 172.21.15.59:0/523207320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf9c073a00 0x7faf9c1f2040 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 >> 172.21.15.59:0/523207320 conn(0x7faf9c06cef0 msgr2=0x7faf9c0b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 shutdown_connections 2024-06-29T14:37:59.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59.629+0000 7fafa2817640 1 -- 172.21.15.59:0/523207320 wait complete. 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:37:59.639 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:59.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.639 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:37:59.656 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:37:59.656 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:37:59.656 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:37:59.656 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.656 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:37:59.656 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.656 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:59.656 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 41771sec preferred_lft 41771sec 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.657 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:37:59.659 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T14:37:59.659 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.659 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:37:59.722 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:37:59.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.723 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:37:59.780 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:37:59.780 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:37:59.780 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:37:59.780 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:37:59.781 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:37:59.781 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:37:59.781 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:37:59.781 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:37:59.781 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:37:59.781 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:37:59.781 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:37:59.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:37:59.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:37:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:37:59.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.911 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:37:59.927 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:37:59.927 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:37:59.927 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:59.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:37:59.993 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:37:59.993 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:00.049 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:38:00.049 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:38:00.049 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:00.104 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:38:00.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:00.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:38:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:38:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:38:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:38:00.209 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:38:00.209 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:38:00.210 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:38:00.210 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:38:00.210 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:38:00.217 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:38:00.217 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:38:00.217 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:38:00.217 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:38:00.217 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:38:00.217 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:38:00.217 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:38:00.218 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:38:00.218 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:38:00.218 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:38:00.218 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:38:00.218 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:38:00.218 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:38:00.219 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:38:00.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:00.220 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:00.245 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:38:00.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:38:00.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:38:00.351 INFO:teuthology.orchestra.run.smithi059.stdout:client7027 2024-06-29T14:38:00.352 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:38:00.352 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:00.369 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:38:00.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.529+0000 7f79bd94a640 1 -- 172.21.15.59:0/1089029340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b80cf120 msgr2=0x7f79b8073500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.529+0000 7f79bd94a640 1 --2- 172.21.15.59:0/1089029340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b80cf120 0x7f79b8073500 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f79a0005fd0 tx=0x7f79a002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.529+0000 7f79bd94a640 1 -- 172.21.15.59:0/1089029340 shutdown_connections 2024-06-29T14:38:00.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.529+0000 7f79bd94a640 1 --2- 172.21.15.59:0/1089029340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b80cf120 0x7f79b8073500 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.530+0000 7f79bd94a640 1 --2- 172.21.15.59:0/1089029340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b8073f30 0x7f79b815a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.530+0000 7f79bd94a640 1 -- 172.21.15.59:0/1089029340 >> 172.21.15.59:0/1089029340 conn(0x7f79b80b24c0 msgr2=0x7f79b80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:00.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.530+0000 7f79bd94a640 1 -- 172.21.15.59:0/1089029340 shutdown_connections 2024-06-29T14:38:00.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.530+0000 7f79bd94a640 1 -- 172.21.15.59:0/1089029340 wait complete. 2024-06-29T14:38:00.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.531+0000 7f79bd94a640 1 Processor -- start 2024-06-29T14:38:00.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.531+0000 7f79bd94a640 1 -- start start 2024-06-29T14:38:00.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79bd94a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b8073a40 0x7f79b8146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79bd94a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b8073f30 0x7f79b815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79bd94a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b80cf120 0x7f79b80b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79bd94a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f79b80cccc0 con 0x7f79b8073f30 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79bd94a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79b806cb00 con 0x7f79b80cf120 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79bd94a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f79b80cce40 con 0x7f79b8073a40 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b8073f30 0x7f79b815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b8073f30 0x7f79b815c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33764/0 (socket says 172.21.15.59:33764) 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b67fc640 1 -- 172.21.15.59:0/3906989468 learned_addr learned my addr 172.21.15.59:0/3906989468 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b77fe640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b80cf120 0x7f79b80b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b6ffd640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b8073a40 0x7f79b8146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b67fc640 1 -- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b8073a40 msgr2=0x7f79b8146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b67fc640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b8073a40 0x7f79b8146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b67fc640 1 -- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b80cf120 msgr2=0x7f79b80b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b67fc640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b80cf120 0x7f79b80b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b67fc640 1 -- 172.21.15.59:0/3906989468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b8146e80 con 0x7f79b8073f30 2024-06-29T14:38:00.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.533+0000 7f79b6ffd640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b8073a40 0x7f79b8146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:00.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.534+0000 7f79b67fc640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b8073f30 0x7f79b815c6f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f79a0034040 tx=0x7f79a00035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:00.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.534+0000 7f79affff640 1 -- 172.21.15.59:0/3906989468 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a003e030 con 0x7f79b8073f30 2024-06-29T14:38:00.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.534+0000 7f79affff640 1 -- 172.21.15.59:0/3906989468 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79a0003ae0 con 0x7f79b8073f30 2024-06-29T14:38:00.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.534+0000 7f79affff640 1 -- 172.21.15.59:0/3906989468 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0035040 con 0x7f79b8073f30 2024-06-29T14:38:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.534+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b8147160 con 0x7f79b8073f30 2024-06-29T14:38:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.534+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b8147720 con 0x7f79b8073f30 2024-06-29T14:38:00.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.535+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b815a400 con 0x7f79b8073f30 2024-06-29T14:38:00.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.535+0000 7f79affff640 1 -- 172.21.15.59:0/3906989468 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f79a0041030 con 0x7f79b8073f30 2024-06-29T14:38:00.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.536+0000 7f79affff640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f79980763f0 0x7f7998078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.536+0000 7f79affff640 1 -- 172.21.15.59:0/3906989468 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f79a00b8040 con 0x7f79b8073f30 2024-06-29T14:38:00.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.536+0000 7f79b6ffd640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f79980763f0 0x7f7998078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.537+0000 7f79b6ffd640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f79980763f0 0x7f7998078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f79b806f700 tx=0x7f79b0009500 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:00.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.541+0000 7f79affff640 1 -- 172.21.15.59:0/3906989468 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f79a0080f10 con 0x7f79b8073f30 2024-06-29T14:38:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.757+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 --> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f79b8147a90 con 0x7f79980763f0 2024-06-29T14:38:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.759+0000 7f79affff640 1 -- 172.21.15.59:0/3906989468 <== mgr.6179 v2:172.21.15.153:6832/1607211985 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f79b8147a90 con 0x7f79980763f0 2024-06-29T14:38:00.757 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:38:00.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:00.757 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T14:38:00.757 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:00.757 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:38:00.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f79980763f0 msgr2=0x7f7998078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f79980763f0 0x7f7998078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f79b806f700 tx=0x7f79b0009500 comp rx=0 tx=0).stop 2024-06-29T14:38:00.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b8073f30 msgr2=0x7f79b815c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b8073f30 0x7f79b815c6f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f79a0034040 tx=0x7f79a00035f0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 shutdown_connections 2024-06-29T14:38:00.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b80cf120 0x7f79b80b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b8073f30 0x7f79b815c6f0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f79980763f0 0x7f7998078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 --2- 172.21.15.59:0/3906989468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b8073a40 0x7f79b8146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 >> 172.21.15.59:0/3906989468 conn(0x7f79b80b24c0 msgr2=0x7f79b8068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:00.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 shutdown_connections 2024-06-29T14:38:00.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.761+0000 7f79bd94a640 1 -- 172.21.15.59:0/3906989468 wait complete. 2024-06-29T14:38:00.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:38:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.891+0000 7fad71911640 1 -- 172.21.15.59:0/1035792502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c162800 msgr2=0x7fad6c162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.891+0000 7fad71911640 1 --2- 172.21.15.59:0/1035792502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c162800 0x7fad6c162c10 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fad64005fd0 tx=0x7fad6402a870 comp rx=0 tx=0).stop 2024-06-29T14:38:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.891+0000 7fad71911640 1 -- 172.21.15.59:0/1035792502 shutdown_connections 2024-06-29T14:38:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.891+0000 7fad71911640 1 --2- 172.21.15.59:0/1035792502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c162800 0x7fad6c162c10 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.891+0000 7fad71911640 1 --2- 172.21.15.59:0/1035792502 >> v2:172.21.15.153:3300/0 conn(0x7fad6c0ce950 0x7fad6c1622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.891+0000 7fad71911640 1 -- 172.21.15.59:0/1035792502 >> 172.21.15.59:0/1035792502 conn(0x7fad6c0b5610 msgr2=0x7fad6c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.892+0000 7fad71911640 1 -- 172.21.15.59:0/1035792502 shutdown_connections 2024-06-29T14:38:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.892+0000 7fad71911640 1 -- 172.21.15.59:0/1035792502 wait complete. 2024-06-29T14:38:00.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.893+0000 7fad71911640 1 Processor -- start 2024-06-29T14:38:00.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.894+0000 7fad71911640 1 -- start start 2024-06-29T14:38:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad71911640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad6c08c190 0x7fad6c151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad71911640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c0ce950 0x7fad6c0cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad71911640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad6c162800 0x7fad6c15bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad71911640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c0b7f60 con 0x7fad6c0ce950 2024-06-29T14:38:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad71911640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c0b7de0 con 0x7fad6c08c190 2024-06-29T14:38:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad71911640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fad6c0b80e0 con 0x7fad6c162800 2024-06-29T14:38:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c0ce950 0x7fad6c0cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c0ce950 0x7fad6c0cb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33776/0 (socket says 172.21.15.59:33776) 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6a7fc640 1 -- 172.21.15.59:0/3580451533 learned_addr learned my addr 172.21.15.59:0/3580451533 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6affd640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad6c08c190 0x7fad6c151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6b7fe640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad6c162800 0x7fad6c15bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6a7fc640 1 -- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad6c162800 msgr2=0x7fad6c15bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6a7fc640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad6c162800 0x7fad6c15bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6a7fc640 1 -- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad6c08c190 msgr2=0x7fad6c151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6a7fc640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad6c08c190 0x7fad6c151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.895+0000 7fad6a7fc640 1 -- 172.21.15.59:0/3580451533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c0cbb00 con 0x7fad6c0ce950 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.896+0000 7fad6a7fc640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c0ce950 0x7fad6c0cb5c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fad6000d700 tx=0x7fad6000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.896+0000 7fad53fff640 1 -- 172.21.15.59:0/3580451533 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60017070 con 0x7fad6c0ce950 2024-06-29T14:38:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.896+0000 7fad53fff640 1 -- 172.21.15.59:0/3580451533 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad60004310 con 0x7fad6c0ce950 2024-06-29T14:38:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.896+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6c0cd300 con 0x7fad6c0ce950 2024-06-29T14:38:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.896+0000 7fad53fff640 1 -- 172.21.15.59:0/3580451533 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60009a80 con 0x7fad6c0ce950 2024-06-29T14:38:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.896+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6c1f8600 con 0x7fad6c0ce950 2024-06-29T14:38:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.897+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad6c1622c0 con 0x7fad6c0ce950 2024-06-29T14:38:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.898+0000 7fad53fff640 1 -- 172.21.15.59:0/3580451533 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fad60009c20 con 0x7fad6c0ce950 2024-06-29T14:38:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.898+0000 7fad53fff640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fad580763f0 0x7fad58078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.898+0000 7fad53fff640 1 -- 172.21.15.59:0/3580451533 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7fad60097c20 con 0x7fad6c0ce950 2024-06-29T14:38:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.899+0000 7fad6affd640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fad580763f0 0x7fad58078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.899+0000 7fad6affd640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fad580763f0 0x7fad58078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad5c007fd0 tx=0x7fad5c009040 comp rx=0 tx=0).ready entity=mgr.6179 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:00.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:00.903+0000 7fad53fff640 1 -- 172.21.15.59:0/3580451533 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fad600611d0 con 0x7fad6c0ce950 2024-06-29T14:38:01.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.142+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fad6c0b5da0 con 0x7fad6c0ce950 2024-06-29T14:38:01.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.419+0000 7fad53fff640 1 -- 172.21.15.59:0/3580451533 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v39) ==== 92+0+0 (secure 0 0 0) 0x7fad60060ad0 con 0x7fad6c0ce950 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fad580763f0 msgr2=0x7fad58078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fad580763f0 0x7fad58078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad5c007fd0 tx=0x7fad5c009040 comp rx=0 tx=0).stop 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c0ce950 msgr2=0x7fad6c0cb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c0ce950 0x7fad6c0cb5c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fad6000d700 tx=0x7fad6000dbc0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 shutdown_connections 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fad580763f0 0x7fad58078900 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad5c007fd0 tx=0x7fad5c009040 comp rx=0 tx=0).stop 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad6c162800 0x7fad6c15bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6c0ce950 0x7fad6c0cb5c0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 --2- 172.21.15.59:0/3580451533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad6c08c190 0x7fad6c151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.422+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 >> 172.21.15.59:0/3580451533 conn(0x7fad6c0b5610 msgr2=0x7fad6c06d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.423+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 shutdown_connections 2024-06-29T14:38:01.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.423+0000 7fad71911640 1 -- 172.21.15.59:0/3580451533 wait complete. 2024-06-29T14:38:01.435 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:38:01.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.561+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2395264179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2fa0 msgr2=0x7fdd540d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.561+0000 7fdd5a8c0640 1 --2- 172.21.15.59:0/2395264179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2fa0 0x7fdd540d1520 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fdd48009830 tx=0x7fdd4802a5c0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.561+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2395264179 shutdown_connections 2024-06-29T14:38:01.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.561+0000 7fdd5a8c0640 1 --2- 172.21.15.59:0/2395264179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2fa0 0x7fdd540d1520 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.561+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2395264179 >> 172.21.15.59:0/2395264179 conn(0x7fdd54073e10 msgr2=0x7fdd54074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.561+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2395264179 shutdown_connections 2024-06-29T14:38:01.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.561+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2395264179 wait complete. 2024-06-29T14:38:01.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.563+0000 7fdd5a8c0640 1 Processor -- start 2024-06-29T14:38:01.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.563+0000 7fdd5a8c0640 1 -- start start 2024-06-29T14:38:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd5a8c0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2270 0x7fdd541f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd5a8c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd540b2fa0 0x7fdd5415d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd5a8c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d1a60 0x7fdd540ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd5a8c0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdd540b65d0 con 0x7fdd540b2270 2024-06-29T14:38:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd5a8c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd540b6450 con 0x7fdd540b2fa0 2024-06-29T14:38:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd5a8c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd540b6750 con 0x7fdd540d1a60 2024-06-29T14:38:01.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2270 0x7fdd541f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2270 0x7fdd541f6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33802/0 (socket says 172.21.15.59:33802) 2024-06-29T14:38:01.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 -- 172.21.15.59:0/2966613171 learned_addr learned my addr 172.21.15.59:0/2966613171 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:01.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd537fe640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd540b2fa0 0x7fdd5415d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd58e5d640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d1a60 0x7fdd540ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 -- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d1a60 msgr2=0x7fdd540ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d1a60 0x7fdd540ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 -- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd540b2fa0 msgr2=0x7fdd5415d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd540b2fa0 0x7fdd5415d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 -- 172.21.15.59:0/2966613171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5415dfb0 con 0x7fdd540b2270 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd53fff640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2270 0x7fdd541f6160 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c3a0 tx=0x7fdd4400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.565+0000 7fdd50ff9640 1 -- 172.21.15.59:0/2966613171 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44017070 con 0x7fdd540b2270 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.566+0000 7fdd50ff9640 1 -- 172.21.15.59:0/2966613171 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd440053c0 con 0x7fdd540b2270 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.566+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd5415e290 con 0x7fdd540b2270 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.566+0000 7fdd50ff9640 1 -- 172.21.15.59:0/2966613171 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44003850 con 0x7fdd540b2270 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.566+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd5415e850 con 0x7fdd540b2270 2024-06-29T14:38:01.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.567+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd540b4fe0 con 0x7fdd540b2270 2024-06-29T14:38:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.568+0000 7fdd50ff9640 1 -- 172.21.15.59:0/2966613171 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 39) ==== 98320+0+0 (secure 0 0 0) 0x7fdd4401a070 con 0x7fdd540b2270 2024-06-29T14:38:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.569+0000 7fdd50ff9640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fdd40076440 0x7fdd40078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.569+0000 7fdd537fe640 1 -- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fdd40076440 msgr2=0x7fdd40078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1607211985 2024-06-29T14:38:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.569+0000 7fdd50ff9640 1 -- 172.21.15.59:0/2966613171 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7fdd44096df0 con 0x7fdd540b2270 2024-06-29T14:38:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.569+0000 7fdd537fe640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fdd40076440 0x7fdd40078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:01.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.578+0000 7fdd50ff9640 1 -- 172.21.15.59:0/2966613171 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd540b4fe0 con 0x7fdd540b2270 2024-06-29T14:38:01.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.770+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fdd540b6c70 con 0x7fdd540b2270 2024-06-29T14:38:01.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.770+0000 7fdd537fe640 1 -- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fdd40076440 msgr2=0x7fdd40078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1607211985 2024-06-29T14:38:01.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.770+0000 7fdd537fe640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fdd40076440 0x7fdd40078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:01.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.776+0000 7fdd50ff9640 1 -- 172.21.15.59:0/2966613171 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) ==== 138+0+0 (secure 0 0 0) 0x7fdd4405fdd0 con 0x7fdd540b2270 2024-06-29T14:38:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fdd40076440 msgr2=0x7fdd40078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fdd40076440 0x7fdd40078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2270 msgr2=0x7fdd541f6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2270 0x7fdd541f6160 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c3a0 tx=0x7fdd4400c860 comp rx=0 tx=0).stop 2024-06-29T14:38:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 shutdown_connections 2024-06-29T14:38:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7fdd40076440 0x7fdd40078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d1a60 0x7fdd540ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd540b2fa0 0x7fdd5415d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 --2- 172.21.15.59:0/2966613171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b2270 0x7fdd541f6160 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 >> 172.21.15.59:0/2966613171 conn(0x7fdd54073e10 msgr2=0x7fdd540d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 shutdown_connections 2024-06-29T14:38:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.779+0000 7fdd5a8c0640 1 -- 172.21.15.59:0/2966613171 wait complete. 2024-06-29T14:38:01.789 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.912+0000 7f921ac95640 1 -- 172.21.15.59:0/3009766198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2270 msgr2=0x7f921415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.912+0000 7f921ac95640 1 --2- 172.21.15.59:0/3009766198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2270 0x7f921415bc20 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f9208005fd0 tx=0x7f920802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.912+0000 7f921ac95640 1 -- 172.21.15.59:0/3009766198 shutdown_connections 2024-06-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.912+0000 7f921ac95640 1 --2- 172.21.15.59:0/3009766198 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b2fa0 0x7f92140d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.913+0000 7f921ac95640 1 --2- 172.21.15.59:0/3009766198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2270 0x7f921415bc20 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.913+0000 7f921ac95640 1 -- 172.21.15.59:0/3009766198 >> 172.21.15.59:0/3009766198 conn(0x7f9214073e10 msgr2=0x7f9214074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.913+0000 7f921ac95640 1 -- 172.21.15.59:0/3009766198 shutdown_connections 2024-06-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.913+0000 7f921ac95640 1 -- 172.21.15.59:0/3009766198 wait complete. 2024-06-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.914+0000 7f921ac95640 1 Processor -- start 2024-06-29T14:38:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.914+0000 7f921ac95640 1 -- start start 2024-06-29T14:38:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f921ac95640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b2270 0x7f92141f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f921ac95640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2fa0 0x7f9214154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f921ac95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92140d1a60 0x7f92140ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f921ac95640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f92140b6cc0 con 0x7f92140b2fa0 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f921ac95640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92140b6b40 con 0x7f92140d1a60 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f921ac95640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f92140cfe90 con 0x7f92140b2270 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f9213fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2fa0 0x7f9214154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f9213fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2fa0 0x7f9214154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33822/0 (socket says 172.21.15.59:33822) 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f9213fff640 1 -- 172.21.15.59:0/1464208073 learned_addr learned my addr 172.21.15.59:0/1464208073 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f9218a31640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b2270 0x7f92141f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.916+0000 7f9219232640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92140d1a60 0x7f92140ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f9213fff640 1 -- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b2270 msgr2=0x7f92141f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f9213fff640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b2270 0x7f92141f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f9213fff640 1 -- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92140d1a60 msgr2=0x7f92140ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f9213fff640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92140d1a60 0x7f92140ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f9213fff640 1 -- 172.21.15.59:0/1464208073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92141f52c0 con 0x7f92140b2fa0 2024-06-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f9219232640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92140d1a60 0x7f92140ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f9213fff640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2fa0 0x7f9214154660 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f920000d770 tx=0x7f920000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f92117fa640 1 -- 172.21.15.59:0/1464208073 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9200017070 con 0x7f92140b2fa0 2024-06-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f92117fa640 1 -- 172.21.15.59:0/1464208073 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9200009d20 con 0x7f92140b2fa0 2024-06-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92141f55a0 con 0x7f92140b2fa0 2024-06-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f92117fa640 1 -- 172.21.15.59:0/1464208073 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9200004280 con 0x7f92140b2fa0 2024-06-29T14:38:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.917+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921406de40 con 0x7f92140b2fa0 2024-06-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.918+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921406dc30 con 0x7f92140b2fa0 2024-06-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.919+0000 7f92117fa640 1 -- 172.21.15.59:0/1464208073 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 39) ==== 98320+0+0 (secure 0 0 0) 0x7f9200005020 con 0x7f92140b2fa0 2024-06-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.920+0000 7f92117fa640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f91f8076440 0x7f91f8078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.920+0000 7f92117fa640 1 -- 172.21.15.59:0/1464208073 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f9200097f20 con 0x7f92140b2fa0 2024-06-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.920+0000 7f9218a31640 1 -- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f91f8076440 msgr2=0x7f91f8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1607211985 2024-06-29T14:38:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.920+0000 7f9218a31640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f91f8076440 0x7f91f8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:01.924+0000 7f92117fa640 1 -- 172.21.15.59:0/1464208073 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9200020020 con 0x7f92140b2fa0 2024-06-29T14:38:02.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.121+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f92140d03e0 con 0x7f92140b2fa0 2024-06-29T14:38:02.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.122+0000 7f9218a31640 1 -- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f91f8076440 msgr2=0x7f91f8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1607211985 2024-06-29T14:38:02.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.122+0000 7f9218a31640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f91f8076440 0x7f91f8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:02.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.125+0000 7f92117fa640 1 -- 172.21.15.59:0/1464208073 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) ==== 133+0+0 (secure 0 0 0) 0x7f9200060d70 con 0x7f92140b2fa0 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.127+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f91f8076440 msgr2=0x7f91f8078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.127+0000 7f921ac95640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f91f8076440 0x7f91f8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.127+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2fa0 msgr2=0x7f9214154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.127+0000 7f921ac95640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2fa0 0x7f9214154660 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f920000d770 tx=0x7f920000dc30 comp rx=0 tx=0).stop 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.128+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 shutdown_connections 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.128+0000 7f921ac95640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92140d1a60 0x7f92140ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.128+0000 7f921ac95640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140b2fa0 0x7f9214154660 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.128+0000 7f921ac95640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f91f8076440 0x7f91f8078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:02.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.128+0000 7f921ac95640 1 --2- 172.21.15.59:0/1464208073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b2270 0x7f92141f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:02.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.128+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 >> 172.21.15.59:0/1464208073 conn(0x7f9214073e10 msgr2=0x7f92140d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:02.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.128+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 shutdown_connections 2024-06-29T14:38:02.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.128+0000 7f921ac95640 1 -- 172.21.15.59:0/1464208073 wait complete. 2024-06-29T14:38:02.138 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-06-29T14:38:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.260+0000 7f50a5a47640 1 -- 172.21.15.59:0/2151264274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a00cf0f0 msgr2=0x7f50a00b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.260+0000 7f50a5a47640 1 --2- 172.21.15.59:0/2151264274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a00cf0f0 0x7f50a00b2bd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5090009860 tx=0x7f509002a530 comp rx=0 tx=0).stop 2024-06-29T14:38:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.260+0000 7f50a5a47640 1 -- 172.21.15.59:0/2151264274 shutdown_connections 2024-06-29T14:38:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.260+0000 7f50a5a47640 1 --2- 172.21.15.59:0/2151264274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a00cf0f0 0x7f50a00b2bd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.260+0000 7f50a5a47640 1 -- 172.21.15.59:0/2151264274 >> 172.21.15.59:0/2151264274 conn(0x7f50a00d3ec0 msgr2=0x7f50a00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.260+0000 7f50a5a47640 1 -- 172.21.15.59:0/2151264274 shutdown_connections 2024-06-29T14:38:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.260+0000 7f50a5a47640 1 -- 172.21.15.59:0/2151264274 wait complete. 2024-06-29T14:38:02.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.262+0000 7f50a5a47640 1 Processor -- start 2024-06-29T14:38:02.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.262+0000 7f50a5a47640 1 -- start start 2024-06-29T14:38:02.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.263+0000 7f50a5a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a00b27c0 0x7f50a0152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:02.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.263+0000 7f50a5a47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a00d32b0 0x7f50a01597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:02.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f50a5a47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a0154830 0x7f50a015e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:02.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f50a5a47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f50a0169550 con 0x7f50a00b27c0 2024-06-29T14:38:02.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f50a5a47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50a00b44f0 con 0x7f50a00d32b0 2024-06-29T14:38:02.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f50a5a47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f50a01696d0 con 0x7f50a0154830 2024-06-29T14:38:02.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a00b27c0 0x7f50a0152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a00b27c0 0x7f50a0152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33834/0 (socket says 172.21.15.59:33834) 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 -- 172.21.15.59:0/700897917 learned_addr learned my addr 172.21.15.59:0/700897917 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509f7fe640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a0154830 0x7f50a015e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509e7fc640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a00d32b0 0x7f50a01597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 -- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a0154830 msgr2=0x7f50a015e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a0154830 0x7f50a015e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 -- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a00d32b0 msgr2=0x7f50a01597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a00d32b0 0x7f50a01597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 -- 172.21.15.59:0/700897917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a0152150 con 0x7f50a00b27c0 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509f7fe640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50a0154830 0x7f50a015e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509e7fc640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50a00d32b0 0x7f50a01597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f509effd640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50a00b27c0 0x7f50a0152f10 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f508800c4d0 tx=0x7f508800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088016020 con 0x7f50a00b27c0 2024-06-29T14:38:02.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5088012920 con 0x7f50a00b27c0 2024-06-29T14:38:02.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f50a5a47640 1 -- 172.21.15.59:0/700897917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a0152430 con 0x7f50a00b27c0 2024-06-29T14:38:02.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088012d30 con 0x7f50a00b27c0 2024-06-29T14:38:02.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.264+0000 7f50a5a47640 1 -- 172.21.15.59:0/700897917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a01529f0 con 0x7f50a00b27c0 2024-06-29T14:38:02.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.266+0000 7f50a5a47640 1 -- 172.21.15.59:0/700897917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a00b2bd0 con 0x7f50a00b27c0 2024-06-29T14:38:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.266+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 39) ==== 98320+0+0 (secure 0 0 0) 0x7f5088019020 con 0x7f50a00b27c0 2024-06-29T14:38:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.267+0000 7f5097fff640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 0x7f5080078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.267+0000 7f509e7fc640 1 -- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 msgr2=0x7f5080078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1607211985 2024-06-29T14:38:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.267+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6101+0+0 (secure 0 0 0) 0x7f50880979d0 con 0x7f50a00b27c0 2024-06-29T14:38:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.267+0000 7f509e7fc640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 0x7f5080078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:02.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.275+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5088020030 con 0x7f50a00b27c0 2024-06-29T14:38:02.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.468+0000 7f509e7fc640 1 -- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 msgr2=0x7f5080078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1607211985 2024-06-29T14:38:02.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.468+0000 7f509e7fc640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 0x7f5080078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:02.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.491+0000 7f50a5a47640 1 -- 172.21.15.59:0/700897917 --> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f50a015b440 con 0x7f5080076440 2024-06-29T14:38:02.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.869+0000 7f509e7fc640 1 -- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 msgr2=0x7f5080078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1607211985 2024-06-29T14:38:02.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:02.869+0000 7f509e7fc640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 0x7f5080078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:38:03.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:03.670+0000 7f509e7fc640 1 -- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 msgr2=0x7f5080078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1607211985 2024-06-29T14:38:03.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:03.670+0000 7f509e7fc640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 0x7f5080078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:38:05.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:05.031+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 40) ==== 97929+0+0 (secure 0 0 0) 0x7f5088095d60 con 0x7f50a00b27c0 2024-06-29T14:38:05.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:05.031+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 msgr2=0x7f5080078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:05.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:05.031+0000 7f5097fff640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/1607211985,v1:172.21.15.153:6833/1607211985] conn(0x7f5080076440 0x7f5080078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1954ee9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1954ee9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1954ee9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1954ee9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1954ee9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1951ee3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1951ee3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1951ee3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1951ee3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.200 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:05.202+0000 7f1951ee3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:06.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.034+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 41) ==== 98056+0+0 (secure 0 0 0) 0x7f508805cb10 con 0x7f50a00b27c0 2024-06-29T14:38:06.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.035+0000 7f5097fff640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f508007e980 0x7f5080080da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:06.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.035+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 --> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f5088020ec0 con 0x7f508007e980 2024-06-29T14:38:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.040+0000 7f509e7fc640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f508007e980 0x7f5080080da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.042+0000 7f509e7fc640 1 --2- 172.21.15.59:0/700897917 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f508007e980 0x7f5080080da0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5090034000 tx=0x7f5090004730 comp rx=0 tx=0).ready entity=mgr.6464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:06.048 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:06.050+0000 7f195bef7640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-06-29T14:38:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.050+0000 7f5097fff640 1 -- 172.21.15.59:0/700897917 <== mgr.6464 v2:172.21.15.153:6832/3952610634 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f5088020ec0 con 0x7f508007e980 2024-06-29T14:38:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-06-29T14:38:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:38:06.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:38:06.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.184+0000 7f0e939bd640 1 -- 172.21.15.59:0/2780966659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c08c190 msgr2=0x7f0e8c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:06.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.184+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2780966659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c08c190 0x7f0e8c0cf080 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009830 tx=0x7f0e8402a510 comp rx=0 tx=0).stop 2024-06-29T14:38:06.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.184+0000 7f0e939bd640 1 -- 172.21.15.59:0/2780966659 shutdown_connections 2024-06-29T14:38:06.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.184+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2780966659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c0cf5c0 0x7f0e8c0d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:06.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.184+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2780966659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c08c190 0x7f0e8c0cf080 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:06.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.184+0000 7f0e939bd640 1 -- 172.21.15.59:0/2780966659 >> 172.21.15.59:0/2780966659 conn(0x7f0e8c0b5610 msgr2=0x7f0e8c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:06.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.184+0000 7f0e939bd640 1 -- 172.21.15.59:0/2780966659 shutdown_connections 2024-06-29T14:38:06.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.184+0000 7f0e939bd640 1 -- 172.21.15.59:0/2780966659 wait complete. 2024-06-29T14:38:06.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.186+0000 7f0e939bd640 1 Processor -- start 2024-06-29T14:38:06.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.186+0000 7f0e939bd640 1 -- start start 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e939bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c08c190 0x7f0e8c14d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e939bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c0ce950 0x7f0e8c14ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e939bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c0cf5c0 0x7f0e8c14f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e939bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c0b7fa0 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e939bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c0b7e20 con 0x7f0e8c08c190 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e939bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8c0b8120 con 0x7f0e8c0ce950 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e91f5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c0cf5c0 0x7f0e8c14f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e91f5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c0cf5c0 0x7f0e8c14f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53996/0 (socket says 172.21.15.59:53996) 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e91f5a640 1 -- 172.21.15.59:0/2310008553 learned_addr learned my addr 172.21.15.59:0/2310008553 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:06.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e90f58640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c0ce950 0x7f0e8c14ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.187+0000 7f0e91759640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c08c190 0x7f0e8c14d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e91f5a640 1 -- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c0ce950 msgr2=0x7f0e8c14ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e91f5a640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c0ce950 0x7f0e8c14ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e91f5a640 1 -- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c08c190 msgr2=0x7f0e8c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e91f5a640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c08c190 0x7f0e8c14d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e91f5a640 1 -- 172.21.15.59:0/2310008553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c0cd340 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e90f58640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c0ce950 0x7f0e8c14ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e91759640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c08c190 0x7f0e8c14d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e91f5a640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c0cf5c0 0x7f0e8c14f280 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00cad0 tx=0x7f0e7c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e89ffb640 1 -- 172.21.15.59:0/2310008553 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c016020 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e89ffb640 1 -- 172.21.15.59:0/2310008553 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e7c00ad40 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8c0cd620 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e89ffb640 1 -- 172.21.15.59:0/2310008553 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c012680 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.188+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8c14e540 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.189+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8c068c00 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.190+0000 7f0e89ffb640 1 -- 172.21.15.59:0/2310008553 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 41) ==== 98056+0+0 (secure 0 0 0) 0x7f0e7c019020 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.190+0000 7f0e89ffb640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0e78076170 0x7f0e78078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:06.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.191+0000 7f0e89ffb640 1 -- 172.21.15.59:0/2310008553 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7f0e7c097e20 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.191+0000 7f0e91759640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0e78076170 0x7f0e78078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:06.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.191+0000 7f0e91759640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0e78076170 0x7f0e78078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0e84034000 tx=0x7f0e84004730 comp rx=0 tx=0).ready entity=mgr.6464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:06.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.195+0000 7f0e89ffb640 1 -- 172.21.15.59:0/2310008553 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0e8c068c00 con 0x7f0e8c0cf5c0 2024-06-29T14:38:06.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:06.444+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f0e8c0cf080 con 0x7f0e8c0cf5c0 2024-06-29T14:38:07.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.085+0000 7f0e89ffb640 1 -- 172.21.15.59:0/2310008553 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v42) ==== 93+0+0 (secure 0 0 0) 0x7f0e7c020020 con 0x7f0e8c0cf5c0 2024-06-29T14:38:07.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0e78076170 msgr2=0x7f0e78078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0e78076170 0x7f0e78078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0e84034000 tx=0x7f0e84004730 comp rx=0 tx=0).stop 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c0cf5c0 msgr2=0x7f0e8c14f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c0cf5c0 0x7f0e8c14f280 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00cad0 tx=0x7f0e7c00cf90 comp rx=0 tx=0).stop 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 shutdown_connections 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0e78076170 0x7f0e78078680 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e8c0cf5c0 0x7f0e8c14f280 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e8c0ce950 0x7f0e8c14ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 --2- 172.21.15.59:0/2310008553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e8c08c190 0x7f0e8c14d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 >> 172.21.15.59:0/2310008553 conn(0x7f0e8c0b5610 msgr2=0x7f0e8c15e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 shutdown_connections 2024-06-29T14:38:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.088+0000 7f0e939bd640 1 -- 172.21.15.59:0/2310008553 wait complete. 2024-06-29T14:38:07.101 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:38:07.101 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:38:07.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:38:07.190 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:38:07.190 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:38:07.190 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:38:07.190 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:07.206 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:38:07.206 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:07.263 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:07.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.424+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1864287597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a0003ac0 msgr2=0x7f54a0003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.424+0000 7f54bdd4a640 1 --2- 172.21.15.59:0/1864287597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54a0003ac0 0x7f54a0003f30 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f54a800c260 tx=0x7f54a0005830 comp rx=0 tx=0).stop 2024-06-29T14:38:07.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.424+0000 7f54b7fff640 1 -- 172.21.15.59:0/1864287597 reap_dead start 2024-06-29T14:38:07.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.424+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1864287597 shutdown_connections 2024-06-29T14:38:07.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.424+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1864287597 >> 172.21.15.59:0/1864287597 conn(0x7f54b80d3ee0 msgr2=0x7f54b80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.424+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1864287597 shutdown_connections 2024-06-29T14:38:07.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.424+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1864287597 wait complete. 2024-06-29T14:38:07.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.426+0000 7f54bdd4a640 1 Processor -- start 2024-06-29T14:38:07.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.426+0000 7f54bdd4a640 1 -- start start 2024-06-29T14:38:07.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54bdd4a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a0003ac0 0x7f54b8156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54bdd4a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54b815bc20 0x7f54b8159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54bdd4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54b80aabf0 0x7f54b815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54bdd4a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f54b815fd40 con 0x7f54b80aabf0 2024-06-29T14:38:07.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54bdd4a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54b815fbc0 con 0x7f54a0003ac0 2024-06-29T14:38:07.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54bdd4a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f54b815fec0 con 0x7f54b815bc20 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54b80aabf0 0x7f54b815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54b80aabf0 0x7f54b815a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54044/0 (socket says 172.21.15.59:54044) 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 -- 172.21.15.59:0/1898759125 learned_addr learned my addr 172.21.15.59:0/1898759125 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b6ffd640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54b815bc20 0x7f54b8159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b77fe640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a0003ac0 0x7f54b8156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 -- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54b815bc20 msgr2=0x7f54b8159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54b815bc20 0x7f54b8159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 -- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a0003ac0 msgr2=0x7f54b8156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a0003ac0 0x7f54b8156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 -- 172.21.15.59:0/1898759125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b80ca2b0 con 0x7f54b80aabf0 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54b7fff640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54b80aabf0 0x7f54b815a360 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f54b0005fd0 tx=0x7f54b0011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f549ffff640 1 -- 172.21.15.59:0/1898759125 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0004b20 con 0x7f54b80aabf0 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f549ffff640 1 -- 172.21.15.59:0/1898759125 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54b0004cc0 con 0x7f54b80aabf0 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.428+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b80ca590 con 0x7f54b80aabf0 2024-06-29T14:38:07.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.429+0000 7f549ffff640 1 -- 172.21.15.59:0/1898759125 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0003bd0 con 0x7f54b80aabf0 2024-06-29T14:38:07.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.429+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b80ab300 con 0x7f54b80aabf0 2024-06-29T14:38:07.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.430+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b8161a50 con 0x7f54b80aabf0 2024-06-29T14:38:07.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.431+0000 7f549ffff640 1 -- 172.21.15.59:0/1898759125 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f54b0004030 con 0x7f54b80aabf0 2024-06-29T14:38:07.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.431+0000 7f549ffff640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f54a40763f0 0x7f54a4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.431+0000 7f54b77fe640 1 -- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f54a40763f0 msgr2=0x7f54a4078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:07.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.432+0000 7f549ffff640 1 -- 172.21.15.59:0/1898759125 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7f54b0098be0 con 0x7f54b80aabf0 2024-06-29T14:38:07.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.432+0000 7f54b77fe640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f54a40763f0 0x7f54a4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:07.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.435+0000 7f549ffff640 1 -- 172.21.15.59:0/1898759125 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f54b8161a50 con 0x7f54b80aabf0 2024-06-29T14:38:07.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.629+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54b8160450 con 0x7f54b80aabf0 2024-06-29T14:38:07.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.630+0000 7f549ffff640 1 -- 172.21.15.59:0/1898759125 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+17469 (secure 0 0 0) 0x7f54b0062130 con 0x7f54b80aabf0 2024-06-29T14:38:07.629 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:07.629 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":102,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:05.022034+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:51.120910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.698419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2366299610":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6833/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6832/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3522491343":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3511527873":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1670969120":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/4085525721":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1421481485":"2024-06-30T14:38:05.021970+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f54a40763f0 msgr2=0x7f54a4078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f54a40763f0 0x7f54a4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54b80aabf0 msgr2=0x7f54b815a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54b80aabf0 0x7f54b815a360 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f54b0005fd0 tx=0x7f54b0011ef0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 shutdown_connections 2024-06-29T14:38:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f54a40763f0 0x7f54a4078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54b80aabf0 0x7f54b815a360 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54b815bc20 0x7f54b8159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 --2- 172.21.15.59:0/1898759125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54a0003ac0 0x7f54b8156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 >> 172.21.15.59:0/1898759125 conn(0x7f54b80d3ee0 msgr2=0x7f54b806c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 shutdown_connections 2024-06-29T14:38:07.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.633+0000 7f54bdd4a640 1 -- 172.21.15.59:0/1898759125 wait complete. 2024-06-29T14:38:07.643 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:37:51.120910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:37:51.698419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:38:07.643 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:38:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.767+0000 7f652e3f5640 1 -- 172.21.15.59:0/2280954055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f652815efe0 msgr2=0x7f652815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.767+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2280954055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f652815efe0 0x7f652815f9b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6520005fd0 tx=0x7f652002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.767+0000 7f652e3f5640 1 -- 172.21.15.59:0/2280954055 shutdown_connections 2024-06-29T14:38:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.767+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2280954055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f652815efe0 0x7f652815f9b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.767+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2280954055 >> v2:172.21.15.153:3300/0 conn(0x7f65280d23c0 0x7f65280b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.767+0000 7f652e3f5640 1 -- 172.21.15.59:0/2280954055 >> 172.21.15.59:0/2280954055 conn(0x7f65280d3ee0 msgr2=0x7f65280d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.767+0000 7f652e3f5640 1 -- 172.21.15.59:0/2280954055 shutdown_connections 2024-06-29T14:38:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.767+0000 7f652e3f5640 1 -- 172.21.15.59:0/2280954055 wait complete. 2024-06-29T14:38:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.769+0000 7f652e3f5640 1 Processor -- start 2024-06-29T14:38:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.769+0000 7f652e3f5640 1 -- start start 2024-06-29T14:38:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f652e3f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65280b3ed0 0x7f65280b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f652e3f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65280d23c0 0x7f652815da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f652e3f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f652815efe0 0x7f652815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f652e3f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6528160280 con 0x7f65280d23c0 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f652e3f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6528160100 con 0x7f65280b3ed0 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f652e3f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6528160400 con 0x7f652815efe0 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f65277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65280d23c0 0x7f652815da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f6527fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65280b3ed0 0x7f65280b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f65277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65280d23c0 0x7f652815da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54050/0 (socket says 172.21.15.59:54050) 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f6527fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65280b3ed0 0x7f65280b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:47628/0 (socket says 172.21.15.59:47628) 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f65277fe640 1 -- 172.21.15.59:0/2853327160 learned_addr learned my addr 172.21.15.59:0/2853327160 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.770+0000 7f652c992640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f652815efe0 0x7f652815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f65277fe640 1 -- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f652815efe0 msgr2=0x7f652815ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f65277fe640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f652815efe0 0x7f652815ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f65277fe640 1 -- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65280b3ed0 msgr2=0x7f65280b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f65277fe640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65280b3ed0 0x7f65280b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f65277fe640 1 -- 172.21.15.59:0/2853327160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65280b30f0 con 0x7f65280d23c0 2024-06-29T14:38:07.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f65277fe640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65280d23c0 0x7f652815da90 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f651c00c730 tx=0x7f651c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:07.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f6524ff9640 1 -- 172.21.15.59:0/2853327160 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c006890 con 0x7f65280d23c0 2024-06-29T14:38:07.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6528159470 con 0x7f65280d23c0 2024-06-29T14:38:07.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f6524ff9640 1 -- 172.21.15.59:0/2853327160 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f651c006a30 con 0x7f65280d23c0 2024-06-29T14:38:07.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f6524ff9640 1 -- 172.21.15.59:0/2853327160 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c007030 con 0x7f65280d23c0 2024-06-29T14:38:07.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.771+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6528159a30 con 0x7f65280d23c0 2024-06-29T14:38:07.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.772+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65280b3990 con 0x7f65280d23c0 2024-06-29T14:38:07.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.773+0000 7f6524ff9640 1 -- 172.21.15.59:0/2853327160 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f651c0071d0 con 0x7f65280d23c0 2024-06-29T14:38:07.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.774+0000 7f6524ff9640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f6514076420 0x7f6514078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.774+0000 7f6527fff640 1 -- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f6514076420 msgr2=0x7f6514078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:07.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.774+0000 7f6524ff9640 1 -- 172.21.15.59:0/2853327160 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7f651c097450 con 0x7f65280d23c0 2024-06-29T14:38:07.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.774+0000 7f6527fff640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f6514076420 0x7f6514078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.778+0000 7f6524ff9640 1 -- 172.21.15.59:0/2853327160 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f65280b42e0 con 0x7f65280d23c0 2024-06-29T14:38:07.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.969+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f65280d2950 con 0x7f65280d23c0 2024-06-29T14:38:07.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.969+0000 7f6524ff9640 1 -- 172.21.15.59:0/2853327160 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7f651c060380 con 0x7f65280d23c0 2024-06-29T14:38:07.968 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:38:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f6514076420 msgr2=0x7f6514078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f6514076420 0x7f6514078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65280d23c0 msgr2=0x7f652815da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65280d23c0 0x7f652815da90 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f651c00c730 tx=0x7f651c00cbf0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 shutdown_connections 2024-06-29T14:38:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f6514076420 0x7f6514078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f652815efe0 0x7f652815ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65280d23c0 0x7f652815da90 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 --2- 172.21.15.59:0/2853327160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65280b3ed0 0x7f65280b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 >> 172.21.15.59:0/2853327160 conn(0x7f65280d3ee0 msgr2=0x7f652806c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 shutdown_connections 2024-06-29T14:38:07.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:07.972+0000 7f652e3f5640 1 -- 172.21.15.59:0/2853327160 wait complete. 2024-06-29T14:38:07.982 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:38:08.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.111+0000 7f6522e26640 1 -- 172.21.15.59:0/546934810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 msgr2=0x7f651c0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.111+0000 7f6522e26640 1 --2- 172.21.15.59:0/546934810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 0x7f651c0b3990 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f650c005fd0 tx=0x7f650c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.111+0000 7f6522e26640 1 -- 172.21.15.59:0/546934810 shutdown_connections 2024-06-29T14:38:08.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.111+0000 7f6522e26640 1 --2- 172.21.15.59:0/546934810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f651c0b3ed0 0x7f651c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.111+0000 7f6522e26640 1 --2- 172.21.15.59:0/546934810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 0x7f651c0b3990 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.111+0000 7f6522e26640 1 -- 172.21.15.59:0/546934810 >> 172.21.15.59:0/546934810 conn(0x7f651c0d3ee0 msgr2=0x7f651c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.111+0000 7f6522e26640 1 -- 172.21.15.59:0/546934810 shutdown_connections 2024-06-29T14:38:08.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.111+0000 7f6522e26640 1 -- 172.21.15.59:0/546934810 wait complete. 2024-06-29T14:38:08.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.113+0000 7f6522e26640 1 Processor -- start 2024-06-29T14:38:08.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.113+0000 7f6522e26640 1 -- start start 2024-06-29T14:38:08.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.114+0000 7f6522e26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f651c0b3ed0 0x7f651c14f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.114+0000 7f6522e26640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 0x7f651c1e9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.114+0000 7f6522e26640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f651c15ee10 0x7f651c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.114+0000 7f6522e26640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f651c160240 con 0x7f651c0d23c0 2024-06-29T14:38:08.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.114+0000 7f6522e26640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f651c1600c0 con 0x7f651c0b3ed0 2024-06-29T14:38:08.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.114+0000 7f6522e26640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f651c1603c0 con 0x7f651c15ee10 2024-06-29T14:38:08.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f651bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 0x7f651c1e9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f651bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 0x7f651c1e9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54066/0 (socket says 172.21.15.59:54066) 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f651bfff640 1 -- 172.21.15.59:0/3375084611 learned_addr learned my addr 172.21.15.59:0/3375084611 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f6520bc2640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f651c0b3ed0 0x7f651c14f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f65213c3640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f651c15ee10 0x7f651c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f6520bc2640 1 -- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f651c15ee10 msgr2=0x7f651c154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f6520bc2640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f651c15ee10 0x7f651c154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f6520bc2640 1 -- 172.21.15.59:0/3375084611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 msgr2=0x7f651c1e9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f6520bc2640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 0x7f651c1e9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f6520bc2640 1 -- 172.21.15.59:0/3375084611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c14f750 con 0x7f651c0b3ed0 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f651bfff640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 0x7f651c1e9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f6520bc2640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f651c0b3ed0 0x7f651c14f210 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f651c0cf190 tx=0x7f650c005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:08.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.115+0000 7f65197fa640 1 -- 172.21.15.59:0/3375084611 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c03e030 con 0x7f651c0b3ed0 2024-06-29T14:38:08.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.116+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651c1e7a30 con 0x7f651c0b3ed0 2024-06-29T14:38:08.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.116+0000 7f65197fa640 1 -- 172.21.15.59:0/3375084611 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f650c003a70 con 0x7f651c0b3ed0 2024-06-29T14:38:08.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.116+0000 7f65197fa640 1 -- 172.21.15.59:0/3375084611 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c035040 con 0x7f651c0b3ed0 2024-06-29T14:38:08.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.116+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651c1e7ff0 con 0x7f651c0b3ed0 2024-06-29T14:38:08.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.116+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f651c15e8d0 con 0x7f651c0b3ed0 2024-06-29T14:38:08.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.117+0000 7f65197fa640 1 -- 172.21.15.59:0/3375084611 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f650c041030 con 0x7f651c0b3ed0 2024-06-29T14:38:08.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.118+0000 7f65197fa640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f64fc0763f0 0x7f64fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.118+0000 7f651bfff640 1 -- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f64fc0763f0 msgr2=0x7f64fc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:08.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.118+0000 7f65197fa640 1 -- 172.21.15.59:0/3375084611 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7f650c0b85f0 con 0x7f651c0b3ed0 2024-06-29T14:38:08.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.118+0000 7f651bfff640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f64fc0763f0 0x7f64fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:08.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.126+0000 7f65197fa640 1 -- 172.21.15.59:0/3375084611 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f651c14f750 con 0x7f651c0b3ed0 2024-06-29T14:38:08.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.317+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f651c0d2950 con 0x7f651c0b3ed0 2024-06-29T14:38:08.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.318+0000 7f65197fa640 1 -- 172.21.15.59:0/3375084611 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7f650c081370 con 0x7f651c0b3ed0 2024-06-29T14:38:08.316 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:38:08.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f651bfff640 1 -- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f64fc0763f0 msgr2=0x7f64fc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:08.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f651bfff640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f64fc0763f0 0x7f64fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:08.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f64fc0763f0 msgr2=0x7f64fc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:08.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f64fc0763f0 0x7f64fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f651c0b3ed0 msgr2=0x7f651c14f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f651c0b3ed0 0x7f651c14f210 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f651c0cf190 tx=0x7f650c005750 comp rx=0 tx=0).stop 2024-06-29T14:38:08.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 shutdown_connections 2024-06-29T14:38:08.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f64fc0763f0 0x7f64fc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f651c15ee10 0x7f651c154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f651c0d23c0 0x7f651c1e9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 --2- 172.21.15.59:0/3375084611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f651c0b3ed0 0x7f651c14f210 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 >> 172.21.15.59:0/3375084611 conn(0x7f651c0d3ee0 msgr2=0x7f651c06d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 shutdown_connections 2024-06-29T14:38:08.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.320+0000 7f6522e26640 1 -- 172.21.15.59:0/3375084611 wait complete. 2024-06-29T14:38:08.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:38:08.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.456+0000 7fce1d065640 1 -- 172.21.15.59:0/2754395151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce00004800 msgr2=0x7fce00004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.456+0000 7fce1d065640 1 --2- 172.21.15.59:0/2754395151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce00004800 0x7fce00004c10 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c260 tx=0x7fce00005830 comp rx=0 tx=0).stop 2024-06-29T14:38:08.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.456+0000 7fce1d065640 1 -- 172.21.15.59:0/2754395151 shutdown_connections 2024-06-29T14:38:08.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.456+0000 7fce1d065640 1 --2- 172.21.15.59:0/2754395151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce00004800 0x7fce00004c10 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.456+0000 7fce1d065640 1 -- 172.21.15.59:0/2754395151 >> 172.21.15.59:0/2754395151 conn(0x7fce18073e10 msgr2=0x7fce18074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.456+0000 7fce1d065640 1 -- 172.21.15.59:0/2754395151 shutdown_connections 2024-06-29T14:38:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.456+0000 7fce1d065640 1 -- 172.21.15.59:0/2754395151 wait complete. 2024-06-29T14:38:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.458+0000 7fce1d065640 1 Processor -- start 2024-06-29T14:38:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.458+0000 7fce1d065640 1 -- start start 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1d065640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fce00004800 0x7fce180aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1d065640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce1814b440 0x7fce180af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1d065640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce1814be90 0x7fce1814c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1d065640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fce180d1960 con 0x7fce1814b440 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1d065640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fce180d17e0 con 0x7fce1814be90 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1d065640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fce1815bc20 con 0x7fce00004800 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce1814b440 0x7fce180af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce1814b440 0x7fce180af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54080/0 (socket says 172.21.15.59:54080) 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1659c640 1 -- 172.21.15.59:0/1891940332 learned_addr learned my addr 172.21.15.59:0/1891940332 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce16d9d640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fce00004800 0x7fce180aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.459+0000 7fce1759e640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce1814be90 0x7fce1814c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fce1659c640 1 -- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fce00004800 msgr2=0x7fce180aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fce1659c640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fce00004800 0x7fce180aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fce1659c640 1 -- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce1814be90 msgr2=0x7fce1814c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fce1659c640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce1814be90 0x7fce1814c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fce1659c640 1 -- 172.21.15.59:0/1891940332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce18155d60 con 0x7fce1814b440 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fce1659c640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce1814b440 0x7fce180af450 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c380 tx=0x7fce0c001d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fcdff7fe640 1 -- 172.21.15.59:0/1891940332 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c031040 con 0x7fce1814b440 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fcdff7fe640 1 -- 172.21.15.59:0/1891940332 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce0c0035f0 con 0x7fce1814b440 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce180adb90 con 0x7fce1814b440 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fcdff7fe640 1 -- 172.21.15.59:0/1891940332 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c003a20 con 0x7fce1814b440 2024-06-29T14:38:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.460+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce180ae150 con 0x7fce1814b440 2024-06-29T14:38:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.462+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce1815c110 con 0x7fce1814b440 2024-06-29T14:38:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.462+0000 7fcdff7fe640 1 -- 172.21.15.59:0/1891940332 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7fce0c034070 con 0x7fce1814b440 2024-06-29T14:38:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.462+0000 7fcdff7fe640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7fce040763f0 0x7fce04078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.463+0000 7fcdff7fe640 1 -- 172.21.15.59:0/1891940332 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7fce0c0b7d20 con 0x7fce1814b440 2024-06-29T14:38:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.463+0000 7fce16d9d640 1 -- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7fce040763f0 msgr2=0x7fce04078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.463+0000 7fce16d9d640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7fce040763f0 0x7fce04078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:08.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.467+0000 7fcdff7fe640 1 -- 172.21.15.59:0/1891940332 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fce0c006630 con 0x7fce1814b440 2024-06-29T14:38:08.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.664+0000 7fce16d9d640 1 -- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7fce040763f0 msgr2=0x7fce04078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.664+0000 7fce16d9d640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7fce040763f0 0x7fce04078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.707+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fce1815b200 con 0x7fce1814b440 2024-06-29T14:38:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.707+0000 7fcdff7fe640 1 -- 172.21.15.59:0/1891940332 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7fce0c089250 con 0x7fce1814b440 2024-06-29T14:38:08.706 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:38:08.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7fce040763f0 msgr2=0x7fce04078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:08.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7fce040763f0 0x7fce04078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce1814b440 msgr2=0x7fce180af450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce1814b440 0x7fce180af450 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c380 tx=0x7fce0c001d70 comp rx=0 tx=0).stop 2024-06-29T14:38:08.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 shutdown_connections 2024-06-29T14:38:08.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7fce040763f0 0x7fce04078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce1814be90 0x7fce1814c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce1814b440 0x7fce180af450 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 --2- 172.21.15.59:0/1891940332 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fce00004800 0x7fce180aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 >> 172.21.15.59:0/1891940332 conn(0x7fce18073e10 msgr2=0x7fce18074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 shutdown_connections 2024-06-29T14:38:08.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.711+0000 7fce1d065640 1 -- 172.21.15.59:0/1891940332 wait complete. 2024-06-29T14:38:08.725 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:38:08.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.846+0000 7f17d092a640 1 -- 172.21.15.59:0/526542764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17cc0d1c30 msgr2=0x7f17cc166a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.846+0000 7f17d092a640 1 --2- 172.21.15.59:0/526542764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17cc0d1c30 0x7f17cc166a20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f17c4005fd0 tx=0x7f17c402a940 comp rx=0 tx=0).stop 2024-06-29T14:38:08.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.847+0000 7f17d092a640 1 -- 172.21.15.59:0/526542764 shutdown_connections 2024-06-29T14:38:08.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.847+0000 7f17d092a640 1 --2- 172.21.15.59:0/526542764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17cc0d1c30 0x7f17cc166a20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.847+0000 7f17d092a640 1 -- 172.21.15.59:0/526542764 >> 172.21.15.59:0/526542764 conn(0x7f17cc073e10 msgr2=0x7f17cc074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.847+0000 7f17d092a640 1 -- 172.21.15.59:0/526542764 shutdown_connections 2024-06-29T14:38:08.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.847+0000 7f17d092a640 1 -- 172.21.15.59:0/526542764 wait complete. 2024-06-29T14:38:08.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.849+0000 7f17d092a640 1 Processor -- start 2024-06-29T14:38:08.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.849+0000 7f17d092a640 1 -- start start 2024-06-29T14:38:08.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17d092a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17cc0b2270 0x7f17cc1f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17d092a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17cc0b2fa0 0x7f17cc153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17d092a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17cc0d1c30 0x7f17cc15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17d092a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc1670e0 con 0x7f17cc0b2270 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17d092a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc166f60 con 0x7f17cc0b2fa0 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17d092a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f17cc167260 con 0x7f17cc0d1c30 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17ca59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17cc0b2270 0x7f17cc1f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17ca59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17cc0b2270 0x7f17cc1f5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54086/0 (socket says 172.21.15.59:54086) 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17ca59c640 1 -- 172.21.15.59:0/3929194370 learned_addr learned my addr 172.21.15.59:0/3929194370 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17cad9d640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17cc0d1c30 0x7f17cc15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.850+0000 7f17c9d9b640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17cc0b2fa0 0x7f17cc153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17ca59c640 1 -- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17cc0d1c30 msgr2=0x7f17cc15e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17ca59c640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17cc0d1c30 0x7f17cc15e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17ca59c640 1 -- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17cc0b2fa0 msgr2=0x7f17cc153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17ca59c640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17cc0b2fa0 0x7f17cc153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17ca59c640 1 -- 172.21.15.59:0/3929194370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc1548b0 con 0x7f17cc0b2270 2024-06-29T14:38:08.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17ca59c640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17cc0b2270 0x7f17cc1f5d00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00c400 tx=0x7f17bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:08.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17c2ffd640 1 -- 172.21.15.59:0/3929194370 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc011790 con 0x7f17cc0b2270 2024-06-29T14:38:08.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17c2ffd640 1 -- 172.21.15.59:0/3929194370 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17bc017070 con 0x7f17cc0b2270 2024-06-29T14:38:08.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17cc15c650 con 0x7f17cc0b2270 2024-06-29T14:38:08.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17c2ffd640 1 -- 172.21.15.59:0/3929194370 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc011cc0 con 0x7f17cc0b2270 2024-06-29T14:38:08.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.851+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17cc15cc10 con 0x7f17cc0b2270 2024-06-29T14:38:08.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.852+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17cc15bc20 con 0x7f17cc0b2270 2024-06-29T14:38:08.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.853+0000 7f17c2ffd640 1 -- 172.21.15.59:0/3929194370 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f17bc01c460 con 0x7f17cc0b2270 2024-06-29T14:38:08.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.854+0000 7f17c2ffd640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f17ac0763f0 0x7f17ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.854+0000 7f17c9d9b640 1 -- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f17ac0763f0 msgr2=0x7f17ac078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:08.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.854+0000 7f17c2ffd640 1 -- 172.21.15.59:0/3929194370 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7f17bc097910 con 0x7f17cc0b2270 2024-06-29T14:38:08.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.854+0000 7f17c9d9b640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f17ac0763f0 0x7f17ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:08.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:08.858+0000 7f17c2ffd640 1 -- 172.21.15.59:0/3929194370 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f17bc0607a0 con 0x7f17cc0b2270 2024-06-29T14:38:09.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.055+0000 7f17c9d9b640 1 -- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f17ac0763f0 msgr2=0x7f17ac078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:09.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.055+0000 7f17c9d9b640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f17ac0763f0 0x7f17ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:09.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.063+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f17cc14e9b0 con 0x7f17cc0b2270 2024-06-29T14:38:09.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.063+0000 7f17c2ffd640 1 -- 172.21.15.59:0/3929194370 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v102) ==== 80+0+478 (secure 0 0 0) 0x7f17bc067e20 con 0x7f17cc0b2270 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2366299610 2024-06-30T14:38:05.021970+0000 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/1607211985 2024-06-30T14:38:05.021970+0000 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/1607211985 2024-06-30T14:38:05.021970+0000 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3522491343 2024-06-30T14:38:05.021970+0000 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3511527873 2024-06-30T14:38:05.021970+0000 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1670969120 2024-06-30T14:38:05.021970+0000 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/4085525721 2024-06-30T14:38:05.021970+0000 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1421481485 2024-06-30T14:38:05.021970+0000 2024-06-29T14:38:09.062 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T14:38:09.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f17ac0763f0 msgr2=0x7f17ac078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:09.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f17ac0763f0 0x7f17ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17cc0b2270 msgr2=0x7f17cc1f5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17cc0b2270 0x7f17cc1f5d00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00c400 tx=0x7f17bc00c8c0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 shutdown_connections 2024-06-29T14:38:09.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f17ac0763f0 0x7f17ac078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17cc0d1c30 0x7f17cc15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17cc0b2fa0 0x7f17cc153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 --2- 172.21.15.59:0/3929194370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17cc0b2270 0x7f17cc1f5d00 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.065+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 >> 172.21.15.59:0/3929194370 conn(0x7f17cc073e10 msgr2=0x7f17cc14df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.066+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 shutdown_connections 2024-06-29T14:38:09.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.066+0000 7f17d092a640 1 -- 172.21.15.59:0/3929194370 wait complete. 2024-06-29T14:38:09.075 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:38:09.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:09.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:09.105 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:38:09.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:09.106 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:38:09.162 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:38:09.162 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:38:09.162 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:09.178 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:38:09.178 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:38:09.178 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:38:09.179 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:09.218 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:38:09.219 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:38:09.219 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:38:09.219 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:09.234 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:38:09.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:09.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.379+0000 7f4584a4e640 1 -- 172.21.15.59:0/2039677709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 msgr2=0x7f45800d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.379+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2039677709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 0x7f45800d1520 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4568009830 tx=0x7f456802a510 comp rx=0 tx=0).stop 2024-06-29T14:38:09.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.379+0000 7f4584a4e640 1 -- 172.21.15.59:0/2039677709 shutdown_connections 2024-06-29T14:38:09.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.379+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2039677709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45800d1a60 0x7f45800b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.379+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2039677709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 0x7f45800d1520 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.379+0000 7f4584a4e640 1 -- 172.21.15.59:0/2039677709 >> 172.21.15.59:0/2039677709 conn(0x7f4580073e10 msgr2=0x7f4580074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.380+0000 7f4584a4e640 1 -- 172.21.15.59:0/2039677709 shutdown_connections 2024-06-29T14:38:09.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.380+0000 7f4584a4e640 1 -- 172.21.15.59:0/2039677709 wait complete. 2024-06-29T14:38:09.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.383+0000 7f4584a4e640 1 Processor -- start 2024-06-29T14:38:09.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.383+0000 7f4584a4e640 1 -- start start 2024-06-29T14:38:09.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f4584a4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45800b2270 0x7f458014de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f4584a4e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 0x7f458014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f4584a4e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45800d1a60 0x7f45800b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f4584a4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f45800b6400 con 0x7f45800b2270 2024-06-29T14:38:09.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f4584a4e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45800b6280 con 0x7f45800b2fa0 2024-06-29T14:38:09.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f4584a4e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45800b6580 con 0x7f45800d1a60 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f457e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45800b2270 0x7f458014de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f457e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45800b2270 0x7f458014de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54094/0 (socket says 172.21.15.59:54094) 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f457e59c640 1 -- 172.21.15.59:0/2563526327 learned_addr learned my addr 172.21.15.59:0/2563526327 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f4577fff640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 0x7f458014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.385+0000 7f457ed9d640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45800d1a60 0x7f45800b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f457e59c640 1 -- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45800d1a60 msgr2=0x7f45800b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f457e59c640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45800d1a60 0x7f45800b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f457e59c640 1 -- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 msgr2=0x7f458014d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f457e59c640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 0x7f458014d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f457e59c640 1 -- 172.21.15.59:0/2563526327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45800b3bd0 con 0x7f45800b2270 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f457ed9d640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45800d1a60 0x7f45800b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f4577fff640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 0x7f458014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:09.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f457e59c640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45800b2270 0x7f458014de40 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f457000c410 tx=0x7f457000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:09.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f4576ffd640 1 -- 172.21.15.59:0/2563526327 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570011790 con 0x7f45800b2270 2024-06-29T14:38:09.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f4576ffd640 1 -- 172.21.15.59:0/2563526327 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4570017070 con 0x7f45800b2270 2024-06-29T14:38:09.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.386+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45800b3eb0 con 0x7f45800b2270 2024-06-29T14:38:09.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.387+0000 7f4576ffd640 1 -- 172.21.15.59:0/2563526327 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570011ce0 con 0x7f45800b2270 2024-06-29T14:38:09.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.387+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f458006ee70 con 0x7f45800b2270 2024-06-29T14:38:09.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.388+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45800b6aa0 con 0x7f45800b2270 2024-06-29T14:38:09.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.389+0000 7f4576ffd640 1 -- 172.21.15.59:0/2563526327 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f457001c460 con 0x7f45800b2270 2024-06-29T14:38:09.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.389+0000 7f4576ffd640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f45600763f0 0x7f4560078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.390+0000 7f4577fff640 1 -- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f45600763f0 msgr2=0x7f4560078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:09.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.390+0000 7f4576ffd640 1 -- 172.21.15.59:0/2563526327 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7f4570097880 con 0x7f45800b2270 2024-06-29T14:38:09.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.390+0000 7f4577fff640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f45600763f0 0x7f4560078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:09.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.394+0000 7f4576ffd640 1 -- 172.21.15.59:0/2563526327 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4570060860 con 0x7f45800b2270 2024-06-29T14:38:09.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.591+0000 7f4577fff640 1 -- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f45600763f0 msgr2=0x7f4560078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:09.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.591+0000 7f4577fff640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f45600763f0 0x7f4560078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:09.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.627+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45800d13c0 con 0x7f45800b2270 2024-06-29T14:38:09.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.628+0000 7f4576ffd640 1 -- 172.21.15.59:0/2563526327 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5228 (secure 0 0 0) 0x7f4570064b60 con 0x7f45800b2270 2024-06-29T14:38:09.626 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:09.626 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":46,"btime":"2024-06-29T14:37:56:391508+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2174310652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2174310652},{"type":"v1","addr":"172.21.15.59:6835","nonce":2174310652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3217643445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3217643445},{"type":"v1","addr":"172.21.15.153:6836","nonce":3217643445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/832666341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":832666341},{"type":"v1","addr":"172.21.15.59:6833","nonce":832666341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.753189+0000","modified":"2024-06-29T14:37:55.781006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7365},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7365":{"gid":7365,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/1942687234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1942687234},{"type":"v1","addr":"172.21.15.153:6837","nonce":1942687234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7365,"qdb_cluster":[7365]},"id":5}]} 2024-06-29T14:38:09.627 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 46 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f45600763f0 msgr2=0x7f4560078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f45600763f0 0x7f4560078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45800b2270 msgr2=0x7f458014de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45800b2270 0x7f458014de40 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f457000c410 tx=0x7f457000c8d0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 shutdown_connections 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f45600763f0 0x7f4560078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45800d1a60 0x7f45800b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45800b2fa0 0x7f458014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 --2- 172.21.15.59:0/2563526327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45800b2270 0x7f458014de40 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 >> 172.21.15.59:0/2563526327 conn(0x7f4580073e10 msgr2=0x7f458015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 shutdown_connections 2024-06-29T14:38:09.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.630+0000 7f4584a4e640 1 -- 172.21.15.59:0/2563526327 wait complete. 2024-06-29T14:38:09.640 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:09.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.761+0000 7f3cc0af6640 1 -- 172.21.15.59:0/2188448528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc162810 msgr2=0x7f3cbc164c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.761+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/2188448528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc162810 0x7f3cbc164c30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0cd760 tx=0x7f3cb402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.761+0000 7f3cc0af6640 1 -- 172.21.15.59:0/2188448528 shutdown_connections 2024-06-29T14:38:09.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.761+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/2188448528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc162810 0x7f3cbc164c30 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.761+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/2188448528 >> v2:172.21.15.153:3300/0 conn(0x7f3cbc0cd9b0 0x7f3cbc15bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.761+0000 7f3cc0af6640 1 -- 172.21.15.59:0/2188448528 >> 172.21.15.59:0/2188448528 conn(0x7f3cbc0b24c0 msgr2=0x7f3cbc0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.761+0000 7f3cc0af6640 1 -- 172.21.15.59:0/2188448528 shutdown_connections 2024-06-29T14:38:09.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.761+0000 7f3cc0af6640 1 -- 172.21.15.59:0/2188448528 wait complete. 2024-06-29T14:38:09.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.763+0000 7f3cc0af6640 1 Processor -- start 2024-06-29T14:38:09.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.763+0000 7f3cc0af6640 1 -- start start 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.764+0000 7f3cc0af6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc06d2f0 0x7f3cbc0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.764+0000 7f3cc0af6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cbc0cd9b0 0x7f3cbc1f8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.764+0000 7f3cc0af6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cbc162810 0x7f3cbc1f8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cc0af6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc1448f0 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cc0af6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc144770 con 0x7f3cbc162810 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cc0af6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3cbc165170 con 0x7f3cbc0cd9b0 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc06d2f0 0x7f3cbc0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc06d2f0 0x7f3cbc0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54118/0 (socket says 172.21.15.59:54118) 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 -- 172.21.15.59:0/3378108305 learned_addr learned my addr 172.21.15.59:0/3378108305 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cbad9d640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cbc162810 0x7f3cbc1f8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cb9d9b640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cbc0cd9b0 0x7f3cbc1f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 -- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cbc0cd9b0 msgr2=0x7f3cbc1f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cbc0cd9b0 0x7f3cbc1f8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 -- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cbc162810 msgr2=0x7f3cbc1f8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cbc162810 0x7f3cbc1f8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 -- 172.21.15.59:0/3378108305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cbc158b20 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cba59c640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc06d2f0 0x7f3cbc0b3640 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00d780 tx=0x7f3cac00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3caaffd640 1 -- 172.21.15.59:0/3378108305 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac017070 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3caaffd640 1 -- 172.21.15.59:0/3378108305 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cac00de90 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3caaffd640 1 -- 172.21.15.59:0/3378108305 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0099d0 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cbc158560 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.765+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cbc0ad950 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.767+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cbc15bc80 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.767+0000 7f3caaffd640 1 -- 172.21.15.59:0/3378108305 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f3cac009b70 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.768+0000 7f3caaffd640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f3c9c0763f0 0x7f3c9c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.768+0000 7f3cb9d9b640 1 -- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f3c9c0763f0 msgr2=0x7f3c9c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:09.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.768+0000 7f3caaffd640 1 -- 172.21.15.59:0/3378108305 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7f3cac097e60 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.768+0000 7f3cb9d9b640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f3c9c0763f0 0x7f3c9c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:09.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.773+0000 7f3caaffd640 1 -- 172.21.15.59:0/3378108305 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3cac020020 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.966+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3cbc13f770 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.967+0000 7f3caaffd640 1 -- 172.21.15.59:0/3378108305 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+17469 (secure 0 0 0) 0x7f3cac0692b0 con 0x7f3cbc06d2f0 2024-06-29T14:38:09.965 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:09.965 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":102,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:05.022034+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:51.120910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.698419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2366299610":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6833/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6832/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3522491343":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3511527873":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1670969120":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/4085525721":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1421481485":"2024-06-30T14:38:05.021970+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:09.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f3c9c0763f0 msgr2=0x7f3c9c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:09.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f3c9c0763f0 0x7f3c9c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc06d2f0 msgr2=0x7f3cbc0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc06d2f0 0x7f3cbc0b3640 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00d780 tx=0x7f3cac00dc40 comp rx=0 tx=0).stop 2024-06-29T14:38:09.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 shutdown_connections 2024-06-29T14:38:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f3c9c0763f0 0x7f3c9c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cbc162810 0x7f3cbc1f8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3cbc0cd9b0 0x7f3cbc1f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 --2- 172.21.15.59:0/3378108305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cbc06d2f0 0x7f3cbc0b3640 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 >> 172.21.15.59:0/3378108305 conn(0x7f3cbc0b24c0 msgr2=0x7f3cbc06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 shutdown_connections 2024-06-29T14:38:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:09.969+0000 7f3cc0af6640 1 -- 172.21.15.59:0/3378108305 wait complete. 2024-06-29T14:38:09.979 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:37:51.120910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:37:51.698419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:38:09.979 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:38:10.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.105+0000 7efdb92e6640 1 -- 172.21.15.59:0/3271983574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdb415efe0 msgr2=0x7efdb415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.105+0000 7efdb92e6640 1 --2- 172.21.15.59:0/3271983574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdb415efe0 0x7efdb415f9b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efdac009be0 tx=0x7efdac02a920 comp rx=0 tx=0).stop 2024-06-29T14:38:10.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.105+0000 7efdb92e6640 1 -- 172.21.15.59:0/3271983574 shutdown_connections 2024-06-29T14:38:10.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.105+0000 7efdb92e6640 1 --2- 172.21.15.59:0/3271983574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdb415efe0 0x7efdb415f9b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.105+0000 7efdb92e6640 1 -- 172.21.15.59:0/3271983574 >> 172.21.15.59:0/3271983574 conn(0x7efdb40d3ee0 msgr2=0x7efdb40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.105+0000 7efdb92e6640 1 -- 172.21.15.59:0/3271983574 shutdown_connections 2024-06-29T14:38:10.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.105+0000 7efdb92e6640 1 -- 172.21.15.59:0/3271983574 wait complete. 2024-06-29T14:38:10.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.107+0000 7efdb92e6640 1 Processor -- start 2024-06-29T14:38:10.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.107+0000 7efdb92e6640 1 -- start start 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb92e6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb40d23c0 0x7efdb4140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb92e6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdb415efe0 0x7efdb415a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb92e6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb415ac60 0x7efdb4153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb92e6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efdb4160470 con 0x7efdb40d23c0 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb92e6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efdb41602f0 con 0x7efdb415ac60 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb92e6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efdb41605f0 con 0x7efdb415efe0 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb40d23c0 0x7efdb4140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb40d23c0 0x7efdb4140d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54130/0 (socket says 172.21.15.59:54130) 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdabfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdb415efe0 0x7efdb415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb2ffd640 1 -- 172.21.15.59:0/1259429157 learned_addr learned my addr 172.21.15.59:0/1259429157 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:10.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.108+0000 7efdb37fe640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb415ac60 0x7efdb4153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdb2ffd640 1 -- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdb415efe0 msgr2=0x7efdb415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdb2ffd640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdb415efe0 0x7efdb415a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdb2ffd640 1 -- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb415ac60 msgr2=0x7efdb4153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdb2ffd640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb415ac60 0x7efdb4153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdb2ffd640 1 -- 172.21.15.59:0/1259429157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb4141250 con 0x7efdb40d23c0 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdb2ffd640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb40d23c0 0x7efdb4140d10 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7efda400c3c0 tx=0x7efda4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdab7fe640 1 -- 172.21.15.59:0/1259429157 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4017070 con 0x7efdb40d23c0 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdab7fe640 1 -- 172.21.15.59:0/1259429157 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efda40053e0 con 0x7efdb40d23c0 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb4159100 con 0x7efdb40d23c0 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdab7fe640 1 -- 172.21.15.59:0/1259429157 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4003850 con 0x7efdb40d23c0 2024-06-29T14:38:10.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.109+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb41596c0 con 0x7efdb40d23c0 2024-06-29T14:38:10.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.110+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb4158e50 con 0x7efdb40d23c0 2024-06-29T14:38:10.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.111+0000 7efdab7fe640 1 -- 172.21.15.59:0/1259429157 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7efda401a070 con 0x7efdb40d23c0 2024-06-29T14:38:10.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.112+0000 7efdab7fe640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7efd940763f0 0x7efd94078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.112+0000 7efdab7fe640 1 -- 172.21.15.59:0/1259429157 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7efda4097690 con 0x7efdb40d23c0 2024-06-29T14:38:10.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.112+0000 7efdabfff640 1 -- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7efd940763f0 msgr2=0x7efd94078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:10.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.112+0000 7efdabfff640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7efd940763f0 0x7efd94078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:10.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.115+0000 7efdab7fe640 1 -- 172.21.15.59:0/1259429157 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efda4021070 con 0x7efdb40d23c0 2024-06-29T14:38:10.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.307+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efdb415f9b0 con 0x7efdb40d23c0 2024-06-29T14:38:10.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.308+0000 7efdab7fe640 1 -- 172.21.15.59:0/1259429157 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7efda40606c0 con 0x7efdb40d23c0 2024-06-29T14:38:10.306 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:38:10.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7efd940763f0 msgr2=0x7efd94078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:10.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7efd940763f0 0x7efd94078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb40d23c0 msgr2=0x7efdb4140d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb40d23c0 0x7efdb4140d10 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7efda400c3c0 tx=0x7efda4001e70 comp rx=0 tx=0).stop 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 shutdown_connections 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7efd940763f0 0x7efd94078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb415ac60 0x7efdb4153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdb415efe0 0x7efdb415a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 --2- 172.21.15.59:0/1259429157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb40d23c0 0x7efdb4140d10 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 >> 172.21.15.59:0/1259429157 conn(0x7efdb40d3ee0 msgr2=0x7efdb406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 shutdown_connections 2024-06-29T14:38:10.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.310+0000 7efdb92e6640 1 -- 172.21.15.59:0/1259429157 wait complete. 2024-06-29T14:38:10.319 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:38:10.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.442+0000 7f0b891fc640 1 -- 172.21.15.59:0/3130586975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b84143840 msgr2=0x7f0b8406e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.442+0000 7f0b891fc640 1 --2- 172.21.15.59:0/3130586975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b84143840 0x7f0b8406e4d0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0b84144290 tx=0x7f0b7802a560 comp rx=0 tx=0).stop 2024-06-29T14:38:10.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.442+0000 7f0b891fc640 1 -- 172.21.15.59:0/3130586975 shutdown_connections 2024-06-29T14:38:10.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.442+0000 7f0b891fc640 1 --2- 172.21.15.59:0/3130586975 >> v2:172.21.15.153:3300/0 conn(0x7f0b8406ea10 0x7f0b840cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.442+0000 7f0b891fc640 1 --2- 172.21.15.59:0/3130586975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b84143840 0x7f0b8406e4d0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.442+0000 7f0b891fc640 1 -- 172.21.15.59:0/3130586975 >> 172.21.15.59:0/3130586975 conn(0x7f0b840b2ab0 msgr2=0x7f0b840b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.442+0000 7f0b891fc640 1 -- 172.21.15.59:0/3130586975 shutdown_connections 2024-06-29T14:38:10.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.442+0000 7f0b891fc640 1 -- 172.21.15.59:0/3130586975 wait complete. 2024-06-29T14:38:10.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.446+0000 7f0b891fc640 1 Processor -- start 2024-06-29T14:38:10.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.446+0000 7f0b891fc640 1 -- start start 2024-06-29T14:38:10.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b891fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b8406ea10 0x7f0b8413dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b891fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b840d0130 0x7f0b8415bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b891fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b84143840 0x7f0b8413d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b891fc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0b840cf650 con 0x7f0b8406ea10 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b891fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b840cf4d0 con 0x7f0b84143840 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b891fc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0b840cf7d0 con 0x7f0b840d0130 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b82d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b8406ea10 0x7f0b8413dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b82d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b8406ea10 0x7f0b8413dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54146/0 (socket says 172.21.15.59:54146) 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b82d9d640 1 -- 172.21.15.59:0/2486273060 learned_addr learned my addr 172.21.15.59:0/2486273060 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b8359e640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b84143840 0x7f0b8413d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.448+0000 7f0b8259c640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b840d0130 0x7f0b8415bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b82d9d640 1 -- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b840d0130 msgr2=0x7f0b8415bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b82d9d640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b840d0130 0x7f0b8415bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b82d9d640 1 -- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b84143840 msgr2=0x7f0b8413d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b82d9d640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b84143840 0x7f0b8413d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b82d9d640 1 -- 172.21.15.59:0/2486273060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8413e5f0 con 0x7f0b8406ea10 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b8359e640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b84143840 0x7f0b8413d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b82d9d640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b8406ea10 0x7f0b8413dda0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400c3d0 tx=0x7f0b7400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b6b7fe640 1 -- 172.21.15.59:0/2486273060 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b74017070 con 0x7f0b8406ea10 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b6b7fe640 1 -- 172.21.15.59:0/2486273060 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b740053c0 con 0x7f0b8406ea10 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b841552d0 con 0x7f0b8406ea10 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b6b7fe640 1 -- 172.21.15.59:0/2486273060 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b74003850 con 0x7f0b8406ea10 2024-06-29T14:38:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.449+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b84155890 con 0x7f0b8406ea10 2024-06-29T14:38:10.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.450+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b840cfbf0 con 0x7f0b8406ea10 2024-06-29T14:38:10.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.451+0000 7f0b6b7fe640 1 -- 172.21.15.59:0/2486273060 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f0b7401a070 con 0x7f0b8406ea10 2024-06-29T14:38:10.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.451+0000 7f0b6b7fe640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0b70076420 0x7f0b70078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.452+0000 7f0b8259c640 1 -- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0b70076420 msgr2=0x7f0b70078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3952610634 2024-06-29T14:38:10.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.452+0000 7f0b6b7fe640 1 -- 172.21.15.59:0/2486273060 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6445+0+0 (secure 0 0 0) 0x7f0b74096f80 con 0x7f0b8406ea10 2024-06-29T14:38:10.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.452+0000 7f0b8259c640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0b70076420 0x7f0b70078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:10.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.456+0000 7f0b6b7fe640 1 -- 172.21.15.59:0/2486273060 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b7405fe10 con 0x7f0b8406ea10 2024-06-29T14:38:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.647+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0b8406e6e0 con 0x7f0b8406ea10 2024-06-29T14:38:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.647+0000 7f0b6b7fe640 1 -- 172.21.15.59:0/2486273060 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7f0b74064110 con 0x7f0b8406ea10 2024-06-29T14:38:10.646 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:38:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.649+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0b70076420 msgr2=0x7f0b70078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0b70076420 0x7f0b70078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b8406ea10 msgr2=0x7f0b8413dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b8406ea10 0x7f0b8413dda0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400c3d0 tx=0x7f0b7400c890 comp rx=0 tx=0).stop 2024-06-29T14:38:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 shutdown_connections 2024-06-29T14:38:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:6832/3952610634,v1:172.21.15.153:6833/3952610634] conn(0x7f0b70076420 0x7f0b70078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b84143840 0x7f0b8413d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b840d0130 0x7f0b8415bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 --2- 172.21.15.59:0/2486273060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b8406ea10 0x7f0b8413dda0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 >> 172.21.15.59:0/2486273060 conn(0x7f0b840b2ab0 msgr2=0x7f0b840b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 shutdown_connections 2024-06-29T14:38:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.650+0000 7f0b891fc640 1 -- 172.21.15.59:0/2486273060 wait complete. 2024-06-29T14:38:10.659 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:38:10.776 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3223f5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.777 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3223f5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.777 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3223f5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.777 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3223f5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.777 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3223f5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.777 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3220f55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.777 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3220f55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.778 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3220f55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.778 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3220f55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.778 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:10.779+0000 7f3220f55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.788+0000 7fafef8df640 1 -- 172.21.15.59:0/3494996452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe806ca70 msgr2=0x7fafe806ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.788+0000 7fafef8df640 1 --2- 172.21.15.59:0/3494996452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe806ca70 0x7fafe806ce80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fafe0005fd0 tx=0x7fafe002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.788+0000 7fafef8df640 1 -- 172.21.15.59:0/3494996452 shutdown_connections 2024-06-29T14:38:10.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.788+0000 7fafef8df640 1 --2- 172.21.15.59:0/3494996452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe806ca70 0x7fafe806ce80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.788+0000 7fafef8df640 1 --2- 172.21.15.59:0/3494996452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe80cf860 0x7fafe80cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.788+0000 7fafef8df640 1 -- 172.21.15.59:0/3494996452 >> 172.21.15.59:0/3494996452 conn(0x7fafe80b30c0 msgr2=0x7fafe80cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.788+0000 7fafef8df640 1 -- 172.21.15.59:0/3494996452 shutdown_connections 2024-06-29T14:38:10.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.788+0000 7fafef8df640 1 -- 172.21.15.59:0/3494996452 wait complete. 2024-06-29T14:38:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.790+0000 7fafef8df640 1 Processor -- start 2024-06-29T14:38:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.790+0000 7fafef8df640 1 -- start start 2024-06-29T14:38:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.791+0000 7fafef8df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe806ca70 0x7fafe81f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.791+0000 7fafef8df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80cf860 0x7fafe813bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.791+0000 7fafef8df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe81423b0 0x7fafe81f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.791+0000 7fafef8df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fafe806d540 con 0x7fafe80cf860 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.791+0000 7fafef8df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafe806d3c0 con 0x7fafe806ca70 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.791+0000 7fafef8df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fafe806d6c0 con 0x7fafe81423b0 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.791+0000 7fafece7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80cf860 0x7fafe813bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.791+0000 7fafece7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80cf860 0x7fafe813bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54152/0 (socket says 172.21.15.59:54152) 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafece7a640 1 -- 172.21.15.59:0/1367804225 learned_addr learned my addr 172.21.15.59:0/1367804225 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafede7c640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe81423b0 0x7fafe81f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafed67b640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe806ca70 0x7fafe81f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafede7c640 1 -- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe806ca70 msgr2=0x7fafe81f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafede7c640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe806ca70 0x7fafe81f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafede7c640 1 -- 172.21.15.59:0/1367804225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80cf860 msgr2=0x7fafe813bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafede7c640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80cf860 0x7fafe813bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafede7c640 1 -- 172.21.15.59:0/1367804225 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe813c650 con 0x7fafe81423b0 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafede7c640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe81423b0 0x7fafe81f6aa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fafe80ce720 tx=0x7fafe00316e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafddffb640 1 -- 172.21.15.59:0/1367804225 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0031990 con 0x7fafe81423b0 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafddffb640 1 -- 172.21.15.59:0/1367804225 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafe0031c40 con 0x7fafe81423b0 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafddffb640 1 -- 172.21.15.59:0/1367804225 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0033330 con 0x7fafe81423b0 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.792+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe813c930 con 0x7fafe81423b0 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.793+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe813cef0 con 0x7fafe81423b0 2024-06-29T14:38:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.793+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe806fce0 con 0x7fafe81423b0 2024-06-29T14:38:10.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.794+0000 7fafddffb640 1 -- 172.21.15.59:0/1367804225 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 43) ==== 97912+0+0 (secure 0 0 0) 0x7fafe00334d0 con 0x7fafe81423b0 2024-06-29T14:38:10.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.795+0000 7fafddffb640 1 -- 172.21.15.59:0/1367804225 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(103..103 src has 1..103) ==== 6789+0+0 (secure 0 0 0) 0x7fafe00b8c10 con 0x7fafe81423b0 2024-06-29T14:38:10.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.798+0000 7fafddffb640 1 -- 172.21.15.59:0/1367804225 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fafe806fce0 con 0x7fafe81423b0 2024-06-29T14:38:10.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.992+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fafe806dbe0 con 0x7fafe81423b0 2024-06-29T14:38:10.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.992+0000 7fafddffb640 1 -- 172.21.15.59:0/1367804225 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7fafe003f030 con 0x7fafe81423b0 2024-06-29T14:38:10.991 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.994+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe81423b0 msgr2=0x7fafe81f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.994+0000 7fafef8df640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe81423b0 0x7fafe81f6aa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fafe80ce720 tx=0x7fafe00316e0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.995+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 shutdown_connections 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.995+0000 7fafef8df640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafe81423b0 0x7fafe81f6aa0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.995+0000 7fafef8df640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafe80cf860 0x7fafe813bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.995+0000 7fafef8df640 1 --2- 172.21.15.59:0/1367804225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe806ca70 0x7fafe81f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.995+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 >> 172.21.15.59:0/1367804225 conn(0x7fafe80b30c0 msgr2=0x7fafe8145210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.995+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 shutdown_connections 2024-06-29T14:38:10.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:10.995+0000 7fafef8df640 1 -- 172.21.15.59:0/1367804225 wait complete. 2024-06-29T14:38:11.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:11.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.126+0000 7f6c1c08f640 1 -- 172.21.15.59:0/4120471668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 msgr2=0x7f6c140b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.126+0000 7f6c1c08f640 1 --2- 172.21.15.59:0/4120471668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 0x7f6c140b1d10 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009830 tx=0x7f6c0c02a510 comp rx=0 tx=0).stop 2024-06-29T14:38:11.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.126+0000 7f6c1c08f640 1 -- 172.21.15.59:0/4120471668 shutdown_connections 2024-06-29T14:38:11.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.126+0000 7f6c1c08f640 1 --2- 172.21.15.59:0/4120471668 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14161b00 0x7f6c14161f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.126+0000 7f6c1c08f640 1 --2- 172.21.15.59:0/4120471668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 0x7f6c140b1d10 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.126+0000 7f6c1c08f640 1 -- 172.21.15.59:0/4120471668 >> 172.21.15.59:0/4120471668 conn(0x7f6c1406c950 msgr2=0x7f6c14073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.126+0000 7f6c1c08f640 1 -- 172.21.15.59:0/4120471668 shutdown_connections 2024-06-29T14:38:11.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.126+0000 7f6c1c08f640 1 -- 172.21.15.59:0/4120471668 wait complete. 2024-06-29T14:38:11.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.127+0000 7f6c1c08f640 1 Processor -- start 2024-06-29T14:38:11.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.128+0000 7f6c1c08f640 1 -- start start 2024-06-29T14:38:11.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1c08f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c1406f8a0 0x7f6c141f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1c08f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 0x7f6c140cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1c08f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14161b00 0x7f6c141e8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1c08f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1415ea90 con 0x7f6c14147370 2024-06-29T14:38:11.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1c08f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1415e910 con 0x7f6c1406f8a0 2024-06-29T14:38:11.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1c08f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1415ec10 con 0x7f6c14161b00 2024-06-29T14:38:11.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 0x7f6c140cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 0x7f6c140cd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54170/0 (socket says 172.21.15.59:54170) 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 -- 172.21.15.59:0/1537966532 learned_addr learned my addr 172.21.15.59:0/1537966532 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1a62c640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14161b00 0x7f6c141e8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c19e2b640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c1406f8a0 0x7f6c141f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 -- 172.21.15.59:0/1537966532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14161b00 msgr2=0x7f6c141e8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14161b00 0x7f6c141e8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 -- 172.21.15.59:0/1537966532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c1406f8a0 msgr2=0x7f6c141f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c1406f8a0 0x7f6c141f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 -- 172.21.15.59:0/1537966532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c140cdf50 con 0x7f6c14147370 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.129+0000 7f6c1962a640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 0x7f6c140cd540 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c3a0 tx=0x7f6c0800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.130+0000 7f6c127fc640 1 -- 172.21.15.59:0/1537966532 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08011790 con 0x7f6c14147370 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.130+0000 7f6c127fc640 1 -- 172.21.15.59:0/1537966532 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c08017070 con 0x7f6c14147370 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.130+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c140ce230 con 0x7f6c14147370 2024-06-29T14:38:11.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.130+0000 7f6c127fc640 1 -- 172.21.15.59:0/1537966532 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08011cc0 con 0x7f6c14147370 2024-06-29T14:38:11.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.130+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c140ce7f0 con 0x7f6c14147370 2024-06-29T14:38:11.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.131+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c1415f1a0 con 0x7f6c14147370 2024-06-29T14:38:11.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.132+0000 7f6c127fc640 1 -- 172.21.15.59:0/1537966532 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 43) ==== 97912+0+0 (secure 0 0 0) 0x7f6c0801c460 con 0x7f6c14147370 2024-06-29T14:38:11.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.132+0000 7f6c127fc640 1 -- 172.21.15.59:0/1537966532 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6789+0+0 (secure 0 0 0) 0x7f6c080971d0 con 0x7f6c14147370 2024-06-29T14:38:11.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.137+0000 7f6c127fc640 1 -- 172.21.15.59:0/1537966532 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6c14162450 con 0x7f6c14147370 2024-06-29T14:38:11.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.364+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c14146bb0 con 0x7f6c14147370 2024-06-29T14:38:11.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.365+0000 7f6c127fc640 1 -- 172.21.15.59:0/1537966532 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5228 (secure 0 0 0) 0x7f6c080604f0 con 0x7f6c14147370 2024-06-29T14:38:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:11.363 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":46,"btime":"2024-06-29T14:37:56:391508+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2174310652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2174310652},{"type":"v1","addr":"172.21.15.59:6835","nonce":2174310652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3217643445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3217643445},{"type":"v1","addr":"172.21.15.153:6836","nonce":3217643445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/832666341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":832666341},{"type":"v1","addr":"172.21.15.59:6833","nonce":832666341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.753189+0000","modified":"2024-06-29T14:37:55.781006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7365},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7365":{"gid":7365,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/1942687234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1942687234},{"type":"v1","addr":"172.21.15.153:6837","nonce":1942687234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7365,"qdb_cluster":[7365]},"id":5}]} 2024-06-29T14:38:11.363 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 46 2024-06-29T14:38:11.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 msgr2=0x7f6c140cd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 0x7f6c140cd540 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c3a0 tx=0x7f6c0800c860 comp rx=0 tx=0).stop 2024-06-29T14:38:11.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 shutdown_connections 2024-06-29T14:38:11.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c14161b00 0x7f6c141e8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14147370 0x7f6c140cd540 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 --2- 172.21.15.59:0/1537966532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c1406f8a0 0x7f6c141f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 >> 172.21.15.59:0/1537966532 conn(0x7f6c1406c950 msgr2=0x7f6c140cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 shutdown_connections 2024-06-29T14:38:11.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.367+0000 7f6c1c08f640 1 -- 172.21.15.59:0/1537966532 wait complete. 2024-06-29T14:38:11.376 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:11.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.496+0000 7f8527735640 1 -- 172.21.15.59:0/3197577116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85201458a0 msgr2=0x7f85200cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.496+0000 7f8527735640 1 --2- 172.21.15.59:0/3197577116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85201458a0 0x7f85200cf870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f851c005fd0 tx=0x7f851c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.496+0000 7f8527735640 1 -- 172.21.15.59:0/3197577116 shutdown_connections 2024-06-29T14:38:11.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.496+0000 7f8527735640 1 --2- 172.21.15.59:0/3197577116 >> v2:172.21.15.153:3300/0 conn(0x7f85200b3740 0x7f85200b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.496+0000 7f8527735640 1 --2- 172.21.15.59:0/3197577116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85201458a0 0x7f85200cf870 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.496+0000 7f8527735640 1 -- 172.21.15.59:0/3197577116 >> 172.21.15.59:0/3197577116 conn(0x7f852006c950 msgr2=0x7f8520073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.496+0000 7f8527735640 1 -- 172.21.15.59:0/3197577116 shutdown_connections 2024-06-29T14:38:11.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.496+0000 7f8527735640 1 -- 172.21.15.59:0/3197577116 wait complete. 2024-06-29T14:38:11.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.498+0000 7f8527735640 1 Processor -- start 2024-06-29T14:38:11.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.498+0000 7f8527735640 1 -- start start 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f8527735640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85200b3740 0x7f85200ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f8527735640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f852015c2f0 0x7f85200ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f8527735640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85201f8820 0x7f85201f8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f8527735640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f85201601a0 con 0x7f85200b3740 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f8527735640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8520160020 con 0x7f85201f8820 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f8527735640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8520160320 con 0x7f852015c2f0 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f85254d1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85200b3740 0x7f85200ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f85254d1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85200b3740 0x7f85200ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54180/0 (socket says 172.21.15.59:54180) 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f85254d1640 1 -- 172.21.15.59:0/4104660032 learned_addr learned my addr 172.21.15.59:0/4104660032 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.499+0000 7f8524cd0640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f852015c2f0 0x7f85200ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f8525cd2640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85201f8820 0x7f85201f8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f85254d1640 1 -- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f852015c2f0 msgr2=0x7f85200ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f85254d1640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f852015c2f0 0x7f85200ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f85254d1640 1 -- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85201f8820 msgr2=0x7f85201f8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f85254d1640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85201f8820 0x7f85201f8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f85254d1640 1 -- 172.21.15.59:0/4104660032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85200ce770 con 0x7f85200b3740 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f8525cd2640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85201f8820 0x7f85201f8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f85254d1640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85200b3740 0x7f85200ccb90 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f851c000c00 tx=0x7f851c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f8515ffb640 1 -- 172.21.15.59:0/4104660032 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c03e030 con 0x7f85200b3740 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85200cbaf0 con 0x7f85200b3740 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f8515ffb640 1 -- 172.21.15.59:0/4104660032 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f851c003ae0 con 0x7f85200b3740 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f8515ffb640 1 -- 172.21.15.59:0/4104660032 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c035040 con 0x7f85200b3740 2024-06-29T14:38:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.500+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85200cc0b0 con 0x7f85200b3740 2024-06-29T14:38:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.501+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85201458a0 con 0x7f85200b3740 2024-06-29T14:38:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.502+0000 7f8515ffb640 1 -- 172.21.15.59:0/4104660032 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 43) ==== 97912+0+0 (secure 0 0 0) 0x7f851c041030 con 0x7f85200b3740 2024-06-29T14:38:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.503+0000 7f8515ffb640 1 -- 172.21.15.59:0/4104660032 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6789+0+0 (secure 0 0 0) 0x7f851c0b7a70 con 0x7f85200b3740 2024-06-29T14:38:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.507+0000 7f8515ffb640 1 -- 172.21.15.59:0/4104660032 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f851c080c80 con 0x7f85200b3740 2024-06-29T14:38:11.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.700+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f852015be10 con 0x7f85200b3740 2024-06-29T14:38:11.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.701+0000 7f8515ffb640 1 -- 172.21.15.59:0/4104660032 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+17979 (secure 0 0 0) 0x7f851c084f80 con 0x7f85200b3740 2024-06-29T14:38:11.699 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:11.699 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":103,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:10.719590+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:51.120910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.698419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1412183002":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2783149049":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2472287695":"2024-06-30T14:38:10.719553+0000","172.21.15.153:6833/3952610634":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2216119053":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/3289012264":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2366299610":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6833/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6832/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3522491343":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6832/3952610634":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/3511527873":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1670969120":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3512251954":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/4085525721":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1421481485":"2024-06-30T14:38:05.021970+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:11.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85200b3740 msgr2=0x7f85200ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85200b3740 0x7f85200ccb90 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f851c000c00 tx=0x7f851c0035f0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 shutdown_connections 2024-06-29T14:38:11.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85201f8820 0x7f85201f8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f852015c2f0 0x7f85200ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 --2- 172.21.15.59:0/4104660032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85200b3740 0x7f85200ccb90 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 >> 172.21.15.59:0/4104660032 conn(0x7f852006c950 msgr2=0x7f8520070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 shutdown_connections 2024-06-29T14:38:11.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.703+0000 7f8527735640 1 -- 172.21.15.59:0/4104660032 wait complete. 2024-06-29T14:38:11.713 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:38:11.713 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:11.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.835+0000 7efe4ca40640 1 -- 172.21.15.59:0/590419305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 msgr2=0x7efe4815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.835+0000 7efe4ca40640 1 --2- 172.21.15.59:0/590419305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 0x7efe4815f9b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7efe40005fd0 tx=0x7efe4002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.835+0000 7efe4ca40640 1 -- 172.21.15.59:0/590419305 shutdown_connections 2024-06-29T14:38:11.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.835+0000 7efe4ca40640 1 --2- 172.21.15.59:0/590419305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 0x7efe4815f9b0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.835+0000 7efe4ca40640 1 --2- 172.21.15.59:0/590419305 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe480b40a0 0x7efe4815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.835+0000 7efe4ca40640 1 -- 172.21.15.59:0/590419305 >> 172.21.15.59:0/590419305 conn(0x7efe480d3ee0 msgr2=0x7efe480d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.835+0000 7efe4ca40640 1 -- 172.21.15.59:0/590419305 shutdown_connections 2024-06-29T14:38:11.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.835+0000 7efe4ca40640 1 -- 172.21.15.59:0/590419305 wait complete. 2024-06-29T14:38:11.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.837+0000 7efe4ca40640 1 Processor -- start 2024-06-29T14:38:11.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.837+0000 7efe4ca40640 1 -- start start 2024-06-29T14:38:11.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe4ca40640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe480d23c0 0x7efe48158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe4ca40640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 0x7efe481f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe4ca40640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48143ac0 0x7efe48156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe4ca40640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efe48160280 con 0x7efe4815ee10 2024-06-29T14:38:11.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe4ca40640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe48160100 con 0x7efe48143ac0 2024-06-29T14:38:11.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe4ca40640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efe48160400 con 0x7efe480d23c0 2024-06-29T14:38:11.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 0x7efe481f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 0x7efe481f4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54208/0 (socket says 172.21.15.59:54208) 2024-06-29T14:38:11.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 -- 172.21.15.59:0/103410776 learned_addr learned my addr 172.21.15.59:0/103410776 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:11.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe4659c640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe480d23c0 0x7efe48158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe46d9d640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48143ac0 0x7efe48156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 -- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe480d23c0 msgr2=0x7efe48158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe480d23c0 0x7efe48158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 -- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48143ac0 msgr2=0x7efe48156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48143ac0 0x7efe48156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 -- 172.21.15.59:0/103410776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe48141250 con 0x7efe4815ee10 2024-06-29T14:38:11.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.839+0000 7efe45d9b640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 0x7efe481f4c30 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7efe3000d770 tx=0x7efe3000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.840+0000 7efe3effd640 1 -- 172.21.15.59:0/103410776 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30017070 con 0x7efe4815ee10 2024-06-29T14:38:11.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.840+0000 7efe3effd640 1 -- 172.21.15.59:0/103410776 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe30009d20 con 0x7efe4815ee10 2024-06-29T14:38:11.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.840+0000 7efe3effd640 1 -- 172.21.15.59:0/103410776 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe300106c0 con 0x7efe4815ee10 2024-06-29T14:38:11.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.840+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe48141530 con 0x7efe4815ee10 2024-06-29T14:38:11.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.840+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe48143ed0 con 0x7efe4815ee10 2024-06-29T14:38:11.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.841+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe4815e8d0 con 0x7efe4815ee10 2024-06-29T14:38:11.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.842+0000 7efe3effd640 1 -- 172.21.15.59:0/103410776 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 44) ==== 98039+0+0 (secure 0 0 0) 0x7efe3001a070 con 0x7efe4815ee10 2024-06-29T14:38:11.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.842+0000 7efe3effd640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efe28076150 0x7efe28078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.842+0000 7efe3effd640 1 -- 172.21.15.59:0/103410776 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6789+0+0 (secure 0 0 0) 0x7efe30097760 con 0x7efe4815ee10 2024-06-29T14:38:11.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.842+0000 7efe4659c640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efe28076150 0x7efe28078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.843+0000 7efe4659c640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efe28076150 0x7efe28078660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efe480d2e40 tx=0x7efe38009040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:11.846+0000 7efe3effd640 1 -- 172.21.15.59:0/103410776 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efe30060970 con 0x7efe4815ee10 2024-06-29T14:38:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.075+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe48159410 con 0x7efe4815ee10 2024-06-29T14:38:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.075+0000 7efe3effd640 1 -- 172.21.15.59:0/103410776 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5228 (secure 0 0 0) 0x7efe30064c70 con 0x7efe4815ee10 2024-06-29T14:38:12.074 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:12.074 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":46,"btime":"2024-06-29T14:37:56:391508+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2174310652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2174310652},{"type":"v1","addr":"172.21.15.59:6835","nonce":2174310652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3217643445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3217643445},{"type":"v1","addr":"172.21.15.153:6836","nonce":3217643445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/832666341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":832666341},{"type":"v1","addr":"172.21.15.59:6833","nonce":832666341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.753189+0000","modified":"2024-06-29T14:37:55.781006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7365},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7365":{"gid":7365,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/1942687234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1942687234},{"type":"v1","addr":"172.21.15.153:6837","nonce":1942687234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7365,"qdb_cluster":[7365]},"id":5}]} 2024-06-29T14:38:12.074 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 46 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efe28076150 msgr2=0x7efe28078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efe28076150 0x7efe28078660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efe480d2e40 tx=0x7efe38009040 comp rx=0 tx=0).stop 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 msgr2=0x7efe481f4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 0x7efe481f4c30 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7efe3000d770 tx=0x7efe3000dc30 comp rx=0 tx=0).stop 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 shutdown_connections 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efe28076150 0x7efe28078660 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe48143ac0 0x7efe48156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe4815ee10 0x7efe481f4c30 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 --2- 172.21.15.59:0/103410776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe480d23c0 0x7efe48158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 >> 172.21.15.59:0/103410776 conn(0x7efe480d3ee0 msgr2=0x7efe481608d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 shutdown_connections 2024-06-29T14:38:12.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.077+0000 7efe4ca40640 1 -- 172.21.15.59:0/103410776 wait complete. 2024-06-29T14:38:12.087 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.206+0000 7fac5bd03640 1 -- 172.21.15.59:0/1040722925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac541458a0 msgr2=0x7fac540cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.206+0000 7fac5bd03640 1 --2- 172.21.15.59:0/1040722925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac541458a0 0x7fac540cf870 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fac540b1c40 tx=0x7fac4c02a530 comp rx=0 tx=0).stop 2024-06-29T14:38:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.206+0000 7fac5bd03640 1 -- 172.21.15.59:0/1040722925 shutdown_connections 2024-06-29T14:38:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.206+0000 7fac5bd03640 1 --2- 172.21.15.59:0/1040722925 >> v2:172.21.15.153:3300/0 conn(0x7fac540b3740 0x7fac540b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.206+0000 7fac5bd03640 1 --2- 172.21.15.59:0/1040722925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac541458a0 0x7fac540cf870 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.206+0000 7fac5bd03640 1 -- 172.21.15.59:0/1040722925 >> 172.21.15.59:0/1040722925 conn(0x7fac5406c950 msgr2=0x7fac54073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.206+0000 7fac5bd03640 1 -- 172.21.15.59:0/1040722925 shutdown_connections 2024-06-29T14:38:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.206+0000 7fac5bd03640 1 -- 172.21.15.59:0/1040722925 wait complete. 2024-06-29T14:38:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.208+0000 7fac5bd03640 1 Processor -- start 2024-06-29T14:38:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.208+0000 7fac5bd03640 1 -- start start 2024-06-29T14:38:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.209+0000 7fac5bd03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac540b3740 0x7fac5415ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.209+0000 7fac5bd03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac5415c2f0 0x7fac5413f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.209+0000 7fac5bd03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac54138a10 0x7fac5413fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.209+0000 7fac5bd03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fac54160020 con 0x7fac5415c2f0 2024-06-29T14:38:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.209+0000 7fac5bd03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac5415cbb0 con 0x7fac54138a10 2024-06-29T14:38:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.209+0000 7fac5bd03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fac541601a0 con 0x7fac540b3740 2024-06-29T14:38:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac5415c2f0 0x7fac5413f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac5415c2f0 0x7fac5413f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54222/0 (socket says 172.21.15.59:54222) 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 -- 172.21.15.59:0/654503802 learned_addr learned my addr 172.21.15.59:0/654503802 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac59a9f640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac540b3740 0x7fac5415ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5a2a0640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac54138a10 0x7fac5413fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 -- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac540b3740 msgr2=0x7fac5415ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac540b3740 0x7fac5415ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 -- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac54138a10 msgr2=0x7fac5413fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac54138a10 0x7fac5413fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 -- 172.21.15.59:0/654503802 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5413ea30 con 0x7fac5415c2f0 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5929e640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac5415c2f0 0x7fac5413f7f0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fac4800d770 tx=0x7fac4800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac527fc640 1 -- 172.21.15.59:0/654503802 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48017070 con 0x7fac5415c2f0 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac527fc640 1 -- 172.21.15.59:0/654503802 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac48009d20 con 0x7fac5415c2f0 2024-06-29T14:38:12.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5413ed10 con 0x7fac5415c2f0 2024-06-29T14:38:12.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac527fc640 1 -- 172.21.15.59:0/654503802 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac480106c0 con 0x7fac5415c2f0 2024-06-29T14:38:12.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.210+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5413f2d0 con 0x7fac5415c2f0 2024-06-29T14:38:12.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.211+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5406f990 con 0x7fac5415c2f0 2024-06-29T14:38:12.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.212+0000 7fac527fc640 1 -- 172.21.15.59:0/654503802 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 44) ==== 98039+0+0 (secure 0 0 0) 0x7fac4801a070 con 0x7fac5415c2f0 2024-06-29T14:38:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.213+0000 7fac527fc640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fac40076120 0x7fac40078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.213+0000 7fac527fc640 1 -- 172.21.15.59:0/654503802 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6789+0+0 (secure 0 0 0) 0x7fac480978a0 con 0x7fac5415c2f0 2024-06-29T14:38:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.213+0000 7fac59a9f640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fac40076120 0x7fac40078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.213+0000 7fac59a9f640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fac40076120 0x7fac40078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fac4c034000 tx=0x7fac4c004730 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.217+0000 7fac527fc640 1 -- 172.21.15.59:0/654503802 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fac48061080 con 0x7fac5415c2f0 2024-06-29T14:38:12.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.446+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac541606c0 con 0x7fac5415c2f0 2024-06-29T14:38:12.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.447+0000 7fac527fc640 1 -- 172.21.15.59:0/654503802 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5228 (secure 0 0 0) 0x7fac48060980 con 0x7fac5415c2f0 2024-06-29T14:38:12.445 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:12.445 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":46,"btime":"2024-06-29T14:37:56:391508+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2174310652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2174310652},{"type":"v1","addr":"172.21.15.59:6835","nonce":2174310652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3217643445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3217643445},{"type":"v1","addr":"172.21.15.153:6836","nonce":3217643445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/832666341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":832666341},{"type":"v1","addr":"172.21.15.59:6833","nonce":832666341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.753189+0000","modified":"2024-06-29T14:37:55.781006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7365},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7365":{"gid":7365,"name":"b","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/1942687234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1942687234},{"type":"v1","addr":"172.21.15.153:6837","nonce":1942687234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7365,"qdb_cluster":[7365]},"id":5}]} 2024-06-29T14:38:12.445 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 46 2024-06-29T14:38:12.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fac40076120 msgr2=0x7fac40078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fac40076120 0x7fac40078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fac4c034000 tx=0x7fac4c004730 comp rx=0 tx=0).stop 2024-06-29T14:38:12.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac5415c2f0 msgr2=0x7fac5413f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac5415c2f0 0x7fac5413f7f0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fac4800d770 tx=0x7fac4800dc30 comp rx=0 tx=0).stop 2024-06-29T14:38:12.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 shutdown_connections 2024-06-29T14:38:12.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fac40076120 0x7fac40078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac54138a10 0x7fac5413fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac5415c2f0 0x7fac5413f7f0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 --2- 172.21.15.59:0/654503802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac540b3740 0x7fac5415ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 >> 172.21.15.59:0/654503802 conn(0x7fac5406c950 msgr2=0x7fac540cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 shutdown_connections 2024-06-29T14:38:12.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.449+0000 7fac5bd03640 1 -- 172.21.15.59:0/654503802 wait complete. 2024-06-29T14:38:12.458 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:12.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.578+0000 7f6a060df640 1 -- 172.21.15.59:0/3088426854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 msgr2=0x7f69e8005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.578+0000 7f6a060df640 1 --2- 172.21.15.59:0/3088426854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 0x7f69e8005150 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f69f800c610 tx=0x7f69e8005830 comp rx=0 tx=0).stop 2024-06-29T14:38:12.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.578+0000 7f69fffff640 1 -- 172.21.15.59:0/3088426854 reap_dead start 2024-06-29T14:38:12.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.578+0000 7f6a060df640 1 -- 172.21.15.59:0/3088426854 shutdown_connections 2024-06-29T14:38:12.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.578+0000 7f6a060df640 1 -- 172.21.15.59:0/3088426854 >> 172.21.15.59:0/3088426854 conn(0x7f6a0006c950 msgr2=0x7f6a00073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.578+0000 7f6a060df640 1 -- 172.21.15.59:0/3088426854 shutdown_connections 2024-06-29T14:38:12.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.578+0000 7f6a060df640 1 -- 172.21.15.59:0/3088426854 wait complete. 2024-06-29T14:38:12.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.580+0000 7f6a060df640 1 Processor -- start 2024-06-29T14:38:12.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.580+0000 7f6a060df640 1 -- start start 2024-06-29T14:38:12.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f6a060df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 0x7f6a001f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f6a060df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0013e510 0x7f6a001f5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f6a060df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a001f57b0 0x7f6a001f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f6a060df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0015c880 con 0x7f69e8004d40 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f6a060df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0015c700 con 0x7f6a001f57b0 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f6a060df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6a0015ca00 con 0x7f6a0013e510 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 0x7f6a001f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 0x7f6a001f2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54236/0 (socket says 172.21.15.59:54236) 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.581+0000 7f69ff7fe640 1 -- 172.21.15.59:0/3776098773 learned_addr learned my addr 172.21.15.59:0/3776098773 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69fffff640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a001f57b0 0x7f6a001f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69feffd640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0013e510 0x7f6a001f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69ff7fe640 1 -- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0013e510 msgr2=0x7f6a001f5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69ff7fe640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0013e510 0x7f6a001f5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69ff7fe640 1 -- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a001f57b0 msgr2=0x7f6a001f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69ff7fe640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a001f57b0 0x7f6a001f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69ff7fe640 1 -- 172.21.15.59:0/3776098773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0013cb30 con 0x7f69e8004d40 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69fffff640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a001f57b0 0x7f6a001f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69feffd640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0013e510 0x7f6a001f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69ff7fe640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 0x7f6a001f2630 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f69f0011b40 tx=0x7f69f0009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69e7fff640 1 -- 172.21.15.59:0/3776098773 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f000fd10 con 0x7f69e8004d40 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69e7fff640 1 -- 172.21.15.59:0/3776098773 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69f0004020 con 0x7f69e8004d40 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a0013ce10 con 0x7f69e8004d40 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f69e7fff640 1 -- 172.21.15.59:0/3776098773 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0004470 con 0x7f69e8004d40 2024-06-29T14:38:12.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.582+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a0013d3d0 con 0x7f69e8004d40 2024-06-29T14:38:12.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.583+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a001458a0 con 0x7f69e8004d40 2024-06-29T14:38:12.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.584+0000 7f69e7fff640 1 -- 172.21.15.59:0/3776098773 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 44) ==== 98039+0+0 (secure 0 0 0) 0x7f69f00035f0 con 0x7f69e8004d40 2024-06-29T14:38:12.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.585+0000 7f69e7fff640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f69ec076150 0x7f69ec078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.585+0000 7f69e7fff640 1 -- 172.21.15.59:0/3776098773 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6789+0+0 (secure 0 0 0) 0x7f69f00982e0 con 0x7f69e8004d40 2024-06-29T14:38:12.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.585+0000 7f69feffd640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f69ec076150 0x7f69ec078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.586+0000 7f69feffd640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f69ec076150 0x7f69ec078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6a00070700 tx=0x7f69f4005f90 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.589+0000 7f69e7fff640 1 -- 172.21.15.59:0/3776098773 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f69f0061b40 con 0x7f69e8004d40 2024-06-29T14:38:12.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.783+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a000b6ab0 con 0x7f69e8004d40 2024-06-29T14:38:12.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.784+0000 7f69e7fff640 1 -- 172.21.15.59:0/3776098773 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+17979 (secure 0 0 0) 0x7f69f0061440 con 0x7f69e8004d40 2024-06-29T14:38:12.782 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:12.782 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":103,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:10.719590+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:51.120910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.698419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1412183002":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2783149049":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2472287695":"2024-06-30T14:38:10.719553+0000","172.21.15.153:6833/3952610634":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2216119053":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/3289012264":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2366299610":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6833/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6832/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3522491343":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6832/3952610634":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/3511527873":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1670969120":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3512251954":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/4085525721":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1421481485":"2024-06-30T14:38:05.021970+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:12.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f69ec076150 msgr2=0x7f69ec078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f69ec076150 0x7f69ec078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6a00070700 tx=0x7f69f4005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 msgr2=0x7f6a001f2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 0x7f6a001f2630 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f69f0011b40 tx=0x7f69f0009f10 comp rx=0 tx=0).stop 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 shutdown_connections 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f69ec076150 0x7f69ec078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a001f57b0 0x7f6a001f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a0013e510 0x7f6a001f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 --2- 172.21.15.59:0/3776098773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69e8004d40 0x7f6a001f2630 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 >> 172.21.15.59:0/3776098773 conn(0x7f6a0006c950 msgr2=0x7f6a000b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 shutdown_connections 2024-06-29T14:38:12.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.786+0000 7f6a060df640 1 -- 172.21.15.59:0/3776098773 wait complete. 2024-06-29T14:38:12.795 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:38:12.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.916+0000 7f4c887f5640 1 -- 172.21.15.59:0/1483049182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c6c004d90 msgr2=0x7f4c6c005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.916+0000 7f4c887f5640 1 --2- 172.21.15.59:0/1483049182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c6c004d90 0x7f4c6c005260 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4c78001dc0 tx=0x7f4c6c005830 comp rx=0 tx=0).stop 2024-06-29T14:38:12.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.916+0000 7f4c86d92640 1 -- 172.21.15.59:0/1483049182 reap_dead start 2024-06-29T14:38:12.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.916+0000 7f4c887f5640 1 -- 172.21.15.59:0/1483049182 shutdown_connections 2024-06-29T14:38:12.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.917+0000 7f4c887f5640 1 -- 172.21.15.59:0/1483049182 >> 172.21.15.59:0/1483049182 conn(0x7f4c800b5610 msgr2=0x7f4c800b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.917+0000 7f4c887f5640 1 -- 172.21.15.59:0/1483049182 shutdown_connections 2024-06-29T14:38:12.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.917+0000 7f4c887f5640 1 -- 172.21.15.59:0/1483049182 wait complete. 2024-06-29T14:38:12.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.918+0000 7f4c887f5640 1 Processor -- start 2024-06-29T14:38:12.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.918+0000 7f4c887f5640 1 -- start start 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c887f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c6c004d90 0x7f4c80161430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c887f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c801f45e0 0x7f4c8006d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c887f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c801f49f0 0x7f4c8006d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c887f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4c80162c10 con 0x7f4c801f45e0 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c887f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c801625c0 con 0x7f4c801f49f0 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c887f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4c80162d90 con 0x7f4c6c004d90 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c801f45e0 0x7f4c8006d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c801f45e0 0x7f4c8006d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58216/0 (socket says 172.21.15.59:58216) 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 -- 172.21.15.59:0/516652681 learned_addr learned my addr 172.21.15.59:0/516652681 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:12.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c86591640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c6c004d90 0x7f4c80161430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c86d92640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c801f49f0 0x7f4c8006d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 -- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c6c004d90 msgr2=0x7f4c80161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c6c004d90 0x7f4c80161430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 -- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c801f49f0 msgr2=0x7f4c8006d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c801f49f0 0x7f4c8006d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 -- 172.21.15.59:0/516652681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c80161970 con 0x7f4c801f45e0 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.920+0000 7f4c85d90640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c801f45e0 0x7f4c8006d230 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4c70011ae0 tx=0x7f4c70009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.921+0000 7f4c76ffd640 1 -- 172.21.15.59:0/516652681 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7001e070 con 0x7f4c801f45e0 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.921+0000 7f4c76ffd640 1 -- 172.21.15.59:0/516652681 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c7000fe70 con 0x7f4c801f45e0 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.921+0000 7f4c76ffd640 1 -- 172.21.15.59:0/516652681 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70004290 con 0x7f4c801f45e0 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.921+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c801f8a10 con 0x7f4c801f45e0 2024-06-29T14:38:12.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.921+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c801f8fd0 con 0x7f4c801f45e0 2024-06-29T14:38:12.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.922+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c800d2f00 con 0x7f4c801f45e0 2024-06-29T14:38:12.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.922+0000 7f4c76ffd640 1 -- 172.21.15.59:0/516652681 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 44) ==== 98039+0+0 (secure 0 0 0) 0x7f4c70021070 con 0x7f4c801f45e0 2024-06-29T14:38:12.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.923+0000 7f4c76ffd640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c64076150 0x7f4c64078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.923+0000 7f4c76ffd640 1 -- 172.21.15.59:0/516652681 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6789+0+0 (secure 0 0 0) 0x7f4c700981f0 con 0x7f4c801f45e0 2024-06-29T14:38:12.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.923+0000 7f4c86591640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c64076150 0x7f4c64078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.924+0000 7f4c86591640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c64076150 0x7f4c64078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4c800b7620 tx=0x7f4c7c000f30 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:12.927+0000 7f4c76ffd640 1 -- 172.21.15.59:0/516652681 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4c800d2f00 con 0x7f4c801f45e0 2024-06-29T14:38:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.097+0000 7f4c76ffd640 1 -- 172.21.15.59:0/516652681 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 45) ==== 98247+0+0 (secure 0 0 0) 0x7f4c7005d750 con 0x7f4c801f45e0 2024-06-29T14:38:13.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.161+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4c800cf080 con 0x7f4c801f45e0 2024-06-29T14:38:13.160 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T14:38:13.162+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:38:13.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.163+0000 7f4c76ffd640 1 -- 172.21.15.59:0/516652681 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v47) ==== 161+0+0 (secure 0 0 0) 0x7f4c700416a0 con 0x7f4c801f45e0 2024-06-29T14:38:13.161 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:38:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c64076150 msgr2=0x7f4c64078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c64076150 0x7f4c64078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4c800b7620 tx=0x7f4c7c000f30 comp rx=0 tx=0).stop 2024-06-29T14:38:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c801f45e0 msgr2=0x7f4c8006d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c801f45e0 0x7f4c8006d230 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4c70011ae0 tx=0x7f4c70009e90 comp rx=0 tx=0).stop 2024-06-29T14:38:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 shutdown_connections 2024-06-29T14:38:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c64076150 0x7f4c64078660 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c801f49f0 0x7f4c8006d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c801f45e0 0x7f4c8006d230 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 --2- 172.21.15.59:0/516652681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c6c004d90 0x7f4c80161430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 >> 172.21.15.59:0/516652681 conn(0x7f4c800b5610 msgr2=0x7f4c800c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:13.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 shutdown_connections 2024-06-29T14:38:13.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.166+0000 7f4c887f5640 1 -- 172.21.15.59:0/516652681 wait complete. 2024-06-29T14:38:13.166 INFO:tasks.ceph.mds.b.smithi153.stderr: -2858> 2024-06-29T14:37:55.343+0000 7f5572dea640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:38:13.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:38:13.222 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:38:13.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.297+0000 7efddc410640 1 -- 172.21.15.59:0/2515438688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd40b2a50 msgr2=0x7efdd4161380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.297+0000 7efddc410640 1 --2- 172.21.15.59:0/2515438688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd40b2a50 0x7efdd4161380 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efdd40d2c60 tx=0x7efdd002a890 comp rx=0 tx=0).stop 2024-06-29T14:38:13.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.297+0000 7efddc410640 1 -- 172.21.15.59:0/2515438688 shutdown_connections 2024-06-29T14:38:13.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.297+0000 7efddc410640 1 --2- 172.21.15.59:0/2515438688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd40b2a50 0x7efdd4161380 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.297+0000 7efddc410640 1 -- 172.21.15.59:0/2515438688 >> 172.21.15.59:0/2515438688 conn(0x7efdd40b5590 msgr2=0x7efdd40b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:13.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.297+0000 7efddc410640 1 -- 172.21.15.59:0/2515438688 shutdown_connections 2024-06-29T14:38:13.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.297+0000 7efddc410640 1 -- 172.21.15.59:0/2515438688 wait complete. 2024-06-29T14:38:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.299+0000 7efddc410640 1 Processor -- start 2024-06-29T14:38:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.299+0000 7efddc410640 1 -- start start 2024-06-29T14:38:13.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efddc410640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b2a50 0x7efdd41f5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efddc410640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd40b7150 0x7efdd41600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efddc410640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd4160620 0x7efdd40cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efddc410640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efdd40b4da0 con 0x7efdd40b2a50 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efddc410640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efdd40b4c20 con 0x7efdd40b7150 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efddc410640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efdd40b7560 con 0x7efdd4160620 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efdda1ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b2a50 0x7efdd41f5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efdda1ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b2a50 0x7efdd41f5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58226/0 (socket says 172.21.15.59:58226) 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.300+0000 7efdda1ac640 1 -- 172.21.15.59:0/1711288959 learned_addr learned my addr 172.21.15.59:0/1711288959 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdd99ab640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd40b7150 0x7efdd41600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdda9ad640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd4160620 0x7efdd40cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdda1ac640 1 -- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd4160620 msgr2=0x7efdd40cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdda1ac640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd4160620 0x7efdd40cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdda1ac640 1 -- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd40b7150 msgr2=0x7efdd41600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdda1ac640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd40b7150 0x7efdd41600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdda1ac640 1 -- 172.21.15.59:0/1711288959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd4158bb0 con 0x7efdd40b2a50 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdda1ac640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b2a50 0x7efdd41f5500 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009b30 tx=0x7efdd0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdcaffd640 1 -- 172.21.15.59:0/1711288959 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0041020 con 0x7efdd40b2a50 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd4158e90 con 0x7efdd40b2a50 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdcaffd640 1 -- 172.21.15.59:0/1711288959 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efdd00324f0 con 0x7efdd40b2a50 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efdcaffd640 1 -- 172.21.15.59:0/1711288959 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0037040 con 0x7efdd40b2a50 2024-06-29T14:38:13.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.301+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd40cd950 con 0x7efdd40b2a50 2024-06-29T14:38:13.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.302+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd4161380 con 0x7efdd40b2a50 2024-06-29T14:38:13.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.303+0000 7efdcaffd640 1 -- 172.21.15.59:0/1711288959 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 45) ==== 98247+0+0 (secure 0 0 0) 0x7efdd0044020 con 0x7efdd40b2a50 2024-06-29T14:38:13.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.303+0000 7efdcaffd640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efdb8076360 0x7efdb8078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:13.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.304+0000 7efdcaffd640 1 -- 172.21.15.59:0/1711288959 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6875+0+0 (secure 0 0 0) 0x7efdd00b85e0 con 0x7efdd40b2a50 2024-06-29T14:38:13.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.304+0000 7efdd99ab640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efdb8076360 0x7efdb8078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:13.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.304+0000 7efdd99ab640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efdb8076360 0x7efdb8078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efdc4000f60 tx=0x7efdc4009040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:13.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.308+0000 7efdcaffd640 1 -- 172.21.15.59:0/1711288959 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efdd004b030 con 0x7efdd40b2a50 2024-06-29T14:38:13.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:13.541+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7efdd408ccd0 con 0x7efdd40b2a50 2024-06-29T14:38:14.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.165+0000 7efdcaffd640 1 -- 172.21.15.59:0/1711288959 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) ==== 104+0+0 (secure 0 0 0) 0x7efdd0081270 con 0x7efdd40b2a50 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efdb8076360 msgr2=0x7efdb8078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efdb8076360 0x7efdb8078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efdc4000f60 tx=0x7efdc4009040 comp rx=0 tx=0).stop 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b2a50 msgr2=0x7efdd41f5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b2a50 0x7efdd41f5500 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009b30 tx=0x7efdd0004430 comp rx=0 tx=0).stop 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 shutdown_connections 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7efdb8076360 0x7efdb8078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd4160620 0x7efdd40cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd40b7150 0x7efdd41600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 --2- 172.21.15.59:0/1711288959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b2a50 0x7efdd41f5500 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 >> 172.21.15.59:0/1711288959 conn(0x7efdd40b5590 msgr2=0x7efdd406d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 shutdown_connections 2024-06-29T14:38:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.168+0000 7efddc410640 1 -- 172.21.15.59:0/1711288959 wait complete. 2024-06-29T14:38:14.180 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:38:14.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.312+0000 7f9087762640 1 -- 172.21.15.59:0/547642937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 msgr2=0x7f90800b6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.312+0000 7f9087762640 1 --2- 172.21.15.59:0/547642937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 0x7f90800b6f70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f90800b3d60 tx=0x7f907002a4a0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.312+0000 7f9087762640 1 -- 172.21.15.59:0/547642937 shutdown_connections 2024-06-29T14:38:14.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.312+0000 7f9087762640 1 --2- 172.21.15.59:0/547642937 >> v2:172.21.15.153:3300/0 conn(0x7f90800d2eb0 0x7f90800b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.312+0000 7f9087762640 1 --2- 172.21.15.59:0/547642937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 0x7f90800b6f70 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.312+0000 7f9087762640 1 -- 172.21.15.59:0/547642937 >> 172.21.15.59:0/547642937 conn(0x7f90800d0f90 msgr2=0x7f90800d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:14.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.312+0000 7f9087762640 1 -- 172.21.15.59:0/547642937 shutdown_connections 2024-06-29T14:38:14.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.312+0000 7f9087762640 1 -- 172.21.15.59:0/547642937 wait complete. 2024-06-29T14:38:14.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.314+0000 7f9087762640 1 Processor -- start 2024-06-29T14:38:14.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.314+0000 7f9087762640 1 -- start start 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f9087762640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 0x7f9080146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f9087762640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90800d1c30 0x7f90800c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f9087762640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90800d2eb0 0x7f908015d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f9087762640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f90800b4730 con 0x7f90800b2980 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f9087762640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90800b45b0 con 0x7f90800d1c30 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f9087762640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f90800b48b0 con 0x7f90800d2eb0 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f90854fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 0x7f9080146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f90854fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 0x7f9080146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58240/0 (socket says 172.21.15.59:58240) 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f90854fe640 1 -- 172.21.15.59:0/1225100832 learned_addr learned my addr 172.21.15.59:0/1225100832 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f9085cff640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90800d2eb0 0x7f908015d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.315+0000 7f9084cfd640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90800d1c30 0x7f90800c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f90854fe640 1 -- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90800d2eb0 msgr2=0x7f908015d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f90854fe640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90800d2eb0 0x7f908015d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f90854fe640 1 -- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90800d1c30 msgr2=0x7f90800c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f90854fe640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90800d1c30 0x7f90800c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f90854fe640 1 -- 172.21.15.59:0/1225100832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90801473c0 con 0x7f90800b2980 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f90854fe640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 0x7f9080146e80 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f907c00c3c0 tx=0x7f907c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f9075ffb640 1 -- 172.21.15.59:0/1225100832 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c017070 con 0x7f90800b2980 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f9075ffb640 1 -- 172.21.15.59:0/1225100832 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f907c0035f0 con 0x7f90800b2980 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f908015c0e0 con 0x7f90800b2980 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f9075ffb640 1 -- 172.21.15.59:0/1225100832 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c003a80 con 0x7f90800b2980 2024-06-29T14:38:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.316+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f908015c6a0 con 0x7f90800b2980 2024-06-29T14:38:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.317+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90801f5d60 con 0x7f90800b2980 2024-06-29T14:38:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.318+0000 7f9075ffb640 1 -- 172.21.15.59:0/1225100832 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 45) ==== 98247+0+0 (secure 0 0 0) 0x7f907c01a070 con 0x7f90800b2980 2024-06-29T14:38:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.318+0000 7f9075ffb640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f9064076390 0x7f90640788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.319+0000 7f9075ffb640 1 -- 172.21.15.59:0/1225100832 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6875+0+0 (secure 0 0 0) 0x7f907c096ff0 con 0x7f90800b2980 2024-06-29T14:38:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.319+0000 7f9084cfd640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f9064076390 0x7f90640788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.319+0000 7f9084cfd640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f9064076390 0x7f90640788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9070034000 tx=0x7f9070004730 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:14.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.323+0000 7f9075ffb640 1 -- 172.21.15.59:0/1225100832 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f90801f5d60 con 0x7f90800b2980 2024-06-29T14:38:14.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:14.513+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f90800b4d60 con 0x7f90800b2980 2024-06-29T14:38:15.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.170+0000 7f9075ffb640 1 -- 172.21.15.59:0/1225100832 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) ==== 181+0+0 (secure 0 0 0) 0x7f907c05fd70 con 0x7f90800b2980 2024-06-29T14:38:15.168 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:38:15.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.175+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f9064076390 msgr2=0x7f90640788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.175+0000 7f9087762640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f9064076390 0x7f90640788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9070034000 tx=0x7f9070004730 comp rx=0 tx=0).stop 2024-06-29T14:38:15.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.175+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 msgr2=0x7f9080146e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.175+0000 7f9087762640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 0x7f9080146e80 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f907c00c3c0 tx=0x7f907c00c880 comp rx=0 tx=0).stop 2024-06-29T14:38:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.176+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 shutdown_connections 2024-06-29T14:38:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.176+0000 7f9087762640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f9064076390 0x7f90640788a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.176+0000 7f9087762640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f90800d2eb0 0x7f908015d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.176+0000 7f9087762640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90800d1c30 0x7f90800c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.176+0000 7f9087762640 1 --2- 172.21.15.59:0/1225100832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90800b2980 0x7f9080146e80 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.176+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 >> 172.21.15.59:0/1225100832 conn(0x7f90800d0f90 msgr2=0x7f908015a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.177+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 shutdown_connections 2024-06-29T14:38:15.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.178+0000 7f9087762640 1 -- 172.21.15.59:0/1225100832 wait complete. 2024-06-29T14:38:15.192 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:38:15.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.322+0000 7fc7baf68640 1 -- 172.21.15.59:0/1297670609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b40d2780 msgr2=0x7fc7b41621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.322+0000 7fc7baf68640 1 --2- 172.21.15.59:0/1297670609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b40d2780 0x7fc7b41621f0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40c7270 tx=0x7fc7a402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.322+0000 7fc7baf68640 1 -- 172.21.15.59:0/1297670609 shutdown_connections 2024-06-29T14:38:15.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.322+0000 7fc7baf68640 1 --2- 172.21.15.59:0/1297670609 >> v2:172.21.15.153:3300/0 conn(0x7fc7b40b3ed0 0x7fc7b40b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.322+0000 7fc7baf68640 1 --2- 172.21.15.59:0/1297670609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b40d2780 0x7fc7b41621f0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.322+0000 7fc7baf68640 1 -- 172.21.15.59:0/1297670609 >> 172.21.15.59:0/1297670609 conn(0x7fc7b40d3e60 msgr2=0x7fc7b40d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:15.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.323+0000 7fc7baf68640 1 -- 172.21.15.59:0/1297670609 shutdown_connections 2024-06-29T14:38:15.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.323+0000 7fc7baf68640 1 -- 172.21.15.59:0/1297670609 wait complete. 2024-06-29T14:38:15.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.325+0000 7fc7baf68640 1 Processor -- start 2024-06-29T14:38:15.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.325+0000 7fc7baf68640 1 -- start start 2024-06-29T14:38:15.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.327+0000 7fc7baf68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b40b3ed0 0x7fc7b4153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:15.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.327+0000 7fc7baf68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b40d2780 0x7fc7b40ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:15.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.327+0000 7fc7baf68640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b415dff0 0x7fc7b40cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:15.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.327+0000 7fc7baf68640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b40b4820 con 0x7fc7b415dff0 2024-06-29T14:38:15.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.327+0000 7fc7baf68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b4162940 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.327+0000 7fc7baf68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b40b49a0 con 0x7fc7b40d2780 2024-06-29T14:38:15.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.327+0000 7fc7b8d04640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b40b3ed0 0x7fc7b4153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:15.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.327+0000 7fc7b9505640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b415dff0 0x7fc7b40cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b8d04640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b40b3ed0 0x7fc7b4153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:33004/0 (socket says 172.21.15.59:33004) 2024-06-29T14:38:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b8d04640 1 -- 172.21.15.59:0/177893374 learned_addr learned my addr 172.21.15.59:0/177893374 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b9505640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b415dff0 0x7fc7b40cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58256/0 (socket says 172.21.15.59:58256) 2024-06-29T14:38:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b3fff640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b40d2780 0x7fc7b40ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b8d04640 1 -- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b40d2780 msgr2=0x7fc7b40ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b8d04640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b40d2780 0x7fc7b40ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b8d04640 1 -- 172.21.15.59:0/177893374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b415dff0 msgr2=0x7fc7b40cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b8d04640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b415dff0 0x7fc7b40cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b8d04640 1 -- 172.21.15.59:0/177893374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b40ac920 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b9505640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b415dff0 0x7fc7b40cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:15.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.328+0000 7fc7b8d04640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b40b3ed0 0x7fc7b4153ea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00c420 tx=0x7fc79c00c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:15.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.329+0000 7fc7b17fa640 1 -- 172.21.15.59:0/177893374 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c017070 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.329+0000 7fc7b17fa640 1 -- 172.21.15.59:0/177893374 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc79c0035f0 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.329+0000 7fc7b17fa640 1 -- 172.21.15.59:0/177893374 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c003a40 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.329+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b40acc00 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.329+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b40ad1c0 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.330+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b40b4e90 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.331+0000 7fc7b17fa640 1 -- 172.21.15.59:0/177893374 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fc79c01a070 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.331+0000 7fc7b17fa640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7940763f0 0x7fc794078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.331+0000 7fc7b17fa640 1 -- 172.21.15.59:0/177893374 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6437+0+0 (secure 0 0 0) 0x7fc79c096e20 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.331+0000 7fc7b3fff640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7940763f0 0x7fc794078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.332+0000 7fc7b3fff640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7940763f0 0x7fc794078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4034000 tx=0x7fc7a4004ab0 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:15.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.336+0000 7fc7b17fa640 1 -- 172.21.15.59:0/177893374 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc7b40d2b90 con 0x7fc7b40b3ed0 2024-06-29T14:38:15.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:15.532+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc7b40b2c20 con 0x7fc7b40b3ed0 2024-06-29T14:38:16.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.182+0000 7fc7b17fa640 1 -- 172.21.15.59:0/177893374 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) ==== 169+0+0 (secure 0 0 0) 0x7fc79c05fdd0 con 0x7fc7b40b3ed0 2024-06-29T14:38:16.180 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T14:38:16.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7940763f0 msgr2=0x7fc794078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7940763f0 0x7fc794078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4034000 tx=0x7fc7a4004ab0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b40b3ed0 msgr2=0x7fc7b4153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b40b3ed0 0x7fc7b4153ea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00c420 tx=0x7fc79c00c8e0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 shutdown_connections 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7940763f0 0x7fc794078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7b415dff0 0x7fc7b40cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7b40d2780 0x7fc7b40ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 --2- 172.21.15.59:0/177893374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b40b3ed0 0x7fc7b4153ea0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 >> 172.21.15.59:0/177893374 conn(0x7fc7b40d3e60 msgr2=0x7fc7b406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 shutdown_connections 2024-06-29T14:38:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.185+0000 7fc7baf68640 1 -- 172.21.15.59:0/177893374 wait complete. 2024-06-29T14:38:16.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.325+0000 7f74326f7640 1 -- 172.21.15.59:0/2193526936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f742c0b3740 msgr2=0x7f742c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.325+0000 7f74326f7640 1 --2- 172.21.15.59:0/2193526936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f742c0b3740 0x7f742c0b3b50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7420005fd0 tx=0x7f742002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.325+0000 7f74326f7640 1 -- 172.21.15.59:0/2193526936 shutdown_connections 2024-06-29T14:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.325+0000 7f74326f7640 1 --2- 172.21.15.59:0/2193526936 >> v2:172.21.15.153:3300/0 conn(0x7f742c15c2f0 0x7f742c15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.325+0000 7f74326f7640 1 --2- 172.21.15.59:0/2193526936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f742c0b3740 0x7f742c0b3b50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.325+0000 7f74326f7640 1 -- 172.21.15.59:0/2193526936 >> 172.21.15.59:0/2193526936 conn(0x7f742c06c950 msgr2=0x7f742c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.325+0000 7f74326f7640 1 -- 172.21.15.59:0/2193526936 shutdown_connections 2024-06-29T14:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.325+0000 7f74326f7640 1 -- 172.21.15.59:0/2193526936 wait complete. 2024-06-29T14:38:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.327+0000 7f74326f7640 1 Processor -- start 2024-06-29T14:38:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.327+0000 7f74326f7640 1 -- start start 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f74326f7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f742c0b3740 0x7f742c13f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f74326f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c15c2f0 0x7f742c0c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f74326f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c0c7080 0x7f742c0c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f74326f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f742c0b5fd0 con 0x7f742c0c7080 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f74326f7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f742c0b5e50 con 0x7f742c15c2f0 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f74326f7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f742c0b6150 con 0x7f742c0b3740 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c0c7080 0x7f742c0c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c0c7080 0x7f742c0c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58272/0 (socket says 172.21.15.59:58272) 2024-06-29T14:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 -- 172.21.15.59:0/1645334203 learned_addr learned my addr 172.21.15.59:0/1645334203 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f742bfff640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f742c0b3740 0x7f742c13f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f742b7fe640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c15c2f0 0x7f742c0c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 -- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f742c0b3740 msgr2=0x7f742c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f742c0b3740 0x7f742c13f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 -- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c15c2f0 msgr2=0x7f742c0c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c15c2f0 0x7f742c0c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 -- 172.21.15.59:0/1645334203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c13fcf0 con 0x7f742c0c7080 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.329+0000 7f7430c94640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c0c7080 0x7f742c0c9330 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f742400ec50 tx=0x7f742400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.330+0000 7f7428ff9640 1 -- 172.21.15.59:0/1645334203 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424003b70 con 0x7f742c0c7080 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.330+0000 7f7428ff9640 1 -- 172.21.15.59:0/1645334203 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7424003d10 con 0x7f742c0c7080 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.330+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742c0c77c0 con 0x7f742c0c7080 2024-06-29T14:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.330+0000 7f7428ff9640 1 -- 172.21.15.59:0/1645334203 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424004280 con 0x7f742c0c7080 2024-06-29T14:38:16.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.330+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742c0c7d80 con 0x7f742c0c7080 2024-06-29T14:38:16.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.331+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742c0c7530 con 0x7f742c0c7080 2024-06-29T14:38:16.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.332+0000 7f7428ff9640 1 -- 172.21.15.59:0/1645334203 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f7424004420 con 0x7f742c0c7080 2024-06-29T14:38:16.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.332+0000 7f7428ff9640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7418076420 0x7f7418078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.332+0000 7f7428ff9640 1 -- 172.21.15.59:0/1645334203 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7f74240987d0 con 0x7f742c0c7080 2024-06-29T14:38:16.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.332+0000 7f742bfff640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7418076420 0x7f7418078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.333+0000 7f742bfff640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7418076420 0x7f7418078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f741c007fd0 tx=0x7f741c009040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:16.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.336+0000 7f7428ff9640 1 -- 172.21.15.59:0/1645334203 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7424061700 con 0x7f742c0c7080 2024-06-29T14:38:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.533+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f742c0b6670 con 0x7f742c0c7080 2024-06-29T14:38:16.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.537+0000 7f7428ff9640 1 -- 172.21.15.59:0/1645334203 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v43) ==== 71+0+0 (secure 0 0 0) 0x7f7424065a00 con 0x7f742c0c7080 2024-06-29T14:38:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.539+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7418076420 msgr2=0x7f7418078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.539+0000 7f74326f7640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7418076420 0x7f7418078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f741c007fd0 tx=0x7f741c009040 comp rx=0 tx=0).stop 2024-06-29T14:38:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.539+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c0c7080 msgr2=0x7f742c0c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.539+0000 7f74326f7640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c0c7080 0x7f742c0c9330 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f742400ec50 tx=0x7f742400c6a0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.539+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 shutdown_connections 2024-06-29T14:38:16.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.539+0000 7f74326f7640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7418076420 0x7f7418078930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.539+0000 7f74326f7640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c0c7080 0x7f742c0c9330 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.540+0000 7f74326f7640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f742c15c2f0 0x7f742c0c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.540+0000 7f74326f7640 1 --2- 172.21.15.59:0/1645334203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f742c0b3740 0x7f742c13f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.540+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 >> 172.21.15.59:0/1645334203 conn(0x7f742c06c950 msgr2=0x7f742c1415a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.540+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 shutdown_connections 2024-06-29T14:38:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.540+0000 7f74326f7640 1 -- 172.21.15.59:0/1645334203 wait complete. 2024-06-29T14:38:16.549 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-06-29T14:38:16.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.669+0000 7fb077e3c640 1 -- 172.21.15.59:0/88757985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 msgr2=0x7fb058004cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.669+0000 7fb077e3c640 1 --2- 172.21.15.59:0/88757985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 0x7fb058004cb0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fb0640017a0 tx=0x7fb058005830 comp rx=0 tx=0).stop 2024-06-29T14:38:16.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.669+0000 7fb0763d9640 1 -- 172.21.15.59:0/88757985 reap_dead start 2024-06-29T14:38:16.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.669+0000 7fb077e3c640 1 -- 172.21.15.59:0/88757985 shutdown_connections 2024-06-29T14:38:16.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.669+0000 7fb077e3c640 1 -- 172.21.15.59:0/88757985 >> 172.21.15.59:0/88757985 conn(0x7fb0700d3ee0 msgr2=0x7fb0700d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:16.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.669+0000 7fb077e3c640 1 -- 172.21.15.59:0/88757985 shutdown_connections 2024-06-29T14:38:16.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.669+0000 7fb077e3c640 1 -- 172.21.15.59:0/88757985 wait complete. 2024-06-29T14:38:16.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.671+0000 7fb077e3c640 1 Processor -- start 2024-06-29T14:38:16.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.671+0000 7fb077e3c640 1 -- start start 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb077e3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 0x7fb0700aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb077e3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb07015da60 0x7fb0700cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb077e3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0700ac060 0x7fb0700ac470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb077e3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb07015feb0 con 0x7fb058004800 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb077e3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb07015fd30 con 0x7fb0700ac060 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb077e3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb070160030 con 0x7fb07015da60 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb075bd8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 0x7fb0700aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb075bd8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 0x7fb0700aae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58284/0 (socket says 172.21.15.59:58284) 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.672+0000 7fb075bd8640 1 -- 172.21.15.59:0/2260997548 learned_addr learned my addr 172.21.15.59:0/2260997548 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:16.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb0753d7640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb07015da60 0x7fb0700cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb0763d9640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0700ac060 0x7fb0700ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb075bd8640 1 -- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb07015da60 msgr2=0x7fb0700cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb075bd8640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb07015da60 0x7fb0700cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb075bd8640 1 -- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0700ac060 msgr2=0x7fb0700ac470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb075bd8640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0700ac060 0x7fb0700ac470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb075bd8640 1 -- 172.21.15.59:0/2260997548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0700ac9b0 con 0x7fb058004800 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb0763d9640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0700ac060 0x7fb0700ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb075bd8640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 0x7fb0700aae80 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb060001be0 tx=0x7fb060006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb06e7fc640 1 -- 172.21.15.59:0/2260997548 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060015070 con 0x7fb058004800 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb06e7fc640 1 -- 172.21.15.59:0/2260997548 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb06000fce0 con 0x7fb058004800 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb07015b120 con 0x7fb058004800 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb06e7fc640 1 -- 172.21.15.59:0/2260997548 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060011690 con 0x7fb058004800 2024-06-29T14:38:16.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.673+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0700ab6d0 con 0x7fb058004800 2024-06-29T14:38:16.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.674+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb07014b970 con 0x7fb058004800 2024-06-29T14:38:16.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.675+0000 7fb06e7fc640 1 -- 172.21.15.59:0/2260997548 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fb060011830 con 0x7fb058004800 2024-06-29T14:38:16.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.676+0000 7fb06e7fc640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fb05c0763f0 0x7fb05c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.676+0000 7fb06e7fc640 1 -- 172.21.15.59:0/2260997548 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7fb060098a50 con 0x7fb058004800 2024-06-29T14:38:16.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.676+0000 7fb0753d7640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fb05c0763f0 0x7fb05c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.676+0000 7fb0753d7640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fb05c0763f0 0x7fb05c078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb064032000 tx=0x7fb064001360 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.680+0000 7fb06e7fc640 1 -- 172.21.15.59:0/2260997548 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb0700ac9b0 con 0x7fb058004800 2024-06-29T14:38:16.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:16.947+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) -- 0x7fb0701605c0 con 0x7fb058004800 2024-06-29T14:38:17.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.184+0000 7fb06e7fc640 1 -- 172.21.15.59:0/2260997548 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v627) ==== 166+0+0 (secure 0 0 0) 0x7fb0600619f0 con 0x7fb058004800 2024-06-29T14:38:17.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.186+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fb05c0763f0 msgr2=0x7fb05c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.186+0000 7fb077e3c640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fb05c0763f0 0x7fb05c078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb064032000 tx=0x7fb064001360 comp rx=0 tx=0).stop 2024-06-29T14:38:17.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.186+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 msgr2=0x7fb0700aae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.186+0000 7fb077e3c640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 0x7fb0700aae80 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb060001be0 tx=0x7fb060006720 comp rx=0 tx=0).stop 2024-06-29T14:38:17.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.187+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 shutdown_connections 2024-06-29T14:38:17.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.187+0000 7fb077e3c640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fb05c0763f0 0x7fb05c078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.187+0000 7fb077e3c640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0700ac060 0x7fb0700ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.187+0000 7fb077e3c640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb07015da60 0x7fb0700cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.187+0000 7fb077e3c640 1 --2- 172.21.15.59:0/2260997548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb058004800 0x7fb0700aae80 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.187+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 >> 172.21.15.59:0/2260997548 conn(0x7fb0700d3ee0 msgr2=0x7fb07006c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.187+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 shutdown_connections 2024-06-29T14:38:17.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.187+0000 7fb077e3c640 1 -- 172.21.15.59:0/2260997548 wait complete. 2024-06-29T14:38:17.196 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-06-29T14:38:17.196 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-06-29T14:38:17.196 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:38:17.197 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.321+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3299782837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc15efe0 msgr2=0x7f45bc15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.321+0000 7f45c3ae7640 1 --2- 172.21.15.59:0/3299782837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc15efe0 0x7f45bc15f9b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f45b4005fd0 tx=0x7f45b402ab40 comp rx=0 tx=0).stop 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.321+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3299782837 shutdown_connections 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.321+0000 7f45c3ae7640 1 --2- 172.21.15.59:0/3299782837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc15efe0 0x7f45bc15f9b0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.321+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3299782837 >> 172.21.15.59:0/3299782837 conn(0x7f45bc0d3ee0 msgr2=0x7f45bc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.321+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3299782837 shutdown_connections 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.321+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3299782837 wait complete. 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.323+0000 7f45c3ae7640 1 Processor -- start 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.323+0000 7f45c3ae7640 1 -- start start 2024-06-29T14:38:17.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c3ae7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45bc0d23c0 0x7f45bc0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c3ae7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45bc15efe0 0x7f45bc0c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c3ae7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc1565a0 0x7f45bc1569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c3ae7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc160470 con 0x7f45bc1565a0 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c3ae7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1602f0 con 0x7f45bc15efe0 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c3ae7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45bc1605f0 con 0x7f45bc0d23c0 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c2084640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc1565a0 0x7f45bc1569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c2084640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc1565a0 0x7f45bc1569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58314/0 (socket says 172.21.15.59:58314) 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c2084640 1 -- 172.21.15.59:0/3608166021 learned_addr learned my addr 172.21.15.59:0/3608166021 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c1883640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45bc0d23c0 0x7f45bc0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c1082640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45bc15efe0 0x7f45bc0c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c2084640 1 -- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45bc0d23c0 msgr2=0x7f45bc0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c2084640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45bc0d23c0 0x7f45bc0cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c2084640 1 -- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45bc15efe0 msgr2=0x7f45bc0c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c2084640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45bc15efe0 0x7f45bc0c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c2084640 1 -- 172.21.15.59:0/3608166021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45bc0ccc20 con 0x7f45bc1565a0 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.324+0000 7f45c1082640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45bc15efe0 0x7f45bc0c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.325+0000 7f45c2084640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc1565a0 0x7f45bc1569b0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f45b4006010 tx=0x7f45b402c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.325+0000 7f45b27fc640 1 -- 172.21.15.59:0/3608166021 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4044070 con 0x7f45bc1565a0 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.325+0000 7f45b27fc640 1 -- 172.21.15.59:0/3608166021 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45b4004250 con 0x7f45bc1565a0 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.325+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45bc0cd0b0 con 0x7f45bc1565a0 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.325+0000 7f45b27fc640 1 -- 172.21.15.59:0/3608166021 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b40046a0 con 0x7f45bc1565a0 2024-06-29T14:38:17.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.325+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45bc0c70f0 con 0x7f45bc1565a0 2024-06-29T14:38:17.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.326+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45bc15f9b0 con 0x7f45bc1565a0 2024-06-29T14:38:17.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.327+0000 7f45b27fc640 1 -- 172.21.15.59:0/3608166021 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f45b4036400 con 0x7f45bc1565a0 2024-06-29T14:38:17.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.327+0000 7f45b27fc640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f45a00763f0 0x7f45a0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.327+0000 7f45b27fc640 1 -- 172.21.15.59:0/3608166021 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7f45b40b83b0 con 0x7f45bc1565a0 2024-06-29T14:38:17.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.328+0000 7f45c1883640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f45a00763f0 0x7f45a0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.328+0000 7f45c1883640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f45a00763f0 0x7f45a0078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f45ac000f60 tx=0x7f45ac009040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.332+0000 7f45b27fc640 1 -- 172.21.15.59:0/3608166021 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f45bc0ccc20 con 0x7f45bc1565a0 2024-06-29T14:38:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.526+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45bc0d2950 con 0x7f45bc1565a0 2024-06-29T14:38:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.526+0000 7f45b27fc640 1 -- 172.21.15.59:0/3608166021 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+14329 (secure 0 0 0) 0x7f45b4032030 con 0x7f45bc1565a0 2024-06-29T14:38:17.525 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:17.525 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:16.171694+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6837/1942687234":"2024-06-30T14:38:13.163548+0000","172.21.15.153:6835/1942687234":"2024-06-30T14:38:13.163548+0000","172.21.15.153:0/1412183002":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2783149049":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2472287695":"2024-06-30T14:38:10.719553+0000","172.21.15.153:6833/3952610634":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2216119053":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/3289012264":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/2366299610":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6833/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6832/1607211985":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3522491343":"2024-06-30T14:38:05.021970+0000","172.21.15.153:6832/3952610634":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/3511527873":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1670969120":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/3512251954":"2024-06-30T14:38:10.719553+0000","172.21.15.153:0/4085525721":"2024-06-30T14:38:05.021970+0000","172.21.15.153:0/1421481485":"2024-06-30T14:38:05.021970+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f45a00763f0 msgr2=0x7f45a0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f45a00763f0 0x7f45a0078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f45ac000f60 tx=0x7f45ac009040 comp rx=0 tx=0).stop 2024-06-29T14:38:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc1565a0 msgr2=0x7f45bc1569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc1565a0 0x7f45bc1569b0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f45b4006010 tx=0x7f45b402c8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 shutdown_connections 2024-06-29T14:38:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f45a00763f0 0x7f45a0078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45bc1565a0 0x7f45bc1569b0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45bc15efe0 0x7f45bc0c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 --2- 172.21.15.59:0/3608166021 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45bc0d23c0 0x7f45bc0cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 >> 172.21.15.59:0/3608166021 conn(0x7f45bc0d3ee0 msgr2=0x7f45bc150820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 shutdown_connections 2024-06-29T14:38:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.528+0000 7f45c3ae7640 1 -- 172.21.15.59:0/3608166021 wait complete. 2024-06-29T14:38:17.538 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:38:17.538 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:38:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.661+0000 7eff8890a640 1 -- 172.21.15.59:0/3083556130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 msgr2=0x7eff840ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.661+0000 7eff8890a640 1 --2- 172.21.15.59:0/3083556130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 0x7eff840ce550 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7eff74005fd0 tx=0x7eff7402aa30 comp rx=0 tx=0).stop 2024-06-29T14:38:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.661+0000 7eff8890a640 1 -- 172.21.15.59:0/3083556130 shutdown_connections 2024-06-29T14:38:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.661+0000 7eff8890a640 1 --2- 172.21.15.59:0/3083556130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff8415e0a0 0x7eff84161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.661+0000 7eff8890a640 1 --2- 172.21.15.59:0/3083556130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 0x7eff840ce550 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.661+0000 7eff8890a640 1 -- 172.21.15.59:0/3083556130 >> 172.21.15.59:0/3083556130 conn(0x7eff840d45b0 msgr2=0x7eff840d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.661+0000 7eff8890a640 1 -- 172.21.15.59:0/3083556130 shutdown_connections 2024-06-29T14:38:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.661+0000 7eff8890a640 1 -- 172.21.15.59:0/3083556130 wait complete. 2024-06-29T14:38:17.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.663+0000 7eff8890a640 1 Processor -- start 2024-06-29T14:38:17.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.663+0000 7eff8890a640 1 -- start start 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff8890a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff8415e0a0 0x7eff84153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff8890a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff84161fc0 0x7eff840c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff8890a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 0x7eff84155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff8890a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7eff840d2d80 con 0x7eff84162770 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff8890a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7eff840b4b00 con 0x7eff84161fc0 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff8890a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7eff840d2f00 con 0x7eff8415e0a0 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff82d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 0x7eff84155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff82d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 0x7eff84155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58338/0 (socket says 172.21.15.59:58338) 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.664+0000 7eff82d9d640 1 -- 172.21.15.59:0/3921052830 learned_addr learned my addr 172.21.15.59:0/3921052830 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff8259c640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff8415e0a0 0x7eff84153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff81d9b640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff84161fc0 0x7eff840c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff82d9d640 1 -- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff8415e0a0 msgr2=0x7eff84153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff82d9d640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff8415e0a0 0x7eff84153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff82d9d640 1 -- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff84161fc0 msgr2=0x7eff840c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff82d9d640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff84161fc0 0x7eff840c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff82d9d640 1 -- 172.21.15.59:0/3921052830 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff841549a0 con 0x7eff84162770 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff82d9d640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 0x7eff84155460 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00c3f0 tx=0x7eff7c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff7affd640 1 -- 172.21.15.59:0/3921052830 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c017070 con 0x7eff84162770 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff7affd640 1 -- 172.21.15.59:0/3921052830 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff7c0053e0 con 0x7eff84162770 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff840ca400 con 0x7eff84162770 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff7affd640 1 -- 172.21.15.59:0/3921052830 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c003870 con 0x7eff84162770 2024-06-29T14:38:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.665+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8415ee20 con 0x7eff84162770 2024-06-29T14:38:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.666+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff840d3430 con 0x7eff84162770 2024-06-29T14:38:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.667+0000 7eff7affd640 1 -- 172.21.15.59:0/3921052830 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7eff7c01a070 con 0x7eff84162770 2024-06-29T14:38:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.667+0000 7eff7affd640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7eff64076420 0x7eff64078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.668+0000 7eff7affd640 1 -- 172.21.15.59:0/3921052830 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7eff7c096c80 con 0x7eff84162770 2024-06-29T14:38:17.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.668+0000 7eff8259c640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7eff64076420 0x7eff64078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.668+0000 7eff8259c640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7eff64076420 0x7eff64078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7eff74035000 tx=0x7eff7403b040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.672+0000 7eff7affd640 1 -- 172.21.15.59:0/3921052830 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7eff841623d0 con 0x7eff84162770 2024-06-29T14:38:17.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.863+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff840ca6e0 con 0x7eff84162770 2024-06-29T14:38:17.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.864+0000 7eff7affd640 1 -- 172.21.15.59:0/3921052830 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7eff7c05fe00 con 0x7eff84162770 2024-06-29T14:38:17.862 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:38:17.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.865+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7eff64076420 msgr2=0x7eff64078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.865+0000 7eff8890a640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7eff64076420 0x7eff64078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7eff74035000 tx=0x7eff7403b040 comp rx=0 tx=0).stop 2024-06-29T14:38:17.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.865+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 msgr2=0x7eff84155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.865+0000 7eff8890a640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 0x7eff84155460 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00c3f0 tx=0x7eff7c00c8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.866+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 shutdown_connections 2024-06-29T14:38:17.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.866+0000 7eff8890a640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7eff64076420 0x7eff64078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.866+0000 7eff8890a640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff84162770 0x7eff84155460 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.866+0000 7eff8890a640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff84161fc0 0x7eff840c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.866+0000 7eff8890a640 1 --2- 172.21.15.59:0/3921052830 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7eff8415e0a0 0x7eff84153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.866+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 >> 172.21.15.59:0/3921052830 conn(0x7eff840d45b0 msgr2=0x7eff840d2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.866+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 shutdown_connections 2024-06-29T14:38:17.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.866+0000 7eff8890a640 1 -- 172.21.15.59:0/3921052830 wait complete. 2024-06-29T14:38:17.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-06-29T14:38:17.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.995+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/345713012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bd80b40a0 msgr2=0x7f4bd815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.995+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/345713012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bd80b40a0 0x7f4bd815e8d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009830 tx=0x7f4bc002a500 comp rx=0 tx=0).stop 2024-06-29T14:38:17.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.995+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/345713012 shutdown_connections 2024-06-29T14:38:17.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.995+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/345713012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd815ee10 0x7f4bd815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.995+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/345713012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bd80b40a0 0x7f4bd815e8d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.995+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/345713012 >> 172.21.15.59:0/345713012 conn(0x7f4bd80d3ee0 msgr2=0x7f4bd80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.995+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/345713012 shutdown_connections 2024-06-29T14:38:17.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.995+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/345713012 wait complete. 2024-06-29T14:38:17.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.997+0000 7f4bdc9d4640 1 Processor -- start 2024-06-29T14:38:17.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.997+0000 7f4bdc9d4640 1 -- start start 2024-06-29T14:38:17.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bdc9d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80d23c0 0x7f4bd815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bdc9d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bd815ee10 0x7f4bd8149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bdc9d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4bd81f55c0 0x7f4bd81f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bdc9d4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd8160280 con 0x7f4bd81f55c0 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bdc9d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd8160100 con 0x7f4bd815ee10 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bdc9d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd8160400 con 0x7f4bd80d23c0 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bd6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4bd81f55c0 0x7f4bd81f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bd6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4bd81f55c0 0x7f4bd81f59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58342/0 (socket says 172.21.15.59:58342) 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bd6d9d640 1 -- 172.21.15.59:0/3741729112 learned_addr learned my addr 172.21.15.59:0/3741729112 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bd5d9b640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bd815ee10 0x7f4bd8149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.998+0000 7f4bd659c640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80d23c0 0x7f4bd815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bd6d9d640 1 -- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80d23c0 msgr2=0x7f4bd815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bd6d9d640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80d23c0 0x7f4bd815adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bd6d9d640 1 -- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bd815ee10 msgr2=0x7f4bd8149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bd6d9d640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bd815ee10 0x7f4bd8149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bd6d9d640 1 -- 172.21.15.59:0/3741729112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd815a000 con 0x7f4bd81f55c0 2024-06-29T14:38:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bd6d9d640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4bd81f55c0 0x7f4bd81f59d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000e940 tx=0x7f4bd000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bceffd640 1 -- 172.21.15.59:0/3741729112 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000cd20 con 0x7f4bd81f55c0 2024-06-29T14:38:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bceffd640 1 -- 172.21.15.59:0/3741729112 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4bd0004020 con 0x7f4bd81f55c0 2024-06-29T14:38:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd815a2e0 con 0x7f4bd81f55c0 2024-06-29T14:38:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bceffd640 1 -- 172.21.15.59:0/3741729112 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0004470 con 0x7f4bd81f55c0 2024-06-29T14:38:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:17.999+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd815a8a0 con 0x7f4bd81f55c0 2024-06-29T14:38:18.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:18.000+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd81ea3f0 con 0x7f4bd81f55c0 2024-06-29T14:38:18.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:18.001+0000 7f4bceffd640 1 -- 172.21.15.59:0/3741729112 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f4bd00035f0 con 0x7f4bd81f55c0 2024-06-29T14:38:18.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:18.001+0000 7f4bceffd640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4bb80763f0 0x7f4bb8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:18.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:18.001+0000 7f4bceffd640 1 -- 172.21.15.59:0/3741729112 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7f4bd0098270 con 0x7f4bd81f55c0 2024-06-29T14:38:18.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:18.002+0000 7f4bd659c640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4bb80763f0 0x7f4bb8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:18.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:18.002+0000 7f4bd659c640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4bb80763f0 0x7f4bb8078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80017a0 tx=0x7f4bc8001730 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:18.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:18.006+0000 7f4bceffd640 1 -- 172.21.15.59:0/3741729112 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4bd00612b0 con 0x7f4bd81f55c0 2024-06-29T14:38:18.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:18.272+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) -- 0x7f4bd80d2950 con 0x7f4bd81f55c0 2024-06-29T14:38:19.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.194+0000 7f4bceffd640 1 -- 172.21.15.59:0/3741729112 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v629) ==== 186+0+0 (secure 0 0 0) 0x7f4bd00655b0 con 0x7f4bd81f55c0 2024-06-29T14:38:19.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4bb80763f0 msgr2=0x7f4bb8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4bb80763f0 0x7f4bb8078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80017a0 tx=0x7f4bc8001730 comp rx=0 tx=0).stop 2024-06-29T14:38:19.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4bd81f55c0 msgr2=0x7f4bd81f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4bd81f55c0 0x7f4bd81f59d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000e940 tx=0x7f4bd000ee00 comp rx=0 tx=0).stop 2024-06-29T14:38:19.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 shutdown_connections 2024-06-29T14:38:19.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4bb80763f0 0x7f4bb8078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4bd81f55c0 0x7f4bd81f59d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bd815ee10 0x7f4bd8149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 --2- 172.21.15.59:0/3741729112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4bd80d23c0 0x7f4bd815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 >> 172.21.15.59:0/3741729112 conn(0x7f4bd80d3ee0 msgr2=0x7f4bd8157f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 shutdown_connections 2024-06-29T14:38:19.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.197+0000 7f4bdc9d4640 1 -- 172.21.15.59:0/3741729112 wait complete. 2024-06-29T14:38:19.209 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:38:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.336+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1325022361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fc4004d40 msgr2=0x7f4fc4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.336+0000 7f4fe2b9a640 1 --2- 172.21.15.59:0/1325022361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fc4004d40 0x7f4fc4005150 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4005fd0 tx=0x7f4fc4005830 comp rx=0 tx=0).stop 2024-06-29T14:38:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.336+0000 7f4fe1137640 1 -- 172.21.15.59:0/1325022361 reap_dead start 2024-06-29T14:38:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.336+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1325022361 shutdown_connections 2024-06-29T14:38:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.336+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1325022361 >> 172.21.15.59:0/1325022361 conn(0x7f4fdc0b5610 msgr2=0x7f4fdc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.336+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1325022361 shutdown_connections 2024-06-29T14:38:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.336+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1325022361 wait complete. 2024-06-29T14:38:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.338+0000 7f4fe2b9a640 1 Processor -- start 2024-06-29T14:38:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.338+0000 7f4fe2b9a640 1 -- start start 2024-06-29T14:38:19.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe2b9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fdc1f8d90 0x7f4fdc14cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe2b9a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4fdc15cd40 0x7f4fdc15d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe2b9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fdc15ff40 0x7f4fdc160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe2b9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc162fa0 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe2b9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc162e20 con 0x7f4fdc15ff40 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe2b9a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4fdc0b7bd0 con 0x7f4fdc15cd40 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fdc1f8d90 0x7f4fdc14cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fdc1f8d90 0x7f4fdc14cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58358/0 (socket says 172.21.15.59:58358) 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 -- 172.21.15.59:0/1427966671 learned_addr learned my addr 172.21.15.59:0/1427966671 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fdbfff640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4fdc15cd40 0x7f4fdc15d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe1137640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fdc15ff40 0x7f4fdc160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 -- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4fdc15cd40 msgr2=0x7f4fdc15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4fdc15cd40 0x7f4fdc15d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 -- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fdc15ff40 msgr2=0x7f4fdc160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fdc15ff40 0x7f4fdc160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 -- 172.21.15.59:0/1427966671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fdc15d690 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fdbfff640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4fdc15cd40 0x7f4fdc15d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe1137640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fdc15ff40 0x7f4fdc160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.340+0000 7f4fe0936640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fdc1f8d90 0x7f4fdc14cf30 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc006660 tx=0x7f4fcc003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.341+0000 7f4fd97fa640 1 -- 172.21.15.59:0/1427966671 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc015070 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.341+0000 7f4fd97fa640 1 -- 172.21.15.59:0/1427966671 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fcc011920 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.341+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fdc0cafb0 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.341+0000 7f4fd97fa640 1 -- 172.21.15.59:0/1427966671 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc011d70 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.341+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fdc0cb570 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.342+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fdc1622c0 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.342+0000 7f4fd97fa640 1 -- 172.21.15.59:0/1427966671 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f4fcc00f430 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.343+0000 7f4fd97fa640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4fc80763f0 0x7f4fc8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.343+0000 7f4fd97fa640 1 -- 172.21.15.59:0/1427966671 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7f4fcc098830 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.343+0000 7f4fdbfff640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4fc80763f0 0x7f4fc8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.344+0000 7f4fdbfff640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4fc80763f0 0x7f4fc8078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc0aa5a0 tx=0x7f4fd0005f90 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:19.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.347+0000 7f4fd97fa640 1 -- 172.21.15.59:0/1427966671 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4fdc15d690 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.538+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f4fdc0cf400 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.539+0000 7f4fd97fa640 1 -- 172.21.15.59:0/1427966671 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v43) ==== 59+0+1998 (secure 0 0 0) 0x7f4fcc061e40 con 0x7f4fdc1f8d90 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T14:38:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:38:19.538 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:38:19.539 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:38:19.539 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T14:38:19.539 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T14:38:19.539 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T14:38:19.539 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:38:19.539 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:38:19.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.540+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4fc80763f0 msgr2=0x7f4fc8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.540+0000 7f4fe2b9a640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4fc80763f0 0x7f4fc8078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc0aa5a0 tx=0x7f4fd0005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:19.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.540+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fdc1f8d90 msgr2=0x7f4fdc14cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.540+0000 7f4fe2b9a640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fdc1f8d90 0x7f4fdc14cf30 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc006660 tx=0x7f4fcc003aa0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.541+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 shutdown_connections 2024-06-29T14:38:19.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.541+0000 7f4fe2b9a640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4fc80763f0 0x7f4fc8078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.541+0000 7f4fe2b9a640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fdc15ff40 0x7f4fdc160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.541+0000 7f4fe2b9a640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4fdc15cd40 0x7f4fdc15d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.541+0000 7f4fe2b9a640 1 --2- 172.21.15.59:0/1427966671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fdc1f8d90 0x7f4fdc14cf30 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.541+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 >> 172.21.15.59:0/1427966671 conn(0x7f4fdc0b5610 msgr2=0x7f4fdc0b8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.541+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 shutdown_connections 2024-06-29T14:38:19.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.541+0000 7f4fe2b9a640 1 -- 172.21.15.59:0/1427966671 wait complete. 2024-06-29T14:38:19.550 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:38:19.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 -- 172.21.15.59:0/1934775109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 msgr2=0x7fd5640d2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 --2- 172.21.15.59:0/1934775109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 0x7fd5640d2670 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fd5640b35d0 tx=0x7fd55402a970 comp rx=0 tx=0).stop 2024-06-29T14:38:19.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 -- 172.21.15.59:0/1934775109 shutdown_connections 2024-06-29T14:38:19.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 --2- 172.21.15.59:0/1934775109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 0x7fd5640d2670 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 --2- 172.21.15.59:0/1934775109 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd56415e0a0 0x7fd564161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 --2- 172.21.15.59:0/1934775109 >> v2:172.21.15.153:3300/0 conn(0x7fd564162770 0x7fd5640ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 -- 172.21.15.59:0/1934775109 >> 172.21.15.59:0/1934775109 conn(0x7fd5640d45b0 msgr2=0x7fd5640d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 -- 172.21.15.59:0/1934775109 shutdown_connections 2024-06-29T14:38:19.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.672+0000 7fd569881640 1 -- 172.21.15.59:0/1934775109 wait complete. 2024-06-29T14:38:19.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.674+0000 7fd569881640 1 Processor -- start 2024-06-29T14:38:19.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.674+0000 7fd569881640 1 -- start start 2024-06-29T14:38:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd569881640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd56415e0a0 0x7fd564156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd569881640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 0x7fd564161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd569881640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564162770 0x7fd564153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd569881640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd5640d2f40 con 0x7fd564161fc0 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd569881640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd5640d2dc0 con 0x7fd564162770 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd569881640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd5640d30c0 con 0x7fd56415e0a0 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd5627fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 0x7fd564161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd5627fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 0x7fd564161190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58368/0 (socket says 172.21.15.59:58368) 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd5627fc640 1 -- 172.21.15.59:0/117730019 learned_addr learned my addr 172.21.15.59:0/117730019 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd562ffd640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd56415e0a0 0x7fd564156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.675+0000 7fd5637fe640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564162770 0x7fd564153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd5627fc640 1 -- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd56415e0a0 msgr2=0x7fd564156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd5627fc640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd56415e0a0 0x7fd564156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd5627fc640 1 -- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564162770 msgr2=0x7fd564153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd5627fc640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564162770 0x7fd564153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd5627fc640 1 -- 172.21.15.59:0/117730019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5641f65c0 con 0x7fd564161fc0 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd5627fc640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 0x7fd564161190 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00d5f0 tx=0x7fd54c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd55bfff640 1 -- 172.21.15.59:0/117730019 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54c017070 con 0x7fd564161fc0 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd55bfff640 1 -- 172.21.15.59:0/117730019 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd54c005330 con 0x7fd564161fc0 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5641f68a0 con 0x7fd564161fc0 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd55bfff640 1 -- 172.21.15.59:0/117730019 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54c0047a0 con 0x7fd564161fc0 2024-06-29T14:38:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.676+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5641f6e60 con 0x7fd564161fc0 2024-06-29T14:38:19.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.677+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5641f6330 con 0x7fd564161fc0 2024-06-29T14:38:19.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.678+0000 7fd55bfff640 1 -- 172.21.15.59:0/117730019 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fd54c004940 con 0x7fd564161fc0 2024-06-29T14:38:19.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.678+0000 7fd55bfff640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fd5440763f0 0x7fd544078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.678+0000 7fd55bfff640 1 -- 172.21.15.59:0/117730019 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7fd54c097b60 con 0x7fd564161fc0 2024-06-29T14:38:19.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.679+0000 7fd562ffd640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fd5440763f0 0x7fd544078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.679+0000 7fd562ffd640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fd5440763f0 0x7fd544078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00bb50 tx=0x7fd55c00b2e0 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:19.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.683+0000 7fd55bfff640 1 -- 172.21.15.59:0/117730019 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd54c060a80 con 0x7fd564161fc0 2024-06-29T14:38:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.874+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd5640cdd90 con 0x7fd564161fc0 2024-06-29T14:38:19.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.878+0000 7fd55bfff640 1 -- 172.21.15.59:0/117730019 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7fd54c069150 con 0x7fd564161fc0 2024-06-29T14:38:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fd5440763f0 msgr2=0x7fd544078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fd5440763f0 0x7fd544078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00bb50 tx=0x7fd55c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 msgr2=0x7fd564161190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 0x7fd564161190 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00d5f0 tx=0x7fd54c00dab0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 shutdown_connections 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fd5440763f0 0x7fd544078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd564162770 0x7fd564153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd564161fc0 0x7fd564161190 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 --2- 172.21.15.59:0/117730019 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd56415e0a0 0x7fd564156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 >> 172.21.15.59:0/117730019 conn(0x7fd5640d45b0 msgr2=0x7fd5640b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 shutdown_connections 2024-06-29T14:38:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:19.880+0000 7fd569881640 1 -- 172.21.15.59:0/117730019 wait complete. 2024-06-29T14:38:19.890 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:38:19.890 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:38:19.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:38:19.977 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:38:19.977 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:38:19.977 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:19.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:38:19.993 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:38:19.993 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:20.035 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:38:20.035 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:38:20.035 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:20.051 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:38:20.051 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:38:20.051 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:20.092 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:38:20.093 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:38:20.093 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.211+0000 7fe764281640 1 -- 172.21.15.59:0/1471497051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0ce950 msgr2=0x7fe75c0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.211+0000 7fe764281640 1 --2- 172.21.15.59:0/1471497051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0ce950 0x7fe75c0d2130 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fe754005fd0 tx=0x7fe75402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.212+0000 7fe764281640 1 -- 172.21.15.59:0/1471497051 shutdown_connections 2024-06-29T14:38:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.212+0000 7fe764281640 1 --2- 172.21.15.59:0/1471497051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0ce950 0x7fe75c0d2130 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.212+0000 7fe764281640 1 --2- 172.21.15.59:0/1471497051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe75c08c190 0x7fe75c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.212+0000 7fe764281640 1 -- 172.21.15.59:0/1471497051 >> 172.21.15.59:0/1471497051 conn(0x7fe75c0b5610 msgr2=0x7fe75c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.212+0000 7fe764281640 1 -- 172.21.15.59:0/1471497051 shutdown_connections 2024-06-29T14:38:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.212+0000 7fe764281640 1 -- 172.21.15.59:0/1471497051 wait complete. 2024-06-29T14:38:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.213+0000 7fe764281640 1 Processor -- start 2024-06-29T14:38:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.213+0000 7fe764281640 1 -- start start 2024-06-29T14:38:20.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe764281640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c08c190 0x7fe75c14ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe764281640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe75c0ce950 0x7fe75c160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe764281640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0cf5c0 0x7fe75c156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe764281640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c0b7fa0 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe764281640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c0b7e20 con 0x7fe75c08c190 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe764281640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe75c0b8120 con 0x7fe75c0ce950 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0cf5c0 0x7fe75c156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0cf5c0 0x7fe75c156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58376/0 (socket says 172.21.15.59:58376) 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 -- 172.21.15.59:0/3809298882 learned_addr learned my addr 172.21.15.59:0/3809298882 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76181c640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe75c0ce950 0x7fe75c160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76201d640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c08c190 0x7fe75c14ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 -- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe75c0ce950 msgr2=0x7fe75c160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe75c0ce950 0x7fe75c160ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 -- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c08c190 msgr2=0x7fe75c14ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c08c190 0x7fe75c14ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 -- 172.21.15.59:0/3809298882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75c14b370 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76201d640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c08c190 0x7fe75c14ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.215+0000 7fe76281e640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0cf5c0 0x7fe75c156e70 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fe754009cd0 tx=0x7fe7540039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.216+0000 7fe752ffd640 1 -- 172.21.15.59:0/3809298882 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754031990 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.216+0000 7fe752ffd640 1 -- 172.21.15.59:0/3809298882 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe754031c20 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.216+0000 7fe752ffd640 1 -- 172.21.15.59:0/3809298882 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7540330d0 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.216+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe75c161550 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.216+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe75c0cb9f0 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.217+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe75c0cf080 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.218+0000 7fe752ffd640 1 -- 172.21.15.59:0/3809298882 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fe754033270 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.218+0000 7fe752ffd640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe7400763e0 0x7fe7400788f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.218+0000 7fe752ffd640 1 -- 172.21.15.59:0/3809298882 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7fe7540b8fe0 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.218+0000 7fe76201d640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe7400763e0 0x7fe7400788f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.219+0000 7fe76201d640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe7400763e0 0x7fe7400788f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe758009aa0 tx=0x7fe758005db0 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:20.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.223+0000 7fe752ffd640 1 -- 172.21.15.59:0/3809298882 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe75403f030 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.451+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe75c06d9c0 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.451+0000 7fe752ffd640 1 -- 172.21.15.59:0/3809298882 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+3914 (secure 0 0 0) 0x7fe7540825e0 con 0x7fe75c0cf5c0 2024-06-29T14:38:20.450 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:20.450 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":48,"btime":"2024-06-29T14:38:14:165151+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2174310652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2174310652},{"type":"v1","addr":"172.21.15.59:6835","nonce":2174310652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":6356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3217643445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3217643445},{"type":"v1","addr":"172.21.15.153:6836","nonce":3217643445}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":7279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1843182753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1843182753},{"type":"v1","addr":"172.21.15.153:6837","nonce":1843182753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":7347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/832666341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":832666341},{"type":"v1","addr":"172.21.15.59:6833","nonce":832666341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[]} 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 48 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe7400763e0 msgr2=0x7fe7400788f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe7400763e0 0x7fe7400788f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe758009aa0 tx=0x7fe758005db0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0cf5c0 msgr2=0x7fe75c156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0cf5c0 0x7fe75c156e70 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fe754009cd0 tx=0x7fe7540039f0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 shutdown_connections 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe7400763e0 0x7fe7400788f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe75c0cf5c0 0x7fe75c156e70 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe75c0ce950 0x7fe75c160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 --2- 172.21.15.59:0/3809298882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe75c08c190 0x7fe75c14ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 >> 172.21.15.59:0/3809298882 conn(0x7fe75c0b5610 msgr2=0x7fe75c0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 shutdown_connections 2024-06-29T14:38:20.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.453+0000 7fe764281640 1 -- 172.21.15.59:0/3809298882 wait complete. 2024-06-29T14:38:20.463 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:38:20.463 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:38:20.463 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:38:20.463 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:38:20.463 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:38:20.463 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:38:20.464 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:38:20.464 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:38:20.464 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:38:20.464 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:38:20.464 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:38:20.464 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:38:20.465 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:38:20.465 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:38:20.465 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:38:20.465 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:38:20.473 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:38:20.473 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:38:20.474 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:38:20.474 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:38:20.479 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:38:20.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:38:20.481 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:38:20.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:38:20.515 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:38:20.516 DEBUG:teuthology.parallel:result is None 2024-06-29T14:38:20.516 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:38:20.517 DEBUG:teuthology.parallel:result is None 2024-06-29T14:38:20.523 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:38:20.523 DEBUG:teuthology.parallel:result is None 2024-06-29T14:38:20.525 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:38:20.525 DEBUG:teuthology.parallel:result is None 2024-06-29T14:38:20.525 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:38:20.605 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:38:20.606 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:38:20.619 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:38:20.620 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:38:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.652+0000 7f256f59a640 1 -- 172.21.15.59:0/3146571037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25680b3ed0 msgr2=0x7f256815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.652+0000 7f256f59a640 1 --2- 172.21.15.59:0/3146571037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25680b3ed0 0x7f256815e8d0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f2558005fd0 tx=0x7f255802a530 comp rx=0 tx=0).stop 2024-06-29T14:38:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.652+0000 7f256f59a640 1 -- 172.21.15.59:0/3146571037 shutdown_connections 2024-06-29T14:38:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.652+0000 7f256f59a640 1 --2- 172.21.15.59:0/3146571037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25680b3ed0 0x7f256815e8d0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.652+0000 7f256f59a640 1 --2- 172.21.15.59:0/3146571037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25680d23c0 0x7f25680b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.652+0000 7f256f59a640 1 -- 172.21.15.59:0/3146571037 >> 172.21.15.59:0/3146571037 conn(0x7f25680d3ee0 msgr2=0x7f25680d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.653+0000 7f256f59a640 1 -- 172.21.15.59:0/3146571037 shutdown_connections 2024-06-29T14:38:20.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.653+0000 7f256f59a640 1 -- 172.21.15.59:0/3146571037 wait complete. 2024-06-29T14:38:20.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.654+0000 7f256f59a640 1 Processor -- start 2024-06-29T14:38:20.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.654+0000 7f256f59a640 1 -- start start 2024-06-29T14:38:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256f59a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25680b3ed0 0x7f256815dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256f59a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25680d23c0 0x7f2568148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256f59a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f256815ee10 0x7f25681e6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256f59a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2568160240 con 0x7f256815ee10 2024-06-29T14:38:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256f59a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f25681600c0 con 0x7f25680d23c0 2024-06-29T14:38:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256f59a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f25681603c0 con 0x7f25680b3ed0 2024-06-29T14:38:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f256815ee10 0x7f25681e6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f256815ee10 0x7f25681e6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58412/0 (socket says 172.21.15.59:58412) 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 -- 172.21.15.59:0/380324167 learned_addr learned my addr 172.21.15.59:0/380324167 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256cb35640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25680d23c0 0x7f2568148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256d336640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25680b3ed0 0x7f256815dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 -- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25680b3ed0 msgr2=0x7f256815dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25680b3ed0 0x7f256815dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 -- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25680d23c0 msgr2=0x7f2568148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25680d23c0 0x7f2568148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 -- 172.21.15.59:0/380324167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256815ad60 con 0x7f256815ee10 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256db37640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f256815ee10 0x7f25681e6a40 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f256000b9a0 tx=0x7f256000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f255dffb640 1 -- 172.21.15.59:0/380324167 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560019070 con 0x7f256815ee10 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f255dffb640 1 -- 172.21.15.59:0/380324167 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f256000cc90 con 0x7f256815ee10 2024-06-29T14:38:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f255dffb640 1 -- 172.21.15.59:0/380324167 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25600146a0 con 0x7f256815ee10 2024-06-29T14:38:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.656+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256815b040 con 0x7f256815ee10 2024-06-29T14:38:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.657+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f256815b600 con 0x7f256815ee10 2024-06-29T14:38:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.658+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f256815e8d0 con 0x7f256815ee10 2024-06-29T14:38:20.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.658+0000 7f255dffb640 1 -- 172.21.15.59:0/380324167 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f256001c070 con 0x7f256815ee10 2024-06-29T14:38:20.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.659+0000 7f255dffb640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f254c0763f0 0x7f254c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.659+0000 7f255dffb640 1 -- 172.21.15.59:0/380324167 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6023+0+0 (secure 0 0 0) 0x7f2560097710 con 0x7f256815ee10 2024-06-29T14:38:20.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.659+0000 7f256d336640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f254c0763f0 0x7f254c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.659+0000 7f256d336640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f254c0763f0 0x7f254c078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25680d2e40 tx=0x7f256400a250 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:20.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.663+0000 7f255dffb640 1 -- 172.21.15.59:0/380324167 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2560060740 con 0x7f256815ee10 2024-06-29T14:38:20.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:20.872+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f25680d2950 con 0x7f256815ee10 2024-06-29T14:38:21.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.206+0000 7f255dffb640 1 -- 172.21.15.59:0/380324167 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v108) ==== 97+0+0 (secure 0 0 0) 0x7f2560064a40 con 0x7f256815ee10 2024-06-29T14:38:21.204 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T14:38:21.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f254c0763f0 msgr2=0x7f254c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f254c0763f0 0x7f254c078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25680d2e40 tx=0x7f256400a250 comp rx=0 tx=0).stop 2024-06-29T14:38:21.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f256815ee10 msgr2=0x7f25681e6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f256815ee10 0x7f25681e6a40 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f256000b9a0 tx=0x7f256000be60 comp rx=0 tx=0).stop 2024-06-29T14:38:21.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 shutdown_connections 2024-06-29T14:38:21.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f254c0763f0 0x7f254c078900 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25680d2e40 tx=0x7f256400a250 comp rx=0 tx=0).stop 2024-06-29T14:38:21.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f256815ee10 0x7f25681e6a40 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25680d23c0 0x7f2568148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 --2- 172.21.15.59:0/380324167 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25680b3ed0 0x7f256815dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 >> 172.21.15.59:0/380324167 conn(0x7f25680d3ee0 msgr2=0x7f25680ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 shutdown_connections 2024-06-29T14:38:21.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.209+0000 7f256f59a640 1 -- 172.21.15.59:0/380324167 wait complete. 2024-06-29T14:38:21.220 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:38:21.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.341+0000 7f4e8e447640 1 -- 172.21.15.59:0/3190312572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 msgr2=0x7f4e70003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.341+0000 7f4e8e447640 1 --2- 172.21.15.59:0/3190312572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 0x7f4e70003f30 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f4e78006fd0 tx=0x7f4e70005830 comp rx=0 tx=0).stop 2024-06-29T14:38:21.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.341+0000 7f4e8e447640 1 -- 172.21.15.59:0/3190312572 shutdown_connections 2024-06-29T14:38:21.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.341+0000 7f4e8e447640 1 --2- 172.21.15.59:0/3190312572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 0x7f4e70003f30 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.341+0000 7f4e8e447640 1 -- 172.21.15.59:0/3190312572 >> 172.21.15.59:0/3190312572 conn(0x7f4e880d45b0 msgr2=0x7f4e880d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.341+0000 7f4e8e447640 1 -- 172.21.15.59:0/3190312572 shutdown_connections 2024-06-29T14:38:21.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.341+0000 7f4e8e447640 1 -- 172.21.15.59:0/3190312572 wait complete. 2024-06-29T14:38:21.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.343+0000 7f4e8e447640 1 Processor -- start 2024-06-29T14:38:21.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.343+0000 7f4e8e447640 1 -- start start 2024-06-29T14:38:21.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e8e447640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 0x7f4e8815a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e8e447640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e880b2850 0x7f4e880c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e8e447640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8815ad20 0x7f4e8815b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e8e447640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88161a80 con 0x7f4e70003ac0 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e8e447640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e880ce7c0 con 0x7f4e8815ad20 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e8e447640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4e88161c00 con 0x7f4e880b2850 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e87fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 0x7f4e8815a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e87fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 0x7f4e8815a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58434/0 (socket says 172.21.15.59:58434) 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.344+0000 7f4e87fff640 1 -- 172.21.15.59:0/1196720908 learned_addr learned my addr 172.21.15.59:0/1196720908 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e877fe640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e880b2850 0x7f4e880c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e8c9e4640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8815ad20 0x7f4e8815b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e87fff640 1 -- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e880b2850 msgr2=0x7f4e880c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e87fff640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e880b2850 0x7f4e880c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e87fff640 1 -- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8815ad20 msgr2=0x7f4e8815b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e87fff640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8815ad20 0x7f4e8815b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e87fff640 1 -- 172.21.15.59:0/1196720908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8815b670 con 0x7f4e70003ac0 2024-06-29T14:38:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e8c9e4640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8815ad20 0x7f4e8815b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e87fff640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 0x7f4e8815a0d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f4e7802b130 tx=0x7f4e78004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e84ff9640 1 -- 172.21.15.59:0/1196720908 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78031040 con 0x7f4e70003ac0 2024-06-29T14:38:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e84ff9640 1 -- 172.21.15.59:0/1196720908 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e78006440 con 0x7f4e70003ac0 2024-06-29T14:38:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e84ff9640 1 -- 172.21.15.59:0/1196720908 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78006ca0 con 0x7f4e70003ac0 2024-06-29T14:38:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e880c8210 con 0x7f4e70003ac0 2024-06-29T14:38:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.345+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e880c87d0 con 0x7f4e70003ac0 2024-06-29T14:38:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.346+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e88162160 con 0x7f4e70003ac0 2024-06-29T14:38:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.347+0000 7f4e84ff9640 1 -- 172.21.15.59:0/1196720908 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f4e78034070 con 0x7f4e70003ac0 2024-06-29T14:38:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.347+0000 7f4e84ff9640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4e740763f0 0x7f4e74078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.348+0000 7f4e84ff9640 1 -- 172.21.15.59:0/1196720908 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5249+0+0 (secure 0 0 0) 0x7f4e780b76e0 con 0x7f4e70003ac0 2024-06-29T14:38:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.348+0000 7f4e877fe640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4e740763f0 0x7f4e74078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.348+0000 7f4e877fe640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4e740763f0 0x7f4e74078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e880caa20 tx=0x7f4e7c006d90 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.352+0000 7f4e84ff9640 1 -- 172.21.15.59:0/1196720908 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4e8815b670 con 0x7f4e70003ac0 2024-06-29T14:38:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.614+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4e880aa4b0 con 0x7f4e70003ac0 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.614+0000 7f4e84ff9640 1 -- 172.21.15.59:0/1196720908 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7f4e880aa4b0 con 0x7f4e70003ac0 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:38:21.613 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.615 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T14:38:21.615 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T14:38:21.615 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.615 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:38:21.615 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:38:21.615 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:38:21.617 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.617 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.617 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.617 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.617 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.618 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.619 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.620 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T14:38:21.621 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.622 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T14:38:21.623 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:38:21.624 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.625 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T14:38:21.626 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.627 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T14:38:21.628 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4e740763f0 msgr2=0x7f4e74078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4e740763f0 0x7f4e74078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e880caa20 tx=0x7f4e7c006d90 comp rx=0 tx=0).stop 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 msgr2=0x7f4e8815a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 0x7f4e8815a0d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f4e7802b130 tx=0x7f4e78004df0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 shutdown_connections 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4e740763f0 0x7f4e74078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e8815ad20 0x7f4e8815b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e880b2850 0x7f4e880c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 --2- 172.21.15.59:0/1196720908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e70003ac0 0x7f4e8815a0d0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.616+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 >> 172.21.15.59:0/1196720908 conn(0x7f4e880d45b0 msgr2=0x7f4e8815be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.617+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 shutdown_connections 2024-06-29T14:38:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.617+0000 7f4e8e447640 1 -- 172.21.15.59:0/1196720908 wait complete. 2024-06-29T14:38:21.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:21.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.752+0000 7f58687a1640 1 -- 172.21.15.59:0/3093339383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584c004800 msgr2=0x7f584c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.752+0000 7f58687a1640 1 --2- 172.21.15.59:0/3093339383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584c004800 0x7f584c004c10 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f585000c470 tx=0x7f584c005830 comp rx=0 tx=0).stop 2024-06-29T14:38:21.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.752+0000 7f5866d3e640 1 -- 172.21.15.59:0/3093339383 reap_dead start 2024-06-29T14:38:21.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.752+0000 7f58687a1640 1 -- 172.21.15.59:0/3093339383 shutdown_connections 2024-06-29T14:38:21.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.752+0000 7f58687a1640 1 -- 172.21.15.59:0/3093339383 >> 172.21.15.59:0/3093339383 conn(0x7f58600d3ee0 msgr2=0x7f58600d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.752+0000 7f58687a1640 1 -- 172.21.15.59:0/3093339383 shutdown_connections 2024-06-29T14:38:21.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.752+0000 7f58687a1640 1 -- 172.21.15.59:0/3093339383 wait complete. 2024-06-29T14:38:21.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.754+0000 7f58687a1640 1 Processor -- start 2024-06-29T14:38:21.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.754+0000 7f58687a1640 1 -- start start 2024-06-29T14:38:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f58687a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f584c004800 0x7f58600c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f58687a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5860153930 0x7f58600cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f58687a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5860152090 0x7f58601524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f58687a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f586015fd40 con 0x7f5860152090 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f58687a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f586015fbc0 con 0x7f584c004800 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f58687a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f586015fec0 con 0x7f5860153930 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f5866d3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5860152090 0x7f58601524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f5866d3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5860152090 0x7f58601524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58466/0 (socket says 172.21.15.59:58466) 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f5866d3e640 1 -- 172.21.15.59:0/2277653665 learned_addr learned my addr 172.21.15.59:0/2277653665 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f586653d640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f584c004800 0x7f58600c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.755+0000 7f5865d3c640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5860153930 0x7f58600cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5866d3e640 1 -- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5860153930 msgr2=0x7f58600cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5866d3e640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5860153930 0x7f58600cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5866d3e640 1 -- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f584c004800 msgr2=0x7f58600c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5866d3e640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f584c004800 0x7f58600c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5866d3e640 1 -- 172.21.15.59:0/2277653665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58600cbd60 con 0x7f5860152090 2024-06-29T14:38:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5866d3e640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5860152090 0x7f58601524a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5860161d40 tx=0x7f585800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5856ffd640 1 -- 172.21.15.59:0/2277653665 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858004b20 con 0x7f5860152090 2024-06-29T14:38:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5856ffd640 1 -- 172.21.15.59:0/2277653665 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5858004cc0 con 0x7f5860152090 2024-06-29T14:38:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f5856ffd640 1 -- 172.21.15.59:0/2277653665 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858003bd0 con 0x7f5860152090 2024-06-29T14:38:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58600cc040 con 0x7f5860152090 2024-06-29T14:38:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.756+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58600cc600 con 0x7f5860152090 2024-06-29T14:38:21.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.757+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58600b3990 con 0x7f5860152090 2024-06-29T14:38:21.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.758+0000 7f5856ffd640 1 -- 172.21.15.59:0/2277653665 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f5858004030 con 0x7f5860152090 2024-06-29T14:38:21.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.758+0000 7f5856ffd640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f58440763f0 0x7f5844078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.758+0000 7f5856ffd640 1 -- 172.21.15.59:0/2277653665 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5249+0+0 (secure 0 0 0) 0x7f5858097cb0 con 0x7f5860152090 2024-06-29T14:38:21.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.759+0000 7f586653d640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f58440763f0 0x7f5844078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.759+0000 7f586653d640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f58440763f0 0x7f5844078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f585c005e00 tx=0x7f585c005f90 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.763+0000 7f5856ffd640 1 -- 172.21.15.59:0/2277653665 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5858060e30 con 0x7f5860152090 2024-06-29T14:38:21.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.954+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58600d2950 con 0x7f5860152090 2024-06-29T14:38:21.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.955+0000 7f5856ffd640 1 -- 172.21.15.59:0/2277653665 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+13178 (secure 0 0 0) 0x7f5858065130 con 0x7f5860152090 2024-06-29T14:38:21.954 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:21.954 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":110,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:21.217121+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.957+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f58440763f0 msgr2=0x7f5844078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.957+0000 7f58687a1640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f58440763f0 0x7f5844078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f585c005e00 tx=0x7f585c005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5860152090 msgr2=0x7f58601524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5860152090 0x7f58601524a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5860161d40 tx=0x7f585800f3c0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 shutdown_connections 2024-06-29T14:38:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f58440763f0 0x7f5844078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5860152090 0x7f58601524a0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5860153930 0x7f58600cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 --2- 172.21.15.59:0/2277653665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f584c004800 0x7f58600c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 >> 172.21.15.59:0/2277653665 conn(0x7f58600d3ee0 msgr2=0x7f586006cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 shutdown_connections 2024-06-29T14:38:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:21.958+0000 7f58687a1640 1 -- 172.21.15.59:0/2277653665 wait complete. 2024-06-29T14:38:21.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:38:21.968 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:38:22.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.091+0000 7f67e597a640 1 -- 172.21.15.59:0/2842017030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e01458a0 msgr2=0x7f67e00cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.091+0000 7f67e597a640 1 --2- 172.21.15.59:0/2842017030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e01458a0 0x7f67e00cf870 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f67d0005fd0 tx=0x7f67d002aa90 comp rx=0 tx=0).stop 2024-06-29T14:38:22.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.092+0000 7f67e597a640 1 -- 172.21.15.59:0/2842017030 shutdown_connections 2024-06-29T14:38:22.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.092+0000 7f67e597a640 1 --2- 172.21.15.59:0/2842017030 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67e00b3b50 0x7f67e00b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.092+0000 7f67e597a640 1 --2- 172.21.15.59:0/2842017030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e01458a0 0x7f67e00cf870 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.092+0000 7f67e597a640 1 -- 172.21.15.59:0/2842017030 >> 172.21.15.59:0/2842017030 conn(0x7f67e006c950 msgr2=0x7f67e0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.092+0000 7f67e597a640 1 -- 172.21.15.59:0/2842017030 shutdown_connections 2024-06-29T14:38:22.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.092+0000 7f67e597a640 1 -- 172.21.15.59:0/2842017030 wait complete. 2024-06-29T14:38:22.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.093+0000 7f67e597a640 1 Processor -- start 2024-06-29T14:38:22.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.094+0000 7f67e597a640 1 -- start start 2024-06-29T14:38:22.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67e597a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e00b3740 0x7f67e013fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67e597a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67e00b3b50 0x7f67e006db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67e597a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67e01f7ee0 0x7f67e01f82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67e597a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f67e00b6150 con 0x7f67e00b3740 2024-06-29T14:38:22.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67e597a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67e00b5fd0 con 0x7f67e00b3b50 2024-06-29T14:38:22.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67e597a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f67e00b62d0 con 0x7f67e01f7ee0 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67deffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e00b3740 0x7f67e013fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67deffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e00b3740 0x7f67e013fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58484/0 (socket says 172.21.15.59:58484) 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67deffd640 1 -- 172.21.15.59:0/173663123 learned_addr learned my addr 172.21.15.59:0/173663123 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67df7fe640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67e01f7ee0 0x7f67e01f82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67de7fc640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67e00b3b50 0x7f67e006db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67deffd640 1 -- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67e01f7ee0 msgr2=0x7f67e01f82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67deffd640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67e01f7ee0 0x7f67e01f82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67deffd640 1 -- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67e00b3b50 msgr2=0x7f67e006db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67deffd640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67e00b3b50 0x7f67e006db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67deffd640 1 -- 172.21.15.59:0/173663123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e01f8830 con 0x7f67e00b3740 2024-06-29T14:38:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.095+0000 7f67df7fe640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67e01f7ee0 0x7f67e01f82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.096+0000 7f67deffd640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e00b3740 0x7f67e013fe40 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f67d0000f80 tx=0x7f67d0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.096+0000 7f67d7fff640 1 -- 172.21.15.59:0/173663123 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0044070 con 0x7f67e00b3740 2024-06-29T14:38:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.096+0000 7f67d7fff640 1 -- 172.21.15.59:0/173663123 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67d0004250 con 0x7f67e00b3740 2024-06-29T14:38:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.096+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e006e0c0 con 0x7f67e00b3740 2024-06-29T14:38:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.096+0000 7f67d7fff640 1 -- 172.21.15.59:0/173663123 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0004640 con 0x7f67e00b3740 2024-06-29T14:38:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.096+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e006e8b0 con 0x7f67e00b3740 2024-06-29T14:38:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.097+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e01458a0 con 0x7f67e00b3740 2024-06-29T14:38:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.098+0000 7f67d7fff640 1 -- 172.21.15.59:0/173663123 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f67d0047070 con 0x7f67e00b3740 2024-06-29T14:38:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.098+0000 7f67d7fff640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f67c0076420 0x7f67c0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.098+0000 7f67d7fff640 1 -- 172.21.15.59:0/173663123 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5249+0+0 (secure 0 0 0) 0x7f67d00b87f0 con 0x7f67e00b3740 2024-06-29T14:38:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.098+0000 7f67de7fc640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f67c0076420 0x7f67c0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.099+0000 7f67de7fc640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f67c0076420 0x7f67c0078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f67e00d2060 tx=0x7f67c8009040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.103+0000 7f67d7fff640 1 -- 172.21.15.59:0/173663123 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f67e01f8830 con 0x7f67e00b3740 2024-06-29T14:38:22.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.297+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67e00b1f10 con 0x7f67e00b3740 2024-06-29T14:38:22.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.298+0000 7f67d7fff640 1 -- 172.21.15.59:0/173663123 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f67d0032030 con 0x7f67e00b3740 2024-06-29T14:38:22.296 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f67c0076420 msgr2=0x7f67c0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f67c0076420 0x7f67c0078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f67e00d2060 tx=0x7f67c8009040 comp rx=0 tx=0).stop 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e00b3740 msgr2=0x7f67e013fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e00b3740 0x7f67e013fe40 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f67d0000f80 tx=0x7f67d0003f10 comp rx=0 tx=0).stop 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 shutdown_connections 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f67c0076420 0x7f67c0078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67e01f7ee0 0x7f67e01f82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67e00b3b50 0x7f67e006db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 --2- 172.21.15.59:0/173663123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e00b3740 0x7f67e013fe40 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 >> 172.21.15.59:0/173663123 conn(0x7f67e006c950 msgr2=0x7f67e0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 shutdown_connections 2024-06-29T14:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.300+0000 7f67e597a640 1 -- 172.21.15.59:0/173663123 wait complete. 2024-06-29T14:38:22.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:38:22.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 -- 172.21.15.59:0/215168788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd00cf5c0 msgr2=0x7fcdd00d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/215168788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd00cf5c0 0x7fcdd00d1a20 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4005fd0 tx=0x7fcdc402a580 comp rx=0 tx=0).stop 2024-06-29T14:38:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 -- 172.21.15.59:0/215168788 shutdown_connections 2024-06-29T14:38:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/215168788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdd00ce950 0x7fcdd00d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/215168788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd00cf5c0 0x7fcdd00d1a20 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/215168788 >> v2:172.21.15.153:3300/0 conn(0x7fcdd008c190 0x7fcdd00cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 -- 172.21.15.59:0/215168788 >> 172.21.15.59:0/215168788 conn(0x7fcdd00b5610 msgr2=0x7fcdd00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 -- 172.21.15.59:0/215168788 shutdown_connections 2024-06-29T14:38:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.429+0000 7fcdd7c47640 1 -- 172.21.15.59:0/215168788 wait complete. 2024-06-29T14:38:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.431+0000 7fcdd7c47640 1 Processor -- start 2024-06-29T14:38:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.431+0000 7fcdd7c47640 1 -- start start 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd7c47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd008c190 0x7fcdd014d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd7c47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd00ce950 0x7fcdd014f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd7c47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdd00cf5c0 0x7fcdd00b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd7c47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd0162440 con 0x7fcdd008c190 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd7c47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd01622c0 con 0x7fcdd00ce950 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd7c47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd01625c0 con 0x7fcdd00cf5c0 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd59e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd008c190 0x7fcdd014d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd59e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd008c190 0x7fcdd014d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58496/0 (socket says 172.21.15.59:58496) 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd59e3640 1 -- 172.21.15.59:0/2159834939 learned_addr learned my addr 172.21.15.59:0/2159834939 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd51e2640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd00ce950 0x7fcdd014f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.432+0000 7fcdd61e4640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdd00cf5c0 0x7fcdd00b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdd59e3640 1 -- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdd00cf5c0 msgr2=0x7fcdd00b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdd59e3640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdd00cf5c0 0x7fcdd00b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdd59e3640 1 -- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd00ce950 msgr2=0x7fcdd014f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdd59e3640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd00ce950 0x7fcdd014f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdd59e3640 1 -- 172.21.15.59:0/2159834939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd014df80 con 0x7fcdd008c190 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdd59e3640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd008c190 0x7fcdd014d610 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0005fd0 tx=0x7fcdc000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdce7fc640 1 -- 172.21.15.59:0/2159834939 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc000f450 con 0x7fcdd008c190 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdce7fc640 1 -- 172.21.15.59:0/2159834939 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdc000f6e0 con 0x7fcdd008c190 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd014e260 con 0x7fcdd008c190 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdce7fc640 1 -- 172.21.15.59:0/2159834939 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc000fb10 con 0x7fcdd008c190 2024-06-29T14:38:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.433+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd014e820 con 0x7fcdd008c190 2024-06-29T14:38:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.434+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd00cf080 con 0x7fcdd008c190 2024-06-29T14:38:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.435+0000 7fcdce7fc640 1 -- 172.21.15.59:0/2159834939 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fcdc000fcb0 con 0x7fcdd008c190 2024-06-29T14:38:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.435+0000 7fcdce7fc640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fcdbc0763f0 0x7fcdbc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.436+0000 7fcdce7fc640 1 -- 172.21.15.59:0/2159834939 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5249+0+0 (secure 0 0 0) 0x7fcdc00981e0 con 0x7fcdd008c190 2024-06-29T14:38:22.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.436+0000 7fcdd51e2640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fcdbc0763f0 0x7fcdbc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.436+0000 7fcdd51e2640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fcdbc0763f0 0x7fcdbc078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4033000 tx=0x7fcdc4004730 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.440+0000 7fcdce7fc640 1 -- 172.21.15.59:0/2159834939 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcdc0014030 con 0x7fcdd008c190 2024-06-29T14:38:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.635+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fcdd015c400 con 0x7fcdd008c190 2024-06-29T14:38:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.636+0000 7fcdce7fc640 1 -- 172.21.15.59:0/2159834939 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v110) ==== 84+0+65 (secure 0 0 0) 0x7fcdc0061400 con 0x7fcdd008c190 2024-06-29T14:38:22.634 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:22.634 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:38:22.634 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:22.634 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:38:22.635 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:38:22.635 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:22.635 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:38:22.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fcdbc0763f0 msgr2=0x7fcdbc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fcdbc0763f0 0x7fcdbc078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4033000 tx=0x7fcdc4004730 comp rx=0 tx=0).stop 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd008c190 msgr2=0x7fcdd014d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd008c190 0x7fcdd014d610 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0005fd0 tx=0x7fcdc000a620 comp rx=0 tx=0).stop 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 shutdown_connections 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fcdbc0763f0 0x7fcdbc078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdd00cf5c0 0x7fcdd00b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdd00ce950 0x7fcdd014f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 --2- 172.21.15.59:0/2159834939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdd008c190 0x7fcdd014d610 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 >> 172.21.15.59:0/2159834939 conn(0x7fcdd00b5610 msgr2=0x7fcdd00c88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 shutdown_connections 2024-06-29T14:38:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.638+0000 7fcdd7c47640 1 -- 172.21.15.59:0/2159834939 wait complete. 2024-06-29T14:38:22.648 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:38:22.648 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:38:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.772+0000 7ff72f247640 1 -- 172.21.15.59:0/3483406664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff714005210 msgr2=0x7ff71400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.772+0000 7ff72f247640 1 --2- 172.21.15.59:0/3483406664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff714005210 0x7ff71400b9c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff72000c560 tx=0x7ff714005830 comp rx=0 tx=0).stop 2024-06-29T14:38:22.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.772+0000 7ff72d7e4640 1 -- 172.21.15.59:0/3483406664 reap_dead start 2024-06-29T14:38:22.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.772+0000 7ff72f247640 1 -- 172.21.15.59:0/3483406664 shutdown_connections 2024-06-29T14:38:22.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.772+0000 7ff72f247640 1 -- 172.21.15.59:0/3483406664 >> 172.21.15.59:0/3483406664 conn(0x7ff7280b5590 msgr2=0x7ff7280b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.772+0000 7ff72f247640 1 -- 172.21.15.59:0/3483406664 shutdown_connections 2024-06-29T14:38:22.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.772+0000 7ff72f247640 1 -- 172.21.15.59:0/3483406664 wait complete. 2024-06-29T14:38:22.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.774+0000 7ff72f247640 1 Processor -- start 2024-06-29T14:38:22.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.774+0000 7ff72f247640 1 -- start start 2024-06-29T14:38:22.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff72f247640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff714005210 0x7ff7281f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff72f247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7280ce300 0x7ff72806d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff72f247640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7280ce710 0x7ff72806d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff72f247640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff7280b78f0 con 0x7ff7280ce300 2024-06-29T14:38:22.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff72f247640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7280b7770 con 0x7ff7280ce710 2024-06-29T14:38:22.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff72f247640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff7280b4510 con 0x7ff714005210 2024-06-29T14:38:22.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff727fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7280ce300 0x7ff72806d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff727fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7280ce300 0x7ff72806d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53152/0 (socket says 172.21.15.59:53152) 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff727fff640 1 -- 172.21.15.59:0/648498768 learned_addr learned my addr 172.21.15.59:0/648498768 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff72cfe3640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff714005210 0x7ff7281f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.775+0000 7ff72d7e4640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7280ce710 0x7ff72806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff727fff640 1 -- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff714005210 msgr2=0x7ff7281f59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff727fff640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff714005210 0x7ff7281f59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff727fff640 1 -- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7280ce710 msgr2=0x7ff72806d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff727fff640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7280ce710 0x7ff72806d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff727fff640 1 -- 172.21.15.59:0/648498768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7280aa760 con 0x7ff7280ce300 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff727fff640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7280ce300 0x7ff72806d1f0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff7280cf860 tx=0x7ff710011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff7257fa640 1 -- 172.21.15.59:0/648498768 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71001f070 con 0x7ff7280ce300 2024-06-29T14:38:22.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff7257fa640 1 -- 172.21.15.59:0/648498768 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff71000fe40 con 0x7ff7280ce300 2024-06-29T14:38:22.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7280aaa40 con 0x7ff7280ce300 2024-06-29T14:38:22.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff7257fa640 1 -- 172.21.15.59:0/648498768 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7100126b0 con 0x7ff7280ce300 2024-06-29T14:38:22.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.776+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7280ab000 con 0x7ff7280ce300 2024-06-29T14:38:22.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.777+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7280b4a70 con 0x7ff7280ce300 2024-06-29T14:38:22.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.778+0000 7ff7257fa640 1 -- 172.21.15.59:0/648498768 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7ff710022070 con 0x7ff7280ce300 2024-06-29T14:38:22.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.779+0000 7ff7257fa640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7ff7080763f0 0x7ff708078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.779+0000 7ff7257fa640 1 -- 172.21.15.59:0/648498768 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5249+0+0 (secure 0 0 0) 0x7ff7100982e0 con 0x7ff7280ce300 2024-06-29T14:38:22.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.779+0000 7ff72cfe3640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7ff7080763f0 0x7ff708078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.779+0000 7ff72cfe3640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7ff7080763f0 0x7ff708078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff718000f60 tx=0x7ff718000ed0 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.783+0000 7ff7257fa640 1 -- 172.21.15.59:0/648498768 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff710061600 con 0x7ff7280ce300 2024-06-29T14:38:22.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:22.975+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff7280b94e0 con 0x7ff7280ce300 2024-06-29T14:38:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.215+0000 7ff7257fa640 1 -- 172.21.15.59:0/648498768 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v111) ==== 138+0+0 (secure 0 0 0) 0x7ff710065900 con 0x7ff7280ce300 2024-06-29T14:38:23.214 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:38:23.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.217+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7ff7080763f0 msgr2=0x7ff708078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.217+0000 7ff72f247640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7ff7080763f0 0x7ff708078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff718000f60 tx=0x7ff718000ed0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.217+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7280ce300 msgr2=0x7ff72806d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.217+0000 7ff72f247640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7280ce300 0x7ff72806d1f0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff7280cf860 tx=0x7ff710011e10 comp rx=0 tx=0).stop 2024-06-29T14:38:23.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.218+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 shutdown_connections 2024-06-29T14:38:23.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.218+0000 7ff72f247640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7ff7080763f0 0x7ff708078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.218+0000 7ff72f247640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7280ce710 0x7ff72806d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.218+0000 7ff72f247640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7280ce300 0x7ff72806d1f0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.218+0000 7ff72f247640 1 --2- 172.21.15.59:0/648498768 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff714005210 0x7ff7281f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.218+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 >> 172.21.15.59:0/648498768 conn(0x7ff7280b5590 msgr2=0x7ff728160190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:23.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.218+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 shutdown_connections 2024-06-29T14:38:23.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.218+0000 7ff72f247640 1 -- 172.21.15.59:0/648498768 wait complete. 2024-06-29T14:38:23.231 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:38:23.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.357+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/2789922884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00b4c80 msgr2=0x7f8ef00b7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.357+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/2789922884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00b4c80 0x7f8ef00b7060 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f8ef008c7a0 tx=0x7f8ee802aa30 comp rx=0 tx=0).stop 2024-06-29T14:38:23.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.357+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/2789922884 shutdown_connections 2024-06-29T14:38:23.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.357+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/2789922884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00b4c80 0x7f8ef00b7060 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.357+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/2789922884 >> v2:172.21.15.153:3300/0 conn(0x7f8ef00d3260 0x7f8ef015e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.357+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/2789922884 >> 172.21.15.59:0/2789922884 conn(0x7f8ef00d04e0 msgr2=0x7f8ef00d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:23.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.357+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/2789922884 shutdown_connections 2024-06-29T14:38:23.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.357+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/2789922884 wait complete. 2024-06-29T14:38:23.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.359+0000 7f8ef6ff5640 1 Processor -- start 2024-06-29T14:38:23.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.359+0000 7f8ef6ff5640 1 -- start start 2024-06-29T14:38:23.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ef6ff5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ef00b2c40 0x7f8ef00cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:23.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ef6ff5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00d3260 0x7f8ef015cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:23.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ef6ff5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ef015d180 0x7f8ef0151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:23.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ef6ff5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef00b7930 con 0x7f8ef00d3260 2024-06-29T14:38:23.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ef6ff5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef00b77b0 con 0x7f8ef015d180 2024-06-29T14:38:23.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ef6ff5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef00b7ab0 con 0x7f8ef00b2c40 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00d3260 0x7f8ef015cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00d3260 0x7f8ef015cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53166/0 (socket says 172.21.15.59:53166) 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 -- 172.21.15.59:0/853196000 learned_addr learned my addr 172.21.15.59:0/853196000 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ef4d91640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ef00b2c40 0x7f8ef00cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ef5592640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ef015d180 0x7f8ef0151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 -- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ef00b2c40 msgr2=0x7f8ef00cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ef00b2c40 0x7f8ef00cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 -- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ef015d180 msgr2=0x7f8ef0151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ef015d180 0x7f8ef0151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 -- 172.21.15.59:0/853196000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef00cd0b0 con 0x7f8ef00d3260 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee7fff640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00d3260 0x7f8ef015cc40 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009fd0 tx=0x7f8ee000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.361+0000 7f8ee57fa640 1 -- 172.21.15.59:0/853196000 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0015020 con 0x7f8ef00d3260 2024-06-29T14:38:23.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.362+0000 7f8ee57fa640 1 -- 172.21.15.59:0/853196000 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ee0011920 con 0x7f8ef00d3260 2024-06-29T14:38:23.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.362+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef01f4a70 con 0x7f8ef00d3260 2024-06-29T14:38:23.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.362+0000 7f8ee57fa640 1 -- 172.21.15.59:0/853196000 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0011d50 con 0x7f8ef00d3260 2024-06-29T14:38:23.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.362+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef01f4ff0 con 0x7f8ef00d3260 2024-06-29T14:38:23.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.363+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef00b7ff0 con 0x7f8ef00d3260 2024-06-29T14:38:23.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.363+0000 7f8ee57fa640 1 -- 172.21.15.59:0/853196000 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f8ee0018020 con 0x7f8ef00d3260 2024-06-29T14:38:23.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.364+0000 7f8ee57fa640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8edc0763f0 0x7f8edc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:23.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.364+0000 7f8ee57fa640 1 -- 172.21.15.59:0/853196000 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5619+0+0 (secure 0 0 0) 0x7f8ee0097750 con 0x7f8ef00d3260 2024-06-29T14:38:23.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.364+0000 7f8ef4d91640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8edc0763f0 0x7f8edc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:23.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.365+0000 7f8ef4d91640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8edc0763f0 0x7f8edc078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8eec004630 tx=0x7f8eec00b2e0 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:23.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.369+0000 7f8ee57fa640 1 -- 172.21.15.59:0/853196000 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8ee001f020 con 0x7f8ef00d3260 2024-06-29T14:38:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:23.559+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8ef006c8b0 con 0x7f8ef00d3260 2024-06-29T14:38:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.218+0000 7f8ee57fa640 1 -- 172.21.15.59:0/853196000 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v112) ==== 170+0+0 (secure 0 0 0) 0x7f8ef01f4a70 con 0x7f8ef00d3260 2024-06-29T14:38:24.217 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.221+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8edc0763f0 msgr2=0x7f8edc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.221+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8edc0763f0 0x7f8edc078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8eec004630 tx=0x7f8eec00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.221+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00d3260 msgr2=0x7f8ef015cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.221+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00d3260 0x7f8ef015cc40 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009fd0 tx=0x7f8ee000b870 comp rx=0 tx=0).stop 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.222+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 shutdown_connections 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.222+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8edc0763f0 0x7f8edc078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.222+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ef015d180 0x7f8ef0151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.222+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ef00d3260 0x7f8ef015cc40 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.222+0000 7f8ef6ff5640 1 --2- 172.21.15.59:0/853196000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8ef00b2c40 0x7f8ef00cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.222+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 >> 172.21.15.59:0/853196000 conn(0x7f8ef00d04e0 msgr2=0x7f8ef00b8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:24.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.222+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 shutdown_connections 2024-06-29T14:38:24.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.222+0000 7f8ef6ff5640 1 -- 172.21.15.59:0/853196000 wait complete. 2024-06-29T14:38:24.248 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:38:24.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.376+0000 7f8eb95d4640 1 -- 172.21.15.59:0/4011735513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 msgr2=0x7f8eb40cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:24.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.376+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/4011735513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 0x7f8eb40cf080 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4005fd0 tx=0x7f8ea402a5a0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.376+0000 7f8eb95d4640 1 -- 172.21.15.59:0/4011735513 shutdown_connections 2024-06-29T14:38:24.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.376+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/4011735513 >> v2:172.21.15.153:3300/0 conn(0x7f8eb40ce950 0x7f8eb40d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.376+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/4011735513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 0x7f8eb40cf080 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.376+0000 7f8eb95d4640 1 -- 172.21.15.59:0/4011735513 >> 172.21.15.59:0/4011735513 conn(0x7f8eb40b5610 msgr2=0x7f8eb40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:24.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.376+0000 7f8eb95d4640 1 -- 172.21.15.59:0/4011735513 shutdown_connections 2024-06-29T14:38:24.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.376+0000 7f8eb95d4640 1 -- 172.21.15.59:0/4011735513 wait complete. 2024-06-29T14:38:24.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.378+0000 7f8eb95d4640 1 Processor -- start 2024-06-29T14:38:24.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.378+0000 7f8eb95d4640 1 -- start start 2024-06-29T14:38:24.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb95d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 0x7f8eb4158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:24.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb95d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb40ce950 0x7f8eb41f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:24.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb95d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb40cf5c0 0x7f8eb415b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:24.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb95d4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb40b7f60 con 0x7f8eb408c190 2024-06-29T14:38:24.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb95d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb40b7de0 con 0x7f8eb40cf5c0 2024-06-29T14:38:24.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb95d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8eb40b80e0 con 0x7f8eb40ce950 2024-06-29T14:38:24.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 0x7f8eb4158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 0x7f8eb4158890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53190/0 (socket says 172.21.15.59:53190) 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 -- 172.21.15.59:0/3328520992 learned_addr learned my addr 172.21.15.59:0/3328520992 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 -- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb40ce950 msgr2=0x7f8eb41f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb40ce950 0x7f8eb41f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 -- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb40cf5c0 msgr2=0x7f8eb415b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb40cf5c0 0x7f8eb415b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 -- 172.21.15.59:0/3328520992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb4159510 con 0x7f8eb408c190 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.380+0000 7f8eb2ffd640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 0x7f8eb4158890 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4034040 tx=0x7f8ea40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.381+0000 7f8eabfff640 1 -- 172.21.15.59:0/3328520992 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea403e030 con 0x7f8eb408c190 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.381+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb41597f0 con 0x7f8eb408c190 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.381+0000 7f8eabfff640 1 -- 172.21.15.59:0/3328520992 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ea4003d90 con 0x7f8eb408c190 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.381+0000 7f8eabfff640 1 -- 172.21.15.59:0/3328520992 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4035040 con 0x7f8eb408c190 2024-06-29T14:38:24.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.381+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb4159db0 con 0x7f8eb408c190 2024-06-29T14:38:24.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.382+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eb40cf080 con 0x7f8eb408c190 2024-06-29T14:38:24.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.383+0000 7f8eabfff640 1 -- 172.21.15.59:0/3328520992 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f8ea4041030 con 0x7f8eb408c190 2024-06-29T14:38:24.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.383+0000 7f8eabfff640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8e94076420 0x7f8e94078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:24.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.383+0000 7f8eabfff640 1 -- 172.21.15.59:0/3328520992 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 6001+0+0 (secure 0 0 0) 0x7f8ea40b8750 con 0x7f8eb408c190 2024-06-29T14:38:24.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.383+0000 7f8eaa7fc640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8e94076420 0x7f8e94078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:24.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.384+0000 7f8eaa7fc640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8e94076420 0x7f8e94078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8eb415d500 tx=0x7f8e9c007970 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:24.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.387+0000 7f8eabfff640 1 -- 172.21.15.59:0/3328520992 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8ea4081cd0 con 0x7f8eb408c190 2024-06-29T14:38:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:24.620+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f8eb415de70 con 0x7f8eb408c190 2024-06-29T14:38:25.242 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 13 and data pool 14 2024-06-29T14:38:25.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.243+0000 7f8eabfff640 1 -- 172.21.15.59:0/3328520992 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v51) ==== 174+0+0 (secure 0 0 0) 0x7f8ea40815d0 con 0x7f8eb408c190 2024-06-29T14:38:25.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.256+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8e94076420 msgr2=0x7f8e94078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.256+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8e94076420 0x7f8e94078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8eb415d500 tx=0x7f8e9c007970 comp rx=0 tx=0).stop 2024-06-29T14:38:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.256+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 msgr2=0x7f8eb4158890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.256+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 0x7f8eb4158890 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4034040 tx=0x7f8ea40038a0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.257+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 shutdown_connections 2024-06-29T14:38:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.257+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f8e94076420 0x7f8e94078930 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8eb415d500 tx=0x7f8e9c007970 comp rx=0 tx=0).stop 2024-06-29T14:38:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.257+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb40cf5c0 0x7f8eb415b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.257+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb40ce950 0x7f8eb41f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.257+0000 7f8eb95d4640 1 --2- 172.21.15.59:0/3328520992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb408c190 0x7f8eb4158890 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.257+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 >> 172.21.15.59:0/3328520992 conn(0x7f8eb40b5610 msgr2=0x7f8eb40b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.259+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 shutdown_connections 2024-06-29T14:38:25.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.259+0000 7f8eb95d4640 1 -- 172.21.15.59:0/3328520992 wait complete. 2024-06-29T14:38:25.281 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:25.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.406+0000 7f6c2d9be640 1 -- 172.21.15.59:0/4215385850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14003ac0 msgr2=0x7f6c14003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.406+0000 7f6c2d9be640 1 --2- 172.21.15.59:0/4215385850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c14003ac0 0x7f6c14003f30 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f6c18005fd0 tx=0x7f6c14005830 comp rx=0 tx=0).stop 2024-06-29T14:38:25.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.406+0000 7f6c277fe640 1 -- 172.21.15.59:0/4215385850 reap_dead start 2024-06-29T14:38:25.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.406+0000 7f6c2d9be640 1 -- 172.21.15.59:0/4215385850 shutdown_connections 2024-06-29T14:38:25.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.406+0000 7f6c2d9be640 1 -- 172.21.15.59:0/4215385850 >> 172.21.15.59:0/4215385850 conn(0x7f6c280d45b0 msgr2=0x7f6c280d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.406+0000 7f6c2d9be640 1 -- 172.21.15.59:0/4215385850 shutdown_connections 2024-06-29T14:38:25.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.406+0000 7f6c2d9be640 1 -- 172.21.15.59:0/4215385850 wait complete. 2024-06-29T14:38:25.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.408+0000 7f6c2d9be640 1 Processor -- start 2024-06-29T14:38:25.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.408+0000 7f6c2d9be640 1 -- start start 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c2d9be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14003ac0 0x7f6c28160e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c2d9be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c2815b410 0x7f6c2815e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c2d9be640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c280c9ca0 0x7f6c280cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c2d9be640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6c280b38e0 con 0x7f6c2815b410 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c2d9be640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c281623a0 con 0x7f6c14003ac0 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c2d9be640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6c280b3a60 con 0x7f6c280c9ca0 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c1e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c2815b410 0x7f6c2815e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c1e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c2815b410 0x7f6c2815e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53208/0 (socket says 172.21.15.59:53208) 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.409+0000 7f6c1e7fc640 1 -- 172.21.15.59:0/2359693774 learned_addr learned my addr 172.21.15.59:0/2359693774 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c26ffd640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14003ac0 0x7f6c28160e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c277fe640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c280c9ca0 0x7f6c280cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c1e7fc640 1 -- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c280c9ca0 msgr2=0x7f6c280cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c1e7fc640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c280c9ca0 0x7f6c280cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c1e7fc640 1 -- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14003ac0 msgr2=0x7f6c28160e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c1e7fc640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14003ac0 0x7f6c28160e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c1e7fc640 1 -- 172.21.15.59:0/2359693774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2815ea80 con 0x7f6c2815b410 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c1e7fc640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c2815b410 0x7f6c2815e540 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f6c10001160 tx=0x7f6c100039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c24ff9640 1 -- 172.21.15.59:0/2359693774 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10015070 con 0x7f6c2815b410 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c24ff9640 1 -- 172.21.15.59:0/2359693774 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c10011920 con 0x7f6c2815b410 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c280c8c00 con 0x7f6c2815b410 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c24ff9640 1 -- 172.21.15.59:0/2359693774 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10011d70 con 0x7f6c2815b410 2024-06-29T14:38:25.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.410+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c280c91c0 con 0x7f6c2815b410 2024-06-29T14:38:25.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.411+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c280ce550 con 0x7f6c2815b410 2024-06-29T14:38:25.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.412+0000 7f6c24ff9640 1 -- 172.21.15.59:0/2359693774 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f6c10018070 con 0x7f6c2815b410 2024-06-29T14:38:25.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.413+0000 7f6c24ff9640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f6c080763f0 0x7f6c08078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.413+0000 7f6c24ff9640 1 -- 172.21.15.59:0/2359693774 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 6101+0+0 (secure 0 0 0) 0x7f6c100989e0 con 0x7f6c2815b410 2024-06-29T14:38:25.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.413+0000 7f6c26ffd640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f6c080763f0 0x7f6c08078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.413+0000 7f6c26ffd640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f6c080763f0 0x7f6c08078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6c18036000 tx=0x7f6c180046c0 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.417+0000 7f6c24ff9640 1 -- 172.21.15.59:0/2359693774 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6c10061f00 con 0x7f6c2815b410 2024-06-29T14:38:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.613+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c280c8990 con 0x7f6c2815b410 2024-06-29T14:38:25.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.614+0000 7f6c24ff9640 1 -- 172.21.15.59:0/2359693774 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+16921 (secure 0 0 0) 0x7f6c10061800 con 0x7f6c2815b410 2024-06-29T14:38:25.613 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:25.613 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":113,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:24.621757+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.981044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.565073+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f6c080763f0 msgr2=0x7f6c08078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f6c080763f0 0x7f6c08078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6c18036000 tx=0x7f6c180046c0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c2815b410 msgr2=0x7f6c2815e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c2815b410 0x7f6c2815e540 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f6c10001160 tx=0x7f6c100039c0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 shutdown_connections 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f6c080763f0 0x7f6c08078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c280c9ca0 0x7f6c280cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c2815b410 0x7f6c2815e540 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 --2- 172.21.15.59:0/2359693774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c14003ac0 0x7f6c28160e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.616+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 >> 172.21.15.59:0/2359693774 conn(0x7f6c280d45b0 msgr2=0x7f6c2815bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.617+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 shutdown_connections 2024-06-29T14:38:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.617+0000 7f6c2d9be640 1 -- 172.21.15.59:0/2359693774 wait complete. 2024-06-29T14:38:25.626 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.746+0000 7f9818511640 1 -- 172.21.15.59:0/3886075745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98100ce950 msgr2=0x7f98101622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.746+0000 7f9818511640 1 --2- 172.21.15.59:0/3886075745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98100ce950 0x7f98101622c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f98100b78c0 tx=0x7f980002a540 comp rx=0 tx=0).stop 2024-06-29T14:38:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.746+0000 7f9818511640 1 -- 172.21.15.59:0/3886075745 shutdown_connections 2024-06-29T14:38:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.746+0000 7f9818511640 1 --2- 172.21.15.59:0/3886075745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98100ce950 0x7f98101622c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.746+0000 7f9818511640 1 -- 172.21.15.59:0/3886075745 >> 172.21.15.59:0/3886075745 conn(0x7f98100b5610 msgr2=0x7f98100b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.746+0000 7f9818511640 1 -- 172.21.15.59:0/3886075745 shutdown_connections 2024-06-29T14:38:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.746+0000 7f9818511640 1 -- 172.21.15.59:0/3886075745 wait complete. 2024-06-29T14:38:25.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.748+0000 7f9818511640 1 Processor -- start 2024-06-29T14:38:25.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.748+0000 7f9818511640 1 -- start start 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9818511640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f981008c190 0x7f98101f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9818511640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98100ce950 0x7f9810153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9818511640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810162800 0x7f981015a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9818511640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f98100b82e0 con 0x7f9810162800 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9818511640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9810162fe0 con 0x7f98100ce950 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9818511640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f98100b8460 con 0x7f981008c190 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9816aae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810162800 0x7f981015a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9816aae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810162800 0x7f981015a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53214/0 (socket says 172.21.15.59:53214) 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.749+0000 7f9816aae640 1 -- 172.21.15.59:0/3091115586 learned_addr learned my addr 172.21.15.59:0/3091115586 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9815aac640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98100ce950 0x7f9810153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f98162ad640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f981008c190 0x7f98101f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9816aae640 1 -- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f981008c190 msgr2=0x7f98101f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9816aae640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f981008c190 0x7f98101f5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9816aae640 1 -- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98100ce950 msgr2=0x7f9810153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9816aae640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98100ce950 0x7f9810153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9816aae640 1 -- 172.21.15.59:0/3091115586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98100cac70 con 0x7f9810162800 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9816aae640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810162800 0x7f981015a9a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f980800c700 tx=0x7f980800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9806ffd640 1 -- 172.21.15.59:0/3091115586 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9808003b70 con 0x7f9810162800 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9806ffd640 1 -- 172.21.15.59:0/3091115586 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9808003d10 con 0x7f9810162800 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98100caf10 con 0x7f9810162800 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9806ffd640 1 -- 172.21.15.59:0/3091115586 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980800aa50 con 0x7f9810162800 2024-06-29T14:38:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.750+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98100cd6c0 con 0x7f9810162800 2024-06-29T14:38:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.751+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98100cd430 con 0x7f9810162800 2024-06-29T14:38:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.752+0000 7f9806ffd640 1 -- 172.21.15.59:0/3091115586 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f980800abf0 con 0x7f9810162800 2024-06-29T14:38:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.753+0000 7f9806ffd640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f97f40763f0 0x7f97f4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.753+0000 7f9806ffd640 1 -- 172.21.15.59:0/3091115586 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 6101+0+0 (secure 0 0 0) 0x7f98080977d0 con 0x7f9810162800 2024-06-29T14:38:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.753+0000 7f98162ad640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f97f40763f0 0x7f97f4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.754+0000 7f98162ad640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f97f40763f0 0x7f97f4078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f980c007fd0 tx=0x7f980c007eb0 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.757+0000 7f9806ffd640 1 -- 172.21.15.59:0/3091115586 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f98080607b0 con 0x7f9810162800 2024-06-29T14:38:25.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.949+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98101622c0 con 0x7f9810162800 2024-06-29T14:38:25.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.950+0000 7f9806ffd640 1 -- 172.21.15.59:0/3091115586 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+16921 (secure 0 0 0) 0x7f9808064ab0 con 0x7f9810162800 2024-06-29T14:38:25.948 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:25.948 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":113,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:24.621757+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.981044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.565073+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f97f40763f0 msgr2=0x7f97f4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f97f40763f0 0x7f97f4078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f980c007fd0 tx=0x7f980c007eb0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810162800 msgr2=0x7f981015a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810162800 0x7f981015a9a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f980800c700 tx=0x7f980800cbc0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 shutdown_connections 2024-06-29T14:38:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f97f40763f0 0x7f97f4078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9810162800 0x7f981015a9a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98100ce950 0x7f9810153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 --2- 172.21.15.59:0/3091115586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f981008c190 0x7f98101f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 >> 172.21.15.59:0/3091115586 conn(0x7f98100b5610 msgr2=0x7f981006c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 shutdown_connections 2024-06-29T14:38:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:25.952+0000 7f9818511640 1 -- 172.21.15.59:0/3091115586 wait complete. 2024-06-29T14:38:25.962 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:38:26.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.084+0000 7f40b668f640 1 -- 172.21.15.59:0/2184128996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4098005270 msgr2=0x7f409800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:26.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.084+0000 7f40b668f640 1 --2- 172.21.15.59:0/2184128996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4098005270 0x7f409800b9c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f40a800c640 tx=0x7f4098005830 comp rx=0 tx=0).stop 2024-06-29T14:38:26.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.084+0000 7f40b668f640 1 -- 172.21.15.59:0/2184128996 shutdown_connections 2024-06-29T14:38:26.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.084+0000 7f40b668f640 1 --2- 172.21.15.59:0/2184128996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4098005270 0x7f409800b9c0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.084+0000 7f40b668f640 1 -- 172.21.15.59:0/2184128996 >> 172.21.15.59:0/2184128996 conn(0x7f40b00d3ec0 msgr2=0x7f40b00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:26.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.085+0000 7f40b668f640 1 -- 172.21.15.59:0/2184128996 shutdown_connections 2024-06-29T14:38:26.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.085+0000 7f40b668f640 1 -- 172.21.15.59:0/2184128996 wait complete. 2024-06-29T14:38:26.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.086+0000 7f40b668f640 1 Processor -- start 2024-06-29T14:38:26.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.087+0000 7f40b668f640 1 -- start start 2024-06-29T14:38:26.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40b668f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4098005270 0x7f40b01e53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:26.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40b668f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40b0158c60 0x7f40b01e5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:26.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40b668f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b00cd590 0x7f40b00ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:26.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40b668f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f40b00b2d50 con 0x7f40b0158c60 2024-06-29T14:38:26.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40b668f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f40b00b2bd0 con 0x7f4098005270 2024-06-29T14:38:26.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40b668f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f40b00b2ed0 con 0x7f40b00cd590 2024-06-29T14:38:26.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40b0158c60 0x7f40b01e5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40b0158c60 0x7f40b01e5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53226/0 (socket says 172.21.15.59:53226) 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 -- 172.21.15.59:0/439079896 learned_addr learned my addr 172.21.15.59:0/439079896 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40b4c2c640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b00cd590 0x7f40b00ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40affff640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4098005270 0x7f40b01e53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 -- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b00cd590 msgr2=0x7f40b00ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b00cd590 0x7f40b00ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 -- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4098005270 msgr2=0x7f40b01e53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4098005270 0x7f40b01e53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 -- 172.21.15.59:0/439079896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b01f4350 con 0x7f40b0158c60 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.088+0000 7f40af7fe640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40b0158c60 0x7f40b01e5900 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f40a4011ab0 tx=0x7f40a4011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.089+0000 7f40acff9640 1 -- 172.21.15.59:0/439079896 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a401e070 con 0x7f40b0158c60 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.089+0000 7f40acff9640 1 -- 172.21.15.59:0/439079896 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40a400fe70 con 0x7f40b0158c60 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.089+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b00c7550 con 0x7f40b0158c60 2024-06-29T14:38:26.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.089+0000 7f40acff9640 1 -- 172.21.15.59:0/439079896 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a40042b0 con 0x7f40b0158c60 2024-06-29T14:38:26.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.089+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b00c7d30 con 0x7f40b0158c60 2024-06-29T14:38:26.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.090+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b00c77c0 con 0x7f40b0158c60 2024-06-29T14:38:26.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.091+0000 7f40acff9640 1 -- 172.21.15.59:0/439079896 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f40a4021070 con 0x7f40b0158c60 2024-06-29T14:38:26.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.091+0000 7f40acff9640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f409c076420 0x7f409c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:26.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.091+0000 7f40acff9640 1 -- 172.21.15.59:0/439079896 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 6101+0+0 (secure 0 0 0) 0x7f40a40985c0 con 0x7f40b0158c60 2024-06-29T14:38:26.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.091+0000 7f40affff640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f409c076420 0x7f409c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:26.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.092+0000 7f40affff640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f409c076420 0x7f409c078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006fd0 tx=0x7f40a0006b10 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:26.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.095+0000 7f40acff9640 1 -- 172.21.15.59:0/439079896 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f40a40614a0 con 0x7f40b0158c60 2024-06-29T14:38:26.180 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:38:26.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:38:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:26.328+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f40b00b92b0 con 0x7f40b0158c60 2024-06-29T14:38:27.212 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T14:38:27.214+0000 7f188fbfc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:38:27.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.265+0000 7f40acff9640 1 -- 172.21.15.59:0/439079896 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) ==== 118+0+0 (secure 0 0 0) 0x7f40a40657a0 con 0x7f40b0158c60 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.268+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f409c076420 msgr2=0x7f409c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.268+0000 7f40b668f640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f409c076420 0x7f409c078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006fd0 tx=0x7f40a0006b10 comp rx=0 tx=0).stop 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.268+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40b0158c60 msgr2=0x7f40b01e5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.268+0000 7f40b668f640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40b0158c60 0x7f40b01e5900 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f40a4011ab0 tx=0x7f40a4011f70 comp rx=0 tx=0).stop 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.268+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 shutdown_connections 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.268+0000 7f40b668f640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f409c076420 0x7f409c078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.268+0000 7f40b668f640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40b00cd590 0x7f40b00ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.269+0000 7f40b668f640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40b0158c60 0x7f40b01e5900 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.269+0000 7f40b668f640 1 --2- 172.21.15.59:0/439079896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4098005270 0x7f40b01e53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.269+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 >> 172.21.15.59:0/439079896 conn(0x7f40b00d3ec0 msgr2=0x7f40b00b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.269+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 shutdown_connections 2024-06-29T14:38:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.269+0000 7f40b668f640 1 -- 172.21.15.59:0/439079896 wait complete. 2024-06-29T14:38:27.280 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:38:27.280 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:38:27.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.410+0000 7fc93359e640 1 -- 172.21.15.59:0/1248358383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc93406d1f0 msgr2=0x7fc93406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.410+0000 7fc93359e640 1 --2- 172.21.15.59:0/1248358383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc93406d1f0 0x7fc93406d600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc928009830 tx=0x7fc92802a510 comp rx=0 tx=0).stop 2024-06-29T14:38:27.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.411+0000 7fc93359e640 1 -- 172.21.15.59:0/1248358383 shutdown_connections 2024-06-29T14:38:27.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.411+0000 7fc93359e640 1 --2- 172.21.15.59:0/1248358383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc93406d1f0 0x7fc93406d600 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.411+0000 7fc93359e640 1 --2- 172.21.15.59:0/1248358383 >> v2:172.21.15.153:3300/0 conn(0x7fc9340ce070 0x7fc93415e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.411+0000 7fc93359e640 1 -- 172.21.15.59:0/1248358383 >> 172.21.15.59:0/1248358383 conn(0x7fc9340d0950 msgr2=0x7fc9340b5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:27.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.411+0000 7fc93359e640 1 -- 172.21.15.59:0/1248358383 shutdown_connections 2024-06-29T14:38:27.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.411+0000 7fc93359e640 1 -- 172.21.15.59:0/1248358383 wait complete. 2024-06-29T14:38:27.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.413+0000 7fc93359e640 1 Processor -- start 2024-06-29T14:38:27.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.413+0000 7fc93359e640 1 -- start start 2024-06-29T14:38:27.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc93359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc93406d1f0 0x7fc934148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:27.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc93359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9340ce070 0x7fc9340c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:27.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc93359e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9341618a0 0x7fc9340c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc93359e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc9340d2950 con 0x7fc9340ce070 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc93359e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc9340d27d0 con 0x7fc93406d1f0 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc93359e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc9340d2ad0 con 0x7fc9341618a0 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc931d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9340ce070 0x7fc9340c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc931d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9340ce070 0x7fc9340c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53228/0 (socket says 172.21.15.59:53228) 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc931d9b640 1 -- 172.21.15.59:0/2960771145 learned_addr learned my addr 172.21.15.59:0/2960771145 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc93259c640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc93406d1f0 0x7fc934148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.414+0000 7fc932d9d640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9341618a0 0x7fc9340c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc931d9b640 1 -- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9341618a0 msgr2=0x7fc9340c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc931d9b640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9341618a0 0x7fc9340c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc931d9b640 1 -- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc93406d1f0 msgr2=0x7fc934148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc931d9b640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc93406d1f0 0x7fc934148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc931d9b640 1 -- 172.21.15.59:0/2960771145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9340c8a40 con 0x7fc9340ce070 2024-06-29T14:38:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc931d9b640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9340ce070 0x7fc9340c7d20 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fc92802ab00 tx=0x7fc928033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc922ffd640 1 -- 172.21.15.59:0/2960771145 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc928005c90 con 0x7fc9340ce070 2024-06-29T14:38:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9340cbb40 con 0x7fc9340ce070 2024-06-29T14:38:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc922ffd640 1 -- 172.21.15.59:0/2960771145 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc928031e50 con 0x7fc9340ce070 2024-06-29T14:38:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc922ffd640 1 -- 172.21.15.59:0/2960771145 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc928005260 con 0x7fc9340ce070 2024-06-29T14:38:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.415+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9341530b0 con 0x7fc9340ce070 2024-06-29T14:38:27.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.416+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9340cb930 con 0x7fc9340ce070 2024-06-29T14:38:27.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.417+0000 7fc922ffd640 1 -- 172.21.15.59:0/2960771145 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fc928005400 con 0x7fc9340ce070 2024-06-29T14:38:27.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.417+0000 7fc922ffd640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc9140763f0 0x7fc914078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:27.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.417+0000 7fc922ffd640 1 -- 172.21.15.59:0/2960771145 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7fc9280b8ba0 con 0x7fc9340ce070 2024-06-29T14:38:27.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.418+0000 7fc93259c640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc9140763f0 0x7fc914078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:27.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.418+0000 7fc93259c640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc9140763f0 0x7fc914078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc9240099a0 tx=0x7fc924009500 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.422+0000 7fc922ffd640 1 -- 172.21.15.59:0/2960771145 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc92803f030 con 0x7fc9340ce070 2024-06-29T14:38:27.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:27.657+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fc93406d600 con 0x7fc9340ce070 2024-06-29T14:38:28.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.267+0000 7fc922ffd640 1 -- 172.21.15.59:0/2960771145 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v55) ==== 122+0+0 (secure 0 0 0) 0x7fc9280819f0 con 0x7fc9340ce070 2024-06-29T14:38:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.270+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc9140763f0 msgr2=0x7fc914078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.270+0000 7fc93359e640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc9140763f0 0x7fc914078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc9240099a0 tx=0x7fc924009500 comp rx=0 tx=0).stop 2024-06-29T14:38:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.270+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9340ce070 msgr2=0x7fc9340c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.270+0000 7fc93359e640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9340ce070 0x7fc9340c7d20 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fc92802ab00 tx=0x7fc928033c20 comp rx=0 tx=0).stop 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.270+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 shutdown_connections 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.271+0000 7fc93359e640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc9140763f0 0x7fc914078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.271+0000 7fc93359e640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc9341618a0 0x7fc9340c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.271+0000 7fc93359e640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9340ce070 0x7fc9340c7d20 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.271+0000 7fc93359e640 1 --2- 172.21.15.59:0/2960771145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc93406d1f0 0x7fc934148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.271+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 >> 172.21.15.59:0/2960771145 conn(0x7fc9340d0950 msgr2=0x7fc93415fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.271+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 shutdown_connections 2024-06-29T14:38:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.271+0000 7fc93359e640 1 -- 172.21.15.59:0/2960771145 wait complete. 2024-06-29T14:38:28.283 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:38:28.283 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.409+0000 7f835cb11640 1 -- 172.21.15.59:0/544818317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83581604d0 msgr2=0x7f835813ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.409+0000 7f835cb11640 1 --2- 172.21.15.59:0/544818317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83581604d0 0x7f835813ebd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f8350009830 tx=0x7f835002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.409+0000 7f835cb11640 1 -- 172.21.15.59:0/544818317 shutdown_connections 2024-06-29T14:38:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.409+0000 7f835cb11640 1 --2- 172.21.15.59:0/544818317 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83581604d0 0x7f835813ebd0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.409+0000 7f835cb11640 1 -- 172.21.15.59:0/544818317 >> 172.21.15.59:0/544818317 conn(0x7f83580b06e0 msgr2=0x7f83580b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.409+0000 7f835cb11640 1 -- 172.21.15.59:0/544818317 shutdown_connections 2024-06-29T14:38:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.409+0000 7f835cb11640 1 -- 172.21.15.59:0/544818317 wait complete. 2024-06-29T14:38:28.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.411+0000 7f835cb11640 1 Processor -- start 2024-06-29T14:38:28.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.411+0000 7f835cb11640 1 -- start start 2024-06-29T14:38:28.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835cb11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83580b1b00 0x7f835813e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835cb11640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358141360 0x7f83580b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835cb11640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83581604d0 0x7f83580b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835cb11640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8358167d70 con 0x7f83580b1b00 2024-06-29T14:38:28.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835cb11640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8358167bf0 con 0x7f83581604d0 2024-06-29T14:38:28.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835cb11640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8358167ef0 con 0x7f8358141360 2024-06-29T14:38:28.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83580b1b00 0x7f835813e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83580b1b00 0x7f835813e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53248/0 (socket says 172.21.15.59:53248) 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 -- 172.21.15.59:0/1232876796 learned_addr learned my addr 172.21.15.59:0/1232876796 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f8355d9b640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358141360 0x7f83580b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f8356d9d640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83581604d0 0x7f83580b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 -- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358141360 msgr2=0x7f83580b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358141360 0x7f83580b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 -- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83581604d0 msgr2=0x7f83580b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83581604d0 0x7f83580b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 -- 172.21.15.59:0/1232876796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8358156ac0 con 0x7f83580b1b00 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835659c640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83580b1b00 0x7f835813e510 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f835002abb0 tx=0x7f8350033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f834effd640 1 -- 172.21.15.59:0/1232876796 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350031b50 con 0x7f83580b1b00 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f834effd640 1 -- 172.21.15.59:0/1232876796 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8350031de0 con 0x7f83580b1b00 2024-06-29T14:38:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8358156da0 con 0x7f83580b1b00 2024-06-29T14:38:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.413+0000 7f834effd640 1 -- 172.21.15.59:0/1232876796 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350005280 con 0x7f83580b1b00 2024-06-29T14:38:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.414+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8358157360 con 0x7f83580b1b00 2024-06-29T14:38:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.414+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f835815ebe0 con 0x7f83580b1b00 2024-06-29T14:38:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.415+0000 7f834effd640 1 -- 172.21.15.59:0/1232876796 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f8350005420 con 0x7f83580b1b00 2024-06-29T14:38:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.416+0000 7f834effd640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f83380763f0 0x7f8338078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.416+0000 7f834effd640 1 -- 172.21.15.59:0/1232876796 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f83500b8b50 con 0x7f83580b1b00 2024-06-29T14:38:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.416+0000 7f8355d9b640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f83380763f0 0x7f8338078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.417+0000 7f8355d9b640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f83380763f0 0x7f8338078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f83581599b0 tx=0x7f8340009040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.420+0000 7f834effd640 1 -- 172.21.15.59:0/1232876796 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f835003f030 con 0x7f83580b1b00 2024-06-29T14:38:28.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.654+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f835815e900 con 0x7f83580b1b00 2024-06-29T14:38:28.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.655+0000 7f834effd640 1 -- 172.21.15.59:0/1232876796 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5225 (secure 0 0 0) 0x7f8350082100 con 0x7f83580b1b00 2024-06-29T14:38:28.653 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:28.653 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"btime":"2024-06-29T14:38:28:267220+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3208152037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3208152037},{"type":"v1","addr":"172.21.15.153:6836","nonce":3208152037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6716,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/590918917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":590918917},{"type":"v1","addr":"172.21.15.153:6837","nonce":590918917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/546301975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":546301975},{"type":"v1","addr":"172.21.15.59:6833","nonce":546301975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.621714+0000","modified":"2024-06-29T14:38:27.659142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/2011834805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2011834805},{"type":"v1","addr":"172.21.15.59:6835","nonce":2011834805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":6}]} 2024-06-29T14:38:28.653 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 55 2024-06-29T14:38:28.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f83380763f0 msgr2=0x7f8338078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f83380763f0 0x7f8338078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f83581599b0 tx=0x7f8340009040 comp rx=0 tx=0).stop 2024-06-29T14:38:28.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83580b1b00 msgr2=0x7f835813e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83580b1b00 0x7f835813e510 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f835002abb0 tx=0x7f8350033ba0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 shutdown_connections 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f83380763f0 0x7f8338078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83581604d0 0x7f83580b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8358141360 0x7f83580b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 --2- 172.21.15.59:0/1232876796 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83580b1b00 0x7f835813e510 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 >> 172.21.15.59:0/1232876796 conn(0x7f83580b06e0 msgr2=0x7f835815af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 shutdown_connections 2024-06-29T14:38:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.657+0000 7f835cb11640 1 -- 172.21.15.59:0/1232876796 wait complete. 2024-06-29T14:38:28.666 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:28.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.786+0000 7f031b7e0640 1 -- 172.21.15.59:0/1992476499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 msgr2=0x7f030000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.786+0000 7f031b7e0640 1 --2- 172.21.15.59:0/1992476499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 0x7f030000ba20 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f030c005fd0 tx=0x7f0300005830 comp rx=0 tx=0).stop 2024-06-29T14:38:28.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.786+0000 7f031b7e0640 1 -- 172.21.15.59:0/1992476499 shutdown_connections 2024-06-29T14:38:28.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.786+0000 7f031b7e0640 1 --2- 172.21.15.59:0/1992476499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 0x7f030000ba20 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.787+0000 7f031b7e0640 1 -- 172.21.15.59:0/1992476499 >> 172.21.15.59:0/1992476499 conn(0x7f03140d45b0 msgr2=0x7f03140d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.787+0000 7f031b7e0640 1 -- 172.21.15.59:0/1992476499 shutdown_connections 2024-06-29T14:38:28.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.787+0000 7f031b7e0640 1 -- 172.21.15.59:0/1992476499 wait complete. 2024-06-29T14:38:28.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.788+0000 7f031b7e0640 1 Processor -- start 2024-06-29T14:38:28.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.788+0000 7f031b7e0640 1 -- start start 2024-06-29T14:38:28.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031b7e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 0x7f03140cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031b7e0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f031415b290 0x7f03140cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031b7e0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0314160c20 0x7f0314161030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031b7e0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0314161a80 con 0x7f03000053a0 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031b7e0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03140ce7b0 con 0x7f031415b290 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031b7e0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0314161c00 con 0x7f0314160c20 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 0x7f03140cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 0x7f03140cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53254/0 (socket says 172.21.15.59:53254) 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 -- 172.21.15.59:0/1414863100 learned_addr learned my addr 172.21.15.59:0/1414863100 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f0319d7d640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0314160c20 0x7f0314161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f0318d7b640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f031415b290 0x7f03140cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 -- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0314160c20 msgr2=0x7f0314161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0314160c20 0x7f0314161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 -- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f031415b290 msgr2=0x7f03140cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f031415b290 0x7f03140cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 -- 172.21.15.59:0/1414863100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0314161570 con 0x7f03000053a0 2024-06-29T14:38:28.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f0319d7d640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0314160c20 0x7f0314161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:28.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f0318d7b640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f031415b290 0x7f03140cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:28.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f031957c640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 0x7f03140cc6d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0310004e50 tx=0x7f0310001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.790+0000 7f0309ffb640 1 -- 172.21.15.59:0/1414863100 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03100107d0 con 0x7f03000053a0 2024-06-29T14:38:28.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.791+0000 7f0309ffb640 1 -- 172.21.15.59:0/1414863100 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0310016070 con 0x7f03000053a0 2024-06-29T14:38:28.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.791+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03140cb630 con 0x7f03000053a0 2024-06-29T14:38:28.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.791+0000 7f0309ffb640 1 -- 172.21.15.59:0/1414863100 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310010cc0 con 0x7f03000053a0 2024-06-29T14:38:28.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.791+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03140cbbf0 con 0x7f03000053a0 2024-06-29T14:38:28.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.792+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f031415c650 con 0x7f03000053a0 2024-06-29T14:38:28.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.792+0000 7f0309ffb640 1 -- 172.21.15.59:0/1414863100 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f031001b460 con 0x7f03000053a0 2024-06-29T14:38:28.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.793+0000 7f0309ffb640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f02f80763f0 0x7f02f8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.793+0000 7f0309ffb640 1 -- 172.21.15.59:0/1414863100 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f03100983b0 con 0x7f03000053a0 2024-06-29T14:38:28.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.793+0000 7f0318d7b640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f02f80763f0 0x7f02f8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.794+0000 7f0318d7b640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f02f80763f0 0x7f02f8078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0304000fd0 tx=0x7f0304001c90 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.797+0000 7f0309ffb640 1 -- 172.21.15.59:0/1414863100 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f031415c650 con 0x7f03000053a0 2024-06-29T14:38:28.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.988+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0314162160 con 0x7f03000053a0 2024-06-29T14:38:28.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.989+0000 7f0309ffb640 1 -- 172.21.15.59:0/1414863100 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+16908 (secure 0 0 0) 0x7f0314161570 con 0x7f03000053a0 2024-06-29T14:38:28.988 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:28.988 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":114,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:38:26.244498+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.981044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.565073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:28.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f02f80763f0 msgr2=0x7f02f8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f02f80763f0 0x7f02f8078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0304000fd0 tx=0x7f0304001c90 comp rx=0 tx=0).stop 2024-06-29T14:38:28.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 msgr2=0x7f03140cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 0x7f03140cc6d0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0310004e50 tx=0x7f0310001630 comp rx=0 tx=0).stop 2024-06-29T14:38:28.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 shutdown_connections 2024-06-29T14:38:28.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f02f80763f0 0x7f02f8078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0314160c20 0x7f0314161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f031415b290 0x7f03140cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 --2- 172.21.15.59:0/1414863100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03000053a0 0x7f03140cc6d0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 >> 172.21.15.59:0/1414863100 conn(0x7f03140d45b0 msgr2=0x7f03140b78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 shutdown_connections 2024-06-29T14:38:28.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:28.992+0000 7f031b7e0640 1 -- 172.21.15.59:0/1414863100 wait complete. 2024-06-29T14:38:29.001 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:38:29.002 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:38:29.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.122+0000 7f4c5be81640 1 -- 172.21.15.59:0/1847820423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 msgr2=0x7f4c4000ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.122+0000 7f4c5be81640 1 --2- 172.21.15.59:0/1847820423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 0x7f4c4000ba50 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c001f00 tx=0x7f4c40005830 comp rx=0 tx=0).stop 2024-06-29T14:38:29.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.122+0000 7f4c5be81640 1 -- 172.21.15.59:0/1847820423 shutdown_connections 2024-06-29T14:38:29.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.122+0000 7f4c5be81640 1 --2- 172.21.15.59:0/1847820423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 0x7f4c4000ba50 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.122+0000 7f4c5be81640 1 -- 172.21.15.59:0/1847820423 >> 172.21.15.59:0/1847820423 conn(0x7f4c540b5610 msgr2=0x7f4c540b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.122+0000 7f4c5be81640 1 -- 172.21.15.59:0/1847820423 shutdown_connections 2024-06-29T14:38:29.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.122+0000 7f4c5be81640 1 -- 172.21.15.59:0/1847820423 wait complete. 2024-06-29T14:38:29.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.124+0000 7f4c5be81640 1 Processor -- start 2024-06-29T14:38:29.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.124+0000 7f4c5be81640 1 -- start start 2024-06-29T14:38:29.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c5be81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 0x7f4c5415b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c5be81640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c541577b0 0x7f4c541f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c5be81640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c54156170 0x7f4c54158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c5be81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4c540d1a20 con 0x7f4c400053a0 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c5be81640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c540cf3a0 con 0x7f4c541577b0 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c5be81640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4c540d1ba0 con 0x7f4c54156170 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c59c1d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 0x7f4c5415b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c59c1d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 0x7f4c5415b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53278/0 (socket says 172.21.15.59:53278) 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c59c1d640 1 -- 172.21.15.59:0/1697028190 learned_addr learned my addr 172.21.15.59:0/1697028190 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c5a41e640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c54156170 0x7f4c54158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.125+0000 7f4c5941c640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c541577b0 0x7f4c541f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c59c1d640 1 -- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c54156170 msgr2=0x7f4c54158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c59c1d640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c54156170 0x7f4c54158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c59c1d640 1 -- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c541577b0 msgr2=0x7f4c541f5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c59c1d640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c541577b0 0x7f4c541f5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c59c1d640 1 -- 172.21.15.59:0/1697028190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c541f5b50 con 0x7f4c400053a0 2024-06-29T14:38:29.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c59c1d640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 0x7f4c5415b3c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4c50011ab0 tx=0x7f4c50011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c4a7fc640 1 -- 172.21.15.59:0/1697028190 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5001e070 con 0x7f4c400053a0 2024-06-29T14:38:29.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c4a7fc640 1 -- 172.21.15.59:0/1697028190 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c5000fe70 con 0x7f4c400053a0 2024-06-29T14:38:29.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c54156710 con 0x7f4c400053a0 2024-06-29T14:38:29.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c4a7fc640 1 -- 172.21.15.59:0/1697028190 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c500042b0 con 0x7f4c400053a0 2024-06-29T14:38:29.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.126+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c54156cd0 con 0x7f4c400053a0 2024-06-29T14:38:29.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.127+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c540d2100 con 0x7f4c400053a0 2024-06-29T14:38:29.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.128+0000 7f4c4a7fc640 1 -- 172.21.15.59:0/1697028190 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f4c50021070 con 0x7f4c400053a0 2024-06-29T14:38:29.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.128+0000 7f4c4a7fc640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c380763f0 0x7f4c38078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.129+0000 7f4c4a7fc640 1 -- 172.21.15.59:0/1697028190 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f4c50098540 con 0x7f4c400053a0 2024-06-29T14:38:29.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.129+0000 7f4c5941c640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c380763f0 0x7f4c38078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.129+0000 7f4c5941c640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c380763f0 0x7f4c38078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4c54156100 tx=0x7f4c44000f80 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.133+0000 7f4c4a7fc640 1 -- 172.21.15.59:0/1697028190 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4c541f5b50 con 0x7f4c400053a0 2024-06-29T14:38:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.321+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 --> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4c5415b900 con 0x7f4c380763f0 2024-06-29T14:38:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.327+0000 7f4c4a7fc640 1 -- 172.21.15.59:0/1697028190 <== mgr.7132 v2:172.21.15.153:6832/1090607011 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252448 (secure 0 0 0) 0x7f4c5415b900 con 0x7f4c380763f0 2024-06-29T14:38:29.327 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:29.329 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:38:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c380763f0 msgr2=0x7f4c38078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c380763f0 0x7f4c38078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4c54156100 tx=0x7f4c44000f80 comp rx=0 tx=0).stop 2024-06-29T14:38:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 msgr2=0x7f4c5415b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 0x7f4c5415b3c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4c50011ab0 tx=0x7f4c50011f70 comp rx=0 tx=0).stop 2024-06-29T14:38:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 shutdown_connections 2024-06-29T14:38:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f4c380763f0 0x7f4c38078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c54156170 0x7f4c54158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c541577b0 0x7f4c541f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 --2- 172.21.15.59:0/1697028190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c400053a0 0x7f4c5415b3c0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 >> 172.21.15.59:0/1697028190 conn(0x7f4c540b5610 msgr2=0x7f4c540c94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 shutdown_connections 2024-06-29T14:38:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.329+0000 7f4c5be81640 1 -- 172.21.15.59:0/1697028190 wait complete. 2024-06-29T14:38:29.339 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:38:28.740640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":242,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167,"ondisk_log_size":167,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":344820,"kb_used_data":12088,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754629900,"statfs":{"total":773094113280,"available":772741017600,"internally_reserved":0,"allocated":12378112,"data_stored":6793348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":340577842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.522227"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.505064+0000","last_change":"2024-06-29T14:38:25.251254+0000","last_active":"2024-06-29T14:38:26.505064+0000","last_peered":"2024-06-29T14:38:26.505064+0000","last_clean":"2024-06-29T14:38:26.505064+0000","last_became_active":"2024-06-29T14:38:25.250919+0000","last_became_peered":"2024-06-29T14:38:25.250919+0000","last_unstale":"2024-06-29T14:38:26.505064+0000","last_undegraded":"2024-06-29T14:38:26.505064+0000","last_fullsized":"2024-06-29T14:38:26.505064+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:50:52.656540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258974+0000","last_change":"2024-06-29T14:38:25.260653+0000","last_active":"2024-06-29T14:38:26.258974+0000","last_peered":"2024-06-29T14:38:26.258974+0000","last_clean":"2024-06-29T14:38:26.258974+0000","last_became_active":"2024-06-29T14:38:25.260350+0000","last_became_peered":"2024-06-29T14:38:25.260350+0000","last_unstale":"2024-06-29T14:38:26.258974+0000","last_undegraded":"2024-06-29T14:38:26.258974+0000","last_fullsized":"2024-06-29T14:38:26.258974+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:46:11.716533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504872+0000","last_change":"2024-06-29T14:38:25.264427+0000","last_active":"2024-06-29T14:38:26.504872+0000","last_peered":"2024-06-29T14:38:26.504872+0000","last_clean":"2024-06-29T14:38:26.504872+0000","last_became_active":"2024-06-29T14:38:25.264072+0000","last_became_peered":"2024-06-29T14:38:25.264072+0000","last_unstale":"2024-06-29T14:38:26.504872+0000","last_undegraded":"2024-06-29T14:38:26.504872+0000","last_fullsized":"2024-06-29T14:38:26.504872+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:36.334047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266296+0000","last_change":"2024-06-29T14:38:25.263449+0000","last_active":"2024-06-29T14:38:25.266296+0000","last_peered":"2024-06-29T14:38:25.266296+0000","last_clean":"2024-06-29T14:38:25.266296+0000","last_became_active":"2024-06-29T14:38:25.262216+0000","last_became_peered":"2024-06-29T14:38:25.262216+0000","last_unstale":"2024-06-29T14:38:25.266296+0000","last_undegraded":"2024-06-29T14:38:25.266296+0000","last_fullsized":"2024-06-29T14:38:25.266296+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:50.699909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253701+0000","last_change":"2024-06-29T14:38:25.255488+0000","last_active":"2024-06-29T14:38:26.253701+0000","last_peered":"2024-06-29T14:38:26.253701+0000","last_clean":"2024-06-29T14:38:26.253701+0000","last_became_active":"2024-06-29T14:38:25.255169+0000","last_became_peered":"2024-06-29T14:38:25.255169+0000","last_unstale":"2024-06-29T14:38:26.253701+0000","last_undegraded":"2024-06-29T14:38:26.253701+0000","last_fullsized":"2024-06-29T14:38:26.253701+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:57.275912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.275022+0000","last_change":"2024-06-29T14:38:25.269662+0000","last_active":"2024-06-29T14:38:25.275022+0000","last_peered":"2024-06-29T14:38:25.275022+0000","last_clean":"2024-06-29T14:38:25.275022+0000","last_became_active":"2024-06-29T14:38:25.269330+0000","last_became_peered":"2024-06-29T14:38:25.269330+0000","last_unstale":"2024-06-29T14:38:25.275022+0000","last_undegraded":"2024-06-29T14:38:25.275022+0000","last_fullsized":"2024-06-29T14:38:25.275022+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:53.949358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.253234+0000","last_change":"2024-06-29T14:38:25.252640+0000","last_active":"2024-06-29T14:38:25.253234+0000","last_peered":"2024-06-29T14:38:25.253234+0000","last_clean":"2024-06-29T14:38:25.253234+0000","last_became_active":"2024-06-29T14:38:25.252245+0000","last_became_peered":"2024-06-29T14:38:25.252245+0000","last_unstale":"2024-06-29T14:38:25.253234+0000","last_undegraded":"2024-06-29T14:38:25.253234+0000","last_fullsized":"2024-06-29T14:38:25.253234+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:35:11.434093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.262486+0000","last_change":"2024-06-29T14:38:25.255036+0000","last_active":"2024-06-29T14:38:25.262486+0000","last_peered":"2024-06-29T14:38:25.262486+0000","last_clean":"2024-06-29T14:38:25.262486+0000","last_became_active":"2024-06-29T14:38:25.254537+0000","last_became_peered":"2024-06-29T14:38:25.254537+0000","last_unstale":"2024-06-29T14:38:25.262486+0000","last_undegraded":"2024-06-29T14:38:25.262486+0000","last_fullsized":"2024-06-29T14:38:25.262486+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:14:05.543579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253930+0000","last_change":"2024-06-29T14:38:25.252228+0000","last_active":"2024-06-29T14:38:26.253930+0000","last_peered":"2024-06-29T14:38:26.253930+0000","last_clean":"2024-06-29T14:38:26.253930+0000","last_became_active":"2024-06-29T14:38:25.251771+0000","last_became_peered":"2024-06-29T14:38:25.251771+0000","last_unstale":"2024-06-29T14:38:26.253930+0000","last_undegraded":"2024-06-29T14:38:26.253930+0000","last_fullsized":"2024-06-29T14:38:26.253930+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:58.577643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252631+0000","last_change":"2024-06-29T14:38:25.250901+0000","last_active":"2024-06-29T14:38:26.252631+0000","last_peered":"2024-06-29T14:38:26.252631+0000","last_clean":"2024-06-29T14:38:26.252631+0000","last_became_active":"2024-06-29T14:38:25.250583+0000","last_became_peered":"2024-06-29T14:38:25.250583+0000","last_unstale":"2024-06-29T14:38:26.252631+0000","last_undegraded":"2024-06-29T14:38:26.252631+0000","last_fullsized":"2024-06-29T14:38:26.252631+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:53:59.235820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254676+0000","last_change":"2024-06-29T14:38:25.258357+0000","last_active":"2024-06-29T14:38:26.254676+0000","last_peered":"2024-06-29T14:38:26.254676+0000","last_clean":"2024-06-29T14:38:26.254676+0000","last_became_active":"2024-06-29T14:38:25.256883+0000","last_became_peered":"2024-06-29T14:38:25.256883+0000","last_unstale":"2024-06-29T14:38:26.254676+0000","last_undegraded":"2024-06-29T14:38:26.254676+0000","last_fullsized":"2024-06-29T14:38:26.254676+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:37:17.660568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252233+0000","last_change":"2024-06-29T14:38:25.243893+0000","last_active":"2024-06-29T14:38:26.252233+0000","last_peered":"2024-06-29T14:38:26.252233+0000","last_clean":"2024-06-29T14:38:26.252233+0000","last_became_active":"2024-06-29T14:38:25.243515+0000","last_became_peered":"2024-06-29T14:38:25.243515+0000","last_unstale":"2024-06-29T14:38:26.252233+0000","last_undegraded":"2024-06-29T14:38:26.252233+0000","last_fullsized":"2024-06-29T14:38:26.252233+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:15.690906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.264263+0000","last_change":"2024-06-29T14:38:25.254080+0000","last_active":"2024-06-29T14:38:25.264263+0000","last_peered":"2024-06-29T14:38:25.264263+0000","last_clean":"2024-06-29T14:38:25.264263+0000","last_became_active":"2024-06-29T14:38:25.253671+0000","last_became_peered":"2024-06-29T14:38:25.253671+0000","last_unstale":"2024-06-29T14:38:25.264263+0000","last_undegraded":"2024-06-29T14:38:25.264263+0000","last_fullsized":"2024-06-29T14:38:25.264263+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:49.583434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253521+0000","last_change":"2024-06-29T14:38:25.270035+0000","last_active":"2024-06-29T14:38:26.253521+0000","last_peered":"2024-06-29T14:38:26.253521+0000","last_clean":"2024-06-29T14:38:26.253521+0000","last_became_active":"2024-06-29T14:38:25.269583+0000","last_became_peered":"2024-06-29T14:38:25.269583+0000","last_unstale":"2024-06-29T14:38:26.253521+0000","last_undegraded":"2024-06-29T14:38:26.253521+0000","last_fullsized":"2024-06-29T14:38:26.253521+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:18:27.204498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.267015+0000","last_change":"2024-06-29T14:38:25.263566+0000","last_active":"2024-06-29T14:38:25.267015+0000","last_peered":"2024-06-29T14:38:25.267015+0000","last_clean":"2024-06-29T14:38:25.267015+0000","last_became_active":"2024-06-29T14:38:25.263220+0000","last_became_peered":"2024-06-29T14:38:25.263220+0000","last_unstale":"2024-06-29T14:38:25.267015+0000","last_undegraded":"2024-06-29T14:38:25.267015+0000","last_fullsized":"2024-06-29T14:38:25.267015+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:47.144025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276282+0000","last_change":"2024-06-29T14:38:25.273414+0000","last_active":"2024-06-29T14:38:25.276282+0000","last_peered":"2024-06-29T14:38:25.276282+0000","last_clean":"2024-06-29T14:38:25.276282+0000","last_became_active":"2024-06-29T14:38:25.273072+0000","last_became_peered":"2024-06-29T14:38:25.273072+0000","last_unstale":"2024-06-29T14:38:25.276282+0000","last_undegraded":"2024-06-29T14:38:25.276282+0000","last_fullsized":"2024-06-29T14:38:25.276282+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:36.384799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276046+0000","last_change":"2024-06-29T14:38:25.273963+0000","last_active":"2024-06-29T14:38:25.276046+0000","last_peered":"2024-06-29T14:38:25.276046+0000","last_clean":"2024-06-29T14:38:25.276046+0000","last_became_active":"2024-06-29T14:38:25.272976+0000","last_became_peered":"2024-06-29T14:38:25.272976+0000","last_unstale":"2024-06-29T14:38:25.276046+0000","last_undegraded":"2024-06-29T14:38:25.276046+0000","last_fullsized":"2024-06-29T14:38:25.276046+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:21.083178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504923+0000","last_change":"2024-06-29T14:38:25.257391+0000","last_active":"2024-06-29T14:38:26.504923+0000","last_peered":"2024-06-29T14:38:26.504923+0000","last_clean":"2024-06-29T14:38:26.504923+0000","last_became_active":"2024-06-29T14:38:25.257054+0000","last_became_peered":"2024-06-29T14:38:25.257054+0000","last_unstale":"2024-06-29T14:38:26.504923+0000","last_undegraded":"2024-06-29T14:38:26.504923+0000","last_fullsized":"2024-06-29T14:38:26.504923+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:32.242764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254714+0000","last_change":"2024-06-29T14:38:25.256102+0000","last_active":"2024-06-29T14:38:26.254714+0000","last_peered":"2024-06-29T14:38:26.254714+0000","last_clean":"2024-06-29T14:38:26.254714+0000","last_became_active":"2024-06-29T14:38:25.254366+0000","last_became_peered":"2024-06-29T14:38:25.254366+0000","last_unstale":"2024-06-29T14:38:26.254714+0000","last_undegraded":"2024-06-29T14:38:26.254714+0000","last_fullsized":"2024-06-29T14:38:26.254714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:41:53.035546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.267850+0000","last_change":"2024-06-29T14:38:25.264084+0000","last_active":"2024-06-29T14:38:25.267850+0000","last_peered":"2024-06-29T14:38:25.267850+0000","last_clean":"2024-06-29T14:38:25.267850+0000","last_became_active":"2024-06-29T14:38:25.263745+0000","last_became_peered":"2024-06-29T14:38:25.263745+0000","last_unstale":"2024-06-29T14:38:25.267850+0000","last_undegraded":"2024-06-29T14:38:25.267850+0000","last_fullsized":"2024-06-29T14:38:25.267850+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:39.430531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254186+0000","last_change":"2024-06-29T14:38:25.254695+0000","last_active":"2024-06-29T14:38:26.254186+0000","last_peered":"2024-06-29T14:38:26.254186+0000","last_clean":"2024-06-29T14:38:26.254186+0000","last_became_active":"2024-06-29T14:38:25.253362+0000","last_became_peered":"2024-06-29T14:38:25.253362+0000","last_unstale":"2024-06-29T14:38:26.254186+0000","last_undegraded":"2024-06-29T14:38:26.254186+0000","last_fullsized":"2024-06-29T14:38:26.254186+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:04:19.098699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253723+0000","last_change":"2024-06-29T14:38:25.253571+0000","last_active":"2024-06-29T14:38:26.253723+0000","last_peered":"2024-06-29T14:38:26.253723+0000","last_clean":"2024-06-29T14:38:26.253723+0000","last_became_active":"2024-06-29T14:38:25.253255+0000","last_became_peered":"2024-06-29T14:38:25.253255+0000","last_unstale":"2024-06-29T14:38:26.253723+0000","last_undegraded":"2024-06-29T14:38:26.253723+0000","last_fullsized":"2024-06-29T14:38:26.253723+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:31:19.873490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253253+0000","last_change":"2024-06-29T14:38:25.265270+0000","last_active":"2024-06-29T14:38:26.253253+0000","last_peered":"2024-06-29T14:38:26.253253+0000","last_clean":"2024-06-29T14:38:26.253253+0000","last_became_active":"2024-06-29T14:38:25.264939+0000","last_became_peered":"2024-06-29T14:38:25.264939+0000","last_unstale":"2024-06-29T14:38:26.253253+0000","last_undegraded":"2024-06-29T14:38:26.253253+0000","last_fullsized":"2024-06-29T14:38:26.253253+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:21:47.305292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504797+0000","last_change":"2024-06-29T14:38:25.263039+0000","last_active":"2024-06-29T14:38:26.504797+0000","last_peered":"2024-06-29T14:38:26.504797+0000","last_clean":"2024-06-29T14:38:26.504797+0000","last_became_active":"2024-06-29T14:38:25.262709+0000","last_became_peered":"2024-06-29T14:38:25.262709+0000","last_unstale":"2024-06-29T14:38:26.504797+0000","last_undegraded":"2024-06-29T14:38:26.504797+0000","last_fullsized":"2024-06-29T14:38:26.504797+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:31.041814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.259150+0000","last_change":"2024-06-29T14:38:25.256751+0000","last_active":"2024-06-29T14:38:26.259150+0000","last_peered":"2024-06-29T14:38:26.259150+0000","last_clean":"2024-06-29T14:38:26.259150+0000","last_became_active":"2024-06-29T14:38:25.256315+0000","last_became_peered":"2024-06-29T14:38:25.256315+0000","last_unstale":"2024-06-29T14:38:26.259150+0000","last_undegraded":"2024-06-29T14:38:26.259150+0000","last_fullsized":"2024-06-29T14:38:26.259150+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:27.779944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254436+0000","last_change":"2024-06-29T14:38:25.269405+0000","last_active":"2024-06-29T14:38:26.254436+0000","last_peered":"2024-06-29T14:38:26.254436+0000","last_clean":"2024-06-29T14:38:26.254436+0000","last_became_active":"2024-06-29T14:38:25.269079+0000","last_became_peered":"2024-06-29T14:38:25.269079+0000","last_unstale":"2024-06-29T14:38:26.254436+0000","last_undegraded":"2024-06-29T14:38:26.254436+0000","last_fullsized":"2024-06-29T14:38:26.254436+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:32.358721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252356+0000","last_change":"2024-06-29T14:38:25.246307+0000","last_active":"2024-06-29T14:38:26.252356+0000","last_peered":"2024-06-29T14:38:26.252356+0000","last_clean":"2024-06-29T14:38:26.252356+0000","last_became_active":"2024-06-29T14:38:25.243960+0000","last_became_peered":"2024-06-29T14:38:25.243960+0000","last_unstale":"2024-06-29T14:38:26.252356+0000","last_undegraded":"2024-06-29T14:38:26.252356+0000","last_fullsized":"2024-06-29T14:38:26.252356+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:16.044436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.272649+0000","last_change":"2024-06-29T14:38:25.267684+0000","last_active":"2024-06-29T14:38:25.272649+0000","last_peered":"2024-06-29T14:38:25.272649+0000","last_clean":"2024-06-29T14:38:25.272649+0000","last_became_active":"2024-06-29T14:38:25.267324+0000","last_became_peered":"2024-06-29T14:38:25.267324+0000","last_unstale":"2024-06-29T14:38:25.272649+0000","last_undegraded":"2024-06-29T14:38:25.272649+0000","last_fullsized":"2024-06-29T14:38:25.272649+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:02.422398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.251400+0000","last_change":"2024-06-29T14:38:25.250278+0000","last_active":"2024-06-29T14:38:25.251400+0000","last_peered":"2024-06-29T14:38:25.251400+0000","last_clean":"2024-06-29T14:38:25.251400+0000","last_became_active":"2024-06-29T14:38:25.249953+0000","last_became_peered":"2024-06-29T14:38:25.249953+0000","last_unstale":"2024-06-29T14:38:25.251400+0000","last_undegraded":"2024-06-29T14:38:25.251400+0000","last_fullsized":"2024-06-29T14:38:25.251400+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:00:03.882845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254229+0000","last_change":"2024-06-29T14:38:25.252943+0000","last_active":"2024-06-29T14:38:26.254229+0000","last_peered":"2024-06-29T14:38:26.254229+0000","last_clean":"2024-06-29T14:38:26.254229+0000","last_became_active":"2024-06-29T14:38:25.251716+0000","last_became_peered":"2024-06-29T14:38:25.251716+0000","last_unstale":"2024-06-29T14:38:26.254229+0000","last_undegraded":"2024-06-29T14:38:26.254229+0000","last_fullsized":"2024-06-29T14:38:26.254229+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:41.771814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258531+0000","last_change":"2024-06-29T14:38:25.267924+0000","last_active":"2024-06-29T14:38:26.258531+0000","last_peered":"2024-06-29T14:38:26.258531+0000","last_clean":"2024-06-29T14:38:26.258531+0000","last_became_active":"2024-06-29T14:38:25.267655+0000","last_became_peered":"2024-06-29T14:38:25.267655+0000","last_unstale":"2024-06-29T14:38:26.258531+0000","last_undegraded":"2024-06-29T14:38:26.258531+0000","last_fullsized":"2024-06-29T14:38:26.258531+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:41:45.854837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.277128+0000","last_change":"2024-06-29T14:38:25.274341+0000","last_active":"2024-06-29T14:38:25.277128+0000","last_peered":"2024-06-29T14:38:25.277128+0000","last_clean":"2024-06-29T14:38:25.277128+0000","last_became_active":"2024-06-29T14:38:25.273809+0000","last_became_peered":"2024-06-29T14:38:25.273809+0000","last_unstale":"2024-06-29T14:38:25.277128+0000","last_undegraded":"2024-06-29T14:38:25.277128+0000","last_fullsized":"2024-06-29T14:38:25.277128+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:24:14.699898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.267225+0000","last_change":"2024-06-29T14:38:25.260039+0000","last_active":"2024-06-29T14:38:25.267225+0000","last_peered":"2024-06-29T14:38:25.267225+0000","last_clean":"2024-06-29T14:38:25.267225+0000","last_became_active":"2024-06-29T14:38:25.259705+0000","last_became_peered":"2024-06-29T14:38:25.259705+0000","last_unstale":"2024-06-29T14:38:25.267225+0000","last_undegraded":"2024-06-29T14:38:25.267225+0000","last_fullsized":"2024-06-29T14:38:25.267225+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:04:29.412189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"113'4","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254572+0000","last_change":"2024-06-29T14:38:24.246482+0000","last_active":"2024-06-29T14:38:26.254572+0000","last_peered":"2024-06-29T14:38:26.254572+0000","last_clean":"2024-06-29T14:38:26.254572+0000","last_became_active":"2024-06-29T14:38:24.246228+0000","last_became_peered":"2024-06-29T14:38:24.246228+0000","last_unstale":"2024-06-29T14:38:26.254572+0000","last_undegraded":"2024-06-29T14:38:26.254572+0000","last_fullsized":"2024-06-29T14:38:26.254572+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:55.908391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266482+0000","last_change":"2024-06-29T14:38:25.261983+0000","last_active":"2024-06-29T14:38:25.266482+0000","last_peered":"2024-06-29T14:38:25.266482+0000","last_clean":"2024-06-29T14:38:25.266482+0000","last_became_active":"2024-06-29T14:38:25.261676+0000","last_became_peered":"2024-06-29T14:38:25.261676+0000","last_unstale":"2024-06-29T14:38:25.266482+0000","last_undegraded":"2024-06-29T14:38:25.266482+0000","last_fullsized":"2024-06-29T14:38:25.266482+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:44.155424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254384+0000","last_change":"2024-06-29T14:38:24.236757+0000","last_active":"2024-06-29T14:38:26.254384+0000","last_peered":"2024-06-29T14:38:26.254384+0000","last_clean":"2024-06-29T14:38:26.254384+0000","last_became_active":"2024-06-29T14:38:24.236433+0000","last_became_peered":"2024-06-29T14:38:24.236433+0000","last_unstale":"2024-06-29T14:38:26.254384+0000","last_undegraded":"2024-06-29T14:38:26.254384+0000","last_fullsized":"2024-06-29T14:38:26.254384+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:31.110203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.270299+0000","last_change":"2024-06-29T14:38:25.269356+0000","last_active":"2024-06-29T14:38:25.270299+0000","last_peered":"2024-06-29T14:38:25.270299+0000","last_clean":"2024-06-29T14:38:25.270299+0000","last_became_active":"2024-06-29T14:38:25.269005+0000","last_became_peered":"2024-06-29T14:38:25.269005+0000","last_unstale":"2024-06-29T14:38:25.270299+0000","last_undegraded":"2024-06-29T14:38:25.270299+0000","last_fullsized":"2024-06-29T14:38:25.270299+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:56.668689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"113'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266510+0000","last_change":"2024-06-29T14:38:24.241221+0000","last_active":"2024-06-29T14:38:25.266510+0000","last_peered":"2024-06-29T14:38:25.266510+0000","last_clean":"2024-06-29T14:38:25.266510+0000","last_became_active":"2024-06-29T14:38:24.240897+0000","last_became_peered":"2024-06-29T14:38:24.240897+0000","last_unstale":"2024-06-29T14:38:25.266510+0000","last_undegraded":"2024-06-29T14:38:25.266510+0000","last_fullsized":"2024-06-29T14:38:25.266510+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:06:47.300493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.260213+0000","last_change":"2024-06-29T14:38:25.256937+0000","last_active":"2024-06-29T14:38:25.260213+0000","last_peered":"2024-06-29T14:38:25.260213+0000","last_clean":"2024-06-29T14:38:25.260213+0000","last_became_active":"2024-06-29T14:38:25.256610+0000","last_became_peered":"2024-06-29T14:38:25.256610+0000","last_unstale":"2024-06-29T14:38:25.260213+0000","last_undegraded":"2024-06-29T14:38:25.260213+0000","last_fullsized":"2024-06-29T14:38:25.260213+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:11.574455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254236+0000","last_change":"2024-06-29T14:38:24.238658+0000","last_active":"2024-06-29T14:38:26.254236+0000","last_peered":"2024-06-29T14:38:26.254236+0000","last_clean":"2024-06-29T14:38:26.254236+0000","last_became_active":"2024-06-29T14:38:24.238287+0000","last_became_peered":"2024-06-29T14:38:24.238287+0000","last_unstale":"2024-06-29T14:38:26.254236+0000","last_undegraded":"2024-06-29T14:38:26.254236+0000","last_fullsized":"2024-06-29T14:38:26.254236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:31.110203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.265786+0000","last_change":"2024-06-29T14:38:25.262627+0000","last_active":"2024-06-29T14:38:25.265786+0000","last_peered":"2024-06-29T14:38:25.265786+0000","last_clean":"2024-06-29T14:38:25.265786+0000","last_became_active":"2024-06-29T14:38:25.262247+0000","last_became_peered":"2024-06-29T14:38:25.262247+0000","last_unstale":"2024-06-29T14:38:25.265786+0000","last_undegraded":"2024-06-29T14:38:25.265786+0000","last_fullsized":"2024-06-29T14:38:25.265786+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:47.811090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253872+0000","last_change":"2024-06-29T14:38:24.240712+0000","last_active":"2024-06-29T14:38:26.253872+0000","last_peered":"2024-06-29T14:38:26.253872+0000","last_clean":"2024-06-29T14:38:26.253872+0000","last_became_active":"2024-06-29T14:38:24.240398+0000","last_became_peered":"2024-06-29T14:38:24.240398+0000","last_unstale":"2024-06-29T14:38:26.253872+0000","last_undegraded":"2024-06-29T14:38:26.253872+0000","last_fullsized":"2024-06-29T14:38:26.253872+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:50:51.069888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253254+0000","last_change":"2024-06-29T14:38:25.253336+0000","last_active":"2024-06-29T14:38:26.253254+0000","last_peered":"2024-06-29T14:38:26.253254+0000","last_clean":"2024-06-29T14:38:26.253254+0000","last_became_active":"2024-06-29T14:38:25.252964+0000","last_became_peered":"2024-06-29T14:38:25.252964+0000","last_unstale":"2024-06-29T14:38:26.253254+0000","last_undegraded":"2024-06-29T14:38:26.253254+0000","last_fullsized":"2024-06-29T14:38:26.253254+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:50.419536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"113'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.279054+0000","last_change":"2024-06-29T14:38:24.241135+0000","last_active":"2024-06-29T14:38:25.279054+0000","last_peered":"2024-06-29T14:38:25.279054+0000","last_clean":"2024-06-29T14:38:25.279054+0000","last_became_active":"2024-06-29T14:38:24.240821+0000","last_became_peered":"2024-06-29T14:38:24.240821+0000","last_unstale":"2024-06-29T14:38:25.279054+0000","last_undegraded":"2024-06-29T14:38:25.279054+0000","last_fullsized":"2024-06-29T14:38:25.279054+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:15.375607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.273069+0000","last_change":"2024-06-29T14:38:25.268819+0000","last_active":"2024-06-29T14:38:25.273069+0000","last_peered":"2024-06-29T14:38:25.273069+0000","last_clean":"2024-06-29T14:38:25.273069+0000","last_became_active":"2024-06-29T14:38:25.268493+0000","last_became_peered":"2024-06-29T14:38:25.268493+0000","last_unstale":"2024-06-29T14:38:25.273069+0000","last_undegraded":"2024-06-29T14:38:25.273069+0000","last_fullsized":"2024-06-29T14:38:25.273069+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:00.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.505016+0000","last_change":"2024-06-29T14:38:24.234597+0000","last_active":"2024-06-29T14:38:26.505016+0000","last_peered":"2024-06-29T14:38:26.505016+0000","last_clean":"2024-06-29T14:38:26.505016+0000","last_became_active":"2024-06-29T14:38:24.234219+0000","last_became_peered":"2024-06-29T14:38:24.234219+0000","last_unstale":"2024-06-29T14:38:26.505016+0000","last_undegraded":"2024-06-29T14:38:26.505016+0000","last_fullsized":"2024-06-29T14:38:26.505016+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:24:13.693101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266686+0000","last_change":"2024-06-29T14:38:25.263143+0000","last_active":"2024-06-29T14:38:25.266686+0000","last_peered":"2024-06-29T14:38:25.266686+0000","last_clean":"2024-06-29T14:38:25.266686+0000","last_became_active":"2024-06-29T14:38:25.262782+0000","last_became_peered":"2024-06-29T14:38:25.262782+0000","last_unstale":"2024-06-29T14:38:25.266686+0000","last_undegraded":"2024-06-29T14:38:25.266686+0000","last_fullsized":"2024-06-29T14:38:25.266686+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:07.068662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.246801+0000","last_change":"2024-06-29T14:38:24.247874+0000","last_active":"2024-06-29T14:38:25.246801+0000","last_peered":"2024-06-29T14:38:25.246801+0000","last_clean":"2024-06-29T14:38:25.246801+0000","last_became_active":"2024-06-29T14:38:24.247440+0000","last_became_peered":"2024-06-29T14:38:24.247440+0000","last_unstale":"2024-06-29T14:38:25.246801+0000","last_undegraded":"2024-06-29T14:38:25.246801+0000","last_fullsized":"2024-06-29T14:38:25.246801+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:37:38.067294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253932+0000","last_change":"2024-06-29T14:38:25.255741+0000","last_active":"2024-06-29T14:38:26.253932+0000","last_peered":"2024-06-29T14:38:26.253932+0000","last_clean":"2024-06-29T14:38:26.253932+0000","last_became_active":"2024-06-29T14:38:25.253494+0000","last_became_peered":"2024-06-29T14:38:25.253494+0000","last_unstale":"2024-06-29T14:38:26.253932+0000","last_undegraded":"2024-06-29T14:38:26.253932+0000","last_fullsized":"2024-06-29T14:38:26.253932+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:58.283980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.259270+0000","last_change":"2024-06-29T14:38:24.239910+0000","last_active":"2024-06-29T14:38:26.259270+0000","last_peered":"2024-06-29T14:38:26.259270+0000","last_clean":"2024-06-29T14:38:26.259270+0000","last_became_active":"2024-06-29T14:38:24.239581+0000","last_became_peered":"2024-06-29T14:38:24.239581+0000","last_unstale":"2024-06-29T14:38:26.259270+0000","last_undegraded":"2024-06-29T14:38:26.259270+0000","last_fullsized":"2024-06-29T14:38:26.259270+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:31.235967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.257479+0000","last_change":"2024-06-29T14:38:25.256951+0000","last_active":"2024-06-29T14:38:25.257479+0000","last_peered":"2024-06-29T14:38:25.257479+0000","last_clean":"2024-06-29T14:38:25.257479+0000","last_became_active":"2024-06-29T14:38:25.256618+0000","last_became_peered":"2024-06-29T14:38:25.256618+0000","last_unstale":"2024-06-29T14:38:25.257479+0000","last_undegraded":"2024-06-29T14:38:25.257479+0000","last_fullsized":"2024-06-29T14:38:25.257479+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:06:57.038946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.505001+0000","last_change":"2024-06-29T14:38:24.234071+0000","last_active":"2024-06-29T14:38:26.505001+0000","last_peered":"2024-06-29T14:38:26.505001+0000","last_clean":"2024-06-29T14:38:26.505001+0000","last_became_active":"2024-06-29T14:38:24.233724+0000","last_became_peered":"2024-06-29T14:38:24.233724+0000","last_unstale":"2024-06-29T14:38:26.505001+0000","last_undegraded":"2024-06-29T14:38:26.505001+0000","last_fullsized":"2024-06-29T14:38:26.505001+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:07:29.952013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504974+0000","last_change":"2024-06-29T14:38:25.262192+0000","last_active":"2024-06-29T14:38:26.504974+0000","last_peered":"2024-06-29T14:38:26.504974+0000","last_clean":"2024-06-29T14:38:26.504974+0000","last_became_active":"2024-06-29T14:38:25.261843+0000","last_became_peered":"2024-06-29T14:38:25.261843+0000","last_unstale":"2024-06-29T14:38:26.504974+0000","last_undegraded":"2024-06-29T14:38:26.504974+0000","last_fullsized":"2024-06-29T14:38:26.504974+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:11.203600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"113'3","reported_seq":17,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254852+0000","last_change":"2024-06-29T14:38:24.247790+0000","last_active":"2024-06-29T14:38:26.254852+0000","last_peered":"2024-06-29T14:38:26.254852+0000","last_clean":"2024-06-29T14:38:26.254852+0000","last_became_active":"2024-06-29T14:38:24.247492+0000","last_became_peered":"2024-06-29T14:38:24.247492+0000","last_unstale":"2024-06-29T14:38:26.254852+0000","last_undegraded":"2024-06-29T14:38:26.254852+0000","last_fullsized":"2024-06-29T14:38:26.254852+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:07.555389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266873+0000","last_change":"2024-06-29T14:38:25.261727+0000","last_active":"2024-06-29T14:38:25.266873+0000","last_peered":"2024-06-29T14:38:25.266873+0000","last_clean":"2024-06-29T14:38:25.266873+0000","last_became_active":"2024-06-29T14:38:25.260935+0000","last_became_peered":"2024-06-29T14:38:25.260935+0000","last_unstale":"2024-06-29T14:38:25.266873+0000","last_undegraded":"2024-06-29T14:38:25.266873+0000","last_fullsized":"2024-06-29T14:38:25.266873+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:18:47.480133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"113'6","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.279096+0000","last_change":"2024-06-29T14:38:24.240662+0000","last_active":"2024-06-29T14:38:25.279096+0000","last_peered":"2024-06-29T14:38:25.279096+0000","last_clean":"2024-06-29T14:38:25.279096+0000","last_became_active":"2024-06-29T14:38:24.240227+0000","last_became_peered":"2024-06-29T14:38:24.240227+0000","last_unstale":"2024-06-29T14:38:25.279096+0000","last_undegraded":"2024-06-29T14:38:25.279096+0000","last_fullsized":"2024-06-29T14:38:25.279096+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:14:06.509455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.262817+0000","last_change":"2024-06-29T14:38:25.254856+0000","last_active":"2024-06-29T14:38:25.262817+0000","last_peered":"2024-06-29T14:38:25.262817+0000","last_clean":"2024-06-29T14:38:25.262817+0000","last_became_active":"2024-06-29T14:38:25.254119+0000","last_became_peered":"2024-06-29T14:38:25.254119+0000","last_unstale":"2024-06-29T14:38:25.262817+0000","last_undegraded":"2024-06-29T14:38:25.262817+0000","last_fullsized":"2024-06-29T14:38:25.262817+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:55.347262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"113'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.277664+0000","last_change":"2024-06-29T14:38:24.247772+0000","last_active":"2024-06-29T14:38:25.277664+0000","last_peered":"2024-06-29T14:38:25.277664+0000","last_clean":"2024-06-29T14:38:25.277664+0000","last_became_active":"2024-06-29T14:38:24.247394+0000","last_became_peered":"2024-06-29T14:38:24.247394+0000","last_unstale":"2024-06-29T14:38:25.277664+0000","last_undegraded":"2024-06-29T14:38:25.277664+0000","last_fullsized":"2024-06-29T14:38:25.277664+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:25:43.149108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254426+0000","last_change":"2024-06-29T14:38:25.252014+0000","last_active":"2024-06-29T14:38:26.254426+0000","last_peered":"2024-06-29T14:38:26.254426+0000","last_clean":"2024-06-29T14:38:26.254426+0000","last_became_active":"2024-06-29T14:38:25.251666+0000","last_became_peered":"2024-06-29T14:38:25.251666+0000","last_unstale":"2024-06-29T14:38:26.254426+0000","last_undegraded":"2024-06-29T14:38:26.254426+0000","last_fullsized":"2024-06-29T14:38:26.254426+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:55:35.230309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.251493+0000","last_change":"2024-06-29T14:38:24.246143+0000","last_active":"2024-06-29T14:38:25.251493+0000","last_peered":"2024-06-29T14:38:25.251493+0000","last_clean":"2024-06-29T14:38:25.251493+0000","last_became_active":"2024-06-29T14:38:24.245765+0000","last_became_peered":"2024-06-29T14:38:24.245765+0000","last_unstale":"2024-06-29T14:38:25.251493+0000","last_undegraded":"2024-06-29T14:38:25.251493+0000","last_fullsized":"2024-06-29T14:38:25.251493+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:02:10.504001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252071+0000","last_change":"2024-06-29T14:38:25.251826+0000","last_active":"2024-06-29T14:38:26.252071+0000","last_peered":"2024-06-29T14:38:26.252071+0000","last_clean":"2024-06-29T14:38:26.252071+0000","last_became_active":"2024-06-29T14:38:25.251516+0000","last_became_peered":"2024-06-29T14:38:25.251516+0000","last_unstale":"2024-06-29T14:38:26.252071+0000","last_undegraded":"2024-06-29T14:38:26.252071+0000","last_fullsized":"2024-06-29T14:38:26.252071+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:47.200322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.241404+0000","last_change":"2024-06-29T14:38:24.235783+0000","last_active":"2024-06-29T14:38:25.241404+0000","last_peered":"2024-06-29T14:38:25.241404+0000","last_clean":"2024-06-29T14:38:25.241404+0000","last_became_active":"2024-06-29T14:38:24.235437+0000","last_became_peered":"2024-06-29T14:38:24.235437+0000","last_unstale":"2024-06-29T14:38:25.241404+0000","last_undegraded":"2024-06-29T14:38:25.241404+0000","last_fullsized":"2024-06-29T14:38:25.241404+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:19.676731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252129+0000","last_change":"2024-06-29T14:38:25.264403+0000","last_active":"2024-06-29T14:38:26.252129+0000","last_peered":"2024-06-29T14:38:26.252129+0000","last_clean":"2024-06-29T14:38:26.252129+0000","last_became_active":"2024-06-29T14:38:25.263244+0000","last_became_peered":"2024-06-29T14:38:25.263244+0000","last_unstale":"2024-06-29T14:38:26.252129+0000","last_undegraded":"2024-06-29T14:38:26.252129+0000","last_fullsized":"2024-06-29T14:38:26.252129+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:28.522036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"113'8","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.280217+0000","last_change":"2024-06-29T14:38:24.242860+0000","last_active":"2024-06-29T14:38:25.280217+0000","last_peered":"2024-06-29T14:38:25.280217+0000","last_clean":"2024-06-29T14:38:25.280217+0000","last_became_active":"2024-06-29T14:38:24.242551+0000","last_became_peered":"2024-06-29T14:38:24.242551+0000","last_unstale":"2024-06-29T14:38:25.280217+0000","last_undegraded":"2024-06-29T14:38:25.280217+0000","last_fullsized":"2024-06-29T14:38:25.280217+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:01.650803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.267200+0000","last_change":"2024-06-29T14:38:25.257913+0000","last_active":"2024-06-29T14:38:25.267200+0000","last_peered":"2024-06-29T14:38:25.267200+0000","last_clean":"2024-06-29T14:38:25.267200+0000","last_became_active":"2024-06-29T14:38:25.257587+0000","last_became_peered":"2024-06-29T14:38:25.257587+0000","last_unstale":"2024-06-29T14:38:25.267200+0000","last_undegraded":"2024-06-29T14:38:25.267200+0000","last_fullsized":"2024-06-29T14:38:25.267200+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:33.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254650+0000","last_change":"2024-06-29T14:38:24.235635+0000","last_active":"2024-06-29T14:38:26.254650+0000","last_peered":"2024-06-29T14:38:26.254650+0000","last_clean":"2024-06-29T14:38:26.254650+0000","last_became_active":"2024-06-29T14:38:24.235329+0000","last_became_peered":"2024-06-29T14:38:24.235329+0000","last_unstale":"2024-06-29T14:38:26.254650+0000","last_undegraded":"2024-06-29T14:38:26.254650+0000","last_fullsized":"2024-06-29T14:38:26.254650+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:23.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253719+0000","last_change":"2024-06-29T14:38:25.250431+0000","last_active":"2024-06-29T14:38:26.253719+0000","last_peered":"2024-06-29T14:38:26.253719+0000","last_clean":"2024-06-29T14:38:26.253719+0000","last_became_active":"2024-06-29T14:38:25.250125+0000","last_became_peered":"2024-06-29T14:38:25.250125+0000","last_unstale":"2024-06-29T14:38:26.253719+0000","last_undegraded":"2024-06-29T14:38:26.253719+0000","last_fullsized":"2024-06-29T14:38:26.253719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:48.095292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"113'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.264464+0000","last_change":"2024-06-29T14:38:24.236235+0000","last_active":"2024-06-29T14:38:25.264464+0000","last_peered":"2024-06-29T14:38:25.264464+0000","last_clean":"2024-06-29T14:38:25.264464+0000","last_became_active":"2024-06-29T14:38:24.235850+0000","last_became_peered":"2024-06-29T14:38:24.235850+0000","last_unstale":"2024-06-29T14:38:25.264464+0000","last_undegraded":"2024-06-29T14:38:25.264464+0000","last_fullsized":"2024-06-29T14:38:25.264464+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:14.484323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254258+0000","last_change":"2024-06-29T14:38:25.253942+0000","last_active":"2024-06-29T14:38:26.254258+0000","last_peered":"2024-06-29T14:38:26.254258+0000","last_clean":"2024-06-29T14:38:26.254258+0000","last_became_active":"2024-06-29T14:38:25.253266+0000","last_became_peered":"2024-06-29T14:38:25.253266+0000","last_unstale":"2024-06-29T14:38:26.254258+0000","last_undegraded":"2024-06-29T14:38:26.254258+0000","last_fullsized":"2024-06-29T14:38:26.254258+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:26:22.525190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254999+0000","last_change":"2024-06-29T14:38:24.233439+0000","last_active":"2024-06-29T14:38:26.254999+0000","last_peered":"2024-06-29T14:38:26.254999+0000","last_clean":"2024-06-29T14:38:26.254999+0000","last_became_active":"2024-06-29T14:38:24.233085+0000","last_became_peered":"2024-06-29T14:38:24.233085+0000","last_unstale":"2024-06-29T14:38:26.254999+0000","last_undegraded":"2024-06-29T14:38:26.254999+0000","last_fullsized":"2024-06-29T14:38:26.254999+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:35.922312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.261180+0000","last_change":"2024-06-29T14:38:25.256840+0000","last_active":"2024-06-29T14:38:25.261180+0000","last_peered":"2024-06-29T14:38:25.261180+0000","last_clean":"2024-06-29T14:38:25.261180+0000","last_became_active":"2024-06-29T14:38:25.256473+0000","last_became_peered":"2024-06-29T14:38:25.256473+0000","last_unstale":"2024-06-29T14:38:25.261180+0000","last_undegraded":"2024-06-29T14:38:25.261180+0000","last_fullsized":"2024-06-29T14:38:25.261180+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:30:50.230950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258821+0000","last_change":"2024-06-29T14:38:24.240662+0000","last_active":"2024-06-29T14:38:26.258821+0000","last_peered":"2024-06-29T14:38:26.258821+0000","last_clean":"2024-06-29T14:38:26.258821+0000","last_became_active":"2024-06-29T14:38:24.239874+0000","last_became_peered":"2024-06-29T14:38:24.239874+0000","last_unstale":"2024-06-29T14:38:26.258821+0000","last_undegraded":"2024-06-29T14:38:26.258821+0000","last_fullsized":"2024-06-29T14:38:26.258821+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:10.478176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258550+0000","last_change":"2024-06-29T14:38:25.259659+0000","last_active":"2024-06-29T14:38:26.258550+0000","last_peered":"2024-06-29T14:38:26.258550+0000","last_clean":"2024-06-29T14:38:26.258550+0000","last_became_active":"2024-06-29T14:38:25.259353+0000","last_became_peered":"2024-06-29T14:38:25.259353+0000","last_unstale":"2024-06-29T14:38:26.258550+0000","last_undegraded":"2024-06-29T14:38:26.258550+0000","last_fullsized":"2024-06-29T14:38:26.258550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:07.841508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.251998+0000","last_change":"2024-06-29T14:38:24.242984+0000","last_active":"2024-06-29T14:38:26.251998+0000","last_peered":"2024-06-29T14:38:26.251998+0000","last_clean":"2024-06-29T14:38:26.251998+0000","last_became_active":"2024-06-29T14:38:24.242419+0000","last_became_peered":"2024-06-29T14:38:24.242419+0000","last_unstale":"2024-06-29T14:38:26.251998+0000","last_undegraded":"2024-06-29T14:38:26.251998+0000","last_fullsized":"2024-06-29T14:38:26.251998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:13.989024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258711+0000","last_change":"2024-06-29T14:38:25.268979+0000","last_active":"2024-06-29T14:38:26.258711+0000","last_peered":"2024-06-29T14:38:26.258711+0000","last_clean":"2024-06-29T14:38:26.258711+0000","last_became_active":"2024-06-29T14:38:25.268617+0000","last_became_peered":"2024-06-29T14:38:25.268617+0000","last_unstale":"2024-06-29T14:38:26.258711+0000","last_undegraded":"2024-06-29T14:38:26.258711+0000","last_fullsized":"2024-06-29T14:38:26.258711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:59:46.751191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258646+0000","last_change":"2024-06-29T14:38:24.247720+0000","last_active":"2024-06-29T14:38:26.258646+0000","last_peered":"2024-06-29T14:38:26.258646+0000","last_clean":"2024-06-29T14:38:26.258646+0000","last_became_active":"2024-06-29T14:38:24.247407+0000","last_became_peered":"2024-06-29T14:38:24.247407+0000","last_unstale":"2024-06-29T14:38:26.258646+0000","last_undegraded":"2024-06-29T14:38:26.258646+0000","last_fullsized":"2024-06-29T14:38:26.258646+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:30.035017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276930+0000","last_change":"2024-06-29T14:38:25.273618+0000","last_active":"2024-06-29T14:38:25.276930+0000","last_peered":"2024-06-29T14:38:25.276930+0000","last_clean":"2024-06-29T14:38:25.276930+0000","last_became_active":"2024-06-29T14:38:25.273232+0000","last_became_peered":"2024-06-29T14:38:25.273232+0000","last_unstale":"2024-06-29T14:38:25.276930+0000","last_undegraded":"2024-06-29T14:38:25.276930+0000","last_fullsized":"2024-06-29T14:38:25.276930+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:03.235771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252002+0000","last_change":"2024-06-29T14:38:24.238128+0000","last_active":"2024-06-29T14:38:26.252002+0000","last_peered":"2024-06-29T14:38:26.252002+0000","last_clean":"2024-06-29T14:38:26.252002+0000","last_became_active":"2024-06-29T14:38:24.237794+0000","last_became_peered":"2024-06-29T14:38:24.237794+0000","last_unstale":"2024-06-29T14:38:26.252002+0000","last_undegraded":"2024-06-29T14:38:26.252002+0000","last_fullsized":"2024-06-29T14:38:26.252002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:24.723402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254101+0000","last_change":"2024-06-29T14:38:25.249778+0000","last_active":"2024-06-29T14:38:26.254101+0000","last_peered":"2024-06-29T14:38:26.254101+0000","last_clean":"2024-06-29T14:38:26.254101+0000","last_became_active":"2024-06-29T14:38:25.249455+0000","last_became_peered":"2024-06-29T14:38:25.249455+0000","last_unstale":"2024-06-29T14:38:26.254101+0000","last_undegraded":"2024-06-29T14:38:26.254101+0000","last_fullsized":"2024-06-29T14:38:26.254101+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:32.993979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"113'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266180+0000","last_change":"2024-06-29T14:38:24.234583+0000","last_active":"2024-06-29T14:38:25.266180+0000","last_peered":"2024-06-29T14:38:25.266180+0000","last_clean":"2024-06-29T14:38:25.266180+0000","last_became_active":"2024-06-29T14:38:24.234249+0000","last_became_peered":"2024-06-29T14:38:24.234249+0000","last_unstale":"2024-06-29T14:38:25.266180+0000","last_undegraded":"2024-06-29T14:38:25.266180+0000","last_fullsized":"2024-06-29T14:38:25.266180+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:17:35.184659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266452+0000","last_change":"2024-06-29T14:38:25.261017+0000","last_active":"2024-06-29T14:38:25.266452+0000","last_peered":"2024-06-29T14:38:25.266452+0000","last_clean":"2024-06-29T14:38:25.266452+0000","last_became_active":"2024-06-29T14:38:25.260710+0000","last_became_peered":"2024-06-29T14:38:25.260710+0000","last_unstale":"2024-06-29T14:38:25.266452+0000","last_undegraded":"2024-06-29T14:38:25.266452+0000","last_fullsized":"2024-06-29T14:38:25.266452+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:50.078684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"114'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.257955+0000","last_change":"2024-06-29T14:38:24.237559+0000","last_active":"2024-06-29T14:38:26.257955+0000","last_peered":"2024-06-29T14:38:26.257955+0000","last_clean":"2024-06-29T14:38:26.257955+0000","last_became_active":"2024-06-29T14:38:24.237199+0000","last_became_peered":"2024-06-29T14:38:24.237199+0000","last_unstale":"2024-06-29T14:38:26.257955+0000","last_undegraded":"2024-06-29T14:38:26.257955+0000","last_fullsized":"2024-06-29T14:38:26.257955+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:23.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276322+0000","last_change":"2024-06-29T14:38:25.274329+0000","last_active":"2024-06-29T14:38:25.276322+0000","last_peered":"2024-06-29T14:38:25.276322+0000","last_clean":"2024-06-29T14:38:25.276322+0000","last_became_active":"2024-06-29T14:38:25.271073+0000","last_became_peered":"2024-06-29T14:38:25.271073+0000","last_unstale":"2024-06-29T14:38:25.276322+0000","last_undegraded":"2024-06-29T14:38:25.276322+0000","last_fullsized":"2024-06-29T14:38:25.276322+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:52.208296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254984+0000","last_change":"2024-06-29T14:38:24.247753+0000","last_active":"2024-06-29T14:38:26.254984+0000","last_peered":"2024-06-29T14:38:26.254984+0000","last_clean":"2024-06-29T14:38:26.254984+0000","last_became_active":"2024-06-29T14:38:24.247448+0000","last_became_peered":"2024-06-29T14:38:24.247448+0000","last_unstale":"2024-06-29T14:38:26.254984+0000","last_undegraded":"2024-06-29T14:38:26.254984+0000","last_fullsized":"2024-06-29T14:38:26.254984+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:55.908391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.259150+0000","last_change":"2024-06-29T14:38:25.258035+0000","last_active":"2024-06-29T14:38:26.259150+0000","last_peered":"2024-06-29T14:38:26.259150+0000","last_clean":"2024-06-29T14:38:26.259150+0000","last_became_active":"2024-06-29T14:38:25.257716+0000","last_became_peered":"2024-06-29T14:38:25.257716+0000","last_unstale":"2024-06-29T14:38:26.259150+0000","last_undegraded":"2024-06-29T14:38:26.259150+0000","last_fullsized":"2024-06-29T14:38:26.259150+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:23.293066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254421+0000","last_change":"2024-06-29T14:38:24.240706+0000","last_active":"2024-06-29T14:38:26.254421+0000","last_peered":"2024-06-29T14:38:26.254421+0000","last_clean":"2024-06-29T14:38:26.254421+0000","last_became_active":"2024-06-29T14:38:24.240371+0000","last_became_peered":"2024-06-29T14:38:24.240371+0000","last_unstale":"2024-06-29T14:38:26.254421+0000","last_undegraded":"2024-06-29T14:38:26.254421+0000","last_fullsized":"2024-06-29T14:38:26.254421+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:11:21.632342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.251888+0000","last_change":"2024-06-29T14:38:25.259943+0000","last_active":"2024-06-29T14:38:26.251888+0000","last_peered":"2024-06-29T14:38:26.251888+0000","last_clean":"2024-06-29T14:38:26.251888+0000","last_became_active":"2024-06-29T14:38:25.259562+0000","last_became_peered":"2024-06-29T14:38:25.259562+0000","last_unstale":"2024-06-29T14:38:26.251888+0000","last_undegraded":"2024-06-29T14:38:26.251888+0000","last_fullsized":"2024-06-29T14:38:26.251888+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:14:27.069812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.254134+0000","last_change":"2024-06-29T14:38:24.247118+0000","last_active":"2024-06-29T14:38:25.254134+0000","last_peered":"2024-06-29T14:38:25.254134+0000","last_clean":"2024-06-29T14:38:25.254134+0000","last_became_active":"2024-06-29T14:38:24.246726+0000","last_became_peered":"2024-06-29T14:38:24.246726+0000","last_unstale":"2024-06-29T14:38:25.254134+0000","last_undegraded":"2024-06-29T14:38:25.254134+0000","last_fullsized":"2024-06-29T14:38:25.254134+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:29.359547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504783+0000","last_change":"2024-06-29T14:38:25.266715+0000","last_active":"2024-06-29T14:38:26.504783+0000","last_peered":"2024-06-29T14:38:26.504783+0000","last_clean":"2024-06-29T14:38:26.504783+0000","last_became_active":"2024-06-29T14:38:25.264793+0000","last_became_peered":"2024-06-29T14:38:25.264793+0000","last_unstale":"2024-06-29T14:38:26.504783+0000","last_undegraded":"2024-06-29T14:38:26.504783+0000","last_fullsized":"2024-06-29T14:38:26.504783+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:11.484973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258828+0000","last_change":"2024-06-29T14:38:24.245329+0000","last_active":"2024-06-29T14:38:26.258828+0000","last_peered":"2024-06-29T14:38:26.258828+0000","last_clean":"2024-06-29T14:38:26.258828+0000","last_became_active":"2024-06-29T14:38:24.244794+0000","last_became_peered":"2024-06-29T14:38:24.244794+0000","last_unstale":"2024-06-29T14:38:26.258828+0000","last_undegraded":"2024-06-29T14:38:26.258828+0000","last_fullsized":"2024-06-29T14:38:26.258828+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:35.327250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253236+0000","last_change":"2024-06-29T14:38:25.264996+0000","last_active":"2024-06-29T14:38:26.253236+0000","last_peered":"2024-06-29T14:38:26.253236+0000","last_clean":"2024-06-29T14:38:26.253236+0000","last_became_active":"2024-06-29T14:38:25.264703+0000","last_became_peered":"2024-06-29T14:38:25.264703+0000","last_unstale":"2024-06-29T14:38:26.253236+0000","last_undegraded":"2024-06-29T14:38:26.253236+0000","last_fullsized":"2024-06-29T14:38:26.253236+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:01.897966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253406+0000","last_change":"2024-06-29T14:38:24.240670+0000","last_active":"2024-06-29T14:38:26.253406+0000","last_peered":"2024-06-29T14:38:26.253406+0000","last_clean":"2024-06-29T14:38:26.253406+0000","last_became_active":"2024-06-29T14:38:24.240355+0000","last_became_peered":"2024-06-29T14:38:24.240355+0000","last_unstale":"2024-06-29T14:38:26.253406+0000","last_undegraded":"2024-06-29T14:38:26.253406+0000","last_fullsized":"2024-06-29T14:38:26.253406+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:24.373775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276955+0000","last_change":"2024-06-29T14:38:25.274356+0000","last_active":"2024-06-29T14:38:25.276955+0000","last_peered":"2024-06-29T14:38:25.276955+0000","last_clean":"2024-06-29T14:38:25.276955+0000","last_became_active":"2024-06-29T14:38:25.274025+0000","last_became_peered":"2024-06-29T14:38:25.274025+0000","last_unstale":"2024-06-29T14:38:25.276955+0000","last_undegraded":"2024-06-29T14:38:25.276955+0000","last_fullsized":"2024-06-29T14:38:25.276955+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:39:48.564333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.246804+0000","last_change":"2024-06-29T14:38:24.242251+0000","last_active":"2024-06-29T14:38:25.246804+0000","last_peered":"2024-06-29T14:38:25.246804+0000","last_clean":"2024-06-29T14:38:25.246804+0000","last_became_active":"2024-06-29T14:38:24.241944+0000","last_became_peered":"2024-06-29T14:38:24.241944+0000","last_unstale":"2024-06-29T14:38:25.246804+0000","last_undegraded":"2024-06-29T14:38:25.246804+0000","last_fullsized":"2024-06-29T14:38:25.246804+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:14.114639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.270941+0000","last_change":"2024-06-29T14:38:25.270227+0000","last_active":"2024-06-29T14:38:25.270941+0000","last_peered":"2024-06-29T14:38:25.270941+0000","last_clean":"2024-06-29T14:38:25.270941+0000","last_became_active":"2024-06-29T14:38:25.269922+0000","last_became_peered":"2024-06-29T14:38:25.269922+0000","last_unstale":"2024-06-29T14:38:25.270941+0000","last_undegraded":"2024-06-29T14:38:25.270941+0000","last_fullsized":"2024-06-29T14:38:25.270941+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:25:39.450578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.242584+0000","last_change":"2024-06-29T14:38:24.238508+0000","last_active":"2024-06-29T14:38:25.242584+0000","last_peered":"2024-06-29T14:38:25.242584+0000","last_clean":"2024-06-29T14:38:25.242584+0000","last_became_active":"2024-06-29T14:38:24.238160+0000","last_became_peered":"2024-06-29T14:38:24.238160+0000","last_unstale":"2024-06-29T14:38:25.242584+0000","last_undegraded":"2024-06-29T14:38:25.242584+0000","last_fullsized":"2024-06-29T14:38:25.242584+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:31.226738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"103'128","reported_seq":489,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.246878+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:38:25.246878+0000","last_peered":"2024-06-29T14:38:25.246878+0000","last_clean":"2024-06-29T14:38:25.246878+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:38:25.246878+0000","last_undegraded":"2024-06-29T14:38:25.246878+0000","last_fullsized":"2024-06-29T14:38:25.246878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":197,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":197,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542345,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59488,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312352,"statfs":{"total":96636764160,"available":96575848448,"internally_reserved":0,"allocated":1986560,"data_stored":1288052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542346,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1417216,"data_stored":706536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542345,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50600,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321240,"statfs":{"total":96636764160,"available":96584949760,"internally_reserved":0,"allocated":1396736,"data_stored":704320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542344,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1392640,"data_stored":697788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1396736,"data_stored":704320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575050,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58920,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312920,"statfs":{"total":96636764160,"available":96576430080,"internally_reserved":0,"allocated":1396736,"data_stored":704320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509641,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1388544,"data_stored":697698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509643,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37296,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334544,"statfs":{"total":96636764160,"available":96598573056,"internally_reserved":0,"allocated":2002944,"data_stored":1290314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:38:29.343 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:38:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.498+0000 7f99ebb11640 1 -- 172.21.15.59:0/3351439692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 msgr2=0x7f99e40b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.498+0000 7f99ebb11640 1 --2- 172.21.15.59:0/3351439692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 0x7f99e40b5880 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f99dc005fd0 tx=0x7f99dc02aac0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.498+0000 7f99ebb11640 1 -- 172.21.15.59:0/3351439692 shutdown_connections 2024-06-29T14:38:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.498+0000 7f99ebb11640 1 --2- 172.21.15.59:0/3351439692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 0x7f99e40b5880 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.498+0000 7f99ebb11640 1 --2- 172.21.15.59:0/3351439692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e41458a0 0x7f99e40cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.498+0000 7f99ebb11640 1 -- 172.21.15.59:0/3351439692 >> 172.21.15.59:0/3351439692 conn(0x7f99e406c950 msgr2=0x7f99e4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.498+0000 7f99ebb11640 1 -- 172.21.15.59:0/3351439692 shutdown_connections 2024-06-29T14:38:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.498+0000 7f99ebb11640 1 -- 172.21.15.59:0/3351439692 wait complete. 2024-06-29T14:38:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.500+0000 7f99ebb11640 1 Processor -- start 2024-06-29T14:38:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.500+0000 7f99ebb11640 1 -- start start 2024-06-29T14:38:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.501+0000 7f99ebb11640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e40b3740 0x7f99e40adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.501+0000 7f99ebb11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 0x7f99e415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99ebb11640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e415b4a0 0x7f99e40aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99ebb11640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f99e40b6150 con 0x7f99e40b3b50 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99ebb11640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99e40b5fd0 con 0x7f99e40b3740 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99ebb11640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f99e40b62d0 con 0x7f99e415b4a0 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 0x7f99e415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 0x7f99e415adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53300/0 (socket says 172.21.15.59:53300) 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 -- 172.21.15.59:0/2199167053 learned_addr learned my addr 172.21.15.59:0/2199167053 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99ea0ae640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e415b4a0 0x7f99e40aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e98ad640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e40b3740 0x7f99e40adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 -- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e415b4a0 msgr2=0x7f99e40aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e415b4a0 0x7f99e40aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 -- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e40b3740 msgr2=0x7f99e40adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e40b3740 0x7f99e40adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 -- 172.21.15.59:0/2199167053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e40b1020 con 0x7f99e40b3b50 2024-06-29T14:38:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99ea0ae640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e415b4a0 0x7f99e40aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e98ad640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e40b3740 0x7f99e40adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99e90ac640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 0x7f99e415adc0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f99d400c3c0 tx=0x7f99d400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99da7fc640 1 -- 172.21.15.59:0/2199167053 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d4017070 con 0x7f99e40b3b50 2024-06-29T14:38:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99da7fc640 1 -- 172.21.15.59:0/2199167053 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99d40053e0 con 0x7f99e40b3b50 2024-06-29T14:38:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e40b12c0 con 0x7f99e40b3b50 2024-06-29T14:38:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99da7fc640 1 -- 172.21.15.59:0/2199167053 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d4003850 con 0x7f99e40b3b50 2024-06-29T14:38:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.502+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e40ad740 con 0x7f99e40b3b50 2024-06-29T14:38:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.503+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e40b1490 con 0x7f99e40b3b50 2024-06-29T14:38:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.504+0000 7f99da7fc640 1 -- 172.21.15.59:0/2199167053 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f99d401a070 con 0x7f99e40b3b50 2024-06-29T14:38:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.505+0000 7f99da7fc640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f99c8076420 0x7f99c8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.505+0000 7f99da7fc640 1 -- 172.21.15.59:0/2199167053 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f99d4096e30 con 0x7f99e40b3b50 2024-06-29T14:38:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.505+0000 7f99e98ad640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f99c8076420 0x7f99c8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.506+0000 7f99e98ad640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f99c8076420 0x7f99c8078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005fd0 tx=0x7f99e000f360 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.509+0000 7f99da7fc640 1 -- 172.21.15.59:0/2199167053 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99d40603c0 con 0x7f99e40b3b50 2024-06-29T14:38:29.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.697+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 --> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f99e41458a0 con 0x7f99c8076420 2024-06-29T14:38:29.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.702+0000 7f99da7fc640 1 -- 172.21.15.59:0/2199167053 <== mgr.7132 v2:172.21.15.153:6832/1090607011 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252448 (secure 0 0 0) 0x7f99e41458a0 con 0x7f99c8076420 2024-06-29T14:38:29.702 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:29.703 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f99c8076420 msgr2=0x7f99c8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f99c8076420 0x7f99c8078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005fd0 tx=0x7f99e000f360 comp rx=0 tx=0).stop 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 msgr2=0x7f99e415adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 0x7f99e415adc0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f99d400c3c0 tx=0x7f99d400c880 comp rx=0 tx=0).stop 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 shutdown_connections 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f99c8076420 0x7f99c8078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99e415b4a0 0x7f99e40aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99e40b3b50 0x7f99e415adc0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 --2- 172.21.15.59:0/2199167053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99e40b3740 0x7f99e40adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 >> 172.21.15.59:0/2199167053 conn(0x7f99e406c950 msgr2=0x7f99e4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 shutdown_connections 2024-06-29T14:38:29.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.705+0000 7f99ebb11640 1 -- 172.21.15.59:0/2199167053 wait complete. 2024-06-29T14:38:29.714 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:38:28.740640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":242,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167,"ondisk_log_size":167,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":344820,"kb_used_data":12088,"kb_used_omap":12,"kb_used_meta":332595,"kb_avail":754629900,"statfs":{"total":773094113280,"available":772741017600,"internally_reserved":0,"allocated":12378112,"data_stored":6793348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":340577842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.522227"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.505064+0000","last_change":"2024-06-29T14:38:25.251254+0000","last_active":"2024-06-29T14:38:26.505064+0000","last_peered":"2024-06-29T14:38:26.505064+0000","last_clean":"2024-06-29T14:38:26.505064+0000","last_became_active":"2024-06-29T14:38:25.250919+0000","last_became_peered":"2024-06-29T14:38:25.250919+0000","last_unstale":"2024-06-29T14:38:26.505064+0000","last_undegraded":"2024-06-29T14:38:26.505064+0000","last_fullsized":"2024-06-29T14:38:26.505064+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:50:52.656540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258974+0000","last_change":"2024-06-29T14:38:25.260653+0000","last_active":"2024-06-29T14:38:26.258974+0000","last_peered":"2024-06-29T14:38:26.258974+0000","last_clean":"2024-06-29T14:38:26.258974+0000","last_became_active":"2024-06-29T14:38:25.260350+0000","last_became_peered":"2024-06-29T14:38:25.260350+0000","last_unstale":"2024-06-29T14:38:26.258974+0000","last_undegraded":"2024-06-29T14:38:26.258974+0000","last_fullsized":"2024-06-29T14:38:26.258974+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:46:11.716533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504872+0000","last_change":"2024-06-29T14:38:25.264427+0000","last_active":"2024-06-29T14:38:26.504872+0000","last_peered":"2024-06-29T14:38:26.504872+0000","last_clean":"2024-06-29T14:38:26.504872+0000","last_became_active":"2024-06-29T14:38:25.264072+0000","last_became_peered":"2024-06-29T14:38:25.264072+0000","last_unstale":"2024-06-29T14:38:26.504872+0000","last_undegraded":"2024-06-29T14:38:26.504872+0000","last_fullsized":"2024-06-29T14:38:26.504872+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:36.334047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266296+0000","last_change":"2024-06-29T14:38:25.263449+0000","last_active":"2024-06-29T14:38:25.266296+0000","last_peered":"2024-06-29T14:38:25.266296+0000","last_clean":"2024-06-29T14:38:25.266296+0000","last_became_active":"2024-06-29T14:38:25.262216+0000","last_became_peered":"2024-06-29T14:38:25.262216+0000","last_unstale":"2024-06-29T14:38:25.266296+0000","last_undegraded":"2024-06-29T14:38:25.266296+0000","last_fullsized":"2024-06-29T14:38:25.266296+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:50.699909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253701+0000","last_change":"2024-06-29T14:38:25.255488+0000","last_active":"2024-06-29T14:38:26.253701+0000","last_peered":"2024-06-29T14:38:26.253701+0000","last_clean":"2024-06-29T14:38:26.253701+0000","last_became_active":"2024-06-29T14:38:25.255169+0000","last_became_peered":"2024-06-29T14:38:25.255169+0000","last_unstale":"2024-06-29T14:38:26.253701+0000","last_undegraded":"2024-06-29T14:38:26.253701+0000","last_fullsized":"2024-06-29T14:38:26.253701+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:57.275912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.275022+0000","last_change":"2024-06-29T14:38:25.269662+0000","last_active":"2024-06-29T14:38:25.275022+0000","last_peered":"2024-06-29T14:38:25.275022+0000","last_clean":"2024-06-29T14:38:25.275022+0000","last_became_active":"2024-06-29T14:38:25.269330+0000","last_became_peered":"2024-06-29T14:38:25.269330+0000","last_unstale":"2024-06-29T14:38:25.275022+0000","last_undegraded":"2024-06-29T14:38:25.275022+0000","last_fullsized":"2024-06-29T14:38:25.275022+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:53.949358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.253234+0000","last_change":"2024-06-29T14:38:25.252640+0000","last_active":"2024-06-29T14:38:25.253234+0000","last_peered":"2024-06-29T14:38:25.253234+0000","last_clean":"2024-06-29T14:38:25.253234+0000","last_became_active":"2024-06-29T14:38:25.252245+0000","last_became_peered":"2024-06-29T14:38:25.252245+0000","last_unstale":"2024-06-29T14:38:25.253234+0000","last_undegraded":"2024-06-29T14:38:25.253234+0000","last_fullsized":"2024-06-29T14:38:25.253234+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:35:11.434093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.262486+0000","last_change":"2024-06-29T14:38:25.255036+0000","last_active":"2024-06-29T14:38:25.262486+0000","last_peered":"2024-06-29T14:38:25.262486+0000","last_clean":"2024-06-29T14:38:25.262486+0000","last_became_active":"2024-06-29T14:38:25.254537+0000","last_became_peered":"2024-06-29T14:38:25.254537+0000","last_unstale":"2024-06-29T14:38:25.262486+0000","last_undegraded":"2024-06-29T14:38:25.262486+0000","last_fullsized":"2024-06-29T14:38:25.262486+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:14:05.543579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253930+0000","last_change":"2024-06-29T14:38:25.252228+0000","last_active":"2024-06-29T14:38:26.253930+0000","last_peered":"2024-06-29T14:38:26.253930+0000","last_clean":"2024-06-29T14:38:26.253930+0000","last_became_active":"2024-06-29T14:38:25.251771+0000","last_became_peered":"2024-06-29T14:38:25.251771+0000","last_unstale":"2024-06-29T14:38:26.253930+0000","last_undegraded":"2024-06-29T14:38:26.253930+0000","last_fullsized":"2024-06-29T14:38:26.253930+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:58.577643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252631+0000","last_change":"2024-06-29T14:38:25.250901+0000","last_active":"2024-06-29T14:38:26.252631+0000","last_peered":"2024-06-29T14:38:26.252631+0000","last_clean":"2024-06-29T14:38:26.252631+0000","last_became_active":"2024-06-29T14:38:25.250583+0000","last_became_peered":"2024-06-29T14:38:25.250583+0000","last_unstale":"2024-06-29T14:38:26.252631+0000","last_undegraded":"2024-06-29T14:38:26.252631+0000","last_fullsized":"2024-06-29T14:38:26.252631+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:53:59.235820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254676+0000","last_change":"2024-06-29T14:38:25.258357+0000","last_active":"2024-06-29T14:38:26.254676+0000","last_peered":"2024-06-29T14:38:26.254676+0000","last_clean":"2024-06-29T14:38:26.254676+0000","last_became_active":"2024-06-29T14:38:25.256883+0000","last_became_peered":"2024-06-29T14:38:25.256883+0000","last_unstale":"2024-06-29T14:38:26.254676+0000","last_undegraded":"2024-06-29T14:38:26.254676+0000","last_fullsized":"2024-06-29T14:38:26.254676+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:37:17.660568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252233+0000","last_change":"2024-06-29T14:38:25.243893+0000","last_active":"2024-06-29T14:38:26.252233+0000","last_peered":"2024-06-29T14:38:26.252233+0000","last_clean":"2024-06-29T14:38:26.252233+0000","last_became_active":"2024-06-29T14:38:25.243515+0000","last_became_peered":"2024-06-29T14:38:25.243515+0000","last_unstale":"2024-06-29T14:38:26.252233+0000","last_undegraded":"2024-06-29T14:38:26.252233+0000","last_fullsized":"2024-06-29T14:38:26.252233+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:15.690906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.264263+0000","last_change":"2024-06-29T14:38:25.254080+0000","last_active":"2024-06-29T14:38:25.264263+0000","last_peered":"2024-06-29T14:38:25.264263+0000","last_clean":"2024-06-29T14:38:25.264263+0000","last_became_active":"2024-06-29T14:38:25.253671+0000","last_became_peered":"2024-06-29T14:38:25.253671+0000","last_unstale":"2024-06-29T14:38:25.264263+0000","last_undegraded":"2024-06-29T14:38:25.264263+0000","last_fullsized":"2024-06-29T14:38:25.264263+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:49.583434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253521+0000","last_change":"2024-06-29T14:38:25.270035+0000","last_active":"2024-06-29T14:38:26.253521+0000","last_peered":"2024-06-29T14:38:26.253521+0000","last_clean":"2024-06-29T14:38:26.253521+0000","last_became_active":"2024-06-29T14:38:25.269583+0000","last_became_peered":"2024-06-29T14:38:25.269583+0000","last_unstale":"2024-06-29T14:38:26.253521+0000","last_undegraded":"2024-06-29T14:38:26.253521+0000","last_fullsized":"2024-06-29T14:38:26.253521+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:18:27.204498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.267015+0000","last_change":"2024-06-29T14:38:25.263566+0000","last_active":"2024-06-29T14:38:25.267015+0000","last_peered":"2024-06-29T14:38:25.267015+0000","last_clean":"2024-06-29T14:38:25.267015+0000","last_became_active":"2024-06-29T14:38:25.263220+0000","last_became_peered":"2024-06-29T14:38:25.263220+0000","last_unstale":"2024-06-29T14:38:25.267015+0000","last_undegraded":"2024-06-29T14:38:25.267015+0000","last_fullsized":"2024-06-29T14:38:25.267015+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:47.144025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276282+0000","last_change":"2024-06-29T14:38:25.273414+0000","last_active":"2024-06-29T14:38:25.276282+0000","last_peered":"2024-06-29T14:38:25.276282+0000","last_clean":"2024-06-29T14:38:25.276282+0000","last_became_active":"2024-06-29T14:38:25.273072+0000","last_became_peered":"2024-06-29T14:38:25.273072+0000","last_unstale":"2024-06-29T14:38:25.276282+0000","last_undegraded":"2024-06-29T14:38:25.276282+0000","last_fullsized":"2024-06-29T14:38:25.276282+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:36.384799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276046+0000","last_change":"2024-06-29T14:38:25.273963+0000","last_active":"2024-06-29T14:38:25.276046+0000","last_peered":"2024-06-29T14:38:25.276046+0000","last_clean":"2024-06-29T14:38:25.276046+0000","last_became_active":"2024-06-29T14:38:25.272976+0000","last_became_peered":"2024-06-29T14:38:25.272976+0000","last_unstale":"2024-06-29T14:38:25.276046+0000","last_undegraded":"2024-06-29T14:38:25.276046+0000","last_fullsized":"2024-06-29T14:38:25.276046+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:21.083178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504923+0000","last_change":"2024-06-29T14:38:25.257391+0000","last_active":"2024-06-29T14:38:26.504923+0000","last_peered":"2024-06-29T14:38:26.504923+0000","last_clean":"2024-06-29T14:38:26.504923+0000","last_became_active":"2024-06-29T14:38:25.257054+0000","last_became_peered":"2024-06-29T14:38:25.257054+0000","last_unstale":"2024-06-29T14:38:26.504923+0000","last_undegraded":"2024-06-29T14:38:26.504923+0000","last_fullsized":"2024-06-29T14:38:26.504923+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:32.242764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254714+0000","last_change":"2024-06-29T14:38:25.256102+0000","last_active":"2024-06-29T14:38:26.254714+0000","last_peered":"2024-06-29T14:38:26.254714+0000","last_clean":"2024-06-29T14:38:26.254714+0000","last_became_active":"2024-06-29T14:38:25.254366+0000","last_became_peered":"2024-06-29T14:38:25.254366+0000","last_unstale":"2024-06-29T14:38:26.254714+0000","last_undegraded":"2024-06-29T14:38:26.254714+0000","last_fullsized":"2024-06-29T14:38:26.254714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:41:53.035546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.267850+0000","last_change":"2024-06-29T14:38:25.264084+0000","last_active":"2024-06-29T14:38:25.267850+0000","last_peered":"2024-06-29T14:38:25.267850+0000","last_clean":"2024-06-29T14:38:25.267850+0000","last_became_active":"2024-06-29T14:38:25.263745+0000","last_became_peered":"2024-06-29T14:38:25.263745+0000","last_unstale":"2024-06-29T14:38:25.267850+0000","last_undegraded":"2024-06-29T14:38:25.267850+0000","last_fullsized":"2024-06-29T14:38:25.267850+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:39.430531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254186+0000","last_change":"2024-06-29T14:38:25.254695+0000","last_active":"2024-06-29T14:38:26.254186+0000","last_peered":"2024-06-29T14:38:26.254186+0000","last_clean":"2024-06-29T14:38:26.254186+0000","last_became_active":"2024-06-29T14:38:25.253362+0000","last_became_peered":"2024-06-29T14:38:25.253362+0000","last_unstale":"2024-06-29T14:38:26.254186+0000","last_undegraded":"2024-06-29T14:38:26.254186+0000","last_fullsized":"2024-06-29T14:38:26.254186+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:04:19.098699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253723+0000","last_change":"2024-06-29T14:38:25.253571+0000","last_active":"2024-06-29T14:38:26.253723+0000","last_peered":"2024-06-29T14:38:26.253723+0000","last_clean":"2024-06-29T14:38:26.253723+0000","last_became_active":"2024-06-29T14:38:25.253255+0000","last_became_peered":"2024-06-29T14:38:25.253255+0000","last_unstale":"2024-06-29T14:38:26.253723+0000","last_undegraded":"2024-06-29T14:38:26.253723+0000","last_fullsized":"2024-06-29T14:38:26.253723+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:31:19.873490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253253+0000","last_change":"2024-06-29T14:38:25.265270+0000","last_active":"2024-06-29T14:38:26.253253+0000","last_peered":"2024-06-29T14:38:26.253253+0000","last_clean":"2024-06-29T14:38:26.253253+0000","last_became_active":"2024-06-29T14:38:25.264939+0000","last_became_peered":"2024-06-29T14:38:25.264939+0000","last_unstale":"2024-06-29T14:38:26.253253+0000","last_undegraded":"2024-06-29T14:38:26.253253+0000","last_fullsized":"2024-06-29T14:38:26.253253+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:21:47.305292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504797+0000","last_change":"2024-06-29T14:38:25.263039+0000","last_active":"2024-06-29T14:38:26.504797+0000","last_peered":"2024-06-29T14:38:26.504797+0000","last_clean":"2024-06-29T14:38:26.504797+0000","last_became_active":"2024-06-29T14:38:25.262709+0000","last_became_peered":"2024-06-29T14:38:25.262709+0000","last_unstale":"2024-06-29T14:38:26.504797+0000","last_undegraded":"2024-06-29T14:38:26.504797+0000","last_fullsized":"2024-06-29T14:38:26.504797+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:31.041814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.259150+0000","last_change":"2024-06-29T14:38:25.256751+0000","last_active":"2024-06-29T14:38:26.259150+0000","last_peered":"2024-06-29T14:38:26.259150+0000","last_clean":"2024-06-29T14:38:26.259150+0000","last_became_active":"2024-06-29T14:38:25.256315+0000","last_became_peered":"2024-06-29T14:38:25.256315+0000","last_unstale":"2024-06-29T14:38:26.259150+0000","last_undegraded":"2024-06-29T14:38:26.259150+0000","last_fullsized":"2024-06-29T14:38:26.259150+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:27.779944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254436+0000","last_change":"2024-06-29T14:38:25.269405+0000","last_active":"2024-06-29T14:38:26.254436+0000","last_peered":"2024-06-29T14:38:26.254436+0000","last_clean":"2024-06-29T14:38:26.254436+0000","last_became_active":"2024-06-29T14:38:25.269079+0000","last_became_peered":"2024-06-29T14:38:25.269079+0000","last_unstale":"2024-06-29T14:38:26.254436+0000","last_undegraded":"2024-06-29T14:38:26.254436+0000","last_fullsized":"2024-06-29T14:38:26.254436+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:32.358721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252356+0000","last_change":"2024-06-29T14:38:25.246307+0000","last_active":"2024-06-29T14:38:26.252356+0000","last_peered":"2024-06-29T14:38:26.252356+0000","last_clean":"2024-06-29T14:38:26.252356+0000","last_became_active":"2024-06-29T14:38:25.243960+0000","last_became_peered":"2024-06-29T14:38:25.243960+0000","last_unstale":"2024-06-29T14:38:26.252356+0000","last_undegraded":"2024-06-29T14:38:26.252356+0000","last_fullsized":"2024-06-29T14:38:26.252356+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:16.044436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.272649+0000","last_change":"2024-06-29T14:38:25.267684+0000","last_active":"2024-06-29T14:38:25.272649+0000","last_peered":"2024-06-29T14:38:25.272649+0000","last_clean":"2024-06-29T14:38:25.272649+0000","last_became_active":"2024-06-29T14:38:25.267324+0000","last_became_peered":"2024-06-29T14:38:25.267324+0000","last_unstale":"2024-06-29T14:38:25.272649+0000","last_undegraded":"2024-06-29T14:38:25.272649+0000","last_fullsized":"2024-06-29T14:38:25.272649+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:02.422398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.251400+0000","last_change":"2024-06-29T14:38:25.250278+0000","last_active":"2024-06-29T14:38:25.251400+0000","last_peered":"2024-06-29T14:38:25.251400+0000","last_clean":"2024-06-29T14:38:25.251400+0000","last_became_active":"2024-06-29T14:38:25.249953+0000","last_became_peered":"2024-06-29T14:38:25.249953+0000","last_unstale":"2024-06-29T14:38:25.251400+0000","last_undegraded":"2024-06-29T14:38:25.251400+0000","last_fullsized":"2024-06-29T14:38:25.251400+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:00:03.882845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254229+0000","last_change":"2024-06-29T14:38:25.252943+0000","last_active":"2024-06-29T14:38:26.254229+0000","last_peered":"2024-06-29T14:38:26.254229+0000","last_clean":"2024-06-29T14:38:26.254229+0000","last_became_active":"2024-06-29T14:38:25.251716+0000","last_became_peered":"2024-06-29T14:38:25.251716+0000","last_unstale":"2024-06-29T14:38:26.254229+0000","last_undegraded":"2024-06-29T14:38:26.254229+0000","last_fullsized":"2024-06-29T14:38:26.254229+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:41.771814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258531+0000","last_change":"2024-06-29T14:38:25.267924+0000","last_active":"2024-06-29T14:38:26.258531+0000","last_peered":"2024-06-29T14:38:26.258531+0000","last_clean":"2024-06-29T14:38:26.258531+0000","last_became_active":"2024-06-29T14:38:25.267655+0000","last_became_peered":"2024-06-29T14:38:25.267655+0000","last_unstale":"2024-06-29T14:38:26.258531+0000","last_undegraded":"2024-06-29T14:38:26.258531+0000","last_fullsized":"2024-06-29T14:38:26.258531+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:41:45.854837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.277128+0000","last_change":"2024-06-29T14:38:25.274341+0000","last_active":"2024-06-29T14:38:25.277128+0000","last_peered":"2024-06-29T14:38:25.277128+0000","last_clean":"2024-06-29T14:38:25.277128+0000","last_became_active":"2024-06-29T14:38:25.273809+0000","last_became_peered":"2024-06-29T14:38:25.273809+0000","last_unstale":"2024-06-29T14:38:25.277128+0000","last_undegraded":"2024-06-29T14:38:25.277128+0000","last_fullsized":"2024-06-29T14:38:25.277128+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:24:14.699898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.267225+0000","last_change":"2024-06-29T14:38:25.260039+0000","last_active":"2024-06-29T14:38:25.267225+0000","last_peered":"2024-06-29T14:38:25.267225+0000","last_clean":"2024-06-29T14:38:25.267225+0000","last_became_active":"2024-06-29T14:38:25.259705+0000","last_became_peered":"2024-06-29T14:38:25.259705+0000","last_unstale":"2024-06-29T14:38:25.267225+0000","last_undegraded":"2024-06-29T14:38:25.267225+0000","last_fullsized":"2024-06-29T14:38:25.267225+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:04:29.412189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"113'4","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254572+0000","last_change":"2024-06-29T14:38:24.246482+0000","last_active":"2024-06-29T14:38:26.254572+0000","last_peered":"2024-06-29T14:38:26.254572+0000","last_clean":"2024-06-29T14:38:26.254572+0000","last_became_active":"2024-06-29T14:38:24.246228+0000","last_became_peered":"2024-06-29T14:38:24.246228+0000","last_unstale":"2024-06-29T14:38:26.254572+0000","last_undegraded":"2024-06-29T14:38:26.254572+0000","last_fullsized":"2024-06-29T14:38:26.254572+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:55.908391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266482+0000","last_change":"2024-06-29T14:38:25.261983+0000","last_active":"2024-06-29T14:38:25.266482+0000","last_peered":"2024-06-29T14:38:25.266482+0000","last_clean":"2024-06-29T14:38:25.266482+0000","last_became_active":"2024-06-29T14:38:25.261676+0000","last_became_peered":"2024-06-29T14:38:25.261676+0000","last_unstale":"2024-06-29T14:38:25.266482+0000","last_undegraded":"2024-06-29T14:38:25.266482+0000","last_fullsized":"2024-06-29T14:38:25.266482+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:44.155424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254384+0000","last_change":"2024-06-29T14:38:24.236757+0000","last_active":"2024-06-29T14:38:26.254384+0000","last_peered":"2024-06-29T14:38:26.254384+0000","last_clean":"2024-06-29T14:38:26.254384+0000","last_became_active":"2024-06-29T14:38:24.236433+0000","last_became_peered":"2024-06-29T14:38:24.236433+0000","last_unstale":"2024-06-29T14:38:26.254384+0000","last_undegraded":"2024-06-29T14:38:26.254384+0000","last_fullsized":"2024-06-29T14:38:26.254384+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:31.110203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.270299+0000","last_change":"2024-06-29T14:38:25.269356+0000","last_active":"2024-06-29T14:38:25.270299+0000","last_peered":"2024-06-29T14:38:25.270299+0000","last_clean":"2024-06-29T14:38:25.270299+0000","last_became_active":"2024-06-29T14:38:25.269005+0000","last_became_peered":"2024-06-29T14:38:25.269005+0000","last_unstale":"2024-06-29T14:38:25.270299+0000","last_undegraded":"2024-06-29T14:38:25.270299+0000","last_fullsized":"2024-06-29T14:38:25.270299+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:56.668689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"113'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266510+0000","last_change":"2024-06-29T14:38:24.241221+0000","last_active":"2024-06-29T14:38:25.266510+0000","last_peered":"2024-06-29T14:38:25.266510+0000","last_clean":"2024-06-29T14:38:25.266510+0000","last_became_active":"2024-06-29T14:38:24.240897+0000","last_became_peered":"2024-06-29T14:38:24.240897+0000","last_unstale":"2024-06-29T14:38:25.266510+0000","last_undegraded":"2024-06-29T14:38:25.266510+0000","last_fullsized":"2024-06-29T14:38:25.266510+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:06:47.300493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.260213+0000","last_change":"2024-06-29T14:38:25.256937+0000","last_active":"2024-06-29T14:38:25.260213+0000","last_peered":"2024-06-29T14:38:25.260213+0000","last_clean":"2024-06-29T14:38:25.260213+0000","last_became_active":"2024-06-29T14:38:25.256610+0000","last_became_peered":"2024-06-29T14:38:25.256610+0000","last_unstale":"2024-06-29T14:38:25.260213+0000","last_undegraded":"2024-06-29T14:38:25.260213+0000","last_fullsized":"2024-06-29T14:38:25.260213+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:11.574455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254236+0000","last_change":"2024-06-29T14:38:24.238658+0000","last_active":"2024-06-29T14:38:26.254236+0000","last_peered":"2024-06-29T14:38:26.254236+0000","last_clean":"2024-06-29T14:38:26.254236+0000","last_became_active":"2024-06-29T14:38:24.238287+0000","last_became_peered":"2024-06-29T14:38:24.238287+0000","last_unstale":"2024-06-29T14:38:26.254236+0000","last_undegraded":"2024-06-29T14:38:26.254236+0000","last_fullsized":"2024-06-29T14:38:26.254236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:31.110203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.265786+0000","last_change":"2024-06-29T14:38:25.262627+0000","last_active":"2024-06-29T14:38:25.265786+0000","last_peered":"2024-06-29T14:38:25.265786+0000","last_clean":"2024-06-29T14:38:25.265786+0000","last_became_active":"2024-06-29T14:38:25.262247+0000","last_became_peered":"2024-06-29T14:38:25.262247+0000","last_unstale":"2024-06-29T14:38:25.265786+0000","last_undegraded":"2024-06-29T14:38:25.265786+0000","last_fullsized":"2024-06-29T14:38:25.265786+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:47.811090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253872+0000","last_change":"2024-06-29T14:38:24.240712+0000","last_active":"2024-06-29T14:38:26.253872+0000","last_peered":"2024-06-29T14:38:26.253872+0000","last_clean":"2024-06-29T14:38:26.253872+0000","last_became_active":"2024-06-29T14:38:24.240398+0000","last_became_peered":"2024-06-29T14:38:24.240398+0000","last_unstale":"2024-06-29T14:38:26.253872+0000","last_undegraded":"2024-06-29T14:38:26.253872+0000","last_fullsized":"2024-06-29T14:38:26.253872+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:50:51.069888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253254+0000","last_change":"2024-06-29T14:38:25.253336+0000","last_active":"2024-06-29T14:38:26.253254+0000","last_peered":"2024-06-29T14:38:26.253254+0000","last_clean":"2024-06-29T14:38:26.253254+0000","last_became_active":"2024-06-29T14:38:25.252964+0000","last_became_peered":"2024-06-29T14:38:25.252964+0000","last_unstale":"2024-06-29T14:38:26.253254+0000","last_undegraded":"2024-06-29T14:38:26.253254+0000","last_fullsized":"2024-06-29T14:38:26.253254+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:42:50.419536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"113'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.279054+0000","last_change":"2024-06-29T14:38:24.241135+0000","last_active":"2024-06-29T14:38:25.279054+0000","last_peered":"2024-06-29T14:38:25.279054+0000","last_clean":"2024-06-29T14:38:25.279054+0000","last_became_active":"2024-06-29T14:38:24.240821+0000","last_became_peered":"2024-06-29T14:38:24.240821+0000","last_unstale":"2024-06-29T14:38:25.279054+0000","last_undegraded":"2024-06-29T14:38:25.279054+0000","last_fullsized":"2024-06-29T14:38:25.279054+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:15.375607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.273069+0000","last_change":"2024-06-29T14:38:25.268819+0000","last_active":"2024-06-29T14:38:25.273069+0000","last_peered":"2024-06-29T14:38:25.273069+0000","last_clean":"2024-06-29T14:38:25.273069+0000","last_became_active":"2024-06-29T14:38:25.268493+0000","last_became_peered":"2024-06-29T14:38:25.268493+0000","last_unstale":"2024-06-29T14:38:25.273069+0000","last_undegraded":"2024-06-29T14:38:25.273069+0000","last_fullsized":"2024-06-29T14:38:25.273069+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:00.758863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.505016+0000","last_change":"2024-06-29T14:38:24.234597+0000","last_active":"2024-06-29T14:38:26.505016+0000","last_peered":"2024-06-29T14:38:26.505016+0000","last_clean":"2024-06-29T14:38:26.505016+0000","last_became_active":"2024-06-29T14:38:24.234219+0000","last_became_peered":"2024-06-29T14:38:24.234219+0000","last_unstale":"2024-06-29T14:38:26.505016+0000","last_undegraded":"2024-06-29T14:38:26.505016+0000","last_fullsized":"2024-06-29T14:38:26.505016+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:24:13.693101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266686+0000","last_change":"2024-06-29T14:38:25.263143+0000","last_active":"2024-06-29T14:38:25.266686+0000","last_peered":"2024-06-29T14:38:25.266686+0000","last_clean":"2024-06-29T14:38:25.266686+0000","last_became_active":"2024-06-29T14:38:25.262782+0000","last_became_peered":"2024-06-29T14:38:25.262782+0000","last_unstale":"2024-06-29T14:38:25.266686+0000","last_undegraded":"2024-06-29T14:38:25.266686+0000","last_fullsized":"2024-06-29T14:38:25.266686+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:07.068662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.246801+0000","last_change":"2024-06-29T14:38:24.247874+0000","last_active":"2024-06-29T14:38:25.246801+0000","last_peered":"2024-06-29T14:38:25.246801+0000","last_clean":"2024-06-29T14:38:25.246801+0000","last_became_active":"2024-06-29T14:38:24.247440+0000","last_became_peered":"2024-06-29T14:38:24.247440+0000","last_unstale":"2024-06-29T14:38:25.246801+0000","last_undegraded":"2024-06-29T14:38:25.246801+0000","last_fullsized":"2024-06-29T14:38:25.246801+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:37:38.067294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253932+0000","last_change":"2024-06-29T14:38:25.255741+0000","last_active":"2024-06-29T14:38:26.253932+0000","last_peered":"2024-06-29T14:38:26.253932+0000","last_clean":"2024-06-29T14:38:26.253932+0000","last_became_active":"2024-06-29T14:38:25.253494+0000","last_became_peered":"2024-06-29T14:38:25.253494+0000","last_unstale":"2024-06-29T14:38:26.253932+0000","last_undegraded":"2024-06-29T14:38:26.253932+0000","last_fullsized":"2024-06-29T14:38:26.253932+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:58.283980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.259270+0000","last_change":"2024-06-29T14:38:24.239910+0000","last_active":"2024-06-29T14:38:26.259270+0000","last_peered":"2024-06-29T14:38:26.259270+0000","last_clean":"2024-06-29T14:38:26.259270+0000","last_became_active":"2024-06-29T14:38:24.239581+0000","last_became_peered":"2024-06-29T14:38:24.239581+0000","last_unstale":"2024-06-29T14:38:26.259270+0000","last_undegraded":"2024-06-29T14:38:26.259270+0000","last_fullsized":"2024-06-29T14:38:26.259270+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:31.235967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.257479+0000","last_change":"2024-06-29T14:38:25.256951+0000","last_active":"2024-06-29T14:38:25.257479+0000","last_peered":"2024-06-29T14:38:25.257479+0000","last_clean":"2024-06-29T14:38:25.257479+0000","last_became_active":"2024-06-29T14:38:25.256618+0000","last_became_peered":"2024-06-29T14:38:25.256618+0000","last_unstale":"2024-06-29T14:38:25.257479+0000","last_undegraded":"2024-06-29T14:38:25.257479+0000","last_fullsized":"2024-06-29T14:38:25.257479+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:06:57.038946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.505001+0000","last_change":"2024-06-29T14:38:24.234071+0000","last_active":"2024-06-29T14:38:26.505001+0000","last_peered":"2024-06-29T14:38:26.505001+0000","last_clean":"2024-06-29T14:38:26.505001+0000","last_became_active":"2024-06-29T14:38:24.233724+0000","last_became_peered":"2024-06-29T14:38:24.233724+0000","last_unstale":"2024-06-29T14:38:26.505001+0000","last_undegraded":"2024-06-29T14:38:26.505001+0000","last_fullsized":"2024-06-29T14:38:26.505001+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:07:29.952013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504974+0000","last_change":"2024-06-29T14:38:25.262192+0000","last_active":"2024-06-29T14:38:26.504974+0000","last_peered":"2024-06-29T14:38:26.504974+0000","last_clean":"2024-06-29T14:38:26.504974+0000","last_became_active":"2024-06-29T14:38:25.261843+0000","last_became_peered":"2024-06-29T14:38:25.261843+0000","last_unstale":"2024-06-29T14:38:26.504974+0000","last_undegraded":"2024-06-29T14:38:26.504974+0000","last_fullsized":"2024-06-29T14:38:26.504974+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:11.203600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"113'3","reported_seq":17,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254852+0000","last_change":"2024-06-29T14:38:24.247790+0000","last_active":"2024-06-29T14:38:26.254852+0000","last_peered":"2024-06-29T14:38:26.254852+0000","last_clean":"2024-06-29T14:38:26.254852+0000","last_became_active":"2024-06-29T14:38:24.247492+0000","last_became_peered":"2024-06-29T14:38:24.247492+0000","last_unstale":"2024-06-29T14:38:26.254852+0000","last_undegraded":"2024-06-29T14:38:26.254852+0000","last_fullsized":"2024-06-29T14:38:26.254852+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:07.555389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266873+0000","last_change":"2024-06-29T14:38:25.261727+0000","last_active":"2024-06-29T14:38:25.266873+0000","last_peered":"2024-06-29T14:38:25.266873+0000","last_clean":"2024-06-29T14:38:25.266873+0000","last_became_active":"2024-06-29T14:38:25.260935+0000","last_became_peered":"2024-06-29T14:38:25.260935+0000","last_unstale":"2024-06-29T14:38:25.266873+0000","last_undegraded":"2024-06-29T14:38:25.266873+0000","last_fullsized":"2024-06-29T14:38:25.266873+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:18:47.480133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"113'6","reported_seq":18,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.279096+0000","last_change":"2024-06-29T14:38:24.240662+0000","last_active":"2024-06-29T14:38:25.279096+0000","last_peered":"2024-06-29T14:38:25.279096+0000","last_clean":"2024-06-29T14:38:25.279096+0000","last_became_active":"2024-06-29T14:38:24.240227+0000","last_became_peered":"2024-06-29T14:38:24.240227+0000","last_unstale":"2024-06-29T14:38:25.279096+0000","last_undegraded":"2024-06-29T14:38:25.279096+0000","last_fullsized":"2024-06-29T14:38:25.279096+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:14:06.509455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.262817+0000","last_change":"2024-06-29T14:38:25.254856+0000","last_active":"2024-06-29T14:38:25.262817+0000","last_peered":"2024-06-29T14:38:25.262817+0000","last_clean":"2024-06-29T14:38:25.262817+0000","last_became_active":"2024-06-29T14:38:25.254119+0000","last_became_peered":"2024-06-29T14:38:25.254119+0000","last_unstale":"2024-06-29T14:38:25.262817+0000","last_undegraded":"2024-06-29T14:38:25.262817+0000","last_fullsized":"2024-06-29T14:38:25.262817+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:55.347262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"113'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.277664+0000","last_change":"2024-06-29T14:38:24.247772+0000","last_active":"2024-06-29T14:38:25.277664+0000","last_peered":"2024-06-29T14:38:25.277664+0000","last_clean":"2024-06-29T14:38:25.277664+0000","last_became_active":"2024-06-29T14:38:24.247394+0000","last_became_peered":"2024-06-29T14:38:24.247394+0000","last_unstale":"2024-06-29T14:38:25.277664+0000","last_undegraded":"2024-06-29T14:38:25.277664+0000","last_fullsized":"2024-06-29T14:38:25.277664+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:25:43.149108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254426+0000","last_change":"2024-06-29T14:38:25.252014+0000","last_active":"2024-06-29T14:38:26.254426+0000","last_peered":"2024-06-29T14:38:26.254426+0000","last_clean":"2024-06-29T14:38:26.254426+0000","last_became_active":"2024-06-29T14:38:25.251666+0000","last_became_peered":"2024-06-29T14:38:25.251666+0000","last_unstale":"2024-06-29T14:38:26.254426+0000","last_undegraded":"2024-06-29T14:38:26.254426+0000","last_fullsized":"2024-06-29T14:38:26.254426+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:55:35.230309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.251493+0000","last_change":"2024-06-29T14:38:24.246143+0000","last_active":"2024-06-29T14:38:25.251493+0000","last_peered":"2024-06-29T14:38:25.251493+0000","last_clean":"2024-06-29T14:38:25.251493+0000","last_became_active":"2024-06-29T14:38:24.245765+0000","last_became_peered":"2024-06-29T14:38:24.245765+0000","last_unstale":"2024-06-29T14:38:25.251493+0000","last_undegraded":"2024-06-29T14:38:25.251493+0000","last_fullsized":"2024-06-29T14:38:25.251493+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:02:10.504001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252071+0000","last_change":"2024-06-29T14:38:25.251826+0000","last_active":"2024-06-29T14:38:26.252071+0000","last_peered":"2024-06-29T14:38:26.252071+0000","last_clean":"2024-06-29T14:38:26.252071+0000","last_became_active":"2024-06-29T14:38:25.251516+0000","last_became_peered":"2024-06-29T14:38:25.251516+0000","last_unstale":"2024-06-29T14:38:26.252071+0000","last_undegraded":"2024-06-29T14:38:26.252071+0000","last_fullsized":"2024-06-29T14:38:26.252071+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:47.200322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.241404+0000","last_change":"2024-06-29T14:38:24.235783+0000","last_active":"2024-06-29T14:38:25.241404+0000","last_peered":"2024-06-29T14:38:25.241404+0000","last_clean":"2024-06-29T14:38:25.241404+0000","last_became_active":"2024-06-29T14:38:24.235437+0000","last_became_peered":"2024-06-29T14:38:24.235437+0000","last_unstale":"2024-06-29T14:38:25.241404+0000","last_undegraded":"2024-06-29T14:38:25.241404+0000","last_fullsized":"2024-06-29T14:38:25.241404+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:19.676731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252129+0000","last_change":"2024-06-29T14:38:25.264403+0000","last_active":"2024-06-29T14:38:26.252129+0000","last_peered":"2024-06-29T14:38:26.252129+0000","last_clean":"2024-06-29T14:38:26.252129+0000","last_became_active":"2024-06-29T14:38:25.263244+0000","last_became_peered":"2024-06-29T14:38:25.263244+0000","last_unstale":"2024-06-29T14:38:26.252129+0000","last_undegraded":"2024-06-29T14:38:26.252129+0000","last_fullsized":"2024-06-29T14:38:26.252129+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:28.522036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"113'8","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.280217+0000","last_change":"2024-06-29T14:38:24.242860+0000","last_active":"2024-06-29T14:38:25.280217+0000","last_peered":"2024-06-29T14:38:25.280217+0000","last_clean":"2024-06-29T14:38:25.280217+0000","last_became_active":"2024-06-29T14:38:24.242551+0000","last_became_peered":"2024-06-29T14:38:24.242551+0000","last_unstale":"2024-06-29T14:38:25.280217+0000","last_undegraded":"2024-06-29T14:38:25.280217+0000","last_fullsized":"2024-06-29T14:38:25.280217+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:01.650803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.267200+0000","last_change":"2024-06-29T14:38:25.257913+0000","last_active":"2024-06-29T14:38:25.267200+0000","last_peered":"2024-06-29T14:38:25.267200+0000","last_clean":"2024-06-29T14:38:25.267200+0000","last_became_active":"2024-06-29T14:38:25.257587+0000","last_became_peered":"2024-06-29T14:38:25.257587+0000","last_unstale":"2024-06-29T14:38:25.267200+0000","last_undegraded":"2024-06-29T14:38:25.267200+0000","last_fullsized":"2024-06-29T14:38:25.267200+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:33.252621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254650+0000","last_change":"2024-06-29T14:38:24.235635+0000","last_active":"2024-06-29T14:38:26.254650+0000","last_peered":"2024-06-29T14:38:26.254650+0000","last_clean":"2024-06-29T14:38:26.254650+0000","last_became_active":"2024-06-29T14:38:24.235329+0000","last_became_peered":"2024-06-29T14:38:24.235329+0000","last_unstale":"2024-06-29T14:38:26.254650+0000","last_undegraded":"2024-06-29T14:38:26.254650+0000","last_fullsized":"2024-06-29T14:38:26.254650+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:23.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253719+0000","last_change":"2024-06-29T14:38:25.250431+0000","last_active":"2024-06-29T14:38:26.253719+0000","last_peered":"2024-06-29T14:38:26.253719+0000","last_clean":"2024-06-29T14:38:26.253719+0000","last_became_active":"2024-06-29T14:38:25.250125+0000","last_became_peered":"2024-06-29T14:38:25.250125+0000","last_unstale":"2024-06-29T14:38:26.253719+0000","last_undegraded":"2024-06-29T14:38:26.253719+0000","last_fullsized":"2024-06-29T14:38:26.253719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:48.095292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"113'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.264464+0000","last_change":"2024-06-29T14:38:24.236235+0000","last_active":"2024-06-29T14:38:25.264464+0000","last_peered":"2024-06-29T14:38:25.264464+0000","last_clean":"2024-06-29T14:38:25.264464+0000","last_became_active":"2024-06-29T14:38:24.235850+0000","last_became_peered":"2024-06-29T14:38:24.235850+0000","last_unstale":"2024-06-29T14:38:25.264464+0000","last_undegraded":"2024-06-29T14:38:25.264464+0000","last_fullsized":"2024-06-29T14:38:25.264464+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:14.484323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254258+0000","last_change":"2024-06-29T14:38:25.253942+0000","last_active":"2024-06-29T14:38:26.254258+0000","last_peered":"2024-06-29T14:38:26.254258+0000","last_clean":"2024-06-29T14:38:26.254258+0000","last_became_active":"2024-06-29T14:38:25.253266+0000","last_became_peered":"2024-06-29T14:38:25.253266+0000","last_unstale":"2024-06-29T14:38:26.254258+0000","last_undegraded":"2024-06-29T14:38:26.254258+0000","last_fullsized":"2024-06-29T14:38:26.254258+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:26:22.525190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"113'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254999+0000","last_change":"2024-06-29T14:38:24.233439+0000","last_active":"2024-06-29T14:38:26.254999+0000","last_peered":"2024-06-29T14:38:26.254999+0000","last_clean":"2024-06-29T14:38:26.254999+0000","last_became_active":"2024-06-29T14:38:24.233085+0000","last_became_peered":"2024-06-29T14:38:24.233085+0000","last_unstale":"2024-06-29T14:38:26.254999+0000","last_undegraded":"2024-06-29T14:38:26.254999+0000","last_fullsized":"2024-06-29T14:38:26.254999+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:35.922312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.261180+0000","last_change":"2024-06-29T14:38:25.256840+0000","last_active":"2024-06-29T14:38:25.261180+0000","last_peered":"2024-06-29T14:38:25.261180+0000","last_clean":"2024-06-29T14:38:25.261180+0000","last_became_active":"2024-06-29T14:38:25.256473+0000","last_became_peered":"2024-06-29T14:38:25.256473+0000","last_unstale":"2024-06-29T14:38:25.261180+0000","last_undegraded":"2024-06-29T14:38:25.261180+0000","last_fullsized":"2024-06-29T14:38:25.261180+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:30:50.230950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"113'2","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258821+0000","last_change":"2024-06-29T14:38:24.240662+0000","last_active":"2024-06-29T14:38:26.258821+0000","last_peered":"2024-06-29T14:38:26.258821+0000","last_clean":"2024-06-29T14:38:26.258821+0000","last_became_active":"2024-06-29T14:38:24.239874+0000","last_became_peered":"2024-06-29T14:38:24.239874+0000","last_unstale":"2024-06-29T14:38:26.258821+0000","last_undegraded":"2024-06-29T14:38:26.258821+0000","last_fullsized":"2024-06-29T14:38:26.258821+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:10.478176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258550+0000","last_change":"2024-06-29T14:38:25.259659+0000","last_active":"2024-06-29T14:38:26.258550+0000","last_peered":"2024-06-29T14:38:26.258550+0000","last_clean":"2024-06-29T14:38:26.258550+0000","last_became_active":"2024-06-29T14:38:25.259353+0000","last_became_peered":"2024-06-29T14:38:25.259353+0000","last_unstale":"2024-06-29T14:38:26.258550+0000","last_undegraded":"2024-06-29T14:38:26.258550+0000","last_fullsized":"2024-06-29T14:38:26.258550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:07.841508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.251998+0000","last_change":"2024-06-29T14:38:24.242984+0000","last_active":"2024-06-29T14:38:26.251998+0000","last_peered":"2024-06-29T14:38:26.251998+0000","last_clean":"2024-06-29T14:38:26.251998+0000","last_became_active":"2024-06-29T14:38:24.242419+0000","last_became_peered":"2024-06-29T14:38:24.242419+0000","last_unstale":"2024-06-29T14:38:26.251998+0000","last_undegraded":"2024-06-29T14:38:26.251998+0000","last_fullsized":"2024-06-29T14:38:26.251998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:13.989024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258711+0000","last_change":"2024-06-29T14:38:25.268979+0000","last_active":"2024-06-29T14:38:26.258711+0000","last_peered":"2024-06-29T14:38:26.258711+0000","last_clean":"2024-06-29T14:38:26.258711+0000","last_became_active":"2024-06-29T14:38:25.268617+0000","last_became_peered":"2024-06-29T14:38:25.268617+0000","last_unstale":"2024-06-29T14:38:26.258711+0000","last_undegraded":"2024-06-29T14:38:26.258711+0000","last_fullsized":"2024-06-29T14:38:26.258711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:59:46.751191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258646+0000","last_change":"2024-06-29T14:38:24.247720+0000","last_active":"2024-06-29T14:38:26.258646+0000","last_peered":"2024-06-29T14:38:26.258646+0000","last_clean":"2024-06-29T14:38:26.258646+0000","last_became_active":"2024-06-29T14:38:24.247407+0000","last_became_peered":"2024-06-29T14:38:24.247407+0000","last_unstale":"2024-06-29T14:38:26.258646+0000","last_undegraded":"2024-06-29T14:38:26.258646+0000","last_fullsized":"2024-06-29T14:38:26.258646+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:30.035017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276930+0000","last_change":"2024-06-29T14:38:25.273618+0000","last_active":"2024-06-29T14:38:25.276930+0000","last_peered":"2024-06-29T14:38:25.276930+0000","last_clean":"2024-06-29T14:38:25.276930+0000","last_became_active":"2024-06-29T14:38:25.273232+0000","last_became_peered":"2024-06-29T14:38:25.273232+0000","last_unstale":"2024-06-29T14:38:25.276930+0000","last_undegraded":"2024-06-29T14:38:25.276930+0000","last_fullsized":"2024-06-29T14:38:25.276930+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:03.235771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.252002+0000","last_change":"2024-06-29T14:38:24.238128+0000","last_active":"2024-06-29T14:38:26.252002+0000","last_peered":"2024-06-29T14:38:26.252002+0000","last_clean":"2024-06-29T14:38:26.252002+0000","last_became_active":"2024-06-29T14:38:24.237794+0000","last_became_peered":"2024-06-29T14:38:24.237794+0000","last_unstale":"2024-06-29T14:38:26.252002+0000","last_undegraded":"2024-06-29T14:38:26.252002+0000","last_fullsized":"2024-06-29T14:38:26.252002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:24.723402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254101+0000","last_change":"2024-06-29T14:38:25.249778+0000","last_active":"2024-06-29T14:38:26.254101+0000","last_peered":"2024-06-29T14:38:26.254101+0000","last_clean":"2024-06-29T14:38:26.254101+0000","last_became_active":"2024-06-29T14:38:25.249455+0000","last_became_peered":"2024-06-29T14:38:25.249455+0000","last_unstale":"2024-06-29T14:38:26.254101+0000","last_undegraded":"2024-06-29T14:38:26.254101+0000","last_fullsized":"2024-06-29T14:38:26.254101+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:32.993979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"113'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266180+0000","last_change":"2024-06-29T14:38:24.234583+0000","last_active":"2024-06-29T14:38:25.266180+0000","last_peered":"2024-06-29T14:38:25.266180+0000","last_clean":"2024-06-29T14:38:25.266180+0000","last_became_active":"2024-06-29T14:38:24.234249+0000","last_became_peered":"2024-06-29T14:38:24.234249+0000","last_unstale":"2024-06-29T14:38:25.266180+0000","last_undegraded":"2024-06-29T14:38:25.266180+0000","last_fullsized":"2024-06-29T14:38:25.266180+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:17:35.184659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.266452+0000","last_change":"2024-06-29T14:38:25.261017+0000","last_active":"2024-06-29T14:38:25.266452+0000","last_peered":"2024-06-29T14:38:25.266452+0000","last_clean":"2024-06-29T14:38:25.266452+0000","last_became_active":"2024-06-29T14:38:25.260710+0000","last_became_peered":"2024-06-29T14:38:25.260710+0000","last_unstale":"2024-06-29T14:38:25.266452+0000","last_undegraded":"2024-06-29T14:38:25.266452+0000","last_fullsized":"2024-06-29T14:38:25.266452+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:50.078684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"114'1","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.257955+0000","last_change":"2024-06-29T14:38:24.237559+0000","last_active":"2024-06-29T14:38:26.257955+0000","last_peered":"2024-06-29T14:38:26.257955+0000","last_clean":"2024-06-29T14:38:26.257955+0000","last_became_active":"2024-06-29T14:38:24.237199+0000","last_became_peered":"2024-06-29T14:38:24.237199+0000","last_unstale":"2024-06-29T14:38:26.257955+0000","last_undegraded":"2024-06-29T14:38:26.257955+0000","last_fullsized":"2024-06-29T14:38:26.257955+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:23.142501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276322+0000","last_change":"2024-06-29T14:38:25.274329+0000","last_active":"2024-06-29T14:38:25.276322+0000","last_peered":"2024-06-29T14:38:25.276322+0000","last_clean":"2024-06-29T14:38:25.276322+0000","last_became_active":"2024-06-29T14:38:25.271073+0000","last_became_peered":"2024-06-29T14:38:25.271073+0000","last_unstale":"2024-06-29T14:38:25.276322+0000","last_undegraded":"2024-06-29T14:38:25.276322+0000","last_fullsized":"2024-06-29T14:38:25.276322+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:52.208296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254984+0000","last_change":"2024-06-29T14:38:24.247753+0000","last_active":"2024-06-29T14:38:26.254984+0000","last_peered":"2024-06-29T14:38:26.254984+0000","last_clean":"2024-06-29T14:38:26.254984+0000","last_became_active":"2024-06-29T14:38:24.247448+0000","last_became_peered":"2024-06-29T14:38:24.247448+0000","last_unstale":"2024-06-29T14:38:26.254984+0000","last_undegraded":"2024-06-29T14:38:26.254984+0000","last_fullsized":"2024-06-29T14:38:26.254984+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:55.908391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.259150+0000","last_change":"2024-06-29T14:38:25.258035+0000","last_active":"2024-06-29T14:38:26.259150+0000","last_peered":"2024-06-29T14:38:26.259150+0000","last_clean":"2024-06-29T14:38:26.259150+0000","last_became_active":"2024-06-29T14:38:25.257716+0000","last_became_peered":"2024-06-29T14:38:25.257716+0000","last_unstale":"2024-06-29T14:38:26.259150+0000","last_undegraded":"2024-06-29T14:38:26.259150+0000","last_fullsized":"2024-06-29T14:38:26.259150+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:23.293066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.254421+0000","last_change":"2024-06-29T14:38:24.240706+0000","last_active":"2024-06-29T14:38:26.254421+0000","last_peered":"2024-06-29T14:38:26.254421+0000","last_clean":"2024-06-29T14:38:26.254421+0000","last_became_active":"2024-06-29T14:38:24.240371+0000","last_became_peered":"2024-06-29T14:38:24.240371+0000","last_unstale":"2024-06-29T14:38:26.254421+0000","last_undegraded":"2024-06-29T14:38:26.254421+0000","last_fullsized":"2024-06-29T14:38:26.254421+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:11:21.632342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.251888+0000","last_change":"2024-06-29T14:38:25.259943+0000","last_active":"2024-06-29T14:38:26.251888+0000","last_peered":"2024-06-29T14:38:26.251888+0000","last_clean":"2024-06-29T14:38:26.251888+0000","last_became_active":"2024-06-29T14:38:25.259562+0000","last_became_peered":"2024-06-29T14:38:25.259562+0000","last_unstale":"2024-06-29T14:38:26.251888+0000","last_undegraded":"2024-06-29T14:38:26.251888+0000","last_fullsized":"2024-06-29T14:38:26.251888+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:14:27.069812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.254134+0000","last_change":"2024-06-29T14:38:24.247118+0000","last_active":"2024-06-29T14:38:25.254134+0000","last_peered":"2024-06-29T14:38:25.254134+0000","last_clean":"2024-06-29T14:38:25.254134+0000","last_became_active":"2024-06-29T14:38:24.246726+0000","last_became_peered":"2024-06-29T14:38:24.246726+0000","last_unstale":"2024-06-29T14:38:25.254134+0000","last_undegraded":"2024-06-29T14:38:25.254134+0000","last_fullsized":"2024-06-29T14:38:25.254134+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:29.359547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.504783+0000","last_change":"2024-06-29T14:38:25.266715+0000","last_active":"2024-06-29T14:38:26.504783+0000","last_peered":"2024-06-29T14:38:26.504783+0000","last_clean":"2024-06-29T14:38:26.504783+0000","last_became_active":"2024-06-29T14:38:25.264793+0000","last_became_peered":"2024-06-29T14:38:25.264793+0000","last_unstale":"2024-06-29T14:38:26.504783+0000","last_undegraded":"2024-06-29T14:38:26.504783+0000","last_fullsized":"2024-06-29T14:38:26.504783+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:11.484973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.258828+0000","last_change":"2024-06-29T14:38:24.245329+0000","last_active":"2024-06-29T14:38:26.258828+0000","last_peered":"2024-06-29T14:38:26.258828+0000","last_clean":"2024-06-29T14:38:26.258828+0000","last_became_active":"2024-06-29T14:38:24.244794+0000","last_became_peered":"2024-06-29T14:38:24.244794+0000","last_unstale":"2024-06-29T14:38:26.258828+0000","last_undegraded":"2024-06-29T14:38:26.258828+0000","last_fullsized":"2024-06-29T14:38:26.258828+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:35.327250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253236+0000","last_change":"2024-06-29T14:38:25.264996+0000","last_active":"2024-06-29T14:38:26.253236+0000","last_peered":"2024-06-29T14:38:26.253236+0000","last_clean":"2024-06-29T14:38:26.253236+0000","last_became_active":"2024-06-29T14:38:25.264703+0000","last_became_peered":"2024-06-29T14:38:25.264703+0000","last_unstale":"2024-06-29T14:38:26.253236+0000","last_undegraded":"2024-06-29T14:38:26.253236+0000","last_fullsized":"2024-06-29T14:38:26.253236+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:01.897966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.253406+0000","last_change":"2024-06-29T14:38:24.240670+0000","last_active":"2024-06-29T14:38:26.253406+0000","last_peered":"2024-06-29T14:38:26.253406+0000","last_clean":"2024-06-29T14:38:26.253406+0000","last_became_active":"2024-06-29T14:38:24.240355+0000","last_became_peered":"2024-06-29T14:38:24.240355+0000","last_unstale":"2024-06-29T14:38:26.253406+0000","last_undegraded":"2024-06-29T14:38:26.253406+0000","last_fullsized":"2024-06-29T14:38:26.253406+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:24.373775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.276955+0000","last_change":"2024-06-29T14:38:25.274356+0000","last_active":"2024-06-29T14:38:25.276955+0000","last_peered":"2024-06-29T14:38:25.276955+0000","last_clean":"2024-06-29T14:38:25.276955+0000","last_became_active":"2024-06-29T14:38:25.274025+0000","last_became_peered":"2024-06-29T14:38:25.274025+0000","last_unstale":"2024-06-29T14:38:25.276955+0000","last_undegraded":"2024-06-29T14:38:25.276955+0000","last_fullsized":"2024-06-29T14:38:25.276955+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:39:48.564333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.246804+0000","last_change":"2024-06-29T14:38:24.242251+0000","last_active":"2024-06-29T14:38:25.246804+0000","last_peered":"2024-06-29T14:38:25.246804+0000","last_clean":"2024-06-29T14:38:25.246804+0000","last_became_active":"2024-06-29T14:38:24.241944+0000","last_became_peered":"2024-06-29T14:38:24.241944+0000","last_unstale":"2024-06-29T14:38:25.246804+0000","last_undegraded":"2024-06-29T14:38:25.246804+0000","last_fullsized":"2024-06-29T14:38:25.246804+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:14.114639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.270941+0000","last_change":"2024-06-29T14:38:25.270227+0000","last_active":"2024-06-29T14:38:25.270941+0000","last_peered":"2024-06-29T14:38:25.270941+0000","last_clean":"2024-06-29T14:38:25.270941+0000","last_became_active":"2024-06-29T14:38:25.269922+0000","last_became_peered":"2024-06-29T14:38:25.269922+0000","last_unstale":"2024-06-29T14:38:25.270941+0000","last_undegraded":"2024-06-29T14:38:25.270941+0000","last_fullsized":"2024-06-29T14:38:25.270941+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:24.216988+0000","last_clean_scrub_stamp":"2024-06-29T14:38:24.216988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:25:39.450578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.242584+0000","last_change":"2024-06-29T14:38:24.238508+0000","last_active":"2024-06-29T14:38:25.242584+0000","last_peered":"2024-06-29T14:38:25.242584+0000","last_clean":"2024-06-29T14:38:25.242584+0000","last_became_active":"2024-06-29T14:38:24.238160+0000","last_became_peered":"2024-06-29T14:38:24.238160+0000","last_unstale":"2024-06-29T14:38:25.242584+0000","last_undegraded":"2024-06-29T14:38:25.242584+0000","last_fullsized":"2024-06-29T14:38:25.242584+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.210191+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.210191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:31.226738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"103'128","reported_seq":489,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.246878+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:38:25.246878+0000","last_peered":"2024-06-29T14:38:25.246878+0000","last_clean":"2024-06-29T14:38:25.246878+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:38:25.246878+0000","last_undegraded":"2024-06-29T14:38:25.246878+0000","last_fullsized":"2024-06-29T14:38:25.246878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":197,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":197,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542345,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59488,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312352,"statfs":{"total":96636764160,"available":96575848448,"internally_reserved":0,"allocated":1986560,"data_stored":1288052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542346,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1417216,"data_stored":706536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542345,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50600,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321240,"statfs":{"total":96636764160,"available":96584949760,"internally_reserved":0,"allocated":1396736,"data_stored":704320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542344,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1392640,"data_stored":697788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1396736,"data_stored":704320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575050,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58920,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312920,"statfs":{"total":96636764160,"available":96576430080,"internally_reserved":0,"allocated":1396736,"data_stored":704320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509641,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1388544,"data_stored":697698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509643,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37296,"kb_used_data":1956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334544,"statfs":{"total":96636764160,"available":96598573056,"internally_reserved":0,"allocated":2002944,"data_stored":1290314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:38:29.718 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:38:29.718 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:38:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.881+0000 7f04b799e640 1 -- 172.21.15.59:0/2997269969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 msgr2=0x7f04b00b5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.881+0000 7f04b799e640 1 --2- 172.21.15.59:0/2997269969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 0x7f04b00b5bc0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f04b015c850 tx=0x7f04a802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.882+0000 7f04b799e640 1 -- 172.21.15.59:0/2997269969 shutdown_connections 2024-06-29T14:38:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.882+0000 7f04b799e640 1 --2- 172.21.15.59:0/2997269969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 0x7f04b00b5bc0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.882+0000 7f04b799e640 1 --2- 172.21.15.59:0/2997269969 >> v2:172.21.15.153:3300/0 conn(0x7f04b01564f0 0x7f04b00d1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.882+0000 7f04b799e640 1 -- 172.21.15.59:0/2997269969 >> 172.21.15.59:0/2997269969 conn(0x7f04b0073e10 msgr2=0x7f04b0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.882+0000 7f04b799e640 1 -- 172.21.15.59:0/2997269969 shutdown_connections 2024-06-29T14:38:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.882+0000 7f04b799e640 1 -- 172.21.15.59:0/2997269969 wait complete. 2024-06-29T14:38:29.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.883+0000 7f04b799e640 1 Processor -- start 2024-06-29T14:38:29.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.884+0000 7f04b799e640 1 -- start start 2024-06-29T14:38:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b799e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 0x7f04b013f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b799e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04b01564f0 0x7f04b0152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b799e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b015c440 0x7f04b01e46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b799e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f04b00b64d0 con 0x7f04b00d1aa0 2024-06-29T14:38:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b799e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f04b00b6350 con 0x7f04b015c440 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b799e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f04b00b6650 con 0x7f04b01564f0 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b573a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 0x7f04b013f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b573a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 0x7f04b013f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53310/0 (socket says 172.21.15.59:53310) 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b573a640 1 -- 172.21.15.59:0/783673235 learned_addr learned my addr 172.21.15.59:0/783673235 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b5f3b640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b015c440 0x7f04b01e46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b4f39640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04b01564f0 0x7f04b0152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b573a640 1 -- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04b01564f0 msgr2=0x7f04b0152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b573a640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04b01564f0 0x7f04b0152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b573a640 1 -- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b015c440 msgr2=0x7f04b01e46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b573a640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b015c440 0x7f04b01e46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.885+0000 7f04b573a640 1 -- 172.21.15.59:0/783673235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b013fdf0 con 0x7f04b00d1aa0 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.886+0000 7f04b573a640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 0x7f04b013f210 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c3c0 tx=0x7f04ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.886+0000 7f04a1ffb640 1 -- 172.21.15.59:0/783673235 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac017070 con 0x7f04b00d1aa0 2024-06-29T14:38:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.886+0000 7f04a1ffb640 1 -- 172.21.15.59:0/783673235 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04ac0035f0 con 0x7f04b00d1aa0 2024-06-29T14:38:29.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.886+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b01400d0 con 0x7f04b00d1aa0 2024-06-29T14:38:29.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.886+0000 7f04a1ffb640 1 -- 172.21.15.59:0/783673235 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac003a80 con 0x7f04b00d1aa0 2024-06-29T14:38:29.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.886+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b013d500 con 0x7f04b00d1aa0 2024-06-29T14:38:29.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.887+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b00b6b90 con 0x7f04b00d1aa0 2024-06-29T14:38:29.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.888+0000 7f04a1ffb640 1 -- 172.21.15.59:0/783673235 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f04ac01a070 con 0x7f04b00d1aa0 2024-06-29T14:38:29.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.888+0000 7f04a1ffb640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f0494076420 0x7f0494078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.889+0000 7f04a1ffb640 1 -- 172.21.15.59:0/783673235 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f04ac096d70 con 0x7f04b00d1aa0 2024-06-29T14:38:29.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.889+0000 7f04b4f39640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f0494076420 0x7f0494078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.890+0000 7f04b4f39640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f0494076420 0x7f0494078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04a400a490 tx=0x7f04a400a420 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:29.893+0000 7f04a1ffb640 1 -- 172.21.15.59:0/783673235 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f04ac05fd50 con 0x7f04b00d1aa0 2024-06-29T14:38:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.155+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f04b015f540 con 0x7f04b00d1aa0 2024-06-29T14:38:30.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.159+0000 7f04a1ffb640 1 -- 172.21.15.59:0/783673235 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7f04ac064050 con 0x7f04b00d1aa0 2024-06-29T14:38:30.157 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:38:30.157 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:38:30.157 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T14:38:30.157 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T14:38:30.157 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:38:30.157 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T14:38:30.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f0494076420 msgr2=0x7f0494078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f0494076420 0x7f0494078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04a400a490 tx=0x7f04a400a420 comp rx=0 tx=0).stop 2024-06-29T14:38:30.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 msgr2=0x7f04b013f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 0x7f04b013f210 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c3c0 tx=0x7f04ac00c880 comp rx=0 tx=0).stop 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 shutdown_connections 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f0494076420 0x7f0494078930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04b015c440 0x7f04b01e46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04b01564f0 0x7f04b0152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 --2- 172.21.15.59:0/783673235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b00d1aa0 0x7f04b013f210 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 >> 172.21.15.59:0/783673235 conn(0x7f04b0073e10 msgr2=0x7f04b0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 shutdown_connections 2024-06-29T14:38:30.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.162+0000 7f04b799e640 1 -- 172.21.15.59:0/783673235 wait complete. 2024-06-29T14:38:30.171 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:30.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.291+0000 7f7b838df640 1 -- 172.21.15.59:0/4223040656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c15e0a0 msgr2=0x7f7b7c161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.291+0000 7f7b838df640 1 --2- 172.21.15.59:0/4223040656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c15e0a0 0x7f7b7c161a80 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0b5060 tx=0x7f7b7002a490 comp rx=0 tx=0).stop 2024-06-29T14:38:30.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.291+0000 7f7b838df640 1 -- 172.21.15.59:0/4223040656 shutdown_connections 2024-06-29T14:38:30.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.291+0000 7f7b838df640 1 --2- 172.21.15.59:0/4223040656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c15e0a0 0x7f7b7c161a80 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.291+0000 7f7b838df640 1 --2- 172.21.15.59:0/4223040656 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b7c162770 0x7f7b7c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.291+0000 7f7b838df640 1 -- 172.21.15.59:0/4223040656 >> 172.21.15.59:0/4223040656 conn(0x7f7b7c0d45b0 msgr2=0x7f7b7c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.291+0000 7f7b838df640 1 -- 172.21.15.59:0/4223040656 shutdown_connections 2024-06-29T14:38:30.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.291+0000 7f7b838df640 1 -- 172.21.15.59:0/4223040656 wait complete. 2024-06-29T14:38:30.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.293+0000 7f7b838df640 1 Processor -- start 2024-06-29T14:38:30.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.293+0000 7f7b838df640 1 -- start start 2024-06-29T14:38:30.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b838df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b7c15e0a0 0x7f7b7c1f8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b838df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c161fc0 0x7f7b7c150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b838df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b7c162770 0x7f7b7c160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b838df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c0d2d80 con 0x7f7b7c161fc0 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b838df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c0b4b00 con 0x7f7b7c162770 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b838df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7c0d2f00 con 0x7f7b7c15e0a0 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b80e7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c161fc0 0x7f7b7c150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b80e7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c161fc0 0x7f7b7c150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53328/0 (socket says 172.21.15.59:53328) 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b80e7a640 1 -- 172.21.15.59:0/3070728373 learned_addr learned my addr 172.21.15.59:0/3070728373 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b8167b640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b7c15e0a0 0x7f7b7c1f8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.294+0000 7f7b81e7c640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b7c162770 0x7f7b7c160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b80e7a640 1 -- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b7c15e0a0 msgr2=0x7f7b7c1f8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b80e7a640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b7c15e0a0 0x7f7b7c1f8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b80e7a640 1 -- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b7c162770 msgr2=0x7f7b7c160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b80e7a640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b7c162770 0x7f7b7c160af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b80e7a640 1 -- 172.21.15.59:0/3070728373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7c1f8fe0 con 0x7f7b7c161fc0 2024-06-29T14:38:30.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b80e7a640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c161fc0 0x7f7b7c150e60 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f7b70000c00 tx=0x7f7b700039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b79ffb640 1 -- 172.21.15.59:0/3070728373 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7003e030 con 0x7f7b7c161fc0 2024-06-29T14:38:30.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b79ffb640 1 -- 172.21.15.59:0/3070728373 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b700047a0 con 0x7f7b7c161fc0 2024-06-29T14:38:30.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7c0ccd70 con 0x7f7b7c161fc0 2024-06-29T14:38:30.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b79ffb640 1 -- 172.21.15.59:0/3070728373 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b70035040 con 0x7f7b7c161fc0 2024-06-29T14:38:30.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.295+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7c0cd330 con 0x7f7b7c161fc0 2024-06-29T14:38:30.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.296+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b7c0d33e0 con 0x7f7b7c161fc0 2024-06-29T14:38:30.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.297+0000 7f7b79ffb640 1 -- 172.21.15.59:0/3070728373 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f7b70041030 con 0x7f7b7c161fc0 2024-06-29T14:38:30.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.298+0000 7f7b79ffb640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7b68076420 0x7f7b68078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.298+0000 7f7b79ffb640 1 -- 172.21.15.59:0/3070728373 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f7b700b8300 con 0x7f7b7c161fc0 2024-06-29T14:38:30.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.298+0000 7f7b8167b640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7b68076420 0x7f7b68078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.298+0000 7f7b8167b640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7b68076420 0x7f7b68078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00a400 tx=0x7f7b6c00d040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.302+0000 7f7b79ffb640 1 -- 172.21.15.59:0/3070728373 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7b7c1f8fe0 con 0x7f7b7c161fc0 2024-06-29T14:38:30.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.531+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b7c06dca0 con 0x7f7b7c161fc0 2024-06-29T14:38:30.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.531+0000 7f7b79ffb640 1 -- 172.21.15.59:0/3070728373 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5225 (secure 0 0 0) 0x7f7b7c1623d0 con 0x7f7b7c161fc0 2024-06-29T14:38:30.530 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:30.530 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"btime":"2024-06-29T14:38:28:267220+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3208152037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3208152037},{"type":"v1","addr":"172.21.15.153:6836","nonce":3208152037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6716,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/590918917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":590918917},{"type":"v1","addr":"172.21.15.153:6837","nonce":590918917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/546301975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":546301975},{"type":"v1","addr":"172.21.15.59:6833","nonce":546301975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.621714+0000","modified":"2024-06-29T14:38:27.659142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/2011834805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2011834805},{"type":"v1","addr":"172.21.15.59:6835","nonce":2011834805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":6}]} 2024-06-29T14:38:30.530 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 55 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.533+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7b68076420 msgr2=0x7f7b68078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.533+0000 7f7b838df640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7b68076420 0x7f7b68078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00a400 tx=0x7f7b6c00d040 comp rx=0 tx=0).stop 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.533+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c161fc0 msgr2=0x7f7b7c150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.533+0000 7f7b838df640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c161fc0 0x7f7b7c150e60 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f7b70000c00 tx=0x7f7b700039e0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.534+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 shutdown_connections 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.534+0000 7f7b838df640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f7b68076420 0x7f7b68078930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.534+0000 7f7b838df640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b7c162770 0x7f7b7c160af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.534+0000 7f7b838df640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b7c161fc0 0x7f7b7c150e60 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.534+0000 7f7b838df640 1 --2- 172.21.15.59:0/3070728373 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b7c15e0a0 0x7f7b7c1f8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.534+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 >> 172.21.15.59:0/3070728373 conn(0x7f7b7c0d45b0 msgr2=0x7f7b7c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.534+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 shutdown_connections 2024-06-29T14:38:30.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:30.534+0000 7f7b838df640 1 -- 172.21.15.59:0/3070728373 wait complete. 2024-06-29T14:38:30.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:38:24.621714+0000', 'modified': '2024-06-29T14:38:27.659142+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7854}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7854': {'gid': 7854, 'name': 'a', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/2011834805', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 2011834805}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 2011834805}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7854, 'qdb_cluster': [7854]} 2024-06-29T14:38:30.946 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:38:30.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:38:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.071+0000 7f218ed68640 1 -- 172.21.15.59:0/1853943950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 msgr2=0x7f218815f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.071+0000 7f218ed68640 1 --2- 172.21.15.59:0/1853943950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 0x7f218815f240 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f2180005fd0 tx=0x7f218002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.071+0000 7f218ed68640 1 -- 172.21.15.59:0/1853943950 shutdown_connections 2024-06-29T14:38:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.071+0000 7f218ed68640 1 --2- 172.21.15.59:0/1853943950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 0x7f218815f240 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.071+0000 7f218ed68640 1 -- 172.21.15.59:0/1853943950 >> 172.21.15.59:0/1853943950 conn(0x7f21880b24c0 msgr2=0x7f21880cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.071+0000 7f218ed68640 1 -- 172.21.15.59:0/1853943950 shutdown_connections 2024-06-29T14:38:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.071+0000 7f218ed68640 1 -- 172.21.15.59:0/1853943950 wait complete. 2024-06-29T14:38:31.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.073+0000 7f218ed68640 1 Processor -- start 2024-06-29T14:38:31.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.073+0000 7f218ed68640 1 -- start start 2024-06-29T14:38:31.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218ed68640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 0x7f218814b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218ed68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21881444c0 0x7f21881e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218ed68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21881f8cd0 0x7f21880b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218ed68640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2188073d10 con 0x7f21880cd9b0 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218ed68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2188073b90 con 0x7f21881f8cd0 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218ed68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2188073e90 con 0x7f21881444c0 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218cb04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 0x7f218814b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218cb04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 0x7f218814b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53340/0 (socket says 172.21.15.59:53340) 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218cb04640 1 -- 172.21.15.59:0/2643192481 learned_addr learned my addr 172.21.15.59:0/2643192481 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f2187fff640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21881444c0 0x7f21881e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.074+0000 7f218d305640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21881f8cd0 0x7f21880b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f218cb04640 1 -- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21881444c0 msgr2=0x7f21881e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f218cb04640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21881444c0 0x7f21881e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f218cb04640 1 -- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21881f8cd0 msgr2=0x7f21880b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f218cb04640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21881f8cd0 0x7f21880b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f218cb04640 1 -- 172.21.15.59:0/2643192481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2188140db0 con 0x7f21880cd9b0 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f218cb04640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 0x7f218814b8c0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2180009cd0 tx=0x7f2180003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f21857fa640 1 -- 172.21.15.59:0/2643192481 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180031b90 con 0x7f21880cd9b0 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f21857fa640 1 -- 172.21.15.59:0/2643192481 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2180031e20 con 0x7f21880cd9b0 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f218ed68640 1 -- 172.21.15.59:0/2643192481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21881f81f0 con 0x7f21880cd9b0 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f21857fa640 1 -- 172.21.15.59:0/2643192481 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21800332d0 con 0x7f21880cd9b0 2024-06-29T14:38:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.075+0000 7f218ed68640 1 -- 172.21.15.59:0/2643192481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21881f87b0 con 0x7f21880cd9b0 2024-06-29T14:38:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.077+0000 7f21857fa640 1 -- 172.21.15.59:0/2643192481 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f2180033470 con 0x7f21880cd9b0 2024-06-29T14:38:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.077+0000 7f21857fa640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f21740763f0 0x7f2174078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.078+0000 7f21857fa640 1 -- 172.21.15.59:0/2643192481 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f21800b91f0 con 0x7f21880cd9b0 2024-06-29T14:38:31.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.078+0000 7f2187fff640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f21740763f0 0x7f2174078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.078+0000 7f2187fff640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f21740763f0 0x7f2174078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f217c006fd0 tx=0x7f217c008040 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.078+0000 7f2195c24740 1 Processor -- start 2024-06-29T14:38:31.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.078+0000 7f2195c24740 1 -- start start 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2195c24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2195c24740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67963a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2195c24740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63a5590 0x55bde67968e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2195c24740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55bde672ebc0 con 0x55bde639d3b0 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2195c24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bde66dd630 con 0x55bde63c6270 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2195c24740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55bde66ccbd0 con 0x55bde63a5590 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53348/0 (socket says 172.21.15.59:53348) 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 -- 172.21.15.59:0/1360378287 learned_addr learned my addr 172.21.15.59:0/1360378287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f218cb04640 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f218d305640 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63a5590 0x55bde67968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 -- 172.21.15.59:0/1360378287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63a5590 msgr2=0x55bde67968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63a5590 0x55bde67968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 -- 172.21.15.59:0/1360378287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 msgr2=0x55bde67b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 -- 172.21.15.59:0/1360378287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bde66c9ca0 con 0x55bde639d3b0 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.080+0000 7f2187fff640 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67963a0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f217c00d8e0 tx=0x7f217c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16551fc183db8afe server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f216e7fc640 1 -- 172.21.15.59:0/1360378287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c01a040 con 0x55bde639d3b0 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f216e7fc640 1 -- 172.21.15.59:0/1360378287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f217c023070 con 0x55bde639d3b0 2024-06-29T14:38:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f216e7fc640 1 -- 172.21.15.59:0/1360378287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c014bf0 con 0x55bde639d3b0 2024-06-29T14:38:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 -- 172.21.15.59:0/1360378287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 msgr2=0x55bde67963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67963a0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f217c00d8e0 tx=0x7f217c010000 comp rx=0 tx=0).stop 2024-06-29T14:38:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 -- 172.21.15.59:0/1360378287 shutdown_connections 2024-06-29T14:38:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63a5590 0x55bde67968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67963a0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 --2- 172.21.15.59:0/1360378287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 -- 172.21.15.59:0/1360378287 >> 172.21.15.59:0/1360378287 conn(0x55bde63ba3c0 msgr2=0x55bde6769f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 -- 172.21.15.59:0/1360378287 shutdown_connections 2024-06-29T14:38:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.081+0000 7f2195c24740 1 -- 172.21.15.59:0/1360378287 wait complete. 2024-06-29T14:38:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.082+0000 7f2195c24740 1 -- start start 2024-06-29T14:38:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.083+0000 7f2195c24740 1 Processor -- start 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2195c24740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde639d3b0 0x55bde67a87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2195c24740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde63a5590 0x55bde67a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2195c24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2195c24740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55bde666aae0 con 0x55bde63a5590 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2195c24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bde66d0c10 con 0x55bde63c6270 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2195c24740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55bde6440620 con 0x55bde639d3b0 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2195c24740 10 client.0 _ll_register_callbacks cb 0x55bde6677480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde63a5590 0x55bde67a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde63a5590 0x55bde67a7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53358/0 (socket says 172.21.15.59:53358) 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 -- 172.21.15.59:0/739919234 learned_addr learned my addr 172.21.15.59:0/739919234 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f218d305640 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f218cb04640 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde639d3b0 0x55bde67a87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 -- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde639d3b0 msgr2=0x55bde67a87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde639d3b0 0x55bde67a87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 -- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 msgr2=0x55bde67c3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67c3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 -- 172.21.15.59:0/739919234 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bde6431850 con 0x55bde63a5590 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f218d305640 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f2187fff640 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde63a5590 0x55bde67a7bd0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x55bde67b4420 tx=0x7f217c0159a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674156ae483fca74 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.084+0000 7f21677fe640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f21677fe640 1 -- 172.21.15.59:0/739919234 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c02c070 con 0x55bde63a5590 2024-06-29T14:38:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f21677fe640 1 -- 172.21.15.59:0/739919234 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f217c005b10 con 0x55bde63a5590 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 1 -- 172.21.15.59:0/739919234 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f217c023070 con 0x55bde63a5590 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f21677fe640 1 -- 172.21.15.59:0/739919234 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f217c037050 con 0x55bde63a5590 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 10 client.7995 fetch_fsmap learned FSMap version 55 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 1 -- 172.21.15.59:0/739919234 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) -- 0x55bde64a86b0 con 0x55bde63a5590 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f21677fe640 1 -- 172.21.15.59:0/739919234 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f217c023070 con 0x55bde63a5590 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f21677fe640 1 -- 172.21.15.59:0/739919234 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 55) ==== 3502+0+0 (secure 0 0 0) 0x7f217c03a020 con 0x55bde63a5590 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 10 client.7995 fetch_fsmap finished waiting for FSMap version 55 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 10 client.7995 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 20 client.7995 populate_metadata read hostname 'smithi059' 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] conn(0x55bde6384e40 0x55bde67c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 4 client.7995 mds_command: new command op to 7854 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:38:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f2195c24740 1 -- 172.21.15.59:0/739919234 --> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bde66df580 con 0x55bde6384e40 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.085+0000 7f218cb04640 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] conn(0x55bde6384e40 0x55bde67c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.086+0000 7f21677fe640 10 client.7995 ms_handle_connect on v2:172.21.15.59:6834/2011834805 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.086+0000 7f218cb04640 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] conn(0x55bde6384e40 0x55bde67c05e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37587df6480a48d3 server_cookie=b0062d5d0b0275a1 in_seq=0 out_seq=0 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.086+0000 7f21677fe640 1 -- 172.21.15.59:0/739919234 <== mds.0 v2:172.21.15.59:6834/2011834805 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55bde66df580 con 0x55bde6384e40 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.086+0000 7f21677fe640 10 client.7995 handle_command_reply: tid=0 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.086+0000 7f2195c24740 1 client.7995 shutdown 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 -- 172.21.15.59:0/739919234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde63a5590 msgr2=0x55bde67a7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde63a5590 0x55bde67a7bd0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x55bde67b4420 tx=0x7f217c0159a0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 -- 172.21.15.59:0/739919234 shutdown_connections 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] conn(0x55bde6384e40 0x55bde67c05e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63c6270 0x55bde67c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde63a5590 0x55bde67a7bd0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 --2- 172.21.15.59:0/739919234 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde639d3b0 0x55bde67a87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f21677fe640 0 client.7995 ms_handle_reset on v2:172.21.15.59:6834/2011834805 2024-06-29T14:38:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 -- 172.21.15.59:0/739919234 >> 172.21.15.59:0/739919234 conn(0x55bde63ba3c0 msgr2=0x55bde6792c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 -- 172.21.15.59:0/739919234 shutdown_connections 2024-06-29T14:38:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 1 -- 172.21.15.59:0/739919234 wait complete. 2024-06-29T14:38:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.087+0000 7f2195c24740 20 client.7995 trim_cache size 0 max 16384 2024-06-29T14:38:31.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.106+0000 7f2195c24740 1 Processor -- start 2024-06-29T14:38:31.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.106+0000 7f2195c24740 1 -- start start 2024-06-29T14:38:31.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.107+0000 7f2195c24740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde6796e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.107+0000 7f2195c24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 0x55bde67bf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.107+0000 7f2195c24740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 0x55bde67bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.107+0000 7f2195c24740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2188201ca0 con 0x55bde639d3b0 2024-06-29T14:38:31.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.107+0000 7f2195c24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2188142f60 con 0x55bde63a5590 2024-06-29T14:38:31.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.107+0000 7f2195c24740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f21880b13f0 con 0x55bde63c6270 2024-06-29T14:38:31.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.107+0000 7f218cb04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde6796e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f218cb04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde6796e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53374/0 (socket says 172.21.15.59:53374) 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f218cb04640 1 -- 172.21.15.59:0/820345201 learned_addr learned my addr 172.21.15.59:0/820345201 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f2187fff640 1 --2- 172.21.15.59:0/820345201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 0x55bde67bf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f218cb04640 1 -- 172.21.15.59:0/820345201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 msgr2=0x55bde67bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f218cb04640 1 --2- 172.21.15.59:0/820345201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 0x55bde67bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f218cb04640 1 -- 172.21.15.59:0/820345201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 msgr2=0x55bde67bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f218cb04640 1 --2- 172.21.15.59:0/820345201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 0x55bde67bf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f218cb04640 1 -- 172.21.15.59:0/820345201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bde6793230 con 0x55bde639d3b0 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f218cb04640 1 --2- 172.21.15.59:0/820345201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde6796e40 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f2180081d10 tx=0x7f218002a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a527ea5f2a9434a server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f216dffb640 1 -- 172.21.15.59:0/820345201 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21800863c0 con 0x55bde639d3b0 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f216dffb640 1 -- 172.21.15.59:0/820345201 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21800a0070 con 0x55bde639d3b0 2024-06-29T14:38:31.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.108+0000 7f216dffb640 1 -- 172.21.15.59:0/820345201 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218008ae70 con 0x55bde639d3b0 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 -- 172.21.15.59:0/820345201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 msgr2=0x55bde6796e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 --2- 172.21.15.59:0/820345201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde6796e40 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f2180081d10 tx=0x7f218002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 -- 172.21.15.59:0/820345201 shutdown_connections 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 --2- 172.21.15.59:0/820345201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 0x55bde67bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 --2- 172.21.15.59:0/820345201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 0x55bde67bf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 --2- 172.21.15.59:0/820345201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde6796e40 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 -- 172.21.15.59:0/820345201 >> 172.21.15.59:0/820345201 conn(0x55bde63ba3c0 msgr2=0x55bde67abf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 -- 172.21.15.59:0/820345201 shutdown_connections 2024-06-29T14:38:31.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.109+0000 7f2195c24740 1 -- 172.21.15.59:0/820345201 wait complete. 2024-06-29T14:38:31.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.110+0000 7f2195c24740 1 -- start start 2024-06-29T14:38:31.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.110+0000 7f2195c24740 1 Processor -- start 2024-06-29T14:38:31.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f2195c24740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67661c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f2195c24740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 0x55bde6766700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f2195c24740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 0x55bde682e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f2195c24740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55bde67c49c0 con 0x55bde639d3b0 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f2195c24740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2188142910 con 0x55bde63a5590 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f2195c24740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55bde6794a60 con 0x55bde63c6270 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f2195c24740 10 client.0 _ll_register_callbacks cb 0x7f21800098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f218cb04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f218cb04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67661c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53384/0 (socket says 172.21.15.59:53384) 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.111+0000 7f218cb04640 1 -- 172.21.15.59:0/3155925053 learned_addr learned my addr 172.21.15.59:0/3155925053 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f2187fff640 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 0x55bde6766700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f218d305640 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 0x55bde682e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f218cb04640 1 -- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 msgr2=0x55bde682e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f218cb04640 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 0x55bde682e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f218cb04640 1 -- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 msgr2=0x55bde6766700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f218cb04640 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 0x55bde6766700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f218cb04640 1 -- 172.21.15.59:0/3155925053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bde67a7970 con 0x55bde639d3b0 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f218d305640 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 0x55bde682e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f218cb04640 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67661c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f2180009ca0 tx=0x7f2180009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9da9b1d16a8c21 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f21677fe640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f21677fe640 1 -- 172.21.15.59:0/3155925053 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218008f070 con 0x55bde639d3b0 2024-06-29T14:38:31.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f21677fe640 1 -- 172.21.15.59:0/3155925053 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f218007e2a0 con 0x55bde639d3b0 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f2195c24740 1 -- 172.21.15.59:0/3155925053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21800a0070 con 0x55bde639d3b0 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f21677fe640 1 -- 172.21.15.59:0/3155925053 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f2180003e10 con 0x55bde639d3b0 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f2195c24740 10 client.8001 fetch_fsmap learned FSMap version 55 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f2195c24740 1 -- 172.21.15.59:0/3155925053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) -- 0x55bde67a7970 con 0x55bde639d3b0 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.112+0000 7f21677fe640 1 -- 172.21.15.59:0/3155925053 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21800a0070 con 0x55bde639d3b0 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f21677fe640 1 -- 172.21.15.59:0/3155925053 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 55) ==== 3502+0+0 (secure 0 0 0) 0x7f21800a6070 con 0x55bde639d3b0 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2195c24740 10 client.8001 fetch_fsmap finished waiting for FSMap version 55 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2195c24740 10 client.8001 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2195c24740 20 client.8001 populate_metadata read hostname 'smithi059' 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2195c24740 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] conn(0x55bde682eca0 0x55bde67b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2195c24740 4 client.8001 mds_command: new command op to 7854 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2195c24740 1 -- 172.21.15.59:0/3155925053 --> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] -- command(tid 0: {"prefix": "status"}) -- 0x55bde66db020 con 0x55bde682eca0 2024-06-29T14:38:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2187fff640 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] conn(0x55bde682eca0 0x55bde67b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f21677fe640 10 client.8001 ms_handle_connect on v2:172.21.15.59:6834/2011834805 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2187fff640 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] conn(0x55bde682eca0 0x55bde67b0dd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b9cd963fe96c9f2 server_cookie=f111dbc2d2a218c8 in_seq=0 out_seq=0 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f21677fe640 1 -- 172.21.15.59:0/3155925053 <== mds.0 v2:172.21.15.59:6834/2011834805 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x55bde66db020 con 0x55bde682eca0 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f21677fe640 10 client.8001 handle_command_reply: tid=0 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.113+0000 7f2195c24740 1 client.8001 shutdown 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 -- 172.21.15.59:0/3155925053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 msgr2=0x55bde67661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67661c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f2180009ca0 tx=0x7f2180009520 comp rx=0 tx=0).stop 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 -- 172.21.15.59:0/3155925053 shutdown_connections 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.59:6834/2011834805,v1:172.21.15.59:6835/2011834805] conn(0x55bde682eca0 0x55bde67b0dd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55bde63c6270 0x55bde682e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bde63a5590 0x55bde6766700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f21677fe640 0 client.8001 ms_handle_reset on v2:172.21.15.59:6834/2011834805 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 --2- 172.21.15.59:0/3155925053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bde639d3b0 0x55bde67661c0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 -- 172.21.15.59:0/3155925053 >> 172.21.15.59:0/3155925053 conn(0x55bde63ba3c0 msgr2=0x55bde67b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 -- 172.21.15.59:0/3155925053 shutdown_connections 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 1 -- 172.21.15.59:0/3155925053 wait complete. 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f2195c24740 20 client.8001 trim_cache size 0 max 16384 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 7854, 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:38:31.118 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:38:31.119 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8650571879999998, 2024-06-29T14:38:31.119 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 55, 2024-06-29T14:38:31.119 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 114, 2024-06-29T14:38:31.119 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 114, 2024-06-29T14:38:31.119 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.502231804999999 2024-06-29T14:38:31.119 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f218ed68640 1 -- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f21740763f0 msgr2=0x7f2174078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f218ed68640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f21740763f0 0x7f2174078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f217c006fd0 tx=0x7f217c008040 comp rx=0 tx=0).stop 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f218ed68640 1 -- 172.21.15.59:0/2643192481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 msgr2=0x7f218814b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.114+0000 7f218ed68640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 0x7f218814b8c0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2180009cd0 tx=0x7f2180003a10 comp rx=0 tx=0).stop 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.115+0000 7f218ed68640 1 -- 172.21.15.59:0/2643192481 shutdown_connections 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.115+0000 7f218ed68640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f21740763f0 0x7f2174078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.115+0000 7f218ed68640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21881f8cd0 0x7f21880b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.115+0000 7f218ed68640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f21881444c0 0x7f21881e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.115+0000 7f218ed68640 1 --2- 172.21.15.59:0/2643192481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21880cd9b0 0x7f218814b8c0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.115+0000 7f218ed68640 1 -- 172.21.15.59:0/2643192481 >> 172.21.15.59:0/2643192481 conn(0x7f21880b24c0 msgr2=0x7f21880683a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.115+0000 7f218ed68640 1 -- 172.21.15.59:0/2643192481 shutdown_connections 2024-06-29T14:38:31.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.115+0000 7f218ed68640 1 -- 172.21.15.59:0/2643192481 wait complete. 2024-06-29T14:38:31.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:38:31.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.242+0000 7f186970a640 1 -- 172.21.15.59:0/1214613939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 msgr2=0x7f18640cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.242+0000 7f186970a640 1 --2- 172.21.15.59:0/1214613939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 0x7f18640cf080 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f1854005fd0 tx=0x7f185402a580 comp rx=0 tx=0).stop 2024-06-29T14:38:31.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.242+0000 7f186970a640 1 -- 172.21.15.59:0/1214613939 shutdown_connections 2024-06-29T14:38:31.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.242+0000 7f186970a640 1 --2- 172.21.15.59:0/1214613939 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18640ce950 0x7f18640d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.242+0000 7f186970a640 1 --2- 172.21.15.59:0/1214613939 >> v2:172.21.15.153:3300/0 conn(0x7f18640cf5c0 0x7f18640d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.242+0000 7f186970a640 1 --2- 172.21.15.59:0/1214613939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 0x7f18640cf080 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.242+0000 7f186970a640 1 -- 172.21.15.59:0/1214613939 >> 172.21.15.59:0/1214613939 conn(0x7f18640b5610 msgr2=0x7f18640b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.242+0000 7f186970a640 1 -- 172.21.15.59:0/1214613939 shutdown_connections 2024-06-29T14:38:31.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.243+0000 7f186970a640 1 -- 172.21.15.59:0/1214613939 wait complete. 2024-06-29T14:38:31.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.244+0000 7f186970a640 1 Processor -- start 2024-06-29T14:38:31.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.244+0000 7f186970a640 1 -- start start 2024-06-29T14:38:31.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f186970a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 0x7f18640c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f186970a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18640ce950 0x7f18640cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f186970a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18640cf5c0 0x7f18640ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f186970a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1864162440 con 0x7f186408c190 2024-06-29T14:38:31.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f186970a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f18641622c0 con 0x7f18640cf5c0 2024-06-29T14:38:31.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f186970a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f18641625c0 con 0x7f18640ce950 2024-06-29T14:38:31.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 0x7f18640c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 0x7f18640c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53398/0 (socket says 172.21.15.59:53398) 2024-06-29T14:38:31.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 -- 172.21.15.59:0/353519229 learned_addr learned my addr 172.21.15.59:0/353519229 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:31.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f18637fe640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18640cf5c0 0x7f18640ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f18627fc640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18640ce950 0x7f18640cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 -- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18640ce950 msgr2=0x7f18640cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18640ce950 0x7f18640cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 -- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18640cf5c0 msgr2=0x7f18640ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18640cf5c0 0x7f18640ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 -- 172.21.15.59:0/353519229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18641f4d70 con 0x7f186408c190 2024-06-29T14:38:31.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f18637fe640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18640cf5c0 0x7f18640ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:31.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f1862ffd640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 0x7f18640c8040 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f1854034040 tx=0x7f1854005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.246+0000 7f185b7fe640 1 -- 172.21.15.59:0/353519229 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185403e030 con 0x7f186408c190 2024-06-29T14:38:31.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.247+0000 7f185b7fe640 1 -- 172.21.15.59:0/353519229 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1854003c30 con 0x7f186408c190 2024-06-29T14:38:31.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.247+0000 7f185b7fe640 1 -- 172.21.15.59:0/353519229 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1854035040 con 0x7f186408c190 2024-06-29T14:38:31.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.247+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18640aa740 con 0x7f186408c190 2024-06-29T14:38:31.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.247+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18640aad00 con 0x7f186408c190 2024-06-29T14:38:31.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.248+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18640cf080 con 0x7f186408c190 2024-06-29T14:38:31.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.248+0000 7f185b7fe640 1 -- 172.21.15.59:0/353519229 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f1854041030 con 0x7f186408c190 2024-06-29T14:38:31.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.249+0000 7f185b7fe640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f18440763f0 0x7f1844078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.249+0000 7f185b7fe640 1 -- 172.21.15.59:0/353519229 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f18540b8220 con 0x7f186408c190 2024-06-29T14:38:31.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.249+0000 7f18627fc640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f18440763f0 0x7f1844078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.250+0000 7f18627fc640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f18440763f0 0x7f1844078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f18640b4c40 tx=0x7f184c005f90 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.253+0000 7f185b7fe640 1 -- 172.21.15.59:0/353519229 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1854081740 con 0x7f186408c190 2024-06-29T14:38:31.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.484+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f186415c000 con 0x7f186408c190 2024-06-29T14:38:31.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.485+0000 7f185b7fe640 1 -- 172.21.15.59:0/353519229 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) ==== 53+0+73 (secure 0 0 0) 0x7f1854081040 con 0x7f186408c190 2024-06-29T14:38:31.483 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.486+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f18440763f0 msgr2=0x7f1844078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.486+0000 7f186970a640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f18440763f0 0x7f1844078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f18640b4c40 tx=0x7f184c005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 msgr2=0x7f18640c8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 0x7f18640c8040 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f1854034040 tx=0x7f1854005e80 comp rx=0 tx=0).stop 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 shutdown_connections 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f18440763f0 0x7f1844078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18640cf5c0 0x7f18640ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18640ce950 0x7f18640cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 --2- 172.21.15.59:0/353519229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f186408c190 0x7f18640c8040 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 >> 172.21.15.59:0/353519229 conn(0x7f18640b5610 msgr2=0x7f1864160f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 shutdown_connections 2024-06-29T14:38:31.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31.487+0000 7f186970a640 1 -- 172.21.15.59:0/353519229 wait complete. 2024-06-29T14:38:31.496 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:38:31.496 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:38:31.496 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:38:31.496 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:38:31.497 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:31.497 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:38:31.497 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:38:31.497 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:38:31.497 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:38:31.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.497 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:38:31.514 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 41739sec preferred_lft 41739sec 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:31.515 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T14:38:31.516 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.517 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:38:31.578 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:38:31.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.580 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:38:31.637 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:38:31.637 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:38:31.637 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:38:31.637 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:38:31.637 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:38:31.637 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:38:31.637 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:38:31.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:38:31.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:38:31.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:38:31.638 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:38:31.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:38:31.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.767 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:38:31.783 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:38:31.783 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:38:31.783 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:38:31.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.783 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:38:31.850 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:38:31.850 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:31.906 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:38:31.907 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:38:31.907 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:31.962 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:38:31.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:38:32.058 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:38:32.058 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:38:32.058 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:38:32.069 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:38:32.069 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:38:32.069 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:38:32.069 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:38:32.069 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:38:32.075 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:38:32.075 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:38:32.075 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:38:32.075 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:38:32.075 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:38:32.075 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:38:32.075 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:38:32.076 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:38:32.076 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:38:32.076 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:38:32.076 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:38:32.076 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:38:32.076 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:38:32.078 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:38:32.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:32.079 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:32.104 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:38:32.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:38:32.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:38:32.210 INFO:teuthology.orchestra.run.smithi059.stdout:client8010 2024-06-29T14:38:32.210 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:38:32.211 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:32.227 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:38:32.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.387+0000 7fe879ba1640 1 -- 172.21.15.59:0/355018498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87415ee10 msgr2=0x7fe87415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.387+0000 7fe879ba1640 1 --2- 172.21.15.59:0/355018498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87415ee10 0x7fe87415f9b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fe87408c4f0 tx=0x7fe86c02a560 comp rx=0 tx=0).stop 2024-06-29T14:38:32.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.387+0000 7fe879ba1640 1 -- 172.21.15.59:0/355018498 shutdown_connections 2024-06-29T14:38:32.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.387+0000 7fe879ba1640 1 --2- 172.21.15.59:0/355018498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87415ee10 0x7fe87415f9b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.387+0000 7fe879ba1640 1 --2- 172.21.15.59:0/355018498 >> v2:172.21.15.153:3300/0 conn(0x7fe8740b40a0 0x7fe87415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.387+0000 7fe879ba1640 1 -- 172.21.15.59:0/355018498 >> 172.21.15.59:0/355018498 conn(0x7fe8740d3ee0 msgr2=0x7fe8740d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:32.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.387+0000 7fe879ba1640 1 -- 172.21.15.59:0/355018498 shutdown_connections 2024-06-29T14:38:32.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.387+0000 7fe879ba1640 1 -- 172.21.15.59:0/355018498 wait complete. 2024-06-29T14:38:32.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.389+0000 7fe879ba1640 1 Processor -- start 2024-06-29T14:38:32.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.389+0000 7fe879ba1640 1 -- start start 2024-06-29T14:38:32.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe879ba1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740d23c0 0x7fe8741f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe879ba1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87415ee10 0x7fe87414e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe879ba1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe87414ef70 0x7fe874154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe879ba1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe874160280 con 0x7fe8740d23c0 2024-06-29T14:38:32.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe879ba1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe874160100 con 0x7fe87414ef70 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe879ba1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe874160400 con 0x7fe87415ee10 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740d23c0 0x7fe8741f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740d23c0 0x7fe8741f4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53410/0 (socket says 172.21.15.59:53410) 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 -- 172.21.15.59:0/4269223805 learned_addr learned my addr 172.21.15.59:0/4269223805 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe873fff640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe87414ef70 0x7fe874154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe872ffd640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87415ee10 0x7fe87414e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 -- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87415ee10 msgr2=0x7fe87414e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87415ee10 0x7fe87414e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 -- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe87414ef70 msgr2=0x7fe874154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe87414ef70 0x7fe874154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 -- 172.21.15.59:0/4269223805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8741528a0 con 0x7fe8740d23c0 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.391+0000 7fe8737fe640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740d23c0 0x7fe8741f4ef0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b9c0 tx=0x7fe86400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:32.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.392+0000 7fe86bfff640 1 -- 172.21.15.59:0/4269223805 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86400cba0 con 0x7fe8740d23c0 2024-06-29T14:38:32.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.392+0000 7fe86bfff640 1 -- 172.21.15.59:0/4269223805 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe86400ce30 con 0x7fe8740d23c0 2024-06-29T14:38:32.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.392+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe874155f80 con 0x7fe8740d23c0 2024-06-29T14:38:32.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.392+0000 7fe86bfff640 1 -- 172.21.15.59:0/4269223805 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864003bd0 con 0x7fe8740d23c0 2024-06-29T14:38:32.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.392+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8741f5430 con 0x7fe8740d23c0 2024-06-29T14:38:32.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.393+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87415e8d0 con 0x7fe8740d23c0 2024-06-29T14:38:32.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.394+0000 7fe86bfff640 1 -- 172.21.15.59:0/4269223805 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fe864014440 con 0x7fe8740d23c0 2024-06-29T14:38:32.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.394+0000 7fe86bfff640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe8540763f0 0x7fe854078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.394+0000 7fe86bfff640 1 -- 172.21.15.59:0/4269223805 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7fe8640981e0 con 0x7fe8740d23c0 2024-06-29T14:38:32.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.394+0000 7fe872ffd640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe8540763f0 0x7fe854078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.395+0000 7fe872ffd640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe8540763f0 0x7fe854078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe874152bc0 tx=0x7fe85c00a250 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:32.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.398+0000 7fe86bfff640 1 -- 172.21.15.59:0/4269223805 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe864061250 con 0x7fe8740d23c0 2024-06-29T14:38:32.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.613+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 --> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fe8741564a0 con 0x7fe8540763f0 2024-06-29T14:38:32.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.616+0000 7fe86bfff640 1 -- 172.21.15.59:0/4269223805 <== mgr.7132 v2:172.21.15.153:6832/1090607011 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fe8741564a0 con 0x7fe8540763f0 2024-06-29T14:38:32.614 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:38:32.614 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:38:32.614 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T14:38:32.614 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:38:32.614 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:38:32.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe8540763f0 msgr2=0x7fe854078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe8540763f0 0x7fe854078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe874152bc0 tx=0x7fe85c00a250 comp rx=0 tx=0).stop 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740d23c0 msgr2=0x7fe8741f4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740d23c0 0x7fe8741f4ef0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b9c0 tx=0x7fe86400be80 comp rx=0 tx=0).stop 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 shutdown_connections 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe8540763f0 0x7fe854078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe87414ef70 0x7fe874154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87415ee10 0x7fe87414e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 --2- 172.21.15.59:0/4269223805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740d23c0 0x7fe8741f4ef0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 >> 172.21.15.59:0/4269223805 conn(0x7fe8740d3ee0 msgr2=0x7fe8740d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 shutdown_connections 2024-06-29T14:38:32.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.617+0000 7fe879ba1640 1 -- 172.21.15.59:0/4269223805 wait complete. 2024-06-29T14:38:32.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:38:32.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.748+0000 7fc74314a640 1 -- 172.21.15.59:0/2983173133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc72c005150 msgr2=0x7fc72c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.748+0000 7fc74314a640 1 --2- 172.21.15.59:0/2983173133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc72c005150 0x7fc72c00b9c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fc734009e20 tx=0x7fc72c005830 comp rx=0 tx=0).stop 2024-06-29T14:38:32.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.748+0000 7fc7416e7640 1 -- 172.21.15.59:0/2983173133 reap_dead start 2024-06-29T14:38:32.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.748+0000 7fc74314a640 1 -- 172.21.15.59:0/2983173133 shutdown_connections 2024-06-29T14:38:32.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.748+0000 7fc74314a640 1 -- 172.21.15.59:0/2983173133 >> 172.21.15.59:0/2983173133 conn(0x7fc73c0d3ee0 msgr2=0x7fc73c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:32.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.748+0000 7fc74314a640 1 -- 172.21.15.59:0/2983173133 shutdown_connections 2024-06-29T14:38:32.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.748+0000 7fc74314a640 1 -- 172.21.15.59:0/2983173133 wait complete. 2024-06-29T14:38:32.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.750+0000 7fc74314a640 1 Processor -- start 2024-06-29T14:38:32.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.750+0000 7fc74314a640 1 -- start start 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc74314a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc72c005150 0x7fc73c15c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc74314a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f5970 0x7fc73c155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc74314a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f2380 0x7fc73c156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc74314a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c15e8d0 con 0x7fc73c1f5970 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc74314a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c0b42b0 con 0x7fc72c005150 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc74314a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc73c15ea50 con 0x7fc73c1f2380 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc73bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f5970 0x7fc73c155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc73bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f5970 0x7fc73c155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45434/0 (socket says 172.21.15.59:45434) 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.751+0000 7fc73bfff640 1 -- 172.21.15.59:0/3796704893 learned_addr learned my addr 172.21.15.59:0/3796704893 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc7416e7640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f2380 0x7fc73c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc740ee6640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc72c005150 0x7fc73c15c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc73bfff640 1 -- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f2380 msgr2=0x7fc73c156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc73bfff640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f2380 0x7fc73c156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc73bfff640 1 -- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc72c005150 msgr2=0x7fc73c15c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc73bfff640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc72c005150 0x7fc73c15c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc73bfff640 1 -- 172.21.15.59:0/3796704893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c1f6070 con 0x7fc73c1f5970 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc7416e7640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f2380 0x7fc73c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc740ee6640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc72c005150 0x7fc73c15c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc73bfff640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f5970 0x7fc73c155ad0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc72800f9d0 tx=0x7fc72800fe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc7397fa640 1 -- 172.21.15.59:0/3796704893 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72800bcb0 con 0x7fc73c1f5970 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc7397fa640 1 -- 172.21.15.59:0/3796704893 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc728011440 con 0x7fc73c1f5970 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc7397fa640 1 -- 172.21.15.59:0/3796704893 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc728011870 con 0x7fc73c1f5970 2024-06-29T14:38:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73c1f6350 con 0x7fc73c1f5970 2024-06-29T14:38:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.752+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc73c1f6910 con 0x7fc73c1f5970 2024-06-29T14:38:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.753+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc73c15cda0 con 0x7fc73c1f5970 2024-06-29T14:38:32.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.754+0000 7fc7397fa640 1 -- 172.21.15.59:0/3796704893 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fc728019020 con 0x7fc73c1f5970 2024-06-29T14:38:32.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.755+0000 7fc7397fa640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7200763f0 0x7fc720078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.755+0000 7fc7397fa640 1 -- 172.21.15.59:0/3796704893 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7fc728098b60 con 0x7fc73c1f5970 2024-06-29T14:38:32.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.755+0000 7fc740ee6640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7200763f0 0x7fc720078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.756+0000 7fc740ee6640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7200763f0 0x7fc720078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc730001860 tx=0x7fc730001cf0 comp rx=0 tx=0).ready entity=mgr.7132 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:32.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.758+0000 7fc7397fa640 1 -- 172.21.15.59:0/3796704893 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc728029020 con 0x7fc73c1f5970 2024-06-29T14:38:32.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:32.998+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fc73c15f9b0 con 0x7fc73c1f5970 2024-06-29T14:38:33.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.289+0000 7fc7397fa640 1 -- 172.21.15.59:0/3796704893 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v47) ==== 92+0+0 (secure 0 0 0) 0x7fc7280620f0 con 0x7fc73c1f5970 2024-06-29T14:38:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.292+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7200763f0 msgr2=0x7fc720078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.292+0000 7fc74314a640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7200763f0 0x7fc720078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc730001860 tx=0x7fc730001cf0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.293+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f5970 msgr2=0x7fc73c155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.293+0000 7fc74314a640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f5970 0x7fc73c155ad0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc72800f9d0 tx=0x7fc72800fe90 comp rx=0 tx=0).stop 2024-06-29T14:38:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.293+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 shutdown_connections 2024-06-29T14:38:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.293+0000 7fc74314a640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fc7200763f0 0x7fc720078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.293+0000 7fc74314a640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f2380 0x7fc73c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.293+0000 7fc74314a640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f5970 0x7fc73c155ad0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.293+0000 7fc74314a640 1 --2- 172.21.15.59:0/3796704893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc72c005150 0x7fc73c15c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.293+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 >> 172.21.15.59:0/3796704893 conn(0x7fc73c0d3ee0 msgr2=0x7fc73c161060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.294+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 shutdown_connections 2024-06-29T14:38:33.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.294+0000 7fc74314a640 1 -- 172.21.15.59:0/3796704893 wait complete. 2024-06-29T14:38:33.305 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:38:33.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.431+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/3026033317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9f80b40a0 msgr2=0x7fe9f815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.431+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/3026033317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9f80b40a0 0x7fe9f815e8d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009830 tx=0x7fe9e802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.431+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/3026033317 shutdown_connections 2024-06-29T14:38:33.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.431+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/3026033317 >> v2:172.21.15.153:3300/0 conn(0x7fe9f815ee10 0x7fe9f815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.431+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/3026033317 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9f80b40a0 0x7fe9f815e8d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.431+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/3026033317 >> 172.21.15.59:0/3026033317 conn(0x7fe9f80d3ee0 msgr2=0x7fe9f80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.431+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/3026033317 shutdown_connections 2024-06-29T14:38:33.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.431+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/3026033317 wait complete. 2024-06-29T14:38:33.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.433+0000 7fe9ff1a5640 1 Processor -- start 2024-06-29T14:38:33.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.433+0000 7fe9ff1a5640 1 -- start start 2024-06-29T14:38:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9ff1a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f80d23c0 0x7fe9f815b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9ff1a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9f815ee10 0x7fe9f81f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9ff1a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9f81f5240 0x7fe9f8155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9ff1a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f8160280 con 0x7fe9f815ee10 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9ff1a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f8160100 con 0x7fe9f80d23c0 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9ff1a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f8160400 con 0x7fe9f81f5240 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9f815ee10 0x7fe9f81f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9f815ee10 0x7fe9f81f4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45460/0 (socket says 172.21.15.59:45460) 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 -- 172.21.15.59:0/4050617181 learned_addr learned my addr 172.21.15.59:0/4050617181 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9fcf41640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f80d23c0 0x7fe9f815b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9fd742640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9f81f5240 0x7fe9f8155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 -- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9f81f5240 msgr2=0x7fe9f8155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9f81f5240 0x7fe9f8155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 -- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f80d23c0 msgr2=0x7fe9f815b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f80d23c0 0x7fe9f815b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 -- 172.21.15.59:0/4050617181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f815c160 con 0x7fe9f815ee10 2024-06-29T14:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9fcf41640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f80d23c0 0x7fe9f815b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:33.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.435+0000 7fe9f7fff640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9f815ee10 0x7fe9f81f4d00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009950 tx=0x7fe9e8030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:33.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.436+0000 7fe9f57fa640 1 -- 172.21.15.59:0/4050617181 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e803f070 con 0x7fe9f815ee10 2024-06-29T14:38:33.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.436+0000 7fe9f57fa640 1 -- 172.21.15.59:0/4050617181 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9e8030c50 con 0x7fe9f815ee10 2024-06-29T14:38:33.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.436+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f815c440 con 0x7fe9f815ee10 2024-06-29T14:38:33.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.436+0000 7fe9f57fa640 1 -- 172.21.15.59:0/4050617181 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8005c90 con 0x7fe9f815ee10 2024-06-29T14:38:33.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.436+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f815ca00 con 0x7fe9f815ee10 2024-06-29T14:38:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.437+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f815bed0 con 0x7fe9f815ee10 2024-06-29T14:38:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.437+0000 7fe9f57fa640 1 -- 172.21.15.59:0/4050617181 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 47) ==== 98320+0+0 (secure 0 0 0) 0x7fe9e8004280 con 0x7fe9f815ee10 2024-06-29T14:38:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.438+0000 7fe9f57fa640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe9d8076440 0x7fe9d8078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.438+0000 7fe9fcf41640 1 -- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe9d8076440 msgr2=0x7fe9d8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1090607011 2024-06-29T14:38:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.438+0000 7fe9f57fa640 1 -- 172.21.15.59:0/4050617181 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7fe9e80b8310 con 0x7fe9f815ee10 2024-06-29T14:38:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.438+0000 7fe9fcf41640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe9d8076440 0x7fe9d8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.442+0000 7fe9f57fa640 1 -- 172.21.15.59:0/4050617181 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe9e80811e0 con 0x7fe9f815ee10 2024-06-29T14:38:33.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.640+0000 7fe9fcf41640 1 -- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe9d8076440 msgr2=0x7fe9d8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1090607011 2024-06-29T14:38:33.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.640+0000 7fe9fcf41640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe9d8076440 0x7fe9d8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.662+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fe9f8069de0 con 0x7fe9f815ee10 2024-06-29T14:38:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.666+0000 7fe9f57fa640 1 -- 172.21.15.59:0/4050617181 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v45)=0 v45) ==== 138+0+0 (secure 0 0 0) 0x7fe9f8069de0 con 0x7fe9f815ee10 2024-06-29T14:38:33.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe9d8076440 msgr2=0x7fe9d8078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:33.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe9d8076440 0x7fe9d8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9f815ee10 msgr2=0x7fe9f81f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9f815ee10 0x7fe9f81f4d00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009950 tx=0x7fe9e8030650 comp rx=0 tx=0).stop 2024-06-29T14:38:33.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 shutdown_connections 2024-06-29T14:38:33.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe9d8076440 0x7fe9d8078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe9f81f5240 0x7fe9f8155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9f815ee10 0x7fe9f81f4d00 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 --2- 172.21.15.59:0/4050617181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9f80d23c0 0x7fe9f815b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 >> 172.21.15.59:0/4050617181 conn(0x7fe9f80d3ee0 msgr2=0x7fe9f80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 shutdown_connections 2024-06-29T14:38:33.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.669+0000 7fe9ff1a5640 1 -- 172.21.15.59:0/4050617181 wait complete. 2024-06-29T14:38:33.679 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:38:33.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.803+0000 7fe1a3711640 1 -- 172.21.15.59:0/3177511804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0d0130 msgr2=0x7fe19c143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.803+0000 7fe1a3711640 1 --2- 172.21.15.59:0/3177511804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0d0130 0x7fe19c143840 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe198009830 tx=0x7fe19802a5c0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.803+0000 7fe1a3711640 1 -- 172.21.15.59:0/3177511804 shutdown_connections 2024-06-29T14:38:33.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.803+0000 7fe1a3711640 1 --2- 172.21.15.59:0/3177511804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0d0130 0x7fe19c143840 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.804+0000 7fe1a3711640 1 -- 172.21.15.59:0/3177511804 >> 172.21.15.59:0/3177511804 conn(0x7fe19c0b2ab0 msgr2=0x7fe19c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.804+0000 7fe1a3711640 1 -- 172.21.15.59:0/3177511804 shutdown_connections 2024-06-29T14:38:33.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.804+0000 7fe1a3711640 1 -- 172.21.15.59:0/3177511804 wait complete. 2024-06-29T14:38:33.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.805+0000 7fe1a3711640 1 Processor -- start 2024-06-29T14:38:33.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.805+0000 7fe1a3711640 1 -- start start 2024-06-29T14:38:33.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a3711640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe19c06ebe0 0x7fe19c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a3711640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0b4ee0 0x7fe19c0b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a3711640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19c0d0130 0x7fe19c0b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a3711640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c0cfa30 con 0x7fe19c0d0130 2024-06-29T14:38:33.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a3711640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c0cf8b0 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a3711640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe19c0cfbb0 con 0x7fe19c06ebe0 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a1cae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19c0d0130 0x7fe19c0b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a1cae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19c0d0130 0x7fe19c0b0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45474/0 (socket says 172.21.15.59:45474) 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a1cae640 1 -- 172.21.15.59:0/2374819354 learned_addr learned my addr 172.21.15.59:0/2374819354 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a0cac640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0b4ee0 0x7fe19c0b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a14ad640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe19c06ebe0 0x7fe19c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a0cac640 1 -- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe19c06ebe0 msgr2=0x7fe19c0acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a0cac640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe19c06ebe0 0x7fe19c0acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a0cac640 1 -- 172.21.15.59:0/2374819354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19c0d0130 msgr2=0x7fe19c0b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a0cac640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19c0d0130 0x7fe19c0b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a0cac640 1 -- 172.21.15.59:0/2374819354 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c1f2350 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.807+0000 7fe1a1cae640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19c0d0130 0x7fe19c0b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.808+0000 7fe1a0cac640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0b4ee0 0x7fe19c0b0030 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c370 tx=0x7fe18c00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.808+0000 7fe191ffb640 1 -- 172.21.15.59:0/2374819354 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c011790 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.808+0000 7fe191ffb640 1 -- 172.21.15.59:0/2374819354 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe18c017070 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.808+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19c1f2630 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.808+0000 7fe191ffb640 1 -- 172.21.15.59:0/2374819354 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c011cc0 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.808+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19c0ad300 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.809+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe19c1f2b50 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.810+0000 7fe191ffb640 1 -- 172.21.15.59:0/2374819354 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 47) ==== 98320+0+0 (secure 0 0 0) 0x7fe18c01c460 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.810+0000 7fe191ffb640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe180076440 0x7fe180078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.810+0000 7fe1a14ad640 1 -- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe180076440 msgr2=0x7fe180078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1090607011 2024-06-29T14:38:33.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.810+0000 7fe191ffb640 1 -- 172.21.15.59:0/2374819354 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7fe18c097760 con 0x7fe19c0b4ee0 2024-06-29T14:38:33.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.810+0000 7fe1a14ad640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe180076440 0x7fe180078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:33.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:33.815+0000 7fe191ffb640 1 -- 172.21.15.59:0/2374819354 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe19c1f2b50 con 0x7fe19c0b4ee0 2024-06-29T14:38:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.010+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fe19c143840 con 0x7fe19c0b4ee0 2024-06-29T14:38:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.012+0000 7fe1a14ad640 1 -- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe180076440 msgr2=0x7fe180078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1090607011 2024-06-29T14:38:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.012+0000 7fe1a14ad640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe180076440 0x7fe180078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.015+0000 7fe191ffb640 1 -- 172.21.15.59:0/2374819354 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v46)=0 v46) ==== 133+0+0 (secure 0 0 0) 0x7fe18c060740 con 0x7fe19c0b4ee0 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.017+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe180076440 msgr2=0x7fe180078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.017+0000 7fe1a3711640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe180076440 0x7fe180078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.017+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0b4ee0 msgr2=0x7fe19c0b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.017+0000 7fe1a3711640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0b4ee0 0x7fe19c0b0030 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c370 tx=0x7fe18c00c830 comp rx=0 tx=0).stop 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.018+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 shutdown_connections 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.018+0000 7fe1a3711640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7fe180076440 0x7fe180078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.018+0000 7fe1a3711640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19c0d0130 0x7fe19c0b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.018+0000 7fe1a3711640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c0b4ee0 0x7fe19c0b0030 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.018+0000 7fe1a3711640 1 --2- 172.21.15.59:0/2374819354 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe19c06ebe0 0x7fe19c0acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.018+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 >> 172.21.15.59:0/2374819354 conn(0x7fe19c0b2ab0 msgr2=0x7fe19c1427c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:34.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.018+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 shutdown_connections 2024-06-29T14:38:34.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.018+0000 7fe1a3711640 1 -- 172.21.15.59:0/2374819354 wait complete. 2024-06-29T14:38:34.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-06-29T14:38:34.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.151+0000 7f95f32ac640 1 -- 172.21.15.59:0/2012553402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95ec143840 msgr2=0x7f95ec06e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:34.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.151+0000 7f95f32ac640 1 --2- 172.21.15.59:0/2012553402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95ec143840 0x7f95ec06e4d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009830 tx=0x7f95e002a480 comp rx=0 tx=0).stop 2024-06-29T14:38:34.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.151+0000 7f95f32ac640 1 -- 172.21.15.59:0/2012553402 shutdown_connections 2024-06-29T14:38:34.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.151+0000 7f95f32ac640 1 --2- 172.21.15.59:0/2012553402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95ec06ea10 0x7f95ec0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.151+0000 7f95f32ac640 1 --2- 172.21.15.59:0/2012553402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95ec143840 0x7f95ec06e4d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.151+0000 7f95f32ac640 1 -- 172.21.15.59:0/2012553402 >> 172.21.15.59:0/2012553402 conn(0x7f95ec0b2ab0 msgr2=0x7f95ec0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:34.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.151+0000 7f95f32ac640 1 -- 172.21.15.59:0/2012553402 shutdown_connections 2024-06-29T14:38:34.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.151+0000 7f95f32ac640 1 -- 172.21.15.59:0/2012553402 wait complete. 2024-06-29T14:38:34.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.152+0000 7f95f32ac640 1 Processor -- start 2024-06-29T14:38:34.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.153+0000 7f95f32ac640 1 -- start start 2024-06-29T14:38:34.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f32ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95ec06ea10 0x7f95ec154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:34.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f32ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95ec0d0130 0x7f95ec157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:34.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f32ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95ec143840 0x7f95ec157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:34.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f32ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec0cf650 con 0x7f95ec06ea10 2024-06-29T14:38:34.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f32ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec0cf4d0 con 0x7f95ec0d0130 2024-06-29T14:38:34.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f32ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f95ec0cf7d0 con 0x7f95ec143840 2024-06-29T14:38:34.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f1048640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95ec06ea10 0x7f95ec154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:34.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f1048640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95ec06ea10 0x7f95ec154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45486/0 (socket says 172.21.15.59:45486) 2024-06-29T14:38:34.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f1048640 1 -- 172.21.15.59:0/2522102981 learned_addr learned my addr 172.21.15.59:0/2522102981 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:34.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f0847640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95ec0d0130 0x7f95ec157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:34.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f1849640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95ec143840 0x7f95ec157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:34.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f0847640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95ec143840 msgr2=0x7f95ec157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:34.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f0847640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95ec143840 0x7f95ec157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f0847640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95ec06ea10 msgr2=0x7f95ec154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:34.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f0847640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95ec06ea10 0x7f95ec154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f0847640 1 -- 172.21.15.59:0/2522102981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ec13dce0 con 0x7f95ec0d0130 2024-06-29T14:38:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f1048640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95ec06ea10 0x7f95ec154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.154+0000 7f95f1849640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95ec143840 0x7f95ec157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.155+0000 7f95f0847640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95ec0d0130 0x7f95ec157620 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0b4610 tx=0x7f95e0033c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.155+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0005c90 con 0x7f95ec0d0130 2024-06-29T14:38:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.155+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95e0031e50 con 0x7f95ec0d0130 2024-06-29T14:38:34.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.155+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ec13dfc0 con 0x7f95ec0d0130 2024-06-29T14:38:34.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.155+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0005280 con 0x7f95ec0d0130 2024-06-29T14:38:34.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.155+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ec13e580 con 0x7f95ec0d0130 2024-06-29T14:38:34.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.156+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95ec0cfc20 con 0x7f95ec0d0130 2024-06-29T14:38:34.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.157+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 47) ==== 98320+0+0 (secure 0 0 0) 0x7f95e0005420 con 0x7f95ec0d0130 2024-06-29T14:38:34.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.157+0000 7f95ddffb640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 0x7f95d0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:34.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.157+0000 7f95f1048640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 msgr2=0x7f95d0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1090607011 2024-06-29T14:38:34.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.157+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6101+0+0 (secure 0 0 0) 0x7f95e00b89e0 con 0x7f95ec0d0130 2024-06-29T14:38:34.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.157+0000 7f95f1048640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 0x7f95d0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:34.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.162+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95e003f030 con 0x7f95ec0d0130 2024-06-29T14:38:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.358+0000 7f95f1048640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 msgr2=0x7f95d0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1090607011 2024-06-29T14:38:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.358+0000 7f95f1048640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 0x7f95d0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:34.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.378+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 --> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f95ec159f30 con 0x7f95d0076440 2024-06-29T14:38:34.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.760+0000 7f95f1048640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 msgr2=0x7f95d0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1090607011 2024-06-29T14:38:34.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:34.760+0000 7f95f1048640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 0x7f95d0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:38:35.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:35.562+0000 7f95f1048640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 msgr2=0x7f95d0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1090607011 2024-06-29T14:38:35.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:35.562+0000 7f95f1048640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 0x7f95d0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:38:36.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:36.899+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 48) ==== 97929+0+0 (secure 0 0 0) 0x7f95e00b6d70 con 0x7f95ec0d0130 2024-06-29T14:38:36.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:36.899+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 msgr2=0x7f95d0078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:36.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:36.899+0000 7f95ddffb640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/1090607011,v1:172.21.15.153:6833/1090607011] conn(0x7f95d0076440 0x7f95d0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:37.051 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9f0857640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9f0857640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9f0857640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9f0857640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9f0857640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9ed851640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9ed851640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9ed851640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9ed851640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:38:37.054+0000 7fb9ed851640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.901+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mgrmap(e 49) ==== 98056+0+0 (secure 0 0 0) 0x7f95e007dbd0 con 0x7f95ec0d0130 2024-06-29T14:38:37.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.901+0000 7f95ddffb640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f95d003afd0 0x7f95d00809a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:37.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.901+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 --> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f95e003ae50 con 0x7f95d003afd0 2024-06-29T14:38:37.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.904+0000 7f95f1048640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f95d003afd0 0x7f95d00809a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:37.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.907+0000 7f95f1048640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f95d003afd0 0x7f95d00809a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f95e8007b10 tx=0x7f95e8007a60 comp rx=0 tx=0).ready entity=mgr.7510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:37.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.976+0000 7f95ddffb640 1 -- 172.21.15.59:0/2522102981 <== mgr.7510 v2:172.21.15.153:6832/716775118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f95e003ae50 con 0x7f95d003afd0 2024-06-29T14:38:37.974 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /bad-path 2024-06-29T14:38:37.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f95d003afd0 msgr2=0x7f95d00809a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:37.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f32ac640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f95d003afd0 0x7f95d00809a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f95e8007b10 tx=0x7f95e8007a60 comp rx=0 tx=0).stop 2024-06-29T14:38:37.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95ec0d0130 msgr2=0x7f95ec157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:37.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f32ac640 1 --2- 172.21.15.59:0/2522102981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95ec0d0130 0x7f95ec157620 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0b4610 tx=0x7f95e0033c20 comp rx=0 tx=0).stop 2024-06-29T14:38:37.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f1849640 1 -- 172.21.15.59:0/2522102981 reap_dead start 2024-06-29T14:38:37.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 shutdown_connections 2024-06-29T14:38:37.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 >> 172.21.15.59:0/2522102981 conn(0x7f95ec0b2ab0 msgr2=0x7f95ec159020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:37.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 shutdown_connections 2024-06-29T14:38:37.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:37.978+0000 7f95f32ac640 1 -- 172.21.15.59:0/2522102981 wait complete. 2024-06-29T14:38:37.989 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-06-29T14:38:37.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-06-29T14:38:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 -- 172.21.15.59:0/3857127491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f591415e0a0 msgr2=0x7f5914161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 --2- 172.21.15.59:0/3857127491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f591415e0a0 0x7f5914161a80 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f59140b35d0 tx=0x7f590802a4b0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 -- 172.21.15.59:0/3857127491 shutdown_connections 2024-06-29T14:38:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 --2- 172.21.15.59:0/3857127491 >> v2:172.21.15.153:3300/0 conn(0x7f5914161fc0 0x7f59140d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 --2- 172.21.15.59:0/3857127491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f591415e0a0 0x7f5914161a80 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 --2- 172.21.15.59:0/3857127491 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5914162770 0x7f59140ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 -- 172.21.15.59:0/3857127491 >> 172.21.15.59:0/3857127491 conn(0x7f59140d45b0 msgr2=0x7f59140d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 -- 172.21.15.59:0/3857127491 shutdown_connections 2024-06-29T14:38:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.116+0000 7f591a826640 1 -- 172.21.15.59:0/3857127491 wait complete. 2024-06-29T14:38:38.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.118+0000 7f591a826640 1 Processor -- start 2024-06-29T14:38:38.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.118+0000 7f591a826640 1 -- start start 2024-06-29T14:38:38.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f591a826640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591415e0a0 0x7f5914150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f591a826640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5914161fc0 0x7f5914160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f591a826640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5914162770 0x7f59140c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f591a826640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f59140d2f40 con 0x7f5914162770 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f591a826640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59140d2dc0 con 0x7f591415e0a0 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f591a826640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f59140d30c0 con 0x7f5914161fc0 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f5913fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591415e0a0 0x7f5914150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f5918dc3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5914162770 0x7f59140c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f5913fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591415e0a0 0x7f5914150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:49302/0 (socket says 172.21.15.59:49302) 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f5913fff640 1 -- 172.21.15.59:0/2829436753 learned_addr learned my addr 172.21.15.59:0/2829436753 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.119+0000 7f59137fe640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5914161fc0 0x7f5914160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5918dc3640 1 -- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5914161fc0 msgr2=0x7f5914160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5918dc3640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5914161fc0 0x7f5914160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5918dc3640 1 -- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591415e0a0 msgr2=0x7f5914150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5918dc3640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591415e0a0 0x7f5914150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5918dc3640 1 -- 172.21.15.59:0/2829436753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5914160f80 con 0x7f5914162770 2024-06-29T14:38:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5918dc3640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5914162770 0x7f59140c9f50 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f590c005fd0 tx=0x7f590c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5910ff9640 1 -- 172.21.15.59:0/2829436753 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c004d20 con 0x7f5914162770 2024-06-29T14:38:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5910ff9640 1 -- 172.21.15.59:0/2829436753 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f590c00ca60 con 0x7f5914162770 2024-06-29T14:38:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5914161260 con 0x7f5914162770 2024-06-29T14:38:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f5910ff9640 1 -- 172.21.15.59:0/2829436753 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c005020 con 0x7f5914162770 2024-06-29T14:38:38.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.120+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5914161820 con 0x7f5914162770 2024-06-29T14:38:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.121+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5914161da0 con 0x7f5914162770 2024-06-29T14:38:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.122+0000 7f5910ff9640 1 -- 172.21.15.59:0/2829436753 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 49) ==== 98056+0+0 (secure 0 0 0) 0x7f590c0051c0 con 0x7f5914162770 2024-06-29T14:38:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.122+0000 7f5910ff9640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f5900076170 0x7f5900078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.122+0000 7f5910ff9640 1 -- 172.21.15.59:0/2829436753 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6445+0+0 (secure 0 0 0) 0x7f590c097dc0 con 0x7f5914162770 2024-06-29T14:38:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.123+0000 7f5913fff640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f5900076170 0x7f5900078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:38.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.123+0000 7f5913fff640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f5900076170 0x7f5900078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f590400bc00 tx=0x7f590400b390 comp rx=0 tx=0).ready entity=mgr.7510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:38.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.127+0000 7f5910ff9640 1 -- 172.21.15.59:0/2829436753 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f590c016030 con 0x7f5914162770 2024-06-29T14:38:38.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.339+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 --> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f59140b95e0 con 0x7f5900076170 2024-06-29T14:38:38.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.346+0000 7f5910ff9640 1 -- 172.21.15.59:0/2829436753 <== mgr.7510 v2:172.21.15.153:6832/716775118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7f59140b95e0 con 0x7f5900076170 2024-06-29T14:38:38.344 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:38:38.344 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:38:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-06-29T14:38:38.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f5900076170 msgr2=0x7f5900078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:38.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f5900076170 0x7f5900078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f590400bc00 tx=0x7f590400b390 comp rx=0 tx=0).stop 2024-06-29T14:38:38.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5914162770 msgr2=0x7f59140c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:38.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5914162770 0x7f59140c9f50 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f590c005fd0 tx=0x7f590c00edf0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 shutdown_connections 2024-06-29T14:38:38.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f5900076170 0x7f5900078680 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5914162770 0x7f59140c9f50 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5914161fc0 0x7f5914160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 --2- 172.21.15.59:0/2829436753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f591415e0a0 0x7f5914150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 >> 172.21.15.59:0/2829436753 conn(0x7f59140d45b0 msgr2=0x7f59140b8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:38.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 shutdown_connections 2024-06-29T14:38:38.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:38:38.348+0000 7f591a826640 1 -- 172.21.15.59:0/2829436753 wait complete. 2024-06-29T14:38:38.357 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:38:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-06-29T14:38:56.257 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:38:56.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:00.984 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:00.986+0000 7fb9e0036640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:00.984 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:00.986+0000 7fb9e0036640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:00.984 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:00.986+0000 7fb9e0036640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:00.984 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:00.986+0000 7fb9e0036640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:00.984 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:00.986+0000 7fb9e0036640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:26.511 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:26.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:38.359 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-06-29T14:39:38.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.493+0000 7f89db903640 1 -- 172.21.15.59:0/2661748154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 msgr2=0x7f89d415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.493+0000 7f89db903640 1 --2- 172.21.15.59:0/2661748154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 0x7f89d415e8d0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f89d40d15d0 tx=0x7f89c402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.494+0000 7f89db903640 1 -- 172.21.15.59:0/2661748154 shutdown_connections 2024-06-29T14:39:38.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.494+0000 7f89db903640 1 --2- 172.21.15.59:0/2661748154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 0x7f89d415e8d0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.494+0000 7f89db903640 1 --2- 172.21.15.59:0/2661748154 >> v2:172.21.15.153:3300/0 conn(0x7f89d40d23c0 0x7f89d40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.494+0000 7f89db903640 1 -- 172.21.15.59:0/2661748154 >> 172.21.15.59:0/2661748154 conn(0x7f89d40d3ee0 msgr2=0x7f89d40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.494+0000 7f89db903640 1 -- 172.21.15.59:0/2661748154 shutdown_connections 2024-06-29T14:39:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.494+0000 7f89db903640 1 -- 172.21.15.59:0/2661748154 wait complete. 2024-06-29T14:39:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.495+0000 7f89db903640 1 Processor -- start 2024-06-29T14:39:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.496+0000 7f89db903640 1 -- start start 2024-06-29T14:39:38.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89db903640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 0x7f89d40ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89db903640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89d40d23c0 0x7f89d41557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89db903640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89d415ee10 0x7f89d41f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89db903640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4160240 con 0x7f89d40b3ed0 2024-06-29T14:39:38.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89db903640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f89d41600c0 con 0x7f89d40d23c0 2024-06-29T14:39:38.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89db903640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f89d41603c0 con 0x7f89d415ee10 2024-06-29T14:39:38.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d969f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 0x7f89d40ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d969f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 0x7f89d40ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34700/0 (socket says 172.21.15.59:34700) 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d969f640 1 -- 172.21.15.59:0/3655044226 learned_addr learned my addr 172.21.15.59:0/3655044226 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d8e9e640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89d40d23c0 0x7f89d41557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d9ea0640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89d415ee10 0x7f89d41f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d969f640 1 -- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89d415ee10 msgr2=0x7f89d41f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d969f640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89d415ee10 0x7f89d41f5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d969f640 1 -- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89d40d23c0 msgr2=0x7f89d41557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d969f640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89d40d23c0 0x7f89d41557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d969f640 1 -- 172.21.15.59:0/3655044226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d40c7800 con 0x7f89d40b3ed0 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.497+0000 7f89d9ea0640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89d415ee10 0x7f89d41f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.498+0000 7f89d969f640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 0x7f89d40ca030 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f89d000d780 tx=0x7f89d000dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.498+0000 7f89c9ffb640 1 -- 172.21.15.59:0/3655044226 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d0017070 con 0x7f89d40b3ed0 2024-06-29T14:39:38.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.498+0000 7f89c9ffb640 1 -- 172.21.15.59:0/3655044226 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89d0009b10 con 0x7f89d40b3ed0 2024-06-29T14:39:38.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.498+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d4155d20 con 0x7f89d40b3ed0 2024-06-29T14:39:38.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.498+0000 7f89c9ffb640 1 -- 172.21.15.59:0/3655044226 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d0010460 con 0x7f89d40b3ed0 2024-06-29T14:39:38.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.498+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d40c7dc0 con 0x7f89d40b3ed0 2024-06-29T14:39:38.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.499+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d415e8d0 con 0x7f89d40b3ed0 2024-06-29T14:39:38.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.500+0000 7f89c9ffb640 1 -- 172.21.15.59:0/3655044226 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 52) ==== 98434+0+0 (secure 0 0 0) 0x7f89d001a070 con 0x7f89d40b3ed0 2024-06-29T14:39:38.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.500+0000 7f89c9ffb640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f89b8076590 0x7f89b8078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.500+0000 7f89c9ffb640 1 -- 172.21.15.59:0/3655044226 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7f89d0098110 con 0x7f89d40b3ed0 2024-06-29T14:39:38.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.500+0000 7f89d8e9e640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f89b8076590 0x7f89b8078aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.501+0000 7f89d8e9e640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f89b8076590 0x7f89b8078aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f89c4033000 tx=0x7f89c4004730 comp rx=0 tx=0).ready entity=mgr.7510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.505+0000 7f89c9ffb640 1 -- 172.21.15.59:0/3655044226 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f89d0060c50 con 0x7f89d40b3ed0 2024-06-29T14:39:38.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.723+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 --> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89d40c7370 con 0x7f89b8076590 2024-06-29T14:39:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.730+0000 7f89c9ffb640 1 -- 172.21.15.59:0/3655044226 <== mgr.7510 v2:172.21.15.153:6832/716775118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+303 (secure 0 0 0) 0x7f89d40c7370 con 0x7f89b8076590 2024-06-29T14:39:38.729 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:38.729 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:38:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-06-29T14:39:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.732+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f89b8076590 msgr2=0x7f89b8078aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.732+0000 7f89db903640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f89b8076590 0x7f89b8078aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f89c4033000 tx=0x7f89c4004730 comp rx=0 tx=0).stop 2024-06-29T14:39:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.732+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 msgr2=0x7f89d40ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.732+0000 7f89db903640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 0x7f89d40ca030 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f89d000d780 tx=0x7f89d000dc40 comp rx=0 tx=0).stop 2024-06-29T14:39:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.733+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 shutdown_connections 2024-06-29T14:39:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.733+0000 7f89db903640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f89b8076590 0x7f89b8078aa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.733+0000 7f89db903640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89d415ee10 0x7f89d41f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.733+0000 7f89db903640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89d40d23c0 0x7f89d41557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.733+0000 7f89db903640 1 --2- 172.21.15.59:0/3655044226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89d40b3ed0 0x7f89d40ca030 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.733+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 >> 172.21.15.59:0/3655044226 conn(0x7f89d40d3ee0 msgr2=0x7f89d406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.733+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 shutdown_connections 2024-06-29T14:39:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.733+0000 7f89db903640 1 -- 172.21.15.59:0/3655044226 wait complete. 2024-06-29T14:39:38.742 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:38:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-06-29T14:39:38.742 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-06-29T14:39:38.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.865+0000 7efe4b4ac640 1 -- 172.21.15.59:0/97901020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe2c0051a0 msgr2=0x7efe2c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.865+0000 7efe4b4ac640 1 --2- 172.21.15.59:0/97901020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe2c0051a0 0x7efe2c00b9c0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7efe44074d20 tx=0x7efe2c005830 comp rx=0 tx=0).stop 2024-06-29T14:39:38.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.865+0000 7efe49a49640 1 -- 172.21.15.59:0/97901020 reap_dead start 2024-06-29T14:39:38.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.865+0000 7efe4b4ac640 1 -- 172.21.15.59:0/97901020 shutdown_connections 2024-06-29T14:39:38.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.865+0000 7efe4b4ac640 1 -- 172.21.15.59:0/97901020 >> 172.21.15.59:0/97901020 conn(0x7efe4406c950 msgr2=0x7efe44073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:38.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.865+0000 7efe4b4ac640 1 -- 172.21.15.59:0/97901020 shutdown_connections 2024-06-29T14:39:38.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.865+0000 7efe4b4ac640 1 -- 172.21.15.59:0/97901020 wait complete. 2024-06-29T14:39:38.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.867+0000 7efe4b4ac640 1 Processor -- start 2024-06-29T14:39:38.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.867+0000 7efe4b4ac640 1 -- start start 2024-06-29T14:39:38.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe4b4ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe440b6c40 0x7efe44165f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe4b4ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe440ced20 0x7efe44141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe4b4ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe44141aa0 0x7efe44164120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe4b4ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efe440b20a0 con 0x7efe440ced20 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe4b4ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe440b1f20 con 0x7efe440b6c40 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe4b4ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efe44166c70 con 0x7efe44141aa0 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe48a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe440ced20 0x7efe44141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe48a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe440ced20 0x7efe44141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34708/0 (socket says 172.21.15.59:34708) 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe48a47640 1 -- 172.21.15.59:0/2157613492 learned_addr learned my addr 172.21.15.59:0/2157613492 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe49a49640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe44141aa0 0x7efe44164120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe49248640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe440b6c40 0x7efe44165f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe48a47640 1 -- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe44141aa0 msgr2=0x7efe44164120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe48a47640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe44141aa0 0x7efe44164120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe48a47640 1 -- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe440b6c40 msgr2=0x7efe44165f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe48a47640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe440b6c40 0x7efe44165f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe48a47640 1 -- 172.21.15.59:0/2157613492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe440b6480 con 0x7efe440ced20 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.868+0000 7efe49248640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe440b6c40 0x7efe44165f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.869+0000 7efe48a47640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe440ced20 0x7efe44141560 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7efe3800e590 tx=0x7efe3800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.869+0000 7efe41ffb640 1 -- 172.21.15.59:0/2157613492 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38015070 con 0x7efe440ced20 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.869+0000 7efe41ffb640 1 -- 172.21.15.59:0/2157613492 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe38010980 con 0x7efe440ced20 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.869+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe44166500 con 0x7efe440ced20 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.869+0000 7efe41ffb640 1 -- 172.21.15.59:0/2157613492 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38010db0 con 0x7efe440ced20 2024-06-29T14:39:38.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.869+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe440cf130 con 0x7efe440ced20 2024-06-29T14:39:38.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.870+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe440cf3c0 con 0x7efe440ced20 2024-06-29T14:39:38.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.871+0000 7efe41ffb640 1 -- 172.21.15.59:0/2157613492 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 52) ==== 98434+0+0 (secure 0 0 0) 0x7efe38018070 con 0x7efe440ced20 2024-06-29T14:39:38.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.871+0000 7efe41ffb640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7efe30076560 0x7efe30078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.871+0000 7efe41ffb640 1 -- 172.21.15.59:0/2157613492 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7efe38098330 con 0x7efe440ced20 2024-06-29T14:39:38.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.872+0000 7efe49248640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7efe30076560 0x7efe30078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.872+0000 7efe49248640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7efe30076560 0x7efe30078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe34006b70 tx=0x7efe34006b00 comp rx=0 tx=0).ready entity=mgr.7510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:38.876+0000 7efe41ffb640 1 -- 172.21.15.59:0/2157613492 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efe38060e70 con 0x7efe440ced20 2024-06-29T14:39:39.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.089+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 --> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) -- 0x7efe44070660 con 0x7efe30076560 2024-06-29T14:39:39.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.165+0000 7efe41ffb640 1 -- 172.21.15.59:0/2157613492 <== mgr.7510 v2:172.21.15.153:6832/716775118 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+36 (secure 0 0 0) 0x7efe44070660 con 0x7efe30076560 2024-06-29T14:39:39.164 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /bad-path 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7efe30076560 msgr2=0x7efe30078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7efe30076560 0x7efe30078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efe34006b70 tx=0x7efe34006b00 comp rx=0 tx=0).stop 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe440ced20 msgr2=0x7efe44141560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe440ced20 0x7efe44141560 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7efe3800e590 tx=0x7efe3800e990 comp rx=0 tx=0).stop 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 shutdown_connections 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7efe30076560 0x7efe30078a70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe44141aa0 0x7efe44164120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe440ced20 0x7efe44141560 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 --2- 172.21.15.59:0/2157613492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe440b6c40 0x7efe44165f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 >> 172.21.15.59:0/2157613492 conn(0x7efe4406c950 msgr2=0x7efe440cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:39.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 shutdown_connections 2024-06-29T14:39:39.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.167+0000 7efe4b4ac640 1 -- 172.21.15.59:0/2157613492 wait complete. 2024-06-29T14:39:39.177 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-06-29T14:39:39.177 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:39:39.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.298+0000 7f79b6f18640 1 -- 172.21.15.59:0/2099018441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7998004800 msgr2=0x7f7998004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:39.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.298+0000 7f79b6f18640 1 --2- 172.21.15.59:0/2099018441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7998004800 0x7f7998004c10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f79b008c6c0 tx=0x7f7998005830 comp rx=0 tx=0).stop 2024-06-29T14:39:39.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.298+0000 7f79b54b5640 1 -- 172.21.15.59:0/2099018441 reap_dead start 2024-06-29T14:39:39.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.298+0000 7f79b6f18640 1 -- 172.21.15.59:0/2099018441 shutdown_connections 2024-06-29T14:39:39.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.298+0000 7f79b6f18640 1 -- 172.21.15.59:0/2099018441 >> 172.21.15.59:0/2099018441 conn(0x7f79b00d45b0 msgr2=0x7f79b00d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:39.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.298+0000 7f79b6f18640 1 -- 172.21.15.59:0/2099018441 shutdown_connections 2024-06-29T14:39:39.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.298+0000 7f79b6f18640 1 -- 172.21.15.59:0/2099018441 wait complete. 2024-06-29T14:39:39.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.300+0000 7f79b6f18640 1 Processor -- start 2024-06-29T14:39:39.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.300+0000 7f79b6f18640 1 -- start start 2024-06-29T14:39:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b6f18640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7998004800 0x7f79b0159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b6f18640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b0152910 0x7f79b015fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b6f18640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b0152dd0 0x7f79b0160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b6f18640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f79b00ce6d0 con 0x7f79b0152dd0 2024-06-29T14:39:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b6f18640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79b00ce550 con 0x7f7998004800 2024-06-29T14:39:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b6f18640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f79b00ce850 con 0x7f79b0152910 2024-06-29T14:39:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b0152dd0 0x7f79b0160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b0152dd0 0x7f79b0160070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34730/0 (socket says 172.21.15.59:34730) 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 -- 172.21.15.59:0/2745138553 learned_addr learned my addr 172.21.15.59:0/2745138553 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79affff640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b0152910 0x7f79b015fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b4cb4640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7998004800 0x7f79b0159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 -- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b0152910 msgr2=0x7f79b015fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b0152910 0x7f79b015fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 -- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7998004800 msgr2=0x7f79b0159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7998004800 0x7f79b0159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 -- 172.21.15.59:0/2745138553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b0157b40 con 0x7f79b0152dd0 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79affff640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b0152910 0x7f79b015fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b4cb4640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7998004800 0x7f79b0159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.302+0000 7f79b54b5640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b0152dd0 0x7f79b0160070 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f79b00d3730 tx=0x7f79a80064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.303+0000 7f79ad7fa640 1 -- 172.21.15.59:0/2745138553 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a8011900 con 0x7f79b0152dd0 2024-06-29T14:39:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.303+0000 7f79ad7fa640 1 -- 172.21.15.59:0/2745138553 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79a8011ac0 con 0x7f79b0152dd0 2024-06-29T14:39:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.303+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b00cbc70 con 0x7f79b0152dd0 2024-06-29T14:39:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.303+0000 7f79ad7fa640 1 -- 172.21.15.59:0/2745138553 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a8010420 con 0x7f79b0152dd0 2024-06-29T14:39:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.303+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b00cc230 con 0x7f79b0152dd0 2024-06-29T14:39:39.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.304+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b00b38e0 con 0x7f79b0152dd0 2024-06-29T14:39:39.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.304+0000 7f79ad7fa640 1 -- 172.21.15.59:0/2745138553 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 52) ==== 98434+0+0 (secure 0 0 0) 0x7f79a80105c0 con 0x7f79b0152dd0 2024-06-29T14:39:39.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.305+0000 7f79ad7fa640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f799c076560 0x7f799c078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:39.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.305+0000 7f79ad7fa640 1 -- 172.21.15.59:0/2745138553 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7f79a8098780 con 0x7f79b0152dd0 2024-06-29T14:39:39.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.305+0000 7f79b4cb4640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f799c076560 0x7f799c078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:39.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.306+0000 7f79b4cb4640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f799c076560 0x7f799c078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79a0000c00 tx=0x7f79a0006b70 comp rx=0 tx=0).ready entity=mgr.7510 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.309+0000 7f79ad7fa640 1 -- 172.21.15.59:0/2745138553 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f79a8061a20 con 0x7f79b0152dd0 2024-06-29T14:39:39.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:39.549+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f79b006c840 con 0x7f79b0152dd0 2024-06-29T14:39:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.346+0000 7f79ad7fa640 1 -- 172.21.15.59:0/2745138553 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v53) ==== 93+0+0 (secure 0 0 0) 0x7f79a8061320 con 0x7f79b0152dd0 2024-06-29T14:39:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.349+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f799c076560 msgr2=0x7f799c078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.349+0000 7f79b6f18640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f799c076560 0x7f799c078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79a0000c00 tx=0x7f79a0006b70 comp rx=0 tx=0).stop 2024-06-29T14:39:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.349+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b0152dd0 msgr2=0x7f79b0160070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.349+0000 7f79b6f18640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b0152dd0 0x7f79b0160070 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f79b00d3730 tx=0x7f79a80064f0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.350+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 shutdown_connections 2024-06-29T14:39:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.350+0000 7f79b6f18640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f799c076560 0x7f799c078a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.350+0000 7f79b6f18640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b0152dd0 0x7f79b0160070 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.350+0000 7f79b6f18640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b0152910 0x7f79b015fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.350+0000 7f79b6f18640 1 --2- 172.21.15.59:0/2745138553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7998004800 0x7f79b0159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.350+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 >> 172.21.15.59:0/2745138553 conn(0x7f79b00d45b0 msgr2=0x7f79b00b8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.351+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 shutdown_connections 2024-06-29T14:39:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.351+0000 7f79b6f18640 1 -- 172.21.15.59:0/2745138553 wait complete. 2024-06-29T14:39:40.365 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:39:40.365 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:39:40.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:39:40.453 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:39:40.453 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:39:40.453 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:39:40.453 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:40.470 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:39:40.470 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:40.527 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 -- 172.21.15.59:0/3620044288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 msgr2=0x7f97880cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 --2- 172.21.15.59:0/3620044288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 0x7f97880cf780 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f9788074c50 tx=0x7f978002aa30 comp rx=0 tx=0).stop 2024-06-29T14:39:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 -- 172.21.15.59:0/3620044288 shutdown_connections 2024-06-29T14:39:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 --2- 172.21.15.59:0/3620044288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 0x7f97880cf780 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 --2- 172.21.15.59:0/3620044288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97880b2fa0 0x7f97880d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 --2- 172.21.15.59:0/3620044288 >> v2:172.21.15.153:3300/0 conn(0x7f97880b2270 0x7f978815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 -- 172.21.15.59:0/3620044288 >> 172.21.15.59:0/3620044288 conn(0x7f9788073e10 msgr2=0x7f9788074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 -- 172.21.15.59:0/3620044288 shutdown_connections 2024-06-29T14:39:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.687+0000 7f978f768640 1 -- 172.21.15.59:0/3620044288 wait complete. 2024-06-29T14:39:40.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.689+0000 7f978f768640 1 Processor -- start 2024-06-29T14:39:40.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.689+0000 7f978f768640 1 -- start start 2024-06-29T14:39:40.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.690+0000 7f978f768640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97880b2270 0x7f97881f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.690+0000 7f978f768640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97880b2fa0 0x7f97881f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.690+0000 7f978f768640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 0x7f9788156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.690+0000 7f978f768640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97880cfe40 con 0x7f97880d1a60 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.690+0000 7f978f768640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97880cfcc0 con 0x7f97880b2270 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.690+0000 7f978f768640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f97880cffc0 con 0x7f97880b2fa0 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.690+0000 7f978dd05640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 0x7f9788156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.690+0000 7f978dd05640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 0x7f9788156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34762/0 (socket says 172.21.15.59:34762) 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978dd05640 1 -- 172.21.15.59:0/503061449 learned_addr learned my addr 172.21.15.59:0/503061449 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978d504640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97880b2270 0x7f97881f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978cd03640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97880b2fa0 0x7f97881f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978dd05640 1 -- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97880b2fa0 msgr2=0x7f97881f1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978dd05640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97880b2fa0 0x7f97881f1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978dd05640 1 -- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97880b2270 msgr2=0x7f97881f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978dd05640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97880b2270 0x7f97881f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978dd05640 1 -- 172.21.15.59:0/503061449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97881f5210 con 0x7f97880d1a60 2024-06-29T14:39:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978dd05640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 0x7f9788156880 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f978002b020 tx=0x7f9780003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f9779ffb640 1 -- 172.21.15.59:0/503061449 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9780033050 con 0x7f97880d1a60 2024-06-29T14:39:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97881f54f0 con 0x7f97880d1a60 2024-06-29T14:39:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f9779ffb640 1 -- 172.21.15.59:0/503061449 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9780004450 con 0x7f97880d1a60 2024-06-29T14:39:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f9779ffb640 1 -- 172.21.15.59:0/503061449 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9780004840 con 0x7f97880d1a60 2024-06-29T14:39:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.691+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97880ada50 con 0x7f97880d1a60 2024-06-29T14:39:40.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.693+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97880d0450 con 0x7f97880d1a60 2024-06-29T14:39:40.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.693+0000 7f9779ffb640 1 -- 172.21.15.59:0/503061449 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f9780036070 con 0x7f97880d1a60 2024-06-29T14:39:40.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.694+0000 7f9779ffb640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f976c076510 0x7f976c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.694+0000 7f978d504640 1 -- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f976c076510 msgr2=0x7f976c078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:40.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.694+0000 7f9779ffb640 1 -- 172.21.15.59:0/503061449 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7f97800b94d0 con 0x7f97880d1a60 2024-06-29T14:39:40.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.694+0000 7f978d504640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f976c076510 0x7f976c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:40.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.699+0000 7f9779ffb640 1 -- 172.21.15.59:0/503061449 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9780081f00 con 0x7f97880d1a60 2024-06-29T14:39:40.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.895+0000 7f978d504640 1 -- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f976c076510 msgr2=0x7f976c078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:40.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.895+0000 7f978d504640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f976c076510 0x7f976c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:40.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.895+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9788155f70 con 0x7f97880d1a60 2024-06-29T14:39:40.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.896+0000 7f9779ffb640 1 -- 172.21.15.59:0/503061449 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+17608 (secure 0 0 0) 0x7f9780086200 con 0x7f97880d1a60 2024-06-29T14:39:40.895 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:40.895 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":117,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:38.318732+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.981044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.565073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.a","mappings":[{"from":1,"to":2}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2502091213":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3749556168":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3059657409":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3367163578":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6832/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/441578565":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6833/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/578976042":"2024-06-30T14:38:36.889100+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f976c076510 msgr2=0x7f976c078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f976c076510 0x7f976c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 msgr2=0x7f9788156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 0x7f9788156880 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f978002b020 tx=0x7f9780003f10 comp rx=0 tx=0).stop 2024-06-29T14:39:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 shutdown_connections 2024-06-29T14:39:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f976c076510 0x7f976c078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97880d1a60 0x7f9788156880 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97880b2fa0 0x7f97881f1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 --2- 172.21.15.59:0/503061449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97880b2270 0x7f97881f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 >> 172.21.15.59:0/503061449 conn(0x7f9788073e10 msgr2=0x7f9788154410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 shutdown_connections 2024-06-29T14:39:40.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:40.899+0000 7f978f768640 1 -- 172.21.15.59:0/503061449 wait complete. 2024-06-29T14:39:40.909 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:38:22.981044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:38:23.565073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:39:40.909 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:39:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.033+0000 7fc0c7689640 1 -- 172.21.15.59:0/1576792957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00b2fa0 msgr2=0x7fc0c00d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.033+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1576792957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00b2fa0 0x7fc0c00d1520 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0074c50 tx=0x7fc0b002a530 comp rx=0 tx=0).stop 2024-06-29T14:39:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.033+0000 7fc0c7689640 1 -- 172.21.15.59:0/1576792957 shutdown_connections 2024-06-29T14:39:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.033+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1576792957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00b2fa0 0x7fc0c00d1520 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.033+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1576792957 >> v2:172.21.15.153:3300/0 conn(0x7fc0c00b2270 0x7fc0c015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.033+0000 7fc0c7689640 1 -- 172.21.15.59:0/1576792957 >> 172.21.15.59:0/1576792957 conn(0x7fc0c0073e10 msgr2=0x7fc0c0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.033+0000 7fc0c7689640 1 -- 172.21.15.59:0/1576792957 shutdown_connections 2024-06-29T14:39:41.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.033+0000 7fc0c7689640 1 -- 172.21.15.59:0/1576792957 wait complete. 2024-06-29T14:39:41.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.036+0000 7fc0c7689640 1 Processor -- start 2024-06-29T14:39:41.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.036+0000 7fc0c7689640 1 -- start start 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c7689640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0c00b2270 0x7fc0c01500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c7689640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0c00b2fa0 0x7fc0c01e6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c7689640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00d1a60 0x7fc0c015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c7689640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c00b6cc0 con 0x7fc0c00d1a60 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c7689640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c00b6b40 con 0x7fc0c00b2fa0 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c7689640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c00cfe90 con 0x7fc0c00b2270 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00d1a60 0x7fc0c015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00d1a60 0x7fc0c015c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34784/0 (socket says 172.21.15.59:34784) 2024-06-29T14:39:41.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 -- 172.21.15.59:0/1568456917 learned_addr learned my addr 172.21.15.59:0/1568456917 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5425640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0c00b2270 0x7fc0c01500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c4c24640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0c00b2fa0 0x7fc0c01e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 -- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0c00b2270 msgr2=0x7fc0c01500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0c00b2270 0x7fc0c01500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 -- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0c00b2fa0 msgr2=0x7fc0c01e6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0c00b2fa0 0x7fc0c01e6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 -- 172.21.15.59:0/1568456917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c01e56b0 con 0x7fc0c00d1a60 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c4c24640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0c00b2fa0 0x7fc0c01e6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.039+0000 7fc0c5c26640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00d1a60 0x7fc0c015c460 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800cb40 tx=0x7fc0b800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.040+0000 7fc0b5ffb640 1 -- 172.21.15.59:0/1568456917 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8016020 con 0x7fc0c00d1a60 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.040+0000 7fc0b5ffb640 1 -- 172.21.15.59:0/1568456917 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0b800ad50 con 0x7fc0c00d1a60 2024-06-29T14:39:41.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.040+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c01e5950 con 0x7fc0c00d1a60 2024-06-29T14:39:41.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.040+0000 7fc0b5ffb640 1 -- 172.21.15.59:0/1568456917 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8012680 con 0x7fc0c00d1a60 2024-06-29T14:39:41.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.040+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c0150610 con 0x7fc0c00d1a60 2024-06-29T14:39:41.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.041+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c00d03b0 con 0x7fc0c00d1a60 2024-06-29T14:39:41.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.041+0000 7fc0b5ffb640 1 -- 172.21.15.59:0/1568456917 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7fc0b8019020 con 0x7fc0c00d1a60 2024-06-29T14:39:41.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.042+0000 7fc0b5ffb640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc0a4076510 0x7fc0a4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.042+0000 7fc0b5ffb640 1 -- 172.21.15.59:0/1568456917 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7fc0b8098410 con 0x7fc0c00d1a60 2024-06-29T14:39:41.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.042+0000 7fc0c5425640 1 -- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc0a4076510 msgr2=0x7fc0a4078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:41.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.042+0000 7fc0c5425640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc0a4076510 0x7fc0a4078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:41.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.046+0000 7fc0b5ffb640 1 -- 172.21.15.59:0/1568456917 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0b8020020 con 0x7fc0c00d1a60 2024-06-29T14:39:41.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.244+0000 7fc0c5425640 1 -- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc0a4076510 msgr2=0x7fc0a4078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:41.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.244+0000 7fc0c5425640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc0a4076510 0x7fc0a4078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.280+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0c015bd40 con 0x7fc0c00d1a60 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.281+0000 7fc0b5ffb640 1 -- 172.21.15.59:0/1568456917 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) ==== 93+0+10 (secure 0 0 0) 0x7fc0b8061060 con 0x7fc0c00d1a60 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:39:41.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc0a4076510 msgr2=0x7fc0a4078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:41.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc0a4076510 0x7fc0a4078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00d1a60 msgr2=0x7fc0c015c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00d1a60 0x7fc0c015c460 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800cb40 tx=0x7fc0b800a060 comp rx=0 tx=0).stop 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 shutdown_connections 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc0a4076510 0x7fc0a4078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c00d1a60 0x7fc0c015c460 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0c00b2fa0 0x7fc0c01e6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 --2- 172.21.15.59:0/1568456917 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0c00b2270 0x7fc0c01500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 >> 172.21.15.59:0/1568456917 conn(0x7fc0c0073e10 msgr2=0x7fc0c00d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 shutdown_connections 2024-06-29T14:39:41.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.283+0000 7fc0c7689640 1 -- 172.21.15.59:0/1568456917 wait complete. 2024-06-29T14:39:41.293 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:39:41.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.419+0000 7fdf1e152640 1 -- 172.21.15.59:0/1606978402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf1815efe0 msgr2=0x7fdf1815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.419+0000 7fdf1e152640 1 --2- 172.21.15.59:0/1606978402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf1815efe0 0x7fdf1815f9b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdf10005fd0 tx=0x7fdf1002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.419+0000 7fdf1e152640 1 -- 172.21.15.59:0/1606978402 shutdown_connections 2024-06-29T14:39:41.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.419+0000 7fdf1e152640 1 --2- 172.21.15.59:0/1606978402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf1815efe0 0x7fdf1815f9b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.419+0000 7fdf1e152640 1 --2- 172.21.15.59:0/1606978402 >> v2:172.21.15.153:3300/0 conn(0x7fdf180d23c0 0x7fdf180b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.419+0000 7fdf1e152640 1 -- 172.21.15.59:0/1606978402 >> 172.21.15.59:0/1606978402 conn(0x7fdf180d3ee0 msgr2=0x7fdf180d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.419+0000 7fdf1e152640 1 -- 172.21.15.59:0/1606978402 shutdown_connections 2024-06-29T14:39:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.419+0000 7fdf1e152640 1 -- 172.21.15.59:0/1606978402 wait complete. 2024-06-29T14:39:41.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.421+0000 7fdf1e152640 1 Processor -- start 2024-06-29T14:39:41.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.421+0000 7fdf1e152640 1 -- start start 2024-06-29T14:39:41.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.422+0000 7fdf1e152640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf180b3ed0 0x7fdf181e6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.422+0000 7fdf1e152640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf180d23c0 0x7fdf18154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.422+0000 7fdf1e152640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1815efe0 0x7fdf18154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.422+0000 7fdf1e152640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdf18160280 con 0x7fdf1815efe0 2024-06-29T14:39:41.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.422+0000 7fdf1e152640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf18160100 con 0x7fdf180b3ed0 2024-06-29T14:39:41.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.422+0000 7fdf1e152640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdf18160400 con 0x7fdf180d23c0 2024-06-29T14:39:41.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1815efe0 0x7fdf18154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1815efe0 0x7fdf18154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34800/0 (socket says 172.21.15.59:34800) 2024-06-29T14:39:41.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 -- 172.21.15.59:0/817762729 learned_addr learned my addr 172.21.15.59:0/817762729 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:41.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf177fe640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf180b3ed0 0x7fdf181e6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf16ffd640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf180d23c0 0x7fdf18154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 -- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf180d23c0 msgr2=0x7fdf18154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf180d23c0 0x7fdf18154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 -- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf180b3ed0 msgr2=0x7fdf181e6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf180b3ed0 0x7fdf181e6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 -- 172.21.15.59:0/817762729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf180cca10 con 0x7fdf1815efe0 2024-06-29T14:39:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf177fe640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf180b3ed0 0x7fdf181e6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:39:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf17fff640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1815efe0 0x7fdf18154750 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009cd0 tx=0x7fdf10031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdefffff640 1 -- 172.21.15.59:0/817762729 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10031990 con 0x7fdf1815efe0 2024-06-29T14:39:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdefffff640 1 -- 172.21.15.59:0/817762729 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf10031c20 con 0x7fdf1815efe0 2024-06-29T14:39:41.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdefffff640 1 -- 172.21.15.59:0/817762729 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10033070 con 0x7fdf1815efe0 2024-06-29T14:39:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.423+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf181507e0 con 0x7fdf1815efe0 2024-06-29T14:39:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.424+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf18150da0 con 0x7fdf1815efe0 2024-06-29T14:39:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.425+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf180b3990 con 0x7fdf1815efe0 2024-06-29T14:39:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.426+0000 7fdefffff640 1 -- 172.21.15.59:0/817762729 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7fdf10033210 con 0x7fdf1815efe0 2024-06-29T14:39:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.426+0000 7fdefffff640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fdf04076540 0x7fdf04078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.426+0000 7fdf177fe640 1 -- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fdf04076540 msgr2=0x7fdf04078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:41.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.426+0000 7fdefffff640 1 -- 172.21.15.59:0/817762729 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7fdf100b98c0 con 0x7fdf1815efe0 2024-06-29T14:39:41.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.426+0000 7fdf177fe640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fdf04076540 0x7fdf04078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:41.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.430+0000 7fdefffff640 1 -- 172.21.15.59:0/817762729 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf10051020 con 0x7fdf1815efe0 2024-06-29T14:39:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.620+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdf18156780 con 0x7fdf1815efe0 2024-06-29T14:39:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.621+0000 7fdefffff640 1 -- 172.21.15.59:0/817762729 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) ==== 104+0+11 (secure 0 0 0) 0x7fdf1003f030 con 0x7fdf1815efe0 2024-06-29T14:39:41.619 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:39:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fdf04076540 msgr2=0x7fdf04078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fdf04076540 0x7fdf04078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1815efe0 msgr2=0x7fdf18154750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1815efe0 0x7fdf18154750 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009cd0 tx=0x7fdf10031700 comp rx=0 tx=0).stop 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 shutdown_connections 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fdf04076540 0x7fdf04078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1815efe0 0x7fdf18154750 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf180d23c0 0x7fdf18154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 --2- 172.21.15.59:0/817762729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf180b3ed0 0x7fdf181e6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 >> 172.21.15.59:0/817762729 conn(0x7fdf180d3ee0 msgr2=0x7fdf181569c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 shutdown_connections 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.623+0000 7fdf1e152640 1 -- 172.21.15.59:0/817762729 wait complete. 2024-06-29T14:39:41.633 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:39:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.757+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/558545477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 msgr2=0x7f7edc15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.757+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/558545477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 0x7f7edc15f9b0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f7edc08c4f0 tx=0x7f7ed402aac0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.757+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/558545477 shutdown_connections 2024-06-29T14:39:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.757+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/558545477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 0x7f7edc15f9b0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.757+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/558545477 >> v2:172.21.15.153:3300/0 conn(0x7f7edc0b40a0 0x7f7edc15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.757+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/558545477 >> 172.21.15.59:0/558545477 conn(0x7f7edc0d3ee0 msgr2=0x7f7edc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.757+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/558545477 shutdown_connections 2024-06-29T14:39:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.757+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/558545477 wait complete. 2024-06-29T14:39:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.759+0000 7f7ee1cf6640 1 Processor -- start 2024-06-29T14:39:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.759+0000 7f7ee1cf6640 1 -- start start 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.760+0000 7f7ee1cf6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc0d23c0 0x7f7edc14db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.760+0000 7f7ee1cf6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 0x7f7edc150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.760+0000 7f7ee1cf6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc152f50 0x7f7edc153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.760+0000 7f7ee1cf6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc160280 con 0x7f7edc15ee10 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.760+0000 7f7ee1cf6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc160100 con 0x7f7edc0d23c0 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.760+0000 7f7ee1cf6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7edc160400 con 0x7f7edc152f50 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 0x7f7edc150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 0x7f7edc150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34818/0 (socket says 172.21.15.59:34818) 2024-06-29T14:39:41.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 -- 172.21.15.59:0/3811963203 learned_addr learned my addr 172.21.15.59:0/3811963203 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edb7fe640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc0d23c0 0x7f7edc14db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edbfff640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc152f50 0x7f7edc153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 -- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc152f50 msgr2=0x7f7edc153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc152f50 0x7f7edc153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 -- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc0d23c0 msgr2=0x7f7edc14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc0d23c0 0x7f7edc14db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 -- 172.21.15.59:0/3811963203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7edc1538a0 con 0x7f7edc15ee10 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7edaffd640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 0x7f7edc150760 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000d700 tx=0x7f7ed000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7ec3fff640 1 -- 172.21.15.59:0/3811963203 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0017070 con 0x7f7edc15ee10 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7ec3fff640 1 -- 172.21.15.59:0/3811963203 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ed0009b10 con 0x7f7edc15ee10 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7ec3fff640 1 -- 172.21.15.59:0/3811963203 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0010460 con 0x7f7edc15ee10 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.761+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7edc1512d0 con 0x7f7edc15ee10 2024-06-29T14:39:41.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.762+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7edc151890 con 0x7f7edc15ee10 2024-06-29T14:39:41.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.763+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7edc074570 con 0x7f7edc15ee10 2024-06-29T14:39:41.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.763+0000 7f7ec3fff640 1 -- 172.21.15.59:0/3811963203 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f7ed001a070 con 0x7f7edc15ee10 2024-06-29T14:39:41.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.764+0000 7f7ec3fff640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f7ec8076540 0x7f7ec8078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.764+0000 7f7ec3fff640 1 -- 172.21.15.59:0/3811963203 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7f7ed00980a0 con 0x7f7edc15ee10 2024-06-29T14:39:41.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.764+0000 7f7edb7fe640 1 -- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f7ec8076540 msgr2=0x7f7ec8078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:41.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.764+0000 7f7edb7fe640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f7ec8076540 0x7f7ec8078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:41.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.768+0000 7f7ec3fff640 1 -- 172.21.15.59:0/3811963203 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7edc074570 con 0x7f7edc15ee10 2024-06-29T14:39:41.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.965+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7edc0cdae0 con 0x7f7edc15ee10 2024-06-29T14:39:41.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.965+0000 7f7ec3fff640 1 -- 172.21.15.59:0/3811963203 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) ==== 100+0+11 (secure 0 0 0) 0x7f7edc0cdae0 con 0x7f7edc15ee10 2024-06-29T14:39:41.964 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:39:41.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.965+0000 7f7edb7fe640 1 -- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f7ec8076540 msgr2=0x7f7ec8078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:41.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.965+0000 7f7edb7fe640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f7ec8076540 0x7f7ec8078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:41.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f7ec8076540 msgr2=0x7f7ec8078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:41.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f7ec8076540 0x7f7ec8078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 msgr2=0x7f7edc150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 0x7f7edc150760 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000d700 tx=0x7f7ed000dbc0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 shutdown_connections 2024-06-29T14:39:41.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f7ec8076540 0x7f7ec8078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7edc152f50 0x7f7edc153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7edc15ee10 0x7f7edc150760 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 --2- 172.21.15.59:0/3811963203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7edc0d23c0 0x7f7edc14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 >> 172.21.15.59:0/3811963203 conn(0x7f7edc0d3ee0 msgr2=0x7f7edc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 shutdown_connections 2024-06-29T14:39:41.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:41.968+0000 7f7ee1cf6640 1 -- 172.21.15.59:0/3811963203 wait complete. 2024-06-29T14:39:41.978 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:39:42.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.099+0000 7f2ce463f640 1 -- 172.21.15.59:0/981664773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc0d23c0 msgr2=0x7f2cdc0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.099+0000 7f2ce463f640 1 --2- 172.21.15.59:0/981664773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc0d23c0 0x7f2cdc0b3990 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc161d40 tx=0x7f2cd802a4d0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.099+0000 7f2ce463f640 1 -- 172.21.15.59:0/981664773 shutdown_connections 2024-06-29T14:39:42.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.099+0000 7f2ce463f640 1 --2- 172.21.15.59:0/981664773 >> v2:172.21.15.153:3300/0 conn(0x7f2cdc15efe0 0x7f2cdc15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.099+0000 7f2ce463f640 1 --2- 172.21.15.59:0/981664773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc0d23c0 0x7f2cdc0b3990 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.099+0000 7f2ce463f640 1 -- 172.21.15.59:0/981664773 >> 172.21.15.59:0/981664773 conn(0x7f2cdc0d3ee0 msgr2=0x7f2cdc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.099+0000 7f2ce463f640 1 -- 172.21.15.59:0/981664773 shutdown_connections 2024-06-29T14:39:42.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.099+0000 7f2ce463f640 1 -- 172.21.15.59:0/981664773 wait complete. 2024-06-29T14:39:42.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.101+0000 7f2ce463f640 1 Processor -- start 2024-06-29T14:39:42.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.101+0000 7f2ce463f640 1 -- start start 2024-06-29T14:39:42.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce463f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cdc0b3ed0 0x7f2cdc1f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce463f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cdc0d23c0 0x7f2cdc15d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce463f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc15efe0 0x7f2cdc15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce463f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc160280 con 0x7f2cdc15efe0 2024-06-29T14:39:42.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce463f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc160100 con 0x7f2cdc0b3ed0 2024-06-29T14:39:42.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce463f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc160400 con 0x7f2cdc0d23c0 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc15efe0 0x7f2cdc15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc15efe0 0x7f2cdc15afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34824/0 (socket says 172.21.15.59:34824) 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 -- 172.21.15.59:0/3246361013 learned_addr learned my addr 172.21.15.59:0/3246361013 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce23db640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cdc0b3ed0 0x7f2cdc1f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce1bda640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cdc0d23c0 0x7f2cdc15d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 -- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cdc0d23c0 msgr2=0x7f2cdc15d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cdc0d23c0 0x7f2cdc15d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 -- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cdc0b3ed0 msgr2=0x7f2cdc1f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cdc0b3ed0 0x7f2cdc1f5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 -- 172.21.15.59:0/3246361013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc15d670 con 0x7f2cdc15efe0 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.103+0000 7f2ce2bdc640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc15efe0 0x7f2cdc15afa0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400ea40 tx=0x7f2cd400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.104+0000 7f2cceffd640 1 -- 172.21.15.59:0/3246361013 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd400ce40 con 0x7f2cdc15efe0 2024-06-29T14:39:42.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.104+0000 7f2cceffd640 1 -- 172.21.15.59:0/3246361013 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cd4004b20 con 0x7f2cdc15efe0 2024-06-29T14:39:42.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.104+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc15daf0 con 0x7f2cdc15efe0 2024-06-29T14:39:42.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.104+0000 7f2cceffd640 1 -- 172.21.15.59:0/3246361013 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4005020 con 0x7f2cdc15efe0 2024-06-29T14:39:42.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.104+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc15e0b0 con 0x7f2cdc15efe0 2024-06-29T14:39:42.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.105+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cdc0b3990 con 0x7f2cdc15efe0 2024-06-29T14:39:42.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.105+0000 7f2cceffd640 1 -- 172.21.15.59:0/3246361013 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f2cd40051c0 con 0x7f2cdc15efe0 2024-06-29T14:39:42.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.106+0000 7f2cceffd640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f2cc0076510 0x7f2cc0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.106+0000 7f2cceffd640 1 -- 172.21.15.59:0/3246361013 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7f2cd4098b60 con 0x7f2cdc15efe0 2024-06-29T14:39:42.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.106+0000 7f2ce23db640 1 -- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f2cc0076510 msgr2=0x7f2cc0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:42.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.106+0000 7f2ce23db640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f2cc0076510 0x7f2cc0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:42.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.110+0000 7f2cceffd640 1 -- 172.21.15.59:0/3246361013 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2cd40616a0 con 0x7f2cdc15efe0 2024-06-29T14:39:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.307+0000 7f2ce23db640 1 -- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f2cc0076510 msgr2=0x7f2cc0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.307+0000 7f2ce23db640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f2cc0076510 0x7f2cc0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:42.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.314+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2cdc0d2950 con 0x7f2cdc15efe0 2024-06-29T14:39:42.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.315+0000 7f2cceffd640 1 -- 172.21.15.59:0/3246361013 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v117) ==== 80+0+476 (secure 0 0 0) 0x7f2cd40659a0 con 0x7f2cdc15efe0 2024-06-29T14:39:42.313 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2502091213 2024-06-30T14:38:36.889100+0000 2024-06-29T14:39:42.313 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3749556168 2024-06-30T14:38:36.889100+0000 2024-06-29T14:39:42.313 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3059657409 2024-06-30T14:38:36.889100+0000 2024-06-29T14:39:42.313 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3367163578 2024-06-30T14:38:36.889100+0000 2024-06-29T14:39:42.313 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/1090607011 2024-06-30T14:38:36.889100+0000 2024-06-29T14:39:42.314 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/441578565 2024-06-30T14:38:36.889100+0000 2024-06-29T14:39:42.314 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/1090607011 2024-06-30T14:38:36.889100+0000 2024-06-29T14:39:42.314 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/578976042 2024-06-30T14:38:36.889100+0000 2024-06-29T14:39:42.314 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f2cc0076510 msgr2=0x7f2cc0078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f2cc0076510 0x7f2cc0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc15efe0 msgr2=0x7f2cdc15afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc15efe0 0x7f2cdc15afa0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400ea40 tx=0x7f2cd400ef00 comp rx=0 tx=0).stop 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 shutdown_connections 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f2cc0076510 0x7f2cc0078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cdc15efe0 0x7f2cdc15afa0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cdc0d23c0 0x7f2cdc15d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 --2- 172.21.15.59:0/3246361013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cdc0b3ed0 0x7f2cdc1f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.317+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 >> 172.21.15.59:0/3246361013 conn(0x7f2cdc0d3ee0 msgr2=0x7f2cdc157260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.318+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 shutdown_connections 2024-06-29T14:39:42.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.318+0000 7f2ce463f640 1 -- 172.21.15.59:0/3246361013 wait complete. 2024-06-29T14:39:42.328 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:39:42.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:39:42.328 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:39:42.358 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:39:42.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:39:42.358 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:39:42.414 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:39:42.414 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:39:42.414 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:42.430 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:39:42.430 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:39:42.430 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:39:42.430 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:42.471 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:39:42.471 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:39:42.471 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:39:42.471 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:42.487 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:39:42.487 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:42.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.631+0000 7fc6623eb640 1 -- 172.21.15.59:0/14280657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 msgr2=0x7fc65c15f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.631+0000 7fc6623eb640 1 --2- 172.21.15.59:0/14280657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 0x7fc65c15f240 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0726d0 tx=0x7fc64c02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.632+0000 7fc6623eb640 1 -- 172.21.15.59:0/14280657 shutdown_connections 2024-06-29T14:39:42.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.632+0000 7fc6623eb640 1 --2- 172.21.15.59:0/14280657 >> v2:172.21.15.153:3300/0 conn(0x7fc65c06da00 0x7fc65c073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.632+0000 7fc6623eb640 1 --2- 172.21.15.59:0/14280657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 0x7fc65c15f240 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.632+0000 7fc6623eb640 1 -- 172.21.15.59:0/14280657 >> 172.21.15.59:0/14280657 conn(0x7fc65c0b24c0 msgr2=0x7fc65c0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.632+0000 7fc6623eb640 1 -- 172.21.15.59:0/14280657 shutdown_connections 2024-06-29T14:39:42.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.632+0000 7fc6623eb640 1 -- 172.21.15.59:0/14280657 wait complete. 2024-06-29T14:39:42.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.633+0000 7fc6623eb640 1 Processor -- start 2024-06-29T14:39:42.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.634+0000 7fc6623eb640 1 -- start start 2024-06-29T14:39:42.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc6623eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 0x7fc65c15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc6623eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc65c1444c0 0x7fc65c0ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc6623eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc65c13bb80 0x7fc65c15db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc6623eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c073b40 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc6623eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc65c0739c0 con 0x7fc65c13bb80 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc6623eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc65c073cc0 con 0x7fc65c1444c0 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 0x7fc65c15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 0x7fc65c15a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34840/0 (socket says 172.21.15.59:34840) 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65bfff640 1 -- 172.21.15.59:0/4032090676 learned_addr learned my addr 172.21.15.59:0/4032090676 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc660988640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc65c13bb80 0x7fc65c15db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65b7fe640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc65c1444c0 0x7fc65c0ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65bfff640 1 -- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc65c1444c0 msgr2=0x7fc65c0ae780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65bfff640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc65c1444c0 0x7fc65c0ae780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65bfff640 1 -- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc65c13bb80 msgr2=0x7fc65c15db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65bfff640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc65c13bb80 0x7fc65c15db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.635+0000 7fc65bfff640 1 -- 172.21.15.59:0/4032090676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65c140c20 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.636+0000 7fc65bfff640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 0x7fc65c15a480 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc64c000c00 tx=0x7fc64c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.636+0000 7fc658ff9640 1 -- 172.21.15.59:0/4032090676 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c03e030 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.636+0000 7fc658ff9640 1 -- 172.21.15.59:0/4032090676 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc64c003e00 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.636+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc65c140f00 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.636+0000 7fc658ff9640 1 -- 172.21.15.59:0/4032090676 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c035040 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.636+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc65c15a9c0 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.637+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc65c0741e0 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.638+0000 7fc658ff9640 1 -- 172.21.15.59:0/4032090676 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7fc64c041030 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.638+0000 7fc658ff9640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc648076540 0x7fc648078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.639+0000 7fc65b7fe640 1 -- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc648076540 msgr2=0x7fc648078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.639+0000 7fc658ff9640 1 -- 172.21.15.59:0/4032090676 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7fc64c0b8600 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.639+0000 7fc65b7fe640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc648076540 0x7fc648078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:42.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.643+0000 7fc658ff9640 1 -- 172.21.15.59:0/4032090676 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc64c081210 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.840+0000 7fc65b7fe640 1 -- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc648076540 msgr2=0x7fc648078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:42.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.840+0000 7fc65b7fe640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc648076540 0x7fc648078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:42.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.876+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc65c0b1490 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.876+0000 7fc658ff9640 1 -- 172.21.15.59:0/4032090676 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5225 (secure 0 0 0) 0x7fc64c085510 con 0x7fc65c0cd9b0 2024-06-29T14:39:42.874 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:42.875 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"btime":"2024-06-29T14:38:28:267220+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3208152037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3208152037},{"type":"v1","addr":"172.21.15.153:6836","nonce":3208152037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6716,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/590918917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":590918917},{"type":"v1","addr":"172.21.15.153:6837","nonce":590918917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/546301975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":546301975},{"type":"v1","addr":"172.21.15.59:6833","nonce":546301975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.621714+0000","modified":"2024-06-29T14:38:27.659142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/2011834805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2011834805},{"type":"v1","addr":"172.21.15.59:6835","nonce":2011834805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":6}]} 2024-06-29T14:39:42.875 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 55 2024-06-29T14:39:42.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc648076540 msgr2=0x7fc648078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:42.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc648076540 0x7fc648078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 msgr2=0x7fc65c15a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 0x7fc65c15a480 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc64c000c00 tx=0x7fc64c003910 comp rx=0 tx=0).stop 2024-06-29T14:39:42.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 shutdown_connections 2024-06-29T14:39:42.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fc648076540 0x7fc648078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc65c13bb80 0x7fc65c15db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc65c1444c0 0x7fc65c0ae780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 --2- 172.21.15.59:0/4032090676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc65c0cd9b0 0x7fc65c15a480 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 >> 172.21.15.59:0/4032090676 conn(0x7fc65c0b24c0 msgr2=0x7fc65c136d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 shutdown_connections 2024-06-29T14:39:42.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:42.878+0000 7fc6623eb640 1 -- 172.21.15.59:0/4032090676 wait complete. 2024-06-29T14:39:42.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:43.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.008+0000 7f911257b640 1 -- 172.21.15.59:0/2520841566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c145680 msgr2=0x7f910c0b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.008+0000 7f911257b640 1 --2- 172.21.15.59:0/2520841566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c145680 0x7f910c0b3a70 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f910c075020 tx=0x7f90fc02a560 comp rx=0 tx=0).stop 2024-06-29T14:39:43.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.008+0000 7f911257b640 1 -- 172.21.15.59:0/2520841566 shutdown_connections 2024-06-29T14:39:43.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.008+0000 7f911257b640 1 --2- 172.21.15.59:0/2520841566 >> v2:172.21.15.153:3300/0 conn(0x7f910c06ca70 0x7f910c06ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.008+0000 7f911257b640 1 --2- 172.21.15.59:0/2520841566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c145680 0x7f910c0b3a70 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.008+0000 7f911257b640 1 -- 172.21.15.59:0/2520841566 >> 172.21.15.59:0/2520841566 conn(0x7f910c072d80 msgr2=0x7f910c073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.008+0000 7f911257b640 1 -- 172.21.15.59:0/2520841566 shutdown_connections 2024-06-29T14:39:43.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.008+0000 7f911257b640 1 -- 172.21.15.59:0/2520841566 wait complete. 2024-06-29T14:39:43.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.011+0000 7f911257b640 1 Processor -- start 2024-06-29T14:39:43.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.011+0000 7f911257b640 1 -- start start 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f911257b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f910c06ca70 0x7f910c15b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f911257b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f910c145680 0x7f910c13c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f911257b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c15f860 0x7f910c13c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f911257b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f910c0cfbb0 con 0x7f910c15f860 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f911257b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f910c0b41c0 con 0x7f910c06ca70 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f911257b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f910c0cfd30 con 0x7f910c145680 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c15f860 0x7f910c13c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c15f860 0x7f910c13c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43978/0 (socket says 172.21.15.59:43978) 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 -- 172.21.15.59:0/1412272933 learned_addr learned my addr 172.21.15.59:0/1412272933 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f910bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f910c06ca70 0x7f910c15b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f910b7fe640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f910c145680 0x7f910c13c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 -- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f910c145680 msgr2=0x7f910c13c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f910c145680 0x7f910c13c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 -- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f910c06ca70 msgr2=0x7f910c15b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f910c06ca70 0x7f910c15b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 -- 172.21.15.59:0/1412272933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910c15b9e0 con 0x7f910c15f860 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.014+0000 7f9110b18640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c15f860 0x7f910c13c600 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f910400c700 tx=0x7f910400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.015+0000 7f9108ff9640 1 -- 172.21.15.59:0/1412272933 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9104003b70 con 0x7f910c15f860 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.015+0000 7f9108ff9640 1 -- 172.21.15.59:0/1412272933 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9104003d10 con 0x7f910c15f860 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.015+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910c143880 con 0x7f910c15f860 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.015+0000 7f9108ff9640 1 -- 172.21.15.59:0/1412272933 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910400aab0 con 0x7f910c15f860 2024-06-29T14:39:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.015+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f910c143e40 con 0x7f910c15f860 2024-06-29T14:39:43.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.016+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910c1604f0 con 0x7f910c15f860 2024-06-29T14:39:43.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.017+0000 7f9108ff9640 1 -- 172.21.15.59:0/1412272933 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f910400ac50 con 0x7f910c15f860 2024-06-29T14:39:43.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.017+0000 7f9108ff9640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f90ec076540 0x7f90ec078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.018+0000 7f910bfff640 1 -- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f90ec076540 msgr2=0x7f90ec078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:43.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.018+0000 7f9108ff9640 1 -- 172.21.15.59:0/1412272933 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7f9104097d20 con 0x7f910c15f860 2024-06-29T14:39:43.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.018+0000 7f910bfff640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f90ec076540 0x7f90ec078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.022+0000 7f9108ff9640 1 -- 172.21.15.59:0/1412272933 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f910c06d3c0 con 0x7f910c15f860 2024-06-29T14:39:43.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.218+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f910c0b6990 con 0x7f910c15f860 2024-06-29T14:39:43.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.219+0000 7f9108ff9640 1 -- 172.21.15.59:0/1412272933 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+17608 (secure 0 0 0) 0x7f9104060850 con 0x7f910c15f860 2024-06-29T14:39:43.217 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:43.217 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":117,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:38.318732+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.981044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.565073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.a","mappings":[{"from":1,"to":2}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2502091213":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3749556168":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3059657409":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3367163578":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6832/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/441578565":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6833/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/578976042":"2024-06-30T14:38:36.889100+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:43.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.219+0000 7f910bfff640 1 -- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f90ec076540 msgr2=0x7f90ec078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:43.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.219+0000 7f910bfff640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f90ec076540 0x7f90ec078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f90ec076540 msgr2=0x7f90ec078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f90ec076540 0x7f90ec078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c15f860 msgr2=0x7f910c13c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c15f860 0x7f910c13c600 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f910400c700 tx=0x7f910400cbc0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 shutdown_connections 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f90ec076540 0x7f90ec078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f910c15f860 0x7f910c13c600 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f910c145680 0x7f910c13c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 --2- 172.21.15.59:0/1412272933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f910c06ca70 0x7f910c15b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 >> 172.21.15.59:0/1412272933 conn(0x7f910c072d80 msgr2=0x7f910c15dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 shutdown_connections 2024-06-29T14:39:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.221+0000 7f911257b640 1 -- 172.21.15.59:0/1412272933 wait complete. 2024-06-29T14:39:43.231 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:38:22.981044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:38:23.565073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:39:43.231 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:39:43.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.355+0000 7f79155a5640 1 -- 172.21.15.59:0/2666780151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79100d23c0 msgr2=0x7f79100b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.355+0000 7f79155a5640 1 --2- 172.21.15.59:0/2666780151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79100d23c0 0x7f79100b3990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7900009830 tx=0x7f790002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.355+0000 7f79155a5640 1 -- 172.21.15.59:0/2666780151 shutdown_connections 2024-06-29T14:39:43.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.355+0000 7f79155a5640 1 --2- 172.21.15.59:0/2666780151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79100d23c0 0x7f79100b3990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.355+0000 7f79155a5640 1 -- 172.21.15.59:0/2666780151 >> 172.21.15.59:0/2666780151 conn(0x7f79100d3ee0 msgr2=0x7f79100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.355+0000 7f79155a5640 1 -- 172.21.15.59:0/2666780151 shutdown_connections 2024-06-29T14:39:43.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.355+0000 7f79155a5640 1 -- 172.21.15.59:0/2666780151 wait complete. 2024-06-29T14:39:43.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.357+0000 7f79155a5640 1 Processor -- start 2024-06-29T14:39:43.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.357+0000 7f79155a5640 1 -- start start 2024-06-29T14:39:43.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f79155a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79100b3ed0 0x7f7910153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f79155a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79100d23c0 0x7f791015cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f79155a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f791015efe0 0x7f791015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f79155a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7910160450 con 0x7f79100d23c0 2024-06-29T14:39:43.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f79155a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79101602d0 con 0x7f79100b3ed0 2024-06-29T14:39:43.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f79155a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f79101605d0 con 0x7f791015efe0 2024-06-29T14:39:43.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79100d23c0 0x7f791015cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79100d23c0 0x7f791015cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43994/0 (socket says 172.21.15.59:43994) 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 -- 172.21.15.59:0/2046247835 learned_addr learned my addr 172.21.15.59:0/2046247835 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790effd640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79100b3ed0 0x7f7910153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790f7fe640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f791015efe0 0x7f791015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 -- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f791015efe0 msgr2=0x7f791015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f791015efe0 0x7f791015d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 -- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79100b3ed0 msgr2=0x7f7910153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79100b3ed0 0x7f7910153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 -- 172.21.15.59:0/2046247835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7910159050 con 0x7f79100d23c0 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.359+0000 7f790e7fc640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79100d23c0 0x7f791015cc70 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f7904001eb0 tx=0x7f790400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.360+0000 7f78f77fe640 1 -- 172.21.15.59:0/2046247835 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904011790 con 0x7f79100d23c0 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.360+0000 7f78f77fe640 1 -- 172.21.15.59:0/2046247835 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7904017070 con 0x7f79100d23c0 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.360+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79101592f0 con 0x7f79100d23c0 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.360+0000 7f78f77fe640 1 -- 172.21.15.59:0/2046247835 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904011ca0 con 0x7f79100d23c0 2024-06-29T14:39:43.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.360+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79101598b0 con 0x7f79100d23c0 2024-06-29T14:39:43.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.361+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79100b3990 con 0x7f79100d23c0 2024-06-29T14:39:43.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.362+0000 7f78f77fe640 1 -- 172.21.15.59:0/2046247835 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f790401c460 con 0x7f79100d23c0 2024-06-29T14:39:43.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.362+0000 7f78f77fe640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f78fc076540 0x7f78fc078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.362+0000 7f78f77fe640 1 -- 172.21.15.59:0/2046247835 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7f7904097bf0 con 0x7f79100d23c0 2024-06-29T14:39:43.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.362+0000 7f790effd640 1 -- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f78fc076540 msgr2=0x7f78fc078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:43.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.362+0000 7f790effd640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f78fc076540 0x7f78fc078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:43.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.367+0000 7f78f77fe640 1 -- 172.21.15.59:0/2046247835 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f79100b42e0 con 0x7f79100d23c0 2024-06-29T14:39:43.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.560+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79100d2950 con 0x7f79100d23c0 2024-06-29T14:39:43.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.560+0000 7f78f77fe640 1 -- 172.21.15.59:0/2046247835 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) ==== 93+0+10 (secure 0 0 0) 0x7f7904060800 con 0x7f79100d23c0 2024-06-29T14:39:43.559 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:39:43.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.562+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f78fc076540 msgr2=0x7f78fc078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:43.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.562+0000 7f79155a5640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f78fc076540 0x7f78fc078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.562+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79100d23c0 msgr2=0x7f791015cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.562+0000 7f79155a5640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79100d23c0 0x7f791015cc70 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f7904001eb0 tx=0x7f790400c890 comp rx=0 tx=0).stop 2024-06-29T14:39:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.563+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 shutdown_connections 2024-06-29T14:39:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.563+0000 7f79155a5640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7f78fc076540 0x7f78fc078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.563+0000 7f79155a5640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f791015efe0 0x7f791015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.563+0000 7f79155a5640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79100d23c0 0x7f791015cc70 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.563+0000 7f79155a5640 1 --2- 172.21.15.59:0/2046247835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79100b3ed0 0x7f7910153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.563+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 >> 172.21.15.59:0/2046247835 conn(0x7f79100d3ee0 msgr2=0x7f791006cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.563+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 shutdown_connections 2024-06-29T14:39:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.563+0000 7f79155a5640 1 -- 172.21.15.59:0/2046247835 wait complete. 2024-06-29T14:39:43.572 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:39:43.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.695+0000 7fb177e78640 1 -- 172.21.15.59:0/1286366022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700d1c30 msgr2=0x7fb170166a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.695+0000 7fb177e78640 1 --2- 172.21.15.59:0/1286366022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700d1c30 0x7fb170166a20 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb168005fd0 tx=0x7fb16802a840 comp rx=0 tx=0).stop 2024-06-29T14:39:43.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.695+0000 7fb177e78640 1 -- 172.21.15.59:0/1286366022 shutdown_connections 2024-06-29T14:39:43.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.695+0000 7fb177e78640 1 --2- 172.21.15.59:0/1286366022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700d1c30 0x7fb170166a20 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.695+0000 7fb177e78640 1 -- 172.21.15.59:0/1286366022 >> 172.21.15.59:0/1286366022 conn(0x7fb170073e10 msgr2=0x7fb170074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.696+0000 7fb177e78640 1 -- 172.21.15.59:0/1286366022 shutdown_connections 2024-06-29T14:39:43.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.696+0000 7fb177e78640 1 -- 172.21.15.59:0/1286366022 wait complete. 2024-06-29T14:39:43.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.697+0000 7fb177e78640 1 Processor -- start 2024-06-29T14:39:43.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.697+0000 7fb177e78640 1 -- start start 2024-06-29T14:39:43.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb177e78640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1700b2270 0x7fb17014db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb177e78640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700b2fa0 0x7fb17015eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb177e78640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700d1c30 0x7fb17015f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb177e78640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb1701670e0 con 0x7fb1700b2fa0 2024-06-29T14:39:43.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb177e78640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb170166f60 con 0x7fb1700d1c30 2024-06-29T14:39:43.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb177e78640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb170167260 con 0x7fb1700b2270 2024-06-29T14:39:43.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700b2fa0 0x7fb17015eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700b2fa0 0x7fb17015eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44014/0 (socket says 172.21.15.59:44014) 2024-06-29T14:39:43.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 -- 172.21.15.59:0/482788580 learned_addr learned my addr 172.21.15.59:0/482788580 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:43.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175c14640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1700b2270 0x7fb17014db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb176415640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700d1c30 0x7fb17015f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 -- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1700b2270 msgr2=0x7fb17014db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1700b2270 0x7fb17014db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 -- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700d1c30 msgr2=0x7fb17015f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700d1c30 0x7fb17015f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 -- 172.21.15.59:0/482788580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17014e080 con 0x7fb1700b2fa0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.699+0000 7fb175413640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700b2fa0 0x7fb17015eb00 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fb16400c420 tx=0x7fb16400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.700+0000 7fb1627fc640 1 -- 172.21.15.59:0/482788580 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164017070 con 0x7fb1700b2fa0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.700+0000 7fb1627fc640 1 -- 172.21.15.59:0/482788580 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1640035f0 con 0x7fb1700b2fa0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.700+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17015d730 con 0x7fb1700b2fa0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.700+0000 7fb1627fc640 1 -- 172.21.15.59:0/482788580 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164003a40 con 0x7fb1700b2fa0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.700+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb17015dcf0 con 0x7fb1700b2fa0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.701+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb17014ee80 con 0x7fb1700b2fa0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.701+0000 7fb1627fc640 1 -- 172.21.15.59:0/482788580 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7fb16401a070 con 0x7fb1700b2fa0 2024-06-29T14:39:43.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.702+0000 7fb1627fc640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fb154076540 0x7fb154078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.702+0000 7fb175c14640 1 -- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fb154076540 msgr2=0x7fb154078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/716775118 2024-06-29T14:39:43.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.702+0000 7fb1627fc640 1 -- 172.21.15.59:0/482788580 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6561+0+0 (secure 0 0 0) 0x7fb1640971d0 con 0x7fb1700b2fa0 2024-06-29T14:39:43.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.702+0000 7fb175c14640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fb154076540 0x7fb154078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:43.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.707+0000 7fb1627fc640 1 -- 172.21.15.59:0/482788580 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb17014ee80 con 0x7fb1700b2fa0 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.897+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb17015bc20 con 0x7fb1700b2fa0 2024-06-29T14:39:43.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.898+0000 7fb1627fc640 1 -- 172.21.15.59:0/482788580 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) ==== 104+0+11 (secure 0 0 0) 0x7fb16405fd90 con 0x7fb1700b2fa0 2024-06-29T14:39:43.896 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:39:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fb154076540 msgr2=0x7fb154078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fb154076540 0x7fb154078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700b2fa0 msgr2=0x7fb17015eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700b2fa0 0x7fb17015eb00 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fb16400c420 tx=0x7fb16400c8e0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 shutdown_connections 2024-06-29T14:39:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:6832/716775118,v1:172.21.15.153:6833/716775118] conn(0x7fb154076540 0x7fb154078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1700d1c30 0x7fb17015f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1700b2fa0 0x7fb17015eb00 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 --2- 172.21.15.59:0/482788580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1700b2270 0x7fb17014db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 >> 172.21.15.59:0/482788580 conn(0x7fb170073e10 msgr2=0x7fb1701579d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 shutdown_connections 2024-06-29T14:39:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:43.900+0000 7fb177e78640 1 -- 172.21.15.59:0/482788580 wait complete. 2024-06-29T14:39:43.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.034+0000 7fc64a11b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.034+0000 7fc64a11b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.034+0000 7fc64a11b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.035+0000 7fc64a11b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.035+0000 7fc64a11b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.035+0000 7fc647115640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.035+0000 7fc647115640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.035+0000 7fc647115640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.035+0000 7fc647115640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.033 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:39:44.035+0000 7fc647115640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:44.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.039+0000 7f8255a7f640 1 -- 172.21.15.59:0/3173585288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 msgr2=0x7f825015b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.039+0000 7f8255a7f640 1 --2- 172.21.15.59:0/3173585288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 0x7f825015b660 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8244009860 tx=0x7f824402a5a0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.039+0000 7f8255a7f640 1 -- 172.21.15.59:0/3173585288 shutdown_connections 2024-06-29T14:39:44.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.039+0000 7f8255a7f640 1 --2- 172.21.15.59:0/3173585288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 0x7f825015b660 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.039+0000 7f8255a7f640 1 -- 172.21.15.59:0/3173585288 >> 172.21.15.59:0/3173585288 conn(0x7f8250072d80 msgr2=0x7f8250073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.039+0000 7f8255a7f640 1 -- 172.21.15.59:0/3173585288 shutdown_connections 2024-06-29T14:39:44.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.039+0000 7f8255a7f640 1 -- 172.21.15.59:0/3173585288 wait complete. 2024-06-29T14:39:44.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.041+0000 7f8255a7f640 1 Processor -- start 2024-06-29T14:39:44.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.042+0000 7f8255a7f640 1 -- start start 2024-06-29T14:39:44.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f8255a7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8250140350 0x7f82501f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f8255a7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 0x7f825013c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f8255a7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825015bba0 0x7f8250154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f8255a7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f825015c130 con 0x7f8250140350 2024-06-29T14:39:44.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f8255a7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f825015bfb0 con 0x7f825015bba0 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f8255a7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f82500cffe0 con 0x7f825015b250 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8250140350 0x7f82501f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8250140350 0x7f82501f69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44020/0 (socket says 172.21.15.59:44020) 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824f7fe640 1 -- 172.21.15.59:0/4263599979 learned_addr learned my addr 172.21.15.59:0/4263599979 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824effd640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 0x7f825013c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824ffff640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825015bba0 0x7f8250154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824f7fe640 1 -- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 msgr2=0x7f825013c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824f7fe640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 0x7f825013c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824f7fe640 1 -- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825015bba0 msgr2=0x7f8250154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824f7fe640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825015bba0 0x7f8250154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824f7fe640 1 -- 172.21.15.59:0/4263599979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8250142cd0 con 0x7f8250140350 2024-06-29T14:39:44.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824effd640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 0x7f825013c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:39:44.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.043+0000 7f824ffff640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825015bba0 0x7f8250154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:39:44.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.044+0000 7f824f7fe640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8250140350 0x7f82501f69f0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f824000c460 tx=0x7f824000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:44.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.044+0000 7f8237fff640 1 -- 172.21.15.59:0/4263599979 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82400052d0 con 0x7f8250140350 2024-06-29T14:39:44.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.044+0000 7f8237fff640 1 -- 172.21.15.59:0/4263599979 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8240005490 con 0x7f8250140350 2024-06-29T14:39:44.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.044+0000 7f8237fff640 1 -- 172.21.15.59:0/4263599979 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240012870 con 0x7f8250140350 2024-06-29T14:39:44.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.044+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82501e58f0 con 0x7f8250140350 2024-06-29T14:39:44.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.044+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8250143250 con 0x7f8250140350 2024-06-29T14:39:44.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.045+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f825015b660 con 0x7f8250140350 2024-06-29T14:39:44.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.046+0000 7f8237fff640 1 -- 172.21.15.59:0/4263599979 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 54) ==== 97912+0+0 (secure 0 0 0) 0x7f8240012a10 con 0x7f8250140350 2024-06-29T14:39:44.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.046+0000 7f8237fff640 1 -- 172.21.15.59:0/4263599979 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6991+0+0 (secure 0 0 0) 0x7f82400970a0 con 0x7f8250140350 2024-06-29T14:39:44.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.051+0000 7f8237fff640 1 -- 172.21.15.59:0/4263599979 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8240060300 con 0x7f8250140350 2024-06-29T14:39:44.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.242+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f825006db80 con 0x7f8250140350 2024-06-29T14:39:44.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.243+0000 7f8237fff640 1 -- 172.21.15.59:0/4263599979 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) ==== 100+0+11 (secure 0 0 0) 0x7f8240064600 con 0x7f8250140350 2024-06-29T14:39:44.241 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:39:44.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8250140350 msgr2=0x7f82501f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8250140350 0x7f82501f69f0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f824000c460 tx=0x7f824000c920 comp rx=0 tx=0).stop 2024-06-29T14:39:44.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 shutdown_connections 2024-06-29T14:39:44.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825015bba0 0x7f8250154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825015b250 0x7f825013c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 --2- 172.21.15.59:0/4263599979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8250140350 0x7f82501f69f0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 >> 172.21.15.59:0/4263599979 conn(0x7f8250072d80 msgr2=0x7f8250068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 shutdown_connections 2024-06-29T14:39:44.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.245+0000 7f8255a7f640 1 -- 172.21.15.59:0/4263599979 wait complete. 2024-06-29T14:39:44.255 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:44.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.378+0000 7f11c6295640 1 -- 172.21.15.59:0/1799822506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c015c5a0 msgr2=0x7f11c015c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.378+0000 7f11c6295640 1 --2- 172.21.15.59:0/1799822506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c015c5a0 0x7f11c015c9b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009be0 tx=0x7f11b802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.378+0000 7f11c6295640 1 -- 172.21.15.59:0/1799822506 shutdown_connections 2024-06-29T14:39:44.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.378+0000 7f11c6295640 1 --2- 172.21.15.59:0/1799822506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c015c5a0 0x7f11c015c9b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.378+0000 7f11c6295640 1 -- 172.21.15.59:0/1799822506 >> 172.21.15.59:0/1799822506 conn(0x7f11c0073e50 msgr2=0x7f11c00b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.378+0000 7f11c6295640 1 -- 172.21.15.59:0/1799822506 shutdown_connections 2024-06-29T14:39:44.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.378+0000 7f11c6295640 1 -- 172.21.15.59:0/1799822506 wait complete. 2024-06-29T14:39:44.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.380+0000 7f11c6295640 1 Processor -- start 2024-06-29T14:39:44.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.380+0000 7f11c6295640 1 -- start start 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c6295640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c00b5320 0x7f11c015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c6295640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c015c5a0 0x7f11c00cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c6295640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c015d580 0x7f11c01f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c6295640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f11c006e860 con 0x7f11c015d580 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c6295640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11c006e6e0 con 0x7f11c015c5a0 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c6295640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11c006e9e0 con 0x7f11c00b5320 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c4832640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c015d580 0x7f11c01f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c4832640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c015d580 0x7f11c01f2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44024/0 (socket says 172.21.15.59:44024) 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.381+0000 7f11c4832640 1 -- 172.21.15.59:0/2589698410 learned_addr learned my addr 172.21.15.59:0/2589698410 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11bffff640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c00b5320 0x7f11c015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11bf7fe640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c015c5a0 0x7f11c00cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11c4832640 1 -- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c00b5320 msgr2=0x7f11c015b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11c4832640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c00b5320 0x7f11c015b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11c4832640 1 -- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c015c5a0 msgr2=0x7f11c00cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11c4832640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c015c5a0 0x7f11c00cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11c4832640 1 -- 172.21.15.59:0/2589698410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c015bce0 con 0x7f11c015d580 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11bffff640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c00b5320 0x7f11c015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11bf7fe640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c015c5a0 0x7f11c00cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11c4832640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c015d580 0x7f11c01f2ab0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f11b8035040 tx=0x7f11b8003e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11bcff9640 1 -- 172.21.15.59:0/2589698410 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8042020 con 0x7f11c015d580 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c00ca910 con 0x7f11c015d580 2024-06-29T14:39:44.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11bcff9640 1 -- 172.21.15.59:0/2589698410 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11b8033b00 con 0x7f11c015d580 2024-06-29T14:39:44.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11bcff9640 1 -- 172.21.15.59:0/2589698410 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8037040 con 0x7f11c015d580 2024-06-29T14:39:44.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.382+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c00caed0 con 0x7f11c015d580 2024-06-29T14:39:44.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.383+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c00cc8f0 con 0x7f11c015d580 2024-06-29T14:39:44.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.384+0000 7f11bcff9640 1 -- 172.21.15.59:0/2589698410 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 54) ==== 97912+0+0 (secure 0 0 0) 0x7f11b8045020 con 0x7f11c015d580 2024-06-29T14:39:44.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.384+0000 7f11bcff9640 1 -- 172.21.15.59:0/2589698410 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6991+0+0 (secure 0 0 0) 0x7f11b80b7fa0 con 0x7f11c015d580 2024-06-29T14:39:44.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.389+0000 7f11bcff9640 1 -- 172.21.15.59:0/2589698410 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11b804c070 con 0x7f11c015d580 2024-06-29T14:39:44.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.617+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11c00cce50 con 0x7f11c015d580 2024-06-29T14:39:44.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.617+0000 7f11bcff9640 1 -- 172.21.15.59:0/2589698410 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5225 (secure 0 0 0) 0x7f11c015bce0 con 0x7f11c015d580 2024-06-29T14:39:44.616 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:44.616 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"btime":"2024-06-29T14:38:28:267220+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3208152037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3208152037},{"type":"v1","addr":"172.21.15.153:6836","nonce":3208152037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6716,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/590918917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":590918917},{"type":"v1","addr":"172.21.15.153:6837","nonce":590918917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/546301975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":546301975},{"type":"v1","addr":"172.21.15.59:6833","nonce":546301975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.621714+0000","modified":"2024-06-29T14:38:27.659142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/2011834805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2011834805},{"type":"v1","addr":"172.21.15.59:6835","nonce":2011834805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":6}]} 2024-06-29T14:39:44.616 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 55 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.619+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c015d580 msgr2=0x7f11c01f2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.619+0000 7f11c6295640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c015d580 0x7f11c01f2ab0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f11b8035040 tx=0x7f11b8003e70 comp rx=0 tx=0).stop 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.619+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 shutdown_connections 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.619+0000 7f11c6295640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c015d580 0x7f11c01f2ab0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.619+0000 7f11c6295640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c015c5a0 0x7f11c00cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.619+0000 7f11c6295640 1 --2- 172.21.15.59:0/2589698410 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c00b5320 0x7f11c015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.620+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 >> 172.21.15.59:0/2589698410 conn(0x7f11c0073e50 msgr2=0x7f11c0141680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.620+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 shutdown_connections 2024-06-29T14:39:44.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.620+0000 7f11c6295640 1 -- 172.21.15.59:0/2589698410 wait complete. 2024-06-29T14:39:44.629 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:44.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.748+0000 7f5db16fd640 1 -- 172.21.15.59:0/3001233276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b8be0 msgr2=0x7f5dac1629d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.748+0000 7f5db16fd640 1 --2- 172.21.15.59:0/3001233276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b8be0 0x7f5dac1629d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0b78c0 tx=0x7f5da402a4a0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.748+0000 7f5db16fd640 1 -- 172.21.15.59:0/3001233276 shutdown_connections 2024-06-29T14:39:44.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.748+0000 7f5db16fd640 1 --2- 172.21.15.59:0/3001233276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b8be0 0x7f5dac1629d0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.748+0000 7f5db16fd640 1 -- 172.21.15.59:0/3001233276 >> 172.21.15.59:0/3001233276 conn(0x7f5dac0b5610 msgr2=0x7f5dac0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.748+0000 7f5db16fd640 1 -- 172.21.15.59:0/3001233276 shutdown_connections 2024-06-29T14:39:44.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.748+0000 7f5db16fd640 1 -- 172.21.15.59:0/3001233276 wait complete. 2024-06-29T14:39:44.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.750+0000 7f5db16fd640 1 Processor -- start 2024-06-29T14:39:44.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.750+0000 7f5db16fd640 1 -- start start 2024-06-29T14:39:44.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5db16fd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac08c190 0x7f5dac1514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5db16fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dac0b8be0 0x7f5dac14edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5db16fd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dac0ce950 0x7f5dac153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5db16fd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5dac169690 con 0x7f5dac08c190 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5db16fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5dac169510 con 0x7f5dac0b8be0 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5db16fd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5dac169810 con 0x7f5dac0ce950 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5daaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac08c190 0x7f5dac1514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5daaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac08c190 0x7f5dac1514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44040/0 (socket says 172.21.15.59:44040) 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5daaffd640 1 -- 172.21.15.59:0/1967010365 learned_addr learned my addr 172.21.15.59:0/1967010365 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.751+0000 7f5dab7fe640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dac0ce950 0x7f5dac153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5daa7fc640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dac0b8be0 0x7f5dac14edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5daaffd640 1 -- 172.21.15.59:0/1967010365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dac0ce950 msgr2=0x7f5dac153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5daaffd640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dac0ce950 0x7f5dac153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5daaffd640 1 -- 172.21.15.59:0/1967010365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dac0b8be0 msgr2=0x7f5dac14edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5daaffd640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dac0b8be0 0x7f5dac14edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5daaffd640 1 -- 172.21.15.59:0/1967010365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dac151a10 con 0x7f5dac08c190 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5daaffd640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac08c190 0x7f5dac1514d0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00cbf0 tx=0x7f5d9c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5da37fe640 1 -- 172.21.15.59:0/1967010365 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c016020 con 0x7f5dac08c190 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5da37fe640 1 -- 172.21.15.59:0/1967010365 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d9c00ad90 con 0x7f5dac08c190 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dac14f2f0 con 0x7f5dac08c190 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5da37fe640 1 -- 172.21.15.59:0/1967010365 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c012680 con 0x7f5dac08c190 2024-06-29T14:39:44.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.752+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dac159ee0 con 0x7f5dac08c190 2024-06-29T14:39:44.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.753+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dac0cf080 con 0x7f5dac08c190 2024-06-29T14:39:44.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.754+0000 7f5da37fe640 1 -- 172.21.15.59:0/1967010365 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 54) ==== 97912+0+0 (secure 0 0 0) 0x7f5d9c019020 con 0x7f5dac08c190 2024-06-29T14:39:44.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.754+0000 7f5da37fe640 1 -- 172.21.15.59:0/1967010365 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6991+0+0 (secure 0 0 0) 0x7f5d9c097e80 con 0x7f5dac08c190 2024-06-29T14:39:44.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.758+0000 7f5da37fe640 1 -- 172.21.15.59:0/1967010365 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d9c020020 con 0x7f5dac08c190 2024-06-29T14:39:44.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.953+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5dac15a0b0 con 0x7f5dac08c190 2024-06-29T14:39:44.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.954+0000 7f5da37fe640 1 -- 172.21.15.59:0/1967010365 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+18240 (secure 0 0 0) 0x7f5d9c061730 con 0x7f5dac08c190 2024-06-29T14:39:44.953 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:44.953 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":118,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:43.971617+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.981044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.565073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.a","mappings":[{"from":1,"to":2}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1864308943":"2024-06-30T14:39:43.971541+0000","172.21.15.153:6833/716775118":"2024-06-30T14:39:43.971541+0000","172.21.15.153:6832/716775118":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3495517559":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3803261247":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3992874705":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/2786718243":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/493501963":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/1628521659":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/2502091213":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3749556168":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3059657409":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/438874239":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3367163578":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6832/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/441578565":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6833/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/578976042":"2024-06-30T14:38:36.889100+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac08c190 msgr2=0x7f5dac1514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac08c190 0x7f5dac1514d0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00cbf0 tx=0x7f5d9c00a060 comp rx=0 tx=0).stop 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 shutdown_connections 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dac0ce950 0x7f5dac153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dac0b8be0 0x7f5dac14edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 --2- 172.21.15.59:0/1967010365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac08c190 0x7f5dac1514d0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 >> 172.21.15.59:0/1967010365 conn(0x7f5dac0b5610 msgr2=0x7f5dac0b7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 shutdown_connections 2024-06-29T14:39:44.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:44.956+0000 7f5db16fd640 1 -- 172.21.15.59:0/1967010365 wait complete. 2024-06-29T14:39:44.966 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:39:44.966 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:45.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.088+0000 7f9285c40640 1 -- 172.21.15.59:0/4019128133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92800b40a0 msgr2=0x7f928015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.088+0000 7f9285c40640 1 --2- 172.21.15.59:0/4019128133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92800b40a0 0x7f928015e8d0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f9274009830 tx=0x7f927402a510 comp rx=0 tx=0).stop 2024-06-29T14:39:45.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.088+0000 7f9285c40640 1 -- 172.21.15.59:0/4019128133 shutdown_connections 2024-06-29T14:39:45.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.088+0000 7f9285c40640 1 --2- 172.21.15.59:0/4019128133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92800b40a0 0x7f928015e8d0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.088+0000 7f9285c40640 1 -- 172.21.15.59:0/4019128133 >> 172.21.15.59:0/4019128133 conn(0x7f92800d3ee0 msgr2=0x7f92800d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.088+0000 7f9285c40640 1 -- 172.21.15.59:0/4019128133 shutdown_connections 2024-06-29T14:39:45.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.088+0000 7f9285c40640 1 -- 172.21.15.59:0/4019128133 wait complete. 2024-06-29T14:39:45.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.090+0000 7f9285c40640 1 Processor -- start 2024-06-29T14:39:45.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.090+0000 7f9285c40640 1 -- start start 2024-06-29T14:39:45.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f9285c40640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92800d23c0 0x7f928014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f9285c40640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f928015ee10 0x7f928014a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f9285c40640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92801f87c0 0x7f92801f8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f9285c40640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9280160450 con 0x7f928015ee10 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f9285c40640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92801602d0 con 0x7f92800d23c0 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f9285c40640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f92801605d0 con 0x7f92801f87c0 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f927effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f928015ee10 0x7f928014a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f927effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f928015ee10 0x7f928014a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44064/0 (socket says 172.21.15.59:44064) 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.091+0000 7f927effd640 1 -- 172.21.15.59:0/2889762853 learned_addr learned my addr 172.21.15.59:0/2889762853 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927f7fe640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92800d23c0 0x7f928014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927ffff640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92801f87c0 0x7f92801f8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927effd640 1 -- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92801f87c0 msgr2=0x7f92801f8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927effd640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92801f87c0 0x7f92801f8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927effd640 1 -- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92800d23c0 msgr2=0x7f928014f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927effd640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92800d23c0 0x7f928014f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927effd640 1 -- 172.21.15.59:0/2889762853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9280154f90 con 0x7f928015ee10 2024-06-29T14:39:45.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927ffff640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92801f87c0 0x7f92801f8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927f7fe640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92800d23c0 0x7f928014f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f927effd640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f928015ee10 0x7f928014a7a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f927402ab00 tx=0x7f9274033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f9267fff640 1 -- 172.21.15.59:0/2889762853 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9274005c90 con 0x7f928015ee10 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f9267fff640 1 -- 172.21.15.59:0/2889762853 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9274031e50 con 0x7f928015ee10 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f928014fc60 con 0x7f928015ee10 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f9267fff640 1 -- 172.21.15.59:0/2889762853 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9274005260 con 0x7f928015ee10 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.092+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9280150220 con 0x7f928015ee10 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.093+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f928015e8d0 con 0x7f928015ee10 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.094+0000 7f9267fff640 1 -- 172.21.15.59:0/2889762853 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 55) ==== 98039+0+0 (secure 0 0 0) 0x7f9274005400 con 0x7f928015ee10 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.095+0000 7f9267fff640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f926c076120 0x7f926c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.095+0000 7f9267fff640 1 -- 172.21.15.59:0/2889762853 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6991+0+0 (secure 0 0 0) 0x7f92740b8a50 con 0x7f928015ee10 2024-06-29T14:39:45.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.095+0000 7f927f7fe640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f926c076120 0x7f926c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.095+0000 7f927f7fe640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f926c076120 0x7f926c078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9270007a50 tx=0x7f9270009040 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.099+0000 7f9267fff640 1 -- 172.21.15.59:0/2889762853 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f927403f030 con 0x7f928015ee10 2024-06-29T14:39:45.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.332+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9280158510 con 0x7f928015ee10 2024-06-29T14:39:45.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.333+0000 7f9267fff640 1 -- 172.21.15.59:0/2889762853 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5225 (secure 0 0 0) 0x7f92740820a0 con 0x7f928015ee10 2024-06-29T14:39:45.331 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:45.331 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"btime":"2024-06-29T14:38:28:267220+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3208152037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3208152037},{"type":"v1","addr":"172.21.15.153:6836","nonce":3208152037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6716,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/590918917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":590918917},{"type":"v1","addr":"172.21.15.153:6837","nonce":590918917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/546301975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":546301975},{"type":"v1","addr":"172.21.15.59:6833","nonce":546301975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.621714+0000","modified":"2024-06-29T14:38:27.659142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/2011834805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2011834805},{"type":"v1","addr":"172.21.15.59:6835","nonce":2011834805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":6}]} 2024-06-29T14:39:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 55 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f926c076120 msgr2=0x7f926c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f926c076120 0x7f926c078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9270007a50 tx=0x7f9270009040 comp rx=0 tx=0).stop 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f928015ee10 msgr2=0x7f928014a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f928015ee10 0x7f928014a7a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f927402ab00 tx=0x7f9274033c20 comp rx=0 tx=0).stop 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 shutdown_connections 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92801f87c0 0x7f92801f8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f926c076120 0x7f926c078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f928015ee10 0x7f928014a7a0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 --2- 172.21.15.59:0/2889762853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92800d23c0 0x7f928014f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 >> 172.21.15.59:0/2889762853 conn(0x7f92800d3ee0 msgr2=0x7f9280151150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 shutdown_connections 2024-06-29T14:39:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.335+0000 7f9285c40640 1 -- 172.21.15.59:0/2889762853 wait complete. 2024-06-29T14:39:45.344 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.464+0000 7f87231ac640 1 -- 172.21.15.59:0/3453991482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8708004040 msgr2=0x7f8708004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.464+0000 7f87231ac640 1 --2- 172.21.15.59:0/3453991482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8708004040 0x7f8708004800 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f8704009ed0 tx=0x7f8708005830 comp rx=0 tx=0).stop 2024-06-29T14:39:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.464+0000 7f8721749640 1 -- 172.21.15.59:0/3453991482 reap_dead start 2024-06-29T14:39:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.464+0000 7f87231ac640 1 -- 172.21.15.59:0/3453991482 shutdown_connections 2024-06-29T14:39:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.464+0000 7f87231ac640 1 -- 172.21.15.59:0/3453991482 >> 172.21.15.59:0/3453991482 conn(0x7f871c0b5610 msgr2=0x7f871c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.464+0000 7f87231ac640 1 -- 172.21.15.59:0/3453991482 shutdown_connections 2024-06-29T14:39:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.464+0000 7f87231ac640 1 -- 172.21.15.59:0/3453991482 wait complete. 2024-06-29T14:39:45.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.466+0000 7f87231ac640 1 Processor -- start 2024-06-29T14:39:45.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.466+0000 7f87231ac640 1 -- start start 2024-06-29T14:39:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87231ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8708004040 0x7f871c1f5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87231ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871c1f5af0 0x7f871c1f8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87231ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1f8ba0 0x7f871c1f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87231ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f871c0cf410 con 0x7f871c1f8ba0 2024-06-29T14:39:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87231ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f871c0cf290 con 0x7f8708004040 2024-06-29T14:39:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87231ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f871c0b7bd0 con 0x7f871c1f5af0 2024-06-29T14:39:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1f8ba0 0x7f871c1f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1f8ba0 0x7f871c1f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44088/0 (socket says 172.21.15.59:44088) 2024-06-29T14:39:45.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 -- 172.21.15.59:0/2800000347 learned_addr learned my addr 172.21.15.59:0/2800000347 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:45.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f871bfff640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871c1f5af0 0x7f871c1f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8720f48640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8708004040 0x7f871c1f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 -- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871c1f5af0 msgr2=0x7f871c1f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871c1f5af0 0x7f871c1f8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 -- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8708004040 msgr2=0x7f871c1f5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8708004040 0x7f871c1f5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 -- 172.21.15.59:0/2800000347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c156d00 con 0x7f871c1f8ba0 2024-06-29T14:39:45.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f8721749640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1f8ba0 0x7f871c1f8fb0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f871c0b7620 tx=0x7f87140039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87197fa640 1 -- 172.21.15.59:0/2800000347 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714011840 con 0x7f871c1f8ba0 2024-06-29T14:39:45.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87197fa640 1 -- 172.21.15.59:0/2800000347 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8714017070 con 0x7f871c1f8ba0 2024-06-29T14:39:45.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87197fa640 1 -- 172.21.15.59:0/2800000347 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714011c00 con 0x7f871c1f8ba0 2024-06-29T14:39:45.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871c155c60 con 0x7f871c1f8ba0 2024-06-29T14:39:45.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.468+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f871c156220 con 0x7f871c1f8ba0 2024-06-29T14:39:45.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.470+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871c068ba0 con 0x7f871c1f8ba0 2024-06-29T14:39:45.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.470+0000 7f87197fa640 1 -- 172.21.15.59:0/2800000347 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 55) ==== 98039+0+0 (secure 0 0 0) 0x7f871401c460 con 0x7f871c1f8ba0 2024-06-29T14:39:45.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.471+0000 7f87197fa640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f86fc076120 0x7f86fc078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.471+0000 7f87197fa640 1 -- 172.21.15.59:0/2800000347 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6991+0+0 (secure 0 0 0) 0x7f8714097e90 con 0x7f871c1f8ba0 2024-06-29T14:39:45.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.471+0000 7f8720f48640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f86fc076120 0x7f86fc078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.471+0000 7f8720f48640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f86fc076120 0x7f86fc078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f870c009b30 tx=0x7f870c005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.475+0000 7f87197fa640 1 -- 172.21.15.59:0/2800000347 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f871c068ba0 con 0x7f871c1f8ba0 2024-06-29T14:39:45.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.703+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f871c0d1a20 con 0x7f871c1f8ba0 2024-06-29T14:39:45.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.703+0000 7f87197fa640 1 -- 172.21.15.59:0/2800000347 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5225 (secure 0 0 0) 0x7f8714061630 con 0x7f871c1f8ba0 2024-06-29T14:39:45.702 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:45.702 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"btime":"2024-06-29T14:38:28:267220+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3208152037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3208152037},{"type":"v1","addr":"172.21.15.153:6836","nonce":3208152037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6716,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/590918917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":590918917},{"type":"v1","addr":"172.21.15.153:6837","nonce":590918917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/546301975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":546301975},{"type":"v1","addr":"172.21.15.59:6833","nonce":546301975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.621714+0000","modified":"2024-06-29T14:38:27.659142+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/2011834805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2011834805},{"type":"v1","addr":"172.21.15.59:6835","nonce":2011834805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":6}]} 2024-06-29T14:39:45.702 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 55 2024-06-29T14:39:45.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f86fc076120 msgr2=0x7f86fc078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f86fc076120 0x7f86fc078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f870c009b30 tx=0x7f870c005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:45.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1f8ba0 msgr2=0x7f871c1f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1f8ba0 0x7f871c1f8fb0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f871c0b7620 tx=0x7f87140039c0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 shutdown_connections 2024-06-29T14:39:45.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f86fc076120 0x7f86fc078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c1f8ba0 0x7f871c1f8fb0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871c1f5af0 0x7f871c1f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 --2- 172.21.15.59:0/2800000347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8708004040 0x7f871c1f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 >> 172.21.15.59:0/2800000347 conn(0x7f871c0b5610 msgr2=0x7f871c160500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 shutdown_connections 2024-06-29T14:39:45.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.705+0000 7f87231ac640 1 -- 172.21.15.59:0/2800000347 wait complete. 2024-06-29T14:39:45.715 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:45.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.834+0000 7fa3579bd640 1 -- 172.21.15.59:0/419747018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa35006e840 msgr2=0x7fa3500cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.834+0000 7fa3579bd640 1 --2- 172.21.15.59:0/419747018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa35006e840 0x7fa3500cf1d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa340005fd0 tx=0x7fa34002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.834+0000 7fa3579bd640 1 -- 172.21.15.59:0/419747018 shutdown_connections 2024-06-29T14:39:45.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.834+0000 7fa3579bd640 1 --2- 172.21.15.59:0/419747018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa35006e840 0x7fa3500cf1d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.834+0000 7fa3579bd640 1 --2- 172.21.15.59:0/419747018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3500b2210 0x7fa35006e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.834+0000 7fa3579bd640 1 -- 172.21.15.59:0/419747018 >> 172.21.15.59:0/419747018 conn(0x7fa3500b06e0 msgr2=0x7fa3500b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.834+0000 7fa3579bd640 1 -- 172.21.15.59:0/419747018 shutdown_connections 2024-06-29T14:39:45.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.834+0000 7fa3579bd640 1 -- 172.21.15.59:0/419747018 wait complete. 2024-06-29T14:39:45.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.836+0000 7fa3579bd640 1 Processor -- start 2024-06-29T14:39:45.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.836+0000 7fa3579bd640 1 -- start start 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.837+0000 7fa3579bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35006e840 0x7fa35013f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.837+0000 7fa3579bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa350144470 0x7fa35006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.837+0000 7fa3579bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa35015a3a0 0x7fa35006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.837+0000 7fa3579bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa3500cfaa0 con 0x7fa35006e840 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.837+0000 7fa3579bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa3500cf920 con 0x7fa35015a3a0 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.837+0000 7fa3579bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa3500cfc20 con 0x7fa350144470 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.837+0000 7fa355759640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35006e840 0x7fa35013f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355759640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35006e840 0x7fa35013f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44098/0 (socket says 172.21.15.59:44098) 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355759640 1 -- 172.21.15.59:0/2418757216 learned_addr learned my addr 172.21.15.59:0/2418757216 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355f5a640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa35015a3a0 0x7fa35006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa354f58640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa350144470 0x7fa35006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355759640 1 -- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa350144470 msgr2=0x7fa35006ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355759640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa350144470 0x7fa35006ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355759640 1 -- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa35015a3a0 msgr2=0x7fa35006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355759640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa35015a3a0 0x7fa35006cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355759640 1 -- 172.21.15.59:0/2418757216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3501f6ae0 con 0x7fa35006e840 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa355759640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35006e840 0x7fa35013f0d0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fa34800c3a0 tx=0x7fa34800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa34dffb640 1 -- 172.21.15.59:0/2418757216 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348011790 con 0x7fa35006e840 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa34dffb640 1 -- 172.21.15.59:0/2418757216 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa348017070 con 0x7fa35006e840 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa350157b60 con 0x7fa35006e840 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.838+0000 7fa34dffb640 1 -- 172.21.15.59:0/2418757216 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348011cc0 con 0x7fa35006e840 2024-06-29T14:39:45.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.839+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa350157e00 con 0x7fa35006e840 2024-06-29T14:39:45.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.840+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3500d0140 con 0x7fa35006e840 2024-06-29T14:39:45.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.840+0000 7fa34dffb640 1 -- 172.21.15.59:0/2418757216 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 55) ==== 98039+0+0 (secure 0 0 0) 0x7fa34801c460 con 0x7fa35006e840 2024-06-29T14:39:45.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.841+0000 7fa34dffb640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa33c076120 0x7fa33c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.841+0000 7fa34dffb640 1 -- 172.21.15.59:0/2418757216 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6991+0+0 (secure 0 0 0) 0x7fa348097670 con 0x7fa35006e840 2024-06-29T14:39:45.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.841+0000 7fa354f58640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa33c076120 0x7fa33c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.841+0000 7fa354f58640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa33c076120 0x7fa33c078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa3500b5d90 tx=0x7fa344005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:45.845+0000 7fa34dffb640 1 -- 172.21.15.59:0/2418757216 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa3480606f0 con 0x7fa35006e840 2024-06-29T14:39:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.039+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa350071040 con 0x7fa35006e840 2024-06-29T14:39:46.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.040+0000 7fa34dffb640 1 -- 172.21.15.59:0/2418757216 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+18240 (secure 0 0 0) 0x7fa3480649f0 con 0x7fa35006e840 2024-06-29T14:39:46.038 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:46.038 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":118,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:43.971617+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.981044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.565073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.a","mappings":[{"from":1,"to":2}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1b","mappings":[{"from":1,"to":2}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1864308943":"2024-06-30T14:39:43.971541+0000","172.21.15.153:6833/716775118":"2024-06-30T14:39:43.971541+0000","172.21.15.153:6832/716775118":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3495517559":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3803261247":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3992874705":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/2786718243":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/493501963":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/1628521659":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/2502091213":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3749556168":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3059657409":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/438874239":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3367163578":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6832/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/441578565":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6833/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/578976042":"2024-06-30T14:38:36.889100+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:46.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa33c076120 msgr2=0x7fa33c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa33c076120 0x7fa33c078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa3500b5d90 tx=0x7fa344005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:46.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35006e840 msgr2=0x7fa35013f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35006e840 0x7fa35013f0d0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fa34800c3a0 tx=0x7fa34800c860 comp rx=0 tx=0).stop 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 shutdown_connections 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa35015a3a0 0x7fa35006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa350144470 0x7fa35006ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa33c076120 0x7fa33c078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 --2- 172.21.15.59:0/2418757216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35006e840 0x7fa35013f0d0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 >> 172.21.15.59:0/2418757216 conn(0x7fa3500b06e0 msgr2=0x7fa3501412f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 shutdown_connections 2024-06-29T14:39:46.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.042+0000 7fa3579bd640 1 -- 172.21.15.59:0/2418757216 wait complete. 2024-06-29T14:39:46.051 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:39:46.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.172+0000 7f1be472a640 1 -- 172.21.15.59:0/2148631243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 msgr2=0x7f1bc8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.172+0000 7f1be472a640 1 --2- 172.21.15.59:0/2148631243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 0x7f1bc8004c10 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00c2e0 tx=0x7f1bc8005830 comp rx=0 tx=0).stop 2024-06-29T14:39:46.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.172+0000 7f1be2cc7640 1 -- 172.21.15.59:0/2148631243 reap_dead start 2024-06-29T14:39:46.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.172+0000 7f1be472a640 1 -- 172.21.15.59:0/2148631243 shutdown_connections 2024-06-29T14:39:46.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.172+0000 7f1be472a640 1 -- 172.21.15.59:0/2148631243 >> 172.21.15.59:0/2148631243 conn(0x7f1bdc0b5610 msgr2=0x7f1bdc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:46.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.172+0000 7f1be472a640 1 -- 172.21.15.59:0/2148631243 shutdown_connections 2024-06-29T14:39:46.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.172+0000 7f1be472a640 1 -- 172.21.15.59:0/2148631243 wait complete. 2024-06-29T14:39:46.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.174+0000 7f1be472a640 1 Processor -- start 2024-06-29T14:39:46.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.174+0000 7f1be472a640 1 -- start start 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be472a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 0x7f1bdc153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be472a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0ada60 0x7f1bdc0ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be472a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc0b00a0 0x7f1bdc0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be472a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1bdc0b7bd0 con 0x7f1bc8004800 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be472a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1bdc0cf2d0 con 0x7f1bdc0ada60 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be472a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1bdc0b7d50 con 0x7f1bdc0b00a0 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 0x7f1bdc153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 0x7f1bdc153f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44118/0 (socket says 172.21.15.59:44118) 2024-06-29T14:39:46.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 -- 172.21.15.59:0/2339203505 learned_addr learned my addr 172.21.15.59:0/2339203505 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be2cc7640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc0b00a0 0x7f1bdc0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 -- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc0b00a0 msgr2=0x7f1bdc0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be1cc5640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0ada60 0x7f1bdc0ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc0b00a0 0x7f1bdc0b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 -- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0ada60 msgr2=0x7f1bdc0ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0ada60 0x7f1bdc0ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 -- 172.21.15.59:0/2339203505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bdc0b09f0 con 0x7f1bc8004800 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be24c6640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 0x7f1bdc153f80 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8003f80 tx=0x7f1bd800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1bd2ffd640 1 -- 172.21.15.59:0/2339203505 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8010a60 con 0x7f1bc8004800 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1bd2ffd640 1 -- 172.21.15.59:0/2339203505 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bd8015070 con 0x7f1bc8004800 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bdc1544c0 con 0x7f1bc8004800 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1bd2ffd640 1 -- 172.21.15.59:0/2339203505 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8010e80 con 0x7f1bc8004800 2024-06-29T14:39:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.176+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bdc0aea20 con 0x7f1bc8004800 2024-06-29T14:39:46.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.177+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bdc0d1a20 con 0x7f1bc8004800 2024-06-29T14:39:46.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.178+0000 7f1bd2ffd640 1 -- 172.21.15.59:0/2339203505 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 55) ==== 98039+0+0 (secure 0 0 0) 0x7f1bd801a470 con 0x7f1bc8004800 2024-06-29T14:39:46.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.179+0000 7f1bd2ffd640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1bc0076120 0x7f1bc0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.179+0000 7f1bd2ffd640 1 -- 172.21.15.59:0/2339203505 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6991+0+0 (secure 0 0 0) 0x7f1bd8098670 con 0x7f1bc8004800 2024-06-29T14:39:46.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.179+0000 7f1be1cc5640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1bc0076120 0x7f1bc0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.180+0000 7f1be1cc5640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1bc0076120 0x7f1bc0078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc033000 tx=0x7f1bcc02d000 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:46.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.183+0000 7f1bd2ffd640 1 -- 172.21.15.59:0/2339203505 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1bdc0b09f0 con 0x7f1bc8004800 2024-06-29T14:39:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.363+0000 7f1bd2ffd640 1 -- 172.21.15.59:0/2339203505 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 56) ==== 98247+0+0 (secure 0 0 0) 0x7f1bd805d920 con 0x7f1bc8004800 2024-06-29T14:39:46.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.416+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1bdc0b9920 con 0x7f1bc8004800 2024-06-29T14:39:46.415 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T14:39:46.417+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:39:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.420+0000 7f1bd2ffd640 1 -- 172.21.15.59:0/2339203505 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v56) ==== 161+0+0 (secure 0 0 0) 0x7f1bd8061d90 con 0x7f1bc8004800 2024-06-29T14:39:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:39:46.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1bc0076120 msgr2=0x7f1bc0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1bc0076120 0x7f1bc0078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc033000 tx=0x7f1bcc02d000 comp rx=0 tx=0).stop 2024-06-29T14:39:46.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 msgr2=0x7f1bdc153f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 0x7f1bdc153f80 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8003f80 tx=0x7f1bd800ba40 comp rx=0 tx=0).stop 2024-06-29T14:39:46.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 shutdown_connections 2024-06-29T14:39:46.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc0b00a0 0x7f1bdc0b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0ada60 0x7f1bdc0ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1bc0076120 0x7f1bc0078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 --2- 172.21.15.59:0/2339203505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bc8004800 0x7f1bdc153f80 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 >> 172.21.15.59:0/2339203505 conn(0x7f1bdc0b5610 msgr2=0x7f1bdc1618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:46.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 shutdown_connections 2024-06-29T14:39:46.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.423+0000 7f1be472a640 1 -- 172.21.15.59:0/2339203505 wait complete. 2024-06-29T14:39:46.425 INFO:tasks.ceph.mds.a.smithi059.stderr: -7963> 2024-06-29T14:38:27.214+0000 7f188fbfc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:39:46.433 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:39:46.484 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:39:46.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.558+0000 7fb9ee911640 1 -- 172.21.15.59:0/3090028160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815eed0 msgr2=0x7fb9e815d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.558+0000 7fb9ee911640 1 --2- 172.21.15.59:0/3090028160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815eed0 0x7fb9e815d2c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8009860 tx=0x7fb9d802a5c0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.558+0000 7fb9ee911640 1 -- 172.21.15.59:0/3090028160 shutdown_connections 2024-06-29T14:39:46.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.558+0000 7fb9ee911640 1 --2- 172.21.15.59:0/3090028160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815eed0 0x7fb9e815d2c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.558+0000 7fb9ee911640 1 -- 172.21.15.59:0/3090028160 >> 172.21.15.59:0/3090028160 conn(0x7fb9e8073da0 msgr2=0x7fb9e80741b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:46.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.559+0000 7fb9ee911640 1 -- 172.21.15.59:0/3090028160 shutdown_connections 2024-06-29T14:39:46.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.559+0000 7fb9ee911640 1 -- 172.21.15.59:0/3090028160 wait complete. 2024-06-29T14:39:46.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.560+0000 7fb9ee911640 1 Processor -- start 2024-06-29T14:39:46.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.560+0000 7fb9ee911640 1 -- start start 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9ee911640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815a450 0x7fb9e8141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9ee911640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e815d800 0x7fb9e80a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9ee911640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e815eed0 0x7fb9e80d0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9ee911640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e815cbe0 con 0x7fb9e815eed0 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9ee911640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e815ca60 con 0x7fb9e815d800 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9ee911640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e815cd60 con 0x7fb9e815a450 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e815eed0 0x7fb9e80d0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e815eed0 0x7fb9e80d0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44134/0 (socket says 172.21.15.59:44134) 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 -- 172.21.15.59:0/2494752623 learned_addr learned my addr 172.21.15.59:0/2494752623 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9e7fff640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815a450 0x7fb9e8141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9e77fe640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e815d800 0x7fb9e80a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 -- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815a450 msgr2=0x7fb9e8141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815a450 0x7fb9e8141330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 -- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e815d800 msgr2=0x7fb9e80a9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e815d800 0x7fb9e80a9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 -- 172.21.15.59:0/2494752623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e81f28c0 con 0x7fb9e815eed0 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9e7fff640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815a450 0x7fb9e8141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9e77fe640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e815d800 0x7fb9e80a9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.562+0000 7fb9eceae640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e815eed0 0x7fb9e80d0fc0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000c700 tx=0x7fb9e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.563+0000 7fb9e4ff9640 1 -- 172.21.15.59:0/2494752623 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0003b70 con 0x7fb9e815eed0 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.563+0000 7fb9e4ff9640 1 -- 172.21.15.59:0/2494752623 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9e0003d10 con 0x7fb9e815eed0 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.563+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e81f2b80 con 0x7fb9e815eed0 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.563+0000 7fb9e4ff9640 1 -- 172.21.15.59:0/2494752623 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0005330 con 0x7fb9e815eed0 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.563+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e81f3140 con 0x7fb9e815eed0 2024-06-29T14:39:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.564+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e80aa630 con 0x7fb9e815eed0 2024-06-29T14:39:46.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.565+0000 7fb9e4ff9640 1 -- 172.21.15.59:0/2494752623 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 56) ==== 98247+0+0 (secure 0 0 0) 0x7fb9e00054d0 con 0x7fb9e815eed0 2024-06-29T14:39:46.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.565+0000 7fb9e4ff9640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb9d4076360 0x7fb9d4078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.565+0000 7fb9e4ff9640 1 -- 172.21.15.59:0/2494752623 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 7077+0+0 (secure 0 0 0) 0x7fb9e0097ab0 con 0x7fb9e815eed0 2024-06-29T14:39:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.565+0000 7fb9e7fff640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb9d4076360 0x7fb9d4078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.566+0000 7fb9e7fff640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb9d4076360 0x7fb9d4078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8034000 tx=0x7fb9d8004730 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:46.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.569+0000 7fb9e4ff9640 1 -- 172.21.15.59:0/2494752623 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb9e0060ec0 con 0x7fb9e815eed0 2024-06-29T14:39:46.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:46.800+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb9e80cebf0 con 0x7fb9e815eed0 2024-06-29T14:39:47.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.422+0000 7fb9e4ff9640 1 -- 172.21.15.59:0/2494752623 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v57) ==== 104+0+0 (secure 0 0 0) 0x7fb9e80cebf0 con 0x7fb9e815eed0 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb9d4076360 msgr2=0x7fb9d4078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb9d4076360 0x7fb9d4078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8034000 tx=0x7fb9d8004730 comp rx=0 tx=0).stop 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e815eed0 msgr2=0x7fb9e80d0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e815eed0 0x7fb9e80d0fc0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000c700 tx=0x7fb9e000cbc0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 shutdown_connections 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb9d4076360 0x7fb9d4078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e815eed0 0x7fb9e80d0fc0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e815d800 0x7fb9e80a9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 --2- 172.21.15.59:0/2494752623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e815a450 0x7fb9e8141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.424+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 >> 172.21.15.59:0/2494752623 conn(0x7fb9e8073da0 msgr2=0x7fb9e80b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:47.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.425+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 shutdown_connections 2024-06-29T14:39:47.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.425+0000 7fb9ee911640 1 -- 172.21.15.59:0/2494752623 wait complete. 2024-06-29T14:39:47.437 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:39:47.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 -- 172.21.15.59:0/3209520099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99540743d0 msgr2=0x7f99540cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 --2- 172.21.15.59:0/3209520099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99540743d0 0x7f99540cea30 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f99540b6c80 tx=0x7f994402aa30 comp rx=0 tx=0).stop 2024-06-29T14:39:47.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 -- 172.21.15.59:0/3209520099 shutdown_connections 2024-06-29T14:39:47.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 --2- 172.21.15.59:0/3209520099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99540743d0 0x7f99540cea30 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 --2- 172.21.15.59:0/3209520099 >> v2:172.21.15.153:3300/0 conn(0x7f995415ce00 0x7f9954073e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 --2- 172.21.15.59:0/3209520099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9954145760 0x7f99540d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 -- 172.21.15.59:0/3209520099 >> 172.21.15.59:0/3209520099 conn(0x7f99540cd630 msgr2=0x7f99540cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 -- 172.21.15.59:0/3209520099 shutdown_connections 2024-06-29T14:39:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.568+0000 7f995977b640 1 -- 172.21.15.59:0/3209520099 wait complete. 2024-06-29T14:39:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.570+0000 7f995977b640 1 Processor -- start 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.570+0000 7f995977b640 1 -- start start 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f995977b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99540743d0 0x7f99541f8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f995977b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9954145760 0x7f99541f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f995977b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995415ce00 0x7f99541f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f995977b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f99540cf300 con 0x7f995415ce00 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f995977b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99540cf180 con 0x7f99540743d0 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f995977b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f99540cf480 con 0x7f9954145760 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f99537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995415ce00 0x7f99541f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f99537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995415ce00 0x7f99541f7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44146/0 (socket says 172.21.15.59:44146) 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.571+0000 7f99537fe640 1 -- 172.21.15.59:0/492877508 learned_addr learned my addr 172.21.15.59:0/492877508 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f9952ffd640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99540743d0 0x7f99541f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f99527fc640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9954145760 0x7f99541f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f99537fe640 1 -- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9954145760 msgr2=0x7f99541f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f99537fe640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9954145760 0x7f99541f8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f99537fe640 1 -- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99540743d0 msgr2=0x7f99541f8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f99537fe640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99540743d0 0x7f99541f8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f99537fe640 1 -- 172.21.15.59:0/492877508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99540cc740 con 0x7f995415ce00 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f99537fe640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995415ce00 0x7f99541f7520 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f994402b020 tx=0x7f9944003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f994b7fe640 1 -- 172.21.15.59:0/492877508 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944033050 con 0x7f995415ce00 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f994b7fe640 1 -- 172.21.15.59:0/492877508 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9944004450 con 0x7f995415ce00 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f994b7fe640 1 -- 172.21.15.59:0/492877508 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944004840 con 0x7f995415ce00 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99540b4f20 con 0x7f995415ce00 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.572+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f995413d5d0 con 0x7f995415ce00 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.573+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99540d04a0 con 0x7f995415ce00 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.574+0000 7f994b7fe640 1 -- 172.21.15.59:0/492877508 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 56) ==== 98247+0+0 (secure 0 0 0) 0x7f9944036070 con 0x7f995415ce00 2024-06-29T14:39:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.574+0000 7f994b7fe640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9934076390 0x7f99340788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:47.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.574+0000 7f994b7fe640 1 -- 172.21.15.59:0/492877508 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 7077+0+0 (secure 0 0 0) 0x7f99440b9390 con 0x7f995415ce00 2024-06-29T14:39:47.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.575+0000 7f9952ffd640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9934076390 0x7f99340788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:47.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.575+0000 7f9952ffd640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9934076390 0x7f99340788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9954072090 tx=0x7f994c005c70 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:47.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.579+0000 7f994b7fe640 1 -- 172.21.15.59:0/492877508 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99440825a0 con 0x7f995415ce00 2024-06-29T14:39:47.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:47.776+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f99540b4c70 con 0x7f995415ce00 2024-06-29T14:39:48.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.427+0000 7f994b7fe640 1 -- 172.21.15.59:0/492877508 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v120) ==== 181+0+0 (secure 0 0 0) 0x7f9944081ea0 con 0x7f995415ce00 2024-06-29T14:39:48.426 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:39:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.434+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9934076390 msgr2=0x7f99340788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.434+0000 7f995977b640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9934076390 0x7f99340788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9954072090 tx=0x7f994c005c70 comp rx=0 tx=0).stop 2024-06-29T14:39:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995415ce00 msgr2=0x7f99541f7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f994b7fe640 1 -- 172.21.15.59:0/492877508 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f994402d070 con 0x7f995415ce00 2024-06-29T14:39:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995415ce00 0x7f99541f7520 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f994402b020 tx=0x7f9944003f10 comp rx=0 tx=0).stop 2024-06-29T14:39:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 shutdown_connections 2024-06-29T14:39:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9934076390 0x7f99340788a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f995415ce00 0x7f99541f7520 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9954145760 0x7f99541f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 --2- 172.21.15.59:0/492877508 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99540743d0 0x7f99541f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 >> 172.21.15.59:0/492877508 conn(0x7f99540cd630 msgr2=0x7f99540b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:48.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.435+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 shutdown_connections 2024-06-29T14:39:48.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.436+0000 7f995977b640 1 -- 172.21.15.59:0/492877508 wait complete. 2024-06-29T14:39:48.449 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:39:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.580+0000 7fb05c01f640 1 -- 172.21.15.59:0/968840927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb054162600 msgr2=0x7fb0540b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.580+0000 7fb05c01f640 1 --2- 172.21.15.59:0/968840927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb054162600 0x7fb0540b3ed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb04c005fd0 tx=0x7fb04c02aa90 comp rx=0 tx=0).stop 2024-06-29T14:39:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.580+0000 7fb05c01f640 1 -- 172.21.15.59:0/968840927 shutdown_connections 2024-06-29T14:39:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.580+0000 7fb05c01f640 1 --2- 172.21.15.59:0/968840927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb054162600 0x7fb0540b3ed0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.580+0000 7fb05c01f640 1 --2- 172.21.15.59:0/968840927 >> v2:172.21.15.153:3300/0 conn(0x7fb05415dff0 0x7fb0540d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.580+0000 7fb05c01f640 1 -- 172.21.15.59:0/968840927 >> 172.21.15.59:0/968840927 conn(0x7fb0540d3e60 msgr2=0x7fb0540d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:48.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.581+0000 7fb05c01f640 1 -- 172.21.15.59:0/968840927 shutdown_connections 2024-06-29T14:39:48.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.581+0000 7fb05c01f640 1 -- 172.21.15.59:0/968840927 wait complete. 2024-06-29T14:39:48.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.582+0000 7fb05c01f640 1 Processor -- start 2024-06-29T14:39:48.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.582+0000 7fb05c01f640 1 -- start start 2024-06-29T14:39:48.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05c01f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb05415dff0 0x7fb0540aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:48.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05c01f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb054161ae0 0x7fb05415c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05c01f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb054162600 0x7fb0541f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05c01f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb0540b4860 con 0x7fb054162600 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05c01f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb0540b46e0 con 0x7fb054161ae0 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05c01f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb0540b49e0 con 0x7fb05415dff0 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb054162600 0x7fb0541f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb054162600 0x7fb0541f83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44152/0 (socket says 172.21.15.59:44152) 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 -- 172.21.15.59:0/479093209 learned_addr learned my addr 172.21.15.59:0/479093209 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb059dbb640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb05415dff0 0x7fb0540aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb0595ba640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb054161ae0 0x7fb05415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 -- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb05415dff0 msgr2=0x7fb0540aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb05415dff0 0x7fb0540aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 -- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb054161ae0 msgr2=0x7fb05415c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb054161ae0 0x7fb05415c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 -- 172.21.15.59:0/479093209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0540ab8a0 con 0x7fb054162600 2024-06-29T14:39:48.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb0595ba640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb054161ae0 0x7fb05415c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:48.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.584+0000 7fb05a5bc640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb054162600 0x7fb0541f83a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009d90 tx=0x7fb04c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:48.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.585+0000 7fb04a7fc640 1 -- 172.21.15.59:0/479093209 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c044070 con 0x7fb054162600 2024-06-29T14:39:48.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.585+0000 7fb04a7fc640 1 -- 172.21.15.59:0/479093209 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb04c004040 con 0x7fb054162600 2024-06-29T14:39:48.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.585+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0540abb80 con 0x7fb054162600 2024-06-29T14:39:48.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.585+0000 7fb04a7fc640 1 -- 172.21.15.59:0/479093209 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c004430 con 0x7fb054162600 2024-06-29T14:39:48.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.585+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0540ac140 con 0x7fb054162600 2024-06-29T14:39:48.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.586+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0540b4f00 con 0x7fb054162600 2024-06-29T14:39:48.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.586+0000 7fb04a7fc640 1 -- 172.21.15.59:0/479093209 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fb04c047070 con 0x7fb054162600 2024-06-29T14:39:48.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.587+0000 7fb04a7fc640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb038076420 0x7fb038078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:48.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.587+0000 7fb04a7fc640 1 -- 172.21.15.59:0/479093209 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6639+0+0 (secure 0 0 0) 0x7fb04c0b8aa0 con 0x7fb054162600 2024-06-29T14:39:48.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.587+0000 7fb059dbb640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb038076420 0x7fb038078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:48.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.588+0000 7fb059dbb640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb038076420 0x7fb038078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb050009ac0 tx=0x7fb050005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:48.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.591+0000 7fb04a7fc640 1 -- 172.21.15.59:0/479093209 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb04c032030 con 0x7fb054162600 2024-06-29T14:39:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:48.782+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb0540ab630 con 0x7fb054162600 2024-06-29T14:39:49.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.433+0000 7fb04a7fc640 1 -- 172.21.15.59:0/479093209 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v121) ==== 169+0+0 (secure 0 0 0) 0x7fb04c081e50 con 0x7fb054162600 2024-06-29T14:39:49.432 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T14:39:49.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.436+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb038076420 msgr2=0x7fb038078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.436+0000 7fb05c01f640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb038076420 0x7fb038078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb050009ac0 tx=0x7fb050005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:49.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.436+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb054162600 msgr2=0x7fb0541f83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.436+0000 7fb05c01f640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb054162600 0x7fb0541f83a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009d90 tx=0x7fb04c003ce0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.438+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 shutdown_connections 2024-06-29T14:39:49.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.438+0000 7fb05c01f640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb038076420 0x7fb038078930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.438+0000 7fb05c01f640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb054162600 0x7fb0541f83a0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.438+0000 7fb05c01f640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb054161ae0 0x7fb05415c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.438+0000 7fb05c01f640 1 --2- 172.21.15.59:0/479093209 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb05415dff0 0x7fb0540aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.438+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 >> 172.21.15.59:0/479093209 conn(0x7fb0540d3e60 msgr2=0x7fb054160850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:49.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.438+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 shutdown_connections 2024-06-29T14:39:49.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.438+0000 7fb05c01f640 1 -- 172.21.15.59:0/479093209 wait complete. 2024-06-29T14:39:49.451 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:39:49.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.576+0000 7f4395474640 1 -- 172.21.15.59:0/3225923552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f439015ceb0 msgr2=0x7f43900b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.576+0000 7f4395474640 1 --2- 172.21.15.59:0/3225923552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f439015ceb0 0x7f43900b56d0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f43900b5590 tx=0x7f438802a4d0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.576+0000 7f4395474640 1 -- 172.21.15.59:0/3225923552 shutdown_connections 2024-06-29T14:39:49.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.576+0000 7f4395474640 1 --2- 172.21.15.59:0/3225923552 >> v2:172.21.15.153:3300/0 conn(0x7f439013e9b0 0x7f439013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.576+0000 7f4395474640 1 --2- 172.21.15.59:0/3225923552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f439015ceb0 0x7f43900b56d0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.576+0000 7f4395474640 1 -- 172.21.15.59:0/3225923552 >> 172.21.15.59:0/3225923552 conn(0x7f43900b6fb0 msgr2=0x7f43900b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:49.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.576+0000 7f4395474640 1 -- 172.21.15.59:0/3225923552 shutdown_connections 2024-06-29T14:39:49.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.576+0000 7f4395474640 1 -- 172.21.15.59:0/3225923552 wait complete. 2024-06-29T14:39:49.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.578+0000 7f4395474640 1 Processor -- start 2024-06-29T14:39:49.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.578+0000 7f4395474640 1 -- start start 2024-06-29T14:39:49.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f4395474640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439013e9b0 0x7f43901eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f4395474640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43901414c0 0x7f43900cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f4395474640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439015ceb0 0x7f43900cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f4395474640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f43900b29d0 con 0x7f43901414c0 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f4395474640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43900b2850 con 0x7f439013e9b0 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f4395474640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f43900b2b50 con 0x7f439015ceb0 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f438e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43901414c0 0x7f43900cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f438e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43901414c0 0x7f43900cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44178/0 (socket says 172.21.15.59:44178) 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f438e7fc640 1 -- 172.21.15.59:0/2903127507 learned_addr learned my addr 172.21.15.59:0/2903127507 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f438effd640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439013e9b0 0x7f43901eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.579+0000 7f438f7fe640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439015ceb0 0x7f43900cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f438e7fc640 1 -- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439015ceb0 msgr2=0x7f43900cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f438e7fc640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439015ceb0 0x7f43900cd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f438e7fc640 1 -- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439013e9b0 msgr2=0x7f43901eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f438e7fc640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439013e9b0 0x7f43901eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f438e7fc640 1 -- 172.21.15.59:0/2903127507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43900cb650 con 0x7f43901414c0 2024-06-29T14:39:49.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f438e7fc640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43901414c0 0x7f43900cd460 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f437800b410 tx=0x7f437800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:49.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f43877fe640 1 -- 172.21.15.59:0/2903127507 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378015020 con 0x7f43901414c0 2024-06-29T14:39:49.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f43877fe640 1 -- 172.21.15.59:0/2903127507 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4378011920 con 0x7f43901414c0 2024-06-29T14:39:49.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43900cb930 con 0x7f43901414c0 2024-06-29T14:39:49.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f43877fe640 1 -- 172.21.15.59:0/2903127507 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378011d70 con 0x7f43901414c0 2024-06-29T14:39:49.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.580+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43900cbef0 con 0x7f43901414c0 2024-06-29T14:39:49.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.581+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f439015f500 con 0x7f43901414c0 2024-06-29T14:39:49.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.582+0000 7f43877fe640 1 -- 172.21.15.59:0/2903127507 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f4378018020 con 0x7f43901414c0 2024-06-29T14:39:49.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.582+0000 7f43877fe640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f43700763f0 0x7f4370078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.582+0000 7f43877fe640 1 -- 172.21.15.59:0/2903127507 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7f4378097a50 con 0x7f43901414c0 2024-06-29T14:39:49.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.583+0000 7f438effd640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f43700763f0 0x7f4370078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.583+0000 7f438effd640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f43700763f0 0x7f4370078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4388033000 tx=0x7f4388004730 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:49.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.587+0000 7f43877fe640 1 -- 172.21.15.59:0/2903127507 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f437801f020 con 0x7f43901414c0 2024-06-29T14:39:49.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.776+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f439013dfd0 con 0x7f43901414c0 2024-06-29T14:39:49.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.780+0000 7f43877fe640 1 -- 172.21.15.59:0/2903127507 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v47) ==== 71+0+0 (secure 0 0 0) 0x7f4378061080 con 0x7f43901414c0 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f43700763f0 msgr2=0x7f4370078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f43700763f0 0x7f4370078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4388033000 tx=0x7f4388004730 comp rx=0 tx=0).stop 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43901414c0 msgr2=0x7f43900cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43901414c0 0x7f43900cd460 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f437800b410 tx=0x7f437800b8d0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 shutdown_connections 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f439015ceb0 0x7f43900cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f43700763f0 0x7f4370078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43901414c0 0x7f43900cd460 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 --2- 172.21.15.59:0/2903127507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439013e9b0 0x7f43901eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 >> 172.21.15.59:0/2903127507 conn(0x7f43900b6fb0 msgr2=0x7f43900b3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 shutdown_connections 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.782+0000 7f4395474640 1 -- 172.21.15.59:0/2903127507 wait complete. 2024-06-29T14:39:49.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-06-29T14:39:49.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.911+0000 7fb20e77f640 1 -- 172.21.15.59:0/2522324397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080b2fa0 msgr2=0x7fb2080d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.911+0000 7fb20e77f640 1 --2- 172.21.15.59:0/2522324397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080b2fa0 0x7fb2080d1520 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fb208074c50 tx=0x7fb1fc02a560 comp rx=0 tx=0).stop 2024-06-29T14:39:49.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.911+0000 7fb20e77f640 1 -- 172.21.15.59:0/2522324397 shutdown_connections 2024-06-29T14:39:49.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.911+0000 7fb20e77f640 1 --2- 172.21.15.59:0/2522324397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080b2fa0 0x7fb2080d1520 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.911+0000 7fb20e77f640 1 --2- 172.21.15.59:0/2522324397 >> v2:172.21.15.153:3300/0 conn(0x7fb2080b2270 0x7fb20815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.911+0000 7fb20e77f640 1 -- 172.21.15.59:0/2522324397 >> 172.21.15.59:0/2522324397 conn(0x7fb208073e10 msgr2=0x7fb208074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:49.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.911+0000 7fb20e77f640 1 -- 172.21.15.59:0/2522324397 shutdown_connections 2024-06-29T14:39:49.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.911+0000 7fb20e77f640 1 -- 172.21.15.59:0/2522324397 wait complete. 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.913+0000 7fb20e77f640 1 Processor -- start 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.913+0000 7fb20e77f640 1 -- start start 2024-06-29T14:39:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20e77f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2080b2270 0x7fb2081eff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20e77f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2080b2fa0 0x7fb2080a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20e77f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080d1a60 0x7fb2080b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20e77f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb2080b6cc0 con 0x7fb2080d1a60 2024-06-29T14:39:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20e77f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb2080b6b40 con 0x7fb2080b2270 2024-06-29T14:39:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20e77f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb2080cfe90 con 0x7fb2080b2fa0 2024-06-29T14:39:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080d1a60 0x7fb2080b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080d1a60 0x7fb2080b34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44210/0 (socket says 172.21.15.59:44210) 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 -- 172.21.15.59:0/733361526 learned_addr learned my addr 172.21.15.59:0/733361526 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb207fff640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2080b2270 0x7fb2081eff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb2077fe640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2080b2fa0 0x7fb2080a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 -- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2080b2fa0 msgr2=0x7fb2080a9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2080b2fa0 0x7fb2080a9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 -- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2080b2270 msgr2=0x7fb2081eff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2080b2270 0x7fb2081eff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 -- 172.21.15.59:0/733361526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2080aa8d0 con 0x7fb2080d1a60 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb207fff640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2080b2270 0x7fb2081eff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20cd1c640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080d1a60 0x7fb2080b34b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb20000b970 tx=0x7fb20000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb204ff9640 1 -- 172.21.15.59:0/733361526 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200019070 con 0x7fb2080d1a60 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb204ff9640 1 -- 172.21.15.59:0/733361526 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb20000cc30 con 0x7fb2080d1a60 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb204ff9640 1 -- 172.21.15.59:0/733361526 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2000146a0 con 0x7fb2080d1a60 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.915+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2081f2c80 con 0x7fb2080d1a60 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.916+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2081f3240 con 0x7fb2080d1a60 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.916+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2080d03b0 con 0x7fb2080d1a60 2024-06-29T14:39:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.917+0000 7fb204ff9640 1 -- 172.21.15.59:0/733361526 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fb20001c070 con 0x7fb2080d1a60 2024-06-29T14:39:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.918+0000 7fb204ff9640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb1f40763f0 0x7fb1f4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.918+0000 7fb204ff9640 1 -- 172.21.15.59:0/733361526 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7fb200097770 con 0x7fb2080d1a60 2024-06-29T14:39:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.918+0000 7fb207fff640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb1f40763f0 0x7fb1f4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.918+0000 7fb207fff640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb1f40763f0 0x7fb1f4078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8005fd0 tx=0x7fb1f8005e20 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:49.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:49.922+0000 7fb204ff9640 1 -- 172.21.15.59:0/733361526 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb200060da0 con 0x7fb2080d1a60 2024-06-29T14:39:50.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.193+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) -- 0x7fb208075b40 con 0x7fb2080d1a60 2024-06-29T14:39:50.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.435+0000 7fb204ff9640 1 -- 172.21.15.59:0/733361526 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v690) ==== 183+0+0 (secure 0 0 0) 0x7fb2000606a0 con 0x7fb2080d1a60 2024-06-29T14:39:50.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb1f40763f0 msgr2=0x7fb1f4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb1f40763f0 0x7fb1f4078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8005fd0 tx=0x7fb1f8005e20 comp rx=0 tx=0).stop 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080d1a60 msgr2=0x7fb2080b34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080d1a60 0x7fb2080b34b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb20000b970 tx=0x7fb20000be30 comp rx=0 tx=0).stop 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 shutdown_connections 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb1f40763f0 0x7fb1f4078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2080d1a60 0x7fb2080b34b0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2080b2fa0 0x7fb2080a9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 --2- 172.21.15.59:0/733361526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2080b2270 0x7fb2081eff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 >> 172.21.15.59:0/733361526 conn(0x7fb208073e10 msgr2=0x7fb208074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 shutdown_connections 2024-06-29T14:39:50.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.437+0000 7fb20e77f640 1 -- 172.21.15.59:0/733361526 wait complete. 2024-06-29T14:39:50.447 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-06-29T14:39:50.447 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-06-29T14:39:50.447 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:39:50.447 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:50.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.567+0000 7fa45499a640 1 -- 172.21.15.59:0/2231320435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4500b3740 msgr2=0x7fa4500b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.567+0000 7fa45499a640 1 --2- 172.21.15.59:0/2231320435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4500b3740 0x7fa4500b3b50 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fa4500b4b50 tx=0x7fa43802a530 comp rx=0 tx=0).stop 2024-06-29T14:39:50.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.567+0000 7fa45499a640 1 -- 172.21.15.59:0/2231320435 shutdown_connections 2024-06-29T14:39:50.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.567+0000 7fa45499a640 1 --2- 172.21.15.59:0/2231320435 >> v2:172.21.15.153:3300/0 conn(0x7fa45015c2f0 0x7fa45015c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.567+0000 7fa45499a640 1 --2- 172.21.15.59:0/2231320435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4500b3740 0x7fa4500b3b50 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.567+0000 7fa45499a640 1 -- 172.21.15.59:0/2231320435 >> 172.21.15.59:0/2231320435 conn(0x7fa45006c950 msgr2=0x7fa450073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:50.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.567+0000 7fa45499a640 1 -- 172.21.15.59:0/2231320435 shutdown_connections 2024-06-29T14:39:50.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.567+0000 7fa45499a640 1 -- 172.21.15.59:0/2231320435 wait complete. 2024-06-29T14:39:50.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.569+0000 7fa45499a640 1 Processor -- start 2024-06-29T14:39:50.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.569+0000 7fa45499a640 1 -- start start 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.570+0000 7fa45499a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4500b3740 0x7fa450143440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.570+0000 7fa45499a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45015c2f0 0x7fa45014c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.570+0000 7fa45499a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa45014c6e0 0x7fa45014caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa45499a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa4500b5fd0 con 0x7fa45015c2f0 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa45499a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4500b5e50 con 0x7fa45014c6e0 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa45499a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa4500b6150 con 0x7fa4500b3740 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45015c2f0 0x7fa45014c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45015c2f0 0x7fa45014c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44234/0 (socket says 172.21.15.59:44234) 2024-06-29T14:39:50.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 -- 172.21.15.59:0/1449346941 learned_addr learned my addr 172.21.15.59:0/1449346941 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44ed9d640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa45014c6e0 0x7fa45014caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44e59c640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4500b3740 0x7fa450143440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 -- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4500b3740 msgr2=0x7fa450143440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4500b3740 0x7fa450143440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 -- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa45014c6e0 msgr2=0x7fa45014caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa45014c6e0 0x7fa45014caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 -- 172.21.15.59:0/1449346941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa450143ee0 con 0x7fa45015c2f0 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa44dd9b640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45015c2f0 0x7fa45014c1a0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fa43802ab20 tx=0x7fa438033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa446ffd640 1 -- 172.21.15.59:0/1449346941 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438031e20 con 0x7fa45015c2f0 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa446ffd640 1 -- 172.21.15.59:0/1449346941 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa438005110 con 0x7fa45015c2f0 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4501441c0 con 0x7fa45015c2f0 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.571+0000 7fa446ffd640 1 -- 172.21.15.59:0/1449346941 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438005560 con 0x7fa45015c2f0 2024-06-29T14:39:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.572+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa450144780 con 0x7fa45015c2f0 2024-06-29T14:39:50.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.572+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4500b6670 con 0x7fa45015c2f0 2024-06-29T14:39:50.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.573+0000 7fa446ffd640 1 -- 172.21.15.59:0/1449346941 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fa4380047a0 con 0x7fa45015c2f0 2024-06-29T14:39:50.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.574+0000 7fa446ffd640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa4300763f0 0x7fa430078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.574+0000 7fa446ffd640 1 -- 172.21.15.59:0/1449346941 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7fa4380b8e90 con 0x7fa45015c2f0 2024-06-29T14:39:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.574+0000 7fa44e59c640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa4300763f0 0x7fa430078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.575+0000 7fa44e59c640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa4300763f0 0x7fa430078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa4400020b0 tx=0x7fa440009040 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:50.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.578+0000 7fa446ffd640 1 -- 172.21.15.59:0/1449346941 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa43803f030 con 0x7fa45015c2f0 2024-06-29T14:39:50.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.773+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4500742a0 con 0x7fa45015c2f0 2024-06-29T14:39:50.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.774+0000 7fa446ffd640 1 -- 172.21.15.59:0/1449346941 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+14455 (secure 0 0 0) 0x7fa438082320 con 0x7fa45015c2f0 2024-06-29T14:39:50.772 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:50.772 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":121,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:49.429355+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/2011834805":"2024-06-30T14:39:46.418321+0000","172.21.15.153:0/1864308943":"2024-06-30T14:39:43.971541+0000","172.21.15.153:6833/716775118":"2024-06-30T14:39:43.971541+0000","172.21.15.153:6832/716775118":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3495517559":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3803261247":"2024-06-30T14:39:43.971541+0000","172.21.15.59:6835/2011834805":"2024-06-30T14:39:46.418321+0000","172.21.15.153:0/3992874705":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/2786718243":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/493501963":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/1628521659":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/2502091213":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3749556168":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/3059657409":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/438874239":"2024-06-30T14:39:43.971541+0000","172.21.15.153:0/3367163578":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6832/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/441578565":"2024-06-30T14:38:36.889100+0000","172.21.15.153:6833/1090607011":"2024-06-30T14:38:36.889100+0000","172.21.15.153:0/578976042":"2024-06-30T14:38:36.889100+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa4300763f0 msgr2=0x7fa430078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa4300763f0 0x7fa430078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa4400020b0 tx=0x7fa440009040 comp rx=0 tx=0).stop 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45015c2f0 msgr2=0x7fa45014c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45015c2f0 0x7fa45014c1a0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fa43802ab20 tx=0x7fa438033d60 comp rx=0 tx=0).stop 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 shutdown_connections 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa45014c6e0 0x7fa45014caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa4300763f0 0x7fa430078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa45015c2f0 0x7fa45014c1a0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 --2- 172.21.15.59:0/1449346941 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4500b3740 0x7fa450143440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 >> 172.21.15.59:0/1449346941 conn(0x7fa45006c950 msgr2=0x7fa45006fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 shutdown_connections 2024-06-29T14:39:50.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.776+0000 7fa45499a640 1 -- 172.21.15.59:0/1449346941 wait complete. 2024-06-29T14:39:50.786 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:39:50.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:39:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.909+0000 7fd37e519640 1 -- 172.21.15.59:0/3160048106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd364004120 msgr2=0x7fd364004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.909+0000 7fd37e519640 1 --2- 172.21.15.59:0/3160048106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd364004120 0x7fd364004800 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fd360001690 tx=0x7fd364005830 comp rx=0 tx=0).stop 2024-06-29T14:39:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.909+0000 7fd37cab6640 1 -- 172.21.15.59:0/3160048106 reap_dead start 2024-06-29T14:39:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.909+0000 7fd37e519640 1 -- 172.21.15.59:0/3160048106 shutdown_connections 2024-06-29T14:39:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.909+0000 7fd37e519640 1 -- 172.21.15.59:0/3160048106 >> 172.21.15.59:0/3160048106 conn(0x7fd3780b06e0 msgr2=0x7fd3780b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.909+0000 7fd37e519640 1 -- 172.21.15.59:0/3160048106 shutdown_connections 2024-06-29T14:39:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.909+0000 7fd37e519640 1 -- 172.21.15.59:0/3160048106 wait complete. 2024-06-29T14:39:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.911+0000 7fd37e519640 1 Processor -- start 2024-06-29T14:39:50.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.911+0000 7fd37e519640 1 -- start start 2024-06-29T14:39:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37e519640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd364004120 0x7fd3781f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37e519640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3781412d0 0x7fd3781541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37e519640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3781f52e0 0x7fd37815c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37e519640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd378163a40 con 0x7fd3781f52e0 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37e519640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd3781638c0 con 0x7fd3781412d0 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37e519640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd378163bc0 con 0x7fd364004120 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37cab6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3781f52e0 0x7fd37815c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37cab6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3781f52e0 0x7fd37815c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44258/0 (socket says 172.21.15.59:44258) 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37cab6640 1 -- 172.21.15.59:0/3666742435 learned_addr learned my addr 172.21.15.59:0/3666742435 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd377fff640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd364004120 0x7fd3781f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd36ffff640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3781412d0 0x7fd3781541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37cab6640 1 -- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd364004120 msgr2=0x7fd3781f7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37cab6640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd364004120 0x7fd3781f7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37cab6640 1 -- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3781412d0 msgr2=0x7fd3781541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37cab6640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3781412d0 0x7fd3781541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd37cab6640 1 -- 172.21.15.59:0/3666742435 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3781f87b0 con 0x7fd3781f52e0 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.912+0000 7fd36ffff640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3781412d0 0x7fd3781541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.913+0000 7fd37cab6640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3781f52e0 0x7fd37815c320 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fd378072d60 tx=0x7fd370009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.913+0000 7fd3757fa640 1 -- 172.21.15.59:0/3666742435 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd370004b20 con 0x7fd3781f52e0 2024-06-29T14:39:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.913+0000 7fd3757fa640 1 -- 172.21.15.59:0/3666742435 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd370004db0 con 0x7fd3781f52e0 2024-06-29T14:39:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.913+0000 7fd3757fa640 1 -- 172.21.15.59:0/3666742435 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd370011970 con 0x7fd3781f52e0 2024-06-29T14:39:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.913+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3781f8a90 con 0x7fd3781f52e0 2024-06-29T14:39:50.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.913+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3781546f0 con 0x7fd3781f52e0 2024-06-29T14:39:50.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.914+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd378154ca0 con 0x7fd3781f52e0 2024-06-29T14:39:50.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.915+0000 7fd3757fa640 1 -- 172.21.15.59:0/3666742435 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fd37001b020 con 0x7fd3781f52e0 2024-06-29T14:39:50.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.915+0000 7fd3757fa640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fd358076420 0x7fd358078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.915+0000 7fd3757fa640 1 -- 172.21.15.59:0/3666742435 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7fd370097e20 con 0x7fd3781f52e0 2024-06-29T14:39:50.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.915+0000 7fd377fff640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fd358076420 0x7fd358078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.916+0000 7fd377fff640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fd358076420 0x7fd358078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd368001050 tx=0x7fd3680067a0 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:50.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:50.919+0000 7fd3757fa640 1 -- 172.21.15.59:0/3666742435 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd370061340 con 0x7fd3781f52e0 2024-06-29T14:39:51.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.111+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd3781f8540 con 0x7fd3781f52e0 2024-06-29T14:39:51.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.112+0000 7fd3757fa640 1 -- 172.21.15.59:0/3666742435 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) ==== 93+0+10 (secure 0 0 0) 0x7fd3700693c0 con 0x7fd3781f52e0 2024-06-29T14:39:51.110 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fd358076420 msgr2=0x7fd358078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fd358076420 0x7fd358078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd368001050 tx=0x7fd3680067a0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3781f52e0 msgr2=0x7fd37815c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3781f52e0 0x7fd37815c320 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fd378072d60 tx=0x7fd370009a40 comp rx=0 tx=0).stop 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 shutdown_connections 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fd358076420 0x7fd358078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3781f52e0 0x7fd37815c320 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3781412d0 0x7fd3781541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 --2- 172.21.15.59:0/3666742435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd364004120 0x7fd3781f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 >> 172.21.15.59:0/3666742435 conn(0x7fd3780b06e0 msgr2=0x7fd37815b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 shutdown_connections 2024-06-29T14:39:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.114+0000 7fd37e519640 1 -- 172.21.15.59:0/3666742435 wait complete. 2024-06-29T14:39:51.123 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-06-29T14:39:51.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 -- 172.21.15.59:0/2736131532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 msgr2=0x7f00a80caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/2736131532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 0x7f00a80caf40 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f00a8074850 tx=0x7f009c02a560 comp rx=0 tx=0).stop 2024-06-29T14:39:51.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 -- 172.21.15.59:0/2736131532 shutdown_connections 2024-06-29T14:39:51.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/2736131532 >> v2:172.21.15.153:3300/0 conn(0x7f00a80cb480 0x7f00a80cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/2736131532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 0x7f00a80caf40 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/2736131532 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00a80b52b0 0x7f00a80b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 -- 172.21.15.59:0/2736131532 >> 172.21.15.59:0/2736131532 conn(0x7f00a80b1ea0 msgr2=0x7f00a80b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:51.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 -- 172.21.15.59:0/2736131532 shutdown_connections 2024-06-29T14:39:51.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.243+0000 7f00ae5a1640 1 -- 172.21.15.59:0/2736131532 wait complete. 2024-06-29T14:39:51.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.245+0000 7f00ae5a1640 1 Processor -- start 2024-06-29T14:39:51.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.245+0000 7f00ae5a1640 1 -- start start 2024-06-29T14:39:51.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.246+0000 7f00ae5a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00a80b52b0 0x7f00a81489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:51.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.246+0000 7f00ae5a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a80cb480 0x7f00a815ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:51.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00ae5a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 0x7f00a815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:51.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00ae5a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f00a815fe20 con 0x7f00a815be90 2024-06-29T14:39:51.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00ae5a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00a815fca0 con 0x7f00a80cb480 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00ae5a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f00a815ffa0 con 0x7f00a80b52b0 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 0x7f00a815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 0x7f00a815ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44280/0 (socket says 172.21.15.59:44280) 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 -- 172.21.15.59:0/1107269664 learned_addr learned my addr 172.21.15.59:0/1107269664 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00a77fe640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a80cb480 0x7f00a815ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00a7fff640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00a80b52b0 0x7f00a81489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 -- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00a80b52b0 msgr2=0x7f00a81489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00a80b52b0 0x7f00a81489d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 -- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a80cb480 msgr2=0x7f00a815ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a80cb480 0x7f00a815ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 -- 172.21.15.59:0/1107269664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a8148f10 con 0x7f00a815be90 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00a77fe640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a80cb480 0x7f00a815ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00acb3e640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 0x7f00a815ef80 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f00a000cab0 tx=0x7f00a000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00a4ff9640 1 -- 172.21.15.59:0/1107269664 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0007d90 con 0x7f00a815be90 2024-06-29T14:39:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00a4ff9640 1 -- 172.21.15.59:0/1107269664 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00a000f560 con 0x7f00a815be90 2024-06-29T14:39:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a809b730 con 0x7f00a815be90 2024-06-29T14:39:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00a4ff9640 1 -- 172.21.15.59:0/1107269664 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a000f9b0 con 0x7f00a815be90 2024-06-29T14:39:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.247+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a809bcf0 con 0x7f00a815be90 2024-06-29T14:39:51.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.249+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a80cb890 con 0x7f00a815be90 2024-06-29T14:39:51.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.249+0000 7f00a4ff9640 1 -- 172.21.15.59:0/1107269664 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f00a000fb50 con 0x7f00a815be90 2024-06-29T14:39:51.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.250+0000 7f00a4ff9640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f00940763f0 0x7f0094078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:51.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.250+0000 7f00a7fff640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f00940763f0 0x7f0094078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:51.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.250+0000 7f00a4ff9640 1 -- 172.21.15.59:0/1107269664 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7f00a0098500 con 0x7f00a815be90 2024-06-29T14:39:51.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.251+0000 7f00a7fff640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f00940763f0 0x7f0094078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f009800b890 tx=0x7f0098005b10 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.255+0000 7f00a4ff9640 1 -- 172.21.15.59:0/1107269664 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f00a0014070 con 0x7f00a815be90 2024-06-29T14:39:51.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:51.522+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) -- 0x7f00a809bf30 con 0x7f00a815be90 2024-06-29T14:39:52.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.443+0000 7f00a4ff9640 1 -- 172.21.15.59:0/1107269664 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v692) ==== 151+0+0 (secure 0 0 0) 0x7f00a0061ab0 con 0x7f00a815be90 2024-06-29T14:39:52.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f00940763f0 msgr2=0x7f0094078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f00940763f0 0x7f0094078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f009800b890 tx=0x7f0098005b10 comp rx=0 tx=0).stop 2024-06-29T14:39:52.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 msgr2=0x7f00a815ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 0x7f00a815ef80 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f00a000cab0 tx=0x7f00a000cf70 comp rx=0 tx=0).stop 2024-06-29T14:39:52.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 shutdown_connections 2024-06-29T14:39:52.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f00940763f0 0x7f0094078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00a815be90 0x7f00a815ef80 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a80cb480 0x7f00a815ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 --2- 172.21.15.59:0/1107269664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00a80b52b0 0x7f00a81489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 >> 172.21.15.59:0/1107269664 conn(0x7f00a80b1ea0 msgr2=0x7f00a80cc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 shutdown_connections 2024-06-29T14:39:52.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.446+0000 7f00ae5a1640 1 -- 172.21.15.59:0/1107269664 wait complete. 2024-06-29T14:39:52.458 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:39:52.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.585+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/3238008046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 msgr2=0x7f0f380d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.585+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/3238008046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 0x7f0f380d1a20 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f0f20005fd0 tx=0x7f0f2002a5a0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.585+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/3238008046 shutdown_connections 2024-06-29T14:39:52.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.585+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/3238008046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 0x7f0f380d1a20 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.585+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/3238008046 >> v2:172.21.15.153:3300/0 conn(0x7f0f3808c190 0x7f0f380cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.585+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/3238008046 >> 172.21.15.59:0/3238008046 conn(0x7f0f380b5610 msgr2=0x7f0f380b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.585+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/3238008046 shutdown_connections 2024-06-29T14:39:52.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.585+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/3238008046 wait complete. 2024-06-29T14:39:52.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.587+0000 7f0f3d0c5640 1 Processor -- start 2024-06-29T14:39:52.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.587+0000 7f0f3d0c5640 1 -- start start 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3d0c5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f3808c190 0x7f0f381511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3d0c5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f380ce950 0x7f0f38151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3d0c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 0x7f0f38154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3d0c5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0f380b7f60 con 0x7f0f380cf5c0 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3d0c5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f380b7de0 con 0x7f0f3808c190 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3d0c5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0f380b80e0 con 0x7f0f380ce950 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 0x7f0f38154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 0x7f0f38154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44292/0 (socket says 172.21.15.59:44292) 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3759e640 1 -- 172.21.15.59:0/2994209820 learned_addr learned my addr 172.21.15.59:0/2994209820 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:52.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f36d9d640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f3808c190 0x7f0f381511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.588+0000 7f0f3659c640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f380ce950 0x7f0f38151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f3759e640 1 -- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f380ce950 msgr2=0x7f0f38151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f3759e640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f380ce950 0x7f0f38151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f3759e640 1 -- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f3808c190 msgr2=0x7f0f381511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f3759e640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f3808c190 0x7f0f381511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f3759e640 1 -- 172.21.15.59:0/2994209820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f381554b0 con 0x7f0f380cf5c0 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f3759e640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 0x7f0f38154670 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c730 tx=0x7f0f2800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f2f7fe640 1 -- 172.21.15.59:0/2994209820 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28003b70 con 0x7f0f380cf5c0 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f2f7fe640 1 -- 172.21.15.59:0/2994209820 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f28003d10 con 0x7f0f380cf5c0 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3814f200 con 0x7f0f380cf5c0 2024-06-29T14:39:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f2f7fe640 1 -- 172.21.15.59:0/2994209820 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28005330 con 0x7f0f380cf5c0 2024-06-29T14:39:52.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.589+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f3814f7c0 con 0x7f0f380cf5c0 2024-06-29T14:39:52.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.590+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f380abde0 con 0x7f0f380cf5c0 2024-06-29T14:39:52.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.591+0000 7f0f2f7fe640 1 -- 172.21.15.59:0/2994209820 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f0f280054d0 con 0x7f0f380cf5c0 2024-06-29T14:39:52.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.591+0000 7f0f2f7fe640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0f180763f0 0x7f0f18078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.591+0000 7f0f2f7fe640 1 -- 172.21.15.59:0/2994209820 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7f0f28097760 con 0x7f0f380cf5c0 2024-06-29T14:39:52.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.592+0000 7f0f36d9d640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0f180763f0 0x7f0f18078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.592+0000 7f0f36d9d640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0f180763f0 0x7f0f18078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0f30005fd0 tx=0x7f0f3000b2e0 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:52.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.596+0000 7f0f2f7fe640 1 -- 172.21.15.59:0/2994209820 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f28060740 con 0x7f0f380cf5c0 2024-06-29T14:39:52.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.786+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f0f380cf080 con 0x7f0f380cf5c0 2024-06-29T14:39:52.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.787+0000 7f0f2f7fe640 1 -- 172.21.15.59:0/2994209820 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v47) ==== 59+0+1998 (secure 0 0 0) 0x7f0f28064a40 con 0x7f0f380cf5c0 2024-06-29T14:39:52.785 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:39:52.785 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:39:52.785 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:39:52.785 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:39:52.785 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:39:52.786 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:39:52.787 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:39:52.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0f180763f0 msgr2=0x7f0f18078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0f180763f0 0x7f0f18078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0f30005fd0 tx=0x7f0f3000b2e0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 msgr2=0x7f0f38154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 0x7f0f38154670 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c730 tx=0x7f0f2800cb30 comp rx=0 tx=0).stop 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 shutdown_connections 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0f180763f0 0x7f0f18078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f380cf5c0 0x7f0f38154670 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f380ce950 0x7f0f38151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 --2- 172.21.15.59:0/2994209820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f3808c190 0x7f0f381511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 >> 172.21.15.59:0/2994209820 conn(0x7f0f380b5610 msgr2=0x7f0f3815e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 shutdown_connections 2024-06-29T14:39:52.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.789+0000 7f0f3d0c5640 1 -- 172.21.15.59:0/2994209820 wait complete. 2024-06-29T14:39:52.798 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:39:52.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.921+0000 7f129ef52640 1 -- 172.21.15.59:0/292701397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f129808c190 msgr2=0x7f12980cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.921+0000 7f129ef52640 1 --2- 172.21.15.59:0/292701397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f129808c190 0x7f12980cf080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f128c009830 tx=0x7f128c02a510 comp rx=0 tx=0).stop 2024-06-29T14:39:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.921+0000 7f129ef52640 1 -- 172.21.15.59:0/292701397 shutdown_connections 2024-06-29T14:39:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.921+0000 7f129ef52640 1 --2- 172.21.15.59:0/292701397 >> v2:172.21.15.153:3300/0 conn(0x7f12980cf5c0 0x7f12980d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.921+0000 7f129ef52640 1 --2- 172.21.15.59:0/292701397 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f129808c190 0x7f12980cf080 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.921+0000 7f129ef52640 1 -- 172.21.15.59:0/292701397 >> 172.21.15.59:0/292701397 conn(0x7f12980b5610 msgr2=0x7f12980b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.921+0000 7f129ef52640 1 -- 172.21.15.59:0/292701397 shutdown_connections 2024-06-29T14:39:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.921+0000 7f129ef52640 1 -- 172.21.15.59:0/292701397 wait complete. 2024-06-29T14:39:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.923+0000 7f129ef52640 1 Processor -- start 2024-06-29T14:39:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.923+0000 7f129ef52640 1 -- start start 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129ef52640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f129808c190 0x7f12980aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129ef52640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12980ce950 0x7f12981f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129ef52640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12980cf5c0 0x7f1298160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129ef52640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f12980b7fa0 con 0x7f12980cf5c0 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129ef52640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12980b7e20 con 0x7f12980ce950 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129ef52640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f12980b8120 con 0x7f129808c190 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129d4ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12980cf5c0 0x7f1298160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129d4ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12980cf5c0 0x7f1298160470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43908/0 (socket says 172.21.15.59:43908) 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.924+0000 7f129d4ef640 1 -- 172.21.15.59:0/1730727911 learned_addr learned my addr 172.21.15.59:0/1730727911 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:52.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129ccee640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f129808c190 0x7f12980aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f1297fff640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12980ce950 0x7f12981f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129d4ef640 1 -- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f129808c190 msgr2=0x7f12980aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129d4ef640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f129808c190 0x7f12980aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129d4ef640 1 -- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12980ce950 msgr2=0x7f12981f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129d4ef640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12980ce950 0x7f12981f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129d4ef640 1 -- 172.21.15.59:0/1730727911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12981f7f00 con 0x7f12980cf5c0 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129d4ef640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12980cf5c0 0x7f1298160470 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f1290002230 tx=0x7f129000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f12957fa640 1 -- 172.21.15.59:0/1730727911 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290016020 con 0x7f12980cf5c0 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f12957fa640 1 -- 172.21.15.59:0/1730727911 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f129000acc0 con 0x7f12980cf5c0 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12980ae940 con 0x7f12980cf5c0 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f12957fa640 1 -- 172.21.15.59:0/1730727911 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290012680 con 0x7f12980cf5c0 2024-06-29T14:39:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.925+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12980abff0 con 0x7f12980cf5c0 2024-06-29T14:39:52.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.926+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12980abb00 con 0x7f12980cf5c0 2024-06-29T14:39:52.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.927+0000 7f12957fa640 1 -- 172.21.15.59:0/1730727911 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f1290019020 con 0x7f12980cf5c0 2024-06-29T14:39:52.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.927+0000 7f12957fa640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f127c0763f0 0x7f127c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.927+0000 7f12957fa640 1 -- 172.21.15.59:0/1730727911 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7f1290097d00 con 0x7f12980cf5c0 2024-06-29T14:39:52.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.928+0000 7f129ccee640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f127c0763f0 0x7f127c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.928+0000 7f129ccee640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f127c0763f0 0x7f127c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f128c034000 tx=0x7f128c004730 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:52.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:52.932+0000 7f12957fa640 1 -- 172.21.15.59:0/1730727911 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1290020030 con 0x7f12980cf5c0 2024-06-29T14:39:53.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.121+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f12980cf080 con 0x7f12980cf5c0 2024-06-29T14:39:53.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.125+0000 7f12957fa640 1 -- 172.21.15.59:0/1730727911 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7f12981f7f00 con 0x7f12980cf5c0 2024-06-29T14:39:53.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f127c0763f0 msgr2=0x7f127c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f127c0763f0 0x7f127c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f128c034000 tx=0x7f128c004730 comp rx=0 tx=0).stop 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12980cf5c0 msgr2=0x7f1298160470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12980cf5c0 0x7f1298160470 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f1290002230 tx=0x7f129000cf70 comp rx=0 tx=0).stop 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 shutdown_connections 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f127c0763f0 0x7f127c078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12980cf5c0 0x7f1298160470 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12980ce950 0x7f12981f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 --2- 172.21.15.59:0/1730727911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f129808c190 0x7f12980aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 >> 172.21.15.59:0/1730727911 conn(0x7f12980b5610 msgr2=0x7f129806d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 shutdown_connections 2024-06-29T14:39:53.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.127+0000 7f129ef52640 1 -- 172.21.15.59:0/1730727911 wait complete. 2024-06-29T14:39:53.136 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:39:53.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:39:53.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:39:53.223 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:39:53.223 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:39:53.223 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:53.239 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:39:53.239 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:39:53.240 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:53.281 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:39:53.281 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:39:53.281 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:53.297 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:39:53.297 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:39:53.297 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:53.339 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:39:53.339 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:39:53.339 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:53.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.459+0000 7f8e0f523640 1 -- 172.21.15.59:0/1121511586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8df4004040 msgr2=0x7f8df4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.459+0000 7f8e0f523640 1 --2- 172.21.15.59:0/1121511586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8df4004040 0x7f8df4004800 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009b50 tx=0x7f8df4005830 comp rx=0 tx=0).stop 2024-06-29T14:39:53.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.459+0000 7f8e0dac0640 1 -- 172.21.15.59:0/1121511586 reap_dead start 2024-06-29T14:39:53.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.459+0000 7f8e0f523640 1 -- 172.21.15.59:0/1121511586 shutdown_connections 2024-06-29T14:39:53.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.459+0000 7f8e0f523640 1 -- 172.21.15.59:0/1121511586 >> 172.21.15.59:0/1121511586 conn(0x7f8e080b2ab0 msgr2=0x7f8e080b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:53.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.459+0000 7f8e0f523640 1 -- 172.21.15.59:0/1121511586 shutdown_connections 2024-06-29T14:39:53.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.459+0000 7f8e0f523640 1 -- 172.21.15.59:0/1121511586 wait complete. 2024-06-29T14:39:53.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.461+0000 7f8e0f523640 1 Processor -- start 2024-06-29T14:39:53.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.461+0000 7f8e0f523640 1 -- start start 2024-06-29T14:39:53.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.462+0000 7f8e0f523640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8df4004040 0x7f8e080caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.462+0000 7f8e0f523640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e080ccb10 0x7f8e0815d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.462+0000 7f8e0f523640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e0813ecd0 0x7f8e0815d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.462+0000 7f8e0f523640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8e080cf110 con 0x7f8e0813ecd0 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.462+0000 7f8e0f523640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8e080cef90 con 0x7f8e080ccb10 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.462+0000 7f8e0f523640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8e080cf290 con 0x7f8df4004040 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.462+0000 7f8e0dac0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e0813ecd0 0x7f8e0815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.462+0000 7f8e0dac0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e0813ecd0 0x7f8e0815d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43920/0 (socket says 172.21.15.59:43920) 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0dac0640 1 -- 172.21.15.59:0/3236185069 learned_addr learned my addr 172.21.15.59:0/3236185069 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0cabe640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e080ccb10 0x7f8e0815d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0d2bf640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8df4004040 0x7f8e080caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0dac0640 1 -- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8df4004040 msgr2=0x7f8e080caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0dac0640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8df4004040 0x7f8e080caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0dac0640 1 -- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e080ccb10 msgr2=0x7f8e0815d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0dac0640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e080ccb10 0x7f8e0815d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0dac0640 1 -- 172.21.15.59:0/3236185069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e080cba70 con 0x7f8e0813ecd0 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0dac0640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e0813ecd0 0x7f8e0815d640 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f8e00005fd0 tx=0x7f8e00009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8dfdffb640 1 -- 172.21.15.59:0/3236185069 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000fd70 con 0x7f8e0813ecd0 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8dfdffb640 1 -- 172.21.15.59:0/3236185069 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e00004b20 con 0x7f8e0813ecd0 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e080cbd50 con 0x7f8e0813ecd0 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.463+0000 7f8dfdffb640 1 -- 172.21.15.59:0/3236185069 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00005020 con 0x7f8e0813ecd0 2024-06-29T14:39:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.464+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e080cc310 con 0x7f8e0813ecd0 2024-06-29T14:39:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.464+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e080cb790 con 0x7f8e0813ecd0 2024-06-29T14:39:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.465+0000 7f8dfdffb640 1 -- 172.21.15.59:0/3236185069 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f8e00020020 con 0x7f8e0813ecd0 2024-06-29T14:39:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.466+0000 7f8dfdffb640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f8dec0763f0 0x7f8dec078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.466+0000 7f8dfdffb640 1 -- 172.21.15.59:0/3236185069 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7f8e00097ee0 con 0x7f8e0813ecd0 2024-06-29T14:39:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.466+0000 7f8e0d2bf640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f8dec0763f0 0x7f8dec078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.467+0000 7f8e0d2bf640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f8dec0763f0 0x7f8dec078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8e04001220 tx=0x7f8e040067a0 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:53.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.470+0000 7f8dfdffb640 1 -- 172.21.15.59:0/3236185069 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8e00060db0 con 0x7f8e0813ecd0 2024-06-29T14:39:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.703+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e080ca830 con 0x7f8e0813ecd0 2024-06-29T14:39:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.704+0000 7f8dfdffb640 1 -- 172.21.15.59:0/3236185069 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+3911 (secure 0 0 0) 0x7f8e000650b0 con 0x7f8e0813ecd0 2024-06-29T14:39:53.702 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:53.702 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":57,"btime":"2024-06-29T14:39:47:421961+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3208152037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3208152037},{"type":"v1","addr":"172.21.15.153:6836","nonce":3208152037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":6716,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/590918917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":590918917},{"type":"v1","addr":"172.21.15.153:6837","nonce":590918917}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":50},{"gid":7851,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/546301975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":546301975},{"type":"v1","addr":"172.21.15.59:6833","nonce":546301975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":8277,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/4026115176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":4026115176},{"type":"v1","addr":"172.21.15.59:6835","nonce":4026115176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[]} 2024-06-29T14:39:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 57 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f8dec0763f0 msgr2=0x7f8dec078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f8dec0763f0 0x7f8dec078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8e04001220 tx=0x7f8e040067a0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e0813ecd0 msgr2=0x7f8e0815d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e0813ecd0 0x7f8e0815d640 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f8e00005fd0 tx=0x7f8e00009f70 comp rx=0 tx=0).stop 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 shutdown_connections 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f8dec0763f0 0x7f8dec078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e0813ecd0 0x7f8e0815d640 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e080ccb10 0x7f8e0815d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 --2- 172.21.15.59:0/3236185069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8df4004040 0x7f8e080caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 >> 172.21.15.59:0/3236185069 conn(0x7f8e080b2ab0 msgr2=0x7f8e08141350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 shutdown_connections 2024-06-29T14:39:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.706+0000 7f8e0f523640 1 -- 172.21.15.59:0/3236185069 wait complete. 2024-06-29T14:39:53.716 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:39:53.716 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:39:53.716 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:39:53.716 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:39:53.717 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:39:53.717 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:39:53.717 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:39:53.717 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:39:53.717 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:39:53.717 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:39:53.717 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:39:53.717 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:39:53.718 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:39:53.718 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:39:53.718 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:39:53.718 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:39:53.727 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:39:53.727 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:39:53.728 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:39:53.728 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:39:53.729 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:39:53.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:39:53.730 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:39:53.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:39:53.771 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:39:53.772 DEBUG:teuthology.parallel:result is None 2024-06-29T14:39:53.772 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:39:53.773 DEBUG:teuthology.parallel:result is None 2024-06-29T14:39:53.774 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:39:53.774 DEBUG:teuthology.parallel:result is None 2024-06-29T14:39:53.774 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:39:53.775 DEBUG:teuthology.parallel:result is None 2024-06-29T14:39:53.775 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:39:53.861 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:39:53.862 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:39:53.866 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:39:53.867 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:39:53.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.902+0000 7fb59463f640 1 -- 172.21.15.59:0/230685960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0d1c30 msgr2=0x7fb58c0b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.902+0000 7fb59463f640 1 --2- 172.21.15.59:0/230685960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0d1c30 0x7fb58c0b5b30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb584005fd0 tx=0x7fb58402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.902+0000 7fb59463f640 1 -- 172.21.15.59:0/230685960 shutdown_connections 2024-06-29T14:39:53.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.902+0000 7fb59463f640 1 --2- 172.21.15.59:0/230685960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0d1c30 0x7fb58c0b5b30 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.902+0000 7fb59463f640 1 --2- 172.21.15.59:0/230685960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb58c0b2270 0x7fb58c15bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.902+0000 7fb59463f640 1 -- 172.21.15.59:0/230685960 >> 172.21.15.59:0/230685960 conn(0x7fb58c073e10 msgr2=0x7fb58c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:53.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.902+0000 7fb59463f640 1 -- 172.21.15.59:0/230685960 shutdown_connections 2024-06-29T14:39:53.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.902+0000 7fb59463f640 1 -- 172.21.15.59:0/230685960 wait complete. 2024-06-29T14:39:53.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.904+0000 7fb59463f640 1 Processor -- start 2024-06-29T14:39:53.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.904+0000 7fb59463f640 1 -- start start 2024-06-29T14:39:53.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb59463f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0b2270 0x7fb58c15f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb59463f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb58c0b2fa0 0x7fb58c159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb59463f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb58c0d1c30 0x7fb58c13c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb59463f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c0b6400 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb59463f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c0b6280 con 0x7fb58c0b2270 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb59463f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb58c0b6580 con 0x7fb58c0d1c30 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb58c0b2fa0 0x7fb58c159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb58c0b2fa0 0x7fb58c159cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43972/0 (socket says 172.21.15.59:43972) 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 -- 172.21.15.59:0/1187979452 learned_addr learned my addr 172.21.15.59:0/1187979452 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb5923db640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0b2270 0x7fb58c15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb592bdc640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb58c0d1c30 0x7fb58c13c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 -- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb58c0d1c30 msgr2=0x7fb58c13c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb58c0d1c30 0x7fb58c13c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 -- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0b2270 msgr2=0x7fb58c15f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0b2270 0x7fb58c15f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 -- 172.21.15.59:0/1187979452 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c13c770 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb5923db640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0b2270 0x7fb58c15f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb592bdc640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb58c0d1c30 0x7fb58c13c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb591bda640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb58c0b2fa0 0x7fb58c159cb0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c3c0 tx=0x7fb57c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb582ffd640 1 -- 172.21.15.59:0/1187979452 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c011790 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb582ffd640 1 -- 172.21.15.59:0/1187979452 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb57c017070 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb582ffd640 1 -- 172.21.15.59:0/1187979452 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c011d20 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb58c15a3e0 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.906+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb58c15a9a0 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.907+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb58c15f6c0 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.908+0000 7fb582ffd640 1 -- 172.21.15.59:0/1187979452 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fb57c01c460 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.909+0000 7fb582ffd640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb5700763f0 0x7fb570078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.909+0000 7fb582ffd640 1 -- 172.21.15.59:0/1187979452 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6109+0+0 (secure 0 0 0) 0x7fb57c0977a0 con 0x7fb58c0b2fa0 2024-06-29T14:39:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.909+0000 7fb5923db640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb5700763f0 0x7fb570078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.909+0000 7fb5923db640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb5700763f0 0x7fb570078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb588009ab0 tx=0x7fb588005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:53.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:53.913+0000 7fb582ffd640 1 -- 172.21.15.59:0/1187979452 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb58c15f6c0 con 0x7fb58c0b2fa0 2024-06-29T14:39:54.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.119+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb58c0ced70 con 0x7fb58c0b2fa0 2024-06-29T14:39:54.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.461+0000 7fb582ffd640 1 -- 172.21.15.59:0/1187979452 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v123) ==== 97+0+0 (secure 0 0 0) 0x7fb57c060e60 con 0x7fb58c0b2fa0 2024-06-29T14:39:54.460 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb5700763f0 msgr2=0x7fb570078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb5700763f0 0x7fb570078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb588009ab0 tx=0x7fb588005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb58c0b2fa0 msgr2=0x7fb58c159cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb58c0b2fa0 0x7fb58c159cb0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c3c0 tx=0x7fb57c00c880 comp rx=0 tx=0).stop 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 shutdown_connections 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb58c0d1c30 0x7fb58c13c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fb5700763f0 0x7fb570078900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb58c0b2fa0 0x7fb58c159cb0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 --2- 172.21.15.59:0/1187979452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb58c0b2270 0x7fb58c15f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 >> 172.21.15.59:0/1187979452 conn(0x7fb58c073e10 msgr2=0x7fb58c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.464+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 shutdown_connections 2024-06-29T14:39:54.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.465+0000 7fb59463f640 1 -- 172.21.15.59:0/1187979452 wait complete. 2024-06-29T14:39:54.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:39:54.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.598+0000 7fc2f69f5640 1 -- 172.21.15.59:0/2964772951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00ce950 msgr2=0x7fc2f00d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.598+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/2964772951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00ce950 0x7fc2f00d2130 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8005fd0 tx=0x7fc2e802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.598+0000 7fc2f69f5640 1 -- 172.21.15.59:0/2964772951 shutdown_connections 2024-06-29T14:39:54.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.598+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/2964772951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00ce950 0x7fc2f00d2130 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.598+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/2964772951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f008c190 0x7fc2f00cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.598+0000 7fc2f69f5640 1 -- 172.21.15.59:0/2964772951 >> 172.21.15.59:0/2964772951 conn(0x7fc2f00b5610 msgr2=0x7fc2f00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.598+0000 7fc2f69f5640 1 -- 172.21.15.59:0/2964772951 shutdown_connections 2024-06-29T14:39:54.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.598+0000 7fc2f69f5640 1 -- 172.21.15.59:0/2964772951 wait complete. 2024-06-29T14:39:54.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.600+0000 7fc2f69f5640 1 Processor -- start 2024-06-29T14:39:54.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.600+0000 7fc2f69f5640 1 -- start start 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.601+0000 7fc2f69f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f008c190 0x7fc2f01f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.601+0000 7fc2f69f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00ce950 0x7fc2f01f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.601+0000 7fc2f69f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f00cf5c0 0x7fc2f01f5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.601+0000 7fc2f69f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f00b7fa0 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.601+0000 7fc2f69f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f00b7e20 con 0x7fc2f00ce950 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.601+0000 7fc2f69f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f00b8120 con 0x7fc2f008c190 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f00cf5c0 0x7fc2f01f5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f00cf5c0 0x7fc2f01f5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43974/0 (socket says 172.21.15.59:43974) 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 -- 172.21.15.59:0/1162636107 learned_addr learned my addr 172.21.15.59:0/1162636107 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2effff640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f008c190 0x7fc2f01f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2ef7fe640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00ce950 0x7fc2f01f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 -- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f008c190 msgr2=0x7fc2f01f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f008c190 0x7fc2f01f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 -- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00ce950 msgr2=0x7fc2f01f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00ce950 0x7fc2f01f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 -- 172.21.15.59:0/1162636107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f01f5b70 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f4f92640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f00cf5c0 0x7fc2f01f5d60 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009cd0 tx=0x7fc2e80316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2ecff9640 1 -- 172.21.15.59:0/1162636107 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8031da0 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2ecff9640 1 -- 172.21.15.59:0/1162636107 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2e8003a60 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f01f6a00 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2ecff9640 1 -- 172.21.15.59:0/1162636107 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80330d0 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.602+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f01f6fc0 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.603+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f00cf080 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.604+0000 7fc2ecff9640 1 -- 172.21.15.59:0/1162636107 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fc2e8033270 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.605+0000 7fc2ecff9640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fc2dc076420 0x7fc2dc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.605+0000 7fc2ecff9640 1 -- 172.21.15.59:0/1162636107 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5249+0+0 (secure 0 0 0) 0x7fc2e80b8a10 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.605+0000 7fc2effff640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fc2dc076420 0x7fc2dc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.606+0000 7fc2effff640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fc2dc076420 0x7fc2dc078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009530 tx=0x7fc2e0009260 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:54.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.610+0000 7fc2ecff9640 1 -- 172.21.15.59:0/1162636107 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2e803f030 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.882+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc2f00ca130 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.883+0000 7fc2ecff9640 1 -- 172.21.15.59:0/1162636107 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) ==== 80+0+5304 (secure 0 0 0) 0x7fc2e8081cc0 con 0x7fc2f00cf5c0 2024-06-29T14:39:54.881 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:54.881 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:39:54.881 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T14:39:54.881 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.881 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:39:54.882 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.883 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.884 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.885 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.886 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.887 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T14:39:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T14:39:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:39:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T14:39:54.895 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fc2dc076420 msgr2=0x7fc2dc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fc2dc076420 0x7fc2dc078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009530 tx=0x7fc2e0009260 comp rx=0 tx=0).stop 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f00cf5c0 msgr2=0x7fc2f01f5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f00cf5c0 0x7fc2f01f5d60 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009cd0 tx=0x7fc2e80316f0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 shutdown_connections 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fc2dc076420 0x7fc2dc078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f00cf5c0 0x7fc2f01f5d60 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f00ce950 0x7fc2f01f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 --2- 172.21.15.59:0/1162636107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f008c190 0x7fc2f01f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 >> 172.21.15.59:0/1162636107 conn(0x7fc2f00b5610 msgr2=0x7fc2f006d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 shutdown_connections 2024-06-29T14:39:54.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:54.885+0000 7fc2f69f5640 1 -- 172.21.15.59:0/1162636107 wait complete. 2024-06-29T14:39:54.897 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:55.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.017+0000 7fa74212e640 1 -- 172.21.15.59:0/3384334713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa724004d40 msgr2=0x7fa724005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.017+0000 7fa74212e640 1 --2- 172.21.15.59:0/3384334713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa724004d40 0x7fa724005150 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fa734009ed0 tx=0x7fa724005830 comp rx=0 tx=0).stop 2024-06-29T14:39:55.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.017+0000 7fa73bfff640 1 -- 172.21.15.59:0/3384334713 reap_dead start 2024-06-29T14:39:55.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.017+0000 7fa74212e640 1 -- 172.21.15.59:0/3384334713 shutdown_connections 2024-06-29T14:39:55.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.017+0000 7fa74212e640 1 -- 172.21.15.59:0/3384334713 >> 172.21.15.59:0/3384334713 conn(0x7fa73c073e10 msgr2=0x7fa73c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.017+0000 7fa74212e640 1 -- 172.21.15.59:0/3384334713 shutdown_connections 2024-06-29T14:39:55.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.017+0000 7fa74212e640 1 -- 172.21.15.59:0/3384334713 wait complete. 2024-06-29T14:39:55.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.019+0000 7fa74212e640 1 Processor -- start 2024-06-29T14:39:55.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.019+0000 7fa74212e640 1 -- start start 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.020+0000 7fa74212e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa724004d40 0x7fa73c14e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.020+0000 7fa74212e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c14f5a0 0x7fa73c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.020+0000 7fa74212e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c15e2f0 0x7fa73c154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.020+0000 7fa74212e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa73c15c0c0 con 0x7fa73c14f5a0 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.020+0000 7fa74212e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa73c15bf40 con 0x7fa73c15e2f0 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.020+0000 7fa74212e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa73c15c240 con 0x7fa724004d40 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.020+0000 7fa73affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c14f5a0 0x7fa73c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c14f5a0 0x7fa73c15ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43998/0 (socket says 172.21.15.59:43998) 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73affd640 1 -- 172.21.15.59:0/4292477911 learned_addr learned my addr 172.21.15.59:0/4292477911 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:55.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73bfff640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c15e2f0 0x7fa73c154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73b7fe640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa724004d40 0x7fa73c14e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73affd640 1 -- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa724004d40 msgr2=0x7fa73c14e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73affd640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa724004d40 0x7fa73c14e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73affd640 1 -- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c15e2f0 msgr2=0x7fa73c154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73affd640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c15e2f0 0x7fa73c154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73affd640 1 -- 172.21.15.59:0/4292477911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73c15e700 con 0x7fa73c14f5a0 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa73affd640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c14f5a0 0x7fa73c15ddb0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fa730006600 tx=0x7fa7300039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa723fff640 1 -- 172.21.15.59:0/4292477911 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730015070 con 0x7fa73c14f5a0 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa723fff640 1 -- 172.21.15.59:0/4292477911 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa730011920 con 0x7fa73c14f5a0 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73c153890 con 0x7fa73c14f5a0 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa723fff640 1 -- 172.21.15.59:0/4292477911 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730011d70 con 0x7fa73c14f5a0 2024-06-29T14:39:55.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.021+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73c153e50 con 0x7fa73c14f5a0 2024-06-29T14:39:55.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.022+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa73c0b5b30 con 0x7fa73c14f5a0 2024-06-29T14:39:55.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.023+0000 7fa723fff640 1 -- 172.21.15.59:0/4292477911 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fa730018070 con 0x7fa73c14f5a0 2024-06-29T14:39:55.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.024+0000 7fa723fff640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa7280763f0 0x7fa728078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.024+0000 7fa723fff640 1 -- 172.21.15.59:0/4292477911 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5249+0+0 (secure 0 0 0) 0x7fa730098450 con 0x7fa73c14f5a0 2024-06-29T14:39:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.024+0000 7fa73b7fe640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa7280763f0 0x7fa728078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.025+0000 7fa73b7fe640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa7280763f0 0x7fa728078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009c50 tx=0x7fa72c005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.028+0000 7fa723fff640 1 -- 172.21.15.59:0/4292477911 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa73c15e700 con 0x7fa73c14f5a0 2024-06-29T14:39:55.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.221+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa73c0cf0f0 con 0x7fa73c14f5a0 2024-06-29T14:39:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.222+0000 7fa723fff640 1 -- 172.21.15.59:0/4292477911 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) ==== 74+0+13186 (secure 0 0 0) 0x7fa730061710 con 0x7fa73c14f5a0 2024-06-29T14:39:55.220 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:55.220 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":125,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:54.471353+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:55.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa7280763f0 msgr2=0x7fa728078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa7280763f0 0x7fa728078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009c50 tx=0x7fa72c005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:55.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c14f5a0 msgr2=0x7fa73c15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c14f5a0 0x7fa73c15ddb0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fa730006600 tx=0x7fa7300039c0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 shutdown_connections 2024-06-29T14:39:55.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa73c15e2f0 0x7fa73c154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa7280763f0 0x7fa728078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa73c14f5a0 0x7fa73c15ddb0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 --2- 172.21.15.59:0/4292477911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa724004d40 0x7fa73c14e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 >> 172.21.15.59:0/4292477911 conn(0x7fa73c073e10 msgr2=0x7fa73c15ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 shutdown_connections 2024-06-29T14:39:55.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.224+0000 7fa74212e640 1 -- 172.21.15.59:0/4292477911 wait complete. 2024-06-29T14:39:55.234 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:39:55.234 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:39:55.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.357+0000 7f3d0cffb640 1 -- 172.21.15.59:0/6079670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 msgr2=0x7f3cf4003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.357+0000 7f3d0cffb640 1 --2- 172.21.15.59:0/6079670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 0x7f3cf4003f10 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80016c0 tx=0x7f3cf4005830 comp rx=0 tx=0).stop 2024-06-29T14:39:55.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.357+0000 7f3d06d9d640 1 -- 172.21.15.59:0/6079670 reap_dead start 2024-06-29T14:39:55.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.358+0000 7f3d0cffb640 1 -- 172.21.15.59:0/6079670 shutdown_connections 2024-06-29T14:39:55.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.358+0000 7f3d0cffb640 1 -- 172.21.15.59:0/6079670 >> 172.21.15.59:0/6079670 conn(0x7f3d080b5610 msgr2=0x7f3d080b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.358+0000 7f3d0cffb640 1 -- 172.21.15.59:0/6079670 shutdown_connections 2024-06-29T14:39:55.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.358+0000 7f3d0cffb640 1 -- 172.21.15.59:0/6079670 wait complete. 2024-06-29T14:39:55.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.359+0000 7f3d0cffb640 1 Processor -- start 2024-06-29T14:39:55.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.360+0000 7f3d0cffb640 1 -- start start 2024-06-29T14:39:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0cffb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 0x7f3d08144a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0cffb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d080cd820 0x7f3d080cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0cffb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d0806c9a0 0x7f3d0806cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0cffb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08162c10 con 0x7f3cf4003aa0 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0cffb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08162610 con 0x7f3d0806c9a0 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0cffb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3d08162d90 con 0x7f3d080cd820 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 0x7f3d08144a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 0x7f3d08144a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44022/0 (socket says 172.21.15.59:44022) 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0659c640 1 -- 172.21.15.59:0/496418481 learned_addr learned my addr 172.21.15.59:0/496418481 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d06d9d640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d0806c9a0 0x7f3d0806cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3cfffff640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d080cd820 0x7f3d080cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0659c640 1 -- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d080cd820 msgr2=0x7f3d080cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0659c640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d080cd820 0x7f3d080cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0659c640 1 -- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d0806c9a0 msgr2=0x7f3d0806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0659c640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d0806c9a0 0x7f3d0806cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d0659c640 1 -- 172.21.15.59:0/496418481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d0806d2f0 con 0x7f3cf4003aa0 2024-06-29T14:39:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3d06d9d640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d0806c9a0 0x7f3d0806cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.361+0000 7f3cfffff640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d080cd820 0x7f3d080cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.362+0000 7f3d0659c640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 0x7f3d08144a30 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f3cf802a930 tx=0x7f3cf8006310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.362+0000 7f3cfeffd640 1 -- 172.21.15.59:0/496418481 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8041070 con 0x7f3cf4003aa0 2024-06-29T14:39:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.362+0000 7f3cfeffd640 1 -- 172.21.15.59:0/496418481 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cf8003af0 con 0x7f3cf4003aa0 2024-06-29T14:39:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.362+0000 7f3cfeffd640 1 -- 172.21.15.59:0/496418481 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8038070 con 0x7f3cf4003aa0 2024-06-29T14:39:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.362+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d080cc780 con 0x7f3cf4003aa0 2024-06-29T14:39:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.362+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d080ccd40 con 0x7f3cf4003aa0 2024-06-29T14:39:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.363+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d080cf080 con 0x7f3cf4003aa0 2024-06-29T14:39:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.364+0000 7f3cfeffd640 1 -- 172.21.15.59:0/496418481 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f3cf8044070 con 0x7f3cf4003aa0 2024-06-29T14:39:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.364+0000 7f3cfeffd640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f3ce80763f0 0x7f3ce8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.364+0000 7f3cfeffd640 1 -- 172.21.15.59:0/496418481 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5249+0+0 (secure 0 0 0) 0x7f3cf80b6ee0 con 0x7f3cf4003aa0 2024-06-29T14:39:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.364+0000 7f3cfffff640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f3ce80763f0 0x7f3ce8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.365+0000 7f3cfffff640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f3ce80763f0 0x7f3ce8078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d08069740 tx=0x7f3cf0005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.368+0000 7f3cfeffd640 1 -- 172.21.15.59:0/496418481 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3d0806d2f0 con 0x7f3cf4003aa0 2024-06-29T14:39:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.560+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3d080b4d80 con 0x7f3cf4003aa0 2024-06-29T14:39:55.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.560+0000 7f3cfeffd640 1 -- 172.21.15.59:0/496418481 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v125) ==== 93+0+10 (secure 0 0 0) 0x7f3cf8006500 con 0x7f3cf4003aa0 2024-06-29T14:39:55.559 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:39:55.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.562+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f3ce80763f0 msgr2=0x7f3ce8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.562+0000 7f3d0cffb640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f3ce80763f0 0x7f3ce8078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d08069740 tx=0x7f3cf0005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:55.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 msgr2=0x7f3d08144a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 0x7f3d08144a30 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f3cf802a930 tx=0x7f3cf8006310 comp rx=0 tx=0).stop 2024-06-29T14:39:55.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 shutdown_connections 2024-06-29T14:39:55.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d0806c9a0 0x7f3d0806cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d080cd820 0x7f3d080cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f3ce80763f0 0x7f3ce8078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 --2- 172.21.15.59:0/496418481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cf4003aa0 0x7f3d08144a30 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 >> 172.21.15.59:0/496418481 conn(0x7f3d080b5610 msgr2=0x7f3d081609c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 shutdown_connections 2024-06-29T14:39:55.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.563+0000 7f3d0cffb640 1 -- 172.21.15.59:0/496418481 wait complete. 2024-06-29T14:39:55.572 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:39:55.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.692+0000 7fe49a543640 1 -- 172.21.15.59:0/2264105505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 msgr2=0x7fe49415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.692+0000 7fe49a543640 1 --2- 172.21.15.59:0/2264105505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 0x7fe49415f9b0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe48c005fd0 tx=0x7fe48c02aa00 comp rx=0 tx=0).stop 2024-06-29T14:39:55.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.692+0000 7fe49a543640 1 -- 172.21.15.59:0/2264105505 shutdown_connections 2024-06-29T14:39:55.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.692+0000 7fe49a543640 1 --2- 172.21.15.59:0/2264105505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 0x7fe49415f9b0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.692+0000 7fe49a543640 1 --2- 172.21.15.59:0/2264105505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4940b40a0 0x7fe49415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.692+0000 7fe49a543640 1 -- 172.21.15.59:0/2264105505 >> 172.21.15.59:0/2264105505 conn(0x7fe4940d3ee0 msgr2=0x7fe4940d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.692+0000 7fe49a543640 1 -- 172.21.15.59:0/2264105505 shutdown_connections 2024-06-29T14:39:55.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.692+0000 7fe49a543640 1 -- 172.21.15.59:0/2264105505 wait complete. 2024-06-29T14:39:55.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.694+0000 7fe49a543640 1 Processor -- start 2024-06-29T14:39:55.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.694+0000 7fe49a543640 1 -- start start 2024-06-29T14:39:55.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe49a543640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4940d23c0 0x7fe49414e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe49a543640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 0x7fe49415baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe49a543640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe494156140 0x7fe494159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe49a543640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe494160280 con 0x7fe49415ee10 2024-06-29T14:39:55.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe49a543640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe494160100 con 0x7fe494156140 2024-06-29T14:39:55.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe49a543640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe494160400 con 0x7fe4940d23c0 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 0x7fe49415baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 0x7fe49415baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44030/0 (socket says 172.21.15.59:44030) 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 -- 172.21.15.59:0/70910334 learned_addr learned my addr 172.21.15.59:0/70910334 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe498ae0640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe494156140 0x7fe494159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe493fff640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4940d23c0 0x7fe49414e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 -- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4940d23c0 msgr2=0x7fe49414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4940d23c0 0x7fe49414e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 -- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe494156140 msgr2=0x7fe494159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe494156140 0x7fe494159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 -- 172.21.15.59:0/70910334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49414f4b0 con 0x7fe49415ee10 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe493fff640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4940d23c0 0x7fe49414e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe4937fe640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 0x7fe49415baf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c970 tx=0x7fe48800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.696+0000 7fe490ff9640 1 -- 172.21.15.59:0/70910334 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488007bb0 con 0x7fe49415ee10 2024-06-29T14:39:55.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.697+0000 7fe490ff9640 1 -- 172.21.15.59:0/70910334 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe488007e40 con 0x7fe49415ee10 2024-06-29T14:39:55.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.697+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4941550a0 con 0x7fe49415ee10 2024-06-29T14:39:55.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.697+0000 7fe490ff9640 1 -- 172.21.15.59:0/70910334 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488005280 con 0x7fe49415ee10 2024-06-29T14:39:55.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.697+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe494155660 con 0x7fe49415ee10 2024-06-29T14:39:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.698+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe49415e8d0 con 0x7fe49415ee10 2024-06-29T14:39:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.698+0000 7fe490ff9640 1 -- 172.21.15.59:0/70910334 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fe488005420 con 0x7fe49415ee10 2024-06-29T14:39:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.699+0000 7fe490ff9640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fe4800763f0 0x7fe480078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.699+0000 7fe490ff9640 1 -- 172.21.15.59:0/70910334 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5249+0+0 (secure 0 0 0) 0x7fe488097cb0 con 0x7fe49415ee10 2024-06-29T14:39:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.699+0000 7fe493fff640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fe4800763f0 0x7fe480078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.700+0000 7fe493fff640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fe4800763f0 0x7fe480078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe484006fd0 tx=0x7fe484008040 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.703+0000 7fe490ff9640 1 -- 172.21.15.59:0/70910334 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe488014030 con 0x7fe49415ee10 2024-06-29T14:39:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.895+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe4940d2950 con 0x7fe49415ee10 2024-06-29T14:39:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.896+0000 7fe490ff9640 1 -- 172.21.15.59:0/70910334 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v125) ==== 84+0+65 (secure 0 0 0) 0x7fe488060f10 con 0x7fe49415ee10 2024-06-29T14:39:55.894 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:55.894 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:39:55.894 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:39:55.895 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:39:55.895 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:39:55.895 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:39:55.895 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:39:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fe4800763f0 msgr2=0x7fe480078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fe4800763f0 0x7fe480078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe484006fd0 tx=0x7fe484008040 comp rx=0 tx=0).stop 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 msgr2=0x7fe49415baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 0x7fe49415baf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c970 tx=0x7fe48800ce30 comp rx=0 tx=0).stop 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 shutdown_connections 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe494156140 0x7fe494159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fe4800763f0 0x7fe480078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe49415ee10 0x7fe49415baf0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 --2- 172.21.15.59:0/70910334 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4940d23c0 0x7fe49414e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 >> 172.21.15.59:0/70910334 conn(0x7fe4940d3ee0 msgr2=0x7fe4941608d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 shutdown_connections 2024-06-29T14:39:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:55.898+0000 7fe49a543640 1 -- 172.21.15.59:0/70910334 wait complete. 2024-06-29T14:39:55.908 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:39:55.908 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:39:56.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.031+0000 7f1a400e6640 1 -- 172.21.15.59:0/4177341107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38145650 msgr2=0x7f1a380b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.032+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4177341107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38145650 0x7f1a380b5cb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009830 tx=0x7f1a3002a510 comp rx=0 tx=0).stop 2024-06-29T14:39:56.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.032+0000 7f1a400e6640 1 -- 172.21.15.59:0/4177341107 shutdown_connections 2024-06-29T14:39:56.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.032+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4177341107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a3815ff00 0x7f1a3815d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.032+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4177341107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38145650 0x7f1a380b5cb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.032+0000 7f1a400e6640 1 -- 172.21.15.59:0/4177341107 >> 172.21.15.59:0/4177341107 conn(0x7f1a380b6ed0 msgr2=0x7f1a380b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:56.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.032+0000 7f1a400e6640 1 -- 172.21.15.59:0/4177341107 shutdown_connections 2024-06-29T14:39:56.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.032+0000 7f1a400e6640 1 -- 172.21.15.59:0/4177341107 wait complete. 2024-06-29T14:39:56.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.033+0000 7f1a400e6640 1 Processor -- start 2024-06-29T14:39:56.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.033+0000 7f1a400e6640 1 -- start start 2024-06-29T14:39:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a400e6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a380d1130 0x7f1a380ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a400e6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a38145650 0x7f1a38143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a400e6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38144ed0 0x7f1a381440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a400e6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3815e3c0 con 0x7f1a380d1130 2024-06-29T14:39:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a400e6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3815e240 con 0x7f1a38144ed0 2024-06-29T14:39:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a400e6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3815e540 con 0x7f1a38145650 2024-06-29T14:39:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a380d1130 0x7f1a380ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a380d1130 0x7f1a380ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44034/0 (socket says 172.21.15.59:44034) 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 -- 172.21.15.59:0/4259762781 learned_addr learned my addr 172.21.15.59:0/4259762781 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3d681640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a38145650 0x7f1a38143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3e683640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38144ed0 0x7f1a381440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 -- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a38145650 msgr2=0x7f1a38143bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a38145650 0x7f1a38143bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 -- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38144ed0 msgr2=0x7f1a381440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38144ed0 0x7f1a381440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 -- 172.21.15.59:0/4259762781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a380ce7c0 con 0x7f1a380d1130 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.035+0000 7f1a3de82640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a380d1130 0x7f1a380ce280 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009950 tx=0x7f1a30033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.036+0000 7f1a367fc640 1 -- 172.21.15.59:0/4259762781 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30031d60 con 0x7f1a380d1130 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.036+0000 7f1a367fc640 1 -- 172.21.15.59:0/4259762781 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a30033e10 con 0x7f1a380d1130 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.036+0000 7f1a367fc640 1 -- 172.21.15.59:0/4259762781 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30005260 con 0x7f1a380d1130 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.036+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a381f2c60 con 0x7f1a380d1130 2024-06-29T14:39:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.036+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a381f3220 con 0x7f1a380d1130 2024-06-29T14:39:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.037+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a3815ea60 con 0x7f1a380d1130 2024-06-29T14:39:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.037+0000 7f1a367fc640 1 -- 172.21.15.59:0/4259762781 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f1a30005400 con 0x7f1a380d1130 2024-06-29T14:39:56.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.038+0000 7f1a367fc640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1a24076420 0x7f1a24078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.038+0000 7f1a367fc640 1 -- 172.21.15.59:0/4259762781 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5249+0+0 (secure 0 0 0) 0x7f1a300b8750 con 0x7f1a380d1130 2024-06-29T14:39:56.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.038+0000 7f1a3d681640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1a24076420 0x7f1a24078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.039+0000 7f1a3d681640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1a24076420 0x7f1a24078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1a380c8e20 tx=0x7f1a2c009040 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.042+0000 7f1a367fc640 1 -- 172.21.15.59:0/4259762781 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a380ce7c0 con 0x7f1a380d1130 2024-06-29T14:39:56.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.237+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1a380b7530 con 0x7f1a380d1130 2024-06-29T14:39:56.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.471+0000 7f1a367fc640 1 -- 172.21.15.59:0/4259762781 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v126) ==== 138+0+0 (secure 0 0 0) 0x7f1a3003f030 con 0x7f1a380d1130 2024-06-29T14:39:56.470 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1a24076420 msgr2=0x7f1a24078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1a24076420 0x7f1a24078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1a380c8e20 tx=0x7f1a2c009040 comp rx=0 tx=0).stop 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a380d1130 msgr2=0x7f1a380ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a380d1130 0x7f1a380ce280 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009950 tx=0x7f1a30033c20 comp rx=0 tx=0).stop 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 shutdown_connections 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a38144ed0 0x7f1a381440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a38145650 0x7f1a38143bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1a24076420 0x7f1a24078930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 --2- 172.21.15.59:0/4259762781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a380d1130 0x7f1a380ce280 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 >> 172.21.15.59:0/4259762781 conn(0x7f1a380b6ed0 msgr2=0x7f1a38140150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 shutdown_connections 2024-06-29T14:39:56.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.474+0000 7f1a400e6640 1 -- 172.21.15.59:0/4259762781 wait complete. 2024-06-29T14:39:56.487 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:39:56.581 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:56.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:56.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.615+0000 7efdd8b2e640 1 -- 172.21.15.59:0/1592851234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd4073de0 msgr2=0x7efdd40741f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.615+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/1592851234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd4073de0 0x7efdd40741f0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7efdd4072510 tx=0x7efdbc02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.615+0000 7efdd8b2e640 1 -- 172.21.15.59:0/1592851234 shutdown_connections 2024-06-29T14:39:56.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.615+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/1592851234 >> v2:172.21.15.153:3300/0 conn(0x7efdd40b5500 0x7efdd40b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.615+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/1592851234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd4073de0 0x7efdd40741f0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.615+0000 7efdd8b2e640 1 -- 172.21.15.59:0/1592851234 >> 172.21.15.59:0/1592851234 conn(0x7efdd40cd870 msgr2=0x7efdd40cdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:56.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.615+0000 7efdd8b2e640 1 -- 172.21.15.59:0/1592851234 shutdown_connections 2024-06-29T14:39:56.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.615+0000 7efdd8b2e640 1 -- 172.21.15.59:0/1592851234 wait complete. 2024-06-29T14:39:56.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.617+0000 7efdd8b2e640 1 Processor -- start 2024-06-29T14:39:56.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.617+0000 7efdd8b2e640 1 -- start start 2024-06-29T14:39:56.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd8b2e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd4073de0 0x7efdd415a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd8b2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b5500 0x7efdd406de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd8b2e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd415bb50 0x7efdd414fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd8b2e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efdd413df40 con 0x7efdd40b5500 2024-06-29T14:39:56.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd8b2e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efdd413ddc0 con 0x7efdd4073de0 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd8b2e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efdd415bf60 con 0x7efdd415bb50 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b5500 0x7efdd406de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b5500 0x7efdd406de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44056/0 (socket says 172.21.15.59:44056) 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 -- 172.21.15.59:0/2881266639 learned_addr learned my addr 172.21.15.59:0/2881266639 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd259c640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd4073de0 0x7efdd415a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd2d9d640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd415bb50 0x7efdd414fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 -- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd415bb50 msgr2=0x7efdd414fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd415bb50 0x7efdd414fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 -- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd4073de0 msgr2=0x7efdd415a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd4073de0 0x7efdd415a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 -- 172.21.15.59:0/2881266639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd406e630 con 0x7efdd40b5500 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.619+0000 7efdd1d9b640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b5500 0x7efdd406de00 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7efdbc02aa90 tx=0x7efdbc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.620+0000 7efdcaffd640 1 -- 172.21.15.59:0/2881266639 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc005c90 con 0x7efdd40b5500 2024-06-29T14:39:56.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.620+0000 7efdcaffd640 1 -- 172.21.15.59:0/2881266639 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efdbc031e50 con 0x7efdd40b5500 2024-06-29T14:39:56.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.620+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd406e910 con 0x7efdd40b5500 2024-06-29T14:39:56.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.620+0000 7efdcaffd640 1 -- 172.21.15.59:0/2881266639 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc005240 con 0x7efdd40b5500 2024-06-29T14:39:56.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.620+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd4150750 con 0x7efdd40b5500 2024-06-29T14:39:56.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.621+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdd40b4560 con 0x7efdd40b5500 2024-06-29T14:39:56.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.622+0000 7efdcaffd640 1 -- 172.21.15.59:0/2881266639 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7efdbc0053e0 con 0x7efdd40b5500 2024-06-29T14:39:56.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.623+0000 7efdcaffd640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7efdb40763f0 0x7efdb4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.623+0000 7efdcaffd640 1 -- 172.21.15.59:0/2881266639 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5619+0+0 (secure 0 0 0) 0x7efdbc0b8dd0 con 0x7efdd40b5500 2024-06-29T14:39:56.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.623+0000 7efdd259c640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7efdb40763f0 0x7efdb4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.623+0000 7efdd259c640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7efdb40763f0 0x7efdb4078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efdc4007fd0 tx=0x7efdc4009040 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:56.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.627+0000 7efdcaffd640 1 -- 172.21.15.59:0/2881266639 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efdd40b4560 con 0x7efdd40b5500 2024-06-29T14:39:56.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:56.819+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7efdd406e3c0 con 0x7efdd40b5500 2024-06-29T14:39:57.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.482+0000 7efdcaffd640 1 -- 172.21.15.59:0/2881266639 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v127) ==== 170+0+0 (secure 0 0 0) 0x7efdbc03f030 con 0x7efdd40b5500 2024-06-29T14:39:57.480 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:39:57.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7efdb40763f0 msgr2=0x7efdb4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7efdb40763f0 0x7efdb4078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efdc4007fd0 tx=0x7efdc4009040 comp rx=0 tx=0).stop 2024-06-29T14:39:57.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b5500 msgr2=0x7efdd406de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b5500 0x7efdd406de00 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7efdbc02aa90 tx=0x7efdbc033c20 comp rx=0 tx=0).stop 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 shutdown_connections 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdd415bb50 0x7efdd414fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7efdb40763f0 0x7efdb4078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdd40b5500 0x7efdd406de00 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 --2- 172.21.15.59:0/2881266639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdd4073de0 0x7efdd415a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 >> 172.21.15.59:0/2881266639 conn(0x7efdd40cd870 msgr2=0x7efdd40b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.486+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 shutdown_connections 2024-06-29T14:39:57.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.487+0000 7efdd8b2e640 1 -- 172.21.15.59:0/2881266639 wait complete. 2024-06-29T14:39:57.508 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:39:57.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.637+0000 7f961d738640 1 -- 172.21.15.59:0/1367881107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9604005350 msgr2=0x7f960400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.637+0000 7f961d738640 1 --2- 172.21.15.59:0/1367881107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9604005350 0x7f960400b9c0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f96180b55a0 tx=0x7f9604005830 comp rx=0 tx=0).stop 2024-06-29T14:39:57.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.637+0000 7f961d738640 1 -- 172.21.15.59:0/1367881107 shutdown_connections 2024-06-29T14:39:57.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.637+0000 7f961d738640 1 --2- 172.21.15.59:0/1367881107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9604005350 0x7f960400b9c0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.637+0000 7f961d738640 1 -- 172.21.15.59:0/1367881107 >> 172.21.15.59:0/1367881107 conn(0x7f961806c950 msgr2=0x7f9618073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:57.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.637+0000 7f961d738640 1 -- 172.21.15.59:0/1367881107 shutdown_connections 2024-06-29T14:39:57.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.637+0000 7f961d738640 1 -- 172.21.15.59:0/1367881107 wait complete. 2024-06-29T14:39:57.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.639+0000 7f961d738640 1 Processor -- start 2024-06-29T14:39:57.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.639+0000 7f961d738640 1 -- start start 2024-06-29T14:39:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.640+0000 7f961d738640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9604005350 0x7f961815b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.640+0000 7f961d738640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96180acd50 0x7f961815e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.640+0000 7f961d738640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961815e640 0x7f96181f6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.640+0000 7f961d738640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f96180cf9f0 con 0x7f961815e640 2024-06-29T14:39:57.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.640+0000 7f961d738640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96180cf870 con 0x7f96180acd50 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f961d738640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f96180cfb70 con 0x7f9604005350 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961815e640 0x7f96181f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961815e640 0x7f96181f6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44072/0 (socket says 172.21.15.59:44072) 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 -- 172.21.15.59:0/2245412743 learned_addr learned my addr 172.21.15.59:0/2245412743 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f960e7fc640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96180acd50 0x7f961815e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f9616ffd640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9604005350 0x7f961815b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 -- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9604005350 msgr2=0x7f961815b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9604005350 0x7f961815b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 -- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96180acd50 msgr2=0x7f961815e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96180acd50 0x7f961815e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 -- 172.21.15.59:0/2245412743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961815be60 con 0x7f961815e640 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f960e7fc640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96180acd50 0x7f961815e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:57.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f96177fe640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961815e640 0x7f96181f6110 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f961002b410 tx=0x7f9610004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:57.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f960ffff640 1 -- 172.21.15.59:0/2245412743 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610039070 con 0x7f961815e640 2024-06-29T14:39:57.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f960ffff640 1 -- 172.21.15.59:0/2245412743 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f961002bc60 con 0x7f961815e640 2024-06-29T14:39:57.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f960ffff640 1 -- 172.21.15.59:0/2245412743 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610006250 con 0x7f961815e640 2024-06-29T14:39:57.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f961815ea50 con 0x7f961815e640 2024-06-29T14:39:57.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.641+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96180a9e80 con 0x7f961815e640 2024-06-29T14:39:57.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.642+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96180cfd50 con 0x7f961815e640 2024-06-29T14:39:57.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.644+0000 7f960ffff640 1 -- 172.21.15.59:0/2245412743 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f96100063f0 con 0x7f961815e640 2024-06-29T14:39:57.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.644+0000 7f960ffff640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f95f8076420 0x7f95f8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:57.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.644+0000 7f960ffff640 1 -- 172.21.15.59:0/2245412743 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 6001+0+0 (secure 0 0 0) 0x7f96100b81f0 con 0x7f961815e640 2024-06-29T14:39:57.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.644+0000 7f9616ffd640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f95f8076420 0x7f95f8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:57.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.645+0000 7f9616ffd640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f95f8076420 0x7f95f8078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9608009a20 tx=0x7f9608005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:57.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.648+0000 7f960ffff640 1 -- 172.21.15.59:0/2245412743 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f961815ea50 con 0x7f961815e640 2024-06-29T14:39:57.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:57.884+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f96181417b0 con 0x7f961815e640 2024-06-29T14:39:58.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.496+0000 7f960ffff640 1 -- 172.21.15.59:0/2245412743 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v60) ==== 174+0+0 (secure 0 0 0) 0x7f96100819f0 con 0x7f961815e640 2024-06-29T14:39:58.500 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 15 and data pool 16 2024-06-29T14:39:58.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.509+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f95f8076420 msgr2=0x7f95f8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.509+0000 7f961d738640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f95f8076420 0x7f95f8078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9608009a20 tx=0x7f9608005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:58.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.509+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961815e640 msgr2=0x7f96181f6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.509+0000 7f961d738640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961815e640 0x7f96181f6110 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f961002b410 tx=0x7f9610004bb0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.509+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 shutdown_connections 2024-06-29T14:39:58.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.509+0000 7f961d738640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f95f8076420 0x7f95f8078930 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.510+0000 7f961d738640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961815e640 0x7f96181f6110 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.510+0000 7f961d738640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96180acd50 0x7f961815e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.510+0000 7f961d738640 1 --2- 172.21.15.59:0/2245412743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9604005350 0x7f961815b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.510+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 >> 172.21.15.59:0/2245412743 conn(0x7f961806c950 msgr2=0x7f9618140c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:58.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.511+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 shutdown_connections 2024-06-29T14:39:58.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.512+0000 7f961d738640 1 -- 172.21.15.59:0/2245412743 wait complete. 2024-06-29T14:39:58.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:58.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.654+0000 7f2f850e7640 1 -- 172.21.15.59:0/2716069122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 msgr2=0x7f2f8015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.654+0000 7f2f850e7640 1 --2- 172.21.15.59:0/2716069122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 0x7f2f8015f9b0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2f800d15d0 tx=0x7f2f7802a530 comp rx=0 tx=0).stop 2024-06-29T14:39:58.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.654+0000 7f2f850e7640 1 -- 172.21.15.59:0/2716069122 shutdown_connections 2024-06-29T14:39:58.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.654+0000 7f2f850e7640 1 --2- 172.21.15.59:0/2716069122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 0x7f2f8015f9b0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.654+0000 7f2f850e7640 1 --2- 172.21.15.59:0/2716069122 >> v2:172.21.15.153:3300/0 conn(0x7f2f800d23c0 0x7f2f800b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.654+0000 7f2f850e7640 1 -- 172.21.15.59:0/2716069122 >> 172.21.15.59:0/2716069122 conn(0x7f2f800d3ee0 msgr2=0x7f2f800d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:58.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.654+0000 7f2f850e7640 1 -- 172.21.15.59:0/2716069122 shutdown_connections 2024-06-29T14:39:58.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.654+0000 7f2f850e7640 1 -- 172.21.15.59:0/2716069122 wait complete. 2024-06-29T14:39:58.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.656+0000 7f2f850e7640 1 Processor -- start 2024-06-29T14:39:58.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.656+0000 7f2f850e7640 1 -- start start 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f850e7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f800b3ed0 0x7f2f80153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f850e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f800d23c0 0x7f2f80159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f850e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 0x7f2f800b8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f850e7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2f80160280 con 0x7f2f8015efe0 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f850e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2f80160100 con 0x7f2f800d23c0 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f850e7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2f80160400 con 0x7f2f800b3ed0 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 0x7f2f800b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 0x7f2f800b8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44096/0 (socket says 172.21.15.59:44096) 2024-06-29T14:39:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 -- 172.21.15.59:0/1548620193 learned_addr learned my addr 172.21.15.59:0/1548620193 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7ed9d640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f800b3ed0 0x7f2f80153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7e59c640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f800d23c0 0x7f2f80159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 -- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f800b3ed0 msgr2=0x7f2f80153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f800b3ed0 0x7f2f80153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 -- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f800d23c0 msgr2=0x7f2f80159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f800d23c0 0x7f2f80159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 -- 172.21.15.59:0/1548620193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f80153860 con 0x7f2f8015efe0 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f7f59e640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 0x7f2f800b8120 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f2f78009920 tx=0x7f2f78033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f777fe640 1 -- 172.21.15.59:0/1548620193 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f78031b60 con 0x7f2f8015efe0 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.658+0000 7f2f777fe640 1 -- 172.21.15.59:0/1548620193 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f78031e10 con 0x7f2f8015efe0 2024-06-29T14:39:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.659+0000 7f2f777fe640 1 -- 172.21.15.59:0/1548620193 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f78005260 con 0x7f2f8015efe0 2024-06-29T14:39:58.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.659+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f80159d70 con 0x7f2f8015efe0 2024-06-29T14:39:58.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.659+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f8015cb10 con 0x7f2f8015efe0 2024-06-29T14:39:58.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.659+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f80159ae0 con 0x7f2f8015efe0 2024-06-29T14:39:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.660+0000 7f2f777fe640 1 -- 172.21.15.59:0/1548620193 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f2f78005400 con 0x7f2f8015efe0 2024-06-29T14:39:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.661+0000 7f2f777fe640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f2f60076420 0x7f2f60078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.661+0000 7f2f777fe640 1 -- 172.21.15.59:0/1548620193 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6101+0+0 (secure 0 0 0) 0x7f2f780b8d10 con 0x7f2f8015efe0 2024-06-29T14:39:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.661+0000 7f2f7ed9d640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f2f60076420 0x7f2f60078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.662+0000 7f2f7ed9d640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f2f60076420 0x7f2f60078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2f800d2e40 tx=0x7f2f700095d0 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:58.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.665+0000 7f2f777fe640 1 -- 172.21.15.59:0/1548620193 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2f780be020 con 0x7f2f8015efe0 2024-06-29T14:39:58.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.857+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f800b3990 con 0x7f2f8015efe0 2024-06-29T14:39:58.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.858+0000 7f2f777fe640 1 -- 172.21.15.59:0/1548620193 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+16909 (secure 0 0 0) 0x7f2f7803f030 con 0x7f2f8015efe0 2024-06-29T14:39:58.857 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:58.857 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":128,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:57.885798+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:56.243419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.825874+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:58.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f2f60076420 msgr2=0x7f2f60078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f2f60076420 0x7f2f60078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2f800d2e40 tx=0x7f2f700095d0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 msgr2=0x7f2f800b8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 0x7f2f800b8120 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f2f78009920 tx=0x7f2f78033c10 comp rx=0 tx=0).stop 2024-06-29T14:39:58.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 shutdown_connections 2024-06-29T14:39:58.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f2f60076420 0x7f2f60078930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f8015efe0 0x7f2f800b8120 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f800d23c0 0x7f2f80159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 --2- 172.21.15.59:0/1548620193 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f800b3ed0 0x7f2f80153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 >> 172.21.15.59:0/1548620193 conn(0x7f2f800d3ee0 msgr2=0x7f2f800d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:58.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.860+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 shutdown_connections 2024-06-29T14:39:58.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.861+0000 7f2f850e7640 1 -- 172.21.15.59:0/1548620193 wait complete. 2024-06-29T14:39:58.870 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.990+0000 7f98bc03c640 1 -- 172.21.15.59:0/1384907958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b415e0a0 msgr2=0x7f98b4161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.990+0000 7f98bc03c640 1 --2- 172.21.15.59:0/1384907958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b415e0a0 0x7f98b4161a80 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f98b408c200 tx=0x7f98a802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.990+0000 7f98bc03c640 1 -- 172.21.15.59:0/1384907958 shutdown_connections 2024-06-29T14:39:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.990+0000 7f98bc03c640 1 --2- 172.21.15.59:0/1384907958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b415e0a0 0x7f98b4161a80 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.990+0000 7f98bc03c640 1 --2- 172.21.15.59:0/1384907958 >> v2:172.21.15.153:3300/0 conn(0x7f98b4162770 0x7f98b40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.990+0000 7f98bc03c640 1 -- 172.21.15.59:0/1384907958 >> 172.21.15.59:0/1384907958 conn(0x7f98b40d45b0 msgr2=0x7f98b40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.990+0000 7f98bc03c640 1 -- 172.21.15.59:0/1384907958 shutdown_connections 2024-06-29T14:39:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.990+0000 7f98bc03c640 1 -- 172.21.15.59:0/1384907958 wait complete. 2024-06-29T14:39:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.992+0000 7f98bc03c640 1 Processor -- start 2024-06-29T14:39:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.992+0000 7f98bc03c640 1 -- start start 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.993+0000 7f98bc03c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b415e0a0 0x7f98b4160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.993+0000 7f98bc03c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98b4161fc0 0x7f98b406c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.993+0000 7f98bc03c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b4162770 0x7f98b406c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.993+0000 7f98bc03c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f98b40d2d80 con 0x7f98b4162770 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.993+0000 7f98bc03c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98b40b4b00 con 0x7f98b415e0a0 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.993+0000 7f98bc03c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f98b40d2f00 con 0x7f98b4161fc0 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b4162770 0x7f98b406c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b4162770 0x7f98b406c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44104/0 (socket says 172.21.15.59:44104) 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 -- 172.21.15.59:0/2178853331 learned_addr learned my addr 172.21.15.59:0/2178853331 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98b9dd8640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b415e0a0 0x7f98b4160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98b95d7640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98b4161fc0 0x7f98b406c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 -- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98b4161fc0 msgr2=0x7f98b406c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98b4161fc0 0x7f98b406c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 -- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b415e0a0 msgr2=0x7f98b4160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b415e0a0 0x7f98b4160ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 -- 172.21.15.59:0/2178853331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b41f4d80 con 0x7f98b4162770 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98ba5d9640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b4162770 0x7f98b406c720 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00c3c0 tx=0x7f98ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98b27fc640 1 -- 172.21.15.59:0/2178853331 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac006d60 con 0x7f98b4162770 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98b27fc640 1 -- 172.21.15.59:0/2178853331 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98ac007050 con 0x7f98b4162770 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b41f5060 con 0x7f98b4162770 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98b27fc640 1 -- 172.21.15.59:0/2178853331 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0074c0 con 0x7f98b4162770 2024-06-29T14:39:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.994+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b41f5620 con 0x7f98b4162770 2024-06-29T14:39:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.995+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b40d3420 con 0x7f98b4162770 2024-06-29T14:39:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.996+0000 7f98b27fc640 1 -- 172.21.15.59:0/2178853331 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f98ac01b400 con 0x7f98b4162770 2024-06-29T14:39:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.997+0000 7f98b27fc640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f98a0076420 0x7f98a0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.997+0000 7f98b27fc640 1 -- 172.21.15.59:0/2178853331 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6101+0+0 (secure 0 0 0) 0x7f98ac0975c0 con 0x7f98b4162770 2024-06-29T14:39:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.997+0000 7f98b9dd8640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f98a0076420 0x7f98a0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:58.998+0000 7f98b9dd8640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f98a0076420 0x7f98a0078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f98a4005fd0 tx=0x7f98a40095e0 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:58.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.002+0000 7f98b27fc640 1 -- 172.21.15.59:0/2178853331 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f98b41623d0 con 0x7f98b4162770 2024-06-29T14:39:59.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.197+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98b4161420 con 0x7f98b4162770 2024-06-29T14:39:59.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.198+0000 7f98b27fc640 1 -- 172.21.15.59:0/2178853331 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+16909 (secure 0 0 0) 0x7f98ac0605a0 con 0x7f98b4162770 2024-06-29T14:39:59.197 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:39:59.197 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":128,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:57.885798+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:56.243419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.825874+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.200+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f98a0076420 msgr2=0x7f98a0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.200+0000 7f98bc03c640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f98a0076420 0x7f98a0078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f98a4005fd0 tx=0x7f98a40095e0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.200+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b4162770 msgr2=0x7f98b406c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.200+0000 7f98bc03c640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b4162770 0x7f98b406c720 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00c3c0 tx=0x7f98ac00c880 comp rx=0 tx=0).stop 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.201+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 shutdown_connections 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.201+0000 7f98bc03c640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f98a0076420 0x7f98a0078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.201+0000 7f98bc03c640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98b4162770 0x7f98b406c720 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.201+0000 7f98bc03c640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98b4161fc0 0x7f98b406c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.201+0000 7f98bc03c640 1 --2- 172.21.15.59:0/2178853331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b415e0a0 0x7f98b4160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.201+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 >> 172.21.15.59:0/2178853331 conn(0x7f98b40d45b0 msgr2=0x7f98b40682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:59.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.201+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 shutdown_connections 2024-06-29T14:39:59.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.201+0000 7f98bc03c640 1 -- 172.21.15.59:0/2178853331 wait complete. 2024-06-29T14:39:59.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:39:59.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.333+0000 7f29fee3f640 1 -- 172.21.15.59:0/2139508307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 msgr2=0x7f29f815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.333+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2139508307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 0x7f29f815bc20 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f29f0005fd0 tx=0x7f29f002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.333+0000 7f29fee3f640 1 -- 172.21.15.59:0/2139508307 shutdown_connections 2024-06-29T14:39:59.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.333+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2139508307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29f80d1c30 0x7f29f80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.333+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2139508307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 0x7f29f815bc20 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.333+0000 7f29fee3f640 1 -- 172.21.15.59:0/2139508307 >> 172.21.15.59:0/2139508307 conn(0x7f29f8073e10 msgr2=0x7f29f8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:59.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.333+0000 7f29fee3f640 1 -- 172.21.15.59:0/2139508307 shutdown_connections 2024-06-29T14:39:59.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.333+0000 7f29fee3f640 1 -- 172.21.15.59:0/2139508307 wait complete. 2024-06-29T14:39:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.335+0000 7f29fee3f640 1 Processor -- start 2024-06-29T14:39:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.335+0000 7f29fee3f640 1 -- start start 2024-06-29T14:39:59.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fee3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 0x7f29f81f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fee3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29f80b2fa0 0x7f29f814c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fee3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29f80d1c30 0x7f29f8158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fee3f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f29f80b6400 con 0x7f29f80b2270 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fee3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29f80b6280 con 0x7f29f80d1c30 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fee3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f29f80b6580 con 0x7f29f80b2fa0 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 0x7f29f81f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 0x7f29f81f5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44122/0 (socket says 172.21.15.59:44122) 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 -- 172.21.15.59:0/2396705862 learned_addr learned my addr 172.21.15.59:0/2396705862 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29f7fff640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29f80b2fa0 0x7f29f814c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fd3dc640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29f80d1c30 0x7f29f8158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 -- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29f80b2fa0 msgr2=0x7f29f814c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29f80b2fa0 0x7f29f814c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 -- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29f80d1c30 msgr2=0x7f29f8158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29f80d1c30 0x7f29f8158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 -- 172.21.15.59:0/2396705862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f81f5770 con 0x7f29f80b2270 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29f7fff640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29f80b2fa0 0x7f29f814c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fd3dc640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29f80d1c30 0x7f29f8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.337+0000 7f29fcbdb640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 0x7f29f81f5230 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f29f002aea0 tx=0x7f29f0003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.338+0000 7f29f57fa640 1 -- 172.21.15.59:0/2396705862 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0031b90 con 0x7f29f80b2270 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.338+0000 7f29f57fa640 1 -- 172.21.15.59:0/2396705862 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29f0031e20 con 0x7f29f80b2270 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.338+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f814c890 con 0x7f29f80b2270 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.338+0000 7f29f57fa640 1 -- 172.21.15.59:0/2396705862 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f00332f0 con 0x7f29f80b2270 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.338+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f80aea30 con 0x7f29f80b2270 2024-06-29T14:39:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.339+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f80b6ac0 con 0x7f29f80b2270 2024-06-29T14:39:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.339+0000 7f29f57fa640 1 -- 172.21.15.59:0/2396705862 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f29f0033490 con 0x7f29f80b2270 2024-06-29T14:39:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.340+0000 7f29f57fa640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29e4076420 0x7f29e4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.340+0000 7f29f57fa640 1 -- 172.21.15.59:0/2396705862 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6101+0+0 (secure 0 0 0) 0x7f29f00b9160 con 0x7f29f80b2270 2024-06-29T14:39:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.340+0000 7f29f7fff640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29e4076420 0x7f29e4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.341+0000 7f29f7fff640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29e4076420 0x7f29e4078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f29f80cef40 tx=0x7f29ec009040 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:59.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.344+0000 7f29f57fa640 1 -- 172.21.15.59:0/2396705862 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f29f814c890 con 0x7f29f80b2270 2024-06-29T14:39:59.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:39:59.581+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f29f80c7310 con 0x7f29f80b2270 2024-06-29T14:40:00.466 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T14:40:00.468+0000 7ff4629d1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:40:00.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.505+0000 7f29f57fa640 1 -- 172.21.15.59:0/2396705862 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) ==== 118+0+0 (secure 0 0 0) 0x7f29f81f5770 con 0x7f29f80b2270 2024-06-29T14:40:00.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29e4076420 msgr2=0x7f29e4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29e4076420 0x7f29e4078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f29f80cef40 tx=0x7f29ec009040 comp rx=0 tx=0).stop 2024-06-29T14:40:00.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 msgr2=0x7f29f81f5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 0x7f29f81f5230 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f29f002aea0 tx=0x7f29f0003a10 comp rx=0 tx=0).stop 2024-06-29T14:40:00.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 shutdown_connections 2024-06-29T14:40:00.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29f80d1c30 0x7f29f8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29f80b2fa0 0x7f29f814c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29e4076420 0x7f29e4078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 --2- 172.21.15.59:0/2396705862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29f80b2270 0x7f29f81f5230 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 >> 172.21.15.59:0/2396705862 conn(0x7f29f8073e10 msgr2=0x7f29f815af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:00.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 shutdown_connections 2024-06-29T14:40:00.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.508+0000 7f29fee3f640 1 -- 172.21.15.59:0/2396705862 wait complete. 2024-06-29T14:40:00.520 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:40:00.521 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:40:00.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 -- 172.21.15.59:0/1310262882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c08c190 msgr2=0x7fab0c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 --2- 172.21.15.59:0/1310262882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c08c190 0x7fab0c0cf080 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0cfbc0 tx=0x7fab0802a560 comp rx=0 tx=0).stop 2024-06-29T14:40:00.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 -- 172.21.15.59:0/1310262882 shutdown_connections 2024-06-29T14:40:00.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 --2- 172.21.15.59:0/1310262882 >> v2:172.21.15.153:3300/0 conn(0x7fab0c0ce950 0x7fab0c0d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 --2- 172.21.15.59:0/1310262882 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab0c0cf5c0 0x7fab0c0d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 --2- 172.21.15.59:0/1310262882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c08c190 0x7fab0c0cf080 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 -- 172.21.15.59:0/1310262882 >> 172.21.15.59:0/1310262882 conn(0x7fab0c0b5610 msgr2=0x7fab0c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:00.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 -- 172.21.15.59:0/1310262882 shutdown_connections 2024-06-29T14:40:00.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.653+0000 7fab13410640 1 -- 172.21.15.59:0/1310262882 wait complete. 2024-06-29T14:40:00.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.655+0000 7fab13410640 1 Processor -- start 2024-06-29T14:40:00.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.655+0000 7fab13410640 1 -- start start 2024-06-29T14:40:00.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab13410640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab0c08c190 0x7fab0c1550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:00.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab13410640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab0c0ce950 0x7fab0c0ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab13410640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c0cf5c0 0x7fab0c0c9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab13410640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c162440 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab13410640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1622c0 con 0x7fab0c08c190 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab13410640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fab0c1625c0 con 0x7fab0c0ce950 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab119ad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c0cf5c0 0x7fab0c0c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab119ad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c0cf5c0 0x7fab0c0c9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44140/0 (socket says 172.21.15.59:44140) 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab119ad640 1 -- 172.21.15.59:0/1770741035 learned_addr learned my addr 172.21.15.59:0/1770741035 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab109ab640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab0c0ce950 0x7fab0c0ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab111ac640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab0c08c190 0x7fab0c1550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab119ad640 1 -- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab0c0ce950 msgr2=0x7fab0c0ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab119ad640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab0c0ce950 0x7fab0c0ae4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab119ad640 1 -- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab0c08c190 msgr2=0x7fab0c1550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab119ad640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab0c08c190 0x7fab0c1550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.656+0000 7fab119ad640 1 -- 172.21.15.59:0/1770741035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c0ac7f0 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.657+0000 7fab119ad640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c0cf5c0 0x7fab0c0c9750 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fab0400cab0 tx=0x7fab0400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:00.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.657+0000 7fab01ffb640 1 -- 172.21.15.59:0/1770741035 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab04007d90 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.657+0000 7fab01ffb640 1 -- 172.21.15.59:0/1770741035 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab0400f560 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.657+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab0c0acad0 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.657+0000 7fab01ffb640 1 -- 172.21.15.59:0/1770741035 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0400f9b0 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.657+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab0c0ad090 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.658+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab0c15e9a0 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.659+0000 7fab01ffb640 1 -- 172.21.15.59:0/1770741035 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fab0400fb50 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.659+0000 7fab01ffb640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7faaf00763f0 0x7faaf0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:00.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.659+0000 7fab01ffb640 1 -- 172.21.15.59:0/1770741035 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7fab04098590 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.660+0000 7fab111ac640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7faaf00763f0 0x7faaf0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:00.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.660+0000 7fab111ac640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7faaf00763f0 0x7faaf0078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab08033000 tx=0x7fab08004730 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:00.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.664+0000 7fab01ffb640 1 -- 172.21.15.59:0/1770741035 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fab0c15e9a0 con 0x7fab0c0cf5c0 2024-06-29T14:40:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:00.899+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fab0c0cf080 con 0x7fab0c0cf5c0 2024-06-29T14:40:01.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.506+0000 7fab01ffb640 1 -- 172.21.15.59:0/1770741035 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v63) ==== 122+0+0 (secure 0 0 0) 0x7fab04014070 con 0x7fab0c0cf5c0 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7faaf00763f0 msgr2=0x7faaf0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7faaf00763f0 0x7faaf0078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab08033000 tx=0x7fab08004730 comp rx=0 tx=0).stop 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c0cf5c0 msgr2=0x7fab0c0c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c0cf5c0 0x7fab0c0c9750 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fab0400cab0 tx=0x7fab0400cf70 comp rx=0 tx=0).stop 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 shutdown_connections 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7faaf00763f0 0x7faaf0078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab0c0cf5c0 0x7fab0c0c9750 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab0c0ce950 0x7fab0c0ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 --2- 172.21.15.59:0/1770741035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab0c08c190 0x7fab0c1550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 >> 172.21.15.59:0/1770741035 conn(0x7fab0c0b5610 msgr2=0x7fab0c06d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 shutdown_connections 2024-06-29T14:40:01.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.509+0000 7fab13410640 1 -- 172.21.15.59:0/1770741035 wait complete. 2024-06-29T14:40:01.521 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:40:01.521 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:40:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.648+0000 7fa9e475b640 1 -- 172.21.15.59:0/3352124763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 msgr2=0x7fa9c4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.648+0000 7fa9e475b640 1 --2- 172.21.15.59:0/3352124763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 0x7fa9c4004c10 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000c2e0 tx=0x7fa9c4005830 comp rx=0 tx=0).stop 2024-06-29T14:40:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.648+0000 7fa9e475b640 1 -- 172.21.15.59:0/3352124763 shutdown_connections 2024-06-29T14:40:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.648+0000 7fa9e475b640 1 --2- 172.21.15.59:0/3352124763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 0x7fa9c4004c10 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.648+0000 7fa9e475b640 1 -- 172.21.15.59:0/3352124763 >> 172.21.15.59:0/3352124763 conn(0x7fa9dc0d3ee0 msgr2=0x7fa9dc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.648+0000 7fa9e475b640 1 -- 172.21.15.59:0/3352124763 shutdown_connections 2024-06-29T14:40:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.648+0000 7fa9e475b640 1 -- 172.21.15.59:0/3352124763 wait complete. 2024-06-29T14:40:01.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.650+0000 7fa9e475b640 1 Processor -- start 2024-06-29T14:40:01.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.650+0000 7fa9e475b640 1 -- start start 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.651+0000 7fa9e475b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 0x7fa9dc15da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.651+0000 7fa9e475b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9dc14adb0 0x7fa9dc0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.651+0000 7fa9e475b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc14b1c0 0x7fa9dc0d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.651+0000 7fa9e475b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc15ed10 con 0x7fa9c4004800 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.651+0000 7fa9e475b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc15eb90 con 0x7fa9dc14b1c0 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.651+0000 7fa9e475b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa9dc0b3990 con 0x7fa9dc14adb0 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.651+0000 7fa9e24f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 0x7fa9dc15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e24f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 0x7fa9dc15da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44154/0 (socket says 172.21.15.59:44154) 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e24f7640 1 -- 172.21.15.59:0/634210219 learned_addr learned my addr 172.21.15.59:0/634210219 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:01.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e2cf8640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc14b1c0 0x7fa9dc0d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e1cf6640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9dc14adb0 0x7fa9dc0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e24f7640 1 -- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9dc14adb0 msgr2=0x7fa9dc0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e24f7640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9dc14adb0 0x7fa9dc0d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e24f7640 1 -- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc14b1c0 msgr2=0x7fa9dc0d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e24f7640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc14b1c0 0x7fa9dc0d0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e24f7640 1 -- 172.21.15.59:0/634210219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9dc1f8d80 con 0x7fa9c4004800 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e24f7640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 0x7fa9dc15da70 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0021d0 tx=0x7fa9cc0064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9daffd640 1 -- 172.21.15.59:0/634210219 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc015070 con 0x7fa9c4004800 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9daffd640 1 -- 172.21.15.59:0/634210219 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9cc00fa00 con 0x7fa9c4004800 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9dc1f9060 con 0x7fa9c4004800 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9daffd640 1 -- 172.21.15.59:0/634210219 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00fe30 con 0x7fa9c4004800 2024-06-29T14:40:01.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.652+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9dc0c7720 con 0x7fa9c4004800 2024-06-29T14:40:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.653+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9dc0b3ef0 con 0x7fa9c4004800 2024-06-29T14:40:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.654+0000 7fa9daffd640 1 -- 172.21.15.59:0/634210219 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fa9cc011430 con 0x7fa9c4004800 2024-06-29T14:40:01.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.655+0000 7fa9daffd640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa9c80763f0 0x7fa9c8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.655+0000 7fa9daffd640 1 -- 172.21.15.59:0/634210219 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7fa9cc098a00 con 0x7fa9c4004800 2024-06-29T14:40:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.655+0000 7fa9e1cf6640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa9c80763f0 0x7fa9c8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.655+0000 7fa9e1cf6640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa9c80763f0 0x7fa9c8078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0033000 tx=0x7fa9d00019f0 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:01.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.659+0000 7fa9daffd640 1 -- 172.21.15.59:0/634210219 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa9cc061f10 con 0x7fa9c4004800 2024-06-29T14:40:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.888+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9dc157600 con 0x7fa9c4004800 2024-06-29T14:40:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.889+0000 7fa9daffd640 1 -- 172.21.15.59:0/634210219 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5228 (secure 0 0 0) 0x7fa9cc061810 con 0x7fa9c4004800 2024-06-29T14:40:01.887 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:40:01.887 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":63,"btime":"2024-06-29T14:40:01:506455+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2473404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2473404084},{"type":"v1","addr":"172.21.15.153:6836","nonce":2473404084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7831,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3605263971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3605263971},{"type":"v1","addr":"172.21.15.153:6837","nonce":3605263971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/842001425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":842001425},{"type":"v1","addr":"172.21.15.59:6833","nonce":842001425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.885754+0000","modified":"2024-06-29T14:40:00.901266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3950841523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3950841523},{"type":"v1","addr":"172.21.15.59:6835","nonce":3950841523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8361,"qdb_cluster":[8361]},"id":7}]} 2024-06-29T14:40:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 63 2024-06-29T14:40:01.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa9c80763f0 msgr2=0x7fa9c8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa9c80763f0 0x7fa9c8078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0033000 tx=0x7fa9d00019f0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 msgr2=0x7fa9dc15da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 0x7fa9dc15da70 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0021d0 tx=0x7fa9cc0064f0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 shutdown_connections 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc14b1c0 0x7fa9dc0d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa9dc14adb0 0x7fa9dc0d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fa9c80763f0 0x7fa9c8078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 --2- 172.21.15.59:0/634210219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c4004800 0x7fa9dc15da70 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 >> 172.21.15.59:0/634210219 conn(0x7fa9dc0d3ee0 msgr2=0x7fa9dc15e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 shutdown_connections 2024-06-29T14:40:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:01.891+0000 7fa9e475b640 1 -- 172.21.15.59:0/634210219 wait complete. 2024-06-29T14:40:01.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:40:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 -- 172.21.15.59:0/1879669880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f80ce6c0 msgr2=0x7f31f815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 --2- 172.21.15.59:0/1879669880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f80ce6c0 0x7f31f815bee0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f31f8073fc0 tx=0x7f31e802a540 comp rx=0 tx=0).stop 2024-06-29T14:40:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 -- 172.21.15.59:0/1879669880 shutdown_connections 2024-06-29T14:40:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 --2- 172.21.15.59:0/1879669880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31f8073a00 0x7f31f80b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 --2- 172.21.15.59:0/1879669880 >> v2:172.21.15.153:3300/0 conn(0x7f31f8144760 0x7f31f80734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 --2- 172.21.15.59:0/1879669880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f80ce6c0 0x7f31f815bee0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 -- 172.21.15.59:0/1879669880 >> 172.21.15.59:0/1879669880 conn(0x7f31f806cef0 msgr2=0x7f31f806d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 -- 172.21.15.59:0/1879669880 shutdown_connections 2024-06-29T14:40:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.021+0000 7f31fe57a640 1 -- 172.21.15.59:0/1879669880 wait complete. 2024-06-29T14:40:02.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.023+0000 7f31fe57a640 1 Processor -- start 2024-06-29T14:40:02.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.023+0000 7f31fe57a640 1 -- start start 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.024+0000 7f31fe57a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f8073a00 0x7f31f81585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.024+0000 7f31fe57a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31f80ce6c0 0x7f31f8132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.024+0000 7f31fe57a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31f8144760 0x7f31f813e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.024+0000 7f31fe57a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f31f80b48c0 con 0x7f31f8073a00 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.024+0000 7f31fe57a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31f80b4740 con 0x7f31f8144760 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31fe57a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f31f80b4a40 con 0x7f31f80ce6c0 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f8073a00 0x7f31f81585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f8073a00 0x7f31f81585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44170/0 (socket says 172.21.15.59:44170) 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 -- 172.21.15.59:0/2069357360 learned_addr learned my addr 172.21.15.59:0/2069357360 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f77fe640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31f80ce6c0 0x7f31f8132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31fcb17640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31f8144760 0x7f31f813e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 -- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31f80ce6c0 msgr2=0x7f31f8132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31f80ce6c0 0x7f31f8132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 -- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31f8144760 msgr2=0x7f31f813e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31f8144760 0x7f31f813e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 -- 172.21.15.59:0/2069357360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f81318d0 con 0x7f31f8073a00 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f7fff640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f8073a00 0x7f31f81585e0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f31e8034040 tx=0x7f31e80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f4ff9640 1 -- 172.21.15.59:0/2069357360 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e803e030 con 0x7f31f8073a00 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f4ff9640 1 -- 172.21.15.59:0/2069357360 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31e80047a0 con 0x7f31f8073a00 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f8131bb0 con 0x7f31f8073a00 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31f4ff9640 1 -- 172.21.15.59:0/2069357360 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e8035040 con 0x7f31f8073a00 2024-06-29T14:40:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.025+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f8132170 con 0x7f31f8073a00 2024-06-29T14:40:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.026+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f815bee0 con 0x7f31f8073a00 2024-06-29T14:40:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.027+0000 7f31f4ff9640 1 -- 172.21.15.59:0/2069357360 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f31e8041030 con 0x7f31f8073a00 2024-06-29T14:40:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.028+0000 7f31f4ff9640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f31e4076420 0x7f31e4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.028+0000 7f31f4ff9640 1 -- 172.21.15.59:0/2069357360 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f31e80b8350 con 0x7f31f8073a00 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.028+0000 7f31f77fe640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f31e4076420 0x7f31e4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.029+0000 7f31f77fe640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f31e4076420 0x7f31e4078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31ec005fd0 tx=0x7f31ec005da0 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.032+0000 7f31f4ff9640 1 -- 172.21.15.59:0/2069357360 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f31e8081910 con 0x7f31f8073a00 2024-06-29T14:40:02.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.222+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31f80701e0 con 0x7f31f8073a00 2024-06-29T14:40:02.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.223+0000 7f31f4ff9640 1 -- 172.21.15.59:0/2069357360 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+16896 (secure 0 0 0) 0x7f31e8081210 con 0x7f31f8073a00 2024-06-29T14:40:02.222 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:40:02.222 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":129,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:39:59.485324+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:56.243419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.825874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f31e4076420 msgr2=0x7f31e4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f31e4076420 0x7f31e4078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31ec005fd0 tx=0x7f31ec005da0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f8073a00 msgr2=0x7f31f81585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f8073a00 0x7f31f81585e0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f31e8034040 tx=0x7f31e80039e0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 shutdown_connections 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31f8144760 0x7f31f813e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31f80ce6c0 0x7f31f8132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f31e4076420 0x7f31e4078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 --2- 172.21.15.59:0/2069357360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f8073a00 0x7f31f81585e0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.225+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 >> 172.21.15.59:0/2069357360 conn(0x7f31f806cef0 msgr2=0x7f31f815dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.226+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 shutdown_connections 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.226+0000 7f31fe57a640 1 -- 172.21.15.59:0/2069357360 wait complete. 2024-06-29T14:40:02.235 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:40:02.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:40:02.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.356+0000 7f496543f640 1 -- 172.21.15.59:0/799477130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 msgr2=0x7f494c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.356+0000 7f496543f640 1 --2- 172.21.15.59:0/799477130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 0x7f494c003ee0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f495000c2e0 tx=0x7f494c005830 comp rx=0 tx=0).stop 2024-06-29T14:40:02.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.356+0000 7f495f7fe640 1 -- 172.21.15.59:0/799477130 reap_dead start 2024-06-29T14:40:02.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.356+0000 7f496543f640 1 -- 172.21.15.59:0/799477130 shutdown_connections 2024-06-29T14:40:02.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.356+0000 7f496543f640 1 -- 172.21.15.59:0/799477130 >> 172.21.15.59:0/799477130 conn(0x7f4960073e10 msgr2=0x7f4960074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.356+0000 7f496543f640 1 -- 172.21.15.59:0/799477130 shutdown_connections 2024-06-29T14:40:02.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.356+0000 7f496543f640 1 -- 172.21.15.59:0/799477130 wait complete. 2024-06-29T14:40:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.358+0000 7f496543f640 1 Processor -- start 2024-06-29T14:40:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.358+0000 7f496543f640 1 -- start start 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f496543f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 0x7f496014dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f496543f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f496015e000 0x7f4960158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f496543f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f496015e410 0x7f49601586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f496543f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f49600b5ec0 con 0x7f494c003a70 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f496543f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f49600b5d40 con 0x7f496015e410 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f496543f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f49600b6040 con 0x7f496015e000 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f495effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 0x7f496014dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f495effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 0x7f496014dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44202/0 (socket says 172.21.15.59:44202) 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f495effd640 1 -- 172.21.15.59:0/4010594367 learned_addr learned my addr 172.21.15.59:0/4010594367 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f49567fc640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f496015e000 0x7f4960158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.359+0000 7f495f7fe640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f496015e410 0x7f49601586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f495effd640 1 -- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f496015e000 msgr2=0x7f4960158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f495effd640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f496015e000 0x7f4960158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f495effd640 1 -- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f496015e410 msgr2=0x7f49601586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f495effd640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f496015e410 0x7f49601586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f495effd640 1 -- 172.21.15.59:0/4010594367 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496014e7c0 con 0x7f494c003a70 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f495effd640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 0x7f496014dd00 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f495000c400 tx=0x7f4950006ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f4957fff640 1 -- 172.21.15.59:0/4010594367 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950031040 con 0x7f494c003a70 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f4957fff640 1 -- 172.21.15.59:0/4010594367 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4950003820 con 0x7f494c003a70 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496014e240 con 0x7f494c003a70 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f4957fff640 1 -- 172.21.15.59:0/4010594367 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950006600 con 0x7f494c003a70 2024-06-29T14:40:02.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.360+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4960071460 con 0x7f494c003a70 2024-06-29T14:40:02.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.361+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49600b4cc0 con 0x7f494c003a70 2024-06-29T14:40:02.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.362+0000 7f4957fff640 1 -- 172.21.15.59:0/4010594367 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f4950034070 con 0x7f494c003a70 2024-06-29T14:40:02.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.362+0000 7f4957fff640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4940076420 0x7f4940078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.362+0000 7f4957fff640 1 -- 172.21.15.59:0/4010594367 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f49500b7ac0 con 0x7f494c003a70 2024-06-29T14:40:02.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.363+0000 7f49567fc640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4940076420 0x7f4940078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.363+0000 7f49567fc640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4940076420 0x7f4940078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4960070cf0 tx=0x7f4948005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.367+0000 7f4957fff640 1 -- 172.21.15.59:0/4010594367 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f49600b4cc0 con 0x7f494c003a70 2024-06-29T14:40:02.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.558+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 --> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4960070f30 con 0x7f4940076420 2024-06-29T14:40:02.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.564+0000 7f4957fff640 1 -- 172.21.15.59:0/4010594367 <== mgr.7615 v2:172.21.15.153:6832/892574025 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252487 (secure 0 0 0) 0x7f4960070f30 con 0x7f4940076420 2024-06-29T14:40:02.565 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:40:02.566 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:40:02.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4940076420 msgr2=0x7f4940078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4940076420 0x7f4940078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4960070cf0 tx=0x7f4948005f90 comp rx=0 tx=0).stop 2024-06-29T14:40:02.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 msgr2=0x7f496014dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 0x7f496014dd00 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f495000c400 tx=0x7f4950006ce0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 shutdown_connections 2024-06-29T14:40:02.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f496015e410 0x7f49601586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f496015e000 0x7f4960158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4940076420 0x7f4940078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 --2- 172.21.15.59:0/4010594367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f494c003a70 0x7f496014dd00 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 >> 172.21.15.59:0/4010594367 conn(0x7f4960073e10 msgr2=0x7f49601578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 shutdown_connections 2024-06-29T14:40:02.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.566+0000 7f496543f640 1 -- 172.21.15.59:0/4010594367 wait complete. 2024-06-29T14:40:02.576 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:40:01.992960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":266,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":350480,"kb_used_data":13580,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754624240,"statfs":{"total":773094113280,"available":772735221760,"internally_reserved":0,"allocated":13905920,"data_stored":7585072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":344837682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.516125"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507016+0000","last_change":"2024-06-29T14:39:58.522340+0000","last_active":"2024-06-29T14:39:59.507016+0000","last_peered":"2024-06-29T14:39:59.507016+0000","last_clean":"2024-06-29T14:39:59.507016+0000","last_became_active":"2024-06-29T14:39:58.521203+0000","last_became_peered":"2024-06-29T14:39:58.521203+0000","last_unstale":"2024-06-29T14:39:59.507016+0000","last_undegraded":"2024-06-29T14:39:59.507016+0000","last_fullsized":"2024-06-29T14:39:59.507016+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:03.881878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493801+0000","last_change":"2024-06-29T14:39:58.506632+0000","last_active":"2024-06-29T14:39:59.493801+0000","last_peered":"2024-06-29T14:39:59.493801+0000","last_clean":"2024-06-29T14:39:59.493801+0000","last_became_active":"2024-06-29T14:39:58.506327+0000","last_became_peered":"2024-06-29T14:39:58.506327+0000","last_unstale":"2024-06-29T14:39:59.493801+0000","last_undegraded":"2024-06-29T14:39:59.493801+0000","last_fullsized":"2024-06-29T14:39:59.493801+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:23.806553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507081+0000","last_change":"2024-06-29T14:39:58.523645+0000","last_active":"2024-06-29T14:39:59.507081+0000","last_peered":"2024-06-29T14:39:59.507081+0000","last_clean":"2024-06-29T14:39:59.507081+0000","last_became_active":"2024-06-29T14:39:58.523242+0000","last_became_peered":"2024-06-29T14:39:58.523242+0000","last_unstale":"2024-06-29T14:39:59.507081+0000","last_undegraded":"2024-06-29T14:39:59.507081+0000","last_fullsized":"2024-06-29T14:39:59.507081+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:05.348037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986720+0000","last_change":"2024-06-29T14:39:58.503281+0000","last_active":"2024-06-29T14:39:59.986720+0000","last_peered":"2024-06-29T14:39:59.986720+0000","last_clean":"2024-06-29T14:39:59.986720+0000","last_became_active":"2024-06-29T14:39:58.502960+0000","last_became_peered":"2024-06-29T14:39:58.502960+0000","last_unstale":"2024-06-29T14:39:59.986720+0000","last_undegraded":"2024-06-29T14:39:59.986720+0000","last_fullsized":"2024-06-29T14:39:59.986720+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:08:45.202818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489008+0000","last_change":"2024-06-29T14:39:58.500771+0000","last_active":"2024-06-29T14:39:59.489008+0000","last_peered":"2024-06-29T14:39:59.489008+0000","last_clean":"2024-06-29T14:39:59.489008+0000","last_became_active":"2024-06-29T14:39:58.500448+0000","last_became_peered":"2024-06-29T14:39:58.500448+0000","last_unstale":"2024-06-29T14:39:59.489008+0000","last_undegraded":"2024-06-29T14:39:59.489008+0000","last_fullsized":"2024-06-29T14:39:59.489008+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:07:27.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494192+0000","last_change":"2024-06-29T14:39:58.499653+0000","last_active":"2024-06-29T14:39:59.494192+0000","last_peered":"2024-06-29T14:39:59.494192+0000","last_clean":"2024-06-29T14:39:59.494192+0000","last_became_active":"2024-06-29T14:39:58.498677+0000","last_became_peered":"2024-06-29T14:39:58.498677+0000","last_unstale":"2024-06-29T14:39:59.494192+0000","last_undegraded":"2024-06-29T14:39:59.494192+0000","last_fullsized":"2024-06-29T14:39:59.494192+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:48.747706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494312+0000","last_change":"2024-06-29T14:39:58.513104+0000","last_active":"2024-06-29T14:39:59.494312+0000","last_peered":"2024-06-29T14:39:59.494312+0000","last_clean":"2024-06-29T14:39:59.494312+0000","last_became_active":"2024-06-29T14:39:58.511342+0000","last_became_peered":"2024-06-29T14:39:58.511342+0000","last_unstale":"2024-06-29T14:39:59.494312+0000","last_undegraded":"2024-06-29T14:39:59.494312+0000","last_fullsized":"2024-06-29T14:39:59.494312+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:09.448042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539440+0000","last_change":"2024-06-29T14:39:58.510498+0000","last_active":"2024-06-29T14:39:59.539440+0000","last_peered":"2024-06-29T14:39:59.539440+0000","last_clean":"2024-06-29T14:39:59.539440+0000","last_became_active":"2024-06-29T14:39:58.510139+0000","last_became_peered":"2024-06-29T14:39:58.510139+0000","last_unstale":"2024-06-29T14:39:59.539440+0000","last_undegraded":"2024-06-29T14:39:59.539440+0000","last_fullsized":"2024-06-29T14:39:59.539440+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:31.834229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494086+0000","last_change":"2024-06-29T14:39:58.512240+0000","last_active":"2024-06-29T14:39:59.494086+0000","last_peered":"2024-06-29T14:39:59.494086+0000","last_clean":"2024-06-29T14:39:59.494086+0000","last_became_active":"2024-06-29T14:39:58.510727+0000","last_became_peered":"2024-06-29T14:39:58.510727+0000","last_unstale":"2024-06-29T14:39:59.494086+0000","last_undegraded":"2024-06-29T14:39:59.494086+0000","last_fullsized":"2024-06-29T14:39:59.494086+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:53.606347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489152+0000","last_change":"2024-06-29T14:39:58.503141+0000","last_active":"2024-06-29T14:39:59.489152+0000","last_peered":"2024-06-29T14:39:59.489152+0000","last_clean":"2024-06-29T14:39:59.489152+0000","last_became_active":"2024-06-29T14:39:58.502682+0000","last_became_peered":"2024-06-29T14:39:58.502682+0000","last_unstale":"2024-06-29T14:39:59.489152+0000","last_undegraded":"2024-06-29T14:39:59.489152+0000","last_fullsized":"2024-06-29T14:39:59.489152+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:26:36.492357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509194+0000","last_change":"2024-06-29T14:39:58.516336+0000","last_active":"2024-06-29T14:39:59.509194+0000","last_peered":"2024-06-29T14:39:59.509194+0000","last_clean":"2024-06-29T14:39:59.509194+0000","last_became_active":"2024-06-29T14:39:58.515090+0000","last_became_peered":"2024-06-29T14:39:58.515090+0000","last_unstale":"2024-06-29T14:39:59.509194+0000","last_undegraded":"2024-06-29T14:39:59.509194+0000","last_fullsized":"2024-06-29T14:39:59.509194+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:14:57.642394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492890+0000","last_change":"2024-06-29T14:39:58.505973+0000","last_active":"2024-06-29T14:39:59.492890+0000","last_peered":"2024-06-29T14:39:59.492890+0000","last_clean":"2024-06-29T14:39:59.492890+0000","last_became_active":"2024-06-29T14:39:58.505477+0000","last_became_peered":"2024-06-29T14:39:58.505477+0000","last_unstale":"2024-06-29T14:39:59.492890+0000","last_undegraded":"2024-06-29T14:39:59.492890+0000","last_fullsized":"2024-06-29T14:39:59.492890+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:23:48.378022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986435+0000","last_change":"2024-06-29T14:39:58.514529+0000","last_active":"2024-06-29T14:39:59.986435+0000","last_peered":"2024-06-29T14:39:59.986435+0000","last_clean":"2024-06-29T14:39:59.986435+0000","last_became_active":"2024-06-29T14:39:58.514139+0000","last_became_peered":"2024-06-29T14:39:58.514139+0000","last_unstale":"2024-06-29T14:39:59.986435+0000","last_undegraded":"2024-06-29T14:39:59.986435+0000","last_fullsized":"2024-06-29T14:39:59.986435+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:27.970923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986733+0000","last_change":"2024-06-29T14:39:58.517145+0000","last_active":"2024-06-29T14:39:59.986733+0000","last_peered":"2024-06-29T14:39:59.986733+0000","last_clean":"2024-06-29T14:39:59.986733+0000","last_became_active":"2024-06-29T14:39:58.516817+0000","last_became_peered":"2024-06-29T14:39:58.516817+0000","last_unstale":"2024-06-29T14:39:59.986733+0000","last_undegraded":"2024-06-29T14:39:59.986733+0000","last_fullsized":"2024-06-29T14:39:59.986733+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:25:19.271179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986641+0000","last_change":"2024-06-29T14:39:58.508090+0000","last_active":"2024-06-29T14:39:59.986641+0000","last_peered":"2024-06-29T14:39:59.986641+0000","last_clean":"2024-06-29T14:39:59.986641+0000","last_became_active":"2024-06-29T14:39:58.507749+0000","last_became_peered":"2024-06-29T14:39:58.507749+0000","last_unstale":"2024-06-29T14:39:59.986641+0000","last_undegraded":"2024-06-29T14:39:59.986641+0000","last_fullsized":"2024-06-29T14:39:59.986641+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:41:26.853858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986622+0000","last_change":"2024-06-29T14:39:58.512353+0000","last_active":"2024-06-29T14:39:59.986622+0000","last_peered":"2024-06-29T14:39:59.986622+0000","last_clean":"2024-06-29T14:39:59.986622+0000","last_became_active":"2024-06-29T14:39:58.512028+0000","last_became_peered":"2024-06-29T14:39:58.512028+0000","last_unstale":"2024-06-29T14:39:59.986622+0000","last_undegraded":"2024-06-29T14:39:59.986622+0000","last_fullsized":"2024-06-29T14:39:59.986622+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:19.295385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507139+0000","last_change":"2024-06-29T14:39:58.521980+0000","last_active":"2024-06-29T14:39:59.507139+0000","last_peered":"2024-06-29T14:39:59.507139+0000","last_clean":"2024-06-29T14:39:59.507139+0000","last_became_active":"2024-06-29T14:39:58.520815+0000","last_became_peered":"2024-06-29T14:39:58.520815+0000","last_unstale":"2024-06-29T14:39:59.507139+0000","last_undegraded":"2024-06-29T14:39:59.507139+0000","last_fullsized":"2024-06-29T14:39:59.507139+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:30.767893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493837+0000","last_change":"2024-06-29T14:39:58.498229+0000","last_active":"2024-06-29T14:39:59.493837+0000","last_peered":"2024-06-29T14:39:59.493837+0000","last_clean":"2024-06-29T14:39:59.493837+0000","last_became_active":"2024-06-29T14:39:58.496059+0000","last_became_peered":"2024-06-29T14:39:58.496059+0000","last_unstale":"2024-06-29T14:39:59.493837+0000","last_undegraded":"2024-06-29T14:39:59.493837+0000","last_fullsized":"2024-06-29T14:39:59.493837+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:03.396218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507085+0000","last_change":"2024-06-29T14:39:58.516130+0000","last_active":"2024-06-29T14:39:59.507085+0000","last_peered":"2024-06-29T14:39:59.507085+0000","last_clean":"2024-06-29T14:39:59.507085+0000","last_became_active":"2024-06-29T14:39:58.513799+0000","last_became_peered":"2024-06-29T14:39:58.513799+0000","last_unstale":"2024-06-29T14:39:59.507085+0000","last_undegraded":"2024-06-29T14:39:59.507085+0000","last_fullsized":"2024-06-29T14:39:59.507085+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:14:57.642394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493372+0000","last_change":"2024-06-29T14:39:58.498448+0000","last_active":"2024-06-29T14:39:59.493372+0000","last_peered":"2024-06-29T14:39:59.493372+0000","last_clean":"2024-06-29T14:39:59.493372+0000","last_became_active":"2024-06-29T14:39:58.498128+0000","last_became_peered":"2024-06-29T14:39:58.498128+0000","last_unstale":"2024-06-29T14:39:59.493372+0000","last_undegraded":"2024-06-29T14:39:59.493372+0000","last_fullsized":"2024-06-29T14:39:59.493372+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:30.295533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539822+0000","last_change":"2024-06-29T14:39:58.508313+0000","last_active":"2024-06-29T14:39:59.539822+0000","last_peered":"2024-06-29T14:39:59.539822+0000","last_clean":"2024-06-29T14:39:59.539822+0000","last_became_active":"2024-06-29T14:39:58.507947+0000","last_became_peered":"2024-06-29T14:39:58.507947+0000","last_unstale":"2024-06-29T14:39:59.539822+0000","last_undegraded":"2024-06-29T14:39:59.539822+0000","last_fullsized":"2024-06-29T14:39:59.539822+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:36.389730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.488899+0000","last_change":"2024-06-29T14:39:58.511653+0000","last_active":"2024-06-29T14:39:59.488899+0000","last_peered":"2024-06-29T14:39:59.488899+0000","last_clean":"2024-06-29T14:39:59.488899+0000","last_became_active":"2024-06-29T14:39:58.511317+0000","last_became_peered":"2024-06-29T14:39:58.511317+0000","last_unstale":"2024-06-29T14:39:59.488899+0000","last_undegraded":"2024-06-29T14:39:59.488899+0000","last_fullsized":"2024-06-29T14:39:59.488899+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:17.412011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506758+0000","last_change":"2024-06-29T14:39:58.517032+0000","last_active":"2024-06-29T14:39:59.506758+0000","last_peered":"2024-06-29T14:39:59.506758+0000","last_clean":"2024-06-29T14:39:59.506758+0000","last_became_active":"2024-06-29T14:39:58.516697+0000","last_became_peered":"2024-06-29T14:39:58.516697+0000","last_unstale":"2024-06-29T14:39:59.506758+0000","last_undegraded":"2024-06-29T14:39:59.506758+0000","last_fullsized":"2024-06-29T14:39:59.506758+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:14:59.300903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.488577+0000","last_change":"2024-06-29T14:39:58.513491+0000","last_active":"2024-06-29T14:39:59.488577+0000","last_peered":"2024-06-29T14:39:59.488577+0000","last_clean":"2024-06-29T14:39:59.488577+0000","last_became_active":"2024-06-29T14:39:58.512947+0000","last_became_peered":"2024-06-29T14:39:58.512947+0000","last_unstale":"2024-06-29T14:39:59.488577+0000","last_undegraded":"2024-06-29T14:39:59.488577+0000","last_fullsized":"2024-06-29T14:39:59.488577+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:02.668775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493434+0000","last_change":"2024-06-29T14:39:58.508977+0000","last_active":"2024-06-29T14:39:59.493434+0000","last_peered":"2024-06-29T14:39:59.493434+0000","last_clean":"2024-06-29T14:39:59.493434+0000","last_became_active":"2024-06-29T14:39:58.508680+0000","last_became_peered":"2024-06-29T14:39:58.508680+0000","last_unstale":"2024-06-29T14:39:59.493434+0000","last_undegraded":"2024-06-29T14:39:59.493434+0000","last_fullsized":"2024-06-29T14:39:59.493434+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:24:29.925276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493722+0000","last_change":"2024-06-29T14:39:58.499983+0000","last_active":"2024-06-29T14:39:59.493722+0000","last_peered":"2024-06-29T14:39:59.493722+0000","last_clean":"2024-06-29T14:39:59.493722+0000","last_became_active":"2024-06-29T14:39:58.499618+0000","last_became_peered":"2024-06-29T14:39:58.499618+0000","last_unstale":"2024-06-29T14:39:59.493722+0000","last_undegraded":"2024-06-29T14:39:59.493722+0000","last_fullsized":"2024-06-29T14:39:59.493722+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:12.707165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509245+0000","last_change":"2024-06-29T14:39:58.507714+0000","last_active":"2024-06-29T14:39:59.509245+0000","last_peered":"2024-06-29T14:39:59.509245+0000","last_clean":"2024-06-29T14:39:59.509245+0000","last_became_active":"2024-06-29T14:39:58.507375+0000","last_became_peered":"2024-06-29T14:39:58.507375+0000","last_unstale":"2024-06-29T14:39:59.509245+0000","last_undegraded":"2024-06-29T14:39:59.509245+0000","last_fullsized":"2024-06-29T14:39:59.509245+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:35.075707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493515+0000","last_change":"2024-06-29T14:39:58.511346+0000","last_active":"2024-06-29T14:39:59.493515+0000","last_peered":"2024-06-29T14:39:59.493515+0000","last_clean":"2024-06-29T14:39:59.493515+0000","last_became_active":"2024-06-29T14:39:58.511007+0000","last_became_peered":"2024-06-29T14:39:58.511007+0000","last_unstale":"2024-06-29T14:39:59.493515+0000","last_undegraded":"2024-06-29T14:39:59.493515+0000","last_fullsized":"2024-06-29T14:39:59.493515+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:37:46.847863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539769+0000","last_change":"2024-06-29T14:39:58.499715+0000","last_active":"2024-06-29T14:39:59.539769+0000","last_peered":"2024-06-29T14:39:59.539769+0000","last_clean":"2024-06-29T14:39:59.539769+0000","last_became_active":"2024-06-29T14:39:58.499339+0000","last_became_peered":"2024-06-29T14:39:58.499339+0000","last_unstale":"2024-06-29T14:39:59.539769+0000","last_undegraded":"2024-06-29T14:39:59.539769+0000","last_fullsized":"2024-06-29T14:39:59.539769+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:05.615307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492992+0000","last_change":"2024-06-29T14:39:58.506765+0000","last_active":"2024-06-29T14:39:59.492992+0000","last_peered":"2024-06-29T14:39:59.492992+0000","last_clean":"2024-06-29T14:39:59.492992+0000","last_became_active":"2024-06-29T14:39:58.506432+0000","last_became_peered":"2024-06-29T14:39:58.506432+0000","last_unstale":"2024-06-29T14:39:59.492992+0000","last_undegraded":"2024-06-29T14:39:59.492992+0000","last_fullsized":"2024-06-29T14:39:59.492992+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:37.139432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494473+0000","last_change":"2024-06-29T14:39:58.510610+0000","last_active":"2024-06-29T14:39:59.494473+0000","last_peered":"2024-06-29T14:39:59.494473+0000","last_clean":"2024-06-29T14:39:59.494473+0000","last_became_active":"2024-06-29T14:39:58.510273+0000","last_became_peered":"2024-06-29T14:39:58.510273+0000","last_unstale":"2024-06-29T14:39:59.494473+0000","last_undegraded":"2024-06-29T14:39:59.494473+0000","last_fullsized":"2024-06-29T14:39:59.494473+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:12.330677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539528+0000","last_change":"2024-06-29T14:39:58.500723+0000","last_active":"2024-06-29T14:39:59.539528+0000","last_peered":"2024-06-29T14:39:59.539528+0000","last_clean":"2024-06-29T14:39:59.539528+0000","last_became_active":"2024-06-29T14:39:58.500364+0000","last_became_peered":"2024-06-29T14:39:58.500364+0000","last_unstale":"2024-06-29T14:39:59.539528+0000","last_undegraded":"2024-06-29T14:39:59.539528+0000","last_fullsized":"2024-06-29T14:39:59.539528+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:33:15.028401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506909+0000","last_change":"2024-06-29T14:39:58.507123+0000","last_active":"2024-06-29T14:39:59.506909+0000","last_peered":"2024-06-29T14:39:59.506909+0000","last_clean":"2024-06-29T14:39:59.506909+0000","last_became_active":"2024-06-29T14:39:58.506774+0000","last_became_peered":"2024-06-29T14:39:58.506774+0000","last_unstale":"2024-06-29T14:39:59.506909+0000","last_undegraded":"2024-06-29T14:39:59.506909+0000","last_fullsized":"2024-06-29T14:39:59.506909+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:14.884180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"128'4","reported_seq":18,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986710+0000","last_change":"2024-06-29T14:39:57.506787+0000","last_active":"2024-06-29T14:39:59.986710+0000","last_peered":"2024-06-29T14:39:59.986710+0000","last_clean":"2024-06-29T14:39:59.986710+0000","last_became_active":"2024-06-29T14:39:57.506456+0000","last_became_peered":"2024-06-29T14:39:57.506456+0000","last_unstale":"2024-06-29T14:39:59.986710+0000","last_undegraded":"2024-06-29T14:39:59.986710+0000","last_fullsized":"2024-06-29T14:39:59.986710+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:43.544640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539960+0000","last_change":"2024-06-29T14:39:58.515482+0000","last_active":"2024-06-29T14:39:59.539960+0000","last_peered":"2024-06-29T14:39:59.539960+0000","last_clean":"2024-06-29T14:39:59.539960+0000","last_became_active":"2024-06-29T14:39:58.515157+0000","last_became_peered":"2024-06-29T14:39:58.515157+0000","last_unstale":"2024-06-29T14:39:59.539960+0000","last_undegraded":"2024-06-29T14:39:59.539960+0000","last_fullsized":"2024-06-29T14:39:59.539960+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:18.776871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539824+0000","last_change":"2024-06-29T14:39:57.505345+0000","last_active":"2024-06-29T14:39:59.539824+0000","last_peered":"2024-06-29T14:39:59.539824+0000","last_clean":"2024-06-29T14:39:59.539824+0000","last_became_active":"2024-06-29T14:39:57.505048+0000","last_became_peered":"2024-06-29T14:39:57.505048+0000","last_unstale":"2024-06-29T14:39:59.539824+0000","last_undegraded":"2024-06-29T14:39:59.539824+0000","last_fullsized":"2024-06-29T14:39:59.539824+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:57.628691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493175+0000","last_change":"2024-06-29T14:39:58.512570+0000","last_active":"2024-06-29T14:39:59.493175+0000","last_peered":"2024-06-29T14:39:59.493175+0000","last_clean":"2024-06-29T14:39:59.493175+0000","last_became_active":"2024-06-29T14:39:58.510857+0000","last_became_peered":"2024-06-29T14:39:58.510857+0000","last_unstale":"2024-06-29T14:39:59.493175+0000","last_undegraded":"2024-06-29T14:39:59.493175+0000","last_fullsized":"2024-06-29T14:39:59.493175+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:44.690680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493941+0000","last_change":"2024-06-29T14:39:57.503458+0000","last_active":"2024-06-29T14:39:59.493941+0000","last_peered":"2024-06-29T14:39:59.493941+0000","last_clean":"2024-06-29T14:39:59.493941+0000","last_became_active":"2024-06-29T14:39:57.502753+0000","last_became_peered":"2024-06-29T14:39:57.502753+0000","last_unstale":"2024-06-29T14:39:59.493941+0000","last_undegraded":"2024-06-29T14:39:59.493941+0000","last_fullsized":"2024-06-29T14:39:59.493941+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:48.128122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509225+0000","last_change":"2024-06-29T14:39:58.506490+0000","last_active":"2024-06-29T14:39:59.509225+0000","last_peered":"2024-06-29T14:39:59.509225+0000","last_clean":"2024-06-29T14:39:59.509225+0000","last_became_active":"2024-06-29T14:39:58.506147+0000","last_became_peered":"2024-06-29T14:39:58.506147+0000","last_unstale":"2024-06-29T14:39:59.509225+0000","last_undegraded":"2024-06-29T14:39:59.509225+0000","last_fullsized":"2024-06-29T14:39:59.509225+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:30.767893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489024+0000","last_change":"2024-06-29T14:39:57.503888+0000","last_active":"2024-06-29T14:39:59.489024+0000","last_peered":"2024-06-29T14:39:59.489024+0000","last_clean":"2024-06-29T14:39:59.489024+0000","last_became_active":"2024-06-29T14:39:57.503572+0000","last_became_peered":"2024-06-29T14:39:57.503572+0000","last_unstale":"2024-06-29T14:39:59.489024+0000","last_undegraded":"2024-06-29T14:39:59.489024+0000","last_fullsized":"2024-06-29T14:39:59.489024+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:43.886933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986602+0000","last_change":"2024-06-29T14:39:58.498126+0000","last_active":"2024-06-29T14:39:59.986602+0000","last_peered":"2024-06-29T14:39:59.986602+0000","last_clean":"2024-06-29T14:39:59.986602+0000","last_became_active":"2024-06-29T14:39:58.497782+0000","last_became_peered":"2024-06-29T14:39:58.497782+0000","last_unstale":"2024-06-29T14:39:59.986602+0000","last_undegraded":"2024-06-29T14:39:59.986602+0000","last_fullsized":"2024-06-29T14:39:59.986602+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:07:15.431378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986751+0000","last_change":"2024-06-29T14:39:57.504774+0000","last_active":"2024-06-29T14:39:59.986751+0000","last_peered":"2024-06-29T14:39:59.986751+0000","last_clean":"2024-06-29T14:39:59.986751+0000","last_became_active":"2024-06-29T14:39:57.504416+0000","last_became_peered":"2024-06-29T14:39:57.504416+0000","last_unstale":"2024-06-29T14:39:59.986751+0000","last_undegraded":"2024-06-29T14:39:59.986751+0000","last_fullsized":"2024-06-29T14:39:59.986751+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:15.327087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507278+0000","last_change":"2024-06-29T14:39:58.513890+0000","last_active":"2024-06-29T14:39:59.507278+0000","last_peered":"2024-06-29T14:39:59.507278+0000","last_clean":"2024-06-29T14:39:59.507278+0000","last_became_active":"2024-06-29T14:39:58.513565+0000","last_became_peered":"2024-06-29T14:39:58.513565+0000","last_unstale":"2024-06-29T14:39:59.507278+0000","last_undegraded":"2024-06-29T14:39:59.507278+0000","last_fullsized":"2024-06-29T14:39:59.507278+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:57.500067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.508941+0000","last_change":"2024-06-29T14:39:57.507715+0000","last_active":"2024-06-29T14:39:59.508941+0000","last_peered":"2024-06-29T14:39:59.508941+0000","last_clean":"2024-06-29T14:39:59.508941+0000","last_became_active":"2024-06-29T14:39:57.507235+0000","last_became_peered":"2024-06-29T14:39:57.507235+0000","last_unstale":"2024-06-29T14:39:59.508941+0000","last_undegraded":"2024-06-29T14:39:59.508941+0000","last_fullsized":"2024-06-29T14:39:59.508941+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:07:14.422911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494017+0000","last_change":"2024-06-29T14:39:58.509639+0000","last_active":"2024-06-29T14:39:59.494017+0000","last_peered":"2024-06-29T14:39:59.494017+0000","last_clean":"2024-06-29T14:39:59.494017+0000","last_became_active":"2024-06-29T14:39:58.509351+0000","last_became_peered":"2024-06-29T14:39:58.509351+0000","last_unstale":"2024-06-29T14:39:59.494017+0000","last_undegraded":"2024-06-29T14:39:59.494017+0000","last_fullsized":"2024-06-29T14:39:59.494017+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:04:51.367262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492816+0000","last_change":"2024-06-29T14:39:57.505036+0000","last_active":"2024-06-29T14:39:59.492816+0000","last_peered":"2024-06-29T14:39:59.492816+0000","last_clean":"2024-06-29T14:39:59.492816+0000","last_became_active":"2024-06-29T14:39:57.504626+0000","last_became_peered":"2024-06-29T14:39:57.504626+0000","last_unstale":"2024-06-29T14:39:59.492816+0000","last_undegraded":"2024-06-29T14:39:59.492816+0000","last_fullsized":"2024-06-29T14:39:59.492816+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:22.798087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986688+0000","last_change":"2024-06-29T14:39:58.510125+0000","last_active":"2024-06-29T14:39:59.986688+0000","last_peered":"2024-06-29T14:39:59.986688+0000","last_clean":"2024-06-29T14:39:59.986688+0000","last_became_active":"2024-06-29T14:39:58.509223+0000","last_became_peered":"2024-06-29T14:39:58.509223+0000","last_unstale":"2024-06-29T14:39:59.986688+0000","last_undegraded":"2024-06-29T14:39:59.986688+0000","last_fullsized":"2024-06-29T14:39:59.986688+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:49.773813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986230+0000","last_change":"2024-06-29T14:39:57.499002+0000","last_active":"2024-06-29T14:39:59.986230+0000","last_peered":"2024-06-29T14:39:59.986230+0000","last_clean":"2024-06-29T14:39:59.986230+0000","last_became_active":"2024-06-29T14:39:57.498690+0000","last_became_peered":"2024-06-29T14:39:57.498690+0000","last_unstale":"2024-06-29T14:39:59.986230+0000","last_undegraded":"2024-06-29T14:39:59.986230+0000","last_fullsized":"2024-06-29T14:39:59.986230+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:41:24.206994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506749+0000","last_change":"2024-06-29T14:39:58.517426+0000","last_active":"2024-06-29T14:39:59.506749+0000","last_peered":"2024-06-29T14:39:59.506749+0000","last_clean":"2024-06-29T14:39:59.506749+0000","last_became_active":"2024-06-29T14:39:58.516663+0000","last_became_peered":"2024-06-29T14:39:58.516663+0000","last_unstale":"2024-06-29T14:39:59.506749+0000","last_undegraded":"2024-06-29T14:39:59.506749+0000","last_fullsized":"2024-06-29T14:39:59.506749+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:07:27.868633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493572+0000","last_change":"2024-06-29T14:39:57.502293+0000","last_active":"2024-06-29T14:39:59.493572+0000","last_peered":"2024-06-29T14:39:59.493572+0000","last_clean":"2024-06-29T14:39:59.493572+0000","last_became_active":"2024-06-29T14:39:57.501652+0000","last_became_peered":"2024-06-29T14:39:57.501652+0000","last_unstale":"2024-06-29T14:39:59.493572+0000","last_undegraded":"2024-06-29T14:39:59.493572+0000","last_fullsized":"2024-06-29T14:39:59.493572+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:37:45.839396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.488690+0000","last_change":"2024-06-29T14:39:58.498818+0000","last_active":"2024-06-29T14:39:59.488690+0000","last_peered":"2024-06-29T14:39:59.488690+0000","last_clean":"2024-06-29T14:39:59.488690+0000","last_became_active":"2024-06-29T14:39:58.498478+0000","last_became_peered":"2024-06-29T14:39:58.498478+0000","last_unstale":"2024-06-29T14:39:59.488690+0000","last_undegraded":"2024-06-29T14:39:59.488690+0000","last_fullsized":"2024-06-29T14:39:59.488690+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:23.335270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506788+0000","last_change":"2024-06-29T14:39:57.509463+0000","last_active":"2024-06-29T14:39:59.506788+0000","last_peered":"2024-06-29T14:39:59.506788+0000","last_clean":"2024-06-29T14:39:59.506788+0000","last_became_active":"2024-06-29T14:39:57.509148+0000","last_became_peered":"2024-06-29T14:39:57.509148+0000","last_unstale":"2024-06-29T14:39:59.506788+0000","last_undegraded":"2024-06-29T14:39:59.506788+0000","last_fullsized":"2024-06-29T14:39:59.506788+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:19:18.491023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539518+0000","last_change":"2024-06-29T14:39:58.510586+0000","last_active":"2024-06-29T14:39:59.539518+0000","last_peered":"2024-06-29T14:39:59.539518+0000","last_clean":"2024-06-29T14:39:59.539518+0000","last_became_active":"2024-06-29T14:39:58.510220+0000","last_became_peered":"2024-06-29T14:39:58.510220+0000","last_unstale":"2024-06-29T14:39:59.539518+0000","last_undegraded":"2024-06-29T14:39:59.539518+0000","last_fullsized":"2024-06-29T14:39:59.539518+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:37:31.540566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"128'3","reported_seq":17,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986827+0000","last_change":"2024-06-29T14:39:57.504857+0000","last_active":"2024-06-29T14:39:59.986827+0000","last_peered":"2024-06-29T14:39:59.986827+0000","last_clean":"2024-06-29T14:39:59.986827+0000","last_became_active":"2024-06-29T14:39:57.504503+0000","last_became_peered":"2024-06-29T14:39:57.504503+0000","last_unstale":"2024-06-29T14:39:59.986827+0000","last_undegraded":"2024-06-29T14:39:59.986827+0000","last_fullsized":"2024-06-29T14:39:59.986827+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:54.134703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489155+0000","last_change":"2024-06-29T14:39:58.498888+0000","last_active":"2024-06-29T14:39:59.489155+0000","last_peered":"2024-06-29T14:39:59.489155+0000","last_clean":"2024-06-29T14:39:59.489155+0000","last_became_active":"2024-06-29T14:39:58.498520+0000","last_became_peered":"2024-06-29T14:39:58.498520+0000","last_unstale":"2024-06-29T14:39:59.489155+0000","last_undegraded":"2024-06-29T14:39:59.489155+0000","last_fullsized":"2024-06-29T14:39:59.489155+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:07:09.641385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"128'6","reported_seq":20,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494187+0000","last_change":"2024-06-29T14:39:57.510734+0000","last_active":"2024-06-29T14:39:59.494187+0000","last_peered":"2024-06-29T14:39:59.494187+0000","last_clean":"2024-06-29T14:39:59.494187+0000","last_became_active":"2024-06-29T14:39:57.510383+0000","last_became_peered":"2024-06-29T14:39:57.510383+0000","last_unstale":"2024-06-29T14:39:59.494187+0000","last_undegraded":"2024-06-29T14:39:59.494187+0000","last_fullsized":"2024-06-29T14:39:59.494187+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:42.036419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509087+0000","last_change":"2024-06-29T14:39:58.514874+0000","last_active":"2024-06-29T14:39:59.509087+0000","last_peered":"2024-06-29T14:39:59.509087+0000","last_clean":"2024-06-29T14:39:59.509087+0000","last_became_active":"2024-06-29T14:39:58.514576+0000","last_became_peered":"2024-06-29T14:39:58.514576+0000","last_unstale":"2024-06-29T14:39:59.509087+0000","last_undegraded":"2024-06-29T14:39:59.509087+0000","last_fullsized":"2024-06-29T14:39:59.509087+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:05.348037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489318+0000","last_change":"2024-06-29T14:39:57.502783+0000","last_active":"2024-06-29T14:39:59.489318+0000","last_peered":"2024-06-29T14:39:59.489318+0000","last_clean":"2024-06-29T14:39:59.489318+0000","last_became_active":"2024-06-29T14:39:57.501603+0000","last_became_peered":"2024-06-29T14:39:57.501603+0000","last_unstale":"2024-06-29T14:39:59.489318+0000","last_undegraded":"2024-06-29T14:39:59.489318+0000","last_fullsized":"2024-06-29T14:39:59.489318+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:18:17.222919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494326+0000","last_change":"2024-06-29T14:39:58.508361+0000","last_active":"2024-06-29T14:39:59.494326+0000","last_peered":"2024-06-29T14:39:59.494326+0000","last_clean":"2024-06-29T14:39:59.494326+0000","last_became_active":"2024-06-29T14:39:58.507954+0000","last_became_peered":"2024-06-29T14:39:58.507954+0000","last_unstale":"2024-06-29T14:39:59.494326+0000","last_undegraded":"2024-06-29T14:39:59.494326+0000","last_fullsized":"2024-06-29T14:39:59.494326+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:17.412492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986419+0000","last_change":"2024-06-29T14:39:57.508598+0000","last_active":"2024-06-29T14:39:59.986419+0000","last_peered":"2024-06-29T14:39:59.986419+0000","last_clean":"2024-06-29T14:39:59.986419+0000","last_became_active":"2024-06-29T14:39:57.508268+0000","last_became_peered":"2024-06-29T14:39:57.508268+0000","last_unstale":"2024-06-29T14:39:59.986419+0000","last_undegraded":"2024-06-29T14:39:59.986419+0000","last_fullsized":"2024-06-29T14:39:59.986419+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:19:18.491023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539711+0000","last_change":"2024-06-29T14:39:58.501478+0000","last_active":"2024-06-29T14:39:59.539711+0000","last_peered":"2024-06-29T14:39:59.539711+0000","last_clean":"2024-06-29T14:39:59.539711+0000","last_became_active":"2024-06-29T14:39:58.501162+0000","last_became_peered":"2024-06-29T14:39:58.501162+0000","last_unstale":"2024-06-29T14:39:59.539711+0000","last_undegraded":"2024-06-29T14:39:59.539711+0000","last_fullsized":"2024-06-29T14:39:59.539711+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:55.781777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539590+0000","last_change":"2024-06-29T14:39:57.504008+0000","last_active":"2024-06-29T14:39:59.539590+0000","last_peered":"2024-06-29T14:39:59.539590+0000","last_clean":"2024-06-29T14:39:59.539590+0000","last_became_active":"2024-06-29T14:39:57.503722+0000","last_became_peered":"2024-06-29T14:39:57.503722+0000","last_unstale":"2024-06-29T14:39:59.539590+0000","last_undegraded":"2024-06-29T14:39:59.539590+0000","last_fullsized":"2024-06-29T14:39:59.539590+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:47:40.810306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507213+0000","last_change":"2024-06-29T14:39:58.520203+0000","last_active":"2024-06-29T14:39:59.507213+0000","last_peered":"2024-06-29T14:39:59.507213+0000","last_clean":"2024-06-29T14:39:59.507213+0000","last_became_active":"2024-06-29T14:39:58.519860+0000","last_became_peered":"2024-06-29T14:39:58.519860+0000","last_unstale":"2024-06-29T14:39:59.507213+0000","last_undegraded":"2024-06-29T14:39:59.507213+0000","last_fullsized":"2024-06-29T14:39:59.507213+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:23.420360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"128'8","reported_seq":22,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492757+0000","last_change":"2024-06-29T14:39:57.502251+0000","last_active":"2024-06-29T14:39:59.492757+0000","last_peered":"2024-06-29T14:39:59.492757+0000","last_clean":"2024-06-29T14:39:59.492757+0000","last_became_active":"2024-06-29T14:39:57.501389+0000","last_became_peered":"2024-06-29T14:39:57.501389+0000","last_unstale":"2024-06-29T14:39:59.492757+0000","last_undegraded":"2024-06-29T14:39:59.492757+0000","last_fullsized":"2024-06-29T14:39:59.492757+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:28:46.311128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539779+0000","last_change":"2024-06-29T14:39:58.509671+0000","last_active":"2024-06-29T14:39:59.539779+0000","last_peered":"2024-06-29T14:39:59.539779+0000","last_clean":"2024-06-29T14:39:59.539779+0000","last_became_active":"2024-06-29T14:39:58.509357+0000","last_became_peered":"2024-06-29T14:39:58.509357+0000","last_unstale":"2024-06-29T14:39:59.539779+0000","last_undegraded":"2024-06-29T14:39:59.539779+0000","last_fullsized":"2024-06-29T14:39:59.539779+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:50.917154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494267+0000","last_change":"2024-06-29T14:39:57.504227+0000","last_active":"2024-06-29T14:39:59.494267+0000","last_peered":"2024-06-29T14:39:59.494267+0000","last_clean":"2024-06-29T14:39:59.494267+0000","last_became_active":"2024-06-29T14:39:57.503904+0000","last_became_peered":"2024-06-29T14:39:57.503904+0000","last_unstale":"2024-06-29T14:39:59.494267+0000","last_undegraded":"2024-06-29T14:39:59.494267+0000","last_fullsized":"2024-06-29T14:39:59.494267+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:48:21.626126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494115+0000","last_change":"2024-06-29T14:39:58.508416+0000","last_active":"2024-06-29T14:39:59.494115+0000","last_peered":"2024-06-29T14:39:59.494115+0000","last_clean":"2024-06-29T14:39:59.494115+0000","last_became_active":"2024-06-29T14:39:58.507931+0000","last_became_peered":"2024-06-29T14:39:58.507931+0000","last_unstale":"2024-06-29T14:39:59.494115+0000","last_undegraded":"2024-06-29T14:39:59.494115+0000","last_fullsized":"2024-06-29T14:39:59.494115+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:21.563876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986929+0000","last_change":"2024-06-29T14:39:57.501506+0000","last_active":"2024-06-29T14:39:59.986929+0000","last_peered":"2024-06-29T14:39:59.986929+0000","last_clean":"2024-06-29T14:39:59.986929+0000","last_became_active":"2024-06-29T14:39:57.501135+0000","last_became_peered":"2024-06-29T14:39:57.501135+0000","last_unstale":"2024-06-29T14:39:59.986929+0000","last_undegraded":"2024-06-29T14:39:59.986929+0000","last_fullsized":"2024-06-29T14:39:59.986929+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:00.440766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.488945+0000","last_change":"2024-06-29T14:39:58.499715+0000","last_active":"2024-06-29T14:39:59.488945+0000","last_peered":"2024-06-29T14:39:59.488945+0000","last_clean":"2024-06-29T14:39:59.488945+0000","last_became_active":"2024-06-29T14:39:58.499339+0000","last_became_peered":"2024-06-29T14:39:58.499339+0000","last_unstale":"2024-06-29T14:39:59.488945+0000","last_undegraded":"2024-06-29T14:39:59.488945+0000","last_fullsized":"2024-06-29T14:39:59.488945+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:34.015450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509055+0000","last_change":"2024-06-29T14:39:57.507850+0000","last_active":"2024-06-29T14:39:59.509055+0000","last_peered":"2024-06-29T14:39:59.509055+0000","last_clean":"2024-06-29T14:39:59.509055+0000","last_became_active":"2024-06-29T14:39:57.507261+0000","last_became_peered":"2024-06-29T14:39:57.507261+0000","last_unstale":"2024-06-29T14:39:59.509055+0000","last_undegraded":"2024-06-29T14:39:59.509055+0000","last_fullsized":"2024-06-29T14:39:59.509055+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:26.962456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506914+0000","last_change":"2024-06-29T14:39:58.523318+0000","last_active":"2024-06-29T14:39:59.506914+0000","last_peered":"2024-06-29T14:39:59.506914+0000","last_clean":"2024-06-29T14:39:59.506914+0000","last_became_active":"2024-06-29T14:39:58.522946+0000","last_became_peered":"2024-06-29T14:39:58.522946+0000","last_unstale":"2024-06-29T14:39:59.506914+0000","last_undegraded":"2024-06-29T14:39:59.506914+0000","last_fullsized":"2024-06-29T14:39:59.506914+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:30.310799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539959+0000","last_change":"2024-06-29T14:39:57.506162+0000","last_active":"2024-06-29T14:39:59.539959+0000","last_peered":"2024-06-29T14:39:59.539959+0000","last_clean":"2024-06-29T14:39:59.539959+0000","last_became_active":"2024-06-29T14:39:57.504914+0000","last_became_peered":"2024-06-29T14:39:57.504914+0000","last_unstale":"2024-06-29T14:39:59.539959+0000","last_undegraded":"2024-06-29T14:39:59.539959+0000","last_fullsized":"2024-06-29T14:39:59.539959+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:33.841274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507260+0000","last_change":"2024-06-29T14:39:58.511527+0000","last_active":"2024-06-29T14:39:59.507260+0000","last_peered":"2024-06-29T14:39:59.507260+0000","last_clean":"2024-06-29T14:39:59.507260+0000","last_became_active":"2024-06-29T14:39:58.511188+0000","last_became_peered":"2024-06-29T14:39:58.511188+0000","last_unstale":"2024-06-29T14:39:59.507260+0000","last_undegraded":"2024-06-29T14:39:59.507260+0000","last_fullsized":"2024-06-29T14:39:59.507260+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:35.075707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539637+0000","last_change":"2024-06-29T14:39:57.504730+0000","last_active":"2024-06-29T14:39:59.539637+0000","last_peered":"2024-06-29T14:39:59.539637+0000","last_clean":"2024-06-29T14:39:59.539637+0000","last_became_active":"2024-06-29T14:39:57.504088+0000","last_became_peered":"2024-06-29T14:39:57.504088+0000","last_unstale":"2024-06-29T14:39:59.539637+0000","last_undegraded":"2024-06-29T14:39:59.539637+0000","last_fullsized":"2024-06-29T14:39:59.539637+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:09.177229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986721+0000","last_change":"2024-06-29T14:39:58.505928+0000","last_active":"2024-06-29T14:39:59.986721+0000","last_peered":"2024-06-29T14:39:59.986721+0000","last_clean":"2024-06-29T14:39:59.986721+0000","last_became_active":"2024-06-29T14:39:58.504876+0000","last_became_peered":"2024-06-29T14:39:58.504876+0000","last_unstale":"2024-06-29T14:39:59.986721+0000","last_undegraded":"2024-06-29T14:39:59.986721+0000","last_fullsized":"2024-06-29T14:39:59.986721+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:31:31.162145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494436+0000","last_change":"2024-06-29T14:39:57.500171+0000","last_active":"2024-06-29T14:39:59.494436+0000","last_peered":"2024-06-29T14:39:59.494436+0000","last_clean":"2024-06-29T14:39:59.494436+0000","last_became_active":"2024-06-29T14:39:57.499804+0000","last_became_peered":"2024-06-29T14:39:57.499804+0000","last_unstale":"2024-06-29T14:39:59.494436+0000","last_undegraded":"2024-06-29T14:39:59.494436+0000","last_fullsized":"2024-06-29T14:39:59.494436+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:12.743273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539673+0000","last_change":"2024-06-29T14:39:58.511398+0000","last_active":"2024-06-29T14:39:59.539673+0000","last_peered":"2024-06-29T14:39:59.539673+0000","last_clean":"2024-06-29T14:39:59.539673+0000","last_became_active":"2024-06-29T14:39:58.511074+0000","last_became_peered":"2024-06-29T14:39:58.511074+0000","last_unstale":"2024-06-29T14:39:59.539673+0000","last_undegraded":"2024-06-29T14:39:59.539673+0000","last_fullsized":"2024-06-29T14:39:59.539673+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:52.355286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539877+0000","last_change":"2024-06-29T14:39:57.505691+0000","last_active":"2024-06-29T14:39:59.539877+0000","last_peered":"2024-06-29T14:39:59.539877+0000","last_clean":"2024-06-29T14:39:59.539877+0000","last_became_active":"2024-06-29T14:39:57.505157+0000","last_became_peered":"2024-06-29T14:39:57.505157+0000","last_unstale":"2024-06-29T14:39:59.539877+0000","last_undegraded":"2024-06-29T14:39:59.539877+0000","last_fullsized":"2024-06-29T14:39:59.539877+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:11:56.397418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506718+0000","last_change":"2024-06-29T14:39:58.520282+0000","last_active":"2024-06-29T14:39:59.506718+0000","last_peered":"2024-06-29T14:39:59.506718+0000","last_clean":"2024-06-29T14:39:59.506718+0000","last_became_active":"2024-06-29T14:39:58.519984+0000","last_became_peered":"2024-06-29T14:39:58.519984+0000","last_unstale":"2024-06-29T14:39:59.506718+0000","last_undegraded":"2024-06-29T14:39:59.506718+0000","last_fullsized":"2024-06-29T14:39:59.506718+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:31:31.162145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489190+0000","last_change":"2024-06-29T14:39:57.501627+0000","last_active":"2024-06-29T14:39:59.489190+0000","last_peered":"2024-06-29T14:39:59.489190+0000","last_clean":"2024-06-29T14:39:59.489190+0000","last_became_active":"2024-06-29T14:39:57.501274+0000","last_became_peered":"2024-06-29T14:39:57.501274+0000","last_unstale":"2024-06-29T14:39:59.489190+0000","last_undegraded":"2024-06-29T14:39:59.489190+0000","last_fullsized":"2024-06-29T14:39:59.489190+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:28:28.094143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492875+0000","last_change":"2024-06-29T14:39:58.506728+0000","last_active":"2024-06-29T14:39:59.492875+0000","last_peered":"2024-06-29T14:39:59.492875+0000","last_clean":"2024-06-29T14:39:59.492875+0000","last_became_active":"2024-06-29T14:39:58.506400+0000","last_became_peered":"2024-06-29T14:39:58.506400+0000","last_unstale":"2024-06-29T14:39:59.492875+0000","last_undegraded":"2024-06-29T14:39:59.492875+0000","last_fullsized":"2024-06-29T14:39:59.492875+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:12.330677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"129'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.508408+0000","last_change":"2024-06-29T14:39:57.507418+0000","last_active":"2024-06-29T14:39:59.508408+0000","last_peered":"2024-06-29T14:39:59.508408+0000","last_clean":"2024-06-29T14:39:59.508408+0000","last_became_active":"2024-06-29T14:39:57.507093+0000","last_became_peered":"2024-06-29T14:39:57.507093+0000","last_unstale":"2024-06-29T14:39:59.508408+0000","last_undegraded":"2024-06-29T14:39:59.508408+0000","last_fullsized":"2024-06-29T14:39:59.508408+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:43.544640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986395+0000","last_change":"2024-06-29T14:39:58.506238+0000","last_active":"2024-06-29T14:39:59.986395+0000","last_peered":"2024-06-29T14:39:59.986395+0000","last_clean":"2024-06-29T14:39:59.986395+0000","last_became_active":"2024-06-29T14:39:58.504550+0000","last_became_peered":"2024-06-29T14:39:58.504550+0000","last_unstale":"2024-06-29T14:39:59.986395+0000","last_undegraded":"2024-06-29T14:39:59.986395+0000","last_fullsized":"2024-06-29T14:39:59.986395+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:50.697755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494287+0000","last_change":"2024-06-29T14:39:57.511540+0000","last_active":"2024-06-29T14:39:59.494287+0000","last_peered":"2024-06-29T14:39:59.494287+0000","last_clean":"2024-06-29T14:39:59.494287+0000","last_became_active":"2024-06-29T14:39:57.511270+0000","last_became_peered":"2024-06-29T14:39:57.511270+0000","last_unstale":"2024-06-29T14:39:59.494287+0000","last_undegraded":"2024-06-29T14:39:59.494287+0000","last_fullsized":"2024-06-29T14:39:59.494287+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:23.317944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509247+0000","last_change":"2024-06-29T14:39:58.513690+0000","last_active":"2024-06-29T14:39:59.509247+0000","last_peered":"2024-06-29T14:39:59.509247+0000","last_clean":"2024-06-29T14:39:59.509247+0000","last_became_active":"2024-06-29T14:39:58.513016+0000","last_became_peered":"2024-06-29T14:39:58.513016+0000","last_unstale":"2024-06-29T14:39:59.509247+0000","last_undegraded":"2024-06-29T14:39:59.509247+0000","last_fullsized":"2024-06-29T14:39:59.509247+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:23.420360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489083+0000","last_change":"2024-06-29T14:39:57.504751+0000","last_active":"2024-06-29T14:39:59.489083+0000","last_peered":"2024-06-29T14:39:59.489083+0000","last_clean":"2024-06-29T14:39:59.489083+0000","last_became_active":"2024-06-29T14:39:57.504435+0000","last_became_peered":"2024-06-29T14:39:57.504435+0000","last_unstale":"2024-06-29T14:39:59.489083+0000","last_undegraded":"2024-06-29T14:39:59.489083+0000","last_fullsized":"2024-06-29T14:39:59.489083+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:30.390652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539772+0000","last_change":"2024-06-29T14:39:58.495560+0000","last_active":"2024-06-29T14:39:59.539772+0000","last_peered":"2024-06-29T14:39:59.539772+0000","last_clean":"2024-06-29T14:39:59.539772+0000","last_became_active":"2024-06-29T14:39:58.495237+0000","last_became_peered":"2024-06-29T14:39:58.495237+0000","last_unstale":"2024-06-29T14:39:59.539772+0000","last_undegraded":"2024-06-29T14:39:59.539772+0000","last_fullsized":"2024-06-29T14:39:59.539772+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:08.486896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493136+0000","last_change":"2024-06-29T14:39:57.501055+0000","last_active":"2024-06-29T14:39:59.493136+0000","last_peered":"2024-06-29T14:39:59.493136+0000","last_clean":"2024-06-29T14:39:59.493136+0000","last_became_active":"2024-06-29T14:39:57.500420+0000","last_became_peered":"2024-06-29T14:39:57.500420+0000","last_unstale":"2024-06-29T14:39:59.493136+0000","last_undegraded":"2024-06-29T14:39:59.493136+0000","last_fullsized":"2024-06-29T14:39:59.493136+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:30.374480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539808+0000","last_change":"2024-06-29T14:39:58.516176+0000","last_active":"2024-06-29T14:39:59.539808+0000","last_peered":"2024-06-29T14:39:59.539808+0000","last_clean":"2024-06-29T14:39:59.539808+0000","last_became_active":"2024-06-29T14:39:58.515859+0000","last_became_peered":"2024-06-29T14:39:58.515859+0000","last_unstale":"2024-06-29T14:39:59.539808+0000","last_undegraded":"2024-06-29T14:39:59.539808+0000","last_fullsized":"2024-06-29T14:39:59.539808+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:43:26.292133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539660+0000","last_change":"2024-06-29T14:39:57.504470+0000","last_active":"2024-06-29T14:39:59.539660+0000","last_peered":"2024-06-29T14:39:59.539660+0000","last_clean":"2024-06-29T14:39:59.539660+0000","last_became_active":"2024-06-29T14:39:57.503659+0000","last_became_peered":"2024-06-29T14:39:57.503659+0000","last_unstale":"2024-06-29T14:39:59.539660+0000","last_undegraded":"2024-06-29T14:39:59.539660+0000","last_fullsized":"2024-06-29T14:39:59.539660+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:04.365120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986903+0000","last_change":"2024-06-29T14:39:58.516028+0000","last_active":"2024-06-29T14:39:59.986903+0000","last_peered":"2024-06-29T14:39:59.986903+0000","last_clean":"2024-06-29T14:39:59.986903+0000","last_became_active":"2024-06-29T14:39:58.515647+0000","last_became_peered":"2024-06-29T14:39:58.515647+0000","last_unstale":"2024-06-29T14:39:59.986903+0000","last_undegraded":"2024-06-29T14:39:59.986903+0000","last_fullsized":"2024-06-29T14:39:59.986903+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:12.317950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494542+0000","last_change":"2024-06-29T14:39:57.503907+0000","last_active":"2024-06-29T14:39:59.494542+0000","last_peered":"2024-06-29T14:39:59.494542+0000","last_clean":"2024-06-29T14:39:59.494542+0000","last_became_active":"2024-06-29T14:39:57.503557+0000","last_became_peered":"2024-06-29T14:39:57.503557+0000","last_unstale":"2024-06-29T14:39:59.494542+0000","last_undegraded":"2024-06-29T14:39:59.494542+0000","last_fullsized":"2024-06-29T14:39:59.494542+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:18:07.624984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509389+0000","last_change":"2024-06-29T14:39:58.507541+0000","last_active":"2024-06-29T14:39:59.509389+0000","last_peered":"2024-06-29T14:39:59.509389+0000","last_clean":"2024-06-29T14:39:59.509389+0000","last_became_active":"2024-06-29T14:39:58.507240+0000","last_became_peered":"2024-06-29T14:39:58.507240+0000","last_unstale":"2024-06-29T14:39:59.509389+0000","last_undegraded":"2024-06-29T14:39:59.509389+0000","last_fullsized":"2024-06-29T14:39:59.509389+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:57.500067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539952+0000","last_change":"2024-06-29T14:39:57.497210+0000","last_active":"2024-06-29T14:39:59.539952+0000","last_peered":"2024-06-29T14:39:59.539952+0000","last_clean":"2024-06-29T14:39:59.539952+0000","last_became_active":"2024-06-29T14:39:57.496895+0000","last_became_peered":"2024-06-29T14:39:57.496895+0000","last_unstale":"2024-06-29T14:39:59.539952+0000","last_undegraded":"2024-06-29T14:39:59.539952+0000","last_fullsized":"2024-06-29T14:39:59.539952+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:29.163307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.508982+0000","last_change":"2024-06-29T14:39:58.513738+0000","last_active":"2024-06-29T14:39:59.508982+0000","last_peered":"2024-06-29T14:39:59.508982+0000","last_clean":"2024-06-29T14:39:59.508982+0000","last_became_active":"2024-06-29T14:39:58.511414+0000","last_became_peered":"2024-06-29T14:39:58.511414+0000","last_unstale":"2024-06-29T14:39:59.508982+0000","last_undegraded":"2024-06-29T14:39:59.508982+0000","last_fullsized":"2024-06-29T14:39:59.508982+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:03.881878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539956+0000","last_change":"2024-06-29T14:39:57.500699+0000","last_active":"2024-06-29T14:39:59.539956+0000","last_peered":"2024-06-29T14:39:59.539956+0000","last_clean":"2024-06-29T14:39:59.539956+0000","last_became_active":"2024-06-29T14:39:57.500405+0000","last_became_peered":"2024-06-29T14:39:57.500405+0000","last_unstale":"2024-06-29T14:39:59.539956+0000","last_undegraded":"2024-06-29T14:39:59.539956+0000","last_fullsized":"2024-06-29T14:39:59.539956+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:01.456219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"118'142","reported_seq":569,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986386+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:39:59.986386+0000","last_peered":"2024-06-29T14:39:59.986386+0000","last_clean":"2024-06-29T14:39:59.986386+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:39:59.986386+0000","last_undegraded":"2024-06-29T14:39:59.986386+0000","last_fullsized":"2024-06-29T14:39:59.986386+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":221,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":221,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542363,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59676,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312164,"statfs":{"total":96636764160,"available":96575655936,"internally_reserved":0,"allocated":2179072,"data_stored":1387958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542364,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1597440,"data_stored":799646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54932,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316908,"statfs":{"total":96636764160,"available":96580513792,"internally_reserved":0,"allocated":1576960,"data_stored":797430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542362,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1597440,"data_stored":804214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509660,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36892,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334948,"statfs":{"total":96636764160,"available":96598986752,"internally_reserved":0,"allocated":1581056,"data_stored":798012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575068,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59112,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312728,"statfs":{"total":96636764160,"available":96576233472,"internally_reserved":0,"allocated":1593344,"data_stored":804156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509659,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1581056,"data_stored":797464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509661,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37492,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334348,"statfs":{"total":96636764160,"available":96598372352,"internally_reserved":0,"allocated":2199552,"data_stored":1396192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:40:02.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:40:02.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.735+0000 7f09d9681640 1 -- 172.21.15.59:0/3140719907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 msgr2=0x7f09d40ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.735+0000 7f09d9681640 1 --2- 172.21.15.59:0/3140719907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 0x7f09d40ce110 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f09c4005fd0 tx=0x7f09c402a530 comp rx=0 tx=0).stop 2024-06-29T14:40:02.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.735+0000 7f09d9681640 1 -- 172.21.15.59:0/3140719907 shutdown_connections 2024-06-29T14:40:02.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.735+0000 7f09d9681640 1 --2- 172.21.15.59:0/3140719907 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09d40b40a0 0x7f09d40b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.735+0000 7f09d9681640 1 --2- 172.21.15.59:0/3140719907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 0x7f09d40ce110 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.735+0000 7f09d9681640 1 -- 172.21.15.59:0/3140719907 >> 172.21.15.59:0/3140719907 conn(0x7f09d40d3ee0 msgr2=0x7f09d40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.735+0000 7f09d9681640 1 -- 172.21.15.59:0/3140719907 shutdown_connections 2024-06-29T14:40:02.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.735+0000 7f09d9681640 1 -- 172.21.15.59:0/3140719907 wait complete. 2024-06-29T14:40:02.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.737+0000 7f09d9681640 1 Processor -- start 2024-06-29T14:40:02.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.737+0000 7f09d9681640 1 -- start start 2024-06-29T14:40:02.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09d9681640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09d40ce650 0x7f09d40b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09d9681640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 0x7f09d4153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09d9681640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09d4153890 0x7f09d4159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09d9681640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f09d40b7cc0 con 0x7f09d415e290 2024-06-29T14:40:02.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09d9681640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09d40b7b40 con 0x7f09d4153890 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09d9681640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f09d40b7e40 con 0x7f09d40ce650 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 0x7f09d4153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 0x7f09d4153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56916/0 (socket says 172.21.15.59:56916) 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 -- 172.21.15.59:0/4121042516 learned_addr learned my addr 172.21.15.59:0/4121042516 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09d40ce650 0x7f09d40b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09d37fe640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09d4153890 0x7f09d4159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 -- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09d40ce650 msgr2=0x7f09d40b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09d40ce650 0x7f09d40b1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 -- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09d4153890 msgr2=0x7f09d4159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09d4153890 0x7f09d4159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 -- 172.21.15.59:0/4121042516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d41f52e0 con 0x7f09d415e290 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09ca7fc640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 0x7f09d4153350 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009fd0 tx=0x7f09bc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.739+0000 7f09cbfff640 1 -- 172.21.15.59:0/4121042516 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc015020 con 0x7f09d415e290 2024-06-29T14:40:02.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.740+0000 7f09cbfff640 1 -- 172.21.15.59:0/4121042516 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09bc011920 con 0x7f09d415e290 2024-06-29T14:40:02.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.740+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d41f55c0 con 0x7f09d415e290 2024-06-29T14:40:02.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.740+0000 7f09cbfff640 1 -- 172.21.15.59:0/4121042516 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc011d50 con 0x7f09d415e290 2024-06-29T14:40:02.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.740+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d41f5b80 con 0x7f09d415e290 2024-06-29T14:40:02.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.741+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09d40b8b60 con 0x7f09d415e290 2024-06-29T14:40:02.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.741+0000 7f09cbfff640 1 -- 172.21.15.59:0/4121042516 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f09bc018020 con 0x7f09d415e290 2024-06-29T14:40:02.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.742+0000 7f09cbfff640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f09b4076420 0x7f09b4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.742+0000 7f09cbfff640 1 -- 172.21.15.59:0/4121042516 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f09bc097590 con 0x7f09d415e290 2024-06-29T14:40:02.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.742+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f09b4076420 0x7f09b4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.743+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f09b4076420 0x7f09b4078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f09c4033000 tx=0x7f09c4004730 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.747+0000 7f09cbfff640 1 -- 172.21.15.59:0/4121042516 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f09bc01f030 con 0x7f09d415e290 2024-06-29T14:40:02.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.937+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 --> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f09d41f5d50 con 0x7f09b4076420 2024-06-29T14:40:02.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.942+0000 7f09cbfff640 1 -- 172.21.15.59:0/4121042516 <== mgr.7615 v2:172.21.15.153:6832/892574025 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252487 (secure 0 0 0) 0x7f09c40055e0 con 0x7f09b4076420 2024-06-29T14:40:02.942 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:40:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:40:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f09b4076420 msgr2=0x7f09b4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f09b4076420 0x7f09b4078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f09c4033000 tx=0x7f09c4004730 comp rx=0 tx=0).stop 2024-06-29T14:40:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 msgr2=0x7f09d4153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 0x7f09d4153350 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009fd0 tx=0x7f09bc00b870 comp rx=0 tx=0).stop 2024-06-29T14:40:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 shutdown_connections 2024-06-29T14:40:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09d4153890 0x7f09d4159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f09b4076420 0x7f09b4078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415e290 0x7f09d4153350 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 --2- 172.21.15.59:0/4121042516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09d40ce650 0x7f09d40b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 >> 172.21.15.59:0/4121042516 conn(0x7f09d40d3ee0 msgr2=0x7f09d415b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 shutdown_connections 2024-06-29T14:40:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:02.945+0000 7f09d9681640 1 -- 172.21.15.59:0/4121042516 wait complete. 2024-06-29T14:40:02.955 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:40:01.992960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":266,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":350480,"kb_used_data":13580,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754624240,"statfs":{"total":773094113280,"available":772735221760,"internally_reserved":0,"allocated":13905920,"data_stored":7585072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":344837682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.516125"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507016+0000","last_change":"2024-06-29T14:39:58.522340+0000","last_active":"2024-06-29T14:39:59.507016+0000","last_peered":"2024-06-29T14:39:59.507016+0000","last_clean":"2024-06-29T14:39:59.507016+0000","last_became_active":"2024-06-29T14:39:58.521203+0000","last_became_peered":"2024-06-29T14:39:58.521203+0000","last_unstale":"2024-06-29T14:39:59.507016+0000","last_undegraded":"2024-06-29T14:39:59.507016+0000","last_fullsized":"2024-06-29T14:39:59.507016+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:03.881878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493801+0000","last_change":"2024-06-29T14:39:58.506632+0000","last_active":"2024-06-29T14:39:59.493801+0000","last_peered":"2024-06-29T14:39:59.493801+0000","last_clean":"2024-06-29T14:39:59.493801+0000","last_became_active":"2024-06-29T14:39:58.506327+0000","last_became_peered":"2024-06-29T14:39:58.506327+0000","last_unstale":"2024-06-29T14:39:59.493801+0000","last_undegraded":"2024-06-29T14:39:59.493801+0000","last_fullsized":"2024-06-29T14:39:59.493801+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:23.806553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507081+0000","last_change":"2024-06-29T14:39:58.523645+0000","last_active":"2024-06-29T14:39:59.507081+0000","last_peered":"2024-06-29T14:39:59.507081+0000","last_clean":"2024-06-29T14:39:59.507081+0000","last_became_active":"2024-06-29T14:39:58.523242+0000","last_became_peered":"2024-06-29T14:39:58.523242+0000","last_unstale":"2024-06-29T14:39:59.507081+0000","last_undegraded":"2024-06-29T14:39:59.507081+0000","last_fullsized":"2024-06-29T14:39:59.507081+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:05.348037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986720+0000","last_change":"2024-06-29T14:39:58.503281+0000","last_active":"2024-06-29T14:39:59.986720+0000","last_peered":"2024-06-29T14:39:59.986720+0000","last_clean":"2024-06-29T14:39:59.986720+0000","last_became_active":"2024-06-29T14:39:58.502960+0000","last_became_peered":"2024-06-29T14:39:58.502960+0000","last_unstale":"2024-06-29T14:39:59.986720+0000","last_undegraded":"2024-06-29T14:39:59.986720+0000","last_fullsized":"2024-06-29T14:39:59.986720+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:08:45.202818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489008+0000","last_change":"2024-06-29T14:39:58.500771+0000","last_active":"2024-06-29T14:39:59.489008+0000","last_peered":"2024-06-29T14:39:59.489008+0000","last_clean":"2024-06-29T14:39:59.489008+0000","last_became_active":"2024-06-29T14:39:58.500448+0000","last_became_peered":"2024-06-29T14:39:58.500448+0000","last_unstale":"2024-06-29T14:39:59.489008+0000","last_undegraded":"2024-06-29T14:39:59.489008+0000","last_fullsized":"2024-06-29T14:39:59.489008+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:07:27.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494192+0000","last_change":"2024-06-29T14:39:58.499653+0000","last_active":"2024-06-29T14:39:59.494192+0000","last_peered":"2024-06-29T14:39:59.494192+0000","last_clean":"2024-06-29T14:39:59.494192+0000","last_became_active":"2024-06-29T14:39:58.498677+0000","last_became_peered":"2024-06-29T14:39:58.498677+0000","last_unstale":"2024-06-29T14:39:59.494192+0000","last_undegraded":"2024-06-29T14:39:59.494192+0000","last_fullsized":"2024-06-29T14:39:59.494192+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:48.747706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494312+0000","last_change":"2024-06-29T14:39:58.513104+0000","last_active":"2024-06-29T14:39:59.494312+0000","last_peered":"2024-06-29T14:39:59.494312+0000","last_clean":"2024-06-29T14:39:59.494312+0000","last_became_active":"2024-06-29T14:39:58.511342+0000","last_became_peered":"2024-06-29T14:39:58.511342+0000","last_unstale":"2024-06-29T14:39:59.494312+0000","last_undegraded":"2024-06-29T14:39:59.494312+0000","last_fullsized":"2024-06-29T14:39:59.494312+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:09.448042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539440+0000","last_change":"2024-06-29T14:39:58.510498+0000","last_active":"2024-06-29T14:39:59.539440+0000","last_peered":"2024-06-29T14:39:59.539440+0000","last_clean":"2024-06-29T14:39:59.539440+0000","last_became_active":"2024-06-29T14:39:58.510139+0000","last_became_peered":"2024-06-29T14:39:58.510139+0000","last_unstale":"2024-06-29T14:39:59.539440+0000","last_undegraded":"2024-06-29T14:39:59.539440+0000","last_fullsized":"2024-06-29T14:39:59.539440+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:31.834229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494086+0000","last_change":"2024-06-29T14:39:58.512240+0000","last_active":"2024-06-29T14:39:59.494086+0000","last_peered":"2024-06-29T14:39:59.494086+0000","last_clean":"2024-06-29T14:39:59.494086+0000","last_became_active":"2024-06-29T14:39:58.510727+0000","last_became_peered":"2024-06-29T14:39:58.510727+0000","last_unstale":"2024-06-29T14:39:59.494086+0000","last_undegraded":"2024-06-29T14:39:59.494086+0000","last_fullsized":"2024-06-29T14:39:59.494086+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:53.606347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489152+0000","last_change":"2024-06-29T14:39:58.503141+0000","last_active":"2024-06-29T14:39:59.489152+0000","last_peered":"2024-06-29T14:39:59.489152+0000","last_clean":"2024-06-29T14:39:59.489152+0000","last_became_active":"2024-06-29T14:39:58.502682+0000","last_became_peered":"2024-06-29T14:39:58.502682+0000","last_unstale":"2024-06-29T14:39:59.489152+0000","last_undegraded":"2024-06-29T14:39:59.489152+0000","last_fullsized":"2024-06-29T14:39:59.489152+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:26:36.492357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509194+0000","last_change":"2024-06-29T14:39:58.516336+0000","last_active":"2024-06-29T14:39:59.509194+0000","last_peered":"2024-06-29T14:39:59.509194+0000","last_clean":"2024-06-29T14:39:59.509194+0000","last_became_active":"2024-06-29T14:39:58.515090+0000","last_became_peered":"2024-06-29T14:39:58.515090+0000","last_unstale":"2024-06-29T14:39:59.509194+0000","last_undegraded":"2024-06-29T14:39:59.509194+0000","last_fullsized":"2024-06-29T14:39:59.509194+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:14:57.642394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492890+0000","last_change":"2024-06-29T14:39:58.505973+0000","last_active":"2024-06-29T14:39:59.492890+0000","last_peered":"2024-06-29T14:39:59.492890+0000","last_clean":"2024-06-29T14:39:59.492890+0000","last_became_active":"2024-06-29T14:39:58.505477+0000","last_became_peered":"2024-06-29T14:39:58.505477+0000","last_unstale":"2024-06-29T14:39:59.492890+0000","last_undegraded":"2024-06-29T14:39:59.492890+0000","last_fullsized":"2024-06-29T14:39:59.492890+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:23:48.378022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986435+0000","last_change":"2024-06-29T14:39:58.514529+0000","last_active":"2024-06-29T14:39:59.986435+0000","last_peered":"2024-06-29T14:39:59.986435+0000","last_clean":"2024-06-29T14:39:59.986435+0000","last_became_active":"2024-06-29T14:39:58.514139+0000","last_became_peered":"2024-06-29T14:39:58.514139+0000","last_unstale":"2024-06-29T14:39:59.986435+0000","last_undegraded":"2024-06-29T14:39:59.986435+0000","last_fullsized":"2024-06-29T14:39:59.986435+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:27.970923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986733+0000","last_change":"2024-06-29T14:39:58.517145+0000","last_active":"2024-06-29T14:39:59.986733+0000","last_peered":"2024-06-29T14:39:59.986733+0000","last_clean":"2024-06-29T14:39:59.986733+0000","last_became_active":"2024-06-29T14:39:58.516817+0000","last_became_peered":"2024-06-29T14:39:58.516817+0000","last_unstale":"2024-06-29T14:39:59.986733+0000","last_undegraded":"2024-06-29T14:39:59.986733+0000","last_fullsized":"2024-06-29T14:39:59.986733+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:25:19.271179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986641+0000","last_change":"2024-06-29T14:39:58.508090+0000","last_active":"2024-06-29T14:39:59.986641+0000","last_peered":"2024-06-29T14:39:59.986641+0000","last_clean":"2024-06-29T14:39:59.986641+0000","last_became_active":"2024-06-29T14:39:58.507749+0000","last_became_peered":"2024-06-29T14:39:58.507749+0000","last_unstale":"2024-06-29T14:39:59.986641+0000","last_undegraded":"2024-06-29T14:39:59.986641+0000","last_fullsized":"2024-06-29T14:39:59.986641+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:41:26.853858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986622+0000","last_change":"2024-06-29T14:39:58.512353+0000","last_active":"2024-06-29T14:39:59.986622+0000","last_peered":"2024-06-29T14:39:59.986622+0000","last_clean":"2024-06-29T14:39:59.986622+0000","last_became_active":"2024-06-29T14:39:58.512028+0000","last_became_peered":"2024-06-29T14:39:58.512028+0000","last_unstale":"2024-06-29T14:39:59.986622+0000","last_undegraded":"2024-06-29T14:39:59.986622+0000","last_fullsized":"2024-06-29T14:39:59.986622+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:19.295385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507139+0000","last_change":"2024-06-29T14:39:58.521980+0000","last_active":"2024-06-29T14:39:59.507139+0000","last_peered":"2024-06-29T14:39:59.507139+0000","last_clean":"2024-06-29T14:39:59.507139+0000","last_became_active":"2024-06-29T14:39:58.520815+0000","last_became_peered":"2024-06-29T14:39:58.520815+0000","last_unstale":"2024-06-29T14:39:59.507139+0000","last_undegraded":"2024-06-29T14:39:59.507139+0000","last_fullsized":"2024-06-29T14:39:59.507139+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:30.767893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493837+0000","last_change":"2024-06-29T14:39:58.498229+0000","last_active":"2024-06-29T14:39:59.493837+0000","last_peered":"2024-06-29T14:39:59.493837+0000","last_clean":"2024-06-29T14:39:59.493837+0000","last_became_active":"2024-06-29T14:39:58.496059+0000","last_became_peered":"2024-06-29T14:39:58.496059+0000","last_unstale":"2024-06-29T14:39:59.493837+0000","last_undegraded":"2024-06-29T14:39:59.493837+0000","last_fullsized":"2024-06-29T14:39:59.493837+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:03.396218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507085+0000","last_change":"2024-06-29T14:39:58.516130+0000","last_active":"2024-06-29T14:39:59.507085+0000","last_peered":"2024-06-29T14:39:59.507085+0000","last_clean":"2024-06-29T14:39:59.507085+0000","last_became_active":"2024-06-29T14:39:58.513799+0000","last_became_peered":"2024-06-29T14:39:58.513799+0000","last_unstale":"2024-06-29T14:39:59.507085+0000","last_undegraded":"2024-06-29T14:39:59.507085+0000","last_fullsized":"2024-06-29T14:39:59.507085+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:14:57.642394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493372+0000","last_change":"2024-06-29T14:39:58.498448+0000","last_active":"2024-06-29T14:39:59.493372+0000","last_peered":"2024-06-29T14:39:59.493372+0000","last_clean":"2024-06-29T14:39:59.493372+0000","last_became_active":"2024-06-29T14:39:58.498128+0000","last_became_peered":"2024-06-29T14:39:58.498128+0000","last_unstale":"2024-06-29T14:39:59.493372+0000","last_undegraded":"2024-06-29T14:39:59.493372+0000","last_fullsized":"2024-06-29T14:39:59.493372+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:30.295533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539822+0000","last_change":"2024-06-29T14:39:58.508313+0000","last_active":"2024-06-29T14:39:59.539822+0000","last_peered":"2024-06-29T14:39:59.539822+0000","last_clean":"2024-06-29T14:39:59.539822+0000","last_became_active":"2024-06-29T14:39:58.507947+0000","last_became_peered":"2024-06-29T14:39:58.507947+0000","last_unstale":"2024-06-29T14:39:59.539822+0000","last_undegraded":"2024-06-29T14:39:59.539822+0000","last_fullsized":"2024-06-29T14:39:59.539822+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:36.389730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.488899+0000","last_change":"2024-06-29T14:39:58.511653+0000","last_active":"2024-06-29T14:39:59.488899+0000","last_peered":"2024-06-29T14:39:59.488899+0000","last_clean":"2024-06-29T14:39:59.488899+0000","last_became_active":"2024-06-29T14:39:58.511317+0000","last_became_peered":"2024-06-29T14:39:58.511317+0000","last_unstale":"2024-06-29T14:39:59.488899+0000","last_undegraded":"2024-06-29T14:39:59.488899+0000","last_fullsized":"2024-06-29T14:39:59.488899+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:17.412011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506758+0000","last_change":"2024-06-29T14:39:58.517032+0000","last_active":"2024-06-29T14:39:59.506758+0000","last_peered":"2024-06-29T14:39:59.506758+0000","last_clean":"2024-06-29T14:39:59.506758+0000","last_became_active":"2024-06-29T14:39:58.516697+0000","last_became_peered":"2024-06-29T14:39:58.516697+0000","last_unstale":"2024-06-29T14:39:59.506758+0000","last_undegraded":"2024-06-29T14:39:59.506758+0000","last_fullsized":"2024-06-29T14:39:59.506758+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:14:59.300903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.488577+0000","last_change":"2024-06-29T14:39:58.513491+0000","last_active":"2024-06-29T14:39:59.488577+0000","last_peered":"2024-06-29T14:39:59.488577+0000","last_clean":"2024-06-29T14:39:59.488577+0000","last_became_active":"2024-06-29T14:39:58.512947+0000","last_became_peered":"2024-06-29T14:39:58.512947+0000","last_unstale":"2024-06-29T14:39:59.488577+0000","last_undegraded":"2024-06-29T14:39:59.488577+0000","last_fullsized":"2024-06-29T14:39:59.488577+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:02.668775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493434+0000","last_change":"2024-06-29T14:39:58.508977+0000","last_active":"2024-06-29T14:39:59.493434+0000","last_peered":"2024-06-29T14:39:59.493434+0000","last_clean":"2024-06-29T14:39:59.493434+0000","last_became_active":"2024-06-29T14:39:58.508680+0000","last_became_peered":"2024-06-29T14:39:58.508680+0000","last_unstale":"2024-06-29T14:39:59.493434+0000","last_undegraded":"2024-06-29T14:39:59.493434+0000","last_fullsized":"2024-06-29T14:39:59.493434+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:24:29.925276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493722+0000","last_change":"2024-06-29T14:39:58.499983+0000","last_active":"2024-06-29T14:39:59.493722+0000","last_peered":"2024-06-29T14:39:59.493722+0000","last_clean":"2024-06-29T14:39:59.493722+0000","last_became_active":"2024-06-29T14:39:58.499618+0000","last_became_peered":"2024-06-29T14:39:58.499618+0000","last_unstale":"2024-06-29T14:39:59.493722+0000","last_undegraded":"2024-06-29T14:39:59.493722+0000","last_fullsized":"2024-06-29T14:39:59.493722+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:12.707165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509245+0000","last_change":"2024-06-29T14:39:58.507714+0000","last_active":"2024-06-29T14:39:59.509245+0000","last_peered":"2024-06-29T14:39:59.509245+0000","last_clean":"2024-06-29T14:39:59.509245+0000","last_became_active":"2024-06-29T14:39:58.507375+0000","last_became_peered":"2024-06-29T14:39:58.507375+0000","last_unstale":"2024-06-29T14:39:59.509245+0000","last_undegraded":"2024-06-29T14:39:59.509245+0000","last_fullsized":"2024-06-29T14:39:59.509245+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:35.075707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493515+0000","last_change":"2024-06-29T14:39:58.511346+0000","last_active":"2024-06-29T14:39:59.493515+0000","last_peered":"2024-06-29T14:39:59.493515+0000","last_clean":"2024-06-29T14:39:59.493515+0000","last_became_active":"2024-06-29T14:39:58.511007+0000","last_became_peered":"2024-06-29T14:39:58.511007+0000","last_unstale":"2024-06-29T14:39:59.493515+0000","last_undegraded":"2024-06-29T14:39:59.493515+0000","last_fullsized":"2024-06-29T14:39:59.493515+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:37:46.847863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539769+0000","last_change":"2024-06-29T14:39:58.499715+0000","last_active":"2024-06-29T14:39:59.539769+0000","last_peered":"2024-06-29T14:39:59.539769+0000","last_clean":"2024-06-29T14:39:59.539769+0000","last_became_active":"2024-06-29T14:39:58.499339+0000","last_became_peered":"2024-06-29T14:39:58.499339+0000","last_unstale":"2024-06-29T14:39:59.539769+0000","last_undegraded":"2024-06-29T14:39:59.539769+0000","last_fullsized":"2024-06-29T14:39:59.539769+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:05.615307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492992+0000","last_change":"2024-06-29T14:39:58.506765+0000","last_active":"2024-06-29T14:39:59.492992+0000","last_peered":"2024-06-29T14:39:59.492992+0000","last_clean":"2024-06-29T14:39:59.492992+0000","last_became_active":"2024-06-29T14:39:58.506432+0000","last_became_peered":"2024-06-29T14:39:58.506432+0000","last_unstale":"2024-06-29T14:39:59.492992+0000","last_undegraded":"2024-06-29T14:39:59.492992+0000","last_fullsized":"2024-06-29T14:39:59.492992+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:37.139432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494473+0000","last_change":"2024-06-29T14:39:58.510610+0000","last_active":"2024-06-29T14:39:59.494473+0000","last_peered":"2024-06-29T14:39:59.494473+0000","last_clean":"2024-06-29T14:39:59.494473+0000","last_became_active":"2024-06-29T14:39:58.510273+0000","last_became_peered":"2024-06-29T14:39:58.510273+0000","last_unstale":"2024-06-29T14:39:59.494473+0000","last_undegraded":"2024-06-29T14:39:59.494473+0000","last_fullsized":"2024-06-29T14:39:59.494473+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:12.330677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539528+0000","last_change":"2024-06-29T14:39:58.500723+0000","last_active":"2024-06-29T14:39:59.539528+0000","last_peered":"2024-06-29T14:39:59.539528+0000","last_clean":"2024-06-29T14:39:59.539528+0000","last_became_active":"2024-06-29T14:39:58.500364+0000","last_became_peered":"2024-06-29T14:39:58.500364+0000","last_unstale":"2024-06-29T14:39:59.539528+0000","last_undegraded":"2024-06-29T14:39:59.539528+0000","last_fullsized":"2024-06-29T14:39:59.539528+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:33:15.028401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506909+0000","last_change":"2024-06-29T14:39:58.507123+0000","last_active":"2024-06-29T14:39:59.506909+0000","last_peered":"2024-06-29T14:39:59.506909+0000","last_clean":"2024-06-29T14:39:59.506909+0000","last_became_active":"2024-06-29T14:39:58.506774+0000","last_became_peered":"2024-06-29T14:39:58.506774+0000","last_unstale":"2024-06-29T14:39:59.506909+0000","last_undegraded":"2024-06-29T14:39:59.506909+0000","last_fullsized":"2024-06-29T14:39:59.506909+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:14.884180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"128'4","reported_seq":18,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986710+0000","last_change":"2024-06-29T14:39:57.506787+0000","last_active":"2024-06-29T14:39:59.986710+0000","last_peered":"2024-06-29T14:39:59.986710+0000","last_clean":"2024-06-29T14:39:59.986710+0000","last_became_active":"2024-06-29T14:39:57.506456+0000","last_became_peered":"2024-06-29T14:39:57.506456+0000","last_unstale":"2024-06-29T14:39:59.986710+0000","last_undegraded":"2024-06-29T14:39:59.986710+0000","last_fullsized":"2024-06-29T14:39:59.986710+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:43.544640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539960+0000","last_change":"2024-06-29T14:39:58.515482+0000","last_active":"2024-06-29T14:39:59.539960+0000","last_peered":"2024-06-29T14:39:59.539960+0000","last_clean":"2024-06-29T14:39:59.539960+0000","last_became_active":"2024-06-29T14:39:58.515157+0000","last_became_peered":"2024-06-29T14:39:58.515157+0000","last_unstale":"2024-06-29T14:39:59.539960+0000","last_undegraded":"2024-06-29T14:39:59.539960+0000","last_fullsized":"2024-06-29T14:39:59.539960+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:18.776871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539824+0000","last_change":"2024-06-29T14:39:57.505345+0000","last_active":"2024-06-29T14:39:59.539824+0000","last_peered":"2024-06-29T14:39:59.539824+0000","last_clean":"2024-06-29T14:39:59.539824+0000","last_became_active":"2024-06-29T14:39:57.505048+0000","last_became_peered":"2024-06-29T14:39:57.505048+0000","last_unstale":"2024-06-29T14:39:59.539824+0000","last_undegraded":"2024-06-29T14:39:59.539824+0000","last_fullsized":"2024-06-29T14:39:59.539824+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:57.628691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493175+0000","last_change":"2024-06-29T14:39:58.512570+0000","last_active":"2024-06-29T14:39:59.493175+0000","last_peered":"2024-06-29T14:39:59.493175+0000","last_clean":"2024-06-29T14:39:59.493175+0000","last_became_active":"2024-06-29T14:39:58.510857+0000","last_became_peered":"2024-06-29T14:39:58.510857+0000","last_unstale":"2024-06-29T14:39:59.493175+0000","last_undegraded":"2024-06-29T14:39:59.493175+0000","last_fullsized":"2024-06-29T14:39:59.493175+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:44.690680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493941+0000","last_change":"2024-06-29T14:39:57.503458+0000","last_active":"2024-06-29T14:39:59.493941+0000","last_peered":"2024-06-29T14:39:59.493941+0000","last_clean":"2024-06-29T14:39:59.493941+0000","last_became_active":"2024-06-29T14:39:57.502753+0000","last_became_peered":"2024-06-29T14:39:57.502753+0000","last_unstale":"2024-06-29T14:39:59.493941+0000","last_undegraded":"2024-06-29T14:39:59.493941+0000","last_fullsized":"2024-06-29T14:39:59.493941+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:48.128122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509225+0000","last_change":"2024-06-29T14:39:58.506490+0000","last_active":"2024-06-29T14:39:59.509225+0000","last_peered":"2024-06-29T14:39:59.509225+0000","last_clean":"2024-06-29T14:39:59.509225+0000","last_became_active":"2024-06-29T14:39:58.506147+0000","last_became_peered":"2024-06-29T14:39:58.506147+0000","last_unstale":"2024-06-29T14:39:59.509225+0000","last_undegraded":"2024-06-29T14:39:59.509225+0000","last_fullsized":"2024-06-29T14:39:59.509225+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:30.767893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489024+0000","last_change":"2024-06-29T14:39:57.503888+0000","last_active":"2024-06-29T14:39:59.489024+0000","last_peered":"2024-06-29T14:39:59.489024+0000","last_clean":"2024-06-29T14:39:59.489024+0000","last_became_active":"2024-06-29T14:39:57.503572+0000","last_became_peered":"2024-06-29T14:39:57.503572+0000","last_unstale":"2024-06-29T14:39:59.489024+0000","last_undegraded":"2024-06-29T14:39:59.489024+0000","last_fullsized":"2024-06-29T14:39:59.489024+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:43.886933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986602+0000","last_change":"2024-06-29T14:39:58.498126+0000","last_active":"2024-06-29T14:39:59.986602+0000","last_peered":"2024-06-29T14:39:59.986602+0000","last_clean":"2024-06-29T14:39:59.986602+0000","last_became_active":"2024-06-29T14:39:58.497782+0000","last_became_peered":"2024-06-29T14:39:58.497782+0000","last_unstale":"2024-06-29T14:39:59.986602+0000","last_undegraded":"2024-06-29T14:39:59.986602+0000","last_fullsized":"2024-06-29T14:39:59.986602+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:07:15.431378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986751+0000","last_change":"2024-06-29T14:39:57.504774+0000","last_active":"2024-06-29T14:39:59.986751+0000","last_peered":"2024-06-29T14:39:59.986751+0000","last_clean":"2024-06-29T14:39:59.986751+0000","last_became_active":"2024-06-29T14:39:57.504416+0000","last_became_peered":"2024-06-29T14:39:57.504416+0000","last_unstale":"2024-06-29T14:39:59.986751+0000","last_undegraded":"2024-06-29T14:39:59.986751+0000","last_fullsized":"2024-06-29T14:39:59.986751+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:15.327087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507278+0000","last_change":"2024-06-29T14:39:58.513890+0000","last_active":"2024-06-29T14:39:59.507278+0000","last_peered":"2024-06-29T14:39:59.507278+0000","last_clean":"2024-06-29T14:39:59.507278+0000","last_became_active":"2024-06-29T14:39:58.513565+0000","last_became_peered":"2024-06-29T14:39:58.513565+0000","last_unstale":"2024-06-29T14:39:59.507278+0000","last_undegraded":"2024-06-29T14:39:59.507278+0000","last_fullsized":"2024-06-29T14:39:59.507278+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:57.500067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.508941+0000","last_change":"2024-06-29T14:39:57.507715+0000","last_active":"2024-06-29T14:39:59.508941+0000","last_peered":"2024-06-29T14:39:59.508941+0000","last_clean":"2024-06-29T14:39:59.508941+0000","last_became_active":"2024-06-29T14:39:57.507235+0000","last_became_peered":"2024-06-29T14:39:57.507235+0000","last_unstale":"2024-06-29T14:39:59.508941+0000","last_undegraded":"2024-06-29T14:39:59.508941+0000","last_fullsized":"2024-06-29T14:39:59.508941+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:07:14.422911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494017+0000","last_change":"2024-06-29T14:39:58.509639+0000","last_active":"2024-06-29T14:39:59.494017+0000","last_peered":"2024-06-29T14:39:59.494017+0000","last_clean":"2024-06-29T14:39:59.494017+0000","last_became_active":"2024-06-29T14:39:58.509351+0000","last_became_peered":"2024-06-29T14:39:58.509351+0000","last_unstale":"2024-06-29T14:39:59.494017+0000","last_undegraded":"2024-06-29T14:39:59.494017+0000","last_fullsized":"2024-06-29T14:39:59.494017+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:04:51.367262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492816+0000","last_change":"2024-06-29T14:39:57.505036+0000","last_active":"2024-06-29T14:39:59.492816+0000","last_peered":"2024-06-29T14:39:59.492816+0000","last_clean":"2024-06-29T14:39:59.492816+0000","last_became_active":"2024-06-29T14:39:57.504626+0000","last_became_peered":"2024-06-29T14:39:57.504626+0000","last_unstale":"2024-06-29T14:39:59.492816+0000","last_undegraded":"2024-06-29T14:39:59.492816+0000","last_fullsized":"2024-06-29T14:39:59.492816+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:22.798087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986688+0000","last_change":"2024-06-29T14:39:58.510125+0000","last_active":"2024-06-29T14:39:59.986688+0000","last_peered":"2024-06-29T14:39:59.986688+0000","last_clean":"2024-06-29T14:39:59.986688+0000","last_became_active":"2024-06-29T14:39:58.509223+0000","last_became_peered":"2024-06-29T14:39:58.509223+0000","last_unstale":"2024-06-29T14:39:59.986688+0000","last_undegraded":"2024-06-29T14:39:59.986688+0000","last_fullsized":"2024-06-29T14:39:59.986688+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:49.773813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986230+0000","last_change":"2024-06-29T14:39:57.499002+0000","last_active":"2024-06-29T14:39:59.986230+0000","last_peered":"2024-06-29T14:39:59.986230+0000","last_clean":"2024-06-29T14:39:59.986230+0000","last_became_active":"2024-06-29T14:39:57.498690+0000","last_became_peered":"2024-06-29T14:39:57.498690+0000","last_unstale":"2024-06-29T14:39:59.986230+0000","last_undegraded":"2024-06-29T14:39:59.986230+0000","last_fullsized":"2024-06-29T14:39:59.986230+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:41:24.206994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506749+0000","last_change":"2024-06-29T14:39:58.517426+0000","last_active":"2024-06-29T14:39:59.506749+0000","last_peered":"2024-06-29T14:39:59.506749+0000","last_clean":"2024-06-29T14:39:59.506749+0000","last_became_active":"2024-06-29T14:39:58.516663+0000","last_became_peered":"2024-06-29T14:39:58.516663+0000","last_unstale":"2024-06-29T14:39:59.506749+0000","last_undegraded":"2024-06-29T14:39:59.506749+0000","last_fullsized":"2024-06-29T14:39:59.506749+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:07:27.868633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493572+0000","last_change":"2024-06-29T14:39:57.502293+0000","last_active":"2024-06-29T14:39:59.493572+0000","last_peered":"2024-06-29T14:39:59.493572+0000","last_clean":"2024-06-29T14:39:59.493572+0000","last_became_active":"2024-06-29T14:39:57.501652+0000","last_became_peered":"2024-06-29T14:39:57.501652+0000","last_unstale":"2024-06-29T14:39:59.493572+0000","last_undegraded":"2024-06-29T14:39:59.493572+0000","last_fullsized":"2024-06-29T14:39:59.493572+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:37:45.839396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.488690+0000","last_change":"2024-06-29T14:39:58.498818+0000","last_active":"2024-06-29T14:39:59.488690+0000","last_peered":"2024-06-29T14:39:59.488690+0000","last_clean":"2024-06-29T14:39:59.488690+0000","last_became_active":"2024-06-29T14:39:58.498478+0000","last_became_peered":"2024-06-29T14:39:58.498478+0000","last_unstale":"2024-06-29T14:39:59.488690+0000","last_undegraded":"2024-06-29T14:39:59.488690+0000","last_fullsized":"2024-06-29T14:39:59.488690+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:23.335270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506788+0000","last_change":"2024-06-29T14:39:57.509463+0000","last_active":"2024-06-29T14:39:59.506788+0000","last_peered":"2024-06-29T14:39:59.506788+0000","last_clean":"2024-06-29T14:39:59.506788+0000","last_became_active":"2024-06-29T14:39:57.509148+0000","last_became_peered":"2024-06-29T14:39:57.509148+0000","last_unstale":"2024-06-29T14:39:59.506788+0000","last_undegraded":"2024-06-29T14:39:59.506788+0000","last_fullsized":"2024-06-29T14:39:59.506788+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:19:18.491023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539518+0000","last_change":"2024-06-29T14:39:58.510586+0000","last_active":"2024-06-29T14:39:59.539518+0000","last_peered":"2024-06-29T14:39:59.539518+0000","last_clean":"2024-06-29T14:39:59.539518+0000","last_became_active":"2024-06-29T14:39:58.510220+0000","last_became_peered":"2024-06-29T14:39:58.510220+0000","last_unstale":"2024-06-29T14:39:59.539518+0000","last_undegraded":"2024-06-29T14:39:59.539518+0000","last_fullsized":"2024-06-29T14:39:59.539518+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:37:31.540566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"128'3","reported_seq":17,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986827+0000","last_change":"2024-06-29T14:39:57.504857+0000","last_active":"2024-06-29T14:39:59.986827+0000","last_peered":"2024-06-29T14:39:59.986827+0000","last_clean":"2024-06-29T14:39:59.986827+0000","last_became_active":"2024-06-29T14:39:57.504503+0000","last_became_peered":"2024-06-29T14:39:57.504503+0000","last_unstale":"2024-06-29T14:39:59.986827+0000","last_undegraded":"2024-06-29T14:39:59.986827+0000","last_fullsized":"2024-06-29T14:39:59.986827+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:54.134703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489155+0000","last_change":"2024-06-29T14:39:58.498888+0000","last_active":"2024-06-29T14:39:59.489155+0000","last_peered":"2024-06-29T14:39:59.489155+0000","last_clean":"2024-06-29T14:39:59.489155+0000","last_became_active":"2024-06-29T14:39:58.498520+0000","last_became_peered":"2024-06-29T14:39:58.498520+0000","last_unstale":"2024-06-29T14:39:59.489155+0000","last_undegraded":"2024-06-29T14:39:59.489155+0000","last_fullsized":"2024-06-29T14:39:59.489155+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:07:09.641385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"128'6","reported_seq":20,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494187+0000","last_change":"2024-06-29T14:39:57.510734+0000","last_active":"2024-06-29T14:39:59.494187+0000","last_peered":"2024-06-29T14:39:59.494187+0000","last_clean":"2024-06-29T14:39:59.494187+0000","last_became_active":"2024-06-29T14:39:57.510383+0000","last_became_peered":"2024-06-29T14:39:57.510383+0000","last_unstale":"2024-06-29T14:39:59.494187+0000","last_undegraded":"2024-06-29T14:39:59.494187+0000","last_fullsized":"2024-06-29T14:39:59.494187+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:42.036419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509087+0000","last_change":"2024-06-29T14:39:58.514874+0000","last_active":"2024-06-29T14:39:59.509087+0000","last_peered":"2024-06-29T14:39:59.509087+0000","last_clean":"2024-06-29T14:39:59.509087+0000","last_became_active":"2024-06-29T14:39:58.514576+0000","last_became_peered":"2024-06-29T14:39:58.514576+0000","last_unstale":"2024-06-29T14:39:59.509087+0000","last_undegraded":"2024-06-29T14:39:59.509087+0000","last_fullsized":"2024-06-29T14:39:59.509087+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:05.348037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489318+0000","last_change":"2024-06-29T14:39:57.502783+0000","last_active":"2024-06-29T14:39:59.489318+0000","last_peered":"2024-06-29T14:39:59.489318+0000","last_clean":"2024-06-29T14:39:59.489318+0000","last_became_active":"2024-06-29T14:39:57.501603+0000","last_became_peered":"2024-06-29T14:39:57.501603+0000","last_unstale":"2024-06-29T14:39:59.489318+0000","last_undegraded":"2024-06-29T14:39:59.489318+0000","last_fullsized":"2024-06-29T14:39:59.489318+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:18:17.222919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494326+0000","last_change":"2024-06-29T14:39:58.508361+0000","last_active":"2024-06-29T14:39:59.494326+0000","last_peered":"2024-06-29T14:39:59.494326+0000","last_clean":"2024-06-29T14:39:59.494326+0000","last_became_active":"2024-06-29T14:39:58.507954+0000","last_became_peered":"2024-06-29T14:39:58.507954+0000","last_unstale":"2024-06-29T14:39:59.494326+0000","last_undegraded":"2024-06-29T14:39:59.494326+0000","last_fullsized":"2024-06-29T14:39:59.494326+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:17.412492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986419+0000","last_change":"2024-06-29T14:39:57.508598+0000","last_active":"2024-06-29T14:39:59.986419+0000","last_peered":"2024-06-29T14:39:59.986419+0000","last_clean":"2024-06-29T14:39:59.986419+0000","last_became_active":"2024-06-29T14:39:57.508268+0000","last_became_peered":"2024-06-29T14:39:57.508268+0000","last_unstale":"2024-06-29T14:39:59.986419+0000","last_undegraded":"2024-06-29T14:39:59.986419+0000","last_fullsized":"2024-06-29T14:39:59.986419+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:19:18.491023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539711+0000","last_change":"2024-06-29T14:39:58.501478+0000","last_active":"2024-06-29T14:39:59.539711+0000","last_peered":"2024-06-29T14:39:59.539711+0000","last_clean":"2024-06-29T14:39:59.539711+0000","last_became_active":"2024-06-29T14:39:58.501162+0000","last_became_peered":"2024-06-29T14:39:58.501162+0000","last_unstale":"2024-06-29T14:39:59.539711+0000","last_undegraded":"2024-06-29T14:39:59.539711+0000","last_fullsized":"2024-06-29T14:39:59.539711+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:55.781777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539590+0000","last_change":"2024-06-29T14:39:57.504008+0000","last_active":"2024-06-29T14:39:59.539590+0000","last_peered":"2024-06-29T14:39:59.539590+0000","last_clean":"2024-06-29T14:39:59.539590+0000","last_became_active":"2024-06-29T14:39:57.503722+0000","last_became_peered":"2024-06-29T14:39:57.503722+0000","last_unstale":"2024-06-29T14:39:59.539590+0000","last_undegraded":"2024-06-29T14:39:59.539590+0000","last_fullsized":"2024-06-29T14:39:59.539590+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:47:40.810306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507213+0000","last_change":"2024-06-29T14:39:58.520203+0000","last_active":"2024-06-29T14:39:59.507213+0000","last_peered":"2024-06-29T14:39:59.507213+0000","last_clean":"2024-06-29T14:39:59.507213+0000","last_became_active":"2024-06-29T14:39:58.519860+0000","last_became_peered":"2024-06-29T14:39:58.519860+0000","last_unstale":"2024-06-29T14:39:59.507213+0000","last_undegraded":"2024-06-29T14:39:59.507213+0000","last_fullsized":"2024-06-29T14:39:59.507213+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:23.420360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"128'8","reported_seq":22,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492757+0000","last_change":"2024-06-29T14:39:57.502251+0000","last_active":"2024-06-29T14:39:59.492757+0000","last_peered":"2024-06-29T14:39:59.492757+0000","last_clean":"2024-06-29T14:39:59.492757+0000","last_became_active":"2024-06-29T14:39:57.501389+0000","last_became_peered":"2024-06-29T14:39:57.501389+0000","last_unstale":"2024-06-29T14:39:59.492757+0000","last_undegraded":"2024-06-29T14:39:59.492757+0000","last_fullsized":"2024-06-29T14:39:59.492757+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:28:46.311128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539779+0000","last_change":"2024-06-29T14:39:58.509671+0000","last_active":"2024-06-29T14:39:59.539779+0000","last_peered":"2024-06-29T14:39:59.539779+0000","last_clean":"2024-06-29T14:39:59.539779+0000","last_became_active":"2024-06-29T14:39:58.509357+0000","last_became_peered":"2024-06-29T14:39:58.509357+0000","last_unstale":"2024-06-29T14:39:59.539779+0000","last_undegraded":"2024-06-29T14:39:59.539779+0000","last_fullsized":"2024-06-29T14:39:59.539779+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:50.917154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494267+0000","last_change":"2024-06-29T14:39:57.504227+0000","last_active":"2024-06-29T14:39:59.494267+0000","last_peered":"2024-06-29T14:39:59.494267+0000","last_clean":"2024-06-29T14:39:59.494267+0000","last_became_active":"2024-06-29T14:39:57.503904+0000","last_became_peered":"2024-06-29T14:39:57.503904+0000","last_unstale":"2024-06-29T14:39:59.494267+0000","last_undegraded":"2024-06-29T14:39:59.494267+0000","last_fullsized":"2024-06-29T14:39:59.494267+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:48:21.626126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494115+0000","last_change":"2024-06-29T14:39:58.508416+0000","last_active":"2024-06-29T14:39:59.494115+0000","last_peered":"2024-06-29T14:39:59.494115+0000","last_clean":"2024-06-29T14:39:59.494115+0000","last_became_active":"2024-06-29T14:39:58.507931+0000","last_became_peered":"2024-06-29T14:39:58.507931+0000","last_unstale":"2024-06-29T14:39:59.494115+0000","last_undegraded":"2024-06-29T14:39:59.494115+0000","last_fullsized":"2024-06-29T14:39:59.494115+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:21.563876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"128'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986929+0000","last_change":"2024-06-29T14:39:57.501506+0000","last_active":"2024-06-29T14:39:59.986929+0000","last_peered":"2024-06-29T14:39:59.986929+0000","last_clean":"2024-06-29T14:39:59.986929+0000","last_became_active":"2024-06-29T14:39:57.501135+0000","last_became_peered":"2024-06-29T14:39:57.501135+0000","last_unstale":"2024-06-29T14:39:59.986929+0000","last_undegraded":"2024-06-29T14:39:59.986929+0000","last_fullsized":"2024-06-29T14:39:59.986929+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:00.440766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.488945+0000","last_change":"2024-06-29T14:39:58.499715+0000","last_active":"2024-06-29T14:39:59.488945+0000","last_peered":"2024-06-29T14:39:59.488945+0000","last_clean":"2024-06-29T14:39:59.488945+0000","last_became_active":"2024-06-29T14:39:58.499339+0000","last_became_peered":"2024-06-29T14:39:58.499339+0000","last_unstale":"2024-06-29T14:39:59.488945+0000","last_undegraded":"2024-06-29T14:39:59.488945+0000","last_fullsized":"2024-06-29T14:39:59.488945+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:34.015450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509055+0000","last_change":"2024-06-29T14:39:57.507850+0000","last_active":"2024-06-29T14:39:59.509055+0000","last_peered":"2024-06-29T14:39:59.509055+0000","last_clean":"2024-06-29T14:39:59.509055+0000","last_became_active":"2024-06-29T14:39:57.507261+0000","last_became_peered":"2024-06-29T14:39:57.507261+0000","last_unstale":"2024-06-29T14:39:59.509055+0000","last_undegraded":"2024-06-29T14:39:59.509055+0000","last_fullsized":"2024-06-29T14:39:59.509055+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:26.962456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506914+0000","last_change":"2024-06-29T14:39:58.523318+0000","last_active":"2024-06-29T14:39:59.506914+0000","last_peered":"2024-06-29T14:39:59.506914+0000","last_clean":"2024-06-29T14:39:59.506914+0000","last_became_active":"2024-06-29T14:39:58.522946+0000","last_became_peered":"2024-06-29T14:39:58.522946+0000","last_unstale":"2024-06-29T14:39:59.506914+0000","last_undegraded":"2024-06-29T14:39:59.506914+0000","last_fullsized":"2024-06-29T14:39:59.506914+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:30.310799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539959+0000","last_change":"2024-06-29T14:39:57.506162+0000","last_active":"2024-06-29T14:39:59.539959+0000","last_peered":"2024-06-29T14:39:59.539959+0000","last_clean":"2024-06-29T14:39:59.539959+0000","last_became_active":"2024-06-29T14:39:57.504914+0000","last_became_peered":"2024-06-29T14:39:57.504914+0000","last_unstale":"2024-06-29T14:39:59.539959+0000","last_undegraded":"2024-06-29T14:39:59.539959+0000","last_fullsized":"2024-06-29T14:39:59.539959+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:33.841274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.507260+0000","last_change":"2024-06-29T14:39:58.511527+0000","last_active":"2024-06-29T14:39:59.507260+0000","last_peered":"2024-06-29T14:39:59.507260+0000","last_clean":"2024-06-29T14:39:59.507260+0000","last_became_active":"2024-06-29T14:39:58.511188+0000","last_became_peered":"2024-06-29T14:39:58.511188+0000","last_unstale":"2024-06-29T14:39:59.507260+0000","last_undegraded":"2024-06-29T14:39:59.507260+0000","last_fullsized":"2024-06-29T14:39:59.507260+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:35.075707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539637+0000","last_change":"2024-06-29T14:39:57.504730+0000","last_active":"2024-06-29T14:39:59.539637+0000","last_peered":"2024-06-29T14:39:59.539637+0000","last_clean":"2024-06-29T14:39:59.539637+0000","last_became_active":"2024-06-29T14:39:57.504088+0000","last_became_peered":"2024-06-29T14:39:57.504088+0000","last_unstale":"2024-06-29T14:39:59.539637+0000","last_undegraded":"2024-06-29T14:39:59.539637+0000","last_fullsized":"2024-06-29T14:39:59.539637+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:09.177229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986721+0000","last_change":"2024-06-29T14:39:58.505928+0000","last_active":"2024-06-29T14:39:59.986721+0000","last_peered":"2024-06-29T14:39:59.986721+0000","last_clean":"2024-06-29T14:39:59.986721+0000","last_became_active":"2024-06-29T14:39:58.504876+0000","last_became_peered":"2024-06-29T14:39:58.504876+0000","last_unstale":"2024-06-29T14:39:59.986721+0000","last_undegraded":"2024-06-29T14:39:59.986721+0000","last_fullsized":"2024-06-29T14:39:59.986721+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:31:31.162145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494436+0000","last_change":"2024-06-29T14:39:57.500171+0000","last_active":"2024-06-29T14:39:59.494436+0000","last_peered":"2024-06-29T14:39:59.494436+0000","last_clean":"2024-06-29T14:39:59.494436+0000","last_became_active":"2024-06-29T14:39:57.499804+0000","last_became_peered":"2024-06-29T14:39:57.499804+0000","last_unstale":"2024-06-29T14:39:59.494436+0000","last_undegraded":"2024-06-29T14:39:59.494436+0000","last_fullsized":"2024-06-29T14:39:59.494436+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:12.743273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539673+0000","last_change":"2024-06-29T14:39:58.511398+0000","last_active":"2024-06-29T14:39:59.539673+0000","last_peered":"2024-06-29T14:39:59.539673+0000","last_clean":"2024-06-29T14:39:59.539673+0000","last_became_active":"2024-06-29T14:39:58.511074+0000","last_became_peered":"2024-06-29T14:39:58.511074+0000","last_unstale":"2024-06-29T14:39:59.539673+0000","last_undegraded":"2024-06-29T14:39:59.539673+0000","last_fullsized":"2024-06-29T14:39:59.539673+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:52.355286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539877+0000","last_change":"2024-06-29T14:39:57.505691+0000","last_active":"2024-06-29T14:39:59.539877+0000","last_peered":"2024-06-29T14:39:59.539877+0000","last_clean":"2024-06-29T14:39:59.539877+0000","last_became_active":"2024-06-29T14:39:57.505157+0000","last_became_peered":"2024-06-29T14:39:57.505157+0000","last_unstale":"2024-06-29T14:39:59.539877+0000","last_undegraded":"2024-06-29T14:39:59.539877+0000","last_fullsized":"2024-06-29T14:39:59.539877+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:11:56.397418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.506718+0000","last_change":"2024-06-29T14:39:58.520282+0000","last_active":"2024-06-29T14:39:59.506718+0000","last_peered":"2024-06-29T14:39:59.506718+0000","last_clean":"2024-06-29T14:39:59.506718+0000","last_became_active":"2024-06-29T14:39:58.519984+0000","last_became_peered":"2024-06-29T14:39:58.519984+0000","last_unstale":"2024-06-29T14:39:59.506718+0000","last_undegraded":"2024-06-29T14:39:59.506718+0000","last_fullsized":"2024-06-29T14:39:59.506718+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:31:31.162145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489190+0000","last_change":"2024-06-29T14:39:57.501627+0000","last_active":"2024-06-29T14:39:59.489190+0000","last_peered":"2024-06-29T14:39:59.489190+0000","last_clean":"2024-06-29T14:39:59.489190+0000","last_became_active":"2024-06-29T14:39:57.501274+0000","last_became_peered":"2024-06-29T14:39:57.501274+0000","last_unstale":"2024-06-29T14:39:59.489190+0000","last_undegraded":"2024-06-29T14:39:59.489190+0000","last_fullsized":"2024-06-29T14:39:59.489190+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:28:28.094143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.492875+0000","last_change":"2024-06-29T14:39:58.506728+0000","last_active":"2024-06-29T14:39:59.492875+0000","last_peered":"2024-06-29T14:39:59.492875+0000","last_clean":"2024-06-29T14:39:59.492875+0000","last_became_active":"2024-06-29T14:39:58.506400+0000","last_became_peered":"2024-06-29T14:39:58.506400+0000","last_unstale":"2024-06-29T14:39:59.492875+0000","last_undegraded":"2024-06-29T14:39:59.492875+0000","last_fullsized":"2024-06-29T14:39:59.492875+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:12.330677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"129'1","reported_seq":15,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.508408+0000","last_change":"2024-06-29T14:39:57.507418+0000","last_active":"2024-06-29T14:39:59.508408+0000","last_peered":"2024-06-29T14:39:59.508408+0000","last_clean":"2024-06-29T14:39:59.508408+0000","last_became_active":"2024-06-29T14:39:57.507093+0000","last_became_peered":"2024-06-29T14:39:57.507093+0000","last_unstale":"2024-06-29T14:39:59.508408+0000","last_undegraded":"2024-06-29T14:39:59.508408+0000","last_fullsized":"2024-06-29T14:39:59.508408+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:43.544640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986395+0000","last_change":"2024-06-29T14:39:58.506238+0000","last_active":"2024-06-29T14:39:59.986395+0000","last_peered":"2024-06-29T14:39:59.986395+0000","last_clean":"2024-06-29T14:39:59.986395+0000","last_became_active":"2024-06-29T14:39:58.504550+0000","last_became_peered":"2024-06-29T14:39:58.504550+0000","last_unstale":"2024-06-29T14:39:59.986395+0000","last_undegraded":"2024-06-29T14:39:59.986395+0000","last_fullsized":"2024-06-29T14:39:59.986395+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:50.697755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494287+0000","last_change":"2024-06-29T14:39:57.511540+0000","last_active":"2024-06-29T14:39:59.494287+0000","last_peered":"2024-06-29T14:39:59.494287+0000","last_clean":"2024-06-29T14:39:59.494287+0000","last_became_active":"2024-06-29T14:39:57.511270+0000","last_became_peered":"2024-06-29T14:39:57.511270+0000","last_unstale":"2024-06-29T14:39:59.494287+0000","last_undegraded":"2024-06-29T14:39:59.494287+0000","last_fullsized":"2024-06-29T14:39:59.494287+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:23.317944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509247+0000","last_change":"2024-06-29T14:39:58.513690+0000","last_active":"2024-06-29T14:39:59.509247+0000","last_peered":"2024-06-29T14:39:59.509247+0000","last_clean":"2024-06-29T14:39:59.509247+0000","last_became_active":"2024-06-29T14:39:58.513016+0000","last_became_peered":"2024-06-29T14:39:58.513016+0000","last_unstale":"2024-06-29T14:39:59.509247+0000","last_undegraded":"2024-06-29T14:39:59.509247+0000","last_fullsized":"2024-06-29T14:39:59.509247+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:23.420360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.489083+0000","last_change":"2024-06-29T14:39:57.504751+0000","last_active":"2024-06-29T14:39:59.489083+0000","last_peered":"2024-06-29T14:39:59.489083+0000","last_clean":"2024-06-29T14:39:59.489083+0000","last_became_active":"2024-06-29T14:39:57.504435+0000","last_became_peered":"2024-06-29T14:39:57.504435+0000","last_unstale":"2024-06-29T14:39:59.489083+0000","last_undegraded":"2024-06-29T14:39:59.489083+0000","last_fullsized":"2024-06-29T14:39:59.489083+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:30.390652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539772+0000","last_change":"2024-06-29T14:39:58.495560+0000","last_active":"2024-06-29T14:39:59.539772+0000","last_peered":"2024-06-29T14:39:59.539772+0000","last_clean":"2024-06-29T14:39:59.539772+0000","last_became_active":"2024-06-29T14:39:58.495237+0000","last_became_peered":"2024-06-29T14:39:58.495237+0000","last_unstale":"2024-06-29T14:39:59.539772+0000","last_undegraded":"2024-06-29T14:39:59.539772+0000","last_fullsized":"2024-06-29T14:39:59.539772+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:08.486896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.493136+0000","last_change":"2024-06-29T14:39:57.501055+0000","last_active":"2024-06-29T14:39:59.493136+0000","last_peered":"2024-06-29T14:39:59.493136+0000","last_clean":"2024-06-29T14:39:59.493136+0000","last_became_active":"2024-06-29T14:39:57.500420+0000","last_became_peered":"2024-06-29T14:39:57.500420+0000","last_unstale":"2024-06-29T14:39:59.493136+0000","last_undegraded":"2024-06-29T14:39:59.493136+0000","last_fullsized":"2024-06-29T14:39:59.493136+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:30.374480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539808+0000","last_change":"2024-06-29T14:39:58.516176+0000","last_active":"2024-06-29T14:39:59.539808+0000","last_peered":"2024-06-29T14:39:59.539808+0000","last_clean":"2024-06-29T14:39:59.539808+0000","last_became_active":"2024-06-29T14:39:58.515859+0000","last_became_peered":"2024-06-29T14:39:58.515859+0000","last_unstale":"2024-06-29T14:39:59.539808+0000","last_undegraded":"2024-06-29T14:39:59.539808+0000","last_fullsized":"2024-06-29T14:39:59.539808+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:43:26.292133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539660+0000","last_change":"2024-06-29T14:39:57.504470+0000","last_active":"2024-06-29T14:39:59.539660+0000","last_peered":"2024-06-29T14:39:59.539660+0000","last_clean":"2024-06-29T14:39:59.539660+0000","last_became_active":"2024-06-29T14:39:57.503659+0000","last_became_peered":"2024-06-29T14:39:57.503659+0000","last_unstale":"2024-06-29T14:39:59.539660+0000","last_undegraded":"2024-06-29T14:39:59.539660+0000","last_fullsized":"2024-06-29T14:39:59.539660+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:04.365120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986903+0000","last_change":"2024-06-29T14:39:58.516028+0000","last_active":"2024-06-29T14:39:59.986903+0000","last_peered":"2024-06-29T14:39:59.986903+0000","last_clean":"2024-06-29T14:39:59.986903+0000","last_became_active":"2024-06-29T14:39:58.515647+0000","last_became_peered":"2024-06-29T14:39:58.515647+0000","last_unstale":"2024-06-29T14:39:59.986903+0000","last_undegraded":"2024-06-29T14:39:59.986903+0000","last_fullsized":"2024-06-29T14:39:59.986903+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:12.317950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.494542+0000","last_change":"2024-06-29T14:39:57.503907+0000","last_active":"2024-06-29T14:39:59.494542+0000","last_peered":"2024-06-29T14:39:59.494542+0000","last_clean":"2024-06-29T14:39:59.494542+0000","last_became_active":"2024-06-29T14:39:57.503557+0000","last_became_peered":"2024-06-29T14:39:57.503557+0000","last_unstale":"2024-06-29T14:39:59.494542+0000","last_undegraded":"2024-06-29T14:39:59.494542+0000","last_fullsized":"2024-06-29T14:39:59.494542+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:18:07.624984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.509389+0000","last_change":"2024-06-29T14:39:58.507541+0000","last_active":"2024-06-29T14:39:59.509389+0000","last_peered":"2024-06-29T14:39:59.509389+0000","last_clean":"2024-06-29T14:39:59.509389+0000","last_became_active":"2024-06-29T14:39:58.507240+0000","last_became_peered":"2024-06-29T14:39:58.507240+0000","last_unstale":"2024-06-29T14:39:59.509389+0000","last_undegraded":"2024-06-29T14:39:59.509389+0000","last_fullsized":"2024-06-29T14:39:59.509389+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:57.500067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539952+0000","last_change":"2024-06-29T14:39:57.497210+0000","last_active":"2024-06-29T14:39:59.539952+0000","last_peered":"2024-06-29T14:39:59.539952+0000","last_clean":"2024-06-29T14:39:59.539952+0000","last_became_active":"2024-06-29T14:39:57.496895+0000","last_became_peered":"2024-06-29T14:39:57.496895+0000","last_unstale":"2024-06-29T14:39:59.539952+0000","last_undegraded":"2024-06-29T14:39:59.539952+0000","last_fullsized":"2024-06-29T14:39:59.539952+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:29.163307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.508982+0000","last_change":"2024-06-29T14:39:58.513738+0000","last_active":"2024-06-29T14:39:59.508982+0000","last_peered":"2024-06-29T14:39:59.508982+0000","last_clean":"2024-06-29T14:39:59.508982+0000","last_became_active":"2024-06-29T14:39:58.511414+0000","last_became_peered":"2024-06-29T14:39:58.511414+0000","last_unstale":"2024-06-29T14:39:59.508982+0000","last_undegraded":"2024-06-29T14:39:59.508982+0000","last_fullsized":"2024-06-29T14:39:59.508982+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:57.473574+0000","last_clean_scrub_stamp":"2024-06-29T14:39:57.473574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:03.881878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"128'2","reported_seq":16,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.539956+0000","last_change":"2024-06-29T14:39:57.500699+0000","last_active":"2024-06-29T14:39:59.539956+0000","last_peered":"2024-06-29T14:39:59.539956+0000","last_clean":"2024-06-29T14:39:59.539956+0000","last_became_active":"2024-06-29T14:39:57.500405+0000","last_became_peered":"2024-06-29T14:39:57.500405+0000","last_unstale":"2024-06-29T14:39:59.539956+0000","last_undegraded":"2024-06-29T14:39:59.539956+0000","last_fullsized":"2024-06-29T14:39:59.539956+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.465108+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.465108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:01.456219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"118'142","reported_seq":569,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.986386+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:39:59.986386+0000","last_peered":"2024-06-29T14:39:59.986386+0000","last_clean":"2024-06-29T14:39:59.986386+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:39:59.986386+0000","last_undegraded":"2024-06-29T14:39:59.986386+0000","last_fullsized":"2024-06-29T14:39:59.986386+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":221,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":221,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542363,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59676,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312164,"statfs":{"total":96636764160,"available":96575655936,"internally_reserved":0,"allocated":2179072,"data_stored":1387958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542364,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1597440,"data_stored":799646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54932,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316908,"statfs":{"total":96636764160,"available":96580513792,"internally_reserved":0,"allocated":1576960,"data_stored":797430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542362,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1597440,"data_stored":804214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509660,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36892,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334948,"statfs":{"total":96636764160,"available":96598986752,"internally_reserved":0,"allocated":1581056,"data_stored":798012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575068,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59112,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312728,"statfs":{"total":96636764160,"available":96576233472,"internally_reserved":0,"allocated":1593344,"data_stored":804156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509659,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1581056,"data_stored":797464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509661,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37492,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334348,"statfs":{"total":96636764160,"available":96598372352,"internally_reserved":0,"allocated":2199552,"data_stored":1396192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:40:02.958 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:40:02.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:40:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 -- 172.21.15.59:0/2711340757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0b4a50 msgr2=0x7f29cc0d1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 --2- 172.21.15.59:0/2711340757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0b4a50 0x7f29cc0d1610 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f29cc08ca00 tx=0x7f29bc02a980 comp rx=0 tx=0).stop 2024-06-29T14:40:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 -- 172.21.15.59:0/2711340757 shutdown_connections 2024-06-29T14:40:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 --2- 172.21.15.59:0/2711340757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0b4a50 0x7f29cc0d1610 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 --2- 172.21.15.59:0/2711340757 >> v2:172.21.15.153:3300/0 conn(0x7f29cc0cefb0 0x7f29cc0b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 --2- 172.21.15.59:0/2711340757 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29cc0ce090 0x7f29cc0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 -- 172.21.15.59:0/2711340757 >> 172.21.15.59:0/2711340757 conn(0x7f29cc0d4070 msgr2=0x7f29cc0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 -- 172.21.15.59:0/2711340757 shutdown_connections 2024-06-29T14:40:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.122+0000 7f29d0deb640 1 -- 172.21.15.59:0/2711340757 wait complete. 2024-06-29T14:40:03.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.124+0000 7f29d0deb640 1 Processor -- start 2024-06-29T14:40:03.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.124+0000 7f29d0deb640 1 -- start start 2024-06-29T14:40:03.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.125+0000 7f29d0deb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29cc0b4a50 0x7f29cc148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.125+0000 7f29d0deb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29cc0ce090 0x7f29cc1e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.125+0000 7f29d0deb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0cefb0 0x7f29cc0ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.125+0000 7f29d0deb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc0d1ee0 con 0x7f29cc0cefb0 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.125+0000 7f29d0deb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc0d1d60 con 0x7f29cc0b4a50 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.125+0000 7f29d0deb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f29cc0d2060 con 0x7f29cc0ce090 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.125+0000 7f29cad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0cefb0 0x7f29cc0ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29cad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0cefb0 0x7f29cc0ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56942/0 (socket says 172.21.15.59:56942) 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29cad9d640 1 -- 172.21.15.59:0/539209533 learned_addr learned my addr 172.21.15.59:0/539209533 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29ca59c640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29cc0b4a50 0x7f29cc148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29c9d9b640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29cc0ce090 0x7f29cc1e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29cad9d640 1 -- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29cc0ce090 msgr2=0x7f29cc1e9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29cad9d640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29cc0ce090 0x7f29cc1e9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29cad9d640 1 -- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29cc0b4a50 msgr2=0x7f29cc148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29cad9d640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29cc0b4a50 0x7f29cc148870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29cad9d640 1 -- 172.21.15.59:0/539209533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc148db0 con 0x7f29cc0cefb0 2024-06-29T14:40:03.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29cad9d640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0cefb0 0x7f29cc0ccb60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f29bc033040 tx=0x7f29bc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29c2ffd640 1 -- 172.21.15.59:0/539209533 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc034040 con 0x7f29cc0cefb0 2024-06-29T14:40:03.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29c2ffd640 1 -- 172.21.15.59:0/539209533 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29bc004540 con 0x7f29cc0cefb0 2024-06-29T14:40:03.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29cc1e7e30 con 0x7f29cc0cefb0 2024-06-29T14:40:03.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29c2ffd640 1 -- 172.21.15.59:0/539209533 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc045480 con 0x7f29cc0cefb0 2024-06-29T14:40:03.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.126+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29cc1e83f0 con 0x7f29cc0cefb0 2024-06-29T14:40:03.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.128+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29cc0b2e60 con 0x7f29cc0cefb0 2024-06-29T14:40:03.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.128+0000 7f29c2ffd640 1 -- 172.21.15.59:0/539209533 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f29bc037020 con 0x7f29cc0cefb0 2024-06-29T14:40:03.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.128+0000 7f29c2ffd640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29ac0763f0 0x7f29ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.129+0000 7f29c2ffd640 1 -- 172.21.15.59:0/539209533 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f29bc0b91e0 con 0x7f29cc0cefb0 2024-06-29T14:40:03.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.129+0000 7f29ca59c640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29ac0763f0 0x7f29ac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.129+0000 7f29ca59c640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29ac0763f0 0x7f29ac078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009b30 tx=0x7f29c4005f90 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.133+0000 7f29c2ffd640 1 -- 172.21.15.59:0/539209533 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f29bc082260 con 0x7f29cc0cefb0 2024-06-29T14:40:03.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.398+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f29cc1ea030 con 0x7f29cc0cefb0 2024-06-29T14:40:03.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.400+0000 7f29c2ffd640 1 -- 172.21.15.59:0/539209533 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) ==== 190+0+147 (secure 0 0 0) 0x7f29bc086560 con 0x7f29cc0cefb0 2024-06-29T14:40:03.399 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:40:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:40:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T14:40:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T14:40:03.399 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:40:03.399 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29ac0763f0 msgr2=0x7f29ac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29ac0763f0 0x7f29ac078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009b30 tx=0x7f29c4005f90 comp rx=0 tx=0).stop 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0cefb0 msgr2=0x7f29cc0ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0cefb0 0x7f29cc0ccb60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f29bc033040 tx=0x7f29bc003f10 comp rx=0 tx=0).stop 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 shutdown_connections 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f29ac0763f0 0x7f29ac078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29cc0cefb0 0x7f29cc0ccb60 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29cc0ce090 0x7f29cc1e9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 --2- 172.21.15.59:0/539209533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29cc0b4a50 0x7f29cc148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 >> 172.21.15.59:0/539209533 conn(0x7f29cc0d4070 msgr2=0x7f29cc160890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 shutdown_connections 2024-06-29T14:40:03.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.403+0000 7f29d0deb640 1 -- 172.21.15.59:0/539209533 wait complete. 2024-06-29T14:40:03.412 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:40:03.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.532+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/178791481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e3815efe0 msgr2=0x7f0e3815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.532+0000 7f0e3dfbd640 1 --2- 172.21.15.59:0/178791481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e3815efe0 0x7f0e3815f9b0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009830 tx=0x7f0e3002a510 comp rx=0 tx=0).stop 2024-06-29T14:40:03.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.532+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/178791481 shutdown_connections 2024-06-29T14:40:03.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.532+0000 7f0e3dfbd640 1 --2- 172.21.15.59:0/178791481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e3815efe0 0x7f0e3815f9b0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.532+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/178791481 >> 172.21.15.59:0/178791481 conn(0x7f0e380d3ee0 msgr2=0x7f0e380d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.532+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/178791481 shutdown_connections 2024-06-29T14:40:03.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.532+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/178791481 wait complete. 2024-06-29T14:40:03.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.534+0000 7f0e3dfbd640 1 Processor -- start 2024-06-29T14:40:03.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.534+0000 7f0e3dfbd640 1 -- start start 2024-06-29T14:40:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e3dfbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e380d23c0 0x7f0e3814a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e3dfbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e3815efe0 0x7f0e380b8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e3dfbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e380b88b0 0x7f0e3814f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e3dfbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38160420 con 0x7f0e380b88b0 2024-06-29T14:40:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e3dfbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381602a0 con 0x7f0e380d23c0 2024-06-29T14:40:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e3dfbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0e381605a0 con 0x7f0e3815efe0 2024-06-29T14:40:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e380b88b0 0x7f0e3814f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e380b88b0 0x7f0e3814f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56966/0 (socket says 172.21.15.59:56966) 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 -- 172.21.15.59:0/1731482281 learned_addr learned my addr 172.21.15.59:0/1731482281 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e36ffd640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e3815efe0 0x7f0e380b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e377fe640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e380d23c0 0x7f0e3814a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 -- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e3815efe0 msgr2=0x7f0e380b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e3815efe0 0x7f0e380b8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 -- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e380d23c0 msgr2=0x7f0e3814a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e380d23c0 0x7f0e3814a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 -- 172.21.15.59:0/1731482281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e380c8960 con 0x7f0e380b88b0 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e377fe640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e380d23c0 0x7f0e3814a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e36ffd640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e3815efe0 0x7f0e380b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e37fff640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e380b88b0 0x7f0e3814f9e0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009950 tx=0x7f0e30033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e2ffff640 1 -- 172.21.15.59:0/1731482281 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30031d60 con 0x7f0e380b88b0 2024-06-29T14:40:03.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e2ffff640 1 -- 172.21.15.59:0/1731482281 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e30005020 con 0x7f0e380b88b0 2024-06-29T14:40:03.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e380c8c40 con 0x7f0e380b88b0 2024-06-29T14:40:03.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e2ffff640 1 -- 172.21.15.59:0/1731482281 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30005450 con 0x7f0e380b88b0 2024-06-29T14:40:03.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.536+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e380c9200 con 0x7f0e380b88b0 2024-06-29T14:40:03.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.537+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e3815f9b0 con 0x7f0e380b88b0 2024-06-29T14:40:03.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.538+0000 7f0e2ffff640 1 -- 172.21.15.59:0/1731482281 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f0e300047a0 con 0x7f0e380b88b0 2024-06-29T14:40:03.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.539+0000 7f0e2ffff640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0e180763f0 0x7f0e18078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.539+0000 7f0e2ffff640 1 -- 172.21.15.59:0/1731482281 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f0e300b8ea0 con 0x7f0e380b88b0 2024-06-29T14:40:03.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.539+0000 7f0e377fe640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0e180763f0 0x7f0e18078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.540+0000 7f0e377fe640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0e180763f0 0x7f0e18078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e380d2e40 tx=0x7f0e28009040 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.543+0000 7f0e2ffff640 1 -- 172.21.15.59:0/1731482281 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0e3003f030 con 0x7f0e380b88b0 2024-06-29T14:40:03.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.772+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e380b8dc0 con 0x7f0e380b88b0 2024-06-29T14:40:03.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.772+0000 7f0e2ffff640 1 -- 172.21.15.59:0/1731482281 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5228 (secure 0 0 0) 0x7f0e30031770 con 0x7f0e380b88b0 2024-06-29T14:40:03.771 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:40:03.771 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":63,"btime":"2024-06-29T14:40:01:506455+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2473404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2473404084},{"type":"v1","addr":"172.21.15.153:6836","nonce":2473404084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7831,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3605263971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3605263971},{"type":"v1","addr":"172.21.15.153:6837","nonce":3605263971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/842001425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":842001425},{"type":"v1","addr":"172.21.15.59:6833","nonce":842001425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.885754+0000","modified":"2024-06-29T14:40:00.901266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3950841523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3950841523},{"type":"v1","addr":"172.21.15.59:6835","nonce":3950841523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8361,"qdb_cluster":[8361]},"id":7}]} 2024-06-29T14:40:03.771 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 63 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0e180763f0 msgr2=0x7f0e18078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0e180763f0 0x7f0e18078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e380d2e40 tx=0x7f0e28009040 comp rx=0 tx=0).stop 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e380b88b0 msgr2=0x7f0e3814f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e380b88b0 0x7f0e3814f9e0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f0e30009950 tx=0x7f0e30033c90 comp rx=0 tx=0).stop 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 shutdown_connections 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f0e180763f0 0x7f0e18078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e380b88b0 0x7f0e3814f9e0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0e3815efe0 0x7f0e380b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 --2- 172.21.15.59:0/1731482281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e380d23c0 0x7f0e3814a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.774+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 >> 172.21.15.59:0/1731482281 conn(0x7f0e380d3ee0 msgr2=0x7f0e3806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.775+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 shutdown_connections 2024-06-29T14:40:03.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:03.775+0000 7f0e3dfbd640 1 -- 172.21.15.59:0/1731482281 wait complete. 2024-06-29T14:40:04.186 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:39:57.885754+0000', 'modified': '2024-06-29T14:40:00.901266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8361}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8361': {'gid': 8361, 'name': 'a', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/3950841523', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 3950841523}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 3950841523}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8361, 'qdb_cluster': [8361]} 2024-06-29T14:40:04.187 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:40:04.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 -- 172.21.15.59:0/541086453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0d23c0 msgr2=0x7f20fc0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 --2- 172.21.15.59:0/541086453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0d23c0 0x7f20fc0b3990 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f20fc0d04f0 tx=0x7f20f802a560 comp rx=0 tx=0).stop 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 -- 172.21.15.59:0/541086453 shutdown_connections 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 --2- 172.21.15.59:0/541086453 >> v2:172.21.15.153:3300/0 conn(0x7f20fc15ee10 0x7f20fc15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 --2- 172.21.15.59:0/541086453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20fc0b3ed0 0x7f20fc15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 --2- 172.21.15.59:0/541086453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0d23c0 0x7f20fc0b3990 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 -- 172.21.15.59:0/541086453 >> 172.21.15.59:0/541086453 conn(0x7f20fc0d3ee0 msgr2=0x7f20fc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 -- 172.21.15.59:0/541086453 shutdown_connections 2024-06-29T14:40:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.311+0000 7f2103427640 1 -- 172.21.15.59:0/541086453 wait complete. 2024-06-29T14:40:04.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.313+0000 7f2103427640 1 Processor -- start 2024-06-29T14:40:04.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.313+0000 7f2103427640 1 -- start start 2024-06-29T14:40:04.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.314+0000 7f2103427640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0b3ed0 0x7f20fc0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.314+0000 7f2103427640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20fc0d23c0 0x7f20fc1e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f2103427640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20fc15ee10 0x7f20fc1e9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f2103427640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc160070 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f2103427640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc15fef0 con 0x7f20fc0d23c0 2024-06-29T14:40:04.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f2103427640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f20fc1601f0 con 0x7f20fc15ee10 2024-06-29T14:40:04.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0b3ed0 0x7f20fc0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0b3ed0 0x7f20fc0cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56988/0 (socket says 172.21.15.59:56988) 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 -- 172.21.15.59:0/3981402058 learned_addr learned my addr 172.21.15.59:0/3981402058 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21009c2640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20fc0d23c0 0x7f20fc1e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21019c4640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20fc15ee10 0x7f20fc1e9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 -- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20fc15ee10 msgr2=0x7f20fc1e9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20fc15ee10 0x7f20fc1e9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 -- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20fc0d23c0 msgr2=0x7f20fc1e9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20fc0d23c0 0x7f20fc1e9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 -- 172.21.15.59:0/3981402058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20fc15a3d0 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21009c2640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20fc0d23c0 0x7f20fc1e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21019c4640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20fc15ee10 0x7f20fc1e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f21011c3640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0b3ed0 0x7f20fc0cc760 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f20f80099c0 tx=0x7f20f80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f20f1ffb640 1 -- 172.21.15.59:0/3981402058 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f803e030 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f20f1ffb640 1 -- 172.21.15.59:0/3981402058 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f80047a0 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f2103427640 1 -- 172.21.15.59:0/3981402058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20fc1e8270 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.315+0000 7f20f1ffb640 1 -- 172.21.15.59:0/3981402058 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8035040 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.316+0000 7f2103427640 1 -- 172.21.15.59:0/3981402058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20fc1e8830 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.318+0000 7f20f1ffb640 1 -- 172.21.15.59:0/3981402058 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f20f8041030 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.318+0000 7f20f1ffb640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f20e0076420 0x7f20e0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.319+0000 7f20f1ffb640 1 -- 172.21.15.59:0/3981402058 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f20f80b83a0 con 0x7f20fc0b3ed0 2024-06-29T14:40:04.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.319+0000 7f21009c2640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f20e0076420 0x7f20e0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.319+0000 7f210a3bd740 1 Processor -- start 2024-06-29T14:40:04.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.319+0000 7f210a3bd740 1 -- start start 2024-06-29T14:40:04.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.319+0000 7f21009c2640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f20e0076420 0x7f20e0078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20fc0cd4f0 tx=0x7f20ec005d50 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.320+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88c16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.320+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.320+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.320+0000 7f210a3bd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555bb883dbc0 con 0x555bb84d5270 2024-06-29T14:40:04.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.320+0000 7f210a3bd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555bb87ec630 con 0x555bb84b4590 2024-06-29T14:40:04.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.320+0000 7f210a3bd740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555bb87dbbd0 con 0x555bb84ac3b0 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88c16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88c16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57004/0 (socket says 172.21.15.59:57004) 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 -- 172.21.15.59:0/461961510 learned_addr learned my addr 172.21.15.59:0/461961510 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21009c2640 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21019c4640 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 -- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 msgr2=0x555bb88a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 -- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 msgr2=0x555bb88a58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88a58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 -- 172.21.15.59:0/461961510 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bb87d8ca0 con 0x555bb84d5270 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21009c2640 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21019c4640 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88a58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f21011c3640 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88c16e0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f20f8081150 tx=0x7f20f8080b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf59f66b591d0f0b server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f20e6ffd640 1 -- 172.21.15.59:0/461961510 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8048070 con 0x555bb84d5270 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f20e6ffd640 1 -- 172.21.15.59:0/461961510 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f8080cd0 con 0x555bb84d5270 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.321+0000 7f20e6ffd640 1 -- 172.21.15.59:0/461961510 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80818c0 con 0x555bb84d5270 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 -- 172.21.15.59:0/461961510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 msgr2=0x555bb88c16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88c16e0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f20f8081150 tx=0x7f20f8080b10 comp rx=0 tx=0).stop 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 -- 172.21.15.59:0/461961510 shutdown_connections 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88a58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 --2- 172.21.15.59:0/461961510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88c16e0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 -- 172.21.15.59:0/461961510 >> 172.21.15.59:0/461961510 conn(0x555bb84c93c0 msgr2=0x555bb8878f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 -- 172.21.15.59:0/461961510 shutdown_connections 2024-06-29T14:40:04.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.322+0000 7f210a3bd740 1 -- 172.21.15.59:0/461961510 wait complete. 2024-06-29T14:40:04.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.323+0000 7f210a3bd740 1 -- start start 2024-06-29T14:40:04.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.323+0000 7f210a3bd740 1 Processor -- start 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.324+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84ac3b0 0x555bb8899b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.324+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84b4590 0x555bb88d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.324+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.324+0000 7f210a3bd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555bb8779ae0 con 0x555bb84d5270 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.324+0000 7f210a3bd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555bb87dfc10 con 0x555bb84ac3b0 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.324+0000 7f210a3bd740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555bb854f620 con 0x555bb84b4590 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f210a3bd740 10 client.0 _ll_register_callbacks cb 0x555bb8786480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21011c3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84ac3b0 0x555bb8899b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57016/0 (socket says 172.21.15.59:57016) 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 -- 172.21.15.59:0/201956711 learned_addr learned my addr 172.21.15.59:0/201956711 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21009c2640 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84b4590 0x555bb88d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 -- 172.21.15.59:0/201956711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84b4590 msgr2=0x555bb88d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84b4590 0x555bb88d2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 -- 172.21.15.59:0/201956711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84ac3b0 msgr2=0x555bb8899b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84ac3b0 0x555bb8899b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 -- 172.21.15.59:0/201956711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bb8540850 con 0x555bb84d5270 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f21019c4640 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3500 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f20f4010500 tx=0x7f20f40109c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68365e0ff216509 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f20dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f20dbfff640 1 -- 172.21.15.59:0/201956711 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4011970 con 0x555bb84d5270 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f20dbfff640 1 -- 172.21.15.59:0/201956711 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f20f4011b10 con 0x555bb84d5270 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.325+0000 7f210a3bd740 1 -- 172.21.15.59:0/201956711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20f8080cd0 con 0x555bb84d5270 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f20dbfff640 1 -- 172.21.15.59:0/201956711 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f20f4011d10 con 0x555bb84d5270 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f210a3bd740 10 client.8496 fetch_fsmap learned FSMap version 63 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f210a3bd740 1 -- 172.21.15.59:0/201956711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) -- 0x555bb85b76b0 con 0x555bb84d5270 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f20dbfff640 1 -- 172.21.15.59:0/201956711 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f8080cd0 con 0x555bb84d5270 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f20dbfff640 1 -- 172.21.15.59:0/201956711 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 63) ==== 3502+0+0 (secure 0 0 0) 0x7f20f4026080 con 0x555bb84d5270 2024-06-29T14:40:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f210a3bd740 10 client.8496 fetch_fsmap finished waiting for FSMap version 63 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f210a3bd740 10 client.8496 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f210a3bd740 20 client.8496 populate_metadata read hostname 'smithi059' 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f210a3bd740 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] conn(0x555bb8493e40 0x555bb88dd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f210a3bd740 4 client.8496 mds_command: new command op to 8361 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f210a3bd740 1 -- 172.21.15.59:0/201956711 --> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bb87ee580 con 0x555bb8493e40 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f21011c3640 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] conn(0x555bb8493e40 0x555bb88dd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.326+0000 7f20dbfff640 10 client.8496 ms_handle_connect on v2:172.21.15.59:6834/3950841523 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f21011c3640 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] conn(0x555bb8493e40 0x555bb88dd2d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca66744aef5fe3d server_cookie=4e6dbcdfd3608fcc in_seq=0 out_seq=0 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f20dbfff640 1 -- 172.21.15.59:0/201956711 <== mds.0 v2:172.21.15.59:6834/3950841523 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555bb87ee580 con 0x555bb8493e40 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f20dbfff640 10 client.8496 handle_command_reply: tid=0 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 client.8496 shutdown 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 -- 172.21.15.59:0/201956711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 msgr2=0x555bb88d3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3500 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f20f4010500 tx=0x7f20f40109c0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 -- 172.21.15.59:0/201956711 shutdown_connections 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] conn(0x555bb8493e40 0x555bb88dd2d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3500 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84b4590 0x555bb88d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f20dbfff640 0 client.8496 ms_handle_reset on v2:172.21.15.59:6834/3950841523 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 --2- 172.21.15.59:0/201956711 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84ac3b0 0x555bb8899b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.327+0000 7f210a3bd740 1 -- 172.21.15.59:0/201956711 >> 172.21.15.59:0/201956711 conn(0x555bb84c93c0 msgr2=0x555bb88a1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.328+0000 7f210a3bd740 1 -- 172.21.15.59:0/201956711 shutdown_connections 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.328+0000 7f210a3bd740 1 -- 172.21.15.59:0/201956711 wait complete. 2024-06-29T14:40:04.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.328+0000 7f210a3bd740 20 client.8496 trim_cache size 0 max 16384 2024-06-29T14:40:04.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.347+0000 7f210a3bd740 1 Processor -- start 2024-06-29T14:40:04.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.347+0000 7f210a3bd740 1 -- start start 2024-06-29T14:40:04.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88772f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f210a3bd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc200d50 con 0x555bb84d5270 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f210a3bd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc1f9790 con 0x555bb84b4590 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f210a3bd740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f20fc159a90 con 0x555bb84ac3b0 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f21019c4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f21019c4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57032/0 (socket says 172.21.15.59:57032) 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.348+0000 7f21019c4640 1 -- 172.21.15.59:0/1946860576 learned_addr learned my addr 172.21.15.59:0/1946860576 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21011c3640 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88772f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21009c2640 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21019c4640 1 -- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 msgr2=0x555bb88772f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21019c4640 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88772f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21019c4640 1 -- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 msgr2=0x555bb88d2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21019c4640 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21019c4640 1 -- 172.21.15.59:0/1946860576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bb87aae50 con 0x555bb84d5270 2024-06-29T14:40:04.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21009c2640 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f21019c4640 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3420 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009d30 tx=0x7f20f4005f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aced6dbf7bf0985 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f20e67fc640 1 -- 172.21.15.59:0/1946860576 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4009890 con 0x555bb84d5270 2024-06-29T14:40:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f20e67fc640 1 -- 172.21.15.59:0/1946860576 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f4004d00 con 0x555bb84d5270 2024-06-29T14:40:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.349+0000 7f20e67fc640 1 -- 172.21.15.59:0/1946860576 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f40119b0 con 0x555bb84d5270 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 -- 172.21.15.59:0/1946860576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 msgr2=0x555bb88d3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3420 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009d30 tx=0x7f20f4005f50 comp rx=0 tx=0).stop 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 -- 172.21.15.59:0/1946860576 shutdown_connections 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d3420 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 --2- 172.21.15.59:0/1946860576 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb88772f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 -- 172.21.15.59:0/1946860576 >> 172.21.15.59:0/1946860576 conn(0x555bb84c93c0 msgr2=0x555bb8871340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 -- 172.21.15.59:0/1946860576 shutdown_connections 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.350+0000 7f210a3bd740 1 -- 172.21.15.59:0/1946860576 wait complete. 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.351+0000 7f210a3bd740 1 -- start start 2024-06-29T14:40:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.351+0000 7f210a3bd740 1 Processor -- start 2024-06-29T14:40:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.352+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb893cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.352+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.352+0000 7f210a3bd740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.352+0000 7f210a3bd740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555bb889a070 con 0x555bb84d5270 2024-06-29T14:40:04.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.352+0000 7f210a3bd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555bb88cfd60 con 0x555bb84b4590 2024-06-29T14:40:04.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.352+0000 7f210a3bd740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555bb88a0050 con 0x555bb84ac3b0 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.352+0000 7f210a3bd740 10 client.0 _ll_register_callbacks cb 0x7f20f4009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.352+0000 7f21019c4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21019c4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57042/0 (socket says 172.21.15.59:57042) 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21019c4640 1 -- 172.21.15.59:0/2188847122 learned_addr learned my addr 172.21.15.59:0/2188847122 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21011c3640 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb893cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21009c2640 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21019c4640 1 -- 172.21.15.59:0/2188847122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 msgr2=0x555bb893cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21019c4640 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb893cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21019c4640 1 -- 172.21.15.59:0/2188847122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 msgr2=0x555bb88d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21019c4640 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21019c4640 1 -- 172.21.15.59:0/2188847122 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bb887cfa0 con 0x555bb84d5270 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f21019c4640 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d33c0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009bd0 tx=0x7f20f4010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae33df8ae6657f0f server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f20dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f20dbfff640 1 -- 172.21.15.59:0/2188847122 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4036470 con 0x555bb84d5270 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f20dbfff640 1 -- 172.21.15.59:0/2188847122 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f20f4036610 con 0x555bb84d5270 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f210a3bd740 1 -- 172.21.15.59:0/2188847122 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20f4004d00 con 0x555bb84d5270 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f20dbfff640 1 -- 172.21.15.59:0/2188847122 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f20f4042050 con 0x555bb84d5270 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f210a3bd740 10 client.8502 fetch_fsmap learned FSMap version 63 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.353+0000 7f210a3bd740 1 -- 172.21.15.59:0/2188847122 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) -- 0x555bb88a39a0 con 0x555bb84d5270 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f20dbfff640 1 -- 172.21.15.59:0/2188847122 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f4004d00 con 0x555bb84d5270 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f20dbfff640 1 -- 172.21.15.59:0/2188847122 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 63) ==== 3502+0+0 (secure 0 0 0) 0x7f20f4045020 con 0x555bb84d5270 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f210a3bd740 10 client.8502 fetch_fsmap finished waiting for FSMap version 63 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f210a3bd740 10 client.8502 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f210a3bd740 20 client.8502 populate_metadata read hostname 'smithi059' 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f210a3bd740 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] conn(0x555bb89413e0 0x555bb88a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f210a3bd740 4 client.8502 mds_command: new command op to 8361 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:40:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f210a3bd740 1 -- 172.21.15.59:0/2188847122 --> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] -- command(tid 0: {"prefix": "status"}) -- 0x555bb87ea020 con 0x555bb89413e0 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f21011c3640 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] conn(0x555bb89413e0 0x555bb88a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f20dbfff640 10 client.8502 ms_handle_connect on v2:172.21.15.59:6834/3950841523 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.354+0000 7f21011c3640 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] conn(0x555bb89413e0 0x555bb88a0d50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aadb0492f6a947f0 server_cookie=36bdcc06f2454a79 in_seq=0 out_seq=0 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f20dbfff640 1 -- 172.21.15.59:0/2188847122 <== mds.0 v2:172.21.15.59:6834/3950841523 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x555bb87ea020 con 0x555bb89413e0 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f20dbfff640 10 client.8502 handle_command_reply: tid=0 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 client.8502 shutdown 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 -- 172.21.15.59:0/2188847122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 msgr2=0x555bb88d33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d33c0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009bd0 tx=0x7f20f4010f40 comp rx=0 tx=0).stop 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 -- 172.21.15.59:0/2188847122 shutdown_connections 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.59:6834/3950841523,v1:172.21.15.59:6835/3950841523] conn(0x555bb89413e0 0x555bb88a0d50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555bb84d5270 0x555bb88d33c0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bb84b4590 0x555bb88d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 --2- 172.21.15.59:0/2188847122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555bb84ac3b0 0x555bb893cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f20dbfff640 0 client.8502 ms_handle_reset on v2:172.21.15.59:6834/3950841523 2024-06-29T14:40:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 -- 172.21.15.59:0/2188847122 >> 172.21.15.59:0/2188847122 conn(0x555bb84c93c0 msgr2=0x555bb88d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 -- 172.21.15.59:0/2188847122 shutdown_connections 2024-06-29T14:40:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 1 -- 172.21.15.59:0/2188847122 wait complete. 2024-06-29T14:40:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.355+0000 7f210a3bd740 20 client.8502 trim_cache size 0 max 16384 2024-06-29T14:40:04.361 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:40:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:40:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:40:04.361 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 8361, 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8594783689999996, 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 63, 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 129, 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 129, 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.489144187999999 2024-06-29T14:40:04.362 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:40:04.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 -- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f20e0076420 msgr2=0x7f20e0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f20e0076420 0x7f20e0078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20fc0cd4f0 tx=0x7f20ec005d50 comp rx=0 tx=0).stop 2024-06-29T14:40:04.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 -- 172.21.15.59:0/3981402058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0b3ed0 msgr2=0x7f20fc0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0b3ed0 0x7f20fc0cc760 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f20f80099c0 tx=0x7f20f80039e0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 -- 172.21.15.59:0/3981402058 shutdown_connections 2024-06-29T14:40:04.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20fc15ee10 0x7f20fc1e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20fc0d23c0 0x7f20fc1e9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f20e0076420 0x7f20e0078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 --2- 172.21.15.59:0/3981402058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20fc0b3ed0 0x7f20fc0cc760 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 -- 172.21.15.59:0/3981402058 >> 172.21.15.59:0/3981402058 conn(0x7f20fc0d3ee0 msgr2=0x7f20fc06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 -- 172.21.15.59:0/3981402058 shutdown_connections 2024-06-29T14:40:04.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.356+0000 7f2103427640 1 -- 172.21.15.59:0/3981402058 wait complete. 2024-06-29T14:40:04.365 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:40:04.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.484+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/94931041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 msgr2=0x7f6c60003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.484+0000 7f6c7e3a5640 1 --2- 172.21.15.59:0/94931041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 0x7f6c60003ee0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800bbe0 tx=0x7f6c60005830 comp rx=0 tx=0).stop 2024-06-29T14:40:04.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.484+0000 7f6c7c942640 1 -- 172.21.15.59:0/94931041 reap_dead start 2024-06-29T14:40:04.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.485+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/94931041 shutdown_connections 2024-06-29T14:40:04.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.485+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/94931041 >> 172.21.15.59:0/94931041 conn(0x7f6c780b5610 msgr2=0x7f6c780b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.485+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/94931041 shutdown_connections 2024-06-29T14:40:04.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.485+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/94931041 wait complete. 2024-06-29T14:40:04.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.486+0000 7f6c7e3a5640 1 Processor -- start 2024-06-29T14:40:04.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.487+0000 7f6c7e3a5640 1 -- start start 2024-06-29T14:40:04.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c7e3a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 0x7f6c78159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c7e3a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c781f85f0 0x7f6c781f8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c7e3a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c780c9510 0x7f6c780c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c7e3a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6c780b7bd0 con 0x7f6c60003a70 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c7e3a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c780d1c70 con 0x7f6c780c9510 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c7e3a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6c780b7d50 con 0x7f6c781f85f0 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 0x7f6c78159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 0x7f6c78159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57064/0 (socket says 172.21.15.59:57064) 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 -- 172.21.15.59:0/44382071 learned_addr learned my addr 172.21.15.59:0/44382071 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c777fe640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c781f85f0 0x7f6c781f8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c7c942640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c780c9510 0x7f6c780c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 -- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c781f85f0 msgr2=0x7f6c781f8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c781f85f0 0x7f6c781f8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 -- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c780c9510 msgr2=0x7f6c780c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c780c9510 0x7f6c780c9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 -- 172.21.15.59:0/44382071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c780c9e60 con 0x7f6c60003a70 2024-06-29T14:40:04.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c777fe640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c781f85f0 0x7f6c781f8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:40:04.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.488+0000 7f6c77fff640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 0x7f6c78159ac0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800bb00 tx=0x7f6c68031c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.489+0000 7f6c74ff9640 1 -- 172.21.15.59:0/44382071 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68034040 con 0x7f6c60003a70 2024-06-29T14:40:04.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.489+0000 7f6c74ff9640 1 -- 172.21.15.59:0/44382071 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c6803c070 con 0x7f6c60003a70 2024-06-29T14:40:04.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.489+0000 7f6c74ff9640 1 -- 172.21.15.59:0/44382071 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6800b7b0 con 0x7f6c60003a70 2024-06-29T14:40:04.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.489+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c781e5a10 con 0x7f6c60003a70 2024-06-29T14:40:04.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.489+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c781e5fd0 con 0x7f6c60003a70 2024-06-29T14:40:04.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.490+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c780cf080 con 0x7f6c60003a70 2024-06-29T14:40:04.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.490+0000 7f6c74ff9640 1 -- 172.21.15.59:0/44382071 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f6c6802e420 con 0x7f6c60003a70 2024-06-29T14:40:04.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.491+0000 7f6c74ff9640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6c640763f0 0x7f6c64078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.491+0000 7f6c74ff9640 1 -- 172.21.15.59:0/44382071 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f6c680b8b20 con 0x7f6c60003a70 2024-06-29T14:40:04.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.491+0000 7f6c777fe640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6c640763f0 0x7f6c64078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.492+0000 7f6c777fe640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6c640763f0 0x7f6c64078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c005fd0 tx=0x7f6c6c00eea0 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.495+0000 7f6c74ff9640 1 -- 172.21.15.59:0/44382071 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6c780c9e60 con 0x7f6c60003a70 2024-06-29T14:40:04.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.724+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6c78154ed0 con 0x7f6c60003a70 2024-06-29T14:40:04.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.725+0000 7f6c74ff9640 1 -- 172.21.15.59:0/44382071 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v63) ==== 53+0+73 (secure 0 0 0) 0x7f6c68082150 con 0x7f6c60003a70 2024-06-29T14:40:04.723 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:40:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.726+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6c640763f0 msgr2=0x7f6c64078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.726+0000 7f6c7e3a5640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6c640763f0 0x7f6c64078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c005fd0 tx=0x7f6c6c00eea0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.726+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 msgr2=0x7f6c78159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.726+0000 7f6c7e3a5640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 0x7f6c78159ac0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800bb00 tx=0x7f6c68031c50 comp rx=0 tx=0).stop 2024-06-29T14:40:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.727+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 shutdown_connections 2024-06-29T14:40:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.727+0000 7f6c7e3a5640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c780c9510 0x7f6c780c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.727+0000 7f6c7e3a5640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c781f85f0 0x7f6c781f8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.727+0000 7f6c7e3a5640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6c640763f0 0x7f6c64078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.727+0000 7f6c7e3a5640 1 --2- 172.21.15.59:0/44382071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c60003a70 0x7f6c78159ac0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.727+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 >> 172.21.15.59:0/44382071 conn(0x7f6c780b5610 msgr2=0x7f6c7806d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.727+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 shutdown_connections 2024-06-29T14:40:04.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04.727+0000 7f6c7e3a5640 1 -- 172.21.15.59:0/44382071 wait complete. 2024-06-29T14:40:04.736 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:40:04.736 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:40:04.736 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:40:04.736 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:40:04.736 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:40:04.736 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:40:04.737 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:40:04.737 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:40:04.737 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:40:04.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:04.737 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:40:04.753 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:40:04.754 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 41646sec preferred_lft 41646sec 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:04.755 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T14:40:04.756 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:04.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:04.757 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:40:04.818 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:40:04.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:04.820 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:40:04.876 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:40:04.876 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:40:04.876 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:40:04.876 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:40:04.876 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:40:04.877 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:40:04.877 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:40:04.877 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:40:04.877 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:40:04.877 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:40:04.877 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:40:04.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:40:05.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:40:05.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:05.008 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:40:05.023 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:40:05.024 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:40:05.024 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:40:05.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:05.024 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:40:05.090 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:40:05.091 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:40:05.147 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:40:05.147 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:40:05.148 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:40:05.204 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:40:05.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:05.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:40:05.299 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:40:05.299 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:40:05.299 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:40:05.310 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:40:05.310 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:40:05.310 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:40:05.310 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:40:05.310 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:40:05.316 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:40:05.317 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:40:05.317 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:40:05.317 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:40:05.317 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:40:05.317 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:40:05.317 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:40:05.317 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:40:05.317 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:40:05.318 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:40:05.318 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:40:05.318 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:40:05.318 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:40:05.320 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:40:05.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:05.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:40:05.346 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:40:05.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:40:05.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:40:05.449 INFO:teuthology.orchestra.run.smithi059.stdout:client7942 2024-06-29T14:40:05.450 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:40:05.450 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:40:05.467 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:40:05.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 -- 172.21.15.59:0/2405521570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 msgr2=0x7f4ecc0736c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 --2- 172.21.15.59:0/2405521570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 0x7f4ecc0736c0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc06f960 tx=0x7f4ebc02aa30 comp rx=0 tx=0).stop 2024-06-29T14:40:05.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 -- 172.21.15.59:0/2405521570 shutdown_connections 2024-06-29T14:40:05.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 --2- 172.21.15.59:0/2405521570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 0x7f4ecc0736c0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 --2- 172.21.15.59:0/2405521570 >> v2:172.21.15.153:3300/0 conn(0x7f4ecc06d2f0 0x7f4ecc06d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 --2- 172.21.15.59:0/2405521570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ecc0cd9b0 0x7f4ecc15bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 -- 172.21.15.59:0/2405521570 >> 172.21.15.59:0/2405521570 conn(0x7f4ecc0b24c0 msgr2=0x7f4ecc0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:05.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 -- 172.21.15.59:0/2405521570 shutdown_connections 2024-06-29T14:40:05.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.627+0000 7f4ed352a640 1 -- 172.21.15.59:0/2405521570 wait complete. 2024-06-29T14:40:05.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.629+0000 7f4ed352a640 1 Processor -- start 2024-06-29T14:40:05.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.629+0000 7f4ed352a640 1 -- start start 2024-06-29T14:40:05.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed352a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ecc06d2f0 0x7f4ecc138c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed352a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 0x7f4ecc137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed352a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ecc0cd9b0 0x7f4ecc0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed352a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc1448f0 con 0x7f4ecc0732b0 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed352a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc144770 con 0x7f4ecc06d2f0 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed352a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4ecc073c00 con 0x7f4ecc0cd9b0 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed0ac5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 0x7f4ecc137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed0ac5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 0x7f4ecc137530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57080/0 (socket says 172.21.15.59:57080) 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed0ac5640 1 -- 172.21.15.59:0/935747826 learned_addr learned my addr 172.21.15.59:0/935747826 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed1ac7640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ecc0cd9b0 0x7f4ecc0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.630+0000 7f4ed12c6640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ecc06d2f0 0x7f4ecc138c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed0ac5640 1 -- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ecc0cd9b0 msgr2=0x7f4ecc0b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed0ac5640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ecc0cd9b0 0x7f4ecc0b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed0ac5640 1 -- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ecc06d2f0 msgr2=0x7f4ecc138c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed0ac5640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ecc06d2f0 0x7f4ecc138c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed0ac5640 1 -- 172.21.15.59:0/935747826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ecc13b8c0 con 0x7f4ecc0732b0 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed1ac7640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ecc0cd9b0 0x7f4ecc0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed12c6640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ecc06d2f0 0x7f4ecc138c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed0ac5640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 0x7f4ecc137530 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000d750 tx=0x7f4ec000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ec9ffb640 1 -- 172.21.15.59:0/935747826 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0017070 con 0x7f4ecc0732b0 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ec9ffb640 1 -- 172.21.15.59:0/935747826 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ec0009cc0 con 0x7f4ecc0732b0 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ecc13be40 con 0x7f4ecc0732b0 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ec9ffb640 1 -- 172.21.15.59:0/935747826 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00106c0 con 0x7f4ecc0732b0 2024-06-29T14:40:05.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.631+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ecc13bb60 con 0x7f4ecc0732b0 2024-06-29T14:40:05.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.632+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ecc074100 con 0x7f4ecc0732b0 2024-06-29T14:40:05.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.633+0000 7f4ec9ffb640 1 -- 172.21.15.59:0/935747826 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f4ec001a070 con 0x7f4ecc0732b0 2024-06-29T14:40:05.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.634+0000 7f4ec9ffb640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4eb8076420 0x7f4eb8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.634+0000 7f4ec9ffb640 1 -- 172.21.15.59:0/935747826 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f4ec0097c80 con 0x7f4ecc0732b0 2024-06-29T14:40:05.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.634+0000 7f4ed12c6640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4eb8076420 0x7f4eb8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.635+0000 7f4ed12c6640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4eb8076420 0x7f4eb8078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4005fd0 tx=0x7f4ec400f360 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:05.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.639+0000 7f4ec9ffb640 1 -- 172.21.15.59:0/935747826 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ec000de60 con 0x7f4ecc0732b0 2024-06-29T14:40:05.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.853+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 --> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f4ecc0b0590 con 0x7f4eb8076420 2024-06-29T14:40:05.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.855+0000 7f4ec9ffb640 1 -- 172.21.15.59:0/935747826 <== mgr.7615 v2:172.21.15.153:6832/892574025 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f4ecc0b0590 con 0x7f4eb8076420 2024-06-29T14:40:05.853 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:40:05.853 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:40:05.853 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T14:40:05.853 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:40:05.853 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4eb8076420 msgr2=0x7f4eb8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4eb8076420 0x7f4eb8078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4005fd0 tx=0x7f4ec400f360 comp rx=0 tx=0).stop 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 msgr2=0x7f4ecc137530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 0x7f4ecc137530 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000d750 tx=0x7f4ec000dc10 comp rx=0 tx=0).stop 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 shutdown_connections 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ecc0cd9b0 0x7f4ecc0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f4eb8076420 0x7f4eb8078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ecc0732b0 0x7f4ecc137530 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 --2- 172.21.15.59:0/935747826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ecc06d2f0 0x7f4ecc138c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.857+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 >> 172.21.15.59:0/935747826 conn(0x7f4ecc0b24c0 msgr2=0x7f4ecc0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:05.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.858+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 shutdown_connections 2024-06-29T14:40:05.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.858+0000 7f4ed352a640 1 -- 172.21.15.59:0/935747826 wait complete. 2024-06-29T14:40:05.867 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:40:05.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.989+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3101246278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda44004800 msgr2=0x7fda44004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.989+0000 7fda5e2ce640 1 --2- 172.21.15.59:0/3101246278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda44004800 0x7fda44004c90 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fda40009b50 tx=0x7fda44005830 comp rx=0 tx=0).stop 2024-06-29T14:40:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.989+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3101246278 shutdown_connections 2024-06-29T14:40:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.989+0000 7fda5e2ce640 1 --2- 172.21.15.59:0/3101246278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda44004800 0x7fda44004c90 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.989+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3101246278 >> 172.21.15.59:0/3101246278 conn(0x7fda5806c950 msgr2=0x7fda58073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.989+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3101246278 shutdown_connections 2024-06-29T14:40:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.989+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3101246278 wait complete. 2024-06-29T14:40:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.991+0000 7fda5e2ce640 1 Processor -- start 2024-06-29T14:40:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.991+0000 7fda5e2ce640 1 -- start start 2024-06-29T14:40:05.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5e2ce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda44004800 0x7fda58138f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5e2ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5813e4b0 0x7fda581f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5e2ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5813b480 0x7fda5813b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5e2ce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fda5815c700 con 0x7fda5813b480 2024-06-29T14:40:05.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5e2ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda580b3fd0 con 0x7fda5813e4b0 2024-06-29T14:40:05.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5e2ce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda5815c880 con 0x7fda44004800 2024-06-29T14:40:05.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5c86b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5813b480 0x7fda5813b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5c86b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5813b480 0x7fda5813b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57104/0 (socket says 172.21.15.59:57104) 2024-06-29T14:40:05.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.992+0000 7fda5c86b640 1 -- 172.21.15.59:0/3351589877 learned_addr learned my addr 172.21.15.59:0/3351589877 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:05.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda57fff640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda44004800 0x7fda58138f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda4ffff640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5813e4b0 0x7fda581f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda5c86b640 1 -- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda44004800 msgr2=0x7fda58138f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda5c86b640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda44004800 0x7fda58138f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda5c86b640 1 -- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5813e4b0 msgr2=0x7fda581f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda5c86b640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5813e4b0 0x7fda581f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda5c86b640 1 -- 172.21.15.59:0/3351589877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5813bdd0 con 0x7fda5813b480 2024-06-29T14:40:05.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda5c86b640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5813b480 0x7fda5813b890 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fda580b55a0 tx=0x7fda50006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:05.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda557fa640 1 -- 172.21.15.59:0/3351589877 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5000f9b0 con 0x7fda5813b480 2024-06-29T14:40:05.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda557fa640 1 -- 172.21.15.59:0/3351589877 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda5000fb50 con 0x7fda5813b480 2024-06-29T14:40:05.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda58139780 con 0x7fda5813b480 2024-06-29T14:40:05.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda557fa640 1 -- 172.21.15.59:0/3351589877 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50011430 con 0x7fda5813b480 2024-06-29T14:40:05.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.993+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda58139d40 con 0x7fda5813b480 2024-06-29T14:40:05.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.994+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda580cf8c0 con 0x7fda5813b480 2024-06-29T14:40:05.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.995+0000 7fda557fa640 1 -- 172.21.15.59:0/3351589877 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fda500115d0 con 0x7fda5813b480 2024-06-29T14:40:05.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.995+0000 7fda557fa640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fda38076420 0x7fda38078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.996+0000 7fda557fa640 1 -- 172.21.15.59:0/3351589877 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7fda50098f30 con 0x7fda5813b480 2024-06-29T14:40:05.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.996+0000 7fda57fff640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fda38076420 0x7fda38078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.997+0000 7fda57fff640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fda38076420 0x7fda38078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda480064a0 tx=0x7fda48006b10 comp rx=0 tx=0).ready entity=mgr.7615 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:05.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:05.999+0000 7fda557fa640 1 -- 172.21.15.59:0/3351589877 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda500c8af0 con 0x7fda5813b480 2024-06-29T14:40:06.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.243+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fda58139510 con 0x7fda5813b480 2024-06-29T14:40:06.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.535+0000 7fda557fa640 1 -- 172.21.15.59:0/3351589877 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v58) ==== 92+0+0 (secure 0 0 0) 0x7fda5813bdd0 con 0x7fda5813b480 2024-06-29T14:40:06.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.537+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fda38076420 msgr2=0x7fda38078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.537+0000 7fda5e2ce640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fda38076420 0x7fda38078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda480064a0 tx=0x7fda48006b10 comp rx=0 tx=0).stop 2024-06-29T14:40:06.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.537+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5813b480 msgr2=0x7fda5813b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.537+0000 7fda5e2ce640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5813b480 0x7fda5813b890 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fda580b55a0 tx=0x7fda50006720 comp rx=0 tx=0).stop 2024-06-29T14:40:06.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.538+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 shutdown_connections 2024-06-29T14:40:06.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.538+0000 7fda5e2ce640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7fda38076420 0x7fda38078930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.538+0000 7fda5e2ce640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5813b480 0x7fda5813b890 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.538+0000 7fda5e2ce640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5813e4b0 0x7fda581f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.538+0000 7fda5e2ce640 1 --2- 172.21.15.59:0/3351589877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda44004800 0x7fda58138f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.538+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 >> 172.21.15.59:0/3351589877 conn(0x7fda5806c950 msgr2=0x7fda5806fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:06.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.538+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 shutdown_connections 2024-06-29T14:40:06.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.538+0000 7fda5e2ce640 1 -- 172.21.15.59:0/3351589877 wait complete. 2024-06-29T14:40:06.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:40:06.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.680+0000 7f97ae610640 1 -- 172.21.15.59:0/4122061023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2fa0 msgr2=0x7f97a80d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.680+0000 7f97ae610640 1 --2- 172.21.15.59:0/4122061023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2fa0 0x7f97a80d1520 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f979c009860 tx=0x7f979c02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.680+0000 7f97ae610640 1 -- 172.21.15.59:0/4122061023 shutdown_connections 2024-06-29T14:40:06.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.680+0000 7f97ae610640 1 --2- 172.21.15.59:0/4122061023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2fa0 0x7f97a80d1520 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.680+0000 7f97ae610640 1 -- 172.21.15.59:0/4122061023 >> 172.21.15.59:0/4122061023 conn(0x7f97a8073e10 msgr2=0x7f97a8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:06.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.680+0000 7f97ae610640 1 -- 172.21.15.59:0/4122061023 shutdown_connections 2024-06-29T14:40:06.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.680+0000 7f97ae610640 1 -- 172.21.15.59:0/4122061023 wait complete. 2024-06-29T14:40:06.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.682+0000 7f97ae610640 1 Processor -- start 2024-06-29T14:40:06.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.682+0000 7f97ae610640 1 -- start start 2024-06-29T14:40:06.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97ae610640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2270 0x7f97a815e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97ae610640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80b2fa0 0x7f97a81f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97ae610640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a80d1a60 0x7f97a80b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97ae610640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97a80b65d0 con 0x7f97a80b2fa0 2024-06-29T14:40:06.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97ae610640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97a80b6450 con 0x7f97a80d1a60 2024-06-29T14:40:06.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97ae610640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f97a80b6750 con 0x7f97a80b2270 2024-06-29T14:40:06.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80b2fa0 0x7f97a81f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80b2fa0 0x7f97a81f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57128/0 (socket says 172.21.15.59:57128) 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2270 0x7f97a815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a77fe640 1 -- 172.21.15.59:0/267763946 learned_addr learned my addr 172.21.15.59:0/267763946 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97acbad640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a80d1a60 0x7f97a80b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a7fff640 1 -- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a80d1a60 msgr2=0x7f97a80b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a7fff640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a80d1a60 0x7f97a80b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a7fff640 1 -- 172.21.15.59:0/267763946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80b2fa0 msgr2=0x7f97a81f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a7fff640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80b2fa0 0x7f97a81f3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.684+0000 7f97a7fff640 1 -- 172.21.15.59:0/267763946 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a815c5a0 con 0x7f97a80b2270 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.685+0000 7f97a7fff640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2270 0x7f97a815e010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f979800c3c0 tx=0x7f979800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.685+0000 7f97a4ff9640 1 -- 172.21.15.59:0/267763946 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798017070 con 0x7f97a80b2270 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.685+0000 7f97a4ff9640 1 -- 172.21.15.59:0/267763946 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97980053c0 con 0x7f97a80b2270 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.685+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a815c880 con 0x7f97a80b2270 2024-06-29T14:40:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.685+0000 7f97a4ff9640 1 -- 172.21.15.59:0/267763946 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798003850 con 0x7f97a80b2270 2024-06-29T14:40:06.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.685+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a815ce40 con 0x7f97a80b2270 2024-06-29T14:40:06.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.686+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a80b6c00 con 0x7f97a80b2270 2024-06-29T14:40:06.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.687+0000 7f97a4ff9640 1 -- 172.21.15.59:0/267763946 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 58) ==== 98320+0+0 (secure 0 0 0) 0x7f979801a070 con 0x7f97a80b2270 2024-06-29T14:40:06.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.687+0000 7f97a4ff9640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9794076440 0x7f9794078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.687+0000 7f97a77fe640 1 -- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9794076440 msgr2=0x7f9794078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/892574025 2024-06-29T14:40:06.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.687+0000 7f97a4ff9640 1 -- 172.21.15.59:0/267763946 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f9798096e50 con 0x7f97a80b2270 2024-06-29T14:40:06.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.687+0000 7f97a77fe640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9794076440 0x7f9794078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:40:06.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.692+0000 7f97a4ff9640 1 -- 172.21.15.59:0/267763946 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f979805fe30 con 0x7f97a80b2270 2024-06-29T14:40:06.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.885+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f97a8075b40 con 0x7f97a80b2270 2024-06-29T14:40:06.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.888+0000 7f97a77fe640 1 -- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9794076440 msgr2=0x7f9794078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/892574025 2024-06-29T14:40:06.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.889+0000 7f97a77fe640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9794076440 0x7f9794078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:40:06.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.893+0000 7f97a4ff9640 1 -- 172.21.15.59:0/267763946 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v49)=0 v49) ==== 138+0+0 (secure 0 0 0) 0x7f9798064130 con 0x7f97a80b2270 2024-06-29T14:40:06.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9794076440 msgr2=0x7f9794078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:40:06.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9794076440 0x7f9794078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2270 msgr2=0x7f97a815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2270 0x7f97a815e010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f979800c3c0 tx=0x7f979800c880 comp rx=0 tx=0).stop 2024-06-29T14:40:06.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 shutdown_connections 2024-06-29T14:40:06.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a80d1a60 0x7f97a80b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f9794076440 0x7f9794078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a80b2fa0 0x7f97a81f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 --2- 172.21.15.59:0/267763946 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a80b2270 0x7f97a815e010 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 >> 172.21.15.59:0/267763946 conn(0x7f97a8073e10 msgr2=0x7f97a815bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:06.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 shutdown_connections 2024-06-29T14:40:06.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:06.895+0000 7f97ae610640 1 -- 172.21.15.59:0/267763946 wait complete. 2024-06-29T14:40:06.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:40:07.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.028+0000 7f1dab7b9640 1 -- 172.21.15.59:0/3852958357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 msgr2=0x7f1da40d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:07.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.028+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/3852958357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 0x7f1da40d1520 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f1da40cf310 tx=0x7f1da002a530 comp rx=0 tx=0).stop 2024-06-29T14:40:07.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.028+0000 7f1dab7b9640 1 -- 172.21.15.59:0/3852958357 shutdown_connections 2024-06-29T14:40:07.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.028+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/3852958357 >> v2:172.21.15.153:3300/0 conn(0x7f1da40d1a60 0x7f1da40b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.028+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/3852958357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 0x7f1da40d1520 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.028+0000 7f1dab7b9640 1 -- 172.21.15.59:0/3852958357 >> 172.21.15.59:0/3852958357 conn(0x7f1da4073e10 msgr2=0x7f1da4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:07.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.028+0000 7f1dab7b9640 1 -- 172.21.15.59:0/3852958357 shutdown_connections 2024-06-29T14:40:07.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.028+0000 7f1dab7b9640 1 -- 172.21.15.59:0/3852958357 wait complete. 2024-06-29T14:40:07.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.030+0000 7f1dab7b9640 1 Processor -- start 2024-06-29T14:40:07.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.030+0000 7f1dab7b9640 1 -- start start 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.031+0000 7f1dab7b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da40b2270 0x7f1da4146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.031+0000 7f1dab7b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 0x7f1da4146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.031+0000 7f1dab7b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da40d1a60 0x7f1da415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.031+0000 7f1dab7b9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40b6400 con 0x7f1da40b2fa0 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.031+0000 7f1dab7b9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40b6280 con 0x7f1da40d1a60 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.031+0000 7f1dab7b9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1da40b6580 con 0x7f1da40b2270 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 0x7f1da4146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 0x7f1da4146c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57152/0 (socket says 172.21.15.59:57152) 2024-06-29T14:40:07.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 -- 172.21.15.59:0/1026958662 learned_addr learned my addr 172.21.15.59:0/1026958662 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da9555640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da40b2270 0x7f1da4146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da9d56640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da40d1a60 0x7f1da415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 -- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da40b2270 msgr2=0x7f1da4146360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da40b2270 0x7f1da4146360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 -- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da40d1a60 msgr2=0x7f1da415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da40d1a60 0x7f1da415eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 -- 172.21.15.59:0/1026958662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da415df50 con 0x7f1da40b2fa0 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da9d56640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da40d1a60 0x7f1da415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1da8d54640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 0x7f1da4146c20 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009920 tx=0x7f1da0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.032+0000 7f1d95ffb640 1 -- 172.21.15.59:0/1026958662 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0031d60 con 0x7f1da40b2fa0 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.033+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da415e230 con 0x7f1da40b2fa0 2024-06-29T14:40:07.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.033+0000 7f1d95ffb640 1 -- 172.21.15.59:0/1026958662 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1da0033e10 con 0x7f1da40b2fa0 2024-06-29T14:40:07.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.033+0000 7f1d95ffb640 1 -- 172.21.15.59:0/1026958662 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0005260 con 0x7f1da40b2fa0 2024-06-29T14:40:07.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.033+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da415e7f0 con 0x7f1da40b2fa0 2024-06-29T14:40:07.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.034+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da40b6aa0 con 0x7f1da40b2fa0 2024-06-29T14:40:07.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.035+0000 7f1d95ffb640 1 -- 172.21.15.59:0/1026958662 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 58) ==== 98320+0+0 (secure 0 0 0) 0x7f1da0005400 con 0x7f1da40b2fa0 2024-06-29T14:40:07.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.035+0000 7f1d95ffb640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1d88076440 0x7f1d88078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:07.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.035+0000 7f1da9555640 1 -- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1d88076440 msgr2=0x7f1d88078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/892574025 2024-06-29T14:40:07.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.035+0000 7f1d95ffb640 1 -- 172.21.15.59:0/1026958662 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f1da00b9120 con 0x7f1da40b2fa0 2024-06-29T14:40:07.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.035+0000 7f1da9555640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1d88076440 0x7f1d88078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:40:07.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.039+0000 7f1d95ffb640 1 -- 172.21.15.59:0/1026958662 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1da003f030 con 0x7f1da40b2fa0 2024-06-29T14:40:07.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.230+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f1da40ceb80 con 0x7f1da40b2fa0 2024-06-29T14:40:07.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.233+0000 7f1d95ffb640 1 -- 172.21.15.59:0/1026958662 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v50)=0 v50) ==== 133+0+0 (secure 0 0 0) 0x7f1da0081ff0 con 0x7f1da40b2fa0 2024-06-29T14:40:07.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1d88076440 msgr2=0x7f1d88078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:40:07.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1d88076440 0x7f1d88078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 msgr2=0x7f1da4146c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 0x7f1da4146c20 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009920 tx=0x7f1da0033c20 comp rx=0 tx=0).stop 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 shutdown_connections 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1da40d1a60 0x7f1da415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f1d88076440 0x7f1d88078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da40b2fa0 0x7f1da4146c20 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 --2- 172.21.15.59:0/1026958662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da40b2270 0x7f1da4146360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 >> 172.21.15.59:0/1026958662 conn(0x7f1da4073e10 msgr2=0x7f1da40d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 shutdown_connections 2024-06-29T14:40:07.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.236+0000 7f1dab7b9640 1 -- 172.21.15.59:0/1026958662 wait complete. 2024-06-29T14:40:07.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:07.246 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:40:07.269 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-06-29T14:40:07.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.432+0000 7f6475b9a640 1 -- 172.21.15.59:0/2886366714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f647015ee10 msgr2=0x7f647015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:07.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.432+0000 7f6475b9a640 1 --2- 172.21.15.59:0/2886366714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f647015ee10 0x7f647015f9b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6468005fd0 tx=0x7f646802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.432+0000 7f6475b9a640 1 -- 172.21.15.59:0/2886366714 shutdown_connections 2024-06-29T14:40:07.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.432+0000 7f6475b9a640 1 --2- 172.21.15.59:0/2886366714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f647015ee10 0x7f647015f9b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.432+0000 7f6475b9a640 1 --2- 172.21.15.59:0/2886366714 >> v2:172.21.15.153:3300/0 conn(0x7f64700b40a0 0x7f647015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.432+0000 7f6475b9a640 1 -- 172.21.15.59:0/2886366714 >> 172.21.15.59:0/2886366714 conn(0x7f64700d3ee0 msgr2=0x7f64700d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:07.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.432+0000 7f6475b9a640 1 -- 172.21.15.59:0/2886366714 shutdown_connections 2024-06-29T14:40:07.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.432+0000 7f6475b9a640 1 -- 172.21.15.59:0/2886366714 wait complete. 2024-06-29T14:40:07.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.434+0000 7f6475b9a640 1 Processor -- start 2024-06-29T14:40:07.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.434+0000 7f6475b9a640 1 -- start start 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.435+0000 7f6475b9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64700d23c0 0x7f64701f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.435+0000 7f6475b9a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f647015ee10 0x7f64700ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.435+0000 7f6475b9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647015c090 0x7f647015c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.435+0000 7f6475b9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6470160280 con 0x7f647015c090 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.435+0000 7f6475b9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6470160100 con 0x7f64700d23c0 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.435+0000 7f6475b9a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6470160400 con 0x7f647015ee10 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647015c090 0x7f647015c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647015c090 0x7f647015c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57160/0 (socket says 172.21.15.59:57160) 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 -- 172.21.15.59:0/1274168156 learned_addr learned my addr 172.21.15.59:0/1274168156 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:07.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f6467fff640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f647015ee10 0x7f64700ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646f7fe640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64700d23c0 0x7f64701f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f647015ee10 msgr2=0x7f64700ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f647015ee10 0x7f64700ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64700d23c0 msgr2=0x7f64701f4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64700d23c0 0x7f64701f4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 -- 172.21.15.59:0/1274168156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647015aff0 con 0x7f647015c090 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646ffff640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647015c090 0x7f647015c4a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f6468005fd0 tx=0x7f64680316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6468004040 con 0x7f647015c090 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64680042d0 con 0x7f647015c090 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f647015b2d0 con 0x7f647015c090 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.436+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64680047a0 con 0x7f647015c090 2024-06-29T14:40:07.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.437+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f647015b890 con 0x7f647015c090 2024-06-29T14:40:07.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.438+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f647015e8d0 con 0x7f647015c090 2024-06-29T14:40:07.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.438+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 58) ==== 98320+0+0 (secure 0 0 0) 0x7f646803b410 con 0x7f647015c090 2024-06-29T14:40:07.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.439+0000 7f646cff9640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 0x7f6450078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:07.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.439+0000 7f646f7fe640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 msgr2=0x7f6450078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/892574025 2024-06-29T14:40:07.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.439+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6101+0+0 (secure 0 0 0) 0x7f64680b8f50 con 0x7f647015c090 2024-06-29T14:40:07.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.439+0000 7f646f7fe640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 0x7f6450078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:40:07.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.443+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f646803f030 con 0x7f647015c090 2024-06-29T14:40:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.640+0000 7f646f7fe640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 msgr2=0x7f6450078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/892574025 2024-06-29T14:40:07.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.640+0000 7f646f7fe640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 0x7f6450078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:40:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:07.660+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 --> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f64700d2950 con 0x7f6450076440 2024-06-29T14:40:08.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:08.041+0000 7f646f7fe640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 msgr2=0x7f6450078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/892574025 2024-06-29T14:40:08.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:08.041+0000 7f646f7fe640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 0x7f6450078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:40:08.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:08.842+0000 7f646f7fe640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 msgr2=0x7f6450078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/892574025 2024-06-29T14:40:08.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:08.842+0000 7f646f7fe640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 0x7f6450078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:40:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:10.149+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 59) ==== 97929+0+0 (secure 0 0 0) 0x7f64680b72e0 con 0x7f647015c090 2024-06-29T14:40:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:10.149+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 msgr2=0x7f6450078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:40:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:10.149+0000 7f646cff9640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/892574025,v1:172.21.15.153:6833/892574025] conn(0x7f6450076440 0x7f6450078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d40077640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d40077640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d40077640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d40077640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d40077640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d4307d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d4307d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d4307d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d4307d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.309 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:40:10.310+0000 7f6d4307d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:11.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.151+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 60) ==== 98056+0+0 (secure 0 0 0) 0x7f646807e1d0 con 0x7f647015c090 2024-06-29T14:40:11.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.152+0000 7f646cff9640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f645007e990 0x7f6450080db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:11.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.152+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 --> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f64700d2950 con 0x7f645007e990 2024-06-29T14:40:11.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.152+0000 7f646f7fe640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f645007e990 0x7f6450080db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:11.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.157+0000 7f646f7fe640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f645007e990 0x7f6450080db0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6460007a50 tx=0x7f64600079e0 comp rx=0 tx=0).ready entity=mgr.7175 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:11.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.234+0000 7f646cff9640 1 -- 172.21.15.59:0/1274168156 <== mgr.7175 v2:172.21.15.153:6832/3741042522 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f64700d2950 con 0x7f645007e990 2024-06-29T14:40:11.233 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.236+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f645007e990 msgr2=0x7f6450080db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.236+0000 7f6475b9a640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f645007e990 0x7f6450080db0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6460007a50 tx=0x7f64600079e0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.236+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647015c090 msgr2=0x7f647015c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.236+0000 7f6475b9a640 1 --2- 172.21.15.59:0/1274168156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647015c090 0x7f647015c4a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f6468005fd0 tx=0x7f64680316f0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.236+0000 7f646ffff640 1 -- 172.21.15.59:0/1274168156 reap_dead start 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.236+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 shutdown_connections 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.236+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 >> 172.21.15.59:0/1274168156 conn(0x7f64700d3ee0 msgr2=0x7f647006ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.237+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 shutdown_connections 2024-06-29T14:40:11.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.237+0000 7f6475b9a640 1 -- 172.21.15.59:0/1274168156 wait complete. 2024-06-29T14:40:11.248 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T14:40:11.248 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-14_41_00 in ~49s... 2024-06-29T14:40:11.248 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-06-29T14:40:11.248 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-06-29T14:40:11.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.374+0000 7f174d0f6640 1 -- 172.21.15.59:0/168484520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17481444c0 msgr2=0x7f174806d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.374+0000 7f174d0f6640 1 --2- 172.21.15.59:0/168484520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17481444c0 0x7f174806d2f0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f1748070d30 tx=0x7f173002a560 comp rx=0 tx=0).stop 2024-06-29T14:40:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.374+0000 7f174d0f6640 1 -- 172.21.15.59:0/168484520 shutdown_connections 2024-06-29T14:40:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.374+0000 7f174d0f6640 1 --2- 172.21.15.59:0/168484520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17481444c0 0x7f174806d2f0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.374+0000 7f174d0f6640 1 --2- 172.21.15.59:0/168484520 >> v2:172.21.15.153:3300/0 conn(0x7f17480cd9b0 0x7f174815f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.374+0000 7f174d0f6640 1 -- 172.21.15.59:0/168484520 >> 172.21.15.59:0/168484520 conn(0x7f17480b24c0 msgr2=0x7f17480cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.374+0000 7f174d0f6640 1 -- 172.21.15.59:0/168484520 shutdown_connections 2024-06-29T14:40:11.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.374+0000 7f174d0f6640 1 -- 172.21.15.59:0/168484520 wait complete. 2024-06-29T14:40:11.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.376+0000 7f174d0f6640 1 Processor -- start 2024-06-29T14:40:11.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.376+0000 7f174d0f6640 1 -- start start 2024-06-29T14:40:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.377+0000 7f174d0f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f174806d830 0x7f17480ae450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.377+0000 7f174d0f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17480cd9b0 0x7f17481f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.377+0000 7f174d0f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17481444c0 0x7f174806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.377+0000 7f174d0f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f174815b9b0 con 0x7f17480cd9b0 2024-06-29T14:40:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.377+0000 7f174d0f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17481448d0 con 0x7f174806d830 2024-06-29T14:40:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.377+0000 7f174d0f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f174815bb30 con 0x7f17481444c0 2024-06-29T14:40:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17480cd9b0 0x7f17481f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17480cd9b0 0x7f17481f4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57192/0 (socket says 172.21.15.59:57192) 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 -- 172.21.15.59:0/995191819 learned_addr learned my addr 172.21.15.59:0/995191819 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f1746d9d640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f174806d830 0x7f17480ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174759e640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17481444c0 0x7f174806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 -- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17481444c0 msgr2=0x7f174806c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17481444c0 0x7f174806c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 -- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f174806d830 msgr2=0x7f17480ae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f174806d830 0x7f17480ae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 -- 172.21.15.59:0/995191819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174813a0c0 con 0x7f17480cd9b0 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174659c640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17480cd9b0 0x7f17481f4780 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f1730034040 tx=0x7f17300039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f173f7fe640 1 -- 172.21.15.59:0/995191819 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173003e030 con 0x7f17480cd9b0 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f173f7fe640 1 -- 172.21.15.59:0/995191819 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17300047a0 con 0x7f17480cd9b0 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174813bb60 con 0x7f17480cd9b0 2024-06-29T14:40:11.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f173f7fe640 1 -- 172.21.15.59:0/995191819 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730035040 con 0x7f17480cd9b0 2024-06-29T14:40:11.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.378+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174813c120 con 0x7f17480cd9b0 2024-06-29T14:40:11.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.380+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174815c010 con 0x7f17480cd9b0 2024-06-29T14:40:11.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.380+0000 7f173f7fe640 1 -- 172.21.15.59:0/995191819 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 60) ==== 98056+0+0 (secure 0 0 0) 0x7f1730041030 con 0x7f17480cd9b0 2024-06-29T14:40:11.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.380+0000 7f173f7fe640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f1728076170 0x7f1728078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:11.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.380+0000 7f173f7fe640 1 -- 172.21.15.59:0/995191819 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7f17300b80d0 con 0x7f17480cd9b0 2024-06-29T14:40:11.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.380+0000 7f1746d9d640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f1728076170 0x7f1728078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:11.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.380+0000 7f1746d9d640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f1728076170 0x7f1728078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f173800b810 tx=0x7f1738008040 comp rx=0 tx=0).ready entity=mgr.7175 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:11.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.385+0000 7f173f7fe640 1 -- 172.21.15.59:0/995191819 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f174806dc40 con 0x7f17480cd9b0 2024-06-29T14:40:11.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.597+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 --> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1748139e50 con 0x7f1728076170 2024-06-29T14:40:11.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.605+0000 7f173f7fe640 1 -- 172.21.15.59:0/995191819 <== mgr.7175 v2:172.21.15.153:6832/3741042522 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+66 (secure 0 0 0) 0x7f1748139e50 con 0x7f1728076170 2024-06-29T14:40:11.602 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:40:11.602 INFO:teuthology.orchestra.run.smithi059.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-06-29T14:40:11.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f1728076170 msgr2=0x7f1728078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:11.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f1728076170 0x7f1728078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f173800b810 tx=0x7f1738008040 comp rx=0 tx=0).stop 2024-06-29T14:40:11.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17480cd9b0 msgr2=0x7f17481f4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17480cd9b0 0x7f17481f4780 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f1730034040 tx=0x7f17300039e0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 shutdown_connections 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f1728076170 0x7f1728078680 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17481444c0 0x7f174806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17480cd9b0 0x7f17481f4780 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 --2- 172.21.15.59:0/995191819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f174806d830 0x7f17480ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 >> 172.21.15.59:0/995191819 conn(0x7f17480b24c0 msgr2=0x7f17480682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 shutdown_connections 2024-06-29T14:40:11.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:40:11.607+0000 7f174d0f6640 1 -- 172.21.15.59:0/995191819 wait complete. 2024-06-29T14:40:11.616 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-06-29T14:40:11.616 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-06-29T14:40:11.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:11.616 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:11.633 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:11.633 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:11.633 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:12.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:12.634 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:12.655 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:12.655 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:12.655 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:13.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:13.656 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:13.677 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:13.677 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:13.677 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:14.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:14.678 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:14.699 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:14.699 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:14.699 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:15.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:15.700 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:15.721 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:15.721 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:15.721 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:16.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:16.722 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:16.743 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:16.743 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:16.743 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:17.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:17.745 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:17.765 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:17.766 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:17.766 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:18.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:18.766 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:18.788 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:18.788 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:18.788 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:19.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:19.789 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:19.810 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:19.810 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:19.810 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:20.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:20.811 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:20.832 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:20.833 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:20.833 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:21.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:21.834 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:21.855 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:21.855 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:21.855 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:22.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:22.856 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:22.877 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:22.877 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:22.877 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:23.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:23.879 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:23.900 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:23.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:23.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:24.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:24.901 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:24.922 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:24.922 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:24.922 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:25.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:25.923 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:25.944 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:25.944 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:25.944 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:26.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:40:26.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:40:26.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:26.945 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:26.965 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:26.965 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:26.965 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:27.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:27.966 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:27.987 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:27.987 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:27.987 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:28.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:28.989 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:29.010 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:29.010 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:29.010 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:30.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:30.011 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:30.032 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:30.032 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:30.032 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:31.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:31.033 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:31.054 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:31.054 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:31.054 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:32.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:32.055 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:32.077 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:32.077 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:32.077 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:33.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:33.079 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:33.099 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:33.100 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:33.100 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:34.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:34.101 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:34.121 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:34.121 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:34.121 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:35.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:35.122 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:35.142 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:35.142 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:35.142 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:36.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:36.144 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:36.164 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:36.165 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:36.165 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:37.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:37.166 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:37.187 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:37.187 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:37.187 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:38.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:38.188 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:38.209 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:38.209 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:38.209 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:39.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:39.210 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:39.231 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:39.232 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:39.232 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:40.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:40.233 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:40.254 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:40.254 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:40.254 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:41.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:41.255 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:41.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:41.276 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:41.276 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:42.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:42.276 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:42.297 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:42.298 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:42.298 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:43.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:43.299 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:43.320 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:43.320 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:43.320 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:44.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:44.321 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:44.342 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:44.342 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:44.342 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:45.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:45.343 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:45.362 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:45.363 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:45.363 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:46.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:46.364 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:46.385 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:46.385 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:46.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:47.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:47.386 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:47.406 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:47.407 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:47.407 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:48.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:48.408 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:48.428 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:48.429 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:48.429 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:49.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:49.429 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:49.451 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:49.451 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:49.451 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:50.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:50.451 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:50.472 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:50.473 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:50.473 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:51.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:51.474 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:51.495 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:51.495 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:51.495 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:52.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:52.496 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:52.517 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:52.517 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:52.517 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:53.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:53.519 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:53.539 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:53.539 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:53.539 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:54.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:54.540 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:54.560 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:54.560 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:54.560 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:55.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:55.562 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:55.583 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:55.583 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:55.583 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:56.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:56.584 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:56.605 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:56.605 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:56.605 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:56.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:40:56.802 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:40:57.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:57.606 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:57.627 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:57.627 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:57.627 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:58.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:58.628 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:58.649 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:58.649 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:58.649 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:59.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:59.650 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:59.671 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:59.671 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:59.672 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:41:00.243 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:00.246+0000 7f6d3285c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.244 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:00.246+0000 7f6d3285c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.244 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:00.246+0000 7f6d3285c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.244 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:00.246+0000 7f6d3285c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.244 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:00.246+0000 7f6d3285c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:00.673 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:41:00.693 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_41_00_UTC 2024-06-29T14:41:00.694 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_41_00_UTC'} 2024-06-29T14:41:00.694 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-14_41_00_UTC'} 2024-06-29T14:41:00.694 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:41:00.694 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-06-29-14_41_00_UTC'] 2024-06-29T14:41:00.694 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719672011.2483132, now = 1719672060.6945572, timo = 49 2024-06-29T14:41:01.695 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T14:41:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.832+0000 7f3876ac0640 1 -- 172.21.15.59:0/366138880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38701458a0 msgr2=0x7f38700cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.832+0000 7f3876ac0640 1 --2- 172.21.15.59:0/366138880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38701458a0 0x7f38700cf870 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f386c005fd0 tx=0x7f386c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.832+0000 7f3876ac0640 1 -- 172.21.15.59:0/366138880 shutdown_connections 2024-06-29T14:41:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.832+0000 7f3876ac0640 1 --2- 172.21.15.59:0/366138880 >> v2:172.21.15.153:3300/0 conn(0x7f38700b3740 0x7f38700b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.832+0000 7f3876ac0640 1 --2- 172.21.15.59:0/366138880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38701458a0 0x7f38700cf870 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.832+0000 7f3876ac0640 1 -- 172.21.15.59:0/366138880 >> 172.21.15.59:0/366138880 conn(0x7f387006c950 msgr2=0x7f3870073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.832+0000 7f3876ac0640 1 -- 172.21.15.59:0/366138880 shutdown_connections 2024-06-29T14:41:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.832+0000 7f3876ac0640 1 -- 172.21.15.59:0/366138880 wait complete. 2024-06-29T14:41:01.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.834+0000 7f3876ac0640 1 Processor -- start 2024-06-29T14:41:01.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.834+0000 7f3876ac0640 1 -- start start 2024-06-29T14:41:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3876ac0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38700b3740 0x7f38701f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3876ac0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387015c2f0 0x7f38701f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3876ac0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38700a9c90 0x7f38700aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3876ac0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f38701601a0 con 0x7f387015c2f0 2024-06-29T14:41:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3876ac0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3870160020 con 0x7f38700b3740 2024-06-29T14:41:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3876ac0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3870160320 con 0x7f38700a9c90 2024-06-29T14:41:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387015c2f0 0x7f38701f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387015c2f0 0x7f38701f2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34422/0 (socket says 172.21.15.59:34422) 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 -- 172.21.15.59:0/1840171601 learned_addr learned my addr 172.21.15.59:0/1840171601 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f387485c640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38700b3740 0x7f38701f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f387505d640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38700a9c90 0x7f38700aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 -- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38700a9c90 msgr2=0x7f38700aa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38700a9c90 0x7f38700aa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 -- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38700b3740 msgr2=0x7f38701f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38700b3740 0x7f38701f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 -- 172.21.15.59:0/1840171601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38701f3820 con 0x7f387015c2f0 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f3867fff640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387015c2f0 0x7f38701f2140 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f386000e6d0 tx=0x7f386000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.836+0000 7f38657fa640 1 -- 172.21.15.59:0/1840171601 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3860004220 con 0x7f387015c2f0 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.837+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38700aa5e0 con 0x7f387015c2f0 2024-06-29T14:41:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.837+0000 7f38657fa640 1 -- 172.21.15.59:0/1840171601 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38600043c0 con 0x7f387015c2f0 2024-06-29T14:41:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.837+0000 7f38657fa640 1 -- 172.21.15.59:0/1840171601 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386000cb10 con 0x7f387015c2f0 2024-06-29T14:41:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.837+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38701f4270 con 0x7f387015c2f0 2024-06-29T14:41:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.838+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38701458a0 con 0x7f387015c2f0 2024-06-29T14:41:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.838+0000 7f38657fa640 1 -- 172.21.15.59:0/1840171601 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 62) ==== 98377+0+0 (secure 0 0 0) 0x7f386000ccb0 con 0x7f387015c2f0 2024-06-29T14:41:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.839+0000 7f38657fa640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f385c076500 0x7f385c078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.839+0000 7f38657fa640 1 -- 172.21.15.59:0/1840171601 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7f3860098800 con 0x7f387015c2f0 2024-06-29T14:41:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.839+0000 7f387485c640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f385c076500 0x7f385c078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.840+0000 7f387485c640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f385c076500 0x7f385c078a10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f386c033000 tx=0x7f386c004730 comp rx=0 tx=0).ready entity=mgr.7175 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:01.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:01.843+0000 7f38657fa640 1 -- 172.21.15.59:0/1840171601 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f38700aa5e0 con 0x7f387015c2f0 2024-06-29T14:41:02.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.057+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 --> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f38700d26f0 con 0x7f385c076500 2024-06-29T14:41:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.135+0000 7f38657fa640 1 -- 172.21.15.59:0/1840171601 <== mgr.7175 v2:172.21.15.153:6832/3741042522 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f38700d26f0 con 0x7f385c076500 2024-06-29T14:41:02.133 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path snap_test_dir1 2024-06-29T14:41:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f385c076500 msgr2=0x7f385c078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f385c076500 0x7f385c078a10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f386c033000 tx=0x7f386c004730 comp rx=0 tx=0).stop 2024-06-29T14:41:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387015c2f0 msgr2=0x7f38701f2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387015c2f0 0x7f38701f2140 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f386000e6d0 tx=0x7f386000eb90 comp rx=0 tx=0).stop 2024-06-29T14:41:02.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 shutdown_connections 2024-06-29T14:41:02.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f385c076500 0x7f385c078a10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38700a9c90 0x7f38700aa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387015c2f0 0x7f38701f2140 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 --2- 172.21.15.59:0/1840171601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38700b3740 0x7f38701f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 >> 172.21.15.59:0/1840171601 conn(0x7f387006c950 msgr2=0x7f38700d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:02.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 shutdown_connections 2024-06-29T14:41:02.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.137+0000 7f3876ac0640 1 -- 172.21.15.59:0/1840171601 wait complete. 2024-06-29T14:41:02.146 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T14:41:02.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:02.146 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:41:02.163 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_41_00_UTC 2024-06-29T14:41:02.163 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_41_00_UTC 2024-06-29T14:41:02.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:02.163 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_41_00_UTC) 2024-06-29T14:41:02.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:02.224 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T14:41:02.283 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:41:02.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.444+0000 7f2faccd6640 1 -- 172.21.15.59:0/2486122957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 msgr2=0x7f2fa80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.444+0000 7f2faccd6640 1 --2- 172.21.15.59:0/2486122957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 0x7f2fa80b3990 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0005fd0 tx=0x7f2fa002a940 comp rx=0 tx=0).stop 2024-06-29T14:41:02.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.444+0000 7f2faccd6640 1 -- 172.21.15.59:0/2486122957 shutdown_connections 2024-06-29T14:41:02.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.444+0000 7f2faccd6640 1 --2- 172.21.15.59:0/2486122957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 0x7f2fa80b3990 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.444+0000 7f2faccd6640 1 -- 172.21.15.59:0/2486122957 >> 172.21.15.59:0/2486122957 conn(0x7f2fa80d3ee0 msgr2=0x7f2fa80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:02.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.444+0000 7f2faccd6640 1 -- 172.21.15.59:0/2486122957 shutdown_connections 2024-06-29T14:41:02.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.444+0000 7f2faccd6640 1 -- 172.21.15.59:0/2486122957 wait complete. 2024-06-29T14:41:02.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.446+0000 7f2faccd6640 1 Processor -- start 2024-06-29T14:41:02.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.446+0000 7f2faccd6640 1 -- start start 2024-06-29T14:41:02.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2faccd6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa80b3ed0 0x7f2fa814a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2faccd6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 0x7f2fa814aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2faccd6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fa815efe0 0x7f2fa814afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2faccd6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa8160450 con 0x7f2fa80d23c0 2024-06-29T14:41:02.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2faccd6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa81602d0 con 0x7f2fa80b3ed0 2024-06-29T14:41:02.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2faccd6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa81605d0 con 0x7f2fa815efe0 2024-06-29T14:41:02.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 0x7f2fa814aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 0x7f2fa814aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34452/0 (socket says 172.21.15.59:34452) 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 -- 172.21.15.59:0/500266090 learned_addr learned my addr 172.21.15.59:0/500266090 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa659c640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa80b3ed0 0x7f2fa814a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa6d9d640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fa815efe0 0x7f2fa814afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 -- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fa815efe0 msgr2=0x7f2fa814afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fa815efe0 0x7f2fa814afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 -- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa80b3ed0 msgr2=0x7f2fa814a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa80b3ed0 0x7f2fa814a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 -- 172.21.15.59:0/500266090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa814db90 con 0x7f2fa80d23c0 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2fa5d9b640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 0x7f2fa814aa70 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000c340 tx=0x7f2f9000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2f9effd640 1 -- 172.21.15.59:0/500266090 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90017070 con 0x7f2fa80d23c0 2024-06-29T14:41:02.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2f9effd640 1 -- 172.21.15.59:0/500266090 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f900035f0 con 0x7f2fa80d23c0 2024-06-29T14:41:02.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa8157ea0 con 0x7f2fa80d23c0 2024-06-29T14:41:02.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.448+0000 7f2f9effd640 1 -- 172.21.15.59:0/500266090 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90003a80 con 0x7f2fa80d23c0 2024-06-29T14:41:02.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.449+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa814e0e0 con 0x7f2fa80d23c0 2024-06-29T14:41:02.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.450+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa8161a30 con 0x7f2fa80d23c0 2024-06-29T14:41:02.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.450+0000 7f2f9effd640 1 -- 172.21.15.59:0/500266090 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 63) ==== 98434+0+0 (secure 0 0 0) 0x7f2f9001a070 con 0x7f2fa80d23c0 2024-06-29T14:41:02.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.451+0000 7f2f9effd640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f2f88076560 0x7f2f88078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.451+0000 7f2f9effd640 1 -- 172.21.15.59:0/500266090 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7f2f900970f0 con 0x7f2fa80d23c0 2024-06-29T14:41:02.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.451+0000 7f2fa659c640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f2f88076560 0x7f2f88078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.452+0000 7f2fa659c640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f2f88076560 0x7f2f88078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0034000 tx=0x7f2fa0004ab0 comp rx=0 tx=0).ready entity=mgr.7175 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:02.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.456+0000 7f2f9effd640 1 -- 172.21.15.59:0/500266090 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2fa80b42e0 con 0x7f2fa80d23c0 2024-06-29T14:41:02.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:02.702+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f2fa80b3990 con 0x7f2fa80d23c0 2024-06-29T14:41:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.301+0000 7f2f9effd640 1 -- 172.21.15.59:0/500266090 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v64) ==== 93+0+0 (secure 0 0 0) 0x7f2f9005fe50 con 0x7f2fa80d23c0 2024-06-29T14:41:03.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.306+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f2f88076560 msgr2=0x7f2f88078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.306+0000 7f2faccd6640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f2f88076560 0x7f2f88078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0034000 tx=0x7f2fa0004ab0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.306+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 msgr2=0x7f2fa814aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2f9effd640 1 -- 172.21.15.59:0/500266090 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f2f9005c0e0 con 0x7f2fa80d23c0 2024-06-29T14:41:03.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 0x7f2fa814aa70 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000c340 tx=0x7f2f9000c800 comp rx=0 tx=0).stop 2024-06-29T14:41:03.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 shutdown_connections 2024-06-29T14:41:03.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f2f88076560 0x7f2f88078a70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fa815efe0 0x7f2fa814afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fa80d23c0 0x7f2fa814aa70 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 --2- 172.21.15.59:0/500266090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fa80b3ed0 0x7f2fa814a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 >> 172.21.15.59:0/500266090 conn(0x7f2fa80d3ee0 msgr2=0x7f2fa806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:03.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 shutdown_connections 2024-06-29T14:41:03.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.307+0000 7f2faccd6640 1 -- 172.21.15.59:0/500266090 wait complete. 2024-06-29T14:41:03.320 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:41:03.320 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:03.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:03.402 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:41:03.402 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:03.403 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:03.403 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:03.419 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:03.419 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:03.476 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.637+0000 7efc83047640 1 -- 172.21.15.59:0/330601604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cd7c0 msgr2=0x7efc7c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.637+0000 7efc83047640 1 --2- 172.21.15.59:0/330601604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cd7c0 0x7efc7c06ca70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7efc74005fd0 tx=0x7efc7402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.637+0000 7efc83047640 1 -- 172.21.15.59:0/330601604 shutdown_connections 2024-06-29T14:41:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.637+0000 7efc83047640 1 --2- 172.21.15.59:0/330601604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cd7c0 0x7efc7c06ca70 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.637+0000 7efc83047640 1 --2- 172.21.15.59:0/330601604 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc7c0d02f0 0x7efc7c0cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.637+0000 7efc83047640 1 -- 172.21.15.59:0/330601604 >> 172.21.15.59:0/330601604 conn(0x7efc7c0b2ab0 msgr2=0x7efc7c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:03.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.637+0000 7efc83047640 1 -- 172.21.15.59:0/330601604 shutdown_connections 2024-06-29T14:41:03.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.637+0000 7efc83047640 1 -- 172.21.15.59:0/330601604 wait complete. 2024-06-29T14:41:03.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.639+0000 7efc83047640 1 Processor -- start 2024-06-29T14:41:03.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.639+0000 7efc83047640 1 -- start start 2024-06-29T14:41:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc83047640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7c0cd7c0 0x7efc7c15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc83047640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc7c1448e0 0x7efc7c0aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc83047640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cfbe0 0x7efc7c0cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc83047640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c06d4b0 con 0x7efc7c0cd7c0 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc83047640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c06d330 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc83047640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efc7c06d630 con 0x7efc7c1448e0 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc80de3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7c0cd7c0 0x7efc7c15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc73fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc7c1448e0 0x7efc7c0aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc815e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cfbe0 0x7efc7c0cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc73fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc7c1448e0 0x7efc7c0aaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:58944/0 (socket says 172.21.15.59:58944) 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc80de3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7c0cd7c0 0x7efc7c15a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53980/0 (socket says 172.21.15.59:53980) 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.640+0000 7efc73fff640 1 -- 172.21.15.59:0/2141909093 learned_addr learned my addr 172.21.15.59:0/2141909093 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc815e4640 1 -- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc7c1448e0 msgr2=0x7efc7c0aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc815e4640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc7c1448e0 0x7efc7c0aaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc815e4640 1 -- 172.21.15.59:0/2141909093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7c0cd7c0 msgr2=0x7efc7c15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc815e4640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7c0cd7c0 0x7efc7c15a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc815e4640 1 -- 172.21.15.59:0/2141909093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7c0d0530 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc73fff640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc7c1448e0 0x7efc7c0aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc815e4640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cfbe0 0x7efc7c0cfff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7efc74009000 tx=0x7efc740316d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc717fa640 1 -- 172.21.15.59:0/2141909093 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc74031990 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc717fa640 1 -- 172.21.15.59:0/2141909093 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc74031c20 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc7c15e0b0 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc717fa640 1 -- 172.21.15.59:0/2141909093 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc740330d0 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.641+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc7c15e670 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.642+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc7c06f750 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.643+0000 7efc717fa640 1 -- 172.21.15.59:0/2141909093 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7efc74033270 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.644+0000 7efc717fa640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7efc68076500 0x7efc68078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.644+0000 7efc80de3640 1 -- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7efc68076500 msgr2=0x7efc68078a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:03.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.644+0000 7efc717fa640 1 -- 172.21.15.59:0/2141909093 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7efc740b9850 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.644+0000 7efc80de3640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7efc68076500 0x7efc68078a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.647+0000 7efc717fa640 1 -- 172.21.15.59:0/2141909093 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efc7c06f750 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.842+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc7c06dae0 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.843+0000 7efc717fa640 1 -- 172.21.15.59:0/2141909093 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+17401 (secure 0 0 0) 0x7efc7c0d0530 con 0x7efc7c0cfbe0 2024-06-29T14:41:03.841 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:03.841 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":130,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:40:10.139466+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:56.243419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.825874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/473258225":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2246604721":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/1279390380":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2067735430":"2024-06-30T14:40:10.139401+0000","172.21.15.153:6833/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:6832/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2996335365":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/399775492":"2024-06-30T14:40:10.139401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc80de3640 1 -- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7efc68076500 msgr2=0x7efc68078a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc80de3640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7efc68076500 0x7efc68078a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7efc68076500 msgr2=0x7efc68078a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7efc68076500 0x7efc68078a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cfbe0 msgr2=0x7efc7c0cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cfbe0 0x7efc7c0cfff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7efc74009000 tx=0x7efc740316d0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 shutdown_connections 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7efc68076500 0x7efc68078a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc7c0cfbe0 0x7efc7c0cfff0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc7c1448e0 0x7efc7c0aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 --2- 172.21.15.59:0/2141909093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7c0cd7c0 0x7efc7c15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 >> 172.21.15.59:0/2141909093 conn(0x7efc7c0b2ab0 msgr2=0x7efc7c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:03.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 shutdown_connections 2024-06-29T14:41:03.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.845+0000 7efc83047640 1 -- 172.21.15.59:0/2141909093 wait complete. 2024-06-29T14:41:03.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:39:56.243419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:39:56.825874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:03.863 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.987+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/2886028007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a40b2fa0 msgr2=0x7fd6a40d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.987+0000 7fd6aa0e4640 1 --2- 172.21.15.59:0/2886028007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a40b2fa0 0x7fd6a40d1520 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd698009830 tx=0x7fd69802a5c0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.987+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/2886028007 shutdown_connections 2024-06-29T14:41:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.987+0000 7fd6aa0e4640 1 --2- 172.21.15.59:0/2886028007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a40b2fa0 0x7fd6a40d1520 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.987+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/2886028007 >> 172.21.15.59:0/2886028007 conn(0x7fd6a4073e10 msgr2=0x7fd6a4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.987+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/2886028007 shutdown_connections 2024-06-29T14:41:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.987+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/2886028007 wait complete. 2024-06-29T14:41:03.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.989+0000 7fd6aa0e4640 1 Processor -- start 2024-06-29T14:41:03.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.989+0000 7fd6aa0e4640 1 -- start start 2024-06-29T14:41:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6aa0e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a40b2270 0x7fd6a406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6aa0e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a40b2fa0 0x7fd6a414b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6aa0e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a40d1a60 0x7fd6a4158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6aa0e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a40b65d0 con 0x7fd6a40d1a60 2024-06-29T14:41:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6aa0e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a40b6450 con 0x7fd6a40b2270 2024-06-29T14:41:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6aa0e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a40b6750 con 0x7fd6a40b2fa0 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a40d1a60 0x7fd6a4158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a40d1a60 0x7fd6a4158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53982/0 (socket says 172.21.15.59:53982) 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 -- 172.21.15.59:0/3348985028 learned_addr learned my addr 172.21.15.59:0/3348985028 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a37fe640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a40b2270 0x7fd6a406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a2ffd640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a40b2fa0 0x7fd6a414b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 -- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a40b2fa0 msgr2=0x7fd6a414b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a40b2fa0 0x7fd6a414b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 -- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a40b2270 msgr2=0x7fd6a406ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a40b2270 0x7fd6a406ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 -- 172.21.15.59:0/3348985028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a40ada50 con 0x7fd6a40d1a60 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd6a3fff640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a40d1a60 0x7fd6a4158170 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c760 tx=0x7fd69c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.991+0000 7fd68bfff640 1 -- 172.21.15.59:0/3348985028 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c003b70 con 0x7fd6a40d1a60 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.992+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a40add30 con 0x7fd6a40d1a60 2024-06-29T14:41:03.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.992+0000 7fd68bfff640 1 -- 172.21.15.59:0/3348985028 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd69c003d10 con 0x7fd6a40d1a60 2024-06-29T14:41:03.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.992+0000 7fd68bfff640 1 -- 172.21.15.59:0/3348985028 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0052d0 con 0x7fd6a40d1a60 2024-06-29T14:41:03.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.992+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a40ae2f0 con 0x7fd6a40d1a60 2024-06-29T14:41:03.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.993+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a40b6c90 con 0x7fd6a40d1a60 2024-06-29T14:41:03.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.993+0000 7fd68bfff640 1 -- 172.21.15.59:0/3348985028 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7fd69c005470 con 0x7fd6a40d1a60 2024-06-29T14:41:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.994+0000 7fd68bfff640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7fd690076510 0x7fd690078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.994+0000 7fd6a37fe640 1 -- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7fd690076510 msgr2=0x7fd690078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.994+0000 7fd68bfff640 1 -- 172.21.15.59:0/3348985028 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7fd69c097c70 con 0x7fd6a40d1a60 2024-06-29T14:41:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.994+0000 7fd6a37fe640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7fd690076510 0x7fd690078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:03.998+0000 7fd68bfff640 1 -- 172.21.15.59:0/3348985028 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd69c060830 con 0x7fd6a40d1a60 2024-06-29T14:41:04.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.190+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd6a40d08d0 con 0x7fd6a40d1a60 2024-06-29T14:41:04.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.190+0000 7fd68bfff640 1 -- 172.21.15.59:0/3348985028 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) ==== 93+0+10 (secure 0 0 0) 0x7fd69c064b30 con 0x7fd6a40d1a60 2024-06-29T14:41:04.188 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:41:04.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7fd690076510 msgr2=0x7fd690078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7fd690076510 0x7fd690078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a40d1a60 msgr2=0x7fd6a4158170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a40d1a60 0x7fd6a4158170 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00c760 tx=0x7fd69c00cb60 comp rx=0 tx=0).stop 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 shutdown_connections 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7fd690076510 0x7fd690078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a40d1a60 0x7fd6a4158170 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a40b2fa0 0x7fd6a414b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 --2- 172.21.15.59:0/3348985028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6a40b2270 0x7fd6a406ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 >> 172.21.15.59:0/3348985028 conn(0x7fd6a4073e10 msgr2=0x7fd6a415bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 shutdown_connections 2024-06-29T14:41:04.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.192+0000 7fd6aa0e4640 1 -- 172.21.15.59:0/3348985028 wait complete. 2024-06-29T14:41:04.202 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:41:04.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.328+0000 7f4168168640 1 -- 172.21.15.59:0/2620337438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 msgr2=0x7f41600b5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.328+0000 7f4168168640 1 --2- 172.21.15.59:0/2620337438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 0x7f41600b5730 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f4160071a30 tx=0x7f415802a820 comp rx=0 tx=0).stop 2024-06-29T14:41:04.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.328+0000 7f4168168640 1 -- 172.21.15.59:0/2620337438 shutdown_connections 2024-06-29T14:41:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.328+0000 7f4168168640 1 --2- 172.21.15.59:0/2620337438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 0x7f41600b5730 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.328+0000 7f4168168640 1 --2- 172.21.15.59:0/2620337438 >> v2:172.21.15.153:3300/0 conn(0x7f416006eb40 0x7f41601443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.328+0000 7f4168168640 1 -- 172.21.15.59:0/2620337438 >> 172.21.15.59:0/2620337438 conn(0x7f4160073e50 msgr2=0x7f41600b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.328+0000 7f4168168640 1 -- 172.21.15.59:0/2620337438 shutdown_connections 2024-06-29T14:41:04.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.328+0000 7f4168168640 1 -- 172.21.15.59:0/2620337438 wait complete. 2024-06-29T14:41:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.330+0000 7f4168168640 1 Processor -- start 2024-06-29T14:41:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.330+0000 7f4168168640 1 -- start start 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4168168640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f416006eb40 0x7f41601f90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4168168640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4160071620 0x7f41600afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4168168640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 0x7f4160150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4168168640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f41600cf470 con 0x7f41600b5320 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4168168640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f41600cf2f0 con 0x7f4160071620 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4168168640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f41600cf5f0 con 0x7f416006eb40 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4166705640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 0x7f4160150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4165703640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4160071620 0x7f41600afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4165703640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4160071620 0x7f41600afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:39192/0 (socket says 172.21.15.59:39192) 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4165703640 1 -- 172.21.15.59:0/3887934641 learned_addr learned my addr 172.21.15.59:0/3887934641 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4166705640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 0x7f4160150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54008/0 (socket says 172.21.15.59:54008) 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.331+0000 7f4165f04640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f416006eb40 0x7f41601f90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4166705640 1 -- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f416006eb40 msgr2=0x7f41601f90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4166705640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f416006eb40 0x7f41601f90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4166705640 1 -- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4160071620 msgr2=0x7f41600afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4166705640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4160071620 0x7f41600afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4166705640 1 -- 172.21.15.59:0/3887934641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41601513a0 con 0x7f41600b5320 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4165703640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4160071620 0x7f41600afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4166705640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 0x7f4160150890 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f4158009000 tx=0x7f4158032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f41567fc640 1 -- 172.21.15.59:0/3887934641 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158037040 con 0x7f41600b5320 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f41567fc640 1 -- 172.21.15.59:0/3887934641 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4158042070 con 0x7f41600b5320 2024-06-29T14:41:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4160151680 con 0x7f41600b5320 2024-06-29T14:41:04.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f41567fc640 1 -- 172.21.15.59:0/3887934641 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580042a0 con 0x7f41600b5320 2024-06-29T14:41:04.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.332+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41601f8ba0 con 0x7f41600b5320 2024-06-29T14:41:04.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.333+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4160151110 con 0x7f41600b5320 2024-06-29T14:41:04.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.334+0000 7f41567fc640 1 -- 172.21.15.59:0/3887934641 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f4158004440 con 0x7f41600b5320 2024-06-29T14:41:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.334+0000 7f41567fc640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f4144076510 0x7f4144078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.335+0000 7f41567fc640 1 -- 172.21.15.59:0/3887934641 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7f41580b8ad0 con 0x7f41600b5320 2024-06-29T14:41:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.335+0000 7f4165f04640 1 -- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f4144076510 msgr2=0x7f4144078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:04.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.335+0000 7f4165f04640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f4144076510 0x7f4144078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:04.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.339+0000 7f41567fc640 1 -- 172.21.15.59:0/3887934641 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f41600b5c70 con 0x7f41600b5320 2024-06-29T14:41:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.531+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f41601f86b0 con 0x7f41600b5320 2024-06-29T14:41:04.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.532+0000 7f41567fc640 1 -- 172.21.15.59:0/3887934641 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) ==== 104+0+11 (secure 0 0 0) 0x7f4158081690 con 0x7f41600b5320 2024-06-29T14:41:04.530 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:41:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f4144076510 msgr2=0x7f4144078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f4144076510 0x7f4144078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 msgr2=0x7f4160150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 0x7f4160150890 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f4158009000 tx=0x7f4158032c60 comp rx=0 tx=0).stop 2024-06-29T14:41:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 shutdown_connections 2024-06-29T14:41:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f4144076510 0x7f4144078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41600b5320 0x7f4160150890 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4160071620 0x7f41600afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 --2- 172.21.15.59:0/3887934641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f416006eb40 0x7f41601f90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 >> 172.21.15.59:0/3887934641 conn(0x7f4160073e50 msgr2=0x7f416015b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 shutdown_connections 2024-06-29T14:41:04.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.534+0000 7f4168168640 1 -- 172.21.15.59:0/3887934641 wait complete. 2024-06-29T14:41:04.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:41:04.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.666+0000 7ff76e5e0640 1 -- 172.21.15.59:0/2826189354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 msgr2=0x7ff7680d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.666+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/2826189354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 0x7ff7680d1520 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7ff768074c50 tx=0x7ff75c02a530 comp rx=0 tx=0).stop 2024-06-29T14:41:04.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.666+0000 7ff76e5e0640 1 -- 172.21.15.59:0/2826189354 shutdown_connections 2024-06-29T14:41:04.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.666+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/2826189354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 0x7ff7680d1520 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.666+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/2826189354 >> v2:172.21.15.153:3300/0 conn(0x7ff7680b2270 0x7ff76815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.667+0000 7ff76e5e0640 1 -- 172.21.15.59:0/2826189354 >> 172.21.15.59:0/2826189354 conn(0x7ff768073e10 msgr2=0x7ff768074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.667+0000 7ff76e5e0640 1 -- 172.21.15.59:0/2826189354 shutdown_connections 2024-06-29T14:41:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.667+0000 7ff76e5e0640 1 -- 172.21.15.59:0/2826189354 wait complete. 2024-06-29T14:41:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.668+0000 7ff76e5e0640 1 Processor -- start 2024-06-29T14:41:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.669+0000 7ff76e5e0640 1 -- start start 2024-06-29T14:41:04.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff76e5e0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7680b2270 0x7ff76814d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff76e5e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 0x7ff7680b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff76e5e0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7680d1a60 0x7ff7680ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff76e5e0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff7680b6cc0 con 0x7ff7680b2fa0 2024-06-29T14:41:04.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff76e5e0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7680b6b40 con 0x7ff7680b2270 2024-06-29T14:41:04.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff76e5e0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff7680cfe90 con 0x7ff7680d1a60 2024-06-29T14:41:04.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff7677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 0x7ff7680b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff7677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 0x7ff7680b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54028/0 (socket says 172.21.15.59:54028) 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff7677fe640 1 -- 172.21.15.59:0/4084797753 learned_addr learned my addr 172.21.15.59:0/4084797753 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff767fff640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7680b2270 0x7ff76814d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff76cb7d640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7680d1a60 0x7ff7680ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff767fff640 1 -- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7680d1a60 msgr2=0x7ff7680ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff767fff640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7680d1a60 0x7ff7680ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff767fff640 1 -- 172.21.15.59:0/4084797753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 msgr2=0x7ff7680b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff767fff640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 0x7ff7680b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff767fff640 1 -- 172.21.15.59:0/4084797753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7680adb90 con 0x7ff7680b2270 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.670+0000 7ff7677fe640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 0x7ff7680b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.671+0000 7ff767fff640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7680b2270 0x7ff76814d460 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff75800e6d0 tx=0x7ff75800eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.671+0000 7ff764ff9640 1 -- 172.21.15.59:0/4084797753 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75800cab0 con 0x7ff7680b2270 2024-06-29T14:41:04.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.671+0000 7ff764ff9640 1 -- 172.21.15.59:0/4084797753 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff75800cd40 con 0x7ff7680b2270 2024-06-29T14:41:04.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.671+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7680ade70 con 0x7ff7680b2270 2024-06-29T14:41:04.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.671+0000 7ff764ff9640 1 -- 172.21.15.59:0/4084797753 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758010690 con 0x7ff7680b2270 2024-06-29T14:41:04.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.671+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff76806f150 con 0x7ff7680b2270 2024-06-29T14:41:04.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.672+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7681575f0 con 0x7ff7680b2270 2024-06-29T14:41:04.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.673+0000 7ff764ff9640 1 -- 172.21.15.59:0/4084797753 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7ff758010830 con 0x7ff7680b2270 2024-06-29T14:41:04.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.673+0000 7ff764ff9640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff754076510 0x7ff754078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.673+0000 7ff7677fe640 1 -- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff754076510 msgr2=0x7ff754078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:04.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.673+0000 7ff764ff9640 1 -- 172.21.15.59:0/4084797753 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7ff758098820 con 0x7ff7680b2270 2024-06-29T14:41:04.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.673+0000 7ff7677fe640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff754076510 0x7ff754078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:04.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.678+0000 7ff764ff9640 1 -- 172.21.15.59:0/4084797753 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff7681575f0 con 0x7ff7680b2270 2024-06-29T14:41:04.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.869+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff7680d03b0 con 0x7ff7680b2270 2024-06-29T14:41:04.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.869+0000 7ff764ff9640 1 -- 172.21.15.59:0/4084797753 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) ==== 100+0+11 (secure 0 0 0) 0x7ff758061340 con 0x7ff7680b2270 2024-06-29T14:41:04.867 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.871+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff754076510 msgr2=0x7ff754078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.871+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff754076510 0x7ff754078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.871+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7680b2270 msgr2=0x7ff76814d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.871+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7680b2270 0x7ff76814d460 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff75800e6d0 tx=0x7ff75800eb90 comp rx=0 tx=0).stop 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.871+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 shutdown_connections 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.871+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff754076510 0x7ff754078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.871+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7680d1a60 0x7ff7680ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.872+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7680b2fa0 0x7ff7680b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.872+0000 7ff76e5e0640 1 --2- 172.21.15.59:0/4084797753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7680b2270 0x7ff76814d460 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.872+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 >> 172.21.15.59:0/4084797753 conn(0x7ff768073e10 msgr2=0x7ff768070a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.872+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 shutdown_connections 2024-06-29T14:41:04.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:04.872+0000 7ff76e5e0640 1 -- 172.21.15.59:0/4084797753 wait complete. 2024-06-29T14:41:04.882 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:41:05.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.003+0000 7f26dac88640 1 -- 172.21.15.59:0/4011056685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40b3ed0 msgr2=0x7f26d415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.003+0000 7f26dac88640 1 --2- 172.21.15.59:0/4011056685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40b3ed0 0x7f26d415e8d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f26c4005fd0 tx=0x7f26c402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.003+0000 7f26dac88640 1 -- 172.21.15.59:0/4011056685 shutdown_connections 2024-06-29T14:41:05.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.003+0000 7f26dac88640 1 --2- 172.21.15.59:0/4011056685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40b3ed0 0x7f26d415e8d0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.003+0000 7f26dac88640 1 --2- 172.21.15.59:0/4011056685 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26d40d23c0 0x7f26d40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.003+0000 7f26dac88640 1 -- 172.21.15.59:0/4011056685 >> 172.21.15.59:0/4011056685 conn(0x7f26d40d3ee0 msgr2=0x7f26d40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.003+0000 7f26dac88640 1 -- 172.21.15.59:0/4011056685 shutdown_connections 2024-06-29T14:41:05.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.003+0000 7f26dac88640 1 -- 172.21.15.59:0/4011056685 wait complete. 2024-06-29T14:41:05.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.005+0000 7f26dac88640 1 Processor -- start 2024-06-29T14:41:05.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.005+0000 7f26dac88640 1 -- start start 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26dac88640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d40b3ed0 0x7f26d41f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26dac88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40d23c0 0x7f26d40c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26dac88640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26d415ee10 0x7f26d414db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26dac88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f26d4160240 con 0x7f26d40d23c0 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26dac88640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26d41600c0 con 0x7f26d40b3ed0 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26dac88640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f26d41603c0 con 0x7f26d415ee10 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40d23c0 0x7f26d40c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40d23c0 0x7f26d40c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54042/0 (socket says 172.21.15.59:54042) 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.006+0000 7f26d3fff640 1 -- 172.21.15.59:0/667059624 learned_addr learned my addr 172.21.15.59:0/667059624 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d8a24640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d40b3ed0 0x7f26d41f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d9225640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26d415ee10 0x7f26d414db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d3fff640 1 -- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26d415ee10 msgr2=0x7f26d414db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d3fff640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26d415ee10 0x7f26d414db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d3fff640 1 -- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d40b3ed0 msgr2=0x7f26d41f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d3fff640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d40b3ed0 0x7f26d41f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d3fff640 1 -- 172.21.15.59:0/667059624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d415e410 con 0x7f26d40d23c0 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d9225640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26d415ee10 0x7f26d414db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d3fff640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40d23c0 0x7f26d40c8e70 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f26c4034040 tx=0x7f26c40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d17fa640 1 -- 172.21.15.59:0/667059624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c403e030 con 0x7f26d40d23c0 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d17fa640 1 -- 172.21.15.59:0/667059624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26c4003ca0 con 0x7f26d40d23c0 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d415db70 con 0x7f26d40d23c0 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26d17fa640 1 -- 172.21.15.59:0/667059624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c4035040 con 0x7f26d40d23c0 2024-06-29T14:41:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.007+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d415e130 con 0x7f26d40d23c0 2024-06-29T14:41:05.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.009+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26d415e8d0 con 0x7f26d40d23c0 2024-06-29T14:41:05.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.009+0000 7f26d17fa640 1 -- 172.21.15.59:0/667059624 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f26c4041030 con 0x7f26d40d23c0 2024-06-29T14:41:05.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.010+0000 7f26d17fa640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f26b4076540 0x7f26b4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.010+0000 7f26d8a24640 1 -- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f26b4076540 msgr2=0x7f26b4078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:05.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.010+0000 7f26d17fa640 1 -- 172.21.15.59:0/667059624 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7f26c40b8440 con 0x7f26d40d23c0 2024-06-29T14:41:05.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.010+0000 7f26d8a24640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f26b4076540 0x7f26b4078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:05.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.014+0000 7f26d17fa640 1 -- 172.21.15.59:0/667059624 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f26c4080fc0 con 0x7f26d40d23c0 2024-06-29T14:41:05.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.211+0000 7f26d8a24640 1 -- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f26b4076540 msgr2=0x7f26b4078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:05.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.211+0000 7f26d8a24640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f26b4076540 0x7f26b4078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:05.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.219+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f26d40d2950 con 0x7f26d40d23c0 2024-06-29T14:41:05.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.219+0000 7f26d17fa640 1 -- 172.21.15.59:0/667059624 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v130) ==== 80+0+474 (secure 0 0 0) 0x7f26c40852c0 con 0x7f26d40d23c0 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/473258225 2024-06-30T14:40:10.139401+0000 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2246604721 2024-06-30T14:40:10.139401+0000 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1279390380 2024-06-30T14:40:10.139401+0000 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2067735430 2024-06-30T14:40:10.139401+0000 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/892574025 2024-06-30T14:40:10.139401+0000 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/892574025 2024-06-30T14:40:10.139401+0000 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2996335365 2024-06-30T14:40:10.139401+0000 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/399775492 2024-06-30T14:40:10.139401+0000 2024-06-29T14:41:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f26b4076540 msgr2=0x7f26b4078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f26b4076540 0x7f26b4078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40d23c0 msgr2=0x7f26d40c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40d23c0 0x7f26d40c8e70 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f26c4034040 tx=0x7f26c40038a0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 shutdown_connections 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f26b4076540 0x7f26b4078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26d415ee10 0x7f26d414db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26d40d23c0 0x7f26d40c8e70 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 --2- 172.21.15.59:0/667059624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26d40b3ed0 0x7f26d41f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 >> 172.21.15.59:0/667059624 conn(0x7f26d40d3ee0 msgr2=0x7f26d406cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 shutdown_connections 2024-06-29T14:41:05.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.222+0000 7f26dac88640 1 -- 172.21.15.59:0/667059624 wait complete. 2024-06-29T14:41:05.232 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:41:05.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:05.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:05.263 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:41:05.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:05.263 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:41:05.321 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:05.321 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:41:05.321 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:05.337 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:05.337 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:05.337 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:05.337 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:05.377 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:05.377 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:05.377 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:41:05.377 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:05.393 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:05.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:05.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 -- 172.21.15.59:0/2902737394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83480b3ed0 msgr2=0x7f834815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 --2- 172.21.15.59:0/2902737394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83480b3ed0 0x7f834815e8d0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f83480d04f0 tx=0x7f833802a560 comp rx=0 tx=0).stop 2024-06-29T14:41:05.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 -- 172.21.15.59:0/2902737394 shutdown_connections 2024-06-29T14:41:05.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 --2- 172.21.15.59:0/2902737394 >> v2:172.21.15.153:3300/0 conn(0x7f834815ee10 0x7f834815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 --2- 172.21.15.59:0/2902737394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83480b3ed0 0x7f834815e8d0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 --2- 172.21.15.59:0/2902737394 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83480d23c0 0x7f83480b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 -- 172.21.15.59:0/2902737394 >> 172.21.15.59:0/2902737394 conn(0x7f83480d3ee0 msgr2=0x7f83480d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 -- 172.21.15.59:0/2902737394 shutdown_connections 2024-06-29T14:41:05.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.538+0000 7f834d311640 1 -- 172.21.15.59:0/2902737394 wait complete. 2024-06-29T14:41:05.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.540+0000 7f834d311640 1 Processor -- start 2024-06-29T14:41:05.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.540+0000 7f834d311640 1 -- start start 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f834d311640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83480b3ed0 0x7f83480cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f834d311640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83480d23c0 0x7f834815b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f834d311640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f834815ee10 0x7f8348160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f834d311640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8348160070 con 0x7f834815ee10 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f834d311640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f834815fef0 con 0x7f83480d23c0 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f834d311640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f83481601f0 con 0x7f83480b3ed0 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f83477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f834815ee10 0x7f8348160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f83477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f834815ee10 0x7f8348160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54064/0 (socket says 172.21.15.59:54064) 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f83477fe640 1 -- 172.21.15.59:0/563642734 learned_addr learned my addr 172.21.15.59:0/563642734 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.541+0000 7f83467fc640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83480d23c0 0x7f834815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f8346ffd640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83480b3ed0 0x7f83480cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f83477fe640 1 -- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83480b3ed0 msgr2=0x7f83480cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f83477fe640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83480b3ed0 0x7f83480cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f83477fe640 1 -- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83480d23c0 msgr2=0x7f834815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f83477fe640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83480d23c0 0x7f834815b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f83477fe640 1 -- 172.21.15.59:0/563642734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834815e530 con 0x7f834815ee10 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f83477fe640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f834815ee10 0x7f8348160730 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f834000eab0 tx=0x7f834000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f833f7fe640 1 -- 172.21.15.59:0/563642734 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834000cea0 con 0x7f834815ee10 2024-06-29T14:41:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f833f7fe640 1 -- 172.21.15.59:0/563642734 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8340004b20 con 0x7f834815ee10 2024-06-29T14:41:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8348158fa0 con 0x7f834815ee10 2024-06-29T14:41:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f833f7fe640 1 -- 172.21.15.59:0/563642734 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340005020 con 0x7f834815ee10 2024-06-29T14:41:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.542+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f834815df70 con 0x7f834815ee10 2024-06-29T14:41:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.544+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83480b3990 con 0x7f834815ee10 2024-06-29T14:41:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.544+0000 7f833f7fe640 1 -- 172.21.15.59:0/563642734 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f83400051c0 con 0x7f834815ee10 2024-06-29T14:41:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.545+0000 7f833f7fe640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f8328076510 0x7f8328078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.545+0000 7f833f7fe640 1 -- 172.21.15.59:0/563642734 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7f8340098ac0 con 0x7f834815ee10 2024-06-29T14:41:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.545+0000 7f8346ffd640 1 -- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f8328076510 msgr2=0x7f8328078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.545+0000 7f8346ffd640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f8328076510 0x7f8328078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:05.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.549+0000 7f833f7fe640 1 -- 172.21.15.59:0/563642734 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8340061680 con 0x7f834815ee10 2024-06-29T14:41:05.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.746+0000 7f8346ffd640 1 -- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f8328076510 msgr2=0x7f8328078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:05.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.746+0000 7f8346ffd640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f8328076510 0x7f8328078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:05.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.779+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83480d2950 con 0x7f834815ee10 2024-06-29T14:41:05.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.780+0000 7f833f7fe640 1 -- 172.21.15.59:0/563642734 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5228 (secure 0 0 0) 0x7f8340065980 con 0x7f834815ee10 2024-06-29T14:41:05.778 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:05.778 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":63,"btime":"2024-06-29T14:40:01:506455+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2473404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2473404084},{"type":"v1","addr":"172.21.15.153:6836","nonce":2473404084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7831,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3605263971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3605263971},{"type":"v1","addr":"172.21.15.153:6837","nonce":3605263971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/842001425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":842001425},{"type":"v1","addr":"172.21.15.59:6833","nonce":842001425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.885754+0000","modified":"2024-06-29T14:40:00.901266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3950841523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3950841523},{"type":"v1","addr":"172.21.15.59:6835","nonce":3950841523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8361,"qdb_cluster":[8361]},"id":7}]} 2024-06-29T14:41:05.778 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 63 2024-06-29T14:41:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f8328076510 msgr2=0x7f8328078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f8328076510 0x7f8328078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f834815ee10 msgr2=0x7f8348160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f834815ee10 0x7f8348160730 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f834000eab0 tx=0x7f834000ef70 comp rx=0 tx=0).stop 2024-06-29T14:41:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 shutdown_connections 2024-06-29T14:41:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f8328076510 0x7f8328078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f834815ee10 0x7f8348160730 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83480d23c0 0x7f834815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 --2- 172.21.15.59:0/563642734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83480b3ed0 0x7f83480cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 >> 172.21.15.59:0/563642734 conn(0x7f83480d3ee0 msgr2=0x7f834806cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 shutdown_connections 2024-06-29T14:41:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.782+0000 7f834d311640 1 -- 172.21.15.59:0/563642734 wait complete. 2024-06-29T14:41:05.795 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:05.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.917+0000 7f9701810640 1 -- 172.21.15.59:0/984669514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b3a20 msgr2=0x7f96fc15ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.917+0000 7f9701810640 1 --2- 172.21.15.59:0/984669514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b3a20 0x7f96fc15ac50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009830 tx=0x7f96ec02a510 comp rx=0 tx=0).stop 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.917+0000 7f9701810640 1 -- 172.21.15.59:0/984669514 shutdown_connections 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.917+0000 7f9701810640 1 --2- 172.21.15.59:0/984669514 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96fc0cf080 0x7f96fc06d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.917+0000 7f9701810640 1 --2- 172.21.15.59:0/984669514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b3a20 0x7f96fc15ac50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.917+0000 7f9701810640 1 -- 172.21.15.59:0/984669514 >> 172.21.15.59:0/984669514 conn(0x7f96fc06f8d0 msgr2=0x7f96fc06fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.917+0000 7f9701810640 1 -- 172.21.15.59:0/984669514 shutdown_connections 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.917+0000 7f9701810640 1 -- 172.21.15.59:0/984669514 wait complete. 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.919+0000 7f9701810640 1 Processor -- start 2024-06-29T14:41:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.919+0000 7f9701810640 1 -- start start 2024-06-29T14:41:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f9701810640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96fc0b3a20 0x7f96fc0af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f9701810640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b5020 0x7f96fc0726d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f9701810640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96fc072c10 0x7f96fc156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f9701810640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc06dba0 con 0x7f96fc0b3a20 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f9701810640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc06da20 con 0x7f96fc0b5020 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f9701810640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f96fc06dd20 con 0x7f96fc072c10 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f96faffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96fc0b3a20 0x7f96fc0af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f96faffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96fc0b3a20 0x7f96fc0af150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54066/0 (socket says 172.21.15.59:54066) 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f96faffd640 1 -- 172.21.15.59:0/3920324944 learned_addr learned my addr 172.21.15.59:0/3920324944 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f96fa7fc640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b5020 0x7f96fc0726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.920+0000 7f96fb7fe640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96fc072c10 0x7f96fc156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96faffd640 1 -- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96fc072c10 msgr2=0x7f96fc156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96faffd640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96fc072c10 0x7f96fc156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96faffd640 1 -- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b5020 msgr2=0x7f96fc0726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96faffd640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b5020 0x7f96fc0726d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96faffd640 1 -- 172.21.15.59:0/3920324944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96fc1457f0 con 0x7f96fc0b3a20 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96fa7fc640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b5020 0x7f96fc0726d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96faffd640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96fc0b3a20 0x7f96fc0af150 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f96ec000c00 tx=0x7f96ec033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96e3fff640 1 -- 172.21.15.59:0/3920324944 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec005c90 con 0x7f96fc0b3a20 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96e3fff640 1 -- 172.21.15.59:0/3920324944 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96ec031e50 con 0x7f96fc0b3a20 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96fc157b80 con 0x7f96fc0b3a20 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.921+0000 7f96e3fff640 1 -- 172.21.15.59:0/3920324944 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec0332f0 con 0x7f96fc0b3a20 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.922+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96fc0ae170 con 0x7f96fc0b3a20 2024-06-29T14:41:05.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.923+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96fc06e1d0 con 0x7f96fc0b3a20 2024-06-29T14:41:05.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.924+0000 7f96e3fff640 1 -- 172.21.15.59:0/3920324944 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f96ec033490 con 0x7f96fc0b3a20 2024-06-29T14:41:05.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.924+0000 7f96e3fff640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f96e8076540 0x7f96e8078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.925+0000 7f96e3fff640 1 -- 172.21.15.59:0/3920324944 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7f96ec0b8dd0 con 0x7f96fc0b3a20 2024-06-29T14:41:05.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.925+0000 7f96fa7fc640 1 -- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f96e8076540 msgr2=0x7f96e8078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:05.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.925+0000 7f96fa7fc640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f96e8076540 0x7f96e8078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:05.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:05.929+0000 7f96e3fff640 1 -- 172.21.15.59:0/3920324944 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f96ec04a030 con 0x7f96fc0b3a20 2024-06-29T14:41:06.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.121+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96fc0b1860 con 0x7f96fc0b3a20 2024-06-29T14:41:06.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.122+0000 7f96e3fff640 1 -- 172.21.15.59:0/3920324944 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+17401 (secure 0 0 0) 0x7f96ec03f030 con 0x7f96fc0b3a20 2024-06-29T14:41:06.120 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:06.121 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":130,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:40:10.139466+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:56.243419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.825874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/473258225":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2246604721":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/1279390380":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2067735430":"2024-06-30T14:40:10.139401+0000","172.21.15.153:6833/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:6832/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2996335365":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/399775492":"2024-06-30T14:40:10.139401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:06.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f96e8076540 msgr2=0x7f96e8078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f96e8076540 0x7f96e8078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96fc0b3a20 msgr2=0x7f96fc0af150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96fc0b3a20 0x7f96fc0af150 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f96ec000c00 tx=0x7f96ec033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 shutdown_connections 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f96e8076540 0x7f96e8078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96fc072c10 0x7f96fc156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96fc0b5020 0x7f96fc0726d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 --2- 172.21.15.59:0/3920324944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96fc0b3a20 0x7f96fc0af150 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 >> 172.21.15.59:0/3920324944 conn(0x7f96fc06f8d0 msgr2=0x7f96fc073bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 shutdown_connections 2024-06-29T14:41:06.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.124+0000 7f9701810640 1 -- 172.21.15.59:0/3920324944 wait complete. 2024-06-29T14:41:06.135 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:39:56.243419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:39:56.825874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:06.135 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:06.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.260+0000 7f965fd44640 1 -- 172.21.15.59:0/1529194273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96580b40a0 msgr2=0x7f965815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.260+0000 7f965fd44640 1 --2- 172.21.15.59:0/1529194273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96580b40a0 0x7f965815e8d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9648009830 tx=0x7f964802a500 comp rx=0 tx=0).stop 2024-06-29T14:41:06.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.260+0000 7f965fd44640 1 -- 172.21.15.59:0/1529194273 shutdown_connections 2024-06-29T14:41:06.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.260+0000 7f965fd44640 1 --2- 172.21.15.59:0/1529194273 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f965815ee10 0x7f965815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.260+0000 7f965fd44640 1 --2- 172.21.15.59:0/1529194273 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96580b40a0 0x7f965815e8d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.260+0000 7f965fd44640 1 -- 172.21.15.59:0/1529194273 >> 172.21.15.59:0/1529194273 conn(0x7f96580d3ee0 msgr2=0x7f96580d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.260+0000 7f965fd44640 1 -- 172.21.15.59:0/1529194273 shutdown_connections 2024-06-29T14:41:06.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.260+0000 7f965fd44640 1 -- 172.21.15.59:0/1529194273 wait complete. 2024-06-29T14:41:06.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.262+0000 7f965fd44640 1 Processor -- start 2024-06-29T14:41:06.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.262+0000 7f965fd44640 1 -- start start 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965fd44640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96580d23c0 0x7f965814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965fd44640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965815ee10 0x7f96580c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965fd44640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96580c7b30 0x7f965806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965fd44640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9658160280 con 0x7f96580d23c0 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965fd44640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9658160100 con 0x7f965815ee10 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965fd44640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9658160400 con 0x7f96580c7b30 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965dae0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96580d23c0 0x7f965814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965dae0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96580d23c0 0x7f965814ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54078/0 (socket says 172.21.15.59:54078) 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965dae0640 1 -- 172.21.15.59:0/1129781308 learned_addr learned my addr 172.21.15.59:0/1129781308 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965d2df640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965815ee10 0x7f96580c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.263+0000 7f965e2e1640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96580c7b30 0x7f965806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f965dae0640 1 -- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96580c7b30 msgr2=0x7f965806c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f965dae0640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96580c7b30 0x7f965806c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f965dae0640 1 -- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965815ee10 msgr2=0x7f96580c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f965dae0640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965815ee10 0x7f96580c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f965dae0640 1 -- 172.21.15.59:0/1129781308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96580c81b0 con 0x7f96580d23c0 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f965dae0640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96580d23c0 0x7f965814ae20 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f965400c400 tx=0x7f965400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f964e7fc640 1 -- 172.21.15.59:0/1129781308 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654011790 con 0x7f96580d23c0 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f964e7fc640 1 -- 172.21.15.59:0/1129781308 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9654017070 con 0x7f96580d23c0 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96580c8490 con 0x7f96580d23c0 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f964e7fc640 1 -- 172.21.15.59:0/1129781308 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654011cc0 con 0x7f96580d23c0 2024-06-29T14:41:06.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.264+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96580c8a50 con 0x7f96580d23c0 2024-06-29T14:41:06.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.266+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f965815e8d0 con 0x7f96580d23c0 2024-06-29T14:41:06.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.266+0000 7f964e7fc640 1 -- 172.21.15.59:0/1129781308 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f965401c460 con 0x7f96580d23c0 2024-06-29T14:41:06.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.267+0000 7f964e7fc640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f963c076510 0x7f963c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.267+0000 7f964e7fc640 1 -- 172.21.15.59:0/1129781308 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7f9654097890 con 0x7f96580d23c0 2024-06-29T14:41:06.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.267+0000 7f965d2df640 1 -- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f963c076510 msgr2=0x7f963c078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:06.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.267+0000 7f965d2df640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f963c076510 0x7f963c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:06.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.271+0000 7f964e7fc640 1 -- 172.21.15.59:0/1129781308 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9654021020 con 0x7f96580d23c0 2024-06-29T14:41:06.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.464+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f96580d2950 con 0x7f96580d23c0 2024-06-29T14:41:06.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.464+0000 7f964e7fc640 1 -- 172.21.15.59:0/1129781308 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) ==== 93+0+10 (secure 0 0 0) 0x7f9654060ca0 con 0x7f96580d23c0 2024-06-29T14:41:06.463 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:41:06.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f963c076510 msgr2=0x7f963c078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f963c076510 0x7f963c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96580d23c0 msgr2=0x7f965814ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96580d23c0 0x7f965814ae20 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f965400c400 tx=0x7f965400c800 comp rx=0 tx=0).stop 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 shutdown_connections 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7f963c076510 0x7f963c078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96580c7b30 0x7f965806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f965815ee10 0x7f96580c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 --2- 172.21.15.59:0/1129781308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96580d23c0 0x7f965814ae20 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 >> 172.21.15.59:0/1129781308 conn(0x7f96580d3ee0 msgr2=0x7f965806dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 shutdown_connections 2024-06-29T14:41:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.466+0000 7f965fd44640 1 -- 172.21.15.59:0/1129781308 wait complete. 2024-06-29T14:41:06.476 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:41:06.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.601+0000 7ff31d77f640 1 -- 172.21.15.59:0/1807300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 msgr2=0x7ff31815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.601+0000 7ff31d77f640 1 --2- 172.21.15.59:0/1807300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 0x7ff31815f9b0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7ff3180d15d0 tx=0x7ff31002aac0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.601+0000 7ff31d77f640 1 -- 172.21.15.59:0/1807300 shutdown_connections 2024-06-29T14:41:06.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.601+0000 7ff31d77f640 1 --2- 172.21.15.59:0/1807300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 0x7ff31815f9b0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.601+0000 7ff31d77f640 1 --2- 172.21.15.59:0/1807300 >> v2:172.21.15.153:3300/0 conn(0x7ff3180d23c0 0x7ff3180b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.601+0000 7ff31d77f640 1 -- 172.21.15.59:0/1807300 >> 172.21.15.59:0/1807300 conn(0x7ff3180d3ee0 msgr2=0x7ff3180d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.601+0000 7ff31d77f640 1 -- 172.21.15.59:0/1807300 shutdown_connections 2024-06-29T14:41:06.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.601+0000 7ff31d77f640 1 -- 172.21.15.59:0/1807300 wait complete. 2024-06-29T14:41:06.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.603+0000 7ff31d77f640 1 Processor -- start 2024-06-29T14:41:06.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.603+0000 7ff31d77f640 1 -- start start 2024-06-29T14:41:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff31d77f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3180b3ed0 0x7ff318155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff31d77f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3180d23c0 0x7ff3180d0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff31d77f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 0x7ff31815a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff31d77f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff318160280 con 0x7ff31815efe0 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff31d77f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff318160100 con 0x7ff3180d23c0 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff31d77f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff318160400 con 0x7ff3180b3ed0 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff3177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 0x7ff31815a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff30ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3180d23c0 0x7ff3180d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff316ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3180b3ed0 0x7ff318155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff3177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 0x7ff31815a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54092/0 (socket says 172.21.15.59:54092) 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff3177fe640 1 -- 172.21.15.59:0/4220172925 learned_addr learned my addr 172.21.15.59:0/4220172925 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.604+0000 7ff30ffff640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3180d23c0 0x7ff3180d0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:39314/0 (socket says 172.21.15.59:39314) 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff3177fe640 1 -- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3180b3ed0 msgr2=0x7ff318155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff3177fe640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3180b3ed0 0x7ff318155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff3177fe640 1 -- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3180d23c0 msgr2=0x7ff3180d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff3177fe640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3180d23c0 0x7ff3180d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff3177fe640 1 -- 172.21.15.59:0/4220172925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff318156530 con 0x7ff31815efe0 2024-06-29T14:41:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff3177fe640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 0x7ff31815a740 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7ff310009d60 tx=0x7ff310003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:06.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff30f7fe640 1 -- 172.21.15.59:0/4220172925 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff310044070 con 0x7ff31815efe0 2024-06-29T14:41:06.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff30f7fe640 1 -- 172.21.15.59:0/4220172925 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff310003e00 con 0x7ff31815efe0 2024-06-29T14:41:06.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff30f7fe640 1 -- 172.21.15.59:0/4220172925 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3100044b0 con 0x7ff31815efe0 2024-06-29T14:41:06.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff318159200 con 0x7ff31815efe0 2024-06-29T14:41:06.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.605+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3181597c0 con 0x7ff31815efe0 2024-06-29T14:41:06.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.606+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3180b3990 con 0x7ff31815efe0 2024-06-29T14:41:06.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.607+0000 7ff30f7fe640 1 -- 172.21.15.59:0/4220172925 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7ff310004650 con 0x7ff31815efe0 2024-06-29T14:41:06.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.608+0000 7ff30f7fe640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff2f8076510 0x7ff2f8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.608+0000 7ff316ffd640 1 -- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff2f8076510 msgr2=0x7ff2f8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3741042522 2024-06-29T14:41:06.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.608+0000 7ff30f7fe640 1 -- 172.21.15.59:0/4220172925 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6445+0+0 (secure 0 0 0) 0x7ff3100b86c0 con 0x7ff31815efe0 2024-06-29T14:41:06.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.608+0000 7ff316ffd640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff2f8076510 0x7ff2f8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.612+0000 7ff30f7fe640 1 -- 172.21.15.59:0/4220172925 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff318156530 con 0x7ff31815efe0 2024-06-29T14:41:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.802+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff3180d2f00 con 0x7ff31815efe0 2024-06-29T14:41:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.802+0000 7ff30f7fe640 1 -- 172.21.15.59:0/4220172925 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) ==== 104+0+11 (secure 0 0 0) 0x7ff310032030 con 0x7ff31815efe0 2024-06-29T14:41:06.801 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff2f8076510 msgr2=0x7ff2f8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff2f8076510 0x7ff2f8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 msgr2=0x7ff31815a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 0x7ff31815a740 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7ff310009d60 tx=0x7ff310003c10 comp rx=0 tx=0).stop 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 shutdown_connections 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:6832/3741042522,v1:172.21.15.153:6833/3741042522] conn(0x7ff2f8076510 0x7ff2f8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff31815efe0 0x7ff31815a740 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3180d23c0 0x7ff3180d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 --2- 172.21.15.59:0/4220172925 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3180b3ed0 0x7ff318155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 >> 172.21.15.59:0/4220172925 conn(0x7ff3180d3ee0 msgr2=0x7ff3180b8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 shutdown_connections 2024-06-29T14:41:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.805+0000 7ff31d77f640 1 -- 172.21.15.59:0/4220172925 wait complete. 2024-06-29T14:41:06.814 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:41:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.938+0000 7f1350078640 1 -- 172.21.15.59:0/633738658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13480b3ed0 msgr2=0x7f134815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.938+0000 7f1350078640 1 --2- 172.21.15.59:0/633738658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13480b3ed0 0x7f134815e8d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1338005fd0 tx=0x7f133802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.938+0000 7f1350078640 1 -- 172.21.15.59:0/633738658 shutdown_connections 2024-06-29T14:41:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.938+0000 7f1350078640 1 --2- 172.21.15.59:0/633738658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13480b3ed0 0x7f134815e8d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.938+0000 7f1350078640 1 --2- 172.21.15.59:0/633738658 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13480d23c0 0x7f13480b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.938+0000 7f1350078640 1 -- 172.21.15.59:0/633738658 >> 172.21.15.59:0/633738658 conn(0x7f13480d3ee0 msgr2=0x7f13480d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.938+0000 7f1350078640 1 -- 172.21.15.59:0/633738658 shutdown_connections 2024-06-29T14:41:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.938+0000 7f1350078640 1 -- 172.21.15.59:0/633738658 wait complete. 2024-06-29T14:41:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.941+0000 7f1350078640 1 Processor -- start 2024-06-29T14:41:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.941+0000 7f1350078640 1 -- start start 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.944+0000 7f1350078640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13480b3ed0 0x7f13481f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.944+0000 7f1350078640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13480d23c0 0x7f13480c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.944+0000 7f1350078640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f134815ee10 0x7f134814db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.944+0000 7f1350078640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1348160240 con 0x7f13480d23c0 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.944+0000 7f1350078640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13481600c0 con 0x7f134815ee10 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.944+0000 7f1350078640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f13481603c0 con 0x7f13480b3ed0 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.945+0000 7f134d613640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13480d23c0 0x7f13480c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.945+0000 7f134d613640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13480d23c0 0x7f13480c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54104/0 (socket says 172.21.15.59:54104) 2024-06-29T14:41:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.945+0000 7f134d613640 1 -- 172.21.15.59:0/2865999149 learned_addr learned my addr 172.21.15.59:0/2865999149 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.946+0000 7f134de14640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13480b3ed0 0x7f13481f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.946+0000 7f134e615640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f134815ee10 0x7f134814db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.946+0000 7f134d613640 1 -- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13480b3ed0 msgr2=0x7f13481f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.946+0000 7f134d613640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13480b3ed0 0x7f13481f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.946+0000 7f134d613640 1 -- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f134815ee10 msgr2=0x7f134814db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.946+0000 7f134d613640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f134815ee10 0x7f134814db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.946+0000 7f134d613640 1 -- 172.21.15.59:0/2865999149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134815e410 con 0x7f13480d23c0 2024-06-29T14:41:06.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.946+0000 7f134e615640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f134815ee10 0x7f134814db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:06.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.947+0000 7f134d613640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13480d23c0 0x7f13480c8e70 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1338034040 tx=0x7f13380035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.950+0000 7f133e7fc640 1 -- 172.21.15.59:0/2865999149 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133803e030 con 0x7f13480d23c0 2024-06-29T14:41:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.950+0000 7f133e7fc640 1 -- 172.21.15.59:0/2865999149 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1338003ae0 con 0x7f13480d23c0 2024-06-29T14:41:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.950+0000 7f133e7fc640 1 -- 172.21.15.59:0/2865999149 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338035040 con 0x7f13480d23c0 2024-06-29T14:41:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.950+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134815db70 con 0x7f13480d23c0 2024-06-29T14:41:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.950+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134815e130 con 0x7f13480d23c0 2024-06-29T14:41:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.951+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134815e8d0 con 0x7f13480d23c0 2024-06-29T14:41:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.952+0000 7f133e7fc640 1 -- 172.21.15.59:0/2865999149 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 65) ==== 97912+0+0 (secure 0 0 0) 0x7f1338041030 con 0x7f13480d23c0 2024-06-29T14:41:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.952+0000 7f133e7fc640 1 -- 172.21.15.59:0/2865999149 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 6875+0+0 (secure 0 0 0) 0x7f13380b7ee0 con 0x7f13480d23c0 2024-06-29T14:41:06.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:06.957+0000 7f133e7fc640 1 -- 172.21.15.59:0/2865999149 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1338081770 con 0x7f13480d23c0 2024-06-29T14:41:06.996 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.998+0000 7f92b8522640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.996 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.998+0000 7f92b8522640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.996 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.998+0000 7f92b8522640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.996 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.998+0000 7f92b8522640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.996 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.998+0000 7f92b8522640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.997 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.999+0000 7f92b551c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.997 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.999+0000 7f92b551c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.997 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.999+0000 7f92b551c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.997 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.999+0000 7f92b551c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:06.997 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:06.999+0000 7f92b551c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.152+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f13480d2950 con 0x7f13480d23c0 2024-06-29T14:41:07.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.152+0000 7f133e7fc640 1 -- 172.21.15.59:0/2865999149 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) ==== 100+0+11 (secure 0 0 0) 0x7f1338081070 con 0x7f13480d23c0 2024-06-29T14:41:07.150 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:41:07.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.154+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13480d23c0 msgr2=0x7f13480c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.154+0000 7f1350078640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13480d23c0 0x7f13480c8e70 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f1338034040 tx=0x7f13380035f0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.154+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 shutdown_connections 2024-06-29T14:41:07.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.155+0000 7f1350078640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f134815ee10 0x7f134814db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.155+0000 7f1350078640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13480d23c0 0x7f13480c8e70 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.155+0000 7f1350078640 1 --2- 172.21.15.59:0/2865999149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13480b3ed0 0x7f13481f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.155+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 >> 172.21.15.59:0/2865999149 conn(0x7f13480d3ee0 msgr2=0x7f134806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.155+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 shutdown_connections 2024-06-29T14:41:07.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.155+0000 7f1350078640 1 -- 172.21.15.59:0/2865999149 wait complete. 2024-06-29T14:41:07.165 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:07.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.287+0000 7f0b06636640 1 -- 172.21.15.59:0/1832482609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b000d1c30 msgr2=0x7f0b000b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.287+0000 7f0b06636640 1 --2- 172.21.15.59:0/1832482609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b000d1c30 0x7f0b000b5b30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0af8005fd0 tx=0x7f0af802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.287+0000 7f0b06636640 1 -- 172.21.15.59:0/1832482609 shutdown_connections 2024-06-29T14:41:07.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.287+0000 7f0b06636640 1 --2- 172.21.15.59:0/1832482609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b000d1c30 0x7f0b000b5b30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.287+0000 7f0b06636640 1 --2- 172.21.15.59:0/1832482609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b000b2270 0x7f0b0015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.287+0000 7f0b06636640 1 -- 172.21.15.59:0/1832482609 >> 172.21.15.59:0/1832482609 conn(0x7f0b00073e10 msgr2=0x7f0b00074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.287+0000 7f0b06636640 1 -- 172.21.15.59:0/1832482609 shutdown_connections 2024-06-29T14:41:07.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.287+0000 7f0b06636640 1 -- 172.21.15.59:0/1832482609 wait complete. 2024-06-29T14:41:07.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.289+0000 7f0b06636640 1 Processor -- start 2024-06-29T14:41:07.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.289+0000 7f0b06636640 1 -- start start 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.290+0000 7f0b06636640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b000b2270 0x7f0b0015f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.290+0000 7f0b06636640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b000b2fa0 0x7f0b00159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.290+0000 7f0b06636640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b000d1c30 0x7f0b0013c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.290+0000 7f0b06636640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0b000b6400 con 0x7f0b000d1c30 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.290+0000 7f0b06636640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b000b6280 con 0x7f0b000b2fa0 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.290+0000 7f0b06636640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0b000b6580 con 0x7f0b000b2270 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.290+0000 7f0b04bd3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b000d1c30 0x7f0b0013c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.290+0000 7f0b04bd3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b000d1c30 0x7f0b0013c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54114/0 (socket says 172.21.15.59:54114) 2024-06-29T14:41:07.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0b04bd3640 1 -- 172.21.15.59:0/3986073812 learned_addr learned my addr 172.21.15.59:0/3986073812 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0afffff640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b000b2270 0x7f0b0015f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0aff7fe640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b000b2fa0 0x7f0b00159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0b04bd3640 1 -- 172.21.15.59:0/3986073812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b000b2270 msgr2=0x7f0b0015f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0b04bd3640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b000b2270 0x7f0b0015f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0b04bd3640 1 -- 172.21.15.59:0/3986073812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b000b2fa0 msgr2=0x7f0b00159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0b04bd3640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b000b2fa0 0x7f0b00159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0b04bd3640 1 -- 172.21.15.59:0/3986073812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0013c770 con 0x7f0b000d1c30 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0b04bd3640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b000d1c30 0x7f0b0013c230 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009cd0 tx=0x7f0af80316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0afcff9640 1 -- 172.21.15.59:0/3986073812 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8031da0 con 0x7f0b000d1c30 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0afcff9640 1 -- 172.21.15.59:0/3986073812 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0af8003a60 con 0x7f0b000d1c30 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0afcff9640 1 -- 172.21.15.59:0/3986073812 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af80330d0 con 0x7f0b000d1c30 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.291+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b0015a3e0 con 0x7f0b000d1c30 2024-06-29T14:41:07.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.292+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b0015a9a0 con 0x7f0b000d1c30 2024-06-29T14:41:07.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.292+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b0015f6c0 con 0x7f0b000d1c30 2024-06-29T14:41:07.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.293+0000 7f0afcff9640 1 -- 172.21.15.59:0/3986073812 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 65) ==== 97912+0+0 (secure 0 0 0) 0x7f0af8033270 con 0x7f0b000d1c30 2024-06-29T14:41:07.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.294+0000 7f0afcff9640 1 -- 172.21.15.59:0/3986073812 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 6875+0+0 (secure 0 0 0) 0x7f0af80b8ab0 con 0x7f0b000d1c30 2024-06-29T14:41:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.298+0000 7f0afcff9640 1 -- 172.21.15.59:0/3986073812 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b0015f6c0 con 0x7f0b000d1c30 2024-06-29T14:41:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.527+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b000ced70 con 0x7f0b000d1c30 2024-06-29T14:41:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.527+0000 7f0afcff9640 1 -- 172.21.15.59:0/3986073812 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5228 (secure 0 0 0) 0x7f0af803f030 con 0x7f0b000d1c30 2024-06-29T14:41:07.526 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:07.526 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":63,"btime":"2024-06-29T14:40:01:506455+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2473404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2473404084},{"type":"v1","addr":"172.21.15.153:6836","nonce":2473404084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7831,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3605263971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3605263971},{"type":"v1","addr":"172.21.15.153:6837","nonce":3605263971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/842001425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":842001425},{"type":"v1","addr":"172.21.15.59:6833","nonce":842001425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.885754+0000","modified":"2024-06-29T14:40:00.901266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3950841523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3950841523},{"type":"v1","addr":"172.21.15.59:6835","nonce":3950841523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8361,"qdb_cluster":[8361]},"id":7}]} 2024-06-29T14:41:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 63 2024-06-29T14:41:07.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b000d1c30 msgr2=0x7f0b0013c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b000d1c30 0x7f0b0013c230 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009cd0 tx=0x7f0af80316f0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 shutdown_connections 2024-06-29T14:41:07.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b000d1c30 0x7f0b0013c230 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b000b2fa0 0x7f0b00159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 --2- 172.21.15.59:0/3986073812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b000b2270 0x7f0b0015f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 >> 172.21.15.59:0/3986073812 conn(0x7f0b00073e10 msgr2=0x7f0b00074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 shutdown_connections 2024-06-29T14:41:07.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.530+0000 7f0b06636640 1 -- 172.21.15.59:0/3986073812 wait complete. 2024-06-29T14:41:07.539 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.659+0000 7f82ab081640 1 -- 172.21.15.59:0/2838714525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2270 msgr2=0x7f82a415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.659+0000 7f82ab081640 1 --2- 172.21.15.59:0/2838714525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2270 0x7f82a415bc20 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8298009830 tx=0x7f829802a510 comp rx=0 tx=0).stop 2024-06-29T14:41:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.659+0000 7f82ab081640 1 -- 172.21.15.59:0/2838714525 shutdown_connections 2024-06-29T14:41:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.659+0000 7f82ab081640 1 --2- 172.21.15.59:0/2838714525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a40d1c30 0x7f82a40b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.659+0000 7f82ab081640 1 --2- 172.21.15.59:0/2838714525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2270 0x7f82a415bc20 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.659+0000 7f82ab081640 1 -- 172.21.15.59:0/2838714525 >> 172.21.15.59:0/2838714525 conn(0x7f82a4073e10 msgr2=0x7f82a4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.659+0000 7f82ab081640 1 -- 172.21.15.59:0/2838714525 shutdown_connections 2024-06-29T14:41:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.659+0000 7f82ab081640 1 -- 172.21.15.59:0/2838714525 wait complete. 2024-06-29T14:41:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.660+0000 7f82ab081640 1 Processor -- start 2024-06-29T14:41:07.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.660+0000 7f82ab081640 1 -- start start 2024-06-29T14:41:07.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82ab081640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a40b2270 0x7f82a4144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82ab081640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2fa0 0x7f82a41474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82ab081640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a40d1c30 0x7f82a4147a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82ab081640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f82a40b6400 con 0x7f82a40b2fa0 2024-06-29T14:41:07.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82ab081640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82a40b6280 con 0x7f82a40b2270 2024-06-29T14:41:07.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82ab081640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f82a40b6580 con 0x7f82a40d1c30 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2fa0 0x7f82a41474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2fa0 0x7f82a41474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54122/0 (socket says 172.21.15.59:54122) 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 -- 172.21.15.59:0/3755334243 learned_addr learned my addr 172.21.15.59:0/3755334243 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a8e1d640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a40b2270 0x7f82a4144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a961e640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a40d1c30 0x7f82a4147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 -- 172.21.15.59:0/3755334243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a40d1c30 msgr2=0x7f82a4147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a40d1c30 0x7f82a4147a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 -- 172.21.15.59:0/3755334243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a40b2270 msgr2=0x7f82a4144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a40b2270 0x7f82a4144bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 -- 172.21.15.59:0/3755334243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a4145110 con 0x7f82a40b2fa0 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.662+0000 7f82a3fff640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2fa0 0x7f82a41474d0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f829000c3e0 tx=0x7f829000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.663+0000 7f82a17fa640 1 -- 172.21.15.59:0/3755334243 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290011790 con 0x7f82a40b2fa0 2024-06-29T14:41:07.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.663+0000 7f82a17fa640 1 -- 172.21.15.59:0/3755334243 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8290017070 con 0x7f82a40b2fa0 2024-06-29T14:41:07.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.663+0000 7f82a17fa640 1 -- 172.21.15.59:0/3755334243 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290011ca0 con 0x7f82a40b2fa0 2024-06-29T14:41:07.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.663+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a4145600 con 0x7f82a40b2fa0 2024-06-29T14:41:07.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.663+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a4145bc0 con 0x7f82a40b2fa0 2024-06-29T14:41:07.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.664+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a40b6aa0 con 0x7f82a40b2fa0 2024-06-29T14:41:07.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.664+0000 7f82a17fa640 1 -- 172.21.15.59:0/3755334243 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 65) ==== 97912+0+0 (secure 0 0 0) 0x7f829001c460 con 0x7f82a40b2fa0 2024-06-29T14:41:07.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.665+0000 7f82a17fa640 1 -- 172.21.15.59:0/3755334243 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 6875+0+0 (secure 0 0 0) 0x7f82900973a0 con 0x7f82a40b2fa0 2024-06-29T14:41:07.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.669+0000 7f82a17fa640 1 -- 172.21.15.59:0/3755334243 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8290060730 con 0x7f82a40b2fa0 2024-06-29T14:41:07.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.864+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82a4157c30 con 0x7f82a40b2fa0 2024-06-29T14:41:07.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.866+0000 7f82a17fa640 1 -- 172.21.15.59:0/3755334243 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) ==== 74+0+18034 (secure 0 0 0) 0x7f8290064a30 con 0x7f82a40b2fa0 2024-06-29T14:41:07.864 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:07.864 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":131,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:06.937863+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:56.243419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.825874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3471373028":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/13104752":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2821906341":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6833/3741042522":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6832/3741042522":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/3120890145":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/473258225":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/700951802":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/3413132296":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2246604721":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/1279390380":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2067735430":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/1732960037":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6833/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:6832/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/3736955455":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2996335365":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/399775492":"2024-06-30T14:40:10.139401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2fa0 msgr2=0x7f82a41474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2fa0 0x7f82a41474d0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f829000c3e0 tx=0x7f829000c8a0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 shutdown_connections 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a40d1c30 0x7f82a4147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a40b2fa0 0x7f82a41474d0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 --2- 172.21.15.59:0/3755334243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a40b2270 0x7f82a4144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 >> 172.21.15.59:0/3755334243 conn(0x7f82a4073e10 msgr2=0x7f82a4070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 shutdown_connections 2024-06-29T14:41:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:07.868+0000 7f82ab081640 1 -- 172.21.15.59:0/3755334243 wait complete. 2024-06-29T14:41:07.878 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:41:07.878 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:08.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.003+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/4257952903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc80ce950 msgr2=0x7f5dc81622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.003+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/4257952903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc80ce950 0x7f5dc81622c0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc005fd0 tx=0x7f5dbc02a530 comp rx=0 tx=0).stop 2024-06-29T14:41:08.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.003+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/4257952903 shutdown_connections 2024-06-29T14:41:08.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.003+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/4257952903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc8162800 0x7f5dc8162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.003+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/4257952903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc80ce950 0x7f5dc81622c0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.003+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/4257952903 >> 172.21.15.59:0/4257952903 conn(0x7f5dc80b5610 msgr2=0x7f5dc80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.003+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/4257952903 shutdown_connections 2024-06-29T14:41:08.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.003+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/4257952903 wait complete. 2024-06-29T14:41:08.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.004+0000 7f5dcdbe4640 1 Processor -- start 2024-06-29T14:41:08.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.005+0000 7f5dcdbe4640 1 -- start start 2024-06-29T14:41:08.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dcdbe4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc808c190 0x7f5dc80cc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dcdbe4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc80ce950 0x7f5dc80cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dcdbe4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc8162800 0x7f5dc81f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dcdbe4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc80b7f60 con 0x7f5dc808c190 2024-06-29T14:41:08.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dcdbe4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc80b7de0 con 0x7f5dc8162800 2024-06-29T14:41:08.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dcdbe4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5dc80b80e0 con 0x7f5dc80ce950 2024-06-29T14:41:08.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dc77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc808c190 0x7f5dc80cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dc77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc808c190 0x7f5dc80cc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54144/0 (socket says 172.21.15.59:54144) 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dc77fe640 1 -- 172.21.15.59:0/1827365824 learned_addr learned my addr 172.21.15.59:0/1827365824 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dc6ffd640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc80ce950 0x7f5dc80cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.006+0000 7f5dc7fff640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc8162800 0x7f5dc81f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5dc77fe640 1 -- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc80ce950 msgr2=0x7f5dc80cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5dc77fe640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc80ce950 0x7f5dc80cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5dc77fe640 1 -- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc8162800 msgr2=0x7f5dc81f48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5dc77fe640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc8162800 0x7f5dc81f48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5dc77fe640 1 -- 172.21.15.59:0/1827365824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc80cc9d0 con 0x7f5dc808c190 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5dc77fe640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc808c190 0x7f5dc80cc210 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5db800b3f0 tx=0x7f5db800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5daffff640 1 -- 172.21.15.59:0/1827365824 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8015020 con 0x7f5dc808c190 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5daffff640 1 -- 172.21.15.59:0/1827365824 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5db8011920 con 0x7f5dc808c190 2024-06-29T14:41:08.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc80cccb0 con 0x7f5dc808c190 2024-06-29T14:41:08.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5daffff640 1 -- 172.21.15.59:0/1827365824 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db8011d70 con 0x7f5dc808c190 2024-06-29T14:41:08.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.007+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc80cd270 con 0x7f5dc808c190 2024-06-29T14:41:08.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.008+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc81622c0 con 0x7f5dc808c190 2024-06-29T14:41:08.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.009+0000 7f5daffff640 1 -- 172.21.15.59:0/1827365824 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 66) ==== 98039+0+0 (secure 0 0 0) 0x7f5db8018020 con 0x7f5dc808c190 2024-06-29T14:41:08.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.009+0000 7f5daffff640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5db4076120 0x7f5db4078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.009+0000 7f5daffff640 1 -- 172.21.15.59:0/1827365824 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 6931+0+0 (secure 0 0 0) 0x7f5db8097430 con 0x7f5dc808c190 2024-06-29T14:41:08.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.010+0000 7f5dc6ffd640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5db4076120 0x7f5db4078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.010+0000 7f5dc6ffd640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5db4076120 0x7f5db4078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc033000 tx=0x7f5dbc004730 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.014+0000 7f5daffff640 1 -- 172.21.15.59:0/1827365824 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5db801f030 con 0x7f5dc808c190 2024-06-29T14:41:08.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.249+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dc80b5ac0 con 0x7f5dc808c190 2024-06-29T14:41:08.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.249+0000 7f5daffff640 1 -- 172.21.15.59:0/1827365824 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5228 (secure 0 0 0) 0x7f5db8060520 con 0x7f5dc808c190 2024-06-29T14:41:08.248 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:08.248 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":63,"btime":"2024-06-29T14:40:01:506455+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2473404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2473404084},{"type":"v1","addr":"172.21.15.153:6836","nonce":2473404084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7831,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3605263971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3605263971},{"type":"v1","addr":"172.21.15.153:6837","nonce":3605263971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/842001425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":842001425},{"type":"v1","addr":"172.21.15.59:6833","nonce":842001425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.885754+0000","modified":"2024-06-29T14:40:00.901266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3950841523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3950841523},{"type":"v1","addr":"172.21.15.59:6835","nonce":3950841523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8361,"qdb_cluster":[8361]},"id":7}]} 2024-06-29T14:41:08.248 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 63 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5db4076120 msgr2=0x7f5db4078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5db4076120 0x7f5db4078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc033000 tx=0x7f5dbc004730 comp rx=0 tx=0).stop 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc808c190 msgr2=0x7f5dc80cc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc808c190 0x7f5dc80cc210 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f5db800b3f0 tx=0x7f5db800b8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 shutdown_connections 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc8162800 0x7f5dc81f48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5db4076120 0x7f5db4078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc80ce950 0x7f5dc80cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 --2- 172.21.15.59:0/1827365824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc808c190 0x7f5dc80cc210 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.251+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 >> 172.21.15.59:0/1827365824 conn(0x7f5dc80b5610 msgr2=0x7f5dc80cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.252+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 shutdown_connections 2024-06-29T14:41:08.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.252+0000 7f5dcdbe4640 1 -- 172.21.15.59:0/1827365824 wait complete. 2024-06-29T14:41:08.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:08.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.381+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3875172399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 msgr2=0x7ffa1815eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.381+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3875172399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 0x7ffa1815eed0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7ffa180b06b0 tx=0x7ffa0802a820 comp rx=0 tx=0).stop 2024-06-29T14:41:08.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.381+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3875172399 shutdown_connections 2024-06-29T14:41:08.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.381+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3875172399 >> v2:172.21.15.153:3300/0 conn(0x7ffa18144350 0x7ffa18144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.381+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3875172399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 0x7ffa1815eed0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.381+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3875172399 >> 172.21.15.59:0/3875172399 conn(0x7ffa180b06e0 msgr2=0x7ffa180b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.381+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3875172399 shutdown_connections 2024-06-29T14:41:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.381+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3875172399 wait complete. 2024-06-29T14:41:08.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.383+0000 7ffa1fb52640 1 Processor -- start 2024-06-29T14:41:08.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.383+0000 7ffa1fb52640 1 -- start start 2024-06-29T14:41:08.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1fb52640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa180b1b00 0x7ffa180ac570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1fb52640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa18144350 0x7ffa180c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1fb52640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 0x7ffa18131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1fb52640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ffa18160e50 con 0x7ffa1815f5a0 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1fb52640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa18160cd0 con 0x7ffa180b1b00 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1fb52640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffa18160fd0 con 0x7ffa18144350 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1e0ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 0x7ffa18131290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1e0ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 0x7ffa18131290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54164/0 (socket says 172.21.15.59:54164) 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1e0ef640 1 -- 172.21.15.59:0/3953257325 learned_addr learned my addr 172.21.15.59:0/3953257325 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.384+0000 7ffa1d8ee640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa180b1b00 0x7ffa180ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1d0ed640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa18144350 0x7ffa180c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1e0ef640 1 -- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa18144350 msgr2=0x7ffa180c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1e0ef640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa18144350 0x7ffa180c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1e0ef640 1 -- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa180b1b00 msgr2=0x7ffa180ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1e0ef640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa180b1b00 0x7ffa180ac570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1e0ef640 1 -- 172.21.15.59:0/3953257325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1812f590 con 0x7ffa1815f5a0 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1e0ef640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 0x7ffa18131290 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000c390 tx=0x7ffa1000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa167fc640 1 -- 172.21.15.59:0/3953257325 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10011790 con 0x7ffa1815f5a0 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa167fc640 1 -- 172.21.15.59:0/3953257325 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa10017070 con 0x7ffa1815f5a0 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1812f870 con 0x7ffa1815f5a0 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa167fc640 1 -- 172.21.15.59:0/3953257325 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10011c40 con 0x7ffa1815f5a0 2024-06-29T14:41:08.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.385+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1812fe30 con 0x7ffa1815f5a0 2024-06-29T14:41:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.386+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa181614f0 con 0x7ffa1815f5a0 2024-06-29T14:41:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.387+0000 7ffa167fc640 1 -- 172.21.15.59:0/3953257325 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 66) ==== 98039+0+0 (secure 0 0 0) 0x7ffa1001c460 con 0x7ffa1815f5a0 2024-06-29T14:41:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.387+0000 7ffa167fc640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ffa04076150 0x7ffa04078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.387+0000 7ffa167fc640 1 -- 172.21.15.59:0/3953257325 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 6931+0+0 (secure 0 0 0) 0x7ffa10097780 con 0x7ffa1815f5a0 2024-06-29T14:41:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.388+0000 7ffa1d8ee640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ffa04076150 0x7ffa04078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.388+0000 7ffa1d8ee640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ffa04076150 0x7ffa04078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffa08034000 tx=0x7ffa08004ab0 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.392+0000 7ffa167fc640 1 -- 172.21.15.59:0/3953257325 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa1815f9b0 con 0x7ffa1815f5a0 2024-06-29T14:41:08.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.622+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa18162600 con 0x7ffa1815f5a0 2024-06-29T14:41:08.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.623+0000 7ffa167fc640 1 -- 172.21.15.59:0/3953257325 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5228 (secure 0 0 0) 0x7ffa10060d70 con 0x7ffa1815f5a0 2024-06-29T14:41:08.621 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:08.621 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":63,"btime":"2024-06-29T14:40:01:506455+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2473404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2473404084},{"type":"v1","addr":"172.21.15.153:6836","nonce":2473404084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7831,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3605263971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3605263971},{"type":"v1","addr":"172.21.15.153:6837","nonce":3605263971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/842001425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":842001425},{"type":"v1","addr":"172.21.15.59:6833","nonce":842001425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.885754+0000","modified":"2024-06-29T14:40:00.901266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8361},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8361":{"gid":8361,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3950841523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3950841523},{"type":"v1","addr":"172.21.15.59:6835","nonce":3950841523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8361,"qdb_cluster":[8361]},"id":7}]} 2024-06-29T14:41:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 63 2024-06-29T14:41:08.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ffa04076150 msgr2=0x7ffa04078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ffa04076150 0x7ffa04078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffa08034000 tx=0x7ffa08004ab0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 msgr2=0x7ffa18131290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 0x7ffa18131290 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000c390 tx=0x7ffa1000c850 comp rx=0 tx=0).stop 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 shutdown_connections 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa1815f5a0 0x7ffa18131290 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ffa04076150 0x7ffa04078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa18144350 0x7ffa180c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 --2- 172.21.15.59:0/3953257325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa180b1b00 0x7ffa180ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 >> 172.21.15.59:0/3953257325 conn(0x7ffa180b06e0 msgr2=0x7ffa181457d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 shutdown_connections 2024-06-29T14:41:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.625+0000 7ffa1fb52640 1 -- 172.21.15.59:0/3953257325 wait complete. 2024-06-29T14:41:08.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:08.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.754+0000 7fad2999a640 1 -- 172.21.15.59:0/1029458281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad2415ceb0 msgr2=0x7fad240b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.754+0000 7fad2999a640 1 --2- 172.21.15.59:0/1029458281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad2415ceb0 0x7fad240b56d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fad14005fd0 tx=0x7fad1402a4a0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.754+0000 7fad2999a640 1 -- 172.21.15.59:0/1029458281 shutdown_connections 2024-06-29T14:41:08.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.754+0000 7fad2999a640 1 --2- 172.21.15.59:0/1029458281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2413e9b0 0x7fad2413edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.754+0000 7fad2999a640 1 --2- 172.21.15.59:0/1029458281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad2415ceb0 0x7fad240b56d0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.754+0000 7fad2999a640 1 -- 172.21.15.59:0/1029458281 >> 172.21.15.59:0/1029458281 conn(0x7fad240b6fb0 msgr2=0x7fad240b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.754+0000 7fad2999a640 1 -- 172.21.15.59:0/1029458281 shutdown_connections 2024-06-29T14:41:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.754+0000 7fad2999a640 1 -- 172.21.15.59:0/1029458281 wait complete. 2024-06-29T14:41:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.755+0000 7fad2999a640 1 Processor -- start 2024-06-29T14:41:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.755+0000 7fad2999a640 1 -- start start 2024-06-29T14:41:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad2999a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2413e9b0 0x7fad241f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad2999a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad241414c0 0x7fad2413d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad2999a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2415ceb0 0x7fad240cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad2999a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fad240b2ab0 con 0x7fad241414c0 2024-06-29T14:41:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad2999a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fad240b2930 con 0x7fad2413e9b0 2024-06-29T14:41:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad2999a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fad240b2c30 con 0x7fad2415ceb0 2024-06-29T14:41:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad241414c0 0x7fad2413d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad241414c0 0x7fad2413d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54168/0 (socket says 172.21.15.59:54168) 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 -- 172.21.15.59:0/694331809 learned_addr learned my addr 172.21.15.59:0/694331809 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad22ffd640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2413e9b0 0x7fad241f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad237fe640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2415ceb0 0x7fad240cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 -- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2415ceb0 msgr2=0x7fad240cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2415ceb0 0x7fad240cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 -- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2413e9b0 msgr2=0x7fad241f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2413e9b0 0x7fad241f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 -- 172.21.15.59:0/694331809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad240cc2f0 con 0x7fad241414c0 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.757+0000 7fad1bfff640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad241414c0 0x7fad2413d030 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00d750 tx=0x7fad0c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.758+0000 7fad20ff9640 1 -- 172.21.15.59:0/694331809 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c017070 con 0x7fad241414c0 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.758+0000 7fad20ff9640 1 -- 172.21.15.59:0/694331809 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad0c009b10 con 0x7fad241414c0 2024-06-29T14:41:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.758+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad240cc5d0 con 0x7fad241414c0 2024-06-29T14:41:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.758+0000 7fad20ff9640 1 -- 172.21.15.59:0/694331809 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c010460 con 0x7fad241414c0 2024-06-29T14:41:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.758+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad240ccb90 con 0x7fad241414c0 2024-06-29T14:41:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.759+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad2413edc0 con 0x7fad241414c0 2024-06-29T14:41:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.759+0000 7fad20ff9640 1 -- 172.21.15.59:0/694331809 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 66) ==== 98039+0+0 (secure 0 0 0) 0x7fad0c01a070 con 0x7fad241414c0 2024-06-29T14:41:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.760+0000 7fad20ff9640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad04076120 0x7fad04078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.760+0000 7fad20ff9640 1 -- 172.21.15.59:0/694331809 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 6931+0+0 (secure 0 0 0) 0x7fad0c097d30 con 0x7fad241414c0 2024-06-29T14:41:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.760+0000 7fad22ffd640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad04076120 0x7fad04078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.761+0000 7fad22ffd640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad04076120 0x7fad04078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad14033000 tx=0x7fad14004730 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.764+0000 7fad20ff9640 1 -- 172.21.15.59:0/694331809 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fad0c061480 con 0x7fad241414c0 2024-06-29T14:41:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.958+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad2415c600 con 0x7fad241414c0 2024-06-29T14:41:08.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.959+0000 7fad20ff9640 1 -- 172.21.15.59:0/694331809 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+18145 (secure 0 0 0) 0x7fad0c060d80 con 0x7fad241414c0 2024-06-29T14:41:08.958 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:08.958 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":132,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:07.940217+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:56.243419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.825874+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3471373028":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/13104752":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2821906341":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6833/3741042522":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6832/3741042522":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/3120890145":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/473258225":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/700951802":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/3413132296":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2246604721":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/1279390380":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2067735430":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/1732960037":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6833/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:6832/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/3736955455":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2996335365":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/399775492":"2024-06-30T14:40:10.139401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.961+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad04076120 msgr2=0x7fad04078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.961+0000 7fad2999a640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad04076120 0x7fad04078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad14033000 tx=0x7fad14004730 comp rx=0 tx=0).stop 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad241414c0 msgr2=0x7fad2413d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad241414c0 0x7fad2413d030 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00d750 tx=0x7fad0c00dc10 comp rx=0 tx=0).stop 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 shutdown_connections 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad04076120 0x7fad04078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad2415ceb0 0x7fad240cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad241414c0 0x7fad2413d030 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 --2- 172.21.15.59:0/694331809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2413e9b0 0x7fad241f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 >> 172.21.15.59:0/694331809 conn(0x7fad240b6fb0 msgr2=0x7fad2413f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 shutdown_connections 2024-06-29T14:41:08.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:08.962+0000 7fad2999a640 1 -- 172.21.15.59:0/694331809 wait complete. 2024-06-29T14:41:08.971 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:41:09.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.092+0000 7f8b4613c640 1 -- 172.21.15.59:0/3443240732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b280099b0 msgr2=0x7f8b2800bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.092+0000 7f8b4613c640 1 --2- 172.21.15.59:0/3443240732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b280099b0 0x7f8b2800bdd0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f8b38005fd0 tx=0x7f8b28005830 comp rx=0 tx=0).stop 2024-06-29T14:41:09.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.092+0000 7f8b4613c640 1 -- 172.21.15.59:0/3443240732 shutdown_connections 2024-06-29T14:41:09.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.092+0000 7f8b4613c640 1 --2- 172.21.15.59:0/3443240732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b280099b0 0x7f8b2800bdd0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.092+0000 7f8b4613c640 1 -- 172.21.15.59:0/3443240732 >> 172.21.15.59:0/3443240732 conn(0x7f8b400b5610 msgr2=0x7f8b400b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:09.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.092+0000 7f8b4613c640 1 -- 172.21.15.59:0/3443240732 shutdown_connections 2024-06-29T14:41:09.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.092+0000 7f8b4613c640 1 -- 172.21.15.59:0/3443240732 wait complete. 2024-06-29T14:41:09.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.093+0000 7f8b4613c640 1 Processor -- start 2024-06-29T14:41:09.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.094+0000 7f8b4613c640 1 -- start start 2024-06-29T14:41:09.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b4613c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b280099b0 0x7f8b401592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b4613c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b401576e0 0x7f8b401550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b4613c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b40157af0 0x7f8b40153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b4613c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8b400cf6e0 con 0x7f8b401576e0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b4613c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b400cf560 con 0x7f8b40157af0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b4613c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8b401622c0 con 0x7f8b280099b0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b401576e0 0x7f8b401550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b401576e0 0x7f8b401550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54178/0 (socket says 172.21.15.59:54178) 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3effd640 1 -- 172.21.15.59:0/2186809272 learned_addr learned my addr 172.21.15.59:0/2186809272 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3f7fe640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b280099b0 0x7f8b401592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3ffff640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b40157af0 0x7f8b40153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3effd640 1 -- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b280099b0 msgr2=0x7f8b401592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3effd640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b280099b0 0x7f8b401592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3effd640 1 -- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b40157af0 msgr2=0x7f8b40153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3effd640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b40157af0 0x7f8b40153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.095+0000 7f8b3effd640 1 -- 172.21.15.59:0/2186809272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b40154020 con 0x7f8b401576e0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.096+0000 7f8b3effd640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b401576e0 0x7f8b401550f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400e590 tx=0x7f8b3400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.096+0000 7f8b27fff640 1 -- 172.21.15.59:0/2186809272 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34015070 con 0x7f8b401576e0 2024-06-29T14:41:09.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.096+0000 7f8b27fff640 1 -- 172.21.15.59:0/2186809272 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b34010980 con 0x7f8b401576e0 2024-06-29T14:41:09.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.096+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b401587f0 con 0x7f8b401576e0 2024-06-29T14:41:09.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.096+0000 7f8b27fff640 1 -- 172.21.15.59:0/2186809272 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34010dd0 con 0x7f8b401576e0 2024-06-29T14:41:09.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.096+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b40158ab0 con 0x7f8b401576e0 2024-06-29T14:41:09.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.097+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b40162820 con 0x7f8b401576e0 2024-06-29T14:41:09.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.098+0000 7f8b27fff640 1 -- 172.21.15.59:0/2186809272 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 66) ==== 98039+0+0 (secure 0 0 0) 0x7f8b34018070 con 0x7f8b401576e0 2024-06-29T14:41:09.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.098+0000 7f8b27fff640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8b2c076120 0x7f8b2c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.098+0000 7f8b27fff640 1 -- 172.21.15.59:0/2186809272 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 6931+0+0 (secure 0 0 0) 0x7f8b34097a10 con 0x7f8b401576e0 2024-06-29T14:41:09.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.098+0000 7f8b3f7fe640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8b2c076120 0x7f8b2c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.099+0000 7f8b3f7fe640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8b2c076120 0x7f8b2c078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b30000c00 tx=0x7f8b30006b70 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:09.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.103+0000 7f8b27fff640 1 -- 172.21.15.59:0/2186809272 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8b34060b90 con 0x7f8b401576e0 2024-06-29T14:41:09.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.325+0000 7f8b27fff640 1 -- 172.21.15.59:0/2186809272 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 67) ==== 98247+0+0 (secure 0 0 0) 0x7f8b3405ce20 con 0x7f8b401576e0 2024-06-29T14:41:09.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.334+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8b400b8310 con 0x7f8b401576e0 2024-06-29T14:41:09.333 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T14:41:09.335+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:41:09.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.337+0000 7f8b27fff640 1 -- 172.21.15.59:0/2186809272 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v64) ==== 161+0+0 (secure 0 0 0) 0x7f8b3406b2c0 con 0x7f8b401576e0 2024-06-29T14:41:09.335 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:41:09.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8b2c076120 msgr2=0x7f8b2c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8b2c076120 0x7f8b2c078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b30000c00 tx=0x7f8b30006b70 comp rx=0 tx=0).stop 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b401576e0 msgr2=0x7f8b401550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b401576e0 0x7f8b401550f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400e590 tx=0x7f8b3400ea50 comp rx=0 tx=0).stop 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 shutdown_connections 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b40157af0 0x7f8b40153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b401576e0 0x7f8b401550f0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8b2c076120 0x7f8b2c078630 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8b30000c00 tx=0x7f8b30006b70 comp rx=0 tx=0).stop 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 --2- 172.21.15.59:0/2186809272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b280099b0 0x7f8b401592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.339+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 >> 172.21.15.59:0/2186809272 conn(0x7f8b400b5610 msgr2=0x7f8b4006d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.340+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 shutdown_connections 2024-06-29T14:41:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.340+0000 7f8b4613c640 1 -- 172.21.15.59:0/2186809272 wait complete. 2024-06-29T14:41:09.350 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:41:09.401 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:41:09.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.475+0000 7f9b222d6640 1 -- 172.21.15.59:0/2200258884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b1c13dad0 msgr2=0x7f9b1c15bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.475+0000 7f9b222d6640 1 --2- 172.21.15.59:0/2200258884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b1c13dad0 0x7f9b1c15bb40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c005fd0 tx=0x7f9b0c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.475+0000 7f9b222d6640 1 -- 172.21.15.59:0/2200258884 shutdown_connections 2024-06-29T14:41:09.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.475+0000 7f9b222d6640 1 --2- 172.21.15.59:0/2200258884 >> v2:172.21.15.153:3300/0 conn(0x7f9b1c15e740 0x7f9b1c15eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.475+0000 7f9b222d6640 1 --2- 172.21.15.59:0/2200258884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b1c13dad0 0x7f9b1c15bb40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.475+0000 7f9b222d6640 1 -- 172.21.15.59:0/2200258884 >> 172.21.15.59:0/2200258884 conn(0x7f9b1c06cb30 msgr2=0x7f9b1c06cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:09.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.475+0000 7f9b222d6640 1 -- 172.21.15.59:0/2200258884 shutdown_connections 2024-06-29T14:41:09.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.475+0000 7f9b222d6640 1 -- 172.21.15.59:0/2200258884 wait complete. 2024-06-29T14:41:09.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.477+0000 7f9b222d6640 1 Processor -- start 2024-06-29T14:41:09.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.477+0000 7f9b222d6640 1 -- start start 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b222d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b1c06f180 0x7f9b1c0aef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b222d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b1c13dad0 0x7f9b1c150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b222d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b1c15e740 0x7f9b1c13bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b222d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c0ce6e0 con 0x7f9b1c13dad0 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b222d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c0ce560 con 0x7f9b1c06f180 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b222d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1c0ce860 con 0x7f9b1c15e740 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b1b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b1c13dad0 0x7f9b1c150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b1b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b1c13dad0 0x7f9b1c150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54200/0 (socket says 172.21.15.59:54200) 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.478+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/3009163656 learned_addr learned my addr 172.21.15.59:0/3009163656 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b20873640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b1c15e740 0x7f9b1c13bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b1bfff640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b1c06f180 0x7f9b1c0aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b1c15e740 msgr2=0x7f9b1c13bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b1b7fe640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b1c15e740 0x7f9b1c13bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b1c06f180 msgr2=0x7f9b1c0aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b1b7fe640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b1c06f180 0x7f9b1c0aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b1b7fe640 1 -- 172.21.15.59:0/3009163656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1c1f3300 con 0x7f9b1c13dad0 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b1b7fe640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b1c13dad0 0x7f9b1c150260 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000e790 tx=0x7f9b1000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b18ff9640 1 -- 172.21.15.59:0/3009163656 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1000eea0 con 0x7f9b1c13dad0 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b18ff9640 1 -- 172.21.15.59:0/3009163656 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b1000cd50 con 0x7f9b1c13dad0 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b1c1f35e0 con 0x7f9b1c13dad0 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b18ff9640 1 -- 172.21.15.59:0/3009163656 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b10010690 con 0x7f9b1c13dad0 2024-06-29T14:41:09.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.479+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b1c1f60e0 con 0x7f9b1c13dad0 2024-06-29T14:41:09.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.480+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b1c0ced20 con 0x7f9b1c13dad0 2024-06-29T14:41:09.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.481+0000 7f9b18ff9640 1 -- 172.21.15.59:0/3009163656 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 67) ==== 98247+0+0 (secure 0 0 0) 0x7f9b10010830 con 0x7f9b1c13dad0 2024-06-29T14:41:09.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.481+0000 7f9b18ff9640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9b08076360 0x7f9b08078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.482+0000 7f9b18ff9640 1 -- 172.21.15.59:0/3009163656 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 6989+0+0 (secure 0 0 0) 0x7f9b10098600 con 0x7f9b1c13dad0 2024-06-29T14:41:09.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.482+0000 7f9b1bfff640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9b08076360 0x7f9b08078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.482+0000 7f9b1bfff640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9b08076360 0x7f9b08078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c033000 tx=0x7f9b0c004730 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.486+0000 7f9b18ff9640 1 -- 172.21.15.59:0/3009163656 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9b10061310 con 0x7f9b1c13dad0 2024-06-29T14:41:09.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:09.720+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9b1c1f3050 con 0x7f9b1c13dad0 2024-06-29T14:41:10.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.340+0000 7f9b18ff9640 1 -- 172.21.15.59:0/3009163656 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v65) ==== 104+0+0 (secure 0 0 0) 0x7f9b10065610 con 0x7f9b1c13dad0 2024-06-29T14:41:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9b08076360 msgr2=0x7f9b08078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9b08076360 0x7f9b08078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c033000 tx=0x7f9b0c004730 comp rx=0 tx=0).stop 2024-06-29T14:41:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b1c13dad0 msgr2=0x7f9b1c150260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b1c13dad0 0x7f9b1c150260 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000e790 tx=0x7f9b1000ec50 comp rx=0 tx=0).stop 2024-06-29T14:41:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 shutdown_connections 2024-06-29T14:41:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9b08076360 0x7f9b08078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9b1c15e740 0x7f9b1c13bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b1c13dad0 0x7f9b1c150260 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 --2- 172.21.15.59:0/3009163656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b1c06f180 0x7f9b1c0aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.342+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 >> 172.21.15.59:0/3009163656 conn(0x7f9b1c06cb30 msgr2=0x7f9b1c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.343+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 shutdown_connections 2024-06-29T14:41:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.343+0000 7f9b222d6640 1 -- 172.21.15.59:0/3009163656 wait complete. 2024-06-29T14:41:10.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:41:10.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.485+0000 7fe456c88640 1 -- 172.21.15.59:0/4263988295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe438004800 msgr2=0x7fe438004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.485+0000 7fe456c88640 1 --2- 172.21.15.59:0/4263988295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe438004800 0x7fe438004c10 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe4400016c0 tx=0x7fe438005830 comp rx=0 tx=0).stop 2024-06-29T14:41:10.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.485+0000 7fe456c88640 1 -- 172.21.15.59:0/4263988295 shutdown_connections 2024-06-29T14:41:10.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.485+0000 7fe456c88640 1 --2- 172.21.15.59:0/4263988295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe438004800 0x7fe438004c10 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.485+0000 7fe456c88640 1 -- 172.21.15.59:0/4263988295 >> 172.21.15.59:0/4263988295 conn(0x7fe4500d3e60 msgr2=0x7fe4500d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:10.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.485+0000 7fe456c88640 1 -- 172.21.15.59:0/4263988295 shutdown_connections 2024-06-29T14:41:10.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.485+0000 7fe456c88640 1 -- 172.21.15.59:0/4263988295 wait complete. 2024-06-29T14:41:10.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.487+0000 7fe456c88640 1 Processor -- start 2024-06-29T14:41:10.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.487+0000 7fe456c88640 1 -- start start 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.488+0000 7fe456c88640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe438004800 0x7fe4501531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe456c88640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe450157510 0x7fe4501f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe456c88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe450154560 0x7fe450159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe456c88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe450161e70 con 0x7fe450154560 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe456c88640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe450161cf0 con 0x7fe438004800 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe456c88640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe450161ff0 con 0x7fe450157510 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe450154560 0x7fe450159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe450154560 0x7fe450159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54220/0 (socket says 172.21.15.59:54220) 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 -- 172.21.15.59:0/2603844158 learned_addr learned my addr 172.21.15.59:0/2603844158 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe447fff640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe450157510 0x7fe4501f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe454a24640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe438004800 0x7fe4501531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 -- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe450157510 msgr2=0x7fe4501f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe450157510 0x7fe4501f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 -- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe438004800 msgr2=0x7fe4501531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe438004800 0x7fe4501531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 -- 172.21.15.59:0/2603844158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4501f5d00 con 0x7fe450154560 2024-06-29T14:41:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe455225640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe450154560 0x7fe450159e60 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe4500caac0 tx=0x7fe448010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe4457fa640 1 -- 172.21.15.59:0/2603844158 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44800ee10 con 0x7fe450154560 2024-06-29T14:41:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe4457fa640 1 -- 172.21.15.59:0/2603844158 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe448015070 con 0x7fe450154560 2024-06-29T14:41:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.489+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe450157ab0 con 0x7fe450154560 2024-06-29T14:41:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.490+0000 7fe4457fa640 1 -- 172.21.15.59:0/2603844158 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448003d00 con 0x7fe450154560 2024-06-29T14:41:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.490+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe450159380 con 0x7fe450154560 2024-06-29T14:41:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.491+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe450162550 con 0x7fe450154560 2024-06-29T14:41:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.491+0000 7fe4457fa640 1 -- 172.21.15.59:0/2603844158 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 67) ==== 98247+0+0 (secure 0 0 0) 0x7fe448004030 con 0x7fe450154560 2024-06-29T14:41:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.492+0000 7fe4457fa640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe43c076390 0x7fe43c0788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.492+0000 7fe4457fa640 1 -- 172.21.15.59:0/2603844158 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 6989+0+0 (secure 0 0 0) 0x7fe448098950 con 0x7fe450154560 2024-06-29T14:41:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.492+0000 7fe454a24640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe43c076390 0x7fe43c0788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.493+0000 7fe454a24640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe43c076390 0x7fe43c0788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe44c001410 tx=0x7fe44c006910 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:10.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.497+0000 7fe4457fa640 1 -- 172.21.15.59:0/2603844158 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe4501f5d00 con 0x7fe450154560 2024-06-29T14:41:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:10.690+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe4500b2c20 con 0x7fe450154560 2024-06-29T14:41:11.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.347+0000 7fe4457fa640 1 -- 172.21.15.59:0/2603844158 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v134) ==== 181+0+0 (secure 0 0 0) 0x7fe448061d50 con 0x7fe450154560 2024-06-29T14:41:11.346 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe43c076390 msgr2=0x7fe43c0788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe43c076390 0x7fe43c0788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe44c001410 tx=0x7fe44c006910 comp rx=0 tx=0).stop 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe450154560 msgr2=0x7fe450159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe450154560 0x7fe450159e60 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe4500caac0 tx=0x7fe448010f40 comp rx=0 tx=0).stop 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 shutdown_connections 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe450154560 0x7fe450159e60 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe43c076390 0x7fe43c0788a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe450157510 0x7fe4501f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 --2- 172.21.15.59:0/2603844158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe438004800 0x7fe4501531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 >> 172.21.15.59:0/2603844158 conn(0x7fe4500d3e60 msgr2=0x7fe4500b7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:11.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 shutdown_connections 2024-06-29T14:41:11.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.350+0000 7fe456c88640 1 -- 172.21.15.59:0/2603844158 wait complete. 2024-06-29T14:41:11.367 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:41:11.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.498+0000 7fe5add9a640 1 -- 172.21.15.59:0/3798987771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe590003a00 msgr2=0x7fe590003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.498+0000 7fe5add9a640 1 --2- 172.21.15.59:0/3798987771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe590003a00 0x7fe590003e70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe59800c290 tx=0x7fe590005830 comp rx=0 tx=0).stop 2024-06-29T14:41:11.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.498+0000 7fe5a7fff640 1 -- 172.21.15.59:0/3798987771 reap_dead start 2024-06-29T14:41:11.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.498+0000 7fe5add9a640 1 -- 172.21.15.59:0/3798987771 shutdown_connections 2024-06-29T14:41:11.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.498+0000 7fe5add9a640 1 -- 172.21.15.59:0/3798987771 >> 172.21.15.59:0/3798987771 conn(0x7fe5a80b7690 msgr2=0x7fe5a8073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:11.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.499+0000 7fe5add9a640 1 -- 172.21.15.59:0/3798987771 shutdown_connections 2024-06-29T14:41:11.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.499+0000 7fe5add9a640 1 -- 172.21.15.59:0/3798987771 wait complete. 2024-06-29T14:41:11.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.500+0000 7fe5add9a640 1 Processor -- start 2024-06-29T14:41:11.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.501+0000 7fe5add9a640 1 -- start start 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5add9a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe590003a00 0x7fe5a81586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5add9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5a806f020 0x7fe5a81efb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5add9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5a80d10d0 0x7fe5a80d14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5add9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a8159a30 con 0x7fe5a806f020 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5add9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a81598b0 con 0x7fe5a80d10d0 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5add9a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe5a80b1080 con 0x7fe590003a00 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5a806f020 0x7fe5a81efb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5a806f020 0x7fe5a81efb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54234/0 (socket says 172.21.15.59:54234) 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 -- 172.21.15.59:0/403327868 learned_addr learned my addr 172.21.15.59:0/403327868 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a7fff640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5a80d10d0 0x7fe5a80d14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a77fe640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe590003a00 0x7fe5a81586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 -- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe590003a00 msgr2=0x7fe5a81586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe590003a00 0x7fe5a81586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 -- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5a80d10d0 msgr2=0x7fe5a80d14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5a80d10d0 0x7fe5a80d14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 -- 172.21.15.59:0/403327868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a80d1a20 con 0x7fe5a806f020 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.502+0000 7fe5a6ffd640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5a806f020 0x7fe5a81efb00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fe59c011ae0 tx=0x7fe59c009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.503+0000 7fe58ffff640 1 -- 172.21.15.59:0/403327868 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c01e070 con 0x7fe5a806f020 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.503+0000 7fe58ffff640 1 -- 172.21.15.59:0/403327868 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe59c004480 con 0x7fe5a806f020 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.503+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5a8150660 con 0x7fe5a806f020 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.503+0000 7fe58ffff640 1 -- 172.21.15.59:0/403327868 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00fc60 con 0x7fe5a806f020 2024-06-29T14:41:11.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.503+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5a8150c20 con 0x7fe5a806f020 2024-06-29T14:41:11.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.504+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5a80b4940 con 0x7fe5a806f020 2024-06-29T14:41:11.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.504+0000 7fe58ffff640 1 -- 172.21.15.59:0/403327868 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fe59c021070 con 0x7fe5a806f020 2024-06-29T14:41:11.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.505+0000 7fe58ffff640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe5940763f0 0x7fe594078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:11.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.505+0000 7fe58ffff640 1 -- 172.21.15.59:0/403327868 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6551+0+0 (secure 0 0 0) 0x7fe59c098820 con 0x7fe5a806f020 2024-06-29T14:41:11.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.505+0000 7fe5a77fe640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe5940763f0 0x7fe594078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:11.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.506+0000 7fe5a77fe640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe5940763f0 0x7fe594078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe598033000 tx=0x7fe59802d000 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.509+0000 7fe58ffff640 1 -- 172.21.15.59:0/403327868 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe5a80d1a20 con 0x7fe5a806f020 2024-06-29T14:41:11.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:11.702+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe5a8153ec0 con 0x7fe5a806f020 2024-06-29T14:41:12.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.349+0000 7fe58ffff640 1 -- 172.21.15.59:0/403327868 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) ==== 169+0+0 (secure 0 0 0) 0x7fe59c096e20 con 0x7fe5a806f020 2024-06-29T14:41:12.347 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T14:41:12.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.351+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe5940763f0 msgr2=0x7fe594078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.351+0000 7fe5add9a640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe5940763f0 0x7fe594078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe598033000 tx=0x7fe59802d000 comp rx=0 tx=0).stop 2024-06-29T14:41:12.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.351+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5a806f020 msgr2=0x7fe5a81efb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.351+0000 7fe5add9a640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5a806f020 0x7fe5a81efb00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fe59c011ae0 tx=0x7fe59c009e40 comp rx=0 tx=0).stop 2024-06-29T14:41:12.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.355+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 shutdown_connections 2024-06-29T14:41:12.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.355+0000 7fe5add9a640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5a80d10d0 0x7fe5a80d14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.355+0000 7fe5add9a640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5a806f020 0x7fe5a81efb00 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.355+0000 7fe5add9a640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe5940763f0 0x7fe594078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.355+0000 7fe5add9a640 1 --2- 172.21.15.59:0/403327868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe590003a00 0x7fe5a81586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.355+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 >> 172.21.15.59:0/403327868 conn(0x7fe5a80b7690 msgr2=0x7fe5a80cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:12.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.355+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 shutdown_connections 2024-06-29T14:41:12.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.355+0000 7fe5add9a640 1 -- 172.21.15.59:0/403327868 wait complete. 2024-06-29T14:41:12.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:41:12.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.492+0000 7f6d2bc26640 1 -- 172.21.15.59:0/3690564507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d240b40a0 msgr2=0x7f6d2415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.492+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/3690564507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d240b40a0 0x7f6d2415e8d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009830 tx=0x7f6d1802a500 comp rx=0 tx=0).stop 2024-06-29T14:41:12.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.492+0000 7f6d2bc26640 1 -- 172.21.15.59:0/3690564507 shutdown_connections 2024-06-29T14:41:12.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.492+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/3690564507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d2415ee10 0x7f6d2415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.492+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/3690564507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d240b40a0 0x7f6d2415e8d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.492+0000 7f6d2bc26640 1 -- 172.21.15.59:0/3690564507 >> 172.21.15.59:0/3690564507 conn(0x7f6d240d3ee0 msgr2=0x7f6d240d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:12.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.492+0000 7f6d2bc26640 1 -- 172.21.15.59:0/3690564507 shutdown_connections 2024-06-29T14:41:12.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.492+0000 7f6d2bc26640 1 -- 172.21.15.59:0/3690564507 wait complete. 2024-06-29T14:41:12.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.494+0000 7f6d2bc26640 1 Processor -- start 2024-06-29T14:41:12.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.494+0000 7f6d2bc26640 1 -- start start 2024-06-29T14:41:12.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2bc26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d240d23c0 0x7f6d241410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2bc26640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d2415ee10 0x7f6d24152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2bc26640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d24142550 0x7f6d241441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2bc26640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6d24160280 con 0x7f6d24142550 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2bc26640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d24160100 con 0x7f6d240d23c0 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2bc26640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6d24160400 con 0x7f6d2415ee10 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2a1c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d24142550 0x7f6d241441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2a1c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d24142550 0x7f6d241441d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54240/0 (socket says 172.21.15.59:54240) 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2a1c3640 1 -- 172.21.15.59:0/175299988 learned_addr learned my addr 172.21.15.59:0/175299988 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d291c1640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d2415ee10 0x7f6d24152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d299c2640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d240d23c0 0x7f6d241410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2a1c3640 1 -- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d2415ee10 msgr2=0x7f6d24152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2a1c3640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d2415ee10 0x7f6d24152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2a1c3640 1 -- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d240d23c0 msgr2=0x7f6d241410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.495+0000 7f6d2a1c3640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d240d23c0 0x7f6d241410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.496+0000 7f6d2a1c3640 1 -- 172.21.15.59:0/175299988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d24141790 con 0x7f6d24142550 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.496+0000 7f6d2a1c3640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d24142550 0x7f6d241441d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0058c0 tx=0x7f6d1c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.496+0000 7f6d227fc640 1 -- 172.21.15.59:0/175299988 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c00ccc0 con 0x7f6d24142550 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.496+0000 7f6d227fc640 1 -- 172.21.15.59:0/175299988 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d1c004030 con 0x7f6d24142550 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.496+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d24141a70 con 0x7f6d24142550 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.496+0000 7f6d227fc640 1 -- 172.21.15.59:0/175299988 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c00ce60 con 0x7f6d24142550 2024-06-29T14:41:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.496+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d24142030 con 0x7f6d24142550 2024-06-29T14:41:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.497+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d2415e8d0 con 0x7f6d24142550 2024-06-29T14:41:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.498+0000 7f6d227fc640 1 -- 172.21.15.59:0/175299988 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f6d1c0035f0 con 0x7f6d24142550 2024-06-29T14:41:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.498+0000 7f6d227fc640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6d100763f0 0x7f6d10078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.499+0000 7f6d227fc640 1 -- 172.21.15.59:0/175299988 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7f6d1c098400 con 0x7f6d24142550 2024-06-29T14:41:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.499+0000 7f6d299c2640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6d100763f0 0x7f6d10078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.499+0000 7f6d299c2640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6d100763f0 0x7f6d10078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6d240d2e40 tx=0x7f6d14009040 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:12.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.503+0000 7f6d227fc640 1 -- 172.21.15.59:0/175299988 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6d1c061340 con 0x7f6d24142550 2024-06-29T14:41:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.703+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f6d240d2950 con 0x7f6d24142550 2024-06-29T14:41:12.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.707+0000 7f6d227fc640 1 -- 172.21.15.59:0/175299988 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v51) ==== 71+0+0 (secure 0 0 0) 0x7f6d1c065640 con 0x7f6d24142550 2024-06-29T14:41:12.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6d100763f0 msgr2=0x7f6d10078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6d100763f0 0x7f6d10078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6d240d2e40 tx=0x7f6d14009040 comp rx=0 tx=0).stop 2024-06-29T14:41:12.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d24142550 msgr2=0x7f6d241441d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d24142550 0x7f6d241441d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0058c0 tx=0x7f6d1c00ed20 comp rx=0 tx=0).stop 2024-06-29T14:41:12.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 shutdown_connections 2024-06-29T14:41:12.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d24142550 0x7f6d241441d0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6d100763f0 0x7f6d10078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d2415ee10 0x7f6d24152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 --2- 172.21.15.59:0/175299988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d240d23c0 0x7f6d241410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 >> 172.21.15.59:0/175299988 conn(0x7f6d240d3ee0 msgr2=0x7f6d2406cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:12.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 shutdown_connections 2024-06-29T14:41:12.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.709+0000 7f6d2bc26640 1 -- 172.21.15.59:0/175299988 wait complete. 2024-06-29T14:41:12.718 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-06-29T14:41:12.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 -- 172.21.15.59:0/4022131646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42140b3740 msgr2=0x7f42140b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 --2- 172.21.15.59:0/4022131646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42140b3740 0x7f42140b3b50 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f42140b4b50 tx=0x7f420802a560 comp rx=0 tx=0).stop 2024-06-29T14:41:12.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 -- 172.21.15.59:0/4022131646 shutdown_connections 2024-06-29T14:41:12.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 --2- 172.21.15.59:0/4022131646 >> v2:172.21.15.153:3300/0 conn(0x7f421415c2f0 0x7f421415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 --2- 172.21.15.59:0/4022131646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42140b3740 0x7f42140b3b50 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 --2- 172.21.15.59:0/4022131646 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42141458a0 0x7f42140cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 -- 172.21.15.59:0/4022131646 >> 172.21.15.59:0/4022131646 conn(0x7f421406c950 msgr2=0x7f4214073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:12.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 -- 172.21.15.59:0/4022131646 shutdown_connections 2024-06-29T14:41:12.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.838+0000 7f421a518640 1 -- 172.21.15.59:0/4022131646 wait complete. 2024-06-29T14:41:12.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.839+0000 7f421a518640 1 Processor -- start 2024-06-29T14:41:12.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.840+0000 7f421a518640 1 -- start start 2024-06-29T14:41:12.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f421a518640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42140b3740 0x7f42141f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f421a518640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f421415c2f0 0x7f42141f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f421a518640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421414e780 0x7f42141f8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f421a518640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f421415ff40 con 0x7f421414e780 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f421a518640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f421415fdc0 con 0x7f421415c2f0 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f421a518640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f42141600c0 con 0x7f42140b3740 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4218ab5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421414e780 0x7f42141f8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4218ab5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421414e780 0x7f42141f8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57836/0 (socket says 172.21.15.59:57836) 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4218ab5640 1 -- 172.21.15.59:0/2570503937 learned_addr learned my addr 172.21.15.59:0/2570503937 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4213fff640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42140b3740 0x7f42141f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f42137fe640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f421415c2f0 0x7f42141f89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4218ab5640 1 -- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42140b3740 msgr2=0x7f42141f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4218ab5640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42140b3740 0x7f42141f5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4218ab5640 1 -- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f421415c2f0 msgr2=0x7f42141f89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4218ab5640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f421415c2f0 0x7f42141f89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.841+0000 7f4218ab5640 1 -- 172.21.15.59:0/2570503937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42140b13f0 con 0x7f421414e780 2024-06-29T14:41:12.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.842+0000 7f4218ab5640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421414e780 0x7f42141f8ee0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f420c00ca80 tx=0x7f420c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:12.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.842+0000 7f4210ff9640 1 -- 172.21.15.59:0/2570503937 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c007d90 con 0x7f421414e780 2024-06-29T14:41:12.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.842+0000 7f4210ff9640 1 -- 172.21.15.59:0/2570503937 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f420c004c30 con 0x7f421414e780 2024-06-29T14:41:12.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.842+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421406dd50 con 0x7f421414e780 2024-06-29T14:41:12.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.842+0000 7f4210ff9640 1 -- 172.21.15.59:0/2570503937 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c005280 con 0x7f421414e780 2024-06-29T14:41:12.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.842+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f421406e310 con 0x7f421414e780 2024-06-29T14:41:12.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.843+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42141f6130 con 0x7f421414e780 2024-06-29T14:41:12.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.844+0000 7f4210ff9640 1 -- 172.21.15.59:0/2570503937 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f420c005420 con 0x7f421414e780 2024-06-29T14:41:12.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.844+0000 7f4210ff9640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f42000763f0 0x7f4200078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.844+0000 7f4210ff9640 1 -- 172.21.15.59:0/2570503937 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7f420c0985f0 con 0x7f421414e780 2024-06-29T14:41:12.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.844+0000 7f4213fff640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f42000763f0 0x7f4200078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.845+0000 7f4213fff640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f42000763f0 0x7f4200078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f420400aa10 tx=0x7f4204005ed0 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:12.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:12.848+0000 7f4210ff9640 1 -- 172.21.15.59:0/2570503937 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f420c014070 con 0x7f421414e780 2024-06-29T14:41:13.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.114+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) -- 0x7f42141605e0 con 0x7f421414e780 2024-06-29T14:41:13.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.357+0000 7f4210ff9640 1 -- 172.21.15.59:0/2570503937 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v746) ==== 148+0+0 (secure 0 0 0) 0x7f420c061420 con 0x7f421414e780 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.359+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f42000763f0 msgr2=0x7f4200078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.359+0000 7f421a518640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f42000763f0 0x7f4200078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f420400aa10 tx=0x7f4204005ed0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.359+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421414e780 msgr2=0x7f42141f8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.359+0000 7f421a518640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421414e780 0x7f42141f8ee0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f420c00ca80 tx=0x7f420c00cf40 comp rx=0 tx=0).stop 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.359+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 shutdown_connections 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.359+0000 7f421a518640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421414e780 0x7f42141f8ee0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.359+0000 7f421a518640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f421415c2f0 0x7f42141f89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.359+0000 7f421a518640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f42000763f0 0x7f4200078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.360+0000 7f421a518640 1 --2- 172.21.15.59:0/2570503937 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42140b3740 0x7f42141f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.360+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 >> 172.21.15.59:0/2570503937 conn(0x7f421406c950 msgr2=0x7f42140b6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.360+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 shutdown_connections 2024-06-29T14:41:13.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.360+0000 7f421a518640 1 -- 172.21.15.59:0/2570503937 wait complete. 2024-06-29T14:41:13.369 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-06-29T14:41:13.370 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:41:13.370 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:13.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.491+0000 7f1f71eef640 1 -- 172.21.15.59:0/58911796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c0b3ed0 msgr2=0x7f1f6c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.491+0000 7f1f71eef640 1 --2- 172.21.15.59:0/58911796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c0b3ed0 0x7f1f6c15e8d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009830 tx=0x7f1f6002a510 comp rx=0 tx=0).stop 2024-06-29T14:41:13.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.491+0000 7f1f71eef640 1 -- 172.21.15.59:0/58911796 shutdown_connections 2024-06-29T14:41:13.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.491+0000 7f1f71eef640 1 --2- 172.21.15.59:0/58911796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c0b3ed0 0x7f1f6c15e8d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.491+0000 7f1f71eef640 1 --2- 172.21.15.59:0/58911796 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f6c0d23c0 0x7f1f6c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.491+0000 7f1f71eef640 1 -- 172.21.15.59:0/58911796 >> 172.21.15.59:0/58911796 conn(0x7f1f6c0d3ee0 msgr2=0x7f1f6c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.491+0000 7f1f71eef640 1 -- 172.21.15.59:0/58911796 shutdown_connections 2024-06-29T14:41:13.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.491+0000 7f1f71eef640 1 -- 172.21.15.59:0/58911796 wait complete. 2024-06-29T14:41:13.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.493+0000 7f1f71eef640 1 Processor -- start 2024-06-29T14:41:13.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.493+0000 7f1f71eef640 1 -- start start 2024-06-29T14:41:13.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f71eef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f6c0b3ed0 0x7f1f6c14ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f71eef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f6c0d23c0 0x7f1f6c1e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f71eef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c15ee10 0x7f1f6c1e9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f71eef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c160240 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f71eef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c1600c0 con 0x7f1f6c15ee10 2024-06-29T14:41:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f71eef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6c1603c0 con 0x7f1f6c0d23c0 2024-06-29T14:41:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f6b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f6c0b3ed0 0x7f1f6c14ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f6b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f6c0b3ed0 0x7f1f6c14ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57844/0 (socket says 172.21.15.59:57844) 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f6b7fe640 1 -- 172.21.15.59:0/1663867938 learned_addr learned my addr 172.21.15.59:0/1663867938 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f6affd640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f6c0d23c0 0x7f1f6c1e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.494+0000 7f1f6bfff640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c15ee10 0x7f1f6c1e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f6b7fe640 1 -- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f6c0d23c0 msgr2=0x7f1f6c1e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f6b7fe640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f6c0d23c0 0x7f1f6c1e8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f6b7fe640 1 -- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c15ee10 msgr2=0x7f1f6c1e9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f6b7fe640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c15ee10 0x7f1f6c1e9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f6b7fe640 1 -- 172.21.15.59:0/1663867938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c155330 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f6affd640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f6c0d23c0 0x7f1f6c1e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f6bfff640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c15ee10 0x7f1f6c1e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f6b7fe640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f6c0b3ed0 0x7f1f6c14ee10 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00e9f0 tx=0x7f1f5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f53fff640 1 -- 172.21.15.59:0/1663867938 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c00cd20 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6c154a90 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f53fff640 1 -- 172.21.15.59:0/1663867938 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f5c010430 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f53fff640 1 -- 172.21.15.59:0/1663867938 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c010860 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.495+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f6c155050 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.496+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6c15e8d0 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.497+0000 7f1f53fff640 1 -- 172.21.15.59:0/1663867938 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f1f5c009400 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.498+0000 7f1f53fff640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f1f580763f0 0x7f1f58078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.498+0000 7f1f53fff640 1 -- 172.21.15.59:0/1663867938 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7f1f5c0986f0 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.498+0000 7f1f6affd640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f1f580763f0 0x7f1f58078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.498+0000 7f1f6affd640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f1f580763f0 0x7f1f58078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1f60034000 tx=0x7f1f60004730 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:13.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.502+0000 7f1f53fff640 1 -- 172.21.15.59:0/1663867938 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1f5c054af0 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.692+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f6c0d2950 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.693+0000 7f1f53fff640 1 -- 172.21.15.59:0/1663867938 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+14498 (secure 0 0 0) 0x7f1f5c061b70 con 0x7f1f6c0b3ed0 2024-06-29T14:41:13.691 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:13.692 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":135,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:12.349002+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/3950841523":"2024-06-30T14:41:09.336655+0000","172.21.15.153:0/3471373028":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/13104752":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2821906341":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6833/3741042522":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6832/3741042522":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/3120890145":"2024-06-30T14:41:06.937819+0000","172.21.15.59:6835/3950841523":"2024-06-30T14:41:09.336655+0000","172.21.15.153:0/473258225":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/700951802":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/3413132296":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2246604721":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/1279390380":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/2067735430":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/1732960037":"2024-06-30T14:41:06.937819+0000","172.21.15.153:6833/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:6832/892574025":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/3736955455":"2024-06-30T14:41:06.937819+0000","172.21.15.153:0/2996335365":"2024-06-30T14:40:10.139401+0000","172.21.15.153:0/399775492":"2024-06-30T14:40:10.139401+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:13.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f1f580763f0 msgr2=0x7f1f58078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f1f580763f0 0x7f1f58078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1f60034000 tx=0x7f1f60004730 comp rx=0 tx=0).stop 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f6c0b3ed0 msgr2=0x7f1f6c14ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f6c0b3ed0 0x7f1f6c14ee10 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00e9f0 tx=0x7f1f5c00edf0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 shutdown_connections 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c15ee10 0x7f1f6c1e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f1f580763f0 0x7f1f58078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f6c0d23c0 0x7f1f6c1e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 --2- 172.21.15.59:0/1663867938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f6c0b3ed0 0x7f1f6c14ee10 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 >> 172.21.15.59:0/1663867938 conn(0x7f1f6c0d3ee0 msgr2=0x7f1f6c06d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 shutdown_connections 2024-06-29T14:41:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.695+0000 7f1f71eef640 1 -- 172.21.15.59:0/1663867938 wait complete. 2024-06-29T14:41:13.705 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:13.705 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:13.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 -- 172.21.15.59:0/477685482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc1458a0 msgr2=0x7f5bfc0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 --2- 172.21.15.59:0/477685482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc1458a0 0x7f5bfc0cf870 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc0d0730 tx=0x7f5bec02aa30 comp rx=0 tx=0).stop 2024-06-29T14:41:13.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 -- 172.21.15.59:0/477685482 shutdown_connections 2024-06-29T14:41:13.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 --2- 172.21.15.59:0/477685482 >> v2:172.21.15.153:3300/0 conn(0x7f5bfc15c2f0 0x7f5bfc15c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 --2- 172.21.15.59:0/477685482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bfc0b3740 0x7f5bfc0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 --2- 172.21.15.59:0/477685482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc1458a0 0x7f5bfc0cf870 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 -- 172.21.15.59:0/477685482 >> 172.21.15.59:0/477685482 conn(0x7f5bfc06c950 msgr2=0x7f5bfc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 -- 172.21.15.59:0/477685482 shutdown_connections 2024-06-29T14:41:13.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.827+0000 7f5c01163640 1 -- 172.21.15.59:0/477685482 wait complete. 2024-06-29T14:41:13.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.829+0000 7f5c01163640 1 Processor -- start 2024-06-29T14:41:13.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.829+0000 7f5c01163640 1 -- start start 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5c01163640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bfc0b3740 0x7f5bfc151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5c01163640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bfc15c2f0 0x7f5bfc0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5c01163640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc0d1e40 0x7f5bfc14e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5c01163640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc15fe40 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5c01163640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc15fcc0 con 0x7f5bfc0b3740 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5c01163640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5bfc15ffc0 con 0x7f5bfc15c2f0 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5bfb59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc0d1e40 0x7f5bfc14e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5bfb59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc0d1e40 0x7f5bfc14e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57866/0 (socket says 172.21.15.59:57866) 2024-06-29T14:41:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5bfb59e640 1 -- 172.21.15.59:0/3729321992 learned_addr learned my addr 172.21.15.59:0/3729321992 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.830+0000 7f5bfad9d640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bfc0b3740 0x7f5bfc151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bfa59c640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bfc15c2f0 0x7f5bfc0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bfb59e640 1 -- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bfc15c2f0 msgr2=0x7f5bfc0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bfb59e640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bfc15c2f0 0x7f5bfc0d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bfb59e640 1 -- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bfc0b3740 msgr2=0x7f5bfc151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bfb59e640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bfc0b3740 0x7f5bfc151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bfb59e640 1 -- 172.21.15.59:0/3729321992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bfc0d2250 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bfb59e640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc0d1e40 0x7f5bfc14e080 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c970 tx=0x7f5bf400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bf37fe640 1 -- 172.21.15.59:0/3729321992 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4007bb0 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bf37fe640 1 -- 172.21.15.59:0/3729321992 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bf400f450 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5bf37fe640 1 -- 172.21.15.59:0/3729321992 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4007e60 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bfc14cfe0 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.831+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bfc14d5a0 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.832+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bfc1458a0 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.833+0000 7f5bf37fe640 1 -- 172.21.15.59:0/3729321992 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f5bf400f8a0 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.833+0000 7f5bf37fe640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5bdc0763f0 0x7f5bdc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.834+0000 7f5bf37fe640 1 -- 172.21.15.59:0/3729321992 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7f5bf4098350 con 0x7f5bfc0d1e40 2024-06-29T14:41:13.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.834+0000 7f5bfad9d640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5bdc0763f0 0x7f5bdc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.834+0000 7f5bfad9d640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5bdc0763f0 0x7f5bdc078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5bec035000 tx=0x7f5bec03b040 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:13.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:13.838+0000 7f5bf37fe640 1 -- 172.21.15.59:0/3729321992 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5bf4014030 con 0x7f5bfc0d1e40 2024-06-29T14:41:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.029+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5bfc0b1f10 con 0x7f5bfc0d1e40 2024-06-29T14:41:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.029+0000 7f5bf37fe640 1 -- 172.21.15.59:0/3729321992 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) ==== 93+0+10 (secure 0 0 0) 0x7f5bf40610e0 con 0x7f5bfc0d1e40 2024-06-29T14:41:14.028 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5bdc0763f0 msgr2=0x7f5bdc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5bdc0763f0 0x7f5bdc078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5bec035000 tx=0x7f5bec03b040 comp rx=0 tx=0).stop 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc0d1e40 msgr2=0x7f5bfc14e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc0d1e40 0x7f5bfc14e080 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c970 tx=0x7f5bf400ce30 comp rx=0 tx=0).stop 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 shutdown_connections 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bfc0d1e40 0x7f5bfc14e080 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5bdc0763f0 0x7f5bdc078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5bfc15c2f0 0x7f5bfc0d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.031+0000 7f5c01163640 1 --2- 172.21.15.59:0/3729321992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bfc0b3740 0x7f5bfc151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.032+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 >> 172.21.15.59:0/3729321992 conn(0x7f5bfc06c950 msgr2=0x7f5bfc144f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:14.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.032+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 shutdown_connections 2024-06-29T14:41:14.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.032+0000 7f5c01163640 1 -- 172.21.15.59:0/3729321992 wait complete. 2024-06-29T14:41:14.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-06-29T14:41:14.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.161+0000 7f9d37617640 1 -- 172.21.15.59:0/1559247367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d18005150 msgr2=0x7f9d1800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.161+0000 7f9d37617640 1 --2- 172.21.15.59:0/1559247367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d18005150 0x7f9d1800b9c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f9d20006fd0 tx=0x7f9d18005830 comp rx=0 tx=0).stop 2024-06-29T14:41:14.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.161+0000 7f9d35bb4640 1 -- 172.21.15.59:0/1559247367 reap_dead start 2024-06-29T14:41:14.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.161+0000 7f9d37617640 1 -- 172.21.15.59:0/1559247367 shutdown_connections 2024-06-29T14:41:14.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.161+0000 7f9d37617640 1 -- 172.21.15.59:0/1559247367 >> 172.21.15.59:0/1559247367 conn(0x7f9d3006c950 msgr2=0x7f9d30073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:14.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.161+0000 7f9d37617640 1 -- 172.21.15.59:0/1559247367 shutdown_connections 2024-06-29T14:41:14.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.161+0000 7f9d37617640 1 -- 172.21.15.59:0/1559247367 wait complete. 2024-06-29T14:41:14.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.163+0000 7f9d37617640 1 Processor -- start 2024-06-29T14:41:14.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.163+0000 7f9d37617640 1 -- start start 2024-06-29T14:41:14.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d37617640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d301f3840 0x7f9d301f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d37617640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d300af4f0 0x7f9d301f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d37617640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d301445f0 0x7f9d30144a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d37617640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9d300cfc00 con 0x7f9d301445f0 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d37617640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9d300cfa80 con 0x7f9d300af4f0 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d37617640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9d3015c700 con 0x7f9d301f3840 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d35bb4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d301445f0 0x7f9d30144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d35bb4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d301445f0 0x7f9d30144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57874/0 (socket says 172.21.15.59:57874) 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d35bb4640 1 -- 172.21.15.59:0/1171060877 learned_addr learned my addr 172.21.15.59:0/1171060877 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d353b3640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d301f3840 0x7f9d301f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d34bb2640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d300af4f0 0x7f9d301f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d35bb4640 1 -- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d301f3840 msgr2=0x7f9d301f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.164+0000 7f9d35bb4640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d301f3840 0x7f9d301f3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d35bb4640 1 -- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d300af4f0 msgr2=0x7f9d301f1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d35bb4640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d300af4f0 0x7f9d301f1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d35bb4640 1 -- 172.21.15.59:0/1171060877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d30144f40 con 0x7f9d301445f0 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d35bb4640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d301445f0 0x7f9d30144a00 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000c6d0 tx=0x7f9d20004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d2dffb640 1 -- 172.21.15.59:0/1171060877 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2002bbf0 con 0x7f9d301445f0 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d2dffb640 1 -- 172.21.15.59:0/1171060877 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d2002bd90 con 0x7f9d301445f0 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d300afc90 con 0x7f9d301445f0 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d2dffb640 1 -- 172.21.15.59:0/1171060877 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d200062d0 con 0x7f9d301445f0 2024-06-29T14:41:14.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.165+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d300ace50 con 0x7f9d301445f0 2024-06-29T14:41:14.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.166+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d300b2350 con 0x7f9d301445f0 2024-06-29T14:41:14.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.167+0000 7f9d2dffb640 1 -- 172.21.15.59:0/1171060877 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f9d20006470 con 0x7f9d301445f0 2024-06-29T14:41:14.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.167+0000 7f9d2dffb640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9d1c0763f0 0x7f9d1c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.168+0000 7f9d2dffb640 1 -- 172.21.15.59:0/1171060877 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7f9d2003b020 con 0x7f9d301445f0 2024-06-29T14:41:14.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.168+0000 7f9d353b3640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9d1c0763f0 0x7f9d1c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.168+0000 7f9d353b3640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9d1c0763f0 0x7f9d1c078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9d30071260 tx=0x7f9d28005f90 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:14.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.172+0000 7f9d2dffb640 1 -- 172.21.15.59:0/1171060877 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9d300b2350 con 0x7f9d301445f0 2024-06-29T14:41:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:14.442+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) -- 0x7f9d300b3b50 con 0x7f9d301445f0 2024-06-29T14:41:15.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.366+0000 7f9d2dffb640 1 -- 172.21.15.59:0/1171060877 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v748) ==== 163+0+0 (secure 0 0 0) 0x7f9d30144f40 con 0x7f9d301445f0 2024-06-29T14:41:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.368+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9d1c0763f0 msgr2=0x7f9d1c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.368+0000 7f9d37617640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9d1c0763f0 0x7f9d1c078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9d30071260 tx=0x7f9d28005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.368+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d301445f0 msgr2=0x7f9d30144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.368+0000 7f9d37617640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d301445f0 0x7f9d30144a00 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000c6d0 tx=0x7f9d20004c70 comp rx=0 tx=0).stop 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.369+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 shutdown_connections 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.369+0000 7f9d37617640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d301445f0 0x7f9d30144a00 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.369+0000 7f9d37617640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d300af4f0 0x7f9d301f1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.369+0000 7f9d37617640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f9d1c0763f0 0x7f9d1c078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.369+0000 7f9d37617640 1 --2- 172.21.15.59:0/1171060877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d301f3840 0x7f9d301f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.369+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 >> 172.21.15.59:0/1171060877 conn(0x7f9d3006c950 msgr2=0x7f9d30073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.369+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 shutdown_connections 2024-06-29T14:41:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.369+0000 7f9d37617640 1 -- 172.21.15.59:0/1171060877 wait complete. 2024-06-29T14:41:15.381 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:41:15.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.508+0000 7f6d14895640 1 -- 172.21.15.59:0/1958483557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d1008c190 msgr2=0x7f6d100cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.508+0000 7f6d14895640 1 --2- 172.21.15.59:0/1958483557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d1008c190 0x7f6d100cf080 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009830 tx=0x7f6d0002a510 comp rx=0 tx=0).stop 2024-06-29T14:41:15.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.508+0000 7f6d14895640 1 -- 172.21.15.59:0/1958483557 shutdown_connections 2024-06-29T14:41:15.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.508+0000 7f6d14895640 1 --2- 172.21.15.59:0/1958483557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d100cf5c0 0x7f6d100d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.508+0000 7f6d14895640 1 --2- 172.21.15.59:0/1958483557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d1008c190 0x7f6d100cf080 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.508+0000 7f6d14895640 1 -- 172.21.15.59:0/1958483557 >> 172.21.15.59:0/1958483557 conn(0x7f6d100b5610 msgr2=0x7f6d100b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:15.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.508+0000 7f6d14895640 1 -- 172.21.15.59:0/1958483557 shutdown_connections 2024-06-29T14:41:15.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.508+0000 7f6d14895640 1 -- 172.21.15.59:0/1958483557 wait complete. 2024-06-29T14:41:15.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.510+0000 7f6d14895640 1 Processor -- start 2024-06-29T14:41:15.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.510+0000 7f6d14895640 1 -- start start 2024-06-29T14:41:15.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d14895640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d1008c190 0x7f6d100aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d14895640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d100ce950 0x7f6d100cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d14895640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d100cf5c0 0x7f6d1014c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d14895640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6d100b7fa0 con 0x7f6d100cf5c0 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d14895640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d100b7e20 con 0x7f6d100ce950 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d14895640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6d100b8120 con 0x7f6d1008c190 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d100cf5c0 0x7f6d1014c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d100cf5c0 0x7f6d1014c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57890/0 (socket says 172.21.15.59:57890) 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 -- 172.21.15.59:0/3365166555 learned_addr learned my addr 172.21.15.59:0/3365166555 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0e59c640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d1008c190 0x7f6d100aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0dd9b640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d100ce950 0x7f6d100cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 -- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d1008c190 msgr2=0x7f6d100aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d1008c190 0x7f6d100aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 -- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d100ce950 msgr2=0x7f6d100cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d100ce950 0x7f6d100cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 -- 172.21.15.59:0/3365166555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d101e8ea0 con 0x7f6d100cf5c0 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0e59c640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d1008c190 0x7f6d100aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0dd9b640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d100ce950 0x7f6d100cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.512+0000 7f6d0ed9d640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d100cf5c0 0x7f6d1014c770 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800cab0 tx=0x7f6d0800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.513+0000 7f6cfeffd640 1 -- 172.21.15.59:0/3365166555 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08016020 con 0x7f6d100cf5c0 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.513+0000 7f6cfeffd640 1 -- 172.21.15.59:0/3365166555 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d0800ad40 con 0x7f6d100cf5c0 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.513+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d101e9180 con 0x7f6d100cf5c0 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.513+0000 7f6cfeffd640 1 -- 172.21.15.59:0/3365166555 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08012680 con 0x7f6d100cf5c0 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.513+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d101e9740 con 0x7f6d100cf5c0 2024-06-29T14:41:15.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.514+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d100cf080 con 0x7f6d100cf5c0 2024-06-29T14:41:15.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.515+0000 7f6cfeffd640 1 -- 172.21.15.59:0/3365166555 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f6d08019020 con 0x7f6d100cf5c0 2024-06-29T14:41:15.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.515+0000 7f6cfeffd640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6cf00763f0 0x7f6cf0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.515+0000 7f6cfeffd640 1 -- 172.21.15.59:0/3365166555 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7f6d08098160 con 0x7f6d100cf5c0 2024-06-29T14:41:15.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.515+0000 7f6d0e59c640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6cf00763f0 0x7f6cf0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.516+0000 7f6d0e59c640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6cf00763f0 0x7f6cf0078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d00034000 tx=0x7f6d00004730 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:15.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.519+0000 7f6cfeffd640 1 -- 172.21.15.59:0/3365166555 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6d08020020 con 0x7f6d100cf5c0 2024-06-29T14:41:15.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.713+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f6d100b4b30 con 0x7f6d100cf5c0 2024-06-29T14:41:15.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.713+0000 7f6cfeffd640 1 -- 172.21.15.59:0/3365166555 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v51) ==== 59+0+1998 (secure 0 0 0) 0x7f6d08061800 con 0x7f6d100cf5c0 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T14:41:15.712 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T14:41:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T14:41:15.714 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T14:41:15.714 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:41:15.714 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:41:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6cf00763f0 msgr2=0x7f6cf0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6cf00763f0 0x7f6cf0078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d00034000 tx=0x7f6d00004730 comp rx=0 tx=0).stop 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d100cf5c0 msgr2=0x7f6d1014c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d100cf5c0 0x7f6d1014c770 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800cab0 tx=0x7f6d0800cf70 comp rx=0 tx=0).stop 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 shutdown_connections 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d100cf5c0 0x7f6d1014c770 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d100ce950 0x7f6d100cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f6cf00763f0 0x7f6cf0078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 --2- 172.21.15.59:0/3365166555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d1008c190 0x7f6d100aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 >> 172.21.15.59:0/3365166555 conn(0x7f6d100b5610 msgr2=0x7f6d1006d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 shutdown_connections 2024-06-29T14:41:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.716+0000 7f6d14895640 1 -- 172.21.15.59:0/3365166555 wait complete. 2024-06-29T14:41:15.725 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:41:15.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.847+0000 7f91f0822640 1 -- 172.21.15.59:0/53829579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec1414c0 msgr2=0x7f91ec1418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.847+0000 7f91f0822640 1 --2- 172.21.15.59:0/53829579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec1414c0 0x7f91ec1418d0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0b5590 tx=0x7f91e402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.847+0000 7f91f0822640 1 -- 172.21.15.59:0/53829579 shutdown_connections 2024-06-29T14:41:15.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.847+0000 7f91f0822640 1 --2- 172.21.15.59:0/53829579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec1414c0 0x7f91ec1418d0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.847+0000 7f91f0822640 1 --2- 172.21.15.59:0/53829579 >> v2:172.21.15.153:3300/0 conn(0x7f91ec13e9b0 0x7f91ec13edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.847+0000 7f91f0822640 1 -- 172.21.15.59:0/53829579 >> 172.21.15.59:0/53829579 conn(0x7f91ec0b6fb0 msgr2=0x7f91ec0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:15.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.847+0000 7f91f0822640 1 -- 172.21.15.59:0/53829579 shutdown_connections 2024-06-29T14:41:15.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.847+0000 7f91f0822640 1 -- 172.21.15.59:0/53829579 wait complete. 2024-06-29T14:41:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.849+0000 7f91f0822640 1 Processor -- start 2024-06-29T14:41:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.849+0000 7f91f0822640 1 -- start start 2024-06-29T14:41:15.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91f0822640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec13e9b0 0x7f91ec139790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91f0822640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91ec1414c0 0x7f91ec139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91f0822640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec15ceb0 0x7f91ec151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91f0822640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec0b29d0 con 0x7f91ec13e9b0 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91f0822640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec0b2850 con 0x7f91ec15ceb0 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91f0822640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f91ec0b2b50 con 0x7f91ec1414c0 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec13e9b0 0x7f91ec139790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec13e9b0 0x7f91ec139790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57906/0 (socket says 172.21.15.59:57906) 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.850+0000 7f91ea59c640 1 -- 172.21.15.59:0/4021608421 learned_addr learned my addr 172.21.15.59:0/4021608421 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91e9d9b640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91ec1414c0 0x7f91ec139cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91ead9d640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec15ceb0 0x7f91ec151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91ea59c640 1 -- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91ec1414c0 msgr2=0x7f91ec139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91ea59c640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91ec1414c0 0x7f91ec139cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91ea59c640 1 -- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec15ceb0 msgr2=0x7f91ec151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91ea59c640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec15ceb0 0x7f91ec151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91ea59c640 1 -- 172.21.15.59:0/4021608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec13c140 con 0x7f91ec13e9b0 2024-06-29T14:41:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91ead9d640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec15ceb0 0x7f91ec151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91e9d9b640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91ec1414c0 0x7f91ec139cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91ea59c640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec13e9b0 0x7f91ec139790 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00ba80 tx=0x7f91dc00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91e2ffd640 1 -- 172.21.15.59:0/4021608421 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00cc00 con 0x7f91ec13e9b0 2024-06-29T14:41:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91e2ffd640 1 -- 172.21.15.59:0/4021608421 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91dc003970 con 0x7f91ec13e9b0 2024-06-29T14:41:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ec137840 con 0x7f91ec13e9b0 2024-06-29T14:41:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91e2ffd640 1 -- 172.21.15.59:0/4021608421 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc003e00 con 0x7f91ec13e9b0 2024-06-29T14:41:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.851+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ec137e00 con 0x7f91ec13e9b0 2024-06-29T14:41:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.852+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ec0b3000 con 0x7f91ec13e9b0 2024-06-29T14:41:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.853+0000 7f91e2ffd640 1 -- 172.21.15.59:0/4021608421 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f91dc004000 con 0x7f91ec13e9b0 2024-06-29T14:41:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.853+0000 7f91e2ffd640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f91cc0763f0 0x7f91cc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.854+0000 7f91e2ffd640 1 -- 172.21.15.59:0/4021608421 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7f91dc097b50 con 0x7f91ec13e9b0 2024-06-29T14:41:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.854+0000 7f91e9d9b640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f91cc0763f0 0x7f91cc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.854+0000 7f91e9d9b640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f91cc0763f0 0x7f91cc078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f91ec13fb40 tx=0x7f91d4005f10 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:15.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:15.858+0000 7f91e2ffd640 1 -- 172.21.15.59:0/4021608421 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f91dc0610c0 con 0x7f91ec13e9b0 2024-06-29T14:41:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.049+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f91ec13e530 con 0x7f91ec13e9b0 2024-06-29T14:41:16.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.052+0000 7f91e2ffd640 1 -- 172.21.15.59:0/4021608421 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) ==== 122+0+0 (secure 0 0 0) 0x7f91dc0609c0 con 0x7f91ec13e9b0 2024-06-29T14:41:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f91cc0763f0 msgr2=0x7f91cc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f91cc0763f0 0x7f91cc078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f91ec13fb40 tx=0x7f91d4005f10 comp rx=0 tx=0).stop 2024-06-29T14:41:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec13e9b0 msgr2=0x7f91ec139790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec13e9b0 0x7f91ec139790 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00ba80 tx=0x7f91dc00bf40 comp rx=0 tx=0).stop 2024-06-29T14:41:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 shutdown_connections 2024-06-29T14:41:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91ec15ceb0 0x7f91ec151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f91cc0763f0 0x7f91cc078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f91ec1414c0 0x7f91ec139cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 --2- 172.21.15.59:0/4021608421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91ec13e9b0 0x7f91ec139790 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 >> 172.21.15.59:0/4021608421 conn(0x7f91ec0b6fb0 msgr2=0x7f91ec0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:16.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 shutdown_connections 2024-06-29T14:41:16.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.054+0000 7f91f0822640 1 -- 172.21.15.59:0/4021608421 wait complete. 2024-06-29T14:41:16.063 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:41:16.064 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:16.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:16.166 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:41:16.166 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:16.166 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:16.183 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:16.183 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:41:16.183 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:16.201 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:16.202 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:16.202 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:16.239 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:16.239 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:41:16.239 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:16.260 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:16.260 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:41:16.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:16.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 -- 172.21.15.59:0/500683493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d008c190 msgr2=0x7f76d00cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 --2- 172.21.15.59:0/500683493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d008c190 0x7f76d00cf080 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f76d00cfbc0 tx=0x7f76c002aa30 comp rx=0 tx=0).stop 2024-06-29T14:41:16.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 -- 172.21.15.59:0/500683493 shutdown_connections 2024-06-29T14:41:16.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 --2- 172.21.15.59:0/500683493 >> v2:172.21.15.153:3300/0 conn(0x7f76d00ce950 0x7f76d00d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 --2- 172.21.15.59:0/500683493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76d00cf5c0 0x7f76d00d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 --2- 172.21.15.59:0/500683493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d008c190 0x7f76d00cf080 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 -- 172.21.15.59:0/500683493 >> 172.21.15.59:0/500683493 conn(0x7f76d00b5610 msgr2=0x7f76d00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:16.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 -- 172.21.15.59:0/500683493 shutdown_connections 2024-06-29T14:41:16.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.399+0000 7f76d8344640 1 -- 172.21.15.59:0/500683493 wait complete. 2024-06-29T14:41:16.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.401+0000 7f76d8344640 1 Processor -- start 2024-06-29T14:41:16.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.401+0000 7f76d8344640 1 -- start start 2024-06-29T14:41:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d8344640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76d008c190 0x7f76d0152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d8344640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d00ce950 0x7f76d00acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d8344640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d00cf5c0 0x7f76d00ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d8344640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f76d0162440 con 0x7f76d00ce950 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d8344640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76d01622c0 con 0x7f76d00cf5c0 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d8344640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f76d01625c0 con 0x7f76d008c190 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d58df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d00ce950 0x7f76d00acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d58df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d00ce950 0x7f76d00acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57922/0 (socket says 172.21.15.59:57922) 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d58df640 1 -- 172.21.15.59:0/1052151972 learned_addr learned my addr 172.21.15.59:0/1052151972 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d60e0640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76d008c190 0x7f76d0152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.402+0000 7f76d68e1640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d00cf5c0 0x7f76d00ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d58df640 1 -- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76d008c190 msgr2=0x7f76d0152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d58df640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76d008c190 0x7f76d0152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d58df640 1 -- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d00cf5c0 msgr2=0x7f76d00ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d58df640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d00cf5c0 0x7f76d00ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d58df640 1 -- 172.21.15.59:0/1052151972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d01534f0 con 0x7f76d00ce950 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d60e0640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76d008c190 0x7f76d0152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d58df640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d00ce950 0x7f76d00acb00 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f76c400d700 tx=0x7f76c400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76ceffd640 1 -- 172.21.15.59:0/1052151972 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4017070 con 0x7f76d00ce950 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d00ad7f0 con 0x7f76d00ce950 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76ceffd640 1 -- 172.21.15.59:0/1052151972 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76c4005310 con 0x7f76d00ce950 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76ceffd640 1 -- 172.21.15.59:0/1052151972 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4009820 con 0x7f76d00ce950 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.403+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d00addb0 con 0x7f76d00ce950 2024-06-29T14:41:16.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.404+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d00cf080 con 0x7f76d00ce950 2024-06-29T14:41:16.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.405+0000 7f76ceffd640 1 -- 172.21.15.59:0/1052151972 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f76c40099c0 con 0x7f76d00ce950 2024-06-29T14:41:16.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.405+0000 7f76ceffd640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f76bc0763f0 0x7f76bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.405+0000 7f76ceffd640 1 -- 172.21.15.59:0/1052151972 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7f76c4097d40 con 0x7f76d00ce950 2024-06-29T14:41:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.406+0000 7f76d60e0640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f76bc0763f0 0x7f76bc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.406+0000 7f76d60e0640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f76bc0763f0 0x7f76bc078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f76c0035000 tx=0x7f76c0004ab0 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:16.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.410+0000 7f76ceffd640 1 -- 172.21.15.59:0/1052151972 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76d01534f0 con 0x7f76d00ce950 2024-06-29T14:41:16.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.639+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76d00c73f0 con 0x7f76d00ce950 2024-06-29T14:41:16.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.640+0000 7f76ceffd640 1 -- 172.21.15.59:0/1052151972 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+3914 (secure 0 0 0) 0x7f76c40611c0 con 0x7f76d00ce950 2024-06-29T14:41:16.638 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:16.638 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":65,"btime":"2024-06-29T14:41:10:338379+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2473404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2473404084},{"type":"v1","addr":"172.21.15.153:6836","nonce":2473404084}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":7831,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3605263971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3605263971},{"type":"v1","addr":"172.21.15.153:6837","nonce":3605263971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59},{"gid":8352,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/842001425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":842001425},{"type":"v1","addr":"172.21.15.59:6833","nonce":842001425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58},{"gid":8778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3663956172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3663956172},{"type":"v1","addr":"172.21.15.59:6835","nonce":3663956172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":65}],"filesystems":[]} 2024-06-29T14:41:16.639 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 65 2024-06-29T14:41:16.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f76bc0763f0 msgr2=0x7f76bc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f76bc0763f0 0x7f76bc078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f76c0035000 tx=0x7f76c0004ab0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d00ce950 msgr2=0x7f76d00acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d00ce950 0x7f76d00acb00 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f76c400d700 tx=0x7f76c400dbc0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 shutdown_connections 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76d00cf5c0 0x7f76d00ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76d00ce950 0x7f76d00acb00 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f76bc0763f0 0x7f76bc078900 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f76c0035000 tx=0x7f76c0004ab0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 --2- 172.21.15.59:0/1052151972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76d008c190 0x7f76d0152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 >> 172.21.15.59:0/1052151972 conn(0x7f76d00b5610 msgr2=0x7f76d00c8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 shutdown_connections 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.642+0000 7f76d8344640 1 -- 172.21.15.59:0/1052151972 wait complete. 2024-06-29T14:41:16.651 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:41:16.652 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:41:16.652 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:41:16.652 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:16.652 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:41:16.652 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:41:16.652 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:41:16.652 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:16.653 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:41:16.653 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:41:16.653 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:41:16.653 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:16.653 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:41:16.653 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:41:16.653 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:41:16.654 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:16.662 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:41:16.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:41:16.664 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:41:16.664 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:41:16.665 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:41:16.665 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:41:16.666 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:41:16.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:41:16.705 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:41:16.706 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:16.707 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:41:16.707 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:16.708 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:41:16.708 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:16.709 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:41:16.709 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:16.709 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:41:16.796 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:41:16.796 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:41:16.803 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:41:16.803 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:41:16.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.835+0000 7fb315c40640 1 -- 172.21.15.59:0/699074056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 msgr2=0x7fb31015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.835+0000 7fb315c40640 1 --2- 172.21.15.59:0/699074056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 0x7fb31015f9b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb308005fd0 tx=0x7fb30802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.835+0000 7fb315c40640 1 -- 172.21.15.59:0/699074056 shutdown_connections 2024-06-29T14:41:16.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.835+0000 7fb315c40640 1 --2- 172.21.15.59:0/699074056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 0x7fb31015f9b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.835+0000 7fb315c40640 1 --2- 172.21.15.59:0/699074056 >> v2:172.21.15.153:3300/0 conn(0x7fb3100b40a0 0x7fb31015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.835+0000 7fb315c40640 1 -- 172.21.15.59:0/699074056 >> 172.21.15.59:0/699074056 conn(0x7fb3100d3ee0 msgr2=0x7fb3100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:16.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.835+0000 7fb315c40640 1 -- 172.21.15.59:0/699074056 shutdown_connections 2024-06-29T14:41:16.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.835+0000 7fb315c40640 1 -- 172.21.15.59:0/699074056 wait complete. 2024-06-29T14:41:16.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.837+0000 7fb315c40640 1 Processor -- start 2024-06-29T14:41:16.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.837+0000 7fb315c40640 1 -- start start 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb315c40640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3100d23c0 0x7fb31014f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb315c40640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 0x7fb310151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb315c40640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb310152200 0x7fb310152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb315c40640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb310160280 con 0x7fb310152200 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb315c40640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb310160100 con 0x7fb3100d23c0 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb315c40640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb310160400 con 0x7fb31015ee10 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb30ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb310152200 0x7fb310152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 0x7fb310151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb30f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3100d23c0 0x7fb31014f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 0x7fb310151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:47526/0 (socket says 172.21.15.59:47526) 2024-06-29T14:41:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 -- 172.21.15.59:0/1787284981 learned_addr learned my addr 172.21.15.59:0/1787284981 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 -- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3100d23c0 msgr2=0x7fb31014f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3100d23c0 0x7fb31014f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 -- 172.21.15.59:0/1787284981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb310152200 msgr2=0x7fb310152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb310152200 0x7fb310152610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 -- 172.21.15.59:0/1787284981 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3101561d0 con 0x7fb31015ee10 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.839+0000 7fb306ffd640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 0x7fb310151cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009fd0 tx=0x7fb2f800ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.840+0000 7fb30cff9640 1 -- 172.21.15.59:0/1787284981 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f800ccc0 con 0x7fb31015ee10 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.840+0000 7fb30cff9640 1 -- 172.21.15.59:0/1787284981 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2f800ee70 con 0x7fb31015ee10 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.840+0000 7fb30cff9640 1 -- 172.21.15.59:0/1787284981 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f800ce60 con 0x7fb31015ee10 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.840+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb310154c30 con 0x7fb31015ee10 2024-06-29T14:41:16.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.840+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3101f60e0 con 0x7fb31015ee10 2024-06-29T14:41:16.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.841+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb310154a50 con 0x7fb31015ee10 2024-06-29T14:41:16.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.842+0000 7fb30cff9640 1 -- 172.21.15.59:0/1787284981 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fb2f8005260 con 0x7fb31015ee10 2024-06-29T14:41:16.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.842+0000 7fb30cff9640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb2f0076420 0x7fb2f0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.842+0000 7fb30cff9640 1 -- 172.21.15.59:0/1787284981 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(135..135 src has 1..135) ==== 6137+0+0 (secure 0 0 0) 0x7fb2f8098380 con 0x7fb31015ee10 2024-06-29T14:41:16.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.842+0000 7fb30f7fe640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb2f0076420 0x7fb2f0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.843+0000 7fb30f7fe640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb2f0076420 0x7fb2f0078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb300007fd0 tx=0x7fb300001370 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:16.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:16.847+0000 7fb30cff9640 1 -- 172.21.15.59:0/1787284981 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb310154a50 con 0x7fb31015ee10 2024-06-29T14:41:17.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.049+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb31015e8d0 con 0x7fb31015ee10 2024-06-29T14:41:17.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.388+0000 7fb30cff9640 1 -- 172.21.15.59:0/1787284981 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v137) ==== 97+0+0 (secure 0 0 0) 0x7fb2f8061230 con 0x7fb31015ee10 2024-06-29T14:41:17.387 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.390+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb2f0076420 msgr2=0x7fb2f0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.390+0000 7fb315c40640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb2f0076420 0x7fb2f0078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb300007fd0 tx=0x7fb300001370 comp rx=0 tx=0).stop 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.390+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 msgr2=0x7fb310151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.390+0000 7fb315c40640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 0x7fb310151cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009fd0 tx=0x7fb2f800ec20 comp rx=0 tx=0).stop 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.391+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 shutdown_connections 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.391+0000 7fb315c40640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb310152200 0x7fb310152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.391+0000 7fb315c40640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb2f0076420 0x7fb2f0078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.391+0000 7fb315c40640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb31015ee10 0x7fb310151cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.391+0000 7fb315c40640 1 --2- 172.21.15.59:0/1787284981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3100d23c0 0x7fb31014f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.391+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 >> 172.21.15.59:0/1787284981 conn(0x7fb3100d3ee0 msgr2=0x7fb3100d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:17.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.391+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 shutdown_connections 2024-06-29T14:41:17.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.391+0000 7fb315c40640 1 -- 172.21.15.59:0/1787284981 wait complete. 2024-06-29T14:41:17.402 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:41:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.523+0000 7f89c7c47640 1 -- 172.21.15.59:0/346408697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89ac004d40 msgr2=0x7f89ac005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.523+0000 7f89c7c47640 1 --2- 172.21.15.59:0/346408697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89ac004d40 0x7f89ac005150 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009ed0 tx=0x7f89ac005830 comp rx=0 tx=0).stop 2024-06-29T14:41:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.523+0000 7f89c61e4640 1 -- 172.21.15.59:0/346408697 reap_dead start 2024-06-29T14:41:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.523+0000 7f89c7c47640 1 -- 172.21.15.59:0/346408697 shutdown_connections 2024-06-29T14:41:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.523+0000 7f89c7c47640 1 -- 172.21.15.59:0/346408697 >> 172.21.15.59:0/346408697 conn(0x7f89c006c950 msgr2=0x7f89c0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.523+0000 7f89c7c47640 1 -- 172.21.15.59:0/346408697 shutdown_connections 2024-06-29T14:41:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.523+0000 7f89c7c47640 1 -- 172.21.15.59:0/346408697 wait complete. 2024-06-29T14:41:17.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.525+0000 7f89c7c47640 1 Processor -- start 2024-06-29T14:41:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.525+0000 7f89c7c47640 1 -- start start 2024-06-29T14:41:17.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c7c47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89ac004d40 0x7f89c014c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c7c47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c015afd0 0x7f89c013a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c7c47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89c015cdb0 0x7f89c015d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c7c47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f89c00b5a00 con 0x7f89c015afd0 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c7c47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f89c00b5880 con 0x7f89c015cdb0 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c7c47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f89c00b5b80 con 0x7f89ac004d40 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c51e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c015afd0 0x7f89c013a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c51e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c015afd0 0x7f89c013a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57976/0 (socket says 172.21.15.59:57976) 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c51e2640 1 -- 172.21.15.59:0/3905433921 learned_addr learned my addr 172.21.15.59:0/3905433921 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c59e3640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89ac004d40 0x7f89c014c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c61e4640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89c015cdb0 0x7f89c015d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c51e2640 1 -- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89ac004d40 msgr2=0x7f89c014c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c51e2640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89ac004d40 0x7f89c014c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c51e2640 1 -- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89c015cdb0 msgr2=0x7f89c015d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c51e2640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89c015cdb0 0x7f89c015d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.526+0000 7f89c51e2640 1 -- 172.21.15.59:0/3905433921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c014c730 con 0x7f89c015afd0 2024-06-29T14:41:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.527+0000 7f89c51e2640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c015afd0 0x7f89c013a560 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006620 tx=0x7f89b00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.527+0000 7f89b67fc640 1 -- 172.21.15.59:0/3905433921 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0011700 con 0x7f89c015afd0 2024-06-29T14:41:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.527+0000 7f89b67fc640 1 -- 172.21.15.59:0/3905433921 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89b0016020 con 0x7f89c015afd0 2024-06-29T14:41:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.527+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c01394c0 con 0x7f89c015afd0 2024-06-29T14:41:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.527+0000 7f89b67fc640 1 -- 172.21.15.59:0/3905433921 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0011b40 con 0x7f89c015afd0 2024-06-29T14:41:17.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.527+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c0139a80 con 0x7f89c015afd0 2024-06-29T14:41:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.528+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c01458a0 con 0x7f89c015afd0 2024-06-29T14:41:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.529+0000 7f89b67fc640 1 -- 172.21.15.59:0/3905433921 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f89b0011ce0 con 0x7f89c015afd0 2024-06-29T14:41:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.529+0000 7f89b67fc640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f89a40763f0 0x7f89a4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.530+0000 7f89b67fc640 1 -- 172.21.15.59:0/3905433921 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5277+0+0 (secure 0 0 0) 0x7f89b00982e0 con 0x7f89c015afd0 2024-06-29T14:41:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.530+0000 7f89c59e3640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f89a40763f0 0x7f89a4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.530+0000 7f89c59e3640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f89a40763f0 0x7f89a4078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89bc009c50 tx=0x7f89bc005f90 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:17.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.534+0000 7f89b67fc640 1 -- 172.21.15.59:0/3905433921 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f89b0020020 con 0x7f89c015afd0 2024-06-29T14:41:17.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.801+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f89c0139250 con 0x7f89c015afd0 2024-06-29T14:41:17.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.802+0000 7f89b67fc640 1 -- 172.21.15.59:0/3905433921 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) ==== 80+0+5304 (secure 0 0 0) 0x7f89b0061b50 con 0x7f89c015afd0 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.801 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T14:41:17.802 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:17.803 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.804 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:17.805 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T14:41:17.807 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:17.808 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.809 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.810 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:41:17.811 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.812 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:41:17.813 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:17.814 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.814 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:17.814 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T14:41:17.814 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:41:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.803+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f89a40763f0 msgr2=0x7f89a4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.803+0000 7f89c7c47640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f89a40763f0 0x7f89a4078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89bc009c50 tx=0x7f89bc005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.803+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c015afd0 msgr2=0x7f89c013a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.803+0000 7f89c7c47640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c015afd0 0x7f89c013a560 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006620 tx=0x7f89b00039c0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.804+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 shutdown_connections 2024-06-29T14:41:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.804+0000 7f89c7c47640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89c015cdb0 0x7f89c015d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.804+0000 7f89c7c47640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c015afd0 0x7f89c013a560 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.804+0000 7f89c7c47640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f89a40763f0 0x7f89a4078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.804+0000 7f89c7c47640 1 --2- 172.21.15.59:0/3905433921 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89ac004d40 0x7f89c014c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.804+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 >> 172.21.15.59:0/3905433921 conn(0x7f89c006c950 msgr2=0x7f89c01433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.804+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 shutdown_connections 2024-06-29T14:41:17.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.804+0000 7f89c7c47640 1 -- 172.21.15.59:0/3905433921 wait complete. 2024-06-29T14:41:17.817 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:17.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.936+0000 7f5ac3ece640 1 -- 172.21.15.59:0/2288533462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc15ceb0 msgr2=0x7f5abc0b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.936+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/2288533462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc15ceb0 0x7f5abc0b56d0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f5aac005fd0 tx=0x7f5aac02a970 comp rx=0 tx=0).stop 2024-06-29T14:41:17.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.936+0000 7f5ac3ece640 1 -- 172.21.15.59:0/2288533462 shutdown_connections 2024-06-29T14:41:17.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.936+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/2288533462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5abc141bd0 0x7f5abc13edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.936+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/2288533462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc15ceb0 0x7f5abc0b56d0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.936+0000 7f5ac3ece640 1 -- 172.21.15.59:0/2288533462 >> 172.21.15.59:0/2288533462 conn(0x7f5abc0b6fb0 msgr2=0x7f5abc0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:17.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.936+0000 7f5ac3ece640 1 -- 172.21.15.59:0/2288533462 shutdown_connections 2024-06-29T14:41:17.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.936+0000 7f5ac3ece640 1 -- 172.21.15.59:0/2288533462 wait complete. 2024-06-29T14:41:17.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.938+0000 7f5ac3ece640 1 Processor -- start 2024-06-29T14:41:17.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.938+0000 7f5ac3ece640 1 -- start start 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.939+0000 7f5ac3ece640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5abc13e9b0 0x7f5abc1534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.939+0000 7f5ac3ece640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5abc15ceb0 0x7f5abc15b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.939+0000 7f5ac3ece640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc1f5b30 0x7f5abc151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.939+0000 7f5ac3ece640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5abc0b2a00 con 0x7f5abc1f5b30 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.939+0000 7f5ac3ece640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5abc0b2880 con 0x7f5abc13e9b0 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.939+0000 7f5ac3ece640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5abc0b2b80 con 0x7f5abc15ceb0 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.939+0000 7f5ac246b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc1f5b30 0x7f5abc151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac246b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc1f5b30 0x7f5abc151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57990/0 (socket says 172.21.15.59:57990) 2024-06-29T14:41:17.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac246b640 1 -- 172.21.15.59:0/3381309641 learned_addr learned my addr 172.21.15.59:0/3381309641 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:17.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac1c6a640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5abc13e9b0 0x7f5abc1534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac1469640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5abc15ceb0 0x7f5abc15b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac246b640 1 -- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5abc15ceb0 msgr2=0x7f5abc15b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac246b640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5abc15ceb0 0x7f5abc15b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac246b640 1 -- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5abc13e9b0 msgr2=0x7f5abc1534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac246b640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5abc13e9b0 0x7f5abc1534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac246b640 1 -- 172.21.15.59:0/3381309641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5abc153a20 con 0x7f5abc1f5b30 2024-06-29T14:41:17.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac1469640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5abc15ceb0 0x7f5abc15b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:17.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac246b640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc1f5b30 0x7f5abc151b80 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400d650 tx=0x7f5ab400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:17.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5aba7fc640 1 -- 172.21.15.59:0/3381309641 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4017070 con 0x7f5abc1f5b30 2024-06-29T14:41:17.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5aba7fc640 1 -- 172.21.15.59:0/3381309641 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ab4005310 con 0x7f5abc1f5b30 2024-06-29T14:41:17.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5abc1f6760 con 0x7f5abc1f5b30 2024-06-29T14:41:17.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5aba7fc640 1 -- 172.21.15.59:0/3381309641 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4009820 con 0x7f5abc1f5b30 2024-06-29T14:41:17.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.940+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5abc1510a0 con 0x7f5abc1f5b30 2024-06-29T14:41:17.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.941+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5abc1f64d0 con 0x7f5abc1f5b30 2024-06-29T14:41:17.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.942+0000 7f5aba7fc640 1 -- 172.21.15.59:0/3381309641 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f5ab40099c0 con 0x7f5abc1f5b30 2024-06-29T14:41:17.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.943+0000 7f5aba7fc640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5aa80763f0 0x7f5aa8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.943+0000 7f5aba7fc640 1 -- 172.21.15.59:0/3381309641 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5277+0+0 (secure 0 0 0) 0x7f5ab4097bc0 con 0x7f5abc1f5b30 2024-06-29T14:41:17.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.943+0000 7f5ac1c6a640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5aa80763f0 0x7f5aa8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.944+0000 7f5ac1c6a640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5aa80763f0 0x7f5aa8078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5aac035000 tx=0x7f5aac004ab0 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:17.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:17.946+0000 7f5aba7fc640 1 -- 172.21.15.59:0/3381309641 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ab4060eb0 con 0x7f5abc1f5b30 2024-06-29T14:41:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.138+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5abc13edc0 con 0x7f5abc1f5b30 2024-06-29T14:41:18.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.139+0000 7f5aba7fc640 1 -- 172.21.15.59:0/3381309641 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+13230 (secure 0 0 0) 0x7f5ab40651b0 con 0x7f5abc1f5b30 2024-06-29T14:41:18.137 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:18.137 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":139,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:17.394654+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:18.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5aa80763f0 msgr2=0x7f5aa8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5aa80763f0 0x7f5aa8078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5aac035000 tx=0x7f5aac004ab0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc1f5b30 msgr2=0x7f5abc151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc1f5b30 0x7f5abc151b80 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400d650 tx=0x7f5ab400db10 comp rx=0 tx=0).stop 2024-06-29T14:41:18.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 shutdown_connections 2024-06-29T14:41:18.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5abc1f5b30 0x7f5abc151b80 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f5aa80763f0 0x7f5aa8078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5abc15ceb0 0x7f5abc15b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 --2- 172.21.15.59:0/3381309641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5abc13e9b0 0x7f5abc1534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 >> 172.21.15.59:0/3381309641 conn(0x7f5abc0b6fb0 msgr2=0x7f5abc15de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 shutdown_connections 2024-06-29T14:41:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.141+0000 7f5ac3ece640 1 -- 172.21.15.59:0/3381309641 wait complete. 2024-06-29T14:41:18.150 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:18.151 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:18.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.274+0000 7f53a6355640 1 -- 172.21.15.59:0/1372931739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5388003a70 msgr2=0x7f5388003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.274+0000 7f53a6355640 1 --2- 172.21.15.59:0/1372931739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5388003a70 0x7f5388003ee0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f5390009b20 tx=0x7f5388005830 comp rx=0 tx=0).stop 2024-06-29T14:41:18.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.274+0000 7f53a48f2640 1 -- 172.21.15.59:0/1372931739 reap_dead start 2024-06-29T14:41:18.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.274+0000 7f53a6355640 1 -- 172.21.15.59:0/1372931739 shutdown_connections 2024-06-29T14:41:18.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.274+0000 7f53a6355640 1 -- 172.21.15.59:0/1372931739 >> 172.21.15.59:0/1372931739 conn(0x7f53a00d3ee0 msgr2=0x7f53a00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.274+0000 7f53a6355640 1 -- 172.21.15.59:0/1372931739 shutdown_connections 2024-06-29T14:41:18.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.274+0000 7f53a6355640 1 -- 172.21.15.59:0/1372931739 wait complete. 2024-06-29T14:41:18.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.276+0000 7f53a6355640 1 Processor -- start 2024-06-29T14:41:18.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.276+0000 7f53a6355640 1 -- start start 2024-06-29T14:41:18.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.277+0000 7f53a6355640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5388003a70 0x7f53a015b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.277+0000 7f53a6355640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a01f8360 0x7f53a01ea860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.277+0000 7f53a6355640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53a01eada0 0x7f53a0158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.277+0000 7f53a6355640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f53a015ea50 con 0x7f53a01eada0 2024-06-29T14:41:18.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.277+0000 7f53a6355640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53a015e8d0 con 0x7f53a01f8360 2024-06-29T14:41:18.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.277+0000 7f53a6355640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f53a015ebd0 con 0x7f5388003a70 2024-06-29T14:41:18.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53a01eada0 0x7f53a0158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53a01eada0 0x7f53a0158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58016/0 (socket says 172.21.15.59:58016) 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 -- 172.21.15.59:0/902818789 learned_addr learned my addr 172.21.15.59:0/902818789 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f539ffff640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5388003a70 0x7f53a015b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f539f7fe640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a01f8360 0x7f53a01ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 -- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5388003a70 msgr2=0x7f53a015b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5388003a70 0x7f53a015b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 -- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a01f8360 msgr2=0x7f53a01ea860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a01f8360 0x7f53a01ea860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 -- 172.21.15.59:0/902818789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a01eb1b0 con 0x7f53a01eada0 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f539ffff640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5388003a70 0x7f53a015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:18.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f539f7fe640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a01f8360 0x7f53a01ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:18.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a48f2640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53a01eada0 0x7f53a0158eb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f53a00d2920 tx=0x7f5398006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f539cff9640 1 -- 172.21.15.59:0/902818789 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539800fa00 con 0x7f53a01eada0 2024-06-29T14:41:18.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f539cff9640 1 -- 172.21.15.59:0/902818789 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f539800fba0 con 0x7f53a01eada0 2024-06-29T14:41:18.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a015ba30 con 0x7f53a01eada0 2024-06-29T14:41:18.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f539cff9640 1 -- 172.21.15.59:0/902818789 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398011430 con 0x7f53a01eada0 2024-06-29T14:41:18.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.278+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a015bff0 con 0x7f53a01eada0 2024-06-29T14:41:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.279+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a015b7a0 con 0x7f53a01eada0 2024-06-29T14:41:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.280+0000 7f539cff9640 1 -- 172.21.15.59:0/902818789 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f53980115d0 con 0x7f53a01eada0 2024-06-29T14:41:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.281+0000 7f539cff9640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f538c0763f0 0x7f538c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.281+0000 7f539cff9640 1 -- 172.21.15.59:0/902818789 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5277+0+0 (secure 0 0 0) 0x7f53980981f0 con 0x7f53a01eada0 2024-06-29T14:41:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.281+0000 7f539ffff640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f538c0763f0 0x7f538c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.282+0000 7f539ffff640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f538c0763f0 0x7f538c078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5390035000 tx=0x7f539002d040 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.285+0000 7f539cff9640 1 -- 172.21.15.59:0/902818789 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f53a01eb1b0 con 0x7f53a01eada0 2024-06-29T14:41:18.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.481+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53a00b3990 con 0x7f53a01eada0 2024-06-29T14:41:18.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.481+0000 7f539cff9640 1 -- 172.21.15.59:0/902818789 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v139) ==== 93+0+10 (secure 0 0 0) 0x7f5398015070 con 0x7f53a01eada0 2024-06-29T14:41:18.480 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f538c0763f0 msgr2=0x7f538c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f538c0763f0 0x7f538c078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5390035000 tx=0x7f539002d040 comp rx=0 tx=0).stop 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53a01eada0 msgr2=0x7f53a0158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53a01eada0 0x7f53a0158eb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f53a00d2920 tx=0x7f5398006720 comp rx=0 tx=0).stop 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 shutdown_connections 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53a01eada0 0x7f53a0158eb0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a01f8360 0x7f53a01ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f538c0763f0 0x7f538c078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 --2- 172.21.15.59:0/902818789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5388003a70 0x7f53a015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 >> 172.21.15.59:0/902818789 conn(0x7f53a00d3ee0 msgr2=0x7f53a006d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 shutdown_connections 2024-06-29T14:41:18.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.484+0000 7f53a6355640 1 -- 172.21.15.59:0/902818789 wait complete. 2024-06-29T14:41:18.493 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:41:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.613+0000 7f65aebc7640 1 -- 172.21.15.59:0/3237739662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65a808c190 msgr2=0x7f65a80cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.613+0000 7f65aebc7640 1 --2- 172.21.15.59:0/3237739662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65a808c190 0x7f65a80cf080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6598005fd0 tx=0x7f659802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.613+0000 7f65aebc7640 1 -- 172.21.15.59:0/3237739662 shutdown_connections 2024-06-29T14:41:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.613+0000 7f65aebc7640 1 --2- 172.21.15.59:0/3237739662 >> v2:172.21.15.153:3300/0 conn(0x7f65a80cf5c0 0x7f65a80d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.613+0000 7f65aebc7640 1 --2- 172.21.15.59:0/3237739662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65a808c190 0x7f65a80cf080 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.613+0000 7f65aebc7640 1 -- 172.21.15.59:0/3237739662 >> 172.21.15.59:0/3237739662 conn(0x7f65a80b5610 msgr2=0x7f65a80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.613+0000 7f65aebc7640 1 -- 172.21.15.59:0/3237739662 shutdown_connections 2024-06-29T14:41:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.613+0000 7f65aebc7640 1 -- 172.21.15.59:0/3237739662 wait complete. 2024-06-29T14:41:18.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.615+0000 7f65aebc7640 1 Processor -- start 2024-06-29T14:41:18.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.615+0000 7f65aebc7640 1 -- start start 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65aebc7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65a808c190 0x7f65a8150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65aebc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65a80ce950 0x7f65a81f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65aebc7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65a80cf5c0 0x7f65a81f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65aebc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f65a80b7fa0 con 0x7f65a80ce950 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65aebc7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65a80b7e20 con 0x7f65a808c190 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65aebc7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f65a80b8120 con 0x7f65a80cf5c0 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65a80ce950 0x7f65a81f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65a80ce950 0x7f65a81f6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58028/0 (socket says 172.21.15.59:58028) 2024-06-29T14:41:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 -- 172.21.15.59:0/2480489969 learned_addr learned my addr 172.21.15.59:0/2480489969 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65ac963640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65a808c190 0x7f65a8150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65ad164640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65a80cf5c0 0x7f65a81f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 -- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65a80cf5c0 msgr2=0x7f65a81f7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65a80cf5c0 0x7f65a81f7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 -- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65a808c190 msgr2=0x7f65a8150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65a808c190 0x7f65a8150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 -- 172.21.15.59:0/2480489969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a81f5d90 con 0x7f65a80ce950 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a7fff640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65a80ce950 0x7f65a81f6e30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f659000ece0 tx=0x7f659000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.617+0000 7f65a57fa640 1 -- 172.21.15.59:0/2480489969 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590003b70 con 0x7f65a80ce950 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.618+0000 7f65a57fa640 1 -- 172.21.15.59:0/2480489969 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6590003d10 con 0x7f65a80ce950 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.618+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a81f6070 con 0x7f65a80ce950 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.618+0000 7f65a57fa640 1 -- 172.21.15.59:0/2480489969 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590004280 con 0x7f65a80ce950 2024-06-29T14:41:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.618+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a81f6630 con 0x7f65a80ce950 2024-06-29T14:41:18.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.619+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a80d2f00 con 0x7f65a80ce950 2024-06-29T14:41:18.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.619+0000 7f65a57fa640 1 -- 172.21.15.59:0/2480489969 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f6590004420 con 0x7f65a80ce950 2024-06-29T14:41:18.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.620+0000 7f65a57fa640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f65880763f0 0x7f6588078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.620+0000 7f65a57fa640 1 -- 172.21.15.59:0/2480489969 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5277+0+0 (secure 0 0 0) 0x7f65900983f0 con 0x7f65a80ce950 2024-06-29T14:41:18.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.620+0000 7f65ac963640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f65880763f0 0x7f6588078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.621+0000 7f65ac963640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f65880763f0 0x7f6588078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6598033000 tx=0x7f6598004730 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.624+0000 7f65a57fa640 1 -- 172.21.15.59:0/2480489969 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f65a80d2f00 con 0x7f65a80ce950 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.822+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f65a81f5b20 con 0x7f65a80ce950 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.823+0000 7f65a57fa640 1 -- 172.21.15.59:0/2480489969 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v139) ==== 84+0+65 (secure 0 0 0) 0x7f6590061600 con 0x7f65a80ce950 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:18.821 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f65880763f0 msgr2=0x7f6588078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f65880763f0 0x7f6588078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6598033000 tx=0x7f6598004730 comp rx=0 tx=0).stop 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65a80ce950 msgr2=0x7f65a81f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65a80ce950 0x7f65a81f6e30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f659000ece0 tx=0x7f659000c6a0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 shutdown_connections 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f65880763f0 0x7f6588078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65a80cf5c0 0x7f65a81f7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65a80ce950 0x7f65a81f6e30 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 --2- 172.21.15.59:0/2480489969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65a808c190 0x7f65a8150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 >> 172.21.15.59:0/2480489969 conn(0x7f65a80b5610 msgr2=0x7f65a80c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 shutdown_connections 2024-06-29T14:41:18.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.825+0000 7f65aebc7640 1 -- 172.21.15.59:0/2480489969 wait complete. 2024-06-29T14:41:18.834 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:41:18.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:41:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/2386862887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f28145650 msgr2=0x7f4f280b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/2386862887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f28145650 0x7f4f280b5cb0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4f280d0ea0 tx=0x7f4f2402a560 comp rx=0 tx=0).stop 2024-06-29T14:41:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/2386862887 shutdown_connections 2024-06-29T14:41:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/2386862887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f2815fd30 0x7f4f2815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/2386862887 >> v2:172.21.15.153:3300/0 conn(0x7f4f280d1130 0x7f4f2815f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/2386862887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f28145650 0x7f4f280b5cb0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/2386862887 >> 172.21.15.59:0/2386862887 conn(0x7f4f280b6ed0 msgr2=0x7f4f280b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/2386862887 shutdown_connections 2024-06-29T14:41:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.958+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/2386862887 wait complete. 2024-06-29T14:41:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.960+0000 7f4f2f8e4640 1 Processor -- start 2024-06-29T14:41:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.960+0000 7f4f2f8e4640 1 -- start start 2024-06-29T14:41:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2f8e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280d1130 0x7f4f281f1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2f8e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f28145650 0x7f4f280c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2f8e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f2815fd30 0x7f4f280cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2f8e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2815cad0 con 0x7f4f2815fd30 2024-06-29T14:41:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2f8e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2815c950 con 0x7f4f28145650 2024-06-29T14:41:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2f8e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4f2815cc50 con 0x7f4f280d1130 2024-06-29T14:41:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2de81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f2815fd30 0x7f4f280cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2de81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f2815fd30 0x7f4f280cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58044/0 (socket says 172.21.15.59:58044) 2024-06-29T14:41:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2de81640 1 -- 172.21.15.59:0/3173476037 learned_addr learned my addr 172.21.15.59:0/3173476037 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2d680640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280d1130 0x7f4f281f1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.961+0000 7f4f2ce7f640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f28145650 0x7f4f280c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2de81640 1 -- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280d1130 msgr2=0x7f4f281f1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2de81640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280d1130 0x7f4f281f1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2de81640 1 -- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f28145650 msgr2=0x7f4f280c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2de81640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f28145650 0x7f4f280c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2de81640 1 -- 172.21.15.59:0/3173476037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f280c8e20 con 0x7f4f2815fd30 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2d680640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280d1130 0x7f4f281f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2ce7f640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f28145650 0x7f4f280c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2de81640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f2815fd30 0x7f4f280cc960 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000cab0 tx=0x7f4f2000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f1dffb640 1 -- 172.21.15.59:0/3173476037 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2000ae20 con 0x7f4f2815fd30 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f1dffb640 1 -- 172.21.15.59:0/3173476037 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f20003a60 con 0x7f4f2815fd30 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f280ca320 con 0x7f4f2815fd30 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f1dffb640 1 -- 172.21.15.59:0/3173476037 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20004030 con 0x7f4f2815fd30 2024-06-29T14:41:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.962+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f280ca8e0 con 0x7f4f2815fd30 2024-06-29T14:41:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.963+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f2813e650 con 0x7f4f2815fd30 2024-06-29T14:41:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.964+0000 7f4f1dffb640 1 -- 172.21.15.59:0/3173476037 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f4f200041d0 con 0x7f4f2815fd30 2024-06-29T14:41:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.964+0000 7f4f1dffb640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4f0c076420 0x7f4f0c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.965+0000 7f4f1dffb640 1 -- 172.21.15.59:0/3173476037 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5277+0+0 (secure 0 0 0) 0x7f4f20097f10 con 0x7f4f2815fd30 2024-06-29T14:41:18.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.965+0000 7f4f2d680640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4f0c076420 0x7f4f0c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.966+0000 7f4f2d680640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4f0c076420 0x7f4f0c078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4f24033000 tx=0x7f4f24004730 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:18.969+0000 7f4f1dffb640 1 -- 172.21.15.59:0/3173476037 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f20014070 con 0x7f4f2815fd30 2024-06-29T14:41:19.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.165+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4f280b5cb0 con 0x7f4f2815fd30 2024-06-29T14:41:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.390+0000 7f4f1dffb640 1 -- 172.21.15.59:0/3173476037 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v140) ==== 138+0+0 (secure 0 0 0) 0x7f4f20061800 con 0x7f4f2815fd30 2024-06-29T14:41:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4f0c076420 msgr2=0x7f4f0c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4f0c076420 0x7f4f0c078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4f24033000 tx=0x7f4f24004730 comp rx=0 tx=0).stop 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f2815fd30 msgr2=0x7f4f280cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f2815fd30 0x7f4f280cc960 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000cab0 tx=0x7f4f2000cf70 comp rx=0 tx=0).stop 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 shutdown_connections 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f2815fd30 0x7f4f280cc960 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f28145650 0x7f4f280c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4f0c076420 0x7f4f0c078930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 --2- 172.21.15.59:0/3173476037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280d1130 0x7f4f281f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.392+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 >> 172.21.15.59:0/3173476037 conn(0x7f4f280b6ed0 msgr2=0x7f4f2815d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.393+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 shutdown_connections 2024-06-29T14:41:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.393+0000 7f4f2f8e4640 1 -- 172.21.15.59:0/3173476037 wait complete. 2024-06-29T14:41:19.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:41:19.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.534+0000 7fc0424a4640 1 -- 172.21.15.59:0/558324799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 msgr2=0x7fc024004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.534+0000 7fc0424a4640 1 --2- 172.21.15.59:0/558324799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 0x7fc024004c10 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc0300016c0 tx=0x7fc024005830 comp rx=0 tx=0).stop 2024-06-29T14:41:19.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.534+0000 7fc040a41640 1 -- 172.21.15.59:0/558324799 reap_dead start 2024-06-29T14:41:19.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.534+0000 7fc0424a4640 1 -- 172.21.15.59:0/558324799 shutdown_connections 2024-06-29T14:41:19.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.534+0000 7fc0424a4640 1 -- 172.21.15.59:0/558324799 >> 172.21.15.59:0/558324799 conn(0x7fc03c073ef0 msgr2=0x7fc03c074300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:19.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.534+0000 7fc0424a4640 1 -- 172.21.15.59:0/558324799 shutdown_connections 2024-06-29T14:41:19.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.534+0000 7fc0424a4640 1 -- 172.21.15.59:0/558324799 wait complete. 2024-06-29T14:41:19.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.535+0000 7fc0424a4640 1 Processor -- start 2024-06-29T14:41:19.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.536+0000 7fc0424a4640 1 -- start start 2024-06-29T14:41:19.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc0424a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 0x7fc03c06ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc0424a4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc03c1453b0 0x7fc03c06f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc0424a4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc03c143150 0x7fc03c160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc0424a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c160190 con 0x7fc024004800 2024-06-29T14:41:19.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc0424a4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c160010 con 0x7fc03c143150 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc0424a4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc03c164a30 con 0x7fc03c1453b0 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 0x7fc03c06ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 0x7fc03c06ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58066/0 (socket says 172.21.15.59:58066) 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03bfff640 1 -- 172.21.15.59:0/1894653629 learned_addr learned my addr 172.21.15.59:0/1894653629 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03b7fe640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc03c1453b0 0x7fc03c06f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc040a41640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc03c143150 0x7fc03c160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03bfff640 1 -- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc03c1453b0 msgr2=0x7fc03c06f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03bfff640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc03c1453b0 0x7fc03c06f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03bfff640 1 -- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc03c143150 msgr2=0x7fc03c160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03bfff640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc03c143150 0x7fc03c160aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc03bfff640 1 -- 172.21.15.59:0/1894653629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03c06dcb0 con 0x7fc024004800 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.537+0000 7fc040a41640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc03c143150 0x7fc03c160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.538+0000 7fc03bfff640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 0x7fc03c06ea70 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0066e0 tx=0x7fc02c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:19.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.538+0000 7fc038ff9640 1 -- 172.21.15.59:0/1894653629 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c00f760 con 0x7fc024004800 2024-06-29T14:41:19.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.538+0000 7fc038ff9640 1 -- 172.21.15.59:0/1894653629 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc02c016020 con 0x7fc024004800 2024-06-29T14:41:19.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.538+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03c06df90 con 0x7fc024004800 2024-06-29T14:41:19.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.538+0000 7fc038ff9640 1 -- 172.21.15.59:0/1894653629 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c00fc00 con 0x7fc024004800 2024-06-29T14:41:19.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.538+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03c06e550 con 0x7fc024004800 2024-06-29T14:41:19.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.539+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc03c0b1e10 con 0x7fc024004800 2024-06-29T14:41:19.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.541+0000 7fc038ff9640 1 -- 172.21.15.59:0/1894653629 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fc02c025440 con 0x7fc024004800 2024-06-29T14:41:19.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.541+0000 7fc038ff9640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc028076420 0x7fc028078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.541+0000 7fc038ff9640 1 -- 172.21.15.59:0/1894653629 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5647+0+0 (secure 0 0 0) 0x7fc02c098a60 con 0x7fc024004800 2024-06-29T14:41:19.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.541+0000 7fc03b7fe640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc028076420 0x7fc028078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.542+0000 7fc03b7fe640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc028076420 0x7fc028078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc030032000 tx=0x7fc030001190 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:19.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.545+0000 7fc038ff9640 1 -- 172.21.15.59:0/1894653629 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc02c020020 con 0x7fc024004800 2024-06-29T14:41:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:19.738+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc03c0cde70 con 0x7fc024004800 2024-06-29T14:41:20.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.392+0000 7fc038ff9640 1 -- 172.21.15.59:0/1894653629 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v141) ==== 170+0+0 (secure 0 0 0) 0x7fc02c0621d0 con 0x7fc024004800 2024-06-29T14:41:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:41:20.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.395+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc028076420 msgr2=0x7fc028078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.395+0000 7fc0424a4640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc028076420 0x7fc028078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc030032000 tx=0x7fc030001190 comp rx=0 tx=0).stop 2024-06-29T14:41:20.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.395+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 msgr2=0x7fc03c06ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.395+0000 7fc0424a4640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 0x7fc03c06ea70 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0066e0 tx=0x7fc02c0039c0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.396+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 shutdown_connections 2024-06-29T14:41:20.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.396+0000 7fc0424a4640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc03c143150 0x7fc03c160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.396+0000 7fc0424a4640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc028076420 0x7fc028078930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.396+0000 7fc0424a4640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc03c1453b0 0x7fc03c06f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.396+0000 7fc0424a4640 1 --2- 172.21.15.59:0/1894653629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc024004800 0x7fc03c06ea70 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.396+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 >> 172.21.15.59:0/1894653629 conn(0x7fc03c073ef0 msgr2=0x7fc03c143c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:20.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.396+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 shutdown_connections 2024-06-29T14:41:20.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.396+0000 7fc0424a4640 1 -- 172.21.15.59:0/1894653629 wait complete. 2024-06-29T14:41:20.420 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:41:20.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.546+0000 7fd2cdd24640 1 -- 172.21.15.59:0/828278647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 msgr2=0x7fd2c80d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.546+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/828278647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 0x7fd2c80d2130 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0005fd0 tx=0x7fd2c002aa90 comp rx=0 tx=0).stop 2024-06-29T14:41:20.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.546+0000 7fd2cdd24640 1 -- 172.21.15.59:0/828278647 shutdown_connections 2024-06-29T14:41:20.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.546+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/828278647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 0x7fd2c80d2130 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.546+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/828278647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c808c190 0x7fd2c80cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.546+0000 7fd2cdd24640 1 -- 172.21.15.59:0/828278647 >> 172.21.15.59:0/828278647 conn(0x7fd2c80b5610 msgr2=0x7fd2c80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:20.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.546+0000 7fd2cdd24640 1 -- 172.21.15.59:0/828278647 shutdown_connections 2024-06-29T14:41:20.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.546+0000 7fd2cdd24640 1 -- 172.21.15.59:0/828278647 wait complete. 2024-06-29T14:41:20.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.548+0000 7fd2cdd24640 1 Processor -- start 2024-06-29T14:41:20.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.548+0000 7fd2cdd24640 1 -- start start 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.549+0000 7fd2cdd24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c808c190 0x7fd2c815edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.549+0000 7fd2cdd24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 0x7fd2c815f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.549+0000 7fd2cdd24640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c80cf5c0 0x7fd2c80aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.549+0000 7fd2cdd24640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c80b7fa0 con 0x7fd2c80ce950 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.549+0000 7fd2cdd24640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c80b7e20 con 0x7fd2c808c190 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.549+0000 7fd2cdd24640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c80b8120 con 0x7fd2c80cf5c0 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 0x7fd2c815f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 0x7fd2c815f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58092/0 (socket says 172.21.15.59:58092) 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 -- 172.21.15.59:0/859142200 learned_addr learned my addr 172.21.15.59:0/859142200 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c7fff640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c80cf5c0 0x7fd2c80aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c77fe640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c808c190 0x7fd2c815edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:20.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 -- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c80cf5c0 msgr2=0x7fd2c80aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c80cf5c0 0x7fd2c80aa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 -- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c808c190 msgr2=0x7fd2c815edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c808c190 0x7fd2c815edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 -- 172.21.15.59:0/859142200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c815fad0 con 0x7fd2c80ce950 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2c6ffd640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 0x7fd2c815f300 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00c3c0 tx=0x7fd2bc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2affff640 1 -- 172.21.15.59:0/859142200 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc017070 con 0x7fd2c80ce950 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2affff640 1 -- 172.21.15.59:0/859142200 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2bc0053c0 con 0x7fd2c80ce950 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c815fdb0 con 0x7fd2c80ce950 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2affff640 1 -- 172.21.15.59:0/859142200 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc003850 con 0x7fd2c80ce950 2024-06-29T14:41:20.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.550+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c8160370 con 0x7fd2c80ce950 2024-06-29T14:41:20.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.551+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c80cf080 con 0x7fd2c80ce950 2024-06-29T14:41:20.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.552+0000 7fd2affff640 1 -- 172.21.15.59:0/859142200 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fd2bc01a070 con 0x7fd2c80ce950 2024-06-29T14:41:20.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.553+0000 7fd2affff640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fd2b40763f0 0x7fd2b4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:20.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.553+0000 7fd2affff640 1 -- 172.21.15.59:0/859142200 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 6029+0+0 (secure 0 0 0) 0x7fd2bc097280 con 0x7fd2c80ce950 2024-06-29T14:41:20.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.553+0000 7fd2c77fe640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fd2b40763f0 0x7fd2b4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:20.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.554+0000 7fd2c77fe640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fd2b40763f0 0x7fd2b4078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8005fd0 tx=0x7fd2b800a250 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:20.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.557+0000 7fd2affff640 1 -- 172.21.15.59:0/859142200 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd2bc021020 con 0x7fd2c80ce950 2024-06-29T14:41:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:20.793+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd2c815f860 con 0x7fd2c80ce950 2024-06-29T14:41:21.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.416+0000 7fd2affff640 1 -- 172.21.15.59:0/859142200 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v68) ==== 174+0+0 (secure 0 0 0) 0x7fd2bc060290 con 0x7fd2c80ce950 2024-06-29T14:41:21.415 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 17 and data pool 18 2024-06-29T14:41:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.432+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fd2b40763f0 msgr2=0x7fd2b4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.432+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fd2b40763f0 0x7fd2b4078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8005fd0 tx=0x7fd2b800a250 comp rx=0 tx=0).stop 2024-06-29T14:41:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.432+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 msgr2=0x7fd2c815f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.432+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 0x7fd2c815f300 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00c3c0 tx=0x7fd2bc00c880 comp rx=0 tx=0).stop 2024-06-29T14:41:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.433+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 shutdown_connections 2024-06-29T14:41:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.433+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fd2b40763f0 0x7fd2b4078900 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8005fd0 tx=0x7fd2b800a250 comp rx=0 tx=0).stop 2024-06-29T14:41:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.433+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2c80cf5c0 0x7fd2c80aa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.433+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80ce950 0x7fd2c815f300 secure :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00c3c0 tx=0x7fd2bc00c880 comp rx=0 tx=0).stop 2024-06-29T14:41:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.433+0000 7fd2cdd24640 1 --2- 172.21.15.59:0/859142200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2c808c190 0x7fd2c815edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.433+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 >> 172.21.15.59:0/859142200 conn(0x7fd2c80b5610 msgr2=0x7fd2c815dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.440+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 shutdown_connections 2024-06-29T14:41:21.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.444+0000 7fd2cdd24640 1 -- 172.21.15.59:0/859142200 wait complete. 2024-06-29T14:41:21.460 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:21.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.585+0000 7fa36121f640 1 -- 172.21.15.59:0/94921857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa348005150 msgr2=0x7fa34800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.585+0000 7fa36121f640 1 --2- 172.21.15.59:0/94921857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa348005150 0x7fa34800b9c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009f00 tx=0x7fa348005830 comp rx=0 tx=0).stop 2024-06-29T14:41:21.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.585+0000 7fa35b59e640 1 -- 172.21.15.59:0/94921857 reap_dead start 2024-06-29T14:41:21.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.585+0000 7fa36121f640 1 -- 172.21.15.59:0/94921857 shutdown_connections 2024-06-29T14:41:21.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.585+0000 7fa36121f640 1 -- 172.21.15.59:0/94921857 >> 172.21.15.59:0/94921857 conn(0x7fa35c0d3ee0 msgr2=0x7fa35c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:21.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.585+0000 7fa36121f640 1 -- 172.21.15.59:0/94921857 shutdown_connections 2024-06-29T14:41:21.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.585+0000 7fa36121f640 1 -- 172.21.15.59:0/94921857 wait complete. 2024-06-29T14:41:21.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.587+0000 7fa36121f640 1 Processor -- start 2024-06-29T14:41:21.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.587+0000 7fa36121f640 1 -- start start 2024-06-29T14:41:21.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa36121f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa348005150 0x7fa35c1f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa36121f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35c0c8410 0x7fa35c0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa36121f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35c0c96c0 0x7fa35c0c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa36121f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa35c15e8d0 con 0x7fa35c0c96c0 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa36121f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa35c0b42b0 con 0x7fa348005150 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa36121f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa35c15ea50 con 0x7fa35c0c8410 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa35b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35c0c96c0 0x7fa35c0c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa35b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35c0c96c0 0x7fa35c0c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58120/0 (socket says 172.21.15.59:58120) 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.588+0000 7fa35b59e640 1 -- 172.21.15.59:0/3261956125 learned_addr learned my addr 172.21.15.59:0/3261956125 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa35ad9d640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa348005150 0x7fa35c1f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa35a59c640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35c0c8410 0x7fa35c0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa35b59e640 1 -- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35c0c8410 msgr2=0x7fa35c0c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa35b59e640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35c0c8410 0x7fa35c0c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa35b59e640 1 -- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa348005150 msgr2=0x7fa35c1f7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa35b59e640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa348005150 0x7fa35c1f7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa35b59e640 1 -- 172.21.15.59:0/3261956125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35c15b780 con 0x7fa35c0c96c0 2024-06-29T14:41:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa35b59e640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35c0c96c0 0x7fa35c0c9ad0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fa34c005fd0 tx=0x7fa34c033dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa3537fe640 1 -- 172.21.15.59:0/3261956125 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c043070 con 0x7fa35c0c96c0 2024-06-29T14:41:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa3537fe640 1 -- 172.21.15.59:0/3261956125 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa34c003c80 con 0x7fa35c0c96c0 2024-06-29T14:41:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa3537fe640 1 -- 172.21.15.59:0/3261956125 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c038040 con 0x7fa35c0c96c0 2024-06-29T14:41:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa35c15ba60 con 0x7fa35c0c96c0 2024-06-29T14:41:21.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.589+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa35c15c020 con 0x7fa35c0c96c0 2024-06-29T14:41:21.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.590+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa35c160170 con 0x7fa35c0c96c0 2024-06-29T14:41:21.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.591+0000 7fa3537fe640 1 -- 172.21.15.59:0/3261956125 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fa34c004290 con 0x7fa35c0c96c0 2024-06-29T14:41:21.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.592+0000 7fa3537fe640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fa33c0763f0 0x7fa33c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.592+0000 7fa3537fe640 1 -- 172.21.15.59:0/3261956125 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 6129+0+0 (secure 0 0 0) 0x7fa34c0b81b0 con 0x7fa35c0c96c0 2024-06-29T14:41:21.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.592+0000 7fa35ad9d640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fa33c0763f0 0x7fa33c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.593+0000 7fa35ad9d640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fa33c0763f0 0x7fa33c078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa354000ff0 tx=0x7fa354001b90 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:21.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.596+0000 7fa3537fe640 1 -- 172.21.15.59:0/3261956125 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa34c081170 con 0x7fa35c0c96c0 2024-06-29T14:41:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.787+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa35c0d2950 con 0x7fa35c0c96c0 2024-06-29T14:41:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.789+0000 7fa3537fe640 1 -- 172.21.15.59:0/3261956125 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) ==== 74+0+17021 (secure 0 0 0) 0x7fa34c085470 con 0x7fa35c0c96c0 2024-06-29T14:41:21.787 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:21.787 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":142,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:20.795483+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.171960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:19.744696+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fa33c0763f0 msgr2=0x7fa33c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fa33c0763f0 0x7fa33c078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa354000ff0 tx=0x7fa354001b90 comp rx=0 tx=0).stop 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35c0c96c0 msgr2=0x7fa35c0c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35c0c96c0 0x7fa35c0c9ad0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fa34c005fd0 tx=0x7fa34c033dc0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 shutdown_connections 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35c0c96c0 0x7fa35c0c9ad0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fa33c0763f0 0x7fa33c078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35c0c8410 0x7fa35c0c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 --2- 172.21.15.59:0/3261956125 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa348005150 0x7fa35c1f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 >> 172.21.15.59:0/3261956125 conn(0x7fa35c0d3ee0 msgr2=0x7fa35c0d2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 shutdown_connections 2024-06-29T14:41:21.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.791+0000 7fa36121f640 1 -- 172.21.15.59:0/3261956125 wait complete. 2024-06-29T14:41:21.801 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:21.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.919+0000 7ff403e3c640 1 -- 172.21.15.59:0/3063034089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 msgr2=0x7ff3fc15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.919+0000 7ff403e3c640 1 --2- 172.21.15.59:0/3063034089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 0x7ff3fc15e8d0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0d15d0 tx=0x7ff3ec02a560 comp rx=0 tx=0).stop 2024-06-29T14:41:21.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.919+0000 7ff403e3c640 1 -- 172.21.15.59:0/3063034089 shutdown_connections 2024-06-29T14:41:21.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.919+0000 7ff403e3c640 1 --2- 172.21.15.59:0/3063034089 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3fc15ee10 0x7ff3fc15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.919+0000 7ff403e3c640 1 --2- 172.21.15.59:0/3063034089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 0x7ff3fc15e8d0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.919+0000 7ff403e3c640 1 --2- 172.21.15.59:0/3063034089 >> v2:172.21.15.153:3300/0 conn(0x7ff3fc0d23c0 0x7ff3fc0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.919+0000 7ff403e3c640 1 -- 172.21.15.59:0/3063034089 >> 172.21.15.59:0/3063034089 conn(0x7ff3fc0d3ee0 msgr2=0x7ff3fc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:21.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.920+0000 7ff403e3c640 1 -- 172.21.15.59:0/3063034089 shutdown_connections 2024-06-29T14:41:21.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.920+0000 7ff403e3c640 1 -- 172.21.15.59:0/3063034089 wait complete. 2024-06-29T14:41:21.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.921+0000 7ff403e3c640 1 Processor -- start 2024-06-29T14:41:21.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.921+0000 7ff403e3c640 1 -- start start 2024-06-29T14:41:21.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff403e3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 0x7ff3fc156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff403e3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3fc0d23c0 0x7ff3fc15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff403e3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3fc15ee10 0x7ff3fc15d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff403e3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc160070 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff403e3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc15fef0 con 0x7ff3fc0d23c0 2024-06-29T14:41:21.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff403e3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff3fc1601f0 con 0x7ff3fc15ee10 2024-06-29T14:41:21.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 0x7ff3fc156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 0x7ff3fc156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58150/0 (socket says 172.21.15.59:58150) 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 -- 172.21.15.59:0/1483468383 learned_addr learned my addr 172.21.15.59:0/1483468383 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff4013d7640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3fc0d23c0 0x7ff3fc15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff4023d9640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3fc15ee10 0x7ff3fc15d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 -- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3fc15ee10 msgr2=0x7ff3fc15d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3fc15ee10 0x7ff3fc15d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 -- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3fc0d23c0 msgr2=0x7ff3fc15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3fc0d23c0 0x7ff3fc15ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 -- 172.21.15.59:0/1483468383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc15b890 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff401bd8640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 0x7ff3fc156880 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800d680 tx=0x7ff3f800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.923+0000 7ff3f27fc640 1 -- 172.21.15.59:0/1483468383 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8017070 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.924+0000 7ff3f27fc640 1 -- 172.21.15.59:0/1483468383 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3f8009b10 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.924+0000 7ff3f27fc640 1 -- 172.21.15.59:0/1483468383 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8010460 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.924+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3fc15bb70 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.924+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3fc159230 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.925+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3fc15e330 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.925+0000 7ff3f27fc640 1 -- 172.21.15.59:0/1483468383 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7ff3f801a070 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.926+0000 7ff3f27fc640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ff3e00763f0 0x7ff3e0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.926+0000 7ff3f27fc640 1 -- 172.21.15.59:0/1483468383 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 6129+0+0 (secure 0 0 0) 0x7ff3f8097c60 con 0x7ff3fc0b3ed0 2024-06-29T14:41:21.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.926+0000 7ff4013d7640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ff3e00763f0 0x7ff3e0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.927+0000 7ff4013d7640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ff3e00763f0 0x7ff3e0078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec033000 tx=0x7ff3ec004730 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:21.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:21.930+0000 7ff3f27fc640 1 -- 172.21.15.59:0/1483468383 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff3f8061150 con 0x7ff3fc0b3ed0 2024-06-29T14:41:22.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.123+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3fc0b3990 con 0x7ff3fc0b3ed0 2024-06-29T14:41:22.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.124+0000 7ff3f27fc640 1 -- 172.21.15.59:0/1483468383 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) ==== 74+0+17021 (secure 0 0 0) 0x7ff3f8060a50 con 0x7ff3fc0b3ed0 2024-06-29T14:41:22.122 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:22.122 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":142,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:20.795483+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.171960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:19.744696+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ff3e00763f0 msgr2=0x7ff3e0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ff3e00763f0 0x7ff3e0078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec033000 tx=0x7ff3ec004730 comp rx=0 tx=0).stop 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 msgr2=0x7ff3fc156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 0x7ff3fc156880 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800d680 tx=0x7ff3f800db40 comp rx=0 tx=0).stop 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 shutdown_connections 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7ff3e00763f0 0x7ff3e0078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3fc15ee10 0x7ff3fc15d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3fc0d23c0 0x7ff3fc15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 --2- 172.21.15.59:0/1483468383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3fc0b3ed0 0x7ff3fc156880 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 >> 172.21.15.59:0/1483468383 conn(0x7ff3fc0d3ee0 msgr2=0x7ff3fc160730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 shutdown_connections 2024-06-29T14:41:22.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.126+0000 7ff403e3c640 1 -- 172.21.15.59:0/1483468383 wait complete. 2024-06-29T14:41:22.136 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 -- 172.21.15.59:0/2980645312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 msgr2=0x7f11d40b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 --2- 172.21.15.59:0/2980645312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 0x7f11d40b56d0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f11d4073f30 tx=0x7f11c402a980 comp rx=0 tx=0).stop 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 -- 172.21.15.59:0/2980645312 shutdown_connections 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 --2- 172.21.15.59:0/2980645312 >> v2:172.21.15.153:3300/0 conn(0x7f11d41414c0 0x7f11d41418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 --2- 172.21.15.59:0/2980645312 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11d413e9b0 0x7f11d413edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 --2- 172.21.15.59:0/2980645312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 0x7f11d40b56d0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 -- 172.21.15.59:0/2980645312 >> 172.21.15.59:0/2980645312 conn(0x7f11d40b6fb0 msgr2=0x7f11d40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 -- 172.21.15.59:0/2980645312 shutdown_connections 2024-06-29T14:41:22.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.259+0000 7f11dc763640 1 -- 172.21.15.59:0/2980645312 wait complete. 2024-06-29T14:41:22.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.261+0000 7f11dc763640 1 Processor -- start 2024-06-29T14:41:22.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.261+0000 7f11dc763640 1 -- start start 2024-06-29T14:41:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.262+0000 7f11dc763640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11d413e9b0 0x7f11d4159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.262+0000 7f11dc763640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d41414c0 0x7f11d406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.262+0000 7f11dc763640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 0x7f11d406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.262+0000 7f11dc763640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f11d40b28c0 con 0x7f11d415ceb0 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.262+0000 7f11dc763640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11d40b2740 con 0x7f11d41414c0 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.262+0000 7f11dc763640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11d40b2a40 con 0x7f11d413e9b0 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.262+0000 7f11dad00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 0x7f11d406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dad00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 0x7f11d406ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58164/0 (socket says 172.21.15.59:58164) 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dad00640 1 -- 172.21.15.59:0/72237246 learned_addr learned my addr 172.21.15.59:0/72237246 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11da4ff640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11d413e9b0 0x7f11d4159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11d9cfe640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d41414c0 0x7f11d406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dad00640 1 -- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11d413e9b0 msgr2=0x7f11d4159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dad00640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11d413e9b0 0x7f11d4159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dad00640 1 -- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d41414c0 msgr2=0x7f11d406e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dad00640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d41414c0 0x7f11d406e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dad00640 1 -- 172.21.15.59:0/72237246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d406db80 con 0x7f11d415ceb0 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11da4ff640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11d413e9b0 0x7f11d4159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11d9cfe640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d41414c0 0x7f11d406e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dad00640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 0x7f11d406ee60 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00d690 tx=0x7f11cc00db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11d2ffd640 1 -- 172.21.15.59:0/72237246 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc017070 con 0x7f11d415ceb0 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11d2ffd640 1 -- 172.21.15.59:0/72237246 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11cc005310 con 0x7f11d415ceb0 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11d2ffd640 1 -- 172.21.15.59:0/72237246 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc009820 con 0x7f11d415ceb0 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d415a860 con 0x7f11d415ceb0 2024-06-29T14:41:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.263+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d406e120 con 0x7f11d415ceb0 2024-06-29T14:41:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.264+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d40b2f60 con 0x7f11d415ceb0 2024-06-29T14:41:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.265+0000 7f11d2ffd640 1 -- 172.21.15.59:0/72237246 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f11cc0099c0 con 0x7f11d415ceb0 2024-06-29T14:41:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.265+0000 7f11d2ffd640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f11c00763f0 0x7f11c0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.266+0000 7f11d2ffd640 1 -- 172.21.15.59:0/72237246 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 6129+0+0 (secure 0 0 0) 0x7f11cc097d20 con 0x7f11d415ceb0 2024-06-29T14:41:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.266+0000 7f11da4ff640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f11c00763f0 0x7f11c0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.266+0000 7f11da4ff640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f11c00763f0 0x7f11c0078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f11c4035000 tx=0x7f11c4004ab0 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:22.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.270+0000 7f11d2ffd640 1 -- 172.21.15.59:0/72237246 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11cc061100 con 0x7f11d415ceb0 2024-06-29T14:41:22.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:22.502+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f11d40ce160 con 0x7f11d415ceb0 2024-06-29T14:41:23.383 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T14:41:23.386+0000 7f90a49af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:41:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.428+0000 7f11d2ffd640 1 -- 172.21.15.59:0/72237246 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v71) ==== 118+0+0 (secure 0 0 0) 0x7f11cc069180 con 0x7f11d415ceb0 2024-06-29T14:41:23.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f11c00763f0 msgr2=0x7f11c0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f11c00763f0 0x7f11c0078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f11c4035000 tx=0x7f11c4004ab0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 msgr2=0x7f11d406ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 0x7f11d406ee60 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00d690 tx=0x7f11cc00db50 comp rx=0 tx=0).stop 2024-06-29T14:41:23.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 shutdown_connections 2024-06-29T14:41:23.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d415ceb0 0x7f11d406ee60 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11d41414c0 0x7f11d406e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f11c00763f0 0x7f11c0078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 --2- 172.21.15.59:0/72237246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11d413e9b0 0x7f11d4159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 >> 172.21.15.59:0/72237246 conn(0x7f11d40b6fb0 msgr2=0x7f11d413cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:23.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 shutdown_connections 2024-06-29T14:41:23.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.431+0000 7f11dc763640 1 -- 172.21.15.59:0/72237246 wait complete. 2024-06-29T14:41:23.443 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:41:23.443 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:41:23.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/2859382732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb813e9b0 msgr2=0x7f0eb813edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/2859382732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb813e9b0 0x7f0eb813edc0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8069670 tx=0x7f0eac02a4d0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/2859382732 shutdown_connections 2024-06-29T14:41:23.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/2859382732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb81414c0 0x7f0eb81418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/2859382732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb813e9b0 0x7f0eb813edc0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/2859382732 >> v2:172.21.15.153:3300/0 conn(0x7f0eb815ceb0 0x7f0eb80b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/2859382732 >> 172.21.15.59:0/2859382732 conn(0x7f0eb80b6fb0 msgr2=0x7f0eb80b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:23.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/2859382732 shutdown_connections 2024-06-29T14:41:23.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.573+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/2859382732 wait complete. 2024-06-29T14:41:23.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.575+0000 7f0ebe1b9640 1 Processor -- start 2024-06-29T14:41:23.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.575+0000 7f0ebe1b9640 1 -- start start 2024-06-29T14:41:23.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0ebe1b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb813e9b0 0x7f0eb81e59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:23.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0ebe1b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb81414c0 0x7f0eb81e5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:23.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0ebe1b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb815ceb0 0x7f0eb81e6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:23.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0ebe1b9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb80b28c0 con 0x7f0eb81414c0 2024-06-29T14:41:23.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0ebe1b9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb80b2740 con 0x7f0eb815ceb0 2024-06-29T14:41:23.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0ebe1b9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb80b2a40 con 0x7f0eb813e9b0 2024-06-29T14:41:23.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb81414c0 0x7f0eb81e5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:23.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb81414c0 0x7f0eb81e5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37704/0 (socket says 172.21.15.59:37704) 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 -- 172.21.15.59:0/4205078826 learned_addr learned my addr 172.21.15.59:0/4205078826 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb7fff640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb815ceb0 0x7f0eb81e6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb77fe640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb813e9b0 0x7f0eb81e59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 -- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb813e9b0 msgr2=0x7f0eb81e59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb813e9b0 0x7f0eb81e59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 -- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb815ceb0 msgr2=0x7f0eb81e6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb815ceb0 0x7f0eb81e6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 -- 172.21.15.59:0/4205078826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb80c7fe0 con 0x7f0eb81414c0 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb7fff640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb815ceb0 0x7f0eb81e6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.577+0000 7f0eb6ffd640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb81414c0 0x7f0eb81e5f30 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0eac000c00 tx=0x7f0eac003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.578+0000 7f0e9ffff640 1 -- 172.21.15.59:0/4205078826 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac03e030 con 0x7f0eb81414c0 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.578+0000 7f0e9ffff640 1 -- 172.21.15.59:0/4205078826 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eac003e00 con 0x7f0eb81414c0 2024-06-29T14:41:23.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.578+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb80c82c0 con 0x7f0eb81414c0 2024-06-29T14:41:23.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.578+0000 7f0e9ffff640 1 -- 172.21.15.59:0/4205078826 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac035040 con 0x7f0eb81414c0 2024-06-29T14:41:23.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.578+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb80c8880 con 0x7f0eb81414c0 2024-06-29T14:41:23.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.579+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb80c7d50 con 0x7f0eb81414c0 2024-06-29T14:41:23.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.579+0000 7f0e9ffff640 1 -- 172.21.15.59:0/4205078826 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f0eac041030 con 0x7f0eb81414c0 2024-06-29T14:41:23.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.580+0000 7f0e9ffff640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0ea40763f0 0x7f0ea4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:23.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.580+0000 7f0e9ffff640 1 -- 172.21.15.59:0/4205078826 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7f0eac0b83c0 con 0x7f0eb81414c0 2024-06-29T14:41:23.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.580+0000 7f0eb77fe640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0ea40763f0 0x7f0ea4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:23.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.581+0000 7f0eb77fe640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0ea40763f0 0x7f0ea4078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8005d70 tx=0x7f0ea8005d00 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:23.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.585+0000 7f0e9ffff640 1 -- 172.21.15.59:0/4205078826 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0eac0818c0 con 0x7f0eb81414c0 2024-06-29T14:41:23.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:23.820+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0eb80b2f60 con 0x7f0eb81414c0 2024-06-29T14:41:24.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.429+0000 7f0e9ffff640 1 -- 172.21.15.59:0/4205078826 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v72) ==== 122+0+0 (secure 0 0 0) 0x7f0eac0811c0 con 0x7f0eb81414c0 2024-06-29T14:41:24.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0ea40763f0 msgr2=0x7f0ea4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0ea40763f0 0x7f0ea4078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8005d70 tx=0x7f0ea8005d00 comp rx=0 tx=0).stop 2024-06-29T14:41:24.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb81414c0 msgr2=0x7f0eb81e5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb81414c0 0x7f0eb81e5f30 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0eac000c00 tx=0x7f0eac003910 comp rx=0 tx=0).stop 2024-06-29T14:41:24.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 shutdown_connections 2024-06-29T14:41:24.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb815ceb0 0x7f0eb81e6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb81414c0 0x7f0eb81e5f30 secure :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0eac000c00 tx=0x7f0eac003910 comp rx=0 tx=0).stop 2024-06-29T14:41:24.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0ea40763f0 0x7f0ea4078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 --2- 172.21.15.59:0/4205078826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb813e9b0 0x7f0eb81e59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.433+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 >> 172.21.15.59:0/4205078826 conn(0x7f0eb80b6fb0 msgr2=0x7f0eb815e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:24.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.434+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 shutdown_connections 2024-06-29T14:41:24.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.434+0000 7f0ebe1b9640 1 -- 172.21.15.59:0/4205078826 wait complete. 2024-06-29T14:41:24.446 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:41:24.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.574+0000 7f59ee1f5640 1 -- 172.21.15.59:0/3306061924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59d0004120 msgr2=0x7f59d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.574+0000 7f59ee1f5640 1 --2- 172.21.15.59:0/3306061924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59d0004120 0x7f59d0004800 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0016c0 tx=0x7f59d0005830 comp rx=0 tx=0).stop 2024-06-29T14:41:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.574+0000 7f59e7fff640 1 -- 172.21.15.59:0/3306061924 reap_dead start 2024-06-29T14:41:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.574+0000 7f59ee1f5640 1 -- 172.21.15.59:0/3306061924 shutdown_connections 2024-06-29T14:41:24.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.574+0000 7f59ee1f5640 1 -- 172.21.15.59:0/3306061924 >> 172.21.15.59:0/3306061924 conn(0x7f59e806c950 msgr2=0x7f59e8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:24.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.574+0000 7f59ee1f5640 1 -- 172.21.15.59:0/3306061924 shutdown_connections 2024-06-29T14:41:24.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.574+0000 7f59ee1f5640 1 -- 172.21.15.59:0/3306061924 wait complete. 2024-06-29T14:41:24.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.576+0000 7f59ee1f5640 1 Processor -- start 2024-06-29T14:41:24.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.576+0000 7f59ee1f5640 1 -- start start 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59ee1f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0004120 0x7f59e81f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59ee1f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59e80b10a0 0x7f59e80abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59ee1f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59e80ac180 0x7f59e8162b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59ee1f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f59e8165020 con 0x7f59e80ac180 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59ee1f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59e8164ea0 con 0x7f59d0004120 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59ee1f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f59e81651a0 con 0x7f59e80b10a0 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59e7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59e80ac180 0x7f59e8162b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59e7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59e80ac180 0x7f59e8162b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37728/0 (socket says 172.21.15.59:37728) 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59e7fff640 1 -- 172.21.15.59:0/2129260542 learned_addr learned my addr 172.21.15.59:0/2129260542 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59e6ffd640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59e80b10a0 0x7f59e80abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.577+0000 7f59e77fe640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0004120 0x7f59e81f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59e7fff640 1 -- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59e80b10a0 msgr2=0x7f59e80abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59e7fff640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59e80b10a0 0x7f59e80abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59e7fff640 1 -- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0004120 msgr2=0x7f59e81f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59e7fff640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0004120 0x7f59e81f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59e7fff640 1 -- 172.21.15.59:0/2129260542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e80aaba0 con 0x7f59e80ac180 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59e7fff640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59e80ac180 0x7f59e8162b90 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f59e80b5030 tx=0x7f59e0010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59cffff640 1 -- 172.21.15.59:0/2129260542 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0003c40 con 0x7f59e80ac180 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59cffff640 1 -- 172.21.15.59:0/2129260542 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59e0003e00 con 0x7f59e80ac180 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e80aae80 con 0x7f59e80ac180 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59cffff640 1 -- 172.21.15.59:0/2129260542 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0004d80 con 0x7f59e80ac180 2024-06-29T14:41:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.578+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e80ab440 con 0x7f59e80ac180 2024-06-29T14:41:24.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.579+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e81458a0 con 0x7f59e80ac180 2024-06-29T14:41:24.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.580+0000 7f59cffff640 1 -- 172.21.15.59:0/2129260542 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f59e001c020 con 0x7f59e80ac180 2024-06-29T14:41:24.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.580+0000 7f59cffff640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f59d40763f0 0x7f59d4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.580+0000 7f59cffff640 1 -- 172.21.15.59:0/2129260542 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7f59e0097d50 con 0x7f59e80ac180 2024-06-29T14:41:24.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.581+0000 7f59e77fe640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f59d40763f0 0x7f59d4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.581+0000 7f59e77fe640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f59d40763f0 0x7f59d4078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59d8001220 tx=0x7f59d80067a0 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:24.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.585+0000 7f59cffff640 1 -- 172.21.15.59:0/2129260542 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f59e0060bf0 con 0x7f59e80ac180 2024-06-29T14:41:24.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.815+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59e80b1f10 con 0x7f59e80ac180 2024-06-29T14:41:24.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.816+0000 7f59cffff640 1 -- 172.21.15.59:0/2129260542 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5228 (secure 0 0 0) 0x7f59e0064ef0 con 0x7f59e80ac180 2024-06-29T14:41:24.814 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:24.814 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":72,"btime":"2024-06-29T14:41:24:429207+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7442,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/4194962668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4194962668},{"type":"v1","addr":"172.21.15.153:6836","nonce":4194962668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7445,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1840307553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1840307553},{"type":"v1","addr":"172.21.15.153:6837","nonce":1840307553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1010388227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1010388227},{"type":"v1","addr":"172.21.15.59:6834","nonce":1010388227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:20.795436+0000","modified":"2024-06-29T14:41:23.821908+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8844":{"gid":8844,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/957887336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":957887336},{"type":"v1","addr":"172.21.15.59:6835","nonce":957887336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8844,"qdb_cluster":[8844]},"id":8}]} 2024-06-29T14:41:24.814 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 72 2024-06-29T14:41:24.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f59d40763f0 msgr2=0x7f59d4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f59d40763f0 0x7f59d4078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59d8001220 tx=0x7f59d80067a0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59e80ac180 msgr2=0x7f59e8162b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59e80ac180 0x7f59e8162b90 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f59e80b5030 tx=0x7f59e0010f00 comp rx=0 tx=0).stop 2024-06-29T14:41:24.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 shutdown_connections 2024-06-29T14:41:24.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59e80ac180 0x7f59e8162b90 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f59d40763f0 0x7f59d4078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f59e80b10a0 0x7f59e80abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 --2- 172.21.15.59:0/2129260542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0004120 0x7f59e81f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 >> 172.21.15.59:0/2129260542 conn(0x7f59e806c950 msgr2=0x7f59e815cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:24.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 shutdown_connections 2024-06-29T14:41:24.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.818+0000 7f59ee1f5640 1 -- 172.21.15.59:0/2129260542 wait complete. 2024-06-29T14:41:24.828 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.947+0000 7fe729289640 1 -- 172.21.15.59:0/149418291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724162190 msgr2=0x7fe7240ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.947+0000 7fe729289640 1 --2- 172.21.15.59:0/149418291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724162190 0x7fe7240ce550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe70c005fd0 tx=0x7fe70c02a540 comp rx=0 tx=0).stop 2024-06-29T14:41:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.947+0000 7fe729289640 1 -- 172.21.15.59:0/149418291 shutdown_connections 2024-06-29T14:41:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.947+0000 7fe729289640 1 --2- 172.21.15.59:0/149418291 >> v2:172.21.15.153:3300/0 conn(0x7fe72415e0a0 0x7fe7240b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.947+0000 7fe729289640 1 --2- 172.21.15.59:0/149418291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724162190 0x7fe7240ce550 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.947+0000 7fe729289640 1 -- 172.21.15.59:0/149418291 >> 172.21.15.59:0/149418291 conn(0x7fe7240d45b0 msgr2=0x7fe7240d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.947+0000 7fe729289640 1 -- 172.21.15.59:0/149418291 shutdown_connections 2024-06-29T14:41:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.947+0000 7fe729289640 1 -- 172.21.15.59:0/149418291 wait complete. 2024-06-29T14:41:24.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.949+0000 7fe729289640 1 Processor -- start 2024-06-29T14:41:24.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.949+0000 7fe729289640 1 -- start start 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.950+0000 7fe729289640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72415e0a0 0x7fe7241570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe729289640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724162770 0x7fe7241f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe729289640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe724159ab0 0x7fe724159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe729289640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe7240b43a0 con 0x7fe724159ab0 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe729289640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe7240b4220 con 0x7fe72415e0a0 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe729289640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe7240b4520 con 0x7fe724162770 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe724159ab0 0x7fe724159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe724159ab0 0x7fe724159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37740/0 (socket says 172.21.15.59:37740) 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 -- 172.21.15.59:0/2485241838 learned_addr learned my addr 172.21.15.59:0/2485241838 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe722ffd640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72415e0a0 0x7fe7241570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7227fc640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724162770 0x7fe7241f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 -- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724162770 msgr2=0x7fe7241f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724162770 0x7fe7241f42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 -- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72415e0a0 msgr2=0x7fe7241570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72415e0a0 0x7fe7241570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 -- 172.21.15.59:0/2485241838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72415a400 con 0x7fe724159ab0 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe7237fe640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe724159ab0 0x7fe724159ec0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00e6d0 tx=0x7fe71c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe71b7fe640 1 -- 172.21.15.59:0/2485241838 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c005220 con 0x7fe724159ab0 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe71b7fe640 1 -- 172.21.15.59:0/2485241838 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe71c0053c0 con 0x7fe724159ab0 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7241f4bf0 con 0x7fe724159ab0 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe71b7fe640 1 -- 172.21.15.59:0/2485241838 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c00c850 con 0x7fe724159ab0 2024-06-29T14:41:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.951+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7241f51b0 con 0x7fe724159ab0 2024-06-29T14:41:24.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.952+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7240b49d0 con 0x7fe724159ab0 2024-06-29T14:41:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.953+0000 7fe71b7fe640 1 -- 172.21.15.59:0/2485241838 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fe71c00ca80 con 0x7fe724159ab0 2024-06-29T14:41:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.954+0000 7fe71b7fe640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe704076420 0x7fe704078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.954+0000 7fe71b7fe640 1 -- 172.21.15.59:0/2485241838 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7fe71c0981d0 con 0x7fe724159ab0 2024-06-29T14:41:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.954+0000 7fe722ffd640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe704076420 0x7fe704078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.954+0000 7fe722ffd640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe704076420 0x7fe704078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7240b5480 tx=0x7fe714009040 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:24.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:24.958+0000 7fe71b7fe640 1 -- 172.21.15.59:0/2485241838 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe72415a400 con 0x7fe724159ab0 2024-06-29T14:41:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.151+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe72406cd70 con 0x7fe724159ab0 2024-06-29T14:41:25.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.152+0000 7fe71b7fe640 1 -- 172.21.15.59:0/2485241838 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+17008 (secure 0 0 0) 0x7fe71c061120 con 0x7fe724159ab0 2024-06-29T14:41:25.150 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:25.151 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":143,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:22.417959+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.171960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:19.744696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe704076420 msgr2=0x7fe704078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe704076420 0x7fe704078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7240b5480 tx=0x7fe714009040 comp rx=0 tx=0).stop 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe724159ab0 msgr2=0x7fe724159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe724159ab0 0x7fe724159ec0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00e6d0 tx=0x7fe71c00eb90 comp rx=0 tx=0).stop 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 shutdown_connections 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe724159ab0 0x7fe724159ec0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fe704076420 0x7fe704078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724162770 0x7fe7241f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 --2- 172.21.15.59:0/2485241838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72415e0a0 0x7fe7241570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.154+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 >> 172.21.15.59:0/2485241838 conn(0x7fe7240d45b0 msgr2=0x7fe7240b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.155+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 shutdown_connections 2024-06-29T14:41:25.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.155+0000 7fe729289640 1 -- 172.21.15.59:0/2485241838 wait complete. 2024-06-29T14:41:25.164 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:41:25.164 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:41:25.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.285+0000 7f7892dbd640 1 -- 172.21.15.59:0/3548722577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 msgr2=0x7f787800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.285+0000 7f7892dbd640 1 --2- 172.21.15.59:0/3548722577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 0x7f787800ba20 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f788c0b5480 tx=0x7f7878005830 comp rx=0 tx=0).stop 2024-06-29T14:41:25.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.285+0000 7f7892dbd640 1 -- 172.21.15.59:0/3548722577 shutdown_connections 2024-06-29T14:41:25.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.285+0000 7f7892dbd640 1 --2- 172.21.15.59:0/3548722577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 0x7f787800ba20 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.285+0000 7f7892dbd640 1 -- 172.21.15.59:0/3548722577 >> 172.21.15.59:0/3548722577 conn(0x7f788c0d45b0 msgr2=0x7f788c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.285+0000 7f7892dbd640 1 -- 172.21.15.59:0/3548722577 shutdown_connections 2024-06-29T14:41:25.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.285+0000 7f7892dbd640 1 -- 172.21.15.59:0/3548722577 wait complete. 2024-06-29T14:41:25.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.287+0000 7f7892dbd640 1 Processor -- start 2024-06-29T14:41:25.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.287+0000 7f7892dbd640 1 -- start start 2024-06-29T14:41:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7892dbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 0x7f788c149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7892dbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f788c15a730 0x7f788c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7892dbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f788c0cd190 0x7f788c0c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7892dbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f788c161a80 con 0x7f78780053a0 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7892dbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f788c0ce7c0 con 0x7f788c15a730 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7892dbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f788c161c00 con 0x7f788c0cd190 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7890b59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 0x7f788c149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7890b59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 0x7f788c149930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37768/0 (socket says 172.21.15.59:37768) 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f7890b59640 1 -- 172.21.15.59:0/1901889593 learned_addr learned my addr 172.21.15.59:0/1901889593 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f789135a640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f788c0cd190 0x7f788c0c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.288+0000 7f788bfff640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f788c15a730 0x7f788c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f7890b59640 1 -- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f788c0cd190 msgr2=0x7f788c0c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f7890b59640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f788c0cd190 0x7f788c0c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f7890b59640 1 -- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f788c15a730 msgr2=0x7f788c0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f7890b59640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f788c15a730 0x7f788c0ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f7890b59640 1 -- 172.21.15.59:0/1901889593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788c0cbc20 con 0x7f78780053a0 2024-06-29T14:41:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f7890b59640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 0x7f788c149930 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f7874004e50 tx=0x7f7874001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f78897fa640 1 -- 172.21.15.59:0/1901889593 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78740107d0 con 0x7f78780053a0 2024-06-29T14:41:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f78897fa640 1 -- 172.21.15.59:0/1901889593 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7874016070 con 0x7f78780053a0 2024-06-29T14:41:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788c0cbe90 con 0x7f78780053a0 2024-06-29T14:41:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f78897fa640 1 -- 172.21.15.59:0/1901889593 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874010cc0 con 0x7f78780053a0 2024-06-29T14:41:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.289+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f788c0cc450 con 0x7f78780053a0 2024-06-29T14:41:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.290+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f788c0b4810 con 0x7f78780053a0 2024-06-29T14:41:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.291+0000 7f78897fa640 1 -- 172.21.15.59:0/1901889593 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f787401b460 con 0x7f78780053a0 2024-06-29T14:41:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.291+0000 7f78897fa640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f786c0763f0 0x7f786c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.291+0000 7f78897fa640 1 -- 172.21.15.59:0/1901889593 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7f78740982a0 con 0x7f78780053a0 2024-06-29T14:41:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.292+0000 7f788bfff640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f786c0763f0 0x7f786c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.292+0000 7f788bfff640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f786c0763f0 0x7f786c078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f788c1f80a0 tx=0x7f787c009500 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.296+0000 7f78897fa640 1 -- 172.21.15.59:0/1901889593 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f788c0b4810 con 0x7f78780053a0 2024-06-29T14:41:25.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.488+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 --> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f788c162160 con 0x7f786c0763f0 2024-06-29T14:41:25.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.494+0000 7f78897fa640 1 -- 172.21.15.59:0/1901889593 <== mgr.7259 v2:172.21.15.153:6832/776633953 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252447 (secure 0 0 0) 0x7f788c162160 con 0x7f786c0763f0 2024-06-29T14:41:25.494 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:25.496 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:41:25.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f786c0763f0 msgr2=0x7f786c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f786c0763f0 0x7f786c078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f788c1f80a0 tx=0x7f787c009500 comp rx=0 tx=0).stop 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 msgr2=0x7f788c149930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 0x7f788c149930 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f7874004e50 tx=0x7f7874001630 comp rx=0 tx=0).stop 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 shutdown_connections 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f786c0763f0 0x7f786c078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f788c0cd190 0x7f788c0c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f788c15a730 0x7f788c0ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 --2- 172.21.15.59:0/1901889593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78780053a0 0x7f788c149930 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 >> 172.21.15.59:0/1901889593 conn(0x7f788c0d45b0 msgr2=0x7f788c06c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 shutdown_connections 2024-06-29T14:41:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.496+0000 7f7892dbd640 1 -- 172.21.15.59:0/1901889593 wait complete. 2024-06-29T14:41:25.506 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-06-29T14:41:24.960065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":290,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":356008,"kb_used_data":14964,"kb_used_omap":12,"kb_used_meta":340915,"kb_avail":754618712,"statfs":{"total":773094113280,"available":772729561088,"internally_reserved":0,"allocated":15323136,"data_stored":8320212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":349097522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.567798"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.434595+0000","last_change":"2024-06-29T14:41:21.432149+0000","last_active":"2024-06-29T14:41:21.434595+0000","last_peered":"2024-06-29T14:41:21.434595+0000","last_clean":"2024-06-29T14:41:21.434595+0000","last_became_active":"2024-06-29T14:41:21.431528+0000","last_became_peered":"2024-06-29T14:41:21.431528+0000","last_unstale":"2024-06-29T14:41:21.434595+0000","last_undegraded":"2024-06-29T14:41:21.434595+0000","last_fullsized":"2024-06-29T14:41:21.434595+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:42.146353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439601+0000","last_change":"2024-06-29T14:41:21.422105+0000","last_active":"2024-06-29T14:41:21.439601+0000","last_peered":"2024-06-29T14:41:21.439601+0000","last_clean":"2024-06-29T14:41:21.439601+0000","last_became_active":"2024-06-29T14:41:21.421776+0000","last_became_peered":"2024-06-29T14:41:21.421776+0000","last_unstale":"2024-06-29T14:41:21.439601+0000","last_undegraded":"2024-06-29T14:41:21.439601+0000","last_fullsized":"2024-06-29T14:41:21.439601+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:44.107767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.441309+0000","last_change":"2024-06-29T14:41:21.423301+0000","last_active":"2024-06-29T14:41:21.441309+0000","last_peered":"2024-06-29T14:41:21.441309+0000","last_clean":"2024-06-29T14:41:21.441309+0000","last_became_active":"2024-06-29T14:41:21.422401+0000","last_became_peered":"2024-06-29T14:41:21.422401+0000","last_unstale":"2024-06-29T14:41:21.441309+0000","last_undegraded":"2024-06-29T14:41:21.441309+0000","last_fullsized":"2024-06-29T14:41:21.441309+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:20.998587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.430560+0000","last_change":"2024-06-29T14:41:21.427478+0000","last_active":"2024-06-29T14:41:21.430560+0000","last_peered":"2024-06-29T14:41:21.430560+0000","last_clean":"2024-06-29T14:41:21.430560+0000","last_became_active":"2024-06-29T14:41:21.427170+0000","last_became_peered":"2024-06-29T14:41:21.427170+0000","last_unstale":"2024-06-29T14:41:21.430560+0000","last_undegraded":"2024-06-29T14:41:21.430560+0000","last_fullsized":"2024-06-29T14:41:21.430560+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:15.824908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.432550+0000","last_change":"2024-06-29T14:41:21.430183+0000","last_active":"2024-06-29T14:41:21.432550+0000","last_peered":"2024-06-29T14:41:21.432550+0000","last_clean":"2024-06-29T14:41:21.432550+0000","last_became_active":"2024-06-29T14:41:21.429803+0000","last_became_peered":"2024-06-29T14:41:21.429803+0000","last_unstale":"2024-06-29T14:41:21.432550+0000","last_undegraded":"2024-06-29T14:41:21.432550+0000","last_fullsized":"2024-06-29T14:41:21.432550+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:40.853602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426280+0000","last_change":"2024-06-29T14:41:21.444312+0000","last_active":"2024-06-29T14:41:22.426280+0000","last_peered":"2024-06-29T14:41:22.426280+0000","last_clean":"2024-06-29T14:41:22.426280+0000","last_became_active":"2024-06-29T14:41:21.443961+0000","last_became_peered":"2024-06-29T14:41:21.443961+0000","last_unstale":"2024-06-29T14:41:22.426280+0000","last_undegraded":"2024-06-29T14:41:22.426280+0000","last_fullsized":"2024-06-29T14:41:22.426280+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:09.781784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425575+0000","last_change":"2024-06-29T14:41:21.436399+0000","last_active":"2024-06-29T14:41:22.425575+0000","last_peered":"2024-06-29T14:41:22.425575+0000","last_clean":"2024-06-29T14:41:22.425575+0000","last_became_active":"2024-06-29T14:41:21.436032+0000","last_became_peered":"2024-06-29T14:41:21.436032+0000","last_unstale":"2024-06-29T14:41:22.425575+0000","last_undegraded":"2024-06-29T14:41:22.425575+0000","last_fullsized":"2024-06-29T14:41:22.425575+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:29.933829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.420813+0000","last_change":"2024-06-29T14:41:21.418878+0000","last_active":"2024-06-29T14:41:21.420813+0000","last_peered":"2024-06-29T14:41:21.420813+0000","last_clean":"2024-06-29T14:41:21.420813+0000","last_became_active":"2024-06-29T14:41:21.418550+0000","last_became_peered":"2024-06-29T14:41:21.418550+0000","last_unstale":"2024-06-29T14:41:21.420813+0000","last_undegraded":"2024-06-29T14:41:21.420813+0000","last_fullsized":"2024-06-29T14:41:21.420813+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:24.799928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.424595+0000","last_change":"2024-06-29T14:41:21.422527+0000","last_active":"2024-06-29T14:41:21.424595+0000","last_peered":"2024-06-29T14:41:21.424595+0000","last_clean":"2024-06-29T14:41:21.424595+0000","last_became_active":"2024-06-29T14:41:21.422223+0000","last_became_peered":"2024-06-29T14:41:21.422223+0000","last_unstale":"2024-06-29T14:41:21.424595+0000","last_undegraded":"2024-06-29T14:41:21.424595+0000","last_fullsized":"2024-06-29T14:41:21.424595+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:48.505228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425909+0000","last_change":"2024-06-29T14:41:21.443188+0000","last_active":"2024-06-29T14:41:22.425909+0000","last_peered":"2024-06-29T14:41:22.425909+0000","last_clean":"2024-06-29T14:41:22.425909+0000","last_became_active":"2024-06-29T14:41:21.442834+0000","last_became_peered":"2024-06-29T14:41:21.442834+0000","last_unstale":"2024-06-29T14:41:22.425909+0000","last_undegraded":"2024-06-29T14:41:22.425909+0000","last_fullsized":"2024-06-29T14:41:22.425909+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:34.786229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439941+0000","last_change":"2024-06-29T14:41:21.436905+0000","last_active":"2024-06-29T14:41:21.439941+0000","last_peered":"2024-06-29T14:41:21.439941+0000","last_clean":"2024-06-29T14:41:21.439941+0000","last_became_active":"2024-06-29T14:41:21.436544+0000","last_became_peered":"2024-06-29T14:41:21.436544+0000","last_unstale":"2024-06-29T14:41:21.439941+0000","last_undegraded":"2024-06-29T14:41:21.439941+0000","last_fullsized":"2024-06-29T14:41:21.439941+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:49.771666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426604+0000","last_change":"2024-06-29T14:41:21.430583+0000","last_active":"2024-06-29T14:41:22.426604+0000","last_peered":"2024-06-29T14:41:22.426604+0000","last_clean":"2024-06-29T14:41:22.426604+0000","last_became_active":"2024-06-29T14:41:21.430215+0000","last_became_peered":"2024-06-29T14:41:21.430215+0000","last_unstale":"2024-06-29T14:41:22.426604+0000","last_undegraded":"2024-06-29T14:41:22.426604+0000","last_fullsized":"2024-06-29T14:41:22.426604+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:30:26.767490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425185+0000","last_change":"2024-06-29T14:41:21.432562+0000","last_active":"2024-06-29T14:41:22.425185+0000","last_peered":"2024-06-29T14:41:22.425185+0000","last_clean":"2024-06-29T14:41:22.425185+0000","last_became_active":"2024-06-29T14:41:21.432193+0000","last_became_peered":"2024-06-29T14:41:21.432193+0000","last_unstale":"2024-06-29T14:41:22.425185+0000","last_undegraded":"2024-06-29T14:41:22.425185+0000","last_fullsized":"2024-06-29T14:41:22.425185+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:41.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.421185+0000","last_change":"2024-06-29T14:41:21.418762+0000","last_active":"2024-06-29T14:41:21.421185+0000","last_peered":"2024-06-29T14:41:21.421185+0000","last_clean":"2024-06-29T14:41:21.421185+0000","last_became_active":"2024-06-29T14:41:21.418246+0000","last_became_peered":"2024-06-29T14:41:21.418246+0000","last_unstale":"2024-06-29T14:41:21.421185+0000","last_undegraded":"2024-06-29T14:41:21.421185+0000","last_fullsized":"2024-06-29T14:41:21.421185+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:30.919446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439937+0000","last_change":"2024-06-29T14:41:21.436102+0000","last_active":"2024-06-29T14:41:21.439937+0000","last_peered":"2024-06-29T14:41:21.439937+0000","last_clean":"2024-06-29T14:41:21.439937+0000","last_became_active":"2024-06-29T14:41:21.435767+0000","last_became_peered":"2024-06-29T14:41:21.435767+0000","last_unstale":"2024-06-29T14:41:21.439937+0000","last_undegraded":"2024-06-29T14:41:21.439937+0000","last_fullsized":"2024-06-29T14:41:21.439937+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:33:42.213193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425529+0000","last_change":"2024-06-29T14:41:21.438179+0000","last_active":"2024-06-29T14:41:22.425529+0000","last_peered":"2024-06-29T14:41:22.425529+0000","last_clean":"2024-06-29T14:41:22.425529+0000","last_became_active":"2024-06-29T14:41:21.437804+0000","last_became_peered":"2024-06-29T14:41:21.437804+0000","last_unstale":"2024-06-29T14:41:22.425529+0000","last_undegraded":"2024-06-29T14:41:22.425529+0000","last_fullsized":"2024-06-29T14:41:22.425529+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:45:58.642581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425674+0000","last_change":"2024-06-29T14:41:21.432638+0000","last_active":"2024-06-29T14:41:22.425674+0000","last_peered":"2024-06-29T14:41:22.425674+0000","last_clean":"2024-06-29T14:41:22.425674+0000","last_became_active":"2024-06-29T14:41:21.432281+0000","last_became_peered":"2024-06-29T14:41:21.432281+0000","last_unstale":"2024-06-29T14:41:22.425674+0000","last_undegraded":"2024-06-29T14:41:22.425674+0000","last_fullsized":"2024-06-29T14:41:22.425674+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:27:03.389186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439628+0000","last_change":"2024-06-29T14:41:21.422385+0000","last_active":"2024-06-29T14:41:21.439628+0000","last_peered":"2024-06-29T14:41:21.439628+0000","last_clean":"2024-06-29T14:41:21.439628+0000","last_became_active":"2024-06-29T14:41:21.422061+0000","last_became_peered":"2024-06-29T14:41:21.422061+0000","last_unstale":"2024-06-29T14:41:21.439628+0000","last_undegraded":"2024-06-29T14:41:21.439628+0000","last_fullsized":"2024-06-29T14:41:21.439628+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:21.208186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425977+0000","last_change":"2024-06-29T14:41:21.419016+0000","last_active":"2024-06-29T14:41:22.425977+0000","last_peered":"2024-06-29T14:41:22.425977+0000","last_clean":"2024-06-29T14:41:22.425977+0000","last_became_active":"2024-06-29T14:41:21.418663+0000","last_became_peered":"2024-06-29T14:41:21.418663+0000","last_unstale":"2024-06-29T14:41:22.425977+0000","last_undegraded":"2024-06-29T14:41:22.425977+0000","last_fullsized":"2024-06-29T14:41:22.425977+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:35.235758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.448800+0000","last_change":"2024-06-29T14:41:21.437211+0000","last_active":"2024-06-29T14:41:21.448800+0000","last_peered":"2024-06-29T14:41:21.448800+0000","last_clean":"2024-06-29T14:41:21.448800+0000","last_became_active":"2024-06-29T14:41:21.436885+0000","last_became_peered":"2024-06-29T14:41:21.436885+0000","last_unstale":"2024-06-29T14:41:21.448800+0000","last_undegraded":"2024-06-29T14:41:21.448800+0000","last_fullsized":"2024-06-29T14:41:21.448800+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:11.665514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439913+0000","last_change":"2024-06-29T14:41:21.436287+0000","last_active":"2024-06-29T14:41:21.439913+0000","last_peered":"2024-06-29T14:41:21.439913+0000","last_clean":"2024-06-29T14:41:21.439913+0000","last_became_active":"2024-06-29T14:41:21.435356+0000","last_became_peered":"2024-06-29T14:41:21.435356+0000","last_unstale":"2024-06-29T14:41:21.439913+0000","last_undegraded":"2024-06-29T14:41:21.439913+0000","last_fullsized":"2024-06-29T14:41:21.439913+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:57.964925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.436340+0000","last_change":"2024-06-29T14:41:21.435488+0000","last_active":"2024-06-29T14:41:21.436340+0000","last_peered":"2024-06-29T14:41:21.436340+0000","last_clean":"2024-06-29T14:41:21.436340+0000","last_became_active":"2024-06-29T14:41:21.435098+0000","last_became_peered":"2024-06-29T14:41:21.435098+0000","last_unstale":"2024-06-29T14:41:21.436340+0000","last_undegraded":"2024-06-29T14:41:21.436340+0000","last_fullsized":"2024-06-29T14:41:21.436340+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:14:45.301566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.426299+0000","last_change":"2024-06-29T14:41:21.424553+0000","last_active":"2024-06-29T14:41:21.426299+0000","last_peered":"2024-06-29T14:41:21.426299+0000","last_clean":"2024-06-29T14:41:21.426299+0000","last_became_active":"2024-06-29T14:41:21.424176+0000","last_became_peered":"2024-06-29T14:41:21.424176+0000","last_unstale":"2024-06-29T14:41:21.426299+0000","last_undegraded":"2024-06-29T14:41:21.426299+0000","last_fullsized":"2024-06-29T14:41:21.426299+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:43:57.320978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.433989+0000","last_change":"2024-06-29T14:41:21.433206+0000","last_active":"2024-06-29T14:41:21.433989+0000","last_peered":"2024-06-29T14:41:21.433989+0000","last_clean":"2024-06-29T14:41:21.433989+0000","last_became_active":"2024-06-29T14:41:21.432123+0000","last_became_peered":"2024-06-29T14:41:21.432123+0000","last_unstale":"2024-06-29T14:41:21.433989+0000","last_undegraded":"2024-06-29T14:41:21.433989+0000","last_fullsized":"2024-06-29T14:41:21.433989+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:28.594542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.433288+0000","last_change":"2024-06-29T14:41:21.431360+0000","last_active":"2024-06-29T14:41:21.433288+0000","last_peered":"2024-06-29T14:41:21.433288+0000","last_clean":"2024-06-29T14:41:21.433288+0000","last_became_active":"2024-06-29T14:41:21.430202+0000","last_became_peered":"2024-06-29T14:41:21.430202+0000","last_unstale":"2024-06-29T14:41:21.433288+0000","last_undegraded":"2024-06-29T14:41:21.433288+0000","last_fullsized":"2024-06-29T14:41:21.433288+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:30.495511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.448751+0000","last_change":"2024-06-29T14:41:21.437704+0000","last_active":"2024-06-29T14:41:21.448751+0000","last_peered":"2024-06-29T14:41:21.448751+0000","last_clean":"2024-06-29T14:41:21.448751+0000","last_became_active":"2024-06-29T14:41:21.437329+0000","last_became_peered":"2024-06-29T14:41:21.437329+0000","last_unstale":"2024-06-29T14:41:21.448751+0000","last_undegraded":"2024-06-29T14:41:21.448751+0000","last_fullsized":"2024-06-29T14:41:21.448751+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:32.365851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426002+0000","last_change":"2024-06-29T14:41:21.419002+0000","last_active":"2024-06-29T14:41:22.426002+0000","last_peered":"2024-06-29T14:41:22.426002+0000","last_clean":"2024-06-29T14:41:22.426002+0000","last_became_active":"2024-06-29T14:41:21.418637+0000","last_became_peered":"2024-06-29T14:41:21.418637+0000","last_unstale":"2024-06-29T14:41:22.426002+0000","last_undegraded":"2024-06-29T14:41:22.426002+0000","last_fullsized":"2024-06-29T14:41:22.426002+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:46.771288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426695+0000","last_change":"2024-06-29T14:41:21.435163+0000","last_active":"2024-06-29T14:41:22.426695+0000","last_peered":"2024-06-29T14:41:22.426695+0000","last_clean":"2024-06-29T14:41:22.426695+0000","last_became_active":"2024-06-29T14:41:21.434814+0000","last_became_peered":"2024-06-29T14:41:21.434814+0000","last_unstale":"2024-06-29T14:41:22.426695+0000","last_undegraded":"2024-06-29T14:41:22.426695+0000","last_fullsized":"2024-06-29T14:41:22.426695+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:33.399206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.448837+0000","last_change":"2024-06-29T14:41:21.437020+0000","last_active":"2024-06-29T14:41:21.448837+0000","last_peered":"2024-06-29T14:41:21.448837+0000","last_clean":"2024-06-29T14:41:21.448837+0000","last_became_active":"2024-06-29T14:41:21.435398+0000","last_became_peered":"2024-06-29T14:41:21.435398+0000","last_unstale":"2024-06-29T14:41:21.448837+0000","last_undegraded":"2024-06-29T14:41:21.448837+0000","last_fullsized":"2024-06-29T14:41:21.448837+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:44.481684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446271+0000","last_change":"2024-06-29T14:41:21.432746+0000","last_active":"2024-06-29T14:41:21.446271+0000","last_peered":"2024-06-29T14:41:21.446271+0000","last_clean":"2024-06-29T14:41:21.446271+0000","last_became_active":"2024-06-29T14:41:21.432190+0000","last_became_peered":"2024-06-29T14:41:21.432190+0000","last_unstale":"2024-06-29T14:41:21.446271+0000","last_undegraded":"2024-06-29T14:41:21.446271+0000","last_fullsized":"2024-06-29T14:41:21.446271+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:10.402070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.431891+0000","last_change":"2024-06-29T14:41:21.430220+0000","last_active":"2024-06-29T14:41:21.431891+0000","last_peered":"2024-06-29T14:41:21.431891+0000","last_clean":"2024-06-29T14:41:21.431891+0000","last_became_active":"2024-06-29T14:41:21.429853+0000","last_became_peered":"2024-06-29T14:41:21.429853+0000","last_unstale":"2024-06-29T14:41:21.431891+0000","last_undegraded":"2024-06-29T14:41:21.431891+0000","last_fullsized":"2024-06-29T14:41:21.431891+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:03:26.279064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.433950+0000","last_change":"2024-06-29T14:41:21.432543+0000","last_active":"2024-06-29T14:41:21.433950+0000","last_peered":"2024-06-29T14:41:21.433950+0000","last_clean":"2024-06-29T14:41:21.433950+0000","last_became_active":"2024-06-29T14:41:21.432021+0000","last_became_peered":"2024-06-29T14:41:21.432021+0000","last_unstale":"2024-06-29T14:41:21.433950+0000","last_undegraded":"2024-06-29T14:41:21.433950+0000","last_fullsized":"2024-06-29T14:41:21.433950+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:28.987473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426187+0000","last_change":"2024-06-29T14:41:21.441622+0000","last_active":"2024-06-29T14:41:22.426187+0000","last_peered":"2024-06-29T14:41:22.426187+0000","last_clean":"2024-06-29T14:41:22.426187+0000","last_became_active":"2024-06-29T14:41:21.441192+0000","last_became_peered":"2024-06-29T14:41:21.441192+0000","last_unstale":"2024-06-29T14:41:22.426187+0000","last_undegraded":"2024-06-29T14:41:22.426187+0000","last_fullsized":"2024-06-29T14:41:22.426187+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:40:13.615563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"142'4","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425792+0000","last_change":"2024-06-29T14:41:20.413102+0000","last_active":"2024-06-29T14:41:22.425792+0000","last_peered":"2024-06-29T14:41:22.425792+0000","last_clean":"2024-06-29T14:41:22.425792+0000","last_became_active":"2024-06-29T14:41:20.412752+0000","last_became_peered":"2024-06-29T14:41:20.412752+0000","last_unstale":"2024-06-29T14:41:22.425792+0000","last_undegraded":"2024-06-29T14:41:22.425792+0000","last_fullsized":"2024-06-29T14:41:22.425792+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:38.247088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426437+0000","last_change":"2024-06-29T14:41:21.443247+0000","last_active":"2024-06-29T14:41:22.426437+0000","last_peered":"2024-06-29T14:41:22.426437+0000","last_clean":"2024-06-29T14:41:22.426437+0000","last_became_active":"2024-06-29T14:41:21.442913+0000","last_became_peered":"2024-06-29T14:41:21.442913+0000","last_unstale":"2024-06-29T14:41:22.426437+0000","last_undegraded":"2024-06-29T14:41:22.426437+0000","last_fullsized":"2024-06-29T14:41:22.426437+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:38:32.792002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"142'1","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426021+0000","last_change":"2024-06-29T14:41:20.410994+0000","last_active":"2024-06-29T14:41:22.426021+0000","last_peered":"2024-06-29T14:41:22.426021+0000","last_clean":"2024-06-29T14:41:22.426021+0000","last_became_active":"2024-06-29T14:41:20.410665+0000","last_became_peered":"2024-06-29T14:41:20.410665+0000","last_unstale":"2024-06-29T14:41:22.426021+0000","last_undegraded":"2024-06-29T14:41:22.426021+0000","last_fullsized":"2024-06-29T14:41:22.426021+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:09.848965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.444128+0000","last_change":"2024-06-29T14:41:21.438943+0000","last_active":"2024-06-29T14:41:21.444128+0000","last_peered":"2024-06-29T14:41:21.444128+0000","last_clean":"2024-06-29T14:41:21.444128+0000","last_became_active":"2024-06-29T14:41:21.438650+0000","last_became_peered":"2024-06-29T14:41:21.438650+0000","last_unstale":"2024-06-29T14:41:21.444128+0000","last_undegraded":"2024-06-29T14:41:21.444128+0000","last_fullsized":"2024-06-29T14:41:21.444128+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:10.229870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"142'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.450810+0000","last_change":"2024-06-29T14:41:20.416624+0000","last_active":"2024-06-29T14:41:21.450810+0000","last_peered":"2024-06-29T14:41:21.450810+0000","last_clean":"2024-06-29T14:41:21.450810+0000","last_became_active":"2024-06-29T14:41:20.416297+0000","last_became_peered":"2024-06-29T14:41:20.416297+0000","last_unstale":"2024-06-29T14:41:21.450810+0000","last_undegraded":"2024-06-29T14:41:21.450810+0000","last_fullsized":"2024-06-29T14:41:21.450810+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:21.833779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.435058+0000","last_change":"2024-06-29T14:41:21.428280+0000","last_active":"2024-06-29T14:41:21.435058+0000","last_peered":"2024-06-29T14:41:21.435058+0000","last_clean":"2024-06-29T14:41:21.435058+0000","last_became_active":"2024-06-29T14:41:21.427880+0000","last_became_peered":"2024-06-29T14:41:21.427880+0000","last_unstale":"2024-06-29T14:41:21.435058+0000","last_undegraded":"2024-06-29T14:41:21.435058+0000","last_fullsized":"2024-06-29T14:41:21.435058+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:26.036865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"142'2","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426011+0000","last_change":"2024-06-29T14:41:20.412590+0000","last_active":"2024-06-29T14:41:22.426011+0000","last_peered":"2024-06-29T14:41:22.426011+0000","last_clean":"2024-06-29T14:41:22.426011+0000","last_became_active":"2024-06-29T14:41:20.412276+0000","last_became_peered":"2024-06-29T14:41:20.412276+0000","last_unstale":"2024-06-29T14:41:22.426011+0000","last_undegraded":"2024-06-29T14:41:22.426011+0000","last_fullsized":"2024-06-29T14:41:22.426011+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:46.271143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.424980+0000","last_change":"2024-06-29T14:41:21.436350+0000","last_active":"2024-06-29T14:41:22.424980+0000","last_peered":"2024-06-29T14:41:22.424980+0000","last_clean":"2024-06-29T14:41:22.424980+0000","last_became_active":"2024-06-29T14:41:21.435872+0000","last_became_peered":"2024-06-29T14:41:21.435872+0000","last_unstale":"2024-06-29T14:41:22.424980+0000","last_undegraded":"2024-06-29T14:41:22.424980+0000","last_fullsized":"2024-06-29T14:41:22.424980+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:07.470915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"142'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.453304+0000","last_change":"2024-06-29T14:41:20.418669+0000","last_active":"2024-06-29T14:41:21.453304+0000","last_peered":"2024-06-29T14:41:21.453304+0000","last_clean":"2024-06-29T14:41:21.453304+0000","last_became_active":"2024-06-29T14:41:20.418380+0000","last_became_peered":"2024-06-29T14:41:20.418380+0000","last_unstale":"2024-06-29T14:41:21.453304+0000","last_undegraded":"2024-06-29T14:41:21.453304+0000","last_fullsized":"2024-06-29T14:41:21.453304+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:17.054703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.429284+0000","last_change":"2024-06-29T14:41:21.427521+0000","last_active":"2024-06-29T14:41:21.429284+0000","last_peered":"2024-06-29T14:41:21.429284+0000","last_clean":"2024-06-29T14:41:21.429284+0000","last_became_active":"2024-06-29T14:41:21.427197+0000","last_became_peered":"2024-06-29T14:41:21.427197+0000","last_unstale":"2024-06-29T14:41:21.429284+0000","last_undegraded":"2024-06-29T14:41:21.429284+0000","last_fullsized":"2024-06-29T14:41:21.429284+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:22.323516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"142'1","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426208+0000","last_change":"2024-06-29T14:41:20.423395+0000","last_active":"2024-06-29T14:41:22.426208+0000","last_peered":"2024-06-29T14:41:22.426208+0000","last_clean":"2024-06-29T14:41:22.426208+0000","last_became_active":"2024-06-29T14:41:20.423044+0000","last_became_peered":"2024-06-29T14:41:20.423044+0000","last_unstale":"2024-06-29T14:41:22.426208+0000","last_undegraded":"2024-06-29T14:41:22.426208+0000","last_fullsized":"2024-06-29T14:41:22.426208+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:42.806858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446299+0000","last_change":"2024-06-29T14:41:21.433522+0000","last_active":"2024-06-29T14:41:21.446299+0000","last_peered":"2024-06-29T14:41:21.446299+0000","last_clean":"2024-06-29T14:41:21.446299+0000","last_became_active":"2024-06-29T14:41:21.433171+0000","last_became_peered":"2024-06-29T14:41:21.433171+0000","last_unstale":"2024-06-29T14:41:21.446299+0000","last_undegraded":"2024-06-29T14:41:21.446299+0000","last_fullsized":"2024-06-29T14:41:21.446299+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:18.242404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.425120+0000","last_change":"2024-06-29T14:41:20.412134+0000","last_active":"2024-06-29T14:41:21.425120+0000","last_peered":"2024-06-29T14:41:21.425120+0000","last_clean":"2024-06-29T14:41:21.425120+0000","last_became_active":"2024-06-29T14:41:20.411776+0000","last_became_peered":"2024-06-29T14:41:20.411776+0000","last_unstale":"2024-06-29T14:41:21.425120+0000","last_undegraded":"2024-06-29T14:41:21.425120+0000","last_fullsized":"2024-06-29T14:41:21.425120+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:49.488844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425125+0000","last_change":"2024-06-29T14:41:21.433915+0000","last_active":"2024-06-29T14:41:22.425125+0000","last_peered":"2024-06-29T14:41:22.425125+0000","last_clean":"2024-06-29T14:41:22.425125+0000","last_became_active":"2024-06-29T14:41:21.433596+0000","last_became_peered":"2024-06-29T14:41:21.433596+0000","last_unstale":"2024-06-29T14:41:22.425125+0000","last_undegraded":"2024-06-29T14:41:22.425125+0000","last_fullsized":"2024-06-29T14:41:22.425125+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:33.742338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425711+0000","last_change":"2024-06-29T14:41:20.421426+0000","last_active":"2024-06-29T14:41:22.425711+0000","last_peered":"2024-06-29T14:41:22.425711+0000","last_clean":"2024-06-29T14:41:22.425711+0000","last_became_active":"2024-06-29T14:41:20.421092+0000","last_became_peered":"2024-06-29T14:41:20.421092+0000","last_unstale":"2024-06-29T14:41:22.425711+0000","last_undegraded":"2024-06-29T14:41:22.425711+0000","last_fullsized":"2024-06-29T14:41:22.425711+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:47.861510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425602+0000","last_change":"2024-06-29T14:41:21.418489+0000","last_active":"2024-06-29T14:41:22.425602+0000","last_peered":"2024-06-29T14:41:22.425602+0000","last_clean":"2024-06-29T14:41:22.425602+0000","last_became_active":"2024-06-29T14:41:21.418142+0000","last_became_peered":"2024-06-29T14:41:21.418142+0000","last_unstale":"2024-06-29T14:41:22.425602+0000","last_undegraded":"2024-06-29T14:41:22.425602+0000","last_fullsized":"2024-06-29T14:41:22.425602+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:26:42.188988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425780+0000","last_change":"2024-06-29T14:41:20.417276+0000","last_active":"2024-06-29T14:41:22.425780+0000","last_peered":"2024-06-29T14:41:22.425780+0000","last_clean":"2024-06-29T14:41:22.425780+0000","last_became_active":"2024-06-29T14:41:20.416974+0000","last_became_peered":"2024-06-29T14:41:20.416974+0000","last_unstale":"2024-06-29T14:41:22.425780+0000","last_undegraded":"2024-06-29T14:41:22.425780+0000","last_fullsized":"2024-06-29T14:41:22.425780+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:17.054703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.434525+0000","last_change":"2024-06-29T14:41:21.433462+0000","last_active":"2024-06-29T14:41:21.434525+0000","last_peered":"2024-06-29T14:41:21.434525+0000","last_clean":"2024-06-29T14:41:21.434525+0000","last_became_active":"2024-06-29T14:41:21.433151+0000","last_became_peered":"2024-06-29T14:41:21.433151+0000","last_unstale":"2024-06-29T14:41:21.434525+0000","last_undegraded":"2024-06-29T14:41:21.434525+0000","last_fullsized":"2024-06-29T14:41:21.434525+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:43.813132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.425075+0000","last_change":"2024-06-29T14:41:20.411001+0000","last_active":"2024-06-29T14:41:21.425075+0000","last_peered":"2024-06-29T14:41:21.425075+0000","last_clean":"2024-06-29T14:41:21.425075+0000","last_became_active":"2024-06-29T14:41:20.410641+0000","last_became_peered":"2024-06-29T14:41:20.410641+0000","last_unstale":"2024-06-29T14:41:21.425075+0000","last_undegraded":"2024-06-29T14:41:21.425075+0000","last_fullsized":"2024-06-29T14:41:21.425075+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:50:37.352177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.427997+0000","last_change":"2024-06-29T14:41:21.420840+0000","last_active":"2024-06-29T14:41:21.427997+0000","last_peered":"2024-06-29T14:41:21.427997+0000","last_clean":"2024-06-29T14:41:21.427997+0000","last_became_active":"2024-06-29T14:41:21.420493+0000","last_became_peered":"2024-06-29T14:41:21.420493+0000","last_unstale":"2024-06-29T14:41:21.427997+0000","last_undegraded":"2024-06-29T14:41:21.427997+0000","last_fullsized":"2024-06-29T14:41:21.427997+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:44.946059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"142'3","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446625+0000","last_change":"2024-06-29T14:41:20.411557+0000","last_active":"2024-06-29T14:41:21.446625+0000","last_peered":"2024-06-29T14:41:21.446625+0000","last_clean":"2024-06-29T14:41:21.446625+0000","last_became_active":"2024-06-29T14:41:20.411264+0000","last_became_peered":"2024-06-29T14:41:20.411264+0000","last_unstale":"2024-06-29T14:41:21.446625+0000","last_undegraded":"2024-06-29T14:41:21.446625+0000","last_fullsized":"2024-06-29T14:41:21.446625+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:19.992312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.447002+0000","last_change":"2024-06-29T14:41:21.446570+0000","last_active":"2024-06-29T14:41:21.447002+0000","last_peered":"2024-06-29T14:41:21.447002+0000","last_clean":"2024-06-29T14:41:21.447002+0000","last_became_active":"2024-06-29T14:41:21.446099+0000","last_became_peered":"2024-06-29T14:41:21.446099+0000","last_unstale":"2024-06-29T14:41:21.447002+0000","last_undegraded":"2024-06-29T14:41:21.447002+0000","last_fullsized":"2024-06-29T14:41:21.447002+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.430952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"142'6","reported_seq":18,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.455150+0000","last_change":"2024-06-29T14:41:20.416297+0000","last_active":"2024-06-29T14:41:21.455150+0000","last_peered":"2024-06-29T14:41:21.455150+0000","last_clean":"2024-06-29T14:41:21.455150+0000","last_became_active":"2024-06-29T14:41:20.416005+0000","last_became_peered":"2024-06-29T14:41:20.416005+0000","last_unstale":"2024-06-29T14:41:21.455150+0000","last_undegraded":"2024-06-29T14:41:21.455150+0000","last_fullsized":"2024-06-29T14:41:21.455150+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:38.631988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426161+0000","last_change":"2024-06-29T14:41:21.422586+0000","last_active":"2024-06-29T14:41:22.426161+0000","last_peered":"2024-06-29T14:41:22.426161+0000","last_clean":"2024-06-29T14:41:22.426161+0000","last_became_active":"2024-06-29T14:41:21.422287+0000","last_became_peered":"2024-06-29T14:41:21.422287+0000","last_unstale":"2024-06-29T14:41:22.426161+0000","last_undegraded":"2024-06-29T14:41:22.426161+0000","last_fullsized":"2024-06-29T14:41:22.426161+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:58:05.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"142'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.449801+0000","last_change":"2024-06-29T14:41:20.418823+0000","last_active":"2024-06-29T14:41:21.449801+0000","last_peered":"2024-06-29T14:41:21.449801+0000","last_clean":"2024-06-29T14:41:21.449801+0000","last_became_active":"2024-06-29T14:41:20.418526+0000","last_became_peered":"2024-06-29T14:41:20.418526+0000","last_unstale":"2024-06-29T14:41:21.449801+0000","last_undegraded":"2024-06-29T14:41:21.449801+0000","last_fullsized":"2024-06-29T14:41:21.449801+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:29.886679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425034+0000","last_change":"2024-06-29T14:41:21.435826+0000","last_active":"2024-06-29T14:41:22.425034+0000","last_peered":"2024-06-29T14:41:22.425034+0000","last_clean":"2024-06-29T14:41:22.425034+0000","last_became_active":"2024-06-29T14:41:21.435293+0000","last_became_peered":"2024-06-29T14:41:21.435293+0000","last_unstale":"2024-06-29T14:41:22.425034+0000","last_undegraded":"2024-06-29T14:41:22.425034+0000","last_fullsized":"2024-06-29T14:41:22.425034+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:16.572821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.414888+0000","last_change":"2024-06-29T14:41:20.413868+0000","last_active":"2024-06-29T14:41:21.414888+0000","last_peered":"2024-06-29T14:41:21.414888+0000","last_clean":"2024-06-29T14:41:21.414888+0000","last_became_active":"2024-06-29T14:41:20.413540+0000","last_became_peered":"2024-06-29T14:41:20.413540+0000","last_unstale":"2024-06-29T14:41:21.414888+0000","last_undegraded":"2024-06-29T14:41:21.414888+0000","last_fullsized":"2024-06-29T14:41:21.414888+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:28:06.006143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.431573+0000","last_change":"2024-06-29T14:41:21.429057+0000","last_active":"2024-06-29T14:41:21.431573+0000","last_peered":"2024-06-29T14:41:21.431573+0000","last_clean":"2024-06-29T14:41:21.431573+0000","last_became_active":"2024-06-29T14:41:21.428695+0000","last_became_peered":"2024-06-29T14:41:21.428695+0000","last_unstale":"2024-06-29T14:41:21.431573+0000","last_undegraded":"2024-06-29T14:41:21.431573+0000","last_fullsized":"2024-06-29T14:41:21.431573+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:14.285071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.413805+0000","last_change":"2024-06-29T14:41:20.415218+0000","last_active":"2024-06-29T14:41:21.413805+0000","last_peered":"2024-06-29T14:41:21.413805+0000","last_clean":"2024-06-29T14:41:21.413805+0000","last_became_active":"2024-06-29T14:41:20.412865+0000","last_became_peered":"2024-06-29T14:41:20.412865+0000","last_unstale":"2024-06-29T14:41:21.413805+0000","last_undegraded":"2024-06-29T14:41:21.413805+0000","last_fullsized":"2024-06-29T14:41:21.413805+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:28.404030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.431948+0000","last_change":"2024-06-29T14:41:21.429817+0000","last_active":"2024-06-29T14:41:21.431948+0000","last_peered":"2024-06-29T14:41:21.431948+0000","last_clean":"2024-06-29T14:41:21.431948+0000","last_became_active":"2024-06-29T14:41:21.429513+0000","last_became_peered":"2024-06-29T14:41:21.429513+0000","last_unstale":"2024-06-29T14:41:21.431948+0000","last_undegraded":"2024-06-29T14:41:21.431948+0000","last_fullsized":"2024-06-29T14:41:21.431948+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:18.242404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"142'1","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425684+0000","last_change":"2024-06-29T14:41:20.419691+0000","last_active":"2024-06-29T14:41:22.425684+0000","last_peered":"2024-06-29T14:41:22.425684+0000","last_clean":"2024-06-29T14:41:22.425684+0000","last_became_active":"2024-06-29T14:41:20.419297+0000","last_became_peered":"2024-06-29T14:41:20.419297+0000","last_unstale":"2024-06-29T14:41:22.425684+0000","last_undegraded":"2024-06-29T14:41:22.425684+0000","last_fullsized":"2024-06-29T14:41:22.425684+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:46.716724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.444046+0000","last_change":"2024-06-29T14:41:21.443431+0000","last_active":"2024-06-29T14:41:21.444046+0000","last_peered":"2024-06-29T14:41:21.444046+0000","last_clean":"2024-06-29T14:41:21.444046+0000","last_became_active":"2024-06-29T14:41:21.442501+0000","last_became_peered":"2024-06-29T14:41:21.442501+0000","last_unstale":"2024-06-29T14:41:21.444046+0000","last_undegraded":"2024-06-29T14:41:21.444046+0000","last_fullsized":"2024-06-29T14:41:21.444046+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:26:42.188988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"142'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.451239+0000","last_change":"2024-06-29T14:41:20.413941+0000","last_active":"2024-06-29T14:41:21.451239+0000","last_peered":"2024-06-29T14:41:21.451239+0000","last_clean":"2024-06-29T14:41:21.451239+0000","last_became_active":"2024-06-29T14:41:20.413489+0000","last_became_peered":"2024-06-29T14:41:20.413489+0000","last_unstale":"2024-06-29T14:41:21.451239+0000","last_undegraded":"2024-06-29T14:41:21.451239+0000","last_fullsized":"2024-06-29T14:41:21.451239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:25.307752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426646+0000","last_change":"2024-06-29T14:41:21.439759+0000","last_active":"2024-06-29T14:41:22.426646+0000","last_peered":"2024-06-29T14:41:22.426646+0000","last_clean":"2024-06-29T14:41:22.426646+0000","last_became_active":"2024-06-29T14:41:21.438457+0000","last_became_peered":"2024-06-29T14:41:21.438457+0000","last_unstale":"2024-06-29T14:41:22.426646+0000","last_undegraded":"2024-06-29T14:41:22.426646+0000","last_fullsized":"2024-06-29T14:41:22.426646+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:49:10.598608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"142'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446729+0000","last_change":"2024-06-29T14:41:20.415077+0000","last_active":"2024-06-29T14:41:21.446729+0000","last_peered":"2024-06-29T14:41:21.446729+0000","last_clean":"2024-06-29T14:41:21.446729+0000","last_became_active":"2024-06-29T14:41:20.414748+0000","last_became_peered":"2024-06-29T14:41:20.414748+0000","last_unstale":"2024-06-29T14:41:21.446729+0000","last_undegraded":"2024-06-29T14:41:21.446729+0000","last_fullsized":"2024-06-29T14:41:21.446729+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:15.607913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.431641+0000","last_change":"2024-06-29T14:41:21.428709+0000","last_active":"2024-06-29T14:41:21.431641+0000","last_peered":"2024-06-29T14:41:21.431641+0000","last_clean":"2024-06-29T14:41:21.431641+0000","last_became_active":"2024-06-29T14:41:21.428293+0000","last_became_peered":"2024-06-29T14:41:21.428293+0000","last_unstale":"2024-06-29T14:41:21.431641+0000","last_undegraded":"2024-06-29T14:41:21.431641+0000","last_fullsized":"2024-06-29T14:41:21.431641+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:10.402070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.413696+0000","last_change":"2024-06-29T14:41:20.417402+0000","last_active":"2024-06-29T14:41:21.413696+0000","last_peered":"2024-06-29T14:41:21.413696+0000","last_clean":"2024-06-29T14:41:21.413696+0000","last_became_active":"2024-06-29T14:41:20.416938+0000","last_became_peered":"2024-06-29T14:41:20.416938+0000","last_unstale":"2024-06-29T14:41:21.413696+0000","last_undegraded":"2024-06-29T14:41:21.413696+0000","last_fullsized":"2024-06-29T14:41:21.413696+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:05.059264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425592+0000","last_change":"2024-06-29T14:41:21.431968+0000","last_active":"2024-06-29T14:41:22.425592+0000","last_peered":"2024-06-29T14:41:22.425592+0000","last_clean":"2024-06-29T14:41:22.425592+0000","last_became_active":"2024-06-29T14:41:21.431636+0000","last_became_peered":"2024-06-29T14:41:21.431636+0000","last_unstale":"2024-06-29T14:41:22.425592+0000","last_undegraded":"2024-06-29T14:41:22.425592+0000","last_fullsized":"2024-06-29T14:41:22.425592+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:36:08.237326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"142'2","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426220+0000","last_change":"2024-06-29T14:41:20.416570+0000","last_active":"2024-06-29T14:41:22.426220+0000","last_peered":"2024-06-29T14:41:22.426220+0000","last_clean":"2024-06-29T14:41:22.426220+0000","last_became_active":"2024-06-29T14:41:20.416245+0000","last_became_peered":"2024-06-29T14:41:20.416245+0000","last_unstale":"2024-06-29T14:41:22.426220+0000","last_undegraded":"2024-06-29T14:41:22.426220+0000","last_fullsized":"2024-06-29T14:41:22.426220+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:27.588268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.444144+0000","last_change":"2024-06-29T14:41:21.431994+0000","last_active":"2024-06-29T14:41:21.444144+0000","last_peered":"2024-06-29T14:41:21.444144+0000","last_clean":"2024-06-29T14:41:21.444144+0000","last_became_active":"2024-06-29T14:41:21.430221+0000","last_became_peered":"2024-06-29T14:41:21.430221+0000","last_unstale":"2024-06-29T14:41:21.444144+0000","last_undegraded":"2024-06-29T14:41:21.444144+0000","last_fullsized":"2024-06-29T14:41:21.444144+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:50:38.358451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.412403+0000","last_change":"2024-06-29T14:41:20.413126+0000","last_active":"2024-06-29T14:41:21.412403+0000","last_peered":"2024-06-29T14:41:21.412403+0000","last_clean":"2024-06-29T14:41:21.412403+0000","last_became_active":"2024-06-29T14:41:20.412807+0000","last_became_peered":"2024-06-29T14:41:20.412807+0000","last_unstale":"2024-06-29T14:41:21.412403+0000","last_undegraded":"2024-06-29T14:41:21.412403+0000","last_fullsized":"2024-06-29T14:41:21.412403+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:19.142792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.441298+0000","last_change":"2024-06-29T14:41:21.426823+0000","last_active":"2024-06-29T14:41:21.441298+0000","last_peered":"2024-06-29T14:41:21.441298+0000","last_clean":"2024-06-29T14:41:21.441298+0000","last_became_active":"2024-06-29T14:41:21.426433+0000","last_became_peered":"2024-06-29T14:41:21.426433+0000","last_unstale":"2024-06-29T14:41:21.441298+0000","last_undegraded":"2024-06-29T14:41:21.441298+0000","last_fullsized":"2024-06-29T14:41:21.441298+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:50.495118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.414165+0000","last_change":"2024-06-29T14:41:20.421115+0000","last_active":"2024-06-29T14:41:21.414165+0000","last_peered":"2024-06-29T14:41:21.414165+0000","last_clean":"2024-06-29T14:41:21.414165+0000","last_became_active":"2024-06-29T14:41:20.420750+0000","last_became_peered":"2024-06-29T14:41:20.420750+0000","last_unstale":"2024-06-29T14:41:21.414165+0000","last_undegraded":"2024-06-29T14:41:21.414165+0000","last_fullsized":"2024-06-29T14:41:21.414165+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:04.578774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.445998+0000","last_change":"2024-06-29T14:41:21.444979+0000","last_active":"2024-06-29T14:41:21.445998+0000","last_peered":"2024-06-29T14:41:21.445998+0000","last_clean":"2024-06-29T14:41:21.445998+0000","last_became_active":"2024-06-29T14:41:21.444624+0000","last_became_peered":"2024-06-29T14:41:21.444624+0000","last_unstale":"2024-06-29T14:41:21.445998+0000","last_undegraded":"2024-06-29T14:41:21.445998+0000","last_fullsized":"2024-06-29T14:41:21.445998+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:48.867784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.412104+0000","last_change":"2024-06-29T14:41:20.416766+0000","last_active":"2024-06-29T14:41:21.412104+0000","last_peered":"2024-06-29T14:41:21.412104+0000","last_clean":"2024-06-29T14:41:21.412104+0000","last_became_active":"2024-06-29T14:41:20.416418+0000","last_became_peered":"2024-06-29T14:41:20.416418+0000","last_unstale":"2024-06-29T14:41:21.412104+0000","last_undegraded":"2024-06-29T14:41:21.412104+0000","last_fullsized":"2024-06-29T14:41:21.412104+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:45.259800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426717+0000","last_change":"2024-06-29T14:41:21.442681+0000","last_active":"2024-06-29T14:41:22.426717+0000","last_peered":"2024-06-29T14:41:22.426717+0000","last_clean":"2024-06-29T14:41:22.426717+0000","last_became_active":"2024-06-29T14:41:21.442334+0000","last_became_peered":"2024-06-29T14:41:21.442334+0000","last_unstale":"2024-06-29T14:41:22.426717+0000","last_undegraded":"2024-06-29T14:41:22.426717+0000","last_fullsized":"2024-06-29T14:41:22.426717+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:56.871226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"142'2","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426523+0000","last_change":"2024-06-29T14:41:20.418325+0000","last_active":"2024-06-29T14:41:22.426523+0000","last_peered":"2024-06-29T14:41:22.426523+0000","last_clean":"2024-06-29T14:41:22.426523+0000","last_became_active":"2024-06-29T14:41:20.417963+0000","last_became_peered":"2024-06-29T14:41:20.417963+0000","last_unstale":"2024-06-29T14:41:22.426523+0000","last_undegraded":"2024-06-29T14:41:22.426523+0000","last_fullsized":"2024-06-29T14:41:22.426523+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:09.223595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425551+0000","last_change":"2024-06-29T14:41:21.436827+0000","last_active":"2024-06-29T14:41:22.425551+0000","last_peered":"2024-06-29T14:41:22.425551+0000","last_clean":"2024-06-29T14:41:22.425551+0000","last_became_active":"2024-06-29T14:41:21.436028+0000","last_became_peered":"2024-06-29T14:41:21.436028+0000","last_unstale":"2024-06-29T14:41:22.425551+0000","last_undegraded":"2024-06-29T14:41:22.425551+0000","last_fullsized":"2024-06-29T14:41:22.425551+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:18:51.081360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"143'1","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.438712+0000","last_change":"2024-06-29T14:41:20.421040+0000","last_active":"2024-06-29T14:41:22.438712+0000","last_peered":"2024-06-29T14:41:22.438712+0000","last_clean":"2024-06-29T14:41:22.438712+0000","last_became_active":"2024-06-29T14:41:20.420781+0000","last_became_peered":"2024-06-29T14:41:20.420781+0000","last_unstale":"2024-06-29T14:41:22.438712+0000","last_undegraded":"2024-06-29T14:41:22.438712+0000","last_fullsized":"2024-06-29T14:41:22.438712+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:23.360766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.434693+0000","last_change":"2024-06-29T14:41:21.428481+0000","last_active":"2024-06-29T14:41:21.434693+0000","last_peered":"2024-06-29T14:41:21.434693+0000","last_clean":"2024-06-29T14:41:21.434693+0000","last_became_active":"2024-06-29T14:41:21.427719+0000","last_became_peered":"2024-06-29T14:41:21.427719+0000","last_unstale":"2024-06-29T14:41:21.434693+0000","last_undegraded":"2024-06-29T14:41:21.434693+0000","last_fullsized":"2024-06-29T14:41:21.434693+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:37.502969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426086+0000","last_change":"2024-06-29T14:41:20.416455+0000","last_active":"2024-06-29T14:41:22.426086+0000","last_peered":"2024-06-29T14:41:22.426086+0000","last_clean":"2024-06-29T14:41:22.426086+0000","last_became_active":"2024-06-29T14:41:20.416115+0000","last_became_peered":"2024-06-29T14:41:20.416115+0000","last_unstale":"2024-06-29T14:41:22.426086+0000","last_undegraded":"2024-06-29T14:41:22.426086+0000","last_fullsized":"2024-06-29T14:41:22.426086+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:21.317242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.445223+0000","last_change":"2024-06-29T14:41:21.436636+0000","last_active":"2024-06-29T14:41:21.445223+0000","last_peered":"2024-06-29T14:41:21.445223+0000","last_clean":"2024-06-29T14:41:21.445223+0000","last_became_active":"2024-06-29T14:41:21.436290+0000","last_became_peered":"2024-06-29T14:41:21.436290+0000","last_unstale":"2024-06-29T14:41:21.445223+0000","last_undegraded":"2024-06-29T14:41:21.445223+0000","last_fullsized":"2024-06-29T14:41:21.445223+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:59:16.524155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426812+0000","last_change":"2024-06-29T14:41:20.420444+0000","last_active":"2024-06-29T14:41:22.426812+0000","last_peered":"2024-06-29T14:41:22.426812+0000","last_clean":"2024-06-29T14:41:22.426812+0000","last_became_active":"2024-06-29T14:41:20.420135+0000","last_became_peered":"2024-06-29T14:41:20.420135+0000","last_unstale":"2024-06-29T14:41:22.426812+0000","last_undegraded":"2024-06-29T14:41:22.426812+0000","last_fullsized":"2024-06-29T14:41:22.426812+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:48.424677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425715+0000","last_change":"2024-06-29T14:41:21.430690+0000","last_active":"2024-06-29T14:41:22.425715+0000","last_peered":"2024-06-29T14:41:22.425715+0000","last_clean":"2024-06-29T14:41:22.425715+0000","last_became_active":"2024-06-29T14:41:21.430299+0000","last_became_peered":"2024-06-29T14:41:21.430299+0000","last_unstale":"2024-06-29T14:41:22.425715+0000","last_undegraded":"2024-06-29T14:41:22.425715+0000","last_fullsized":"2024-06-29T14:41:22.425715+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:03.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.421972+0000","last_change":"2024-06-29T14:41:20.416367+0000","last_active":"2024-06-29T14:41:21.421972+0000","last_peered":"2024-06-29T14:41:21.421972+0000","last_clean":"2024-06-29T14:41:21.421972+0000","last_became_active":"2024-06-29T14:41:20.416045+0000","last_became_peered":"2024-06-29T14:41:20.416045+0000","last_unstale":"2024-06-29T14:41:21.421972+0000","last_undegraded":"2024-06-29T14:41:21.421972+0000","last_fullsized":"2024-06-29T14:41:21.421972+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:23.360766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.422601+0000","last_change":"2024-06-29T14:41:21.420512+0000","last_active":"2024-06-29T14:41:21.422601+0000","last_peered":"2024-06-29T14:41:21.422601+0000","last_clean":"2024-06-29T14:41:21.422601+0000","last_became_active":"2024-06-29T14:41:21.419330+0000","last_became_peered":"2024-06-29T14:41:21.419330+0000","last_unstale":"2024-06-29T14:41:21.422601+0000","last_undegraded":"2024-06-29T14:41:21.422601+0000","last_fullsized":"2024-06-29T14:41:21.422601+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:00.239358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.420945+0000","last_change":"2024-06-29T14:41:20.410434+0000","last_active":"2024-06-29T14:41:21.420945+0000","last_peered":"2024-06-29T14:41:21.420945+0000","last_clean":"2024-06-29T14:41:21.420945+0000","last_became_active":"2024-06-29T14:41:20.410059+0000","last_became_peered":"2024-06-29T14:41:20.410059+0000","last_unstale":"2024-06-29T14:41:21.420945+0000","last_undegraded":"2024-06-29T14:41:21.420945+0000","last_fullsized":"2024-06-29T14:41:21.420945+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:38.247088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.434894+0000","last_change":"2024-06-29T14:41:21.429353+0000","last_active":"2024-06-29T14:41:21.434894+0000","last_peered":"2024-06-29T14:41:21.434894+0000","last_clean":"2024-06-29T14:41:21.434894+0000","last_became_active":"2024-06-29T14:41:21.429031+0000","last_became_peered":"2024-06-29T14:41:21.429031+0000","last_unstale":"2024-06-29T14:41:21.434894+0000","last_undegraded":"2024-06-29T14:41:21.434894+0000","last_fullsized":"2024-06-29T14:41:21.434894+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:52.963873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425757+0000","last_change":"2024-06-29T14:41:20.421511+0000","last_active":"2024-06-29T14:41:22.425757+0000","last_peered":"2024-06-29T14:41:22.425757+0000","last_clean":"2024-06-29T14:41:22.425757+0000","last_became_active":"2024-06-29T14:41:20.420971+0000","last_became_peered":"2024-06-29T14:41:20.420971+0000","last_unstale":"2024-06-29T14:41:22.425757+0000","last_undegraded":"2024-06-29T14:41:22.425757+0000","last_fullsized":"2024-06-29T14:41:22.425757+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:27.981198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425351+0000","last_change":"2024-06-29T14:41:21.431579+0000","last_active":"2024-06-29T14:41:22.425351+0000","last_peered":"2024-06-29T14:41:22.425351+0000","last_clean":"2024-06-29T14:41:22.425351+0000","last_became_active":"2024-06-29T14:41:21.431183+0000","last_became_peered":"2024-06-29T14:41:21.431183+0000","last_unstale":"2024-06-29T14:41:22.425351+0000","last_undegraded":"2024-06-29T14:41:22.425351+0000","last_fullsized":"2024-06-29T14:41:22.425351+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:51.161729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.412481+0000","last_change":"2024-06-29T14:41:20.411196+0000","last_active":"2024-06-29T14:41:21.412481+0000","last_peered":"2024-06-29T14:41:21.412481+0000","last_clean":"2024-06-29T14:41:21.412481+0000","last_became_active":"2024-06-29T14:41:20.410842+0000","last_became_peered":"2024-06-29T14:41:20.410842+0000","last_unstale":"2024-06-29T14:41:21.412481+0000","last_undegraded":"2024-06-29T14:41:21.412481+0000","last_fullsized":"2024-06-29T14:41:21.412481+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:23.902951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446079+0000","last_change":"2024-06-29T14:41:21.444006+0000","last_active":"2024-06-29T14:41:21.446079+0000","last_peered":"2024-06-29T14:41:21.446079+0000","last_clean":"2024-06-29T14:41:21.446079+0000","last_became_active":"2024-06-29T14:41:21.443672+0000","last_became_peered":"2024-06-29T14:41:21.443672+0000","last_unstale":"2024-06-29T14:41:21.446079+0000","last_undegraded":"2024-06-29T14:41:21.446079+0000","last_fullsized":"2024-06-29T14:41:21.446079+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:17.965041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"142'8","reported_seq":20,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.452219+0000","last_change":"2024-06-29T14:41:20.410691+0000","last_active":"2024-06-29T14:41:21.452219+0000","last_peered":"2024-06-29T14:41:21.452219+0000","last_clean":"2024-06-29T14:41:21.452219+0000","last_became_active":"2024-06-29T14:41:20.410281+0000","last_became_peered":"2024-06-29T14:41:20.410281+0000","last_unstale":"2024-06-29T14:41:21.452219+0000","last_undegraded":"2024-06-29T14:41:21.452219+0000","last_fullsized":"2024-06-29T14:41:21.452219+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:56.592360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'156","reported_seq":643,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.412274+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:41:21.412274+0000","last_peered":"2024-06-29T14:41:21.412274+0000","last_clean":"2024-06-29T14:41:21.412274+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:41:21.412274+0000","last_undegraded":"2024-06-29T14:41:21.412274+0000","last_fullsized":"2024-06-29T14:41:21.412274+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":245,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":245,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542379,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":2367488,"data_stored":1482136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542380,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1761280,"data_stored":891574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542379,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55112,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316728,"statfs":{"total":96636764160,"available":96580329472,"internally_reserved":0,"allocated":1765376,"data_stored":891152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542378,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1769472,"data_stored":892604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509676,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":892638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575084,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59264,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312576,"statfs":{"total":96636764160,"available":96576077824,"internally_reserved":0,"allocated":1757184,"data_stored":890992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1773568,"data_stored":897734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509677,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37648,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334192,"statfs":{"total":96636764160,"available":96598212608,"internally_reserved":0,"allocated":2355200,"data_stored":1481382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:41:25.511 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:41:25.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.666+0000 7f97a7474640 1 -- 172.21.15.59:0/4017294033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9788004800 msgr2=0x7f9788004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.666+0000 7f97a7474640 1 --2- 172.21.15.59:0/4017294033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9788004800 0x7f9788004c10 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f979400c290 tx=0x7f9788005830 comp rx=0 tx=0).stop 2024-06-29T14:41:25.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.666+0000 7f97a5a11640 1 -- 172.21.15.59:0/4017294033 reap_dead start 2024-06-29T14:41:25.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.666+0000 7f97a7474640 1 -- 172.21.15.59:0/4017294033 shutdown_connections 2024-06-29T14:41:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.666+0000 7f97a7474640 1 -- 172.21.15.59:0/4017294033 >> 172.21.15.59:0/4017294033 conn(0x7f97a006c950 msgr2=0x7f97a0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.666+0000 7f97a7474640 1 -- 172.21.15.59:0/4017294033 shutdown_connections 2024-06-29T14:41:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.666+0000 7f97a7474640 1 -- 172.21.15.59:0/4017294033 wait complete. 2024-06-29T14:41:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.668+0000 7f97a7474640 1 Processor -- start 2024-06-29T14:41:25.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.668+0000 7f97a7474640 1 -- start start 2024-06-29T14:41:25.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a7474640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788004800 0x7f97a0143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a7474640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a00ceee0 0x7f97a00cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a7474640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a00cd9b0 0x7f97a00cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a7474640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97a00b20a0 con 0x7f97a00ceee0 2024-06-29T14:41:25.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a7474640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97a00b1f20 con 0x7f9788004800 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a7474640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f97a0166c70 con 0x7f97a00cd9b0 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a4a0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a00ceee0 0x7f97a00cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a4a0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a00ceee0 0x7f97a00cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37782/0 (socket says 172.21.15.59:37782) 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a4a0f640 1 -- 172.21.15.59:0/2388660817 learned_addr learned my addr 172.21.15.59:0/2388660817 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a5210640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788004800 0x7f97a0143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a5a11640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a00cd9b0 0x7f97a00cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a4a0f640 1 -- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a00cd9b0 msgr2=0x7f97a00cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a4a0f640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a00cd9b0 0x7f97a00cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a4a0f640 1 -- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788004800 msgr2=0x7f97a0143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a4a0f640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788004800 0x7f97a0143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a4a0f640 1 -- 172.21.15.59:0/2388660817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a00cb030 con 0x7f97a00ceee0 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.669+0000 7f97a5a11640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a00cd9b0 0x7f97a00cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.670+0000 7f97a4a0f640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a00ceee0 0x7f97a00cd470 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f979400c3b0 tx=0x7f9794006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.670+0000 7f979dffb640 1 -- 172.21.15.59:0/2388660817 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9794031040 con 0x7f97a00ceee0 2024-06-29T14:41:25.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.670+0000 7f979dffb640 1 -- 172.21.15.59:0/2388660817 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9794006520 con 0x7f97a00ceee0 2024-06-29T14:41:25.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.670+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a00ce780 con 0x7f97a00ceee0 2024-06-29T14:41:25.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.670+0000 7f979dffb640 1 -- 172.21.15.59:0/2388660817 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9794003a60 con 0x7f97a00ceee0 2024-06-29T14:41:25.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.670+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a01639c0 con 0x7f97a00ceee0 2024-06-29T14:41:25.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.671+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a0070d40 con 0x7f97a00ceee0 2024-06-29T14:41:25.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.672+0000 7f979dffb640 1 -- 172.21.15.59:0/2388660817 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f9794034070 con 0x7f97a00ceee0 2024-06-29T14:41:25.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.673+0000 7f979dffb640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f978c0763f0 0x7f978c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.673+0000 7f979dffb640 1 -- 172.21.15.59:0/2388660817 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7f97940b7ac0 con 0x7f97a00ceee0 2024-06-29T14:41:25.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.673+0000 7f97a5210640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f978c0763f0 0x7f978c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.674+0000 7f97a5210640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f978c0763f0 0x7f978c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f97980036b0 tx=0x7f9798006b70 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.677+0000 7f979dffb640 1 -- 172.21.15.59:0/2388660817 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f97a0070d40 con 0x7f97a00ceee0 2024-06-29T14:41:25.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.863+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 --> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97a0161b00 con 0x7f978c0763f0 2024-06-29T14:41:25.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.868+0000 7f979dffb640 1 -- 172.21.15.59:0/2388660817 <== mgr.7259 v2:172.21.15.153:6832/776633953 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252447 (secure 0 0 0) 0x7f97a0161b00 con 0x7f978c0763f0 2024-06-29T14:41:25.868 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:25.869 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f978c0763f0 msgr2=0x7f978c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f978c0763f0 0x7f978c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f97980036b0 tx=0x7f9798006b70 comp rx=0 tx=0).stop 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a00ceee0 msgr2=0x7f97a00cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a00ceee0 0x7f97a00cd470 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f979400c3b0 tx=0x7f9794006ca0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 shutdown_connections 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f978c0763f0 0x7f978c078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97a00cd9b0 0x7f97a00cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a00ceee0 0x7f97a00cd470 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 --2- 172.21.15.59:0/2388660817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9788004800 0x7f97a0143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 >> 172.21.15.59:0/2388660817 conn(0x7f97a006c950 msgr2=0x7f97a0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 shutdown_connections 2024-06-29T14:41:25.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:25.870+0000 7f97a7474640 1 -- 172.21.15.59:0/2388660817 wait complete. 2024-06-29T14:41:25.880 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-06-29T14:41:24.960065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":290,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":356008,"kb_used_data":14964,"kb_used_omap":12,"kb_used_meta":340915,"kb_avail":754618712,"statfs":{"total":773094113280,"available":772729561088,"internally_reserved":0,"allocated":15323136,"data_stored":8320212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":349097522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.567798"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.434595+0000","last_change":"2024-06-29T14:41:21.432149+0000","last_active":"2024-06-29T14:41:21.434595+0000","last_peered":"2024-06-29T14:41:21.434595+0000","last_clean":"2024-06-29T14:41:21.434595+0000","last_became_active":"2024-06-29T14:41:21.431528+0000","last_became_peered":"2024-06-29T14:41:21.431528+0000","last_unstale":"2024-06-29T14:41:21.434595+0000","last_undegraded":"2024-06-29T14:41:21.434595+0000","last_fullsized":"2024-06-29T14:41:21.434595+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:42.146353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439601+0000","last_change":"2024-06-29T14:41:21.422105+0000","last_active":"2024-06-29T14:41:21.439601+0000","last_peered":"2024-06-29T14:41:21.439601+0000","last_clean":"2024-06-29T14:41:21.439601+0000","last_became_active":"2024-06-29T14:41:21.421776+0000","last_became_peered":"2024-06-29T14:41:21.421776+0000","last_unstale":"2024-06-29T14:41:21.439601+0000","last_undegraded":"2024-06-29T14:41:21.439601+0000","last_fullsized":"2024-06-29T14:41:21.439601+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:44.107767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.441309+0000","last_change":"2024-06-29T14:41:21.423301+0000","last_active":"2024-06-29T14:41:21.441309+0000","last_peered":"2024-06-29T14:41:21.441309+0000","last_clean":"2024-06-29T14:41:21.441309+0000","last_became_active":"2024-06-29T14:41:21.422401+0000","last_became_peered":"2024-06-29T14:41:21.422401+0000","last_unstale":"2024-06-29T14:41:21.441309+0000","last_undegraded":"2024-06-29T14:41:21.441309+0000","last_fullsized":"2024-06-29T14:41:21.441309+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:20.998587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.430560+0000","last_change":"2024-06-29T14:41:21.427478+0000","last_active":"2024-06-29T14:41:21.430560+0000","last_peered":"2024-06-29T14:41:21.430560+0000","last_clean":"2024-06-29T14:41:21.430560+0000","last_became_active":"2024-06-29T14:41:21.427170+0000","last_became_peered":"2024-06-29T14:41:21.427170+0000","last_unstale":"2024-06-29T14:41:21.430560+0000","last_undegraded":"2024-06-29T14:41:21.430560+0000","last_fullsized":"2024-06-29T14:41:21.430560+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:15.824908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.432550+0000","last_change":"2024-06-29T14:41:21.430183+0000","last_active":"2024-06-29T14:41:21.432550+0000","last_peered":"2024-06-29T14:41:21.432550+0000","last_clean":"2024-06-29T14:41:21.432550+0000","last_became_active":"2024-06-29T14:41:21.429803+0000","last_became_peered":"2024-06-29T14:41:21.429803+0000","last_unstale":"2024-06-29T14:41:21.432550+0000","last_undegraded":"2024-06-29T14:41:21.432550+0000","last_fullsized":"2024-06-29T14:41:21.432550+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:40.853602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426280+0000","last_change":"2024-06-29T14:41:21.444312+0000","last_active":"2024-06-29T14:41:22.426280+0000","last_peered":"2024-06-29T14:41:22.426280+0000","last_clean":"2024-06-29T14:41:22.426280+0000","last_became_active":"2024-06-29T14:41:21.443961+0000","last_became_peered":"2024-06-29T14:41:21.443961+0000","last_unstale":"2024-06-29T14:41:22.426280+0000","last_undegraded":"2024-06-29T14:41:22.426280+0000","last_fullsized":"2024-06-29T14:41:22.426280+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:57:09.781784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425575+0000","last_change":"2024-06-29T14:41:21.436399+0000","last_active":"2024-06-29T14:41:22.425575+0000","last_peered":"2024-06-29T14:41:22.425575+0000","last_clean":"2024-06-29T14:41:22.425575+0000","last_became_active":"2024-06-29T14:41:21.436032+0000","last_became_peered":"2024-06-29T14:41:21.436032+0000","last_unstale":"2024-06-29T14:41:22.425575+0000","last_undegraded":"2024-06-29T14:41:22.425575+0000","last_fullsized":"2024-06-29T14:41:22.425575+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:29.933829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.420813+0000","last_change":"2024-06-29T14:41:21.418878+0000","last_active":"2024-06-29T14:41:21.420813+0000","last_peered":"2024-06-29T14:41:21.420813+0000","last_clean":"2024-06-29T14:41:21.420813+0000","last_became_active":"2024-06-29T14:41:21.418550+0000","last_became_peered":"2024-06-29T14:41:21.418550+0000","last_unstale":"2024-06-29T14:41:21.420813+0000","last_undegraded":"2024-06-29T14:41:21.420813+0000","last_fullsized":"2024-06-29T14:41:21.420813+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:24.799928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.424595+0000","last_change":"2024-06-29T14:41:21.422527+0000","last_active":"2024-06-29T14:41:21.424595+0000","last_peered":"2024-06-29T14:41:21.424595+0000","last_clean":"2024-06-29T14:41:21.424595+0000","last_became_active":"2024-06-29T14:41:21.422223+0000","last_became_peered":"2024-06-29T14:41:21.422223+0000","last_unstale":"2024-06-29T14:41:21.424595+0000","last_undegraded":"2024-06-29T14:41:21.424595+0000","last_fullsized":"2024-06-29T14:41:21.424595+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:48.505228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425909+0000","last_change":"2024-06-29T14:41:21.443188+0000","last_active":"2024-06-29T14:41:22.425909+0000","last_peered":"2024-06-29T14:41:22.425909+0000","last_clean":"2024-06-29T14:41:22.425909+0000","last_became_active":"2024-06-29T14:41:21.442834+0000","last_became_peered":"2024-06-29T14:41:21.442834+0000","last_unstale":"2024-06-29T14:41:22.425909+0000","last_undegraded":"2024-06-29T14:41:22.425909+0000","last_fullsized":"2024-06-29T14:41:22.425909+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:34.786229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439941+0000","last_change":"2024-06-29T14:41:21.436905+0000","last_active":"2024-06-29T14:41:21.439941+0000","last_peered":"2024-06-29T14:41:21.439941+0000","last_clean":"2024-06-29T14:41:21.439941+0000","last_became_active":"2024-06-29T14:41:21.436544+0000","last_became_peered":"2024-06-29T14:41:21.436544+0000","last_unstale":"2024-06-29T14:41:21.439941+0000","last_undegraded":"2024-06-29T14:41:21.439941+0000","last_fullsized":"2024-06-29T14:41:21.439941+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:49.771666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426604+0000","last_change":"2024-06-29T14:41:21.430583+0000","last_active":"2024-06-29T14:41:22.426604+0000","last_peered":"2024-06-29T14:41:22.426604+0000","last_clean":"2024-06-29T14:41:22.426604+0000","last_became_active":"2024-06-29T14:41:21.430215+0000","last_became_peered":"2024-06-29T14:41:21.430215+0000","last_unstale":"2024-06-29T14:41:22.426604+0000","last_undegraded":"2024-06-29T14:41:22.426604+0000","last_fullsized":"2024-06-29T14:41:22.426604+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:30:26.767490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425185+0000","last_change":"2024-06-29T14:41:21.432562+0000","last_active":"2024-06-29T14:41:22.425185+0000","last_peered":"2024-06-29T14:41:22.425185+0000","last_clean":"2024-06-29T14:41:22.425185+0000","last_became_active":"2024-06-29T14:41:21.432193+0000","last_became_peered":"2024-06-29T14:41:21.432193+0000","last_unstale":"2024-06-29T14:41:22.425185+0000","last_undegraded":"2024-06-29T14:41:22.425185+0000","last_fullsized":"2024-06-29T14:41:22.425185+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:41.751317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.421185+0000","last_change":"2024-06-29T14:41:21.418762+0000","last_active":"2024-06-29T14:41:21.421185+0000","last_peered":"2024-06-29T14:41:21.421185+0000","last_clean":"2024-06-29T14:41:21.421185+0000","last_became_active":"2024-06-29T14:41:21.418246+0000","last_became_peered":"2024-06-29T14:41:21.418246+0000","last_unstale":"2024-06-29T14:41:21.421185+0000","last_undegraded":"2024-06-29T14:41:21.421185+0000","last_fullsized":"2024-06-29T14:41:21.421185+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:30.919446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439937+0000","last_change":"2024-06-29T14:41:21.436102+0000","last_active":"2024-06-29T14:41:21.439937+0000","last_peered":"2024-06-29T14:41:21.439937+0000","last_clean":"2024-06-29T14:41:21.439937+0000","last_became_active":"2024-06-29T14:41:21.435767+0000","last_became_peered":"2024-06-29T14:41:21.435767+0000","last_unstale":"2024-06-29T14:41:21.439937+0000","last_undegraded":"2024-06-29T14:41:21.439937+0000","last_fullsized":"2024-06-29T14:41:21.439937+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:33:42.213193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425529+0000","last_change":"2024-06-29T14:41:21.438179+0000","last_active":"2024-06-29T14:41:22.425529+0000","last_peered":"2024-06-29T14:41:22.425529+0000","last_clean":"2024-06-29T14:41:22.425529+0000","last_became_active":"2024-06-29T14:41:21.437804+0000","last_became_peered":"2024-06-29T14:41:21.437804+0000","last_unstale":"2024-06-29T14:41:22.425529+0000","last_undegraded":"2024-06-29T14:41:22.425529+0000","last_fullsized":"2024-06-29T14:41:22.425529+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:45:58.642581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425674+0000","last_change":"2024-06-29T14:41:21.432638+0000","last_active":"2024-06-29T14:41:22.425674+0000","last_peered":"2024-06-29T14:41:22.425674+0000","last_clean":"2024-06-29T14:41:22.425674+0000","last_became_active":"2024-06-29T14:41:21.432281+0000","last_became_peered":"2024-06-29T14:41:21.432281+0000","last_unstale":"2024-06-29T14:41:22.425674+0000","last_undegraded":"2024-06-29T14:41:22.425674+0000","last_fullsized":"2024-06-29T14:41:22.425674+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:27:03.389186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439628+0000","last_change":"2024-06-29T14:41:21.422385+0000","last_active":"2024-06-29T14:41:21.439628+0000","last_peered":"2024-06-29T14:41:21.439628+0000","last_clean":"2024-06-29T14:41:21.439628+0000","last_became_active":"2024-06-29T14:41:21.422061+0000","last_became_peered":"2024-06-29T14:41:21.422061+0000","last_unstale":"2024-06-29T14:41:21.439628+0000","last_undegraded":"2024-06-29T14:41:21.439628+0000","last_fullsized":"2024-06-29T14:41:21.439628+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:21.208186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425977+0000","last_change":"2024-06-29T14:41:21.419016+0000","last_active":"2024-06-29T14:41:22.425977+0000","last_peered":"2024-06-29T14:41:22.425977+0000","last_clean":"2024-06-29T14:41:22.425977+0000","last_became_active":"2024-06-29T14:41:21.418663+0000","last_became_peered":"2024-06-29T14:41:21.418663+0000","last_unstale":"2024-06-29T14:41:22.425977+0000","last_undegraded":"2024-06-29T14:41:22.425977+0000","last_fullsized":"2024-06-29T14:41:22.425977+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:35.235758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.448800+0000","last_change":"2024-06-29T14:41:21.437211+0000","last_active":"2024-06-29T14:41:21.448800+0000","last_peered":"2024-06-29T14:41:21.448800+0000","last_clean":"2024-06-29T14:41:21.448800+0000","last_became_active":"2024-06-29T14:41:21.436885+0000","last_became_peered":"2024-06-29T14:41:21.436885+0000","last_unstale":"2024-06-29T14:41:21.448800+0000","last_undegraded":"2024-06-29T14:41:21.448800+0000","last_fullsized":"2024-06-29T14:41:21.448800+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:11.665514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.439913+0000","last_change":"2024-06-29T14:41:21.436287+0000","last_active":"2024-06-29T14:41:21.439913+0000","last_peered":"2024-06-29T14:41:21.439913+0000","last_clean":"2024-06-29T14:41:21.439913+0000","last_became_active":"2024-06-29T14:41:21.435356+0000","last_became_peered":"2024-06-29T14:41:21.435356+0000","last_unstale":"2024-06-29T14:41:21.439913+0000","last_undegraded":"2024-06-29T14:41:21.439913+0000","last_fullsized":"2024-06-29T14:41:21.439913+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:57.964925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.436340+0000","last_change":"2024-06-29T14:41:21.435488+0000","last_active":"2024-06-29T14:41:21.436340+0000","last_peered":"2024-06-29T14:41:21.436340+0000","last_clean":"2024-06-29T14:41:21.436340+0000","last_became_active":"2024-06-29T14:41:21.435098+0000","last_became_peered":"2024-06-29T14:41:21.435098+0000","last_unstale":"2024-06-29T14:41:21.436340+0000","last_undegraded":"2024-06-29T14:41:21.436340+0000","last_fullsized":"2024-06-29T14:41:21.436340+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:14:45.301566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.426299+0000","last_change":"2024-06-29T14:41:21.424553+0000","last_active":"2024-06-29T14:41:21.426299+0000","last_peered":"2024-06-29T14:41:21.426299+0000","last_clean":"2024-06-29T14:41:21.426299+0000","last_became_active":"2024-06-29T14:41:21.424176+0000","last_became_peered":"2024-06-29T14:41:21.424176+0000","last_unstale":"2024-06-29T14:41:21.426299+0000","last_undegraded":"2024-06-29T14:41:21.426299+0000","last_fullsized":"2024-06-29T14:41:21.426299+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:43:57.320978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.433989+0000","last_change":"2024-06-29T14:41:21.433206+0000","last_active":"2024-06-29T14:41:21.433989+0000","last_peered":"2024-06-29T14:41:21.433989+0000","last_clean":"2024-06-29T14:41:21.433989+0000","last_became_active":"2024-06-29T14:41:21.432123+0000","last_became_peered":"2024-06-29T14:41:21.432123+0000","last_unstale":"2024-06-29T14:41:21.433989+0000","last_undegraded":"2024-06-29T14:41:21.433989+0000","last_fullsized":"2024-06-29T14:41:21.433989+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:28.594542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.433288+0000","last_change":"2024-06-29T14:41:21.431360+0000","last_active":"2024-06-29T14:41:21.433288+0000","last_peered":"2024-06-29T14:41:21.433288+0000","last_clean":"2024-06-29T14:41:21.433288+0000","last_became_active":"2024-06-29T14:41:21.430202+0000","last_became_peered":"2024-06-29T14:41:21.430202+0000","last_unstale":"2024-06-29T14:41:21.433288+0000","last_undegraded":"2024-06-29T14:41:21.433288+0000","last_fullsized":"2024-06-29T14:41:21.433288+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:30.495511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.448751+0000","last_change":"2024-06-29T14:41:21.437704+0000","last_active":"2024-06-29T14:41:21.448751+0000","last_peered":"2024-06-29T14:41:21.448751+0000","last_clean":"2024-06-29T14:41:21.448751+0000","last_became_active":"2024-06-29T14:41:21.437329+0000","last_became_peered":"2024-06-29T14:41:21.437329+0000","last_unstale":"2024-06-29T14:41:21.448751+0000","last_undegraded":"2024-06-29T14:41:21.448751+0000","last_fullsized":"2024-06-29T14:41:21.448751+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:32.365851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426002+0000","last_change":"2024-06-29T14:41:21.419002+0000","last_active":"2024-06-29T14:41:22.426002+0000","last_peered":"2024-06-29T14:41:22.426002+0000","last_clean":"2024-06-29T14:41:22.426002+0000","last_became_active":"2024-06-29T14:41:21.418637+0000","last_became_peered":"2024-06-29T14:41:21.418637+0000","last_unstale":"2024-06-29T14:41:22.426002+0000","last_undegraded":"2024-06-29T14:41:22.426002+0000","last_fullsized":"2024-06-29T14:41:22.426002+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:46.771288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426695+0000","last_change":"2024-06-29T14:41:21.435163+0000","last_active":"2024-06-29T14:41:22.426695+0000","last_peered":"2024-06-29T14:41:22.426695+0000","last_clean":"2024-06-29T14:41:22.426695+0000","last_became_active":"2024-06-29T14:41:21.434814+0000","last_became_peered":"2024-06-29T14:41:21.434814+0000","last_unstale":"2024-06-29T14:41:22.426695+0000","last_undegraded":"2024-06-29T14:41:22.426695+0000","last_fullsized":"2024-06-29T14:41:22.426695+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:33.399206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.448837+0000","last_change":"2024-06-29T14:41:21.437020+0000","last_active":"2024-06-29T14:41:21.448837+0000","last_peered":"2024-06-29T14:41:21.448837+0000","last_clean":"2024-06-29T14:41:21.448837+0000","last_became_active":"2024-06-29T14:41:21.435398+0000","last_became_peered":"2024-06-29T14:41:21.435398+0000","last_unstale":"2024-06-29T14:41:21.448837+0000","last_undegraded":"2024-06-29T14:41:21.448837+0000","last_fullsized":"2024-06-29T14:41:21.448837+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:44.481684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446271+0000","last_change":"2024-06-29T14:41:21.432746+0000","last_active":"2024-06-29T14:41:21.446271+0000","last_peered":"2024-06-29T14:41:21.446271+0000","last_clean":"2024-06-29T14:41:21.446271+0000","last_became_active":"2024-06-29T14:41:21.432190+0000","last_became_peered":"2024-06-29T14:41:21.432190+0000","last_unstale":"2024-06-29T14:41:21.446271+0000","last_undegraded":"2024-06-29T14:41:21.446271+0000","last_fullsized":"2024-06-29T14:41:21.446271+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:10.402070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.431891+0000","last_change":"2024-06-29T14:41:21.430220+0000","last_active":"2024-06-29T14:41:21.431891+0000","last_peered":"2024-06-29T14:41:21.431891+0000","last_clean":"2024-06-29T14:41:21.431891+0000","last_became_active":"2024-06-29T14:41:21.429853+0000","last_became_peered":"2024-06-29T14:41:21.429853+0000","last_unstale":"2024-06-29T14:41:21.431891+0000","last_undegraded":"2024-06-29T14:41:21.431891+0000","last_fullsized":"2024-06-29T14:41:21.431891+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:03:26.279064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.433950+0000","last_change":"2024-06-29T14:41:21.432543+0000","last_active":"2024-06-29T14:41:21.433950+0000","last_peered":"2024-06-29T14:41:21.433950+0000","last_clean":"2024-06-29T14:41:21.433950+0000","last_became_active":"2024-06-29T14:41:21.432021+0000","last_became_peered":"2024-06-29T14:41:21.432021+0000","last_unstale":"2024-06-29T14:41:21.433950+0000","last_undegraded":"2024-06-29T14:41:21.433950+0000","last_fullsized":"2024-06-29T14:41:21.433950+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:28.987473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426187+0000","last_change":"2024-06-29T14:41:21.441622+0000","last_active":"2024-06-29T14:41:22.426187+0000","last_peered":"2024-06-29T14:41:22.426187+0000","last_clean":"2024-06-29T14:41:22.426187+0000","last_became_active":"2024-06-29T14:41:21.441192+0000","last_became_peered":"2024-06-29T14:41:21.441192+0000","last_unstale":"2024-06-29T14:41:22.426187+0000","last_undegraded":"2024-06-29T14:41:22.426187+0000","last_fullsized":"2024-06-29T14:41:22.426187+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:40:13.615563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"142'4","reported_seq":18,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425792+0000","last_change":"2024-06-29T14:41:20.413102+0000","last_active":"2024-06-29T14:41:22.425792+0000","last_peered":"2024-06-29T14:41:22.425792+0000","last_clean":"2024-06-29T14:41:22.425792+0000","last_became_active":"2024-06-29T14:41:20.412752+0000","last_became_peered":"2024-06-29T14:41:20.412752+0000","last_unstale":"2024-06-29T14:41:22.425792+0000","last_undegraded":"2024-06-29T14:41:22.425792+0000","last_fullsized":"2024-06-29T14:41:22.425792+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:38.247088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426437+0000","last_change":"2024-06-29T14:41:21.443247+0000","last_active":"2024-06-29T14:41:22.426437+0000","last_peered":"2024-06-29T14:41:22.426437+0000","last_clean":"2024-06-29T14:41:22.426437+0000","last_became_active":"2024-06-29T14:41:21.442913+0000","last_became_peered":"2024-06-29T14:41:21.442913+0000","last_unstale":"2024-06-29T14:41:22.426437+0000","last_undegraded":"2024-06-29T14:41:22.426437+0000","last_fullsized":"2024-06-29T14:41:22.426437+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:38:32.792002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"142'1","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426021+0000","last_change":"2024-06-29T14:41:20.410994+0000","last_active":"2024-06-29T14:41:22.426021+0000","last_peered":"2024-06-29T14:41:22.426021+0000","last_clean":"2024-06-29T14:41:22.426021+0000","last_became_active":"2024-06-29T14:41:20.410665+0000","last_became_peered":"2024-06-29T14:41:20.410665+0000","last_unstale":"2024-06-29T14:41:22.426021+0000","last_undegraded":"2024-06-29T14:41:22.426021+0000","last_fullsized":"2024-06-29T14:41:22.426021+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:09.848965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.444128+0000","last_change":"2024-06-29T14:41:21.438943+0000","last_active":"2024-06-29T14:41:21.444128+0000","last_peered":"2024-06-29T14:41:21.444128+0000","last_clean":"2024-06-29T14:41:21.444128+0000","last_became_active":"2024-06-29T14:41:21.438650+0000","last_became_peered":"2024-06-29T14:41:21.438650+0000","last_unstale":"2024-06-29T14:41:21.444128+0000","last_undegraded":"2024-06-29T14:41:21.444128+0000","last_fullsized":"2024-06-29T14:41:21.444128+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:10.229870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"142'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.450810+0000","last_change":"2024-06-29T14:41:20.416624+0000","last_active":"2024-06-29T14:41:21.450810+0000","last_peered":"2024-06-29T14:41:21.450810+0000","last_clean":"2024-06-29T14:41:21.450810+0000","last_became_active":"2024-06-29T14:41:20.416297+0000","last_became_peered":"2024-06-29T14:41:20.416297+0000","last_unstale":"2024-06-29T14:41:21.450810+0000","last_undegraded":"2024-06-29T14:41:21.450810+0000","last_fullsized":"2024-06-29T14:41:21.450810+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:21.833779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.435058+0000","last_change":"2024-06-29T14:41:21.428280+0000","last_active":"2024-06-29T14:41:21.435058+0000","last_peered":"2024-06-29T14:41:21.435058+0000","last_clean":"2024-06-29T14:41:21.435058+0000","last_became_active":"2024-06-29T14:41:21.427880+0000","last_became_peered":"2024-06-29T14:41:21.427880+0000","last_unstale":"2024-06-29T14:41:21.435058+0000","last_undegraded":"2024-06-29T14:41:21.435058+0000","last_fullsized":"2024-06-29T14:41:21.435058+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:26.036865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"142'2","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426011+0000","last_change":"2024-06-29T14:41:20.412590+0000","last_active":"2024-06-29T14:41:22.426011+0000","last_peered":"2024-06-29T14:41:22.426011+0000","last_clean":"2024-06-29T14:41:22.426011+0000","last_became_active":"2024-06-29T14:41:20.412276+0000","last_became_peered":"2024-06-29T14:41:20.412276+0000","last_unstale":"2024-06-29T14:41:22.426011+0000","last_undegraded":"2024-06-29T14:41:22.426011+0000","last_fullsized":"2024-06-29T14:41:22.426011+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:46.271143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.424980+0000","last_change":"2024-06-29T14:41:21.436350+0000","last_active":"2024-06-29T14:41:22.424980+0000","last_peered":"2024-06-29T14:41:22.424980+0000","last_clean":"2024-06-29T14:41:22.424980+0000","last_became_active":"2024-06-29T14:41:21.435872+0000","last_became_peered":"2024-06-29T14:41:21.435872+0000","last_unstale":"2024-06-29T14:41:22.424980+0000","last_undegraded":"2024-06-29T14:41:22.424980+0000","last_fullsized":"2024-06-29T14:41:22.424980+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:07.470915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"142'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.453304+0000","last_change":"2024-06-29T14:41:20.418669+0000","last_active":"2024-06-29T14:41:21.453304+0000","last_peered":"2024-06-29T14:41:21.453304+0000","last_clean":"2024-06-29T14:41:21.453304+0000","last_became_active":"2024-06-29T14:41:20.418380+0000","last_became_peered":"2024-06-29T14:41:20.418380+0000","last_unstale":"2024-06-29T14:41:21.453304+0000","last_undegraded":"2024-06-29T14:41:21.453304+0000","last_fullsized":"2024-06-29T14:41:21.453304+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:17.054703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.429284+0000","last_change":"2024-06-29T14:41:21.427521+0000","last_active":"2024-06-29T14:41:21.429284+0000","last_peered":"2024-06-29T14:41:21.429284+0000","last_clean":"2024-06-29T14:41:21.429284+0000","last_became_active":"2024-06-29T14:41:21.427197+0000","last_became_peered":"2024-06-29T14:41:21.427197+0000","last_unstale":"2024-06-29T14:41:21.429284+0000","last_undegraded":"2024-06-29T14:41:21.429284+0000","last_fullsized":"2024-06-29T14:41:21.429284+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:22.323516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"142'1","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426208+0000","last_change":"2024-06-29T14:41:20.423395+0000","last_active":"2024-06-29T14:41:22.426208+0000","last_peered":"2024-06-29T14:41:22.426208+0000","last_clean":"2024-06-29T14:41:22.426208+0000","last_became_active":"2024-06-29T14:41:20.423044+0000","last_became_peered":"2024-06-29T14:41:20.423044+0000","last_unstale":"2024-06-29T14:41:22.426208+0000","last_undegraded":"2024-06-29T14:41:22.426208+0000","last_fullsized":"2024-06-29T14:41:22.426208+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:42.806858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446299+0000","last_change":"2024-06-29T14:41:21.433522+0000","last_active":"2024-06-29T14:41:21.446299+0000","last_peered":"2024-06-29T14:41:21.446299+0000","last_clean":"2024-06-29T14:41:21.446299+0000","last_became_active":"2024-06-29T14:41:21.433171+0000","last_became_peered":"2024-06-29T14:41:21.433171+0000","last_unstale":"2024-06-29T14:41:21.446299+0000","last_undegraded":"2024-06-29T14:41:21.446299+0000","last_fullsized":"2024-06-29T14:41:21.446299+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:18.242404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.425120+0000","last_change":"2024-06-29T14:41:20.412134+0000","last_active":"2024-06-29T14:41:21.425120+0000","last_peered":"2024-06-29T14:41:21.425120+0000","last_clean":"2024-06-29T14:41:21.425120+0000","last_became_active":"2024-06-29T14:41:20.411776+0000","last_became_peered":"2024-06-29T14:41:20.411776+0000","last_unstale":"2024-06-29T14:41:21.425120+0000","last_undegraded":"2024-06-29T14:41:21.425120+0000","last_fullsized":"2024-06-29T14:41:21.425120+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:49.488844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425125+0000","last_change":"2024-06-29T14:41:21.433915+0000","last_active":"2024-06-29T14:41:22.425125+0000","last_peered":"2024-06-29T14:41:22.425125+0000","last_clean":"2024-06-29T14:41:22.425125+0000","last_became_active":"2024-06-29T14:41:21.433596+0000","last_became_peered":"2024-06-29T14:41:21.433596+0000","last_unstale":"2024-06-29T14:41:22.425125+0000","last_undegraded":"2024-06-29T14:41:22.425125+0000","last_fullsized":"2024-06-29T14:41:22.425125+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:33.742338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425711+0000","last_change":"2024-06-29T14:41:20.421426+0000","last_active":"2024-06-29T14:41:22.425711+0000","last_peered":"2024-06-29T14:41:22.425711+0000","last_clean":"2024-06-29T14:41:22.425711+0000","last_became_active":"2024-06-29T14:41:20.421092+0000","last_became_peered":"2024-06-29T14:41:20.421092+0000","last_unstale":"2024-06-29T14:41:22.425711+0000","last_undegraded":"2024-06-29T14:41:22.425711+0000","last_fullsized":"2024-06-29T14:41:22.425711+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:47.861510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425602+0000","last_change":"2024-06-29T14:41:21.418489+0000","last_active":"2024-06-29T14:41:22.425602+0000","last_peered":"2024-06-29T14:41:22.425602+0000","last_clean":"2024-06-29T14:41:22.425602+0000","last_became_active":"2024-06-29T14:41:21.418142+0000","last_became_peered":"2024-06-29T14:41:21.418142+0000","last_unstale":"2024-06-29T14:41:22.425602+0000","last_undegraded":"2024-06-29T14:41:22.425602+0000","last_fullsized":"2024-06-29T14:41:22.425602+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:26:42.188988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425780+0000","last_change":"2024-06-29T14:41:20.417276+0000","last_active":"2024-06-29T14:41:22.425780+0000","last_peered":"2024-06-29T14:41:22.425780+0000","last_clean":"2024-06-29T14:41:22.425780+0000","last_became_active":"2024-06-29T14:41:20.416974+0000","last_became_peered":"2024-06-29T14:41:20.416974+0000","last_unstale":"2024-06-29T14:41:22.425780+0000","last_undegraded":"2024-06-29T14:41:22.425780+0000","last_fullsized":"2024-06-29T14:41:22.425780+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:05:17.054703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.434525+0000","last_change":"2024-06-29T14:41:21.433462+0000","last_active":"2024-06-29T14:41:21.434525+0000","last_peered":"2024-06-29T14:41:21.434525+0000","last_clean":"2024-06-29T14:41:21.434525+0000","last_became_active":"2024-06-29T14:41:21.433151+0000","last_became_peered":"2024-06-29T14:41:21.433151+0000","last_unstale":"2024-06-29T14:41:21.434525+0000","last_undegraded":"2024-06-29T14:41:21.434525+0000","last_fullsized":"2024-06-29T14:41:21.434525+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:43.813132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.425075+0000","last_change":"2024-06-29T14:41:20.411001+0000","last_active":"2024-06-29T14:41:21.425075+0000","last_peered":"2024-06-29T14:41:21.425075+0000","last_clean":"2024-06-29T14:41:21.425075+0000","last_became_active":"2024-06-29T14:41:20.410641+0000","last_became_peered":"2024-06-29T14:41:20.410641+0000","last_unstale":"2024-06-29T14:41:21.425075+0000","last_undegraded":"2024-06-29T14:41:21.425075+0000","last_fullsized":"2024-06-29T14:41:21.425075+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:50:37.352177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.427997+0000","last_change":"2024-06-29T14:41:21.420840+0000","last_active":"2024-06-29T14:41:21.427997+0000","last_peered":"2024-06-29T14:41:21.427997+0000","last_clean":"2024-06-29T14:41:21.427997+0000","last_became_active":"2024-06-29T14:41:21.420493+0000","last_became_peered":"2024-06-29T14:41:21.420493+0000","last_unstale":"2024-06-29T14:41:21.427997+0000","last_undegraded":"2024-06-29T14:41:21.427997+0000","last_fullsized":"2024-06-29T14:41:21.427997+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:44.946059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"142'3","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446625+0000","last_change":"2024-06-29T14:41:20.411557+0000","last_active":"2024-06-29T14:41:21.446625+0000","last_peered":"2024-06-29T14:41:21.446625+0000","last_clean":"2024-06-29T14:41:21.446625+0000","last_became_active":"2024-06-29T14:41:20.411264+0000","last_became_peered":"2024-06-29T14:41:20.411264+0000","last_unstale":"2024-06-29T14:41:21.446625+0000","last_undegraded":"2024-06-29T14:41:21.446625+0000","last_fullsized":"2024-06-29T14:41:21.446625+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:19.992312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.447002+0000","last_change":"2024-06-29T14:41:21.446570+0000","last_active":"2024-06-29T14:41:21.447002+0000","last_peered":"2024-06-29T14:41:21.447002+0000","last_clean":"2024-06-29T14:41:21.447002+0000","last_became_active":"2024-06-29T14:41:21.446099+0000","last_became_peered":"2024-06-29T14:41:21.446099+0000","last_unstale":"2024-06-29T14:41:21.447002+0000","last_undegraded":"2024-06-29T14:41:21.447002+0000","last_fullsized":"2024-06-29T14:41:21.447002+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.430952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"142'6","reported_seq":18,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.455150+0000","last_change":"2024-06-29T14:41:20.416297+0000","last_active":"2024-06-29T14:41:21.455150+0000","last_peered":"2024-06-29T14:41:21.455150+0000","last_clean":"2024-06-29T14:41:21.455150+0000","last_became_active":"2024-06-29T14:41:20.416005+0000","last_became_peered":"2024-06-29T14:41:20.416005+0000","last_unstale":"2024-06-29T14:41:21.455150+0000","last_undegraded":"2024-06-29T14:41:21.455150+0000","last_fullsized":"2024-06-29T14:41:21.455150+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:38.631988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426161+0000","last_change":"2024-06-29T14:41:21.422586+0000","last_active":"2024-06-29T14:41:22.426161+0000","last_peered":"2024-06-29T14:41:22.426161+0000","last_clean":"2024-06-29T14:41:22.426161+0000","last_became_active":"2024-06-29T14:41:21.422287+0000","last_became_peered":"2024-06-29T14:41:21.422287+0000","last_unstale":"2024-06-29T14:41:22.426161+0000","last_undegraded":"2024-06-29T14:41:22.426161+0000","last_fullsized":"2024-06-29T14:41:22.426161+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:58:05.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"142'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.449801+0000","last_change":"2024-06-29T14:41:20.418823+0000","last_active":"2024-06-29T14:41:21.449801+0000","last_peered":"2024-06-29T14:41:21.449801+0000","last_clean":"2024-06-29T14:41:21.449801+0000","last_became_active":"2024-06-29T14:41:20.418526+0000","last_became_peered":"2024-06-29T14:41:20.418526+0000","last_unstale":"2024-06-29T14:41:21.449801+0000","last_undegraded":"2024-06-29T14:41:21.449801+0000","last_fullsized":"2024-06-29T14:41:21.449801+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:29.886679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425034+0000","last_change":"2024-06-29T14:41:21.435826+0000","last_active":"2024-06-29T14:41:22.425034+0000","last_peered":"2024-06-29T14:41:22.425034+0000","last_clean":"2024-06-29T14:41:22.425034+0000","last_became_active":"2024-06-29T14:41:21.435293+0000","last_became_peered":"2024-06-29T14:41:21.435293+0000","last_unstale":"2024-06-29T14:41:22.425034+0000","last_undegraded":"2024-06-29T14:41:22.425034+0000","last_fullsized":"2024-06-29T14:41:22.425034+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:16.572821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.414888+0000","last_change":"2024-06-29T14:41:20.413868+0000","last_active":"2024-06-29T14:41:21.414888+0000","last_peered":"2024-06-29T14:41:21.414888+0000","last_clean":"2024-06-29T14:41:21.414888+0000","last_became_active":"2024-06-29T14:41:20.413540+0000","last_became_peered":"2024-06-29T14:41:20.413540+0000","last_unstale":"2024-06-29T14:41:21.414888+0000","last_undegraded":"2024-06-29T14:41:21.414888+0000","last_fullsized":"2024-06-29T14:41:21.414888+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:28:06.006143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.431573+0000","last_change":"2024-06-29T14:41:21.429057+0000","last_active":"2024-06-29T14:41:21.431573+0000","last_peered":"2024-06-29T14:41:21.431573+0000","last_clean":"2024-06-29T14:41:21.431573+0000","last_became_active":"2024-06-29T14:41:21.428695+0000","last_became_peered":"2024-06-29T14:41:21.428695+0000","last_unstale":"2024-06-29T14:41:21.431573+0000","last_undegraded":"2024-06-29T14:41:21.431573+0000","last_fullsized":"2024-06-29T14:41:21.431573+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:14.285071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.413805+0000","last_change":"2024-06-29T14:41:20.415218+0000","last_active":"2024-06-29T14:41:21.413805+0000","last_peered":"2024-06-29T14:41:21.413805+0000","last_clean":"2024-06-29T14:41:21.413805+0000","last_became_active":"2024-06-29T14:41:20.412865+0000","last_became_peered":"2024-06-29T14:41:20.412865+0000","last_unstale":"2024-06-29T14:41:21.413805+0000","last_undegraded":"2024-06-29T14:41:21.413805+0000","last_fullsized":"2024-06-29T14:41:21.413805+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:28.404030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.431948+0000","last_change":"2024-06-29T14:41:21.429817+0000","last_active":"2024-06-29T14:41:21.431948+0000","last_peered":"2024-06-29T14:41:21.431948+0000","last_clean":"2024-06-29T14:41:21.431948+0000","last_became_active":"2024-06-29T14:41:21.429513+0000","last_became_peered":"2024-06-29T14:41:21.429513+0000","last_unstale":"2024-06-29T14:41:21.431948+0000","last_undegraded":"2024-06-29T14:41:21.431948+0000","last_fullsized":"2024-06-29T14:41:21.431948+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:12:18.242404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"142'1","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425684+0000","last_change":"2024-06-29T14:41:20.419691+0000","last_active":"2024-06-29T14:41:22.425684+0000","last_peered":"2024-06-29T14:41:22.425684+0000","last_clean":"2024-06-29T14:41:22.425684+0000","last_became_active":"2024-06-29T14:41:20.419297+0000","last_became_peered":"2024-06-29T14:41:20.419297+0000","last_unstale":"2024-06-29T14:41:22.425684+0000","last_undegraded":"2024-06-29T14:41:22.425684+0000","last_fullsized":"2024-06-29T14:41:22.425684+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:46.716724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.444046+0000","last_change":"2024-06-29T14:41:21.443431+0000","last_active":"2024-06-29T14:41:21.444046+0000","last_peered":"2024-06-29T14:41:21.444046+0000","last_clean":"2024-06-29T14:41:21.444046+0000","last_became_active":"2024-06-29T14:41:21.442501+0000","last_became_peered":"2024-06-29T14:41:21.442501+0000","last_unstale":"2024-06-29T14:41:21.444046+0000","last_undegraded":"2024-06-29T14:41:21.444046+0000","last_fullsized":"2024-06-29T14:41:21.444046+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:26:42.188988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"142'2","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.451239+0000","last_change":"2024-06-29T14:41:20.413941+0000","last_active":"2024-06-29T14:41:21.451239+0000","last_peered":"2024-06-29T14:41:21.451239+0000","last_clean":"2024-06-29T14:41:21.451239+0000","last_became_active":"2024-06-29T14:41:20.413489+0000","last_became_peered":"2024-06-29T14:41:20.413489+0000","last_unstale":"2024-06-29T14:41:21.451239+0000","last_undegraded":"2024-06-29T14:41:21.451239+0000","last_fullsized":"2024-06-29T14:41:21.451239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:25.307752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426646+0000","last_change":"2024-06-29T14:41:21.439759+0000","last_active":"2024-06-29T14:41:22.426646+0000","last_peered":"2024-06-29T14:41:22.426646+0000","last_clean":"2024-06-29T14:41:22.426646+0000","last_became_active":"2024-06-29T14:41:21.438457+0000","last_became_peered":"2024-06-29T14:41:21.438457+0000","last_unstale":"2024-06-29T14:41:22.426646+0000","last_undegraded":"2024-06-29T14:41:22.426646+0000","last_fullsized":"2024-06-29T14:41:22.426646+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:49:10.598608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"142'1","reported_seq":13,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446729+0000","last_change":"2024-06-29T14:41:20.415077+0000","last_active":"2024-06-29T14:41:21.446729+0000","last_peered":"2024-06-29T14:41:21.446729+0000","last_clean":"2024-06-29T14:41:21.446729+0000","last_became_active":"2024-06-29T14:41:20.414748+0000","last_became_peered":"2024-06-29T14:41:20.414748+0000","last_unstale":"2024-06-29T14:41:21.446729+0000","last_undegraded":"2024-06-29T14:41:21.446729+0000","last_fullsized":"2024-06-29T14:41:21.446729+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:15.607913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.431641+0000","last_change":"2024-06-29T14:41:21.428709+0000","last_active":"2024-06-29T14:41:21.431641+0000","last_peered":"2024-06-29T14:41:21.431641+0000","last_clean":"2024-06-29T14:41:21.431641+0000","last_became_active":"2024-06-29T14:41:21.428293+0000","last_became_peered":"2024-06-29T14:41:21.428293+0000","last_unstale":"2024-06-29T14:41:21.431641+0000","last_undegraded":"2024-06-29T14:41:21.431641+0000","last_fullsized":"2024-06-29T14:41:21.431641+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:10.402070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.413696+0000","last_change":"2024-06-29T14:41:20.417402+0000","last_active":"2024-06-29T14:41:21.413696+0000","last_peered":"2024-06-29T14:41:21.413696+0000","last_clean":"2024-06-29T14:41:21.413696+0000","last_became_active":"2024-06-29T14:41:20.416938+0000","last_became_peered":"2024-06-29T14:41:20.416938+0000","last_unstale":"2024-06-29T14:41:21.413696+0000","last_undegraded":"2024-06-29T14:41:21.413696+0000","last_fullsized":"2024-06-29T14:41:21.413696+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:05.059264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425592+0000","last_change":"2024-06-29T14:41:21.431968+0000","last_active":"2024-06-29T14:41:22.425592+0000","last_peered":"2024-06-29T14:41:22.425592+0000","last_clean":"2024-06-29T14:41:22.425592+0000","last_became_active":"2024-06-29T14:41:21.431636+0000","last_became_peered":"2024-06-29T14:41:21.431636+0000","last_unstale":"2024-06-29T14:41:22.425592+0000","last_undegraded":"2024-06-29T14:41:22.425592+0000","last_fullsized":"2024-06-29T14:41:22.425592+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:36:08.237326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"142'2","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426220+0000","last_change":"2024-06-29T14:41:20.416570+0000","last_active":"2024-06-29T14:41:22.426220+0000","last_peered":"2024-06-29T14:41:22.426220+0000","last_clean":"2024-06-29T14:41:22.426220+0000","last_became_active":"2024-06-29T14:41:20.416245+0000","last_became_peered":"2024-06-29T14:41:20.416245+0000","last_unstale":"2024-06-29T14:41:22.426220+0000","last_undegraded":"2024-06-29T14:41:22.426220+0000","last_fullsized":"2024-06-29T14:41:22.426220+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:46:27.588268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.444144+0000","last_change":"2024-06-29T14:41:21.431994+0000","last_active":"2024-06-29T14:41:21.444144+0000","last_peered":"2024-06-29T14:41:21.444144+0000","last_clean":"2024-06-29T14:41:21.444144+0000","last_became_active":"2024-06-29T14:41:21.430221+0000","last_became_peered":"2024-06-29T14:41:21.430221+0000","last_unstale":"2024-06-29T14:41:21.444144+0000","last_undegraded":"2024-06-29T14:41:21.444144+0000","last_fullsized":"2024-06-29T14:41:21.444144+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:50:38.358451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.412403+0000","last_change":"2024-06-29T14:41:20.413126+0000","last_active":"2024-06-29T14:41:21.412403+0000","last_peered":"2024-06-29T14:41:21.412403+0000","last_clean":"2024-06-29T14:41:21.412403+0000","last_became_active":"2024-06-29T14:41:20.412807+0000","last_became_peered":"2024-06-29T14:41:20.412807+0000","last_unstale":"2024-06-29T14:41:21.412403+0000","last_undegraded":"2024-06-29T14:41:21.412403+0000","last_fullsized":"2024-06-29T14:41:21.412403+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:19.142792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.441298+0000","last_change":"2024-06-29T14:41:21.426823+0000","last_active":"2024-06-29T14:41:21.441298+0000","last_peered":"2024-06-29T14:41:21.441298+0000","last_clean":"2024-06-29T14:41:21.441298+0000","last_became_active":"2024-06-29T14:41:21.426433+0000","last_became_peered":"2024-06-29T14:41:21.426433+0000","last_unstale":"2024-06-29T14:41:21.441298+0000","last_undegraded":"2024-06-29T14:41:21.441298+0000","last_fullsized":"2024-06-29T14:41:21.441298+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:50.495118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.414165+0000","last_change":"2024-06-29T14:41:20.421115+0000","last_active":"2024-06-29T14:41:21.414165+0000","last_peered":"2024-06-29T14:41:21.414165+0000","last_clean":"2024-06-29T14:41:21.414165+0000","last_became_active":"2024-06-29T14:41:20.420750+0000","last_became_peered":"2024-06-29T14:41:20.420750+0000","last_unstale":"2024-06-29T14:41:21.414165+0000","last_undegraded":"2024-06-29T14:41:21.414165+0000","last_fullsized":"2024-06-29T14:41:21.414165+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:04.578774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.445998+0000","last_change":"2024-06-29T14:41:21.444979+0000","last_active":"2024-06-29T14:41:21.445998+0000","last_peered":"2024-06-29T14:41:21.445998+0000","last_clean":"2024-06-29T14:41:21.445998+0000","last_became_active":"2024-06-29T14:41:21.444624+0000","last_became_peered":"2024-06-29T14:41:21.444624+0000","last_unstale":"2024-06-29T14:41:21.445998+0000","last_undegraded":"2024-06-29T14:41:21.445998+0000","last_fullsized":"2024-06-29T14:41:21.445998+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:48.867784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.412104+0000","last_change":"2024-06-29T14:41:20.416766+0000","last_active":"2024-06-29T14:41:21.412104+0000","last_peered":"2024-06-29T14:41:21.412104+0000","last_clean":"2024-06-29T14:41:21.412104+0000","last_became_active":"2024-06-29T14:41:20.416418+0000","last_became_peered":"2024-06-29T14:41:20.416418+0000","last_unstale":"2024-06-29T14:41:21.412104+0000","last_undegraded":"2024-06-29T14:41:21.412104+0000","last_fullsized":"2024-06-29T14:41:21.412104+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:45.259800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426717+0000","last_change":"2024-06-29T14:41:21.442681+0000","last_active":"2024-06-29T14:41:22.426717+0000","last_peered":"2024-06-29T14:41:22.426717+0000","last_clean":"2024-06-29T14:41:22.426717+0000","last_became_active":"2024-06-29T14:41:21.442334+0000","last_became_peered":"2024-06-29T14:41:21.442334+0000","last_unstale":"2024-06-29T14:41:22.426717+0000","last_undegraded":"2024-06-29T14:41:22.426717+0000","last_fullsized":"2024-06-29T14:41:22.426717+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:56.871226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"142'2","reported_seq":16,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426523+0000","last_change":"2024-06-29T14:41:20.418325+0000","last_active":"2024-06-29T14:41:22.426523+0000","last_peered":"2024-06-29T14:41:22.426523+0000","last_clean":"2024-06-29T14:41:22.426523+0000","last_became_active":"2024-06-29T14:41:20.417963+0000","last_became_peered":"2024-06-29T14:41:20.417963+0000","last_unstale":"2024-06-29T14:41:22.426523+0000","last_undegraded":"2024-06-29T14:41:22.426523+0000","last_fullsized":"2024-06-29T14:41:22.426523+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:09.223595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425551+0000","last_change":"2024-06-29T14:41:21.436827+0000","last_active":"2024-06-29T14:41:22.425551+0000","last_peered":"2024-06-29T14:41:22.425551+0000","last_clean":"2024-06-29T14:41:22.425551+0000","last_became_active":"2024-06-29T14:41:21.436028+0000","last_became_peered":"2024-06-29T14:41:21.436028+0000","last_unstale":"2024-06-29T14:41:22.425551+0000","last_undegraded":"2024-06-29T14:41:22.425551+0000","last_fullsized":"2024-06-29T14:41:22.425551+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:18:51.081360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"143'1","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.438712+0000","last_change":"2024-06-29T14:41:20.421040+0000","last_active":"2024-06-29T14:41:22.438712+0000","last_peered":"2024-06-29T14:41:22.438712+0000","last_clean":"2024-06-29T14:41:22.438712+0000","last_became_active":"2024-06-29T14:41:20.420781+0000","last_became_peered":"2024-06-29T14:41:20.420781+0000","last_unstale":"2024-06-29T14:41:22.438712+0000","last_undegraded":"2024-06-29T14:41:22.438712+0000","last_fullsized":"2024-06-29T14:41:22.438712+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:23.360766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.434693+0000","last_change":"2024-06-29T14:41:21.428481+0000","last_active":"2024-06-29T14:41:21.434693+0000","last_peered":"2024-06-29T14:41:21.434693+0000","last_clean":"2024-06-29T14:41:21.434693+0000","last_became_active":"2024-06-29T14:41:21.427719+0000","last_became_peered":"2024-06-29T14:41:21.427719+0000","last_unstale":"2024-06-29T14:41:21.434693+0000","last_undegraded":"2024-06-29T14:41:21.434693+0000","last_fullsized":"2024-06-29T14:41:21.434693+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:37.502969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426086+0000","last_change":"2024-06-29T14:41:20.416455+0000","last_active":"2024-06-29T14:41:22.426086+0000","last_peered":"2024-06-29T14:41:22.426086+0000","last_clean":"2024-06-29T14:41:22.426086+0000","last_became_active":"2024-06-29T14:41:20.416115+0000","last_became_peered":"2024-06-29T14:41:20.416115+0000","last_unstale":"2024-06-29T14:41:22.426086+0000","last_undegraded":"2024-06-29T14:41:22.426086+0000","last_fullsized":"2024-06-29T14:41:22.426086+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:21.317242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.445223+0000","last_change":"2024-06-29T14:41:21.436636+0000","last_active":"2024-06-29T14:41:21.445223+0000","last_peered":"2024-06-29T14:41:21.445223+0000","last_clean":"2024-06-29T14:41:21.445223+0000","last_became_active":"2024-06-29T14:41:21.436290+0000","last_became_peered":"2024-06-29T14:41:21.436290+0000","last_unstale":"2024-06-29T14:41:21.445223+0000","last_undegraded":"2024-06-29T14:41:21.445223+0000","last_fullsized":"2024-06-29T14:41:21.445223+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:59:16.524155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.426812+0000","last_change":"2024-06-29T14:41:20.420444+0000","last_active":"2024-06-29T14:41:22.426812+0000","last_peered":"2024-06-29T14:41:22.426812+0000","last_clean":"2024-06-29T14:41:22.426812+0000","last_became_active":"2024-06-29T14:41:20.420135+0000","last_became_peered":"2024-06-29T14:41:20.420135+0000","last_unstale":"2024-06-29T14:41:22.426812+0000","last_undegraded":"2024-06-29T14:41:22.426812+0000","last_fullsized":"2024-06-29T14:41:22.426812+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:48.424677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425715+0000","last_change":"2024-06-29T14:41:21.430690+0000","last_active":"2024-06-29T14:41:22.425715+0000","last_peered":"2024-06-29T14:41:22.425715+0000","last_clean":"2024-06-29T14:41:22.425715+0000","last_became_active":"2024-06-29T14:41:21.430299+0000","last_became_peered":"2024-06-29T14:41:21.430299+0000","last_unstale":"2024-06-29T14:41:22.425715+0000","last_undegraded":"2024-06-29T14:41:22.425715+0000","last_fullsized":"2024-06-29T14:41:22.425715+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:03.051126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.421972+0000","last_change":"2024-06-29T14:41:20.416367+0000","last_active":"2024-06-29T14:41:21.421972+0000","last_peered":"2024-06-29T14:41:21.421972+0000","last_clean":"2024-06-29T14:41:21.421972+0000","last_became_active":"2024-06-29T14:41:20.416045+0000","last_became_peered":"2024-06-29T14:41:20.416045+0000","last_unstale":"2024-06-29T14:41:21.421972+0000","last_undegraded":"2024-06-29T14:41:21.421972+0000","last_fullsized":"2024-06-29T14:41:21.421972+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:23.360766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.422601+0000","last_change":"2024-06-29T14:41:21.420512+0000","last_active":"2024-06-29T14:41:21.422601+0000","last_peered":"2024-06-29T14:41:21.422601+0000","last_clean":"2024-06-29T14:41:21.422601+0000","last_became_active":"2024-06-29T14:41:21.419330+0000","last_became_peered":"2024-06-29T14:41:21.419330+0000","last_unstale":"2024-06-29T14:41:21.422601+0000","last_undegraded":"2024-06-29T14:41:21.422601+0000","last_fullsized":"2024-06-29T14:41:21.422601+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:00.239358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.420945+0000","last_change":"2024-06-29T14:41:20.410434+0000","last_active":"2024-06-29T14:41:21.420945+0000","last_peered":"2024-06-29T14:41:21.420945+0000","last_clean":"2024-06-29T14:41:21.420945+0000","last_became_active":"2024-06-29T14:41:20.410059+0000","last_became_peered":"2024-06-29T14:41:20.410059+0000","last_unstale":"2024-06-29T14:41:21.420945+0000","last_undegraded":"2024-06-29T14:41:21.420945+0000","last_fullsized":"2024-06-29T14:41:21.420945+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:38.247088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.434894+0000","last_change":"2024-06-29T14:41:21.429353+0000","last_active":"2024-06-29T14:41:21.434894+0000","last_peered":"2024-06-29T14:41:21.434894+0000","last_clean":"2024-06-29T14:41:21.434894+0000","last_became_active":"2024-06-29T14:41:21.429031+0000","last_became_peered":"2024-06-29T14:41:21.429031+0000","last_unstale":"2024-06-29T14:41:21.434894+0000","last_undegraded":"2024-06-29T14:41:21.434894+0000","last_fullsized":"2024-06-29T14:41:21.434894+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:52.963873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425757+0000","last_change":"2024-06-29T14:41:20.421511+0000","last_active":"2024-06-29T14:41:22.425757+0000","last_peered":"2024-06-29T14:41:22.425757+0000","last_clean":"2024-06-29T14:41:22.425757+0000","last_became_active":"2024-06-29T14:41:20.420971+0000","last_became_peered":"2024-06-29T14:41:20.420971+0000","last_unstale":"2024-06-29T14:41:22.425757+0000","last_undegraded":"2024-06-29T14:41:22.425757+0000","last_fullsized":"2024-06-29T14:41:22.425757+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:27.981198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.425351+0000","last_change":"2024-06-29T14:41:21.431579+0000","last_active":"2024-06-29T14:41:22.425351+0000","last_peered":"2024-06-29T14:41:22.425351+0000","last_clean":"2024-06-29T14:41:22.425351+0000","last_became_active":"2024-06-29T14:41:21.431183+0000","last_became_peered":"2024-06-29T14:41:21.431183+0000","last_unstale":"2024-06-29T14:41:22.425351+0000","last_undegraded":"2024-06-29T14:41:22.425351+0000","last_fullsized":"2024-06-29T14:41:22.425351+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:51.161729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.412481+0000","last_change":"2024-06-29T14:41:20.411196+0000","last_active":"2024-06-29T14:41:21.412481+0000","last_peered":"2024-06-29T14:41:21.412481+0000","last_clean":"2024-06-29T14:41:21.412481+0000","last_became_active":"2024-06-29T14:41:20.410842+0000","last_became_peered":"2024-06-29T14:41:20.410842+0000","last_unstale":"2024-06-29T14:41:21.412481+0000","last_undegraded":"2024-06-29T14:41:21.412481+0000","last_fullsized":"2024-06-29T14:41:21.412481+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:23.902951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.446079+0000","last_change":"2024-06-29T14:41:21.444006+0000","last_active":"2024-06-29T14:41:21.446079+0000","last_peered":"2024-06-29T14:41:21.446079+0000","last_clean":"2024-06-29T14:41:21.446079+0000","last_became_active":"2024-06-29T14:41:21.443672+0000","last_became_peered":"2024-06-29T14:41:21.443672+0000","last_unstale":"2024-06-29T14:41:21.446079+0000","last_undegraded":"2024-06-29T14:41:21.446079+0000","last_fullsized":"2024-06-29T14:41:21.446079+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.391383+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.391383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:17.965041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"142'8","reported_seq":20,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.452219+0000","last_change":"2024-06-29T14:41:20.410691+0000","last_active":"2024-06-29T14:41:21.452219+0000","last_peered":"2024-06-29T14:41:21.452219+0000","last_clean":"2024-06-29T14:41:21.452219+0000","last_became_active":"2024-06-29T14:41:20.410281+0000","last_became_peered":"2024-06-29T14:41:20.410281+0000","last_unstale":"2024-06-29T14:41:21.452219+0000","last_undegraded":"2024-06-29T14:41:21.452219+0000","last_fullsized":"2024-06-29T14:41:21.452219+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.385108+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.385108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:56.592360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'156","reported_seq":643,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.412274+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:41:21.412274+0000","last_peered":"2024-06-29T14:41:21.412274+0000","last_clean":"2024-06-29T14:41:21.412274+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:41:21.412274+0000","last_undegraded":"2024-06-29T14:41:21.412274+0000","last_fullsized":"2024-06-29T14:41:21.412274+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":245,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":245,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542379,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59864,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311976,"statfs":{"total":96636764160,"available":96575463424,"internally_reserved":0,"allocated":2367488,"data_stored":1482136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542380,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1761280,"data_stored":891574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542379,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55112,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316728,"statfs":{"total":96636764160,"available":96580329472,"internally_reserved":0,"allocated":1765376,"data_stored":891152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542378,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1769472,"data_stored":892604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509676,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1773568,"data_stored":892638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575084,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59264,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312576,"statfs":{"total":96636764160,"available":96576077824,"internally_reserved":0,"allocated":1757184,"data_stored":890992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32916,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1773568,"data_stored":897734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509677,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37648,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334192,"statfs":{"total":96636764160,"available":96598212608,"internally_reserved":0,"allocated":2355200,"data_stored":1481382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:41:25.940 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:41:25.940 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:41:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.067+0000 7f8e671e0640 1 -- 172.21.15.59:0/1122504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 msgr2=0x7f8e4800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.067+0000 7f8e671e0640 1 --2- 172.21.15.59:0/1122504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 0x7f8e4800b9c0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800c770 tx=0x7f8e48005830 comp rx=0 tx=0).stop 2024-06-29T14:41:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.067+0000 7f8e671e0640 1 -- 172.21.15.59:0/1122504 shutdown_connections 2024-06-29T14:41:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.067+0000 7f8e671e0640 1 --2- 172.21.15.59:0/1122504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 0x7f8e4800b9c0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.067+0000 7f8e671e0640 1 -- 172.21.15.59:0/1122504 >> 172.21.15.59:0/1122504 conn(0x7f8e600d3e40 msgr2=0x7f8e600d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.067+0000 7f8e671e0640 1 -- 172.21.15.59:0/1122504 shutdown_connections 2024-06-29T14:41:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.067+0000 7f8e671e0640 1 -- 172.21.15.59:0/1122504 wait complete. 2024-06-29T14:41:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.069+0000 7f8e671e0640 1 Processor -- start 2024-06-29T14:41:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.069+0000 7f8e671e0640 1 -- start start 2024-06-29T14:41:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e671e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 0x7f8e600cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e671e0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e6014fe60 0x7f8e600ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e671e0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e600c81a0 0x7f8e600c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e671e0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8e600cf000 con 0x7f8e48005250 2024-06-29T14:41:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e671e0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8e600b2c20 con 0x7f8e600c81a0 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e671e0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8e600cf180 con 0x7f8e6014fe60 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 0x7f8e600cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 0x7f8e600cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37790/0 (socket says 172.21.15.59:37790) 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 -- 172.21.15.59:0/3093518910 learned_addr learned my addr 172.21.15.59:0/3093518910 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e6577d640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e600c81a0 0x7f8e600c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e5ffff640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e6014fe60 0x7f8e600ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 -- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e6014fe60 msgr2=0x7f8e600ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e6014fe60 0x7f8e600ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 -- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e600c81a0 msgr2=0x7f8e600c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e600c81a0 0x7f8e600c85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 -- 172.21.15.59:0/3093518910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e600ccaf0 con 0x7f8e48005250 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e6577d640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e600c81a0 0x7f8e600c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.071+0000 7f8e64f7c640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 0x7f8e600cd8b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f8e50004e50 tx=0x7f8e50001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:26.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.072+0000 7f8e5d7fa640 1 -- 172.21.15.59:0/3093518910 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500107d0 con 0x7f8e48005250 2024-06-29T14:41:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.072+0000 7f8e5d7fa640 1 -- 172.21.15.59:0/3093518910 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e50016070 con 0x7f8e48005250 2024-06-29T14:41:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.072+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e600ccdd0 con 0x7f8e48005250 2024-06-29T14:41:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.072+0000 7f8e5d7fa640 1 -- 172.21.15.59:0/3093518910 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e50010cc0 con 0x7f8e48005250 2024-06-29T14:41:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.072+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e600cd390 con 0x7f8e48005250 2024-06-29T14:41:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.073+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e60050470 con 0x7f8e48005250 2024-06-29T14:41:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.073+0000 7f8e5d7fa640 1 -- 172.21.15.59:0/3093518910 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f8e5001b460 con 0x7f8e48005250 2024-06-29T14:41:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.074+0000 7f8e5d7fa640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8e4c0763f0 0x7f8e4c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.074+0000 7f8e5d7fa640 1 -- 172.21.15.59:0/3093518910 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7f8e50098340 con 0x7f8e48005250 2024-06-29T14:41:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.074+0000 7f8e5ffff640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8e4c0763f0 0x7f8e4c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.075+0000 7f8e5ffff640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8e4c0763f0 0x7f8e4c078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8e6006dc70 tx=0x7f8e54005f90 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:26.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.078+0000 7f8e5d7fa640 1 -- 172.21.15.59:0/3093518910 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8e60050470 con 0x7f8e48005250 2024-06-29T14:41:26.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.345+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8e6015dd80 con 0x7f8e48005250 2024-06-29T14:41:26.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.348+0000 7f8e5d7fa640 1 -- 172.21.15.59:0/3093518910 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) ==== 190+0+147 (secure 0 0 0) 0x7f8e500618d0 con 0x7f8e48005250 2024-06-29T14:41:26.346 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:41:26.346 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:41:26.346 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T14:41:26.346 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T14:41:26.346 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:41:26.346 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T14:41:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8e4c0763f0 msgr2=0x7f8e4c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8e4c0763f0 0x7f8e4c078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8e6006dc70 tx=0x7f8e54005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 msgr2=0x7f8e600cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 0x7f8e600cd8b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f8e50004e50 tx=0x7f8e50001630 comp rx=0 tx=0).stop 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 shutdown_connections 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e600c81a0 0x7f8e600c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f8e4c0763f0 0x7f8e4c078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8e6014fe60 0x7f8e600ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 --2- 172.21.15.59:0/3093518910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e48005250 0x7f8e600cd8b0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 >> 172.21.15.59:0/3093518910 conn(0x7f8e600d3e40 msgr2=0x7f8e600d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 shutdown_connections 2024-06-29T14:41:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.350+0000 7f8e671e0640 1 -- 172.21.15.59:0/3093518910 wait complete. 2024-06-29T14:41:26.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.479+0000 7fdc499cf640 1 -- 172.21.15.59:0/966064501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440d23c0 msgr2=0x7fdc440b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.479+0000 7fdc499cf640 1 --2- 172.21.15.59:0/966064501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440d23c0 0x7fdc440b3990 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009830 tx=0x7fdc3402a510 comp rx=0 tx=0).stop 2024-06-29T14:41:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.479+0000 7fdc499cf640 1 -- 172.21.15.59:0/966064501 shutdown_connections 2024-06-29T14:41:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.479+0000 7fdc499cf640 1 --2- 172.21.15.59:0/966064501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc440b3ed0 0x7fdc4415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.479+0000 7fdc499cf640 1 --2- 172.21.15.59:0/966064501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440d23c0 0x7fdc440b3990 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.479+0000 7fdc499cf640 1 -- 172.21.15.59:0/966064501 >> 172.21.15.59:0/966064501 conn(0x7fdc440d3ee0 msgr2=0x7fdc440d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.479+0000 7fdc499cf640 1 -- 172.21.15.59:0/966064501 shutdown_connections 2024-06-29T14:41:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.479+0000 7fdc499cf640 1 -- 172.21.15.59:0/966064501 wait complete. 2024-06-29T14:41:26.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.481+0000 7fdc499cf640 1 Processor -- start 2024-06-29T14:41:26.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.481+0000 7fdc499cf640 1 -- start start 2024-06-29T14:41:26.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc499cf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440b3ed0 0x7fdc44143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc499cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc440d23c0 0x7fdc44154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc499cf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc4415ee10 0x7fdc44154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc499cf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdc44160240 con 0x7fdc440b3ed0 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc499cf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441600c0 con 0x7fdc440d23c0 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc499cf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdc441603c0 con 0x7fdc4415ee10 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440b3ed0 0x7fdc44143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440b3ed0 0x7fdc44143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37818/0 (socket says 172.21.15.59:37818) 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 -- 172.21.15.59:0/3830431543 learned_addr learned my addr 172.21.15.59:0/3830431543 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc427fc640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc440d23c0 0x7fdc44154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 -- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc4415ee10 msgr2=0x7fdc44154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc437fe640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc4415ee10 0x7fdc44154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc4415ee10 0x7fdc44154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 -- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc440d23c0 msgr2=0x7fdc44154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc440d23c0 0x7fdc44154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 -- 172.21.15.59:0/3830431543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc441552c0 con 0x7fdc440b3ed0 2024-06-29T14:41:26.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc427fc640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc440d23c0 0x7fdc44154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:26.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc437fe640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc4415ee10 0x7fdc44154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:26.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc42ffd640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440b3ed0 0x7fdc44143600 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009950 tx=0x7fdc34033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:26.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc2bfff640 1 -- 172.21.15.59:0/3830431543 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34005c90 con 0x7fdc440b3ed0 2024-06-29T14:41:26.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc2bfff640 1 -- 172.21.15.59:0/3830431543 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc34031e50 con 0x7fdc440b3ed0 2024-06-29T14:41:26.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc441555a0 con 0x7fdc440b3ed0 2024-06-29T14:41:26.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.483+0000 7fdc2bfff640 1 -- 172.21.15.59:0/3830431543 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34005260 con 0x7fdc440b3ed0 2024-06-29T14:41:26.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.484+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc44155b60 con 0x7fdc440b3ed0 2024-06-29T14:41:26.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.485+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4415e8d0 con 0x7fdc440b3ed0 2024-06-29T14:41:26.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.485+0000 7fdc2bfff640 1 -- 172.21.15.59:0/3830431543 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fdc34005400 con 0x7fdc440b3ed0 2024-06-29T14:41:26.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.486+0000 7fdc2bfff640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fdc300763f0 0x7fdc30078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.486+0000 7fdc2bfff640 1 -- 172.21.15.59:0/3830431543 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7fdc340b8b90 con 0x7fdc440b3ed0 2024-06-29T14:41:26.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.486+0000 7fdc427fc640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fdc300763f0 0x7fdc30078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.487+0000 7fdc427fc640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fdc300763f0 0x7fdc30078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc4414f870 tx=0x7fdc38005dd0 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:26.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.491+0000 7fdc2bfff640 1 -- 172.21.15.59:0/3830431543 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc3403f030 con 0x7fdc440b3ed0 2024-06-29T14:41:26.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.725+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc44141650 con 0x7fdc440b3ed0 2024-06-29T14:41:26.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.725+0000 7fdc2bfff640 1 -- 172.21.15.59:0/3830431543 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5228 (secure 0 0 0) 0x7fdc340821b0 con 0x7fdc440b3ed0 2024-06-29T14:41:26.724 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:26.724 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":72,"btime":"2024-06-29T14:41:24:429207+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7442,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/4194962668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4194962668},{"type":"v1","addr":"172.21.15.153:6836","nonce":4194962668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7445,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1840307553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1840307553},{"type":"v1","addr":"172.21.15.153:6837","nonce":1840307553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1010388227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1010388227},{"type":"v1","addr":"172.21.15.59:6834","nonce":1010388227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:20.795436+0000","modified":"2024-06-29T14:41:23.821908+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8844":{"gid":8844,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/957887336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":957887336},{"type":"v1","addr":"172.21.15.59:6835","nonce":957887336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8844,"qdb_cluster":[8844]},"id":8}]} 2024-06-29T14:41:26.724 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 72 2024-06-29T14:41:26.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fdc300763f0 msgr2=0x7fdc30078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fdc300763f0 0x7fdc30078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc4414f870 tx=0x7fdc38005dd0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440b3ed0 msgr2=0x7fdc44143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440b3ed0 0x7fdc44143600 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009950 tx=0x7fdc34033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 shutdown_connections 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fdc300763f0 0x7fdc30078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdc4415ee10 0x7fdc44154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc440d23c0 0x7fdc44154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 --2- 172.21.15.59:0/3830431543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc440b3ed0 0x7fdc44143600 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 >> 172.21.15.59:0/3830431543 conn(0x7fdc440d3ee0 msgr2=0x7fdc440ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 shutdown_connections 2024-06-29T14:41:26.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:26.728+0000 7fdc499cf640 1 -- 172.21.15.59:0/3830431543 wait complete. 2024-06-29T14:41:26.957 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:41:26.960 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:41:27.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:41:20.795436+0000', 'modified': '2024-06-29T14:41:23.821908+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8844}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8844': {'gid': 8844, 'name': 'a', 'rank': 0, 'incarnation': 68, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/957887336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 957887336}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 957887336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8844, 'qdb_cluster': [8844]} 2024-06-29T14:41:27.141 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:41:27.141 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:41:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.263+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1553255580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 msgr2=0x7fc79c166ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.263+0000 7fc7a3ee4640 1 --2- 172.21.15.59:0/1553255580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 0x7fc79c166ca0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fc794009830 tx=0x7fc79402a480 comp rx=0 tx=0).stop 2024-06-29T14:41:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.263+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1553255580 shutdown_connections 2024-06-29T14:41:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.263+0000 7fc7a3ee4640 1 --2- 172.21.15.59:0/1553255580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 0x7fc79c166ca0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.263+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1553255580 >> 172.21.15.59:0/1553255580 conn(0x7fc79c06c950 msgr2=0x7fc79c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.263+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1553255580 shutdown_connections 2024-06-29T14:41:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.263+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1553255580 wait complete. 2024-06-29T14:41:27.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.265+0000 7fc7a3ee4640 1 Processor -- start 2024-06-29T14:41:27.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.265+0000 7fc7a3ee4640 1 -- start start 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.266+0000 7fc7a3ee4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc79c0b3740 0x7fc79c1624c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.266+0000 7fc7a3ee4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 0x7fc79c158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.266+0000 7fc7a3ee4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc79c13cf60 0x7fc79c13d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.266+0000 7fc7a3ee4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c167570 con 0x7fc79c0b3b50 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.266+0000 7fc7a3ee4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c1673f0 con 0x7fc79c0b3740 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.266+0000 7fc7a3ee4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc79c1676f0 con 0x7fc79c13cf60 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.266+0000 7fc7a147f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 0x7fc79c158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a147f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 0x7fc79c158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37844/0 (socket says 172.21.15.59:37844) 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a147f640 1 -- 172.21.15.59:0/1000561101 learned_addr learned my addr 172.21.15.59:0/1000561101 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc79c0b3740 0x7fc79c1624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a2481640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc79c13cf60 0x7fc79c13d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a147f640 1 -- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc79c13cf60 msgr2=0x7fc79c13d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a147f640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc79c13cf60 0x7fc79c13d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a147f640 1 -- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc79c0b3740 msgr2=0x7fc79c1624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a147f640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc79c0b3740 0x7fc79c1624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a147f640 1 -- 172.21.15.59:0/1000561101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c13d8b0 con 0x7fc79c0b3b50 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a147f640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 0x7fc79c158e10 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fc79000b3f0 tx=0x7fc79000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc79a7fc640 1 -- 172.21.15.59:0/1000561101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790015020 con 0x7fc79c0b3b50 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc79a7fc640 1 -- 172.21.15.59:0/1000561101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc790011940 con 0x7fc79c0b3b50 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1000561101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79c1f2e30 con 0x7fc79c0b3b50 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc79a7fc640 1 -- 172.21.15.59:0/1000561101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790011d90 con 0x7fc79c0b3b50 2024-06-29T14:41:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.267+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1000561101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79c1f33d0 con 0x7fc79c0b3b50 2024-06-29T14:41:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.269+0000 7fc79a7fc640 1 -- 172.21.15.59:0/1000561101 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fc790018020 con 0x7fc79c0b3b50 2024-06-29T14:41:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.270+0000 7fc79a7fc640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc7880763f0 0x7fc788078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.270+0000 7fc79a7fc640 1 -- 172.21.15.59:0/1000561101 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7fc7900978b0 con 0x7fc79c0b3b50 2024-06-29T14:41:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.270+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc7880763f0 0x7fc788078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.271+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc7880763f0 0x7fc788078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b450 tx=0x7fc78c00d040 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.271+0000 7fc7aada7740 1 Processor -- start 2024-06-29T14:41:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.271+0000 7fc7aada7740 1 -- start start 2024-06-29T14:41:27.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.272+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c665270 0x55f86ca516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.272+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca353a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.272+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca358e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.272+0000 7fc7aada7740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55f86c9cdbc0 con 0x55f86c63c3b0 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.272+0000 7fc7aada7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f86c97c630 con 0x55f86c644590 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.272+0000 7fc7aada7740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55f86c96bbd0 con 0x55f86c665270 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca353a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37852/0 (socket says 172.21.15.59:37852) 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 -- 172.21.15.59:0/3900892406 learned_addr learned my addr 172.21.15.59:0/3900892406 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c665270 0x55f86ca516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a2481640 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 -- 172.21.15.59:0/3900892406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c665270 msgr2=0x55f86ca516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c665270 0x55f86ca516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 -- 172.21.15.59:0/3900892406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 msgr2=0x55f86ca358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca358e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 -- 172.21.15.59:0/3900892406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f86c968ca0 con 0x55f86c63c3b0 2024-06-29T14:41:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7a147f640 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca353a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fc790060690 tx=0x7fc790060990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef7d8d8079d4291 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7837fe640 1 -- 172.21.15.59:0/3900892406 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7900696b0 con 0x55f86c63c3b0 2024-06-29T14:41:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7837fe640 1 -- 172.21.15.59:0/3900892406 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc79005cc20 con 0x55f86c63c3b0 2024-06-29T14:41:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.273+0000 7fc7837fe640 1 -- 172.21.15.59:0/3900892406 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79005fe50 con 0x55f86c63c3b0 2024-06-29T14:41:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 -- 172.21.15.59:0/3900892406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 msgr2=0x55f86ca353a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca353a0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fc790060690 tx=0x7fc790060990 comp rx=0 tx=0).stop 2024-06-29T14:41:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 -- 172.21.15.59:0/3900892406 shutdown_connections 2024-06-29T14:41:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca353a0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 --2- 172.21.15.59:0/3900892406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c665270 0x55f86ca516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 -- 172.21.15.59:0/3900892406 >> 172.21.15.59:0/3900892406 conn(0x55f86c6593c0 msgr2=0x55f86ca08f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 -- 172.21.15.59:0/3900892406 shutdown_connections 2024-06-29T14:41:27.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.274+0000 7fc7aada7740 1 -- 172.21.15.59:0/3900892406 wait complete. 2024-06-29T14:41:27.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.275+0000 7fc7aada7740 1 -- start start 2024-06-29T14:41:27.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.275+0000 7fc7aada7740 1 Processor -- start 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 0x55f86ca29b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca4ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c665270 0x55f86ca4f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7aada7740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55f86c909ae0 con 0x55f86c665270 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7aada7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f86c96fc10 con 0x55f86c644590 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7aada7740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55f86c6df620 con 0x55f86c63c3b0 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7aada7740 10 client.0 _ll_register_callbacks cb 0x55f86c916480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c665270 0x55f86ca4f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c665270 0x55f86ca4f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37862/0 (socket says 172.21.15.59:37862) 2024-06-29T14:41:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 -- 172.21.15.59:0/561281535 learned_addr learned my addr 172.21.15.59:0/561281535 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 0x55f86ca29b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a147f640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca4ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 -- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 msgr2=0x55f86ca29b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 0x55f86ca29b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 -- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 msgr2=0x55f86ca4ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca4ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 -- 172.21.15.59:0/561281535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f86c6d0850 con 0x55f86c665270 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 0x55f86ca29b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a147f640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca4ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc7a2481640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c665270 0x55f86ca4f200 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc79402aa70 tx=0x7fc794003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da22f37270b166d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc780ff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.277+0000 7fc780ff9640 1 -- 172.21.15.59:0/561281535 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794003ba0 con 0x55f86c665270 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc780ff9640 1 -- 172.21.15.59:0/561281535 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7fc794003d40 con 0x55f86c665270 2024-06-29T14:41:27.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 1 -- 172.21.15.59:0/561281535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc79005cc20 con 0x55f86c665270 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc780ff9640 1 -- 172.21.15.59:0/561281535 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fc794031420 con 0x55f86c665270 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 10 client.8976 fetch_fsmap learned FSMap version 72 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 1 -- 172.21.15.59:0/561281535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) -- 0x55f86c7476b0 con 0x55f86c665270 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc780ff9640 1 -- 172.21.15.59:0/561281535 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc79005cc20 con 0x55f86c665270 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc780ff9640 1 -- 172.21.15.59:0/561281535 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 72) ==== 3502+0+0 (secure 0 0 0) 0x7fc79404d050 con 0x55f86c665270 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 10 client.8976 fetch_fsmap finished waiting for FSMap version 72 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 10 client.8976 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 20 client.8976 populate_metadata read hostname 'smithi059' 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] conn(0x55f86c623e40 0x55f86ca62b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 4 client.8976 mds_command: new command op to 8844 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7aada7740 1 -- 172.21.15.59:0/561281535 --> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f86c97e580 con 0x55f86c623e40 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.278+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] conn(0x55f86c623e40 0x55f86ca62b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.279+0000 7fc780ff9640 10 client.8976 ms_handle_connect on v2:172.21.15.59:6833/957887336 2024-06-29T14:41:27.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.279+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] conn(0x55f86c623e40 0x55f86ca62b40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c29a4dcdb66dfe01 server_cookie=410a6d3322e5f30e in_seq=0 out_seq=0 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.279+0000 7fc780ff9640 1 -- 172.21.15.59:0/561281535 <== mds.0 v2:172.21.15.59:6833/957887336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55f86c97e580 con 0x55f86c623e40 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.279+0000 7fc780ff9640 10 client.8976 handle_command_reply: tid=0 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.279+0000 7fc7aada7740 1 client.8976 shutdown 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 -- 172.21.15.59:0/561281535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c665270 msgr2=0x55f86ca4f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c665270 0x55f86ca4f200 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc79402aa70 tx=0x7fc794003820 comp rx=0 tx=0).stop 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 -- 172.21.15.59:0/561281535 shutdown_connections 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] conn(0x55f86c623e40 0x55f86ca62b40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c665270 0x55f86ca4f200 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c644590 0x55f86ca4ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 --2- 172.21.15.59:0/561281535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 0x55f86ca29b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc780ff9640 0 client.8976 ms_handle_reset on v2:172.21.15.59:6833/957887336 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 -- 172.21.15.59:0/561281535 >> 172.21.15.59:0/561281535 conn(0x55f86c6593c0 msgr2=0x55f86ca31c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 -- 172.21.15.59:0/561281535 shutdown_connections 2024-06-29T14:41:27.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 1 -- 172.21.15.59:0/561281535 wait complete. 2024-06-29T14:41:27.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.280+0000 7fc7aada7740 20 client.8976 trim_cache size 0 max 16384 2024-06-29T14:41:27.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.299+0000 7fc7aada7740 1 Processor -- start 2024-06-29T14:41:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.299+0000 7fc7aada7740 1 -- start start 2024-06-29T14:41:27.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 0x55f86ca07040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c644590 0x55f86ca73590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 0x55f86ca4fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7aada7740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c1fed00 con 0x55f86c644590 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7aada7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c1f6d30 con 0x55f86c665270 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7aada7740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc79c1fe7b0 con 0x55f86c63c3b0 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c644590 0x55f86ca73590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c644590 0x55f86ca73590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37868/0 (socket says 172.21.15.59:37868) 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 -- 172.21.15.59:0/2781742780 learned_addr learned my addr 172.21.15.59:0/2781742780 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 -- 172.21.15.59:0/2781742780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 msgr2=0x55f86ca07040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 --2- 172.21.15.59:0/2781742780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 0x55f86ca07040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 -- 172.21.15.59:0/2781742780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 msgr2=0x55f86ca4fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 --2- 172.21.15.59:0/2781742780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 0x55f86ca4fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 -- 172.21.15.59:0/2781742780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f86ca31610 con 0x55f86c644590 2024-06-29T14:41:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc7a147f640 1 --2- 172.21.15.59:0/2781742780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c644590 0x55f86ca73590 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc79005dc00 tx=0x7fc790009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1255c7af31fadbc server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc782ffd640 1 -- 172.21.15.59:0/2781742780 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79007f070 con 0x55f86c644590 2024-06-29T14:41:27.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc782ffd640 1 -- 172.21.15.59:0/2781742780 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7900696b0 con 0x55f86c644590 2024-06-29T14:41:27.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.301+0000 7fc782ffd640 1 -- 172.21.15.59:0/2781742780 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790095070 con 0x55f86c644590 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 -- 172.21.15.59:0/2781742780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c644590 msgr2=0x55f86ca73590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2781742780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c644590 0x55f86ca73590 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc79005dc00 tx=0x7fc790009220 comp rx=0 tx=0).stop 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 -- 172.21.15.59:0/2781742780 shutdown_connections 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2781742780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 0x55f86ca4fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2781742780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c644590 0x55f86ca73590 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2781742780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c63c3b0 0x55f86ca07040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 -- 172.21.15.59:0/2781742780 >> 172.21.15.59:0/2781742780 conn(0x55f86c6593c0 msgr2=0x55f86ca31bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 -- 172.21.15.59:0/2781742780 shutdown_connections 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.302+0000 7fc7aada7740 1 -- 172.21.15.59:0/2781742780 wait complete. 2024-06-29T14:41:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.303+0000 7fc7aada7740 1 -- start start 2024-06-29T14:41:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.303+0000 7fc7aada7740 1 Processor -- start 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca5d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c644590 0x55f86caccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7aada7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 0x55f86ca31b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7aada7740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55f86ca2d8a0 con 0x55f86c63c3b0 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7aada7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f86ca332a0 con 0x55f86c665270 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7aada7740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55f86ca33c90 con 0x55f86c644590 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7aada7740 10 client.0 _ll_register_callbacks cb 0x7fc794005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca5d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca5d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37876/0 (socket says 172.21.15.59:37876) 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 -- 172.21.15.59:0/2644428549 learned_addr learned my addr 172.21.15.59:0/2644428549 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:27.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a2481640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 0x55f86ca31b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a147f640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c644590 0x55f86caccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 -- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c644590 msgr2=0x55f86caccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c644590 0x55f86caccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 -- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 msgr2=0x55f86ca31b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 0x55f86ca31b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 -- 172.21.15.59:0/2644428549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f86cacd2e0 con 0x55f86c63c3b0 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a147f640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c644590 0x55f86caccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a2481640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 0x55f86ca31b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7a1c80640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca5d830 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x55f86ca340f0 tx=0x7fc78c0096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bdf82e46e992efb server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc780ff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc780ff9640 1 -- 172.21.15.59:0/2644428549 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c032060 con 0x55f86c63c3b0 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc780ff9640 1 -- 172.21.15.59:0/2644428549 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7fc78c016bb0 con 0x55f86c63c3b0 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.305+0000 7fc7aada7740 1 -- 172.21.15.59:0/2644428549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7900696b0 con 0x55f86c63c3b0 2024-06-29T14:41:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc780ff9640 1 -- 172.21.15.59:0/2644428549 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7fc78c038050 con 0x55f86c63c3b0 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7aada7740 10 client.8982 fetch_fsmap learned FSMap version 72 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7aada7740 1 -- 172.21.15.59:0/2644428549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) -- 0x55f86ca616c0 con 0x55f86c63c3b0 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc780ff9640 1 -- 172.21.15.59:0/2644428549 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7900696b0 con 0x55f86c63c3b0 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc780ff9640 1 -- 172.21.15.59:0/2644428549 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 72) ==== 3502+0+0 (secure 0 0 0) 0x7fc78c03b020 con 0x55f86c63c3b0 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7aada7740 10 client.8982 fetch_fsmap finished waiting for FSMap version 72 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7aada7740 10 client.8982 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7aada7740 20 client.8982 populate_metadata read hostname 'smithi059' 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] conn(0x55f86ca50210 0x55f86ca2c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7aada7740 4 client.8982 mds_command: new command op to 8844 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7aada7740 1 -- 172.21.15.59:0/2644428549 --> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] -- command(tid 0: {"prefix": "status"}) -- 0x55f86c97a020 con 0x55f86ca50210 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7a147f640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] conn(0x55f86ca50210 0x55f86ca2c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc780ff9640 10 client.8982 ms_handle_connect on v2:172.21.15.59:6833/957887336 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.306+0000 7fc7a147f640 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] conn(0x55f86ca50210 0x55f86ca2c200 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f98b29525862fb8 server_cookie=8c77dacf6f9fca1c in_seq=0 out_seq=0 2024-06-29T14:41:27.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc780ff9640 1 -- 172.21.15.59:0/2644428549 <== mds.0 v2:172.21.15.59:6833/957887336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x55f86c97a020 con 0x55f86ca50210 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc780ff9640 10 client.8982 handle_command_reply: tid=0 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 client.8982 shutdown 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 -- 172.21.15.59:0/2644428549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 msgr2=0x55f86ca5d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca5d830 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x55f86ca340f0 tx=0x7fc78c0096b0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 -- 172.21.15.59:0/2644428549 shutdown_connections 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.59:6833/957887336,v1:172.21.15.59:6835/957887336] conn(0x55f86ca50210 0x55f86ca2c200 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f86c665270 0x55f86ca31b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55f86c644590 0x55f86caccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc780ff9640 0 client.8982 ms_handle_reset on v2:172.21.15.59:6833/957887336 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 --2- 172.21.15.59:0/2644428549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55f86c63c3b0 0x55f86ca5d830 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 -- 172.21.15.59:0/2644428549 >> 172.21.15.59:0/2644428549 conn(0x55f86c6593c0 msgr2=0x55f86ca06d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 -- 172.21.15.59:0/2644428549 shutdown_connections 2024-06-29T14:41:27.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 1 -- 172.21.15.59:0/2644428549 wait complete. 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.307+0000 7fc7aada7740 20 client.8982 trim_cache size 0 max 16384 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 8844, 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8894616470000001, 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 72, 2024-06-29T14:41:27.311 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 143, 2024-06-29T14:41:27.312 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 143, 2024-06-29T14:41:27.312 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.506632382999999 2024-06-29T14:41:27.312 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:41:27.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc7880763f0 msgr2=0x7fc788078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc7880763f0 0x7fc788078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b450 tx=0x7fc78c00d040 comp rx=0 tx=0).stop 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1000561101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 msgr2=0x7fc79c158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 0x7fc79c158e10 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fc79000b3f0 tx=0x7fc79000b8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1000561101 shutdown_connections 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc7880763f0 0x7fc788078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc79c13cf60 0x7fc79c13d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc79c0b3b50 0x7fc79c158e10 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 --2- 172.21.15.59:0/1000561101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc79c0b3740 0x7fc79c1624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1000561101 >> 172.21.15.59:0/1000561101 conn(0x7fc79c06c950 msgr2=0x7fc79c071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1000561101 shutdown_connections 2024-06-29T14:41:27.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.308+0000 7fc7a3ee4640 1 -- 172.21.15.59:0/1000561101 wait complete. 2024-06-29T14:41:27.317 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:41:27.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.437+0000 7fad2c168640 1 -- 172.21.15.59:0/2465635168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad2415ee10 msgr2=0x7fad2415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.437+0000 7fad2c168640 1 --2- 172.21.15.59:0/2465635168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad2415ee10 0x7fad2415f9b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fad240d15d0 tx=0x7fad1c02aa30 comp rx=0 tx=0).stop 2024-06-29T14:41:27.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.437+0000 7fad2c168640 1 -- 172.21.15.59:0/2465635168 shutdown_connections 2024-06-29T14:41:27.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.437+0000 7fad2c168640 1 --2- 172.21.15.59:0/2465635168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad2415ee10 0x7fad2415f9b0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.437+0000 7fad2c168640 1 --2- 172.21.15.59:0/2465635168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad240b3ed0 0x7fad2415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.437+0000 7fad2c168640 1 --2- 172.21.15.59:0/2465635168 >> v2:172.21.15.153:3300/0 conn(0x7fad240d23c0 0x7fad240b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.437+0000 7fad2c168640 1 -- 172.21.15.59:0/2465635168 >> 172.21.15.59:0/2465635168 conn(0x7fad240d3ee0 msgr2=0x7fad240d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.438+0000 7fad2c168640 1 -- 172.21.15.59:0/2465635168 shutdown_connections 2024-06-29T14:41:27.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.438+0000 7fad2c168640 1 -- 172.21.15.59:0/2465635168 wait complete. 2024-06-29T14:41:27.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.440+0000 7fad2c168640 1 Processor -- start 2024-06-29T14:41:27.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.440+0000 7fad2c168640 1 -- start start 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad2c168640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad240b3ed0 0x7fad2415ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad2c168640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad240d23c0 0x7fad24155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad2c168640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2415ee10 0x7fad24155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad2c168640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fad24160070 con 0x7fad240b3ed0 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad2c168640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fad2415fef0 con 0x7fad2415ee10 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad2c168640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fad241601f0 con 0x7fad240d23c0 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29f04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad240b3ed0 0x7fad2415ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29f04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad240b3ed0 0x7fad2415ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37894/0 (socket says 172.21.15.59:37894) 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29f04640 1 -- 172.21.15.59:0/3193742627 learned_addr learned my addr 172.21.15.59:0/3193742627 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad2a705640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2415ee10 0x7fad24155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29703640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad240d23c0 0x7fad24155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29f04640 1 -- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad240d23c0 msgr2=0x7fad24155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29f04640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad240d23c0 0x7fad24155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29f04640 1 -- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2415ee10 msgr2=0x7fad24155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29f04640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2415ee10 0x7fad24155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.441+0000 7fad29f04640 1 -- 172.21.15.59:0/3193742627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2415ab10 con 0x7fad240b3ed0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.442+0000 7fad29f04640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad240b3ed0 0x7fad2415ddc0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fad2000d700 tx=0x7fad2000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.442+0000 7fad1a7fc640 1 -- 172.21.15.59:0/3193742627 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20017070 con 0x7fad240b3ed0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.442+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2415adf0 con 0x7fad240b3ed0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.442+0000 7fad1a7fc640 1 -- 172.21.15.59:0/3193742627 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad20004310 con 0x7fad240b3ed0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.442+0000 7fad1a7fc640 1 -- 172.21.15.59:0/3193742627 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20009a20 con 0x7fad240b3ed0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.442+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2415b3b0 con 0x7fad240b3ed0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.443+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad240b3990 con 0x7fad240b3ed0 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.447+0000 7fad1a7fc640 1 -- 172.21.15.59:0/3193742627 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fad20009bc0 con 0x7fad240b3ed0 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.447+0000 7fad1a7fc640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad080763f0 0x7fad08078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.447+0000 7fad1a7fc640 1 -- 172.21.15.59:0/3193742627 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7fad20097fb0 con 0x7fad240b3ed0 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.447+0000 7fad29703640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad080763f0 0x7fad08078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.448+0000 7fad29703640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad080763f0 0x7fad08078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fad240cd1a0 tx=0x7fad1400b260 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.450+0000 7fad1a7fc640 1 -- 172.21.15.59:0/3193742627 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fad20060e60 con 0x7fad240b3ed0 2024-06-29T14:41:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.682+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fad240cda80 con 0x7fad240b3ed0 2024-06-29T14:41:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.682+0000 7fad1a7fc640 1 -- 172.21.15.59:0/3193742627 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v72) ==== 53+0+73 (secure 0 0 0) 0x7fad20065160 con 0x7fad240b3ed0 2024-06-29T14:41:27.681 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad080763f0 msgr2=0x7fad08078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad080763f0 0x7fad08078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fad240cd1a0 tx=0x7fad1400b260 comp rx=0 tx=0).stop 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad240b3ed0 msgr2=0x7fad2415ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad240b3ed0 0x7fad2415ddc0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fad2000d700 tx=0x7fad2000dbc0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 shutdown_connections 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad2415ee10 0x7fad24155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fad080763f0 0x7fad08078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad240d23c0 0x7fad24155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 --2- 172.21.15.59:0/3193742627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad240b3ed0 0x7fad2415ddc0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 >> 172.21.15.59:0/3193742627 conn(0x7fad240d3ee0 msgr2=0x7fad2406c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 shutdown_connections 2024-06-29T14:41:27.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27.685+0000 7fad2c168640 1 -- 172.21.15.59:0/3193742627 wait complete. 2024-06-29T14:41:27.694 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:41:27.694 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:41:27.694 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:41:27.694 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:41:27.694 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:27.695 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:41:27.695 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:41:27.695 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:41:27.695 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:41:27.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:27.695 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:41:27.711 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:41:27.711 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:41:27.711 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:41:27.712 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 41563sec preferred_lft 41563sec 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:41:27.713 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T14:41:27.714 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.714 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:41:27.714 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:41:27.714 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T14:41:27.714 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:27.714 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:41:27.777 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:41:27.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:27.778 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:41:27.838 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:41:27.838 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:41:27.838 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:41:27.838 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:41:27.838 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:41:27.839 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:41:27.839 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:41:27.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:41:27.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:41:27.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:41:27.839 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:41:27.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:41:27.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:41:27.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:27.967 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:41:27.983 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:41:27.983 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:41:27.983 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:41:27.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:27.984 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:41:28.051 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:41:28.051 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:28.106 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:41:28.107 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:41:28.107 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:28.162 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:41:28.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:28.162 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:41:28.258 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:41:28.258 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:41:28.258 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:41:28.268 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:41:28.269 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:41:28.269 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:41:28.269 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:41:28.269 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:41:28.275 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:41:28.275 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:41:28.275 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:41:28.275 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:41:28.275 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:41:28.276 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:41:28.276 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:41:28.276 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:41:28.276 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:41:28.276 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:41:28.276 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:41:28.276 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:41:28.276 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:41:28.277 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:41:28.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:28.278 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:28.303 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:41:28.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:41:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:41:28.407 INFO:teuthology.orchestra.run.smithi059.stdout:client8398 2024-06-29T14:41:28.407 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:28.407 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:28.424 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:41:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.584+0000 7f95f9a38640 1 -- 172.21.15.59:0/579838140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 msgr2=0x7f95f40ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.584+0000 7f95f9a38640 1 --2- 172.21.15.59:0/579838140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 0x7f95f40ce550 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f95ec005fd0 tx=0x7f95ec02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.584+0000 7f95f9a38640 1 -- 172.21.15.59:0/579838140 shutdown_connections 2024-06-29T14:41:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.584+0000 7f95f9a38640 1 --2- 172.21.15.59:0/579838140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 0x7f95f40ce550 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.584+0000 7f95f9a38640 1 -- 172.21.15.59:0/579838140 >> 172.21.15.59:0/579838140 conn(0x7f95f40d45b0 msgr2=0x7f95f40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.585+0000 7f95f9a38640 1 -- 172.21.15.59:0/579838140 shutdown_connections 2024-06-29T14:41:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.585+0000 7f95f9a38640 1 -- 172.21.15.59:0/579838140 wait complete. 2024-06-29T14:41:28.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.586+0000 7f95f9a38640 1 Processor -- start 2024-06-29T14:41:28.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.586+0000 7f95f9a38640 1 -- start start 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f9a38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95f415e0a0 0x7f95f4159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f9a38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 0x7f95f40cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f9a38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95f40cc290 0x7f95f4158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f9a38640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f95f40b4490 con 0x7f95f4162770 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f9a38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95f40b4310 con 0x7f95f40cc290 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f9a38640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f95f40b4610 con 0x7f95f415e0a0 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 0x7f95f40cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 0x7f95f40cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37906/0 (socket says 172.21.15.59:37906) 2024-06-29T14:41:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 -- 172.21.15.59:0/2328856135 learned_addr learned my addr 172.21.15.59:0/2328856135 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 -- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95f415e0a0 msgr2=0x7f95f4159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95f415e0a0 0x7f95f4159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 -- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95f40cc290 msgr2=0x7f95f4158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95f40cc290 0x7f95f4158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 -- 172.21.15.59:0/2328856135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f40c73e0 con 0x7f95f4162770 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95f27fc640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 0x7f95f40cbd50 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00c3d0 tx=0x7f95dc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.588+0000 7f95ebfff640 1 -- 172.21.15.59:0/2328856135 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc017070 con 0x7f95f4162770 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.589+0000 7f95ebfff640 1 -- 172.21.15.59:0/2328856135 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95dc0053c0 con 0x7f95f4162770 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.589+0000 7f95ebfff640 1 -- 172.21.15.59:0/2328856135 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc003850 con 0x7f95f4162770 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.589+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f40ca220 con 0x7f95f4162770 2024-06-29T14:41:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.589+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f40cb270 con 0x7f95f4162770 2024-06-29T14:41:28.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.590+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f40b4b30 con 0x7f95f4162770 2024-06-29T14:41:28.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.590+0000 7f95ebfff640 1 -- 172.21.15.59:0/2328856135 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f95dc01a070 con 0x7f95f4162770 2024-06-29T14:41:28.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.591+0000 7f95ebfff640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f95d4076420 0x7f95d4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.591+0000 7f95ebfff640 1 -- 172.21.15.59:0/2328856135 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7f95dc096fb0 con 0x7f95f4162770 2024-06-29T14:41:28.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.591+0000 7f95f2ffd640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f95d4076420 0x7f95d4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.592+0000 7f95f2ffd640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f95d4076420 0x7f95d4078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f95ec034000 tx=0x7f95ec004a40 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:28.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.595+0000 7f95ebfff640 1 -- 172.21.15.59:0/2328856135 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95dc060520 con 0x7f95f4162770 2024-06-29T14:41:28.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.814+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 --> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f95f4068210 con 0x7f95d4076420 2024-06-29T14:41:28.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.817+0000 7f95ebfff640 1 -- 172.21.15.59:0/2328856135 <== mgr.7259 v2:172.21.15.153:6832/776633953 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f95f4068210 con 0x7f95d4076420 2024-06-29T14:41:28.814 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:41:28.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:28.814 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T14:41:28.815 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:28.815 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f95d4076420 msgr2=0x7f95d4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f95d4076420 0x7f95d4078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f95ec034000 tx=0x7f95ec004a40 comp rx=0 tx=0).stop 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 msgr2=0x7f95f40cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 0x7f95f40cbd50 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f95dc00c3d0 tx=0x7f95dc00c890 comp rx=0 tx=0).stop 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 shutdown_connections 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95f40cc290 0x7f95f4158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95f4162770 0x7f95f40cbd50 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f95d4076420 0x7f95d4078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 --2- 172.21.15.59:0/2328856135 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95f415e0a0 0x7f95f4159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 >> 172.21.15.59:0/2328856135 conn(0x7f95f40d45b0 msgr2=0x7f95f40b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:28.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 shutdown_connections 2024-06-29T14:41:28.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.818+0000 7f95f9a38640 1 -- 172.21.15.59:0/2328856135 wait complete. 2024-06-29T14:41:28.828 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:41:28.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.949+0000 7f424c09e640 1 -- 172.21.15.59:0/3547868412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440b3740 msgr2=0x7f42440b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.949+0000 7f424c09e640 1 --2- 172.21.15.59:0/3547868412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440b3740 0x7f42440b3b50 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f4238009830 tx=0x7f423802a510 comp rx=0 tx=0).stop 2024-06-29T14:41:28.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.949+0000 7f424c09e640 1 -- 172.21.15.59:0/3547868412 shutdown_connections 2024-06-29T14:41:28.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.949+0000 7f424c09e640 1 --2- 172.21.15.59:0/3547868412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f424415c2f0 0x7f424415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.949+0000 7f424c09e640 1 --2- 172.21.15.59:0/3547868412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440b3740 0x7f42440b3b50 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.949+0000 7f424c09e640 1 -- 172.21.15.59:0/3547868412 >> 172.21.15.59:0/3547868412 conn(0x7f424406c950 msgr2=0x7f4244073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:28.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.949+0000 7f424c09e640 1 -- 172.21.15.59:0/3547868412 shutdown_connections 2024-06-29T14:41:28.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.949+0000 7f424c09e640 1 -- 172.21.15.59:0/3547868412 wait complete. 2024-06-29T14:41:28.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.951+0000 7f424c09e640 1 Processor -- start 2024-06-29T14:41:28.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.951+0000 7f424c09e640 1 -- start start 2024-06-29T14:41:28.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.952+0000 7f424c09e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42440b3740 0x7f42441f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.952+0000 7f424c09e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f424415c2f0 0x7f42440b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.952+0000 7f424c09e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440ae140 0x7f42440ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.952+0000 7f424c09e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f42440b5fd0 con 0x7f42440ae140 2024-06-29T14:41:28.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.952+0000 7f424c09e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42440b5e50 con 0x7f42440b3740 2024-06-29T14:41:28.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.952+0000 7f424c09e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f42440b6150 con 0x7f424415c2f0 2024-06-29T14:41:28.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440ae140 0x7f42440ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440ae140 0x7f42440ae550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37918/0 (socket says 172.21.15.59:37918) 2024-06-29T14:41:28.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 -- 172.21.15.59:0/552983380 learned_addr learned my addr 172.21.15.59:0/552983380 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:28.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f4249639640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f424415c2f0 0x7f42440b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f4249e3a640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42440b3740 0x7f42441f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 -- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f424415c2f0 msgr2=0x7f42440b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f424415c2f0 0x7f42440b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 -- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42440b3740 msgr2=0x7f42441f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42440b3740 0x7f42441f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 -- 172.21.15.59:0/552983380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42440aea90 con 0x7f42440ae140 2024-06-29T14:41:28.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f4249639640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f424415c2f0 0x7f42440b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:28.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f4249e3a640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42440b3740 0x7f42441f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:28.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424a63b640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440ae140 0x7f42440ae550 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f423c00e770 tx=0x7f423c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:28.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f42427fc640 1 -- 172.21.15.59:0/552983380 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c00ccc0 con 0x7f42440ae140 2024-06-29T14:41:28.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f42427fc640 1 -- 172.21.15.59:0/552983380 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f423c00ee80 con 0x7f42440ae140 2024-06-29T14:41:28.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42440ac4e0 con 0x7f42440ae140 2024-06-29T14:41:28.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f42427fc640 1 -- 172.21.15.59:0/552983380 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c005280 con 0x7f42440ae140 2024-06-29T14:41:28.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.953+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42440acaa0 con 0x7f42440ae140 2024-06-29T14:41:28.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.954+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42440b6670 con 0x7f42440ae140 2024-06-29T14:41:28.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.955+0000 7f42427fc640 1 -- 172.21.15.59:0/552983380 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f423c005420 con 0x7f42440ae140 2024-06-29T14:41:28.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.956+0000 7f42427fc640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4230076420 0x7f4230078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.956+0000 7f42427fc640 1 -- 172.21.15.59:0/552983380 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7f423c0984c0 con 0x7f42440ae140 2024-06-29T14:41:28.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.956+0000 7f4249e3a640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4230076420 0x7f4230078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.956+0000 7f4249e3a640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4230076420 0x7f4230078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f42340017a0 tx=0x7f4234009040 comp rx=0 tx=0).ready entity=mgr.7259 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:28.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:28.960+0000 7f42427fc640 1 -- 172.21.15.59:0/552983380 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f42440aea90 con 0x7f42440ae140 2024-06-29T14:41:29.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.202+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f42440b1f10 con 0x7f42440ae140 2024-06-29T14:41:29.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.449+0000 7f42427fc640 1 -- 172.21.15.59:0/552983380 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v69) ==== 92+0+0 (secure 0 0 0) 0x7f423c061940 con 0x7f42440ae140 2024-06-29T14:41:29.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.453+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4230076420 msgr2=0x7f4230078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.453+0000 7f424c09e640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4230076420 0x7f4230078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f42340017a0 tx=0x7f4234009040 comp rx=0 tx=0).stop 2024-06-29T14:41:29.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.453+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440ae140 msgr2=0x7f42440ae550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f42427fc640 1 -- 172.21.15.59:0/552983380 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 69) ==== 98320+0+0 (secure 0 0 0) 0x7f423c096830 con 0x7f42440ae140 2024-06-29T14:41:29.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440ae140 0x7f42440ae550 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f423c00e770 tx=0x7f423c00ec30 comp rx=0 tx=0).stop 2024-06-29T14:41:29.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 shutdown_connections 2024-06-29T14:41:29.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42440ae140 0x7f42440ae550 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f4230076420 0x7f4230078930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f424415c2f0 0x7f42440b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 --2- 172.21.15.59:0/552983380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42440b3740 0x7f42441f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 >> 172.21.15.59:0/552983380 conn(0x7f424406c950 msgr2=0x7f4244071640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:29.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 shutdown_connections 2024-06-29T14:41:29.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.454+0000 7f424c09e640 1 -- 172.21.15.59:0/552983380 wait complete. 2024-06-29T14:41:29.464 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:41:29.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.591+0000 7fc33fa17640 1 -- 172.21.15.59:0/2829029629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3380d1c30 msgr2=0x7fc338166a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.591+0000 7fc33fa17640 1 --2- 172.21.15.59:0/2829029629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3380d1c30 0x7fc338166a20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc330009860 tx=0x7fc33002a5a0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.591+0000 7fc33fa17640 1 -- 172.21.15.59:0/2829029629 shutdown_connections 2024-06-29T14:41:29.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.591+0000 7fc33fa17640 1 --2- 172.21.15.59:0/2829029629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3380d1c30 0x7fc338166a20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.591+0000 7fc33fa17640 1 -- 172.21.15.59:0/2829029629 >> 172.21.15.59:0/2829029629 conn(0x7fc338073e10 msgr2=0x7fc338074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:29.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.591+0000 7fc33fa17640 1 -- 172.21.15.59:0/2829029629 shutdown_connections 2024-06-29T14:41:29.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.591+0000 7fc33fa17640 1 -- 172.21.15.59:0/2829029629 wait complete. 2024-06-29T14:41:29.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.593+0000 7fc33fa17640 1 Processor -- start 2024-06-29T14:41:29.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.593+0000 7fc33fa17640 1 -- start start 2024-06-29T14:41:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.594+0000 7fc33fa17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3380b2270 0x7fc338152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.594+0000 7fc33fa17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3380b2fa0 0x7fc33806f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.594+0000 7fc33fa17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3380d1c30 0x7fc33815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.594+0000 7fc33fa17640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc3381670e0 con 0x7fc3380d1c30 2024-06-29T14:41:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.594+0000 7fc33fa17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc338166f60 con 0x7fc3380b2270 2024-06-29T14:41:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.594+0000 7fc33fa17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc338167260 con 0x7fc3380b2fa0 2024-06-29T14:41:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33dfb4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3380d1c30 0x7fc33815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33dfb4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3380d1c30 0x7fc33815a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37950/0 (socket says 172.21.15.59:37950) 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33dfb4640 1 -- 172.21.15.59:0/3492649300 learned_addr learned my addr 172.21.15.59:0/3492649300 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33d7b3640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3380b2270 0x7fc338152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33cfb2640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3380b2fa0 0x7fc33806f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33d7b3640 1 -- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3380b2fa0 msgr2=0x7fc33806f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33d7b3640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3380b2fa0 0x7fc33806f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33d7b3640 1 -- 172.21.15.59:0/3492649300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3380d1c30 msgr2=0x7fc33815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33d7b3640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3380d1c30 0x7fc33815a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33d7b3640 1 -- 172.21.15.59:0/3492649300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33806f8c0 con 0x7fc3380b2270 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33dfb4640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3380d1c30 0x7fc33815a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.595+0000 7fc33d7b3640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3380b2270 0x7fc338152540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc32800cae0 tx=0x7fc32800a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.596+0000 7fc335ffb640 1 -- 172.21.15.59:0/3492649300 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328016020 con 0x7fc3380b2270 2024-06-29T14:41:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.596+0000 7fc335ffb640 1 -- 172.21.15.59:0/3492649300 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc32800acd0 con 0x7fc3380b2270 2024-06-29T14:41:29.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.596+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc33815e2c0 con 0x7fc3380b2270 2024-06-29T14:41:29.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.596+0000 7fc335ffb640 1 -- 172.21.15.59:0/3492649300 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328012680 con 0x7fc3380b2270 2024-06-29T14:41:29.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.596+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33815e860 con 0x7fc3380b2270 2024-06-29T14:41:29.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.597+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc33815bc20 con 0x7fc3380b2270 2024-06-29T14:41:29.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.598+0000 7fc335ffb640 1 -- 172.21.15.59:0/3492649300 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 69) ==== 98320+0+0 (secure 0 0 0) 0x7fc328019020 con 0x7fc3380b2270 2024-06-29T14:41:29.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.598+0000 7fc335ffb640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc324076440 0x7fc324078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.598+0000 7fc335ffb640 1 -- 172.21.15.59:0/3492649300 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7fc328097da0 con 0x7fc3380b2270 2024-06-29T14:41:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.598+0000 7fc33cfb2640 1 -- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc324076440 msgr2=0x7fc324078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/776633953 2024-06-29T14:41:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.598+0000 7fc33cfb2640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc324076440 0x7fc324078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:29.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.603+0000 7fc335ffb640 1 -- 172.21.15.59:0/3492649300 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc328020030 con 0x7fc3380b2270 2024-06-29T14:41:29.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.797+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fc338156380 con 0x7fc3380b2270 2024-06-29T14:41:29.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.799+0000 7fc33cfb2640 1 -- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc324076440 msgr2=0x7fc324078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/776633953 2024-06-29T14:41:29.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.799+0000 7fc33cfb2640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc324076440 0x7fc324078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:29.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.806+0000 7fc335ffb640 1 -- 172.21.15.59:0/3492649300 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) ==== 138+0+0 (secure 0 0 0) 0x7fc328060d60 con 0x7fc3380b2270 2024-06-29T14:41:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc324076440 msgr2=0x7fc324078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc324076440 0x7fc324078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3380b2270 msgr2=0x7fc338152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3380b2270 0x7fc338152540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc32800cae0 tx=0x7fc32800a060 comp rx=0 tx=0).stop 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 shutdown_connections 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3380d1c30 0x7fc33815a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fc324076440 0x7fc324078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3380b2fa0 0x7fc33806f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 --2- 172.21.15.59:0/3492649300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3380b2270 0x7fc338152540 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 >> 172.21.15.59:0/3492649300 conn(0x7fc338073e10 msgr2=0x7fc3380b5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 shutdown_connections 2024-06-29T14:41:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.808+0000 7fc33fa17640 1 -- 172.21.15.59:0/3492649300 wait complete. 2024-06-29T14:41:29.818 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:41:29.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.942+0000 7fb8b4ace640 1 -- 172.21.15.59:0/2112830349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8b00d0840 msgr2=0x7fb8b015f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.942+0000 7fb8b4ace640 1 --2- 172.21.15.59:0/2112830349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8b00d0840 0x7fb8b015f240 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009860 tx=0x7fb8a002a530 comp rx=0 tx=0).stop 2024-06-29T14:41:29.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.942+0000 7fb8b4ace640 1 -- 172.21.15.59:0/2112830349 shutdown_connections 2024-06-29T14:41:29.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.942+0000 7fb8b4ace640 1 --2- 172.21.15.59:0/2112830349 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8b00d0840 0x7fb8b015f240 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.942+0000 7fb8b4ace640 1 -- 172.21.15.59:0/2112830349 >> 172.21.15.59:0/2112830349 conn(0x7fb8b00b3560 msgr2=0x7fb8b00b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:29.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.942+0000 7fb8b4ace640 1 -- 172.21.15.59:0/2112830349 shutdown_connections 2024-06-29T14:41:29.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.942+0000 7fb8b4ace640 1 -- 172.21.15.59:0/2112830349 wait complete. 2024-06-29T14:41:29.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.944+0000 7fb8b4ace640 1 Processor -- start 2024-06-29T14:41:29.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.944+0000 7fb8b4ace640 1 -- start start 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.945+0000 7fb8b4ace640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8b00b13d0 0x7fb8b01e5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.945+0000 7fb8b4ace640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8b00b1ef0 0x7fb8b0145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.945+0000 7fb8b4ace640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b00d0840 0x7fb8b0146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.945+0000 7fb8b4ace640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b0160290 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.945+0000 7fb8b4ace640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b0160110 con 0x7fb8b00d0840 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8b4ace640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b0160410 con 0x7fb8b00b13d0 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8b00b1ef0 0x7fb8b0145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8b00b1ef0 0x7fb8b0145d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37954/0 (socket says 172.21.15.59:37954) 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 -- 172.21.15.59:0/189792159 learned_addr learned my addr 172.21.15.59:0/189792159 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8aed9d640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b00d0840 0x7fb8b0146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8ae59c640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8b00b13d0 0x7fb8b01e5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 -- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8b00b13d0 msgr2=0x7fb8b01e5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8b00b13d0 0x7fb8b01e5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 -- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b00d0840 msgr2=0x7fb8b0146280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b00d0840 0x7fb8b0146280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 -- 172.21.15.59:0/189792159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b01e5950 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a7fff640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8b00b1ef0 0x7fb8b0145d40 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fb89800c4b0 tx=0x7fb89800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a6ffd640 1 -- 172.21.15.59:0/189792159 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898017070 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a6ffd640 1 -- 172.21.15.59:0/189792159 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8980035f0 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b0157dd0 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.946+0000 7fb8a6ffd640 1 -- 172.21.15.59:0/189792159 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898003a40 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.947+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b0158390 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.948+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b00b17e0 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.948+0000 7fb8a6ffd640 1 -- 172.21.15.59:0/189792159 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 69) ==== 98320+0+0 (secure 0 0 0) 0x7fb89801a070 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.949+0000 7fb8a6ffd640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb890076440 0x7fb890078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.949+0000 7fb8ae59c640 1 -- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb890076440 msgr2=0x7fb890078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/776633953 2024-06-29T14:41:29.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.949+0000 7fb8a6ffd640 1 -- 172.21.15.59:0/189792159 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7fb898096f10 con 0x7fb8b00b1ef0 2024-06-29T14:41:29.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.949+0000 7fb8ae59c640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb890076440 0x7fb890078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:29.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:29.953+0000 7fb8a6ffd640 1 -- 172.21.15.59:0/189792159 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb89805fed0 con 0x7fb8b00b1ef0 2024-06-29T14:41:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.145+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fb8b0157af0 con 0x7fb8b00b1ef0 2024-06-29T14:41:30.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.150+0000 7fb8ae59c640 1 -- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb890076440 msgr2=0x7fb890078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/776633953 2024-06-29T14:41:30.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.150+0000 7fb8ae59c640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb890076440 0x7fb890078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:30.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.151+0000 7fb8a6ffd640 1 -- 172.21.15.59:0/189792159 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) ==== 133+0+0 (secure 0 0 0) 0x7fb8980641d0 con 0x7fb8b00b1ef0 2024-06-29T14:41:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb890076440 msgr2=0x7fb890078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb890076440 0x7fb890078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8b00b1ef0 msgr2=0x7fb8b0145d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8b00b1ef0 0x7fb8b0145d40 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fb89800c4b0 tx=0x7fb89800c970 comp rx=0 tx=0).stop 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 shutdown_connections 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b00d0840 0x7fb8b0146280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8b00b1ef0 0x7fb8b0145d40 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7fb890076440 0x7fb890078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 --2- 172.21.15.59:0/189792159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8b00b13d0 0x7fb8b01e5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 >> 172.21.15.59:0/189792159 conn(0x7fb8b00b3560 msgr2=0x7fb8b006fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 shutdown_connections 2024-06-29T14:41:30.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.153+0000 7fb8b4ace640 1 -- 172.21.15.59:0/189792159 wait complete. 2024-06-29T14:41:30.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:30.163 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-06-29T14:41:30.185 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-06-29T14:41:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.351+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3825173331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 msgr2=0x7f0f740b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.351+0000 7f0f7bf31640 1 --2- 172.21.15.59:0/3825173331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 0x7f0f740b3990 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0f74161d40 tx=0x7f0f7002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.351+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3825173331 shutdown_connections 2024-06-29T14:41:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.351+0000 7f0f7bf31640 1 --2- 172.21.15.59:0/3825173331 >> v2:172.21.15.153:3300/0 conn(0x7f0f7415efe0 0x7f0f7415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.351+0000 7f0f7bf31640 1 --2- 172.21.15.59:0/3825173331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 0x7f0f740b3990 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.351+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3825173331 >> 172.21.15.59:0/3825173331 conn(0x7f0f740d3ee0 msgr2=0x7f0f740d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:30.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.351+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3825173331 shutdown_connections 2024-06-29T14:41:30.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.351+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3825173331 wait complete. 2024-06-29T14:41:30.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.353+0000 7f0f7bf31640 1 Processor -- start 2024-06-29T14:41:30.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.353+0000 7f0f7bf31640 1 -- start start 2024-06-29T14:41:30.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f7bf31640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f740b3ed0 0x7f0f74148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f7bf31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 0x7f0f74146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f7bf31640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f7415efe0 0x7f0f741ea690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f7bf31640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0f74160280 con 0x7f0f740d23c0 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f7bf31640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f74160100 con 0x7f0f740b3ed0 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f7bf31640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0f74160400 con 0x7f0f7415efe0 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f794cc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 0x7f0f74146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f794cc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 0x7f0f74146430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37980/0 (socket says 172.21.15.59:37980) 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f794cc640 1 -- 172.21.15.59:0/3695462525 learned_addr learned my addr 172.21.15.59:0/3695462525 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f79ccd640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f740b3ed0 0x7f0f74148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.354+0000 7f0f7a4ce640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f7415efe0 0x7f0f741ea690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f794cc640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f7415efe0 msgr2=0x7f0f741ea690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f794cc640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f7415efe0 0x7f0f741ea690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f794cc640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f740b3ed0 msgr2=0x7f0f74148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f794cc640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f740b3ed0 0x7f0f74148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f794cc640 1 -- 172.21.15.59:0/3695462525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f74145390 con 0x7f0f740d23c0 2024-06-29T14:41:30.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f79ccd640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f740b3ed0 0x7f0f74148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:30.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f7a4ce640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f7415efe0 0x7f0f741ea690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:30.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f794cc640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 0x7f0f74146430 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400c370 tx=0x7f0f6400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:30.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64017070 con 0x7f0f740d23c0 2024-06-29T14:41:30.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f640053c0 con 0x7f0f740d23c0 2024-06-29T14:41:30.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f74145670 con 0x7f0f740d23c0 2024-06-29T14:41:30.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64003850 con 0x7f0f740d23c0 2024-06-29T14:41:30.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.355+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f74145c30 con 0x7f0f740d23c0 2024-06-29T14:41:30.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.356+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f741450e0 con 0x7f0f740d23c0 2024-06-29T14:41:30.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.357+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 69) ==== 98320+0+0 (secure 0 0 0) 0x7f0f6401a070 con 0x7f0f740d23c0 2024-06-29T14:41:30.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.357+0000 7f0f6a7fc640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 0x7f0f58078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.357+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6129+0+0 (secure 0 0 0) 0x7f0f64096e30 con 0x7f0f740d23c0 2024-06-29T14:41:30.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.357+0000 7f0f79ccd640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 msgr2=0x7f0f58078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/776633953 2024-06-29T14:41:30.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.358+0000 7f0f79ccd640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 0x7f0f58078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:30.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.362+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f740b42e0 con 0x7f0f740d23c0 2024-06-29T14:41:30.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.559+0000 7f0f79ccd640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 msgr2=0x7f0f58078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/776633953 2024-06-29T14:41:30.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.559+0000 7f0f79ccd640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 0x7f0f58078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:30.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.602+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 --> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f0f740b3990 con 0x7f0f58076440 2024-06-29T14:41:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.961+0000 7f0f79ccd640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 msgr2=0x7f0f58078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/776633953 2024-06-29T14:41:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:30.961+0000 7f0f79ccd640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 0x7f0f58078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:41:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:31.762+0000 7f0f79ccd640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 msgr2=0x7f0f58078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/776633953 2024-06-29T14:41:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:31.762+0000 7f0f79ccd640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 0x7f0f58078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:41:33.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:33.051+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 70) ==== 97929+0+0 (secure 0 0 0) 0x7f0f640951a0 con 0x7f0f740d23c0 2024-06-29T14:41:33.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:33.051+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 msgr2=0x7f0f58078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:33.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:33.051+0000 7f0f6a7fc640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/776633953,v1:172.21.15.153:6833/776633953] conn(0x7f0f58076440 0x7f0f58078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:33.190 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.191+0000 7f37087f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.190 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.191+0000 7f37087f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.191 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.191+0000 7f37087f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.191 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.191+0000 7f37087f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.191 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.191+0000 7f37087f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.191 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.192+0000 7f370b7fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.191 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.192+0000 7f370b7fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.191 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.192+0000 7f370b7fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.191 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.192+0000 7f370b7fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.191 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:33.192+0000 7f370b7fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:34.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.059+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 71) ==== 98056+0+0 (secure 0 0 0) 0x7f0f6405c110 con 0x7f0f740d23c0 2024-06-29T14:41:34.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.059+0000 7f0f6a7fc640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f5803afd0 0x7f0f580809f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.059+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f0f640666d0 con 0x7f0f5803afd0 2024-06-29T14:41:34.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.060+0000 7f0f79ccd640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f5803afd0 0x7f0f580809f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.061+0000 7f0f79ccd640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f5803afd0 0x7f0f580809f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0f70033000 tx=0x7f0f70004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:34.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.130+0000 7f0f6a7fc640 1 -- 172.21.15.59:0/3695462525 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+45 (secure 0 0 0) 0x7f0f640666d0 con 0x7f0f5803afd0 2024-06-29T14:41:34.129 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1/minutes 2024-06-29T14:41:34.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f5803afd0 msgr2=0x7f0f580809f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7bf31640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f5803afd0 0x7f0f580809f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0f70033000 tx=0x7f0f70004730 comp rx=0 tx=0).stop 2024-06-29T14:41:34.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 msgr2=0x7f0f74146430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7bf31640 1 --2- 172.21.15.59:0/3695462525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f740d23c0 0x7f0f74146430 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400c370 tx=0x7f0f6400c830 comp rx=0 tx=0).stop 2024-06-29T14:41:34.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7a4ce640 1 -- 172.21.15.59:0/3695462525 reap_dead start 2024-06-29T14:41:34.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 shutdown_connections 2024-06-29T14:41:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 >> 172.21.15.59:0/3695462525 conn(0x7f0f740d3ee0 msgr2=0x7f0f740ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 shutdown_connections 2024-06-29T14:41:34.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.133+0000 7f0f7bf31640 1 -- 172.21.15.59:0/3695462525 wait complete. 2024-06-29T14:41:34.144 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-06-29T14:41:34.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:34.144 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-06-29T14:41:34.162 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-06-29T14:41:34.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.326+0000 7f117ef7f640 1 -- 172.21.15.59:0/2993285187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780d1a60 msgr2=0x7f11780b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.326+0000 7f117ef7f640 1 --2- 172.21.15.59:0/2993285187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780d1a60 0x7f11780b5b30 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f11780b2680 tx=0x7f117002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.326+0000 7f117ef7f640 1 -- 172.21.15.59:0/2993285187 shutdown_connections 2024-06-29T14:41:34.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.326+0000 7f117ef7f640 1 --2- 172.21.15.59:0/2993285187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780d1a60 0x7f11780b5b30 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.327+0000 7f117ef7f640 1 --2- 172.21.15.59:0/2993285187 >> v2:172.21.15.153:3300/0 conn(0x7f11780b2fa0 0x7f11780d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.327+0000 7f117ef7f640 1 -- 172.21.15.59:0/2993285187 >> 172.21.15.59:0/2993285187 conn(0x7f1178073e10 msgr2=0x7f1178074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:34.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.327+0000 7f117ef7f640 1 -- 172.21.15.59:0/2993285187 shutdown_connections 2024-06-29T14:41:34.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.327+0000 7f117ef7f640 1 -- 172.21.15.59:0/2993285187 wait complete. 2024-06-29T14:41:34.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.328+0000 7f117ef7f640 1 Processor -- start 2024-06-29T14:41:34.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.328+0000 7f117ef7f640 1 -- start start 2024-06-29T14:41:34.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117ef7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780b2270 0x7f11780adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117ef7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11780b2fa0 0x7f1178148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117ef7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11780d1a60 0x7f11780b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117ef7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f11780b6400 con 0x7f11780b2270 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117ef7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11780b6280 con 0x7f11780d1a60 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117ef7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11780b6580 con 0x7f11780b2fa0 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780b2270 0x7f11780adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780b2270 0x7f11780adba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33984/0 (socket says 172.21.15.59:33984) 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 -- 172.21.15.59:0/1216078706 learned_addr learned my addr 172.21.15.59:0/1216078706 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117d51c640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11780d1a60 0x7f11780b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f1177fff640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11780b2fa0 0x7f1178148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 -- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11780b2fa0 msgr2=0x7f1178148320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11780b2fa0 0x7f1178148320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 -- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11780d1a60 msgr2=0x7f11780b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11780d1a60 0x7f11780b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 -- 172.21.15.59:0/1216078706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1178148d60 con 0x7f11780b2270 2024-06-29T14:41:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f1177fff640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11780b2fa0 0x7f1178148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.330+0000 7f117cd1b640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780b2270 0x7f11780adba0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f116c00c3f0 tx=0x7f116c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.331+0000 7f11757fa640 1 -- 172.21.15.59:0/1216078706 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c017070 con 0x7f11780b2270 2024-06-29T14:41:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.331+0000 7f11757fa640 1 -- 172.21.15.59:0/1216078706 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f116c0035f0 con 0x7f11780b2270 2024-06-29T14:41:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.331+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1178149040 con 0x7f11780b2270 2024-06-29T14:41:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.331+0000 7f11757fa640 1 -- 172.21.15.59:0/1216078706 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c003a40 con 0x7f11780b2270 2024-06-29T14:41:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.331+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1178149600 con 0x7f11780b2270 2024-06-29T14:41:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.332+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11780b6aa0 con 0x7f11780b2270 2024-06-29T14:41:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.332+0000 7f11757fa640 1 -- 172.21.15.59:0/1216078706 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 71) ==== 98056+0+0 (secure 0 0 0) 0x7f116c01a070 con 0x7f11780b2270 2024-06-29T14:41:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.333+0000 7f11757fa640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f115c076170 0x7f115c078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.333+0000 7f11757fa640 1 -- 172.21.15.59:0/1216078706 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f116c096b10 con 0x7f11780b2270 2024-06-29T14:41:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.333+0000 7f1177fff640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f115c076170 0x7f115c078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.334+0000 7f1177fff640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f115c076170 0x7f115c078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f117806fe40 tx=0x7f1164005f90 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:34.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.338+0000 7f11757fa640 1 -- 172.21.15.59:0/1216078706 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f116c060420 con 0x7f11780b2270 2024-06-29T14:41:34.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.550+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7f11780702c0 con 0x7f115c076170 2024-06-29T14:41:34.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.610+0000 7f11757fa640 1 -- 172.21.15.59:0/1216078706 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7f11780702c0 con 0x7f115c076170 2024-06-29T14:41:34.608 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1/hourly 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f115c076170 msgr2=0x7f115c078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f115c076170 0x7f115c078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f117806fe40 tx=0x7f1164005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780b2270 msgr2=0x7f11780adba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780b2270 0x7f11780adba0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f116c00c3f0 tx=0x7f116c00c8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 shutdown_connections 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f115c076170 0x7f115c078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11780d1a60 0x7f11780b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11780b2fa0 0x7f1178148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 --2- 172.21.15.59:0/1216078706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11780b2270 0x7f11780adba0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 >> 172.21.15.59:0/1216078706 conn(0x7f1178073e10 msgr2=0x7f117815bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:34.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 shutdown_connections 2024-06-29T14:41:34.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.613+0000 7f117ef7f640 1 -- 172.21.15.59:0/1216078706 wait complete. 2024-06-29T14:41:34.622 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-06-29T14:41:34.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:34.622 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-06-29T14:41:34.639 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-06-29T14:41:34.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.801+0000 7fe5671d9640 1 -- 172.21.15.59:0/3538661511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5600d23c0 msgr2=0x7fe5600b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.801+0000 7fe5671d9640 1 --2- 172.21.15.59:0/3538661511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5600d23c0 0x7fe5600b3990 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fe548009830 tx=0x7fe54802a510 comp rx=0 tx=0).stop 2024-06-29T14:41:34.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.801+0000 7fe5671d9640 1 -- 172.21.15.59:0/3538661511 shutdown_connections 2024-06-29T14:41:34.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.801+0000 7fe5671d9640 1 --2- 172.21.15.59:0/3538661511 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe56015efe0 0x7fe56015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.801+0000 7fe5671d9640 1 --2- 172.21.15.59:0/3538661511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5600d23c0 0x7fe5600b3990 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.801+0000 7fe5671d9640 1 -- 172.21.15.59:0/3538661511 >> 172.21.15.59:0/3538661511 conn(0x7fe5600d3ee0 msgr2=0x7fe5600d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:34.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.801+0000 7fe5671d9640 1 -- 172.21.15.59:0/3538661511 shutdown_connections 2024-06-29T14:41:34.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.801+0000 7fe5671d9640 1 -- 172.21.15.59:0/3538661511 wait complete. 2024-06-29T14:41:34.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.803+0000 7fe5671d9640 1 Processor -- start 2024-06-29T14:41:34.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.803+0000 7fe5671d9640 1 -- start start 2024-06-29T14:41:34.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe5671d9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5600b3ed0 0x7fe560148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe5671d9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5600d23c0 0x7fe5601e8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe5671d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56015efe0 0x7fe5601598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe5671d9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe560160280 con 0x7fe56015efe0 2024-06-29T14:41:34.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe5671d9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe560160100 con 0x7fe5600d23c0 2024-06-29T14:41:34.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe5671d9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe560160400 con 0x7fe5600b3ed0 2024-06-29T14:41:34.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56015efe0 0x7fe5601598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56015efe0 0x7fe5601598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33998/0 (socket says 172.21.15.59:33998) 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 -- 172.21.15.59:0/2059513310 learned_addr learned my addr 172.21.15.59:0/2059513310 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe564f75640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5600b3ed0 0x7fe560148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe55ffff640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5600d23c0 0x7fe5601e8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 -- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5600b3ed0 msgr2=0x7fe560148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5600b3ed0 0x7fe560148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 -- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5600d23c0 msgr2=0x7fe5601e8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5600d23c0 0x7fe5601e8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 -- 172.21.15.59:0/2059513310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56015c8b0 con 0x7fe56015efe0 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe565776640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56015efe0 0x7fe5601598e0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fe55800e990 tx=0x7fe55800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe55d7fa640 1 -- 172.21.15.59:0/2059513310 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558004d20 con 0x7fe56015efe0 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.805+0000 7fe55d7fa640 1 -- 172.21.15.59:0/2059513310 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe55800caa0 con 0x7fe56015efe0 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.806+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe56015cb90 con 0x7fe56015efe0 2024-06-29T14:41:34.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.806+0000 7fe55d7fa640 1 -- 172.21.15.59:0/2059513310 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558005020 con 0x7fe56015efe0 2024-06-29T14:41:34.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.806+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe56015d150 con 0x7fe56015efe0 2024-06-29T14:41:34.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.807+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5600b3990 con 0x7fe56015efe0 2024-06-29T14:41:34.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.807+0000 7fe55d7fa640 1 -- 172.21.15.59:0/2059513310 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 71) ==== 98056+0+0 (secure 0 0 0) 0x7fe5580051c0 con 0x7fe56015efe0 2024-06-29T14:41:34.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.808+0000 7fe55d7fa640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe5400761a0 0x7fe5400786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.808+0000 7fe55d7fa640 1 -- 172.21.15.59:0/2059513310 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7fe558097f10 con 0x7fe56015efe0 2024-06-29T14:41:34.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.808+0000 7fe564f75640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe5400761a0 0x7fe5400786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.809+0000 7fe564f75640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe5400761a0 0x7fe5400786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe548034000 tx=0x7fe548004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:34.813+0000 7fe55d7fa640 1 -- 172.21.15.59:0/2059513310 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe558016030 con 0x7fe56015efe0 2024-06-29T14:41:35.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.027+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) -- 0x7fe56015c600 con 0x7fe5400761a0 2024-06-29T14:41:35.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.090+0000 7fe55d7fa640 1 -- 172.21.15.59:0/2059513310 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fe56015c600 con 0x7fe5400761a0 2024-06-29T14:41:35.089 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1/daily 2024-06-29T14:41:35.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.092+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe5400761a0 msgr2=0x7fe5400786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.092+0000 7fe5671d9640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe5400761a0 0x7fe5400786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe548034000 tx=0x7fe548004730 comp rx=0 tx=0).stop 2024-06-29T14:41:35.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.092+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56015efe0 msgr2=0x7fe5601598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.092+0000 7fe5671d9640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56015efe0 0x7fe5601598e0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fe55800e990 tx=0x7fe55800ee50 comp rx=0 tx=0).stop 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.093+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 shutdown_connections 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.093+0000 7fe5671d9640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe5400761a0 0x7fe5400786b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.093+0000 7fe5671d9640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56015efe0 0x7fe5601598e0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.093+0000 7fe5671d9640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5600d23c0 0x7fe5601e8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.093+0000 7fe5671d9640 1 --2- 172.21.15.59:0/2059513310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5600b3ed0 0x7fe560148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.093+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 >> 172.21.15.59:0/2059513310 conn(0x7fe5600d3ee0 msgr2=0x7fe56006d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.093+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 shutdown_connections 2024-06-29T14:41:35.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.093+0000 7fe5671d9640 1 -- 172.21.15.59:0/2059513310 wait complete. 2024-06-29T14:41:35.102 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-06-29T14:41:35.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:35.103 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-06-29T14:41:35.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-06-29T14:41:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.283+0000 7fcb66246640 1 -- 172.21.15.59:0/2497236887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6015efe0 msgr2=0x7fcb6015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.283+0000 7fcb66246640 1 --2- 172.21.15.59:0/2497236887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6015efe0 0x7fcb6015f9b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcb58005fd0 tx=0x7fcb5802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.283+0000 7fcb66246640 1 -- 172.21.15.59:0/2497236887 shutdown_connections 2024-06-29T14:41:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.283+0000 7fcb66246640 1 --2- 172.21.15.59:0/2497236887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6015efe0 0x7fcb6015f9b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.283+0000 7fcb66246640 1 --2- 172.21.15.59:0/2497236887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb600d23c0 0x7fcb600b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.283+0000 7fcb66246640 1 -- 172.21.15.59:0/2497236887 >> 172.21.15.59:0/2497236887 conn(0x7fcb600d3ee0 msgr2=0x7fcb600d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.283+0000 7fcb66246640 1 -- 172.21.15.59:0/2497236887 shutdown_connections 2024-06-29T14:41:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.283+0000 7fcb66246640 1 -- 172.21.15.59:0/2497236887 wait complete. 2024-06-29T14:41:35.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.285+0000 7fcb66246640 1 Processor -- start 2024-06-29T14:41:35.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.285+0000 7fcb66246640 1 -- start start 2024-06-29T14:41:35.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.287+0000 7fcb66246640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb600b3ed0 0x7fcb601e83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.287+0000 7fcb66246640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb600d23c0 0x7fcb6015def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.287+0000 7fcb66246640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6015efe0 0x7fcb601596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb66246640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcb60160280 con 0x7fcb600d23c0 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb66246640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb60160100 con 0x7fcb6015efe0 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb66246640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcb60160400 con 0x7fcb600b3ed0 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb600d23c0 0x7fcb6015def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb600d23c0 0x7fcb6015def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34014/0 (socket says 172.21.15.59:34014) 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 -- 172.21.15.59:0/43985358 learned_addr learned my addr 172.21.15.59:0/43985358 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 -- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb600b3ed0 msgr2=0x7fcb601e83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb600b3ed0 0x7fcb601e83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 -- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6015efe0 msgr2=0x7fcb601596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6015efe0 0x7fcb601596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 -- 172.21.15.59:0/43985358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb601e8910 con 0x7fcb600d23c0 2024-06-29T14:41:35.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.288+0000 7fcb5effd640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb600d23c0 0x7fcb6015def0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c760 tx=0x7fcb5400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:35.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.289+0000 7fcb47fff640 1 -- 172.21.15.59:0/43985358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54016020 con 0x7fcb600d23c0 2024-06-29T14:41:35.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.289+0000 7fcb47fff640 1 -- 172.21.15.59:0/43985358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb5400a950 con 0x7fcb600d23c0 2024-06-29T14:41:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.289+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6015e430 con 0x7fcb600d23c0 2024-06-29T14:41:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.289+0000 7fcb47fff640 1 -- 172.21.15.59:0/43985358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5400ada0 con 0x7fcb600d23c0 2024-06-29T14:41:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.289+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb601e9100 con 0x7fcb600d23c0 2024-06-29T14:41:35.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.291+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb600b3990 con 0x7fcb600d23c0 2024-06-29T14:41:35.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.291+0000 7fcb47fff640 1 -- 172.21.15.59:0/43985358 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7fcb54019020 con 0x7fcb600d23c0 2024-06-29T14:41:35.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.292+0000 7fcb47fff640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fcb4c076440 0x7fcb4c078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.292+0000 7fcb47fff640 1 -- 172.21.15.59:0/43985358 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7fcb54097e00 con 0x7fcb600d23c0 2024-06-29T14:41:35.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.292+0000 7fcb5f7fe640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fcb4c076440 0x7fcb4c078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.292+0000 7fcb5f7fe640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fcb4c076440 0x7fcb4c078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb50005d00 tx=0x7fcb50005c50 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.297+0000 7fcb47fff640 1 -- 172.21.15.59:0/43985358 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcb54020020 con 0x7fcb600d23c0 2024-06-29T14:41:35.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.508+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) -- 0x7fcb601e8bb0 con 0x7fcb4c076440 2024-06-29T14:41:35.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.569+0000 7fcb47fff640 1 -- 172.21.15.59:0/43985358 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7fcb601e8bb0 con 0x7fcb4c076440 2024-06-29T14:41:35.567 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1/weekly 2024-06-29T14:41:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.571+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fcb4c076440 msgr2=0x7fcb4c078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.571+0000 7fcb66246640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fcb4c076440 0x7fcb4c078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb50005d00 tx=0x7fcb50005c50 comp rx=0 tx=0).stop 2024-06-29T14:41:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb600d23c0 msgr2=0x7fcb6015def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb600d23c0 0x7fcb6015def0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c760 tx=0x7fcb5400cc20 comp rx=0 tx=0).stop 2024-06-29T14:41:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 shutdown_connections 2024-06-29T14:41:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fcb4c076440 0x7fcb4c078950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb6015efe0 0x7fcb601596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb600d23c0 0x7fcb6015def0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 --2- 172.21.15.59:0/43985358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcb600b3ed0 0x7fcb601e83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 >> 172.21.15.59:0/43985358 conn(0x7fcb600d3ee0 msgr2=0x7fcb6006d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 shutdown_connections 2024-06-29T14:41:35.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.572+0000 7fcb66246640 1 -- 172.21.15.59:0/43985358 wait complete. 2024-06-29T14:41:35.580 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-06-29T14:41:35.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:35.580 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-06-29T14:41:35.601 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-06-29T14:41:35.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.763+0000 7f955e39e640 1 -- 172.21.15.59:0/2199184590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 msgr2=0x7f9544003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.763+0000 7f955e39e640 1 --2- 172.21.15.59:0/2199184590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 0x7f9544003ee0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f95580b1fc0 tx=0x7f9544005830 comp rx=0 tx=0).stop 2024-06-29T14:41:35.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.763+0000 7f955c93b640 1 -- 172.21.15.59:0/2199184590 reap_dead start 2024-06-29T14:41:35.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.763+0000 7f955e39e640 1 -- 172.21.15.59:0/2199184590 shutdown_connections 2024-06-29T14:41:35.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.763+0000 7f955e39e640 1 -- 172.21.15.59:0/2199184590 >> 172.21.15.59:0/2199184590 conn(0x7f9558073e10 msgr2=0x7f9558074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:35.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.763+0000 7f955e39e640 1 -- 172.21.15.59:0/2199184590 shutdown_connections 2024-06-29T14:41:35.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.763+0000 7f955e39e640 1 -- 172.21.15.59:0/2199184590 wait complete. 2024-06-29T14:41:35.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.765+0000 7f955e39e640 1 Processor -- start 2024-06-29T14:41:35.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.765+0000 7f955e39e640 1 -- start start 2024-06-29T14:41:35.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f955e39e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 0x7f95581f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f955e39e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95581f4510 0x7f955806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f955e39e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f955806f520 0x7f95580af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f955e39e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f95580b5ec0 con 0x7f9544003a70 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f955e39e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95580b5d40 con 0x7f955806f520 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f955e39e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f95580b6040 con 0x7f95581f4510 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f9557fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 0x7f95581f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f9557fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 0x7f95581f7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34038/0 (socket says 172.21.15.59:34038) 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f9557fff640 1 -- 172.21.15.59:0/2442223259 learned_addr learned my addr 172.21.15.59:0/2442223259 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.766+0000 7f955c93b640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f955806f520 0x7f95580af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f954ffff640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95581f4510 0x7f955806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f9557fff640 1 -- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95581f4510 msgr2=0x7f955806efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f9557fff640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95581f4510 0x7f955806efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f9557fff640 1 -- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f955806f520 msgr2=0x7f95580af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f9557fff640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f955806f520 0x7f95580af2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f9557fff640 1 -- 172.21.15.59:0/2442223259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955806df40 con 0x7f9544003a70 2024-06-29T14:41:35.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f954ffff640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95581f4510 0x7f955806efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:35.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f9557fff640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 0x7f95581f7520 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f954800c3a0 tx=0x7f9548006c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:35.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f95557fa640 1 -- 172.21.15.59:0/2442223259 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548031040 con 0x7f9544003a70 2024-06-29T14:41:35.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f95557fa640 1 -- 172.21.15.59:0/2442223259 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9548003610 con 0x7f9544003a70 2024-06-29T14:41:35.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955806e220 con 0x7f9544003a70 2024-06-29T14:41:35.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f95557fa640 1 -- 172.21.15.59:0/2442223259 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548003a40 con 0x7f9544003a70 2024-06-29T14:41:35.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.767+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f955806e7e0 con 0x7f9544003a70 2024-06-29T14:41:35.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.768+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95580d1520 con 0x7f9544003a70 2024-06-29T14:41:35.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.769+0000 7f95557fa640 1 -- 172.21.15.59:0/2442223259 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7f9548034070 con 0x7f9544003a70 2024-06-29T14:41:35.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.769+0000 7f95557fa640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f9538076470 0x7f9538078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.769+0000 7f95557fa640 1 -- 172.21.15.59:0/2442223259 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f95480b7c80 con 0x7f9544003a70 2024-06-29T14:41:35.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.769+0000 7f954ffff640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f9538076470 0x7f9538078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.770+0000 7f954ffff640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f9538076470 0x7f9538078980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95580712a0 tx=0x7f9540005f90 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:35.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.774+0000 7f95557fa640 1 -- 172.21.15.59:0/2442223259 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9548081110 con 0x7f9544003a70 2024-06-29T14:41:35.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:35.986+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f9558075b40 con 0x7f9538076470 2024-06-29T14:41:36.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.046+0000 7f95557fa640 1 -- 172.21.15.59:0/2442223259 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+45 (secure 0 0 0) 0x7f9558075b40 con 0x7f9538076470 2024-06-29T14:41:36.044 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1/monthly 2024-06-29T14:41:36.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f9538076470 msgr2=0x7f9538078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f9538076470 0x7f9538078980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95580712a0 tx=0x7f9540005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 msgr2=0x7f95581f7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 0x7f95581f7520 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f954800c3a0 tx=0x7f9548006c50 comp rx=0 tx=0).stop 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 shutdown_connections 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f9538076470 0x7f9538078980 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f955806f520 0x7f95580af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f95581f4510 0x7f955806efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 --2- 172.21.15.59:0/2442223259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9544003a70 0x7f95581f7520 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 >> 172.21.15.59:0/2442223259 conn(0x7f9558073e10 msgr2=0x7f9558155570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 shutdown_connections 2024-06-29T14:41:36.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.048+0000 7f955e39e640 1 -- 172.21.15.59:0/2442223259 wait complete. 2024-06-29T14:41:36.057 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-06-29T14:41:36.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:36.057 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-06-29T14:41:36.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1y 2024-06-29T14:41:36.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.239+0000 7fab9ed7b640 1 -- 172.21.15.59:0/200495244 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab9808c190 msgr2=0x7fab980cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.239+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/200495244 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab9808c190 0x7fab980cf080 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fab88005fd0 tx=0x7fab8802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.239+0000 7fab9ed7b640 1 -- 172.21.15.59:0/200495244 shutdown_connections 2024-06-29T14:41:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.239+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/200495244 >> v2:172.21.15.153:3300/0 conn(0x7fab980ce950 0x7fab980d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.239+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/200495244 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab9808c190 0x7fab980cf080 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.239+0000 7fab9ed7b640 1 -- 172.21.15.59:0/200495244 >> 172.21.15.59:0/200495244 conn(0x7fab980b5610 msgr2=0x7fab980b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.239+0000 7fab9ed7b640 1 -- 172.21.15.59:0/200495244 shutdown_connections 2024-06-29T14:41:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.239+0000 7fab9ed7b640 1 -- 172.21.15.59:0/200495244 wait complete. 2024-06-29T14:41:36.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.241+0000 7fab9ed7b640 1 Processor -- start 2024-06-29T14:41:36.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.241+0000 7fab9ed7b640 1 -- start start 2024-06-29T14:41:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9ed7b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab9808c190 0x7fab981f5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9ed7b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab980ce950 0x7fab980cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9ed7b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab980cf5c0 0x7fab980ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9ed7b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fab980b7fa0 con 0x7fab9808c190 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9ed7b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fab980b7e20 con 0x7fab980cf5c0 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9ed7b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fab980b8120 con 0x7fab980ce950 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9cb17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab9808c190 0x7fab981f5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9cb17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab9808c190 0x7fab981f5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34044/0 (socket says 172.21.15.59:34044) 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab9cb17640 1 -- 172.21.15.59:0/4144602980 learned_addr learned my addr 172.21.15.59:0/4144602980 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.242+0000 7fab97fff640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab980ce950 0x7fab980cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9d318640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab980cf5c0 0x7fab980ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9cb17640 1 -- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab980ce950 msgr2=0x7fab980cbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9cb17640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab980ce950 0x7fab980cbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9cb17640 1 -- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab980cf5c0 msgr2=0x7fab980ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9cb17640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab980cf5c0 0x7fab980ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9cb17640 1 -- 172.21.15.59:0/4144602980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab980cc8c0 con 0x7fab9808c190 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab97fff640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab980ce950 0x7fab980cbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9d318640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab980cf5c0 0x7fab980ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9cb17640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab9808c190 0x7fab981f5e10 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fab88034040 tx=0x7fab880035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab957fa640 1 -- 172.21.15.59:0/4144602980 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8803e030 con 0x7fab9808c190 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab957fa640 1 -- 172.21.15.59:0/4144602980 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab88003ae0 con 0x7fab9808c190 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab957fa640 1 -- 172.21.15.59:0/4144602980 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88035040 con 0x7fab9808c190 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab980ccba0 con 0x7fab9808c190 2024-06-29T14:41:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.243+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab980cd160 con 0x7fab9808c190 2024-06-29T14:41:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.244+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab980cf080 con 0x7fab9808c190 2024-06-29T14:41:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.245+0000 7fab957fa640 1 -- 172.21.15.59:0/4144602980 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7fab88041030 con 0x7fab9808c190 2024-06-29T14:41:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.246+0000 7fab957fa640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fab84076440 0x7fab84078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.246+0000 7fab957fa640 1 -- 172.21.15.59:0/4144602980 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7fab880b7f80 con 0x7fab9808c190 2024-06-29T14:41:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.246+0000 7fab97fff640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fab84076440 0x7fab84078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.246+0000 7fab97fff640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fab84076440 0x7fab84078950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab980c8be0 tx=0x7fab8c009040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.250+0000 7fab957fa640 1 -- 172.21.15.59:0/4144602980 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fab88080e00 con 0x7fab9808c190 2024-06-29T14:41:36.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.470+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1y", "target": ["mon-mgr", ""]}) -- 0x7fab981f6350 con 0x7fab84076440 2024-06-29T14:41:36.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.528+0000 7fab957fa640 1 -- 172.21.15.59:0/4144602980 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7fab981f6350 con 0x7fab84076440 2024-06-29T14:41:36.526 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1/yearly 2024-06-29T14:41:36.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fab84076440 msgr2=0x7fab84078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fab84076440 0x7fab84078950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab980c8be0 tx=0x7fab8c009040 comp rx=0 tx=0).stop 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab9808c190 msgr2=0x7fab981f5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab9808c190 0x7fab981f5e10 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fab88034040 tx=0x7fab880035f0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 shutdown_connections 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fab84076440 0x7fab84078950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab980cf5c0 0x7fab980ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab980ce950 0x7fab980cbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 --2- 172.21.15.59:0/4144602980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab9808c190 0x7fab981f5e10 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 >> 172.21.15.59:0/4144602980 conn(0x7fab980b5610 msgr2=0x7fab98160dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 shutdown_connections 2024-06-29T14:41:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.531+0000 7fab9ed7b640 1 -- 172.21.15.59:0/4144602980 wait complete. 2024-06-29T14:41:36.540 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-06-29T14:41:36.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:36.540 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-06-29T14:41:36.557 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-06-29T14:41:36.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.722+0000 7f5512b11640 1 -- 172.21.15.59:0/2530649462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c162190 msgr2=0x7f550c0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.722+0000 7f5512b11640 1 --2- 172.21.15.59:0/2530649462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c162190 0x7f550c0b38e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f550c08c200 tx=0x7f550402aa30 comp rx=0 tx=0).stop 2024-06-29T14:41:36.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.722+0000 7f5512b11640 1 -- 172.21.15.59:0/2530649462 shutdown_connections 2024-06-29T14:41:36.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.722+0000 7f5512b11640 1 --2- 172.21.15.59:0/2530649462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c162190 0x7f550c0b38e0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.722+0000 7f5512b11640 1 --2- 172.21.15.59:0/2530649462 >> v2:172.21.15.153:3300/0 conn(0x7f550c162770 0x7f550c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.722+0000 7f5512b11640 1 -- 172.21.15.59:0/2530649462 >> 172.21.15.59:0/2530649462 conn(0x7f550c0d45b0 msgr2=0x7f550c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.722+0000 7f5512b11640 1 -- 172.21.15.59:0/2530649462 shutdown_connections 2024-06-29T14:41:36.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.722+0000 7f5512b11640 1 -- 172.21.15.59:0/2530649462 wait complete. 2024-06-29T14:41:36.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.724+0000 7f5512b11640 1 Processor -- start 2024-06-29T14:41:36.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.724+0000 7f5512b11640 1 -- start start 2024-06-29T14:41:36.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f5512b11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c15e0a0 0x7f550c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f5512b11640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f550c162770 0x7f550c0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f5512b11640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c1f7f30 0x7f550c1f8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f5512b11640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f550c0b4270 con 0x7f550c15e0a0 2024-06-29T14:41:36.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f5512b11640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f550c0b40f0 con 0x7f550c1f7f30 2024-06-29T14:41:36.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f5512b11640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f550c0b43f0 con 0x7f550c162770 2024-06-29T14:41:36.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f55108ad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c15e0a0 0x7f550c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f55108ad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c15e0a0 0x7f550c157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34056/0 (socket says 172.21.15.59:34056) 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f55108ad640 1 -- 172.21.15.59:0/1686932067 learned_addr learned my addr 172.21.15.59:0/1686932067 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f550bfff640 1 --2- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f550c162770 0x7f550c0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.725+0000 7f55110ae640 1 --2- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c1f7f30 0x7f550c1f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55108ad640 1 -- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f550c162770 msgr2=0x7f550c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55108ad640 1 --2- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f550c162770 0x7f550c0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55108ad640 1 -- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c1f7f30 msgr2=0x7f550c1f8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55108ad640 1 --2- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f550c1f7f30 0x7f550c1f8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55108ad640 1 -- 172.21.15.59:0/1686932067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550c1f8880 con 0x7f550c15e0a0 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55108ad640 1 --2- 172.21.15.59:0/1686932067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f550c15e0a0 0x7f550c157ba0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00d780 tx=0x7f54fc00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55097fa640 1 -- 172.21.15.59:0/1686932067 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc017070 con 0x7f550c15e0a0 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55097fa640 1 -- 172.21.15.59:0/1686932067 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54fc00de90 con 0x7f550c15e0a0 2024-06-29T14:41:36.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f5512b11640 1 -- 172.21.15.59:0/1686932067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550c1580e0 con 0x7f550c15e0a0 2024-06-29T14:41:36.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f55097fa640 1 -- 172.21.15.59:0/1686932067 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc0099d0 con 0x7f550c15e0a0 2024-06-29T14:41:36.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.726+0000 7f5512b11640 1 -- 172.21.15.59:0/1686932067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f550c0c7720 con 0x7f550c15e0a0 2024-06-29T14:41:36.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.727+0000 7f5512b11640 1 -- 172.21.15.59:0/1686932067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f550c0b7aa0 con 0x7f550c15e0a0 2024-06-29T14:41:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.728+0000 7f55097fa640 1 -- 172.21.15.59:0/1686932067 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7f54fc009b70 con 0x7f550c15e0a0 2024-06-29T14:41:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.728+0000 7f55097fa640 1 --2- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f54f8076470 0x7f54f8078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.728+0000 7f55097fa640 1 -- 172.21.15.59:0/1686932067 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f54fc098060 con 0x7f550c15e0a0 2024-06-29T14:41:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.729+0000 7f550bfff640 1 --2- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f54f8076470 0x7f54f8078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.729+0000 7f550bfff640 1 --2- 172.21.15.59:0/1686932067 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f54f8076470 0x7f54f8078980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f550c156ca0 tx=0x7f5500009040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.733+0000 7f55097fa640 1 -- 172.21.15.59:0/1686932067 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f550c0b7aa0 con 0x7f550c15e0a0 2024-06-29T14:41:36.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.946+0000 7f5512b11640 1 -- 172.21.15.59:0/1686932067 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) -- 0x7f550c0b4910 con 0x7f54f8076470 2024-06-29T14:41:36.944 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:36.947+0000 7f371280b640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,y 2024-06-29T14:41:36.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:36.948+0000 7f55097fa640 1 -- 172.21.15.59:0/1686932067 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,y) ==== 74+0+0 (secure 0 0 0) 0x7f550c0b4910 con 0x7f54f8076470 2024-06-29T14:41:36.945 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,y 2024-06-29T14:41:36.950 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:36.951 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-06-29T14:41:37.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.074+0000 7fe22fa10640 1 -- 172.21.15.59:0/3531759410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2280ad6b0 msgr2=0x7fe2281658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.074+0000 7fe22fa10640 1 --2- 172.21.15.59:0/3531759410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2280ad6b0 0x7fe2281658b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fe220009830 tx=0x7fe22002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.074+0000 7fe22fa10640 1 -- 172.21.15.59:0/3531759410 shutdown_connections 2024-06-29T14:41:37.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.074+0000 7fe22fa10640 1 --2- 172.21.15.59:0/3531759410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2280ad6b0 0x7fe2281658b0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.074+0000 7fe22fa10640 1 -- 172.21.15.59:0/3531759410 >> 172.21.15.59:0/3531759410 conn(0x7fe2280b1ea0 msgr2=0x7fe2280b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:37.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.074+0000 7fe22fa10640 1 -- 172.21.15.59:0/3531759410 shutdown_connections 2024-06-29T14:41:37.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.074+0000 7fe22fa10640 1 -- 172.21.15.59:0/3531759410 wait complete. 2024-06-29T14:41:37.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.076+0000 7fe22fa10640 1 Processor -- start 2024-06-29T14:41:37.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.076+0000 7fe22fa10640 1 -- start start 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22fa10640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2280cb490 0x7fe228155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22fa10640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2280cc060 0x7fe22809d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22fa10640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2280b4010 0x7fe2280b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22fa10640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281663f0 con 0x7fe2280b4010 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22fa10640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe228166270 con 0x7fe2280cb490 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22fa10640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe228166570 con 0x7fe2280cc060 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22dfad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2280b4010 0x7fe2280b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22dfad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2280b4010 0x7fe2280b4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34078/0 (socket says 172.21.15.59:34078) 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22dfad640 1 -- 172.21.15.59:0/1854454948 learned_addr learned my addr 172.21.15.59:0/1854454948 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22d7ac640 1 --2- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2280cb490 0x7fe228155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.077+0000 7fe22cfab640 1 --2- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2280cc060 0x7fe22809d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe22dfad640 1 -- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2280cc060 msgr2=0x7fe22809d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe22dfad640 1 --2- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2280cc060 0x7fe22809d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe22dfad640 1 -- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2280cb490 msgr2=0x7fe228155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe22dfad640 1 --2- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2280cb490 0x7fe228155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe22dfad640 1 -- 172.21.15.59:0/1854454948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22806c500 con 0x7fe2280b4010 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe22dfad640 1 --2- 172.21.15.59:0/1854454948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2280b4010 0x7fe2280b4420 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fe220009920 tx=0x7fe220033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe225ffb640 1 -- 172.21.15.59:0/1854454948 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220031d60 con 0x7fe2280b4010 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe225ffb640 1 -- 172.21.15.59:0/1854454948 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe220033e10 con 0x7fe2280b4010 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe22fa10640 1 -- 172.21.15.59:0/1854454948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe22809c940 con 0x7fe2280b4010 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe225ffb640 1 -- 172.21.15.59:0/1854454948 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220005260 con 0x7fe2280b4010 2024-06-29T14:41:37.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.078+0000 7fe22fa10640 1 -- 172.21.15.59:0/1854454948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2280b5070 con 0x7fe2280b4010 2024-06-29T14:41:37.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.079+0000 7fe22fa10640 1 -- 172.21.15.59:0/1854454948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe228073be0 con 0x7fe2280b4010 2024-06-29T14:41:37.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.080+0000 7fe225ffb640 1 -- 172.21.15.59:0/1854454948 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7fe220005400 con 0x7fe2280b4010 2024-06-29T14:41:37.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.081+0000 7fe225ffb640 1 --2- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe214076440 0x7fe214078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.081+0000 7fe225ffb640 1 -- 172.21.15.59:0/1854454948 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7fe2200b8ff0 con 0x7fe2280b4010 2024-06-29T14:41:37.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.081+0000 7fe22d7ac640 1 --2- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe214076440 0x7fe214078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.081+0000 7fe22d7ac640 1 --2- 172.21.15.59:0/1854454948 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe214076440 0x7fe214078950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe2180012e0 tx=0x7fe218009040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.085+0000 7fe225ffb640 1 -- 172.21.15.59:0/1854454948 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe22003f030 con 0x7fe2280b4010 2024-06-29T14:41:37.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.298+0000 7fe22fa10640 1 -- 172.21.15.59:0/1854454948 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) -- 0x7fe2280cc780 con 0x7fe214076440 2024-06-29T14:41:37.297 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:37.299+0000 7f371280b640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:37.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.300+0000 7fe225ffb640 1 -- 172.21.15.59:0/1854454948 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7fe2280cc780 con 0x7fe214076440 2024-06-29T14:41:37.297 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:37.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.301+0000 7fe225ffb640 1 -- 172.21.15.59:0/1854454948 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7fe22007df50 con 0x7fe2280b4010 2024-06-29T14:41:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:37.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-06-29T14:41:37.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.429+0000 7ff89ed03640 1 -- 172.21.15.59:0/1957702844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff888003ac0 msgr2=0x7ff888003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.429+0000 7ff89ed03640 1 --2- 172.21.15.59:0/1957702844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff888003ac0 0x7ff888003f30 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00c2e0 tx=0x7ff888005830 comp rx=0 tx=0).stop 2024-06-29T14:41:37.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.429+0000 7ff89ed03640 1 -- 172.21.15.59:0/1957702844 shutdown_connections 2024-06-29T14:41:37.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.429+0000 7ff89ed03640 1 --2- 172.21.15.59:0/1957702844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff888003ac0 0x7ff888003f30 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.429+0000 7ff89ed03640 1 -- 172.21.15.59:0/1957702844 >> 172.21.15.59:0/1957702844 conn(0x7ff8980d3ee0 msgr2=0x7ff8980d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:37.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.429+0000 7ff89ed03640 1 -- 172.21.15.59:0/1957702844 shutdown_connections 2024-06-29T14:41:37.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.429+0000 7ff89ed03640 1 -- 172.21.15.59:0/1957702844 wait complete. 2024-06-29T14:41:37.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.431+0000 7ff89ed03640 1 Processor -- start 2024-06-29T14:41:37.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.431+0000 7ff89ed03640 1 -- start start 2024-06-29T14:41:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff89ed03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff888003ac0 0x7ff89814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff89ed03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff89814f730 0x7ff8980c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff89ed03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8980c9d00 0x7ff898154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff89ed03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff89815ec60 con 0x7ff89814f730 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff89ed03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff89815eae0 con 0x7ff888003ac0 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff89ed03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff8980b3990 con 0x7ff8980c9d00 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff897fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff89814f730 0x7ff8980c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff89ca9f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff888003ac0 0x7ff89814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff897fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff89814f730 0x7ff8980c97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34092/0 (socket says 172.21.15.59:34092) 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff89d2a0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8980c9d00 0x7ff898154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff897fff640 1 -- 172.21.15.59:0/4061094149 learned_addr learned my addr 172.21.15.59:0/4061094149 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff897fff640 1 -- 172.21.15.59:0/4061094149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8980c9d00 msgr2=0x7ff898154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.433+0000 7ff897fff640 1 --2- 172.21.15.59:0/4061094149 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8980c9d00 0x7ff898154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff897fff640 1 -- 172.21.15.59:0/4061094149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff888003ac0 msgr2=0x7ff89814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff897fff640 1 --2- 172.21.15.59:0/4061094149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff888003ac0 0x7ff89814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff897fff640 1 -- 172.21.15.59:0/4061094149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8980ca110 con 0x7ff89814f730 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff897fff640 1 --2- 172.21.15.59:0/4061094149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff89814f730 0x7ff8980c97c0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7ff884010aa0 tx=0x7ff884010f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff8957fa640 1 -- 172.21.15.59:0/4061094149 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88400ec20 con 0x7ff89814f730 2024-06-29T14:41:37.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff8957fa640 1 -- 172.21.15.59:0/4061094149 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff88400edc0 con 0x7ff89814f730 2024-06-29T14:41:37.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff89ed03640 1 -- 172.21.15.59:0/4061094149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8981563e0 con 0x7ff89814f730 2024-06-29T14:41:37.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff8957fa640 1 -- 172.21.15.59:0/4061094149 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884005260 con 0x7ff89814f730 2024-06-29T14:41:37.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.434+0000 7ff89ed03640 1 -- 172.21.15.59:0/4061094149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8980d0c50 con 0x7ff89814f730 2024-06-29T14:41:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.435+0000 7ff89ed03640 1 -- 172.21.15.59:0/4061094149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8980b3ef0 con 0x7ff89814f730 2024-06-29T14:41:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.436+0000 7ff8957fa640 1 -- 172.21.15.59:0/4061094149 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7ff884005400 con 0x7ff89814f730 2024-06-29T14:41:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.437+0000 7ff8957fa640 1 --2- 172.21.15.59:0/4061094149 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7ff87c0764d0 0x7ff87c0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.437+0000 7ff8957fa640 1 -- 172.21.15.59:0/4061094149 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7ff884098a20 con 0x7ff89814f730 2024-06-29T14:41:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.437+0000 7ff89ca9f640 1 --2- 172.21.15.59:0/4061094149 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7ff87c0764d0 0x7ff87c0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.438+0000 7ff89ca9f640 1 --2- 172.21.15.59:0/4061094149 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7ff87c0764d0 0x7ff87c0789e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff88c033000 tx=0x7ff88c038040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.441+0000 7ff8957fa640 1 -- 172.21.15.59:0/4061094149 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff8980ca110 con 0x7ff89814f730 2024-06-29T14:41:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.656+0000 7ff89ed03640 1 -- 172.21.15.59:0/4061094149 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) -- 0x7ff898156190 con 0x7ff87c0764d0 2024-06-29T14:41:37.654 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:37.657+0000 7f371280b640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:37.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.657+0000 7ff8957fa640 1 -- 172.21.15.59:0/4061094149 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7ff898156190 con 0x7ff87c0764d0 2024-06-29T14:41:37.655 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:37.660 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-06-29T14:41:37.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.785+0000 7faa53bfc640 1 -- 172.21.15.59:0/3686349616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa34004800 msgr2=0x7faa34004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.785+0000 7faa53bfc640 1 --2- 172.21.15.59:0/3686349616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa34004800 0x7faa34004c10 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7faa4c162220 tx=0x7faa34005830 comp rx=0 tx=0).stop 2024-06-29T14:41:37.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.785+0000 7faa53bfc640 1 -- 172.21.15.59:0/3686349616 shutdown_connections 2024-06-29T14:41:37.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.785+0000 7faa53bfc640 1 --2- 172.21.15.59:0/3686349616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa34004800 0x7faa34004c10 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.785+0000 7faa53bfc640 1 -- 172.21.15.59:0/3686349616 >> 172.21.15.59:0/3686349616 conn(0x7faa4c0b5610 msgr2=0x7faa4c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:37.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.785+0000 7faa53bfc640 1 -- 172.21.15.59:0/3686349616 shutdown_connections 2024-06-29T14:41:37.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.785+0000 7faa53bfc640 1 -- 172.21.15.59:0/3686349616 wait complete. 2024-06-29T14:41:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.787+0000 7faa53bfc640 1 Processor -- start 2024-06-29T14:41:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.787+0000 7faa53bfc640 1 -- start start 2024-06-29T14:41:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa53bfc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa34004800 0x7faa4c0cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa53bfc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c1548c0 0x7faa4c1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa53bfc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c155e10 0x7faa4c15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa53bfc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0cf680 con 0x7faa34004800 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa53bfc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0cf500 con 0x7faa4c1548c0 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa53bfc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c0cf800 con 0x7faa4c155e10 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51998640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa34004800 0x7faa4c0cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51998640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa34004800 0x7faa4c0cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34110/0 (socket says 172.21.15.59:34110) 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51998640 1 -- 172.21.15.59:0/2276810013 learned_addr learned my addr 172.21.15.59:0/2276810013 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51197640 1 --2- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c1548c0 0x7faa4c1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa52199640 1 --2- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c155e10 0x7faa4c15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51998640 1 -- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c155e10 msgr2=0x7faa4c15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51998640 1 --2- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c155e10 0x7faa4c15a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51998640 1 -- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c1548c0 msgr2=0x7faa4c1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51998640 1 --2- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c1548c0 0x7faa4c1558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.788+0000 7faa51998640 1 -- 172.21.15.59:0/2276810013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c0cc120 con 0x7faa34004800 2024-06-29T14:41:37.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.789+0000 7faa51998640 1 --2- 172.21.15.59:0/2276810013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa34004800 0x7faa4c0cd1c0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7faa3c004d90 tx=0x7faa3c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.789+0000 7faa4a7fc640 1 -- 172.21.15.59:0/2276810013 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c017070 con 0x7faa34004800 2024-06-29T14:41:37.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.789+0000 7faa4a7fc640 1 -- 172.21.15.59:0/2276810013 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa3c0108c0 con 0x7faa34004800 2024-06-29T14:41:37.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.789+0000 7faa53bfc640 1 -- 172.21.15.59:0/2276810013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c0cc400 con 0x7faa34004800 2024-06-29T14:41:37.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.789+0000 7faa4a7fc640 1 -- 172.21.15.59:0/2276810013 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c001d40 con 0x7faa34004800 2024-06-29T14:41:37.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.789+0000 7faa53bfc640 1 -- 172.21.15.59:0/2276810013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c0cc9c0 con 0x7faa34004800 2024-06-29T14:41:37.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.790+0000 7faa53bfc640 1 -- 172.21.15.59:0/2276810013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c162650 con 0x7faa34004800 2024-06-29T14:41:37.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.791+0000 7faa4a7fc640 1 -- 172.21.15.59:0/2276810013 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7faa3c01a070 con 0x7faa34004800 2024-06-29T14:41:37.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.791+0000 7faa4a7fc640 1 --2- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7faa380764d0 0x7faa380789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.791+0000 7faa4a7fc640 1 -- 172.21.15.59:0/2276810013 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7faa3c097a20 con 0x7faa34004800 2024-06-29T14:41:37.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.791+0000 7faa51197640 1 --2- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7faa380764d0 0x7faa380789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.792+0000 7faa51197640 1 --2- 172.21.15.59:0/2276810013 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7faa380764d0 0x7faa380789e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa40033000 tx=0x7faa40000fe0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:37.796+0000 7faa4a7fc640 1 -- 172.21.15.59:0/2276810013 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faa3c060750 con 0x7faa34004800 2024-06-29T14:41:38.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.011+0000 7faa53bfc640 1 -- 172.21.15.59:0/2276810013 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) -- 0x7faa4c0b4d80 con 0x7faa380764d0 2024-06-29T14:41:38.009 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:38.011+0000 7f371280b640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:38.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.012+0000 7faa4a7fc640 1 -- 172.21.15.59:0/2276810013 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7faa4c0b4d80 con 0x7faa380764d0 2024-06-29T14:41:38.010 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:38.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-06-29T14:41:38.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.139+0000 7f88ffdcf640 1 -- 172.21.15.59:0/3636054114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e4005350 msgr2=0x7f88e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.139+0000 7f88ffdcf640 1 --2- 172.21.15.59:0/3636054114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e4005350 0x7f88e400b9c0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f88f0001dc0 tx=0x7f88e4005830 comp rx=0 tx=0).stop 2024-06-29T14:41:38.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.139+0000 7f88ffdcf640 1 -- 172.21.15.59:0/3636054114 shutdown_connections 2024-06-29T14:41:38.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.139+0000 7f88ffdcf640 1 --2- 172.21.15.59:0/3636054114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e4005350 0x7f88e400b9c0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.139+0000 7f88ffdcf640 1 -- 172.21.15.59:0/3636054114 >> 172.21.15.59:0/3636054114 conn(0x7f88f8073e10 msgr2=0x7f88f8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:38.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.139+0000 7f88ffdcf640 1 -- 172.21.15.59:0/3636054114 shutdown_connections 2024-06-29T14:41:38.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.139+0000 7f88ffdcf640 1 -- 172.21.15.59:0/3636054114 wait complete. 2024-06-29T14:41:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.141+0000 7f88ffdcf640 1 Processor -- start 2024-06-29T14:41:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.141+0000 7f88ffdcf640 1 -- start start 2024-06-29T14:41:38.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.142+0000 7f88ffdcf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e4005350 0x7f88f8159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.142+0000 7f88ffdcf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f80aa0e0 0x7f88f80aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.142+0000 7f88ffdcf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88f80b5840 0x7f88f80b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.142+0000 7f88ffdcf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f88f80d1960 con 0x7f88e4005350 2024-06-29T14:41:38.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.142+0000 7f88ffdcf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88f80d17e0 con 0x7f88f80aa0e0 2024-06-29T14:41:38.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.142+0000 7f88ffdcf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f88f815e590 con 0x7f88f80b5840 2024-06-29T14:41:38.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.142+0000 7f88fdb6b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e4005350 0x7f88f8159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fdb6b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e4005350 0x7f88f8159f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34122/0 (socket says 172.21.15.59:34122) 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fdb6b640 1 -- 172.21.15.59:0/2181601872 learned_addr learned my addr 172.21.15.59:0/2181601872 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fd36a640 1 --2- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f80aa0e0 0x7f88f80aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fe36c640 1 --2- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88f80b5840 0x7f88f80b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fdb6b640 1 -- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88f80b5840 msgr2=0x7f88f80b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fdb6b640 1 --2- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88f80b5840 0x7f88f80b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fdb6b640 1 -- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f80aa0e0 msgr2=0x7f88f80aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fdb6b640 1 --2- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f80aa0e0 0x7f88f80aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fdb6b640 1 -- 172.21.15.59:0/2181601872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f80b6190 con 0x7f88e4005350 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88fdb6b640 1 --2- 172.21.15.59:0/2181601872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e4005350 0x7f88f8159f50 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f88f4011ab0 tx=0x7f88f4011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88ee7fc640 1 -- 172.21.15.59:0/2181601872 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f401e070 con 0x7f88e4005350 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88ee7fc640 1 -- 172.21.15.59:0/2181601872 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88f4004460 con 0x7f88e4005350 2024-06-29T14:41:38.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88ffdcf640 1 -- 172.21.15.59:0/2181601872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f80cd370 con 0x7f88e4005350 2024-06-29T14:41:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88ee7fc640 1 -- 172.21.15.59:0/2181601872 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f400fc60 con 0x7f88e4005350 2024-06-29T14:41:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.143+0000 7f88ffdcf640 1 -- 172.21.15.59:0/2181601872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f80cd930 con 0x7f88e4005350 2024-06-29T14:41:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.144+0000 7f88ffdcf640 1 -- 172.21.15.59:0/2181601872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f8161aa0 con 0x7f88e4005350 2024-06-29T14:41:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.145+0000 7f88ee7fc640 1 -- 172.21.15.59:0/2181601872 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7f88f4021070 con 0x7f88e4005350 2024-06-29T14:41:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.146+0000 7f88ee7fc640 1 --2- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f88dc0764d0 0x7f88dc0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.146+0000 7f88ee7fc640 1 -- 172.21.15.59:0/2181601872 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f88f40988a0 con 0x7f88e4005350 2024-06-29T14:41:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.146+0000 7f88fd36a640 1 --2- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f88dc0764d0 0x7f88dc0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.147+0000 7f88fd36a640 1 --2- 172.21.15.59:0/2181601872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f88dc0764d0 0x7f88dc0789e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88f80b6af0 tx=0x7f88e80067a0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.150+0000 7f88ee7fc640 1 -- 172.21.15.59:0/2181601872 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f88f80b6190 con 0x7f88e4005350 2024-06-29T14:41:38.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.362+0000 7f88ffdcf640 1 -- 172.21.15.59:0/2181601872 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) -- 0x7f88f8071790 con 0x7f88dc0764d0 2024-06-29T14:41:38.360 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:38.363+0000 7f371280b640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-06-29T14:41:38.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.364+0000 7f88ee7fc640 1 -- 172.21.15.59:0/2181601872 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) ==== 96+0+0 (secure 0 0 0) 0x7f88f8071790 con 0x7f88dc0764d0 2024-06-29T14:41:38.361 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-06-29T14:41:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:38.367 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-06-29T14:41:38.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.491+0000 7fd6b9481640 1 -- 172.21.15.59:0/3135866807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a0004d40 msgr2=0x7fd6a0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.491+0000 7fd6b9481640 1 --2- 172.21.15.59:0/3135866807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6a0004d40 0x7fd6a0005150 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009ed0 tx=0x7fd6a0005830 comp rx=0 tx=0).stop 2024-06-29T14:41:38.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.491+0000 7fd6b37fe640 1 -- 172.21.15.59:0/3135866807 reap_dead start 2024-06-29T14:41:38.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.492+0000 7fd6b9481640 1 -- 172.21.15.59:0/3135866807 shutdown_connections 2024-06-29T14:41:38.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.492+0000 7fd6b9481640 1 -- 172.21.15.59:0/3135866807 >> 172.21.15.59:0/3135866807 conn(0x7fd6b40b6fb0 msgr2=0x7fd6b40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:38.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.492+0000 7fd6b9481640 1 -- 172.21.15.59:0/3135866807 shutdown_connections 2024-06-29T14:41:38.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.492+0000 7fd6b9481640 1 -- 172.21.15.59:0/3135866807 wait complete. 2024-06-29T14:41:38.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.493+0000 7fd6b9481640 1 Processor -- start 2024-06-29T14:41:38.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.494+0000 7fd6b9481640 1 -- start start 2024-06-29T14:41:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b9481640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a0004d40 0x7fd6b4159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b9481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b40c7a50 0x7fd6b40c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b9481640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b41f2580 0x7fd6b41f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b9481640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b40b2530 con 0x7fd6b40c7a50 2024-06-29T14:41:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b9481640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b413f0b0 con 0x7fd6b41f2580 2024-06-29T14:41:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b9481640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b40b26b0 con 0x7fd6a0004d40 2024-06-29T14:41:38.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b40c7a50 0x7fd6b40c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b40c7a50 0x7fd6b40c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34136/0 (socket says 172.21.15.59:34136) 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 -- 172.21.15.59:0/3238173104 learned_addr learned my addr 172.21.15.59:0/3238173104 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b2ffd640 1 --2- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a0004d40 0x7fd6b4159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b37fe640 1 --2- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b41f2580 0x7fd6b41f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 -- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a0004d40 msgr2=0x7fd6b4159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 --2- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6a0004d40 0x7fd6b4159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 -- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b41f2580 msgr2=0x7fd6b41f2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 --2- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b41f2580 0x7fd6b41f2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 -- 172.21.15.59:0/3238173104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b40c8610 con 0x7fd6b40c7a50 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.495+0000 7fd6b27fc640 1 --2- 172.21.15.59:0/3238173104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b40c7a50 0x7fd6b40c7e60 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd69c011ae0 tx=0x7fd69c009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.496+0000 7fd6ab7fe640 1 -- 172.21.15.59:0/3238173104 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c01e070 con 0x7fd6b40c7a50 2024-06-29T14:41:38.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.496+0000 7fd6ab7fe640 1 -- 172.21.15.59:0/3238173104 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd69c004460 con 0x7fd6b40c7a50 2024-06-29T14:41:38.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.496+0000 7fd6b9481640 1 -- 172.21.15.59:0/3238173104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b40c88f0 con 0x7fd6b40c7a50 2024-06-29T14:41:38.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.496+0000 7fd6ab7fe640 1 -- 172.21.15.59:0/3238173104 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c00fc60 con 0x7fd6b40c7a50 2024-06-29T14:41:38.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.496+0000 7fd6b9481640 1 -- 172.21.15.59:0/3238173104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6b40c8eb0 con 0x7fd6b40c7a50 2024-06-29T14:41:38.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.497+0000 7fd6b9481640 1 -- 172.21.15.59:0/3238173104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b40710f0 con 0x7fd6b40c7a50 2024-06-29T14:41:38.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.497+0000 7fd6ab7fe640 1 -- 172.21.15.59:0/3238173104 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7fd69c021070 con 0x7fd6b40c7a50 2024-06-29T14:41:38.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.498+0000 7fd6ab7fe640 1 --2- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fd6940764d0 0x7fd6940789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.498+0000 7fd6ab7fe640 1 -- 172.21.15.59:0/3238173104 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7fd69c0988a0 con 0x7fd6b40c7a50 2024-06-29T14:41:38.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.498+0000 7fd6b2ffd640 1 --2- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fd6940764d0 0x7fd6940789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.499+0000 7fd6b2ffd640 1 --2- 172.21.15.59:0/3238173104 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fd6940764d0 0x7fd6940789e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac001410 tx=0x7fd6ac006910 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.502+0000 7fd6ab7fe640 1 -- 172.21.15.59:0/3238173104 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd69c061520 con 0x7fd6b40c7a50 2024-06-29T14:41:38.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.718+0000 7fd6b9481640 1 -- 172.21.15.59:0/3238173104 --> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) -- 0x7fd6b41f3490 con 0x7fd6940764d0 2024-06-29T14:41:38.717 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:38.719+0000 7f371280b640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:38.720+0000 7fd6ab7fe640 1 -- 172.21.15.59:0/3238173104 <== mgr.8419 v2:172.21.15.153:6832/2556013753 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7fd6ac0065b0 con 0x7fd6940764d0 2024-06-29T14:41:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:38.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.723 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-06-29T14:41:38.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.740 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-06-29T14:41:38.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.799 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-06-29T14:41:38.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.856 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-06-29T14:41:38.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.913 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-06-29T14:41:38.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.970 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-06-29T14:41:39.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:39.027 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-06-29T14:41:39.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:41:39.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.248+0000 7f0f9b511640 1 -- 172.21.15.59:0/2159203289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f9408c190 msgr2=0x7f0f940cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:39.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.248+0000 7f0f9b511640 1 --2- 172.21.15.59:0/2159203289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f9408c190 0x7f0f940cf080 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f0f90005fd0 tx=0x7f0f9002a5a0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.248+0000 7f0f9b511640 1 -- 172.21.15.59:0/2159203289 shutdown_connections 2024-06-29T14:41:39.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.248+0000 7f0f9b511640 1 --2- 172.21.15.59:0/2159203289 >> v2:172.21.15.153:3300/0 conn(0x7f0f940ce950 0x7f0f940d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.248+0000 7f0f9b511640 1 --2- 172.21.15.59:0/2159203289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f9408c190 0x7f0f940cf080 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.248+0000 7f0f9b511640 1 -- 172.21.15.59:0/2159203289 >> 172.21.15.59:0/2159203289 conn(0x7f0f940b5610 msgr2=0x7f0f940b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:39.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.248+0000 7f0f9b511640 1 -- 172.21.15.59:0/2159203289 shutdown_connections 2024-06-29T14:41:39.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.248+0000 7f0f9b511640 1 -- 172.21.15.59:0/2159203289 wait complete. 2024-06-29T14:41:39.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.250+0000 7f0f9b511640 1 Processor -- start 2024-06-29T14:41:39.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.250+0000 7f0f9b511640 1 -- start start 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.251+0000 7f0f9b511640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f9408c190 0x7f0f941f50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.251+0000 7f0f9b511640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f940ce950 0x7f0f941f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.251+0000 7f0f9b511640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f940cf5c0 0x7f0f94157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.251+0000 7f0f9b511640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0f940b7f60 con 0x7f0f940ce950 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.251+0000 7f0f9b511640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f940b7de0 con 0x7f0f940cf5c0 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.251+0000 7f0f9b511640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0f940b80e0 con 0x7f0f9408c190 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f940ce950 0x7f0f941f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f940ce950 0x7f0f941f5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34154/0 (socket says 172.21.15.59:34154) 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 -- 172.21.15.59:0/1489275443 learned_addr learned my addr 172.21.15.59:0/1489275443 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f99aae640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f940cf5c0 0x7f0f94157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f992ad640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f9408c190 0x7f0f941f50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:39.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 -- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f9408c190 msgr2=0x7f0f941f50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f9408c190 0x7f0f941f50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 -- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f940cf5c0 msgr2=0x7f0f94157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f940cf5c0 0x7f0f94157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 -- 172.21.15.59:0/1489275443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f94156490 con 0x7f0f940ce950 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f98aac640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f940ce950 0x7f0f941f5bc0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400c370 tx=0x7f0f8400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f89ffb640 1 -- 172.21.15.59:0/1489275443 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f84017070 con 0x7f0f940ce950 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f89ffb640 1 -- 172.21.15.59:0/1489275443 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f840053c0 con 0x7f0f940ce950 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f94156770 con 0x7f0f940ce950 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.252+0000 7f0f89ffb640 1 -- 172.21.15.59:0/1489275443 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f84003850 con 0x7f0f940ce950 2024-06-29T14:41:39.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.253+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f94156d30 con 0x7f0f940ce950 2024-06-29T14:41:39.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.254+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f940cf080 con 0x7f0f940ce950 2024-06-29T14:41:39.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.254+0000 7f0f89ffb640 1 -- 172.21.15.59:0/1489275443 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7f0f8401a070 con 0x7f0f940ce950 2024-06-29T14:41:39.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.255+0000 7f0f89ffb640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f78076500 0x7f0f78078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:39.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.255+0000 7f0f89ffb640 1 -- 172.21.15.59:0/1489275443 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f0f84097200 con 0x7f0f940ce950 2024-06-29T14:41:39.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.255+0000 7f0f992ad640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f78076500 0x7f0f78078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:39.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.256+0000 7f0f992ad640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f78076500 0x7f0f78078a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0f90033000 tx=0x7f0f90004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:39.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.260+0000 7f0f89ffb640 1 -- 172.21.15.59:0/1489275443 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f84060560 con 0x7f0f940ce950 2024-06-29T14:41:39.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:39.505+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f0f9415c970 con 0x7f0f940ce950 2024-06-29T14:41:40.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.311+0000 7f0f89ffb640 1 -- 172.21.15.59:0/1489275443 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v74) ==== 93+0+0 (secure 0 0 0) 0x7f0f8405fe60 con 0x7f0f940ce950 2024-06-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.315+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f78076500 msgr2=0x7f0f78078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.315+0000 7f0f9b511640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f78076500 0x7f0f78078a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0f90033000 tx=0x7f0f90004730 comp rx=0 tx=0).stop 2024-06-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.316+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f940ce950 msgr2=0x7f0f941f5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.316+0000 7f0f9b511640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f940ce950 0x7f0f941f5bc0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400c370 tx=0x7f0f8400c830 comp rx=0 tx=0).stop 2024-06-29T14:41:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.316+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 shutdown_connections 2024-06-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.316+0000 7f0f9b511640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0f78076500 0x7f0f78078a10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.316+0000 7f0f9b511640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f940cf5c0 0x7f0f94157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.316+0000 7f0f9b511640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f940ce950 0x7f0f941f5bc0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.316+0000 7f0f9b511640 1 --2- 172.21.15.59:0/1489275443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f9408c190 0x7f0f941f50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.317+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 >> 172.21.15.59:0/1489275443 conn(0x7f0f940b5610 msgr2=0x7f0f941552a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.317+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 shutdown_connections 2024-06-29T14:41:40.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.317+0000 7f0f9b511640 1 -- 172.21.15.59:0/1489275443 wait complete. 2024-06-29T14:41:40.329 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:41:40.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:40.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:40.436 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:41:40.436 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:40.437 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:40.437 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:40.453 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:40.453 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:40.509 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.671+0000 7f46961bd640 1 -- 172.21.15.59:0/14052534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46900b40a0 msgr2=0x7f469015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.671+0000 7f46961bd640 1 --2- 172.21.15.59:0/14052534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46900b40a0 0x7f469015e8d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4684009830 tx=0x7f468402a540 comp rx=0 tx=0).stop 2024-06-29T14:41:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.672+0000 7f46961bd640 1 -- 172.21.15.59:0/14052534 shutdown_connections 2024-06-29T14:41:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.672+0000 7f46961bd640 1 --2- 172.21.15.59:0/14052534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46900b40a0 0x7f469015e8d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.672+0000 7f46961bd640 1 -- 172.21.15.59:0/14052534 >> 172.21.15.59:0/14052534 conn(0x7f46900d3ee0 msgr2=0x7f46900d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.672+0000 7f46961bd640 1 -- 172.21.15.59:0/14052534 shutdown_connections 2024-06-29T14:41:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.672+0000 7f46961bd640 1 -- 172.21.15.59:0/14052534 wait complete. 2024-06-29T14:41:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.673+0000 7f46961bd640 1 Processor -- start 2024-06-29T14:41:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.674+0000 7f46961bd640 1 -- start start 2024-06-29T14:41:40.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f46961bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46900d23c0 0x7f469014fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:40.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f46961bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f469015ee10 0x7f46901f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:40.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f46961bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4690157370 0x7f4690157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:40.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f46961bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4690160450 con 0x7f4690157370 2024-06-29T14:41:40.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f46961bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46901602d0 con 0x7f469015ee10 2024-06-29T14:41:40.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f46961bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46901605d0 con 0x7f46900d23c0 2024-06-29T14:41:40.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4690157370 0x7f4690157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4690157370 0x7f4690157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34180/0 (socket says 172.21.15.59:34180) 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468ffff640 1 -- 172.21.15.59:0/3704669872 learned_addr learned my addr 172.21.15.59:0/3704669872 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468effd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f469015ee10 0x7f46901f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468f7fe640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46900d23c0 0x7f469014fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468effd640 1 -- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46900d23c0 msgr2=0x7f469014fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468effd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46900d23c0 0x7f469014fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468effd640 1 -- 172.21.15.59:0/3704669872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4690157370 msgr2=0x7f4690157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468effd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4690157370 0x7f4690157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.675+0000 7f468effd640 1 -- 172.21.15.59:0/3704669872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4690157cc0 con 0x7f469015ee10 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.676+0000 7f468effd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f469015ee10 0x7f46901f5cd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4684031000 tx=0x7f4684003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.676+0000 7f4677fff640 1 -- 172.21.15.59:0/3704669872 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684005c90 con 0x7f469015ee10 2024-06-29T14:41:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.676+0000 7f4677fff640 1 -- 172.21.15.59:0/3704669872 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4684005110 con 0x7f469015ee10 2024-06-29T14:41:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.676+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469015aac0 con 0x7f469015ee10 2024-06-29T14:41:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.676+0000 7f4677fff640 1 -- 172.21.15.59:0/3704669872 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684005540 con 0x7f469015ee10 2024-06-29T14:41:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.676+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469015ada0 con 0x7f469015ee10 2024-06-29T14:41:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.677+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4690069de0 con 0x7f469015ee10 2024-06-29T14:41:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.678+0000 7f4677fff640 1 -- 172.21.15.59:0/3704669872 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f46840317c0 con 0x7f469015ee10 2024-06-29T14:41:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.678+0000 7f4677fff640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f467c076480 0x7f467c078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.678+0000 7f468f7fe640 1 -- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f467c076480 msgr2=0x7f467c078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.678+0000 7f4677fff640 1 -- 172.21.15.59:0/3704669872 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f46840b8e20 con 0x7f469015ee10 2024-06-29T14:41:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.678+0000 7f468f7fe640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f467c076480 0x7f467c078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.683+0000 7f4677fff640 1 -- 172.21.15.59:0/3704669872 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4690069de0 con 0x7f469015ee10 2024-06-29T14:41:40.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.876+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f469015e8d0 con 0x7f469015ee10 2024-06-29T14:41:40.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.877+0000 7f4677fff640 1 -- 172.21.15.59:0/3704669872 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+17514 (secure 0 0 0) 0x7f4690157cc0 con 0x7f469015ee10 2024-06-29T14:41:40.874 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:40.874 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":144,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:33.044625+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.171960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:19.744696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3137107702":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1492442042":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/2519658617":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1609759314":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6833/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6832/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1585009476":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/706205231":"2024-06-30T14:41:33.044584+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:40.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f468f7fe640 1 -- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f467c076480 msgr2=0x7f467c078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:40.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f468f7fe640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f467c076480 0x7f467c078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f467c076480 msgr2=0x7f467c078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f467c076480 0x7f467c078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f469015ee10 msgr2=0x7f46901f5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f469015ee10 0x7f46901f5cd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4684031000 tx=0x7f4684003c10 comp rx=0 tx=0).stop 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 shutdown_connections 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f467c076480 0x7f467c078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4690157370 0x7f4690157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f469015ee10 0x7f46901f5cd0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 --2- 172.21.15.59:0/3704669872 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46900d23c0 0x7f469014fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 >> 172.21.15.59:0/3704669872 conn(0x7f46900d3ee0 msgr2=0x7f46900b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 shutdown_connections 2024-06-29T14:41:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:40.879+0000 7f46961bd640 1 -- 172.21.15.59:0/3704669872 wait complete. 2024-06-29T14:41:40.888 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:41:19.171960+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:41:19.744696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:40.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:41.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.013+0000 7fe012d0a640 1 -- 172.21.15.59:0/1736219600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c0b3ed0 msgr2=0x7fe00c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.013+0000 7fe012d0a640 1 --2- 172.21.15.59:0/1736219600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c0b3ed0 0x7fe00c15e8d0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fdff8005fd0 tx=0x7fdff802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.013+0000 7fe012d0a640 1 -- 172.21.15.59:0/1736219600 shutdown_connections 2024-06-29T14:41:41.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.013+0000 7fe012d0a640 1 --2- 172.21.15.59:0/1736219600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c0b3ed0 0x7fe00c15e8d0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.013+0000 7fe012d0a640 1 --2- 172.21.15.59:0/1736219600 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe00c0d23c0 0x7fe00c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.013+0000 7fe012d0a640 1 -- 172.21.15.59:0/1736219600 >> 172.21.15.59:0/1736219600 conn(0x7fe00c0d3ee0 msgr2=0x7fe00c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.013+0000 7fe012d0a640 1 -- 172.21.15.59:0/1736219600 shutdown_connections 2024-06-29T14:41:41.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.013+0000 7fe012d0a640 1 -- 172.21.15.59:0/1736219600 wait complete. 2024-06-29T14:41:41.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.015+0000 7fe012d0a640 1 Processor -- start 2024-06-29T14:41:41.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.015+0000 7fe012d0a640 1 -- start start 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.016+0000 7fe012d0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe00c0b3ed0 0x7fe00c15a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.016+0000 7fe012d0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe00c0d23c0 0x7fe00c15b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.016+0000 7fe012d0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c15ee10 0x7fe00c15b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.016+0000 7fe012d0a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe00c160240 con 0x7fe00c15ee10 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.016+0000 7fe012d0a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe00c1600c0 con 0x7fe00c0d23c0 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.016+0000 7fe012d0a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe00c1603c0 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe0112a7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c15ee10 0x7fe00c15b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe0112a7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c15ee10 0x7fe00c15b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34198/0 (socket says 172.21.15.59:34198) 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe0112a7640 1 -- 172.21.15.59:0/3143488551 learned_addr learned my addr 172.21.15.59:0/3143488551 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:41.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe010aa6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe00c0b3ed0 0x7fe00c15a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe00bfff640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe00c0d23c0 0x7fe00c15b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe010aa6640 1 -- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe00c0d23c0 msgr2=0x7fe00c15b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe010aa6640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe00c0d23c0 0x7fe00c15b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe010aa6640 1 -- 172.21.15.59:0/3143488551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c15ee10 msgr2=0x7fe00c15b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe010aa6640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c15ee10 0x7fe00c15b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe010aa6640 1 -- 172.21.15.59:0/3143488551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00c1e93c0 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe0112a7640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c15ee10 0x7fe00c15b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe010aa6640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe00c0b3ed0 0x7fe00c15a680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe00000e6f0 tx=0x7fe00000ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe0097fa640 1 -- 172.21.15.59:0/3143488551 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe000004230 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe0097fa640 1 -- 172.21.15.59:0/3143488551 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe000004410 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe00c1e96a0 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.017+0000 7fe0097fa640 1 -- 172.21.15.59:0/3143488551 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00000cb10 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.018+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe00c1e9c60 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.018+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe00c15e8d0 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.019+0000 7fe0097fa640 1 -- 172.21.15.59:0/3143488551 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7fe00000ccb0 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.020+0000 7fe0097fa640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fdff0076480 0x7fdff0078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.020+0000 7fe0097fa640 1 -- 172.21.15.59:0/3143488551 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7fe0000986b0 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.020+0000 7fe00bfff640 1 -- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fdff0076480 msgr2=0x7fdff0078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:41.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.020+0000 7fe00bfff640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fdff0076480 0x7fdff0078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:41.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.024+0000 7fe0097fa640 1 -- 172.21.15.59:0/3143488551 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe000061280 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.216+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe00c0d2950 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.216+0000 7fe0097fa640 1 -- 172.21.15.59:0/3143488551 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) ==== 93+0+10 (secure 0 0 0) 0x7fe000065580 con 0x7fe00c0b3ed0 2024-06-29T14:41:41.214 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:41:41.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fdff0076480 msgr2=0x7fdff0078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:41.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fdff0076480 0x7fdff0078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe00c0b3ed0 msgr2=0x7fe00c15a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe00c0b3ed0 0x7fe00c15a680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe00000e6f0 tx=0x7fe00000ebb0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 shutdown_connections 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fdff0076480 0x7fdff0078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe00c15ee10 0x7fe00c15b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe00c0d23c0 0x7fe00c15b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 --2- 172.21.15.59:0/3143488551 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe00c0b3ed0 0x7fe00c15a680 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 >> 172.21.15.59:0/3143488551 conn(0x7fe00c0d3ee0 msgr2=0x7fe00c1576e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 shutdown_connections 2024-06-29T14:41:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.219+0000 7fe012d0a640 1 -- 172.21.15.59:0/3143488551 wait complete. 2024-06-29T14:41:41.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:41:41.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.354+0000 7f69d24ce640 1 -- 172.21.15.59:0/492191847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69cc0d23c0 msgr2=0x7f69cc0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.354+0000 7f69d24ce640 1 --2- 172.21.15.59:0/492191847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69cc0d23c0 0x7f69cc0b3990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009830 tx=0x7f69bc02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.354+0000 7f69d24ce640 1 -- 172.21.15.59:0/492191847 shutdown_connections 2024-06-29T14:41:41.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.354+0000 7f69d24ce640 1 --2- 172.21.15.59:0/492191847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69cc0d23c0 0x7f69cc0b3990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.354+0000 7f69d24ce640 1 -- 172.21.15.59:0/492191847 >> 172.21.15.59:0/492191847 conn(0x7f69cc0d3ee0 msgr2=0x7f69cc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.354+0000 7f69d24ce640 1 -- 172.21.15.59:0/492191847 shutdown_connections 2024-06-29T14:41:41.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.355+0000 7f69d24ce640 1 -- 172.21.15.59:0/492191847 wait complete. 2024-06-29T14:41:41.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.356+0000 7f69d24ce640 1 Processor -- start 2024-06-29T14:41:41.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.357+0000 7f69d24ce640 1 -- start start 2024-06-29T14:41:41.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69d24ce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69cc0b3ed0 0x7f69cc151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69d24ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69cc0d23c0 0x7f69cc14a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69d24ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69cc15efe0 0x7f69cc1522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69d24ce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f69cc160450 con 0x7f69cc0d23c0 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69d24ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f69cc1602d0 con 0x7f69cc15efe0 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69d24ce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f69cc1605d0 con 0x7f69cc0b3ed0 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69cc0d23c0 0x7f69cc14a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69cc0d23c0 0x7f69cc14a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34200/0 (socket says 172.21.15.59:34200) 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cb7fe640 1 -- 172.21.15.59:0/668585340 learned_addr learned my addr 172.21.15.59:0/668585340 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69d0a6b640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69cc15efe0 0x7f69cc1522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cbfff640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69cc0b3ed0 0x7f69cc151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cb7fe640 1 -- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69cc0b3ed0 msgr2=0x7f69cc151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cb7fe640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69cc0b3ed0 0x7f69cc151190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cb7fe640 1 -- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69cc15efe0 msgr2=0x7f69cc1522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cb7fe640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69cc15efe0 0x7f69cc1522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.358+0000 7f69cb7fe640 1 -- 172.21.15.59:0/668585340 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69cc0c6030 con 0x7f69cc0d23c0 2024-06-29T14:41:41.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.359+0000 7f69cb7fe640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69cc0d23c0 0x7f69cc14a250 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f69b400c760 tx=0x7f69b400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:41.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.359+0000 7f69c8ff9640 1 -- 172.21.15.59:0/668585340 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4006a90 con 0x7f69cc0d23c0 2024-06-29T14:41:41.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.359+0000 7f69c8ff9640 1 -- 172.21.15.59:0/668585340 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69b4017070 con 0x7f69cc0d23c0 2024-06-29T14:41:41.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.359+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69cc151b00 con 0x7f69cc0d23c0 2024-06-29T14:41:41.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.359+0000 7f69c8ff9640 1 -- 172.21.15.59:0/668585340 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b400ce10 con 0x7f69cc0d23c0 2024-06-29T14:41:41.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.359+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69cc0c65b0 con 0x7f69cc0d23c0 2024-06-29T14:41:41.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.360+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69cc0b7c60 con 0x7f69cc0d23c0 2024-06-29T14:41:41.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.361+0000 7f69c8ff9640 1 -- 172.21.15.59:0/668585340 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f69b4007030 con 0x7f69cc0d23c0 2024-06-29T14:41:41.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.361+0000 7f69c8ff9640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f69ac0764b0 0x7f69ac0789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.361+0000 7f69cbfff640 1 -- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f69ac0764b0 msgr2=0x7f69ac0789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:41.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.361+0000 7f69c8ff9640 1 -- 172.21.15.59:0/668585340 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f69b4097a70 con 0x7f69cc0d23c0 2024-06-29T14:41:41.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.361+0000 7f69cbfff640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f69ac0764b0 0x7f69ac0789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:41.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.366+0000 7f69c8ff9640 1 -- 172.21.15.59:0/668585340 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f69cc0b7c60 con 0x7f69cc0d23c0 2024-06-29T14:41:41.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.558+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69cc151850 con 0x7f69cc0d23c0 2024-06-29T14:41:41.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.558+0000 7f69c8ff9640 1 -- 172.21.15.59:0/668585340 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) ==== 104+0+11 (secure 0 0 0) 0x7f69cc0b42e0 con 0x7f69cc0d23c0 2024-06-29T14:41:41.555 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f69ac0764b0 msgr2=0x7f69ac0789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f69ac0764b0 0x7f69ac0789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69cc0d23c0 msgr2=0x7f69cc14a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69cc0d23c0 0x7f69cc14a250 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f69b400c760 tx=0x7f69b400cc20 comp rx=0 tx=0).stop 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 shutdown_connections 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f69ac0764b0 0x7f69ac0789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69cc15efe0 0x7f69cc1522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69cc0d23c0 0x7f69cc14a250 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 --2- 172.21.15.59:0/668585340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69cc0b3ed0 0x7f69cc151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 >> 172.21.15.59:0/668585340 conn(0x7f69cc0d3ee0 msgr2=0x7f69cc0b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.560+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 shutdown_connections 2024-06-29T14:41:41.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.561+0000 7f69d24ce640 1 -- 172.21.15.59:0/668585340 wait complete. 2024-06-29T14:41:41.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:41:41.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.694+0000 7fe686417640 1 -- 172.21.15.59:0/2300691578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800d1c30 msgr2=0x7fe6800b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.694+0000 7fe686417640 1 --2- 172.21.15.59:0/2300691578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800d1c30 0x7fe6800b5b30 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fe678005fd0 tx=0x7fe67802aa90 comp rx=0 tx=0).stop 2024-06-29T14:41:41.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.694+0000 7fe686417640 1 -- 172.21.15.59:0/2300691578 shutdown_connections 2024-06-29T14:41:41.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.694+0000 7fe686417640 1 --2- 172.21.15.59:0/2300691578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800d1c30 0x7fe6800b5b30 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.694+0000 7fe686417640 1 --2- 172.21.15.59:0/2300691578 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6800b2270 0x7fe68015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.694+0000 7fe686417640 1 -- 172.21.15.59:0/2300691578 >> 172.21.15.59:0/2300691578 conn(0x7fe680073e10 msgr2=0x7fe680074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.694+0000 7fe686417640 1 -- 172.21.15.59:0/2300691578 shutdown_connections 2024-06-29T14:41:41.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.694+0000 7fe686417640 1 -- 172.21.15.59:0/2300691578 wait complete. 2024-06-29T14:41:41.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.695+0000 7fe686417640 1 Processor -- start 2024-06-29T14:41:41.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.696+0000 7fe686417640 1 -- start start 2024-06-29T14:41:41.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe686417640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800b2270 0x7fe6801f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe686417640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6800b2fa0 0x7fe6800ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe686417640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6800d1c30 0x7fe6800adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe686417640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe6800b6400 con 0x7fe6800b2270 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe686417640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6800b6280 con 0x7fe6800b2fa0 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe686417640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe6800b6580 con 0x7fe6800d1c30 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe67ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800b2270 0x7fe6801f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe67ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800b2270 0x7fe6801f5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34224/0 (socket says 172.21.15.59:34224) 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe67ffff640 1 -- 172.21.15.59:0/1095294399 learned_addr learned my addr 172.21.15.59:0/1095294399 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe6849b4640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6800d1c30 0x7fe6800adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.697+0000 7fe67f7fe640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6800b2fa0 0x7fe6800ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67ffff640 1 -- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6800d1c30 msgr2=0x7fe6800adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67ffff640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6800d1c30 0x7fe6800adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67ffff640 1 -- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6800b2fa0 msgr2=0x7fe6800ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67ffff640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6800b2fa0 0x7fe6800ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67ffff640 1 -- 172.21.15.59:0/1095294399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68015d4f0 con 0x7fe6800b2270 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67ffff640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800b2270 0x7fe6801f5170 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fe67000e9f0 tx=0x7fe67000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67cff9640 1 -- 172.21.15.59:0/1095294399 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67000cd20 con 0x7fe6800b2270 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67cff9640 1 -- 172.21.15.59:0/1095294399 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe670004020 con 0x7fe6800b2270 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6800ae7c0 con 0x7fe6800b2270 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe67cff9640 1 -- 172.21.15.59:0/1095294399 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670004470 con 0x7fe6800b2270 2024-06-29T14:41:41.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.698+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68006de20 con 0x7fe6800b2270 2024-06-29T14:41:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.699+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6800b6ac0 con 0x7fe6800b2270 2024-06-29T14:41:41.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.700+0000 7fe67cff9640 1 -- 172.21.15.59:0/1095294399 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7fe6700035f0 con 0x7fe6800b2270 2024-06-29T14:41:41.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.700+0000 7fe67cff9640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe66c0764b0 0x7fe66c0789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.701+0000 7fe67f7fe640 1 -- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe66c0764b0 msgr2=0x7fe66c0789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:41.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.701+0000 7fe67cff9640 1 -- 172.21.15.59:0/1095294399 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7fe670098810 con 0x7fe6800b2270 2024-06-29T14:41:41.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.701+0000 7fe67f7fe640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe66c0764b0 0x7fe66c0789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:41.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.709+0000 7fe67cff9640 1 -- 172.21.15.59:0/1095294399 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe670061360 con 0x7fe6800b2270 2024-06-29T14:41:41.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.901+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe680075b40 con 0x7fe6800b2270 2024-06-29T14:41:41.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.901+0000 7fe67cff9640 1 -- 172.21.15.59:0/1095294399 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v144) ==== 100+0+11 (secure 0 0 0) 0x7fe670065660 con 0x7fe6800b2270 2024-06-29T14:41:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:41:41.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.902+0000 7fe67f7fe640 1 -- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe66c0764b0 msgr2=0x7fe66c0789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:41.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.902+0000 7fe67f7fe640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe66c0764b0 0x7fe66c0789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:41.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.903+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe66c0764b0 msgr2=0x7fe66c0789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:41.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.903+0000 7fe686417640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe66c0764b0 0x7fe66c0789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.903+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800b2270 msgr2=0x7fe6801f5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.903+0000 7fe686417640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800b2270 0x7fe6801f5170 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fe67000e9f0 tx=0x7fe67000eeb0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.904+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 shutdown_connections 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.904+0000 7fe686417640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7fe66c0764b0 0x7fe66c0789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.904+0000 7fe686417640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe6800d1c30 0x7fe6800adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.904+0000 7fe686417640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6800b2fa0 0x7fe6800ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.904+0000 7fe686417640 1 --2- 172.21.15.59:0/1095294399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6800b2270 0x7fe6801f5170 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.904+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 >> 172.21.15.59:0/1095294399 conn(0x7fe680073e10 msgr2=0x7fe680155520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.904+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 shutdown_connections 2024-06-29T14:41:41.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:41.904+0000 7fe686417640 1 -- 172.21.15.59:0/1095294399 wait complete. 2024-06-29T14:41:41.912 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:41:42.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.034+0000 7f0d1b54b640 1 -- 172.21.15.59:0/17127424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1406ea10 msgr2=0x7f0d140cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.035+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/17127424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1406ea10 0x7f0d140cef90 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c005fd0 tx=0x7f0d0c02a820 comp rx=0 tx=0).stop 2024-06-29T14:41:42.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.035+0000 7f0d1b54b640 1 -- 172.21.15.59:0/17127424 shutdown_connections 2024-06-29T14:41:42.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.035+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/17127424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d1406ea10 0x7f0d140cef90 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.035+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/17127424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d14143840 0x7f0d1406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.035+0000 7f0d1b54b640 1 -- 172.21.15.59:0/17127424 >> 172.21.15.59:0/17127424 conn(0x7f0d140b2ab0 msgr2=0x7f0d140b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.035+0000 7f0d1b54b640 1 -- 172.21.15.59:0/17127424 shutdown_connections 2024-06-29T14:41:42.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.035+0000 7f0d1b54b640 1 -- 172.21.15.59:0/17127424 wait complete. 2024-06-29T14:41:42.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.036+0000 7f0d1b54b640 1 Processor -- start 2024-06-29T14:41:42.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.036+0000 7f0d1b54b640 1 -- start start 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d1b54b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d1406ea10 0x7f0d14154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d1b54b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d140d0130 0x7f0d14158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d1b54b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d14143840 0x7f0d140b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d1b54b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0d140cf690 con 0x7f0d140d0130 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d1b54b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d140cf510 con 0x7f0d14143840 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d1b54b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0d140cf810 con 0x7f0d1406ea10 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d18ae6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d140d0130 0x7f0d14158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d18ae6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d140d0130 0x7f0d14158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34240/0 (socket says 172.21.15.59:34240) 2024-06-29T14:41:42.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d18ae6640 1 -- 172.21.15.59:0/1513182902 learned_addr learned my addr 172.21.15.59:0/1513182902 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d192e7640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d1406ea10 0x7f0d14154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d19ae8640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d14143840 0x7f0d140b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d19ae8640 1 -- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d1406ea10 msgr2=0x7f0d14154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d19ae8640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d1406ea10 0x7f0d14154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d19ae8640 1 -- 172.21.15.59:0/1513182902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d140d0130 msgr2=0x7f0d14158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d19ae8640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d140d0130 0x7f0d14158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d19ae8640 1 -- 172.21.15.59:0/1513182902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d141f69c0 con 0x7f0d14143840 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.038+0000 7f0d18ae6640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d140d0130 0x7f0d14158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.039+0000 7f0d19ae8640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d14143840 0x7f0d140b1590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009000 tx=0x7f0d0c032c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.039+0000 7f0d05ffb640 1 -- 172.21.15.59:0/1513182902 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c037040 con 0x7f0d14143840 2024-06-29T14:41:42.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.039+0000 7f0d05ffb640 1 -- 172.21.15.59:0/1513182902 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d0c042070 con 0x7f0d14143840 2024-06-29T14:41:42.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.039+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d141f6e30 con 0x7f0d14143840 2024-06-29T14:41:42.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.039+0000 7f0d05ffb640 1 -- 172.21.15.59:0/1513182902 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c0042a0 con 0x7f0d14143840 2024-06-29T14:41:42.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.039+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d14141ab0 con 0x7f0d14143840 2024-06-29T14:41:42.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.040+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d1413d1d0 con 0x7f0d14143840 2024-06-29T14:41:42.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.041+0000 7f0d05ffb640 1 -- 172.21.15.59:0/1513182902 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f0d0c004440 con 0x7f0d14143840 2024-06-29T14:41:42.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.041+0000 7f0d05ffb640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0cf8076480 0x7f0cf8078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.041+0000 7f0d192e7640 1 -- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0cf8076480 msgr2=0x7f0cf8078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:42.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.041+0000 7f0d05ffb640 1 -- 172.21.15.59:0/1513182902 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f0d0c0b8b20 con 0x7f0d14143840 2024-06-29T14:41:42.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.041+0000 7f0d192e7640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0cf8076480 0x7f0cf8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:42.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.046+0000 7f0d05ffb640 1 -- 172.21.15.59:0/1513182902 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0d1413d1d0 con 0x7f0d14143840 2024-06-29T14:41:42.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.242+0000 7f0d192e7640 1 -- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0cf8076480 msgr2=0x7f0cf8078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:42.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.242+0000 7f0d192e7640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0cf8076480 0x7f0cf8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:42.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.249+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0d141f6b90 con 0x7f0d14143840 2024-06-29T14:41:42.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.250+0000 7f0d05ffb640 1 -- 172.21.15.59:0/1513182902 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v144) ==== 80+0+475 (secure 0 0 0) 0x7f0d141f6b90 con 0x7f0d14143840 2024-06-29T14:41:42.247 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3137107702 2024-06-30T14:41:33.044584+0000 2024-06-29T14:41:42.247 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1492442042 2024-06-30T14:41:33.044584+0000 2024-06-29T14:41:42.247 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2519658617 2024-06-30T14:41:33.044584+0000 2024-06-29T14:41:42.247 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1609759314 2024-06-30T14:41:33.044584+0000 2024-06-29T14:41:42.247 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/776633953 2024-06-30T14:41:33.044584+0000 2024-06-29T14:41:42.247 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/776633953 2024-06-30T14:41:33.044584+0000 2024-06-29T14:41:42.247 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1585009476 2024-06-30T14:41:33.044584+0000 2024-06-29T14:41:42.248 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/706205231 2024-06-30T14:41:33.044584+0000 2024-06-29T14:41:42.248 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T14:41:42.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0cf8076480 msgr2=0x7f0cf8078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:42.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0cf8076480 0x7f0cf8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d14143840 msgr2=0x7f0d140b1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d14143840 0x7f0d140b1590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009000 tx=0x7f0d0c032c90 comp rx=0 tx=0).stop 2024-06-29T14:41:42.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 shutdown_connections 2024-06-29T14:41:42.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f0cf8076480 0x7f0cf8078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d14143840 0x7f0d140b1590 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d140d0130 0x7f0d14158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 --2- 172.21.15.59:0/1513182902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d1406ea10 0x7f0d14154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 >> 172.21.15.59:0/1513182902 conn(0x7f0d140b2ab0 msgr2=0x7f0d1413cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 shutdown_connections 2024-06-29T14:41:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.252+0000 7f0d1b54b640 1 -- 172.21.15.59:0/1513182902 wait complete. 2024-06-29T14:41:42.261 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:41:42.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:42.261 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:42.292 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T14:41:42.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:42.292 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:41:42.350 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:42.350 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:41:42.350 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:42.366 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:42.366 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:42.366 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:42.366 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:42.406 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:42.406 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:42.407 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:41:42.407 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:42.423 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:42.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:42.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.569+0000 7f1e50ac5640 1 -- 172.21.15.59:0/3867105733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c0b40a0 msgr2=0x7f1e4c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.569+0000 7f1e50ac5640 1 --2- 172.21.15.59:0/3867105733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c0b40a0 0x7f1e4c15e8d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1e40009860 tx=0x7f1e4002a5a0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.570+0000 7f1e50ac5640 1 -- 172.21.15.59:0/3867105733 shutdown_connections 2024-06-29T14:41:42.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.570+0000 7f1e50ac5640 1 --2- 172.21.15.59:0/3867105733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c0b40a0 0x7f1e4c15e8d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.570+0000 7f1e50ac5640 1 -- 172.21.15.59:0/3867105733 >> 172.21.15.59:0/3867105733 conn(0x7f1e4c0d3ee0 msgr2=0x7f1e4c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.570+0000 7f1e50ac5640 1 -- 172.21.15.59:0/3867105733 shutdown_connections 2024-06-29T14:41:42.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.570+0000 7f1e50ac5640 1 -- 172.21.15.59:0/3867105733 wait complete. 2024-06-29T14:41:42.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.571+0000 7f1e50ac5640 1 Processor -- start 2024-06-29T14:41:42.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.572+0000 7f1e50ac5640 1 -- start start 2024-06-29T14:41:42.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e50ac5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e4c0d23c0 0x7f1e4c157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e50ac5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c15ee10 0x7f1e4c159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e50ac5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c157ee0 0x7f1e4c0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e50ac5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c160450 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e50ac5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c1602d0 con 0x7f1e4c157ee0 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e50ac5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1e4c1605d0 con 0x7f1e4c15ee10 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e4c0d23c0 0x7f1e4c157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e4c0d23c0 0x7f1e4c157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34256/0 (socket says 172.21.15.59:34256) 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4a59c640 1 -- 172.21.15.59:0/2697325327 learned_addr learned my addr 172.21.15.59:0/2697325327 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e49d9b640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c15ee10 0x7f1e4c159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4ad9d640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c157ee0 0x7f1e4c0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4a59c640 1 -- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c15ee10 msgr2=0x7f1e4c159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4a59c640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c15ee10 0x7f1e4c159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4a59c640 1 -- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c157ee0 msgr2=0x7f1e4c0cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4a59c640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c157ee0 0x7f1e4c0cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.573+0000 7f1e4a59c640 1 -- 172.21.15.59:0/2697325327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4c158f50 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.574+0000 7f1e4a59c640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e4c0d23c0 0x7f1e4c157240 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00c430 tx=0x7f1e3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.574+0000 7f1e3affd640 1 -- 172.21.15.59:0/2697325327 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c017070 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.574+0000 7f1e3affd640 1 -- 172.21.15.59:0/2697325327 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e3c0053c0 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.574+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4c1591c0 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.574+0000 7f1e3affd640 1 -- 172.21.15.59:0/2697325327 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c003850 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.574+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e4c14e480 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.575+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e4c15e8d0 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.576+0000 7f1e3affd640 1 -- 172.21.15.59:0/2697325327 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f1e3c01a070 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.576+0000 7f1e3affd640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f1e2c0764b0 0x7f1e2c0789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.577+0000 7f1e49d9b640 1 -- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f1e2c0764b0 msgr2=0x7f1e2c0789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:42.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.577+0000 7f1e3affd640 1 -- 172.21.15.59:0/2697325327 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f1e3c0970b0 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.577+0000 7f1e49d9b640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f1e2c0764b0 0x7f1e2c0789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:42.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.585+0000 7f1e3affd640 1 -- 172.21.15.59:0/2697325327 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1e3c05fe60 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.778+0000 7f1e49d9b640 1 -- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f1e2c0764b0 msgr2=0x7f1e2c0789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.778+0000 7f1e49d9b640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f1e2c0764b0 0x7f1e2c0789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:42.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.818+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e4c0d2950 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.818+0000 7f1e3affd640 1 -- 172.21.15.59:0/2697325327 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5228 (secure 0 0 0) 0x7f1e3c064160 con 0x7f1e4c0d23c0 2024-06-29T14:41:42.816 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:42.816 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":72,"btime":"2024-06-29T14:41:24:429207+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7442,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/4194962668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4194962668},{"type":"v1","addr":"172.21.15.153:6836","nonce":4194962668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7445,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1840307553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1840307553},{"type":"v1","addr":"172.21.15.153:6837","nonce":1840307553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1010388227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1010388227},{"type":"v1","addr":"172.21.15.59:6834","nonce":1010388227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:20.795436+0000","modified":"2024-06-29T14:41:23.821908+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8844":{"gid":8844,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/957887336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":957887336},{"type":"v1","addr":"172.21.15.59:6835","nonce":957887336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8844,"qdb_cluster":[8844]},"id":8}]} 2024-06-29T14:41:42.816 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 72 2024-06-29T14:41:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f1e2c0764b0 msgr2=0x7f1e2c0789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f1e2c0764b0 0x7f1e2c0789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e4c0d23c0 msgr2=0x7f1e4c157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e4c0d23c0 0x7f1e4c157240 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00c430 tx=0x7f1e3c00c8f0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 shutdown_connections 2024-06-29T14:41:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f1e2c0764b0 0x7f1e2c0789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e4c157ee0 0x7f1e4c0cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e4c15ee10 0x7f1e4c159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 --2- 172.21.15.59:0/2697325327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e4c0d23c0 0x7f1e4c157240 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 >> 172.21.15.59:0/2697325327 conn(0x7f1e4c0d3ee0 msgr2=0x7f1e4c154330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 shutdown_connections 2024-06-29T14:41:42.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.821+0000 7f1e50ac5640 1 -- 172.21.15.59:0/2697325327 wait complete. 2024-06-29T14:41:42.829 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:42.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.950+0000 7f01aae7f640 1 -- 172.21.15.59:0/2077348061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a415efe0 msgr2=0x7f01a415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.950+0000 7f01aae7f640 1 --2- 172.21.15.59:0/2077348061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a415efe0 0x7f01a415f9b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f019c005fd0 tx=0x7f019c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.950+0000 7f01aae7f640 1 -- 172.21.15.59:0/2077348061 shutdown_connections 2024-06-29T14:41:42.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.950+0000 7f01aae7f640 1 --2- 172.21.15.59:0/2077348061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a415efe0 0x7f01a415f9b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.950+0000 7f01aae7f640 1 --2- 172.21.15.59:0/2077348061 >> v2:172.21.15.153:3300/0 conn(0x7f01a40d23c0 0x7f01a40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.950+0000 7f01aae7f640 1 -- 172.21.15.59:0/2077348061 >> 172.21.15.59:0/2077348061 conn(0x7f01a40d3ee0 msgr2=0x7f01a40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.950+0000 7f01aae7f640 1 -- 172.21.15.59:0/2077348061 shutdown_connections 2024-06-29T14:41:42.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.950+0000 7f01aae7f640 1 -- 172.21.15.59:0/2077348061 wait complete. 2024-06-29T14:41:42.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.952+0000 7f01aae7f640 1 Processor -- start 2024-06-29T14:41:42.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.952+0000 7f01aae7f640 1 -- start start 2024-06-29T14:41:42.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01aae7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a40b3ed0 0x7f01a40b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01aae7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a40d23c0 0x7f01a415da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01aae7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a415efe0 0x7f01a415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01aae7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f01a4160280 con 0x7f01a40b3ed0 2024-06-29T14:41:42.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01aae7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01a4160100 con 0x7f01a40d23c0 2024-06-29T14:41:42.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01aae7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f01a4160400 con 0x7f01a415efe0 2024-06-29T14:41:42.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a40b3ed0 0x7f01a40b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a40b3ed0 0x7f01a40b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51634/0 (socket says 172.21.15.59:51634) 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 -- 172.21.15.59:0/927545214 learned_addr learned my addr 172.21.15.59:0/927545214 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a941c640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a415efe0 0x7f01a415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a3fff640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a40d23c0 0x7f01a415da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 -- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a415efe0 msgr2=0x7f01a415ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a415efe0 0x7f01a415ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 -- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a40d23c0 msgr2=0x7f01a415da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a40d23c0 0x7f01a415da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 -- 172.21.15.59:0/927545214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a40b30f0 con 0x7f01a40b3ed0 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.954+0000 7f01a8c1b640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a40b3ed0 0x7f01a40b2bb0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f019800e910 tx=0x7f019800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.955+0000 7f01a17fa640 1 -- 172.21.15.59:0/927545214 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019800cd80 con 0x7f01a40b3ed0 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.955+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a4159470 con 0x7f01a40b3ed0 2024-06-29T14:41:42.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.955+0000 7f01a17fa640 1 -- 172.21.15.59:0/927545214 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0198010430 con 0x7f01a40b3ed0 2024-06-29T14:41:42.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.955+0000 7f01a17fa640 1 -- 172.21.15.59:0/927545214 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198010860 con 0x7f01a40b3ed0 2024-06-29T14:41:42.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.955+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a4159a30 con 0x7f01a40b3ed0 2024-06-29T14:41:42.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.956+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a40b3990 con 0x7f01a40b3ed0 2024-06-29T14:41:42.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.957+0000 7f01a17fa640 1 -- 172.21.15.59:0/927545214 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f019801f020 con 0x7f01a40b3ed0 2024-06-29T14:41:42.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.958+0000 7f01a17fa640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f01880764b0 0x7f01880789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.958+0000 7f01a3fff640 1 -- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f01880764b0 msgr2=0x7f01880789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:42.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.958+0000 7f01a17fa640 1 -- 172.21.15.59:0/927545214 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f0198098a10 con 0x7f01a40b3ed0 2024-06-29T14:41:42.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.958+0000 7f01a3fff640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f01880764b0 0x7f01880789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:42.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:42.961+0000 7f01a17fa640 1 -- 172.21.15.59:0/927545214 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0198061720 con 0x7f01a40b3ed0 2024-06-29T14:41:43.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.154+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01a40d2950 con 0x7f01a40b3ed0 2024-06-29T14:41:43.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.155+0000 7f01a17fa640 1 -- 172.21.15.59:0/927545214 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+17514 (secure 0 0 0) 0x7f0198065a20 con 0x7f01a40b3ed0 2024-06-29T14:41:43.152 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:43.152 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":144,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:33.044625+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.171960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:19.744696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3137107702":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1492442042":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/2519658617":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1609759314":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6833/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6832/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1585009476":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/706205231":"2024-06-30T14:41:33.044584+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:43.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f01880764b0 msgr2=0x7f01880789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f01880764b0 0x7f01880789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a40b3ed0 msgr2=0x7f01a40b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a40b3ed0 0x7f01a40b2bb0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f019800e910 tx=0x7f019800edd0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 shutdown_connections 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f01880764b0 0x7f01880789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a415efe0 0x7f01a415ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a40d23c0 0x7f01a415da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 --2- 172.21.15.59:0/927545214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a40b3ed0 0x7f01a40b2bb0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 >> 172.21.15.59:0/927545214 conn(0x7f01a40d3ee0 msgr2=0x7f01a406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 shutdown_connections 2024-06-29T14:41:43.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.157+0000 7f01aae7f640 1 -- 172.21.15.59:0/927545214 wait complete. 2024-06-29T14:41:43.166 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:41:19.171960+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:41:19.744696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:43.166 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:43.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.291+0000 7f549ae8f640 1 -- 172.21.15.59:0/1704062218 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f549415efe0 msgr2=0x7f549415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.291+0000 7f549ae8f640 1 --2- 172.21.15.59:0/1704062218 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f549415efe0 0x7f549415f9b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f548c009be0 tx=0x7f548c02a920 comp rx=0 tx=0).stop 2024-06-29T14:41:43.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.291+0000 7f549ae8f640 1 -- 172.21.15.59:0/1704062218 shutdown_connections 2024-06-29T14:41:43.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.291+0000 7f549ae8f640 1 --2- 172.21.15.59:0/1704062218 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f549415efe0 0x7f549415f9b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.291+0000 7f549ae8f640 1 -- 172.21.15.59:0/1704062218 >> 172.21.15.59:0/1704062218 conn(0x7f54940d3ee0 msgr2=0x7f54940d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.291+0000 7f549ae8f640 1 -- 172.21.15.59:0/1704062218 shutdown_connections 2024-06-29T14:41:43.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.291+0000 7f549ae8f640 1 -- 172.21.15.59:0/1704062218 wait complete. 2024-06-29T14:41:43.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.292+0000 7f549ae8f640 1 Processor -- start 2024-06-29T14:41:43.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.293+0000 7f549ae8f640 1 -- start start 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f549ae8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54940d23c0 0x7f549414d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f549ae8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f549415efe0 0x7f5494152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f549ae8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5494153f70 0x7f549414da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f549ae8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5494160470 con 0x7f54940d23c0 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f549ae8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54941602f0 con 0x7f549415efe0 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f549ae8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f54941605f0 con 0x7f5494153f70 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f5498c2b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54940d23c0 0x7f549414d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f5498c2b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54940d23c0 0x7f549414d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51648/0 (socket says 172.21.15.59:51648) 2024-06-29T14:41:43.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f5498c2b640 1 -- 172.21.15.59:0/2007126897 learned_addr learned my addr 172.21.15.59:0/2007126897 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f5493fff640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f549415efe0 0x7f5494152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.294+0000 7f549942c640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5494153f70 0x7f549414da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f5493fff640 1 -- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5494153f70 msgr2=0x7f549414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f5493fff640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5494153f70 0x7f549414da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f5493fff640 1 -- 172.21.15.59:0/2007126897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54940d23c0 msgr2=0x7f549414d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f5493fff640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54940d23c0 0x7f549414d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f5493fff640 1 -- 172.21.15.59:0/2007126897 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5494151160 con 0x7f549415efe0 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f5498c2b640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54940d23c0 0x7f549414d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:43.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f549942c640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5494153f70 0x7f549414da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:43.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f5493fff640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f549415efe0 0x7f5494152200 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f54940d2cd0 tx=0x7f548000c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:43.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f54917fa640 1 -- 172.21.15.59:0/2007126897 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5480017070 con 0x7f549415efe0 2024-06-29T14:41:43.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f54917fa640 1 -- 172.21.15.59:0/2007126897 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54800053c0 con 0x7f549415efe0 2024-06-29T14:41:43.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5494151440 con 0x7f549415efe0 2024-06-29T14:41:43.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f54917fa640 1 -- 172.21.15.59:0/2007126897 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5480003850 con 0x7f549415efe0 2024-06-29T14:41:43.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.295+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5494151a00 con 0x7f549415efe0 2024-06-29T14:41:43.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.296+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549415f9b0 con 0x7f549415efe0 2024-06-29T14:41:43.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.297+0000 7f54917fa640 1 -- 172.21.15.59:0/2007126897 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f548001a070 con 0x7f549415efe0 2024-06-29T14:41:43.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.297+0000 7f54917fa640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f5478076480 0x7f5478078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.298+0000 7f5498c2b640 1 -- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f5478076480 msgr2=0x7f5478078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:43.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.298+0000 7f54917fa640 1 -- 172.21.15.59:0/2007126897 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f54800971b0 con 0x7f549415efe0 2024-06-29T14:41:43.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.298+0000 7f5498c2b640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f5478076480 0x7f5478078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.306+0000 7f54917fa640 1 -- 172.21.15.59:0/2007126897 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f548005fee0 con 0x7f549415efe0 2024-06-29T14:41:43.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.496+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f54940b52b0 con 0x7f549415efe0 2024-06-29T14:41:43.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.497+0000 7f54917fa640 1 -- 172.21.15.59:0/2007126897 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) ==== 93+0+10 (secure 0 0 0) 0x7f54800641e0 con 0x7f549415efe0 2024-06-29T14:41:43.494 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:41:43.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f5478076480 msgr2=0x7f5478078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:43.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f5478076480 0x7f5478078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f549415efe0 msgr2=0x7f5494152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f549415efe0 0x7f5494152200 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f54940d2cd0 tx=0x7f548000c820 comp rx=0 tx=0).stop 2024-06-29T14:41:43.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 shutdown_connections 2024-06-29T14:41:43.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f5478076480 0x7f5478078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5494153f70 0x7f549414da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f549415efe0 0x7f5494152200 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 --2- 172.21.15.59:0/2007126897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54940d23c0 0x7f549414d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 >> 172.21.15.59:0/2007126897 conn(0x7f54940d3ee0 msgr2=0x7f54941544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 shutdown_connections 2024-06-29T14:41:43.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.499+0000 7f549ae8f640 1 -- 172.21.15.59:0/2007126897 wait complete. 2024-06-29T14:41:43.507 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:41:43.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.633+0000 7f223c265640 1 -- 172.21.15.59:0/253569648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340b2270 msgr2=0x7f223415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.633+0000 7f223c265640 1 --2- 172.21.15.59:0/253569648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340b2270 0x7f223415bc20 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f22340cf310 tx=0x7f223002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.633+0000 7f223c265640 1 -- 172.21.15.59:0/253569648 shutdown_connections 2024-06-29T14:41:43.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.633+0000 7f223c265640 1 --2- 172.21.15.59:0/253569648 >> v2:172.21.15.153:3300/0 conn(0x7f22340d1c30 0x7f22340b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.633+0000 7f223c265640 1 --2- 172.21.15.59:0/253569648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340b2270 0x7f223415bc20 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.633+0000 7f223c265640 1 -- 172.21.15.59:0/253569648 >> 172.21.15.59:0/253569648 conn(0x7f2234073e10 msgr2=0x7f2234074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.633+0000 7f223c265640 1 -- 172.21.15.59:0/253569648 shutdown_connections 2024-06-29T14:41:43.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.633+0000 7f223c265640 1 -- 172.21.15.59:0/253569648 wait complete. 2024-06-29T14:41:43.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.635+0000 7f223c265640 1 Processor -- start 2024-06-29T14:41:43.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.635+0000 7f223c265640 1 -- start start 2024-06-29T14:41:43.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223c265640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22340b2270 0x7f22340c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223c265640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22340b2fa0 0x7f223414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223c265640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340d1c30 0x7f22341f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223c265640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f22340b6400 con 0x7f22340d1c30 2024-06-29T14:41:43.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223c265640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22340b6280 con 0x7f22340b2fa0 2024-06-29T14:41:43.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223c265640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f22340b6580 con 0x7f22340b2270 2024-06-29T14:41:43.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223a802640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340d1c30 0x7f22341f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223a802640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340d1c30 0x7f22341f4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51664/0 (socket says 172.21.15.59:51664) 2024-06-29T14:41:43.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223a802640 1 -- 172.21.15.59:0/1761614859 learned_addr learned my addr 172.21.15.59:0/1761614859 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:43.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f2239800640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22340b2fa0 0x7f223414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.636+0000 7f223a001640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22340b2270 0x7f22340c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f223a802640 1 -- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22340b2270 msgr2=0x7f22340c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f223a802640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22340b2270 0x7f22340c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f223a802640 1 -- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22340b2fa0 msgr2=0x7f223414f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f223a802640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22340b2fa0 0x7f223414f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f223a802640 1 -- 172.21.15.59:0/1761614859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223414f8a0 con 0x7f22340d1c30 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f2239800640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22340b2fa0 0x7f223414f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f223a802640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340d1c30 0x7f22341f4aa0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f222c00eab0 tx=0x7f222c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f22267fc640 1 -- 172.21.15.59:0/1761614859 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c00cea0 con 0x7f22340d1c30 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f22267fc640 1 -- 172.21.15.59:0/1761614859 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f222c004b20 con 0x7f22340d1c30 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223415ece0 con 0x7f22340d1c30 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f22267fc640 1 -- 172.21.15.59:0/1761614859 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c005020 con 0x7f22340d1c30 2024-06-29T14:41:43.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.637+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223406e6c0 con 0x7f22340d1c30 2024-06-29T14:41:43.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.638+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22340b6aa0 con 0x7f22340d1c30 2024-06-29T14:41:43.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.640+0000 7f22267fc640 1 -- 172.21.15.59:0/1761614859 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f222c0051c0 con 0x7f22340d1c30 2024-06-29T14:41:43.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.641+0000 7f22267fc640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f2218076480 0x7f2218078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.641+0000 7f223a001640 1 -- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f2218076480 msgr2=0x7f2218078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2556013753 2024-06-29T14:41:43.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.641+0000 7f22267fc640 1 -- 172.21.15.59:0/1761614859 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6473+0+0 (secure 0 0 0) 0x7f222c098910 con 0x7f22340d1c30 2024-06-29T14:41:43.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.641+0000 7f223a001640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f2218076480 0x7f2218078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:43.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.645+0000 7f22267fc640 1 -- 172.21.15.59:0/1761614859 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f222c061590 con 0x7f22340d1c30 2024-06-29T14:41:43.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.837+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f223415f200 con 0x7f22340d1c30 2024-06-29T14:41:43.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.838+0000 7f22267fc640 1 -- 172.21.15.59:0/1761614859 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) ==== 104+0+11 (secure 0 0 0) 0x7f222c065890 con 0x7f22340d1c30 2024-06-29T14:41:43.835 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f2218076480 msgr2=0x7f2218078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f2218076480 0x7f2218078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340d1c30 msgr2=0x7f22341f4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340d1c30 0x7f22341f4aa0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f222c00eab0 tx=0x7f222c00ef70 comp rx=0 tx=0).stop 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 shutdown_connections 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:6832/2556013753,v1:172.21.15.153:6833/2556013753] conn(0x7f2218076480 0x7f2218078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22340d1c30 0x7f22341f4aa0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22340b2fa0 0x7f223414f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 --2- 172.21.15.59:0/1761614859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22340b2270 0x7f22340c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 >> 172.21.15.59:0/1761614859 conn(0x7f2234073e10 msgr2=0x7f2234070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.840+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 shutdown_connections 2024-06-29T14:41:43.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.841+0000 7f223c265640 1 -- 172.21.15.59:0/1761614859 wait complete. 2024-06-29T14:41:43.849 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:41:43.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.979+0000 7fb3917f5640 1 -- 172.21.15.59:0/949362438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb38c0b40a0 msgr2=0x7fb38c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.979+0000 7fb3917f5640 1 --2- 172.21.15.59:0/949362438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb38c0b40a0 0x7fb38c15e8d0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb38c08c4f0 tx=0x7fb38002a530 comp rx=0 tx=0).stop 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.979+0000 7fb3917f5640 1 -- 172.21.15.59:0/949362438 shutdown_connections 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.979+0000 7fb3917f5640 1 --2- 172.21.15.59:0/949362438 >> v2:172.21.15.59:3300/0 conn(0x7fb38c15ee10 0x7fb38c15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.979+0000 7fb3917f5640 1 --2- 172.21.15.59:0/949362438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb38c0b40a0 0x7fb38c15e8d0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.979+0000 7fb3917f5640 1 -- 172.21.15.59:0/949362438 >> 172.21.15.59:0/949362438 conn(0x7fb38c0d3ee0 msgr2=0x7fb38c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.979+0000 7fb3917f5640 1 -- 172.21.15.59:0/949362438 shutdown_connections 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.979+0000 7fb3917f5640 1 -- 172.21.15.59:0/949362438 wait complete. 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.982+0000 7fb3917f5640 1 Processor -- start 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.982+0000 7fb3917f5640 1 -- start start 2024-06-29T14:41:43.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb3917f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0d23c0 0x7fb38c0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb3917f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb38c15ee10 0x7fb38c0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb3917f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb38c14c150 0x7fb38c0b86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb3917f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb38c160280 con 0x7fb38c0d23c0 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb3917f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb38c160100 con 0x7fb38c15ee10 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb3917f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb38c160400 con 0x7fb38c14c150 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb38affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0d23c0 0x7fb38c0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb38affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0d23c0 0x7fb38c0c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51686/0 (socket says 172.21.15.59:51686) 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb38affd640 1 -- 172.21.15.59:0/2863973566 learned_addr learned my addr 172.21.15.59:0/2863973566 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb38a7fc640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb38c15ee10 0x7fb38c0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.983+0000 7fb38b7fe640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb38c14c150 0x7fb38c0b86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb38affd640 1 -- 172.21.15.59:0/2863973566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb38c14c150 msgr2=0x7fb38c0b86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb38affd640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb38c14c150 0x7fb38c0b86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb38affd640 1 -- 172.21.15.59:0/2863973566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb38c15ee10 msgr2=0x7fb38c0b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb38affd640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb38c15ee10 0x7fb38c0b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb38affd640 1 -- 172.21.15.59:0/2863973566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38c0c84e0 con 0x7fb38c0d23c0 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb38affd640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0d23c0 0x7fb38c0c7fa0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c400 tx=0x7fb37c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb3737fe640 1 -- 172.21.15.59:0/2863973566 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c011790 con 0x7fb38c0d23c0 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb38c0ccc60 con 0x7fb38c0d23c0 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb3737fe640 1 -- 172.21.15.59:0/2863973566 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb37c017070 con 0x7fb38c0d23c0 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb3737fe640 1 -- 172.21.15.59:0/2863973566 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c011c60 con 0x7fb38c0d23c0 2024-06-29T14:41:43.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.984+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb38c0cd220 con 0x7fb38c0d23c0 2024-06-29T14:41:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.985+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb38c15e8d0 con 0x7fb38c0d23c0 2024-06-29T14:41:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.986+0000 7fb3737fe640 1 -- 172.21.15.59:0/2863973566 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 75) ==== 97912+0+0 (secure 0 0 0) 0x7fb37c01c460 con 0x7fb38c0d23c0 2024-06-29T14:41:43.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.987+0000 7fb3737fe640 1 -- 172.21.15.59:0/2863973566 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6860+0+0 (secure 0 0 0) 0x7fb37c097380 con 0x7fb38c0d23c0 2024-06-29T14:41:43.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:43.991+0000 7fb3737fe640 1 -- 172.21.15.59:0/2863973566 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb37c0605e0 con 0x7fb38c0d23c0 2024-06-29T14:41:44.023 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.025+0000 7f78dba82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.023 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.025+0000 7f78dba82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.023 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.025+0000 7f78dba82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.023 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.025+0000 7f78dba82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.023 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.025+0000 7f78dba82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.024 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.026+0000 7f78dea88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.024 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.026+0000 7f78dea88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.024 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.026+0000 7f78dea88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.024 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.026+0000 7f78dea88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.024 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:41:44.026+0000 7f78dea88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.187+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb38c0d2950 con 0x7fb38c0d23c0 2024-06-29T14:41:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.187+0000 7fb3737fe640 1 -- 172.21.15.59:0/2863973566 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v145) ==== 100+0+11 (secure 0 0 0) 0x7fb37c0648e0 con 0x7fb38c0d23c0 2024-06-29T14:41:44.185 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T14:41:44.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0d23c0 msgr2=0x7fb38c0c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0d23c0 0x7fb38c0c7fa0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c400 tx=0x7fb37c00c8c0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 shutdown_connections 2024-06-29T14:41:44.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb38c14c150 0x7fb38c0b86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb38c15ee10 0x7fb38c0b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 --2- 172.21.15.59:0/2863973566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0d23c0 0x7fb38c0c7fa0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 >> 172.21.15.59:0/2863973566 conn(0x7fb38c0d3ee0 msgr2=0x7fb38c1608d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 shutdown_connections 2024-06-29T14:41:44.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.190+0000 7fb3917f5640 1 -- 172.21.15.59:0/2863973566 wait complete. 2024-06-29T14:41:44.202 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.324+0000 7fe77c666640 1 -- 172.21.15.59:0/3823725451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 msgr2=0x7fe774143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.324+0000 7fe77c666640 1 --2- 172.21.15.59:0/3823725451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 0x7fe774143840 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe764009830 tx=0x7fe76402a510 comp rx=0 tx=0).stop 2024-06-29T14:41:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.324+0000 7fe77c666640 1 -- 172.21.15.59:0/3823725451 shutdown_connections 2024-06-29T14:41:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.324+0000 7fe77c666640 1 --2- 172.21.15.59:0/3823725451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7740b4ee0 0x7fe77406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.324+0000 7fe77c666640 1 --2- 172.21.15.59:0/3823725451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 0x7fe774143840 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.324+0000 7fe77c666640 1 -- 172.21.15.59:0/3823725451 >> 172.21.15.59:0/3823725451 conn(0x7fe7740b2ab0 msgr2=0x7fe7740b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.324+0000 7fe77c666640 1 -- 172.21.15.59:0/3823725451 shutdown_connections 2024-06-29T14:41:44.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.324+0000 7fe77c666640 1 -- 172.21.15.59:0/3823725451 wait complete. 2024-06-29T14:41:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.326+0000 7fe77c666640 1 Processor -- start 2024-06-29T14:41:44.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.326+0000 7fe77c666640 1 -- start start 2024-06-29T14:41:44.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.327+0000 7fe77c666640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe77406ea10 0x7fe7741f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.327+0000 7fe77c666640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740b4ee0 0x7fe7741e4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe77c666640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 0x7fe7741e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe77c666640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe77415ed80 con 0x7fe7740b4ee0 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe77c666640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe77406f030 con 0x7fe7740d0130 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe77c666640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe77415ef00 con 0x7fe77406ea10 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe779c01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740b4ee0 0x7fe7741e4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe779c01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740b4ee0 0x7fe7741e4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51694/0 (socket says 172.21.15.59:51694) 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe779c01640 1 -- 172.21.15.59:0/4058170564 learned_addr learned my addr 172.21.15.59:0/4058170564 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe77ac03640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 0x7fe7741e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe77a402640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe77406ea10 0x7fe7741f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe779c01640 1 -- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe77406ea10 msgr2=0x7fe7741f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe779c01640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe77406ea10 0x7fe7741f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe779c01640 1 -- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 msgr2=0x7fe7741e4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe779c01640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 0x7fe7741e4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe779c01640 1 -- 172.21.15.59:0/4058170564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7741f2480 con 0x7fe7740b4ee0 2024-06-29T14:41:44.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.328+0000 7fe77ac03640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 0x7fe7741e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.329+0000 7fe779c01640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740b4ee0 0x7fe7741e4a00 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fe77000e9f0 tx=0x7fe77000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.329+0000 7fe76affd640 1 -- 172.21.15.59:0/4058170564 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77000cd80 con 0x7fe7740b4ee0 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.329+0000 7fe76affd640 1 -- 172.21.15.59:0/4058170564 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe770004020 con 0x7fe7740b4ee0 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.329+0000 7fe76affd640 1 -- 172.21.15.59:0/4058170564 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770004470 con 0x7fe7740b4ee0 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.329+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe77413dcf0 con 0x7fe7740b4ee0 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.329+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe77413e2b0 con 0x7fe7740b4ee0 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.330+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe77415f3b0 con 0x7fe7740b4ee0 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.331+0000 7fe76affd640 1 -- 172.21.15.59:0/4058170564 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 75) ==== 97912+0+0 (secure 0 0 0) 0x7fe7700035f0 con 0x7fe7740b4ee0 2024-06-29T14:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.331+0000 7fe76affd640 1 -- 172.21.15.59:0/4058170564 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6860+0+0 (secure 0 0 0) 0x7fe770097ef0 con 0x7fe7740b4ee0 2024-06-29T14:41:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.336+0000 7fe76affd640 1 -- 172.21.15.59:0/4058170564 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe770061820 con 0x7fe7740b4ee0 2024-06-29T14:41:44.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.568+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7740b21d0 con 0x7fe7740b4ee0 2024-06-29T14:41:44.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.569+0000 7fe76affd640 1 -- 172.21.15.59:0/4058170564 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5228 (secure 0 0 0) 0x7fe770061120 con 0x7fe7740b4ee0 2024-06-29T14:41:44.566 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:44.566 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":72,"btime":"2024-06-29T14:41:24:429207+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7442,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/4194962668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4194962668},{"type":"v1","addr":"172.21.15.153:6836","nonce":4194962668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7445,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1840307553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1840307553},{"type":"v1","addr":"172.21.15.153:6837","nonce":1840307553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1010388227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1010388227},{"type":"v1","addr":"172.21.15.59:6834","nonce":1010388227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:20.795436+0000","modified":"2024-06-29T14:41:23.821908+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8844":{"gid":8844,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/957887336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":957887336},{"type":"v1","addr":"172.21.15.59:6835","nonce":957887336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8844,"qdb_cluster":[8844]},"id":8}]} 2024-06-29T14:41:44.567 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 72 2024-06-29T14:41:44.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740b4ee0 msgr2=0x7fe7741e4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740b4ee0 0x7fe7741e4a00 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fe77000e9f0 tx=0x7fe77000eeb0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 shutdown_connections 2024-06-29T14:41:44.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7740d0130 0x7fe7741e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740b4ee0 0x7fe7741e4a00 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 --2- 172.21.15.59:0/4058170564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe77406ea10 0x7fe7741f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 >> 172.21.15.59:0/4058170564 conn(0x7fe7740b2ab0 msgr2=0x7fe77406f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 shutdown_connections 2024-06-29T14:41:44.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.571+0000 7fe77c666640 1 -- 172.21.15.59:0/4058170564 wait complete. 2024-06-29T14:41:44.579 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:44.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.701+0000 7fbe27226640 1 -- 172.21.15.59:0/2286829423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c003aa0 msgr2=0x7fbe0c003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.701+0000 7fbe27226640 1 --2- 172.21.15.59:0/2286829423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c003aa0 0x7fbe0c003f10 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fbe2006fab0 tx=0x7fbe0c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:44.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.701+0000 7fbe257c3640 1 -- 172.21.15.59:0/2286829423 reap_dead start 2024-06-29T14:41:44.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.701+0000 7fbe27226640 1 -- 172.21.15.59:0/2286829423 shutdown_connections 2024-06-29T14:41:44.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.701+0000 7fbe27226640 1 -- 172.21.15.59:0/2286829423 >> 172.21.15.59:0/2286829423 conn(0x7fbe200b24c0 msgr2=0x7fbe200cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.701+0000 7fbe27226640 1 -- 172.21.15.59:0/2286829423 shutdown_connections 2024-06-29T14:41:44.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.701+0000 7fbe27226640 1 -- 172.21.15.59:0/2286829423 wait complete. 2024-06-29T14:41:44.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.703+0000 7fbe27226640 1 Processor -- start 2024-06-29T14:41:44.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.703+0000 7fbe27226640 1 -- start start 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe27226640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe0c003aa0 0x7fbe20146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe27226640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe200ad8b0 0x7fbe20155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe27226640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe200af120 0x7fbe201559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe27226640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbe200cf2a0 con 0x7fbe200af120 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe27226640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbe200cf120 con 0x7fbe200ad8b0 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe27226640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbe200cf420 con 0x7fbe0c003aa0 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe257c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe200af120 0x7fbe201559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe257c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe200af120 0x7fbe201559d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51708/0 (socket says 172.21.15.59:51708) 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe257c3640 1 -- 172.21.15.59:0/939234177 learned_addr learned my addr 172.21.15.59:0/939234177 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:44.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe24fc2640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe0c003aa0 0x7fbe20146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe1ffff640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe200ad8b0 0x7fbe20155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe257c3640 1 -- 172.21.15.59:0/939234177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe0c003aa0 msgr2=0x7fbe20146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe257c3640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe0c003aa0 0x7fbe20146600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe257c3640 1 -- 172.21.15.59:0/939234177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe200ad8b0 msgr2=0x7fbe20155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe257c3640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe200ad8b0 0x7fbe20155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.704+0000 7fbe257c3640 1 -- 172.21.15.59:0/939234177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe20157f50 con 0x7fbe200af120 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.705+0000 7fbe257c3640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe200af120 0x7fbe201559d0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fbe2006fa80 tx=0x7fbe08011c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.705+0000 7fbe1d7fa640 1 -- 172.21.15.59:0/939234177 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe08004440 con 0x7fbe200af120 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.705+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe200ae020 con 0x7fbe200af120 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.705+0000 7fbe1d7fa640 1 -- 172.21.15.59:0/939234177 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe08011e50 con 0x7fbe200af120 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.705+0000 7fbe1d7fa640 1 -- 172.21.15.59:0/939234177 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0800faf0 con 0x7fbe200af120 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.705+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe200ae5e0 con 0x7fbe200af120 2024-06-29T14:41:44.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.706+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe200cead0 con 0x7fbe200af120 2024-06-29T14:41:44.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.707+0000 7fbe1d7fa640 1 -- 172.21.15.59:0/939234177 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 75) ==== 97912+0+0 (secure 0 0 0) 0x7fbe0800fc90 con 0x7fbe200af120 2024-06-29T14:41:44.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.707+0000 7fbe1d7fa640 1 -- 172.21.15.59:0/939234177 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6860+0+0 (secure 0 0 0) 0x7fbe08097a60 con 0x7fbe200af120 2024-06-29T14:41:44.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.712+0000 7fbe1d7fa640 1 -- 172.21.15.59:0/939234177 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbe200cead0 con 0x7fbe200af120 2024-06-29T14:41:44.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.906+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe2006c5f0 con 0x7fbe200af120 2024-06-29T14:41:44.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.907+0000 7fbe1d7fa640 1 -- 172.21.15.59:0/939234177 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) ==== 74+0+18085 (secure 0 0 0) 0x7fbe08020030 con 0x7fbe200af120 2024-06-29T14:41:44.904 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:44.905 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":145,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:43.965694+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.171960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:19.744696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6833/2556013753":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/2928572805":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3238748033":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/76946022":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3684291134":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/2658087184":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3137107702":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1455078472":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/1492442042":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/2519658617":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1609759314":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6832/2556013753":"2024-06-30T14:41:43.965640+0000","172.21.15.153:6833/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6832/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1585009476":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/3926664495":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/706205231":"2024-06-30T14:41:33.044584+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:44.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe200af120 msgr2=0x7fbe201559d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe200af120 0x7fbe201559d0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fbe2006fa80 tx=0x7fbe08011c60 comp rx=0 tx=0).stop 2024-06-29T14:41:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 shutdown_connections 2024-06-29T14:41:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe200af120 0x7fbe201559d0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe200ad8b0 0x7fbe20155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 --2- 172.21.15.59:0/939234177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbe0c003aa0 0x7fbe20146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 >> 172.21.15.59:0/939234177 conn(0x7fbe200b24c0 msgr2=0x7fbe2014c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 shutdown_connections 2024-06-29T14:41:44.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:44.909+0000 7fbe27226640 1 -- 172.21.15.59:0/939234177 wait complete. 2024-06-29T14:41:44.917 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:41:44.917 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.041+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1580332684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 msgr2=0x7fbcf40cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.041+0000 7fbcfb7bd640 1 --2- 172.21.15.59:0/1580332684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 0x7fbcf40cd9b0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbce4005fd0 tx=0x7fbce402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.041+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1580332684 shutdown_connections 2024-06-29T14:41:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.041+0000 7fbcfb7bd640 1 --2- 172.21.15.59:0/1580332684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 0x7fbcf40cd9b0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.041+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1580332684 >> 172.21.15.59:0/1580332684 conn(0x7fbcf40b24c0 msgr2=0x7fbcf40cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.041+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1580332684 shutdown_connections 2024-06-29T14:41:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.041+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1580332684 wait complete. 2024-06-29T14:41:45.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.043+0000 7fbcfb7bd640 1 Processor -- start 2024-06-29T14:41:45.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.043+0000 7fbcfb7bd640 1 -- start start 2024-06-29T14:41:45.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcfb7bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 0x7fbcf413e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcfb7bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4160380 0x7fbcf415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcfb7bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf40aa170 0x7fbcf40aad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcfb7bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf406cc80 con 0x7fbcf4145710 2024-06-29T14:41:45.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcfb7bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf406cb00 con 0x7fbcf40aa170 2024-06-29T14:41:45.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcfb7bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf406ce00 con 0x7fbcf4160380 2024-06-29T14:41:45.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcf9559640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 0x7fbcf413e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcf9559640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 0x7fbcf413e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51720/0 (socket says 172.21.15.59:51720) 2024-06-29T14:41:45.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcf9559640 1 -- 172.21.15.59:0/1251988151 learned_addr learned my addr 172.21.15.59:0/1251988151 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:45.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcf8d58640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4160380 0x7fbcf415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.044+0000 7fbcf9d5a640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf40aa170 0x7fbcf40aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf9559640 1 -- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4160380 msgr2=0x7fbcf415b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf9559640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4160380 0x7fbcf415b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf9559640 1 -- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf40aa170 msgr2=0x7fbcf40aad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf9559640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf40aa170 0x7fbcf40aad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf9559640 1 -- 172.21.15.59:0/1251988151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf413ec40 con 0x7fbcf4145710 2024-06-29T14:41:45.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf9559640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 0x7fbcf413e380 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009b30 tx=0x7fbce4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf1ffb640 1 -- 172.21.15.59:0/1251988151 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4031da0 con 0x7fbcf4145710 2024-06-29T14:41:45.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf1ffb640 1 -- 172.21.15.59:0/1251988151 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbce4033070 con 0x7fbcf4145710 2024-06-29T14:41:45.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf413bd30 con 0x7fbcf4145710 2024-06-29T14:41:45.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcf1ffb640 1 -- 172.21.15.59:0/1251988151 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce40334c0 con 0x7fbcf4145710 2024-06-29T14:41:45.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.045+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf413c2f0 con 0x7fbcf4145710 2024-06-29T14:41:45.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.046+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf40cd9b0 con 0x7fbcf4145710 2024-06-29T14:41:45.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.047+0000 7fbcf1ffb640 1 -- 172.21.15.59:0/1251988151 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7fbce4033660 con 0x7fbcf4145710 2024-06-29T14:41:45.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.048+0000 7fbcf1ffb640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbce0076120 0x7fbce0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.048+0000 7fbcf1ffb640 1 -- 172.21.15.59:0/1251988151 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6860+0+0 (secure 0 0 0) 0x7fbce40b9300 con 0x7fbcf4145710 2024-06-29T14:41:45.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.048+0000 7fbcf8d58640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbce0076120 0x7fbce0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.049+0000 7fbcf8d58640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbce0076120 0x7fbce0078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbcf406c3c0 tx=0x7fbce8009060 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.051+0000 7fbcf1ffb640 1 -- 172.21.15.59:0/1251988151 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbce4052080 con 0x7fbcf4145710 2024-06-29T14:41:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.282+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcf4141ce0 con 0x7fbcf4145710 2024-06-29T14:41:45.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.282+0000 7fbcf1ffb640 1 -- 172.21.15.59:0/1251988151 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5228 (secure 0 0 0) 0x7fbce403f030 con 0x7fbcf4145710 2024-06-29T14:41:45.356 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:45.356 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":72,"btime":"2024-06-29T14:41:24:429207+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7442,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/4194962668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4194962668},{"type":"v1","addr":"172.21.15.153:6836","nonce":4194962668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7445,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1840307553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1840307553},{"type":"v1","addr":"172.21.15.153:6837","nonce":1840307553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1010388227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1010388227},{"type":"v1","addr":"172.21.15.59:6834","nonce":1010388227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:20.795436+0000","modified":"2024-06-29T14:41:23.821908+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8844":{"gid":8844,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/957887336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":957887336},{"type":"v1","addr":"172.21.15.59:6835","nonce":957887336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8844,"qdb_cluster":[8844]},"id":8}]} 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 72 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbce0076120 msgr2=0x7fbce0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbce0076120 0x7fbce0078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbcf406c3c0 tx=0x7fbce8009060 comp rx=0 tx=0).stop 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 msgr2=0x7fbcf413e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 0x7fbcf413e380 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009b30 tx=0x7fbce4003c20 comp rx=0 tx=0).stop 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 shutdown_connections 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbce0076120 0x7fbce0078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf40aa170 0x7fbcf40aad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbcf4160380 0x7fbcf415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 --2- 172.21.15.59:0/1251988151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcf4145710 0x7fbcf413e380 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 >> 172.21.15.59:0/1251988151 conn(0x7fbcf40b24c0 msgr2=0x7fbcf4068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 shutdown_connections 2024-06-29T14:41:45.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.284+0000 7fbcfb7bd640 1 -- 172.21.15.59:0/1251988151 wait complete. 2024-06-29T14:41:45.359 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:45.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.483+0000 7fe25c1fc640 1 -- 172.21.15.59:0/325106108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2540b2fa0 msgr2=0x7fe2540d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.483+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/325106108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2540b2fa0 0x7fe2540d1520 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe244009830 tx=0x7fe24402a510 comp rx=0 tx=0).stop 2024-06-29T14:41:45.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.483+0000 7fe25c1fc640 1 -- 172.21.15.59:0/325106108 shutdown_connections 2024-06-29T14:41:45.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.483+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/325106108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2540b2fa0 0x7fe2540d1520 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.483+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/325106108 >> v2:172.21.15.153:3300/0 conn(0x7fe2540b2270 0x7fe25415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.483+0000 7fe25c1fc640 1 -- 172.21.15.59:0/325106108 >> 172.21.15.59:0/325106108 conn(0x7fe254073e10 msgr2=0x7fe254074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.483+0000 7fe25c1fc640 1 -- 172.21.15.59:0/325106108 shutdown_connections 2024-06-29T14:41:45.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.483+0000 7fe25c1fc640 1 -- 172.21.15.59:0/325106108 wait complete. 2024-06-29T14:41:45.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.485+0000 7fe25c1fc640 1 Processor -- start 2024-06-29T14:41:45.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.485+0000 7fe25c1fc640 1 -- start start 2024-06-29T14:41:45.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe25c1fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2540b2270 0x7fe25406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe25c1fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2540b2fa0 0x7fe2540ad990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe25c1fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2540d1a60 0x7fe254070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe25c1fc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe2540b6cc0 con 0x7fe2540b2270 2024-06-29T14:41:45.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe25c1fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe2540b6b40 con 0x7fe2540b2fa0 2024-06-29T14:41:45.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe25c1fc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe2540cfe90 con 0x7fe2540d1a60 2024-06-29T14:41:45.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe259f98640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2540b2270 0x7fe25406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe259f98640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2540b2270 0x7fe25406eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51722/0 (socket says 172.21.15.59:51722) 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe259f98640 1 -- 172.21.15.59:0/1465583223 learned_addr learned my addr 172.21.15.59:0/1465583223 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe259797640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2540b2fa0 0x7fe2540ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.486+0000 7fe25a799640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2540d1a60 0x7fe254070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe259f98640 1 -- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2540d1a60 msgr2=0x7fe254070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe259f98640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2540d1a60 0x7fe254070270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe259f98640 1 -- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2540b2fa0 msgr2=0x7fe2540ad990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe259f98640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2540b2fa0 0x7fe2540ad990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe259f98640 1 -- 172.21.15.59:0/1465583223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2540ae700 con 0x7fe2540b2270 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe259f98640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2540b2270 0x7fe25406eee0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fe25000ece0 tx=0x7fe25000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe24a7fc640 1 -- 172.21.15.59:0/1465583223 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe250003b70 con 0x7fe2540b2270 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2540ae9e0 con 0x7fe2540b2270 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe24a7fc640 1 -- 172.21.15.59:0/1465583223 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe250003d10 con 0x7fe2540b2270 2024-06-29T14:41:45.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe24a7fc640 1 -- 172.21.15.59:0/1465583223 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe250004280 con 0x7fe2540b2270 2024-06-29T14:41:45.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.487+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2540aefa0 con 0x7fe2540b2270 2024-06-29T14:41:45.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.488+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2540b2680 con 0x7fe2540b2270 2024-06-29T14:41:45.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.489+0000 7fe24a7fc640 1 -- 172.21.15.59:0/1465583223 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7fe250004420 con 0x7fe2540b2270 2024-06-29T14:41:45.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.489+0000 7fe24a7fc640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe238076120 0x7fe238078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.489+0000 7fe24a7fc640 1 -- 172.21.15.59:0/1465583223 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6860+0+0 (secure 0 0 0) 0x7fe2500983f0 con 0x7fe2540b2270 2024-06-29T14:41:45.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.489+0000 7fe259797640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe238076120 0x7fe238078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.490+0000 7fe259797640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe238076120 0x7fe238078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe244034000 tx=0x7fe244004730 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.494+0000 7fe24a7fc640 1 -- 172.21.15.59:0/1465583223 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe2540b2680 con 0x7fe2540b2270 2024-06-29T14:41:45.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.722+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2540d03e0 con 0x7fe2540b2270 2024-06-29T14:41:45.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.723+0000 7fe24a7fc640 1 -- 172.21.15.59:0/1465583223 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+5228 (secure 0 0 0) 0x7fe2500614b0 con 0x7fe2540b2270 2024-06-29T14:41:45.720 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:45.720 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":72,"btime":"2024-06-29T14:41:24:429207+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7442,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/4194962668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4194962668},{"type":"v1","addr":"172.21.15.153:6836","nonce":4194962668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7445,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1840307553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1840307553},{"type":"v1","addr":"172.21.15.153:6837","nonce":1840307553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1010388227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1010388227},{"type":"v1","addr":"172.21.15.59:6834","nonce":1010388227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:20.795436+0000","modified":"2024-06-29T14:41:23.821908+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8844":{"gid":8844,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/957887336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":957887336},{"type":"v1","addr":"172.21.15.59:6835","nonce":957887336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8844,"qdb_cluster":[8844]},"id":8}]} 2024-06-29T14:41:45.720 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 72 2024-06-29T14:41:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe238076120 msgr2=0x7fe238078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe238076120 0x7fe238078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe244034000 tx=0x7fe244004730 comp rx=0 tx=0).stop 2024-06-29T14:41:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2540b2270 msgr2=0x7fe25406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2540b2270 0x7fe25406eee0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fe25000ece0 tx=0x7fe25000c6a0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 shutdown_connections 2024-06-29T14:41:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe238076120 0x7fe238078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2540d1a60 0x7fe254070270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2540b2fa0 0x7fe2540ad990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 --2- 172.21.15.59:0/1465583223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2540b2270 0x7fe25406eee0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 >> 172.21.15.59:0/1465583223 conn(0x7fe254073e10 msgr2=0x7fe254155570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 shutdown_connections 2024-06-29T14:41:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.725+0000 7fe25c1fc640 1 -- 172.21.15.59:0/1465583223 wait complete. 2024-06-29T14:41:45.733 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:45.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.854+0000 7f78bc6ef640 1 -- 172.21.15.59:0/2372941943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b40d23c0 msgr2=0x7f78b40b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.854+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/2372941943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b40d23c0 0x7f78b40b3990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009830 tx=0x7f78b002a510 comp rx=0 tx=0).stop 2024-06-29T14:41:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.854+0000 7f78bc6ef640 1 -- 172.21.15.59:0/2372941943 shutdown_connections 2024-06-29T14:41:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.854+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/2372941943 >> v2:172.21.15.153:3300/0 conn(0x7f78b415efe0 0x7f78b415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.854+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/2372941943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b40d23c0 0x7f78b40b3990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.854+0000 7f78bc6ef640 1 -- 172.21.15.59:0/2372941943 >> 172.21.15.59:0/2372941943 conn(0x7f78b40d3ee0 msgr2=0x7f78b40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.854+0000 7f78bc6ef640 1 -- 172.21.15.59:0/2372941943 shutdown_connections 2024-06-29T14:41:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.854+0000 7f78bc6ef640 1 -- 172.21.15.59:0/2372941943 wait complete. 2024-06-29T14:41:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.856+0000 7f78bc6ef640 1 Processor -- start 2024-06-29T14:41:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.856+0000 7f78bc6ef640 1 -- start start 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78bc6ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40b3ed0 0x7f78b4152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78bc6ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b40d23c0 0x7f78b414e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78bc6ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b415efe0 0x7f78b41f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78bc6ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f78b4160280 con 0x7f78b40b3ed0 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78bc6ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f78b4160100 con 0x7f78b415efe0 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78bc6ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f78b4160400 con 0x7f78b40d23c0 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78ba48b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40b3ed0 0x7f78b4152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78ba48b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40b3ed0 0x7f78b4152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51734/0 (socket says 172.21.15.59:51734) 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78ba48b640 1 -- 172.21.15.59:0/265873949 learned_addr learned my addr 172.21.15.59:0/265873949 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.857+0000 7f78b9c8a640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b40d23c0 0x7f78b414e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78bac8c640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b415efe0 0x7f78b41f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78ba48b640 1 -- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b40d23c0 msgr2=0x7f78b414e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78ba48b640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b40d23c0 0x7f78b414e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78ba48b640 1 -- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b415efe0 msgr2=0x7f78b41f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78ba48b640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b415efe0 0x7f78b41f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78ba48b640 1 -- 172.21.15.59:0/265873949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b41514d0 con 0x7f78b40b3ed0 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78ba48b640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40b3ed0 0x7f78b4152570 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f78b002ab00 tx=0x7f78b0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78aaffd640 1 -- 172.21.15.59:0/265873949 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0005c90 con 0x7f78b40b3ed0 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78aaffd640 1 -- 172.21.15.59:0/265873949 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78b0031e50 con 0x7f78b40b3ed0 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78aaffd640 1 -- 172.21.15.59:0/265873949 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0005260 con 0x7f78b40b3ed0 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b41517b0 con 0x7f78b40b3ed0 2024-06-29T14:41:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.858+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b4151d70 con 0x7f78b40b3ed0 2024-06-29T14:41:45.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.859+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b4069de0 con 0x7f78b40b3ed0 2024-06-29T14:41:45.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.860+0000 7f78aaffd640 1 -- 172.21.15.59:0/265873949 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7f78b0005400 con 0x7f78b40b3ed0 2024-06-29T14:41:45.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.861+0000 7f78aaffd640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f7898076120 0x7f7898078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.861+0000 7f78aaffd640 1 -- 172.21.15.59:0/265873949 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6860+0+0 (secure 0 0 0) 0x7f78b00b87e0 con 0x7f78b40b3ed0 2024-06-29T14:41:45.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.861+0000 7f78b9c8a640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f7898076120 0x7f7898078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.862+0000 7f78b9c8a640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f7898076120 0x7f7898078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f78a4007d60 tx=0x7f78a4007cf0 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:45.865+0000 7f78aaffd640 1 -- 172.21.15.59:0/265873949 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f78b4069de0 con 0x7f78b40b3ed0 2024-06-29T14:41:46.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.059+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78b40b3990 con 0x7f78b40b3ed0 2024-06-29T14:41:46.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.060+0000 7f78aaffd640 1 -- 172.21.15.59:0/265873949 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) ==== 74+0+18085 (secure 0 0 0) 0x7f78b003f030 con 0x7f78b40b3ed0 2024-06-29T14:41:46.058 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:46.058 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":145,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:43.965694+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.171960+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:19.744696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6833/2556013753":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/2928572805":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3238748033":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/76946022":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3684291134":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/2658087184":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3137107702":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1455078472":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/1492442042":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/2519658617":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1609759314":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6832/2556013753":"2024-06-30T14:41:43.965640+0000","172.21.15.153:6833/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6832/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1585009476":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/3926664495":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/706205231":"2024-06-30T14:41:33.044584+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:46.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f7898076120 msgr2=0x7f7898078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f7898076120 0x7f7898078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f78a4007d60 tx=0x7f78a4007cf0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40b3ed0 msgr2=0x7f78b4152570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40b3ed0 0x7f78b4152570 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f78b002ab00 tx=0x7f78b0033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 shutdown_connections 2024-06-29T14:41:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f7898076120 0x7f7898078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78b415efe0 0x7f78b41f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f78b40d23c0 0x7f78b414e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.062+0000 7f78bc6ef640 1 --2- 172.21.15.59:0/265873949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40b3ed0 0x7f78b4152570 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.063+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 >> 172.21.15.59:0/265873949 conn(0x7f78b40d3ee0 msgr2=0x7f78b406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:46.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.063+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 shutdown_connections 2024-06-29T14:41:46.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.063+0000 7f78bc6ef640 1 -- 172.21.15.59:0/265873949 wait complete. 2024-06-29T14:41:46.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:41:46.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 -- 172.21.15.59:0/3943880881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df00d23c0 msgr2=0x7f4df00b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 --2- 172.21.15.59:0/3943880881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df00d23c0 0x7f4df00b3990 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f4df00d04f0 tx=0x7f4de002a560 comp rx=0 tx=0).stop 2024-06-29T14:41:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 -- 172.21.15.59:0/3943880881 shutdown_connections 2024-06-29T14:41:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 --2- 172.21.15.59:0/3943880881 >> v2:172.21.15.153:3300/0 conn(0x7f4df015ee10 0x7f4df015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 --2- 172.21.15.59:0/3943880881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df00b3ed0 0x7f4df015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 --2- 172.21.15.59:0/3943880881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df00d23c0 0x7f4df00b3990 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 -- 172.21.15.59:0/3943880881 >> 172.21.15.59:0/3943880881 conn(0x7f4df00d3ee0 msgr2=0x7f4df00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 -- 172.21.15.59:0/3943880881 shutdown_connections 2024-06-29T14:41:46.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.193+0000 7f4df6363640 1 -- 172.21.15.59:0/3943880881 wait complete. 2024-06-29T14:41:46.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.195+0000 7f4df6363640 1 Processor -- start 2024-06-29T14:41:46.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.195+0000 7f4df6363640 1 -- start start 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df6363640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df00b3ed0 0x7f4df015d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df6363640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df00d23c0 0x7f4df01e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df6363640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df015ee10 0x7f4df01e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df6363640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4df0160070 con 0x7f4df015ee10 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df6363640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4df015fef0 con 0x7f4df00d23c0 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df6363640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4df01601f0 con 0x7f4df00b3ed0 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df015ee10 0x7f4df01e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df015ee10 0x7f4df01e94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51762/0 (socket says 172.21.15.59:51762) 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 -- 172.21.15.59:0/3644604762 learned_addr learned my addr 172.21.15.59:0/3644604762 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4def7fe640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df00d23c0 0x7f4df01e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4deffff640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df00b3ed0 0x7f4df015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 -- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df00b3ed0 msgr2=0x7f4df015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df00b3ed0 0x7f4df015d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 -- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df00d23c0 msgr2=0x7f4df01e8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df00d23c0 0x7f4df01e8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 -- 172.21.15.59:0/3644604762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df01e8530 con 0x7f4df015ee10 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df4900640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df015ee10 0x7f4df01e94e0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f4de800e990 tx=0x7f4de800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4decff9640 1 -- 172.21.15.59:0/3644604762 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8004d20 con 0x7f4df015ee10 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4decff9640 1 -- 172.21.15.59:0/3644604762 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4de800cb00 con 0x7f4df015ee10 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df015e1d0 con 0x7f4df015ee10 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4decff9640 1 -- 172.21.15.59:0/3644604762 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8005020 con 0x7f4df015ee10 2024-06-29T14:41:46.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.197+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df015dc50 con 0x7f4df015ee10 2024-06-29T14:41:46.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.198+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df00b3990 con 0x7f4df015ee10 2024-06-29T14:41:46.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.199+0000 7f4decff9640 1 -- 172.21.15.59:0/3644604762 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7f4de80051c0 con 0x7f4df015ee10 2024-06-29T14:41:46.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.200+0000 7f4decff9640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4ddc076120 0x7f4ddc078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.200+0000 7f4decff9640 1 -- 172.21.15.59:0/3644604762 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6860+0+0 (secure 0 0 0) 0x7f4de8098160 con 0x7f4df015ee10 2024-06-29T14:41:46.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.200+0000 7f4deffff640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4ddc076120 0x7f4ddc078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.200+0000 7f4deffff640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4ddc076120 0x7f4ddc078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4de0033000 tx=0x7f4de0004730 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:46.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.204+0000 7f4decff9640 1 -- 172.21.15.59:0/3644604762 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4de8016030 con 0x7f4df015ee10 2024-06-29T14:41:46.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.326+0000 7f4decff9640 1 -- 172.21.15.59:0/3644604762 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 77) ==== 98247+0+0 (secure 0 0 0) 0x7f4de805d400 con 0x7f4df015ee10 2024-06-29T14:41:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.437+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4df00d2950 con 0x7f4df015ee10 2024-06-29T14:41:46.434 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T14:41:46.438+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:41:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.439+0000 7f4decff9640 1 -- 172.21.15.59:0/3644604762 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) ==== 161+0+0 (secure 0 0 0) 0x7f4de8061870 con 0x7f4df015ee10 2024-06-29T14:41:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:41:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4ddc076120 msgr2=0x7f4ddc078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4ddc076120 0x7f4ddc078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4de0033000 tx=0x7f4de0004730 comp rx=0 tx=0).stop 2024-06-29T14:41:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df015ee10 msgr2=0x7f4df01e94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df015ee10 0x7f4df01e94e0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f4de800e990 tx=0x7f4de800ee50 comp rx=0 tx=0).stop 2024-06-29T14:41:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 shutdown_connections 2024-06-29T14:41:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4ddc076120 0x7f4ddc078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df015ee10 0x7f4df01e94e0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4df00d23c0 0x7f4df01e8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 --2- 172.21.15.59:0/3644604762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4df00b3ed0 0x7f4df015d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 >> 172.21.15.59:0/3644604762 conn(0x7f4df00d3ee0 msgr2=0x7f4df006d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 shutdown_connections 2024-06-29T14:41:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.442+0000 7f4df6363640 1 -- 172.21.15.59:0/3644604762 wait complete. 2024-06-29T14:41:46.442 INFO:tasks.ceph.mds.a.smithi059.stderr: -9019> 2024-06-29T14:41:23.386+0000 7f90a49af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:41:46.451 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:41:46.503 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:41:46.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.577+0000 7f54572c1640 1 -- 172.21.15.59:0/3703113629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 msgr2=0x7f543c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.577+0000 7f54572c1640 1 --2- 172.21.15.59:0/3703113629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 0x7f543c003ec0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f544400bb20 tx=0x7f543c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:46.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.578+0000 7f54572c1640 1 -- 172.21.15.59:0/3703113629 shutdown_connections 2024-06-29T14:41:46.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.578+0000 7f54572c1640 1 --2- 172.21.15.59:0/3703113629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 0x7f543c003ec0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.578+0000 7f54572c1640 1 -- 172.21.15.59:0/3703113629 >> 172.21.15.59:0/3703113629 conn(0x7f545006cb30 msgr2=0x7f545006cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:46.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.578+0000 7f54572c1640 1 -- 172.21.15.59:0/3703113629 shutdown_connections 2024-06-29T14:41:46.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.578+0000 7f54572c1640 1 -- 172.21.15.59:0/3703113629 wait complete. 2024-06-29T14:41:46.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.579+0000 7f54572c1640 1 Processor -- start 2024-06-29T14:41:46.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.579+0000 7f54572c1640 1 -- start start 2024-06-29T14:41:46.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f54572c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 0x7f5450140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f54572c1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54501f3390 0x7f545013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f54572c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54501f37a0 0x7f545013f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f54572c1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f545015bb40 con 0x7f543c003a50 2024-06-29T14:41:46.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f54572c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f545015ef20 con 0x7f54501f37a0 2024-06-29T14:41:46.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f54572c1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f545015bcc0 con 0x7f54501f3390 2024-06-29T14:41:46.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 0x7f5450140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 0x7f5450140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51780/0 (socket says 172.21.15.59:51780) 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 -- 172.21.15.59:0/3615384573 learned_addr learned my addr 172.21.15.59:0/3615384573 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 -- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54501f3390 msgr2=0x7f545013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54501f3390 0x7f545013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 -- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54501f37a0 msgr2=0x7f545013f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54501f37a0 0x7f545013f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 -- 172.21.15.59:0/3615384573 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54501411a0 con 0x7f543c003a50 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.581+0000 7f545505d640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 0x7f5450140950 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f544400bc40 tx=0x7f5444031430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.582+0000 7f5441ffb640 1 -- 172.21.15.59:0/3615384573 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5444035040 con 0x7f543c003a50 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.582+0000 7f5441ffb640 1 -- 172.21.15.59:0/3615384573 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f544403e070 con 0x7f543c003a50 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.582+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f545013e200 con 0x7f543c003a50 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.582+0000 7f5441ffb640 1 -- 172.21.15.59:0/3615384573 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f544403a630 con 0x7f543c003a50 2024-06-29T14:41:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.582+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f545013e7c0 con 0x7f543c003a50 2024-06-29T14:41:46.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.583+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f545006f180 con 0x7f543c003a50 2024-06-29T14:41:46.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.584+0000 7f5441ffb640 1 -- 172.21.15.59:0/3615384573 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 77) ==== 98247+0+0 (secure 0 0 0) 0x7f5444038020 con 0x7f543c003a50 2024-06-29T14:41:46.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.584+0000 7f5441ffb640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5434076390 0x7f54340788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.584+0000 7f5441ffb640 1 -- 172.21.15.59:0/3615384573 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 6946+0+0 (secure 0 0 0) 0x7f54440b8a20 con 0x7f543c003a50 2024-06-29T14:41:46.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.584+0000 7f545485c640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5434076390 0x7f54340788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.585+0000 7f545485c640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5434076390 0x7f54340788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f54500b0dc0 tx=0x7f544c001110 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:46.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.588+0000 7f5441ffb640 1 -- 172.21.15.59:0/3615384573 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5444081e20 con 0x7f543c003a50 2024-06-29T14:41:46.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:46.819+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f545015e2a0 con 0x7f543c003a50 2024-06-29T14:41:47.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.441+0000 7f5441ffb640 1 -- 172.21.15.59:0/3615384573 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) ==== 104+0+0 (secure 0 0 0) 0x7f5444081720 con 0x7f543c003a50 2024-06-29T14:41:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5434076390 msgr2=0x7f54340788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5434076390 0x7f54340788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f54500b0dc0 tx=0x7f544c001110 comp rx=0 tx=0).stop 2024-06-29T14:41:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 msgr2=0x7f5450140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 0x7f5450140950 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f544400bc40 tx=0x7f5444031430 comp rx=0 tx=0).stop 2024-06-29T14:41:47.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 shutdown_connections 2024-06-29T14:41:47.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5434076390 0x7f54340788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54501f37a0 0x7f545013f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54501f3390 0x7f545013efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 --2- 172.21.15.59:0/3615384573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f543c003a50 0x7f5450140950 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 >> 172.21.15.59:0/3615384573 conn(0x7f545006cb30 msgr2=0x7f54500cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:47.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 shutdown_connections 2024-06-29T14:41:47.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.444+0000 7f54572c1640 1 -- 172.21.15.59:0/3615384573 wait complete. 2024-06-29T14:41:47.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:41:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.587+0000 7ff267881640 1 -- 172.21.15.59:0/1376313199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26015f820 msgr2=0x7ff2600cfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.587+0000 7ff267881640 1 --2- 172.21.15.59:0/1376313199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26015f820 0x7ff2600cfa70 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff260145a50 tx=0x7ff25002a820 comp rx=0 tx=0).stop 2024-06-29T14:41:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.587+0000 7ff267881640 1 -- 172.21.15.59:0/1376313199 shutdown_connections 2024-06-29T14:41:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.587+0000 7ff267881640 1 --2- 172.21.15.59:0/1376313199 >> v2:172.21.15.153:3300/0 conn(0x7ff26006dd20 0x7ff26006e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.587+0000 7ff267881640 1 --2- 172.21.15.59:0/1376313199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26015f820 0x7ff2600cfa70 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.587+0000 7ff267881640 1 -- 172.21.15.59:0/1376313199 >> 172.21.15.59:0/1376313199 conn(0x7ff2600b2ab0 msgr2=0x7ff2600b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.587+0000 7ff267881640 1 -- 172.21.15.59:0/1376313199 shutdown_connections 2024-06-29T14:41:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.587+0000 7ff267881640 1 -- 172.21.15.59:0/1376313199 wait complete. 2024-06-29T14:41:47.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.589+0000 7ff267881640 1 Processor -- start 2024-06-29T14:41:47.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.589+0000 7ff267881640 1 -- start start 2024-06-29T14:41:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff267881640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff26006ca70 0x7ff260143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff267881640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26006dd20 0x7ff26015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff267881640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff26015f820 0x7ff2600aa200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff267881640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff26015d870 con 0x7ff26006dd20 2024-06-29T14:41:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff267881640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff260144cf0 con 0x7ff26015f820 2024-06-29T14:41:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff267881640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff26015d9f0 con 0x7ff26006ca70 2024-06-29T14:41:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26006dd20 0x7ff26015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26006dd20 0x7ff26015a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51806/0 (socket says 172.21.15.59:51806) 2024-06-29T14:41:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 -- 172.21.15.59:0/3517664013 learned_addr learned my addr 172.21.15.59:0/3517664013 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff26561d640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff26006ca70 0x7ff260143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff265e1e640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff26015f820 0x7ff2600aa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 -- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff26006ca70 msgr2=0x7ff260143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff26006ca70 0x7ff260143220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 -- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff26015f820 msgr2=0x7ff2600aa200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff26015f820 0x7ff2600aa200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 -- 172.21.15.59:0/3517664013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26013c3a0 con 0x7ff26006dd20 2024-06-29T14:41:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff26561d640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff26006ca70 0x7ff260143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff264e1c640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26006dd20 0x7ff26015a610 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7ff25400d740 tx=0x7ff25400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.591+0000 7ff25dffb640 1 -- 172.21.15.59:0/3517664013 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254017070 con 0x7ff26006dd20 2024-06-29T14:41:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.592+0000 7ff25dffb640 1 -- 172.21.15.59:0/3517664013 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff254009d20 con 0x7ff26006dd20 2024-06-29T14:41:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.592+0000 7ff25dffb640 1 -- 172.21.15.59:0/3517664013 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2540106c0 con 0x7ff26006dd20 2024-06-29T14:41:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.592+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff260143b20 con 0x7ff26006dd20 2024-06-29T14:41:47.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.592+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff26013c920 con 0x7ff26006dd20 2024-06-29T14:41:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.593+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff26015ab50 con 0x7ff26006dd20 2024-06-29T14:41:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.594+0000 7ff25dffb640 1 -- 172.21.15.59:0/3517664013 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 77) ==== 98247+0+0 (secure 0 0 0) 0x7ff25401a070 con 0x7ff26006dd20 2024-06-29T14:41:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.594+0000 7ff25dffb640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff24c076360 0x7ff24c078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.594+0000 7ff25dffb640 1 -- 172.21.15.59:0/3517664013 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 6946+0+0 (secure 0 0 0) 0x7ff254097ff0 con 0x7ff26006dd20 2024-06-29T14:41:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.595+0000 7ff26561d640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff24c076360 0x7ff24c078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.595+0000 7ff26561d640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff24c076360 0x7ff24c078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff250034000 tx=0x7ff250004ab0 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:47.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.599+0000 7ff25dffb640 1 -- 172.21.15.59:0/3517664013 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff26015ab50 con 0x7ff26006dd20 2024-06-29T14:41:47.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:47.789+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff26006d7c0 con 0x7ff26006dd20 2024-06-29T14:41:48.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.452+0000 7ff25dffb640 1 -- 172.21.15.59:0/3517664013 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v147) ==== 181+0+0 (secure 0 0 0) 0x7ff254061260 con 0x7ff26006dd20 2024-06-29T14:41:48.451 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:41:48.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff24c076360 msgr2=0x7ff24c078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff24c076360 0x7ff24c078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff250034000 tx=0x7ff250004ab0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26006dd20 msgr2=0x7ff26015a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26006dd20 0x7ff26015a610 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7ff25400d740 tx=0x7ff25400dc00 comp rx=0 tx=0).stop 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 shutdown_connections 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff24c076360 0x7ff24c078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff26015f820 0x7ff2600aa200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff26006dd20 0x7ff26015a610 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 --2- 172.21.15.59:0/3517664013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff26006ca70 0x7ff260143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 >> 172.21.15.59:0/3517664013 conn(0x7ff2600b2ab0 msgr2=0x7ff26015e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 shutdown_connections 2024-06-29T14:41:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.455+0000 7ff267881640 1 -- 172.21.15.59:0/3517664013 wait complete. 2024-06-29T14:41:48.471 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:41:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.602+0000 7f60dcf62640 1 -- 172.21.15.59:0/532773619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 msgr2=0x7f60d80b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.602+0000 7f60dcf62640 1 --2- 172.21.15.59:0/532773619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 0x7f60d80b5cb0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f60d0005fd0 tx=0x7f60d002a530 comp rx=0 tx=0).stop 2024-06-29T14:41:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.602+0000 7f60dcf62640 1 -- 172.21.15.59:0/532773619 shutdown_connections 2024-06-29T14:41:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.602+0000 7f60dcf62640 1 --2- 172.21.15.59:0/532773619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d815ff00 0x7f60d815d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.602+0000 7f60dcf62640 1 --2- 172.21.15.59:0/532773619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 0x7f60d80b5cb0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.602+0000 7f60dcf62640 1 -- 172.21.15.59:0/532773619 >> 172.21.15.59:0/532773619 conn(0x7f60d80b6ed0 msgr2=0x7f60d80b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.602+0000 7f60dcf62640 1 -- 172.21.15.59:0/532773619 shutdown_connections 2024-06-29T14:41:48.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.602+0000 7f60dcf62640 1 -- 172.21.15.59:0/532773619 wait complete. 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.604+0000 7f60dcf62640 1 Processor -- start 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.604+0000 7f60dcf62640 1 -- start start 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60dcf62640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d1130 0x7f60d8150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60dcf62640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 0x7f60d80c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60dcf62640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d813d940 0x7f60d813dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60dcf62640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f60d815e3c0 con 0x7f60d8145650 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60dcf62640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60d815e240 con 0x7f60d80d1130 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60dcf62640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f60d815e540 con 0x7f60d813d940 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 0x7f60d80c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 0x7f60d80c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51818/0 (socket says 172.21.15.59:51818) 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 -- 172.21.15.59:0/2070410818 learned_addr learned my addr 172.21.15.59:0/2070410818 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d6d9d640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d813d940 0x7f60d813dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:48.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d659c640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d1130 0x7f60d8150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 -- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d813d940 msgr2=0x7f60d813dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d813d940 0x7f60d813dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 -- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d1130 msgr2=0x7f60d8150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d1130 0x7f60d8150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 -- 172.21.15.59:0/2070410818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d8144f40 con 0x7f60d8145650 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60d5d9b640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 0x7f60d80c6920 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f60c000b3f0 tx=0x7f60c000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60ceffd640 1 -- 172.21.15.59:0/2070410818 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0015020 con 0x7f60d8145650 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60ceffd640 1 -- 172.21.15.59:0/2070410818 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60c0011920 con 0x7f60d8145650 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d813e290 con 0x7f60d8145650 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.606+0000 7f60ceffd640 1 -- 172.21.15.59:0/2070410818 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0011d70 con 0x7f60d8145650 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.607+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d8144180 con 0x7f60d8145650 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.607+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d815ea60 con 0x7f60d8145650 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.608+0000 7f60ceffd640 1 -- 172.21.15.59:0/2070410818 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f60c0018020 con 0x7f60d8145650 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.609+0000 7f60ceffd640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f60b8076420 0x7f60b8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.609+0000 7f60ceffd640 1 -- 172.21.15.59:0/2070410818 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 6508+0+0 (secure 0 0 0) 0x7f60c00977a0 con 0x7f60d8145650 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.609+0000 7f60d659c640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f60b8076420 0x7f60b8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.610+0000 7f60d659c640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f60b8076420 0x7f60b8078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60d0033000 tx=0x7f60d0004730 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:48.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.613+0000 7f60ceffd640 1 -- 172.21.15.59:0/2070410818 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f60c001f030 con 0x7f60d8145650 2024-06-29T14:41:48.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:48.804+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f60d80b2f40 con 0x7f60d8145650 2024-06-29T14:41:49.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.454+0000 7f60ceffd640 1 -- 172.21.15.59:0/2070410818 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v148) ==== 169+0+0 (secure 0 0 0) 0x7f60d813e290 con 0x7f60d8145650 2024-06-29T14:41:49.451 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T14:41:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f60b8076420 msgr2=0x7f60b8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f60b8076420 0x7f60b8078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60d0033000 tx=0x7f60d0004730 comp rx=0 tx=0).stop 2024-06-29T14:41:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 msgr2=0x7f60d80c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 0x7f60d80c6920 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f60c000b3f0 tx=0x7f60c000b8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 shutdown_connections 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f60b8076420 0x7f60b8078930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60d813d940 0x7f60d813dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60d8145650 0x7f60d80c6920 secure :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f60c000b3f0 tx=0x7f60c000b8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 --2- 172.21.15.59:0/2070410818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d80d1130 0x7f60d8150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 >> 172.21.15.59:0/2070410818 conn(0x7f60d80b6ed0 msgr2=0x7f60d8071630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 shutdown_connections 2024-06-29T14:41:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.458+0000 7f60dcf62640 1 -- 172.21.15.59:0/2070410818 wait complete. 2024-06-29T14:41:49.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:41:49.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.595+0000 7ff4366d6640 1 -- 172.21.15.59:0/3162582532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff43015e290 msgr2=0x7ff4300ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.595+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3162582532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff43015e290 0x7ff4300ce110 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7ff420005fd0 tx=0x7ff42002a580 comp rx=0 tx=0).stop 2024-06-29T14:41:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.595+0000 7ff4366d6640 1 -- 172.21.15.59:0/3162582532 shutdown_connections 2024-06-29T14:41:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.595+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3162582532 >> v2:172.21.15.153:3300/0 conn(0x7ff4300b40a0 0x7ff4300b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.595+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3162582532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff43015e290 0x7ff4300ce110 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.595+0000 7ff4366d6640 1 -- 172.21.15.59:0/3162582532 >> 172.21.15.59:0/3162582532 conn(0x7ff4300d3ee0 msgr2=0x7ff4300d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.595+0000 7ff4366d6640 1 -- 172.21.15.59:0/3162582532 shutdown_connections 2024-06-29T14:41:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.595+0000 7ff4366d6640 1 -- 172.21.15.59:0/3162582532 wait complete. 2024-06-29T14:41:49.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.597+0000 7ff4366d6640 1 Processor -- start 2024-06-29T14:41:49.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.597+0000 7ff4366d6640 1 -- start start 2024-06-29T14:41:49.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.598+0000 7ff4366d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4300ce650 0x7ff43014e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.598+0000 7ff4366d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e290 0x7ff4301f88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.598+0000 7ff4366d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430153200 0x7ff4301f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.598+0000 7ff4366d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff4300b7cc0 con 0x7ff430153200 2024-06-29T14:41:49.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.598+0000 7ff4366d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4300b7b40 con 0x7ff4300ce650 2024-06-29T14:41:49.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.598+0000 7ff4366d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff4300b7e40 con 0x7ff43015e290 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.598+0000 7ff434c73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430153200 0x7ff4301f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.598+0000 7ff434c73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430153200 0x7ff4301f8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51838/0 (socket says 172.21.15.59:51838) 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff434c73640 1 -- 172.21.15.59:0/3706909296 learned_addr learned my addr 172.21.15.59:0/3706909296 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff42ffff640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4300ce650 0x7ff43014e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff42f7fe640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e290 0x7ff4301f88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff434c73640 1 -- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e290 msgr2=0x7ff4301f88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff434c73640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e290 0x7ff4301f88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff434c73640 1 -- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4300ce650 msgr2=0x7ff43014e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff434c73640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4300ce650 0x7ff43014e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff434c73640 1 -- 172.21.15.59:0/3706909296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43014e540 con 0x7ff430153200 2024-06-29T14:41:49.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff434c73640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430153200 0x7ff4301f8de0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7ff42800e990 tx=0x7ff42800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:49.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff42cff9640 1 -- 172.21.15.59:0/3706909296 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428004d20 con 0x7ff430153200 2024-06-29T14:41:49.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff42cff9640 1 -- 172.21.15.59:0/3706909296 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff42800ca40 con 0x7ff430153200 2024-06-29T14:41:49.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff430160a80 con 0x7ff430153200 2024-06-29T14:41:49.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff42cff9640 1 -- 172.21.15.59:0/3706909296 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428005020 con 0x7ff430153200 2024-06-29T14:41:49.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.599+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff430161020 con 0x7ff430153200 2024-06-29T14:41:49.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.600+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4300b8b80 con 0x7ff430153200 2024-06-29T14:41:49.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.601+0000 7ff42cff9640 1 -- 172.21.15.59:0/3706909296 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7ff4280051c0 con 0x7ff430153200 2024-06-29T14:41:49.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.602+0000 7ff42cff9640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff41c0763f0 0x7ff41c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.602+0000 7ff42cff9640 1 -- 172.21.15.59:0/3706909296 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7ff428098330 con 0x7ff430153200 2024-06-29T14:41:49.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.602+0000 7ff42ffff640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff41c0763f0 0x7ff41c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.603+0000 7ff42ffff640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff41c0763f0 0x7ff41c078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff420033000 tx=0x7ff420004730 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:49.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.606+0000 7ff42cff9640 1 -- 172.21.15.59:0/3706909296 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff43014e540 con 0x7ff430153200 2024-06-29T14:41:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.797+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7ff43015af40 con 0x7ff430153200 2024-06-29T14:41:49.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.801+0000 7ff42cff9640 1 -- 172.21.15.59:0/3706909296 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v55) ==== 71+0+0 (secure 0 0 0) 0x7ff428016030 con 0x7ff430153200 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.803+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff41c0763f0 msgr2=0x7ff41c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.803+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff41c0763f0 0x7ff41c078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff420033000 tx=0x7ff420004730 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.803+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430153200 msgr2=0x7ff4301f8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.803+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430153200 0x7ff4301f8de0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7ff42800e990 tx=0x7ff42800ee50 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.804+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 shutdown_connections 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.804+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff41c0763f0 0x7ff41c078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.804+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430153200 0x7ff4301f8de0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.804+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e290 0x7ff4301f88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.804+0000 7ff4366d6640 1 --2- 172.21.15.59:0/3706909296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4300ce650 0x7ff43014e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.804+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 >> 172.21.15.59:0/3706909296 conn(0x7ff4300d3ee0 msgr2=0x7ff43015bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.804+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 shutdown_connections 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.804+0000 7ff4366d6640 1 -- 172.21.15.59:0/3706909296 wait complete. 2024-06-29T14:41:49.812 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-06-29T14:41:49.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.933+0000 7f50a2695640 1 -- 172.21.15.59:0/1423430612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 msgr2=0x7f509c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.933+0000 7f50a2695640 1 --2- 172.21.15.59:0/1423430612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 0x7f509c0cf080 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f508c009860 tx=0x7f508c02a540 comp rx=0 tx=0).stop 2024-06-29T14:41:49.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.933+0000 7f50a2695640 1 -- 172.21.15.59:0/1423430612 shutdown_connections 2024-06-29T14:41:49.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.933+0000 7f50a2695640 1 --2- 172.21.15.59:0/1423430612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c0ce950 0x7f509c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.933+0000 7f50a2695640 1 --2- 172.21.15.59:0/1423430612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 0x7f509c0cf080 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.933+0000 7f50a2695640 1 -- 172.21.15.59:0/1423430612 >> 172.21.15.59:0/1423430612 conn(0x7f509c0b5610 msgr2=0x7f509c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.933+0000 7f50a2695640 1 -- 172.21.15.59:0/1423430612 shutdown_connections 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.933+0000 7f50a2695640 1 -- 172.21.15.59:0/1423430612 wait complete. 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.935+0000 7f50a2695640 1 Processor -- start 2024-06-29T14:41:49.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.935+0000 7f50a2695640 1 -- start start 2024-06-29T14:41:49.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f50a2695640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 0x7f509c1e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f50a2695640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c0ce950 0x7f509c0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f50a2695640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c0cf5c0 0x7f509c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f50a2695640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f509c0b7f60 con 0x7f509c08c190 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f50a2695640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f509c0b7de0 con 0x7f509c0cf5c0 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f50a2695640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f509c0b80e0 con 0x7f509c0ce950 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f509bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 0x7f509c1e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f509bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 0x7f509c1e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51852/0 (socket says 172.21.15.59:51852) 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f509bfff640 1 -- 172.21.15.59:0/4060622693 learned_addr learned my addr 172.21.15.59:0/4060622693 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f50a0c32640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c0cf5c0 0x7f509c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.936+0000 7f509b7fe640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c0ce950 0x7f509c0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f509bfff640 1 -- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c0ce950 msgr2=0x7f509c0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f509bfff640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c0ce950 0x7f509c0ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f509bfff640 1 -- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c0cf5c0 msgr2=0x7f509c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f509bfff640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c0cf5c0 0x7f509c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f509bfff640 1 -- 172.21.15.59:0/4060622693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509c1e8430 con 0x7f509c08c190 2024-06-29T14:41:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f509b7fe640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c0ce950 0x7f509c0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:49.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f50a0c32640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c0cf5c0 0x7f509c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:49.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f509bfff640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 0x7f509c1e7970 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f508c02ab30 tx=0x7f508c033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:49.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f5098ff9640 1 -- 172.21.15.59:0/4060622693 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c005c90 con 0x7f509c08c190 2024-06-29T14:41:49.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f5098ff9640 1 -- 172.21.15.59:0/4060622693 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f508c031e50 con 0x7f509c08c190 2024-06-29T14:41:49.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f5098ff9640 1 -- 172.21.15.59:0/4060622693 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c005260 con 0x7f509c08c190 2024-06-29T14:41:49.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509c1e8710 con 0x7f509c08c190 2024-06-29T14:41:49.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.937+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f509c1e7eb0 con 0x7f509c08c190 2024-06-29T14:41:49.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.938+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509c1e8950 con 0x7f509c08c190 2024-06-29T14:41:49.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.939+0000 7f5098ff9640 1 -- 172.21.15.59:0/4060622693 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f508c005400 con 0x7f509c08c190 2024-06-29T14:41:49.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.939+0000 7f5098ff9640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f50880763f0 0x7f5088078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.939+0000 7f5098ff9640 1 -- 172.21.15.59:0/4060622693 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7f508c0b8c70 con 0x7f509c08c190 2024-06-29T14:41:49.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.939+0000 7f509b7fe640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f50880763f0 0x7f5088078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.940+0000 7f509b7fe640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f50880763f0 0x7f5088078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f509c1f8920 tx=0x7f5090009040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:49.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:49.944+0000 7f5098ff9640 1 -- 172.21.15.59:0/4060622693 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f509c1e8950 con 0x7f509c08c190 2024-06-29T14:41:50.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.217+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) -- 0x7f509c0cf080 con 0x7f509c08c190 2024-06-29T14:41:50.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.455+0000 7f5098ff9640 1 -- 172.21.15.59:0/4060622693 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v781) ==== 160+0+0 (secure 0 0 0) 0x7f508c03f030 con 0x7f509c08c190 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.457+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f50880763f0 msgr2=0x7f5088078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.457+0000 7f50a2695640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f50880763f0 0x7f5088078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f509c1f8920 tx=0x7f5090009040 comp rx=0 tx=0).stop 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 msgr2=0x7f509c1e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 0x7f509c1e7970 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f508c02ab30 tx=0x7f508c033c90 comp rx=0 tx=0).stop 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 shutdown_connections 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f50880763f0 0x7f5088078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509c0cf5c0 0x7f509c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f509c0ce950 0x7f509c0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 --2- 172.21.15.59:0/4060622693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509c08c190 0x7f509c1e7970 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 >> 172.21.15.59:0/4060622693 conn(0x7f509c0b5610 msgr2=0x7f509c1552a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 shutdown_connections 2024-06-29T14:41:50.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.458+0000 7f50a2695640 1 -- 172.21.15.59:0/4060622693 wait complete. 2024-06-29T14:41:50.470 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-06-29T14:41:50.470 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-06-29T14:41:50.470 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:41:50.470 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.593+0000 7efd29768640 1 -- 172.21.15.59:0/175569892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10004090 msgr2=0x7efd10004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.593+0000 7efd29768640 1 --2- 172.21.15.59:0/175569892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd10004090 0x7efd10004800 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7efd0c009b20 tx=0x7efd10005830 comp rx=0 tx=0).stop 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.593+0000 7efd237fe640 1 -- 172.21.15.59:0/175569892 reap_dead start 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.593+0000 7efd29768640 1 -- 172.21.15.59:0/175569892 shutdown_connections 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.593+0000 7efd29768640 1 -- 172.21.15.59:0/175569892 >> 172.21.15.59:0/175569892 conn(0x7efd240d3ee0 msgr2=0x7efd240d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.593+0000 7efd29768640 1 -- 172.21.15.59:0/175569892 shutdown_connections 2024-06-29T14:41:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.593+0000 7efd29768640 1 -- 172.21.15.59:0/175569892 wait complete. 2024-06-29T14:41:50.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.595+0000 7efd29768640 1 Processor -- start 2024-06-29T14:41:50.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.595+0000 7efd29768640 1 -- start start 2024-06-29T14:41:50.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd29768640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd10004090 0x7efd241598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd29768640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd241f5640 0x7efd240ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd29768640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd241f5a50 0x7efd240cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd29768640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efd2415fd40 con 0x7efd241f5640 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd29768640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd2415fbc0 con 0x7efd10004090 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd29768640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efd2415fec0 con 0x7efd241f5a50 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd227fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd241f5640 0x7efd240ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd227fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd241f5640 0x7efd240ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51860/0 (socket says 172.21.15.59:51860) 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd227fc640 1 -- 172.21.15.59:0/3022838412 learned_addr learned my addr 172.21.15.59:0/3022838412 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd22ffd640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd10004090 0x7efd241598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd237fe640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd241f5a50 0x7efd240cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd227fc640 1 -- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd241f5a50 msgr2=0x7efd240cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd227fc640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd241f5a50 0x7efd240cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd227fc640 1 -- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd10004090 msgr2=0x7efd241598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.596+0000 7efd227fc640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd10004090 0x7efd241598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.597+0000 7efd227fc640 1 -- 172.21.15.59:0/3022838412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd24155230 con 0x7efd241f5640 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.597+0000 7efd237fe640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd241f5a50 0x7efd240cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.597+0000 7efd227fc640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd241f5640 0x7efd240ccd40 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7efd0c036040 tx=0x7efd0c0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.597+0000 7efd1b7fe640 1 -- 172.21.15.59:0/3022838412 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c038040 con 0x7efd241f5640 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.597+0000 7efd1b7fe640 1 -- 172.21.15.59:0/3022838412 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd0c02b7e0 con 0x7efd241f5640 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.597+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2415a360 con 0x7efd241f5640 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.597+0000 7efd1b7fe640 1 -- 172.21.15.59:0/3022838412 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c033b40 con 0x7efd241f5640 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.597+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd241557b0 con 0x7efd241f5640 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.598+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd2415a0b0 con 0x7efd241f5640 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.599+0000 7efd1b7fe640 1 -- 172.21.15.59:0/3022838412 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7efd0c043020 con 0x7efd241f5640 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.599+0000 7efd1b7fe640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7efd040763f0 0x7efd04078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.599+0000 7efd1b7fe640 1 -- 172.21.15.59:0/3022838412 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7efd0c0b7b50 con 0x7efd241f5640 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.599+0000 7efd22ffd640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7efd040763f0 0x7efd04078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.600+0000 7efd22ffd640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7efd040763f0 0x7efd04078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efd14001be0 tx=0x7efd14006040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:50.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.604+0000 7efd1b7fe640 1 -- 172.21.15.59:0/3022838412 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd0c04a020 con 0x7efd241f5640 2024-06-29T14:41:50.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.799+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd24160420 con 0x7efd241f5640 2024-06-29T14:41:50.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.799+0000 7efd1b7fe640 1 -- 172.21.15.59:0/3022838412 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) ==== 74+0+14435 (secure 0 0 0) 0x7efd0c02b9b0 con 0x7efd241f5640 2024-06-29T14:41:50.797 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:50.797 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":148,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:49.452609+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6833/957887336":"2024-06-30T14:41:46.438037+0000","172.21.15.153:6833/2556013753":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/2928572805":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3238748033":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/76946022":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3684291134":"2024-06-30T14:41:43.965640+0000","172.21.15.59:6835/957887336":"2024-06-30T14:41:46.438037+0000","172.21.15.153:0/2658087184":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/3137107702":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1455078472":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/1492442042":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/2519658617":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1609759314":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6832/2556013753":"2024-06-30T14:41:43.965640+0000","172.21.15.153:6833/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:6832/776633953":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/1585009476":"2024-06-30T14:41:33.044584+0000","172.21.15.153:0/3926664495":"2024-06-30T14:41:43.965640+0000","172.21.15.153:0/706205231":"2024-06-30T14:41:33.044584+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:50.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7efd040763f0 msgr2=0x7efd04078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7efd040763f0 0x7efd04078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efd14001be0 tx=0x7efd14006040 comp rx=0 tx=0).stop 2024-06-29T14:41:50.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd241f5640 msgr2=0x7efd240ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd241f5640 0x7efd240ccd40 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7efd0c036040 tx=0x7efd0c0336f0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 shutdown_connections 2024-06-29T14:41:50.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7efd040763f0 0x7efd04078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd241f5a50 0x7efd240cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd241f5640 0x7efd240ccd40 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 --2- 172.21.15.59:0/3022838412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd10004090 0x7efd241598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 >> 172.21.15.59:0/3022838412 conn(0x7efd240d3ee0 msgr2=0x7efd2406cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:50.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 shutdown_connections 2024-06-29T14:41:50.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.802+0000 7efd29768640 1 -- 172.21.15.59:0/3022838412 wait complete. 2024-06-29T14:41:50.811 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:50.811 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.934+0000 7f3c9473c640 1 -- 172.21.15.59:0/572851076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 msgr2=0x7f3c8c15a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.934+0000 7f3c9473c640 1 --2- 172.21.15.59:0/572851076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 0x7f3c8c15a400 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0b4720 tx=0x7f3c8402aa30 comp rx=0 tx=0).stop 2024-06-29T14:41:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.934+0000 7f3c9473c640 1 -- 172.21.15.59:0/572851076 shutdown_connections 2024-06-29T14:41:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.934+0000 7f3c9473c640 1 --2- 172.21.15.59:0/572851076 >> v2:172.21.15.153:3300/0 conn(0x7f3c8c0cf120 0x7f3c8c073500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.934+0000 7f3c9473c640 1 --2- 172.21.15.59:0/572851076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 0x7f3c8c15a400 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.934+0000 7f3c9473c640 1 -- 172.21.15.59:0/572851076 >> 172.21.15.59:0/572851076 conn(0x7f3c8c0b24c0 msgr2=0x7f3c8c0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.934+0000 7f3c9473c640 1 -- 172.21.15.59:0/572851076 shutdown_connections 2024-06-29T14:41:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.934+0000 7f3c9473c640 1 -- 172.21.15.59:0/572851076 wait complete. 2024-06-29T14:41:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.936+0000 7f3c9473c640 1 Processor -- start 2024-06-29T14:41:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.936+0000 7f3c9473c640 1 -- start start 2024-06-29T14:41:50.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c9473c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8c073a40 0x7f3c8c14f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c9473c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 0x7f3c8c0aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c9473c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c8c0cf120 0x7f3c8c15a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c9473c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0b3c00 con 0x7f3c8c073f30 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c9473c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0b3a80 con 0x7f3c8c0cf120 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c9473c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c0b3d80 con 0x7f3c8c073a40 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c91cd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 0x7f3c8c0aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c91cd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 0x7f3c8c0aed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51890/0 (socket says 172.21.15.59:51890) 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.937+0000 7f3c91cd7640 1 -- 172.21.15.59:0/2780343322 learned_addr learned my addr 172.21.15.59:0/2780343322 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c924d8640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8c073a40 0x7f3c8c14f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c92cd9640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c8c0cf120 0x7f3c8c15a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c91cd7640 1 -- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8c073a40 msgr2=0x7f3c8c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c91cd7640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8c073a40 0x7f3c8c14f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c91cd7640 1 -- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c8c0cf120 msgr2=0x7f3c8c15a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c91cd7640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c8c0cf120 0x7f3c8c15a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c91cd7640 1 -- 172.21.15.59:0/2780343322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c0af240 con 0x7f3c8c073f30 2024-06-29T14:41:50.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c91cd7640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 0x7f3c8c0aed00 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000c970 tx=0x7f3c8000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:50.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c8affd640 1 -- 172.21.15.59:0/2780343322 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80007bb0 con 0x7f3c8c073f30 2024-06-29T14:41:50.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c8affd640 1 -- 172.21.15.59:0/2780343322 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c80007e40 con 0x7f3c8c073f30 2024-06-29T14:41:50.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c8affd640 1 -- 172.21.15.59:0/2780343322 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8000f690 con 0x7f3c8c073f30 2024-06-29T14:41:50.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8c1f5900 con 0x7f3c8c073f30 2024-06-29T14:41:50.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.938+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c8c0ae500 con 0x7f3c8c073f30 2024-06-29T14:41:50.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.939+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c8c0b4250 con 0x7f3c8c073f30 2024-06-29T14:41:50.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.940+0000 7f3c8affd640 1 -- 172.21.15.59:0/2780343322 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f3c8000f830 con 0x7f3c8c073f30 2024-06-29T14:41:50.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.941+0000 7f3c8affd640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3c78076420 0x7f3c78078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.941+0000 7f3c8affd640 1 -- 172.21.15.59:0/2780343322 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7f3c800981f0 con 0x7f3c8c073f30 2024-06-29T14:41:50.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.941+0000 7f3c924d8640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3c78076420 0x7f3c78078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.941+0000 7f3c924d8640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3c78076420 0x7f3c78078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3c84035000 tx=0x7f3c8403b040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:50.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:50.945+0000 7f3c8affd640 1 -- 172.21.15.59:0/2780343322 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3c80014030 con 0x7f3c8c073f30 2024-06-29T14:41:51.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.134+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3c8c1f5650 con 0x7f3c8c073f30 2024-06-29T14:41:51.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.134+0000 7f3c8affd640 1 -- 172.21.15.59:0/2780343322 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) ==== 93+0+10 (secure 0 0 0) 0x7f3c800610d0 con 0x7f3c8c073f30 2024-06-29T14:41:51.132 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:41:51.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3c78076420 msgr2=0x7f3c78078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3c78076420 0x7f3c78078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3c84035000 tx=0x7f3c8403b040 comp rx=0 tx=0).stop 2024-06-29T14:41:51.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 msgr2=0x7f3c8c0aed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 0x7f3c8c0aed00 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000c970 tx=0x7f3c8000ce30 comp rx=0 tx=0).stop 2024-06-29T14:41:51.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 shutdown_connections 2024-06-29T14:41:51.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3c78076420 0x7f3c78078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c8c0cf120 0x7f3c8c15a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c073f30 0x7f3c8c0aed00 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 --2- 172.21.15.59:0/2780343322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c8c073a40 0x7f3c8c14f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 >> 172.21.15.59:0/2780343322 conn(0x7f3c8c0b24c0 msgr2=0x7f3c8c06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:51.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 shutdown_connections 2024-06-29T14:41:51.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.136+0000 7f3c9473c640 1 -- 172.21.15.59:0/2780343322 wait complete. 2024-06-29T14:41:51.144 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-06-29T14:41:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.265+0000 7fe9472d6640 1 -- 172.21.15.59:0/1383861888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c005350 msgr2=0x7fe92c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.265+0000 7fe9472d6640 1 --2- 172.21.15.59:0/1383861888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c005350 0x7fe92c00b9c0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fe9400cfa80 tx=0x7fe92c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.265+0000 7fe9472d6640 1 -- 172.21.15.59:0/1383861888 shutdown_connections 2024-06-29T14:41:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.265+0000 7fe9472d6640 1 --2- 172.21.15.59:0/1383861888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe92c005350 0x7fe92c00b9c0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.265+0000 7fe9472d6640 1 -- 172.21.15.59:0/1383861888 >> 172.21.15.59:0/1383861888 conn(0x7fe9400b24c0 msgr2=0x7fe9400cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.265+0000 7fe9472d6640 1 -- 172.21.15.59:0/1383861888 shutdown_connections 2024-06-29T14:41:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.265+0000 7fe9472d6640 1 -- 172.21.15.59:0/1383861888 wait complete. 2024-06-29T14:41:51.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.267+0000 7fe9472d6640 1 Processor -- start 2024-06-29T14:41:51.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.267+0000 7fe9472d6640 1 -- start start 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.268+0000 7fe9472d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92c005350 0x7fe9400ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.268+0000 7fe9472d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9401594f0 0x7fe94015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.268+0000 7fe9472d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe940159900 0x7fe94015c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.268+0000 7fe9472d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe94015f5d0 con 0x7fe9401594f0 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.268+0000 7fe9472d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe94015f450 con 0x7fe92c005350 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.268+0000 7fe9472d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe94015f750 con 0x7fe940159900 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9401594f0 0x7fe94015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9401594f0 0x7fe94015c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51918/0 (socket says 172.21.15.59:51918) 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 -- 172.21.15.59:0/2528001602 learned_addr learned my addr 172.21.15.59:0/2528001602 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:51.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe945873640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe940159900 0x7fe94015c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe945072640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92c005350 0x7fe9400ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 -- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe940159900 msgr2=0x7fe94015c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe940159900 0x7fe94015c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 -- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92c005350 msgr2=0x7fe9400ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92c005350 0x7fe9400ae2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 -- 172.21.15.59:0/2528001602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe940140a10 con 0x7fe9401594f0 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe944871640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9401594f0 0x7fe94015c3f0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe93000e570 tx=0x7fe93000ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe935ffb640 1 -- 172.21.15.59:0/2528001602 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe930015070 con 0x7fe9401594f0 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe935ffb640 1 -- 172.21.15.59:0/2528001602 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe93000c9e0 con 0x7fe9401594f0 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe935ffb640 1 -- 172.21.15.59:0/2528001602 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93000cdf0 con 0x7fe9401594f0 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.269+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9401e47f0 con 0x7fe9401594f0 2024-06-29T14:41:51.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.270+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe94013c2f0 con 0x7fe9401594f0 2024-06-29T14:41:51.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.271+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe94006d900 con 0x7fe9401594f0 2024-06-29T14:41:51.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.271+0000 7fe935ffb640 1 -- 172.21.15.59:0/2528001602 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fe930018070 con 0x7fe9401594f0 2024-06-29T14:41:51.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.272+0000 7fe935ffb640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe9240763f0 0x7fe924078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:51.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.272+0000 7fe935ffb640 1 -- 172.21.15.59:0/2528001602 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7fe930097e50 con 0x7fe9401594f0 2024-06-29T14:41:51.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.272+0000 7fe945072640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe9240763f0 0x7fe924078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:51.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.273+0000 7fe945072640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe9240763f0 0x7fe924078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009a10 tx=0x7fe93c005f90 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:51.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.276+0000 7fe935ffb640 1 -- 172.21.15.59:0/2528001602 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe94006d900 con 0x7fe9401594f0 2024-06-29T14:41:51.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:51.545+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) -- 0x7fe94015b680 con 0x7fe9401594f0 2024-06-29T14:41:52.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.464+0000 7fe935ffb640 1 -- 172.21.15.59:0/2528001602 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v783) ==== 181+0+0 (secure 0 0 0) 0x7fe930061410 con 0x7fe9401594f0 2024-06-29T14:41:52.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe9240763f0 msgr2=0x7fe924078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe9240763f0 0x7fe924078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009a10 tx=0x7fe93c005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:52.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9401594f0 msgr2=0x7fe94015c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9401594f0 0x7fe94015c3f0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe93000e570 tx=0x7fe93000ea30 comp rx=0 tx=0).stop 2024-06-29T14:41:52.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 shutdown_connections 2024-06-29T14:41:52.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe9240763f0 0x7fe924078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe940159900 0x7fe94015c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9401594f0 0x7fe94015c3f0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 --2- 172.21.15.59:0/2528001602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe92c005350 0x7fe9400ae2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 >> 172.21.15.59:0/2528001602 conn(0x7fe9400b24c0 msgr2=0x7fe94015de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:52.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 shutdown_connections 2024-06-29T14:41:52.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.467+0000 7fe9472d6640 1 -- 172.21.15.59:0/2528001602 wait complete. 2024-06-29T14:41:52.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:41:52.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.605+0000 7fe408994640 1 -- 172.21.15.59:0/1619272426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040d23c0 msgr2=0x7fe4040b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.605+0000 7fe408994640 1 --2- 172.21.15.59:0/1619272426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040d23c0 0x7fe4040b3990 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009830 tx=0x7fe3f402a510 comp rx=0 tx=0).stop 2024-06-29T14:41:52.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.605+0000 7fe408994640 1 -- 172.21.15.59:0/1619272426 shutdown_connections 2024-06-29T14:41:52.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.605+0000 7fe408994640 1 --2- 172.21.15.59:0/1619272426 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe40415efe0 0x7fe40415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.605+0000 7fe408994640 1 --2- 172.21.15.59:0/1619272426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040d23c0 0x7fe4040b3990 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.605+0000 7fe408994640 1 -- 172.21.15.59:0/1619272426 >> 172.21.15.59:0/1619272426 conn(0x7fe4040d3ee0 msgr2=0x7fe4040d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:52.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.606+0000 7fe408994640 1 -- 172.21.15.59:0/1619272426 shutdown_connections 2024-06-29T14:41:52.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.606+0000 7fe408994640 1 -- 172.21.15.59:0/1619272426 wait complete. 2024-06-29T14:41:52.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.607+0000 7fe408994640 1 Processor -- start 2024-06-29T14:41:52.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.608+0000 7fe408994640 1 -- start start 2024-06-29T14:41:52.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe408994640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040b3ed0 0x7fe4041e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe408994640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4040d23c0 0x7fe404151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe408994640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40415efe0 0x7fe404152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe408994640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe404160280 con 0x7fe4040b3ed0 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe408994640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe404160100 con 0x7fe40415efe0 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe408994640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe404160400 con 0x7fe4040d23c0 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe40259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040b3ed0 0x7fe4041e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe40259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040b3ed0 0x7fe4041e6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51946/0 (socket says 172.21.15.59:51946) 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe40259c640 1 -- 172.21.15.59:0/1413156886 learned_addr learned my addr 172.21.15.59:0/1413156886 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe3fbfff640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4040d23c0 0x7fe404151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe402d9d640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40415efe0 0x7fe404152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe40259c640 1 -- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4040d23c0 msgr2=0x7fe404151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe40259c640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4040d23c0 0x7fe404151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe40259c640 1 -- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40415efe0 msgr2=0x7fe404152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe40259c640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40415efe0 0x7fe404152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.609+0000 7fe40259c640 1 -- 172.21.15.59:0/1413156886 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe404144290 con 0x7fe4040b3ed0 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.610+0000 7fe40259c640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040b3ed0 0x7fe4041e6ba0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4000c00 tx=0x7fe3f4033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:52.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.610+0000 7fe3faffd640 1 -- 172.21.15.59:0/1413156886 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4005c90 con 0x7fe4040b3ed0 2024-06-29T14:41:52.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.610+0000 7fe3faffd640 1 -- 172.21.15.59:0/1413156886 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3f4031e50 con 0x7fe4040b3ed0 2024-06-29T14:41:52.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.610+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4041e7780 con 0x7fe4040b3ed0 2024-06-29T14:41:52.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.610+0000 7fe3faffd640 1 -- 172.21.15.59:0/1413156886 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40332b0 con 0x7fe4040b3ed0 2024-06-29T14:41:52.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.610+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4041e7d40 con 0x7fe4040b3ed0 2024-06-29T14:41:52.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.611+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe404158890 con 0x7fe4040b3ed0 2024-06-29T14:41:52.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.612+0000 7fe3faffd640 1 -- 172.21.15.59:0/1413156886 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fe3f4033450 con 0x7fe4040b3ed0 2024-06-29T14:41:52.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.612+0000 7fe3faffd640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe3e40763f0 0x7fe3e4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.612+0000 7fe3faffd640 1 -- 172.21.15.59:0/1413156886 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7fe3f40b8e80 con 0x7fe4040b3ed0 2024-06-29T14:41:52.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.613+0000 7fe3fbfff640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe3e40763f0 0x7fe3e4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.613+0000 7fe3fbfff640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe3e40763f0 0x7fe3e4078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe404153ce0 tx=0x7fe3ec009040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:52.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.617+0000 7fe3faffd640 1 -- 172.21.15.59:0/1413156886 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe404158890 con 0x7fe4040b3ed0 2024-06-29T14:41:52.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.807+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fe4040b3990 con 0x7fe4040b3ed0 2024-06-29T14:41:52.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.808+0000 7fe3faffd640 1 -- 172.21.15.59:0/1413156886 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v55) ==== 59+0+1998 (secure 0 0 0) 0x7fe3f403f030 con 0x7fe4040b3ed0 2024-06-29T14:41:52.807 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:41:52.808 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:41:52.808 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:41:52.808 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:41:52.808 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T14:41:52.808 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T14:41:52.808 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T14:41:52.808 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T14:41:52.809 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:41:52.810 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:41:52.810 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T14:41:52.810 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:41:52.810 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:41:52.810 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:41:52.810 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T14:41:52.810 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T14:41:52.810 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T14:41:52.811 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:41:52.811 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:41:52.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.809+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe3e40763f0 msgr2=0x7fe3e4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.809+0000 7fe408994640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe3e40763f0 0x7fe3e4078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe404153ce0 tx=0x7fe3ec009040 comp rx=0 tx=0).stop 2024-06-29T14:41:52.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.809+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040b3ed0 msgr2=0x7fe4041e6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.809+0000 7fe408994640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040b3ed0 0x7fe4041e6ba0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4000c00 tx=0x7fe3f4033c90 comp rx=0 tx=0).stop 2024-06-29T14:41:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.810+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 shutdown_connections 2024-06-29T14:41:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.810+0000 7fe408994640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fe3e40763f0 0x7fe3e4078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.810+0000 7fe408994640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40415efe0 0x7fe404152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.810+0000 7fe408994640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4040d23c0 0x7fe404151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.810+0000 7fe408994640 1 --2- 172.21.15.59:0/1413156886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4040b3ed0 0x7fe4041e6ba0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.810+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 >> 172.21.15.59:0/1413156886 conn(0x7fe4040d3ee0 msgr2=0x7fe4040d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.810+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 shutdown_connections 2024-06-29T14:41:52.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.810+0000 7fe408994640 1 -- 172.21.15.59:0/1413156886 wait complete. 2024-06-29T14:41:52.818 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:41:52.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.942+0000 7f5ce0718640 1 -- 172.21.15.59:0/2406611962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cc0004800 msgr2=0x7f5cc0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.942+0000 7f5ce0718640 1 --2- 172.21.15.59:0/2406611962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cc0004800 0x7f5cc0004c10 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009b20 tx=0x7f5cc0005830 comp rx=0 tx=0).stop 2024-06-29T14:41:52.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.942+0000 7f5ce0718640 1 -- 172.21.15.59:0/2406611962 shutdown_connections 2024-06-29T14:41:52.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.942+0000 7f5ce0718640 1 --2- 172.21.15.59:0/2406611962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cc0004800 0x7f5cc0004c10 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.942+0000 7f5ce0718640 1 -- 172.21.15.59:0/2406611962 >> 172.21.15.59:0/2406611962 conn(0x7f5cd80b6fb0 msgr2=0x7f5cd80b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:52.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.942+0000 7f5ce0718640 1 -- 172.21.15.59:0/2406611962 shutdown_connections 2024-06-29T14:41:52.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.942+0000 7f5ce0718640 1 -- 172.21.15.59:0/2406611962 wait complete. 2024-06-29T14:41:52.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.943+0000 7f5ce0718640 1 Processor -- start 2024-06-29T14:41:52.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.944+0000 7f5ce0718640 1 -- start start 2024-06-29T14:41:52.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5ce0718640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cc0004800 0x7f5cd815ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5ce0718640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cd81f30b0 0x7f5cd81f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5ce0718640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cd80ce750 0x7f5cd80ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5ce0718640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd80b5850 con 0x7f5cd81f30b0 2024-06-29T14:41:52.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5ce0718640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd80b56d0 con 0x7f5cd80ce750 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5ce0718640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd80b59d0 con 0x7f5cc0004800 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cd81f30b0 0x7f5cd81f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cd81f30b0 0x7f5cd81f34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49352/0 (socket says 172.21.15.59:49352) 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 -- 172.21.15.59:0/1409165483 learned_addr learned my addr 172.21.15.59:0/1409165483 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cde4b4640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cc0004800 0x7f5cd815ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cdecb5640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cd80ce750 0x7f5cd80ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 -- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cc0004800 msgr2=0x7f5cd815ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cc0004800 0x7f5cd815ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 -- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cd80ce750 msgr2=0x7f5cd80ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cd80ce750 0x7f5cd80ceb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 -- 172.21.15.59:0/1409165483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd80cf0a0 con 0x7f5cd81f30b0 2024-06-29T14:41:52.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cdecb5640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cd80ce750 0x7f5cd80ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:52.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.945+0000 7f5cddcb3640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cd81f30b0 0x7f5cd81f34c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009b20 tx=0x7f5ccc0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:52.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.946+0000 7f5cd6ffd640 1 -- 172.21.15.59:0/1409165483 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc038040 con 0x7f5cd81f30b0 2024-06-29T14:41:52.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.946+0000 7f5cd6ffd640 1 -- 172.21.15.59:0/1409165483 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ccc043070 con 0x7f5cd81f30b0 2024-06-29T14:41:52.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.946+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd815b090 con 0x7f5cd81f30b0 2024-06-29T14:41:52.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.946+0000 7f5cd6ffd640 1 -- 172.21.15.59:0/1409165483 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc033c50 con 0x7f5cd81f30b0 2024-06-29T14:41:52.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.946+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd8159a50 con 0x7f5cd81f30b0 2024-06-29T14:41:52.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.947+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd81419b0 con 0x7f5cd81f30b0 2024-06-29T14:41:52.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.947+0000 7f5cd6ffd640 1 -- 172.21.15.59:0/1409165483 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f5ccc048460 con 0x7f5cd81f30b0 2024-06-29T14:41:52.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.948+0000 7f5cd6ffd640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5cc40763f0 0x7f5cc4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.948+0000 7f5cd6ffd640 1 -- 172.21.15.59:0/1409165483 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7f5ccc0b8c00 con 0x7f5cd81f30b0 2024-06-29T14:41:52.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.948+0000 7f5cde4b4640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5cc40763f0 0x7f5cc4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.949+0000 7f5cde4b4640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5cc40763f0 0x7f5cc4078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5cc80015a0 tx=0x7f5cc8006aa0 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:52.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:52.952+0000 7f5cd6ffd640 1 -- 172.21.15.59:0/1409165483 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5cd80cf0a0 con 0x7f5cd81f30b0 2024-06-29T14:41:53.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.146+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5cd8068b70 con 0x7f5cd81f30b0 2024-06-29T14:41:53.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.150+0000 7f5cd6ffd640 1 -- 172.21.15.59:0/1409165483 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v56)=0 v56) ==== 122+0+0 (secure 0 0 0) 0x7f5ccc081be0 con 0x7f5cd81f30b0 2024-06-29T14:41:53.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5cc40763f0 msgr2=0x7f5cc4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5cc40763f0 0x7f5cc4078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5cc80015a0 tx=0x7f5cc8006aa0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cd81f30b0 msgr2=0x7f5cd81f34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cd81f30b0 0x7f5cd81f34c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009b20 tx=0x7f5ccc0336f0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 shutdown_connections 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5cc40763f0 0x7f5cc4078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5cd80ce750 0x7f5cd80ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cd81f30b0 0x7f5cd81f34c0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 --2- 172.21.15.59:0/1409165483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cc0004800 0x7f5cd815ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 >> 172.21.15.59:0/1409165483 conn(0x7f5cd80b6fb0 msgr2=0x7f5cd815f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 shutdown_connections 2024-06-29T14:41:53.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.152+0000 7f5ce0718640 1 -- 172.21.15.59:0/1409165483 wait complete. 2024-06-29T14:41:53.161 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:41:53.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:53.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:53.246 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:41:53.247 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:53.247 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:53.264 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:53.264 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:41:53.264 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:53.308 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:53.309 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:41:53.309 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:53.326 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:53.326 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T14:41:53.326 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:53.367 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:53.367 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:41:53.367 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:53.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.488+0000 7f94b552a640 1 -- 172.21.15.59:0/3714384638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c004000 msgr2=0x7f949c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.488+0000 7f94b552a640 1 --2- 172.21.15.59:0/3714384638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c004000 0x7f949c004800 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f9498001700 tx=0x7f949c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:53.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.488+0000 7f94af7fe640 1 -- 172.21.15.59:0/3714384638 reap_dead start 2024-06-29T14:41:53.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.488+0000 7f94b552a640 1 -- 172.21.15.59:0/3714384638 shutdown_connections 2024-06-29T14:41:53.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.488+0000 7f94b552a640 1 -- 172.21.15.59:0/3714384638 >> 172.21.15.59:0/3714384638 conn(0x7f94b00b6fb0 msgr2=0x7f94b00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:53.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.488+0000 7f94b552a640 1 -- 172.21.15.59:0/3714384638 shutdown_connections 2024-06-29T14:41:53.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.488+0000 7f94b552a640 1 -- 172.21.15.59:0/3714384638 wait complete. 2024-06-29T14:41:53.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.490+0000 7f94b552a640 1 Processor -- start 2024-06-29T14:41:53.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.490+0000 7f94b552a640 1 -- start start 2024-06-29T14:41:53.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94b552a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f949c004000 0x7f94b013ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94b552a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94b014f960 0x7f94b0155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94b552a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0155870 0x7f94b01f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94b552a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f94b00b5850 con 0x7f94b0155870 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94b552a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94b00b56d0 con 0x7f949c004000 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94b552a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f94b00b59d0 con 0x7f94b014f960 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0155870 0x7f94b01f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0155870 0x7f94b01f57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49372/0 (socket says 172.21.15.59:49372) 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 -- 172.21.15.59:0/1805203357 learned_addr learned my addr 172.21.15.59:0/1805203357 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94ae7fc640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94b014f960 0x7f94b0155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94aeffd640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f949c004000 0x7f94b013ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 -- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94b014f960 msgr2=0x7f94b0155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94b014f960 0x7f94b0155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 -- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f949c004000 msgr2=0x7f94b013ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f949c004000 0x7f94b013ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 -- 172.21.15.59:0/1805203357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b0155c80 con 0x7f94b0155870 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94ae7fc640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94b014f960 0x7f94b0155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94af7fe640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0155870 0x7f94b01f57c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f94b00d1940 tx=0x7f94a80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.492+0000 7f94a77fe640 1 -- 172.21.15.59:0/1805203357 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8011630 con 0x7f94b0155870 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.493+0000 7f94a77fe640 1 -- 172.21.15.59:0/1805203357 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94a80117d0 con 0x7f94b0155870 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.493+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b0157230 con 0x7f94b0155870 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.493+0000 7f94a77fe640 1 -- 172.21.15.59:0/1805203357 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8011c20 con 0x7f94b0155870 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.493+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b01577f0 con 0x7f94b0155870 2024-06-29T14:41:53.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.494+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b0156fa0 con 0x7f94b0155870 2024-06-29T14:41:53.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.494+0000 7f94a77fe640 1 -- 172.21.15.59:0/1805203357 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f94a801d470 con 0x7f94b0155870 2024-06-29T14:41:53.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.495+0000 7f94a77fe640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f94900763f0 0x7f9490078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.495+0000 7f94a77fe640 1 -- 172.21.15.59:0/1805203357 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7f94a80983f0 con 0x7f94b0155870 2024-06-29T14:41:53.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.495+0000 7f94aeffd640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f94900763f0 0x7f9490078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.496+0000 7f94aeffd640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f94900763f0 0x7f9490078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f94a0005fd0 tx=0x7f94a0009500 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:53.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.499+0000 7f94a77fe640 1 -- 172.21.15.59:0/1805203357 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f94b0155c80 con 0x7f94b0155870 2024-06-29T14:41:53.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.729+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94b00b5bb0 con 0x7f94b0155870 2024-06-29T14:41:53.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.730+0000 7f94a77fe640 1 -- 172.21.15.59:0/1805203357 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+3917 (secure 0 0 0) 0x7f94a8061ac0 con 0x7f94b0155870 2024-06-29T14:41:53.727 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:53.727 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":74,"btime":"2024-06-29T14:41:47:439912+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7442,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/4194962668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":4194962668},{"type":"v1","addr":"172.21.15.153:6836","nonce":4194962668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7445,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1840307553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1840307553},{"type":"v1","addr":"172.21.15.153:6837","nonce":1840307553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1010388227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1010388227},{"type":"v1","addr":"172.21.15.59:6834","nonce":1010388227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":9288,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1018860728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1018860728},{"type":"v1","addr":"172.21.15.59:6835","nonce":1018860728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[]} 2024-06-29T14:41:53.727 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 74 2024-06-29T14:41:53.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f94900763f0 msgr2=0x7f9490078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f94900763f0 0x7f9490078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f94a0005fd0 tx=0x7f94a0009500 comp rx=0 tx=0).stop 2024-06-29T14:41:53.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0155870 msgr2=0x7f94b01f57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0155870 0x7f94b01f57c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f94b00d1940 tx=0x7f94a80035f0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 shutdown_connections 2024-06-29T14:41:53.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f94900763f0 0x7f9490078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94b0155870 0x7f94b01f57c0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94b014f960 0x7f94b0155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 --2- 172.21.15.59:0/1805203357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f949c004000 0x7f94b013ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 >> 172.21.15.59:0/1805203357 conn(0x7f94b00b6fb0 msgr2=0x7f94b015e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:53.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 shutdown_connections 2024-06-29T14:41:53.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.732+0000 7f94b552a640 1 -- 172.21.15.59:0/1805203357 wait complete. 2024-06-29T14:41:53.740 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:41:53.740 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:41:53.741 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:41:53.741 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:53.741 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:41:53.741 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:41:53.741 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:41:53.741 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:53.742 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:41:53.742 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:41:53.742 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:41:53.742 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:53.742 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:41:53.742 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:41:53.742 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:41:53.743 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:53.751 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:41:53.751 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:41:53.753 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:41:53.753 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:41:53.754 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:41:53.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:41:53.755 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:41:53.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:41:53.795 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:41:53.795 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:53.796 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:41:53.796 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:53.797 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:41:53.797 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:53.798 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:41:53.798 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:53.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:41:53.884 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T14:41:53.884 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T14:41:53.890 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T14:41:53.891 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T14:41:53.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.926+0000 7f8fe44c0640 1 -- 172.21.15.59:0/3886873230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0b40a0 msgr2=0x7f8fdc15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.926+0000 7f8fe44c0640 1 --2- 172.21.15.59:0/3886873230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0b40a0 0x7f8fdc15e8d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009860 tx=0x7f8fcc02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.926+0000 7f8fe44c0640 1 -- 172.21.15.59:0/3886873230 shutdown_connections 2024-06-29T14:41:53.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.926+0000 7f8fe44c0640 1 --2- 172.21.15.59:0/3886873230 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc0b40a0 0x7f8fdc15e8d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.926+0000 7f8fe44c0640 1 -- 172.21.15.59:0/3886873230 >> 172.21.15.59:0/3886873230 conn(0x7f8fdc0d3ee0 msgr2=0x7f8fdc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:53.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.926+0000 7f8fe44c0640 1 -- 172.21.15.59:0/3886873230 shutdown_connections 2024-06-29T14:41:53.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.926+0000 7f8fe44c0640 1 -- 172.21.15.59:0/3886873230 wait complete. 2024-06-29T14:41:53.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.928+0000 7f8fe44c0640 1 Processor -- start 2024-06-29T14:41:53.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.928+0000 7f8fe44c0640 1 -- start start 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe44c0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc0d23c0 0x7f8fdc14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe44c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc15ee10 0x7f8fdc152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe44c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc15db40 0x7f8fdc1528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe44c0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc160450 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe44c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc1602d0 con 0x7f8fdc15ee10 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc0d23c0 0x7f8fdc14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe44c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8fdc1605d0 con 0x7f8fdc15db40 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc0d23c0 0x7f8fdc14ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49414/0 (socket says 172.21.15.59:49414) 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 -- 172.21.15.59:0/1990516476 learned_addr learned my addr 172.21.15.59:0/1990516476 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe2a5d640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc15db40 0x7f8fdc1528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe1a5b640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc15ee10 0x7f8fdc152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 -- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc15db40 msgr2=0x7f8fdc1528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc15db40 0x7f8fdc1528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 -- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc15ee10 msgr2=0x7f8fdc152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc15ee10 0x7f8fdc152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 -- 172.21.15.59:0/1990516476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fdc159870 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe1a5b640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc15ee10 0x7f8fdc152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.930+0000 7f8fe225c640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc0d23c0 0x7f8fdc14ff70 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800c400 tx=0x7f8fd800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.931+0000 7f8fd2ffd640 1 -- 172.21.15.59:0/1990516476 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8011850 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.931+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fdc1504b0 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.931+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fdc1592c0 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.931+0000 7f8fd2ffd640 1 -- 172.21.15.59:0/1990516476 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fd8017070 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.931+0000 7f8fd2ffd640 1 -- 172.21.15.59:0/1990516476 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8011850 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.932+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc4005290 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.933+0000 7f8fd2ffd640 1 -- 172.21.15.59:0/1990516476 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f8fd8011b20 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.933+0000 7f8fd2ffd640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f8fc0076820 0x7f8fc0078d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.933+0000 7f8fd2ffd640 1 -- 172.21.15.59:0/1990516476 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 6094+0+0 (secure 0 0 0) 0x7f8fd8011650 con 0x7f8fdc0d23c0 2024-06-29T14:41:53.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.933+0000 7f8fe1a5b640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f8fc0076820 0x7f8fc0078d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.934+0000 7f8fe1a5b640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f8fc0076820 0x7f8fc0078d30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc034000 tx=0x7f8fcc004730 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:53.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:53.938+0000 7f8fd2ffd640 1 -- 172.21.15.59:0/1990516476 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8fd8020020 con 0x7f8fdc0d23c0 2024-06-29T14:41:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.144+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f8fc4005850 con 0x7f8fdc0d23c0 2024-06-29T14:41:54.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.478+0000 7f8fd2ffd640 1 -- 172.21.15.59:0/1990516476 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v150) ==== 97+0+0 (secure 0 0 0) 0x7f8fd8063ee0 con 0x7f8fdc0d23c0 2024-06-29T14:41:54.475 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T14:41:54.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f8fc0076820 msgr2=0x7f8fc0078d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f8fc0076820 0x7f8fc0078d30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc034000 tx=0x7f8fcc004730 comp rx=0 tx=0).stop 2024-06-29T14:41:54.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc0d23c0 msgr2=0x7f8fdc14ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc0d23c0 0x7f8fdc14ff70 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800c400 tx=0x7f8fd800c8c0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 shutdown_connections 2024-06-29T14:41:54.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f8fc0076820 0x7f8fc0078d30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fdc15db40 0x7f8fdc1528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fdc15ee10 0x7f8fdc152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 --2- 172.21.15.59:0/1990516476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fdc0d23c0 0x7f8fdc14ff70 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 >> 172.21.15.59:0/1990516476 conn(0x7f8fdc0d3ee0 msgr2=0x7f8fdc160940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:54.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 shutdown_connections 2024-06-29T14:41:54.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.481+0000 7f8fe44c0640 1 -- 172.21.15.59:0/1990516476 wait complete. 2024-06-29T14:41:54.493 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:41:54.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.616+0000 7ff08e3be640 1 -- 172.21.15.59:0/295361572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff088162770 msgr2=0x7ff0880ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.616+0000 7ff08e3be640 1 --2- 172.21.15.59:0/295361572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff088162770 0x7ff0880ce550 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff078009830 tx=0x7ff07802a410 comp rx=0 tx=0).stop 2024-06-29T14:41:54.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.616+0000 7ff08e3be640 1 -- 172.21.15.59:0/295361572 shutdown_connections 2024-06-29T14:41:54.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.616+0000 7ff08e3be640 1 --2- 172.21.15.59:0/295361572 >> v2:172.21.15.153:3300/0 conn(0x7ff08815e0a0 0x7ff088161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.616+0000 7ff08e3be640 1 --2- 172.21.15.59:0/295361572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff088162770 0x7ff0880ce550 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.616+0000 7ff08e3be640 1 -- 172.21.15.59:0/295361572 >> 172.21.15.59:0/295361572 conn(0x7ff0880d45b0 msgr2=0x7ff0880d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.616+0000 7ff08e3be640 1 -- 172.21.15.59:0/295361572 shutdown_connections 2024-06-29T14:41:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.616+0000 7ff08e3be640 1 -- 172.21.15.59:0/295361572 wait complete. 2024-06-29T14:41:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.617+0000 7ff08e3be640 1 Processor -- start 2024-06-29T14:41:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.617+0000 7ff08e3be640 1 -- start start 2024-06-29T14:41:54.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff08e3be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08815e0a0 0x7ff08814d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:54.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff08e3be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff088161fc0 0x7ff08814f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:54.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff08e3be640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff088162770 0x7ff08815f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:54.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff08e3be640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff0880d2d80 con 0x7ff088161fc0 2024-06-29T14:41:54.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff08e3be640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0880b4b00 con 0x7ff08815e0a0 2024-06-29T14:41:54.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff08e3be640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff0880d2f00 con 0x7ff088162770 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff088161fc0 0x7ff08814f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff088161fc0 0x7ff08814f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49428/0 (socket says 172.21.15.59:49428) 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 -- 172.21.15.59:0/2617864788 learned_addr learned my addr 172.21.15.59:0/2617864788 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff087fff640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08815e0a0 0x7ff08814d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff08c95b640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff088162770 0x7ff08815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 -- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff088162770 msgr2=0x7ff08815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff088162770 0x7ff08815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 -- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08815e0a0 msgr2=0x7ff08814d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08815e0a0 0x7ff08814d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 -- 172.21.15.59:0/2617864788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08814d830 con 0x7ff088161fc0 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.619+0000 7ff07ffff640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff088161fc0 0x7ff08814f560 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7ff07000e9f0 tx=0x7ff07000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.620+0000 7ff0857fa640 1 -- 172.21.15.59:0/2617864788 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07000cd80 con 0x7ff088161fc0 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.620+0000 7ff0857fa640 1 -- 172.21.15.59:0/2617864788 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff070004020 con 0x7ff088161fc0 2024-06-29T14:41:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.620+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08814dd40 con 0x7ff088161fc0 2024-06-29T14:41:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.620+0000 7ff0857fa640 1 -- 172.21.15.59:0/2617864788 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070004450 con 0x7ff088161fc0 2024-06-29T14:41:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.620+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff08814e300 con 0x7ff088161fc0 2024-06-29T14:41:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.621+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff08814dab0 con 0x7ff088161fc0 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.621+0000 7ff0857fa640 1 -- 172.21.15.59:0/2617864788 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7ff0700035f0 con 0x7ff088161fc0 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.622+0000 7ff0857fa640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff068076420 0x7ff068078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.622+0000 7ff0857fa640 1 -- 172.21.15.59:0/2617864788 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5277+0+0 (secure 0 0 0) 0x7ff0700980c0 con 0x7ff088161fc0 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.622+0000 7ff087fff640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff068076420 0x7ff068078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.623+0000 7ff087fff640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff068076420 0x7ff068078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff078034000 tx=0x7ff078004730 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:54.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.626+0000 7ff0857fa640 1 -- 172.21.15.59:0/2617864788 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff070061920 con 0x7ff088161fc0 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.891+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ff088068330 con 0x7ff088161fc0 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.891+0000 7ff0857fa640 1 -- 172.21.15.59:0/2617864788 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) ==== 80+0+5304 (secure 0 0 0) 0x7ff070061220 con 0x7ff088161fc0 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.892 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.894 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T14:41:54.895 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.896 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T14:41:54.897 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:41:54.898 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.899 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:41:54.900 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T14:41:54.901 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:41:54.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff068076420 msgr2=0x7ff068078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff068076420 0x7ff068078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff078034000 tx=0x7ff078004730 comp rx=0 tx=0).stop 2024-06-29T14:41:54.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff088161fc0 msgr2=0x7ff08814f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff088161fc0 0x7ff08814f560 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7ff07000e9f0 tx=0x7ff07000eeb0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 shutdown_connections 2024-06-29T14:41:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff068076420 0x7ff068078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff088162770 0x7ff08815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff088161fc0 0x7ff08814f560 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 --2- 172.21.15.59:0/2617864788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff08815e0a0 0x7ff08814d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 >> 172.21.15.59:0/2617864788 conn(0x7ff0880d45b0 msgr2=0x7ff0880b76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 shutdown_connections 2024-06-29T14:41:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:54.893+0000 7ff08e3be640 1 -- 172.21.15.59:0/2617864788 wait complete. 2024-06-29T14:41:54.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.025+0000 7f5792da1640 1 -- 172.21.15.59:0/4143415357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 msgr2=0x7f578c073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.025+0000 7f5792da1640 1 --2- 172.21.15.59:0/4143415357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 0x7f578c073be0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f578c0705a0 tx=0x7f577c02a530 comp rx=0 tx=0).stop 2024-06-29T14:41:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.025+0000 7f5792da1640 1 -- 172.21.15.59:0/4143415357 shutdown_connections 2024-06-29T14:41:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.025+0000 7f5792da1640 1 --2- 172.21.15.59:0/4143415357 >> v2:172.21.15.153:3300/0 conn(0x7f578c0cc060 0x7f578c0acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.025+0000 7f5792da1640 1 --2- 172.21.15.59:0/4143415357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 0x7f578c073be0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.025+0000 7f5792da1640 1 -- 172.21.15.59:0/4143415357 >> 172.21.15.59:0/4143415357 conn(0x7f578c0b1ea0 msgr2=0x7f578c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.025+0000 7f5792da1640 1 -- 172.21.15.59:0/4143415357 shutdown_connections 2024-06-29T14:41:55.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.025+0000 7f5792da1640 1 -- 172.21.15.59:0/4143415357 wait complete. 2024-06-29T14:41:55.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.027+0000 7f5792da1640 1 Processor -- start 2024-06-29T14:41:55.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.027+0000 7f5792da1640 1 -- start start 2024-06-29T14:41:55.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f5792da1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f578c0ad4e0 0x7f578c09c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f5792da1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 0x7f578c1efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f5792da1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f578c0cc060 0x7f578c156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f5792da1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f578c0ce750 con 0x7f578c0cb490 2024-06-29T14:41:55.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f5792da1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f578c074230 con 0x7f578c0ad4e0 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f5792da1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f578c0ce8d0 con 0x7f578c0cc060 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 0x7f578c1efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 0x7f578c1efda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49438/0 (socket says 172.21.15.59:49438) 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 -- 172.21.15.59:0/3225896431 learned_addr learned my addr 172.21.15.59:0/3225896431 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f5790b3d640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f578c0ad4e0 0x7f578c09c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f579133e640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f578c0cc060 0x7f578c156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 -- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f578c0cc060 msgr2=0x7f578c156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f578c0cc060 0x7f578c156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 -- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f578c0ad4e0 msgr2=0x7f578c09c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f578c0ad4e0 0x7f578c09c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 -- 172.21.15.59:0/3225896431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578c09d030 con 0x7f578c0cb490 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f578bfff640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 0x7f578c1efda0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f578000c9a0 tx=0x7f578000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f57897fa640 1 -- 172.21.15.59:0/3225896431 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780007bb0 con 0x7f578c0cb490 2024-06-29T14:41:55.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f57897fa640 1 -- 172.21.15.59:0/3225896431 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5780007e40 con 0x7f578c0cb490 2024-06-29T14:41:55.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.029+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f578c1f2bc0 con 0x7f578c0cb490 2024-06-29T14:41:55.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.030+0000 7f57897fa640 1 -- 172.21.15.59:0/3225896431 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780005280 con 0x7f578c0cb490 2024-06-29T14:41:55.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.030+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f578c1f3180 con 0x7f578c0cb490 2024-06-29T14:41:55.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.031+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f578c09cda0 con 0x7f578c0cb490 2024-06-29T14:41:55.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.031+0000 7f57897fa640 1 -- 172.21.15.59:0/3225896431 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f5780005420 con 0x7f578c0cb490 2024-06-29T14:41:55.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.032+0000 7f57897fa640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f57780763f0 0x7f5778078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.032+0000 7f57897fa640 1 -- 172.21.15.59:0/3225896431 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5277+0+0 (secure 0 0 0) 0x7f5780097ce0 con 0x7f578c0cb490 2024-06-29T14:41:55.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.032+0000 7f5790b3d640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f57780763f0 0x7f5778078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.033+0000 7f5790b3d640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f57780763f0 0x7f5778078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f577c034000 tx=0x7f577c004730 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.036+0000 7f57897fa640 1 -- 172.21.15.59:0/3225896431 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5780014030 con 0x7f578c0cb490 2024-06-29T14:41:55.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.230+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f578c1f29b0 con 0x7f578c0cb490 2024-06-29T14:41:55.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.231+0000 7f57897fa640 1 -- 172.21.15.59:0/3225896431 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+13230 (secure 0 0 0) 0x7f5780061610 con 0x7f578c0cb490 2024-06-29T14:41:55.228 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:55.228 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":152,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:54.487875+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.233+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f57780763f0 msgr2=0x7f5778078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.233+0000 7f5792da1640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f57780763f0 0x7f5778078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f577c034000 tx=0x7f577c004730 comp rx=0 tx=0).stop 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.233+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 msgr2=0x7f578c1efda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.233+0000 7f5792da1640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 0x7f578c1efda0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f578000c9a0 tx=0x7f578000ce60 comp rx=0 tx=0).stop 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.234+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 shutdown_connections 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.234+0000 7f5792da1640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f57780763f0 0x7f5778078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.234+0000 7f5792da1640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f578c0cc060 0x7f578c156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.234+0000 7f5792da1640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0cb490 0x7f578c1efda0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.234+0000 7f5792da1640 1 --2- 172.21.15.59:0/3225896431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f578c0ad4e0 0x7f578c09c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.234+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 >> 172.21.15.59:0/3225896431 conn(0x7f578c0b1ea0 msgr2=0x7f578c0cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.234+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 shutdown_connections 2024-06-29T14:41:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.234+0000 7f5792da1640 1 -- 172.21.15.59:0/3225896431 wait complete. 2024-06-29T14:41:55.242 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:55.243 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.366+0000 7fbffc6fd640 1 -- 172.21.15.59:0/606149622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff406ea10 msgr2=0x7fbff40cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.366+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/606149622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff406ea10 0x7fbff40cf1d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009830 tx=0x7fbfec02a510 comp rx=0 tx=0).stop 2024-06-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.366+0000 7fbffc6fd640 1 -- 172.21.15.59:0/606149622 shutdown_connections 2024-06-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.366+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/606149622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff406ea10 0x7fbff40cf1d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.366+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/606149622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff4144470 0x7fbff40b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.366+0000 7fbffc6fd640 1 -- 172.21.15.59:0/606149622 >> 172.21.15.59:0/606149622 conn(0x7fbff40b06e0 msgr2=0x7fbff40b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.366+0000 7fbffc6fd640 1 -- 172.21.15.59:0/606149622 shutdown_connections 2024-06-29T14:41:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.366+0000 7fbffc6fd640 1 -- 172.21.15.59:0/606149622 wait complete. 2024-06-29T14:41:55.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.368+0000 7fbffc6fd640 1 Processor -- start 2024-06-29T14:41:55.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.368+0000 7fbffc6fd640 1 -- start start 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.369+0000 7fbffc6fd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff406ea10 0x7fbff40a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.369+0000 7fbffc6fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff40b2040 0x7fbff41f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.369+0000 7fbffc6fd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff4144470 0x7fbff415d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.369+0000 7fbffc6fd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbff40cfaa0 con 0x7fbff4144470 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.369+0000 7fbffc6fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbff40cf920 con 0x7fbff40b2040 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.369+0000 7fbffc6fd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbff40cfc20 con 0x7fbff406ea10 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.369+0000 7fbffac9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff4144470 0x7fbff415d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffac9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff4144470 0x7fbff415d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49450/0 (socket says 172.21.15.59:49450) 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffac9a640 1 -- 172.21.15.59:0/1924307849 learned_addr learned my addr 172.21.15.59:0/1924307849 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffa499640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff406ea10 0x7fbff40a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbff9c98640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff40b2040 0x7fbff41f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffac9a640 1 -- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff406ea10 msgr2=0x7fbff40a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffac9a640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff406ea10 0x7fbff40a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffac9a640 1 -- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff40b2040 msgr2=0x7fbff41f57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffac9a640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff40b2040 0x7fbff41f57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffac9a640 1 -- 172.21.15.59:0/1924307849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff41f5d30 con 0x7fbff4144470 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffac9a640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff4144470 0x7fbff415d7a0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fbfec02ab00 tx=0x7fbfec033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbff2ffd640 1 -- 172.21.15.59:0/1924307849 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec005c90 con 0x7fbff4144470 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbff2ffd640 1 -- 172.21.15.59:0/1924307849 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfec031e50 con 0x7fbff4144470 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff41f29d0 con 0x7fbff4144470 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbff2ffd640 1 -- 172.21.15.59:0/1924307849 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec033330 con 0x7fbff4144470 2024-06-29T14:41:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.370+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff40aa080 con 0x7fbff4144470 2024-06-29T14:41:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.371+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff40a9b10 con 0x7fbff4144470 2024-06-29T14:41:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.372+0000 7fbff2ffd640 1 -- 172.21.15.59:0/1924307849 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fbfec0334d0 con 0x7fbff4144470 2024-06-29T14:41:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.373+0000 7fbff2ffd640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbfe00763f0 0x7fbfe0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.373+0000 7fbff2ffd640 1 -- 172.21.15.59:0/1924307849 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5277+0+0 (secure 0 0 0) 0x7fbfec0b8b10 con 0x7fbff4144470 2024-06-29T14:41:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.373+0000 7fbffa499640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbfe00763f0 0x7fbfe0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.373+0000 7fbffa499640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbfe00763f0 0x7fbfe0078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009a60 tx=0x7fbfe4010040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.377+0000 7fbff2ffd640 1 -- 172.21.15.59:0/1924307849 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbff41f5d30 con 0x7fbff4144470 2024-06-29T14:41:55.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.569+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbff406a7c0 con 0x7fbff4144470 2024-06-29T14:41:55.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.569+0000 7fbff2ffd640 1 -- 172.21.15.59:0/1924307849 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v152) ==== 93+0+10 (secure 0 0 0) 0x7fbff406a7c0 con 0x7fbff4144470 2024-06-29T14:41:55.566 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T14:41:55.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbfe00763f0 msgr2=0x7fbfe0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbfe00763f0 0x7fbfe0078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009a60 tx=0x7fbfe4010040 comp rx=0 tx=0).stop 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff4144470 msgr2=0x7fbff415d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff4144470 0x7fbff415d7a0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fbfec02ab00 tx=0x7fbfec033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 shutdown_connections 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fbfe00763f0 0x7fbfe0078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff4144470 0x7fbff415d7a0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff40b2040 0x7fbff41f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 --2- 172.21.15.59:0/1924307849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff406ea10 0x7fbff40a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 >> 172.21.15.59:0/1924307849 conn(0x7fbff40b06e0 msgr2=0x7fbff415f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 shutdown_connections 2024-06-29T14:41:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.571+0000 7fbffc6fd640 1 -- 172.21.15.59:0/1924307849 wait complete. 2024-06-29T14:41:55.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:41:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.700+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1472308144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a40b3740 msgr2=0x7fc5a40b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.700+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1472308144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a40b3740 0x7fc5a40b3b50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc58c005fd0 tx=0x7fc58c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.700+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1472308144 shutdown_connections 2024-06-29T14:41:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.700+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1472308144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a40b3740 0x7fc5a40b3b50 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.700+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1472308144 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5a41458a0 0x7fc5a40cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.700+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1472308144 >> 172.21.15.59:0/1472308144 conn(0x7fc5a406c950 msgr2=0x7fc5a4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.700+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1472308144 shutdown_connections 2024-06-29T14:41:55.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.700+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1472308144 wait complete. 2024-06-29T14:41:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.702+0000 7fc5aa3be640 1 Processor -- start 2024-06-29T14:41:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.702+0000 7fc5aa3be640 1 -- start start 2024-06-29T14:41:55.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5aa3be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a40b3740 0x7fc5a40d2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5aa3be640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5a415c2f0 0x7fc5a40ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5aa3be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5a40d2670 0x7fc5a41340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5aa3be640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a41601a0 con 0x7fc5a40d2670 2024-06-29T14:41:55.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5aa3be640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a4160020 con 0x7fc5a40b3740 2024-06-29T14:41:55.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5aa3be640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a4160320 con 0x7fc5a415c2f0 2024-06-29T14:41:55.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5a40d2670 0x7fc5a41340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5a40d2670 0x7fc5a41340f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49462/0 (socket says 172.21.15.59:49462) 2024-06-29T14:41:55.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 -- 172.21.15.59:0/1587988506 learned_addr learned my addr 172.21.15.59:0/1587988506 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:55.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a3fff640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a40b3740 0x7fc5a40d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc59bfff640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5a415c2f0 0x7fc5a40ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 -- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5a415c2f0 msgr2=0x7fc5a40ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5a415c2f0 0x7fc5a40ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 -- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a40b3740 msgr2=0x7fc5a40d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a40b3740 0x7fc5a40d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 -- 172.21.15.59:0/1587988506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a40a9620 con 0x7fc5a40d2670 2024-06-29T14:41:55.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.704+0000 7fc5a895b640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5a40d2670 0x7fc5a41340f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fc59c00c730 tx=0x7fc59c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.705+0000 7fc5a17fa640 1 -- 172.21.15.59:0/1587988506 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c003b70 con 0x7fc5a40d2670 2024-06-29T14:41:55.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.705+0000 7fc5a17fa640 1 -- 172.21.15.59:0/1587988506 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc59c003d30 con 0x7fc5a40d2670 2024-06-29T14:41:55.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.705+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a40a9900 con 0x7fc5a40d2670 2024-06-29T14:41:55.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.705+0000 7fc5a17fa640 1 -- 172.21.15.59:0/1587988506 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c0052d0 con 0x7fc5a40d2670 2024-06-29T14:41:55.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.705+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a40a9ec0 con 0x7fc5a40d2670 2024-06-29T14:41:55.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.706+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a41458a0 con 0x7fc5a40d2670 2024-06-29T14:41:55.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.706+0000 7fc5a17fa640 1 -- 172.21.15.59:0/1587988506 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fc59c005680 con 0x7fc5a40d2670 2024-06-29T14:41:55.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.707+0000 7fc5a17fa640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5840763f0 0x7fc584078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.707+0000 7fc5a17fa640 1 -- 172.21.15.59:0/1587988506 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5277+0+0 (secure 0 0 0) 0x7fc59c0976e0 con 0x7fc5a40d2670 2024-06-29T14:41:55.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.707+0000 7fc5a3fff640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5840763f0 0x7fc584078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.708+0000 7fc5a3fff640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5840763f0 0x7fc584078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc5940095f0 tx=0x7fc594005f90 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.711+0000 7fc5a17fa640 1 -- 172.21.15.59:0/1587988506 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc59c060a10 con 0x7fc5a40d2670 2024-06-29T14:41:55.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.905+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc5a40b1f10 con 0x7fc5a40d2670 2024-06-29T14:41:55.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.905+0000 7fc5a17fa640 1 -- 172.21.15.59:0/1587988506 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v152) ==== 84+0+65 (secure 0 0 0) 0x7fc59c064d10 con 0x7fc5a40d2670 2024-06-29T14:41:55.902 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:55.903 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:41:55.903 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:41:55.903 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T14:41:55.903 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T14:41:55.903 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:41:55.903 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:41:55.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.907+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5840763f0 msgr2=0x7fc584078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.907+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5840763f0 0x7fc584078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc5940095f0 tx=0x7fc594005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:55.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5a40d2670 msgr2=0x7fc5a41340f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5a40d2670 0x7fc5a41340f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fc59c00c730 tx=0x7fc59c00cbf0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 shutdown_connections 2024-06-29T14:41:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5840763f0 0x7fc584078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5a40d2670 0x7fc5a41340f0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5a415c2f0 0x7fc5a40ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 --2- 172.21.15.59:0/1587988506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5a40b3740 0x7fc5a40d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 >> 172.21.15.59:0/1587988506 conn(0x7fc5a406c950 msgr2=0x7fc5a40706a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 shutdown_connections 2024-06-29T14:41:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:55.908+0000 7fc5aa3be640 1 -- 172.21.15.59:0/1587988506 wait complete. 2024-06-29T14:41:55.916 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:41:55.916 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:41:56.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.040+0000 7f1f456a4640 1 -- 172.21.15.59:0/2253826108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 msgr2=0x7f1f400b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.040+0000 7f1f456a4640 1 --2- 172.21.15.59:0/2253826108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 0x7f1f400b5cb0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f1f4015d310 tx=0x7f1f3002aa30 comp rx=0 tx=0).stop 2024-06-29T14:41:56.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.040+0000 7f1f456a4640 1 -- 172.21.15.59:0/2253826108 shutdown_connections 2024-06-29T14:41:56.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.040+0000 7f1f456a4640 1 --2- 172.21.15.59:0/2253826108 >> v2:172.21.15.153:3300/0 conn(0x7f1f4015ff00 0x7f1f4015d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.040+0000 7f1f456a4640 1 --2- 172.21.15.59:0/2253826108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 0x7f1f400b5cb0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.041+0000 7f1f456a4640 1 -- 172.21.15.59:0/2253826108 >> 172.21.15.59:0/2253826108 conn(0x7f1f400b6ed0 msgr2=0x7f1f400b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:56.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.041+0000 7f1f456a4640 1 -- 172.21.15.59:0/2253826108 shutdown_connections 2024-06-29T14:41:56.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.041+0000 7f1f456a4640 1 -- 172.21.15.59:0/2253826108 wait complete. 2024-06-29T14:41:56.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.042+0000 7f1f456a4640 1 Processor -- start 2024-06-29T14:41:56.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.042+0000 7f1f456a4640 1 -- start start 2024-06-29T14:41:56.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f456a4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f400d1130 0x7f1f401f5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f456a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 0x7f1f401401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f456a4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4006f960 0x7f1f400cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f456a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4015e2d0 con 0x7f1f40145650 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f456a4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4015e150 con 0x7f1f400d1130 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f456a4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1f4015e450 con 0x7f1f4006f960 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 0x7f1f401401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 0x7f1f401401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49470/0 (socket says 172.21.15.59:49470) 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 -- 172.21.15.59:0/1993816699 learned_addr learned my addr 172.21.15.59:0/1993816699 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3effd640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f400d1130 0x7f1f401f5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3f7fe640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4006f960 0x7f1f400cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 -- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4006f960 msgr2=0x7f1f400cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4006f960 0x7f1f400cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 -- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f400d1130 msgr2=0x7f1f401f5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f400d1130 0x7f1f401f5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 -- 172.21.15.59:0/1993816699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f400cbd80 con 0x7f1f40145650 2024-06-29T14:41:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3f7fe640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4006f960 0x7f1f400cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:56.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.044+0000 7f1f3e7fc640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 0x7f1f401401b0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800b420 tx=0x7f1f2800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:56.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.045+0000 7f1f377fe640 1 -- 172.21.15.59:0/1993816699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28011830 con 0x7f1f40145650 2024-06-29T14:41:56.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.045+0000 7f1f377fe640 1 -- 172.21.15.59:0/1993816699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f28016070 con 0x7f1f40145650 2024-06-29T14:41:56.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.045+0000 7f1f377fe640 1 -- 172.21.15.59:0/1993816699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28011d60 con 0x7f1f40145650 2024-06-29T14:41:56.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.045+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f400cc060 con 0x7f1f40145650 2024-06-29T14:41:56.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.045+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f400cc620 con 0x7f1f40145650 2024-06-29T14:41:56.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.046+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f4015e970 con 0x7f1f40145650 2024-06-29T14:41:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.046+0000 7f1f377fe640 1 -- 172.21.15.59:0/1993816699 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f1f2801b460 con 0x7f1f40145650 2024-06-29T14:41:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.047+0000 7f1f377fe640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1f200763f0 0x7f1f20078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.047+0000 7f1f377fe640 1 -- 172.21.15.59:0/1993816699 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5277+0+0 (secure 0 0 0) 0x7f1f280971f0 con 0x7f1f40145650 2024-06-29T14:41:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.047+0000 7f1f3effd640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1f200763f0 0x7f1f20078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.048+0000 7f1f3effd640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1f200763f0 0x7f1f20078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1f30035000 tx=0x7f1f3003b040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:56.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.051+0000 7f1f377fe640 1 -- 172.21.15.59:0/1993816699 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1f28020020 con 0x7f1f40145650 2024-06-29T14:41:56.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.245+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1f40070220 con 0x7f1f40145650 2024-06-29T14:41:56.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.487+0000 7f1f377fe640 1 -- 172.21.15.59:0/1993816699 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v153) ==== 138+0+0 (secure 0 0 0) 0x7f1f28060bf0 con 0x7f1f40145650 2024-06-29T14:41:56.485 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T14:41:56.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1f200763f0 msgr2=0x7f1f20078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1f200763f0 0x7f1f20078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1f30035000 tx=0x7f1f3003b040 comp rx=0 tx=0).stop 2024-06-29T14:41:56.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 msgr2=0x7f1f401401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 0x7f1f401401b0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800b420 tx=0x7f1f2800b8e0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 shutdown_connections 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1f200763f0 0x7f1f20078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4006f960 0x7f1f400cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f40145650 0x7f1f401401b0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 --2- 172.21.15.59:0/1993816699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f400d1130 0x7f1f401f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 >> 172.21.15.59:0/1993816699 conn(0x7f1f400b6ed0 msgr2=0x7f1f400b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 shutdown_connections 2024-06-29T14:41:56.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.490+0000 7f1f456a4640 1 -- 172.21.15.59:0/1993816699 wait complete. 2024-06-29T14:41:56.503 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:41:56.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.632+0000 7fa35d8a7640 1 -- 172.21.15.59:0/4205630990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581453d0 msgr2=0x7fa3580d0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.632+0000 7fa35d8a7640 1 --2- 172.21.15.59:0/4205630990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581453d0 0x7fa3580d0960 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fa350005fd0 tx=0x7fa35002a670 comp rx=0 tx=0).stop 2024-06-29T14:41:56.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.632+0000 7fa35d8a7640 1 -- 172.21.15.59:0/4205630990 shutdown_connections 2024-06-29T14:41:56.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.632+0000 7fa35d8a7640 1 --2- 172.21.15.59:0/4205630990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581453d0 0x7fa3580d0960 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.632+0000 7fa35d8a7640 1 -- 172.21.15.59:0/4205630990 >> 172.21.15.59:0/4205630990 conn(0x7fa358073ef0 msgr2=0x7fa358074300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:56.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.632+0000 7fa35d8a7640 1 -- 172.21.15.59:0/4205630990 shutdown_connections 2024-06-29T14:41:56.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.632+0000 7fa35d8a7640 1 -- 172.21.15.59:0/4205630990 wait complete. 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.634+0000 7fa35d8a7640 1 Processor -- start 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.634+0000 7fa35d8a7640 1 -- start start 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa35d8a7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3580ce320 0x7fa3581f8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa35d8a7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3581453d0 0x7fa35815b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa35d8a7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581f4670 0x7fa3580cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa35d8a7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa3580b53e0 con 0x7fa3581f4670 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa35d8a7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa3580b5260 con 0x7fa3580ce320 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa35d8a7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa3580b5560 con 0x7fa3581453d0 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa3577fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581f4670 0x7fa3580cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa3577fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581f4670 0x7fa3580cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49484/0 (socket says 172.21.15.59:49484) 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.635+0000 7fa3577fe640 1 -- 172.21.15.59:0/306406002 learned_addr learned my addr 172.21.15.59:0/306406002 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa3567fc640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3581453d0 0x7fa35815b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa356ffd640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3580ce320 0x7fa3581f8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa3577fe640 1 -- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3581453d0 msgr2=0x7fa35815b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa3577fe640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3581453d0 0x7fa35815b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa3577fe640 1 -- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3580ce320 msgr2=0x7fa3581f8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa3577fe640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3580ce320 0x7fa3581f8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa3577fe640 1 -- 172.21.15.59:0/306406002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3581f7c60 con 0x7fa3581f4670 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa3577fe640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581f4670 0x7fa3580cacc0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa350009230 tx=0x7fa350003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa34ffff640 1 -- 172.21.15.59:0/306406002 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35003e030 con 0x7fa3581f4670 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa34ffff640 1 -- 172.21.15.59:0/306406002 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa350003e00 con 0x7fa3581f4670 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa34ffff640 1 -- 172.21.15.59:0/306406002 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350035040 con 0x7fa3581f4670 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3581f51a0 con 0x7fa3581f4670 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.636+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3581f7f20 con 0x7fa3581f4670 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.637+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3580d0960 con 0x7fa3581f4670 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.638+0000 7fa34ffff640 1 -- 172.21.15.59:0/306406002 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fa350041030 con 0x7fa3581f4670 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.639+0000 7fa34ffff640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fa338076420 0x7fa338078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.639+0000 7fa34ffff640 1 -- 172.21.15.59:0/306406002 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5647+0+0 (secure 0 0 0) 0x7fa3500b8550 con 0x7fa3581f4670 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.639+0000 7fa356ffd640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fa338076420 0x7fa338078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.640+0000 7fa356ffd640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fa338076420 0x7fa338078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa3480012e0 tx=0x7fa348009040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:56.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.643+0000 7fa34ffff640 1 -- 172.21.15.59:0/306406002 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa350081680 con 0x7fa3581f4670 2024-06-29T14:41:56.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:56.836+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa3581f4f30 con 0x7fa3581f4670 2024-06-29T14:41:57.020 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:41:57.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:41:57.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.490+0000 7fa34ffff640 1 -- 172.21.15.59:0/306406002 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v154) ==== 170+0+0 (secure 0 0 0) 0x7fa350085980 con 0x7fa3581f4670 2024-06-29T14:41:57.487 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T14:41:57.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fa338076420 msgr2=0x7fa338078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fa338076420 0x7fa338078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa3480012e0 tx=0x7fa348009040 comp rx=0 tx=0).stop 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581f4670 msgr2=0x7fa3580cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581f4670 0x7fa3580cacc0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa350009230 tx=0x7fa350003910 comp rx=0 tx=0).stop 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 shutdown_connections 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fa338076420 0x7fa338078930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3581f4670 0x7fa3580cacc0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3581453d0 0x7fa35815b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 --2- 172.21.15.59:0/306406002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3580ce320 0x7fa3581f8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 >> 172.21.15.59:0/306406002 conn(0x7fa358073ef0 msgr2=0x7fa3580b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.493+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 shutdown_connections 2024-06-29T14:41:57.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.494+0000 7fa35d8a7640 1 -- 172.21.15.59:0/306406002 wait complete. 2024-06-29T14:41:57.517 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:41:57.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.646+0000 7fc3327bd640 1 -- 172.21.15.59:0/3796854995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc314003ac0 msgr2=0x7fc314003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.646+0000 7fc3327bd640 1 --2- 172.21.15.59:0/3796854995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc314003ac0 0x7fc314003f30 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00c260 tx=0x7fc314005830 comp rx=0 tx=0).stop 2024-06-29T14:41:57.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.646+0000 7fc330d5a640 1 -- 172.21.15.59:0/3796854995 reap_dead start 2024-06-29T14:41:57.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.646+0000 7fc3327bd640 1 -- 172.21.15.59:0/3796854995 shutdown_connections 2024-06-29T14:41:57.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.646+0000 7fc3327bd640 1 -- 172.21.15.59:0/3796854995 >> 172.21.15.59:0/3796854995 conn(0x7fc32c0d3ee0 msgr2=0x7fc32c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:57.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.646+0000 7fc3327bd640 1 -- 172.21.15.59:0/3796854995 shutdown_connections 2024-06-29T14:41:57.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.646+0000 7fc3327bd640 1 -- 172.21.15.59:0/3796854995 wait complete. 2024-06-29T14:41:57.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.648+0000 7fc3327bd640 1 Processor -- start 2024-06-29T14:41:57.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.648+0000 7fc3327bd640 1 -- start start 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.649+0000 7fc3327bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc314003ac0 0x7fc32c14ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.649+0000 7fc3327bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc32c142be0 0x7fc32c142ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.649+0000 7fc3327bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c0cc7d0 0x7fc32c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc3327bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c15fd40 con 0x7fc32c142be0 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc3327bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c15fbc0 con 0x7fc32c0cc7d0 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc3327bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc32c15fec0 con 0x7fc314003ac0 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc32c142be0 0x7fc32c142ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc32c142be0 0x7fc32c142ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49508/0 (socket says 172.21.15.59:49508) 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 -- 172.21.15.59:0/1723087172 learned_addr learned my addr 172.21.15.59:0/1723087172 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc330d5a640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c0cc7d0 0x7fc32c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32bfff640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc314003ac0 0x7fc32c14ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 -- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc314003ac0 msgr2=0x7fc32c14ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc314003ac0 0x7fc32c14ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 -- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c0cc7d0 msgr2=0x7fc32c0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c0cc7d0 0x7fc32c0ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 -- 172.21.15.59:0/1723087172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c150810 con 0x7fc32c142be0 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32bfff640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc314003ac0 0x7fc32c14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc32b7fe640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc32c142be0 0x7fc32c142ff0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc320011a50 tx=0x7fc320011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc328ff9640 1 -- 172.21.15.59:0/1723087172 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32000fcd0 con 0x7fc32c142be0 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc328ff9640 1 -- 172.21.15.59:0/1723087172 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc320004020 con 0x7fc32c142be0 2024-06-29T14:41:57.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c0cbcf0 con 0x7fc32c142be0 2024-06-29T14:41:57.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.650+0000 7fc328ff9640 1 -- 172.21.15.59:0/1723087172 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320004430 con 0x7fc32c142be0 2024-06-29T14:41:57.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.651+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c0cc2b0 con 0x7fc32c142be0 2024-06-29T14:41:57.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.651+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc32c06c940 con 0x7fc32c142be0 2024-06-29T14:41:57.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.652+0000 7fc328ff9640 1 -- 172.21.15.59:0/1723087172 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fc3200035f0 con 0x7fc32c142be0 2024-06-29T14:41:57.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.653+0000 7fc328ff9640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc3180763f0 0x7fc318078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:57.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.653+0000 7fc328ff9640 1 -- 172.21.15.59:0/1723087172 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 6029+0+0 (secure 0 0 0) 0x7fc320098e90 con 0x7fc32c142be0 2024-06-29T14:41:57.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.653+0000 7fc32bfff640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc3180763f0 0x7fc318078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:57.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.654+0000 7fc32bfff640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc3180763f0 0x7fc318078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc31c033000 tx=0x7fc31c02d000 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:57.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.657+0000 7fc328ff9640 1 -- 172.21.15.59:0/1723087172 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc32c06c940 con 0x7fc32c142be0 2024-06-29T14:41:57.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:57.890+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc32c1505a0 con 0x7fc32c142be0 2024-06-29T14:41:58.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.504+0000 7fc328ff9640 1 -- 172.21.15.59:0/1723087172 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v77) ==== 174+0+0 (secure 0 0 0) 0x7fc320062580 con 0x7fc32c142be0 2024-06-29T14:41:58.501 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 19 and data pool 20 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.507+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc3180763f0 msgr2=0x7fc318078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.507+0000 7fc3327bd640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc3180763f0 0x7fc318078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc31c033000 tx=0x7fc31c02d000 comp rx=0 tx=0).stop 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc32c142be0 msgr2=0x7fc32c142ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc32c142be0 0x7fc32c142ff0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc320011a50 tx=0x7fc320011e50 comp rx=0 tx=0).stop 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 shutdown_connections 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc3180763f0 0x7fc318078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc32c0cc7d0 0x7fc32c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc32c142be0 0x7fc32c142ff0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 --2- 172.21.15.59:0/1723087172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc314003ac0 0x7fc32c14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 >> 172.21.15.59:0/1723087172 conn(0x7fc32c0d3ee0 msgr2=0x7fc32c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:58.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 shutdown_connections 2024-06-29T14:41:58.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.508+0000 7fc3327bd640 1 -- 172.21.15.59:0/1723087172 wait complete. 2024-06-29T14:41:58.530 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:58.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.658+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3999131920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47815efe0 msgr2=0x7ff47815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.658+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3999131920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47815efe0 0x7ff47815f9b0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7ff4780d15d0 tx=0x7ff47002aac0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.658+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3999131920 shutdown_connections 2024-06-29T14:41:58.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.658+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3999131920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47815efe0 0x7ff47815f9b0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.658+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3999131920 >> v2:172.21.15.153:3300/0 conn(0x7ff4780d23c0 0x7ff4780b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.658+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3999131920 >> 172.21.15.59:0/3999131920 conn(0x7ff4780d3ee0 msgr2=0x7ff4780d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:58.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.659+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3999131920 shutdown_connections 2024-06-29T14:41:58.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.659+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3999131920 wait complete. 2024-06-29T14:41:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.660+0000 7ff47fbe7640 1 Processor -- start 2024-06-29T14:41:58.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.660+0000 7ff47fbe7640 1 -- start start 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47fbe7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4780b3ed0 0x7ff4780c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47fbe7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780d23c0 0x7ff47815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47fbe7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff47815efe0 0x7ff4781572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47fbe7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff478160280 con 0x7ff4780d23c0 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47fbe7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff478160100 con 0x7ff4780b3ed0 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47fbe7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff478160400 con 0x7ff47815efe0 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780d23c0 0x7ff47815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780d23c0 0x7ff47815dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49512/0 (socket says 172.21.15.59:49512) 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 -- 172.21.15.59:0/3980849638 learned_addr learned my addr 172.21.15.59:0/3980849638 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d983640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4780b3ed0 0x7ff4780c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47e184640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff47815efe0 0x7ff4781572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 -- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff47815efe0 msgr2=0x7ff4781572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff47815efe0 0x7ff4781572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 -- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4780b3ed0 msgr2=0x7ff4780c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4780b3ed0 0x7ff4780c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 -- 172.21.15.59:0/3980849638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff478159340 con 0x7ff4780d23c0 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47d182640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780d23c0 0x7ff47815dde0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff46800c400 tx=0x7ff46800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff46e7fc640 1 -- 172.21.15.59:0/3980849638 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468017070 con 0x7ff4780d23c0 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff46e7fc640 1 -- 172.21.15.59:0/3980849638 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4680053c0 con 0x7ff4780d23c0 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.662+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff478159620 con 0x7ff4780d23c0 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.663+0000 7ff46e7fc640 1 -- 172.21.15.59:0/3980849638 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff468003830 con 0x7ff4780d23c0 2024-06-29T14:41:58.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.663+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff478159be0 con 0x7ff4780d23c0 2024-06-29T14:41:58.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.664+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4781590b0 con 0x7ff4780d23c0 2024-06-29T14:41:58.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.664+0000 7ff46e7fc640 1 -- 172.21.15.59:0/3980849638 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7ff46801a070 con 0x7ff4780d23c0 2024-06-29T14:41:58.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.665+0000 7ff46e7fc640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff45c0763f0 0x7ff45c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.665+0000 7ff46e7fc640 1 -- 172.21.15.59:0/3980849638 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6129+0+0 (secure 0 0 0) 0x7ff468096d90 con 0x7ff4780d23c0 2024-06-29T14:41:58.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.665+0000 7ff47d983640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff45c0763f0 0x7ff45c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.666+0000 7ff47d983640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff45c0763f0 0x7ff45c078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4780d2e40 tx=0x7ff474005f90 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:58.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.669+0000 7ff46e7fc640 1 -- 172.21.15.59:0/3980849638 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff4780b42e0 con 0x7ff4780d23c0 2024-06-29T14:41:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.861+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4780b3990 con 0x7ff4780d23c0 2024-06-29T14:41:58.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.862+0000 7ff46e7fc640 1 -- 172.21.15.59:0/3980849638 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+16969 (secure 0 0 0) 0x7ff46805fe70 con 0x7ff4780d23c0 2024-06-29T14:41:58.859 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:58.859 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":155,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:57.891235+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.250305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.841623+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:58.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff45c0763f0 msgr2=0x7ff45c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff45c0763f0 0x7ff45c078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4780d2e40 tx=0x7ff474005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:58.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780d23c0 msgr2=0x7ff47815dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780d23c0 0x7ff47815dde0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff46800c400 tx=0x7ff46800c8c0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 shutdown_connections 2024-06-29T14:41:58.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7ff45c0763f0 0x7ff45c078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff47815efe0 0x7ff4781572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4780d23c0 0x7ff47815dde0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 --2- 172.21.15.59:0/3980849638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4780b3ed0 0x7ff4780c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 >> 172.21.15.59:0/3980849638 conn(0x7ff4780d3ee0 msgr2=0x7ff47806d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:58.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 shutdown_connections 2024-06-29T14:41:58.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.864+0000 7ff47fbe7640 1 -- 172.21.15.59:0/3980849638 wait complete. 2024-06-29T14:41:58.872 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.993+0000 7f5eeff91640 1 -- 172.21.15.59:0/1755062656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 msgr2=0x7f5ed0003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.993+0000 7f5eeff91640 1 --2- 172.21.15.59:0/1755062656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 0x7f5ed0003f30 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8006fd0 tx=0x7f5ed0005830 comp rx=0 tx=0).stop 2024-06-29T14:41:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.993+0000 7f5eeff91640 1 -- 172.21.15.59:0/1755062656 shutdown_connections 2024-06-29T14:41:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.993+0000 7f5eeff91640 1 --2- 172.21.15.59:0/1755062656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 0x7f5ed0003f30 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.993+0000 7f5eeff91640 1 -- 172.21.15.59:0/1755062656 >> 172.21.15.59:0/1755062656 conn(0x7f5ee806c950 msgr2=0x7f5ee8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.994+0000 7f5eeff91640 1 -- 172.21.15.59:0/1755062656 shutdown_connections 2024-06-29T14:41:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.994+0000 7f5eeff91640 1 -- 172.21.15.59:0/1755062656 wait complete. 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.995+0000 7f5eeff91640 1 Processor -- start 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.995+0000 7f5eeff91640 1 -- start start 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eeff91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 0x7f5ee815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eeff91640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ee81e99e0 0x7f5ee81602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eeff91640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ee81e9df0 0x7f5ee8160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eeff91640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee80cfc00 con 0x7f5ed0003ac0 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eeff91640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee80cfa80 con 0x7f5ee81e9df0 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eeff91640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee80cfd80 con 0x7f5ee81e99e0 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 0x7f5ee815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 0x7f5ee815dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49534/0 (socket says 172.21.15.59:49534) 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 -- 172.21.15.59:0/842962067 learned_addr learned my addr 172.21.15.59:0/842962067 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 -- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ee81e99e0 msgr2=0x7f5ee81602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ee81e99e0 0x7f5ee81602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 -- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ee81e9df0 msgr2=0x7f5ee8160800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ee81e9df0 0x7f5ee8160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 -- 172.21.15.59:0/842962067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee8141fa0 con 0x7f5ed0003ac0 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5eedd2d640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 0x7f5ee815dc00 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5ed802b470 tx=0x7f5ed8001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.997+0000 7f5ee67fc640 1 -- 172.21.15.59:0/842962067 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8002040 con 0x7f5ed0003ac0 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.998+0000 7f5ee67fc640 1 -- 172.21.15.59:0/842962067 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ed8001250 con 0x7f5ed0003ac0 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.998+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee8142280 con 0x7f5ed0003ac0 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.998+0000 7f5ee67fc640 1 -- 172.21.15.59:0/842962067 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed802bbf0 con 0x7f5ed0003ac0 2024-06-29T14:41:58.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.998+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee8142840 con 0x7f5ed0003ac0 2024-06-29T14:41:58.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.998+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee8161cc0 con 0x7f5ed0003ac0 2024-06-29T14:41:58.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:58.999+0000 7f5ee67fc640 1 -- 172.21.15.59:0/842962067 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f5ed8006070 con 0x7f5ed0003ac0 2024-06-29T14:41:58.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.000+0000 7f5ee67fc640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5ed4076420 0x7f5ed4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.000+0000 7f5ee67fc640 1 -- 172.21.15.59:0/842962067 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6129+0+0 (secure 0 0 0) 0x7f5ed80b7b70 con 0x7f5ed0003ac0 2024-06-29T14:41:58.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.000+0000 7f5eed52c640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5ed4076420 0x7f5ed4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.001+0000 7f5eed52c640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5ed4076420 0x7f5ed4078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5edc001050 tx=0x7f5edc000fe0 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:59.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.004+0000 7f5ee67fc640 1 -- 172.21.15.59:0/842962067 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ed8081100 con 0x7f5ed0003ac0 2024-06-29T14:41:59.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.198+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ee80cf210 con 0x7f5ed0003ac0 2024-06-29T14:41:59.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.199+0000 7f5ee67fc640 1 -- 172.21.15.59:0/842962067 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+16969 (secure 0 0 0) 0x7f5ed8089180 con 0x7f5ed0003ac0 2024-06-29T14:41:59.196 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:41:59.197 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":155,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:57.891235+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.250305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.841623+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5ed4076420 msgr2=0x7f5ed4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5ed4076420 0x7f5ed4078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5edc001050 tx=0x7f5edc000fe0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 msgr2=0x7f5ee815dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 0x7f5ee815dc00 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5ed802b470 tx=0x7f5ed8001aa0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 shutdown_connections 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5ed4076420 0x7f5ed4078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ee81e9df0 0x7f5ee8160800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ee81e99e0 0x7f5ee81602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 --2- 172.21.15.59:0/842962067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ed0003ac0 0x7f5ee815dc00 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 >> 172.21.15.59:0/842962067 conn(0x7f5ee806c950 msgr2=0x7f5ee8146510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:59.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 shutdown_connections 2024-06-29T14:41:59.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.201+0000 7f5eeff91640 1 -- 172.21.15.59:0/842962067 wait complete. 2024-06-29T14:41:59.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:41:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.335+0000 7fc523ec3640 1 -- 172.21.15.59:0/2556112337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc508003ac0 msgr2=0x7fc508003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.335+0000 7fc523ec3640 1 --2- 172.21.15.59:0/2556112337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc508003ac0 0x7fc508003f30 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fc518009b50 tx=0x7fc508005830 comp rx=0 tx=0).stop 2024-06-29T14:41:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.335+0000 7fc523ec3640 1 -- 172.21.15.59:0/2556112337 shutdown_connections 2024-06-29T14:41:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.335+0000 7fc523ec3640 1 --2- 172.21.15.59:0/2556112337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc508003ac0 0x7fc508003f30 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.335+0000 7fc523ec3640 1 -- 172.21.15.59:0/2556112337 >> 172.21.15.59:0/2556112337 conn(0x7fc51c06c950 msgr2=0x7fc51c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.335+0000 7fc523ec3640 1 -- 172.21.15.59:0/2556112337 shutdown_connections 2024-06-29T14:41:59.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.335+0000 7fc523ec3640 1 -- 172.21.15.59:0/2556112337 wait complete. 2024-06-29T14:41:59.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.337+0000 7fc523ec3640 1 Processor -- start 2024-06-29T14:41:59.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.337+0000 7fc523ec3640 1 -- start start 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc523ec3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc508003ac0 0x7fc51c15b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc523ec3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc51c15b810 0x7fc51c1f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc523ec3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c1f54f0 0x7fc51c1f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc523ec3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc51c0cf9f0 con 0x7fc51c1f54f0 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc523ec3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc51c0cf870 con 0x7fc508003ac0 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc523ec3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc51c0cfb70 con 0x7fc51c15b810 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c1f54f0 0x7fc51c1f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c1f54f0 0x7fc51c1f5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49548/0 (socket says 172.21.15.59:49548) 2024-06-29T14:41:59.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 -- 172.21.15.59:0/4028462861 learned_addr learned my addr 172.21.15.59:0/4028462861 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:41:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc52145e640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc51c15b810 0x7fc51c1f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc521c5f640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc508003ac0 0x7fc51c15b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 -- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc51c15b810 msgr2=0x7fc51c1f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc51c15b810 0x7fc51c1f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 -- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc508003ac0 msgr2=0x7fc51c15b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc508003ac0 0x7fc51c15b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 -- 172.21.15.59:0/4028462861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51c1e4470 con 0x7fc51c1f54f0 2024-06-29T14:41:59.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc52145e640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc51c15b810 0x7fc51c1f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:59.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc521c5f640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc508003ac0 0x7fc51c15b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:59.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc522460640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c1f54f0 0x7fc51c1f5900 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0b5030 tx=0x7fc51400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:59.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc5127fc640 1 -- 172.21.15.59:0/4028462861 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51400c920 con 0x7fc51c1f54f0 2024-06-29T14:41:59.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc5127fc640 1 -- 172.21.15.59:0/4028462861 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc514017070 con 0x7fc51c1f54f0 2024-06-29T14:41:59.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc5127fc640 1 -- 172.21.15.59:0/4028462861 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51400ce70 con 0x7fc51c1f54f0 2024-06-29T14:41:59.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.339+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc51c15bc20 con 0x7fc51c1f54f0 2024-06-29T14:41:59.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.340+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc51c0ae220 con 0x7fc51c1f54f0 2024-06-29T14:41:59.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.340+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc51c0adcb0 con 0x7fc51c1f54f0 2024-06-29T14:41:59.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.341+0000 7fc5127fc640 1 -- 172.21.15.59:0/4028462861 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fc51401c460 con 0x7fc51c1f54f0 2024-06-29T14:41:59.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.342+0000 7fc5127fc640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5000763f0 0x7fc500078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:59.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.342+0000 7fc5127fc640 1 -- 172.21.15.59:0/4028462861 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6129+0+0 (secure 0 0 0) 0x7fc514098010 con 0x7fc51c1f54f0 2024-06-29T14:41:59.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.342+0000 7fc521c5f640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5000763f0 0x7fc500078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:59.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.343+0000 7fc521c5f640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5000763f0 0x7fc500078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc518035000 tx=0x7fc5180046c0 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:59.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.346+0000 7fc5127fc640 1 -- 172.21.15.59:0/4028462861 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc514061510 con 0x7fc51c1f54f0 2024-06-29T14:41:59.477 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T14:41:59.480+0000 7f74777ba640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:41:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:41:59.586+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc51c0cfd50 con 0x7fc51c1f54f0 2024-06-29T14:42:00.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.534+0000 7fc5127fc640 1 -- 172.21.15.59:0/4028462861 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v79) ==== 118+0+0 (secure 0 0 0) 0x7fc514060e10 con 0x7fc51c1f54f0 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5000763f0 msgr2=0x7fc500078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5000763f0 0x7fc500078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc518035000 tx=0x7fc5180046c0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c1f54f0 msgr2=0x7fc51c1f5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c1f54f0 0x7fc51c1f5900 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0b5030 tx=0x7fc51400ed10 comp rx=0 tx=0).stop 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 shutdown_connections 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc5000763f0 0x7fc500078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c1f54f0 0x7fc51c1f5900 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc51c15b810 0x7fc51c1f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 --2- 172.21.15.59:0/4028462861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc508003ac0 0x7fc51c15b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 >> 172.21.15.59:0/4028462861 conn(0x7fc51c06c950 msgr2=0x7fc51c143a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:00.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 shutdown_connections 2024-06-29T14:42:00.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.538+0000 7fc523ec3640 1 -- 172.21.15.59:0/4028462861 wait complete. 2024-06-29T14:42:00.549 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:42:00.549 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:42:00.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.680+0000 7f238824d640 1 -- 172.21.15.59:0/1448725323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 msgr2=0x7f238006e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.680+0000 7f238824d640 1 --2- 172.21.15.59:0/1448725323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 0x7f238006e9e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2378005fd0 tx=0x7f237802aa90 comp rx=0 tx=0).stop 2024-06-29T14:42:00.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.680+0000 7f238824d640 1 -- 172.21.15.59:0/1448725323 shutdown_connections 2024-06-29T14:42:00.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.680+0000 7f238824d640 1 --2- 172.21.15.59:0/1448725323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 0x7f238006e9e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.680+0000 7f238824d640 1 --2- 172.21.15.59:0/1448725323 >> v2:172.21.15.153:3300/0 conn(0x7f23800d1ce0 0x7f238015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.680+0000 7f238824d640 1 -- 172.21.15.59:0/1448725323 >> 172.21.15.59:0/1448725323 conn(0x7f2380073e10 msgr2=0x7f2380074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:00.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.680+0000 7f238824d640 1 -- 172.21.15.59:0/1448725323 shutdown_connections 2024-06-29T14:42:00.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.680+0000 7f238824d640 1 -- 172.21.15.59:0/1448725323 wait complete. 2024-06-29T14:42:00.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.682+0000 7f238824d640 1 Processor -- start 2024-06-29T14:42:00.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.682+0000 7f238824d640 1 -- start start 2024-06-29T14:42:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f238824d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23800d0dc0 0x7f23800cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f238824d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23800d1ce0 0x7f23801598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f238824d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 0x7f23801f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f238824d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f238006f2b0 con 0x7f23800d0dc0 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f238824d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f238006f130 con 0x7f23800d1ce0 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f238824d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f238006f430 con 0x7f238015ead0 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f2385fe9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23800d0dc0 0x7f23800cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f2385fe9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23800d0dc0 0x7f23800cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49560/0 (socket says 172.21.15.59:49560) 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f2385fe9640 1 -- 172.21.15.59:0/2412566351 learned_addr learned my addr 172.21.15.59:0/2412566351 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.683+0000 7f23867ea640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 0x7f23801f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f23857e8640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23800d1ce0 0x7f23801598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f2385fe9640 1 -- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 msgr2=0x7f23801f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f2385fe9640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 0x7f23801f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f2385fe9640 1 -- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23800d1ce0 msgr2=0x7f23801598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f2385fe9640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23800d1ce0 0x7f23801598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f2385fe9640 1 -- 172.21.15.59:0/2412566351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23800cbe20 con 0x7f23800d0dc0 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f23867ea640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 0x7f23801f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f23857e8640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23800d1ce0 0x7f23801598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f2385fe9640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23800d0dc0 0x7f23800cb5d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f237c00c3d0 tx=0x7f237c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f23767fc640 1 -- 172.21.15.59:0/2412566351 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237c017070 con 0x7f23800d0dc0 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f23767fc640 1 -- 172.21.15.59:0/2412566351 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f237c0053c0 con 0x7f23800d0dc0 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f23767fc640 1 -- 172.21.15.59:0/2412566351 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237c003850 con 0x7f23800d0dc0 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23800cc100 con 0x7f23800d0dc0 2024-06-29T14:42:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.684+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23800cc6c0 con 0x7f23800d0dc0 2024-06-29T14:42:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.685+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f238006f950 con 0x7f23800d0dc0 2024-06-29T14:42:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.686+0000 7f23767fc640 1 -- 172.21.15.59:0/2412566351 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f237c01a070 con 0x7f23800d0dc0 2024-06-29T14:42:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.686+0000 7f23767fc640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f2364076420 0x7f2364078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.687+0000 7f23767fc640 1 -- 172.21.15.59:0/2412566351 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f237c096f20 con 0x7f23800d0dc0 2024-06-29T14:42:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.687+0000 7f23857e8640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f2364076420 0x7f2364078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.687+0000 7f23857e8640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f2364076420 0x7f2364078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f23800748e0 tx=0x7f2370005f90 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.691+0000 7f23767fc640 1 -- 172.21.15.59:0/2412566351 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f237c060520 con 0x7f23800d0dc0 2024-06-29T14:42:00.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:00.923+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f23800d09a0 con 0x7f23800d0dc0 2024-06-29T14:42:01.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.535+0000 7f23767fc640 1 -- 172.21.15.59:0/2412566351 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v80) ==== 122+0+0 (secure 0 0 0) 0x7f237c05fe20 con 0x7f23800d0dc0 2024-06-29T14:42:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f2364076420 msgr2=0x7f2364078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f2364076420 0x7f2364078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f23800748e0 tx=0x7f2370005f90 comp rx=0 tx=0).stop 2024-06-29T14:42:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23800d0dc0 msgr2=0x7f23800cb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23800d0dc0 0x7f23800cb5d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f237c00c3d0 tx=0x7f237c00c890 comp rx=0 tx=0).stop 2024-06-29T14:42:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 shutdown_connections 2024-06-29T14:42:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f2364076420 0x7f2364078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f238015ead0 0x7f23801f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23800d1ce0 0x7f23801598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.537+0000 7f238824d640 1 --2- 172.21.15.59:0/2412566351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23800d0dc0 0x7f23800cb5d0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.538+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 >> 172.21.15.59:0/2412566351 conn(0x7f2380073e10 msgr2=0x7f23800d0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.538+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 shutdown_connections 2024-06-29T14:42:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.538+0000 7f238824d640 1 -- 172.21.15.59:0/2412566351 wait complete. 2024-06-29T14:42:01.549 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:42:01.549 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:42:01.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.676+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3662348228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 msgr2=0x7f1cf00b5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.676+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3662348228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 0x7f1cf00b5730 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0071a30 tx=0x7f1ce802aa30 comp rx=0 tx=0).stop 2024-06-29T14:42:01.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.676+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3662348228 shutdown_connections 2024-06-29T14:42:01.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.676+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3662348228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 0x7f1cf00b5730 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.676+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3662348228 >> v2:172.21.15.153:3300/0 conn(0x7f1cf006eb40 0x7f1cf01443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.676+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3662348228 >> 172.21.15.59:0/3662348228 conn(0x7f1cf0073e50 msgr2=0x7f1cf00b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:01.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.676+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3662348228 shutdown_connections 2024-06-29T14:42:01.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.676+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3662348228 wait complete. 2024-06-29T14:42:01.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.677+0000 7f1cf7d8c640 1 Processor -- start 2024-06-29T14:42:01.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.678+0000 7f1cf7d8c640 1 -- start start 2024-06-29T14:42:01.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf7d8c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cf006eb40 0x7f1cf015a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:01.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf7d8c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cf0071620 0x7f1cf00ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:01.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf7d8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 0x7f1cf00aca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:01.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf7d8c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf00cf470 con 0x7f1cf00b5320 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf7d8c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf00cf2f0 con 0x7f1cf0071620 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf7d8c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf00cf5f0 con 0x7f1cf006eb40 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 0x7f1cf00aca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 0x7f1cf00aca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49572/0 (socket says 172.21.15.59:49572) 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 -- 172.21.15.59:0/3835250583 learned_addr learned my addr 172.21.15.59:0/3835250583 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf5b28640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cf006eb40 0x7f1cf015a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf5327640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cf0071620 0x7f1cf00ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 -- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cf006eb40 msgr2=0x7f1cf015a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cf006eb40 0x7f1cf015a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 -- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cf0071620 msgr2=0x7f1cf00ac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cf0071620 0x7f1cf00ac550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 -- 172.21.15.59:0/3835250583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf015a6c0 con 0x7f1cf00b5320 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf5327640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cf0071620 0x7f1cf00ac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.679+0000 7f1cf6329640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 0x7f1cf00aca90 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8005fd0 tx=0x7f1ce8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.680+0000 7f1ce67fc640 1 -- 172.21.15.59:0/3835250583 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8044020 con 0x7f1cf00b5320 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.680+0000 7f1ce67fc640 1 -- 172.21.15.59:0/3835250583 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ce80046e0 con 0x7f1cf00b5320 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.680+0000 7f1ce67fc640 1 -- 172.21.15.59:0/3835250583 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8033070 con 0x7f1cf00b5320 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.680+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf00aa820 con 0x7f1cf00b5320 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.680+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf00abd50 con 0x7f1cf00b5320 2024-06-29T14:42:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.681+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf01443e0 con 0x7f1cf00b5320 2024-06-29T14:42:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.682+0000 7f1ce67fc640 1 -- 172.21.15.59:0/3835250583 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f1ce8047020 con 0x7f1cf00b5320 2024-06-29T14:42:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.682+0000 7f1ce67fc640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1cd4076420 0x7f1cd4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.682+0000 7f1ce67fc640 1 -- 172.21.15.59:0/3835250583 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f1ce80bc090 con 0x7f1cf00b5320 2024-06-29T14:42:01.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.683+0000 7f1cf5b28640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1cd4076420 0x7f1cd4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:01.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.683+0000 7f1cf5b28640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1cd4076420 0x7f1cd4078930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1cec000f60 tx=0x7f1cec009040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.686+0000 7f1ce67fc640 1 -- 172.21.15.59:0/3835250583 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1ce804e020 con 0x7f1cf00b5320 2024-06-29T14:42:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.917+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cf00aa5b0 con 0x7f1cf00b5320 2024-06-29T14:42:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.918+0000 7f1ce67fc640 1 -- 172.21.15.59:0/3835250583 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5225 (secure 0 0 0) 0x7f1cf00b5c70 con 0x7f1cf00b5320 2024-06-29T14:42:01.915 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:42:01.915 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"btime":"2024-06-29T14:42:01:534262+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/507949731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":507949731},{"type":"v1","addr":"172.21.15.153:6837","nonce":507949731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":8812,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1668688055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1668688055},{"type":"v1","addr":"172.21.15.153:6836","nonce":1668688055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":9372,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3535921856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3535921856},{"type":"v1","addr":"172.21.15.59:6834","nonce":3535921856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.891192+0000","modified":"2024-06-29T14:42:00.924713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/178954889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":178954889},{"type":"v1","addr":"172.21.15.59:6835","nonce":178954889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9375,"qdb_cluster":[9375]},"id":9}]} 2024-06-29T14:42:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 80 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1cd4076420 msgr2=0x7f1cd4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1cd4076420 0x7f1cd4078930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1cec000f60 tx=0x7f1cec009040 comp rx=0 tx=0).stop 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 msgr2=0x7f1cf00aca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 0x7f1cf00aca90 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8005fd0 tx=0x7f1ce8003f90 comp rx=0 tx=0).stop 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 shutdown_connections 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1cd4076420 0x7f1cd4078930 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cf00b5320 0x7f1cf00aca90 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cf0071620 0x7f1cf00ac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 --2- 172.21.15.59:0/3835250583 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cf006eb40 0x7f1cf015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 >> 172.21.15.59:0/3835250583 conn(0x7f1cf0073e50 msgr2=0x7f1cf006e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 shutdown_connections 2024-06-29T14:42:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:01.920+0000 7f1cf7d8c640 1 -- 172.21.15.59:0/3835250583 wait complete. 2024-06-29T14:42:01.928 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:42:02.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.049+0000 7f699151c640 1 -- 172.21.15.59:0/3391180581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b3740 msgr2=0x7f698c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.049+0000 7f699151c640 1 --2- 172.21.15.59:0/3391180581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b3740 0x7f698c0b3b50 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f6974005fd0 tx=0x7f697402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.049+0000 7f699151c640 1 -- 172.21.15.59:0/3391180581 shutdown_connections 2024-06-29T14:42:02.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.049+0000 7f699151c640 1 --2- 172.21.15.59:0/3391180581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b3740 0x7f698c0b3b50 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.049+0000 7f699151c640 1 -- 172.21.15.59:0/3391180581 >> 172.21.15.59:0/3391180581 conn(0x7f698c06c950 msgr2=0x7f698c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.049+0000 7f699151c640 1 -- 172.21.15.59:0/3391180581 shutdown_connections 2024-06-29T14:42:02.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.049+0000 7f699151c640 1 -- 172.21.15.59:0/3391180581 wait complete. 2024-06-29T14:42:02.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.051+0000 7f699151c640 1 Processor -- start 2024-06-29T14:42:02.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.051+0000 7f699151c640 1 -- start start 2024-06-29T14:42:02.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f699151c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f698c0b3740 0x7f698c142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f699151c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f698c15c2f0 0x7f698c0b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f699151c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b10a0 0x7f698c13ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f699151c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b6320 con 0x7f698c0b10a0 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f699151c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b61a0 con 0x7f698c0b3740 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f699151c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f698c0b64a0 con 0x7f698c15c2f0 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f698b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b10a0 0x7f698c13ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f698b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b10a0 0x7f698c13ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49600/0 (socket says 172.21.15.59:49600) 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f698b7fe640 1 -- 172.21.15.59:0/3575335614 learned_addr learned my addr 172.21.15.59:0/3575335614 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f698a7fc640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f698c15c2f0 0x7f698c0b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.052+0000 7f698affd640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f698c0b3740 0x7f698c142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f698b7fe640 1 -- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f698c15c2f0 msgr2=0x7f698c0b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f698b7fe640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f698c15c2f0 0x7f698c0b0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f698b7fe640 1 -- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f698c0b3740 msgr2=0x7f698c142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f698b7fe640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f698c0b3740 0x7f698c142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f698b7fe640 1 -- 172.21.15.59:0/3575335614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c1f2f50 con 0x7f698c0b10a0 2024-06-29T14:42:02.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f698affd640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f698c0b3740 0x7f698c142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:02.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f698b7fe640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b10a0 0x7f698c13ffb0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f698400b410 tx=0x7f698400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f69837fe640 1 -- 172.21.15.59:0/3575335614 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984015030 con 0x7f698c0b10a0 2024-06-29T14:42:02.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f69837fe640 1 -- 172.21.15.59:0/3575335614 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6984004020 con 0x7f698c0b10a0 2024-06-29T14:42:02.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c1f3230 con 0x7f698c0b10a0 2024-06-29T14:42:02.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f69837fe640 1 -- 172.21.15.59:0/3575335614 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984011630 con 0x7f698c0b10a0 2024-06-29T14:42:02.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.053+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c1f37f0 con 0x7f698c0b10a0 2024-06-29T14:42:02.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.054+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f698c1f2cc0 con 0x7f698c0b10a0 2024-06-29T14:42:02.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.055+0000 7f69837fe640 1 -- 172.21.15.59:0/3575335614 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f6984018070 con 0x7f698c0b10a0 2024-06-29T14:42:02.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.055+0000 7f69837fe640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f696c076420 0x7f696c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.056+0000 7f69837fe640 1 -- 172.21.15.59:0/3575335614 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f6984097510 con 0x7f698c0b10a0 2024-06-29T14:42:02.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.056+0000 7f698affd640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f696c076420 0x7f696c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.056+0000 7f698affd640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f696c076420 0x7f696c078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f697c001da0 tx=0x7f697c009040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.060+0000 7f69837fe640 1 -- 172.21.15.59:0/3575335614 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6984060b20 con 0x7f698c0b10a0 2024-06-29T14:42:02.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.256+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f698c0706e0 con 0x7f698c0b10a0 2024-06-29T14:42:02.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.257+0000 7f69837fe640 1 -- 172.21.15.59:0/3575335614 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) ==== 74+0+16956 (secure 0 0 0) 0x7f698c0706e0 con 0x7f698c0b10a0 2024-06-29T14:42:02.254 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:42:02.254 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":156,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T14:41:59.519498+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.250305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.841623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:42:02.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f696c076420 msgr2=0x7f696c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f696c076420 0x7f696c078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f697c001da0 tx=0x7f697c009040 comp rx=0 tx=0).stop 2024-06-29T14:42:02.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b10a0 msgr2=0x7f698c13ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b10a0 0x7f698c13ffb0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f698400b410 tx=0x7f698400b8d0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 shutdown_connections 2024-06-29T14:42:02.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f696c076420 0x7f696c078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f698c0b10a0 0x7f698c13ffb0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f698c15c2f0 0x7f698c0b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 --2- 172.21.15.59:0/3575335614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f698c0b3740 0x7f698c142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 >> 172.21.15.59:0/3575335614 conn(0x7f698c06c950 msgr2=0x7f698c071610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 shutdown_connections 2024-06-29T14:42:02.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.259+0000 7f699151c640 1 -- 172.21.15.59:0/3575335614 wait complete. 2024-06-29T14:42:02.267 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:42:02.267 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:42:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.388+0000 7fc48c794640 1 -- 172.21.15.59:0/3091977394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4840ce950 msgr2=0x7fc4841622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.388+0000 7fc48c794640 1 --2- 172.21.15.59:0/3091977394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4840ce950 0x7fc4841622c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fc478005fd0 tx=0x7fc47802a4f0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.388+0000 7fc48c794640 1 -- 172.21.15.59:0/3091977394 shutdown_connections 2024-06-29T14:42:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.388+0000 7fc48c794640 1 --2- 172.21.15.59:0/3091977394 >> v2:172.21.15.153:3300/0 conn(0x7fc484162800 0x7fc484162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.388+0000 7fc48c794640 1 --2- 172.21.15.59:0/3091977394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4840ce950 0x7fc4841622c0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.388+0000 7fc48c794640 1 -- 172.21.15.59:0/3091977394 >> 172.21.15.59:0/3091977394 conn(0x7fc4840b5610 msgr2=0x7fc4840b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.388+0000 7fc48c794640 1 -- 172.21.15.59:0/3091977394 shutdown_connections 2024-06-29T14:42:02.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.388+0000 7fc48c794640 1 -- 172.21.15.59:0/3091977394 wait complete. 2024-06-29T14:42:02.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.390+0000 7fc48c794640 1 Processor -- start 2024-06-29T14:42:02.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.390+0000 7fc48c794640 1 -- start start 2024-06-29T14:42:02.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48c794640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc48408c190 0x7fc4840cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48c794640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4840ce950 0x7fc4840aace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48c794640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc484162800 0x7fc4840ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48c794640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc4840b7f60 con 0x7fc484162800 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48c794640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc4840b7de0 con 0x7fc48408c190 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48c794640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc4840b80e0 con 0x7fc4840ce950 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48ad31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc484162800 0x7fc4840ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48ad31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc484162800 0x7fc4840ac090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49604/0 (socket says 172.21.15.59:49604) 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48ad31640 1 -- 172.21.15.59:0/3750179121 learned_addr learned my addr 172.21.15.59:0/3750179121 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc489d2f640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4840ce950 0x7fc4840aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48a530640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc48408c190 0x7fc4840cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48ad31640 1 -- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4840ce950 msgr2=0x7fc4840aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48ad31640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4840ce950 0x7fc4840aace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48ad31640 1 -- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc48408c190 msgr2=0x7fc4840cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48ad31640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc48408c190 0x7fc4840cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.391+0000 7fc48ad31640 1 -- 172.21.15.59:0/3750179121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4840cad00 con 0x7fc484162800 2024-06-29T14:42:02.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.392+0000 7fc48ad31640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc484162800 0x7fc4840ac090 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fc47400e9c0 tx=0x7fc47400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.392+0000 7fc482ffd640 1 -- 172.21.15.59:0/3750179121 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc474004d20 con 0x7fc484162800 2024-06-29T14:42:02.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.392+0000 7fc482ffd640 1 -- 172.21.15.59:0/3750179121 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc47400cb20 con 0x7fc484162800 2024-06-29T14:42:02.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.392+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4840cafe0 con 0x7fc484162800 2024-06-29T14:42:02.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.392+0000 7fc482ffd640 1 -- 172.21.15.59:0/3750179121 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc474005020 con 0x7fc484162800 2024-06-29T14:42:02.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.392+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4840cb5a0 con 0x7fc484162800 2024-06-29T14:42:02.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.393+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4840cd980 con 0x7fc484162800 2024-06-29T14:42:02.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.394+0000 7fc482ffd640 1 -- 172.21.15.59:0/3750179121 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fc4740051c0 con 0x7fc484162800 2024-06-29T14:42:02.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.394+0000 7fc482ffd640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc470076420 0x7fc470078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.394+0000 7fc482ffd640 1 -- 172.21.15.59:0/3750179121 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7fc474098360 con 0x7fc484162800 2024-06-29T14:42:02.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.395+0000 7fc48a530640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc470076420 0x7fc470078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.395+0000 7fc48a530640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc470076420 0x7fc470078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc47c006fd0 tx=0x7fc47c009060 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.399+0000 7fc482ffd640 1 -- 172.21.15.59:0/3750179121 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc474016030 con 0x7fc484162800 2024-06-29T14:42:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.591+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 --> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc4840cdc10 con 0x7fc470076420 2024-06-29T14:42:02.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.597+0000 7fc482ffd640 1 -- 172.21.15.59:0/3750179121 <== mgr.7658 v2:172.21.15.153:6832/3378196672 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252485 (secure 0 0 0) 0x7fc4840cdc10 con 0x7fc470076420 2024-06-29T14:42:02.597 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:42:02.598 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc470076420 msgr2=0x7fc470078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc470076420 0x7fc470078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc47c006fd0 tx=0x7fc47c009060 comp rx=0 tx=0).stop 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc484162800 msgr2=0x7fc4840ac090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc484162800 0x7fc4840ac090 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fc47400e9c0 tx=0x7fc47400ee80 comp rx=0 tx=0).stop 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 shutdown_connections 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fc470076420 0x7fc470078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc484162800 0x7fc4840ac090 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4840ce950 0x7fc4840aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 --2- 172.21.15.59:0/3750179121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc48408c190 0x7fc4840cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 >> 172.21.15.59:0/3750179121 conn(0x7fc4840b5610 msgr2=0x7fc48406d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 shutdown_connections 2024-06-29T14:42:02.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.600+0000 7fc48c794640 1 -- 172.21.15.59:0/3750179121 wait complete. 2024-06-29T14:42:02.608 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:42:01.986038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":314,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":361536,"kb_used_data":16324,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754613184,"statfs":{"total":773094113280,"available":772723900416,"internally_reserved":0,"allocated":16715776,"data_stored":9033452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":353357362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.496476"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533901+0000","last_change":"2024-06-29T14:41:58.534161+0000","last_active":"2024-06-29T14:41:59.533901+0000","last_peered":"2024-06-29T14:41:59.533901+0000","last_clean":"2024-06-29T14:41:59.533901+0000","last_became_active":"2024-06-29T14:41:58.532120+0000","last_became_peered":"2024-06-29T14:41:58.532120+0000","last_unstale":"2024-06-29T14:41:59.533901+0000","last_undegraded":"2024-06-29T14:41:59.533901+0000","last_fullsized":"2024-06-29T14:41:59.533901+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:16.350098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533658+0000","last_change":"2024-06-29T14:41:58.518549+0000","last_active":"2024-06-29T14:41:59.533658+0000","last_peered":"2024-06-29T14:41:59.533658+0000","last_clean":"2024-06-29T14:41:59.533658+0000","last_became_active":"2024-06-29T14:41:58.518244+0000","last_became_peered":"2024-06-29T14:41:58.518244+0000","last_unstale":"2024-06-29T14:41:59.533658+0000","last_undegraded":"2024-06-29T14:41:59.533658+0000","last_fullsized":"2024-06-29T14:41:59.533658+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:15.787250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525589+0000","last_change":"2024-06-29T14:41:58.525851+0000","last_active":"2024-06-29T14:41:59.525589+0000","last_peered":"2024-06-29T14:41:59.525589+0000","last_clean":"2024-06-29T14:41:59.525589+0000","last_became_active":"2024-06-29T14:41:58.524386+0000","last_became_peered":"2024-06-29T14:41:58.524386+0000","last_unstale":"2024-06-29T14:41:59.525589+0000","last_undegraded":"2024-06-29T14:41:59.525589+0000","last_fullsized":"2024-06-29T14:41:59.525589+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:26:57.805185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533027+0000","last_change":"2024-06-29T14:41:58.527479+0000","last_active":"2024-06-29T14:41:59.533027+0000","last_peered":"2024-06-29T14:41:59.533027+0000","last_clean":"2024-06-29T14:41:59.533027+0000","last_became_active":"2024-06-29T14:41:58.527091+0000","last_became_peered":"2024-06-29T14:41:58.527091+0000","last_unstale":"2024-06-29T14:41:59.533027+0000","last_undegraded":"2024-06-29T14:41:59.533027+0000","last_fullsized":"2024-06-29T14:41:59.533027+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:37.536052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523583+0000","last_change":"2024-06-29T14:41:58.524582+0000","last_active":"2024-06-29T14:41:59.523583+0000","last_peered":"2024-06-29T14:41:59.523583+0000","last_clean":"2024-06-29T14:41:59.523583+0000","last_became_active":"2024-06-29T14:41:58.524257+0000","last_became_peered":"2024-06-29T14:41:58.524257+0000","last_unstale":"2024-06-29T14:41:59.523583+0000","last_undegraded":"2024-06-29T14:41:59.523583+0000","last_fullsized":"2024-06-29T14:41:59.523583+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:03.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532826+0000","last_change":"2024-06-29T14:41:58.518349+0000","last_active":"2024-06-29T14:41:59.532826+0000","last_peered":"2024-06-29T14:41:59.532826+0000","last_clean":"2024-06-29T14:41:59.532826+0000","last_became_active":"2024-06-29T14:41:58.517991+0000","last_became_peered":"2024-06-29T14:41:58.517991+0000","last_unstale":"2024-06-29T14:41:59.532826+0000","last_undegraded":"2024-06-29T14:41:59.532826+0000","last_fullsized":"2024-06-29T14:41:59.532826+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:33.500996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526671+0000","last_change":"2024-06-29T14:41:58.529976+0000","last_active":"2024-06-29T14:41:59.526671+0000","last_peered":"2024-06-29T14:41:59.526671+0000","last_clean":"2024-06-29T14:41:59.526671+0000","last_became_active":"2024-06-29T14:41:58.529667+0000","last_became_peered":"2024-06-29T14:41:58.529667+0000","last_unstale":"2024-06-29T14:41:59.526671+0000","last_undegraded":"2024-06-29T14:41:59.526671+0000","last_fullsized":"2024-06-29T14:41:59.526671+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:19.720982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532519+0000","last_change":"2024-06-29T14:41:58.526786+0000","last_active":"2024-06-29T14:41:59.532519+0000","last_peered":"2024-06-29T14:41:59.532519+0000","last_clean":"2024-06-29T14:41:59.532519+0000","last_became_active":"2024-06-29T14:41:58.526479+0000","last_became_peered":"2024-06-29T14:41:58.526479+0000","last_unstale":"2024-06-29T14:41:59.532519+0000","last_undegraded":"2024-06-29T14:41:59.532519+0000","last_fullsized":"2024-06-29T14:41:59.532519+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:15:32.748190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533363+0000","last_change":"2024-06-29T14:41:58.517597+0000","last_active":"2024-06-29T14:41:59.533363+0000","last_peered":"2024-06-29T14:41:59.533363+0000","last_clean":"2024-06-29T14:41:59.533363+0000","last_became_active":"2024-06-29T14:41:58.517218+0000","last_became_peered":"2024-06-29T14:41:58.517218+0000","last_unstale":"2024-06-29T14:41:59.533363+0000","last_undegraded":"2024-06-29T14:41:59.533363+0000","last_fullsized":"2024-06-29T14:41:59.533363+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:06.338200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533191+0000","last_change":"2024-06-29T14:41:58.530549+0000","last_active":"2024-06-29T14:41:59.533191+0000","last_peered":"2024-06-29T14:41:59.533191+0000","last_clean":"2024-06-29T14:41:59.533191+0000","last_became_active":"2024-06-29T14:41:58.530202+0000","last_became_peered":"2024-06-29T14:41:58.530202+0000","last_unstale":"2024-06-29T14:41:59.533191+0000","last_undegraded":"2024-06-29T14:41:59.533191+0000","last_fullsized":"2024-06-29T14:41:59.533191+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:30.652977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533760+0000","last_change":"2024-06-29T14:41:58.516764+0000","last_active":"2024-06-29T14:41:59.533760+0000","last_peered":"2024-06-29T14:41:59.533760+0000","last_clean":"2024-06-29T14:41:59.533760+0000","last_became_active":"2024-06-29T14:41:58.516446+0000","last_became_peered":"2024-06-29T14:41:58.516446+0000","last_unstale":"2024-06-29T14:41:59.533760+0000","last_undegraded":"2024-06-29T14:41:59.533760+0000","last_fullsized":"2024-06-29T14:41:59.533760+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:54.490869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533695+0000","last_change":"2024-06-29T14:41:58.534977+0000","last_active":"2024-06-29T14:41:59.533695+0000","last_peered":"2024-06-29T14:41:59.533695+0000","last_clean":"2024-06-29T14:41:59.533695+0000","last_became_active":"2024-06-29T14:41:58.534644+0000","last_became_peered":"2024-06-29T14:41:58.534644+0000","last_unstale":"2024-06-29T14:41:59.533695+0000","last_undegraded":"2024-06-29T14:41:59.533695+0000","last_fullsized":"2024-06-29T14:41:59.533695+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:57.768219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532551+0000","last_change":"2024-06-29T14:41:58.519217+0000","last_active":"2024-06-29T14:41:59.532551+0000","last_peered":"2024-06-29T14:41:59.532551+0000","last_clean":"2024-06-29T14:41:59.532551+0000","last_became_active":"2024-06-29T14:41:58.518637+0000","last_became_peered":"2024-06-29T14:41:58.518637+0000","last_unstale":"2024-06-29T14:41:59.532551+0000","last_undegraded":"2024-06-29T14:41:59.532551+0000","last_fullsized":"2024-06-29T14:41:59.532551+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:17.761380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533131+0000","last_change":"2024-06-29T14:41:58.538984+0000","last_active":"2024-06-29T14:41:59.533131+0000","last_peered":"2024-06-29T14:41:59.533131+0000","last_clean":"2024-06-29T14:41:59.533131+0000","last_became_active":"2024-06-29T14:41:58.538653+0000","last_became_peered":"2024-06-29T14:41:58.538653+0000","last_unstale":"2024-06-29T14:41:59.533131+0000","last_undegraded":"2024-06-29T14:41:59.533131+0000","last_fullsized":"2024-06-29T14:41:59.533131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:56:52.580045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533639+0000","last_change":"2024-06-29T14:41:58.539744+0000","last_active":"2024-06-29T14:41:59.533639+0000","last_peered":"2024-06-29T14:41:59.533639+0000","last_clean":"2024-06-29T14:41:59.533639+0000","last_became_active":"2024-06-29T14:41:58.539449+0000","last_became_peered":"2024-06-29T14:41:58.539449+0000","last_unstale":"2024-06-29T14:41:59.533639+0000","last_undegraded":"2024-06-29T14:41:59.533639+0000","last_fullsized":"2024-06-29T14:41:59.533639+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:51.884562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532709+0000","last_change":"2024-06-29T14:41:58.524113+0000","last_active":"2024-06-29T14:41:59.532709+0000","last_peered":"2024-06-29T14:41:59.532709+0000","last_clean":"2024-06-29T14:41:59.532709+0000","last_became_active":"2024-06-29T14:41:58.523762+0000","last_became_peered":"2024-06-29T14:41:58.523762+0000","last_unstale":"2024-06-29T14:41:59.532709+0000","last_undegraded":"2024-06-29T14:41:59.532709+0000","last_fullsized":"2024-06-29T14:41:59.532709+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:53:26.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533494+0000","last_change":"2024-06-29T14:41:58.524869+0000","last_active":"2024-06-29T14:41:59.533494+0000","last_peered":"2024-06-29T14:41:59.533494+0000","last_clean":"2024-06-29T14:41:59.533494+0000","last_became_active":"2024-06-29T14:41:58.524553+0000","last_became_peered":"2024-06-29T14:41:58.524553+0000","last_unstale":"2024-06-29T14:41:59.533494+0000","last_undegraded":"2024-06-29T14:41:59.533494+0000","last_fullsized":"2024-06-29T14:41:59.533494+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:55:34.695370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523970+0000","last_change":"2024-06-29T14:41:58.521070+0000","last_active":"2024-06-29T14:41:59.523970+0000","last_peered":"2024-06-29T14:41:59.523970+0000","last_clean":"2024-06-29T14:41:59.523970+0000","last_became_active":"2024-06-29T14:41:58.520583+0000","last_became_peered":"2024-06-29T14:41:58.520583+0000","last_unstale":"2024-06-29T14:41:59.523970+0000","last_undegraded":"2024-06-29T14:41:59.523970+0000","last_fullsized":"2024-06-29T14:41:59.523970+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:07:38.672786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526983+0000","last_change":"2024-06-29T14:41:58.539172+0000","last_active":"2024-06-29T14:41:59.526983+0000","last_peered":"2024-06-29T14:41:59.526983+0000","last_clean":"2024-06-29T14:41:59.526983+0000","last_became_active":"2024-06-29T14:41:58.538721+0000","last_became_peered":"2024-06-29T14:41:58.538721+0000","last_unstale":"2024-06-29T14:41:59.526983+0000","last_undegraded":"2024-06-29T14:41:59.526983+0000","last_fullsized":"2024-06-29T14:41:59.526983+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:12:48.524845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523783+0000","last_change":"2024-06-29T14:41:58.539117+0000","last_active":"2024-06-29T14:41:59.523783+0000","last_peered":"2024-06-29T14:41:59.523783+0000","last_clean":"2024-06-29T14:41:59.523783+0000","last_became_active":"2024-06-29T14:41:58.538793+0000","last_became_peered":"2024-06-29T14:41:58.538793+0000","last_unstale":"2024-06-29T14:41:59.523783+0000","last_undegraded":"2024-06-29T14:41:59.523783+0000","last_fullsized":"2024-06-29T14:41:59.523783+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:10.667868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532720+0000","last_change":"2024-06-29T14:41:58.520306+0000","last_active":"2024-06-29T14:41:59.532720+0000","last_peered":"2024-06-29T14:41:59.532720+0000","last_clean":"2024-06-29T14:41:59.532720+0000","last_became_active":"2024-06-29T14:41:58.520005+0000","last_became_peered":"2024-06-29T14:41:58.520005+0000","last_unstale":"2024-06-29T14:41:59.532720+0000","last_undegraded":"2024-06-29T14:41:59.532720+0000","last_fullsized":"2024-06-29T14:41:59.532720+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:05.489135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533437+0000","last_change":"2024-06-29T14:41:58.532514+0000","last_active":"2024-06-29T14:41:59.533437+0000","last_peered":"2024-06-29T14:41:59.533437+0000","last_clean":"2024-06-29T14:41:59.533437+0000","last_became_active":"2024-06-29T14:41:58.530528+0000","last_became_peered":"2024-06-29T14:41:58.530528+0000","last_unstale":"2024-06-29T14:41:59.533437+0000","last_undegraded":"2024-06-29T14:41:59.533437+0000","last_fullsized":"2024-06-29T14:41:59.533437+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:41.642248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533230+0000","last_change":"2024-06-29T14:41:58.527374+0000","last_active":"2024-06-29T14:41:59.533230+0000","last_peered":"2024-06-29T14:41:59.533230+0000","last_clean":"2024-06-29T14:41:59.533230+0000","last_became_active":"2024-06-29T14:41:58.526244+0000","last_became_peered":"2024-06-29T14:41:58.526244+0000","last_unstale":"2024-06-29T14:41:59.533230+0000","last_undegraded":"2024-06-29T14:41:59.533230+0000","last_fullsized":"2024-06-29T14:41:59.533230+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:44.948443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533161+0000","last_change":"2024-06-29T14:41:58.517873+0000","last_active":"2024-06-29T14:41:59.533161+0000","last_peered":"2024-06-29T14:41:59.533161+0000","last_clean":"2024-06-29T14:41:59.533161+0000","last_became_active":"2024-06-29T14:41:58.517300+0000","last_became_peered":"2024-06-29T14:41:58.517300+0000","last_unstale":"2024-06-29T14:41:59.533161+0000","last_undegraded":"2024-06-29T14:41:59.533161+0000","last_fullsized":"2024-06-29T14:41:59.533161+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:21:08.860536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526221+0000","last_change":"2024-06-29T14:41:58.519361+0000","last_active":"2024-06-29T14:41:59.526221+0000","last_peered":"2024-06-29T14:41:59.526221+0000","last_clean":"2024-06-29T14:41:59.526221+0000","last_became_active":"2024-06-29T14:41:58.519042+0000","last_became_peered":"2024-06-29T14:41:58.519042+0000","last_unstale":"2024-06-29T14:41:59.526221+0000","last_undegraded":"2024-06-29T14:41:59.526221+0000","last_fullsized":"2024-06-29T14:41:59.526221+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:43.162274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525654+0000","last_change":"2024-06-29T14:41:58.535961+0000","last_active":"2024-06-29T14:41:59.525654+0000","last_peered":"2024-06-29T14:41:59.525654+0000","last_clean":"2024-06-29T14:41:59.525654+0000","last_became_active":"2024-06-29T14:41:58.535649+0000","last_became_peered":"2024-06-29T14:41:58.535649+0000","last_unstale":"2024-06-29T14:41:59.525654+0000","last_undegraded":"2024-06-29T14:41:59.525654+0000","last_fullsized":"2024-06-29T14:41:59.525654+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:23.362810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527197+0000","last_change":"2024-06-29T14:41:58.539230+0000","last_active":"2024-06-29T14:41:59.527197+0000","last_peered":"2024-06-29T14:41:59.527197+0000","last_clean":"2024-06-29T14:41:59.527197+0000","last_became_active":"2024-06-29T14:41:58.538844+0000","last_became_peered":"2024-06-29T14:41:58.538844+0000","last_unstale":"2024-06-29T14:41:59.527197+0000","last_undegraded":"2024-06-29T14:41:59.527197+0000","last_fullsized":"2024-06-29T14:41:59.527197+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:18:01.237841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532237+0000","last_change":"2024-06-29T14:41:58.529623+0000","last_active":"2024-06-29T14:41:59.532237+0000","last_peered":"2024-06-29T14:41:59.532237+0000","last_clean":"2024-06-29T14:41:59.532237+0000","last_became_active":"2024-06-29T14:41:58.528984+0000","last_became_peered":"2024-06-29T14:41:58.528984+0000","last_unstale":"2024-06-29T14:41:59.532237+0000","last_undegraded":"2024-06-29T14:41:59.532237+0000","last_fullsized":"2024-06-29T14:41:59.532237+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:44.948443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525645+0000","last_change":"2024-06-29T14:41:58.522389+0000","last_active":"2024-06-29T14:41:59.525645+0000","last_peered":"2024-06-29T14:41:59.525645+0000","last_clean":"2024-06-29T14:41:59.525645+0000","last_became_active":"2024-06-29T14:41:58.522074+0000","last_became_peered":"2024-06-29T14:41:58.522074+0000","last_unstale":"2024-06-29T14:41:59.525645+0000","last_undegraded":"2024-06-29T14:41:59.525645+0000","last_fullsized":"2024-06-29T14:41:59.525645+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:02:26.307183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525411+0000","last_change":"2024-06-29T14:41:58.518761+0000","last_active":"2024-06-29T14:41:59.525411+0000","last_peered":"2024-06-29T14:41:59.525411+0000","last_clean":"2024-06-29T14:41:59.525411+0000","last_became_active":"2024-06-29T14:41:58.518383+0000","last_became_peered":"2024-06-29T14:41:58.518383+0000","last_unstale":"2024-06-29T14:41:59.525411+0000","last_undegraded":"2024-06-29T14:41:59.525411+0000","last_fullsized":"2024-06-29T14:41:59.525411+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.936045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532519+0000","last_change":"2024-06-29T14:41:58.534685+0000","last_active":"2024-06-29T14:41:59.532519+0000","last_peered":"2024-06-29T14:41:59.532519+0000","last_clean":"2024-06-29T14:41:59.532519+0000","last_became_active":"2024-06-29T14:41:58.534358+0000","last_became_peered":"2024-06-29T14:41:58.534358+0000","last_unstale":"2024-06-29T14:41:59.532519+0000","last_undegraded":"2024-06-29T14:41:59.532519+0000","last_fullsized":"2024-06-29T14:41:59.532519+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:40.532010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532070+0000","last_change":"2024-06-29T14:41:58.541428+0000","last_active":"2024-06-29T14:41:59.532070+0000","last_peered":"2024-06-29T14:41:59.532070+0000","last_clean":"2024-06-29T14:41:59.532070+0000","last_became_active":"2024-06-29T14:41:58.540979+0000","last_became_peered":"2024-06-29T14:41:58.540979+0000","last_unstale":"2024-06-29T14:41:59.532070+0000","last_undegraded":"2024-06-29T14:41:59.532070+0000","last_fullsized":"2024-06-29T14:41:59.532070+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:37.536052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532733+0000","last_change":"2024-06-29T14:41:58.518095+0000","last_active":"2024-06-29T14:41:59.532733+0000","last_peered":"2024-06-29T14:41:59.532733+0000","last_clean":"2024-06-29T14:41:59.532733+0000","last_became_active":"2024-06-29T14:41:58.517776+0000","last_became_peered":"2024-06-29T14:41:58.517776+0000","last_unstale":"2024-06-29T14:41:59.532733+0000","last_undegraded":"2024-06-29T14:41:59.532733+0000","last_fullsized":"2024-06-29T14:41:59.532733+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:46.537755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526349+0000","last_change":"2024-06-29T14:41:57.509834+0000","last_active":"2024-06-29T14:41:59.526349+0000","last_peered":"2024-06-29T14:41:59.526349+0000","last_clean":"2024-06-29T14:41:59.526349+0000","last_became_active":"2024-06-29T14:41:57.509525+0000","last_became_peered":"2024-06-29T14:41:57.509525+0000","last_unstale":"2024-06-29T14:41:59.526349+0000","last_undegraded":"2024-06-29T14:41:59.526349+0000","last_fullsized":"2024-06-29T14:41:59.526349+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:36.662713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533824+0000","last_change":"2024-06-29T14:41:58.533253+0000","last_active":"2024-06-29T14:41:59.533824+0000","last_peered":"2024-06-29T14:41:59.533824+0000","last_clean":"2024-06-29T14:41:59.533824+0000","last_became_active":"2024-06-29T14:41:58.532927+0000","last_became_peered":"2024-06-29T14:41:58.532927+0000","last_unstale":"2024-06-29T14:41:59.533824+0000","last_undegraded":"2024-06-29T14:41:59.533824+0000","last_fullsized":"2024-06-29T14:41:59.533824+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:41.902208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.524154+0000","last_change":"2024-06-29T14:41:57.511427+0000","last_active":"2024-06-29T14:41:59.524154+0000","last_peered":"2024-06-29T14:41:59.524154+0000","last_clean":"2024-06-29T14:41:59.524154+0000","last_became_active":"2024-06-29T14:41:57.511108+0000","last_became_peered":"2024-06-29T14:41:57.511108+0000","last_unstale":"2024-06-29T14:41:59.524154+0000","last_undegraded":"2024-06-29T14:41:59.524154+0000","last_fullsized":"2024-06-29T14:41:59.524154+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:59.499233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532665+0000","last_change":"2024-06-29T14:41:58.525359+0000","last_active":"2024-06-29T14:41:59.532665+0000","last_peered":"2024-06-29T14:41:59.532665+0000","last_clean":"2024-06-29T14:41:59.532665+0000","last_became_active":"2024-06-29T14:41:58.524403+0000","last_became_peered":"2024-06-29T14:41:58.524403+0000","last_unstale":"2024-06-29T14:41:59.532665+0000","last_undegraded":"2024-06-29T14:41:59.532665+0000","last_fullsized":"2024-06-29T14:41:59.532665+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:22.027012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527197+0000","last_change":"2024-06-29T14:41:57.512273+0000","last_active":"2024-06-29T14:41:59.527197+0000","last_peered":"2024-06-29T14:41:59.527197+0000","last_clean":"2024-06-29T14:41:59.527197+0000","last_became_active":"2024-06-29T14:41:57.511958+0000","last_became_peered":"2024-06-29T14:41:57.511958+0000","last_unstale":"2024-06-29T14:41:59.527197+0000","last_undegraded":"2024-06-29T14:41:59.527197+0000","last_fullsized":"2024-06-29T14:41:59.527197+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:26.744219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533061+0000","last_change":"2024-06-29T14:41:58.538082+0000","last_active":"2024-06-29T14:41:59.533061+0000","last_peered":"2024-06-29T14:41:59.533061+0000","last_clean":"2024-06-29T14:41:59.533061+0000","last_became_active":"2024-06-29T14:41:58.537792+0000","last_became_peered":"2024-06-29T14:41:58.537792+0000","last_unstale":"2024-06-29T14:41:59.533061+0000","last_undegraded":"2024-06-29T14:41:59.533061+0000","last_fullsized":"2024-06-29T14:41:59.533061+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:52.542145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526890+0000","last_change":"2024-06-29T14:41:57.506735+0000","last_active":"2024-06-29T14:41:59.526890+0000","last_peered":"2024-06-29T14:41:59.526890+0000","last_clean":"2024-06-29T14:41:59.526890+0000","last_became_active":"2024-06-29T14:41:57.506402+0000","last_became_peered":"2024-06-29T14:41:57.506402+0000","last_unstale":"2024-06-29T14:41:59.526890+0000","last_undegraded":"2024-06-29T14:41:59.526890+0000","last_fullsized":"2024-06-29T14:41:59.526890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:30:42.868318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533710+0000","last_change":"2024-06-29T14:41:58.534462+0000","last_active":"2024-06-29T14:41:59.533710+0000","last_peered":"2024-06-29T14:41:59.533710+0000","last_clean":"2024-06-29T14:41:59.533710+0000","last_became_active":"2024-06-29T14:41:58.534164+0000","last_became_peered":"2024-06-29T14:41:58.534164+0000","last_unstale":"2024-06-29T14:41:59.533710+0000","last_undegraded":"2024-06-29T14:41:59.533710+0000","last_fullsized":"2024-06-29T14:41:59.533710+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:26:57.805185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532809+0000","last_change":"2024-06-29T14:41:57.509711+0000","last_active":"2024-06-29T14:41:59.532809+0000","last_peered":"2024-06-29T14:41:59.532809+0000","last_clean":"2024-06-29T14:41:59.532809+0000","last_became_active":"2024-06-29T14:41:57.509425+0000","last_became_peered":"2024-06-29T14:41:57.509425+0000","last_unstale":"2024-06-29T14:41:59.532809+0000","last_undegraded":"2024-06-29T14:41:59.532809+0000","last_fullsized":"2024-06-29T14:41:59.532809+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:49.202916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532595+0000","last_change":"2024-06-29T14:41:58.525958+0000","last_active":"2024-06-29T14:41:59.532595+0000","last_peered":"2024-06-29T14:41:59.532595+0000","last_clean":"2024-06-29T14:41:59.532595+0000","last_became_active":"2024-06-29T14:41:58.525490+0000","last_became_peered":"2024-06-29T14:41:58.525490+0000","last_unstale":"2024-06-29T14:41:59.532595+0000","last_undegraded":"2024-06-29T14:41:59.532595+0000","last_fullsized":"2024-06-29T14:41:59.532595+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.936045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523507+0000","last_change":"2024-06-29T14:41:57.513810+0000","last_active":"2024-06-29T14:41:59.523507+0000","last_peered":"2024-06-29T14:41:59.523507+0000","last_clean":"2024-06-29T14:41:59.523507+0000","last_became_active":"2024-06-29T14:41:57.513515+0000","last_became_peered":"2024-06-29T14:41:57.513515+0000","last_unstale":"2024-06-29T14:41:59.523507+0000","last_undegraded":"2024-06-29T14:41:59.523507+0000","last_fullsized":"2024-06-29T14:41:59.523507+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:32.960566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.524032+0000","last_change":"2024-06-29T14:41:58.536776+0000","last_active":"2024-06-29T14:41:59.524032+0000","last_peered":"2024-06-29T14:41:59.524032+0000","last_clean":"2024-06-29T14:41:59.524032+0000","last_became_active":"2024-06-29T14:41:58.536415+0000","last_became_peered":"2024-06-29T14:41:58.536415+0000","last_unstale":"2024-06-29T14:41:59.524032+0000","last_undegraded":"2024-06-29T14:41:59.524032+0000","last_fullsized":"2024-06-29T14:41:59.524032+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:55:44.478520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532194+0000","last_change":"2024-06-29T14:41:57.517631+0000","last_active":"2024-06-29T14:41:59.532194+0000","last_peered":"2024-06-29T14:41:59.532194+0000","last_clean":"2024-06-29T14:41:59.532194+0000","last_became_active":"2024-06-29T14:41:57.517287+0000","last_became_peered":"2024-06-29T14:41:57.517287+0000","last_unstale":"2024-06-29T14:41:59.532194+0000","last_undegraded":"2024-06-29T14:41:59.532194+0000","last_fullsized":"2024-06-29T14:41:59.532194+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:56.507215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526148+0000","last_change":"2024-06-29T14:41:58.539564+0000","last_active":"2024-06-29T14:41:59.526148+0000","last_peered":"2024-06-29T14:41:59.526148+0000","last_clean":"2024-06-29T14:41:59.526148+0000","last_became_active":"2024-06-29T14:41:58.539031+0000","last_became_peered":"2024-06-29T14:41:58.539031+0000","last_unstale":"2024-06-29T14:41:59.526148+0000","last_undegraded":"2024-06-29T14:41:59.526148+0000","last_fullsized":"2024-06-29T14:41:59.526148+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:08.305916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525142+0000","last_change":"2024-06-29T14:41:57.513604+0000","last_active":"2024-06-29T14:41:59.525142+0000","last_peered":"2024-06-29T14:41:59.525142+0000","last_clean":"2024-06-29T14:41:59.525142+0000","last_became_active":"2024-06-29T14:41:57.513290+0000","last_became_peered":"2024-06-29T14:41:57.513290+0000","last_unstale":"2024-06-29T14:41:59.525142+0000","last_undegraded":"2024-06-29T14:41:59.525142+0000","last_fullsized":"2024-06-29T14:41:59.525142+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:36.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532920+0000","last_change":"2024-06-29T14:41:58.530597+0000","last_active":"2024-06-29T14:41:59.532920+0000","last_peered":"2024-06-29T14:41:59.532920+0000","last_clean":"2024-06-29T14:41:59.532920+0000","last_became_active":"2024-06-29T14:41:58.530246+0000","last_became_peered":"2024-06-29T14:41:58.530246+0000","last_unstale":"2024-06-29T14:41:59.532920+0000","last_undegraded":"2024-06-29T14:41:59.532920+0000","last_fullsized":"2024-06-29T14:41:59.532920+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:15:41.729768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533512+0000","last_change":"2024-06-29T14:41:57.515396+0000","last_active":"2024-06-29T14:41:59.533512+0000","last_peered":"2024-06-29T14:41:59.533512+0000","last_clean":"2024-06-29T14:41:59.533512+0000","last_became_active":"2024-06-29T14:41:57.515164+0000","last_became_peered":"2024-06-29T14:41:57.515164+0000","last_unstale":"2024-06-29T14:41:59.533512+0000","last_undegraded":"2024-06-29T14:41:59.533512+0000","last_fullsized":"2024-06-29T14:41:59.533512+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:33.025091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532368+0000","last_change":"2024-06-29T14:41:58.525131+0000","last_active":"2024-06-29T14:41:59.532368+0000","last_peered":"2024-06-29T14:41:59.532368+0000","last_clean":"2024-06-29T14:41:59.532368+0000","last_became_active":"2024-06-29T14:41:58.524299+0000","last_became_peered":"2024-06-29T14:41:58.524299+0000","last_unstale":"2024-06-29T14:41:59.532368+0000","last_undegraded":"2024-06-29T14:41:59.532368+0000","last_fullsized":"2024-06-29T14:41:59.532368+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:24:38.730535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533291+0000","last_change":"2024-06-29T14:41:57.510159+0000","last_active":"2024-06-29T14:41:59.533291+0000","last_peered":"2024-06-29T14:41:59.533291+0000","last_clean":"2024-06-29T14:41:59.533291+0000","last_became_active":"2024-06-29T14:41:57.509825+0000","last_became_peered":"2024-06-29T14:41:57.509825+0000","last_unstale":"2024-06-29T14:41:59.533291+0000","last_undegraded":"2024-06-29T14:41:59.533291+0000","last_fullsized":"2024-06-29T14:41:59.533291+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:53.480344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525488+0000","last_change":"2024-06-29T14:41:58.515800+0000","last_active":"2024-06-29T14:41:59.525488+0000","last_peered":"2024-06-29T14:41:59.525488+0000","last_clean":"2024-06-29T14:41:59.525488+0000","last_became_active":"2024-06-29T14:41:58.514758+0000","last_became_peered":"2024-06-29T14:41:58.514758+0000","last_unstale":"2024-06-29T14:41:59.525488+0000","last_undegraded":"2024-06-29T14:41:59.525488+0000","last_fullsized":"2024-06-29T14:41:59.525488+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:46.537755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527321+0000","last_change":"2024-06-29T14:41:57.511545+0000","last_active":"2024-06-29T14:41:59.527321+0000","last_peered":"2024-06-29T14:41:59.527321+0000","last_clean":"2024-06-29T14:41:59.527321+0000","last_became_active":"2024-06-29T14:41:57.510065+0000","last_became_peered":"2024-06-29T14:41:57.510065+0000","last_unstale":"2024-06-29T14:41:59.527321+0000","last_undegraded":"2024-06-29T14:41:59.527321+0000","last_fullsized":"2024-06-29T14:41:59.527321+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:56.648158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525707+0000","last_change":"2024-06-29T14:41:58.529387+0000","last_active":"2024-06-29T14:41:59.525707+0000","last_peered":"2024-06-29T14:41:59.525707+0000","last_clean":"2024-06-29T14:41:59.525707+0000","last_became_active":"2024-06-29T14:41:58.528942+0000","last_became_peered":"2024-06-29T14:41:58.528942+0000","last_unstale":"2024-06-29T14:41:59.525707+0000","last_undegraded":"2024-06-29T14:41:59.525707+0000","last_fullsized":"2024-06-29T14:41:59.525707+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:57.245802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527029+0000","last_change":"2024-06-29T14:41:57.516426+0000","last_active":"2024-06-29T14:41:59.527029+0000","last_peered":"2024-06-29T14:41:59.527029+0000","last_clean":"2024-06-29T14:41:59.527029+0000","last_became_active":"2024-06-29T14:41:57.516048+0000","last_became_peered":"2024-06-29T14:41:57.516048+0000","last_unstale":"2024-06-29T14:41:59.527029+0000","last_undegraded":"2024-06-29T14:41:59.527029+0000","last_fullsized":"2024-06-29T14:41:59.527029+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:38.386996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532407+0000","last_change":"2024-06-29T14:41:58.533628+0000","last_active":"2024-06-29T14:41:59.532407+0000","last_peered":"2024-06-29T14:41:59.532407+0000","last_clean":"2024-06-29T14:41:59.532407+0000","last_became_active":"2024-06-29T14:41:58.533280+0000","last_became_peered":"2024-06-29T14:41:58.533280+0000","last_unstale":"2024-06-29T14:41:59.532407+0000","last_undegraded":"2024-06-29T14:41:59.532407+0000","last_fullsized":"2024-06-29T14:41:59.532407+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:42.807014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533198+0000","last_change":"2024-06-29T14:41:57.512737+0000","last_active":"2024-06-29T14:41:59.533198+0000","last_peered":"2024-06-29T14:41:59.533198+0000","last_clean":"2024-06-29T14:41:59.533198+0000","last_became_active":"2024-06-29T14:41:57.512003+0000","last_became_peered":"2024-06-29T14:41:57.512003+0000","last_unstale":"2024-06-29T14:41:59.533198+0000","last_undegraded":"2024-06-29T14:41:59.533198+0000","last_fullsized":"2024-06-29T14:41:59.533198+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:49.503288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526256+0000","last_change":"2024-06-29T14:41:58.535064+0000","last_active":"2024-06-29T14:41:59.526256+0000","last_peered":"2024-06-29T14:41:59.526256+0000","last_clean":"2024-06-29T14:41:59.526256+0000","last_became_active":"2024-06-29T14:41:58.534654+0000","last_became_peered":"2024-06-29T14:41:58.534654+0000","last_unstale":"2024-06-29T14:41:59.526256+0000","last_undegraded":"2024-06-29T14:41:59.526256+0000","last_fullsized":"2024-06-29T14:41:59.526256+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:45.765189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525010+0000","last_change":"2024-06-29T14:41:57.511650+0000","last_active":"2024-06-29T14:41:59.525010+0000","last_peered":"2024-06-29T14:41:59.525010+0000","last_clean":"2024-06-29T14:41:59.525010+0000","last_became_active":"2024-06-29T14:41:57.511224+0000","last_became_peered":"2024-06-29T14:41:57.511224+0000","last_unstale":"2024-06-29T14:41:59.525010+0000","last_undegraded":"2024-06-29T14:41:59.525010+0000","last_fullsized":"2024-06-29T14:41:59.525010+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:53:28.408782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.524133+0000","last_change":"2024-06-29T14:41:58.531677+0000","last_active":"2024-06-29T14:41:59.524133+0000","last_peered":"2024-06-29T14:41:59.524133+0000","last_clean":"2024-06-29T14:41:59.524133+0000","last_became_active":"2024-06-29T14:41:58.531328+0000","last_became_peered":"2024-06-29T14:41:58.531328+0000","last_unstale":"2024-06-29T14:41:59.524133+0000","last_undegraded":"2024-06-29T14:41:59.524133+0000","last_fullsized":"2024-06-29T14:41:59.524133+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:05.489135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533289+0000","last_change":"2024-06-29T14:41:57.506405+0000","last_active":"2024-06-29T14:41:59.533289+0000","last_peered":"2024-06-29T14:41:59.533289+0000","last_clean":"2024-06-29T14:41:59.533289+0000","last_became_active":"2024-06-29T14:41:57.506117+0000","last_became_peered":"2024-06-29T14:41:57.506117+0000","last_unstale":"2024-06-29T14:41:59.533289+0000","last_undegraded":"2024-06-29T14:41:59.533289+0000","last_fullsized":"2024-06-29T14:41:59.533289+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:17.449570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525262+0000","last_change":"2024-06-29T14:41:58.523820+0000","last_active":"2024-06-29T14:41:59.525262+0000","last_peered":"2024-06-29T14:41:59.525262+0000","last_clean":"2024-06-29T14:41:59.525262+0000","last_became_active":"2024-06-29T14:41:58.523477+0000","last_became_peered":"2024-06-29T14:41:58.523477+0000","last_unstale":"2024-06-29T14:41:59.525262+0000","last_undegraded":"2024-06-29T14:41:59.525262+0000","last_fullsized":"2024-06-29T14:41:59.525262+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:40.532010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533635+0000","last_change":"2024-06-29T14:41:57.512385+0000","last_active":"2024-06-29T14:41:59.533635+0000","last_peered":"2024-06-29T14:41:59.533635+0000","last_clean":"2024-06-29T14:41:59.533635+0000","last_became_active":"2024-06-29T14:41:57.512034+0000","last_became_peered":"2024-06-29T14:41:57.512034+0000","last_unstale":"2024-06-29T14:41:59.533635+0000","last_undegraded":"2024-06-29T14:41:59.533635+0000","last_fullsized":"2024-06-29T14:41:59.533635+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:29.246165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526797+0000","last_change":"2024-06-29T14:41:58.537987+0000","last_active":"2024-06-29T14:41:59.526797+0000","last_peered":"2024-06-29T14:41:59.526797+0000","last_clean":"2024-06-29T14:41:59.526797+0000","last_became_active":"2024-06-29T14:41:58.537327+0000","last_became_peered":"2024-06-29T14:41:58.537327+0000","last_unstale":"2024-06-29T14:41:59.526797+0000","last_undegraded":"2024-06-29T14:41:59.526797+0000","last_fullsized":"2024-06-29T14:41:59.526797+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:30.664437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525925+0000","last_change":"2024-06-29T14:41:57.508397+0000","last_active":"2024-06-29T14:41:59.525925+0000","last_peered":"2024-06-29T14:41:59.525925+0000","last_clean":"2024-06-29T14:41:59.525925+0000","last_became_active":"2024-06-29T14:41:57.508099+0000","last_became_peered":"2024-06-29T14:41:57.508099+0000","last_unstale":"2024-06-29T14:41:59.525925+0000","last_undegraded":"2024-06-29T14:41:59.525925+0000","last_fullsized":"2024-06-29T14:41:59.525925+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:33.619707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527291+0000","last_change":"2024-06-29T14:41:58.539325+0000","last_active":"2024-06-29T14:41:59.527291+0000","last_peered":"2024-06-29T14:41:59.527291+0000","last_clean":"2024-06-29T14:41:59.527291+0000","last_became_active":"2024-06-29T14:41:58.538983+0000","last_became_peered":"2024-06-29T14:41:58.538983+0000","last_unstale":"2024-06-29T14:41:59.527291+0000","last_undegraded":"2024-06-29T14:41:59.527291+0000","last_fullsized":"2024-06-29T14:41:59.527291+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:51.468237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533027+0000","last_change":"2024-06-29T14:41:57.516894+0000","last_active":"2024-06-29T14:41:59.533027+0000","last_peered":"2024-06-29T14:41:59.533027+0000","last_clean":"2024-06-29T14:41:59.533027+0000","last_became_active":"2024-06-29T14:41:57.516556+0000","last_became_peered":"2024-06-29T14:41:57.516556+0000","last_unstale":"2024-06-29T14:41:59.533027+0000","last_undegraded":"2024-06-29T14:41:59.533027+0000","last_fullsized":"2024-06-29T14:41:59.533027+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:10.507355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532698+0000","last_change":"2024-06-29T14:41:58.530241+0000","last_active":"2024-06-29T14:41:59.532698+0000","last_peered":"2024-06-29T14:41:59.532698+0000","last_clean":"2024-06-29T14:41:59.532698+0000","last_became_active":"2024-06-29T14:41:58.529847+0000","last_became_peered":"2024-06-29T14:41:58.529847+0000","last_unstale":"2024-06-29T14:41:59.532698+0000","last_undegraded":"2024-06-29T14:41:59.532698+0000","last_fullsized":"2024-06-29T14:41:59.532698+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:15.787250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523891+0000","last_change":"2024-06-29T14:41:57.510198+0000","last_active":"2024-06-29T14:41:59.523891+0000","last_peered":"2024-06-29T14:41:59.523891+0000","last_clean":"2024-06-29T14:41:59.523891+0000","last_became_active":"2024-06-29T14:41:57.509901+0000","last_became_peered":"2024-06-29T14:41:57.509901+0000","last_unstale":"2024-06-29T14:41:59.523891+0000","last_undegraded":"2024-06-29T14:41:59.523891+0000","last_fullsized":"2024-06-29T14:41:59.523891+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:59:04.655016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523852+0000","last_change":"2024-06-29T14:41:58.527478+0000","last_active":"2024-06-29T14:41:59.523852+0000","last_peered":"2024-06-29T14:41:59.523852+0000","last_clean":"2024-06-29T14:41:59.523852+0000","last_became_active":"2024-06-29T14:41:58.527151+0000","last_became_peered":"2024-06-29T14:41:58.527151+0000","last_unstale":"2024-06-29T14:41:59.523852+0000","last_undegraded":"2024-06-29T14:41:59.523852+0000","last_fullsized":"2024-06-29T14:41:59.523852+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:08.124503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533062+0000","last_change":"2024-06-29T14:41:57.507138+0000","last_active":"2024-06-29T14:41:59.533062+0000","last_peered":"2024-06-29T14:41:59.533062+0000","last_clean":"2024-06-29T14:41:59.533062+0000","last_became_active":"2024-06-29T14:41:57.506796+0000","last_became_peered":"2024-06-29T14:41:57.506796+0000","last_unstale":"2024-06-29T14:41:59.533062+0000","last_undegraded":"2024-06-29T14:41:59.533062+0000","last_fullsized":"2024-06-29T14:41:59.533062+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:39:55.074608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533363+0000","last_change":"2024-06-29T14:41:58.541451+0000","last_active":"2024-06-29T14:41:59.533363+0000","last_peered":"2024-06-29T14:41:59.533363+0000","last_clean":"2024-06-29T14:41:59.533363+0000","last_became_active":"2024-06-29T14:41:58.541145+0000","last_became_peered":"2024-06-29T14:41:58.541145+0000","last_unstale":"2024-06-29T14:41:59.533363+0000","last_undegraded":"2024-06-29T14:41:59.533363+0000","last_fullsized":"2024-06-29T14:41:59.533363+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:56.963338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533538+0000","last_change":"2024-06-29T14:41:57.517317+0000","last_active":"2024-06-29T14:41:59.533538+0000","last_peered":"2024-06-29T14:41:59.533538+0000","last_clean":"2024-06-29T14:41:59.533538+0000","last_became_active":"2024-06-29T14:41:57.516971+0000","last_became_peered":"2024-06-29T14:41:57.516971+0000","last_unstale":"2024-06-29T14:41:59.533538+0000","last_undegraded":"2024-06-29T14:41:59.533538+0000","last_fullsized":"2024-06-29T14:41:59.533538+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:55.425911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.524097+0000","last_change":"2024-06-29T14:41:58.526944+0000","last_active":"2024-06-29T14:41:59.524097+0000","last_peered":"2024-06-29T14:41:59.524097+0000","last_clean":"2024-06-29T14:41:59.524097+0000","last_became_active":"2024-06-29T14:41:58.526617+0000","last_became_peered":"2024-06-29T14:41:58.526617+0000","last_unstale":"2024-06-29T14:41:59.524097+0000","last_undegraded":"2024-06-29T14:41:59.524097+0000","last_fullsized":"2024-06-29T14:41:59.524097+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:22.027012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526898+0000","last_change":"2024-06-29T14:41:57.510993+0000","last_active":"2024-06-29T14:41:59.526898+0000","last_peered":"2024-06-29T14:41:59.526898+0000","last_clean":"2024-06-29T14:41:59.526898+0000","last_became_active":"2024-06-29T14:41:57.510650+0000","last_became_peered":"2024-06-29T14:41:57.510650+0000","last_unstale":"2024-06-29T14:41:59.526898+0000","last_undegraded":"2024-06-29T14:41:59.526898+0000","last_fullsized":"2024-06-29T14:41:59.526898+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:06.750598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533376+0000","last_change":"2024-06-29T14:41:58.517623+0000","last_active":"2024-06-29T14:41:59.533376+0000","last_peered":"2024-06-29T14:41:59.533376+0000","last_clean":"2024-06-29T14:41:59.533376+0000","last_became_active":"2024-06-29T14:41:58.517323+0000","last_became_peered":"2024-06-29T14:41:58.517323+0000","last_unstale":"2024-06-29T14:41:59.533376+0000","last_undegraded":"2024-06-29T14:41:59.533376+0000","last_fullsized":"2024-06-29T14:41:59.533376+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:42.807014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533409+0000","last_change":"2024-06-29T14:41:57.510104+0000","last_active":"2024-06-29T14:41:59.533409+0000","last_peered":"2024-06-29T14:41:59.533409+0000","last_clean":"2024-06-29T14:41:59.533409+0000","last_became_active":"2024-06-29T14:41:57.509745+0000","last_became_peered":"2024-06-29T14:41:57.509745+0000","last_unstale":"2024-06-29T14:41:59.533409+0000","last_undegraded":"2024-06-29T14:41:59.533409+0000","last_fullsized":"2024-06-29T14:41:59.533409+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:49.707683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523706+0000","last_change":"2024-06-29T14:41:58.526235+0000","last_active":"2024-06-29T14:41:59.523706+0000","last_peered":"2024-06-29T14:41:59.523706+0000","last_clean":"2024-06-29T14:41:59.523706+0000","last_became_active":"2024-06-29T14:41:58.525871+0000","last_became_peered":"2024-06-29T14:41:58.525871+0000","last_unstale":"2024-06-29T14:41:59.523706+0000","last_undegraded":"2024-06-29T14:41:59.523706+0000","last_fullsized":"2024-06-29T14:41:59.523706+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:49.063709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533972+0000","last_change":"2024-06-29T14:41:57.515930+0000","last_active":"2024-06-29T14:41:59.533972+0000","last_peered":"2024-06-29T14:41:59.533972+0000","last_clean":"2024-06-29T14:41:59.533972+0000","last_became_active":"2024-06-29T14:41:57.515603+0000","last_became_peered":"2024-06-29T14:41:57.515603+0000","last_unstale":"2024-06-29T14:41:59.533972+0000","last_undegraded":"2024-06-29T14:41:59.533972+0000","last_fullsized":"2024-06-29T14:41:59.533972+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:19.701123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525136+0000","last_change":"2024-06-29T14:41:58.529192+0000","last_active":"2024-06-29T14:41:59.525136+0000","last_peered":"2024-06-29T14:41:59.525136+0000","last_clean":"2024-06-29T14:41:59.525136+0000","last_became_active":"2024-06-29T14:41:58.528804+0000","last_became_peered":"2024-06-29T14:41:58.528804+0000","last_unstale":"2024-06-29T14:41:59.525136+0000","last_undegraded":"2024-06-29T14:41:59.525136+0000","last_fullsized":"2024-06-29T14:41:59.525136+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:15:32.748190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533247+0000","last_change":"2024-06-29T14:41:57.512008+0000","last_active":"2024-06-29T14:41:59.533247+0000","last_peered":"2024-06-29T14:41:59.533247+0000","last_clean":"2024-06-29T14:41:59.533247+0000","last_became_active":"2024-06-29T14:41:57.511681+0000","last_became_peered":"2024-06-29T14:41:57.511681+0000","last_unstale":"2024-06-29T14:41:59.533247+0000","last_undegraded":"2024-06-29T14:41:59.533247+0000","last_fullsized":"2024-06-29T14:41:59.533247+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:34.612389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527017+0000","last_change":"2024-06-29T14:41:58.534327+0000","last_active":"2024-06-29T14:41:59.527017+0000","last_peered":"2024-06-29T14:41:59.527017+0000","last_clean":"2024-06-29T14:41:59.527017+0000","last_became_active":"2024-06-29T14:41:58.533374+0000","last_became_peered":"2024-06-29T14:41:58.533374+0000","last_unstale":"2024-06-29T14:41:59.527017+0000","last_undegraded":"2024-06-29T14:41:59.527017+0000","last_fullsized":"2024-06-29T14:41:59.527017+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:35.653870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532890+0000","last_change":"2024-06-29T14:41:57.509359+0000","last_active":"2024-06-29T14:41:59.532890+0000","last_peered":"2024-06-29T14:41:59.532890+0000","last_clean":"2024-06-29T14:41:59.532890+0000","last_became_active":"2024-06-29T14:41:57.509067+0000","last_became_peered":"2024-06-29T14:41:57.509067+0000","last_unstale":"2024-06-29T14:41:59.532890+0000","last_undegraded":"2024-06-29T14:41:59.532890+0000","last_fullsized":"2024-06-29T14:41:59.532890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:52:22.009454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532863+0000","last_change":"2024-06-29T14:41:58.529484+0000","last_active":"2024-06-29T14:41:59.532863+0000","last_peered":"2024-06-29T14:41:59.532863+0000","last_clean":"2024-06-29T14:41:59.532863+0000","last_became_active":"2024-06-29T14:41:58.529160+0000","last_became_peered":"2024-06-29T14:41:58.529160+0000","last_unstale":"2024-06-29T14:41:59.532863+0000","last_undegraded":"2024-06-29T14:41:59.532863+0000","last_fullsized":"2024-06-29T14:41:59.532863+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:54.490869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532833+0000","last_change":"2024-06-29T14:41:57.504027+0000","last_active":"2024-06-29T14:41:59.532833+0000","last_peered":"2024-06-29T14:41:59.532833+0000","last_clean":"2024-06-29T14:41:59.532833+0000","last_became_active":"2024-06-29T14:41:57.503586+0000","last_became_peered":"2024-06-29T14:41:57.503586+0000","last_unstale":"2024-06-29T14:41:59.532833+0000","last_undegraded":"2024-06-29T14:41:59.532833+0000","last_fullsized":"2024-06-29T14:41:59.532833+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:15.788541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532731+0000","last_change":"2024-06-29T14:41:58.533438+0000","last_active":"2024-06-29T14:41:59.532731+0000","last_peered":"2024-06-29T14:41:59.532731+0000","last_clean":"2024-06-29T14:41:59.532731+0000","last_became_active":"2024-06-29T14:41:58.533140+0000","last_became_peered":"2024-06-29T14:41:58.533140+0000","last_unstale":"2024-06-29T14:41:59.532731+0000","last_undegraded":"2024-06-29T14:41:59.532731+0000","last_fullsized":"2024-06-29T14:41:59.532731+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:51.449930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532833+0000","last_change":"2024-06-29T14:41:57.513038+0000","last_active":"2024-06-29T14:41:59.532833+0000","last_peered":"2024-06-29T14:41:59.532833+0000","last_clean":"2024-06-29T14:41:59.532833+0000","last_became_active":"2024-06-29T14:41:57.512729+0000","last_became_peered":"2024-06-29T14:41:57.512729+0000","last_unstale":"2024-06-29T14:41:59.532833+0000","last_undegraded":"2024-06-29T14:41:59.532833+0000","last_fullsized":"2024-06-29T14:41:59.532833+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:18.745856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533034+0000","last_change":"2024-06-29T14:41:58.531798+0000","last_active":"2024-06-29T14:41:59.533034+0000","last_peered":"2024-06-29T14:41:59.533034+0000","last_clean":"2024-06-29T14:41:59.533034+0000","last_became_active":"2024-06-29T14:41:58.531499+0000","last_became_peered":"2024-06-29T14:41:58.531499+0000","last_unstale":"2024-06-29T14:41:59.533034+0000","last_undegraded":"2024-06-29T14:41:59.533034+0000","last_fullsized":"2024-06-29T14:41:59.533034+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:28.287087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527282+0000","last_change":"2024-06-29T14:41:57.506196+0000","last_active":"2024-06-29T14:41:59.527282+0000","last_peered":"2024-06-29T14:41:59.527282+0000","last_clean":"2024-06-29T14:41:59.527282+0000","last_became_active":"2024-06-29T14:41:57.505842+0000","last_became_peered":"2024-06-29T14:41:57.505842+0000","last_unstale":"2024-06-29T14:41:59.527282+0000","last_undegraded":"2024-06-29T14:41:59.527282+0000","last_fullsized":"2024-06-29T14:41:59.527282+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:36.993862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532603+0000","last_change":"2024-06-29T14:41:58.527859+0000","last_active":"2024-06-29T14:41:59.532603+0000","last_peered":"2024-06-29T14:41:59.532603+0000","last_clean":"2024-06-29T14:41:59.532603+0000","last_became_active":"2024-06-29T14:41:58.527463+0000","last_became_peered":"2024-06-29T14:41:58.527463+0000","last_unstale":"2024-06-29T14:41:59.532603+0000","last_undegraded":"2024-06-29T14:41:59.532603+0000","last_fullsized":"2024-06-29T14:41:59.532603+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:03.105937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532494+0000","last_change":"2024-06-29T14:41:57.505760+0000","last_active":"2024-06-29T14:41:59.532494+0000","last_peered":"2024-06-29T14:41:59.532494+0000","last_clean":"2024-06-29T14:41:59.532494+0000","last_became_active":"2024-06-29T14:41:57.505476+0000","last_became_peered":"2024-06-29T14:41:57.505476+0000","last_unstale":"2024-06-29T14:41:59.532494+0000","last_undegraded":"2024-06-29T14:41:59.532494+0000","last_fullsized":"2024-06-29T14:41:59.532494+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:15.695601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526324+0000","last_change":"2024-06-29T14:41:58.533381+0000","last_active":"2024-06-29T14:41:59.526324+0000","last_peered":"2024-06-29T14:41:59.526324+0000","last_clean":"2024-06-29T14:41:59.526324+0000","last_became_active":"2024-06-29T14:41:58.533071+0000","last_became_peered":"2024-06-29T14:41:58.533071+0000","last_unstale":"2024-06-29T14:41:59.526324+0000","last_undegraded":"2024-06-29T14:41:59.526324+0000","last_fullsized":"2024-06-29T14:41:59.526324+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:24:41.804258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523827+0000","last_change":"2024-06-29T14:41:57.515433+0000","last_active":"2024-06-29T14:41:59.523827+0000","last_peered":"2024-06-29T14:41:59.523827+0000","last_clean":"2024-06-29T14:41:59.523827+0000","last_became_active":"2024-06-29T14:41:57.515134+0000","last_became_peered":"2024-06-29T14:41:57.515134+0000","last_unstale":"2024-06-29T14:41:59.523827+0000","last_undegraded":"2024-06-29T14:41:59.523827+0000","last_fullsized":"2024-06-29T14:41:59.523827+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:09.488545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525725+0000","last_change":"2024-06-29T14:41:58.524691+0000","last_active":"2024-06-29T14:41:59.525725+0000","last_peered":"2024-06-29T14:41:59.525725+0000","last_clean":"2024-06-29T14:41:59.525725+0000","last_became_active":"2024-06-29T14:41:58.523946+0000","last_became_peered":"2024-06-29T14:41:58.523946+0000","last_unstale":"2024-06-29T14:41:59.525725+0000","last_undegraded":"2024-06-29T14:41:59.525725+0000","last_fullsized":"2024-06-29T14:41:59.525725+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:37:02.005962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527130+0000","last_change":"2024-06-29T14:41:57.512547+0000","last_active":"2024-06-29T14:41:59.527130+0000","last_peered":"2024-06-29T14:41:59.527130+0000","last_clean":"2024-06-29T14:41:59.527130+0000","last_became_active":"2024-06-29T14:41:57.512183+0000","last_became_peered":"2024-06-29T14:41:57.512183+0000","last_unstale":"2024-06-29T14:41:59.527130+0000","last_undegraded":"2024-06-29T14:41:59.527130+0000","last_fullsized":"2024-06-29T14:41:59.527130+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:16:51.038914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"145'170","reported_seq":719,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525766+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:41:59.525766+0000","last_peered":"2024-06-29T14:41:59.525766+0000","last_clean":"2024-06-29T14:41:59.525766+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:41:59.525766+0000","last_undegraded":"2024-06-29T14:41:59.525766+0000","last_fullsized":"2024-06-29T14:41:59.525766+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":269,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":269,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542386,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60040,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311800,"statfs":{"total":96636764160,"available":96575283200,"internally_reserved":0,"allocated":2547712,"data_stored":1575577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542387,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1933312,"data_stored":977459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542386,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1929216,"data_stored":976809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542385,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1937408,"data_stored":983507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509683,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1953792,"data_stored":985189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575091,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59456,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312384,"statfs":{"total":96636764160,"available":96575881216,"internally_reserved":0,"allocated":1945600,"data_stored":984123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509682,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1941504,"data_stored":983577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509684,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37808,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334032,"statfs":{"total":96636764160,"available":96598048768,"internally_reserved":0,"allocated":2527232,"data_stored":1567211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:42:02.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:42:02.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.770+0000 7fcc04963640 1 -- 172.21.15.59:0/1218217831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 msgr2=0x7fcbec004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.770+0000 7fcc04963640 1 --2- 172.21.15.59:0/1218217831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 0x7fcbec004c10 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009b20 tx=0x7fcbec005830 comp rx=0 tx=0).stop 2024-06-29T14:42:02.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.770+0000 7fcc04963640 1 -- 172.21.15.59:0/1218217831 shutdown_connections 2024-06-29T14:42:02.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.770+0000 7fcc04963640 1 --2- 172.21.15.59:0/1218217831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 0x7fcbec004c10 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.770+0000 7fcc04963640 1 -- 172.21.15.59:0/1218217831 >> 172.21.15.59:0/1218217831 conn(0x7fcc000b6fb0 msgr2=0x7fcc000b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.770+0000 7fcc04963640 1 -- 172.21.15.59:0/1218217831 shutdown_connections 2024-06-29T14:42:02.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.770+0000 7fcc04963640 1 -- 172.21.15.59:0/1218217831 wait complete. 2024-06-29T14:42:02.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.772+0000 7fcc04963640 1 Processor -- start 2024-06-29T14:42:02.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.772+0000 7fcc04963640 1 -- start start 2024-06-29T14:42:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.773+0000 7fcc04963640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 0x7fcc00155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.773+0000 7fcc04963640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc00148220 0x7fcc00158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.773+0000 7fcc04963640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc0006ed50 0x7fcc0006f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.773+0000 7fcc04963640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcc000b5850 con 0x7fcbec004800 2024-06-29T14:42:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.773+0000 7fcc04963640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc000b56d0 con 0x7fcc00148220 2024-06-29T14:42:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.773+0000 7fcc04963640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcc000b59d0 con 0x7fcc0006ed50 2024-06-29T14:42:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.773+0000 7fcbfe59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 0x7fcc00155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.773+0000 7fcbfe59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 0x7fcc00155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36502/0 (socket says 172.21.15.59:36502) 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfe59c640 1 -- 172.21.15.59:0/874567621 learned_addr learned my addr 172.21.15.59:0/874567621 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfed9d640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc0006ed50 0x7fcc0006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfdd9b640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc00148220 0x7fcc00158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfe59c640 1 -- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc0006ed50 msgr2=0x7fcc0006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfe59c640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc0006ed50 0x7fcc0006f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfe59c640 1 -- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc00148220 msgr2=0x7fcc00158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfe59c640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc00148220 0x7fcc00158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfe59c640 1 -- 172.21.15.59:0/874567621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc00158f90 con 0x7fcbec004800 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbfe59c640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 0x7fcc00155150 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80066b0 tx=0x7fcbf80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbf6ffd640 1 -- 172.21.15.59:0/874567621 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8011700 con 0x7fcbec004800 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbf6ffd640 1 -- 172.21.15.59:0/874567621 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbf8016020 con 0x7fcbec004800 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcbf6ffd640 1 -- 172.21.15.59:0/874567621 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8011b60 con 0x7fcbec004800 2024-06-29T14:42:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0006dcb0 con 0x7fcbec004800 2024-06-29T14:42:02.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.774+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0006e270 con 0x7fcbec004800 2024-06-29T14:42:02.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.775+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc001419b0 con 0x7fcbec004800 2024-06-29T14:42:02.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.776+0000 7fcbf6ffd640 1 -- 172.21.15.59:0/874567621 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fcbf8011d00 con 0x7fcbec004800 2024-06-29T14:42:02.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.777+0000 7fcbf6ffd640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fcbe00763f0 0x7fcbe0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.777+0000 7fcbf6ffd640 1 -- 172.21.15.59:0/874567621 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7fcbf80984a0 con 0x7fcbec004800 2024-06-29T14:42:02.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.777+0000 7fcbfdd9b640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fcbe00763f0 0x7fcbe0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.777+0000 7fcbfdd9b640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fcbe00763f0 0x7fcbe0078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8035000 tx=0x7fcbe802d040 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.781+0000 7fcbf6ffd640 1 -- 172.21.15.59:0/874567621 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcbf8020020 con 0x7fcbec004800 2024-06-29T14:42:02.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.968+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 --> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc00160660 con 0x7fcbe00763f0 2024-06-29T14:42:02.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.973+0000 7fcbf6ffd640 1 -- 172.21.15.59:0/874567621 <== mgr.7658 v2:172.21.15.153:6832/3378196672 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252485 (secure 0 0 0) 0x7fcc00160660 con 0x7fcbe00763f0 2024-06-29T14:42:02.972 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:42:02.973 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T14:42:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fcbe00763f0 msgr2=0x7fcbe0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fcbe00763f0 0x7fcbe0078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8035000 tx=0x7fcbe802d040 comp rx=0 tx=0).stop 2024-06-29T14:42:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 msgr2=0x7fcc00155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 0x7fcc00155150 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80066b0 tx=0x7fcbf80039c0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 shutdown_connections 2024-06-29T14:42:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fcbe00763f0 0x7fcbe0078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc0006ed50 0x7fcc0006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc00148220 0x7fcc00158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 --2- 172.21.15.59:0/874567621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec004800 0x7fcc00155150 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 >> 172.21.15.59:0/874567621 conn(0x7fcc000b6fb0 msgr2=0x7fcc0013e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 shutdown_connections 2024-06-29T14:42:02.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:02.976+0000 7fcc04963640 1 -- 172.21.15.59:0/874567621 wait complete. 2024-06-29T14:42:02.984 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:42:01.986038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":314,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":361536,"kb_used_data":16324,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754613184,"statfs":{"total":773094113280,"available":772723900416,"internally_reserved":0,"allocated":16715776,"data_stored":9033452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":353357362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.496476"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533901+0000","last_change":"2024-06-29T14:41:58.534161+0000","last_active":"2024-06-29T14:41:59.533901+0000","last_peered":"2024-06-29T14:41:59.533901+0000","last_clean":"2024-06-29T14:41:59.533901+0000","last_became_active":"2024-06-29T14:41:58.532120+0000","last_became_peered":"2024-06-29T14:41:58.532120+0000","last_unstale":"2024-06-29T14:41:59.533901+0000","last_undegraded":"2024-06-29T14:41:59.533901+0000","last_fullsized":"2024-06-29T14:41:59.533901+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:16.350098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533658+0000","last_change":"2024-06-29T14:41:58.518549+0000","last_active":"2024-06-29T14:41:59.533658+0000","last_peered":"2024-06-29T14:41:59.533658+0000","last_clean":"2024-06-29T14:41:59.533658+0000","last_became_active":"2024-06-29T14:41:58.518244+0000","last_became_peered":"2024-06-29T14:41:58.518244+0000","last_unstale":"2024-06-29T14:41:59.533658+0000","last_undegraded":"2024-06-29T14:41:59.533658+0000","last_fullsized":"2024-06-29T14:41:59.533658+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:15.787250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525589+0000","last_change":"2024-06-29T14:41:58.525851+0000","last_active":"2024-06-29T14:41:59.525589+0000","last_peered":"2024-06-29T14:41:59.525589+0000","last_clean":"2024-06-29T14:41:59.525589+0000","last_became_active":"2024-06-29T14:41:58.524386+0000","last_became_peered":"2024-06-29T14:41:58.524386+0000","last_unstale":"2024-06-29T14:41:59.525589+0000","last_undegraded":"2024-06-29T14:41:59.525589+0000","last_fullsized":"2024-06-29T14:41:59.525589+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:26:57.805185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533027+0000","last_change":"2024-06-29T14:41:58.527479+0000","last_active":"2024-06-29T14:41:59.533027+0000","last_peered":"2024-06-29T14:41:59.533027+0000","last_clean":"2024-06-29T14:41:59.533027+0000","last_became_active":"2024-06-29T14:41:58.527091+0000","last_became_peered":"2024-06-29T14:41:58.527091+0000","last_unstale":"2024-06-29T14:41:59.533027+0000","last_undegraded":"2024-06-29T14:41:59.533027+0000","last_fullsized":"2024-06-29T14:41:59.533027+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:37.536052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523583+0000","last_change":"2024-06-29T14:41:58.524582+0000","last_active":"2024-06-29T14:41:59.523583+0000","last_peered":"2024-06-29T14:41:59.523583+0000","last_clean":"2024-06-29T14:41:59.523583+0000","last_became_active":"2024-06-29T14:41:58.524257+0000","last_became_peered":"2024-06-29T14:41:58.524257+0000","last_unstale":"2024-06-29T14:41:59.523583+0000","last_undegraded":"2024-06-29T14:41:59.523583+0000","last_fullsized":"2024-06-29T14:41:59.523583+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:03.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532826+0000","last_change":"2024-06-29T14:41:58.518349+0000","last_active":"2024-06-29T14:41:59.532826+0000","last_peered":"2024-06-29T14:41:59.532826+0000","last_clean":"2024-06-29T14:41:59.532826+0000","last_became_active":"2024-06-29T14:41:58.517991+0000","last_became_peered":"2024-06-29T14:41:58.517991+0000","last_unstale":"2024-06-29T14:41:59.532826+0000","last_undegraded":"2024-06-29T14:41:59.532826+0000","last_fullsized":"2024-06-29T14:41:59.532826+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:33.500996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526671+0000","last_change":"2024-06-29T14:41:58.529976+0000","last_active":"2024-06-29T14:41:59.526671+0000","last_peered":"2024-06-29T14:41:59.526671+0000","last_clean":"2024-06-29T14:41:59.526671+0000","last_became_active":"2024-06-29T14:41:58.529667+0000","last_became_peered":"2024-06-29T14:41:58.529667+0000","last_unstale":"2024-06-29T14:41:59.526671+0000","last_undegraded":"2024-06-29T14:41:59.526671+0000","last_fullsized":"2024-06-29T14:41:59.526671+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:53:19.720982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532519+0000","last_change":"2024-06-29T14:41:58.526786+0000","last_active":"2024-06-29T14:41:59.532519+0000","last_peered":"2024-06-29T14:41:59.532519+0000","last_clean":"2024-06-29T14:41:59.532519+0000","last_became_active":"2024-06-29T14:41:58.526479+0000","last_became_peered":"2024-06-29T14:41:58.526479+0000","last_unstale":"2024-06-29T14:41:59.532519+0000","last_undegraded":"2024-06-29T14:41:59.532519+0000","last_fullsized":"2024-06-29T14:41:59.532519+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:15:32.748190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533363+0000","last_change":"2024-06-29T14:41:58.517597+0000","last_active":"2024-06-29T14:41:59.533363+0000","last_peered":"2024-06-29T14:41:59.533363+0000","last_clean":"2024-06-29T14:41:59.533363+0000","last_became_active":"2024-06-29T14:41:58.517218+0000","last_became_peered":"2024-06-29T14:41:58.517218+0000","last_unstale":"2024-06-29T14:41:59.533363+0000","last_undegraded":"2024-06-29T14:41:59.533363+0000","last_fullsized":"2024-06-29T14:41:59.533363+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:06.338200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533191+0000","last_change":"2024-06-29T14:41:58.530549+0000","last_active":"2024-06-29T14:41:59.533191+0000","last_peered":"2024-06-29T14:41:59.533191+0000","last_clean":"2024-06-29T14:41:59.533191+0000","last_became_active":"2024-06-29T14:41:58.530202+0000","last_became_peered":"2024-06-29T14:41:58.530202+0000","last_unstale":"2024-06-29T14:41:59.533191+0000","last_undegraded":"2024-06-29T14:41:59.533191+0000","last_fullsized":"2024-06-29T14:41:59.533191+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:30.652977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533760+0000","last_change":"2024-06-29T14:41:58.516764+0000","last_active":"2024-06-29T14:41:59.533760+0000","last_peered":"2024-06-29T14:41:59.533760+0000","last_clean":"2024-06-29T14:41:59.533760+0000","last_became_active":"2024-06-29T14:41:58.516446+0000","last_became_peered":"2024-06-29T14:41:58.516446+0000","last_unstale":"2024-06-29T14:41:59.533760+0000","last_undegraded":"2024-06-29T14:41:59.533760+0000","last_fullsized":"2024-06-29T14:41:59.533760+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:54.490869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533695+0000","last_change":"2024-06-29T14:41:58.534977+0000","last_active":"2024-06-29T14:41:59.533695+0000","last_peered":"2024-06-29T14:41:59.533695+0000","last_clean":"2024-06-29T14:41:59.533695+0000","last_became_active":"2024-06-29T14:41:58.534644+0000","last_became_peered":"2024-06-29T14:41:58.534644+0000","last_unstale":"2024-06-29T14:41:59.533695+0000","last_undegraded":"2024-06-29T14:41:59.533695+0000","last_fullsized":"2024-06-29T14:41:59.533695+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:57.768219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532551+0000","last_change":"2024-06-29T14:41:58.519217+0000","last_active":"2024-06-29T14:41:59.532551+0000","last_peered":"2024-06-29T14:41:59.532551+0000","last_clean":"2024-06-29T14:41:59.532551+0000","last_became_active":"2024-06-29T14:41:58.518637+0000","last_became_peered":"2024-06-29T14:41:58.518637+0000","last_unstale":"2024-06-29T14:41:59.532551+0000","last_undegraded":"2024-06-29T14:41:59.532551+0000","last_fullsized":"2024-06-29T14:41:59.532551+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:17.761380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533131+0000","last_change":"2024-06-29T14:41:58.538984+0000","last_active":"2024-06-29T14:41:59.533131+0000","last_peered":"2024-06-29T14:41:59.533131+0000","last_clean":"2024-06-29T14:41:59.533131+0000","last_became_active":"2024-06-29T14:41:58.538653+0000","last_became_peered":"2024-06-29T14:41:58.538653+0000","last_unstale":"2024-06-29T14:41:59.533131+0000","last_undegraded":"2024-06-29T14:41:59.533131+0000","last_fullsized":"2024-06-29T14:41:59.533131+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:56:52.580045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533639+0000","last_change":"2024-06-29T14:41:58.539744+0000","last_active":"2024-06-29T14:41:59.533639+0000","last_peered":"2024-06-29T14:41:59.533639+0000","last_clean":"2024-06-29T14:41:59.533639+0000","last_became_active":"2024-06-29T14:41:58.539449+0000","last_became_peered":"2024-06-29T14:41:58.539449+0000","last_unstale":"2024-06-29T14:41:59.533639+0000","last_undegraded":"2024-06-29T14:41:59.533639+0000","last_fullsized":"2024-06-29T14:41:59.533639+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:51.884562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532709+0000","last_change":"2024-06-29T14:41:58.524113+0000","last_active":"2024-06-29T14:41:59.532709+0000","last_peered":"2024-06-29T14:41:59.532709+0000","last_clean":"2024-06-29T14:41:59.532709+0000","last_became_active":"2024-06-29T14:41:58.523762+0000","last_became_peered":"2024-06-29T14:41:58.523762+0000","last_unstale":"2024-06-29T14:41:59.532709+0000","last_undegraded":"2024-06-29T14:41:59.532709+0000","last_fullsized":"2024-06-29T14:41:59.532709+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:53:26.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533494+0000","last_change":"2024-06-29T14:41:58.524869+0000","last_active":"2024-06-29T14:41:59.533494+0000","last_peered":"2024-06-29T14:41:59.533494+0000","last_clean":"2024-06-29T14:41:59.533494+0000","last_became_active":"2024-06-29T14:41:58.524553+0000","last_became_peered":"2024-06-29T14:41:58.524553+0000","last_unstale":"2024-06-29T14:41:59.533494+0000","last_undegraded":"2024-06-29T14:41:59.533494+0000","last_fullsized":"2024-06-29T14:41:59.533494+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:55:34.695370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523970+0000","last_change":"2024-06-29T14:41:58.521070+0000","last_active":"2024-06-29T14:41:59.523970+0000","last_peered":"2024-06-29T14:41:59.523970+0000","last_clean":"2024-06-29T14:41:59.523970+0000","last_became_active":"2024-06-29T14:41:58.520583+0000","last_became_peered":"2024-06-29T14:41:58.520583+0000","last_unstale":"2024-06-29T14:41:59.523970+0000","last_undegraded":"2024-06-29T14:41:59.523970+0000","last_fullsized":"2024-06-29T14:41:59.523970+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:07:38.672786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526983+0000","last_change":"2024-06-29T14:41:58.539172+0000","last_active":"2024-06-29T14:41:59.526983+0000","last_peered":"2024-06-29T14:41:59.526983+0000","last_clean":"2024-06-29T14:41:59.526983+0000","last_became_active":"2024-06-29T14:41:58.538721+0000","last_became_peered":"2024-06-29T14:41:58.538721+0000","last_unstale":"2024-06-29T14:41:59.526983+0000","last_undegraded":"2024-06-29T14:41:59.526983+0000","last_fullsized":"2024-06-29T14:41:59.526983+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:12:48.524845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523783+0000","last_change":"2024-06-29T14:41:58.539117+0000","last_active":"2024-06-29T14:41:59.523783+0000","last_peered":"2024-06-29T14:41:59.523783+0000","last_clean":"2024-06-29T14:41:59.523783+0000","last_became_active":"2024-06-29T14:41:58.538793+0000","last_became_peered":"2024-06-29T14:41:58.538793+0000","last_unstale":"2024-06-29T14:41:59.523783+0000","last_undegraded":"2024-06-29T14:41:59.523783+0000","last_fullsized":"2024-06-29T14:41:59.523783+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:10.667868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532720+0000","last_change":"2024-06-29T14:41:58.520306+0000","last_active":"2024-06-29T14:41:59.532720+0000","last_peered":"2024-06-29T14:41:59.532720+0000","last_clean":"2024-06-29T14:41:59.532720+0000","last_became_active":"2024-06-29T14:41:58.520005+0000","last_became_peered":"2024-06-29T14:41:58.520005+0000","last_unstale":"2024-06-29T14:41:59.532720+0000","last_undegraded":"2024-06-29T14:41:59.532720+0000","last_fullsized":"2024-06-29T14:41:59.532720+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:05.489135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533437+0000","last_change":"2024-06-29T14:41:58.532514+0000","last_active":"2024-06-29T14:41:59.533437+0000","last_peered":"2024-06-29T14:41:59.533437+0000","last_clean":"2024-06-29T14:41:59.533437+0000","last_became_active":"2024-06-29T14:41:58.530528+0000","last_became_peered":"2024-06-29T14:41:58.530528+0000","last_unstale":"2024-06-29T14:41:59.533437+0000","last_undegraded":"2024-06-29T14:41:59.533437+0000","last_fullsized":"2024-06-29T14:41:59.533437+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:41.642248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533230+0000","last_change":"2024-06-29T14:41:58.527374+0000","last_active":"2024-06-29T14:41:59.533230+0000","last_peered":"2024-06-29T14:41:59.533230+0000","last_clean":"2024-06-29T14:41:59.533230+0000","last_became_active":"2024-06-29T14:41:58.526244+0000","last_became_peered":"2024-06-29T14:41:58.526244+0000","last_unstale":"2024-06-29T14:41:59.533230+0000","last_undegraded":"2024-06-29T14:41:59.533230+0000","last_fullsized":"2024-06-29T14:41:59.533230+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:44.948443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533161+0000","last_change":"2024-06-29T14:41:58.517873+0000","last_active":"2024-06-29T14:41:59.533161+0000","last_peered":"2024-06-29T14:41:59.533161+0000","last_clean":"2024-06-29T14:41:59.533161+0000","last_became_active":"2024-06-29T14:41:58.517300+0000","last_became_peered":"2024-06-29T14:41:58.517300+0000","last_unstale":"2024-06-29T14:41:59.533161+0000","last_undegraded":"2024-06-29T14:41:59.533161+0000","last_fullsized":"2024-06-29T14:41:59.533161+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:21:08.860536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526221+0000","last_change":"2024-06-29T14:41:58.519361+0000","last_active":"2024-06-29T14:41:59.526221+0000","last_peered":"2024-06-29T14:41:59.526221+0000","last_clean":"2024-06-29T14:41:59.526221+0000","last_became_active":"2024-06-29T14:41:58.519042+0000","last_became_peered":"2024-06-29T14:41:58.519042+0000","last_unstale":"2024-06-29T14:41:59.526221+0000","last_undegraded":"2024-06-29T14:41:59.526221+0000","last_fullsized":"2024-06-29T14:41:59.526221+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:43.162274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525654+0000","last_change":"2024-06-29T14:41:58.535961+0000","last_active":"2024-06-29T14:41:59.525654+0000","last_peered":"2024-06-29T14:41:59.525654+0000","last_clean":"2024-06-29T14:41:59.525654+0000","last_became_active":"2024-06-29T14:41:58.535649+0000","last_became_peered":"2024-06-29T14:41:58.535649+0000","last_unstale":"2024-06-29T14:41:59.525654+0000","last_undegraded":"2024-06-29T14:41:59.525654+0000","last_fullsized":"2024-06-29T14:41:59.525654+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:23.362810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527197+0000","last_change":"2024-06-29T14:41:58.539230+0000","last_active":"2024-06-29T14:41:59.527197+0000","last_peered":"2024-06-29T14:41:59.527197+0000","last_clean":"2024-06-29T14:41:59.527197+0000","last_became_active":"2024-06-29T14:41:58.538844+0000","last_became_peered":"2024-06-29T14:41:58.538844+0000","last_unstale":"2024-06-29T14:41:59.527197+0000","last_undegraded":"2024-06-29T14:41:59.527197+0000","last_fullsized":"2024-06-29T14:41:59.527197+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:18:01.237841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532237+0000","last_change":"2024-06-29T14:41:58.529623+0000","last_active":"2024-06-29T14:41:59.532237+0000","last_peered":"2024-06-29T14:41:59.532237+0000","last_clean":"2024-06-29T14:41:59.532237+0000","last_became_active":"2024-06-29T14:41:58.528984+0000","last_became_peered":"2024-06-29T14:41:58.528984+0000","last_unstale":"2024-06-29T14:41:59.532237+0000","last_undegraded":"2024-06-29T14:41:59.532237+0000","last_fullsized":"2024-06-29T14:41:59.532237+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:44.948443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525645+0000","last_change":"2024-06-29T14:41:58.522389+0000","last_active":"2024-06-29T14:41:59.525645+0000","last_peered":"2024-06-29T14:41:59.525645+0000","last_clean":"2024-06-29T14:41:59.525645+0000","last_became_active":"2024-06-29T14:41:58.522074+0000","last_became_peered":"2024-06-29T14:41:58.522074+0000","last_unstale":"2024-06-29T14:41:59.525645+0000","last_undegraded":"2024-06-29T14:41:59.525645+0000","last_fullsized":"2024-06-29T14:41:59.525645+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:02:26.307183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525411+0000","last_change":"2024-06-29T14:41:58.518761+0000","last_active":"2024-06-29T14:41:59.525411+0000","last_peered":"2024-06-29T14:41:59.525411+0000","last_clean":"2024-06-29T14:41:59.525411+0000","last_became_active":"2024-06-29T14:41:58.518383+0000","last_became_peered":"2024-06-29T14:41:58.518383+0000","last_unstale":"2024-06-29T14:41:59.525411+0000","last_undegraded":"2024-06-29T14:41:59.525411+0000","last_fullsized":"2024-06-29T14:41:59.525411+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.936045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532519+0000","last_change":"2024-06-29T14:41:58.534685+0000","last_active":"2024-06-29T14:41:59.532519+0000","last_peered":"2024-06-29T14:41:59.532519+0000","last_clean":"2024-06-29T14:41:59.532519+0000","last_became_active":"2024-06-29T14:41:58.534358+0000","last_became_peered":"2024-06-29T14:41:58.534358+0000","last_unstale":"2024-06-29T14:41:59.532519+0000","last_undegraded":"2024-06-29T14:41:59.532519+0000","last_fullsized":"2024-06-29T14:41:59.532519+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:40.532010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532070+0000","last_change":"2024-06-29T14:41:58.541428+0000","last_active":"2024-06-29T14:41:59.532070+0000","last_peered":"2024-06-29T14:41:59.532070+0000","last_clean":"2024-06-29T14:41:59.532070+0000","last_became_active":"2024-06-29T14:41:58.540979+0000","last_became_peered":"2024-06-29T14:41:58.540979+0000","last_unstale":"2024-06-29T14:41:59.532070+0000","last_undegraded":"2024-06-29T14:41:59.532070+0000","last_fullsized":"2024-06-29T14:41:59.532070+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:37.536052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532733+0000","last_change":"2024-06-29T14:41:58.518095+0000","last_active":"2024-06-29T14:41:59.532733+0000","last_peered":"2024-06-29T14:41:59.532733+0000","last_clean":"2024-06-29T14:41:59.532733+0000","last_became_active":"2024-06-29T14:41:58.517776+0000","last_became_peered":"2024-06-29T14:41:58.517776+0000","last_unstale":"2024-06-29T14:41:59.532733+0000","last_undegraded":"2024-06-29T14:41:59.532733+0000","last_fullsized":"2024-06-29T14:41:59.532733+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:46.537755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526349+0000","last_change":"2024-06-29T14:41:57.509834+0000","last_active":"2024-06-29T14:41:59.526349+0000","last_peered":"2024-06-29T14:41:59.526349+0000","last_clean":"2024-06-29T14:41:59.526349+0000","last_became_active":"2024-06-29T14:41:57.509525+0000","last_became_peered":"2024-06-29T14:41:57.509525+0000","last_unstale":"2024-06-29T14:41:59.526349+0000","last_undegraded":"2024-06-29T14:41:59.526349+0000","last_fullsized":"2024-06-29T14:41:59.526349+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:36.662713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533824+0000","last_change":"2024-06-29T14:41:58.533253+0000","last_active":"2024-06-29T14:41:59.533824+0000","last_peered":"2024-06-29T14:41:59.533824+0000","last_clean":"2024-06-29T14:41:59.533824+0000","last_became_active":"2024-06-29T14:41:58.532927+0000","last_became_peered":"2024-06-29T14:41:58.532927+0000","last_unstale":"2024-06-29T14:41:59.533824+0000","last_undegraded":"2024-06-29T14:41:59.533824+0000","last_fullsized":"2024-06-29T14:41:59.533824+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:41.902208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.524154+0000","last_change":"2024-06-29T14:41:57.511427+0000","last_active":"2024-06-29T14:41:59.524154+0000","last_peered":"2024-06-29T14:41:59.524154+0000","last_clean":"2024-06-29T14:41:59.524154+0000","last_became_active":"2024-06-29T14:41:57.511108+0000","last_became_peered":"2024-06-29T14:41:57.511108+0000","last_unstale":"2024-06-29T14:41:59.524154+0000","last_undegraded":"2024-06-29T14:41:59.524154+0000","last_fullsized":"2024-06-29T14:41:59.524154+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:59.499233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532665+0000","last_change":"2024-06-29T14:41:58.525359+0000","last_active":"2024-06-29T14:41:59.532665+0000","last_peered":"2024-06-29T14:41:59.532665+0000","last_clean":"2024-06-29T14:41:59.532665+0000","last_became_active":"2024-06-29T14:41:58.524403+0000","last_became_peered":"2024-06-29T14:41:58.524403+0000","last_unstale":"2024-06-29T14:41:59.532665+0000","last_undegraded":"2024-06-29T14:41:59.532665+0000","last_fullsized":"2024-06-29T14:41:59.532665+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:22.027012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527197+0000","last_change":"2024-06-29T14:41:57.512273+0000","last_active":"2024-06-29T14:41:59.527197+0000","last_peered":"2024-06-29T14:41:59.527197+0000","last_clean":"2024-06-29T14:41:59.527197+0000","last_became_active":"2024-06-29T14:41:57.511958+0000","last_became_peered":"2024-06-29T14:41:57.511958+0000","last_unstale":"2024-06-29T14:41:59.527197+0000","last_undegraded":"2024-06-29T14:41:59.527197+0000","last_fullsized":"2024-06-29T14:41:59.527197+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:26.744219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533061+0000","last_change":"2024-06-29T14:41:58.538082+0000","last_active":"2024-06-29T14:41:59.533061+0000","last_peered":"2024-06-29T14:41:59.533061+0000","last_clean":"2024-06-29T14:41:59.533061+0000","last_became_active":"2024-06-29T14:41:58.537792+0000","last_became_peered":"2024-06-29T14:41:58.537792+0000","last_unstale":"2024-06-29T14:41:59.533061+0000","last_undegraded":"2024-06-29T14:41:59.533061+0000","last_fullsized":"2024-06-29T14:41:59.533061+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:52.542145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526890+0000","last_change":"2024-06-29T14:41:57.506735+0000","last_active":"2024-06-29T14:41:59.526890+0000","last_peered":"2024-06-29T14:41:59.526890+0000","last_clean":"2024-06-29T14:41:59.526890+0000","last_became_active":"2024-06-29T14:41:57.506402+0000","last_became_peered":"2024-06-29T14:41:57.506402+0000","last_unstale":"2024-06-29T14:41:59.526890+0000","last_undegraded":"2024-06-29T14:41:59.526890+0000","last_fullsized":"2024-06-29T14:41:59.526890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:30:42.868318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533710+0000","last_change":"2024-06-29T14:41:58.534462+0000","last_active":"2024-06-29T14:41:59.533710+0000","last_peered":"2024-06-29T14:41:59.533710+0000","last_clean":"2024-06-29T14:41:59.533710+0000","last_became_active":"2024-06-29T14:41:58.534164+0000","last_became_peered":"2024-06-29T14:41:58.534164+0000","last_unstale":"2024-06-29T14:41:59.533710+0000","last_undegraded":"2024-06-29T14:41:59.533710+0000","last_fullsized":"2024-06-29T14:41:59.533710+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:26:57.805185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532809+0000","last_change":"2024-06-29T14:41:57.509711+0000","last_active":"2024-06-29T14:41:59.532809+0000","last_peered":"2024-06-29T14:41:59.532809+0000","last_clean":"2024-06-29T14:41:59.532809+0000","last_became_active":"2024-06-29T14:41:57.509425+0000","last_became_peered":"2024-06-29T14:41:57.509425+0000","last_unstale":"2024-06-29T14:41:59.532809+0000","last_undegraded":"2024-06-29T14:41:59.532809+0000","last_fullsized":"2024-06-29T14:41:59.532809+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:49.202916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532595+0000","last_change":"2024-06-29T14:41:58.525958+0000","last_active":"2024-06-29T14:41:59.532595+0000","last_peered":"2024-06-29T14:41:59.532595+0000","last_clean":"2024-06-29T14:41:59.532595+0000","last_became_active":"2024-06-29T14:41:58.525490+0000","last_became_peered":"2024-06-29T14:41:58.525490+0000","last_unstale":"2024-06-29T14:41:59.532595+0000","last_undegraded":"2024-06-29T14:41:59.532595+0000","last_fullsized":"2024-06-29T14:41:59.532595+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.936045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523507+0000","last_change":"2024-06-29T14:41:57.513810+0000","last_active":"2024-06-29T14:41:59.523507+0000","last_peered":"2024-06-29T14:41:59.523507+0000","last_clean":"2024-06-29T14:41:59.523507+0000","last_became_active":"2024-06-29T14:41:57.513515+0000","last_became_peered":"2024-06-29T14:41:57.513515+0000","last_unstale":"2024-06-29T14:41:59.523507+0000","last_undegraded":"2024-06-29T14:41:59.523507+0000","last_fullsized":"2024-06-29T14:41:59.523507+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:32.960566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.524032+0000","last_change":"2024-06-29T14:41:58.536776+0000","last_active":"2024-06-29T14:41:59.524032+0000","last_peered":"2024-06-29T14:41:59.524032+0000","last_clean":"2024-06-29T14:41:59.524032+0000","last_became_active":"2024-06-29T14:41:58.536415+0000","last_became_peered":"2024-06-29T14:41:58.536415+0000","last_unstale":"2024-06-29T14:41:59.524032+0000","last_undegraded":"2024-06-29T14:41:59.524032+0000","last_fullsized":"2024-06-29T14:41:59.524032+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:55:44.478520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532194+0000","last_change":"2024-06-29T14:41:57.517631+0000","last_active":"2024-06-29T14:41:59.532194+0000","last_peered":"2024-06-29T14:41:59.532194+0000","last_clean":"2024-06-29T14:41:59.532194+0000","last_became_active":"2024-06-29T14:41:57.517287+0000","last_became_peered":"2024-06-29T14:41:57.517287+0000","last_unstale":"2024-06-29T14:41:59.532194+0000","last_undegraded":"2024-06-29T14:41:59.532194+0000","last_fullsized":"2024-06-29T14:41:59.532194+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:56.507215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526148+0000","last_change":"2024-06-29T14:41:58.539564+0000","last_active":"2024-06-29T14:41:59.526148+0000","last_peered":"2024-06-29T14:41:59.526148+0000","last_clean":"2024-06-29T14:41:59.526148+0000","last_became_active":"2024-06-29T14:41:58.539031+0000","last_became_peered":"2024-06-29T14:41:58.539031+0000","last_unstale":"2024-06-29T14:41:59.526148+0000","last_undegraded":"2024-06-29T14:41:59.526148+0000","last_fullsized":"2024-06-29T14:41:59.526148+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:08.305916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525142+0000","last_change":"2024-06-29T14:41:57.513604+0000","last_active":"2024-06-29T14:41:59.525142+0000","last_peered":"2024-06-29T14:41:59.525142+0000","last_clean":"2024-06-29T14:41:59.525142+0000","last_became_active":"2024-06-29T14:41:57.513290+0000","last_became_peered":"2024-06-29T14:41:57.513290+0000","last_unstale":"2024-06-29T14:41:59.525142+0000","last_undegraded":"2024-06-29T14:41:59.525142+0000","last_fullsized":"2024-06-29T14:41:59.525142+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:36.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532920+0000","last_change":"2024-06-29T14:41:58.530597+0000","last_active":"2024-06-29T14:41:59.532920+0000","last_peered":"2024-06-29T14:41:59.532920+0000","last_clean":"2024-06-29T14:41:59.532920+0000","last_became_active":"2024-06-29T14:41:58.530246+0000","last_became_peered":"2024-06-29T14:41:58.530246+0000","last_unstale":"2024-06-29T14:41:59.532920+0000","last_undegraded":"2024-06-29T14:41:59.532920+0000","last_fullsized":"2024-06-29T14:41:59.532920+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:15:41.729768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533512+0000","last_change":"2024-06-29T14:41:57.515396+0000","last_active":"2024-06-29T14:41:59.533512+0000","last_peered":"2024-06-29T14:41:59.533512+0000","last_clean":"2024-06-29T14:41:59.533512+0000","last_became_active":"2024-06-29T14:41:57.515164+0000","last_became_peered":"2024-06-29T14:41:57.515164+0000","last_unstale":"2024-06-29T14:41:59.533512+0000","last_undegraded":"2024-06-29T14:41:59.533512+0000","last_fullsized":"2024-06-29T14:41:59.533512+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:33.025091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532368+0000","last_change":"2024-06-29T14:41:58.525131+0000","last_active":"2024-06-29T14:41:59.532368+0000","last_peered":"2024-06-29T14:41:59.532368+0000","last_clean":"2024-06-29T14:41:59.532368+0000","last_became_active":"2024-06-29T14:41:58.524299+0000","last_became_peered":"2024-06-29T14:41:58.524299+0000","last_unstale":"2024-06-29T14:41:59.532368+0000","last_undegraded":"2024-06-29T14:41:59.532368+0000","last_fullsized":"2024-06-29T14:41:59.532368+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:24:38.730535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533291+0000","last_change":"2024-06-29T14:41:57.510159+0000","last_active":"2024-06-29T14:41:59.533291+0000","last_peered":"2024-06-29T14:41:59.533291+0000","last_clean":"2024-06-29T14:41:59.533291+0000","last_became_active":"2024-06-29T14:41:57.509825+0000","last_became_peered":"2024-06-29T14:41:57.509825+0000","last_unstale":"2024-06-29T14:41:59.533291+0000","last_undegraded":"2024-06-29T14:41:59.533291+0000","last_fullsized":"2024-06-29T14:41:59.533291+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:53.480344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525488+0000","last_change":"2024-06-29T14:41:58.515800+0000","last_active":"2024-06-29T14:41:59.525488+0000","last_peered":"2024-06-29T14:41:59.525488+0000","last_clean":"2024-06-29T14:41:59.525488+0000","last_became_active":"2024-06-29T14:41:58.514758+0000","last_became_peered":"2024-06-29T14:41:58.514758+0000","last_unstale":"2024-06-29T14:41:59.525488+0000","last_undegraded":"2024-06-29T14:41:59.525488+0000","last_fullsized":"2024-06-29T14:41:59.525488+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:46.537755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527321+0000","last_change":"2024-06-29T14:41:57.511545+0000","last_active":"2024-06-29T14:41:59.527321+0000","last_peered":"2024-06-29T14:41:59.527321+0000","last_clean":"2024-06-29T14:41:59.527321+0000","last_became_active":"2024-06-29T14:41:57.510065+0000","last_became_peered":"2024-06-29T14:41:57.510065+0000","last_unstale":"2024-06-29T14:41:59.527321+0000","last_undegraded":"2024-06-29T14:41:59.527321+0000","last_fullsized":"2024-06-29T14:41:59.527321+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:56.648158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525707+0000","last_change":"2024-06-29T14:41:58.529387+0000","last_active":"2024-06-29T14:41:59.525707+0000","last_peered":"2024-06-29T14:41:59.525707+0000","last_clean":"2024-06-29T14:41:59.525707+0000","last_became_active":"2024-06-29T14:41:58.528942+0000","last_became_peered":"2024-06-29T14:41:58.528942+0000","last_unstale":"2024-06-29T14:41:59.525707+0000","last_undegraded":"2024-06-29T14:41:59.525707+0000","last_fullsized":"2024-06-29T14:41:59.525707+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:57.245802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527029+0000","last_change":"2024-06-29T14:41:57.516426+0000","last_active":"2024-06-29T14:41:59.527029+0000","last_peered":"2024-06-29T14:41:59.527029+0000","last_clean":"2024-06-29T14:41:59.527029+0000","last_became_active":"2024-06-29T14:41:57.516048+0000","last_became_peered":"2024-06-29T14:41:57.516048+0000","last_unstale":"2024-06-29T14:41:59.527029+0000","last_undegraded":"2024-06-29T14:41:59.527029+0000","last_fullsized":"2024-06-29T14:41:59.527029+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:38.386996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532407+0000","last_change":"2024-06-29T14:41:58.533628+0000","last_active":"2024-06-29T14:41:59.532407+0000","last_peered":"2024-06-29T14:41:59.532407+0000","last_clean":"2024-06-29T14:41:59.532407+0000","last_became_active":"2024-06-29T14:41:58.533280+0000","last_became_peered":"2024-06-29T14:41:58.533280+0000","last_unstale":"2024-06-29T14:41:59.532407+0000","last_undegraded":"2024-06-29T14:41:59.532407+0000","last_fullsized":"2024-06-29T14:41:59.532407+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:42.807014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533198+0000","last_change":"2024-06-29T14:41:57.512737+0000","last_active":"2024-06-29T14:41:59.533198+0000","last_peered":"2024-06-29T14:41:59.533198+0000","last_clean":"2024-06-29T14:41:59.533198+0000","last_became_active":"2024-06-29T14:41:57.512003+0000","last_became_peered":"2024-06-29T14:41:57.512003+0000","last_unstale":"2024-06-29T14:41:59.533198+0000","last_undegraded":"2024-06-29T14:41:59.533198+0000","last_fullsized":"2024-06-29T14:41:59.533198+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:49.503288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526256+0000","last_change":"2024-06-29T14:41:58.535064+0000","last_active":"2024-06-29T14:41:59.526256+0000","last_peered":"2024-06-29T14:41:59.526256+0000","last_clean":"2024-06-29T14:41:59.526256+0000","last_became_active":"2024-06-29T14:41:58.534654+0000","last_became_peered":"2024-06-29T14:41:58.534654+0000","last_unstale":"2024-06-29T14:41:59.526256+0000","last_undegraded":"2024-06-29T14:41:59.526256+0000","last_fullsized":"2024-06-29T14:41:59.526256+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:45.765189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525010+0000","last_change":"2024-06-29T14:41:57.511650+0000","last_active":"2024-06-29T14:41:59.525010+0000","last_peered":"2024-06-29T14:41:59.525010+0000","last_clean":"2024-06-29T14:41:59.525010+0000","last_became_active":"2024-06-29T14:41:57.511224+0000","last_became_peered":"2024-06-29T14:41:57.511224+0000","last_unstale":"2024-06-29T14:41:59.525010+0000","last_undegraded":"2024-06-29T14:41:59.525010+0000","last_fullsized":"2024-06-29T14:41:59.525010+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:53:28.408782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.524133+0000","last_change":"2024-06-29T14:41:58.531677+0000","last_active":"2024-06-29T14:41:59.524133+0000","last_peered":"2024-06-29T14:41:59.524133+0000","last_clean":"2024-06-29T14:41:59.524133+0000","last_became_active":"2024-06-29T14:41:58.531328+0000","last_became_peered":"2024-06-29T14:41:58.531328+0000","last_unstale":"2024-06-29T14:41:59.524133+0000","last_undegraded":"2024-06-29T14:41:59.524133+0000","last_fullsized":"2024-06-29T14:41:59.524133+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:05.489135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533289+0000","last_change":"2024-06-29T14:41:57.506405+0000","last_active":"2024-06-29T14:41:59.533289+0000","last_peered":"2024-06-29T14:41:59.533289+0000","last_clean":"2024-06-29T14:41:59.533289+0000","last_became_active":"2024-06-29T14:41:57.506117+0000","last_became_peered":"2024-06-29T14:41:57.506117+0000","last_unstale":"2024-06-29T14:41:59.533289+0000","last_undegraded":"2024-06-29T14:41:59.533289+0000","last_fullsized":"2024-06-29T14:41:59.533289+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:17.449570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525262+0000","last_change":"2024-06-29T14:41:58.523820+0000","last_active":"2024-06-29T14:41:59.525262+0000","last_peered":"2024-06-29T14:41:59.525262+0000","last_clean":"2024-06-29T14:41:59.525262+0000","last_became_active":"2024-06-29T14:41:58.523477+0000","last_became_peered":"2024-06-29T14:41:58.523477+0000","last_unstale":"2024-06-29T14:41:59.525262+0000","last_undegraded":"2024-06-29T14:41:59.525262+0000","last_fullsized":"2024-06-29T14:41:59.525262+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:40.532010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533635+0000","last_change":"2024-06-29T14:41:57.512385+0000","last_active":"2024-06-29T14:41:59.533635+0000","last_peered":"2024-06-29T14:41:59.533635+0000","last_clean":"2024-06-29T14:41:59.533635+0000","last_became_active":"2024-06-29T14:41:57.512034+0000","last_became_peered":"2024-06-29T14:41:57.512034+0000","last_unstale":"2024-06-29T14:41:59.533635+0000","last_undegraded":"2024-06-29T14:41:59.533635+0000","last_fullsized":"2024-06-29T14:41:59.533635+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:29.246165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526797+0000","last_change":"2024-06-29T14:41:58.537987+0000","last_active":"2024-06-29T14:41:59.526797+0000","last_peered":"2024-06-29T14:41:59.526797+0000","last_clean":"2024-06-29T14:41:59.526797+0000","last_became_active":"2024-06-29T14:41:58.537327+0000","last_became_peered":"2024-06-29T14:41:58.537327+0000","last_unstale":"2024-06-29T14:41:59.526797+0000","last_undegraded":"2024-06-29T14:41:59.526797+0000","last_fullsized":"2024-06-29T14:41:59.526797+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:30.664437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525925+0000","last_change":"2024-06-29T14:41:57.508397+0000","last_active":"2024-06-29T14:41:59.525925+0000","last_peered":"2024-06-29T14:41:59.525925+0000","last_clean":"2024-06-29T14:41:59.525925+0000","last_became_active":"2024-06-29T14:41:57.508099+0000","last_became_peered":"2024-06-29T14:41:57.508099+0000","last_unstale":"2024-06-29T14:41:59.525925+0000","last_undegraded":"2024-06-29T14:41:59.525925+0000","last_fullsized":"2024-06-29T14:41:59.525925+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:33.619707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527291+0000","last_change":"2024-06-29T14:41:58.539325+0000","last_active":"2024-06-29T14:41:59.527291+0000","last_peered":"2024-06-29T14:41:59.527291+0000","last_clean":"2024-06-29T14:41:59.527291+0000","last_became_active":"2024-06-29T14:41:58.538983+0000","last_became_peered":"2024-06-29T14:41:58.538983+0000","last_unstale":"2024-06-29T14:41:59.527291+0000","last_undegraded":"2024-06-29T14:41:59.527291+0000","last_fullsized":"2024-06-29T14:41:59.527291+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:51.468237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533027+0000","last_change":"2024-06-29T14:41:57.516894+0000","last_active":"2024-06-29T14:41:59.533027+0000","last_peered":"2024-06-29T14:41:59.533027+0000","last_clean":"2024-06-29T14:41:59.533027+0000","last_became_active":"2024-06-29T14:41:57.516556+0000","last_became_peered":"2024-06-29T14:41:57.516556+0000","last_unstale":"2024-06-29T14:41:59.533027+0000","last_undegraded":"2024-06-29T14:41:59.533027+0000","last_fullsized":"2024-06-29T14:41:59.533027+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:10.507355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532698+0000","last_change":"2024-06-29T14:41:58.530241+0000","last_active":"2024-06-29T14:41:59.532698+0000","last_peered":"2024-06-29T14:41:59.532698+0000","last_clean":"2024-06-29T14:41:59.532698+0000","last_became_active":"2024-06-29T14:41:58.529847+0000","last_became_peered":"2024-06-29T14:41:58.529847+0000","last_unstale":"2024-06-29T14:41:59.532698+0000","last_undegraded":"2024-06-29T14:41:59.532698+0000","last_fullsized":"2024-06-29T14:41:59.532698+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:15.787250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523891+0000","last_change":"2024-06-29T14:41:57.510198+0000","last_active":"2024-06-29T14:41:59.523891+0000","last_peered":"2024-06-29T14:41:59.523891+0000","last_clean":"2024-06-29T14:41:59.523891+0000","last_became_active":"2024-06-29T14:41:57.509901+0000","last_became_peered":"2024-06-29T14:41:57.509901+0000","last_unstale":"2024-06-29T14:41:59.523891+0000","last_undegraded":"2024-06-29T14:41:59.523891+0000","last_fullsized":"2024-06-29T14:41:59.523891+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:59:04.655016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523852+0000","last_change":"2024-06-29T14:41:58.527478+0000","last_active":"2024-06-29T14:41:59.523852+0000","last_peered":"2024-06-29T14:41:59.523852+0000","last_clean":"2024-06-29T14:41:59.523852+0000","last_became_active":"2024-06-29T14:41:58.527151+0000","last_became_peered":"2024-06-29T14:41:58.527151+0000","last_unstale":"2024-06-29T14:41:59.523852+0000","last_undegraded":"2024-06-29T14:41:59.523852+0000","last_fullsized":"2024-06-29T14:41:59.523852+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:08.124503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533062+0000","last_change":"2024-06-29T14:41:57.507138+0000","last_active":"2024-06-29T14:41:59.533062+0000","last_peered":"2024-06-29T14:41:59.533062+0000","last_clean":"2024-06-29T14:41:59.533062+0000","last_became_active":"2024-06-29T14:41:57.506796+0000","last_became_peered":"2024-06-29T14:41:57.506796+0000","last_unstale":"2024-06-29T14:41:59.533062+0000","last_undegraded":"2024-06-29T14:41:59.533062+0000","last_fullsized":"2024-06-29T14:41:59.533062+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:39:55.074608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533363+0000","last_change":"2024-06-29T14:41:58.541451+0000","last_active":"2024-06-29T14:41:59.533363+0000","last_peered":"2024-06-29T14:41:59.533363+0000","last_clean":"2024-06-29T14:41:59.533363+0000","last_became_active":"2024-06-29T14:41:58.541145+0000","last_became_peered":"2024-06-29T14:41:58.541145+0000","last_unstale":"2024-06-29T14:41:59.533363+0000","last_undegraded":"2024-06-29T14:41:59.533363+0000","last_fullsized":"2024-06-29T14:41:59.533363+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:56.963338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533538+0000","last_change":"2024-06-29T14:41:57.517317+0000","last_active":"2024-06-29T14:41:59.533538+0000","last_peered":"2024-06-29T14:41:59.533538+0000","last_clean":"2024-06-29T14:41:59.533538+0000","last_became_active":"2024-06-29T14:41:57.516971+0000","last_became_peered":"2024-06-29T14:41:57.516971+0000","last_unstale":"2024-06-29T14:41:59.533538+0000","last_undegraded":"2024-06-29T14:41:59.533538+0000","last_fullsized":"2024-06-29T14:41:59.533538+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:55.425911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.524097+0000","last_change":"2024-06-29T14:41:58.526944+0000","last_active":"2024-06-29T14:41:59.524097+0000","last_peered":"2024-06-29T14:41:59.524097+0000","last_clean":"2024-06-29T14:41:59.524097+0000","last_became_active":"2024-06-29T14:41:58.526617+0000","last_became_peered":"2024-06-29T14:41:58.526617+0000","last_unstale":"2024-06-29T14:41:59.524097+0000","last_undegraded":"2024-06-29T14:41:59.524097+0000","last_fullsized":"2024-06-29T14:41:59.524097+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:22.027012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526898+0000","last_change":"2024-06-29T14:41:57.510993+0000","last_active":"2024-06-29T14:41:59.526898+0000","last_peered":"2024-06-29T14:41:59.526898+0000","last_clean":"2024-06-29T14:41:59.526898+0000","last_became_active":"2024-06-29T14:41:57.510650+0000","last_became_peered":"2024-06-29T14:41:57.510650+0000","last_unstale":"2024-06-29T14:41:59.526898+0000","last_undegraded":"2024-06-29T14:41:59.526898+0000","last_fullsized":"2024-06-29T14:41:59.526898+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:06.750598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533376+0000","last_change":"2024-06-29T14:41:58.517623+0000","last_active":"2024-06-29T14:41:59.533376+0000","last_peered":"2024-06-29T14:41:59.533376+0000","last_clean":"2024-06-29T14:41:59.533376+0000","last_became_active":"2024-06-29T14:41:58.517323+0000","last_became_peered":"2024-06-29T14:41:58.517323+0000","last_unstale":"2024-06-29T14:41:59.533376+0000","last_undegraded":"2024-06-29T14:41:59.533376+0000","last_fullsized":"2024-06-29T14:41:59.533376+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:42.807014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533409+0000","last_change":"2024-06-29T14:41:57.510104+0000","last_active":"2024-06-29T14:41:59.533409+0000","last_peered":"2024-06-29T14:41:59.533409+0000","last_clean":"2024-06-29T14:41:59.533409+0000","last_became_active":"2024-06-29T14:41:57.509745+0000","last_became_peered":"2024-06-29T14:41:57.509745+0000","last_unstale":"2024-06-29T14:41:59.533409+0000","last_undegraded":"2024-06-29T14:41:59.533409+0000","last_fullsized":"2024-06-29T14:41:59.533409+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:49.707683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523706+0000","last_change":"2024-06-29T14:41:58.526235+0000","last_active":"2024-06-29T14:41:59.523706+0000","last_peered":"2024-06-29T14:41:59.523706+0000","last_clean":"2024-06-29T14:41:59.523706+0000","last_became_active":"2024-06-29T14:41:58.525871+0000","last_became_peered":"2024-06-29T14:41:58.525871+0000","last_unstale":"2024-06-29T14:41:59.523706+0000","last_undegraded":"2024-06-29T14:41:59.523706+0000","last_fullsized":"2024-06-29T14:41:59.523706+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:49.063709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533972+0000","last_change":"2024-06-29T14:41:57.515930+0000","last_active":"2024-06-29T14:41:59.533972+0000","last_peered":"2024-06-29T14:41:59.533972+0000","last_clean":"2024-06-29T14:41:59.533972+0000","last_became_active":"2024-06-29T14:41:57.515603+0000","last_became_peered":"2024-06-29T14:41:57.515603+0000","last_unstale":"2024-06-29T14:41:59.533972+0000","last_undegraded":"2024-06-29T14:41:59.533972+0000","last_fullsized":"2024-06-29T14:41:59.533972+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:19.701123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525136+0000","last_change":"2024-06-29T14:41:58.529192+0000","last_active":"2024-06-29T14:41:59.525136+0000","last_peered":"2024-06-29T14:41:59.525136+0000","last_clean":"2024-06-29T14:41:59.525136+0000","last_became_active":"2024-06-29T14:41:58.528804+0000","last_became_peered":"2024-06-29T14:41:58.528804+0000","last_unstale":"2024-06-29T14:41:59.525136+0000","last_undegraded":"2024-06-29T14:41:59.525136+0000","last_fullsized":"2024-06-29T14:41:59.525136+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:15:32.748190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533247+0000","last_change":"2024-06-29T14:41:57.512008+0000","last_active":"2024-06-29T14:41:59.533247+0000","last_peered":"2024-06-29T14:41:59.533247+0000","last_clean":"2024-06-29T14:41:59.533247+0000","last_became_active":"2024-06-29T14:41:57.511681+0000","last_became_peered":"2024-06-29T14:41:57.511681+0000","last_unstale":"2024-06-29T14:41:59.533247+0000","last_undegraded":"2024-06-29T14:41:59.533247+0000","last_fullsized":"2024-06-29T14:41:59.533247+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:34.612389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527017+0000","last_change":"2024-06-29T14:41:58.534327+0000","last_active":"2024-06-29T14:41:59.527017+0000","last_peered":"2024-06-29T14:41:59.527017+0000","last_clean":"2024-06-29T14:41:59.527017+0000","last_became_active":"2024-06-29T14:41:58.533374+0000","last_became_peered":"2024-06-29T14:41:58.533374+0000","last_unstale":"2024-06-29T14:41:59.527017+0000","last_undegraded":"2024-06-29T14:41:59.527017+0000","last_fullsized":"2024-06-29T14:41:59.527017+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:35.653870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532890+0000","last_change":"2024-06-29T14:41:57.509359+0000","last_active":"2024-06-29T14:41:59.532890+0000","last_peered":"2024-06-29T14:41:59.532890+0000","last_clean":"2024-06-29T14:41:59.532890+0000","last_became_active":"2024-06-29T14:41:57.509067+0000","last_became_peered":"2024-06-29T14:41:57.509067+0000","last_unstale":"2024-06-29T14:41:59.532890+0000","last_undegraded":"2024-06-29T14:41:59.532890+0000","last_fullsized":"2024-06-29T14:41:59.532890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:52:22.009454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532863+0000","last_change":"2024-06-29T14:41:58.529484+0000","last_active":"2024-06-29T14:41:59.532863+0000","last_peered":"2024-06-29T14:41:59.532863+0000","last_clean":"2024-06-29T14:41:59.532863+0000","last_became_active":"2024-06-29T14:41:58.529160+0000","last_became_peered":"2024-06-29T14:41:58.529160+0000","last_unstale":"2024-06-29T14:41:59.532863+0000","last_undegraded":"2024-06-29T14:41:59.532863+0000","last_fullsized":"2024-06-29T14:41:59.532863+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:54.490869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532833+0000","last_change":"2024-06-29T14:41:57.504027+0000","last_active":"2024-06-29T14:41:59.532833+0000","last_peered":"2024-06-29T14:41:59.532833+0000","last_clean":"2024-06-29T14:41:59.532833+0000","last_became_active":"2024-06-29T14:41:57.503586+0000","last_became_peered":"2024-06-29T14:41:57.503586+0000","last_unstale":"2024-06-29T14:41:59.532833+0000","last_undegraded":"2024-06-29T14:41:59.532833+0000","last_fullsized":"2024-06-29T14:41:59.532833+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:15.788541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532731+0000","last_change":"2024-06-29T14:41:58.533438+0000","last_active":"2024-06-29T14:41:59.532731+0000","last_peered":"2024-06-29T14:41:59.532731+0000","last_clean":"2024-06-29T14:41:59.532731+0000","last_became_active":"2024-06-29T14:41:58.533140+0000","last_became_peered":"2024-06-29T14:41:58.533140+0000","last_unstale":"2024-06-29T14:41:59.532731+0000","last_undegraded":"2024-06-29T14:41:59.532731+0000","last_fullsized":"2024-06-29T14:41:59.532731+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:51.449930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532833+0000","last_change":"2024-06-29T14:41:57.513038+0000","last_active":"2024-06-29T14:41:59.532833+0000","last_peered":"2024-06-29T14:41:59.532833+0000","last_clean":"2024-06-29T14:41:59.532833+0000","last_became_active":"2024-06-29T14:41:57.512729+0000","last_became_peered":"2024-06-29T14:41:57.512729+0000","last_unstale":"2024-06-29T14:41:59.532833+0000","last_undegraded":"2024-06-29T14:41:59.532833+0000","last_fullsized":"2024-06-29T14:41:59.532833+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:18.745856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.533034+0000","last_change":"2024-06-29T14:41:58.531798+0000","last_active":"2024-06-29T14:41:59.533034+0000","last_peered":"2024-06-29T14:41:59.533034+0000","last_clean":"2024-06-29T14:41:59.533034+0000","last_became_active":"2024-06-29T14:41:58.531499+0000","last_became_peered":"2024-06-29T14:41:58.531499+0000","last_unstale":"2024-06-29T14:41:59.533034+0000","last_undegraded":"2024-06-29T14:41:59.533034+0000","last_fullsized":"2024-06-29T14:41:59.533034+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:28.287087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527282+0000","last_change":"2024-06-29T14:41:57.506196+0000","last_active":"2024-06-29T14:41:59.527282+0000","last_peered":"2024-06-29T14:41:59.527282+0000","last_clean":"2024-06-29T14:41:59.527282+0000","last_became_active":"2024-06-29T14:41:57.505842+0000","last_became_peered":"2024-06-29T14:41:57.505842+0000","last_unstale":"2024-06-29T14:41:59.527282+0000","last_undegraded":"2024-06-29T14:41:59.527282+0000","last_fullsized":"2024-06-29T14:41:59.527282+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:36.993862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532603+0000","last_change":"2024-06-29T14:41:58.527859+0000","last_active":"2024-06-29T14:41:59.532603+0000","last_peered":"2024-06-29T14:41:59.532603+0000","last_clean":"2024-06-29T14:41:59.532603+0000","last_became_active":"2024-06-29T14:41:58.527463+0000","last_became_peered":"2024-06-29T14:41:58.527463+0000","last_unstale":"2024-06-29T14:41:59.532603+0000","last_undegraded":"2024-06-29T14:41:59.532603+0000","last_fullsized":"2024-06-29T14:41:59.532603+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:03.105937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.532494+0000","last_change":"2024-06-29T14:41:57.505760+0000","last_active":"2024-06-29T14:41:59.532494+0000","last_peered":"2024-06-29T14:41:59.532494+0000","last_clean":"2024-06-29T14:41:59.532494+0000","last_became_active":"2024-06-29T14:41:57.505476+0000","last_became_peered":"2024-06-29T14:41:57.505476+0000","last_unstale":"2024-06-29T14:41:59.532494+0000","last_undegraded":"2024-06-29T14:41:59.532494+0000","last_fullsized":"2024-06-29T14:41:59.532494+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:15.695601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.526324+0000","last_change":"2024-06-29T14:41:58.533381+0000","last_active":"2024-06-29T14:41:59.526324+0000","last_peered":"2024-06-29T14:41:59.526324+0000","last_clean":"2024-06-29T14:41:59.526324+0000","last_became_active":"2024-06-29T14:41:58.533071+0000","last_became_peered":"2024-06-29T14:41:58.533071+0000","last_unstale":"2024-06-29T14:41:59.526324+0000","last_undegraded":"2024-06-29T14:41:59.526324+0000","last_fullsized":"2024-06-29T14:41:59.526324+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:24:41.804258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.523827+0000","last_change":"2024-06-29T14:41:57.515433+0000","last_active":"2024-06-29T14:41:59.523827+0000","last_peered":"2024-06-29T14:41:59.523827+0000","last_clean":"2024-06-29T14:41:59.523827+0000","last_became_active":"2024-06-29T14:41:57.515134+0000","last_became_peered":"2024-06-29T14:41:57.515134+0000","last_unstale":"2024-06-29T14:41:59.523827+0000","last_undegraded":"2024-06-29T14:41:59.523827+0000","last_fullsized":"2024-06-29T14:41:59.523827+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:09.488545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525725+0000","last_change":"2024-06-29T14:41:58.524691+0000","last_active":"2024-06-29T14:41:59.525725+0000","last_peered":"2024-06-29T14:41:59.525725+0000","last_clean":"2024-06-29T14:41:59.525725+0000","last_became_active":"2024-06-29T14:41:58.523946+0000","last_became_peered":"2024-06-29T14:41:58.523946+0000","last_unstale":"2024-06-29T14:41:59.525725+0000","last_undegraded":"2024-06-29T14:41:59.525725+0000","last_fullsized":"2024-06-29T14:41:59.525725+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.488118+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.488118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:37:02.005962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.527130+0000","last_change":"2024-06-29T14:41:57.512547+0000","last_active":"2024-06-29T14:41:59.527130+0000","last_peered":"2024-06-29T14:41:59.527130+0000","last_clean":"2024-06-29T14:41:59.527130+0000","last_became_active":"2024-06-29T14:41:57.512183+0000","last_became_peered":"2024-06-29T14:41:57.512183+0000","last_unstale":"2024-06-29T14:41:59.527130+0000","last_undegraded":"2024-06-29T14:41:59.527130+0000","last_fullsized":"2024-06-29T14:41:59.527130+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.480722+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.480722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:16:51.038914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"145'170","reported_seq":719,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.525766+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T14:41:59.525766+0000","last_peered":"2024-06-29T14:41:59.525766+0000","last_clean":"2024-06-29T14:41:59.525766+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T14:41:59.525766+0000","last_undegraded":"2024-06-29T14:41:59.525766+0000","last_fullsized":"2024-06-29T14:41:59.525766+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":269,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":269,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542386,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60040,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311800,"statfs":{"total":96636764160,"available":96575283200,"internally_reserved":0,"allocated":2547712,"data_stored":1575577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542387,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1933312,"data_stored":977459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542386,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":1929216,"data_stored":976809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542385,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1937408,"data_stored":983507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509683,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":1953792,"data_stored":985189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575091,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59456,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312384,"statfs":{"total":96636764160,"available":96575881216,"internally_reserved":0,"allocated":1945600,"data_stored":984123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509682,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1941504,"data_stored":983577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509684,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37808,"kb_used_data":2468,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334032,"statfs":{"total":96636764160,"available":96598048768,"internally_reserved":0,"allocated":2527232,"data_stored":1567211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:42:02.988 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:42:02.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:42:03.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.153+0000 7f4859dbd640 1 -- 172.21.15.59:0/2936824778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48540cefb0 msgr2=0x7f48540b7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.153+0000 7f4859dbd640 1 --2- 172.21.15.59:0/2936824778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48540cefb0 0x7f48540b7fb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f484c005fd0 tx=0x7f484c02aa10 comp rx=0 tx=0).stop 2024-06-29T14:42:03.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.153+0000 7f4859dbd640 1 -- 172.21.15.59:0/2936824778 shutdown_connections 2024-06-29T14:42:03.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.153+0000 7f4859dbd640 1 --2- 172.21.15.59:0/2936824778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48540cefb0 0x7f48540b7fb0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.153+0000 7f4859dbd640 1 --2- 172.21.15.59:0/2936824778 >> v2:172.21.15.153:3300/0 conn(0x7f48540b4c20 0x7f48540b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.153+0000 7f4859dbd640 1 -- 172.21.15.59:0/2936824778 >> 172.21.15.59:0/2936824778 conn(0x7f48540d4070 msgr2=0x7f48540d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:03.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.153+0000 7f4859dbd640 1 -- 172.21.15.59:0/2936824778 shutdown_connections 2024-06-29T14:42:03.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.153+0000 7f4859dbd640 1 -- 172.21.15.59:0/2936824778 wait complete. 2024-06-29T14:42:03.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.155+0000 7f4859dbd640 1 Processor -- start 2024-06-29T14:42:03.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.155+0000 7f4859dbd640 1 -- start start 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f4859dbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48540ce090 0x7f4854158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f4859dbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48540cefb0 0x7f485415eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f4859dbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f485415f060 0x7f485415f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f4859dbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f48540b8880 con 0x7f48540ce090 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f4859dbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48540b8700 con 0x7f48540cefb0 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f4859dbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f48540b8a00 con 0x7f485415f060 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f48537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48540ce090 0x7f4854158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f4853fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f485415f060 0x7f485415f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f48537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48540ce090 0x7f4854158b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36508/0 (socket says 172.21.15.59:36508) 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f48537fe640 1 -- 172.21.15.59:0/691081027 learned_addr learned my addr 172.21.15.59:0/691081027 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.156+0000 7f4852ffd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48540cefb0 0x7f485415eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f48537fe640 1 -- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f485415f060 msgr2=0x7f485415f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f48537fe640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f485415f060 0x7f485415f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f48537fe640 1 -- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48540cefb0 msgr2=0x7f485415eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f48537fe640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48540cefb0 0x7f485415eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f48537fe640 1 -- 172.21.15.59:0/691081027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485415f9b0 con 0x7f48540ce090 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f48537fe640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48540ce090 0x7f4854158b20 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f484400c3d0 tx=0x7f484400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f483bfff640 1 -- 172.21.15.59:0/691081027 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844017070 con 0x7f48540ce090 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f483bfff640 1 -- 172.21.15.59:0/691081027 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48440053c0 con 0x7f48540ce090 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4854159490 con 0x7f48540ce090 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f483bfff640 1 -- 172.21.15.59:0/691081027 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844003850 con 0x7f48540ce090 2024-06-29T14:42:03.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.157+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4854159a50 con 0x7f48540ce090 2024-06-29T14:42:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.158+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48540b8f90 con 0x7f48540ce090 2024-06-29T14:42:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.159+0000 7f483bfff640 1 -- 172.21.15.59:0/691081027 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f484401a070 con 0x7f48540ce090 2024-06-29T14:42:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.160+0000 7f483bfff640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f48400763f0 0x7f4840078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.160+0000 7f483bfff640 1 -- 172.21.15.59:0/691081027 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f48440971c0 con 0x7f48540ce090 2024-06-29T14:42:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.160+0000 7f4852ffd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f48400763f0 0x7f4840078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.160+0000 7f4852ffd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f48400763f0 0x7f4840078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48540b33e0 tx=0x7f4848005f90 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:03.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.163+0000 7f483bfff640 1 -- 172.21.15.59:0/691081027 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4844021020 con 0x7f48540ce090 2024-06-29T14:42:03.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.432+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f48540cfed0 con 0x7f48540ce090 2024-06-29T14:42:03.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.435+0000 7f483bfff640 1 -- 172.21.15.59:0/691081027 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) ==== 190+0+147 (secure 0 0 0) 0x7f4844060260 con 0x7f48540ce090 2024-06-29T14:42:03.432 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T14:42:03.432 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T14:42:03.433 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T14:42:03.433 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T14:42:03.433 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:42:03.433 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f48400763f0 msgr2=0x7f4840078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f48400763f0 0x7f4840078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48540b33e0 tx=0x7f4848005f90 comp rx=0 tx=0).stop 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48540ce090 msgr2=0x7f4854158b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48540ce090 0x7f4854158b20 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f484400c3d0 tx=0x7f484400c890 comp rx=0 tx=0).stop 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 shutdown_connections 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f48400763f0 0x7f4840078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f485415f060 0x7f485415f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48540cefb0 0x7f485415eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 --2- 172.21.15.59:0/691081027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48540ce090 0x7f4854158b20 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 >> 172.21.15.59:0/691081027 conn(0x7f48540d4070 msgr2=0x7f4854158380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:03.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 shutdown_connections 2024-06-29T14:42:03.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.437+0000 7f4859dbd640 1 -- 172.21.15.59:0/691081027 wait complete. 2024-06-29T14:42:03.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:42:03.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 -- 172.21.15.59:0/1998768638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100ce950 msgr2=0x7f33100d1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 --2- 172.21.15.59:0/1998768638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100ce950 0x7f33100d1f60 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f3310069f10 tx=0x7f330802aa30 comp rx=0 tx=0).stop 2024-06-29T14:42:03.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 -- 172.21.15.59:0/1998768638 shutdown_connections 2024-06-29T14:42:03.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 --2- 172.21.15.59:0/1998768638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100ce950 0x7f33100d1f60 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 --2- 172.21.15.59:0/1998768638 >> v2:172.21.15.153:3300/0 conn(0x7f33100cf5c0 0x7f33100d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 --2- 172.21.15.59:0/1998768638 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f331008c190 0x7f33100cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 -- 172.21.15.59:0/1998768638 >> 172.21.15.59:0/1998768638 conn(0x7f33100b5610 msgr2=0x7f33100b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:03.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 -- 172.21.15.59:0/1998768638 shutdown_connections 2024-06-29T14:42:03.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.567+0000 7f3316a8f640 1 -- 172.21.15.59:0/1998768638 wait complete. 2024-06-29T14:42:03.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.568+0000 7f3316a8f640 1 Processor -- start 2024-06-29T14:42:03.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.569+0000 7f3316a8f640 1 -- start start 2024-06-29T14:42:03.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f3316a8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f331008c190 0x7f33101543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f3316a8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33100ce950 0x7f33101f4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f3316a8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100cf5c0 0x7f3310160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f3316a8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3310162440 con 0x7f33100cf5c0 2024-06-29T14:42:03.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f3316a8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f33101622c0 con 0x7f331008c190 2024-06-29T14:42:03.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f3316a8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f33101625c0 con 0x7f33100ce950 2024-06-29T14:42:03.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331502c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100cf5c0 0x7f3310160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331502c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100cf5c0 0x7f3310160f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36534/0 (socket says 172.21.15.59:36534) 2024-06-29T14:42:03.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331502c640 1 -- 172.21.15.59:0/12487597 learned_addr learned my addr 172.21.15.59:0/12487597 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:03.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f330ffff640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33100ce950 0x7f33101f4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331482b640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f331008c190 0x7f33101543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331502c640 1 -- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33100ce950 msgr2=0x7f33101f4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331502c640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33100ce950 0x7f33101f4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331502c640 1 -- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f331008c190 msgr2=0x7f33101543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331502c640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f331008c190 0x7f33101543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.570+0000 7f331502c640 1 -- 172.21.15.59:0/12487597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3310149240 con 0x7f33100cf5c0 2024-06-29T14:42:03.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.571+0000 7f331502c640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100cf5c0 0x7f3310160f90 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f330802b020 tx=0x7f3308003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:03.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.571+0000 7f330d7fa640 1 -- 172.21.15.59:0/12487597 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308033050 con 0x7f33100cf5c0 2024-06-29T14:42:03.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.571+0000 7f330d7fa640 1 -- 172.21.15.59:0/12487597 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3308004550 con 0x7f33100cf5c0 2024-06-29T14:42:03.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.571+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33100caeb0 con 0x7f33100cf5c0 2024-06-29T14:42:03.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.571+0000 7f330d7fa640 1 -- 172.21.15.59:0/12487597 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308038450 con 0x7f33100cf5c0 2024-06-29T14:42:03.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.571+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33101497e0 con 0x7f33100cf5c0 2024-06-29T14:42:03.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.572+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33100cf080 con 0x7f33100cf5c0 2024-06-29T14:42:03.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.573+0000 7f330d7fa640 1 -- 172.21.15.59:0/12487597 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f3308036070 con 0x7f33100cf5c0 2024-06-29T14:42:03.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.573+0000 7f330d7fa640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f32f0076420 0x7f32f0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.573+0000 7f330d7fa640 1 -- 172.21.15.59:0/12487597 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f33080b8fd0 con 0x7f33100cf5c0 2024-06-29T14:42:03.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.574+0000 7f331482b640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f32f0076420 0x7f32f0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.574+0000 7f331482b640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f32f0076420 0x7f32f0078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f32f800aa10 tx=0x7f32f8005ef0 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:03.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.578+0000 7f330d7fa640 1 -- 172.21.15.59:0/12487597 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3308081e80 con 0x7f33100cf5c0 2024-06-29T14:42:03.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.811+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33100cac00 con 0x7f33100cf5c0 2024-06-29T14:42:03.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.811+0000 7f330d7fa640 1 -- 172.21.15.59:0/12487597 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5225 (secure 0 0 0) 0x7f3308086180 con 0x7f33100cf5c0 2024-06-29T14:42:03.808 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:42:03.808 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"btime":"2024-06-29T14:42:01:534262+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/507949731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":507949731},{"type":"v1","addr":"172.21.15.153:6837","nonce":507949731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":8812,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1668688055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1668688055},{"type":"v1","addr":"172.21.15.153:6836","nonce":1668688055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":9372,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3535921856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3535921856},{"type":"v1","addr":"172.21.15.59:6834","nonce":3535921856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.891192+0000","modified":"2024-06-29T14:42:00.924713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/178954889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":178954889},{"type":"v1","addr":"172.21.15.59:6835","nonce":178954889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9375,"qdb_cluster":[9375]},"id":9}]} 2024-06-29T14:42:03.809 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 80 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f32f0076420 msgr2=0x7f32f0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f32f0076420 0x7f32f0078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f32f800aa10 tx=0x7f32f8005ef0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100cf5c0 msgr2=0x7f3310160f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100cf5c0 0x7f3310160f90 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f330802b020 tx=0x7f3308003f10 comp rx=0 tx=0).stop 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 shutdown_connections 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f32f0076420 0x7f32f0078930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100cf5c0 0x7f3310160f90 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33100ce950 0x7f33101f4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 --2- 172.21.15.59:0/12487597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f331008c190 0x7f33101543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 >> 172.21.15.59:0/12487597 conn(0x7f33100b5610 msgr2=0x7f33100b89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 shutdown_connections 2024-06-29T14:42:03.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:03.813+0000 7f3316a8f640 1 -- 172.21.15.59:0/12487597 wait complete. 2024-06-29T14:42:04.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:41:57.891192+0000', 'modified': '2024-06-29T14:42:00.924713+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/178954889', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 178954889}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 178954889}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9375, 'qdb_cluster': [9375]} 2024-06-29T14:42:04.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:42:04.225 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:42:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.352+0000 7f3b01563640 1 -- 172.21.15.59:0/1876031122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 msgr2=0x7f3afc0b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.352+0000 7f3b01563640 1 --2- 172.21.15.59:0/1876031122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 0x7f3afc0b56d0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0b5590 tx=0x7f3aec02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.352+0000 7f3b01563640 1 -- 172.21.15.59:0/1876031122 shutdown_connections 2024-06-29T14:42:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.352+0000 7f3b01563640 1 --2- 172.21.15.59:0/1876031122 >> v2:172.21.15.153:3300/0 conn(0x7f3afc13e9b0 0x7f3afc13edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.352+0000 7f3b01563640 1 --2- 172.21.15.59:0/1876031122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 0x7f3afc0b56d0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.352+0000 7f3b01563640 1 -- 172.21.15.59:0/1876031122 >> 172.21.15.59:0/1876031122 conn(0x7f3afc0b6fb0 msgr2=0x7f3afc0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.352+0000 7f3b01563640 1 -- 172.21.15.59:0/1876031122 shutdown_connections 2024-06-29T14:42:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.352+0000 7f3b01563640 1 -- 172.21.15.59:0/1876031122 wait complete. 2024-06-29T14:42:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.353+0000 7f3b01563640 1 Processor -- start 2024-06-29T14:42:04.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.353+0000 7f3b01563640 1 -- start start 2024-06-29T14:42:04.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3b01563640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3afc13e9b0 0x7f3afc0c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3b01563640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3afc1414c0 0x7f3afc0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3b01563640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 0x7f3afc13c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3b01563640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc0b29d0 con 0x7f3afc15ceb0 2024-06-29T14:42:04.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3b01563640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc0b2850 con 0x7f3afc13e9b0 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3b01563640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3afc0b2b50 con 0x7f3afc1414c0 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 0x7f3afc13c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 0x7f3afc13c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36544/0 (socket says 172.21.15.59:36544) 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 -- 172.21.15.59:0/3406492737 learned_addr learned my addr 172.21.15.59:0/3406492737 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afaffd640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3afc13e9b0 0x7f3afc0c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3af3fff640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3afc1414c0 0x7f3afc0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 -- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3afc1414c0 msgr2=0x7f3afc0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3afc1414c0 0x7f3afc0ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 -- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3afc13e9b0 msgr2=0x7f3afc0c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3afc13e9b0 0x7f3afc0c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 -- 172.21.15.59:0/3406492737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3afc0c8040 con 0x7f3afc15ceb0 2024-06-29T14:42:04.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3af3fff640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3afc1414c0 0x7f3afc0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.355+0000 7f3afb7fe640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 0x7f3afc13c6f0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f3af400cab0 tx=0x7f3af400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.356+0000 7f3af37fe640 1 -- 172.21.15.59:0/3406492737 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4016020 con 0x7f3afc15ceb0 2024-06-29T14:42:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.356+0000 7f3af37fe640 1 -- 172.21.15.59:0/3406492737 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3af400ad20 con 0x7f3afc15ceb0 2024-06-29T14:42:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.356+0000 7f3af37fe640 1 -- 172.21.15.59:0/3406492737 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4012660 con 0x7f3afc15ceb0 2024-06-29T14:42:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.356+0000 7f3b01563640 1 -- 172.21.15.59:0/3406492737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3afc1f2c00 con 0x7f3afc15ceb0 2024-06-29T14:42:04.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.356+0000 7f3b01563640 1 -- 172.21.15.59:0/3406492737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3afc1f31c0 con 0x7f3afc15ceb0 2024-06-29T14:42:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.357+0000 7f3af37fe640 1 -- 172.21.15.59:0/3406492737 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f3af4019020 con 0x7f3afc15ceb0 2024-06-29T14:42:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.358+0000 7f3af37fe640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3adc076420 0x7f3adc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.358+0000 7f3af37fe640 1 -- 172.21.15.59:0/3406492737 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f3af4097d30 con 0x7f3afc15ceb0 2024-06-29T14:42:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.358+0000 7f3afaffd640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3adc076420 0x7f3adc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.359+0000 7f3afaffd640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3adc076420 0x7f3adc078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3aec034000 tx=0x7f3aec004730 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.359+0000 7f3b0841c740 1 Processor -- start 2024-06-29T14:42:04.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.359+0000 7f3b0841c740 1 -- start start 2024-06-29T14:42:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.360+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77e0f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779fa3b0 0x555a77df33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a77a02590 0x555a77df38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3b0841c740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555a77d8bbc0 con 0x555a779fa3b0 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3b0841c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555a77d3a630 con 0x555a77a02590 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3b0841c740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555a77d29bd0 con 0x555a77a23270 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779fa3b0 0x555a77df33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779fa3b0 0x555a77df33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36548/0 (socket says 172.21.15.59:36548) 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 -- 172.21.15.59:0/105202163 learned_addr learned my addr 172.21.15.59:0/105202163 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3afaffd640 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77e0f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3afb7fe640 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a77a02590 0x555a77df38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 -- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 msgr2=0x555a77e0f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77e0f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 -- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a77a02590 msgr2=0x555a77df38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a77a02590 0x555a77df38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 -- 172.21.15.59:0/105202163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a77d26ca0 con 0x555a779fa3b0 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3afaffd640 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77e0f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af3fff640 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779fa3b0 0x555a77df33a0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400cbb0 tx=0x7f3ae400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c763351466da3270 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af0ff9640 1 -- 172.21.15.59:0/105202163 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4005220 con 0x555a779fa3b0 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af0ff9640 1 -- 172.21.15.59:0/105202163 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ae40053c0 con 0x555a779fa3b0 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.361+0000 7f3af0ff9640 1 -- 172.21.15.59:0/105202163 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae40047a0 con 0x555a779fa3b0 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 -- 172.21.15.59:0/105202163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779fa3b0 msgr2=0x555a77df33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779fa3b0 0x555a77df33a0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400cbb0 tx=0x7f3ae400e720 comp rx=0 tx=0).stop 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 -- 172.21.15.59:0/105202163 shutdown_connections 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a77a02590 0x555a77df38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779fa3b0 0x555a77df33a0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 --2- 172.21.15.59:0/105202163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77e0f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 -- 172.21.15.59:0/105202163 >> 172.21.15.59:0/105202163 conn(0x555a77a173c0 msgr2=0x555a77dc6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 -- 172.21.15.59:0/105202163 shutdown_connections 2024-06-29T14:42:04.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.362+0000 7f3b0841c740 1 -- 172.21.15.59:0/105202163 wait complete. 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.363+0000 7f3b0841c740 1 -- start start 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.363+0000 7f3b0841c740 1 Processor -- start 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 0x555a77e0d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77deb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77df3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3b0841c740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555a77cc7ae0 con 0x555a77a02590 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3b0841c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555a77d2dc10 con 0x555a779fa3b0 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3b0841c740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555a77a9d620 con 0x555a77a23270 2024-06-29T14:42:04.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3b0841c740 10 client.0 _ll_register_callbacks cb 0x555a77cd4480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77deb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77deb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36564/0 (socket says 172.21.15.59:36564) 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 -- 172.21.15.59:0/1054849971 learned_addr learned my addr 172.21.15.59:0/1054849971 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3afb7fe640 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77df3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3afaffd640 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 0x555a77e0d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 -- 172.21.15.59:0/1054849971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 msgr2=0x555a77df3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77df3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 -- 172.21.15.59:0/1054849971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 msgr2=0x555a77e0d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 0x555a77e0d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 -- 172.21.15.59:0/1054849971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a77a8e850 con 0x555a77a02590 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3af3fff640 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77deb340 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4016000 tx=0x7f3ae401aab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fa93fb8a2932d7 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3ad9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:42:04.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/1054849971 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4005630 con 0x555a77a02590 2024-06-29T14:42:04.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/1054849971 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f3ae4025c80 con 0x555a77a02590 2024-06-29T14:42:04.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.365+0000 7f3b0841c740 1 -- 172.21.15.59:0/1054849971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ae40053c0 con 0x555a77a02590 2024-06-29T14:42:04.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/1054849971 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f3ae4025e80 con 0x555a77a02590 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3b0841c740 10 client.9513 fetch_fsmap learned FSMap version 80 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3b0841c740 1 -- 172.21.15.59:0/1054849971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) -- 0x555a77b056b0 con 0x555a77a02590 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/1054849971 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ae40053c0 con 0x555a77a02590 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/1054849971 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 80) ==== 3502+0+0 (secure 0 0 0) 0x7f3ae402c050 con 0x555a77a02590 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3b0841c740 10 client.9513 fetch_fsmap finished waiting for FSMap version 80 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3b0841c740 10 client.9513 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3b0841c740 20 client.9513 populate_metadata read hostname 'smithi059' 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] conn(0x555a779e1e40 0x555a77df1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3b0841c740 4 client.9513 mds_command: new command op to 9375 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3b0841c740 1 -- 172.21.15.59:0/1054849971 --> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a77d3c580 con 0x555a779e1e40 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3afaffd640 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] conn(0x555a779e1e40 0x555a77df1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3afaffd640 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] conn(0x555a779e1e40 0x555a77df1350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=969e1c5af2392958 server_cookie=ca47ef0f48232750 in_seq=0 out_seq=0 2024-06-29T14:42:04.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.366+0000 7f3ad9ffb640 10 client.9513 ms_handle_connect on v2:172.21.15.59:6833/178954889 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/1054849971 <== mds.0 v2:172.21.15.59:6833/178954889 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555a77d3c580 con 0x555a779e1e40 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3ad9ffb640 10 client.9513 handle_command_reply: tid=0 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 client.9513 shutdown 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 -- 172.21.15.59:0/1054849971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 msgr2=0x555a77deb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77deb340 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4016000 tx=0x7f3ae401aab0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 -- 172.21.15.59:0/1054849971 shutdown_connections 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a23270 0x555a77df3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] conn(0x555a779e1e40 0x555a77df1350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3ad9ffb640 0 client.9513 ms_handle_reset on v2:172.21.15.59:6833/178954889 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77deb340 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1054849971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 0x555a77e0d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 -- 172.21.15.59:0/1054849971 >> 172.21.15.59:0/1054849971 conn(0x555a77a173c0 msgr2=0x555a77dc2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 -- 172.21.15.59:0/1054849971 shutdown_connections 2024-06-29T14:42:04.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.367+0000 7f3b0841c740 1 -- 172.21.15.59:0/1054849971 wait complete. 2024-06-29T14:42:04.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.368+0000 7f3b0841c740 20 client.9513 trim_cache size 0 max 16384 2024-06-29T14:42:04.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.387+0000 7f3b0841c740 1 Processor -- start 2024-06-29T14:42:04.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.387+0000 7f3b0841c740 1 -- start start 2024-06-29T14:42:04.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779e1e40 0x555a77e0ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 0x555a77e84b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a02590 0x555a77e85060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3b0841c740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc1fea90 con 0x555a779e1e40 2024-06-29T14:42:04.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3b0841c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc137b90 con 0x555a779fa3b0 2024-06-29T14:42:04.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3b0841c740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3afc1fe420 con 0x555a77a02590 2024-06-29T14:42:04.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3afaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779e1e40 0x555a77e0ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3afaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779e1e40 0x555a77e0ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36576/0 (socket says 172.21.15.59:36576) 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.388+0000 7f3afaffd640 1 -- 172.21.15.59:0/1966320221 learned_addr learned my addr 172.21.15.59:0/1966320221 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3afaffd640 1 -- 172.21.15.59:0/1966320221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a02590 msgr2=0x555a77e85060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3afaffd640 1 --2- 172.21.15.59:0/1966320221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a02590 0x555a77e85060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3afaffd640 1 -- 172.21.15.59:0/1966320221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 msgr2=0x555a77e84b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3afaffd640 1 --2- 172.21.15.59:0/1966320221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 0x555a77e84b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3afaffd640 1 -- 172.21.15.59:0/1966320221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a77e21920 con 0x555a779e1e40 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3afaffd640 1 --2- 172.21.15.59:0/1966320221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779e1e40 0x555a77e0ea20 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009780 tx=0x7f3aec03d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e049dd66a47526 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3adbfff640 1 -- 172.21.15.59:0/1966320221 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec032e30 con 0x555a779e1e40 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3adbfff640 1 -- 172.21.15.59:0/1966320221 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3aec055070 con 0x555a779e1e40 2024-06-29T14:42:04.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.389+0000 7f3adbfff640 1 -- 172.21.15.59:0/1966320221 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec0505b0 con 0x555a779e1e40 2024-06-29T14:42:04.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 -- 172.21.15.59:0/1966320221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779e1e40 msgr2=0x555a77e0ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1966320221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779e1e40 0x555a77e0ea20 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009780 tx=0x7f3aec03d070 comp rx=0 tx=0).stop 2024-06-29T14:42:04.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 -- 172.21.15.59:0/1966320221 shutdown_connections 2024-06-29T14:42:04.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1966320221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a77a02590 0x555a77e85060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1966320221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779fa3b0 0x555a77e84b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 --2- 172.21.15.59:0/1966320221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a779e1e40 0x555a77e0ea20 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 -- 172.21.15.59:0/1966320221 >> 172.21.15.59:0/1966320221 conn(0x555a77a173c0 msgr2=0x555a77deb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 -- 172.21.15.59:0/1966320221 shutdown_connections 2024-06-29T14:42:04.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.390+0000 7f3b0841c740 1 -- 172.21.15.59:0/1966320221 wait complete. 2024-06-29T14:42:04.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.391+0000 7f3b0841c740 1 -- start start 2024-06-29T14:42:04.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.391+0000 7f3b0841c740 1 Processor -- start 2024-06-29T14:42:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779e1e40 0x555a77e309f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a779fa3b0 0x555a77e825f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3b0841c740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77e82b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3b0841c740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555a77e0e100 con 0x555a77a02590 2024-06-29T14:42:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3b0841c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555a77df24b0 con 0x555a779e1e40 2024-06-29T14:42:04.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3b0841c740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555a77e20f40 con 0x555a779fa3b0 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3b0841c740 10 client.0 _ll_register_callbacks cb 0x7f3ae4014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77e82b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77e82b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36592/0 (socket says 172.21.15.59:36592) 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afb7fe640 1 -- 172.21.15.59:0/3899090288 learned_addr learned my addr 172.21.15.59:0/3899090288 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3af3fff640 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a779fa3b0 0x555a77e825f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afaffd640 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779e1e40 0x555a77e309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afb7fe640 1 -- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a779fa3b0 msgr2=0x555a77e825f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afb7fe640 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a779fa3b0 0x555a77e825f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afb7fe640 1 -- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779e1e40 msgr2=0x555a77e309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afb7fe640 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779e1e40 0x555a77e309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afb7fe640 1 -- 172.21.15.59:0/3899090288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a77e842b0 con 0x555a77a02590 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.392+0000 7f3afaffd640 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779e1e40 0x555a77e309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:04.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3afb7fe640 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77e82b30 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f3af405d2a0 tx=0x7f3af405d1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a896587daac086e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3ad9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/3899090288 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4079070 con 0x555a77a02590 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/3899090288 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f3af4067260 con 0x555a77a02590 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 1 -- 172.21.15.59:0/3899090288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aec055070 con 0x555a77a02590 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/3899090288 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f3af40607c0 con 0x555a77a02590 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 10 client.9519 fetch_fsmap learned FSMap version 80 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 1 -- 172.21.15.59:0/3899090288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) -- 0x555a77e0b2a0 con 0x555a77a02590 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/3899090288 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3aec055070 con 0x555a77a02590 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/3899090288 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 80) ==== 3502+0+0 (secure 0 0 0) 0x7f3af405c620 con 0x555a77a02590 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 10 client.9519 fetch_fsmap finished waiting for FSMap version 80 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 10 client.9519 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 20 client.9519 populate_metadata read hostname 'smithi059' 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] conn(0x555a779b7280 0x555a77df17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 4 client.9519 mds_command: new command op to 9375 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3b0841c740 1 -- 172.21.15.59:0/3899090288 --> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] -- command(tid 0: {"prefix": "status"}) -- 0x555a77d38020 con 0x555a779b7280 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.393+0000 7f3afaffd640 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] conn(0x555a779b7280 0x555a77df17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3ad9ffb640 10 client.9519 ms_handle_connect on v2:172.21.15.59:6833/178954889 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3afaffd640 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] conn(0x555a779b7280 0x555a77df17d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59aa91243395c9a server_cookie=cbbf4f3d4adc801d in_seq=0 out_seq=0 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3ad9ffb640 1 -- 172.21.15.59:0/3899090288 <== mds.0 v2:172.21.15.59:6833/178954889 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x555a77d38020 con 0x555a779b7280 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3ad9ffb640 10 client.9519 handle_command_reply: tid=0 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 client.9519 shutdown 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 -- 172.21.15.59:0/3899090288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 msgr2=0x555a77e82b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77e82b30 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f3af405d2a0 tx=0x7f3af405d1f0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 -- 172.21.15.59:0/3899090288 shutdown_connections 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.59:6833/178954889,v1:172.21.15.59:6835/178954889] conn(0x555a779b7280 0x555a77df17d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555a77a02590 0x555a77e82b30 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3ad9ffb640 0 client.9519 ms_handle_reset on v2:172.21.15.59:6833/178954889 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555a779fa3b0 0x555a77e825f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 --2- 172.21.15.59:0/3899090288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555a779e1e40 0x555a77e309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 -- 172.21.15.59:0/3899090288 >> 172.21.15.59:0/3899090288 conn(0x555a77a173c0 msgr2=0x555a77e0ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 -- 172.21.15.59:0/3899090288 shutdown_connections 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 1 -- 172.21.15.59:0/3899090288 wait complete. 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.394+0000 7f3b0841c740 20 client.9519 trim_cache size 0 max 16384 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T14:42:04.397 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 9375, 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8893021970000001, 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 80, 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 156, 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 156, 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.505285285999999 2024-06-29T14:42:04.398 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T14:42:04.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 -- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3adc076420 msgr2=0x7f3adc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3adc076420 0x7f3adc078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3aec034000 tx=0x7f3aec004730 comp rx=0 tx=0).stop 2024-06-29T14:42:04.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 -- 172.21.15.59:0/3406492737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 msgr2=0x7f3afc13c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 0x7f3afc13c6f0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f3af400cab0 tx=0x7f3af400cf70 comp rx=0 tx=0).stop 2024-06-29T14:42:04.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 -- 172.21.15.59:0/3406492737 shutdown_connections 2024-06-29T14:42:04.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f3adc076420 0x7f3adc078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3afc15ceb0 0x7f3afc13c6f0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3afc1414c0 0x7f3afc0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 --2- 172.21.15.59:0/3406492737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3afc13e9b0 0x7f3afc0c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 -- 172.21.15.59:0/3406492737 >> 172.21.15.59:0/3406492737 conn(0x7f3afc0b6fb0 msgr2=0x7f3afc1401e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 -- 172.21.15.59:0/3406492737 shutdown_connections 2024-06-29T14:42:04.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.395+0000 7f3b01563640 1 -- 172.21.15.59:0/3406492737 wait complete. 2024-06-29T14:42:04.403 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:42:04.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.524+0000 7f64fa67f640 1 -- 172.21.15.59:0/1569863164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64dc004d90 msgr2=0x7f64dc005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.524+0000 7f64fa67f640 1 --2- 172.21.15.59:0/1569863164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64dc004d90 0x7f64dc005260 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00c750 tx=0x7f64dc005830 comp rx=0 tx=0).stop 2024-06-29T14:42:04.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.524+0000 7f64f8c1c640 1 -- 172.21.15.59:0/1569863164 reap_dead start 2024-06-29T14:42:04.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.524+0000 7f64fa67f640 1 -- 172.21.15.59:0/1569863164 shutdown_connections 2024-06-29T14:42:04.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.524+0000 7f64fa67f640 1 -- 172.21.15.59:0/1569863164 >> 172.21.15.59:0/1569863164 conn(0x7f64f406cef0 msgr2=0x7f64f406d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.524+0000 7f64fa67f640 1 -- 172.21.15.59:0/1569863164 shutdown_connections 2024-06-29T14:42:04.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.524+0000 7f64fa67f640 1 -- 172.21.15.59:0/1569863164 wait complete. 2024-06-29T14:42:04.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.526+0000 7f64fa67f640 1 Processor -- start 2024-06-29T14:42:04.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.526+0000 7f64fa67f640 1 -- start start 2024-06-29T14:42:04.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64fa67f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64dc004d90 0x7f64f40cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64fa67f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64f415b690 0x7f64f413d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64fa67f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64f415baa0 0x7f64f413d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64fa67f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f64f40737a0 con 0x7f64f415baa0 2024-06-29T14:42:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64fa67f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4073620 con 0x7f64dc004d90 2024-06-29T14:42:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64fa67f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f64f415d1c0 con 0x7f64f415b690 2024-06-29T14:42:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64f8c1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64f415baa0 0x7f64f413d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64f8c1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64f415baa0 0x7f64f413d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36610/0 (socket says 172.21.15.59:36610) 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.527+0000 7f64f8c1c640 1 -- 172.21.15.59:0/2701233315 learned_addr learned my addr 172.21.15.59:0/2701233315 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f37fe640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64f415b690 0x7f64f413d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f3fff640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64dc004d90 0x7f64f40cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f8c1c640 1 -- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64f415b690 msgr2=0x7f64f413d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f8c1c640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64f415b690 0x7f64f413d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f8c1c640 1 -- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64dc004d90 msgr2=0x7f64f40cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f8c1c640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64dc004d90 0x7f64f40cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f8c1c640 1 -- 172.21.15.59:0/2701233315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f406e400 con 0x7f64f415baa0 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f8c1c640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64f415baa0 0x7f64f413d750 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00c840 tx=0x7f64ec0011e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f0ff9640 1 -- 172.21.15.59:0/2701233315 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec004d40 con 0x7f64f415baa0 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f0ff9640 1 -- 172.21.15.59:0/2701233315 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64ec02b950 con 0x7f64f415baa0 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64f0ff9640 1 -- 172.21.15.59:0/2701233315 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec02bd60 con 0x7f64f415baa0 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f4140e80 con 0x7f64f415baa0 2024-06-29T14:42:04.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.528+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f406e6a0 con 0x7f64f415baa0 2024-06-29T14:42:04.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.529+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f415cab0 con 0x7f64f415baa0 2024-06-29T14:42:04.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.530+0000 7f64f0ff9640 1 -- 172.21.15.59:0/2701233315 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f64ec006030 con 0x7f64f415baa0 2024-06-29T14:42:04.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.530+0000 7f64f0ff9640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f64e0076420 0x7f64e0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.530+0000 7f64f0ff9640 1 -- 172.21.15.59:0/2701233315 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f64ec03b020 con 0x7f64f415baa0 2024-06-29T14:42:04.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.531+0000 7f64f3fff640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f64e0076420 0x7f64e0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.531+0000 7f64f3fff640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f64e0076420 0x7f64e0078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64e4000c00 tx=0x7f64e4006b70 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.535+0000 7f64f0ff9640 1 -- 172.21.15.59:0/2701233315 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f64f4145030 con 0x7f64f415baa0 2024-06-29T14:42:04.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.765+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f64f4140c10 con 0x7f64f415baa0 2024-06-29T14:42:04.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.766+0000 7f64f0ff9640 1 -- 172.21.15.59:0/2701233315 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v80) ==== 53+0+73 (secure 0 0 0) 0x7f64ec081000 con 0x7f64f415baa0 2024-06-29T14:42:04.763 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:42:04.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f64e0076420 msgr2=0x7f64e0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f64e0076420 0x7f64e0078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64e4000c00 tx=0x7f64e4006b70 comp rx=0 tx=0).stop 2024-06-29T14:42:04.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64f415baa0 msgr2=0x7f64f413d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64f415baa0 0x7f64f413d750 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00c840 tx=0x7f64ec0011e0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 shutdown_connections 2024-06-29T14:42:04.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f64e0076420 0x7f64e0078930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64f415baa0 0x7f64f413d750 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64f415b690 0x7f64f413d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 --2- 172.21.15.59:0/2701233315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64dc004d90 0x7f64f40cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 >> 172.21.15.59:0/2701233315 conn(0x7f64f406cef0 msgr2=0x7f64f415f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 shutdown_connections 2024-06-29T14:42:04.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:04.768+0000 7f64fa67f640 1 -- 172.21.15.59:0/2701233315 wait complete. 2024-06-29T14:42:04.776 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:42:04.777 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:42:04.777 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T14:42:04.777 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:42:04.777 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:42:04.777 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:42:04.777 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:42:04.777 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:42:04.777 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:42:04.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:04.777 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:42:04.794 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 41526sec preferred_lft 41526sec 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:42:04.795 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:42:04.796 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T14:42:04.797 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:04.797 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:42:04.858 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T14:42:04.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:04.860 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T14:42:04.917 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T14:42:04.917 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:42:04.917 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T14:42:04.918 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T14:42:04.918 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:42:04.918 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:42:04.918 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T14:42:04.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T14:42:04.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T14:42:04.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T14:42:04.918 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T14:42:05.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:42:05.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:42:05.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:05.047 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T14:42:05.063 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:42:05.063 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:42:05.063 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:42:05.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:05.064 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T14:42:05.133 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:42:05.133 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:42:05.190 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:42:05.190 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:42:05.190 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:42:05.246 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T14:42:05.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:05.246 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:42:05.341 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:42:05.341 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:42:05.341 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:42:05.352 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:42:05.352 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:42:05.352 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:42:05.352 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:42:05.352 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:42:05.358 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T14:42:05.358 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T14:42:05.358 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T14:42:05.359 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T14:42:05.359 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T14:42:05.359 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T14:42:05.359 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T14:42:05.359 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T14:42:05.359 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:42:05.359 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:42:05.360 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:42:05.360 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:42:05.360 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:42:05.361 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:42:05.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:05.362 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:42:05.388 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:42:05.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:42:05.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:42:05.490 INFO:teuthology.orchestra.run.smithi059.stdout:client7928 2024-06-29T14:42:05.491 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T14:42:05.491 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:42:05.511 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 -- 172.21.15.59:0/771937238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000b3ed0 msgr2=0x7f4c0015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 --2- 172.21.15.59:0/771937238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000b3ed0 0x7f4c0015e8d0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f4c000d04f0 tx=0x7f4bf402a560 comp rx=0 tx=0).stop 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 -- 172.21.15.59:0/771937238 shutdown_connections 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 --2- 172.21.15.59:0/771937238 >> v2:172.21.15.153:3300/0 conn(0x7f4c0015ee10 0x7f4c0015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 --2- 172.21.15.59:0/771937238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000b3ed0 0x7f4c0015e8d0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 --2- 172.21.15.59:0/771937238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c000d23c0 0x7f4c000b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 -- 172.21.15.59:0/771937238 >> 172.21.15.59:0/771937238 conn(0x7f4c000d3ee0 msgr2=0x7f4c000d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 -- 172.21.15.59:0/771937238 shutdown_connections 2024-06-29T14:42:05.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.672+0000 7f4c05f43640 1 -- 172.21.15.59:0/771937238 wait complete. 2024-06-29T14:42:05.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.674+0000 7f4c05f43640 1 Processor -- start 2024-06-29T14:42:05.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.674+0000 7f4c05f43640 1 -- start start 2024-06-29T14:42:05.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4c05f43640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c000b3ed0 0x7f4c0015cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4c05f43640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000d23c0 0x7f4c00158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4c05f43640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c0015ee10 0x7f4c001593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4c05f43640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4c00160070 con 0x7f4c000d23c0 2024-06-29T14:42:05.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4c05f43640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0015fef0 con 0x7f4c0015ee10 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4c05f43640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4c001601f0 con 0x7f4c000b3ed0 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000d23c0 0x7f4c00158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000d23c0 0x7f4c00158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36630/0 (socket says 172.21.15.59:36630) 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3554745659 learned_addr learned my addr 172.21.15.59:0/3554745659 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bff7fe640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c000b3ed0 0x7f4c0015cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfffff640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c0015ee10 0x7f4c001593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c000b3ed0 msgr2=0x7f4c0015cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c000b3ed0 0x7f4c0015cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c0015ee10 msgr2=0x7f4c001593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c0015ee10 0x7f4c001593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3554745659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0015d7f0 con 0x7f4c000d23c0 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4bfeffd640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000d23c0 0x7f4c00158e70 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4034040 tx=0x7f4bf4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.676+0000 7f4be7fff640 1 -- 172.21.15.59:0/3554745659 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf403e030 con 0x7f4c000d23c0 2024-06-29T14:42:05.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.677+0000 7f4be7fff640 1 -- 172.21.15.59:0/3554745659 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4bf4003e00 con 0x7f4c000d23c0 2024-06-29T14:42:05.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.677+0000 7f4be7fff640 1 -- 172.21.15.59:0/3554745659 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4035040 con 0x7f4c000d23c0 2024-06-29T14:42:05.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.677+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0015dad0 con 0x7f4c000d23c0 2024-06-29T14:42:05.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.677+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c0015e090 con 0x7f4c000d23c0 2024-06-29T14:42:05.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.678+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c0015d560 con 0x7f4c000d23c0 2024-06-29T14:42:05.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.678+0000 7f4be7fff640 1 -- 172.21.15.59:0/3554745659 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f4bf4041030 con 0x7f4c000d23c0 2024-06-29T14:42:05.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.679+0000 7f4be7fff640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4bec0763f0 0x7f4bec078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.679+0000 7f4be7fff640 1 -- 172.21.15.59:0/3554745659 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f4bf40b8270 con 0x7f4c000d23c0 2024-06-29T14:42:05.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.679+0000 7f4bff7fe640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4bec0763f0 0x7f4bec078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.680+0000 7f4bff7fe640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4bec0763f0 0x7f4bec078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4c000d2e40 tx=0x7f4bf000b260 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:05.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.683+0000 7f4be7fff640 1 -- 172.21.15.59:0/3554745659 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4bf40811b0 con 0x7f4c000d23c0 2024-06-29T14:42:05.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.898+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 --> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f4c000b3990 con 0x7f4bec0763f0 2024-06-29T14:42:05.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.900+0000 7f4be7fff640 1 -- 172.21.15.59:0/3554745659 <== mgr.7658 v2:172.21.15.153:6832/3378196672 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f4c000b3990 con 0x7f4bec0763f0 2024-06-29T14:42:05.897 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T14:42:05.897 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T14:42:05.897 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T14:42:05.897 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T14:42:05.897 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T14:42:05.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4bec0763f0 msgr2=0x7f4bec078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4bec0763f0 0x7f4bec078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4c000d2e40 tx=0x7f4bf000b260 comp rx=0 tx=0).stop 2024-06-29T14:42:05.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000d23c0 msgr2=0x7f4c00158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000d23c0 0x7f4c00158e70 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4034040 tx=0x7f4bf4003910 comp rx=0 tx=0).stop 2024-06-29T14:42:05.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 shutdown_connections 2024-06-29T14:42:05.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f4bec0763f0 0x7f4bec078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c0015ee10 0x7f4c001593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c000d23c0 0x7f4c00158e70 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 --2- 172.21.15.59:0/3554745659 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c000b3ed0 0x7f4c0015cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 >> 172.21.15.59:0/3554745659 conn(0x7f4c000d3ee0 msgr2=0x7f4c00160730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:05.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 shutdown_connections 2024-06-29T14:42:05.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:05.902+0000 7f4c05f43640 1 -- 172.21.15.59:0/3554745659 wait complete. 2024-06-29T14:42:05.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:42:06.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 -- 172.21.15.59:0/4292233477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 msgr2=0x7fdf6c15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/4292233477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 0x7fdf6c15f9b0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0d15d0 tx=0x7fdf5c02aa30 comp rx=0 tx=0).stop 2024-06-29T14:42:06.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 -- 172.21.15.59:0/4292233477 shutdown_connections 2024-06-29T14:42:06.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/4292233477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 0x7fdf6c15f9b0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/4292233477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c0b3ed0 0x7fdf6c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/4292233477 >> v2:172.21.15.153:3300/0 conn(0x7fdf6c0d23c0 0x7fdf6c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 -- 172.21.15.59:0/4292233477 >> 172.21.15.59:0/4292233477 conn(0x7fdf6c0d3ee0 msgr2=0x7fdf6c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:06.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 -- 172.21.15.59:0/4292233477 shutdown_connections 2024-06-29T14:42:06.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.032+0000 7fdf70a3c640 1 -- 172.21.15.59:0/4292233477 wait complete. 2024-06-29T14:42:06.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.034+0000 7fdf70a3c640 1 Processor -- start 2024-06-29T14:42:06.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.034+0000 7fdf70a3c640 1 -- start start 2024-06-29T14:42:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf70a3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c0b3ed0 0x7fdf6c1ea450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf70a3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c0d23c0 0x7fdf6c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf70a3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 0x7fdf6c160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf70a3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c160070 con 0x7fdf6c15ee10 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf70a3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c15fef0 con 0x7fdf6c0b3ed0 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf70a3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6c1601f0 con 0x7fdf6c0d23c0 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 0x7fdf6c160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 0x7fdf6c160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36642/0 (socket says 172.21.15.59:36642) 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 -- 172.21.15.59:0/1585381162 learned_addr learned my addr 172.21.15.59:0/1585381162 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf69d9b640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c0d23c0 0x7fdf6c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6a59c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c0b3ed0 0x7fdf6c1ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 -- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c0d23c0 msgr2=0x7fdf6c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c0d23c0 0x7fdf6c15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 -- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c0b3ed0 msgr2=0x7fdf6c1ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c0b3ed0 0x7fdf6c1ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 -- 172.21.15.59:0/1585381162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c15d400 con 0x7fdf6c15ee10 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf69d9b640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c0d23c0 0x7fdf6c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6a59c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c0b3ed0 0x7fdf6c1ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.036+0000 7fdf6ad9d640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 0x7fdf6c160730 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009d60 tx=0x7fdf5c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.037+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1585381162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c033050 con 0x7fdf6c15ee10 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.037+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1585381162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf5c004550 con 0x7fdf6c15ee10 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.037+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6c15d6e0 con 0x7fdf6c15ee10 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.037+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1585381162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c038450 con 0x7fdf6c15ee10 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.037+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6c15dca0 con 0x7fdf6c15ee10 2024-06-29T14:42:06.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.038+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6c0b3990 con 0x7fdf6c15ee10 2024-06-29T14:42:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.039+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1585381162 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fdf5c036070 con 0x7fdf6c15ee10 2024-06-29T14:42:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.039+0000 7fdf62ffd640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fdf4c0763f0 0x7fdf4c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.039+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1585381162 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7fdf5c0b9270 con 0x7fdf6c15ee10 2024-06-29T14:42:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.039+0000 7fdf6a59c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fdf4c0763f0 0x7fdf4c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.040+0000 7fdf6a59c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fdf4c0763f0 0x7fdf4c078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0d2e40 tx=0x7fdf6400a4c0 comp rx=0 tx=0).ready entity=mgr.7658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.044+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1585381162 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf5c082770 con 0x7fdf6c15ee10 2024-06-29T14:42:06.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.291+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fdf6c1579b0 con 0x7fdf6c15ee10 2024-06-29T14:42:06.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.567+0000 7fdf62ffd640 1 -- 172.21.15.59:0/1585381162 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v79) ==== 92+0+0 (secure 0 0 0) 0x7fdf5c082070 con 0x7fdf6c15ee10 2024-06-29T14:42:06.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fdf4c0763f0 msgr2=0x7fdf4c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fdf4c0763f0 0x7fdf4c078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0d2e40 tx=0x7fdf6400a4c0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 msgr2=0x7fdf6c160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 0x7fdf6c160730 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009d60 tx=0x7fdf5c003f10 comp rx=0 tx=0).stop 2024-06-29T14:42:06.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 shutdown_connections 2024-06-29T14:42:06.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7fdf4c0763f0 0x7fdf4c078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf6c15ee10 0x7fdf6c160730 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdf6c0d23c0 0x7fdf6c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 --2- 172.21.15.59:0/1585381162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf6c0b3ed0 0x7fdf6c1ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.570+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 >> 172.21.15.59:0/1585381162 conn(0x7fdf6c0d3ee0 msgr2=0x7fdf6c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:06.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.571+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 shutdown_connections 2024-06-29T14:42:06.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.571+0000 7fdf70a3c640 1 -- 172.21.15.59:0/1585381162 wait complete. 2024-06-29T14:42:06.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:42:06.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.709+0000 7f1b37e8f640 1 -- 172.21.15.59:0/333739923 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b300b3990 msgr2=0x7f1b300b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.709+0000 7f1b37e8f640 1 --2- 172.21.15.59:0/333739923 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b300b3990 0x7f1b300b3da0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1b24009860 tx=0x7f1b2402a530 comp rx=0 tx=0).stop 2024-06-29T14:42:06.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.710+0000 7f1b37e8f640 1 -- 172.21.15.59:0/333739923 shutdown_connections 2024-06-29T14:42:06.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.710+0000 7f1b37e8f640 1 --2- 172.21.15.59:0/333739923 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b300b3990 0x7f1b300b3da0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.710+0000 7f1b37e8f640 1 -- 172.21.15.59:0/333739923 >> 172.21.15.59:0/333739923 conn(0x7f1b300d3ee0 msgr2=0x7f1b300d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:06.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.710+0000 7f1b37e8f640 1 -- 172.21.15.59:0/333739923 shutdown_connections 2024-06-29T14:42:06.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.710+0000 7f1b37e8f640 1 -- 172.21.15.59:0/333739923 wait complete. 2024-06-29T14:42:06.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.711+0000 7f1b37e8f640 1 Processor -- start 2024-06-29T14:42:06.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.711+0000 7f1b37e8f640 1 -- start start 2024-06-29T14:42:06.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b37e8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b300b3990 0x7f1b30158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b37e8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b300b73f0 0x7f1b30159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b37e8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b3015e290 0x7f1b3015a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b37e8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1b300b7f10 con 0x7f1b300b3990 2024-06-29T14:42:06.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b37e8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1b300cea30 con 0x7f1b3015e290 2024-06-29T14:42:06.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b37e8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1b300b8090 con 0x7f1b300b73f0 2024-06-29T14:42:06.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b300b3990 0x7f1b30158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b300b3990 0x7f1b30158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36664/0 (socket says 172.21.15.59:36664) 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 -- 172.21.15.59:0/3705108516 learned_addr learned my addr 172.21.15.59:0/3705108516 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b3542a640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b300b73f0 0x7f1b30159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b3642c640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b3015e290 0x7f1b3015a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 -- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b300b73f0 msgr2=0x7f1b30159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b300b73f0 0x7f1b30159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 -- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b3015e290 msgr2=0x7f1b3015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b3015e290 0x7f1b3015a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 -- 172.21.15.59:0/3705108516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b301f8dd0 con 0x7f1b300b3990 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.713+0000 7f1b35c2b640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b300b3990 0x7f1b30158110 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000cc50 tx=0x7f1b2000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.714+0000 7f1b2e7fc640 1 -- 172.21.15.59:0/3705108516 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b20016020 con 0x7f1b300b3990 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.714+0000 7f1b2e7fc640 1 -- 172.21.15.59:0/3705108516 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b20012420 con 0x7f1b300b3990 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.714+0000 7f1b2e7fc640 1 -- 172.21.15.59:0/3705108516 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b20012830 con 0x7f1b300b3990 2024-06-29T14:42:06.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.714+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b301e88e0 con 0x7f1b300b3990 2024-06-29T14:42:06.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.714+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b301e8ea0 con 0x7f1b300b3990 2024-06-29T14:42:06.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.715+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b300b3da0 con 0x7f1b300b3990 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.716+0000 7f1b2e7fc640 1 -- 172.21.15.59:0/3705108516 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 79) ==== 98320+0+0 (secure 0 0 0) 0x7f1b20019020 con 0x7f1b300b3990 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.716+0000 7f1b2e7fc640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1b1c076440 0x7f1b1c078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.716+0000 7f1b3542a640 1 -- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1b1c076440 msgr2=0x7f1b1c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3378196672 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.716+0000 7f1b2e7fc640 1 -- 172.21.15.59:0/3705108516 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f1b20098150 con 0x7f1b300b3990 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.716+0000 7f1b3542a640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1b1c076440 0x7f1b1c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:42:06.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.725+0000 7f1b2e7fc640 1 -- 172.21.15.59:0/3705108516 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1b20020020 con 0x7f1b300b3990 2024-06-29T14:42:06.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.917+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f1b301f92d0 con 0x7f1b300b3990 2024-06-29T14:42:06.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.918+0000 7f1b3542a640 1 -- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1b1c076440 msgr2=0x7f1b1c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3378196672 2024-06-29T14:42:06.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.918+0000 7f1b3542a640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1b1c076440 0x7f1b1c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:42:06.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.920+0000 7f1b2e7fc640 1 -- 172.21.15.59:0/3705108516 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v57)=0 v57) ==== 138+0+0 (secure 0 0 0) 0x7f1b200611a0 con 0x7f1b300b3990 2024-06-29T14:42:06.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1b1c076440 msgr2=0x7f1b1c078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:42:06.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1b1c076440 0x7f1b1c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b300b3990 msgr2=0x7f1b30158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b300b3990 0x7f1b30158110 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000cc50 tx=0x7f1b2000a430 comp rx=0 tx=0).stop 2024-06-29T14:42:06.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 shutdown_connections 2024-06-29T14:42:06.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f1b1c076440 0x7f1b1c078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b3015e290 0x7f1b3015a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b300b73f0 0x7f1b30159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 --2- 172.21.15.59:0/3705108516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b300b3990 0x7f1b30158110 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 >> 172.21.15.59:0/3705108516 conn(0x7f1b300d3ee0 msgr2=0x7f1b300ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:06.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 shutdown_connections 2024-06-29T14:42:06.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:06.923+0000 7f1b37e8f640 1 -- 172.21.15.59:0/3705108516 wait complete. 2024-06-29T14:42:06.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:42:07.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 -- 172.21.15.59:0/846896587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 msgr2=0x7f5d340d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 --2- 172.21.15.59:0/846896587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 0x7f5d340d1520 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f5d34074c50 tx=0x7f5d2402a560 comp rx=0 tx=0).stop 2024-06-29T14:42:07.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 -- 172.21.15.59:0/846896587 shutdown_connections 2024-06-29T14:42:07.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 --2- 172.21.15.59:0/846896587 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d340d1a60 0x7f5d340cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 --2- 172.21.15.59:0/846896587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 0x7f5d340d1520 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 --2- 172.21.15.59:0/846896587 >> v2:172.21.15.153:3300/0 conn(0x7f5d340b2270 0x7f5d3415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 -- 172.21.15.59:0/846896587 >> 172.21.15.59:0/846896587 conn(0x7f5d34073e10 msgr2=0x7f5d34074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:07.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 -- 172.21.15.59:0/846896587 shutdown_connections 2024-06-29T14:42:07.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.057+0000 7f5d3b773640 1 -- 172.21.15.59:0/846896587 wait complete. 2024-06-29T14:42:07.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.058+0000 7f5d3b773640 1 Processor -- start 2024-06-29T14:42:07.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.058+0000 7f5d3b773640 1 -- start start 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d3b773640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d340b2270 0x7f5d34141e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d3b773640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 0x7f5d34159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d3b773640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d340d1a60 0x7f5d340a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d3b773640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5d340cfe40 con 0x7f5d340b2fa0 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d3b773640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d340cfcc0 con 0x7f5d340b2270 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d3b773640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5d340cffc0 con 0x7f5d340d1a60 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d38d0e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 0x7f5d34159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d38d0e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 0x7f5d34159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36686/0 (socket says 172.21.15.59:36686) 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d38d0e640 1 -- 172.21.15.59:0/374414733 learned_addr learned my addr 172.21.15.59:0/374414733 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:07.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d3950f640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d340b2270 0x7f5d34141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.059+0000 7f5d39d10640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d340d1a60 0x7f5d340a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d3950f640 1 -- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d340d1a60 msgr2=0x7f5d340a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d3950f640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d340d1a60 0x7f5d340a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d3950f640 1 -- 172.21.15.59:0/374414733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 msgr2=0x7f5d34159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d3950f640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 0x7f5d34159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d3950f640 1 -- 172.21.15.59:0/374414733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3415aa60 con 0x7f5d340b2270 2024-06-29T14:42:07.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d38d0e640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 0x7f5d34159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:42:07.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d3950f640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d340b2270 0x7f5d34141e40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000e6d0 tx=0x7f5d3000eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:07.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d29ffb640 1 -- 172.21.15.59:0/374414733 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3000cab0 con 0x7f5d340b2270 2024-06-29T14:42:07.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d29ffb640 1 -- 172.21.15.59:0/374414733 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d3000cd40 con 0x7f5d340b2270 2024-06-29T14:42:07.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d29ffb640 1 -- 172.21.15.59:0/374414733 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30010690 con 0x7f5d340b2270 2024-06-29T14:42:07.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.060+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d3415ad40 con 0x7f5d340b2270 2024-06-29T14:42:07.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.061+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d34142b20 con 0x7f5d340b2270 2024-06-29T14:42:07.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.061+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d340d0450 con 0x7f5d340b2270 2024-06-29T14:42:07.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.062+0000 7f5d29ffb640 1 -- 172.21.15.59:0/374414733 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 79) ==== 98320+0+0 (secure 0 0 0) 0x7f5d30010830 con 0x7f5d340b2270 2024-06-29T14:42:07.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.063+0000 7f5d29ffb640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5d18076440 0x7f5d18078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.063+0000 7f5d29ffb640 1 -- 172.21.15.59:0/374414733 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f5d300984d0 con 0x7f5d340b2270 2024-06-29T14:42:07.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.063+0000 7f5d38d0e640 1 -- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5d18076440 msgr2=0x7f5d18078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3378196672 2024-06-29T14:42:07.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.063+0000 7f5d38d0e640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5d18076440 0x7f5d18078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:42:07.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.067+0000 7f5d29ffb640 1 -- 172.21.15.59:0/374414733 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d300612e0 con 0x7f5d340b2270 2024-06-29T14:42:07.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.260+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f5d34074500 con 0x7f5d340b2270 2024-06-29T14:42:07.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.264+0000 7f5d38d0e640 1 -- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5d18076440 msgr2=0x7f5d18078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3378196672 2024-06-29T14:42:07.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.264+0000 7f5d38d0e640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5d18076440 0x7f5d18078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:42:07.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.265+0000 7f5d29ffb640 1 -- 172.21.15.59:0/374414733 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v58)=0 v58) ==== 133+0+0 (secure 0 0 0) 0x7f5d300655e0 con 0x7f5d340b2270 2024-06-29T14:42:07.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5d18076440 msgr2=0x7f5d18078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5d18076440 0x7f5d18078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d340b2270 msgr2=0x7f5d34141e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d340b2270 0x7f5d34141e40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000e6d0 tx=0x7f5d3000eb90 comp rx=0 tx=0).stop 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 shutdown_connections 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f5d18076440 0x7f5d18078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d340d1a60 0x7f5d340a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d340b2fa0 0x7f5d34159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 --2- 172.21.15.59:0/374414733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d340b2270 0x7f5d34141e40 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 >> 172.21.15.59:0/374414733 conn(0x7f5d34073e10 msgr2=0x7f5d340d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 shutdown_connections 2024-06-29T14:42:07.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.267+0000 7f5d3b773640 1 -- 172.21.15.59:0/374414733 wait complete. 2024-06-29T14:42:07.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:07.277 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:42:07.298 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-06-29T14:42:07.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.462+0000 7f9f10864640 1 -- 172.21.15.59:0/761212239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f0c0d1c50 msgr2=0x7f9f0c0d2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.462+0000 7f9f10864640 1 --2- 172.21.15.59:0/761212239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f0c0d1c50 0x7f9f0c0d2060 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9f04005fd0 tx=0x7f9f0402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.463+0000 7f9f10864640 1 -- 172.21.15.59:0/761212239 shutdown_connections 2024-06-29T14:42:07.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.463+0000 7f9f10864640 1 --2- 172.21.15.59:0/761212239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f0c0d1c50 0x7f9f0c0d2060 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.463+0000 7f9f10864640 1 --2- 172.21.15.59:0/761212239 >> v2:172.21.15.153:3300/0 conn(0x7f9f0c0d12f0 0x7f9f0c0b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.463+0000 7f9f10864640 1 -- 172.21.15.59:0/761212239 >> 172.21.15.59:0/761212239 conn(0x7f9f0c073e80 msgr2=0x7f9f0c074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:07.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.463+0000 7f9f10864640 1 -- 172.21.15.59:0/761212239 shutdown_connections 2024-06-29T14:42:07.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.463+0000 7f9f10864640 1 -- 172.21.15.59:0/761212239 wait complete. 2024-06-29T14:42:07.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.465+0000 7f9f10864640 1 Processor -- start 2024-06-29T14:42:07.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.465+0000 7f9f10864640 1 -- start start 2024-06-29T14:42:07.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f10864640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c0d12f0 0x7f9f0c1f4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f10864640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f0c0d1c50 0x7f9f0c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f10864640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f0c145570 0x7f9f0c0addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f10864640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c0b64c0 con 0x7f9f0c0d1c50 2024-06-29T14:42:07.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f10864640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c0b6340 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f10864640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9f0c0b6640 con 0x7f9f0c145570 2024-06-29T14:42:07.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f09d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f0c0d1c50 0x7f9f0c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f09d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f0c0d1c50 0x7f9f0c0ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36692/0 (socket says 172.21.15.59:36692) 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f0a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c0d12f0 0x7f9f0c1f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f09d9b640 1 -- 172.21.15.59:0/3416275881 learned_addr learned my addr 172.21.15.59:0/3416275881 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f0ad9d640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f0c145570 0x7f9f0c0addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f0a59c640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f0c145570 msgr2=0x7f9f0c0addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f0a59c640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f0c145570 0x7f9f0c0addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f0a59c640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f0c0d1c50 msgr2=0x7f9f0c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f0a59c640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f0c0d1c50 0x7f9f0c0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f0a59c640 1 -- 172.21.15.59:0/3416275881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c1f4b80 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.466+0000 7f9f09d9b640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f0c0d1c50 0x7f9f0c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.467+0000 7f9f0a59c640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c0d12f0 0x7f9f0c1f4640 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c3c0 tx=0x7f9efc00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.467+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc011790 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.467+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9efc017070 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.467+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc011cc0 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.467+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0c0aff70 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.467+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0c0b0530 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.468+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f0c0b6ad0 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.469+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 79) ==== 98320+0+0 (secure 0 0 0) 0x7f9efc01c460 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.469+0000 7f9efaffd640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 0x7f9eec078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.469+0000 7f9f09d9b640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 msgr2=0x7f9eec078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3378196672 2024-06-29T14:42:07.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.469+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6129+0+0 (secure 0 0 0) 0x7f9efc097440 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.469+0000 7f9f09d9b640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 0x7f9eec078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:42:07.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.474+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9efc021020 con 0x7f9f0c0d12f0 2024-06-29T14:42:07.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.670+0000 7f9f09d9b640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 msgr2=0x7f9eec078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3378196672 2024-06-29T14:42:07.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.670+0000 7f9f09d9b640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 0x7f9eec078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:42:07.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:07.689+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 --> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9f0c142bd0 con 0x7f9eec076440 2024-06-29T14:42:08.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:08.071+0000 7f9f09d9b640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 msgr2=0x7f9eec078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3378196672 2024-06-29T14:42:08.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:08.071+0000 7f9f09d9b640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 0x7f9eec078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:42:08.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:08.872+0000 7f9f09d9b640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 msgr2=0x7f9eec078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3378196672 2024-06-29T14:42:08.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:08.872+0000 7f9f09d9b640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 0x7f9eec078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:42:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:10.238+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 80) ==== 97929+0+0 (secure 0 0 0) 0x7f9efc05c700 con 0x7f9f0c0d12f0 2024-06-29T14:42:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:10.238+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 msgr2=0x7f9eec078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:42:10.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:10.238+0000 7f9efaffd640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3378196672,v1:172.21.15.153:6833/3378196672] conn(0x7f9eec076440 0x7f9eec078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:10.392 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.394+0000 7f4284950640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.392 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.394+0000 7f4284950640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.392 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.394+0000 7f4284950640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.392 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.394+0000 7f4284950640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.392 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.394+0000 7f4284950640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.393 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.395+0000 7f428194a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.393 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.395+0000 7f428194a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.393 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.395+0000 7f428194a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.393 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.395+0000 7f428194a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.393 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:42:10.395+0000 7f428194a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:11.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.239+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mgrmap(e 81) ==== 98056+0+0 (secure 0 0 0) 0x7f9efc06ab30 con 0x7f9f0c0d12f0 2024-06-29T14:42:11.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.240+0000 7f9efaffd640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f9eec07e9f0 0x7f9eec080e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.240+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 --> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9f0c142bd0 con 0x7f9eec07e9f0 2024-06-29T14:42:11.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.244+0000 7f9f09d9b640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f9eec07e9f0 0x7f9eec080e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.247+0000 7f9f09d9b640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f9eec07e9f0 0x7f9eec080e10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c155e90 tx=0x7f9f00005f90 comp rx=0 tx=0).ready entity=mgr.8953 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.318+0000 7f9efaffd640 1 -- 172.21.15.59:0/3416275881 <== mgr.8953 v2:172.21.15.153:6832/3376352564 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9f0c142bd0 con 0x7f9eec07e9f0 2024-06-29T14:42:11.317 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1 2024-06-29T14:42:11.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f9eec07e9f0 msgr2=0x7f9eec080e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f10864640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f9eec07e9f0 0x7f9eec080e10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c155e90 tx=0x7f9f00005f90 comp rx=0 tx=0).stop 2024-06-29T14:42:11.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c0d12f0 msgr2=0x7f9f0c1f4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f10864640 1 --2- 172.21.15.59:0/3416275881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c0d12f0 0x7f9f0c1f4640 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c3c0 tx=0x7f9efc00c880 comp rx=0 tx=0).stop 2024-06-29T14:42:11.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f0ad9d640 1 -- 172.21.15.59:0/3416275881 reap_dead start 2024-06-29T14:42:11.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 shutdown_connections 2024-06-29T14:42:11.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 >> 172.21.15.59:0/3416275881 conn(0x7f9f0c073e80 msgr2=0x7f9f0c15eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:11.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 shutdown_connections 2024-06-29T14:42:11.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.321+0000 7f9f10864640 1 -- 172.21.15.59:0/3416275881 wait complete. 2024-06-29T14:42:11.332 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-06-29T14:42:11.332 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-06-29T14:42:11.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.460+0000 7f142df54640 1 -- 172.21.15.59:0/1215071540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280d2780 msgr2=0x7f14281621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.460+0000 7f142df54640 1 --2- 172.21.15.59:0/1215071540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280d2780 0x7f14281621f0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f14280b3bc0 tx=0x7f141c02a920 comp rx=0 tx=0).stop 2024-06-29T14:42:11.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.460+0000 7f142df54640 1 -- 172.21.15.59:0/1215071540 shutdown_connections 2024-06-29T14:42:11.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.460+0000 7f142df54640 1 --2- 172.21.15.59:0/1215071540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280d2780 0x7f14281621f0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.460+0000 7f142df54640 1 -- 172.21.15.59:0/1215071540 >> 172.21.15.59:0/1215071540 conn(0x7f14280d3e60 msgr2=0x7f14280d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:11.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.460+0000 7f142df54640 1 -- 172.21.15.59:0/1215071540 shutdown_connections 2024-06-29T14:42:11.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.460+0000 7f142df54640 1 -- 172.21.15.59:0/1215071540 wait complete. 2024-06-29T14:42:11.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.462+0000 7f142df54640 1 Processor -- start 2024-06-29T14:42:11.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.462+0000 7f142df54640 1 -- start start 2024-06-29T14:42:11.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f142df54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280b3ed0 0x7f14281f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f142df54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14280d2780 0x7f14280c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f142df54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142815dff0 0x7f142814c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f142df54640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f14280b4b70 con 0x7f14280b3ed0 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f142df54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14280b49f0 con 0x7f142815dff0 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f142df54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f14280b4cf0 con 0x7f14280d2780 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f14277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280b3ed0 0x7f14281f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f14277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280b3ed0 0x7f14281f54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36710/0 (socket says 172.21.15.59:36710) 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f14277fe640 1 -- 172.21.15.59:0/3065203595 learned_addr learned my addr 172.21.15.59:0/3065203595 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f1426ffd640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14280d2780 0x7f14280c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f1427fff640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142815dff0 0x7f142814c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f14277fe640 1 -- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14280d2780 msgr2=0x7f14280c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f14277fe640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14280d2780 0x7f14280c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f14277fe640 1 -- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142815dff0 msgr2=0x7f142814c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f14277fe640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142815dff0 0x7f142814c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f14277fe640 1 -- 172.21.15.59:0/3065203595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142815e640 con 0x7f14280b3ed0 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.464+0000 7f1427fff640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142815dff0 0x7f142814c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.465+0000 7f14277fe640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280b3ed0 0x7f14281f54b0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f141800c3a0 tx=0x7f141800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.465+0000 7f140ffff640 1 -- 172.21.15.59:0/3065203595 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418017070 con 0x7f14280b3ed0 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.465+0000 7f140ffff640 1 -- 172.21.15.59:0/3065203595 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14180053c0 con 0x7f14280b3ed0 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.465+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142815e900 con 0x7f14280b3ed0 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.465+0000 7f140ffff640 1 -- 172.21.15.59:0/3065203595 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418003850 con 0x7f14280b3ed0 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.465+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f142815eec0 con 0x7f14280b3ed0 2024-06-29T14:42:11.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.466+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14280b42e0 con 0x7f14280b3ed0 2024-06-29T14:42:11.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.467+0000 7f140ffff640 1 -- 172.21.15.59:0/3065203595 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 81) ==== 98056+0+0 (secure 0 0 0) 0x7f141801a070 con 0x7f14280b3ed0 2024-06-29T14:42:11.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.467+0000 7f140ffff640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f1414076170 0x7f1414078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.467+0000 7f140ffff640 1 -- 172.21.15.59:0/3065203595 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6473+0+0 (secure 0 0 0) 0x7f1418096b90 con 0x7f14280b3ed0 2024-06-29T14:42:11.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.467+0000 7f1426ffd640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f1414076170 0x7f1414078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.468+0000 7f1426ffd640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f1414076170 0x7f1414078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f141c034000 tx=0x7f141c004ab0 comp rx=0 tx=0).ready entity=mgr.8953 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.472+0000 7f140ffff640 1 -- 172.21.15.59:0/3065203595 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f14180604a0 con 0x7f14280b3ed0 2024-06-29T14:42:11.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.686+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 --> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) -- 0x7f142815f6c0 con 0x7f1414076170 2024-06-29T14:42:11.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.748+0000 7f140ffff640 1 -- 172.21.15.59:0/3065203595 <== mgr.8953 v2:172.21.15.153:6832/3376352564 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+40 (secure 0 0 0) 0x7f141c003af0 con 0x7f1414076170 2024-06-29T14:42:11.747 INFO:teuthology.orchestra.run.smithi059.stdout:Retention added to path /snap_test_dir1 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f1414076170 msgr2=0x7f1414078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f1414076170 0x7f1414078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f141c034000 tx=0x7f141c004ab0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280b3ed0 msgr2=0x7f14281f54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280b3ed0 0x7f14281f54b0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f141800c3a0 tx=0x7f141800c860 comp rx=0 tx=0).stop 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 shutdown_connections 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f1414076170 0x7f1414078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f142815dff0 0x7f142814c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14280d2780 0x7f14280c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 --2- 172.21.15.59:0/3065203595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14280b3ed0 0x7f14281f54b0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.750+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 >> 172.21.15.59:0/3065203595 conn(0x7f14280d3e60 msgr2=0x7f14280b8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:11.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.751+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 shutdown_connections 2024-06-29T14:42:11.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.751+0000 7f142df54640 1 -- 172.21.15.59:0/3065203595 wait complete. 2024-06-29T14:42:11.760 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-06-29T14:42:11.760 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-06-29T14:42:11.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-06-29T14:42:11.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.882+0000 7fed58f24640 1 -- 172.21.15.59:0/3255689583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed5415efe0 msgr2=0x7fed5415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.882+0000 7fed58f24640 1 --2- 172.21.15.59:0/3255689583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed5415efe0 0x7fed5415f9b0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fed540d15d0 tx=0x7fed4402aa30 comp rx=0 tx=0).stop 2024-06-29T14:42:11.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.882+0000 7fed58f24640 1 -- 172.21.15.59:0/3255689583 shutdown_connections 2024-06-29T14:42:11.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.882+0000 7fed58f24640 1 --2- 172.21.15.59:0/3255689583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed5415efe0 0x7fed5415f9b0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.882+0000 7fed58f24640 1 --2- 172.21.15.59:0/3255689583 >> v2:172.21.15.153:3300/0 conn(0x7fed540d23c0 0x7fed540b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.882+0000 7fed58f24640 1 -- 172.21.15.59:0/3255689583 >> 172.21.15.59:0/3255689583 conn(0x7fed540d3ee0 msgr2=0x7fed540d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:11.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.882+0000 7fed58f24640 1 -- 172.21.15.59:0/3255689583 shutdown_connections 2024-06-29T14:42:11.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.882+0000 7fed58f24640 1 -- 172.21.15.59:0/3255689583 wait complete. 2024-06-29T14:42:11.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.884+0000 7fed58f24640 1 Processor -- start 2024-06-29T14:42:11.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.884+0000 7fed58f24640 1 -- start start 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed58f24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed540b3ed0 0x7fed54148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed58f24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed540d23c0 0x7fed541598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed58f24640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5415efe0 0x7fed5415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed58f24640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fed54160280 con 0x7fed540d23c0 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed58f24640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed54160100 con 0x7fed540b3ed0 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed58f24640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fed54160400 con 0x7fed5415efe0 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed540d23c0 0x7fed541598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed540d23c0 0x7fed541598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36722/0 (socket says 172.21.15.59:36722) 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 -- 172.21.15.59:0/2759271246 learned_addr learned my addr 172.21.15.59:0/2759271246 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T14:42:11.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed5259c640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed540b3ed0 0x7fed54148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed52d9d640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5415efe0 0x7fed5415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 -- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5415efe0 msgr2=0x7fed5415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5415efe0 0x7fed5415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 -- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed540b3ed0 msgr2=0x7fed54148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed540b3ed0 0x7fed54148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 -- 172.21.15.59:0/2759271246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed54149460 con 0x7fed540d23c0 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.886+0000 7fed51d9b640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed540d23c0 0x7fed541598c0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00b3f0 tx=0x7fed3c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.887+0000 7fed4affd640 1 -- 172.21.15.59:0/2759271246 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c015020 con 0x7fed540d23c0 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.887+0000 7fed4affd640 1 -- 172.21.15.59:0/2759271246 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed3c011920 con 0x7fed540d23c0 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.887+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed54156890 con 0x7fed540d23c0 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.887+0000 7fed4affd640 1 -- 172.21.15.59:0/2759271246 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c011d70 con 0x7fed540d23c0 2024-06-29T14:42:11.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.887+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed5415a780 con 0x7fed540d23c0 2024-06-29T14:42:11.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.888+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed540b3990 con 0x7fed540d23c0 2024-06-29T14:42:11.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.889+0000 7fed4affd640 1 -- 172.21.15.59:0/2759271246 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 81) ==== 98056+0+0 (secure 0 0 0) 0x7fed3c018020 con 0x7fed540d23c0 2024-06-29T14:42:11.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.889+0000 7fed4affd640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fed34076170 0x7fed34078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.889+0000 7fed4affd640 1 -- 172.21.15.59:0/2759271246 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6473+0+0 (secure 0 0 0) 0x7fed3c097290 con 0x7fed540d23c0 2024-06-29T14:42:11.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.889+0000 7fed5259c640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fed34076170 0x7fed34078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.890+0000 7fed5259c640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fed34076170 0x7fed34078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed540d2e40 tx=0x7fed4c00b2e0 comp rx=0 tx=0).ready entity=mgr.8953 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:11.893+0000 7fed4affd640 1 -- 172.21.15.59:0/2759271246 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fed3c01f030 con 0x7fed540d23c0 2024-06-29T14:42:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.105+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 --> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fed541ea5a0 con 0x7fed34076170 2024-06-29T14:42:12.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.112+0000 7fed4affd640 1 -- 172.21.15.59:0/2759271246 <== mgr.8953 v2:172.21.15.153:6832/3376352564 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+74 (secure 0 0 0) 0x7fed4c015070 con 0x7fed34076170 2024-06-29T14:42:12.110 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T14:42:12.110 INFO:teuthology.orchestra.run.smithi059.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-06-29T14:42:12.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fed34076170 msgr2=0x7fed34078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:12.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fed34076170 0x7fed34078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed540d2e40 tx=0x7fed4c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed540d23c0 msgr2=0x7fed541598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:12.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed540d23c0 0x7fed541598c0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00b3f0 tx=0x7fed3c00b8b0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 shutdown_connections 2024-06-29T14:42:12.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fed34076170 0x7fed34078680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5415efe0 0x7fed5415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed540d23c0 0x7fed541598c0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 --2- 172.21.15.59:0/2759271246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed540b3ed0 0x7fed54148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 >> 172.21.15.59:0/2759271246 conn(0x7fed540d3ee0 msgr2=0x7fed5406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:12.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 shutdown_connections 2024-06-29T14:42:12.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T14:42:12.114+0000 7fed58f24640 1 -- 172.21.15.59:0/2759271246 wait complete. 2024-06-29T14:42:12.124 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-06-29T14:42:12.124 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-06-29T14:42:27.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:42:27.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:42:57.153 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:42:57.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:43:00.757 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:43:00.759+0000 7f427392e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:00.757 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:43:00.759+0000 7f427392e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:00.757 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:43:00.759+0000 7f427392e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:00.757 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:43:00.759+0000 7f427392e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:00.757 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T14:43:00.759+0000 7f427392e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:27.278 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:43:27.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:43:57.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:43:57.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:44:27.572 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:44:27.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:44:57.636 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:44:57.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:45:27.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:45:27.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:45:57.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:45:57.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:46:27.993 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:46:27.997 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:46:58.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:46:58.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:47:28.225 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:47:28.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:47:58.300 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:47:58.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:48:28.363 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:48:28.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:48:58.427 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:48:58.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:49:28.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:49:28.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:49:58.565 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:49:58.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:50:28.644 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:50:28.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:50:58.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:50:58.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:51:28.909 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:51:28.912 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:51:59.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:51:59.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:52:29.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:52:29.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:52:59.369 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:52:59.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:53:29.517 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:53:29.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:53:59.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:53:59.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:54:29.776 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:54:29.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:54:59.898 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:54:59.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:55:30.045 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:55:30.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:56:00.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:56:00.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:56:30.264 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:56:30.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:57:00.410 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:57:00.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:57:30.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:57:30.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:58:00.605 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:58:00.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:58:30.759 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:58:30.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:59:00.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:59:00.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:59:30.903 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:59:30.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:00:00.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:00:00.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:00:31.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:00:31.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:01:01.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:01:01.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:01:31.181 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:01:31.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:02:01.247 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:02:01.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:02:31.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:02:31.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:03:01.418 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:03:01.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:03:31.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:03:31.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:04:01.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:04:01.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:04:31.870 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:04:31.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:05:01.976 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:05:01.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:05:32.064 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:05:32.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:06:02.194 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:06:02.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:06:32.316 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:06:32.320 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:07:02.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:07:02.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:07:32.584 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:07:32.587 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:08:02.646 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:08:02.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:08:32.777 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:08:32.782 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:09:02.916 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:09:02.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:09:33.013 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:09:33.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:10:03.087 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:10:03.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:10:33.164 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:10:33.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:11:03.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:11:03.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:11:33.306 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:11:33.310 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:12:03.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:12:03.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:12:33.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:12:33.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:13:03.508 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:13:03.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:13:33.579 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:13:33.583 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:14:03.659 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:14:03.663 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:14:33.848 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:14:33.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:15:03.907 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:15:03.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:15:34.067 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:15:34.071 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:16:04.188 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:16:04.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:16:34.344 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:16:34.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:17:04.473 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:17:04.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:17:34.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:17:34.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:18:04.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:18:04.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:18:34.747 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:18:34.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:19:04.843 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:19:04.847 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:19:34.982 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:19:34.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:20:05.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:20:05.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:20:35.208 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:20:35.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:21:05.389 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:21:05.392 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:21:35.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:21:35.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:22:05.520 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:22:05.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:22:35.585 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:22:35.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:23:05.650 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:23:05.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:23:35.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:23:35.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:24:05.819 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:24:05.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:24:35.883 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:24:35.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:25:05.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:25:05.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:25:36.017 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:25:36.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:26:06.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:26:06.089 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:26:36.178 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:26:36.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:27:06.301 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:27:06.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:27:36.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:27:36.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:28:06.552 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:28:06.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:28:36.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:28:36.719 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:29:06.846 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:29:06.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:29:36.969 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:29:36.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:30:07.092 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:30:07.095 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:30:37.230 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:30:37.233 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:31:07.344 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:31:07.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:31:37.457 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:31:37.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:32:07.596 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:32:07.600 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:32:37.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:32:37.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:33:07.832 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:33:07.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:33:37.904 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:33:37.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:34:07.967 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:34:07.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:34:38.032 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:34:38.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:35:08.094 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:35:08.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:35:38.208 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:35:38.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:36:08.291 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:36:08.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:36:38.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:36:38.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:37:08.715 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:37:08.719 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:37:38.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:37:38.782 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:08.909 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:08.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:16.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:16.125 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:38:16.149 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_44_00_UTC 2024-06-29T15:38:16.149 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_45_00_UTC 2024-06-29T15:38:16.149 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_46_00_UTC 2024-06-29T15:38:16.149 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_47_00_UTC 2024-06-29T15:38:16.149 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_48_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_49_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_50_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_51_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_52_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_53_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_54_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_55_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_56_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_57_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_58_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_59_00_UTC 2024-06-29T15:38:16.150 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_00_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_01_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_02_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_03_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_04_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_05_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_06_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_07_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_08_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_09_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_10_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_11_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_12_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_13_00_UTC 2024-06-29T15:38:16.151 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_14_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_15_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_16_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_17_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_18_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_19_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_20_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_21_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_22_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_23_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_24_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_25_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_26_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_27_00_UTC 2024-06-29T15:38:16.152 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_28_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_29_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_30_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_31_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_32_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_33_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_34_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_35_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_36_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_37_00_UTC 2024-06-29T15:38:16.153 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_38_00_UTC 2024-06-29T15:38:16.153 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_44_00_UTC', 'scheduled-2024-06-29-14_45_00_UTC', 'scheduled-2024-06-29-14_46_00_UTC', 'scheduled-2024-06-29-14_47_00_UTC', 'scheduled-2024-06-29-14_48_00_UTC', 'scheduled-2024-06-29-14_49_00_UTC', 'scheduled-2024-06-29-14_50_00_UTC', 'scheduled-2024-06-29-14_51_00_UTC', 'scheduled-2024-06-29-14_52_00_UTC', 'scheduled-2024-06-29-14_53_00_UTC', 'scheduled-2024-06-29-14_54_00_UTC', 'scheduled-2024-06-29-14_55_00_UTC', 'scheduled-2024-06-29-14_56_00_UTC', 'scheduled-2024-06-29-14_57_00_UTC', 'scheduled-2024-06-29-14_58_00_UTC', 'scheduled-2024-06-29-14_59_00_UTC', 'scheduled-2024-06-29-15_00_00_UTC', 'scheduled-2024-06-29-15_01_00_UTC', 'scheduled-2024-06-29-15_02_00_UTC', 'scheduled-2024-06-29-15_03_00_UTC', 'scheduled-2024-06-29-15_04_00_UTC', 'scheduled-2024-06-29-15_05_00_UTC', 'scheduled-2024-06-29-15_06_00_UTC', 'scheduled-2024-06-29-15_07_00_UTC', 'scheduled-2024-06-29-15_08_00_UTC', 'scheduled-2024-06-29-15_09_00_UTC', 'scheduled-2024-06-29-15_10_00_UTC', 'scheduled-2024-06-29-15_11_00_UTC', 'scheduled-2024-06-29-15_12_00_UTC', 'scheduled-2024-06-29-15_13_00_UTC', 'scheduled-2024-06-29-15_14_00_UTC', 'scheduled-2024-06-29-15_15_00_UTC', 'scheduled-2024-06-29-15_16_00_UTC', 'scheduled-2024-06-29-15_17_00_UTC', 'scheduled-2024-06-29-15_18_00_UTC', 'scheduled-2024-06-29-15_19_00_UTC', 'scheduled-2024-06-29-15_20_00_UTC', 'scheduled-2024-06-29-15_21_00_UTC', 'scheduled-2024-06-29-15_22_00_UTC', 'scheduled-2024-06-29-15_23_00_UTC', 'scheduled-2024-06-29-15_24_00_UTC', 'scheduled-2024-06-29-15_25_00_UTC', 'scheduled-2024-06-29-15_26_00_UTC', 'scheduled-2024-06-29-15_27_00_UTC', 'scheduled-2024-06-29-15_28_00_UTC', 'scheduled-2024-06-29-15_29_00_UTC', 'scheduled-2024-06-29-15_30_00_UTC', 'scheduled-2024-06-29-15_31_00_UTC', 'scheduled-2024-06-29-15_32_00_UTC', 'scheduled-2024-06-29-15_33_00_UTC', 'scheduled-2024-06-29-15_34_00_UTC', 'scheduled-2024-06-29-15_35_00_UTC', 'scheduled-2024-06-29-15_36_00_UTC', 'scheduled-2024-06-29-15_37_00_UTC', 'scheduled-2024-06-29-15_38_00_UTC'] 2024-06-29T15:38:16.154 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-06-29T15:38:16.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.323+0000 7fd99f1a1640 1 -- 172.21.15.59:0/1707369973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd980003ac0 msgr2=0x7fd980003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.323+0000 7fd99f1a1640 1 --2- 172.21.15.59:0/1707369973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd980003ac0 0x7fd980003f30 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fd988001dc0 tx=0x7fd980005830 comp rx=0 tx=0).stop 2024-06-29T15:38:16.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.323+0000 7fd99f1a1640 1 -- 172.21.15.59:0/1707369973 shutdown_connections 2024-06-29T15:38:16.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.323+0000 7fd99f1a1640 1 --2- 172.21.15.59:0/1707369973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd980003ac0 0x7fd980003f30 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.323+0000 7fd99f1a1640 1 -- 172.21.15.59:0/1707369973 >> 172.21.15.59:0/1707369973 conn(0x7fd9980b49f0 msgr2=0x7fd9980b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.323+0000 7fd99f1a1640 1 -- 172.21.15.59:0/1707369973 shutdown_connections 2024-06-29T15:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.323+0000 7fd99f1a1640 1 -- 172.21.15.59:0/1707369973 wait complete. 2024-06-29T15:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.325+0000 7fd99f1a1640 1 Processor -- start 2024-06-29T15:38:16.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.325+0000 7fd99f1a1640 1 -- start start 2024-06-29T15:38:16.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99f1a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980003ac0 0x7fd9981f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99f1a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9981f4160 0x7fd998140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99f1a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9981f4570 0x7fd998141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99f1a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd9980c7370 con 0x7fd9981f4570 2024-06-29T15:38:16.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99f1a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd99815c6c0 con 0x7fd980003ac0 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99f1a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd9980c74f0 con 0x7fd9981f4160 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e9a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9981f4570 0x7fd998141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e9a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9981f4570 0x7fd998141120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36060/0 (socket says 172.21.15.59:36060) 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e9a0640 1 -- 172.21.15.59:0/2765903571 learned_addr learned my addr 172.21.15.59:0/2765903571 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99d99e640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9981f4160 0x7fd998140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e19f640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980003ac0 0x7fd9981f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e9a0640 1 -- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9981f4160 msgr2=0x7fd998140be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e9a0640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9981f4160 0x7fd998140be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e9a0640 1 -- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980003ac0 msgr2=0x7fd9981f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e9a0640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980003ac0 0x7fd9981f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.326+0000 7fd99e9a0640 1 -- 172.21.15.59:0/2765903571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9981f4cd0 con 0x7fd9981f4570 2024-06-29T15:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.327+0000 7fd99e9a0640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9981f4570 0x7fd998141120 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fd9981568b0 tx=0x7fd990011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.327+0000 7fd996ffd640 1 -- 172.21.15.59:0/2765903571 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990004240 con 0x7fd9981f4570 2024-06-29T15:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.327+0000 7fd996ffd640 1 -- 172.21.15.59:0/2765903571 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd990016070 con 0x7fd9981f4570 2024-06-29T15:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.327+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9981f4fb0 con 0x7fd9981f4570 2024-06-29T15:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.327+0000 7fd996ffd640 1 -- 172.21.15.59:0/2765903571 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99000faf0 con 0x7fd9981f4570 2024-06-29T15:38:16.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.327+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9981f5570 con 0x7fd9981f4570 2024-06-29T15:38:16.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.328+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd998069db0 con 0x7fd9981f4570 2024-06-29T15:38:16.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.329+0000 7fd996ffd640 1 -- 172.21.15.59:0/2765903571 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 85) ==== 98332+0+0 (secure 0 0 0) 0x7fd99000fc90 con 0x7fd9981f4570 2024-06-29T15:38:16.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.330+0000 7fd996ffd640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fd984076440 0x7fd984078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.330+0000 7fd996ffd640 1 -- 172.21.15.59:0/2765903571 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 6517+0+0 (secure 0 0 0) 0x7fd990098620 con 0x7fd9981f4570 2024-06-29T15:38:16.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.330+0000 7fd99e19f640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fd984076440 0x7fd984078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.330+0000 7fd99e19f640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fd984076440 0x7fd984078950 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd988032000 tx=0x7fd988001510 comp rx=0 tx=0).ready entity=mgr.8953 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:16.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.334+0000 7fd996ffd640 1 -- 172.21.15.59:0/2765903571 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd998069db0 con 0x7fd9981f4570 2024-06-29T15:38:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.549+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 --> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9980c6b90 con 0x7fd984076440 2024-06-29T15:38:16.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.559+0000 7fd996ffd640 1 -- 172.21.15.59:0/2765903571 <== mgr.8953 v2:172.21.15.153:6832/3376352564 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+373 (secure 0 0 0) 0x7fd9980c6b90 con 0x7fd984076440 2024-06-29T15:38:16.559 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:16.559 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:42:11", "first": "2024-06-29T14:43:00", "last": "2024-06-29T15:38:00", "last_pruned": "2024-06-29T15:38:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fd984076440 msgr2=0x7fd984078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fd984076440 0x7fd984078950 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd988032000 tx=0x7fd988001510 comp rx=0 tx=0).stop 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9981f4570 msgr2=0x7fd998141120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9981f4570 0x7fd998141120 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fd9981568b0 tx=0x7fd990011b30 comp rx=0 tx=0).stop 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 shutdown_connections 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fd984076440 0x7fd984078950 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9981f4570 0x7fd998141120 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9981f4160 0x7fd998140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 --2- 172.21.15.59:0/2765903571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd980003ac0 0x7fd9981f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 >> 172.21.15.59:0/2765903571 conn(0x7fd9980b49f0 msgr2=0x7fd998159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:16.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 shutdown_connections 2024-06-29T15:38:16.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.561+0000 7fd99f1a1640 1 -- 172.21.15.59:0/2765903571 wait complete. 2024-06-29T15:38:16.572 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:42:11", "first": "2024-06-29T14:43:00", "last": "2024-06-29T15:38:00", "last_pruned": "2024-06-29T15:38:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-06-29T15:38:16.572 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:42:11', 'first': '2024-06-29T14:43:00', 'last': '2024-06-29T15:38:00', 'last_pruned': '2024-06-29T15:38:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-06-29T15:38:16.572 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-06-29T15:38:16.572 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-06-29T15:38:16.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-06-29T15:38:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.692+0000 7ff6e64b3640 1 -- 172.21.15.59:0/4143454874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e0072720 msgr2=0x7ff6e00cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.692+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/4143454874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e0072720 0x7ff6e00cd9b0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff6e006f810 tx=0x7ff6c802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.692+0000 7ff6e64b3640 1 -- 172.21.15.59:0/4143454874 shutdown_connections 2024-06-29T15:38:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.692+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/4143454874 >> v2:172.21.15.153:3300/0 conn(0x7ff6e0160380 0x7ff6e006c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.692+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/4143454874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e0072720 0x7ff6e00cd9b0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.692+0000 7ff6e64b3640 1 -- 172.21.15.59:0/4143454874 >> 172.21.15.59:0/4143454874 conn(0x7ff6e00b24c0 msgr2=0x7ff6e00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.692+0000 7ff6e64b3640 1 -- 172.21.15.59:0/4143454874 shutdown_connections 2024-06-29T15:38:16.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.692+0000 7ff6e64b3640 1 -- 172.21.15.59:0/4143454874 wait complete. 2024-06-29T15:38:16.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.694+0000 7ff6e64b3640 1 Processor -- start 2024-06-29T15:38:16.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.694+0000 7ff6e64b3640 1 -- start start 2024-06-29T15:38:16.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e64b3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e0145710 0x7ff6e01f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e64b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e0160380 0x7ff6e0138420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e64b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e015b7d0 0x7ff6e0139f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e64b3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e006cab0 con 0x7ff6e015b7d0 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e64b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e006c930 con 0x7ff6e0160380 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e64b3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e006cc30 con 0x7ff6e0145710 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e4a50640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e015b7d0 0x7ff6e0139f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e4a50640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e015b7d0 0x7ff6e0139f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36076/0 (socket says 172.21.15.59:36076) 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.695+0000 7ff6e4a50640 1 -- 172.21.15.59:0/1675281094 learned_addr learned my addr 172.21.15.59:0/1675281094 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6d7fff640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e0160380 0x7ff6e0138420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6dffff640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e0145710 0x7ff6e01f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6e4a50640 1 -- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e0145710 msgr2=0x7ff6e01f2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6e4a50640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e0145710 0x7ff6e01f2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6e4a50640 1 -- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e0160380 msgr2=0x7ff6e0138420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6e4a50640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e0160380 0x7ff6e0138420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6e4a50640 1 -- 172.21.15.59:0/1675281094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e013bfb0 con 0x7ff6e015b7d0 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6dffff640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e0145710 0x7ff6e01f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6e4a50640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e015b7d0 0x7ff6e0139f50 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800b970 tx=0x7ff6d800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6dd7fa640 1 -- 172.21.15.59:0/1675281094 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8019070 con 0x7ff6e015b7d0 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6dd7fa640 1 -- 172.21.15.59:0/1675281094 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6d800cc50 con 0x7ff6e015b7d0 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e0138eb0 con 0x7ff6e015b7d0 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6dd7fa640 1 -- 172.21.15.59:0/1675281094 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d80146a0 con 0x7ff6e015b7d0 2024-06-29T15:38:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.696+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e0139470 con 0x7ff6e015b7d0 2024-06-29T15:38:16.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.697+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e006d1c0 con 0x7ff6e015b7d0 2024-06-29T15:38:16.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.698+0000 7ff6dd7fa640 1 -- 172.21.15.59:0/1675281094 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 85) ==== 98332+0+0 (secure 0 0 0) 0x7ff6d801c070 con 0x7ff6e015b7d0 2024-06-29T15:38:16.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.699+0000 7ff6dd7fa640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7ff6c0076440 0x7ff6c0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.699+0000 7ff6dd7fa640 1 -- 172.21.15.59:0/1675281094 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 6517+0+0 (secure 0 0 0) 0x7ff6d8097ef0 con 0x7ff6e015b7d0 2024-06-29T15:38:16.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.699+0000 7ff6dffff640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7ff6c0076440 0x7ff6c0078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.699+0000 7ff6dffff640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7ff6c0076440 0x7ff6c0078950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0007fd0 tx=0x7ff6d0009040 comp rx=0 tx=0).ready entity=mgr.8953 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:16.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.702+0000 7ff6dd7fa640 1 -- 172.21.15.59:0/1675281094 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff6d8023020 con 0x7ff6e015b7d0 2024-06-29T15:38:16.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:16.916+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 --> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7ff6e0138c60 con 0x7ff6c0076440 2024-06-29T15:38:17.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.003+0000 7ff6dd7fa640 1 -- 172.21.15.59:0/1675281094 <== mgr.8953 v2:172.21.15.153:6832/3376352564 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+42 (secure 0 0 0) 0x7ff6e0138c60 con 0x7ff6c0076440 2024-06-29T15:38:17.004 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1 2024-06-29T15:38:17.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7ff6c0076440 msgr2=0x7ff6c0078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:17.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7ff6c0076440 0x7ff6c0078950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0007fd0 tx=0x7ff6d0009040 comp rx=0 tx=0).stop 2024-06-29T15:38:17.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e015b7d0 msgr2=0x7ff6e0139f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:17.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e015b7d0 0x7ff6e0139f50 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800b970 tx=0x7ff6d800be30 comp rx=0 tx=0).stop 2024-06-29T15:38:17.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 shutdown_connections 2024-06-29T15:38:17.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7ff6c0076440 0x7ff6c0078950 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:17.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6e015b7d0 0x7ff6e0139f50 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:17.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6e0160380 0x7ff6e0138420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:17.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 --2- 172.21.15.59:0/1675281094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6e0145710 0x7ff6e01f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:17.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 >> 172.21.15.59:0/1675281094 conn(0x7ff6e00b24c0 msgr2=0x7ff6e00afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:17.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 shutdown_connections 2024-06-29T15:38:17.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:17.006+0000 7ff6e64b3640 1 -- 172.21.15.59:0/1675281094 wait complete. 2024-06-29T15:38:17.017 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-06-29T15:38:17.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.017 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:38:17.035 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_44_00_UTC 2024-06-29T15:38:17.035 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_45_00_UTC 2024-06-29T15:38:17.035 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_46_00_UTC 2024-06-29T15:38:17.035 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_47_00_UTC 2024-06-29T15:38:17.035 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_48_00_UTC 2024-06-29T15:38:17.035 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_49_00_UTC 2024-06-29T15:38:17.035 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_50_00_UTC 2024-06-29T15:38:17.035 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_51_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_52_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_53_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_54_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_55_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_56_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_57_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_58_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-14_59_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_00_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_01_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_02_00_UTC 2024-06-29T15:38:17.036 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_03_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_04_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_05_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_06_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_07_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_08_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_09_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_10_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_11_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_12_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_13_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_14_00_UTC 2024-06-29T15:38:17.037 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_15_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_16_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_17_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_18_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_19_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_20_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_21_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_22_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_23_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_24_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_25_00_UTC 2024-06-29T15:38:17.038 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_26_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_27_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_28_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_29_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_30_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_31_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_32_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_33_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_34_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_35_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_36_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_37_00_UTC 2024-06-29T15:38:17.039 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_38_00_UTC 2024-06-29T15:38:17.040 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_44_00_UTC 2024-06-29T15:38:17.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.040 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_44_00_UTC) 2024-06-29T15:38:17.097 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_45_00_UTC 2024-06-29T15:38:17.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.097 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_45_00_UTC) 2024-06-29T15:38:17.159 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_46_00_UTC 2024-06-29T15:38:17.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.159 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_46_00_UTC) 2024-06-29T15:38:17.220 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_47_00_UTC 2024-06-29T15:38:17.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.220 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_47_00_UTC) 2024-06-29T15:38:17.283 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_48_00_UTC 2024-06-29T15:38:17.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.283 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_48_00_UTC) 2024-06-29T15:38:17.345 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_49_00_UTC 2024-06-29T15:38:17.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.346 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_49_00_UTC) 2024-06-29T15:38:17.408 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_50_00_UTC 2024-06-29T15:38:17.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.408 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_50_00_UTC) 2024-06-29T15:38:17.470 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_51_00_UTC 2024-06-29T15:38:17.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.470 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_51_00_UTC) 2024-06-29T15:38:17.534 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_52_00_UTC 2024-06-29T15:38:17.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.534 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_52_00_UTC) 2024-06-29T15:38:17.596 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_53_00_UTC 2024-06-29T15:38:17.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.596 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_53_00_UTC) 2024-06-29T15:38:17.658 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_54_00_UTC 2024-06-29T15:38:17.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.658 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_54_00_UTC) 2024-06-29T15:38:17.720 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_55_00_UTC 2024-06-29T15:38:17.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.721 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_55_00_UTC) 2024-06-29T15:38:17.783 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_56_00_UTC 2024-06-29T15:38:17.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.783 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_56_00_UTC) 2024-06-29T15:38:17.845 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_57_00_UTC 2024-06-29T15:38:17.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.845 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_57_00_UTC) 2024-06-29T15:38:17.906 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_58_00_UTC 2024-06-29T15:38:17.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.906 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_58_00_UTC) 2024-06-29T15:38:17.969 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_59_00_UTC 2024-06-29T15:38:17.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.969 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_59_00_UTC) 2024-06-29T15:38:18.031 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_00_00_UTC 2024-06-29T15:38:18.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.031 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_00_00_UTC) 2024-06-29T15:38:18.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_01_00_UTC 2024-06-29T15:38:18.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.094 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_01_00_UTC) 2024-06-29T15:38:18.156 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_02_00_UTC 2024-06-29T15:38:18.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.156 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_02_00_UTC) 2024-06-29T15:38:18.218 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_03_00_UTC 2024-06-29T15:38:18.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.218 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_03_00_UTC) 2024-06-29T15:38:18.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_04_00_UTC 2024-06-29T15:38:18.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.281 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_04_00_UTC) 2024-06-29T15:38:18.343 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_05_00_UTC 2024-06-29T15:38:18.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.344 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_05_00_UTC) 2024-06-29T15:38:18.406 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_06_00_UTC 2024-06-29T15:38:18.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.407 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_06_00_UTC) 2024-06-29T15:38:18.469 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_07_00_UTC 2024-06-29T15:38:18.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.470 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_07_00_UTC) 2024-06-29T15:38:18.532 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_08_00_UTC 2024-06-29T15:38:18.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.533 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_08_00_UTC) 2024-06-29T15:38:18.595 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_09_00_UTC 2024-06-29T15:38:18.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.596 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_09_00_UTC) 2024-06-29T15:38:18.658 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_10_00_UTC 2024-06-29T15:38:18.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.659 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_10_00_UTC) 2024-06-29T15:38:18.721 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_11_00_UTC 2024-06-29T15:38:18.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.721 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_11_00_UTC) 2024-06-29T15:38:18.783 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_12_00_UTC 2024-06-29T15:38:18.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.783 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_12_00_UTC) 2024-06-29T15:38:18.845 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_13_00_UTC 2024-06-29T15:38:18.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.845 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_13_00_UTC) 2024-06-29T15:38:18.907 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_14_00_UTC 2024-06-29T15:38:18.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.907 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_14_00_UTC) 2024-06-29T15:38:18.969 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_15_00_UTC 2024-06-29T15:38:18.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.970 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_15_00_UTC) 2024-06-29T15:38:19.032 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_16_00_UTC 2024-06-29T15:38:19.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.032 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_16_00_UTC) 2024-06-29T15:38:19.136 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_17_00_UTC 2024-06-29T15:38:19.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.136 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_17_00_UTC) 2024-06-29T15:38:19.205 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_18_00_UTC 2024-06-29T15:38:19.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.206 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_18_00_UTC) 2024-06-29T15:38:19.255 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_19_00_UTC 2024-06-29T15:38:19.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.255 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_19_00_UTC) 2024-06-29T15:38:19.295 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_20_00_UTC 2024-06-29T15:38:19.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.295 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_20_00_UTC) 2024-06-29T15:38:19.356 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_21_00_UTC 2024-06-29T15:38:19.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.357 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_21_00_UTC) 2024-06-29T15:38:19.418 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_22_00_UTC 2024-06-29T15:38:19.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.418 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_22_00_UTC) 2024-06-29T15:38:19.479 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_23_00_UTC 2024-06-29T15:38:19.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.479 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_23_00_UTC) 2024-06-29T15:38:19.541 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_24_00_UTC 2024-06-29T15:38:19.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.541 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_24_00_UTC) 2024-06-29T15:38:19.602 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_25_00_UTC 2024-06-29T15:38:19.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.602 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_25_00_UTC) 2024-06-29T15:38:19.664 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_26_00_UTC 2024-06-29T15:38:19.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.664 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_26_00_UTC) 2024-06-29T15:38:19.726 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_27_00_UTC 2024-06-29T15:38:19.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.727 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_27_00_UTC) 2024-06-29T15:38:19.788 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_28_00_UTC 2024-06-29T15:38:19.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.789 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_28_00_UTC) 2024-06-29T15:38:19.851 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_29_00_UTC 2024-06-29T15:38:19.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.851 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_29_00_UTC) 2024-06-29T15:38:19.912 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_30_00_UTC 2024-06-29T15:38:19.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.913 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_30_00_UTC) 2024-06-29T15:38:19.974 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_31_00_UTC 2024-06-29T15:38:19.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.975 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_31_00_UTC) 2024-06-29T15:38:20.036 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_32_00_UTC 2024-06-29T15:38:20.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.036 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_32_00_UTC) 2024-06-29T15:38:20.098 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_33_00_UTC 2024-06-29T15:38:20.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.099 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_33_00_UTC) 2024-06-29T15:38:20.161 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_34_00_UTC 2024-06-29T15:38:20.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.162 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_34_00_UTC) 2024-06-29T15:38:20.223 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_35_00_UTC 2024-06-29T15:38:20.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.224 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_35_00_UTC) 2024-06-29T15:38:20.285 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_36_00_UTC 2024-06-29T15:38:20.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.286 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_36_00_UTC) 2024-06-29T15:38:20.348 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_37_00_UTC 2024-06-29T15:38:20.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.348 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_37_00_UTC) 2024-06-29T15:38:20.409 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_38_00_UTC 2024-06-29T15:38:20.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.409 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_38_00_UTC) 2024-06-29T15:38:20.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.470 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T15:38:20.530 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:38:20.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.690+0000 7f01c6d7a640 1 -- 172.21.15.59:0/285159584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0053a0 msgr2=0x7f01ac00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:20.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.690+0000 7f01c6d7a640 1 --2- 172.21.15.59:0/285159584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0053a0 0x7f01ac00ba20 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009b20 tx=0x7f01ac005830 comp rx=0 tx=0).stop 2024-06-29T15:38:20.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.690+0000 7f01c6d7a640 1 -- 172.21.15.59:0/285159584 shutdown_connections 2024-06-29T15:38:20.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.690+0000 7f01c6d7a640 1 --2- 172.21.15.59:0/285159584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0053a0 0x7f01ac00ba20 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:20.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.690+0000 7f01c6d7a640 1 -- 172.21.15.59:0/285159584 >> 172.21.15.59:0/285159584 conn(0x7f01c00b6fb0 msgr2=0x7f01c00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:20.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.690+0000 7f01c6d7a640 1 -- 172.21.15.59:0/285159584 shutdown_connections 2024-06-29T15:38:20.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.690+0000 7f01c6d7a640 1 -- 172.21.15.59:0/285159584 wait complete. 2024-06-29T15:38:20.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.692+0000 7f01c6d7a640 1 Processor -- start 2024-06-29T15:38:20.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.692+0000 7f01c6d7a640 1 -- start start 2024-06-29T15:38:20.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01c6d7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ac0053a0 0x7f01c0155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:20.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01c6d7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01c0133970 0x7f01c0152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01c6d7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01c015e570 0x7f01c015e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01c6d7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f01c00b5850 con 0x7f01c0133970 2024-06-29T15:38:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01c6d7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01c00b56d0 con 0x7f01ac0053a0 2024-06-29T15:38:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01c6d7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f01c00b59d0 con 0x7f01c015e570 2024-06-29T15:38:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01c0133970 0x7f01c0152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01c0133970 0x7f01c0152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36092/0 (socket says 172.21.15.59:36092) 2024-06-29T15:38:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01bffff640 1 -- 172.21.15.59:0/1121958102 learned_addr learned my addr 172.21.15.59:0/1121958102 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01c5317640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01c015e570 0x7f01c015e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.693+0000 7f01c4b16640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ac0053a0 0x7f01c0155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bffff640 1 -- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01c015e570 msgr2=0x7f01c015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bffff640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01c015e570 0x7f01c015e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bffff640 1 -- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ac0053a0 msgr2=0x7f01c0155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bffff640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ac0053a0 0x7f01c0155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bffff640 1 -- 172.21.15.59:0/1121958102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c015eec0 con 0x7f01c0133970 2024-06-29T15:38:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bffff640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01c0133970 0x7f01c0152700 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f01b00018c0 tx=0x7f01b0001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bd7fa640 1 -- 172.21.15.59:0/1121958102 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0015070 con 0x7f01c0133970 2024-06-29T15:38:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c01530c0 con 0x7f01c0133970 2024-06-29T15:38:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bd7fa640 1 -- 172.21.15.59:0/1121958102 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01b000eb50 con 0x7f01c0133970 2024-06-29T15:38:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01bd7fa640 1 -- 172.21.15.59:0/1121958102 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b00020c0 con 0x7f01c0133970 2024-06-29T15:38:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.694+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c0153680 con 0x7f01c0133970 2024-06-29T15:38:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.695+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c01415a0 con 0x7f01c0133970 2024-06-29T15:38:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.696+0000 7f01bd7fa640 1 -- 172.21.15.59:0/1121958102 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 85) ==== 98332+0+0 (secure 0 0 0) 0x7f01b0018070 con 0x7f01c0133970 2024-06-29T15:38:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.696+0000 7f01bd7fa640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f01a0076440 0x7f01a0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.696+0000 7f01bd7fa640 1 -- 172.21.15.59:0/1121958102 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 6561+0+0 (secure 0 0 0) 0x7f01b00985a0 con 0x7f01c0133970 2024-06-29T15:38:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.696+0000 7f01c4b16640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f01a0076440 0x7f01a0078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.697+0000 7f01c4b16640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f01a0076440 0x7f01a0078950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f01a8006b90 tx=0x7f01a80067a0 comp rx=0 tx=0).ready entity=mgr.8953 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:20.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.701+0000 7f01bd7fa640 1 -- 172.21.15.59:0/1121958102 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f01c015eec0 con 0x7f01c0133970 2024-06-29T15:38:20.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:20.941+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f01c015a6e0 con 0x7f01c0133970 2024-06-29T15:38:21.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.046+0000 7f01bd7fa640 1 -- 172.21.15.59:0/1121958102 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v86) ==== 93+0+0 (secure 0 0 0) 0x7f01b0061900 con 0x7f01c0133970 2024-06-29T15:38:21.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f01a0076440 msgr2=0x7f01a0078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f01a0076440 0x7f01a0078950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f01a8006b90 tx=0x7f01a80067a0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01c0133970 msgr2=0x7f01c0152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01c0133970 0x7f01c0152700 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f01b00018c0 tx=0x7f01b0001d80 comp rx=0 tx=0).stop 2024-06-29T15:38:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 shutdown_connections 2024-06-29T15:38:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f01a0076440 0x7f01a0078950 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01c015e570 0x7f01c015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01c0133970 0x7f01c0152700 secure :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f01b00018c0 tx=0x7f01b0001d80 comp rx=0 tx=0).stop 2024-06-29T15:38:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 --2- 172.21.15.59:0/1121958102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01ac0053a0 0x7f01c0155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.051+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 >> 172.21.15.59:0/1121958102 conn(0x7f01c00b6fb0 msgr2=0x7f01c00b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.053+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 shutdown_connections 2024-06-29T15:38:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.054+0000 7f01c6d7a640 1 -- 172.21.15.59:0/1121958102 wait complete. 2024-06-29T15:38:21.073 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:38:21.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:21.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:21.186 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:38:21.186 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:21.186 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:21.187 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:21.210 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:21.210 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:21.280 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 -- 172.21.15.59:0/2315748655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 msgr2=0x7fcd7c15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 --2- 172.21.15.59:0/2315748655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 0x7fcd7c15f9b0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0d15d0 tx=0x7fcd7402aa30 comp rx=0 tx=0).stop 2024-06-29T15:38:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 -- 172.21.15.59:0/2315748655 shutdown_connections 2024-06-29T15:38:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 --2- 172.21.15.59:0/2315748655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 0x7fcd7c15f9b0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 --2- 172.21.15.59:0/2315748655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0b3ed0 0x7fcd7c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 --2- 172.21.15.59:0/2315748655 >> v2:172.21.15.153:3300/0 conn(0x7fcd7c0d23c0 0x7fcd7c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 -- 172.21.15.59:0/2315748655 >> 172.21.15.59:0/2315748655 conn(0x7fcd7c0d3ee0 msgr2=0x7fcd7c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:21.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 -- 172.21.15.59:0/2315748655 shutdown_connections 2024-06-29T15:38:21.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.440+0000 7fcd823ac640 1 -- 172.21.15.59:0/2315748655 wait complete. 2024-06-29T15:38:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.443+0000 7fcd823ac640 1 Processor -- start 2024-06-29T15:38:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.444+0000 7fcd823ac640 1 -- start start 2024-06-29T15:38:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd823ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0b3ed0 0x7fcd7c15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd823ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c0d23c0 0x7fcd7c155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd823ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 0x7fcd7c155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd823ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c160070 con 0x7fcd7c15ee10 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd823ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c15fef0 con 0x7fcd7c0d23c0 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd823ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7c1601f0 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd80949640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 0x7fcd7c155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd7bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0b3ed0 0x7fcd7c15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd80949640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 0x7fcd7c155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36140/0 (socket says 172.21.15.59:36140) 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd80949640 1 -- 172.21.15.59:0/584827017 learned_addr learned my addr 172.21.15.59:0/584827017 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.446+0000 7fcd7b7fe640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c0d23c0 0x7fcd7c155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd7bfff640 1 -- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c0d23c0 msgr2=0x7fcd7c155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd7bfff640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c0d23c0 0x7fcd7c155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd7bfff640 1 -- 172.21.15.59:0/584827017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 msgr2=0x7fcd7c155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd7bfff640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 0x7fcd7c155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd7bfff640 1 -- 172.21.15.59:0/584827017 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c15ab10 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd7bfff640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0b3ed0 0x7fcd7c15ddc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00e6a0 tx=0x7fcd6c00eb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd78ff9640 1 -- 172.21.15.59:0/584827017 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c004220 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd78ff9640 1 -- 172.21.15.59:0/584827017 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd6c0043c0 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd78ff9640 1 -- 172.21.15.59:0/584827017 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c00cb10 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7c15adf0 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.447+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd7c15b3b0 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.448+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd7c0b3990 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.449+0000 7fcd78ff9640 1 -- 172.21.15.59:0/584827017 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fcd6c00ccb0 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.450+0000 7fcd78ff9640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fcd680763f0 0x7fcd68078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.450+0000 7fcd7b7fe640 1 -- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fcd680763f0 msgr2=0x7fcd68078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.450+0000 7fcd78ff9640 1 -- 172.21.15.59:0/584827017 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(162..162 src has 1..162) ==== 6561+0+0 (secure 0 0 0) 0x7fcd6c0984d0 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.450+0000 7fcd7b7fe640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fcd680763f0 0x7fcd68078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.454+0000 7fcd78ff9640 1 -- 172.21.15.59:0/584827017 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcd6c061130 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.651+0000 7fcd7b7fe640 1 -- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fcd680763f0 msgr2=0x7fcd68078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.651+0000 7fcd7b7fe640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fcd680763f0 0x7fcd68078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:21.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.787+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd7c0d2950 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.789+0000 7fcd78ff9640 1 -- 172.21.15.59:0/584827017 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) ==== 74+0+17630 (secure 0 0 0) 0x7fcd6c065430 con 0x7fcd7c0b3ed0 2024-06-29T15:38:21.789 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:21.789 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":162,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:21.038087+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.250305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.841623+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":105,"snap_epoch":162,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6833/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/2522719166":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/784386967":"2024-06-30T14:42:10.228325+0000","172.21.15.153:6832/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3466261848":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/568570198":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/2676841282":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3008499766":"2024-06-30T14:42:10.228325+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":47},{"begin":59,"length":47}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":34,"length":16},{"begin":90,"length":16}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fcd680763f0 msgr2=0x7fcd68078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fcd680763f0 0x7fcd68078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0b3ed0 msgr2=0x7fcd7c15ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0b3ed0 0x7fcd7c15ddc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00e6a0 tx=0x7fcd6c00eb60 comp rx=0 tx=0).stop 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 shutdown_connections 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fcd680763f0 0x7fcd68078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd7c15ee10 0x7fcd7c155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c0d23c0 0x7fcd7c155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 --2- 172.21.15.59:0/584827017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd7c0b3ed0 0x7fcd7c15ddc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 >> 172.21.15.59:0/584827017 conn(0x7fcd7c0d3ee0 msgr2=0x7fcd7c06c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 shutdown_connections 2024-06-29T15:38:21.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.792+0000 7fcd823ac640 1 -- 172.21.15.59:0/584827017 wait complete. 2024-06-29T15:38:21.809 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:41:56.250305+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:41:56.841623+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 105, 'snap_epoch': 162, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:21.810 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:21.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.932+0000 7fe4a922e640 1 -- 172.21.15.59:0/3969212951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 msgr2=0x7fe4a415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.932+0000 7fe4a922e640 1 --2- 172.21.15.59:0/3969212951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 0x7fe4a415f9b0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40d15d0 tx=0x7fe49c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.932+0000 7fe4a922e640 1 -- 172.21.15.59:0/3969212951 shutdown_connections 2024-06-29T15:38:21.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.932+0000 7fe4a922e640 1 --2- 172.21.15.59:0/3969212951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 0x7fe4a415f9b0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.932+0000 7fe4a922e640 1 --2- 172.21.15.59:0/3969212951 >> v2:172.21.15.153:3300/0 conn(0x7fe4a40d23c0 0x7fe4a40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.932+0000 7fe4a922e640 1 -- 172.21.15.59:0/3969212951 >> 172.21.15.59:0/3969212951 conn(0x7fe4a40d3ee0 msgr2=0x7fe4a40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:21.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.932+0000 7fe4a922e640 1 -- 172.21.15.59:0/3969212951 shutdown_connections 2024-06-29T15:38:21.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.932+0000 7fe4a922e640 1 -- 172.21.15.59:0/3969212951 wait complete. 2024-06-29T15:38:21.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.934+0000 7fe4a922e640 1 Processor -- start 2024-06-29T15:38:21.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.934+0000 7fe4a922e640 1 -- start start 2024-06-29T15:38:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a922e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4a40b3ed0 0x7fe4a415b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a922e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4a40d23c0 0x7fe4a4148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a922e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 0x7fe4a41570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a922e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a4160280 con 0x7fe4a415efe0 2024-06-29T15:38:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a922e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a4160100 con 0x7fe4a40d23c0 2024-06-29T15:38:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a922e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe4a4160400 con 0x7fe4a40b3ed0 2024-06-29T15:38:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 0x7fe4a41570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 0x7fe4a41570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36158/0 (socket says 172.21.15.59:36158) 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a359e640 1 -- 172.21.15.59:0/1880128664 learned_addr learned my addr 172.21.15.59:0/1880128664 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a259c640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4a40d23c0 0x7fe4a4148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a2d9d640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4a40b3ed0 0x7fe4a415b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a359e640 1 -- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4a40b3ed0 msgr2=0x7fe4a415b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a359e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4a40b3ed0 0x7fe4a415b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a359e640 1 -- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4a40d23c0 msgr2=0x7fe4a4148cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a359e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4a40d23c0 0x7fe4a4148cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a359e640 1 -- 172.21.15.59:0/1880128664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a415d3e0 con 0x7fe4a415efe0 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a259c640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4a40d23c0 0x7fe4a4148cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.936+0000 7fe4a2d9d640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4a40b3ed0 0x7fe4a415b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.937+0000 7fe4a359e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 0x7fe4a41570c0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fe49c02aea0 tx=0x7fe49c033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.937+0000 7fe48b7fe640 1 -- 172.21.15.59:0/1880128664 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c031990 con 0x7fe4a415efe0 2024-06-29T15:38:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.937+0000 7fe48b7fe640 1 -- 172.21.15.59:0/1880128664 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe49c031c20 con 0x7fe4a415efe0 2024-06-29T15:38:21.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.937+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a415d6c0 con 0x7fe4a415efe0 2024-06-29T15:38:21.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.937+0000 7fe48b7fe640 1 -- 172.21.15.59:0/1880128664 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c033070 con 0x7fe4a415efe0 2024-06-29T15:38:21.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.937+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a415dc80 con 0x7fe4a415efe0 2024-06-29T15:38:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.938+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a415e0c0 con 0x7fe4a415efe0 2024-06-29T15:38:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.939+0000 7fe48b7fe640 1 -- 172.21.15.59:0/1880128664 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fe49c033210 con 0x7fe4a415efe0 2024-06-29T15:38:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.939+0000 7fe48b7fe640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe4900763f0 0x7fe490078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.940+0000 7fe48b7fe640 1 -- 172.21.15.59:0/1880128664 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 6561+0+0 (secure 0 0 0) 0x7fe49c0b9150 con 0x7fe4a415efe0 2024-06-29T15:38:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.940+0000 7fe4a2d9d640 1 -- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe4900763f0 msgr2=0x7fe490078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.940+0000 7fe4a2d9d640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe4900763f0 0x7fe490078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:21.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:21.944+0000 7fe48b7fe640 1 -- 172.21.15.59:0/1880128664 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe49c03f030 con 0x7fe4a415efe0 2024-06-29T15:38:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.141+0000 7fe4a2d9d640 1 -- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe4900763f0 msgr2=0x7fe490078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.141+0000 7fe4a2d9d640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe4900763f0 0x7fe490078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:22.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.152+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe4a40b3990 con 0x7fe4a415efe0 2024-06-29T15:38:22.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.152+0000 7fe48b7fe640 1 -- 172.21.15.59:0/1880128664 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v163) ==== 93+0+10 (secure 0 0 0) 0x7fe49c081e40 con 0x7fe4a415efe0 2024-06-29T15:38:22.153 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:38:22.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe4900763f0 msgr2=0x7fe490078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:22.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe4900763f0 0x7fe490078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 msgr2=0x7fe4a41570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 0x7fe4a41570c0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fe49c02aea0 tx=0x7fe49c033f70 comp rx=0 tx=0).stop 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 shutdown_connections 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe4900763f0 0x7fe490078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4a415efe0 0x7fe4a41570c0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4a40d23c0 0x7fe4a4148cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 --2- 172.21.15.59:0/1880128664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4a40b3ed0 0x7fe4a415b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 >> 172.21.15.59:0/1880128664 conn(0x7fe4a40d3ee0 msgr2=0x7fe4a406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 shutdown_connections 2024-06-29T15:38:22.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.155+0000 7fe4a922e640 1 -- 172.21.15.59:0/1880128664 wait complete. 2024-06-29T15:38:22.167 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.297+0000 7f7edbe66640 1 -- 172.21.15.59:0/1078831921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ed40d23c0 msgr2=0x7f7ed40b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.297+0000 7f7edbe66640 1 --2- 172.21.15.59:0/1078831921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ed40d23c0 0x7f7ed40b3990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009830 tx=0x7f7ed002a5c0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.297+0000 7f7edbe66640 1 -- 172.21.15.59:0/1078831921 shutdown_connections 2024-06-29T15:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.297+0000 7f7edbe66640 1 --2- 172.21.15.59:0/1078831921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ed40d23c0 0x7f7ed40b3990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.297+0000 7f7edbe66640 1 -- 172.21.15.59:0/1078831921 >> 172.21.15.59:0/1078831921 conn(0x7f7ed40d3ee0 msgr2=0x7f7ed40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.297+0000 7f7edbe66640 1 -- 172.21.15.59:0/1078831921 shutdown_connections 2024-06-29T15:38:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.297+0000 7f7edbe66640 1 -- 172.21.15.59:0/1078831921 wait complete. 2024-06-29T15:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.299+0000 7f7edbe66640 1 Processor -- start 2024-06-29T15:38:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.299+0000 7f7edbe66640 1 -- start start 2024-06-29T15:38:22.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7edbe66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ed40b3ed0 0x7f7ed4151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7edbe66640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ed40d23c0 0x7f7ed414a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7edbe66640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ed415efe0 0x7f7ed41522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7edbe66640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed4160450 con 0x7f7ed40d23c0 2024-06-29T15:38:22.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7edbe66640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41602d0 con 0x7f7ed40b3ed0 2024-06-29T15:38:22.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7edbe66640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed41605d0 con 0x7f7ed415efe0 2024-06-29T15:38:22.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ed40d23c0 0x7f7ed414a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ed40d23c0 0x7f7ed414a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36162/0 (socket says 172.21.15.59:36162) 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9c02640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ed40b3ed0 0x7f7ed4151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 -- 172.21.15.59:0/552460580 learned_addr learned my addr 172.21.15.59:0/552460580 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7eda403640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ed415efe0 0x7f7ed41522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 -- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ed415efe0 msgr2=0x7f7ed41522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ed415efe0 0x7f7ed41522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 -- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ed40b3ed0 msgr2=0x7f7ed4151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ed40b3ed0 0x7f7ed4151190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 -- 172.21.15.59:0/552460580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed40c6030 con 0x7f7ed40d23c0 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.301+0000 7f7ed9401640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ed40d23c0 0x7f7ed414a250 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c3f0 tx=0x7f7ec4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.302+0000 7f7eca7fc640 1 -- 172.21.15.59:0/552460580 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4017070 con 0x7f7ed40d23c0 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.302+0000 7f7eca7fc640 1 -- 172.21.15.59:0/552460580 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ec40053e0 con 0x7f7ed40d23c0 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.302+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed4151b00 con 0x7f7ed40d23c0 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.302+0000 7f7eca7fc640 1 -- 172.21.15.59:0/552460580 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4003850 con 0x7f7ed40d23c0 2024-06-29T15:38:22.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.302+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed40c65b0 con 0x7f7ed40d23c0 2024-06-29T15:38:22.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.303+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed40b7c60 con 0x7f7ed40d23c0 2024-06-29T15:38:22.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.304+0000 7f7eca7fc640 1 -- 172.21.15.59:0/552460580 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f7ec401a070 con 0x7f7ed40d23c0 2024-06-29T15:38:22.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.305+0000 7f7eca7fc640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7eb8076420 0x7f7eb8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.305+0000 7f7ed9c02640 1 -- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7eb8076420 msgr2=0x7f7eb8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:22.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.305+0000 7f7eca7fc640 1 -- 172.21.15.59:0/552460580 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 6561+0+0 (secure 0 0 0) 0x7f7ec4097100 con 0x7f7ed40d23c0 2024-06-29T15:38:22.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.305+0000 7f7ed9c02640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7eb8076420 0x7f7eb8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:22.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.309+0000 7f7eca7fc640 1 -- 172.21.15.59:0/552460580 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7ed40b7c60 con 0x7f7ed40d23c0 2024-06-29T15:38:22.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.506+0000 7f7ed9c02640 1 -- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7eb8076420 msgr2=0x7f7eb8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:22.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.506+0000 7f7ed9c02640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7eb8076420 0x7f7eb8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:22.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.527+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7ed4151850 con 0x7f7ed40d23c0 2024-06-29T15:38:22.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.527+0000 7f7eca7fc640 1 -- 172.21.15.59:0/552460580 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v163) ==== 104+0+11 (secure 0 0 0) 0x7f7ed40b42e0 con 0x7f7ed40d23c0 2024-06-29T15:38:22.527 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7eb8076420 msgr2=0x7f7eb8078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7eb8076420 0x7f7eb8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ed40d23c0 msgr2=0x7f7ed414a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ed40d23c0 0x7f7ed414a250 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c3f0 tx=0x7f7ec4001e70 comp rx=0 tx=0).stop 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 shutdown_connections 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7eb8076420 0x7f7eb8078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ed415efe0 0x7f7ed41522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ed40d23c0 0x7f7ed414a250 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 --2- 172.21.15.59:0/552460580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ed40b3ed0 0x7f7ed4151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 >> 172.21.15.59:0/552460580 conn(0x7f7ed40d3ee0 msgr2=0x7f7ed40b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 shutdown_connections 2024-06-29T15:38:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.529+0000 7f7edbe66640 1 -- 172.21.15.59:0/552460580 wait complete. 2024-06-29T15:38:22.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:38:22.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.668+0000 7fc1eda22640 1 -- 172.21.15.59:0/3213339147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 msgr2=0x7fc1e806ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.668+0000 7fc1eda22640 1 --2- 172.21.15.59:0/3213339147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 0x7fc1e806ce80 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80752d0 tx=0x7fc1dc02a560 comp rx=0 tx=0).stop 2024-06-29T15:38:22.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.669+0000 7fc1eda22640 1 -- 172.21.15.59:0/3213339147 shutdown_connections 2024-06-29T15:38:22.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.669+0000 7fc1eda22640 1 --2- 172.21.15.59:0/3213339147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 0x7fc1e806ce80 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.669+0000 7fc1eda22640 1 --2- 172.21.15.59:0/3213339147 >> v2:172.21.15.153:3300/0 conn(0x7fc1e8145680 0x7fc1e80b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.669+0000 7fc1eda22640 1 -- 172.21.15.59:0/3213339147 >> 172.21.15.59:0/3213339147 conn(0x7fc1e8072d80 msgr2=0x7fc1e8073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.669+0000 7fc1eda22640 1 -- 172.21.15.59:0/3213339147 shutdown_connections 2024-06-29T15:38:22.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.669+0000 7fc1eda22640 1 -- 172.21.15.59:0/3213339147 wait complete. 2024-06-29T15:38:22.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.670+0000 7fc1eda22640 1 Processor -- start 2024-06-29T15:38:22.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.671+0000 7fc1eda22640 1 -- start start 2024-06-29T15:38:22.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1eda22640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 0x7fc1e813dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1eda22640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1e8145680 0x7fc1e80af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1eda22640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1e815f860 0x7fc1e81443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1eda22640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80cfbb0 con 0x7fc1e806ca70 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1eda22640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80b41c0 con 0x7fc1e8145680 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1eda22640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e80cfd30 con 0x7fc1e815f860 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1e6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 0x7fc1e813dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1e6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 0x7fc1e813dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36186/0 (socket says 172.21.15.59:36186) 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1e6ffd640 1 -- 172.21.15.59:0/432956302 learned_addr learned my addr 172.21.15.59:0/432956302 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1e67fc640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1e8145680 0x7fc1e80af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1e77fe640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1e815f860 0x7fc1e81443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1e6ffd640 1 -- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1e815f860 msgr2=0x7fc1e81443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.672+0000 7fc1e6ffd640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1e815f860 0x7fc1e81443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1e6ffd640 1 -- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1e8145680 msgr2=0x7fc1e80af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1e6ffd640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1e8145680 0x7fc1e80af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1e6ffd640 1 -- 172.21.15.59:0/432956302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e813e2e0 con 0x7fc1e806ca70 2024-06-29T15:38:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1e6ffd640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 0x7fc1e813dda0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800e7c0 tx=0x7fc1d800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1cffff640 1 -- 172.21.15.59:0/432956302 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d800ccc0 con 0x7fc1e806ca70 2024-06-29T15:38:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1cffff640 1 -- 172.21.15.59:0/432956302 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1d8010430 con 0x7fc1e806ca70 2024-06-29T15:38:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e80b0450 con 0x7fc1e806ca70 2024-06-29T15:38:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1cffff640 1 -- 172.21.15.59:0/432956302 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8010860 con 0x7fc1e806ca70 2024-06-29T15:38:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.673+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e80afe90 con 0x7fc1e806ca70 2024-06-29T15:38:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.674+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e81604f0 con 0x7fc1e806ca70 2024-06-29T15:38:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.675+0000 7fc1cffff640 1 -- 172.21.15.59:0/432956302 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fc1d8010a00 con 0x7fc1e806ca70 2024-06-29T15:38:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.675+0000 7fc1cffff640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fc1d40763f0 0x7fc1d4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.675+0000 7fc1e67fc640 1 -- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fc1d40763f0 msgr2=0x7fc1d4078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.675+0000 7fc1cffff640 1 -- 172.21.15.59:0/432956302 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 6561+0+0 (secure 0 0 0) 0x7fc1d8098790 con 0x7fc1e806ca70 2024-06-29T15:38:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.676+0000 7fc1e67fc640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fc1d40763f0 0x7fc1d4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:22.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.680+0000 7fc1cffff640 1 -- 172.21.15.59:0/432956302 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc1d80613f0 con 0x7fc1e806ca70 2024-06-29T15:38:22.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.871+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc1e80af670 con 0x7fc1e806ca70 2024-06-29T15:38:22.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.871+0000 7fc1cffff640 1 -- 172.21.15.59:0/432956302 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v163) ==== 100+0+11 (secure 0 0 0) 0x7fc1d80656f0 con 0x7fc1e806ca70 2024-06-29T15:38:22.872 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fc1d40763f0 msgr2=0x7fc1d4078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fc1d40763f0 0x7fc1d4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 msgr2=0x7fc1e813dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 0x7fc1e813dda0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800e7c0 tx=0x7fc1d800ec80 comp rx=0 tx=0).stop 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 shutdown_connections 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fc1d40763f0 0x7fc1d4078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1e815f860 0x7fc1e81443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1e8145680 0x7fc1e80af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 --2- 172.21.15.59:0/432956302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e806ca70 0x7fc1e813dda0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 >> 172.21.15.59:0/432956302 conn(0x7fc1e8072d80 msgr2=0x7fc1e80b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 shutdown_connections 2024-06-29T15:38:22.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:22.873+0000 7fc1eda22640 1 -- 172.21.15.59:0/432956302 wait complete. 2024-06-29T15:38:22.885 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:38:23.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.004+0000 7f0679674640 1 -- 172.21.15.59:0/1541919218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740d0130 msgr2=0x7f0674143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.004+0000 7f0679674640 1 --2- 172.21.15.59:0/1541919218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740d0130 0x7f0674143840 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f0674074350 tx=0x7f066402a5f0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.004+0000 7f0679674640 1 -- 172.21.15.59:0/1541919218 shutdown_connections 2024-06-29T15:38:23.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.004+0000 7f0679674640 1 --2- 172.21.15.59:0/1541919218 >> v2:172.21.15.153:3300/0 conn(0x7f06740b4ee0 0x7f067406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.004+0000 7f0679674640 1 --2- 172.21.15.59:0/1541919218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740d0130 0x7f0674143840 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.004+0000 7f0679674640 1 -- 172.21.15.59:0/1541919218 >> 172.21.15.59:0/1541919218 conn(0x7f06740b2ab0 msgr2=0x7f06740b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.004+0000 7f0679674640 1 -- 172.21.15.59:0/1541919218 shutdown_connections 2024-06-29T15:38:23.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.005+0000 7f0679674640 1 -- 172.21.15.59:0/1541919218 wait complete. 2024-06-29T15:38:23.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.006+0000 7f0679674640 1 Processor -- start 2024-06-29T15:38:23.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.006+0000 7f0679674640 1 -- start start 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f0679674640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f067406ea10 0x7f06741f8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f0679674640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740b4ee0 0x7f067415a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f0679674640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06740d0130 0x7f06740aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f0679674640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f067415ed80 con 0x7f06740b4ee0 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f0679674640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f067406f030 con 0x7f067406ea10 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f0679674640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f067415ef00 con 0x7f06740d0130 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740b4ee0 0x7f067415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f0672ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f067406ea10 0x7f06741f8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740b4ee0 0x7f067415a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59130/0 (socket says 172.21.15.59:59130) 2024-06-29T15:38:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 -- 172.21.15.59:0/204190805 learned_addr learned my addr 172.21.15.59:0/204190805 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f0672ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f067406ea10 0x7f06741f8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:45698/0 (socket says 172.21.15.59:45698) 2024-06-29T15:38:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06737fe640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06740d0130 0x7f06740aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 -- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06740d0130 msgr2=0x7f06740aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06740d0130 0x7f06740aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 -- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f067406ea10 msgr2=0x7f06741f8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f067406ea10 0x7f06741f8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 -- 172.21.15.59:0/204190805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06740cd420 con 0x7f06740b4ee0 2024-06-29T15:38:23.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.008+0000 7f06727fc640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740b4ee0 0x7f067415a420 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f066800e790 tx=0x7f066800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:23.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.009+0000 7f065b7fe640 1 -- 172.21.15.59:0/204190805 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066800eea0 con 0x7f06740b4ee0 2024-06-29T15:38:23.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.009+0000 7f065b7fe640 1 -- 172.21.15.59:0/204190805 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f066800cdb0 con 0x7f06740b4ee0 2024-06-29T15:38:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.009+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0674141ee0 con 0x7f06740b4ee0 2024-06-29T15:38:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.009+0000 7f065b7fe640 1 -- 172.21.15.59:0/204190805 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668010690 con 0x7f06740b4ee0 2024-06-29T15:38:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.009+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06741f4c80 con 0x7f06740b4ee0 2024-06-29T15:38:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.010+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f067415f3d0 con 0x7f06740b4ee0 2024-06-29T15:38:23.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.011+0000 7f065b7fe640 1 -- 172.21.15.59:0/204190805 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f0668010830 con 0x7f06740b4ee0 2024-06-29T15:38:23.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.011+0000 7f065b7fe640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f06600763f0 0x7f0660078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.011+0000 7f0672ffd640 1 -- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f06600763f0 msgr2=0x7f0660078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:23.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.011+0000 7f065b7fe640 1 -- 172.21.15.59:0/204190805 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 6561+0+0 (secure 0 0 0) 0x7f0668098de0 con 0x7f06740b4ee0 2024-06-29T15:38:23.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.011+0000 7f0672ffd640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f06600763f0 0x7f0660078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.015+0000 7f065b7fe640 1 -- 172.21.15.59:0/204190805 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0668061c50 con 0x7f06740b4ee0 2024-06-29T15:38:23.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.212+0000 7f0672ffd640 1 -- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f06600763f0 msgr2=0x7f0660078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:23.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.212+0000 7f0672ffd640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f06600763f0 0x7f0660078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:23.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.219+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f067406d6d0 con 0x7f06740b4ee0 2024-06-29T15:38:23.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.220+0000 7f065b7fe640 1 -- 172.21.15.59:0/204190805 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v163) ==== 80+0+476 (secure 0 0 0) 0x7f0668065f50 con 0x7f06740b4ee0 2024-06-29T15:38:23.220 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/3378196672 2024-06-30T14:42:10.228325+0000 2024-06-29T15:38:23.220 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2522719166 2024-06-30T14:42:10.228325+0000 2024-06-29T15:38:23.220 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/784386967 2024-06-30T14:42:10.228325+0000 2024-06-29T15:38:23.221 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/3378196672 2024-06-30T14:42:10.228325+0000 2024-06-29T15:38:23.221 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3466261848 2024-06-30T14:42:10.228325+0000 2024-06-29T15:38:23.221 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/568570198 2024-06-30T14:42:10.228325+0000 2024-06-29T15:38:23.221 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2676841282 2024-06-30T14:42:10.228325+0000 2024-06-29T15:38:23.221 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3008499766 2024-06-30T14:42:10.228325+0000 2024-06-29T15:38:23.221 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f06600763f0 msgr2=0x7f0660078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f06600763f0 0x7f0660078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740b4ee0 msgr2=0x7f067415a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740b4ee0 0x7f067415a420 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f066800e790 tx=0x7f066800ec50 comp rx=0 tx=0).stop 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 shutdown_connections 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f06600763f0 0x7f0660078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06740d0130 0x7f06740aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06740b4ee0 0x7f067415a420 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 --2- 172.21.15.59:0/204190805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f067406ea10 0x7f06741f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 >> 172.21.15.59:0/204190805 conn(0x7f06740b2ab0 msgr2=0x7f0674143840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 shutdown_connections 2024-06-29T15:38:23.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.222+0000 7f0679674640 1 -- 172.21.15.59:0/204190805 wait complete. 2024-06-29T15:38:23.234 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:38:23.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:23.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:23.268 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T15:38:23.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:23.268 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:38:23.325 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:23.326 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:38:23.326 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:23.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:23.342 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:23.342 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:23.342 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:23.382 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:38:23.382 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:23.382 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:38:23.382 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:23.398 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:23.399 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:23.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.541+0000 7fbc295d2640 1 -- 172.21.15.59:0/3040924786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc241448e0 msgr2=0x7fbc240cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.541+0000 7fbc295d2640 1 --2- 172.21.15.59:0/3040924786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc241448e0 0x7fbc240cfbe0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009830 tx=0x7fbc1402a540 comp rx=0 tx=0).stop 2024-06-29T15:38:23.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.542+0000 7fbc295d2640 1 -- 172.21.15.59:0/3040924786 shutdown_connections 2024-06-29T15:38:23.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.542+0000 7fbc295d2640 1 --2- 172.21.15.59:0/3040924786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc241448e0 0x7fbc240cfbe0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.542+0000 7fbc295d2640 1 -- 172.21.15.59:0/3040924786 >> 172.21.15.59:0/3040924786 conn(0x7fbc240b2ab0 msgr2=0x7fbc240b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.542+0000 7fbc295d2640 1 -- 172.21.15.59:0/3040924786 shutdown_connections 2024-06-29T15:38:23.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.542+0000 7fbc295d2640 1 -- 172.21.15.59:0/3040924786 wait complete. 2024-06-29T15:38:23.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.543+0000 7fbc295d2640 1 Processor -- start 2024-06-29T15:38:23.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.544+0000 7fbc295d2640 1 -- start start 2024-06-29T15:38:23.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc295d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc240cd990 0x7fbc2415abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc295d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc240d0120 0x7fbc240ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc295d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc241448e0 0x7fbc240abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc295d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2406d680 con 0x7fbc241448e0 2024-06-29T15:38:23.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc295d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2406d500 con 0x7fbc240d0120 2024-06-29T15:38:23.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc295d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbc2406d800 con 0x7fbc240cd990 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc241448e0 0x7fbc240abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc241448e0 0x7fbc240abc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59136/0 (socket says 172.21.15.59:59136) 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc237fe640 1 -- 172.21.15.59:0/4201918039 learned_addr learned my addr 172.21.15.59:0/4201918039 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc22ffd640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc240cd990 0x7fbc2415abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc227fc640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc240d0120 0x7fbc240ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc237fe640 1 -- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc240cd990 msgr2=0x7fbc2415abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc237fe640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc240cd990 0x7fbc2415abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc237fe640 1 -- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc240d0120 msgr2=0x7fbc240ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc237fe640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc240d0120 0x7fbc240ab700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.545+0000 7fbc237fe640 1 -- 172.21.15.59:0/4201918039 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2415eb80 con 0x7fbc241448e0 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.546+0000 7fbc237fe640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc241448e0 0x7fbc240abc40 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00cbc0 tx=0x7fbc1c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.546+0000 7fbc0b7fe640 1 -- 172.21.15.59:0/4201918039 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c016020 con 0x7fbc241448e0 2024-06-29T15:38:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.546+0000 7fbc0b7fe640 1 -- 172.21.15.59:0/4201918039 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc1c00ae50 con 0x7fbc241448e0 2024-06-29T15:38:23.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.546+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc240ae810 con 0x7fbc241448e0 2024-06-29T15:38:23.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.546+0000 7fbc0b7fe640 1 -- 172.21.15.59:0/4201918039 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c012680 con 0x7fbc241448e0 2024-06-29T15:38:23.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.546+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc240aedd0 con 0x7fbc241448e0 2024-06-29T15:38:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.547+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2406dcd0 con 0x7fbc241448e0 2024-06-29T15:38:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.548+0000 7fbc0b7fe640 1 -- 172.21.15.59:0/4201918039 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fbc1c019020 con 0x7fbc241448e0 2024-06-29T15:38:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.548+0000 7fbc0b7fe640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fbc10076420 0x7fbc10078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.549+0000 7fbc22ffd640 1 -- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fbc10076420 msgr2=0x7fbc10078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.549+0000 7fbc0b7fe640 1 -- 172.21.15.59:0/4201918039 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 6561+0+0 (secure 0 0 0) 0x7fbc1c097fe0 con 0x7fbc241448e0 2024-06-29T15:38:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.549+0000 7fbc22ffd640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fbc10076420 0x7fbc10078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:23.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.553+0000 7fbc0b7fe640 1 -- 172.21.15.59:0/4201918039 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbc1c020030 con 0x7fbc241448e0 2024-06-29T15:38:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.750+0000 7fbc22ffd640 1 -- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fbc10076420 msgr2=0x7fbc10078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.750+0000 7fbc22ffd640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fbc10076420 0x7fbc10078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:23.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.782+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc2415f440 con 0x7fbc241448e0 2024-06-29T15:38:23.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.782+0000 7fbc0b7fe640 1 -- 172.21.15.59:0/4201918039 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5225 (secure 0 0 0) 0x7fbc240d0530 con 0x7fbc241448e0 2024-06-29T15:38:23.783 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:23.783 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"btime":"2024-06-29T14:42:01:534262+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/507949731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":507949731},{"type":"v1","addr":"172.21.15.153:6837","nonce":507949731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":8812,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1668688055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1668688055},{"type":"v1","addr":"172.21.15.153:6836","nonce":1668688055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":9372,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3535921856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3535921856},{"type":"v1","addr":"172.21.15.59:6834","nonce":3535921856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.891192+0000","modified":"2024-06-29T14:42:00.924713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/178954889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":178954889},{"type":"v1","addr":"172.21.15.59:6835","nonce":178954889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9375,"qdb_cluster":[9375]},"id":9}]} 2024-06-29T15:38:23.783 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 80 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fbc10076420 msgr2=0x7fbc10078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fbc10076420 0x7fbc10078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc241448e0 msgr2=0x7fbc240abc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc241448e0 0x7fbc240abc40 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c00cbc0 tx=0x7fbc1c00a430 comp rx=0 tx=0).stop 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 shutdown_connections 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fbc10076420 0x7fbc10078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc241448e0 0x7fbc240abc40 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc240d0120 0x7fbc240ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 --2- 172.21.15.59:0/4201918039 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc240cd990 0x7fbc2415abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 >> 172.21.15.59:0/4201918039 conn(0x7fbc240b2ab0 msgr2=0x7fbc24144e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 shutdown_connections 2024-06-29T15:38:23.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.784+0000 7fbc295d2640 1 -- 172.21.15.59:0/4201918039 wait complete. 2024-06-29T15:38:23.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:23.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.915+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/937382136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 msgr2=0x7fa788003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.915+0000 7fa7a3c2e640 1 --2- 172.21.15.59:0/937382136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 0x7fa788003f30 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0d2e40 tx=0x7fa788005830 comp rx=0 tx=0).stop 2024-06-29T15:38:23.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.916+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/937382136 shutdown_connections 2024-06-29T15:38:23.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.916+0000 7fa7a3c2e640 1 --2- 172.21.15.59:0/937382136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 0x7fa788003f30 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.916+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/937382136 >> 172.21.15.59:0/937382136 conn(0x7fa79c0d3ee0 msgr2=0x7fa79c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.916+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/937382136 shutdown_connections 2024-06-29T15:38:23.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.916+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/937382136 wait complete. 2024-06-29T15:38:23.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.917+0000 7fa7a3c2e640 1 Processor -- start 2024-06-29T15:38:23.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.917+0000 7fa7a3c2e640 1 -- start start 2024-06-29T15:38:23.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a3c2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 0x7fa79c14f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a3c2e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c150200 0x7fa79c150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a3c2e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c1557b0 0x7fa79c155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a3c2e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c15ed10 con 0x7fa788003ac0 2024-06-29T15:38:23.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a3c2e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c15eb90 con 0x7fa79c150200 2024-06-29T15:38:23.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a3c2e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa79c0b3990 con 0x7fa79c1557b0 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 0x7fa79c14f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 0x7fa79c14f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59154/0 (socket says 172.21.15.59:59154) 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 -- 172.21.15.59:0/818057724 learned_addr learned my addr 172.21.15.59:0/818057724 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a11c9640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c150200 0x7fa79c150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a21cb640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c1557b0 0x7fa79c155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 -- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c1557b0 msgr2=0x7fa79c155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c1557b0 0x7fa79c155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 -- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c150200 msgr2=0x7fa79c150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c150200 0x7fa79c150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 -- 172.21.15.59:0/818057724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c156100 con 0x7fa788003ac0 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a11c9640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c150200 0x7fa79c150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.919+0000 7fa7a19ca640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 0x7fa79c14f2a0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fa79000c350 tx=0x7fa7900064a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.920+0000 7fa78e7fc640 1 -- 172.21.15.59:0/818057724 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790003a00 con 0x7fa788003ac0 2024-06-29T15:38:23.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.920+0000 7fa78e7fc640 1 -- 172.21.15.59:0/818057724 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa790003c90 con 0x7fa788003ac0 2024-06-29T15:38:23.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.920+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79c1f84b0 con 0x7fa788003ac0 2024-06-29T15:38:23.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.920+0000 7fa78e7fc640 1 -- 172.21.15.59:0/818057724 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79002e6e0 con 0x7fa788003ac0 2024-06-29T15:38:23.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.920+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa79c1f8a70 con 0x7fa788003ac0 2024-06-29T15:38:23.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.921+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa79c0b3ef0 con 0x7fa788003ac0 2024-06-29T15:38:23.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.922+0000 7fa78e7fc640 1 -- 172.21.15.59:0/818057724 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fa790035020 con 0x7fa788003ac0 2024-06-29T15:38:23.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.923+0000 7fa78e7fc640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fa780076420 0x7fa780078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.923+0000 7fa7a11c9640 1 -- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fa780076420 msgr2=0x7fa780078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:23.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.923+0000 7fa78e7fc640 1 -- 172.21.15.59:0/818057724 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 6561+0+0 (secure 0 0 0) 0x7fa7900b8560 con 0x7fa788003ac0 2024-06-29T15:38:23.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.923+0000 7fa7a11c9640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fa780076420 0x7fa780078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:23.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:23.927+0000 7fa78e7fc640 1 -- 172.21.15.59:0/818057724 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa790047020 con 0x7fa788003ac0 2024-06-29T15:38:24.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.124+0000 7fa7a11c9640 1 -- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fa780076420 msgr2=0x7fa780078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:24.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.124+0000 7fa7a11c9640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fa780076420 0x7fa780078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:24.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.126+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa79c0d2950 con 0x7fa788003ac0 2024-06-29T15:38:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.128+0000 7fa78e7fc640 1 -- 172.21.15.59:0/818057724 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) ==== 74+0+17629 (secure 0 0 0) 0x7fa79c156100 con 0x7fa788003ac0 2024-06-29T15:38:24.128 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:24.129 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":164,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:24.054718+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.250305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.841623+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6833/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/2522719166":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/784386967":"2024-06-30T14:42:10.228325+0000","172.21.15.153:6832/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3466261848":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/568570198":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/2676841282":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3008499766":"2024-06-30T14:42:10.228325+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fa780076420 msgr2=0x7fa780078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fa780076420 0x7fa780078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 msgr2=0x7fa79c14f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 0x7fa79c14f2a0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fa79000c350 tx=0x7fa7900064a0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 shutdown_connections 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fa780076420 0x7fa780078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa79c1557b0 0x7fa79c155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa79c150200 0x7fa79c150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 --2- 172.21.15.59:0/818057724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa788003ac0 0x7fa79c14f2a0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 >> 172.21.15.59:0/818057724 conn(0x7fa79c0d3ee0 msgr2=0x7fa79c161440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 shutdown_connections 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.130+0000 7fa7a3c2e640 1 -- 172.21.15.59:0/818057724 wait complete. 2024-06-29T15:38:24.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:41:56.250305+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:41:56.841623+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 164, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:24.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:24.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.268+0000 7fe72a047640 1 -- 172.21.15.59:0/3702477684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 msgr2=0x7fe7241443e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.268+0000 7fe72a047640 1 --2- 172.21.15.59:0/3702477684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 0x7fe7241443e0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe718005fd0 tx=0x7fe71802a540 comp rx=0 tx=0).stop 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.268+0000 7fe72a047640 1 -- 172.21.15.59:0/3702477684 shutdown_connections 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.268+0000 7fe72a047640 1 --2- 172.21.15.59:0/3702477684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 0x7fe7241443e0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.268+0000 7fe72a047640 1 --2- 172.21.15.59:0/3702477684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724071620 0x7fe72415bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.268+0000 7fe72a047640 1 -- 172.21.15.59:0/3702477684 >> 172.21.15.59:0/3702477684 conn(0x7fe724073e50 msgr2=0x7fe7240b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.268+0000 7fe72a047640 1 -- 172.21.15.59:0/3702477684 shutdown_connections 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.268+0000 7fe72a047640 1 -- 172.21.15.59:0/3702477684 wait complete. 2024-06-29T15:38:24.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.271+0000 7fe72a047640 1 Processor -- start 2024-06-29T15:38:24.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.271+0000 7fe72a047640 1 -- start start 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.273+0000 7fe72a047640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 0x7fe7240a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.273+0000 7fe72a047640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724071620 0x7fe72413ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.273+0000 7fe72a047640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7240b5320 0x7fe72414d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.273+0000 7fe72a047640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe7240cf530 con 0x7fe72406eb40 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.273+0000 7fe72a047640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe7240cf3b0 con 0x7fe7240b5320 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.273+0000 7fe72a047640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe7240cf6b0 con 0x7fe724071620 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 0x7fe7240a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 0x7fe7240a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59180/0 (socket says 172.21.15.59:59180) 2024-06-29T15:38:24.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 -- 172.21.15.59:0/2253790885 learned_addr learned my addr 172.21.15.59:0/2253790885 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe722ffd640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724071620 0x7fe72413ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe723fff640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7240b5320 0x7fe72414d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 -- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724071620 msgr2=0x7fe72413ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724071620 0x7fe72413ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 -- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7240b5320 msgr2=0x7fe72414d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7240b5320 0x7fe72414d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 -- 172.21.15.59:0/2253790885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72413bd60 con 0x7fe72406eb40 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe7237fe640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 0x7fe7240a94a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fe71400e790 tx=0x7fe71400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe70bfff640 1 -- 172.21.15.59:0/2253790885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71400eea0 con 0x7fe72406eb40 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72413c040 con 0x7fe72406eb40 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe70bfff640 1 -- 172.21.15.59:0/2253790885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe71400cdb0 con 0x7fe72406eb40 2024-06-29T15:38:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe70bfff640 1 -- 172.21.15.59:0/2253790885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe714010690 con 0x7fe72406eb40 2024-06-29T15:38:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.274+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72413c600 con 0x7fe72406eb40 2024-06-29T15:38:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.276+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7241443e0 con 0x7fe72406eb40 2024-06-29T15:38:24.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.276+0000 7fe70bfff640 1 -- 172.21.15.59:0/2253790885 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fe714010830 con 0x7fe72406eb40 2024-06-29T15:38:24.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.277+0000 7fe70bfff640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe7100763f0 0x7fe710078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.277+0000 7fe70bfff640 1 -- 172.21.15.59:0/2253790885 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 6561+0+0 (secure 0 0 0) 0x7fe714098700 con 0x7fe72406eb40 2024-06-29T15:38:24.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.277+0000 7fe722ffd640 1 -- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe7100763f0 msgr2=0x7fe710078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:24.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.277+0000 7fe722ffd640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe7100763f0 0x7fe710078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.281+0000 7fe70bfff640 1 -- 172.21.15.59:0/2253790885 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe714061410 con 0x7fe72406eb40 2024-06-29T15:38:24.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.478+0000 7fe722ffd640 1 -- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe7100763f0 msgr2=0x7fe710078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:24.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.478+0000 7fe722ffd640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe7100763f0 0x7fe710078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:24.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.483+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe7240cec90 con 0x7fe72406eb40 2024-06-29T15:38:24.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.483+0000 7fe70bfff640 1 -- 172.21.15.59:0/2253790885 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) ==== 93+0+10 (secure 0 0 0) 0x7fe714065710 con 0x7fe72406eb40 2024-06-29T15:38:24.483 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:38:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe7100763f0 msgr2=0x7fe710078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe7100763f0 0x7fe710078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 msgr2=0x7fe7240a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 0x7fe7240a94a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fe71400e790 tx=0x7fe71400ec50 comp rx=0 tx=0).stop 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 shutdown_connections 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7fe7100763f0 0x7fe710078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7240b5320 0x7fe72414d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe724071620 0x7fe72413ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 --2- 172.21.15.59:0/2253790885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe72406eb40 0x7fe7240a94a0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 >> 172.21.15.59:0/2253790885 conn(0x7fe724073e50 msgr2=0x7fe7240b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 shutdown_connections 2024-06-29T15:38:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.485+0000 7fe72a047640 1 -- 172.21.15.59:0/2253790885 wait complete. 2024-06-29T15:38:24.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:38:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.621+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/1447900732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf40b40a0 msgr2=0x7f7cf40b73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.621+0000 7f7cfc2f6640 1 --2- 172.21.15.59:0/1447900732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf40b40a0 0x7f7cf40b73f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7cec005fd0 tx=0x7f7cec02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.621+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/1447900732 shutdown_connections 2024-06-29T15:38:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.621+0000 7f7cfc2f6640 1 --2- 172.21.15.59:0/1447900732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf40b40a0 0x7f7cf40b73f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.621+0000 7f7cfc2f6640 1 --2- 172.21.15.59:0/1447900732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf415e290 0x7f7cf40ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.621+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/1447900732 >> 172.21.15.59:0/1447900732 conn(0x7f7cf40d3ee0 msgr2=0x7f7cf40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.621+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/1447900732 shutdown_connections 2024-06-29T15:38:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.621+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/1447900732 wait complete. 2024-06-29T15:38:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.622+0000 7f7cfc2f6640 1 Processor -- start 2024-06-29T15:38:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.623+0000 7f7cfc2f6640 1 -- start start 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfc2f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf40ce650 0x7f7cf4158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfc2f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf415e290 0x7f7cf4156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfc2f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf41f43a0 0x7f7cf4143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfc2f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf40b7cc0 con 0x7f7cf41f43a0 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfc2f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf40b7b40 con 0x7f7cf415e290 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfc2f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf40b7e40 con 0x7f7cf40ce650 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa092640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf40ce650 0x7f7cf4158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa893640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf41f43a0 0x7f7cf4143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa092640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf40ce650 0x7f7cf4158fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:53674/0 (socket says 172.21.15.59:53674) 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa092640 1 -- 172.21.15.59:0/3287389340 learned_addr learned my addr 172.21.15.59:0/3287389340 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cf9891640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf415e290 0x7f7cf4156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa092640 1 -- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf415e290 msgr2=0x7f7cf4156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa092640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf415e290 0x7f7cf4156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa092640 1 -- 172.21.15.59:0/3287389340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf41f43a0 msgr2=0x7f7cf4143da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa092640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf41f43a0 0x7f7cf4143da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa092640 1 -- 172.21.15.59:0/3287389340 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf4157f00 con 0x7f7cf40ce650 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.624+0000 7f7cfa893640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf41f43a0 0x7f7cf4143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.625+0000 7f7cfa092640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf40ce650 0x7f7cf4158fa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000ea20 tx=0x7f7cf000eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.625+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf000ce40 con 0x7f7cf40ce650 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.625+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cf0004020 con 0x7f7cf40ce650 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.625+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0004450 con 0x7f7cf40ce650 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.625+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf41581e0 con 0x7f7cf40ce650 2024-06-29T15:38:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.625+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf41587a0 con 0x7f7cf40ce650 2024-06-29T15:38:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.626+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf40d0190 con 0x7f7cf40ce650 2024-06-29T15:38:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.627+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f7cf00035f0 con 0x7f7cf40ce650 2024-06-29T15:38:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.627+0000 7f7ceaffd640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7cd8076420 0x7f7cd8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.627+0000 7f7cf9891640 1 -- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7cd8076420 msgr2=0x7f7cd8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3376352564 2024-06-29T15:38:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.627+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(164..164 src has 1..164) ==== 6561+0+0 (secure 0 0 0) 0x7f7cf0098540 con 0x7f7cf40ce650 2024-06-29T15:38:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.627+0000 7f7cf9891640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7cd8076420 0x7f7cd8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.632+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7cf40d0190 con 0x7f7cf40ce650 2024-06-29T15:38:24.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.818+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mgrmap(e 87) ==== 97912+0+0 (secure 0 0 0) 0x7f7cf0009400 con 0x7f7cf40ce650 2024-06-29T15:38:24.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.818+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7cd8076420 msgr2=0x7f7cd8078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:24.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.818+0000 7f7ceaffd640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7cd8076420 0x7f7cd8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.832+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7cf41568a0 con 0x7f7cf40ce650 2024-06-29T15:38:24.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.832+0000 7f7ceaffd640 1 -- 172.21.15.59:0/3287389340 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v165) ==== 104+0+11 (secure 0 0 0) 0x7f7cf006da50 con 0x7f7cf40ce650 2024-06-29T15:38:24.833 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:38:24.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf40ce650 msgr2=0x7f7cf4158fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf40ce650 0x7f7cf4158fa0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000ea20 tx=0x7f7cf000eee0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 shutdown_connections 2024-06-29T15:38:24.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:6832/3376352564,v1:172.21.15.153:6833/3376352564] conn(0x7f7cd8076420 0x7f7cd8078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf41f43a0 0x7f7cf4143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf415e290 0x7f7cf4156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 --2- 172.21.15.59:0/3287389340 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf40ce650 0x7f7cf4158fa0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 >> 172.21.15.59:0/3287389340 conn(0x7f7cf40d3ee0 msgr2=0x7f7cf40d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 shutdown_connections 2024-06-29T15:38:24.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.835+0000 7f7cfc2f6640 1 -- 172.21.15.59:0/3287389340 wait complete. 2024-06-29T15:38:24.847 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:38:24.868 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f281fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.868 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f281fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.868 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f281fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.868 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f281fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.868 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f281fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.869 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f2822a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.869 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f2822a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.869 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f2822a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.869 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f2822a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.869 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:24.867+0000 7f2822a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:24.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.969+0000 7f1661163640 1 -- 172.21.15.59:0/1207278141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f165c0b2270 msgr2=0x7f165c15bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.969+0000 7f1661163640 1 --2- 172.21.15.59:0/1207278141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f165c0b2270 0x7f165c15bc20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f164c009860 tx=0x7f164c02a5a0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.969+0000 7f1661163640 1 -- 172.21.15.59:0/1207278141 shutdown_connections 2024-06-29T15:38:24.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.969+0000 7f1661163640 1 --2- 172.21.15.59:0/1207278141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f165c0b2270 0x7f165c15bc20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.969+0000 7f1661163640 1 -- 172.21.15.59:0/1207278141 >> 172.21.15.59:0/1207278141 conn(0x7f165c073e10 msgr2=0x7f165c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.969+0000 7f1661163640 1 -- 172.21.15.59:0/1207278141 shutdown_connections 2024-06-29T15:38:24.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.969+0000 7f1661163640 1 -- 172.21.15.59:0/1207278141 wait complete. 2024-06-29T15:38:24.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.971+0000 7f1661163640 1 Processor -- start 2024-06-29T15:38:24.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.971+0000 7f1661163640 1 -- start start 2024-06-29T15:38:24.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.972+0000 7f1661163640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f165c0b2270 0x7f165c146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f1661163640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f165c0b2fa0 0x7f165c144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f1661163640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c0d1c30 0x7f165c153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f1661163640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f165c0b65d0 con 0x7f165c0b2270 2024-06-29T15:38:24.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f1661163640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f165c0b6450 con 0x7f165c0d1c30 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f1661163640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f165c0b6750 con 0x7f165c0b2fa0 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f165c0b2270 0x7f165c146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f165c0b2270 0x7f165c146a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59210/0 (socket says 172.21.15.59:59210) 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 -- 172.21.15.59:0/3805482520 learned_addr learned my addr 172.21.15.59:0/3805482520 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165b59e640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c0d1c30 0x7f165c153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165a59c640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f165c0b2fa0 0x7f165c144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 -- 172.21.15.59:0/3805482520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f165c0b2fa0 msgr2=0x7f165c144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f165c0b2fa0 0x7f165c144ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 -- 172.21.15.59:0/3805482520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c0d1c30 msgr2=0x7f165c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c0d1c30 0x7f165c153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 -- 172.21.15.59:0/3805482520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165c154410 con 0x7f165c0b2270 2024-06-29T15:38:24.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.973+0000 7f165ad9d640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f165c0b2270 0x7f165c146a00 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f164c02ab90 tx=0x7f164c033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:24.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.974+0000 7f16537fe640 1 -- 172.21.15.59:0/3805482520 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c031b50 con 0x7f165c0b2270 2024-06-29T15:38:24.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.974+0000 7f16537fe640 1 -- 172.21.15.59:0/3805482520 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f164c031de0 con 0x7f165c0b2270 2024-06-29T15:38:24.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.974+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165c145960 con 0x7f165c0b2270 2024-06-29T15:38:24.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.974+0000 7f16537fe640 1 -- 172.21.15.59:0/3805482520 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c033330 con 0x7f165c0b2270 2024-06-29T15:38:24.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.974+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f165c145f20 con 0x7f165c0b2270 2024-06-29T15:38:24.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.975+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f165c159010 con 0x7f165c0b2270 2024-06-29T15:38:24.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.976+0000 7f16537fe640 1 -- 172.21.15.59:0/3805482520 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 87) ==== 97912+0+0 (secure 0 0 0) 0x7f164c0334d0 con 0x7f165c0b2270 2024-06-29T15:38:24.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.976+0000 7f16537fe640 1 -- 172.21.15.59:0/3805482520 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6861+0+0 (secure 0 0 0) 0x7f164c0b8bf0 con 0x7f165c0b2270 2024-06-29T15:38:24.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:24.980+0000 7f16537fe640 1 -- 172.21.15.59:0/3805482520 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f165c159010 con 0x7f165c0b2270 2024-06-29T15:38:25.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.174+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f165c0b6c40 con 0x7f165c0b2270 2024-06-29T15:38:25.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.175+0000 7f16537fe640 1 -- 172.21.15.59:0/3805482520 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v165) ==== 100+0+11 (secure 0 0 0) 0x7f164c052080 con 0x7f165c0b2270 2024-06-29T15:38:25.175 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:38:25.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f165c0b2270 msgr2=0x7f165c146a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f165c0b2270 0x7f165c146a00 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f164c02ab90 tx=0x7f164c033ba0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 shutdown_connections 2024-06-29T15:38:25.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f165c0d1c30 0x7f165c153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f165c0b2fa0 0x7f165c144ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 --2- 172.21.15.59:0/3805482520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f165c0b2270 0x7f165c146a00 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 >> 172.21.15.59:0/3805482520 conn(0x7f165c073e10 msgr2=0x7f165c06fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 shutdown_connections 2024-06-29T15:38:25.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.177+0000 7f1661163640 1 -- 172.21.15.59:0/3805482520 wait complete. 2024-06-29T15:38:25.189 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:25.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.308+0000 7fb069074640 1 -- 172.21.15.59:0/3672114106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162190 msgr2=0x7fb0640b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.308+0000 7fb069074640 1 --2- 172.21.15.59:0/3672114106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162190 0x7fb0640b38e0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fb05c005fd0 tx=0x7fb05c02aa10 comp rx=0 tx=0).stop 2024-06-29T15:38:25.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.308+0000 7fb069074640 1 -- 172.21.15.59:0/3672114106 shutdown_connections 2024-06-29T15:38:25.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.308+0000 7fb069074640 1 --2- 172.21.15.59:0/3672114106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162190 0x7fb0640b38e0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.308+0000 7fb069074640 1 --2- 172.21.15.59:0/3672114106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb064162770 0x7fb0640ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.308+0000 7fb069074640 1 -- 172.21.15.59:0/3672114106 >> 172.21.15.59:0/3672114106 conn(0x7fb0640d45b0 msgr2=0x7fb0640d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.308+0000 7fb069074640 1 -- 172.21.15.59:0/3672114106 shutdown_connections 2024-06-29T15:38:25.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.308+0000 7fb069074640 1 -- 172.21.15.59:0/3672114106 wait complete. 2024-06-29T15:38:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.310+0000 7fb069074640 1 Processor -- start 2024-06-29T15:38:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.310+0000 7fb069074640 1 -- start start 2024-06-29T15:38:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb069074640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb06415e0a0 0x7fb064150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb069074640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162770 0x7fb0640c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb069074640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0640c84f0 0x7fb0640c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb069074640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb0640b43a0 con 0x7fb064162770 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb069074640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb0640b4220 con 0x7fb0640c84f0 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb069074640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb0640b4520 con 0x7fb06415e0a0 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb06259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162770 0x7fb0640c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb062d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb06415e0a0 0x7fb064150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb06359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0640c84f0 0x7fb0640c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb062d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb06415e0a0 0x7fb064150d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:53716/0 (socket says 172.21.15.59:53716) 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb062d9d640 1 -- 172.21.15.59:0/2531099634 learned_addr learned my addr 172.21.15.59:0/2531099634 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.311+0000 7fb06259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162770 0x7fb0640c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59222/0 (socket says 172.21.15.59:59222) 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb062d9d640 1 -- 172.21.15.59:0/2531099634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0640c84f0 msgr2=0x7fb0640c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb062d9d640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0640c84f0 0x7fb0640c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb062d9d640 1 -- 172.21.15.59:0/2531099634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162770 msgr2=0x7fb0640c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb062d9d640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162770 0x7fb0640c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb062d9d640 1 -- 172.21.15.59:0/2531099634 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0640c8a90 con 0x7fb06415e0a0 2024-06-29T15:38:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb06259c640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162770 0x7fb0640c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:25.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb06359e640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0640c84f0 0x7fb0640c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:25.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb062d9d640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb06415e0a0 0x7fb064150d30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb05400d6d0 tx=0x7fb05400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:25.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb04b7fe640 1 -- 172.21.15.59:0/2531099634 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb054017070 con 0x7fb06415e0a0 2024-06-29T15:38:25.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0640c8d70 con 0x7fb06415e0a0 2024-06-29T15:38:25.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb04b7fe640 1 -- 172.21.15.59:0/2531099634 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb054005330 con 0x7fb06415e0a0 2024-06-29T15:38:25.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb04b7fe640 1 -- 172.21.15.59:0/2531099634 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0540097c0 con 0x7fb06415e0a0 2024-06-29T15:38:25.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.312+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0640ccab0 con 0x7fb06415e0a0 2024-06-29T15:38:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.313+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0640b49d0 con 0x7fb06415e0a0 2024-06-29T15:38:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.314+0000 7fb04b7fe640 1 -- 172.21.15.59:0/2531099634 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 87) ==== 97912+0+0 (secure 0 0 0) 0x7fb054009b70 con 0x7fb06415e0a0 2024-06-29T15:38:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.315+0000 7fb04b7fe640 1 -- 172.21.15.59:0/2531099634 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(165..165 src has 1..165) ==== 6861+0+0 (secure 0 0 0) 0x7fb054097960 con 0x7fb06415e0a0 2024-06-29T15:38:25.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.319+0000 7fb04b7fe640 1 -- 172.21.15.59:0/2531099634 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb054060b60 con 0x7fb06415e0a0 2024-06-29T15:38:25.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.548+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb064151270 con 0x7fb06415e0a0 2024-06-29T15:38:25.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.549+0000 7fb04b7fe640 1 -- 172.21.15.59:0/2531099634 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5225 (secure 0 0 0) 0x7fb0540691a0 con 0x7fb06415e0a0 2024-06-29T15:38:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:25.549 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"btime":"2024-06-29T14:42:01:534262+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/507949731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":507949731},{"type":"v1","addr":"172.21.15.153:6837","nonce":507949731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":8812,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1668688055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1668688055},{"type":"v1","addr":"172.21.15.153:6836","nonce":1668688055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":9372,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3535921856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3535921856},{"type":"v1","addr":"172.21.15.59:6834","nonce":3535921856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.891192+0000","modified":"2024-06-29T14:42:00.924713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/178954889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":178954889},{"type":"v1","addr":"172.21.15.59:6835","nonce":178954889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9375,"qdb_cluster":[9375]},"id":9}]} 2024-06-29T15:38:25.550 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 80 2024-06-29T15:38:25.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb06415e0a0 msgr2=0x7fb064150d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb06415e0a0 0x7fb064150d30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb05400d6d0 tx=0x7fb05400db90 comp rx=0 tx=0).stop 2024-06-29T15:38:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 shutdown_connections 2024-06-29T15:38:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0640c84f0 0x7fb0640c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb064162770 0x7fb0640c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 --2- 172.21.15.59:0/2531099634 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb06415e0a0 0x7fb064150d30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 >> 172.21.15.59:0/2531099634 conn(0x7fb0640d45b0 msgr2=0x7fb0640b7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 shutdown_connections 2024-06-29T15:38:25.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.551+0000 7fb069074640 1 -- 172.21.15.59:0/2531099634 wait complete. 2024-06-29T15:38:25.564 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:25.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.681+0000 7f63d5be4640 1 -- 172.21.15.59:0/2033492577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d015ceb0 msgr2=0x7f63d00b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.681+0000 7f63d5be4640 1 --2- 172.21.15.59:0/2033492577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d015ceb0 0x7f63d00b56d0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f63c0005fd0 tx=0x7f63c002a540 comp rx=0 tx=0).stop 2024-06-29T15:38:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.681+0000 7f63d5be4640 1 -- 172.21.15.59:0/2033492577 shutdown_connections 2024-06-29T15:38:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.681+0000 7f63d5be4640 1 --2- 172.21.15.59:0/2033492577 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63d0141bd0 0x7f63d013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.681+0000 7f63d5be4640 1 --2- 172.21.15.59:0/2033492577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d015ceb0 0x7f63d00b56d0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.681+0000 7f63d5be4640 1 -- 172.21.15.59:0/2033492577 >> 172.21.15.59:0/2033492577 conn(0x7f63d00b6fb0 msgr2=0x7f63d00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.681+0000 7f63d5be4640 1 -- 172.21.15.59:0/2033492577 shutdown_connections 2024-06-29T15:38:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.681+0000 7f63d5be4640 1 -- 172.21.15.59:0/2033492577 wait complete. 2024-06-29T15:38:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.683+0000 7f63d5be4640 1 Processor -- start 2024-06-29T15:38:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.683+0000 7f63d5be4640 1 -- start start 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63d5be4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d013e9b0 0x7f63d0138780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63d5be4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63d015ceb0 0x7f63d0150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63d5be4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63d01580f0 0x7f63d0158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63d5be4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f63d00b2a00 con 0x7f63d013e9b0 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63d5be4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f63d00b2880 con 0x7f63d01580f0 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63d5be4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f63d00b2b80 con 0x7f63d015ceb0 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63cf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d013e9b0 0x7f63d0138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63cf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d013e9b0 0x7f63d0138780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59234/0 (socket says 172.21.15.59:59234) 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63cf7fe640 1 -- 172.21.15.59:0/1199321192 learned_addr learned my addr 172.21.15.59:0/1199321192 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63cffff640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63d01580f0 0x7f63d0158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.684+0000 7f63ceffd640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63d015ceb0 0x7f63d0150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63cf7fe640 1 -- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63d015ceb0 msgr2=0x7f63d0150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63cf7fe640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63d015ceb0 0x7f63d0150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63cf7fe640 1 -- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63d01580f0 msgr2=0x7f63d0158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63cf7fe640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63d01580f0 0x7f63d0158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63cf7fe640 1 -- 172.21.15.59:0/1199321192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d0158a40 con 0x7f63d013e9b0 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63cf7fe640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d013e9b0 0x7f63d0138780 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f63c0000c00 tx=0x7f63c0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63b7fff640 1 -- 172.21.15.59:0/1199321192 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c003e030 con 0x7f63d013e9b0 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63b7fff640 1 -- 172.21.15.59:0/1199321192 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63c0003e00 con 0x7f63d013e9b0 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d0151690 con 0x7f63d013e9b0 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63b7fff640 1 -- 172.21.15.59:0/1199321192 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0035040 con 0x7f63d013e9b0 2024-06-29T15:38:25.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.685+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d0151c50 con 0x7f63d013e9b0 2024-06-29T15:38:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.686+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63d00d16e0 con 0x7f63d013e9b0 2024-06-29T15:38:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.687+0000 7f63b7fff640 1 -- 172.21.15.59:0/1199321192 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 87) ==== 97912+0+0 (secure 0 0 0) 0x7f63c0041030 con 0x7f63d013e9b0 2024-06-29T15:38:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.687+0000 7f63b7fff640 1 -- 172.21.15.59:0/1199321192 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6861+0+0 (secure 0 0 0) 0x7f63c00b7d40 con 0x7f63d013e9b0 2024-06-29T15:38:25.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.692+0000 7f63b7fff640 1 -- 172.21.15.59:0/1199321192 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f63d00d16e0 con 0x7f63d013e9b0 2024-06-29T15:38:25.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.819+0000 7f63b7fff640 1 -- 172.21.15.59:0/1199321192 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 88) ==== 98039+0+0 (secure 0 0 0) 0x7f63c007c820 con 0x7f63d013e9b0 2024-06-29T15:38:25.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.820+0000 7f63b7fff640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f63bc07c160 0x7f63bc07e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.821+0000 7f63ceffd640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f63bc07c160 0x7f63bc07e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.825+0000 7f63ceffd640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f63bc07c160 0x7f63bc07e580 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f63d013c2d0 tx=0x7f63c4009040 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:25.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.890+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63d00b40f0 con 0x7f63d013e9b0 2024-06-29T15:38:25.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.891+0000 7f63b7fff640 1 -- 172.21.15.59:0/1199321192 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) ==== 74+0+18069 (secure 0 0 0) 0x7f63d0158a40 con 0x7f63d013e9b0 2024-06-29T15:38:25.891 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:25.892 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":165,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:24.808530+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.250305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.841623+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3579543037":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1931238090":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/2801509304":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1002441703":"2024-06-30T15:38:24.808471+0000","172.21.15.153:6832/3376352564":"2024-06-30T15:38:24.808471+0000","172.21.15.153:6833/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/2522719166":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/784386967":"2024-06-30T14:42:10.228325+0000","172.21.15.153:6832/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3466261848":"2024-06-30T14:42:10.228325+0000","172.21.15.153:6833/3376352564":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/568570198":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/1350737373":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1560085074":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/2676841282":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3008499766":"2024-06-30T14:42:10.228325+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:25.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f63bc07c160 msgr2=0x7f63bc07e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f63bc07c160 0x7f63bc07e580 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f63d013c2d0 tx=0x7f63c4009040 comp rx=0 tx=0).stop 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d013e9b0 msgr2=0x7f63d0138780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d013e9b0 0x7f63d0138780 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f63c0000c00 tx=0x7f63c0003910 comp rx=0 tx=0).stop 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 shutdown_connections 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63d01580f0 0x7f63d0158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63d015ceb0 0x7f63d0150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f63bc07c160 0x7f63bc07e580 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 --2- 172.21.15.59:0/1199321192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63d013e9b0 0x7f63d0138780 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 >> 172.21.15.59:0/1199321192 conn(0x7f63d00b6fb0 msgr2=0x7f63d015de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 shutdown_connections 2024-06-29T15:38:25.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:25.893+0000 7f63d5be4640 1 -- 172.21.15.59:0/1199321192 wait complete. 2024-06-29T15:38:25.905 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:38:25.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.022+0000 7fc560c66640 1 -- 172.21.15.59:0/3632652112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c144470 msgr2=0x7fc55c0b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.022+0000 7fc560c66640 1 --2- 172.21.15.59:0/3632652112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c144470 0x7fc55c0b1b00 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fc54c005fd0 tx=0x7fc54c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.022+0000 7fc560c66640 1 -- 172.21.15.59:0/3632652112 shutdown_connections 2024-06-29T15:38:26.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.022+0000 7fc560c66640 1 --2- 172.21.15.59:0/3632652112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc55c06ea10 0x7fc55c0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.022+0000 7fc560c66640 1 --2- 172.21.15.59:0/3632652112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c144470 0x7fc55c0b1b00 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.022+0000 7fc560c66640 1 -- 172.21.15.59:0/3632652112 >> 172.21.15.59:0/3632652112 conn(0x7fc55c0b06e0 msgr2=0x7fc55c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.022+0000 7fc560c66640 1 -- 172.21.15.59:0/3632652112 shutdown_connections 2024-06-29T15:38:26.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.022+0000 7fc560c66640 1 -- 172.21.15.59:0/3632652112 wait complete. 2024-06-29T15:38:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.024+0000 7fc560c66640 1 Processor -- start 2024-06-29T15:38:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.024+0000 7fc560c66640 1 -- start start 2024-06-29T15:38:26.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc560c66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc55c06ea10 0x7fc55c13bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc560c66640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0b2040 0x7fc55c156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc560c66640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc55c144470 0x7fc55c157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc560c66640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c0cfaa0 con 0x7fc55c0b2040 2024-06-29T15:38:26.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc560c66640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c0cf920 con 0x7fc55c06ea10 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc560c66640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc55c0cfc20 con 0x7fc55c144470 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc559d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0b2040 0x7fc55c156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc559d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0b2040 0x7fc55c156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59254/0 (socket says 172.21.15.59:59254) 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc559d9b640 1 -- 172.21.15.59:0/522723213 learned_addr learned my addr 172.21.15.59:0/522723213 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc55ad9d640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc55c144470 0x7fc55c157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.025+0000 7fc55a59c640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc55c06ea10 0x7fc55c13bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc559d9b640 1 -- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc55c144470 msgr2=0x7fc55c157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc559d9b640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc55c144470 0x7fc55c157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc559d9b640 1 -- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc55c06ea10 msgr2=0x7fc55c13bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc559d9b640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc55c06ea10 0x7fc55c13bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc559d9b640 1 -- 172.21.15.59:0/522723213 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55c0b04d0 con 0x7fc55c0b2040 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc55a59c640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc55c06ea10 0x7fc55c13bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc559d9b640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0b2040 0x7fc55c156ff0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fc54400c3a0 tx=0x7fc54400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc552ffd640 1 -- 172.21.15.59:0/522723213 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc544011790 con 0x7fc55c0b2040 2024-06-29T15:38:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc552ffd640 1 -- 172.21.15.59:0/522723213 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc544017070 con 0x7fc55c0b2040 2024-06-29T15:38:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55c15ab80 con 0x7fc55c0b2040 2024-06-29T15:38:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc552ffd640 1 -- 172.21.15.59:0/522723213 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc544011c80 con 0x7fc55c0b2040 2024-06-29T15:38:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.026+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc55c15a250 con 0x7fc55c0b2040 2024-06-29T15:38:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.027+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc55c0d0140 con 0x7fc55c0b2040 2024-06-29T15:38:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.028+0000 7fc552ffd640 1 -- 172.21.15.59:0/522723213 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 88) ==== 98039+0+0 (secure 0 0 0) 0x7fc54401c460 con 0x7fc55c0b2040 2024-06-29T15:38:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.029+0000 7fc552ffd640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc53c076120 0x7fc53c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.029+0000 7fc552ffd640 1 -- 172.21.15.59:0/522723213 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6861+0+0 (secure 0 0 0) 0x7fc544097730 con 0x7fc55c0b2040 2024-06-29T15:38:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.029+0000 7fc55a59c640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc53c076120 0x7fc53c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.029+0000 7fc55a59c640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc53c076120 0x7fc53c078630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc54c034000 tx=0x7fc54c004ab0 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.033+0000 7fc552ffd640 1 -- 172.21.15.59:0/522723213 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc55c0b2450 con 0x7fc55c0b2040 2024-06-29T15:38:26.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.262+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc55c159d00 con 0x7fc55c0b2040 2024-06-29T15:38:26.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.263+0000 7fc552ffd640 1 -- 172.21.15.59:0/522723213 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5225 (secure 0 0 0) 0x7fc544060ec0 con 0x7fc55c0b2040 2024-06-29T15:38:26.263 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:26.263 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"btime":"2024-06-29T14:42:01:534262+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/507949731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":507949731},{"type":"v1","addr":"172.21.15.153:6837","nonce":507949731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":8812,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1668688055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1668688055},{"type":"v1","addr":"172.21.15.153:6836","nonce":1668688055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":9372,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3535921856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3535921856},{"type":"v1","addr":"172.21.15.59:6834","nonce":3535921856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.891192+0000","modified":"2024-06-29T14:42:00.924713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/178954889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":178954889},{"type":"v1","addr":"172.21.15.59:6835","nonce":178954889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9375,"qdb_cluster":[9375]},"id":9}]} 2024-06-29T15:38:26.264 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 80 2024-06-29T15:38:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc53c076120 msgr2=0x7fc53c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc53c076120 0x7fc53c078630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc54c034000 tx=0x7fc54c004ab0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0b2040 msgr2=0x7fc55c156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0b2040 0x7fc55c156ff0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fc54400c3a0 tx=0x7fc54400c860 comp rx=0 tx=0).stop 2024-06-29T15:38:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 shutdown_connections 2024-06-29T15:38:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc55c144470 0x7fc55c157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc53c076120 0x7fc53c078630 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0b2040 0x7fc55c156ff0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 --2- 172.21.15.59:0/522723213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc55c06ea10 0x7fc55c13bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 >> 172.21.15.59:0/522723213 conn(0x7fc55c0b06e0 msgr2=0x7fc55c15e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 shutdown_connections 2024-06-29T15:38:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.265+0000 7fc560c66640 1 -- 172.21.15.59:0/522723213 wait complete. 2024-06-29T15:38:26.276 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:26.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.393+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/3486001211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b415c2f0 msgr2=0x7ff4b415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.393+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/3486001211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b415c2f0 0x7ff4b415c700 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40b1c40 tx=0x7ff4a402aa30 comp rx=0 tx=0).stop 2024-06-29T15:38:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.393+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/3486001211 shutdown_connections 2024-06-29T15:38:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.393+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/3486001211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b415c2f0 0x7ff4b415c700 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.393+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/3486001211 >> v2:172.21.15.153:3300/0 conn(0x7ff4b40b3740 0x7ff4b40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.393+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/3486001211 >> 172.21.15.59:0/3486001211 conn(0x7ff4b406c950 msgr2=0x7ff4b4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.393+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/3486001211 shutdown_connections 2024-06-29T15:38:26.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.393+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/3486001211 wait complete. 2024-06-29T15:38:26.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.395+0000 7ff4bc3d2640 1 Processor -- start 2024-06-29T15:38:26.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.395+0000 7ff4bc3d2640 1 -- start start 2024-06-29T15:38:26.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.396+0000 7ff4bc3d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4b40b3740 0x7ff4b41405e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.396+0000 7ff4bc3d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4b415c2f0 0x7ff4b4153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.396+0000 7ff4bc3d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b41543e0 0x7ff4b4070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.396+0000 7ff4bc3d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b40b5fd0 con 0x7ff4b41543e0 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.396+0000 7ff4bc3d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b40b5e50 con 0x7ff4b415c2f0 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.396+0000 7ff4bc3d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b40b6150 con 0x7ff4b40b3740 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.396+0000 7ff4ba96f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b41543e0 0x7ff4b4070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba96f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b41543e0 0x7ff4b4070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59278/0 (socket says 172.21.15.59:59278) 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba96f640 1 -- 172.21.15.59:0/151560272 learned_addr learned my addr 172.21.15.59:0/151560272 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba16e640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4b40b3740 0x7ff4b41405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4b996d640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4b415c2f0 0x7ff4b4153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba96f640 1 -- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4b40b3740 msgr2=0x7ff4b41405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba96f640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4b40b3740 0x7ff4b41405e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba96f640 1 -- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4b415c2f0 msgr2=0x7ff4b4153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba96f640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4b415c2f0 0x7ff4b4153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba96f640 1 -- 172.21.15.59:0/151560272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b406df60 con 0x7ff4b41543e0 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4b996d640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4b415c2f0 0x7ff4b4153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4ba96f640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b41543e0 0x7ff4b4070660 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7ff4a402b020 tx=0x7ff4a4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4b2ffd640 1 -- 172.21.15.59:0/151560272 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4033050 con 0x7ff4b41543e0 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4b2ffd640 1 -- 172.21.15.59:0/151560272 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4a40045f0 con 0x7ff4b41543e0 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b406e240 con 0x7ff4b41543e0 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4b2ffd640 1 -- 172.21.15.59:0/151560272 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4038450 con 0x7ff4b41543e0 2024-06-29T15:38:26.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.397+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b406e800 con 0x7ff4b41543e0 2024-06-29T15:38:26.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.398+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b406dcb0 con 0x7ff4b41543e0 2024-06-29T15:38:26.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.399+0000 7ff4b2ffd640 1 -- 172.21.15.59:0/151560272 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 88) ==== 98039+0+0 (secure 0 0 0) 0x7ff4a4036070 con 0x7ff4b41543e0 2024-06-29T15:38:26.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.400+0000 7ff4b2ffd640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7ff4a0076150 0x7ff4a0078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.400+0000 7ff4b2ffd640 1 -- 172.21.15.59:0/151560272 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6861+0+0 (secure 0 0 0) 0x7ff4a40b8ce0 con 0x7ff4b41543e0 2024-06-29T15:38:26.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.400+0000 7ff4ba16e640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7ff4a0076150 0x7ff4a0078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.401+0000 7ff4ba16e640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7ff4a0076150 0x7ff4a0078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac006fd0 tx=0x7ff4ac009060 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.404+0000 7ff4b2ffd640 1 -- 172.21.15.59:0/151560272 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff4a40bd070 con 0x7ff4b41543e0 2024-06-29T15:38:26.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.632+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4b40b66f0 con 0x7ff4b41543e0 2024-06-29T15:38:26.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.632+0000 7ff4b2ffd640 1 -- 172.21.15.59:0/151560272 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5225 (secure 0 0 0) 0x7ff4a40b8090 con 0x7ff4b41543e0 2024-06-29T15:38:26.632 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:26.633 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"btime":"2024-06-29T14:42:01:534262+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/507949731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":507949731},{"type":"v1","addr":"172.21.15.153:6837","nonce":507949731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":8812,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1668688055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1668688055},{"type":"v1","addr":"172.21.15.153:6836","nonce":1668688055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":9372,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3535921856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3535921856},{"type":"v1","addr":"172.21.15.59:6834","nonce":3535921856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.891192+0000","modified":"2024-06-29T14:42:00.924713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/178954889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":178954889},{"type":"v1","addr":"172.21.15.59:6835","nonce":178954889}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9375,"qdb_cluster":[9375]},"id":9}]} 2024-06-29T15:38:26.633 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 80 2024-06-29T15:38:26.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7ff4a0076150 msgr2=0x7ff4a0078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7ff4a0076150 0x7ff4a0078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac006fd0 tx=0x7ff4ac009060 comp rx=0 tx=0).stop 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b41543e0 msgr2=0x7ff4b4070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b41543e0 0x7ff4b4070660 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7ff4a402b020 tx=0x7ff4a4003f90 comp rx=0 tx=0).stop 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 shutdown_connections 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7ff4a0076150 0x7ff4a0078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4b41543e0 0x7ff4b4070660 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4b415c2f0 0x7ff4b4153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 --2- 172.21.15.59:0/151560272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4b40b3740 0x7ff4b41405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 >> 172.21.15.59:0/151560272 conn(0x7ff4b406c950 msgr2=0x7ff4b4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 shutdown_connections 2024-06-29T15:38:26.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.634+0000 7ff4bc3d2640 1 -- 172.21.15.59:0/151560272 wait complete. 2024-06-29T15:38:26.645 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:26.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.763+0000 7f2e0417f640 1 -- 172.21.15.59:0/1164913002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc0ce950 msgr2=0x7f2dfc0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.763+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1164913002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc0ce950 0x7f2dfc0d2130 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f2df4005fd0 tx=0x7f2df402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.763+0000 7f2e0417f640 1 -- 172.21.15.59:0/1164913002 shutdown_connections 2024-06-29T15:38:26.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.763+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1164913002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc0ce950 0x7f2dfc0d2130 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.763+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1164913002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dfc08c190 0x7f2dfc0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.763+0000 7f2e0417f640 1 -- 172.21.15.59:0/1164913002 >> 172.21.15.59:0/1164913002 conn(0x7f2dfc0b5610 msgr2=0x7f2dfc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.763+0000 7f2e0417f640 1 -- 172.21.15.59:0/1164913002 shutdown_connections 2024-06-29T15:38:26.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.763+0000 7f2e0417f640 1 -- 172.21.15.59:0/1164913002 wait complete. 2024-06-29T15:38:26.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.765+0000 7f2e0417f640 1 Processor -- start 2024-06-29T15:38:26.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.765+0000 7f2e0417f640 1 -- start start 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e0417f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc08c190 0x7f2dfc0b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e0417f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dfc0ce950 0x7f2dfc0cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e0417f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dfc0cf5c0 0x7f2dfc1f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e0417f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc0b7fa0 con 0x7f2dfc08c190 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e0417f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc0b7e20 con 0x7f2dfc0ce950 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e0417f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2dfc0b8120 con 0x7f2dfc0cf5c0 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e01f1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc08c190 0x7f2dfc0b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e01f1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc08c190 0x7f2dfc0b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59306/0 (socket says 172.21.15.59:59306) 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e01f1b640 1 -- 172.21.15.59:0/1442596736 learned_addr learned my addr 172.21.15.59:0/1442596736 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:26.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e0271c640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dfc0cf5c0 0x7f2dfc1f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.766+0000 7f2e0171a640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dfc0ce950 0x7f2dfc0cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e01f1b640 1 -- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dfc0cf5c0 msgr2=0x7f2dfc1f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e01f1b640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dfc0cf5c0 0x7f2dfc1f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e01f1b640 1 -- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dfc0ce950 msgr2=0x7f2dfc0cbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e01f1b640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dfc0ce950 0x7f2dfc0cbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e01f1b640 1 -- 172.21.15.59:0/1442596736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc0b47d0 con 0x7f2dfc08c190 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e0171a640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dfc0ce950 0x7f2dfc0cbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e01f1b640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc08c190 0x7f2dfc0b4290 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f2df000d740 tx=0x7f2df000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2dee7fc640 1 -- 172.21.15.59:0/1442596736 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0017070 con 0x7f2dfc08c190 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2dee7fc640 1 -- 172.21.15.59:0/1442596736 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2df0009cc0 con 0x7f2dfc08c190 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dfc0cca80 con 0x7f2dfc08c190 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2dee7fc640 1 -- 172.21.15.59:0/1442596736 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df00106c0 con 0x7f2dfc08c190 2024-06-29T15:38:26.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.767+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dfc0cd040 con 0x7f2dfc08c190 2024-06-29T15:38:26.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.768+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dfc0cf080 con 0x7f2dfc08c190 2024-06-29T15:38:26.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.769+0000 7f2dee7fc640 1 -- 172.21.15.59:0/1442596736 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 88) ==== 98039+0+0 (secure 0 0 0) 0x7f2df001a070 con 0x7f2dfc08c190 2024-06-29T15:38:26.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.770+0000 7f2dee7fc640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2de0076120 0x7f2de0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.770+0000 7f2dee7fc640 1 -- 172.21.15.59:0/1442596736 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6861+0+0 (secure 0 0 0) 0x7f2df0097810 con 0x7f2dfc08c190 2024-06-29T15:38:26.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.770+0000 7f2e0171a640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2de0076120 0x7f2de0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.770+0000 7f2e0171a640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2de0076120 0x7f2de0078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0b8c70 tx=0x7f2df8009040 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.774+0000 7f2dee7fc640 1 -- 172.21.15.59:0/1442596736 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2df00609e0 con 0x7f2dfc08c190 2024-06-29T15:38:26.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.973+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2dfc0c8270 con 0x7f2dfc08c190 2024-06-29T15:38:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.974+0000 7f2dee7fc640 1 -- 172.21.15.59:0/1442596736 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) ==== 74+0+18069 (secure 0 0 0) 0x7f2df0064ce0 con 0x7f2dfc08c190 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":165,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:24.808530+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.250305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.841623+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3579543037":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1931238090":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/2801509304":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1002441703":"2024-06-30T15:38:24.808471+0000","172.21.15.153:6832/3376352564":"2024-06-30T15:38:24.808471+0000","172.21.15.153:6833/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/2522719166":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/784386967":"2024-06-30T14:42:10.228325+0000","172.21.15.153:6832/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3466261848":"2024-06-30T14:42:10.228325+0000","172.21.15.153:6833/3376352564":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/568570198":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/1350737373":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1560085074":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/2676841282":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3008499766":"2024-06-30T14:42:10.228325+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2de0076120 msgr2=0x7f2de0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2de0076120 0x7f2de0078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0b8c70 tx=0x7f2df8009040 comp rx=0 tx=0).stop 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc08c190 msgr2=0x7f2dfc0b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc08c190 0x7f2dfc0b4290 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f2df000d740 tx=0x7f2df000dc00 comp rx=0 tx=0).stop 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 shutdown_connections 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dfc0cf5c0 0x7f2dfc1f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dfc0ce950 0x7f2dfc0cbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2de0076120 0x7f2de0078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 --2- 172.21.15.59:0/1442596736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc08c190 0x7f2dfc0b4290 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.976+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 >> 172.21.15.59:0/1442596736 conn(0x7f2dfc0b5610 msgr2=0x7f2dfc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.977+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 shutdown_connections 2024-06-29T15:38:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:26.977+0000 7f2e0417f640 1 -- 172.21.15.59:0/1442596736 wait complete. 2024-06-29T15:38:26.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:38:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.108+0000 7fe37610a640 1 -- 172.21.15.59:0/4150150159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe370162190 msgr2=0x7fe3700ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.108+0000 7fe37610a640 1 --2- 172.21.15.59:0/4150150159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe370162190 0x7fe3700ce550 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fe358005fd0 tx=0x7fe35802a540 comp rx=0 tx=0).stop 2024-06-29T15:38:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.108+0000 7fe37610a640 1 -- 172.21.15.59:0/4150150159 shutdown_connections 2024-06-29T15:38:27.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.108+0000 7fe37610a640 1 --2- 172.21.15.59:0/4150150159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe37015e0a0 0x7fe3700b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.108+0000 7fe37610a640 1 --2- 172.21.15.59:0/4150150159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe370162190 0x7fe3700ce550 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.108+0000 7fe37610a640 1 -- 172.21.15.59:0/4150150159 >> 172.21.15.59:0/4150150159 conn(0x7fe3700d45b0 msgr2=0x7fe3700d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:27.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.108+0000 7fe37610a640 1 -- 172.21.15.59:0/4150150159 shutdown_connections 2024-06-29T15:38:27.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.108+0000 7fe37610a640 1 -- 172.21.15.59:0/4150150159 wait complete. 2024-06-29T15:38:27.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.110+0000 7fe37610a640 1 Processor -- start 2024-06-29T15:38:27.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.110+0000 7fe37610a640 1 -- start start 2024-06-29T15:38:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe37610a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe37015e0a0 0x7fe3701578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe37610a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe370162770 0x7fe37015e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe37610a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3700c98c0 0x7fe3700c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe37610a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700b42c0 con 0x7fe37015e0a0 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe37610a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700b4140 con 0x7fe3700c98c0 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe37610a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe3700b4440 con 0x7fe370162770 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe37015e0a0 0x7fe3701578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe37015e0a0 0x7fe3701578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59328/0 (socket says 172.21.15.59:59328) 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 -- 172.21.15.59:0/4133983101 learned_addr learned my addr 172.21.15.59:0/4133983101 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe367fff640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe370162770 0x7fe37015e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36ffff640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3700c98c0 0x7fe3700c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 -- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe370162770 msgr2=0x7fe37015e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe370162770 0x7fe37015e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 -- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3700c98c0 msgr2=0x7fe3700c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3700c98c0 0x7fe3700c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 -- 172.21.15.59:0/4133983101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37015a790 con 0x7fe37015e0a0 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36f7fe640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe37015e0a0 0x7fe3701578c0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fe36000c3a0 tx=0x7fe36000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36cff9640 1 -- 172.21.15.59:0/4133983101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360017070 con 0x7fe37015e0a0 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36cff9640 1 -- 172.21.15.59:0/4133983101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3600053c0 con 0x7fe37015e0a0 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3700ca430 con 0x7fe37015e0a0 2024-06-29T15:38:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe36cff9640 1 -- 172.21.15.59:0/4133983101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360003850 con 0x7fe37015e0a0 2024-06-29T15:38:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.112+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3700c8690 con 0x7fe37015e0a0 2024-06-29T15:38:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.114+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3700b49d0 con 0x7fe37015e0a0 2024-06-29T15:38:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.114+0000 7fe36cff9640 1 -- 172.21.15.59:0/4133983101 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 89) ==== 98247+0+0 (secure 0 0 0) 0x7fe36001a070 con 0x7fe37015e0a0 2024-06-29T15:38:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.115+0000 7fe36cff9640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fe350076390 0x7fe3500788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.115+0000 7fe36cff9640 1 -- 172.21.15.59:0/4133983101 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6861+0+0 (secure 0 0 0) 0x7fe360096eb0 con 0x7fe37015e0a0 2024-06-29T15:38:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.115+0000 7fe367fff640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fe350076390 0x7fe3500788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.116+0000 7fe367fff640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fe350076390 0x7fe3500788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe358033000 tx=0x7fe358004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.119+0000 7fe36cff9640 1 -- 172.21.15.59:0/4133983101 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe36005fd50 con 0x7fe37015e0a0 2024-06-29T15:38:27.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.350+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe37006dbc0 con 0x7fe37015e0a0 2024-06-29T15:38:27.350 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:38:27.351+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:38:27.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.353+0000 7fe36cff9640 1 -- 172.21.15.59:0/4133983101 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v81) ==== 161+0+0 (secure 0 0 0) 0x7fe360068390 con 0x7fe37015e0a0 2024-06-29T15:38:27.353 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:38:27.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fe350076390 msgr2=0x7fe3500788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fe350076390 0x7fe3500788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe358033000 tx=0x7fe358004730 comp rx=0 tx=0).stop 2024-06-29T15:38:27.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe37015e0a0 msgr2=0x7fe3701578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe37015e0a0 0x7fe3701578c0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fe36000c3a0 tx=0x7fe36000c860 comp rx=0 tx=0).stop 2024-06-29T15:38:27.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 shutdown_connections 2024-06-29T15:38:27.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3700c98c0 0x7fe3700c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe370162770 0x7fe37015e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fe350076390 0x7fe3500788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 --2- 172.21.15.59:0/4133983101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe37015e0a0 0x7fe3701578c0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 >> 172.21.15.59:0/4133983101 conn(0x7fe3700d45b0 msgr2=0x7fe3700b7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:27.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 shutdown_connections 2024-06-29T15:38:27.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.356+0000 7fe37610a640 1 -- 172.21.15.59:0/4133983101 wait complete. 2024-06-29T15:38:27.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:38:27.419 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:38:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.490+0000 7fb533131640 1 -- 172.21.15.59:0/165857731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 msgr2=0x7fb514004fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.490+0000 7fb533131640 1 --2- 172.21.15.59:0/165857731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 0x7fb514004fd0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fb52c15a800 tx=0x7fb514006940 comp rx=0 tx=0).stop 2024-06-29T15:38:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.490+0000 7fb533131640 1 -- 172.21.15.59:0/165857731 shutdown_connections 2024-06-29T15:38:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.490+0000 7fb533131640 1 --2- 172.21.15.59:0/165857731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 0x7fb514004fd0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.490+0000 7fb533131640 1 -- 172.21.15.59:0/165857731 >> 172.21.15.59:0/165857731 conn(0x7fb52c06cb30 msgr2=0x7fb52c06cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.490+0000 7fb533131640 1 -- 172.21.15.59:0/165857731 shutdown_connections 2024-06-29T15:38:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.490+0000 7fb533131640 1 -- 172.21.15.59:0/165857731 wait complete. 2024-06-29T15:38:27.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.492+0000 7fb533131640 1 Processor -- start 2024-06-29T15:38:27.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.492+0000 7fb533131640 1 -- start start 2024-06-29T15:38:27.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.493+0000 7fb533131640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 0x7fb52c15e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.493+0000 7fb533131640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c14da70 0x7fb52c142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.493+0000 7fb533131640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb52c154ab0 0x7fb52c154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.493+0000 7fb533131640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c06ff60 con 0x7fb514004b60 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.493+0000 7fb533131640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c1548e0 con 0x7fb52c14da70 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.493+0000 7fb533131640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb52c0700e0 con 0x7fb52c154ab0 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 0x7fb52c15e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 0x7fb52c15e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59366/0 (socket says 172.21.15.59:59366) 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 -- 172.21.15.59:0/3778672525 learned_addr learned my addr 172.21.15.59:0/3778672525 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb5316ce640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb52c154ab0 0x7fb52c154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb523fff640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c14da70 0x7fb52c142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 -- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb52c154ab0 msgr2=0x7fb52c154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb52c154ab0 0x7fb52c154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 -- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c14da70 msgr2=0x7fb52c142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c14da70 0x7fb52c142420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 -- 172.21.15.59:0/3778672525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c155400 con 0x7fb514004b60 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb523fff640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c14da70 0x7fb52c142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb530ecd640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 0x7fb52c15e350 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fb52802b070 tx=0x7fb528006cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb5217fa640 1 -- 172.21.15.59:0/3778672525 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528032030 con 0x7fb514004b60 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb5217fa640 1 -- 172.21.15.59:0/3778672525 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb528003900 con 0x7fb514004b60 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52c15f280 con 0x7fb514004b60 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb5217fa640 1 -- 172.21.15.59:0/3778672525 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528003cf0 con 0x7fb514004b60 2024-06-29T15:38:27.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.494+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52c15efa0 con 0x7fb514004b60 2024-06-29T15:38:27.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.495+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb52c0b5420 con 0x7fb514004b60 2024-06-29T15:38:27.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.496+0000 7fb5217fa640 1 -- 172.21.15.59:0/3778672525 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 89) ==== 98247+0+0 (secure 0 0 0) 0x7fb528035020 con 0x7fb514004b60 2024-06-29T15:38:27.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.496+0000 7fb5217fa640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb518076360 0x7fb518078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.497+0000 7fb5217fa640 1 -- 172.21.15.59:0/3778672525 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 6947+0+0 (secure 0 0 0) 0x7fb5280b8220 con 0x7fb514004b60 2024-06-29T15:38:27.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.497+0000 7fb523fff640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb518076360 0x7fb518078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.497+0000 7fb523fff640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb518076360 0x7fb518078870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb51c006fd0 tx=0x7fb51c006b10 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:27.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.501+0000 7fb5217fa640 1 -- 172.21.15.59:0/3778672525 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb528047030 con 0x7fb514004b60 2024-06-29T15:38:27.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:27.730+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb52c15d5a0 con 0x7fb514004b60 2024-06-29T15:38:28.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.355+0000 7fb5217fa640 1 -- 172.21.15.59:0/3778672525 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v82) ==== 104+0+0 (secure 0 0 0) 0x7fb52c155400 con 0x7fb514004b60 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb518076360 msgr2=0x7fb518078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb518076360 0x7fb518078870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb51c006fd0 tx=0x7fb51c006b10 comp rx=0 tx=0).stop 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 msgr2=0x7fb52c15e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 0x7fb52c15e350 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fb52802b070 tx=0x7fb528006cd0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 shutdown_connections 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb52c154ab0 0x7fb52c154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb52c14da70 0x7fb52c142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb518076360 0x7fb518078870 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 --2- 172.21.15.59:0/3778672525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb514004b60 0x7fb52c15e350 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.358+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 >> 172.21.15.59:0/3778672525 conn(0x7fb52c06cb30 msgr2=0x7fb52c1532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.359+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 shutdown_connections 2024-06-29T15:38:28.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.359+0000 7fb533131640 1 -- 172.21.15.59:0/3778672525 wait complete. 2024-06-29T15:38:28.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:38:28.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.500+0000 7f880daac640 1 -- 172.21.15.59:0/3360965708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080b3ed0 msgr2=0x7f88080b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.500+0000 7f880daac640 1 --2- 172.21.15.59:0/3360965708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080b3ed0 0x7f88080b42e0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f880808cdb0 tx=0x7f880002aac0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.500+0000 7f880daac640 1 -- 172.21.15.59:0/3360965708 shutdown_connections 2024-06-29T15:38:28.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.500+0000 7f880daac640 1 --2- 172.21.15.59:0/3360965708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080b3ed0 0x7f88080b42e0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.500+0000 7f880daac640 1 --2- 172.21.15.59:0/3360965708 >> v2:172.21.15.153:3300/0 conn(0x7f88080d2780 0x7f88081621f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.500+0000 7f880daac640 1 -- 172.21.15.59:0/3360965708 >> 172.21.15.59:0/3360965708 conn(0x7f88080d3e60 msgr2=0x7f88080d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:28.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.501+0000 7f880daac640 1 -- 172.21.15.59:0/3360965708 shutdown_connections 2024-06-29T15:38:28.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.501+0000 7f880daac640 1 -- 172.21.15.59:0/3360965708 wait complete. 2024-06-29T15:38:28.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.502+0000 7f880daac640 1 Processor -- start 2024-06-29T15:38:28.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.503+0000 7f880daac640 1 -- start start 2024-06-29T15:38:28.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f880daac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88080b3ed0 0x7f8808157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:28.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f880daac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080d2780 0x7f88080aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:28.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f880daac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f880815dff0 0x7f88080b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:28.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f880daac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f88080b4820 con 0x7f88080d2780 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f880daac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8808162940 con 0x7f880815dff0 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f880daac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f88080b49a0 con 0x7f88080b3ed0 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8806ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080d2780 0x7f88080aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8806ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080d2780 0x7f88080aa9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59392/0 (socket says 172.21.15.59:59392) 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8806ffd640 1 -- 172.21.15.59:0/2461372820 learned_addr learned my addr 172.21.15.59:0/2461372820 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f88077fe640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88080b3ed0 0x7f8808157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8807fff640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f880815dff0 0x7f88080b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8806ffd640 1 -- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88080b3ed0 msgr2=0x7f8808157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8806ffd640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88080b3ed0 0x7f8808157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8806ffd640 1 -- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f880815dff0 msgr2=0x7f88080b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8806ffd640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f880815dff0 0x7f88080b19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.504+0000 7f8806ffd640 1 -- 172.21.15.59:0/2461372820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88080ab8b0 con 0x7f88080d2780 2024-06-29T15:38:28.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.505+0000 7f8806ffd640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080d2780 0x7f88080aa9d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00d750 tx=0x7f87fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:28.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.505+0000 7f87effff640 1 -- 172.21.15.59:0/2461372820 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc017070 con 0x7f88080d2780 2024-06-29T15:38:28.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.505+0000 7f87effff640 1 -- 172.21.15.59:0/2461372820 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87fc004310 con 0x7f88080d2780 2024-06-29T15:38:28.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.505+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88080abb90 con 0x7f88080d2780 2024-06-29T15:38:28.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.505+0000 7f87effff640 1 -- 172.21.15.59:0/2461372820 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc00de60 con 0x7f88080d2780 2024-06-29T15:38:28.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.505+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88080ac150 con 0x7f88080d2780 2024-06-29T15:38:28.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.506+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88080b4e90 con 0x7f88080d2780 2024-06-29T15:38:28.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.507+0000 7f87effff640 1 -- 172.21.15.59:0/2461372820 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 89) ==== 98247+0+0 (secure 0 0 0) 0x7f87fc009c30 con 0x7f88080d2780 2024-06-29T15:38:28.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.507+0000 7f87effff640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f87f4076390 0x7f87f40788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:28.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.507+0000 7f87effff640 1 -- 172.21.15.59:0/2461372820 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 6947+0+0 (secure 0 0 0) 0x7f87fc097f70 con 0x7f88080d2780 2024-06-29T15:38:28.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.508+0000 7f88077fe640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f87f4076390 0x7f87f40788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:28.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.508+0000 7f88077fe640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f87f4076390 0x7f87f40788a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f87f8001720 tx=0x7f87f8009040 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:28.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.512+0000 7f87effff640 1 -- 172.21.15.59:0/2461372820 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f87fc020020 con 0x7f88080d2780 2024-06-29T15:38:28.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:28.705+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f88080b2c20 con 0x7f88080d2780 2024-06-29T15:38:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.362+0000 7f87effff640 1 -- 172.21.15.59:0/2461372820 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v167) ==== 181+0+0 (secure 0 0 0) 0x7f87fc0692e0 con 0x7f88080d2780 2024-06-29T15:38:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:38:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.366+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f87f4076390 msgr2=0x7f87f40788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.366+0000 7f880daac640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f87f4076390 0x7f87f40788a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f87f8001720 tx=0x7f87f8009040 comp rx=0 tx=0).stop 2024-06-29T15:38:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.367+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080d2780 msgr2=0x7f88080aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.367+0000 7f880daac640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080d2780 0x7f88080aa9d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00d750 tx=0x7f87fc00dc10 comp rx=0 tx=0).stop 2024-06-29T15:38:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.369+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 shutdown_connections 2024-06-29T15:38:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.369+0000 7f880daac640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f880815dff0 0x7f88080b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.369+0000 7f880daac640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f87f4076390 0x7f87f40788a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.369+0000 7f880daac640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88080d2780 0x7f88080aa9d0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.369+0000 7f880daac640 1 --2- 172.21.15.59:0/2461372820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f88080b3ed0 0x7f8808157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.369+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 >> 172.21.15.59:0/2461372820 conn(0x7f88080d3e60 msgr2=0x7f8808160850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.369+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 shutdown_connections 2024-06-29T15:38:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.369+0000 7f880daac640 1 -- 172.21.15.59:0/2461372820 wait complete. 2024-06-29T15:38:29.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 -- 172.21.15.59:0/1906595679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec15e740 msgr2=0x7fb7ec15eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/1906595679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec15e740 0x7fb7ec15eb50 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec06de60 tx=0x7fb7e002a560 comp rx=0 tx=0).stop 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 -- 172.21.15.59:0/1906595679 shutdown_connections 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/1906595679 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec06f180 0x7fb7ec06f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/1906595679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec15e740 0x7fb7ec15eb50 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/1906595679 >> v2:172.21.15.153:3300/0 conn(0x7fb7ec13dad0 0x7fb7ec15bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 -- 172.21.15.59:0/1906595679 >> 172.21.15.59:0/1906595679 conn(0x7fb7ec06cb30 msgr2=0x7fb7ec06cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 -- 172.21.15.59:0/1906595679 shutdown_connections 2024-06-29T15:38:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.516+0000 7fb7f24a4640 1 -- 172.21.15.59:0/1906595679 wait complete. 2024-06-29T15:38:29.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.518+0000 7fb7f24a4640 1 Processor -- start 2024-06-29T15:38:29.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.518+0000 7fb7f24a4640 1 -- start start 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.519+0000 7fb7f24a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec06f180 0x7fb7ec140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.519+0000 7fb7f24a4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec13dad0 0x7fb7ec071e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.519+0000 7fb7f24a4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec15e740 0x7fb7ec0723b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.519+0000 7fb7f24a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec0ce4d0 con 0x7fb7ec06f180 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.519+0000 7fb7f24a4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec0ce350 con 0x7fb7ec15e740 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.519+0000 7fb7f24a4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb7ec0ce650 con 0x7fb7ec13dad0 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec06f180 0x7fb7ec140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec06f180 0x7fb7ec140ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59414/0 (socket says 172.21.15.59:59414) 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 -- 172.21.15.59:0/652980329 learned_addr learned my addr 172.21.15.59:0/652980329 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7eb7fe640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec13dad0 0x7fb7ec071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7f0a41640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec15e740 0x7fb7ec0723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 -- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec13dad0 msgr2=0x7fb7ec071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec13dad0 0x7fb7ec071e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 -- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec15e740 msgr2=0x7fb7ec0723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec15e740 0x7fb7ec0723b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 -- 172.21.15.59:0/652980329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec141020 con 0x7fb7ec06f180 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7f0a41640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec15e740 0x7fb7ec0723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7ebfff640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec06f180 0x7fb7ec140ae0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00c970 tx=0x7fb7dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7e8ff9640 1 -- 172.21.15.59:0/652980329 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc007bb0 con 0x7fb7ec06f180 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7e8ff9640 1 -- 172.21.15.59:0/652980329 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7dc007e40 con 0x7fb7ec06f180 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ec13e4d0 con 0x7fb7ec06f180 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7e8ff9640 1 -- 172.21.15.59:0/652980329 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc005280 con 0x7fb7ec06f180 2024-06-29T15:38:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.520+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ec13ea90 con 0x7fb7ec06f180 2024-06-29T15:38:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.522+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ec15bb40 con 0x7fb7ec06f180 2024-06-29T15:38:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.522+0000 7fb7e8ff9640 1 -- 172.21.15.59:0/652980329 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fb7dc005420 con 0x7fb7ec06f180 2024-06-29T15:38:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.522+0000 7fb7e8ff9640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb7d80763f0 0x7fb7d8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.523+0000 7fb7e8ff9640 1 -- 172.21.15.59:0/652980329 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 6509+0+0 (secure 0 0 0) 0x7fb7dc0980e0 con 0x7fb7ec06f180 2024-06-29T15:38:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.523+0000 7fb7eb7fe640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb7d80763f0 0x7fb7d8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.523+0000 7fb7eb7fe640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb7d80763f0 0x7fb7d8078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0033000 tx=0x7fb7e0004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:29.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.527+0000 7fb7e8ff9640 1 -- 172.21.15.59:0/652980329 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7ec141020 con 0x7fb7ec06f180 2024-06-29T15:38:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.723+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb7ec139c00 con 0x7fb7ec06f180 2024-06-29T15:38:29.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.993+0000 7fb7e8ff9640 1 -- 172.21.15.59:0/652980329 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v168) ==== 169+0+0 (secure 0 0 0) 0x7fb7dc014030 con 0x7fb7ec06f180 2024-06-29T15:38:29.993 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T15:38:29.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb7d80763f0 msgr2=0x7fb7d8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb7d80763f0 0x7fb7d8078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0033000 tx=0x7fb7e0004730 comp rx=0 tx=0).stop 2024-06-29T15:38:29.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec06f180 msgr2=0x7fb7ec140ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec06f180 0x7fb7ec140ae0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00c970 tx=0x7fb7dc00ce30 comp rx=0 tx=0).stop 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 shutdown_connections 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7ec15e740 0x7fb7ec0723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb7ec13dad0 0x7fb7ec071e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb7d80763f0 0x7fb7d8078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.996+0000 7fb7f24a4640 1 --2- 172.21.15.59:0/652980329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7ec06f180 0x7fb7ec140ae0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.997+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 >> 172.21.15.59:0/652980329 conn(0x7fb7ec06cb30 msgr2=0x7fb7ec0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.997+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 shutdown_connections 2024-06-29T15:38:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:29.997+0000 7fb7f24a4640 1 -- 172.21.15.59:0/652980329 wait complete. 2024-06-29T15:38:30.012 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:38:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.136+0000 7fc28352a640 1 -- 172.21.15.59:0/4235674696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc268004d40 msgr2=0x7fc2680051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.136+0000 7fc28352a640 1 --2- 172.21.15.59:0/4235674696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc268004d40 0x7fc2680051d0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0d3730 tx=0x7fc268005830 comp rx=0 tx=0).stop 2024-06-29T15:38:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.136+0000 7fc281ac7640 1 -- 172.21.15.59:0/4235674696 reap_dead start 2024-06-29T15:38:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.136+0000 7fc28352a640 1 -- 172.21.15.59:0/4235674696 shutdown_connections 2024-06-29T15:38:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.136+0000 7fc28352a640 1 -- 172.21.15.59:0/4235674696 >> 172.21.15.59:0/4235674696 conn(0x7fc27c0d45b0 msgr2=0x7fc27c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.137+0000 7fc28352a640 1 -- 172.21.15.59:0/4235674696 shutdown_connections 2024-06-29T15:38:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.137+0000 7fc28352a640 1 -- 172.21.15.59:0/4235674696 wait complete. 2024-06-29T15:38:30.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.138+0000 7fc28352a640 1 Processor -- start 2024-06-29T15:38:30.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.138+0000 7fc28352a640 1 -- start start 2024-06-29T15:38:30.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc28352a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc27c0c9f90 0x7fc27c0cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc28352a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc27c159770 0x7fc27c1f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc28352a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc27c1f8380 0x7fc27c1f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc28352a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc27c0b4170 con 0x7fc27c159770 2024-06-29T15:38:30.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc28352a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc27c0b3ff0 con 0x7fc27c0c9f90 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc28352a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc27c0b42f0 con 0x7fc27c1f8380 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc27c159770 0x7fc27c1f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc27c159770 0x7fc27c1f7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59440/0 (socket says 172.21.15.59:59440) 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 -- 172.21.15.59:0/777406545 learned_addr learned my addr 172.21.15.59:0/777406545 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc281ac7640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc27c1f8380 0x7fc27c1f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc2812c6640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc27c0c9f90 0x7fc27c0cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 -- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc27c1f8380 msgr2=0x7fc27c1f4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc27c1f8380 0x7fc27c1f4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 -- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc27c0c9f90 msgr2=0x7fc27c0cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc27c0c9f90 0x7fc27c0cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 -- 172.21.15.59:0/777406545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27c1f8790 con 0x7fc27c159770 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc280ac5640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc27c159770 0x7fc27c1f7e40 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fc26c003ed0 tx=0x7fc26c0016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.140+0000 7fc271ffb640 1 -- 172.21.15.59:0/777406545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c015070 con 0x7fc27c159770 2024-06-29T15:38:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.141+0000 7fc271ffb640 1 -- 172.21.15.59:0/777406545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc26c011840 con 0x7fc27c159770 2024-06-29T15:38:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.141+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc27c0ca3a0 con 0x7fc27c159770 2024-06-29T15:38:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.141+0000 7fc271ffb640 1 -- 172.21.15.59:0/777406545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c011c70 con 0x7fc27c159770 2024-06-29T15:38:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.141+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc27c0ccac0 con 0x7fc27c159770 2024-06-29T15:38:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.142+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc27c0b4880 con 0x7fc27c159770 2024-06-29T15:38:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.143+0000 7fc271ffb640 1 -- 172.21.15.59:0/777406545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fc26c018070 con 0x7fc27c159770 2024-06-29T15:38:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.143+0000 7fc271ffb640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc2600763f0 0x7fc260078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.143+0000 7fc271ffb640 1 -- 172.21.15.59:0/777406545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7fc26c098640 con 0x7fc27c159770 2024-06-29T15:38:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.143+0000 7fc2812c6640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc2600763f0 0x7fc260078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.144+0000 7fc2812c6640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc2600763f0 0x7fc260078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc278009c50 tx=0x7fc278005f90 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:30.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.148+0000 7fc271ffb640 1 -- 172.21.15.59:0/777406545 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc27c1f8790 con 0x7fc27c159770 2024-06-29T15:38:30.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.342+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fc27c15d450 con 0x7fc27c159770 2024-06-29T15:38:30.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.346+0000 7fc271ffb640 1 -- 172.21.15.59:0/777406545 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v59) ==== 71+0+0 (secure 0 0 0) 0x7fc26c061640 con 0x7fc27c159770 2024-06-29T15:38:30.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc2600763f0 msgr2=0x7fc260078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc2600763f0 0x7fc260078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc278009c50 tx=0x7fc278005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc27c159770 msgr2=0x7fc27c1f7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc27c159770 0x7fc27c1f7e40 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fc26c003ed0 tx=0x7fc26c0016b0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 shutdown_connections 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc27c1f8380 0x7fc27c1f4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fc2600763f0 0x7fc260078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc27c159770 0x7fc27c1f7e40 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 --2- 172.21.15.59:0/777406545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc27c0c9f90 0x7fc27c0cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 >> 172.21.15.59:0/777406545 conn(0x7fc27c0d45b0 msgr2=0x7fc27c0ceec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 shutdown_connections 2024-06-29T15:38:30.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.348+0000 7fc28352a640 1 -- 172.21.15.59:0/777406545 wait complete. 2024-06-29T15:38:30.359 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-06-29T15:38:30.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.476+0000 7f83e0ecc640 1 -- 172.21.15.59:0/2599164747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc1458a0 msgr2=0x7f83dc0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.476+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/2599164747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc1458a0 0x7f83dc0cf870 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f83cc005fd0 tx=0x7f83cc02a530 comp rx=0 tx=0).stop 2024-06-29T15:38:30.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.476+0000 7f83e0ecc640 1 -- 172.21.15.59:0/2599164747 shutdown_connections 2024-06-29T15:38:30.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.476+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/2599164747 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83dc0b3740 0x7f83dc0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.477+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/2599164747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc1458a0 0x7f83dc0cf870 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.477+0000 7f83e0ecc640 1 -- 172.21.15.59:0/2599164747 >> 172.21.15.59:0/2599164747 conn(0x7f83dc06c950 msgr2=0x7f83dc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:30.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.477+0000 7f83e0ecc640 1 -- 172.21.15.59:0/2599164747 shutdown_connections 2024-06-29T15:38:30.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.477+0000 7f83e0ecc640 1 -- 172.21.15.59:0/2599164747 wait complete. 2024-06-29T15:38:30.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.479+0000 7f83e0ecc640 1 Processor -- start 2024-06-29T15:38:30.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.479+0000 7f83e0ecc640 1 -- start start 2024-06-29T15:38:30.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83e0ecc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc0b3740 0x7f83dc15b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83e0ecc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83dc15c2f0 0x7f83dc0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83e0ecc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83dc1401b0 0x7f83dc0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83e0ecc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc160020 con 0x7f83dc0b3740 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83e0ecc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc15cbb0 con 0x7f83dc15c2f0 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83e0ecc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f83dc1601a0 con 0x7f83dc1401b0 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83da59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc0b3740 0x7f83dc15b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83da59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc0b3740 0x7f83dc15b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59448/0 (socket says 172.21.15.59:59448) 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83da59c640 1 -- 172.21.15.59:0/1818448756 learned_addr learned my addr 172.21.15.59:0/1818448756 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83d9d9b640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83dc15c2f0 0x7f83dc0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83dad9d640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83dc1401b0 0x7f83dc0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83da59c640 1 -- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83dc1401b0 msgr2=0x7f83dc0b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83da59c640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83dc1401b0 0x7f83dc0b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83da59c640 1 -- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83dc15c2f0 msgr2=0x7f83dc0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83da59c640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83dc15c2f0 0x7f83dc0b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83da59c640 1 -- 172.21.15.59:0/1818448756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc0af240 con 0x7f83dc0b3740 2024-06-29T15:38:30.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.480+0000 7f83d9d9b640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83dc15c2f0 0x7f83dc0b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:30.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.481+0000 7f83da59c640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc0b3740 0x7f83dc15b5e0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:30.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.481+0000 7f83caffd640 1 -- 172.21.15.59:0/1818448756 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc03e030 con 0x7f83dc0b3740 2024-06-29T15:38:30.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.481+0000 7f83caffd640 1 -- 172.21.15.59:0/1818448756 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83cc003e60 con 0x7f83dc0b3740 2024-06-29T15:38:30.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.481+0000 7f83caffd640 1 -- 172.21.15.59:0/1818448756 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc035040 con 0x7f83dc0b3740 2024-06-29T15:38:30.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.481+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83dc0af520 con 0x7f83dc0b3740 2024-06-29T15:38:30.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.481+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83dc0afae0 con 0x7f83dc0b3740 2024-06-29T15:38:30.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.482+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83dc1606c0 con 0x7f83dc0b3740 2024-06-29T15:38:30.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.483+0000 7f83caffd640 1 -- 172.21.15.59:0/1818448756 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f83cc041030 con 0x7f83dc0b3740 2024-06-29T15:38:30.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.483+0000 7f83caffd640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f83bc076420 0x7f83bc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.483+0000 7f83caffd640 1 -- 172.21.15.59:0/1818448756 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7f83cc0b8290 con 0x7f83dc0b3740 2024-06-29T15:38:30.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.483+0000 7f83d9d9b640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f83bc076420 0x7f83bc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.484+0000 7f83d9d9b640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f83bc076420 0x7f83bc078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f83d0005fd0 tx=0x7f83d0005e20 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:30.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.488+0000 7f83caffd640 1 -- 172.21.15.59:0/1818448756 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f83cc0818e0 con 0x7f83dc0b3740 2024-06-29T15:38:30.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:30.769+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) -- 0x7f83dc0b6ab0 con 0x7f83dc0b3740 2024-06-29T15:38:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.083+0000 7f83caffd640 1 -- 172.21.15.59:0/1818448756 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2500) ==== 178+0+0 (secure 0 0 0) 0x7f83cc0811e0 con 0x7f83dc0b3740 2024-06-29T15:38:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f83bc076420 msgr2=0x7f83bc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f83bc076420 0x7f83bc078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f83d0005fd0 tx=0x7f83d0005e20 comp rx=0 tx=0).stop 2024-06-29T15:38:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc0b3740 msgr2=0x7f83dc15b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc0b3740 0x7f83dc15b5e0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc003970 comp rx=0 tx=0).stop 2024-06-29T15:38:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 shutdown_connections 2024-06-29T15:38:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f83dc1401b0 0x7f83dc0b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83dc15c2f0 0x7f83dc0b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f83bc076420 0x7f83bc078930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 --2- 172.21.15.59:0/1818448756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83dc0b3740 0x7f83dc15b5e0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 >> 172.21.15.59:0/1818448756 conn(0x7f83dc06c950 msgr2=0x7f83dc0d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 shutdown_connections 2024-06-29T15:38:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.085+0000 7f83e0ecc640 1 -- 172.21.15.59:0/1818448756 wait complete. 2024-06-29T15:38:31.097 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-06-29T15:38:31.097 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-06-29T15:38:31.097 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T15:38:31.098 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:31.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.216+0000 7fb46979a640 1 -- 172.21.15.59:0/3369510117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb46408c190 msgr2=0x7fb4640cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.216+0000 7fb46979a640 1 --2- 172.21.15.59:0/3369510117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb46408c190 0x7fb4640cf080 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009830 tx=0x7fb45c02a4d0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.216+0000 7fb46979a640 1 -- 172.21.15.59:0/3369510117 shutdown_connections 2024-06-29T15:38:31.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.216+0000 7fb46979a640 1 --2- 172.21.15.59:0/3369510117 >> v2:172.21.15.153:3300/0 conn(0x7fb4640ce950 0x7fb4640d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.216+0000 7fb46979a640 1 --2- 172.21.15.59:0/3369510117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb46408c190 0x7fb4640cf080 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.216+0000 7fb46979a640 1 -- 172.21.15.59:0/3369510117 >> 172.21.15.59:0/3369510117 conn(0x7fb4640b5610 msgr2=0x7fb4640b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.216+0000 7fb46979a640 1 -- 172.21.15.59:0/3369510117 shutdown_connections 2024-06-29T15:38:31.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.216+0000 7fb46979a640 1 -- 172.21.15.59:0/3369510117 wait complete. 2024-06-29T15:38:31.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.218+0000 7fb46979a640 1 Processor -- start 2024-06-29T15:38:31.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.218+0000 7fb46979a640 1 -- start start 2024-06-29T15:38:31.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb46979a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb46408c190 0x7fb4641e9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb46979a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4640ce950 0x7fb4641f8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb46979a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4640cf5c0 0x7fb4641f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb46979a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb4640b7f60 con 0x7fb4640ce950 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb46979a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4640b7de0 con 0x7fb4640cf5c0 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb46979a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb4640b80e0 con 0x7fb46408c190 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb4627fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4640ce950 0x7fb4641f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb4627fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4640ce950 0x7fb4641f8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59470/0 (socket says 172.21.15.59:59470) 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb4627fc640 1 -- 172.21.15.59:0/2327868984 learned_addr learned my addr 172.21.15.59:0/2327868984 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb4637fe640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4640cf5c0 0x7fb4641f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.219+0000 7fb462ffd640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb46408c190 0x7fb4641e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb4627fc640 1 -- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb46408c190 msgr2=0x7fb4641e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb4627fc640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb46408c190 0x7fb4641e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb4627fc640 1 -- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4640cf5c0 msgr2=0x7fb4641f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb4627fc640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4640cf5c0 0x7fb4641f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb4627fc640 1 -- 172.21.15.59:0/2327868984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb464158b00 con 0x7fb4640ce950 2024-06-29T15:38:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb4627fc640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4640ce950 0x7fb4641f8980 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00c3c0 tx=0x7fb44c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb45b7fe640 1 -- 172.21.15.59:0/2327868984 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c017070 con 0x7fb4640ce950 2024-06-29T15:38:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb45b7fe640 1 -- 172.21.15.59:0/2327868984 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb44c0035f0 con 0x7fb4640ce950 2024-06-29T15:38:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb464155990 con 0x7fb4640ce950 2024-06-29T15:38:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb45b7fe640 1 -- 172.21.15.59:0/2327868984 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c003a20 con 0x7fb4640ce950 2024-06-29T15:38:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.220+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb464155c50 con 0x7fb4640ce950 2024-06-29T15:38:31.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.221+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4640cf080 con 0x7fb4640ce950 2024-06-29T15:38:31.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.222+0000 7fb45b7fe640 1 -- 172.21.15.59:0/2327868984 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fb44c01a070 con 0x7fb4640ce950 2024-06-29T15:38:31.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.222+0000 7fb45b7fe640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb444076420 0x7fb444078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.222+0000 7fb45b7fe640 1 -- 172.21.15.59:0/2327868984 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7fb44c096de0 con 0x7fb4640ce950 2024-06-29T15:38:31.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.223+0000 7fb462ffd640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb444076420 0x7fb444078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.223+0000 7fb462ffd640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb444076420 0x7fb444078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb45c034000 tx=0x7fb45c004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.227+0000 7fb45b7fe640 1 -- 172.21.15.59:0/2327868984 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb44c05fd50 con 0x7fb4640ce950 2024-06-29T15:38:31.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.421+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4640b4d80 con 0x7fb4640ce950 2024-06-29T15:38:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.422+0000 7fb45b7fe640 1 -- 172.21.15.59:0/2327868984 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) ==== 74+0+14445 (secure 0 0 0) 0x7fb44c064050 con 0x7fb4640ce950 2024-06-29T15:38:31.422 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:31.422 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":168,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:29.991263+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6833/178954889":"2024-06-30T15:38:27.351157+0000","172.21.15.153:0/3579543037":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1931238090":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/2801509304":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1002441703":"2024-06-30T15:38:24.808471+0000","172.21.15.153:6832/3376352564":"2024-06-30T15:38:24.808471+0000","172.21.15.153:6833/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.59:6835/178954889":"2024-06-30T15:38:27.351157+0000","172.21.15.153:0/2522719166":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/784386967":"2024-06-30T14:42:10.228325+0000","172.21.15.153:6832/3378196672":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3466261848":"2024-06-30T14:42:10.228325+0000","172.21.15.153:6833/3376352564":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/568570198":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/1350737373":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/1560085074":"2024-06-30T15:38:24.808471+0000","172.21.15.153:0/2676841282":"2024-06-30T14:42:10.228325+0000","172.21.15.153:0/3008499766":"2024-06-30T14:42:10.228325+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb444076420 msgr2=0x7fb444078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb444076420 0x7fb444078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb45c034000 tx=0x7fb45c004730 comp rx=0 tx=0).stop 2024-06-29T15:38:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4640ce950 msgr2=0x7fb4641f8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4640ce950 0x7fb4641f8980 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00c3c0 tx=0x7fb44c00c880 comp rx=0 tx=0).stop 2024-06-29T15:38:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 shutdown_connections 2024-06-29T15:38:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4640cf5c0 0x7fb4641f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb444076420 0x7fb444078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4640ce950 0x7fb4641f8980 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 --2- 172.21.15.59:0/2327868984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb46408c190 0x7fb4641e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 >> 172.21.15.59:0/2327868984 conn(0x7fb4640b5610 msgr2=0x7fb4640686b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.424+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 shutdown_connections 2024-06-29T15:38:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.425+0000 7fb46979a640 1 -- 172.21.15.59:0/2327868984 wait complete. 2024-06-29T15:38:31.436 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:31.436 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.556+0000 7f68ff173640 1 -- 172.21.15.59:0/4124600627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162190 msgr2=0x7f68f80b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.556+0000 7f68ff173640 1 --2- 172.21.15.59:0/4124600627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162190 0x7f68f80b38e0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f68f0005fd0 tx=0x7f68f002a810 comp rx=0 tx=0).stop 2024-06-29T15:38:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.556+0000 7f68ff173640 1 -- 172.21.15.59:0/4124600627 shutdown_connections 2024-06-29T15:38:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.556+0000 7f68ff173640 1 --2- 172.21.15.59:0/4124600627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162190 0x7f68f80b38e0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.556+0000 7f68ff173640 1 --2- 172.21.15.59:0/4124600627 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f8162770 0x7f68f80ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.556+0000 7f68ff173640 1 -- 172.21.15.59:0/4124600627 >> 172.21.15.59:0/4124600627 conn(0x7f68f80d45b0 msgr2=0x7f68f80d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.556+0000 7f68ff173640 1 -- 172.21.15.59:0/4124600627 shutdown_connections 2024-06-29T15:38:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.556+0000 7f68ff173640 1 -- 172.21.15.59:0/4124600627 wait complete. 2024-06-29T15:38:31.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.558+0000 7f68ff173640 1 Processor -- start 2024-06-29T15:38:31.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.558+0000 7f68ff173640 1 -- start start 2024-06-29T15:38:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68ff173640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f815e0a0 0x7f68f8161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68ff173640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162770 0x7f68f815ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68ff173640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f80ca530 0x7f68f815f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68ff173640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f68f80b42f0 con 0x7f68f8162770 2024-06-29T15:38:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68ff173640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f68f80b4170 con 0x7f68f815e0a0 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68ff173640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f68f80b4470 con 0x7f68f80ca530 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162770 0x7f68f815ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162770 0x7f68f815ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59480/0 (socket says 172.21.15.59:59480) 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 -- 172.21.15.59:0/909148165 learned_addr learned my addr 172.21.15.59:0/909148165 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68fcf0f640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f815e0a0 0x7f68f8161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68fd710640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f80ca530 0x7f68f815f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 -- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f80ca530 msgr2=0x7f68f815f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f80ca530 0x7f68f815f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 -- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f815e0a0 msgr2=0x7f68f8161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f815e0a0 0x7f68f8161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 -- 172.21.15.59:0/909148165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f80c9b60 con 0x7f68f8162770 2024-06-29T15:38:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f7fff640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162770 0x7f68f815ecb0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00c3a0 tx=0x7f68ec00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f57fa640 1 -- 172.21.15.59:0/909148165 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec011790 con 0x7f68f8162770 2024-06-29T15:38:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68f57fa640 1 -- 172.21.15.59:0/909148165 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68ec017070 con 0x7f68f8162770 2024-06-29T15:38:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.560+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f80c9e40 con 0x7f68f8162770 2024-06-29T15:38:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.561+0000 7f68f57fa640 1 -- 172.21.15.59:0/909148165 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec011ce0 con 0x7f68f8162770 2024-06-29T15:38:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.561+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f80c7230 con 0x7f68f8162770 2024-06-29T15:38:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.561+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f8162430 con 0x7f68f8162770 2024-06-29T15:38:31.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.562+0000 7f68f57fa640 1 -- 172.21.15.59:0/909148165 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f68ec01c460 con 0x7f68f8162770 2024-06-29T15:38:31.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.563+0000 7f68f57fa640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f68e4076420 0x7f68e4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.563+0000 7f68f57fa640 1 -- 172.21.15.59:0/909148165 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7f68ec097690 con 0x7f68f8162770 2024-06-29T15:38:31.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.563+0000 7f68fcf0f640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f68e4076420 0x7f68e4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.564+0000 7f68fcf0f640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f68e4076420 0x7f68e4078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68e800bb50 tx=0x7f68e800b2e0 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.567+0000 7f68f57fa640 1 -- 172.21.15.59:0/909148165 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f68ec060710 con 0x7f68f8162770 2024-06-29T15:38:31.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.756+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68f80b4920 con 0x7f68f8162770 2024-06-29T15:38:31.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.757+0000 7f68f57fa640 1 -- 172.21.15.59:0/909148165 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) ==== 93+0+10 (secure 0 0 0) 0x7f68ec064a10 con 0x7f68f8162770 2024-06-29T15:38:31.757 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:38:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f68e4076420 msgr2=0x7f68e4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f68e4076420 0x7f68e4078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68e800bb50 tx=0x7f68e800b2e0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162770 msgr2=0x7f68f815ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162770 0x7f68f815ecb0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00c3a0 tx=0x7f68ec00c860 comp rx=0 tx=0).stop 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 shutdown_connections 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f68f80ca530 0x7f68f815f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f68e4076420 0x7f68e4078930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68f8162770 0x7f68f815ecb0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 --2- 172.21.15.59:0/909148165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68f815e0a0 0x7f68f8161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 >> 172.21.15.59:0/909148165 conn(0x7f68f80d45b0 msgr2=0x7f68f815bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 shutdown_connections 2024-06-29T15:38:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.759+0000 7f68ff173640 1 -- 172.21.15.59:0/909148165 wait complete. 2024-06-29T15:38:31.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-06-29T15:38:31.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.889+0000 7fabb3674640 1 -- 172.21.15.59:0/1761218339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabac141bd0 msgr2=0x7fabac13edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.889+0000 7fabb3674640 1 --2- 172.21.15.59:0/1761218339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabac141bd0 0x7fabac13edc0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7faba4005fd0 tx=0x7faba402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.889+0000 7fabb3674640 1 -- 172.21.15.59:0/1761218339 shutdown_connections 2024-06-29T15:38:31.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.889+0000 7fabb3674640 1 --2- 172.21.15.59:0/1761218339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabac141bd0 0x7fabac13edc0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.889+0000 7fabb3674640 1 -- 172.21.15.59:0/1761218339 >> 172.21.15.59:0/1761218339 conn(0x7fabac0b6fb0 msgr2=0x7fabac0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.889+0000 7fabb3674640 1 -- 172.21.15.59:0/1761218339 shutdown_connections 2024-06-29T15:38:31.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.889+0000 7fabb3674640 1 -- 172.21.15.59:0/1761218339 wait complete. 2024-06-29T15:38:31.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.891+0000 7fabb3674640 1 Processor -- start 2024-06-29T15:38:31.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.891+0000 7fabb3674640 1 -- start start 2024-06-29T15:38:31.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb3674640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabac13e9b0 0x7fabac0c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb3674640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabac15ceb0 0x7fabac15a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb3674640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabac13c040 0x7fabac15a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb3674640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fabac0b2900 con 0x7fabac13c040 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb3674640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fabac0b2780 con 0x7fabac13e9b0 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb3674640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fabac0b2a80 con 0x7fabac15ceb0 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1c11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabac13c040 0x7fabac15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1c11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabac13c040 0x7fabac15a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59490/0 (socket says 172.21.15.59:59490) 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1c11640 1 -- 172.21.15.59:0/2410551430 learned_addr learned my addr 172.21.15.59:0/2410551430 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1410640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabac13e9b0 0x7fabac0c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb0c0f640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabac15ceb0 0x7fabac15a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1c11640 1 -- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabac15ceb0 msgr2=0x7fabac15a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1c11640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabac15ceb0 0x7fabac15a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1c11640 1 -- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabac13e9b0 msgr2=0x7fabac0c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1c11640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabac13e9b0 0x7fabac0c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.892+0000 7fabb1c11640 1 -- 172.21.15.59:0/2410551430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabac158270 con 0x7fabac13c040 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.893+0000 7fabb1c11640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabac13c040 0x7fabac15a720 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7faba4009cd0 tx=0x7faba40316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.893+0000 7faba1ffb640 1 -- 172.21.15.59:0/2410551430 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba4031990 con 0x7fabac13c040 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.893+0000 7faba1ffb640 1 -- 172.21.15.59:0/2410551430 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faba4031c40 con 0x7fabac13c040 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.893+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabac158550 con 0x7fabac13c040 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.893+0000 7faba1ffb640 1 -- 172.21.15.59:0/2410551430 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba4033330 con 0x7fabac13c040 2024-06-29T15:38:31.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.893+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabac15b260 con 0x7fabac13c040 2024-06-29T15:38:31.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.894+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabac0c78d0 con 0x7fabac13c040 2024-06-29T15:38:31.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.895+0000 7faba1ffb640 1 -- 172.21.15.59:0/2410551430 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7faba40334d0 con 0x7fabac13c040 2024-06-29T15:38:31.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.895+0000 7faba1ffb640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fab900763f0 0x7fab90078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.895+0000 7faba1ffb640 1 -- 172.21.15.59:0/2410551430 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7faba40b8f30 con 0x7fabac13c040 2024-06-29T15:38:31.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.896+0000 7fabb1410640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fab900763f0 0x7fab90078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.896+0000 7fabb1410640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fab900763f0 0x7fab90078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faba8007fd0 tx=0x7faba8001370 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:31.900+0000 7faba1ffb640 1 -- 172.21.15.59:0/2410551430 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faba403f030 con 0x7fabac13c040 2024-06-29T15:38:32.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:32.164+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) -- 0x7fabac15b4c0 con 0x7fabac13c040 2024-06-29T15:38:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.091+0000 7faba1ffb640 1 -- 172.21.15.59:0/2410551430 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2502) ==== 177+0+0 (secure 0 0 0) 0x7faba4081e10 con 0x7fabac13c040 2024-06-29T15:38:33.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fab900763f0 msgr2=0x7fab90078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fab900763f0 0x7fab90078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faba8007fd0 tx=0x7faba8001370 comp rx=0 tx=0).stop 2024-06-29T15:38:33.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabac13c040 msgr2=0x7fabac15a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabac13c040 0x7fabac15a720 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7faba4009cd0 tx=0x7faba40316f0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 shutdown_connections 2024-06-29T15:38:33.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fab900763f0 0x7fab90078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabac13c040 0x7fabac15a720 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabac15ceb0 0x7fabac15a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 --2- 172.21.15.59:0/2410551430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabac13e9b0 0x7fabac0c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 >> 172.21.15.59:0/2410551430 conn(0x7fabac0b6fb0 msgr2=0x7fabac13dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 shutdown_connections 2024-06-29T15:38:33.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.094+0000 7fabb3674640 1 -- 172.21.15.59:0/2410551430 wait complete. 2024-06-29T15:38:33.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:38:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.233+0000 7f77c794a640 1 -- 172.21.15.59:0/2927352074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 msgr2=0x7f77c0161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.233+0000 7f77c794a640 1 --2- 172.21.15.59:0/2927352074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 0x7f77c0161a80 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f77c008c200 tx=0x7f77b402a560 comp rx=0 tx=0).stop 2024-06-29T15:38:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.233+0000 7f77c794a640 1 -- 172.21.15.59:0/2927352074 shutdown_connections 2024-06-29T15:38:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.233+0000 7f77c794a640 1 --2- 172.21.15.59:0/2927352074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 0x7f77c0161a80 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.233+0000 7f77c794a640 1 --2- 172.21.15.59:0/2927352074 >> v2:172.21.15.153:3300/0 conn(0x7f77c0162770 0x7f77c00ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.233+0000 7f77c794a640 1 -- 172.21.15.59:0/2927352074 >> 172.21.15.59:0/2927352074 conn(0x7f77c00d45b0 msgr2=0x7f77c00d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.233+0000 7f77c794a640 1 -- 172.21.15.59:0/2927352074 shutdown_connections 2024-06-29T15:38:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.233+0000 7f77c794a640 1 -- 172.21.15.59:0/2927352074 wait complete. 2024-06-29T15:38:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.235+0000 7f77c794a640 1 Processor -- start 2024-06-29T15:38:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.235+0000 7f77c794a640 1 -- start start 2024-06-29T15:38:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.236+0000 7f77c794a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 0x7f77c0160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c794a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77c0161fc0 0x7f77c006c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c794a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77c0162770 0x7f77c006c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c794a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f77c00d2d80 con 0x7f77c015e0a0 2024-06-29T15:38:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c794a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f77c00b4b00 con 0x7f77c0162770 2024-06-29T15:38:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c794a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f77c00d2f00 con 0x7f77c0161fc0 2024-06-29T15:38:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 0x7f77c0160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 0x7f77c0160ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57786/0 (socket says 172.21.15.59:57786) 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 -- 172.21.15.59:0/1435363692 learned_addr learned my addr 172.21.15.59:0/1435363692 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c4ee5640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77c0161fc0 0x7f77c006c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c5ee7640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77c0162770 0x7f77c006c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 -- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77c0161fc0 msgr2=0x7f77c006c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77c0161fc0 0x7f77c006c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 -- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77c0162770 msgr2=0x7f77c006c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77c0162770 0x7f77c006c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 -- 172.21.15.59:0/1435363692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c01f4d80 con 0x7f77c015e0a0 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c5ee7640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77c0162770 0x7f77c006c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c56e6640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 0x7f77c0160ee0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f77b800e790 tx=0x7f77b800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77bdffb640 1 -- 172.21.15.59:0/1435363692 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b800ca60 con 0x7f77c015e0a0 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77bdffb640 1 -- 172.21.15.59:0/1435363692 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77b800ccf0 con 0x7f77c015e0a0 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77bdffb640 1 -- 172.21.15.59:0/1435363692 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8010690 con 0x7f77c015e0a0 2024-06-29T15:38:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.237+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c01f5060 con 0x7f77c015e0a0 2024-06-29T15:38:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.238+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c01f5620 con 0x7f77c015e0a0 2024-06-29T15:38:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.239+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c00d3420 con 0x7f77c015e0a0 2024-06-29T15:38:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.239+0000 7f77bdffb640 1 -- 172.21.15.59:0/1435363692 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f77b8010830 con 0x7f77c015e0a0 2024-06-29T15:38:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.240+0000 7f77bdffb640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f77ac076420 0x7f77ac078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.240+0000 7f77bdffb640 1 -- 172.21.15.59:0/1435363692 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7f77b8098430 con 0x7f77c015e0a0 2024-06-29T15:38:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.240+0000 7f77c4ee5640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f77ac076420 0x7f77ac078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.241+0000 7f77c4ee5640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f77ac076420 0x7f77ac078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f77b4033000 tx=0x7f77b4004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:33.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.244+0000 7f77bdffb640 1 -- 172.21.15.59:0/1435363692 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f77b8061310 con 0x7f77c015e0a0 2024-06-29T15:38:33.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.435+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f77c0075b40 con 0x7f77c015e0a0 2024-06-29T15:38:33.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.436+0000 7f77bdffb640 1 -- 172.21.15.59:0/1435363692 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v59) ==== 59+0+1998 (secure 0 0 0) 0x7f77b8065610 con 0x7f77c015e0a0 2024-06-29T15:38:33.436 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:38:33.436 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:38:33.436 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:38:33.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:38:33.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T15:38:33.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T15:38:33.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:38:33.437 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T15:38:33.438 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:38:33.438 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:38:33.438 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:38:33.438 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T15:38:33.438 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T15:38:33.438 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T15:38:33.438 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:38:33.438 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:38:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f77ac076420 msgr2=0x7f77ac078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f77ac076420 0x7f77ac078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f77b4033000 tx=0x7f77b4004730 comp rx=0 tx=0).stop 2024-06-29T15:38:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 msgr2=0x7f77c0160ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 0x7f77c0160ee0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f77b800e790 tx=0x7f77b800ec50 comp rx=0 tx=0).stop 2024-06-29T15:38:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 shutdown_connections 2024-06-29T15:38:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77c0162770 0x7f77c006c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77c0161fc0 0x7f77c006c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f77ac076420 0x7f77ac078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 --2- 172.21.15.59:0/1435363692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77c015e0a0 0x7f77c0160ee0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 >> 172.21.15.59:0/1435363692 conn(0x7f77c00d45b0 msgr2=0x7f77c00682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 shutdown_connections 2024-06-29T15:38:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.438+0000 7f77c794a640 1 -- 172.21.15.59:0/1435363692 wait complete. 2024-06-29T15:38:33.449 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:38:33.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 -- 172.21.15.59:0/2002611358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 msgr2=0x7fabe00b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 --2- 172.21.15.59:0/2002611358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 0x7fabe00b56d0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fabe00b5590 tx=0x7fabd802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 -- 172.21.15.59:0/2002611358 shutdown_connections 2024-06-29T15:38:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 --2- 172.21.15.59:0/2002611358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabe01414c0 0x7fabe01418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 --2- 172.21.15.59:0/2002611358 >> v2:172.21.15.153:3300/0 conn(0x7fabe013e9b0 0x7fabe013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 --2- 172.21.15.59:0/2002611358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 0x7fabe00b56d0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 -- 172.21.15.59:0/2002611358 >> 172.21.15.59:0/2002611358 conn(0x7fabe00b6fb0 msgr2=0x7fabe00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 -- 172.21.15.59:0/2002611358 shutdown_connections 2024-06-29T15:38:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.570+0000 7fabe5962640 1 -- 172.21.15.59:0/2002611358 wait complete. 2024-06-29T15:38:33.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.572+0000 7fabe5962640 1 Processor -- start 2024-06-29T15:38:33.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.572+0000 7fabe5962640 1 -- start start 2024-06-29T15:38:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabe5962640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabe013e9b0 0x7fabe015af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabe5962640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01414c0 0x7fabe00cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabe5962640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 0x7fabe00cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabe5962640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fabe00b28c0 con 0x7fabe015ceb0 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabe5962640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fabe00b2740 con 0x7fabe01414c0 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabe5962640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fabe00b2a40 con 0x7fabe013e9b0 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 0x7fabe00cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 0x7fabe00cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57796/0 (socket says 172.21.15.59:57796) 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.573+0000 7fabdf7fe640 1 -- 172.21.15.59:0/271027610 learned_addr learned my addr 172.21.15.59:0/271027610 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabde7fc640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01414c0 0x7fabe00cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabdeffd640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabe013e9b0 0x7fabe015af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabdf7fe640 1 -- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabe013e9b0 msgr2=0x7fabe015af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabdf7fe640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabe013e9b0 0x7fabe015af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabdf7fe640 1 -- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01414c0 msgr2=0x7fabe00cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabdf7fe640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01414c0 0x7fabe00cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabdf7fe640 1 -- 172.21.15.59:0/271027610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe00cb230 con 0x7fabe015ceb0 2024-06-29T15:38:33.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabdf7fe640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 0x7fabe00cd500 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fabd000e990 tx=0x7fabd000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabd7fff640 1 -- 172.21.15.59:0/271027610 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0004d20 con 0x7fabe015ceb0 2024-06-29T15:38:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabd7fff640 1 -- 172.21.15.59:0/271027610 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabd000cb00 con 0x7fabe015ceb0 2024-06-29T15:38:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe00c8d50 con 0x7fabe015ceb0 2024-06-29T15:38:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabd7fff640 1 -- 172.21.15.59:0/271027610 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0005020 con 0x7fabe015ceb0 2024-06-29T15:38:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.574+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe00c9310 con 0x7fabe015ceb0 2024-06-29T15:38:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.575+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe00b2f60 con 0x7fabe015ceb0 2024-06-29T15:38:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.576+0000 7fabd7fff640 1 -- 172.21.15.59:0/271027610 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fabd00051c0 con 0x7fabe015ceb0 2024-06-29T15:38:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.576+0000 7fabd7fff640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fabc0076420 0x7fabc0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.576+0000 7fabd7fff640 1 -- 172.21.15.59:0/271027610 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7fabd0098370 con 0x7fabe015ceb0 2024-06-29T15:38:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.577+0000 7fabdeffd640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fabc0076420 0x7fabc0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.577+0000 7fabdeffd640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fabc0076420 0x7fabc0078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fabd8033000 tx=0x7fabd8004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.581+0000 7fabd7fff640 1 -- 172.21.15.59:0/271027610 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fabe00cb230 con 0x7fabe015ceb0 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.774+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fabe00b57e0 con 0x7fabe015ceb0 2024-06-29T15:38:33.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.778+0000 7fabd7fff640 1 -- 172.21.15.59:0/271027610 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v60)=0 v60) ==== 122+0+0 (secure 0 0 0) 0x7fabd0016030 con 0x7fabe015ceb0 2024-06-29T15:38:33.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fabc0076420 msgr2=0x7fabc0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fabc0076420 0x7fabc0078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fabd8033000 tx=0x7fabd8004730 comp rx=0 tx=0).stop 2024-06-29T15:38:33.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 msgr2=0x7fabe00cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 0x7fabe00cd500 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fabd000e990 tx=0x7fabd000ee50 comp rx=0 tx=0).stop 2024-06-29T15:38:33.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 shutdown_connections 2024-06-29T15:38:33.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fabc0076420 0x7fabc0078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe015ceb0 0x7fabe00cd500 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe01414c0 0x7fabe00cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 --2- 172.21.15.59:0/271027610 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fabe013e9b0 0x7fabe015af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 >> 172.21.15.59:0/271027610 conn(0x7fabe00b6fb0 msgr2=0x7fabe013e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 shutdown_connections 2024-06-29T15:38:33.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:33.780+0000 7fabe5962640 1 -- 172.21.15.59:0/271027610 wait complete. 2024-06-29T15:38:33.791 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:38:33.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:33.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:33.880 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:38:33.880 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:33.881 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:33.897 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:38:33.898 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:38:33.898 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:33.916 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:33.916 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:33.916 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:33.955 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:38:33.956 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:38:33.956 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:33.974 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:33.974 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:38:33.975 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:34.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.115+0000 7f436ff54640 1 -- 172.21.15.59:0/1961569427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4354004230 msgr2=0x7f4354004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.115+0000 7f436ff54640 1 --2- 172.21.15.59:0/1961569427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4354004230 0x7f4354004800 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f43680b2f50 tx=0x7f4354005830 comp rx=0 tx=0).stop 2024-06-29T15:38:34.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.115+0000 7f436e4f1640 1 -- 172.21.15.59:0/1961569427 reap_dead start 2024-06-29T15:38:34.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.115+0000 7f436ff54640 1 -- 172.21.15.59:0/1961569427 shutdown_connections 2024-06-29T15:38:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.115+0000 7f436ff54640 1 -- 172.21.15.59:0/1961569427 >> 172.21.15.59:0/1961569427 conn(0x7f4368073e10 msgr2=0x7f4368074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.115+0000 7f436ff54640 1 -- 172.21.15.59:0/1961569427 shutdown_connections 2024-06-29T15:38:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.115+0000 7f436ff54640 1 -- 172.21.15.59:0/1961569427 wait complete. 2024-06-29T15:38:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.117+0000 7f436ff54640 1 Processor -- start 2024-06-29T15:38:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.117+0000 7f436ff54640 1 -- start start 2024-06-29T15:38:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436ff54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4354004230 0x7f4368151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436ff54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43681f2900 0x7f43681f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436ff54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4368153a10 0x7f4368153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436ff54640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f436815c0c0 con 0x7f4368153a10 2024-06-29T15:38:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436ff54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f436815bf40 con 0x7f43681f2900 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436ff54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f436815c240 con 0x7f4354004230 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436e4f1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4368153a10 0x7f4368153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436e4f1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4368153a10 0x7f4368153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57806/0 (socket says 172.21.15.59:57806) 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436e4f1640 1 -- 172.21.15.59:0/2926225164 learned_addr learned my addr 172.21.15.59:0/2926225164 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436d4ef640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43681f2900 0x7f43681f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.118+0000 7f436dcf0640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4354004230 0x7f4368151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f436e4f1640 1 -- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4354004230 msgr2=0x7f4368151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f436e4f1640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4354004230 0x7f4368151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f436e4f1640 1 -- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43681f2900 msgr2=0x7f43681f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f436e4f1640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43681f2900 0x7f43681f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f436e4f1640 1 -- 172.21.15.59:0/2926225164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4368154360 con 0x7f4368153a10 2024-06-29T15:38:34.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f436e4f1640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4368153a10 0x7f4368153e20 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f43680b5470 tx=0x7f4360006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:34.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f435e7fc640 1 -- 172.21.15.59:0/2926225164 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360011900 con 0x7f4368153a10 2024-06-29T15:38:34.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f435e7fc640 1 -- 172.21.15.59:0/2926225164 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4360018070 con 0x7f4368153a10 2024-06-29T15:38:34.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f435e7fc640 1 -- 172.21.15.59:0/2926225164 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360011d40 con 0x7f4368153a10 2024-06-29T15:38:34.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43681523a0 con 0x7f4368153a10 2024-06-29T15:38:34.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.119+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4368152960 con 0x7f4368153a10 2024-06-29T15:38:34.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.120+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43680b4fe0 con 0x7f4368153a10 2024-06-29T15:38:34.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.121+0000 7f435e7fc640 1 -- 172.21.15.59:0/2926225164 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f436001d460 con 0x7f4368153a10 2024-06-29T15:38:34.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.121+0000 7f435e7fc640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f434c076420 0x7f434c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.122+0000 7f435e7fc640 1 -- 172.21.15.59:0/2926225164 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7f43600983c0 con 0x7f4368153a10 2024-06-29T15:38:34.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.122+0000 7f436dcf0640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f434c076420 0x7f434c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.122+0000 7f436dcf0640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f434c076420 0x7f434c078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4364009bb0 tx=0x7f4364005f90 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:34.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.126+0000 7f435e7fc640 1 -- 172.21.15.59:0/2926225164 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f43680b4fe0 con 0x7f4368153a10 2024-06-29T15:38:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.356+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43680d1520 con 0x7f4368153a10 2024-06-29T15:38:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.357+0000 7f435e7fc640 1 -- 172.21.15.59:0/2926225164 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+3914 (secure 0 0 0) 0x7f4368154360 con 0x7f4368153a10 2024-06-29T15:38:34.357 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:34.357 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":82,"btime":"2024-06-29T15:38:28:354010+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/507949731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":507949731},{"type":"v1","addr":"172.21.15.153:6837","nonce":507949731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":8812,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1668688055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1668688055},{"type":"v1","addr":"172.21.15.153:6836","nonce":1668688055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":9372,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3535921856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3535921856},{"type":"v1","addr":"172.21.15.59:6834","nonce":3535921856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76},{"gid":9813,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1953280627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1953280627},{"type":"v1","addr":"172.21.15.59:6835","nonce":1953280627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[]} 2024-06-29T15:38:34.358 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 82 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f434c076420 msgr2=0x7f434c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f434c076420 0x7f434c078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4364009bb0 tx=0x7f4364005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4368153a10 msgr2=0x7f4368153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4368153a10 0x7f4368153e20 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f43680b5470 tx=0x7f4360006430 comp rx=0 tx=0).stop 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 shutdown_connections 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f434c076420 0x7f434c078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4368153a10 0x7f4368153e20 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43681f2900 0x7f43681f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 --2- 172.21.15.59:0/2926225164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4354004230 0x7f4368151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 >> 172.21.15.59:0/2926225164 conn(0x7f4368073e10 msgr2=0x7f4368074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:34.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 shutdown_connections 2024-06-29T15:38:34.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.359+0000 7f436ff54640 1 -- 172.21.15.59:0/2926225164 wait complete. 2024-06-29T15:38:34.370 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:38:34.370 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:38:34.371 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:38:34.371 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:38:34.371 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:38:34.371 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:38:34.371 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:38:34.371 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:38:34.372 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:38:34.372 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:38:34.372 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:38:34.372 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:38:34.372 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:38:34.372 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:38:34.372 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:38:34.372 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:38:34.382 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:38:34.382 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:38:34.383 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:38:34.383 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:38:34.386 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:38:34.386 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:38:34.387 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:38:34.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:38:34.426 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:38:34.426 DEBUG:teuthology.parallel:result is None 2024-06-29T15:38:34.427 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:38:34.427 DEBUG:teuthology.parallel:result is None 2024-06-29T15:38:34.430 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:38:34.430 DEBUG:teuthology.parallel:result is None 2024-06-29T15:38:34.430 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:38:34.431 DEBUG:teuthology.parallel:result is None 2024-06-29T15:38:34.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:38:34.516 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:38:34.517 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:38:34.524 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:38:34.524 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T15:38:34.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.555+0000 7f6b81ece640 1 -- 172.21.15.59:0/3514256294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0b3ed0 msgr2=0x7f6b7c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.555+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3514256294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0b3ed0 0x7f6b7c15e8d0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009830 tx=0x7f6b7002a510 comp rx=0 tx=0).stop 2024-06-29T15:38:34.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.555+0000 7f6b81ece640 1 -- 172.21.15.59:0/3514256294 shutdown_connections 2024-06-29T15:38:34.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.555+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3514256294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0b3ed0 0x7f6b7c15e8d0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.555+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3514256294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b7c0d23c0 0x7f6b7c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.555+0000 7f6b81ece640 1 -- 172.21.15.59:0/3514256294 >> 172.21.15.59:0/3514256294 conn(0x7f6b7c0d3ee0 msgr2=0x7f6b7c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:34.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.555+0000 7f6b81ece640 1 -- 172.21.15.59:0/3514256294 shutdown_connections 2024-06-29T15:38:34.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.555+0000 7f6b81ece640 1 -- 172.21.15.59:0/3514256294 wait complete. 2024-06-29T15:38:34.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.557+0000 7f6b81ece640 1 Processor -- start 2024-06-29T15:38:34.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.557+0000 7f6b81ece640 1 -- start start 2024-06-29T15:38:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.558+0000 7f6b81ece640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c0b3ed0 0x7f6b7c152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b81ece640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0d23c0 0x7f6b7c14f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b81ece640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b7c15ee10 0x7f6b7c1e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b81ece640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c160240 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b81ece640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c1600c0 con 0x7f6b7c0b3ed0 2024-06-29T15:38:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b81ece640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7c1603c0 con 0x7f6b7c15ee10 2024-06-29T15:38:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0d23c0 0x7f6b7c14f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0d23c0 0x7f6b7c14f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57838/0 (socket says 172.21.15.59:57838) 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 -- 172.21.15.59:0/3801392849 learned_addr learned my addr 172.21.15.59:0/3801392849 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7b7fe640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c0b3ed0 0x7f6b7c152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7bfff640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b7c15ee10 0x7f6b7c1e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 -- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b7c15ee10 msgr2=0x7f6b7c1e8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b7c15ee10 0x7f6b7c1e8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 -- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c0b3ed0 msgr2=0x7f6b7c152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c0b3ed0 0x7f6b7c152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 -- 172.21.15.59:0/3801392849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7c1533f0 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7b7fe640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c0b3ed0 0x7f6b7c152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b7affd640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0d23c0 0x7f6b7c14f110 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009950 tx=0x7f6b70033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b63fff640 1 -- 172.21.15.59:0/3801392849 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70005c90 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b63fff640 1 -- 172.21.15.59:0/3801392849 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b70031e50 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.559+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7c153900 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.560+0000 7f6b63fff640 1 -- 172.21.15.59:0/3801392849 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70033290 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.560+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7c153ec0 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.561+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b7c153650 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.561+0000 7f6b63fff640 1 -- 172.21.15.59:0/3801392849 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f6b70033430 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.562+0000 7f6b63fff640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f6b68076420 0x7f6b68078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.562+0000 7f6b63fff640 1 -- 172.21.15.59:0/3801392849 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6095+0+0 (secure 0 0 0) 0x7f6b700b8eb0 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.562+0000 7f6b7b7fe640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f6b68076420 0x7f6b68078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.563+0000 7f6b7b7fe640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f6b68076420 0x7f6b68078930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0d2e40 tx=0x7f6b6c005f90 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:34.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.566+0000 7f6b63fff640 1 -- 172.21.15.59:0/3801392849 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b7003f030 con 0x7f6b7c0d23c0 2024-06-29T15:38:34.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:34.769+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6b7c15e8d0 con 0x7f6b7c0d23c0 2024-06-29T15:38:35.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.109+0000 7f6b63fff640 1 -- 172.21.15.59:0/3801392849 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v170) ==== 97+0+0 (secure 0 0 0) 0x7f6b700823f0 con 0x7f6b7c0d23c0 2024-06-29T15:38:35.110 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f6b68076420 msgr2=0x7f6b68078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f6b68076420 0x7f6b68078930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0d2e40 tx=0x7f6b6c005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0d23c0 msgr2=0x7f6b7c14f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0d23c0 0x7f6b7c14f110 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009950 tx=0x7f6b70033c20 comp rx=0 tx=0).stop 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 shutdown_connections 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6b7c15ee10 0x7f6b7c1e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f6b68076420 0x7f6b68078930 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b7c0d23c0 0x7f6b7c14f110 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 --2- 172.21.15.59:0/3801392849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7c0b3ed0 0x7f6b7c152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 >> 172.21.15.59:0/3801392849 conn(0x7f6b7c0d3ee0 msgr2=0x7f6b7c0ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 shutdown_connections 2024-06-29T15:38:35.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.112+0000 7f6b81ece640 1 -- 172.21.15.59:0/3801392849 wait complete. 2024-06-29T15:38:35.126 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:38:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.245+0000 7f94c3018640 1 -- 172.21.15.59:0/357899581 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94bc0cd9b0 msgr2=0x7f94bc15bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.245+0000 7f94c3018640 1 --2- 172.21.15.59:0/357899581 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94bc0cd9b0 0x7f94bc15bc80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009830 tx=0x7f94b802a510 comp rx=0 tx=0).stop 2024-06-29T15:38:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.245+0000 7f94c3018640 1 -- 172.21.15.59:0/357899581 shutdown_connections 2024-06-29T15:38:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.245+0000 7f94c3018640 1 --2- 172.21.15.59:0/357899581 >> v2:172.21.15.59:3300/0 conn(0x7f94bc162810 0x7f94bc164c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.245+0000 7f94c3018640 1 --2- 172.21.15.59:0/357899581 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94bc0cd9b0 0x7f94bc15bc80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.245+0000 7f94c3018640 1 -- 172.21.15.59:0/357899581 >> 172.21.15.59:0/357899581 conn(0x7f94bc0b24c0 msgr2=0x7f94bc0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.245+0000 7f94c3018640 1 -- 172.21.15.59:0/357899581 shutdown_connections 2024-06-29T15:38:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.245+0000 7f94c3018640 1 -- 172.21.15.59:0/357899581 wait complete. 2024-06-29T15:38:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.247+0000 7f94c3018640 1 Processor -- start 2024-06-29T15:38:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.247+0000 7f94c3018640 1 -- start start 2024-06-29T15:38:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c3018640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94bc06d2f0 0x7f94bc158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c3018640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94bc0cd9b0 0x7f94bc0aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c3018640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc162810 0x7f94bc073220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c3018640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1448f0 con 0x7f94bc162810 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c3018640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc144770 con 0x7f94bc0cd9b0 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c3018640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f94bc165170 con 0x7f94bc06d2f0 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c15b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc162810 0x7f94bc073220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c15b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc162810 0x7f94bc073220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57856/0 (socket says 172.21.15.59:57856) 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c15b5640 1 -- 172.21.15.59:0/1415018154 learned_addr learned my addr 172.21.15.59:0/1415018154 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94c0db4640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94bc06d2f0 0x7f94bc158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.248+0000 7f94b3fff640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94bc0cd9b0 0x7f94bc0aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94c15b5640 1 -- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94bc06d2f0 msgr2=0x7f94bc158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94c15b5640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94bc06d2f0 0x7f94bc158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94c15b5640 1 -- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94bc0cd9b0 msgr2=0x7f94bc0aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94c15b5640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94bc0cd9b0 0x7f94bc0aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94c15b5640 1 -- 172.21.15.59:0/1415018154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc158aa0 con 0x7f94bc162810 2024-06-29T15:38:35.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94c15b5640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc162810 0x7f94bc073220 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f94b400ea70 tx=0x7f94b400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:35.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94b17fa640 1 -- 172.21.15.59:0/1415018154 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b400ce40 con 0x7f94bc162810 2024-06-29T15:38:35.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94b17fa640 1 -- 172.21.15.59:0/1415018154 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94b4004c10 con 0x7f94bc162810 2024-06-29T15:38:35.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94b17fa640 1 -- 172.21.15.59:0/1415018154 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4005020 con 0x7f94bc162810 2024-06-29T15:38:35.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc1f8750 con 0x7f94bc162810 2024-06-29T15:38:35.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.249+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc1f8d10 con 0x7f94bc162810 2024-06-29T15:38:35.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.250+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc15bc80 con 0x7f94bc162810 2024-06-29T15:38:35.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.251+0000 7f94b17fa640 1 -- 172.21.15.59:0/1415018154 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f94b40051c0 con 0x7f94bc162810 2024-06-29T15:38:35.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.252+0000 7f94b17fa640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f94a8076420 0x7f94a8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.252+0000 7f94b17fa640 1 -- 172.21.15.59:0/1415018154 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 5277+0+0 (secure 0 0 0) 0x7f94b40984c0 con 0x7f94bc162810 2024-06-29T15:38:35.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.252+0000 7f94c0db4640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f94a8076420 0x7f94a8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.252+0000 7f94c0db4640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f94a8076420 0x7f94a8078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f94b8034000 tx=0x7f94b8004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:35.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.256+0000 7f94b17fa640 1 -- 172.21.15.59:0/1415018154 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f94bc158aa0 con 0x7f94bc162810 2024-06-29T15:38:35.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.520+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f94bc13a120 con 0x7f94bc162810 2024-06-29T15:38:35.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.521+0000 7f94b17fa640 1 -- 172.21.15.59:0/1415018154 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) ==== 80+0+5304 (secure 0 0 0) 0x7f94b40616c0 con 0x7f94bc162810 2024-06-29T15:38:35.522 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:35.522 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.523 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:38:35.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T15:38:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.529 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T15:38:35.531 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:38:35.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.534 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:35.535 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T15:38:35.536 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.522+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f94a8076420 msgr2=0x7f94a8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.522+0000 7f94c3018640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f94a8076420 0x7f94a8078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f94b8034000 tx=0x7f94b8004730 comp rx=0 tx=0).stop 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc162810 msgr2=0x7f94bc073220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc162810 0x7f94bc073220 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f94b400ea70 tx=0x7f94b400ef30 comp rx=0 tx=0).stop 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 shutdown_connections 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f94a8076420 0x7f94a8078930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94bc162810 0x7f94bc073220 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94bc0cd9b0 0x7f94bc0aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 --2- 172.21.15.59:0/1415018154 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94bc06d2f0 0x7f94bc158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 >> 172.21.15.59:0/1415018154 conn(0x7f94bc0b24c0 msgr2=0x7f94bc13acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 shutdown_connections 2024-06-29T15:38:35.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.523+0000 7f94c3018640 1 -- 172.21.15.59:0/1415018154 wait complete. 2024-06-29T15:38:35.538 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:35.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.656+0000 7f8924ac1640 1 -- 172.21.15.59:0/1134903508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 msgr2=0x7f8908004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.656+0000 7f8924ac1640 1 --2- 172.21.15.59:0/1134903508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 0x7f8908004c10 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f8914009b20 tx=0x7f8908005830 comp rx=0 tx=0).stop 2024-06-29T15:38:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.656+0000 7f891ed9d640 1 -- 172.21.15.59:0/1134903508 reap_dead start 2024-06-29T15:38:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.656+0000 7f8924ac1640 1 -- 172.21.15.59:0/1134903508 shutdown_connections 2024-06-29T15:38:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.656+0000 7f8924ac1640 1 -- 172.21.15.59:0/1134903508 >> 172.21.15.59:0/1134903508 conn(0x7f89200b5610 msgr2=0x7f89200b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.656+0000 7f8924ac1640 1 -- 172.21.15.59:0/1134903508 shutdown_connections 2024-06-29T15:38:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.656+0000 7f8924ac1640 1 -- 172.21.15.59:0/1134903508 wait complete. 2024-06-29T15:38:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.658+0000 7f8924ac1640 1 Processor -- start 2024-06-29T15:38:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.658+0000 7f8924ac1640 1 -- start start 2024-06-29T15:38:35.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.659+0000 7f8924ac1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 0x7f89200aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f8924ac1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89200ca250 0x7f892015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f8924ac1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89200c8de0 0x7f89200c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f8924ac1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8920162fa0 con 0x7f8908004800 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f8924ac1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8920162e20 con 0x7f89200ca250 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f8924ac1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f89200b7bd0 con 0x7f89200c8de0 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 0x7f89200aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 0x7f89200aaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57868/0 (socket says 172.21.15.59:57868) 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 -- 172.21.15.59:0/2195083857 learned_addr learned my addr 172.21.15.59:0/2195083857 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891dd9b640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89200ca250 0x7f892015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891ed9d640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89200c8de0 0x7f89200c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 -- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89200c8de0 msgr2=0x7f89200c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89200c8de0 0x7f89200c91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 -- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89200ca250 msgr2=0x7f892015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89200ca250 0x7f892015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 -- 172.21.15.59:0/2195083857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89200c9730 con 0x7f8908004800 2024-06-29T15:38:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f891e59c640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 0x7f89200aaec0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f8910006600 tx=0x7f89100039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f890effd640 1 -- 172.21.15.59:0/2195083857 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910015070 con 0x7f8908004800 2024-06-29T15:38:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.660+0000 7f890effd640 1 -- 172.21.15.59:0/2195083857 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8910011900 con 0x7f8908004800 2024-06-29T15:38:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.661+0000 7f890effd640 1 -- 172.21.15.59:0/2195083857 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910011d50 con 0x7f8908004800 2024-06-29T15:38:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.661+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89201f5600 con 0x7f8908004800 2024-06-29T15:38:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.661+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89200b44a0 con 0x7f8908004800 2024-06-29T15:38:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.662+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89201f5af0 con 0x7f8908004800 2024-06-29T15:38:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.662+0000 7f890effd640 1 -- 172.21.15.59:0/2195083857 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f8910018070 con 0x7f8908004800 2024-06-29T15:38:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.663+0000 7f890effd640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f89000763f0 0x7f8900078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.663+0000 7f890effd640 1 -- 172.21.15.59:0/2195083857 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 5277+0+0 (secure 0 0 0) 0x7f8910098540 con 0x7f8908004800 2024-06-29T15:38:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.663+0000 7f891dd9b640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f89000763f0 0x7f8900078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.664+0000 7f891dd9b640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f89000763f0 0x7f8900078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8914035000 tx=0x7f891402d040 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:35.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.667+0000 7f890effd640 1 -- 172.21.15.59:0/2195083857 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f89200c9730 con 0x7f8908004800 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.862+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89201622c0 con 0x7f8908004800 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.863+0000 7f890effd640 1 -- 172.21.15.59:0/2195083857 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) ==== 74+0+13230 (secure 0 0 0) 0x7f89100617f0 con 0x7f8908004800 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":172,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:35.119610+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f89000763f0 msgr2=0x7f8900078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f89000763f0 0x7f8900078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8914035000 tx=0x7f891402d040 comp rx=0 tx=0).stop 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 msgr2=0x7f89200aaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 0x7f89200aaec0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f8910006600 tx=0x7f89100039c0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 shutdown_connections 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f89200c8de0 0x7f89200c91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89200ca250 0x7f892015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f89000763f0 0x7f8900078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 --2- 172.21.15.59:0/2195083857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8908004800 0x7f89200aaec0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 >> 172.21.15.59:0/2195083857 conn(0x7f89200b5610 msgr2=0x7f89200cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 shutdown_connections 2024-06-29T15:38:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.865+0000 7f8924ac1640 1 -- 172.21.15.59:0/2195083857 wait complete. 2024-06-29T15:38:35.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:35.877 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:35.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 -- 172.21.15.59:0/1156912086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a480cf5c0 msgr2=0x7f5a480d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/1156912086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a480cf5c0 0x7f5a480d1a20 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5a480cfbc0 tx=0x7f5a3c02a560 comp rx=0 tx=0).stop 2024-06-29T15:38:35.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 -- 172.21.15.59:0/1156912086 shutdown_connections 2024-06-29T15:38:35.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/1156912086 >> v2:172.21.15.153:3300/0 conn(0x7f5a480ce950 0x7f5a480d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/1156912086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a480cf5c0 0x7f5a480d1a20 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/1156912086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a4808c190 0x7f5a480cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 -- 172.21.15.59:0/1156912086 >> 172.21.15.59:0/1156912086 conn(0x7f5a480b5610 msgr2=0x7f5a480b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 -- 172.21.15.59:0/1156912086 shutdown_connections 2024-06-29T15:38:35.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:35.998+0000 7f5a4e63c640 1 -- 172.21.15.59:0/1156912086 wait complete. 2024-06-29T15:38:36.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.000+0000 7f5a4e63c640 1 Processor -- start 2024-06-29T15:38:36.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.000+0000 7f5a4e63c640 1 -- start start 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a4e63c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a4808c190 0x7f5a481f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a4e63c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a480ce950 0x7f5a4815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a4e63c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a480cf5c0 0x7f5a4815bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a4e63c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48162440 con 0x7f5a4808c190 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a4e63c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5a481622c0 con 0x7f5a480cf5c0 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a4e63c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5a481625c0 con 0x7f5a480ce950 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a47fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a4808c190 0x7f5a481f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a47fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a4808c190 0x7f5a481f7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57882/0 (socket says 172.21.15.59:57882) 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.001+0000 7f5a47fff640 1 -- 172.21.15.59:0/3683646095 learned_addr learned my addr 172.21.15.59:0/3683646095 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a477fe640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a480ce950 0x7f5a4815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a4cbd9640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a480cf5c0 0x7f5a4815bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a47fff640 1 -- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a480ce950 msgr2=0x7f5a4815a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a47fff640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a480ce950 0x7f5a4815a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a47fff640 1 -- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a480cf5c0 msgr2=0x7f5a4815bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a47fff640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a480cf5c0 0x7f5a4815bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a47fff640 1 -- 172.21.15.59:0/3683646095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a480cbd50 con 0x7f5a4808c190 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a47fff640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a4808c190 0x7f5a481f7ac0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800d690 tx=0x7f5a3800db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a44ff9640 1 -- 172.21.15.59:0/3683646095 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38017070 con 0x7f5a4808c190 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a44ff9640 1 -- 172.21.15.59:0/3683646095 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a38005310 con 0x7f5a4808c190 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a480cc030 con 0x7f5a4808c190 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a44ff9640 1 -- 172.21.15.59:0/3683646095 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a380097c0 con 0x7f5a4808c190 2024-06-29T15:38:36.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.002+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a4815ade0 con 0x7f5a4808c190 2024-06-29T15:38:36.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.003+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a480c73f0 con 0x7f5a4808c190 2024-06-29T15:38:36.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.004+0000 7f5a44ff9640 1 -- 172.21.15.59:0/3683646095 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f5a38009960 con 0x7f5a4808c190 2024-06-29T15:38:36.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.005+0000 7f5a44ff9640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5a340763f0 0x7f5a34078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.005+0000 7f5a44ff9640 1 -- 172.21.15.59:0/3683646095 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 5277+0+0 (secure 0 0 0) 0x7f5a380977b0 con 0x7f5a4808c190 2024-06-29T15:38:36.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.005+0000 7f5a477fe640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5a340763f0 0x7f5a34078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.006+0000 7f5a477fe640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5a340763f0 0x7f5a34078900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c033000 tx=0x7f5a3c004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.009+0000 7f5a44ff9640 1 -- 172.21.15.59:0/3683646095 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5a480c73f0 con 0x7f5a4808c190 2024-06-29T15:38:36.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.201+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5a480cf080 con 0x7f5a4808c190 2024-06-29T15:38:36.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.202+0000 7f5a44ff9640 1 -- 172.21.15.59:0/3683646095 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v172) ==== 93+0+10 (secure 0 0 0) 0x7f5a38069090 con 0x7f5a4808c190 2024-06-29T15:38:36.202 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:38:36.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5a340763f0 msgr2=0x7f5a34078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5a340763f0 0x7f5a34078900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c033000 tx=0x7f5a3c004730 comp rx=0 tx=0).stop 2024-06-29T15:38:36.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a4808c190 msgr2=0x7f5a481f7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a4808c190 0x7f5a481f7ac0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800d690 tx=0x7f5a3800db50 comp rx=0 tx=0).stop 2024-06-29T15:38:36.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 shutdown_connections 2024-06-29T15:38:36.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a480cf5c0 0x7f5a4815bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a480ce950 0x7f5a4815a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5a340763f0 0x7f5a34078900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 --2- 172.21.15.59:0/3683646095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a4808c190 0x7f5a481f7ac0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 >> 172.21.15.59:0/3683646095 conn(0x7f5a480b5610 msgr2=0x7f5a480d1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 shutdown_connections 2024-06-29T15:38:36.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.204+0000 7f5a4e63c640 1 -- 172.21.15.59:0/3683646095 wait complete. 2024-06-29T15:38:36.215 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:38:36.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.333+0000 7f49797aa640 1 -- 172.21.15.59:0/3201410022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 msgr2=0x7f49741622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.333+0000 7f49797aa640 1 --2- 172.21.15.59:0/3201410022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 0x7f49741622c0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f495c005fd0 tx=0x7f495c02a5a0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.333+0000 7f49797aa640 1 -- 172.21.15.59:0/3201410022 shutdown_connections 2024-06-29T15:38:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.333+0000 7f49797aa640 1 --2- 172.21.15.59:0/3201410022 >> v2:172.21.15.153:3300/0 conn(0x7f4974162800 0x7f4974162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.333+0000 7f49797aa640 1 --2- 172.21.15.59:0/3201410022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 0x7f49741622c0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.333+0000 7f49797aa640 1 -- 172.21.15.59:0/3201410022 >> 172.21.15.59:0/3201410022 conn(0x7f49740b5610 msgr2=0x7f49740b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.333+0000 7f49797aa640 1 -- 172.21.15.59:0/3201410022 shutdown_connections 2024-06-29T15:38:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.333+0000 7f49797aa640 1 -- 172.21.15.59:0/3201410022 wait complete. 2024-06-29T15:38:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.335+0000 7f49797aa640 1 Processor -- start 2024-06-29T15:38:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.335+0000 7f49797aa640 1 -- start start 2024-06-29T15:38:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49797aa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f497408c190 0x7f49741f7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49797aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 0x7f49740cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49797aa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4974162800 0x7f4974160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49797aa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f49740b7f60 con 0x7f49740ce950 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49797aa640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f49740b7de0 con 0x7f497408c190 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49797aa640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f49740b80e0 con 0x7f4974162800 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49727fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 0x7f49740cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49727fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 0x7f49740cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57902/0 (socket says 172.21.15.59:57902) 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49727fc640 1 -- 172.21.15.59:0/824917794 learned_addr learned my addr 172.21.15.59:0/824917794 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f4972ffd640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f497408c190 0x7f49741f7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.336+0000 7f49737fe640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4974162800 0x7f4974160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f49727fc640 1 -- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4974162800 msgr2=0x7f4974160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f49727fc640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4974162800 0x7f4974160440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f49727fc640 1 -- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f497408c190 msgr2=0x7f49741f7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f49727fc640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f497408c190 0x7f49741f7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f49727fc640 1 -- 172.21.15.59:0/824917794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4974155180 con 0x7f49740ce950 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f49727fc640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 0x7f49740cb5e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f495c034040 tx=0x7f495c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f496b7fe640 1 -- 172.21.15.59:0/824917794 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c03e030 con 0x7f49740ce950 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f496b7fe640 1 -- 172.21.15.59:0/824917794 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f495c003d90 con 0x7f49740ce950 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f496b7fe640 1 -- 172.21.15.59:0/824917794 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c035040 con 0x7f49740ce950 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49740cbb20 con 0x7f49740ce950 2024-06-29T15:38:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.337+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49741555f0 con 0x7f49740ce950 2024-06-29T15:38:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.338+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4974155350 con 0x7f49740ce950 2024-06-29T15:38:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.339+0000 7f496b7fe640 1 -- 172.21.15.59:0/824917794 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f495c041030 con 0x7f49740ce950 2024-06-29T15:38:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.339+0000 7f496b7fe640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f49540763f0 0x7f4954078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.339+0000 7f496b7fe640 1 -- 172.21.15.59:0/824917794 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 5277+0+0 (secure 0 0 0) 0x7f495c0b81c0 con 0x7f49740ce950 2024-06-29T15:38:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.340+0000 7f4972ffd640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f49540763f0 0x7f4954078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.340+0000 7f4972ffd640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f49540763f0 0x7f4954078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4964001e20 tx=0x7f4964009040 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.344+0000 7f496b7fe640 1 -- 172.21.15.59:0/824917794 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4974155350 con 0x7f49740ce950 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.536+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f49741622c0 con 0x7f49740ce950 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.537+0000 7f496b7fe640 1 -- 172.21.15.59:0/824917794 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v172) ==== 84+0+65 (secure 0 0 0) 0x7f495c0814f0 con 0x7f49740ce950 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:36.537 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:38:36.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f49540763f0 msgr2=0x7f4954078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f49540763f0 0x7f4954078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4964001e20 tx=0x7f4964009040 comp rx=0 tx=0).stop 2024-06-29T15:38:36.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 msgr2=0x7f49740cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 0x7f49740cb5e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f495c034040 tx=0x7f495c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 shutdown_connections 2024-06-29T15:38:36.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4974162800 0x7f4974160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f49540763f0 0x7f4954078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49740ce950 0x7f49740cb5e0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 --2- 172.21.15.59:0/824917794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f497408c190 0x7f49741f7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 >> 172.21.15.59:0/824917794 conn(0x7f49740b5610 msgr2=0x7f497406d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 shutdown_connections 2024-06-29T15:38:36.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.539+0000 7f49797aa640 1 -- 172.21.15.59:0/824917794 wait complete. 2024-06-29T15:38:36.550 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:38:36.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:38:36.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.673+0000 7f33fea09640 1 -- 172.21.15.59:0/2433045005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f80d1130 msgr2=0x7f33f815f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.673+0000 7f33fea09640 1 --2- 172.21.15.59:0/2433045005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f80d1130 0x7f33f815f7f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009830 tx=0x7f33ec02a510 comp rx=0 tx=0).stop 2024-06-29T15:38:36.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.673+0000 7f33fea09640 1 -- 172.21.15.59:0/2433045005 shutdown_connections 2024-06-29T15:38:36.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.673+0000 7f33fea09640 1 --2- 172.21.15.59:0/2433045005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f80d1130 0x7f33f815f7f0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.673+0000 7f33fea09640 1 --2- 172.21.15.59:0/2433045005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33f8145650 0x7f33f80b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.673+0000 7f33fea09640 1 -- 172.21.15.59:0/2433045005 >> 172.21.15.59:0/2433045005 conn(0x7f33f80b6ed0 msgr2=0x7f33f80b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.673+0000 7f33fea09640 1 -- 172.21.15.59:0/2433045005 shutdown_connections 2024-06-29T15:38:36.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.673+0000 7f33fea09640 1 -- 172.21.15.59:0/2433045005 wait complete. 2024-06-29T15:38:36.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.674+0000 7f33fea09640 1 Processor -- start 2024-06-29T15:38:36.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.675+0000 7f33fea09640 1 -- start start 2024-06-29T15:38:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fea09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f80d1130 0x7f33f81e7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fea09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33f8145650 0x7f33f80c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fea09640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f815fd30 0x7f33f81f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fea09640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f33f815c910 con 0x7f33f815fd30 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fea09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f33f815eba0 con 0x7f33f80d1130 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fea09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f33f815ca90 con 0x7f33f8145650 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fcfa6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f815fd30 0x7f33f81f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fcfa6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f815fd30 0x7f33f81f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57930/0 (socket says 172.21.15.59:57930) 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fcfa6640 1 -- 172.21.15.59:0/3545979264 learned_addr learned my addr 172.21.15.59:0/3545979264 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33f77fe640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33f8145650 0x7f33f80c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33f7fff640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f80d1130 0x7f33f81e7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fcfa6640 1 -- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33f8145650 msgr2=0x7f33f80c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fcfa6640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33f8145650 0x7f33f80c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fcfa6640 1 -- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f80d1130 msgr2=0x7f33f81e7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fcfa6640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f80d1130 0x7f33f81e7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.676+0000 7f33fcfa6640 1 -- 172.21.15.59:0/3545979264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f81f2a00 con 0x7f33f815fd30 2024-06-29T15:38:36.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.677+0000 7f33fcfa6640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f815fd30 0x7f33f81f24c0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f33f000ca20 tx=0x7f33f000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.677+0000 7f33f4ff9640 1 -- 172.21.15.59:0/3545979264 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0016020 con 0x7f33f815fd30 2024-06-29T15:38:36.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.677+0000 7f33f4ff9640 1 -- 172.21.15.59:0/3545979264 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33f000ac60 con 0x7f33f815fd30 2024-06-29T15:38:36.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.677+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f81e6e20 con 0x7f33f815fd30 2024-06-29T15:38:36.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.677+0000 7f33f4ff9640 1 -- 172.21.15.59:0/3545979264 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0012680 con 0x7f33f815fd30 2024-06-29T15:38:36.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.677+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f81e73e0 con 0x7f33f815fd30 2024-06-29T15:38:36.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.678+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33f815cf80 con 0x7f33f815fd30 2024-06-29T15:38:36.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.679+0000 7f33f4ff9640 1 -- 172.21.15.59:0/3545979264 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f33f0019020 con 0x7f33f815fd30 2024-06-29T15:38:36.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.679+0000 7f33f4ff9640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f33e40763f0 0x7f33e4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.679+0000 7f33f4ff9640 1 -- 172.21.15.59:0/3545979264 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 5277+0+0 (secure 0 0 0) 0x7f33f0097ca0 con 0x7f33f815fd30 2024-06-29T15:38:36.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.679+0000 7f33f7fff640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f33e40763f0 0x7f33e4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.680+0000 7f33f7fff640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f33e40763f0 0x7f33e4078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f33e800a490 tx=0x7f33e800a420 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.684+0000 7f33f4ff9640 1 -- 172.21.15.59:0/3545979264 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f33f0020020 con 0x7f33f815fd30 2024-06-29T15:38:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:36.875+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f33f81415d0 con 0x7f33f815fd30 2024-06-29T15:38:37.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.116+0000 7f33f4ff9640 1 -- 172.21.15.59:0/3545979264 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v173) ==== 138+0+0 (secure 0 0 0) 0x7f33f8160140 con 0x7f33f815fd30 2024-06-29T15:38:37.116 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T15:38:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f33e40763f0 msgr2=0x7f33e4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f33e40763f0 0x7f33e4078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f33e800a490 tx=0x7f33e800a420 comp rx=0 tx=0).stop 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f815fd30 msgr2=0x7f33f81f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f815fd30 0x7f33f81f24c0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f33f000ca20 tx=0x7f33f000cee0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 shutdown_connections 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f33e40763f0 0x7f33e4078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33f815fd30 0x7f33f81f24c0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33f8145650 0x7f33f80c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 --2- 172.21.15.59:0/3545979264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f80d1130 0x7f33f81e7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 >> 172.21.15.59:0/3545979264 conn(0x7f33f80b6ed0 msgr2=0x7f33f806fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 shutdown_connections 2024-06-29T15:38:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.119+0000 7f33fea09640 1 -- 172.21.15.59:0/3545979264 wait complete. 2024-06-29T15:38:37.134 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:38:37.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.260+0000 7f1ce473c640 1 -- 172.21.15.59:0/892471907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cc8003f90 msgr2=0x7f1cc8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.260+0000 7f1ce473c640 1 --2- 172.21.15.59:0/892471907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cc8003f90 0x7f1cc8004800 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0017c0 tx=0x7f1cc8005830 comp rx=0 tx=0).stop 2024-06-29T15:38:37.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.260+0000 7f1ce2cd9640 1 -- 172.21.15.59:0/892471907 reap_dead start 2024-06-29T15:38:37.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.260+0000 7f1ce473c640 1 -- 172.21.15.59:0/892471907 shutdown_connections 2024-06-29T15:38:37.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.260+0000 7f1ce473c640 1 -- 172.21.15.59:0/892471907 >> 172.21.15.59:0/892471907 conn(0x7f1cdc0d4070 msgr2=0x7f1cdc0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:37.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.260+0000 7f1ce473c640 1 -- 172.21.15.59:0/892471907 shutdown_connections 2024-06-29T15:38:37.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.260+0000 7f1ce473c640 1 -- 172.21.15.59:0/892471907 wait complete. 2024-06-29T15:38:37.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.262+0000 7f1ce473c640 1 Processor -- start 2024-06-29T15:38:37.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.262+0000 7f1ce473c640 1 -- start start 2024-06-29T15:38:37.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce473c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cc8003f90 0x7f1cdc160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:37.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce473c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc156cc0 0x7f1cdc1e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:37.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce473c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cdc1570d0 0x7f1cdc1e8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce473c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc0b86c0 con 0x7f1cdc156cc0 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce473c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1cdc0b30c0 con 0x7f1cdc1570d0 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce473c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1cdc0b8840 con 0x7f1cc8003f90 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce1cd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc156cc0 0x7f1cdc1e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce1cd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc156cc0 0x7f1cdc1e7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57952/0 (socket says 172.21.15.59:57952) 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce1cd7640 1 -- 172.21.15.59:0/2802721281 learned_addr learned my addr 172.21.15.59:0/2802721281 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce24d8640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cc8003f90 0x7f1cdc160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.263+0000 7f1ce2cd9640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cdc1570d0 0x7f1cdc1e8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1ce1cd7640 1 -- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cc8003f90 msgr2=0x7f1cdc160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1ce1cd7640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cc8003f90 0x7f1cdc160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1ce1cd7640 1 -- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cdc1570d0 msgr2=0x7f1cdc1e8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1ce1cd7640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cdc1570d0 0x7f1cdc1e8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1ce1cd7640 1 -- 172.21.15.59:0/2802721281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cdc0cac70 con 0x7f1cdc156cc0 2024-06-29T15:38:37.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1ce1cd7640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc156cc0 0x7f1cdc1e7b00 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0018e0 tx=0x7f1ccc006170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:37.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1cd2ffd640 1 -- 172.21.15.59:0/2802721281 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc042070 con 0x7f1cdc156cc0 2024-06-29T15:38:37.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1cd2ffd640 1 -- 172.21.15.59:0/2802721281 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ccc0037f0 con 0x7f1cdc156cc0 2024-06-29T15:38:37.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cdc0caf50 con 0x7f1cdc156cc0 2024-06-29T15:38:37.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1cd2ffd640 1 -- 172.21.15.59:0/2802721281 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc037040 con 0x7f1cdc156cc0 2024-06-29T15:38:37.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.264+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cdc0cb510 con 0x7f1cdc156cc0 2024-06-29T15:38:37.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.265+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cdc160b70 con 0x7f1cdc156cc0 2024-06-29T15:38:37.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.266+0000 7f1cd2ffd640 1 -- 172.21.15.59:0/2802721281 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f1ccc003c00 con 0x7f1cdc156cc0 2024-06-29T15:38:37.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.266+0000 7f1cd2ffd640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f1cc00763f0 0x7f1cc0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:37.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.267+0000 7f1cd2ffd640 1 -- 172.21.15.59:0/2802721281 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5647+0+0 (secure 0 0 0) 0x7f1ccc0b7740 con 0x7f1cdc156cc0 2024-06-29T15:38:37.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.267+0000 7f1ce24d8640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f1cc00763f0 0x7f1cc0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:37.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.267+0000 7f1ce24d8640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f1cc00763f0 0x7f1cc0078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80015c0 tx=0x7f1cd8006aa0 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:37.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.271+0000 7f1cd2ffd640 1 -- 172.21.15.59:0/2802721281 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1ccc080750 con 0x7f1cdc156cc0 2024-06-29T15:38:37.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:37.462+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1cdc0b7fb0 con 0x7f1cdc156cc0 2024-06-29T15:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.119+0000 7f1cd2ffd640 1 -- 172.21.15.59:0/2802721281 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v174) ==== 170+0+0 (secure 0 0 0) 0x7f1ccc084a50 con 0x7f1cdc156cc0 2024-06-29T15:38:38.119 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T15:38:38.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f1cc00763f0 msgr2=0x7f1cc0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f1cc00763f0 0x7f1cc0078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80015c0 tx=0x7f1cd8006aa0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc156cc0 msgr2=0x7f1cdc1e7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc156cc0 0x7f1cdc1e7b00 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0018e0 tx=0x7f1ccc006170 comp rx=0 tx=0).stop 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 shutdown_connections 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cdc1570d0 0x7f1cdc1e8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f1cc00763f0 0x7f1cc0078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cdc156cc0 0x7f1cdc1e7b00 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 --2- 172.21.15.59:0/2802721281 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1cc8003f90 0x7f1cdc160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 >> 172.21.15.59:0/2802721281 conn(0x7f1cdc0d4070 msgr2=0x7f1cdc06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 shutdown_connections 2024-06-29T15:38:38.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.122+0000 7f1ce473c640 1 -- 172.21.15.59:0/2802721281 wait complete. 2024-06-29T15:38:38.148 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:38:38.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.275+0000 7fbd953eb640 1 -- 172.21.15.59:0/3716991564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd90141bd0 msgr2=0x7fbd9013edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.275+0000 7fbd953eb640 1 --2- 172.21.15.59:0/3716991564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd90141bd0 0x7fbd9013edc0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fbd80005fd0 tx=0x7fbd8002aa10 comp rx=0 tx=0).stop 2024-06-29T15:38:38.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.275+0000 7fbd953eb640 1 -- 172.21.15.59:0/3716991564 shutdown_connections 2024-06-29T15:38:38.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.275+0000 7fbd953eb640 1 --2- 172.21.15.59:0/3716991564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd90141bd0 0x7fbd9013edc0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.275+0000 7fbd953eb640 1 --2- 172.21.15.59:0/3716991564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd9015ceb0 0x7fbd900b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.275+0000 7fbd953eb640 1 -- 172.21.15.59:0/3716991564 >> 172.21.15.59:0/3716991564 conn(0x7fbd900b6fb0 msgr2=0x7fbd900b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:38.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.275+0000 7fbd953eb640 1 -- 172.21.15.59:0/3716991564 shutdown_connections 2024-06-29T15:38:38.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.275+0000 7fbd953eb640 1 -- 172.21.15.59:0/3716991564 wait complete. 2024-06-29T15:38:38.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.277+0000 7fbd953eb640 1 Processor -- start 2024-06-29T15:38:38.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.277+0000 7fbd953eb640 1 -- start start 2024-06-29T15:38:38.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd953eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9013e9b0 0x7fbd9014bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:38.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd953eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd9015ceb0 0x7fbd901f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:38.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd953eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9013e0c0 0x7fbd9014eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:38.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd953eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbd900b2a80 con 0x7fbd9013e0c0 2024-06-29T15:38:38.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd953eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd900b2900 con 0x7fbd9013e9b0 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd953eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbd900b2c00 con 0x7fbd9015ceb0 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd8f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9013e0c0 0x7fbd9014eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd8f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9013e0c0 0x7fbd9014eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57966/0 (socket says 172.21.15.59:57966) 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd8f7fe640 1 -- 172.21.15.59:0/2197312178 learned_addr learned my addr 172.21.15.59:0/2197312178 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd8effd640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9013e9b0 0x7fbd9014bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.278+0000 7fbd8e7fc640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd9015ceb0 0x7fbd901f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd8f7fe640 1 -- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd9015ceb0 msgr2=0x7fbd901f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd8f7fe640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd9015ceb0 0x7fbd901f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd8f7fe640 1 -- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9013e9b0 msgr2=0x7fbd9014bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd8f7fe640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9013e9b0 0x7fbd9014bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd8f7fe640 1 -- 172.21.15.59:0/2197312178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9014de30 con 0x7fbd9013e0c0 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd8f7fe640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9013e0c0 0x7fbd9014eed0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fbd8002afd0 tx=0x7fbd80003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:38.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd777fe640 1 -- 172.21.15.59:0/2197312178 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80033050 con 0x7fbd9013e0c0 2024-06-29T15:38:38.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9014e110 con 0x7fbd9013e0c0 2024-06-29T15:38:38.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd777fe640 1 -- 172.21.15.59:0/2197312178 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd80004250 con 0x7fbd9013e0c0 2024-06-29T15:38:38.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd777fe640 1 -- 172.21.15.59:0/2197312178 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80004680 con 0x7fbd9013e0c0 2024-06-29T15:38:38.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.279+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd9014e6d0 con 0x7fbd9013e0c0 2024-06-29T15:38:38.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.280+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd9013edc0 con 0x7fbd9013e0c0 2024-06-29T15:38:38.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.281+0000 7fbd777fe640 1 -- 172.21.15.59:0/2197312178 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fbd80036070 con 0x7fbd9013e0c0 2024-06-29T15:38:38.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.282+0000 7fbd777fe640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fbd7c076420 0x7fbd7c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:38.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.282+0000 7fbd777fe640 1 -- 172.21.15.59:0/2197312178 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 6029+0+0 (secure 0 0 0) 0x7fbd800b8f30 con 0x7fbd9013e0c0 2024-06-29T15:38:38.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.282+0000 7fbd8effd640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fbd7c076420 0x7fbd7c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:38.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.283+0000 7fbd8effd640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fbd7c076420 0x7fbd7c078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009b30 tx=0x7fbd88005f90 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:38.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.286+0000 7fbd777fe640 1 -- 172.21.15.59:0/2197312178 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbd80004820 con 0x7fbd9013e0c0 2024-06-29T15:38:38.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:38.522+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbd900708d0 con 0x7fbd9013e0c0 2024-06-29T15:38:38.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:38.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:39.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.153+0000 7fbd777fe640 1 -- 172.21.15.59:0/2197312178 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v85) ==== 174+0+0 (secure 0 0 0) 0x7fbd80081e50 con 0x7fbd9013e0c0 2024-06-29T15:38:39.155 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 21 and data pool 22 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.164+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fbd7c076420 msgr2=0x7fbd7c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.164+0000 7fbd953eb640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fbd7c076420 0x7fbd7c078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009b30 tx=0x7fbd88005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.164+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9013e0c0 msgr2=0x7fbd9014eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.164+0000 7fbd953eb640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9013e0c0 0x7fbd9014eed0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fbd8002afd0 tx=0x7fbd80003f10 comp rx=0 tx=0).stop 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.165+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 shutdown_connections 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.165+0000 7fbd953eb640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fbd7c076420 0x7fbd7c078930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.165+0000 7fbd953eb640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9013e0c0 0x7fbd9014eed0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.165+0000 7fbd953eb640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd9015ceb0 0x7fbd901f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.165+0000 7fbd953eb640 1 --2- 172.21.15.59:0/2197312178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd9013e9b0 0x7fbd9014bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.165+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 >> 172.21.15.59:0/2197312178 conn(0x7fbd900b6fb0 msgr2=0x7fbd90070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.166+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 shutdown_connections 2024-06-29T15:38:39.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.166+0000 7fbd953eb640 1 -- 172.21.15.59:0/2197312178 wait complete. 2024-06-29T15:38:39.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.304+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1112132434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4162770 msgr2=0x7fd6b40ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.304+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1112132434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4162770 0x7fd6b40ce550 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4005fd0 tx=0x7fd6a402a540 comp rx=0 tx=0).stop 2024-06-29T15:38:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.304+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1112132434 shutdown_connections 2024-06-29T15:38:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.304+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1112132434 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b415e0a0 0x7fd6b4161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.304+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1112132434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4162770 0x7fd6b40ce550 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.304+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1112132434 >> 172.21.15.59:0/1112132434 conn(0x7fd6b40d45b0 msgr2=0x7fd6b40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.304+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1112132434 shutdown_connections 2024-06-29T15:38:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.304+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1112132434 wait complete. 2024-06-29T15:38:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.306+0000 7fd6bb48f640 1 Processor -- start 2024-06-29T15:38:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.306+0000 7fd6bb48f640 1 -- start start 2024-06-29T15:38:39.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6bb48f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b415e0a0 0x7fd6b415ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6bb48f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4161fc0 0x7fd6b41f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6bb48f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b4162770 0x7fd6b41f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6bb48f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b40d2d80 con 0x7fd6b4161fc0 2024-06-29T15:38:39.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6bb48f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b40b4b00 con 0x7fd6b4162770 2024-06-29T15:38:39.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6bb48f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b40d2f00 con 0x7fd6b415e0a0 2024-06-29T15:38:39.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6b8a2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4161fc0 0x7fd6b41f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6b8a2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4161fc0 0x7fd6b41f8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57972/0 (socket says 172.21.15.59:57972) 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.307+0000 7fd6b8a2a640 1 -- 172.21.15.59:0/1299025490 learned_addr learned my addr 172.21.15.59:0/1299025490 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b9a2c640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b4162770 0x7fd6b41f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b922b640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b415e0a0 0x7fd6b415ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b8a2a640 1 -- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b415e0a0 msgr2=0x7fd6b415ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b8a2a640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b415e0a0 0x7fd6b415ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b8a2a640 1 -- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b4162770 msgr2=0x7fd6b41f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b8a2a640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b4162770 0x7fd6b41f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b8a2a640 1 -- 172.21.15.59:0/1299025490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b40c9570 con 0x7fd6b4161fc0 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b922b640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b415e0a0 0x7fd6b415ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b8a2a640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4161fc0 0x7fd6b41f8280 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800c970 tx=0x7fd6a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b1ffb640 1 -- 172.21.15.59:0/1299025490 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8007bb0 con 0x7fd6b4161fc0 2024-06-29T15:38:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b1ffb640 1 -- 172.21.15.59:0/1299025490 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6a8007e40 con 0x7fd6b4161fc0 2024-06-29T15:38:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b40c9850 con 0x7fd6b4161fc0 2024-06-29T15:38:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6b1ffb640 1 -- 172.21.15.59:0/1299025490 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8005280 con 0x7fd6b4161fc0 2024-06-29T15:38:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.308+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6b40c9e10 con 0x7fd6b4161fc0 2024-06-29T15:38:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.309+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b40d33d0 con 0x7fd6b4161fc0 2024-06-29T15:38:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.310+0000 7fd6b1ffb640 1 -- 172.21.15.59:0/1299025490 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fd6a8005420 con 0x7fd6b4161fc0 2024-06-29T15:38:39.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.311+0000 7fd6b1ffb640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fd6a00763f0 0x7fd6a0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.311+0000 7fd6b1ffb640 1 -- 172.21.15.59:0/1299025490 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 6129+0+0 (secure 0 0 0) 0x7fd6a8097ec0 con 0x7fd6b4161fc0 2024-06-29T15:38:39.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.311+0000 7fd6b922b640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fd6a00763f0 0x7fd6a0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.312+0000 7fd6b922b640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fd6a00763f0 0x7fd6a0078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4033000 tx=0x7fd6a4004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.315+0000 7fd6b1ffb640 1 -- 172.21.15.59:0/1299025490 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd6a8014030 con 0x7fd6b4161fc0 2024-06-29T15:38:39.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.508+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6b415db30 con 0x7fd6b4161fc0 2024-06-29T15:38:39.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.510+0000 7fd6b1ffb640 1 -- 172.21.15.59:0/1299025490 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) ==== 74+0+16961 (secure 0 0 0) 0x7fd6a8060e00 con 0x7fd6b4161fc0 2024-06-29T15:38:39.510 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:39.511 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":175,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:38.523116+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:36.882222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.468804+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fd6a00763f0 msgr2=0x7fd6a0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fd6a00763f0 0x7fd6a0078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4033000 tx=0x7fd6a4004730 comp rx=0 tx=0).stop 2024-06-29T15:38:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4161fc0 msgr2=0x7fd6b41f8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4161fc0 0x7fd6b41f8280 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800c970 tx=0x7fd6a800ce30 comp rx=0 tx=0).stop 2024-06-29T15:38:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 shutdown_connections 2024-06-29T15:38:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b4162770 0x7fd6b41f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fd6a00763f0 0x7fd6a0078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6b4161fc0 0x7fd6b41f8280 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 --2- 172.21.15.59:0/1299025490 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6b415e0a0 0x7fd6b415ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 >> 172.21.15.59:0/1299025490 conn(0x7fd6b40d45b0 msgr2=0x7fd6b40b73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 shutdown_connections 2024-06-29T15:38:39.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.512+0000 7fd6bb48f640 1 -- 172.21.15.59:0/1299025490 wait complete. 2024-06-29T15:38:39.524 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.643+0000 7f5d225ba640 1 -- 172.21.15.59:0/4084275077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d04003c80 msgr2=0x7f5d040040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.643+0000 7f5d225ba640 1 --2- 172.21.15.59:0/4084275077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d04003c80 0x7f5d040040d0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c006f90 tx=0x7f5d04005830 comp rx=0 tx=0).stop 2024-06-29T15:38:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.643+0000 7f5d20b57640 1 -- 172.21.15.59:0/4084275077 reap_dead start 2024-06-29T15:38:39.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.643+0000 7f5d225ba640 1 -- 172.21.15.59:0/4084275077 shutdown_connections 2024-06-29T15:38:39.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.643+0000 7f5d225ba640 1 -- 172.21.15.59:0/4084275077 >> 172.21.15.59:0/4084275077 conn(0x7f5d1c0b6fb0 msgr2=0x7f5d1c0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.643+0000 7f5d225ba640 1 -- 172.21.15.59:0/4084275077 shutdown_connections 2024-06-29T15:38:39.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.643+0000 7f5d225ba640 1 -- 172.21.15.59:0/4084275077 wait complete. 2024-06-29T15:38:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.645+0000 7f5d225ba640 1 Processor -- start 2024-06-29T15:38:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.645+0000 7f5d225ba640 1 -- start start 2024-06-29T15:38:39.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d225ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d04003c80 0x7f5d1c06f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d225ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d1c1f7fe0 0x7f5d1c0c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d225ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d1c1f5020 0x7f5d1c1f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d225ba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c13f150 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d225ba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c13efd0 con 0x7f5d1c1f5020 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d225ba640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1c0b2530 con 0x7f5d04003c80 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d1c1f7fe0 0x7f5d1c0c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d1c1f7fe0 0x7f5d1c0c8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57990/0 (socket says 172.21.15.59:57990) 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d1b7fe640 1 -- 172.21.15.59:0/907237287 learned_addr learned my addr 172.21.15.59:0/907237287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d1bfff640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d04003c80 0x7f5d1c06f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.646+0000 7f5d20b57640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d1c1f5020 0x7f5d1c1f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d1b7fe640 1 -- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d04003c80 msgr2=0x7f5d1c06f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d1b7fe640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d04003c80 0x7f5d1c06f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d1b7fe640 1 -- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d1c1f5020 msgr2=0x7f5d1c1f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d1b7fe640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d1c1f5020 0x7f5d1c1f5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d1b7fe640 1 -- 172.21.15.59:0/907237287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c06e060 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d1b7fe640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d1c1f7fe0 0x7f5d1c0c8400 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f5d100118e0 tx=0x7f5d10011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d18ff9640 1 -- 172.21.15.59:0/907237287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1001e070 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d18ff9640 1 -- 172.21.15.59:0/907237287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d1000f9a0 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d18ff9640 1 -- 172.21.15.59:0/907237287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1000fdd0 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1c06e340 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.647+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d1c1f4b00 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.648+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d1c0b56d0 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.649+0000 7f5d18ff9640 1 -- 172.21.15.59:0/907237287 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f5d10021070 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.649+0000 7f5d18ff9640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5d080763f0 0x7f5d08078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.649+0000 7f5d18ff9640 1 -- 172.21.15.59:0/907237287 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 6129+0+0 (secure 0 0 0) 0x7f5d10098660 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.650+0000 7f5d1bfff640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5d080763f0 0x7f5d08078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.650+0000 7f5d1bfff640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5d080763f0 0x7f5d08078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c033000 tx=0x7f5d1c0b60b0 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.654+0000 7f5d18ff9640 1 -- 172.21.15.59:0/907237287 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d10061510 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.844+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d1c0c8020 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.845+0000 7f5d18ff9640 1 -- 172.21.15.59:0/907237287 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) ==== 74+0+16961 (secure 0 0 0) 0x7f5d10065810 con 0x7f5d1c1f7fe0 2024-06-29T15:38:39.846 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:39.846 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":175,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:38.523116+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:36.882222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.468804+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5d080763f0 msgr2=0x7f5d08078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5d080763f0 0x7f5d08078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c033000 tx=0x7f5d1c0b60b0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d1c1f7fe0 msgr2=0x7f5d1c0c8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d1c1f7fe0 0x7f5d1c0c8400 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f5d100118e0 tx=0x7f5d10011da0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 shutdown_connections 2024-06-29T15:38:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d1c1f5020 0x7f5d1c1f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f5d080763f0 0x7f5d08078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d1c1f7fe0 0x7f5d1c0c8400 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 --2- 172.21.15.59:0/907237287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d04003c80 0x7f5d1c06f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 >> 172.21.15.59:0/907237287 conn(0x7f5d1c0b6fb0 msgr2=0x7f5d1c13e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 shutdown_connections 2024-06-29T15:38:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.848+0000 7f5d225ba640 1 -- 172.21.15.59:0/907237287 wait complete. 2024-06-29T15:38:39.860 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:38:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.981+0000 7f55209bd640 1 -- 172.21.15.59:0/1804413485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5508003c80 msgr2=0x7f55080040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.981+0000 7f55209bd640 1 --2- 172.21.15.59:0/1804413485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5508003c80 0x7f55080040d0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f550c006f80 tx=0x7f5508005830 comp rx=0 tx=0).stop 2024-06-29T15:38:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.981+0000 7f551ad9d640 1 -- 172.21.15.59:0/1804413485 reap_dead start 2024-06-29T15:38:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.981+0000 7f55209bd640 1 -- 172.21.15.59:0/1804413485 shutdown_connections 2024-06-29T15:38:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.981+0000 7f55209bd640 1 -- 172.21.15.59:0/1804413485 >> 172.21.15.59:0/1804413485 conn(0x7f551c0d3ee0 msgr2=0x7f551c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.981+0000 7f55209bd640 1 -- 172.21.15.59:0/1804413485 shutdown_connections 2024-06-29T15:38:39.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.981+0000 7f55209bd640 1 -- 172.21.15.59:0/1804413485 wait complete. 2024-06-29T15:38:39.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.983+0000 7f55209bd640 1 Processor -- start 2024-06-29T15:38:39.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.983+0000 7f55209bd640 1 -- start start 2024-06-29T15:38:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.984+0000 7f55209bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5508003c80 0x7f551c151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.984+0000 7f55209bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cd380 0x7f551c0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.984+0000 7f55209bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c0cd790 0x7f551c0c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.984+0000 7f55209bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f551c15e8d0 con 0x7f551c0cd380 2024-06-29T15:38:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.984+0000 7f55209bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f551c0b42b0 con 0x7f551c0cd790 2024-06-29T15:38:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.984+0000 7f55209bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f551c15ea50 con 0x7f5508003c80 2024-06-29T15:38:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cd380 0x7f551c0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cd380 0x7f551c0c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58000/0 (socket says 172.21.15.59:58000) 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 -- 172.21.15.59:0/323055709 learned_addr learned my addr 172.21.15.59:0/323055709 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f551a59c640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5508003c80 0x7f551c151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f551ad9d640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c0cd790 0x7f551c0c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 -- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5508003c80 msgr2=0x7f551c151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5508003c80 0x7f551c151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 -- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c0cd790 msgr2=0x7f551c0c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c0cd790 0x7f551c0c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 -- 172.21.15.59:0/323055709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c0c7a40 con 0x7f551c0cd380 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f5511d9b640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cd380 0x7f551c0c9440 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f55040119b0 tx=0x7f5504011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f55137fe640 1 -- 172.21.15.59:0/323055709 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550400fcf0 con 0x7f551c0cd380 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f55137fe640 1 -- 172.21.15.59:0/323055709 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5504005020 con 0x7f551c0cd380 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f55137fe640 1 -- 172.21.15.59:0/323055709 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504005470 con 0x7f551c0cd380 2024-06-29T15:38:39.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551c0c7d20 con 0x7f551c0cd380 2024-06-29T15:38:39.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.985+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551c0c82e0 con 0x7f551c0cd380 2024-06-29T15:38:39.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.986+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551c0c77b0 con 0x7f551c0cd380 2024-06-29T15:38:39.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.987+0000 7f55137fe640 1 -- 172.21.15.59:0/323055709 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f55040047a0 con 0x7f551c0cd380 2024-06-29T15:38:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.988+0000 7f55137fe640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f54fc0763f0 0x7f54fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.988+0000 7f55137fe640 1 -- 172.21.15.59:0/323055709 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 6129+0+0 (secure 0 0 0) 0x7f55040986d0 con 0x7f551c0cd380 2024-06-29T15:38:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.988+0000 7f551a59c640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f54fc0763f0 0x7f54fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.989+0000 7f551a59c640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f54fc0763f0 0x7f54fc078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f550c033000 tx=0x7f550c02d000 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:39.992+0000 7f55137fe640 1 -- 172.21.15.59:0/323055709 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5504061720 con 0x7f551c0cd380 2024-06-29T15:38:40.111 INFO:tasks.ceph.mds.c.smithi059.stderr:2024-06-29T15:38:40.111+0000 7f9f9f3d9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:38:40.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:40.247+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f551c0b3990 con 0x7f551c0cd380 2024-06-29T15:38:41.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.163+0000 7f55137fe640 1 -- 172.21.15.59:0/323055709 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v87) ==== 118+0+0 (secure 0 0 0) 0x7f5504065a20 con 0x7f551c0cd380 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f54fc0763f0 msgr2=0x7f54fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f54fc0763f0 0x7f54fc078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f550c033000 tx=0x7f550c02d000 comp rx=0 tx=0).stop 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cd380 msgr2=0x7f551c0c9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cd380 0x7f551c0c9440 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f55040119b0 tx=0x7f5504011e70 comp rx=0 tx=0).stop 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 shutdown_connections 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c0cd790 0x7f551c0c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f54fc0763f0 0x7f54fc078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cd380 0x7f551c0c9440 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 --2- 172.21.15.59:0/323055709 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5508003c80 0x7f551c151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.166+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 >> 172.21.15.59:0/323055709 conn(0x7f551c0d3ee0 msgr2=0x7f551c160150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:41.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.167+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 shutdown_connections 2024-06-29T15:38:41.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.167+0000 7f55209bd640 1 -- 172.21.15.59:0/323055709 wait complete. 2024-06-29T15:38:41.181 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:38:41.181 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:38:41.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.309+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/2750168348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8ac004d40 msgr2=0x7fb8ac005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.309+0000 7fb8ca0ef640 1 --2- 172.21.15.59:0/2750168348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8ac004d40 0x7fb8ac005150 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc005fd0 tx=0x7fb8ac005830 comp rx=0 tx=0).stop 2024-06-29T15:38:41.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.309+0000 7fb8c3fff640 1 -- 172.21.15.59:0/2750168348 reap_dead start 2024-06-29T15:38:41.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.310+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/2750168348 shutdown_connections 2024-06-29T15:38:41.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.310+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/2750168348 >> 172.21.15.59:0/2750168348 conn(0x7fb8c40b24c0 msgr2=0x7fb8c40cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:41.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.310+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/2750168348 shutdown_connections 2024-06-29T15:38:41.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.310+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/2750168348 wait complete. 2024-06-29T15:38:41.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.311+0000 7fb8ca0ef640 1 Processor -- start 2024-06-29T15:38:41.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.312+0000 7fb8ca0ef640 1 -- start start 2024-06-29T15:38:41.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8ca0ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8ac004d40 0x7fb8c406c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:41.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8ca0ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c41436d0 0x7fb8c406c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:41.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8ca0ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8c4153740 0x7fb8c413e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8ca0ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c40733f0 con 0x7fb8c41436d0 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8ca0ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c4073270 con 0x7fb8ac004d40 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8ca0ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c4073570 con 0x7fb8c4153740 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c41436d0 0x7fb8c406c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c41436d0 0x7fb8c406c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58020/0 (socket says 172.21.15.59:58020) 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c2ffd640 1 -- 172.21.15.59:0/745267951 learned_addr learned my addr 172.21.15.59:0/745267951 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c37fe640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8ac004d40 0x7fb8c406c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c3fff640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8c4153740 0x7fb8c413e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c2ffd640 1 -- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8c4153740 msgr2=0x7fb8c413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c2ffd640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8c4153740 0x7fb8c413e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c2ffd640 1 -- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8ac004d40 msgr2=0x7fb8c406c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c2ffd640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8ac004d40 0x7fb8c406c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c2ffd640 1 -- 172.21.15.59:0/745267951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c41f56c0 con 0x7fb8c41436d0 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.313+0000 7fb8c37fe640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8ac004d40 0x7fb8c406c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.314+0000 7fb8c2ffd640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c41436d0 0x7fb8c406c720 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8011b10 tx=0x7fb8b8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.314+0000 7fb8abfff640 1 -- 172.21.15.59:0/745267951 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b800fc20 con 0x7fb8c41436d0 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.314+0000 7fb8abfff640 1 -- 172.21.15.59:0/745267951 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8b800fdc0 con 0x7fb8c41436d0 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.314+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c41f7b80 con 0x7fb8c41436d0 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.314+0000 7fb8abfff640 1 -- 172.21.15.59:0/745267951 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8004280 con 0x7fb8c41436d0 2024-06-29T15:38:41.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.314+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8c41f8140 con 0x7fb8c41436d0 2024-06-29T15:38:41.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.315+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8c415f240 con 0x7fb8c41436d0 2024-06-29T15:38:41.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.315+0000 7fb8abfff640 1 -- 172.21.15.59:0/745267951 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fb8b80035f0 con 0x7fb8c41436d0 2024-06-29T15:38:41.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.316+0000 7fb8abfff640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb8b0076420 0x7fb8b0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:41.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.316+0000 7fb8abfff640 1 -- 172.21.15.59:0/745267951 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7fb8b8098550 con 0x7fb8c41436d0 2024-06-29T15:38:41.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.316+0000 7fb8c37fe640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb8b0076420 0x7fb8b0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:41.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.317+0000 7fb8c37fe640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb8b0076420 0x7fb8b0078930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4006b30 tx=0x7fb8b40067a0 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:41.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.320+0000 7fb8abfff640 1 -- 172.21.15.59:0/745267951 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb8b8061ae0 con 0x7fb8c41436d0 2024-06-29T15:38:41.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:41.553+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fb8c415e770 con 0x7fb8c41436d0 2024-06-29T15:38:42.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.166+0000 7fb8abfff640 1 -- 172.21.15.59:0/745267951 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v88) ==== 122+0+0 (secure 0 0 0) 0x7fb8b80613e0 con 0x7fb8c41436d0 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.168+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb8b0076420 msgr2=0x7fb8b0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.168+0000 7fb8ca0ef640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb8b0076420 0x7fb8b0078930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4006b30 tx=0x7fb8b40067a0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.168+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c41436d0 msgr2=0x7fb8c406c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c41436d0 0x7fb8c406c720 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8011b10 tx=0x7fb8b8009f10 comp rx=0 tx=0).stop 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 shutdown_connections 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb8c4153740 0x7fb8c413e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fb8b0076420 0x7fb8b0078930 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8c41436d0 0x7fb8c406c720 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 --2- 172.21.15.59:0/745267951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8ac004d40 0x7fb8c406c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 >> 172.21.15.59:0/745267951 conn(0x7fb8c40b24c0 msgr2=0x7fb8c415c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 shutdown_connections 2024-06-29T15:38:42.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.169+0000 7fb8ca0ef640 1 -- 172.21.15.59:0/745267951 wait complete. 2024-06-29T15:38:42.182 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:38:42.182 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.304+0000 7f60ec5f5640 1 -- 172.21.15.59:0/1213803588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40b2fa0 msgr2=0x7f60e40d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.304+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/1213803588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40b2fa0 0x7f60e40d1520 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f60d8005fd0 tx=0x7f60d802a530 comp rx=0 tx=0).stop 2024-06-29T15:38:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.304+0000 7f60ec5f5640 1 -- 172.21.15.59:0/1213803588 shutdown_connections 2024-06-29T15:38:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.304+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/1213803588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40b2fa0 0x7f60e40d1520 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.304+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/1213803588 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60e40b2270 0x7f60e415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.304+0000 7f60ec5f5640 1 -- 172.21.15.59:0/1213803588 >> 172.21.15.59:0/1213803588 conn(0x7f60e4073e10 msgr2=0x7f60e4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.305+0000 7f60ec5f5640 1 -- 172.21.15.59:0/1213803588 shutdown_connections 2024-06-29T15:38:42.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.305+0000 7f60ec5f5640 1 -- 172.21.15.59:0/1213803588 wait complete. 2024-06-29T15:38:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.306+0000 7f60ec5f5640 1 Processor -- start 2024-06-29T15:38:42.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.306+0000 7f60ec5f5640 1 -- start start 2024-06-29T15:38:42.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60ec5f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60e40b2270 0x7f60e415c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60ec5f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60e40b2fa0 0x7f60e4147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60ec5f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40d1a60 0x7f60e4146130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60ec5f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f60e40b6cc0 con 0x7f60e40d1a60 2024-06-29T15:38:42.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60ec5f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60e40b6b40 con 0x7f60e40b2fa0 2024-06-29T15:38:42.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60ec5f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f60e40cfe90 con 0x7f60e40b2270 2024-06-29T15:38:42.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40d1a60 0x7f60e4146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40d1a60 0x7f60e4146130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58040/0 (socket says 172.21.15.59:58040) 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 -- 172.21.15.59:0/3465507875 learned_addr learned my addr 172.21.15.59:0/3465507875 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60ea391640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60e40b2270 0x7f60e415c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60e9b90640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60e40b2fa0 0x7f60e4147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 -- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60e40b2270 msgr2=0x7f60e415c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60e40b2270 0x7f60e415c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 -- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60e40b2fa0 msgr2=0x7f60e4147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60e40b2fa0 0x7f60e4147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 -- 172.21.15.59:0/3465507875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e4147a80 con 0x7f60e40d1a60 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60e9b90640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60e40b2fa0 0x7f60e4147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60eab92640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40d1a60 0x7f60e4146130 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00b440 tx=0x7f60dc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60e2ffd640 1 -- 172.21.15.59:0/3465507875 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc015020 con 0x7f60e40d1a60 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.308+0000 7f60e2ffd640 1 -- 172.21.15.59:0/3465507875 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60dc011920 con 0x7f60e40d1a60 2024-06-29T15:38:42.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.309+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e4145090 con 0x7f60e40d1a60 2024-06-29T15:38:42.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.309+0000 7f60e2ffd640 1 -- 172.21.15.59:0/3465507875 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc011d70 con 0x7f60e40d1a60 2024-06-29T15:38:42.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.309+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e4145650 con 0x7f60e40d1a60 2024-06-29T15:38:42.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.309+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e40d03b0 con 0x7f60e40d1a60 2024-06-29T15:38:42.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.310+0000 7f60e2ffd640 1 -- 172.21.15.59:0/3465507875 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f60dc018020 con 0x7f60e40d1a60 2024-06-29T15:38:42.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.311+0000 7f60e2ffd640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f60d0076420 0x7f60d0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.311+0000 7f60e2ffd640 1 -- 172.21.15.59:0/3465507875 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f60dc0979f0 con 0x7f60e40d1a60 2024-06-29T15:38:42.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.311+0000 7f60ea391640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f60d0076420 0x7f60d0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.312+0000 7f60ea391640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f60d0076420 0x7f60d0078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f60d400ae50 tx=0x7f60d400a540 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:42.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.315+0000 7f60e2ffd640 1 -- 172.21.15.59:0/3465507875 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f60dc01f020 con 0x7f60e40d1a60 2024-06-29T15:38:42.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.546+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60e41597d0 con 0x7f60e40d1a60 2024-06-29T15:38:42.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.547+0000 7f60e2ffd640 1 -- 172.21.15.59:0/3465507875 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5233 (secure 0 0 0) 0x7f60dc060f60 con 0x7f60e40d1a60 2024-06-29T15:38:42.547 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:42.547 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"btime":"2024-06-29T15:38:42:164025+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1947749735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1947749735},{"type":"v1","addr":"172.21.15.59:6835","nonce":1947749735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3010048644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3010048644},{"type":"v1","addr":"172.21.15.153:6836","nonce":3010048644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9283,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1912783986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1912783986},{"type":"v1","addr":"172.21.15.153:6837","nonce":1912783986}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.523073+0000","modified":"2024-06-29T15:38:41.554713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1057565935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1057565935},{"type":"v1","addr":"172.21.15.59:6834","nonce":1057565935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9894,"qdb_cluster":[9894]},"id":10}]} 2024-06-29T15:38:42.548 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f60d0076420 msgr2=0x7f60d0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f60d0076420 0x7f60d0078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f60d400ae50 tx=0x7f60d400a540 comp rx=0 tx=0).stop 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40d1a60 msgr2=0x7f60e4146130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40d1a60 0x7f60e4146130 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00b440 tx=0x7f60dc00b900 comp rx=0 tx=0).stop 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 shutdown_connections 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f60d0076420 0x7f60d0078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60e40d1a60 0x7f60e4146130 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60e40b2fa0 0x7f60e4147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 --2- 172.21.15.59:0/3465507875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60e40b2270 0x7f60e415c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 >> 172.21.15.59:0/3465507875 conn(0x7f60e4073e10 msgr2=0x7f60e40cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 shutdown_connections 2024-06-29T15:38:42.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.549+0000 7f60ec5f5640 1 -- 172.21.15.59:0/3465507875 wait complete. 2024-06-29T15:38:42.561 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:42.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.678+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/1961729494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780d1c30 msgr2=0x7f2b780b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.678+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/1961729494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780d1c30 0x7f2b780b5b30 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f2b68005fd0 tx=0x7f2b6802aa00 comp rx=0 tx=0).stop 2024-06-29T15:38:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.678+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/1961729494 shutdown_connections 2024-06-29T15:38:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.678+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/1961729494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780d1c30 0x7f2b780b5b30 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.678+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/1961729494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b780b2270 0x7f2b7815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.678+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/1961729494 >> 172.21.15.59:0/1961729494 conn(0x7f2b78073e10 msgr2=0x7f2b78074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.678+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/1961729494 shutdown_connections 2024-06-29T15:38:42.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.678+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/1961729494 wait complete. 2024-06-29T15:38:42.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.680+0000 7f2b7d0f6640 1 Processor -- start 2024-06-29T15:38:42.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.680+0000 7f2b7d0f6640 1 -- start start 2024-06-29T15:38:42.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.681+0000 7f2b7d0f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b780b2270 0x7f2b7806f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.681+0000 7f2b7d0f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780b2fa0 0x7f2b7814dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.681+0000 7f2b7d0f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b780d1c30 0x7f2b78158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.681+0000 7f2b7d0f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2b780b6400 con 0x7f2b780b2fa0 2024-06-29T15:38:42.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.681+0000 7f2b7d0f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b780b6280 con 0x7f2b780b2270 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.681+0000 7f2b7d0f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2b780b6580 con 0x7f2b780d1c30 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780b2fa0 0x7f2b7814dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780b2fa0 0x7f2b7814dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58062/0 (socket says 172.21.15.59:58062) 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 -- 172.21.15.59:0/2328156282 learned_addr learned my addr 172.21.15.59:0/2328156282 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7759e640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b780d1c30 0x7f2b78158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b76d9d640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b780b2270 0x7f2b7806f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 -- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b780d1c30 msgr2=0x7f2b78158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b780d1c30 0x7f2b78158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 -- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b780b2270 msgr2=0x7f2b7806f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b780b2270 0x7f2b7806f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 -- 172.21.15.59:0/2328156282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7806db80 con 0x7f2b780b2fa0 2024-06-29T15:38:42.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7659c640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780b2fa0 0x7f2b7814dd10 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000b3e0 tx=0x7f2b60009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:42.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b6f7fe640 1 -- 172.21.15.59:0/2328156282 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60015030 con 0x7f2b780b2fa0 2024-06-29T15:38:42.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b6f7fe640 1 -- 172.21.15.59:0/2328156282 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b60004110 con 0x7f2b780b2fa0 2024-06-29T15:38:42.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7814e250 con 0x7f2b780b2fa0 2024-06-29T15:38:42.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.682+0000 7f2b6f7fe640 1 -- 172.21.15.59:0/2328156282 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60011630 con 0x7f2b780b2fa0 2024-06-29T15:38:42.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.683+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b781f9680 con 0x7f2b780b2fa0 2024-06-29T15:38:42.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.684+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b780b6ae0 con 0x7f2b780b2fa0 2024-06-29T15:38:42.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.684+0000 7f2b6f7fe640 1 -- 172.21.15.59:0/2328156282 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f2b60018070 con 0x7f2b780b2fa0 2024-06-29T15:38:42.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.685+0000 7f2b6f7fe640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2b580763f0 0x7f2b58078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.685+0000 7f2b6f7fe640 1 -- 172.21.15.59:0/2328156282 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f2b60097530 con 0x7f2b780b2fa0 2024-06-29T15:38:42.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.686+0000 7f2b76d9d640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2b580763f0 0x7f2b58078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.686+0000 7f2b76d9d640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2b580763f0 0x7f2b58078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b700058c0 tx=0x7f2b70005f60 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:42.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.690+0000 7f2b6f7fe640 1 -- 172.21.15.59:0/2328156282 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2b600604f0 con 0x7f2b780b2fa0 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.881+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b78071990 con 0x7f2b780b2fa0 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.882+0000 7f2b6f7fe640 1 -- 172.21.15.59:0/2328156282 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) ==== 74+0+16948 (secure 0 0 0) 0x7f2b600647f0 con 0x7f2b780b2fa0 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":176,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:40.151277+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:36.882222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.468804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2b580763f0 msgr2=0x7f2b58078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2b580763f0 0x7f2b58078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b700058c0 tx=0x7f2b70005f60 comp rx=0 tx=0).stop 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780b2fa0 msgr2=0x7f2b7814dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780b2fa0 0x7f2b7814dd10 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000b3e0 tx=0x7f2b60009f90 comp rx=0 tx=0).stop 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 shutdown_connections 2024-06-29T15:38:42.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2b780d1c30 0x7f2b78158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f2b580763f0 0x7f2b58078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b780b2fa0 0x7f2b7814dd10 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 --2- 172.21.15.59:0/2328156282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b780b2270 0x7f2b7806f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 >> 172.21.15.59:0/2328156282 conn(0x7f2b78073e10 msgr2=0x7f2b78070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 shutdown_connections 2024-06-29T15:38:42.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:42.884+0000 7f2b7d0f6640 1 -- 172.21.15.59:0/2328156282 wait complete. 2024-06-29T15:38:42.896 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:38:42.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:38:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.014+0000 7f4da8ffc640 1 -- 172.21.15.59:0/1965892321 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40b8be0 msgr2=0x7f4da41629d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.014+0000 7f4da8ffc640 1 --2- 172.21.15.59:0/1965892321 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40b8be0 0x7f4da41629d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f4da40b78c0 tx=0x7f4d9c02a840 comp rx=0 tx=0).stop 2024-06-29T15:38:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.014+0000 7f4da8ffc640 1 -- 172.21.15.59:0/1965892321 shutdown_connections 2024-06-29T15:38:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.014+0000 7f4da8ffc640 1 --2- 172.21.15.59:0/1965892321 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40b8be0 0x7f4da41629d0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.014+0000 7f4da8ffc640 1 -- 172.21.15.59:0/1965892321 >> 172.21.15.59:0/1965892321 conn(0x7f4da40b5610 msgr2=0x7f4da40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.014+0000 7f4da8ffc640 1 -- 172.21.15.59:0/1965892321 shutdown_connections 2024-06-29T15:38:43.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.014+0000 7f4da8ffc640 1 -- 172.21.15.59:0/1965892321 wait complete. 2024-06-29T15:38:43.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.016+0000 7f4da8ffc640 1 Processor -- start 2024-06-29T15:38:43.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.016+0000 7f4da8ffc640 1 -- start start 2024-06-29T15:38:43.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da8ffc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da408c190 0x7f4da4161490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da8ffc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4da40b8be0 0x7f4da4157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da8ffc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40ce950 0x7f4da415a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da8ffc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4da4169690 con 0x7f4da40ce950 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da8ffc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4da4169510 con 0x7f4da408c190 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da8ffc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4da4169810 con 0x7f4da40b8be0 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40ce950 0x7f4da415a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40ce950 0x7f4da415a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56730/0 (socket says 172.21.15.59:56730) 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da2d9d640 1 -- 172.21.15.59:0/2359517696 learned_addr learned my addr 172.21.15.59:0/2359517696 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da259c640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da408c190 0x7f4da4161490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.017+0000 7f4da1d9b640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4da40b8be0 0x7f4da4157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4da2d9d640 1 -- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4da40b8be0 msgr2=0x7f4da4157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4da2d9d640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4da40b8be0 0x7f4da4157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4da2d9d640 1 -- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da408c190 msgr2=0x7f4da4161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4da2d9d640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da408c190 0x7f4da4161490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4da2d9d640 1 -- 172.21.15.59:0/2359517696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da4158090 con 0x7f4da40ce950 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4da2d9d640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40ce950 0x7f4da415a0f0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c035040 tx=0x7f4d9c033e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4d92ffd640 1 -- 172.21.15.59:0/2359517696 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c042020 con 0x7f4da40ce950 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4d92ffd640 1 -- 172.21.15.59:0/2359517696 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d9c004330 con 0x7f4da40ce950 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da40cae20 con 0x7f4da40ce950 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4d92ffd640 1 -- 172.21.15.59:0/2359517696 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c037040 con 0x7f4da40ce950 2024-06-29T15:38:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.018+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da40c7920 con 0x7f4da40ce950 2024-06-29T15:38:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.019+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da40cf080 con 0x7f4da40ce950 2024-06-29T15:38:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.020+0000 7f4d92ffd640 1 -- 172.21.15.59:0/2359517696 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f4d9c045020 con 0x7f4da40ce950 2024-06-29T15:38:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.020+0000 7f4d92ffd640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f4d840763f0 0x7f4d84078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.020+0000 7f4d92ffd640 1 -- 172.21.15.59:0/2359517696 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f4d9c0b8b30 con 0x7f4da40ce950 2024-06-29T15:38:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.021+0000 7f4da259c640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f4d840763f0 0x7f4d84078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.021+0000 7f4da259c640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f4d840763f0 0x7f4d84078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4d94006fd0 tx=0x7f4d94009060 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.025+0000 7f4d92ffd640 1 -- 172.21.15.59:0/2359517696 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d9c04c030 con 0x7f4da40ce950 2024-06-29T15:38:43.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.213+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 --> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4da4157e20 con 0x7f4d840763f0 2024-06-29T15:38:43.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.218+0000 7f4d92ffd640 1 -- 172.21.15.59:0/2359517696 <== mgr.8039 v2:172.21.15.153:6832/2627795750 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252472 (secure 0 0 0) 0x7f4da4157e20 con 0x7f4d840763f0 2024-06-29T15:38:43.221 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:43.222 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:38:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f4d840763f0 msgr2=0x7f4d84078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f4d840763f0 0x7f4d84078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4d94006fd0 tx=0x7f4d94009060 comp rx=0 tx=0).stop 2024-06-29T15:38:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40ce950 msgr2=0x7f4da415a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40ce950 0x7f4da415a0f0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c035040 tx=0x7f4d9c033e80 comp rx=0 tx=0).stop 2024-06-29T15:38:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 shutdown_connections 2024-06-29T15:38:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f4d840763f0 0x7f4d84078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4da40ce950 0x7f4da415a0f0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4da40b8be0 0x7f4da4157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 --2- 172.21.15.59:0/2359517696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4da408c190 0x7f4da4161490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 >> 172.21.15.59:0/2359517696 conn(0x7f4da40b5610 msgr2=0x7f4da406cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 shutdown_connections 2024-06-29T15:38:43.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.221+0000 7f4da8ffc640 1 -- 172.21.15.59:0/2359517696 wait complete. 2024-06-29T15:38:43.233 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:38:42.828383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":338,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":369052,"kb_used_data":18284,"kb_used_omap":12,"kb_used_meta":350643,"kb_avail":754605668,"statfs":{"total":773094113280,"available":772716204032,"internally_reserved":0,"allocated":18722816,"data_stored":10105348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":359058994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.710772"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167034+0000","last_change":"2024-06-29T15:38:39.148245+0000","last_active":"2024-06-29T15:38:40.167034+0000","last_peered":"2024-06-29T15:38:40.167034+0000","last_clean":"2024-06-29T15:38:40.167034+0000","last_became_active":"2024-06-29T15:38:39.147398+0000","last_became_peered":"2024-06-29T15:38:39.147398+0000","last_unstale":"2024-06-29T15:38:40.167034+0000","last_undegraded":"2024-06-29T15:38:40.167034+0000","last_fullsized":"2024-06-29T15:38:40.167034+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:11:20.353442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154787+0000","last_change":"2024-06-29T15:38:39.143268+0000","last_active":"2024-06-29T15:38:40.154787+0000","last_peered":"2024-06-29T15:38:40.154787+0000","last_clean":"2024-06-29T15:38:40.154787+0000","last_became_active":"2024-06-29T15:38:39.142921+0000","last_became_peered":"2024-06-29T15:38:39.142921+0000","last_unstale":"2024-06-29T15:38:40.154787+0000","last_undegraded":"2024-06-29T15:38:40.154787+0000","last_fullsized":"2024-06-29T15:38:40.154787+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:12.898181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154435+0000","last_change":"2024-06-29T15:38:39.151169+0000","last_active":"2024-06-29T15:38:40.154435+0000","last_peered":"2024-06-29T15:38:40.154435+0000","last_clean":"2024-06-29T15:38:40.154435+0000","last_became_active":"2024-06-29T15:38:39.150852+0000","last_became_peered":"2024-06-29T15:38:39.150852+0000","last_unstale":"2024-06-29T15:38:40.154435+0000","last_undegraded":"2024-06-29T15:38:40.154435+0000","last_fullsized":"2024-06-29T15:38:40.154435+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:35:53.798820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155803+0000","last_change":"2024-06-29T15:38:39.151072+0000","last_active":"2024-06-29T15:38:40.155803+0000","last_peered":"2024-06-29T15:38:40.155803+0000","last_clean":"2024-06-29T15:38:40.155803+0000","last_became_active":"2024-06-29T15:38:39.150733+0000","last_became_peered":"2024-06-29T15:38:39.150733+0000","last_unstale":"2024-06-29T15:38:40.155803+0000","last_undegraded":"2024-06-29T15:38:40.155803+0000","last_fullsized":"2024-06-29T15:38:40.155803+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:51.775965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.160732+0000","last_change":"2024-06-29T15:38:39.158840+0000","last_active":"2024-06-29T15:38:39.160732+0000","last_peered":"2024-06-29T15:38:39.160732+0000","last_clean":"2024-06-29T15:38:39.160732+0000","last_became_active":"2024-06-29T15:38:39.158536+0000","last_became_peered":"2024-06-29T15:38:39.158536+0000","last_unstale":"2024-06-29T15:38:39.160732+0000","last_undegraded":"2024-06-29T15:38:39.160732+0000","last_fullsized":"2024-06-29T15:38:39.160732+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:41.407945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.160760+0000","last_change":"2024-06-29T15:38:39.159678+0000","last_active":"2024-06-29T15:38:39.160760+0000","last_peered":"2024-06-29T15:38:39.160760+0000","last_clean":"2024-06-29T15:38:39.160760+0000","last_became_active":"2024-06-29T15:38:39.159352+0000","last_became_peered":"2024-06-29T15:38:39.159352+0000","last_unstale":"2024-06-29T15:38:39.160760+0000","last_undegraded":"2024-06-29T15:38:39.160760+0000","last_fullsized":"2024-06-29T15:38:39.160760+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:34.550503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166788+0000","last_change":"2024-06-29T15:38:39.153277+0000","last_active":"2024-06-29T15:38:40.166788+0000","last_peered":"2024-06-29T15:38:40.166788+0000","last_clean":"2024-06-29T15:38:40.166788+0000","last_became_active":"2024-06-29T15:38:39.152953+0000","last_became_peered":"2024-06-29T15:38:39.152953+0000","last_unstale":"2024-06-29T15:38:40.166788+0000","last_undegraded":"2024-06-29T15:38:40.166788+0000","last_fullsized":"2024-06-29T15:38:40.166788+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:03.933083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155472+0000","last_change":"2024-06-29T15:38:39.156812+0000","last_active":"2024-06-29T15:38:40.155472+0000","last_peered":"2024-06-29T15:38:40.155472+0000","last_clean":"2024-06-29T15:38:40.155472+0000","last_became_active":"2024-06-29T15:38:39.156230+0000","last_became_peered":"2024-06-29T15:38:39.156230+0000","last_unstale":"2024-06-29T15:38:40.155472+0000","last_undegraded":"2024-06-29T15:38:40.155472+0000","last_fullsized":"2024-06-29T15:38:40.155472+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:02.681259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.161163+0000","last_change":"2024-06-29T15:38:39.158599+0000","last_active":"2024-06-29T15:38:39.161163+0000","last_peered":"2024-06-29T15:38:39.161163+0000","last_clean":"2024-06-29T15:38:39.161163+0000","last_became_active":"2024-06-29T15:38:39.158267+0000","last_became_peered":"2024-06-29T15:38:39.158267+0000","last_unstale":"2024-06-29T15:38:39.161163+0000","last_undegraded":"2024-06-29T15:38:39.161163+0000","last_fullsized":"2024-06-29T15:38:39.161163+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:30:11.113187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.161859+0000","last_change":"2024-06-29T15:38:39.160534+0000","last_active":"2024-06-29T15:38:39.161859+0000","last_peered":"2024-06-29T15:38:39.161859+0000","last_clean":"2024-06-29T15:38:39.161859+0000","last_became_active":"2024-06-29T15:38:39.160148+0000","last_became_peered":"2024-06-29T15:38:39.160148+0000","last_unstale":"2024-06-29T15:38:39.161859+0000","last_undegraded":"2024-06-29T15:38:39.161859+0000","last_fullsized":"2024-06-29T15:38:39.161859+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:26.668399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164424+0000","last_change":"2024-06-29T15:38:39.159356+0000","last_active":"2024-06-29T15:38:39.164424+0000","last_peered":"2024-06-29T15:38:39.164424+0000","last_clean":"2024-06-29T15:38:39.164424+0000","last_became_active":"2024-06-29T15:38:39.159044+0000","last_became_peered":"2024-06-29T15:38:39.159044+0000","last_unstale":"2024-06-29T15:38:39.164424+0000","last_undegraded":"2024-06-29T15:38:39.164424+0000","last_fullsized":"2024-06-29T15:38:39.164424+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:26:40.028297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155830+0000","last_change":"2024-06-29T15:38:39.154464+0000","last_active":"2024-06-29T15:38:40.155830+0000","last_peered":"2024-06-29T15:38:40.155830+0000","last_clean":"2024-06-29T15:38:40.155830+0000","last_became_active":"2024-06-29T15:38:39.153957+0000","last_became_peered":"2024-06-29T15:38:39.153957+0000","last_unstale":"2024-06-29T15:38:40.155830+0000","last_undegraded":"2024-06-29T15:38:40.155830+0000","last_fullsized":"2024-06-29T15:38:40.155830+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:10.999985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.162117+0000","last_change":"2024-06-29T15:38:39.160768+0000","last_active":"2024-06-29T15:38:39.162117+0000","last_peered":"2024-06-29T15:38:39.162117+0000","last_clean":"2024-06-29T15:38:39.162117+0000","last_became_active":"2024-06-29T15:38:39.160329+0000","last_became_peered":"2024-06-29T15:38:39.160329+0000","last_unstale":"2024-06-29T15:38:39.162117+0000","last_undegraded":"2024-06-29T15:38:39.162117+0000","last_fullsized":"2024-06-29T15:38:39.162117+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:18:51.865063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155362+0000","last_change":"2024-06-29T15:38:39.154417+0000","last_active":"2024-06-29T15:38:40.155362+0000","last_peered":"2024-06-29T15:38:40.155362+0000","last_clean":"2024-06-29T15:38:40.155362+0000","last_became_active":"2024-06-29T15:38:39.153946+0000","last_became_peered":"2024-06-29T15:38:39.153946+0000","last_unstale":"2024-06-29T15:38:40.155362+0000","last_undegraded":"2024-06-29T15:38:40.155362+0000","last_fullsized":"2024-06-29T15:38:40.155362+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:43.946244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154406+0000","last_change":"2024-06-29T15:38:39.143494+0000","last_active":"2024-06-29T15:38:40.154406+0000","last_peered":"2024-06-29T15:38:40.154406+0000","last_clean":"2024-06-29T15:38:40.154406+0000","last_became_active":"2024-06-29T15:38:39.142463+0000","last_became_peered":"2024-06-29T15:38:39.142463+0000","last_unstale":"2024-06-29T15:38:40.154406+0000","last_undegraded":"2024-06-29T15:38:40.154406+0000","last_fullsized":"2024-06-29T15:38:40.154406+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:39.264607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.165166+0000","last_change":"2024-06-29T15:38:39.163630+0000","last_active":"2024-06-29T15:38:39.165166+0000","last_peered":"2024-06-29T15:38:39.165166+0000","last_clean":"2024-06-29T15:38:39.165166+0000","last_became_active":"2024-06-29T15:38:39.163314+0000","last_became_peered":"2024-06-29T15:38:39.163314+0000","last_unstale":"2024-06-29T15:38:39.165166+0000","last_undegraded":"2024-06-29T15:38:39.165166+0000","last_fullsized":"2024-06-29T15:38:39.165166+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:03.933083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156173+0000","last_change":"2024-06-29T15:38:39.149776+0000","last_active":"2024-06-29T15:38:40.156173+0000","last_peered":"2024-06-29T15:38:40.156173+0000","last_clean":"2024-06-29T15:38:40.156173+0000","last_became_active":"2024-06-29T15:38:39.149457+0000","last_became_peered":"2024-06-29T15:38:39.149457+0000","last_unstale":"2024-06-29T15:38:40.156173+0000","last_undegraded":"2024-06-29T15:38:40.156173+0000","last_fullsized":"2024-06-29T15:38:40.156173+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:04:09.294171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164377+0000","last_change":"2024-06-29T15:38:39.162226+0000","last_active":"2024-06-29T15:38:39.164377+0000","last_peered":"2024-06-29T15:38:39.164377+0000","last_clean":"2024-06-29T15:38:39.164377+0000","last_became_active":"2024-06-29T15:38:39.160999+0000","last_became_peered":"2024-06-29T15:38:39.160999+0000","last_unstale":"2024-06-29T15:38:39.164377+0000","last_undegraded":"2024-06-29T15:38:39.164377+0000","last_fullsized":"2024-06-29T15:38:39.164377+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:10:21.982586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169086+0000","last_change":"2024-06-29T15:38:39.150984+0000","last_active":"2024-06-29T15:38:40.169086+0000","last_peered":"2024-06-29T15:38:40.169086+0000","last_clean":"2024-06-29T15:38:40.169086+0000","last_became_active":"2024-06-29T15:38:39.149691+0000","last_became_peered":"2024-06-29T15:38:39.149691+0000","last_unstale":"2024-06-29T15:38:40.169086+0000","last_undegraded":"2024-06-29T15:38:40.169086+0000","last_fullsized":"2024-06-29T15:38:40.169086+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:10.383784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.165920+0000","last_change":"2024-06-29T15:38:39.163067+0000","last_active":"2024-06-29T15:38:39.165920+0000","last_peered":"2024-06-29T15:38:39.165920+0000","last_clean":"2024-06-29T15:38:39.165920+0000","last_became_active":"2024-06-29T15:38:39.162707+0000","last_became_peered":"2024-06-29T15:38:39.162707+0000","last_unstale":"2024-06-29T15:38:39.165920+0000","last_undegraded":"2024-06-29T15:38:39.165920+0000","last_fullsized":"2024-06-29T15:38:39.165920+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:36.688998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164703+0000","last_change":"2024-06-29T15:38:39.163087+0000","last_active":"2024-06-29T15:38:39.164703+0000","last_peered":"2024-06-29T15:38:39.164703+0000","last_clean":"2024-06-29T15:38:39.164703+0000","last_became_active":"2024-06-29T15:38:39.162735+0000","last_became_peered":"2024-06-29T15:38:39.162735+0000","last_unstale":"2024-06-29T15:38:39.164703+0000","last_undegraded":"2024-06-29T15:38:39.164703+0000","last_fullsized":"2024-06-29T15:38:39.164703+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:18:31.626323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166814+0000","last_change":"2024-06-29T15:38:39.152932+0000","last_active":"2024-06-29T15:38:40.166814+0000","last_peered":"2024-06-29T15:38:40.166814+0000","last_clean":"2024-06-29T15:38:40.166814+0000","last_became_active":"2024-06-29T15:38:39.152598+0000","last_became_peered":"2024-06-29T15:38:39.152598+0000","last_unstale":"2024-06-29T15:38:40.166814+0000","last_undegraded":"2024-06-29T15:38:40.166814+0000","last_fullsized":"2024-06-29T15:38:40.166814+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:44.861477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155539+0000","last_change":"2024-06-29T15:38:39.150934+0000","last_active":"2024-06-29T15:38:40.155539+0000","last_peered":"2024-06-29T15:38:40.155539+0000","last_clean":"2024-06-29T15:38:40.155539+0000","last_became_active":"2024-06-29T15:38:39.150597+0000","last_became_peered":"2024-06-29T15:38:39.150597+0000","last_unstale":"2024-06-29T15:38:40.155539+0000","last_undegraded":"2024-06-29T15:38:40.155539+0000","last_fullsized":"2024-06-29T15:38:40.155539+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:38.700745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.157297+0000","last_change":"2024-06-29T15:38:39.154955+0000","last_active":"2024-06-29T15:38:39.157297+0000","last_peered":"2024-06-29T15:38:39.157297+0000","last_clean":"2024-06-29T15:38:39.157297+0000","last_became_active":"2024-06-29T15:38:39.154618+0000","last_became_peered":"2024-06-29T15:38:39.154618+0000","last_unstale":"2024-06-29T15:38:39.157297+0000","last_undegraded":"2024-06-29T15:38:39.157297+0000","last_fullsized":"2024-06-29T15:38:39.157297+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:13:26.331444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167047+0000","last_change":"2024-06-29T15:38:39.148536+0000","last_active":"2024-06-29T15:38:40.167047+0000","last_peered":"2024-06-29T15:38:40.167047+0000","last_clean":"2024-06-29T15:38:40.167047+0000","last_became_active":"2024-06-29T15:38:39.147923+0000","last_became_peered":"2024-06-29T15:38:39.147923+0000","last_unstale":"2024-06-29T15:38:40.167047+0000","last_undegraded":"2024-06-29T15:38:40.167047+0000","last_fullsized":"2024-06-29T15:38:40.167047+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:32.659808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166963+0000","last_change":"2024-06-29T15:38:39.156524+0000","last_active":"2024-06-29T15:38:40.166963+0000","last_peered":"2024-06-29T15:38:40.166963+0000","last_clean":"2024-06-29T15:38:40.166963+0000","last_became_active":"2024-06-29T15:38:39.156242+0000","last_became_peered":"2024-06-29T15:38:39.156242+0000","last_unstale":"2024-06-29T15:38:40.166963+0000","last_undegraded":"2024-06-29T15:38:40.166963+0000","last_fullsized":"2024-06-29T15:38:40.166963+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:40.455445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.165128+0000","last_change":"2024-06-29T15:38:39.163694+0000","last_active":"2024-06-29T15:38:39.165128+0000","last_peered":"2024-06-29T15:38:39.165128+0000","last_clean":"2024-06-29T15:38:39.165128+0000","last_became_active":"2024-06-29T15:38:39.163358+0000","last_became_peered":"2024-06-29T15:38:39.163358+0000","last_unstale":"2024-06-29T15:38:39.165128+0000","last_undegraded":"2024-06-29T15:38:39.165128+0000","last_fullsized":"2024-06-29T15:38:39.165128+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:44.861477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.155278+0000","last_change":"2024-06-29T15:38:39.153537+0000","last_active":"2024-06-29T15:38:39.155278+0000","last_peered":"2024-06-29T15:38:39.155278+0000","last_clean":"2024-06-29T15:38:39.155278+0000","last_became_active":"2024-06-29T15:38:39.153149+0000","last_became_peered":"2024-06-29T15:38:39.153149+0000","last_unstale":"2024-06-29T15:38:39.155278+0000","last_undegraded":"2024-06-29T15:38:39.155278+0000","last_fullsized":"2024-06-29T15:38:39.155278+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:18.095124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156658+0000","last_change":"2024-06-29T15:38:39.157793+0000","last_active":"2024-06-29T15:38:40.156658+0000","last_peered":"2024-06-29T15:38:40.156658+0000","last_clean":"2024-06-29T15:38:40.156658+0000","last_became_active":"2024-06-29T15:38:39.157478+0000","last_became_peered":"2024-06-29T15:38:39.157478+0000","last_unstale":"2024-06-29T15:38:40.156658+0000","last_undegraded":"2024-06-29T15:38:40.156658+0000","last_fullsized":"2024-06-29T15:38:40.156658+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:55.635046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.155655+0000","last_change":"2024-06-29T15:38:39.154641+0000","last_active":"2024-06-29T15:38:39.155655+0000","last_peered":"2024-06-29T15:38:39.155655+0000","last_clean":"2024-06-29T15:38:39.155655+0000","last_became_active":"2024-06-29T15:38:39.154284+0000","last_became_peered":"2024-06-29T15:38:39.154284+0000","last_unstale":"2024-06-29T15:38:39.155655+0000","last_undegraded":"2024-06-29T15:38:39.155655+0000","last_fullsized":"2024-06-29T15:38:39.155655+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:27.600161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.163983+0000","last_change":"2024-06-29T15:38:39.162675+0000","last_active":"2024-06-29T15:38:39.163983+0000","last_peered":"2024-06-29T15:38:39.163983+0000","last_clean":"2024-06-29T15:38:39.163983+0000","last_became_active":"2024-06-29T15:38:39.162343+0000","last_became_peered":"2024-06-29T15:38:39.162343+0000","last_unstale":"2024-06-29T15:38:39.163983+0000","last_undegraded":"2024-06-29T15:38:39.163983+0000","last_fullsized":"2024-06-29T15:38:39.163983+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:07.530421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155785+0000","last_change":"2024-06-29T15:38:39.157488+0000","last_active":"2024-06-29T15:38:40.155785+0000","last_peered":"2024-06-29T15:38:40.155785+0000","last_clean":"2024-06-29T15:38:40.155785+0000","last_became_active":"2024-06-29T15:38:39.157135+0000","last_became_peered":"2024-06-29T15:38:39.157135+0000","last_unstale":"2024-06-29T15:38:40.155785+0000","last_undegraded":"2024-06-29T15:38:40.155785+0000","last_fullsized":"2024-06-29T15:38:40.155785+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:04.592718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154460+0000","last_change":"2024-06-29T15:38:39.151336+0000","last_active":"2024-06-29T15:38:40.154460+0000","last_peered":"2024-06-29T15:38:40.154460+0000","last_clean":"2024-06-29T15:38:40.154460+0000","last_became_active":"2024-06-29T15:38:39.151021+0000","last_became_peered":"2024-06-29T15:38:39.151021+0000","last_unstale":"2024-06-29T15:38:40.154460+0000","last_undegraded":"2024-06-29T15:38:40.154460+0000","last_fullsized":"2024-06-29T15:38:40.154460+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:46.972831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"175'4","reported_seq":16,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.174339+0000","last_change":"2024-06-29T15:38:38.141732+0000","last_active":"2024-06-29T15:38:39.174339+0000","last_peered":"2024-06-29T15:38:39.174339+0000","last_clean":"2024-06-29T15:38:39.174339+0000","last_became_active":"2024-06-29T15:38:38.141417+0000","last_became_peered":"2024-06-29T15:38:38.141417+0000","last_unstale":"2024-06-29T15:38:39.174339+0000","last_undegraded":"2024-06-29T15:38:39.174339+0000","last_fullsized":"2024-06-29T15:38:39.174339+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:39.934824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154646+0000","last_change":"2024-06-29T15:38:39.144744+0000","last_active":"2024-06-29T15:38:40.154646+0000","last_peered":"2024-06-29T15:38:40.154646+0000","last_clean":"2024-06-29T15:38:40.154646+0000","last_became_active":"2024-06-29T15:38:39.144421+0000","last_became_peered":"2024-06-29T15:38:39.144421+0000","last_unstale":"2024-06-29T15:38:40.154646+0000","last_undegraded":"2024-06-29T15:38:40.154646+0000","last_fullsized":"2024-06-29T15:38:40.154646+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:50.512535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"175'1","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169063+0000","last_change":"2024-06-29T15:38:38.139246+0000","last_active":"2024-06-29T15:38:40.169063+0000","last_peered":"2024-06-29T15:38:40.169063+0000","last_clean":"2024-06-29T15:38:40.169063+0000","last_became_active":"2024-06-29T15:38:38.138916+0000","last_became_peered":"2024-06-29T15:38:38.138916+0000","last_unstale":"2024-06-29T15:38:40.169063+0000","last_undegraded":"2024-06-29T15:38:40.169063+0000","last_fullsized":"2024-06-29T15:38:40.169063+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:18:53.281121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155640+0000","last_change":"2024-06-29T15:38:39.141695+0000","last_active":"2024-06-29T15:38:40.155640+0000","last_peered":"2024-06-29T15:38:40.155640+0000","last_clean":"2024-06-29T15:38:40.155640+0000","last_became_active":"2024-06-29T15:38:39.141366+0000","last_became_peered":"2024-06-29T15:38:39.141366+0000","last_unstale":"2024-06-29T15:38:40.155640+0000","last_undegraded":"2024-06-29T15:38:40.155640+0000","last_fullsized":"2024-06-29T15:38:40.155640+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:45.889033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"175'2","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169204+0000","last_change":"2024-06-29T15:38:38.144313+0000","last_active":"2024-06-29T15:38:40.169204+0000","last_peered":"2024-06-29T15:38:40.169204+0000","last_clean":"2024-06-29T15:38:40.169204+0000","last_became_active":"2024-06-29T15:38:38.143961+0000","last_became_peered":"2024-06-29T15:38:38.143961+0000","last_unstale":"2024-06-29T15:38:40.169204+0000","last_undegraded":"2024-06-29T15:38:40.169204+0000","last_fullsized":"2024-06-29T15:38:40.169204+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:43.324328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169311+0000","last_change":"2024-06-29T15:38:39.150505+0000","last_active":"2024-06-29T15:38:40.169311+0000","last_peered":"2024-06-29T15:38:40.169311+0000","last_clean":"2024-06-29T15:38:40.169311+0000","last_became_active":"2024-06-29T15:38:39.150183+0000","last_became_peered":"2024-06-29T15:38:39.150183+0000","last_unstale":"2024-06-29T15:38:40.169311+0000","last_undegraded":"2024-06-29T15:38:40.169311+0000","last_fullsized":"2024-06-29T15:38:40.169311+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:25.906801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"175'2","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154469+0000","last_change":"2024-06-29T15:38:38.132916+0000","last_active":"2024-06-29T15:38:40.154469+0000","last_peered":"2024-06-29T15:38:40.154469+0000","last_clean":"2024-06-29T15:38:40.154469+0000","last_became_active":"2024-06-29T15:38:38.132528+0000","last_became_peered":"2024-06-29T15:38:38.132528+0000","last_unstale":"2024-06-29T15:38:40.154469+0000","last_undegraded":"2024-06-29T15:38:40.154469+0000","last_fullsized":"2024-06-29T15:38:40.154469+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:02.141063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.152348+0000","last_change":"2024-06-29T15:38:39.150947+0000","last_active":"2024-06-29T15:38:39.152348+0000","last_peered":"2024-06-29T15:38:39.152348+0000","last_clean":"2024-06-29T15:38:39.152348+0000","last_became_active":"2024-06-29T15:38:39.150632+0000","last_became_peered":"2024-06-29T15:38:39.150632+0000","last_unstale":"2024-06-29T15:38:39.152348+0000","last_undegraded":"2024-06-29T15:38:39.152348+0000","last_fullsized":"2024-06-29T15:38:39.152348+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:43.078957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"175'1","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156575+0000","last_change":"2024-06-29T15:38:38.135133+0000","last_active":"2024-06-29T15:38:40.156575+0000","last_peered":"2024-06-29T15:38:40.156575+0000","last_clean":"2024-06-29T15:38:40.156575+0000","last_became_active":"2024-06-29T15:38:38.134784+0000","last_became_peered":"2024-06-29T15:38:38.134784+0000","last_unstale":"2024-06-29T15:38:40.156575+0000","last_undegraded":"2024-06-29T15:38:40.156575+0000","last_fullsized":"2024-06-29T15:38:40.156575+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:56:58.991428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155258+0000","last_change":"2024-06-29T15:38:39.146362+0000","last_active":"2024-06-29T15:38:40.155258+0000","last_peered":"2024-06-29T15:38:40.155258+0000","last_clean":"2024-06-29T15:38:40.155258+0000","last_became_active":"2024-06-29T15:38:39.146036+0000","last_became_peered":"2024-06-29T15:38:39.146036+0000","last_unstale":"2024-06-29T15:38:40.155258+0000","last_undegraded":"2024-06-29T15:38:40.155258+0000","last_fullsized":"2024-06-29T15:38:40.155258+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:37.349591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"175'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.171178+0000","last_change":"2024-06-29T15:38:38.138923+0000","last_active":"2024-06-29T15:38:39.171178+0000","last_peered":"2024-06-29T15:38:39.171178+0000","last_clean":"2024-06-29T15:38:39.171178+0000","last_became_active":"2024-06-29T15:38:38.138547+0000","last_became_peered":"2024-06-29T15:38:38.138547+0000","last_unstale":"2024-06-29T15:38:39.171178+0000","last_undegraded":"2024-06-29T15:38:39.171178+0000","last_fullsized":"2024-06-29T15:38:39.171178+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:15:37.990480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154388+0000","last_change":"2024-06-29T15:38:39.152156+0000","last_active":"2024-06-29T15:38:40.154388+0000","last_peered":"2024-06-29T15:38:40.154388+0000","last_clean":"2024-06-29T15:38:40.154388+0000","last_became_active":"2024-06-29T15:38:39.151822+0000","last_became_peered":"2024-06-29T15:38:39.151822+0000","last_unstale":"2024-06-29T15:38:40.154388+0000","last_undegraded":"2024-06-29T15:38:40.154388+0000","last_fullsized":"2024-06-29T15:38:40.154388+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:38.865518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166702+0000","last_change":"2024-06-29T15:38:38.138910+0000","last_active":"2024-06-29T15:38:40.166702+0000","last_peered":"2024-06-29T15:38:40.166702+0000","last_clean":"2024-06-29T15:38:40.166702+0000","last_became_active":"2024-06-29T15:38:38.138568+0000","last_became_peered":"2024-06-29T15:38:38.138568+0000","last_unstale":"2024-06-29T15:38:40.166702+0000","last_undegraded":"2024-06-29T15:38:40.166702+0000","last_fullsized":"2024-06-29T15:38:40.166702+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:20.492267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166880+0000","last_change":"2024-06-29T15:38:39.145534+0000","last_active":"2024-06-29T15:38:40.166880+0000","last_peered":"2024-06-29T15:38:40.166880+0000","last_clean":"2024-06-29T15:38:40.166880+0000","last_became_active":"2024-06-29T15:38:39.145176+0000","last_became_peered":"2024-06-29T15:38:39.145176+0000","last_unstale":"2024-06-29T15:38:40.166880+0000","last_undegraded":"2024-06-29T15:38:40.166880+0000","last_fullsized":"2024-06-29T15:38:40.166880+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.415989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.147198+0000","last_change":"2024-06-29T15:38:38.137170+0000","last_active":"2024-06-29T15:38:39.147198+0000","last_peered":"2024-06-29T15:38:39.147198+0000","last_clean":"2024-06-29T15:38:39.147198+0000","last_became_active":"2024-06-29T15:38:38.136844+0000","last_became_peered":"2024-06-29T15:38:38.136844+0000","last_unstale":"2024-06-29T15:38:39.147198+0000","last_undegraded":"2024-06-29T15:38:39.147198+0000","last_fullsized":"2024-06-29T15:38:39.147198+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:51.813146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156354+0000","last_change":"2024-06-29T15:38:39.155195+0000","last_active":"2024-06-29T15:38:40.156354+0000","last_peered":"2024-06-29T15:38:40.156354+0000","last_clean":"2024-06-29T15:38:40.156354+0000","last_became_active":"2024-06-29T15:38:39.154820+0000","last_became_peered":"2024-06-29T15:38:39.154820+0000","last_unstale":"2024-06-29T15:38:40.156354+0000","last_undegraded":"2024-06-29T15:38:40.156354+0000","last_fullsized":"2024-06-29T15:38:40.156354+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:13:52.543192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169295+0000","last_change":"2024-06-29T15:38:38.143363+0000","last_active":"2024-06-29T15:38:40.169295+0000","last_peered":"2024-06-29T15:38:40.169295+0000","last_clean":"2024-06-29T15:38:40.169295+0000","last_became_active":"2024-06-29T15:38:38.142990+0000","last_became_peered":"2024-06-29T15:38:38.142990+0000","last_unstale":"2024-06-29T15:38:40.169295+0000","last_undegraded":"2024-06-29T15:38:40.169295+0000","last_fullsized":"2024-06-29T15:38:40.169295+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:10.863020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164405+0000","last_change":"2024-06-29T15:38:39.158686+0000","last_active":"2024-06-29T15:38:39.164405+0000","last_peered":"2024-06-29T15:38:39.164405+0000","last_clean":"2024-06-29T15:38:39.164405+0000","last_became_active":"2024-06-29T15:38:39.158299+0000","last_became_peered":"2024-06-29T15:38:39.158299+0000","last_unstale":"2024-06-29T15:38:39.164405+0000","last_undegraded":"2024-06-29T15:38:39.164405+0000","last_fullsized":"2024-06-29T15:38:39.164405+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:37.147618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167091+0000","last_change":"2024-06-29T15:38:38.141828+0000","last_active":"2024-06-29T15:38:40.167091+0000","last_peered":"2024-06-29T15:38:40.167091+0000","last_clean":"2024-06-29T15:38:40.167091+0000","last_became_active":"2024-06-29T15:38:38.140799+0000","last_became_peered":"2024-06-29T15:38:38.140799+0000","last_unstale":"2024-06-29T15:38:40.167091+0000","last_undegraded":"2024-06-29T15:38:40.167091+0000","last_fullsized":"2024-06-29T15:38:40.167091+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:26.107298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164348+0000","last_change":"2024-06-29T15:38:39.160500+0000","last_active":"2024-06-29T15:38:39.164348+0000","last_peered":"2024-06-29T15:38:39.164348+0000","last_clean":"2024-06-29T15:38:39.164348+0000","last_became_active":"2024-06-29T15:38:39.160185+0000","last_became_peered":"2024-06-29T15:38:39.160185+0000","last_unstale":"2024-06-29T15:38:39.164348+0000","last_undegraded":"2024-06-29T15:38:39.164348+0000","last_fullsized":"2024-06-29T15:38:39.164348+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:23:16.829589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"175'3","reported_seq":15,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.173805+0000","last_change":"2024-06-29T15:38:38.138511+0000","last_active":"2024-06-29T15:38:39.173805+0000","last_peered":"2024-06-29T15:38:39.173805+0000","last_clean":"2024-06-29T15:38:39.173805+0000","last_became_active":"2024-06-29T15:38:38.137954+0000","last_became_peered":"2024-06-29T15:38:38.137954+0000","last_unstale":"2024-06-29T15:38:39.173805+0000","last_undegraded":"2024-06-29T15:38:39.173805+0000","last_fullsized":"2024-06-29T15:38:39.173805+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:06:30.869270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155988+0000","last_change":"2024-06-29T15:38:39.156569+0000","last_active":"2024-06-29T15:38:40.155988+0000","last_peered":"2024-06-29T15:38:40.155988+0000","last_clean":"2024-06-29T15:38:40.155988+0000","last_became_active":"2024-06-29T15:38:39.155825+0000","last_became_peered":"2024-06-29T15:38:39.155825+0000","last_unstale":"2024-06-29T15:38:40.155988+0000","last_undegraded":"2024-06-29T15:38:40.155988+0000","last_fullsized":"2024-06-29T15:38:40.155988+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:17.413535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"175'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.174042+0000","last_change":"2024-06-29T15:38:38.145348+0000","last_active":"2024-06-29T15:38:39.174042+0000","last_peered":"2024-06-29T15:38:39.174042+0000","last_clean":"2024-06-29T15:38:39.174042+0000","last_became_active":"2024-06-29T15:38:38.145000+0000","last_became_peered":"2024-06-29T15:38:38.145000+0000","last_unstale":"2024-06-29T15:38:39.174042+0000","last_undegraded":"2024-06-29T15:38:39.174042+0000","last_fullsized":"2024-06-29T15:38:39.174042+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:26.107298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154824+0000","last_change":"2024-06-29T15:38:39.142722+0000","last_active":"2024-06-29T15:38:40.154824+0000","last_peered":"2024-06-29T15:38:40.154824+0000","last_clean":"2024-06-29T15:38:40.154824+0000","last_became_active":"2024-06-29T15:38:39.142389+0000","last_became_peered":"2024-06-29T15:38:39.142389+0000","last_unstale":"2024-06-29T15:38:40.154824+0000","last_undegraded":"2024-06-29T15:38:40.154824+0000","last_fullsized":"2024-06-29T15:38:40.154824+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:02.433053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155474+0000","last_change":"2024-06-29T15:38:38.130694+0000","last_active":"2024-06-29T15:38:40.155474+0000","last_peered":"2024-06-29T15:38:40.155474+0000","last_clean":"2024-06-29T15:38:40.155474+0000","last_became_active":"2024-06-29T15:38:38.130322+0000","last_became_peered":"2024-06-29T15:38:38.130322+0000","last_unstale":"2024-06-29T15:38:40.155474+0000","last_undegraded":"2024-06-29T15:38:40.155474+0000","last_fullsized":"2024-06-29T15:38:40.155474+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:47:32.079450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.155806+0000","last_change":"2024-06-29T15:38:39.154958+0000","last_active":"2024-06-29T15:38:39.155806+0000","last_peered":"2024-06-29T15:38:39.155806+0000","last_clean":"2024-06-29T15:38:39.155806+0000","last_became_active":"2024-06-29T15:38:39.154586+0000","last_became_peered":"2024-06-29T15:38:39.154586+0000","last_unstale":"2024-06-29T15:38:39.155806+0000","last_undegraded":"2024-06-29T15:38:39.155806+0000","last_fullsized":"2024-06-29T15:38:39.155806+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:49.723604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155560+0000","last_change":"2024-06-29T15:38:38.132854+0000","last_active":"2024-06-29T15:38:40.155560+0000","last_peered":"2024-06-29T15:38:40.155560+0000","last_clean":"2024-06-29T15:38:40.155560+0000","last_became_active":"2024-06-29T15:38:38.132557+0000","last_became_peered":"2024-06-29T15:38:38.132557+0000","last_unstale":"2024-06-29T15:38:40.155560+0000","last_undegraded":"2024-06-29T15:38:40.155560+0000","last_fullsized":"2024-06-29T15:38:40.155560+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:50.690292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154654+0000","last_change":"2024-06-29T15:38:39.156679+0000","last_active":"2024-06-29T15:38:40.154654+0000","last_peered":"2024-06-29T15:38:40.154654+0000","last_clean":"2024-06-29T15:38:40.154654+0000","last_became_active":"2024-06-29T15:38:39.156324+0000","last_became_peered":"2024-06-29T15:38:39.156324+0000","last_unstale":"2024-06-29T15:38:40.154654+0000","last_undegraded":"2024-06-29T15:38:40.154654+0000","last_fullsized":"2024-06-29T15:38:40.154654+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:10:31.025124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"175'8","reported_seq":22,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166561+0000","last_change":"2024-06-29T15:38:38.140553+0000","last_active":"2024-06-29T15:38:40.166561+0000","last_peered":"2024-06-29T15:38:40.166561+0000","last_clean":"2024-06-29T15:38:40.166561+0000","last_became_active":"2024-06-29T15:38:38.140232+0000","last_became_peered":"2024-06-29T15:38:38.140232+0000","last_unstale":"2024-06-29T15:38:40.166561+0000","last_undegraded":"2024-06-29T15:38:40.166561+0000","last_fullsized":"2024-06-29T15:38:40.166561+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:15:37.990480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169061+0000","last_change":"2024-06-29T15:38:39.157939+0000","last_active":"2024-06-29T15:38:40.169061+0000","last_peered":"2024-06-29T15:38:40.169061+0000","last_clean":"2024-06-29T15:38:40.169061+0000","last_became_active":"2024-06-29T15:38:39.157597+0000","last_became_peered":"2024-06-29T15:38:39.157597+0000","last_unstale":"2024-06-29T15:38:40.169061+0000","last_undegraded":"2024-06-29T15:38:40.169061+0000","last_fullsized":"2024-06-29T15:38:40.169061+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:22:03.658729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"175'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.170095+0000","last_change":"2024-06-29T15:38:38.140498+0000","last_active":"2024-06-29T15:38:39.170095+0000","last_peered":"2024-06-29T15:38:39.170095+0000","last_clean":"2024-06-29T15:38:39.170095+0000","last_became_active":"2024-06-29T15:38:38.140147+0000","last_became_peered":"2024-06-29T15:38:38.140147+0000","last_unstale":"2024-06-29T15:38:39.170095+0000","last_undegraded":"2024-06-29T15:38:39.170095+0000","last_fullsized":"2024-06-29T15:38:39.170095+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:31.453886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169209+0000","last_change":"2024-06-29T15:38:39.150687+0000","last_active":"2024-06-29T15:38:40.169209+0000","last_peered":"2024-06-29T15:38:40.169209+0000","last_clean":"2024-06-29T15:38:40.169209+0000","last_became_active":"2024-06-29T15:38:39.149667+0000","last_became_peered":"2024-06-29T15:38:39.149667+0000","last_unstale":"2024-06-29T15:38:40.169209+0000","last_undegraded":"2024-06-29T15:38:40.169209+0000","last_fullsized":"2024-06-29T15:38:40.169209+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:25.506511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"175'2","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169214+0000","last_change":"2024-06-29T15:38:38.144684+0000","last_active":"2024-06-29T15:38:40.169214+0000","last_peered":"2024-06-29T15:38:40.169214+0000","last_clean":"2024-06-29T15:38:40.169214+0000","last_became_active":"2024-06-29T15:38:38.144291+0000","last_became_peered":"2024-06-29T15:38:38.144291+0000","last_unstale":"2024-06-29T15:38:40.169214+0000","last_undegraded":"2024-06-29T15:38:40.169214+0000","last_fullsized":"2024-06-29T15:38:40.169214+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:56:49.937960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167018+0000","last_change":"2024-06-29T15:38:39.155855+0000","last_active":"2024-06-29T15:38:40.167018+0000","last_peered":"2024-06-29T15:38:40.167018+0000","last_clean":"2024-06-29T15:38:40.167018+0000","last_became_active":"2024-06-29T15:38:39.155502+0000","last_became_peered":"2024-06-29T15:38:39.155502+0000","last_unstale":"2024-06-29T15:38:40.167018+0000","last_undegraded":"2024-06-29T15:38:40.167018+0000","last_fullsized":"2024-06-29T15:38:40.167018+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:22.941213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"175'1","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154684+0000","last_change":"2024-06-29T15:38:38.138559+0000","last_active":"2024-06-29T15:38:40.154684+0000","last_peered":"2024-06-29T15:38:40.154684+0000","last_clean":"2024-06-29T15:38:40.154684+0000","last_became_active":"2024-06-29T15:38:38.138061+0000","last_became_peered":"2024-06-29T15:38:38.138061+0000","last_unstale":"2024-06-29T15:38:40.154684+0000","last_undegraded":"2024-06-29T15:38:40.154684+0000","last_fullsized":"2024-06-29T15:38:40.154684+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:38.686599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167192+0000","last_change":"2024-06-29T15:38:39.151788+0000","last_active":"2024-06-29T15:38:40.167192+0000","last_peered":"2024-06-29T15:38:40.167192+0000","last_clean":"2024-06-29T15:38:40.167192+0000","last_became_active":"2024-06-29T15:38:39.151448+0000","last_became_peered":"2024-06-29T15:38:39.151448+0000","last_unstale":"2024-06-29T15:38:40.167192+0000","last_undegraded":"2024-06-29T15:38:40.167192+0000","last_fullsized":"2024-06-29T15:38:40.167192+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:09:52.321508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155934+0000","last_change":"2024-06-29T15:38:38.131522+0000","last_active":"2024-06-29T15:38:40.155934+0000","last_peered":"2024-06-29T15:38:40.155934+0000","last_clean":"2024-06-29T15:38:40.155934+0000","last_became_active":"2024-06-29T15:38:38.131234+0000","last_became_peered":"2024-06-29T15:38:38.131234+0000","last_unstale":"2024-06-29T15:38:40.155934+0000","last_undegraded":"2024-06-29T15:38:40.155934+0000","last_fullsized":"2024-06-29T15:38:40.155934+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:10.222146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168918+0000","last_change":"2024-06-29T15:38:39.160210+0000","last_active":"2024-06-29T15:38:40.168918+0000","last_peered":"2024-06-29T15:38:40.168918+0000","last_clean":"2024-06-29T15:38:40.168918+0000","last_became_active":"2024-06-29T15:38:39.159426+0000","last_became_peered":"2024-06-29T15:38:39.159426+0000","last_unstale":"2024-06-29T15:38:40.168918+0000","last_undegraded":"2024-06-29T15:38:40.168918+0000","last_fullsized":"2024-06-29T15:38:40.168918+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:14:37.705141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"175'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.174625+0000","last_change":"2024-06-29T15:38:38.146272+0000","last_active":"2024-06-29T15:38:39.174625+0000","last_peered":"2024-06-29T15:38:39.174625+0000","last_clean":"2024-06-29T15:38:39.174625+0000","last_became_active":"2024-06-29T15:38:38.145947+0000","last_became_peered":"2024-06-29T15:38:38.145947+0000","last_unstale":"2024-06-29T15:38:39.174625+0000","last_undegraded":"2024-06-29T15:38:39.174625+0000","last_fullsized":"2024-06-29T15:38:39.174625+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:24.046559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154576+0000","last_change":"2024-06-29T15:38:39.146746+0000","last_active":"2024-06-29T15:38:40.154576+0000","last_peered":"2024-06-29T15:38:40.154576+0000","last_clean":"2024-06-29T15:38:40.154576+0000","last_became_active":"2024-06-29T15:38:39.146438+0000","last_became_peered":"2024-06-29T15:38:39.146438+0000","last_unstale":"2024-06-29T15:38:40.154576+0000","last_undegraded":"2024-06-29T15:38:40.154576+0000","last_fullsized":"2024-06-29T15:38:40.154576+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:19:50.874594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155934+0000","last_change":"2024-06-29T15:38:38.138462+0000","last_active":"2024-06-29T15:38:40.155934+0000","last_peered":"2024-06-29T15:38:40.155934+0000","last_clean":"2024-06-29T15:38:40.155934+0000","last_became_active":"2024-06-29T15:38:38.138152+0000","last_became_peered":"2024-06-29T15:38:38.138152+0000","last_unstale":"2024-06-29T15:38:40.155934+0000","last_undegraded":"2024-06-29T15:38:40.155934+0000","last_fullsized":"2024-06-29T15:38:40.155934+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:17:29.816811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168924+0000","last_change":"2024-06-29T15:38:39.160918+0000","last_active":"2024-06-29T15:38:40.168924+0000","last_peered":"2024-06-29T15:38:40.168924+0000","last_clean":"2024-06-29T15:38:40.168924+0000","last_became_active":"2024-06-29T15:38:39.160596+0000","last_became_peered":"2024-06-29T15:38:39.160596+0000","last_unstale":"2024-06-29T15:38:40.168924+0000","last_undegraded":"2024-06-29T15:38:40.168924+0000","last_fullsized":"2024-06-29T15:38:40.168924+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:57.655611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.143568+0000","last_change":"2024-06-29T15:38:38.142854+0000","last_active":"2024-06-29T15:38:39.143568+0000","last_peered":"2024-06-29T15:38:39.143568+0000","last_clean":"2024-06-29T15:38:39.143568+0000","last_became_active":"2024-06-29T15:38:38.142470+0000","last_became_peered":"2024-06-29T15:38:38.142470+0000","last_unstale":"2024-06-29T15:38:39.143568+0000","last_undegraded":"2024-06-29T15:38:39.143568+0000","last_fullsized":"2024-06-29T15:38:39.143568+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:43:29.315174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154566+0000","last_change":"2024-06-29T15:38:39.147121+0000","last_active":"2024-06-29T15:38:40.154566+0000","last_peered":"2024-06-29T15:38:40.154566+0000","last_clean":"2024-06-29T15:38:40.154566+0000","last_became_active":"2024-06-29T15:38:39.146785+0000","last_became_peered":"2024-06-29T15:38:39.146785+0000","last_unstale":"2024-06-29T15:38:40.154566+0000","last_undegraded":"2024-06-29T15:38:40.154566+0000","last_fullsized":"2024-06-29T15:38:40.154566+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:15.147451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156290+0000","last_change":"2024-06-29T15:38:38.141415+0000","last_active":"2024-06-29T15:38:40.156290+0000","last_peered":"2024-06-29T15:38:40.156290+0000","last_clean":"2024-06-29T15:38:40.156290+0000","last_became_active":"2024-06-29T15:38:38.141065+0000","last_became_peered":"2024-06-29T15:38:38.141065+0000","last_unstale":"2024-06-29T15:38:40.156290+0000","last_undegraded":"2024-06-29T15:38:40.156290+0000","last_fullsized":"2024-06-29T15:38:40.156290+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:50.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.156023+0000","last_change":"2024-06-29T15:38:39.155001+0000","last_active":"2024-06-29T15:38:39.156023+0000","last_peered":"2024-06-29T15:38:39.156023+0000","last_clean":"2024-06-29T15:38:39.156023+0000","last_became_active":"2024-06-29T15:38:39.154693+0000","last_became_peered":"2024-06-29T15:38:39.154693+0000","last_unstale":"2024-06-29T15:38:39.156023+0000","last_undegraded":"2024-06-29T15:38:39.156023+0000","last_fullsized":"2024-06-29T15:38:39.156023+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:01.959176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"175'2","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156291+0000","last_change":"2024-06-29T15:38:38.130633+0000","last_active":"2024-06-29T15:38:40.156291+0000","last_peered":"2024-06-29T15:38:40.156291+0000","last_clean":"2024-06-29T15:38:40.156291+0000","last_became_active":"2024-06-29T15:38:38.130299+0000","last_became_peered":"2024-06-29T15:38:38.130299+0000","last_unstale":"2024-06-29T15:38:40.156291+0000","last_undegraded":"2024-06-29T15:38:40.156291+0000","last_fullsized":"2024-06-29T15:38:40.156291+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:56.239100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155832+0000","last_change":"2024-06-29T15:38:39.154685+0000","last_active":"2024-06-29T15:38:40.155832+0000","last_peered":"2024-06-29T15:38:40.155832+0000","last_clean":"2024-06-29T15:38:40.155832+0000","last_became_active":"2024-06-29T15:38:39.154233+0000","last_became_peered":"2024-06-29T15:38:39.154233+0000","last_unstale":"2024-06-29T15:38:40.155832+0000","last_undegraded":"2024-06-29T15:38:40.155832+0000","last_fullsized":"2024-06-29T15:38:40.155832+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:42.139941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"176'1","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168409+0000","last_change":"2024-06-29T15:38:38.142120+0000","last_active":"2024-06-29T15:38:40.168409+0000","last_peered":"2024-06-29T15:38:40.168409+0000","last_clean":"2024-06-29T15:38:40.168409+0000","last_became_active":"2024-06-29T15:38:38.141514+0000","last_became_peered":"2024-06-29T15:38:38.141514+0000","last_unstale":"2024-06-29T15:38:40.168409+0000","last_undegraded":"2024-06-29T15:38:40.168409+0000","last_fullsized":"2024-06-29T15:38:40.168409+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:48.717596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167135+0000","last_change":"2024-06-29T15:38:39.146017+0000","last_active":"2024-06-29T15:38:40.167135+0000","last_peered":"2024-06-29T15:38:40.167135+0000","last_clean":"2024-06-29T15:38:40.167135+0000","last_became_active":"2024-06-29T15:38:39.145436+0000","last_became_peered":"2024-06-29T15:38:39.145436+0000","last_unstale":"2024-06-29T15:38:40.167135+0000","last_undegraded":"2024-06-29T15:38:40.167135+0000","last_fullsized":"2024-06-29T15:38:40.167135+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:09.876961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154740+0000","last_change":"2024-06-29T15:38:38.133986+0000","last_active":"2024-06-29T15:38:40.154740+0000","last_peered":"2024-06-29T15:38:40.154740+0000","last_clean":"2024-06-29T15:38:40.154740+0000","last_became_active":"2024-06-29T15:38:38.133667+0000","last_became_peered":"2024-06-29T15:38:38.133667+0000","last_unstale":"2024-06-29T15:38:40.154740+0000","last_undegraded":"2024-06-29T15:38:40.154740+0000","last_fullsized":"2024-06-29T15:38:40.154740+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:46.816540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.162086+0000","last_change":"2024-06-29T15:38:39.161000+0000","last_active":"2024-06-29T15:38:39.162086+0000","last_peered":"2024-06-29T15:38:39.162086+0000","last_clean":"2024-06-29T15:38:39.162086+0000","last_became_active":"2024-06-29T15:38:39.160344+0000","last_became_peered":"2024-06-29T15:38:39.160344+0000","last_unstale":"2024-06-29T15:38:39.162086+0000","last_undegraded":"2024-06-29T15:38:39.162086+0000","last_fullsized":"2024-06-29T15:38:39.162086+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:21.918534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.146157+0000","last_change":"2024-06-29T15:38:38.144774+0000","last_active":"2024-06-29T15:38:39.146157+0000","last_peered":"2024-06-29T15:38:39.146157+0000","last_clean":"2024-06-29T15:38:39.146157+0000","last_became_active":"2024-06-29T15:38:38.144421+0000","last_became_peered":"2024-06-29T15:38:38.144421+0000","last_unstale":"2024-06-29T15:38:39.146157+0000","last_undegraded":"2024-06-29T15:38:39.146157+0000","last_fullsized":"2024-06-29T15:38:39.146157+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:20.492267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156155+0000","last_change":"2024-06-29T15:38:39.150069+0000","last_active":"2024-06-29T15:38:40.156155+0000","last_peered":"2024-06-29T15:38:40.156155+0000","last_clean":"2024-06-29T15:38:40.156155+0000","last_became_active":"2024-06-29T15:38:39.149757+0000","last_became_peered":"2024-06-29T15:38:39.149757+0000","last_unstale":"2024-06-29T15:38:40.156155+0000","last_undegraded":"2024-06-29T15:38:40.156155+0000","last_fullsized":"2024-06-29T15:38:40.156155+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:08:29.754780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166756+0000","last_change":"2024-06-29T15:38:38.140417+0000","last_active":"2024-06-29T15:38:40.166756+0000","last_peered":"2024-06-29T15:38:40.166756+0000","last_clean":"2024-06-29T15:38:40.166756+0000","last_became_active":"2024-06-29T15:38:38.139582+0000","last_became_peered":"2024-06-29T15:38:38.139582+0000","last_unstale":"2024-06-29T15:38:40.166756+0000","last_undegraded":"2024-06-29T15:38:40.166756+0000","last_fullsized":"2024-06-29T15:38:40.166756+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:39.934824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155439+0000","last_change":"2024-06-29T15:38:39.154748+0000","last_active":"2024-06-29T15:38:40.155439+0000","last_peered":"2024-06-29T15:38:40.155439+0000","last_clean":"2024-06-29T15:38:40.155439+0000","last_became_active":"2024-06-29T15:38:39.153793+0000","last_became_peered":"2024-06-29T15:38:39.153793+0000","last_unstale":"2024-06-29T15:38:40.155439+0000","last_undegraded":"2024-06-29T15:38:40.155439+0000","last_fullsized":"2024-06-29T15:38:40.155439+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:34.159767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.147794+0000","last_change":"2024-06-29T15:38:38.138047+0000","last_active":"2024-06-29T15:38:39.147794+0000","last_peered":"2024-06-29T15:38:39.147794+0000","last_clean":"2024-06-29T15:38:39.147794+0000","last_became_active":"2024-06-29T15:38:38.137678+0000","last_became_peered":"2024-06-29T15:38:38.137678+0000","last_unstale":"2024-06-29T15:38:39.147794+0000","last_undegraded":"2024-06-29T15:38:39.147794+0000","last_fullsized":"2024-06-29T15:38:39.147794+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:31:12.113302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154596+0000","last_change":"2024-06-29T15:38:39.148136+0000","last_active":"2024-06-29T15:38:40.154596+0000","last_peered":"2024-06-29T15:38:40.154596+0000","last_clean":"2024-06-29T15:38:40.154596+0000","last_became_active":"2024-06-29T15:38:39.147595+0000","last_became_peered":"2024-06-29T15:38:39.147595+0000","last_unstale":"2024-06-29T15:38:40.154596+0000","last_undegraded":"2024-06-29T15:38:40.154596+0000","last_fullsized":"2024-06-29T15:38:40.154596+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:39:49.753251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166845+0000","last_change":"2024-06-29T15:38:38.141300+0000","last_active":"2024-06-29T15:38:40.166845+0000","last_peered":"2024-06-29T15:38:40.166845+0000","last_clean":"2024-06-29T15:38:40.166845+0000","last_became_active":"2024-06-29T15:38:38.140988+0000","last_became_peered":"2024-06-29T15:38:38.140988+0000","last_unstale":"2024-06-29T15:38:40.166845+0000","last_undegraded":"2024-06-29T15:38:40.166845+0000","last_fullsized":"2024-06-29T15:38:40.166845+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:51.813146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156484+0000","last_change":"2024-06-29T15:38:39.158962+0000","last_active":"2024-06-29T15:38:40.156484+0000","last_peered":"2024-06-29T15:38:40.156484+0000","last_clean":"2024-06-29T15:38:40.156484+0000","last_became_active":"2024-06-29T15:38:39.158645+0000","last_became_peered":"2024-06-29T15:38:39.158645+0000","last_unstale":"2024-06-29T15:38:40.156484+0000","last_undegraded":"2024-06-29T15:38:40.156484+0000","last_fullsized":"2024-06-29T15:38:40.156484+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:53:46.096993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168813+0000","last_change":"2024-06-29T15:38:38.143179+0000","last_active":"2024-06-29T15:38:40.168813+0000","last_peered":"2024-06-29T15:38:40.168813+0000","last_clean":"2024-06-29T15:38:40.168813+0000","last_became_active":"2024-06-29T15:38:38.142852+0000","last_became_peered":"2024-06-29T15:38:38.142852+0000","last_unstale":"2024-06-29T15:38:40.168813+0000","last_undegraded":"2024-06-29T15:38:40.168813+0000","last_fullsized":"2024-06-29T15:38:40.168813+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:04.468652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156425+0000","last_change":"2024-06-29T15:38:39.147936+0000","last_active":"2024-06-29T15:38:40.156425+0000","last_peered":"2024-06-29T15:38:40.156425+0000","last_clean":"2024-06-29T15:38:40.156425+0000","last_became_active":"2024-06-29T15:38:39.147453+0000","last_became_peered":"2024-06-29T15:38:39.147453+0000","last_unstale":"2024-06-29T15:38:40.156425+0000","last_undegraded":"2024-06-29T15:38:40.156425+0000","last_fullsized":"2024-06-29T15:38:40.156425+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:05.170580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"175'6","reported_seq":18,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.176602+0000","last_change":"2024-06-29T15:38:38.145282+0000","last_active":"2024-06-29T15:38:39.176602+0000","last_peered":"2024-06-29T15:38:39.176602+0000","last_clean":"2024-06-29T15:38:39.176602+0000","last_became_active":"2024-06-29T15:38:38.144976+0000","last_became_peered":"2024-06-29T15:38:38.144976+0000","last_unstale":"2024-06-29T15:38:39.176602+0000","last_undegraded":"2024-06-29T15:38:39.176602+0000","last_fullsized":"2024-06-29T15:38:39.176602+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:09.078878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"165'184","reported_seq":834,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168901+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:38:40.168901+0000","last_peered":"2024-06-29T15:38:40.168901+0000","last_clean":"2024-06-29T15:38:40.168901+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:38:40.168901+0000","last_undegraded":"2024-06-29T15:38:40.168901+0000","last_fullsized":"2024-06-29T15:38:40.168901+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":293,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":293,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2789376,"data_stored":1709564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543066,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41636,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330204,"statfs":{"total":96636764160,"available":96594128896,"internally_reserved":0,"allocated":2183168,"data_stored":1112524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47208,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324632,"statfs":{"total":96636764160,"available":96588423168,"internally_reserved":0,"allocated":2191360,"data_stored":1119174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543064,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29176,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342664,"statfs":{"total":96636764160,"available":96606887936,"internally_reserved":0,"allocated":2203648,"data_stored":1120786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510362,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59688,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312152,"statfs":{"total":96636764160,"available":96575643648,"internally_reserved":0,"allocated":2187264,"data_stored":1113070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575770,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63844,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307996,"statfs":{"total":96636764160,"available":96571387904,"internally_reserved":0,"allocated":2187264,"data_stored":1112566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510361,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41632,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330208,"statfs":{"total":96636764160,"available":96594132992,"internally_reserved":0,"allocated":2183168,"data_stored":1112524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510363,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38072,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333768,"statfs":{"total":96636764160,"available":96597778432,"internally_reserved":0,"allocated":2797568,"data_stored":1705140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:38:43.237 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:38:43.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.390+0000 7fcf118e4640 1 -- 172.21.15.59:0/518511154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c0b2210 msgr2=0x7fcf0c13ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.390+0000 7fcf118e4640 1 --2- 172.21.15.59:0/518511154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c0b2210 0x7fcf0c13ebd0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fcef4005fd0 tx=0x7fcef402a670 comp rx=0 tx=0).stop 2024-06-29T15:38:43.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.391+0000 7fcf118e4640 1 -- 172.21.15.59:0/518511154 shutdown_connections 2024-06-29T15:38:43.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.391+0000 7fcf118e4640 1 --2- 172.21.15.59:0/518511154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c0b2210 0x7fcf0c13ebd0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.391+0000 7fcf118e4640 1 -- 172.21.15.59:0/518511154 >> 172.21.15.59:0/518511154 conn(0x7fcf0c0b06e0 msgr2=0x7fcf0c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.391+0000 7fcf118e4640 1 -- 172.21.15.59:0/518511154 shutdown_connections 2024-06-29T15:38:43.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.391+0000 7fcf118e4640 1 -- 172.21.15.59:0/518511154 wait complete. 2024-06-29T15:38:43.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.392+0000 7fcf118e4640 1 Processor -- start 2024-06-29T15:38:43.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.392+0000 7fcf118e4640 1 -- start start 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf118e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c141360 0x7fcf0c151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf118e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf0c1604d0 0x7fcf0c0cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf118e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf0c15e520 0x7fcf0c15e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf118e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c167d70 con 0x7fcf0c141360 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf118e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c167bf0 con 0x7fcf0c1604d0 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf118e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c167ef0 con 0x7fcf0c15e520 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c141360 0x7fcf0c151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c141360 0x7fcf0c151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56748/0 (socket says 172.21.15.59:56748) 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 -- 172.21.15.59:0/2198669513 learned_addr learned my addr 172.21.15.59:0/2198669513 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf03fff640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf0c1604d0 0x7fcf0c0cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0b7fe640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf0c15e520 0x7fcf0c15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 -- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf0c15e520 msgr2=0x7fcf0c15e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf0c15e520 0x7fcf0c15e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 -- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf0c1604d0 msgr2=0x7fcf0c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf0c1604d0 0x7fcf0c0cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 -- 172.21.15.59:0/2198669513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c15ee70 con 0x7fcf0c141360 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf03fff640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf0c1604d0 0x7fcf0c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf0affd640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c141360 0x7fcf0c151c10 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00c430 tx=0x7fcefc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.394+0000 7fcf08ff9640 1 -- 172.21.15.59:0/2198669513 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc017070 con 0x7fcf0c141360 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.395+0000 7fcf08ff9640 1 -- 172.21.15.59:0/2198669513 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcefc0053c0 con 0x7fcf0c141360 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.395+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0c0ce760 con 0x7fcf0c141360 2024-06-29T15:38:43.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.395+0000 7fcf08ff9640 1 -- 172.21.15.59:0/2198669513 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc003850 con 0x7fcf0c141360 2024-06-29T15:38:43.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.395+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0c0ced20 con 0x7fcf0c141360 2024-06-29T15:38:43.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.396+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf0c0ccf10 con 0x7fcf0c141360 2024-06-29T15:38:43.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.396+0000 7fcf08ff9640 1 -- 172.21.15.59:0/2198669513 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fcefc01a070 con 0x7fcf0c141360 2024-06-29T15:38:43.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.397+0000 7fcf08ff9640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fceec076420 0x7fceec078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.397+0000 7fcf08ff9640 1 -- 172.21.15.59:0/2198669513 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7fcefc096f30 con 0x7fcf0c141360 2024-06-29T15:38:43.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.397+0000 7fcf03fff640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fceec076420 0x7fceec078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.398+0000 7fcf03fff640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fceec076420 0x7fceec078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcef4033000 tx=0x7fcef4004730 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.401+0000 7fcf08ff9640 1 -- 172.21.15.59:0/2198669513 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcf0c15ee70 con 0x7fcf0c141360 2024-06-29T15:38:43.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.594+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 --> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcf0c06cc10 con 0x7fceec076420 2024-06-29T15:38:43.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.598+0000 7fcf08ff9640 1 -- 172.21.15.59:0/2198669513 <== mgr.8039 v2:172.21.15.153:6832/2627795750 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252472 (secure 0 0 0) 0x7fcf0c06cc10 con 0x7fceec076420 2024-06-29T15:38:43.601 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:43.602 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:38:43.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fceec076420 msgr2=0x7fceec078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fceec076420 0x7fceec078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcef4033000 tx=0x7fcef4004730 comp rx=0 tx=0).stop 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c141360 msgr2=0x7fcf0c151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c141360 0x7fcf0c151c10 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00c430 tx=0x7fcefc00c8f0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 shutdown_connections 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf0c15e520 0x7fcf0c15e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf0c1604d0 0x7fcf0c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fceec076420 0x7fceec078930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 --2- 172.21.15.59:0/2198669513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf0c141360 0x7fcf0c151c10 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 >> 172.21.15.59:0/2198669513 conn(0x7fcf0c0b06e0 msgr2=0x7fcf0c15bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 shutdown_connections 2024-06-29T15:38:43.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.601+0000 7fcf118e4640 1 -- 172.21.15.59:0/2198669513 wait complete. 2024-06-29T15:38:43.613 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:38:42.828383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":338,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":369052,"kb_used_data":18284,"kb_used_omap":12,"kb_used_meta":350643,"kb_avail":754605668,"statfs":{"total":773094113280,"available":772716204032,"internally_reserved":0,"allocated":18722816,"data_stored":10105348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":359058994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.710772"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167034+0000","last_change":"2024-06-29T15:38:39.148245+0000","last_active":"2024-06-29T15:38:40.167034+0000","last_peered":"2024-06-29T15:38:40.167034+0000","last_clean":"2024-06-29T15:38:40.167034+0000","last_became_active":"2024-06-29T15:38:39.147398+0000","last_became_peered":"2024-06-29T15:38:39.147398+0000","last_unstale":"2024-06-29T15:38:40.167034+0000","last_undegraded":"2024-06-29T15:38:40.167034+0000","last_fullsized":"2024-06-29T15:38:40.167034+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:11:20.353442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154787+0000","last_change":"2024-06-29T15:38:39.143268+0000","last_active":"2024-06-29T15:38:40.154787+0000","last_peered":"2024-06-29T15:38:40.154787+0000","last_clean":"2024-06-29T15:38:40.154787+0000","last_became_active":"2024-06-29T15:38:39.142921+0000","last_became_peered":"2024-06-29T15:38:39.142921+0000","last_unstale":"2024-06-29T15:38:40.154787+0000","last_undegraded":"2024-06-29T15:38:40.154787+0000","last_fullsized":"2024-06-29T15:38:40.154787+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:12.898181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154435+0000","last_change":"2024-06-29T15:38:39.151169+0000","last_active":"2024-06-29T15:38:40.154435+0000","last_peered":"2024-06-29T15:38:40.154435+0000","last_clean":"2024-06-29T15:38:40.154435+0000","last_became_active":"2024-06-29T15:38:39.150852+0000","last_became_peered":"2024-06-29T15:38:39.150852+0000","last_unstale":"2024-06-29T15:38:40.154435+0000","last_undegraded":"2024-06-29T15:38:40.154435+0000","last_fullsized":"2024-06-29T15:38:40.154435+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:35:53.798820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155803+0000","last_change":"2024-06-29T15:38:39.151072+0000","last_active":"2024-06-29T15:38:40.155803+0000","last_peered":"2024-06-29T15:38:40.155803+0000","last_clean":"2024-06-29T15:38:40.155803+0000","last_became_active":"2024-06-29T15:38:39.150733+0000","last_became_peered":"2024-06-29T15:38:39.150733+0000","last_unstale":"2024-06-29T15:38:40.155803+0000","last_undegraded":"2024-06-29T15:38:40.155803+0000","last_fullsized":"2024-06-29T15:38:40.155803+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:51.775965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.160732+0000","last_change":"2024-06-29T15:38:39.158840+0000","last_active":"2024-06-29T15:38:39.160732+0000","last_peered":"2024-06-29T15:38:39.160732+0000","last_clean":"2024-06-29T15:38:39.160732+0000","last_became_active":"2024-06-29T15:38:39.158536+0000","last_became_peered":"2024-06-29T15:38:39.158536+0000","last_unstale":"2024-06-29T15:38:39.160732+0000","last_undegraded":"2024-06-29T15:38:39.160732+0000","last_fullsized":"2024-06-29T15:38:39.160732+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:41.407945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.160760+0000","last_change":"2024-06-29T15:38:39.159678+0000","last_active":"2024-06-29T15:38:39.160760+0000","last_peered":"2024-06-29T15:38:39.160760+0000","last_clean":"2024-06-29T15:38:39.160760+0000","last_became_active":"2024-06-29T15:38:39.159352+0000","last_became_peered":"2024-06-29T15:38:39.159352+0000","last_unstale":"2024-06-29T15:38:39.160760+0000","last_undegraded":"2024-06-29T15:38:39.160760+0000","last_fullsized":"2024-06-29T15:38:39.160760+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:34.550503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166788+0000","last_change":"2024-06-29T15:38:39.153277+0000","last_active":"2024-06-29T15:38:40.166788+0000","last_peered":"2024-06-29T15:38:40.166788+0000","last_clean":"2024-06-29T15:38:40.166788+0000","last_became_active":"2024-06-29T15:38:39.152953+0000","last_became_peered":"2024-06-29T15:38:39.152953+0000","last_unstale":"2024-06-29T15:38:40.166788+0000","last_undegraded":"2024-06-29T15:38:40.166788+0000","last_fullsized":"2024-06-29T15:38:40.166788+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:03.933083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155472+0000","last_change":"2024-06-29T15:38:39.156812+0000","last_active":"2024-06-29T15:38:40.155472+0000","last_peered":"2024-06-29T15:38:40.155472+0000","last_clean":"2024-06-29T15:38:40.155472+0000","last_became_active":"2024-06-29T15:38:39.156230+0000","last_became_peered":"2024-06-29T15:38:39.156230+0000","last_unstale":"2024-06-29T15:38:40.155472+0000","last_undegraded":"2024-06-29T15:38:40.155472+0000","last_fullsized":"2024-06-29T15:38:40.155472+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:02.681259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.161163+0000","last_change":"2024-06-29T15:38:39.158599+0000","last_active":"2024-06-29T15:38:39.161163+0000","last_peered":"2024-06-29T15:38:39.161163+0000","last_clean":"2024-06-29T15:38:39.161163+0000","last_became_active":"2024-06-29T15:38:39.158267+0000","last_became_peered":"2024-06-29T15:38:39.158267+0000","last_unstale":"2024-06-29T15:38:39.161163+0000","last_undegraded":"2024-06-29T15:38:39.161163+0000","last_fullsized":"2024-06-29T15:38:39.161163+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:30:11.113187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.161859+0000","last_change":"2024-06-29T15:38:39.160534+0000","last_active":"2024-06-29T15:38:39.161859+0000","last_peered":"2024-06-29T15:38:39.161859+0000","last_clean":"2024-06-29T15:38:39.161859+0000","last_became_active":"2024-06-29T15:38:39.160148+0000","last_became_peered":"2024-06-29T15:38:39.160148+0000","last_unstale":"2024-06-29T15:38:39.161859+0000","last_undegraded":"2024-06-29T15:38:39.161859+0000","last_fullsized":"2024-06-29T15:38:39.161859+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:26.668399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164424+0000","last_change":"2024-06-29T15:38:39.159356+0000","last_active":"2024-06-29T15:38:39.164424+0000","last_peered":"2024-06-29T15:38:39.164424+0000","last_clean":"2024-06-29T15:38:39.164424+0000","last_became_active":"2024-06-29T15:38:39.159044+0000","last_became_peered":"2024-06-29T15:38:39.159044+0000","last_unstale":"2024-06-29T15:38:39.164424+0000","last_undegraded":"2024-06-29T15:38:39.164424+0000","last_fullsized":"2024-06-29T15:38:39.164424+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:26:40.028297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155830+0000","last_change":"2024-06-29T15:38:39.154464+0000","last_active":"2024-06-29T15:38:40.155830+0000","last_peered":"2024-06-29T15:38:40.155830+0000","last_clean":"2024-06-29T15:38:40.155830+0000","last_became_active":"2024-06-29T15:38:39.153957+0000","last_became_peered":"2024-06-29T15:38:39.153957+0000","last_unstale":"2024-06-29T15:38:40.155830+0000","last_undegraded":"2024-06-29T15:38:40.155830+0000","last_fullsized":"2024-06-29T15:38:40.155830+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:10.999985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.162117+0000","last_change":"2024-06-29T15:38:39.160768+0000","last_active":"2024-06-29T15:38:39.162117+0000","last_peered":"2024-06-29T15:38:39.162117+0000","last_clean":"2024-06-29T15:38:39.162117+0000","last_became_active":"2024-06-29T15:38:39.160329+0000","last_became_peered":"2024-06-29T15:38:39.160329+0000","last_unstale":"2024-06-29T15:38:39.162117+0000","last_undegraded":"2024-06-29T15:38:39.162117+0000","last_fullsized":"2024-06-29T15:38:39.162117+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:18:51.865063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155362+0000","last_change":"2024-06-29T15:38:39.154417+0000","last_active":"2024-06-29T15:38:40.155362+0000","last_peered":"2024-06-29T15:38:40.155362+0000","last_clean":"2024-06-29T15:38:40.155362+0000","last_became_active":"2024-06-29T15:38:39.153946+0000","last_became_peered":"2024-06-29T15:38:39.153946+0000","last_unstale":"2024-06-29T15:38:40.155362+0000","last_undegraded":"2024-06-29T15:38:40.155362+0000","last_fullsized":"2024-06-29T15:38:40.155362+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:43.946244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154406+0000","last_change":"2024-06-29T15:38:39.143494+0000","last_active":"2024-06-29T15:38:40.154406+0000","last_peered":"2024-06-29T15:38:40.154406+0000","last_clean":"2024-06-29T15:38:40.154406+0000","last_became_active":"2024-06-29T15:38:39.142463+0000","last_became_peered":"2024-06-29T15:38:39.142463+0000","last_unstale":"2024-06-29T15:38:40.154406+0000","last_undegraded":"2024-06-29T15:38:40.154406+0000","last_fullsized":"2024-06-29T15:38:40.154406+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:39.264607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.165166+0000","last_change":"2024-06-29T15:38:39.163630+0000","last_active":"2024-06-29T15:38:39.165166+0000","last_peered":"2024-06-29T15:38:39.165166+0000","last_clean":"2024-06-29T15:38:39.165166+0000","last_became_active":"2024-06-29T15:38:39.163314+0000","last_became_peered":"2024-06-29T15:38:39.163314+0000","last_unstale":"2024-06-29T15:38:39.165166+0000","last_undegraded":"2024-06-29T15:38:39.165166+0000","last_fullsized":"2024-06-29T15:38:39.165166+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:03.933083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156173+0000","last_change":"2024-06-29T15:38:39.149776+0000","last_active":"2024-06-29T15:38:40.156173+0000","last_peered":"2024-06-29T15:38:40.156173+0000","last_clean":"2024-06-29T15:38:40.156173+0000","last_became_active":"2024-06-29T15:38:39.149457+0000","last_became_peered":"2024-06-29T15:38:39.149457+0000","last_unstale":"2024-06-29T15:38:40.156173+0000","last_undegraded":"2024-06-29T15:38:40.156173+0000","last_fullsized":"2024-06-29T15:38:40.156173+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:04:09.294171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164377+0000","last_change":"2024-06-29T15:38:39.162226+0000","last_active":"2024-06-29T15:38:39.164377+0000","last_peered":"2024-06-29T15:38:39.164377+0000","last_clean":"2024-06-29T15:38:39.164377+0000","last_became_active":"2024-06-29T15:38:39.160999+0000","last_became_peered":"2024-06-29T15:38:39.160999+0000","last_unstale":"2024-06-29T15:38:39.164377+0000","last_undegraded":"2024-06-29T15:38:39.164377+0000","last_fullsized":"2024-06-29T15:38:39.164377+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:10:21.982586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169086+0000","last_change":"2024-06-29T15:38:39.150984+0000","last_active":"2024-06-29T15:38:40.169086+0000","last_peered":"2024-06-29T15:38:40.169086+0000","last_clean":"2024-06-29T15:38:40.169086+0000","last_became_active":"2024-06-29T15:38:39.149691+0000","last_became_peered":"2024-06-29T15:38:39.149691+0000","last_unstale":"2024-06-29T15:38:40.169086+0000","last_undegraded":"2024-06-29T15:38:40.169086+0000","last_fullsized":"2024-06-29T15:38:40.169086+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:10.383784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.165920+0000","last_change":"2024-06-29T15:38:39.163067+0000","last_active":"2024-06-29T15:38:39.165920+0000","last_peered":"2024-06-29T15:38:39.165920+0000","last_clean":"2024-06-29T15:38:39.165920+0000","last_became_active":"2024-06-29T15:38:39.162707+0000","last_became_peered":"2024-06-29T15:38:39.162707+0000","last_unstale":"2024-06-29T15:38:39.165920+0000","last_undegraded":"2024-06-29T15:38:39.165920+0000","last_fullsized":"2024-06-29T15:38:39.165920+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:36.688998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164703+0000","last_change":"2024-06-29T15:38:39.163087+0000","last_active":"2024-06-29T15:38:39.164703+0000","last_peered":"2024-06-29T15:38:39.164703+0000","last_clean":"2024-06-29T15:38:39.164703+0000","last_became_active":"2024-06-29T15:38:39.162735+0000","last_became_peered":"2024-06-29T15:38:39.162735+0000","last_unstale":"2024-06-29T15:38:39.164703+0000","last_undegraded":"2024-06-29T15:38:39.164703+0000","last_fullsized":"2024-06-29T15:38:39.164703+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:18:31.626323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166814+0000","last_change":"2024-06-29T15:38:39.152932+0000","last_active":"2024-06-29T15:38:40.166814+0000","last_peered":"2024-06-29T15:38:40.166814+0000","last_clean":"2024-06-29T15:38:40.166814+0000","last_became_active":"2024-06-29T15:38:39.152598+0000","last_became_peered":"2024-06-29T15:38:39.152598+0000","last_unstale":"2024-06-29T15:38:40.166814+0000","last_undegraded":"2024-06-29T15:38:40.166814+0000","last_fullsized":"2024-06-29T15:38:40.166814+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:44.861477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155539+0000","last_change":"2024-06-29T15:38:39.150934+0000","last_active":"2024-06-29T15:38:40.155539+0000","last_peered":"2024-06-29T15:38:40.155539+0000","last_clean":"2024-06-29T15:38:40.155539+0000","last_became_active":"2024-06-29T15:38:39.150597+0000","last_became_peered":"2024-06-29T15:38:39.150597+0000","last_unstale":"2024-06-29T15:38:40.155539+0000","last_undegraded":"2024-06-29T15:38:40.155539+0000","last_fullsized":"2024-06-29T15:38:40.155539+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:38.700745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.157297+0000","last_change":"2024-06-29T15:38:39.154955+0000","last_active":"2024-06-29T15:38:39.157297+0000","last_peered":"2024-06-29T15:38:39.157297+0000","last_clean":"2024-06-29T15:38:39.157297+0000","last_became_active":"2024-06-29T15:38:39.154618+0000","last_became_peered":"2024-06-29T15:38:39.154618+0000","last_unstale":"2024-06-29T15:38:39.157297+0000","last_undegraded":"2024-06-29T15:38:39.157297+0000","last_fullsized":"2024-06-29T15:38:39.157297+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:13:26.331444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167047+0000","last_change":"2024-06-29T15:38:39.148536+0000","last_active":"2024-06-29T15:38:40.167047+0000","last_peered":"2024-06-29T15:38:40.167047+0000","last_clean":"2024-06-29T15:38:40.167047+0000","last_became_active":"2024-06-29T15:38:39.147923+0000","last_became_peered":"2024-06-29T15:38:39.147923+0000","last_unstale":"2024-06-29T15:38:40.167047+0000","last_undegraded":"2024-06-29T15:38:40.167047+0000","last_fullsized":"2024-06-29T15:38:40.167047+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:32.659808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166963+0000","last_change":"2024-06-29T15:38:39.156524+0000","last_active":"2024-06-29T15:38:40.166963+0000","last_peered":"2024-06-29T15:38:40.166963+0000","last_clean":"2024-06-29T15:38:40.166963+0000","last_became_active":"2024-06-29T15:38:39.156242+0000","last_became_peered":"2024-06-29T15:38:39.156242+0000","last_unstale":"2024-06-29T15:38:40.166963+0000","last_undegraded":"2024-06-29T15:38:40.166963+0000","last_fullsized":"2024-06-29T15:38:40.166963+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:40.455445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.165128+0000","last_change":"2024-06-29T15:38:39.163694+0000","last_active":"2024-06-29T15:38:39.165128+0000","last_peered":"2024-06-29T15:38:39.165128+0000","last_clean":"2024-06-29T15:38:39.165128+0000","last_became_active":"2024-06-29T15:38:39.163358+0000","last_became_peered":"2024-06-29T15:38:39.163358+0000","last_unstale":"2024-06-29T15:38:39.165128+0000","last_undegraded":"2024-06-29T15:38:39.165128+0000","last_fullsized":"2024-06-29T15:38:39.165128+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:44.861477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.155278+0000","last_change":"2024-06-29T15:38:39.153537+0000","last_active":"2024-06-29T15:38:39.155278+0000","last_peered":"2024-06-29T15:38:39.155278+0000","last_clean":"2024-06-29T15:38:39.155278+0000","last_became_active":"2024-06-29T15:38:39.153149+0000","last_became_peered":"2024-06-29T15:38:39.153149+0000","last_unstale":"2024-06-29T15:38:39.155278+0000","last_undegraded":"2024-06-29T15:38:39.155278+0000","last_fullsized":"2024-06-29T15:38:39.155278+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:18.095124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156658+0000","last_change":"2024-06-29T15:38:39.157793+0000","last_active":"2024-06-29T15:38:40.156658+0000","last_peered":"2024-06-29T15:38:40.156658+0000","last_clean":"2024-06-29T15:38:40.156658+0000","last_became_active":"2024-06-29T15:38:39.157478+0000","last_became_peered":"2024-06-29T15:38:39.157478+0000","last_unstale":"2024-06-29T15:38:40.156658+0000","last_undegraded":"2024-06-29T15:38:40.156658+0000","last_fullsized":"2024-06-29T15:38:40.156658+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:55.635046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.155655+0000","last_change":"2024-06-29T15:38:39.154641+0000","last_active":"2024-06-29T15:38:39.155655+0000","last_peered":"2024-06-29T15:38:39.155655+0000","last_clean":"2024-06-29T15:38:39.155655+0000","last_became_active":"2024-06-29T15:38:39.154284+0000","last_became_peered":"2024-06-29T15:38:39.154284+0000","last_unstale":"2024-06-29T15:38:39.155655+0000","last_undegraded":"2024-06-29T15:38:39.155655+0000","last_fullsized":"2024-06-29T15:38:39.155655+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:27.600161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.163983+0000","last_change":"2024-06-29T15:38:39.162675+0000","last_active":"2024-06-29T15:38:39.163983+0000","last_peered":"2024-06-29T15:38:39.163983+0000","last_clean":"2024-06-29T15:38:39.163983+0000","last_became_active":"2024-06-29T15:38:39.162343+0000","last_became_peered":"2024-06-29T15:38:39.162343+0000","last_unstale":"2024-06-29T15:38:39.163983+0000","last_undegraded":"2024-06-29T15:38:39.163983+0000","last_fullsized":"2024-06-29T15:38:39.163983+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:07.530421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155785+0000","last_change":"2024-06-29T15:38:39.157488+0000","last_active":"2024-06-29T15:38:40.155785+0000","last_peered":"2024-06-29T15:38:40.155785+0000","last_clean":"2024-06-29T15:38:40.155785+0000","last_became_active":"2024-06-29T15:38:39.157135+0000","last_became_peered":"2024-06-29T15:38:39.157135+0000","last_unstale":"2024-06-29T15:38:40.155785+0000","last_undegraded":"2024-06-29T15:38:40.155785+0000","last_fullsized":"2024-06-29T15:38:40.155785+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:04.592718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154460+0000","last_change":"2024-06-29T15:38:39.151336+0000","last_active":"2024-06-29T15:38:40.154460+0000","last_peered":"2024-06-29T15:38:40.154460+0000","last_clean":"2024-06-29T15:38:40.154460+0000","last_became_active":"2024-06-29T15:38:39.151021+0000","last_became_peered":"2024-06-29T15:38:39.151021+0000","last_unstale":"2024-06-29T15:38:40.154460+0000","last_undegraded":"2024-06-29T15:38:40.154460+0000","last_fullsized":"2024-06-29T15:38:40.154460+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:46.972831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"175'4","reported_seq":16,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.174339+0000","last_change":"2024-06-29T15:38:38.141732+0000","last_active":"2024-06-29T15:38:39.174339+0000","last_peered":"2024-06-29T15:38:39.174339+0000","last_clean":"2024-06-29T15:38:39.174339+0000","last_became_active":"2024-06-29T15:38:38.141417+0000","last_became_peered":"2024-06-29T15:38:38.141417+0000","last_unstale":"2024-06-29T15:38:39.174339+0000","last_undegraded":"2024-06-29T15:38:39.174339+0000","last_fullsized":"2024-06-29T15:38:39.174339+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:39.934824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154646+0000","last_change":"2024-06-29T15:38:39.144744+0000","last_active":"2024-06-29T15:38:40.154646+0000","last_peered":"2024-06-29T15:38:40.154646+0000","last_clean":"2024-06-29T15:38:40.154646+0000","last_became_active":"2024-06-29T15:38:39.144421+0000","last_became_peered":"2024-06-29T15:38:39.144421+0000","last_unstale":"2024-06-29T15:38:40.154646+0000","last_undegraded":"2024-06-29T15:38:40.154646+0000","last_fullsized":"2024-06-29T15:38:40.154646+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:50.512535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"175'1","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169063+0000","last_change":"2024-06-29T15:38:38.139246+0000","last_active":"2024-06-29T15:38:40.169063+0000","last_peered":"2024-06-29T15:38:40.169063+0000","last_clean":"2024-06-29T15:38:40.169063+0000","last_became_active":"2024-06-29T15:38:38.138916+0000","last_became_peered":"2024-06-29T15:38:38.138916+0000","last_unstale":"2024-06-29T15:38:40.169063+0000","last_undegraded":"2024-06-29T15:38:40.169063+0000","last_fullsized":"2024-06-29T15:38:40.169063+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:18:53.281121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155640+0000","last_change":"2024-06-29T15:38:39.141695+0000","last_active":"2024-06-29T15:38:40.155640+0000","last_peered":"2024-06-29T15:38:40.155640+0000","last_clean":"2024-06-29T15:38:40.155640+0000","last_became_active":"2024-06-29T15:38:39.141366+0000","last_became_peered":"2024-06-29T15:38:39.141366+0000","last_unstale":"2024-06-29T15:38:40.155640+0000","last_undegraded":"2024-06-29T15:38:40.155640+0000","last_fullsized":"2024-06-29T15:38:40.155640+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:45.889033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"175'2","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169204+0000","last_change":"2024-06-29T15:38:38.144313+0000","last_active":"2024-06-29T15:38:40.169204+0000","last_peered":"2024-06-29T15:38:40.169204+0000","last_clean":"2024-06-29T15:38:40.169204+0000","last_became_active":"2024-06-29T15:38:38.143961+0000","last_became_peered":"2024-06-29T15:38:38.143961+0000","last_unstale":"2024-06-29T15:38:40.169204+0000","last_undegraded":"2024-06-29T15:38:40.169204+0000","last_fullsized":"2024-06-29T15:38:40.169204+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:43.324328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169311+0000","last_change":"2024-06-29T15:38:39.150505+0000","last_active":"2024-06-29T15:38:40.169311+0000","last_peered":"2024-06-29T15:38:40.169311+0000","last_clean":"2024-06-29T15:38:40.169311+0000","last_became_active":"2024-06-29T15:38:39.150183+0000","last_became_peered":"2024-06-29T15:38:39.150183+0000","last_unstale":"2024-06-29T15:38:40.169311+0000","last_undegraded":"2024-06-29T15:38:40.169311+0000","last_fullsized":"2024-06-29T15:38:40.169311+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:25.906801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"175'2","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154469+0000","last_change":"2024-06-29T15:38:38.132916+0000","last_active":"2024-06-29T15:38:40.154469+0000","last_peered":"2024-06-29T15:38:40.154469+0000","last_clean":"2024-06-29T15:38:40.154469+0000","last_became_active":"2024-06-29T15:38:38.132528+0000","last_became_peered":"2024-06-29T15:38:38.132528+0000","last_unstale":"2024-06-29T15:38:40.154469+0000","last_undegraded":"2024-06-29T15:38:40.154469+0000","last_fullsized":"2024-06-29T15:38:40.154469+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:02.141063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.152348+0000","last_change":"2024-06-29T15:38:39.150947+0000","last_active":"2024-06-29T15:38:39.152348+0000","last_peered":"2024-06-29T15:38:39.152348+0000","last_clean":"2024-06-29T15:38:39.152348+0000","last_became_active":"2024-06-29T15:38:39.150632+0000","last_became_peered":"2024-06-29T15:38:39.150632+0000","last_unstale":"2024-06-29T15:38:39.152348+0000","last_undegraded":"2024-06-29T15:38:39.152348+0000","last_fullsized":"2024-06-29T15:38:39.152348+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:43.078957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"175'1","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156575+0000","last_change":"2024-06-29T15:38:38.135133+0000","last_active":"2024-06-29T15:38:40.156575+0000","last_peered":"2024-06-29T15:38:40.156575+0000","last_clean":"2024-06-29T15:38:40.156575+0000","last_became_active":"2024-06-29T15:38:38.134784+0000","last_became_peered":"2024-06-29T15:38:38.134784+0000","last_unstale":"2024-06-29T15:38:40.156575+0000","last_undegraded":"2024-06-29T15:38:40.156575+0000","last_fullsized":"2024-06-29T15:38:40.156575+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:56:58.991428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155258+0000","last_change":"2024-06-29T15:38:39.146362+0000","last_active":"2024-06-29T15:38:40.155258+0000","last_peered":"2024-06-29T15:38:40.155258+0000","last_clean":"2024-06-29T15:38:40.155258+0000","last_became_active":"2024-06-29T15:38:39.146036+0000","last_became_peered":"2024-06-29T15:38:39.146036+0000","last_unstale":"2024-06-29T15:38:40.155258+0000","last_undegraded":"2024-06-29T15:38:40.155258+0000","last_fullsized":"2024-06-29T15:38:40.155258+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:37.349591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"175'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.171178+0000","last_change":"2024-06-29T15:38:38.138923+0000","last_active":"2024-06-29T15:38:39.171178+0000","last_peered":"2024-06-29T15:38:39.171178+0000","last_clean":"2024-06-29T15:38:39.171178+0000","last_became_active":"2024-06-29T15:38:38.138547+0000","last_became_peered":"2024-06-29T15:38:38.138547+0000","last_unstale":"2024-06-29T15:38:39.171178+0000","last_undegraded":"2024-06-29T15:38:39.171178+0000","last_fullsized":"2024-06-29T15:38:39.171178+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:15:37.990480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154388+0000","last_change":"2024-06-29T15:38:39.152156+0000","last_active":"2024-06-29T15:38:40.154388+0000","last_peered":"2024-06-29T15:38:40.154388+0000","last_clean":"2024-06-29T15:38:40.154388+0000","last_became_active":"2024-06-29T15:38:39.151822+0000","last_became_peered":"2024-06-29T15:38:39.151822+0000","last_unstale":"2024-06-29T15:38:40.154388+0000","last_undegraded":"2024-06-29T15:38:40.154388+0000","last_fullsized":"2024-06-29T15:38:40.154388+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:38.865518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166702+0000","last_change":"2024-06-29T15:38:38.138910+0000","last_active":"2024-06-29T15:38:40.166702+0000","last_peered":"2024-06-29T15:38:40.166702+0000","last_clean":"2024-06-29T15:38:40.166702+0000","last_became_active":"2024-06-29T15:38:38.138568+0000","last_became_peered":"2024-06-29T15:38:38.138568+0000","last_unstale":"2024-06-29T15:38:40.166702+0000","last_undegraded":"2024-06-29T15:38:40.166702+0000","last_fullsized":"2024-06-29T15:38:40.166702+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:20.492267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166880+0000","last_change":"2024-06-29T15:38:39.145534+0000","last_active":"2024-06-29T15:38:40.166880+0000","last_peered":"2024-06-29T15:38:40.166880+0000","last_clean":"2024-06-29T15:38:40.166880+0000","last_became_active":"2024-06-29T15:38:39.145176+0000","last_became_peered":"2024-06-29T15:38:39.145176+0000","last_unstale":"2024-06-29T15:38:40.166880+0000","last_undegraded":"2024-06-29T15:38:40.166880+0000","last_fullsized":"2024-06-29T15:38:40.166880+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.415989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.147198+0000","last_change":"2024-06-29T15:38:38.137170+0000","last_active":"2024-06-29T15:38:39.147198+0000","last_peered":"2024-06-29T15:38:39.147198+0000","last_clean":"2024-06-29T15:38:39.147198+0000","last_became_active":"2024-06-29T15:38:38.136844+0000","last_became_peered":"2024-06-29T15:38:38.136844+0000","last_unstale":"2024-06-29T15:38:39.147198+0000","last_undegraded":"2024-06-29T15:38:39.147198+0000","last_fullsized":"2024-06-29T15:38:39.147198+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:51.813146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156354+0000","last_change":"2024-06-29T15:38:39.155195+0000","last_active":"2024-06-29T15:38:40.156354+0000","last_peered":"2024-06-29T15:38:40.156354+0000","last_clean":"2024-06-29T15:38:40.156354+0000","last_became_active":"2024-06-29T15:38:39.154820+0000","last_became_peered":"2024-06-29T15:38:39.154820+0000","last_unstale":"2024-06-29T15:38:40.156354+0000","last_undegraded":"2024-06-29T15:38:40.156354+0000","last_fullsized":"2024-06-29T15:38:40.156354+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:13:52.543192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169295+0000","last_change":"2024-06-29T15:38:38.143363+0000","last_active":"2024-06-29T15:38:40.169295+0000","last_peered":"2024-06-29T15:38:40.169295+0000","last_clean":"2024-06-29T15:38:40.169295+0000","last_became_active":"2024-06-29T15:38:38.142990+0000","last_became_peered":"2024-06-29T15:38:38.142990+0000","last_unstale":"2024-06-29T15:38:40.169295+0000","last_undegraded":"2024-06-29T15:38:40.169295+0000","last_fullsized":"2024-06-29T15:38:40.169295+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:10.863020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164405+0000","last_change":"2024-06-29T15:38:39.158686+0000","last_active":"2024-06-29T15:38:39.164405+0000","last_peered":"2024-06-29T15:38:39.164405+0000","last_clean":"2024-06-29T15:38:39.164405+0000","last_became_active":"2024-06-29T15:38:39.158299+0000","last_became_peered":"2024-06-29T15:38:39.158299+0000","last_unstale":"2024-06-29T15:38:39.164405+0000","last_undegraded":"2024-06-29T15:38:39.164405+0000","last_fullsized":"2024-06-29T15:38:39.164405+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:37.147618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167091+0000","last_change":"2024-06-29T15:38:38.141828+0000","last_active":"2024-06-29T15:38:40.167091+0000","last_peered":"2024-06-29T15:38:40.167091+0000","last_clean":"2024-06-29T15:38:40.167091+0000","last_became_active":"2024-06-29T15:38:38.140799+0000","last_became_peered":"2024-06-29T15:38:38.140799+0000","last_unstale":"2024-06-29T15:38:40.167091+0000","last_undegraded":"2024-06-29T15:38:40.167091+0000","last_fullsized":"2024-06-29T15:38:40.167091+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:26.107298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.164348+0000","last_change":"2024-06-29T15:38:39.160500+0000","last_active":"2024-06-29T15:38:39.164348+0000","last_peered":"2024-06-29T15:38:39.164348+0000","last_clean":"2024-06-29T15:38:39.164348+0000","last_became_active":"2024-06-29T15:38:39.160185+0000","last_became_peered":"2024-06-29T15:38:39.160185+0000","last_unstale":"2024-06-29T15:38:39.164348+0000","last_undegraded":"2024-06-29T15:38:39.164348+0000","last_fullsized":"2024-06-29T15:38:39.164348+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:23:16.829589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"175'3","reported_seq":15,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.173805+0000","last_change":"2024-06-29T15:38:38.138511+0000","last_active":"2024-06-29T15:38:39.173805+0000","last_peered":"2024-06-29T15:38:39.173805+0000","last_clean":"2024-06-29T15:38:39.173805+0000","last_became_active":"2024-06-29T15:38:38.137954+0000","last_became_peered":"2024-06-29T15:38:38.137954+0000","last_unstale":"2024-06-29T15:38:39.173805+0000","last_undegraded":"2024-06-29T15:38:39.173805+0000","last_fullsized":"2024-06-29T15:38:39.173805+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:06:30.869270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155988+0000","last_change":"2024-06-29T15:38:39.156569+0000","last_active":"2024-06-29T15:38:40.155988+0000","last_peered":"2024-06-29T15:38:40.155988+0000","last_clean":"2024-06-29T15:38:40.155988+0000","last_became_active":"2024-06-29T15:38:39.155825+0000","last_became_peered":"2024-06-29T15:38:39.155825+0000","last_unstale":"2024-06-29T15:38:40.155988+0000","last_undegraded":"2024-06-29T15:38:40.155988+0000","last_fullsized":"2024-06-29T15:38:40.155988+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:17.413535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"175'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.174042+0000","last_change":"2024-06-29T15:38:38.145348+0000","last_active":"2024-06-29T15:38:39.174042+0000","last_peered":"2024-06-29T15:38:39.174042+0000","last_clean":"2024-06-29T15:38:39.174042+0000","last_became_active":"2024-06-29T15:38:38.145000+0000","last_became_peered":"2024-06-29T15:38:38.145000+0000","last_unstale":"2024-06-29T15:38:39.174042+0000","last_undegraded":"2024-06-29T15:38:39.174042+0000","last_fullsized":"2024-06-29T15:38:39.174042+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:26.107298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154824+0000","last_change":"2024-06-29T15:38:39.142722+0000","last_active":"2024-06-29T15:38:40.154824+0000","last_peered":"2024-06-29T15:38:40.154824+0000","last_clean":"2024-06-29T15:38:40.154824+0000","last_became_active":"2024-06-29T15:38:39.142389+0000","last_became_peered":"2024-06-29T15:38:39.142389+0000","last_unstale":"2024-06-29T15:38:40.154824+0000","last_undegraded":"2024-06-29T15:38:40.154824+0000","last_fullsized":"2024-06-29T15:38:40.154824+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:02.433053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155474+0000","last_change":"2024-06-29T15:38:38.130694+0000","last_active":"2024-06-29T15:38:40.155474+0000","last_peered":"2024-06-29T15:38:40.155474+0000","last_clean":"2024-06-29T15:38:40.155474+0000","last_became_active":"2024-06-29T15:38:38.130322+0000","last_became_peered":"2024-06-29T15:38:38.130322+0000","last_unstale":"2024-06-29T15:38:40.155474+0000","last_undegraded":"2024-06-29T15:38:40.155474+0000","last_fullsized":"2024-06-29T15:38:40.155474+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:47:32.079450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.155806+0000","last_change":"2024-06-29T15:38:39.154958+0000","last_active":"2024-06-29T15:38:39.155806+0000","last_peered":"2024-06-29T15:38:39.155806+0000","last_clean":"2024-06-29T15:38:39.155806+0000","last_became_active":"2024-06-29T15:38:39.154586+0000","last_became_peered":"2024-06-29T15:38:39.154586+0000","last_unstale":"2024-06-29T15:38:39.155806+0000","last_undegraded":"2024-06-29T15:38:39.155806+0000","last_fullsized":"2024-06-29T15:38:39.155806+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:49.723604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155560+0000","last_change":"2024-06-29T15:38:38.132854+0000","last_active":"2024-06-29T15:38:40.155560+0000","last_peered":"2024-06-29T15:38:40.155560+0000","last_clean":"2024-06-29T15:38:40.155560+0000","last_became_active":"2024-06-29T15:38:38.132557+0000","last_became_peered":"2024-06-29T15:38:38.132557+0000","last_unstale":"2024-06-29T15:38:40.155560+0000","last_undegraded":"2024-06-29T15:38:40.155560+0000","last_fullsized":"2024-06-29T15:38:40.155560+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:50.690292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154654+0000","last_change":"2024-06-29T15:38:39.156679+0000","last_active":"2024-06-29T15:38:40.154654+0000","last_peered":"2024-06-29T15:38:40.154654+0000","last_clean":"2024-06-29T15:38:40.154654+0000","last_became_active":"2024-06-29T15:38:39.156324+0000","last_became_peered":"2024-06-29T15:38:39.156324+0000","last_unstale":"2024-06-29T15:38:40.154654+0000","last_undegraded":"2024-06-29T15:38:40.154654+0000","last_fullsized":"2024-06-29T15:38:40.154654+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:10:31.025124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"175'8","reported_seq":22,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166561+0000","last_change":"2024-06-29T15:38:38.140553+0000","last_active":"2024-06-29T15:38:40.166561+0000","last_peered":"2024-06-29T15:38:40.166561+0000","last_clean":"2024-06-29T15:38:40.166561+0000","last_became_active":"2024-06-29T15:38:38.140232+0000","last_became_peered":"2024-06-29T15:38:38.140232+0000","last_unstale":"2024-06-29T15:38:40.166561+0000","last_undegraded":"2024-06-29T15:38:40.166561+0000","last_fullsized":"2024-06-29T15:38:40.166561+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:15:37.990480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169061+0000","last_change":"2024-06-29T15:38:39.157939+0000","last_active":"2024-06-29T15:38:40.169061+0000","last_peered":"2024-06-29T15:38:40.169061+0000","last_clean":"2024-06-29T15:38:40.169061+0000","last_became_active":"2024-06-29T15:38:39.157597+0000","last_became_peered":"2024-06-29T15:38:39.157597+0000","last_unstale":"2024-06-29T15:38:40.169061+0000","last_undegraded":"2024-06-29T15:38:40.169061+0000","last_fullsized":"2024-06-29T15:38:40.169061+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:22:03.658729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"175'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.170095+0000","last_change":"2024-06-29T15:38:38.140498+0000","last_active":"2024-06-29T15:38:39.170095+0000","last_peered":"2024-06-29T15:38:39.170095+0000","last_clean":"2024-06-29T15:38:39.170095+0000","last_became_active":"2024-06-29T15:38:38.140147+0000","last_became_peered":"2024-06-29T15:38:38.140147+0000","last_unstale":"2024-06-29T15:38:39.170095+0000","last_undegraded":"2024-06-29T15:38:39.170095+0000","last_fullsized":"2024-06-29T15:38:39.170095+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:31.453886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169209+0000","last_change":"2024-06-29T15:38:39.150687+0000","last_active":"2024-06-29T15:38:40.169209+0000","last_peered":"2024-06-29T15:38:40.169209+0000","last_clean":"2024-06-29T15:38:40.169209+0000","last_became_active":"2024-06-29T15:38:39.149667+0000","last_became_peered":"2024-06-29T15:38:39.149667+0000","last_unstale":"2024-06-29T15:38:40.169209+0000","last_undegraded":"2024-06-29T15:38:40.169209+0000","last_fullsized":"2024-06-29T15:38:40.169209+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:25.506511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"175'2","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.169214+0000","last_change":"2024-06-29T15:38:38.144684+0000","last_active":"2024-06-29T15:38:40.169214+0000","last_peered":"2024-06-29T15:38:40.169214+0000","last_clean":"2024-06-29T15:38:40.169214+0000","last_became_active":"2024-06-29T15:38:38.144291+0000","last_became_peered":"2024-06-29T15:38:38.144291+0000","last_unstale":"2024-06-29T15:38:40.169214+0000","last_undegraded":"2024-06-29T15:38:40.169214+0000","last_fullsized":"2024-06-29T15:38:40.169214+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:56:49.937960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167018+0000","last_change":"2024-06-29T15:38:39.155855+0000","last_active":"2024-06-29T15:38:40.167018+0000","last_peered":"2024-06-29T15:38:40.167018+0000","last_clean":"2024-06-29T15:38:40.167018+0000","last_became_active":"2024-06-29T15:38:39.155502+0000","last_became_peered":"2024-06-29T15:38:39.155502+0000","last_unstale":"2024-06-29T15:38:40.167018+0000","last_undegraded":"2024-06-29T15:38:40.167018+0000","last_fullsized":"2024-06-29T15:38:40.167018+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:22.941213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"175'1","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154684+0000","last_change":"2024-06-29T15:38:38.138559+0000","last_active":"2024-06-29T15:38:40.154684+0000","last_peered":"2024-06-29T15:38:40.154684+0000","last_clean":"2024-06-29T15:38:40.154684+0000","last_became_active":"2024-06-29T15:38:38.138061+0000","last_became_peered":"2024-06-29T15:38:38.138061+0000","last_unstale":"2024-06-29T15:38:40.154684+0000","last_undegraded":"2024-06-29T15:38:40.154684+0000","last_fullsized":"2024-06-29T15:38:40.154684+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:38.686599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167192+0000","last_change":"2024-06-29T15:38:39.151788+0000","last_active":"2024-06-29T15:38:40.167192+0000","last_peered":"2024-06-29T15:38:40.167192+0000","last_clean":"2024-06-29T15:38:40.167192+0000","last_became_active":"2024-06-29T15:38:39.151448+0000","last_became_peered":"2024-06-29T15:38:39.151448+0000","last_unstale":"2024-06-29T15:38:40.167192+0000","last_undegraded":"2024-06-29T15:38:40.167192+0000","last_fullsized":"2024-06-29T15:38:40.167192+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:09:52.321508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155934+0000","last_change":"2024-06-29T15:38:38.131522+0000","last_active":"2024-06-29T15:38:40.155934+0000","last_peered":"2024-06-29T15:38:40.155934+0000","last_clean":"2024-06-29T15:38:40.155934+0000","last_became_active":"2024-06-29T15:38:38.131234+0000","last_became_peered":"2024-06-29T15:38:38.131234+0000","last_unstale":"2024-06-29T15:38:40.155934+0000","last_undegraded":"2024-06-29T15:38:40.155934+0000","last_fullsized":"2024-06-29T15:38:40.155934+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:10.222146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168918+0000","last_change":"2024-06-29T15:38:39.160210+0000","last_active":"2024-06-29T15:38:40.168918+0000","last_peered":"2024-06-29T15:38:40.168918+0000","last_clean":"2024-06-29T15:38:40.168918+0000","last_became_active":"2024-06-29T15:38:39.159426+0000","last_became_peered":"2024-06-29T15:38:39.159426+0000","last_unstale":"2024-06-29T15:38:40.168918+0000","last_undegraded":"2024-06-29T15:38:40.168918+0000","last_fullsized":"2024-06-29T15:38:40.168918+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:14:37.705141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"175'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.174625+0000","last_change":"2024-06-29T15:38:38.146272+0000","last_active":"2024-06-29T15:38:39.174625+0000","last_peered":"2024-06-29T15:38:39.174625+0000","last_clean":"2024-06-29T15:38:39.174625+0000","last_became_active":"2024-06-29T15:38:38.145947+0000","last_became_peered":"2024-06-29T15:38:38.145947+0000","last_unstale":"2024-06-29T15:38:39.174625+0000","last_undegraded":"2024-06-29T15:38:39.174625+0000","last_fullsized":"2024-06-29T15:38:39.174625+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:24.046559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154576+0000","last_change":"2024-06-29T15:38:39.146746+0000","last_active":"2024-06-29T15:38:40.154576+0000","last_peered":"2024-06-29T15:38:40.154576+0000","last_clean":"2024-06-29T15:38:40.154576+0000","last_became_active":"2024-06-29T15:38:39.146438+0000","last_became_peered":"2024-06-29T15:38:39.146438+0000","last_unstale":"2024-06-29T15:38:40.154576+0000","last_undegraded":"2024-06-29T15:38:40.154576+0000","last_fullsized":"2024-06-29T15:38:40.154576+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:19:50.874594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155934+0000","last_change":"2024-06-29T15:38:38.138462+0000","last_active":"2024-06-29T15:38:40.155934+0000","last_peered":"2024-06-29T15:38:40.155934+0000","last_clean":"2024-06-29T15:38:40.155934+0000","last_became_active":"2024-06-29T15:38:38.138152+0000","last_became_peered":"2024-06-29T15:38:38.138152+0000","last_unstale":"2024-06-29T15:38:40.155934+0000","last_undegraded":"2024-06-29T15:38:40.155934+0000","last_fullsized":"2024-06-29T15:38:40.155934+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:17:29.816811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168924+0000","last_change":"2024-06-29T15:38:39.160918+0000","last_active":"2024-06-29T15:38:40.168924+0000","last_peered":"2024-06-29T15:38:40.168924+0000","last_clean":"2024-06-29T15:38:40.168924+0000","last_became_active":"2024-06-29T15:38:39.160596+0000","last_became_peered":"2024-06-29T15:38:39.160596+0000","last_unstale":"2024-06-29T15:38:40.168924+0000","last_undegraded":"2024-06-29T15:38:40.168924+0000","last_fullsized":"2024-06-29T15:38:40.168924+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:57.655611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.143568+0000","last_change":"2024-06-29T15:38:38.142854+0000","last_active":"2024-06-29T15:38:39.143568+0000","last_peered":"2024-06-29T15:38:39.143568+0000","last_clean":"2024-06-29T15:38:39.143568+0000","last_became_active":"2024-06-29T15:38:38.142470+0000","last_became_peered":"2024-06-29T15:38:38.142470+0000","last_unstale":"2024-06-29T15:38:39.143568+0000","last_undegraded":"2024-06-29T15:38:39.143568+0000","last_fullsized":"2024-06-29T15:38:39.143568+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:43:29.315174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154566+0000","last_change":"2024-06-29T15:38:39.147121+0000","last_active":"2024-06-29T15:38:40.154566+0000","last_peered":"2024-06-29T15:38:40.154566+0000","last_clean":"2024-06-29T15:38:40.154566+0000","last_became_active":"2024-06-29T15:38:39.146785+0000","last_became_peered":"2024-06-29T15:38:39.146785+0000","last_unstale":"2024-06-29T15:38:40.154566+0000","last_undegraded":"2024-06-29T15:38:40.154566+0000","last_fullsized":"2024-06-29T15:38:40.154566+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:15.147451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156290+0000","last_change":"2024-06-29T15:38:38.141415+0000","last_active":"2024-06-29T15:38:40.156290+0000","last_peered":"2024-06-29T15:38:40.156290+0000","last_clean":"2024-06-29T15:38:40.156290+0000","last_became_active":"2024-06-29T15:38:38.141065+0000","last_became_peered":"2024-06-29T15:38:38.141065+0000","last_unstale":"2024-06-29T15:38:40.156290+0000","last_undegraded":"2024-06-29T15:38:40.156290+0000","last_fullsized":"2024-06-29T15:38:40.156290+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:50.722131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.156023+0000","last_change":"2024-06-29T15:38:39.155001+0000","last_active":"2024-06-29T15:38:39.156023+0000","last_peered":"2024-06-29T15:38:39.156023+0000","last_clean":"2024-06-29T15:38:39.156023+0000","last_became_active":"2024-06-29T15:38:39.154693+0000","last_became_peered":"2024-06-29T15:38:39.154693+0000","last_unstale":"2024-06-29T15:38:39.156023+0000","last_undegraded":"2024-06-29T15:38:39.156023+0000","last_fullsized":"2024-06-29T15:38:39.156023+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:01.959176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"175'2","reported_seq":16,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156291+0000","last_change":"2024-06-29T15:38:38.130633+0000","last_active":"2024-06-29T15:38:40.156291+0000","last_peered":"2024-06-29T15:38:40.156291+0000","last_clean":"2024-06-29T15:38:40.156291+0000","last_became_active":"2024-06-29T15:38:38.130299+0000","last_became_peered":"2024-06-29T15:38:38.130299+0000","last_unstale":"2024-06-29T15:38:40.156291+0000","last_undegraded":"2024-06-29T15:38:40.156291+0000","last_fullsized":"2024-06-29T15:38:40.156291+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:56.239100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155832+0000","last_change":"2024-06-29T15:38:39.154685+0000","last_active":"2024-06-29T15:38:40.155832+0000","last_peered":"2024-06-29T15:38:40.155832+0000","last_clean":"2024-06-29T15:38:40.155832+0000","last_became_active":"2024-06-29T15:38:39.154233+0000","last_became_peered":"2024-06-29T15:38:39.154233+0000","last_unstale":"2024-06-29T15:38:40.155832+0000","last_undegraded":"2024-06-29T15:38:40.155832+0000","last_fullsized":"2024-06-29T15:38:40.155832+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:42.139941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"176'1","reported_seq":15,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168409+0000","last_change":"2024-06-29T15:38:38.142120+0000","last_active":"2024-06-29T15:38:40.168409+0000","last_peered":"2024-06-29T15:38:40.168409+0000","last_clean":"2024-06-29T15:38:40.168409+0000","last_became_active":"2024-06-29T15:38:38.141514+0000","last_became_peered":"2024-06-29T15:38:38.141514+0000","last_unstale":"2024-06-29T15:38:40.168409+0000","last_undegraded":"2024-06-29T15:38:40.168409+0000","last_fullsized":"2024-06-29T15:38:40.168409+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:48.717596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.167135+0000","last_change":"2024-06-29T15:38:39.146017+0000","last_active":"2024-06-29T15:38:40.167135+0000","last_peered":"2024-06-29T15:38:40.167135+0000","last_clean":"2024-06-29T15:38:40.167135+0000","last_became_active":"2024-06-29T15:38:39.145436+0000","last_became_peered":"2024-06-29T15:38:39.145436+0000","last_unstale":"2024-06-29T15:38:40.167135+0000","last_undegraded":"2024-06-29T15:38:40.167135+0000","last_fullsized":"2024-06-29T15:38:40.167135+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:09.876961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154740+0000","last_change":"2024-06-29T15:38:38.133986+0000","last_active":"2024-06-29T15:38:40.154740+0000","last_peered":"2024-06-29T15:38:40.154740+0000","last_clean":"2024-06-29T15:38:40.154740+0000","last_became_active":"2024-06-29T15:38:38.133667+0000","last_became_peered":"2024-06-29T15:38:38.133667+0000","last_unstale":"2024-06-29T15:38:40.154740+0000","last_undegraded":"2024-06-29T15:38:40.154740+0000","last_fullsized":"2024-06-29T15:38:40.154740+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:46.816540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.162086+0000","last_change":"2024-06-29T15:38:39.161000+0000","last_active":"2024-06-29T15:38:39.162086+0000","last_peered":"2024-06-29T15:38:39.162086+0000","last_clean":"2024-06-29T15:38:39.162086+0000","last_became_active":"2024-06-29T15:38:39.160344+0000","last_became_peered":"2024-06-29T15:38:39.160344+0000","last_unstale":"2024-06-29T15:38:39.162086+0000","last_undegraded":"2024-06-29T15:38:39.162086+0000","last_fullsized":"2024-06-29T15:38:39.162086+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:21.918534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.146157+0000","last_change":"2024-06-29T15:38:38.144774+0000","last_active":"2024-06-29T15:38:39.146157+0000","last_peered":"2024-06-29T15:38:39.146157+0000","last_clean":"2024-06-29T15:38:39.146157+0000","last_became_active":"2024-06-29T15:38:38.144421+0000","last_became_peered":"2024-06-29T15:38:38.144421+0000","last_unstale":"2024-06-29T15:38:39.146157+0000","last_undegraded":"2024-06-29T15:38:39.146157+0000","last_fullsized":"2024-06-29T15:38:39.146157+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:20.492267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156155+0000","last_change":"2024-06-29T15:38:39.150069+0000","last_active":"2024-06-29T15:38:40.156155+0000","last_peered":"2024-06-29T15:38:40.156155+0000","last_clean":"2024-06-29T15:38:40.156155+0000","last_became_active":"2024-06-29T15:38:39.149757+0000","last_became_peered":"2024-06-29T15:38:39.149757+0000","last_unstale":"2024-06-29T15:38:40.156155+0000","last_undegraded":"2024-06-29T15:38:40.156155+0000","last_fullsized":"2024-06-29T15:38:40.156155+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:08:29.754780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166756+0000","last_change":"2024-06-29T15:38:38.140417+0000","last_active":"2024-06-29T15:38:40.166756+0000","last_peered":"2024-06-29T15:38:40.166756+0000","last_clean":"2024-06-29T15:38:40.166756+0000","last_became_active":"2024-06-29T15:38:38.139582+0000","last_became_peered":"2024-06-29T15:38:38.139582+0000","last_unstale":"2024-06-29T15:38:40.166756+0000","last_undegraded":"2024-06-29T15:38:40.166756+0000","last_fullsized":"2024-06-29T15:38:40.166756+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:39.934824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.155439+0000","last_change":"2024-06-29T15:38:39.154748+0000","last_active":"2024-06-29T15:38:40.155439+0000","last_peered":"2024-06-29T15:38:40.155439+0000","last_clean":"2024-06-29T15:38:40.155439+0000","last_became_active":"2024-06-29T15:38:39.153793+0000","last_became_peered":"2024-06-29T15:38:39.153793+0000","last_unstale":"2024-06-29T15:38:40.155439+0000","last_undegraded":"2024-06-29T15:38:40.155439+0000","last_fullsized":"2024-06-29T15:38:40.155439+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:34.159767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.147794+0000","last_change":"2024-06-29T15:38:38.138047+0000","last_active":"2024-06-29T15:38:39.147794+0000","last_peered":"2024-06-29T15:38:39.147794+0000","last_clean":"2024-06-29T15:38:39.147794+0000","last_became_active":"2024-06-29T15:38:38.137678+0000","last_became_peered":"2024-06-29T15:38:38.137678+0000","last_unstale":"2024-06-29T15:38:39.147794+0000","last_undegraded":"2024-06-29T15:38:39.147794+0000","last_fullsized":"2024-06-29T15:38:39.147794+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:31:12.113302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.154596+0000","last_change":"2024-06-29T15:38:39.148136+0000","last_active":"2024-06-29T15:38:40.154596+0000","last_peered":"2024-06-29T15:38:40.154596+0000","last_clean":"2024-06-29T15:38:40.154596+0000","last_became_active":"2024-06-29T15:38:39.147595+0000","last_became_peered":"2024-06-29T15:38:39.147595+0000","last_unstale":"2024-06-29T15:38:40.154596+0000","last_undegraded":"2024-06-29T15:38:40.154596+0000","last_fullsized":"2024-06-29T15:38:40.154596+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:39:49.753251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.166845+0000","last_change":"2024-06-29T15:38:38.141300+0000","last_active":"2024-06-29T15:38:40.166845+0000","last_peered":"2024-06-29T15:38:40.166845+0000","last_clean":"2024-06-29T15:38:40.166845+0000","last_became_active":"2024-06-29T15:38:38.140988+0000","last_became_peered":"2024-06-29T15:38:38.140988+0000","last_unstale":"2024-06-29T15:38:40.166845+0000","last_undegraded":"2024-06-29T15:38:40.166845+0000","last_fullsized":"2024-06-29T15:38:40.166845+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:51.813146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156484+0000","last_change":"2024-06-29T15:38:39.158962+0000","last_active":"2024-06-29T15:38:40.156484+0000","last_peered":"2024-06-29T15:38:40.156484+0000","last_clean":"2024-06-29T15:38:40.156484+0000","last_became_active":"2024-06-29T15:38:39.158645+0000","last_became_peered":"2024-06-29T15:38:39.158645+0000","last_unstale":"2024-06-29T15:38:40.156484+0000","last_undegraded":"2024-06-29T15:38:40.156484+0000","last_fullsized":"2024-06-29T15:38:40.156484+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:53:46.096993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168813+0000","last_change":"2024-06-29T15:38:38.143179+0000","last_active":"2024-06-29T15:38:40.168813+0000","last_peered":"2024-06-29T15:38:40.168813+0000","last_clean":"2024-06-29T15:38:40.168813+0000","last_became_active":"2024-06-29T15:38:38.142852+0000","last_became_peered":"2024-06-29T15:38:38.142852+0000","last_unstale":"2024-06-29T15:38:40.168813+0000","last_undegraded":"2024-06-29T15:38:40.168813+0000","last_fullsized":"2024-06-29T15:38:40.168813+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:04.468652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.156425+0000","last_change":"2024-06-29T15:38:39.147936+0000","last_active":"2024-06-29T15:38:40.156425+0000","last_peered":"2024-06-29T15:38:40.156425+0000","last_clean":"2024-06-29T15:38:40.156425+0000","last_became_active":"2024-06-29T15:38:39.147453+0000","last_became_peered":"2024-06-29T15:38:39.147453+0000","last_unstale":"2024-06-29T15:38:40.156425+0000","last_undegraded":"2024-06-29T15:38:40.156425+0000","last_fullsized":"2024-06-29T15:38:40.156425+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":174,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.116819+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.116819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:05.170580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"175'6","reported_seq":18,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-29T15:38:39.176602+0000","last_change":"2024-06-29T15:38:38.145282+0000","last_active":"2024-06-29T15:38:39.176602+0000","last_peered":"2024-06-29T15:38:39.176602+0000","last_clean":"2024-06-29T15:38:39.176602+0000","last_became_active":"2024-06-29T15:38:38.144976+0000","last_became_peered":"2024-06-29T15:38:38.144976+0000","last_unstale":"2024-06-29T15:38:39.176602+0000","last_undegraded":"2024-06-29T15:38:39.176602+0000","last_fullsized":"2024-06-29T15:38:39.176602+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.110811+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.110811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:09.078878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"165'184","reported_seq":834,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.168901+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:38:40.168901+0000","last_peered":"2024-06-29T15:38:40.168901+0000","last_clean":"2024-06-29T15:38:40.168901+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:38:40.168901+0000","last_undegraded":"2024-06-29T15:38:40.168901+0000","last_fullsized":"2024-06-29T15:38:40.168901+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":293,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":293,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2789376,"data_stored":1709564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543066,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41636,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330204,"statfs":{"total":96636764160,"available":96594128896,"internally_reserved":0,"allocated":2183168,"data_stored":1112524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47208,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324632,"statfs":{"total":96636764160,"available":96588423168,"internally_reserved":0,"allocated":2191360,"data_stored":1119174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543064,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29176,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342664,"statfs":{"total":96636764160,"available":96606887936,"internally_reserved":0,"allocated":2203648,"data_stored":1120786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510362,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59688,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312152,"statfs":{"total":96636764160,"available":96575643648,"internally_reserved":0,"allocated":2187264,"data_stored":1113070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575770,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63844,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307996,"statfs":{"total":96636764160,"available":96571387904,"internally_reserved":0,"allocated":2187264,"data_stored":1112566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510361,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41632,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330208,"statfs":{"total":96636764160,"available":96594132992,"internally_reserved":0,"allocated":2183168,"data_stored":1112524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510363,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38072,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333768,"statfs":{"total":96636764160,"available":96597778432,"internally_reserved":0,"allocated":2797568,"data_stored":1705140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:38:43.616 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:38:43.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:38:43.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 -- 172.21.15.59:0/4228523905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 msgr2=0x7f566815dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 --2- 172.21.15.59:0/4228523905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 0x7f566815dc10 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f56680d3990 tx=0x7f566402a560 comp rx=0 tx=0).stop 2024-06-29T15:38:43.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 -- 172.21.15.59:0/4228523905 shutdown_connections 2024-06-29T15:38:43.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 --2- 172.21.15.59:0/4228523905 >> v2:172.21.15.153:3300/0 conn(0x7f5668161980 0x7f5668161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 --2- 172.21.15.59:0/4228523905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f56680cebf0 0x7f56680cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 --2- 172.21.15.59:0/4228523905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 0x7f566815dc10 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 -- 172.21.15.59:0/4228523905 >> 172.21.15.59:0/4228523905 conn(0x7f56680d3e40 msgr2=0x7f56680d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 -- 172.21.15.59:0/4228523905 shutdown_connections 2024-06-29T15:38:43.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.778+0000 7f566fe4e640 1 -- 172.21.15.59:0/4228523905 wait complete. 2024-06-29T15:38:43.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.780+0000 7f566fe4e640 1 Processor -- start 2024-06-29T15:38:43.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.780+0000 7f566fe4e640 1 -- start start 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566fe4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 0x7f56680c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566fe4e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56680cebf0 0x7f566815b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566fe4e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5668161980 0x7f56680cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566fe4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f56680b2b90 con 0x7f56680ce510 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566fe4e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f56680b2a10 con 0x7f56680cebf0 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566fe4e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f56681622d0 con 0x7f5668161980 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566dbea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 0x7f56680c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566dbea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 0x7f56680c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56778/0 (socket says 172.21.15.59:56778) 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566dbea640 1 -- 172.21.15.59:0/2963761748 learned_addr learned my addr 172.21.15.59:0/2963761748 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566d3e9640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56680cebf0 0x7f566815b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566e3eb640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5668161980 0x7f56680cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566dbea640 1 -- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5668161980 msgr2=0x7f56680cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566dbea640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5668161980 0x7f56680cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566dbea640 1 -- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56680cebf0 msgr2=0x7f566815b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566dbea640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56680cebf0 0x7f566815b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.781+0000 7f566dbea640 1 -- 172.21.15.59:0/2963761748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56680c8a80 con 0x7f56680ce510 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.782+0000 7f566dbea640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 0x7f56680c8150 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f5664034040 tx=0x7f5664003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.782+0000 7f565e7fc640 1 -- 172.21.15.59:0/2963761748 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566403e030 con 0x7f56680ce510 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.782+0000 7f565e7fc640 1 -- 172.21.15.59:0/2963761748 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5664003e00 con 0x7f56680ce510 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.782+0000 7f565e7fc640 1 -- 172.21.15.59:0/2963761748 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5664035040 con 0x7f56680ce510 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.782+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56680c8d60 con 0x7f56680ce510 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.782+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56680ca4b0 con 0x7f56680ce510 2024-06-29T15:38:43.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.783+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5668162730 con 0x7f56680ce510 2024-06-29T15:38:43.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.784+0000 7f565e7fc640 1 -- 172.21.15.59:0/2963761748 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f5664041030 con 0x7f56680ce510 2024-06-29T15:38:43.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.784+0000 7f565e7fc640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f564c0763f0 0x7f564c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.784+0000 7f565e7fc640 1 -- 172.21.15.59:0/2963761748 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f56640b8470 con 0x7f56680ce510 2024-06-29T15:38:43.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.784+0000 7f566d3e9640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f564c0763f0 0x7f564c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.785+0000 7f566d3e9640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f564c0763f0 0x7f564c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5658005fd0 tx=0x7f5658005850 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:43.788+0000 7f565e7fc640 1 -- 172.21.15.59:0/2963761748 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f56640818f0 con 0x7f56680ce510 2024-06-29T15:38:44.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.056+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f566815c5d0 con 0x7f56680ce510 2024-06-29T15:38:44.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.058+0000 7f565e7fc640 1 -- 172.21.15.59:0/2963761748 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v43) ==== 190+0+147 (secure 0 0 0) 0x7f56640811f0 con 0x7f56680ce510 2024-06-29T15:38:44.059 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T15:38:44.060 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T15:38:44.060 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T15:38:44.060 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T15:38:44.060 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:38:44.060 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.060+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f564c0763f0 msgr2=0x7f564c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.060+0000 7f566fe4e640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f564c0763f0 0x7f564c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5658005fd0 tx=0x7f5658005850 comp rx=0 tx=0).stop 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.060+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 msgr2=0x7f56680c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.060+0000 7f566fe4e640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 0x7f56680c8150 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f5664034040 tx=0x7f5664003910 comp rx=0 tx=0).stop 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.061+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 shutdown_connections 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.061+0000 7f566fe4e640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5668161980 0x7f56680cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.061+0000 7f566fe4e640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56680cebf0 0x7f566815b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.061+0000 7f566fe4e640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f564c0763f0 0x7f564c078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.061+0000 7f566fe4e640 1 --2- 172.21.15.59:0/2963761748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56680ce510 0x7f56680c8150 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.061+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 >> 172.21.15.59:0/2963761748 conn(0x7f56680d3e40 msgr2=0x7f56680b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.061+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 shutdown_connections 2024-06-29T15:38:44.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.061+0000 7f566fe4e640 1 -- 172.21.15.59:0/2963761748 wait complete. 2024-06-29T15:38:44.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:44.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.190+0000 7fa635312640 1 -- 172.21.15.59:0/62911471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6300b2fa0 msgr2=0x7fa6300d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.190+0000 7fa635312640 1 --2- 172.21.15.59:0/62911471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6300b2fa0 0x7fa6300d1520 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa624005fd0 tx=0x7fa62402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.191+0000 7fa635312640 1 -- 172.21.15.59:0/62911471 shutdown_connections 2024-06-29T15:38:44.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.191+0000 7fa635312640 1 --2- 172.21.15.59:0/62911471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6300b2fa0 0x7fa6300d1520 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.191+0000 7fa635312640 1 --2- 172.21.15.59:0/62911471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6300b2270 0x7fa63015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.191+0000 7fa635312640 1 -- 172.21.15.59:0/62911471 >> 172.21.15.59:0/62911471 conn(0x7fa630073e10 msgr2=0x7fa630074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.191+0000 7fa635312640 1 -- 172.21.15.59:0/62911471 shutdown_connections 2024-06-29T15:38:44.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.191+0000 7fa635312640 1 -- 172.21.15.59:0/62911471 wait complete. 2024-06-29T15:38:44.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.192+0000 7fa635312640 1 Processor -- start 2024-06-29T15:38:44.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.193+0000 7fa635312640 1 -- start start 2024-06-29T15:38:44.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa635312640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6300b2270 0x7fa63006f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa635312640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300b2fa0 0x7fa6301461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa635312640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6300d1a60 0x7fa630158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa635312640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa6300b6cc0 con 0x7fa6300b2fa0 2024-06-29T15:38:44.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa635312640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6300b6b40 con 0x7fa6300d1a60 2024-06-29T15:38:44.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa635312640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa6300cfe90 con 0x7fa6300b2270 2024-06-29T15:38:44.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300b2fa0 0x7fa6301461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300b2fa0 0x7fa6301461f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56782/0 (socket says 172.21.15.59:56782) 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62e7fc640 1 -- 172.21.15.59:0/3824854261 learned_addr learned my addr 172.21.15.59:0/3824854261 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62effd640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6300b2270 0x7fa63006f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62f7fe640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6300d1a60 0x7fa630158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62e7fc640 1 -- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6300b2270 msgr2=0x7fa63006f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62e7fc640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6300b2270 0x7fa63006f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62e7fc640 1 -- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6300d1a60 msgr2=0x7fa630158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62e7fc640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6300d1a60 0x7fa630158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.194+0000 7fa62e7fc640 1 -- 172.21.15.59:0/3824854261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa630150960 con 0x7fa6300b2fa0 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.195+0000 7fa62e7fc640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300b2fa0 0x7fa6301461f0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa624034040 tx=0x7fa6240035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.195+0000 7fa6177fe640 1 -- 172.21.15.59:0/3824854261 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62403e030 con 0x7fa6300b2fa0 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.195+0000 7fa6177fe640 1 -- 172.21.15.59:0/3824854261 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa624003ae0 con 0x7fa6300b2fa0 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.195+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6300b0690 con 0x7fa6300b2fa0 2024-06-29T15:38:44.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.195+0000 7fa6177fe640 1 -- 172.21.15.59:0/3824854261 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624035040 con 0x7fa6300b2fa0 2024-06-29T15:38:44.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.195+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6300b00f0 con 0x7fa6300b2fa0 2024-06-29T15:38:44.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.196+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6300adab0 con 0x7fa6300b2fa0 2024-06-29T15:38:44.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.197+0000 7fa6177fe640 1 -- 172.21.15.59:0/3824854261 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fa624041030 con 0x7fa6300b2fa0 2024-06-29T15:38:44.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.198+0000 7fa6177fe640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fa61c076420 0x7fa61c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.198+0000 7fa6177fe640 1 -- 172.21.15.59:0/3824854261 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7fa6240b8490 con 0x7fa6300b2fa0 2024-06-29T15:38:44.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.198+0000 7fa62effd640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fa61c076420 0x7fa61c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.199+0000 7fa62effd640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fa61c076420 0x7fa61c078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa6200095f0 tx=0x7fa620005f90 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:44.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.201+0000 7fa6177fe640 1 -- 172.21.15.59:0/3824854261 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa6240813d0 con 0x7fa6300b2fa0 2024-06-29T15:38:44.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.429+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6300d0340 con 0x7fa6300b2fa0 2024-06-29T15:38:44.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.429+0000 7fa6177fe640 1 -- 172.21.15.59:0/3824854261 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5233 (secure 0 0 0) 0x7fa6240856d0 con 0x7fa6300b2fa0 2024-06-29T15:38:44.431 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:44.431 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"btime":"2024-06-29T15:38:42:164025+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1947749735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1947749735},{"type":"v1","addr":"172.21.15.59:6835","nonce":1947749735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3010048644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3010048644},{"type":"v1","addr":"172.21.15.153:6836","nonce":3010048644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9283,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1912783986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1912783986},{"type":"v1","addr":"172.21.15.153:6837","nonce":1912783986}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.523073+0000","modified":"2024-06-29T15:38:41.554713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1057565935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1057565935},{"type":"v1","addr":"172.21.15.59:6834","nonce":1057565935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9894,"qdb_cluster":[9894]},"id":10}]} 2024-06-29T15:38:44.431 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.431+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fa61c076420 msgr2=0x7fa61c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fa61c076420 0x7fa61c078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa6200095f0 tx=0x7fa620005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300b2fa0 msgr2=0x7fa6301461f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300b2fa0 0x7fa6301461f0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa624034040 tx=0x7fa6240035f0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 shutdown_connections 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6300d1a60 0x7fa630158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fa61c076420 0x7fa61c078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300b2fa0 0x7fa6301461f0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 --2- 172.21.15.59:0/3824854261 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa6300b2270 0x7fa63006f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 >> 172.21.15.59:0/3824854261 conn(0x7fa630073e10 msgr2=0x7fa630074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 shutdown_connections 2024-06-29T15:38:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.432+0000 7fa635312640 1 -- 172.21.15.59:0/3824854261 wait complete. 2024-06-29T15:38:44.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:38:38.523073+0000', 'modified': '2024-06-29T15:38:41.554713+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9894}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9894': {'gid': 9894, 'name': 'c', 'rank': 0, 'incarnation': 85, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6834/1057565935', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1057565935}, {'type': 'v1', 'addr': '172.21.15.59:6834', 'nonce': 1057565935}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9894, 'qdb_cluster': [9894]} 2024-06-29T15:38:44.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:38:44.847 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-06-29T15:38:44.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.969+0000 7f85c237a640 1 -- 172.21.15.59:0/812057431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc0ce950 msgr2=0x7f85bc1622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.969+0000 7f85c237a640 1 --2- 172.21.15.59:0/812057431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc0ce950 0x7f85bc1622c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0b78c0 tx=0x7f85b002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.969+0000 7f85c237a640 1 -- 172.21.15.59:0/812057431 shutdown_connections 2024-06-29T15:38:44.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.969+0000 7f85c237a640 1 --2- 172.21.15.59:0/812057431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc0ce950 0x7f85bc1622c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.969+0000 7f85c237a640 1 -- 172.21.15.59:0/812057431 >> 172.21.15.59:0/812057431 conn(0x7f85bc0b5610 msgr2=0x7f85bc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.969+0000 7f85c237a640 1 -- 172.21.15.59:0/812057431 shutdown_connections 2024-06-29T15:38:44.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.969+0000 7f85c237a640 1 -- 172.21.15.59:0/812057431 wait complete. 2024-06-29T15:38:44.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.971+0000 7f85c237a640 1 Processor -- start 2024-06-29T15:38:44.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.971+0000 7f85c237a640 1 -- start start 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c237a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc08c190 0x7f85bc0b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c237a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85bc0ce950 0x7f85bc1f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c237a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc162800 0x7f85bc15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c237a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc163110 con 0x7f85bc162800 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c237a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc162f90 con 0x7f85bc08c190 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c237a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f85bc0b82e0 con 0x7f85bc0ce950 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc162800 0x7f85bc15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc162800 0x7f85bc15bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56788/0 (socket says 172.21.15.59:56788) 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 -- 172.21.15.59:0/1078871545 learned_addr learned my addr 172.21.15.59:0/1078871545 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:44.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85bbfff640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc08c190 0x7f85bc0b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85bc0ce950 0x7f85bc1f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 -- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85bc0ce950 msgr2=0x7f85bc1f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85bc0ce950 0x7f85bc1f4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 -- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc08c190 msgr2=0x7f85bc0b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc08c190 0x7f85bc0b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 -- 172.21.15.59:0/1078871545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85bc0aaa40 con 0x7f85bc162800 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85bbfff640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc08c190 0x7f85bc0b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85c0917640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc162800 0x7f85bc15bd80 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f85b4002230 tx=0x7f85b400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.973+0000 7f85b8ff9640 1 -- 172.21.15.59:0/1078871545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4003b70 con 0x7f85bc162800 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.974+0000 7f85b8ff9640 1 -- 172.21.15.59:0/1078871545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85b4003d10 con 0x7f85bc162800 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.974+0000 7f85c237a640 1 -- 172.21.15.59:0/1078871545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85bc0aad20 con 0x7f85bc162800 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.974+0000 7f85b8ff9640 1 -- 172.21.15.59:0/1078871545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4005330 con 0x7f85bc162800 2024-06-29T15:38:44.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.974+0000 7f85c237a640 1 -- 172.21.15.59:0/1078871545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85bc0ab2e0 con 0x7f85bc162800 2024-06-29T15:38:44.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.975+0000 7f85b8ff9640 1 -- 172.21.15.59:0/1078871545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f85b40056e0 con 0x7f85bc162800 2024-06-29T15:38:44.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.976+0000 7f85b8ff9640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f85a80763f0 0x7f85a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.976+0000 7f85b8ff9640 1 -- 172.21.15.59:0/1078871545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f85b4097ce0 con 0x7f85bc162800 2024-06-29T15:38:44.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.976+0000 7f85bbfff640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f85a80763f0 0x7f85a8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.977+0000 7f85c9237740 1 Processor -- start 2024-06-29T15:38:44.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.977+0000 7f85bbfff640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f85a80763f0 0x7f85a8078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f85ac001860 tx=0x7f85ac009040 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:44.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.977+0000 7f85c9237740 1 -- start start 2024-06-29T15:38:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85c9237740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 0x563df74c16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85c9237740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85c9237740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70b4590 0x563df74a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85c9237740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x563df743dbc0 con 0x563df70ac3b0 2024-06-29T15:38:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85c9237740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563df73ec630 con 0x563df70d5270 2024-06-29T15:38:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85c9237740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x563df73dbbd0 con 0x563df70b4590 2024-06-29T15:38:44.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56796/0 (socket says 172.21.15.59:56796) 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85bb7fe640 1 -- 172.21.15.59:0/2777698424 learned_addr learned my addr 172.21.15.59:0/2777698424 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85bbfff640 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 0x563df74c16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.978+0000 7f85c0917640 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70b4590 0x563df74a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85bb7fe640 1 -- 172.21.15.59:0/2777698424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70b4590 msgr2=0x563df74a58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70b4590 0x563df74a58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85bb7fe640 1 -- 172.21.15.59:0/2777698424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 msgr2=0x563df74c16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 0x563df74c16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85bb7fe640 1 -- 172.21.15.59:0/2777698424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563df73d8ca0 con 0x563df70ac3b0 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74a53a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009950 tx=0x7f85b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c54cdccf5faeed server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85a1ffb640 1 -- 172.21.15.59:0/2777698424 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0040070 con 0x563df70ac3b0 2024-06-29T15:38:44.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85a1ffb640 1 -- 172.21.15.59:0/2777698424 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85b0005020 con 0x563df70ac3b0 2024-06-29T15:38:44.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.979+0000 7f85a1ffb640 1 -- 172.21.15.59:0/2777698424 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0005450 con 0x563df70ac3b0 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 -- 172.21.15.59:0/2777698424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 msgr2=0x563df74a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74a53a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009950 tx=0x7f85b00046c0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 -- 172.21.15.59:0/2777698424 shutdown_connections 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70b4590 0x563df74a58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74a53a0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 --2- 172.21.15.59:0/2777698424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 0x563df74c16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 -- 172.21.15.59:0/2777698424 >> 172.21.15.59:0/2777698424 conn(0x563df70c93c0 msgr2=0x563df7478f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 -- 172.21.15.59:0/2777698424 shutdown_connections 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.980+0000 7f85c9237740 1 -- 172.21.15.59:0/2777698424 wait complete. 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.981+0000 7f85c9237740 1 -- start start 2024-06-29T15:38:44.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.981+0000 7f85c9237740 1 Processor -- start 2024-06-29T15:38:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.982+0000 7f85c9237740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70ac3b0 0x563df74bced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.982+0000 7f85c9237740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.982+0000 7f85c9237740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 0x563df74a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.982+0000 7f85c9237740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x563df7379ae0 con 0x563df70b4590 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.982+0000 7f85c9237740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563df73dfc10 con 0x563df70d5270 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.982+0000 7f85c9237740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x563df714f620 con 0x563df70ac3b0 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.982+0000 7f85c9237740 10 client.0 _ll_register_callbacks cb 0x563df7386480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74a4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56806/0 (socket says 172.21.15.59:56806) 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 -- 172.21.15.59:0/248990791 learned_addr learned my addr 172.21.15.59:0/248990791 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bbfff640 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70ac3b0 0x563df74bced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85c0917640 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 0x563df74a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 -- 172.21.15.59:0/248990791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70ac3b0 msgr2=0x563df74bced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70ac3b0 0x563df74bced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 -- 172.21.15.59:0/248990791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 msgr2=0x563df74a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 0x563df74a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 -- 172.21.15.59:0/248990791 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563df7140850 con 0x563df70b4590 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74a4eb0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x563df74c3420 tx=0x7f85b002ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a590472a3021c728 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f859affd640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f859affd640 1 -- 172.21.15.59:0/248990791 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0040040 con 0x563df70b4590 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f859affd640 1 -- 172.21.15.59:0/248990791 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f85b0004240 con 0x563df70b4590 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85c9237740 1 -- 172.21.15.59:0/248990791 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85b0005020 con 0x563df70b4590 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f859affd640 1 -- 172.21.15.59:0/248990791 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f85b0055050 con 0x563df70b4590 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85c9237740 10 client.10041 fetch_fsmap learned FSMap version 88 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.983+0000 7f85c9237740 1 -- 172.21.15.59:0/248990791 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x563df71b76b0 con 0x563df70b4590 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f859affd640 1 -- 172.21.15.59:0/248990791 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85b0005020 con 0x563df70b4590 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f859affd640 1 -- 172.21.15.59:0/248990791 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 88) ==== 3502+0+0 (secure 0 0 0) 0x7f85b0058020 con 0x563df70b4590 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f85c9237740 10 client.10041 fetch_fsmap finished waiting for FSMap version 88 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f85c9237740 10 client.10041 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:38:44.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f85c9237740 20 client.10041 populate_metadata read hostname 'smithi059' 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f85c9237740 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] conn(0x563df7093e40 0x563df74d37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f85c9237740 4 client.10041 mds_command: new command op to 9894 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f85c9237740 1 -- 172.21.15.59:0/248990791 --> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563df73ee580 con 0x563df7093e40 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f85bbfff640 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] conn(0x563df7093e40 0x563df74d37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f859affd640 10 client.10041 ms_handle_connect on v2:172.21.15.59:6832/1057565935 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.984+0000 7f85bbfff640 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] conn(0x563df7093e40 0x563df74d37d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb417ada82e5a73 server_cookie=591f0434221f76b1 in_seq=0 out_seq=0 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f859affd640 1 -- 172.21.15.59:0/248990791 <== mds.0 v2:172.21.15.59:6832/1057565935 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x563df73ee580 con 0x563df7093e40 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f859affd640 10 client.10041 handle_command_reply: tid=0 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 client.10041 shutdown 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 -- 172.21.15.59:0/248990791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 msgr2=0x563df74a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74a4eb0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x563df74c3420 tx=0x7f85b002ab90 comp rx=0 tx=0).stop 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 -- 172.21.15.59:0/248990791 shutdown_connections 2024-06-29T15:38:44.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] conn(0x563df7093e40 0x563df74d37d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70d5270 0x563df74a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74a4eb0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 --2- 172.21.15.59:0/248990791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70ac3b0 0x563df74bced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f859affd640 0 client.10041 ms_handle_reset on v2:172.21.15.59:6832/1057565935 2024-06-29T15:38:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 -- 172.21.15.59:0/248990791 >> 172.21.15.59:0/248990791 conn(0x563df70c93c0 msgr2=0x563df7474b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 -- 172.21.15.59:0/248990791 shutdown_connections 2024-06-29T15:38:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.985+0000 7f85c9237740 1 -- 172.21.15.59:0/248990791 wait complete. 2024-06-29T15:38:44.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:44.986+0000 7f85c9237740 20 client.10041 trim_cache size 0 max 16384 2024-06-29T15:38:45.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.005+0000 7f85c9237740 1 Processor -- start 2024-06-29T15:38:45.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.005+0000 7f85c9237740 1 -- start start 2024-06-29T15:38:45.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.006+0000 7f85c9237740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74bdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.006+0000 7f85c9237740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70b4590 0x563df747ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.006+0000 7f85c9237740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 0x563df747b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.006+0000 7f85c9237740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc200bb0 con 0x563df70ac3b0 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.006+0000 7f85c9237740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc0cb4e0 con 0x563df70b4590 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.006+0000 7f85c9237740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f85bc200660 con 0x563df70d5270 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74bdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74bdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56814/0 (socket says 172.21.15.59:56814) 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 -- 172.21.15.59:0/2890385300 learned_addr learned my addr 172.21.15.59:0/2890385300 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85c0917640 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 0x563df747b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70b4590 0x563df747ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 -- 172.21.15.59:0/2890385300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 msgr2=0x563df747b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 0x563df747b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 -- 172.21.15.59:0/2890385300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70b4590 msgr2=0x563df747ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70b4590 0x563df747ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 -- 172.21.15.59:0/2890385300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563df74a2440 con 0x563df70ac3b0 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85bbfff640 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74bdd50 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f85ac0013c0 tx=0x7f85ac013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71bc75b450e8e07 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85a17fa640 1 -- 172.21.15.59:0/2890385300 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac023820 con 0x563df70ac3b0 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85a17fa640 1 -- 172.21.15.59:0/2890385300 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85ac02f070 con 0x563df70ac3b0 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.007+0000 7f85a17fa640 1 -- 172.21.15.59:0/2890385300 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0239c0 con 0x563df70ac3b0 2024-06-29T15:38:45.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 -- 172.21.15.59:0/2890385300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 msgr2=0x563df74bdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74bdd50 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f85ac0013c0 tx=0x7f85ac013000 comp rx=0 tx=0).stop 2024-06-29T15:38:45.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 -- 172.21.15.59:0/2890385300 shutdown_connections 2024-06-29T15:38:45.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 0x563df747b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70b4590 0x563df747ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 --2- 172.21.15.59:0/2890385300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70ac3b0 0x563df74bdd50 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 -- 172.21.15.59:0/2890385300 >> 172.21.15.59:0/2890385300 conn(0x563df70c93c0 msgr2=0x563df74c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 -- 172.21.15.59:0/2890385300 shutdown_connections 2024-06-29T15:38:45.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.008+0000 7f85c9237740 1 -- 172.21.15.59:0/2890385300 wait complete. 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.009+0000 7f85c9237740 1 -- start start 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.009+0000 7f85c9237740 1 Processor -- start 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85c9237740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70ac3b0 0x563df74d5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85c9237740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74d5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85c9237740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 0x563df753d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85c9237740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x563df74a59b0 con 0x563df70b4590 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85c9237740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563df74a4460 con 0x563df70ac3b0 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85c9237740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x563df74befc0 con 0x563df70d5270 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85c9237740 10 client.0 _ll_register_callbacks cb 0x7f85ac007ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:38:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74d5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74d5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56822/0 (socket says 172.21.15.59:56822) 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 -- 172.21.15.59:0/1208237069 learned_addr learned my addr 172.21.15.59:0/1208237069 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bbfff640 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70ac3b0 0x563df74d5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85c0917640 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 0x563df753d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 -- 172.21.15.59:0/1208237069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 msgr2=0x563df753d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 0x563df753d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 -- 172.21.15.59:0/1208237069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70ac3b0 msgr2=0x563df74d5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70ac3b0 0x563df74d5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 -- 172.21.15.59:0/1208237069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563df74a4e20 con 0x563df70b4590 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f85bb7fe640 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74d5590 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f85b0005cb0 tx=0x7f85b0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca9353dbf2a381a server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f859affd640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.011+0000 7f859affd640 1 -- 172.21.15.59:0/1208237069 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0040070 con 0x563df70b4590 2024-06-29T15:38:45.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f859affd640 1 -- 172.21.15.59:0/1208237069 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f85b005ac50 con 0x563df70b4590 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 1 -- 172.21.15.59:0/1208237069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85ac02f070 con 0x563df70b4590 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f859affd640 1 -- 172.21.15.59:0/1208237069 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f85b005ae50 con 0x563df70b4590 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 10 client.10047 fetch_fsmap learned FSMap version 88 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 1 -- 172.21.15.59:0/1208237069 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) -- 0x563df753dca0 con 0x563df70b4590 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f859affd640 1 -- 172.21.15.59:0/1208237069 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85ac02f070 con 0x563df70b4590 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f859affd640 1 -- 172.21.15.59:0/1208237069 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 88) ==== 3502+0+0 (secure 0 0 0) 0x7f85b0058040 con 0x563df70b4590 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 10 client.10047 fetch_fsmap finished waiting for FSMap version 88 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 10 client.10047 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 20 client.10047 populate_metadata read hostname 'smithi059' 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] conn(0x563df753d6f0 0x563df74a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 4 client.10047 mds_command: new command op to 9894 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:38:45.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85c9237740 1 -- 172.21.15.59:0/1208237069 --> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] -- command(tid 0: {"prefix": "status"}) -- 0x563df74a4ab0 con 0x563df753d6f0 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.012+0000 7f85bbfff640 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] conn(0x563df753d6f0 0x563df74a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f859affd640 10 client.10047 ms_handle_connect on v2:172.21.15.59:6832/1057565935 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85bbfff640 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] conn(0x563df753d6f0 0x563df74a08c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d54403c7739d650 server_cookie=d86bd605bdb45e38 in_seq=0 out_seq=0 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f859affd640 1 -- 172.21.15.59:0/1208237069 <== mds.0 v2:172.21.15.59:6832/1057565935 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x563df74a4ab0 con 0x563df753d6f0 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f859affd640 10 client.10047 handle_command_reply: tid=0 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 client.10047 shutdown 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 -- 172.21.15.59:0/1208237069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 msgr2=0x563df74d5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74d5590 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f85b0005cb0 tx=0x7f85b0005d30 comp rx=0 tx=0).stop 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 -- 172.21.15.59:0/1208237069 shutdown_connections 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.59:6832/1057565935,v1:172.21.15.59:6834/1057565935] conn(0x563df753d6f0 0x563df74a08c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x563df70d5270 0x563df753d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f859affd640 0 client.10047 ms_handle_reset on v2:172.21.15.59:6832/1057565935 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563df70b4590 0x563df74d5590 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 --2- 172.21.15.59:0/1208237069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563df70ac3b0 0x563df74d5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 -- 172.21.15.59:0/1208237069 >> 172.21.15.59:0/1208237069 conn(0x563df70c93c0 msgr2=0x563df74c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 -- 172.21.15.59:0/1208237069 shutdown_connections 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 1 -- 172.21.15.59:0/1208237069 wait complete. 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.013+0000 7f85c9237740 20 client.10047 trim_cache size 0 max 16384 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 9894, 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:38:45.021 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T15:38:45.022 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8613597869999996, 2024-06-29T15:38:45.022 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 88, 2024-06-29T15:38:45.022 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 176, 2024-06-29T15:38:45.022 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 176, 2024-06-29T15:38:45.022 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.496065659999999 2024-06-29T15:38:45.022 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 -- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f85a80763f0 msgr2=0x7f85a8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f85a80763f0 0x7f85a8078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f85ac001860 tx=0x7f85ac009040 comp rx=0 tx=0).stop 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 -- 172.21.15.59:0/1078871545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc162800 msgr2=0x7f85bc15bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc162800 0x7f85bc15bd80 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f85b4002230 tx=0x7f85b400cb90 comp rx=0 tx=0).stop 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 -- 172.21.15.59:0/1078871545 shutdown_connections 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f85a80763f0 0x7f85a8078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85bc162800 0x7f85bc15bd80 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85bc0ce950 0x7f85bc1f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 --2- 172.21.15.59:0/1078871545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc08c190 0x7f85bc0b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 -- 172.21.15.59:0/1078871545 >> 172.21.15.59:0/1078871545 conn(0x7f85bc0b5610 msgr2=0x7f85bc15dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 -- 172.21.15.59:0/1078871545 shutdown_connections 2024-06-29T15:38:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.014+0000 7f85c237a640 1 -- 172.21.15.59:0/1078871545 wait complete. 2024-06-29T15:38:45.026 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:38:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.143+0000 7f3e06f4a640 1 -- 172.21.15.59:0/1683410743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dec003ac0 msgr2=0x7f3dec003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.143+0000 7f3e06f4a640 1 --2- 172.21.15.59:0/1683410743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dec003ac0 0x7f3dec003f30 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009ed0 tx=0x7f3dec005830 comp rx=0 tx=0).stop 2024-06-29T15:38:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.143+0000 7f3e054e7640 1 -- 172.21.15.59:0/1683410743 reap_dead start 2024-06-29T15:38:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.143+0000 7f3e06f4a640 1 -- 172.21.15.59:0/1683410743 shutdown_connections 2024-06-29T15:38:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.143+0000 7f3e06f4a640 1 -- 172.21.15.59:0/1683410743 >> 172.21.15.59:0/1683410743 conn(0x7f3e000b24c0 msgr2=0x7f3e000cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.143+0000 7f3e06f4a640 1 -- 172.21.15.59:0/1683410743 shutdown_connections 2024-06-29T15:38:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.143+0000 7f3e06f4a640 1 -- 172.21.15.59:0/1683410743 wait complete. 2024-06-29T15:38:45.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.145+0000 7f3e06f4a640 1 Processor -- start 2024-06-29T15:38:45.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.145+0000 7f3e06f4a640 1 -- start start 2024-06-29T15:38:45.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3e06f4a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dec003ac0 0x7f3e00139a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3e06f4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e001f4c60 0x7f3e00137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3e06f4a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e00138d60 0x7f3e0013ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3e06f4a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3e001448f0 con 0x7f3e001f4c60 2024-06-29T15:38:45.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3e06f4a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e00144770 con 0x7f3dec003ac0 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3e06f4a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0006d700 con 0x7f3e00138d60 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e001f4c60 0x7f3e00137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e001f4c60 0x7f3e00137190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56842/0 (socket says 172.21.15.59:56842) 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 -- 172.21.15.59:0/2732932289 learned_addr learned my addr 172.21.15.59:0/2732932289 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3e04ce6640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dec003ac0 0x7f3e00139a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3e054e7640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e00138d60 0x7f3e0013ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 -- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e00138d60 msgr2=0x7f3e0013ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e00138d60 0x7f3e0013ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 -- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dec003ac0 msgr2=0x7f3e00139a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dec003ac0 0x7f3e00139a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 -- 172.21.15.59:0/2732932289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e001f5440 con 0x7f3e001f4c60 2024-06-29T15:38:45.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.147+0000 7f3dfffff640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e001f4c60 0x7f3e00137190 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f3de8011ab0 tx=0x7f3de8011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.148+0000 7f3dfd7fa640 1 -- 172.21.15.59:0/2732932289 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de801e070 con 0x7f3e001f4c60 2024-06-29T15:38:45.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.148+0000 7f3dfd7fa640 1 -- 172.21.15.59:0/2732932289 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3de8004460 con 0x7f3e001f4c60 2024-06-29T15:38:45.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.148+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e001f5720 con 0x7f3e001f4c60 2024-06-29T15:38:45.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.148+0000 7f3dfd7fa640 1 -- 172.21.15.59:0/2732932289 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de800fc60 con 0x7f3e001f4c60 2024-06-29T15:38:45.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.148+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e00139170 con 0x7f3e001f4c60 2024-06-29T15:38:45.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.149+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e0015bc80 con 0x7f3e001f4c60 2024-06-29T15:38:45.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.150+0000 7f3dfd7fa640 1 -- 172.21.15.59:0/2732932289 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f3de8021070 con 0x7f3e001f4c60 2024-06-29T15:38:45.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.150+0000 7f3dfd7fa640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f3de0076420 0x7f3de0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.150+0000 7f3dfd7fa640 1 -- 172.21.15.59:0/2732932289 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f3de8098470 con 0x7f3e001f4c60 2024-06-29T15:38:45.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.150+0000 7f3e04ce6640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f3de0076420 0x7f3de0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.151+0000 7f3e04ce6640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f3de0076420 0x7f3de0078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3df0035000 tx=0x7f3df002d040 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.155+0000 7f3dfd7fa640 1 -- 172.21.15.59:0/2732932289 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3de80613b0 con 0x7f3e001f4c60 2024-06-29T15:38:45.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.392+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3e000b0840 con 0x7f3e001f4c60 2024-06-29T15:38:45.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.393+0000 7f3dfd7fa640 1 -- 172.21.15.59:0/2732932289 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v88) ==== 53+0+73 (secure 0 0 0) 0x7f3de80656b0 con 0x7f3e001f4c60 2024-06-29T15:38:45.394 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f3de0076420 msgr2=0x7f3de0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f3de0076420 0x7f3de0078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3df0035000 tx=0x7f3df002d040 comp rx=0 tx=0).stop 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e001f4c60 msgr2=0x7f3e00137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e001f4c60 0x7f3e00137190 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f3de8011ab0 tx=0x7f3de8011f70 comp rx=0 tx=0).stop 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 shutdown_connections 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e00138d60 0x7f3e0013ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f3de0076420 0x7f3de0078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e001f4c60 0x7f3e00137190 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 --2- 172.21.15.59:0/2732932289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dec003ac0 0x7f3e00139a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 >> 172.21.15.59:0/2732932289 conn(0x7f3e000b24c0 msgr2=0x7f3e0015c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 shutdown_connections 2024-06-29T15:38:45.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45.395+0000 7f3e06f4a640 1 -- 172.21.15.59:0/2732932289 wait complete. 2024-06-29T15:38:45.407 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:38:45.407 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:38:45.408 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T15:38:45.408 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:38:45.408 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:45.408 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:38:45.408 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:38:45.408 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:38:45.408 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:38:45.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.408 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T15:38:45.425 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:38:45.425 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:38:45.425 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:38:45.425 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T15:38:45.426 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 38125sec preferred_lft 38125sec 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T15:38:45.427 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.428 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:38:45.428 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:38:45.428 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T15:38:45.428 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.428 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:38:45.490 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T15:38:45.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.492 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T15:38:45.549 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T15:38:45.549 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:38:45.550 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T15:38:45.550 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T15:38:45.550 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:38:45.550 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:38:45.550 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T15:38:45.550 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T15:38:45.550 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T15:38:45.550 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T15:38:45.550 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T15:38:45.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:38:45.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:38:45.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.681 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T15:38:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:38:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:38:45.697 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:38:45.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.697 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T15:38:45.762 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:38:45.763 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:45.819 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:38:45.819 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:38:45.819 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:45.874 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T15:38:45.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:38:45.970 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:38:45.970 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:38:45.970 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:38:45.981 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:38:45.981 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:38:45.981 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:38:45.981 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:38:45.981 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:38:45.988 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:38:45.988 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:38:45.988 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:38:45.989 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:38:45.989 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:38:45.989 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T15:38:45.989 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T15:38:45.989 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T15:38:45.989 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:38:45.989 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:38:45.989 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:38:45.990 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:38:45.990 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:38:45.991 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:38:45.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:46.022 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:38:46.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:38:46.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:38:46.124 INFO:teuthology.orchestra.run.smithi059.stdout:client8324 2024-06-29T15:38:46.124 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:46.125 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:46.142 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:38:46.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.299+0000 7f0ccf41f640 1 -- 172.21.15.59:0/984011799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc80cd9b0 msgr2=0x7f0cc815f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.299+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/984011799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc80cd9b0 0x7f0cc815f240 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8005fd0 tx=0x7f0cb802aa00 comp rx=0 tx=0).stop 2024-06-29T15:38:46.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.299+0000 7f0ccf41f640 1 -- 172.21.15.59:0/984011799 shutdown_connections 2024-06-29T15:38:46.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.299+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/984011799 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cc81444c0 0x7f0cc806d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.299+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/984011799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc80cd9b0 0x7f0cc815f240 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.299+0000 7f0ccf41f640 1 -- 172.21.15.59:0/984011799 >> 172.21.15.59:0/984011799 conn(0x7f0cc80b24c0 msgr2=0x7f0cc80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:46.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.299+0000 7f0ccf41f640 1 -- 172.21.15.59:0/984011799 shutdown_connections 2024-06-29T15:38:46.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.299+0000 7f0ccf41f640 1 -- 172.21.15.59:0/984011799 wait complete. 2024-06-29T15:38:46.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.301+0000 7f0ccf41f640 1 Processor -- start 2024-06-29T15:38:46.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.301+0000 7f0ccf41f640 1 -- start start 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.302+0000 7f0ccf41f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc806d830 0x7f0cc806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.302+0000 7f0ccf41f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cc80cd9b0 0x7f0cc806c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.302+0000 7f0ccf41f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cc81444c0 0x7f0cc80ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.302+0000 7f0ccf41f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc815b9b0 con 0x7f0cc806d830 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.302+0000 7f0ccf41f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc8074300 con 0x7f0cc80cd9b0 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.302+0000 7f0ccf41f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0cc815bb30 con 0x7f0cc81444c0 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.302+0000 7f0ccd1bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc806d830 0x7f0cc806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd1bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc806d830 0x7f0cc806c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56856/0 (socket says 172.21.15.59:56856) 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd1bb640 1 -- 172.21.15.59:0/4114360986 learned_addr learned my addr 172.21.15.59:0/4114360986 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:46.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccc9ba640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cc80cd9b0 0x7f0cc806c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd9bc640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cc81444c0 0x7f0cc80ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd1bb640 1 -- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cc81444c0 msgr2=0x7f0cc80ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd1bb640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cc81444c0 0x7f0cc80ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd1bb640 1 -- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cc80cd9b0 msgr2=0x7f0cc806c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd1bb640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cc80cd9b0 0x7f0cc806c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd1bb640 1 -- 172.21.15.59:0/4114360986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc8140f80 con 0x7f0cc806d830 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccd1bb640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc806d830 0x7f0cc806c1e0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009d90 tx=0x7f0cb8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0cc5ffb640 1 -- 172.21.15.59:0/4114360986 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8033050 con 0x7f0cc806d830 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0cc5ffb640 1 -- 172.21.15.59:0/4114360986 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cb80045b0 con 0x7f0cc806d830 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cc8141260 con 0x7f0cc806d830 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.303+0000 7f0cc5ffb640 1 -- 172.21.15.59:0/4114360986 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8038450 con 0x7f0cc806d830 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.304+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cc8141820 con 0x7f0cc806d830 2024-06-29T15:38:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.304+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cc815bf20 con 0x7f0cc806d830 2024-06-29T15:38:46.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.305+0000 7f0cc5ffb640 1 -- 172.21.15.59:0/4114360986 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f0cb8036070 con 0x7f0cc806d830 2024-06-29T15:38:46.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.306+0000 7f0cc5ffb640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f0cb40763f0 0x7f0cb4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.306+0000 7f0cc5ffb640 1 -- 172.21.15.59:0/4114360986 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f0cb80b9020 con 0x7f0cc806d830 2024-06-29T15:38:46.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.306+0000 7f0ccc9ba640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f0cb40763f0 0x7f0cb4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.307+0000 7f0ccc9ba640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f0cb40763f0 0x7f0cb4078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8143a10 tx=0x7f0cbc00a250 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:46.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.310+0000 7f0cc5ffb640 1 -- 172.21.15.59:0/4114360986 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0cc806dc40 con 0x7f0cc806d830 2024-06-29T15:38:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.526+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 --> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f0cc8140cc0 con 0x7f0cb40763f0 2024-06-29T15:38:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.528+0000 7f0cc5ffb640 1 -- 172.21.15.59:0/4114360986 <== mgr.8039 v2:172.21.15.153:6832/2627795750 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f0cc8140cc0 con 0x7f0cb40763f0 2024-06-29T15:38:46.529 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:38:46.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:38:46.529 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T15:38:46.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:38:46.529 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:38:46.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f0cb40763f0 msgr2=0x7f0cb4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f0cb40763f0 0x7f0cb4078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8143a10 tx=0x7f0cbc00a250 comp rx=0 tx=0).stop 2024-06-29T15:38:46.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc806d830 msgr2=0x7f0cc806c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc806d830 0x7f0cc806c1e0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009d90 tx=0x7f0cb8003f10 comp rx=0 tx=0).stop 2024-06-29T15:38:46.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 shutdown_connections 2024-06-29T15:38:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0cc81444c0 0x7f0cc80ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cc80cd9b0 0x7f0cc806c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f0cb40763f0 0x7f0cb4078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 --2- 172.21.15.59:0/4114360986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cc806d830 0x7f0cc806c1e0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 >> 172.21.15.59:0/4114360986 conn(0x7f0cc80b24c0 msgr2=0x7f0cc815c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 shutdown_connections 2024-06-29T15:38:46.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.530+0000 7f0ccf41f640 1 -- 172.21.15.59:0/4114360986 wait complete. 2024-06-29T15:38:46.542 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:38:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.661+0000 7fae04af7640 1 -- 172.21.15.59:0/2273190231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0015ceb0 msgr2=0x7fae000b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.661+0000 7fae04af7640 1 --2- 172.21.15.59:0/2273190231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0015ceb0 0x7fae000b56d0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fae000b5590 tx=0x7fade802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.661+0000 7fae04af7640 1 -- 172.21.15.59:0/2273190231 shutdown_connections 2024-06-29T15:38:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.661+0000 7fae04af7640 1 --2- 172.21.15.59:0/2273190231 >> v2:172.21.15.153:3300/0 conn(0x7fae0013e9b0 0x7fae0013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.661+0000 7fae04af7640 1 --2- 172.21.15.59:0/2273190231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0015ceb0 0x7fae000b56d0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.661+0000 7fae04af7640 1 -- 172.21.15.59:0/2273190231 >> 172.21.15.59:0/2273190231 conn(0x7fae000b6fb0 msgr2=0x7fae000b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.661+0000 7fae04af7640 1 -- 172.21.15.59:0/2273190231 shutdown_connections 2024-06-29T15:38:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.661+0000 7fae04af7640 1 -- 172.21.15.59:0/2273190231 wait complete. 2024-06-29T15:38:46.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.663+0000 7fae04af7640 1 Processor -- start 2024-06-29T15:38:46.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.663+0000 7fae04af7640 1 -- start start 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fae04af7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0013e9b0 0x7fae001584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fae04af7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae001414c0 0x7fae00159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fae04af7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae0015ceb0 0x7fae0015a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fae04af7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fae000b29d0 con 0x7fae0013e9b0 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fae04af7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae000b2850 con 0x7fae0015ceb0 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fae04af7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fae000b2b50 con 0x7fae001414c0 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0013e9b0 0x7fae001584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0013e9b0 0x7fae001584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56872/0 (socket says 172.21.15.59:56872) 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 -- 172.21.15.59:0/2897607831 learned_addr learned my addr 172.21.15.59:0/2897607831 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfed9d640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae0015ceb0 0x7fae0015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfdd9b640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae001414c0 0x7fae00159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 -- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae001414c0 msgr2=0x7fae00159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae001414c0 0x7fae00159af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 -- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae0015ceb0 msgr2=0x7fae0015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae0015ceb0 0x7fae0015a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 -- 172.21.15.59:0/2897607831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0013e530 con 0x7fae0013e9b0 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfdd9b640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae001414c0 0x7fae00159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.665+0000 7fadfe59c640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0013e9b0 0x7fae001584d0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fade8034040 tx=0x7fade80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.666+0000 7fadf6ffd640 1 -- 172.21.15.59:0/2897607831 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade803e030 con 0x7fae0013e9b0 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.666+0000 7fadf6ffd640 1 -- 172.21.15.59:0/2897607831 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fade80047a0 con 0x7fae0013e9b0 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.666+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae00157430 con 0x7fae0013e9b0 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.666+0000 7fadf6ffd640 1 -- 172.21.15.59:0/2897607831 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8035040 con 0x7fae0013e9b0 2024-06-29T15:38:46.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.666+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae001579f0 con 0x7fae0013e9b0 2024-06-29T15:38:46.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.667+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae000b3000 con 0x7fae0013e9b0 2024-06-29T15:38:46.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.668+0000 7fadf6ffd640 1 -- 172.21.15.59:0/2897607831 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fade8041030 con 0x7fae0013e9b0 2024-06-29T15:38:46.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.668+0000 7fadf6ffd640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fade00763f0 0x7fade0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.668+0000 7fadf6ffd640 1 -- 172.21.15.59:0/2897607831 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7fade80b8430 con 0x7fae0013e9b0 2024-06-29T15:38:46.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.668+0000 7fadfdd9b640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fade00763f0 0x7fade0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.669+0000 7fadfdd9b640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fade00763f0 0x7fade0078900 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae00159260 tx=0x7fadf0005cd0 comp rx=0 tx=0).ready entity=mgr.8039 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:46.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.673+0000 7fadf6ffd640 1 -- 172.21.15.59:0/2897607831 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fade8081940 con 0x7fae0013e9b0 2024-06-29T15:38:46.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:46.915+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fae000b7880 con 0x7fae0013e9b0 2024-06-29T15:38:47.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.189+0000 7fadf6ffd640 1 -- 172.21.15.59:0/2897607831 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v91) ==== 92+0+0 (secure 0 0 0) 0x7fade8081240 con 0x7fae0013e9b0 2024-06-29T15:38:47.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.191+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fade00763f0 msgr2=0x7fade0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.191+0000 7fae04af7640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fade00763f0 0x7fade0078900 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae00159260 tx=0x7fadf0005cd0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.191+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0013e9b0 msgr2=0x7fae001584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.191+0000 7fae04af7640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0013e9b0 0x7fae001584d0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fade8034040 tx=0x7fade80039e0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.192+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 shutdown_connections 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.192+0000 7fae04af7640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae0015ceb0 0x7fae0015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.192+0000 7fae04af7640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae001414c0 0x7fae00159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.192+0000 7fae04af7640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7fade00763f0 0x7fade0078900 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae00159260 tx=0x7fadf0005cd0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.192+0000 7fae04af7640 1 --2- 172.21.15.59:0/2897607831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae0013e9b0 0x7fae001584d0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.192+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 >> 172.21.15.59:0/2897607831 conn(0x7fae000b6fb0 msgr2=0x7fae001409b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.192+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 shutdown_connections 2024-06-29T15:38:47.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.192+0000 7fae04af7640 1 -- 172.21.15.59:0/2897607831 wait complete. 2024-06-29T15:38:47.206 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:38:47.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.330+0000 7f961941f640 1 -- 172.21.15.59:0/752650675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961406da80 msgr2=0x7f96140cd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.330+0000 7f961941f640 1 --2- 172.21.15.59:0/752650675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961406da80 0x7f96140cd990 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f960c005fd0 tx=0x7f960c02ab40 comp rx=0 tx=0).stop 2024-06-29T15:38:47.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.330+0000 7f961941f640 1 -- 172.21.15.59:0/752650675 shutdown_connections 2024-06-29T15:38:47.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.330+0000 7f961941f640 1 --2- 172.21.15.59:0/752650675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f961406da80 0x7f96140cd990 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.330+0000 7f961941f640 1 -- 172.21.15.59:0/752650675 >> 172.21.15.59:0/752650675 conn(0x7f96140b2ab0 msgr2=0x7f96140b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.330+0000 7f961941f640 1 -- 172.21.15.59:0/752650675 shutdown_connections 2024-06-29T15:38:47.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.330+0000 7f961941f640 1 -- 172.21.15.59:0/752650675 wait complete. 2024-06-29T15:38:47.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.332+0000 7f961941f640 1 Processor -- start 2024-06-29T15:38:47.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.332+0000 7f961941f640 1 -- start start 2024-06-29T15:38:47.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f961941f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f961406da80 0x7f961413de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f961941f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96141448e0 0x7f96141f8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f961941f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961415b2f0 0x7f96141f8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f961941f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9614166ed0 con 0x7f96141448e0 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f961941f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9614166d50 con 0x7f961415b2f0 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f961941f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9614167050 con 0x7f961406da80 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f96127fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96141448e0 0x7f96141f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f96127fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96141448e0 0x7f96141f8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56908/0 (socket says 172.21.15.59:56908) 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f96127fc640 1 -- 172.21.15.59:0/4004296494 learned_addr learned my addr 172.21.15.59:0/4004296494 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f96137fe640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961415b2f0 0x7f96141f8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.333+0000 7f9612ffd640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f961406da80 0x7f961413de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f96127fc640 1 -- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f961406da80 msgr2=0x7f961413de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f96127fc640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f961406da80 0x7f961413de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f96127fc640 1 -- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961415b2f0 msgr2=0x7f96141f8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f96127fc640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961415b2f0 0x7f96141f8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f96127fc640 1 -- 172.21.15.59:0/4004296494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9614159620 con 0x7f96141448e0 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f9612ffd640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f961406da80 0x7f961413de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f96127fc640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96141448e0 0x7f96141f8110 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00c370 tx=0x7f95fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f960b7fe640 1 -- 172.21.15.59:0/4004296494 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc017070 con 0x7f96141448e0 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f960b7fe640 1 -- 172.21.15.59:0/4004296494 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95fc0035f0 con 0x7f96141448e0 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9614159900 con 0x7f96141448e0 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f960b7fe640 1 -- 172.21.15.59:0/4004296494 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc003a40 con 0x7f96141448e0 2024-06-29T15:38:47.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.334+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96140b4400 con 0x7f96141448e0 2024-06-29T15:38:47.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.335+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96140cfbe0 con 0x7f96141448e0 2024-06-29T15:38:47.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.336+0000 7f960b7fe640 1 -- 172.21.15.59:0/4004296494 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 91) ==== 98320+0+0 (secure 0 0 0) 0x7f95fc01a070 con 0x7f96141448e0 2024-06-29T15:38:47.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.337+0000 7f960b7fe640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f95f4076470 0x7f95f4078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.337+0000 7f9612ffd640 1 -- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f95f4076470 msgr2=0x7f95f4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2627795750 2024-06-29T15:38:47.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.337+0000 7f960b7fe640 1 -- 172.21.15.59:0/4004296494 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f95fc0970c0 con 0x7f96141448e0 2024-06-29T15:38:47.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.337+0000 7f9612ffd640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f95f4076470 0x7f95f4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.340+0000 7f960b7fe640 1 -- 172.21.15.59:0/4004296494 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95fc021020 con 0x7f96141448e0 2024-06-29T15:38:47.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.533+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f9614073a80 con 0x7f96141448e0 2024-06-29T15:38:47.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.537+0000 7f960b7fe640 1 -- 172.21.15.59:0/4004296494 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v61)=0 v61) ==== 138+0+0 (secure 0 0 0) 0x7f95fc0601f0 con 0x7f96141448e0 2024-06-29T15:38:47.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f9612ffd640 1 -- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f95f4076470 msgr2=0x7f95f4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2627795750 2024-06-29T15:38:47.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f9612ffd640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f95f4076470 0x7f95f4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f95f4076470 msgr2=0x7f95f4078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f95f4076470 0x7f95f4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96141448e0 msgr2=0x7f96141f8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96141448e0 0x7f96141f8110 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00c370 tx=0x7f95fc00c830 comp rx=0 tx=0).stop 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 shutdown_connections 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961415b2f0 0x7f96141f8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f95f4076470 0x7f95f4078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96141448e0 0x7f96141f8110 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 --2- 172.21.15.59:0/4004296494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f961406da80 0x7f961413de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.539+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 >> 172.21.15.59:0/4004296494 conn(0x7f96140b2ab0 msgr2=0x7f96140b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.540+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 shutdown_connections 2024-06-29T15:38:47.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.540+0000 7f961941f640 1 -- 172.21.15.59:0/4004296494 wait complete. 2024-06-29T15:38:47.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:38:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.673+0000 7f2746903640 1 -- 172.21.15.59:0/711952545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 msgr2=0x7f274006e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.673+0000 7f2746903640 1 --2- 172.21.15.59:0/711952545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 0x7f274006e4d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f2740144290 tx=0x7f273402a560 comp rx=0 tx=0).stop 2024-06-29T15:38:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.673+0000 7f2746903640 1 -- 172.21.15.59:0/711952545 shutdown_connections 2024-06-29T15:38:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.673+0000 7f2746903640 1 --2- 172.21.15.59:0/711952545 >> v2:172.21.15.153:3300/0 conn(0x7f274006ea10 0x7f27400cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.673+0000 7f2746903640 1 --2- 172.21.15.59:0/711952545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 0x7f274006e4d0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.673+0000 7f2746903640 1 -- 172.21.15.59:0/711952545 >> 172.21.15.59:0/711952545 conn(0x7f27400b2ab0 msgr2=0x7f27400b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.674+0000 7f2746903640 1 -- 172.21.15.59:0/711952545 shutdown_connections 2024-06-29T15:38:47.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.674+0000 7f2746903640 1 -- 172.21.15.59:0/711952545 wait complete. 2024-06-29T15:38:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.676+0000 7f2746903640 1 Processor -- start 2024-06-29T15:38:47.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.677+0000 7f2746903640 1 -- start start 2024-06-29T15:38:47.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2746903640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274006ea10 0x7f274013dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2746903640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27400d0130 0x7f274015bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2746903640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 0x7f274013d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2746903640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f27400cf650 con 0x7f2740143840 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2746903640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27400cf4d0 con 0x7f27400d0130 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2746903640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f27400cf7d0 con 0x7f274006ea10 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2744ea0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 0x7f274013d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2744ea0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 0x7f274013d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56922/0 (socket says 172.21.15.59:56922) 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f2744ea0640 1 -- 172.21.15.59:0/3526681420 learned_addr learned my addr 172.21.15.59:0/3526681420 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f273f7fe640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27400d0130 0x7f274015bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.679+0000 7f273ffff640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274006ea10 0x7f274013dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f2744ea0640 1 -- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274006ea10 msgr2=0x7f274013dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f2744ea0640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274006ea10 0x7f274013dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f2744ea0640 1 -- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27400d0130 msgr2=0x7f274015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f2744ea0640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27400d0130 0x7f274015bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f2744ea0640 1 -- 172.21.15.59:0/3526681420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274013e5f0 con 0x7f2740143840 2024-06-29T15:38:47.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f2744ea0640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 0x7f274013d370 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f273800ea70 tx=0x7f273800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f273cff9640 1 -- 172.21.15.59:0/3526681420 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273800ce40 con 0x7f2740143840 2024-06-29T15:38:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f273cff9640 1 -- 172.21.15.59:0/3526681420 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2738004c10 con 0x7f2740143840 2024-06-29T15:38:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27401552d0 con 0x7f2740143840 2024-06-29T15:38:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f273cff9640 1 -- 172.21.15.59:0/3526681420 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738005020 con 0x7f2740143840 2024-06-29T15:38:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.680+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2740155890 con 0x7f2740143840 2024-06-29T15:38:47.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.681+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27400cfbf0 con 0x7f2740143840 2024-06-29T15:38:47.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.682+0000 7f273cff9640 1 -- 172.21.15.59:0/3526681420 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 91) ==== 98320+0+0 (secure 0 0 0) 0x7f27380051c0 con 0x7f2740143840 2024-06-29T15:38:47.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.682+0000 7f273cff9640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f272c076470 0x7f272c078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.683+0000 7f273ffff640 1 -- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f272c076470 msgr2=0x7f272c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2627795750 2024-06-29T15:38:47.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.683+0000 7f273cff9640 1 -- 172.21.15.59:0/3526681420 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f2738098e40 con 0x7f2740143840 2024-06-29T15:38:47.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.683+0000 7f273ffff640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f272c076470 0x7f272c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:47.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.687+0000 7f273cff9640 1 -- 172.21.15.59:0/3526681420 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2738026020 con 0x7f2740143840 2024-06-29T15:38:47.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.878+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f274013e310 con 0x7f2740143840 2024-06-29T15:38:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.882+0000 7f273cff9640 1 -- 172.21.15.59:0/3526681420 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v62)=0 v62) ==== 133+0+0 (secure 0 0 0) 0x7f2738061dd0 con 0x7f2740143840 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f273ffff640 1 -- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f272c076470 msgr2=0x7f272c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2627795750 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f273ffff640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f272c076470 0x7f272c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f272c076470 msgr2=0x7f272c078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f272c076470 0x7f272c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 msgr2=0x7f274013d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 0x7f274013d370 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f273800ea70 tx=0x7f273800ef30 comp rx=0 tx=0).stop 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 shutdown_connections 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f272c076470 0x7f272c078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2740143840 0x7f274013d370 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27400d0130 0x7f274015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 --2- 172.21.15.59:0/3526681420 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f274006ea10 0x7f274013dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 >> 172.21.15.59:0/3526681420 conn(0x7f27400b2ab0 msgr2=0x7f27400b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 shutdown_connections 2024-06-29T15:38:47.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:47.884+0000 7f2746903640 1 -- 172.21.15.59:0/3526681420 wait complete. 2024-06-29T15:38:47.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:47.897 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T15:38:47.920 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-06-29T15:38:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.080+0000 7f7e57b68640 1 -- 172.21.15.59:0/1191285853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 msgr2=0x7f7e5015f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.080+0000 7f7e57b68640 1 --2- 172.21.15.59:0/1191285853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 0x7f7e5015f240 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009830 tx=0x7f7e4c02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.080+0000 7f7e57b68640 1 -- 172.21.15.59:0/1191285853 shutdown_connections 2024-06-29T15:38:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.080+0000 7f7e57b68640 1 --2- 172.21.15.59:0/1191285853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 0x7f7e5015f240 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.080+0000 7f7e57b68640 1 -- 172.21.15.59:0/1191285853 >> 172.21.15.59:0/1191285853 conn(0x7f7e500b24c0 msgr2=0x7f7e500cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.081+0000 7f7e57b68640 1 -- 172.21.15.59:0/1191285853 shutdown_connections 2024-06-29T15:38:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.081+0000 7f7e57b68640 1 -- 172.21.15.59:0/1191285853 wait complete. 2024-06-29T15:38:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.082+0000 7f7e57b68640 1 Processor -- start 2024-06-29T15:38:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.082+0000 7f7e57b68640 1 -- start start 2024-06-29T15:38:48.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e57b68640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 0x7f7e500aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:48.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e57b68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501444c0 0x7f7e5015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:48.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e57b68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e501f5f00 0x7f7e501f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e57b68640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7e50073d10 con 0x7f7e500cd9b0 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e57b68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e50073b90 con 0x7f7e501444c0 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e57b68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7e50073e90 con 0x7f7e501f5f00 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 0x7f7e500aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 0x7f7e500aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56940/0 (socket says 172.21.15.59:56940) 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 -- 172.21.15.59:0/534332714 learned_addr learned my addr 172.21.15.59:0/534332714 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55103640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501444c0 0x7f7e5015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e56105640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e501f5f00 0x7f7e501f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e501f5f00 msgr2=0x7f7e501f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e501f5f00 0x7f7e501f3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501444c0 msgr2=0x7f7e5015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e501444c0 0x7f7e5015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 -- 172.21.15.59:0/534332714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e501f69a0 con 0x7f7e500cd9b0 2024-06-29T15:38:48.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.084+0000 7f7e55904640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 0x7f7e500aef80 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009950 tx=0x7f7e4c033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:48.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.085+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c031d60 con 0x7f7e500cd9b0 2024-06-29T15:38:48.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.085+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e4c005020 con 0x7f7e500cd9b0 2024-06-29T15:38:48.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.085+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c005470 con 0x7f7e500cd9b0 2024-06-29T15:38:48.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.085+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e501f6c80 con 0x7f7e500cd9b0 2024-06-29T15:38:48.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.085+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e500aea60 con 0x7f7e500cd9b0 2024-06-29T15:38:48.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.086+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e500b33d0 con 0x7f7e500cd9b0 2024-06-29T15:38:48.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.087+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 91) ==== 98320+0+0 (secure 0 0 0) 0x7f7e4c0047a0 con 0x7f7e500cd9b0 2024-06-29T15:38:48.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.088+0000 7f7e427fc640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 0x7f7e34078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:48.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.088+0000 7f7e55103640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 msgr2=0x7f7e34078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2627795750 2024-06-29T15:38:48.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.088+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 6129+0+0 (secure 0 0 0) 0x7f7e4c0b8d80 con 0x7f7e500cd9b0 2024-06-29T15:38:48.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.088+0000 7f7e55103640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 0x7f7e34078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:48.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.093+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7e4c03f030 con 0x7f7e500cd9b0 2024-06-29T15:38:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.289+0000 7f7e55103640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 msgr2=0x7f7e34078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2627795750 2024-06-29T15:38:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.289+0000 7f7e55103640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 0x7f7e34078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:48.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.308+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 --> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7f7e500b30f0 con 0x7f7e34076470 2024-06-29T15:38:48.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.690+0000 7f7e55103640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 msgr2=0x7f7e34078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2627795750 2024-06-29T15:38:48.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:48.690+0000 7f7e55103640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 0x7f7e34078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:38:49.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:49.491+0000 7f7e55103640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 msgr2=0x7f7e34078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2627795750 2024-06-29T15:38:49.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:49.491+0000 7f7e55103640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 0x7f7e34078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:38:50.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:50.833+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 92) ==== 97929+0+0 (secure 0 0 0) 0x7f7e4c0b70f0 con 0x7f7e500cd9b0 2024-06-29T15:38:50.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:50.833+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 msgr2=0x7f7e34078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:50.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:50.833+0000 7f7e427fc640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/2627795750,v1:172.21.15.153:6833/2627795750] conn(0x7f7e34076470 0x7f7e34078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:50.990 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd781377640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd781377640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd781377640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd781377640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd781377640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd77e371640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd77e371640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd77e371640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd77e371640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:50.991 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:50.989+0000 7fd77e371640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.837+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 93) ==== 98056+0+0 (secure 0 0 0) 0x7f7e4c07ded0 con 0x7f7e500cd9b0 2024-06-29T15:38:51.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.838+0000 7f7e427fc640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f7e3403ad60 0x7f7e34080a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:51.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.838+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 --> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7f7e4c0a3350 con 0x7f7e3403ad60 2024-06-29T15:38:51.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.843+0000 7f7e55103640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f7e3403ad60 0x7f7e34080a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:51.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.845+0000 7f7e55103640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f7e3403ad60 0x7f7e34080a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e5013d820 tx=0x7f7e44007e30 comp rx=0 tx=0).ready entity=mgr.9415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:51.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.917+0000 7f7e427fc640 1 -- 172.21.15.59:0/534332714 <== mgr.9415 v2:172.21.15.153:6832/770286644 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7e4c0a3350 con 0x7f7e3403ad60 2024-06-29T15:38:51.918 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1 2024-06-29T15:38:51.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f7e3403ad60 msgr2=0x7f7e34080a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:51.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e57b68640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f7e3403ad60 0x7f7e34080a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e5013d820 tx=0x7f7e44007e30 comp rx=0 tx=0).stop 2024-06-29T15:38:51.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 msgr2=0x7f7e500aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:51.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e57b68640 1 --2- 172.21.15.59:0/534332714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e500cd9b0 0x7f7e500aef80 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009950 tx=0x7f7e4c033ce0 comp rx=0 tx=0).stop 2024-06-29T15:38:51.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e56105640 1 -- 172.21.15.59:0/534332714 reap_dead start 2024-06-29T15:38:51.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 shutdown_connections 2024-06-29T15:38:51.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 >> 172.21.15.59:0/534332714 conn(0x7f7e500b24c0 msgr2=0x7f7e50068520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:51.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 shutdown_connections 2024-06-29T15:38:51.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:51.919+0000 7f7e57b68640 1 -- 172.21.15.59:0/534332714 wait complete. 2024-06-29T15:38:51.933 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T15:38:51.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T15:38:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.055+0000 7f4b03488640 1 -- 172.21.15.59:0/1620781322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc0b40a0 msgr2=0x7f4afc15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.055+0000 7f4b03488640 1 --2- 172.21.15.59:0/1620781322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc0b40a0 0x7f4afc15e8d0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f4aec005fd0 tx=0x7f4aec02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.055+0000 7f4b03488640 1 -- 172.21.15.59:0/1620781322 shutdown_connections 2024-06-29T15:38:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.055+0000 7f4b03488640 1 --2- 172.21.15.59:0/1620781322 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4afc15ee10 0x7f4afc15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.055+0000 7f4b03488640 1 --2- 172.21.15.59:0/1620781322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc0b40a0 0x7f4afc15e8d0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.055+0000 7f4b03488640 1 -- 172.21.15.59:0/1620781322 >> 172.21.15.59:0/1620781322 conn(0x7f4afc0d3ee0 msgr2=0x7f4afc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.055+0000 7f4b03488640 1 -- 172.21.15.59:0/1620781322 shutdown_connections 2024-06-29T15:38:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.055+0000 7f4b03488640 1 -- 172.21.15.59:0/1620781322 wait complete. 2024-06-29T15:38:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.057+0000 7f4b03488640 1 Processor -- start 2024-06-29T15:38:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.057+0000 7f4b03488640 1 -- start start 2024-06-29T15:38:52.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b03488640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4afc0d23c0 0x7f4afc144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b03488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc15ee10 0x7f4afc1f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b03488640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4afc1467c0 0x7f4afc1559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b03488640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc160280 con 0x7f4afc15ee10 2024-06-29T15:38:52.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b03488640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc160100 con 0x7f4afc1467c0 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b03488640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4afc160400 con 0x7f4afc0d23c0 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc15ee10 0x7f4afc1f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc15ee10 0x7f4afc1f4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56962/0 (socket says 172.21.15.59:56962) 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 -- 172.21.15.59:0/1039102839 learned_addr learned my addr 172.21.15.59:0/1039102839 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b01224640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4afc0d23c0 0x7f4afc144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b01a25640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4afc1467c0 0x7f4afc1559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 -- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4afc0d23c0 msgr2=0x7f4afc144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4afc0d23c0 0x7f4afc144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 -- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4afc1467c0 msgr2=0x7f4afc1559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4afc1467c0 0x7f4afc1559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 -- 172.21.15.59:0/1039102839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc145720 con 0x7f4afc15ee10 2024-06-29T15:38:52.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b00a23640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc15ee10 0x7f4afc1f4740 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f4aec006010 tx=0x7f4aec0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4af1ffb640 1 -- 172.21.15.59:0/1039102839 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec03e030 con 0x7f4afc15ee10 2024-06-29T15:38:52.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4af1ffb640 1 -- 172.21.15.59:0/1039102839 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4aec003d90 con 0x7f4afc15ee10 2024-06-29T15:38:52.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4afc145a00 con 0x7f4afc15ee10 2024-06-29T15:38:52.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.059+0000 7f4af1ffb640 1 -- 172.21.15.59:0/1039102839 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec035040 con 0x7f4afc15ee10 2024-06-29T15:38:52.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.060+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4afc145fc0 con 0x7f4afc15ee10 2024-06-29T15:38:52.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.061+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4afc15e8d0 con 0x7f4afc15ee10 2024-06-29T15:38:52.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.062+0000 7f4af1ffb640 1 -- 172.21.15.59:0/1039102839 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 93) ==== 98056+0+0 (secure 0 0 0) 0x7f4aec041030 con 0x7f4afc15ee10 2024-06-29T15:38:52.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.062+0000 7f4af1ffb640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f4ae0076170 0x7f4ae0078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.062+0000 7f4af1ffb640 1 -- 172.21.15.59:0/1039102839 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7f4aec0b7ed0 con 0x7f4afc15ee10 2024-06-29T15:38:52.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.062+0000 7f4b01224640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f4ae0076170 0x7f4ae0078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.063+0000 7f4b01224640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f4ae0076170 0x7f4ae0078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0d2e40 tx=0x7f4af8009040 comp rx=0 tx=0).ready entity=mgr.9415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.067+0000 7f4af1ffb640 1 -- 172.21.15.59:0/1039102839 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4aec081190 con 0x7f4afc15ee10 2024-06-29T15:38:52.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.283+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 --> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f4afc0d2950 con 0x7f4ae0076170 2024-06-29T15:38:52.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.362+0000 7f4af1ffb640 1 -- 172.21.15.59:0/1039102839 <== mgr.9415 v2:172.21.15.153:6832/770286644 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f4afc0d2950 con 0x7f4ae0076170 2024-06-29T15:38:52.364 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path snap_test_dir1 2024-06-29T15:38:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f4ae0076170 msgr2=0x7f4ae0078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f4ae0076170 0x7f4ae0078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0d2e40 tx=0x7f4af8009040 comp rx=0 tx=0).stop 2024-06-29T15:38:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc15ee10 msgr2=0x7f4afc1f4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc15ee10 0x7f4afc1f4740 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f4aec006010 tx=0x7f4aec0038a0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 shutdown_connections 2024-06-29T15:38:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f4ae0076170 0x7f4ae0078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4afc1467c0 0x7f4afc1559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4afc15ee10 0x7f4afc1f4740 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 --2- 172.21.15.59:0/1039102839 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4afc0d23c0 0x7f4afc144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 >> 172.21.15.59:0/1039102839 conn(0x7f4afc0d3ee0 msgr2=0x7f4afc06cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 shutdown_connections 2024-06-29T15:38:52.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.365+0000 7f4b03488640 1 -- 172.21.15.59:0/1039102839 wait complete. 2024-06-29T15:38:52.377 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T15:38:52.377 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-06-29T15:38:52.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.496+0000 7f61cfb9e640 1 -- 172.21.15.59:0/3976805535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61c80d23c0 msgr2=0x7f61c80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.496+0000 7f61cfb9e640 1 --2- 172.21.15.59:0/3976805535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61c80d23c0 0x7f61c80b3990 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f61c808c4f0 tx=0x7f61b802aac0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.496+0000 7f61cfb9e640 1 -- 172.21.15.59:0/3976805535 shutdown_connections 2024-06-29T15:38:52.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.496+0000 7f61cfb9e640 1 --2- 172.21.15.59:0/3976805535 >> v2:172.21.15.153:3300/0 conn(0x7f61c80b3ed0 0x7f61c815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.496+0000 7f61cfb9e640 1 --2- 172.21.15.59:0/3976805535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61c80d23c0 0x7f61c80b3990 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.496+0000 7f61cfb9e640 1 -- 172.21.15.59:0/3976805535 >> 172.21.15.59:0/3976805535 conn(0x7f61c80d3ee0 msgr2=0x7f61c80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:52.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.496+0000 7f61cfb9e640 1 -- 172.21.15.59:0/3976805535 shutdown_connections 2024-06-29T15:38:52.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.496+0000 7f61cfb9e640 1 -- 172.21.15.59:0/3976805535 wait complete. 2024-06-29T15:38:52.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.498+0000 7f61cfb9e640 1 Processor -- start 2024-06-29T15:38:52.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.498+0000 7f61cfb9e640 1 -- start start 2024-06-29T15:38:52.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cfb9e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61c80b3ed0 0x7f61c815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cfb9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61c80d23c0 0x7f61c8148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cfb9e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61c815ee10 0x7f61c8160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cfb9e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f61c8160240 con 0x7f61c80b3ed0 2024-06-29T15:38:52.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cfb9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f61c81600c0 con 0x7f61c80d23c0 2024-06-29T15:38:52.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cfb9e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f61c81603c0 con 0x7f61c815ee10 2024-06-29T15:38:52.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61c80b3ed0 0x7f61c815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61c80b3ed0 0x7f61c815b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56984/0 (socket says 172.21.15.59:56984) 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 -- 172.21.15.59:0/2725173817 learned_addr learned my addr 172.21.15.59:0/2725173817 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd139640 1 --2- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61c80d23c0 0x7f61c8148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61ce13b640 1 --2- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61c815ee10 0x7f61c8160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 -- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61c815ee10 msgr2=0x7f61c8160900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 --2- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61c815ee10 0x7f61c8160900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 -- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61c80d23c0 msgr2=0x7f61c8148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 --2- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61c80d23c0 0x7f61c8148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 -- 172.21.15.59:0/2725173817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c81494f0 con 0x7f61c80b3ed0 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cd93a640 1 --2- 172.21.15.59:0/2725173817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61c80b3ed0 0x7f61c815b0a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009d60 tx=0x7f61b8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61c67fc640 1 -- 172.21.15.59:0/2725173817 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b8004250 con 0x7f61c80b3ed0 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61c67fc640 1 -- 172.21.15.59:0/2725173817 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61b8004410 con 0x7f61c80b3ed0 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61cfb9e640 1 -- 172.21.15.59:0/2725173817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c8159fa0 con 0x7f61c80b3ed0 2024-06-29T15:38:52.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.500+0000 7f61c67fc640 1 -- 172.21.15.59:0/2725173817 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b8004860 con 0x7f61c80b3ed0 2024-06-29T15:38:52.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.501+0000 7f61cfb9e640 1 -- 172.21.15.59:0/2725173817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c815cde0 con 0x7f61c80b3ed0 2024-06-29T15:38:52.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.502+0000 7f61cfb9e640 1 -- 172.21.15.59:0/2725173817 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c815e8d0 con 0x7f61c80b3ed0 2024-06-29T15:38:52.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.502+0000 7f61c67fc640 1 -- 172.21.15.59:0/2725173817 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 93) ==== 98056+0+0 (secure 0 0 0) 0x7f61b8049430 con 0x7f61c80b3ed0 2024-06-29T15:38:52.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.503+0000 7f61c67fc640 1 --2- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f61b4076170 0x7f61b4078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.503+0000 7f61cd139640 1 --2- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f61b4076170 0x7f61b4078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.503+0000 7f61c67fc640 1 -- 172.21.15.59:0/2725173817 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7f61b80b8440 con 0x7f61c80b3ed0 2024-06-29T15:38:52.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.504+0000 7f61cd139640 1 --2- 172.21.15.59:0/2725173817 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f61b4076170 0x7f61b4078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f61c81f8710 tx=0x7f61bc005f90 comp rx=0 tx=0).ready entity=mgr.9415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.508+0000 7f61c67fc640 1 -- 172.21.15.59:0/2725173817 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f61c81494f0 con 0x7f61c80b3ed0 2024-06-29T15:38:52.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.720+0000 7f61cfb9e640 1 -- 172.21.15.59:0/2725173817 --> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f61c80d2950 con 0x7f61b4076170 2024-06-29T15:38:52.727 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:52.727+0000 7fd788385640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-06-29T15:38:52.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.727+0000 7f61c67fc640 1 -- 172.21.15.59:0/2725173817 <== mgr.9415 v2:172.21.15.153:6832/770286644 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f61c80d2950 con 0x7f61b4076170 2024-06-29T15:38:52.728 INFO:teuthology.orchestra.run.smithi059.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-06-29T15:38:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T15:38:52.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:52.734 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T15:38:52.751 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:38:52.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.914+0000 7fcbce090640 1 -- 172.21.15.59:0/2477890957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 msgr2=0x7fcbc815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.914+0000 7fcbce090640 1 --2- 172.21.15.59:0/2477890957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 0x7fcbc815f9b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fcbc808c4f0 tx=0x7fcbc002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.914+0000 7fcbce090640 1 -- 172.21.15.59:0/2477890957 shutdown_connections 2024-06-29T15:38:52.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.914+0000 7fcbce090640 1 --2- 172.21.15.59:0/2477890957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 0x7fcbc815f9b0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.914+0000 7fcbce090640 1 --2- 172.21.15.59:0/2477890957 >> v2:172.21.15.153:3300/0 conn(0x7fcbc80b40a0 0x7fcbc815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.914+0000 7fcbce090640 1 -- 172.21.15.59:0/2477890957 >> 172.21.15.59:0/2477890957 conn(0x7fcbc80d3ee0 msgr2=0x7fcbc80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:52.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.914+0000 7fcbce090640 1 -- 172.21.15.59:0/2477890957 shutdown_connections 2024-06-29T15:38:52.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.914+0000 7fcbce090640 1 -- 172.21.15.59:0/2477890957 wait complete. 2024-06-29T15:38:52.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.915+0000 7fcbce090640 1 Processor -- start 2024-06-29T15:38:52.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.916+0000 7fcbce090640 1 -- start start 2024-06-29T15:38:52.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbce090640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc80d23c0 0x7fcbc81f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbce090640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 0x7fcbc815e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbce090640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc815c080 0x7fcbc815c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbce090640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc8160280 con 0x7fcbc815ee10 2024-06-29T15:38:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbce090640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc8160100 con 0x7fcbc815c080 2024-06-29T15:38:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbce090640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc8160400 con 0x7fcbc80d23c0 2024-06-29T15:38:52.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 0x7fcbc815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 0x7fcbc815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59350/0 (socket says 172.21.15.59:59350) 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc6ffd640 1 -- 172.21.15.59:0/3384262358 learned_addr learned my addr 172.21.15.59:0/3384262358 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc7fff640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc815c080 0x7fcbc815c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc77fe640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc80d23c0 0x7fcbc81f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc6ffd640 1 -- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc80d23c0 msgr2=0x7fcbc81f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc6ffd640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc80d23c0 0x7fcbc81f4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc6ffd640 1 -- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc815c080 msgr2=0x7fcbc815c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc6ffd640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc815c080 0x7fcbc815c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.917+0000 7fcbc6ffd640 1 -- 172.21.15.59:0/3384262358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc815e550 con 0x7fcbc815ee10 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.918+0000 7fcbc6ffd640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 0x7fcbc815e010 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00d720 tx=0x7fcbbc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.918+0000 7fcbaffff640 1 -- 172.21.15.59:0/3384262358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc017070 con 0x7fcbc815ee10 2024-06-29T15:38:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.918+0000 7fcbaffff640 1 -- 172.21.15.59:0/3384262358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbbc004310 con 0x7fcbc815ee10 2024-06-29T15:38:52.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.918+0000 7fcbaffff640 1 -- 172.21.15.59:0/3384262358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc009c30 con 0x7fcbc815ee10 2024-06-29T15:38:52.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.918+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc8159db0 con 0x7fcbc815ee10 2024-06-29T15:38:52.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.918+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc815a370 con 0x7fcbc815ee10 2024-06-29T15:38:52.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.919+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc815e8d0 con 0x7fcbc815ee10 2024-06-29T15:38:52.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.920+0000 7fcbaffff640 1 -- 172.21.15.59:0/3384262358 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 94) ==== 98321+0+0 (secure 0 0 0) 0x7fcbbc005020 con 0x7fcbc815ee10 2024-06-29T15:38:52.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.920+0000 7fcbaffff640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcbb4076440 0x7fcbb4078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.920+0000 7fcbaffff640 1 -- 172.21.15.59:0/3384262358 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7fcbbc098010 con 0x7fcbc815ee10 2024-06-29T15:38:52.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.920+0000 7fcbc77fe640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcbb4076440 0x7fcbb4078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.921+0000 7fcbc77fe640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcbb4076440 0x7fcbb4078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80d2e40 tx=0x7fcbb8007e90 comp rx=0 tx=0).ready entity=mgr.9415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:52.924+0000 7fcbaffff640 1 -- 172.21.15.59:0/3384262358 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcbbc020020 con 0x7fcbc815ee10 2024-06-29T15:38:53.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.168+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7fcbc80d3830 con 0x7fcbc815ee10 2024-06-29T15:38:53.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.894+0000 7fcbaffff640 1 -- 172.21.15.59:0/3384262358 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) ==== 93+0+0 (secure 0 0 0) 0x7fcbbc0693c0 con 0x7fcbc815ee10 2024-06-29T15:38:53.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.896+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcbb4076440 msgr2=0x7fcbb4078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:53.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.896+0000 7fcbce090640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcbb4076440 0x7fcbb4078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80d2e40 tx=0x7fcbb8007e90 comp rx=0 tx=0).stop 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.896+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 msgr2=0x7fcbc815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.896+0000 7fcbce090640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 0x7fcbc815e010 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00d720 tx=0x7fcbbc00dbe0 comp rx=0 tx=0).stop 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.897+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 shutdown_connections 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.897+0000 7fcbce090640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcbb4076440 0x7fcbb4078950 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80d2e40 tx=0x7fcbb8007e90 comp rx=0 tx=0).stop 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.897+0000 7fcbce090640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbc815c080 0x7fcbc815c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.897+0000 7fcbce090640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbc815ee10 0x7fcbc815e010 secure :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00d720 tx=0x7fcbbc00dbe0 comp rx=0 tx=0).stop 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.897+0000 7fcbce090640 1 --2- 172.21.15.59:0/3384262358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbc80d23c0 0x7fcbc81f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.897+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 >> 172.21.15.59:0/3384262358 conn(0x7fcbc80d3ee0 msgr2=0x7fcbc80b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.897+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 shutdown_connections 2024-06-29T15:38:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:53.897+0000 7fcbce090640 1 -- 172.21.15.59:0/3384262358 wait complete. 2024-06-29T15:38:53.918 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:38:53.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:53.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:54.006 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:38:54.006 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:54.007 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:54.007 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:54.023 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:54.023 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:54.080 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:54.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 -- 172.21.15.59:0/2752192748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880b2fa0 msgr2=0x7fe4880d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/2752192748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880b2fa0 0x7fe4880d1520 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fe4880cf310 tx=0x7fe47c02a560 comp rx=0 tx=0).stop 2024-06-29T15:38:54.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 -- 172.21.15.59:0/2752192748 shutdown_connections 2024-06-29T15:38:54.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/2752192748 >> v2:172.21.15.153:3300/0 conn(0x7fe4880d1a60 0x7fe4880cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/2752192748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880b2fa0 0x7fe4880d1520 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/2752192748 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4880b2270 0x7fe48815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 -- 172.21.15.59:0/2752192748 >> 172.21.15.59:0/2752192748 conn(0x7fe488073e10 msgr2=0x7fe488074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 -- 172.21.15.59:0/2752192748 shutdown_connections 2024-06-29T15:38:54.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.238+0000 7fe48cb5b640 1 -- 172.21.15.59:0/2752192748 wait complete. 2024-06-29T15:38:54.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.240+0000 7fe48cb5b640 1 Processor -- start 2024-06-29T15:38:54.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.240+0000 7fe48cb5b640 1 -- start start 2024-06-29T15:38:54.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.241+0000 7fe48cb5b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4880b2270 0x7fe488143c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.241+0000 7fe48cb5b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4880b2fa0 0x7fe48815f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.241+0000 7fe48cb5b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880d1a60 0x7fe488158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.241+0000 7fe48cb5b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe4880cfe40 con 0x7fe4880d1a60 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.241+0000 7fe48cb5b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4880cfcc0 con 0x7fe4880b2fa0 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.241+0000 7fe48cb5b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe4880cffc0 con 0x7fe4880b2270 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.241+0000 7fe486d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880d1a60 0x7fe488158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe486d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880d1a60 0x7fe488158170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59380/0 (socket says 172.21.15.59:59380) 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe486d9d640 1 -- 172.21.15.59:0/1254481025 learned_addr learned my addr 172.21.15.59:0/1254481025 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe48659c640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4880b2270 0x7fe488143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe485d9b640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4880b2fa0 0x7fe48815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe486d9d640 1 -- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4880b2270 msgr2=0x7fe488143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe486d9d640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4880b2270 0x7fe488143c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe486d9d640 1 -- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4880b2fa0 msgr2=0x7fe48815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe486d9d640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4880b2fa0 0x7fe48815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe486d9d640 1 -- 172.21.15.59:0/1254481025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe488151ca0 con 0x7fe4880d1a60 2024-06-29T15:38:54.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe486d9d640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880d1a60 0x7fe488158170 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fe48000ea80 tx=0x7fe48000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:54.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe476ffd640 1 -- 172.21.15.59:0/1254481025 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480004b20 con 0x7fe4880d1a60 2024-06-29T15:38:54.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe48815e2f0 con 0x7fe4880d1a60 2024-06-29T15:38:54.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe476ffd640 1 -- 172.21.15.59:0/1254481025 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe480004ce0 con 0x7fe4880d1a60 2024-06-29T15:38:54.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe476ffd640 1 -- 172.21.15.59:0/1254481025 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480005280 con 0x7fe4880d1a60 2024-06-29T15:38:54.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.242+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4881447e0 con 0x7fe4880d1a60 2024-06-29T15:38:54.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.243+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe48815e060 con 0x7fe4880d1a60 2024-06-29T15:38:54.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.244+0000 7fe476ffd640 1 -- 172.21.15.59:0/1254481025 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fe480005420 con 0x7fe4880d1a60 2024-06-29T15:38:54.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.245+0000 7fe476ffd640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe4680763f0 0x7fe468078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.245+0000 7fe48659c640 1 -- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe4680763f0 msgr2=0x7fe468078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:54.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.245+0000 7fe476ffd640 1 -- 172.21.15.59:0/1254481025 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7fe4800987c0 con 0x7fe4880d1a60 2024-06-29T15:38:54.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.245+0000 7fe48659c640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe4680763f0 0x7fe468078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:54.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.254+0000 7fe476ffd640 1 -- 172.21.15.59:0/1254481025 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe488151ca0 con 0x7fe4880d1a60 2024-06-29T15:38:54.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.446+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4880d0510 con 0x7fe4880d1a60 2024-06-29T15:38:54.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.447+0000 7fe48659c640 1 -- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe4680763f0 msgr2=0x7fe468078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:54.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.447+0000 7fe48659c640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe4680763f0 0x7fe468078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:54.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.447+0000 7fe476ffd640 1 -- 172.21.15.59:0/1254481025 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) ==== 74+0+17456 (secure 0 0 0) 0x7fe4800614a0 con 0x7fe4880d1a60 2024-06-29T15:38:54.449 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:54.449 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":177,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:50.826574+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:36.882222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.468804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2597046756":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/623872135":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/2213101666":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4098785844":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4234893873":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6833/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6832/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4064948250":"2024-06-30T15:38:50.826532+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:54.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe4680763f0 msgr2=0x7fe468078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:54.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe4680763f0 0x7fe468078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880d1a60 msgr2=0x7fe488158170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880d1a60 0x7fe488158170 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fe48000ea80 tx=0x7fe48000ef40 comp rx=0 tx=0).stop 2024-06-29T15:38:54.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 shutdown_connections 2024-06-29T15:38:54.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe4680763f0 0x7fe468078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4880d1a60 0x7fe488158170 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4880b2fa0 0x7fe48815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 --2- 172.21.15.59:0/1254481025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4880b2270 0x7fe488143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 >> 172.21.15.59:0/1254481025 conn(0x7fe488073e10 msgr2=0x7fe488155590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 shutdown_connections 2024-06-29T15:38:54.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.450+0000 7fe48cb5b640 1 -- 172.21.15.59:0/1254481025 wait complete. 2024-06-29T15:38:54.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:38:36.882222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:38:37.468804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '176', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:54.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:54.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.584+0000 7f01af7ac640 1 -- 172.21.15.59:0/710755953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 msgr2=0x7f01a815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.584+0000 7f01af7ac640 1 --2- 172.21.15.59:0/710755953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 0x7f01a815bc20 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f019c005fd0 tx=0x7f019c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.584+0000 7f01af7ac640 1 -- 172.21.15.59:0/710755953 shutdown_connections 2024-06-29T15:38:54.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.584+0000 7f01af7ac640 1 --2- 172.21.15.59:0/710755953 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a80d1c30 0x7f01a80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.584+0000 7f01af7ac640 1 --2- 172.21.15.59:0/710755953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 0x7f01a815bc20 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.584+0000 7f01af7ac640 1 -- 172.21.15.59:0/710755953 >> 172.21.15.59:0/710755953 conn(0x7f01a8073e10 msgr2=0x7f01a8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.584+0000 7f01af7ac640 1 -- 172.21.15.59:0/710755953 shutdown_connections 2024-06-29T15:38:54.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.585+0000 7f01af7ac640 1 -- 172.21.15.59:0/710755953 wait complete. 2024-06-29T15:38:54.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.586+0000 7f01af7ac640 1 Processor -- start 2024-06-29T15:38:54.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.586+0000 7f01af7ac640 1 -- start start 2024-06-29T15:38:54.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01af7ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 0x7f01a81f0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01af7ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a80b2fa0 0x7f01a80ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01af7ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a80d1c30 0x7f01a80ae750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01af7ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f01a80b6400 con 0x7f01a80b2270 2024-06-29T15:38:54.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01af7ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01a80b6280 con 0x7f01a80b2fa0 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01af7ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f01a80b6580 con 0x7f01a80d1c30 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 0x7f01a81f0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 0x7f01a81f0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59396/0 (socket says 172.21.15.59:59396) 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 -- 172.21.15.59:0/242660442 learned_addr learned my addr 172.21.15.59:0/242660442 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01acd47640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a80b2fa0 0x7f01a80ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01add49640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a80d1c30 0x7f01a80ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 -- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a80d1c30 msgr2=0x7f01a80ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a80d1c30 0x7f01a80ae750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 -- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a80b2fa0 msgr2=0x7f01a80ac9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a80b2fa0 0x7f01a80ac9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 -- 172.21.15.59:0/242660442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a80aec90 con 0x7f01a80b2270 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.588+0000 7f01ad548640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 0x7f01a81f0070 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f019c006010 tx=0x7f019c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.589+0000 7f0199ffb640 1 -- 172.21.15.59:0/242660442 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c03e030 con 0x7f01a80b2270 2024-06-29T15:38:54.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.589+0000 7f0199ffb640 1 -- 172.21.15.59:0/242660442 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f019c003ae0 con 0x7f01a80b2270 2024-06-29T15:38:54.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.589+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a80aa280 con 0x7f01a80b2270 2024-06-29T15:38:54.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.589+0000 7f0199ffb640 1 -- 172.21.15.59:0/242660442 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c035040 con 0x7f01a80b2270 2024-06-29T15:38:54.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.589+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a80aa840 con 0x7f01a80b2270 2024-06-29T15:38:54.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.590+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a80b6aa0 con 0x7f01a80b2270 2024-06-29T15:38:54.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.592+0000 7f0199ffb640 1 -- 172.21.15.59:0/242660442 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f019c041030 con 0x7f01a80b2270 2024-06-29T15:38:54.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.592+0000 7f0199ffb640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f018c0763f0 0x7f018c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.592+0000 7f01acd47640 1 -- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f018c0763f0 msgr2=0x7f018c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:54.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.592+0000 7f0199ffb640 1 -- 172.21.15.59:0/242660442 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7f019c0b81b0 con 0x7f01a80b2270 2024-06-29T15:38:54.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.592+0000 7f01acd47640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f018c0763f0 0x7f018c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:54.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.597+0000 7f0199ffb640 1 -- 172.21.15.59:0/242660442 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f019c080f10 con 0x7f01a80b2270 2024-06-29T15:38:54.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.790+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01a80ac7c0 con 0x7f01a80b2270 2024-06-29T15:38:54.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.790+0000 7f0199ffb640 1 -- 172.21.15.59:0/242660442 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) ==== 93+0+10 (secure 0 0 0) 0x7f019c085210 con 0x7f01a80b2270 2024-06-29T15:38:54.792 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:38:54.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01acd47640 1 -- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f018c0763f0 msgr2=0x7f018c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:54.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01acd47640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f018c0763f0 0x7f018c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:54.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f018c0763f0 msgr2=0x7f018c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f018c0763f0 0x7f018c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 msgr2=0x7f01a81f0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 0x7f01a81f0070 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f019c006010 tx=0x7f019c0035f0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 shutdown_connections 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f018c0763f0 0x7f018c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01a80d1c30 0x7f01a80ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a80b2fa0 0x7f01a80ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 --2- 172.21.15.59:0/242660442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01a80b2270 0x7f01a81f0070 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 >> 172.21.15.59:0/242660442 conn(0x7f01a8073e10 msgr2=0x7f01a8158170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 shutdown_connections 2024-06-29T15:38:54.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.793+0000 7f01af7ac640 1 -- 172.21.15.59:0/242660442 wait complete. 2024-06-29T15:38:54.806 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:38:54.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.928+0000 7f8fb144d640 1 -- 172.21.15.59:0/2450216535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0ce9e0 msgr2=0x7f8fac0cedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.928+0000 7f8fb144d640 1 --2- 172.21.15.59:0/2450216535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0ce9e0 0x7f8fac0cedf0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009830 tx=0x7f8f9402a510 comp rx=0 tx=0).stop 2024-06-29T15:38:54.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.929+0000 7f8fb144d640 1 -- 172.21.15.59:0/2450216535 shutdown_connections 2024-06-29T15:38:54.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.929+0000 7f8fb144d640 1 --2- 172.21.15.59:0/2450216535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0ce9e0 0x7f8fac0cedf0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.929+0000 7f8fb144d640 1 --2- 172.21.15.59:0/2450216535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fac0b27c0 0x7f8fac0b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.929+0000 7f8fb144d640 1 -- 172.21.15.59:0/2450216535 >> 172.21.15.59:0/2450216535 conn(0x7f8fac0d3ec0 msgr2=0x7f8fac0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.929+0000 7f8fb144d640 1 -- 172.21.15.59:0/2450216535 shutdown_connections 2024-06-29T15:38:54.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.929+0000 7f8fb144d640 1 -- 172.21.15.59:0/2450216535 wait complete. 2024-06-29T15:38:54.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.930+0000 7f8fb144d640 1 Processor -- start 2024-06-29T15:38:54.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.931+0000 7f8fb144d640 1 -- start start 2024-06-29T15:38:54.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fb144d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0b27c0 0x7f8fac148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fb144d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fac0ce9e0 0x7f8fac146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fb144d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fac0d22a0 0x7f8fac1466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fb144d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac0d2f40 con 0x7f8fac0d22a0 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fb144d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac0d2dc0 con 0x7f8fac0b27c0 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fb144d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8fac0d30c0 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fac0d22a0 0x7f8fac1466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0b27c0 0x7f8fac148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fa3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fac0ce9e0 0x7f8fac146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8faaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0b27c0 0x7f8fac148330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:33042/0 (socket says 172.21.15.59:33042) 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8faaffd640 1 -- 172.21.15.59:0/465927102 learned_addr learned my addr 172.21.15.59:0/465927102 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fa3fff640 1 -- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0b27c0 msgr2=0x7f8fac148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fa3fff640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0b27c0 0x7f8fac148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fa3fff640 1 -- 172.21.15.59:0/465927102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fac0d22a0 msgr2=0x7f8fac1466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fa3fff640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fac0d22a0 0x7f8fac1466b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8fa3fff640 1 -- 172.21.15.59:0/465927102 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac146f10 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.932+0000 7f8faaffd640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0b27c0 0x7f8fac148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:54.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.933+0000 7f8fa3fff640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fac0ce9e0 0x7f8fac146170 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0b5450 tx=0x7f8f94033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:54.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.933+0000 7f8fa37fe640 1 -- 172.21.15.59:0/465927102 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94031b50 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.933+0000 7f8fa37fe640 1 -- 172.21.15.59:0/465927102 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f94031de0 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.933+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac1471f0 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.933+0000 7f8fa37fe640 1 -- 172.21.15.59:0/465927102 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94005280 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.933+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac1477b0 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.934+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fac146c80 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.935+0000 7f8fa37fe640 1 -- 172.21.15.59:0/465927102 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f8f94005420 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.935+0000 7f8fa37fe640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f8f8c0763f0 0x7f8f8c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.935+0000 7f8faaffd640 1 -- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f8f8c0763f0 msgr2=0x7f8f8c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:54.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.935+0000 7f8fa37fe640 1 -- 172.21.15.59:0/465927102 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7f8f940b9120 con 0x7f8fac0ce9e0 2024-06-29T15:38:54.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.935+0000 7f8faaffd640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f8f8c0763f0 0x7f8f8c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:54.940+0000 7f8fa37fe640 1 -- 172.21.15.59:0/465927102 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8f9403f030 con 0x7f8fac0ce9e0 2024-06-29T15:38:55.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.132+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8fac0cedf0 con 0x7f8fac0ce9e0 2024-06-29T15:38:55.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.133+0000 7f8fa37fe640 1 -- 172.21.15.59:0/465927102 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v177) ==== 104+0+11 (secure 0 0 0) 0x7f8f94031770 con 0x7f8fac0ce9e0 2024-06-29T15:38:55.134 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:38:55.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f8f8c0763f0 msgr2=0x7f8f8c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:55.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f8f8c0763f0 0x7f8f8c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fac0ce9e0 msgr2=0x7f8fac146170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fac0ce9e0 0x7f8fac146170 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0b5450 tx=0x7f8f94033ba0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 shutdown_connections 2024-06-29T15:38:55.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f8f8c0763f0 0x7f8f8c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fac0d22a0 0x7f8fac1466b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fac0ce9e0 0x7f8fac146170 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 --2- 172.21.15.59:0/465927102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fac0b27c0 0x7f8fac148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 >> 172.21.15.59:0/465927102 conn(0x7f8fac0d3ec0 msgr2=0x7f8fac0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 shutdown_connections 2024-06-29T15:38:55.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.135+0000 7f8fb144d640 1 -- 172.21.15.59:0/465927102 wait complete. 2024-06-29T15:38:55.149 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:38:55.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.272+0000 7fb500124640 1 -- 172.21.15.59:0/2098611044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f80b40a0 msgr2=0x7fb4f815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.272+0000 7fb500124640 1 --2- 172.21.15.59:0/2098611044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f80b40a0 0x7fb4f815e8d0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009830 tx=0x7fb4ec02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.272+0000 7fb500124640 1 -- 172.21.15.59:0/2098611044 shutdown_connections 2024-06-29T15:38:55.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.272+0000 7fb500124640 1 --2- 172.21.15.59:0/2098611044 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f80b40a0 0x7fb4f815e8d0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.272+0000 7fb500124640 1 -- 172.21.15.59:0/2098611044 >> 172.21.15.59:0/2098611044 conn(0x7fb4f80d3ee0 msgr2=0x7fb4f80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.272+0000 7fb500124640 1 -- 172.21.15.59:0/2098611044 shutdown_connections 2024-06-29T15:38:55.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.272+0000 7fb500124640 1 -- 172.21.15.59:0/2098611044 wait complete. 2024-06-29T15:38:55.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.274+0000 7fb500124640 1 Processor -- start 2024-06-29T15:38:55.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.274+0000 7fb500124640 1 -- start start 2024-06-29T15:38:55.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb500124640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4f80d23c0 0x7fb4f8152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb500124640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f815ee10 0x7fb4f81534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb500124640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f814ddb0 0x7fb4f81577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb500124640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f8160450 con 0x7fb4f815ee10 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb500124640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f81602d0 con 0x7fb4f814ddb0 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb500124640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f81605d0 con 0x7fb4f80d23c0 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f815ee10 0x7fb4f81534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f815ee10 0x7fb4f81534b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59416/0 (socket says 172.21.15.59:59416) 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 -- 172.21.15.59:0/3333891943 learned_addr learned my addr 172.21.15.59:0/3333891943 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fe6c1640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f814ddb0 0x7fb4f81577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fdec0640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4f80d23c0 0x7fb4f8152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 -- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4f80d23c0 msgr2=0x7fb4f8152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4f80d23c0 0x7fb4f8152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 -- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f814ddb0 msgr2=0x7fb4f81577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f814ddb0 0x7fb4f81577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 -- 172.21.15.59:0/3333891943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f814d640 con 0x7fb4f815ee10 2024-06-29T15:38:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.276+0000 7fb4fd6bf640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f815ee10 0x7fb4f81534b0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009950 tx=0x7fb4ec033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.277+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/3333891943 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec031d60 con 0x7fb4f815ee10 2024-06-29T15:38:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.277+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/3333891943 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4ec0330d0 con 0x7fb4f815ee10 2024-06-29T15:38:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.277+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f8153b10 con 0x7fb4f815ee10 2024-06-29T15:38:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.277+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/3333891943 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec033520 con 0x7fb4f815ee10 2024-06-29T15:38:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.277+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f8157050 con 0x7fb4f815ee10 2024-06-29T15:38:55.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.278+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f814da60 con 0x7fb4f815ee10 2024-06-29T15:38:55.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.279+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/3333891943 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fb4ec0035f0 con 0x7fb4f815ee10 2024-06-29T15:38:55.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.279+0000 7fb4ea7fc640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fb4dc076420 0x7fb4dc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.279+0000 7fb4fdec0640 1 -- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fb4dc076420 msgr2=0x7fb4dc078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:55.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.280+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/3333891943 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7fb4ec0b90e0 con 0x7fb4f815ee10 2024-06-29T15:38:55.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.280+0000 7fb4fdec0640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fb4dc076420 0x7fb4dc078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:55.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.284+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/3333891943 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb4ec03f030 con 0x7fb4f815ee10 2024-06-29T15:38:55.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.476+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb4f815e8d0 con 0x7fb4f815ee10 2024-06-29T15:38:55.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.477+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/3333891943 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v177) ==== 100+0+11 (secure 0 0 0) 0x7fb4ec081e50 con 0x7fb4f815ee10 2024-06-29T15:38:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:38:55.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fb4dc076420 msgr2=0x7fb4dc078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:55.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fb4dc076420 0x7fb4dc078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f815ee10 msgr2=0x7fb4f81534b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f815ee10 0x7fb4f81534b0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009950 tx=0x7fb4ec033d60 comp rx=0 tx=0).stop 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 shutdown_connections 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fb4dc076420 0x7fb4dc078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4f814ddb0 0x7fb4f81577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f815ee10 0x7fb4f81534b0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 --2- 172.21.15.59:0/3333891943 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb4f80d23c0 0x7fb4f8152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 >> 172.21.15.59:0/3333891943 conn(0x7fb4f80d3ee0 msgr2=0x7fb4f8160940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 shutdown_connections 2024-06-29T15:38:55.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.479+0000 7fb500124640 1 -- 172.21.15.59:0/3333891943 wait complete. 2024-06-29T15:38:55.492 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:38:55.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.611+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1935175609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c00b40a0 msgr2=0x7fe0c015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.611+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1935175609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c00b40a0 0x7fe0c015e8d0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0161d40 tx=0x7fe0ac02a5f0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.611+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1935175609 shutdown_connections 2024-06-29T15:38:55.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.611+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1935175609 >> v2:172.21.15.153:3300/0 conn(0x7fe0c015ee10 0x7fe0c015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.611+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1935175609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c00b40a0 0x7fe0c015e8d0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.611+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1935175609 >> 172.21.15.59:0/1935175609 conn(0x7fe0c00d3ee0 msgr2=0x7fe0c00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.611+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1935175609 shutdown_connections 2024-06-29T15:38:55.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.611+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1935175609 wait complete. 2024-06-29T15:38:55.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.614+0000 7fe0c71bd640 1 Processor -- start 2024-06-29T15:38:55.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.614+0000 7fe0c71bd640 1 -- start start 2024-06-29T15:38:55.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c71bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0c00d23c0 0x7fe0c014ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c71bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c015ee10 0x7fe0c00cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c71bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c0142970 0x7fe0c0142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c71bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0160280 con 0x7fe0c0142970 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c71bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0160100 con 0x7fe0c015ee10 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c71bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c0160400 con 0x7fe0c00d23c0 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c0142970 0x7fe0c0142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c0142970 0x7fe0c0142d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59432/0 (socket says 172.21.15.59:59432) 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 -- 172.21.15.59:0/1689360497 learned_addr learned my addr 172.21.15.59:0/1689360497 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0bffff640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c015ee10 0x7fe0c00cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c4f59640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0c00d23c0 0x7fe0c014ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 -- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0c00d23c0 msgr2=0x7fe0c014ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0c00d23c0 0x7fe0c014ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 -- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c015ee10 msgr2=0x7fe0c00cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c015ee10 0x7fe0c00cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 -- 172.21.15.59:0/1689360497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c00cccd0 con 0x7fe0c0142970 2024-06-29T15:38:55.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.617+0000 7fe0c575a640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c0142970 0x7fe0c0142d80 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800ea10 tx=0x7fe0b800eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:55.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.618+0000 7fe0bd7fa640 1 -- 172.21.15.59:0/1689360497 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b800cd80 con 0x7fe0c0142970 2024-06-29T15:38:55.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.618+0000 7fe0bd7fa640 1 -- 172.21.15.59:0/1689360497 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0b8004c10 con 0x7fe0c0142970 2024-06-29T15:38:55.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.618+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c00ccfb0 con 0x7fe0c0142970 2024-06-29T15:38:55.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.618+0000 7fe0bd7fa640 1 -- 172.21.15.59:0/1689360497 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8005020 con 0x7fe0c0142970 2024-06-29T15:38:55.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.618+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c00cd570 con 0x7fe0c0142970 2024-06-29T15:38:55.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.619+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c015e8d0 con 0x7fe0c0142970 2024-06-29T15:38:55.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.620+0000 7fe0bd7fa640 1 -- 172.21.15.59:0/1689360497 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fe0b80051c0 con 0x7fe0c0142970 2024-06-29T15:38:55.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.620+0000 7fe0bd7fa640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe0a4076420 0x7fe0a4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.621+0000 7fe0c4f59640 1 -- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe0a4076420 msgr2=0x7fe0a4078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:55.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.621+0000 7fe0bd7fa640 1 -- 172.21.15.59:0/1689360497 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7fe0b8098850 con 0x7fe0c0142970 2024-06-29T15:38:55.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.621+0000 7fe0c4f59640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe0a4076420 0x7fe0a4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:55.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.625+0000 7fe0bd7fa640 1 -- 172.21.15.59:0/1689360497 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe0b8061620 con 0x7fe0c0142970 2024-06-29T15:38:55.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.822+0000 7fe0c4f59640 1 -- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe0a4076420 msgr2=0x7fe0a4078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:55.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.822+0000 7fe0c4f59640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe0a4076420 0x7fe0a4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:55.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.828+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe0c00cc9f0 con 0x7fe0c0142970 2024-06-29T15:38:55.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.829+0000 7fe0bd7fa640 1 -- 172.21.15.59:0/1689360497 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v177) ==== 80+0+477 (secure 0 0 0) 0x7fe0b8065920 con 0x7fe0c0142970 2024-06-29T15:38:55.830 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2597046756 2024-06-30T15:38:50.826532+0000 2024-06-29T15:38:55.830 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/623872135 2024-06-30T15:38:50.826532+0000 2024-06-29T15:38:55.830 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2213101666 2024-06-30T15:38:50.826532+0000 2024-06-29T15:38:55.830 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/4098785844 2024-06-30T15:38:50.826532+0000 2024-06-29T15:38:55.830 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/4234893873 2024-06-30T15:38:50.826532+0000 2024-06-29T15:38:55.830 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/2627795750 2024-06-30T15:38:50.826532+0000 2024-06-29T15:38:55.831 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/2627795750 2024-06-30T15:38:50.826532+0000 2024-06-29T15:38:55.831 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/4064948250 2024-06-30T15:38:50.826532+0000 2024-06-29T15:38:55.831 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T15:38:55.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe0a4076420 msgr2=0x7fe0a4078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe0a4076420 0x7fe0a4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c0142970 msgr2=0x7fe0c0142d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c0142970 0x7fe0c0142d80 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800ea10 tx=0x7fe0b800eed0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 shutdown_connections 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fe0a4076420 0x7fe0a4078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0c0142970 0x7fe0c0142d80 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0c015ee10 0x7fe0c00cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 --2- 172.21.15.59:0/1689360497 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe0c00d23c0 0x7fe0c014ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 >> 172.21.15.59:0/1689360497 conn(0x7fe0c00d3ee0 msgr2=0x7fe0c006c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 shutdown_connections 2024-06-29T15:38:55.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:55.831+0000 7fe0c71bd640 1 -- 172.21.15.59:0/1689360497 wait complete. 2024-06-29T15:38:55.845 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:38:55.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:55.845 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:55.874 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T15:38:55.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:55.875 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:38:55.931 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:55.931 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:38:55.931 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:55.947 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:55.947 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:55.948 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:38:55.948 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:55.987 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:38:55.987 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:55.987 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:38:55.987 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:56.003 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:56.003 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:56.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.145+0000 7fd115a55640 1 -- 172.21.15.59:0/1898258598 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1100d23c0 msgr2=0x7fd1100b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.145+0000 7fd115a55640 1 --2- 172.21.15.59:0/1898258598 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1100d23c0 0x7fd1100b3990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd100009830 tx=0x7fd10002a510 comp rx=0 tx=0).stop 2024-06-29T15:38:56.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.145+0000 7fd115a55640 1 -- 172.21.15.59:0/1898258598 shutdown_connections 2024-06-29T15:38:56.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.145+0000 7fd115a55640 1 --2- 172.21.15.59:0/1898258598 >> v2:172.21.15.153:3300/0 conn(0x7fd11015efe0 0x7fd11015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.146+0000 7fd115a55640 1 --2- 172.21.15.59:0/1898258598 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1100d23c0 0x7fd1100b3990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.146+0000 7fd115a55640 1 -- 172.21.15.59:0/1898258598 >> 172.21.15.59:0/1898258598 conn(0x7fd1100d3ee0 msgr2=0x7fd1100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.146+0000 7fd115a55640 1 -- 172.21.15.59:0/1898258598 shutdown_connections 2024-06-29T15:38:56.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.146+0000 7fd115a55640 1 -- 172.21.15.59:0/1898258598 wait complete. 2024-06-29T15:38:56.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.147+0000 7fd115a55640 1 Processor -- start 2024-06-29T15:38:56.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.147+0000 7fd115a55640 1 -- start start 2024-06-29T15:38:56.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd115a55640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1100b3ed0 0x7fd1100cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd115a55640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1100d23c0 0x7fd11015db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd115a55640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11015efe0 0x7fd110156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd115a55640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd110160280 con 0x7fd1100d23c0 2024-06-29T15:38:56.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd115a55640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd110160100 con 0x7fd1100b3ed0 2024-06-29T15:38:56.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd115a55640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd110160400 con 0x7fd11015efe0 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1100d23c0 0x7fd11015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1100d23c0 0x7fd11015db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59444/0 (socket says 172.21.15.59:59444) 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 -- 172.21.15.59:0/37903360 learned_addr learned my addr 172.21.15.59:0/37903360 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd10effd640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1100b3ed0 0x7fd1100cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd10f7fe640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11015efe0 0x7fd110156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 -- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11015efe0 msgr2=0x7fd110156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11015efe0 0x7fd110156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 -- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1100b3ed0 msgr2=0x7fd1100cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1100b3ed0 0x7fd1100cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 -- 172.21.15.59:0/37903360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1101590c0 con 0x7fd1100d23c0 2024-06-29T15:38:56.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.149+0000 7fd107fff640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1100d23c0 0x7fd11015db30 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800c730 tx=0x7fd0f800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:56.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.150+0000 7fd10cff9640 1 -- 172.21.15.59:0/37903360 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8016020 con 0x7fd1100d23c0 2024-06-29T15:38:56.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.150+0000 7fd10cff9640 1 -- 172.21.15.59:0/37903360 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0f800a950 con 0x7fd1100d23c0 2024-06-29T15:38:56.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.150+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1101593a0 con 0x7fd1100d23c0 2024-06-29T15:38:56.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.150+0000 7fd10cff9640 1 -- 172.21.15.59:0/37903360 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f800ada0 con 0x7fd1100d23c0 2024-06-29T15:38:56.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.150+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd110159960 con 0x7fd1100d23c0 2024-06-29T15:38:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.151+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1100b3990 con 0x7fd1100d23c0 2024-06-29T15:38:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.152+0000 7fd10cff9640 1 -- 172.21.15.59:0/37903360 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fd0f8019020 con 0x7fd1100d23c0 2024-06-29T15:38:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.152+0000 7fd10cff9640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fd0f00763f0 0x7fd0f0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.152+0000 7fd10cff9640 1 -- 172.21.15.59:0/37903360 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7fd0f8097790 con 0x7fd1100d23c0 2024-06-29T15:38:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.152+0000 7fd10effd640 1 -- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fd0f00763f0 msgr2=0x7fd0f0078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.152+0000 7fd10effd640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fd0f00763f0 0x7fd0f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.161+0000 7fd10cff9640 1 -- 172.21.15.59:0/37903360 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd0f8020030 con 0x7fd1100d23c0 2024-06-29T15:38:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.353+0000 7fd10effd640 1 -- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fd0f00763f0 msgr2=0x7fd0f0078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.353+0000 7fd10effd640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fd0f00763f0 0x7fd0f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:56.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.394+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1100d2950 con 0x7fd1100d23c0 2024-06-29T15:38:56.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.394+0000 7fd10cff9640 1 -- 172.21.15.59:0/37903360 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5233 (secure 0 0 0) 0x7fd1100b42e0 con 0x7fd1100d23c0 2024-06-29T15:38:56.396 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:56.396 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"btime":"2024-06-29T15:38:42:164025+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1947749735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1947749735},{"type":"v1","addr":"172.21.15.59:6835","nonce":1947749735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3010048644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3010048644},{"type":"v1","addr":"172.21.15.153:6836","nonce":3010048644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9283,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1912783986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1912783986},{"type":"v1","addr":"172.21.15.153:6837","nonce":1912783986}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.523073+0000","modified":"2024-06-29T15:38:41.554713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1057565935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1057565935},{"type":"v1","addr":"172.21.15.59:6834","nonce":1057565935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9894,"qdb_cluster":[9894]},"id":10}]} 2024-06-29T15:38:56.396 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fd0f00763f0 msgr2=0x7fd0f0078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fd0f00763f0 0x7fd0f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1100d23c0 msgr2=0x7fd11015db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1100d23c0 0x7fd11015db30 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800c730 tx=0x7fd0f800cbf0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 shutdown_connections 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fd0f00763f0 0x7fd0f0078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd11015efe0 0x7fd110156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1100d23c0 0x7fd11015db30 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 --2- 172.21.15.59:0/37903360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1100b3ed0 0x7fd1100cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 >> 172.21.15.59:0/37903360 conn(0x7fd1100d3ee0 msgr2=0x7fd11006d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 shutdown_connections 2024-06-29T15:38:56.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.396+0000 7fd115a55640 1 -- 172.21.15.59:0/37903360 wait complete. 2024-06-29T15:38:56.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:56.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.526+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1638095721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e68162190 msgr2=0x7f5e680ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.526+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1638095721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e68162190 0x7f5e680ce550 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f5e680d3710 tx=0x7f5e5802a560 comp rx=0 tx=0).stop 2024-06-29T15:38:56.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.526+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1638095721 shutdown_connections 2024-06-29T15:38:56.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.526+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1638095721 >> v2:172.21.15.153:3300/0 conn(0x7f5e6815e0a0 0x7f5e680b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.526+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1638095721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e68162190 0x7f5e680ce550 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.526+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1638095721 >> 172.21.15.59:0/1638095721 conn(0x7f5e680d45b0 msgr2=0x7f5e680d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.526+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1638095721 shutdown_connections 2024-06-29T15:38:56.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.526+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1638095721 wait complete. 2024-06-29T15:38:56.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.528+0000 7f5e6d63f640 1 Processor -- start 2024-06-29T15:38:56.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.528+0000 7f5e6d63f640 1 -- start start 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e6d63f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e6815e0a0 0x7f5e680cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e6d63f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e68162770 0x7f5e681f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e6d63f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5e68155fd0 0x7f5e681f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e6d63f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5e680b4230 con 0x7f5e6815e0a0 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e6d63f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5e680b40b0 con 0x7f5e68162770 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e6d63f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5e680b43b0 con 0x7f5e68155fd0 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e66ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e6815e0a0 0x7f5e680cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e667fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e68162770 0x7f5e681f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e66ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e6815e0a0 0x7f5e680cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59460/0 (socket says 172.21.15.59:59460) 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e66ffd640 1 -- 172.21.15.59:0/1542817596 learned_addr learned my addr 172.21.15.59:0/1542817596 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.529+0000 7f5e677fe640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5e68155fd0 0x7f5e681f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e66ffd640 1 -- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5e68155fd0 msgr2=0x7f5e681f5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e66ffd640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5e68155fd0 0x7f5e681f5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e66ffd640 1 -- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e68162770 msgr2=0x7f5e681f55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e66ffd640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e68162770 0x7f5e681f55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e66ffd640 1 -- 172.21.15.59:0/1542817596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e681f5ae0 con 0x7f5e6815e0a0 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e667fc640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e68162770 0x7f5e681f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e66ffd640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e6815e0a0 0x7f5e680cd430 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000c3a0 tx=0x7f5e5000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e5f7fe640 1 -- 172.21.15.59:0/1542817596 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e500052d0 con 0x7f5e6815e0a0 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e5f7fe640 1 -- 172.21.15.59:0/1542817596 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e50007020 con 0x7f5e6815e0a0 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e5f7fe640 1 -- 172.21.15.59:0/1542817596 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50007470 con 0x7f5e6815e0a0 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e680cc390 con 0x7f5e6815e0a0 2024-06-29T15:38:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.530+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e680cc950 con 0x7f5e6815e0a0 2024-06-29T15:38:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.531+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6806a7c0 con 0x7f5e6815e0a0 2024-06-29T15:38:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.533+0000 7f5e5f7fe640 1 -- 172.21.15.59:0/1542817596 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f5e500067d0 con 0x7f5e6815e0a0 2024-06-29T15:38:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.533+0000 7f5e5f7fe640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f5e480763f0 0x7f5e48078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.533+0000 7f5e667fc640 1 -- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f5e480763f0 msgr2=0x7f5e48078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.533+0000 7f5e5f7fe640 1 -- 172.21.15.59:0/1542817596 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7f5e500970b0 con 0x7f5e6815e0a0 2024-06-29T15:38:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.533+0000 7f5e667fc640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f5e480763f0 0x7f5e48078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:56.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.538+0000 7f5e5f7fe640 1 -- 172.21.15.59:0/1542817596 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e6806a7c0 con 0x7f5e6815e0a0 2024-06-29T15:38:56.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.729+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e680b48d0 con 0x7f5e6815e0a0 2024-06-29T15:38:56.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.730+0000 7f5e5f7fe640 1 -- 172.21.15.59:0/1542817596 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) ==== 74+0+17456 (secure 0 0 0) 0x7f5e5005ff20 con 0x7f5e6815e0a0 2024-06-29T15:38:56.732 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:56.732 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":177,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:50.826574+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:36.882222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.468804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2597046756":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/623872135":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/2213101666":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4098785844":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4234893873":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6833/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6832/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4064948250":"2024-06-30T15:38:50.826532+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:56.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f5e480763f0 msgr2=0x7f5e48078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:56.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f5e480763f0 0x7f5e48078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e6815e0a0 msgr2=0x7f5e680cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e6815e0a0 0x7f5e680cd430 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000c3a0 tx=0x7f5e5000c860 comp rx=0 tx=0).stop 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 shutdown_connections 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f5e480763f0 0x7f5e48078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5e68155fd0 0x7f5e681f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e68162770 0x7f5e681f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 --2- 172.21.15.59:0/1542817596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e6815e0a0 0x7f5e680cd430 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 >> 172.21.15.59:0/1542817596 conn(0x7f5e680d45b0 msgr2=0x7f5e680b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 shutdown_connections 2024-06-29T15:38:56.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.733+0000 7f5e6d63f640 1 -- 172.21.15.59:0/1542817596 wait complete. 2024-06-29T15:38:56.746 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:38:36.882222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:38:37.468804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '176', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:56.747 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:56.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.868+0000 7fcfdb37a640 1 -- 172.21.15.59:0/3912278924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40b2270 msgr2=0x7fcfd415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.868+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/3912278924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40b2270 0x7fcfd415bc20 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40cf310 tx=0x7fcfd002a5f0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.868+0000 7fcfdb37a640 1 -- 172.21.15.59:0/3912278924 shutdown_connections 2024-06-29T15:38:56.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.868+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/3912278924 >> v2:172.21.15.153:3300/0 conn(0x7fcfd40d1c30 0x7fcfd40b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.868+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/3912278924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40b2270 0x7fcfd415bc20 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.868+0000 7fcfdb37a640 1 -- 172.21.15.59:0/3912278924 >> 172.21.15.59:0/3912278924 conn(0x7fcfd4073e10 msgr2=0x7fcfd4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.868+0000 7fcfdb37a640 1 -- 172.21.15.59:0/3912278924 shutdown_connections 2024-06-29T15:38:56.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.868+0000 7fcfdb37a640 1 -- 172.21.15.59:0/3912278924 wait complete. 2024-06-29T15:38:56.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.870+0000 7fcfdb37a640 1 Processor -- start 2024-06-29T15:38:56.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.870+0000 7fcfdb37a640 1 -- start start 2024-06-29T15:38:56.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfdb37a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd40b2270 0x7fcfd40c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfdb37a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd40b2fa0 0x7fcfd414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfdb37a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40d1c30 0x7fcfd41f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfdb37a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd40b6400 con 0x7fcfd40d1c30 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfdb37a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd40b6280 con 0x7fcfd40b2270 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfdb37a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd40b6580 con 0x7fcfd40b2fa0 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfd9917640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40d1c30 0x7fcfd41f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfd9917640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40d1c30 0x7fcfd41f4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59478/0 (socket says 172.21.15.59:59478) 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfd9917640 1 -- 172.21.15.59:0/1184934412 learned_addr learned my addr 172.21.15.59:0/1184934412 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfd9116640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd40b2270 0x7fcfd40c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.871+0000 7fcfd8915640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd40b2fa0 0x7fcfd414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfd9917640 1 -- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd40b2fa0 msgr2=0x7fcfd414f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfd9917640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd40b2fa0 0x7fcfd414f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfd9917640 1 -- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd40b2270 msgr2=0x7fcfd40c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfd9917640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd40b2270 0x7fcfd40c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfd9917640 1 -- 172.21.15.59:0/1184934412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd414f8a0 con 0x7fcfd40d1c30 2024-06-29T15:38:56.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfd9917640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40d1c30 0x7fcfd41f4aa0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00eab0 tx=0x7fcfcc00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:56.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfc9ffb640 1 -- 172.21.15.59:0/1184934412 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc00cea0 con 0x7fcfd40d1c30 2024-06-29T15:38:56.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfc9ffb640 1 -- 172.21.15.59:0/1184934412 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfcc004b20 con 0x7fcfd40d1c30 2024-06-29T15:38:56.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd415ece0 con 0x7fcfd40d1c30 2024-06-29T15:38:56.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfc9ffb640 1 -- 172.21.15.59:0/1184934412 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc005020 con 0x7fcfd40d1c30 2024-06-29T15:38:56.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.872+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd406e6c0 con 0x7fcfd40d1c30 2024-06-29T15:38:56.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.873+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd40b6aa0 con 0x7fcfd40d1c30 2024-06-29T15:38:56.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.874+0000 7fcfc9ffb640 1 -- 172.21.15.59:0/1184934412 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fcfcc0051c0 con 0x7fcfd40d1c30 2024-06-29T15:38:56.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.874+0000 7fcfc9ffb640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcfb80763f0 0x7fcfb8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.874+0000 7fcfd9116640 1 -- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcfb80763f0 msgr2=0x7fcfb8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:56.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.874+0000 7fcfc9ffb640 1 -- 172.21.15.59:0/1184934412 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7fcfcc0986f0 con 0x7fcfd40d1c30 2024-06-29T15:38:56.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.874+0000 7fcfd9116640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcfb80763f0 0x7fcfb8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:56.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:56.879+0000 7fcfc9ffb640 1 -- 172.21.15.59:0/1184934412 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcfcc0614e0 con 0x7fcfd40d1c30 2024-06-29T15:38:57.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.070+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcfd415f200 con 0x7fcfd40d1c30 2024-06-29T15:38:57.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.071+0000 7fcfc9ffb640 1 -- 172.21.15.59:0/1184934412 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) ==== 93+0+10 (secure 0 0 0) 0x7fcfcc0657e0 con 0x7fcfd40d1c30 2024-06-29T15:38:57.073 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:38:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcfb80763f0 msgr2=0x7fcfb8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcfb80763f0 0x7fcfb8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40d1c30 msgr2=0x7fcfd41f4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40d1c30 0x7fcfd41f4aa0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00eab0 tx=0x7fcfcc00ef70 comp rx=0 tx=0).stop 2024-06-29T15:38:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 shutdown_connections 2024-06-29T15:38:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7fcfb80763f0 0x7fcfb8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd40d1c30 0x7fcfd41f4aa0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd40b2fa0 0x7fcfd414f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 --2- 172.21.15.59:0/1184934412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd40b2270 0x7fcfd40c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 >> 172.21.15.59:0/1184934412 conn(0x7fcfd4073e10 msgr2=0x7fcfd4070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 shutdown_connections 2024-06-29T15:38:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.073+0000 7fcfdb37a640 1 -- 172.21.15.59:0/1184934412 wait complete. 2024-06-29T15:38:57.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:38:57.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 -- 172.21.15.59:0/3111615968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 msgr2=0x7f887c15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 --2- 172.21.15.59:0/3111615968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 0x7f887c15f9b0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f887c0b4a80 tx=0x7f887402aa30 comp rx=0 tx=0).stop 2024-06-29T15:38:57.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 -- 172.21.15.59:0/3111615968 shutdown_connections 2024-06-29T15:38:57.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 --2- 172.21.15.59:0/3111615968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 0x7f887c15f9b0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 --2- 172.21.15.59:0/3111615968 >> v2:172.21.15.153:3300/0 conn(0x7f887c0b3ed0 0x7f887c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 --2- 172.21.15.59:0/3111615968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f887c0d23c0 0x7f887c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 -- 172.21.15.59:0/3111615968 >> 172.21.15.59:0/3111615968 conn(0x7f887c0d3ee0 msgr2=0x7f887c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 -- 172.21.15.59:0/3111615968 shutdown_connections 2024-06-29T15:38:57.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.207+0000 7f8882a8f640 1 -- 172.21.15.59:0/3111615968 wait complete. 2024-06-29T15:38:57.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.209+0000 7f8882a8f640 1 Processor -- start 2024-06-29T15:38:57.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.209+0000 7f8882a8f640 1 -- start start 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.210+0000 7f8882a8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f887c0b3ed0 0x7f887c147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.210+0000 7f8882a8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f887c0d23c0 0x7f887c0c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.210+0000 7f8882a8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 0x7f887c1ea150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.210+0000 7f8882a8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f887c160070 con 0x7f887c15ee10 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.210+0000 7f8882a8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f887c15fef0 con 0x7f887c0d23c0 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.210+0000 7f8882a8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f887c1601f0 con 0x7f887c0b3ed0 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888102c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 0x7f887c1ea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888082b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f887c0b3ed0 0x7f887c147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f887bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f887c0d23c0 0x7f887c0c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888102c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 0x7f887c1ea150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59492/0 (socket says 172.21.15.59:59492) 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f887bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f887c0d23c0 0x7f887c0c6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:33132/0 (socket says 172.21.15.59:33132) 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888082b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f887c0b3ed0 0x7f887c147120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:40836/0 (socket says 172.21.15.59:40836) 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888102c640 1 -- 172.21.15.59:0/1532810201 learned_addr learned my addr 172.21.15.59:0/1532810201 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888082b640 1 -- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f887c0d23c0 msgr2=0x7f887c0c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888082b640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f887c0d23c0 0x7f887c0c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888082b640 1 -- 172.21.15.59:0/1532810201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 msgr2=0x7f887c1ea150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888082b640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 0x7f887c1ea150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888082b640 1 -- 172.21.15.59:0/1532810201 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887c147660 con 0x7f887c0b3ed0 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f888082b640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f887c0b3ed0 0x7f887c147120 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f886400e6a0 tx=0x7f886400eb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f88797fa640 1 -- 172.21.15.59:0/1532810201 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864004220 con 0x7f887c0b3ed0 2024-06-29T15:38:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f88797fa640 1 -- 172.21.15.59:0/1532810201 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88640043c0 con 0x7f887c0b3ed0 2024-06-29T15:38:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887c0c73b0 con 0x7f887c0b3ed0 2024-06-29T15:38:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f88797fa640 1 -- 172.21.15.59:0/1532810201 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886400cb10 con 0x7f887c0b3ed0 2024-06-29T15:38:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.211+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f887c144ce0 con 0x7f887c0b3ed0 2024-06-29T15:38:57.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.212+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887c0b3990 con 0x7f887c0b3ed0 2024-06-29T15:38:57.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.213+0000 7f88797fa640 1 -- 172.21.15.59:0/1532810201 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f886400ccb0 con 0x7f887c0b3ed0 2024-06-29T15:38:57.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.214+0000 7f88797fa640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f885c076420 0x7f885c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.214+0000 7f887bfff640 1 -- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f885c076420 msgr2=0x7f885c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:57.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.214+0000 7f88797fa640 1 -- 172.21.15.59:0/1532810201 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7f88640983f0 con 0x7f887c0b3ed0 2024-06-29T15:38:57.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.214+0000 7f887bfff640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f885c076420 0x7f885c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:57.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.218+0000 7f88797fa640 1 -- 172.21.15.59:0/1532810201 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f887c0c73b0 con 0x7f887c0b3ed0 2024-06-29T15:38:57.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.408+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f887c0d2950 con 0x7f887c0b3ed0 2024-06-29T15:38:57.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.409+0000 7f88797fa640 1 -- 172.21.15.59:0/1532810201 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v177) ==== 104+0+11 (secure 0 0 0) 0x7f887c147660 con 0x7f887c0b3ed0 2024-06-29T15:38:57.410 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:38:57.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f885c076420 msgr2=0x7f885c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f885c076420 0x7f885c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f887c0b3ed0 msgr2=0x7f887c147120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f887c0b3ed0 0x7f887c147120 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f886400e6a0 tx=0x7f886400eb60 comp rx=0 tx=0).stop 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 shutdown_connections 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7f885c076420 0x7f885c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f887c15ee10 0x7f887c1ea150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f887c0d23c0 0x7f887c0c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 --2- 172.21.15.59:0/1532810201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f887c0b3ed0 0x7f887c147120 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 >> 172.21.15.59:0/1532810201 conn(0x7f887c0d3ee0 msgr2=0x7f887c0b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 shutdown_connections 2024-06-29T15:38:57.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.411+0000 7f8882a8f640 1 -- 172.21.15.59:0/1532810201 wait complete. 2024-06-29T15:38:57.424 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:38:57.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.543+0000 7ffa7318c640 1 -- 172.21.15.59:0/3439996002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 msgr2=0x7ffa6c0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.543+0000 7ffa7318c640 1 --2- 172.21.15.59:0/3439996002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 0x7ffa6c0d1520 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0cf310 tx=0x7ffa5802a560 comp rx=0 tx=0).stop 2024-06-29T15:38:57.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.544+0000 7ffa7318c640 1 -- 172.21.15.59:0/3439996002 shutdown_connections 2024-06-29T15:38:57.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.544+0000 7ffa7318c640 1 --2- 172.21.15.59:0/3439996002 >> v2:172.21.15.153:3300/0 conn(0x7ffa6c0d1a60 0x7ffa6c0cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.544+0000 7ffa7318c640 1 --2- 172.21.15.59:0/3439996002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 0x7ffa6c0d1520 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.544+0000 7ffa7318c640 1 --2- 172.21.15.59:0/3439996002 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa6c0b2270 0x7ffa6c15bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.544+0000 7ffa7318c640 1 -- 172.21.15.59:0/3439996002 >> 172.21.15.59:0/3439996002 conn(0x7ffa6c073e10 msgr2=0x7ffa6c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.544+0000 7ffa7318c640 1 -- 172.21.15.59:0/3439996002 shutdown_connections 2024-06-29T15:38:57.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.544+0000 7ffa7318c640 1 -- 172.21.15.59:0/3439996002 wait complete. 2024-06-29T15:38:57.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.545+0000 7ffa7318c640 1 Processor -- start 2024-06-29T15:38:57.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.546+0000 7ffa7318c640 1 -- start start 2024-06-29T15:38:57.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa7318c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa6c0b2270 0x7ffa6c14b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa7318c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 0x7ffa6c1f66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa7318c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c0d1a60 0x7ffa6c159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa7318c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c0cfe40 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa7318c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c0cfcc0 con 0x7ffa6c0d1a60 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa7318c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6c0cffc0 con 0x7ffa6c0b2270 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 0x7ffa6c1f66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa70f28640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa6c0b2270 0x7ffa6c14b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 0x7ffa6c1f66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59512/0 (socket says 172.21.15.59:59512) 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa6bfff640 1 -- 172.21.15.59:0/2072485547 learned_addr learned my addr 172.21.15.59:0/2072485547 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.547+0000 7ffa71729640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c0d1a60 0x7ffa6c159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.548+0000 7ffa6bfff640 1 -- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa6c0b2270 msgr2=0x7ffa6c14b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.548+0000 7ffa6bfff640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa6c0b2270 0x7ffa6c14b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.548+0000 7ffa6bfff640 1 -- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c0d1a60 msgr2=0x7ffa6c159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.548+0000 7ffa6bfff640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c0d1a60 0x7ffa6c159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.548+0000 7ffa6bfff640 1 -- 172.21.15.59:0/2072485547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c0a9df0 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.548+0000 7ffa70f28640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa6c0b2270 0x7ffa6c14b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:57.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.548+0000 7ffa71729640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c0d1a60 0x7ffa6c159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:57.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.548+0000 7ffa6bfff640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 0x7ffa6c1f66c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7ffa58000c00 tx=0x7ffa580038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.549+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5803e030 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.549+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa58003ca0 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.549+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c0aa0d0 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.549+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58035040 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.549+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c0aa690 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.550+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6c0b2680 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.551+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7ffa58041030 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.551+0000 7ffa697fa640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7ffa500763f0 0x7ffa50078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.551+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 6473+0+0 (secure 0 0 0) 0x7ffa580b82a0 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.552+0000 7ffa70f28640 1 -- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7ffa500763f0 msgr2=0x7ffa50078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/770286644 2024-06-29T15:38:57.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.552+0000 7ffa70f28640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7ffa500763f0 0x7ffa50078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.557+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 96) ==== 97912+0+0 (secure 0 0 0) 0x7ffa580b64c0 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.558+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7ffa500763f0 msgr2=0x7ffa50078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:57.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.558+0000 7ffa697fa640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7ffa500763f0 0x7ffa50078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.568+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa6c0b2680 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.612 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d237c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d237c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d237c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d237c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d237c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d5382640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d5382640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d5382640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d5382640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.613 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:38:57.612+0000 7f76d5382640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.794+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffa6c0d0510 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.795+0000 7ffa697fa640 1 -- 172.21.15.59:0/2072485547 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v178) ==== 100+0+11 (secure 0 0 0) 0x7ffa580b2160 con 0x7ffa6c0b2fa0 2024-06-29T15:38:57.796 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:38:57.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 msgr2=0x7ffa6c1f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 0x7ffa6c1f66c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7ffa58000c00 tx=0x7ffa580038a0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 shutdown_connections 2024-06-29T15:38:57.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:6832/770286644,v1:172.21.15.153:6833/770286644] conn(0x7ffa500763f0 0x7ffa50078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6c0d1a60 0x7ffa6c159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa6c0b2fa0 0x7ffa6c1f66c0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 --2- 172.21.15.59:0/2072485547 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa6c0b2270 0x7ffa6c14b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 >> 172.21.15.59:0/2072485547 conn(0x7ffa6c073e10 msgr2=0x7ffa6c158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 shutdown_connections 2024-06-29T15:38:57.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.798+0000 7ffa7318c640 1 -- 172.21.15.59:0/2072485547 wait complete. 2024-06-29T15:38:57.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:57.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.933+0000 7f5431612640 1 -- 172.21.15.59:0/49914377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c08c190 msgr2=0x7f542c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.933+0000 7f5431612640 1 --2- 172.21.15.59:0/49914377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c08c190 0x7f542c0cf080 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f541c005fd0 tx=0x7f541c02a5a0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.934+0000 7f5431612640 1 -- 172.21.15.59:0/49914377 shutdown_connections 2024-06-29T15:38:57.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.934+0000 7f5431612640 1 --2- 172.21.15.59:0/49914377 >> v2:172.21.15.153:3300/0 conn(0x7f542c0ce950 0x7f542c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.934+0000 7f5431612640 1 --2- 172.21.15.59:0/49914377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c08c190 0x7f542c0cf080 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.934+0000 7f5431612640 1 -- 172.21.15.59:0/49914377 >> 172.21.15.59:0/49914377 conn(0x7f542c0b5610 msgr2=0x7f542c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.934+0000 7f5431612640 1 -- 172.21.15.59:0/49914377 shutdown_connections 2024-06-29T15:38:57.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.934+0000 7f5431612640 1 -- 172.21.15.59:0/49914377 wait complete. 2024-06-29T15:38:57.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.935+0000 7f5431612640 1 Processor -- start 2024-06-29T15:38:57.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.935+0000 7f5431612640 1 -- start start 2024-06-29T15:38:57.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5431612640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f542c08c190 0x7f542c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5431612640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c0ce950 0x7f542c14af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5431612640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f542c0cf5c0 0x7f542c14b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5431612640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f542c0b7f60 con 0x7f542c0ce950 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5431612640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f542c0b7de0 con 0x7f542c0cf5c0 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5431612640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f542c0b80e0 con 0x7f542c08c190 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c0ce950 0x7f542c14af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c0ce950 0x7f542c14af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59542/0 (socket says 172.21.15.59:59542) 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 -- 172.21.15.59:0/246534260 learned_addr learned my addr 172.21.15.59:0/246534260 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f542affd640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f542c08c190 0x7f542c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f542b7fe640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f542c0cf5c0 0x7f542c14b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 -- 172.21.15.59:0/246534260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f542c08c190 msgr2=0x7f542c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f542c08c190 0x7f542c15ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 -- 172.21.15.59:0/246534260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f542c0cf5c0 msgr2=0x7f542c14b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f542c0cf5c0 0x7f542c14b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 -- 172.21.15.59:0/246534260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c15b380 con 0x7f542c0ce950 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.937+0000 7f5423fff640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c0ce950 0x7f542c14af80 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f541400c370 tx=0x7f541400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.938+0000 7f54237fe640 1 -- 172.21.15.59:0/246534260 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414017070 con 0x7f542c0ce950 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.938+0000 7f54237fe640 1 -- 172.21.15.59:0/246534260 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54140053c0 con 0x7f542c0ce950 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.938+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542c1490b0 con 0x7f542c0ce950 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.938+0000 7f54237fe640 1 -- 172.21.15.59:0/246534260 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414003850 con 0x7f542c0ce950 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.938+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542c149670 con 0x7f542c0ce950 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.939+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f542c0cf080 con 0x7f542c0ce950 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.939+0000 7f54237fe640 1 -- 172.21.15.59:0/246534260 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 96) ==== 97912+0+0 (secure 0 0 0) 0x7f541401a070 con 0x7f542c0ce950 2024-06-29T15:38:57.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.940+0000 7f54237fe640 1 -- 172.21.15.59:0/246534260 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 6817+0+0 (secure 0 0 0) 0x7f5414096a10 con 0x7f542c0ce950 2024-06-29T15:38:57.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:57.945+0000 7f54237fe640 1 -- 172.21.15.59:0/246534260 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f541405fc90 con 0x7f542c0ce950 2024-06-29T15:38:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.174+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f542c1e9290 con 0x7f542c0ce950 2024-06-29T15:38:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.174+0000 7f54237fe640 1 -- 172.21.15.59:0/246534260 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5233 (secure 0 0 0) 0x7f54140682d0 con 0x7f542c0ce950 2024-06-29T15:38:58.175 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:58.175 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"btime":"2024-06-29T15:38:42:164025+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1947749735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1947749735},{"type":"v1","addr":"172.21.15.59:6835","nonce":1947749735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3010048644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3010048644},{"type":"v1","addr":"172.21.15.153:6836","nonce":3010048644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9283,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1912783986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1912783986},{"type":"v1","addr":"172.21.15.153:6837","nonce":1912783986}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.523073+0000","modified":"2024-06-29T15:38:41.554713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1057565935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1057565935},{"type":"v1","addr":"172.21.15.59:6834","nonce":1057565935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9894,"qdb_cluster":[9894]},"id":10}]} 2024-06-29T15:38:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-06-29T15:38:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c0ce950 msgr2=0x7f542c14af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c0ce950 0x7f542c14af80 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f541400c370 tx=0x7f541400c830 comp rx=0 tx=0).stop 2024-06-29T15:38:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 shutdown_connections 2024-06-29T15:38:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f542c0cf5c0 0x7f542c14b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f542c0ce950 0x7f542c14af80 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 --2- 172.21.15.59:0/246534260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f542c08c190 0x7f542c15ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 >> 172.21.15.59:0/246534260 conn(0x7f542c0b5610 msgr2=0x7f542c0c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 shutdown_connections 2024-06-29T15:38:58.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.176+0000 7f5431612640 1 -- 172.21.15.59:0/246534260 wait complete. 2024-06-29T15:38:58.188 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:58.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.305+0000 7fa58ee47640 1 -- 172.21.15.59:0/2586997492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 msgr2=0x7fa58806e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.305+0000 7fa58ee47640 1 --2- 172.21.15.59:0/2586997492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 0x7fa58806e300 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fa570005fd0 tx=0x7fa57002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.305+0000 7fa58ee47640 1 -- 172.21.15.59:0/2586997492 shutdown_connections 2024-06-29T15:38:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.305+0000 7fa58ee47640 1 --2- 172.21.15.59:0/2586997492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 0x7fa58806e300 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.305+0000 7fa58ee47640 1 --2- 172.21.15.59:0/2586997492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa588144470 0x7fa5880b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.305+0000 7fa58ee47640 1 -- 172.21.15.59:0/2586997492 >> 172.21.15.59:0/2586997492 conn(0x7fa5880b06e0 msgr2=0x7fa5880b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.305+0000 7fa58ee47640 1 -- 172.21.15.59:0/2586997492 shutdown_connections 2024-06-29T15:38:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.305+0000 7fa58ee47640 1 -- 172.21.15.59:0/2586997492 wait complete. 2024-06-29T15:38:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.307+0000 7fa58ee47640 1 Processor -- start 2024-06-29T15:38:58.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.307+0000 7fa58ee47640 1 -- start start 2024-06-29T15:38:58.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa58ee47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa58806e840 0x7fa588140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa58ee47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 0x7fa5881f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa58ee47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa588144470 0x7fa5881424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa58ee47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa5880d0360 con 0x7fa5880b2040 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa58ee47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5880d01e0 con 0x7fa58806e840 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa58ee47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa58815ee80 con 0x7fa588144470 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa587fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 0x7fa5881f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa587fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 0x7fa5881f4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59548/0 (socket says 172.21.15.59:59548) 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa587fff640 1 -- 172.21.15.59:0/2418760368 learned_addr learned my addr 172.21.15.59:0/2418760368 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa58d3e4640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa588144470 0x7fa5881424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.308+0000 7fa58cbe3640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa58806e840 0x7fa588140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa587fff640 1 -- 172.21.15.59:0/2418760368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa588144470 msgr2=0x7fa5881424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa587fff640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa588144470 0x7fa5881424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa587fff640 1 -- 172.21.15.59:0/2418760368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa58806e840 msgr2=0x7fa588140c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa587fff640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa58806e840 0x7fa588140c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa587fff640 1 -- 172.21.15.59:0/2418760368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5881f5910 con 0x7fa5880b2040 2024-06-29T15:38:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa587fff640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 0x7fa5881f4e50 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fa570034040 tx=0x7fa5700035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa5857fa640 1 -- 172.21.15.59:0/2418760368 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57003e030 con 0x7fa5880b2040 2024-06-29T15:38:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa5857fa640 1 -- 172.21.15.59:0/2418760368 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa570003ae0 con 0x7fa5880b2040 2024-06-29T15:38:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5881f5bf0 con 0x7fa5880b2040 2024-06-29T15:38:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa5857fa640 1 -- 172.21.15.59:0/2418760368 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570035040 con 0x7fa5880b2040 2024-06-29T15:38:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.309+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5881f5390 con 0x7fa5880b2040 2024-06-29T15:38:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.310+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5881f5680 con 0x7fa5880b2040 2024-06-29T15:38:58.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.311+0000 7fa5857fa640 1 -- 172.21.15.59:0/2418760368 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 96) ==== 97912+0+0 (secure 0 0 0) 0x7fa570041030 con 0x7fa5880b2040 2024-06-29T15:38:58.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.311+0000 7fa5857fa640 1 -- 172.21.15.59:0/2418760368 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 6817+0+0 (secure 0 0 0) 0x7fa5700b7e60 con 0x7fa5880b2040 2024-06-29T15:38:58.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.316+0000 7fa5857fa640 1 -- 172.21.15.59:0/2418760368 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa5700817b0 con 0x7fa5880b2040 2024-06-29T15:38:58.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.505+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa58815f3d0 con 0x7fa5880b2040 2024-06-29T15:38:58.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.506+0000 7fa5857fa640 1 -- 172.21.15.59:0/2418760368 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) ==== 74+0+17962 (secure 0 0 0) 0x7fa5700810b0 con 0x7fa5880b2040 2024-06-29T15:38:58.507 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:58.507 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":178,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:57.549395+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:36.882222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.468804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2302300861":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3385295272":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3035080074":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3418701370":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/780112437":"2024-06-30T15:38:57.549343+0000","172.21.15.153:6833/770286644":"2024-06-30T15:38:57.549343+0000","172.21.15.153:6832/770286644":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/2597046756":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/623872135":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/2213101666":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4098785844":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4234893873":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6833/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6832/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/851260222":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/4064948250":"2024-06-30T15:38:50.826532+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:58.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 msgr2=0x7fa5881f4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 0x7fa5881f4e50 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fa570034040 tx=0x7fa5700035f0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 shutdown_connections 2024-06-29T15:38:58.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa588144470 0x7fa5881424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880b2040 0x7fa5881f4e50 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 --2- 172.21.15.59:0/2418760368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa58806e840 0x7fa588140c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 >> 172.21.15.59:0/2418760368 conn(0x7fa5880b06e0 msgr2=0x7fa5880b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 shutdown_connections 2024-06-29T15:38:58.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.508+0000 7fa58ee47640 1 -- 172.21.15.59:0/2418760368 wait complete. 2024-06-29T15:38:58.521 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:38:58.521 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:58.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.640+0000 7f05984d6640 1 -- 172.21.15.59:0/2178402077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 msgr2=0x7f059015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.640+0000 7f05984d6640 1 --2- 172.21.15.59:0/2178402077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 0x7f059015e8d0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f0580005fd0 tx=0x7f058002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.640+0000 7f05984d6640 1 -- 172.21.15.59:0/2178402077 shutdown_connections 2024-06-29T15:38:58.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.640+0000 7f05984d6640 1 --2- 172.21.15.59:0/2178402077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 0x7f059015e8d0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.640+0000 7f05984d6640 1 --2- 172.21.15.59:0/2178402077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05900d23c0 0x7f05900b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.640+0000 7f05984d6640 1 -- 172.21.15.59:0/2178402077 >> 172.21.15.59:0/2178402077 conn(0x7f05900d3ee0 msgr2=0x7f05900d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.640+0000 7f05984d6640 1 -- 172.21.15.59:0/2178402077 shutdown_connections 2024-06-29T15:38:58.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.640+0000 7f05984d6640 1 -- 172.21.15.59:0/2178402077 wait complete. 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.642+0000 7f05984d6640 1 Processor -- start 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.642+0000 7f05984d6640 1 -- start start 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f05984d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 0x7f05900cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f05984d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05900d23c0 0x7f05901594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f05984d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f059015ee10 0x7f0590154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f05984d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0590160240 con 0x7f05900b3ed0 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f05984d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05901600c0 con 0x7f05900d23c0 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f05984d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f05901603c0 con 0x7f059015ee10 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f0596272640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 0x7f05900cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f0596272640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 0x7f05900cccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59568/0 (socket says 172.21.15.59:59568) 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f0596272640 1 -- 172.21.15.59:0/1467676377 learned_addr learned my addr 172.21.15.59:0/1467676377 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:58.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f0595a71640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05900d23c0 0x7f05901594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.643+0000 7f0596a73640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f059015ee10 0x7f0590154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0596272640 1 -- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f059015ee10 msgr2=0x7f0590154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0596272640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f059015ee10 0x7f0590154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0596272640 1 -- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05900d23c0 msgr2=0x7f05901594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0596272640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05900d23c0 0x7f05901594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0596272640 1 -- 172.21.15.59:0/1467676377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0590145ae0 con 0x7f05900b3ed0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0595a71640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05900d23c0 0x7f05901594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0596272640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 0x7f05900cccb0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f058c00d700 tx=0x7f058c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0586ffd640 1 -- 172.21.15.59:0/1467676377 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c017070 con 0x7f05900b3ed0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0586ffd640 1 -- 172.21.15.59:0/1467676377 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f058c004310 con 0x7f05900b3ed0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0590145dc0 con 0x7f05900b3ed0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f0586ffd640 1 -- 172.21.15.59:0/1467676377 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c009a80 con 0x7f05900b3ed0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.644+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0590146380 con 0x7f05900b3ed0 2024-06-29T15:38:58.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.645+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f059006ce50 con 0x7f05900b3ed0 2024-06-29T15:38:58.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.646+0000 7f0586ffd640 1 -- 172.21.15.59:0/1467676377 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7f058c009c20 con 0x7f05900b3ed0 2024-06-29T15:38:58.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.647+0000 7f0586ffd640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0574076120 0x7f0574078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.647+0000 7f0586ffd640 1 -- 172.21.15.59:0/1467676377 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 6817+0+0 (secure 0 0 0) 0x7f058c0978c0 con 0x7f05900b3ed0 2024-06-29T15:38:58.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.647+0000 7f0595a71640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0574076120 0x7f0574078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.647+0000 7f0595a71640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0574076120 0x7f0574078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0580033000 tx=0x7f0580004730 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:58.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.651+0000 7f0586ffd640 1 -- 172.21.15.59:0/1467676377 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f059006ce50 con 0x7f05900b3ed0 2024-06-29T15:38:58.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.880+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f059006a7c0 con 0x7f05900b3ed0 2024-06-29T15:38:58.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.880+0000 7f0586ffd640 1 -- 172.21.15.59:0/1467676377 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5233 (secure 0 0 0) 0x7f059006a7c0 con 0x7f05900b3ed0 2024-06-29T15:38:58.881 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:58.882 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"btime":"2024-06-29T15:38:42:164025+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1947749735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1947749735},{"type":"v1","addr":"172.21.15.59:6835","nonce":1947749735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3010048644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3010048644},{"type":"v1","addr":"172.21.15.153:6836","nonce":3010048644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9283,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1912783986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1912783986},{"type":"v1","addr":"172.21.15.153:6837","nonce":1912783986}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.523073+0000","modified":"2024-06-29T15:38:41.554713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1057565935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1057565935},{"type":"v1","addr":"172.21.15.59:6834","nonce":1057565935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9894,"qdb_cluster":[9894]},"id":10}]} 2024-06-29T15:38:58.882 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-06-29T15:38:58.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0574076120 msgr2=0x7f0574078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0574076120 0x7f0574078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0580033000 tx=0x7f0580004730 comp rx=0 tx=0).stop 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 msgr2=0x7f05900cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 0x7f05900cccb0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f058c00d700 tx=0x7f058c00dbc0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 shutdown_connections 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0574076120 0x7f0574078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f059015ee10 0x7f0590154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05900d23c0 0x7f05901594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 --2- 172.21.15.59:0/1467676377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900b3ed0 0x7f05900cccb0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 >> 172.21.15.59:0/1467676377 conn(0x7f05900d3ee0 msgr2=0x7f05900d2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 shutdown_connections 2024-06-29T15:38:58.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:58.883+0000 7f05984d6640 1 -- 172.21.15.59:0/1467676377 wait complete. 2024-06-29T15:38:58.895 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:59.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.011+0000 7faedb247640 1 -- 172.21.15.59:0/1624269789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed40b40a0 msgr2=0x7faed415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.011+0000 7faedb247640 1 --2- 172.21.15.59:0/1624269789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed40b40a0 0x7faed415e8d0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7faebc009830 tx=0x7faebc02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.011+0000 7faedb247640 1 -- 172.21.15.59:0/1624269789 shutdown_connections 2024-06-29T15:38:59.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.011+0000 7faedb247640 1 --2- 172.21.15.59:0/1624269789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed40b40a0 0x7faed415e8d0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.011+0000 7faedb247640 1 -- 172.21.15.59:0/1624269789 >> 172.21.15.59:0/1624269789 conn(0x7faed40d3ee0 msgr2=0x7faed40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.011+0000 7faedb247640 1 -- 172.21.15.59:0/1624269789 shutdown_connections 2024-06-29T15:38:59.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.011+0000 7faedb247640 1 -- 172.21.15.59:0/1624269789 wait complete. 2024-06-29T15:38:59.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.013+0000 7faedb247640 1 Processor -- start 2024-06-29T15:38:59.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.013+0000 7faedb247640 1 -- start start 2024-06-29T15:38:59.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faedb247640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faed40d23c0 0x7faed4150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faedb247640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faed415ee10 0x7faed4157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faedb247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed414b140 0x7faed414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faedb247640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7faed4160450 con 0x7faed414b140 2024-06-29T15:38:59.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faedb247640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faed41602d0 con 0x7faed40d23c0 2024-06-29T15:38:59.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faedb247640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7faed41605d0 con 0x7faed415ee10 2024-06-29T15:38:59.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed414b140 0x7faed414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed414b140 0x7faed414ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59574/0 (socket says 172.21.15.59:59574) 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 -- 172.21.15.59:0/3587433984 learned_addr learned my addr 172.21.15.59:0/3587433984 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed8fe3640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faed40d23c0 0x7faed4150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed3fff640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faed415ee10 0x7faed4157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 -- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faed415ee10 msgr2=0x7faed4157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faed415ee10 0x7faed4157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 -- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faed40d23c0 msgr2=0x7faed4150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faed40d23c0 0x7faed4150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 -- 172.21.15.59:0/3587433984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed4152ee0 con 0x7faed414b140 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed3fff640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faed415ee10 0x7faed4157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed97e4640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed414b140 0x7faed414ba80 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7faecc00c3f0 tx=0x7faecc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed17fa640 1 -- 172.21.15.59:0/3587433984 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc017070 con 0x7faed414b140 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed17fa640 1 -- 172.21.15.59:0/3587433984 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faecc0035f0 con 0x7faed414b140 2024-06-29T15:38:59.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed4156b90 con 0x7faed414b140 2024-06-29T15:38:59.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.015+0000 7faed17fa640 1 -- 172.21.15.59:0/3587433984 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc003a40 con 0x7faed414b140 2024-06-29T15:38:59.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.016+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed4157150 con 0x7faed414b140 2024-06-29T15:38:59.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.017+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed415e8d0 con 0x7faed414b140 2024-06-29T15:38:59.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.017+0000 7faed17fa640 1 -- 172.21.15.59:0/3587433984 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7faecc01a070 con 0x7faed414b140 2024-06-29T15:38:59.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.018+0000 7faed17fa640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faeb4076150 0x7faeb4078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.018+0000 7faed17fa640 1 -- 172.21.15.59:0/3587433984 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 6817+0+0 (secure 0 0 0) 0x7faecc096a70 con 0x7faed414b140 2024-06-29T15:38:59.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.018+0000 7faed8fe3640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faeb4076150 0x7faeb4078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.019+0000 7faed8fe3640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faeb4076150 0x7faeb4078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faec40012e0 tx=0x7faec4009040 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.022+0000 7faed17fa640 1 -- 172.21.15.59:0/3587433984 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faecc05fbd0 con 0x7faed414b140 2024-06-29T15:38:59.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.251+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faed40d2950 con 0x7faed414b140 2024-06-29T15:38:59.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.251+0000 7faed17fa640 1 -- 172.21.15.59:0/3587433984 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5233 (secure 0 0 0) 0x7faecc0682a0 con 0x7faed414b140 2024-06-29T15:38:59.253 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:59.253 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"btime":"2024-06-29T15:38:42:164025+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1947749735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1947749735},{"type":"v1","addr":"172.21.15.59:6835","nonce":1947749735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3010048644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3010048644},{"type":"v1","addr":"172.21.15.153:6836","nonce":3010048644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9283,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1912783986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1912783986},{"type":"v1","addr":"172.21.15.153:6837","nonce":1912783986}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.523073+0000","modified":"2024-06-29T15:38:41.554713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9894":{"gid":9894,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1057565935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1057565935},{"type":"v1","addr":"172.21.15.59:6834","nonce":1057565935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9894,"qdb_cluster":[9894]},"id":10}]} 2024-06-29T15:38:59.253 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faeb4076150 msgr2=0x7faeb4078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faeb4076150 0x7faeb4078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faec40012e0 tx=0x7faec4009040 comp rx=0 tx=0).stop 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed414b140 msgr2=0x7faed414ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed414b140 0x7faed414ba80 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7faecc00c3f0 tx=0x7faecc00c8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 shutdown_connections 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faeb4076150 0x7faeb4078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed414b140 0x7faed414ba80 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faed415ee10 0x7faed4157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.253+0000 7faedb247640 1 --2- 172.21.15.59:0/3587433984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faed40d23c0 0x7faed4150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.254+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 >> 172.21.15.59:0/3587433984 conn(0x7faed40d3ee0 msgr2=0x7faed4151260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.254+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 shutdown_connections 2024-06-29T15:38:59.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.254+0000 7faedb247640 1 -- 172.21.15.59:0/3587433984 wait complete. 2024-06-29T15:38:59.266 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.382+0000 7fbc36d3c640 1 -- 172.21.15.59:0/1911314905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3015ceb0 msgr2=0x7fbc300b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.382+0000 7fbc36d3c640 1 --2- 172.21.15.59:0/1911314905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3015ceb0 0x7fbc300b56d0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009830 tx=0x7fbc2002a480 comp rx=0 tx=0).stop 2024-06-29T15:38:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.382+0000 7fbc36d3c640 1 -- 172.21.15.59:0/1911314905 shutdown_connections 2024-06-29T15:38:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.382+0000 7fbc36d3c640 1 --2- 172.21.15.59:0/1911314905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3015ceb0 0x7fbc300b56d0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.382+0000 7fbc36d3c640 1 -- 172.21.15.59:0/1911314905 >> 172.21.15.59:0/1911314905 conn(0x7fbc300b6fb0 msgr2=0x7fbc300b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.382+0000 7fbc36d3c640 1 -- 172.21.15.59:0/1911314905 shutdown_connections 2024-06-29T15:38:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.382+0000 7fbc36d3c640 1 -- 172.21.15.59:0/1911314905 wait complete. 2024-06-29T15:38:59.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.384+0000 7fbc36d3c640 1 Processor -- start 2024-06-29T15:38:59.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.384+0000 7fbc36d3c640 1 -- start start 2024-06-29T15:38:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc36d3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc3013e9b0 0x7fbc30152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc36d3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc3015ceb0 0x7fbc301399e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc36d3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3006f920 0x7fbc3006fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc36d3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3013ef40 con 0x7fbc3006f920 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc36d3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3013edc0 con 0x7fbc3013e9b0 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc36d3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3013f0c0 con 0x7fbc3015ceb0 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc352d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3006f920 0x7fbc3006fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc352d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3006f920 0x7fbc3006fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59586/0 (socket says 172.21.15.59:59586) 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.385+0000 7fbc352d9640 1 -- 172.21.15.59:0/2737213779 learned_addr learned my addr 172.21.15.59:0/2737213779 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc34ad8640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc3013e9b0 0x7fbc30152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc2ffff640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc3015ceb0 0x7fbc301399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc352d9640 1 -- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc3015ceb0 msgr2=0x7fbc301399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc352d9640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc3015ceb0 0x7fbc301399e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc352d9640 1 -- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc3013e9b0 msgr2=0x7fbc30152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc352d9640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc3013e9b0 0x7fbc30152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc352d9640 1 -- 172.21.15.59:0/2737213779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc301532c0 con 0x7fbc3006f920 2024-06-29T15:38:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc352d9640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3006f920 0x7fbc3006fd30 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c760 tx=0x7fbc2800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc2d7fa640 1 -- 172.21.15.59:0/2737213779 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc28003b70 con 0x7fbc3006f920 2024-06-29T15:38:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc2d7fa640 1 -- 172.21.15.59:0/2737213779 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc28003d10 con 0x7fbc3006f920 2024-06-29T15:38:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc2d7fa640 1 -- 172.21.15.59:0/2737213779 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc280052d0 con 0x7fbc3006f920 2024-06-29T15:38:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc30070270 con 0x7fbc3006f920 2024-06-29T15:38:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.386+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc30137d80 con 0x7fbc3006f920 2024-06-29T15:38:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.387+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc30153f80 con 0x7fbc3006f920 2024-06-29T15:38:59.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.388+0000 7fbc2d7fa640 1 -- 172.21.15.59:0/2737213779 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7fbc28005470 con 0x7fbc3006f920 2024-06-29T15:38:59.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.389+0000 7fbc2d7fa640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fbc1c076120 0x7fbc1c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.389+0000 7fbc2d7fa640 1 -- 172.21.15.59:0/2737213779 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 6817+0+0 (secure 0 0 0) 0x7fbc28098170 con 0x7fbc3006f920 2024-06-29T15:38:59.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.389+0000 7fbc34ad8640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fbc1c076120 0x7fbc1c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.390+0000 7fbc34ad8640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fbc1c076120 0x7fbc1c078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbc20034000 tx=0x7fbc20004730 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.392+0000 7fbc2d7fa640 1 -- 172.21.15.59:0/2737213779 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbc30153f80 con 0x7fbc3006f920 2024-06-29T15:38:59.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.585+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc30050470 con 0x7fbc3006f920 2024-06-29T15:38:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.586+0000 7fbc2d7fa640 1 -- 172.21.15.59:0/2737213779 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) ==== 74+0+17962 (secure 0 0 0) 0x7fbc30050470 con 0x7fbc3006f920 2024-06-29T15:38:59.588 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:38:59.588 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":178,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:38:57.549395+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:36.882222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.468804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"176","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2302300861":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3385295272":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3035080074":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3418701370":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/780112437":"2024-06-30T15:38:57.549343+0000","172.21.15.153:6833/770286644":"2024-06-30T15:38:57.549343+0000","172.21.15.153:6832/770286644":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/2597046756":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/623872135":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/2213101666":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4098785844":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4234893873":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6833/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6832/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/851260222":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/4064948250":"2024-06-30T15:38:50.826532+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:59.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fbc1c076120 msgr2=0x7fbc1c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fbc1c076120 0x7fbc1c078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbc20034000 tx=0x7fbc20004730 comp rx=0 tx=0).stop 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3006f920 msgr2=0x7fbc3006fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3006f920 0x7fbc3006fd30 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c760 tx=0x7fbc2800cb60 comp rx=0 tx=0).stop 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 shutdown_connections 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fbc1c076120 0x7fbc1c078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3006f920 0x7fbc3006fd30 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbc3015ceb0 0x7fbc301399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 --2- 172.21.15.59:0/2737213779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc3013e9b0 0x7fbc30152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 >> 172.21.15.59:0/2737213779 conn(0x7fbc300b6fb0 msgr2=0x7fbc30153770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 shutdown_connections 2024-06-29T15:38:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.589+0000 7fbc36d3c640 1 -- 172.21.15.59:0/2737213779 wait complete. 2024-06-29T15:38:59.602 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:38:59.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.721+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1719438028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 msgr2=0x7f3d0c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.721+0000 7f3d2c08f640 1 --2- 172.21.15.59:0/1719438028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 0x7f3d0c004c10 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f3d180016c0 tx=0x7f3d0c005830 comp rx=0 tx=0).stop 2024-06-29T15:38:59.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.721+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1719438028 shutdown_connections 2024-06-29T15:38:59.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.721+0000 7f3d2c08f640 1 --2- 172.21.15.59:0/1719438028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 0x7f3d0c004c10 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.721+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1719438028 >> 172.21.15.59:0/1719438028 conn(0x7f3d240d45b0 msgr2=0x7f3d240d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.721+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1719438028 shutdown_connections 2024-06-29T15:38:59.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.721+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1719438028 wait complete. 2024-06-29T15:38:59.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.723+0000 7f3d2c08f640 1 Processor -- start 2024-06-29T15:38:59.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.723+0000 7f3d2c08f640 1 -- start start 2024-06-29T15:38:59.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d2c08f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 0x7f3d240cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d2c08f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d241487b0 0x7f3d240b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d2c08f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240b8120 0x7f3d2414a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d2c08f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3d24161a80 con 0x7f3d0c004800 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d2c08f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3d240ce7c0 con 0x7f3d240b8120 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d2c08f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3d24161c00 con 0x7f3d241487b0 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d29e2b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 0x7f3d240cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d29e2b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 0x7f3d240cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59616/0 (socket says 172.21.15.59:59616) 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d29e2b640 1 -- 172.21.15.59:0/1829860751 learned_addr learned my addr 172.21.15.59:0/1829860751 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d2962a640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d241487b0 0x7f3d240b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d2a62c640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240b8120 0x7f3d2414a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d29e2b640 1 -- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d241487b0 msgr2=0x7f3d240b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d29e2b640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d241487b0 0x7f3d240b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d29e2b640 1 -- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240b8120 msgr2=0x7f3d2414a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.724+0000 7f3d29e2b640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240b8120 0x7f3d2414a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.725+0000 7f3d29e2b640 1 -- 172.21.15.59:0/1829860751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d240b8530 con 0x7f3d0c004800 2024-06-29T15:38:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.725+0000 7f3d2962a640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d241487b0 0x7f3d240b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.725+0000 7f3d29e2b640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 0x7f3d240cd3c0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f3d1400f980 tx=0x7f3d1400fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.725+0000 7f3d227fc640 1 -- 172.21.15.59:0/1829860751 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1400dcd0 con 0x7f3d0c004800 2024-06-29T15:38:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.725+0000 7f3d227fc640 1 -- 172.21.15.59:0/1829860751 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d14011440 con 0x7f3d0c004800 2024-06-29T15:38:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.725+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d24149130 con 0x7f3d0c004800 2024-06-29T15:38:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.725+0000 7f3d227fc640 1 -- 172.21.15.59:0/1829860751 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d140117f0 con 0x7f3d0c004800 2024-06-29T15:38:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.725+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d241496b0 con 0x7f3d0c004800 2024-06-29T15:38:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.726+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d24162160 con 0x7f3d0c004800 2024-06-29T15:38:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.727+0000 7f3d227fc640 1 -- 172.21.15.59:0/1829860751 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7f3d14019020 con 0x7f3d0c004800 2024-06-29T15:38:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.727+0000 7f3d227fc640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3d10076120 0x7f3d10078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.727+0000 7f3d227fc640 1 -- 172.21.15.59:0/1829860751 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 6817+0+0 (secure 0 0 0) 0x7f3d140982e0 con 0x7f3d0c004800 2024-06-29T15:38:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.727+0000 7f3d2962a640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3d10076120 0x7f3d10078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.728+0000 7f3d2962a640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3d10076120 0x7f3d10078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d18032000 tx=0x7f3d18001190 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.732+0000 7f3d227fc640 1 -- 172.21.15.59:0/1829860751 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3d240b8530 con 0x7f3d0c004800 2024-06-29T15:38:59.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.907+0000 7f3d227fc640 1 -- 172.21.15.59:0/1829860751 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 98) ==== 98247+0+0 (secure 0 0 0) 0x7f3d1405d6b0 con 0x7f3d0c004800 2024-06-29T15:38:59.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.969+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3d24075b40 con 0x7f3d0c004800 2024-06-29T15:38:59.970 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:38:59.969+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:38:59.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.971+0000 7f3d227fc640 1 -- 172.21.15.59:0/1829860751 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v89) ==== 161+0+0 (secure 0 0 0) 0x7f3d140103d0 con 0x7f3d0c004800 2024-06-29T15:38:59.972 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.973+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3d10076120 msgr2=0x7f3d10078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.973+0000 7f3d2c08f640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3d10076120 0x7f3d10078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d18032000 tx=0x7f3d18001190 comp rx=0 tx=0).stop 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.973+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 msgr2=0x7f3d240cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.973+0000 7f3d2c08f640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 0x7f3d240cd3c0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f3d1400f980 tx=0x7f3d1400fe40 comp rx=0 tx=0).stop 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.973+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 shutdown_connections 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.974+0000 7f3d2c08f640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3d10076120 0x7f3d10078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.974+0000 7f3d2c08f640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240b8120 0x7f3d2414a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.974+0000 7f3d2c08f640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d241487b0 0x7f3d240b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.974+0000 7f3d2c08f640 1 --2- 172.21.15.59:0/1829860751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c004800 0x7f3d240cd3c0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.974+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 >> 172.21.15.59:0/1829860751 conn(0x7f3d240d45b0 msgr2=0x7f3d240d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.974+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 shutdown_connections 2024-06-29T15:38:59.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:38:59.974+0000 7f3d2c08f640 1 -- 172.21.15.59:0/1829860751 wait complete. 2024-06-29T15:38:59.977 INFO:tasks.ceph.mds.c.smithi059.stderr: -4330> 2024-06-29T15:38:40.111+0000 7f9f9f3d9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:38:59.987 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:39:00.034 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:39:00.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.109+0000 7fec1a4f6640 1 -- 172.21.15.59:0/1208933963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140b2a50 msgr2=0x7fec14161380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.109+0000 7fec1a4f6640 1 --2- 172.21.15.59:0/1208933963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140b2a50 0x7fec14161380 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fec140d2c60 tx=0x7fec0402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.109+0000 7fec1a4f6640 1 -- 172.21.15.59:0/1208933963 shutdown_connections 2024-06-29T15:39:00.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.109+0000 7fec1a4f6640 1 --2- 172.21.15.59:0/1208933963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140b2a50 0x7fec14161380 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.109+0000 7fec1a4f6640 1 -- 172.21.15.59:0/1208933963 >> 172.21.15.59:0/1208933963 conn(0x7fec140b5590 msgr2=0x7fec140b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:00.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.109+0000 7fec1a4f6640 1 -- 172.21.15.59:0/1208933963 shutdown_connections 2024-06-29T15:39:00.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.109+0000 7fec1a4f6640 1 -- 172.21.15.59:0/1208933963 wait complete. 2024-06-29T15:39:00.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.111+0000 7fec1a4f6640 1 Processor -- start 2024-06-29T15:39:00.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.111+0000 7fec1a4f6640 1 -- start start 2024-06-29T15:39:00.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.112+0000 7fec1a4f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec140b2a50 0x7fec141f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:00.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.112+0000 7fec1a4f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec140b7150 0x7fec140c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:00.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec1a4f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140c8ec0 0x7fec140cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:00.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec1a4f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fec140b4da0 con 0x7fec140c8ec0 2024-06-29T15:39:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec1a4f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec140b4c20 con 0x7fec140b2a50 2024-06-29T15:39:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec1a4f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fec140b7560 con 0x7fec140b7150 2024-06-29T15:39:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140c8ec0 0x7fec140cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140c8ec0 0x7fec140cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59652/0 (socket says 172.21.15.59:59652) 2024-06-29T15:39:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 -- 172.21.15.59:0/100199157 learned_addr learned my addr 172.21.15.59:0/100199157 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec13fff640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec140b2a50 0x7fec141f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec137fe640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec140b7150 0x7fec140c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:00.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 -- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec140b7150 msgr2=0x7fec140c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec140b7150 0x7fec140c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 -- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec140b2a50 msgr2=0x7fec141f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec140b2a50 0x7fec141f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 -- 172.21.15.59:0/100199157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec141f7be0 con 0x7fec140c8ec0 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec18a93640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140c8ec0 0x7fec140cd030 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00b9e0 tx=0x7fec0c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec10ff9640 1 -- 172.21.15.59:0/100199157 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c003b70 con 0x7fec140c8ec0 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec10ff9640 1 -- 172.21.15.59:0/100199157 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec0c003e00 con 0x7fec140c8ec0 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec140cab50 con 0x7fec140c8ec0 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.113+0000 7fec10ff9640 1 -- 172.21.15.59:0/100199157 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c00cb20 con 0x7fec140c8ec0 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.114+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec140cb110 con 0x7fec140c8ec0 2024-06-29T15:39:00.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.114+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec140c9400 con 0x7fec140c8ec0 2024-06-29T15:39:00.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.115+0000 7fec10ff9640 1 -- 172.21.15.59:0/100199157 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 98) ==== 98247+0+0 (secure 0 0 0) 0x7fec0c00ccc0 con 0x7fec140c8ec0 2024-06-29T15:39:00.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.116+0000 7fec10ff9640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fec00076390 0x7fec000788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:00.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.116+0000 7fec10ff9640 1 -- 172.21.15.59:0/100199157 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 6903+0+0 (secure 0 0 0) 0x7fec0c0979e0 con 0x7fec140c8ec0 2024-06-29T15:39:00.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.116+0000 7fec13fff640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fec00076390 0x7fec000788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:00.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.117+0000 7fec13fff640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fec00076390 0x7fec000788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fec04033000 tx=0x7fec04004730 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:00.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.120+0000 7fec10ff9640 1 -- 172.21.15.59:0/100199157 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fec0c060850 con 0x7fec140c8ec0 2024-06-29T15:39:00.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.349+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fec1408ccd0 con 0x7fec140c8ec0 2024-06-29T15:39:00.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.972+0000 7fec10ff9640 1 -- 172.21.15.59:0/100199157 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v90) ==== 104+0+0 (secure 0 0 0) 0x7fec0c064b50 con 0x7fec140c8ec0 2024-06-29T15:39:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fec00076390 msgr2=0x7fec000788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fec00076390 0x7fec000788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fec04033000 tx=0x7fec04004730 comp rx=0 tx=0).stop 2024-06-29T15:39:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140c8ec0 msgr2=0x7fec140cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140c8ec0 0x7fec140cd030 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00b9e0 tx=0x7fec0c00bea0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 shutdown_connections 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fec00076390 0x7fec000788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec140c8ec0 0x7fec140cd030 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec140b7150 0x7fec140c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 --2- 172.21.15.59:0/100199157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec140b2a50 0x7fec141f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 >> 172.21.15.59:0/100199157 conn(0x7fec140b5590 msgr2=0x7fec1415e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 shutdown_connections 2024-06-29T15:39:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:00.975+0000 7fec1a4f6640 1 -- 172.21.15.59:0/100199157 wait complete. 2024-06-29T15:39:00.990 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:39:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.118+0000 7f3a72bcb640 1 -- 172.21.15.59:0/1785487561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c0745a0 msgr2=0x7f3a6c0b3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.118+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/1785487561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c0745a0 0x7f3a6c0b3be0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c071a30 tx=0x7f3a6402aac0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.118+0000 7f3a72bcb640 1 -- 172.21.15.59:0/1785487561 shutdown_connections 2024-06-29T15:39:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.118+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/1785487561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c0745a0 0x7f3a6c0b3be0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.118+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/1785487561 >> v2:172.21.15.153:3300/0 conn(0x7f3a6c1440a0 0x7f3a6c143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.118+0000 7f3a72bcb640 1 -- 172.21.15.59:0/1785487561 >> 172.21.15.59:0/1785487561 conn(0x7f3a6c0cd630 msgr2=0x7f3a6c0cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.118+0000 7f3a72bcb640 1 -- 172.21.15.59:0/1785487561 shutdown_connections 2024-06-29T15:39:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.118+0000 7f3a72bcb640 1 -- 172.21.15.59:0/1785487561 wait complete. 2024-06-29T15:39:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.120+0000 7f3a72bcb640 1 Processor -- start 2024-06-29T15:39:01.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.120+0000 7f3a72bcb640 1 -- start start 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a72bcb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1440a0 0x7f3a6c159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a72bcb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c15f1e0 0x7f3a6c0ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a72bcb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a6c0ad900 0x7f3a6c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a72bcb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6c0b44b0 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a72bcb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6c0b4330 con 0x7f3a6c1440a0 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a72bcb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3a6c0b4630 con 0x7f3a6c0ad900 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c15f1e0 0x7f3a6c0ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c15f1e0 0x7f3a6c0ad3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59676/0 (socket says 172.21.15.59:59676) 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.121+0000 7f3a6bfff640 1 -- 172.21.15.59:0/2633146813 learned_addr learned my addr 172.21.15.59:0/2633146813 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a71168640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a6c0ad900 0x7f3a6c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:01.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a70967640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1440a0 0x7f3a6c159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a6bfff640 1 -- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a6c0ad900 msgr2=0x7f3a6c15d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a6bfff640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a6c0ad900 0x7f3a6c15d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a6bfff640 1 -- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1440a0 msgr2=0x7f3a6c159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a6bfff640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1440a0 0x7f3a6c159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a6bfff640 1 -- 172.21.15.59:0/2633146813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a6c0af210 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a70967640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1440a0 0x7f3a6c159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a6bfff640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c15f1e0 0x7f3a6c0ad3c0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000c3c0 tx=0x7f3a6000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a697fa640 1 -- 172.21.15.59:0/2633146813 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60017070 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a697fa640 1 -- 172.21.15.59:0/2633146813 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a600035f0 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a697fa640 1 -- 172.21.15.59:0/2633146813 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60003a20 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a6c15a290 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.122+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a6c0ae450 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.123+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a6c13fcd0 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.124+0000 7f3a697fa640 1 -- 172.21.15.59:0/2633146813 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 98) ==== 98247+0+0 (secure 0 0 0) 0x7f3a6001a070 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.124+0000 7f3a697fa640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3a58076360 0x7f3a58078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:01.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.125+0000 7f3a697fa640 1 -- 172.21.15.59:0/2633146813 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 6903+0+0 (secure 0 0 0) 0x7f3a60096fa0 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.125+0000 7f3a70967640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3a58076360 0x7f3a58078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:01.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.125+0000 7f3a70967640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3a58076360 0x7f3a58078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00a870 tx=0x7f3a5c008040 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:01.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.129+0000 7f3a697fa640 1 -- 172.21.15.59:0/2633146813 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3a6c13fcd0 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.318+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3a6c0adf00 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.982+0000 7f3a697fa640 1 -- 172.21.15.59:0/2633146813 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v180) ==== 181+0+0 (secure 0 0 0) 0x7f3a60060460 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.984 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:39:01.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.989+0000 7f3a697fa640 1 -- 172.21.15.59:0/2633146813 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f3a6005bff0 con 0x7f3a6c15f1e0 2024-06-29T15:39:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.989+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3a58076360 msgr2=0x7f3a58078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.989+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3a58076360 0x7f3a58078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00a870 tx=0x7f3a5c008040 comp rx=0 tx=0).stop 2024-06-29T15:39:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.989+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c15f1e0 msgr2=0x7f3a6c0ad3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.989+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c15f1e0 0x7f3a6c0ad3c0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000c3c0 tx=0x7f3a6000c880 comp rx=0 tx=0).stop 2024-06-29T15:39:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.990+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 shutdown_connections 2024-06-29T15:39:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.990+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3a58076360 0x7f3a58078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.990+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a6c0ad900 0x7f3a6c15d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.990+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a6c15f1e0 0x7f3a6c0ad3c0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.990+0000 7f3a72bcb640 1 --2- 172.21.15.59:0/2633146813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1440a0 0x7f3a6c159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.990+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 >> 172.21.15.59:0/2633146813 conn(0x7f3a6c0cd630 msgr2=0x7f3a6c0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.990+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 shutdown_connections 2024-06-29T15:39:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:01.990+0000 7f3a72bcb640 1 -- 172.21.15.59:0/2633146813 wait complete. 2024-06-29T15:39:02.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:39:02.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.134+0000 7f62e43d2640 1 -- 172.21.15.59:0/976898834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62dc06f910 msgr2=0x7f62dc15eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.134+0000 7f62e43d2640 1 --2- 172.21.15.59:0/976898834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62dc06f910 0x7f62dc15eac0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f62cc005fd0 tx=0x7f62cc02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.134+0000 7f62e43d2640 1 -- 172.21.15.59:0/976898834 shutdown_connections 2024-06-29T15:39:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.134+0000 7f62e43d2640 1 --2- 172.21.15.59:0/976898834 >> v2:172.21.15.153:3300/0 conn(0x7f62dc15f000 0x7f62dc0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.134+0000 7f62e43d2640 1 --2- 172.21.15.59:0/976898834 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62dc06f910 0x7f62dc15eac0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.134+0000 7f62e43d2640 1 -- 172.21.15.59:0/976898834 >> 172.21.15.59:0/976898834 conn(0x7f62dc06d120 msgr2=0x7f62dc06d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.134+0000 7f62e43d2640 1 -- 172.21.15.59:0/976898834 shutdown_connections 2024-06-29T15:39:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.134+0000 7f62e43d2640 1 -- 172.21.15.59:0/976898834 wait complete. 2024-06-29T15:39:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.136+0000 7f62e43d2640 1 Processor -- start 2024-06-29T15:39:02.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.136+0000 7f62e43d2640 1 -- start start 2024-06-29T15:39:02.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.137+0000 7f62e43d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62dc13dc40 0x7f62dc1f8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:02.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.137+0000 7f62e43d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62dc15f000 0x7f62dc138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:02.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.137+0000 7f62e43d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62dc1394e0 0x7f62dc1ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:02.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.137+0000 7f62e43d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc0ced50 con 0x7f62dc13dc40 2024-06-29T15:39:02.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.137+0000 7f62e43d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc0cebd0 con 0x7f62dc1394e0 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.137+0000 7f62e43d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f62dc0ceed0 con 0x7f62dc15f000 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.137+0000 7f62e216e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62dc13dc40 0x7f62dc1f8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.137+0000 7f62e216e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62dc13dc40 0x7f62dc1f8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59686/0 (socket says 172.21.15.59:59686) 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e216e640 1 -- 172.21.15.59:0/3517680810 learned_addr learned my addr 172.21.15.59:0/3517680810 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e196d640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62dc15f000 0x7f62dc138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e296f640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62dc1394e0 0x7f62dc1ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e216e640 1 -- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62dc15f000 msgr2=0x7f62dc138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e216e640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62dc15f000 0x7f62dc138fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e216e640 1 -- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62dc1394e0 msgr2=0x7f62dc1ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e216e640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62dc1394e0 0x7f62dc1ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e216e640 1 -- 172.21.15.59:0/3517680810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc1f4cb0 con 0x7f62dc13dc40 2024-06-29T15:39:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e216e640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62dc13dc40 0x7f62dc1f8cb0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c760 tx=0x7f62d800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62d2ffd640 1 -- 172.21.15.59:0/3517680810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d80052d0 con 0x7f62dc13dc40 2024-06-29T15:39:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62d2ffd640 1 -- 172.21.15.59:0/3517680810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62d8005490 con 0x7f62dc13dc40 2024-06-29T15:39:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc15e190 con 0x7f62dc13dc40 2024-06-29T15:39:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62d2ffd640 1 -- 172.21.15.59:0/3517680810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d800a8a0 con 0x7f62dc13dc40 2024-06-29T15:39:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.138+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc1f51c0 con 0x7f62dc13dc40 2024-06-29T15:39:02.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.139+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc069800 con 0x7f62dc13dc40 2024-06-29T15:39:02.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.140+0000 7f62d2ffd640 1 -- 172.21.15.59:0/3517680810 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f62d800aa40 con 0x7f62dc13dc40 2024-06-29T15:39:02.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.141+0000 7f62d2ffd640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f62c0076420 0x7f62c0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.141+0000 7f62d2ffd640 1 -- 172.21.15.59:0/3517680810 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(180..180 src has 1..180) ==== 6465+0+0 (secure 0 0 0) 0x7f62d8097630 con 0x7f62dc13dc40 2024-06-29T15:39:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.141+0000 7f62e196d640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f62c0076420 0x7f62c0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.142+0000 7f62e196d640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f62c0076420 0x7f62c0078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f62cc033000 tx=0x7f62cc004730 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.146+0000 7f62d2ffd640 1 -- 172.21.15.59:0/3517680810 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f62dc15b210 con 0x7f62dc13dc40 2024-06-29T15:39:02.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.344+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f62dc15ecf0 con 0x7f62dc13dc40 2024-06-29T15:39:02.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.990+0000 7f62d2ffd640 1 -- 172.21.15.59:0/3517680810 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v181) ==== 169+0+0 (secure 0 0 0) 0x7f62d80604a0 con 0x7f62dc13dc40 2024-06-29T15:39:02.991 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.992+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f62c0076420 msgr2=0x7f62c0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.992+0000 7f62e43d2640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f62c0076420 0x7f62c0078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f62cc033000 tx=0x7f62cc004730 comp rx=0 tx=0).stop 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.992+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62dc13dc40 msgr2=0x7f62dc1f8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.992+0000 7f62e43d2640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62dc13dc40 0x7f62dc1f8cb0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c760 tx=0x7f62d800cc20 comp rx=0 tx=0).stop 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.993+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 shutdown_connections 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.993+0000 7f62e43d2640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f62c0076420 0x7f62c0078930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.993+0000 7f62e43d2640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62dc1394e0 0x7f62dc1ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.993+0000 7f62e43d2640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62dc15f000 0x7f62dc138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.993+0000 7f62e43d2640 1 --2- 172.21.15.59:0/3517680810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62dc13dc40 0x7f62dc1f8cb0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.993+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 >> 172.21.15.59:0/3517680810 conn(0x7f62dc06d120 msgr2=0x7f62dc15a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:02.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.993+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 shutdown_connections 2024-06-29T15:39:02.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:02.993+0000 7f62e43d2640 1 -- 172.21.15.59:0/3517680810 wait complete. 2024-06-29T15:39:03.011 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:39:03.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.136+0000 7fd4e9222640 1 -- 172.21.15.59:0/1693917228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc0051a0 msgr2=0x7fd4cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.136+0000 7fd4e9222640 1 --2- 172.21.15.59:0/1693917228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4cc0051a0 0x7fd4cc00b9c0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00c610 tx=0x7fd4cc005830 comp rx=0 tx=0).stop 2024-06-29T15:39:03.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.136+0000 7fd4e359e640 1 -- 172.21.15.59:0/1693917228 reap_dead start 2024-06-29T15:39:03.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.136+0000 7fd4e9222640 1 -- 172.21.15.59:0/1693917228 shutdown_connections 2024-06-29T15:39:03.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.136+0000 7fd4e9222640 1 -- 172.21.15.59:0/1693917228 >> 172.21.15.59:0/1693917228 conn(0x7fd4e40d3ee0 msgr2=0x7fd4e40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:03.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.136+0000 7fd4e9222640 1 -- 172.21.15.59:0/1693917228 shutdown_connections 2024-06-29T15:39:03.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.136+0000 7fd4e9222640 1 -- 172.21.15.59:0/1693917228 wait complete. 2024-06-29T15:39:03.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.138+0000 7fd4e9222640 1 Processor -- start 2024-06-29T15:39:03.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.138+0000 7fd4e9222640 1 -- start start 2024-06-29T15:39:03.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.139+0000 7fd4e9222640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4cc0051a0 0x7fd4e414ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.139+0000 7fd4e9222640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e415b360 0x7fd4e415b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.139+0000 7fd4e9222640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4e41f4d70 0x7fd4e41f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.139+0000 7fd4e9222640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e415e8d0 con 0x7fd4e415b360 2024-06-29T15:39:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.139+0000 7fd4e9222640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e40b42b0 con 0x7fd4e41f4d70 2024-06-29T15:39:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.139+0000 7fd4e9222640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e415ea50 con 0x7fd4cc0051a0 2024-06-29T15:39:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e415b360 0x7fd4e415b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e415b360 0x7fd4e415b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33808/0 (socket says 172.21.15.59:33808) 2024-06-29T15:39:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 -- 172.21.15.59:0/3049255056 learned_addr learned my addr 172.21.15.59:0/3049255056 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e359e640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4e41f4d70 0x7fd4e41f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e2d9d640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4cc0051a0 0x7fd4e414ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 -- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4cc0051a0 msgr2=0x7fd4e414ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4cc0051a0 0x7fd4e414ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 -- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4e41f4d70 msgr2=0x7fd4e41f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4e41f4d70 0x7fd4e41f5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 -- 172.21.15.59:0/3049255056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e40c83c0 con 0x7fd4e415b360 2024-06-29T15:39:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e259c640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e415b360 0x7fd4e415b770 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8011ae0 tx=0x7fd4d8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4cb7fe640 1 -- 172.21.15.59:0/3049255056 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d801e070 con 0x7fd4e415b360 2024-06-29T15:39:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4cb7fe640 1 -- 172.21.15.59:0/3049255056 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4d800fe70 con 0x7fd4e415b360 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e40c86a0 con 0x7fd4e415b360 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4cb7fe640 1 -- 172.21.15.59:0/3049255056 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8004290 con 0x7fd4e415b360 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.140+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e415ae40 con 0x7fd4e415b360 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.141+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4e415f9b0 con 0x7fd4e415b360 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.142+0000 7fd4cb7fe640 1 -- 172.21.15.59:0/3049255056 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fd4d8021070 con 0x7fd4e415b360 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.143+0000 7fd4cb7fe640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd4d00763f0 0x7fd4d0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.143+0000 7fd4cb7fe640 1 -- 172.21.15.59:0/3049255056 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7fd4d8098510 con 0x7fd4e415b360 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.143+0000 7fd4e2d9d640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd4d00763f0 0x7fd4d0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.143+0000 7fd4e2d9d640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd4d00763f0 0x7fd4d0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4000c00 tx=0x7fd4d4006b70 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:03.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.147+0000 7fd4cb7fe640 1 -- 172.21.15.59:0/3049255056 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd4d8061560 con 0x7fd4e415b360 2024-06-29T15:39:03.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.346+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fd4e40c8870 con 0x7fd4e415b360 2024-06-29T15:39:03.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.350+0000 7fd4cb7fe640 1 -- 172.21.15.59:0/3049255056 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v63) ==== 71+0+0 (secure 0 0 0) 0x7fd4d8065860 con 0x7fd4e415b360 2024-06-29T15:39:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd4d00763f0 msgr2=0x7fd4d0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd4d00763f0 0x7fd4d0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4000c00 tx=0x7fd4d4006b70 comp rx=0 tx=0).stop 2024-06-29T15:39:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e415b360 msgr2=0x7fd4e415b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e415b360 0x7fd4e415b770 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8011ae0 tx=0x7fd4d8009e90 comp rx=0 tx=0).stop 2024-06-29T15:39:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 shutdown_connections 2024-06-29T15:39:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd4d00763f0 0x7fd4d0078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4e41f4d70 0x7fd4e41f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4e415b360 0x7fd4e415b770 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 --2- 172.21.15.59:0/3049255056 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4cc0051a0 0x7fd4e414ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 >> 172.21.15.59:0/3049255056 conn(0x7fd4e40d3ee0 msgr2=0x7fd4e415e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:03.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 shutdown_connections 2024-06-29T15:39:03.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.352+0000 7fd4e9222640 1 -- 172.21.15.59:0/3049255056 wait complete. 2024-06-29T15:39:03.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-06-29T15:39:03.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.481+0000 7fa815df0640 1 -- 172.21.15.59:0/1563583909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7f8004d40 msgr2=0x7fa7f8005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.481+0000 7fa815df0640 1 --2- 172.21.15.59:0/1563583909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7f8004d40 0x7fa7f8005150 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fa8100b5090 tx=0x7fa7f8005830 comp rx=0 tx=0).stop 2024-06-29T15:39:03.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.481+0000 7fa80ffff640 1 -- 172.21.15.59:0/1563583909 reap_dead start 2024-06-29T15:39:03.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.481+0000 7fa815df0640 1 -- 172.21.15.59:0/1563583909 shutdown_connections 2024-06-29T15:39:03.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.481+0000 7fa815df0640 1 -- 172.21.15.59:0/1563583909 >> 172.21.15.59:0/1563583909 conn(0x7fa8100b24c0 msgr2=0x7fa8100cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:03.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.481+0000 7fa815df0640 1 -- 172.21.15.59:0/1563583909 shutdown_connections 2024-06-29T15:39:03.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.481+0000 7fa815df0640 1 -- 172.21.15.59:0/1563583909 wait complete. 2024-06-29T15:39:03.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.483+0000 7fa815df0640 1 Processor -- start 2024-06-29T15:39:03.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.483+0000 7fa815df0640 1 -- start start 2024-06-29T15:39:03.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.484+0000 7fa815df0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101f7330 0x7fa8100ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.484+0000 7fa815df0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8101f6640 0x7fa8101f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.484+0000 7fa815df0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa81015dbd0 0x7fa81015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.484+0000 7fa815df0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa810142f70 con 0x7fa81015dbd0 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.484+0000 7fa815df0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa810142df0 con 0x7fa8101f7330 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.484+0000 7fa815df0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa8100ce6e0 con 0x7fa8101f6640 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa81015dbd0 0x7fa81015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa81015dbd0 0x7fa81015dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33816/0 (socket says 172.21.15.59:33816) 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 -- 172.21.15.59:0/2157866052 learned_addr learned my addr 172.21.15.59:0/2157866052 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80f7fe640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101f7330 0x7fa8100ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80effd640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8101f6640 0x7fa8101f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 -- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8101f6640 msgr2=0x7fa8101f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8101f6640 0x7fa8101f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 -- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101f7330 msgr2=0x7fa8100ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101f7330 0x7fa8100ae850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 -- 172.21.15.59:0/2157866052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8101f6f90 con 0x7fa81015dbd0 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80f7fe640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101f7330 0x7fa8100ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa80ffff640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa81015dbd0 0x7fa81015dfe0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fa808009f90 tx=0x7fa808033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa7f7fff640 1 -- 172.21.15.59:0/2157866052 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808043070 con 0x7fa81015dbd0 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa7f7fff640 1 -- 172.21.15.59:0/2157866052 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa808033d10 con 0x7fa81015dbd0 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa7f7fff640 1 -- 172.21.15.59:0/2157866052 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808038040 con 0x7fa81015dbd0 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81013b820 con 0x7fa81015dbd0 2024-06-29T15:39:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.485+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81015d0f0 con 0x7fa81015dbd0 2024-06-29T15:39:03.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.487+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81006c680 con 0x7fa81015dbd0 2024-06-29T15:39:03.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.487+0000 7fa7f7fff640 1 -- 172.21.15.59:0/2157866052 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fa808003bd0 con 0x7fa81015dbd0 2024-06-29T15:39:03.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.488+0000 7fa7f7fff640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fa7fc0763f0 0x7fa7fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.488+0000 7fa7f7fff640 1 -- 172.21.15.59:0/2157866052 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7fa8080b7bc0 con 0x7fa81015dbd0 2024-06-29T15:39:03.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.488+0000 7fa80f7fe640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fa7fc0763f0 0x7fa7fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.489+0000 7fa80f7fe640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fa7fc0763f0 0x7fa7fc078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa81013c2b0 tx=0x7fa800005f90 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:03.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.492+0000 7fa7f7fff640 1 -- 172.21.15.59:0/2157866052 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa8101f6f90 con 0x7fa81015dbd0 2024-06-29T15:39:03.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.757+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) -- 0x7fa81013ec40 con 0x7fa81015dbd0 2024-06-29T15:39:03.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.991+0000 7fa7f7fff640 1 -- 172.21.15.59:0/2157866052 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2531) ==== 174+0+0 (secure 0 0 0) 0x7fa808089220 con 0x7fa81015dbd0 2024-06-29T15:39:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fa7fc0763f0 msgr2=0x7fa7fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fa7fc0763f0 0x7fa7fc078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa81013c2b0 tx=0x7fa800005f90 comp rx=0 tx=0).stop 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa81015dbd0 msgr2=0x7fa81015dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa81015dbd0 0x7fa81015dfe0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fa808009f90 tx=0x7fa808033ba0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 shutdown_connections 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fa7fc0763f0 0x7fa7fc078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa81015dbd0 0x7fa81015dfe0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8101f6640 0x7fa8101f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 --2- 172.21.15.59:0/2157866052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8101f7330 0x7fa8100ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 >> 172.21.15.59:0/2157866052 conn(0x7fa8100b24c0 msgr2=0x7fa81013e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 shutdown_connections 2024-06-29T15:39:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:03.994+0000 7fa815df0640 1 -- 172.21.15.59:0/2157866052 wait complete. 2024-06-29T15:39:04.007 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-06-29T15:39:04.007 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-06-29T15:39:04.007 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T15:39:04.008 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:04.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 -- 172.21.15.59:0/2626532302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 msgr2=0x7f33380b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 --2- 172.21.15.59:0/2626532302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 0x7f33380b3990 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f33380b3520 tx=0x7f332002a560 comp rx=0 tx=0).stop 2024-06-29T15:39:04.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 -- 172.21.15.59:0/2626532302 shutdown_connections 2024-06-29T15:39:04.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 --2- 172.21.15.59:0/2626532302 >> v2:172.21.15.153:3300/0 conn(0x7f33380b3ed0 0x7f33380d1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 --2- 172.21.15.59:0/2626532302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 0x7f33380b3990 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 --2- 172.21.15.59:0/2626532302 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f333815e290 0x7f33380ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 -- 172.21.15.59:0/2626532302 >> 172.21.15.59:0/2626532302 conn(0x7f33380d3ee0 msgr2=0x7f33380d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 -- 172.21.15.59:0/2626532302 shutdown_connections 2024-06-29T15:39:04.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.124+0000 7f333d738640 1 -- 172.21.15.59:0/2626532302 wait complete. 2024-06-29T15:39:04.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.126+0000 7f333d738640 1 Processor -- start 2024-06-29T15:39:04.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.126+0000 7f333d738640 1 -- start start 2024-06-29T15:39:04.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.127+0000 7f333d738640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33380b3ed0 0x7f333814df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.127+0000 7f333d738640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 0x7f33381569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.127+0000 7f333d738640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333815e290 0x7f3338158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.127+0000 7f333d738640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f33380d1fa0 con 0x7f33380ce650 2024-06-29T15:39:04.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.127+0000 7f333d738640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f33380d1e20 con 0x7f333815e290 2024-06-29T15:39:04.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.127+0000 7f333d738640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f33380d2120 con 0x7f33380b3ed0 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 0x7f33381569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 0x7f33381569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33834/0 (socket says 172.21.15.59:33834) 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 -- 172.21.15.59:0/2761558807 learned_addr learned my addr 172.21.15.59:0/2761558807 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f3336ffd640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33380b3ed0 0x7f333814df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f33377fe640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333815e290 0x7f3338158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 -- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33380b3ed0 msgr2=0x7f333814df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33380b3ed0 0x7f333814df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 -- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333815e290 msgr2=0x7f3338158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333815e290 0x7f3338158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 -- 172.21.15.59:0/2761558807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33380c8590 con 0x7f33380ce650 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332ffff640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 0x7f33381569c0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f3320034040 tx=0x7f33200039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332f7fe640 1 -- 172.21.15.59:0/2761558807 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332003e030 con 0x7f33380ce650 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332f7fe640 1 -- 172.21.15.59:0/2761558807 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33200047a0 con 0x7f33380ce650 2024-06-29T15:39:04.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f333814e4b0 con 0x7f33380ce650 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f332f7fe640 1 -- 172.21.15.59:0/2761558807 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3320035040 con 0x7f33380ce650 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.128+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3338157880 con 0x7f33380ce650 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.129+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33380b3990 con 0x7f33380ce650 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.130+0000 7f332f7fe640 1 -- 172.21.15.59:0/2761558807 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f3320041030 con 0x7f33380ce650 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.131+0000 7f332f7fe640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f33180763f0 0x7f3318078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.131+0000 7f332f7fe640 1 -- 172.21.15.59:0/2761558807 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7f33200b83d0 con 0x7f33380ce650 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.131+0000 7f3336ffd640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f33180763f0 0x7f3318078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.131+0000 7f3336ffd640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f33180763f0 0x7f3318078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3328005fd0 tx=0x7f332800a250 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.135+0000 7f332f7fe640 1 -- 172.21.15.59:0/2761558807 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f33200812d0 con 0x7f33380ce650 2024-06-29T15:39:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.327+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f333815b460 con 0x7f33380ce650 2024-06-29T15:39:04.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.327+0000 7f332f7fe640 1 -- 172.21.15.59:0/2761558807 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) ==== 74+0+14374 (secure 0 0 0) 0x7f33200855d0 con 0x7f33380ce650 2024-06-29T15:39:04.329 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:04.329 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":181,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:39:02.984036+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/1057565935":"2024-06-30T15:38:59.970731+0000","172.21.15.153:0/2302300861":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3385295272":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3035080074":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/3418701370":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/780112437":"2024-06-30T15:38:57.549343+0000","172.21.15.153:6833/770286644":"2024-06-30T15:38:57.549343+0000","172.21.15.153:6832/770286644":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/2597046756":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/623872135":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/2213101666":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4098785844":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/4234893873":"2024-06-30T15:38:50.826532+0000","172.21.15.59:6832/1057565935":"2024-06-30T15:38:59.970731+0000","172.21.15.153:6833/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:6832/2627795750":"2024-06-30T15:38:50.826532+0000","172.21.15.153:0/851260222":"2024-06-30T15:38:57.549343+0000","172.21.15.153:0/4064948250":"2024-06-30T15:38:50.826532+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f33180763f0 msgr2=0x7f3318078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f33180763f0 0x7f3318078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3328005fd0 tx=0x7f332800a250 comp rx=0 tx=0).stop 2024-06-29T15:39:04.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 msgr2=0x7f33381569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 0x7f33381569c0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f3320034040 tx=0x7f33200039e0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 shutdown_connections 2024-06-29T15:39:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f33180763f0 0x7f3318078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f333815e290 0x7f3338158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33380ce650 0x7f33381569c0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 --2- 172.21.15.59:0/2761558807 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33380b3ed0 0x7f333814df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 >> 172.21.15.59:0/2761558807 conn(0x7f33380d3ee0 msgr2=0x7f33380b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 shutdown_connections 2024-06-29T15:39:04.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.329+0000 7f333d738640 1 -- 172.21.15.59:0/2761558807 wait complete. 2024-06-29T15:39:04.342 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:39:04.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:39:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.461+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/4234960176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df808c190 msgr2=0x7f9df80cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.461+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/4234960176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df808c190 0x7f9df80cf080 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009900 tx=0x7f9df402a610 comp rx=0 tx=0).stop 2024-06-29T15:39:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.461+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/4234960176 shutdown_connections 2024-06-29T15:39:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.461+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/4234960176 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9df80ce950 0x7f9df80d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.461+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/4234960176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df808c190 0x7f9df80cf080 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.461+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/4234960176 >> 172.21.15.59:0/4234960176 conn(0x7f9df80b5610 msgr2=0x7f9df80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.461+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/4234960176 shutdown_connections 2024-06-29T15:39:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.462+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/4234960176 wait complete. 2024-06-29T15:39:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.463+0000 7f9dfeb3c640 1 Processor -- start 2024-06-29T15:39:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.463+0000 7f9dfeb3c640 1 -- start start 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfeb3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df808c190 0x7f9df8158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfeb3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9df80ce950 0x7f9df8155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfeb3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df80cf5c0 0x7f9df815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfeb3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9df80b7f60 con 0x7f9df80cf5c0 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfeb3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9df80b7de0 con 0x7f9df808c190 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfeb3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9df80b80e0 con 0x7f9df80ce950 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df80cf5c0 0x7f9df815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df80cf5c0 0x7f9df815a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33852/0 (socket says 172.21.15.59:33852) 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 -- 172.21.15.59:0/549618487 learned_addr learned my addr 172.21.15.59:0/549618487 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfc8d8640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df808c190 0x7f9df8158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9deffff640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9df80ce950 0x7f9df8155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 -- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9df80ce950 msgr2=0x7f9df8155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9df80ce950 0x7f9df8155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 -- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df808c190 msgr2=0x7f9df8158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df808c190 0x7f9df8158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 -- 172.21.15.59:0/549618487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df81f92d0 con 0x7f9df80cf5c0 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfc8d8640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df808c190 0x7f9df8158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.465+0000 7f9dfd0d9640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df80cf5c0 0x7f9df815a970 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f9df000ea70 tx=0x7f9df000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.466+0000 7f9ded7fa640 1 -- 172.21.15.59:0/549618487 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df000ce40 con 0x7f9df80cf5c0 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.466+0000 7f9ded7fa640 1 -- 172.21.15.59:0/549618487 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9df0004c10 con 0x7f9df80cf5c0 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.466+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df814bb60 con 0x7f9df80cf5c0 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.466+0000 7f9ded7fa640 1 -- 172.21.15.59:0/549618487 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0005020 con 0x7f9df80cf5c0 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.466+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df81f87f0 con 0x7f9df80cf5c0 2024-06-29T15:39:04.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.467+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df81485a0 con 0x7f9df80cf5c0 2024-06-29T15:39:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.468+0000 7f9ded7fa640 1 -- 172.21.15.59:0/549618487 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f9df00051c0 con 0x7f9df80cf5c0 2024-06-29T15:39:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.468+0000 7f9ded7fa640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f9de4076420 0x7f9de4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.468+0000 7f9ded7fa640 1 -- 172.21.15.59:0/549618487 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7f9df0098910 con 0x7f9df80cf5c0 2024-06-29T15:39:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.468+0000 7f9dfc8d8640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f9de4076420 0x7f9de4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.469+0000 7f9dfc8d8640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f9de4076420 0x7f9de4078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9df4033000 tx=0x7f9df4004730 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.473+0000 7f9ded7fa640 1 -- 172.21.15.59:0/549618487 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9df0061ef0 con 0x7f9df80cf5c0 2024-06-29T15:39:04.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.662+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9df80cf080 con 0x7f9df80cf5c0 2024-06-29T15:39:04.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.662+0000 7f9ded7fa640 1 -- 172.21.15.59:0/549618487 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v181) ==== 93+0+10 (secure 0 0 0) 0x7f9df00680c0 con 0x7f9df80cf5c0 2024-06-29T15:39:04.664 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f9de4076420 msgr2=0x7f9de4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f9de4076420 0x7f9de4078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9df4033000 tx=0x7f9df4004730 comp rx=0 tx=0).stop 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df80cf5c0 msgr2=0x7f9df815a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df80cf5c0 0x7f9df815a970 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f9df000ea70 tx=0x7f9df000ef30 comp rx=0 tx=0).stop 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 shutdown_connections 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f9de4076420 0x7f9de4078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9df80cf5c0 0x7f9df815a970 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9df80ce950 0x7f9df8155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 --2- 172.21.15.59:0/549618487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9df808c190 0x7f9df8158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 >> 172.21.15.59:0/549618487 conn(0x7f9df80b5610 msgr2=0x7f9df80c9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 shutdown_connections 2024-06-29T15:39:04.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.664+0000 7f9dfeb3c640 1 -- 172.21.15.59:0/549618487 wait complete. 2024-06-29T15:39:04.676 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-06-29T15:39:04.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 -- 172.21.15.59:0/789554864 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd015ceb0 msgr2=0x7f1bd00b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/789554864 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd015ceb0 0x7f1bd00b56d0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0073f30 tx=0x7f1bc002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 -- 172.21.15.59:0/789554864 shutdown_connections 2024-06-29T15:39:04.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/789554864 >> v2:172.21.15.153:3300/0 conn(0x7f1bd01414c0 0x7f1bd01418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/789554864 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bd013e9b0 0x7f1bd013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/789554864 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd015ceb0 0x7f1bd00b56d0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 -- 172.21.15.59:0/789554864 >> 172.21.15.59:0/789554864 conn(0x7f1bd00b6fb0 msgr2=0x7f1bd00b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 -- 172.21.15.59:0/789554864 shutdown_connections 2024-06-29T15:39:04.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.793+0000 7f1bd60c0640 1 -- 172.21.15.59:0/789554864 wait complete. 2024-06-29T15:39:04.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.795+0000 7f1bd60c0640 1 Processor -- start 2024-06-29T15:39:04.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.795+0000 7f1bd60c0640 1 -- start start 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.796+0000 7f1bd60c0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd013e9b0 0x7f1bd00c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bd60c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bd01414c0 0x7f1bd00c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bd60c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bd015ceb0 0x7f1bd00c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bd60c0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd00b28c0 con 0x7f1bd013e9b0 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bd60c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd00b2740 con 0x7f1bd01414c0 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bd60c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd00b2a40 con 0x7f1bd015ceb0 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd013e9b0 0x7f1bd00c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd013e9b0 0x7f1bd00c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33880/0 (socket says 172.21.15.59:33880) 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 -- 172.21.15.59:0/4015244052 learned_addr learned my addr 172.21.15.59:0/4015244052 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bceffd640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bd01414c0 0x7f1bd00c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcffff640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bd015ceb0 0x7f1bd00c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 -- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bd015ceb0 msgr2=0x7f1bd00c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bd015ceb0 0x7f1bd00c85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 -- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bd01414c0 msgr2=0x7f1bd00c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bd01414c0 0x7f1bd00c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 -- 172.21.15.59:0/4015244052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd00cb750 con 0x7f1bd013e9b0 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bceffd640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bd01414c0 0x7f1bd00c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bcf7fe640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd013e9b0 0x7f1bd00c9170 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00099c0 tx=0x7f1bc00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bb7fff640 1 -- 172.21.15.59:0/4015244052 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc003e030 con 0x7f1bd013e9b0 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bb7fff640 1 -- 172.21.15.59:0/4015244052 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bc0003db0 con 0x7f1bd013e9b0 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd00cba30 con 0x7f1bd013e9b0 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.797+0000 7f1bb7fff640 1 -- 172.21.15.59:0/4015244052 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0035040 con 0x7f1bd013e9b0 2024-06-29T15:39:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.798+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd00cbff0 con 0x7f1bd013e9b0 2024-06-29T15:39:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.798+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd00b2f60 con 0x7f1bd013e9b0 2024-06-29T15:39:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.799+0000 7f1bb7fff640 1 -- 172.21.15.59:0/4015244052 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f1bc0041030 con 0x7f1bd013e9b0 2024-06-29T15:39:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.800+0000 7f1bb7fff640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f1bbc076420 0x7f1bbc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.800+0000 7f1bb7fff640 1 -- 172.21.15.59:0/4015244052 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7f1bc00b85b0 con 0x7f1bd013e9b0 2024-06-29T15:39:04.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.800+0000 7f1bceffd640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f1bbc076420 0x7f1bbc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.801+0000 7f1bceffd640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f1bbc076420 0x7f1bbc078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1bd01f6290 tx=0x7f1bc4005f90 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:04.804+0000 7f1bb7fff640 1 -- 172.21.15.59:0/4015244052 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1bc0081a80 con 0x7f1bd013e9b0 2024-06-29T15:39:05.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:05.070+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) -- 0x7f1bd00714c0 con 0x7f1bd013e9b0 2024-06-29T15:39:05.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:05.998+0000 7f1bb7fff640 1 -- 172.21.15.59:0/4015244052 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2533) ==== 168+0+0 (secure 0 0 0) 0x7f1bc0081380 con 0x7f1bd013e9b0 2024-06-29T15:39:06.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f1bbc076420 msgr2=0x7f1bbc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f1bbc076420 0x7f1bbc078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1bd01f6290 tx=0x7f1bc4005f90 comp rx=0 tx=0).stop 2024-06-29T15:39:06.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd013e9b0 msgr2=0x7f1bd00c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd013e9b0 0x7f1bd00c9170 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00099c0 tx=0x7f1bc00038a0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 shutdown_connections 2024-06-29T15:39:06.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f1bbc076420 0x7f1bbc078930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bd015ceb0 0x7f1bd00c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bd01414c0 0x7f1bd00c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 --2- 172.21.15.59:0/4015244052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bd013e9b0 0x7f1bd00c9170 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 >> 172.21.15.59:0/4015244052 conn(0x7f1bd00b6fb0 msgr2=0x7f1bd00b3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 shutdown_connections 2024-06-29T15:39:06.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.001+0000 7f1bd60c0640 1 -- 172.21.15.59:0/4015244052 wait complete. 2024-06-29T15:39:06.016 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:39:06.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 -- 172.21.15.59:0/1228835360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0d23c0 msgr2=0x7fc82c0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 --2- 172.21.15.59:0/1228835360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0d23c0 0x7fc82c0b3990 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fc82c08c4f0 tx=0x7fc81c02aa30 comp rx=0 tx=0).stop 2024-06-29T15:39:06.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 -- 172.21.15.59:0/1228835360 shutdown_connections 2024-06-29T15:39:06.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 --2- 172.21.15.59:0/1228835360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc82c15ee10 0x7fc82c15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 --2- 172.21.15.59:0/1228835360 >> v2:172.21.15.153:3300/0 conn(0x7fc82c0b3ed0 0x7fc82c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 --2- 172.21.15.59:0/1228835360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0d23c0 0x7fc82c0b3990 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 -- 172.21.15.59:0/1228835360 >> 172.21.15.59:0/1228835360 conn(0x7fc82c0d3ee0 msgr2=0x7fc82c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 -- 172.21.15.59:0/1228835360 shutdown_connections 2024-06-29T15:39:06.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.139+0000 7fc832f03640 1 -- 172.21.15.59:0/1228835360 wait complete. 2024-06-29T15:39:06.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.141+0000 7fc832f03640 1 Processor -- start 2024-06-29T15:39:06.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.141+0000 7fc832f03640 1 -- start start 2024-06-29T15:39:06.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc832f03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0b3ed0 0x7fc82c1ea560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc832f03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc82c0d23c0 0x7fc82c15a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc832f03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc82c15ee10 0x7fc82c1e9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc832f03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c160070 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc832f03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c15fef0 con 0x7fc82c0d23c0 2024-06-29T15:39:06.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc832f03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc82c1601f0 con 0x7fc82c15ee10 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0b3ed0 0x7fc82c1ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0b3ed0 0x7fc82c1ea560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33896/0 (socket says 172.21.15.59:33896) 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 -- 172.21.15.59:0/1301888356 learned_addr learned my addr 172.21.15.59:0/1301888356 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc8314a0640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc82c15ee10 0x7fc82c1e9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc82bfff640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc82c0d23c0 0x7fc82c15a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 -- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc82c15ee10 msgr2=0x7fc82c1e9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc82c15ee10 0x7fc82c1e9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 -- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc82c0d23c0 msgr2=0x7fc82c15a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc82c0d23c0 0x7fc82c15a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 -- 172.21.15.59:0/1301888356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82c159100 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.143+0000 7fc830c9f640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0b3ed0 0x7fc82c1ea560 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009d60 tx=0x7fc81c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:06.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.144+0000 7fc8297fa640 1 -- 172.21.15.59:0/1301888356 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c033050 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.144+0000 7fc8297fa640 1 -- 172.21.15.59:0/1301888356 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc81c0044f0 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.144+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc82c1593e0 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.144+0000 7fc8297fa640 1 -- 172.21.15.59:0/1301888356 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c038450 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.144+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc82c1599a0 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.145+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82c06c9f0 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.145+0000 7fc8297fa640 1 -- 172.21.15.59:0/1301888356 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fc81c036070 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.146+0000 7fc8297fa640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc80c0763f0 0x7fc80c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.146+0000 7fc8297fa640 1 -- 172.21.15.59:0/1301888356 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7fc81c0b8f40 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.146+0000 7fc82bfff640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc80c0763f0 0x7fc80c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.147+0000 7fc82bfff640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc80c0763f0 0x7fc80c078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc81400a4d0 tx=0x7fc81400a460 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.150+0000 7fc8297fa640 1 -- 172.21.15.59:0/1301888356 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc82c06c9f0 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.343+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fc82c0b3990 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.343+0000 7fc8297fa640 1 -- 172.21.15.59:0/1301888356 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v63) ==== 59+0+1998 (secure 0 0 0) 0x7fc81c081e40 con 0x7fc82c0b3ed0 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T15:39:06.345 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T15:39:06.346 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T15:39:06.347 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T15:39:06.347 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:39:06.347 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc80c0763f0 msgr2=0x7fc80c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc80c0763f0 0x7fc80c078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc81400a4d0 tx=0x7fc81400a460 comp rx=0 tx=0).stop 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0b3ed0 msgr2=0x7fc82c1ea560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0b3ed0 0x7fc82c1ea560 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009d60 tx=0x7fc81c003f10 comp rx=0 tx=0).stop 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 shutdown_connections 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc80c0763f0 0x7fc80c078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc82c15ee10 0x7fc82c1e9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc82c0d23c0 0x7fc82c15a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 --2- 172.21.15.59:0/1301888356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc82c0b3ed0 0x7fc82c1ea560 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 >> 172.21.15.59:0/1301888356 conn(0x7fc82c0d3ee0 msgr2=0x7fc82c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 shutdown_connections 2024-06-29T15:39:06.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.346+0000 7fc832f03640 1 -- 172.21.15.59:0/1301888356 wait complete. 2024-06-29T15:39:06.358 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:39:06.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.478+0000 7f0269970640 1 -- 172.21.15.59:0/1639276238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500099b0 msgr2=0x7f025000bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.478+0000 7f0269970640 1 --2- 172.21.15.59:0/1639276238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500099b0 0x7f025000bdd0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f025c001dc0 tx=0x7f0250005830 comp rx=0 tx=0).stop 2024-06-29T15:39:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.478+0000 7f0269970640 1 -- 172.21.15.59:0/1639276238 shutdown_connections 2024-06-29T15:39:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.478+0000 7f0269970640 1 --2- 172.21.15.59:0/1639276238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500099b0 0x7f025000bdd0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.478+0000 7f0269970640 1 -- 172.21.15.59:0/1639276238 >> 172.21.15.59:0/1639276238 conn(0x7f02640b6fb0 msgr2=0x7f02640b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.478+0000 7f0269970640 1 -- 172.21.15.59:0/1639276238 shutdown_connections 2024-06-29T15:39:06.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.478+0000 7f0269970640 1 -- 172.21.15.59:0/1639276238 wait complete. 2024-06-29T15:39:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.480+0000 7f0269970640 1 Processor -- start 2024-06-29T15:39:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.480+0000 7f0269970640 1 -- start start 2024-06-29T15:39:06.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f0269970640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02500099b0 0x7f0264155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f0269970640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0264154530 0x7f02641f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f0269970640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02641f4a10 0x7f02641f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f0269970640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f026413edc0 con 0x7f02641f4a10 2024-06-29T15:39:06.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f0269970640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02640b5990 con 0x7f0264154530 2024-06-29T15:39:06.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f0269970640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f026413ef40 con 0x7f02500099b0 2024-06-29T15:39:06.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f02637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02641f4a10 0x7f02641f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f02637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02641f4a10 0x7f02641f4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33906/0 (socket says 172.21.15.59:33906) 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f02637fe640 1 -- 172.21.15.59:0/2630767651 learned_addr learned my addr 172.21.15.59:0/2630767651 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f025bfff640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0264154530 0x7f02641f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.481+0000 7f0262ffd640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02500099b0 0x7f0264155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f02637fe640 1 -- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02500099b0 msgr2=0x7f0264155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f02637fe640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02500099b0 0x7f0264155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f02637fe640 1 -- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0264154530 msgr2=0x7f02641f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f02637fe640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0264154530 0x7f02641f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f02637fe640 1 -- 172.21.15.59:0/2630767651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02641f5360 con 0x7f02641f4a10 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f02637fe640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02641f4a10 0x7f02641f4e20 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f025c001eb0 tx=0x7f025c006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f0260ff9640 1 -- 172.21.15.59:0/2630767651 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c003d80 con 0x7f02641f4a10 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f0260ff9640 1 -- 172.21.15.59:0/2630767651 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f025c005b00 con 0x7f02641f4a10 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0264155ca0 con 0x7f02641f4a10 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f0260ff9640 1 -- 172.21.15.59:0/2630767651 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c005f50 con 0x7f02641f4a10 2024-06-29T15:39:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.482+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02641f3cd0 con 0x7f02641f4a10 2024-06-29T15:39:06.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.483+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02640b5ba0 con 0x7f02641f4a10 2024-06-29T15:39:06.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.484+0000 7f0260ff9640 1 -- 172.21.15.59:0/2630767651 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f025c005030 con 0x7f02641f4a10 2024-06-29T15:39:06.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.485+0000 7f0260ff9640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f02440763f0 0x7f0244078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.485+0000 7f0260ff9640 1 -- 172.21.15.59:0/2630767651 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7f025c0b7550 con 0x7f02641f4a10 2024-06-29T15:39:06.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.485+0000 7f0262ffd640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f02440763f0 0x7f0244078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.485+0000 7f0262ffd640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f02440763f0 0x7f0244078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0254005fd0 tx=0x7f025400d360 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:06.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.489+0000 7f0260ff9640 1 -- 172.21.15.59:0/2630767651 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f025c080630 con 0x7f02641f4a10 2024-06-29T15:39:06.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.679+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f02640ce160 con 0x7f02641f4a10 2024-06-29T15:39:06.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.683+0000 7f0260ff9640 1 -- 172.21.15.59:0/2630767651 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) ==== 122+0+0 (secure 0 0 0) 0x7f025c084930 con 0x7f02641f4a10 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f02440763f0 msgr2=0x7f0244078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f02440763f0 0x7f0244078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0254005fd0 tx=0x7f025400d360 comp rx=0 tx=0).stop 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02641f4a10 msgr2=0x7f02641f4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02641f4a10 0x7f02641f4e20 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f025c001eb0 tx=0x7f025c006ca0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 shutdown_connections 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f02440763f0 0x7f0244078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02641f4a10 0x7f02641f4e20 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0264154530 0x7f02641f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 --2- 172.21.15.59:0/2630767651 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02500099b0 0x7f0264155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 >> 172.21.15.59:0/2630767651 conn(0x7f02640b6fb0 msgr2=0x7f026413fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 shutdown_connections 2024-06-29T15:39:06.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:06.685+0000 7f0269970640 1 -- 172.21.15.59:0/2630767651 wait complete. 2024-06-29T15:39:06.698 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:39:06.698 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:39:06.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:39:06.781 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:39:06.782 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:39:06.782 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:06.798 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:39:06.798 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:39:06.798 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:06.816 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:39:06.816 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:39:06.816 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:06.853 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:39:06.853 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:39:06.854 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:06.874 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:39:06.874 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:39:06.874 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:39:07.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.011+0000 7f2af909e640 1 -- 172.21.15.59:0/4072296679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af415e0a0 msgr2=0x7f2af4161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.011+0000 7f2af909e640 1 --2- 172.21.15.59:0/4072296679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af415e0a0 0x7f2af4161a80 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f2af408c200 tx=0x7f2adc02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.011+0000 7f2af909e640 1 -- 172.21.15.59:0/4072296679 shutdown_connections 2024-06-29T15:39:07.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.011+0000 7f2af909e640 1 --2- 172.21.15.59:0/4072296679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af415e0a0 0x7f2af4161a80 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.011+0000 7f2af909e640 1 --2- 172.21.15.59:0/4072296679 >> v2:172.21.15.153:3300/0 conn(0x7f2af4162770 0x7f2af40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.011+0000 7f2af909e640 1 -- 172.21.15.59:0/4072296679 >> 172.21.15.59:0/4072296679 conn(0x7f2af40d45b0 msgr2=0x7f2af40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:07.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.011+0000 7f2af909e640 1 -- 172.21.15.59:0/4072296679 shutdown_connections 2024-06-29T15:39:07.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.011+0000 7f2af909e640 1 -- 172.21.15.59:0/4072296679 wait complete. 2024-06-29T15:39:07.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.013+0000 7f2af909e640 1 Processor -- start 2024-06-29T15:39:07.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.013+0000 7f2af909e640 1 -- start start 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af909e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af415e0a0 0x7f2af41f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af909e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af4161fc0 0x7f2af41f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af909e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2af4162770 0x7f2af41f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af909e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2af40d2d80 con 0x7f2af4161fc0 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af909e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2af40b4b00 con 0x7f2af415e0a0 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af909e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2af40d2f00 con 0x7f2af4162770 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af4161fc0 0x7f2af41f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af4161fc0 0x7f2af41f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33920/0 (socket says 172.21.15.59:33920) 2024-06-29T15:39:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.014+0000 7f2af259c640 1 -- 172.21.15.59:0/1574409734 learned_addr learned my addr 172.21.15.59:0/1574409734 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af2d9d640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af415e0a0 0x7f2af41f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af359e640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2af4162770 0x7f2af41f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af259c640 1 -- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2af4162770 msgr2=0x7f2af41f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af259c640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2af4162770 0x7f2af41f84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af259c640 1 -- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af415e0a0 msgr2=0x7f2af41f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af259c640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af415e0a0 0x7f2af41f1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af259c640 1 -- 172.21.15.59:0/1574409734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af40c8d30 con 0x7f2af4161fc0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af259c640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af4161fc0 0x7f2af41f7f80 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f2adc02aa90 tx=0x7f2adc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2aeb7fe640 1 -- 172.21.15.59:0/1574409734 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc005c90 con 0x7f2af4161fc0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2aeb7fe640 1 -- 172.21.15.59:0/1574409734 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2adc031e50 con 0x7f2af4161fc0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2aeb7fe640 1 -- 172.21.15.59:0/1574409734 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc005240 con 0x7f2af4161fc0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af40c9010 con 0x7f2af4161fc0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.015+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af40c95d0 con 0x7f2af4161fc0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.016+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2af4161ba0 con 0x7f2af4161fc0 2024-06-29T15:39:07.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.017+0000 7f2aeb7fe640 1 -- 172.21.15.59:0/1574409734 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f2adc0053e0 con 0x7f2af4161fc0 2024-06-29T15:39:07.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.018+0000 7f2aeb7fe640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f2ad40763f0 0x7f2ad4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.018+0000 7f2aeb7fe640 1 -- 172.21.15.59:0/1574409734 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7f2adc0b8a40 con 0x7f2af4161fc0 2024-06-29T15:39:07.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.018+0000 7f2af2d9d640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f2ad40763f0 0x7f2ad4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.019+0000 7f2af2d9d640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f2ad40763f0 0x7f2ad4078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4005fd0 tx=0x7f2ae4009500 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:07.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.022+0000 7f2aeb7fe640 1 -- 172.21.15.59:0/1574409734 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2adc03f030 con 0x7f2af4161fc0 2024-06-29T15:39:07.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.250+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2af40d3420 con 0x7f2af4161fc0 2024-06-29T15:39:07.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.251+0000 7f2aeb7fe640 1 -- 172.21.15.59:0/1574409734 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+3915 (secure 0 0 0) 0x7f2adc0b8e20 con 0x7f2af4161fc0 2024-06-29T15:39:07.252 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:07.253 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":90,"btime":"2024-06-29T15:39:00:972455+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8207,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1947749735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1947749735},{"type":"v1","addr":"172.21.15.59:6835","nonce":1947749735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3010048644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3010048644},{"type":"v1","addr":"172.21.15.153:6836","nonce":3010048644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":9283,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1912783986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1912783986},{"type":"v1","addr":"172.21.15.153:6837","nonce":1912783986}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":84},{"gid":10317,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/577314525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":577314525},{"type":"v1","addr":"172.21.15.59:6834","nonce":577314525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[]} 2024-06-29T15:39:07.253 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 90 2024-06-29T15:39:07.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f2ad40763f0 msgr2=0x7f2ad4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f2ad40763f0 0x7f2ad4078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4005fd0 tx=0x7f2ae4009500 comp rx=0 tx=0).stop 2024-06-29T15:39:07.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af4161fc0 msgr2=0x7f2af41f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af4161fc0 0x7f2af41f7f80 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f2adc02aa90 tx=0x7f2adc033c20 comp rx=0 tx=0).stop 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 shutdown_connections 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f2ad40763f0 0x7f2ad4078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2af4162770 0x7f2af41f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2af4161fc0 0x7f2af41f7f80 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 --2- 172.21.15.59:0/1574409734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2af415e0a0 0x7f2af41f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 >> 172.21.15.59:0/1574409734 conn(0x7f2af40d45b0 msgr2=0x7f2af40b7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 shutdown_connections 2024-06-29T15:39:07.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.253+0000 7f2af909e640 1 -- 172.21.15.59:0/1574409734 wait complete. 2024-06-29T15:39:07.265 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:39:07.266 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:39:07.266 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:39:07.266 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:39:07.266 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:39:07.266 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:39:07.267 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:39:07.267 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:39:07.267 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:39:07.267 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:39:07.267 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:39:07.267 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:39:07.267 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:39:07.267 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:39:07.268 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:39:07.268 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:39:07.279 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:39:07.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:39:07.280 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:39:07.280 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:39:07.281 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:39:07.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:39:07.282 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:39:07.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:39:07.321 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:39:07.322 DEBUG:teuthology.parallel:result is None 2024-06-29T15:39:07.322 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:39:07.322 DEBUG:teuthology.parallel:result is None 2024-06-29T15:39:07.325 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:39:07.325 DEBUG:teuthology.parallel:result is None 2024-06-29T15:39:07.325 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:39:07.326 DEBUG:teuthology.parallel:result is None 2024-06-29T15:39:07.326 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:39:07.413 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:39:07.413 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:39:07.421 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T15:39:07.422 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:39:07.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.450+0000 7f34b3743640 1 -- 172.21.15.59:0/3440311139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34ac0b2fa0 msgr2=0x7f34ac0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.450+0000 7f34b3743640 1 --2- 172.21.15.59:0/3440311139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34ac0b2fa0 0x7f34ac0d1520 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f349c009830 tx=0x7f349c02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.450+0000 7f34b3743640 1 -- 172.21.15.59:0/3440311139 shutdown_connections 2024-06-29T15:39:07.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.450+0000 7f34b3743640 1 --2- 172.21.15.59:0/3440311139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34ac0b2fa0 0x7f34ac0d1520 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.450+0000 7f34b3743640 1 -- 172.21.15.59:0/3440311139 >> 172.21.15.59:0/3440311139 conn(0x7f34ac073e10 msgr2=0x7f34ac074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:07.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.451+0000 7f34b3743640 1 -- 172.21.15.59:0/3440311139 shutdown_connections 2024-06-29T15:39:07.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.451+0000 7f34b3743640 1 -- 172.21.15.59:0/3440311139 wait complete. 2024-06-29T15:39:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.452+0000 7f34b3743640 1 Processor -- start 2024-06-29T15:39:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.452+0000 7f34b3743640 1 -- start start 2024-06-29T15:39:07.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b3743640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac0b2270 0x7f34ac1536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b3743640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34ac0b2fa0 0x7f34ac159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b3743640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34ac0d1a60 0x7f34ac159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b3743640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac0b65d0 con 0x7f34ac0b2270 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b3743640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac0b6450 con 0x7f34ac0b2fa0 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b3743640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f34ac0b6750 con 0x7f34ac0d1a60 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac0b2270 0x7f34ac1536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac0b2270 0x7f34ac1536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33958/0 (socket says 172.21.15.59:33958) 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 -- 172.21.15.59:0/694921847 learned_addr learned my addr 172.21.15.59:0/694921847 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 -- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34ac0d1a60 msgr2=0x7f34ac159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34ac0d1a60 0x7f34ac159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 -- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34ac0b2fa0 msgr2=0x7f34ac159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34ac0b2fa0 0x7f34ac159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 -- 172.21.15.59:0/694921847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac0abb20 con 0x7f34ac0b2270 2024-06-29T15:39:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b14df640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac0b2270 0x7f34ac1536d0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f34a800c430 tx=0x7f34a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:07.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34a1ffb640 1 -- 172.21.15.59:0/694921847 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8011790 con 0x7f34ac0b2270 2024-06-29T15:39:07.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34a1ffb640 1 -- 172.21.15.59:0/694921847 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34a8017070 con 0x7f34ac0b2270 2024-06-29T15:39:07.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ac0abe00 con 0x7f34ac0b2270 2024-06-29T15:39:07.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.454+0000 7f34a1ffb640 1 -- 172.21.15.59:0/694921847 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8011ce0 con 0x7f34ac0b2270 2024-06-29T15:39:07.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.455+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ac0aa600 con 0x7f34ac0b2270 2024-06-29T15:39:07.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.456+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34ac0b4cc0 con 0x7f34ac0b2270 2024-06-29T15:39:07.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.456+0000 7f34a1ffb640 1 -- 172.21.15.59:0/694921847 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f34a801c460 con 0x7f34ac0b2270 2024-06-29T15:39:07.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.457+0000 7f34a1ffb640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3490076420 0x7f3490078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.457+0000 7f34a1ffb640 1 -- 172.21.15.59:0/694921847 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6051+0+0 (secure 0 0 0) 0x7f34a8097910 con 0x7f34ac0b2270 2024-06-29T15:39:07.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.457+0000 7f34b0cde640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3490076420 0x7f3490078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.458+0000 7f34b0cde640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3490076420 0x7f3490078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f349c034000 tx=0x7f349c0046c0 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:07.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.461+0000 7f34a1ffb640 1 -- 172.21.15.59:0/694921847 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f34ac0b4cc0 con 0x7f34ac0b2270 2024-06-29T15:39:07.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:07.665+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f34ac0b2d60 con 0x7f34ac0b2270 2024-06-29T15:39:08.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.014+0000 7f34a1ffb640 1 -- 172.21.15.59:0/694921847 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v183) ==== 97+0+0 (secure 0 0 0) 0x7f34ac0b2d60 con 0x7f34ac0b2270 2024-06-29T15:39:08.016 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T15:39:08.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3490076420 msgr2=0x7f3490078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3490076420 0x7f3490078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f349c034000 tx=0x7f349c0046c0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac0b2270 msgr2=0x7f34ac1536d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac0b2270 0x7f34ac1536d0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f34a800c430 tx=0x7f34a800c8f0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 shutdown_connections 2024-06-29T15:39:08.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3490076420 0x7f3490078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34ac0d1a60 0x7f34ac159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34ac0b2fa0 0x7f34ac159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 --2- 172.21.15.59:0/694921847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac0b2270 0x7f34ac1536d0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 >> 172.21.15.59:0/694921847 conn(0x7f34ac073e10 msgr2=0x7f34ac06fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 shutdown_connections 2024-06-29T15:39:08.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.017+0000 7f34b3743640 1 -- 172.21.15.59:0/694921847 wait complete. 2024-06-29T15:39:08.031 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:39:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.150+0000 7fba3e719640 1 -- 172.21.15.59:0/2861887584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3813e9b0 msgr2=0x7fba3813edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.150+0000 7fba3e719640 1 --2- 172.21.15.59:0/2861887584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3813e9b0 0x7fba3813edc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fba20009830 tx=0x7fba2002a530 comp rx=0 tx=0).stop 2024-06-29T15:39:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.150+0000 7fba3e719640 1 -- 172.21.15.59:0/2861887584 shutdown_connections 2024-06-29T15:39:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.150+0000 7fba3e719640 1 --2- 172.21.15.59:0/2861887584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3813e9b0 0x7fba3813edc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.150+0000 7fba3e719640 1 -- 172.21.15.59:0/2861887584 >> 172.21.15.59:0/2861887584 conn(0x7fba380b6fb0 msgr2=0x7fba380b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.150+0000 7fba3e719640 1 -- 172.21.15.59:0/2861887584 shutdown_connections 2024-06-29T15:39:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.150+0000 7fba3e719640 1 -- 172.21.15.59:0/2861887584 wait complete. 2024-06-29T15:39:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.151+0000 7fba3e719640 1 Processor -- start 2024-06-29T15:39:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.151+0000 7fba3e719640 1 -- start start 2024-06-29T15:39:08.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba3e719640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba3813e9b0 0x7fba380cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba3e719640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba381414c0 0x7fba380c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba3e719640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3815ceb0 0x7fba380ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba3e719640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fba380b2900 con 0x7fba3813e9b0 2024-06-29T15:39:08.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba3e719640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba380b2780 con 0x7fba3815ceb0 2024-06-29T15:39:08.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba3e719640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fba380b2a80 con 0x7fba381414c0 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba3813e9b0 0x7fba380cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba3813e9b0 0x7fba380cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33972/0 (socket says 172.21.15.59:33972) 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 -- 172.21.15.59:0/1562228327 learned_addr learned my addr 172.21.15.59:0/1562228327 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba2ffff640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba381414c0 0x7fba380c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba3ccb6640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3815ceb0 0x7fba380ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 -- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba381414c0 msgr2=0x7fba380c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba381414c0 0x7fba380c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 -- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3815ceb0 msgr2=0x7fba380ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3815ceb0 0x7fba380ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 -- 172.21.15.59:0/1562228327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba38153800 con 0x7fba3813e9b0 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba3ccb6640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3815ceb0 0x7fba380ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.153+0000 7fba37fff640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba3813e9b0 0x7fba380cc730 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c3c0 tx=0x7fba2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.154+0000 7fba357fa640 1 -- 172.21.15.59:0/1562228327 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28017070 con 0x7fba3813e9b0 2024-06-29T15:39:08.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.154+0000 7fba357fa640 1 -- 172.21.15.59:0/1562228327 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba280035f0 con 0x7fba3813e9b0 2024-06-29T15:39:08.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.154+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba380cd310 con 0x7fba3813e9b0 2024-06-29T15:39:08.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.154+0000 7fba357fa640 1 -- 172.21.15.59:0/1562228327 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28003a80 con 0x7fba3813e9b0 2024-06-29T15:39:08.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.154+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba380cd8d0 con 0x7fba3813e9b0 2024-06-29T15:39:08.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.155+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba380b2f30 con 0x7fba3813e9b0 2024-06-29T15:39:08.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.155+0000 7fba357fa640 1 -- 172.21.15.59:0/1562228327 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fba2801a070 con 0x7fba3813e9b0 2024-06-29T15:39:08.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.156+0000 7fba357fa640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fba180763f0 0x7fba18078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.156+0000 7fba357fa640 1 -- 172.21.15.59:0/1562228327 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 5277+0+0 (secure 0 0 0) 0x7fba280969f0 con 0x7fba3813e9b0 2024-06-29T15:39:08.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.156+0000 7fba2ffff640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fba180763f0 0x7fba18078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.157+0000 7fba2ffff640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fba180763f0 0x7fba18078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba20034000 tx=0x7fba20004730 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.160+0000 7fba357fa640 1 -- 172.21.15.59:0/1562228327 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fba28060480 con 0x7fba3813e9b0 2024-06-29T15:39:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.422+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fba38140530 con 0x7fba3813e9b0 2024-06-29T15:39:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.422+0000 7fba357fa640 1 -- 172.21.15.59:0/1562228327 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v43) ==== 80+0+5304 (secure 0 0 0) 0x7fba2805fd80 con 0x7fba3813e9b0 2024-06-29T15:39:08.423 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.424 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.425 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:39:08.426 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.427 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T15:39:08.428 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.429 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T15:39:08.430 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.431 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T15:39:08.432 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:39:08.433 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.434 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T15:39:08.435 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.436 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.437 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:08.438 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T15:39:08.438 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:39:08.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fba180763f0 msgr2=0x7fba18078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fba180763f0 0x7fba18078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba20034000 tx=0x7fba20004730 comp rx=0 tx=0).stop 2024-06-29T15:39:08.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba3813e9b0 msgr2=0x7fba380cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba3813e9b0 0x7fba380cc730 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c3c0 tx=0x7fba2800c880 comp rx=0 tx=0).stop 2024-06-29T15:39:08.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 shutdown_connections 2024-06-29T15:39:08.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fba180763f0 0x7fba18078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba3815ceb0 0x7fba380ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba381414c0 0x7fba380c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 --2- 172.21.15.59:0/1562228327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba3813e9b0 0x7fba380cc730 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 >> 172.21.15.59:0/1562228327 conn(0x7fba380b6fb0 msgr2=0x7fba3813e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 shutdown_connections 2024-06-29T15:39:08.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.424+0000 7fba3e719640 1 -- 172.21.15.59:0/1562228327 wait complete. 2024-06-29T15:39:08.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:08.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.558+0000 7fcc3f654640 1 -- 172.21.15.59:0/2437145633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc24003a70 msgr2=0x7fcc24003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.558+0000 7fcc3f654640 1 --2- 172.21.15.59:0/2437145633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc24003a70 0x7fcc24003ee0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400c290 tx=0x7fcc24005830 comp rx=0 tx=0).stop 2024-06-29T15:39:08.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.558+0000 7fcc3dbf1640 1 -- 172.21.15.59:0/2437145633 reap_dead start 2024-06-29T15:39:08.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.558+0000 7fcc3f654640 1 -- 172.21.15.59:0/2437145633 shutdown_connections 2024-06-29T15:39:08.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.558+0000 7fcc3f654640 1 -- 172.21.15.59:0/2437145633 >> 172.21.15.59:0/2437145633 conn(0x7fcc3806c950 msgr2=0x7fcc38073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.558+0000 7fcc3f654640 1 -- 172.21.15.59:0/2437145633 shutdown_connections 2024-06-29T15:39:08.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.558+0000 7fcc3f654640 1 -- 172.21.15.59:0/2437145633 wait complete. 2024-06-29T15:39:08.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.560+0000 7fcc3f654640 1 Processor -- start 2024-06-29T15:39:08.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.560+0000 7fcc3f654640 1 -- start start 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.561+0000 7fcc3f654640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24003a70 0x7fcc381440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.561+0000 7fcc3f654640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc381f27d0 0x7fcc381445e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.561+0000 7fcc3f654640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3815b290 0x7fcc3813d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.561+0000 7fcc3f654640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcc380b5c10 con 0x7fcc3815b290 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.561+0000 7fcc3f654640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc380b5a90 con 0x7fcc381f27d0 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.561+0000 7fcc3f654640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcc380b5d90 con 0x7fcc24003a70 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.561+0000 7fcc3dbf1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3815b290 0x7fcc3813d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3dbf1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3815b290 0x7fcc3813d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33980/0 (socket says 172.21.15.59:33980) 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3dbf1640 1 -- 172.21.15.59:0/4182277370 learned_addr learned my addr 172.21.15.59:0/4182277370 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3cbef640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc381f27d0 0x7fcc381445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3d3f0640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24003a70 0x7fcc381440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3dbf1640 1 -- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24003a70 msgr2=0x7fcc381440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3dbf1640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24003a70 0x7fcc381440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3dbf1640 1 -- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc381f27d0 msgr2=0x7fcc381445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3dbf1640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc381f27d0 0x7fcc381445e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3dbf1640 1 -- 172.21.15.59:0/4182277370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3813b7c0 con 0x7fcc3815b290 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3dbf1640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3815b290 0x7fcc3813d7f0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fcc380b5030 tx=0x7fcc30011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc2dffb640 1 -- 172.21.15.59:0/4182277370 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3000fce0 con 0x7fcc3815b290 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc2dffb640 1 -- 172.21.15.59:0/4182277370 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc3000fe80 con 0x7fcc3815b290 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc2dffb640 1 -- 172.21.15.59:0/4182277370 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30004d60 con 0x7fcc3815b290 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.562+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3814c1d0 con 0x7fcc3815b290 2024-06-29T15:39:08.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.563+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc3814f080 con 0x7fcc3815b290 2024-06-29T15:39:08.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.564+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc381458a0 con 0x7fcc3815b290 2024-06-29T15:39:08.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.564+0000 7fcc2dffb640 1 -- 172.21.15.59:0/4182277370 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fcc30020020 con 0x7fcc3815b290 2024-06-29T15:39:08.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.565+0000 7fcc2dffb640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcc1c0763f0 0x7fcc1c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.565+0000 7fcc2dffb640 1 -- 172.21.15.59:0/4182277370 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 5277+0+0 (secure 0 0 0) 0x7fcc30097db0 con 0x7fcc3815b290 2024-06-29T15:39:08.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.565+0000 7fcc3d3f0640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcc1c0763f0 0x7fcc1c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.566+0000 7fcc3d3f0640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcc1c0763f0 0x7fcc1c078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcc34033000 tx=0x7fcc3402d000 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.569+0000 7fcc2dffb640 1 -- 172.21.15.59:0/4182277370 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc30061610 con 0x7fcc3815b290 2024-06-29T15:39:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.766+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc380b1f10 con 0x7fcc3815b290 2024-06-29T15:39:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.767+0000 7fcc2dffb640 1 -- 172.21.15.59:0/4182277370 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) ==== 74+0+13230 (secure 0 0 0) 0x7fcc30060f10 con 0x7fcc3815b290 2024-06-29T15:39:08.768 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:08.768 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":185,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:39:08.025779+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcc1c0763f0 msgr2=0x7fcc1c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcc1c0763f0 0x7fcc1c078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcc34033000 tx=0x7fcc3402d000 comp rx=0 tx=0).stop 2024-06-29T15:39:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3815b290 msgr2=0x7fcc3813d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3815b290 0x7fcc3813d7f0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fcc380b5030 tx=0x7fcc30011f90 comp rx=0 tx=0).stop 2024-06-29T15:39:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 shutdown_connections 2024-06-29T15:39:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcc1c0763f0 0x7fcc1c078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3815b290 0x7fcc3813d7f0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc381f27d0 0x7fcc381445e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 --2- 172.21.15.59:0/4182277370 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24003a70 0x7fcc381440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 >> 172.21.15.59:0/4182277370 conn(0x7fcc3806c950 msgr2=0x7fcc380cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 shutdown_connections 2024-06-29T15:39:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.769+0000 7fcc3f654640 1 -- 172.21.15.59:0/4182277370 wait complete. 2024-06-29T15:39:08.781 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:39:08.782 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:39:08.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.902+0000 7f4a73931640 1 -- 172.21.15.59:0/3878045043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0b3b50 msgr2=0x7f4a6c0b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.902+0000 7f4a73931640 1 --2- 172.21.15.59:0/3878045043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0b3b50 0x7f4a6c0b5880 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c074b10 tx=0x7f4a5c02a5f0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.902+0000 7f4a73931640 1 -- 172.21.15.59:0/3878045043 shutdown_connections 2024-06-29T15:39:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.902+0000 7f4a73931640 1 --2- 172.21.15.59:0/3878045043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0b3b50 0x7f4a6c0b5880 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.902+0000 7f4a73931640 1 --2- 172.21.15.59:0/3878045043 >> v2:172.21.15.153:3300/0 conn(0x7f4a6c1458a0 0x7f4a6c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.902+0000 7f4a73931640 1 -- 172.21.15.59:0/3878045043 >> 172.21.15.59:0/3878045043 conn(0x7f4a6c06c950 msgr2=0x7f4a6c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.902+0000 7f4a73931640 1 -- 172.21.15.59:0/3878045043 shutdown_connections 2024-06-29T15:39:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.902+0000 7f4a73931640 1 -- 172.21.15.59:0/3878045043 wait complete. 2024-06-29T15:39:08.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.904+0000 7f4a73931640 1 Processor -- start 2024-06-29T15:39:08.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.904+0000 7f4a73931640 1 -- start start 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.905+0000 7f4a73931640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c0b3740 0x7f4a6c06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.905+0000 7f4a73931640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a6c0b3b50 0x7f4a6c0b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.905+0000 7f4a73931640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c1408e0 0x7f4a6c1e6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.905+0000 7f4a73931640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c0b6150 con 0x7f4a6c1408e0 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.905+0000 7f4a73931640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c0b5fd0 con 0x7f4a6c0b3740 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.905+0000 7f4a73931640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4a6c0b62d0 con 0x7f4a6c0b3b50 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.905+0000 7f4a71ece640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c1408e0 0x7f4a6c1e6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a71ece640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c1408e0 0x7f4a6c1e6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33986/0 (socket says 172.21.15.59:33986) 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a71ece640 1 -- 172.21.15.59:0/2534986759 learned_addr learned my addr 172.21.15.59:0/2534986759 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a716cd640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c0b3740 0x7f4a6c06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a70ecc640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a6c0b3b50 0x7f4a6c0b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a71ece640 1 -- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a6c0b3b50 msgr2=0x7f4a6c0b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a71ece640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a6c0b3b50 0x7f4a6c0b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a71ece640 1 -- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c0b3740 msgr2=0x7f4a6c06f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a71ece640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c0b3740 0x7f4a6c06f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a71ece640 1 -- 172.21.15.59:0/2534986759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6c06e140 con 0x7f4a6c1408e0 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a71ece640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c1408e0 0x7f4a6c1e6370 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c034040 tx=0x7f4a5c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a69ffb640 1 -- 172.21.15.59:0/2534986759 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c03e030 con 0x7f4a6c1408e0 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a69ffb640 1 -- 172.21.15.59:0/2534986759 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a5c003dd0 con 0x7f4a6c1408e0 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a6c06e420 con 0x7f4a6c1408e0 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a69ffb640 1 -- 172.21.15.59:0/2534986759 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c035040 con 0x7f4a6c1408e0 2024-06-29T15:39:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.906+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a6c06e9e0 con 0x7f4a6c1408e0 2024-06-29T15:39:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.908+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a6c1458a0 con 0x7f4a6c1408e0 2024-06-29T15:39:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.908+0000 7f4a69ffb640 1 -- 172.21.15.59:0/2534986759 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f4a5c041030 con 0x7f4a6c1408e0 2024-06-29T15:39:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.909+0000 7f4a69ffb640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f4a58076420 0x7f4a58078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.909+0000 7f4a69ffb640 1 -- 172.21.15.59:0/2534986759 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 5277+0+0 (secure 0 0 0) 0x7f4a5c0b7e30 con 0x7f4a6c1408e0 2024-06-29T15:39:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.909+0000 7f4a716cd640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f4a58076420 0x7f4a58078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.910+0000 7f4a716cd640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f4a58076420 0x7f4a58078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4a64005fd0 tx=0x7f4a64009500 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:08.913+0000 7f4a69ffb640 1 -- 172.21.15.59:0/2534986759 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4a5c081030 con 0x7f4a6c1408e0 2024-06-29T15:39:09.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.106+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a6c15be10 con 0x7f4a6c1408e0 2024-06-29T15:39:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.107+0000 7f4a69ffb640 1 -- 172.21.15.59:0/2534986759 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v185) ==== 93+0+10 (secure 0 0 0) 0x7f4a5c085330 con 0x7f4a6c1408e0 2024-06-29T15:39:09.108 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:39:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f4a58076420 msgr2=0x7f4a58078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f4a58076420 0x7f4a58078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4a64005fd0 tx=0x7f4a64009500 comp rx=0 tx=0).stop 2024-06-29T15:39:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c1408e0 msgr2=0x7f4a6c1e6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c1408e0 0x7f4a6c1e6370 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c034040 tx=0x7f4a5c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 shutdown_connections 2024-06-29T15:39:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f4a58076420 0x7f4a58078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c1408e0 0x7f4a6c1e6370 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4a6c0b3b50 0x7f4a6c0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 --2- 172.21.15.59:0/2534986759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a6c0b3740 0x7f4a6c06f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 >> 172.21.15.59:0/2534986759 conn(0x7f4a6c06c950 msgr2=0x7f4a6c06fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 shutdown_connections 2024-06-29T15:39:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.109+0000 7f4a73931640 1 -- 172.21.15.59:0/2534986759 wait complete. 2024-06-29T15:39:09.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:39:09.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:39:09.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:39:09.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.279+0000 7f26069be640 1 -- 172.21.15.59:0/787838472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 msgr2=0x7f260006e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.279+0000 7f26069be640 1 --2- 172.21.15.59:0/787838472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 0x7f260006e300 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f25e8009830 tx=0x7f25e802a510 comp rx=0 tx=0).stop 2024-06-29T15:39:09.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.279+0000 7f26069be640 1 -- 172.21.15.59:0/787838472 shutdown_connections 2024-06-29T15:39:09.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.279+0000 7f26069be640 1 --2- 172.21.15.59:0/787838472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 0x7f260006e300 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.279+0000 7f26069be640 1 --2- 172.21.15.59:0/787838472 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2600144470 0x7f26000b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.279+0000 7f26069be640 1 -- 172.21.15.59:0/787838472 >> 172.21.15.59:0/787838472 conn(0x7f26000b06e0 msgr2=0x7f26000b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:09.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.279+0000 7f26069be640 1 -- 172.21.15.59:0/787838472 shutdown_connections 2024-06-29T15:39:09.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.279+0000 7f26069be640 1 -- 172.21.15.59:0/787838472 wait complete. 2024-06-29T15:39:09.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.281+0000 7f26069be640 1 Processor -- start 2024-06-29T15:39:09.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.281+0000 7f26069be640 1 -- start start 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.282+0000 7f26069be640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f260006e840 0x7f260015a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.282+0000 7f26069be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 0x7f2600159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.282+0000 7f26069be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2600144470 0x7f2600159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.282+0000 7f26069be640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f26000d0360 con 0x7f26000b2040 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.282+0000 7f26069be640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26000d01e0 con 0x7f2600144470 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.282+0000 7f26069be640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f260015ee80 con 0x7f260006e840 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 0x7f2600159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 0x7f2600159960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33998/0 (socket says 172.21.15.59:33998) 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 -- 172.21.15.59:0/1541028557 learned_addr learned my addr 172.21.15.59:0/1541028557 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25fffff640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f260006e840 0x7f260015a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f2604f5b640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2600144470 0x7f2600159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 -- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f260006e840 msgr2=0x7f260015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f260006e840 0x7f260015a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 -- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2600144470 msgr2=0x7f2600159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2600144470 0x7f2600159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 -- 172.21.15.59:0/1541028557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2600157b30 con 0x7f26000b2040 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25f7fff640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 0x7f2600159960 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f25e802ab00 tx=0x7f25e8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25fd7fa640 1 -- 172.21.15.59:0/1541028557 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8031d60 con 0x7f26000b2040 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25fd7fa640 1 -- 172.21.15.59:0/1541028557 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25e8033e10 con 0x7f26000b2040 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2600157e10 con 0x7f26000b2040 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f25fd7fa640 1 -- 172.21.15.59:0/1541028557 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8005260 con 0x7f26000b2040 2024-06-29T15:39:09.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.283+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26000aab70 con 0x7f26000b2040 2024-06-29T15:39:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.284+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f260015f3a0 con 0x7f26000b2040 2024-06-29T15:39:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.285+0000 7f25fd7fa640 1 -- 172.21.15.59:0/1541028557 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f25e8005400 con 0x7f26000b2040 2024-06-29T15:39:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.286+0000 7f25fd7fa640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f25e0076420 0x7f25e0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.286+0000 7f25fd7fa640 1 -- 172.21.15.59:0/1541028557 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 5277+0+0 (secure 0 0 0) 0x7f25e80b8960 con 0x7f26000b2040 2024-06-29T15:39:09.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.286+0000 7f25fffff640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f25e0076420 0x7f25e0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.286+0000 7f25fffff640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f25e0076420 0x7f25e0078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f25f000a490 tx=0x7f25f000a420 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:09.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.290+0000 7f25fd7fa640 1 -- 172.21.15.59:0/1541028557 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f25e803f030 con 0x7f26000b2040 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.486+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f26000682b0 con 0x7f26000b2040 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.487+0000 7f25fd7fa640 1 -- 172.21.15.59:0/1541028557 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v185) ==== 84+0+65 (secure 0 0 0) 0x7f25e808a2d0 con 0x7f26000b2040 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:09.489 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f25e0076420 msgr2=0x7f25e0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f25e0076420 0x7f25e0078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f25f000a490 tx=0x7f25f000a420 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 msgr2=0x7f2600159960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 0x7f2600159960 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f25e802ab00 tx=0x7f25e8033c20 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 shutdown_connections 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f25e0076420 0x7f25e0078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2600144470 0x7f2600159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26000b2040 0x7f2600159960 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 --2- 172.21.15.59:0/1541028557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f260006e840 0x7f260015a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 >> 172.21.15.59:0/1541028557 conn(0x7f26000b06e0 msgr2=0x7f26000b5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:09.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 shutdown_connections 2024-06-29T15:39:09.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.489+0000 7f26069be640 1 -- 172.21.15.59:0/1541028557 wait complete. 2024-06-29T15:39:09.501 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:39:09.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:39:09.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.622+0000 7face05a1640 1 -- 172.21.15.59:0/3896935339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80b0e30 msgr2=0x7facd80b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.622+0000 7face05a1640 1 --2- 172.21.15.59:0/3896935339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80b0e30 0x7facd80b4940 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7facc8005fd0 tx=0x7facc802a970 comp rx=0 tx=0).stop 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.622+0000 7face05a1640 1 -- 172.21.15.59:0/3896935339 shutdown_connections 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.622+0000 7face05a1640 1 --2- 172.21.15.59:0/3896935339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80b0e30 0x7facd80b4940 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.622+0000 7face05a1640 1 --2- 172.21.15.59:0/3896935339 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd80ce2c0 0x7facd81596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.622+0000 7face05a1640 1 -- 172.21.15.59:0/3896935339 >> 172.21.15.59:0/3896935339 conn(0x7facd80b7690 msgr2=0x7facd8073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.622+0000 7face05a1640 1 -- 172.21.15.59:0/3896935339 shutdown_connections 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.622+0000 7face05a1640 1 -- 172.21.15.59:0/3896935339 wait complete. 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.624+0000 7face05a1640 1 Processor -- start 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.624+0000 7face05a1640 1 -- start start 2024-06-29T15:39:09.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7face05a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80ce2c0 0x7facd81545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7face05a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd8160090 0x7facd80c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7face05a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80cd1f0 0x7facd80c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7face05a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7facd80b54c0 con 0x7facd80ce2c0 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7face05a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7facd80b5340 con 0x7facd80cd1f0 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7face05a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7facd80b5640 con 0x7facd8160090 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7facde33d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80ce2c0 0x7facd81545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7facde33d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80ce2c0 0x7facd81545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34016/0 (socket says 172.21.15.59:34016) 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7facde33d640 1 -- 172.21.15.59:0/1721887172 learned_addr learned my addr 172.21.15.59:0/1721887172 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.625+0000 7facddb3c640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd8160090 0x7facd80c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facdeb3e640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80cd1f0 0x7facd80c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facde33d640 1 -- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd8160090 msgr2=0x7facd80c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facde33d640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd8160090 0x7facd80c87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facde33d640 1 -- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80cd1f0 msgr2=0x7facd80c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facde33d640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80cd1f0 0x7facd80c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facde33d640 1 -- 172.21.15.59:0/1721887172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd80cd9e0 con 0x7facd80ce2c0 2024-06-29T15:39:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facde33d640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80ce2c0 0x7facd81545e0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7facd000d770 tx=0x7facd000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facd6ffd640 1 -- 172.21.15.59:0/1721887172 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0017070 con 0x7facd80ce2c0 2024-06-29T15:39:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facd6ffd640 1 -- 172.21.15.59:0/1721887172 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7facd0009cc0 con 0x7facd80ce2c0 2024-06-29T15:39:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd80c7460 con 0x7facd80ce2c0 2024-06-29T15:39:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7facd6ffd640 1 -- 172.21.15.59:0/1721887172 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd00106c0 con 0x7facd80ce2c0 2024-06-29T15:39:09.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.626+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd80c7a20 con 0x7facd80ce2c0 2024-06-29T15:39:09.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.627+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd80b5b60 con 0x7facd80ce2c0 2024-06-29T15:39:09.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.628+0000 7facd6ffd640 1 -- 172.21.15.59:0/1721887172 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7facd001a070 con 0x7facd80ce2c0 2024-06-29T15:39:09.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.629+0000 7facd6ffd640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7facc40763f0 0x7facc4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.629+0000 7facd6ffd640 1 -- 172.21.15.59:0/1721887172 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 5277+0+0 (secure 0 0 0) 0x7facd0097d70 con 0x7facd80ce2c0 2024-06-29T15:39:09.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.629+0000 7facddb3c640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7facc40763f0 0x7facc4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.630+0000 7facddb3c640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7facc40763f0 0x7facc4078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7facd80c7ff0 tx=0x7faccc009060 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:09.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.632+0000 7facd6ffd640 1 -- 172.21.15.59:0/1721887172 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7facd0061000 con 0x7facd80ce2c0 2024-06-29T15:39:09.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:09.826+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7facd8155740 con 0x7facd80ce2c0 2024-06-29T15:39:10.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.021+0000 7facd6ffd640 1 -- 172.21.15.59:0/1721887172 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v186) ==== 138+0+0 (secure 0 0 0) 0x7facd0065300 con 0x7facd80ce2c0 2024-06-29T15:39:10.022 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T15:39:10.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7facc40763f0 msgr2=0x7facc4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7facc40763f0 0x7facc4078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7facd80c7ff0 tx=0x7faccc009060 comp rx=0 tx=0).stop 2024-06-29T15:39:10.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80ce2c0 msgr2=0x7facd81545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80ce2c0 0x7facd81545e0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7facd000d770 tx=0x7facd000dc30 comp rx=0 tx=0).stop 2024-06-29T15:39:10.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 shutdown_connections 2024-06-29T15:39:10.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7facc40763f0 0x7facc4078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facd80cd1f0 0x7facd80c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facd8160090 0x7facd80c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 --2- 172.21.15.59:0/1721887172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facd80ce2c0 0x7facd81545e0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 >> 172.21.15.59:0/1721887172 conn(0x7facd80b7690 msgr2=0x7facd815a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:10.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 shutdown_connections 2024-06-29T15:39:10.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.024+0000 7face05a1640 1 -- 172.21.15.59:0/1721887172 wait complete. 2024-06-29T15:39:10.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:39:10.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.165+0000 7febb9924640 1 -- 172.21.15.59:0/411948565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb9c003a50 msgr2=0x7feb9c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.165+0000 7febb9924640 1 --2- 172.21.15.59:0/411948565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb9c003a50 0x7feb9c003ec0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7feba4006fd0 tx=0x7feb9c005830 comp rx=0 tx=0).stop 2024-06-29T15:39:10.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.165+0000 7febb9924640 1 -- 172.21.15.59:0/411948565 shutdown_connections 2024-06-29T15:39:10.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.165+0000 7febb9924640 1 --2- 172.21.15.59:0/411948565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb9c003a50 0x7feb9c003ec0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.165+0000 7febb9924640 1 -- 172.21.15.59:0/411948565 >> 172.21.15.59:0/411948565 conn(0x7febb4073ef0 msgr2=0x7febb4074300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:10.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.165+0000 7febb9924640 1 -- 172.21.15.59:0/411948565 shutdown_connections 2024-06-29T15:39:10.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.165+0000 7febb9924640 1 -- 172.21.15.59:0/411948565 wait complete. 2024-06-29T15:39:10.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.167+0000 7febb9924640 1 Processor -- start 2024-06-29T15:39:10.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.167+0000 7febb9924640 1 -- start start 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.168+0000 7febb9924640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb9c003a50 0x7febb40cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.168+0000 7febb9924640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febb40cd7d0 0x7febb4141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.168+0000 7febb9924640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febb406ebe0 0x7febb406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.168+0000 7febb9924640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7febb4145760 con 0x7febb406ebe0 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb9924640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7febb41455e0 con 0x7feb9c003a50 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb9924640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7febb40d21f0 con 0x7febb40cd7d0 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febb406ebe0 0x7febb406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febb406ebe0 0x7febb406eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34028/0 (socket says 172.21.15.59:34028) 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 -- 172.21.15.59:0/3973975948 learned_addr learned my addr 172.21.15.59:0/3973975948 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb2ffd640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb9c003a50 0x7febb40cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:10.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb27fc640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febb40cd7d0 0x7febb4141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 -- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febb40cd7d0 msgr2=0x7febb4141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febb40cd7d0 0x7febb4141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 -- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb9c003a50 msgr2=0x7febb40cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb9c003a50 0x7febb40cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 -- 172.21.15.59:0/3973975948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb40cdbe0 con 0x7febb406ebe0 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb37fe640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febb406ebe0 0x7febb406eff0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7febb40b0cf0 tx=0x7febac0019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7feb9bfff640 1 -- 172.21.15.59:0/3973975948 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac011870 con 0x7febb406ebe0 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7feb9bfff640 1 -- 172.21.15.59:0/3973975948 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febac017070 con 0x7febb406ebe0 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb406db80 con 0x7febb406ebe0 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.169+0000 7feb9bfff640 1 -- 172.21.15.59:0/3973975948 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac011d60 con 0x7febb406ebe0 2024-06-29T15:39:10.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.170+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb406de20 con 0x7febb406ebe0 2024-06-29T15:39:10.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.171+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb4141720 con 0x7febb406ebe0 2024-06-29T15:39:10.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.171+0000 7feb9bfff640 1 -- 172.21.15.59:0/3973975948 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7febac01c460 con 0x7febb406ebe0 2024-06-29T15:39:10.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.172+0000 7feb9bfff640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7feba00763f0 0x7feba0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:10.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.172+0000 7feb9bfff640 1 -- 172.21.15.59:0/3973975948 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5647+0+0 (secure 0 0 0) 0x7febac098100 con 0x7febb406ebe0 2024-06-29T15:39:10.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.172+0000 7febb2ffd640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7feba00763f0 0x7feba0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:10.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.173+0000 7febb2ffd640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7feba00763f0 0x7feba0078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feba4033000 tx=0x7feba4038040 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:10.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.177+0000 7feb9bfff640 1 -- 172.21.15.59:0/3973975948 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7febb4141720 con 0x7febb406ebe0 2024-06-29T15:39:10.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:10.369+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7febb40700b0 con 0x7febb406ebe0 2024-06-29T15:39:11.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.030+0000 7feb9bfff640 1 -- 172.21.15.59:0/3973975948 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v187) ==== 170+0+0 (secure 0 0 0) 0x7febb40cdbe0 con 0x7febb406ebe0 2024-06-29T15:39:11.031 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7feba00763f0 msgr2=0x7feba0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7feba00763f0 0x7feba0078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feba4033000 tx=0x7feba4038040 comp rx=0 tx=0).stop 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febb406ebe0 msgr2=0x7febb406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febb406ebe0 0x7febb406eff0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7febb40b0cf0 tx=0x7febac0019b0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 shutdown_connections 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7feba00763f0 0x7feba0078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7febb406ebe0 0x7febb406eff0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7febb40cd7d0 0x7febb4141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 --2- 172.21.15.59:0/3973975948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb9c003a50 0x7febb40cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.033+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 >> 172.21.15.59:0/3973975948 conn(0x7febb4073ef0 msgr2=0x7febb4141f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:11.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.034+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 shutdown_connections 2024-06-29T15:39:11.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.034+0000 7febb9924640 1 -- 172.21.15.59:0/3973975948 wait complete. 2024-06-29T15:39:11.061 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:39:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.186+0000 7f0f881ba640 1 -- 172.21.15.59:0/3543798161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f6c004800 msgr2=0x7f0f6c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.186+0000 7f0f881ba640 1 --2- 172.21.15.59:0/3543798161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f6c004800 0x7f0f6c004c10 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009b50 tx=0x7f0f6c005830 comp rx=0 tx=0).stop 2024-06-29T15:39:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.186+0000 7f0f86757640 1 -- 172.21.15.59:0/3543798161 reap_dead start 2024-06-29T15:39:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.186+0000 7f0f881ba640 1 -- 172.21.15.59:0/3543798161 shutdown_connections 2024-06-29T15:39:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.186+0000 7f0f881ba640 1 -- 172.21.15.59:0/3543798161 >> 172.21.15.59:0/3543798161 conn(0x7f0f800b5610 msgr2=0x7f0f800b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.187+0000 7f0f881ba640 1 -- 172.21.15.59:0/3543798161 shutdown_connections 2024-06-29T15:39:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.187+0000 7f0f881ba640 1 -- 172.21.15.59:0/3543798161 wait complete. 2024-06-29T15:39:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.188+0000 7f0f881ba640 1 Processor -- start 2024-06-29T15:39:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.188+0000 7f0f881ba640 1 -- start start 2024-06-29T15:39:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f881ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f6c004800 0x7f0f801f9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f881ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f801f51d0 0x7f0f8015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f881ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f8015cd30 0x7f0f80153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f881ba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0f800b7bd0 con 0x7f0f801f51d0 2024-06-29T15:39:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f881ba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f80162510 con 0x7f0f8015cd30 2024-06-29T15:39:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f881ba640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0f800b7d50 con 0x7f0f6c004800 2024-06-29T15:39:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f801f51d0 0x7f0f8015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f801f51d0 0x7f0f8015c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34046/0 (socket says 172.21.15.59:34046) 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 -- 172.21.15.59:0/4133718527 learned_addr learned my addr 172.21.15.59:0/4133718527 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85f56640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f6c004800 0x7f0f801f9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f86757640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f8015cd30 0x7f0f80153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 -- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f6c004800 msgr2=0x7f0f801f9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f6c004800 0x7f0f801f9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 -- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f8015cd30 msgr2=0x7f0f80153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f8015cd30 0x7f0f80153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 -- 172.21.15.59:0/4133718527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f801f5990 con 0x7f0f801f51d0 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f85755640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f801f51d0 0x7f0f8015c7f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009a70 tx=0x7f0f700336d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f767fc640 1 -- 172.21.15.59:0/4133718527 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f70043070 con 0x7f0f801f51d0 2024-06-29T15:39:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.190+0000 7f0f767fc640 1 -- 172.21.15.59:0/4133718527 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f70004030 con 0x7f0f801f51d0 2024-06-29T15:39:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.191+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f801f5c70 con 0x7f0f801f51d0 2024-06-29T15:39:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.191+0000 7f0f767fc640 1 -- 172.21.15.59:0/4133718527 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f70038040 con 0x7f0f801f51d0 2024-06-29T15:39:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.191+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f800cad50 con 0x7f0f801f51d0 2024-06-29T15:39:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.192+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f800cf9b0 con 0x7f0f801f51d0 2024-06-29T15:39:11.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.193+0000 7f0f767fc640 1 -- 172.21.15.59:0/4133718527 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f0f70046070 con 0x7f0f801f51d0 2024-06-29T15:39:11.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.194+0000 7f0f767fc640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0f640763f0 0x7f0f64078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:11.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.194+0000 7f0f767fc640 1 -- 172.21.15.59:0/4133718527 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 6029+0+0 (secure 0 0 0) 0x7f0f700b84b0 con 0x7f0f801f51d0 2024-06-29T15:39:11.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.194+0000 7f0f85f56640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0f640763f0 0x7f0f64078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:11.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.195+0000 7f0f85f56640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0f640763f0 0x7f0f64078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f800b7620 tx=0x7f0f7c001610 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:11.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.198+0000 7f0f767fc640 1 -- 172.21.15.59:0/4133718527 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f800cf9b0 con 0x7f0f801f51d0 2024-06-29T15:39:11.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:11.428+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0f801f5720 con 0x7f0f801f51d0 2024-06-29T15:39:12.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.047+0000 7f0f767fc640 1 -- 172.21.15.59:0/4133718527 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v93) ==== 174+0+0 (secure 0 0 0) 0x7f0f70081550 con 0x7f0f801f51d0 2024-06-29T15:39:12.049 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 23 and data pool 24 2024-06-29T15:39:12.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.053+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0f640763f0 msgr2=0x7f0f64078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.053+0000 7f0f881ba640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0f640763f0 0x7f0f64078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f800b7620 tx=0x7f0f7c001610 comp rx=0 tx=0).stop 2024-06-29T15:39:12.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.053+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f801f51d0 msgr2=0x7f0f8015c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.053+0000 7f0f881ba640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f801f51d0 0x7f0f8015c7f0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009a70 tx=0x7f0f700336d0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.058+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 shutdown_connections 2024-06-29T15:39:12.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.058+0000 7f0f881ba640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0f640763f0 0x7f0f64078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.058+0000 7f0f881ba640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f8015cd30 0x7f0f80153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.058+0000 7f0f881ba640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f801f51d0 0x7f0f8015c7f0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.058+0000 7f0f881ba640 1 --2- 172.21.15.59:0/4133718527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f6c004800 0x7f0f801f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.058+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 >> 172.21.15.59:0/4133718527 conn(0x7f0f800b5610 msgr2=0x7f0f8015edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.060+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 shutdown_connections 2024-06-29T15:39:12.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.062+0000 7f0f881ba640 1 -- 172.21.15.59:0/4133718527 wait complete. 2024-06-29T15:39:12.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.206+0000 7f5f43be0640 1 -- 172.21.15.59:0/1972405346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f28003c80 msgr2=0x7f5f280040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.206+0000 7f5f43be0640 1 --2- 172.21.15.59:0/1972405346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f28003c80 0x7f5f280040d0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f5f380017a0 tx=0x7f5f28005830 comp rx=0 tx=0).stop 2024-06-29T15:39:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.206+0000 7f5f4217d640 1 -- 172.21.15.59:0/1972405346 reap_dead start 2024-06-29T15:39:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.206+0000 7f5f43be0640 1 -- 172.21.15.59:0/1972405346 shutdown_connections 2024-06-29T15:39:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.206+0000 7f5f43be0640 1 -- 172.21.15.59:0/1972405346 >> 172.21.15.59:0/1972405346 conn(0x7f5f3c0d3ee0 msgr2=0x7f5f3c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.206+0000 7f5f43be0640 1 -- 172.21.15.59:0/1972405346 shutdown_connections 2024-06-29T15:39:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.206+0000 7f5f43be0640 1 -- 172.21.15.59:0/1972405346 wait complete. 2024-06-29T15:39:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.208+0000 7f5f43be0640 1 Processor -- start 2024-06-29T15:39:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.208+0000 7f5f43be0640 1 -- start start 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f43be0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f28003c80 0x7f5f3c1556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f43be0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f3c153330 0x7f5f3c0c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f43be0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3c0cc790 0x7f5f3c0ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f43be0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c15fe50 con 0x7f5f3c0cc790 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f43be0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c15fcd0 con 0x7f5f3c153330 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f43be0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5f3c15ffd0 con 0x7f5f28003c80 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f4217d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3c0cc790 0x7f5f3c0ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f4217d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3c0cc790 0x7f5f3c0ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34068/0 (socket says 172.21.15.59:34068) 2024-06-29T15:39:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f4217d640 1 -- 172.21.15.59:0/1689366233 learned_addr learned my addr 172.21.15.59:0/1689366233 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.209+0000 7f5f4117b640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f3c153330 0x7f5f3c0c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f4197c640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f28003c80 0x7f5f3c1556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f4217d640 1 -- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f28003c80 msgr2=0x7f5f3c1556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f4217d640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f28003c80 0x7f5f3c1556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f4217d640 1 -- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f3c153330 msgr2=0x7f5f3c0c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f4217d640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f3c153330 0x7f5f3c0c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f4217d640 1 -- 172.21.15.59:0/1689366233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c0cd0e0 con 0x7f5f3c0cc790 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f4217d640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3c0cc790 0x7f5f3c0ccba0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c161d40 tx=0x7f5f3400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f327fc640 1 -- 172.21.15.59:0/1689366233 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34004b20 con 0x7f5f3c0cc790 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f327fc640 1 -- 172.21.15.59:0/1689366233 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f34004cc0 con 0x7f5f3c0cc790 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3c155fb0 con 0x7f5f3c0cc790 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f327fc640 1 -- 172.21.15.59:0/1689366233 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34003bd0 con 0x7f5f3c0cc790 2024-06-29T15:39:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.210+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3c1510f0 con 0x7f5f3c0cc790 2024-06-29T15:39:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.211+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f3c160560 con 0x7f5f3c0cc790 2024-06-29T15:39:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.212+0000 7f5f327fc640 1 -- 172.21.15.59:0/1689366233 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f5f34004030 con 0x7f5f3c0cc790 2024-06-29T15:39:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.213+0000 7f5f327fc640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5f200763f0 0x7f5f20078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.213+0000 7f5f327fc640 1 -- 172.21.15.59:0/1689366233 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6129+0+0 (secure 0 0 0) 0x7f5f34097fe0 con 0x7f5f3c0cc790 2024-06-29T15:39:12.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.213+0000 7f5f4197c640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5f200763f0 0x7f5f20078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.214+0000 7f5f4197c640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5f200763f0 0x7f5f20078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f38032000 tx=0x7f5f38001360 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.217+0000 7f5f327fc640 1 -- 172.21.15.59:0/1689366233 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5f3c0cd0e0 con 0x7f5f3c0cc790 2024-06-29T15:39:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.409+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f3c0d2950 con 0x7f5f3c0cc790 2024-06-29T15:39:12.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.411+0000 7f5f327fc640 1 -- 172.21.15.59:0/1689366233 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) ==== 74+0+16953 (secure 0 0 0) 0x7f5f34060ea0 con 0x7f5f3c0cc790 2024-06-29T15:39:12.412 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:12.412 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":188,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:39:11.430198+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.834000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.377444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:12.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5f200763f0 msgr2=0x7f5f20078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5f200763f0 0x7f5f20078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f38032000 tx=0x7f5f38001360 comp rx=0 tx=0).stop 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3c0cc790 msgr2=0x7f5f3c0ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3c0cc790 0x7f5f3c0ccba0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c161d40 tx=0x7f5f3400f3c0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 shutdown_connections 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5f200763f0 0x7f5f20078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3c0cc790 0x7f5f3c0ccba0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f3c153330 0x7f5f3c0c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 --2- 172.21.15.59:0/1689366233 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5f28003c80 0x7f5f3c1556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 >> 172.21.15.59:0/1689366233 conn(0x7f5f3c0d3ee0 msgr2=0x7f5f3c0d2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 shutdown_connections 2024-06-29T15:39:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.413+0000 7f5f43be0640 1 -- 172.21.15.59:0/1689366233 wait complete. 2024-06-29T15:39:12.426 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:12.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.543+0000 7f27f669d640 1 -- 172.21.15.59:0/3475501967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27d8003a70 msgr2=0x7f27d8003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.543+0000 7f27f669d640 1 --2- 172.21.15.59:0/3475501967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27d8003a70 0x7f27d8003ee0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f27e000c2e0 tx=0x7f27d8005830 comp rx=0 tx=0).stop 2024-06-29T15:39:12.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.543+0000 7f27f4c3a640 1 -- 172.21.15.59:0/3475501967 reap_dead start 2024-06-29T15:39:12.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.543+0000 7f27f669d640 1 -- 172.21.15.59:0/3475501967 shutdown_connections 2024-06-29T15:39:12.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.543+0000 7f27f669d640 1 -- 172.21.15.59:0/3475501967 >> 172.21.15.59:0/3475501967 conn(0x7f27f00b5610 msgr2=0x7f27f00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.543+0000 7f27f669d640 1 -- 172.21.15.59:0/3475501967 shutdown_connections 2024-06-29T15:39:12.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.543+0000 7f27f669d640 1 -- 172.21.15.59:0/3475501967 wait complete. 2024-06-29T15:39:12.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.545+0000 7f27f669d640 1 Processor -- start 2024-06-29T15:39:12.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.546+0000 7f27f669d640 1 -- start start 2024-06-29T15:39:12.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27f669d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d8003a70 0x7f27f00cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27f669d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27f01f8fb0 0x7f27f01f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27f669d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27f01f4d30 0x7f27f01f83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27f669d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0162fa0 con 0x7f27f01f8fb0 2024-06-29T15:39:12.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27f669d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0162e20 con 0x7f27d8003a70 2024-06-29T15:39:12.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27f669d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f27f00b7bd0 con 0x7f27f01f4d30 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27ef7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27f01f8fb0 0x7f27f01f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27ef7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27f01f8fb0 0x7f27f01f47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34072/0 (socket says 172.21.15.59:34072) 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27ef7fe640 1 -- 172.21.15.59:0/2828262200 learned_addr learned my addr 172.21.15.59:0/2828262200 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27effff640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d8003a70 0x7f27f00cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27f4c3a640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27f01f4d30 0x7f27f01f83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27ef7fe640 1 -- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27f01f4d30 msgr2=0x7f27f01f83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27ef7fe640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27f01f4d30 0x7f27f01f83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27ef7fe640 1 -- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d8003a70 msgr2=0x7f27f00cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27ef7fe640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d8003a70 0x7f27f00cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27ef7fe640 1 -- 172.21.15.59:0/2828262200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f00cb4d0 con 0x7f27f01f8fb0 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.547+0000 7f27effff640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d8003a70 0x7f27f00cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:12.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.548+0000 7f27ef7fe640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27f01f8fb0 0x7f27f01f47f0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f27e4005bc0 tx=0x7f27e4011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.548+0000 7f27ecff9640 1 -- 172.21.15.59:0/2828262200 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e400fc70 con 0x7f27f01f8fb0 2024-06-29T15:39:12.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.548+0000 7f27ecff9640 1 -- 172.21.15.59:0/2828262200 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27e400fe10 con 0x7f27f01f8fb0 2024-06-29T15:39:12.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.548+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f00cb7b0 con 0x7f27f01f8fb0 2024-06-29T15:39:12.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.548+0000 7f27ecff9640 1 -- 172.21.15.59:0/2828262200 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4019670 con 0x7f27f01f8fb0 2024-06-29T15:39:12.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.548+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f00cbd70 con 0x7f27f01f8fb0 2024-06-29T15:39:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.549+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f00ce050 con 0x7f27f01f8fb0 2024-06-29T15:39:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.550+0000 7f27ecff9640 1 -- 172.21.15.59:0/2828262200 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f27e4019810 con 0x7f27f01f8fb0 2024-06-29T15:39:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.550+0000 7f27ecff9640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f27dc0763f0 0x7f27dc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.550+0000 7f27ecff9640 1 -- 172.21.15.59:0/2828262200 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6129+0+0 (secure 0 0 0) 0x7f27e4098550 con 0x7f27f01f8fb0 2024-06-29T15:39:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.550+0000 7f27effff640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f27dc0763f0 0x7f27dc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.551+0000 7f27effff640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f27dc0763f0 0x7f27dc078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f27e0033000 tx=0x7f27e002d000 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.555+0000 7f27ecff9640 1 -- 172.21.15.59:0/2828262200 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f27f00ce050 con 0x7f27f01f8fb0 2024-06-29T15:39:12.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.746+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27f01622c0 con 0x7f27f01f8fb0 2024-06-29T15:39:12.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.747+0000 7f27ecff9640 1 -- 172.21.15.59:0/2828262200 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) ==== 74+0+16953 (secure 0 0 0) 0x7f27e4061ae0 con 0x7f27f01f8fb0 2024-06-29T15:39:12.748 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:12.748 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":188,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:39:11.430198+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.834000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.377444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f27dc0763f0 msgr2=0x7f27dc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f27dc0763f0 0x7f27dc078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f27e0033000 tx=0x7f27e002d000 comp rx=0 tx=0).stop 2024-06-29T15:39:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27f01f8fb0 msgr2=0x7f27f01f47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27f01f8fb0 0x7f27f01f47f0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f27e4005bc0 tx=0x7f27e4011f40 comp rx=0 tx=0).stop 2024-06-29T15:39:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 shutdown_connections 2024-06-29T15:39:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f27dc0763f0 0x7f27dc078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27f01f4d30 0x7f27f01f83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27f01f8fb0 0x7f27f01f47f0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 --2- 172.21.15.59:0/2828262200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27d8003a70 0x7f27f00cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 >> 172.21.15.59:0/2828262200 conn(0x7f27f00b5610 msgr2=0x7f27f006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 shutdown_connections 2024-06-29T15:39:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.749+0000 7f27f669d640 1 -- 172.21.15.59:0/2828262200 wait complete. 2024-06-29T15:39:12.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:39:12.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.881+0000 7fcdbb594640 1 -- 172.21.15.59:0/56628618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb40b40a0 msgr2=0x7fcdb415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.881+0000 7fcdbb594640 1 --2- 172.21.15.59:0/56628618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb40b40a0 0x7fcdb415e8d0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4161d40 tx=0x7fcda802a530 comp rx=0 tx=0).stop 2024-06-29T15:39:12.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.881+0000 7fcdbb594640 1 -- 172.21.15.59:0/56628618 shutdown_connections 2024-06-29T15:39:12.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.881+0000 7fcdbb594640 1 --2- 172.21.15.59:0/56628618 >> v2:172.21.15.153:3300/0 conn(0x7fcdb415ee10 0x7fcdb415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.881+0000 7fcdbb594640 1 --2- 172.21.15.59:0/56628618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb40b40a0 0x7fcdb415e8d0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.881+0000 7fcdbb594640 1 -- 172.21.15.59:0/56628618 >> 172.21.15.59:0/56628618 conn(0x7fcdb40d3ee0 msgr2=0x7fcdb40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.882+0000 7fcdbb594640 1 -- 172.21.15.59:0/56628618 shutdown_connections 2024-06-29T15:39:12.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.882+0000 7fcdbb594640 1 -- 172.21.15.59:0/56628618 wait complete. 2024-06-29T15:39:12.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.883+0000 7fcdbb594640 1 Processor -- start 2024-06-29T15:39:12.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.883+0000 7fcdbb594640 1 -- start start 2024-06-29T15:39:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdbb594640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdb40d23c0 0x7fcdb40ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdbb594640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb415ee10 0x7fcdb414ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdbb594640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdb40ccd40 0x7fcdb40cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdbb594640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb4160280 con 0x7fcdb415ee10 2024-06-29T15:39:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdbb594640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb4160100 con 0x7fcdb40ccd40 2024-06-29T15:39:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdbb594640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcdb4160400 con 0x7fcdb40d23c0 2024-06-29T15:39:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb415ee10 0x7fcdb414ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb415ee10 0x7fcdb414ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52836/0 (socket says 172.21.15.59:52836) 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 -- 172.21.15.59:0/2944893392 learned_addr learned my addr 172.21.15.59:0/2944893392 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb9b31640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdb40ccd40 0x7fcdb40cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb9330640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdb40d23c0 0x7fcdb40ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 -- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdb40d23c0 msgr2=0x7fcdb40ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdb40d23c0 0x7fcdb40ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 -- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdb40ccd40 msgr2=0x7fcdb40cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdb40ccd40 0x7fcdb40cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 -- 172.21.15.59:0/2944893392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb40cd690 con 0x7fcdb415ee10 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb8b2f640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb415ee10 0x7fcdb414ee10 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009920 tx=0x7fcda8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.885+0000 7fcdb1ffb640 1 -- 172.21.15.59:0/2944893392 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda8031d60 con 0x7fcdb415ee10 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.886+0000 7fcdb1ffb640 1 -- 172.21.15.59:0/2944893392 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcda8033e10 con 0x7fcdb415ee10 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.886+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb4152cb0 con 0x7fcdb415ee10 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.886+0000 7fcdb1ffb640 1 -- 172.21.15.59:0/2944893392 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda8005260 con 0x7fcdb415ee10 2024-06-29T15:39:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.886+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb40c7a00 con 0x7fcdb415ee10 2024-06-29T15:39:12.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.887+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb415e8d0 con 0x7fcdb415ee10 2024-06-29T15:39:12.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.888+0000 7fcdb1ffb640 1 -- 172.21.15.59:0/2944893392 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fcda8005400 con 0x7fcdb415ee10 2024-06-29T15:39:12.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.888+0000 7fcdb1ffb640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcda00763f0 0x7fcda0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.888+0000 7fcdb1ffb640 1 -- 172.21.15.59:0/2944893392 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6129+0+0 (secure 0 0 0) 0x7fcda80b8bf0 con 0x7fcdb415ee10 2024-06-29T15:39:12.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.888+0000 7fcdb9330640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcda00763f0 0x7fcda0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.889+0000 7fcdb9330640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcda00763f0 0x7fcda0078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcda4000f60 tx=0x7fcda4009040 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:12.892+0000 7fcdb1ffb640 1 -- 172.21.15.59:0/2944893392 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcdb40cd690 con 0x7fcdb415ee10 2024-06-29T15:39:13.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:13.126+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fcdb40d2950 con 0x7fcdb415ee10 2024-06-29T15:39:14.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.057+0000 7fcdb1ffb640 1 -- 172.21.15.59:0/2944893392 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v96) ==== 118+0+0 (secure 0 0 0) 0x7fcda803f030 con 0x7fcdb415ee10 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.060+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcda00763f0 msgr2=0x7fcda0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.060+0000 7fcdbb594640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcda00763f0 0x7fcda0078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcda4000f60 tx=0x7fcda4009040 comp rx=0 tx=0).stop 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.060+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb415ee10 msgr2=0x7fcdb414ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.060+0000 7fcdbb594640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb415ee10 0x7fcdb414ee10 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009920 tx=0x7fcda8033c20 comp rx=0 tx=0).stop 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.061+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 shutdown_connections 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.061+0000 7fcdbb594640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcda00763f0 0x7fcda0078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.061+0000 7fcdbb594640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdb40ccd40 0x7fcdb40cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.061+0000 7fcdbb594640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdb415ee10 0x7fcdb414ee10 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.061+0000 7fcdbb594640 1 --2- 172.21.15.59:0/2944893392 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdb40d23c0 0x7fcdb40ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.061+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 >> 172.21.15.59:0/2944893392 conn(0x7fcdb40d3ee0 msgr2=0x7fcdb406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:14.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.061+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 shutdown_connections 2024-06-29T15:39:14.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.061+0000 7fcdbb594640 1 -- 172.21.15.59:0/2944893392 wait complete. 2024-06-29T15:39:14.076 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:39:14.076 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:39:14.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.204+0000 7fd31609d640 1 -- 172.21.15.59:0/3802410807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3101458a0 msgr2=0x7fd3100cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.204+0000 7fd31609d640 1 --2- 172.21.15.59:0/3802410807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3101458a0 0x7fd3100cf870 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fd300005fd0 tx=0x7fd30002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.204+0000 7fd31609d640 1 -- 172.21.15.59:0/3802410807 shutdown_connections 2024-06-29T15:39:14.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.204+0000 7fd31609d640 1 --2- 172.21.15.59:0/3802410807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3101458a0 0x7fd3100cf870 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.204+0000 7fd31609d640 1 -- 172.21.15.59:0/3802410807 >> 172.21.15.59:0/3802410807 conn(0x7fd31006c950 msgr2=0x7fd310073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:14.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.204+0000 7fd31609d640 1 -- 172.21.15.59:0/3802410807 shutdown_connections 2024-06-29T15:39:14.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.204+0000 7fd31609d640 1 -- 172.21.15.59:0/3802410807 wait complete. 2024-06-29T15:39:14.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.206+0000 7fd31609d640 1 Processor -- start 2024-06-29T15:39:14.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.206+0000 7fd31609d640 1 -- start start 2024-06-29T15:39:14.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.207+0000 7fd31609d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3100b3740 0x7fd31006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.207+0000 7fd31609d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3100b3b50 0x7fd31015f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.207+0000 7fd31609d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31015f780 0x7fd310141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.207+0000 7fd31609d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd3100b6320 con 0x7fd3100b3b50 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.207+0000 7fd31609d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd3100b61a0 con 0x7fd3100b3740 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.207+0000 7fd31609d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd3100b64a0 con 0x7fd31015f780 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3100b3b50 0x7fd31015f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3100b3b50 0x7fd31015f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52866/0 (socket says 172.21.15.59:52866) 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 -- 172.21.15.59:0/4024056738 learned_addr learned my addr 172.21.15.59:0/4024056738 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30f7fe640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3100b3740 0x7fd31006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30ffff640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31015f780 0x7fd310141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 -- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31015f780 msgr2=0x7fd310141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31015f780 0x7fd310141130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 -- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3100b3740 msgr2=0x7fd31006f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3100b3740 0x7fd31006f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 -- 172.21.15.59:0/4024056738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3100aa490 con 0x7fd3100b3b50 2024-06-29T15:39:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd30effd640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3100b3b50 0x7fd31015f240 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fd30400b3f0 tx=0x7fd30400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd2f7fff640 1 -- 172.21.15.59:0/4024056738 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd304015020 con 0x7fd3100b3b50 2024-06-29T15:39:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd2f7fff640 1 -- 172.21.15.59:0/4024056738 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd304011920 con 0x7fd3100b3b50 2024-06-29T15:39:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3100aa770 con 0x7fd3100b3b50 2024-06-29T15:39:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd2f7fff640 1 -- 172.21.15.59:0/4024056738 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd304011d70 con 0x7fd3100b3b50 2024-06-29T15:39:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.208+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3100ac420 con 0x7fd3100b3b50 2024-06-29T15:39:14.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.209+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3101458a0 con 0x7fd3100b3b50 2024-06-29T15:39:14.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.210+0000 7fd2f7fff640 1 -- 172.21.15.59:0/4024056738 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fd304018020 con 0x7fd3100b3b50 2024-06-29T15:39:14.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.211+0000 7fd2f7fff640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd2fc0763f0 0x7fd2fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:14.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.211+0000 7fd2f7fff640 1 -- 172.21.15.59:0/4024056738 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7fd304097740 con 0x7fd3100b3b50 2024-06-29T15:39:14.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.211+0000 7fd30f7fe640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd2fc0763f0 0x7fd2fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:14.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.212+0000 7fd30f7fe640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd2fc0763f0 0x7fd2fc078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd300033000 tx=0x7fd300004730 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:14.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.215+0000 7fd2f7fff640 1 -- 172.21.15.59:0/4024056738 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd30401f030 con 0x7fd3100b3b50 2024-06-29T15:39:14.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:14.446+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd3100d24e0 con 0x7fd3100b3b50 2024-06-29T15:39:15.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.064+0000 7fd2f7fff640 1 -- 172.21.15.59:0/4024056738 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v97) ==== 122+0+0 (secure 0 0 0) 0x7fd304060680 con 0x7fd3100b3b50 2024-06-29T15:39:15.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd2fc0763f0 msgr2=0x7fd2fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd2fc0763f0 0x7fd2fc078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd300033000 tx=0x7fd300004730 comp rx=0 tx=0).stop 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3100b3b50 msgr2=0x7fd31015f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3100b3b50 0x7fd31015f240 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fd30400b3f0 tx=0x7fd30400b8b0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 shutdown_connections 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fd2fc0763f0 0x7fd2fc078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31015f780 0x7fd310141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3100b3b50 0x7fd31015f240 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 --2- 172.21.15.59:0/4024056738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3100b3740 0x7fd31006f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 >> 172.21.15.59:0/4024056738 conn(0x7fd31006c950 msgr2=0x7fd31015cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 shutdown_connections 2024-06-29T15:39:15.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.067+0000 7fd31609d640 1 -- 172.21.15.59:0/4024056738 wait complete. 2024-06-29T15:39:15.082 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:39:15.082 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:39:15.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.206+0000 7efd0e38c640 1 -- 172.21.15.59:0/863941122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf0004e00 msgr2=0x7efcf0005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.206+0000 7efd0e38c640 1 --2- 172.21.15.59:0/863941122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf0004e00 0x7efcf0005290 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7efd00005fd0 tx=0x7efcf0005830 comp rx=0 tx=0).stop 2024-06-29T15:39:15.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.206+0000 7efd0c929640 1 -- 172.21.15.59:0/863941122 reap_dead start 2024-06-29T15:39:15.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.206+0000 7efd0e38c640 1 -- 172.21.15.59:0/863941122 shutdown_connections 2024-06-29T15:39:15.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.206+0000 7efd0e38c640 1 -- 172.21.15.59:0/863941122 >> 172.21.15.59:0/863941122 conn(0x7efd080b5610 msgr2=0x7efd080b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.206+0000 7efd0e38c640 1 -- 172.21.15.59:0/863941122 shutdown_connections 2024-06-29T15:39:15.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.206+0000 7efd0e38c640 1 -- 172.21.15.59:0/863941122 wait complete. 2024-06-29T15:39:15.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.208+0000 7efd0e38c640 1 Processor -- start 2024-06-29T15:39:15.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.208+0000 7efd0e38c640 1 -- start start 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd0e38c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcf0004e00 0x7efd08159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd0e38c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd080cac60 0x7efd080cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd0e38c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0815de50 0x7efd0815e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd0e38c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efd08162df0 con 0x7efd080cac60 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd0e38c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd08162c70 con 0x7efd0815de50 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd0e38c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efd08162f70 con 0x7efcf0004e00 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd080cac60 0x7efd080cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd080cac60 0x7efd080cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52882/0 (socket says 172.21.15.59:52882) 2024-06-29T15:39:15.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 -- 172.21.15.59:0/575255033 learned_addr learned my addr 172.21.15.59:0/575255033 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd07fff640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcf0004e00 0x7efd08159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd0c929640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0815de50 0x7efd0815e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 -- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcf0004e00 msgr2=0x7efd08159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcf0004e00 0x7efd08159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 -- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0815de50 msgr2=0x7efd0815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0815de50 0x7efd0815e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 -- 172.21.15.59:0/575255033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd080cb5b0 con 0x7efd080cac60 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd077fe640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd080cac60 0x7efd080cb070 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7efcfc006fd0 tx=0x7efcfc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.210+0000 7efd04ff9640 1 -- 172.21.15.59:0/575255033 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc015070 con 0x7efd080cac60 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.211+0000 7efd04ff9640 1 -- 172.21.15.59:0/575255033 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcfc011860 con 0x7efd080cac60 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.211+0000 7efd04ff9640 1 -- 172.21.15.59:0/575255033 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc011cb0 con 0x7efd080cac60 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.211+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0815e7a0 con 0x7efd080cac60 2024-06-29T15:39:15.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.211+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd080cbce0 con 0x7efd080cac60 2024-06-29T15:39:15.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.212+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd080cf080 con 0x7efd080cac60 2024-06-29T15:39:15.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.212+0000 7efd04ff9640 1 -- 172.21.15.59:0/575255033 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7efcfc018070 con 0x7efd080cac60 2024-06-29T15:39:15.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.213+0000 7efd04ff9640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7efcf4076420 0x7efcf4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.213+0000 7efd04ff9640 1 -- 172.21.15.59:0/575255033 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7efcfc0987c0 con 0x7efd080cac60 2024-06-29T15:39:15.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.213+0000 7efd07fff640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7efcf4076420 0x7efcf4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.214+0000 7efd07fff640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7efcf4076420 0x7efcf4078930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efcf8006fd0 tx=0x7efcf8006b10 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.217+0000 7efd04ff9640 1 -- 172.21.15.59:0/575255033 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd0815e7a0 con 0x7efd080cac60 2024-06-29T15:39:15.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.448+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd080cb790 con 0x7efd080cac60 2024-06-29T15:39:15.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.449+0000 7efd04ff9640 1 -- 172.21.15.59:0/575255033 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5235 (secure 0 0 0) 0x7efd080cb5b0 con 0x7efd080cac60 2024-06-29T15:39:15.450 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:15.450 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"btime":"2024-06-29T15:39:15:059125+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8588,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3866646629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3866646629},{"type":"v1","addr":"172.21.15.153:6835","nonce":3866646629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9724,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/933275096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":933275096},{"type":"v1","addr":"172.21.15.59:6835","nonce":933275096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9736,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1230229255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1230229255},{"type":"v1","addr":"172.21.15.153:6837","nonce":1230229255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.430152+0000","modified":"2024-06-29T15:39:14.448659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2399523179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2399523179},{"type":"v1","addr":"172.21.15.59:6834","nonce":2399523179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10401,"qdb_cluster":[10401]},"id":11}]} 2024-06-29T15:39:15.450 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 97 2024-06-29T15:39:15.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7efcf4076420 msgr2=0x7efcf4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7efcf4076420 0x7efcf4078930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efcf8006fd0 tx=0x7efcf8006b10 comp rx=0 tx=0).stop 2024-06-29T15:39:15.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd080cac60 msgr2=0x7efd080cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd080cac60 0x7efd080cb070 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7efcfc006fd0 tx=0x7efcfc0039c0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 shutdown_connections 2024-06-29T15:39:15.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7efcf4076420 0x7efcf4078930 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd0815de50 0x7efd0815e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd080cac60 0x7efd080cb070 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 --2- 172.21.15.59:0/575255033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efcf0004e00 0x7efd08159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 >> 172.21.15.59:0/575255033 conn(0x7efd080b5610 msgr2=0x7efd0806c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 shutdown_connections 2024-06-29T15:39:15.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.451+0000 7efd0e38c640 1 -- 172.21.15.59:0/575255033 wait complete. 2024-06-29T15:39:15.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:15.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.579+0000 7f7e797c7640 1 -- 172.21.15.59:0/1143096580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e7415c2f0 msgr2=0x7f7e7415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.579+0000 7f7e797c7640 1 --2- 172.21.15.59:0/1143096580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e7415c2f0 0x7f7e7415c700 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c005fd0 tx=0x7f7e6c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.579+0000 7f7e797c7640 1 -- 172.21.15.59:0/1143096580 shutdown_connections 2024-06-29T15:39:15.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.579+0000 7f7e797c7640 1 --2- 172.21.15.59:0/1143096580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e7415c2f0 0x7f7e7415c700 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.579+0000 7f7e797c7640 1 --2- 172.21.15.59:0/1143096580 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e740b3740 0x7f7e740b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.579+0000 7f7e797c7640 1 -- 172.21.15.59:0/1143096580 >> 172.21.15.59:0/1143096580 conn(0x7f7e7406c950 msgr2=0x7f7e74073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.579+0000 7f7e797c7640 1 -- 172.21.15.59:0/1143096580 shutdown_connections 2024-06-29T15:39:15.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.579+0000 7f7e797c7640 1 -- 172.21.15.59:0/1143096580 wait complete. 2024-06-29T15:39:15.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.581+0000 7f7e797c7640 1 Processor -- start 2024-06-29T15:39:15.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.581+0000 7f7e797c7640 1 -- start start 2024-06-29T15:39:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e797c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e740b3740 0x7f7e740a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e797c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7415c2f0 0x7f7e740ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e797c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e7415ae00 0x7f7e740aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e797c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740b5fd0 con 0x7f7e7415c2f0 2024-06-29T15:39:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e797c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740b5e50 con 0x7f7e740b3740 2024-06-29T15:39:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e797c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7e740b6150 con 0x7f7e7415ae00 2024-06-29T15:39:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7415c2f0 0x7f7e740ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7415c2f0 0x7f7e740ae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52884/0 (socket says 172.21.15.59:52884) 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 -- 172.21.15.59:0/2260153664 learned_addr learned my addr 172.21.15.59:0/2260153664 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e737fe640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e7415ae00 0x7f7e740aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e72ffd640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e740b3740 0x7f7e740a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 -- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e7415ae00 msgr2=0x7f7e740aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e7415ae00 0x7f7e740aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 -- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e740b3740 msgr2=0x7f7e740a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e740b3740 0x7f7e740a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 -- 172.21.15.59:0/2260153664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e740aa010 con 0x7f7e7415c2f0 2024-06-29T15:39:15.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e727fc640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7415c2f0 0x7f7e740ae570 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00ea20 tx=0x7f7e5c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.583+0000 7f7e6b7fe640 1 -- 172.21.15.59:0/2260153664 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c00ce40 con 0x7f7e7415c2f0 2024-06-29T15:39:15.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.584+0000 7f7e6b7fe640 1 -- 172.21.15.59:0/2260153664 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e5c004020 con 0x7f7e7415c2f0 2024-06-29T15:39:15.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.584+0000 7f7e6b7fe640 1 -- 172.21.15.59:0/2260153664 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c004470 con 0x7f7e7415c2f0 2024-06-29T15:39:15.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.584+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e740aa2f0 con 0x7f7e7415c2f0 2024-06-29T15:39:15.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.584+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e740aa8b0 con 0x7f7e7415c2f0 2024-06-29T15:39:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.585+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e740b6670 con 0x7f7e7415c2f0 2024-06-29T15:39:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.586+0000 7f7e6b7fe640 1 -- 172.21.15.59:0/2260153664 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f7e5c0035f0 con 0x7f7e7415c2f0 2024-06-29T15:39:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.586+0000 7f7e6b7fe640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f7e54076420 0x7f7e54078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.586+0000 7f7e6b7fe640 1 -- 172.21.15.59:0/2260153664 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7f7e5c0989a0 con 0x7f7e7415c2f0 2024-06-29T15:39:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.586+0000 7f7e72ffd640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f7e54076420 0x7f7e54078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.587+0000 7f7e72ffd640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f7e54076420 0x7f7e54078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e64001e20 tx=0x7f7e64009040 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.590+0000 7f7e6b7fe640 1 -- 172.21.15.59:0/2260153664 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7e5c061a50 con 0x7f7e7415c2f0 2024-06-29T15:39:15.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.779+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e740a9da0 con 0x7f7e7415c2f0 2024-06-29T15:39:15.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.781+0000 7f7e6b7fe640 1 -- 172.21.15.59:0/2260153664 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) ==== 74+0+16940 (secure 0 0 0) 0x7f7e5c065d50 con 0x7f7e7415c2f0 2024-06-29T15:39:15.782 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:15.782 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":189,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:39:13.044624+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.834000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.377444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:15.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f7e54076420 msgr2=0x7f7e54078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f7e54076420 0x7f7e54078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e64001e20 tx=0x7f7e64009040 comp rx=0 tx=0).stop 2024-06-29T15:39:15.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7415c2f0 msgr2=0x7f7e740ae570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7415c2f0 0x7f7e740ae570 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00ea20 tx=0x7f7e5c00eee0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 shutdown_connections 2024-06-29T15:39:15.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f7e54076420 0x7f7e54078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e7415ae00 0x7f7e740aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7415c2f0 0x7f7e740ae570 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 --2- 172.21.15.59:0/2260153664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e740b3740 0x7f7e740a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 >> 172.21.15.59:0/2260153664 conn(0x7f7e7406c950 msgr2=0x7f7e741415a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 shutdown_connections 2024-06-29T15:39:15.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.783+0000 7f7e797c7640 1 -- 172.21.15.59:0/2260153664 wait complete. 2024-06-29T15:39:15.796 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:39:15.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:39:15.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.913+0000 7f46c228f640 1 -- 172.21.15.59:0/2123241838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46a4003cd0 msgr2=0x7f46a4004120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.913+0000 7f46c228f640 1 --2- 172.21.15.59:0/2123241838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46a4003cd0 0x7f46a4004120 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f46ac006ef0 tx=0x7f46a4005830 comp rx=0 tx=0).stop 2024-06-29T15:39:15.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.913+0000 7f46c082c640 1 -- 172.21.15.59:0/2123241838 reap_dead start 2024-06-29T15:39:15.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.913+0000 7f46c228f640 1 -- 172.21.15.59:0/2123241838 shutdown_connections 2024-06-29T15:39:15.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.913+0000 7f46c228f640 1 -- 172.21.15.59:0/2123241838 >> 172.21.15.59:0/2123241838 conn(0x7f46bc06c950 msgr2=0x7f46bc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.913+0000 7f46c228f640 1 -- 172.21.15.59:0/2123241838 shutdown_connections 2024-06-29T15:39:15.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.913+0000 7f46c228f640 1 -- 172.21.15.59:0/2123241838 wait complete. 2024-06-29T15:39:15.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.915+0000 7f46c228f640 1 Processor -- start 2024-06-29T15:39:15.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.915+0000 7f46c228f640 1 -- start start 2024-06-29T15:39:15.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46c228f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a4003cd0 0x7f46bc15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46c228f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc13e570 0x7f46bc1634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46c228f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46bc1639e0 0x7f46bc0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46c228f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc165020 con 0x7f46bc13e570 2024-06-29T15:39:15.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46c228f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc164ea0 con 0x7f46a4003cd0 2024-06-29T15:39:15.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46c228f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46bc1651a0 con 0x7f46bc1639e0 2024-06-29T15:39:15.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc13e570 0x7f46bc1634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc13e570 0x7f46bc1634a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52890/0 (socket says 172.21.15.59:52890) 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 -- 172.21.15.59:0/1119891325 learned_addr learned my addr 172.21.15.59:0/1119891325 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bbfff640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a4003cd0 0x7f46bc15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46c082c640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46bc1639e0 0x7f46bc0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 -- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46bc1639e0 msgr2=0x7f46bc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46bc1639e0 0x7f46bc0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 -- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a4003cd0 msgr2=0x7f46bc15bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a4003cd0 0x7f46bc15bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 -- 172.21.15.59:0/1119891325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46bc163df0 con 0x7f46bc13e570 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46bb7fe640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc13e570 0x7f46bc1634a0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f46b0003ec0 tx=0x7f46b0001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.917+0000 7f46b8ff9640 1 -- 172.21.15.59:0/1119891325 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0015020 con 0x7f46bc13e570 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.918+0000 7f46b8ff9640 1 -- 172.21.15.59:0/1119891325 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46b0011900 con 0x7f46bc13e570 2024-06-29T15:39:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.918+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46bc13f030 con 0x7f46bc13e570 2024-06-29T15:39:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.918+0000 7f46b8ff9640 1 -- 172.21.15.59:0/1119891325 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0011d10 con 0x7f46bc13e570 2024-06-29T15:39:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.918+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46bc13f5f0 con 0x7f46bc13e570 2024-06-29T15:39:15.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.919+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46bc164ab0 con 0x7f46bc13e570 2024-06-29T15:39:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.919+0000 7f46b8ff9640 1 -- 172.21.15.59:0/1119891325 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f46b0018020 con 0x7f46bc13e570 2024-06-29T15:39:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.920+0000 7f46b8ff9640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f46a80763f0 0x7f46a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.920+0000 7f46b8ff9640 1 -- 172.21.15.59:0/1119891325 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7f46b0098440 con 0x7f46bc13e570 2024-06-29T15:39:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.920+0000 7f46bbfff640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f46a80763f0 0x7f46a8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.921+0000 7f46bbfff640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f46a80763f0 0x7f46a8078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46ac033000 tx=0x7f46ac02d000 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:15.924+0000 7f46b8ff9640 1 -- 172.21.15.59:0/1119891325 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f46bc164ab0 con 0x7f46bc13e570 2024-06-29T15:39:16.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.114+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 --> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46bc1458a0 con 0x7f46a80763f0 2024-06-29T15:39:16.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.119+0000 7f46b8ff9640 1 -- 172.21.15.59:0/1119891325 <== mgr.9505 v2:172.21.15.153:6832/1270545814 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252462 (secure 0 0 0) 0x7f46bc1458a0 con 0x7f46a80763f0 2024-06-29T15:39:16.123 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:16.125 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:39:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f46a80763f0 msgr2=0x7f46a8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f46a80763f0 0x7f46a8078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46ac033000 tx=0x7f46ac02d000 comp rx=0 tx=0).stop 2024-06-29T15:39:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc13e570 msgr2=0x7f46bc1634a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc13e570 0x7f46bc1634a0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f46b0003ec0 tx=0x7f46b0001800 comp rx=0 tx=0).stop 2024-06-29T15:39:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 shutdown_connections 2024-06-29T15:39:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f46a80763f0 0x7f46a8078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46bc1639e0 0x7f46bc0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc13e570 0x7f46bc1634a0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 --2- 172.21.15.59:0/1119891325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a4003cd0 0x7f46bc15bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 >> 172.21.15.59:0/1119891325 conn(0x7f46bc06c950 msgr2=0x7f46bc141df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 shutdown_connections 2024-06-29T15:39:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.122+0000 7f46c228f640 1 -- 172.21.15.59:0/1119891325 wait complete. 2024-06-29T15:39:16.135 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:39:15.572162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1146,"num_read_kb":3223,"num_write":362,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":392572,"kb_used_data":19596,"kb_used_omap":12,"kb_used_meta":372851,"kb_avail":754582148,"statfs":{"total":773094113280,"available":772692119552,"internally_reserved":0,"allocated":20066304,"data_stored":10789364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":381799986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.548503"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.053634+0000","last_change":"2024-06-29T15:39:12.045916+0000","last_active":"2024-06-29T15:39:12.053634+0000","last_peered":"2024-06-29T15:39:12.053634+0000","last_clean":"2024-06-29T15:39:12.053634+0000","last_became_active":"2024-06-29T15:39:12.044457+0000","last_became_peered":"2024-06-29T15:39:12.044457+0000","last_unstale":"2024-06-29T15:39:12.053634+0000","last_undegraded":"2024-06-29T15:39:12.053634+0000","last_fullsized":"2024-06-29T15:39:12.053634+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:24.602613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.059853+0000","last_change":"2024-06-29T15:39:12.057302+0000","last_active":"2024-06-29T15:39:12.059853+0000","last_peered":"2024-06-29T15:39:12.059853+0000","last_clean":"2024-06-29T15:39:12.059853+0000","last_became_active":"2024-06-29T15:39:12.056934+0000","last_became_peered":"2024-06-29T15:39:12.056934+0000","last_unstale":"2024-06-29T15:39:12.059853+0000","last_undegraded":"2024-06-29T15:39:12.059853+0000","last_fullsized":"2024-06-29T15:39:12.059853+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:12.215644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.054441+0000","last_change":"2024-06-29T15:39:12.051318+0000","last_active":"2024-06-29T15:39:12.054441+0000","last_peered":"2024-06-29T15:39:12.054441+0000","last_clean":"2024-06-29T15:39:12.054441+0000","last_became_active":"2024-06-29T15:39:12.050996+0000","last_became_peered":"2024-06-29T15:39:12.050996+0000","last_unstale":"2024-06-29T15:39:12.054441+0000","last_undegraded":"2024-06-29T15:39:12.054441+0000","last_fullsized":"2024-06-29T15:39:12.054441+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:58.914510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.054732+0000","last_change":"2024-06-29T15:39:12.052917+0000","last_active":"2024-06-29T15:39:12.054732+0000","last_peered":"2024-06-29T15:39:12.054732+0000","last_clean":"2024-06-29T15:39:12.054732+0000","last_became_active":"2024-06-29T15:39:12.052601+0000","last_became_peered":"2024-06-29T15:39:12.052601+0000","last_unstale":"2024-06-29T15:39:12.054732+0000","last_undegraded":"2024-06-29T15:39:12.054732+0000","last_fullsized":"2024-06-29T15:39:12.054732+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:41:19.215545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.064761+0000","last_change":"2024-06-29T15:39:12.060412+0000","last_active":"2024-06-29T15:39:12.064761+0000","last_peered":"2024-06-29T15:39:12.064761+0000","last_clean":"2024-06-29T15:39:12.064761+0000","last_became_active":"2024-06-29T15:39:12.060067+0000","last_became_peered":"2024-06-29T15:39:12.060067+0000","last_unstale":"2024-06-29T15:39:12.064761+0000","last_undegraded":"2024-06-29T15:39:12.064761+0000","last_fullsized":"2024-06-29T15:39:12.064761+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:42.783274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.059627+0000","last_change":"2024-06-29T15:39:12.055119+0000","last_active":"2024-06-29T15:39:12.059627+0000","last_peered":"2024-06-29T15:39:12.059627+0000","last_clean":"2024-06-29T15:39:12.059627+0000","last_became_active":"2024-06-29T15:39:12.054795+0000","last_became_peered":"2024-06-29T15:39:12.054795+0000","last_unstale":"2024-06-29T15:39:12.059627+0000","last_undegraded":"2024-06-29T15:39:12.059627+0000","last_fullsized":"2024-06-29T15:39:12.059627+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:52:35.682187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052488+0000","last_change":"2024-06-29T15:39:12.056021+0000","last_active":"2024-06-29T15:39:13.052488+0000","last_peered":"2024-06-29T15:39:13.052488+0000","last_clean":"2024-06-29T15:39:13.052488+0000","last_became_active":"2024-06-29T15:39:12.054045+0000","last_became_peered":"2024-06-29T15:39:12.054045+0000","last_unstale":"2024-06-29T15:39:13.052488+0000","last_undegraded":"2024-06-29T15:39:13.052488+0000","last_fullsized":"2024-06-29T15:39:13.052488+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:54.404587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052433+0000","last_change":"2024-06-29T15:39:12.057833+0000","last_active":"2024-06-29T15:39:13.052433+0000","last_peered":"2024-06-29T15:39:13.052433+0000","last_clean":"2024-06-29T15:39:13.052433+0000","last_became_active":"2024-06-29T15:39:12.057498+0000","last_became_peered":"2024-06-29T15:39:12.057498+0000","last_unstale":"2024-06-29T15:39:13.052433+0000","last_undegraded":"2024-06-29T15:39:13.052433+0000","last_fullsized":"2024-06-29T15:39:13.052433+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:04:26.205915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053097+0000","last_change":"2024-06-29T15:39:12.055967+0000","last_active":"2024-06-29T15:39:13.053097+0000","last_peered":"2024-06-29T15:39:13.053097+0000","last_clean":"2024-06-29T15:39:13.053097+0000","last_became_active":"2024-06-29T15:39:12.054679+0000","last_became_peered":"2024-06-29T15:39:12.054679+0000","last_unstale":"2024-06-29T15:39:13.053097+0000","last_undegraded":"2024-06-29T15:39:13.053097+0000","last_fullsized":"2024-06-29T15:39:13.053097+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:34.865489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053078+0000","last_change":"2024-06-29T15:39:12.061724+0000","last_active":"2024-06-29T15:39:13.053078+0000","last_peered":"2024-06-29T15:39:13.053078+0000","last_clean":"2024-06-29T15:39:13.053078+0000","last_became_active":"2024-06-29T15:39:12.061383+0000","last_became_peered":"2024-06-29T15:39:12.061383+0000","last_unstale":"2024-06-29T15:39:13.053078+0000","last_undegraded":"2024-06-29T15:39:13.053078+0000","last_fullsized":"2024-06-29T15:39:13.053078+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:31:22.629917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055813+0000","last_change":"2024-06-29T15:39:12.068062+0000","last_active":"2024-06-29T15:39:13.055813+0000","last_peered":"2024-06-29T15:39:13.055813+0000","last_clean":"2024-06-29T15:39:13.055813+0000","last_became_active":"2024-06-29T15:39:12.067703+0000","last_became_peered":"2024-06-29T15:39:12.067703+0000","last_unstale":"2024-06-29T15:39:13.055813+0000","last_undegraded":"2024-06-29T15:39:13.055813+0000","last_fullsized":"2024-06-29T15:39:13.055813+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:27.690017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.059616+0000","last_change":"2024-06-29T15:39:12.056563+0000","last_active":"2024-06-29T15:39:12.059616+0000","last_peered":"2024-06-29T15:39:12.059616+0000","last_clean":"2024-06-29T15:39:12.059616+0000","last_became_active":"2024-06-29T15:39:12.055841+0000","last_became_peered":"2024-06-29T15:39:12.055841+0000","last_unstale":"2024-06-29T15:39:12.059616+0000","last_undegraded":"2024-06-29T15:39:12.059616+0000","last_fullsized":"2024-06-29T15:39:12.059616+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:44.332196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055693+0000","last_change":"2024-06-29T15:39:12.057529+0000","last_active":"2024-06-29T15:39:13.055693+0000","last_peered":"2024-06-29T15:39:13.055693+0000","last_clean":"2024-06-29T15:39:13.055693+0000","last_became_active":"2024-06-29T15:39:12.056798+0000","last_became_peered":"2024-06-29T15:39:12.056798+0000","last_unstale":"2024-06-29T15:39:13.055693+0000","last_undegraded":"2024-06-29T15:39:13.055693+0000","last_fullsized":"2024-06-29T15:39:13.055693+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:58.124939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069109+0000","last_change":"2024-06-29T15:39:12.061693+0000","last_active":"2024-06-29T15:39:12.069109+0000","last_peered":"2024-06-29T15:39:12.069109+0000","last_clean":"2024-06-29T15:39:12.069109+0000","last_became_active":"2024-06-29T15:39:12.060832+0000","last_became_peered":"2024-06-29T15:39:12.060832+0000","last_unstale":"2024-06-29T15:39:12.069109+0000","last_undegraded":"2024-06-29T15:39:12.069109+0000","last_fullsized":"2024-06-29T15:39:12.069109+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:25:10.528258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055642+0000","last_change":"2024-06-29T15:39:12.063418+0000","last_active":"2024-06-29T15:39:13.055642+0000","last_peered":"2024-06-29T15:39:13.055642+0000","last_clean":"2024-06-29T15:39:13.055642+0000","last_became_active":"2024-06-29T15:39:12.063087+0000","last_became_peered":"2024-06-29T15:39:12.063087+0000","last_unstale":"2024-06-29T15:39:13.055642+0000","last_undegraded":"2024-06-29T15:39:13.055642+0000","last_fullsized":"2024-06-29T15:39:13.055642+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:16.152546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.059810+0000","last_change":"2024-06-29T15:39:12.058509+0000","last_active":"2024-06-29T15:39:12.059810+0000","last_peered":"2024-06-29T15:39:12.059810+0000","last_clean":"2024-06-29T15:39:12.059810+0000","last_became_active":"2024-06-29T15:39:12.058179+0000","last_became_peered":"2024-06-29T15:39:12.058179+0000","last_unstale":"2024-06-29T15:39:12.059810+0000","last_undegraded":"2024-06-29T15:39:12.059810+0000","last_fullsized":"2024-06-29T15:39:12.059810+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:46:43.779985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053143+0000","last_change":"2024-06-29T15:39:12.055463+0000","last_active":"2024-06-29T15:39:13.053143+0000","last_peered":"2024-06-29T15:39:13.053143+0000","last_clean":"2024-06-29T15:39:13.053143+0000","last_became_active":"2024-06-29T15:39:12.055113+0000","last_became_peered":"2024-06-29T15:39:12.055113+0000","last_unstale":"2024-06-29T15:39:13.053143+0000","last_undegraded":"2024-06-29T15:39:13.053143+0000","last_fullsized":"2024-06-29T15:39:13.053143+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:30:44.019499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055038+0000","last_change":"2024-06-29T15:39:12.048530+0000","last_active":"2024-06-29T15:39:13.055038+0000","last_peered":"2024-06-29T15:39:13.055038+0000","last_clean":"2024-06-29T15:39:13.055038+0000","last_became_active":"2024-06-29T15:39:12.047989+0000","last_became_peered":"2024-06-29T15:39:12.047989+0000","last_unstale":"2024-06-29T15:39:13.055038+0000","last_undegraded":"2024-06-29T15:39:13.055038+0000","last_fullsized":"2024-06-29T15:39:13.055038+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:04:33.628753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056196+0000","last_change":"2024-06-29T15:39:12.063513+0000","last_active":"2024-06-29T15:39:13.056196+0000","last_peered":"2024-06-29T15:39:13.056196+0000","last_clean":"2024-06-29T15:39:13.056196+0000","last_became_active":"2024-06-29T15:39:12.063160+0000","last_became_peered":"2024-06-29T15:39:12.063160+0000","last_unstale":"2024-06-29T15:39:13.056196+0000","last_undegraded":"2024-06-29T15:39:13.056196+0000","last_fullsized":"2024-06-29T15:39:13.056196+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:18:28.925687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135354+0000","last_change":"2024-06-29T15:39:12.055465+0000","last_active":"2024-06-29T15:39:13.135354+0000","last_peered":"2024-06-29T15:39:13.135354+0000","last_clean":"2024-06-29T15:39:13.135354+0000","last_became_active":"2024-06-29T15:39:12.055042+0000","last_became_peered":"2024-06-29T15:39:12.055042+0000","last_unstale":"2024-06-29T15:39:13.135354+0000","last_undegraded":"2024-06-29T15:39:13.135354+0000","last_fullsized":"2024-06-29T15:39:13.135354+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:00.216407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055887+0000","last_change":"2024-06-29T15:39:12.042695+0000","last_active":"2024-06-29T15:39:13.055887+0000","last_peered":"2024-06-29T15:39:13.055887+0000","last_clean":"2024-06-29T15:39:13.055887+0000","last_became_active":"2024-06-29T15:39:12.042324+0000","last_became_peered":"2024-06-29T15:39:12.042324+0000","last_unstale":"2024-06-29T15:39:13.055887+0000","last_undegraded":"2024-06-29T15:39:13.055887+0000","last_fullsized":"2024-06-29T15:39:13.055887+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:52:50.567029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.062627+0000","last_change":"2024-06-29T15:39:12.061442+0000","last_active":"2024-06-29T15:39:12.062627+0000","last_peered":"2024-06-29T15:39:12.062627+0000","last_clean":"2024-06-29T15:39:12.062627+0000","last_became_active":"2024-06-29T15:39:12.061017+0000","last_became_peered":"2024-06-29T15:39:12.061017+0000","last_unstale":"2024-06-29T15:39:12.062627+0000","last_undegraded":"2024-06-29T15:39:12.062627+0000","last_fullsized":"2024-06-29T15:39:12.062627+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:42:19.764925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.064863+0000","last_change":"2024-06-29T15:39:12.061119+0000","last_active":"2024-06-29T15:39:12.064863+0000","last_peered":"2024-06-29T15:39:12.064863+0000","last_clean":"2024-06-29T15:39:12.064863+0000","last_became_active":"2024-06-29T15:39:12.060778+0000","last_became_peered":"2024-06-29T15:39:12.060778+0000","last_unstale":"2024-06-29T15:39:12.064863+0000","last_undegraded":"2024-06-29T15:39:12.064863+0000","last_fullsized":"2024-06-29T15:39:12.064863+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:25.227821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.041153+0000","last_change":"2024-06-29T15:39:12.040384+0000","last_active":"2024-06-29T15:39:12.041153+0000","last_peered":"2024-06-29T15:39:12.041153+0000","last_clean":"2024-06-29T15:39:12.041153+0000","last_became_active":"2024-06-29T15:39:12.040084+0000","last_became_peered":"2024-06-29T15:39:12.040084+0000","last_unstale":"2024-06-29T15:39:12.041153+0000","last_undegraded":"2024-06-29T15:39:12.041153+0000","last_fullsized":"2024-06-29T15:39:12.041153+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:44.134466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056272+0000","last_change":"2024-06-29T15:39:12.052031+0000","last_active":"2024-06-29T15:39:13.056272+0000","last_peered":"2024-06-29T15:39:13.056272+0000","last_clean":"2024-06-29T15:39:13.056272+0000","last_became_active":"2024-06-29T15:39:12.051573+0000","last_became_peered":"2024-06-29T15:39:12.051573+0000","last_unstale":"2024-06-29T15:39:13.056272+0000","last_undegraded":"2024-06-29T15:39:13.056272+0000","last_fullsized":"2024-06-29T15:39:13.056272+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:00.971422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135538+0000","last_change":"2024-06-29T15:39:12.058553+0000","last_active":"2024-06-29T15:39:13.135538+0000","last_peered":"2024-06-29T15:39:13.135538+0000","last_clean":"2024-06-29T15:39:13.135538+0000","last_became_active":"2024-06-29T15:39:12.058234+0000","last_became_peered":"2024-06-29T15:39:12.058234+0000","last_unstale":"2024-06-29T15:39:13.135538+0000","last_undegraded":"2024-06-29T15:39:13.135538+0000","last_fullsized":"2024-06-29T15:39:13.135538+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:52.670054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.061961+0000","last_change":"2024-06-29T15:39:12.061342+0000","last_active":"2024-06-29T15:39:12.061961+0000","last_peered":"2024-06-29T15:39:12.061961+0000","last_clean":"2024-06-29T15:39:12.061961+0000","last_became_active":"2024-06-29T15:39:12.060991+0000","last_became_peered":"2024-06-29T15:39:12.060991+0000","last_unstale":"2024-06-29T15:39:12.061961+0000","last_undegraded":"2024-06-29T15:39:12.061961+0000","last_fullsized":"2024-06-29T15:39:12.061961+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:56.464247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055992+0000","last_change":"2024-06-29T15:39:12.043562+0000","last_active":"2024-06-29T15:39:13.055992+0000","last_peered":"2024-06-29T15:39:13.055992+0000","last_clean":"2024-06-29T15:39:13.055992+0000","last_became_active":"2024-06-29T15:39:12.043218+0000","last_became_peered":"2024-06-29T15:39:12.043218+0000","last_unstale":"2024-06-29T15:39:13.055992+0000","last_undegraded":"2024-06-29T15:39:13.055992+0000","last_fullsized":"2024-06-29T15:39:13.055992+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:47.237775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.057150+0000","last_change":"2024-06-29T15:39:12.052957+0000","last_active":"2024-06-29T15:39:12.057150+0000","last_peered":"2024-06-29T15:39:12.057150+0000","last_clean":"2024-06-29T15:39:12.057150+0000","last_became_active":"2024-06-29T15:39:12.052578+0000","last_became_peered":"2024-06-29T15:39:12.052578+0000","last_unstale":"2024-06-29T15:39:12.057150+0000","last_undegraded":"2024-06-29T15:39:12.057150+0000","last_fullsized":"2024-06-29T15:39:12.057150+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:44.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.054925+0000","last_change":"2024-06-29T15:39:12.053274+0000","last_active":"2024-06-29T15:39:13.054925+0000","last_peered":"2024-06-29T15:39:13.054925+0000","last_clean":"2024-06-29T15:39:13.054925+0000","last_became_active":"2024-06-29T15:39:12.051865+0000","last_became_peered":"2024-06-29T15:39:12.051865+0000","last_unstale":"2024-06-29T15:39:13.054925+0000","last_undegraded":"2024-06-29T15:39:13.054925+0000","last_fullsized":"2024-06-29T15:39:13.054925+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:32.167435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055872+0000","last_change":"2024-06-29T15:39:12.050375+0000","last_active":"2024-06-29T15:39:13.055872+0000","last_peered":"2024-06-29T15:39:13.055872+0000","last_clean":"2024-06-29T15:39:13.055872+0000","last_became_active":"2024-06-29T15:39:12.050053+0000","last_became_peered":"2024-06-29T15:39:12.050053+0000","last_unstale":"2024-06-29T15:39:13.055872+0000","last_undegraded":"2024-06-29T15:39:13.055872+0000","last_fullsized":"2024-06-29T15:39:13.055872+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:36.261145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055000+0000","last_change":"2024-06-29T15:39:12.049996+0000","last_active":"2024-06-29T15:39:13.055000+0000","last_peered":"2024-06-29T15:39:13.055000+0000","last_clean":"2024-06-29T15:39:13.055000+0000","last_became_active":"2024-06-29T15:39:12.049580+0000","last_became_peered":"2024-06-29T15:39:12.049580+0000","last_unstale":"2024-06-29T15:39:13.055000+0000","last_undegraded":"2024-06-29T15:39:13.055000+0000","last_fullsized":"2024-06-29T15:39:13.055000+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:21:58.053003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.063987+0000","last_change":"2024-06-29T15:39:12.063069+0000","last_active":"2024-06-29T15:39:12.063987+0000","last_peered":"2024-06-29T15:39:12.063987+0000","last_clean":"2024-06-29T15:39:12.063987+0000","last_became_active":"2024-06-29T15:39:12.062333+0000","last_became_peered":"2024-06-29T15:39:12.062333+0000","last_unstale":"2024-06-29T15:39:12.063987+0000","last_undegraded":"2024-06-29T15:39:12.063987+0000","last_fullsized":"2024-06-29T15:39:12.063987+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:37.033072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"188'4","reported_seq":16,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.072800+0000","last_change":"2024-06-29T15:39:11.048920+0000","last_active":"2024-06-29T15:39:12.072800+0000","last_peered":"2024-06-29T15:39:12.072800+0000","last_clean":"2024-06-29T15:39:12.072800+0000","last_became_active":"2024-06-29T15:39:11.048586+0000","last_became_peered":"2024-06-29T15:39:11.048586+0000","last_unstale":"2024-06-29T15:39:12.072800+0000","last_undegraded":"2024-06-29T15:39:12.072800+0000","last_fullsized":"2024-06-29T15:39:12.072800+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:29.144496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055507+0000","last_change":"2024-06-29T15:39:12.046257+0000","last_active":"2024-06-29T15:39:13.055507+0000","last_peered":"2024-06-29T15:39:13.055507+0000","last_clean":"2024-06-29T15:39:13.055507+0000","last_became_active":"2024-06-29T15:39:12.045942+0000","last_became_peered":"2024-06-29T15:39:12.045942+0000","last_unstale":"2024-06-29T15:39:13.055507+0000","last_undegraded":"2024-06-29T15:39:13.055507+0000","last_fullsized":"2024-06-29T15:39:13.055507+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:37.129027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"188'1","reported_seq":13,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.073898+0000","last_change":"2024-06-29T15:39:11.058196+0000","last_active":"2024-06-29T15:39:12.073898+0000","last_peered":"2024-06-29T15:39:12.073898+0000","last_clean":"2024-06-29T15:39:12.073898+0000","last_became_active":"2024-06-29T15:39:11.057857+0000","last_became_peered":"2024-06-29T15:39:11.057857+0000","last_unstale":"2024-06-29T15:39:12.073898+0000","last_undegraded":"2024-06-29T15:39:12.073898+0000","last_fullsized":"2024-06-29T15:39:12.073898+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:16.760866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069128+0000","last_change":"2024-06-29T15:39:12.065859+0000","last_active":"2024-06-29T15:39:12.069128+0000","last_peered":"2024-06-29T15:39:12.069128+0000","last_clean":"2024-06-29T15:39:12.069128+0000","last_became_active":"2024-06-29T15:39:12.061304+0000","last_became_peered":"2024-06-29T15:39:12.061304+0000","last_unstale":"2024-06-29T15:39:12.069128+0000","last_undegraded":"2024-06-29T15:39:12.069128+0000","last_fullsized":"2024-06-29T15:39:12.069128+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:09.591584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"188'2","reported_seq":16,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056404+0000","last_change":"2024-06-29T15:39:11.053442+0000","last_active":"2024-06-29T15:39:13.056404+0000","last_peered":"2024-06-29T15:39:13.056404+0000","last_clean":"2024-06-29T15:39:13.056404+0000","last_became_active":"2024-06-29T15:39:11.053111+0000","last_became_peered":"2024-06-29T15:39:11.053111+0000","last_unstale":"2024-06-29T15:39:13.056404+0000","last_undegraded":"2024-06-29T15:39:13.056404+0000","last_fullsized":"2024-06-29T15:39:13.056404+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:47:42.511404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.067946+0000","last_change":"2024-06-29T15:39:12.067147+0000","last_active":"2024-06-29T15:39:12.067946+0000","last_peered":"2024-06-29T15:39:12.067946+0000","last_clean":"2024-06-29T15:39:12.067946+0000","last_became_active":"2024-06-29T15:39:12.066801+0000","last_became_peered":"2024-06-29T15:39:12.066801+0000","last_unstale":"2024-06-29T15:39:12.067946+0000","last_undegraded":"2024-06-29T15:39:12.067946+0000","last_fullsized":"2024-06-29T15:39:12.067946+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:04.667481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"188'2","reported_seq":16,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135319+0000","last_change":"2024-06-29T15:39:11.045162+0000","last_active":"2024-06-29T15:39:13.135319+0000","last_peered":"2024-06-29T15:39:13.135319+0000","last_clean":"2024-06-29T15:39:13.135319+0000","last_became_active":"2024-06-29T15:39:11.044791+0000","last_became_peered":"2024-06-29T15:39:11.044791+0000","last_unstale":"2024-06-29T15:39:13.135319+0000","last_undegraded":"2024-06-29T15:39:13.135319+0000","last_fullsized":"2024-06-29T15:39:13.135319+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:38.779317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052890+0000","last_change":"2024-06-29T15:39:12.062610+0000","last_active":"2024-06-29T15:39:13.052890+0000","last_peered":"2024-06-29T15:39:13.052890+0000","last_clean":"2024-06-29T15:39:13.052890+0000","last_became_active":"2024-06-29T15:39:12.062269+0000","last_became_peered":"2024-06-29T15:39:12.062269+0000","last_unstale":"2024-06-29T15:39:13.052890+0000","last_undegraded":"2024-06-29T15:39:13.052890+0000","last_fullsized":"2024-06-29T15:39:13.052890+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:13.847144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"188'1","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052777+0000","last_change":"2024-06-29T15:39:11.061738+0000","last_active":"2024-06-29T15:39:13.052777+0000","last_peered":"2024-06-29T15:39:13.052777+0000","last_clean":"2024-06-29T15:39:13.052777+0000","last_became_active":"2024-06-29T15:39:11.061385+0000","last_became_peered":"2024-06-29T15:39:11.061385+0000","last_unstale":"2024-06-29T15:39:13.052777+0000","last_undegraded":"2024-06-29T15:39:13.052777+0000","last_fullsized":"2024-06-29T15:39:13.052777+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:49.777466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135415+0000","last_change":"2024-06-29T15:39:12.057660+0000","last_active":"2024-06-29T15:39:13.135415+0000","last_peered":"2024-06-29T15:39:13.135415+0000","last_clean":"2024-06-29T15:39:13.135415+0000","last_became_active":"2024-06-29T15:39:12.057324+0000","last_became_peered":"2024-06-29T15:39:12.057324+0000","last_unstale":"2024-06-29T15:39:13.135415+0000","last_undegraded":"2024-06-29T15:39:13.135415+0000","last_fullsized":"2024-06-29T15:39:13.135415+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:21:57.650253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"188'1","reported_seq":13,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.076497+0000","last_change":"2024-06-29T15:39:11.058866+0000","last_active":"2024-06-29T15:39:12.076497+0000","last_peered":"2024-06-29T15:39:12.076497+0000","last_clean":"2024-06-29T15:39:12.076497+0000","last_became_active":"2024-06-29T15:39:11.058505+0000","last_became_peered":"2024-06-29T15:39:11.058505+0000","last_unstale":"2024-06-29T15:39:12.076497+0000","last_undegraded":"2024-06-29T15:39:12.076497+0000","last_fullsized":"2024-06-29T15:39:12.076497+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:54.840601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135648+0000","last_change":"2024-06-29T15:39:12.046442+0000","last_active":"2024-06-29T15:39:13.135648+0000","last_peered":"2024-06-29T15:39:13.135648+0000","last_clean":"2024-06-29T15:39:13.135648+0000","last_became_active":"2024-06-29T15:39:12.046108+0000","last_became_peered":"2024-06-29T15:39:12.046108+0000","last_unstale":"2024-06-29T15:39:13.135648+0000","last_undegraded":"2024-06-29T15:39:13.135648+0000","last_fullsized":"2024-06-29T15:39:13.135648+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:43:49.484877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135186+0000","last_change":"2024-06-29T15:39:11.050330+0000","last_active":"2024-06-29T15:39:13.135186+0000","last_peered":"2024-06-29T15:39:13.135186+0000","last_clean":"2024-06-29T15:39:13.135186+0000","last_became_active":"2024-06-29T15:39:11.049987+0000","last_became_peered":"2024-06-29T15:39:11.049987+0000","last_unstale":"2024-06-29T15:39:13.135186+0000","last_undegraded":"2024-06-29T15:39:13.135186+0000","last_fullsized":"2024-06-29T15:39:13.135186+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:28.627070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.058420+0000","last_change":"2024-06-29T15:39:12.056637+0000","last_active":"2024-06-29T15:39:12.058420+0000","last_peered":"2024-06-29T15:39:12.058420+0000","last_clean":"2024-06-29T15:39:12.058420+0000","last_became_active":"2024-06-29T15:39:12.056027+0000","last_became_peered":"2024-06-29T15:39:12.056027+0000","last_unstale":"2024-06-29T15:39:12.058420+0000","last_undegraded":"2024-06-29T15:39:12.058420+0000","last_fullsized":"2024-06-29T15:39:12.058420+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:36.971914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052572+0000","last_change":"2024-06-29T15:39:11.053350+0000","last_active":"2024-06-29T15:39:13.052572+0000","last_peered":"2024-06-29T15:39:13.052572+0000","last_clean":"2024-06-29T15:39:13.052572+0000","last_became_active":"2024-06-29T15:39:11.053012+0000","last_became_peered":"2024-06-29T15:39:11.053012+0000","last_unstale":"2024-06-29T15:39:13.052572+0000","last_undegraded":"2024-06-29T15:39:13.052572+0000","last_fullsized":"2024-06-29T15:39:13.052572+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:52.030756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.063964+0000","last_change":"2024-06-29T15:39:12.062973+0000","last_active":"2024-06-29T15:39:12.063964+0000","last_peered":"2024-06-29T15:39:12.063964+0000","last_clean":"2024-06-29T15:39:12.063964+0000","last_became_active":"2024-06-29T15:39:12.062626+0000","last_became_peered":"2024-06-29T15:39:12.062626+0000","last_unstale":"2024-06-29T15:39:12.063964+0000","last_undegraded":"2024-06-29T15:39:12.063964+0000","last_fullsized":"2024-06-29T15:39:12.063964+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:57.350541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056222+0000","last_change":"2024-06-29T15:39:11.056609+0000","last_active":"2024-06-29T15:39:13.056222+0000","last_peered":"2024-06-29T15:39:13.056222+0000","last_clean":"2024-06-29T15:39:13.056222+0000","last_became_active":"2024-06-29T15:39:11.055487+0000","last_became_peered":"2024-06-29T15:39:11.055487+0000","last_unstale":"2024-06-29T15:39:13.056222+0000","last_undegraded":"2024-06-29T15:39:13.056222+0000","last_fullsized":"2024-06-29T15:39:13.056222+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:29.801478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055574+0000","last_change":"2024-06-29T15:39:12.048202+0000","last_active":"2024-06-29T15:39:13.055574+0000","last_peered":"2024-06-29T15:39:13.055574+0000","last_clean":"2024-06-29T15:39:13.055574+0000","last_became_active":"2024-06-29T15:39:12.047848+0000","last_became_peered":"2024-06-29T15:39:12.047848+0000","last_unstale":"2024-06-29T15:39:13.055574+0000","last_undegraded":"2024-06-29T15:39:13.055574+0000","last_fullsized":"2024-06-29T15:39:13.055574+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:57.346505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"188'3","reported_seq":15,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069970+0000","last_change":"2024-06-29T15:39:11.045344+0000","last_active":"2024-06-29T15:39:12.069970+0000","last_peered":"2024-06-29T15:39:12.069970+0000","last_clean":"2024-06-29T15:39:12.069970+0000","last_became_active":"2024-06-29T15:39:11.045007+0000","last_became_peered":"2024-06-29T15:39:11.045007+0000","last_unstale":"2024-06-29T15:39:12.069970+0000","last_undegraded":"2024-06-29T15:39:12.069970+0000","last_fullsized":"2024-06-29T15:39:12.069970+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:04.984846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135498+0000","last_change":"2024-06-29T15:39:12.055384+0000","last_active":"2024-06-29T15:39:13.135498+0000","last_peered":"2024-06-29T15:39:13.135498+0000","last_clean":"2024-06-29T15:39:13.135498+0000","last_became_active":"2024-06-29T15:39:12.054982+0000","last_became_peered":"2024-06-29T15:39:12.054982+0000","last_unstale":"2024-06-29T15:39:13.135498+0000","last_undegraded":"2024-06-29T15:39:13.135498+0000","last_fullsized":"2024-06-29T15:39:13.135498+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:09:05.612525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"188'6","reported_seq":20,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056103+0000","last_change":"2024-06-29T15:39:11.056112+0000","last_active":"2024-06-29T15:39:13.056103+0000","last_peered":"2024-06-29T15:39:13.056103+0000","last_clean":"2024-06-29T15:39:13.056103+0000","last_became_active":"2024-06-29T15:39:11.055775+0000","last_became_peered":"2024-06-29T15:39:11.055775+0000","last_unstale":"2024-06-29T15:39:13.056103+0000","last_undegraded":"2024-06-29T15:39:13.056103+0000","last_fullsized":"2024-06-29T15:39:13.056103+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:32.908998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.054949+0000","last_change":"2024-06-29T15:39:12.050363+0000","last_active":"2024-06-29T15:39:13.054949+0000","last_peered":"2024-06-29T15:39:13.054949+0000","last_clean":"2024-06-29T15:39:13.054949+0000","last_became_active":"2024-06-29T15:39:12.049970+0000","last_became_peered":"2024-06-29T15:39:12.049970+0000","last_unstale":"2024-06-29T15:39:13.054949+0000","last_undegraded":"2024-06-29T15:39:13.054949+0000","last_fullsized":"2024-06-29T15:39:13.054949+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.679970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"188'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.074655+0000","last_change":"2024-06-29T15:39:11.049108+0000","last_active":"2024-06-29T15:39:12.074655+0000","last_peered":"2024-06-29T15:39:12.074655+0000","last_clean":"2024-06-29T15:39:12.074655+0000","last_became_active":"2024-06-29T15:39:11.048312+0000","last_became_peered":"2024-06-29T15:39:11.048312+0000","last_unstale":"2024-06-29T15:39:12.074655+0000","last_undegraded":"2024-06-29T15:39:12.074655+0000","last_fullsized":"2024-06-29T15:39:12.074655+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:11:52.252830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053030+0000","last_change":"2024-06-29T15:39:12.052634+0000","last_active":"2024-06-29T15:39:13.053030+0000","last_peered":"2024-06-29T15:39:13.053030+0000","last_clean":"2024-06-29T15:39:13.053030+0000","last_became_active":"2024-06-29T15:39:12.052249+0000","last_became_peered":"2024-06-29T15:39:12.052249+0000","last_unstale":"2024-06-29T15:39:13.053030+0000","last_undegraded":"2024-06-29T15:39:13.053030+0000","last_fullsized":"2024-06-29T15:39:13.053030+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:59:00.019618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055644+0000","last_change":"2024-06-29T15:39:11.050832+0000","last_active":"2024-06-29T15:39:13.055644+0000","last_peered":"2024-06-29T15:39:13.055644+0000","last_clean":"2024-06-29T15:39:13.055644+0000","last_became_active":"2024-06-29T15:39:11.050273+0000","last_became_peered":"2024-06-29T15:39:11.050273+0000","last_unstale":"2024-06-29T15:39:13.055644+0000","last_undegraded":"2024-06-29T15:39:13.055644+0000","last_fullsized":"2024-06-29T15:39:13.055644+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:32.140104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056212+0000","last_change":"2024-06-29T15:39:12.055589+0000","last_active":"2024-06-29T15:39:13.056212+0000","last_peered":"2024-06-29T15:39:13.056212+0000","last_clean":"2024-06-29T15:39:13.056212+0000","last_became_active":"2024-06-29T15:39:12.055280+0000","last_became_peered":"2024-06-29T15:39:12.055280+0000","last_unstale":"2024-06-29T15:39:13.056212+0000","last_undegraded":"2024-06-29T15:39:13.056212+0000","last_fullsized":"2024-06-29T15:39:13.056212+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:29:30.357235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055975+0000","last_change":"2024-06-29T15:39:11.048698+0000","last_active":"2024-06-29T15:39:13.055975+0000","last_peered":"2024-06-29T15:39:13.055975+0000","last_clean":"2024-06-29T15:39:13.055975+0000","last_became_active":"2024-06-29T15:39:11.048333+0000","last_became_peered":"2024-06-29T15:39:11.048333+0000","last_unstale":"2024-06-29T15:39:13.055975+0000","last_undegraded":"2024-06-29T15:39:13.055975+0000","last_fullsized":"2024-06-29T15:39:13.055975+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:02:38.352100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135595+0000","last_change":"2024-06-29T15:39:12.059109+0000","last_active":"2024-06-29T15:39:13.135595+0000","last_peered":"2024-06-29T15:39:13.135595+0000","last_clean":"2024-06-29T15:39:13.135595+0000","last_became_active":"2024-06-29T15:39:12.058516+0000","last_became_peered":"2024-06-29T15:39:12.058516+0000","last_unstale":"2024-06-29T15:39:13.135595+0000","last_undegraded":"2024-06-29T15:39:13.135595+0000","last_fullsized":"2024-06-29T15:39:13.135595+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:45.712612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"188'8","reported_seq":22,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052570+0000","last_change":"2024-06-29T15:39:11.060192+0000","last_active":"2024-06-29T15:39:13.052570+0000","last_peered":"2024-06-29T15:39:13.052570+0000","last_clean":"2024-06-29T15:39:13.052570+0000","last_became_active":"2024-06-29T15:39:11.059847+0000","last_became_peered":"2024-06-29T15:39:11.059847+0000","last_unstale":"2024-06-29T15:39:13.052570+0000","last_undegraded":"2024-06-29T15:39:13.052570+0000","last_fullsized":"2024-06-29T15:39:13.052570+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:47.347264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052427+0000","last_change":"2024-06-29T15:39:12.055760+0000","last_active":"2024-06-29T15:39:13.052427+0000","last_peered":"2024-06-29T15:39:13.052427+0000","last_clean":"2024-06-29T15:39:13.052427+0000","last_became_active":"2024-06-29T15:39:12.055435+0000","last_became_peered":"2024-06-29T15:39:12.055435+0000","last_unstale":"2024-06-29T15:39:13.052427+0000","last_undegraded":"2024-06-29T15:39:13.052427+0000","last_fullsized":"2024-06-29T15:39:13.052427+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:19.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"188'1","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056347+0000","last_change":"2024-06-29T15:39:11.049084+0000","last_active":"2024-06-29T15:39:13.056347+0000","last_peered":"2024-06-29T15:39:13.056347+0000","last_clean":"2024-06-29T15:39:13.056347+0000","last_became_active":"2024-06-29T15:39:11.048781+0000","last_became_peered":"2024-06-29T15:39:11.048781+0000","last_unstale":"2024-06-29T15:39:13.056347+0000","last_undegraded":"2024-06-29T15:39:13.056347+0000","last_fullsized":"2024-06-29T15:39:13.056347+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:26.683092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055320+0000","last_change":"2024-06-29T15:39:12.052024+0000","last_active":"2024-06-29T15:39:13.055320+0000","last_peered":"2024-06-29T15:39:13.055320+0000","last_clean":"2024-06-29T15:39:13.055320+0000","last_became_active":"2024-06-29T15:39:12.051439+0000","last_became_peered":"2024-06-29T15:39:12.051439+0000","last_unstale":"2024-06-29T15:39:13.055320+0000","last_undegraded":"2024-06-29T15:39:13.055320+0000","last_fullsized":"2024-06-29T15:39:13.055320+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:28.800208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"188'2","reported_seq":16,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053244+0000","last_change":"2024-06-29T15:39:11.055274+0000","last_active":"2024-06-29T15:39:13.053244+0000","last_peered":"2024-06-29T15:39:13.053244+0000","last_clean":"2024-06-29T15:39:13.053244+0000","last_became_active":"2024-06-29T15:39:11.054944+0000","last_became_peered":"2024-06-29T15:39:11.054944+0000","last_unstale":"2024-06-29T15:39:13.053244+0000","last_undegraded":"2024-06-29T15:39:13.053244+0000","last_fullsized":"2024-06-29T15:39:13.053244+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:46.538589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056051+0000","last_change":"2024-06-29T15:39:12.057058+0000","last_active":"2024-06-29T15:39:13.056051+0000","last_peered":"2024-06-29T15:39:13.056051+0000","last_clean":"2024-06-29T15:39:13.056051+0000","last_became_active":"2024-06-29T15:39:12.056473+0000","last_became_peered":"2024-06-29T15:39:12.056473+0000","last_unstale":"2024-06-29T15:39:13.056051+0000","last_undegraded":"2024-06-29T15:39:13.056051+0000","last_fullsized":"2024-06-29T15:39:13.056051+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:13:27.195144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"188'1","reported_seq":13,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.074257+0000","last_change":"2024-06-29T15:39:11.051497+0000","last_active":"2024-06-29T15:39:12.074257+0000","last_peered":"2024-06-29T15:39:12.074257+0000","last_clean":"2024-06-29T15:39:12.074257+0000","last_became_active":"2024-06-29T15:39:11.048355+0000","last_became_peered":"2024-06-29T15:39:11.048355+0000","last_unstale":"2024-06-29T15:39:12.074257+0000","last_undegraded":"2024-06-29T15:39:12.074257+0000","last_fullsized":"2024-06-29T15:39:12.074257+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:57.315377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056064+0000","last_change":"2024-06-29T15:39:12.062507+0000","last_active":"2024-06-29T15:39:13.056064+0000","last_peered":"2024-06-29T15:39:13.056064+0000","last_clean":"2024-06-29T15:39:13.056064+0000","last_became_active":"2024-06-29T15:39:12.062176+0000","last_became_peered":"2024-06-29T15:39:12.062176+0000","last_unstale":"2024-06-29T15:39:13.056064+0000","last_undegraded":"2024-06-29T15:39:13.056064+0000","last_fullsized":"2024-06-29T15:39:13.056064+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:58:06.593426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052990+0000","last_change":"2024-06-29T15:39:11.061085+0000","last_active":"2024-06-29T15:39:13.052990+0000","last_peered":"2024-06-29T15:39:13.052990+0000","last_clean":"2024-06-29T15:39:13.052990+0000","last_became_active":"2024-06-29T15:39:11.060749+0000","last_became_peered":"2024-06-29T15:39:11.060749+0000","last_unstale":"2024-06-29T15:39:13.052990+0000","last_undegraded":"2024-06-29T15:39:13.052990+0000","last_fullsized":"2024-06-29T15:39:13.052990+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:02:03.805873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056027+0000","last_change":"2024-06-29T15:39:12.045265+0000","last_active":"2024-06-29T15:39:13.056027+0000","last_peered":"2024-06-29T15:39:13.056027+0000","last_clean":"2024-06-29T15:39:13.056027+0000","last_became_active":"2024-06-29T15:39:12.044966+0000","last_became_peered":"2024-06-29T15:39:12.044966+0000","last_unstale":"2024-06-29T15:39:13.056027+0000","last_undegraded":"2024-06-29T15:39:13.056027+0000","last_fullsized":"2024-06-29T15:39:13.056027+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:09.072208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"188'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.074547+0000","last_change":"2024-06-29T15:39:11.048845+0000","last_active":"2024-06-29T15:39:12.074547+0000","last_peered":"2024-06-29T15:39:12.074547+0000","last_clean":"2024-06-29T15:39:12.074547+0000","last_became_active":"2024-06-29T15:39:11.047900+0000","last_became_peered":"2024-06-29T15:39:11.047900+0000","last_unstale":"2024-06-29T15:39:12.074547+0000","last_undegraded":"2024-06-29T15:39:12.074547+0000","last_fullsized":"2024-06-29T15:39:12.074547+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:04.559196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069066+0000","last_change":"2024-06-29T15:39:12.065944+0000","last_active":"2024-06-29T15:39:12.069066+0000","last_peered":"2024-06-29T15:39:12.069066+0000","last_clean":"2024-06-29T15:39:12.069066+0000","last_became_active":"2024-06-29T15:39:12.065535+0000","last_became_peered":"2024-06-29T15:39:12.065535+0000","last_unstale":"2024-06-29T15:39:12.069066+0000","last_undegraded":"2024-06-29T15:39:12.069066+0000","last_fullsized":"2024-06-29T15:39:12.069066+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:43:08.398702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055685+0000","last_change":"2024-06-29T15:39:11.049558+0000","last_active":"2024-06-29T15:39:13.055685+0000","last_peered":"2024-06-29T15:39:13.055685+0000","last_clean":"2024-06-29T15:39:13.055685+0000","last_became_active":"2024-06-29T15:39:11.049248+0000","last_became_peered":"2024-06-29T15:39:11.049248+0000","last_unstale":"2024-06-29T15:39:13.055685+0000","last_undegraded":"2024-06-29T15:39:13.055685+0000","last_fullsized":"2024-06-29T15:39:13.055685+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:53.959844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069074+0000","last_change":"2024-06-29T15:39:12.064654+0000","last_active":"2024-06-29T15:39:12.069074+0000","last_peered":"2024-06-29T15:39:12.069074+0000","last_clean":"2024-06-29T15:39:12.069074+0000","last_became_active":"2024-06-29T15:39:12.064307+0000","last_became_peered":"2024-06-29T15:39:12.064307+0000","last_unstale":"2024-06-29T15:39:12.069074+0000","last_undegraded":"2024-06-29T15:39:12.069074+0000","last_fullsized":"2024-06-29T15:39:12.069074+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:02:41.355433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135345+0000","last_change":"2024-06-29T15:39:11.044706+0000","last_active":"2024-06-29T15:39:13.135345+0000","last_peered":"2024-06-29T15:39:13.135345+0000","last_clean":"2024-06-29T15:39:13.135345+0000","last_became_active":"2024-06-29T15:39:11.044352+0000","last_became_peered":"2024-06-29T15:39:11.044352+0000","last_unstale":"2024-06-29T15:39:13.135345+0000","last_undegraded":"2024-06-29T15:39:13.135345+0000","last_fullsized":"2024-06-29T15:39:13.135345+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:22.385508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.054658+0000","last_change":"2024-06-29T15:39:12.052175+0000","last_active":"2024-06-29T15:39:12.054658+0000","last_peered":"2024-06-29T15:39:12.054658+0000","last_clean":"2024-06-29T15:39:12.054658+0000","last_became_active":"2024-06-29T15:39:12.051841+0000","last_became_peered":"2024-06-29T15:39:12.051841+0000","last_unstale":"2024-06-29T15:39:12.054658+0000","last_undegraded":"2024-06-29T15:39:12.054658+0000","last_fullsized":"2024-06-29T15:39:12.054658+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:31.505068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.044363+0000","last_change":"2024-06-29T15:39:11.047344+0000","last_active":"2024-06-29T15:39:12.044363+0000","last_peered":"2024-06-29T15:39:12.044363+0000","last_clean":"2024-06-29T15:39:12.044363+0000","last_became_active":"2024-06-29T15:39:11.046984+0000","last_became_peered":"2024-06-29T15:39:11.046984+0000","last_unstale":"2024-06-29T15:39:12.044363+0000","last_undegraded":"2024-06-29T15:39:12.044363+0000","last_fullsized":"2024-06-29T15:39:12.044363+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:37.637383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055626+0000","last_change":"2024-06-29T15:39:12.050565+0000","last_active":"2024-06-29T15:39:13.055626+0000","last_peered":"2024-06-29T15:39:13.055626+0000","last_clean":"2024-06-29T15:39:13.055626+0000","last_became_active":"2024-06-29T15:39:12.050265+0000","last_became_peered":"2024-06-29T15:39:12.050265+0000","last_unstale":"2024-06-29T15:39:13.055626+0000","last_undegraded":"2024-06-29T15:39:13.055626+0000","last_fullsized":"2024-06-29T15:39:13.055626+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:04.500983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"188'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.074151+0000","last_change":"2024-06-29T15:39:11.049704+0000","last_active":"2024-06-29T15:39:12.074151+0000","last_peered":"2024-06-29T15:39:12.074151+0000","last_clean":"2024-06-29T15:39:12.074151+0000","last_became_active":"2024-06-29T15:39:11.049397+0000","last_became_peered":"2024-06-29T15:39:11.049397+0000","last_unstale":"2024-06-29T15:39:12.074151+0000","last_undegraded":"2024-06-29T15:39:12.074151+0000","last_fullsized":"2024-06-29T15:39:12.074151+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:12.354833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052551+0000","last_change":"2024-06-29T15:39:12.056326+0000","last_active":"2024-06-29T15:39:13.052551+0000","last_peered":"2024-06-29T15:39:13.052551+0000","last_clean":"2024-06-29T15:39:13.052551+0000","last_became_active":"2024-06-29T15:39:12.053118+0000","last_became_peered":"2024-06-29T15:39:12.053118+0000","last_unstale":"2024-06-29T15:39:13.052551+0000","last_undegraded":"2024-06-29T15:39:13.052551+0000","last_fullsized":"2024-06-29T15:39:13.052551+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:25.725466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"189'1","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.057349+0000","last_change":"2024-06-29T15:39:11.056070+0000","last_active":"2024-06-29T15:39:13.057349+0000","last_peered":"2024-06-29T15:39:13.057349+0000","last_clean":"2024-06-29T15:39:13.057349+0000","last_became_active":"2024-06-29T15:39:11.055515+0000","last_became_peered":"2024-06-29T15:39:11.055515+0000","last_unstale":"2024-06-29T15:39:13.057349+0000","last_undegraded":"2024-06-29T15:39:13.057349+0000","last_fullsized":"2024-06-29T15:39:13.057349+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:03.494059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135309+0000","last_change":"2024-06-29T15:39:12.046583+0000","last_active":"2024-06-29T15:39:13.135309+0000","last_peered":"2024-06-29T15:39:13.135309+0000","last_clean":"2024-06-29T15:39:13.135309+0000","last_became_active":"2024-06-29T15:39:12.045167+0000","last_became_peered":"2024-06-29T15:39:12.045167+0000","last_unstale":"2024-06-29T15:39:13.135309+0000","last_undegraded":"2024-06-29T15:39:13.135309+0000","last_fullsized":"2024-06-29T15:39:13.135309+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:28.541358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056003+0000","last_change":"2024-06-29T15:39:11.054037+0000","last_active":"2024-06-29T15:39:13.056003+0000","last_peered":"2024-06-29T15:39:13.056003+0000","last_clean":"2024-06-29T15:39:13.056003+0000","last_became_active":"2024-06-29T15:39:11.053729+0000","last_became_peered":"2024-06-29T15:39:11.053729+0000","last_unstale":"2024-06-29T15:39:13.056003+0000","last_undegraded":"2024-06-29T15:39:13.056003+0000","last_fullsized":"2024-06-29T15:39:13.056003+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:45.609828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052654+0000","last_change":"2024-06-29T15:39:12.056983+0000","last_active":"2024-06-29T15:39:13.052654+0000","last_peered":"2024-06-29T15:39:13.052654+0000","last_clean":"2024-06-29T15:39:13.052654+0000","last_became_active":"2024-06-29T15:39:12.056651+0000","last_became_peered":"2024-06-29T15:39:12.056651+0000","last_unstale":"2024-06-29T15:39:13.052654+0000","last_undegraded":"2024-06-29T15:39:13.052654+0000","last_fullsized":"2024-06-29T15:39:13.052654+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:11.902800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135403+0000","last_change":"2024-06-29T15:39:11.056551+0000","last_active":"2024-06-29T15:39:13.135403+0000","last_peered":"2024-06-29T15:39:13.135403+0000","last_clean":"2024-06-29T15:39:13.135403+0000","last_became_active":"2024-06-29T15:39:11.056230+0000","last_became_peered":"2024-06-29T15:39:11.056230+0000","last_unstale":"2024-06-29T15:39:13.135403+0000","last_undegraded":"2024-06-29T15:39:13.135403+0000","last_fullsized":"2024-06-29T15:39:13.135403+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:45.097306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.058521+0000","last_change":"2024-06-29T15:39:12.057359+0000","last_active":"2024-06-29T15:39:12.058521+0000","last_peered":"2024-06-29T15:39:12.058521+0000","last_clean":"2024-06-29T15:39:12.058521+0000","last_became_active":"2024-06-29T15:39:12.057014+0000","last_became_peered":"2024-06-29T15:39:12.057014+0000","last_unstale":"2024-06-29T15:39:12.058521+0000","last_undegraded":"2024-06-29T15:39:12.058521+0000","last_fullsized":"2024-06-29T15:39:12.058521+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:25:07.655904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.036099+0000","last_change":"2024-06-29T15:39:11.050581+0000","last_active":"2024-06-29T15:39:12.036099+0000","last_peered":"2024-06-29T15:39:12.036099+0000","last_clean":"2024-06-29T15:39:12.036099+0000","last_became_active":"2024-06-29T15:39:11.050183+0000","last_became_peered":"2024-06-29T15:39:11.050183+0000","last_unstale":"2024-06-29T15:39:12.036099+0000","last_undegraded":"2024-06-29T15:39:12.036099+0000","last_fullsized":"2024-06-29T15:39:12.036099+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:02.389518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.064190+0000","last_change":"2024-06-29T15:39:12.063539+0000","last_active":"2024-06-29T15:39:12.064190+0000","last_peered":"2024-06-29T15:39:12.064190+0000","last_clean":"2024-06-29T15:39:12.064190+0000","last_became_active":"2024-06-29T15:39:12.062342+0000","last_became_peered":"2024-06-29T15:39:12.062342+0000","last_unstale":"2024-06-29T15:39:12.064190+0000","last_undegraded":"2024-06-29T15:39:12.064190+0000","last_fullsized":"2024-06-29T15:39:12.064190+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:44.336003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055698+0000","last_change":"2024-06-29T15:39:11.047466+0000","last_active":"2024-06-29T15:39:13.055698+0000","last_peered":"2024-06-29T15:39:13.055698+0000","last_clean":"2024-06-29T15:39:13.055698+0000","last_became_active":"2024-06-29T15:39:11.047148+0000","last_became_peered":"2024-06-29T15:39:11.047148+0000","last_unstale":"2024-06-29T15:39:13.055698+0000","last_undegraded":"2024-06-29T15:39:13.055698+0000","last_fullsized":"2024-06-29T15:39:13.055698+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:15.145621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056046+0000","last_change":"2024-06-29T15:39:12.044506+0000","last_active":"2024-06-29T15:39:13.056046+0000","last_peered":"2024-06-29T15:39:13.056046+0000","last_clean":"2024-06-29T15:39:13.056046+0000","last_became_active":"2024-06-29T15:39:12.044158+0000","last_became_peered":"2024-06-29T15:39:12.044158+0000","last_unstale":"2024-06-29T15:39:13.056046+0000","last_undegraded":"2024-06-29T15:39:13.056046+0000","last_fullsized":"2024-06-29T15:39:13.056046+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:15.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056121+0000","last_change":"2024-06-29T15:39:11.051576+0000","last_active":"2024-06-29T15:39:13.056121+0000","last_peered":"2024-06-29T15:39:13.056121+0000","last_clean":"2024-06-29T15:39:13.056121+0000","last_became_active":"2024-06-29T15:39:11.051337+0000","last_became_peered":"2024-06-29T15:39:11.051337+0000","last_unstale":"2024-06-29T15:39:13.056121+0000","last_undegraded":"2024-06-29T15:39:13.056121+0000","last_fullsized":"2024-06-29T15:39:13.056121+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:20.183519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135551+0000","last_change":"2024-06-29T15:39:12.058842+0000","last_active":"2024-06-29T15:39:13.135551+0000","last_peered":"2024-06-29T15:39:13.135551+0000","last_clean":"2024-06-29T15:39:13.135551+0000","last_became_active":"2024-06-29T15:39:12.058100+0000","last_became_peered":"2024-06-29T15:39:12.058100+0000","last_unstale":"2024-06-29T15:39:13.135551+0000","last_undegraded":"2024-06-29T15:39:13.135551+0000","last_fullsized":"2024-06-29T15:39:13.135551+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:42:53.020860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.045899+0000","last_change":"2024-06-29T15:39:11.047342+0000","last_active":"2024-06-29T15:39:12.045899+0000","last_peered":"2024-06-29T15:39:12.045899+0000","last_clean":"2024-06-29T15:39:12.045899+0000","last_became_active":"2024-06-29T15:39:11.047043+0000","last_became_peered":"2024-06-29T15:39:11.047043+0000","last_unstale":"2024-06-29T15:39:12.045899+0000","last_undegraded":"2024-06-29T15:39:12.045899+0000","last_fullsized":"2024-06-29T15:39:12.045899+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:35.832471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.051922+0000","last_change":"2024-06-29T15:39:12.050802+0000","last_active":"2024-06-29T15:39:12.051922+0000","last_peered":"2024-06-29T15:39:12.051922+0000","last_clean":"2024-06-29T15:39:12.051922+0000","last_became_active":"2024-06-29T15:39:12.050335+0000","last_became_peered":"2024-06-29T15:39:12.050335+0000","last_unstale":"2024-06-29T15:39:12.051922+0000","last_undegraded":"2024-06-29T15:39:12.051922+0000","last_fullsized":"2024-06-29T15:39:12.051922+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:41:01.929978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055965+0000","last_change":"2024-06-29T15:39:11.056912+0000","last_active":"2024-06-29T15:39:13.055965+0000","last_peered":"2024-06-29T15:39:13.055965+0000","last_clean":"2024-06-29T15:39:13.055965+0000","last_became_active":"2024-06-29T15:39:11.056591+0000","last_became_peered":"2024-06-29T15:39:11.056591+0000","last_unstale":"2024-06-29T15:39:13.055965+0000","last_undegraded":"2024-06-29T15:39:13.055965+0000","last_fullsized":"2024-06-29T15:39:13.055965+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:01:18.669685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"178'198","reported_seq":905,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.044474+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:39:12.044474+0000","last_peered":"2024-06-29T15:39:12.044474+0000","last_clean":"2024-06-29T15:39:12.044474+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:39:12.044474+0000","last_undegraded":"2024-06-29T15:39:12.044474+0000","last_fullsized":"2024-06-29T15:39:12.044474+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1146,"num_read_kb":3223,"num_write":317,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1146,"num_read_kb":3223,"num_write":317,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198,"ondisk_log_size":198,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543071,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47940,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323900,"statfs":{"total":96636764160,"available":96587673600,"internally_reserved":0,"allocated":2940928,"data_stored":1788324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41792,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330048,"statfs":{"total":96636764160,"available":96593969152,"internally_reserved":0,"allocated":2347008,"data_stored":1197956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543071,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47380,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324460,"statfs":{"total":96636764160,"available":96588247040,"internally_reserved":0,"allocated":2367488,"data_stored":1204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543070,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29328,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342512,"statfs":{"total":96636764160,"available":96606732288,"internally_reserved":0,"allocated":2363392,"data_stored":1198720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510368,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59880,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311960,"statfs":{"total":96636764160,"available":96575447040,"internally_reserved":0,"allocated":2379776,"data_stored":1206892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575776,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64000,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307840,"statfs":{"total":96636764160,"available":96571228160,"internally_reserved":0,"allocated":2347008,"data_stored":1197956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510367,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59872,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311968,"statfs":{"total":96636764160,"available":96575455232,"internally_reserved":0,"allocated":2371584,"data_stored":1206288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510369,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":2949120,"data_stored":1788428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:39:16.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:39:16.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.292+0000 7fafab5f0640 1 -- 172.21.15.59:0/2252750993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafa40cc060 msgr2=0x7fafa40acfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.292+0000 7fafab5f0640 1 --2- 172.21.15.59:0/2252750993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafa40cc060 0x7fafa40acfa0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faf98005fd0 tx=0x7faf9802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.292+0000 7fafab5f0640 1 -- 172.21.15.59:0/2252750993 shutdown_connections 2024-06-29T15:39:16.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.292+0000 7fafab5f0640 1 --2- 172.21.15.59:0/2252750993 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafa40cc060 0x7fafa40acfa0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.292+0000 7fafab5f0640 1 --2- 172.21.15.59:0/2252750993 >> v2:172.21.15.153:3300/0 conn(0x7fafa40cb490 0x7fafa4073be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.292+0000 7fafab5f0640 1 -- 172.21.15.59:0/2252750993 >> 172.21.15.59:0/2252750993 conn(0x7fafa40b1ea0 msgr2=0x7fafa40b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.292+0000 7fafab5f0640 1 -- 172.21.15.59:0/2252750993 shutdown_connections 2024-06-29T15:39:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.292+0000 7fafab5f0640 1 -- 172.21.15.59:0/2252750993 wait complete. 2024-06-29T15:39:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.293+0000 7fafab5f0640 1 Processor -- start 2024-06-29T15:39:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.294+0000 7fafab5f0640 1 -- start start 2024-06-29T15:39:16.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafab5f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafa40ad4e0 0x7fafa4153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafab5f0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa40cb490 0x7fafa409c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafab5f0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafa40cc060 0x7fafa40c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafab5f0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fafa40ce8d0 con 0x7fafa40cc060 2024-06-29T15:39:16.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafab5f0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafa40ce750 con 0x7fafa40cb490 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafab5f0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fafa40cea50 con 0x7fafa40ad4e0 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa9b8d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafa40cc060 0x7fafa40c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa9b8d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafa40cc060 0x7fafa40c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52898/0 (socket says 172.21.15.59:52898) 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa9b8d640 1 -- 172.21.15.59:0/1340309146 learned_addr learned my addr 172.21.15.59:0/1340309146 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa938c640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafa40ad4e0 0x7fafa4153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa8b8b640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa40cb490 0x7fafa409c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa9b8d640 1 -- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafa40ad4e0 msgr2=0x7fafa4153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa9b8d640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafa40ad4e0 0x7fafa4153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa9b8d640 1 -- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa40cb490 msgr2=0x7fafa409c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa9b8d640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa40cb490 0x7fafa409c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.295+0000 7fafa9b8d640 1 -- 172.21.15.59:0/1340309146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa409da10 con 0x7fafa40cc060 2024-06-29T15:39:16.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.296+0000 7fafa9b8d640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafa40cc060 0x7fafa40c9340 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00c3c0 tx=0x7faf9c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.296+0000 7fafa1ffb640 1 -- 172.21.15.59:0/1340309146 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c0052d0 con 0x7fafa40cc060 2024-06-29T15:39:16.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.296+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa415c550 con 0x7fafa40cc060 2024-06-29T15:39:16.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.296+0000 7fafa1ffb640 1 -- 172.21.15.59:0/1340309146 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf9c007020 con 0x7fafa40cc060 2024-06-29T15:39:16.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.296+0000 7fafa1ffb640 1 -- 172.21.15.59:0/1340309146 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c007470 con 0x7fafa40cc060 2024-06-29T15:39:16.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.296+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa415cb10 con 0x7fafa40cc060 2024-06-29T15:39:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.297+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa4073be0 con 0x7fafa40cc060 2024-06-29T15:39:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.298+0000 7fafa1ffb640 1 -- 172.21.15.59:0/1340309146 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7faf9c0067d0 con 0x7fafa40cc060 2024-06-29T15:39:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.298+0000 7fafa1ffb640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faf900763f0 0x7faf90078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.298+0000 7fafa1ffb640 1 -- 172.21.15.59:0/1340309146 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7faf9c096e90 con 0x7fafa40cc060 2024-06-29T15:39:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.299+0000 7fafa938c640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faf900763f0 0x7faf90078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.299+0000 7fafa938c640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faf900763f0 0x7faf90078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faf94005fd0 tx=0x7faf94009500 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.303+0000 7fafa1ffb640 1 -- 172.21.15.59:0/1340309146 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fafa40ad8f0 con 0x7fafa40cc060 2024-06-29T15:39:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.491+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 --> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fafa40cc780 con 0x7faf900763f0 2024-06-29T15:39:16.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.496+0000 7fafa1ffb640 1 -- 172.21.15.59:0/1340309146 <== mgr.9505 v2:172.21.15.153:6832/1270545814 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252462 (secure 0 0 0) 0x7fafa40cc780 con 0x7faf900763f0 2024-06-29T15:39:16.499 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:16.501 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:39:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faf900763f0 msgr2=0x7faf90078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faf900763f0 0x7faf90078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faf94005fd0 tx=0x7faf94009500 comp rx=0 tx=0).stop 2024-06-29T15:39:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafa40cc060 msgr2=0x7fafa40c9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafa40cc060 0x7fafa40c9340 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00c3c0 tx=0x7faf9c00c880 comp rx=0 tx=0).stop 2024-06-29T15:39:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 shutdown_connections 2024-06-29T15:39:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7faf900763f0 0x7faf90078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafa40cc060 0x7fafa40c9340 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafa40cb490 0x7fafa409c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 --2- 172.21.15.59:0/1340309146 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fafa40ad4e0 0x7fafa4153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.499+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 >> 172.21.15.59:0/1340309146 conn(0x7fafa40b1ea0 msgr2=0x7fafa40b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.500+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 shutdown_connections 2024-06-29T15:39:16.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.500+0000 7fafab5f0640 1 -- 172.21.15.59:0/1340309146 wait complete. 2024-06-29T15:39:16.512 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:39:15.572162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1146,"num_read_kb":3223,"num_write":362,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":392572,"kb_used_data":19596,"kb_used_omap":12,"kb_used_meta":372851,"kb_avail":754582148,"statfs":{"total":773094113280,"available":772692119552,"internally_reserved":0,"allocated":20066304,"data_stored":10789364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":381799986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.548503"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.053634+0000","last_change":"2024-06-29T15:39:12.045916+0000","last_active":"2024-06-29T15:39:12.053634+0000","last_peered":"2024-06-29T15:39:12.053634+0000","last_clean":"2024-06-29T15:39:12.053634+0000","last_became_active":"2024-06-29T15:39:12.044457+0000","last_became_peered":"2024-06-29T15:39:12.044457+0000","last_unstale":"2024-06-29T15:39:12.053634+0000","last_undegraded":"2024-06-29T15:39:12.053634+0000","last_fullsized":"2024-06-29T15:39:12.053634+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:24.602613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.059853+0000","last_change":"2024-06-29T15:39:12.057302+0000","last_active":"2024-06-29T15:39:12.059853+0000","last_peered":"2024-06-29T15:39:12.059853+0000","last_clean":"2024-06-29T15:39:12.059853+0000","last_became_active":"2024-06-29T15:39:12.056934+0000","last_became_peered":"2024-06-29T15:39:12.056934+0000","last_unstale":"2024-06-29T15:39:12.059853+0000","last_undegraded":"2024-06-29T15:39:12.059853+0000","last_fullsized":"2024-06-29T15:39:12.059853+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:12.215644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.054441+0000","last_change":"2024-06-29T15:39:12.051318+0000","last_active":"2024-06-29T15:39:12.054441+0000","last_peered":"2024-06-29T15:39:12.054441+0000","last_clean":"2024-06-29T15:39:12.054441+0000","last_became_active":"2024-06-29T15:39:12.050996+0000","last_became_peered":"2024-06-29T15:39:12.050996+0000","last_unstale":"2024-06-29T15:39:12.054441+0000","last_undegraded":"2024-06-29T15:39:12.054441+0000","last_fullsized":"2024-06-29T15:39:12.054441+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:58.914510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.054732+0000","last_change":"2024-06-29T15:39:12.052917+0000","last_active":"2024-06-29T15:39:12.054732+0000","last_peered":"2024-06-29T15:39:12.054732+0000","last_clean":"2024-06-29T15:39:12.054732+0000","last_became_active":"2024-06-29T15:39:12.052601+0000","last_became_peered":"2024-06-29T15:39:12.052601+0000","last_unstale":"2024-06-29T15:39:12.054732+0000","last_undegraded":"2024-06-29T15:39:12.054732+0000","last_fullsized":"2024-06-29T15:39:12.054732+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:41:19.215545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.064761+0000","last_change":"2024-06-29T15:39:12.060412+0000","last_active":"2024-06-29T15:39:12.064761+0000","last_peered":"2024-06-29T15:39:12.064761+0000","last_clean":"2024-06-29T15:39:12.064761+0000","last_became_active":"2024-06-29T15:39:12.060067+0000","last_became_peered":"2024-06-29T15:39:12.060067+0000","last_unstale":"2024-06-29T15:39:12.064761+0000","last_undegraded":"2024-06-29T15:39:12.064761+0000","last_fullsized":"2024-06-29T15:39:12.064761+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:42.783274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.059627+0000","last_change":"2024-06-29T15:39:12.055119+0000","last_active":"2024-06-29T15:39:12.059627+0000","last_peered":"2024-06-29T15:39:12.059627+0000","last_clean":"2024-06-29T15:39:12.059627+0000","last_became_active":"2024-06-29T15:39:12.054795+0000","last_became_peered":"2024-06-29T15:39:12.054795+0000","last_unstale":"2024-06-29T15:39:12.059627+0000","last_undegraded":"2024-06-29T15:39:12.059627+0000","last_fullsized":"2024-06-29T15:39:12.059627+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:52:35.682187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052488+0000","last_change":"2024-06-29T15:39:12.056021+0000","last_active":"2024-06-29T15:39:13.052488+0000","last_peered":"2024-06-29T15:39:13.052488+0000","last_clean":"2024-06-29T15:39:13.052488+0000","last_became_active":"2024-06-29T15:39:12.054045+0000","last_became_peered":"2024-06-29T15:39:12.054045+0000","last_unstale":"2024-06-29T15:39:13.052488+0000","last_undegraded":"2024-06-29T15:39:13.052488+0000","last_fullsized":"2024-06-29T15:39:13.052488+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:54.404587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052433+0000","last_change":"2024-06-29T15:39:12.057833+0000","last_active":"2024-06-29T15:39:13.052433+0000","last_peered":"2024-06-29T15:39:13.052433+0000","last_clean":"2024-06-29T15:39:13.052433+0000","last_became_active":"2024-06-29T15:39:12.057498+0000","last_became_peered":"2024-06-29T15:39:12.057498+0000","last_unstale":"2024-06-29T15:39:13.052433+0000","last_undegraded":"2024-06-29T15:39:13.052433+0000","last_fullsized":"2024-06-29T15:39:13.052433+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:04:26.205915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053097+0000","last_change":"2024-06-29T15:39:12.055967+0000","last_active":"2024-06-29T15:39:13.053097+0000","last_peered":"2024-06-29T15:39:13.053097+0000","last_clean":"2024-06-29T15:39:13.053097+0000","last_became_active":"2024-06-29T15:39:12.054679+0000","last_became_peered":"2024-06-29T15:39:12.054679+0000","last_unstale":"2024-06-29T15:39:13.053097+0000","last_undegraded":"2024-06-29T15:39:13.053097+0000","last_fullsized":"2024-06-29T15:39:13.053097+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:34.865489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053078+0000","last_change":"2024-06-29T15:39:12.061724+0000","last_active":"2024-06-29T15:39:13.053078+0000","last_peered":"2024-06-29T15:39:13.053078+0000","last_clean":"2024-06-29T15:39:13.053078+0000","last_became_active":"2024-06-29T15:39:12.061383+0000","last_became_peered":"2024-06-29T15:39:12.061383+0000","last_unstale":"2024-06-29T15:39:13.053078+0000","last_undegraded":"2024-06-29T15:39:13.053078+0000","last_fullsized":"2024-06-29T15:39:13.053078+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:31:22.629917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055813+0000","last_change":"2024-06-29T15:39:12.068062+0000","last_active":"2024-06-29T15:39:13.055813+0000","last_peered":"2024-06-29T15:39:13.055813+0000","last_clean":"2024-06-29T15:39:13.055813+0000","last_became_active":"2024-06-29T15:39:12.067703+0000","last_became_peered":"2024-06-29T15:39:12.067703+0000","last_unstale":"2024-06-29T15:39:13.055813+0000","last_undegraded":"2024-06-29T15:39:13.055813+0000","last_fullsized":"2024-06-29T15:39:13.055813+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:27.690017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.059616+0000","last_change":"2024-06-29T15:39:12.056563+0000","last_active":"2024-06-29T15:39:12.059616+0000","last_peered":"2024-06-29T15:39:12.059616+0000","last_clean":"2024-06-29T15:39:12.059616+0000","last_became_active":"2024-06-29T15:39:12.055841+0000","last_became_peered":"2024-06-29T15:39:12.055841+0000","last_unstale":"2024-06-29T15:39:12.059616+0000","last_undegraded":"2024-06-29T15:39:12.059616+0000","last_fullsized":"2024-06-29T15:39:12.059616+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:44.332196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055693+0000","last_change":"2024-06-29T15:39:12.057529+0000","last_active":"2024-06-29T15:39:13.055693+0000","last_peered":"2024-06-29T15:39:13.055693+0000","last_clean":"2024-06-29T15:39:13.055693+0000","last_became_active":"2024-06-29T15:39:12.056798+0000","last_became_peered":"2024-06-29T15:39:12.056798+0000","last_unstale":"2024-06-29T15:39:13.055693+0000","last_undegraded":"2024-06-29T15:39:13.055693+0000","last_fullsized":"2024-06-29T15:39:13.055693+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:58.124939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069109+0000","last_change":"2024-06-29T15:39:12.061693+0000","last_active":"2024-06-29T15:39:12.069109+0000","last_peered":"2024-06-29T15:39:12.069109+0000","last_clean":"2024-06-29T15:39:12.069109+0000","last_became_active":"2024-06-29T15:39:12.060832+0000","last_became_peered":"2024-06-29T15:39:12.060832+0000","last_unstale":"2024-06-29T15:39:12.069109+0000","last_undegraded":"2024-06-29T15:39:12.069109+0000","last_fullsized":"2024-06-29T15:39:12.069109+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:25:10.528258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055642+0000","last_change":"2024-06-29T15:39:12.063418+0000","last_active":"2024-06-29T15:39:13.055642+0000","last_peered":"2024-06-29T15:39:13.055642+0000","last_clean":"2024-06-29T15:39:13.055642+0000","last_became_active":"2024-06-29T15:39:12.063087+0000","last_became_peered":"2024-06-29T15:39:12.063087+0000","last_unstale":"2024-06-29T15:39:13.055642+0000","last_undegraded":"2024-06-29T15:39:13.055642+0000","last_fullsized":"2024-06-29T15:39:13.055642+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:16.152546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.059810+0000","last_change":"2024-06-29T15:39:12.058509+0000","last_active":"2024-06-29T15:39:12.059810+0000","last_peered":"2024-06-29T15:39:12.059810+0000","last_clean":"2024-06-29T15:39:12.059810+0000","last_became_active":"2024-06-29T15:39:12.058179+0000","last_became_peered":"2024-06-29T15:39:12.058179+0000","last_unstale":"2024-06-29T15:39:12.059810+0000","last_undegraded":"2024-06-29T15:39:12.059810+0000","last_fullsized":"2024-06-29T15:39:12.059810+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:46:43.779985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053143+0000","last_change":"2024-06-29T15:39:12.055463+0000","last_active":"2024-06-29T15:39:13.053143+0000","last_peered":"2024-06-29T15:39:13.053143+0000","last_clean":"2024-06-29T15:39:13.053143+0000","last_became_active":"2024-06-29T15:39:12.055113+0000","last_became_peered":"2024-06-29T15:39:12.055113+0000","last_unstale":"2024-06-29T15:39:13.053143+0000","last_undegraded":"2024-06-29T15:39:13.053143+0000","last_fullsized":"2024-06-29T15:39:13.053143+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:30:44.019499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055038+0000","last_change":"2024-06-29T15:39:12.048530+0000","last_active":"2024-06-29T15:39:13.055038+0000","last_peered":"2024-06-29T15:39:13.055038+0000","last_clean":"2024-06-29T15:39:13.055038+0000","last_became_active":"2024-06-29T15:39:12.047989+0000","last_became_peered":"2024-06-29T15:39:12.047989+0000","last_unstale":"2024-06-29T15:39:13.055038+0000","last_undegraded":"2024-06-29T15:39:13.055038+0000","last_fullsized":"2024-06-29T15:39:13.055038+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:04:33.628753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056196+0000","last_change":"2024-06-29T15:39:12.063513+0000","last_active":"2024-06-29T15:39:13.056196+0000","last_peered":"2024-06-29T15:39:13.056196+0000","last_clean":"2024-06-29T15:39:13.056196+0000","last_became_active":"2024-06-29T15:39:12.063160+0000","last_became_peered":"2024-06-29T15:39:12.063160+0000","last_unstale":"2024-06-29T15:39:13.056196+0000","last_undegraded":"2024-06-29T15:39:13.056196+0000","last_fullsized":"2024-06-29T15:39:13.056196+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:18:28.925687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135354+0000","last_change":"2024-06-29T15:39:12.055465+0000","last_active":"2024-06-29T15:39:13.135354+0000","last_peered":"2024-06-29T15:39:13.135354+0000","last_clean":"2024-06-29T15:39:13.135354+0000","last_became_active":"2024-06-29T15:39:12.055042+0000","last_became_peered":"2024-06-29T15:39:12.055042+0000","last_unstale":"2024-06-29T15:39:13.135354+0000","last_undegraded":"2024-06-29T15:39:13.135354+0000","last_fullsized":"2024-06-29T15:39:13.135354+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:00.216407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055887+0000","last_change":"2024-06-29T15:39:12.042695+0000","last_active":"2024-06-29T15:39:13.055887+0000","last_peered":"2024-06-29T15:39:13.055887+0000","last_clean":"2024-06-29T15:39:13.055887+0000","last_became_active":"2024-06-29T15:39:12.042324+0000","last_became_peered":"2024-06-29T15:39:12.042324+0000","last_unstale":"2024-06-29T15:39:13.055887+0000","last_undegraded":"2024-06-29T15:39:13.055887+0000","last_fullsized":"2024-06-29T15:39:13.055887+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:52:50.567029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.062627+0000","last_change":"2024-06-29T15:39:12.061442+0000","last_active":"2024-06-29T15:39:12.062627+0000","last_peered":"2024-06-29T15:39:12.062627+0000","last_clean":"2024-06-29T15:39:12.062627+0000","last_became_active":"2024-06-29T15:39:12.061017+0000","last_became_peered":"2024-06-29T15:39:12.061017+0000","last_unstale":"2024-06-29T15:39:12.062627+0000","last_undegraded":"2024-06-29T15:39:12.062627+0000","last_fullsized":"2024-06-29T15:39:12.062627+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:42:19.764925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.064863+0000","last_change":"2024-06-29T15:39:12.061119+0000","last_active":"2024-06-29T15:39:12.064863+0000","last_peered":"2024-06-29T15:39:12.064863+0000","last_clean":"2024-06-29T15:39:12.064863+0000","last_became_active":"2024-06-29T15:39:12.060778+0000","last_became_peered":"2024-06-29T15:39:12.060778+0000","last_unstale":"2024-06-29T15:39:12.064863+0000","last_undegraded":"2024-06-29T15:39:12.064863+0000","last_fullsized":"2024-06-29T15:39:12.064863+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:25.227821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.041153+0000","last_change":"2024-06-29T15:39:12.040384+0000","last_active":"2024-06-29T15:39:12.041153+0000","last_peered":"2024-06-29T15:39:12.041153+0000","last_clean":"2024-06-29T15:39:12.041153+0000","last_became_active":"2024-06-29T15:39:12.040084+0000","last_became_peered":"2024-06-29T15:39:12.040084+0000","last_unstale":"2024-06-29T15:39:12.041153+0000","last_undegraded":"2024-06-29T15:39:12.041153+0000","last_fullsized":"2024-06-29T15:39:12.041153+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:44.134466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056272+0000","last_change":"2024-06-29T15:39:12.052031+0000","last_active":"2024-06-29T15:39:13.056272+0000","last_peered":"2024-06-29T15:39:13.056272+0000","last_clean":"2024-06-29T15:39:13.056272+0000","last_became_active":"2024-06-29T15:39:12.051573+0000","last_became_peered":"2024-06-29T15:39:12.051573+0000","last_unstale":"2024-06-29T15:39:13.056272+0000","last_undegraded":"2024-06-29T15:39:13.056272+0000","last_fullsized":"2024-06-29T15:39:13.056272+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:00.971422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135538+0000","last_change":"2024-06-29T15:39:12.058553+0000","last_active":"2024-06-29T15:39:13.135538+0000","last_peered":"2024-06-29T15:39:13.135538+0000","last_clean":"2024-06-29T15:39:13.135538+0000","last_became_active":"2024-06-29T15:39:12.058234+0000","last_became_peered":"2024-06-29T15:39:12.058234+0000","last_unstale":"2024-06-29T15:39:13.135538+0000","last_undegraded":"2024-06-29T15:39:13.135538+0000","last_fullsized":"2024-06-29T15:39:13.135538+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:52.670054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.061961+0000","last_change":"2024-06-29T15:39:12.061342+0000","last_active":"2024-06-29T15:39:12.061961+0000","last_peered":"2024-06-29T15:39:12.061961+0000","last_clean":"2024-06-29T15:39:12.061961+0000","last_became_active":"2024-06-29T15:39:12.060991+0000","last_became_peered":"2024-06-29T15:39:12.060991+0000","last_unstale":"2024-06-29T15:39:12.061961+0000","last_undegraded":"2024-06-29T15:39:12.061961+0000","last_fullsized":"2024-06-29T15:39:12.061961+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:56.464247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055992+0000","last_change":"2024-06-29T15:39:12.043562+0000","last_active":"2024-06-29T15:39:13.055992+0000","last_peered":"2024-06-29T15:39:13.055992+0000","last_clean":"2024-06-29T15:39:13.055992+0000","last_became_active":"2024-06-29T15:39:12.043218+0000","last_became_peered":"2024-06-29T15:39:12.043218+0000","last_unstale":"2024-06-29T15:39:13.055992+0000","last_undegraded":"2024-06-29T15:39:13.055992+0000","last_fullsized":"2024-06-29T15:39:13.055992+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:47.237775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.057150+0000","last_change":"2024-06-29T15:39:12.052957+0000","last_active":"2024-06-29T15:39:12.057150+0000","last_peered":"2024-06-29T15:39:12.057150+0000","last_clean":"2024-06-29T15:39:12.057150+0000","last_became_active":"2024-06-29T15:39:12.052578+0000","last_became_peered":"2024-06-29T15:39:12.052578+0000","last_unstale":"2024-06-29T15:39:12.057150+0000","last_undegraded":"2024-06-29T15:39:12.057150+0000","last_fullsized":"2024-06-29T15:39:12.057150+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:44.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.054925+0000","last_change":"2024-06-29T15:39:12.053274+0000","last_active":"2024-06-29T15:39:13.054925+0000","last_peered":"2024-06-29T15:39:13.054925+0000","last_clean":"2024-06-29T15:39:13.054925+0000","last_became_active":"2024-06-29T15:39:12.051865+0000","last_became_peered":"2024-06-29T15:39:12.051865+0000","last_unstale":"2024-06-29T15:39:13.054925+0000","last_undegraded":"2024-06-29T15:39:13.054925+0000","last_fullsized":"2024-06-29T15:39:13.054925+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:32.167435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055872+0000","last_change":"2024-06-29T15:39:12.050375+0000","last_active":"2024-06-29T15:39:13.055872+0000","last_peered":"2024-06-29T15:39:13.055872+0000","last_clean":"2024-06-29T15:39:13.055872+0000","last_became_active":"2024-06-29T15:39:12.050053+0000","last_became_peered":"2024-06-29T15:39:12.050053+0000","last_unstale":"2024-06-29T15:39:13.055872+0000","last_undegraded":"2024-06-29T15:39:13.055872+0000","last_fullsized":"2024-06-29T15:39:13.055872+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:36.261145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055000+0000","last_change":"2024-06-29T15:39:12.049996+0000","last_active":"2024-06-29T15:39:13.055000+0000","last_peered":"2024-06-29T15:39:13.055000+0000","last_clean":"2024-06-29T15:39:13.055000+0000","last_became_active":"2024-06-29T15:39:12.049580+0000","last_became_peered":"2024-06-29T15:39:12.049580+0000","last_unstale":"2024-06-29T15:39:13.055000+0000","last_undegraded":"2024-06-29T15:39:13.055000+0000","last_fullsized":"2024-06-29T15:39:13.055000+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:21:58.053003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.063987+0000","last_change":"2024-06-29T15:39:12.063069+0000","last_active":"2024-06-29T15:39:12.063987+0000","last_peered":"2024-06-29T15:39:12.063987+0000","last_clean":"2024-06-29T15:39:12.063987+0000","last_became_active":"2024-06-29T15:39:12.062333+0000","last_became_peered":"2024-06-29T15:39:12.062333+0000","last_unstale":"2024-06-29T15:39:12.063987+0000","last_undegraded":"2024-06-29T15:39:12.063987+0000","last_fullsized":"2024-06-29T15:39:12.063987+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:37.033072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"188'4","reported_seq":16,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.072800+0000","last_change":"2024-06-29T15:39:11.048920+0000","last_active":"2024-06-29T15:39:12.072800+0000","last_peered":"2024-06-29T15:39:12.072800+0000","last_clean":"2024-06-29T15:39:12.072800+0000","last_became_active":"2024-06-29T15:39:11.048586+0000","last_became_peered":"2024-06-29T15:39:11.048586+0000","last_unstale":"2024-06-29T15:39:12.072800+0000","last_undegraded":"2024-06-29T15:39:12.072800+0000","last_fullsized":"2024-06-29T15:39:12.072800+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:29.144496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055507+0000","last_change":"2024-06-29T15:39:12.046257+0000","last_active":"2024-06-29T15:39:13.055507+0000","last_peered":"2024-06-29T15:39:13.055507+0000","last_clean":"2024-06-29T15:39:13.055507+0000","last_became_active":"2024-06-29T15:39:12.045942+0000","last_became_peered":"2024-06-29T15:39:12.045942+0000","last_unstale":"2024-06-29T15:39:13.055507+0000","last_undegraded":"2024-06-29T15:39:13.055507+0000","last_fullsized":"2024-06-29T15:39:13.055507+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:37.129027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"188'1","reported_seq":13,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.073898+0000","last_change":"2024-06-29T15:39:11.058196+0000","last_active":"2024-06-29T15:39:12.073898+0000","last_peered":"2024-06-29T15:39:12.073898+0000","last_clean":"2024-06-29T15:39:12.073898+0000","last_became_active":"2024-06-29T15:39:11.057857+0000","last_became_peered":"2024-06-29T15:39:11.057857+0000","last_unstale":"2024-06-29T15:39:12.073898+0000","last_undegraded":"2024-06-29T15:39:12.073898+0000","last_fullsized":"2024-06-29T15:39:12.073898+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:16.760866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069128+0000","last_change":"2024-06-29T15:39:12.065859+0000","last_active":"2024-06-29T15:39:12.069128+0000","last_peered":"2024-06-29T15:39:12.069128+0000","last_clean":"2024-06-29T15:39:12.069128+0000","last_became_active":"2024-06-29T15:39:12.061304+0000","last_became_peered":"2024-06-29T15:39:12.061304+0000","last_unstale":"2024-06-29T15:39:12.069128+0000","last_undegraded":"2024-06-29T15:39:12.069128+0000","last_fullsized":"2024-06-29T15:39:12.069128+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:09.591584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"188'2","reported_seq":16,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056404+0000","last_change":"2024-06-29T15:39:11.053442+0000","last_active":"2024-06-29T15:39:13.056404+0000","last_peered":"2024-06-29T15:39:13.056404+0000","last_clean":"2024-06-29T15:39:13.056404+0000","last_became_active":"2024-06-29T15:39:11.053111+0000","last_became_peered":"2024-06-29T15:39:11.053111+0000","last_unstale":"2024-06-29T15:39:13.056404+0000","last_undegraded":"2024-06-29T15:39:13.056404+0000","last_fullsized":"2024-06-29T15:39:13.056404+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:47:42.511404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.067946+0000","last_change":"2024-06-29T15:39:12.067147+0000","last_active":"2024-06-29T15:39:12.067946+0000","last_peered":"2024-06-29T15:39:12.067946+0000","last_clean":"2024-06-29T15:39:12.067946+0000","last_became_active":"2024-06-29T15:39:12.066801+0000","last_became_peered":"2024-06-29T15:39:12.066801+0000","last_unstale":"2024-06-29T15:39:12.067946+0000","last_undegraded":"2024-06-29T15:39:12.067946+0000","last_fullsized":"2024-06-29T15:39:12.067946+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:04.667481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"188'2","reported_seq":16,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135319+0000","last_change":"2024-06-29T15:39:11.045162+0000","last_active":"2024-06-29T15:39:13.135319+0000","last_peered":"2024-06-29T15:39:13.135319+0000","last_clean":"2024-06-29T15:39:13.135319+0000","last_became_active":"2024-06-29T15:39:11.044791+0000","last_became_peered":"2024-06-29T15:39:11.044791+0000","last_unstale":"2024-06-29T15:39:13.135319+0000","last_undegraded":"2024-06-29T15:39:13.135319+0000","last_fullsized":"2024-06-29T15:39:13.135319+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:38.779317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052890+0000","last_change":"2024-06-29T15:39:12.062610+0000","last_active":"2024-06-29T15:39:13.052890+0000","last_peered":"2024-06-29T15:39:13.052890+0000","last_clean":"2024-06-29T15:39:13.052890+0000","last_became_active":"2024-06-29T15:39:12.062269+0000","last_became_peered":"2024-06-29T15:39:12.062269+0000","last_unstale":"2024-06-29T15:39:13.052890+0000","last_undegraded":"2024-06-29T15:39:13.052890+0000","last_fullsized":"2024-06-29T15:39:13.052890+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:13.847144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"188'1","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052777+0000","last_change":"2024-06-29T15:39:11.061738+0000","last_active":"2024-06-29T15:39:13.052777+0000","last_peered":"2024-06-29T15:39:13.052777+0000","last_clean":"2024-06-29T15:39:13.052777+0000","last_became_active":"2024-06-29T15:39:11.061385+0000","last_became_peered":"2024-06-29T15:39:11.061385+0000","last_unstale":"2024-06-29T15:39:13.052777+0000","last_undegraded":"2024-06-29T15:39:13.052777+0000","last_fullsized":"2024-06-29T15:39:13.052777+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:49.777466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135415+0000","last_change":"2024-06-29T15:39:12.057660+0000","last_active":"2024-06-29T15:39:13.135415+0000","last_peered":"2024-06-29T15:39:13.135415+0000","last_clean":"2024-06-29T15:39:13.135415+0000","last_became_active":"2024-06-29T15:39:12.057324+0000","last_became_peered":"2024-06-29T15:39:12.057324+0000","last_unstale":"2024-06-29T15:39:13.135415+0000","last_undegraded":"2024-06-29T15:39:13.135415+0000","last_fullsized":"2024-06-29T15:39:13.135415+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:21:57.650253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"188'1","reported_seq":13,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.076497+0000","last_change":"2024-06-29T15:39:11.058866+0000","last_active":"2024-06-29T15:39:12.076497+0000","last_peered":"2024-06-29T15:39:12.076497+0000","last_clean":"2024-06-29T15:39:12.076497+0000","last_became_active":"2024-06-29T15:39:11.058505+0000","last_became_peered":"2024-06-29T15:39:11.058505+0000","last_unstale":"2024-06-29T15:39:12.076497+0000","last_undegraded":"2024-06-29T15:39:12.076497+0000","last_fullsized":"2024-06-29T15:39:12.076497+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:54.840601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135648+0000","last_change":"2024-06-29T15:39:12.046442+0000","last_active":"2024-06-29T15:39:13.135648+0000","last_peered":"2024-06-29T15:39:13.135648+0000","last_clean":"2024-06-29T15:39:13.135648+0000","last_became_active":"2024-06-29T15:39:12.046108+0000","last_became_peered":"2024-06-29T15:39:12.046108+0000","last_unstale":"2024-06-29T15:39:13.135648+0000","last_undegraded":"2024-06-29T15:39:13.135648+0000","last_fullsized":"2024-06-29T15:39:13.135648+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:43:49.484877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135186+0000","last_change":"2024-06-29T15:39:11.050330+0000","last_active":"2024-06-29T15:39:13.135186+0000","last_peered":"2024-06-29T15:39:13.135186+0000","last_clean":"2024-06-29T15:39:13.135186+0000","last_became_active":"2024-06-29T15:39:11.049987+0000","last_became_peered":"2024-06-29T15:39:11.049987+0000","last_unstale":"2024-06-29T15:39:13.135186+0000","last_undegraded":"2024-06-29T15:39:13.135186+0000","last_fullsized":"2024-06-29T15:39:13.135186+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:28.627070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.058420+0000","last_change":"2024-06-29T15:39:12.056637+0000","last_active":"2024-06-29T15:39:12.058420+0000","last_peered":"2024-06-29T15:39:12.058420+0000","last_clean":"2024-06-29T15:39:12.058420+0000","last_became_active":"2024-06-29T15:39:12.056027+0000","last_became_peered":"2024-06-29T15:39:12.056027+0000","last_unstale":"2024-06-29T15:39:12.058420+0000","last_undegraded":"2024-06-29T15:39:12.058420+0000","last_fullsized":"2024-06-29T15:39:12.058420+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:36.971914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052572+0000","last_change":"2024-06-29T15:39:11.053350+0000","last_active":"2024-06-29T15:39:13.052572+0000","last_peered":"2024-06-29T15:39:13.052572+0000","last_clean":"2024-06-29T15:39:13.052572+0000","last_became_active":"2024-06-29T15:39:11.053012+0000","last_became_peered":"2024-06-29T15:39:11.053012+0000","last_unstale":"2024-06-29T15:39:13.052572+0000","last_undegraded":"2024-06-29T15:39:13.052572+0000","last_fullsized":"2024-06-29T15:39:13.052572+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:52.030756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.063964+0000","last_change":"2024-06-29T15:39:12.062973+0000","last_active":"2024-06-29T15:39:12.063964+0000","last_peered":"2024-06-29T15:39:12.063964+0000","last_clean":"2024-06-29T15:39:12.063964+0000","last_became_active":"2024-06-29T15:39:12.062626+0000","last_became_peered":"2024-06-29T15:39:12.062626+0000","last_unstale":"2024-06-29T15:39:12.063964+0000","last_undegraded":"2024-06-29T15:39:12.063964+0000","last_fullsized":"2024-06-29T15:39:12.063964+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:57.350541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056222+0000","last_change":"2024-06-29T15:39:11.056609+0000","last_active":"2024-06-29T15:39:13.056222+0000","last_peered":"2024-06-29T15:39:13.056222+0000","last_clean":"2024-06-29T15:39:13.056222+0000","last_became_active":"2024-06-29T15:39:11.055487+0000","last_became_peered":"2024-06-29T15:39:11.055487+0000","last_unstale":"2024-06-29T15:39:13.056222+0000","last_undegraded":"2024-06-29T15:39:13.056222+0000","last_fullsized":"2024-06-29T15:39:13.056222+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:29.801478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055574+0000","last_change":"2024-06-29T15:39:12.048202+0000","last_active":"2024-06-29T15:39:13.055574+0000","last_peered":"2024-06-29T15:39:13.055574+0000","last_clean":"2024-06-29T15:39:13.055574+0000","last_became_active":"2024-06-29T15:39:12.047848+0000","last_became_peered":"2024-06-29T15:39:12.047848+0000","last_unstale":"2024-06-29T15:39:13.055574+0000","last_undegraded":"2024-06-29T15:39:13.055574+0000","last_fullsized":"2024-06-29T15:39:13.055574+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:57.346505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"188'3","reported_seq":15,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069970+0000","last_change":"2024-06-29T15:39:11.045344+0000","last_active":"2024-06-29T15:39:12.069970+0000","last_peered":"2024-06-29T15:39:12.069970+0000","last_clean":"2024-06-29T15:39:12.069970+0000","last_became_active":"2024-06-29T15:39:11.045007+0000","last_became_peered":"2024-06-29T15:39:11.045007+0000","last_unstale":"2024-06-29T15:39:12.069970+0000","last_undegraded":"2024-06-29T15:39:12.069970+0000","last_fullsized":"2024-06-29T15:39:12.069970+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:04.984846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135498+0000","last_change":"2024-06-29T15:39:12.055384+0000","last_active":"2024-06-29T15:39:13.135498+0000","last_peered":"2024-06-29T15:39:13.135498+0000","last_clean":"2024-06-29T15:39:13.135498+0000","last_became_active":"2024-06-29T15:39:12.054982+0000","last_became_peered":"2024-06-29T15:39:12.054982+0000","last_unstale":"2024-06-29T15:39:13.135498+0000","last_undegraded":"2024-06-29T15:39:13.135498+0000","last_fullsized":"2024-06-29T15:39:13.135498+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:09:05.612525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"188'6","reported_seq":20,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056103+0000","last_change":"2024-06-29T15:39:11.056112+0000","last_active":"2024-06-29T15:39:13.056103+0000","last_peered":"2024-06-29T15:39:13.056103+0000","last_clean":"2024-06-29T15:39:13.056103+0000","last_became_active":"2024-06-29T15:39:11.055775+0000","last_became_peered":"2024-06-29T15:39:11.055775+0000","last_unstale":"2024-06-29T15:39:13.056103+0000","last_undegraded":"2024-06-29T15:39:13.056103+0000","last_fullsized":"2024-06-29T15:39:13.056103+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:32.908998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.054949+0000","last_change":"2024-06-29T15:39:12.050363+0000","last_active":"2024-06-29T15:39:13.054949+0000","last_peered":"2024-06-29T15:39:13.054949+0000","last_clean":"2024-06-29T15:39:13.054949+0000","last_became_active":"2024-06-29T15:39:12.049970+0000","last_became_peered":"2024-06-29T15:39:12.049970+0000","last_unstale":"2024-06-29T15:39:13.054949+0000","last_undegraded":"2024-06-29T15:39:13.054949+0000","last_fullsized":"2024-06-29T15:39:13.054949+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.679970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"188'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.074655+0000","last_change":"2024-06-29T15:39:11.049108+0000","last_active":"2024-06-29T15:39:12.074655+0000","last_peered":"2024-06-29T15:39:12.074655+0000","last_clean":"2024-06-29T15:39:12.074655+0000","last_became_active":"2024-06-29T15:39:11.048312+0000","last_became_peered":"2024-06-29T15:39:11.048312+0000","last_unstale":"2024-06-29T15:39:12.074655+0000","last_undegraded":"2024-06-29T15:39:12.074655+0000","last_fullsized":"2024-06-29T15:39:12.074655+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:11:52.252830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053030+0000","last_change":"2024-06-29T15:39:12.052634+0000","last_active":"2024-06-29T15:39:13.053030+0000","last_peered":"2024-06-29T15:39:13.053030+0000","last_clean":"2024-06-29T15:39:13.053030+0000","last_became_active":"2024-06-29T15:39:12.052249+0000","last_became_peered":"2024-06-29T15:39:12.052249+0000","last_unstale":"2024-06-29T15:39:13.053030+0000","last_undegraded":"2024-06-29T15:39:13.053030+0000","last_fullsized":"2024-06-29T15:39:13.053030+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:59:00.019618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055644+0000","last_change":"2024-06-29T15:39:11.050832+0000","last_active":"2024-06-29T15:39:13.055644+0000","last_peered":"2024-06-29T15:39:13.055644+0000","last_clean":"2024-06-29T15:39:13.055644+0000","last_became_active":"2024-06-29T15:39:11.050273+0000","last_became_peered":"2024-06-29T15:39:11.050273+0000","last_unstale":"2024-06-29T15:39:13.055644+0000","last_undegraded":"2024-06-29T15:39:13.055644+0000","last_fullsized":"2024-06-29T15:39:13.055644+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:32.140104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056212+0000","last_change":"2024-06-29T15:39:12.055589+0000","last_active":"2024-06-29T15:39:13.056212+0000","last_peered":"2024-06-29T15:39:13.056212+0000","last_clean":"2024-06-29T15:39:13.056212+0000","last_became_active":"2024-06-29T15:39:12.055280+0000","last_became_peered":"2024-06-29T15:39:12.055280+0000","last_unstale":"2024-06-29T15:39:13.056212+0000","last_undegraded":"2024-06-29T15:39:13.056212+0000","last_fullsized":"2024-06-29T15:39:13.056212+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:29:30.357235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055975+0000","last_change":"2024-06-29T15:39:11.048698+0000","last_active":"2024-06-29T15:39:13.055975+0000","last_peered":"2024-06-29T15:39:13.055975+0000","last_clean":"2024-06-29T15:39:13.055975+0000","last_became_active":"2024-06-29T15:39:11.048333+0000","last_became_peered":"2024-06-29T15:39:11.048333+0000","last_unstale":"2024-06-29T15:39:13.055975+0000","last_undegraded":"2024-06-29T15:39:13.055975+0000","last_fullsized":"2024-06-29T15:39:13.055975+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:02:38.352100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135595+0000","last_change":"2024-06-29T15:39:12.059109+0000","last_active":"2024-06-29T15:39:13.135595+0000","last_peered":"2024-06-29T15:39:13.135595+0000","last_clean":"2024-06-29T15:39:13.135595+0000","last_became_active":"2024-06-29T15:39:12.058516+0000","last_became_peered":"2024-06-29T15:39:12.058516+0000","last_unstale":"2024-06-29T15:39:13.135595+0000","last_undegraded":"2024-06-29T15:39:13.135595+0000","last_fullsized":"2024-06-29T15:39:13.135595+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:45.712612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"188'8","reported_seq":22,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052570+0000","last_change":"2024-06-29T15:39:11.060192+0000","last_active":"2024-06-29T15:39:13.052570+0000","last_peered":"2024-06-29T15:39:13.052570+0000","last_clean":"2024-06-29T15:39:13.052570+0000","last_became_active":"2024-06-29T15:39:11.059847+0000","last_became_peered":"2024-06-29T15:39:11.059847+0000","last_unstale":"2024-06-29T15:39:13.052570+0000","last_undegraded":"2024-06-29T15:39:13.052570+0000","last_fullsized":"2024-06-29T15:39:13.052570+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:47.347264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052427+0000","last_change":"2024-06-29T15:39:12.055760+0000","last_active":"2024-06-29T15:39:13.052427+0000","last_peered":"2024-06-29T15:39:13.052427+0000","last_clean":"2024-06-29T15:39:13.052427+0000","last_became_active":"2024-06-29T15:39:12.055435+0000","last_became_peered":"2024-06-29T15:39:12.055435+0000","last_unstale":"2024-06-29T15:39:13.052427+0000","last_undegraded":"2024-06-29T15:39:13.052427+0000","last_fullsized":"2024-06-29T15:39:13.052427+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:19.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"188'1","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056347+0000","last_change":"2024-06-29T15:39:11.049084+0000","last_active":"2024-06-29T15:39:13.056347+0000","last_peered":"2024-06-29T15:39:13.056347+0000","last_clean":"2024-06-29T15:39:13.056347+0000","last_became_active":"2024-06-29T15:39:11.048781+0000","last_became_peered":"2024-06-29T15:39:11.048781+0000","last_unstale":"2024-06-29T15:39:13.056347+0000","last_undegraded":"2024-06-29T15:39:13.056347+0000","last_fullsized":"2024-06-29T15:39:13.056347+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:26.683092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055320+0000","last_change":"2024-06-29T15:39:12.052024+0000","last_active":"2024-06-29T15:39:13.055320+0000","last_peered":"2024-06-29T15:39:13.055320+0000","last_clean":"2024-06-29T15:39:13.055320+0000","last_became_active":"2024-06-29T15:39:12.051439+0000","last_became_peered":"2024-06-29T15:39:12.051439+0000","last_unstale":"2024-06-29T15:39:13.055320+0000","last_undegraded":"2024-06-29T15:39:13.055320+0000","last_fullsized":"2024-06-29T15:39:13.055320+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:28.800208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"188'2","reported_seq":16,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.053244+0000","last_change":"2024-06-29T15:39:11.055274+0000","last_active":"2024-06-29T15:39:13.053244+0000","last_peered":"2024-06-29T15:39:13.053244+0000","last_clean":"2024-06-29T15:39:13.053244+0000","last_became_active":"2024-06-29T15:39:11.054944+0000","last_became_peered":"2024-06-29T15:39:11.054944+0000","last_unstale":"2024-06-29T15:39:13.053244+0000","last_undegraded":"2024-06-29T15:39:13.053244+0000","last_fullsized":"2024-06-29T15:39:13.053244+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:46.538589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056051+0000","last_change":"2024-06-29T15:39:12.057058+0000","last_active":"2024-06-29T15:39:13.056051+0000","last_peered":"2024-06-29T15:39:13.056051+0000","last_clean":"2024-06-29T15:39:13.056051+0000","last_became_active":"2024-06-29T15:39:12.056473+0000","last_became_peered":"2024-06-29T15:39:12.056473+0000","last_unstale":"2024-06-29T15:39:13.056051+0000","last_undegraded":"2024-06-29T15:39:13.056051+0000","last_fullsized":"2024-06-29T15:39:13.056051+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:13:27.195144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"188'1","reported_seq":13,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.074257+0000","last_change":"2024-06-29T15:39:11.051497+0000","last_active":"2024-06-29T15:39:12.074257+0000","last_peered":"2024-06-29T15:39:12.074257+0000","last_clean":"2024-06-29T15:39:12.074257+0000","last_became_active":"2024-06-29T15:39:11.048355+0000","last_became_peered":"2024-06-29T15:39:11.048355+0000","last_unstale":"2024-06-29T15:39:12.074257+0000","last_undegraded":"2024-06-29T15:39:12.074257+0000","last_fullsized":"2024-06-29T15:39:12.074257+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:57.315377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056064+0000","last_change":"2024-06-29T15:39:12.062507+0000","last_active":"2024-06-29T15:39:13.056064+0000","last_peered":"2024-06-29T15:39:13.056064+0000","last_clean":"2024-06-29T15:39:13.056064+0000","last_became_active":"2024-06-29T15:39:12.062176+0000","last_became_peered":"2024-06-29T15:39:12.062176+0000","last_unstale":"2024-06-29T15:39:13.056064+0000","last_undegraded":"2024-06-29T15:39:13.056064+0000","last_fullsized":"2024-06-29T15:39:13.056064+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:58:06.593426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052990+0000","last_change":"2024-06-29T15:39:11.061085+0000","last_active":"2024-06-29T15:39:13.052990+0000","last_peered":"2024-06-29T15:39:13.052990+0000","last_clean":"2024-06-29T15:39:13.052990+0000","last_became_active":"2024-06-29T15:39:11.060749+0000","last_became_peered":"2024-06-29T15:39:11.060749+0000","last_unstale":"2024-06-29T15:39:13.052990+0000","last_undegraded":"2024-06-29T15:39:13.052990+0000","last_fullsized":"2024-06-29T15:39:13.052990+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:02:03.805873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056027+0000","last_change":"2024-06-29T15:39:12.045265+0000","last_active":"2024-06-29T15:39:13.056027+0000","last_peered":"2024-06-29T15:39:13.056027+0000","last_clean":"2024-06-29T15:39:13.056027+0000","last_became_active":"2024-06-29T15:39:12.044966+0000","last_became_peered":"2024-06-29T15:39:12.044966+0000","last_unstale":"2024-06-29T15:39:13.056027+0000","last_undegraded":"2024-06-29T15:39:13.056027+0000","last_fullsized":"2024-06-29T15:39:13.056027+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:09.072208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"188'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.074547+0000","last_change":"2024-06-29T15:39:11.048845+0000","last_active":"2024-06-29T15:39:12.074547+0000","last_peered":"2024-06-29T15:39:12.074547+0000","last_clean":"2024-06-29T15:39:12.074547+0000","last_became_active":"2024-06-29T15:39:11.047900+0000","last_became_peered":"2024-06-29T15:39:11.047900+0000","last_unstale":"2024-06-29T15:39:12.074547+0000","last_undegraded":"2024-06-29T15:39:12.074547+0000","last_fullsized":"2024-06-29T15:39:12.074547+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:04.559196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069066+0000","last_change":"2024-06-29T15:39:12.065944+0000","last_active":"2024-06-29T15:39:12.069066+0000","last_peered":"2024-06-29T15:39:12.069066+0000","last_clean":"2024-06-29T15:39:12.069066+0000","last_became_active":"2024-06-29T15:39:12.065535+0000","last_became_peered":"2024-06-29T15:39:12.065535+0000","last_unstale":"2024-06-29T15:39:12.069066+0000","last_undegraded":"2024-06-29T15:39:12.069066+0000","last_fullsized":"2024-06-29T15:39:12.069066+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:43:08.398702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055685+0000","last_change":"2024-06-29T15:39:11.049558+0000","last_active":"2024-06-29T15:39:13.055685+0000","last_peered":"2024-06-29T15:39:13.055685+0000","last_clean":"2024-06-29T15:39:13.055685+0000","last_became_active":"2024-06-29T15:39:11.049248+0000","last_became_peered":"2024-06-29T15:39:11.049248+0000","last_unstale":"2024-06-29T15:39:13.055685+0000","last_undegraded":"2024-06-29T15:39:13.055685+0000","last_fullsized":"2024-06-29T15:39:13.055685+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:53.959844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.069074+0000","last_change":"2024-06-29T15:39:12.064654+0000","last_active":"2024-06-29T15:39:12.069074+0000","last_peered":"2024-06-29T15:39:12.069074+0000","last_clean":"2024-06-29T15:39:12.069074+0000","last_became_active":"2024-06-29T15:39:12.064307+0000","last_became_peered":"2024-06-29T15:39:12.064307+0000","last_unstale":"2024-06-29T15:39:12.069074+0000","last_undegraded":"2024-06-29T15:39:12.069074+0000","last_fullsized":"2024-06-29T15:39:12.069074+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:02:41.355433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135345+0000","last_change":"2024-06-29T15:39:11.044706+0000","last_active":"2024-06-29T15:39:13.135345+0000","last_peered":"2024-06-29T15:39:13.135345+0000","last_clean":"2024-06-29T15:39:13.135345+0000","last_became_active":"2024-06-29T15:39:11.044352+0000","last_became_peered":"2024-06-29T15:39:11.044352+0000","last_unstale":"2024-06-29T15:39:13.135345+0000","last_undegraded":"2024-06-29T15:39:13.135345+0000","last_fullsized":"2024-06-29T15:39:13.135345+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:22.385508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.054658+0000","last_change":"2024-06-29T15:39:12.052175+0000","last_active":"2024-06-29T15:39:12.054658+0000","last_peered":"2024-06-29T15:39:12.054658+0000","last_clean":"2024-06-29T15:39:12.054658+0000","last_became_active":"2024-06-29T15:39:12.051841+0000","last_became_peered":"2024-06-29T15:39:12.051841+0000","last_unstale":"2024-06-29T15:39:12.054658+0000","last_undegraded":"2024-06-29T15:39:12.054658+0000","last_fullsized":"2024-06-29T15:39:12.054658+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:31.505068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.044363+0000","last_change":"2024-06-29T15:39:11.047344+0000","last_active":"2024-06-29T15:39:12.044363+0000","last_peered":"2024-06-29T15:39:12.044363+0000","last_clean":"2024-06-29T15:39:12.044363+0000","last_became_active":"2024-06-29T15:39:11.046984+0000","last_became_peered":"2024-06-29T15:39:11.046984+0000","last_unstale":"2024-06-29T15:39:12.044363+0000","last_undegraded":"2024-06-29T15:39:12.044363+0000","last_fullsized":"2024-06-29T15:39:12.044363+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:37.637383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055626+0000","last_change":"2024-06-29T15:39:12.050565+0000","last_active":"2024-06-29T15:39:13.055626+0000","last_peered":"2024-06-29T15:39:13.055626+0000","last_clean":"2024-06-29T15:39:13.055626+0000","last_became_active":"2024-06-29T15:39:12.050265+0000","last_became_peered":"2024-06-29T15:39:12.050265+0000","last_unstale":"2024-06-29T15:39:13.055626+0000","last_undegraded":"2024-06-29T15:39:13.055626+0000","last_fullsized":"2024-06-29T15:39:13.055626+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:04.500983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"188'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.074151+0000","last_change":"2024-06-29T15:39:11.049704+0000","last_active":"2024-06-29T15:39:12.074151+0000","last_peered":"2024-06-29T15:39:12.074151+0000","last_clean":"2024-06-29T15:39:12.074151+0000","last_became_active":"2024-06-29T15:39:11.049397+0000","last_became_peered":"2024-06-29T15:39:11.049397+0000","last_unstale":"2024-06-29T15:39:12.074151+0000","last_undegraded":"2024-06-29T15:39:12.074151+0000","last_fullsized":"2024-06-29T15:39:12.074151+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:12.354833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052551+0000","last_change":"2024-06-29T15:39:12.056326+0000","last_active":"2024-06-29T15:39:13.052551+0000","last_peered":"2024-06-29T15:39:13.052551+0000","last_clean":"2024-06-29T15:39:13.052551+0000","last_became_active":"2024-06-29T15:39:12.053118+0000","last_became_peered":"2024-06-29T15:39:12.053118+0000","last_unstale":"2024-06-29T15:39:13.052551+0000","last_undegraded":"2024-06-29T15:39:13.052551+0000","last_fullsized":"2024-06-29T15:39:13.052551+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:25.725466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"189'1","reported_seq":15,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.057349+0000","last_change":"2024-06-29T15:39:11.056070+0000","last_active":"2024-06-29T15:39:13.057349+0000","last_peered":"2024-06-29T15:39:13.057349+0000","last_clean":"2024-06-29T15:39:13.057349+0000","last_became_active":"2024-06-29T15:39:11.055515+0000","last_became_peered":"2024-06-29T15:39:11.055515+0000","last_unstale":"2024-06-29T15:39:13.057349+0000","last_undegraded":"2024-06-29T15:39:13.057349+0000","last_fullsized":"2024-06-29T15:39:13.057349+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:03.494059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135309+0000","last_change":"2024-06-29T15:39:12.046583+0000","last_active":"2024-06-29T15:39:13.135309+0000","last_peered":"2024-06-29T15:39:13.135309+0000","last_clean":"2024-06-29T15:39:13.135309+0000","last_became_active":"2024-06-29T15:39:12.045167+0000","last_became_peered":"2024-06-29T15:39:12.045167+0000","last_unstale":"2024-06-29T15:39:13.135309+0000","last_undegraded":"2024-06-29T15:39:13.135309+0000","last_fullsized":"2024-06-29T15:39:13.135309+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:28.541358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056003+0000","last_change":"2024-06-29T15:39:11.054037+0000","last_active":"2024-06-29T15:39:13.056003+0000","last_peered":"2024-06-29T15:39:13.056003+0000","last_clean":"2024-06-29T15:39:13.056003+0000","last_became_active":"2024-06-29T15:39:11.053729+0000","last_became_peered":"2024-06-29T15:39:11.053729+0000","last_unstale":"2024-06-29T15:39:13.056003+0000","last_undegraded":"2024-06-29T15:39:13.056003+0000","last_fullsized":"2024-06-29T15:39:13.056003+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:45.609828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.052654+0000","last_change":"2024-06-29T15:39:12.056983+0000","last_active":"2024-06-29T15:39:13.052654+0000","last_peered":"2024-06-29T15:39:13.052654+0000","last_clean":"2024-06-29T15:39:13.052654+0000","last_became_active":"2024-06-29T15:39:12.056651+0000","last_became_peered":"2024-06-29T15:39:12.056651+0000","last_unstale":"2024-06-29T15:39:13.052654+0000","last_undegraded":"2024-06-29T15:39:13.052654+0000","last_fullsized":"2024-06-29T15:39:13.052654+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:11.902800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135403+0000","last_change":"2024-06-29T15:39:11.056551+0000","last_active":"2024-06-29T15:39:13.135403+0000","last_peered":"2024-06-29T15:39:13.135403+0000","last_clean":"2024-06-29T15:39:13.135403+0000","last_became_active":"2024-06-29T15:39:11.056230+0000","last_became_peered":"2024-06-29T15:39:11.056230+0000","last_unstale":"2024-06-29T15:39:13.135403+0000","last_undegraded":"2024-06-29T15:39:13.135403+0000","last_fullsized":"2024-06-29T15:39:13.135403+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:45.097306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.058521+0000","last_change":"2024-06-29T15:39:12.057359+0000","last_active":"2024-06-29T15:39:12.058521+0000","last_peered":"2024-06-29T15:39:12.058521+0000","last_clean":"2024-06-29T15:39:12.058521+0000","last_became_active":"2024-06-29T15:39:12.057014+0000","last_became_peered":"2024-06-29T15:39:12.057014+0000","last_unstale":"2024-06-29T15:39:12.058521+0000","last_undegraded":"2024-06-29T15:39:12.058521+0000","last_fullsized":"2024-06-29T15:39:12.058521+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:25:07.655904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.036099+0000","last_change":"2024-06-29T15:39:11.050581+0000","last_active":"2024-06-29T15:39:12.036099+0000","last_peered":"2024-06-29T15:39:12.036099+0000","last_clean":"2024-06-29T15:39:12.036099+0000","last_became_active":"2024-06-29T15:39:11.050183+0000","last_became_peered":"2024-06-29T15:39:11.050183+0000","last_unstale":"2024-06-29T15:39:12.036099+0000","last_undegraded":"2024-06-29T15:39:12.036099+0000","last_fullsized":"2024-06-29T15:39:12.036099+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:02.389518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.064190+0000","last_change":"2024-06-29T15:39:12.063539+0000","last_active":"2024-06-29T15:39:12.064190+0000","last_peered":"2024-06-29T15:39:12.064190+0000","last_clean":"2024-06-29T15:39:12.064190+0000","last_became_active":"2024-06-29T15:39:12.062342+0000","last_became_peered":"2024-06-29T15:39:12.062342+0000","last_unstale":"2024-06-29T15:39:12.064190+0000","last_undegraded":"2024-06-29T15:39:12.064190+0000","last_fullsized":"2024-06-29T15:39:12.064190+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:44.336003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055698+0000","last_change":"2024-06-29T15:39:11.047466+0000","last_active":"2024-06-29T15:39:13.055698+0000","last_peered":"2024-06-29T15:39:13.055698+0000","last_clean":"2024-06-29T15:39:13.055698+0000","last_became_active":"2024-06-29T15:39:11.047148+0000","last_became_peered":"2024-06-29T15:39:11.047148+0000","last_unstale":"2024-06-29T15:39:13.055698+0000","last_undegraded":"2024-06-29T15:39:13.055698+0000","last_fullsized":"2024-06-29T15:39:13.055698+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:15.145621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056046+0000","last_change":"2024-06-29T15:39:12.044506+0000","last_active":"2024-06-29T15:39:13.056046+0000","last_peered":"2024-06-29T15:39:13.056046+0000","last_clean":"2024-06-29T15:39:13.056046+0000","last_became_active":"2024-06-29T15:39:12.044158+0000","last_became_peered":"2024-06-29T15:39:12.044158+0000","last_unstale":"2024-06-29T15:39:13.056046+0000","last_undegraded":"2024-06-29T15:39:13.056046+0000","last_fullsized":"2024-06-29T15:39:13.056046+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:15.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.056121+0000","last_change":"2024-06-29T15:39:11.051576+0000","last_active":"2024-06-29T15:39:13.056121+0000","last_peered":"2024-06-29T15:39:13.056121+0000","last_clean":"2024-06-29T15:39:13.056121+0000","last_became_active":"2024-06-29T15:39:11.051337+0000","last_became_peered":"2024-06-29T15:39:11.051337+0000","last_unstale":"2024-06-29T15:39:13.056121+0000","last_undegraded":"2024-06-29T15:39:13.056121+0000","last_fullsized":"2024-06-29T15:39:13.056121+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:20.183519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":12,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.135551+0000","last_change":"2024-06-29T15:39:12.058842+0000","last_active":"2024-06-29T15:39:13.135551+0000","last_peered":"2024-06-29T15:39:13.135551+0000","last_clean":"2024-06-29T15:39:13.135551+0000","last_became_active":"2024-06-29T15:39:12.058100+0000","last_became_peered":"2024-06-29T15:39:12.058100+0000","last_unstale":"2024-06-29T15:39:13.135551+0000","last_undegraded":"2024-06-29T15:39:13.135551+0000","last_fullsized":"2024-06-29T15:39:13.135551+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:42:53.020860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.045899+0000","last_change":"2024-06-29T15:39:11.047342+0000","last_active":"2024-06-29T15:39:12.045899+0000","last_peered":"2024-06-29T15:39:12.045899+0000","last_clean":"2024-06-29T15:39:12.045899+0000","last_became_active":"2024-06-29T15:39:11.047043+0000","last_became_peered":"2024-06-29T15:39:11.047043+0000","last_unstale":"2024-06-29T15:39:12.045899+0000","last_undegraded":"2024-06-29T15:39:12.045899+0000","last_fullsized":"2024-06-29T15:39:12.045899+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:35.832471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.051922+0000","last_change":"2024-06-29T15:39:12.050802+0000","last_active":"2024-06-29T15:39:12.051922+0000","last_peered":"2024-06-29T15:39:12.051922+0000","last_clean":"2024-06-29T15:39:12.051922+0000","last_became_active":"2024-06-29T15:39:12.050335+0000","last_became_peered":"2024-06-29T15:39:12.050335+0000","last_unstale":"2024-06-29T15:39:12.051922+0000","last_undegraded":"2024-06-29T15:39:12.051922+0000","last_fullsized":"2024-06-29T15:39:12.051922+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":187,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.023132+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.023132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:41:01.929978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.055965+0000","last_change":"2024-06-29T15:39:11.056912+0000","last_active":"2024-06-29T15:39:13.055965+0000","last_peered":"2024-06-29T15:39:13.055965+0000","last_clean":"2024-06-29T15:39:13.055965+0000","last_became_active":"2024-06-29T15:39:11.056591+0000","last_became_peered":"2024-06-29T15:39:11.056591+0000","last_unstale":"2024-06-29T15:39:13.055965+0000","last_undegraded":"2024-06-29T15:39:13.055965+0000","last_fullsized":"2024-06-29T15:39:13.055965+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.016207+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.016207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:01:18.669685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"178'198","reported_seq":905,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-06-29T15:39:12.044474+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:39:12.044474+0000","last_peered":"2024-06-29T15:39:12.044474+0000","last_clean":"2024-06-29T15:39:12.044474+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:39:12.044474+0000","last_undegraded":"2024-06-29T15:39:12.044474+0000","last_fullsized":"2024-06-29T15:39:12.044474+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1146,"num_read_kb":3223,"num_write":317,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1146,"num_read_kb":3223,"num_write":317,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198,"ondisk_log_size":198,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543071,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47940,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323900,"statfs":{"total":96636764160,"available":96587673600,"internally_reserved":0,"allocated":2940928,"data_stored":1788324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543072,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41792,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330048,"statfs":{"total":96636764160,"available":96593969152,"internally_reserved":0,"allocated":2347008,"data_stored":1197956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543071,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47380,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324460,"statfs":{"total":96636764160,"available":96588247040,"internally_reserved":0,"allocated":2367488,"data_stored":1204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543070,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29328,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342512,"statfs":{"total":96636764160,"available":96606732288,"internally_reserved":0,"allocated":2363392,"data_stored":1198720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510368,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59880,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311960,"statfs":{"total":96636764160,"available":96575447040,"internally_reserved":0,"allocated":2379776,"data_stored":1206892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575776,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64000,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307840,"statfs":{"total":96636764160,"available":96571228160,"internally_reserved":0,"allocated":2347008,"data_stored":1197956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510367,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59872,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311968,"statfs":{"total":96636764160,"available":96575455232,"internally_reserved":0,"allocated":2371584,"data_stored":1206288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510369,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":2949120,"data_stored":1788428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:39:16.516 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:39:16.516 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:39:16.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.676+0000 7f0b2d226640 1 -- 172.21.15.59:0/320303824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 msgr2=0x7f0b280cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.676+0000 7f0b2d226640 1 --2- 172.21.15.59:0/320303824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 0x7f0b280cf000 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f0b10005fd0 tx=0x7f0b1002a530 comp rx=0 tx=0).stop 2024-06-29T15:39:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.676+0000 7f0b2d226640 1 -- 172.21.15.59:0/320303824 shutdown_connections 2024-06-29T15:39:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.676+0000 7f0b2d226640 1 --2- 172.21.15.59:0/320303824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 0x7f0b280cf000 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.676+0000 7f0b2d226640 1 --2- 172.21.15.59:0/320303824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b280ce510 0x7f0b2815dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.676+0000 7f0b2d226640 1 -- 172.21.15.59:0/320303824 >> 172.21.15.59:0/320303824 conn(0x7f0b280d3e40 msgr2=0x7f0b280d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.676+0000 7f0b2d226640 1 -- 172.21.15.59:0/320303824 shutdown_connections 2024-06-29T15:39:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.676+0000 7f0b2d226640 1 -- 172.21.15.59:0/320303824 wait complete. 2024-06-29T15:39:16.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.678+0000 7f0b2d226640 1 Processor -- start 2024-06-29T15:39:16.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.678+0000 7f0b2d226640 1 -- start start 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.679+0000 7f0b2d226640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b280ce510 0x7f0b28160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.679+0000 7f0b2d226640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 0x7f0b281e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.679+0000 7f0b2d226640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b28161980 0x7f0b281f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.679+0000 7f0b2d226640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0b280b2b90 con 0x7f0b280cebf0 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.679+0000 7f0b2d226640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b280b2a10 con 0x7f0b280ce510 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.679+0000 7f0b2d226640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0b280b4bc0 con 0x7f0b28161980 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.679+0000 7f0b2659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 0x7f0b281e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.679+0000 7f0b2659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 0x7f0b281e77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52912/0 (socket says 172.21.15.59:52912) 2024-06-29T15:39:16.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2659c640 1 -- 172.21.15.59:0/1337046031 learned_addr learned my addr 172.21.15.59:0/1337046031 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b26d9d640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b280ce510 0x7f0b28160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2759e640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b28161980 0x7f0b281f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2659c640 1 -- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b28161980 msgr2=0x7f0b281f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2659c640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b28161980 0x7f0b281f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2659c640 1 -- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b280ce510 msgr2=0x7f0b28160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2659c640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b280ce510 0x7f0b28160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2659c640 1 -- 172.21.15.59:0/1337046031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b281e8440 con 0x7f0b280cebf0 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2659c640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 0x7f0b281e77c0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f0b10000c00 tx=0x7f0b10003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b1f7fe640 1 -- 172.21.15.59:0/1337046031 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1003e030 con 0x7f0b280cebf0 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b1f7fe640 1 -- 172.21.15.59:0/1337046031 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b10003e00 con 0x7f0b280cebf0 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b281e8720 con 0x7f0b280cebf0 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b1f7fe640 1 -- 172.21.15.59:0/1337046031 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b10035040 con 0x7f0b280cebf0 2024-06-29T15:39:16.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.680+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b28148e10 con 0x7f0b280cebf0 2024-06-29T15:39:16.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.681+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b281e81b0 con 0x7f0b280cebf0 2024-06-29T15:39:16.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.682+0000 7f0b1f7fe640 1 -- 172.21.15.59:0/1337046031 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f0b10041030 con 0x7f0b280cebf0 2024-06-29T15:39:16.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.683+0000 7f0b1f7fe640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0b08076420 0x7f0b08078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.683+0000 7f0b1f7fe640 1 -- 172.21.15.59:0/1337046031 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7f0b100b8300 con 0x7f0b280cebf0 2024-06-29T15:39:16.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.683+0000 7f0b26d9d640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0b08076420 0x7f0b08078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.683+0000 7f0b26d9d640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0b08076420 0x7f0b08078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b20005fd0 tx=0x7f0b200095e0 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.687+0000 7f0b1f7fe640 1 -- 172.21.15.59:0/1337046031 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b280cf540 con 0x7f0b280cebf0 2024-06-29T15:39:16.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.954+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f0b2815df70 con 0x7f0b280cebf0 2024-06-29T15:39:16.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.957+0000 7f0b1f7fe640 1 -- 172.21.15.59:0/1337046031 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v44) ==== 190+0+147 (secure 0 0 0) 0x7f0b2815df70 con 0x7f0b280cebf0 2024-06-29T15:39:16.959 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T15:39:16.959 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T15:39:16.959 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T15:39:16.959 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T15:39:16.959 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:39:16.959 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T15:39:16.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0b08076420 msgr2=0x7f0b08078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0b08076420 0x7f0b08078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b20005fd0 tx=0x7f0b200095e0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 msgr2=0x7f0b281e77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 0x7f0b281e77c0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f0b10000c00 tx=0x7f0b10003910 comp rx=0 tx=0).stop 2024-06-29T15:39:16.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 shutdown_connections 2024-06-29T15:39:16.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f0b08076420 0x7f0b08078930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b28161980 0x7f0b281f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b280cebf0 0x7f0b281e77c0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 --2- 172.21.15.59:0/1337046031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b280ce510 0x7f0b28160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 >> 172.21.15.59:0/1337046031 conn(0x7f0b280d3e40 msgr2=0x7f0b2806c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 shutdown_connections 2024-06-29T15:39:16.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:16.960+0000 7f0b2d226640 1 -- 172.21.15.59:0/1337046031 wait complete. 2024-06-29T15:39:16.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:39:17.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.089+0000 7ffa0d50a640 1 -- 172.21.15.59:0/1501009319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa0806e3a0 msgr2=0x7ffa0815c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.089+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/1501009319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa0806e3a0 0x7ffa0815c730 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8005fd0 tx=0x7ff9f802a4a0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.089+0000 7ffa0d50a640 1 -- 172.21.15.59:0/1501009319 shutdown_connections 2024-06-29T15:39:17.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.089+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/1501009319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa0806e3a0 0x7ffa0815c730 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.089+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/1501009319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa08144970 0x7ffa0815f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.089+0000 7ffa0d50a640 1 -- 172.21.15.59:0/1501009319 >> 172.21.15.59:0/1501009319 conn(0x7ffa080b2ab0 msgr2=0x7ffa080b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.089+0000 7ffa0d50a640 1 -- 172.21.15.59:0/1501009319 shutdown_connections 2024-06-29T15:39:17.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.089+0000 7ffa0d50a640 1 -- 172.21.15.59:0/1501009319 wait complete. 2024-06-29T15:39:17.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.091+0000 7ffa0d50a640 1 Processor -- start 2024-06-29T15:39:17.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.091+0000 7ffa0d50a640 1 -- start start 2024-06-29T15:39:17.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.092+0000 7ffa0d50a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa080cfa00 0x7ffa08143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.092+0000 7ffa0d50a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa08144970 0x7ffa08139210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.092+0000 7ffa0d50a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0813cbf0 0x7ffa08139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.092+0000 7ffa0d50a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0815d2a0 con 0x7ffa080cfa00 2024-06-29T15:39:17.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.092+0000 7ffa0d50a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0815d120 con 0x7ffa0813cbf0 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.092+0000 7ffa0d50a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0815d420 con 0x7ffa08144970 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa080cfa00 0x7ffa08143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa080cfa00 0x7ffa08143d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52920/0 (socket says 172.21.15.59:52920) 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 -- 172.21.15.59:0/3327988184 learned_addr learned my addr 172.21.15.59:0/3327988184 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa077fe640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0813cbf0 0x7ffa08139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa067fc640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa08144970 0x7ffa08139210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 -- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa08144970 msgr2=0x7ffa08139210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa08144970 0x7ffa08139210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 -- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0813cbf0 msgr2=0x7ffa08139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0813cbf0 0x7ffa08139750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 -- 172.21.15.59:0/3327988184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa080a9780 con 0x7ffa080cfa00 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa077fe640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0813cbf0 0x7ffa08139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:17.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa067fc640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa08144970 0x7ffa08139210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:17.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa06ffd640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa080cfa00 0x7ffa08143d10 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000cce0 tx=0x7ffa0000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ff9ff7fe640 1 -- 172.21.15.59:0/3327988184 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000f450 con 0x7ffa080cfa00 2024-06-29T15:39:17.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ff9ff7fe640 1 -- 172.21.15.59:0/3327988184 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa0000f6e0 con 0x7ffa080cfa00 2024-06-29T15:39:17.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0813c110 con 0x7ffa080cfa00 2024-06-29T15:39:17.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ff9ff7fe640 1 -- 172.21.15.59:0/3327988184 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000fb30 con 0x7ffa080cfa00 2024-06-29T15:39:17.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.093+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa0813c6d0 con 0x7ffa080cfa00 2024-06-29T15:39:17.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.094+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa080cf390 con 0x7ffa080cfa00 2024-06-29T15:39:17.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.095+0000 7ff9ff7fe640 1 -- 172.21.15.59:0/3327988184 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7ffa0000fcd0 con 0x7ffa080cfa00 2024-06-29T15:39:17.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.096+0000 7ff9ff7fe640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7ff9e80763f0 0x7ff9e8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.096+0000 7ff9ff7fe640 1 -- 172.21.15.59:0/3327988184 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7ffa00098670 con 0x7ffa080cfa00 2024-06-29T15:39:17.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.096+0000 7ffa067fc640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7ff9e80763f0 0x7ff9e8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.097+0000 7ffa067fc640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7ff9e80763f0 0x7ff9e8078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa08141810 tx=0x7ff9f0008040 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.100+0000 7ff9ff7fe640 1 -- 172.21.15.59:0/3327988184 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa080cf390 con 0x7ffa080cfa00 2024-06-29T15:39:17.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.328+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa0815d940 con 0x7ffa080cfa00 2024-06-29T15:39:17.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.328+0000 7ff9ff7fe640 1 -- 172.21.15.59:0/3327988184 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5235 (secure 0 0 0) 0x7ffa00014030 con 0x7ffa080cfa00 2024-06-29T15:39:17.330 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:39:17.330 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"btime":"2024-06-29T15:39:15:059125+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8588,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3866646629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3866646629},{"type":"v1","addr":"172.21.15.153:6835","nonce":3866646629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9724,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/933275096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":933275096},{"type":"v1","addr":"172.21.15.59:6835","nonce":933275096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9736,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1230229255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1230229255},{"type":"v1","addr":"172.21.15.153:6837","nonce":1230229255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.430152+0000","modified":"2024-06-29T15:39:14.448659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2399523179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2399523179},{"type":"v1","addr":"172.21.15.59:6834","nonce":2399523179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10401,"qdb_cluster":[10401]},"id":11}]} 2024-06-29T15:39:17.330 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 97 2024-06-29T15:39:17.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7ff9e80763f0 msgr2=0x7ff9e8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7ff9e80763f0 0x7ff9e8078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa08141810 tx=0x7ff9f0008040 comp rx=0 tx=0).stop 2024-06-29T15:39:17.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa080cfa00 msgr2=0x7ffa08143d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa080cfa00 0x7ffa08143d10 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000cce0 tx=0x7ffa0000a620 comp rx=0 tx=0).stop 2024-06-29T15:39:17.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 shutdown_connections 2024-06-29T15:39:17.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7ff9e80763f0 0x7ff9e8078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa0813cbf0 0x7ffa08139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa08144970 0x7ffa08139210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 --2- 172.21.15.59:0/3327988184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa080cfa00 0x7ffa08143d10 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 >> 172.21.15.59:0/3327988184 conn(0x7ffa080b2ab0 msgr2=0x7ffa0815b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.330+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 shutdown_connections 2024-06-29T15:39:17.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.331+0000 7ffa0d50a640 1 -- 172.21.15.59:0/3327988184 wait complete. 2024-06-29T15:39:17.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:39:11.430152+0000', 'modified': '2024-06-29T15:39:14.448659+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10401}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10401': {'gid': 10401, 'name': 'c', 'rank': 0, 'incarnation': 93, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6834/2399523179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 2399523179}, {'type': 'v1', 'addr': '172.21.15.59:6834', 'nonce': 2399523179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10401, 'qdb_cluster': [10401]} 2024-06-29T15:39:17.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:39:17.746 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-06-29T15:39:17.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.869+0000 7fc21ff8c640 1 -- 172.21.15.59:0/2037237070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180ad4e0 msgr2=0x7fc21815b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.869+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/2037237070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180ad4e0 0x7fc21815b140 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fc210009830 tx=0x7fc21002a510 comp rx=0 tx=0).stop 2024-06-29T15:39:17.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.869+0000 7fc21ff8c640 1 -- 172.21.15.59:0/2037237070 shutdown_connections 2024-06-29T15:39:17.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.869+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/2037237070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180ad4e0 0x7fc21815b140 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.869+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/2037237070 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2180cc060 0x7fc2180acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.869+0000 7fc21ff8c640 1 -- 172.21.15.59:0/2037237070 >> 172.21.15.59:0/2037237070 conn(0x7fc2180b1ea0 msgr2=0x7fc2180b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.869+0000 7fc21ff8c640 1 -- 172.21.15.59:0/2037237070 shutdown_connections 2024-06-29T15:39:17.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.869+0000 7fc21ff8c640 1 -- 172.21.15.59:0/2037237070 wait complete. 2024-06-29T15:39:17.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.871+0000 7fc21ff8c640 1 Processor -- start 2024-06-29T15:39:17.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.871+0000 7fc21ff8c640 1 -- start start 2024-06-29T15:39:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.872+0000 7fc21ff8c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2180ad4e0 0x7fc218141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.872+0000 7fc21ff8c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2180cb490 0x7fc2180b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.872+0000 7fc21ff8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180cc060 0x7fc2181597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.872+0000 7fc21ff8c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc21815bc00 con 0x7fc2180cc060 2024-06-29T15:39:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.872+0000 7fc21ff8c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc21815ba80 con 0x7fc2180cb490 2024-06-29T15:39:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.872+0000 7fc21ff8c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc21815bd80 con 0x7fc2180ad4e0 2024-06-29T15:39:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.872+0000 7fc21e529640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180cc060 0x7fc2181597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21e529640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180cc060 0x7fc2181597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52930/0 (socket says 172.21.15.59:52930) 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21e529640 1 -- 172.21.15.59:0/855126912 learned_addr learned my addr 172.21.15.59:0/855126912 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21d527640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2180cb490 0x7fc2180b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21dd28640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2180ad4e0 0x7fc218141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21e529640 1 -- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2180ad4e0 msgr2=0x7fc218141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21e529640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2180ad4e0 0x7fc218141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21e529640 1 -- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2180cb490 msgr2=0x7fc2180b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21e529640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2180cb490 0x7fc2180b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21e529640 1 -- 172.21.15.59:0/855126912 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc218143f60 con 0x7fc2180cc060 2024-06-29T15:39:17.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21d527640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2180cb490 0x7fc2180b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:17.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21e529640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180cc060 0x7fc2181597d0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fc21002ab00 tx=0x7fc210033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc20e7fc640 1 -- 172.21.15.59:0/855126912 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210031b50 con 0x7fc2180cc060 2024-06-29T15:39:17.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc20e7fc640 1 -- 172.21.15.59:0/855126912 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc210031de0 con 0x7fc2180cc060 2024-06-29T15:39:17.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21ff8c640 1 -- 172.21.15.59:0/855126912 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc218144240 con 0x7fc2180cc060 2024-06-29T15:39:17.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc20e7fc640 1 -- 172.21.15.59:0/855126912 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210005280 con 0x7fc2180cc060 2024-06-29T15:39:17.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.873+0000 7fc21ff8c640 1 -- 172.21.15.59:0/855126912 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc218144800 con 0x7fc2180cc060 2024-06-29T15:39:17.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.875+0000 7fc20e7fc640 1 -- 172.21.15.59:0/855126912 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fc210005420 con 0x7fc2180cc060 2024-06-29T15:39:17.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.876+0000 7fc20e7fc640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc1fc0763f0 0x7fc1fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.876+0000 7fc20e7fc640 1 -- 172.21.15.59:0/855126912 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7fc2100b8eb0 con 0x7fc2180cc060 2024-06-29T15:39:17.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.876+0000 7fc21dd28640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc1fc0763f0 0x7fc1fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.877+0000 7fc226e48740 1 Processor -- start 2024-06-29T15:39:17.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.877+0000 7fc226e48740 1 -- start start 2024-06-29T15:39:17.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.877+0000 7fc21dd28640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc1fc0763f0 0x7fc1fc078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc214004a30 tx=0x7fc214001a70 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc226e48740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 0x55c74aefd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc226e48740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aee13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc226e48740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aee18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc226e48740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55c74ae79bc0 con 0x55c74aae83b0 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc226e48740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c74ae28630 con 0x55c74aaf0590 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc226e48740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55c74ae17bd0 con 0x55c74ab11270 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21d527640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aee13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21d527640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aee13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52946/0 (socket says 172.21.15.59:52946) 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21d527640 1 -- 172.21.15.59:0/3288692051 learned_addr learned my addr 172.21.15.59:0/3288692051 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21dd28640 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 0x55c74aefd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21e529640 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aee18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21d527640 1 -- 172.21.15.59:0/3288692051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 msgr2=0x55c74aefd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21d527640 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 0x55c74aefd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21d527640 1 -- 172.21.15.59:0/3288692051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 msgr2=0x55c74aee18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21d527640 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aee18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.878+0000 7fc21d527640 1 -- 172.21.15.59:0/3288692051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c74ae14ca0 con 0x55c74aae83b0 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.879+0000 7fc21d527640 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aee13a0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fc20800dcc0 tx=0x7fc20800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37a2069bfe71677 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.879+0000 7fc1fb7fe640 1 -- 172.21.15.59:0/3288692051 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208004030 con 0x55c74aae83b0 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.879+0000 7fc1fb7fe640 1 -- 172.21.15.59:0/3288692051 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2080042c0 con 0x55c74aae83b0 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.879+0000 7fc1fb7fe640 1 -- 172.21.15.59:0/3288692051 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208003850 con 0x55c74aae83b0 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 -- 172.21.15.59:0/3288692051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 msgr2=0x55c74aee13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aee13a0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fc20800dcc0 tx=0x7fc20800f730 comp rx=0 tx=0).stop 2024-06-29T15:39:17.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 -- 172.21.15.59:0/3288692051 shutdown_connections 2024-06-29T15:39:17.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aee18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aee13a0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 --2- 172.21.15.59:0/3288692051 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 0x55c74aefd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 -- 172.21.15.59:0/3288692051 >> 172.21.15.59:0/3288692051 conn(0x55c74ab053c0 msgr2=0x55c74aeb4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 -- 172.21.15.59:0/3288692051 shutdown_connections 2024-06-29T15:39:17.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.880+0000 7fc226e48740 1 -- 172.21.15.59:0/3288692051 wait complete. 2024-06-29T15:39:17.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.881+0000 7fc226e48740 1 -- start start 2024-06-29T15:39:17.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.881+0000 7fc226e48740 1 Processor -- start 2024-06-29T15:39:17.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc226e48740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aae83b0 0x55c74aeccb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc226e48740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aefcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc226e48740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af0f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc226e48740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55c74adb5ae0 con 0x55c74ab11270 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc226e48740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c74ae1bc10 con 0x55c74aaf0590 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc226e48740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55c74ab8b620 con 0x55c74aae83b0 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc226e48740 10 client.0 _ll_register_callbacks cb 0x55c74adc2480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc21e529640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af0f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc21e529640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af0f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52950/0 (socket says 172.21.15.59:52950) 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc21e529640 1 -- 172.21.15.59:0/515495776 learned_addr learned my addr 172.21.15.59:0/515495776 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc21dd28640 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aae83b0 0x55c74aeccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.882+0000 7fc21d527640 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aefcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc21e529640 1 -- 172.21.15.59:0/515495776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aae83b0 msgr2=0x55c74aeccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc21e529640 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aae83b0 0x55c74aeccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc21e529640 1 -- 172.21.15.59:0/515495776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 msgr2=0x55c74aefcca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc21e529640 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aefcca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc21e529640 1 -- 172.21.15.59:0/515495776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c74ab7c850 con 0x55c74ab11270 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc21e529640 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af0f7d0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fc210081b80 tx=0x7fc2100824a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b72a444f9607c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc1f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/515495776 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210048070 con 0x55c74ab11270 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/515495776 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7fc21009cd00 con 0x55c74ab11270 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc226e48740 1 -- 172.21.15.59:0/515495776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2080042c0 con 0x55c74ab11270 2024-06-29T15:39:17.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/515495776 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fc2100a3050 con 0x55c74ab11270 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc226e48740 10 client.10545 fetch_fsmap learned FSMap version 97 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc226e48740 1 -- 172.21.15.59:0/515495776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) -- 0x55c74abf36b0 con 0x55c74ab11270 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/515495776 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2080042c0 con 0x55c74ab11270 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.883+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/515495776 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 97) ==== 3502+0+0 (secure 0 0 0) 0x7fc2100b3030 con 0x55c74ab11270 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc226e48740 10 client.10545 fetch_fsmap finished waiting for FSMap version 97 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc226e48740 10 client.10545 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc226e48740 20 client.10545 populate_metadata read hostname 'smithi059' 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc226e48740 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] conn(0x55c74aacfe40 0x55c74af19ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc226e48740 4 client.10545 mds_command: new command op to 10401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc226e48740 1 -- 172.21.15.59:0/515495776 --> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c74ae2a580 con 0x55c74aacfe40 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc21dd28640 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] conn(0x55c74aacfe40 0x55c74af19ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc1f8ff9640 10 client.10545 ms_handle_connect on v2:172.21.15.59:6832/2399523179 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.884+0000 7fc21dd28640 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] conn(0x55c74aacfe40 0x55c74af19ac0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d03140f9627fdaaa server_cookie=a31643ed09349dfc in_seq=0 out_seq=0 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/515495776 <== mds.0 v2:172.21.15.59:6832/2399523179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c74ae2a580 con 0x55c74aacfe40 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc1f8ff9640 10 client.10545 handle_command_reply: tid=0 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 client.10545 shutdown 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 -- 172.21.15.59:0/515495776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 msgr2=0x55c74af0f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af0f7d0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fc210081b80 tx=0x7fc2100824a0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 -- 172.21.15.59:0/515495776 shutdown_connections 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] conn(0x55c74aacfe40 0x55c74af19ac0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af0f7d0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aefcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc1f8ff9640 0 client.10545 ms_handle_reset on v2:172.21.15.59:6832/2399523179 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 --2- 172.21.15.59:0/515495776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aae83b0 0x55c74aeccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 -- 172.21.15.59:0/515495776 >> 172.21.15.59:0/515495776 conn(0x55c74ab053c0 msgr2=0x55c74aeddc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 -- 172.21.15.59:0/515495776 shutdown_connections 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 1 -- 172.21.15.59:0/515495776 wait complete. 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.885+0000 7fc226e48740 20 client.10545 trim_cache size 0 max 16384 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.904+0000 7fc226e48740 1 Processor -- start 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.905+0000 7fc226e48740 1 -- start start 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc226e48740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aae83b0 0x55c74aead480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc226e48740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aaf0590 0x55c74aefca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc226e48740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af73910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc226e48740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc218200e20 con 0x55c74ab11270 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc226e48740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc218142a10 con 0x55c74aae83b0 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc226e48740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc218143060 con 0x55c74aaf0590 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af73910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af73910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52958/0 (socket says 172.21.15.59:52958) 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 -- 172.21.15.59:0/602495822 learned_addr learned my addr 172.21.15.59:0/602495822 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21d527640 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aaf0590 0x55c74aefca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21dd28640 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aae83b0 0x55c74aead480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 -- 172.21.15.59:0/602495822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aaf0590 msgr2=0x55c74aefca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aaf0590 0x55c74aefca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 -- 172.21.15.59:0/602495822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aae83b0 msgr2=0x55c74aead480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aae83b0 0x55c74aead480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 -- 172.21.15.59:0/602495822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c74aee0600 con 0x55c74ab11270 2024-06-29T15:39:17.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.906+0000 7fc21e529640 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af73910 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fc2100094e0 tx=0x7fc210005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7f4d81177816fb server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc1faffd640 1 -- 172.21.15.59:0/602495822 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210048070 con 0x55c74ab11270 2024-06-29T15:39:17.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc1faffd640 1 -- 172.21.15.59:0/602495822 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc21009a040 con 0x55c74ab11270 2024-06-29T15:39:17.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc1faffd640 1 -- 172.21.15.59:0/602495822 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210031770 con 0x55c74ab11270 2024-06-29T15:39:17.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 -- 172.21.15.59:0/602495822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 msgr2=0x55c74af73910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af73910 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fc2100094e0 tx=0x7fc210005e30 comp rx=0 tx=0).stop 2024-06-29T15:39:17.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 -- 172.21.15.59:0/602495822 shutdown_connections 2024-06-29T15:39:17.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74ab11270 0x55c74af73910 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74aaf0590 0x55c74aefca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 --2- 172.21.15.59:0/602495822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aae83b0 0x55c74aead480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 -- 172.21.15.59:0/602495822 >> 172.21.15.59:0/602495822 conn(0x55c74ab053c0 msgr2=0x55c74aeb88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 -- 172.21.15.59:0/602495822 shutdown_connections 2024-06-29T15:39:17.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.907+0000 7fc226e48740 1 -- 172.21.15.59:0/602495822 wait complete. 2024-06-29T15:39:17.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.908+0000 7fc226e48740 1 -- start start 2024-06-29T15:39:17.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.908+0000 7fc226e48740 1 Processor -- start 2024-06-29T15:39:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc226e48740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aeb8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc226e48740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aeb8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc226e48740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 0x55c74af79eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc226e48740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55c74aefd1e0 con 0x55c74aae83b0 2024-06-29T15:39:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc226e48740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c74aecddf0 con 0x55c74aaf0590 2024-06-29T15:39:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc226e48740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55c74aee03e0 con 0x55c74ab11270 2024-06-29T15:39:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc226e48740 10 client.0 _ll_register_callbacks cb 0x7fc21007dd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aeb8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aeb8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52964/0 (socket says 172.21.15.59:52964) 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 -- 172.21.15.59:0/2376291181 learned_addr learned my addr 172.21.15.59:0/2376291181 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21e529640 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 0x55c74af79eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21d527640 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aeb8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 -- 172.21.15.59:0/2376291181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 msgr2=0x55c74af79eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 0x55c74af79eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 -- 172.21.15.59:0/2376291181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 msgr2=0x55c74aeb8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aeb8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 -- 172.21.15.59:0/2376291181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c74aef6390 con 0x55c74aae83b0 2024-06-29T15:39:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc21dd28640 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aeb8710 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x55c74aee0560 tx=0x7fc214001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f71714143c40fd server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc1f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/2376291181 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21401b040 con 0x55c74aae83b0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/2376291181 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7fc214010040 con 0x55c74aae83b0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.910+0000 7fc226e48740 1 -- 172.21.15.59:0/2376291181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc21009a040 con 0x55c74aae83b0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/2376291181 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) ==== 24+0+0 (secure 0 0 0) 0x7fc214035050 con 0x55c74aae83b0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc226e48740 10 client.10551 fetch_fsmap learned FSMap version 97 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc226e48740 1 -- 172.21.15.59:0/2376291181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) -- 0x55c74af0e4d0 con 0x55c74aae83b0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/2376291181 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc21009a040 con 0x55c74aae83b0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/2376291181 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 97) ==== 3502+0+0 (secure 0 0 0) 0x7fc214038020 con 0x55c74aae83b0 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc226e48740 10 client.10551 fetch_fsmap finished waiting for FSMap version 97 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc226e48740 10 client.10551 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc226e48740 20 client.10551 populate_metadata read hostname 'smithi059' 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc226e48740 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] conn(0x55c74af7a810 0x55c74aef27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc226e48740 4 client.10551 mds_command: new command op to 10401 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc226e48740 1 -- 172.21.15.59:0/2376291181 --> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] -- command(tid 0: {"prefix": "status"}) -- 0x55c74ae26020 con 0x55c74af7a810 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc21d527640 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] conn(0x55c74af7a810 0x55c74aef27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.911+0000 7fc1f8ff9640 10 client.10551 ms_handle_connect on v2:172.21.15.59:6832/2399523179 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc21d527640 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] conn(0x55c74af7a810 0x55c74aef27d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88f883a5c3361661 server_cookie=20073c4d75f37968 in_seq=0 out_seq=0 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc1f8ff9640 1 -- 172.21.15.59:0/2376291181 <== mds.0 v2:172.21.15.59:6832/2399523179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c74ae26020 con 0x55c74af7a810 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc1f8ff9640 10 client.10551 handle_command_reply: tid=0 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 client.10551 shutdown 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 -- 172.21.15.59:0/2376291181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 msgr2=0x55c74aeb8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aeb8710 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x55c74aee0560 tx=0x7fc214001250 comp rx=0 tx=0).stop 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 -- 172.21.15.59:0/2376291181 shutdown_connections 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.59:6832/2399523179,v1:172.21.15.59:6834/2399523179] conn(0x55c74af7a810 0x55c74aef27d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c74ab11270 0x55c74af79eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c74aaf0590 0x55c74aeb8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 --2- 172.21.15.59:0/2376291181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55c74aae83b0 0x55c74aeb8710 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc1f8ff9640 0 client.10551 ms_handle_reset on v2:172.21.15.59:6832/2399523179 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 -- 172.21.15.59:0/2376291181 >> 172.21.15.59:0/2376291181 conn(0x55c74ab053c0 msgr2=0x55c74aefd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 -- 172.21.15.59:0/2376291181 shutdown_connections 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 1 -- 172.21.15.59:0/2376291181 wait complete. 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.912+0000 7fc226e48740 20 client.10551 trim_cache size 0 max 16384 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:39:17.919 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 10401, 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.867952517, 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 97, 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 189, 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 189, 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.498381501000001 2024-06-29T15:39:17.920 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:39:17.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 -- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc1fc0763f0 msgr2=0x7fc1fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc1fc0763f0 0x7fc1fc078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc214004a30 tx=0x7fc214001a70 comp rx=0 tx=0).stop 2024-06-29T15:39:17.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 -- 172.21.15.59:0/855126912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180cc060 msgr2=0x7fc2181597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180cc060 0x7fc2181597d0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fc21002ab00 tx=0x7fc210033ba0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 -- 172.21.15.59:0/855126912 shutdown_connections 2024-06-29T15:39:17.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fc1fc0763f0 0x7fc1fc078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2180cc060 0x7fc2181597d0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2180cb490 0x7fc2180b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 --2- 172.21.15.59:0/855126912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2180ad4e0 0x7fc218141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 -- 172.21.15.59:0/855126912 >> 172.21.15.59:0/855126912 conn(0x7fc2180b1ea0 msgr2=0x7fc2180cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 -- 172.21.15.59:0/855126912 shutdown_connections 2024-06-29T15:39:17.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:17.913+0000 7fc21ff8c640 1 -- 172.21.15.59:0/855126912 wait complete. 2024-06-29T15:39:17.925 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:39:18.015 INFO:tasks.ceph.mds.c.smithi059.stderr:2024-06-29T15:39:18.014+0000 7fac87fa7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:39:18.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.042+0000 7f39b7226640 1 -- 172.21.15.59:0/3256136268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00b3ed0 msgr2=0x7f39b015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.042+0000 7f39b7226640 1 --2- 172.21.15.59:0/3256136268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00b3ed0 0x7f39b015e8d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f399c005fd0 tx=0x7f399c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.042+0000 7f39b7226640 1 -- 172.21.15.59:0/3256136268 shutdown_connections 2024-06-29T15:39:18.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.042+0000 7f39b7226640 1 --2- 172.21.15.59:0/3256136268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00b3ed0 0x7f39b015e8d0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.042+0000 7f39b7226640 1 --2- 172.21.15.59:0/3256136268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39b00d23c0 0x7f39b00b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.042+0000 7f39b7226640 1 -- 172.21.15.59:0/3256136268 >> 172.21.15.59:0/3256136268 conn(0x7f39b00d3ee0 msgr2=0x7f39b00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:18.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.042+0000 7f39b7226640 1 -- 172.21.15.59:0/3256136268 shutdown_connections 2024-06-29T15:39:18.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.042+0000 7f39b7226640 1 -- 172.21.15.59:0/3256136268 wait complete. 2024-06-29T15:39:18.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.044+0000 7f39b7226640 1 Processor -- start 2024-06-29T15:39:18.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.044+0000 7f39b7226640 1 -- start start 2024-06-29T15:39:18.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39b7226640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00b3ed0 0x7f39b0152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:18.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39b7226640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00d23c0 0x7f39b014f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:18.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39b7226640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39b015ee10 0x7f39b01e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:18.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39b7226640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f39b0160240 con 0x7f39b00d23c0 2024-06-29T15:39:18.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39b7226640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39b01600c0 con 0x7f39b00b3ed0 2024-06-29T15:39:18.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39b7226640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f39b01603c0 con 0x7f39b015ee10 2024-06-29T15:39:18.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39affff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00d23c0 0x7f39b014f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:18.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39affff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00d23c0 0x7f39b014f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52988/0 (socket says 172.21.15.59:52988) 2024-06-29T15:39:18.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39affff640 1 -- 172.21.15.59:0/3444211171 learned_addr learned my addr 172.21.15.59:0/3444211171 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:18.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39b4fc2640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00b3ed0 0x7f39b0152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:18.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.045+0000 7f39b57c3640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39b015ee10 0x7f39b01e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:18.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39affff640 1 -- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39b015ee10 msgr2=0x7f39b01e8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39affff640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39b015ee10 0x7f39b01e8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39affff640 1 -- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00b3ed0 msgr2=0x7f39b0152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39affff640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00b3ed0 0x7f39b0152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39affff640 1 -- 172.21.15.59:0/3444211171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b01533f0 con 0x7f39b00d23c0 2024-06-29T15:39:18.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39affff640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00d23c0 0x7f39b014f110 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f399c006010 tx=0x7f399c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:18.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39ad7fa640 1 -- 172.21.15.59:0/3444211171 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c03e030 con 0x7f39b00d23c0 2024-06-29T15:39:18.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39ad7fa640 1 -- 172.21.15.59:0/3444211171 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f399c003ca0 con 0x7f39b00d23c0 2024-06-29T15:39:18.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b0153900 con 0x7f39b00d23c0 2024-06-29T15:39:18.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39ad7fa640 1 -- 172.21.15.59:0/3444211171 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c035040 con 0x7f39b00d23c0 2024-06-29T15:39:18.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.046+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b0153ec0 con 0x7f39b00d23c0 2024-06-29T15:39:18.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.047+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b0153650 con 0x7f39b00d23c0 2024-06-29T15:39:18.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.048+0000 7f39ad7fa640 1 -- 172.21.15.59:0/3444211171 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f399c041030 con 0x7f39b00d23c0 2024-06-29T15:39:18.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.049+0000 7f39ad7fa640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3994076420 0x7f3994078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:18.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.049+0000 7f39ad7fa640 1 -- 172.21.15.59:0/3444211171 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7f399c0b8070 con 0x7f39b00d23c0 2024-06-29T15:39:18.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.049+0000 7f39b4fc2640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3994076420 0x7f3994078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:18.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.049+0000 7f39b4fc2640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3994076420 0x7f3994078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f39b00d2e40 tx=0x7f39a400b2e0 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:18.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.053+0000 7f39ad7fa640 1 -- 172.21.15.59:0/3444211171 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f399c080f30 con 0x7f39b00d23c0 2024-06-29T15:39:18.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.286+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f39b015e8d0 con 0x7f39b00d23c0 2024-06-29T15:39:18.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.286+0000 7f39ad7fa640 1 -- 172.21.15.59:0/3444211171 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) ==== 53+0+73 (secure 0 0 0) 0x7f399c085230 con 0x7f39b00d23c0 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3994076420 msgr2=0x7f3994078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3994076420 0x7f3994078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f39b00d2e40 tx=0x7f39a400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00d23c0 msgr2=0x7f39b014f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00d23c0 0x7f39b014f110 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f399c006010 tx=0x7f399c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 shutdown_connections 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f3994076420 0x7f3994078930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39b015ee10 0x7f39b01e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00d23c0 0x7f39b014f110 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 --2- 172.21.15.59:0/3444211171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b00b3ed0 0x7f39b0152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 >> 172.21.15.59:0/3444211171 conn(0x7f39b00d3ee0 msgr2=0x7f39b00ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 shutdown_connections 2024-06-29T15:39:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18.288+0000 7f39b7226640 1 -- 172.21.15.59:0/3444211171 wait complete. 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:39:18.301 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:39:18.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.302 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T15:39:18.318 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:39:18.318 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:39:18.318 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T15:39:18.319 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 38092sec preferred_lft 38092sec 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.320 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:39:18.321 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:39:18.321 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T15:39:18.321 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.321 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:39:18.321 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:39:18.321 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T15:39:18.321 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.321 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:39:18.382 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T15:39:18.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.384 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T15:39:18.441 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T15:39:18.442 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:39:18.442 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T15:39:18.442 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T15:39:18.442 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:39:18.442 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:39:18.442 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T15:39:18.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T15:39:18.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T15:39:18.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T15:39:18.442 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T15:39:18.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:39:18.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:39:18.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.572 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T15:39:18.587 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:39:18.588 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:39:18.588 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:39:18.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T15:39:18.653 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:39:18.654 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:39:18.709 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:39:18.710 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:39:18.710 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:39:18.765 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T15:39:18.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.765 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:39:18.859 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:39:18.859 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:39:18.859 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:39:18.870 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:39:18.870 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:39:18.870 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:39:18.870 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:39:18.870 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:39:18.876 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:39:18.877 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:39:18.878 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:39:18.878 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:39:18.878 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:39:18.879 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:39:18.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.880 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:39:18.905 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:39:18.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:39:19.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:39:19.009 INFO:teuthology.orchestra.run.smithi059.stdout:client8693 2024-06-29T15:39:19.009 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:39:19.009 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:19.025 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:39:19.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.183+0000 7f54ecb43640 1 -- 172.21.15.59:0/3857551917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 msgr2=0x7f54e80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.183+0000 7f54ecb43640 1 --2- 172.21.15.59:0/3857551917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 0x7f54e80b3990 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f54e0005fd0 tx=0x7f54e002a670 comp rx=0 tx=0).stop 2024-06-29T15:39:19.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.183+0000 7f54ecb43640 1 -- 172.21.15.59:0/3857551917 shutdown_connections 2024-06-29T15:39:19.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.183+0000 7f54ecb43640 1 --2- 172.21.15.59:0/3857551917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 0x7f54e80b3990 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.183+0000 7f54ecb43640 1 -- 172.21.15.59:0/3857551917 >> 172.21.15.59:0/3857551917 conn(0x7f54e80d3ee0 msgr2=0x7f54e80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:19.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.183+0000 7f54ecb43640 1 -- 172.21.15.59:0/3857551917 shutdown_connections 2024-06-29T15:39:19.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.183+0000 7f54ecb43640 1 -- 172.21.15.59:0/3857551917 wait complete. 2024-06-29T15:39:19.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.185+0000 7f54ecb43640 1 Processor -- start 2024-06-29T15:39:19.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.185+0000 7f54ecb43640 1 -- start start 2024-06-29T15:39:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.186+0000 7f54ecb43640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54e80b3ed0 0x7f54e814ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54ecb43640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 0x7f54e81557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54ecb43640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e815efe0 0x7f54e8157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54ecb43640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f54e8160450 con 0x7f54e80d23c0 2024-06-29T15:39:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54ecb43640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81602d0 con 0x7f54e815efe0 2024-06-29T15:39:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54ecb43640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f54e81605d0 con 0x7f54e80b3ed0 2024-06-29T15:39:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 0x7f54e81557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 0x7f54e81557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53004/0 (socket says 172.21.15.59:53004) 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 -- 172.21.15.59:0/3998694244 learned_addr learned my addr 172.21.15.59:0/3998694244 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e659c640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54e80b3ed0 0x7f54e814ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e6d9d640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e815efe0 0x7f54e8157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 -- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54e80b3ed0 msgr2=0x7f54e814ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54e80b3ed0 0x7f54e814ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 -- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e815efe0 msgr2=0x7f54e8157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e815efe0 0x7f54e8157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 -- 172.21.15.59:0/3998694244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e8155d00 con 0x7f54e80d23c0 2024-06-29T15:39:19.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54e5d9b640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 0x7f54e81557c0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f54d000c400 tx=0x7f54d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:19.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54deffd640 1 -- 172.21.15.59:0/3998694244 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0017070 con 0x7f54e80d23c0 2024-06-29T15:39:19.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54deffd640 1 -- 172.21.15.59:0/3998694244 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54d00053c0 con 0x7f54e80d23c0 2024-06-29T15:39:19.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e81584c0 con 0x7f54e80d23c0 2024-06-29T15:39:19.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54deffd640 1 -- 172.21.15.59:0/3998694244 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0003830 con 0x7f54e80d23c0 2024-06-29T15:39:19.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.187+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e814c590 con 0x7f54e80d23c0 2024-06-29T15:39:19.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.188+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e80b3990 con 0x7f54e80d23c0 2024-06-29T15:39:19.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.189+0000 7f54deffd640 1 -- 172.21.15.59:0/3998694244 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f54d001a070 con 0x7f54e80d23c0 2024-06-29T15:39:19.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.190+0000 7f54deffd640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f54c80763f0 0x7f54c8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.190+0000 7f54deffd640 1 -- 172.21.15.59:0/3998694244 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7f54d0096db0 con 0x7f54e80d23c0 2024-06-29T15:39:19.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.190+0000 7f54e659c640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f54c80763f0 0x7f54c8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.191+0000 7f54e659c640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f54c80763f0 0x7f54c8078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f54e0033000 tx=0x7f54e0004730 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:19.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.194+0000 7f54deffd640 1 -- 172.21.15.59:0/3998694244 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f54d005fe00 con 0x7f54e80d23c0 2024-06-29T15:39:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.408+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 --> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f54e80d2950 con 0x7f54c80763f0 2024-06-29T15:39:19.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.410+0000 7f54deffd640 1 -- 172.21.15.59:0/3998694244 <== mgr.9505 v2:172.21.15.153:6832/1270545814 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f54e80d2950 con 0x7f54c80763f0 2024-06-29T15:39:19.411 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:39:19.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:39:19.412 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T15:39:19.412 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:39:19.412 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:39:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f54c80763f0 msgr2=0x7f54c8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f54c80763f0 0x7f54c8078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f54e0033000 tx=0x7f54e0004730 comp rx=0 tx=0).stop 2024-06-29T15:39:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 msgr2=0x7f54e81557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 0x7f54e81557c0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f54d000c400 tx=0x7f54d000c8c0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 shutdown_connections 2024-06-29T15:39:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f54c80763f0 0x7f54c8078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e815efe0 0x7f54e8157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80d23c0 0x7f54e81557c0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 --2- 172.21.15.59:0/3998694244 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f54e80b3ed0 0x7f54e814ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 >> 172.21.15.59:0/3998694244 conn(0x7f54e80d3ee0 msgr2=0x7f54e806d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:19.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 shutdown_connections 2024-06-29T15:39:19.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.413+0000 7f54ecb43640 1 -- 172.21.15.59:0/3998694244 wait complete. 2024-06-29T15:39:19.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:39:19.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.544+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/4259119455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf940cd9b0 msgr2=0x7fcf9415f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.544+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/4259119455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf940cd9b0 0x7fcf9415f240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcf84009be0 tx=0x7fcf8402a8d0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.544+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/4259119455 shutdown_connections 2024-06-29T15:39:19.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.544+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/4259119455 >> v2:172.21.15.153:3300/0 conn(0x7fcf9406da00 0x7fcf94073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.544+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/4259119455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf940cd9b0 0x7fcf9415f240 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.544+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/4259119455 >> 172.21.15.59:0/4259119455 conn(0x7fcf940b24c0 msgr2=0x7fcf940cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:19.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.545+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/4259119455 shutdown_connections 2024-06-29T15:39:19.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.545+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/4259119455 wait complete. 2024-06-29T15:39:19.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.546+0000 7fcf9b9e7640 1 Processor -- start 2024-06-29T15:39:19.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.546+0000 7fcf9b9e7640 1 -- start start 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf9b9e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf940cd9b0 0x7fcf941f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf9b9e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf941444c0 0x7fcf94138b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf9b9e7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf94143900 0x7fcf941f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf9b9e7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcf94073b40 con 0x7fcf940cd9b0 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf9b9e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf940739c0 con 0x7fcf941444c0 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf9b9e7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcf94073cc0 con 0x7fcf94143900 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf940cd9b0 0x7fcf941f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf940cd9b0 0x7fcf941f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53016/0 (socket says 172.21.15.59:53016) 2024-06-29T15:39:19.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 -- 172.21.15.59:0/199941134 learned_addr learned my addr 172.21.15.59:0/199941134 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99f84640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf94143900 0x7fcf941f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf98f82640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf941444c0 0x7fcf94138b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 -- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf94143900 msgr2=0x7fcf941f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf94143900 0x7fcf941f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 -- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf941444c0 msgr2=0x7fcf94138b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf941444c0 0x7fcf94138b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 -- 172.21.15.59:0/199941134 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf941f3280 con 0x7fcf940cd9b0 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf99783640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf940cd9b0 0x7fcf941f3ad0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fcf84009be0 tx=0x7fcf84033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.548+0000 7fcf91ffb640 1 -- 172.21.15.59:0/199941134 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf84031d80 con 0x7fcf940cd9b0 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.549+0000 7fcf91ffb640 1 -- 172.21.15.59:0/199941134 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf84003a60 con 0x7fcf940cd9b0 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.549+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf941f3560 con 0x7fcf940cd9b0 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.549+0000 7fcf91ffb640 1 -- 172.21.15.59:0/199941134 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840330d0 con 0x7fcf940cd9b0 2024-06-29T15:39:19.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.549+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf941369e0 con 0x7fcf940cd9b0 2024-06-29T15:39:19.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.550+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf94074170 con 0x7fcf940cd9b0 2024-06-29T15:39:19.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.551+0000 7fcf91ffb640 1 -- 172.21.15.59:0/199941134 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fcf84033270 con 0x7fcf940cd9b0 2024-06-29T15:39:19.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.551+0000 7fcf91ffb640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcf80076420 0x7fcf80078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.551+0000 7fcf91ffb640 1 -- 172.21.15.59:0/199941134 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7fcf840b8c30 con 0x7fcf940cd9b0 2024-06-29T15:39:19.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.551+0000 7fcf98f82640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcf80076420 0x7fcf80078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.552+0000 7fcf98f82640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcf80076420 0x7fcf80078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf88007fd0 tx=0x7fcf88009040 comp rx=0 tx=0).ready entity=mgr.9505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:19.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.555+0000 7fcf91ffb640 1 -- 172.21.15.59:0/199941134 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcf8403f030 con 0x7fcf940cd9b0 2024-06-29T15:39:19.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:19.795+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fcf9415d2c0 con 0x7fcf940cd9b0 2024-06-29T15:39:20.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.085+0000 7fcf91ffb640 1 -- 172.21.15.59:0/199941134 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v100) ==== 92+0+0 (secure 0 0 0) 0x7fcf84081af0 con 0x7fcf940cd9b0 2024-06-29T15:39:20.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcf80076420 msgr2=0x7fcf80078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcf80076420 0x7fcf80078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf88007fd0 tx=0x7fcf88009040 comp rx=0 tx=0).stop 2024-06-29T15:39:20.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf940cd9b0 msgr2=0x7fcf941f3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf940cd9b0 0x7fcf941f3ad0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fcf84009be0 tx=0x7fcf84033cb0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 shutdown_connections 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fcf80076420 0x7fcf80078930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf94143900 0x7fcf941f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf941444c0 0x7fcf94138b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 --2- 172.21.15.59:0/199941134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf940cd9b0 0x7fcf941f3ad0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 >> 172.21.15.59:0/199941134 conn(0x7fcf940b24c0 msgr2=0x7fcf94135df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.088+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 shutdown_connections 2024-06-29T15:39:20.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.089+0000 7fcf9b9e7640 1 -- 172.21.15.59:0/199941134 wait complete. 2024-06-29T15:39:20.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:39:20.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.229+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/814472577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 msgr2=0x7fb9f415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.229+0000 7fb9fa0fd640 1 --2- 172.21.15.59:0/814472577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 0x7fb9f415bc20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4009830 tx=0x7fb9e402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.229+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/814472577 shutdown_connections 2024-06-29T15:39:20.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.229+0000 7fb9fa0fd640 1 --2- 172.21.15.59:0/814472577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 0x7fb9f415bc20 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.229+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/814472577 >> 172.21.15.59:0/814472577 conn(0x7fb9f4073e10 msgr2=0x7fb9f4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.229+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/814472577 shutdown_connections 2024-06-29T15:39:20.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.229+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/814472577 wait complete. 2024-06-29T15:39:20.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.231+0000 7fb9fa0fd640 1 Processor -- start 2024-06-29T15:39:20.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.231+0000 7fb9fa0fd640 1 -- start start 2024-06-29T15:39:20.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9fa0fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 0x7fb9f40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9fa0fd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f40b2fa0 0x7fb9f4145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9fa0fd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9f40d1c30 0x7fb9f4146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9fa0fd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f40b65d0 con 0x7fb9f40d1c30 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9fa0fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f40b6450 con 0x7fb9f40b2270 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9fa0fd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f40b6750 con 0x7fb9f40b2fa0 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9f3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9f40d1c30 0x7fb9f4146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9f2ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f40b2fa0 0x7fb9f4145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9f37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 0x7fb9f40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9f3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9f40d1c30 0x7fb9f4146040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53032/0 (socket says 172.21.15.59:53032) 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.232+0000 7fb9f3fff640 1 -- 172.21.15.59:0/922284270 learned_addr learned my addr 172.21.15.59:0/922284270 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9f3fff640 1 -- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f40b2fa0 msgr2=0x7fb9f4145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9f3fff640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f40b2fa0 0x7fb9f4145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9f3fff640 1 -- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 msgr2=0x7fb9f40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9f3fff640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 0x7fb9f40ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9f3fff640 1 -- 172.21.15.59:0/922284270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f415eb50 con 0x7fb9f40d1c30 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9f37fe640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 0x7fb9f40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9f3fff640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9f40d1c30 0x7fb9f4146040 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec00c760 tx=0x7fb9ec00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9dbfff640 1 -- 172.21.15.59:0/922284270 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec003b70 con 0x7fb9f40d1c30 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9dbfff640 1 -- 172.21.15.59:0/922284270 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9ec003d10 con 0x7fb9f40d1c30 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f415ee30 con 0x7fb9f40d1c30 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9dbfff640 1 -- 172.21.15.59:0/922284270 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec00aab0 con 0x7fb9f40d1c30 2024-06-29T15:39:20.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.233+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f41455e0 con 0x7fb9f40d1c30 2024-06-29T15:39:20.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.234+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9f4145120 con 0x7fb9f40d1c30 2024-06-29T15:39:20.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.235+0000 7fb9dbfff640 1 -- 172.21.15.59:0/922284270 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 100) ==== 98320+0+0 (secure 0 0 0) 0x7fb9ec00ac50 con 0x7fb9f40d1c30 2024-06-29T15:39:20.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.236+0000 7fb9dbfff640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fb9e0076440 0x7fb9e0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.236+0000 7fb9f37fe640 1 -- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fb9e0076440 msgr2=0x7fb9e0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1270545814 2024-06-29T15:39:20.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.236+0000 7fb9dbfff640 1 -- 172.21.15.59:0/922284270 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7fb9ec0977f0 con 0x7fb9f40d1c30 2024-06-29T15:39:20.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.236+0000 7fb9f37fe640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fb9e0076440 0x7fb9e0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:39:20.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.240+0000 7fb9dbfff640 1 -- 172.21.15.59:0/922284270 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb9f4145120 con 0x7fb9f40d1c30 2024-06-29T15:39:20.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.434+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fb9f40b6c70 con 0x7fb9f40d1c30 2024-06-29T15:39:20.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.437+0000 7fb9f37fe640 1 -- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fb9e0076440 msgr2=0x7fb9e0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1270545814 2024-06-29T15:39:20.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.437+0000 7fb9f37fe640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fb9e0076440 0x7fb9e0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:39:20.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.439+0000 7fb9dbfff640 1 -- 172.21.15.59:0/922284270 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) ==== 138+0+0 (secure 0 0 0) 0x7fb9ec067e20 con 0x7fb9f40d1c30 2024-06-29T15:39:20.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fb9e0076440 msgr2=0x7fb9e0078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:39:20.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fb9e0076440 0x7fb9e0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9f40d1c30 msgr2=0x7fb9f4146040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9f40d1c30 0x7fb9f4146040 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec00c760 tx=0x7fb9ec00cc20 comp rx=0 tx=0).stop 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 shutdown_connections 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7fb9e0076440 0x7fb9e0078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9f40d1c30 0x7fb9f4146040 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f40b2fa0 0x7fb9f4145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 --2- 172.21.15.59:0/922284270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40b2270 0x7fb9f40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 >> 172.21.15.59:0/922284270 conn(0x7fb9f4073e10 msgr2=0x7fb9f4070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 shutdown_connections 2024-06-29T15:39:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.441+0000 7fb9fa0fd640 1 -- 172.21.15.59:0/922284270 wait complete. 2024-06-29T15:39:20.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:39:20.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 -- 172.21.15.59:0/706006940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c0b2040 msgr2=0x7f8c7c06e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 --2- 172.21.15.59:0/706006940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c0b2040 0x7f8c7c06e300 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c06ca20 tx=0x7f8c6402a560 comp rx=0 tx=0).stop 2024-06-29T15:39:20.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 -- 172.21.15.59:0/706006940 shutdown_connections 2024-06-29T15:39:20.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 --2- 172.21.15.59:0/706006940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c7c06e840 0x7f8c7c15e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 --2- 172.21.15.59:0/706006940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c0b2040 0x7f8c7c06e300 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 --2- 172.21.15.59:0/706006940 >> v2:172.21.15.153:3300/0 conn(0x7f8c7c144470 0x7f8c7c0b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 -- 172.21.15.59:0/706006940 >> 172.21.15.59:0/706006940 conn(0x7f8c7c0b06e0 msgr2=0x7f8c7c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 -- 172.21.15.59:0/706006940 shutdown_connections 2024-06-29T15:39:20.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.575+0000 7f8c818f6640 1 -- 172.21.15.59:0/706006940 wait complete. 2024-06-29T15:39:20.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.577+0000 7f8c818f6640 1 Processor -- start 2024-06-29T15:39:20.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.577+0000 7f8c818f6640 1 -- start start 2024-06-29T15:39:20.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c818f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c7c06e840 0x7f8c7c13f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c818f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7c0b2040 0x7f8c7c13faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c818f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c144470 0x7f8c7c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c818f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c15ee30 con 0x7f8c7c144470 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c818f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c15ecb0 con 0x7f8c7c0b2040 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c818f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7c15efb0 con 0x7f8c7c06e840 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c144470 0x7f8c7c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c7b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c144470 0x7f8c7c06d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53050/0 (socket says 172.21.15.59:53050) 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c7b7fe640 1 -- 172.21.15.59:0/3848623531 learned_addr learned my addr 172.21.15.59:0/3848623531 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c7a7fc640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7c0b2040 0x7f8c7c13faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.578+0000 7f8c7affd640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c7c06e840 0x7f8c7c13f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c7b7fe640 1 -- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c7c06e840 msgr2=0x7f8c7c13f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c7b7fe640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c7c06e840 0x7f8c7c13f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c7b7fe640 1 -- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7c0b2040 msgr2=0x7f8c7c13faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c7b7fe640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7c0b2040 0x7f8c7c13faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c7b7fe640 1 -- 172.21.15.59:0/3848623531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7c15c570 con 0x7f8c7c144470 2024-06-29T15:39:20.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c7b7fe640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c144470 0x7f8c7c06d210 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d980 tx=0x7f8c7400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:20.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c73fff640 1 -- 172.21.15.59:0/3848623531 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74016070 con 0x7f8c7c144470 2024-06-29T15:39:20.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c73fff640 1 -- 172.21.15.59:0/3848623531 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c7400bd40 con 0x7f8c7c144470 2024-06-29T15:39:20.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7c15c850 con 0x7f8c7c144470 2024-06-29T15:39:20.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c73fff640 1 -- 172.21.15.59:0/3848623531 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74003bd0 con 0x7f8c7c144470 2024-06-29T15:39:20.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.579+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7c15ce10 con 0x7f8c7c144470 2024-06-29T15:39:20.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.580+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7c0cd0c0 con 0x7f8c7c144470 2024-06-29T15:39:20.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.581+0000 7f8c73fff640 1 -- 172.21.15.59:0/3848623531 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 100) ==== 98320+0+0 (secure 0 0 0) 0x7f8c74010460 con 0x7f8c7c144470 2024-06-29T15:39:20.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.581+0000 7f8c73fff640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f8c5c076440 0x7f8c5c078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.582+0000 7f8c7affd640 1 -- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f8c5c076440 msgr2=0x7f8c5c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1270545814 2024-06-29T15:39:20.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.582+0000 7f8c73fff640 1 -- 172.21.15.59:0/3848623531 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7f8c74098130 con 0x7f8c7c144470 2024-06-29T15:39:20.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.582+0000 7f8c7affd640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f8c5c076440 0x7f8c5c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:39:20.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.586+0000 7f8c73fff640 1 -- 172.21.15.59:0/3848623531 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c7c0cd0c0 con 0x7f8c7c144470 2024-06-29T15:39:20.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.783+0000 7f8c7affd640 1 -- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f8c5c076440 msgr2=0x7f8c5c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1270545814 2024-06-29T15:39:20.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.783+0000 7f8c7affd640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f8c5c076440 0x7f8c5c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:39:20.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.821+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f8c7c15f470 con 0x7f8c7c144470 2024-06-29T15:39:20.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.825+0000 7f8c73fff640 1 -- 172.21.15.59:0/3848623531 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) ==== 133+0+0 (secure 0 0 0) 0x7f8c74060fd0 con 0x7f8c7c144470 2024-06-29T15:39:20.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f8c5c076440 msgr2=0x7f8c5c078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:39:20.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f8c5c076440 0x7f8c5c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c144470 msgr2=0x7f8c7c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c144470 0x7f8c7c06d210 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d980 tx=0x7f8c7400de40 comp rx=0 tx=0).stop 2024-06-29T15:39:20.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 shutdown_connections 2024-06-29T15:39:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f8c5c076440 0x7f8c5c078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7c144470 0x7f8c7c06d210 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7c0b2040 0x7f8c7c13faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 --2- 172.21.15.59:0/3848623531 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c7c06e840 0x7f8c7c13f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 >> 172.21.15.59:0/3848623531 conn(0x7f8c7c0b06e0 msgr2=0x7f8c7c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 shutdown_connections 2024-06-29T15:39:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:20.827+0000 7f8c818f6640 1 -- 172.21.15.59:0/3848623531 wait complete. 2024-06-29T15:39:20.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:20.840 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T15:39:20.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:20.860 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-06-29T15:39:20.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-06-29T15:39:21.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.081+0000 7f5be9036640 1 -- 172.21.15.59:0/3946919953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bcc004230 msgr2=0x7f5bcc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:21.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.081+0000 7f5be9036640 1 --2- 172.21.15.59:0/3946919953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bcc004230 0x7f5bcc004800 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f5be40703d0 tx=0x7f5bcc005830 comp rx=0 tx=0).stop 2024-06-29T15:39:21.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.081+0000 7f5be359e640 1 -- 172.21.15.59:0/3946919953 reap_dead start 2024-06-29T15:39:21.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.081+0000 7f5be9036640 1 -- 172.21.15.59:0/3946919953 shutdown_connections 2024-06-29T15:39:21.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.081+0000 7f5be9036640 1 -- 172.21.15.59:0/3946919953 >> 172.21.15.59:0/3946919953 conn(0x7f5be40b1c50 msgr2=0x7f5be40b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:21.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.081+0000 7f5be9036640 1 -- 172.21.15.59:0/3946919953 shutdown_connections 2024-06-29T15:39:21.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.081+0000 7f5be9036640 1 -- 172.21.15.59:0/3946919953 wait complete. 2024-06-29T15:39:21.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.083+0000 7f5be9036640 1 Processor -- start 2024-06-29T15:39:21.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.083+0000 7f5be9036640 1 -- start start 2024-06-29T15:39:21.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.085+0000 7f5be9036640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bcc004230 0x7f5be414e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:21.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.085+0000 7f5be9036640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be41f11f0 0x7f5be409cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.085+0000 7f5be9036640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5be4157780 0x7f5be40b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.085+0000 7f5be9036640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5be40b5410 con 0x7f5bcc004230 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.085+0000 7f5be9036640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5be40b5290 con 0x7f5be41f11f0 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.085+0000 7f5be9036640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5be40b3820 con 0x7f5be4157780 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.085+0000 7f5be259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be41f11f0 0x7f5be409cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.085+0000 7f5be2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bcc004230 0x7f5be414e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be41f11f0 0x7f5be409cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:44204/0 (socket says 172.21.15.59:44204) 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be259c640 1 -- 172.21.15.59:0/1449305034 learned_addr learned my addr 172.21.15.59:0/1449305034 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be359e640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5be4157780 0x7f5be40b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be259c640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5be4157780 msgr2=0x7f5be40b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be259c640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5be4157780 0x7f5be40b0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be259c640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bcc004230 msgr2=0x7f5be414e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:21.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be259c640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bcc004230 0x7f5be414e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:21.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be259c640 1 -- 172.21.15.59:0/1449305034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be4152170 con 0x7f5be41f11f0 2024-06-29T15:39:21.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be359e640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5be4157780 0x7f5be40b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:39:21.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.086+0000 7f5be259c640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be41f11f0 0x7f5be409cb90 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80018c0 tx=0x7f5bd8006190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:21.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.089+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8003a00 con 0x7f5be41f11f0 2024-06-29T15:39:21.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.089+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bd8003c90 con 0x7f5be41f11f0 2024-06-29T15:39:21.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.089+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be4152450 con 0x7f5be41f11f0 2024-06-29T15:39:21.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.089+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8003e60 con 0x7f5be41f11f0 2024-06-29T15:39:21.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.089+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be409d0d0 con 0x7f5be41f11f0 2024-06-29T15:39:21.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.090+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5be4151ee0 con 0x7f5be41f11f0 2024-06-29T15:39:21.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.091+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 100) ==== 98320+0+0 (secure 0 0 0) 0x7f5bd8045020 con 0x7f5be41f11f0 2024-06-29T15:39:21.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.092+0000 7f5bcb7fe640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 0x7f5bd0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:21.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.092+0000 7f5be2d9d640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 msgr2=0x7f5bd0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1270545814 2024-06-29T15:39:21.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.092+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6129+0+0 (secure 0 0 0) 0x7f5bd80b7c00 con 0x7f5be41f11f0 2024-06-29T15:39:21.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.092+0000 7f5be2d9d640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 0x7f5bd0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:39:21.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.096+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5bd80811b0 con 0x7f5be41f11f0 2024-06-29T15:39:21.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.293+0000 7f5be2d9d640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 msgr2=0x7f5bd0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1270545814 2024-06-29T15:39:21.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.293+0000 7f5be2d9d640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 0x7f5bd0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:39:21.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.312+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 --> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5be415bd60 con 0x7f5bd0076440 2024-06-29T15:39:21.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.694+0000 7f5be2d9d640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 msgr2=0x7f5bd0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1270545814 2024-06-29T15:39:21.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:21.694+0000 7f5be2d9d640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 0x7f5bd0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:39:22.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:22.495+0000 7f5be2d9d640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 msgr2=0x7f5bd0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1270545814 2024-06-29T15:39:22.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:22.495+0000 7f5be2d9d640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 0x7f5bd0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:39:23.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:23.702+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 101) ==== 97929+0+0 (secure 0 0 0) 0x7f5bd807cd40 con 0x7f5be41f11f0 2024-06-29T15:39:23.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:23.703+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 msgr2=0x7f5bd0078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:39:23.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:23.703+0000 7f5bcb7fe640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1270545814,v1:172.21.15.153:6833/1270545814] conn(0x7f5bd0076440 0x7f5bd0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:23.865 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.864+0000 7f0518ff7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.865 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.864+0000 7f0518ff7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.864+0000 7f0518ff7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.864+0000 7f0518ff7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.864+0000 7f0518ff7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.865+0000 7f051bffd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.865+0000 7f051bffd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.865+0000 7f051bffd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.865+0000 7f051bffd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:39:23.865+0000 7f051bffd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:24.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.707+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mgrmap(e 102) ==== 98056+0+0 (secure 0 0 0) 0x7f5bd809a330 con 0x7f5be41f11f0 2024-06-29T15:39:24.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.708+0000 7f5bcb7fe640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f5bd003b070 0x7f5bd0080a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:24.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.708+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 --> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5bd8049eb0 con 0x7f5bd003b070 2024-06-29T15:39:24.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.708+0000 7f5be2d9d640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f5bd003b070 0x7f5bd0080a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:24.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.709+0000 7f5be2d9d640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f5bd003b070 0x7f5bd0080a00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4005eb0 tx=0x7f5bd4005f90 comp rx=0 tx=0).ready entity=mgr.8711 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.787+0000 7f5bcb7fe640 1 -- 172.21.15.59:0/1449305034 <== mgr.8711 v2:172.21.15.153:6832/1682798961 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f5bd8049eb0 con 0x7f5bd003b070 2024-06-29T15:39:24.789 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f5bd003b070 msgr2=0x7f5bd0080a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be9036640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f5bd003b070 0x7f5bd0080a00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4005eb0 tx=0x7f5bd4005f90 comp rx=0 tx=0).stop 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be41f11f0 msgr2=0x7f5be409cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be9036640 1 --2- 172.21.15.59:0/1449305034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5be41f11f0 0x7f5be409cb90 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80018c0 tx=0x7f5bd8006190 comp rx=0 tx=0).stop 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be359e640 1 -- 172.21.15.59:0/1449305034 reap_dead start 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 shutdown_connections 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 >> 172.21.15.59:0/1449305034 conn(0x7f5be40b1c50 msgr2=0x7f5be415b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 shutdown_connections 2024-06-29T15:39:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.790+0000 7f5be9036640 1 -- 172.21.15.59:0/1449305034 wait complete. 2024-06-29T15:39:24.803 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-06-29T15:39:24.803 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 -- 172.21.15.59:0/344975750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3540d23c0 msgr2=0x7fa3540b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 --2- 172.21.15.59:0/344975750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3540d23c0 0x7fa3540b3990 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fa35408c4f0 tx=0x7fa34402a560 comp rx=0 tx=0).stop 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 -- 172.21.15.59:0/344975750 shutdown_connections 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 --2- 172.21.15.59:0/344975750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa35415ee10 0x7fa35415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 --2- 172.21.15.59:0/344975750 >> v2:172.21.15.153:3300/0 conn(0x7fa3540b3ed0 0x7fa35415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 --2- 172.21.15.59:0/344975750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3540d23c0 0x7fa3540b3990 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 -- 172.21.15.59:0/344975750 >> 172.21.15.59:0/344975750 conn(0x7fa3540d3ee0 msgr2=0x7fa3540d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 -- 172.21.15.59:0/344975750 shutdown_connections 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.924+0000 7fa358d12640 1 -- 172.21.15.59:0/344975750 wait complete. 2024-06-29T15:39:24.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.926+0000 7fa358d12640 1 Processor -- start 2024-06-29T15:39:24.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.927+0000 7fa358d12640 1 -- start start 2024-06-29T15:39:24.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa358d12640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3540b3ed0 0x7fa35415e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:24.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa358d12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3540d23c0 0x7fa3541f5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa358d12640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35415ee10 0x7fa3540b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa358d12640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa354160070 con 0x7fa35415ee10 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa358d12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa35415fef0 con 0x7fa3540d23c0 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa358d12640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa3541601f0 con 0x7fa3540b3ed0 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa352d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35415ee10 0x7fa3540b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa352d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35415ee10 0x7fa3540b8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47972/0 (socket says 172.21.15.59:47972) 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa352d9d640 1 -- 172.21.15.59:0/369732298 learned_addr learned my addr 172.21.15.59:0/369732298 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa349d9b640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3540d23c0 0x7fa3541f5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa35259c640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3540b3ed0 0x7fa35415e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa352d9d640 1 -- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3540b3ed0 msgr2=0x7fa35415e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa352d9d640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3540b3ed0 0x7fa35415e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa352d9d640 1 -- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3540d23c0 msgr2=0x7fa3541f5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa352d9d640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3540d23c0 0x7fa3541f5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:24.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.928+0000 7fa352d9d640 1 -- 172.21.15.59:0/369732298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35415b270 con 0x7fa35415ee10 2024-06-29T15:39:24.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.929+0000 7fa352d9d640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35415ee10 0x7fa3540b8c20 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00ea70 tx=0x7fa34c00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:24.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.929+0000 7fa34b7fe640 1 -- 172.21.15.59:0/369732298 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c00ce40 con 0x7fa35415ee10 2024-06-29T15:39:24.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.929+0000 7fa34b7fe640 1 -- 172.21.15.59:0/369732298 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa34c004c10 con 0x7fa35415ee10 2024-06-29T15:39:24.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.929+0000 7fa34b7fe640 1 -- 172.21.15.59:0/369732298 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c005020 con 0x7fa35415ee10 2024-06-29T15:39:24.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.929+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3541f69a0 con 0x7fa35415ee10 2024-06-29T15:39:24.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.929+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3541f6f60 con 0x7fa35415ee10 2024-06-29T15:39:24.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.930+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3540b3990 con 0x7fa35415ee10 2024-06-29T15:39:24.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.931+0000 7fa34b7fe640 1 -- 172.21.15.59:0/369732298 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 102) ==== 98056+0+0 (secure 0 0 0) 0x7fa34c0051c0 con 0x7fa35415ee10 2024-06-29T15:39:24.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.931+0000 7fa34b7fe640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7fa3340761a0 0x7fa3340786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:24.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.932+0000 7fa34b7fe640 1 -- 172.21.15.59:0/369732298 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 6473+0+0 (secure 0 0 0) 0x7fa34c098390 con 0x7fa35415ee10 2024-06-29T15:39:24.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.932+0000 7fa35259c640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7fa3340761a0 0x7fa3340786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:24.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.932+0000 7fa35259c640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7fa3340761a0 0x7fa3340786b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa344033000 tx=0x7fa344004730 comp rx=0 tx=0).ready entity=mgr.8711 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:24.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:24.936+0000 7fa34b7fe640 1 -- 172.21.15.59:0/369732298 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa34c061d30 con 0x7fa35415ee10 2024-06-29T15:39:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.151+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 --> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fa3541f6730 con 0x7fa3340761a0 2024-06-29T15:39:25.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.205+0000 7fa34b7fe640 1 -- 172.21.15.59:0/369732298 <== mgr.8711 v2:172.21.15.153:6832/1682798961 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+57 (secure 0 0 0) 0x7fa3541f6730 con 0x7fa3340761a0 2024-06-29T15:39:25.206 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-06-29T15:39:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7fa3340761a0 msgr2=0x7fa3340786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7fa3340761a0 0x7fa3340786b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa344033000 tx=0x7fa344004730 comp rx=0 tx=0).stop 2024-06-29T15:39:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35415ee10 msgr2=0x7fa3540b8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35415ee10 0x7fa3540b8c20 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00ea70 tx=0x7fa34c00ef30 comp rx=0 tx=0).stop 2024-06-29T15:39:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 shutdown_connections 2024-06-29T15:39:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa35415ee10 0x7fa3540b8c20 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3540d23c0 0x7fa3541f5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7fa3340761a0 0x7fa3340786b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 --2- 172.21.15.59:0/369732298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3540b3ed0 0x7fa35415e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 >> 172.21.15.59:0/369732298 conn(0x7fa3540d3ee0 msgr2=0x7fa354160730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:25.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 shutdown_connections 2024-06-29T15:39:25.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:39:25.207+0000 7fa358d12640 1 -- 172.21.15.59:0/369732298 wait complete. 2024-06-29T15:39:25.219 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-06-29T15:39:39.293 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:39:39.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:40:00.218 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:40:00.217+0000 7f050afdb640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:00.218 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:40:00.217+0000 7f050afdb640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:00.218 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:40:00.217+0000 7f050afdb640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:00.218 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:40:00.217+0000 7f050afdb640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:00.218 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:40:00.217+0000 7f050afdb640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:09.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:40:09.402 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:40:39.529 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:40:39.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:41:09.599 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:41:09.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:41:39.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:41:39.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:42:09.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:42:09.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:42:40.093 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:42:40.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:43:10.156 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:43:10.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:43:40.237 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:43:40.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:44:10.310 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:44:10.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:44:40.384 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:44:40.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:45:10.470 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:45:10.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:45:40.534 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:45:40.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:46:10.597 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:46:10.600 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:46:40.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:46:40.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:47:10.733 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:47:10.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:47:40.813 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:47:40.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:48:11.153 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:48:11.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:48:41.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:48:41.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:49:11.380 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:49:11.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:49:41.503 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:49:41.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:50:11.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:50:11.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:50:16.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:50:16.221 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-06-29T15:50:16.242 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_40_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_41_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_42_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_43_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_44_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_45_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_46_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_47_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_48_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_49_00_UTC 2024-06-29T15:50:16.243 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_50_00_UTC 2024-06-29T15:50:16.244 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-15_40_00_UTC', 'scheduled-2024-06-29-15_41_00_UTC', 'scheduled-2024-06-29-15_42_00_UTC', 'scheduled-2024-06-29-15_43_00_UTC', 'scheduled-2024-06-29-15_44_00_UTC', 'scheduled-2024-06-29-15_45_00_UTC', 'scheduled-2024-06-29-15_46_00_UTC', 'scheduled-2024-06-29-15_47_00_UTC', 'scheduled-2024-06-29-15_48_00_UTC', 'scheduled-2024-06-29-15_49_00_UTC', 'scheduled-2024-06-29-15_50_00_UTC'] 2024-06-29T15:50:16.244 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-06-29T15:50:16.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.415+0000 7f2678fb2640 1 -- 172.21.15.59:0/718514909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 msgr2=0x7f267415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.415+0000 7f2678fb2640 1 --2- 172.21.15.59:0/718514909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 0x7f267415f9b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f26740d15d0 tx=0x7f266c02aa30 comp rx=0 tx=0).stop 2024-06-29T15:50:16.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.416+0000 7f2678fb2640 1 -- 172.21.15.59:0/718514909 shutdown_connections 2024-06-29T15:50:16.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.416+0000 7f2678fb2640 1 --2- 172.21.15.59:0/718514909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 0x7f267415f9b0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.416+0000 7f2678fb2640 1 --2- 172.21.15.59:0/718514909 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26740b3ed0 0x7f267415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.416+0000 7f2678fb2640 1 --2- 172.21.15.59:0/718514909 >> v2:172.21.15.153:3300/0 conn(0x7f26740d23c0 0x7f26740b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.416+0000 7f2678fb2640 1 -- 172.21.15.59:0/718514909 >> 172.21.15.59:0/718514909 conn(0x7f26740d3ee0 msgr2=0x7f26740d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:50:16.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.416+0000 7f2678fb2640 1 -- 172.21.15.59:0/718514909 shutdown_connections 2024-06-29T15:50:16.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.416+0000 7f2678fb2640 1 -- 172.21.15.59:0/718514909 wait complete. 2024-06-29T15:50:16.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.418+0000 7f2678fb2640 1 Processor -- start 2024-06-29T15:50:16.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.418+0000 7f2678fb2640 1 -- start start 2024-06-29T15:50:16.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2678fb2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26740b3ed0 0x7f2674159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2678fb2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26740d23c0 0x7f2674152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2678fb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 0x7f26741e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2678fb2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2674160070 con 0x7f267415ee10 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2678fb2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f267415fef0 con 0x7f26740b3ed0 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2678fb2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f26741601f0 con 0x7f26740d23c0 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2672d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 0x7f26741e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2672d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 0x7f26741e8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52718/0 (socket says 172.21.15.59:52718) 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.419+0000 7f2672d9d640 1 -- 172.21.15.59:0/1392954188 learned_addr learned my addr 172.21.15.59:0/1392954188 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f266bfff640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26740d23c0 0x7f2674152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f267259c640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26740b3ed0 0x7f2674159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f2672d9d640 1 -- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26740d23c0 msgr2=0x7f2674152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f2672d9d640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26740d23c0 0x7f2674152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f2672d9d640 1 -- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26740b3ed0 msgr2=0x7f2674159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f2672d9d640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26740b3ed0 0x7f2674159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f2672d9d640 1 -- 172.21.15.59:0/1392954188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26741e79c0 con 0x7f267415ee10 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f266bfff640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26740d23c0 0x7f2674152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f2672d9d640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 0x7f26741e8a40 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f266c02b020 tx=0x7f266c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f266affd640 1 -- 172.21.15.59:0/1392954188 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266c033050 con 0x7f267415ee10 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f266affd640 1 -- 172.21.15.59:0/1392954188 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f266c004450 con 0x7f267415ee10 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26741e7c80 con 0x7f267415ee10 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f266affd640 1 -- 172.21.15.59:0/1392954188 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266c038450 con 0x7f267415ee10 2024-06-29T15:50:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.420+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26741e8240 con 0x7f267415ee10 2024-06-29T15:50:16.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.421+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26741f9730 con 0x7f267415ee10 2024-06-29T15:50:16.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.422+0000 7f266affd640 1 -- 172.21.15.59:0/1392954188 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 106) ==== 98332+0+0 (secure 0 0 0) 0x7f266c036070 con 0x7f267415ee10 2024-06-29T15:50:16.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.422+0000 7f266affd640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f2654076470 0x7f2654078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.423+0000 7f266affd640 1 -- 172.21.15.59:0/1392954188 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 6473+0+0 (secure 0 0 0) 0x7f266c0b9240 con 0x7f267415ee10 2024-06-29T15:50:16.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.423+0000 7f267259c640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f2654076470 0x7f2654078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.423+0000 7f267259c640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f2654076470 0x7f2654078980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f26740d2e40 tx=0x7f2664005f90 comp rx=0 tx=0).ready entity=mgr.8711 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:50:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.427+0000 7f266affd640 1 -- 172.21.15.59:0/1392954188 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f266c0825f0 con 0x7f267415ee10 2024-06-29T15:50:16.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.638+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 --> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f26740b3990 con 0x7f2654076470 2024-06-29T15:50:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.646+0000 7f266affd640 1 -- 172.21.15.59:0/1392954188 <== mgr.8711 v2:172.21.15.153:6832/1682798961 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+375 (secure 0 0 0) 0x7f26740b3990 con 0x7f2654076470 2024-06-29T15:50:16.647 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:50:16.647 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:39:24", "first": "2024-06-29T15:40:00", "last": "2024-06-29T15:50:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-06-29T15:50:16.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f2654076470 msgr2=0x7f2654078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f2654076470 0x7f2654078980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f26740d2e40 tx=0x7f2664005f90 comp rx=0 tx=0).stop 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 msgr2=0x7f26741e8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 0x7f26741e8a40 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f266c02b020 tx=0x7f266c003f90 comp rx=0 tx=0).stop 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 shutdown_connections 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267415ee10 0x7f26741e8a40 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f2654076470 0x7f2654078980 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26740d23c0 0x7f2674152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 --2- 172.21.15.59:0/1392954188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26740b3ed0 0x7f2674159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 >> 172.21.15.59:0/1392954188 conn(0x7f26740d3ee0 msgr2=0x7f267406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 shutdown_connections 2024-06-29T15:50:16.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.648+0000 7f2678fb2640 1 -- 172.21.15.59:0/1392954188 wait complete. 2024-06-29T15:50:16.660 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:39:24", "first": "2024-06-29T15:40:00", "last": "2024-06-29T15:50:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-06-29T15:50:16.660 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T15:39:24', 'first': '2024-06-29T15:40:00', 'last': '2024-06-29T15:50:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-06-29T15:50:16.660 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T15:50:16.661 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T15:50:16.661 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-06-29T15:50:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.778+0000 7f1a54c3d640 1 -- 172.21.15.59:0/2566251936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a501414c0 msgr2=0x7f1a501418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.778+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/2566251936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a501414c0 0x7f1a501418d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009830 tx=0x7f1a4802a480 comp rx=0 tx=0).stop 2024-06-29T15:50:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.778+0000 7f1a54c3d640 1 -- 172.21.15.59:0/2566251936 shutdown_connections 2024-06-29T15:50:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.778+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/2566251936 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a501414c0 0x7f1a501418d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.778+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/2566251936 >> v2:172.21.15.153:3300/0 conn(0x7f1a5013e9b0 0x7f1a5013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.778+0000 7f1a54c3d640 1 -- 172.21.15.59:0/2566251936 >> 172.21.15.59:0/2566251936 conn(0x7f1a500b6fb0 msgr2=0x7f1a500b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:50:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.778+0000 7f1a54c3d640 1 -- 172.21.15.59:0/2566251936 shutdown_connections 2024-06-29T15:50:16.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.778+0000 7f1a54c3d640 1 -- 172.21.15.59:0/2566251936 wait complete. 2024-06-29T15:50:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.780+0000 7f1a54c3d640 1 Processor -- start 2024-06-29T15:50:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.781+0000 7f1a54c3d640 1 -- start start 2024-06-29T15:50:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a54c3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a5013e9b0 0x7f1a501f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a54c3d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a501414c0 0x7f1a5015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a54c3d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5015ceb0 0x7f1a5015b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a54c3d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1a500b2a00 con 0x7f1a5013e9b0 2024-06-29T15:50:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a54c3d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a500b2880 con 0x7f1a501414c0 2024-06-29T15:50:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a54c3d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1a500b2b80 con 0x7f1a5015ceb0 2024-06-29T15:50:16.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a5013e9b0 0x7f1a501f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a5013e9b0 0x7f1a501f7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52732/0 (socket says 172.21.15.59:52732) 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4e59c640 1 -- 172.21.15.59:0/3620846750 learned_addr learned my addr 172.21.15.59:0/3620846750 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4ed9d640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5015ceb0 0x7f1a5015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4dd9b640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a501414c0 0x7f1a5015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4e59c640 1 -- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5015ceb0 msgr2=0x7f1a5015b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4e59c640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5015ceb0 0x7f1a5015b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4e59c640 1 -- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a501414c0 msgr2=0x7f1a5015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4e59c640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a501414c0 0x7f1a5015ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.782+0000 7f1a4e59c640 1 -- 172.21.15.59:0/3620846750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a501f89c0 con 0x7f1a5013e9b0 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.783+0000 7f1a4e59c640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a5013e9b0 0x7f1a501f7740 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000cb50 tx=0x7f1a4000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.783+0000 7f1a46ffd640 1 -- 172.21.15.59:0/3620846750 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40016020 con 0x7f1a5013e9b0 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.783+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a50138240 con 0x7f1a5013e9b0 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.783+0000 7f1a46ffd640 1 -- 172.21.15.59:0/3620846750 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a4000ae30 con 0x7f1a5013e9b0 2024-06-29T15:50:16.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.783+0000 7f1a46ffd640 1 -- 172.21.15.59:0/3620846750 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40012660 con 0x7f1a5013e9b0 2024-06-29T15:50:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.783+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a50138800 con 0x7f1a5013e9b0 2024-06-29T15:50:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.784+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a500b7700 con 0x7f1a5013e9b0 2024-06-29T15:50:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.784+0000 7f1a46ffd640 1 -- 172.21.15.59:0/3620846750 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 106) ==== 98332+0+0 (secure 0 0 0) 0x7f1a40019020 con 0x7f1a5013e9b0 2024-06-29T15:50:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.785+0000 7f1a46ffd640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f1a30076440 0x7f1a30078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.785+0000 7f1a46ffd640 1 -- 172.21.15.59:0/3620846750 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 6473+0+0 (secure 0 0 0) 0x7f1a400982c0 con 0x7f1a5013e9b0 2024-06-29T15:50:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.785+0000 7f1a4dd9b640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f1a30076440 0x7f1a30078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.786+0000 7f1a4dd9b640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f1a30076440 0x7f1a30078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1a50156220 tx=0x7f1a38005f90 comp rx=0 tx=0).ready entity=mgr.8711 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:50:16.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:16.790+0000 7f1a46ffd640 1 -- 172.21.15.59:0/3620846750 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a40020020 con 0x7f1a5013e9b0 2024-06-29T15:50:17.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.029+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f1a500b3030 con 0x7f1a5013e9b0 2024-06-29T15:50:17.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.032+0000 7f1a46ffd640 1 -- 172.21.15.59:0/3620846750 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v106) ==== 74+0+189739 (secure 0 0 0) 0x7f1a50141e10 con 0x7f1a5013e9b0 2024-06-29T15:50:17.036 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:50:17.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f1a30076440 msgr2=0x7f1a30078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f1a30076440 0x7f1a30078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1a50156220 tx=0x7f1a38005f90 comp rx=0 tx=0).stop 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a5013e9b0 msgr2=0x7f1a501f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a5013e9b0 0x7f1a501f7740 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000cb50 tx=0x7f1a4000a430 comp rx=0 tx=0).stop 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 shutdown_connections 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:6832/1682798961,v1:172.21.15.153:6833/1682798961] conn(0x7f1a30076440 0x7f1a30078950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a5015ceb0 0x7f1a5015b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a501414c0 0x7f1a5015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 --2- 172.21.15.59:0/3620846750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a5013e9b0 0x7f1a501f7740 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 >> 172.21.15.59:0/3620846750 conn(0x7f1a500b6fb0 msgr2=0x7f1a50140210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 shutdown_connections 2024-06-29T15:50:17.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:50:17.035+0000 7f1a54c3d640 1 -- 172.21.15.59:0/3620846750 wait complete. 2024-06-29T15:50:17.047 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"flags":0,"active_gid":8711,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":1682798961},{"type":"v1","addr":"172.21.15.153:6833","nonce":1682798961}]},"active_addr":"172.21.15.153:6833/1682798961","active_change":"2024-06-29T15:39:23.696449+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":10581,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":190,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3711582444}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3023327174}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1052117807}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1412243249}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1918645711}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3420796917}]}]} 2024-06-29T15:50:17.051 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2024-06-29T15:50:17.052 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-06-29T15:50:17.052 INFO:tasks.ceph.mgr.x:Stopping old one... 2024-06-29T15:50:17.052 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-06-29T15:50:17.052 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:50:17.092 INFO:tasks.ceph.mgr.x:Stopped 2024-06-29T15:50:17.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-06-29T15:50:17.135 INFO:tasks.ceph.mgr.x:Started 2024-06-29T15:50:20.909 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f306e6ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.909 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f306e6ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.910 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f306e6ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.910 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f306e6ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.910 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f306e6ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.910 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f30716d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.910 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f30716d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.910 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f30716d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.910 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f30716d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:20.911 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:50:20.908+0000 7f30716d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:41.740 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:50:41.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:51:00.908 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:51:00.907+0000 7f30796e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:00.908 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:51:00.908+0000 7f30796e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:00.909 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:51:00.908+0000 7f30796e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:00.909 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:51:00.908+0000 7f30796e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:00.909 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:51:00.908+0000 7f30796e4640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:11.877 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:51:11.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:51:41.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:51:41.937 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:52:12.046 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:52:12.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:52:42.186 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:52:42.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:53:12.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:53:12.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:53:42.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:53:42.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:54:12.509 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:54:12.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:54:42.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:54:42.652 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:12.713 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:12.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:17.136 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-06-29T15:55:17.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.273+0000 7fb5ae422640 1 -- 172.21.15.59:0/1350140037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb590004800 msgr2=0x7fb590004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.273+0000 7fb5ae422640 1 --2- 172.21.15.59:0/1350140037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb590004800 0x7fb590004c10 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009b50 tx=0x7fb590005830 comp rx=0 tx=0).stop 2024-06-29T15:55:17.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.273+0000 7fb5ae422640 1 -- 172.21.15.59:0/1350140037 shutdown_connections 2024-06-29T15:55:17.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.273+0000 7fb5ae422640 1 --2- 172.21.15.59:0/1350140037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb590004800 0x7fb590004c10 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.273+0000 7fb5ae422640 1 -- 172.21.15.59:0/1350140037 >> 172.21.15.59:0/1350140037 conn(0x7fb5a80b24c0 msgr2=0x7fb5a80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:17.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.273+0000 7fb5ae422640 1 -- 172.21.15.59:0/1350140037 shutdown_connections 2024-06-29T15:55:17.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.273+0000 7fb5ae422640 1 -- 172.21.15.59:0/1350140037 wait complete. 2024-06-29T15:55:17.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.275+0000 7fb5ae422640 1 Processor -- start 2024-06-29T15:55:17.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.275+0000 7fb5ae422640 1 -- start start 2024-06-29T15:55:17.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ae422640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb590004800 0x7fb5a81f68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ae422640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a8140250 0x7fb5a8140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ae422640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a8150f10 0x7fb5a8151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ae422640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a8142d60 con 0x7fb5a8150f10 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ae422640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a8142be0 con 0x7fb5a8140250 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ae422640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a8142ee0 con 0x7fb590004800 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ac9bf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a8150f10 0x7fb5a8151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ac9bf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a8150f10 0x7fb5a8151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54640/0 (socket says 172.21.15.59:54640) 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5ac9bf640 1 -- 172.21.15.59:0/2907994676 learned_addr learned my addr 172.21.15.59:0/2907994676 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5a7fff640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb590004800 0x7fb5a81f68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.276+0000 7fb5a77fe640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a8140250 0x7fb5a8140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5ac9bf640 1 -- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb590004800 msgr2=0x7fb5a81f68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5ac9bf640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb590004800 0x7fb5a81f68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5ac9bf640 1 -- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a8140250 msgr2=0x7fb5a8140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5ac9bf640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a8140250 0x7fb5a8140660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5ac9bf640 1 -- 172.21.15.59:0/2907994676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a8151860 con 0x7fb5a8150f10 2024-06-29T15:55:17.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5ac9bf640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a8150f10 0x7fb5a8151320 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80b4ff0 tx=0x7fb5a00064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:17.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5a4ff9640 1 -- 172.21.15.59:0/2907994676 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00119c0 con 0x7fb5a8150f10 2024-06-29T15:55:17.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5a4ff9640 1 -- 172.21.15.59:0/2907994676 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5a0019070 con 0x7fb5a8150f10 2024-06-29T15:55:17.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a81414a0 con 0x7fb5a8150f10 2024-06-29T15:55:17.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5a4ff9640 1 -- 172.21.15.59:0/2907994676 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0011e00 con 0x7fb5a8150f10 2024-06-29T15:55:17.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.277+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a8141a60 con 0x7fb5a8150f10 2024-06-29T15:55:17.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.278+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a806c6d0 con 0x7fb5a8150f10 2024-06-29T15:55:17.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.279+0000 7fb5a4ff9640 1 -- 172.21.15.59:0/2907994676 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 112) ==== 98332+0+0 (secure 0 0 0) 0x7fb5a001e460 con 0x7fb5a8150f10 2024-06-29T15:55:17.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.279+0000 7fb5a4ff9640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb594076440 0x7fb594078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.280+0000 7fb5a4ff9640 1 -- 172.21.15.59:0/2907994676 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7fb5a00982b0 con 0x7fb5a8150f10 2024-06-29T15:55:17.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.280+0000 7fb5a7fff640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb594076440 0x7fb594078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.280+0000 7fb5a7fff640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb594076440 0x7fb594078950 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb598000c00 tx=0x7fb598006b70 comp rx=0 tx=0).ready entity=mgr.9976 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:17.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.284+0000 7fb5a4ff9640 1 -- 172.21.15.59:0/2907994676 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb5a80cd610 con 0x7fb5a8150f10 2024-06-29T15:55:17.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.498+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 --> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb5a8134c00 con 0x7fb594076440 2024-06-29T15:55:17.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.571+0000 7fb5a4ff9640 1 -- 172.21.15.59:0/2907994676 <== mgr.9976 v2:172.21.15.153:6832/2924556144 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7fb5980062e0 con 0x7fb594076440 2024-06-29T15:55:17.571 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-06-29T15:55:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb594076440 msgr2=0x7fb594078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb594076440 0x7fb594078950 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb598000c00 tx=0x7fb598006b70 comp rx=0 tx=0).stop 2024-06-29T15:55:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a8150f10 msgr2=0x7fb5a8151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a8150f10 0x7fb5a8151320 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80b4ff0 tx=0x7fb5a00064f0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 shutdown_connections 2024-06-29T15:55:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a8150f10 0x7fb5a8151320 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb594076440 0x7fb594078950 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a8140250 0x7fb5a8140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 --2- 172.21.15.59:0/2907994676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb590004800 0x7fb5a81f68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 >> 172.21.15.59:0/2907994676 conn(0x7fb5a80b24c0 msgr2=0x7fb5a815f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 shutdown_connections 2024-06-29T15:55:17.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.573+0000 7fb5ae422640 1 -- 172.21.15.59:0/2907994676 wait complete. 2024-06-29T15:55:17.584 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-06-29T15:55:17.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:17.584 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-06-29T15:55:17.600 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_40_00_UTC 2024-06-29T15:55:17.600 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_41_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_42_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_43_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_44_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_45_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_46_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_47_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_48_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_49_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_50_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_51_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_52_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_53_00_UTC 2024-06-29T15:55:17.601 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_54_00_UTC 2024-06-29T15:55:17.602 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_55_00_UTC 2024-06-29T15:55:17.602 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-15_40_00_UTC', 'scheduled-2024-06-29-15_41_00_UTC', 'scheduled-2024-06-29-15_42_00_UTC', 'scheduled-2024-06-29-15_43_00_UTC', 'scheduled-2024-06-29-15_44_00_UTC', 'scheduled-2024-06-29-15_45_00_UTC', 'scheduled-2024-06-29-15_46_00_UTC', 'scheduled-2024-06-29-15_47_00_UTC', 'scheduled-2024-06-29-15_48_00_UTC', 'scheduled-2024-06-29-15_49_00_UTC', 'scheduled-2024-06-29-15_50_00_UTC', 'scheduled-2024-06-29-15_51_00_UTC', 'scheduled-2024-06-29-15_52_00_UTC', 'scheduled-2024-06-29-15_53_00_UTC', 'scheduled-2024-06-29-15_54_00_UTC', 'scheduled-2024-06-29-15_55_00_UTC'] 2024-06-29T15:55:17.602 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-06-29T15:55:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.761+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1307303902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 msgr2=0x7fe7a40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.761+0000 7fe7bf01f640 1 --2- 172.21.15.59:0/1307303902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 0x7fe7a40051d0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80d3730 tx=0x7fe7a4005830 comp rx=0 tx=0).stop 2024-06-29T15:55:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.761+0000 7fe7bd5bc640 1 -- 172.21.15.59:0/1307303902 reap_dead start 2024-06-29T15:55:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.761+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1307303902 shutdown_connections 2024-06-29T15:55:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.761+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1307303902 >> 172.21.15.59:0/1307303902 conn(0x7fe7b80d45b0 msgr2=0x7fe7b80d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:17.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.761+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1307303902 shutdown_connections 2024-06-29T15:55:17.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.761+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1307303902 wait complete. 2024-06-29T15:55:17.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.763+0000 7fe7bf01f640 1 Processor -- start 2024-06-29T15:55:17.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.763+0000 7fe7bf01f640 1 -- start start 2024-06-29T15:55:17.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.764+0000 7fe7bf01f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 0x7fe7b8152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.764+0000 7fe7bf01f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7b80c7590 0x7fe7b80c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.764+0000 7fe7bf01f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7b80c9580 0x7fe7b80c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.764+0000 7fe7bf01f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b80b4170 con 0x7fe7a4004d40 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.764+0000 7fe7bf01f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b80b3ff0 con 0x7fe7b80c7590 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.764+0000 7fe7bf01f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b80b42f0 con 0x7fe7b80c9580 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.764+0000 7fe7bcdbb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 0x7fe7b8152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bcdbb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 0x7fe7b8152530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54660/0 (socket says 172.21.15.59:54660) 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bcdbb640 1 -- 172.21.15.59:0/1071722405 learned_addr learned my addr 172.21.15.59:0/1071722405 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7b7fff640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7b80c7590 0x7fe7b80c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bd5bc640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7b80c9580 0x7fe7b80c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bcdbb640 1 -- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7b80c9580 msgr2=0x7fe7b80c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bcdbb640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7b80c9580 0x7fe7b80c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bcdbb640 1 -- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7b80c7590 msgr2=0x7fe7b80c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bcdbb640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7b80c7590 0x7fe7b80c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bcdbb640 1 -- 172.21.15.59:0/1071722405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b80ca270 con 0x7fe7a4004d40 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bcdbb640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 0x7fe7b8152530 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8011ae0 tx=0x7fe7a8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7b57fa640 1 -- 172.21.15.59:0/1071722405 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a801e070 con 0x7fe7a4004d40 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7b57fa640 1 -- 172.21.15.59:0/1071722405 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7a8004460 con 0x7fe7a4004d40 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b80ca550 con 0x7fe7a4004d40 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7b57fa640 1 -- 172.21.15.59:0/1071722405 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a800fc60 con 0x7fe7a4004d40 2024-06-29T15:55:17.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.765+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b8157440 con 0x7fe7a4004d40 2024-06-29T15:55:17.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.766+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b80b4880 con 0x7fe7a4004d40 2024-06-29T15:55:17.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.767+0000 7fe7b57fa640 1 -- 172.21.15.59:0/1071722405 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 112) ==== 98332+0+0 (secure 0 0 0) 0x7fe7a8021070 con 0x7fe7a4004d40 2024-06-29T15:55:17.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.768+0000 7fe7b57fa640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe798076440 0x7fe798078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.768+0000 7fe7b57fa640 1 -- 172.21.15.59:0/1071722405 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7fe7a80989a0 con 0x7fe7a4004d40 2024-06-29T15:55:17.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.768+0000 7fe7b7fff640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe798076440 0x7fe798078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.768+0000 7fe7b7fff640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe798076440 0x7fe798078950 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0004fd0 tx=0x7fe7a0004ab0 comp rx=0 tx=0).ready entity=mgr.9976 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:17.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.772+0000 7fe7b57fa640 1 -- 172.21.15.59:0/1071722405 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe7a80615e0 con 0x7fe7a4004d40 2024-06-29T15:55:17.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.987+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 --> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe7b80cdd90 con 0x7fe798076440 2024-06-29T15:55:17.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.995+0000 7fe7b57fa640 1 -- 172.21.15.59:0/1071722405 <== mgr.9976 v2:172.21.15.153:6832/2924556144 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+376 (secure 0 0 0) 0x7fe7b80cdd90 con 0x7fe798076440 2024-06-29T15:55:17.996 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:17.996 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:39:24", "first": "2024-06-29T15:40:00", "last": "2024-06-29T15:55:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe798076440 msgr2=0x7fe798078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe798076440 0x7fe798078950 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0004fd0 tx=0x7fe7a0004ab0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 msgr2=0x7fe7b8152530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 0x7fe7b8152530 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8011ae0 tx=0x7fe7a8009e90 comp rx=0 tx=0).stop 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 shutdown_connections 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7b80c9580 0x7fe7b80c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe798076440 0x7fe798078950 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7b80c7590 0x7fe7b80c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 --2- 172.21.15.59:0/1071722405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a4004d40 0x7fe7b8152530 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 >> 172.21.15.59:0/1071722405 conn(0x7fe7b80d45b0 msgr2=0x7fe7b80ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 shutdown_connections 2024-06-29T15:55:17.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:17.998+0000 7fe7bf01f640 1 -- 172.21.15.59:0/1071722405 wait complete. 2024-06-29T15:55:18.009 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:39:24", "first": "2024-06-29T15:40:00", "last": "2024-06-29T15:55:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-06-29T15:55:18.009 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T15:39:24', 'first': '2024-06-29T15:40:00', 'last': '2024-06-29T15:55:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-06-29T15:55:18.009 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-06-29T15:55:18.009 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-06-29T15:55:18.009 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-06-29T15:55:18.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.130+0000 7feabcdbd640 1 -- 172.21.15.59:0/390868920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab80b3740 msgr2=0x7feab80b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.130+0000 7feabcdbd640 1 --2- 172.21.15.59:0/390868920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab80b3740 0x7feab80b3b50 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009830 tx=0x7feaa002a510 comp rx=0 tx=0).stop 2024-06-29T15:55:18.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.130+0000 7feabcdbd640 1 -- 172.21.15.59:0/390868920 shutdown_connections 2024-06-29T15:55:18.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.130+0000 7feabcdbd640 1 --2- 172.21.15.59:0/390868920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab80b3740 0x7feab80b3b50 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.130+0000 7feabcdbd640 1 --2- 172.21.15.59:0/390868920 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feab81458a0 0x7feab80cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.130+0000 7feabcdbd640 1 -- 172.21.15.59:0/390868920 >> 172.21.15.59:0/390868920 conn(0x7feab806c950 msgr2=0x7feab8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.130+0000 7feabcdbd640 1 -- 172.21.15.59:0/390868920 shutdown_connections 2024-06-29T15:55:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.130+0000 7feabcdbd640 1 -- 172.21.15.59:0/390868920 wait complete. 2024-06-29T15:55:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.132+0000 7feabcdbd640 1 Processor -- start 2024-06-29T15:55:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.132+0000 7feabcdbd640 1 -- start start 2024-06-29T15:55:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feabcdbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab80b3740 0x7feab8140320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feabcdbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab815c2f0 0x7feab806e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feabcdbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feab806ffc0 0x7feab80703d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feabcdbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7feab81601a0 con 0x7feab815c2f0 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feabcdbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feab8160020 con 0x7feab80b3740 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feabcdbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7feab8160320 con 0x7feab806ffc0 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feab5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab815c2f0 0x7feab806e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feab5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab815c2f0 0x7feab806e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54676/0 (socket says 172.21.15.59:54676) 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.133+0000 7feab5d9b640 1 -- 172.21.15.59:0/3600409773 learned_addr learned my addr 172.21.15.59:0/3600409773 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feab6d9d640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feab806ffc0 0x7feab80703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feab659c640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab80b3740 0x7feab8140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feab5d9b640 1 -- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feab806ffc0 msgr2=0x7feab80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feab5d9b640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feab806ffc0 0x7feab80703d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feab5d9b640 1 -- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab80b3740 msgr2=0x7feab8140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feab5d9b640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab80b3740 0x7feab8140320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feab5d9b640 1 -- 172.21.15.59:0/3600409773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab8070910 con 0x7feab815c2f0 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feab5d9b640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab815c2f0 0x7feab806e600 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009950 tx=0x7feaa0030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feaaeffd640 1 -- 172.21.15.59:0/3600409773 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa003f070 con 0x7feab815c2f0 2024-06-29T15:55:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feaaeffd640 1 -- 172.21.15.59:0/3600409773 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feaa0030d40 con 0x7feab815c2f0 2024-06-29T15:55:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab806faa0 con 0x7feab815c2f0 2024-06-29T15:55:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feaaeffd640 1 -- 172.21.15.59:0/3600409773 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0005280 con 0x7feab815c2f0 2024-06-29T15:55:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.134+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab80aebe0 con 0x7feab815c2f0 2024-06-29T15:55:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.135+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab806f810 con 0x7feab815c2f0 2024-06-29T15:55:18.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.136+0000 7feaaeffd640 1 -- 172.21.15.59:0/3600409773 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 112) ==== 98332+0+0 (secure 0 0 0) 0x7feaa0005420 con 0x7feab815c2f0 2024-06-29T15:55:18.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.137+0000 7feaaeffd640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fea98076470 0x7fea98078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:18.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.137+0000 7feaaeffd640 1 -- 172.21.15.59:0/3600409773 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7feaa00b8ff0 con 0x7feab815c2f0 2024-06-29T15:55:18.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.137+0000 7feab659c640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fea98076470 0x7fea98078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:18.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.138+0000 7feab659c640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fea98076470 0x7fea98078980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feab0005fd0 tx=0x7feab000a250 comp rx=0 tx=0).ready entity=mgr.9976 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:18.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.141+0000 7feaaeffd640 1 -- 172.21.15.59:0/3600409773 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feab8070910 con 0x7feab815c2f0 2024-06-29T15:55:18.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.354+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 --> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7feab81458a0 con 0x7fea98076470 2024-06-29T15:55:18.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.433+0000 7feaaeffd640 1 -- 172.21.15.59:0/3600409773 <== mgr.9976 v2:172.21.15.153:6832/2924556144 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+55 (secure 0 0 0) 0x7feab81458a0 con 0x7fea98076470 2024-06-29T15:55:18.433 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fea98076470 msgr2=0x7fea98078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fea98076470 0x7fea98078980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feab0005fd0 tx=0x7feab000a250 comp rx=0 tx=0).stop 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab815c2f0 msgr2=0x7feab806e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab815c2f0 0x7feab806e600 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009950 tx=0x7feaa0030650 comp rx=0 tx=0).stop 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 shutdown_connections 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feab806ffc0 0x7feab80703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab815c2f0 0x7feab806e600 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fea98076470 0x7fea98078980 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 --2- 172.21.15.59:0/3600409773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab80b3740 0x7feab8140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 >> 172.21.15.59:0/3600409773 conn(0x7feab806c950 msgr2=0x7feab815b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:18.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 shutdown_connections 2024-06-29T15:55:18.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:18.435+0000 7feabcdbd640 1 -- 172.21.15.59:0/3600409773 wait complete. 2024-06-29T15:55:18.446 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-06-29T15:55:18.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.447 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-06-29T15:55:18.463 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_40_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_41_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_42_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_43_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_44_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_45_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_46_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_47_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_48_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_49_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_50_00_UTC 2024-06-29T15:55:18.464 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_51_00_UTC 2024-06-29T15:55:18.465 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_52_00_UTC 2024-06-29T15:55:18.465 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_53_00_UTC 2024-06-29T15:55:18.465 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_54_00_UTC 2024-06-29T15:55:18.465 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_55_00_UTC 2024-06-29T15:55:18.465 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_40_00_UTC 2024-06-29T15:55:18.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.465 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_40_00_UTC) 2024-06-29T15:55:18.525 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_41_00_UTC 2024-06-29T15:55:18.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.525 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_41_00_UTC) 2024-06-29T15:55:18.585 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_42_00_UTC 2024-06-29T15:55:18.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.586 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_42_00_UTC) 2024-06-29T15:55:18.647 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_43_00_UTC 2024-06-29T15:55:18.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.647 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_43_00_UTC) 2024-06-29T15:55:18.707 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_44_00_UTC 2024-06-29T15:55:18.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.708 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_44_00_UTC) 2024-06-29T15:55:18.768 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_45_00_UTC 2024-06-29T15:55:18.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.769 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_45_00_UTC) 2024-06-29T15:55:18.829 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_46_00_UTC 2024-06-29T15:55:18.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.829 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_46_00_UTC) 2024-06-29T15:55:18.891 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_47_00_UTC 2024-06-29T15:55:18.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.891 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_47_00_UTC) 2024-06-29T15:55:18.951 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_48_00_UTC 2024-06-29T15:55:18.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.952 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_48_00_UTC) 2024-06-29T15:55:19.012 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_49_00_UTC 2024-06-29T15:55:19.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.012 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_49_00_UTC) 2024-06-29T15:55:19.073 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_50_00_UTC 2024-06-29T15:55:19.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.074 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_50_00_UTC) 2024-06-29T15:55:19.134 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_51_00_UTC 2024-06-29T15:55:19.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.135 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_51_00_UTC) 2024-06-29T15:55:19.195 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_52_00_UTC 2024-06-29T15:55:19.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.196 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_52_00_UTC) 2024-06-29T15:55:19.257 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_53_00_UTC 2024-06-29T15:55:19.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.257 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_53_00_UTC) 2024-06-29T15:55:19.318 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_54_00_UTC 2024-06-29T15:55:19.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.318 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_54_00_UTC) 2024-06-29T15:55:19.378 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_55_00_UTC 2024-06-29T15:55:19.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.379 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_55_00_UTC) 2024-06-29T15:55:19.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.439 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-06-29T15:55:19.499 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:55:19.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.658+0000 7fb9ea681640 1 -- 172.21.15.59:0/998103874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e415e0a0 msgr2=0x7fb9e4161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:19.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.658+0000 7fb9ea681640 1 --2- 172.21.15.59:0/998103874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e415e0a0 0x7fb9e4161a80 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40b29a0 tx=0x7fb9d802a4b0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.658+0000 7fb9ea681640 1 -- 172.21.15.59:0/998103874 shutdown_connections 2024-06-29T15:55:19.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.658+0000 7fb9ea681640 1 --2- 172.21.15.59:0/998103874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e415e0a0 0x7fb9e4161a80 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.658+0000 7fb9ea681640 1 --2- 172.21.15.59:0/998103874 >> v2:172.21.15.153:3300/0 conn(0x7fb9e4162770 0x7fb9e40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.658+0000 7fb9ea681640 1 -- 172.21.15.59:0/998103874 >> 172.21.15.59:0/998103874 conn(0x7fb9e40d45b0 msgr2=0x7fb9e40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:19.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.658+0000 7fb9ea681640 1 -- 172.21.15.59:0/998103874 shutdown_connections 2024-06-29T15:55:19.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.658+0000 7fb9ea681640 1 -- 172.21.15.59:0/998103874 wait complete. 2024-06-29T15:55:19.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.660+0000 7fb9ea681640 1 Processor -- start 2024-06-29T15:55:19.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.660+0000 7fb9ea681640 1 -- start start 2024-06-29T15:55:19.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.661+0000 7fb9ea681640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e415e0a0 0x7fb9e41f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.661+0000 7fb9ea681640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e4161fc0 0x7fb9e41f88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.661+0000 7fb9ea681640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4162770 0x7fb9e40c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.661+0000 7fb9ea681640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e40d2d80 con 0x7fb9e4161fc0 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.661+0000 7fb9ea681640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e40b4b00 con 0x7fb9e4162770 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.661+0000 7fb9ea681640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e40d2f00 con 0x7fb9e415e0a0 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e4161fc0 0x7fb9e41f88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e4161fc0 0x7fb9e41f88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54686/0 (socket says 172.21.15.59:54686) 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 -- 172.21.15.59:0/3597872091 learned_addr learned my addr 172.21.15.59:0/3597872091 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e3fff640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e415e0a0 0x7fb9e41f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e8c1e640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4162770 0x7fb9e40c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 -- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e415e0a0 msgr2=0x7fb9e41f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:19.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e415e0a0 0x7fb9e41f83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 -- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4162770 msgr2=0x7fb9e40c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4162770 0x7fb9e40c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 -- 172.21.15.59:0/3597872091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e415fdc0 con 0x7fb9e4161fc0 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e37fe640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e4161fc0 0x7fb9e41f88e0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8000c00 tx=0x7fb9d8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e0ff9640 1 -- 172.21.15.59:0/3597872091 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d803e030 con 0x7fb9e4161fc0 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e0ff9640 1 -- 172.21.15.59:0/3597872091 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9d8003e00 con 0x7fb9e4161fc0 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e41600a0 con 0x7fb9e4161fc0 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9e0ff9640 1 -- 172.21.15.59:0/3597872091 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8035040 con 0x7fb9e4161fc0 2024-06-29T15:55:19.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.662+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e4160660 con 0x7fb9e4161fc0 2024-06-29T15:55:19.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.663+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e40d3420 con 0x7fb9e4161fc0 2024-06-29T15:55:19.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.664+0000 7fb9e0ff9640 1 -- 172.21.15.59:0/3597872091 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 112) ==== 98332+0+0 (secure 0 0 0) 0x7fb9d8041030 con 0x7fb9e4161fc0 2024-06-29T15:55:19.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.665+0000 7fb9e0ff9640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb9d0076470 0x7fb9d0078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:19.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.665+0000 7fb9e0ff9640 1 -- 172.21.15.59:0/3597872091 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7fb9d80b8bb0 con 0x7fb9e4161fc0 2024-06-29T15:55:19.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.665+0000 7fb9e3fff640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb9d0076470 0x7fb9d0078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:19.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.666+0000 7fb9e3fff640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb9d0076470 0x7fb9d0078980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400bb50 tx=0x7fb9d400b2e0 comp rx=0 tx=0).ready entity=mgr.9976 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:19.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.668+0000 7fb9e0ff9640 1 -- 172.21.15.59:0/3597872091 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb9e41623d0 con 0x7fb9e4161fc0 2024-06-29T15:55:19.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.910+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7fb9e40ca740 con 0x7fb9e4161fc0 2024-06-29T15:55:19.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.944+0000 7fb9e0ff9640 1 -- 172.21.15.59:0/3597872091 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v113) ==== 93+0+0 (secure 0 0 0) 0x7fb9d8081790 con 0x7fb9e4161fc0 2024-06-29T15:55:19.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb9d0076470 msgr2=0x7fb9d0078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:19.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb9d0076470 0x7fb9d0078980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400bb50 tx=0x7fb9d400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e4161fc0 msgr2=0x7fb9e41f88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:19.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e4161fc0 0x7fb9e41f88e0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8000c00 tx=0x7fb9d8003910 comp rx=0 tx=0).stop 2024-06-29T15:55:19.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 shutdown_connections 2024-06-29T15:55:19.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fb9d0076470 0x7fb9d0078980 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400bb50 tx=0x7fb9d400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9e4162770 0x7fb9e40c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9e4161fc0 0x7fb9e41f88e0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 --2- 172.21.15.59:0/3597872091 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9e415e0a0 0x7fb9e41f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 >> 172.21.15.59:0/3597872091 conn(0x7fb9e40d45b0 msgr2=0x7fb9e40ccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:19.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.948+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 shutdown_connections 2024-06-29T15:55:19.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:19.949+0000 7fb9ea681640 1 -- 172.21.15.59:0/3597872091 wait complete. 2024-06-29T15:55:19.963 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:55:19.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:55:20.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:55:20.048 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:55:20.048 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:55:20.049 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:55:20.049 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:20.065 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:55:20.065 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:20.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:20.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.282+0000 7f38d90e4640 1 -- 172.21.15.59:0/1529794931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 msgr2=0x7f38d40b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.282+0000 7f38d90e4640 1 --2- 172.21.15.59:0/1529794931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 0x7f38d40b3990 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f38c4005fd0 tx=0x7f38c402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.282+0000 7f38d90e4640 1 -- 172.21.15.59:0/1529794931 shutdown_connections 2024-06-29T15:55:20.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.282+0000 7f38d90e4640 1 --2- 172.21.15.59:0/1529794931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38d415efe0 0x7f38d415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.282+0000 7f38d90e4640 1 --2- 172.21.15.59:0/1529794931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 0x7f38d40b3990 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.282+0000 7f38d90e4640 1 -- 172.21.15.59:0/1529794931 >> 172.21.15.59:0/1529794931 conn(0x7f38d40d3ee0 msgr2=0x7f38d40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.282+0000 7f38d90e4640 1 -- 172.21.15.59:0/1529794931 shutdown_connections 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.282+0000 7f38d90e4640 1 -- 172.21.15.59:0/1529794931 wait complete. 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.284+0000 7f38d90e4640 1 Processor -- start 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.284+0000 7f38d90e4640 1 -- start start 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d90e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d40b3ed0 0x7f38d4158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d90e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 0x7f38d4159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d90e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38d415efe0 0x7f38d415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d90e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f38d4160280 con 0x7f38d40d23c0 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d90e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f38d4160100 con 0x7f38d40b3ed0 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d90e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f38d4160400 con 0x7f38d415efe0 2024-06-29T15:55:20.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 0x7f38d4159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 0x7f38d4159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54710/0 (socket says 172.21.15.59:54710) 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 -- 172.21.15.59:0/3149584064 learned_addr learned my addr 172.21.15.59:0/3149584064 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d359e640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38d415efe0 0x7f38d415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d2d9d640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d40b3ed0 0x7f38d4158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 -- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38d415efe0 msgr2=0x7f38d415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38d415efe0 0x7f38d415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 -- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d40b3ed0 msgr2=0x7f38d4158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d40b3ed0 0x7f38d4158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 -- 172.21.15.59:0/3149584064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d41e8d70 con 0x7f38d40d23c0 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.286+0000 7f38d259c640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 0x7f38d4159af0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c3c0 tx=0x7f38c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.287+0000 7f38bb7fe640 1 -- 172.21.15.59:0/3149584064 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8017070 con 0x7f38d40d23c0 2024-06-29T15:55:20.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.287+0000 7f38bb7fe640 1 -- 172.21.15.59:0/3149584064 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38c80035f0 con 0x7f38d40d23c0 2024-06-29T15:55:20.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.287+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d41e81f0 con 0x7f38d40d23c0 2024-06-29T15:55:20.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.287+0000 7f38bb7fe640 1 -- 172.21.15.59:0/3149584064 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8003a20 con 0x7f38d40d23c0 2024-06-29T15:55:20.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.287+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d41e87b0 con 0x7f38d40d23c0 2024-06-29T15:55:20.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.288+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d40b3990 con 0x7f38d40d23c0 2024-06-29T15:55:20.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.290+0000 7f38bb7fe640 1 -- 172.21.15.59:0/3149584064 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f38c801a070 con 0x7f38d40d23c0 2024-06-29T15:55:20.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.290+0000 7f38bb7fe640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f38c0076420 0x7f38c0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.290+0000 7f38d2d9d640 1 -- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f38c0076420 msgr2=0x7f38c0078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:20.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.290+0000 7f38bb7fe640 1 -- 172.21.15.59:0/3149584064 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7f38c8097070 con 0x7f38d40d23c0 2024-06-29T15:55:20.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.290+0000 7f38d2d9d640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f38c0076420 0x7f38c0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:20.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.295+0000 7f38bb7fe640 1 -- 172.21.15.59:0/3149584064 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f38d40b42e0 con 0x7f38d40d23c0 2024-06-29T15:55:20.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.488+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38d40d2950 con 0x7f38d40d23c0 2024-06-29T15:55:20.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.490+0000 7f38bb7fe640 1 -- 172.21.15.59:0/3149584064 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) ==== 74+0+17959 (secure 0 0 0) 0x7f38c805fd00 con 0x7f38d40d23c0 2024-06-29T15:55:20.491 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:20.491 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":191,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:50:20.742223+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.834000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.377444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3420796917":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3711582444":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3023327174":"2024-06-30T15:50:20.742173+0000","172.21.15.153:6833/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:6832/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/1412243249":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3586261468":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1851934149":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1918645711":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/4098340029":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/2596446141":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/3821515452":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1052117807":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/2595561638":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6833/1270545814":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6832/1270545814":"2024-06-30T15:39:23.696245+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:20.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.491+0000 7f38d2d9d640 1 -- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f38c0076420 msgr2=0x7f38c0078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:20.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.491+0000 7f38d2d9d640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f38c0076420 0x7f38c0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f38c0076420 msgr2=0x7f38c0078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f38c0076420 0x7f38c0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 msgr2=0x7f38d4159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 0x7f38d4159af0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c3c0 tx=0x7f38c800c880 comp rx=0 tx=0).stop 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 shutdown_connections 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38d415efe0 0x7f38d415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38d40d23c0 0x7f38d4159af0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f38c0076420 0x7f38c0078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 --2- 172.21.15.59:0/3149584064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38d40b3ed0 0x7f38d4158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 >> 172.21.15.59:0/3149584064 conn(0x7f38d40d3ee0 msgr2=0x7f38d40ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 shutdown_connections 2024-06-29T15:55:20.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.492+0000 7f38d90e4640 1 -- 172.21.15.59:0/3149584064 wait complete. 2024-06-29T15:55:20.504 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:39:09.834000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:39:10.377444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:55:20.504 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:55:20.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.626+0000 7f13ba991640 1 -- 172.21.15.59:0/3331197416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b40b3ed0 msgr2=0x7f13b415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.626+0000 7f13ba991640 1 --2- 172.21.15.59:0/3331197416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b40b3ed0 0x7f13b415e8d0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f139c005fd0 tx=0x7f139c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.626+0000 7f13ba991640 1 -- 172.21.15.59:0/3331197416 shutdown_connections 2024-06-29T15:55:20.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.626+0000 7f13ba991640 1 --2- 172.21.15.59:0/3331197416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b40b3ed0 0x7f13b415e8d0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.626+0000 7f13ba991640 1 --2- 172.21.15.59:0/3331197416 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13b40d23c0 0x7f13b40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.626+0000 7f13ba991640 1 -- 172.21.15.59:0/3331197416 >> 172.21.15.59:0/3331197416 conn(0x7f13b40d3ee0 msgr2=0x7f13b40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.626+0000 7f13ba991640 1 -- 172.21.15.59:0/3331197416 shutdown_connections 2024-06-29T15:55:20.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.626+0000 7f13ba991640 1 -- 172.21.15.59:0/3331197416 wait complete. 2024-06-29T15:55:20.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.629+0000 7f13ba991640 1 Processor -- start 2024-06-29T15:55:20.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.629+0000 7f13ba991640 1 -- start start 2024-06-29T15:55:20.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.631+0000 7f13ba991640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b40b3ed0 0x7f13b4152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13ba991640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13b40d23c0 0x7f13b414f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13ba991640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b415ee10 0x7f13b41e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13ba991640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f13b4160240 con 0x7f13b415ee10 2024-06-29T15:55:20.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13ba991640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13b41600c0 con 0x7f13b40b3ed0 2024-06-29T15:55:20.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13ba991640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f13b41603c0 con 0x7f13b40d23c0 2024-06-29T15:55:20.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b415ee10 0x7f13b41e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b415ee10 0x7f13b41e8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54720/0 (socket says 172.21.15.59:54720) 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 -- 172.21.15.59:0/3794817411 learned_addr learned my addr 172.21.15.59:0/3794817411 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13abfff640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13b40d23c0 0x7f13b414f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b3fff640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b40b3ed0 0x7f13b4152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 -- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13b40d23c0 msgr2=0x7f13b414f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13b40d23c0 0x7f13b414f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 -- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b40b3ed0 msgr2=0x7f13b4152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b40b3ed0 0x7f13b4152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 -- 172.21.15.59:0/3794817411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b41533f0 con 0x7f13b415ee10 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b8f2e640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b415ee10 0x7f13b41e8de0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00c6e0 tx=0x7f13ac00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b17fa640 1 -- 172.21.15.59:0/3794817411 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac003b70 con 0x7f13b415ee10 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13b17fa640 1 -- 172.21.15.59:0/3794817411 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13ac003d30 con 0x7f13b415ee10 2024-06-29T15:55:20.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.632+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b4153900 con 0x7f13b415ee10 2024-06-29T15:55:20.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.633+0000 7f13b17fa640 1 -- 172.21.15.59:0/3794817411 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac005330 con 0x7f13b415ee10 2024-06-29T15:55:20.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.633+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b4153ec0 con 0x7f13b415ee10 2024-06-29T15:55:20.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.634+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b4153650 con 0x7f13b415ee10 2024-06-29T15:55:20.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.634+0000 7f13b17fa640 1 -- 172.21.15.59:0/3794817411 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f13ac0054d0 con 0x7f13b415ee10 2024-06-29T15:55:20.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.635+0000 7f13b17fa640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f1394076420 0x7f1394078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.635+0000 7f13b3fff640 1 -- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f1394076420 msgr2=0x7f1394078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:20.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.635+0000 7f13b17fa640 1 -- 172.21.15.59:0/3794817411 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7f13ac097a50 con 0x7f13b415ee10 2024-06-29T15:55:20.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.635+0000 7f13b3fff640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f1394076420 0x7f1394078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:20.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.639+0000 7f13b17fa640 1 -- 172.21.15.59:0/3794817411 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f13b40b42e0 con 0x7f13b415ee10 2024-06-29T15:55:20.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.831+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13b415e8d0 con 0x7f13b415ee10 2024-06-29T15:55:20.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.831+0000 7f13b17fa640 1 -- 172.21.15.59:0/3794817411 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v191) ==== 93+0+10 (secure 0 0 0) 0x7f13ac060770 con 0x7f13b415ee10 2024-06-29T15:55:20.832 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.833+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f1394076420 msgr2=0x7f1394078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.833+0000 7f13ba991640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f1394076420 0x7f1394078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.833+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b415ee10 msgr2=0x7f13b41e8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.833+0000 7f13ba991640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b415ee10 0x7f13b41e8de0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00c6e0 tx=0x7f13ac00cba0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.834+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 shutdown_connections 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.834+0000 7f13ba991640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13b415ee10 0x7f13b41e8de0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.834+0000 7f13ba991640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13b40d23c0 0x7f13b414f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.834+0000 7f13ba991640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f1394076420 0x7f1394078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.834+0000 7f13ba991640 1 --2- 172.21.15.59:0/3794817411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13b40b3ed0 0x7f13b4152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.834+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 >> 172.21.15.59:0/3794817411 conn(0x7f13b40d3ee0 msgr2=0x7f13b40ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.834+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 shutdown_connections 2024-06-29T15:55:20.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.834+0000 7f13ba991640 1 -- 172.21.15.59:0/3794817411 wait complete. 2024-06-29T15:55:20.845 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:55:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.968+0000 7f2adb343640 1 -- 172.21.15.59:0/1329138836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad4071620 msgr2=0x7f2ad415bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.968+0000 7f2adb343640 1 --2- 172.21.15.59:0/1329138836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad4071620 0x7f2ad415bcd0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009830 tx=0x7f2ad002a530 comp rx=0 tx=0).stop 2024-06-29T15:55:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.968+0000 7f2adb343640 1 -- 172.21.15.59:0/1329138836 shutdown_connections 2024-06-29T15:55:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.968+0000 7f2adb343640 1 --2- 172.21.15.59:0/1329138836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad4071620 0x7f2ad415bcd0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.968+0000 7f2adb343640 1 -- 172.21.15.59:0/1329138836 >> 172.21.15.59:0/1329138836 conn(0x7f2ad4073e50 msgr2=0x7f2ad40b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.969+0000 7f2adb343640 1 -- 172.21.15.59:0/1329138836 shutdown_connections 2024-06-29T15:55:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.969+0000 7f2adb343640 1 -- 172.21.15.59:0/1329138836 wait complete. 2024-06-29T15:55:20.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.970+0000 7f2adb343640 1 Processor -- start 2024-06-29T15:55:20.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.970+0000 7f2adb343640 1 -- start start 2024-06-29T15:55:20.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2adb343640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ad406eb40 0x7f2ad4159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2adb343640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad4071620 0x7f2ad40b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2adb343640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad40b5320 0x7f2ad4145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2adb343640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad41661d0 con 0x7f2ad40b5320 2024-06-29T15:55:20.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2adb343640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4166050 con 0x7f2ad4071620 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2adb343640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad4166350 con 0x7f2ad406eb40 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad98e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad40b5320 0x7f2ad4145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad98e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad40b5320 0x7f2ad4145240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54752/0 (socket says 172.21.15.59:54752) 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad98e0640 1 -- 172.21.15.59:0/2865183276 learned_addr learned my addr 172.21.15.59:0/2865183276 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad90df640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ad406eb40 0x7f2ad4159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad88de640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad4071620 0x7f2ad40b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad98e0640 1 -- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ad406eb40 msgr2=0x7f2ad4159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad98e0640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ad406eb40 0x7f2ad4159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad98e0640 1 -- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad4071620 msgr2=0x7f2ad40b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad98e0640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad4071620 0x7f2ad40b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.972+0000 7f2ad98e0640 1 -- 172.21.15.59:0/2865183276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad415a7f0 con 0x7f2ad40b5320 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.973+0000 7f2ad98e0640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad40b5320 0x7f2ad4145240 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c760 tx=0x7f2acc00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:20.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.973+0000 7f2ac9ffb640 1 -- 172.21.15.59:0/2865183276 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc003b70 con 0x7f2ad40b5320 2024-06-29T15:55:20.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.973+0000 7f2ac9ffb640 1 -- 172.21.15.59:0/2865183276 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2acc003d10 con 0x7f2ad40b5320 2024-06-29T15:55:20.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.973+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad4144480 con 0x7f2ad40b5320 2024-06-29T15:55:20.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.973+0000 7f2ac9ffb640 1 -- 172.21.15.59:0/2865183276 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0052d0 con 0x7f2ad40b5320 2024-06-29T15:55:20.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.973+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad4144a40 con 0x7f2ad40b5320 2024-06-29T15:55:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.974+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad415bcd0 con 0x7f2ad40b5320 2024-06-29T15:55:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.975+0000 7f2ac9ffb640 1 -- 172.21.15.59:0/2865183276 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f2acc005470 con 0x7f2ad40b5320 2024-06-29T15:55:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.976+0000 7f2ac9ffb640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2ab8076420 0x7f2ab8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.976+0000 7f2ad90df640 1 -- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2ab8076420 msgr2=0x7f2ab8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.976+0000 7f2ac9ffb640 1 -- 172.21.15.59:0/2865183276 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7f2acc097a00 con 0x7f2ad40b5320 2024-06-29T15:55:20.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.976+0000 7f2ad90df640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2ab8076420 0x7f2ab8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:20.981+0000 7f2ac9ffb640 1 -- 172.21.15.59:0/2865183276 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ad4070300 con 0x7f2ad40b5320 2024-06-29T15:55:21.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.173+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2ad40cec90 con 0x7f2ad40b5320 2024-06-29T15:55:21.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.173+0000 7f2ac9ffb640 1 -- 172.21.15.59:0/2865183276 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v191) ==== 104+0+11 (secure 0 0 0) 0x7f2acc060710 con 0x7f2ad40b5320 2024-06-29T15:55:21.174 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.175+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2ab8076420 msgr2=0x7f2ab8078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.175+0000 7f2adb343640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2ab8076420 0x7f2ab8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.175+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad40b5320 msgr2=0x7f2ad4145240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.175+0000 7f2adb343640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad40b5320 0x7f2ad4145240 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c760 tx=0x7f2acc00cb60 comp rx=0 tx=0).stop 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.176+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 shutdown_connections 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.176+0000 7f2adb343640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ad40b5320 0x7f2ad4145240 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.176+0000 7f2adb343640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2ab8076420 0x7f2ab8078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.176+0000 7f2adb343640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad4071620 0x7f2ad40b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.176+0000 7f2adb343640 1 --2- 172.21.15.59:0/2865183276 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2ad406eb40 0x7f2ad4159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.176+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 >> 172.21.15.59:0/2865183276 conn(0x7f2ad4073e50 msgr2=0x7f2ad40b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.176+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 shutdown_connections 2024-06-29T15:55:21.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.176+0000 7f2adb343640 1 -- 172.21.15.59:0/2865183276 wait complete. 2024-06-29T15:55:21.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:55:21.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.309+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2268766343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e415efe0 msgr2=0x7f73e415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.309+0000 7f73ec0f6640 1 --2- 172.21.15.59:0/2268766343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e415efe0 0x7f73e415f9b0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f73dc005fd0 tx=0x7f73dc02a940 comp rx=0 tx=0).stop 2024-06-29T15:55:21.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.309+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2268766343 shutdown_connections 2024-06-29T15:55:21.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.309+0000 7f73ec0f6640 1 --2- 172.21.15.59:0/2268766343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e415efe0 0x7f73e415f9b0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.309+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2268766343 >> 172.21.15.59:0/2268766343 conn(0x7f73e40d3ee0 msgr2=0x7f73e40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.309+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2268766343 shutdown_connections 2024-06-29T15:55:21.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.309+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2268766343 wait complete. 2024-06-29T15:55:21.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.311+0000 7f73ec0f6640 1 Processor -- start 2024-06-29T15:55:21.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.311+0000 7f73ec0f6640 1 -- start start 2024-06-29T15:55:21.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.312+0000 7f73ec0f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e40d23c0 0x7f73e414fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.312+0000 7f73ec0f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e415efe0 0x7f73e40c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.312+0000 7f73ec0f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e40cbbc0 0x7f73e40c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ec0f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f73e4160420 con 0x7f73e40cbbc0 2024-06-29T15:55:21.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ec0f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73e41602a0 con 0x7f73e415efe0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ec0f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f73e41605a0 con 0x7f73e40d23c0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e40cbbc0 0x7f73e40c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e40cbbc0 0x7f73e40c5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54764/0 (socket says 172.21.15.59:54764) 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73e9691640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e415efe0 0x7f73e40c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 -- 172.21.15.59:0/2740411881 learned_addr learned my addr 172.21.15.59:0/2740411881 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73e9e92640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e40d23c0 0x7f73e414fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 -- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e40d23c0 msgr2=0x7f73e414fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e40d23c0 0x7f73e414fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 -- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e415efe0 msgr2=0x7f73e40c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e415efe0 0x7f73e40c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 -- 172.21.15.59:0/2740411881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e40cd030 con 0x7f73e40cbbc0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.313+0000 7f73ea693640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e40cbbc0 0x7f73e40c5f20 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009cd0 tx=0x7f73dc031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.314+0000 7f73d67fc640 1 -- 172.21.15.59:0/2740411881 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc03e070 con 0x7f73e40cbbc0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.314+0000 7f73d67fc640 1 -- 172.21.15.59:0/2740411881 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73dc004540 con 0x7f73e40cbbc0 2024-06-29T15:55:21.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.314+0000 7f73d67fc640 1 -- 172.21.15.59:0/2740411881 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc043460 con 0x7f73e40cbbc0 2024-06-29T15:55:21.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.314+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e40cd310 con 0x7f73e40cbbc0 2024-06-29T15:55:21.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.314+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e41f60f0 con 0x7f73e40cbbc0 2024-06-29T15:55:21.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.315+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e415f9b0 con 0x7f73e40cbbc0 2024-06-29T15:55:21.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.315+0000 7f73d67fc640 1 -- 172.21.15.59:0/2740411881 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f73dc043600 con 0x7f73e40cbbc0 2024-06-29T15:55:21.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.316+0000 7f73d67fc640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f73c80763f0 0x7f73c8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.316+0000 7f73e9e92640 1 -- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f73c80763f0 msgr2=0x7f73c8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:21.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.316+0000 7f73d67fc640 1 -- 172.21.15.59:0/2740411881 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7f73dc0b96a0 con 0x7f73e40cbbc0 2024-06-29T15:55:21.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.316+0000 7f73e9e92640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f73c80763f0 0x7f73c8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:21.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.321+0000 7f73d67fc640 1 -- 172.21.15.59:0/2740411881 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73dc04a030 con 0x7f73e40cbbc0 2024-06-29T15:55:21.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.516+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f73e40d2950 con 0x7f73e40cbbc0 2024-06-29T15:55:21.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.516+0000 7f73d67fc640 1 -- 172.21.15.59:0/2740411881 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v191) ==== 100+0+11 (secure 0 0 0) 0x7f73dc082220 con 0x7f73e40cbbc0 2024-06-29T15:55:21.516 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:55:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.517+0000 7f73e9e92640 1 -- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f73c80763f0 msgr2=0x7f73c8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.517+0000 7f73e9e92640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f73c80763f0 0x7f73c8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f73c80763f0 msgr2=0x7f73c8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f73c80763f0 0x7f73c8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e40cbbc0 msgr2=0x7f73e40c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e40cbbc0 0x7f73e40c5f20 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009cd0 tx=0x7f73dc031e20 comp rx=0 tx=0).stop 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 shutdown_connections 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e40cbbc0 0x7f73e40c5f20 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f73c80763f0 0x7f73c8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e415efe0 0x7f73e40c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 --2- 172.21.15.59:0/2740411881 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e40d23c0 0x7f73e414fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 >> 172.21.15.59:0/2740411881 conn(0x7f73e40d3ee0 msgr2=0x7f73e40d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 shutdown_connections 2024-06-29T15:55:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.518+0000 7f73ec0f6640 1 -- 172.21.15.59:0/2740411881 wait complete. 2024-06-29T15:55:21.530 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:55:21.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.650+0000 7f22370ac640 1 -- 172.21.15.59:0/1326691616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2270 msgr2=0x7f223015bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.650+0000 7f22370ac640 1 --2- 172.21.15.59:0/1326691616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2270 0x7f223015bc20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2224009830 tx=0x7f222402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.650+0000 7f22370ac640 1 -- 172.21.15.59:0/1326691616 shutdown_connections 2024-06-29T15:55:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.650+0000 7f22370ac640 1 --2- 172.21.15.59:0/1326691616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2270 0x7f223015bc20 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.650+0000 7f22370ac640 1 -- 172.21.15.59:0/1326691616 >> 172.21.15.59:0/1326691616 conn(0x7f2230073e10 msgr2=0x7f2230074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.650+0000 7f22370ac640 1 -- 172.21.15.59:0/1326691616 shutdown_connections 2024-06-29T15:55:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.650+0000 7f22370ac640 1 -- 172.21.15.59:0/1326691616 wait complete. 2024-06-29T15:55:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.652+0000 7f22370ac640 1 Processor -- start 2024-06-29T15:55:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.652+0000 7f22370ac640 1 -- start start 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.653+0000 7f22370ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22300b2270 0x7f22301462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.653+0000 7f22370ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2fa0 0x7f223015d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.653+0000 7f22370ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22300d1c30 0x7f223015e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.653+0000 7f22370ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f22300b65d0 con 0x7f22300b2270 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.653+0000 7f22370ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22300b6450 con 0x7f22300b2fa0 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.653+0000 7f22370ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f22300b6750 con 0x7f22300d1c30 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22300b2270 0x7f22301462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22300b2270 0x7f22301462d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54766/0 (socket says 172.21.15.59:54766) 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 -- 172.21.15.59:0/2969228555 learned_addr learned my addr 172.21.15.59:0/2969228555 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f222ffff640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2fa0 0x7f223015d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2235649640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22300d1c30 0x7f223015e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 -- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22300d1c30 msgr2=0x7f223015e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22300d1c30 0x7f223015e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 -- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2fa0 msgr2=0x7f223015d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2fa0 0x7f223015d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 -- 172.21.15.59:0/2969228555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2230146810 con 0x7f22300b2270 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f222ffff640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2fa0 0x7f223015d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f2234e48640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22300b2270 0x7f22301462d0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f222402abb0 tx=0x7f2224033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f222d7fa640 1 -- 172.21.15.59:0/2969228555 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224031b50 con 0x7f22300b2270 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.654+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2230151aa0 con 0x7f22300b2270 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.655+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223014fc60 con 0x7f22300b2270 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.655+0000 7f222d7fa640 1 -- 172.21.15.59:0/2969228555 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2224031770 con 0x7f22300b2270 2024-06-29T15:55:21.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.655+0000 7f222d7fa640 1 -- 172.21.15.59:0/2969228555 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224048060 con 0x7f22300b2270 2024-06-29T15:55:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.656+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2210005290 con 0x7f22300b2270 2024-06-29T15:55:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.657+0000 7f222d7fa640 1 -- 172.21.15.59:0/2969228555 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f2224005020 con 0x7f22300b2270 2024-06-29T15:55:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.657+0000 7f222d7fa640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f22140763f0 0x7f2214078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.657+0000 7f222ffff640 1 -- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f22140763f0 msgr2=0x7f2214078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.657+0000 7f222d7fa640 1 -- 172.21.15.59:0/2969228555 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7f22240b8d50 con 0x7f22300b2270 2024-06-29T15:55:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.657+0000 7f222ffff640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f22140763f0 0x7f2214078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:21.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.662+0000 7f222d7fa640 1 -- 172.21.15.59:0/2969228555 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f22240bc030 con 0x7f22300b2270 2024-06-29T15:55:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.858+0000 7f222ffff640 1 -- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f22140763f0 msgr2=0x7f2214078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.858+0000 7f222ffff640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f22140763f0 0x7f2214078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:21.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.864+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2210005850 con 0x7f22300b2270 2024-06-29T15:55:21.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.865+0000 7f222d7fa640 1 -- 172.21.15.59:0/2969228555 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v191) ==== 81+0+956 (secure 0 0 0) 0x7f222403f030 con 0x7f22300b2270 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3420796917 2024-06-30T15:50:20.742173+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3711582444 2024-06-30T15:50:20.742173+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3023327174 2024-06-30T15:50:20.742173+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/1682798961 2024-06-30T15:50:20.742173+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/1682798961 2024-06-30T15:50:20.742173+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1412243249 2024-06-30T15:50:20.742173+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3586261468 2024-06-30T15:39:23.696245+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1851934149 2024-06-30T15:39:23.696245+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1918645711 2024-06-30T15:50:20.742173+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/4098340029 2024-06-30T15:39:23.696245+0000 2024-06-29T15:55:21.865 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2596446141 2024-06-30T15:39:23.696245+0000 2024-06-29T15:55:21.866 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3821515452 2024-06-30T15:39:23.696245+0000 2024-06-29T15:55:21.866 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1052117807 2024-06-30T15:50:20.742173+0000 2024-06-29T15:55:21.866 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2595561638 2024-06-30T15:39:23.696245+0000 2024-06-29T15:55:21.866 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/1270545814 2024-06-30T15:39:23.696245+0000 2024-06-29T15:55:21.866 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/1270545814 2024-06-30T15:39:23.696245+0000 2024-06-29T15:55:21.866 INFO:teuthology.orchestra.run.smithi059.stderr:listed 16 entries 2024-06-29T15:55:21.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f22140763f0 msgr2=0x7f2214078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:21.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f22140763f0 0x7f2214078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22300b2270 msgr2=0x7f22301462d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22300b2270 0x7f22301462d0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f222402abb0 tx=0x7f2224033ba0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 shutdown_connections 2024-06-29T15:55:21.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f22300d1c30 0x7f223015e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f22140763f0 0x7f2214078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300b2fa0 0x7f223015d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 --2- 172.21.15.59:0/2969228555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22300b2270 0x7f22301462d0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 >> 172.21.15.59:0/2969228555 conn(0x7f2230073e10 msgr2=0x7f2230070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 shutdown_connections 2024-06-29T15:55:21.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:21.867+0000 7f22370ac640 1 -- 172.21.15.59:0/2969228555 wait complete. 2024-06-29T15:55:21.878 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:55:21.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:21.879 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:21.911 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T15:55:21.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:21.911 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:55:21.968 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:55:21.968 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:55:21.968 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:21.984 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:55:21.984 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:55:21.984 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:55:21.984 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:22.025 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:55:22.025 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:55:22.025 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:55:22.025 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:22.041 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:55:22.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:22.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.184+0000 7ff31857f640 1 -- 172.21.15.59:0/3599033122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3100b3990 msgr2=0x7ff3100b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.184+0000 7ff31857f640 1 --2- 172.21.15.59:0/3599033122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3100b3990 0x7ff3100b3da0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff304009830 tx=0x7ff30402a510 comp rx=0 tx=0).stop 2024-06-29T15:55:22.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.185+0000 7ff31857f640 1 -- 172.21.15.59:0/3599033122 shutdown_connections 2024-06-29T15:55:22.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.185+0000 7ff31857f640 1 --2- 172.21.15.59:0/3599033122 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3100b73f0 0x7ff3100b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.185+0000 7ff31857f640 1 --2- 172.21.15.59:0/3599033122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3100b3990 0x7ff3100b3da0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.185+0000 7ff31857f640 1 -- 172.21.15.59:0/3599033122 >> 172.21.15.59:0/3599033122 conn(0x7ff3100d3ee0 msgr2=0x7ff3100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.185+0000 7ff31857f640 1 -- 172.21.15.59:0/3599033122 shutdown_connections 2024-06-29T15:55:22.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.185+0000 7ff31857f640 1 -- 172.21.15.59:0/3599033122 wait complete. 2024-06-29T15:55:22.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.186+0000 7ff31857f640 1 Processor -- start 2024-06-29T15:55:22.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.187+0000 7ff31857f640 1 -- start start 2024-06-29T15:55:22.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31857f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100b3990 0x7ff3101f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31857f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3100b73f0 0x7ff3101f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31857f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff31015e290 0x7ff3101e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31857f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff3100b7d40 con 0x7ff3100b3990 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31857f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3100cea30 con 0x7ff31015e290 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31857f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff3100b7ec0 con 0x7ff3100b73f0 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31631b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100b3990 0x7ff3101f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31631b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100b3990 0x7ff3101f27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54782/0 (socket says 172.21.15.59:54782) 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31631b640 1 -- 172.21.15.59:0/345598449 learned_addr learned my addr 172.21.15.59:0/345598449 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff316b1c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff31015e290 0x7ff3101e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff315b1a640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3100b73f0 0x7ff3101f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31631b640 1 -- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3100b73f0 msgr2=0x7ff3101f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31631b640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3100b73f0 0x7ff3101f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31631b640 1 -- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff31015e290 msgr2=0x7ff3101e7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31631b640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff31015e290 0x7ff3101e7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.188+0000 7ff31631b640 1 -- 172.21.15.59:0/345598449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3101e97d0 con 0x7ff3100b3990 2024-06-29T15:55:22.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.189+0000 7ff31631b640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100b3990 0x7ff3101f27c0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c430 tx=0x7ff30c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:22.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.189+0000 7ff302ffd640 1 -- 172.21.15.59:0/345598449 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c011790 con 0x7ff3100b3990 2024-06-29T15:55:22.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.189+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3101e9ab0 con 0x7ff3100b3990 2024-06-29T15:55:22.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.189+0000 7ff302ffd640 1 -- 172.21.15.59:0/345598449 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff30c017070 con 0x7ff3100b3990 2024-06-29T15:55:22.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.189+0000 7ff302ffd640 1 -- 172.21.15.59:0/345598449 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c011ce0 con 0x7ff3100b3990 2024-06-29T15:55:22.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.189+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3100c8050 con 0x7ff3100b3990 2024-06-29T15:55:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.190+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3100b8420 con 0x7ff3100b3990 2024-06-29T15:55:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.191+0000 7ff302ffd640 1 -- 172.21.15.59:0/345598449 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7ff30c01c460 con 0x7ff3100b3990 2024-06-29T15:55:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.191+0000 7ff302ffd640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7ff2f4076420 0x7ff2f4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.191+0000 7ff302ffd640 1 -- 172.21.15.59:0/345598449 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7ff30c097b10 con 0x7ff3100b3990 2024-06-29T15:55:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.191+0000 7ff315b1a640 1 -- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7ff2f4076420 msgr2=0x7ff2f4078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.191+0000 7ff315b1a640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7ff2f4076420 0x7ff2f4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:22.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.196+0000 7ff302ffd640 1 -- 172.21.15.59:0/345598449 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff30c0607a0 con 0x7ff3100b3990 2024-06-29T15:55:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.392+0000 7ff315b1a640 1 -- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7ff2f4076420 msgr2=0x7ff2f4078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.392+0000 7ff315b1a640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7ff2f4076420 0x7ff2f4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:22.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.426+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3101f6b50 con 0x7ff3100b3990 2024-06-29T15:55:22.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.427+0000 7ff302ffd640 1 -- 172.21.15.59:0/345598449 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5235 (secure 0 0 0) 0x7ff30c067e20 con 0x7ff3100b3990 2024-06-29T15:55:22.427 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:22.427 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"btime":"2024-06-29T15:39:15:059125+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8588,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3866646629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3866646629},{"type":"v1","addr":"172.21.15.153:6835","nonce":3866646629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9724,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/933275096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":933275096},{"type":"v1","addr":"172.21.15.59:6835","nonce":933275096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9736,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1230229255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1230229255},{"type":"v1","addr":"172.21.15.153:6837","nonce":1230229255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.430152+0000","modified":"2024-06-29T15:39:14.448659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2399523179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2399523179},{"type":"v1","addr":"172.21.15.59:6834","nonce":2399523179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10401,"qdb_cluster":[10401]},"id":11}]} 2024-06-29T15:55:22.427 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 97 2024-06-29T15:55:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7ff2f4076420 msgr2=0x7ff2f4078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7ff2f4076420 0x7ff2f4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100b3990 msgr2=0x7ff3101f27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100b3990 0x7ff3101f27c0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c430 tx=0x7ff30c00c8f0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 shutdown_connections 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7ff2f4076420 0x7ff2f4078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff31015e290 0x7ff3101e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff3100b73f0 0x7ff3101f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 --2- 172.21.15.59:0/345598449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3100b3990 0x7ff3101f27c0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 >> 172.21.15.59:0/345598449 conn(0x7ff3100d3ee0 msgr2=0x7ff3100ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 shutdown_connections 2024-06-29T15:55:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.429+0000 7ff31857f640 1 -- 172.21.15.59:0/345598449 wait complete. 2024-06-29T15:55:22.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:22.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.559+0000 7f480bf0a640 1 -- 172.21.15.59:0/350679124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f480406e3a0 msgr2=0x7f480415f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.559+0000 7f480bf0a640 1 --2- 172.21.15.59:0/350679124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f480406e3a0 0x7f480415f730 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f48040b5d10 tx=0x7f480002a560 comp rx=0 tx=0).stop 2024-06-29T15:55:22.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.559+0000 7f480bf0a640 1 -- 172.21.15.59:0/350679124 shutdown_connections 2024-06-29T15:55:22.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.559+0000 7f480bf0a640 1 --2- 172.21.15.59:0/350679124 >> v2:172.21.15.153:3300/0 conn(0x7f48040cfa00 0x7f480415c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.559+0000 7f480bf0a640 1 --2- 172.21.15.59:0/350679124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f480406e3a0 0x7f480415f730 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.559+0000 7f480bf0a640 1 -- 172.21.15.59:0/350679124 >> 172.21.15.59:0/350679124 conn(0x7f48040b2ab0 msgr2=0x7f48040b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.559+0000 7f480bf0a640 1 -- 172.21.15.59:0/350679124 shutdown_connections 2024-06-29T15:55:22.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.559+0000 7f480bf0a640 1 -- 172.21.15.59:0/350679124 wait complete. 2024-06-29T15:55:22.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.562+0000 7f480bf0a640 1 Processor -- start 2024-06-29T15:55:22.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.563+0000 7f480bf0a640 1 -- start start 2024-06-29T15:55:22.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f480bf0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040cfa00 0x7f4804143b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f480bf0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4804144970 0x7f48040b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f480bf0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040b1ce0 0x7f4804155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f480bf0a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f480415d1f0 con 0x7f4804144970 2024-06-29T15:55:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f480bf0a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f480415d070 con 0x7f48040cfa00 2024-06-29T15:55:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f480bf0a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f480415d370 con 0x7f48040b1ce0 2024-06-29T15:55:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f48094a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4804144970 0x7f48040b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f48094a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4804144970 0x7f48040b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54796/0 (socket says 172.21.15.59:54796) 2024-06-29T15:55:22.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f48094a5640 1 -- 172.21.15.59:0/2074214438 learned_addr learned my addr 172.21.15.59:0/2074214438 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:22.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f4809ca6640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040cfa00 0x7f4804143b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f480a4a7640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040b1ce0 0x7f4804155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f48094a5640 1 -- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040b1ce0 msgr2=0x7f4804155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f48094a5640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040b1ce0 0x7f4804155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f48094a5640 1 -- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040cfa00 msgr2=0x7f4804143b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f48094a5640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040cfa00 0x7f4804143b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.565+0000 7f48094a5640 1 -- 172.21.15.59:0/2074214438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48040b20f0 con 0x7f4804144970 2024-06-29T15:55:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.566+0000 7f48094a5640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4804144970 0x7f48040b17a0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f4800034040 tx=0x7f48000038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.566+0000 7f47f67fc640 1 -- 172.21.15.59:0/2074214438 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480003e030 con 0x7f4804144970 2024-06-29T15:55:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.566+0000 7f47f67fc640 1 -- 172.21.15.59:0/2074214438 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4800003db0 con 0x7f4804144970 2024-06-29T15:55:22.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.566+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48041424d0 con 0x7f4804144970 2024-06-29T15:55:22.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.566+0000 7f47f67fc640 1 -- 172.21.15.59:0/2074214438 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800035040 con 0x7f4804144970 2024-06-29T15:55:22.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.566+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4804142a90 con 0x7f4804144970 2024-06-29T15:55:22.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.567+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f480415d890 con 0x7f4804144970 2024-06-29T15:55:22.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.568+0000 7f47f67fc640 1 -- 172.21.15.59:0/2074214438 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f4800041030 con 0x7f4804144970 2024-06-29T15:55:22.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.568+0000 7f47f67fc640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f47e80763f0 0x7f47e8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.568+0000 7f47f67fc640 1 -- 172.21.15.59:0/2074214438 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7f48000b8820 con 0x7f4804144970 2024-06-29T15:55:22.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.568+0000 7f4809ca6640 1 -- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f47e80763f0 msgr2=0x7f47e8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:22.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.568+0000 7f4809ca6640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f47e80763f0 0x7f47e8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:22.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.573+0000 7f47f67fc640 1 -- 172.21.15.59:0/2074214438 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f48040b20f0 con 0x7f4804144970 2024-06-29T15:55:22.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.765+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48040cf7e0 con 0x7f4804144970 2024-06-29T15:55:22.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.766+0000 7f47f67fc640 1 -- 172.21.15.59:0/2074214438 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) ==== 74+0+17959 (secure 0 0 0) 0x7f4800081420 con 0x7f4804144970 2024-06-29T15:55:22.766 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:22.767 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":191,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:50:20.742223+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.834000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.377444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3420796917":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3711582444":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3023327174":"2024-06-30T15:50:20.742173+0000","172.21.15.153:6833/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:6832/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/1412243249":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3586261468":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1851934149":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1918645711":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/4098340029":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/2596446141":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/3821515452":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1052117807":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/2595561638":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6833/1270545814":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6832/1270545814":"2024-06-30T15:39:23.696245+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f47e80763f0 msgr2=0x7f47e8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f47e80763f0 0x7f47e8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4804144970 msgr2=0x7f48040b17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4804144970 0x7f48040b17a0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f4800034040 tx=0x7f48000038a0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 shutdown_connections 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040b1ce0 0x7f4804155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4804144970 0x7f48040b17a0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f47e80763f0 0x7f47e8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 --2- 172.21.15.59:0/2074214438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040cfa00 0x7f4804143b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 >> 172.21.15.59:0/2074214438 conn(0x7f48040b2ab0 msgr2=0x7f4804140ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 shutdown_connections 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.769+0000 7f480bf0a640 1 -- 172.21.15.59:0/2074214438 wait complete. 2024-06-29T15:55:22.784 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:39:09.834000+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:39:10.377444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:55:22.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:55:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.906+0000 7fe4876df640 1 -- 172.21.15.59:0/3383746093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800d1a60 msgr2=0x7fe4800b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.906+0000 7fe4876df640 1 --2- 172.21.15.59:0/3383746093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800d1a60 0x7fe4800b5b30 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fe478005fd0 tx=0x7fe47802aa90 comp rx=0 tx=0).stop 2024-06-29T15:55:22.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.906+0000 7fe4876df640 1 -- 172.21.15.59:0/3383746093 shutdown_connections 2024-06-29T15:55:22.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.906+0000 7fe4876df640 1 --2- 172.21.15.59:0/3383746093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800d1a60 0x7fe4800b5b30 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.906+0000 7fe4876df640 1 --2- 172.21.15.59:0/3383746093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4800b2fa0 0x7fe4800d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.906+0000 7fe4876df640 1 -- 172.21.15.59:0/3383746093 >> 172.21.15.59:0/3383746093 conn(0x7fe480073e10 msgr2=0x7fe480074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.906+0000 7fe4876df640 1 -- 172.21.15.59:0/3383746093 shutdown_connections 2024-06-29T15:55:22.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.906+0000 7fe4876df640 1 -- 172.21.15.59:0/3383746093 wait complete. 2024-06-29T15:55:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.908+0000 7fe4876df640 1 Processor -- start 2024-06-29T15:55:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.908+0000 7fe4876df640 1 -- start start 2024-06-29T15:55:22.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.909+0000 7fe4876df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4800b2270 0x7fe480071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.909+0000 7fe4876df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800b2fa0 0x7fe480150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.909+0000 7fe4876df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4800d1a60 0x7fe48015ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.909+0000 7fe4876df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe4800b6400 con 0x7fe4800b2fa0 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.909+0000 7fe4876df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4800b6280 con 0x7fe4800b2270 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.909+0000 7fe4876df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe4800b6580 con 0x7fe4800d1a60 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800b2fa0 0x7fe480150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800b2fa0 0x7fe480150720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48592/0 (socket says 172.21.15.59:48592) 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 -- 172.21.15.59:0/679118565 learned_addr learned my addr 172.21.15.59:0/679118565 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe485c7c640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4800d1a60 0x7fe48015ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe48547b640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4800b2270 0x7fe480071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 -- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4800d1a60 msgr2=0x7fe48015ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4800d1a60 0x7fe48015ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 -- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4800b2270 msgr2=0x7fe480071510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4800b2270 0x7fe480071510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 -- 172.21.15.59:0/679118565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4801f4eb0 con 0x7fe4800b2fa0 2024-06-29T15:55:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe484c7a640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800b2fa0 0x7fe480150720 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fe47000e7c0 tx=0x7fe47000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe475ffb640 1 -- 172.21.15.59:0/679118565 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47000ccc0 con 0x7fe4800b2fa0 2024-06-29T15:55:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe475ffb640 1 -- 172.21.15.59:0/679118565 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe470010430 con 0x7fe4800b2fa0 2024-06-29T15:55:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe475ffb640 1 -- 172.21.15.59:0/679118565 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470010860 con 0x7fe4800b2fa0 2024-06-29T15:55:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.910+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4801f5360 con 0x7fe4800b2fa0 2024-06-29T15:55:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.911+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4801f5920 con 0x7fe4800b2fa0 2024-06-29T15:55:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.912+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4800b6aa0 con 0x7fe4800b2fa0 2024-06-29T15:55:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.912+0000 7fe475ffb640 1 -- 172.21.15.59:0/679118565 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7fe470010a00 con 0x7fe4800b2fa0 2024-06-29T15:55:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.913+0000 7fe475ffb640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe464076420 0x7fe464078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.913+0000 7fe475ffb640 1 -- 172.21.15.59:0/679118565 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7fe4700987e0 con 0x7fe4800b2fa0 2024-06-29T15:55:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.913+0000 7fe48547b640 1 -- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe464076420 msgr2=0x7fe464078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.913+0000 7fe48547b640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe464076420 0x7fe464078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:22.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:22.917+0000 7fe475ffb640 1 -- 172.21.15.59:0/679118565 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe4700613d0 con 0x7fe4800b2fa0 2024-06-29T15:55:23.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.114+0000 7fe48547b640 1 -- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe464076420 msgr2=0x7fe464078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:23.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.114+0000 7fe48547b640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe464076420 0x7fe464078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:23.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.207+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe4801f5080 con 0x7fe4800b2fa0 2024-06-29T15:55:23.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.208+0000 7fe475ffb640 1 -- 172.21.15.59:0/679118565 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v192) ==== 93+0+10 (secure 0 0 0) 0x7fe4700656d0 con 0x7fe4800b2fa0 2024-06-29T15:55:23.208 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:55:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe464076420 msgr2=0x7fe464078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe464076420 0x7fe464078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800b2fa0 msgr2=0x7fe480150720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800b2fa0 0x7fe480150720 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fe47000e7c0 tx=0x7fe47000ec80 comp rx=0 tx=0).stop 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 shutdown_connections 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4800d1a60 0x7fe48015ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4800b2fa0 0x7fe480150720 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7fe464076420 0x7fe464078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 --2- 172.21.15.59:0/679118565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4800b2270 0x7fe480071510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 >> 172.21.15.59:0/679118565 conn(0x7fe480073e10 msgr2=0x7fe4800d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 shutdown_connections 2024-06-29T15:55:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.210+0000 7fe4876df640 1 -- 172.21.15.59:0/679118565 wait complete. 2024-06-29T15:55:23.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:55:23.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.388+0000 7f2d159f5640 1 -- 172.21.15.59:0/1232649795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc003a70 msgr2=0x7f2cfc003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.388+0000 7f2d159f5640 1 --2- 172.21.15.59:0/1232649795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc003a70 0x7f2cfc003ee0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f2d00002230 tx=0x7f2cfc005830 comp rx=0 tx=0).stop 2024-06-29T15:55:23.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.388+0000 7f2d0f7fe640 1 -- 172.21.15.59:0/1232649795 reap_dead start 2024-06-29T15:55:23.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.388+0000 7f2d159f5640 1 -- 172.21.15.59:0/1232649795 shutdown_connections 2024-06-29T15:55:23.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.388+0000 7f2d159f5640 1 -- 172.21.15.59:0/1232649795 >> 172.21.15.59:0/1232649795 conn(0x7f2d10073e10 msgr2=0x7f2d10074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.388+0000 7f2d159f5640 1 -- 172.21.15.59:0/1232649795 shutdown_connections 2024-06-29T15:55:23.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.388+0000 7f2d159f5640 1 -- 172.21.15.59:0/1232649795 wait complete. 2024-06-29T15:55:23.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.390+0000 7f2d159f5640 1 Processor -- start 2024-06-29T15:55:23.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.390+0000 7f2d159f5640 1 -- start start 2024-06-29T15:55:23.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d159f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cfc003a70 0x7f2d1006f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d159f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d101544d0 0x7f2d100ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d159f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015d960 0x7f2d1015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d159f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2d100b5ec0 con 0x7f2d101544d0 2024-06-29T15:55:23.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d159f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d100b5d40 con 0x7f2d1015d960 2024-06-29T15:55:23.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d159f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2d100b6040 con 0x7f2cfc003a70 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d101544d0 0x7f2d100ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d101544d0 0x7f2d100ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48620/0 (socket says 172.21.15.59:48620) 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0e7fc640 1 -- 172.21.15.59:0/1988875126 learned_addr learned my addr 172.21.15.59:0/1988875126 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0effd640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cfc003a70 0x7f2d1006f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0f7fe640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015d960 0x7f2d1015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0e7fc640 1 -- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cfc003a70 msgr2=0x7f2d1006f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0e7fc640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cfc003a70 0x7f2d1006f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0e7fc640 1 -- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015d960 msgr2=0x7f2d1015dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0e7fc640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015d960 0x7f2d1015dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0e7fc640 1 -- 172.21.15.59:0/1988875126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d100addf0 con 0x7f2d101544d0 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.392+0000 7f2d0effd640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cfc003a70 0x7f2d1006f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:23.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.393+0000 7f2d0e7fc640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d101544d0 0x7f2d100ad8b0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8011b20 tx=0x7f2cf800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:23.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.393+0000 7f2d07fff640 1 -- 172.21.15.59:0/1988875126 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf801e070 con 0x7f2d101544d0 2024-06-29T15:55:23.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.393+0000 7f2d07fff640 1 -- 172.21.15.59:0/1988875126 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cf800fd40 con 0x7f2d101544d0 2024-06-29T15:55:23.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.393+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d100ae380 con 0x7f2d101544d0 2024-06-29T15:55:23.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.393+0000 7f2d07fff640 1 -- 172.21.15.59:0/1988875126 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8019650 con 0x7f2d101544d0 2024-06-29T15:55:23.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.393+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d100ae940 con 0x7f2d101544d0 2024-06-29T15:55:23.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.394+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d100d1520 con 0x7f2d101544d0 2024-06-29T15:55:23.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.395+0000 7f2d07fff640 1 -- 172.21.15.59:0/1988875126 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f2cf8021070 con 0x7f2d101544d0 2024-06-29T15:55:23.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.396+0000 7f2d07fff640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2cf00763f0 0x7f2cf0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.396+0000 7f2d0effd640 1 -- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2cf00763f0 msgr2=0x7f2cf0078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2924556144 2024-06-29T15:55:23.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.396+0000 7f2d07fff640 1 -- 172.21.15.59:0/1988875126 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 6873+0+0 (secure 0 0 0) 0x7f2cf8098750 con 0x7f2d101544d0 2024-06-29T15:55:23.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.396+0000 7f2d0effd640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2cf00763f0 0x7f2cf0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:23.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.401+0000 7f2d07fff640 1 -- 172.21.15.59:0/1988875126 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2cf8061330 con 0x7f2d101544d0 2024-06-29T15:55:23.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.593+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2d101548e0 con 0x7f2d101544d0 2024-06-29T15:55:23.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.594+0000 7f2d07fff640 1 -- 172.21.15.59:0/1988875126 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v192) ==== 104+0+11 (secure 0 0 0) 0x7f2cf8065630 con 0x7f2d101544d0 2024-06-29T15:55:23.594 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:55:23.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2cf00763f0 msgr2=0x7f2cf0078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:23.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2cf00763f0 0x7f2cf0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d101544d0 msgr2=0x7f2d100ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d101544d0 0x7f2d100ad8b0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8011b20 tx=0x7f2cf800f3c0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 shutdown_connections 2024-06-29T15:55:23.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:6832/2924556144,v1:172.21.15.153:6833/2924556144] conn(0x7f2cf00763f0 0x7f2cf0078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d1015d960 0x7f2d1015dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d101544d0 0x7f2d100ad8b0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 --2- 172.21.15.59:0/1988875126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2cfc003a70 0x7f2d1006f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 >> 172.21.15.59:0/1988875126 conn(0x7f2d10073e10 msgr2=0x7f2d10158170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 shutdown_connections 2024-06-29T15:55:23.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.596+0000 7f2d159f5640 1 -- 172.21.15.59:0/1988875126 wait complete. 2024-06-29T15:55:23.607 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:55:23.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952c99a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952c99a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952c99a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952c99a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952c99a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952a195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952a195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.712 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952a195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.712 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952a195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.712 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:23.710+0000 7f952a195640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.736+0000 7ff1df903640 1 -- 172.21.15.59:0/2936636931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1c4004800 msgr2=0x7ff1c4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.736+0000 7ff1df903640 1 --2- 172.21.15.59:0/2936636931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1c4004800 0x7ff1c4004c10 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80b55e0 tx=0x7ff1c4005830 comp rx=0 tx=0).stop 2024-06-29T15:55:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.736+0000 7ff1df903640 1 -- 172.21.15.59:0/2936636931 shutdown_connections 2024-06-29T15:55:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.736+0000 7ff1df903640 1 --2- 172.21.15.59:0/2936636931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1c4004800 0x7ff1c4004c10 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.736+0000 7ff1df903640 1 -- 172.21.15.59:0/2936636931 >> 172.21.15.59:0/2936636931 conn(0x7ff1d80d3ee0 msgr2=0x7ff1d80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.736+0000 7ff1df903640 1 -- 172.21.15.59:0/2936636931 shutdown_connections 2024-06-29T15:55:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.736+0000 7ff1df903640 1 -- 172.21.15.59:0/2936636931 wait complete. 2024-06-29T15:55:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.739+0000 7ff1df903640 1 Processor -- start 2024-06-29T15:55:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.739+0000 7ff1df903640 1 -- start start 2024-06-29T15:55:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1df903640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1c4004800 0x7ff1d8154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1df903640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1d8153f10 0x7ff1d81e8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1df903640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1d81e8ef0 0x7ff1d81e9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1df903640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d80ce4a0 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1df903640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d80ce320 con 0x7ff1c4004800 2024-06-29T15:55:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1df903640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d80b3990 con 0x7ff1d8153f10 2024-06-29T15:55:23.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1d81e8ef0 0x7ff1d81e9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1d81e8ef0 0x7ff1d81e9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48626/0 (socket says 172.21.15.59:48626) 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 -- 172.21.15.59:0/3615822647 learned_addr learned my addr 172.21.15.59:0/3615822647 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1dce9e640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1d8153f10 0x7ff1d81e8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1dd69f640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1c4004800 0x7ff1d8154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 -- 172.21.15.59:0/3615822647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1d8153f10 msgr2=0x7ff1d81e8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1d8153f10 0x7ff1d81e8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 -- 172.21.15.59:0/3615822647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1c4004800 msgr2=0x7ff1d8154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1c4004800 0x7ff1d8154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 -- 172.21.15.59:0/3615822647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d815eeb0 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.742+0000 7ff1ddea0640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1d81e8ef0 0x7ff1d81e9300 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80b3550 tx=0x7ff1d0006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.743+0000 7ff1c9ffb640 1 -- 172.21.15.59:0/3615822647 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000fa80 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.743+0000 7ff1c9ffb640 1 -- 172.21.15.59:0/3615822647 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1d0018070 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.743+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d81e9840 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.743+0000 7ff1c9ffb640 1 -- 172.21.15.59:0/3615822647 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0011430 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.743+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d8155480 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.744+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d80b3e60 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.745+0000 7ff1c9ffb640 1 -- 172.21.15.59:0/3615822647 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 114) ==== 97912+0+0 (secure 0 0 0) 0x7ff1d00115d0 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.745+0000 7ff1c9ffb640 1 -- 172.21.15.59:0/3615822647 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 7189+0+0 (secure 0 0 0) 0x7ff1d0097ee0 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.750+0000 7ff1c9ffb640 1 -- 172.21.15.59:0/3615822647 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff1d81e9840 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.942+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff1d81599e0 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.943+0000 7ff1c9ffb640 1 -- 172.21.15.59:0/3615822647 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v193) ==== 100+0+11 (secure 0 0 0) 0x7ff1d0061630 con 0x7ff1d81e8ef0 2024-06-29T15:55:23.943 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:55:23.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1d81e8ef0 msgr2=0x7ff1d81e9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1d81e8ef0 0x7ff1d81e9300 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80b3550 tx=0x7ff1d0006660 comp rx=0 tx=0).stop 2024-06-29T15:55:23.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 shutdown_connections 2024-06-29T15:55:23.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1d81e8ef0 0x7ff1d81e9300 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1d8153f10 0x7ff1d81e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 --2- 172.21.15.59:0/3615822647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1c4004800 0x7ff1d8154e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 >> 172.21.15.59:0/3615822647 conn(0x7ff1d80d3ee0 msgr2=0x7ff1d80b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 shutdown_connections 2024-06-29T15:55:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:23.945+0000 7ff1df903640 1 -- 172.21.15.59:0/3615822647 wait complete. 2024-06-29T15:55:23.957 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:24.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.075+0000 7efdef809640 1 -- 172.21.15.59:0/3497589139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde80b3740 msgr2=0x7efde80b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.075+0000 7efdef809640 1 --2- 172.21.15.59:0/3497589139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde80b3740 0x7efde80b3b50 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efde4009830 tx=0x7efde402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.075+0000 7efdef809640 1 -- 172.21.15.59:0/3497589139 shutdown_connections 2024-06-29T15:55:24.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.075+0000 7efdef809640 1 --2- 172.21.15.59:0/3497589139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde80b3740 0x7efde80b3b50 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.075+0000 7efdef809640 1 -- 172.21.15.59:0/3497589139 >> 172.21.15.59:0/3497589139 conn(0x7efde806c950 msgr2=0x7efde8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.075+0000 7efdef809640 1 -- 172.21.15.59:0/3497589139 shutdown_connections 2024-06-29T15:55:24.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.075+0000 7efdef809640 1 -- 172.21.15.59:0/3497589139 wait complete. 2024-06-29T15:55:24.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.077+0000 7efdef809640 1 Processor -- start 2024-06-29T15:55:24.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.077+0000 7efdef809640 1 -- start start 2024-06-29T15:55:24.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efdef809640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde80afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efdef809640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde815c2f0 0x7efde80ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efdef809640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde8137290 0x7efde815ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efdef809640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efde80b6320 con 0x7efde80b3740 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efdef809640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efde80b61a0 con 0x7efde8137290 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efdef809640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efde80b64a0 con 0x7efde815c2f0 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efded5a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde80afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efded5a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde80afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48630/0 (socket says 172.21.15.59:48630) 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.078+0000 7efded5a5640 1 -- 172.21.15.59:0/4275009880 learned_addr learned my addr 172.21.15.59:0/4275009880 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efdedda6640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde8137290 0x7efde815ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efdecda4640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde815c2f0 0x7efde80ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efded5a5640 1 -- 172.21.15.59:0/4275009880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde815c2f0 msgr2=0x7efde80ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efded5a5640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde815c2f0 0x7efde80ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efded5a5640 1 -- 172.21.15.59:0/4275009880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde8137290 msgr2=0x7efde815ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efded5a5640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde8137290 0x7efde815ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efded5a5640 1 -- 172.21.15.59:0/4275009880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde80b0600 con 0x7efde80b3740 2024-06-29T15:55:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efded5a5640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde80afdb0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7efddc00c7d0 tx=0x7efddc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efdd9ffb640 1 -- 172.21.15.59:0/4275009880 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc017070 con 0x7efde80b3740 2024-06-29T15:55:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efdd9ffb640 1 -- 172.21.15.59:0/4275009880 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efddc005070 con 0x7efde80b3740 2024-06-29T15:55:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde80b08e0 con 0x7efde80b3740 2024-06-29T15:55:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efdd9ffb640 1 -- 172.21.15.59:0/4275009880 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc0054a0 con 0x7efde80b3740 2024-06-29T15:55:24.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.079+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde80aadf0 con 0x7efde80b3740 2024-06-29T15:55:24.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.081+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde80b2970 con 0x7efde80b3740 2024-06-29T15:55:24.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.081+0000 7efdd9ffb640 1 -- 172.21.15.59:0/4275009880 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 114) ==== 97912+0+0 (secure 0 0 0) 0x7efddc01a070 con 0x7efde80b3740 2024-06-29T15:55:24.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.082+0000 7efdd9ffb640 1 -- 172.21.15.59:0/4275009880 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 7189+0+0 (secure 0 0 0) 0x7efddc096fd0 con 0x7efde80b3740 2024-06-29T15:55:24.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.086+0000 7efdd9ffb640 1 -- 172.21.15.59:0/4275009880 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efddc060720 con 0x7efde80b3740 2024-06-29T15:55:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.316+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efde81458a0 con 0x7efde80b3740 2024-06-29T15:55:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.317+0000 7efdd9ffb640 1 -- 172.21.15.59:0/4275009880 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5235 (secure 0 0 0) 0x7efddc060020 con 0x7efde80b3740 2024-06-29T15:55:24.317 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:24.317 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"btime":"2024-06-29T15:39:15:059125+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8588,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3866646629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3866646629},{"type":"v1","addr":"172.21.15.153:6835","nonce":3866646629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9724,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/933275096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":933275096},{"type":"v1","addr":"172.21.15.59:6835","nonce":933275096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9736,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1230229255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1230229255},{"type":"v1","addr":"172.21.15.153:6837","nonce":1230229255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.430152+0000","modified":"2024-06-29T15:39:14.448659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2399523179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2399523179},{"type":"v1","addr":"172.21.15.59:6834","nonce":2399523179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10401,"qdb_cluster":[10401]},"id":11}]} 2024-06-29T15:55:24.317 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 97 2024-06-29T15:55:24.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 msgr2=0x7efde80afdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde80afdb0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7efddc00c7d0 tx=0x7efddc00cc90 comp rx=0 tx=0).stop 2024-06-29T15:55:24.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 shutdown_connections 2024-06-29T15:55:24.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde8137290 0x7efde815ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde815c2f0 0x7efde80ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 --2- 172.21.15.59:0/4275009880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde80afdb0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 >> 172.21.15.59:0/4275009880 conn(0x7efde806c950 msgr2=0x7efde8160140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 shutdown_connections 2024-06-29T15:55:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.319+0000 7efdef809640 1 -- 172.21.15.59:0/4275009880 wait complete. 2024-06-29T15:55:24.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 -- 172.21.15.59:0/2731265673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b2040 msgr2=0x7f2e5006e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 --2- 172.21.15.59:0/2731265673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b2040 0x7f2e5006e300 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f2e500b6a80 tx=0x7f2e4402a560 comp rx=0 tx=0).stop 2024-06-29T15:55:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 -- 172.21.15.59:0/2731265673 shutdown_connections 2024-06-29T15:55:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 --2- 172.21.15.59:0/2731265673 >> v2:172.21.15.153:3300/0 conn(0x7f2e5006e840 0x7f2e5015e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 --2- 172.21.15.59:0/2731265673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e500b2040 0x7f2e5006e300 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 --2- 172.21.15.59:0/2731265673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e50144470 0x7f2e500b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 -- 172.21.15.59:0/2731265673 >> 172.21.15.59:0/2731265673 conn(0x7f2e500b06e0 msgr2=0x7f2e500b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 -- 172.21.15.59:0/2731265673 shutdown_connections 2024-06-29T15:55:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.448+0000 7f2e56931640 1 -- 172.21.15.59:0/2731265673 wait complete. 2024-06-29T15:55:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.450+0000 7f2e56931640 1 Processor -- start 2024-06-29T15:55:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.450+0000 7f2e56931640 1 -- start start 2024-06-29T15:55:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e56931640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e5006e840 0x7f2e501f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e56931640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e500b2040 0x7f2e500af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e56931640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e50144470 0x7f2e501f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e56931640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5015ee30 con 0x7f2e5006e840 2024-06-29T15:55:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e56931640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5015ecb0 con 0x7f2e500b2040 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e56931640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5015efb0 con 0x7f2e50144470 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e5006e840 0x7f2e501f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e4ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e5006e840 0x7f2e501f7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48642/0 (socket says 172.21.15.59:48642) 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e4ffff640 1 -- 172.21.15.59:0/4112918177 learned_addr learned my addr 172.21.15.59:0/4112918177 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e4f7fe640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e500b2040 0x7f2e500af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.451+0000 7f2e54ece640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e50144470 0x7f2e501f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4ffff640 1 -- 172.21.15.59:0/4112918177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e50144470 msgr2=0x7f2e501f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4ffff640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e50144470 0x7f2e501f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4ffff640 1 -- 172.21.15.59:0/4112918177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e500b2040 msgr2=0x7f2e500af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4ffff640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e500b2040 0x7f2e500af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4ffff640 1 -- 172.21.15.59:0/4112918177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e50158f10 con 0x7f2e5006e840 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4ffff640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e5006e840 0x7f2e501f7dc0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000d780 tx=0x7f2e4000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4cff9640 1 -- 172.21.15.59:0/4112918177 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40017070 con 0x7f2e5006e840 2024-06-29T15:55:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4cff9640 1 -- 172.21.15.59:0/4112918177 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e40004310 con 0x7f2e5006e840 2024-06-29T15:55:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e500b0050 con 0x7f2e5006e840 2024-06-29T15:55:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e4cff9640 1 -- 172.21.15.59:0/4112918177 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40005280 con 0x7f2e5006e840 2024-06-29T15:55:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.452+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e501591d0 con 0x7f2e5006e840 2024-06-29T15:55:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.453+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e5015c320 con 0x7f2e5006e840 2024-06-29T15:55:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.454+0000 7f2e4cff9640 1 -- 172.21.15.59:0/4112918177 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 114) ==== 97912+0+0 (secure 0 0 0) 0x7f2e40005420 con 0x7f2e5006e840 2024-06-29T15:55:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.454+0000 7f2e4cff9640 1 -- 172.21.15.59:0/4112918177 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 7189+0+0 (secure 0 0 0) 0x7f2e40097ab0 con 0x7f2e5006e840 2024-06-29T15:55:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.459+0000 7f2e4cff9640 1 -- 172.21.15.59:0/4112918177 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2e5015c320 con 0x7f2e5006e840 2024-06-29T15:55:24.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.651+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e5015f470 con 0x7f2e5006e840 2024-06-29T15:55:24.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.652+0000 7f2e4cff9640 1 -- 172.21.15.59:0/4112918177 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) ==== 74+0+18537 (secure 0 0 0) 0x7f2e40069020 con 0x7f2e5006e840 2024-06-29T15:55:24.652 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:24.653 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":193,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:55:23.652664+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.834000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.377444+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1343183333":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6833/2924556144":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6832/2924556144":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/2895806264":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/3420796917":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3711582444":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/1469314326":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/3023327174":"2024-06-30T15:50:20.742173+0000","172.21.15.153:6833/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/2605088182":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/323020462":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6832/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/1412243249":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3586261468":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1851934149":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/771497135":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/1918645711":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/4098340029":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/2596446141":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/3821515452":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1052117807":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/2595561638":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6833/1270545814":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6832/1270545814":"2024-06-30T15:39:23.696245+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:24.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e5006e840 msgr2=0x7f2e501f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e5006e840 0x7f2e501f7dc0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000d780 tx=0x7f2e4000db80 comp rx=0 tx=0).stop 2024-06-29T15:55:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 shutdown_connections 2024-06-29T15:55:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e50144470 0x7f2e501f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e500b2040 0x7f2e500af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 --2- 172.21.15.59:0/4112918177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e5006e840 0x7f2e501f7dc0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 >> 172.21.15.59:0/4112918177 conn(0x7f2e500b06e0 msgr2=0x7f2e5015cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 shutdown_connections 2024-06-29T15:55:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.654+0000 7f2e56931640 1 -- 172.21.15.59:0/4112918177 wait complete. 2024-06-29T15:55:24.668 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:55:24.669 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:24.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.786+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/3541109343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680d2be0 msgr2=0x7ffa680b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.786+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/3541109343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680d2be0 0x7ffa680b3b90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009be0 tx=0x7ffa6002a840 comp rx=0 tx=0).stop 2024-06-29T15:55:24.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.786+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/3541109343 shutdown_connections 2024-06-29T15:55:24.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.786+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/3541109343 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680d2be0 0x7ffa680b3b90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.786+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/3541109343 >> v2:172.21.15.153:3300/0 conn(0x7ffa680cefb0 0x7ffa680d26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.786+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/3541109343 >> 172.21.15.59:0/3541109343 conn(0x7ffa680d3b20 msgr2=0x7ffa680d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.786+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/3541109343 shutdown_connections 2024-06-29T15:55:24.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.786+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/3541109343 wait complete. 2024-06-29T15:55:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.788+0000 7ffa6f5f5640 1 Processor -- start 2024-06-29T15:55:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.788+0000 7ffa6f5f5640 1 -- start start 2024-06-29T15:55:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.789+0000 7ffa6f5f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa680b2980 0x7ffa680c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.789+0000 7ffa6f5f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680cefb0 0x7ffa681e9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.789+0000 7ffa6f5f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa680d2be0 0x7ffa681e9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.789+0000 7ffa6f5f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ffa680b45a0 con 0x7ffa680d2be0 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.789+0000 7ffa6f5f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa680b4420 con 0x7ffa680b2980 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.789+0000 7ffa6f5f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffa680b4720 con 0x7ffa680cefb0 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6db92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa680d2be0 0x7ffa681e9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6db92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa680d2be0 0x7ffa681e9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48654/0 (socket says 172.21.15.59:48654) 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6db92640 1 -- 172.21.15.59:0/1623755721 learned_addr learned my addr 172.21.15.59:0/1623755721 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6d391640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa680b2980 0x7ffa680c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6cb90640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680cefb0 0x7ffa681e9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6d391640 1 -- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680cefb0 msgr2=0x7ffa681e9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6d391640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680cefb0 0x7ffa681e9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6d391640 1 -- 172.21.15.59:0/1623755721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa680d2be0 msgr2=0x7ffa681e9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6d391640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa680d2be0 0x7ffa681e9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6d391640 1 -- 172.21.15.59:0/1623755721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa681e89a0 con 0x7ffa680b2980 2024-06-29T15:55:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6cb90640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680cefb0 0x7ffa681e9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6db92640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa680d2be0 0x7ffa681e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6d391640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa680b2980 0x7ffa680c92b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400c3f0 tx=0x7ffa6400c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa5dffb640 1 -- 172.21.15.59:0/1623755721 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64011790 con 0x7ffa680b2980 2024-06-29T15:55:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa5dffb640 1 -- 172.21.15.59:0/1623755721 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa64017070 con 0x7ffa680b2980 2024-06-29T15:55:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa681e8c80 con 0x7ffa680b2980 2024-06-29T15:55:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.790+0000 7ffa5dffb640 1 -- 172.21.15.59:0/1623755721 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64011cc0 con 0x7ffa680b2980 2024-06-29T15:55:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.791+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa681e9240 con 0x7ffa680b2980 2024-06-29T15:55:24.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.792+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa680b4c40 con 0x7ffa680b2980 2024-06-29T15:55:24.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.792+0000 7ffa5dffb640 1 -- 172.21.15.59:0/1623755721 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 115) ==== 98039+0+0 (secure 0 0 0) 0x7ffa6401c460 con 0x7ffa680b2980 2024-06-29T15:55:24.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.793+0000 7ffa5dffb640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ffa4c076150 0x7ffa4c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.793+0000 7ffa5dffb640 1 -- 172.21.15.59:0/1623755721 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 7189+0+0 (secure 0 0 0) 0x7ffa64098030 con 0x7ffa680b2980 2024-06-29T15:55:24.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.793+0000 7ffa6cb90640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ffa4c076150 0x7ffa4c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.794+0000 7ffa6cb90640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ffa4c076150 0x7ffa4c078660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffa680b8d70 tx=0x7ffa58005f90 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:24.798+0000 7ffa5dffb640 1 -- 172.21.15.59:0/1623755721 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa680d2ff0 con 0x7ffa680b2980 2024-06-29T15:55:25.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.028+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa680b8da0 con 0x7ffa680b2980 2024-06-29T15:55:25.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.029+0000 7ffa5dffb640 1 -- 172.21.15.59:0/1623755721 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5235 (secure 0 0 0) 0x7ffa64061170 con 0x7ffa680b2980 2024-06-29T15:55:25.030 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:25.030 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"btime":"2024-06-29T15:39:15:059125+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8588,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3866646629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3866646629},{"type":"v1","addr":"172.21.15.153:6835","nonce":3866646629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9724,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/933275096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":933275096},{"type":"v1","addr":"172.21.15.59:6835","nonce":933275096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9736,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1230229255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1230229255},{"type":"v1","addr":"172.21.15.153:6837","nonce":1230229255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.430152+0000","modified":"2024-06-29T15:39:14.448659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2399523179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2399523179},{"type":"v1","addr":"172.21.15.59:6834","nonce":2399523179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10401,"qdb_cluster":[10401]},"id":11}]} 2024-06-29T15:55:25.030 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 97 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ffa4c076150 msgr2=0x7ffa4c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ffa4c076150 0x7ffa4c078660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffa680b8d70 tx=0x7ffa58005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa680b2980 msgr2=0x7ffa680c92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa680b2980 0x7ffa680c92b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400c3f0 tx=0x7ffa6400c8b0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 shutdown_connections 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ffa4c076150 0x7ffa4c078660 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa680d2be0 0x7ffa681e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa680cefb0 0x7ffa681e9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 --2- 172.21.15.59:0/1623755721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa680b2980 0x7ffa680c92b0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.031+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 >> 172.21.15.59:0/1623755721 conn(0x7ffa680d3b20 msgr2=0x7ffa680d0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.032+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 shutdown_connections 2024-06-29T15:55:25.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.032+0000 7ffa6f5f5640 1 -- 172.21.15.59:0/1623755721 wait complete. 2024-06-29T15:55:25.043 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:25.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.161+0000 7f62176e4640 1 -- 172.21.15.59:0/611380501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f621015efe0 msgr2=0x7f621015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.161+0000 7f62176e4640 1 --2- 172.21.15.59:0/611380501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f621015efe0 0x7f621015f9b0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6208009830 tx=0x7f620802a510 comp rx=0 tx=0).stop 2024-06-29T15:55:25.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.161+0000 7f62176e4640 1 -- 172.21.15.59:0/611380501 shutdown_connections 2024-06-29T15:55:25.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.161+0000 7f62176e4640 1 --2- 172.21.15.59:0/611380501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f621015efe0 0x7f621015f9b0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.161+0000 7f62176e4640 1 --2- 172.21.15.59:0/611380501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62100d23c0 0x7f62100b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.161+0000 7f62176e4640 1 -- 172.21.15.59:0/611380501 >> 172.21.15.59:0/611380501 conn(0x7f62100d3ee0 msgr2=0x7f62100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.161+0000 7f62176e4640 1 -- 172.21.15.59:0/611380501 shutdown_connections 2024-06-29T15:55:25.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.161+0000 7f62176e4640 1 -- 172.21.15.59:0/611380501 wait complete. 2024-06-29T15:55:25.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.163+0000 7f62176e4640 1 Processor -- start 2024-06-29T15:55:25.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.163+0000 7f62176e4640 1 -- start start 2024-06-29T15:55:25.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f62176e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62100b3ed0 0x7f621015c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f62176e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62100d23c0 0x7f621015b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f62176e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f621015efe0 0x7f621015b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f62176e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6210160280 con 0x7f621015efe0 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f62176e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6210160100 con 0x7f62100b3ed0 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f62176e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6210160400 con 0x7f62100d23c0 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f6215c81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f621015efe0 0x7f621015b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f6215c81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f621015efe0 0x7f621015b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48662/0 (socket says 172.21.15.59:48662) 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f6215c81640 1 -- 172.21.15.59:0/4001602406 learned_addr learned my addr 172.21.15.59:0/4001602406 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f6215480640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62100b3ed0 0x7f621015c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f6214c7f640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62100d23c0 0x7f621015b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f6215c81640 1 -- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62100d23c0 msgr2=0x7f621015b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.164+0000 7f6215c81640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62100d23c0 0x7f621015b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f6215c81640 1 -- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62100b3ed0 msgr2=0x7f621015c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f6215c81640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62100b3ed0 0x7f621015c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f6215c81640 1 -- 172.21.15.59:0/4001602406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62101477c0 con 0x7f621015efe0 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f6214c7f640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62100d23c0 0x7f621015b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f6215480640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62100b3ed0 0x7f621015c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f6215c81640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f621015efe0 0x7f621015b570 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f620802ab00 tx=0x7f6208033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f620dffb640 1 -- 172.21.15.59:0/4001602406 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208005c90 con 0x7f621015efe0 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f620dffb640 1 -- 172.21.15.59:0/4001602406 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6208031e50 con 0x7f621015efe0 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62100c7120 con 0x7f621015efe0 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f620dffb640 1 -- 172.21.15.59:0/4001602406 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62080332d0 con 0x7f621015efe0 2024-06-29T15:55:25.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.165+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f621015a270 con 0x7f621015efe0 2024-06-29T15:55:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.166+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62100b3990 con 0x7f621015efe0 2024-06-29T15:55:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.167+0000 7f620dffb640 1 -- 172.21.15.59:0/4001602406 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 115) ==== 98039+0+0 (secure 0 0 0) 0x7f6208033470 con 0x7f621015efe0 2024-06-29T15:55:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.167+0000 7f620dffb640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f61fc076120 0x7f61fc078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.167+0000 7f620dffb640 1 -- 172.21.15.59:0/4001602406 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 7189+0+0 (secure 0 0 0) 0x7f62080b8b50 con 0x7f621015efe0 2024-06-29T15:55:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.168+0000 7f6215480640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f61fc076120 0x7f61fc078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.168+0000 7f6215480640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f61fc076120 0x7f61fc078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f62100d2e40 tx=0x7f62000095c0 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.172+0000 7f620dffb640 1 -- 172.21.15.59:0/4001602406 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f620804a030 con 0x7f621015efe0 2024-06-29T15:55:25.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.403+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62100d2950 con 0x7f621015efe0 2024-06-29T15:55:25.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.404+0000 7f620dffb640 1 -- 172.21.15.59:0/4001602406 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5235 (secure 0 0 0) 0x7f620803f030 con 0x7f621015efe0 2024-06-29T15:55:25.404 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:25.404 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"btime":"2024-06-29T15:39:15:059125+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8588,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3866646629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3866646629},{"type":"v1","addr":"172.21.15.153:6835","nonce":3866646629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9724,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/933275096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":933275096},{"type":"v1","addr":"172.21.15.59:6835","nonce":933275096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9736,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1230229255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1230229255},{"type":"v1","addr":"172.21.15.153:6837","nonce":1230229255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.430152+0000","modified":"2024-06-29T15:39:14.448659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2399523179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2399523179},{"type":"v1","addr":"172.21.15.59:6834","nonce":2399523179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10401,"qdb_cluster":[10401]},"id":11}]} 2024-06-29T15:55:25.404 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 97 2024-06-29T15:55:25.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f61fc076120 msgr2=0x7f61fc078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f61fc076120 0x7f61fc078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f62100d2e40 tx=0x7f62000095c0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f621015efe0 msgr2=0x7f621015b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f621015efe0 0x7f621015b570 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f620802ab00 tx=0x7f6208033c20 comp rx=0 tx=0).stop 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 shutdown_connections 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f61fc076120 0x7f61fc078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f621015efe0 0x7f621015b570 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62100d23c0 0x7f621015b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 --2- 172.21.15.59:0/4001602406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62100b3ed0 0x7f621015c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 >> 172.21.15.59:0/4001602406 conn(0x7f62100d3ee0 msgr2=0x7f621006c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 shutdown_connections 2024-06-29T15:55:25.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.406+0000 7f62176e4640 1 -- 172.21.15.59:0/4001602406 wait complete. 2024-06-29T15:55:25.417 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 -- 172.21.15.59:0/3726390791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 msgr2=0x7fad3c0d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 --2- 172.21.15.59:0/3726390791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 0x7fad3c0d1a20 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fad30005fd0 tx=0x7fad3002a580 comp rx=0 tx=0).stop 2024-06-29T15:55:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 -- 172.21.15.59:0/3726390791 shutdown_connections 2024-06-29T15:55:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 --2- 172.21.15.59:0/3726390791 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad3c0ce950 0x7fad3c0d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 --2- 172.21.15.59:0/3726390791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 0x7fad3c0d1a20 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 --2- 172.21.15.59:0/3726390791 >> v2:172.21.15.153:3300/0 conn(0x7fad3c08c190 0x7fad3c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 -- 172.21.15.59:0/3726390791 >> 172.21.15.59:0/3726390791 conn(0x7fad3c0b5610 msgr2=0x7fad3c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 -- 172.21.15.59:0/3726390791 shutdown_connections 2024-06-29T15:55:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.535+0000 7fad43768640 1 -- 172.21.15.59:0/3726390791 wait complete. 2024-06-29T15:55:25.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.537+0000 7fad43768640 1 Processor -- start 2024-06-29T15:55:25.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.537+0000 7fad43768640 1 -- start start 2024-06-29T15:55:25.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad43768640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad3c08c190 0x7fad3c1f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad43768640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad3c0ce950 0x7fad3c1588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad43768640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 0x7fad3c0cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad43768640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fad3c162440 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad43768640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fad3c1622c0 con 0x7fad3c08c190 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad43768640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fad3c1625c0 con 0x7fad3c0ce950 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad41d05640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 0x7fad3c0cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad41d05640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 0x7fad3c0cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48688/0 (socket says 172.21.15.59:48688) 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad41d05640 1 -- 172.21.15.59:0/3523841019 learned_addr learned my addr 172.21.15.59:0/3523841019 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad40d03640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad3c0ce950 0x7fad3c1588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.538+0000 7fad41504640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad3c08c190 0x7fad3c1f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad41d05640 1 -- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad3c0ce950 msgr2=0x7fad3c1588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad41d05640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad3c0ce950 0x7fad3c1588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad41d05640 1 -- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad3c08c190 msgr2=0x7fad3c1f7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad41d05640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad3c08c190 0x7fad3c1f7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad41d05640 1 -- 172.21.15.59:0/3523841019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad3c1595e0 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad41d05640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 0x7fad3c0cd0e0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fad3400b410 tx=0x7fad3400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad39ffb640 1 -- 172.21.15.59:0/3523841019 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad34015020 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad39ffb640 1 -- 172.21.15.59:0/3523841019 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad34011920 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad3c1598c0 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad39ffb640 1 -- 172.21.15.59:0/3523841019 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad34011d70 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.539+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad3c159e80 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.540+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad3c159350 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.541+0000 7fad39ffb640 1 -- 172.21.15.59:0/3523841019 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 115) ==== 98039+0+0 (secure 0 0 0) 0x7fad34018020 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.541+0000 7fad39ffb640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fad28076120 0x7fad28078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.542+0000 7fad39ffb640 1 -- 172.21.15.59:0/3523841019 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 7189+0+0 (secure 0 0 0) 0x7fad34097870 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.542+0000 7fad41504640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fad28076120 0x7fad28078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.542+0000 7fad41504640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fad28076120 0x7fad28078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00a400 tx=0x7fad2c00d040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.546+0000 7fad39ffb640 1 -- 172.21.15.59:0/3523841019 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fad3401f020 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.741+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad3c0cf080 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.742+0000 7fad39ffb640 1 -- 172.21.15.59:0/3523841019 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) ==== 74+0+18537 (secure 0 0 0) 0x7fad34060790 con 0x7fad3c0cf5c0 2024-06-29T15:55:25.742 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:25.742 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":193,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:55:23.652664+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.834000+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.377444+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1343183333":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6833/2924556144":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6832/2924556144":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/2895806264":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/3420796917":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3711582444":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/1469314326":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/3023327174":"2024-06-30T15:50:20.742173+0000","172.21.15.153:6833/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/2605088182":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/323020462":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6832/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/1412243249":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3586261468":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1851934149":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/771497135":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/1918645711":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/4098340029":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/2596446141":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/3821515452":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1052117807":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/2595561638":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6833/1270545814":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6832/1270545814":"2024-06-30T15:39:23.696245+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fad28076120 msgr2=0x7fad28078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fad28076120 0x7fad28078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00a400 tx=0x7fad2c00d040 comp rx=0 tx=0).stop 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 msgr2=0x7fad3c0cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 0x7fad3c0cd0e0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fad3400b410 tx=0x7fad3400b810 comp rx=0 tx=0).stop 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 shutdown_connections 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fad28076120 0x7fad28078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c0cf5c0 0x7fad3c0cd0e0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fad3c0ce950 0x7fad3c1588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 --2- 172.21.15.59:0/3523841019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad3c08c190 0x7fad3c1f7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 >> 172.21.15.59:0/3523841019 conn(0x7fad3c0b5610 msgr2=0x7fad3c15c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 shutdown_connections 2024-06-29T15:55:25.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.744+0000 7fad43768640 1 -- 172.21.15.59:0/3523841019 wait complete. 2024-06-29T15:55:25.756 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:55:25.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.876+0000 7f618cd7f640 1 -- 172.21.15.59:0/444801906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740042a0 msgr2=0x7f6174004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.876+0000 7f618cd7f640 1 --2- 172.21.15.59:0/444801906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740042a0 0x7f6174004800 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f618813f300 tx=0x7f6174005830 comp rx=0 tx=0).stop 2024-06-29T15:55:25.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.876+0000 7f6186d9d640 1 -- 172.21.15.59:0/444801906 reap_dead start 2024-06-29T15:55:25.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.876+0000 7f618cd7f640 1 -- 172.21.15.59:0/444801906 shutdown_connections 2024-06-29T15:55:25.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.876+0000 7f618cd7f640 1 -- 172.21.15.59:0/444801906 >> 172.21.15.59:0/444801906 conn(0x7f61880b6fb0 msgr2=0x7f61880b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.876+0000 7f618cd7f640 1 -- 172.21.15.59:0/444801906 shutdown_connections 2024-06-29T15:55:25.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.876+0000 7f618cd7f640 1 -- 172.21.15.59:0/444801906 wait complete. 2024-06-29T15:55:25.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.878+0000 7f618cd7f640 1 Processor -- start 2024-06-29T15:55:25.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.878+0000 7f618cd7f640 1 -- start start 2024-06-29T15:55:25.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f618cd7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61740042a0 0x7f61881590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f618cd7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f618815b820 0x7f61880c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f618cd7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6188159fa0 0x7f618815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f618cd7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f618813f150 con 0x7f618815b820 2024-06-29T15:55:25.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f618cd7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f618813efd0 con 0x7f61740042a0 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f618cd7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f61880b2530 con 0x7f6188159fa0 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6185d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f618815b820 0x7f61880c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6185d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f618815b820 0x7f61880c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48706/0 (socket says 172.21.15.59:48706) 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6185d9b640 1 -- 172.21.15.59:0/2410593654 learned_addr learned my addr 172.21.15.59:0/2410593654 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f618659c640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61740042a0 0x7f61881590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6185d9b640 1 -- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6188159fa0 msgr2=0x7f618815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6186d9d640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6188159fa0 0x7f618815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6185d9b640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6188159fa0 0x7f618815a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6185d9b640 1 -- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61740042a0 msgr2=0x7f61881590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6185d9b640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61740042a0 0x7f61881590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f6185d9b640 1 -- 172.21.15.59:0/2410593654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618815a8f0 con 0x7f618815b820 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.879+0000 7f618659c640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61740042a0 0x7f61881590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.880+0000 7f6185d9b640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f618815b820 0x7f61880c78d0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f6170036040 tx=0x7f6170033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.880+0000 7f617effd640 1 -- 172.21.15.59:0/2410593654 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170038040 con 0x7f618815b820 2024-06-29T15:55:25.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.880+0000 7f617effd640 1 -- 172.21.15.59:0/2410593654 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61700337e0 con 0x7f618815b820 2024-06-29T15:55:25.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.880+0000 7f617effd640 1 -- 172.21.15.59:0/2410593654 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170033c10 con 0x7f618815b820 2024-06-29T15:55:25.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.880+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61880c7e10 con 0x7f618815b820 2024-06-29T15:55:25.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.880+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6188158050 con 0x7f618815b820 2024-06-29T15:55:25.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.881+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61880b56d0 con 0x7f618815b820 2024-06-29T15:55:25.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.882+0000 7f617effd640 1 -- 172.21.15.59:0/2410593654 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 115) ==== 98039+0+0 (secure 0 0 0) 0x7f6170043020 con 0x7f618815b820 2024-06-29T15:55:25.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.882+0000 7f617effd640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6168076120 0x7f6168078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.882+0000 7f617effd640 1 -- 172.21.15.59:0/2410593654 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 7189+0+0 (secure 0 0 0) 0x7f61700b7bc0 con 0x7f618815b820 2024-06-29T15:55:25.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.882+0000 7f618659c640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6168076120 0x7f6168078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.883+0000 7f618659c640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6168076120 0x7f6168078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6180006fd0 tx=0x7f6180001190 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.887+0000 7f617effd640 1 -- 172.21.15.59:0/2410593654 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f617004a020 con 0x7f618815b820 2024-06-29T15:55:25.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:25.981+0000 7f617effd640 1 -- 172.21.15.59:0/2410593654 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 116) ==== 98247+0+0 (secure 0 0 0) 0x7f617007ce20 con 0x7f618815b820 2024-06-29T15:55:26.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.117+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6188074e20 con 0x7f618815b820 2024-06-29T15:55:26.117 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:55:26.118+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:55:26.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.121+0000 7f617effd640 1 -- 172.21.15.59:0/2410593654 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v98) ==== 161+0+0 (secure 0 0 0) 0x7f6170081290 con 0x7f618815b820 2024-06-29T15:55:26.121 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:55:26.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6168076120 msgr2=0x7f6168078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6168076120 0x7f6168078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6180006fd0 tx=0x7f6180001190 comp rx=0 tx=0).stop 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f618815b820 msgr2=0x7f61880c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f618815b820 0x7f61880c78d0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f6170036040 tx=0x7f6170033670 comp rx=0 tx=0).stop 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 shutdown_connections 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6168076120 0x7f6168078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6188159fa0 0x7f618815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f618815b820 0x7f61880c78d0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 --2- 172.21.15.59:0/2410593654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61740042a0 0x7f61881590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 >> 172.21.15.59:0/2410593654 conn(0x7f61880b6fb0 msgr2=0x7f61880ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 shutdown_connections 2024-06-29T15:55:26.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.123+0000 7f618cd7f640 1 -- 172.21.15.59:0/2410593654 wait complete. 2024-06-29T15:55:26.136 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:55:26.188 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:55:26.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.260+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2398232718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a14004d40 msgr2=0x7f0a14005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.260+0000 7f0a2fcef640 1 --2- 172.21.15.59:0/2398232718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a14004d40 0x7f0a14005150 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f0a28073d70 tx=0x7f0a14005830 comp rx=0 tx=0).stop 2024-06-29T15:55:26.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.260+0000 7f0a2e28c640 1 -- 172.21.15.59:0/2398232718 reap_dead start 2024-06-29T15:55:26.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.260+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2398232718 shutdown_connections 2024-06-29T15:55:26.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.260+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2398232718 >> 172.21.15.59:0/2398232718 conn(0x7f0a280b6ed0 msgr2=0x7f0a280b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:26.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.260+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2398232718 shutdown_connections 2024-06-29T15:55:26.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.260+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2398232718 wait complete. 2024-06-29T15:55:26.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.262+0000 7f0a2fcef640 1 Processor -- start 2024-06-29T15:55:26.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.262+0000 7f0a2fcef640 1 -- start start 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.263+0000 7f0a2fcef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a14004d40 0x7f0a280c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2fcef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a280c9c00 0x7f0a281f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2fcef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a281f48e0 0x7f0a281f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2fcef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2815f7f0 con 0x7f0a280c9c00 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2fcef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2815eba0 con 0x7f0a281f48e0 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2fcef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0a2815f970 con 0x7f0a14004d40 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a280c9c00 0x7f0a281f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a280c9c00 0x7f0a281f43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48756/0 (socket says 172.21.15.59:48756) 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 -- 172.21.15.59:0/2826613107 learned_addr learned my addr 172.21.15.59:0/2826613107 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2da8b640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a14004d40 0x7f0a280c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2e28c640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a281f48e0 0x7f0a281f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 -- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a14004d40 msgr2=0x7f0a280c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a14004d40 0x7f0a280c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 -- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a281f48e0 msgr2=0x7f0a281f4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a281f48e0 0x7f0a281f4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 -- 172.21.15.59:0/2826613107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a281f5230 con 0x7f0a280c9c00 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2da8b640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a14004d40 0x7f0a280c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2d28a640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a280c9c00 0x7f0a281f43a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f0a180118b0 tx=0x7f0a18011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a1e7fc640 1 -- 172.21.15.59:0/2826613107 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18004460 con 0x7f0a280c9c00 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a1e7fc640 1 -- 172.21.15.59:0/2826613107 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a1800f9a0 con 0x7f0a280c9c00 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a1e7fc640 1 -- 172.21.15.59:0/2826613107 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1800fdf0 con 0x7f0a280c9c00 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a280ca380 con 0x7f0a280c9c00 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.264+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a281f3ba0 con 0x7f0a280c9c00 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.265+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a280b5cb0 con 0x7f0a280c9c00 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.266+0000 7f0a1e7fc640 1 -- 172.21.15.59:0/2826613107 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 116) ==== 98247+0+0 (secure 0 0 0) 0x7f0a18005020 con 0x7f0a280c9c00 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.266+0000 7f0a1e7fc640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0a0c076360 0x7f0a0c078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.267+0000 7f0a1e7fc640 1 -- 172.21.15.59:0/2826613107 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(194..194 src has 1..194) ==== 7275+0+0 (secure 0 0 0) 0x7f0a18098ad0 con 0x7f0a280c9c00 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.267+0000 7f0a2da8b640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0a0c076360 0x7f0a0c078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.267+0000 7f0a2da8b640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0a0c076360 0x7f0a0c078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009cd0 tx=0x7f0a24005f90 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:26.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.271+0000 7f0a1e7fc640 1 -- 172.21.15.59:0/2826613107 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0a18061d20 con 0x7f0a280c9c00 2024-06-29T15:55:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:26.501+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f0a281417a0 con 0x7f0a280c9c00 2024-06-29T15:55:27.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.124+0000 7f0a1e7fc640 1 -- 172.21.15.59:0/2826613107 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v99) ==== 104+0+0 (secure 0 0 0) 0x7f0a18061620 con 0x7f0a280c9c00 2024-06-29T15:55:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0a0c076360 msgr2=0x7f0a0c078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0a0c076360 0x7f0a0c078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009cd0 tx=0x7f0a24005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a280c9c00 msgr2=0x7f0a281f43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a280c9c00 0x7f0a281f43a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f0a180118b0 tx=0x7f0a18011d70 comp rx=0 tx=0).stop 2024-06-29T15:55:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 shutdown_connections 2024-06-29T15:55:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0a0c076360 0x7f0a0c078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a281f48e0 0x7f0a281f4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a280c9c00 0x7f0a281f43a0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 --2- 172.21.15.59:0/2826613107 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a14004d40 0x7f0a280c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 >> 172.21.15.59:0/2826613107 conn(0x7f0a280b6ed0 msgr2=0x7f0a28068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 shutdown_connections 2024-06-29T15:55:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.127+0000 7f0a2fcef640 1 -- 172.21.15.59:0/2826613107 wait complete. 2024-06-29T15:55:27.141 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:55:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.268+0000 7fb11dbef640 1 -- 172.21.15.59:0/3208645455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 msgr2=0x7fb1181626b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.268+0000 7fb11dbef640 1 --2- 172.21.15.59:0/3208645455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 0x7fb1181626b0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fb10c005fd0 tx=0x7fb10c02a490 comp rx=0 tx=0).stop 2024-06-29T15:55:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.268+0000 7fb11dbef640 1 -- 172.21.15.59:0/3208645455 shutdown_connections 2024-06-29T15:55:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.268+0000 7fb11dbef640 1 --2- 172.21.15.59:0/3208645455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1180b4510 0x7fb1180b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.268+0000 7fb11dbef640 1 --2- 172.21.15.59:0/3208645455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 0x7fb1181626b0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.268+0000 7fb11dbef640 1 -- 172.21.15.59:0/3208645455 >> 172.21.15.59:0/3208645455 conn(0x7fb1180d4070 msgr2=0x7fb1180d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.268+0000 7fb11dbef640 1 -- 172.21.15.59:0/3208645455 shutdown_connections 2024-06-29T15:55:27.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.268+0000 7fb11dbef640 1 -- 172.21.15.59:0/3208645455 wait complete. 2024-06-29T15:55:27.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.270+0000 7fb11dbef640 1 Processor -- start 2024-06-29T15:55:27.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.270+0000 7fb11dbef640 1 -- start start 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb11dbef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1180b4510 0x7fb11814ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb11dbef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 0x7fb1180ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb11dbef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1180d2830 0x7fb1181eab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb11dbef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb1180b77a0 con 0x7fb1180d1610 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb11dbef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1180b7620 con 0x7fb1180d2830 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb11dbef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb1180b7920 con 0x7fb1180b4510 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 0x7fb1180ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 0x7fb1180ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48776/0 (socket says 172.21.15.59:48776) 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 -- 172.21.15.59:0/1219959739 learned_addr learned my addr 172.21.15.59:0/1219959739 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb117fff640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1180d2830 0x7fb1181eab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:27.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb1177fe640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1180b4510 0x7fb11814ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 -- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1180b4510 msgr2=0x7fb11814ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1180b4510 0x7fb11814ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 -- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1180d2830 msgr2=0x7fb1181eab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1180d2830 0x7fb1181eab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 -- 172.21.15.59:0/1219959739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11814f2b0 con 0x7fb1180d1610 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.272+0000 7fb116ffd640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 0x7fb1180ccf10 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb10c034040 tx=0x7fb10c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.273+0000 7fb0fffff640 1 -- 172.21.15.59:0/1219959739 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c03e030 con 0x7fb1180d1610 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.273+0000 7fb0fffff640 1 -- 172.21.15.59:0/1219959739 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb10c0047a0 con 0x7fb1180d1610 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.273+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1181e93b0 con 0x7fb1180d1610 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.273+0000 7fb0fffff640 1 -- 172.21.15.59:0/1219959739 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c035040 con 0x7fb1180d1610 2024-06-29T15:55:27.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.273+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1181e9970 con 0x7fb1180d1610 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.274+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1180b7eb0 con 0x7fb1180d1610 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.275+0000 7fb0fffff640 1 -- 172.21.15.59:0/1219959739 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 116) ==== 98247+0+0 (secure 0 0 0) 0x7fb10c041030 con 0x7fb1180d1610 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.275+0000 7fb0fffff640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb104076360 0x7fb104078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.275+0000 7fb0fffff640 1 -- 172.21.15.59:0/1219959739 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(194..194 src has 1..194) ==== 7275+0+0 (secure 0 0 0) 0x7fb10c0b8720 con 0x7fb1180d1610 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.275+0000 7fb1177fe640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb104076360 0x7fb104078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.276+0000 7fb1177fe640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb104076360 0x7fb104078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb108006fd0 tx=0x7fb108009060 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:27.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.280+0000 7fb0fffff640 1 -- 172.21.15.59:0/1219959739 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb1180b4e60 con 0x7fb1180d1610 2024-06-29T15:55:27.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:27.475+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb1180cde00 con 0x7fb1180d1610 2024-06-29T15:55:28.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.130+0000 7fb0fffff640 1 -- 172.21.15.59:0/1219959739 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v195) ==== 181+0+0 (secure 0 0 0) 0x7fb10c081280 con 0x7fb1180d1610 2024-06-29T15:55:28.131 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:55:28.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb104076360 msgr2=0x7fb104078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb104076360 0x7fb104078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb108006fd0 tx=0x7fb108009060 comp rx=0 tx=0).stop 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 msgr2=0x7fb1180ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 0x7fb1180ccf10 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb10c034040 tx=0x7fb10c0039e0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 shutdown_connections 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb104076360 0x7fb104078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1180d2830 0x7fb1181eab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1180d1610 0x7fb1180ccf10 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 --2- 172.21.15.59:0/1219959739 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1180b4510 0x7fb11814ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 >> 172.21.15.59:0/1219959739 conn(0x7fb1180d4070 msgr2=0x7fb1180b2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:28.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 shutdown_connections 2024-06-29T15:55:28.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.133+0000 7fb11dbef640 1 -- 172.21.15.59:0/1219959739 wait complete. 2024-06-29T15:55:28.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:55:28.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.281+0000 7f0360210640 1 -- 172.21.15.59:0/2904581273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0340005350 msgr2=0x7f034000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.281+0000 7f0360210640 1 --2- 172.21.15.59:0/2904581273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0340005350 0x7f034000b9c0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f0348001dc0 tx=0x7f0340005830 comp rx=0 tx=0).stop 2024-06-29T15:55:28.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.281+0000 7f0360210640 1 -- 172.21.15.59:0/2904581273 shutdown_connections 2024-06-29T15:55:28.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.281+0000 7f0360210640 1 --2- 172.21.15.59:0/2904581273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0340005350 0x7f034000b9c0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.281+0000 7f0360210640 1 -- 172.21.15.59:0/2904581273 >> 172.21.15.59:0/2904581273 conn(0x7f03580b2ab0 msgr2=0x7f03580b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:28.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.281+0000 7f0360210640 1 -- 172.21.15.59:0/2904581273 shutdown_connections 2024-06-29T15:55:28.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.281+0000 7f0360210640 1 -- 172.21.15.59:0/2904581273 wait complete. 2024-06-29T15:55:28.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.283+0000 7f0360210640 1 Processor -- start 2024-06-29T15:55:28.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.283+0000 7f0360210640 1 -- start start 2024-06-29T15:55:28.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f0360210640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340005350 0x7f0358141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f0360210640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f035815a270 0x7f0358141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f0360210640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0358140a50 0x7f03580b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f0360210640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f035815b5c0 con 0x7f035815a270 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f0360210640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f035815b440 con 0x7f0340005350 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f0360210640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f035815b740 con 0x7f0358140a50 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035d7ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f035815a270 0x7f0358141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035d7ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f035815a270 0x7f0358141ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48788/0 (socket says 172.21.15.59:48788) 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035d7ab640 1 -- 172.21.15.59:0/151980745 learned_addr learned my addr 172.21.15.59:0/151980745 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035dfac640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340005350 0x7f0358141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035e7ad640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0358140a50 0x7f03580b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035d7ab640 1 -- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0358140a50 msgr2=0x7f03580b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035d7ab640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0358140a50 0x7f03580b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035d7ab640 1 -- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340005350 msgr2=0x7f0358141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035d7ab640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340005350 0x7f0358141990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.284+0000 7f035d7ab640 1 -- 172.21.15.59:0/151980745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035813ef40 con 0x7f035815a270 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.285+0000 7f035d7ab640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f035815a270 0x7f0358141ed0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f034c00e600 tx=0x7f034c00eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.285+0000 7f03567fc640 1 -- 172.21.15.59:0/151980745 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c015070 con 0x7f035815a270 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.285+0000 7f03567fc640 1 -- 172.21.15.59:0/151980745 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f034c00ca50 con 0x7f035815a270 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.285+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035813f220 con 0x7f035815a270 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.285+0000 7f03567fc640 1 -- 172.21.15.59:0/151980745 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c011460 con 0x7f035815a270 2024-06-29T15:55:28.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.285+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f035813f7e0 con 0x7f035815a270 2024-06-29T15:55:28.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.286+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f035815c2e0 con 0x7f035815a270 2024-06-29T15:55:28.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.287+0000 7f03567fc640 1 -- 172.21.15.59:0/151980745 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f034c018070 con 0x7f035815a270 2024-06-29T15:55:28.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.287+0000 7f03567fc640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0344076420 0x7f0344078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:28.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.287+0000 7f03567fc640 1 -- 172.21.15.59:0/151980745 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 6837+0+0 (secure 0 0 0) 0x7f034c0981f0 con 0x7f035815a270 2024-06-29T15:55:28.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.287+0000 7f035dfac640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0344076420 0x7f0344078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:28.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.288+0000 7f035dfac640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0344076420 0x7f0344078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0350009b10 tx=0x7f0350005f90 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:28.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.292+0000 7f03567fc640 1 -- 172.21.15.59:0/151980745 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f034c061430 con 0x7f035815a270 2024-06-29T15:55:28.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:28.490+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f03580adae0 con 0x7f035815a270 2024-06-29T15:55:29.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.132+0000 7f03567fc640 1 -- 172.21.15.59:0/151980745 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v196) ==== 169+0+0 (secure 0 0 0) 0x7f034c060d30 con 0x7f035815a270 2024-06-29T15:55:29.132 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T15:55:29.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.136+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0344076420 msgr2=0x7f0344078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.136+0000 7f0360210640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0344076420 0x7f0344078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0350009b10 tx=0x7f0350005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:29.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.136+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f035815a270 msgr2=0x7f0358141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.136+0000 7f0360210640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f035815a270 0x7f0358141ed0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f034c00e600 tx=0x7f034c00eac0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.137+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 shutdown_connections 2024-06-29T15:55:29.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.137+0000 7f0360210640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f0344076420 0x7f0344078930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.137+0000 7f0360210640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0358140a50 0x7f03580b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.137+0000 7f0360210640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f035815a270 0x7f0358141ed0 secure :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f034c00e600 tx=0x7f034c00eac0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.137+0000 7f0360210640 1 --2- 172.21.15.59:0/151980745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340005350 0x7f0358141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.137+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 >> 172.21.15.59:0/151980745 conn(0x7f03580b2ab0 msgr2=0x7f03580ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:29.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.137+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 shutdown_connections 2024-06-29T15:55:29.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.137+0000 7f0360210640 1 -- 172.21.15.59:0/151980745 wait complete. 2024-06-29T15:55:29.156 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:55:29.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.280+0000 7fc0ea281640 1 -- 172.21.15.59:0/1124475406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e415efe0 msgr2=0x7fc0e415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.280+0000 7fc0ea281640 1 --2- 172.21.15.59:0/1124475406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e415efe0 0x7fc0e415f9b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc005fd0 tx=0x7fc0dc02a940 comp rx=0 tx=0).stop 2024-06-29T15:55:29.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.280+0000 7fc0ea281640 1 -- 172.21.15.59:0/1124475406 shutdown_connections 2024-06-29T15:55:29.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.280+0000 7fc0ea281640 1 --2- 172.21.15.59:0/1124475406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e415efe0 0x7fc0e415f9b0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.280+0000 7fc0ea281640 1 -- 172.21.15.59:0/1124475406 >> 172.21.15.59:0/1124475406 conn(0x7fc0e40d3ee0 msgr2=0x7fc0e40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:29.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.280+0000 7fc0ea281640 1 -- 172.21.15.59:0/1124475406 shutdown_connections 2024-06-29T15:55:29.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.281+0000 7fc0ea281640 1 -- 172.21.15.59:0/1124475406 wait complete. 2024-06-29T15:55:29.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.282+0000 7fc0ea281640 1 Processor -- start 2024-06-29T15:55:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.283+0000 7fc0ea281640 1 -- start start 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0ea281640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e40d23c0 0x7fc0e40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0ea281640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e415efe0 0x7fc0e4157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0ea281640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0e40cb790 0x7fc0e414fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0ea281640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e4160420 con 0x7fc0e40d23c0 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0ea281640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e41602a0 con 0x7fc0e415efe0 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0ea281640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e41605a0 con 0x7fc0e40cb790 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e40d23c0 0x7fc0e40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e40d23c0 0x7fc0e40cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48812/0 (socket says 172.21.15.59:48812) 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e3fff640 1 -- 172.21.15.59:0/1348945326 learned_addr learned my addr 172.21.15.59:0/1348945326 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e37fe640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e415efe0 0x7fc0e4157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e881e640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0e40cb790 0x7fc0e414fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e3fff640 1 -- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0e40cb790 msgr2=0x7fc0e414fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e3fff640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0e40cb790 0x7fc0e414fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e3fff640 1 -- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e415efe0 msgr2=0x7fc0e4157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e3fff640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e415efe0 0x7fc0e4157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.284+0000 7fc0e3fff640 1 -- 172.21.15.59:0/1348945326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e4157f60 con 0x7fc0e40d23c0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.285+0000 7fc0e3fff640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e40d23c0 0x7fc0e40cd030 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400c400 tx=0x7fc0d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.285+0000 7fc0e0ff9640 1 -- 172.21.15.59:0/1348945326 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4011790 con 0x7fc0e40d23c0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.285+0000 7fc0e0ff9640 1 -- 172.21.15.59:0/1348945326 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0d4017070 con 0x7fc0e40d23c0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.285+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e40cd860 con 0x7fc0e40d23c0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.285+0000 7fc0e0ff9640 1 -- 172.21.15.59:0/1348945326 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4011c60 con 0x7fc0e40d23c0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.285+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e40c8a50 con 0x7fc0e40d23c0 2024-06-29T15:55:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.286+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e415f9b0 con 0x7fc0e40d23c0 2024-06-29T15:55:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.287+0000 7fc0e0ff9640 1 -- 172.21.15.59:0/1348945326 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fc0d401c460 con 0x7fc0e40d23c0 2024-06-29T15:55:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.287+0000 7fc0e0ff9640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fc0d00763f0 0x7fc0d0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.287+0000 7fc0e0ff9640 1 -- 172.21.15.59:0/1348945326 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7fc0d4097910 con 0x7fc0e40d23c0 2024-06-29T15:55:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.287+0000 7fc0e37fe640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fc0d00763f0 0x7fc0d0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.288+0000 7fc0e37fe640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fc0d00763f0 0x7fc0d0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40cdc90 tx=0x7fc0d8009040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.292+0000 7fc0e0ff9640 1 -- 172.21.15.59:0/1348945326 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0d40607c0 con 0x7fc0e40d23c0 2024-06-29T15:55:29.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.483+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fc0e40d2950 con 0x7fc0e40d23c0 2024-06-29T15:55:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.488+0000 7fc0e0ff9640 1 -- 172.21.15.59:0/1348945326 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v67) ==== 71+0+0 (secure 0 0 0) 0x7fc0d4064ac0 con 0x7fc0e40d23c0 2024-06-29T15:55:29.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fc0d00763f0 msgr2=0x7fc0d0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fc0d00763f0 0x7fc0d0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40cdc90 tx=0x7fc0d8009040 comp rx=0 tx=0).stop 2024-06-29T15:55:29.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e40d23c0 msgr2=0x7fc0e40cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e40d23c0 0x7fc0e40cd030 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400c400 tx=0x7fc0d400c8c0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 shutdown_connections 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fc0d00763f0 0x7fc0d0078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0e40cb790 0x7fc0e414fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0e415efe0 0x7fc0e4157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 --2- 172.21.15.59:0/1348945326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0e40d23c0 0x7fc0e40cd030 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 >> 172.21.15.59:0/1348945326 conn(0x7fc0e40d3ee0 msgr2=0x7fc0e4150820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 shutdown_connections 2024-06-29T15:55:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.490+0000 7fc0ea281640 1 -- 172.21.15.59:0/1348945326 wait complete. 2024-06-29T15:55:29.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-06-29T15:55:29.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.619+0000 7fae80554640 1 -- 172.21.15.59:0/4153671038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7813e9b0 msgr2=0x7fae7813edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.619+0000 7fae80554640 1 --2- 172.21.15.59:0/4153671038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7813e9b0 0x7fae7813edc0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fae6c005fd0 tx=0x7fae6c02a540 comp rx=0 tx=0).stop 2024-06-29T15:55:29.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.619+0000 7fae80554640 1 -- 172.21.15.59:0/4153671038 shutdown_connections 2024-06-29T15:55:29.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.619+0000 7fae80554640 1 --2- 172.21.15.59:0/4153671038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7813e9b0 0x7fae7813edc0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.619+0000 7fae80554640 1 -- 172.21.15.59:0/4153671038 >> 172.21.15.59:0/4153671038 conn(0x7fae780b6fb0 msgr2=0x7fae780b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:29.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.619+0000 7fae80554640 1 -- 172.21.15.59:0/4153671038 shutdown_connections 2024-06-29T15:55:29.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.619+0000 7fae80554640 1 -- 172.21.15.59:0/4153671038 wait complete. 2024-06-29T15:55:29.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.621+0000 7fae80554640 1 Processor -- start 2024-06-29T15:55:29.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.621+0000 7fae80554640 1 -- start start 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae80554640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae7813e9b0 0x7fae7813d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae80554640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae781414c0 0x7fae7815b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae80554640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae7815ceb0 0x7fae7815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae80554640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fae780b2900 con 0x7fae781414c0 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae80554640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae780b2780 con 0x7fae7813e9b0 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae80554640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fae780b2a80 con 0x7fae7815ceb0 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae7daef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae781414c0 0x7fae7815b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae7daef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae781414c0 0x7fae7815b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48828/0 (socket says 172.21.15.59:48828) 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae7daef640 1 -- 172.21.15.59:0/3805468729 learned_addr learned my addr 172.21.15.59:0/3805468729 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae7e2f0640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae7813e9b0 0x7fae7813d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.622+0000 7fae7eaf1640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae7815ceb0 0x7fae7815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae7daef640 1 -- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae7815ceb0 msgr2=0x7fae7815b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae7daef640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae7815ceb0 0x7fae7815b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae7daef640 1 -- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae7813e9b0 msgr2=0x7fae7813d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae7daef640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae7813e9b0 0x7fae7813d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae7daef640 1 -- 172.21.15.59:0/3805468729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae780cb590 con 0x7fae781414c0 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae7e2f0640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae7813e9b0 0x7fae7813d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae7daef640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae781414c0 0x7fae7815b230 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000c00 tx=0x7fae6c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae76ffd640 1 -- 172.21.15.59:0/3805468729 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c03e030 con 0x7fae781414c0 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae76ffd640 1 -- 172.21.15.59:0/3805468729 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae6c003e60 con 0x7fae781414c0 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae780cacf0 con 0x7fae781414c0 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae76ffd640 1 -- 172.21.15.59:0/3805468729 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c035040 con 0x7fae781414c0 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.623+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae780cb2b0 con 0x7fae781414c0 2024-06-29T15:55:29.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.624+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae780caa60 con 0x7fae781414c0 2024-06-29T15:55:29.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.625+0000 7fae76ffd640 1 -- 172.21.15.59:0/3805468729 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fae6c041030 con 0x7fae781414c0 2024-06-29T15:55:29.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.625+0000 7fae76ffd640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fae640763f0 0x7fae64078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.626+0000 7fae76ffd640 1 -- 172.21.15.59:0/3805468729 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7fae6c0b84d0 con 0x7fae781414c0 2024-06-29T15:55:29.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.626+0000 7fae7e2f0640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fae640763f0 0x7fae64078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.626+0000 7fae7e2f0640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fae640763f0 0x7fae64078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c420 tx=0x7fae68009570 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:29.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.630+0000 7fae76ffd640 1 -- 172.21.15.59:0/3805468729 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fae6c081930 con 0x7fae781414c0 2024-06-29T15:55:29.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:29.900+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) -- 0x7fae780b2f70 con 0x7fae781414c0 2024-06-29T15:55:30.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.133+0000 7fae76ffd640 1 -- 172.21.15.59:0/3805468729 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3040) ==== 165+0+0 (secure 0 0 0) 0x7fae6c081230 con 0x7fae781414c0 2024-06-29T15:55:30.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.135+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fae640763f0 msgr2=0x7fae64078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.135+0000 7fae80554640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fae640763f0 0x7fae64078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c420 tx=0x7fae68009570 comp rx=0 tx=0).stop 2024-06-29T15:55:30.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae781414c0 msgr2=0x7fae7815b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae781414c0 0x7fae7815b230 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000c00 tx=0x7fae6c003970 comp rx=0 tx=0).stop 2024-06-29T15:55:30.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 shutdown_connections 2024-06-29T15:55:30.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fae640763f0 0x7fae64078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae7815ceb0 0x7fae7815b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae781414c0 0x7fae7815b230 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 --2- 172.21.15.59:0/3805468729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae7813e9b0 0x7fae7813d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 >> 172.21.15.59:0/3805468729 conn(0x7fae780b6fb0 msgr2=0x7fae78140500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 shutdown_connections 2024-06-29T15:55:30.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.136+0000 7fae80554640 1 -- 172.21.15.59:0/3805468729 wait complete. 2024-06-29T15:55:30.147 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-06-29T15:55:30.147 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-06-29T15:55:30.147 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T15:55:30.148 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:30.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.266+0000 7f14baa54640 1 -- 172.21.15.59:0/1093568802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14b40b3740 msgr2=0x7f14b40b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.266+0000 7f14baa54640 1 --2- 172.21.15.59:0/1093568802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14b40b3740 0x7f14b40b3b50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009830 tx=0x7f14a802a510 comp rx=0 tx=0).stop 2024-06-29T15:55:30.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.266+0000 7f14baa54640 1 -- 172.21.15.59:0/1093568802 shutdown_connections 2024-06-29T15:55:30.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.266+0000 7f14baa54640 1 --2- 172.21.15.59:0/1093568802 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14b40b3740 0x7f14b40b3b50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.266+0000 7f14baa54640 1 --2- 172.21.15.59:0/1093568802 >> v2:172.21.15.153:3300/0 conn(0x7f14b41458a0 0x7f14b40cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.267+0000 7f14baa54640 1 -- 172.21.15.59:0/1093568802 >> 172.21.15.59:0/1093568802 conn(0x7f14b406c950 msgr2=0x7f14b4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.267+0000 7f14baa54640 1 -- 172.21.15.59:0/1093568802 shutdown_connections 2024-06-29T15:55:30.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.267+0000 7f14baa54640 1 -- 172.21.15.59:0/1093568802 wait complete. 2024-06-29T15:55:30.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.268+0000 7f14baa54640 1 Processor -- start 2024-06-29T15:55:30.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.268+0000 7f14baa54640 1 -- start start 2024-06-29T15:55:30.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14baa54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b40b3740 0x7f14b40a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14baa54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415c2f0 0x7f14b415e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14baa54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14b415e930 0x7f14b4157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14baa54640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f14b4160020 con 0x7f14b415c2f0 2024-06-29T15:55:30.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14baa54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14b415cbb0 con 0x7f14b40b3740 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14baa54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f14b41601a0 con 0x7f14b415e930 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415c2f0 0x7f14b415e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415c2f0 0x7f14b415e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48838/0 (socket says 172.21.15.59:48838) 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 -- 172.21.15.59:0/380531090 learned_addr learned my addr 172.21.15.59:0/380531090 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b3fff640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b40b3740 0x7f14b40a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b8ff1640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14b415e930 0x7f14b4157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 -- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14b415e930 msgr2=0x7f14b4157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14b415e930 0x7f14b4157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 -- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b40b3740 msgr2=0x7f14b40a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b40b3740 0x7f14b40a9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 -- 172.21.15.59:0/380531090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b415ed40 con 0x7f14b415c2f0 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.270+0000 7f14b37fe640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415c2f0 0x7f14b415e3f0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f14a802ab00 tx=0x7f14a8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.271+0000 7f14b0ff9640 1 -- 172.21.15.59:0/380531090 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8031d60 con 0x7f14b415c2f0 2024-06-29T15:55:30.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.271+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b41f3af0 con 0x7f14b415c2f0 2024-06-29T15:55:30.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.271+0000 7f14b0ff9640 1 -- 172.21.15.59:0/380531090 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14a8033e10 con 0x7f14b415c2f0 2024-06-29T15:55:30.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.271+0000 7f14b0ff9640 1 -- 172.21.15.59:0/380531090 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8005260 con 0x7f14b415c2f0 2024-06-29T15:55:30.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.271+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b4158450 con 0x7f14b415c2f0 2024-06-29T15:55:30.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.272+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b41606c0 con 0x7f14b415c2f0 2024-06-29T15:55:30.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.273+0000 7f14b0ff9640 1 -- 172.21.15.59:0/380531090 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f14a8005400 con 0x7f14b415c2f0 2024-06-29T15:55:30.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.273+0000 7f14b0ff9640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f14a0076420 0x7f14a0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.273+0000 7f14b0ff9640 1 -- 172.21.15.59:0/380531090 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7f14a80b8ca0 con 0x7f14b415c2f0 2024-06-29T15:55:30.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.273+0000 7f14b3fff640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f14a0076420 0x7f14a0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.274+0000 7f14b3fff640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f14a0076420 0x7f14a0078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f14a40095f0 tx=0x7f14a4005f90 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.278+0000 7f14b0ff9640 1 -- 172.21.15.59:0/380531090 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f14b415ed40 con 0x7f14b415c2f0 2024-06-29T15:55:30.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.469+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14b40b1f10 con 0x7f14b415c2f0 2024-06-29T15:55:30.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.469+0000 7f14b0ff9640 1 -- 172.21.15.59:0/380531090 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) ==== 74+0+14933 (secure 0 0 0) 0x7f14a803f030 con 0x7f14b415c2f0 2024-06-29T15:55:30.469 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:30.470 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":196,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:55:29.130838+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/2399523179":"2024-06-30T15:55:26.118289+0000","172.21.15.153:0/1343183333":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6833/2924556144":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6832/2924556144":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/2895806264":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/3420796917":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3711582444":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/1469314326":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/3023327174":"2024-06-30T15:50:20.742173+0000","172.21.15.153:6833/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/2605088182":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/323020462":"2024-06-30T15:55:23.652598+0000","172.21.15.153:6832/1682798961":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/1412243249":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/3586261468":"2024-06-30T15:39:23.696245+0000","172.21.15.59:6832/2399523179":"2024-06-30T15:55:26.118289+0000","172.21.15.153:0/1851934149":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/771497135":"2024-06-30T15:55:23.652598+0000","172.21.15.153:0/1918645711":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/4098340029":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/2596446141":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/3821515452":"2024-06-30T15:39:23.696245+0000","172.21.15.153:0/1052117807":"2024-06-30T15:50:20.742173+0000","172.21.15.153:0/2595561638":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6833/1270545814":"2024-06-30T15:39:23.696245+0000","172.21.15.153:6832/1270545814":"2024-06-30T15:39:23.696245+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.471+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f14a0076420 msgr2=0x7f14a0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.471+0000 7f14baa54640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f14a0076420 0x7f14a0078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f14a40095f0 tx=0x7f14a4005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.471+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415c2f0 msgr2=0x7f14b415e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.471+0000 7f14baa54640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415c2f0 0x7f14b415e3f0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f14a802ab00 tx=0x7f14a8033c20 comp rx=0 tx=0).stop 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.471+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 shutdown_connections 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.471+0000 7f14baa54640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f14a0076420 0x7f14a0078930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.471+0000 7f14baa54640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14b415e930 0x7f14b4157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.471+0000 7f14baa54640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415c2f0 0x7f14b415e3f0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.472+0000 7f14baa54640 1 --2- 172.21.15.59:0/380531090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14b40b3740 0x7f14b40a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.472+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 >> 172.21.15.59:0/380531090 conn(0x7f14b406c950 msgr2=0x7f14b4141450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.472+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 shutdown_connections 2024-06-29T15:55:30.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.472+0000 7f14baa54640 1 -- 172.21.15.59:0/380531090 wait complete. 2024-06-29T15:55:30.483 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:55:30.484 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:55:30.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.605+0000 7f655bcce640 1 -- 172.21.15.59:0/2130637068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540b3ed0 msgr2=0x7f655415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.605+0000 7f655bcce640 1 --2- 172.21.15.59:0/2130637068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540b3ed0 0x7f655415e8d0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f65540d15d0 tx=0x7f654402a560 comp rx=0 tx=0).stop 2024-06-29T15:55:30.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.605+0000 7f655bcce640 1 -- 172.21.15.59:0/2130637068 shutdown_connections 2024-06-29T15:55:30.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.605+0000 7f655bcce640 1 --2- 172.21.15.59:0/2130637068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540b3ed0 0x7f655415e8d0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.605+0000 7f655bcce640 1 --2- 172.21.15.59:0/2130637068 >> v2:172.21.15.153:3300/0 conn(0x7f65540d23c0 0x7f65540b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.605+0000 7f655bcce640 1 -- 172.21.15.59:0/2130637068 >> 172.21.15.59:0/2130637068 conn(0x7f65540d3ee0 msgr2=0x7f65540d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.605+0000 7f655bcce640 1 -- 172.21.15.59:0/2130637068 shutdown_connections 2024-06-29T15:55:30.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.605+0000 7f655bcce640 1 -- 172.21.15.59:0/2130637068 wait complete. 2024-06-29T15:55:30.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.607+0000 7f655bcce640 1 Processor -- start 2024-06-29T15:55:30.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.607+0000 7f655bcce640 1 -- start start 2024-06-29T15:55:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f655bcce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65540b3ed0 0x7f65541e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f655bcce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540d23c0 0x7f65540cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f655bcce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f655415ee10 0x7f6554149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f655bcce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6554160240 con 0x7f65540d23c0 2024-06-29T15:55:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f655bcce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65541600c0 con 0x7f65540b3ed0 2024-06-29T15:55:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f655bcce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f65541603c0 con 0x7f655415ee10 2024-06-29T15:55:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f6559269640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540d23c0 0x7f65540cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f6559269640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540d23c0 0x7f65540cd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48864/0 (socket says 172.21.15.59:48864) 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f6559269640 1 -- 172.21.15.59:0/2104544612 learned_addr learned my addr 172.21.15.59:0/2104544612 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f6559a6a640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65540b3ed0 0x7f65541e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.608+0000 7f655a26b640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f655415ee10 0x7f6554149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f6559269640 1 -- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f655415ee10 msgr2=0x7f6554149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f6559269640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f655415ee10 0x7f6554149be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f6559269640 1 -- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65540b3ed0 msgr2=0x7f65541e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f6559269640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65540b3ed0 0x7f65541e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f6559269640 1 -- 172.21.15.59:0/2104544612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65540cc150 con 0x7f65540d23c0 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f6559269640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540d23c0 0x7f65540cd660 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f6544000c00 tx=0x7f65440039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f654a7fc640 1 -- 172.21.15.59:0/2104544612 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654403e030 con 0x7f65540d23c0 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f654a7fc640 1 -- 172.21.15.59:0/2104544612 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65440047a0 con 0x7f65540d23c0 2024-06-29T15:55:30.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65540cc5c0 con 0x7f65540d23c0 2024-06-29T15:55:30.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f654a7fc640 1 -- 172.21.15.59:0/2104544612 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544035040 con 0x7f65540d23c0 2024-06-29T15:55:30.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.609+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65540ccb80 con 0x7f65540d23c0 2024-06-29T15:55:30.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.610+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65540cc320 con 0x7f65540d23c0 2024-06-29T15:55:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.611+0000 7f654a7fc640 1 -- 172.21.15.59:0/2104544612 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f6544041030 con 0x7f65540d23c0 2024-06-29T15:55:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.611+0000 7f654a7fc640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6538076420 0x7f6538078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.612+0000 7f654a7fc640 1 -- 172.21.15.59:0/2104544612 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7f65440b85a0 con 0x7f65540d23c0 2024-06-29T15:55:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.612+0000 7f6559a6a640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6538076420 0x7f6538078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.612+0000 7f6559a6a640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6538076420 0x7f6538078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f655000a490 tx=0x7f655000a420 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.616+0000 7f654a7fc640 1 -- 172.21.15.59:0/2104544612 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f65540cc320 con 0x7f65540d23c0 2024-06-29T15:55:30.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.806+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f655415e8d0 con 0x7f65540d23c0 2024-06-29T15:55:30.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.806+0000 7f654a7fc640 1 -- 172.21.15.59:0/2104544612 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v196) ==== 93+0+10 (secure 0 0 0) 0x7f65540b42e0 con 0x7f65540d23c0 2024-06-29T15:55:30.806 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:55:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6538076420 msgr2=0x7f6538078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6538076420 0x7f6538078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f655000a490 tx=0x7f655000a420 comp rx=0 tx=0).stop 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540d23c0 msgr2=0x7f65540cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540d23c0 0x7f65540cd660 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f6544000c00 tx=0x7f65440039e0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 shutdown_connections 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6538076420 0x7f6538078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f655415ee10 0x7f6554149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65540d23c0 0x7f65540cd660 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 --2- 172.21.15.59:0/2104544612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65540b3ed0 0x7f65541e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 >> 172.21.15.59:0/2104544612 conn(0x7f65540d3ee0 msgr2=0x7f65540ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 shutdown_connections 2024-06-29T15:55:30.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.808+0000 7f655bcce640 1 -- 172.21.15.59:0/2104544612 wait complete. 2024-06-29T15:55:30.819 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-06-29T15:55:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.937+0000 7f5523017640 1 -- 172.21.15.59:0/2397050150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c08c190 msgr2=0x7f551c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.937+0000 7f5523017640 1 --2- 172.21.15.59:0/2397050150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c08c190 0x7f551c0cf080 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f550c005fd0 tx=0x7f550c02a4f0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.937+0000 7f5523017640 1 -- 172.21.15.59:0/2397050150 shutdown_connections 2024-06-29T15:55:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.937+0000 7f5523017640 1 --2- 172.21.15.59:0/2397050150 >> v2:172.21.15.153:3300/0 conn(0x7f551c0ce950 0x7f551c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.937+0000 7f5523017640 1 --2- 172.21.15.59:0/2397050150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c08c190 0x7f551c0cf080 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.937+0000 7f5523017640 1 -- 172.21.15.59:0/2397050150 >> 172.21.15.59:0/2397050150 conn(0x7f551c0b5610 msgr2=0x7f551c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.937+0000 7f5523017640 1 -- 172.21.15.59:0/2397050150 shutdown_connections 2024-06-29T15:55:30.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.937+0000 7f5523017640 1 -- 172.21.15.59:0/2397050150 wait complete. 2024-06-29T15:55:30.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.938+0000 7f5523017640 1 Processor -- start 2024-06-29T15:55:30.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.939+0000 7f5523017640 1 -- start start 2024-06-29T15:55:30.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f5523017640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c08c190 0x7f551c147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f5523017640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f551c0ce950 0x7f551c157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f5523017640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cf5c0 0x7f551c160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f5523017640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f551c0b7f60 con 0x7f551c0cf5c0 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f5523017640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f551c0b7de0 con 0x7f551c08c190 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f5523017640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f551c0b80e0 con 0x7f551c0ce950 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f55215b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cf5c0 0x7f551c160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f55215b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cf5c0 0x7f551c160dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48874/0 (socket says 172.21.15.59:48874) 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f55215b4640 1 -- 172.21.15.59:0/619315747 learned_addr learned my addr 172.21.15.59:0/619315747 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f551bfff640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f551c0ce950 0x7f551c157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f5520db3640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c08c190 0x7f551c147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f55215b4640 1 -- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f551c0ce950 msgr2=0x7f551c157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f55215b4640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f551c0ce950 0x7f551c157080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f55215b4640 1 -- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c08c190 msgr2=0x7f551c147c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f55215b4640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c08c190 0x7f551c147c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.940+0000 7f55215b4640 1 -- 172.21.15.59:0/619315747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c157df0 con 0x7f551c0cf5c0 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.941+0000 7f551bfff640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f551c0ce950 0x7f551c157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.941+0000 7f55215b4640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cf5c0 0x7f551c160dd0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f551400cab0 tx=0x7f551400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.941+0000 7f55197fa640 1 -- 172.21.15.59:0/619315747 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514007d90 con 0x7f551c0cf5c0 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.941+0000 7f55197fa640 1 -- 172.21.15.59:0/619315747 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f551400f560 con 0x7f551c0cf5c0 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.941+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551c1580d0 con 0x7f551c0cf5c0 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.941+0000 7f55197fa640 1 -- 172.21.15.59:0/619315747 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551400f9b0 con 0x7f551c0cf5c0 2024-06-29T15:55:30.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.941+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551c0cabd0 con 0x7f551c0cf5c0 2024-06-29T15:55:30.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.942+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551c0cf080 con 0x7f551c0cf5c0 2024-06-29T15:55:30.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.943+0000 7f55197fa640 1 -- 172.21.15.59:0/619315747 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f551400fb50 con 0x7f551c0cf5c0 2024-06-29T15:55:30.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.943+0000 7f55197fa640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f55080763f0 0x7f5508078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.943+0000 7f55197fa640 1 -- 172.21.15.59:0/619315747 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7f55140985e0 con 0x7f551c0cf5c0 2024-06-29T15:55:30.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.944+0000 7f5520db3640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f55080763f0 0x7f5508078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.944+0000 7f5520db3640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f55080763f0 0x7f5508078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f550c033000 tx=0x7f550c004730 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:30.948+0000 7f55197fa640 1 -- 172.21.15.59:0/619315747 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5514014070 con 0x7f551c0cf5c0 2024-06-29T15:55:31.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:31.210+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) -- 0x7f551c157b80 con 0x7f551c0cf5c0 2024-06-29T15:55:32.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.141+0000 7f55197fa640 1 -- 172.21.15.59:0/619315747 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3042) ==== 166+0+0 (secure 0 0 0) 0x7f5514061400 con 0x7f551c0cf5c0 2024-06-29T15:55:32.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f55080763f0 msgr2=0x7f5508078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f55080763f0 0x7f5508078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f550c033000 tx=0x7f550c004730 comp rx=0 tx=0).stop 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cf5c0 msgr2=0x7f551c160dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cf5c0 0x7f551c160dd0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f551400cab0 tx=0x7f551400cf70 comp rx=0 tx=0).stop 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 shutdown_connections 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f55080763f0 0x7f5508078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f551c0cf5c0 0x7f551c160dd0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f551c0ce950 0x7f551c157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 --2- 172.21.15.59:0/619315747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f551c08c190 0x7f551c147c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.144+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 >> 172.21.15.59:0/619315747 conn(0x7f551c0b5610 msgr2=0x7f551c0b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.145+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 shutdown_connections 2024-06-29T15:55:32.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.145+0000 7f5523017640 1 -- 172.21.15.59:0/619315747 wait complete. 2024-06-29T15:55:32.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:55:32.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.284+0000 7f909ff63640 1 -- 172.21.15.59:0/2973864807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980b3740 msgr2=0x7f90980b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.284+0000 7f909ff63640 1 --2- 172.21.15.59:0/2973864807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980b3740 0x7f90980b3b50 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f9098074b10 tx=0x7f908c02a560 comp rx=0 tx=0).stop 2024-06-29T15:55:32.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.284+0000 7f909ff63640 1 -- 172.21.15.59:0/2973864807 shutdown_connections 2024-06-29T15:55:32.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.284+0000 7f909ff63640 1 --2- 172.21.15.59:0/2973864807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980b3740 0x7f90980b3b50 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.284+0000 7f909ff63640 1 --2- 172.21.15.59:0/2973864807 >> v2:172.21.15.153:3300/0 conn(0x7f90981458a0 0x7f90980cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.284+0000 7f909ff63640 1 -- 172.21.15.59:0/2973864807 >> 172.21.15.59:0/2973864807 conn(0x7f909806c950 msgr2=0x7f9098073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.284+0000 7f909ff63640 1 -- 172.21.15.59:0/2973864807 shutdown_connections 2024-06-29T15:55:32.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.284+0000 7f909ff63640 1 -- 172.21.15.59:0/2973864807 wait complete. 2024-06-29T15:55:32.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.286+0000 7f909ff63640 1 Processor -- start 2024-06-29T15:55:32.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.286+0000 7f909ff63640 1 -- start start 2024-06-29T15:55:32.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909ff63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980b3740 0x7f9098156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909ff63640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f909815c2f0 0x7f90981f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909ff63640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980a9c70 0x7f909815d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909ff63640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9098160020 con 0x7f90980a9c70 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909ff63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f909815cbb0 con 0x7f90980b3740 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909ff63640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f90981601a0 con 0x7f909815c2f0 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909e500640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980a9c70 0x7f909815d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909e500640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980a9c70 0x7f909815d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48898/0 (socket says 172.21.15.59:48898) 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.287+0000 7f909e500640 1 -- 172.21.15.59:0/2317630483 learned_addr learned my addr 172.21.15.59:0/2317630483 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909d4fe640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f909815c2f0 0x7f90981f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909dcff640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980b3740 0x7f9098156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909e500640 1 -- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f909815c2f0 msgr2=0x7f90981f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909e500640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f909815c2f0 0x7f90981f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909e500640 1 -- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980b3740 msgr2=0x7f9098156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909e500640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980b3740 0x7f9098156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909e500640 1 -- 172.21.15.59:0/2317630483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9098157670 con 0x7f90980a9c70 2024-06-29T15:55:32.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909d4fe640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f909815c2f0 0x7f90981f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:32.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909e500640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980a9c70 0x7f909815d810 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f909000b410 tx=0x7f909000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:32.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f90967fc640 1 -- 172.21.15.59:0/2317630483 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090015030 con 0x7f90980a9c70 2024-06-29T15:55:32.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f90967fc640 1 -- 172.21.15.59:0/2317630483 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9090004110 con 0x7f90980a9c70 2024-06-29T15:55:32.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9098157950 con 0x7f90980a9c70 2024-06-29T15:55:32.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f90967fc640 1 -- 172.21.15.59:0/2317630483 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090011630 con 0x7f90980a9c70 2024-06-29T15:55:32.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.288+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9098158980 con 0x7f90980a9c70 2024-06-29T15:55:32.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.289+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90981606c0 con 0x7f90980a9c70 2024-06-29T15:55:32.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.290+0000 7f90967fc640 1 -- 172.21.15.59:0/2317630483 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f9090018070 con 0x7f90980a9c70 2024-06-29T15:55:32.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.290+0000 7f90967fc640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9084076420 0x7f9084078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.291+0000 7f90967fc640 1 -- 172.21.15.59:0/2317630483 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7f9090097810 con 0x7f90980a9c70 2024-06-29T15:55:32.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.291+0000 7f909dcff640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9084076420 0x7f9084078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.291+0000 7f909dcff640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9084076420 0x7f9084078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f908800a2e0 tx=0x7f908800a270 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:32.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.295+0000 7f90967fc640 1 -- 172.21.15.59:0/2317630483 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9090060d00 con 0x7f90980a9c70 2024-06-29T15:55:32.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.487+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f90980b1f10 con 0x7f90980a9c70 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.488+0000 7f90967fc640 1 -- 172.21.15.59:0/2317630483 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v67) ==== 59+0+1998 (secure 0 0 0) 0x7f9090060600 con 0x7f90980a9c70 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T15:55:32.488 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T15:55:32.489 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T15:55:32.490 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T15:55:32.490 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:55:32.490 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9084076420 msgr2=0x7f9084078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9084076420 0x7f9084078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f908800a2e0 tx=0x7f908800a270 comp rx=0 tx=0).stop 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980a9c70 msgr2=0x7f909815d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980a9c70 0x7f909815d810 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f909000b410 tx=0x7f909000b8d0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 shutdown_connections 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9084076420 0x7f9084078930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980a9c70 0x7f909815d810 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f909815c2f0 0x7f90981f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 --2- 172.21.15.59:0/2317630483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90980b3740 0x7f9098156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 >> 172.21.15.59:0/2317630483 conn(0x7f909806c950 msgr2=0x7f9098070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 shutdown_connections 2024-06-29T15:55:32.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.490+0000 7f909ff63640 1 -- 172.21.15.59:0/2317630483 wait complete. 2024-06-29T15:55:32.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:55:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.621+0000 7ff57fcac640 1 -- 172.21.15.59:0/1014687997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5781458a0 msgr2=0x7ff5780cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.621+0000 7ff57fcac640 1 --2- 172.21.15.59:0/1014687997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5781458a0 0x7ff5780cf870 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7ff5780b4b50 tx=0x7ff57402a560 comp rx=0 tx=0).stop 2024-06-29T15:55:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.622+0000 7ff57fcac640 1 -- 172.21.15.59:0/1014687997 shutdown_connections 2024-06-29T15:55:32.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.622+0000 7ff57fcac640 1 --2- 172.21.15.59:0/1014687997 >> v2:172.21.15.153:3300/0 conn(0x7ff5780b3b50 0x7ff5780b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.622+0000 7ff57fcac640 1 --2- 172.21.15.59:0/1014687997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5781458a0 0x7ff5780cf870 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.622+0000 7ff57fcac640 1 -- 172.21.15.59:0/1014687997 >> 172.21.15.59:0/1014687997 conn(0x7ff57806c950 msgr2=0x7ff578073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.622+0000 7ff57fcac640 1 -- 172.21.15.59:0/1014687997 shutdown_connections 2024-06-29T15:55:32.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.622+0000 7ff57fcac640 1 -- 172.21.15.59:0/1014687997 wait complete. 2024-06-29T15:55:32.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.623+0000 7ff57fcac640 1 Processor -- start 2024-06-29T15:55:32.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.624+0000 7ff57fcac640 1 -- start start 2024-06-29T15:55:32.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57fcac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5780b3740 0x7ff5780b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57fcac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5780b3b50 0x7ff578138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57fcac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff57813c6a0 0x7ff57815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57fcac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff5780b6150 con 0x7ff57813c6a0 2024-06-29T15:55:32.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57fcac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff5780b5fd0 con 0x7ff5780b3740 2024-06-29T15:55:32.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57fcac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff5780b62d0 con 0x7ff5780b3b50 2024-06-29T15:55:32.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff57813c6a0 0x7ff57815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff57813c6a0 0x7ff57815dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48908/0 (socket says 172.21.15.59:48908) 2024-06-29T15:55:32.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 -- 172.21.15.59:0/2887430681 learned_addr learned my addr 172.21.15.59:0/2887430681 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:32.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57d247640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5780b3b50 0x7ff578138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57da48640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5780b3740 0x7ff5780b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 -- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5780b3b50 msgr2=0x7ff578138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5780b3b50 0x7ff578138110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 -- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5780b3740 msgr2=0x7ff5780b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5780b3740 0x7ff5780b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 -- 172.21.15.59:0/2887430681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57813cba0 con 0x7ff57813c6a0 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.625+0000 7ff57e249640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff57813c6a0 0x7ff57815dca0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7ff57000caa0 tx=0x7ff57000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.626+0000 7ff56e7fc640 1 -- 172.21.15.59:0/2887430681 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57000adf0 con 0x7ff57813c6a0 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.626+0000 7ff56e7fc640 1 -- 172.21.15.59:0/2887430681 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff57000f540 con 0x7ff57813c6a0 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.626+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5781391c0 con 0x7ff57813c6a0 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.626+0000 7ff56e7fc640 1 -- 172.21.15.59:0/2887430681 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57000f990 con 0x7ff57813c6a0 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.626+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57813d100 con 0x7ff57813c6a0 2024-06-29T15:55:32.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.627+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5781458a0 con 0x7ff57813c6a0 2024-06-29T15:55:32.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.627+0000 7ff56e7fc640 1 -- 172.21.15.59:0/2887430681 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7ff57000fb30 con 0x7ff57813c6a0 2024-06-29T15:55:32.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.628+0000 7ff56e7fc640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff55c0763f0 0x7ff55c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.628+0000 7ff56e7fc640 1 -- 172.21.15.59:0/2887430681 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7ff570098750 con 0x7ff57813c6a0 2024-06-29T15:55:32.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.628+0000 7ff57da48640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff55c0763f0 0x7ff55c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.629+0000 7ff57da48640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff55c0763f0 0x7ff55c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff574033000 tx=0x7ff574004730 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:32.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.632+0000 7ff56e7fc640 1 -- 172.21.15.59:0/2887430681 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff570014030 con 0x7ff57813c6a0 2024-06-29T15:55:32.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.823+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff578141990 con 0x7ff57813c6a0 2024-06-29T15:55:32.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.827+0000 7ff56e7fc640 1 -- 172.21.15.59:0/2887430681 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v68)=0 v68) ==== 122+0+0 (secure 0 0 0) 0x7ff57000fe20 con 0x7ff57813c6a0 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff55c0763f0 msgr2=0x7ff55c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff55c0763f0 0x7ff55c078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff574033000 tx=0x7ff574004730 comp rx=0 tx=0).stop 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff57813c6a0 msgr2=0x7ff57815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff57813c6a0 0x7ff57815dca0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7ff57000caa0 tx=0x7ff57000cf60 comp rx=0 tx=0).stop 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 shutdown_connections 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff55c0763f0 0x7ff55c078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff57813c6a0 0x7ff57815dca0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5780b3b50 0x7ff578138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 --2- 172.21.15.59:0/2887430681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5780b3740 0x7ff5780b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.829+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 >> 172.21.15.59:0/2887430681 conn(0x7ff57806c950 msgr2=0x7ff5780d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.830+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 shutdown_connections 2024-06-29T15:55:32.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:32.830+0000 7ff57fcac640 1 -- 172.21.15.59:0/2887430681 wait complete. 2024-06-29T15:55:32.840 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:55:32.840 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:55:32.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:55:32.938 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:55:32.939 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:55:32.939 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:32.955 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:55:32.955 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:55:32.955 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:32.974 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:55:32.975 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:55:32.975 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:33.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:55:33.012 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:55:33.012 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:33.033 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:55:33.034 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:55:33.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.170+0000 7f7af7f43640 1 -- 172.21.15.59:0/2385436924 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7af00d23c0 msgr2=0x7f7af00b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.170+0000 7f7af7f43640 1 --2- 172.21.15.59:0/2385436924 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7af00d23c0 0x7f7af00b3990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0005fd0 tx=0x7f7ae002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.170+0000 7f7af7f43640 1 -- 172.21.15.59:0/2385436924 shutdown_connections 2024-06-29T15:55:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.170+0000 7f7af7f43640 1 --2- 172.21.15.59:0/2385436924 >> v2:172.21.15.153:3300/0 conn(0x7f7af015efe0 0x7f7af015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.170+0000 7f7af7f43640 1 --2- 172.21.15.59:0/2385436924 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7af00d23c0 0x7f7af00b3990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.170+0000 7f7af7f43640 1 -- 172.21.15.59:0/2385436924 >> 172.21.15.59:0/2385436924 conn(0x7f7af00d3ee0 msgr2=0x7f7af00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.170+0000 7f7af7f43640 1 -- 172.21.15.59:0/2385436924 shutdown_connections 2024-06-29T15:55:33.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.170+0000 7f7af7f43640 1 -- 172.21.15.59:0/2385436924 wait complete. 2024-06-29T15:55:33.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.172+0000 7f7af7f43640 1 Processor -- start 2024-06-29T15:55:33.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.172+0000 7f7af7f43640 1 -- start start 2024-06-29T15:55:33.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af7f43640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af00b3ed0 0x7f7af015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af7f43640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7af00d23c0 0x7f7af015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af7f43640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7af015efe0 0x7f7af015d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af7f43640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7af0160280 con 0x7f7af00b3ed0 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af7f43640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7af0160100 con 0x7f7af015efe0 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af7f43640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7af0160400 con 0x7f7af00d23c0 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af5cdf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af00b3ed0 0x7f7af015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af5cdf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af00b3ed0 0x7f7af015a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58460/0 (socket says 172.21.15.59:58460) 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af5cdf640 1 -- 172.21.15.59:0/527661933 learned_addr learned my addr 172.21.15.59:0/527661933 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af64e0640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7af015efe0 0x7f7af015d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af54de640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7af00d23c0 0x7f7af015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af5cdf640 1 -- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7af00d23c0 msgr2=0x7f7af015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af5cdf640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7af00d23c0 0x7f7af015cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af5cdf640 1 -- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7af015efe0 msgr2=0x7f7af015d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af5cdf640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7af015efe0 0x7f7af015d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.173+0000 7f7af5cdf640 1 -- 172.21.15.59:0/527661933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af015aad0 con 0x7f7af00b3ed0 2024-06-29T15:55:33.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.174+0000 7f7af5cdf640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af00b3ed0 0x7f7af015a590 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000f80 tx=0x7f7ae0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:33.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.174+0000 7f7aee7fc640 1 -- 172.21.15.59:0/527661933 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0031990 con 0x7f7af00b3ed0 2024-06-29T15:55:33.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.174+0000 7f7aee7fc640 1 -- 172.21.15.59:0/527661933 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ae0031c20 con 0x7f7af00b3ed0 2024-06-29T15:55:33.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.174+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af015daa0 con 0x7f7af00b3ed0 2024-06-29T15:55:33.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.174+0000 7f7aee7fc640 1 -- 172.21.15.59:0/527661933 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0033070 con 0x7f7af00b3ed0 2024-06-29T15:55:33.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.174+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af015e060 con 0x7f7af00b3ed0 2024-06-29T15:55:33.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.175+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af00b3990 con 0x7f7af00b3ed0 2024-06-29T15:55:33.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.176+0000 7f7aee7fc640 1 -- 172.21.15.59:0/527661933 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f7ae0033210 con 0x7f7af00b3ed0 2024-06-29T15:55:33.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.176+0000 7f7aee7fc640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f7adc0763f0 0x7f7adc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.177+0000 7f7aee7fc640 1 -- 172.21.15.59:0/527661933 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7f7ae00b90f0 con 0x7f7af00b3ed0 2024-06-29T15:55:33.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.177+0000 7f7af54de640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f7adc0763f0 0x7f7adc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.177+0000 7f7af54de640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f7adc0763f0 0x7f7adc078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7af006ce80 tx=0x7f7ae4009040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:33.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.181+0000 7f7aee7fc640 1 -- 172.21.15.59:0/527661933 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7ae003f030 con 0x7f7af00b3ed0 2024-06-29T15:55:33.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.408+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7af00d2950 con 0x7f7af00b3ed0 2024-06-29T15:55:33.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.409+0000 7f7aee7fc640 1 -- 172.21.15.59:0/527661933 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+3915 (secure 0 0 0) 0x7f7ae0081e90 con 0x7f7af00b3ed0 2024-06-29T15:55:33.409 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:33.409 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":99,"btime":"2024-06-29T15:55:27:121865+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8588,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3866646629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3866646629},{"type":"v1","addr":"172.21.15.153:6835","nonce":3866646629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9724,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/933275096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":933275096},{"type":"v1","addr":"172.21.15.59:6835","nonce":933275096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":9736,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1230229255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":1230229255},{"type":"v1","addr":"172.21.15.153:6837","nonce":1230229255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92},{"gid":10890,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2532081454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2532081454},{"type":"v1","addr":"172.21.15.59:6834","nonce":2532081454}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[]} 2024-06-29T15:55:33.410 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 99 2024-06-29T15:55:33.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f7adc0763f0 msgr2=0x7f7adc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f7adc0763f0 0x7f7adc078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7af006ce80 tx=0x7f7ae4009040 comp rx=0 tx=0).stop 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af00b3ed0 msgr2=0x7f7af015a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af00b3ed0 0x7f7af015a590 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000f80 tx=0x7f7ae0003970 comp rx=0 tx=0).stop 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 shutdown_connections 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f7adc0763f0 0x7f7adc078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7af015efe0 0x7f7af015d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7af00d23c0 0x7f7af015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 --2- 172.21.15.59:0/527661933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7af00b3ed0 0x7f7af015a590 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 >> 172.21.15.59:0/527661933 conn(0x7f7af00d3ee0 msgr2=0x7f7af0160940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 shutdown_connections 2024-06-29T15:55:33.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.411+0000 7f7af7f43640 1 -- 172.21.15.59:0/527661933 wait complete. 2024-06-29T15:55:33.422 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:55:33.422 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:55:33.423 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:55:33.423 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:55:33.423 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:55:33.423 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:55:33.423 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:55:33.423 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:55:33.423 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:55:33.424 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:55:33.424 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:55:33.424 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:55:33.424 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:55:33.424 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:55:33.424 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:55:33.424 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:55:33.433 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:55:33.433 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:55:33.436 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:55:33.436 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:55:33.437 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:55:33.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:55:33.438 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:55:33.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:55:33.478 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:55:33.478 DEBUG:teuthology.parallel:result is None 2024-06-29T15:55:33.479 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:55:33.479 DEBUG:teuthology.parallel:result is None 2024-06-29T15:55:33.480 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:55:33.480 DEBUG:teuthology.parallel:result is None 2024-06-29T15:55:33.481 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:55:33.481 DEBUG:teuthology.parallel:result is None 2024-06-29T15:55:33.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:55:33.569 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:55:33.570 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:55:33.570 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T15:55:33.570+0000 7fdfb11f6640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-06-29T15:55:33.575 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:55:33.575 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T15:55:33.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.606+0000 7fea9e9cb640 1 -- 172.21.15.59:0/456998654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980d23c0 msgr2=0x7fea980b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.606+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/456998654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980d23c0 0x7fea980b3990 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fea98161d40 tx=0x7fea8802a560 comp rx=0 tx=0).stop 2024-06-29T15:55:33.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.606+0000 7fea9e9cb640 1 -- 172.21.15.59:0/456998654 shutdown_connections 2024-06-29T15:55:33.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.606+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/456998654 >> v2:172.21.15.153:3300/0 conn(0x7fea9815efe0 0x7fea9815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.606+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/456998654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980d23c0 0x7fea980b3990 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.606+0000 7fea9e9cb640 1 -- 172.21.15.59:0/456998654 >> 172.21.15.59:0/456998654 conn(0x7fea980d3ee0 msgr2=0x7fea980d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:33.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.607+0000 7fea9e9cb640 1 -- 172.21.15.59:0/456998654 shutdown_connections 2024-06-29T15:55:33.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.607+0000 7fea9e9cb640 1 -- 172.21.15.59:0/456998654 wait complete. 2024-06-29T15:55:33.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.608+0000 7fea9e9cb640 1 Processor -- start 2024-06-29T15:55:33.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.608+0000 7fea9e9cb640 1 -- start start 2024-06-29T15:55:33.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea9e9cb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980b3ed0 0x7fea980c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea9e9cb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea980d23c0 0x7fea981ea590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea9e9cb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea9815efe0 0x7fea981eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea9e9cb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fea98160280 con 0x7fea980b3ed0 2024-06-29T15:55:33.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea9e9cb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fea98160100 con 0x7fea9815efe0 2024-06-29T15:55:33.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea9e9cb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fea98160400 con 0x7fea980d23c0 2024-06-29T15:55:33.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea97fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980b3ed0 0x7fea980c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea97fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980b3ed0 0x7fea980c6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58516/0 (socket says 172.21.15.59:58516) 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea97fff640 1 -- 172.21.15.59:0/4003525156 learned_addr learned my addr 172.21.15.59:0/4003525156 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea977fe640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea980d23c0 0x7fea981ea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea9cf68640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea9815efe0 0x7fea981eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea97fff640 1 -- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea980d23c0 msgr2=0x7fea981ea590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea97fff640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea980d23c0 0x7fea981ea590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea97fff640 1 -- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea9815efe0 msgr2=0x7fea981eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea97fff640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea9815efe0 0x7fea981eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.610+0000 7fea97fff640 1 -- 172.21.15.59:0/4003525156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea980c6f60 con 0x7fea980b3ed0 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.611+0000 7fea97fff640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980b3ed0 0x7fea980c6a20 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fea88034040 tx=0x7fea880039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.611+0000 7fea94ff9640 1 -- 172.21.15.59:0/4003525156 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8803e030 con 0x7fea980b3ed0 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.611+0000 7fea94ff9640 1 -- 172.21.15.59:0/4003525156 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea880047a0 con 0x7fea980b3ed0 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.611+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea98152780 con 0x7fea980b3ed0 2024-06-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.611+0000 7fea94ff9640 1 -- 172.21.15.59:0/4003525156 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88035040 con 0x7fea980b3ed0 2024-06-29T15:55:33.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.611+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea98156430 con 0x7fea980b3ed0 2024-06-29T15:55:33.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.612+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea980b3990 con 0x7fea980b3ed0 2024-06-29T15:55:33.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.613+0000 7fea94ff9640 1 -- 172.21.15.59:0/4003525156 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fea88041030 con 0x7fea980b3ed0 2024-06-29T15:55:33.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.613+0000 7fea94ff9640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fea84076420 0x7fea84078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.613+0000 7fea94ff9640 1 -- 172.21.15.59:0/4003525156 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6423+0+0 (secure 0 0 0) 0x7fea880b8490 con 0x7fea980b3ed0 2024-06-29T15:55:33.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.613+0000 7fea977fe640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fea84076420 0x7fea84078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.614+0000 7fea977fe640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fea84076420 0x7fea84078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea98151760 tx=0x7fea8c009060 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:33.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.618+0000 7fea94ff9640 1 -- 172.21.15.59:0/4003525156 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fea980b42e0 con 0x7fea980b3ed0 2024-06-29T15:55:33.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:33.823+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fea980d2950 con 0x7fea980b3ed0 2024-06-29T15:55:34.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.157+0000 7fea94ff9640 1 -- 172.21.15.59:0/4003525156 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v198) ==== 97+0+0 (secure 0 0 0) 0x7fea880812b0 con 0x7fea980b3ed0 2024-06-29T15:55:34.157 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fea84076420 msgr2=0x7fea84078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fea84076420 0x7fea84078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea98151760 tx=0x7fea8c009060 comp rx=0 tx=0).stop 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980b3ed0 msgr2=0x7fea980c6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980b3ed0 0x7fea980c6a20 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fea88034040 tx=0x7fea880039e0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 shutdown_connections 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fea84076420 0x7fea84078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea9815efe0 0x7fea981eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fea980d23c0 0x7fea981ea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 --2- 172.21.15.59:0/4003525156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea980b3ed0 0x7fea980c6a20 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 >> 172.21.15.59:0/4003525156 conn(0x7fea980d3ee0 msgr2=0x7fea980ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.159+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 shutdown_connections 2024-06-29T15:55:34.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.160+0000 7fea9e9cb640 1 -- 172.21.15.59:0/4003525156 wait complete. 2024-06-29T15:55:34.173 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:55:34.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.295+0000 7fba24ae3640 1 -- 172.21.15.59:0/2759318849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba200b2040 msgr2=0x7fba2006e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.295+0000 7fba24ae3640 1 --2- 172.21.15.59:0/2759318849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba200b2040 0x7fba2006e300 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fba14009830 tx=0x7fba1402a510 comp rx=0 tx=0).stop 2024-06-29T15:55:34.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.295+0000 7fba24ae3640 1 -- 172.21.15.59:0/2759318849 shutdown_connections 2024-06-29T15:55:34.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.295+0000 7fba24ae3640 1 --2- 172.21.15.59:0/2759318849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba200b2040 0x7fba2006e300 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.295+0000 7fba24ae3640 1 --2- 172.21.15.59:0/2759318849 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba20144470 0x7fba200b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.295+0000 7fba24ae3640 1 -- 172.21.15.59:0/2759318849 >> 172.21.15.59:0/2759318849 conn(0x7fba200b06e0 msgr2=0x7fba200b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.295+0000 7fba24ae3640 1 -- 172.21.15.59:0/2759318849 shutdown_connections 2024-06-29T15:55:34.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.295+0000 7fba24ae3640 1 -- 172.21.15.59:0/2759318849 wait complete. 2024-06-29T15:55:34.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.297+0000 7fba24ae3640 1 Processor -- start 2024-06-29T15:55:34.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.297+0000 7fba24ae3640 1 -- start start 2024-06-29T15:55:34.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba24ae3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba2006e840 0x7fba20158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba24ae3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba200b2040 0x7fba20138170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba24ae3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba20144470 0x7fba200a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba24ae3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fba200d0360 con 0x7fba2006e840 2024-06-29T15:55:34.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba24ae3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba200d01e0 con 0x7fba200b2040 2024-06-29T15:55:34.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba24ae3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fba2015ee80 con 0x7fba20144470 2024-06-29T15:55:34.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba2006e840 0x7fba20158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba2006e840 0x7fba20158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58526/0 (socket says 172.21.15.59:58526) 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1e59c640 1 -- 172.21.15.59:0/3014726260 learned_addr learned my addr 172.21.15.59:0/3014726260 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1ed9d640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba20144470 0x7fba200a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1dd9b640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba200b2040 0x7fba20138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1e59c640 1 -- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba20144470 msgr2=0x7fba200a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1e59c640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba20144470 0x7fba200a9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1e59c640 1 -- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba200b2040 msgr2=0x7fba20138170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1e59c640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba200b2040 0x7fba20138170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.298+0000 7fba1e59c640 1 -- 172.21.15.59:0/3014726260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2015a650 con 0x7fba2006e840 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.299+0000 7fba1e59c640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba2006e840 0x7fba20158e60 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fba1000e790 tx=0x7fba1000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.299+0000 7fba0effd640 1 -- 172.21.15.59:0/3014726260 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1000eea0 con 0x7fba2006e840 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.299+0000 7fba0effd640 1 -- 172.21.15.59:0/3014726260 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba1000cdb0 con 0x7fba2006e840 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.299+0000 7fba0effd640 1 -- 172.21.15.59:0/3014726260 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10010690 con 0x7fba2006e840 2024-06-29T15:55:34.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.299+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba20156a30 con 0x7fba2006e840 2024-06-29T15:55:34.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.299+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba20156ff0 con 0x7fba2006e840 2024-06-29T15:55:34.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.300+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba2015f3a0 con 0x7fba2006e840 2024-06-29T15:55:34.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.301+0000 7fba0effd640 1 -- 172.21.15.59:0/3014726260 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fba10010830 con 0x7fba2006e840 2024-06-29T15:55:34.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.301+0000 7fba0effd640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fba00076420 0x7fba00078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.302+0000 7fba0effd640 1 -- 172.21.15.59:0/3014726260 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 5305+0+0 (secure 0 0 0) 0x7fba10098220 con 0x7fba2006e840 2024-06-29T15:55:34.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.302+0000 7fba1dd9b640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fba00076420 0x7fba00078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.302+0000 7fba1dd9b640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fba00076420 0x7fba00078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba14034000 tx=0x7fba14004730 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:34.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.306+0000 7fba0effd640 1 -- 172.21.15.59:0/3014726260 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fba10061a60 con 0x7fba2006e840 2024-06-29T15:55:34.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.570+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fba2006d520 con 0x7fba2006e840 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.571+0000 7fba0effd640 1 -- 172.21.15.59:0/3014726260 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) ==== 80+0+5304 (secure 0 0 0) 0x7fba10061360 con 0x7fba2006e840 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.571 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.572 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T15:55:34.573 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.574 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T15:55:34.575 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.576 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T15:55:34.577 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.578 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.579 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.580 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T15:55:34.581 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:55:34.582 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.583 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T15:55:34.584 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:55:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fba00076420 msgr2=0x7fba00078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fba00076420 0x7fba00078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba14034000 tx=0x7fba14004730 comp rx=0 tx=0).stop 2024-06-29T15:55:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba2006e840 msgr2=0x7fba20158e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba2006e840 0x7fba20158e60 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fba1000e790 tx=0x7fba1000ec50 comp rx=0 tx=0).stop 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 shutdown_connections 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fba00076420 0x7fba00078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba20144470 0x7fba200a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba200b2040 0x7fba20138170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 --2- 172.21.15.59:0/3014726260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba2006e840 0x7fba20158e60 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 >> 172.21.15.59:0/3014726260 conn(0x7fba200b06e0 msgr2=0x7fba200b5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 shutdown_connections 2024-06-29T15:55:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.573+0000 7fba24ae3640 1 -- 172.21.15.59:0/3014726260 wait complete. 2024-06-29T15:55:34.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.706+0000 7ff9546df640 1 -- 172.21.15.59:0/287390697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff94c162800 msgr2=0x7ff94c162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.706+0000 7ff9546df640 1 --2- 172.21.15.59:0/287390697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff94c162800 0x7ff94c162c10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff944009860 tx=0x7ff94402a540 comp rx=0 tx=0).stop 2024-06-29T15:55:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.706+0000 7ff9546df640 1 -- 172.21.15.59:0/287390697 shutdown_connections 2024-06-29T15:55:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.706+0000 7ff9546df640 1 --2- 172.21.15.59:0/287390697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff94c162800 0x7ff94c162c10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.706+0000 7ff9546df640 1 --2- 172.21.15.59:0/287390697 >> v2:172.21.15.153:3300/0 conn(0x7ff94c0ce950 0x7ff94c1622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.706+0000 7ff9546df640 1 -- 172.21.15.59:0/287390697 >> 172.21.15.59:0/287390697 conn(0x7ff94c0b5610 msgr2=0x7ff94c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.706+0000 7ff9546df640 1 -- 172.21.15.59:0/287390697 shutdown_connections 2024-06-29T15:55:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.707+0000 7ff9546df640 1 -- 172.21.15.59:0/287390697 wait complete. 2024-06-29T15:55:34.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.708+0000 7ff9546df640 1 Processor -- start 2024-06-29T15:55:34.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.708+0000 7ff9546df640 1 -- start start 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff9546df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94c08c190 0x7ff94c1f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff9546df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff94c0ce950 0x7ff94c0c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff9546df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff94c162800 0x7ff94c0c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff9546df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff94c0b7f60 con 0x7ff94c162800 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff9546df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff94c0b7de0 con 0x7ff94c08c190 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff9546df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff94c0b80e0 con 0x7ff94c0ce950 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff94c162800 0x7ff94c0c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff94c162800 0x7ff94c0c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58534/0 (socket says 172.21.15.59:58534) 2024-06-29T15:55:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 -- 172.21.15.59:0/815337236 learned_addr learned my addr 172.21.15.59:0/815337236 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff95247b640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94c08c190 0x7ff94c1f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff951c7a640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff94c0ce950 0x7ff94c0c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 -- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff94c0ce950 msgr2=0x7ff94c0c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff94c0ce950 0x7ff94c0c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 -- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94c08c190 msgr2=0x7ff94c1f4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94c08c190 0x7ff94c1f4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 -- 172.21.15.59:0/815337236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94c1605a0 con 0x7ff94c162800 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.710+0000 7ff952c7c640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff94c162800 0x7ff94c0c9970 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7ff944009950 tx=0x7ff944033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.711+0000 7ff94affd640 1 -- 172.21.15.59:0/815337236 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff944005c90 con 0x7ff94c162800 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.711+0000 7ff94affd640 1 -- 172.21.15.59:0/815337236 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff944031e50 con 0x7ff94c162800 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.711+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff94c1f8aa0 con 0x7ff94c162800 2024-06-29T15:55:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.711+0000 7ff94affd640 1 -- 172.21.15.59:0/815337236 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff944005280 con 0x7ff94c162800 2024-06-29T15:55:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.711+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff94c1f9060 con 0x7ff94c162800 2024-06-29T15:55:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.712+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff94c1622c0 con 0x7ff94c162800 2024-06-29T15:55:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.712+0000 7ff94affd640 1 -- 172.21.15.59:0/815337236 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7ff944005420 con 0x7ff94c162800 2024-06-29T15:55:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.713+0000 7ff94affd640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff938076420 0x7ff938078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.713+0000 7ff94affd640 1 -- 172.21.15.59:0/815337236 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 5305+0+0 (secure 0 0 0) 0x7ff9440b8850 con 0x7ff94c162800 2024-06-29T15:55:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.713+0000 7ff95247b640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff938076420 0x7ff938078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.714+0000 7ff95247b640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff938076420 0x7ff938078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0b7620 tx=0x7ff93c009040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.717+0000 7ff94affd640 1 -- 172.21.15.59:0/815337236 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff94403f030 con 0x7ff94c162800 2024-06-29T15:55:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.909+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff94c0cf220 con 0x7ff94c162800 2024-06-29T15:55:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.910+0000 7ff94affd640 1 -- 172.21.15.59:0/815337236 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) ==== 74+0+13276 (secure 0 0 0) 0x7ff94408a110 con 0x7ff94c162800 2024-06-29T15:55:34.910 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:34.910 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":200,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:55:34.168255+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:34.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff938076420 msgr2=0x7ff938078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff938076420 0x7ff938078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0b7620 tx=0x7ff93c009040 comp rx=0 tx=0).stop 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff94c162800 msgr2=0x7ff94c0c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff94c162800 0x7ff94c0c9970 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7ff944009950 tx=0x7ff944033c20 comp rx=0 tx=0).stop 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 shutdown_connections 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7ff938076420 0x7ff938078930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff94c162800 0x7ff94c0c9970 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff94c0ce950 0x7ff94c0c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 --2- 172.21.15.59:0/815337236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff94c08c190 0x7ff94c1f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 >> 172.21.15.59:0/815337236 conn(0x7ff94c0b5610 msgr2=0x7ff94c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 shutdown_connections 2024-06-29T15:55:34.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:34.912+0000 7ff9546df640 1 -- 172.21.15.59:0/815337236 wait complete. 2024-06-29T15:55:34.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:55:34.924 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:55:35.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.045+0000 7f6d4d481640 1 -- 172.21.15.59:0/1092939884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d4806da00 msgr2=0x7f6d48073270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.045+0000 7f6d4d481640 1 --2- 172.21.15.59:0/1092939884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d4806da00 0x7f6d48073270 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009be0 tx=0x7f6d4002a8d0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.045+0000 7f6d4d481640 1 -- 172.21.15.59:0/1092939884 shutdown_connections 2024-06-29T15:55:35.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.045+0000 7f6d4d481640 1 --2- 172.21.15.59:0/1092939884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d4806da00 0x7f6d48073270 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.045+0000 7f6d4d481640 1 --2- 172.21.15.59:0/1092939884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d480cd9b0 0x7f6d4815f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.045+0000 7f6d4d481640 1 -- 172.21.15.59:0/1092939884 >> 172.21.15.59:0/1092939884 conn(0x7f6d480b24c0 msgr2=0x7f6d480cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.045+0000 7f6d4d481640 1 -- 172.21.15.59:0/1092939884 shutdown_connections 2024-06-29T15:55:35.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.045+0000 7f6d4d481640 1 -- 172.21.15.59:0/1092939884 wait complete. 2024-06-29T15:55:35.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.047+0000 7f6d4d481640 1 Processor -- start 2024-06-29T15:55:35.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.047+0000 7f6d4d481640 1 -- start start 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d4d481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d480cd9b0 0x7f6d48142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d4d481640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d481444c0 0x7f6d4813d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d4d481640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4815a900 0x7f6d480aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d4d481640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48073b40 con 0x7f6d480cd9b0 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d4d481640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d480739c0 con 0x7f6d481444c0 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d4d481640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6d48073cc0 con 0x7f6d4815a900 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d46ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d480cd9b0 0x7f6d48142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d46ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d480cd9b0 0x7f6d48142c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58552/0 (socket says 172.21.15.59:58552) 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.048+0000 7f6d46ffd640 1 -- 172.21.15.59:0/4081045686 learned_addr learned my addr 172.21.15.59:0/4081045686 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d477fe640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4815a900 0x7f6d480aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d467fc640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d481444c0 0x7f6d4813d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d46ffd640 1 -- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4815a900 msgr2=0x7f6d480aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d46ffd640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4815a900 0x7f6d480aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d46ffd640 1 -- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d481444c0 msgr2=0x7f6d4813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d46ffd640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d481444c0 0x7f6d4813d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d46ffd640 1 -- 172.21.15.59:0/4081045686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d48143680 con 0x7f6d480cd9b0 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d477fe640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4815a900 0x7f6d480aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d467fc640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d481444c0 0x7f6d4813d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d46ffd640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d480cd9b0 0x7f6d48142c00 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f6d3800d740 tx=0x7f6d3800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d3f7fe640 1 -- 172.21.15.59:0/4081045686 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d38017070 con 0x7f6d480cd9b0 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d3f7fe640 1 -- 172.21.15.59:0/4081045686 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d38009d20 con 0x7f6d480cd9b0 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d48143960 con 0x7f6d480cd9b0 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d3f7fe640 1 -- 172.21.15.59:0/4081045686 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d380106c0 con 0x7f6d480cd9b0 2024-06-29T15:55:35.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.049+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d48139e80 con 0x7f6d480cd9b0 2024-06-29T15:55:35.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.050+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d481399c0 con 0x7f6d480cd9b0 2024-06-29T15:55:35.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.051+0000 7f6d3f7fe640 1 -- 172.21.15.59:0/4081045686 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f6d3801a070 con 0x7f6d480cd9b0 2024-06-29T15:55:35.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.052+0000 7f6d3f7fe640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6d280763f0 0x7f6d28078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.052+0000 7f6d3f7fe640 1 -- 172.21.15.59:0/4081045686 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 5305+0+0 (secure 0 0 0) 0x7f6d380979b0 con 0x7f6d480cd9b0 2024-06-29T15:55:35.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.052+0000 7f6d467fc640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6d280763f0 0x7f6d28078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.052+0000 7f6d467fc640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6d280763f0 0x7f6d28078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d48141b00 tx=0x7f6d30009040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.056+0000 7f6d3f7fe640 1 -- 172.21.15.59:0/4081045686 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6d481399c0 con 0x7f6d480cd9b0 2024-06-29T15:55:35.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.252+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6d48074170 con 0x7f6d480cd9b0 2024-06-29T15:55:35.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.252+0000 7f6d3f7fe640 1 -- 172.21.15.59:0/4081045686 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v200) ==== 93+0+10 (secure 0 0 0) 0x7f6d380611f0 con 0x7f6d480cd9b0 2024-06-29T15:55:35.252 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:55:35.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6d280763f0 msgr2=0x7f6d28078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6d280763f0 0x7f6d28078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d48141b00 tx=0x7f6d30009040 comp rx=0 tx=0).stop 2024-06-29T15:55:35.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d480cd9b0 msgr2=0x7f6d48142c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d480cd9b0 0x7f6d48142c00 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f6d3800d740 tx=0x7f6d3800dc00 comp rx=0 tx=0).stop 2024-06-29T15:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 shutdown_connections 2024-06-29T15:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f6d280763f0 0x7f6d28078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6d4815a900 0x7f6d480aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d481444c0 0x7f6d4813d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 --2- 172.21.15.59:0/4081045686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d480cd9b0 0x7f6d48142c00 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 >> 172.21.15.59:0/4081045686 conn(0x7f6d480b24c0 msgr2=0x7f6d4815e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 shutdown_connections 2024-06-29T15:55:35.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.254+0000 7f6d4d481640 1 -- 172.21.15.59:0/4081045686 wait complete. 2024-06-29T15:55:35.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:55:35.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.384+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/3274938683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde98003a70 msgr2=0x7fde98003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.384+0000 7fdeb3a1f640 1 --2- 172.21.15.59:0/3274938683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde98003a70 0x7fde98003ec0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0d2e40 tx=0x7fde98005830 comp rx=0 tx=0).stop 2024-06-29T15:55:35.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.384+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/3274938683 shutdown_connections 2024-06-29T15:55:35.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.384+0000 7fdeb3a1f640 1 --2- 172.21.15.59:0/3274938683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde98003a70 0x7fde98003ec0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.384+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/3274938683 >> 172.21.15.59:0/3274938683 conn(0x7fdeac0d3ee0 msgr2=0x7fdeac0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.384+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/3274938683 shutdown_connections 2024-06-29T15:55:35.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.384+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/3274938683 wait complete. 2024-06-29T15:55:35.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.386+0000 7fdeb3a1f640 1 Processor -- start 2024-06-29T15:55:35.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.386+0000 7fdeb3a1f640 1 -- start start 2024-06-29T15:55:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb3a1f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde98003a70 0x7fdeac158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb3a1f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cc5a0 0x7fdeac0cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb3a1f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac14de70 0x7fdeac155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb3a1f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac15ec60 con 0x7fdeac14de70 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb3a1f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac15eae0 con 0x7fde98003a70 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb3a1f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdeac0b3990 con 0x7fdeac0cc5a0 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb1fbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac14de70 0x7fdeac155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb1fbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac14de70 0x7fdeac155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58572/0 (socket says 172.21.15.59:58572) 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb1fbc640 1 -- 172.21.15.59:0/961977180 learned_addr learned my addr 172.21.15.59:0/961977180 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb17bb640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde98003a70 0x7fdeac158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.387+0000 7fdeb0fba640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cc5a0 0x7fdeac0cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdeb1fbc640 1 -- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cc5a0 msgr2=0x7fdeac0cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdeb1fbc640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cc5a0 0x7fdeac0cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdeb1fbc640 1 -- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde98003a70 msgr2=0x7fdeac158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdeb1fbc640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde98003a70 0x7fdeac158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdeb1fbc640 1 -- 172.21.15.59:0/961977180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac155330 con 0x7fdeac14de70 2024-06-29T15:55:35.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdeb1fbc640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac14de70 0x7fdeac155dd0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fdeac15e830 tx=0x7fdea400c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdea1ffb640 1 -- 172.21.15.59:0/961977180 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4019070 con 0x7fdeac14de70 2024-06-29T15:55:35.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdea1ffb640 1 -- 172.21.15.59:0/961977180 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdea40063c0 con 0x7fdeac14de70 2024-06-29T15:55:35.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeac0ccd70 con 0x7fdeac14de70 2024-06-29T15:55:35.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdea1ffb640 1 -- 172.21.15.59:0/961977180 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4003a90 con 0x7fdeac14de70 2024-06-29T15:55:35.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.388+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeac1555d0 con 0x7fdeac14de70 2024-06-29T15:55:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.390+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeac0ccae0 con 0x7fdeac14de70 2024-06-29T15:55:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.390+0000 7fdea1ffb640 1 -- 172.21.15.59:0/961977180 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fdea401c070 con 0x7fdeac14de70 2024-06-29T15:55:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.391+0000 7fdea1ffb640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde900763f0 0x7fde90078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.391+0000 7fdea1ffb640 1 -- 172.21.15.59:0/961977180 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 5305+0+0 (secure 0 0 0) 0x7fdea4096af0 con 0x7fdeac14de70 2024-06-29T15:55:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.391+0000 7fdeb17bb640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde900763f0 0x7fde90078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.392+0000 7fdeb17bb640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde900763f0 0x7fde90078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdea8033000 tx=0x7fdea8038040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.395+0000 7fdea1ffb640 1 -- 172.21.15.59:0/961977180 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdea405fce0 con 0x7fdeac14de70 2024-06-29T15:55:35.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.588+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdeac0d2950 con 0x7fdeac14de70 2024-06-29T15:55:35.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.588+0000 7fdea1ffb640 1 -- 172.21.15.59:0/961977180 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v200) ==== 84+0+65 (secure 0 0 0) 0x7fdea4063fe0 con 0x7fdeac14de70 2024-06-29T15:55:35.589 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:35.589 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:55:35.589 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:35.589 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:55:35.589 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:55:35.589 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:35.589 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde900763f0 msgr2=0x7fde90078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde900763f0 0x7fde90078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdea8033000 tx=0x7fdea8038040 comp rx=0 tx=0).stop 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac14de70 msgr2=0x7fdeac155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac14de70 0x7fdeac155dd0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fdeac15e830 tx=0x7fdea400c630 comp rx=0 tx=0).stop 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 shutdown_connections 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde900763f0 0x7fde90078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac14de70 0x7fdeac155dd0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cc5a0 0x7fdeac0cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 --2- 172.21.15.59:0/961977180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde98003a70 0x7fdeac158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 >> 172.21.15.59:0/961977180 conn(0x7fdeac0d3ee0 msgr2=0x7fdeac15fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.590+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 shutdown_connections 2024-06-29T15:55:35.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.591+0000 7fdeb3a1f640 1 -- 172.21.15.59:0/961977180 wait complete. 2024-06-29T15:55:35.602 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:55:35.602 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:55:35.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.725+0000 7f7593209640 1 -- 172.21.15.59:0/1843222271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 msgr2=0x7f7574004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.725+0000 7f7593209640 1 --2- 172.21.15.59:0/1843222271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 0x7f7574004c10 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f7580009b20 tx=0x7f7574005830 comp rx=0 tx=0).stop 2024-06-29T15:55:35.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.725+0000 7f7593209640 1 -- 172.21.15.59:0/1843222271 shutdown_connections 2024-06-29T15:55:35.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.725+0000 7f7593209640 1 --2- 172.21.15.59:0/1843222271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 0x7f7574004c10 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.725+0000 7f7593209640 1 -- 172.21.15.59:0/1843222271 >> 172.21.15.59:0/1843222271 conn(0x7f758c0d0f90 msgr2=0x7f758c0d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.725+0000 7f7593209640 1 -- 172.21.15.59:0/1843222271 shutdown_connections 2024-06-29T15:55:35.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.725+0000 7f7593209640 1 -- 172.21.15.59:0/1843222271 wait complete. 2024-06-29T15:55:35.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.727+0000 7f7593209640 1 Processor -- start 2024-06-29T15:55:35.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.727+0000 7f7593209640 1 -- start start 2024-06-29T15:55:35.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7593209640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 0x7f758c15f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7593209640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f758c1f44e0 0x7f758c151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7593209640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f758c1f48f0 0x7f758c1523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7593209640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f758c0b70f0 con 0x7f7574004800 2024-06-29T15:55:35.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7593209640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f758c0b6f70 con 0x7f758c1f44e0 2024-06-29T15:55:35.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7593209640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f758c0b7270 con 0x7f758c1f48f0 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 0x7f758c15f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 0x7f758c15f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58588/0 (socket says 172.21.15.59:58588) 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 -- 172.21.15.59:0/382228989 learned_addr learned my addr 172.21.15.59:0/382228989 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f758bfff640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f758c1f44e0 0x7f758c151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f75917a6640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f758c1f48f0 0x7f758c1523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 -- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f758c1f48f0 msgr2=0x7f758c1523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f758c1f48f0 0x7f758c1523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 -- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f758c1f44e0 msgr2=0x7f758c151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f758c1f44e0 0x7f758c151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 -- 172.21.15.59:0/382228989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758c151590 con 0x7f7574004800 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f758bfff640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f758c1f44e0 0x7f758c151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:35.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f75917a6640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f758c1f48f0 0x7f758c1523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:35.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7590fa5640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 0x7f758c15f520 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f757c006680 tx=0x7f757c0066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f75897fa640 1 -- 172.21.15.59:0/382228989 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c015070 con 0x7f7574004800 2024-06-29T15:55:35.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f75897fa640 1 -- 172.21.15.59:0/382228989 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f757c011920 con 0x7f7574004800 2024-06-29T15:55:35.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f758c160100 con 0x7f7574004800 2024-06-29T15:55:35.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.729+0000 7f75897fa640 1 -- 172.21.15.59:0/382228989 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c011d70 con 0x7f7574004800 2024-06-29T15:55:35.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.730+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f758c1606c0 con 0x7f7574004800 2024-06-29T15:55:35.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.731+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f758c0d2960 con 0x7f7574004800 2024-06-29T15:55:35.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.731+0000 7f75897fa640 1 -- 172.21.15.59:0/382228989 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f757c018070 con 0x7f7574004800 2024-06-29T15:55:35.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.732+0000 7f75897fa640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f75780763f0 0x7f7578078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.732+0000 7f75897fa640 1 -- 172.21.15.59:0/382228989 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 5305+0+0 (secure 0 0 0) 0x7f757c098580 con 0x7f7574004800 2024-06-29T15:55:35.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.732+0000 7f758bfff640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f75780763f0 0x7f7578078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.733+0000 7f758bfff640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f75780763f0 0x7f7578078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7580035000 tx=0x7f75800046c0 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.737+0000 7f75897fa640 1 -- 172.21.15.59:0/382228989 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f758c151590 con 0x7f7574004800 2024-06-29T15:55:35.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:35.928+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f758c075b40 con 0x7f7574004800 2024-06-29T15:55:36.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.162+0000 7f75897fa640 1 -- 172.21.15.59:0/382228989 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v201) ==== 138+0+0 (secure 0 0 0) 0x7f757c061dc0 con 0x7f7574004800 2024-06-29T15:55:36.162 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T15:55:36.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f75780763f0 msgr2=0x7f7578078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f75780763f0 0x7f7578078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7580035000 tx=0x7f75800046c0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 msgr2=0x7f758c15f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 0x7f758c15f520 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f757c006680 tx=0x7f757c0066b0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 shutdown_connections 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f75780763f0 0x7f7578078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f758c1f48f0 0x7f758c1523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f758c1f44e0 0x7f758c151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 --2- 172.21.15.59:0/382228989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7574004800 0x7f758c15f520 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 >> 172.21.15.59:0/382228989 conn(0x7f758c0d0f90 msgr2=0x7f758c0b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 shutdown_connections 2024-06-29T15:55:36.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.164+0000 7f7593209640 1 -- 172.21.15.59:0/382228989 wait complete. 2024-06-29T15:55:36.180 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:55:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.306+0000 7f241a70a640 1 -- 172.21.15.59:0/1695154145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23fc003a50 msgr2=0x7f23fc003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.306+0000 7f241a70a640 1 --2- 172.21.15.59:0/1695154145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23fc003a50 0x7f23fc003ec0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f24040016c0 tx=0x7f23fc005830 comp rx=0 tx=0).stop 2024-06-29T15:55:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.306+0000 7f241a70a640 1 -- 172.21.15.59:0/1695154145 shutdown_connections 2024-06-29T15:55:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.306+0000 7f241a70a640 1 --2- 172.21.15.59:0/1695154145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23fc003a50 0x7f23fc003ec0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.306+0000 7f241a70a640 1 -- 172.21.15.59:0/1695154145 >> 172.21.15.59:0/1695154145 conn(0x7f2414072740 msgr2=0x7f24140b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.306+0000 7f241a70a640 1 -- 172.21.15.59:0/1695154145 shutdown_connections 2024-06-29T15:55:36.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.306+0000 7f241a70a640 1 -- 172.21.15.59:0/1695154145 wait complete. 2024-06-29T15:55:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.308+0000 7f241a70a640 1 Processor -- start 2024-06-29T15:55:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.308+0000 7f241a70a640 1 -- start start 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.309+0000 7f241a70a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23fc003a50 0x7f24140997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.309+0000 7f241a70a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2414154890 0x7f2414154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.309+0000 7f241a70a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241409ac40 0x7f241409b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f241a70a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f24140ce4d0 con 0x7f241409ac40 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f241a70a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f24140ce350 con 0x7f2414154890 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f241a70a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f24140ce650 con 0x7f23fc003a50 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241409ac40 0x7f241409b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241409ac40 0x7f241409b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58594/0 (socket says 172.21.15.59:58594) 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 -- 172.21.15.59:0/1818076961 learned_addr learned my addr 172.21.15.59:0/1818076961 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f24137fe640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2414154890 0x7f2414154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2413fff640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23fc003a50 0x7f24140997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 -- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23fc003a50 msgr2=0x7f24140997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23fc003a50 0x7f24140997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 -- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2414154890 msgr2=0x7f2414154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2414154890 0x7f2414154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 -- 172.21.15.59:0/1818076961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2414154000 con 0x7f241409ac40 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2418ca7640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241409ac40 0x7f241409b050 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f2414072310 tx=0x7f240c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2410ff9640 1 -- 172.21.15.59:0/1818076961 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c00ca10 con 0x7f241409ac40 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2410ff9640 1 -- 172.21.15.59:0/1818076961 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f240c017070 con 0x7f241409ac40 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24141551e0 con 0x7f241409ac40 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f2410ff9640 1 -- 172.21.15.59:0/1818076961 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c011460 con 0x7f241409ac40 2024-06-29T15:55:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.310+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2414153760 con 0x7f241409ac40 2024-06-29T15:55:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.312+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24141531f0 con 0x7f241409ac40 2024-06-29T15:55:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.312+0000 7f2410ff9640 1 -- 172.21.15.59:0/1818076961 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f240c011600 con 0x7f241409ac40 2024-06-29T15:55:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.313+0000 7f2410ff9640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f24000763f0 0x7f2400078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.313+0000 7f2410ff9640 1 -- 172.21.15.59:0/1818076961 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 5675+0+0 (secure 0 0 0) 0x7f240c097ea0 con 0x7f241409ac40 2024-06-29T15:55:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.313+0000 7f2413fff640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f24000763f0 0x7f2400078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.314+0000 7f2413fff640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f24000763f0 0x7f2400078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2404032000 tx=0x7f240402c040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:36.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.318+0000 7f2410ff9640 1 -- 172.21.15.59:0/1818076961 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f240c0614f0 con 0x7f241409ac40 2024-06-29T15:55:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:36.511+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f2414072d60 con 0x7f241409ac40 2024-06-29T15:55:37.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.164+0000 7f2410ff9640 1 -- 172.21.15.59:0/1818076961 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v202) ==== 170+0+0 (secure 0 0 0) 0x7f2414072d60 con 0x7f241409ac40 2024-06-29T15:55:37.165 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T15:55:37.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f24000763f0 msgr2=0x7f2400078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f24000763f0 0x7f2400078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2404032000 tx=0x7f240402c040 comp rx=0 tx=0).stop 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241409ac40 msgr2=0x7f241409b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241409ac40 0x7f241409b050 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f2414072310 tx=0x7f240c00ec50 comp rx=0 tx=0).stop 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 shutdown_connections 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f24000763f0 0x7f2400078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241409ac40 0x7f241409b050 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2414154890 0x7f2414154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 --2- 172.21.15.59:0/1818076961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23fc003a50 0x7f24140997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.167+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 >> 172.21.15.59:0/1818076961 conn(0x7f2414072740 msgr2=0x7f2414146930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.168+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 shutdown_connections 2024-06-29T15:55:37.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.168+0000 7f241a70a640 1 -- 172.21.15.59:0/1818076961 wait complete. 2024-06-29T15:55:37.193 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:55:37.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.321+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/2288151812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac06da00 msgr2=0x7fdeac073270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.321+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/2288151812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac06da00 0x7fdeac073270 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fdeac070d30 tx=0x7fdea402aa30 comp rx=0 tx=0).stop 2024-06-29T15:55:37.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.321+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/2288151812 shutdown_connections 2024-06-29T15:55:37.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.321+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/2288151812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac06da00 0x7fdeac073270 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.321+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/2288151812 >> v2:172.21.15.153:3300/0 conn(0x7fdeac0cd9b0 0x7fdeac15f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.321+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/2288151812 >> 172.21.15.59:0/2288151812 conn(0x7fdeac0b24c0 msgr2=0x7fdeac0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:37.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.321+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/2288151812 shutdown_connections 2024-06-29T15:55:37.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.321+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/2288151812 wait complete. 2024-06-29T15:55:37.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.323+0000 7fdeb1e5e640 1 Processor -- start 2024-06-29T15:55:37.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.323+0000 7fdeb1e5e640 1 -- start start 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeb1e5e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cd9b0 0x7fdeac0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeb1e5e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac1444c0 0x7fdeac0a9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeb1e5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac0ae940 0x7fdeac0aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeb1e5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac073b40 con 0x7fdeac0ae940 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeb1e5e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac0739c0 con 0x7fdeac1444c0 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeb1e5e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdeac073cc0 con 0x7fdeac0cd9b0 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeabfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac0ae940 0x7fdeac0aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeabfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac0ae940 0x7fdeac0aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58616/0 (socket says 172.21.15.59:58616) 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeabfff640 1 -- 172.21.15.59:0/3872620119 learned_addr learned my addr 172.21.15.59:0/3872620119 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeabfff640 1 -- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cd9b0 msgr2=0x7fdeac0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeabfff640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cd9b0 0x7fdeac0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeabfff640 1 -- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac1444c0 msgr2=0x7fdeac0a9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.324+0000 7fdeaaffd640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac1444c0 0x7fdeac0a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.325+0000 7fdeabfff640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac1444c0 0x7fdeac0a9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.325+0000 7fdeabfff640 1 -- 172.21.15.59:0/3872620119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac0af290 con 0x7fdeac0ae940 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.325+0000 7fdeabfff640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac0ae940 0x7fdeac0aed50 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fdea402b020 tx=0x7fdea4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.325+0000 7fde93fff640 1 -- 172.21.15.59:0/3872620119 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4033050 con 0x7fdeac0ae940 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.325+0000 7fde93fff640 1 -- 172.21.15.59:0/3872620119 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdea4004450 con 0x7fdeac0ae940 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.325+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeac1f2bd0 con 0x7fdeac0ae940 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.325+0000 7fde93fff640 1 -- 172.21.15.59:0/3872620119 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4004840 con 0x7fdeac0ae940 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.325+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeac0aa0b0 con 0x7fdeac0ae940 2024-06-29T15:55:37.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.326+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeac0741e0 con 0x7fdeac0ae940 2024-06-29T15:55:37.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.327+0000 7fde93fff640 1 -- 172.21.15.59:0/3872620119 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fdea4036070 con 0x7fdeac0ae940 2024-06-29T15:55:37.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.328+0000 7fde93fff640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde980763f0 0x7fde98078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:37.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.328+0000 7fde93fff640 1 -- 172.21.15.59:0/3872620119 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 6057+0+0 (secure 0 0 0) 0x7fdea40b9210 con 0x7fdeac0ae940 2024-06-29T15:55:37.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.328+0000 7fdeab7fe640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde980763f0 0x7fde98078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:37.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.329+0000 7fdeab7fe640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde980763f0 0x7fde98078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00b990 tx=0x7fde9c00b370 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:37.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.332+0000 7fde93fff640 1 -- 172.21.15.59:0/3872620119 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdeac0af290 con 0x7fdeac0ae940 2024-06-29T15:55:37.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:37.566+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdeac0b1290 con 0x7fdeac0ae940 2024-06-29T15:55:38.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.192+0000 7fde93fff640 1 -- 172.21.15.59:0/3872620119 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v102) ==== 174+0+0 (secure 0 0 0) 0x7fdea40828d0 con 0x7fdeac0ae940 2024-06-29T15:55:38.192 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 25 and data pool 26 2024-06-29T15:55:38.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.202+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde980763f0 msgr2=0x7fde98078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.202+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde980763f0 0x7fde98078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde9c00b990 tx=0x7fde9c00b370 comp rx=0 tx=0).stop 2024-06-29T15:55:38.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.202+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac0ae940 msgr2=0x7fdeac0aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.202+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac0ae940 0x7fdeac0aed50 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fdea402b020 tx=0x7fdea4003f10 comp rx=0 tx=0).stop 2024-06-29T15:55:38.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.203+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 shutdown_connections 2024-06-29T15:55:38.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.203+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fde980763f0 0x7fde98078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.203+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdeac0ae940 0x7fdeac0aed50 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.203+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac1444c0 0x7fdeac0a9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.203+0000 7fdeb1e5e640 1 --2- 172.21.15.59:0/3872620119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdeac0cd9b0 0x7fdeac0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.203+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 >> 172.21.15.59:0/3872620119 conn(0x7fdeac0b24c0 msgr2=0x7fdeac140000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.204+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 shutdown_connections 2024-06-29T15:55:38.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.204+0000 7fdeb1e5e640 1 -- 172.21.15.59:0/3872620119 wait complete. 2024-06-29T15:55:38.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:38.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.344+0000 7f963063f640 1 -- 172.21.15.59:0/1032984282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962815efe0 msgr2=0x7f962815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.344+0000 7f963063f640 1 --2- 172.21.15.59:0/1032984282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962815efe0 0x7f962815f9b0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f9620005fd0 tx=0x7f962002aa90 comp rx=0 tx=0).stop 2024-06-29T15:55:38.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.345+0000 7f963063f640 1 -- 172.21.15.59:0/1032984282 shutdown_connections 2024-06-29T15:55:38.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.345+0000 7f963063f640 1 --2- 172.21.15.59:0/1032984282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f962815efe0 0x7f962815f9b0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.345+0000 7f963063f640 1 --2- 172.21.15.59:0/1032984282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96280d23c0 0x7f96280b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.345+0000 7f963063f640 1 -- 172.21.15.59:0/1032984282 >> 172.21.15.59:0/1032984282 conn(0x7f96280d3ee0 msgr2=0x7f96280d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.345+0000 7f963063f640 1 -- 172.21.15.59:0/1032984282 shutdown_connections 2024-06-29T15:55:38.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.345+0000 7f963063f640 1 -- 172.21.15.59:0/1032984282 wait complete. 2024-06-29T15:55:38.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.346+0000 7f963063f640 1 Processor -- start 2024-06-29T15:55:38.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.346+0000 7f963063f640 1 -- start start 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f963063f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96280b3ed0 0x7f9628148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f963063f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96280d23c0 0x7f962815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f963063f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962815efe0 0x7f96281e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f963063f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9628160280 con 0x7f96280b3ed0 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f963063f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9628160100 con 0x7f96280d23c0 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f963063f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9628160400 con 0x7f962815efe0 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96280b3ed0 0x7f9628148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96280b3ed0 0x7f9628148db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58632/0 (socket says 172.21.15.59:58632) 2024-06-29T15:55:38.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 -- 172.21.15.59:0/1316475013 learned_addr learned my addr 172.21.15.59:0/1316475013 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962ebdc640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962815efe0 0x7f96281e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962dbda640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96280d23c0 0x7f962815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 -- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962815efe0 msgr2=0x7f96281e9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962815efe0 0x7f96281e9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 -- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96280d23c0 msgr2=0x7f962815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96280d23c0 0x7f962815dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 -- 172.21.15.59:0/1316475013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96281e8ca0 con 0x7f96280b3ed0 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.348+0000 7f962e3db640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96280b3ed0 0x7f9628148db0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f962400e790 tx=0x7f962400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.349+0000 7f961effd640 1 -- 172.21.15.59:0/1316475013 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962400eea0 con 0x7f96280b3ed0 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.349+0000 7f961effd640 1 -- 172.21.15.59:0/1316475013 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f962400ccc0 con 0x7f96280b3ed0 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.349+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96281492f0 con 0x7f96280b3ed0 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.349+0000 7f961effd640 1 -- 172.21.15.59:0/1316475013 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9624004280 con 0x7f96280b3ed0 2024-06-29T15:55:38.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.349+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96281e9220 con 0x7f96280b3ed0 2024-06-29T15:55:38.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.350+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96280b3990 con 0x7f96280b3ed0 2024-06-29T15:55:38.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.350+0000 7f961effd640 1 -- 172.21.15.59:0/1316475013 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f96240035f0 con 0x7f96280b3ed0 2024-06-29T15:55:38.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.351+0000 7f961effd640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f960c0763f0 0x7f960c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.351+0000 7f961effd640 1 -- 172.21.15.59:0/1316475013 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 6157+0+0 (secure 0 0 0) 0x7f9624098480 con 0x7f96280b3ed0 2024-06-29T15:55:38.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.351+0000 7f962dbda640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f960c0763f0 0x7f960c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.352+0000 7f962dbda640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f960c0763f0 0x7f960c078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f96280b3e80 tx=0x7f9618009040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:38.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.356+0000 7f961effd640 1 -- 172.21.15.59:0/1316475013 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9624061320 con 0x7f96280b3ed0 2024-06-29T15:55:38.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.550+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96280d2950 con 0x7f96280b3ed0 2024-06-29T15:55:38.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.551+0000 7f961effd640 1 -- 172.21.15.59:0/1316475013 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) ==== 74+0+17003 (secure 0 0 0) 0x7f9624065620 con 0x7f96280b3ed0 2024-06-29T15:55:38.551 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:38.552 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":203,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:55:37.567403+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.935517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.518490+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f960c0763f0 msgr2=0x7f960c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f960c0763f0 0x7f960c078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f96280b3e80 tx=0x7f9618009040 comp rx=0 tx=0).stop 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96280b3ed0 msgr2=0x7f9628148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96280b3ed0 0x7f9628148db0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f962400e790 tx=0x7f962400ec50 comp rx=0 tx=0).stop 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 shutdown_connections 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f960c0763f0 0x7f960c078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f962815efe0 0x7f96281e9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96280d23c0 0x7f962815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 --2- 172.21.15.59:0/1316475013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96280b3ed0 0x7f9628148db0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 >> 172.21.15.59:0/1316475013 conn(0x7f96280d3ee0 msgr2=0x7f962806cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 shutdown_connections 2024-06-29T15:55:38.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.554+0000 7f963063f640 1 -- 172.21.15.59:0/1316475013 wait complete. 2024-06-29T15:55:38.565 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:38.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.683+0000 7fcc43103640 1 -- 172.21.15.59:0/1958461492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc24004800 msgr2=0x7fcc24004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.683+0000 7fcc43103640 1 --2- 172.21.15.59:0/1958461492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc24004800 0x7fcc24004c10 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009d20 tx=0x7fcc24005830 comp rx=0 tx=0).stop 2024-06-29T15:55:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.683+0000 7fcc43103640 1 -- 172.21.15.59:0/1958461492 shutdown_connections 2024-06-29T15:55:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.683+0000 7fcc43103640 1 --2- 172.21.15.59:0/1958461492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc24004800 0x7fcc24004c10 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.683+0000 7fcc43103640 1 -- 172.21.15.59:0/1958461492 >> 172.21.15.59:0/1958461492 conn(0x7fcc3c0b24c0 msgr2=0x7fcc3c0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.683+0000 7fcc43103640 1 -- 172.21.15.59:0/1958461492 shutdown_connections 2024-06-29T15:55:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.683+0000 7fcc43103640 1 -- 172.21.15.59:0/1958461492 wait complete. 2024-06-29T15:55:38.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.685+0000 7fcc43103640 1 Processor -- start 2024-06-29T15:55:38.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.685+0000 7fcc43103640 1 -- start start 2024-06-29T15:55:38.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc43103640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24004800 0x7fcc3c15a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc43103640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3c137310 0x7fcc3c137720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc43103640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142c20 0x7fcc3c143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc43103640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c15f5d0 con 0x7fcc3c142c20 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc43103640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c15f450 con 0x7fcc3c137310 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc43103640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcc3c15f750 con 0x7fcc24004800 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142c20 0x7fcc3c143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142c20 0x7fcc3c143030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58644/0 (socket says 172.21.15.59:58644) 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 -- 172.21.15.59:0/2342369896 learned_addr learned my addr 172.21.15.59:0/2342369896 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 -- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24004800 msgr2=0x7fcc3c15a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24004800 0x7fcc3c15a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc33fff640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3c137310 0x7fcc3c137720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 -- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3c137310 msgr2=0x7fcc3c137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3c137310 0x7fcc3c137720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 -- 172.21.15.59:0/2342369896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3c15ab30 con 0x7fcc3c142c20 2024-06-29T15:55:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc416a0640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142c20 0x7fcc3c143030 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fcc34000f80 tx=0x7fcc3400eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc317fa640 1 -- 172.21.15.59:0/2342369896 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3400c9d0 con 0x7fcc3c142c20 2024-06-29T15:55:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc317fa640 1 -- 172.21.15.59:0/2342369896 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc34017070 con 0x7fcc3c142c20 2024-06-29T15:55:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3c143570 con 0x7fcc3c142c20 2024-06-29T15:55:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc317fa640 1 -- 172.21.15.59:0/2342369896 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34011460 con 0x7fcc3c142c20 2024-06-29T15:55:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.687+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc3c138320 con 0x7fcc3c142c20 2024-06-29T15:55:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.689+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc3c06d670 con 0x7fcc3c142c20 2024-06-29T15:55:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.689+0000 7fcc317fa640 1 -- 172.21.15.59:0/2342369896 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fcc34011600 con 0x7fcc3c142c20 2024-06-29T15:55:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.690+0000 7fcc317fa640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fcc28076420 0x7fcc28078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.690+0000 7fcc317fa640 1 -- 172.21.15.59:0/2342369896 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 6157+0+0 (secure 0 0 0) 0x7fcc34097eb0 con 0x7fcc3c142c20 2024-06-29T15:55:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.690+0000 7fcc40e9f640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fcc28076420 0x7fcc28078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.691+0000 7fcc40e9f640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fcc28076420 0x7fcc28078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc380063f0 tx=0x7fcc38006a40 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:38.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.694+0000 7fcc317fa640 1 -- 172.21.15.59:0/2342369896 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc3c143570 con 0x7fcc3c142c20 2024-06-29T15:55:38.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.885+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc3c0b3100 con 0x7fcc3c142c20 2024-06-29T15:55:38.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.887+0000 7fcc317fa640 1 -- 172.21.15.59:0/2342369896 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) ==== 74+0+17003 (secure 0 0 0) 0x7fcc3c15ab30 con 0x7fcc3c142c20 2024-06-29T15:55:38.887 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:38.887 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":203,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:55:37.567403+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.935517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.518490+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:38.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fcc28076420 msgr2=0x7fcc28078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fcc28076420 0x7fcc28078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc380063f0 tx=0x7fcc38006a40 comp rx=0 tx=0).stop 2024-06-29T15:55:38.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142c20 msgr2=0x7fcc3c143030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142c20 0x7fcc3c143030 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fcc34000f80 tx=0x7fcc3400eae0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 shutdown_connections 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fcc28076420 0x7fcc28078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142c20 0x7fcc3c143030 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc3c137310 0x7fcc3c137720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 --2- 172.21.15.59:0/2342369896 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc24004800 0x7fcc3c15a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 >> 172.21.15.59:0/2342369896 conn(0x7fcc3c0b24c0 msgr2=0x7fcc3c135ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 shutdown_connections 2024-06-29T15:55:38.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:38.889+0000 7fcc43103640 1 -- 172.21.15.59:0/2342369896 wait complete. 2024-06-29T15:55:38.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:55:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.020+0000 7fe28a63d640 1 -- 172.21.15.59:0/2371594552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28415e0a0 msgr2=0x7fe284161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.020+0000 7fe28a63d640 1 --2- 172.21.15.59:0/2371594552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28415e0a0 0x7fe284161a80 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fe2840b29a0 tx=0x7fe26c02a4b0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.020+0000 7fe28a63d640 1 -- 172.21.15.59:0/2371594552 shutdown_connections 2024-06-29T15:55:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.020+0000 7fe28a63d640 1 --2- 172.21.15.59:0/2371594552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe28415e0a0 0x7fe284161a80 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.020+0000 7fe28a63d640 1 --2- 172.21.15.59:0/2371594552 >> v2:172.21.15.153:3300/0 conn(0x7fe284162770 0x7fe2840ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.020+0000 7fe28a63d640 1 -- 172.21.15.59:0/2371594552 >> 172.21.15.59:0/2371594552 conn(0x7fe2840d45b0 msgr2=0x7fe2840d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.020+0000 7fe28a63d640 1 -- 172.21.15.59:0/2371594552 shutdown_connections 2024-06-29T15:55:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.020+0000 7fe28a63d640 1 -- 172.21.15.59:0/2371594552 wait complete. 2024-06-29T15:55:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.022+0000 7fe28a63d640 1 Processor -- start 2024-06-29T15:55:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.022+0000 7fe28a63d640 1 -- start start 2024-06-29T15:55:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.023+0000 7fe28a63d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28415e0a0 0x7fe2840c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.023+0000 7fe28a63d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284161fc0 0x7fe2840c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.023+0000 7fe28a63d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe284162770 0x7fe2840c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.023+0000 7fe28a63d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe2840d2d80 con 0x7fe284162770 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.023+0000 7fe28a63d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe2840b4b00 con 0x7fe284161fc0 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.023+0000 7fe28a63d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe2840d2f00 con 0x7fe28415e0a0 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe284162770 0x7fe2840c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe284162770 0x7fe2840c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58654/0 (socket says 172.21.15.59:58654) 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 -- 172.21.15.59:0/1149124741 learned_addr learned my addr 172.21.15.59:0/1149124741 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe27bfff640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284161fc0 0x7fe2840c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe283fff640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28415e0a0 0x7fe2840c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 -- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28415e0a0 msgr2=0x7fe2840c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28415e0a0 0x7fe2840c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 -- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284161fc0 msgr2=0x7fe2840c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284161fc0 0x7fe2840c9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 -- 172.21.15.59:0/1149124741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2840c7e30 con 0x7fe284162770 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe288bda640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe284162770 0x7fe2840c9ad0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00ba10 tx=0x7fe27c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe2817fa640 1 -- 172.21.15.59:0/1149124741 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c019070 con 0x7fe284162770 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2840c8110 con 0x7fe284162770 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe2817fa640 1 -- 172.21.15.59:0/1149124741 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe27c00cc30 con 0x7fe284162770 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe2817fa640 1 -- 172.21.15.59:0/1149124741 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c00ce60 con 0x7fe284162770 2024-06-29T15:55:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.024+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2840c86d0 con 0x7fe284162770 2024-06-29T15:55:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.025+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2841580a0 con 0x7fe284162770 2024-06-29T15:55:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.026+0000 7fe2817fa640 1 -- 172.21.15.59:0/1149124741 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fe27c01c070 con 0x7fe284162770 2024-06-29T15:55:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.027+0000 7fe2817fa640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fe264076420 0x7fe264078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.027+0000 7fe2817fa640 1 -- 172.21.15.59:0/1149124741 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 6157+0+0 (secure 0 0 0) 0x7fe27c097790 con 0x7fe284162770 2024-06-29T15:55:39.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.027+0000 7fe283fff640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fe264076420 0x7fe264078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:39.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.027+0000 7fe283fff640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fe264076420 0x7fe264078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe274004630 tx=0x7fe27400b2e0 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.031+0000 7fe2817fa640 1 -- 172.21.15.59:0/1149124741 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe27c0606b0 con 0x7fe284162770 2024-06-29T15:55:39.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:39.263+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe2840d3440 con 0x7fe284162770 2024-06-29T15:55:40.160 INFO:tasks.ceph.mds.d.smithi153.stderr:2024-06-29T15:55:40.159+0000 7ff23dfcf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:55:40.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.196+0000 7fe2817fa640 1 -- 172.21.15.59:0/1149124741 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v105) ==== 118+0+0 (secure 0 0 0) 0x7fe27c0649b0 con 0x7fe284162770 2024-06-29T15:55:40.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fe264076420 msgr2=0x7fe264078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fe264076420 0x7fe264078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe274004630 tx=0x7fe27400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe284162770 msgr2=0x7fe2840c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe284162770 0x7fe2840c9ad0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00ba10 tx=0x7fe27c00bed0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 shutdown_connections 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fe264076420 0x7fe264078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe284162770 0x7fe2840c9ad0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284161fc0 0x7fe2840c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 --2- 172.21.15.59:0/1149124741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe28415e0a0 0x7fe2840c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.199+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 >> 172.21.15.59:0/1149124741 conn(0x7fe2840d45b0 msgr2=0x7fe28415d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.200+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 shutdown_connections 2024-06-29T15:55:40.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.200+0000 7fe28a63d640 1 -- 172.21.15.59:0/1149124741 wait complete. 2024-06-29T15:55:40.216 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:55:40.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:55:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.346+0000 7f42d1dee640 1 -- 172.21.15.59:0/1674394118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc06ea10 msgr2=0x7f42cc1658d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.346+0000 7f42d1dee640 1 --2- 172.21.15.59:0/1674394118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc06ea10 0x7f42cc1658d0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005fd0 tx=0x7f42c402a940 comp rx=0 tx=0).stop 2024-06-29T15:55:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.346+0000 7f42d1dee640 1 -- 172.21.15.59:0/1674394118 shutdown_connections 2024-06-29T15:55:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.346+0000 7f42d1dee640 1 --2- 172.21.15.59:0/1674394118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc06ea10 0x7f42cc1658d0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.346+0000 7f42d1dee640 1 -- 172.21.15.59:0/1674394118 >> 172.21.15.59:0/1674394118 conn(0x7f42cc0b06e0 msgr2=0x7f42cc0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.346+0000 7f42d1dee640 1 -- 172.21.15.59:0/1674394118 shutdown_connections 2024-06-29T15:55:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.346+0000 7f42d1dee640 1 -- 172.21.15.59:0/1674394118 wait complete. 2024-06-29T15:55:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.348+0000 7f42d1dee640 1 Processor -- start 2024-06-29T15:55:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.348+0000 7f42d1dee640 1 -- start start 2024-06-29T15:55:40.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42d1dee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42cc06ea10 0x7f42cc142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:40.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42d1dee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42cc144470 0x7f42cc13c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:40.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42d1dee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc140720 0x7f42cc140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42d1dee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f42cc165f90 con 0x7f42cc140720 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42d1dee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42cc165e10 con 0x7f42cc06ea10 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42d1dee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f42cc166110 con 0x7f42cc144470 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc140720 0x7f42cc140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc140720 0x7f42cc140b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58678/0 (socket says 172.21.15.59:58678) 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 -- 172.21.15.59:0/3080316141 learned_addr learned my addr 172.21.15.59:0/3080316141 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cb7fe640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42cc06ea10 0x7f42cc142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42caffd640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42cc144470 0x7f42cc13c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 -- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42cc144470 msgr2=0x7f42cc13c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42cc144470 0x7f42cc13c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 -- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42cc06ea10 msgr2=0x7f42cc142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42cc06ea10 0x7f42cc142d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 -- 172.21.15.59:0/3080316141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42cc141070 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42caffd640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42cc144470 0x7f42cc13c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42cbfff640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc140720 0x7f42cc140b30 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f42c402af30 tx=0x7f42c4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.350+0000 7f42b3fff640 1 -- 172.21.15.59:0/3080316141 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4004550 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.351+0000 7f42b3fff640 1 -- 172.21.15.59:0/3080316141 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42c403f070 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.351+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42cc156830 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.351+0000 7f42b3fff640 1 -- 172.21.15.59:0/3080316141 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c40446d0 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.351+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42cc156df0 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.352+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42cc06e300 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.352+0000 7f42b3fff640 1 -- 172.21.15.59:0/3080316141 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f42c403b450 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.353+0000 7f42b3fff640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b8076420 0x7f42b8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.353+0000 7f42b3fff640 1 -- 172.21.15.59:0/3080316141 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f42c40bbc40 con 0x7f42cc140720 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.353+0000 7f42cb7fe640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b8076420 0x7f42b8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.354+0000 7f42cb7fe640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b8076420 0x7f42b8078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0046b0 tx=0x7f42bc007e90 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:40.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.358+0000 7f42b3fff640 1 -- 172.21.15.59:0/3080316141 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f42cc141070 con 0x7f42cc140720 2024-06-29T15:55:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:40.590+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f42cc156550 con 0x7f42cc140720 2024-06-29T15:55:41.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.198+0000 7f42b3fff640 1 -- 172.21.15.59:0/3080316141 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v106) ==== 122+0+0 (secure 0 0 0) 0x7f42c4085240 con 0x7f42cc140720 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.200+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b8076420 msgr2=0x7f42b8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.200+0000 7f42d1dee640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b8076420 0x7f42b8078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0046b0 tx=0x7f42bc007e90 comp rx=0 tx=0).stop 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.200+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc140720 msgr2=0x7f42cc140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.200+0000 7f42d1dee640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc140720 0x7f42cc140b30 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f42c402af30 tx=0x7f42c4003f90 comp rx=0 tx=0).stop 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.200+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 shutdown_connections 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.200+0000 7f42d1dee640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b8076420 0x7f42b8078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.200+0000 7f42d1dee640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42cc140720 0x7f42cc140b30 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.201+0000 7f42d1dee640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42cc144470 0x7f42cc13c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.201+0000 7f42d1dee640 1 --2- 172.21.15.59:0/3080316141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42cc06ea10 0x7f42cc142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.201+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 >> 172.21.15.59:0/3080316141 conn(0x7f42cc0b06e0 msgr2=0x7f42cc0cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.201+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 shutdown_connections 2024-06-29T15:55:41.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.201+0000 7f42d1dee640 1 -- 172.21.15.59:0/3080316141 wait complete. 2024-06-29T15:55:41.216 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:55:41.217 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:41.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.341+0000 7f8b904d2640 1 -- 172.21.15.59:0/2618132619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b8815ceb0 msgr2=0x7f8b880b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.341+0000 7f8b904d2640 1 --2- 172.21.15.59:0/2618132619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b8815ceb0 0x7f8b880b56d0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009830 tx=0x7f8b8002a480 comp rx=0 tx=0).stop 2024-06-29T15:55:41.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.341+0000 7f8b904d2640 1 -- 172.21.15.59:0/2618132619 shutdown_connections 2024-06-29T15:55:41.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.341+0000 7f8b904d2640 1 --2- 172.21.15.59:0/2618132619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b88141bd0 0x7f8b8813edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.341+0000 7f8b904d2640 1 --2- 172.21.15.59:0/2618132619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b8815ceb0 0x7f8b880b56d0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.341+0000 7f8b904d2640 1 -- 172.21.15.59:0/2618132619 >> 172.21.15.59:0/2618132619 conn(0x7f8b880b6fb0 msgr2=0x7f8b880b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.341+0000 7f8b904d2640 1 -- 172.21.15.59:0/2618132619 shutdown_connections 2024-06-29T15:55:41.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.341+0000 7f8b904d2640 1 -- 172.21.15.59:0/2618132619 wait complete. 2024-06-29T15:55:41.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.343+0000 7f8b904d2640 1 Processor -- start 2024-06-29T15:55:41.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.343+0000 7f8b904d2640 1 -- start start 2024-06-29T15:55:41.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b904d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8813e9b0 0x7f8b880cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b904d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b8815ceb0 0x7f8b88137700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b904d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b88154fd0 0x7f8b881553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b904d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880b2a00 con 0x7f8b88154fd0 2024-06-29T15:55:41.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b904d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880b2880 con 0x7f8b8813e9b0 2024-06-29T15:55:41.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b904d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8b880b2b80 con 0x7f8b8815ceb0 2024-06-29T15:55:41.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b8ea6f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b88154fd0 0x7f8b881553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b8ea6f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b88154fd0 0x7f8b881553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58696/0 (socket says 172.21.15.59:58696) 2024-06-29T15:55:41.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b8ea6f640 1 -- 172.21.15.59:0/1154880009 learned_addr learned my addr 172.21.15.59:0/1154880009 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:41.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b8e26e640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8813e9b0 0x7f8b880cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.344+0000 7f8b8da6d640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b8815ceb0 0x7f8b88137700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b8ea6f640 1 -- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b8815ceb0 msgr2=0x7f8b88137700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b8ea6f640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b8815ceb0 0x7f8b88137700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b8ea6f640 1 -- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8813e9b0 msgr2=0x7f8b880cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b8ea6f640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8813e9b0 0x7f8b880cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b8ea6f640 1 -- 172.21.15.59:0/1154880009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b88155920 con 0x7f8b88154fd0 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b8ea6f640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b88154fd0 0x7f8b881553e0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800e990 tx=0x7f8b7800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b86ffd640 1 -- 172.21.15.59:0/1154880009 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78004d20 con 0x7f8b88154fd0 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b86ffd640 1 -- 172.21.15.59:0/1154880009 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b7800caa0 con 0x7f8b88154fd0 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b881381c0 con 0x7f8b88154fd0 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b86ffd640 1 -- 172.21.15.59:0/1154880009 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78005020 con 0x7f8b88154fd0 2024-06-29T15:55:41.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.345+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b881f8330 con 0x7f8b88154fd0 2024-06-29T15:55:41.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.346+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b88137f10 con 0x7f8b88154fd0 2024-06-29T15:55:41.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.347+0000 7f8b86ffd640 1 -- 172.21.15.59:0/1154880009 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f8b780051c0 con 0x7f8b88154fd0 2024-06-29T15:55:41.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.348+0000 7f8b86ffd640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f8b740763f0 0x7f8b74078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.348+0000 7f8b86ffd640 1 -- 172.21.15.59:0/1154880009 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f8b78098470 con 0x7f8b88154fd0 2024-06-29T15:55:41.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.348+0000 7f8b8e26e640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f8b740763f0 0x7f8b74078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.348+0000 7f8b8e26e640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f8b740763f0 0x7f8b74078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8b80034000 tx=0x7f8b80004730 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:41.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.352+0000 7f8b86ffd640 1 -- 172.21.15.59:0/1154880009 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8b88155920 con 0x7f8b88154fd0 2024-06-29T15:55:41.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.582+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b8813edc0 con 0x7f8b88154fd0 2024-06-29T15:55:41.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.583+0000 7f8b86ffd640 1 -- 172.21.15.59:0/1154880009 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5246 (secure 0 0 0) 0x7f8b78016030 con 0x7f8b88154fd0 2024-06-29T15:55:41.584 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:41.584 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"btime":"2024-06-29T15:55:41:198021+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3297476433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3297476433},{"type":"v1","addr":"172.21.15.153:6837","nonce":3297476433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":10974,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3987613682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3987613682},{"type":"v1","addr":"172.21.15.59:6834","nonce":3987613682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":10977,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2501188073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2501188073},{"type":"v1","addr":"172.21.15.59:6835","nonce":2501188073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.567350+0000","modified":"2024-06-29T15:55:40.592165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10983},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10983":{"gid":10983,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2163190416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2163190416},{"type":"v1","addr":"172.21.15.153:6836","nonce":2163190416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10983,"qdb_cluster":[10983]},"id":12}]} 2024-06-29T15:55:41.584 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-06-29T15:55:41.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f8b740763f0 msgr2=0x7f8b74078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f8b740763f0 0x7f8b74078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8b80034000 tx=0x7f8b80004730 comp rx=0 tx=0).stop 2024-06-29T15:55:41.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b88154fd0 msgr2=0x7f8b881553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b88154fd0 0x7f8b881553e0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800e990 tx=0x7f8b7800ee50 comp rx=0 tx=0).stop 2024-06-29T15:55:41.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 shutdown_connections 2024-06-29T15:55:41.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f8b740763f0 0x7f8b74078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b88154fd0 0x7f8b881553e0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8b8815ceb0 0x7f8b88137700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 --2- 172.21.15.59:0/1154880009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8813e9b0 0x7f8b880cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 >> 172.21.15.59:0/1154880009 conn(0x7f8b880b6fb0 msgr2=0x7f8b880b3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 shutdown_connections 2024-06-29T15:55:41.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.585+0000 7f8b904d2640 1 -- 172.21.15.59:0/1154880009 wait complete. 2024-06-29T15:55:41.597 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.715+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/2109978060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 msgr2=0x7f9e54005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.715+0000 7f9e6d1b9640 1 --2- 172.21.15.59:0/2109978060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 0x7f9e54005150 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009ed0 tx=0x7f9e54005830 comp rx=0 tx=0).stop 2024-06-29T15:55:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.715+0000 7f9e6759e640 1 -- 172.21.15.59:0/2109978060 reap_dead start 2024-06-29T15:55:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.715+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/2109978060 shutdown_connections 2024-06-29T15:55:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.715+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/2109978060 >> 172.21.15.59:0/2109978060 conn(0x7f9e680b5610 msgr2=0x7f9e680b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.715+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/2109978060 shutdown_connections 2024-06-29T15:55:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.715+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/2109978060 wait complete. 2024-06-29T15:55:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.717+0000 7f9e6d1b9640 1 Processor -- start 2024-06-29T15:55:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.717+0000 7f9e6d1b9640 1 -- start start 2024-06-29T15:55:41.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.718+0000 7f9e6d1b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 0x7f9e68152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.718+0000 7f9e6d1b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e6815adf0 0x7f9e681506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.718+0000 7f9e6d1b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e68150d10 0x7f9e680cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.718+0000 7f9e6d1b9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9e680cf410 con 0x7f9e54004d40 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.718+0000 7f9e6d1b9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e680cf290 con 0x7f9e6815adf0 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.718+0000 7f9e6d1b9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9e680d2880 con 0x7f9e68150d10 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 0x7f9e68152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 0x7f9e68152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58720/0 (socket says 172.21.15.59:58720) 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 -- 172.21.15.59:0/632858995 learned_addr learned my addr 172.21.15.59:0/632858995 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:41.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e6659c640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e6815adf0 0x7f9e681506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e6759e640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e68150d10 0x7f9e680cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 -- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e68150d10 msgr2=0x7f9e680cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e68150d10 0x7f9e680cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 -- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e6815adf0 msgr2=0x7f9e681506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e6815adf0 0x7f9e681506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 -- 172.21.15.59:0/632858995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6815b5d0 con 0x7f9e54004d40 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e66d9d640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 0x7f9e68152220 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f9e58011a50 tx=0x7f9e58011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e5f7fe640 1 -- 172.21.15.59:0/632858995 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5801f070 con 0x7f9e54004d40 2024-06-29T15:55:41.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e5f7fe640 1 -- 172.21.15.59:0/632858995 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e5800fe80 con 0x7f9e54004d40 2024-06-29T15:55:41.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e6815b8b0 con 0x7f9e54004d40 2024-06-29T15:55:41.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e5f7fe640 1 -- 172.21.15.59:0/632858995 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58012690 con 0x7f9e54004d40 2024-06-29T15:55:41.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.719+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e68151a20 con 0x7f9e54004d40 2024-06-29T15:55:41.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.720+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e680d1a20 con 0x7f9e54004d40 2024-06-29T15:55:41.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.721+0000 7f9e5f7fe640 1 -- 172.21.15.59:0/632858995 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f9e58022070 con 0x7f9e54004d40 2024-06-29T15:55:41.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.722+0000 7f9e5f7fe640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9e480763f0 0x7f9e48078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.722+0000 7f9e5f7fe640 1 -- 172.21.15.59:0/632858995 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f9e58098e00 con 0x7f9e54004d40 2024-06-29T15:55:41.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.722+0000 7f9e6659c640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9e480763f0 0x7f9e48078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.723+0000 7f9e6659c640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9e480763f0 0x7f9e48078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e68154bc0 tx=0x7f9e50006040 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:41.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.725+0000 7f9e5f7fe640 1 -- 172.21.15.59:0/632858995 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9e58061ed0 con 0x7f9e54004d40 2024-06-29T15:55:41.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.915+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e680d23a0 con 0x7f9e54004d40 2024-06-29T15:55:41.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.917+0000 7f9e5f7fe640 1 -- 172.21.15.59:0/632858995 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) ==== 74+0+16990 (secure 0 0 0) 0x7f9e580661d0 con 0x7f9e54004d40 2024-06-29T15:55:41.918 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:41.918 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":204,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:55:39.186933+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.935517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.518490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:41.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9e480763f0 msgr2=0x7f9e48078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9e480763f0 0x7f9e48078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e68154bc0 tx=0x7f9e50006040 comp rx=0 tx=0).stop 2024-06-29T15:55:41.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 msgr2=0x7f9e68152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 0x7f9e68152220 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f9e58011a50 tx=0x7f9e58011f10 comp rx=0 tx=0).stop 2024-06-29T15:55:41.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 shutdown_connections 2024-06-29T15:55:41.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9e480763f0 0x7f9e48078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e68150d10 0x7f9e680cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e6815adf0 0x7f9e681506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 --2- 172.21.15.59:0/632858995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004d40 0x7f9e68152220 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 >> 172.21.15.59:0/632858995 conn(0x7f9e680b5610 msgr2=0x7f9e680c87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 shutdown_connections 2024-06-29T15:55:41.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:41.919+0000 7f9e6d1b9640 1 -- 172.21.15.59:0/632858995 wait complete. 2024-06-29T15:55:41.931 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:55:41.931 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:55:42.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.048+0000 7f52aade0640 1 -- 172.21.15.59:0/1688104688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a41458a0 msgr2=0x7f52a40cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.048+0000 7f52aade0640 1 --2- 172.21.15.59:0/1688104688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a41458a0 0x7f52a40cf870 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5294005fd0 tx=0x7f529402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.048+0000 7f52aade0640 1 -- 172.21.15.59:0/1688104688 shutdown_connections 2024-06-29T15:55:42.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.048+0000 7f52aade0640 1 --2- 172.21.15.59:0/1688104688 >> v2:172.21.15.153:3300/0 conn(0x7f52a40b3740 0x7f52a40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.048+0000 7f52aade0640 1 --2- 172.21.15.59:0/1688104688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a41458a0 0x7f52a40cf870 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.048+0000 7f52aade0640 1 -- 172.21.15.59:0/1688104688 >> 172.21.15.59:0/1688104688 conn(0x7f52a406c950 msgr2=0x7f52a4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.048+0000 7f52aade0640 1 -- 172.21.15.59:0/1688104688 shutdown_connections 2024-06-29T15:55:42.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.048+0000 7f52aade0640 1 -- 172.21.15.59:0/1688104688 wait complete. 2024-06-29T15:55:42.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.050+0000 7f52aade0640 1 Processor -- start 2024-06-29T15:55:42.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.050+0000 7f52aade0640 1 -- start start 2024-06-29T15:55:42.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52aade0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a40b3740 0x7f52a41f4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52aade0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a415c2f0 0x7f52a406eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52aade0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52a415e270 0x7f52a406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52aade0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f52a41601a0 con 0x7f52a415e270 2024-06-29T15:55:42.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52aade0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52a4160020 con 0x7f52a415c2f0 2024-06-29T15:55:42.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52aade0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f52a4160320 con 0x7f52a40b3740 2024-06-29T15:55:42.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52a415e270 0x7f52a406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52a415e270 0x7f52a406f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58734/0 (socket says 172.21.15.59:58734) 2024-06-29T15:55:42.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 -- 172.21.15.59:0/255835795 learned_addr learned my addr 172.21.15.59:0/255835795 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:42.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a3fff640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a415c2f0 0x7f52a406eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a8b7c640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a40b3740 0x7f52a41f4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 -- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a40b3740 msgr2=0x7f52a41f4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a40b3740 0x7f52a41f4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 -- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a415c2f0 msgr2=0x7f52a406eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a415c2f0 0x7f52a406eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 -- 172.21.15.59:0/255835795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a406dd20 con 0x7f52a415e270 2024-06-29T15:55:42.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a8b7c640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a40b3740 0x7f52a41f4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:42.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a3fff640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a415c2f0 0x7f52a406eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a937d640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52a415e270 0x7f52a406f020 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f529c00c730 tx=0x7f529c00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.052+0000 7f52a17fa640 1 -- 172.21.15.59:0/255835795 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c003b70 con 0x7f52a415e270 2024-06-29T15:55:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.053+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a406e000 con 0x7f52a415e270 2024-06-29T15:55:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.053+0000 7f52a17fa640 1 -- 172.21.15.59:0/255835795 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f529c003d10 con 0x7f52a415e270 2024-06-29T15:55:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.053+0000 7f52a17fa640 1 -- 172.21.15.59:0/255835795 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c005330 con 0x7f52a415e270 2024-06-29T15:55:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.053+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a406e5c0 con 0x7f52a415e270 2024-06-29T15:55:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.054+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a41458a0 con 0x7f52a415e270 2024-06-29T15:55:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.054+0000 7f52a17fa640 1 -- 172.21.15.59:0/255835795 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f529c0054d0 con 0x7f52a415e270 2024-06-29T15:55:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.055+0000 7f52a17fa640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f52900763f0 0x7f5290078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.055+0000 7f52a17fa640 1 -- 172.21.15.59:0/255835795 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f529c0979c0 con 0x7f52a415e270 2024-06-29T15:55:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.055+0000 7f52a8b7c640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f52900763f0 0x7f5290078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.056+0000 7f52a8b7c640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f52900763f0 0x7f5290078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5294033000 tx=0x7f5294004730 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.060+0000 7f52a17fa640 1 -- 172.21.15.59:0/255835795 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f529c060ea0 con 0x7f52a415e270 2024-06-29T15:55:42.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.245+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 --> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f52a40d26f0 con 0x7f52900763f0 2024-06-29T15:55:42.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.250+0000 7f52a17fa640 1 -- 172.21.15.59:0/255835795 <== mgr.10063 v2:172.21.15.153:6832/493926731 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252481 (secure 0 0 0) 0x7f52a40d26f0 con 0x7f52900763f0 2024-06-29T15:55:42.254 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:42.255 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f52900763f0 msgr2=0x7f5290078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f52900763f0 0x7f5290078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5294033000 tx=0x7f5294004730 comp rx=0 tx=0).stop 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52a415e270 msgr2=0x7f52a406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52a415e270 0x7f52a406f020 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f529c00c730 tx=0x7f529c00cb30 comp rx=0 tx=0).stop 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 shutdown_connections 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f52900763f0 0x7f5290078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52a415e270 0x7f52a406f020 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52a415c2f0 0x7f52a406eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 --2- 172.21.15.59:0/255835795 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52a40b3740 0x7f52a41f4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 >> 172.21.15.59:0/255835795 conn(0x7f52a406c950 msgr2=0x7f52a40d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 shutdown_connections 2024-06-29T15:55:42.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.253+0000 7f52aade0640 1 -- 172.21.15.59:0/255835795 wait complete. 2024-06-29T15:55:42.265 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:55:41.672243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":398,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":258,"ondisk_log_size":258,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":374704,"kb_used_data":21100,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754600016,"statfs":{"total":773094113280,"available":772710416384,"internally_reserved":0,"allocated":21606400,"data_stored":11610816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":361942578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.504202"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199831+0000","last_change":"2024-06-29T15:55:38.210448+0000","last_active":"2024-06-29T15:55:39.199831+0000","last_peered":"2024-06-29T15:55:39.199831+0000","last_clean":"2024-06-29T15:55:39.199831+0000","last_became_active":"2024-06-29T15:55:38.209396+0000","last_became_peered":"2024-06-29T15:55:38.209396+0000","last_unstale":"2024-06-29T15:55:39.199831+0000","last_undegraded":"2024-06-29T15:55:39.199831+0000","last_fullsized":"2024-06-29T15:55:39.199831+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:52.213826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199692+0000","last_change":"2024-06-29T15:55:38.189981+0000","last_active":"2024-06-29T15:55:39.199692+0000","last_peered":"2024-06-29T15:55:39.199692+0000","last_clean":"2024-06-29T15:55:39.199692+0000","last_became_active":"2024-06-29T15:55:38.189652+0000","last_became_peered":"2024-06-29T15:55:38.189652+0000","last_unstale":"2024-06-29T15:55:39.199692+0000","last_undegraded":"2024-06-29T15:55:39.199692+0000","last_fullsized":"2024-06-29T15:55:39.199692+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:36:47.130148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195206+0000","last_change":"2024-06-29T15:55:38.188572+0000","last_active":"2024-06-29T15:55:39.195206+0000","last_peered":"2024-06-29T15:55:39.195206+0000","last_clean":"2024-06-29T15:55:39.195206+0000","last_became_active":"2024-06-29T15:55:38.188217+0000","last_became_peered":"2024-06-29T15:55:38.188217+0000","last_unstale":"2024-06-29T15:55:39.195206+0000","last_undegraded":"2024-06-29T15:55:39.195206+0000","last_fullsized":"2024-06-29T15:55:39.195206+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:13.262550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637929+0000","last_change":"2024-06-29T15:55:38.211912+0000","last_active":"2024-06-29T15:55:39.637929+0000","last_peered":"2024-06-29T15:55:39.637929+0000","last_clean":"2024-06-29T15:55:39.637929+0000","last_became_active":"2024-06-29T15:55:38.211599+0000","last_became_peered":"2024-06-29T15:55:38.211599+0000","last_unstale":"2024-06-29T15:55:39.637929+0000","last_undegraded":"2024-06-29T15:55:39.637929+0000","last_fullsized":"2024-06-29T15:55:39.637929+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:50.684269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194970+0000","last_change":"2024-06-29T15:55:38.187621+0000","last_active":"2024-06-29T15:55:39.194970+0000","last_peered":"2024-06-29T15:55:39.194970+0000","last_clean":"2024-06-29T15:55:39.194970+0000","last_became_active":"2024-06-29T15:55:38.187292+0000","last_became_peered":"2024-06-29T15:55:38.187292+0000","last_unstale":"2024-06-29T15:55:39.194970+0000","last_undegraded":"2024-06-29T15:55:39.194970+0000","last_fullsized":"2024-06-29T15:55:39.194970+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:40.238763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194771+0000","last_change":"2024-06-29T15:55:38.197810+0000","last_active":"2024-06-29T15:55:39.194771+0000","last_peered":"2024-06-29T15:55:39.194771+0000","last_clean":"2024-06-29T15:55:39.194771+0000","last_became_active":"2024-06-29T15:55:38.197486+0000","last_became_peered":"2024-06-29T15:55:38.197486+0000","last_unstale":"2024-06-29T15:55:39.194771+0000","last_undegraded":"2024-06-29T15:55:39.194771+0000","last_fullsized":"2024-06-29T15:55:39.194771+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:53.933287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197434+0000","last_change":"2024-06-29T15:55:38.203101+0000","last_active":"2024-06-29T15:55:39.197434+0000","last_peered":"2024-06-29T15:55:39.197434+0000","last_clean":"2024-06-29T15:55:39.197434+0000","last_became_active":"2024-06-29T15:55:38.202763+0000","last_became_peered":"2024-06-29T15:55:38.202763+0000","last_unstale":"2024-06-29T15:55:39.197434+0000","last_undegraded":"2024-06-29T15:55:39.197434+0000","last_fullsized":"2024-06-29T15:55:39.197434+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:21.116209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.206380+0000","last_change":"2024-06-29T15:55:38.203503+0000","last_active":"2024-06-29T15:55:38.206380+0000","last_peered":"2024-06-29T15:55:38.206380+0000","last_clean":"2024-06-29T15:55:38.206380+0000","last_became_active":"2024-06-29T15:55:38.203195+0000","last_became_peered":"2024-06-29T15:55:38.203195+0000","last_unstale":"2024-06-29T15:55:38.206380+0000","last_undegraded":"2024-06-29T15:55:38.206380+0000","last_fullsized":"2024-06-29T15:55:38.206380+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:31:25.943016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199990+0000","last_change":"2024-06-29T15:55:38.204545+0000","last_active":"2024-06-29T15:55:39.199990+0000","last_peered":"2024-06-29T15:55:39.199990+0000","last_clean":"2024-06-29T15:55:39.199990+0000","last_became_active":"2024-06-29T15:55:38.204205+0000","last_became_peered":"2024-06-29T15:55:38.204205+0000","last_unstale":"2024-06-29T15:55:39.199990+0000","last_undegraded":"2024-06-29T15:55:39.199990+0000","last_fullsized":"2024-06-29T15:55:39.199990+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:14.722044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195263+0000","last_change":"2024-06-29T15:55:38.204281+0000","last_active":"2024-06-29T15:55:39.195263+0000","last_peered":"2024-06-29T15:55:39.195263+0000","last_clean":"2024-06-29T15:55:39.195263+0000","last_became_active":"2024-06-29T15:55:38.203924+0000","last_became_peered":"2024-06-29T15:55:38.203924+0000","last_unstale":"2024-06-29T15:55:39.195263+0000","last_undegraded":"2024-06-29T15:55:39.195263+0000","last_fullsized":"2024-06-29T15:55:39.195263+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:31.481915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.207616+0000","last_change":"2024-06-29T15:55:38.201374+0000","last_active":"2024-06-29T15:55:38.207616+0000","last_peered":"2024-06-29T15:55:38.207616+0000","last_clean":"2024-06-29T15:55:38.207616+0000","last_became_active":"2024-06-29T15:55:38.200988+0000","last_became_peered":"2024-06-29T15:55:38.200988+0000","last_unstale":"2024-06-29T15:55:38.207616+0000","last_undegraded":"2024-06-29T15:55:38.207616+0000","last_fullsized":"2024-06-29T15:55:38.207616+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:30:18.805904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638002+0000","last_change":"2024-06-29T15:55:38.204934+0000","last_active":"2024-06-29T15:55:39.638002+0000","last_peered":"2024-06-29T15:55:39.638002+0000","last_clean":"2024-06-29T15:55:39.638002+0000","last_became_active":"2024-06-29T15:55:38.204547+0000","last_became_peered":"2024-06-29T15:55:38.204547+0000","last_unstale":"2024-06-29T15:55:39.638002+0000","last_undegraded":"2024-06-29T15:55:39.638002+0000","last_fullsized":"2024-06-29T15:55:39.638002+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:49.473559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195647+0000","last_change":"2024-06-29T15:55:38.209039+0000","last_active":"2024-06-29T15:55:39.195647+0000","last_peered":"2024-06-29T15:55:39.195647+0000","last_clean":"2024-06-29T15:55:39.195647+0000","last_became_active":"2024-06-29T15:55:38.206849+0000","last_became_peered":"2024-06-29T15:55:38.206849+0000","last_unstale":"2024-06-29T15:55:39.195647+0000","last_undegraded":"2024-06-29T15:55:39.195647+0000","last_fullsized":"2024-06-29T15:55:39.195647+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:31.485808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197705+0000","last_change":"2024-06-29T15:55:38.207992+0000","last_active":"2024-06-29T15:55:39.197705+0000","last_peered":"2024-06-29T15:55:39.197705+0000","last_clean":"2024-06-29T15:55:39.197705+0000","last_became_active":"2024-06-29T15:55:38.207669+0000","last_became_peered":"2024-06-29T15:55:38.207669+0000","last_unstale":"2024-06-29T15:55:39.197705+0000","last_undegraded":"2024-06-29T15:55:39.197705+0000","last_fullsized":"2024-06-29T15:55:39.197705+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:38.397012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199532+0000","last_change":"2024-06-29T15:55:38.209811+0000","last_active":"2024-06-29T15:55:39.199532+0000","last_peered":"2024-06-29T15:55:39.199532+0000","last_clean":"2024-06-29T15:55:39.199532+0000","last_became_active":"2024-06-29T15:55:38.209230+0000","last_became_peered":"2024-06-29T15:55:38.209230+0000","last_unstale":"2024-06-29T15:55:39.199532+0000","last_undegraded":"2024-06-29T15:55:39.199532+0000","last_fullsized":"2024-06-29T15:55:39.199532+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:09.365017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194645+0000","last_change":"2024-06-29T15:55:38.189168+0000","last_active":"2024-06-29T15:55:39.194645+0000","last_peered":"2024-06-29T15:55:39.194645+0000","last_clean":"2024-06-29T15:55:39.194645+0000","last_became_active":"2024-06-29T15:55:38.188870+0000","last_became_peered":"2024-06-29T15:55:38.188870+0000","last_unstale":"2024-06-29T15:55:39.194645+0000","last_undegraded":"2024-06-29T15:55:39.194645+0000","last_fullsized":"2024-06-29T15:55:39.194645+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:31.038820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195191+0000","last_change":"2024-06-29T15:55:38.210931+0000","last_active":"2024-06-29T15:55:39.195191+0000","last_peered":"2024-06-29T15:55:39.195191+0000","last_clean":"2024-06-29T15:55:39.195191+0000","last_became_active":"2024-06-29T15:55:38.210574+0000","last_became_peered":"2024-06-29T15:55:38.210574+0000","last_unstale":"2024-06-29T15:55:39.195191+0000","last_undegraded":"2024-06-29T15:55:39.195191+0000","last_fullsized":"2024-06-29T15:55:39.195191+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:34.617520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195628+0000","last_change":"2024-06-29T15:55:38.203400+0000","last_active":"2024-06-29T15:55:39.195628+0000","last_peered":"2024-06-29T15:55:39.195628+0000","last_clean":"2024-06-29T15:55:39.195628+0000","last_became_active":"2024-06-29T15:55:38.203071+0000","last_became_peered":"2024-06-29T15:55:38.203071+0000","last_unstale":"2024-06-29T15:55:39.195628+0000","last_undegraded":"2024-06-29T15:55:39.195628+0000","last_fullsized":"2024-06-29T15:55:39.195628+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:22.269697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195386+0000","last_change":"2024-06-29T15:55:38.207190+0000","last_active":"2024-06-29T15:55:39.195386+0000","last_peered":"2024-06-29T15:55:39.195386+0000","last_clean":"2024-06-29T15:55:39.195386+0000","last_became_active":"2024-06-29T15:55:38.206571+0000","last_became_peered":"2024-06-29T15:55:38.206571+0000","last_unstale":"2024-06-29T15:55:39.195386+0000","last_undegraded":"2024-06-29T15:55:39.195386+0000","last_fullsized":"2024-06-29T15:55:39.195386+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:15:26.759219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195010+0000","last_change":"2024-06-29T15:55:38.204209+0000","last_active":"2024-06-29T15:55:39.195010+0000","last_peered":"2024-06-29T15:55:39.195010+0000","last_clean":"2024-06-29T15:55:39.195010+0000","last_became_active":"2024-06-29T15:55:38.203795+0000","last_became_peered":"2024-06-29T15:55:38.203795+0000","last_unstale":"2024-06-29T15:55:39.195010+0000","last_undegraded":"2024-06-29T15:55:39.195010+0000","last_fullsized":"2024-06-29T15:55:39.195010+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:21.677784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199773+0000","last_change":"2024-06-29T15:55:38.202664+0000","last_active":"2024-06-29T15:55:39.199773+0000","last_peered":"2024-06-29T15:55:39.199773+0000","last_clean":"2024-06-29T15:55:39.199773+0000","last_became_active":"2024-06-29T15:55:38.202314+0000","last_became_peered":"2024-06-29T15:55:38.202314+0000","last_unstale":"2024-06-29T15:55:39.199773+0000","last_undegraded":"2024-06-29T15:55:39.199773+0000","last_fullsized":"2024-06-29T15:55:39.199773+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:04.950646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199673+0000","last_change":"2024-06-29T15:55:38.200899+0000","last_active":"2024-06-29T15:55:39.199673+0000","last_peered":"2024-06-29T15:55:39.199673+0000","last_clean":"2024-06-29T15:55:39.199673+0000","last_became_active":"2024-06-29T15:55:38.200529+0000","last_became_peered":"2024-06-29T15:55:38.200529+0000","last_unstale":"2024-06-29T15:55:39.199673+0000","last_undegraded":"2024-06-29T15:55:39.199673+0000","last_fullsized":"2024-06-29T15:55:39.199673+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:06:25.178759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.202600+0000","last_change":"2024-06-29T15:55:38.199732+0000","last_active":"2024-06-29T15:55:38.202600+0000","last_peered":"2024-06-29T15:55:38.202600+0000","last_clean":"2024-06-29T15:55:38.202600+0000","last_became_active":"2024-06-29T15:55:38.199403+0000","last_became_peered":"2024-06-29T15:55:38.199403+0000","last_unstale":"2024-06-29T15:55:38.202600+0000","last_undegraded":"2024-06-29T15:55:38.202600+0000","last_fullsized":"2024-06-29T15:55:38.202600+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:58:06.377622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199337+0000","last_change":"2024-06-29T15:55:38.206962+0000","last_active":"2024-06-29T15:55:39.199337+0000","last_peered":"2024-06-29T15:55:39.199337+0000","last_clean":"2024-06-29T15:55:39.199337+0000","last_became_active":"2024-06-29T15:55:38.206623+0000","last_became_peered":"2024-06-29T15:55:38.206623+0000","last_unstale":"2024-06-29T15:55:39.199337+0000","last_undegraded":"2024-06-29T15:55:39.199337+0000","last_fullsized":"2024-06-29T15:55:39.199337+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:22:27.907778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197873+0000","last_change":"2024-06-29T15:55:38.197882+0000","last_active":"2024-06-29T15:55:39.197873+0000","last_peered":"2024-06-29T15:55:39.197873+0000","last_clean":"2024-06-29T15:55:39.197873+0000","last_became_active":"2024-06-29T15:55:38.197483+0000","last_became_peered":"2024-06-29T15:55:38.197483+0000","last_unstale":"2024-06-29T15:55:39.197873+0000","last_undegraded":"2024-06-29T15:55:39.197873+0000","last_fullsized":"2024-06-29T15:55:39.197873+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:17.255639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.190459+0000","last_change":"2024-06-29T15:55:38.188563+0000","last_active":"2024-06-29T15:55:38.190459+0000","last_peered":"2024-06-29T15:55:38.190459+0000","last_clean":"2024-06-29T15:55:38.190459+0000","last_became_active":"2024-06-29T15:55:38.188248+0000","last_became_peered":"2024-06-29T15:55:38.188248+0000","last_unstale":"2024-06-29T15:55:38.190459+0000","last_undegraded":"2024-06-29T15:55:38.190459+0000","last_fullsized":"2024-06-29T15:55:38.190459+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:59.274484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.202435+0000","last_change":"2024-06-29T15:55:38.195447+0000","last_active":"2024-06-29T15:55:38.202435+0000","last_peered":"2024-06-29T15:55:38.202435+0000","last_clean":"2024-06-29T15:55:38.202435+0000","last_became_active":"2024-06-29T15:55:38.195078+0000","last_became_peered":"2024-06-29T15:55:38.195078+0000","last_unstale":"2024-06-29T15:55:38.202435+0000","last_undegraded":"2024-06-29T15:55:38.202435+0000","last_fullsized":"2024-06-29T15:55:38.202435+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:42:02.030495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195412+0000","last_change":"2024-06-29T15:55:38.190483+0000","last_active":"2024-06-29T15:55:39.195412+0000","last_peered":"2024-06-29T15:55:39.195412+0000","last_clean":"2024-06-29T15:55:39.195412+0000","last_became_active":"2024-06-29T15:55:38.189760+0000","last_became_peered":"2024-06-29T15:55:38.189760+0000","last_unstale":"2024-06-29T15:55:39.195412+0000","last_undegraded":"2024-06-29T15:55:39.195412+0000","last_fullsized":"2024-06-29T15:55:39.195412+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:09.308429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.205781+0000","last_change":"2024-06-29T15:55:38.195132+0000","last_active":"2024-06-29T15:55:38.205781+0000","last_peered":"2024-06-29T15:55:38.205781+0000","last_clean":"2024-06-29T15:55:38.205781+0000","last_became_active":"2024-06-29T15:55:38.194121+0000","last_became_peered":"2024-06-29T15:55:38.194121+0000","last_unstale":"2024-06-29T15:55:38.205781+0000","last_undegraded":"2024-06-29T15:55:38.205781+0000","last_fullsized":"2024-06-29T15:55:38.205781+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:43.447236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.196310+0000","last_change":"2024-06-29T15:55:38.194814+0000","last_active":"2024-06-29T15:55:38.196310+0000","last_peered":"2024-06-29T15:55:38.196310+0000","last_clean":"2024-06-29T15:55:38.196310+0000","last_became_active":"2024-06-29T15:55:38.194517+0000","last_became_peered":"2024-06-29T15:55:38.194517+0000","last_unstale":"2024-06-29T15:55:38.196310+0000","last_undegraded":"2024-06-29T15:55:38.196310+0000","last_fullsized":"2024-06-29T15:55:38.196310+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:11.853597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197416+0000","last_change":"2024-06-29T15:55:38.200729+0000","last_active":"2024-06-29T15:55:39.197416+0000","last_peered":"2024-06-29T15:55:39.197416+0000","last_clean":"2024-06-29T15:55:39.197416+0000","last_became_active":"2024-06-29T15:55:38.200016+0000","last_became_peered":"2024-06-29T15:55:38.200016+0000","last_unstale":"2024-06-29T15:55:39.197416+0000","last_undegraded":"2024-06-29T15:55:39.197416+0000","last_fullsized":"2024-06-29T15:55:39.197416+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:55:54.952820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194709+0000","last_change":"2024-06-29T15:55:38.210694+0000","last_active":"2024-06-29T15:55:39.194709+0000","last_peered":"2024-06-29T15:55:39.194709+0000","last_clean":"2024-06-29T15:55:39.194709+0000","last_became_active":"2024-06-29T15:55:38.210060+0000","last_became_peered":"2024-06-29T15:55:38.210060+0000","last_unstale":"2024-06-29T15:55:39.194709+0000","last_undegraded":"2024-06-29T15:55:39.194709+0000","last_fullsized":"2024-06-29T15:55:39.194709+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:04:49.433010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.206024+0000","last_change":"2024-06-29T15:55:38.202311+0000","last_active":"2024-06-29T15:55:38.206024+0000","last_peered":"2024-06-29T15:55:38.206024+0000","last_clean":"2024-06-29T15:55:38.206024+0000","last_became_active":"2024-06-29T15:55:38.201990+0000","last_became_peered":"2024-06-29T15:55:38.201990+0000","last_unstale":"2024-06-29T15:55:38.206024+0000","last_undegraded":"2024-06-29T15:55:38.206024+0000","last_fullsized":"2024-06-29T15:55:38.206024+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:47.095705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"203'4","reported_seq":18,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638126+0000","last_change":"2024-06-29T15:55:37.193221+0000","last_active":"2024-06-29T15:55:39.638126+0000","last_peered":"2024-06-29T15:55:39.638126+0000","last_clean":"2024-06-29T15:55:39.638126+0000","last_became_active":"2024-06-29T15:55:37.192892+0000","last_became_peered":"2024-06-29T15:55:37.192892+0000","last_unstale":"2024-06-29T15:55:39.638126+0000","last_undegraded":"2024-06-29T15:55:39.638126+0000","last_fullsized":"2024-06-29T15:55:39.638126+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:13.717012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197876+0000","last_change":"2024-06-29T15:55:38.194911+0000","last_active":"2024-06-29T15:55:39.197876+0000","last_peered":"2024-06-29T15:55:39.197876+0000","last_clean":"2024-06-29T15:55:39.197876+0000","last_became_active":"2024-06-29T15:55:38.193559+0000","last_became_peered":"2024-06-29T15:55:38.193559+0000","last_unstale":"2024-06-29T15:55:39.197876+0000","last_undegraded":"2024-06-29T15:55:39.197876+0000","last_fullsized":"2024-06-29T15:55:39.197876+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:54.371370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"203'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195419+0000","last_change":"2024-06-29T15:55:37.179307+0000","last_active":"2024-06-29T15:55:39.195419+0000","last_peered":"2024-06-29T15:55:39.195419+0000","last_clean":"2024-06-29T15:55:39.195419+0000","last_became_active":"2024-06-29T15:55:37.178982+0000","last_became_peered":"2024-06-29T15:55:37.178982+0000","last_unstale":"2024-06-29T15:55:39.195419+0000","last_undegraded":"2024-06-29T15:55:39.195419+0000","last_fullsized":"2024-06-29T15:55:39.195419+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:58:38.335215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.208242+0000","last_change":"2024-06-29T15:55:38.203950+0000","last_active":"2024-06-29T15:55:38.208242+0000","last_peered":"2024-06-29T15:55:38.208242+0000","last_clean":"2024-06-29T15:55:38.208242+0000","last_became_active":"2024-06-29T15:55:38.203618+0000","last_became_peered":"2024-06-29T15:55:38.203618+0000","last_unstale":"2024-06-29T15:55:38.208242+0000","last_undegraded":"2024-06-29T15:55:38.208242+0000","last_fullsized":"2024-06-29T15:55:38.208242+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:57:12.454921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"203'2","reported_seq":14,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.210082+0000","last_change":"2024-06-29T15:55:37.185771+0000","last_active":"2024-06-29T15:55:38.210082+0000","last_peered":"2024-06-29T15:55:38.210082+0000","last_clean":"2024-06-29T15:55:38.210082+0000","last_became_active":"2024-06-29T15:55:37.185459+0000","last_became_peered":"2024-06-29T15:55:37.185459+0000","last_unstale":"2024-06-29T15:55:38.210082+0000","last_undegraded":"2024-06-29T15:55:38.210082+0000","last_fullsized":"2024-06-29T15:55:38.210082+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:08.402451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194990+0000","last_change":"2024-06-29T15:55:38.191279+0000","last_active":"2024-06-29T15:55:39.194990+0000","last_peered":"2024-06-29T15:55:39.194990+0000","last_clean":"2024-06-29T15:55:39.194990+0000","last_became_active":"2024-06-29T15:55:38.190947+0000","last_became_peered":"2024-06-29T15:55:38.190947+0000","last_unstale":"2024-06-29T15:55:39.194990+0000","last_undegraded":"2024-06-29T15:55:39.194990+0000","last_fullsized":"2024-06-29T15:55:39.194990+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:41.092986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195854+0000","last_change":"2024-06-29T15:55:37.188486+0000","last_active":"2024-06-29T15:55:39.195854+0000","last_peered":"2024-06-29T15:55:39.195854+0000","last_clean":"2024-06-29T15:55:39.195854+0000","last_became_active":"2024-06-29T15:55:37.188132+0000","last_became_peered":"2024-06-29T15:55:37.188132+0000","last_unstale":"2024-06-29T15:55:39.195854+0000","last_undegraded":"2024-06-29T15:55:39.195854+0000","last_fullsized":"2024-06-29T15:55:39.195854+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:24:50.835163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194708+0000","last_change":"2024-06-29T15:55:38.205051+0000","last_active":"2024-06-29T15:55:39.194708+0000","last_peered":"2024-06-29T15:55:39.194708+0000","last_clean":"2024-06-29T15:55:39.194708+0000","last_became_active":"2024-06-29T15:55:38.204733+0000","last_became_peered":"2024-06-29T15:55:38.204733+0000","last_unstale":"2024-06-29T15:55:39.194708+0000","last_undegraded":"2024-06-29T15:55:39.194708+0000","last_fullsized":"2024-06-29T15:55:39.194708+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:34:49.084310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"203'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195081+0000","last_change":"2024-06-29T15:55:37.175986+0000","last_active":"2024-06-29T15:55:39.195081+0000","last_peered":"2024-06-29T15:55:39.195081+0000","last_clean":"2024-06-29T15:55:39.195081+0000","last_became_active":"2024-06-29T15:55:37.175630+0000","last_became_peered":"2024-06-29T15:55:37.175630+0000","last_unstale":"2024-06-29T15:55:39.195081+0000","last_undegraded":"2024-06-29T15:55:39.195081+0000","last_fullsized":"2024-06-29T15:55:39.195081+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:29.015055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199828+0000","last_change":"2024-06-29T15:55:38.187783+0000","last_active":"2024-06-29T15:55:39.199828+0000","last_peered":"2024-06-29T15:55:39.199828+0000","last_clean":"2024-06-29T15:55:39.199828+0000","last_became_active":"2024-06-29T15:55:38.187436+0000","last_became_peered":"2024-06-29T15:55:38.187436+0000","last_unstale":"2024-06-29T15:55:39.199828+0000","last_undegraded":"2024-06-29T15:55:39.199828+0000","last_fullsized":"2024-06-29T15:55:39.199828+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:53.341489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"203'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638054+0000","last_change":"2024-06-29T15:55:37.196118+0000","last_active":"2024-06-29T15:55:39.638054+0000","last_peered":"2024-06-29T15:55:39.638054+0000","last_clean":"2024-06-29T15:55:39.638054+0000","last_became_active":"2024-06-29T15:55:37.195732+0000","last_became_peered":"2024-06-29T15:55:37.195732+0000","last_unstale":"2024-06-29T15:55:39.638054+0000","last_undegraded":"2024-06-29T15:55:39.638054+0000","last_fullsized":"2024-06-29T15:55:39.638054+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:22.730506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199864+0000","last_change":"2024-06-29T15:55:38.207856+0000","last_active":"2024-06-29T15:55:39.199864+0000","last_peered":"2024-06-29T15:55:39.199864+0000","last_clean":"2024-06-29T15:55:39.199864+0000","last_became_active":"2024-06-29T15:55:38.207551+0000","last_became_peered":"2024-06-29T15:55:38.207551+0000","last_unstale":"2024-06-29T15:55:39.199864+0000","last_undegraded":"2024-06-29T15:55:39.199864+0000","last_fullsized":"2024-06-29T15:55:39.199864+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:47:20.285236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195328+0000","last_change":"2024-06-29T15:55:37.186599+0000","last_active":"2024-06-29T15:55:39.195328+0000","last_peered":"2024-06-29T15:55:39.195328+0000","last_clean":"2024-06-29T15:55:39.195328+0000","last_became_active":"2024-06-29T15:55:37.186248+0000","last_became_peered":"2024-06-29T15:55:37.186248+0000","last_unstale":"2024-06-29T15:55:39.195328+0000","last_undegraded":"2024-06-29T15:55:39.195328+0000","last_fullsized":"2024-06-29T15:55:39.195328+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:37.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.205998+0000","last_change":"2024-06-29T15:55:38.201525+0000","last_active":"2024-06-29T15:55:38.205998+0000","last_peered":"2024-06-29T15:55:38.205998+0000","last_clean":"2024-06-29T15:55:38.205998+0000","last_became_active":"2024-06-29T15:55:38.201218+0000","last_became_peered":"2024-06-29T15:55:38.201218+0000","last_unstale":"2024-06-29T15:55:38.205998+0000","last_undegraded":"2024-06-29T15:55:38.205998+0000","last_fullsized":"2024-06-29T15:55:38.205998+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:14:18.988229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195192+0000","last_change":"2024-06-29T15:55:37.195659+0000","last_active":"2024-06-29T15:55:39.195192+0000","last_peered":"2024-06-29T15:55:39.195192+0000","last_clean":"2024-06-29T15:55:39.195192+0000","last_became_active":"2024-06-29T15:55:37.195097+0000","last_became_peered":"2024-06-29T15:55:37.195097+0000","last_unstale":"2024-06-29T15:55:39.195192+0000","last_undegraded":"2024-06-29T15:55:39.195192+0000","last_fullsized":"2024-06-29T15:55:39.195192+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:59:29.566045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195342+0000","last_change":"2024-06-29T15:55:38.196779+0000","last_active":"2024-06-29T15:55:39.195342+0000","last_peered":"2024-06-29T15:55:39.195342+0000","last_clean":"2024-06-29T15:55:39.195342+0000","last_became_active":"2024-06-29T15:55:38.196420+0000","last_became_peered":"2024-06-29T15:55:38.196420+0000","last_unstale":"2024-06-29T15:55:39.195342+0000","last_undegraded":"2024-06-29T15:55:39.195342+0000","last_fullsized":"2024-06-29T15:55:39.195342+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:55:22.062078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199632+0000","last_change":"2024-06-29T15:55:37.184280+0000","last_active":"2024-06-29T15:55:39.199632+0000","last_peered":"2024-06-29T15:55:39.199632+0000","last_clean":"2024-06-29T15:55:39.199632+0000","last_became_active":"2024-06-29T15:55:37.183967+0000","last_became_peered":"2024-06-29T15:55:37.183967+0000","last_unstale":"2024-06-29T15:55:39.199632+0000","last_undegraded":"2024-06-29T15:55:39.199632+0000","last_fullsized":"2024-06-29T15:55:39.199632+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.078739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195735+0000","last_change":"2024-06-29T15:55:38.200107+0000","last_active":"2024-06-29T15:55:39.195735+0000","last_peered":"2024-06-29T15:55:39.195735+0000","last_clean":"2024-06-29T15:55:39.195735+0000","last_became_active":"2024-06-29T15:55:38.199669+0000","last_became_peered":"2024-06-29T15:55:38.199669+0000","last_unstale":"2024-06-29T15:55:39.195735+0000","last_undegraded":"2024-06-29T15:55:39.195735+0000","last_fullsized":"2024-06-29T15:55:39.195735+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:41.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"203'3","reported_seq":17,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637952+0000","last_change":"2024-06-29T15:55:37.197033+0000","last_active":"2024-06-29T15:55:39.637952+0000","last_peered":"2024-06-29T15:55:39.637952+0000","last_clean":"2024-06-29T15:55:39.637952+0000","last_became_active":"2024-06-29T15:55:37.196669+0000","last_became_peered":"2024-06-29T15:55:37.196669+0000","last_unstale":"2024-06-29T15:55:39.637952+0000","last_undegraded":"2024-06-29T15:55:39.637952+0000","last_fullsized":"2024-06-29T15:55:39.637952+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:55.689057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.200002+0000","last_change":"2024-06-29T15:55:38.188546+0000","last_active":"2024-06-29T15:55:39.200002+0000","last_peered":"2024-06-29T15:55:39.200002+0000","last_clean":"2024-06-29T15:55:39.200002+0000","last_became_active":"2024-06-29T15:55:38.188220+0000","last_became_peered":"2024-06-29T15:55:38.188220+0000","last_unstale":"2024-06-29T15:55:39.200002+0000","last_undegraded":"2024-06-29T15:55:39.200002+0000","last_fullsized":"2024-06-29T15:55:39.200002+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:17.474861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"203'6","reported_seq":20,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195988+0000","last_change":"2024-06-29T15:55:37.185494+0000","last_active":"2024-06-29T15:55:39.195988+0000","last_peered":"2024-06-29T15:55:39.195988+0000","last_clean":"2024-06-29T15:55:39.195988+0000","last_became_active":"2024-06-29T15:55:37.184064+0000","last_became_peered":"2024-06-29T15:55:37.184064+0000","last_unstale":"2024-06-29T15:55:39.195988+0000","last_undegraded":"2024-06-29T15:55:39.195988+0000","last_fullsized":"2024-06-29T15:55:39.195988+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:37.797870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197413+0000","last_change":"2024-06-29T15:55:38.194012+0000","last_active":"2024-06-29T15:55:39.197413+0000","last_peered":"2024-06-29T15:55:39.197413+0000","last_clean":"2024-06-29T15:55:39.197413+0000","last_became_active":"2024-06-29T15:55:38.193616+0000","last_became_peered":"2024-06-29T15:55:38.193616+0000","last_unstale":"2024-06-29T15:55:39.197413+0000","last_undegraded":"2024-06-29T15:55:39.197413+0000","last_fullsized":"2024-06-29T15:55:39.197413+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:32.889293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195824+0000","last_change":"2024-06-29T15:55:37.190946+0000","last_active":"2024-06-29T15:55:39.195824+0000","last_peered":"2024-06-29T15:55:39.195824+0000","last_clean":"2024-06-29T15:55:39.195824+0000","last_became_active":"2024-06-29T15:55:37.190639+0000","last_became_peered":"2024-06-29T15:55:37.190639+0000","last_unstale":"2024-06-29T15:55:39.195824+0000","last_undegraded":"2024-06-29T15:55:39.195824+0000","last_fullsized":"2024-06-29T15:55:39.195824+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:11.900910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194708+0000","last_change":"2024-06-29T15:55:38.201962+0000","last_active":"2024-06-29T15:55:39.194708+0000","last_peered":"2024-06-29T15:55:39.194708+0000","last_clean":"2024-06-29T15:55:39.194708+0000","last_became_active":"2024-06-29T15:55:38.201613+0000","last_became_peered":"2024-06-29T15:55:38.201613+0000","last_unstale":"2024-06-29T15:55:39.194708+0000","last_undegraded":"2024-06-29T15:55:39.194708+0000","last_fullsized":"2024-06-29T15:55:39.194708+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:57.852528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.200027+0000","last_change":"2024-06-29T15:55:37.190807+0000","last_active":"2024-06-29T15:55:39.200027+0000","last_peered":"2024-06-29T15:55:39.200027+0000","last_clean":"2024-06-29T15:55:39.200027+0000","last_became_active":"2024-06-29T15:55:37.190477+0000","last_became_peered":"2024-06-29T15:55:37.190477+0000","last_unstale":"2024-06-29T15:55:39.200027+0000","last_undegraded":"2024-06-29T15:55:39.200027+0000","last_fullsized":"2024-06-29T15:55:39.200027+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:06:39.594276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195427+0000","last_change":"2024-06-29T15:55:38.205740+0000","last_active":"2024-06-29T15:55:39.195427+0000","last_peered":"2024-06-29T15:55:39.195427+0000","last_clean":"2024-06-29T15:55:39.195427+0000","last_became_active":"2024-06-29T15:55:38.205406+0000","last_became_peered":"2024-06-29T15:55:38.205406+0000","last_unstale":"2024-06-29T15:55:39.195427+0000","last_undegraded":"2024-06-29T15:55:39.195427+0000","last_fullsized":"2024-06-29T15:55:39.195427+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:57.362943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637711+0000","last_change":"2024-06-29T15:55:37.192976+0000","last_active":"2024-06-29T15:55:39.637711+0000","last_peered":"2024-06-29T15:55:39.637711+0000","last_clean":"2024-06-29T15:55:39.637711+0000","last_became_active":"2024-06-29T15:55:37.192647+0000","last_became_peered":"2024-06-29T15:55:37.192647+0000","last_unstale":"2024-06-29T15:55:39.637711+0000","last_undegraded":"2024-06-29T15:55:39.637711+0000","last_fullsized":"2024-06-29T15:55:39.637711+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:48.150158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195828+0000","last_change":"2024-06-29T15:55:38.211984+0000","last_active":"2024-06-29T15:55:39.195828+0000","last_peered":"2024-06-29T15:55:39.195828+0000","last_clean":"2024-06-29T15:55:39.195828+0000","last_became_active":"2024-06-29T15:55:38.211374+0000","last_became_peered":"2024-06-29T15:55:38.211374+0000","last_unstale":"2024-06-29T15:55:39.195828+0000","last_undegraded":"2024-06-29T15:55:39.195828+0000","last_fullsized":"2024-06-29T15:55:39.195828+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:08:17.996282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"203'8","reported_seq":20,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.214855+0000","last_change":"2024-06-29T15:55:37.187789+0000","last_active":"2024-06-29T15:55:38.214855+0000","last_peered":"2024-06-29T15:55:38.214855+0000","last_clean":"2024-06-29T15:55:38.214855+0000","last_became_active":"2024-06-29T15:55:37.187478+0000","last_became_peered":"2024-06-29T15:55:37.187478+0000","last_unstale":"2024-06-29T15:55:38.214855+0000","last_undegraded":"2024-06-29T15:55:38.214855+0000","last_fullsized":"2024-06-29T15:55:38.214855+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:18.543911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197647+0000","last_change":"2024-06-29T15:55:38.201663+0000","last_active":"2024-06-29T15:55:39.197647+0000","last_peered":"2024-06-29T15:55:39.197647+0000","last_clean":"2024-06-29T15:55:39.197647+0000","last_became_active":"2024-06-29T15:55:38.199999+0000","last_became_peered":"2024-06-29T15:55:38.199999+0000","last_unstale":"2024-06-29T15:55:39.197647+0000","last_undegraded":"2024-06-29T15:55:39.197647+0000","last_fullsized":"2024-06-29T15:55:39.197647+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:19.552281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"203'1","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.213120+0000","last_change":"2024-06-29T15:55:37.190954+0000","last_active":"2024-06-29T15:55:38.213120+0000","last_peered":"2024-06-29T15:55:38.213120+0000","last_clean":"2024-06-29T15:55:38.213120+0000","last_became_active":"2024-06-29T15:55:37.190614+0000","last_became_peered":"2024-06-29T15:55:37.190614+0000","last_unstale":"2024-06-29T15:55:38.213120+0000","last_undegraded":"2024-06-29T15:55:38.213120+0000","last_fullsized":"2024-06-29T15:55:38.213120+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:41:30.536413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195131+0000","last_change":"2024-06-29T15:55:38.209000+0000","last_active":"2024-06-29T15:55:39.195131+0000","last_peered":"2024-06-29T15:55:39.195131+0000","last_clean":"2024-06-29T15:55:39.195131+0000","last_became_active":"2024-06-29T15:55:38.208666+0000","last_became_peered":"2024-06-29T15:55:38.208666+0000","last_unstale":"2024-06-29T15:55:39.195131+0000","last_undegraded":"2024-06-29T15:55:39.195131+0000","last_fullsized":"2024-06-29T15:55:39.195131+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:59.950282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638010+0000","last_change":"2024-06-29T15:55:37.192980+0000","last_active":"2024-06-29T15:55:39.638010+0000","last_peered":"2024-06-29T15:55:39.638010+0000","last_clean":"2024-06-29T15:55:39.638010+0000","last_became_active":"2024-06-29T15:55:37.192601+0000","last_became_peered":"2024-06-29T15:55:37.192601+0000","last_unstale":"2024-06-29T15:55:39.638010+0000","last_undegraded":"2024-06-29T15:55:39.638010+0000","last_fullsized":"2024-06-29T15:55:39.638010+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:52.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.190405+0000","last_change":"2024-06-29T15:55:38.187492+0000","last_active":"2024-06-29T15:55:38.190405+0000","last_peered":"2024-06-29T15:55:38.190405+0000","last_clean":"2024-06-29T15:55:38.190405+0000","last_became_active":"2024-06-29T15:55:38.187160+0000","last_became_peered":"2024-06-29T15:55:38.187160+0000","last_unstale":"2024-06-29T15:55:38.190405+0000","last_undegraded":"2024-06-29T15:55:38.190405+0000","last_fullsized":"2024-06-29T15:55:38.190405+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:46.966818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"203'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195104+0000","last_change":"2024-06-29T15:55:37.181914+0000","last_active":"2024-06-29T15:55:39.195104+0000","last_peered":"2024-06-29T15:55:39.195104+0000","last_clean":"2024-06-29T15:55:39.195104+0000","last_became_active":"2024-06-29T15:55:37.181569+0000","last_became_peered":"2024-06-29T15:55:37.181569+0000","last_unstale":"2024-06-29T15:55:39.195104+0000","last_undegraded":"2024-06-29T15:55:39.195104+0000","last_fullsized":"2024-06-29T15:55:39.195104+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:48.289568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.200119+0000","last_change":"2024-06-29T15:55:38.189467+0000","last_active":"2024-06-29T15:55:39.200119+0000","last_peered":"2024-06-29T15:55:39.200119+0000","last_clean":"2024-06-29T15:55:39.200119+0000","last_became_active":"2024-06-29T15:55:38.189153+0000","last_became_peered":"2024-06-29T15:55:38.189153+0000","last_unstale":"2024-06-29T15:55:39.200119+0000","last_undegraded":"2024-06-29T15:55:39.200119+0000","last_fullsized":"2024-06-29T15:55:39.200119+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:33.643126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.182047+0000","last_change":"2024-06-29T15:55:37.182273+0000","last_active":"2024-06-29T15:55:38.182047+0000","last_peered":"2024-06-29T15:55:38.182047+0000","last_clean":"2024-06-29T15:55:38.182047+0000","last_became_active":"2024-06-29T15:55:37.181918+0000","last_became_peered":"2024-06-29T15:55:37.181918+0000","last_unstale":"2024-06-29T15:55:38.182047+0000","last_undegraded":"2024-06-29T15:55:38.182047+0000","last_fullsized":"2024-06-29T15:55:38.182047+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:42.174671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199302+0000","last_change":"2024-06-29T15:55:38.194184+0000","last_active":"2024-06-29T15:55:39.199302+0000","last_peered":"2024-06-29T15:55:39.199302+0000","last_clean":"2024-06-29T15:55:39.199302+0000","last_became_active":"2024-06-29T15:55:38.193851+0000","last_became_peered":"2024-06-29T15:55:38.193851+0000","last_unstale":"2024-06-29T15:55:39.199302+0000","last_undegraded":"2024-06-29T15:55:39.199302+0000","last_fullsized":"2024-06-29T15:55:39.199302+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:43.333640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195937+0000","last_change":"2024-06-29T15:55:37.193197+0000","last_active":"2024-06-29T15:55:39.195937+0000","last_peered":"2024-06-29T15:55:39.195937+0000","last_clean":"2024-06-29T15:55:39.195937+0000","last_became_active":"2024-06-29T15:55:37.192852+0000","last_became_peered":"2024-06-29T15:55:37.192852+0000","last_unstale":"2024-06-29T15:55:39.195937+0000","last_undegraded":"2024-06-29T15:55:39.195937+0000","last_fullsized":"2024-06-29T15:55:39.195937+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:30.038803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637772+0000","last_change":"2024-06-29T15:55:38.210931+0000","last_active":"2024-06-29T15:55:39.637772+0000","last_peered":"2024-06-29T15:55:39.637772+0000","last_clean":"2024-06-29T15:55:39.637772+0000","last_became_active":"2024-06-29T15:55:38.210611+0000","last_became_peered":"2024-06-29T15:55:38.210611+0000","last_unstale":"2024-06-29T15:55:39.637772+0000","last_undegraded":"2024-06-29T15:55:39.637772+0000","last_fullsized":"2024-06-29T15:55:39.637772+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:12.331030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637857+0000","last_change":"2024-06-29T15:55:37.194538+0000","last_active":"2024-06-29T15:55:39.637857+0000","last_peered":"2024-06-29T15:55:39.637857+0000","last_clean":"2024-06-29T15:55:39.637857+0000","last_became_active":"2024-06-29T15:55:37.193803+0000","last_became_peered":"2024-06-29T15:55:37.193803+0000","last_unstale":"2024-06-29T15:55:39.637857+0000","last_undegraded":"2024-06-29T15:55:39.637857+0000","last_fullsized":"2024-06-29T15:55:39.637857+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:36.758960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637916+0000","last_change":"2024-06-29T15:55:38.200641+0000","last_active":"2024-06-29T15:55:39.637916+0000","last_peered":"2024-06-29T15:55:39.637916+0000","last_clean":"2024-06-29T15:55:39.637916+0000","last_became_active":"2024-06-29T15:55:38.199946+0000","last_became_peered":"2024-06-29T15:55:38.199946+0000","last_unstale":"2024-06-29T15:55:39.637916+0000","last_undegraded":"2024-06-29T15:55:39.637916+0000","last_fullsized":"2024-06-29T15:55:39.637916+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:43.997129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197599+0000","last_change":"2024-06-29T15:55:37.187075+0000","last_active":"2024-06-29T15:55:39.197599+0000","last_peered":"2024-06-29T15:55:39.197599+0000","last_clean":"2024-06-29T15:55:39.197599+0000","last_became_active":"2024-06-29T15:55:37.186757+0000","last_became_peered":"2024-06-29T15:55:37.186757+0000","last_unstale":"2024-06-29T15:55:39.197599+0000","last_undegraded":"2024-06-29T15:55:39.197599+0000","last_fullsized":"2024-06-29T15:55:39.197599+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:19:56.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195475+0000","last_change":"2024-06-29T15:55:38.194353+0000","last_active":"2024-06-29T15:55:39.195475+0000","last_peered":"2024-06-29T15:55:39.195475+0000","last_clean":"2024-06-29T15:55:39.195475+0000","last_became_active":"2024-06-29T15:55:38.194007+0000","last_became_peered":"2024-06-29T15:55:38.194007+0000","last_unstale":"2024-06-29T15:55:39.195475+0000","last_undegraded":"2024-06-29T15:55:39.195475+0000","last_fullsized":"2024-06-29T15:55:39.195475+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:37.766463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194785+0000","last_change":"2024-06-29T15:55:37.183897+0000","last_active":"2024-06-29T15:55:39.194785+0000","last_peered":"2024-06-29T15:55:39.194785+0000","last_clean":"2024-06-29T15:55:39.194785+0000","last_became_active":"2024-06-29T15:55:37.183572+0000","last_became_peered":"2024-06-29T15:55:37.183572+0000","last_unstale":"2024-06-29T15:55:39.194785+0000","last_undegraded":"2024-06-29T15:55:39.194785+0000","last_fullsized":"2024-06-29T15:55:39.194785+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:30.660501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197868+0000","last_change":"2024-06-29T15:55:38.189708+0000","last_active":"2024-06-29T15:55:39.197868+0000","last_peered":"2024-06-29T15:55:39.197868+0000","last_clean":"2024-06-29T15:55:39.197868+0000","last_became_active":"2024-06-29T15:55:38.189342+0000","last_became_peered":"2024-06-29T15:55:38.189342+0000","last_unstale":"2024-06-29T15:55:39.197868+0000","last_undegraded":"2024-06-29T15:55:39.197868+0000","last_fullsized":"2024-06-29T15:55:39.197868+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:01.403464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638108+0000","last_change":"2024-06-29T15:55:37.191777+0000","last_active":"2024-06-29T15:55:39.638108+0000","last_peered":"2024-06-29T15:55:39.638108+0000","last_clean":"2024-06-29T15:55:39.638108+0000","last_became_active":"2024-06-29T15:55:37.191386+0000","last_became_peered":"2024-06-29T15:55:37.191386+0000","last_unstale":"2024-06-29T15:55:39.638108+0000","last_undegraded":"2024-06-29T15:55:39.638108+0000","last_fullsized":"2024-06-29T15:55:39.638108+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:16.469829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199919+0000","last_change":"2024-06-29T15:55:38.188019+0000","last_active":"2024-06-29T15:55:39.199919+0000","last_peered":"2024-06-29T15:55:39.199919+0000","last_clean":"2024-06-29T15:55:39.199919+0000","last_became_active":"2024-06-29T15:55:38.187517+0000","last_became_peered":"2024-06-29T15:55:38.187517+0000","last_unstale":"2024-06-29T15:55:39.199919+0000","last_undegraded":"2024-06-29T15:55:39.199919+0000","last_fullsized":"2024-06-29T15:55:39.199919+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:49.155189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"204'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.198954+0000","last_change":"2024-06-29T15:55:37.191495+0000","last_active":"2024-06-29T15:55:39.198954+0000","last_peered":"2024-06-29T15:55:39.198954+0000","last_clean":"2024-06-29T15:55:39.198954+0000","last_became_active":"2024-06-29T15:55:37.191157+0000","last_became_peered":"2024-06-29T15:55:37.191157+0000","last_unstale":"2024-06-29T15:55:39.198954+0000","last_undegraded":"2024-06-29T15:55:39.198954+0000","last_fullsized":"2024-06-29T15:55:39.198954+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:23:30.328659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.206473+0000","last_change":"2024-06-29T15:55:38.202742+0000","last_active":"2024-06-29T15:55:38.206473+0000","last_peered":"2024-06-29T15:55:38.206473+0000","last_clean":"2024-06-29T15:55:38.206473+0000","last_became_active":"2024-06-29T15:55:38.202447+0000","last_became_peered":"2024-06-29T15:55:38.202447+0000","last_unstale":"2024-06-29T15:55:38.206473+0000","last_undegraded":"2024-06-29T15:55:38.206473+0000","last_fullsized":"2024-06-29T15:55:38.206473+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:29.245610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195329+0000","last_change":"2024-06-29T15:55:37.179052+0000","last_active":"2024-06-29T15:55:39.195329+0000","last_peered":"2024-06-29T15:55:39.195329+0000","last_clean":"2024-06-29T15:55:39.195329+0000","last_became_active":"2024-06-29T15:55:37.178459+0000","last_became_peered":"2024-06-29T15:55:37.178459+0000","last_unstale":"2024-06-29T15:55:39.195329+0000","last_undegraded":"2024-06-29T15:55:39.195329+0000","last_fullsized":"2024-06-29T15:55:39.195329+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:15:26.721841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197935+0000","last_change":"2024-06-29T15:55:38.190604+0000","last_active":"2024-06-29T15:55:39.197935+0000","last_peered":"2024-06-29T15:55:39.197935+0000","last_clean":"2024-06-29T15:55:39.197935+0000","last_became_active":"2024-06-29T15:55:38.190196+0000","last_became_peered":"2024-06-29T15:55:38.190196+0000","last_unstale":"2024-06-29T15:55:39.197935+0000","last_undegraded":"2024-06-29T15:55:39.197935+0000","last_fullsized":"2024-06-29T15:55:39.197935+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:06:14.283587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195165+0000","last_change":"2024-06-29T15:55:37.188647+0000","last_active":"2024-06-29T15:55:39.195165+0000","last_peered":"2024-06-29T15:55:39.195165+0000","last_clean":"2024-06-29T15:55:39.195165+0000","last_became_active":"2024-06-29T15:55:37.188330+0000","last_became_peered":"2024-06-29T15:55:37.188330+0000","last_unstale":"2024-06-29T15:55:39.195165+0000","last_undegraded":"2024-06-29T15:55:39.195165+0000","last_fullsized":"2024-06-29T15:55:39.195165+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:33:04.734367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194644+0000","last_change":"2024-06-29T15:55:38.197753+0000","last_active":"2024-06-29T15:55:39.194644+0000","last_peered":"2024-06-29T15:55:39.194644+0000","last_clean":"2024-06-29T15:55:39.194644+0000","last_became_active":"2024-06-29T15:55:38.197174+0000","last_became_peered":"2024-06-29T15:55:38.197174+0000","last_unstale":"2024-06-29T15:55:39.194644+0000","last_undegraded":"2024-06-29T15:55:39.194644+0000","last_fullsized":"2024-06-29T15:55:39.194644+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:11.837664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.189590+0000","last_change":"2024-06-29T15:55:37.184847+0000","last_active":"2024-06-29T15:55:38.189590+0000","last_peered":"2024-06-29T15:55:38.189590+0000","last_clean":"2024-06-29T15:55:38.189590+0000","last_became_active":"2024-06-29T15:55:37.184537+0000","last_became_peered":"2024-06-29T15:55:37.184537+0000","last_unstale":"2024-06-29T15:55:38.189590+0000","last_undegraded":"2024-06-29T15:55:38.189590+0000","last_fullsized":"2024-06-29T15:55:38.189590+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:48.149496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195229+0000","last_change":"2024-06-29T15:55:38.206586+0000","last_active":"2024-06-29T15:55:39.195229+0000","last_peered":"2024-06-29T15:55:39.195229+0000","last_clean":"2024-06-29T15:55:39.195229+0000","last_became_active":"2024-06-29T15:55:38.206268+0000","last_became_peered":"2024-06-29T15:55:38.206268+0000","last_unstale":"2024-06-29T15:55:39.195229+0000","last_undegraded":"2024-06-29T15:55:39.195229+0000","last_fullsized":"2024-06-29T15:55:39.195229+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:00.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.189134+0000","last_change":"2024-06-29T15:55:37.193390+0000","last_active":"2024-06-29T15:55:38.189134+0000","last_peered":"2024-06-29T15:55:38.189134+0000","last_clean":"2024-06-29T15:55:38.189134+0000","last_became_active":"2024-06-29T15:55:37.193050+0000","last_became_peered":"2024-06-29T15:55:37.193050+0000","last_unstale":"2024-06-29T15:55:38.189134+0000","last_undegraded":"2024-06-29T15:55:38.189134+0000","last_fullsized":"2024-06-29T15:55:38.189134+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:46.897497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199998+0000","last_change":"2024-06-29T15:55:38.192777+0000","last_active":"2024-06-29T15:55:39.199998+0000","last_peered":"2024-06-29T15:55:39.199998+0000","last_clean":"2024-06-29T15:55:39.199998+0000","last_became_active":"2024-06-29T15:55:38.192434+0000","last_became_peered":"2024-06-29T15:55:38.192434+0000","last_unstale":"2024-06-29T15:55:39.199998+0000","last_undegraded":"2024-06-29T15:55:39.199998+0000","last_fullsized":"2024-06-29T15:55:39.199998+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:53.257489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197774+0000","last_change":"2024-06-29T15:55:37.186287+0000","last_active":"2024-06-29T15:55:39.197774+0000","last_peered":"2024-06-29T15:55:39.197774+0000","last_clean":"2024-06-29T15:55:39.197774+0000","last_became_active":"2024-06-29T15:55:37.185949+0000","last_became_peered":"2024-06-29T15:55:37.185949+0000","last_unstale":"2024-06-29T15:55:39.197774+0000","last_undegraded":"2024-06-29T15:55:39.197774+0000","last_fullsized":"2024-06-29T15:55:39.197774+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:01.095749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.205627+0000","last_change":"2024-06-29T15:55:38.200595+0000","last_active":"2024-06-29T15:55:38.205627+0000","last_peered":"2024-06-29T15:55:38.205627+0000","last_clean":"2024-06-29T15:55:38.205627+0000","last_became_active":"2024-06-29T15:55:38.200290+0000","last_became_peered":"2024-06-29T15:55:38.200290+0000","last_unstale":"2024-06-29T15:55:38.205627+0000","last_undegraded":"2024-06-29T15:55:38.205627+0000","last_fullsized":"2024-06-29T15:55:38.205627+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:17:24.117324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194657+0000","last_change":"2024-06-29T15:55:37.177567+0000","last_active":"2024-06-29T15:55:39.194657+0000","last_peered":"2024-06-29T15:55:39.194657+0000","last_clean":"2024-06-29T15:55:39.194657+0000","last_became_active":"2024-06-29T15:55:37.177218+0000","last_became_peered":"2024-06-29T15:55:37.177218+0000","last_unstale":"2024-06-29T15:55:39.194657+0000","last_undegraded":"2024-06-29T15:55:39.194657+0000","last_fullsized":"2024-06-29T15:55:39.194657+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:18.225574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195120+0000","last_change":"2024-06-29T15:55:38.201348+0000","last_active":"2024-06-29T15:55:39.195120+0000","last_peered":"2024-06-29T15:55:39.195120+0000","last_clean":"2024-06-29T15:55:39.195120+0000","last_became_active":"2024-06-29T15:55:38.201045+0000","last_became_peered":"2024-06-29T15:55:38.201045+0000","last_unstale":"2024-06-29T15:55:39.195120+0000","last_undegraded":"2024-06-29T15:55:39.195120+0000","last_fullsized":"2024-06-29T15:55:39.195120+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:15.600066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637731+0000","last_change":"2024-06-29T15:55:37.187445+0000","last_active":"2024-06-29T15:55:39.637731+0000","last_peered":"2024-06-29T15:55:39.637731+0000","last_clean":"2024-06-29T15:55:39.637731+0000","last_became_active":"2024-06-29T15:55:37.186656+0000","last_became_peered":"2024-06-29T15:55:37.186656+0000","last_unstale":"2024-06-29T15:55:39.637731+0000","last_undegraded":"2024-06-29T15:55:39.637731+0000","last_fullsized":"2024-06-29T15:55:39.637731+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:32.638095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"193'219","reported_seq":1016,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.189590+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:55:38.189590+0000","last_peered":"2024-06-29T15:55:38.189590+0000","last_clean":"2024-06-29T15:55:38.189590+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:55:38.189590+0000","last_undegraded":"2024-06-29T15:55:38.189590+0000","last_fullsized":"2024-06-29T15:55:38.189590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543267,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52296,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319544,"statfs":{"total":96636764160,"available":96583213056,"internally_reserved":0,"allocated":3133440,"data_stored":1890995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41996,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329844,"statfs":{"total":96636764160,"available":96593760256,"internally_reserved":0,"allocated":2555904,"data_stored":1307445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543267,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47568,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324272,"statfs":{"total":96636764160,"available":96588054528,"internally_reserved":0,"allocated":2555904,"data_stored":1302261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47584,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324256,"statfs":{"total":96636764160,"available":96588038144,"internally_reserved":0,"allocated":2568192,"data_stored":1309021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47572,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324268,"statfs":{"total":96636764160,"available":96588050432,"internally_reserved":0,"allocated":2560000,"data_stored":1307535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575972,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47560,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324280,"statfs":{"total":96636764160,"available":96588062720,"internally_reserved":0,"allocated":2543616,"data_stored":1301209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510563,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47548,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324292,"statfs":{"total":96636764160,"available":96588075008,"internally_reserved":0,"allocated":2539520,"data_stored":1300627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510565,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42580,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329260,"statfs":{"total":96636764160,"available":96593162240,"internally_reserved":0,"allocated":3149824,"data_stored":1891723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:55:42.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:55:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.422+0000 7fb633319640 1 -- 172.21.15.59:0/961261202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c08c190 msgr2=0x7fb62c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.422+0000 7fb633319640 1 --2- 172.21.15.59:0/961261202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c08c190 0x7fb62c0cf080 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009830 tx=0x7fb61c02a4d0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.422+0000 7fb633319640 1 -- 172.21.15.59:0/961261202 shutdown_connections 2024-06-29T15:55:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.422+0000 7fb633319640 1 --2- 172.21.15.59:0/961261202 >> v2:172.21.15.153:3300/0 conn(0x7fb62c0ce950 0x7fb62c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.422+0000 7fb633319640 1 --2- 172.21.15.59:0/961261202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c08c190 0x7fb62c0cf080 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.422+0000 7fb633319640 1 -- 172.21.15.59:0/961261202 >> 172.21.15.59:0/961261202 conn(0x7fb62c0b5610 msgr2=0x7fb62c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.422+0000 7fb633319640 1 -- 172.21.15.59:0/961261202 shutdown_connections 2024-06-29T15:55:42.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.422+0000 7fb633319640 1 -- 172.21.15.59:0/961261202 wait complete. 2024-06-29T15:55:42.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.424+0000 7fb633319640 1 Processor -- start 2024-06-29T15:55:42.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.424+0000 7fb633319640 1 -- start start 2024-06-29T15:55:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.425+0000 7fb633319640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb62c08c190 0x7fb62c15bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.425+0000 7fb633319640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0ce950 0x7fb62c157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.425+0000 7fb633319640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c0cf5c0 0x7fb62c1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.425+0000 7fb633319640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c0b7f60 con 0x7fb62c0ce950 2024-06-29T15:55:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.425+0000 7fb633319640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c0b7de0 con 0x7fb62c0cf5c0 2024-06-29T15:55:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.425+0000 7fb633319640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb62c0b80e0 con 0x7fb62c08c190 2024-06-29T15:55:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0ce950 0x7fb62c157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0ce950 0x7fb62c157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58748/0 (socket says 172.21.15.59:58748) 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 -- 172.21.15.59:0/3687579446 learned_addr learned my addr 172.21.15.59:0/3687579446 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6318b6640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c0cf5c0 0x7fb62c1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6310b5640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb62c08c190 0x7fb62c15bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 -- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb62c08c190 msgr2=0x7fb62c15bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb62c08c190 0x7fb62c15bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 -- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c0cf5c0 msgr2=0x7fb62c1582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c0cf5c0 0x7fb62c1582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 -- 172.21.15.59:0/3687579446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c1568c0 con 0x7fb62c0ce950 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb6308b4640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0ce950 0x7fb62c157d80 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fb62000c390 tx=0x7fb62000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb629ffb640 1 -- 172.21.15.59:0/3687579446 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb620017070 con 0x7fb62c0ce950 2024-06-29T15:55:42.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb629ffb640 1 -- 172.21.15.59:0/3687579446 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6200035f0 con 0x7fb62c0ce950 2024-06-29T15:55:42.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62c156ba0 con 0x7fb62c0ce950 2024-06-29T15:55:42.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.426+0000 7fb629ffb640 1 -- 172.21.15.59:0/3687579446 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb620003a20 con 0x7fb62c0ce950 2024-06-29T15:55:42.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.427+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb62c157160 con 0x7fb62c0ce950 2024-06-29T15:55:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.428+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb62c156630 con 0x7fb62c0ce950 2024-06-29T15:55:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.428+0000 7fb629ffb640 1 -- 172.21.15.59:0/3687579446 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fb62001a070 con 0x7fb62c0ce950 2024-06-29T15:55:42.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.429+0000 7fb629ffb640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb6180763f0 0x7fb618078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.429+0000 7fb629ffb640 1 -- 172.21.15.59:0/3687579446 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7fb620096d60 con 0x7fb62c0ce950 2024-06-29T15:55:42.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.429+0000 7fb6310b5640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb6180763f0 0x7fb618078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.430+0000 7fb6310b5640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb6180763f0 0x7fb618078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb61c034000 tx=0x7fb61c004730 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.433+0000 7fb629ffb640 1 -- 172.21.15.59:0/3687579446 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb62005fda0 con 0x7fb62c0ce950 2024-06-29T15:55:42.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.625+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 --> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb62c0cf080 con 0x7fb6180763f0 2024-06-29T15:55:42.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.629+0000 7fb629ffb640 1 -- 172.21.15.59:0/3687579446 <== mgr.10063 v2:172.21.15.153:6832/493926731 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252481 (secure 0 0 0) 0x7fb62c0cf080 con 0x7fb6180763f0 2024-06-29T15:55:42.633 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:42.635 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:55:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb6180763f0 msgr2=0x7fb618078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb6180763f0 0x7fb618078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb61c034000 tx=0x7fb61c004730 comp rx=0 tx=0).stop 2024-06-29T15:55:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0ce950 msgr2=0x7fb62c157d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0ce950 0x7fb62c157d80 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fb62000c390 tx=0x7fb62000c850 comp rx=0 tx=0).stop 2024-06-29T15:55:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 shutdown_connections 2024-06-29T15:55:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb6180763f0 0x7fb618078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb62c0cf5c0 0x7fb62c1582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0ce950 0x7fb62c157d80 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 --2- 172.21.15.59:0/3687579446 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb62c08c190 0x7fb62c15bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 >> 172.21.15.59:0/3687579446 conn(0x7fb62c0b5610 msgr2=0x7fb62c0686d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 shutdown_connections 2024-06-29T15:55:42.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.633+0000 7fb633319640 1 -- 172.21.15.59:0/3687579446 wait complete. 2024-06-29T15:55:42.645 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:55:41.672243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":398,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":258,"ondisk_log_size":258,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":374704,"kb_used_data":21100,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754600016,"statfs":{"total":773094113280,"available":772710416384,"internally_reserved":0,"allocated":21606400,"data_stored":11610816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":361942578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.504202"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199831+0000","last_change":"2024-06-29T15:55:38.210448+0000","last_active":"2024-06-29T15:55:39.199831+0000","last_peered":"2024-06-29T15:55:39.199831+0000","last_clean":"2024-06-29T15:55:39.199831+0000","last_became_active":"2024-06-29T15:55:38.209396+0000","last_became_peered":"2024-06-29T15:55:38.209396+0000","last_unstale":"2024-06-29T15:55:39.199831+0000","last_undegraded":"2024-06-29T15:55:39.199831+0000","last_fullsized":"2024-06-29T15:55:39.199831+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:52.213826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199692+0000","last_change":"2024-06-29T15:55:38.189981+0000","last_active":"2024-06-29T15:55:39.199692+0000","last_peered":"2024-06-29T15:55:39.199692+0000","last_clean":"2024-06-29T15:55:39.199692+0000","last_became_active":"2024-06-29T15:55:38.189652+0000","last_became_peered":"2024-06-29T15:55:38.189652+0000","last_unstale":"2024-06-29T15:55:39.199692+0000","last_undegraded":"2024-06-29T15:55:39.199692+0000","last_fullsized":"2024-06-29T15:55:39.199692+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:36:47.130148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195206+0000","last_change":"2024-06-29T15:55:38.188572+0000","last_active":"2024-06-29T15:55:39.195206+0000","last_peered":"2024-06-29T15:55:39.195206+0000","last_clean":"2024-06-29T15:55:39.195206+0000","last_became_active":"2024-06-29T15:55:38.188217+0000","last_became_peered":"2024-06-29T15:55:38.188217+0000","last_unstale":"2024-06-29T15:55:39.195206+0000","last_undegraded":"2024-06-29T15:55:39.195206+0000","last_fullsized":"2024-06-29T15:55:39.195206+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:13.262550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637929+0000","last_change":"2024-06-29T15:55:38.211912+0000","last_active":"2024-06-29T15:55:39.637929+0000","last_peered":"2024-06-29T15:55:39.637929+0000","last_clean":"2024-06-29T15:55:39.637929+0000","last_became_active":"2024-06-29T15:55:38.211599+0000","last_became_peered":"2024-06-29T15:55:38.211599+0000","last_unstale":"2024-06-29T15:55:39.637929+0000","last_undegraded":"2024-06-29T15:55:39.637929+0000","last_fullsized":"2024-06-29T15:55:39.637929+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:50.684269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194970+0000","last_change":"2024-06-29T15:55:38.187621+0000","last_active":"2024-06-29T15:55:39.194970+0000","last_peered":"2024-06-29T15:55:39.194970+0000","last_clean":"2024-06-29T15:55:39.194970+0000","last_became_active":"2024-06-29T15:55:38.187292+0000","last_became_peered":"2024-06-29T15:55:38.187292+0000","last_unstale":"2024-06-29T15:55:39.194970+0000","last_undegraded":"2024-06-29T15:55:39.194970+0000","last_fullsized":"2024-06-29T15:55:39.194970+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:40.238763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194771+0000","last_change":"2024-06-29T15:55:38.197810+0000","last_active":"2024-06-29T15:55:39.194771+0000","last_peered":"2024-06-29T15:55:39.194771+0000","last_clean":"2024-06-29T15:55:39.194771+0000","last_became_active":"2024-06-29T15:55:38.197486+0000","last_became_peered":"2024-06-29T15:55:38.197486+0000","last_unstale":"2024-06-29T15:55:39.194771+0000","last_undegraded":"2024-06-29T15:55:39.194771+0000","last_fullsized":"2024-06-29T15:55:39.194771+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:53.933287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197434+0000","last_change":"2024-06-29T15:55:38.203101+0000","last_active":"2024-06-29T15:55:39.197434+0000","last_peered":"2024-06-29T15:55:39.197434+0000","last_clean":"2024-06-29T15:55:39.197434+0000","last_became_active":"2024-06-29T15:55:38.202763+0000","last_became_peered":"2024-06-29T15:55:38.202763+0000","last_unstale":"2024-06-29T15:55:39.197434+0000","last_undegraded":"2024-06-29T15:55:39.197434+0000","last_fullsized":"2024-06-29T15:55:39.197434+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:21.116209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.206380+0000","last_change":"2024-06-29T15:55:38.203503+0000","last_active":"2024-06-29T15:55:38.206380+0000","last_peered":"2024-06-29T15:55:38.206380+0000","last_clean":"2024-06-29T15:55:38.206380+0000","last_became_active":"2024-06-29T15:55:38.203195+0000","last_became_peered":"2024-06-29T15:55:38.203195+0000","last_unstale":"2024-06-29T15:55:38.206380+0000","last_undegraded":"2024-06-29T15:55:38.206380+0000","last_fullsized":"2024-06-29T15:55:38.206380+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:31:25.943016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199990+0000","last_change":"2024-06-29T15:55:38.204545+0000","last_active":"2024-06-29T15:55:39.199990+0000","last_peered":"2024-06-29T15:55:39.199990+0000","last_clean":"2024-06-29T15:55:39.199990+0000","last_became_active":"2024-06-29T15:55:38.204205+0000","last_became_peered":"2024-06-29T15:55:38.204205+0000","last_unstale":"2024-06-29T15:55:39.199990+0000","last_undegraded":"2024-06-29T15:55:39.199990+0000","last_fullsized":"2024-06-29T15:55:39.199990+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:14.722044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195263+0000","last_change":"2024-06-29T15:55:38.204281+0000","last_active":"2024-06-29T15:55:39.195263+0000","last_peered":"2024-06-29T15:55:39.195263+0000","last_clean":"2024-06-29T15:55:39.195263+0000","last_became_active":"2024-06-29T15:55:38.203924+0000","last_became_peered":"2024-06-29T15:55:38.203924+0000","last_unstale":"2024-06-29T15:55:39.195263+0000","last_undegraded":"2024-06-29T15:55:39.195263+0000","last_fullsized":"2024-06-29T15:55:39.195263+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:31.481915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.207616+0000","last_change":"2024-06-29T15:55:38.201374+0000","last_active":"2024-06-29T15:55:38.207616+0000","last_peered":"2024-06-29T15:55:38.207616+0000","last_clean":"2024-06-29T15:55:38.207616+0000","last_became_active":"2024-06-29T15:55:38.200988+0000","last_became_peered":"2024-06-29T15:55:38.200988+0000","last_unstale":"2024-06-29T15:55:38.207616+0000","last_undegraded":"2024-06-29T15:55:38.207616+0000","last_fullsized":"2024-06-29T15:55:38.207616+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:30:18.805904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638002+0000","last_change":"2024-06-29T15:55:38.204934+0000","last_active":"2024-06-29T15:55:39.638002+0000","last_peered":"2024-06-29T15:55:39.638002+0000","last_clean":"2024-06-29T15:55:39.638002+0000","last_became_active":"2024-06-29T15:55:38.204547+0000","last_became_peered":"2024-06-29T15:55:38.204547+0000","last_unstale":"2024-06-29T15:55:39.638002+0000","last_undegraded":"2024-06-29T15:55:39.638002+0000","last_fullsized":"2024-06-29T15:55:39.638002+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:49.473559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195647+0000","last_change":"2024-06-29T15:55:38.209039+0000","last_active":"2024-06-29T15:55:39.195647+0000","last_peered":"2024-06-29T15:55:39.195647+0000","last_clean":"2024-06-29T15:55:39.195647+0000","last_became_active":"2024-06-29T15:55:38.206849+0000","last_became_peered":"2024-06-29T15:55:38.206849+0000","last_unstale":"2024-06-29T15:55:39.195647+0000","last_undegraded":"2024-06-29T15:55:39.195647+0000","last_fullsized":"2024-06-29T15:55:39.195647+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:10:31.485808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197705+0000","last_change":"2024-06-29T15:55:38.207992+0000","last_active":"2024-06-29T15:55:39.197705+0000","last_peered":"2024-06-29T15:55:39.197705+0000","last_clean":"2024-06-29T15:55:39.197705+0000","last_became_active":"2024-06-29T15:55:38.207669+0000","last_became_peered":"2024-06-29T15:55:38.207669+0000","last_unstale":"2024-06-29T15:55:39.197705+0000","last_undegraded":"2024-06-29T15:55:39.197705+0000","last_fullsized":"2024-06-29T15:55:39.197705+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:38.397012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199532+0000","last_change":"2024-06-29T15:55:38.209811+0000","last_active":"2024-06-29T15:55:39.199532+0000","last_peered":"2024-06-29T15:55:39.199532+0000","last_clean":"2024-06-29T15:55:39.199532+0000","last_became_active":"2024-06-29T15:55:38.209230+0000","last_became_peered":"2024-06-29T15:55:38.209230+0000","last_unstale":"2024-06-29T15:55:39.199532+0000","last_undegraded":"2024-06-29T15:55:39.199532+0000","last_fullsized":"2024-06-29T15:55:39.199532+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:10:09.365017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194645+0000","last_change":"2024-06-29T15:55:38.189168+0000","last_active":"2024-06-29T15:55:39.194645+0000","last_peered":"2024-06-29T15:55:39.194645+0000","last_clean":"2024-06-29T15:55:39.194645+0000","last_became_active":"2024-06-29T15:55:38.188870+0000","last_became_peered":"2024-06-29T15:55:38.188870+0000","last_unstale":"2024-06-29T15:55:39.194645+0000","last_undegraded":"2024-06-29T15:55:39.194645+0000","last_fullsized":"2024-06-29T15:55:39.194645+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:31.038820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195191+0000","last_change":"2024-06-29T15:55:38.210931+0000","last_active":"2024-06-29T15:55:39.195191+0000","last_peered":"2024-06-29T15:55:39.195191+0000","last_clean":"2024-06-29T15:55:39.195191+0000","last_became_active":"2024-06-29T15:55:38.210574+0000","last_became_peered":"2024-06-29T15:55:38.210574+0000","last_unstale":"2024-06-29T15:55:39.195191+0000","last_undegraded":"2024-06-29T15:55:39.195191+0000","last_fullsized":"2024-06-29T15:55:39.195191+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:34.617520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195628+0000","last_change":"2024-06-29T15:55:38.203400+0000","last_active":"2024-06-29T15:55:39.195628+0000","last_peered":"2024-06-29T15:55:39.195628+0000","last_clean":"2024-06-29T15:55:39.195628+0000","last_became_active":"2024-06-29T15:55:38.203071+0000","last_became_peered":"2024-06-29T15:55:38.203071+0000","last_unstale":"2024-06-29T15:55:39.195628+0000","last_undegraded":"2024-06-29T15:55:39.195628+0000","last_fullsized":"2024-06-29T15:55:39.195628+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:22.269697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195386+0000","last_change":"2024-06-29T15:55:38.207190+0000","last_active":"2024-06-29T15:55:39.195386+0000","last_peered":"2024-06-29T15:55:39.195386+0000","last_clean":"2024-06-29T15:55:39.195386+0000","last_became_active":"2024-06-29T15:55:38.206571+0000","last_became_peered":"2024-06-29T15:55:38.206571+0000","last_unstale":"2024-06-29T15:55:39.195386+0000","last_undegraded":"2024-06-29T15:55:39.195386+0000","last_fullsized":"2024-06-29T15:55:39.195386+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:15:26.759219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195010+0000","last_change":"2024-06-29T15:55:38.204209+0000","last_active":"2024-06-29T15:55:39.195010+0000","last_peered":"2024-06-29T15:55:39.195010+0000","last_clean":"2024-06-29T15:55:39.195010+0000","last_became_active":"2024-06-29T15:55:38.203795+0000","last_became_peered":"2024-06-29T15:55:38.203795+0000","last_unstale":"2024-06-29T15:55:39.195010+0000","last_undegraded":"2024-06-29T15:55:39.195010+0000","last_fullsized":"2024-06-29T15:55:39.195010+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:21.677784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199773+0000","last_change":"2024-06-29T15:55:38.202664+0000","last_active":"2024-06-29T15:55:39.199773+0000","last_peered":"2024-06-29T15:55:39.199773+0000","last_clean":"2024-06-29T15:55:39.199773+0000","last_became_active":"2024-06-29T15:55:38.202314+0000","last_became_peered":"2024-06-29T15:55:38.202314+0000","last_unstale":"2024-06-29T15:55:39.199773+0000","last_undegraded":"2024-06-29T15:55:39.199773+0000","last_fullsized":"2024-06-29T15:55:39.199773+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:04.950646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199673+0000","last_change":"2024-06-29T15:55:38.200899+0000","last_active":"2024-06-29T15:55:39.199673+0000","last_peered":"2024-06-29T15:55:39.199673+0000","last_clean":"2024-06-29T15:55:39.199673+0000","last_became_active":"2024-06-29T15:55:38.200529+0000","last_became_peered":"2024-06-29T15:55:38.200529+0000","last_unstale":"2024-06-29T15:55:39.199673+0000","last_undegraded":"2024-06-29T15:55:39.199673+0000","last_fullsized":"2024-06-29T15:55:39.199673+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:06:25.178759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.202600+0000","last_change":"2024-06-29T15:55:38.199732+0000","last_active":"2024-06-29T15:55:38.202600+0000","last_peered":"2024-06-29T15:55:38.202600+0000","last_clean":"2024-06-29T15:55:38.202600+0000","last_became_active":"2024-06-29T15:55:38.199403+0000","last_became_peered":"2024-06-29T15:55:38.199403+0000","last_unstale":"2024-06-29T15:55:38.202600+0000","last_undegraded":"2024-06-29T15:55:38.202600+0000","last_fullsized":"2024-06-29T15:55:38.202600+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:58:06.377622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199337+0000","last_change":"2024-06-29T15:55:38.206962+0000","last_active":"2024-06-29T15:55:39.199337+0000","last_peered":"2024-06-29T15:55:39.199337+0000","last_clean":"2024-06-29T15:55:39.199337+0000","last_became_active":"2024-06-29T15:55:38.206623+0000","last_became_peered":"2024-06-29T15:55:38.206623+0000","last_unstale":"2024-06-29T15:55:39.199337+0000","last_undegraded":"2024-06-29T15:55:39.199337+0000","last_fullsized":"2024-06-29T15:55:39.199337+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:22:27.907778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197873+0000","last_change":"2024-06-29T15:55:38.197882+0000","last_active":"2024-06-29T15:55:39.197873+0000","last_peered":"2024-06-29T15:55:39.197873+0000","last_clean":"2024-06-29T15:55:39.197873+0000","last_became_active":"2024-06-29T15:55:38.197483+0000","last_became_peered":"2024-06-29T15:55:38.197483+0000","last_unstale":"2024-06-29T15:55:39.197873+0000","last_undegraded":"2024-06-29T15:55:39.197873+0000","last_fullsized":"2024-06-29T15:55:39.197873+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:17.255639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.190459+0000","last_change":"2024-06-29T15:55:38.188563+0000","last_active":"2024-06-29T15:55:38.190459+0000","last_peered":"2024-06-29T15:55:38.190459+0000","last_clean":"2024-06-29T15:55:38.190459+0000","last_became_active":"2024-06-29T15:55:38.188248+0000","last_became_peered":"2024-06-29T15:55:38.188248+0000","last_unstale":"2024-06-29T15:55:38.190459+0000","last_undegraded":"2024-06-29T15:55:38.190459+0000","last_fullsized":"2024-06-29T15:55:38.190459+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:59.274484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.202435+0000","last_change":"2024-06-29T15:55:38.195447+0000","last_active":"2024-06-29T15:55:38.202435+0000","last_peered":"2024-06-29T15:55:38.202435+0000","last_clean":"2024-06-29T15:55:38.202435+0000","last_became_active":"2024-06-29T15:55:38.195078+0000","last_became_peered":"2024-06-29T15:55:38.195078+0000","last_unstale":"2024-06-29T15:55:38.202435+0000","last_undegraded":"2024-06-29T15:55:38.202435+0000","last_fullsized":"2024-06-29T15:55:38.202435+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:42:02.030495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195412+0000","last_change":"2024-06-29T15:55:38.190483+0000","last_active":"2024-06-29T15:55:39.195412+0000","last_peered":"2024-06-29T15:55:39.195412+0000","last_clean":"2024-06-29T15:55:39.195412+0000","last_became_active":"2024-06-29T15:55:38.189760+0000","last_became_peered":"2024-06-29T15:55:38.189760+0000","last_unstale":"2024-06-29T15:55:39.195412+0000","last_undegraded":"2024-06-29T15:55:39.195412+0000","last_fullsized":"2024-06-29T15:55:39.195412+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:09.308429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.205781+0000","last_change":"2024-06-29T15:55:38.195132+0000","last_active":"2024-06-29T15:55:38.205781+0000","last_peered":"2024-06-29T15:55:38.205781+0000","last_clean":"2024-06-29T15:55:38.205781+0000","last_became_active":"2024-06-29T15:55:38.194121+0000","last_became_peered":"2024-06-29T15:55:38.194121+0000","last_unstale":"2024-06-29T15:55:38.205781+0000","last_undegraded":"2024-06-29T15:55:38.205781+0000","last_fullsized":"2024-06-29T15:55:38.205781+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:43.447236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.196310+0000","last_change":"2024-06-29T15:55:38.194814+0000","last_active":"2024-06-29T15:55:38.196310+0000","last_peered":"2024-06-29T15:55:38.196310+0000","last_clean":"2024-06-29T15:55:38.196310+0000","last_became_active":"2024-06-29T15:55:38.194517+0000","last_became_peered":"2024-06-29T15:55:38.194517+0000","last_unstale":"2024-06-29T15:55:38.196310+0000","last_undegraded":"2024-06-29T15:55:38.196310+0000","last_fullsized":"2024-06-29T15:55:38.196310+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:11.853597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197416+0000","last_change":"2024-06-29T15:55:38.200729+0000","last_active":"2024-06-29T15:55:39.197416+0000","last_peered":"2024-06-29T15:55:39.197416+0000","last_clean":"2024-06-29T15:55:39.197416+0000","last_became_active":"2024-06-29T15:55:38.200016+0000","last_became_peered":"2024-06-29T15:55:38.200016+0000","last_unstale":"2024-06-29T15:55:39.197416+0000","last_undegraded":"2024-06-29T15:55:39.197416+0000","last_fullsized":"2024-06-29T15:55:39.197416+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:55:54.952820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194709+0000","last_change":"2024-06-29T15:55:38.210694+0000","last_active":"2024-06-29T15:55:39.194709+0000","last_peered":"2024-06-29T15:55:39.194709+0000","last_clean":"2024-06-29T15:55:39.194709+0000","last_became_active":"2024-06-29T15:55:38.210060+0000","last_became_peered":"2024-06-29T15:55:38.210060+0000","last_unstale":"2024-06-29T15:55:39.194709+0000","last_undegraded":"2024-06-29T15:55:39.194709+0000","last_fullsized":"2024-06-29T15:55:39.194709+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:04:49.433010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.206024+0000","last_change":"2024-06-29T15:55:38.202311+0000","last_active":"2024-06-29T15:55:38.206024+0000","last_peered":"2024-06-29T15:55:38.206024+0000","last_clean":"2024-06-29T15:55:38.206024+0000","last_became_active":"2024-06-29T15:55:38.201990+0000","last_became_peered":"2024-06-29T15:55:38.201990+0000","last_unstale":"2024-06-29T15:55:38.206024+0000","last_undegraded":"2024-06-29T15:55:38.206024+0000","last_fullsized":"2024-06-29T15:55:38.206024+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:47.095705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"203'4","reported_seq":18,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638126+0000","last_change":"2024-06-29T15:55:37.193221+0000","last_active":"2024-06-29T15:55:39.638126+0000","last_peered":"2024-06-29T15:55:39.638126+0000","last_clean":"2024-06-29T15:55:39.638126+0000","last_became_active":"2024-06-29T15:55:37.192892+0000","last_became_peered":"2024-06-29T15:55:37.192892+0000","last_unstale":"2024-06-29T15:55:39.638126+0000","last_undegraded":"2024-06-29T15:55:39.638126+0000","last_fullsized":"2024-06-29T15:55:39.638126+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:13.717012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197876+0000","last_change":"2024-06-29T15:55:38.194911+0000","last_active":"2024-06-29T15:55:39.197876+0000","last_peered":"2024-06-29T15:55:39.197876+0000","last_clean":"2024-06-29T15:55:39.197876+0000","last_became_active":"2024-06-29T15:55:38.193559+0000","last_became_peered":"2024-06-29T15:55:38.193559+0000","last_unstale":"2024-06-29T15:55:39.197876+0000","last_undegraded":"2024-06-29T15:55:39.197876+0000","last_fullsized":"2024-06-29T15:55:39.197876+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:54.371370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"203'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195419+0000","last_change":"2024-06-29T15:55:37.179307+0000","last_active":"2024-06-29T15:55:39.195419+0000","last_peered":"2024-06-29T15:55:39.195419+0000","last_clean":"2024-06-29T15:55:39.195419+0000","last_became_active":"2024-06-29T15:55:37.178982+0000","last_became_peered":"2024-06-29T15:55:37.178982+0000","last_unstale":"2024-06-29T15:55:39.195419+0000","last_undegraded":"2024-06-29T15:55:39.195419+0000","last_fullsized":"2024-06-29T15:55:39.195419+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:58:38.335215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.208242+0000","last_change":"2024-06-29T15:55:38.203950+0000","last_active":"2024-06-29T15:55:38.208242+0000","last_peered":"2024-06-29T15:55:38.208242+0000","last_clean":"2024-06-29T15:55:38.208242+0000","last_became_active":"2024-06-29T15:55:38.203618+0000","last_became_peered":"2024-06-29T15:55:38.203618+0000","last_unstale":"2024-06-29T15:55:38.208242+0000","last_undegraded":"2024-06-29T15:55:38.208242+0000","last_fullsized":"2024-06-29T15:55:38.208242+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:57:12.454921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"203'2","reported_seq":14,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.210082+0000","last_change":"2024-06-29T15:55:37.185771+0000","last_active":"2024-06-29T15:55:38.210082+0000","last_peered":"2024-06-29T15:55:38.210082+0000","last_clean":"2024-06-29T15:55:38.210082+0000","last_became_active":"2024-06-29T15:55:37.185459+0000","last_became_peered":"2024-06-29T15:55:37.185459+0000","last_unstale":"2024-06-29T15:55:38.210082+0000","last_undegraded":"2024-06-29T15:55:38.210082+0000","last_fullsized":"2024-06-29T15:55:38.210082+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:08.402451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194990+0000","last_change":"2024-06-29T15:55:38.191279+0000","last_active":"2024-06-29T15:55:39.194990+0000","last_peered":"2024-06-29T15:55:39.194990+0000","last_clean":"2024-06-29T15:55:39.194990+0000","last_became_active":"2024-06-29T15:55:38.190947+0000","last_became_peered":"2024-06-29T15:55:38.190947+0000","last_unstale":"2024-06-29T15:55:39.194990+0000","last_undegraded":"2024-06-29T15:55:39.194990+0000","last_fullsized":"2024-06-29T15:55:39.194990+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:41.092986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195854+0000","last_change":"2024-06-29T15:55:37.188486+0000","last_active":"2024-06-29T15:55:39.195854+0000","last_peered":"2024-06-29T15:55:39.195854+0000","last_clean":"2024-06-29T15:55:39.195854+0000","last_became_active":"2024-06-29T15:55:37.188132+0000","last_became_peered":"2024-06-29T15:55:37.188132+0000","last_unstale":"2024-06-29T15:55:39.195854+0000","last_undegraded":"2024-06-29T15:55:39.195854+0000","last_fullsized":"2024-06-29T15:55:39.195854+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:24:50.835163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194708+0000","last_change":"2024-06-29T15:55:38.205051+0000","last_active":"2024-06-29T15:55:39.194708+0000","last_peered":"2024-06-29T15:55:39.194708+0000","last_clean":"2024-06-29T15:55:39.194708+0000","last_became_active":"2024-06-29T15:55:38.204733+0000","last_became_peered":"2024-06-29T15:55:38.204733+0000","last_unstale":"2024-06-29T15:55:39.194708+0000","last_undegraded":"2024-06-29T15:55:39.194708+0000","last_fullsized":"2024-06-29T15:55:39.194708+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:34:49.084310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"203'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195081+0000","last_change":"2024-06-29T15:55:37.175986+0000","last_active":"2024-06-29T15:55:39.195081+0000","last_peered":"2024-06-29T15:55:39.195081+0000","last_clean":"2024-06-29T15:55:39.195081+0000","last_became_active":"2024-06-29T15:55:37.175630+0000","last_became_peered":"2024-06-29T15:55:37.175630+0000","last_unstale":"2024-06-29T15:55:39.195081+0000","last_undegraded":"2024-06-29T15:55:39.195081+0000","last_fullsized":"2024-06-29T15:55:39.195081+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:29.015055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199828+0000","last_change":"2024-06-29T15:55:38.187783+0000","last_active":"2024-06-29T15:55:39.199828+0000","last_peered":"2024-06-29T15:55:39.199828+0000","last_clean":"2024-06-29T15:55:39.199828+0000","last_became_active":"2024-06-29T15:55:38.187436+0000","last_became_peered":"2024-06-29T15:55:38.187436+0000","last_unstale":"2024-06-29T15:55:39.199828+0000","last_undegraded":"2024-06-29T15:55:39.199828+0000","last_fullsized":"2024-06-29T15:55:39.199828+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:53.341489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"203'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638054+0000","last_change":"2024-06-29T15:55:37.196118+0000","last_active":"2024-06-29T15:55:39.638054+0000","last_peered":"2024-06-29T15:55:39.638054+0000","last_clean":"2024-06-29T15:55:39.638054+0000","last_became_active":"2024-06-29T15:55:37.195732+0000","last_became_peered":"2024-06-29T15:55:37.195732+0000","last_unstale":"2024-06-29T15:55:39.638054+0000","last_undegraded":"2024-06-29T15:55:39.638054+0000","last_fullsized":"2024-06-29T15:55:39.638054+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:22.730506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199864+0000","last_change":"2024-06-29T15:55:38.207856+0000","last_active":"2024-06-29T15:55:39.199864+0000","last_peered":"2024-06-29T15:55:39.199864+0000","last_clean":"2024-06-29T15:55:39.199864+0000","last_became_active":"2024-06-29T15:55:38.207551+0000","last_became_peered":"2024-06-29T15:55:38.207551+0000","last_unstale":"2024-06-29T15:55:39.199864+0000","last_undegraded":"2024-06-29T15:55:39.199864+0000","last_fullsized":"2024-06-29T15:55:39.199864+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:47:20.285236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195328+0000","last_change":"2024-06-29T15:55:37.186599+0000","last_active":"2024-06-29T15:55:39.195328+0000","last_peered":"2024-06-29T15:55:39.195328+0000","last_clean":"2024-06-29T15:55:39.195328+0000","last_became_active":"2024-06-29T15:55:37.186248+0000","last_became_peered":"2024-06-29T15:55:37.186248+0000","last_unstale":"2024-06-29T15:55:39.195328+0000","last_undegraded":"2024-06-29T15:55:39.195328+0000","last_fullsized":"2024-06-29T15:55:39.195328+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:37.582542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.205998+0000","last_change":"2024-06-29T15:55:38.201525+0000","last_active":"2024-06-29T15:55:38.205998+0000","last_peered":"2024-06-29T15:55:38.205998+0000","last_clean":"2024-06-29T15:55:38.205998+0000","last_became_active":"2024-06-29T15:55:38.201218+0000","last_became_peered":"2024-06-29T15:55:38.201218+0000","last_unstale":"2024-06-29T15:55:38.205998+0000","last_undegraded":"2024-06-29T15:55:38.205998+0000","last_fullsized":"2024-06-29T15:55:38.205998+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:14:18.988229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195192+0000","last_change":"2024-06-29T15:55:37.195659+0000","last_active":"2024-06-29T15:55:39.195192+0000","last_peered":"2024-06-29T15:55:39.195192+0000","last_clean":"2024-06-29T15:55:39.195192+0000","last_became_active":"2024-06-29T15:55:37.195097+0000","last_became_peered":"2024-06-29T15:55:37.195097+0000","last_unstale":"2024-06-29T15:55:39.195192+0000","last_undegraded":"2024-06-29T15:55:39.195192+0000","last_fullsized":"2024-06-29T15:55:39.195192+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:59:29.566045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195342+0000","last_change":"2024-06-29T15:55:38.196779+0000","last_active":"2024-06-29T15:55:39.195342+0000","last_peered":"2024-06-29T15:55:39.195342+0000","last_clean":"2024-06-29T15:55:39.195342+0000","last_became_active":"2024-06-29T15:55:38.196420+0000","last_became_peered":"2024-06-29T15:55:38.196420+0000","last_unstale":"2024-06-29T15:55:39.195342+0000","last_undegraded":"2024-06-29T15:55:39.195342+0000","last_fullsized":"2024-06-29T15:55:39.195342+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:55:22.062078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199632+0000","last_change":"2024-06-29T15:55:37.184280+0000","last_active":"2024-06-29T15:55:39.199632+0000","last_peered":"2024-06-29T15:55:39.199632+0000","last_clean":"2024-06-29T15:55:39.199632+0000","last_became_active":"2024-06-29T15:55:37.183967+0000","last_became_peered":"2024-06-29T15:55:37.183967+0000","last_unstale":"2024-06-29T15:55:39.199632+0000","last_undegraded":"2024-06-29T15:55:39.199632+0000","last_fullsized":"2024-06-29T15:55:39.199632+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.078739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195735+0000","last_change":"2024-06-29T15:55:38.200107+0000","last_active":"2024-06-29T15:55:39.195735+0000","last_peered":"2024-06-29T15:55:39.195735+0000","last_clean":"2024-06-29T15:55:39.195735+0000","last_became_active":"2024-06-29T15:55:38.199669+0000","last_became_peered":"2024-06-29T15:55:38.199669+0000","last_unstale":"2024-06-29T15:55:39.195735+0000","last_undegraded":"2024-06-29T15:55:39.195735+0000","last_fullsized":"2024-06-29T15:55:39.195735+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:41.100284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"203'3","reported_seq":17,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637952+0000","last_change":"2024-06-29T15:55:37.197033+0000","last_active":"2024-06-29T15:55:39.637952+0000","last_peered":"2024-06-29T15:55:39.637952+0000","last_clean":"2024-06-29T15:55:39.637952+0000","last_became_active":"2024-06-29T15:55:37.196669+0000","last_became_peered":"2024-06-29T15:55:37.196669+0000","last_unstale":"2024-06-29T15:55:39.637952+0000","last_undegraded":"2024-06-29T15:55:39.637952+0000","last_fullsized":"2024-06-29T15:55:39.637952+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:55.689057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.200002+0000","last_change":"2024-06-29T15:55:38.188546+0000","last_active":"2024-06-29T15:55:39.200002+0000","last_peered":"2024-06-29T15:55:39.200002+0000","last_clean":"2024-06-29T15:55:39.200002+0000","last_became_active":"2024-06-29T15:55:38.188220+0000","last_became_peered":"2024-06-29T15:55:38.188220+0000","last_unstale":"2024-06-29T15:55:39.200002+0000","last_undegraded":"2024-06-29T15:55:39.200002+0000","last_fullsized":"2024-06-29T15:55:39.200002+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:17.474861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"203'6","reported_seq":20,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195988+0000","last_change":"2024-06-29T15:55:37.185494+0000","last_active":"2024-06-29T15:55:39.195988+0000","last_peered":"2024-06-29T15:55:39.195988+0000","last_clean":"2024-06-29T15:55:39.195988+0000","last_became_active":"2024-06-29T15:55:37.184064+0000","last_became_peered":"2024-06-29T15:55:37.184064+0000","last_unstale":"2024-06-29T15:55:39.195988+0000","last_undegraded":"2024-06-29T15:55:39.195988+0000","last_fullsized":"2024-06-29T15:55:39.195988+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:37.797870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197413+0000","last_change":"2024-06-29T15:55:38.194012+0000","last_active":"2024-06-29T15:55:39.197413+0000","last_peered":"2024-06-29T15:55:39.197413+0000","last_clean":"2024-06-29T15:55:39.197413+0000","last_became_active":"2024-06-29T15:55:38.193616+0000","last_became_peered":"2024-06-29T15:55:38.193616+0000","last_unstale":"2024-06-29T15:55:39.197413+0000","last_undegraded":"2024-06-29T15:55:39.197413+0000","last_fullsized":"2024-06-29T15:55:39.197413+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:32.889293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195824+0000","last_change":"2024-06-29T15:55:37.190946+0000","last_active":"2024-06-29T15:55:39.195824+0000","last_peered":"2024-06-29T15:55:39.195824+0000","last_clean":"2024-06-29T15:55:39.195824+0000","last_became_active":"2024-06-29T15:55:37.190639+0000","last_became_peered":"2024-06-29T15:55:37.190639+0000","last_unstale":"2024-06-29T15:55:39.195824+0000","last_undegraded":"2024-06-29T15:55:39.195824+0000","last_fullsized":"2024-06-29T15:55:39.195824+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:11.900910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194708+0000","last_change":"2024-06-29T15:55:38.201962+0000","last_active":"2024-06-29T15:55:39.194708+0000","last_peered":"2024-06-29T15:55:39.194708+0000","last_clean":"2024-06-29T15:55:39.194708+0000","last_became_active":"2024-06-29T15:55:38.201613+0000","last_became_peered":"2024-06-29T15:55:38.201613+0000","last_unstale":"2024-06-29T15:55:39.194708+0000","last_undegraded":"2024-06-29T15:55:39.194708+0000","last_fullsized":"2024-06-29T15:55:39.194708+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:57.852528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.200027+0000","last_change":"2024-06-29T15:55:37.190807+0000","last_active":"2024-06-29T15:55:39.200027+0000","last_peered":"2024-06-29T15:55:39.200027+0000","last_clean":"2024-06-29T15:55:39.200027+0000","last_became_active":"2024-06-29T15:55:37.190477+0000","last_became_peered":"2024-06-29T15:55:37.190477+0000","last_unstale":"2024-06-29T15:55:39.200027+0000","last_undegraded":"2024-06-29T15:55:39.200027+0000","last_fullsized":"2024-06-29T15:55:39.200027+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:06:39.594276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195427+0000","last_change":"2024-06-29T15:55:38.205740+0000","last_active":"2024-06-29T15:55:39.195427+0000","last_peered":"2024-06-29T15:55:39.195427+0000","last_clean":"2024-06-29T15:55:39.195427+0000","last_became_active":"2024-06-29T15:55:38.205406+0000","last_became_peered":"2024-06-29T15:55:38.205406+0000","last_unstale":"2024-06-29T15:55:39.195427+0000","last_undegraded":"2024-06-29T15:55:39.195427+0000","last_fullsized":"2024-06-29T15:55:39.195427+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:57.362943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637711+0000","last_change":"2024-06-29T15:55:37.192976+0000","last_active":"2024-06-29T15:55:39.637711+0000","last_peered":"2024-06-29T15:55:39.637711+0000","last_clean":"2024-06-29T15:55:39.637711+0000","last_became_active":"2024-06-29T15:55:37.192647+0000","last_became_peered":"2024-06-29T15:55:37.192647+0000","last_unstale":"2024-06-29T15:55:39.637711+0000","last_undegraded":"2024-06-29T15:55:39.637711+0000","last_fullsized":"2024-06-29T15:55:39.637711+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:48.150158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195828+0000","last_change":"2024-06-29T15:55:38.211984+0000","last_active":"2024-06-29T15:55:39.195828+0000","last_peered":"2024-06-29T15:55:39.195828+0000","last_clean":"2024-06-29T15:55:39.195828+0000","last_became_active":"2024-06-29T15:55:38.211374+0000","last_became_peered":"2024-06-29T15:55:38.211374+0000","last_unstale":"2024-06-29T15:55:39.195828+0000","last_undegraded":"2024-06-29T15:55:39.195828+0000","last_fullsized":"2024-06-29T15:55:39.195828+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:08:17.996282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"203'8","reported_seq":20,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.214855+0000","last_change":"2024-06-29T15:55:37.187789+0000","last_active":"2024-06-29T15:55:38.214855+0000","last_peered":"2024-06-29T15:55:38.214855+0000","last_clean":"2024-06-29T15:55:38.214855+0000","last_became_active":"2024-06-29T15:55:37.187478+0000","last_became_peered":"2024-06-29T15:55:37.187478+0000","last_unstale":"2024-06-29T15:55:38.214855+0000","last_undegraded":"2024-06-29T15:55:38.214855+0000","last_fullsized":"2024-06-29T15:55:38.214855+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:18.543911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197647+0000","last_change":"2024-06-29T15:55:38.201663+0000","last_active":"2024-06-29T15:55:39.197647+0000","last_peered":"2024-06-29T15:55:39.197647+0000","last_clean":"2024-06-29T15:55:39.197647+0000","last_became_active":"2024-06-29T15:55:38.199999+0000","last_became_peered":"2024-06-29T15:55:38.199999+0000","last_unstale":"2024-06-29T15:55:39.197647+0000","last_undegraded":"2024-06-29T15:55:39.197647+0000","last_fullsized":"2024-06-29T15:55:39.197647+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:19.552281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"203'1","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.213120+0000","last_change":"2024-06-29T15:55:37.190954+0000","last_active":"2024-06-29T15:55:38.213120+0000","last_peered":"2024-06-29T15:55:38.213120+0000","last_clean":"2024-06-29T15:55:38.213120+0000","last_became_active":"2024-06-29T15:55:37.190614+0000","last_became_peered":"2024-06-29T15:55:37.190614+0000","last_unstale":"2024-06-29T15:55:38.213120+0000","last_undegraded":"2024-06-29T15:55:38.213120+0000","last_fullsized":"2024-06-29T15:55:38.213120+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:41:30.536413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195131+0000","last_change":"2024-06-29T15:55:38.209000+0000","last_active":"2024-06-29T15:55:39.195131+0000","last_peered":"2024-06-29T15:55:39.195131+0000","last_clean":"2024-06-29T15:55:39.195131+0000","last_became_active":"2024-06-29T15:55:38.208666+0000","last_became_peered":"2024-06-29T15:55:38.208666+0000","last_unstale":"2024-06-29T15:55:39.195131+0000","last_undegraded":"2024-06-29T15:55:39.195131+0000","last_fullsized":"2024-06-29T15:55:39.195131+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:59.950282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638010+0000","last_change":"2024-06-29T15:55:37.192980+0000","last_active":"2024-06-29T15:55:39.638010+0000","last_peered":"2024-06-29T15:55:39.638010+0000","last_clean":"2024-06-29T15:55:39.638010+0000","last_became_active":"2024-06-29T15:55:37.192601+0000","last_became_peered":"2024-06-29T15:55:37.192601+0000","last_unstale":"2024-06-29T15:55:39.638010+0000","last_undegraded":"2024-06-29T15:55:39.638010+0000","last_fullsized":"2024-06-29T15:55:39.638010+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:52.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.190405+0000","last_change":"2024-06-29T15:55:38.187492+0000","last_active":"2024-06-29T15:55:38.190405+0000","last_peered":"2024-06-29T15:55:38.190405+0000","last_clean":"2024-06-29T15:55:38.190405+0000","last_became_active":"2024-06-29T15:55:38.187160+0000","last_became_peered":"2024-06-29T15:55:38.187160+0000","last_unstale":"2024-06-29T15:55:38.190405+0000","last_undegraded":"2024-06-29T15:55:38.190405+0000","last_fullsized":"2024-06-29T15:55:38.190405+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:46.966818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"203'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195104+0000","last_change":"2024-06-29T15:55:37.181914+0000","last_active":"2024-06-29T15:55:39.195104+0000","last_peered":"2024-06-29T15:55:39.195104+0000","last_clean":"2024-06-29T15:55:39.195104+0000","last_became_active":"2024-06-29T15:55:37.181569+0000","last_became_peered":"2024-06-29T15:55:37.181569+0000","last_unstale":"2024-06-29T15:55:39.195104+0000","last_undegraded":"2024-06-29T15:55:39.195104+0000","last_fullsized":"2024-06-29T15:55:39.195104+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:48.289568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.200119+0000","last_change":"2024-06-29T15:55:38.189467+0000","last_active":"2024-06-29T15:55:39.200119+0000","last_peered":"2024-06-29T15:55:39.200119+0000","last_clean":"2024-06-29T15:55:39.200119+0000","last_became_active":"2024-06-29T15:55:38.189153+0000","last_became_peered":"2024-06-29T15:55:38.189153+0000","last_unstale":"2024-06-29T15:55:39.200119+0000","last_undegraded":"2024-06-29T15:55:39.200119+0000","last_fullsized":"2024-06-29T15:55:39.200119+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:33.643126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.182047+0000","last_change":"2024-06-29T15:55:37.182273+0000","last_active":"2024-06-29T15:55:38.182047+0000","last_peered":"2024-06-29T15:55:38.182047+0000","last_clean":"2024-06-29T15:55:38.182047+0000","last_became_active":"2024-06-29T15:55:37.181918+0000","last_became_peered":"2024-06-29T15:55:37.181918+0000","last_unstale":"2024-06-29T15:55:38.182047+0000","last_undegraded":"2024-06-29T15:55:38.182047+0000","last_fullsized":"2024-06-29T15:55:38.182047+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:42.174671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199302+0000","last_change":"2024-06-29T15:55:38.194184+0000","last_active":"2024-06-29T15:55:39.199302+0000","last_peered":"2024-06-29T15:55:39.199302+0000","last_clean":"2024-06-29T15:55:39.199302+0000","last_became_active":"2024-06-29T15:55:38.193851+0000","last_became_peered":"2024-06-29T15:55:38.193851+0000","last_unstale":"2024-06-29T15:55:39.199302+0000","last_undegraded":"2024-06-29T15:55:39.199302+0000","last_fullsized":"2024-06-29T15:55:39.199302+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:43.333640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195937+0000","last_change":"2024-06-29T15:55:37.193197+0000","last_active":"2024-06-29T15:55:39.195937+0000","last_peered":"2024-06-29T15:55:39.195937+0000","last_clean":"2024-06-29T15:55:39.195937+0000","last_became_active":"2024-06-29T15:55:37.192852+0000","last_became_peered":"2024-06-29T15:55:37.192852+0000","last_unstale":"2024-06-29T15:55:39.195937+0000","last_undegraded":"2024-06-29T15:55:39.195937+0000","last_fullsized":"2024-06-29T15:55:39.195937+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:30.038803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637772+0000","last_change":"2024-06-29T15:55:38.210931+0000","last_active":"2024-06-29T15:55:39.637772+0000","last_peered":"2024-06-29T15:55:39.637772+0000","last_clean":"2024-06-29T15:55:39.637772+0000","last_became_active":"2024-06-29T15:55:38.210611+0000","last_became_peered":"2024-06-29T15:55:38.210611+0000","last_unstale":"2024-06-29T15:55:39.637772+0000","last_undegraded":"2024-06-29T15:55:39.637772+0000","last_fullsized":"2024-06-29T15:55:39.637772+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:12.331030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637857+0000","last_change":"2024-06-29T15:55:37.194538+0000","last_active":"2024-06-29T15:55:39.637857+0000","last_peered":"2024-06-29T15:55:39.637857+0000","last_clean":"2024-06-29T15:55:39.637857+0000","last_became_active":"2024-06-29T15:55:37.193803+0000","last_became_peered":"2024-06-29T15:55:37.193803+0000","last_unstale":"2024-06-29T15:55:39.637857+0000","last_undegraded":"2024-06-29T15:55:39.637857+0000","last_fullsized":"2024-06-29T15:55:39.637857+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:36.758960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637916+0000","last_change":"2024-06-29T15:55:38.200641+0000","last_active":"2024-06-29T15:55:39.637916+0000","last_peered":"2024-06-29T15:55:39.637916+0000","last_clean":"2024-06-29T15:55:39.637916+0000","last_became_active":"2024-06-29T15:55:38.199946+0000","last_became_peered":"2024-06-29T15:55:38.199946+0000","last_unstale":"2024-06-29T15:55:39.637916+0000","last_undegraded":"2024-06-29T15:55:39.637916+0000","last_fullsized":"2024-06-29T15:55:39.637916+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:43.997129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197599+0000","last_change":"2024-06-29T15:55:37.187075+0000","last_active":"2024-06-29T15:55:39.197599+0000","last_peered":"2024-06-29T15:55:39.197599+0000","last_clean":"2024-06-29T15:55:39.197599+0000","last_became_active":"2024-06-29T15:55:37.186757+0000","last_became_peered":"2024-06-29T15:55:37.186757+0000","last_unstale":"2024-06-29T15:55:39.197599+0000","last_undegraded":"2024-06-29T15:55:39.197599+0000","last_fullsized":"2024-06-29T15:55:39.197599+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:19:56.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195475+0000","last_change":"2024-06-29T15:55:38.194353+0000","last_active":"2024-06-29T15:55:39.195475+0000","last_peered":"2024-06-29T15:55:39.195475+0000","last_clean":"2024-06-29T15:55:39.195475+0000","last_became_active":"2024-06-29T15:55:38.194007+0000","last_became_peered":"2024-06-29T15:55:38.194007+0000","last_unstale":"2024-06-29T15:55:39.195475+0000","last_undegraded":"2024-06-29T15:55:39.195475+0000","last_fullsized":"2024-06-29T15:55:39.195475+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:37.766463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194785+0000","last_change":"2024-06-29T15:55:37.183897+0000","last_active":"2024-06-29T15:55:39.194785+0000","last_peered":"2024-06-29T15:55:39.194785+0000","last_clean":"2024-06-29T15:55:39.194785+0000","last_became_active":"2024-06-29T15:55:37.183572+0000","last_became_peered":"2024-06-29T15:55:37.183572+0000","last_unstale":"2024-06-29T15:55:39.194785+0000","last_undegraded":"2024-06-29T15:55:39.194785+0000","last_fullsized":"2024-06-29T15:55:39.194785+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:30.660501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197868+0000","last_change":"2024-06-29T15:55:38.189708+0000","last_active":"2024-06-29T15:55:39.197868+0000","last_peered":"2024-06-29T15:55:39.197868+0000","last_clean":"2024-06-29T15:55:39.197868+0000","last_became_active":"2024-06-29T15:55:38.189342+0000","last_became_peered":"2024-06-29T15:55:38.189342+0000","last_unstale":"2024-06-29T15:55:39.197868+0000","last_undegraded":"2024-06-29T15:55:39.197868+0000","last_fullsized":"2024-06-29T15:55:39.197868+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:01.403464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"203'2","reported_seq":16,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.638108+0000","last_change":"2024-06-29T15:55:37.191777+0000","last_active":"2024-06-29T15:55:39.638108+0000","last_peered":"2024-06-29T15:55:39.638108+0000","last_clean":"2024-06-29T15:55:39.638108+0000","last_became_active":"2024-06-29T15:55:37.191386+0000","last_became_peered":"2024-06-29T15:55:37.191386+0000","last_unstale":"2024-06-29T15:55:39.638108+0000","last_undegraded":"2024-06-29T15:55:39.638108+0000","last_fullsized":"2024-06-29T15:55:39.638108+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:16.469829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199919+0000","last_change":"2024-06-29T15:55:38.188019+0000","last_active":"2024-06-29T15:55:39.199919+0000","last_peered":"2024-06-29T15:55:39.199919+0000","last_clean":"2024-06-29T15:55:39.199919+0000","last_became_active":"2024-06-29T15:55:38.187517+0000","last_became_peered":"2024-06-29T15:55:38.187517+0000","last_unstale":"2024-06-29T15:55:39.199919+0000","last_undegraded":"2024-06-29T15:55:39.199919+0000","last_fullsized":"2024-06-29T15:55:39.199919+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:49.155189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"204'1","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.198954+0000","last_change":"2024-06-29T15:55:37.191495+0000","last_active":"2024-06-29T15:55:39.198954+0000","last_peered":"2024-06-29T15:55:39.198954+0000","last_clean":"2024-06-29T15:55:39.198954+0000","last_became_active":"2024-06-29T15:55:37.191157+0000","last_became_peered":"2024-06-29T15:55:37.191157+0000","last_unstale":"2024-06-29T15:55:39.198954+0000","last_undegraded":"2024-06-29T15:55:39.198954+0000","last_fullsized":"2024-06-29T15:55:39.198954+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:23:30.328659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.206473+0000","last_change":"2024-06-29T15:55:38.202742+0000","last_active":"2024-06-29T15:55:38.206473+0000","last_peered":"2024-06-29T15:55:38.206473+0000","last_clean":"2024-06-29T15:55:38.206473+0000","last_became_active":"2024-06-29T15:55:38.202447+0000","last_became_peered":"2024-06-29T15:55:38.202447+0000","last_unstale":"2024-06-29T15:55:38.206473+0000","last_undegraded":"2024-06-29T15:55:38.206473+0000","last_fullsized":"2024-06-29T15:55:38.206473+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:29.245610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195329+0000","last_change":"2024-06-29T15:55:37.179052+0000","last_active":"2024-06-29T15:55:39.195329+0000","last_peered":"2024-06-29T15:55:39.195329+0000","last_clean":"2024-06-29T15:55:39.195329+0000","last_became_active":"2024-06-29T15:55:37.178459+0000","last_became_peered":"2024-06-29T15:55:37.178459+0000","last_unstale":"2024-06-29T15:55:39.195329+0000","last_undegraded":"2024-06-29T15:55:39.195329+0000","last_fullsized":"2024-06-29T15:55:39.195329+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:15:26.721841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197935+0000","last_change":"2024-06-29T15:55:38.190604+0000","last_active":"2024-06-29T15:55:39.197935+0000","last_peered":"2024-06-29T15:55:39.197935+0000","last_clean":"2024-06-29T15:55:39.197935+0000","last_became_active":"2024-06-29T15:55:38.190196+0000","last_became_peered":"2024-06-29T15:55:38.190196+0000","last_unstale":"2024-06-29T15:55:39.197935+0000","last_undegraded":"2024-06-29T15:55:39.197935+0000","last_fullsized":"2024-06-29T15:55:39.197935+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:06:14.283587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195165+0000","last_change":"2024-06-29T15:55:37.188647+0000","last_active":"2024-06-29T15:55:39.195165+0000","last_peered":"2024-06-29T15:55:39.195165+0000","last_clean":"2024-06-29T15:55:39.195165+0000","last_became_active":"2024-06-29T15:55:37.188330+0000","last_became_peered":"2024-06-29T15:55:37.188330+0000","last_unstale":"2024-06-29T15:55:39.195165+0000","last_undegraded":"2024-06-29T15:55:39.195165+0000","last_fullsized":"2024-06-29T15:55:39.195165+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:33:04.734367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194644+0000","last_change":"2024-06-29T15:55:38.197753+0000","last_active":"2024-06-29T15:55:39.194644+0000","last_peered":"2024-06-29T15:55:39.194644+0000","last_clean":"2024-06-29T15:55:39.194644+0000","last_became_active":"2024-06-29T15:55:38.197174+0000","last_became_peered":"2024-06-29T15:55:38.197174+0000","last_unstale":"2024-06-29T15:55:39.194644+0000","last_undegraded":"2024-06-29T15:55:39.194644+0000","last_fullsized":"2024-06-29T15:55:39.194644+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:11.837664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.189590+0000","last_change":"2024-06-29T15:55:37.184847+0000","last_active":"2024-06-29T15:55:38.189590+0000","last_peered":"2024-06-29T15:55:38.189590+0000","last_clean":"2024-06-29T15:55:38.189590+0000","last_became_active":"2024-06-29T15:55:37.184537+0000","last_became_peered":"2024-06-29T15:55:37.184537+0000","last_unstale":"2024-06-29T15:55:38.189590+0000","last_undegraded":"2024-06-29T15:55:38.189590+0000","last_fullsized":"2024-06-29T15:55:38.189590+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:48.149496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195229+0000","last_change":"2024-06-29T15:55:38.206586+0000","last_active":"2024-06-29T15:55:39.195229+0000","last_peered":"2024-06-29T15:55:39.195229+0000","last_clean":"2024-06-29T15:55:39.195229+0000","last_became_active":"2024-06-29T15:55:38.206268+0000","last_became_peered":"2024-06-29T15:55:38.206268+0000","last_unstale":"2024-06-29T15:55:39.195229+0000","last_undegraded":"2024-06-29T15:55:39.195229+0000","last_fullsized":"2024-06-29T15:55:39.195229+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:00.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.189134+0000","last_change":"2024-06-29T15:55:37.193390+0000","last_active":"2024-06-29T15:55:38.189134+0000","last_peered":"2024-06-29T15:55:38.189134+0000","last_clean":"2024-06-29T15:55:38.189134+0000","last_became_active":"2024-06-29T15:55:37.193050+0000","last_became_peered":"2024-06-29T15:55:37.193050+0000","last_unstale":"2024-06-29T15:55:38.189134+0000","last_undegraded":"2024-06-29T15:55:38.189134+0000","last_fullsized":"2024-06-29T15:55:38.189134+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:46.897497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.199998+0000","last_change":"2024-06-29T15:55:38.192777+0000","last_active":"2024-06-29T15:55:39.199998+0000","last_peered":"2024-06-29T15:55:39.199998+0000","last_clean":"2024-06-29T15:55:39.199998+0000","last_became_active":"2024-06-29T15:55:38.192434+0000","last_became_peered":"2024-06-29T15:55:38.192434+0000","last_unstale":"2024-06-29T15:55:39.199998+0000","last_undegraded":"2024-06-29T15:55:39.199998+0000","last_fullsized":"2024-06-29T15:55:39.199998+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:53.257489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.197774+0000","last_change":"2024-06-29T15:55:37.186287+0000","last_active":"2024-06-29T15:55:39.197774+0000","last_peered":"2024-06-29T15:55:39.197774+0000","last_clean":"2024-06-29T15:55:39.197774+0000","last_became_active":"2024-06-29T15:55:37.185949+0000","last_became_peered":"2024-06-29T15:55:37.185949+0000","last_unstale":"2024-06-29T15:55:39.197774+0000","last_undegraded":"2024-06-29T15:55:39.197774+0000","last_fullsized":"2024-06-29T15:55:39.197774+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:01.095749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.205627+0000","last_change":"2024-06-29T15:55:38.200595+0000","last_active":"2024-06-29T15:55:38.205627+0000","last_peered":"2024-06-29T15:55:38.205627+0000","last_clean":"2024-06-29T15:55:38.205627+0000","last_became_active":"2024-06-29T15:55:38.200290+0000","last_became_peered":"2024-06-29T15:55:38.200290+0000","last_unstale":"2024-06-29T15:55:38.205627+0000","last_undegraded":"2024-06-29T15:55:38.205627+0000","last_fullsized":"2024-06-29T15:55:38.205627+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:17:24.117324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.194657+0000","last_change":"2024-06-29T15:55:37.177567+0000","last_active":"2024-06-29T15:55:39.194657+0000","last_peered":"2024-06-29T15:55:39.194657+0000","last_clean":"2024-06-29T15:55:39.194657+0000","last_became_active":"2024-06-29T15:55:37.177218+0000","last_became_peered":"2024-06-29T15:55:37.177218+0000","last_unstale":"2024-06-29T15:55:39.194657+0000","last_undegraded":"2024-06-29T15:55:39.194657+0000","last_fullsized":"2024-06-29T15:55:39.194657+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:18.225574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.195120+0000","last_change":"2024-06-29T15:55:38.201348+0000","last_active":"2024-06-29T15:55:39.195120+0000","last_peered":"2024-06-29T15:55:39.195120+0000","last_clean":"2024-06-29T15:55:39.195120+0000","last_became_active":"2024-06-29T15:55:38.201045+0000","last_became_peered":"2024-06-29T15:55:38.201045+0000","last_unstale":"2024-06-29T15:55:39.195120+0000","last_undegraded":"2024-06-29T15:55:39.195120+0000","last_fullsized":"2024-06-29T15:55:39.195120+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.162688+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.162688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:15.600066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.637731+0000","last_change":"2024-06-29T15:55:37.187445+0000","last_active":"2024-06-29T15:55:39.637731+0000","last_peered":"2024-06-29T15:55:39.637731+0000","last_clean":"2024-06-29T15:55:39.637731+0000","last_became_active":"2024-06-29T15:55:37.186656+0000","last_became_peered":"2024-06-29T15:55:37.186656+0000","last_unstale":"2024-06-29T15:55:39.637731+0000","last_undegraded":"2024-06-29T15:55:39.637731+0000","last_fullsized":"2024-06-29T15:55:39.637731+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.157656+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.157656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:32.638095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"193'219","reported_seq":1016,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.189590+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:55:38.189590+0000","last_peered":"2024-06-29T15:55:38.189590+0000","last_clean":"2024-06-29T15:55:38.189590+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:55:38.189590+0000","last_undegraded":"2024-06-29T15:55:38.189590+0000","last_fullsized":"2024-06-29T15:55:38.189590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543267,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52296,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319544,"statfs":{"total":96636764160,"available":96583213056,"internally_reserved":0,"allocated":3133440,"data_stored":1890995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41996,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329844,"statfs":{"total":96636764160,"available":96593760256,"internally_reserved":0,"allocated":2555904,"data_stored":1307445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543267,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47568,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324272,"statfs":{"total":96636764160,"available":96588054528,"internally_reserved":0,"allocated":2555904,"data_stored":1302261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47584,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324256,"statfs":{"total":96636764160,"available":96588038144,"internally_reserved":0,"allocated":2568192,"data_stored":1309021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47572,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324268,"statfs":{"total":96636764160,"available":96588050432,"internally_reserved":0,"allocated":2560000,"data_stored":1307535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575972,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47560,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324280,"statfs":{"total":96636764160,"available":96588062720,"internally_reserved":0,"allocated":2543616,"data_stored":1301209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510563,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47548,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324292,"statfs":{"total":96636764160,"available":96588075008,"internally_reserved":0,"allocated":2539520,"data_stored":1300627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510565,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42580,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329260,"statfs":{"total":96636764160,"available":96593162240,"internally_reserved":0,"allocated":3149824,"data_stored":1891723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:55:42.649 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:55:42.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:55:42.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:42.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.811+0000 7fb356b7a640 1 -- 172.21.15.59:0/914058151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c004000 msgr2=0x7fb33c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.811+0000 7fb356b7a640 1 --2- 172.21.15.59:0/914058151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c004000 0x7fb33c004800 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fb340006fd0 tx=0x7fb33c005830 comp rx=0 tx=0).stop 2024-06-29T15:55:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.811+0000 7fb355117640 1 -- 172.21.15.59:0/914058151 reap_dead start 2024-06-29T15:55:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.811+0000 7fb356b7a640 1 -- 172.21.15.59:0/914058151 shutdown_connections 2024-06-29T15:55:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.811+0000 7fb356b7a640 1 -- 172.21.15.59:0/914058151 >> 172.21.15.59:0/914058151 conn(0x7fb350072cb0 msgr2=0x7fb3500730c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.811+0000 7fb356b7a640 1 -- 172.21.15.59:0/914058151 shutdown_connections 2024-06-29T15:55:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.811+0000 7fb356b7a640 1 -- 172.21.15.59:0/914058151 wait complete. 2024-06-29T15:55:42.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.813+0000 7fb356b7a640 1 Processor -- start 2024-06-29T15:55:42.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.813+0000 7fb356b7a640 1 -- start start 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb356b7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb33c004000 0x7fb35015a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb356b7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35013fd90 0x7fb3500ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb356b7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb35015aca0 0x7fb3500aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb356b7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb35015f740 con 0x7fb35015aca0 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb356b7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb35015f8c0 con 0x7fb35013fd90 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb356b7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb35015fa40 con 0x7fb33c004000 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb355117640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb35015aca0 0x7fb3500aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb355117640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb35015aca0 0x7fb3500aeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45816/0 (socket says 172.21.15.59:45816) 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb355117640 1 -- 172.21.15.59:0/738828251 learned_addr learned my addr 172.21.15.59:0/738828251 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:42.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb354916640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb33c004000 0x7fb35015a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.814+0000 7fb34ffff640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35013fd90 0x7fb3500ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb355117640 1 -- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb33c004000 msgr2=0x7fb35015a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb355117640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb33c004000 0x7fb35015a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb355117640 1 -- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35013fd90 msgr2=0x7fb3500ae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb355117640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35013fd90 0x7fb3500ae580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb355117640 1 -- 172.21.15.59:0/738828251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3500ae140 con 0x7fb35015aca0 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb355117640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb35015aca0 0x7fb3500aeac0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fb3500b5e00 tx=0x7fb348011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb34d7fa640 1 -- 172.21.15.59:0/738828251 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348004230 con 0x7fb35015aca0 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb34d7fa640 1 -- 172.21.15.59:0/738828251 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb348016070 con 0x7fb35015aca0 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb350140f00 con 0x7fb35015aca0 2024-06-29T15:55:42.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb34d7fa640 1 -- 172.21.15.59:0/738828251 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34800fac0 con 0x7fb35015aca0 2024-06-29T15:55:42.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.815+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3501414c0 con 0x7fb35015aca0 2024-06-29T15:55:42.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.816+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35006ffd0 con 0x7fb35015aca0 2024-06-29T15:55:42.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.817+0000 7fb34d7fa640 1 -- 172.21.15.59:0/738828251 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fb34800fc60 con 0x7fb35015aca0 2024-06-29T15:55:42.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.818+0000 7fb34d7fa640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb3300763f0 0x7fb330078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.818+0000 7fb34d7fa640 1 -- 172.21.15.59:0/738828251 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7fb348098450 con 0x7fb35015aca0 2024-06-29T15:55:42.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.818+0000 7fb354916640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb3300763f0 0x7fb330078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.818+0000 7fb354916640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb3300763f0 0x7fb330078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb338005eb0 tx=0x7fb338005f90 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:42.822+0000 7fb34d7fa640 1 -- 172.21.15.59:0/738828251 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb348061400 con 0x7fb35015aca0 2024-06-29T15:55:43.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.088+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb3500ce4f0 con 0x7fb35015aca0 2024-06-29T15:55:43.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.090+0000 7fb34d7fa640 1 -- 172.21.15.59:0/738828251 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) ==== 190+0+147 (secure 0 0 0) 0x7fb348065700 con 0x7fb35015aca0 2024-06-29T15:55:43.091 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T15:55:43.092 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T15:55:43.092 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T15:55:43.092 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T15:55:43.092 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:55:43.092 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.092+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb3300763f0 msgr2=0x7fb330078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.092+0000 7fb356b7a640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb3300763f0 0x7fb330078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb338005eb0 tx=0x7fb338005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.092+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb35015aca0 msgr2=0x7fb3500aeac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.092+0000 7fb356b7a640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb35015aca0 0x7fb3500aeac0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fb3500b5e00 tx=0x7fb348011b10 comp rx=0 tx=0).stop 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.093+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 shutdown_connections 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.093+0000 7fb356b7a640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fb3300763f0 0x7fb330078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.093+0000 7fb356b7a640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb35015aca0 0x7fb3500aeac0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.093+0000 7fb356b7a640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb35013fd90 0x7fb3500ae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.093+0000 7fb356b7a640 1 --2- 172.21.15.59:0/738828251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb33c004000 0x7fb35015a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.093+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 >> 172.21.15.59:0/738828251 conn(0x7fb350072cb0 msgr2=0x7fb3500b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:43.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.093+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 shutdown_connections 2024-06-29T15:55:43.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.093+0000 7fb356b7a640 1 -- 172.21.15.59:0/738828251 wait complete. 2024-06-29T15:55:43.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.223+0000 7f193fa47640 1 -- 172.21.15.59:0/3241784783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19200053a0 msgr2=0x7f192000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.223+0000 7f193fa47640 1 --2- 172.21.15.59:0/3241784783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19200053a0 0x7f192000ba20 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f1930001690 tx=0x7f1920005830 comp rx=0 tx=0).stop 2024-06-29T15:55:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.223+0000 7f193fa47640 1 -- 172.21.15.59:0/3241784783 shutdown_connections 2024-06-29T15:55:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.223+0000 7f193fa47640 1 --2- 172.21.15.59:0/3241784783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19200053a0 0x7f192000ba20 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.223+0000 7f193fa47640 1 -- 172.21.15.59:0/3241784783 >> 172.21.15.59:0/3241784783 conn(0x7f19380d45b0 msgr2=0x7f19380d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.223+0000 7f193fa47640 1 -- 172.21.15.59:0/3241784783 shutdown_connections 2024-06-29T15:55:43.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.223+0000 7f193fa47640 1 -- 172.21.15.59:0/3241784783 wait complete. 2024-06-29T15:55:43.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.225+0000 7f193fa47640 1 Processor -- start 2024-06-29T15:55:43.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.225+0000 7f193fa47640 1 -- start start 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193fa47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19200053a0 0x7f19380c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193fa47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f193815ae70 0x7f193806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193fa47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f193806cf90 0x7f193815f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193fa47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1938161a80 con 0x7f193815ae70 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193fa47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f19380ce820 con 0x7f19200053a0 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193fa47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1938161c00 con 0x7f193806cf90 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193cfe2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f193815ae70 0x7f193806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193cfe2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f193815ae70 0x7f193806ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45840/0 (socket says 172.21.15.59:45840) 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193cfe2640 1 -- 172.21.15.59:0/2531571642 learned_addr learned my addr 172.21.15.59:0/2531571642 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193dfe4640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f193806cf90 0x7f193815f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193d7e3640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19200053a0 0x7f19380c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:43.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193cfe2640 1 -- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f193806cf90 msgr2=0x7f193815f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193cfe2640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f193806cf90 0x7f193815f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193cfe2640 1 -- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19200053a0 msgr2=0x7f19380c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193cfe2640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19200053a0 0x7f19380c7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.226+0000 7f193cfe2640 1 -- 172.21.15.59:0/2531571642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193806d3a0 con 0x7f193815ae70 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.227+0000 7f193cfe2640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f193815ae70 0x7f193806ca50 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f192c00d920 tx=0x7f192c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.227+0000 7f1935ffb640 1 -- 172.21.15.59:0/2531571642 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c00b7d0 con 0x7f193815ae70 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.227+0000 7f1935ffb640 1 -- 172.21.15.59:0/2531571642 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f192c00ba60 con 0x7f193815ae70 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.227+0000 7f1935ffb640 1 -- 172.21.15.59:0/2531571642 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c003610 con 0x7f193815ae70 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.227+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19380c93d0 con 0x7f193815ae70 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.227+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19380c9990 con 0x7f193815ae70 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.228+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1938162160 con 0x7f193815ae70 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.229+0000 7f1935ffb640 1 -- 172.21.15.59:0/2531571642 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f192c0037b0 con 0x7f193815ae70 2024-06-29T15:55:43.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.229+0000 7f1935ffb640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f19240763f0 0x7f1924078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:43.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.229+0000 7f1935ffb640 1 -- 172.21.15.59:0/2531571642 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f192c098170 con 0x7f193815ae70 2024-06-29T15:55:43.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.230+0000 7f193d7e3640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f19240763f0 0x7f1924078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:43.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.230+0000 7f193d7e3640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f19240763f0 0x7f1924078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1928005fd0 tx=0x7f1928009500 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:43.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.234+0000 7f1935ffb640 1 -- 172.21.15.59:0/2531571642 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f193806d3a0 con 0x7f193815ae70 2024-06-29T15:55:43.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.465+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1938068590 con 0x7f193815ae70 2024-06-29T15:55:43.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.466+0000 7f1935ffb640 1 -- 172.21.15.59:0/2531571642 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5246 (secure 0 0 0) 0x7f192c061770 con 0x7f193815ae70 2024-06-29T15:55:43.467 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:43.467 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"btime":"2024-06-29T15:55:41:198021+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3297476433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3297476433},{"type":"v1","addr":"172.21.15.153:6837","nonce":3297476433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":10974,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3987613682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3987613682},{"type":"v1","addr":"172.21.15.59:6834","nonce":3987613682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":10977,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2501188073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2501188073},{"type":"v1","addr":"172.21.15.59:6835","nonce":2501188073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.567350+0000","modified":"2024-06-29T15:55:40.592165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10983},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10983":{"gid":10983,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2163190416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2163190416},{"type":"v1","addr":"172.21.15.153:6836","nonce":2163190416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10983,"qdb_cluster":[10983]},"id":12}]} 2024-06-29T15:55:43.467 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f19240763f0 msgr2=0x7f1924078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f19240763f0 0x7f1924078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1928005fd0 tx=0x7f1928009500 comp rx=0 tx=0).stop 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f193815ae70 msgr2=0x7f193806ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f193815ae70 0x7f193806ca50 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f192c00d920 tx=0x7f192c00dde0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 shutdown_connections 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f19240763f0 0x7f1924078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f193806cf90 0x7f193815f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f193815ae70 0x7f193806ca50 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 --2- 172.21.15.59:0/2531571642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19200053a0 0x7f19380c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 >> 172.21.15.59:0/2531571642 conn(0x7f19380d45b0 msgr2=0x7f19380b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:43.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 shutdown_connections 2024-06-29T15:55:43.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:43.468+0000 7f193fa47640 1 -- 172.21.15.59:0/2531571642 wait complete. 2024-06-29T15:55:43.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:55:37.567350+0000', 'modified': '2024-06-29T15:55:40.592165+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10983}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10983': {'gid': 10983, 'name': 'd', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6836/2163190416', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 2163190416}, {'type': 'v1', 'addr': '172.21.15.153:6836', 'nonce': 2163190416}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10983, 'qdb_cluster': [10983]} 2024-06-29T15:55:43.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:55:43.883 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-06-29T15:55:44.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.005+0000 7f5087688640 1 -- 172.21.15.59:0/2155781674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50680051a0 msgr2=0x7f506800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.005+0000 7f5087688640 1 --2- 172.21.15.59:0/2155781674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50680051a0 0x7f506800b9c0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f5070006fd0 tx=0x7f5068005830 comp rx=0 tx=0).stop 2024-06-29T15:55:44.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.005+0000 7f5085c25640 1 -- 172.21.15.59:0/2155781674 reap_dead start 2024-06-29T15:55:44.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.005+0000 7f5087688640 1 -- 172.21.15.59:0/2155781674 shutdown_connections 2024-06-29T15:55:44.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.005+0000 7f5087688640 1 -- 172.21.15.59:0/2155781674 >> 172.21.15.59:0/2155781674 conn(0x7f50800b06e0 msgr2=0x7f50800b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.005+0000 7f5087688640 1 -- 172.21.15.59:0/2155781674 shutdown_connections 2024-06-29T15:55:44.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.005+0000 7f5087688640 1 -- 172.21.15.59:0/2155781674 wait complete. 2024-06-29T15:55:44.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.007+0000 7f5087688640 1 Processor -- start 2024-06-29T15:55:44.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.007+0000 7f5087688640 1 -- start start 2024-06-29T15:55:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5087688640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50800c97e0 0x7f50800ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5087688640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50800cf1f0 0x7f50801f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5087688640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50801f5830 0x7f50801f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5087688640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5080144fd0 con 0x7f50800c97e0 2024-06-29T15:55:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5087688640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5080144e50 con 0x7f50801f5830 2024-06-29T15:55:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5087688640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5080145150 con 0x7f50800cf1f0 2024-06-29T15:55:44.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085424640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50800c97e0 0x7f50800ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085424640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50800c97e0 0x7f50800ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45858/0 (socket says 172.21.15.59:45858) 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085424640 1 -- 172.21.15.59:0/1995555686 learned_addr learned my addr 172.21.15.59:0/1995555686 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5084c23640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50800cf1f0 0x7f50801f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085c25640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50801f5830 0x7f50801f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085424640 1 -- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50800cf1f0 msgr2=0x7f50801f52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085424640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50800cf1f0 0x7f50801f52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085424640 1 -- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50801f5830 msgr2=0x7f50801f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085424640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50801f5830 0x7f50801f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.008+0000 7f5085424640 1 -- 172.21.15.59:0/1995555686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50800c8740 con 0x7f50800c97e0 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.009+0000 7f5085424640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50800c97e0 0x7f50800ca680 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f5078011ba0 tx=0x7f507800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.009+0000 7f507dffb640 1 -- 172.21.15.59:0/1995555686 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507801e070 con 0x7f50800c97e0 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.009+0000 7f507dffb640 1 -- 172.21.15.59:0/1995555686 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f507800e7c0 con 0x7f50800c97e0 2024-06-29T15:55:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.009+0000 7f5087688640 1 -- 172.21.15.59:0/1995555686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50800c8a20 con 0x7f50800c97e0 2024-06-29T15:55:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.009+0000 7f507dffb640 1 -- 172.21.15.59:0/1995555686 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507800ebd0 con 0x7f50800c97e0 2024-06-29T15:55:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.009+0000 7f5087688640 1 -- 172.21.15.59:0/1995555686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50800c8fe0 con 0x7f50800c97e0 2024-06-29T15:55:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.011+0000 7f507dffb640 1 -- 172.21.15.59:0/1995555686 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f5078021070 con 0x7f50800c97e0 2024-06-29T15:55:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.011+0000 7f507dffb640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f506c0763f0 0x7f506c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.011+0000 7f507dffb640 1 -- 172.21.15.59:0/1995555686 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f5078098aa0 con 0x7f50800c97e0 2024-06-29T15:55:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.011+0000 7f5084c23640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f506c0763f0 0x7f506c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.012+0000 7f5084c23640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f506c0763f0 0x7f506c078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5080161b10 tx=0x7f5074005f90 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.012+0000 7f508e61d740 1 Processor -- start 2024-06-29T15:55:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.012+0000 7f508e61d740 1 -- start start 2024-06-29T15:55:44.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.013+0000 7f508e61d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cda16270 0x5597cde026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.013+0000 7f508e61d740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 0x5597cdde63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.013+0000 7f508e61d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cdde68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.013+0000 7f508e61d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5597cdd7ebc0 con 0x5597cd9f5590 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.013+0000 7f508e61d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5597cdd2d630 con 0x5597cda16270 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.013+0000 7f508e61d740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5597cdd1cbd0 con 0x5597cd9ed3b0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cdde68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cdde68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45862/0 (socket says 172.21.15.59:45862) 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 -- 172.21.15.59:0/2007828824 learned_addr learned my addr 172.21.15.59:0/2007828824 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085424640 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cda16270 0x5597cde026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5084c23640 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 0x5597cdde63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 -- 172.21.15.59:0/2007828824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 msgr2=0x5597cdde63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 0x5597cdde63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 -- 172.21.15.59:0/2007828824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cda16270 msgr2=0x5597cde026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cda16270 0x5597cde026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 -- 172.21.15.59:0/2007828824 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597cdd19ca0 con 0x5597cd9f5590 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5085c25640 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cdde68e0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f5070030040 tx=0x7f508015fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64151fa8bd500f1c server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5066ffd640 1 -- 172.21.15.59:0/2007828824 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507000c360 con 0x5597cd9f5590 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5066ffd640 1 -- 172.21.15.59:0/2007828824 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5070001980 con 0x5597cd9f5590 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.014+0000 7f5066ffd640 1 -- 172.21.15.59:0/2007828824 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070004b40 con 0x5597cd9f5590 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 -- 172.21.15.59:0/2007828824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 msgr2=0x5597cdde68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cdde68e0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f5070030040 tx=0x7f508015fe00 comp rx=0 tx=0).stop 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 -- 172.21.15.59:0/2007828824 shutdown_connections 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cdde68e0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 0x5597cdde63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 --2- 172.21.15.59:0/2007828824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cda16270 0x5597cde026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 -- 172.21.15.59:0/2007828824 >> 172.21.15.59:0/2007828824 conn(0x5597cda0a3c0 msgr2=0x5597cddb9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 -- 172.21.15.59:0/2007828824 shutdown_connections 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.015+0000 7f508e61d740 1 -- 172.21.15.59:0/2007828824 wait complete. 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.016+0000 7f508e61d740 1 -- start start 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.016+0000 7f508e61d740 1 Processor -- start 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f508e61d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9ed3b0 0x5597cde00880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f508e61d740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde01e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f508e61d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cda16270 0x5597cddd4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f508e61d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5597cdcbaae0 con 0x5597cda16270 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f508e61d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5597cdd20c10 con 0x5597cd9ed3b0 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f508e61d740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5597cda90620 con 0x5597cd9f5590 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f508e61d740 10 client.0 _ll_register_callbacks cb 0x5597cdcc7480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cda16270 0x5597cddd4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cda16270 0x5597cddd4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45874/0 (socket says 172.21.15.59:45874) 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 -- 172.21.15.59:0/1964821793 learned_addr learned my addr 172.21.15.59:0/1964821793 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085424640 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9ed3b0 0x5597cde00880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5084c23640 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde01e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 -- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 msgr2=0x5597cde01e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde01e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 -- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9ed3b0 msgr2=0x5597cde00880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9ed3b0 0x5597cde00880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 -- 172.21.15.59:0/1964821793 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597cda81850 con 0x5597cda16270 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f5085c25640 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cda16270 0x5597cddd4280 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f50700039f0 tx=0x7f507000c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3f9fbbf722b83b server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f505ffff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.018+0000 7f505ffff640 1 -- 172.21.15.59:0/1964821793 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070001b30 con 0x5597cda16270 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f505ffff640 1 -- 172.21.15.59:0/1964821793 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f5070004b40 con 0x5597cda16270 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 1 -- 172.21.15.59:0/1964821793 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5070001980 con 0x5597cda16270 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f505ffff640 1 -- 172.21.15.59:0/1964821793 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f507004f050 con 0x5597cda16270 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 10 client.11112 fetch_fsmap learned FSMap version 106 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 1 -- 172.21.15.59:0/1964821793 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) -- 0x5597cdaf86b0 con 0x5597cda16270 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f505ffff640 1 -- 172.21.15.59:0/1964821793 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5070001980 con 0x5597cda16270 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f505ffff640 1 -- 172.21.15.59:0/1964821793 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 106) ==== 3502+0+0 (secure 0 0 0) 0x7f507002b8b0 con 0x5597cda16270 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 10 client.11112 fetch_fsmap finished waiting for FSMap version 106 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 10 client.11112 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:55:44.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 20 client.11112 populate_metadata read hostname 'smithi059' 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] conn(0x5597cd9d4e40 0x5597cde1e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 4 client.11112 mds_command: new command op to 10983 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f508e61d740 1 -- 172.21.15.59:0/1964821793 --> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597cdd2f580 con 0x5597cd9d4e40 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.019+0000 7f5085424640 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] conn(0x5597cd9d4e40 0x5597cde1e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.020+0000 7f5085424640 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] conn(0x5597cd9d4e40 0x5597cde1e540 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c4f299bd12d8bd8 server_cookie=6144a1b3bc974467 in_seq=0 out_seq=0 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.020+0000 7f505ffff640 10 client.11112 ms_handle_connect on v2:172.21.15.153:6834/2163190416 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f505ffff640 1 -- 172.21.15.59:0/1964821793 <== mds.0 v2:172.21.15.153:6834/2163190416 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5597cdd2f580 con 0x5597cd9d4e40 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f505ffff640 10 client.11112 handle_command_reply: tid=0 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 client.11112 shutdown 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 -- 172.21.15.59:0/1964821793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cda16270 msgr2=0x5597cddd4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cda16270 0x5597cddd4280 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f50700039f0 tx=0x7f507000c3e0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 -- 172.21.15.59:0/1964821793 shutdown_connections 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] conn(0x5597cd9d4e40 0x5597cde1e540 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cda16270 0x5597cddd4280 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f505ffff640 0 client.11112 ms_handle_reset on v2:172.21.15.153:6834/2163190416 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde01e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 --2- 172.21.15.59:0/1964821793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9ed3b0 0x5597cde00880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 -- 172.21.15.59:0/1964821793 >> 172.21.15.59:0/1964821793 conn(0x5597cda0a3c0 msgr2=0x5597cdde2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 -- 172.21.15.59:0/1964821793 shutdown_connections 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 1 -- 172.21.15.59:0/1964821793 wait complete. 2024-06-29T15:55:44.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.021+0000 7f508e61d740 20 client.11112 trim_cache size 0 max 16384 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.040+0000 7f508e61d740 1 Processor -- start 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.041+0000 7f508e61d740 1 -- start start 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f508e61d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 0x5597cddbbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f508e61d740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 0x5597cde73cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f508e61d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cde741f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f508e61d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5080201920 con 0x5597cd9f5590 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f508e61d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5080200c80 con 0x5597cd9d4e40 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f508e61d740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f50802012d0 con 0x5597cd9ed3b0 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cde741f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cde741f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45878/0 (socket says 172.21.15.59:45878) 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 -- 172.21.15.59:0/3005477809 learned_addr learned my addr 172.21.15.59:0/3005477809 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 -- 172.21.15.59:0/3005477809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 msgr2=0x5597cde73cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 --2- 172.21.15.59:0/3005477809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 0x5597cde73cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 -- 172.21.15.59:0/3005477809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 msgr2=0x5597cddbbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085424640 1 --2- 172.21.15.59:0/3005477809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 0x5597cddbbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 --2- 172.21.15.59:0/3005477809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 0x5597cddbbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 -- 172.21.15.59:0/3005477809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597cde116b0 con 0x5597cd9f5590 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f5085c25640 1 --2- 172.21.15.59:0/3005477809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cde741f0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f507002ba40 tx=0x7f508015fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466cb6d4fcfd2768 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.042+0000 7f50667fc640 1 -- 172.21.15.59:0/3005477809 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070004b40 con 0x5597cd9f5590 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f50667fc640 1 -- 172.21.15.59:0/3005477809 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f507002f070 con 0x5597cd9f5590 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f50667fc640 1 -- 172.21.15.59:0/3005477809 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070001b60 con 0x5597cd9f5590 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 -- 172.21.15.59:0/3005477809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 msgr2=0x5597cde741f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 --2- 172.21.15.59:0/3005477809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cde741f0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f507002ba40 tx=0x7f508015fe00 comp rx=0 tx=0).stop 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 -- 172.21.15.59:0/3005477809 shutdown_connections 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 --2- 172.21.15.59:0/3005477809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9f5590 0x5597cde741f0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 --2- 172.21.15.59:0/3005477809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9ed3b0 0x5597cde73cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 --2- 172.21.15.59:0/3005477809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 0x5597cddbbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 -- 172.21.15.59:0/3005477809 >> 172.21.15.59:0/3005477809 conn(0x5597cda0a3c0 msgr2=0x5597cddb2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 -- 172.21.15.59:0/3005477809 shutdown_connections 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.043+0000 7f508e61d740 1 -- 172.21.15.59:0/3005477809 wait complete. 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.044+0000 7f508e61d740 1 -- start start 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.044+0000 7f508e61d740 1 Processor -- start 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f508e61d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 0x5597cddd3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f508e61d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9ed3b0 0x5597cde6e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f508e61d740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde7d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f508e61d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5597cde003b0 con 0x5597cd9ed3b0 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f508e61d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5597cdcbaae0 con 0x5597cd9d4e40 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f508e61d740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5597cde11460 con 0x5597cd9f5590 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f508e61d740 10 client.0 _ll_register_callbacks cb 0x7f50680036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9ed3b0 0x5597cde6e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9ed3b0 0x5597cde6e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45880/0 (socket says 172.21.15.59:45880) 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 -- 172.21.15.59:0/1901355620 learned_addr learned my addr 172.21.15.59:0/1901355620 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5085424640 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 0x5597cddd3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5085c25640 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde7d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 -- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 msgr2=0x5597cde7d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde7d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 -- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 msgr2=0x5597cddd3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 0x5597cddd3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 -- 172.21.15.59:0/1901355620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597cddf7a60 con 0x5597cd9ed3b0 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5085c25640 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f5084c23640 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9ed3b0 0x5597cde6e870 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f50800b06b0 tx=0x7f5074004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e3638617a91569 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f505ffff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f505ffff640 1 -- 172.21.15.59:0/1901355620 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074005020 con 0x5597cd9ed3b0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f505ffff640 1 -- 172.21.15.59:0/1901355620 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f5074018070 con 0x5597cd9ed3b0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.046+0000 7f508e61d740 1 -- 172.21.15.59:0/1901355620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f507002f070 con 0x5597cd9ed3b0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f505ffff640 1 -- 172.21.15.59:0/1901355620 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (secure 0 0 0) 0x7f5074036050 con 0x5597cd9ed3b0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f508e61d740 10 client.11118 fetch_fsmap learned FSMap version 106 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f508e61d740 1 -- 172.21.15.59:0/1901355620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) -- 0x5597cddd30c0 con 0x5597cd9ed3b0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f505ffff640 1 -- 172.21.15.59:0/1901355620 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f507002f070 con 0x5597cd9ed3b0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f505ffff640 1 -- 172.21.15.59:0/1901355620 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 106) ==== 3502+0+0 (secure 0 0 0) 0x7f5074039020 con 0x5597cd9ed3b0 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f508e61d740 10 client.11118 fetch_fsmap finished waiting for FSMap version 106 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f508e61d740 10 client.11118 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f508e61d740 20 client.11118 populate_metadata read hostname 'smithi059' 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f508e61d740 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] conn(0x5597cd9aa280 0x5597cdde0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f508e61d740 4 client.11118 mds_command: new command op to 10983 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f508e61d740 1 -- 172.21.15.59:0/1901355620 --> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] -- command(tid 0: {"prefix": "status"}) -- 0x5597cdd2b020 con 0x5597cd9aa280 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.047+0000 7f5085424640 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] conn(0x5597cd9aa280 0x5597cdde0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f505ffff640 10 client.11118 ms_handle_connect on v2:172.21.15.153:6834/2163190416 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f5085424640 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] conn(0x5597cd9aa280 0x5597cdde0d70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a9e52ebd661525 server_cookie=f41b76069d593de8 in_seq=0 out_seq=0 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f505ffff640 1 -- 172.21.15.59:0/1901355620 <== mds.0 v2:172.21.15.153:6834/2163190416 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5597cdd2b020 con 0x5597cd9aa280 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f505ffff640 10 client.11118 handle_command_reply: tid=0 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 client.11118 shutdown 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 -- 172.21.15.59:0/1901355620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9ed3b0 msgr2=0x5597cde6e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9ed3b0 0x5597cde6e870 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f50800b06b0 tx=0x7f5074004730 comp rx=0 tx=0).stop 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 -- 172.21.15.59:0/1901355620 shutdown_connections 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:6834/2163190416,v1:172.21.15.153:6836/2163190416] conn(0x5597cd9aa280 0x5597cdde0d70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5597cd9f5590 0x5597cde7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5597cd9ed3b0 0x5597cde6e870 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 --2- 172.21.15.59:0/1901355620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cd9d4e40 0x5597cddd3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 -- 172.21.15.59:0/1901355620 >> 172.21.15.59:0/1901355620 conn(0x5597cda0a3c0 msgr2=0x5597cddb2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f505ffff640 0 client.11118 ms_handle_reset on v2:172.21.15.153:6834/2163190416 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 -- 172.21.15.59:0/1901355620 shutdown_connections 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 1 -- 172.21.15.59:0/1901355620 wait complete. 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.048+0000 7f508e61d740 20 client.11118 trim_cache size 0 max 16384 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 10983, 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.861470143, 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 106, 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 204, 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 204, 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.472399353 2024-06-29T15:55:44.057 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:55:44.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 -- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f506c0763f0 msgr2=0x7f506c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f506c0763f0 0x7f506c078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5080161b10 tx=0x7f5074005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:44.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 -- 172.21.15.59:0/1995555686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50800c97e0 msgr2=0x7f50800ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50800c97e0 0x7f50800ca680 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f5078011ba0 tx=0x7f507800f790 comp rx=0 tx=0).stop 2024-06-29T15:55:44.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 -- 172.21.15.59:0/1995555686 shutdown_connections 2024-06-29T15:55:44.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f506c0763f0 0x7f506c078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50801f5830 0x7f50801f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50800cf1f0 0x7f50801f52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 --2- 172.21.15.59:0/1995555686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50800c97e0 0x7f50800ca680 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 -- 172.21.15.59:0/1995555686 >> 172.21.15.59:0/1995555686 conn(0x7f50800b06e0 msgr2=0x7f50800ab830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 -- 172.21.15.59:0/1995555686 shutdown_connections 2024-06-29T15:55:44.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.049+0000 7f5087688640 1 -- 172.21.15.59:0/1995555686 wait complete. 2024-06-29T15:55:44.061 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:55:44.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.178+0000 7fa390da1640 1 -- 172.21.15.59:0/3766135659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 msgr2=0x7fa374003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.178+0000 7fa390da1640 1 --2- 172.21.15.59:0/3766135659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 0x7fa374003f30 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009b20 tx=0x7fa374005830 comp rx=0 tx=0).stop 2024-06-29T15:55:44.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.178+0000 7fa390da1640 1 -- 172.21.15.59:0/3766135659 shutdown_connections 2024-06-29T15:55:44.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.178+0000 7fa390da1640 1 --2- 172.21.15.59:0/3766135659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 0x7fa374003f30 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.178+0000 7fa390da1640 1 -- 172.21.15.59:0/3766135659 >> 172.21.15.59:0/3766135659 conn(0x7fa38c0d3ee0 msgr2=0x7fa38c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.178+0000 7fa390da1640 1 -- 172.21.15.59:0/3766135659 shutdown_connections 2024-06-29T15:55:44.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.178+0000 7fa390da1640 1 -- 172.21.15.59:0/3766135659 wait complete. 2024-06-29T15:55:44.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.180+0000 7fa390da1640 1 Processor -- start 2024-06-29T15:55:44.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.180+0000 7fa390da1640 1 -- start start 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.181+0000 7fa390da1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 0x7fa38c149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.181+0000 7fa390da1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa38c14a1f0 0x7fa38c149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.181+0000 7fa390da1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa38c144150 0x7fa38c144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.181+0000 7fa390da1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa38c0b3d20 con 0x7fa374003ac0 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.181+0000 7fa390da1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa38c0b3ba0 con 0x7fa38c14a1f0 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.181+0000 7fa390da1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa38c15e8d0 con 0x7fa38c144150 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 0x7fa38c149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 0x7fa38c149660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45892/0 (socket says 172.21.15.59:45892) 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 -- 172.21.15.59:0/3778844375 learned_addr learned my addr 172.21.15.59:0/3778844375 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa389d9b640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa38c14a1f0 0x7fa38c149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38ad9d640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa38c144150 0x7fa38c144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 -- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa38c144150 msgr2=0x7fa38c144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa38c144150 0x7fa38c144560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 -- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa38c14a1f0 msgr2=0x7fa38c149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa38c14a1f0 0x7fa38c149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 -- 172.21.15.59:0/3778844375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa38c0ca840 con 0x7fa374003ac0 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa38a59c640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 0x7fa38c149660 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009a70 tx=0x7fa37c033650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa37affd640 1 -- 172.21.15.59:0/3778844375 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c040020 con 0x7fa374003ac0 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa37affd640 1 -- 172.21.15.59:0/3778844375 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa37c033b30 con 0x7fa374003ac0 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa38c0c82e0 con 0x7fa374003ac0 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa37affd640 1 -- 172.21.15.59:0/3778844375 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c038040 con 0x7fa374003ac0 2024-06-29T15:55:44.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.182+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa38c0c88a0 con 0x7fa374003ac0 2024-06-29T15:55:44.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.183+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa38c15f9b0 con 0x7fa374003ac0 2024-06-29T15:55:44.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.184+0000 7fa37affd640 1 -- 172.21.15.59:0/3778844375 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fa37c043070 con 0x7fa374003ac0 2024-06-29T15:55:44.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.185+0000 7fa37affd640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fa36c0763f0 0x7fa36c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.185+0000 7fa37affd640 1 -- 172.21.15.59:0/3778844375 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7fa37c0b7820 con 0x7fa374003ac0 2024-06-29T15:55:44.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.185+0000 7fa389d9b640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fa36c0763f0 0x7fa36c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.186+0000 7fa389d9b640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fa36c0763f0 0x7fa36c078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0c8210 tx=0x7fa380006910 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.190+0000 7fa37affd640 1 -- 172.21.15.59:0/3778844375 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa37c080860 con 0x7fa374003ac0 2024-06-29T15:55:44.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.421+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa38c0b5320 con 0x7fa374003ac0 2024-06-29T15:55:44.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.422+0000 7fa37affd640 1 -- 172.21.15.59:0/3778844375 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v106) ==== 53+0+73 (secure 0 0 0) 0x7fa37c084b60 con 0x7fa374003ac0 2024-06-29T15:55:44.423 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:55:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fa36c0763f0 msgr2=0x7fa36c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fa36c0763f0 0x7fa36c078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0c8210 tx=0x7fa380006910 comp rx=0 tx=0).stop 2024-06-29T15:55:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 msgr2=0x7fa38c149660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 0x7fa38c149660 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009a70 tx=0x7fa37c033650 comp rx=0 tx=0).stop 2024-06-29T15:55:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 shutdown_connections 2024-06-29T15:55:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fa36c0763f0 0x7fa36c078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa38c144150 0x7fa38c144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa38c14a1f0 0x7fa38c149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 --2- 172.21.15.59:0/3778844375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa374003ac0 0x7fa38c149660 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 >> 172.21.15.59:0/3778844375 conn(0x7fa38c0d3ee0 msgr2=0x7fa38c06d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 shutdown_connections 2024-06-29T15:55:44.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44.424+0000 7fa390da1640 1 -- 172.21.15.59:0/3778844375 wait complete. 2024-06-29T15:55:44.436 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:55:44.436 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:55:44.436 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T15:55:44.436 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:55:44.436 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:44.436 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:55:44.436 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:55:44.437 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:55:44.437 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:55:44.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.437 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T15:55:44.453 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T15:55:44.454 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:55:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T15:55:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T15:55:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 37106sec preferred_lft 37106sec 2024-06-29T15:55:44.455 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T15:55:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.457 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:55:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.457 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:55:44.458 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T15:55:44.459 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.459 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:55:44.518 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T15:55:44.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.519 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T15:55:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T15:55:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:55:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T15:55:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T15:55:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:55:44.577 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:55:44.577 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T15:55:44.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T15:55:44.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T15:55:44.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T15:55:44.578 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T15:55:44.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:55:44.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:55:44.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.706 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T15:55:44.722 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:55:44.722 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:55:44.722 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:55:44.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.723 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T15:55:44.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:55:44.789 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:44.845 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:55:44.845 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:55:44.845 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:44.900 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T15:55:44.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.900 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:55:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:55:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:55:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:55:45.007 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:55:45.007 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:55:45.007 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:55:45.007 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:55:45.007 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:55:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:55:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:55:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:55:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:55:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:55:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T15:55:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T15:55:45.013 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T15:55:45.014 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:55:45.014 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:55:45.014 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:55:45.014 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:55:45.014 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:55:45.016 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:55:45.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:45.016 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:45.042 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:55:45.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:55:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:55:45.151 INFO:teuthology.orchestra.run.smithi059.stdout:client10396 2024-06-29T15:55:45.151 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:55:45.151 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:45.168 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:55:45.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.327+0000 7fee4c57a640 1 -- 172.21.15.59:0/1011792823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee2c003ac0 msgr2=0x7fee2c003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.327+0000 7fee4c57a640 1 --2- 172.21.15.59:0/1011792823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee2c003ac0 0x7fee2c003f30 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fee34009ed0 tx=0x7fee2c005830 comp rx=0 tx=0).stop 2024-06-29T15:55:45.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.327+0000 7fee4c57a640 1 -- 172.21.15.59:0/1011792823 shutdown_connections 2024-06-29T15:55:45.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.327+0000 7fee4c57a640 1 --2- 172.21.15.59:0/1011792823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee2c003ac0 0x7fee2c003f30 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.327+0000 7fee4c57a640 1 -- 172.21.15.59:0/1011792823 >> 172.21.15.59:0/1011792823 conn(0x7fee440b5610 msgr2=0x7fee440b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:45.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.327+0000 7fee4c57a640 1 -- 172.21.15.59:0/1011792823 shutdown_connections 2024-06-29T15:55:45.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.327+0000 7fee4c57a640 1 -- 172.21.15.59:0/1011792823 wait complete. 2024-06-29T15:55:45.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.328+0000 7fee4c57a640 1 Processor -- start 2024-06-29T15:55:45.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.329+0000 7fee4c57a640 1 -- start start 2024-06-29T15:55:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4c57a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee2c003ac0 0x7fee44156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4c57a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee441588d0 0x7fee440cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4c57a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee440cc350 0x7fee4415ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4c57a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fee440d1a20 con 0x7fee440cc350 2024-06-29T15:55:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4c57a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee440cf3a0 con 0x7fee441588d0 2024-06-29T15:55:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4c57a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fee440d1ba0 con 0x7fee2c003ac0 2024-06-29T15:55:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee440cc350 0x7fee4415ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee440cc350 0x7fee4415ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45906/0 (socket says 172.21.15.59:45906) 2024-06-29T15:55:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 -- 172.21.15.59:0/2661828782 learned_addr learned my addr 172.21.15.59:0/2661828782 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee49b15640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee441588d0 0x7fee440cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4a316640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee2c003ac0 0x7fee44156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 -- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee2c003ac0 msgr2=0x7fee44156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee2c003ac0 0x7fee44156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 -- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee441588d0 msgr2=0x7fee440cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee441588d0 0x7fee440cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 -- 172.21.15.59:0/2661828782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee440cad70 con 0x7fee440cc350 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.330+0000 7fee4ab17640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee440cc350 0x7fee4415ef10 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fee440d01f0 tx=0x7fee3c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.331+0000 7fee42ffd640 1 -- 172.21.15.59:0/2661828782 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c011640 con 0x7fee440cc350 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.331+0000 7fee42ffd640 1 -- 172.21.15.59:0/2661828782 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee3c0117e0 con 0x7fee440cc350 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.331+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee440cb050 con 0x7fee440cc350 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.331+0000 7fee42ffd640 1 -- 172.21.15.59:0/2661828782 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c011c30 con 0x7fee440cc350 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.331+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee440cb610 con 0x7fee440cc350 2024-06-29T15:55:45.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.332+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee440d2100 con 0x7fee440cc350 2024-06-29T15:55:45.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.332+0000 7fee42ffd640 1 -- 172.21.15.59:0/2661828782 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fee3c01a020 con 0x7fee440cc350 2024-06-29T15:55:45.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.333+0000 7fee42ffd640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fee30076420 0x7fee30078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.333+0000 7fee42ffd640 1 -- 172.21.15.59:0/2661828782 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7fee3c097c60 con 0x7fee440cc350 2024-06-29T15:55:45.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.333+0000 7fee4a316640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fee30076420 0x7fee30078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.334+0000 7fee4a316640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fee30076420 0x7fee30078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fee34035000 tx=0x7fee34004a40 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:45.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.337+0000 7fee42ffd640 1 -- 172.21.15.59:0/2661828782 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fee3c060b00 con 0x7fee440cc350 2024-06-29T15:55:45.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.551+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 --> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fee4415c9e0 con 0x7fee30076420 2024-06-29T15:55:45.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.554+0000 7fee42ffd640 1 -- 172.21.15.59:0/2661828782 <== mgr.10063 v2:172.21.15.153:6832/493926731 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fee4415c9e0 con 0x7fee30076420 2024-06-29T15:55:45.554 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:55:45.554 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:55:45.555 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T15:55:45.555 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:55:45.555 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fee30076420 msgr2=0x7fee30078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fee30076420 0x7fee30078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fee34035000 tx=0x7fee34004a40 comp rx=0 tx=0).stop 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee440cc350 msgr2=0x7fee4415ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee440cc350 0x7fee4415ef10 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fee440d01f0 tx=0x7fee3c001630 comp rx=0 tx=0).stop 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 shutdown_connections 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7fee30076420 0x7fee30078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee440cc350 0x7fee4415ef10 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee441588d0 0x7fee440cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 --2- 172.21.15.59:0/2661828782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fee2c003ac0 0x7fee44156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 >> 172.21.15.59:0/2661828782 conn(0x7fee440b5610 msgr2=0x7fee44155210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 shutdown_connections 2024-06-29T15:55:45.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.555+0000 7fee4c57a640 1 -- 172.21.15.59:0/2661828782 wait complete. 2024-06-29T15:55:45.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:55:45.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.687+0000 7f42d97d2640 1 -- 172.21.15.59:0/116483836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d408c190 msgr2=0x7f42d40cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.687+0000 7f42d97d2640 1 --2- 172.21.15.59:0/116483836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d408c190 0x7f42d40cf080 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009860 tx=0x7f42cc02a560 comp rx=0 tx=0).stop 2024-06-29T15:55:45.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.687+0000 7f42d97d2640 1 -- 172.21.15.59:0/116483836 shutdown_connections 2024-06-29T15:55:45.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.687+0000 7f42d97d2640 1 --2- 172.21.15.59:0/116483836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d408c190 0x7f42d40cf080 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.687+0000 7f42d97d2640 1 -- 172.21.15.59:0/116483836 >> 172.21.15.59:0/116483836 conn(0x7f42d40b5610 msgr2=0x7f42d40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:45.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.687+0000 7f42d97d2640 1 -- 172.21.15.59:0/116483836 shutdown_connections 2024-06-29T15:55:45.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.687+0000 7f42d97d2640 1 -- 172.21.15.59:0/116483836 wait complete. 2024-06-29T15:55:45.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.689+0000 7f42d97d2640 1 Processor -- start 2024-06-29T15:55:45.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.689+0000 7f42d97d2640 1 -- start start 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d97d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42d408c190 0x7f42d4151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d97d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42d40ce950 0x7f42d4159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d97d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d40cf5c0 0x7f42d4159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d97d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f42d4169690 con 0x7f42d40cf5c0 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d97d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42d4169510 con 0x7f42d408c190 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d97d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f42d4169810 con 0x7f42d40ce950 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d40cf5c0 0x7f42d4159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42d408c190 0x7f42d4151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d40cf5c0 0x7f42d4159b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45924/0 (socket says 172.21.15.59:45924) 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d37fe640 1 -- 172.21.15.59:0/4027047643 learned_addr learned my addr 172.21.15.59:0/4027047643 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d27fc640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42d40ce950 0x7f42d4159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.690+0000 7f42d37fe640 1 -- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42d40ce950 msgr2=0x7f42d4159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42d37fe640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42d40ce950 0x7f42d4159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42d37fe640 1 -- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42d408c190 msgr2=0x7f42d4151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42d37fe640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42d408c190 0x7f42d4151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42d37fe640 1 -- 172.21.15.59:0/4027047643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d4151dd0 con 0x7f42d40cf5c0 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42d37fe640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d40cf5c0 0x7f42d4159b70 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f42c4002230 tx=0x7f42c400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42cb7fe640 1 -- 172.21.15.59:0/4027047643 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4003b70 con 0x7f42d40cf5c0 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42cb7fe640 1 -- 172.21.15.59:0/4027047643 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42c4003d10 con 0x7f42d40cf5c0 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42cb7fe640 1 -- 172.21.15.59:0/4027047643 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4005330 con 0x7f42d40cf5c0 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d41550a0 con 0x7f42d40cf5c0 2024-06-29T15:55:45.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.691+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d4155660 con 0x7f42d40cf5c0 2024-06-29T15:55:45.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.692+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d4154e10 con 0x7f42d40cf5c0 2024-06-29T15:55:45.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.693+0000 7f42cb7fe640 1 -- 172.21.15.59:0/4027047643 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f42c40056e0 con 0x7f42d40cf5c0 2024-06-29T15:55:45.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.693+0000 7f42cb7fe640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b4076420 0x7f42b4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.693+0000 7f42cb7fe640 1 -- 172.21.15.59:0/4027047643 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f42c4097a60 con 0x7f42d40cf5c0 2024-06-29T15:55:45.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.694+0000 7f42d2ffd640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b4076420 0x7f42b4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.694+0000 7f42d2ffd640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b4076420 0x7f42b4078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f42cc034000 tx=0x7f42cc004730 comp rx=0 tx=0).ready entity=mgr.10063 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:45.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.698+0000 7f42cb7fe640 1 -- 172.21.15.59:0/4027047643 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f42c4060a10 con 0x7f42d40cf5c0 2024-06-29T15:55:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:45.938+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f42d40cf080 con 0x7f42d40cf5c0 2024-06-29T15:55:46.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.222+0000 7f42cb7fe640 1 -- 172.21.15.59:0/4027047643 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v118) ==== 92+0+0 (secure 0 0 0) 0x7f42c4064d10 con 0x7f42d40cf5c0 2024-06-29T15:55:46.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.224+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b4076420 msgr2=0x7f42b4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.224+0000 7f42d97d2640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b4076420 0x7f42b4078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f42cc034000 tx=0x7f42cc004730 comp rx=0 tx=0).stop 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.224+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d40cf5c0 msgr2=0x7f42d4159b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.224+0000 7f42d97d2640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d40cf5c0 0x7f42d4159b70 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f42c4002230 tx=0x7f42c400cb90 comp rx=0 tx=0).stop 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.225+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 shutdown_connections 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.225+0000 7f42d97d2640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f42b4076420 0x7f42b4078930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.225+0000 7f42d97d2640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42d40cf5c0 0x7f42d4159b70 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.225+0000 7f42d97d2640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42d40ce950 0x7f42d4159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.225+0000 7f42d97d2640 1 --2- 172.21.15.59:0/4027047643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42d408c190 0x7f42d4151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.225+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 >> 172.21.15.59:0/4027047643 conn(0x7f42d40b5610 msgr2=0x7f42d4161070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.225+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 shutdown_connections 2024-06-29T15:55:46.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.225+0000 7f42d97d2640 1 -- 172.21.15.59:0/4027047643 wait complete. 2024-06-29T15:55:46.240 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:55:46.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.363+0000 7f9a38dd9640 1 -- 172.21.15.59:0/2823814931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a34166da0 msgr2=0x7f9a341691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.363+0000 7f9a38dd9640 1 --2- 172.21.15.59:0/2823814931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a34166da0 0x7f9a341691c0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009ca0 tx=0x7f9a2c02ab60 comp rx=0 tx=0).stop 2024-06-29T15:55:46.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.363+0000 7f9a38dd9640 1 -- 172.21.15.59:0/2823814931 shutdown_connections 2024-06-29T15:55:46.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.363+0000 7f9a38dd9640 1 --2- 172.21.15.59:0/2823814931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a34166da0 0x7f9a341691c0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.363+0000 7f9a38dd9640 1 -- 172.21.15.59:0/2823814931 >> 172.21.15.59:0/2823814931 conn(0x7f9a340d3ee0 msgr2=0x7f9a340d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.363+0000 7f9a38dd9640 1 -- 172.21.15.59:0/2823814931 shutdown_connections 2024-06-29T15:55:46.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.363+0000 7f9a38dd9640 1 -- 172.21.15.59:0/2823814931 wait complete. 2024-06-29T15:55:46.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.365+0000 7f9a38dd9640 1 Processor -- start 2024-06-29T15:55:46.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.366+0000 7f9a38dd9640 1 -- start start 2024-06-29T15:55:46.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a38dd9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a340b3990 0x7f9a34150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a38dd9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3415e290 0x7f9a34153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a38dd9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a34166da0 0x7f9a3415cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a38dd9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34169700 con 0x7f9a340b3990 2024-06-29T15:55:46.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a38dd9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a340cea30 con 0x7f9a3415e290 2024-06-29T15:55:46.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a38dd9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9a34169880 con 0x7f9a34166da0 2024-06-29T15:55:46.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a3259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a340b3990 0x7f9a34150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a3259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a340b3990 0x7f9a34150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45956/0 (socket says 172.21.15.59:45956) 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a3259c640 1 -- 172.21.15.59:0/3104707264 learned_addr learned my addr 172.21.15.59:0/3104707264 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a31d9b640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3415e290 0x7f9a34153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a32d9d640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a34166da0 0x7f9a3415cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a3259c640 1 -- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a34166da0 msgr2=0x7f9a3415cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a3259c640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a34166da0 0x7f9a3415cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a3259c640 1 -- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3415e290 msgr2=0x7f9a34153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a3259c640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3415e290 0x7f9a34153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.367+0000 7f9a3259c640 1 -- 172.21.15.59:0/3104707264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a34155000 con 0x7f9a340b3990 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.368+0000 7f9a3259c640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a340b3990 0x7f9a34150e30 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c370 tx=0x7f9a2400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.368+0000 7f9a2affd640 1 -- 172.21.15.59:0/3104707264 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24017070 con 0x7f9a340b3990 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.368+0000 7f9a2affd640 1 -- 172.21.15.59:0/3104707264 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a240035f0 con 0x7f9a340b3990 2024-06-29T15:55:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.368+0000 7f9a2affd640 1 -- 172.21.15.59:0/3104707264 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24003a40 con 0x7f9a340b3990 2024-06-29T15:55:46.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.368+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a341552e0 con 0x7f9a340b3990 2024-06-29T15:55:46.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.368+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a341558a0 con 0x7f9a340b3990 2024-06-29T15:55:46.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.369+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a34155c80 con 0x7f9a340b3990 2024-06-29T15:55:46.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.370+0000 7f9a2affd640 1 -- 172.21.15.59:0/3104707264 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 118) ==== 98320+0+0 (secure 0 0 0) 0x7f9a2401a070 con 0x7f9a340b3990 2024-06-29T15:55:46.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.370+0000 7f9a2affd640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9a14076440 0x7f9a14078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.371+0000 7f9a31d9b640 1 -- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9a14076440 msgr2=0x7f9a14078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/493926731 2024-06-29T15:55:46.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.371+0000 7f9a2affd640 1 -- 172.21.15.59:0/3104707264 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f9a24096d70 con 0x7f9a340b3990 2024-06-29T15:55:46.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.371+0000 7f9a31d9b640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9a14076440 0x7f9a14078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:46.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.379+0000 7f9a2affd640 1 -- 172.21.15.59:0/3104707264 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a34155c80 con 0x7f9a340b3990 2024-06-29T15:55:46.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.572+0000 7f9a31d9b640 1 -- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9a14076440 msgr2=0x7f9a14078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/493926731 2024-06-29T15:55:46.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.572+0000 7f9a31d9b640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9a14076440 0x7f9a14078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:46.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.573+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f9a341514a0 con 0x7f9a340b3990 2024-06-29T15:55:46.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.576+0000 7f9a2affd640 1 -- 172.21.15.59:0/3104707264 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v69)=0 v69) ==== 138+0+0 (secure 0 0 0) 0x7f9a240683f0 con 0x7f9a340b3990 2024-06-29T15:55:46.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9a14076440 msgr2=0x7f9a14078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9a14076440 0x7f9a14078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a340b3990 msgr2=0x7f9a34150e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a340b3990 0x7f9a34150e30 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c370 tx=0x7f9a2400c830 comp rx=0 tx=0).stop 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 shutdown_connections 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f9a14076440 0x7f9a14078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a34166da0 0x7f9a3415cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3415e290 0x7f9a34153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 --2- 172.21.15.59:0/3104707264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a340b3990 0x7f9a34150e30 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 >> 172.21.15.59:0/3104707264 conn(0x7f9a340d3ee0 msgr2=0x7f9a341577b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 shutdown_connections 2024-06-29T15:55:46.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.578+0000 7f9a38dd9640 1 -- 172.21.15.59:0/3104707264 wait complete. 2024-06-29T15:55:46.591 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:55:46.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.712+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2275889441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 msgr2=0x7f64ac15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.712+0000 7f64b2f7a640 1 --2- 172.21.15.59:0/2275889441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 0x7f64ac15f9b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f64a4005fd0 tx=0x7f64a402a940 comp rx=0 tx=0).stop 2024-06-29T15:55:46.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.712+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2275889441 shutdown_connections 2024-06-29T15:55:46.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.712+0000 7f64b2f7a640 1 --2- 172.21.15.59:0/2275889441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 0x7f64ac15f9b0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.712+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2275889441 >> 172.21.15.59:0/2275889441 conn(0x7f64ac0d3ee0 msgr2=0x7f64ac0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.712+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2275889441 shutdown_connections 2024-06-29T15:55:46.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.712+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2275889441 wait complete. 2024-06-29T15:55:46.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.714+0000 7f64b2f7a640 1 Processor -- start 2024-06-29T15:55:46.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.714+0000 7f64b2f7a640 1 -- start start 2024-06-29T15:55:46.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.715+0000 7f64b2f7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64ac0d23c0 0x7f64ac15b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.715+0000 7f64b2f7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 0x7f64ac0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.715+0000 7f64b2f7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64ac14d4b0 0x7f64ac14f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.715+0000 7f64b2f7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f64ac160420 con 0x7f64ac0d23c0 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.715+0000 7f64b2f7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f64ac1602a0 con 0x7f64ac15efe0 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.715+0000 7f64b2f7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f64ac1605a0 con 0x7f64ac14d4b0 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64b0d16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64ac0d23c0 0x7f64ac15b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64abfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 0x7f64ac0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64b0d16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64ac0d23c0 0x7f64ac15b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45962/0 (socket says 172.21.15.59:45962) 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64b0d16640 1 -- 172.21.15.59:0/2271697988 learned_addr learned my addr 172.21.15.59:0/2271697988 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64b1517640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64ac14d4b0 0x7f64ac14f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64abfff640 1 -- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64ac14d4b0 msgr2=0x7f64ac14f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64abfff640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64ac14d4b0 0x7f64ac14f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64abfff640 1 -- 172.21.15.59:0/2271697988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64ac0d23c0 msgr2=0x7f64ac15b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64abfff640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64ac0d23c0 0x7f64ac15b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64abfff640 1 -- 172.21.15.59:0/2271697988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ac159070 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64b0d16640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64ac0d23c0 0x7f64ac15b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64abfff640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 0x7f64ac0d0950 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0d2cd0 tx=0x7f649400c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64a97fa640 1 -- 172.21.15.59:0/2271697988 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494011790 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64a97fa640 1 -- 172.21.15.59:0/2271697988 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6494017070 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ac159350 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64a97fa640 1 -- 172.21.15.59:0/2271697988 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494011d00 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.716+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ac14f0f0 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.718+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ac15f9b0 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.718+0000 7f64a97fa640 1 -- 172.21.15.59:0/2271697988 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 118) ==== 98320+0+0 (secure 0 0 0) 0x7f649401c460 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.719+0000 7f64a97fa640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f648c076440 0x7f648c078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.719+0000 7f64b0d16640 1 -- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f648c076440 msgr2=0x7f648c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/493926731 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.719+0000 7f64a97fa640 1 -- 172.21.15.59:0/2271697988 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f6494097790 con 0x7f64ac15efe0 2024-06-29T15:55:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.719+0000 7f64b0d16640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f648c076440 0x7f648c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.723+0000 7f64a97fa640 1 -- 172.21.15.59:0/2271697988 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6494060740 con 0x7f64ac15efe0 2024-06-29T15:55:46.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.914+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f64ac0d2950 con 0x7f64ac15efe0 2024-06-29T15:55:46.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.920+0000 7f64b0d16640 1 -- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f648c076440 msgr2=0x7f648c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/493926731 2024-06-29T15:55:46.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.920+0000 7f64b0d16640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f648c076440 0x7f648c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:46.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.920+0000 7f64a97fa640 1 -- 172.21.15.59:0/2271697988 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v70)=0 v70) ==== 133+0+0 (secure 0 0 0) 0x7f6494064a40 con 0x7f64ac15efe0 2024-06-29T15:55:46.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f648c076440 msgr2=0x7f648c078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:46.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f648c076440 0x7f648c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 msgr2=0x7f64ac0d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 0x7f64ac0d0950 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0d2cd0 tx=0x7f649400c760 comp rx=0 tx=0).stop 2024-06-29T15:55:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 shutdown_connections 2024-06-29T15:55:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f648c076440 0x7f648c078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64ac14d4b0 0x7f64ac14f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64ac15efe0 0x7f64ac0d0950 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 --2- 172.21.15.59:0/2271697988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64ac0d23c0 0x7f64ac15b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 >> 172.21.15.59:0/2271697988 conn(0x7f64ac0d3ee0 msgr2=0x7f64ac0d2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 shutdown_connections 2024-06-29T15:55:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:46.923+0000 7f64b2f7a640 1 -- 172.21.15.59:0/2271697988 wait complete. 2024-06-29T15:55:46.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:46.936 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T15:55:46.956 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-06-29T15:55:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.118+0000 7f50e2081640 1 -- 172.21.15.59:0/4101065787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50c4003a70 msgr2=0x7f50c4003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.118+0000 7f50e2081640 1 --2- 172.21.15.59:0/4101065787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50c4003a70 0x7f50c4003ee0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0d2e40 tx=0x7f50c4005830 comp rx=0 tx=0).stop 2024-06-29T15:55:47.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.118+0000 7f50dbfff640 1 -- 172.21.15.59:0/4101065787 reap_dead start 2024-06-29T15:55:47.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.118+0000 7f50e2081640 1 -- 172.21.15.59:0/4101065787 shutdown_connections 2024-06-29T15:55:47.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.118+0000 7f50e2081640 1 -- 172.21.15.59:0/4101065787 >> 172.21.15.59:0/4101065787 conn(0x7f50dc0d3ee0 msgr2=0x7f50dc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:47.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.118+0000 7f50e2081640 1 -- 172.21.15.59:0/4101065787 shutdown_connections 2024-06-29T15:55:47.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.118+0000 7f50e2081640 1 -- 172.21.15.59:0/4101065787 wait complete. 2024-06-29T15:55:47.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.120+0000 7f50e2081640 1 Processor -- start 2024-06-29T15:55:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.120+0000 7f50e2081640 1 -- start start 2024-06-29T15:55:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.121+0000 7f50e2081640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50c4003a70 0x7f50dc14e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.121+0000 7f50e2081640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50dc154630 0x7f50dc156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.121+0000 7f50e2081640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50dc154a40 0x7f50dc156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.121+0000 7f50e2081640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc15e8d0 con 0x7f50dc154a40 2024-06-29T15:55:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.121+0000 7f50e2081640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc0b42b0 con 0x7f50c4003a70 2024-06-29T15:55:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.121+0000 7f50e2081640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f50dc15ea50 con 0x7f50dc154630 2024-06-29T15:55:47.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.121+0000 7f50dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50dc154a40 0x7f50dc156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50dc154a40 0x7f50dc156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45982/0 (socket says 172.21.15.59:45982) 2024-06-29T15:55:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50dbfff640 1 -- 172.21.15.59:0/2185073140 learned_addr learned my addr 172.21.15.59:0/2185073140 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50db7fe640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50c4003a70 0x7f50dc14e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50daffd640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50dc154630 0x7f50dc156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50dbfff640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50dc154630 msgr2=0x7f50dc156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50dbfff640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50dc154630 0x7f50dc156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50dbfff640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50c4003a70 msgr2=0x7f50dc14e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:47.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50dbfff640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50c4003a70 0x7f50dc14e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50dbfff640 1 -- 172.21.15.59:0/2185073140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc14e950 con 0x7f50dc154a40 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50daffd640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f50dc154630 0x7f50dc156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50dbfff640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50dc154a40 0x7f50dc156750 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f50d4005fd0 tx=0x7f50d400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4017070 con 0x7f50dc154a40 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50d4015070 con 0x7f50dc154a40 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50dc0cb150 con 0x7f50dc154a40 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4003970 con 0x7f50dc154a40 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.122+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50dc0cb710 con 0x7f50dc154a40 2024-06-29T15:55:47.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.124+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50dc0caec0 con 0x7f50dc154a40 2024-06-29T15:55:47.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.124+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 118) ==== 98320+0+0 (secure 0 0 0) 0x7f50d4003b10 con 0x7f50dc154a40 2024-06-29T15:55:47.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.125+0000 7f50c3fff640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 0x7f50c8078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:47.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.125+0000 7f50db7fe640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 msgr2=0x7f50c8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/493926731 2024-06-29T15:55:47.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.125+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6157+0+0 (secure 0 0 0) 0x7f50d40982e0 con 0x7f50dc154a40 2024-06-29T15:55:47.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.125+0000 7f50db7fe640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 0x7f50c8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:47.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.129+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f50dc14e950 con 0x7f50dc154a40 2024-06-29T15:55:47.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.326+0000 7f50db7fe640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 msgr2=0x7f50c8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/493926731 2024-06-29T15:55:47.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.326+0000 7f50db7fe640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 0x7f50c8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:47.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.354+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 --> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f50dc153630 con 0x7f50c8076470 2024-06-29T15:55:47.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.727+0000 7f50db7fe640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 msgr2=0x7f50c8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/493926731 2024-06-29T15:55:47.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:47.727+0000 7f50db7fe640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 0x7f50c8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:55:48.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:48.528+0000 7f50db7fe640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 msgr2=0x7f50c8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/493926731 2024-06-29T15:55:48.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:48.528+0000 7f50db7fe640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 0x7f50c8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:55:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:49.828+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 119) ==== 97929+0+0 (secure 0 0 0) 0x7f50d405d480 con 0x7f50dc154a40 2024-06-29T15:55:49.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:49.829+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 msgr2=0x7f50c8078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:49.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:49.829+0000 7f50c3fff640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/493926731,v1:172.21.15.153:6833/493926731] conn(0x7f50c8076470 0x7f50c8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:49.981 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c3446640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.981 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c3446640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.981 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c3446640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.981 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c3446640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c644c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c3446640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c644c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c644c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c644c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:49.982 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:55:49.980+0000 7fe3c644c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.831+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 120) ==== 98056+0+0 (secure 0 0 0) 0x7f50d40540e0 con 0x7f50dc154a40 2024-06-29T15:55:50.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.832+0000 7f50c3fff640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f50c803b000 0x7f50c8080a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:50.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.832+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 --> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f50d407a720 con 0x7f50c803b000 2024-06-29T15:55:50.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.835+0000 7f50db7fe640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f50c803b000 0x7f50c8080a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:50.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.838+0000 7f50db7fe640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f50c803b000 0x7f50c8080a80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f50cc033000 tx=0x7f50cc001a30 comp rx=0 tx=0).ready entity=mgr.9128 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:50.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.909+0000 7f50c3fff640 1 -- 172.21.15.59:0/2185073140 <== mgr.9128 v2:172.21.15.153:6832/1875870697 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f50d407a720 con 0x7f50c803b000 2024-06-29T15:55:50.910 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1 2024-06-29T15:55:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.911+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f50c803b000 msgr2=0x7f50c8080a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.911+0000 7f50e2081640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f50c803b000 0x7f50c8080a80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f50cc033000 tx=0x7f50cc001a30 comp rx=0 tx=0).stop 2024-06-29T15:55:50.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.911+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50dc154a40 msgr2=0x7f50dc156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.911+0000 7f50e2081640 1 --2- 172.21.15.59:0/2185073140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50dc154a40 0x7f50dc156750 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f50d4005fd0 tx=0x7f50d400e720 comp rx=0 tx=0).stop 2024-06-29T15:55:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.911+0000 7f50dbfff640 1 -- 172.21.15.59:0/2185073140 reap_dead start 2024-06-29T15:55:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.911+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 shutdown_connections 2024-06-29T15:55:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.911+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 >> 172.21.15.59:0/2185073140 conn(0x7f50dc0d3ee0 msgr2=0x7f50dc06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.912+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 shutdown_connections 2024-06-29T15:55:50.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:50.912+0000 7f50e2081640 1 -- 172.21.15.59:0/2185073140 wait complete. 2024-06-29T15:55:50.925 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T15:55:50.925 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-06-29T15:55:51.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.048+0000 7fe39b143640 1 -- 172.21.15.59:0/1926937284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3940cec20 msgr2=0x7fe3940d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.048+0000 7fe39b143640 1 --2- 172.21.15.59:0/1926937284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3940cec20 0x7fe3940d1790 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe380009860 tx=0x7fe38002a4b0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.048+0000 7fe39b143640 1 -- 172.21.15.59:0/1926937284 shutdown_connections 2024-06-29T15:55:51.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.048+0000 7fe39b143640 1 --2- 172.21.15.59:0/1926937284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3940cec20 0x7fe3940d1790 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.048+0000 7fe39b143640 1 --2- 172.21.15.59:0/1926937284 >> v2:172.21.15.153:3300/0 conn(0x7fe3941423c0 0x7fe3940b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.048+0000 7fe39b143640 1 -- 172.21.15.59:0/1926937284 >> 172.21.15.59:0/1926937284 conn(0x7fe394073e10 msgr2=0x7fe394074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.048+0000 7fe39b143640 1 -- 172.21.15.59:0/1926937284 shutdown_connections 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.048+0000 7fe39b143640 1 -- 172.21.15.59:0/1926937284 wait complete. 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.050+0000 7fe39b143640 1 Processor -- start 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.050+0000 7fe39b143640 1 -- start start 2024-06-29T15:55:51.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe39b143640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3940cec20 0x7fe3940cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe39b143640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3940d1cd0 0x7fe39415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe39b143640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3941423c0 0x7fe394144a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe39b143640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe3940b2bf0 con 0x7fe3940d1cd0 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe39b143640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe39415c4d0 con 0x7fe3941423c0 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe39b143640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe3940b2d70 con 0x7fe3940cec20 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe393fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3940d1cd0 0x7fe39415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe393fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3940d1cd0 0x7fe39415ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45988/0 (socket says 172.21.15.59:45988) 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe393fff640 1 -- 172.21.15.59:0/3753732136 learned_addr learned my addr 172.21.15.59:0/3753732136 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe398edf640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3940cec20 0x7fe3940cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe3996e0640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3941423c0 0x7fe394144a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe393fff640 1 -- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3940cec20 msgr2=0x7fe3940cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.051+0000 7fe393fff640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3940cec20 0x7fe3940cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe393fff640 1 -- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3941423c0 msgr2=0x7fe394144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe393fff640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3941423c0 0x7fe394144a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe393fff640 1 -- 172.21.15.59:0/3753732136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3941f3430 con 0x7fe3940d1cd0 2024-06-29T15:55:51.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe393fff640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3940d1cd0 0x7fe39415ae60 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fe380009980 tx=0x7fe3800305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe3917fa640 1 -- 172.21.15.59:0/3753732136 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe380030bd0 con 0x7fe3940d1cd0 2024-06-29T15:55:51.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe3917fa640 1 -- 172.21.15.59:0/3753732136 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe38003b410 con 0x7fe3940d1cd0 2024-06-29T15:55:51.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3941f2670 con 0x7fe3940d1cd0 2024-06-29T15:55:51.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe3917fa640 1 -- 172.21.15.59:0/3753732136 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe380030e60 con 0x7fe3940d1cd0 2024-06-29T15:55:51.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.052+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3941f2c30 con 0x7fe3940d1cd0 2024-06-29T15:55:51.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.053+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3940b3290 con 0x7fe3940d1cd0 2024-06-29T15:55:51.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.054+0000 7fe3917fa640 1 -- 172.21.15.59:0/3753732136 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 120) ==== 98056+0+0 (secure 0 0 0) 0x7fe38003b860 con 0x7fe3940d1cd0 2024-06-29T15:55:51.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.054+0000 7fe3917fa640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fe378076170 0x7fe378078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.054+0000 7fe3917fa640 1 -- 172.21.15.59:0/3753732136 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 6501+0+0 (secure 0 0 0) 0x7fe3800b8da0 con 0x7fe3940d1cd0 2024-06-29T15:55:51.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.054+0000 7fe398edf640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fe378076170 0x7fe378078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.055+0000 7fe398edf640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fe378076170 0x7fe378078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe388005fd0 tx=0x7fe388009260 comp rx=0 tx=0).ready entity=mgr.9128 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.059+0000 7fe3917fa640 1 -- 172.21.15.59:0/3753732136 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe38003f030 con 0x7fe3940d1cd0 2024-06-29T15:55:51.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.273+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 --> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) -- 0x7fe3940cb2d0 con 0x7fe378076170 2024-06-29T15:55:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.338+0000 7fe3917fa640 1 -- 172.21.15.59:0/3753732136 <== mgr.9128 v2:172.21.15.153:6832/1875870697 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+39 (secure 0 0 0) 0x7fe3940cb2d0 con 0x7fe378076170 2024-06-29T15:55:51.339 INFO:teuthology.orchestra.run.smithi059.stdout:Retention added to path snap_test_dir1 2024-06-29T15:55:51.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fe378076170 msgr2=0x7fe378078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fe378076170 0x7fe378078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe388005fd0 tx=0x7fe388009260 comp rx=0 tx=0).stop 2024-06-29T15:55:51.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3940d1cd0 msgr2=0x7fe39415ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3940d1cd0 0x7fe39415ae60 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fe380009980 tx=0x7fe3800305d0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 shutdown_connections 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fe378076170 0x7fe378078680 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3941423c0 0x7fe394144a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3940d1cd0 0x7fe39415ae60 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 --2- 172.21.15.59:0/3753732136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3940cec20 0x7fe3940cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 >> 172.21.15.59:0/3753732136 conn(0x7fe394073e10 msgr2=0x7fe394074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 shutdown_connections 2024-06-29T15:55:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.340+0000 7fe39b143640 1 -- 172.21.15.59:0/3753732136 wait complete. 2024-06-29T15:55:51.353 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-06-29T15:55:51.353 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-15_56_00 in ~9s... 2024-06-29T15:55:51.353 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-06-29T15:55:51.353 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-06-29T15:55:51.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.472+0000 7f861df5b640 1 -- 172.21.15.59:0/809307347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 msgr2=0x7f8600005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.472+0000 7f861df5b640 1 --2- 172.21.15.59:0/809307347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 0x7f8600005260 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f8610005fd0 tx=0x7f8600005830 comp rx=0 tx=0).stop 2024-06-29T15:55:51.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.472+0000 7f8617fff640 1 -- 172.21.15.59:0/809307347 reap_dead start 2024-06-29T15:55:51.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.473+0000 7f861df5b640 1 -- 172.21.15.59:0/809307347 shutdown_connections 2024-06-29T15:55:51.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.473+0000 7f861df5b640 1 -- 172.21.15.59:0/809307347 >> 172.21.15.59:0/809307347 conn(0x7f861806c950 msgr2=0x7f8618073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.473+0000 7f861df5b640 1 -- 172.21.15.59:0/809307347 shutdown_connections 2024-06-29T15:55:51.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.473+0000 7f861df5b640 1 -- 172.21.15.59:0/809307347 wait complete. 2024-06-29T15:55:51.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.474+0000 7f861df5b640 1 Processor -- start 2024-06-29T15:55:51.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.475+0000 7f861df5b640 1 -- start start 2024-06-29T15:55:51.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f861df5b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 0x7f86180b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f861df5b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86181f66b0 0x7f86181596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f861df5b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8618159c30 0x7f86180ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f861df5b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f86180b5c10 con 0x7f8600004d90 2024-06-29T15:55:51.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f861df5b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86180b5a90 con 0x7f86181f66b0 2024-06-29T15:55:51.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f861df5b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f86180b5d90 con 0x7f8618159c30 2024-06-29T15:55:51.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 0x7f86180b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 0x7f86180b0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46018/0 (socket says 172.21.15.59:46018) 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 -- 172.21.15.59:0/3044974179 learned_addr learned my addr 172.21.15.59:0/3044974179 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f8616ffd640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86181f66b0 0x7f86181596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f8617fff640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8618159c30 0x7f86180ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 -- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8618159c30 msgr2=0x7f86180ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8618159c30 0x7f86180ad6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 -- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86181f66b0 msgr2=0x7f86181596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86181f66b0 0x7f86181596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 -- 172.21.15.59:0/3044974179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86180af640 con 0x7f8600004d90 2024-06-29T15:55:51.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f8616ffd640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86181f66b0 0x7f86181596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:51.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.476+0000 7f86177fe640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 0x7f86180b0670 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f860800b5b0 tx=0x7f860800ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.477+0000 7f85fffff640 1 -- 172.21.15.59:0/3044974179 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8608010a80 con 0x7f8600004d90 2024-06-29T15:55:51.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.477+0000 7f85fffff640 1 -- 172.21.15.59:0/3044974179 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8608015070 con 0x7f8600004d90 2024-06-29T15:55:51.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.477+0000 7f85fffff640 1 -- 172.21.15.59:0/3044974179 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860801a470 con 0x7f8600004d90 2024-06-29T15:55:51.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.477+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86180af8b0 con 0x7f8600004d90 2024-06-29T15:55:51.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.477+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86180afe70 con 0x7f8600004d90 2024-06-29T15:55:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.478+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86181458a0 con 0x7f8600004d90 2024-06-29T15:55:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.478+0000 7f85fffff640 1 -- 172.21.15.59:0/3044974179 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 120) ==== 98056+0+0 (secure 0 0 0) 0x7f860801a610 con 0x7f8600004d90 2024-06-29T15:55:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.479+0000 7f85fffff640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f8604076170 0x7f8604078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.479+0000 7f85fffff640 1 -- 172.21.15.59:0/3044974179 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 6501+0+0 (secure 0 0 0) 0x7f8608098330 con 0x7f8600004d90 2024-06-29T15:55:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.479+0000 7f8616ffd640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f8604076170 0x7f8604078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.480+0000 7f8616ffd640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f8604076170 0x7f8604078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f860c000ff0 tx=0x7f860c006720 comp rx=0 tx=0).ready entity=mgr.9128 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.483+0000 7f85fffff640 1 -- 172.21.15.59:0/3044974179 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8608061cb0 con 0x7f8600004d90 2024-06-29T15:55:51.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.697+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 --> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f86180b1f10 con 0x7f8604076170 2024-06-29T15:55:51.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.704+0000 7f85fffff640 1 -- 172.21.15.59:0/3044974179 <== mgr.9128 v2:172.21.15.153:6832/1875870697 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+72 (secure 0 0 0) 0x7f86180b1f10 con 0x7f8604076170 2024-06-29T15:55:51.705 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:55:51.705 INFO:teuthology.orchestra.run.smithi059.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-06-29T15:55:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f8604076170 msgr2=0x7f8604078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f8604076170 0x7f8604078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f860c000ff0 tx=0x7f860c006720 comp rx=0 tx=0).stop 2024-06-29T15:55:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 msgr2=0x7f86180b0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 0x7f86180b0670 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f860800b5b0 tx=0x7f860800ba70 comp rx=0 tx=0).stop 2024-06-29T15:55:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 shutdown_connections 2024-06-29T15:55:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f8604076170 0x7f8604078680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8618159c30 0x7f86180ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86181f66b0 0x7f86181596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 --2- 172.21.15.59:0/3044974179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8600004d90 0x7f86180b0670 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 >> 172.21.15.59:0/3044974179 conn(0x7f861806c950 msgr2=0x7f8618141df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 shutdown_connections 2024-06-29T15:55:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:55:51.706+0000 7f861df5b640 1 -- 172.21.15.59:0/3044974179 wait complete. 2024-06-29T15:55:51.719 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-06-29T15:55:51.719 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-06-29T15:55:51.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:51.719 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:51.736 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:51.736 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:51.737 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:52.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:52.737 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:52.758 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:52.759 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:52.759 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:53.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:53.760 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:53.781 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:53.781 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:53.781 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:54.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:54.783 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:54.805 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:54.805 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:54.805 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:55.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:55.806 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:55.827 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:55.827 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:55.827 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:56.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:56.828 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:56.849 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:56.849 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:56.849 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:57.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:57.851 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:57.872 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:57.872 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:57.872 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:58.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:58.873 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:58.895 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:58.895 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:58.895 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:59.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:59.896 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:59.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:59.917 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:59.917 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:00.351 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:56:00.350+0000 7fe3b542a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.351 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:56:00.350+0000 7fe3b542a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.351 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:56:00.350+0000 7fe3b542a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.351 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:56:00.350+0000 7fe3b542a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.351 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:56:00.350+0000 7fe3b542a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:00.918 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:00.939 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:00.940 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:00.940 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:00.940 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:00.940 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-06-29-15_56_00_UTC'] 2024-06-29T15:56:00.940 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719676551.353351, now = 1719676560.9407256, timo = 9 2024-06-29T15:56:01.941 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-06-29-15_56_00 in ~69s... 2024-06-29T15:56:01.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:01.941 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:01.961 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:01.961 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:01.961 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:01.961 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:02.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:02.962 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:02.983 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:02.983 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:02.983 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:02.983 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:03.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:03.984 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:04.005 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:04.006 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:04.006 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:04.006 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:05.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:05.007 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:05.027 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:05.028 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:05.028 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:05.028 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:06.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:06.029 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:06.049 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:06.050 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:06.050 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:06.050 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:07.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:07.052 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:07.072 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:07.073 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:07.073 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:07.073 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:08.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:08.074 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:08.095 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:08.096 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:08.096 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:08.096 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:09.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:09.097 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:09.117 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:09.118 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:09.118 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:09.119 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:10.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:10.120 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:10.141 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:10.141 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:10.142 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:10.142 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:11.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:11.142 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:11.162 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:11.163 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:11.163 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:11.163 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:12.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:12.164 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:12.184 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:12.185 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:12.185 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:12.185 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:12.881 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:56:12.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:56:13.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:13.186 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:13.207 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:13.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:13.208 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:13.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:14.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:14.209 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:14.230 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:14.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:14.231 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:14.231 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:15.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:15.231 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:15.252 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:15.252 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:15.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:15.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:16.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:16.254 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:16.275 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:16.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:16.277 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:16.277 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:17.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:17.278 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:17.299 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:17.300 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:17.300 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:17.300 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:18.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:18.301 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:18.322 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:18.322 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:18.323 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:18.323 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:19.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:19.323 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:19.344 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:19.345 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:19.345 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:19.345 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:20.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:20.346 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:20.367 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:20.367 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:20.367 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:20.367 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:21.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:21.368 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:21.389 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:21.389 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:21.389 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:21.389 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:22.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:22.391 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:22.412 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:22.412 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:22.412 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:22.412 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:23.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:23.413 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:23.433 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:23.434 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:23.434 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:23.434 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:24.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:24.435 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:24.455 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:24.456 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:24.456 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:24.456 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:25.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:25.457 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:25.478 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:25.479 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:25.479 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:25.479 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:26.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:26.480 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:26.501 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:26.502 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:26.502 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:26.502 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:27.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:27.503 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:27.523 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:27.524 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:27.524 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:27.524 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:28.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:28.526 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:28.546 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:28.547 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:28.547 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:28.547 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:29.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:29.548 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:29.569 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:29.570 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:29.570 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:29.570 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:30.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:30.571 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:30.592 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:30.592 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:30.592 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:30.592 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:31.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:31.594 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:31.614 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:31.615 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:31.615 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:31.615 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:32.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:32.616 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:32.637 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:32.637 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:32.638 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:32.638 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:33.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:33.639 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:33.660 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:33.660 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:33.660 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:33.660 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:34.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:34.661 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:34.682 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:34.682 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:34.682 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:34.682 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:35.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:35.684 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:35.706 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:35.707 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:35.707 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:35.707 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:36.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:36.708 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:36.729 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:36.731 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:36.731 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:36.731 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:37.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:37.732 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:37.753 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:37.754 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:37.754 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:37.754 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:38.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:38.755 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:38.777 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:38.777 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:38.777 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:38.777 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:39.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:39.779 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:39.800 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:39.801 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:39.801 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:39.801 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:40.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:40.802 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:40.822 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:40.823 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:40.823 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:40.824 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:41.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:41.825 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:41.846 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:41.846 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:41.846 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:41.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:42.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:42.847 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:42.871 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:42.871 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:42.871 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:42.871 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:42.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:56:42.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:56:43.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:43.872 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:43.892 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:43.893 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:43.893 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:43.893 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:44.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:44.895 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:44.916 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:44.916 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:44.916 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:44.916 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:45.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:45.917 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:45.939 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:45.939 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:45.939 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:45.939 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:46.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:46.940 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:46.960 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:46.961 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:46.961 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:46.961 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:47.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:47.962 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:47.984 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:47.984 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:47.984 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:47.984 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:48.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:48.985 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:49.005 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:49.006 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:49.006 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:49.006 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:50.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:50.007 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:50.028 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:50.028 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:50.028 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:50.028 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:51.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:51.029 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:51.049 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:51.051 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:51.051 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:51.051 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:52.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:52.051 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:52.072 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:52.072 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:52.072 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:52.073 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:53.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:53.074 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:53.095 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:53.095 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:53.095 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:53.095 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:54.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:54.096 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:54.117 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:54.117 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:54.117 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:54.117 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:55.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:55.118 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:55.139 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:55.140 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:55.140 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:55.140 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:56.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:56.141 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:56.161 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:56.162 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:56.162 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:56.162 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:57.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:57.164 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:57.184 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:57.185 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:57.185 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:57.185 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:58.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:58.186 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:58.207 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:58.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:58.208 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:58.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:59.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:59.209 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:59.229 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:59.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:59.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:59.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:57:00.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:00.231 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:57:00.251 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:57:00.252 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:57:00.252 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:57:00.252 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:57:01.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:01.254 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:57:01.274 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_57_00_UTC 2024-06-29T15:57:01.275 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_57_00_UTC'} 2024-06-29T15:57:01.275 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-15_57_00_UTC'} 2024-06-29T15:57:01.275 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:57:01.275 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-06-29-15_56_00_UTC'] 2024-06-29T15:57:01.275 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719676551.353351, now = 1719676621.2757096, timo = 69 2024-06-29T15:57:02.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T15:57:02.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.411+0000 7f330bbe1640 1 -- 172.21.15.59:0/1989394561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040ad6b0 msgr2=0x7f33041658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.411+0000 7f330bbe1640 1 --2- 172.21.15.59:0/1989394561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040ad6b0 0x7f33041658b0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f32fc005fd0 tx=0x7f32fc02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.411+0000 7f330bbe1640 1 -- 172.21.15.59:0/1989394561 shutdown_connections 2024-06-29T15:57:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.411+0000 7f330bbe1640 1 --2- 172.21.15.59:0/1989394561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040ad6b0 0x7f33041658b0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.411+0000 7f330bbe1640 1 -- 172.21.15.59:0/1989394561 >> 172.21.15.59:0/1989394561 conn(0x7f33040b1ea0 msgr2=0x7f33040b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.411+0000 7f330bbe1640 1 -- 172.21.15.59:0/1989394561 shutdown_connections 2024-06-29T15:57:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.411+0000 7f330bbe1640 1 -- 172.21.15.59:0/1989394561 wait complete. 2024-06-29T15:57:02.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.413+0000 7f330bbe1640 1 Processor -- start 2024-06-29T15:57:02.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.413+0000 7f330bbe1640 1 -- start start 2024-06-29T15:57:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330bbe1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040cb490 0x7f330414d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330bbe1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33040cc060 0x7f33040c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330bbe1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f330414d8a0 0x7f33040c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330bbe1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f33041663f0 con 0x7f33040cb490 2024-06-29T15:57:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330bbe1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3304166270 con 0x7f330414d8a0 2024-06-29T15:57:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330bbe1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3304166570 con 0x7f33040cc060 2024-06-29T15:57:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040cb490 0x7f330414d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040cb490 0x7f330414d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45528/0 (socket says 172.21.15.59:45528) 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 -- 172.21.15.59:0/1415142715 learned_addr learned my addr 172.21.15.59:0/1415142715 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330a17e640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f330414d8a0 0x7f33040c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330917c640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33040cc060 0x7f33040c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 -- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33040cc060 msgr2=0x7f33040c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33040cc060 0x7f33040c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 -- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f330414d8a0 msgr2=0x7f33040c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f330414d8a0 0x7f33040c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 -- 172.21.15.59:0/1415142715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3304153be0 con 0x7f33040cb490 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330a17e640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f330414d8a0 0x7f33040c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330917c640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33040cc060 0x7f33040c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330997d640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040cb490 0x7f330414d0f0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f330000c460 tx=0x7f330000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f32fa7fc640 1 -- 172.21.15.59:0/1415142715 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300011790 con 0x7f33040cb490 2024-06-29T15:57:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f32fa7fc640 1 -- 172.21.15.59:0/1415142715 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3300017070 con 0x7f33040cb490 2024-06-29T15:57:02.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f32fa7fc640 1 -- 172.21.15.59:0/1415142715 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300011ce0 con 0x7f33040cb490 2024-06-29T15:57:02.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.415+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33040ce330 con 0x7f33040cb490 2024-06-29T15:57:02.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.416+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3304152e20 con 0x7f33040cb490 2024-06-29T15:57:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.417+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3304073be0 con 0x7f33040cb490 2024-06-29T15:57:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.417+0000 7f32fa7fc640 1 -- 172.21.15.59:0/1415142715 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 123) ==== 98434+0+0 (secure 0 0 0) 0x7f330001c460 con 0x7f33040cb490 2024-06-29T15:57:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.418+0000 7f32fa7fc640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32e8076590 0x7f32e8078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.418+0000 7f32fa7fc640 1 -- 172.21.15.59:0/1415142715 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 6588+0+0 (secure 0 0 0) 0x7f3300097d00 con 0x7f33040cb490 2024-06-29T15:57:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.418+0000 7f330917c640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32e8076590 0x7f32e8078aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.419+0000 7f330917c640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32e8076590 0x7f32e8078aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f33041f7f80 tx=0x7f32f4008040 comp rx=0 tx=0).ready entity=mgr.9128 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:02.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.422+0000 7f32fa7fc640 1 -- 172.21.15.59:0/1415142715 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f33040ce330 con 0x7f33040cb490 2024-06-29T15:57:02.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.636+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 --> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f33040cc780 con 0x7f32e8076590 2024-06-29T15:57:02.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.708+0000 7f32fa7fc640 1 -- 172.21.15.59:0/1415142715 <== mgr.9128 v2:172.21.15.153:6832/1875870697 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f33040cc780 con 0x7f32e8076590 2024-06-29T15:57:02.708 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path snap_test_dir1 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32e8076590 msgr2=0x7f32e8078aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32e8076590 0x7f32e8078aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f33041f7f80 tx=0x7f32f4008040 comp rx=0 tx=0).stop 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040cb490 msgr2=0x7f330414d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040cb490 0x7f330414d0f0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f330000c460 tx=0x7f330000c920 comp rx=0 tx=0).stop 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 shutdown_connections 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32e8076590 0x7f32e8078aa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f330414d8a0 0x7f33040c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33040cc060 0x7f33040c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 --2- 172.21.15.59:0/1415142715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33040cb490 0x7f330414d0f0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.710+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 >> 172.21.15.59:0/1415142715 conn(0x7f33040b1ea0 msgr2=0x7f33040cca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.711+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 shutdown_connections 2024-06-29T15:57:02.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:02.711+0000 7f330bbe1640 1 -- 172.21.15.59:0/1415142715 wait complete. 2024-06-29T15:57:02.721 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T15:57:02.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:02.722 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:57:02.738 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-15_57_00_UTC 2024-06-29T15:57:02.738 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_57_00_UTC 2024-06-29T15:57:02.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:02.739 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_57_00_UTC) 2024-06-29T15:57:02.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:02.799 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T15:57:02.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:57:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.018+0000 7f97bf826640 1 -- 172.21.15.59:0/1070769188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a0004000 msgr2=0x7f97a0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.018+0000 7f97bf826640 1 --2- 172.21.15.59:0/1070769188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97a0004000 0x7f97a0004800 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f97ac001740 tx=0x7f97a0005830 comp rx=0 tx=0).stop 2024-06-29T15:57:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.018+0000 7f97bddc3640 1 -- 172.21.15.59:0/1070769188 reap_dead start 2024-06-29T15:57:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.018+0000 7f97bf826640 1 -- 172.21.15.59:0/1070769188 shutdown_connections 2024-06-29T15:57:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.018+0000 7f97bf826640 1 -- 172.21.15.59:0/1070769188 >> 172.21.15.59:0/1070769188 conn(0x7f97b80b06e0 msgr2=0x7f97b80b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.018+0000 7f97bf826640 1 -- 172.21.15.59:0/1070769188 shutdown_connections 2024-06-29T15:57:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.018+0000 7f97bf826640 1 -- 172.21.15.59:0/1070769188 wait complete. 2024-06-29T15:57:03.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.020+0000 7f97bf826640 1 Processor -- start 2024-06-29T15:57:03.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.020+0000 7f97bf826640 1 -- start start 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bf826640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a0004000 0x7f97b8153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bf826640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97b8141120 0x7f97b8141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bf826640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97b8140430 0x7f97b8140840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bf826640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97b80b1e90 con 0x7f97b8141120 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bf826640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97b80b1d10 con 0x7f97a0004000 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bf826640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f97b80cf8e0 con 0x7f97b8140430 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bcdc1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97b8141120 0x7f97b8141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bcdc1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97b8141120 0x7f97b8141530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51628/0 (socket says 172.21.15.59:51628) 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bcdc1640 1 -- 172.21.15.59:0/1683720528 learned_addr learned my addr 172.21.15.59:0/1683720528 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bd5c2640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a0004000 0x7f97b8153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bddc3640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97b8140430 0x7f97b8140840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:03.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bcdc1640 1 -- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97b8140430 msgr2=0x7f97b8140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bcdc1640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97b8140430 0x7f97b8140840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bcdc1640 1 -- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a0004000 msgr2=0x7f97b8153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bcdc1640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a0004000 0x7f97b8153f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bcdc1640 1 -- 172.21.15.59:0/1683720528 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b8140d80 con 0x7f97b8141120 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.021+0000 7f97bddc3640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97b8140430 0x7f97b8140840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.022+0000 7f97bcdc1640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97b8141120 0x7f97b8141530 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f97ac000fb0 tx=0x7f97ac006190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.022+0000 7f97b5ffb640 1 -- 172.21.15.59:0/1683720528 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac037040 con 0x7f97b8141120 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.022+0000 7f97b5ffb640 1 -- 172.21.15.59:0/1683720528 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97ac0037f0 con 0x7f97b8141120 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.022+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b81e6a40 con 0x7f97b8141120 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.022+0000 7f97b5ffb640 1 -- 172.21.15.59:0/1683720528 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac003c20 con 0x7f97b8141120 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.022+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b813bc30 con 0x7f97b8141120 2024-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.023+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b806d8e0 con 0x7f97b8141120 2024-06-29T15:57:03.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.024+0000 7f97b5ffb640 1 -- 172.21.15.59:0/1683720528 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 123) ==== 98434+0+0 (secure 0 0 0) 0x7f97ac032430 con 0x7f97b8141120 2024-06-29T15:57:03.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.024+0000 7f97b5ffb640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f97a4076560 0x7f97a4078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:03.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.024+0000 7f97b5ffb640 1 -- 172.21.15.59:0/1683720528 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 6588+0+0 (secure 0 0 0) 0x7f97ac0b83a0 con 0x7f97b8141120 2024-06-29T15:57:03.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.025+0000 7f97bd5c2640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f97a4076560 0x7f97a4078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:03.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.025+0000 7f97bd5c2640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f97a4076560 0x7f97a4078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009b20 tx=0x7f97b0005f90 comp rx=0 tx=0).ready entity=mgr.9128 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:03.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.029+0000 7f97b5ffb640 1 -- 172.21.15.59:0/1683720528 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f97b8140d80 con 0x7f97b8141120 2024-06-29T15:57:03.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:03.271+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f97b806d600 con 0x7f97b8141120 2024-06-29T15:57:04.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.080+0000 7f97b5ffb640 1 -- 172.21.15.59:0/1683720528 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v124) ==== 93+0+0 (secure 0 0 0) 0x7f97ac081570 con 0x7f97b8141120 2024-06-29T15:57:04.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.086+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f97a4076560 msgr2=0x7f97a4078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.086+0000 7f97bf826640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f97a4076560 0x7f97a4078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009b20 tx=0x7f97b0005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:04.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.086+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97b8141120 msgr2=0x7f97b8141530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97b5ffb640 1 -- 172.21.15.59:0/1683720528 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f97ac07d100 con 0x7f97b8141120 2024-06-29T15:57:04.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97bf826640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97b8141120 0x7f97b8141530 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f97ac000fb0 tx=0x7f97ac006190 comp rx=0 tx=0).stop 2024-06-29T15:57:04.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 shutdown_connections 2024-06-29T15:57:04.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97bf826640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f97a4076560 0x7f97a4078a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97bf826640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97b8140430 0x7f97b8140840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97bf826640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97b8141120 0x7f97b8141530 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97bf826640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97a0004000 0x7f97b8153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 >> 172.21.15.59:0/1683720528 conn(0x7f97b80b06e0 msgr2=0x7f97b8070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.088+0000 7f97b5ffb640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f97a40394e0 0x7f97a403b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.089+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 shutdown_connections 2024-06-29T15:57:04.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.089+0000 7f97bf826640 1 --2- 172.21.15.59:0/1683720528 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f97a40394e0 0x7f97a403b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.089+0000 7f97bf826640 1 -- 172.21.15.59:0/1683720528 wait complete. 2024-06-29T15:57:04.104 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:57:04.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:04.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:04.190 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:57:04.190 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:04.190 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:04.191 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:04.208 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:04.208 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:04.266 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:04.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.425+0000 7fa1fbece640 1 -- 172.21.15.59:0/304599149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40b2270 msgr2=0x7fa1f415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.425+0000 7fa1fbece640 1 --2- 172.21.15.59:0/304599149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40b2270 0x7fa1f415bc20 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8009830 tx=0x7fa1e802a5c0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.425+0000 7fa1fbece640 1 -- 172.21.15.59:0/304599149 shutdown_connections 2024-06-29T15:57:04.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.425+0000 7fa1fbece640 1 --2- 172.21.15.59:0/304599149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40b2270 0x7fa1f415bc20 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.425+0000 7fa1fbece640 1 -- 172.21.15.59:0/304599149 >> 172.21.15.59:0/304599149 conn(0x7fa1f4073e10 msgr2=0x7fa1f4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.425+0000 7fa1fbece640 1 -- 172.21.15.59:0/304599149 shutdown_connections 2024-06-29T15:57:04.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.425+0000 7fa1fbece640 1 -- 172.21.15.59:0/304599149 wait complete. 2024-06-29T15:57:04.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.427+0000 7fa1fbece640 1 Processor -- start 2024-06-29T15:57:04.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.427+0000 7fa1fbece640 1 -- start start 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.428+0000 7fa1fbece640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1f40b2270 0x7fa1f414d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.428+0000 7fa1fbece640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1f40b2fa0 0x7fa1f406ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.428+0000 7fa1fbece640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40d1c30 0x7fa1f40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.428+0000 7fa1fbece640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f40b65d0 con 0x7fa1f40d1c30 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.428+0000 7fa1fbece640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f40b6450 con 0x7fa1f40b2270 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.428+0000 7fa1fbece640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f40b6750 con 0x7fa1f40b2fa0 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40d1c30 0x7fa1f40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40d1c30 0x7fa1f40ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51676/0 (socket says 172.21.15.59:51676) 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 -- 172.21.15.59:0/1163943496 learned_addr learned my addr 172.21.15.59:0/1163943496 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1f9c6a640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1f40b2270 0x7fa1f414d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1f9469640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1f40b2fa0 0x7fa1f406ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 -- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1f40b2fa0 msgr2=0x7fa1f406ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1f40b2fa0 0x7fa1f406ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 -- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1f40b2270 msgr2=0x7fa1f414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1f40b2270 0x7fa1f414d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 -- 172.21.15.59:0/1163943496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f414dce0 con 0x7fa1f40d1c30 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fa46b640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40d1c30 0x7fa1f40ad8b0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00c730 tx=0x7fa1ec00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1e67fc640 1 -- 172.21.15.59:0/1163943496 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec003b70 con 0x7fa1f40d1c30 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1e67fc640 1 -- 172.21.15.59:0/1163943496 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1ec003d10 con 0x7fa1f40d1c30 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f406f7a0 con 0x7fa1f40d1c30 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.429+0000 7fa1e67fc640 1 -- 172.21.15.59:0/1163943496 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec00aab0 con 0x7fa1f40d1c30 2024-06-29T15:57:04.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.430+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f415da60 con 0x7fa1f40d1c30 2024-06-29T15:57:04.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.431+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1f40b6c70 con 0x7fa1f40d1c30 2024-06-29T15:57:04.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.431+0000 7fa1e67fc640 1 -- 172.21.15.59:0/1163943496 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7fa1ec00ac50 con 0x7fa1f40d1c30 2024-06-29T15:57:04.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.432+0000 7fa1e67fc640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fa1d8076510 0x7fa1d8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.432+0000 7fa1e67fc640 1 -- 172.21.15.59:0/1163943496 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 6676+0+0 (secure 0 0 0) 0x7fa1ec097d00 con 0x7fa1f40d1c30 2024-06-29T15:57:04.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.432+0000 7fa1f9c6a640 1 -- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fa1d8076510 msgr2=0x7fa1d8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:04.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.432+0000 7fa1f9c6a640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fa1d8076510 0x7fa1d8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:04.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.436+0000 7fa1e67fc640 1 -- 172.21.15.59:0/1163943496 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa1ec060860 con 0x7fa1f40d1c30 2024-06-29T15:57:04.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.633+0000 7fa1f9c6a640 1 -- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fa1d8076510 msgr2=0x7fa1d8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:04.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.634+0000 7fa1f9c6a640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fa1d8076510 0x7fa1d8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:04.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.653+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1f406f4c0 con 0x7fa1f40d1c30 2024-06-29T15:57:04.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.654+0000 7fa1e67fc640 1 -- 172.21.15.59:0/1163943496 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) ==== 74+0+17795 (secure 0 0 0) 0x7fa1ec064b60 con 0x7fa1f40d1c30 2024-06-29T15:57:04.655 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:04.655 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":208,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:04.071594+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.935517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.518490+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":208,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.2c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2009747699":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/2580944857":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/4008855742":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6833/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6832/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/719772475":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/4087971292":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/3198120249":"2024-06-30T15:55:49.819811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.656+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fa1d8076510 msgr2=0x7fa1d8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.656+0000 7fa1fbece640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fa1d8076510 0x7fa1d8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.656+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40d1c30 msgr2=0x7fa1f40ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.656+0000 7fa1fbece640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40d1c30 0x7fa1f40ad8b0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00c730 tx=0x7fa1ec00cbf0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.657+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 shutdown_connections 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.657+0000 7fa1fbece640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fa1d8076510 0x7fa1d8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.657+0000 7fa1fbece640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1f40d1c30 0x7fa1f40ad8b0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.657+0000 7fa1fbece640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1f40b2fa0 0x7fa1f406ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.657+0000 7fa1fbece640 1 --2- 172.21.15.59:0/1163943496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1f40b2270 0x7fa1f414d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.657+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 >> 172.21.15.59:0/1163943496 conn(0x7fa1f4073e10 msgr2=0x7fa1f40709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.657+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 shutdown_connections 2024-06-29T15:57:04.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.657+0000 7fa1fbece640 1 -- 172.21.15.59:0/1163943496 wait complete. 2024-06-29T15:57:04.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:55:35.935517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '203', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:55:36.518490+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 208, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:04.669 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:04.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 -- 172.21.15.59:0/1957568992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f40d23c0 msgr2=0x7f99f40b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 --2- 172.21.15.59:0/1957568992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f40d23c0 0x7f99f40b3990 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f99f40d04f0 tx=0x7f99e802a560 comp rx=0 tx=0).stop 2024-06-29T15:57:04.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 -- 172.21.15.59:0/1957568992 shutdown_connections 2024-06-29T15:57:04.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 --2- 172.21.15.59:0/1957568992 >> v2:172.21.15.153:3300/0 conn(0x7f99f415ee10 0x7f99f415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 --2- 172.21.15.59:0/1957568992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99f40b3ed0 0x7f99f415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 --2- 172.21.15.59:0/1957568992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f40d23c0 0x7f99f40b3990 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 -- 172.21.15.59:0/1957568992 >> 172.21.15.59:0/1957568992 conn(0x7f99f40d3ee0 msgr2=0x7f99f40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 -- 172.21.15.59:0/1957568992 shutdown_connections 2024-06-29T15:57:04.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.791+0000 7f99fc038640 1 -- 172.21.15.59:0/1957568992 wait complete. 2024-06-29T15:57:04.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.794+0000 7f99fc038640 1 Processor -- start 2024-06-29T15:57:04.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.794+0000 7f99fc038640 1 -- start start 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.796+0000 7f99fc038640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99f40b3ed0 0x7f99f40ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.796+0000 7f99fc038640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99f40d23c0 0x7f99f415ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fc038640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f415ee10 0x7f99f415d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fc038640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f99f4160070 con 0x7f99f415ee10 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fc038640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f99f415fef0 con 0x7f99f40d23c0 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fc038640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f99f41601f0 con 0x7f99f40b3ed0 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f415ee10 0x7f99f415d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f415ee10 0x7f99f415d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51684/0 (socket says 172.21.15.59:51684) 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 -- 172.21.15.59:0/3401383076 learned_addr learned my addr 172.21.15.59:0/3401383076 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99f95d3640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99f40d23c0 0x7f99f415ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99f9dd4640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99f40b3ed0 0x7f99f40ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 -- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99f40b3ed0 msgr2=0x7f99f40ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99f40b3ed0 0x7f99f40ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 -- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99f40d23c0 msgr2=0x7f99f415ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99f40d23c0 0x7f99f415ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 -- 172.21.15.59:0/3401383076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f41e87c0 con 0x7f99f415ee10 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99fa5d5640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f415ee10 0x7f99f415d3d0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00ea80 tx=0x7f99ec00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.797+0000 7f99e67fc640 1 -- 172.21.15.59:0/3401383076 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec004b20 con 0x7f99f415ee10 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.798+0000 7f99e67fc640 1 -- 172.21.15.59:0/3401383076 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99ec004ce0 con 0x7f99f415ee10 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.798+0000 7f99e67fc640 1 -- 172.21.15.59:0/3401383076 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec005280 con 0x7f99f415ee10 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.798+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f41566a0 con 0x7f99f415ee10 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.798+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f415def0 con 0x7f99f415ee10 2024-06-29T15:57:04.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.799+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99f40b3990 con 0x7f99f415ee10 2024-06-29T15:57:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.799+0000 7f99e67fc640 1 -- 172.21.15.59:0/3401383076 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f99ec005420 con 0x7f99f415ee10 2024-06-29T15:57:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.800+0000 7f99e67fc640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f99d8076510 0x7f99d8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.800+0000 7f99e67fc640 1 -- 172.21.15.59:0/3401383076 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 6676+0+0 (secure 0 0 0) 0x7f99ec098be0 con 0x7f99f415ee10 2024-06-29T15:57:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.800+0000 7f99f9dd4640 1 -- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f99d8076510 msgr2=0x7f99d8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:04.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.800+0000 7f99f9dd4640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f99d8076510 0x7f99d8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:04.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.804+0000 7f99e67fc640 1 -- 172.21.15.59:0/3401383076 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99ec061620 con 0x7f99f415ee10 2024-06-29T15:57:04.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.994+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f99f40d2950 con 0x7f99f415ee10 2024-06-29T15:57:04.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.995+0000 7f99e67fc640 1 -- 172.21.15.59:0/3401383076 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v208) ==== 93+0+10 (secure 0 0 0) 0x7f99ec065920 con 0x7f99f415ee10 2024-06-29T15:57:04.996 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f99d8076510 msgr2=0x7f99d8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f99d8076510 0x7f99d8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f415ee10 msgr2=0x7f99f415d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f415ee10 0x7f99f415d3d0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00ea80 tx=0x7f99ec00ef40 comp rx=0 tx=0).stop 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 shutdown_connections 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f99d8076510 0x7f99d8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99f415ee10 0x7f99f415d3d0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99f40d23c0 0x7f99f415ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 --2- 172.21.15.59:0/3401383076 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f99f40b3ed0 0x7f99f40ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 >> 172.21.15.59:0/3401383076 conn(0x7f99f40d3ee0 msgr2=0x7f99f4160730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 shutdown_connections 2024-06-29T15:57:04.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:04.997+0000 7f99fc038640 1 -- 172.21.15.59:0/3401383076 wait complete. 2024-06-29T15:57:05.009 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:05.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.136+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/3080277678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480d23c0 msgr2=0x7f2c480b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.136+0000 7f2c4f2ce640 1 --2- 172.21.15.59:0/3080277678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480d23c0 0x7f2c480b3990 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009830 tx=0x7f2c4402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.136+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/3080277678 shutdown_connections 2024-06-29T15:57:05.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.136+0000 7f2c4f2ce640 1 --2- 172.21.15.59:0/3080277678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480d23c0 0x7f2c480b3990 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.136+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/3080277678 >> 172.21.15.59:0/3080277678 conn(0x7f2c480d3ee0 msgr2=0x7f2c480d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.136+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/3080277678 shutdown_connections 2024-06-29T15:57:05.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.136+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/3080277678 wait complete. 2024-06-29T15:57:05.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.138+0000 7f2c4f2ce640 1 Processor -- start 2024-06-29T15:57:05.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.138+0000 7f2c4f2ce640 1 -- start start 2024-06-29T15:57:05.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.139+0000 7f2c4f2ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480b3ed0 0x7f2c4815a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.139+0000 7f2c4f2ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c480d23c0 0x7f2c4814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4f2ce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c4815efe0 0x7f2c48156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4f2ce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2c48160450 con 0x7f2c480b3ed0 2024-06-29T15:57:05.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4f2ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c481602d0 con 0x7f2c480d23c0 2024-06-29T15:57:05.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4f2ce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2c481605d0 con 0x7f2c4815efe0 2024-06-29T15:57:05.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480b3ed0 0x7f2c4815a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480b3ed0 0x7f2c4815a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51696/0 (socket says 172.21.15.59:51696) 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 -- 172.21.15.59:0/99045294 learned_addr learned my addr 172.21.15.59:0/99045294 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d86b640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c4815efe0 0x7f2c48156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4c869640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c480d23c0 0x7f2c4814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 -- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c4815efe0 msgr2=0x7f2c48156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c4815efe0 0x7f2c48156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 -- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c480d23c0 msgr2=0x7f2c4814e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c480d23c0 0x7f2c4814e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 -- 172.21.15.59:0/99045294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c48157cd0 con 0x7f2c480b3ed0 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c4d06a640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480b3ed0 0x7f2c4815a2d0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009950 tx=0x7f2c44033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c39ffb640 1 -- 172.21.15.59:0/99045294 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44031d60 con 0x7f2c480b3ed0 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c39ffb640 1 -- 172.21.15.59:0/99045294 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c440330d0 con 0x7f2c480b3ed0 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.140+0000 7f2c39ffb640 1 -- 172.21.15.59:0/99045294 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44033520 con 0x7f2c480b3ed0 2024-06-29T15:57:05.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.141+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c481581d0 con 0x7f2c480b3ed0 2024-06-29T15:57:05.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.141+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c48156730 con 0x7f2c480b3ed0 2024-06-29T15:57:05.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.142+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c480b3990 con 0x7f2c480b3ed0 2024-06-29T15:57:05.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.143+0000 7f2c39ffb640 1 -- 172.21.15.59:0/99045294 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f2c440035f0 con 0x7f2c480b3ed0 2024-06-29T15:57:05.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.143+0000 7f2c39ffb640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f2c2c076510 0x7f2c2c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.144+0000 7f2c4c869640 1 -- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f2c2c076510 msgr2=0x7f2c2c078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:05.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.144+0000 7f2c39ffb640 1 -- 172.21.15.59:0/99045294 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 6660+0+0 (secure 0 0 0) 0x7f2c440b9370 con 0x7f2c480b3ed0 2024-06-29T15:57:05.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.144+0000 7f2c4c869640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f2c2c076510 0x7f2c2c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:05.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.148+0000 7f2c39ffb640 1 -- 172.21.15.59:0/99045294 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2c4403f030 con 0x7f2c480b3ed0 2024-06-29T15:57:05.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.345+0000 7f2c4c869640 1 -- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f2c2c076510 msgr2=0x7f2c2c078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:05.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.345+0000 7f2c4c869640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f2c2c076510 0x7f2c2c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:05.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.346+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2c480d2950 con 0x7f2c480b3ed0 2024-06-29T15:57:05.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.347+0000 7f2c39ffb640 1 -- 172.21.15.59:0/99045294 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v209) ==== 104+0+11 (secure 0 0 0) 0x7f2c44081f20 con 0x7f2c480b3ed0 2024-06-29T15:57:05.348 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:57:05.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f2c2c076510 msgr2=0x7f2c2c078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:05.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f2c2c076510 0x7f2c2c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480b3ed0 msgr2=0x7f2c4815a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480b3ed0 0x7f2c4815a2d0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009950 tx=0x7f2c44033d60 comp rx=0 tx=0).stop 2024-06-29T15:57:05.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 shutdown_connections 2024-06-29T15:57:05.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f2c2c076510 0x7f2c2c078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c4815efe0 0x7f2c48156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c480d23c0 0x7f2c4814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 --2- 172.21.15.59:0/99045294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c480b3ed0 0x7f2c4815a2d0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 >> 172.21.15.59:0/99045294 conn(0x7f2c480d3ee0 msgr2=0x7f2c4806d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 shutdown_connections 2024-06-29T15:57:05.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.349+0000 7f2c4f2ce640 1 -- 172.21.15.59:0/99045294 wait complete. 2024-06-29T15:57:05.362 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:05.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.483+0000 7f2111881640 1 -- 172.21.15.59:0/857195843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c0cd7c0 msgr2=0x7f210c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.483+0000 7f2111881640 1 --2- 172.21.15.59:0/857195843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c0cd7c0 0x7f210c06ca70 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f210c0ccee0 tx=0x7f210402aac0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.483+0000 7f2111881640 1 -- 172.21.15.59:0/857195843 shutdown_connections 2024-06-29T15:57:05.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.483+0000 7f2111881640 1 --2- 172.21.15.59:0/857195843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c0cd7c0 0x7f210c06ca70 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.483+0000 7f2111881640 1 --2- 172.21.15.59:0/857195843 >> v2:172.21.15.153:3300/0 conn(0x7f210c0d02f0 0x7f210c0cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.483+0000 7f2111881640 1 -- 172.21.15.59:0/857195843 >> 172.21.15.59:0/857195843 conn(0x7f210c0b2ab0 msgr2=0x7f210c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.483+0000 7f2111881640 1 -- 172.21.15.59:0/857195843 shutdown_connections 2024-06-29T15:57:05.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.483+0000 7f2111881640 1 -- 172.21.15.59:0/857195843 wait complete. 2024-06-29T15:57:05.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.485+0000 7f2111881640 1 Processor -- start 2024-06-29T15:57:05.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.485+0000 7f2111881640 1 -- start start 2024-06-29T15:57:05.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.486+0000 7f2111881640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f210c0cd7c0 0x7f210c12eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.486+0000 7f2111881640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c1448e0 0x7f210c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.486+0000 7f2111881640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f210c151e40 0x7f210c13fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.486+0000 7f2111881640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f210c06d450 con 0x7f210c1448e0 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.486+0000 7f2111881640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f210c06d2d0 con 0x7f210c151e40 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.486+0000 7f2111881640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f210c06d5d0 con 0x7f210c0cd7c0 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.486+0000 7f210a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c1448e0 0x7f210c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.486+0000 7f210a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c1448e0 0x7f210c154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51712/0 (socket says 172.21.15.59:51712) 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210a7fc640 1 -- 172.21.15.59:0/813150700 learned_addr learned my addr 172.21.15.59:0/813150700 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210b7fe640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f210c151e40 0x7f210c13fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210affd640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f210c0cd7c0 0x7f210c12eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210a7fc640 1 -- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f210c0cd7c0 msgr2=0x7f210c12eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210a7fc640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f210c0cd7c0 0x7f210c12eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210a7fc640 1 -- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f210c151e40 msgr2=0x7f210c13fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210a7fc640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f210c151e40 0x7f210c13fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210a7fc640 1 -- 172.21.15.59:0/813150700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210c1f2850 con 0x7f210c1448e0 2024-06-29T15:57:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f210a7fc640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c1448e0 0x7f210c154b40 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f20f400e7c0 tx=0x7f20f400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f2103fff640 1 -- 172.21.15.59:0/813150700 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f400ccc0 con 0x7f210c1448e0 2024-06-29T15:57:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f2103fff640 1 -- 172.21.15.59:0/813150700 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f4010430 con 0x7f210c1448e0 2024-06-29T15:57:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f210c153aa0 con 0x7f210c1448e0 2024-06-29T15:57:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f2103fff640 1 -- 172.21.15.59:0/813150700 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4010860 con 0x7f210c1448e0 2024-06-29T15:57:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.487+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f210c154060 con 0x7f210c1448e0 2024-06-29T15:57:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.488+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f210c06daf0 con 0x7f210c1448e0 2024-06-29T15:57:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.489+0000 7f2103fff640 1 -- 172.21.15.59:0/813150700 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f20f4010a00 con 0x7f210c1448e0 2024-06-29T15:57:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.490+0000 7f2103fff640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f20ec076510 0x7f20ec078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.490+0000 7f210affd640 1 -- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f20ec076510 msgr2=0x7f20ec078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.490+0000 7f2103fff640 1 -- 172.21.15.59:0/813150700 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 6660+0+0 (secure 0 0 0) 0x7f20f4098b10 con 0x7f210c1448e0 2024-06-29T15:57:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.490+0000 7f210affd640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f20ec076510 0x7f20ec078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.495+0000 7f2103fff640 1 -- 172.21.15.59:0/813150700 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f20f4061440 con 0x7f210c1448e0 2024-06-29T15:57:05.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.687+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f210c15f690 con 0x7f210c1448e0 2024-06-29T15:57:05.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.687+0000 7f2103fff640 1 -- 172.21.15.59:0/813150700 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v209) ==== 100+0+11 (secure 0 0 0) 0x7f20f4065740 con 0x7f210c1448e0 2024-06-29T15:57:05.689 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:57:05.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f20ec076510 msgr2=0x7f20ec078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:05.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f20ec076510 0x7f20ec078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c1448e0 msgr2=0x7f210c154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c1448e0 0x7f210c154b40 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f20f400e7c0 tx=0x7f20f400ec80 comp rx=0 tx=0).stop 2024-06-29T15:57:05.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 shutdown_connections 2024-06-29T15:57:05.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f20ec076510 0x7f20ec078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f210c151e40 0x7f210c13fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f210c1448e0 0x7f210c154b40 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 --2- 172.21.15.59:0/813150700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f210c0cd7c0 0x7f210c12eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 >> 172.21.15.59:0/813150700 conn(0x7f210c0b2ab0 msgr2=0x7f210c144e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 shutdown_connections 2024-06-29T15:57:05.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.690+0000 7f2111881640 1 -- 172.21.15.59:0/813150700 wait complete. 2024-06-29T15:57:05.703 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:57:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.822+0000 7fbffc809640 1 -- 172.21.15.59:0/2425061032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 msgr2=0x7fbff80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.822+0000 7fbffc809640 1 --2- 172.21.15.59:0/2425061032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 0x7fbff80b3990 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fbff8161d40 tx=0x7fbfe802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.822+0000 7fbffc809640 1 -- 172.21.15.59:0/2425061032 shutdown_connections 2024-06-29T15:57:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.822+0000 7fbffc809640 1 --2- 172.21.15.59:0/2425061032 >> v2:172.21.15.153:3300/0 conn(0x7fbff815efe0 0x7fbff815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.822+0000 7fbffc809640 1 --2- 172.21.15.59:0/2425061032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 0x7fbff80b3990 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.822+0000 7fbffc809640 1 -- 172.21.15.59:0/2425061032 >> 172.21.15.59:0/2425061032 conn(0x7fbff80d3ee0 msgr2=0x7fbff80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.822+0000 7fbffc809640 1 -- 172.21.15.59:0/2425061032 shutdown_connections 2024-06-29T15:57:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.822+0000 7fbffc809640 1 -- 172.21.15.59:0/2425061032 wait complete. 2024-06-29T15:57:05.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.824+0000 7fbffc809640 1 Processor -- start 2024-06-29T15:57:05.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.824+0000 7fbffc809640 1 -- start start 2024-06-29T15:57:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.825+0000 7fbffc809640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff80b3ed0 0x7fbff815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.825+0000 7fbffc809640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 0x7fbff80c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.825+0000 7fbffc809640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff815efe0 0x7fbff815cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.825+0000 7fbffc809640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8160280 con 0x7fbff80d23c0 2024-06-29T15:57:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.825+0000 7fbffc809640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8160100 con 0x7fbff815efe0 2024-06-29T15:57:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.825+0000 7fbffc809640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbff8160400 con 0x7fbff80b3ed0 2024-06-29T15:57:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 0x7fbff80c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff6d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff815efe0 0x7fbff815cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 0x7fbff80c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51738/0 (socket says 172.21.15.59:51738) 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 -- 172.21.15.59:0/247650485 learned_addr learned my addr 172.21.15.59:0/247650485 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff659c640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff80b3ed0 0x7fbff815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 -- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff80b3ed0 msgr2=0x7fbff815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff80b3ed0 0x7fbff815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 -- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff815efe0 msgr2=0x7fbff815cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff815efe0 0x7fbff815cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 -- 172.21.15.59:0/247650485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff815c660 con 0x7fbff80d23c0 2024-06-29T15:57:05.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff6d9d640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff815efe0 0x7fbff815cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:05.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbff5d9b640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 0x7fbff80c6c50 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00c3a0 tx=0x7fbfec00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:05.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbfe6ffd640 1 -- 172.21.15.59:0/247650485 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec017070 con 0x7fbff80d23c0 2024-06-29T15:57:05.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbfe6ffd640 1 -- 172.21.15.59:0/247650485 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfec0053c0 con 0x7fbff80d23c0 2024-06-29T15:57:05.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.826+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff815b080 con 0x7fbff80d23c0 2024-06-29T15:57:05.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.827+0000 7fbfe6ffd640 1 -- 172.21.15.59:0/247650485 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec003850 con 0x7fbff80d23c0 2024-06-29T15:57:05.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.827+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff815b640 con 0x7fbff80d23c0 2024-06-29T15:57:05.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.828+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff815add0 con 0x7fbff80d23c0 2024-06-29T15:57:05.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.828+0000 7fbfe6ffd640 1 -- 172.21.15.59:0/247650485 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7fbfec01a070 con 0x7fbff80d23c0 2024-06-29T15:57:05.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.829+0000 7fbfe6ffd640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbfd8076510 0x7fbfd8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.829+0000 7fbff659c640 1 -- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbfd8076510 msgr2=0x7fbfd8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:05.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.829+0000 7fbfe6ffd640 1 -- 172.21.15.59:0/247650485 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 6660+0+0 (secure 0 0 0) 0x7fbfec097240 con 0x7fbff80d23c0 2024-06-29T15:57:05.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.829+0000 7fbff659c640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbfd8076510 0x7fbfd8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:05.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:05.833+0000 7fbfe6ffd640 1 -- 172.21.15.59:0/247650485 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbff80b42e0 con 0x7fbff80d23c0 2024-06-29T15:57:06.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.030+0000 7fbff659c640 1 -- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbfd8076510 msgr2=0x7fbfd8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:06.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.030+0000 7fbff659c640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbfd8076510 0x7fbfd8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.037+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbff80d2950 con 0x7fbff80d23c0 2024-06-29T15:57:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.038+0000 7fbfe6ffd640 1 -- 172.21.15.59:0/247650485 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v209) ==== 80+0+475 (secure 0 0 0) 0x7fbfec064170 con 0x7fbff80d23c0 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2009747699 2024-06-30T15:55:49.819811+0000 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2580944857 2024-06-30T15:55:49.819811+0000 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/4008855742 2024-06-30T15:55:49.819811+0000 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/493926731 2024-06-30T15:55:49.819811+0000 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/493926731 2024-06-30T15:55:49.819811+0000 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/719772475 2024-06-30T15:55:49.819811+0000 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/4087971292 2024-06-30T15:55:49.819811+0000 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3198120249 2024-06-30T15:55:49.819811+0000 2024-06-29T15:57:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbfd8076510 msgr2=0x7fbfd8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbfd8076510 0x7fbfd8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 msgr2=0x7fbff80c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 0x7fbff80c6c50 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00c3a0 tx=0x7fbfec00c860 comp rx=0 tx=0).stop 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 shutdown_connections 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbfd8076510 0x7fbfd8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbff815efe0 0x7fbff815cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbff80d23c0 0x7fbff80c6c50 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 --2- 172.21.15.59:0/247650485 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbff80b3ed0 0x7fbff815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 >> 172.21.15.59:0/247650485 conn(0x7fbff80d3ee0 msgr2=0x7fbff806caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 shutdown_connections 2024-06-29T15:57:06.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.040+0000 7fbffc809640 1 -- 172.21.15.59:0/247650485 wait complete. 2024-06-29T15:57:06.052 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:57:06.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:06.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:06.082 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T15:57:06.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:06.083 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:57:06.139 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:06.139 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:57:06.139 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:06.155 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:06.155 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:06.155 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:06.155 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:06.195 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:57:06.196 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:06.196 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:57:06.196 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:06.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:06.213 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:06.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.353+0000 7fb19a88f640 1 -- 172.21.15.59:0/979969961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb17c003a70 msgr2=0x7fb17c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.353+0000 7fb19a88f640 1 --2- 172.21.15.59:0/979969961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb17c003a70 0x7fb17c003ee0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb184009b50 tx=0x7fb17c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:06.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.353+0000 7fb198e2c640 1 -- 172.21.15.59:0/979969961 reap_dead start 2024-06-29T15:57:06.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.354+0000 7fb19a88f640 1 -- 172.21.15.59:0/979969961 shutdown_connections 2024-06-29T15:57:06.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.354+0000 7fb19a88f640 1 -- 172.21.15.59:0/979969961 >> 172.21.15.59:0/979969961 conn(0x7fb194073e10 msgr2=0x7fb194074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.354+0000 7fb19a88f640 1 -- 172.21.15.59:0/979969961 shutdown_connections 2024-06-29T15:57:06.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.354+0000 7fb19a88f640 1 -- 172.21.15.59:0/979969961 wait complete. 2024-06-29T15:57:06.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.355+0000 7fb19a88f640 1 Processor -- start 2024-06-29T15:57:06.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.356+0000 7fb19a88f640 1 -- start start 2024-06-29T15:57:06.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb19a88f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb17c003a70 0x7fb19415eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb19a88f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940a94a0 0x7fb1940abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb19a88f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940a98b0 0x7fb1940abff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb19a88f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb19415bc20 con 0x7fb1940a94a0 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb19a88f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1940b6d50 con 0x7fb17c003a70 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb19a88f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb19415bda0 con 0x7fb1940a98b0 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb1937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940a94a0 0x7fb1940abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb1937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940a94a0 0x7fb1940abab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51762/0 (socket says 172.21.15.59:51762) 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb1937fe640 1 -- 172.21.15.59:0/2888005888 learned_addr learned my addr 172.21.15.59:0/2888005888 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb198e2c640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940a98b0 0x7fb1940abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb193fff640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb17c003a70 0x7fb19415eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb1937fe640 1 -- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940a98b0 msgr2=0x7fb1940abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb1937fe640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940a98b0 0x7fb1940abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb1937fe640 1 -- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb17c003a70 msgr2=0x7fb19415eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb1937fe640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb17c003a70 0x7fb19415eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.357+0000 7fb1937fe640 1 -- 172.21.15.59:0/2888005888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19415f2e0 con 0x7fb1940a94a0 2024-06-29T15:57:06.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.358+0000 7fb1937fe640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940a94a0 0x7fb1940abab0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fb188011860 tx=0x7fb188011c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:06.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.358+0000 7fb190ff9640 1 -- 172.21.15.59:0/2888005888 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18801e070 con 0x7fb1940a94a0 2024-06-29T15:57:06.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.358+0000 7fb190ff9640 1 -- 172.21.15.59:0/2888005888 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb18800fd30 con 0x7fb1940a94a0 2024-06-29T15:57:06.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.358+0000 7fb190ff9640 1 -- 172.21.15.59:0/2888005888 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188019690 con 0x7fb1940a94a0 2024-06-29T15:57:06.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.358+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1940aa010 con 0x7fb1940a94a0 2024-06-29T15:57:06.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.358+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1940aa5d0 con 0x7fb1940a94a0 2024-06-29T15:57:06.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.359+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb19415e670 con 0x7fb1940a94a0 2024-06-29T15:57:06.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.360+0000 7fb190ff9640 1 -- 172.21.15.59:0/2888005888 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7fb188021070 con 0x7fb1940a94a0 2024-06-29T15:57:06.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.360+0000 7fb190ff9640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fb180076510 0x7fb180078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.361+0000 7fb193fff640 1 -- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fb180076510 msgr2=0x7fb180078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:06.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.361+0000 7fb190ff9640 1 -- 172.21.15.59:0/2888005888 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 6660+0+0 (secure 0 0 0) 0x7fb188098a60 con 0x7fb1940a94a0 2024-06-29T15:57:06.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.361+0000 7fb193fff640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fb180076510 0x7fb180078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.365+0000 7fb190ff9640 1 -- 172.21.15.59:0/2888005888 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb188061430 con 0x7fb1940a94a0 2024-06-29T15:57:06.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.562+0000 7fb193fff640 1 -- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fb180076510 msgr2=0x7fb180078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:06.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.562+0000 7fb193fff640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fb180076510 0x7fb180078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:06.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.598+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1940a9d30 con 0x7fb1940a94a0 2024-06-29T15:57:06.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.598+0000 7fb190ff9640 1 -- 172.21.15.59:0/2888005888 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5246 (secure 0 0 0) 0x7fb188065730 con 0x7fb1940a94a0 2024-06-29T15:57:06.599 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:06.599 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"btime":"2024-06-29T15:55:41:198021+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3297476433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3297476433},{"type":"v1","addr":"172.21.15.153:6837","nonce":3297476433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":10974,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3987613682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3987613682},{"type":"v1","addr":"172.21.15.59:6834","nonce":3987613682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":10977,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2501188073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2501188073},{"type":"v1","addr":"172.21.15.59:6835","nonce":2501188073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.567350+0000","modified":"2024-06-29T15:55:40.592165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10983},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10983":{"gid":10983,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2163190416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2163190416},{"type":"v1","addr":"172.21.15.153:6836","nonce":2163190416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10983,"qdb_cluster":[10983]},"id":12}]} 2024-06-29T15:57:06.600 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-06-29T15:57:06.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fb180076510 msgr2=0x7fb180078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:06.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fb180076510 0x7fb180078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940a94a0 msgr2=0x7fb1940abab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940a94a0 0x7fb1940abab0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fb188011860 tx=0x7fb188011c60 comp rx=0 tx=0).stop 2024-06-29T15:57:06.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 shutdown_connections 2024-06-29T15:57:06.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fb180076510 0x7fb180078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940a98b0 0x7fb1940abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940a94a0 0x7fb1940abab0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 --2- 172.21.15.59:0/2888005888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb17c003a70 0x7fb19415eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 >> 172.21.15.59:0/2888005888 conn(0x7fb194073e10 msgr2=0x7fb1940d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 shutdown_connections 2024-06-29T15:57:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.601+0000 7fb19a88f640 1 -- 172.21.15.59:0/2888005888 wait complete. 2024-06-29T15:57:06.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:06.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.731+0000 7fbee4911640 1 -- 172.21.15.59:0/1268009743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00d1c30 msgr2=0x7fbee00b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.731+0000 7fbee4911640 1 --2- 172.21.15.59:0/1268009743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00d1c30 0x7fbee00b5b30 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fbed8005fd0 tx=0x7fbed802aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:06.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.731+0000 7fbee4911640 1 -- 172.21.15.59:0/1268009743 shutdown_connections 2024-06-29T15:57:06.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.731+0000 7fbee4911640 1 --2- 172.21.15.59:0/1268009743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00d1c30 0x7fbee00b5b30 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.731+0000 7fbee4911640 1 --2- 172.21.15.59:0/1268009743 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbee00b2270 0x7fbee015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.731+0000 7fbee4911640 1 -- 172.21.15.59:0/1268009743 >> 172.21.15.59:0/1268009743 conn(0x7fbee0073e10 msgr2=0x7fbee0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.731+0000 7fbee4911640 1 -- 172.21.15.59:0/1268009743 shutdown_connections 2024-06-29T15:57:06.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.731+0000 7fbee4911640 1 -- 172.21.15.59:0/1268009743 wait complete. 2024-06-29T15:57:06.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.733+0000 7fbee4911640 1 Processor -- start 2024-06-29T15:57:06.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.733+0000 7fbee4911640 1 -- start start 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.734+0000 7fbee4911640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbee00b2270 0x7fbee015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.734+0000 7fbee4911640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00b2fa0 0x7fbee0146f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbee4911640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbee00d1c30 0x7fbee006f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbee4911640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbee00b6400 con 0x7fbee00b2fa0 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbee4911640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbee00b6280 con 0x7fbee00d1c30 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbee4911640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbee00b6580 con 0x7fbee00b2270 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00b2fa0 0x7fbee0146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00b2fa0 0x7fbee0146f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51776/0 (socket says 172.21.15.59:51776) 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbede59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbee00b2270 0x7fbee015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 -- 172.21.15.59:0/2349986495 learned_addr learned my addr 172.21.15.59:0/2349986495 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeded9d640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbee00d1c30 0x7fbee006f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 -- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbee00b2270 msgr2=0x7fbee015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbee00b2270 0x7fbee015d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 -- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbee00d1c30 msgr2=0x7fbee006f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbee00d1c30 0x7fbee006f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 -- 172.21.15.59:0/2349986495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee006e580 con 0x7fbee00b2fa0 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbede59c640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbee00b2270 0x7fbee015d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeddd9b640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00b2fa0 0x7fbee0146f40 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fbed400c3f0 tx=0x7fbed400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.735+0000 7fbeceffd640 1 -- 172.21.15.59:0/2349986495 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4017070 con 0x7fbee00b2fa0 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.736+0000 7fbeceffd640 1 -- 172.21.15.59:0/2349986495 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbed40053e0 con 0x7fbee00b2fa0 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.736+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee006e820 con 0x7fbee00b2fa0 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.736+0000 7fbeceffd640 1 -- 172.21.15.59:0/2349986495 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4003850 con 0x7fbee00b2fa0 2024-06-29T15:57:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.736+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee00b0c20 con 0x7fbee00b2fa0 2024-06-29T15:57:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.737+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee00b6ac0 con 0x7fbee00b2fa0 2024-06-29T15:57:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.738+0000 7fbeceffd640 1 -- 172.21.15.59:0/2349986495 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7fbed401a070 con 0x7fbee00b2fa0 2024-06-29T15:57:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.738+0000 7fbeceffd640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbec0076510 0x7fbec0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.738+0000 7fbeceffd640 1 -- 172.21.15.59:0/2349986495 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 6660+0+0 (secure 0 0 0) 0x7fbed40974f0 con 0x7fbee00b2fa0 2024-06-29T15:57:06.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.738+0000 7fbede59c640 1 -- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbec0076510 msgr2=0x7fbec0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:06.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.738+0000 7fbede59c640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbec0076510 0x7fbec0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.742+0000 7fbeceffd640 1 -- 172.21.15.59:0/2349986495 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbed4021020 con 0x7fbee00b2fa0 2024-06-29T15:57:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.934+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbee0070590 con 0x7fbee00b2fa0 2024-06-29T15:57:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.935+0000 7fbeceffd640 1 -- 172.21.15.59:0/2349986495 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) ==== 74+0+17772 (secure 0 0 0) 0x7fbed40601b0 con 0x7fbee00b2fa0 2024-06-29T15:57:06.936 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:06.936 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":209,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:05.077879+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.935517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.518490+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":209,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.2c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2009747699":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/2580944857":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/4008855742":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6833/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6832/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/719772475":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/4087971292":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/3198120249":"2024-06-30T15:55:49.819811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbec0076510 msgr2=0x7fbec0078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbec0076510 0x7fbec0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00b2fa0 msgr2=0x7fbee0146f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00b2fa0 0x7fbee0146f40 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fbed400c3f0 tx=0x7fbed400c8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 shutdown_connections 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7fbec0076510 0x7fbec0078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbee00d1c30 0x7fbee006f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbee00b2fa0 0x7fbee0146f40 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 --2- 172.21.15.59:0/2349986495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbee00b2270 0x7fbee015d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 >> 172.21.15.59:0/2349986495 conn(0x7fbee0073e10 msgr2=0x7fbee0158170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 shutdown_connections 2024-06-29T15:57:06.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:06.938+0000 7fbee4911640 1 -- 172.21.15.59:0/2349986495 wait complete. 2024-06-29T15:57:06.951 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:55:35.935517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '203', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:55:36.518490+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 209, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:06.952 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:07.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.072+0000 7f9447e4d640 1 -- 172.21.15.59:0/2226911671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 msgr2=0x7f944015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.072+0000 7f9447e4d640 1 --2- 172.21.15.59:0/2226911671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 0x7f944015e8d0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f94400d15d0 tx=0x7f943402a5f0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.072+0000 7f9447e4d640 1 -- 172.21.15.59:0/2226911671 shutdown_connections 2024-06-29T15:57:07.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.072+0000 7f9447e4d640 1 --2- 172.21.15.59:0/2226911671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 0x7f944015e8d0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.072+0000 7f9447e4d640 1 --2- 172.21.15.59:0/2226911671 >> v2:172.21.15.153:3300/0 conn(0x7f94400d23c0 0x7f94400b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.072+0000 7f9447e4d640 1 -- 172.21.15.59:0/2226911671 >> 172.21.15.59:0/2226911671 conn(0x7f94400d3ee0 msgr2=0x7f94400d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.073+0000 7f9447e4d640 1 -- 172.21.15.59:0/2226911671 shutdown_connections 2024-06-29T15:57:07.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.073+0000 7f9447e4d640 1 -- 172.21.15.59:0/2226911671 wait complete. 2024-06-29T15:57:07.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.074+0000 7f9447e4d640 1 Processor -- start 2024-06-29T15:57:07.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.074+0000 7f9447e4d640 1 -- start start 2024-06-29T15:57:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9447e4d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 0x7f94400c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9447e4d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94400d23c0 0x7f9440160ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9447e4d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f944015ee10 0x7f94401e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9447e4d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9440160240 con 0x7f94400b3ed0 2024-06-29T15:57:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9447e4d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94401600c0 con 0x7f944015ee10 2024-06-29T15:57:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9447e4d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f94401603c0 con 0x7f94400d23c0 2024-06-29T15:57:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9445be9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 0x7f94400c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9445be9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 0x7f94400c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51786/0 (socket says 172.21.15.59:51786) 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f9445be9640 1 -- 172.21.15.59:0/4140517940 learned_addr learned my addr 172.21.15.59:0/4140517940 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f94463ea640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f944015ee10 0x7f94401e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f94453e8640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94400d23c0 0x7f9440160ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f94453e8640 1 -- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f944015ee10 msgr2=0x7f94401e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f94453e8640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f944015ee10 0x7f94401e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f94453e8640 1 -- 172.21.15.59:0/4140517940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 msgr2=0x7f94400c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f94453e8640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 0x7f94400c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f94453e8640 1 -- 172.21.15.59:0/4140517940 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94400c7ad0 con 0x7f94400d23c0 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.076+0000 7f94463ea640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f944015ee10 0x7f94401e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.077+0000 7f94453e8640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94400d23c0 0x7f9440160ae0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9434031000 tx=0x7f94340038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.077+0000 7f94327fc640 1 -- 172.21.15.59:0/4140517940 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943403e030 con 0x7f94400d23c0 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.077+0000 7f94327fc640 1 -- 172.21.15.59:0/4140517940 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9434003d90 con 0x7f94400d23c0 2024-06-29T15:57:07.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.077+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94400c7db0 con 0x7f94400d23c0 2024-06-29T15:57:07.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.077+0000 7f94327fc640 1 -- 172.21.15.59:0/4140517940 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9434035040 con 0x7f94400d23c0 2024-06-29T15:57:07.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.077+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94400c75f0 con 0x7f94400d23c0 2024-06-29T15:57:07.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.078+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94400c7820 con 0x7f94400d23c0 2024-06-29T15:57:07.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.079+0000 7f94327fc640 1 -- 172.21.15.59:0/4140517940 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f9434041030 con 0x7f94400d23c0 2024-06-29T15:57:07.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.079+0000 7f94327fc640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f9424076510 0x7f9424078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.079+0000 7f9445be9640 1 -- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f9424076510 msgr2=0x7f9424078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:07.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.079+0000 7f94327fc640 1 -- 172.21.15.59:0/4140517940 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(209..209 src has 1..209) ==== 6660+0+0 (secure 0 0 0) 0x7f94340b85e0 con 0x7f94400d23c0 2024-06-29T15:57:07.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.079+0000 7f9445be9640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f9424076510 0x7f9424078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:07.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.084+0000 7f94327fc640 1 -- 172.21.15.59:0/4140517940 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f94340810e0 con 0x7f94400d23c0 2024-06-29T15:57:07.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.275+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f944015e8d0 con 0x7f94400d23c0 2024-06-29T15:57:07.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.276+0000 7f94327fc640 1 -- 172.21.15.59:0/4140517940 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v209) ==== 93+0+10 (secure 0 0 0) 0x7f94340853e0 con 0x7f94400d23c0 2024-06-29T15:57:07.277 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f9424076510 msgr2=0x7f9424078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f9424076510 0x7f9424078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94400d23c0 msgr2=0x7f9440160ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94400d23c0 0x7f9440160ae0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9434031000 tx=0x7f94340038a0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 shutdown_connections 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f9424076510 0x7f9424078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f944015ee10 0x7f94401e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f94400d23c0 0x7f9440160ae0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 --2- 172.21.15.59:0/4140517940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94400b3ed0 0x7f94400c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 >> 172.21.15.59:0/4140517940 conn(0x7f94400d3ee0 msgr2=0x7f944006cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 shutdown_connections 2024-06-29T15:57:07.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.278+0000 7f9447e4d640 1 -- 172.21.15.59:0/4140517940 wait complete. 2024-06-29T15:57:07.291 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:07.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.411+0000 7f32ca8d6640 1 -- 172.21.15.59:0/551690234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 msgr2=0x7f32c41657d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.411+0000 7f32ca8d6640 1 --2- 172.21.15.59:0/551690234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 0x7f32c41657d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f32bc005fd0 tx=0x7f32bc02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.411+0000 7f32ca8d6640 1 -- 172.21.15.59:0/551690234 shutdown_connections 2024-06-29T15:57:07.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.411+0000 7f32ca8d6640 1 --2- 172.21.15.59:0/551690234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 0x7f32c41657d0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.411+0000 7f32ca8d6640 1 -- 172.21.15.59:0/551690234 >> 172.21.15.59:0/551690234 conn(0x7f32c40b2ab0 msgr2=0x7f32c40b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.411+0000 7f32ca8d6640 1 -- 172.21.15.59:0/551690234 shutdown_connections 2024-06-29T15:57:07.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.411+0000 7f32ca8d6640 1 -- 172.21.15.59:0/551690234 wait complete. 2024-06-29T15:57:07.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.413+0000 7f32ca8d6640 1 Processor -- start 2024-06-29T15:57:07.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.413+0000 7f32ca8d6640 1 -- start start 2024-06-29T15:57:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32ca8d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 0x7f32c415cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32ca8d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32c40d0130 0x7f32c40ab080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32ca8d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32c4143840 0x7f32c4159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32ca8d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f32c406cf40 con 0x7f32c40d0130 2024-06-29T15:57:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32ca8d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32c406cdc0 con 0x7f32c406ebe0 2024-06-29T15:57:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32ca8d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f32c406d0c0 con 0x7f32c4143840 2024-06-29T15:57:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32c40d0130 0x7f32c40ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32c40d0130 0x7f32c40ab080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51788/0 (socket says 172.21.15.59:51788) 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 -- 172.21.15.59:0/1457818783 learned_addr learned my addr 172.21.15.59:0/1457818783 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c3fff640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 0x7f32c415cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c8e73640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32c4143840 0x7f32c4159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 -- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32c4143840 msgr2=0x7f32c4159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32c4143840 0x7f32c4159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 -- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 msgr2=0x7f32c415cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 0x7f32c415cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 -- 172.21.15.59:0/1457818783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c40a9fe0 con 0x7f32c40d0130 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c3fff640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 0x7f32c415cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:07.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.415+0000 7f32c37fe640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32c40d0130 0x7f32c40ab080 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f32b800c430 tx=0x7f32b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:07.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.416+0000 7f32c0ff9640 1 -- 172.21.15.59:0/1457818783 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8017070 con 0x7f32c40d0130 2024-06-29T15:57:07.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.416+0000 7f32c0ff9640 1 -- 172.21.15.59:0/1457818783 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32b80053c0 con 0x7f32c40d0130 2024-06-29T15:57:07.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.416+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c40aa2c0 con 0x7f32c40d0130 2024-06-29T15:57:07.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.416+0000 7f32c0ff9640 1 -- 172.21.15.59:0/1457818783 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8003850 con 0x7f32c40d0130 2024-06-29T15:57:07.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.416+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c40aa880 con 0x7f32c40d0130 2024-06-29T15:57:07.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.417+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c40b4ee0 con 0x7f32c40d0130 2024-06-29T15:57:07.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.418+0000 7f32c0ff9640 1 -- 172.21.15.59:0/1457818783 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f32b801a070 con 0x7f32c40d0130 2024-06-29T15:57:07.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.419+0000 7f32c0ff9640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32b0076510 0x7f32b0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.419+0000 7f32c3fff640 1 -- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32b0076510 msgr2=0x7f32b0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:07.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.419+0000 7f32c0ff9640 1 -- 172.21.15.59:0/1457818783 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 6660+0+0 (secure 0 0 0) 0x7f32b80972f0 con 0x7f32c40d0130 2024-06-29T15:57:07.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.419+0000 7f32c3fff640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32b0076510 0x7f32b0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:07.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.423+0000 7f32c0ff9640 1 -- 172.21.15.59:0/1457818783 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f32b805ff20 con 0x7f32c40d0130 2024-06-29T15:57:07.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.619+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f32c406f200 con 0x7f32c40d0130 2024-06-29T15:57:07.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.620+0000 7f32c0ff9640 1 -- 172.21.15.59:0/1457818783 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v209) ==== 104+0+11 (secure 0 0 0) 0x7f32b8064220 con 0x7f32c40d0130 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.620+0000 7f32c3fff640 1 -- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32b0076510 msgr2=0x7f32b0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1875870697 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.620+0000 7f32c3fff640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32b0076510 0x7f32b0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:07.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32b0076510 msgr2=0x7f32b0078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:07.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32b0076510 0x7f32b0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32c40d0130 msgr2=0x7f32c40ab080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32c40d0130 0x7f32c40ab080 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f32b800c430 tx=0x7f32b800c8f0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 shutdown_connections 2024-06-29T15:57:07.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:6832/1875870697,v1:172.21.15.153:6833/1875870697] conn(0x7f32b0076510 0x7f32b0078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f32c4143840 0x7f32c4159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32c40d0130 0x7f32c40ab080 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 --2- 172.21.15.59:0/1457818783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32c406ebe0 0x7f32c415cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 >> 172.21.15.59:0/1457818783 conn(0x7f32c40b2ab0 msgr2=0x7f32c406e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 shutdown_connections 2024-06-29T15:57:07.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.622+0000 7f32ca8d6640 1 -- 172.21.15.59:0/1457818783 wait complete. 2024-06-29T15:57:07.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:07.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.761+0000 7f609ecc5640 1 -- 172.21.15.59:0/799361494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980d23c0 msgr2=0x7f60980b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.761+0000 7f609ecc5640 1 --2- 172.21.15.59:0/799361494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980d23c0 0x7f60980b3990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f608c005fd0 tx=0x7f608c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.761+0000 7f609ecc5640 1 -- 172.21.15.59:0/799361494 shutdown_connections 2024-06-29T15:57:07.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.761+0000 7f609ecc5640 1 --2- 172.21.15.59:0/799361494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f609815efe0 0x7f609815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.761+0000 7f609ecc5640 1 --2- 172.21.15.59:0/799361494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980d23c0 0x7f60980b3990 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.761+0000 7f609ecc5640 1 -- 172.21.15.59:0/799361494 >> 172.21.15.59:0/799361494 conn(0x7f60980d3ee0 msgr2=0x7f60980d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.761+0000 7f609ecc5640 1 -- 172.21.15.59:0/799361494 shutdown_connections 2024-06-29T15:57:07.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.761+0000 7f609ecc5640 1 -- 172.21.15.59:0/799361494 wait complete. 2024-06-29T15:57:07.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.763+0000 7f609ecc5640 1 Processor -- start 2024-06-29T15:57:07.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.763+0000 7f609ecc5640 1 -- start start 2024-06-29T15:57:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609ecc5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980b3ed0 0x7f609815da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609ecc5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60980d23c0 0x7f6098150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609ecc5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f609815efe0 0x7f60981ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609ecc5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6098160280 con 0x7f60980d23c0 2024-06-29T15:57:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609ecc5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6098160100 con 0x7f60980b3ed0 2024-06-29T15:57:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609ecc5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6098160400 con 0x7f609815efe0 2024-06-29T15:57:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60980d23c0 0x7f6098150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609ca61640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980b3ed0 0x7f609815da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60980d23c0 0x7f6098150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51800/0 (socket says 172.21.15.59:51800) 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 -- 172.21.15.59:0/1434776558 learned_addr learned my addr 172.21.15.59:0/1434776558 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609d262640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f609815efe0 0x7f60981ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 -- 172.21.15.59:0/1434776558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f609815efe0 msgr2=0x7f60981ea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f609815efe0 0x7f60981ea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 -- 172.21.15.59:0/1434776558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980b3ed0 msgr2=0x7f609815da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980b3ed0 0x7f609815da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 -- 172.21.15.59:0/1434776558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609815e550 con 0x7f60980d23c0 2024-06-29T15:57:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f609ca61640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980b3ed0 0x7f609815da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.765+0000 7f6097fff640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60980d23c0 0x7f6098150870 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f608400c370 tx=0x7f608400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.766+0000 7f60957fa640 1 -- 172.21.15.59:0/1434776558 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6084017070 con 0x7f60980d23c0 2024-06-29T15:57:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.766+0000 7f60957fa640 1 -- 172.21.15.59:0/1434776558 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60840053c0 con 0x7f60980d23c0 2024-06-29T15:57:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.766+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609815e270 con 0x7f60980d23c0 2024-06-29T15:57:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.766+0000 7f60957fa640 1 -- 172.21.15.59:0/1434776558 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6084003850 con 0x7f60980d23c0 2024-06-29T15:57:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.766+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60981523c0 con 0x7f60980d23c0 2024-06-29T15:57:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.767+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60980b3990 con 0x7f60980d23c0 2024-06-29T15:57:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.768+0000 7f60957fa640 1 -- 172.21.15.59:0/1434776558 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 125) ==== 97912+0+0 (secure 0 0 0) 0x7f608401a070 con 0x7f60980d23c0 2024-06-29T15:57:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.768+0000 7f60957fa640 1 -- 172.21.15.59:0/1434776558 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7046+0+0 (secure 0 0 0) 0x7f6084096b20 con 0x7f60980d23c0 2024-06-29T15:57:07.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.773+0000 7f60957fa640 1 -- 172.21.15.59:0/1434776558 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f60980b42e0 con 0x7f60980d23c0 2024-06-29T15:57:07.809 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febf93cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.809 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febf93cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.809 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febf93cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.809 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febf93cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.809 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febf93cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.810 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febfc3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.810 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febfc3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.810 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febfc3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.810 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febfc3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.810 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:07.808+0000 7febfc3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.965+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f60980d2950 con 0x7f60980d23c0 2024-06-29T15:57:07.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.966+0000 7f60957fa640 1 -- 172.21.15.59:0/1434776558 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v210) ==== 100+0+11 (secure 0 0 0) 0x7f6084068300 con 0x7f60980d23c0 2024-06-29T15:57:07.967 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:57:07.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60980d23c0 msgr2=0x7f6098150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60980d23c0 0x7f6098150870 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f608400c370 tx=0x7f608400c830 comp rx=0 tx=0).stop 2024-06-29T15:57:07.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 shutdown_connections 2024-06-29T15:57:07.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f609815efe0 0x7f60981ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60980d23c0 0x7f6098150870 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 --2- 172.21.15.59:0/1434776558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60980b3ed0 0x7f609815da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 >> 172.21.15.59:0/1434776558 conn(0x7f60980d3ee0 msgr2=0x7f609806d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 shutdown_connections 2024-06-29T15:57:07.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:07.968+0000 7f609ecc5640 1 -- 172.21.15.59:0/1434776558 wait complete. 2024-06-29T15:57:07.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:08.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.100+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3777493493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc0cff70 msgr2=0x7fc1dc072fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.100+0000 7fc1e42b2640 1 --2- 172.21.15.59:0/3777493493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc0cff70 0x7fc1dc072fe0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4005fd0 tx=0x7fc1d402aa10 comp rx=0 tx=0).stop 2024-06-29T15:57:08.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.100+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3777493493 shutdown_connections 2024-06-29T15:57:08.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.100+0000 7fc1e42b2640 1 --2- 172.21.15.59:0/3777493493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc0cff70 0x7fc1dc072fe0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.100+0000 7fc1e42b2640 1 --2- 172.21.15.59:0/3777493493 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1dc1451b0 0x7fc1dc070060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.100+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3777493493 >> 172.21.15.59:0/3777493493 conn(0x7fc1dc0b30c0 msgr2=0x7fc1dc0cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.100+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3777493493 shutdown_connections 2024-06-29T15:57:08.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.100+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3777493493 wait complete. 2024-06-29T15:57:08.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.101+0000 7fc1e42b2640 1 Processor -- start 2024-06-29T15:57:08.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.102+0000 7fc1e42b2640 1 -- start start 2024-06-29T15:57:08.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e42b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1dc1422c0 0x7fc1dc13f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e42b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc1451b0 0x7fc1dc0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e42b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1dc0aa9c0 0x7fc1dc1e6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e42b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc073a20 con 0x7fc1dc1451b0 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e42b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc0738a0 con 0x7fc1dc0aa9c0 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e42b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc1dc073ba0 con 0x7fc1dc1422c0 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e184d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc1451b0 0x7fc1dc0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e184d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc1451b0 0x7fc1dc0adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51812/0 (socket says 172.21.15.59:51812) 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e184d640 1 -- 172.21.15.59:0/3342017023 learned_addr learned my addr 172.21.15.59:0/3342017023 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e204e640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1dc1422c0 0x7fc1dc13f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e284f640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1dc0aa9c0 0x7fc1dc1e6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e184d640 1 -- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1dc1422c0 msgr2=0x7fc1dc13f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e184d640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1dc1422c0 0x7fc1dc13f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e184d640 1 -- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1dc0aa9c0 msgr2=0x7fc1dc1e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e184d640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1dc0aa9c0 0x7fc1dc1e6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e184d640 1 -- 172.21.15.59:0/3342017023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc0a9920 con 0x7fc1dc1451b0 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e204e640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1dc1422c0 0x7fc1dc13f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.103+0000 7fc1e284f640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1dc0aa9c0 0x7fc1dc1e6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.104+0000 7fc1e184d640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc1451b0 0x7fc1dc0adef0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00c420 tx=0x7fc1cc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.104+0000 7fc1d2ffd640 1 -- 172.21.15.59:0/3342017023 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc017070 con 0x7fc1dc1451b0 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.104+0000 7fc1d2ffd640 1 -- 172.21.15.59:0/3342017023 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1cc0035f0 con 0x7fc1dc1451b0 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.104+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc0a9c00 con 0x7fc1dc1451b0 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.104+0000 7fc1d2ffd640 1 -- 172.21.15.59:0/3342017023 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc003a40 con 0x7fc1dc1451b0 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.104+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1dc0aa1c0 con 0x7fc1dc1451b0 2024-06-29T15:57:08.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.105+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1dc0740c0 con 0x7fc1dc1451b0 2024-06-29T15:57:08.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.106+0000 7fc1d2ffd640 1 -- 172.21.15.59:0/3342017023 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 125) ==== 97912+0+0 (secure 0 0 0) 0x7fc1cc01a070 con 0x7fc1dc1451b0 2024-06-29T15:57:08.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.106+0000 7fc1d2ffd640 1 -- 172.21.15.59:0/3342017023 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7046+0+0 (secure 0 0 0) 0x7fc1cc096bb0 con 0x7fc1dc1451b0 2024-06-29T15:57:08.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.110+0000 7fc1d2ffd640 1 -- 172.21.15.59:0/3342017023 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc1cc09d080 con 0x7fc1dc1451b0 2024-06-29T15:57:08.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.341+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1dc074b50 con 0x7fc1dc1451b0 2024-06-29T15:57:08.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.342+0000 7fc1d2ffd640 1 -- 172.21.15.59:0/3342017023 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5246 (secure 0 0 0) 0x7fc1cc060310 con 0x7fc1dc1451b0 2024-06-29T15:57:08.343 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:08.343 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"btime":"2024-06-29T15:55:41:198021+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3297476433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3297476433},{"type":"v1","addr":"172.21.15.153:6837","nonce":3297476433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":10974,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3987613682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3987613682},{"type":"v1","addr":"172.21.15.59:6834","nonce":3987613682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":10977,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2501188073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2501188073},{"type":"v1","addr":"172.21.15.59:6835","nonce":2501188073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.567350+0000","modified":"2024-06-29T15:55:40.592165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10983},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10983":{"gid":10983,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2163190416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2163190416},{"type":"v1","addr":"172.21.15.153:6836","nonce":2163190416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10983,"qdb_cluster":[10983]},"id":12}]} 2024-06-29T15:57:08.343 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc1451b0 msgr2=0x7fc1dc0adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc1451b0 0x7fc1dc0adef0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00c420 tx=0x7fc1cc00c8e0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 shutdown_connections 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1dc0aa9c0 0x7fc1dc1e6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1dc1451b0 0x7fc1dc0adef0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 --2- 172.21.15.59:0/3342017023 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1dc1422c0 0x7fc1dc13f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 >> 172.21.15.59:0/3342017023 conn(0x7fc1dc0b30c0 msgr2=0x7fc1dc143990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 shutdown_connections 2024-06-29T15:57:08.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.344+0000 7fc1e42b2640 1 -- 172.21.15.59:0/3342017023 wait complete. 2024-06-29T15:57:08.356 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:08.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.472+0000 7f746e712640 1 -- 172.21.15.59:0/793691120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680d23c0 msgr2=0x7f74680b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.472+0000 7f746e712640 1 --2- 172.21.15.59:0/793691120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680d23c0 0x7f74680b3990 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f7468161d40 tx=0x7f745802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.472+0000 7f746e712640 1 -- 172.21.15.59:0/793691120 shutdown_connections 2024-06-29T15:57:08.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.472+0000 7f746e712640 1 --2- 172.21.15.59:0/793691120 >> v2:172.21.15.153:3300/0 conn(0x7f746815efe0 0x7f746815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.472+0000 7f746e712640 1 --2- 172.21.15.59:0/793691120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680d23c0 0x7f74680b3990 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.472+0000 7f746e712640 1 -- 172.21.15.59:0/793691120 >> 172.21.15.59:0/793691120 conn(0x7f74680d3ee0 msgr2=0x7f74680d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.472+0000 7f746e712640 1 -- 172.21.15.59:0/793691120 shutdown_connections 2024-06-29T15:57:08.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.472+0000 7f746e712640 1 -- 172.21.15.59:0/793691120 wait complete. 2024-06-29T15:57:08.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.474+0000 7f746e712640 1 Processor -- start 2024-06-29T15:57:08.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.474+0000 7f746e712640 1 -- start start 2024-06-29T15:57:08.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f746e712640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680b3ed0 0x7f746815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f746e712640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74680d23c0 0x7f74680c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f746e712640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f746815efe0 0x7f746815cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f746e712640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7468160280 con 0x7f74680b3ed0 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f746e712640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7468160100 con 0x7f74680d23c0 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f746e712640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7468160400 con 0x7f746815efe0 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680b3ed0 0x7f746815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680b3ed0 0x7f746815c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51832/0 (socket says 172.21.15.59:51832) 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 -- 172.21.15.59:0/4021082108 learned_addr learned my addr 172.21.15.59:0/4021082108 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f746ccaf640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f746815efe0 0x7f746815cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f74677fe640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74680d23c0 0x7f74680c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 -- 172.21.15.59:0/4021082108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f746815efe0 msgr2=0x7f746815cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f746815efe0 0x7f746815cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 -- 172.21.15.59:0/4021082108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74680d23c0 msgr2=0x7f74680c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74680d23c0 0x7f74680c6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 -- 172.21.15.59:0/4021082108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746815c660 con 0x7f74680b3ed0 2024-06-29T15:57:08.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7467fff640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680b3ed0 0x7f746815c120 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f74580099c0 tx=0x7f74580038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:08.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7464ff9640 1 -- 172.21.15.59:0/4021082108 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745803e030 con 0x7f74680b3ed0 2024-06-29T15:57:08.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7464ff9640 1 -- 172.21.15.59:0/4021082108 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7458003db0 con 0x7f74680b3ed0 2024-06-29T15:57:08.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746815b080 con 0x7f74680b3ed0 2024-06-29T15:57:08.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.476+0000 7f7464ff9640 1 -- 172.21.15.59:0/4021082108 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458035040 con 0x7f74680b3ed0 2024-06-29T15:57:08.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.477+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746815b640 con 0x7f74680b3ed0 2024-06-29T15:57:08.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.478+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74680d2950 con 0x7f74680b3ed0 2024-06-29T15:57:08.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.478+0000 7f7464ff9640 1 -- 172.21.15.59:0/4021082108 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 125) ==== 97912+0+0 (secure 0 0 0) 0x7f7458041030 con 0x7f74680b3ed0 2024-06-29T15:57:08.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.479+0000 7f7464ff9640 1 -- 172.21.15.59:0/4021082108 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7046+0+0 (secure 0 0 0) 0x7f74580b8010 con 0x7f74680b3ed0 2024-06-29T15:57:08.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.483+0000 7f7464ff9640 1 -- 172.21.15.59:0/4021082108 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f74580811a0 con 0x7f74680b3ed0 2024-06-29T15:57:08.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.673+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74680b78d0 con 0x7f74680b3ed0 2024-06-29T15:57:08.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.675+0000 7f7464ff9640 1 -- 172.21.15.59:0/4021082108 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) ==== 74+0+18340 (secure 0 0 0) 0x7f74580854a0 con 0x7f74680b3ed0 2024-06-29T15:57:08.676 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:08.676 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":210,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:07.739937+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.935517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.518490+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":209,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.2c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3494626796":"2024-06-30T15:57:07.739872+0000","172.21.15.153:6833/1875870697":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/1193885100":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/3120983149":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/1750092138":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/3166749975":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/1068787394":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/4060660749":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/2009747699":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/2580944857":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/4008855742":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6833/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6832/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/743192862":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/719772475":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6832/1875870697":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/4087971292":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/3198120249":"2024-06-30T15:55:49.819811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:08.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680b3ed0 msgr2=0x7f746815c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680b3ed0 0x7f746815c120 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f74580099c0 tx=0x7f74580038a0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 shutdown_connections 2024-06-29T15:57:08.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f746815efe0 0x7f746815cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74680d23c0 0x7f74680c6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 --2- 172.21.15.59:0/4021082108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74680b3ed0 0x7f746815c120 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 >> 172.21.15.59:0/4021082108 conn(0x7f74680d3ee0 msgr2=0x7f746806caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 shutdown_connections 2024-06-29T15:57:08.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.677+0000 7f746e712640 1 -- 172.21.15.59:0/4021082108 wait complete. 2024-06-29T15:57:08.689 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:57:08.690 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:08.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.811+0000 7f606317b640 1 -- 172.21.15.59:0/995659320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 msgr2=0x7f6048003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.811+0000 7f606317b640 1 --2- 172.21.15.59:0/995659320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 0x7f6048003f10 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f604c009b50 tx=0x7f6048005830 comp rx=0 tx=0).stop 2024-06-29T15:57:08.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.811+0000 7f6061718640 1 -- 172.21.15.59:0/995659320 reap_dead start 2024-06-29T15:57:08.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.812+0000 7f606317b640 1 -- 172.21.15.59:0/995659320 shutdown_connections 2024-06-29T15:57:08.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.812+0000 7f606317b640 1 -- 172.21.15.59:0/995659320 >> 172.21.15.59:0/995659320 conn(0x7f605c06c950 msgr2=0x7f605c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.812+0000 7f606317b640 1 -- 172.21.15.59:0/995659320 shutdown_connections 2024-06-29T15:57:08.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.812+0000 7f606317b640 1 -- 172.21.15.59:0/995659320 wait complete. 2024-06-29T15:57:08.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.813+0000 7f606317b640 1 Processor -- start 2024-06-29T15:57:08.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.813+0000 7f606317b640 1 -- start start 2024-06-29T15:57:08.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f606317b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 0x7f605c138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f606317b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f605c15b7e0 0x7f605c13d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f606317b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13d810 0x7f605c142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f606317b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f605c0cf870 con 0x7f6048003aa0 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f606317b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f605c145ab0 con 0x7f605c13d810 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f606317b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f605c0cf9f0 con 0x7f605c15b7e0 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6060f17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 0x7f605c138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6060f17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 0x7f605c138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51856/0 (socket says 172.21.15.59:51856) 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6060f17640 1 -- 172.21.15.59:0/1137054737 learned_addr learned my addr 172.21.15.59:0/1137054737 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6060f17640 1 -- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f605c15b7e0 msgr2=0x7f605c13d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6061718640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13d810 0x7f605c142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f605bfff640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f605c15b7e0 0x7f605c13d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6060f17640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f605c15b7e0 0x7f605c13d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6060f17640 1 -- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13d810 msgr2=0x7f605c142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6060f17640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13d810 0x7f605c142120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6060f17640 1 -- 172.21.15.59:0/1137054737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c13dc20 con 0x7f6048003aa0 2024-06-29T15:57:08.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f6061718640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13d810 0x7f605c142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.815+0000 7f605bfff640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f605c15b7e0 0x7f605c13d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.816+0000 7f6060f17640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 0x7f605c138130 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f604c009c70 tx=0x7f604c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.816+0000 7f60597fa640 1 -- 172.21.15.59:0/1137054737 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c038040 con 0x7f6048003aa0 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.816+0000 7f60597fa640 1 -- 172.21.15.59:0/1137054737 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f604c004020 con 0x7f6048003aa0 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.816+0000 7f60597fa640 1 -- 172.21.15.59:0/1137054737 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c004470 con 0x7f6048003aa0 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.816+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c13a920 con 0x7f6048003aa0 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.816+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c13aee0 con 0x7f6048003aa0 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.817+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c15c2f0 con 0x7f6048003aa0 2024-06-29T15:57:08.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.817+0000 7f60597fa640 1 -- 172.21.15.59:0/1137054737 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7f604c045020 con 0x7f6048003aa0 2024-06-29T15:57:08.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.818+0000 7f60597fa640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f603c076150 0x7f603c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.818+0000 7f60597fa640 1 -- 172.21.15.59:0/1137054737 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7046+0+0 (secure 0 0 0) 0x7f604c0b8230 con 0x7f6048003aa0 2024-06-29T15:57:08.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.818+0000 7f605bfff640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f603c076150 0x7f603c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.819+0000 7f605bfff640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f603c076150 0x7f603c078660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f605c0d0d10 tx=0x7f6044001de0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:08.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:08.822+0000 7f60597fa640 1 -- 172.21.15.59:0/1137054737 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f605c13dc20 con 0x7f6048003aa0 2024-06-29T15:57:09.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.055+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f605c0b1f10 con 0x7f6048003aa0 2024-06-29T15:57:09.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.055+0000 7f60597fa640 1 -- 172.21.15.59:0/1137054737 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5246 (secure 0 0 0) 0x7f604c081990 con 0x7f6048003aa0 2024-06-29T15:57:09.056 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:09.056 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"btime":"2024-06-29T15:55:41:198021+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3297476433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3297476433},{"type":"v1","addr":"172.21.15.153:6837","nonce":3297476433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":10974,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3987613682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3987613682},{"type":"v1","addr":"172.21.15.59:6834","nonce":3987613682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":10977,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2501188073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2501188073},{"type":"v1","addr":"172.21.15.59:6835","nonce":2501188073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.567350+0000","modified":"2024-06-29T15:55:40.592165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10983},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10983":{"gid":10983,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2163190416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2163190416},{"type":"v1","addr":"172.21.15.153:6836","nonce":2163190416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10983,"qdb_cluster":[10983]},"id":12}]} 2024-06-29T15:57:09.057 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-06-29T15:57:09.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f603c076150 msgr2=0x7f603c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f603c076150 0x7f603c078660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f605c0d0d10 tx=0x7f6044001de0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 msgr2=0x7f605c138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 0x7f605c138130 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f604c009c70 tx=0x7f604c033670 comp rx=0 tx=0).stop 2024-06-29T15:57:09.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 shutdown_connections 2024-06-29T15:57:09.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f603c076150 0x7f603c078660 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13d810 0x7f605c142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f605c15b7e0 0x7f605c13d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 --2- 172.21.15.59:0/1137054737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6048003aa0 0x7f605c138130 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 >> 172.21.15.59:0/1137054737 conn(0x7f605c06c950 msgr2=0x7f605c15f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 shutdown_connections 2024-06-29T15:57:09.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.058+0000 7f606317b640 1 -- 172.21.15.59:0/1137054737 wait complete. 2024-06-29T15:57:09.070 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:09.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.187+0000 7fd48d78c640 1 -- 172.21.15.59:0/2258015491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4881414c0 msgr2=0x7fd4881418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.187+0000 7fd48d78c640 1 --2- 172.21.15.59:0/2258015491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4881414c0 0x7fd4881418d0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fd4880b5590 tx=0x7fd47802a980 comp rx=0 tx=0).stop 2024-06-29T15:57:09.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.187+0000 7fd48d78c640 1 -- 172.21.15.59:0/2258015491 shutdown_connections 2024-06-29T15:57:09.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.187+0000 7fd48d78c640 1 --2- 172.21.15.59:0/2258015491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4881414c0 0x7fd4881418d0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.187+0000 7fd48d78c640 1 --2- 172.21.15.59:0/2258015491 >> v2:172.21.15.153:3300/0 conn(0x7fd48813e9b0 0x7fd48813edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.187+0000 7fd48d78c640 1 -- 172.21.15.59:0/2258015491 >> 172.21.15.59:0/2258015491 conn(0x7fd4880b6fb0 msgr2=0x7fd4880b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.187+0000 7fd48d78c640 1 -- 172.21.15.59:0/2258015491 shutdown_connections 2024-06-29T15:57:09.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.187+0000 7fd48d78c640 1 -- 172.21.15.59:0/2258015491 wait complete. 2024-06-29T15:57:09.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.189+0000 7fd48d78c640 1 Processor -- start 2024-06-29T15:57:09.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.189+0000 7fd48d78c640 1 -- start start 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd48d78c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd48813e9b0 0x7fd48806f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd48d78c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4881414c0 0x7fd488138830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd48d78c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd48815ceb0 0x7fd4880cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd48d78c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd4880b29d0 con 0x7fd48813e9b0 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd48d78c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd4880b2850 con 0x7fd48815ceb0 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd48d78c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd4880b2b50 con 0x7fd4881414c0 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd486ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd48813e9b0 0x7fd48806f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd486ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd48813e9b0 0x7fd48806f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51874/0 (socket says 172.21.15.59:51874) 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.190+0000 7fd486ffd640 1 -- 172.21.15.59:0/187304619 learned_addr learned my addr 172.21.15.59:0/187304619 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:09.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd4877fe640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd48815ceb0 0x7fd4880cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd4867fc640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4881414c0 0x7fd488138830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd486ffd640 1 -- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4881414c0 msgr2=0x7fd488138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd486ffd640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4881414c0 0x7fd488138830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd486ffd640 1 -- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd48815ceb0 msgr2=0x7fd4880cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd486ffd640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd48815ceb0 0x7fd4880cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd486ffd640 1 -- 172.21.15.59:0/187304619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48806e010 con 0x7fd48813e9b0 2024-06-29T15:57:09.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd486ffd640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd48813e9b0 0x7fd48806f0b0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fd48000b420 tx=0x7fd48000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd47f7fe640 1 -- 172.21.15.59:0/187304619 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480011830 con 0x7fd48813e9b0 2024-06-29T15:57:09.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd47f7fe640 1 -- 172.21.15.59:0/187304619 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd480016070 con 0x7fd48813e9b0 2024-06-29T15:57:09.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd47f7fe640 1 -- 172.21.15.59:0/187304619 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480011d60 con 0x7fd48813e9b0 2024-06-29T15:57:09.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48806e2f0 con 0x7fd48813e9b0 2024-06-29T15:57:09.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.191+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd48806e8b0 con 0x7fd48813e9b0 2024-06-29T15:57:09.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.192+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4880b3000 con 0x7fd48813e9b0 2024-06-29T15:57:09.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.193+0000 7fd47f7fe640 1 -- 172.21.15.59:0/187304619 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7fd48001b460 con 0x7fd48813e9b0 2024-06-29T15:57:09.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.193+0000 7fd47f7fe640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd468076120 0x7fd468078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.194+0000 7fd47f7fe640 1 -- 172.21.15.59:0/187304619 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7046+0+0 (secure 0 0 0) 0x7fd4800975b0 con 0x7fd48813e9b0 2024-06-29T15:57:09.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.194+0000 7fd4867fc640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd468076120 0x7fd468078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.194+0000 7fd4867fc640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd468076120 0x7fd468078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd4881f8370 tx=0x7fd47000b2e0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.198+0000 7fd47f7fe640 1 -- 172.21.15.59:0/187304619 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd480060630 con 0x7fd48813e9b0 2024-06-29T15:57:09.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.425+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd48806db00 con 0x7fd48813e9b0 2024-06-29T15:57:09.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.426+0000 7fd47f7fe640 1 -- 172.21.15.59:0/187304619 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5246 (secure 0 0 0) 0x7fd480064930 con 0x7fd48813e9b0 2024-06-29T15:57:09.427 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:09.427 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"btime":"2024-06-29T15:55:41:198021+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3297476433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3297476433},{"type":"v1","addr":"172.21.15.153:6837","nonce":3297476433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":10974,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3987613682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3987613682},{"type":"v1","addr":"172.21.15.59:6834","nonce":3987613682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":10977,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2501188073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2501188073},{"type":"v1","addr":"172.21.15.59:6835","nonce":2501188073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.567350+0000","modified":"2024-06-29T15:55:40.592165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10983},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10983":{"gid":10983,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6836/2163190416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2163190416},{"type":"v1","addr":"172.21.15.153:6836","nonce":2163190416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10983,"qdb_cluster":[10983]},"id":12}]} 2024-06-29T15:57:09.427 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-06-29T15:57:09.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd468076120 msgr2=0x7fd468078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd468076120 0x7fd468078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd4881f8370 tx=0x7fd47000b2e0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd48813e9b0 msgr2=0x7fd48806f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd48813e9b0 0x7fd48806f0b0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fd48000b420 tx=0x7fd48000b8e0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 shutdown_connections 2024-06-29T15:57:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd468076120 0x7fd468078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd48815ceb0 0x7fd4880cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4881414c0 0x7fd488138830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 --2- 172.21.15.59:0/187304619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd48813e9b0 0x7fd48806f0b0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 >> 172.21.15.59:0/187304619 conn(0x7fd4880b6fb0 msgr2=0x7fd4880ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 shutdown_connections 2024-06-29T15:57:09.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.428+0000 7fd48d78c640 1 -- 172.21.15.59:0/187304619 wait complete. 2024-06-29T15:57:09.440 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:09.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.558+0000 7f28e9e47640 1 -- 172.21.15.59:0/2238759872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28cc004040 msgr2=0x7f28cc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.558+0000 7f28e9e47640 1 --2- 172.21.15.59:0/2238759872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28cc004040 0x7f28cc004800 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009b20 tx=0x7f28cc005830 comp rx=0 tx=0).stop 2024-06-29T15:57:09.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.558+0000 7f28e3fff640 1 -- 172.21.15.59:0/2238759872 reap_dead start 2024-06-29T15:57:09.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.558+0000 7f28e9e47640 1 -- 172.21.15.59:0/2238759872 shutdown_connections 2024-06-29T15:57:09.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.558+0000 7f28e9e47640 1 -- 172.21.15.59:0/2238759872 >> 172.21.15.59:0/2238759872 conn(0x7f28e40b6fb0 msgr2=0x7f28e40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.558+0000 7f28e9e47640 1 -- 172.21.15.59:0/2238759872 shutdown_connections 2024-06-29T15:57:09.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.558+0000 7f28e9e47640 1 -- 172.21.15.59:0/2238759872 wait complete. 2024-06-29T15:57:09.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.559+0000 7f28e9e47640 1 Processor -- start 2024-06-29T15:57:09.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.560+0000 7f28e9e47640 1 -- start start 2024-06-29T15:57:09.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e9e47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc004040 0x7f28e41f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e9e47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e4152c70 0x7f28e415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e9e47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28e415e020 0x7f28e4156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e9e47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f28e40b2530 con 0x7f28e415e020 2024-06-29T15:57:09.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e9e47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28e4141bc0 con 0x7f28e4152c70 2024-06-29T15:57:09.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e9e47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f28e40b26b0 con 0x7f28cc004040 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28e415e020 0x7f28e4156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28e415e020 0x7f28e4156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51896/0 (socket says 172.21.15.59:51896) 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e3fff640 1 -- 172.21.15.59:0/3911419935 learned_addr learned my addr 172.21.15.59:0/3911419935 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e2ffd640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e4152c70 0x7f28e415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e37fe640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc004040 0x7f28e41f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e3fff640 1 -- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc004040 msgr2=0x7f28e41f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e3fff640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc004040 0x7f28e41f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e3fff640 1 -- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e4152c70 msgr2=0x7f28e415dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e3fff640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e4152c70 0x7f28e415dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.561+0000 7f28e3fff640 1 -- 172.21.15.59:0/3911419935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e415e430 con 0x7f28e415e020 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.562+0000 7f28e3fff640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28e415e020 0x7f28e4156980 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f28e4073f30 tx=0x7f28dc009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.562+0000 7f28cbfff640 1 -- 172.21.15.59:0/3911419935 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc00fd40 con 0x7f28e415e020 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.562+0000 7f28cbfff640 1 -- 172.21.15.59:0/3911419935 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28dc004b20 con 0x7f28e415e020 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.562+0000 7f28cbfff640 1 -- 172.21.15.59:0/3911419935 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc005020 con 0x7f28e415e020 2024-06-29T15:57:09.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.562+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e41f3280 con 0x7f28e415e020 2024-06-29T15:57:09.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.562+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e41f3a70 con 0x7f28e415e020 2024-06-29T15:57:09.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.563+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e413edc0 con 0x7f28e415e020 2024-06-29T15:57:09.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.563+0000 7f28cbfff640 1 -- 172.21.15.59:0/3911419935 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7f28dc020020 con 0x7f28e415e020 2024-06-29T15:57:09.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.564+0000 7f28cbfff640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28d0076120 0x7f28d0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.564+0000 7f28cbfff640 1 -- 172.21.15.59:0/3911419935 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7046+0+0 (secure 0 0 0) 0x7f28dc098090 con 0x7f28e415e020 2024-06-29T15:57:09.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.564+0000 7f28e37fe640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28d0076120 0x7f28d0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.565+0000 7f28e37fe640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28d0076120 0x7f28d0078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f28d4000fd0 tx=0x7f28d40013d0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.569+0000 7f28cbfff640 1 -- 172.21.15.59:0/3911419935 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f28e415e430 con 0x7f28e415e020 2024-06-29T15:57:09.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.762+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28e41f3cb0 con 0x7f28e415e020 2024-06-29T15:57:09.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.764+0000 7f28cbfff640 1 -- 172.21.15.59:0/3911419935 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) ==== 74+0+18340 (secure 0 0 0) 0x7f28dc061720 con 0x7f28e415e020 2024-06-29T15:57:09.765 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:09.765 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":210,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:07.739937+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.935517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.518490+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":209,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.5","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]},{"pgid":"26.2c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3494626796":"2024-06-30T15:57:07.739872+0000","172.21.15.153:6833/1875870697":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/1193885100":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/3120983149":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/1750092138":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/3166749975":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/1068787394":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/4060660749":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/2009747699":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/2580944857":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/4008855742":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6833/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6832/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/743192862":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/719772475":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6832/1875870697":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/4087971292":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/3198120249":"2024-06-30T15:55:49.819811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:09.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28d0076120 msgr2=0x7f28d0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28d0076120 0x7f28d0078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f28d4000fd0 tx=0x7f28d40013d0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28e415e020 msgr2=0x7f28e4156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28e415e020 0x7f28e4156980 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f28e4073f30 tx=0x7f28dc009f70 comp rx=0 tx=0).stop 2024-06-29T15:57:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 shutdown_connections 2024-06-29T15:57:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28d0076120 0x7f28d0078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28e415e020 0x7f28e4156980 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e4152c70 0x7f28e415dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 --2- 172.21.15.59:0/3911419935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc004040 0x7f28e41f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 >> 172.21.15.59:0/3911419935 conn(0x7f28e40b6fb0 msgr2=0x7f28e4140300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 shutdown_connections 2024-06-29T15:57:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.766+0000 7f28e9e47640 1 -- 172.21.15.59:0/3911419935 wait complete. 2024-06-29T15:57:09.779 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:57:09.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.896+0000 7f3c67adf640 1 -- 172.21.15.59:0/4091871618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c6015ceb0 msgr2=0x7f3c600b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.896+0000 7f3c67adf640 1 --2- 172.21.15.59:0/4091871618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c6015ceb0 0x7f3c600b56d0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f3c50005fd0 tx=0x7f3c5002a820 comp rx=0 tx=0).stop 2024-06-29T15:57:09.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.896+0000 7f3c67adf640 1 -- 172.21.15.59:0/4091871618 shutdown_connections 2024-06-29T15:57:09.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.896+0000 7f3c67adf640 1 --2- 172.21.15.59:0/4091871618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c60141bd0 0x7f3c6013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.896+0000 7f3c67adf640 1 --2- 172.21.15.59:0/4091871618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c6015ceb0 0x7f3c600b56d0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.896+0000 7f3c67adf640 1 -- 172.21.15.59:0/4091871618 >> 172.21.15.59:0/4091871618 conn(0x7f3c600b6fb0 msgr2=0x7f3c600b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.896+0000 7f3c67adf640 1 -- 172.21.15.59:0/4091871618 shutdown_connections 2024-06-29T15:57:09.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.896+0000 7f3c67adf640 1 -- 172.21.15.59:0/4091871618 wait complete. 2024-06-29T15:57:09.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.898+0000 7f3c67adf640 1 Processor -- start 2024-06-29T15:57:09.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.898+0000 7f3c67adf640 1 -- start start 2024-06-29T15:57:09.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.899+0000 7f3c67adf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c6013e9b0 0x7f3c601560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.899+0000 7f3c67adf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c6015ceb0 0x7f3c601f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.899+0000 7f3c67adf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c601f33b0 0x7f3c601f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.899+0000 7f3c67adf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3c600b2a80 con 0x7f3c601f33b0 2024-06-29T15:57:09.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.899+0000 7f3c67adf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c600b2900 con 0x7f3c6013e9b0 2024-06-29T15:57:09.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.899+0000 7f3c67adf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3c600b2c00 con 0x7f3c6015ceb0 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.899+0000 7f3c6607c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c601f33b0 0x7f3c601f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.899+0000 7f3c6607c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c601f33b0 0x7f3c601f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51910/0 (socket says 172.21.15.59:51910) 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6607c640 1 -- 172.21.15.59:0/547091529 learned_addr learned my addr 172.21.15.59:0/547091529 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6587b640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c6013e9b0 0x7f3c601560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6507a640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c6015ceb0 0x7f3c601f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6607c640 1 -- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c6015ceb0 msgr2=0x7f3c601f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6607c640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c6015ceb0 0x7f3c601f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6607c640 1 -- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c6013e9b0 msgr2=0x7f3c601560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6607c640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c6013e9b0 0x7f3c601560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6607c640 1 -- 172.21.15.59:0/547091529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c60153ef0 con 0x7f3c601f33b0 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c6607c640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c601f33b0 0x7f3c601f2ce0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800c750 tx=0x7f3c5800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c5e7fc640 1 -- 172.21.15.59:0/547091529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5800ce60 con 0x7f3c601f33b0 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c5e7fc640 1 -- 172.21.15.59:0/547091529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c58004340 con 0x7f3c601f33b0 2024-06-29T15:57:09.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c601541d0 con 0x7f3c601f33b0 2024-06-29T15:57:09.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c5e7fc640 1 -- 172.21.15.59:0/547091529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c58007c10 con 0x7f3c601f33b0 2024-06-29T15:57:09.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.900+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c60154790 con 0x7f3c601f33b0 2024-06-29T15:57:09.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.901+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c60156750 con 0x7f3c601f33b0 2024-06-29T15:57:09.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.902+0000 7f3c5e7fc640 1 -- 172.21.15.59:0/547091529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7f3c58003610 con 0x7f3c601f33b0 2024-06-29T15:57:09.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.903+0000 7f3c5e7fc640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3c4c076150 0x7f3c4c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.903+0000 7f3c5e7fc640 1 -- 172.21.15.59:0/547091529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7046+0+0 (secure 0 0 0) 0x7f3c58097d70 con 0x7f3c601f33b0 2024-06-29T15:57:09.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.903+0000 7f3c6587b640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3c4c076150 0x7f3c4c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.904+0000 7f3c6587b640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3c4c076150 0x7f3c4c078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c50034000 tx=0x7f3c50004ab0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:09.907+0000 7f3c5e7fc640 1 -- 172.21.15.59:0/547091529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3c58014070 con 0x7f3c601f33b0 2024-06-29T15:57:10.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.102+0000 7f3c5e7fc640 1 -- 172.21.15.59:0/547091529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 127) ==== 98247+0+0 (secure 0 0 0) 0x7f3c5805cf90 con 0x7f3c601f33b0 2024-06-29T15:57:10.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.139+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3c6013edc0 con 0x7f3c601f33b0 2024-06-29T15:57:10.140 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:57:10.140+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:57:10.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.143+0000 7f3c5e7fc640 1 -- 172.21.15.59:0/547091529 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v107) ==== 161+0+0 (secure 0 0 0) 0x7f3c58061400 con 0x7f3c601f33b0 2024-06-29T15:57:10.144 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:57:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3c4c076150 msgr2=0x7f3c4c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3c4c076150 0x7f3c4c078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c50034000 tx=0x7f3c50004ab0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c601f33b0 msgr2=0x7f3c601f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c601f33b0 0x7f3c601f2ce0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800c750 tx=0x7f3c5800cc10 comp rx=0 tx=0).stop 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 shutdown_connections 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3c4c076150 0x7f3c4c078660 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c601f33b0 0x7f3c601f2ce0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3c6015ceb0 0x7f3c601f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 --2- 172.21.15.59:0/547091529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c6013e9b0 0x7f3c601560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 >> 172.21.15.59:0/547091529 conn(0x7f3c600b6fb0 msgr2=0x7f3c600b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 shutdown_connections 2024-06-29T15:57:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.145+0000 7f3c67adf640 1 -- 172.21.15.59:0/547091529 wait complete. 2024-06-29T15:57:10.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:57:10.208 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:57:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.282+0000 7f1162168640 1 -- 172.21.15.59:0/3703221055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0b4080 msgr2=0x7f115c0b24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.282+0000 7f1162168640 1 --2- 172.21.15.59:0/3703221055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0b4080 0x7f115c0b24d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1150009850 tx=0x7f115002a510 comp rx=0 tx=0).stop 2024-06-29T15:57:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.282+0000 7f1162168640 1 -- 172.21.15.59:0/3703221055 shutdown_connections 2024-06-29T15:57:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.282+0000 7f1162168640 1 --2- 172.21.15.59:0/3703221055 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0b4080 0x7f115c0b24d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.282+0000 7f1162168640 1 -- 172.21.15.59:0/3703221055 >> 172.21.15.59:0/3703221055 conn(0x7f115c0d3e80 msgr2=0x7f115c0d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.283+0000 7f1162168640 1 -- 172.21.15.59:0/3703221055 shutdown_connections 2024-06-29T15:57:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.283+0000 7f1162168640 1 -- 172.21.15.59:0/3703221055 wait complete. 2024-06-29T15:57:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.284+0000 7f1162168640 1 Processor -- start 2024-06-29T15:57:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.284+0000 7f1162168640 1 -- start start 2024-06-29T15:57:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f1162168640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0d2880 0x7f115c0c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f1162168640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f115c162340 0x7f115c161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f1162168640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115c15b190 0x7f115c0b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f1162168640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f115c15eda0 con 0x7f115c162340 2024-06-29T15:57:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f1162168640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f115c15ec20 con 0x7f115c15b190 2024-06-29T15:57:10.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f1162168640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f115c15ef20 con 0x7f115c0d2880 2024-06-29T15:57:10.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f115c162340 0x7f115c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:10.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f115c162340 0x7f115c161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51912/0 (socket says 172.21.15.59:51912) 2024-06-29T15:57:10.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 -- 172.21.15.59:0/2385955959 learned_addr learned my addr 172.21.15.59:0/2385955959 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:10.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115b7fe640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0d2880 0x7f115c0c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115bfff640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115c15b190 0x7f115c0b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 -- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0d2880 msgr2=0x7f115c0c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0d2880 0x7f115c0c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 -- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115c15b190 msgr2=0x7f115c0b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115c15b190 0x7f115c0b8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 -- 172.21.15.59:0/2385955959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115c0c7de0 con 0x7f115c162340 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115b7fe640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0d2880 0x7f115c0c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.286+0000 7f115affd640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f115c162340 0x7f115c161150 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f1150009940 tx=0x7f1150003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:10.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.287+0000 7f1143fff640 1 -- 172.21.15.59:0/2385955959 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115003f070 con 0x7f115c162340 2024-06-29T15:57:10.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.287+0000 7f1143fff640 1 -- 172.21.15.59:0/2385955959 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1150004ca0 con 0x7f115c162340 2024-06-29T15:57:10.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.287+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115c0c80c0 con 0x7f115c162340 2024-06-29T15:57:10.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.287+0000 7f1143fff640 1 -- 172.21.15.59:0/2385955959 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150005c30 con 0x7f115c162340 2024-06-29T15:57:10.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.287+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f115c0c8680 con 0x7f115c162340 2024-06-29T15:57:10.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.288+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115c0c7b30 con 0x7f115c162340 2024-06-29T15:57:10.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.288+0000 7f1143fff640 1 -- 172.21.15.59:0/2385955959 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 127) ==== 98247+0+0 (secure 0 0 0) 0x7f1150042070 con 0x7f115c162340 2024-06-29T15:57:10.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.289+0000 7f1143fff640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1148076390 0x7f11480788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:10.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.289+0000 7f1143fff640 1 -- 172.21.15.59:0/2385955959 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 7132+0+0 (secure 0 0 0) 0x7f11500b8a80 con 0x7f115c162340 2024-06-29T15:57:10.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.289+0000 7f115b7fe640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1148076390 0x7f11480788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:10.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.290+0000 7f115b7fe640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1148076390 0x7f11480788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f114c0013c0 tx=0x7f114c001720 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:10.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.293+0000 7f1143fff640 1 -- 172.21.15.59:0/2385955959 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1150081700 con 0x7f115c162340 2024-06-29T15:57:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:10.525+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f115c0b3970 con 0x7f115c162340 2024-06-29T15:57:11.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.145+0000 7f1143fff640 1 -- 172.21.15.59:0/2385955959 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v108) ==== 104+0+0 (secure 0 0 0) 0x7f1150085a00 con 0x7f115c162340 2024-06-29T15:57:11.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1148076390 msgr2=0x7f11480788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1148076390 0x7f11480788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f114c0013c0 tx=0x7f114c001720 comp rx=0 tx=0).stop 2024-06-29T15:57:11.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f115c162340 msgr2=0x7f115c161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f115c162340 0x7f115c161150 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f1150009940 tx=0x7f1150003c10 comp rx=0 tx=0).stop 2024-06-29T15:57:11.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 shutdown_connections 2024-06-29T15:57:11.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1148076390 0x7f11480788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f115c15b190 0x7f115c0b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f115c162340 0x7f115c161150 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 --2- 172.21.15.59:0/2385955959 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f115c0d2880 0x7f115c0c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 >> 172.21.15.59:0/2385955959 conn(0x7f115c0d3e80 msgr2=0x7f115c15cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:11.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 shutdown_connections 2024-06-29T15:57:11.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.147+0000 7f1162168640 1 -- 172.21.15.59:0/2385955959 wait complete. 2024-06-29T15:57:11.162 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:57:11.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 -- 172.21.15.59:0/2447857093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91815dff0 msgr2=0x7fe9180d2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/2447857093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91815dff0 0x7fe9180d2780 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fe9180d18e0 tx=0x7fe90802a560 comp rx=0 tx=0).stop 2024-06-29T15:57:11.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 -- 172.21.15.59:0/2447857093 shutdown_connections 2024-06-29T15:57:11.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/2447857093 >> v2:172.21.15.153:3300/0 conn(0x7fe918162430 0x7fe9180ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/2447857093 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe918161ae0 0x7fe918161ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/2447857093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91815dff0 0x7fe9180d2780 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 -- 172.21.15.59:0/2447857093 >> 172.21.15.59:0/2447857093 conn(0x7fe9180d3e60 msgr2=0x7fe9180d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:11.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 -- 172.21.15.59:0/2447857093 shutdown_connections 2024-06-29T15:57:11.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.289+0000 7fe91d2a5640 1 -- 172.21.15.59:0/2447857093 wait complete. 2024-06-29T15:57:11.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.291+0000 7fe91d2a5640 1 Processor -- start 2024-06-29T15:57:11.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.291+0000 7fe91d2a5640 1 -- start start 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe91d2a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe91815dff0 0x7fe9180c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe91d2a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918161ae0 0x7fe9181f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe91d2a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe918162430 0x7fe918150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe91d2a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe9180cf0c0 con 0x7fe918162430 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe91d2a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe9180cef40 con 0x7fe918161ae0 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe91d2a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe9180cf240 con 0x7fe91815dff0 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe918162430 0x7fe918150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe918162430 0x7fe918150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51928/0 (socket says 172.21.15.59:51928) 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 -- 172.21.15.59:0/331625506 learned_addr learned my addr 172.21.15.59:0/331625506 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:11.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9167fc640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918161ae0 0x7fe9181f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe916ffd640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe91815dff0 0x7fe9180c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 -- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe91815dff0 msgr2=0x7fe9180c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe91815dff0 0x7fe9180c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 -- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918161ae0 msgr2=0x7fe9181f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918161ae0 0x7fe9181f4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 -- 172.21.15.59:0/331625506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9180ce950 con 0x7fe918162430 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.293+0000 7fe9177fe640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe918162430 0x7fe918150cd0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fe91000e990 tx=0x7fe91000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.294+0000 7fe90f7fe640 1 -- 172.21.15.59:0/331625506 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910004d20 con 0x7fe918162430 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.294+0000 7fe90f7fe640 1 -- 172.21.15.59:0/331625506 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe91000caa0 con 0x7fe918162430 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.294+0000 7fe90f7fe640 1 -- 172.21.15.59:0/331625506 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910005020 con 0x7fe918162430 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.294+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9181f5250 con 0x7fe918162430 2024-06-29T15:57:11.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.294+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9180cb6c0 con 0x7fe918162430 2024-06-29T15:57:11.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.295+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe918161ef0 con 0x7fe918162430 2024-06-29T15:57:11.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.295+0000 7fe90f7fe640 1 -- 172.21.15.59:0/331625506 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 127) ==== 98247+0+0 (secure 0 0 0) 0x7fe9100051c0 con 0x7fe918162430 2024-06-29T15:57:11.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.296+0000 7fe90f7fe640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fe8f8076360 0x7fe8f8078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:11.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.296+0000 7fe90f7fe640 1 -- 172.21.15.59:0/331625506 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 7132+0+0 (secure 0 0 0) 0x7fe910098760 con 0x7fe918162430 2024-06-29T15:57:11.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.296+0000 7fe916ffd640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fe8f8076360 0x7fe8f8078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:11.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.297+0000 7fe916ffd640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fe8f8076360 0x7fe8f8078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe908033000 tx=0x7fe908004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:11.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.300+0000 7fe90f7fe640 1 -- 172.21.15.59:0/331625506 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe910016030 con 0x7fe918162430 2024-06-29T15:57:11.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:11.491+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe9180ca8b0 con 0x7fe918162430 2024-06-29T15:57:12.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.153+0000 7fe90f7fe640 1 -- 172.21.15.59:0/331625506 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v213) ==== 181+0+0 (secure 0 0 0) 0x7fe9100619a0 con 0x7fe918162430 2024-06-29T15:57:12.154 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:57:12.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fe8f8076360 msgr2=0x7fe8f8078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fe8f8076360 0x7fe8f8078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe908033000 tx=0x7fe908004730 comp rx=0 tx=0).stop 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe918162430 msgr2=0x7fe918150cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe918162430 0x7fe918150cd0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fe91000e990 tx=0x7fe91000ee50 comp rx=0 tx=0).stop 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 shutdown_connections 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fe8f8076360 0x7fe8f8078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe918162430 0x7fe918150cd0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe918161ae0 0x7fe9181f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 --2- 172.21.15.59:0/331625506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe91815dff0 0x7fe9180c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.155+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 >> 172.21.15.59:0/331625506 conn(0x7fe9180d3e60 msgr2=0x7fe91815d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.156+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 shutdown_connections 2024-06-29T15:57:12.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.156+0000 7fe91d2a5640 1 -- 172.21.15.59:0/331625506 wait complete. 2024-06-29T15:57:12.173 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:57:12.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.297+0000 7fb2d3343640 1 -- 172.21.15.59:0/3448102020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 msgr2=0x7fb2cc0b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.297+0000 7fb2d3343640 1 --2- 172.21.15.59:0/3448102020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 0x7fb2cc0b5cb0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0d0ea0 tx=0x7fb2c402a560 comp rx=0 tx=0).stop 2024-06-29T15:57:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.297+0000 7fb2d3343640 1 -- 172.21.15.59:0/3448102020 shutdown_connections 2024-06-29T15:57:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.297+0000 7fb2d3343640 1 --2- 172.21.15.59:0/3448102020 >> v2:172.21.15.153:3300/0 conn(0x7fb2cc0d1130 0x7fb2cc15f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.297+0000 7fb2d3343640 1 --2- 172.21.15.59:0/3448102020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 0x7fb2cc0b5cb0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.297+0000 7fb2d3343640 1 -- 172.21.15.59:0/3448102020 >> 172.21.15.59:0/3448102020 conn(0x7fb2cc0b6ed0 msgr2=0x7fb2cc0b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.297+0000 7fb2d3343640 1 -- 172.21.15.59:0/3448102020 shutdown_connections 2024-06-29T15:57:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.297+0000 7fb2d3343640 1 -- 172.21.15.59:0/3448102020 wait complete. 2024-06-29T15:57:12.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.299+0000 7fb2d3343640 1 Processor -- start 2024-06-29T15:57:12.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.299+0000 7fb2d3343640 1 -- start start 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.300+0000 7fb2d3343640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2cc0d1130 0x7fb2cc1f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.300+0000 7fb2d3343640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 0x7fb2cc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.300+0000 7fb2d3343640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2cc15fd30 0x7fb2cc158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.300+0000 7fb2d3343640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc15c910 con 0x7fb2cc145650 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.300+0000 7fb2d3343640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc15eba0 con 0x7fb2cc0d1130 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.300+0000 7fb2d3343640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc15ca90 con 0x7fb2cc15fd30 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 0x7fb2cc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 0x7fb2cc0ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51940/0 (socket says 172.21.15.59:51940) 2024-06-29T15:57:12.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 -- 172.21.15.59:0/1537987071 learned_addr learned my addr 172.21.15.59:0/1537987071 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d10df640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2cc0d1130 0x7fb2cc1f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d18e0640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2cc15fd30 0x7fb2cc158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 -- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2cc15fd30 msgr2=0x7fb2cc158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2cc15fd30 0x7fb2cc158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 -- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2cc0d1130 msgr2=0x7fb2cc1f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2cc0d1130 0x7fb2cc1f6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 -- 172.21.15.59:0/1537987071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc0c85d0 con 0x7fb2cc145650 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d18e0640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2cc15fd30 0x7fb2cc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d08de640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 0x7fb2cc0ce190 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000d6d0 tx=0x7fb2c000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2c9ffb640 1 -- 172.21.15.59:0/1537987071 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0017070 con 0x7fb2cc145650 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2c9ffb640 1 -- 172.21.15.59:0/1537987071 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2c0005330 con 0x7fb2cc145650 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2cc1f6db0 con 0x7fb2cc145650 2024-06-29T15:57:12.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.301+0000 7fb2c9ffb640 1 -- 172.21.15.59:0/1537987071 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0009820 con 0x7fb2cc145650 2024-06-29T15:57:12.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.302+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2cc0c8b90 con 0x7fb2cc145650 2024-06-29T15:57:12.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.303+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2cc15cff0 con 0x7fb2cc145650 2024-06-29T15:57:12.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.303+0000 7fb2c9ffb640 1 -- 172.21.15.59:0/1537987071 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fb2c0009bd0 con 0x7fb2cc145650 2024-06-29T15:57:12.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.304+0000 7fb2c9ffb640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb2b80763f0 0x7fb2b8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:12.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.304+0000 7fb2c9ffb640 1 -- 172.21.15.59:0/1537987071 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 6666+0+0 (secure 0 0 0) 0x7fb2c0098090 con 0x7fb2cc145650 2024-06-29T15:57:12.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.304+0000 7fb2d10df640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb2b80763f0 0x7fb2b8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:12.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.305+0000 7fb2d10df640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb2b80763f0 0x7fb2b8078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4033000 tx=0x7fb2c4004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:12.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.308+0000 7fb2c9ffb640 1 -- 172.21.15.59:0/1537987071 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb2c0060d30 con 0x7fb2cc145650 2024-06-29T15:57:12.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:12.501+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb2cc15b990 con 0x7fb2cc145650 2024-06-29T15:57:13.079 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:57:13.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:57:13.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.158+0000 7fb2c9ffb640 1 -- 172.21.15.59:0/1537987071 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v214) ==== 169+0+0 (secure 0 0 0) 0x7fb2c0069400 con 0x7fb2cc145650 2024-06-29T15:57:13.159 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb2b80763f0 msgr2=0x7fb2b8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb2b80763f0 0x7fb2b8078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4033000 tx=0x7fb2c4004730 comp rx=0 tx=0).stop 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 msgr2=0x7fb2cc0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 0x7fb2cc0ce190 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000d6d0 tx=0x7fb2c000db90 comp rx=0 tx=0).stop 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 shutdown_connections 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb2b80763f0 0x7fb2b8078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2cc15fd30 0x7fb2cc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2cc145650 0x7fb2cc0ce190 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 --2- 172.21.15.59:0/1537987071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2cc0d1130 0x7fb2cc1f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 >> 172.21.15.59:0/1537987071 conn(0x7fb2cc0b6ed0 msgr2=0x7fb2cc0b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:13.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 shutdown_connections 2024-06-29T15:57:13.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.163+0000 7fb2d3343640 1 -- 172.21.15.59:0/1537987071 wait complete. 2024-06-29T15:57:13.177 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.338+0000 7f7a4b51c640 1 -- 172.21.15.59:0/287848423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a440b3ed0 msgr2=0x7f7a4415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.338+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/287848423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a440b3ed0 0x7f7a4415e8d0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f7a440d15d0 tx=0x7f7a3402a560 comp rx=0 tx=0).stop 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.339+0000 7f7a4b51c640 1 -- 172.21.15.59:0/287848423 shutdown_connections 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.339+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/287848423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a4415ee10 0x7f7a4415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.339+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/287848423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a440b3ed0 0x7f7a4415e8d0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.339+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/287848423 >> v2:172.21.15.153:3300/0 conn(0x7f7a440d23c0 0x7f7a440b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.339+0000 7f7a4b51c640 1 -- 172.21.15.59:0/287848423 >> 172.21.15.59:0/287848423 conn(0x7f7a440d3ee0 msgr2=0x7f7a440d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.339+0000 7f7a4b51c640 1 -- 172.21.15.59:0/287848423 shutdown_connections 2024-06-29T15:57:13.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.339+0000 7f7a4b51c640 1 -- 172.21.15.59:0/287848423 wait complete. 2024-06-29T15:57:13.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.340+0000 7f7a4b51c640 1 Processor -- start 2024-06-29T15:57:13.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.340+0000 7f7a4b51c640 1 -- start start 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a4b51c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a440b3ed0 0x7f7a441f4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a4b51c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a440d23c0 0x7f7a441e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a4b51c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a4415ee10 0x7f7a441e8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a4b51c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7a44160070 con 0x7f7a4415ee10 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a4b51c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4415fef0 con 0x7f7a440d23c0 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a4b51c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7a441601f0 con 0x7f7a440b3ed0 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a4415ee10 0x7f7a441e8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a4415ee10 0x7f7a441e8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59404/0 (socket says 172.21.15.59:59404) 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 -- 172.21.15.59:0/1248499810 learned_addr learned my addr 172.21.15.59:0/1248499810 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a492b8640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a440b3ed0 0x7f7a441f4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a48ab7640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a440d23c0 0x7f7a441e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 -- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a440b3ed0 msgr2=0x7f7a441f4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a440b3ed0 0x7f7a441f4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 -- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a440d23c0 msgr2=0x7f7a441e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a440d23c0 0x7f7a441e6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 -- 172.21.15.59:0/1248499810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a441f5450 con 0x7f7a4415ee10 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a48ab7640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a440d23c0 0x7f7a441e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a49ab9640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a4415ee10 0x7f7a441e8b60 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00cab0 tx=0x7f7a3c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.342+0000 7f7a39ffb640 1 -- 172.21.15.59:0/1248499810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c007d90 con 0x7f7a4415ee10 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.343+0000 7f7a39ffb640 1 -- 172.21.15.59:0/1248499810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a3c00f560 con 0x7f7a4415ee10 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.343+0000 7f7a39ffb640 1 -- 172.21.15.59:0/1248499810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c00f9b0 con 0x7f7a4415ee10 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.343+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a441e7ac0 con 0x7f7a4415ee10 2024-06-29T15:57:13.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.343+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a441e8080 con 0x7f7a4415ee10 2024-06-29T15:57:13.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.344+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a440b3990 con 0x7f7a4415ee10 2024-06-29T15:57:13.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.344+0000 7f7a39ffb640 1 -- 172.21.15.59:0/1248499810 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f7a3c00fb50 con 0x7f7a4415ee10 2024-06-29T15:57:13.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.345+0000 7f7a39ffb640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7a280763f0 0x7f7a28078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.345+0000 7f7a39ffb640 1 -- 172.21.15.59:0/1248499810 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7f7a3c0985b0 con 0x7f7a4415ee10 2024-06-29T15:57:13.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.345+0000 7f7a492b8640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7a280763f0 0x7f7a28078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.346+0000 7f7a492b8640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7a280763f0 0x7f7a28078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7a40005c80 tx=0x7f7a40005c10 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:13.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.349+0000 7f7a39ffb640 1 -- 172.21.15.59:0/1248499810 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7a3c014070 con 0x7f7a4415ee10 2024-06-29T15:57:13.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.540+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f7a441e7850 con 0x7f7a4415ee10 2024-06-29T15:57:13.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.544+0000 7f7a39ffb640 1 -- 172.21.15.59:0/1248499810 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v71) ==== 71+0+0 (secure 0 0 0) 0x7f7a3c061a90 con 0x7f7a4415ee10 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.546+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7a280763f0 msgr2=0x7f7a28078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.546+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7a280763f0 0x7f7a28078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7a40005c80 tx=0x7f7a40005c10 comp rx=0 tx=0).stop 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a4415ee10 msgr2=0x7f7a441e8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a4415ee10 0x7f7a441e8b60 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00cab0 tx=0x7f7a3c00cf70 comp rx=0 tx=0).stop 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 shutdown_connections 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7a280763f0 0x7f7a28078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a4415ee10 0x7f7a441e8b60 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a440d23c0 0x7f7a441e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 --2- 172.21.15.59:0/1248499810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7a440b3ed0 0x7f7a441f4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 >> 172.21.15.59:0/1248499810 conn(0x7f7a440d3ee0 msgr2=0x7f7a44160730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:13.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 shutdown_connections 2024-06-29T15:57:13.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.547+0000 7f7a4b51c640 1 -- 172.21.15.59:0/1248499810 wait complete. 2024-06-29T15:57:13.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-06-29T15:57:13.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.675+0000 7fd5f00ac640 1 -- 172.21.15.59:0/2432305457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e81414c0 msgr2=0x7fd5e81418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.675+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/2432305457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e81414c0 0x7fd5e81418d0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0005fd0 tx=0x7fd5e002a540 comp rx=0 tx=0).stop 2024-06-29T15:57:13.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.675+0000 7fd5f00ac640 1 -- 172.21.15.59:0/2432305457 shutdown_connections 2024-06-29T15:57:13.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.675+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/2432305457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e81414c0 0x7fd5e81418d0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.675+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/2432305457 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e813e9b0 0x7fd5e813edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.675+0000 7fd5f00ac640 1 -- 172.21.15.59:0/2432305457 >> 172.21.15.59:0/2432305457 conn(0x7fd5e80b6fb0 msgr2=0x7fd5e80b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:13.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.675+0000 7fd5f00ac640 1 -- 172.21.15.59:0/2432305457 shutdown_connections 2024-06-29T15:57:13.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.675+0000 7fd5f00ac640 1 -- 172.21.15.59:0/2432305457 wait complete. 2024-06-29T15:57:13.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.677+0000 7fd5f00ac640 1 Processor -- start 2024-06-29T15:57:13.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.677+0000 7fd5f00ac640 1 -- start start 2024-06-29T15:57:13.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5f00ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e813e9b0 0x7fd5e8153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5f00ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e81414c0 0x7fd5e813b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5f00ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e815ceb0 0x7fd5e8157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5f00ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e80b2a80 con 0x7fd5e815ceb0 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5f00ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e80b2900 con 0x7fd5e81414c0 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5f00ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e80b2c00 con 0x7fd5e813e9b0 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5ee649640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e815ceb0 0x7fd5e8157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5ee649640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e815ceb0 0x7fd5e8157ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59420/0 (socket says 172.21.15.59:59420) 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.678+0000 7fd5ee649640 1 -- 172.21.15.59:0/3871696733 learned_addr learned my addr 172.21.15.59:0/3871696733 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5ede48640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e813e9b0 0x7fd5e8153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5ed647640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e81414c0 0x7fd5e813b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5ee649640 1 -- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e813e9b0 msgr2=0x7fd5e8153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5ee649640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e813e9b0 0x7fd5e8153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5ee649640 1 -- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e81414c0 msgr2=0x7fd5e813b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5ee649640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e81414c0 0x7fd5e813b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5ee649640 1 -- 172.21.15.59:0/3871696733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e8156d00 con 0x7fd5e815ceb0 2024-06-29T15:57:13.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5ee649640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e815ceb0 0x7fd5e8157ac0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009230 tx=0x7fd5e0003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:13.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5e67fc640 1 -- 172.21.15.59:0/3871696733 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e003e030 con 0x7fd5e815ceb0 2024-06-29T15:57:13.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5e67fc640 1 -- 172.21.15.59:0/3871696733 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5e0003e50 con 0x7fd5e815ceb0 2024-06-29T15:57:13.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5e67fc640 1 -- 172.21.15.59:0/3871696733 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0035040 con 0x7fd5e815ceb0 2024-06-29T15:57:13.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e8156fe0 con 0x7fd5e815ceb0 2024-06-29T15:57:13.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.679+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e81575a0 con 0x7fd5e815ceb0 2024-06-29T15:57:13.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.680+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e815f550 con 0x7fd5e815ceb0 2024-06-29T15:57:13.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.681+0000 7fd5e67fc640 1 -- 172.21.15.59:0/3871696733 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fd5e0041030 con 0x7fd5e815ceb0 2024-06-29T15:57:13.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.682+0000 7fd5e67fc640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd5d40763f0 0x7fd5d4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.682+0000 7fd5e67fc640 1 -- 172.21.15.59:0/3871696733 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7fd5e00b8310 con 0x7fd5e815ceb0 2024-06-29T15:57:13.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.682+0000 7fd5ede48640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd5d40763f0 0x7fd5d4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.683+0000 7fd5ede48640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd5d40763f0 0x7fd5d4078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8001e20 tx=0x7fd5d8009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:13.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.686+0000 7fd5e67fc640 1 -- 172.21.15.59:0/3871696733 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd5e815f550 con 0x7fd5e815ceb0 2024-06-29T15:57:13.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:13.957+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) -- 0x7fd5e80701e0 con 0x7fd5e815ceb0 2024-06-29T15:57:14.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.160+0000 7fd5e67fc640 1 -- 172.21.15.59:0/3871696733 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3110) ==== 163+0+0 (secure 0 0 0) 0x7fd5e80701e0 con 0x7fd5e815ceb0 2024-06-29T15:57:14.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd5d40763f0 msgr2=0x7fd5d4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd5d40763f0 0x7fd5d4078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8001e20 tx=0x7fd5d8009040 comp rx=0 tx=0).stop 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e815ceb0 msgr2=0x7fd5e8157ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e815ceb0 0x7fd5e8157ac0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009230 tx=0x7fd5e0003960 comp rx=0 tx=0).stop 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 shutdown_connections 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd5d40763f0 0x7fd5d4078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5e815ceb0 0x7fd5e8157ac0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5e81414c0 0x7fd5e813b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 --2- 172.21.15.59:0/3871696733 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd5e813e9b0 0x7fd5e8153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 >> 172.21.15.59:0/3871696733 conn(0x7fd5e80b6fb0 msgr2=0x7fd5e813e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.162+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 shutdown_connections 2024-06-29T15:57:14.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.163+0000 7fd5f00ac640 1 -- 172.21.15.59:0/3871696733 wait complete. 2024-06-29T15:57:14.175 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-06-29T15:57:14.175 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-06-29T15:57:14.175 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-06-29T15:57:14.176 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.300+0000 7fa4de1b2640 1 -- 172.21.15.59:0/1315923499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4c0004040 msgr2=0x7fa4c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.300+0000 7fa4de1b2640 1 --2- 172.21.15.59:0/1315923499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4c0004040 0x7fa4c0004800 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009b50 tx=0x7fa4c0005830 comp rx=0 tx=0).stop 2024-06-29T15:57:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.300+0000 7fa4d7fff640 1 -- 172.21.15.59:0/1315923499 reap_dead start 2024-06-29T15:57:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.300+0000 7fa4de1b2640 1 -- 172.21.15.59:0/1315923499 shutdown_connections 2024-06-29T15:57:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.300+0000 7fa4de1b2640 1 -- 172.21.15.59:0/1315923499 >> 172.21.15.59:0/1315923499 conn(0x7fa4d80b5610 msgr2=0x7fa4d80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.300+0000 7fa4de1b2640 1 -- 172.21.15.59:0/1315923499 shutdown_connections 2024-06-29T15:57:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.300+0000 7fa4de1b2640 1 -- 172.21.15.59:0/1315923499 wait complete. 2024-06-29T15:57:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.302+0000 7fa4de1b2640 1 Processor -- start 2024-06-29T15:57:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.302+0000 7fa4de1b2640 1 -- start start 2024-06-29T15:57:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4de1b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c0004040 0x7fa4d8154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4de1b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d80ccdd0 0x7fa4d815ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4de1b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4d815cfb0 0x7fa4d8155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4de1b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d8162650 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4de1b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d81624d0 con 0x7fa4d815cfb0 2024-06-29T15:57:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4de1b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d80b7bd0 con 0x7fa4c0004040 2024-06-29T15:57:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d80ccdd0 0x7fa4d815ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d80ccdd0 0x7fa4d815ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59434/0 (socket says 172.21.15.59:59434) 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 -- 172.21.15.59:0/3533604003 learned_addr learned my addr 172.21.15.59:0/3533604003 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d77fe640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c0004040 0x7fa4d8154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d7fff640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4d815cfb0 0x7fa4d8155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 -- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c0004040 msgr2=0x7fa4d8154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c0004040 0x7fa4d8154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 -- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4d815cfb0 msgr2=0x7fa4d8155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4d815cfb0 0x7fa4d8155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 -- 172.21.15.59:0/3533604003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d815d3c0 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4d6ffd640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d80ccdd0 0x7fa4d815ca70 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc036040 tx=0x7fa4cc0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.304+0000 7fa4bffff640 1 -- 172.21.15.59:0/3533604003 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc038040 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.305+0000 7fa4bffff640 1 -- 172.21.15.59:0/3533604003 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4cc0338e0 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.305+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d8157df0 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.305+0000 7fa4bffff640 1 -- 172.21.15.59:0/3533604003 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc033d10 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.305+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d8154e40 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.305+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d8162c10 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.306+0000 7fa4bffff640 1 -- 172.21.15.59:0/3533604003 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fa4cc043020 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.307+0000 7fa4bffff640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4c40763f0 0x7fa4c4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.307+0000 7fa4bffff640 1 -- 172.21.15.59:0/3533604003 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7fa4cc0b7bf0 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.307+0000 7fa4d77fe640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4c40763f0 0x7fa4c4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.308+0000 7fa4d77fe640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4c40763f0 0x7fa4c4078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8001d70 tx=0x7fa4c8001d00 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.311+0000 7fa4bffff640 1 -- 172.21.15.59:0/3533604003 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa4cc04a020 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.505+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4d80cf570 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.505+0000 7fa4bffff640 1 -- 172.21.15.59:0/3533604003 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) ==== 74+0+14549 (secure 0 0 0) 0x7fa4d815d3c0 con 0x7fa4d80ccdd0 2024-06-29T15:57:14.506 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:14.507 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":214,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:13.154723+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3494626796":"2024-06-30T15:57:07.739872+0000","172.21.15.153:6833/1875870697":"2024-06-30T15:57:07.739872+0000","172.21.15.153:6834/2163190416":"2024-06-30T15:57:10.141264+0000","172.21.15.153:0/1193885100":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/3120983149":"2024-06-30T15:57:07.739872+0000","172.21.15.153:6836/2163190416":"2024-06-30T15:57:10.141264+0000","172.21.15.153:0/1750092138":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/3166749975":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/1068787394":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/4060660749":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/2009747699":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/2580944857":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/4008855742":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6833/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6832/493926731":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/743192862":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/719772475":"2024-06-30T15:55:49.819811+0000","172.21.15.153:6832/1875870697":"2024-06-30T15:57:07.739872+0000","172.21.15.153:0/4087971292":"2024-06-30T15:55:49.819811+0000","172.21.15.153:0/3198120249":"2024-06-30T15:55:49.819811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4c40763f0 msgr2=0x7fa4c4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4c40763f0 0x7fa4c4078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8001d70 tx=0x7fa4c8001d00 comp rx=0 tx=0).stop 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d80ccdd0 msgr2=0x7fa4d815ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d80ccdd0 0x7fa4d815ca70 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc036040 tx=0x7fa4cc0336f0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 shutdown_connections 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4c40763f0 0x7fa4c4078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4d815cfb0 0x7fa4d8155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4d80ccdd0 0x7fa4d815ca70 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 --2- 172.21.15.59:0/3533604003 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa4c0004040 0x7fa4d8154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 >> 172.21.15.59:0/3533604003 conn(0x7fa4d80b5610 msgr2=0x7fa4d815faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 shutdown_connections 2024-06-29T15:57:14.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.508+0000 7fa4de1b2640 1 -- 172.21.15.59:0/3533604003 wait complete. 2024-06-29T15:57:14.520 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:14.520 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:14.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 -- 172.21.15.59:0/1264618698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 msgr2=0x7fa51c15c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 --2- 172.21.15.59:0/1264618698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 0x7fa51c15c700 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0cf610 tx=0x7fa51402a560 comp rx=0 tx=0).stop 2024-06-29T15:57:14.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 -- 172.21.15.59:0/1264618698 shutdown_connections 2024-06-29T15:57:14.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 --2- 172.21.15.59:0/1264618698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 0x7fa51c15c700 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 --2- 172.21.15.59:0/1264618698 >> v2:172.21.15.153:3300/0 conn(0x7fa51c0b3740 0x7fa51c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 --2- 172.21.15.59:0/1264618698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa51c1458a0 0x7fa51c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 -- 172.21.15.59:0/1264618698 >> 172.21.15.59:0/1264618698 conn(0x7fa51c06c950 msgr2=0x7fa51c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 -- 172.21.15.59:0/1264618698 shutdown_connections 2024-06-29T15:57:14.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.641+0000 7fa5219ee640 1 -- 172.21.15.59:0/1264618698 wait complete. 2024-06-29T15:57:14.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.643+0000 7fa5219ee640 1 Processor -- start 2024-06-29T15:57:14.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.643+0000 7fa5219ee640 1 -- start start 2024-06-29T15:57:14.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa5219ee640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c0b3740 0x7fa51c15e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa5219ee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 0x7fa51c140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa5219ee640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa51c06e560 0x7fa51c1437f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa5219ee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c15ff40 con 0x7fa51c15c2f0 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa5219ee640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c15fdc0 con 0x7fa51c0b3740 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa5219ee640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa51c1600c0 con 0x7fa51c06e560 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa51a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 0x7fa51c140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa51a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 0x7fa51c140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59444/0 (socket says 172.21.15.59:59444) 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa51a7fc640 1 -- 172.21.15.59:0/526010615 learned_addr learned my addr 172.21.15.59:0/526010615 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa51affd640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c0b3740 0x7fa51c15e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.644+0000 7fa51b7fe640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa51c06e560 0x7fa51c1437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa51a7fc640 1 -- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa51c06e560 msgr2=0x7fa51c1437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa51a7fc640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa51c06e560 0x7fa51c1437f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa51a7fc640 1 -- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c0b3740 msgr2=0x7fa51c15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa51a7fc640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c0b3740 0x7fa51c15e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa51a7fc640 1 -- 172.21.15.59:0/526010615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c06ec90 con 0x7fa51c15c2f0 2024-06-29T15:57:14.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa51a7fc640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 0x7fa51c140070 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fa50400d6c0 tx=0x7fa50400db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa513fff640 1 -- 172.21.15.59:0/526010615 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504017070 con 0x7fa51c15c2f0 2024-06-29T15:57:14.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa513fff640 1 -- 172.21.15.59:0/526010615 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa504004310 con 0x7fa51c15c2f0 2024-06-29T15:57:14.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51c06ef70 con 0x7fa51c15c2f0 2024-06-29T15:57:14.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa513fff640 1 -- 172.21.15.59:0/526010615 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504009a80 con 0x7fa51c15c2f0 2024-06-29T15:57:14.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.645+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa51c06f530 con 0x7fa51c15c2f0 2024-06-29T15:57:14.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.646+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa51c1605e0 con 0x7fa51c15c2f0 2024-06-29T15:57:14.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.647+0000 7fa513fff640 1 -- 172.21.15.59:0/526010615 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fa504009c20 con 0x7fa51c15c2f0 2024-06-29T15:57:14.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.647+0000 7fa513fff640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4fc076420 0x7fa4fc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.647+0000 7fa513fff640 1 -- 172.21.15.59:0/526010615 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7fa5040979a0 con 0x7fa51c15c2f0 2024-06-29T15:57:14.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.648+0000 7fa51affd640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4fc076420 0x7fa4fc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.648+0000 7fa51affd640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4fc076420 0x7fa4fc078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa50c005fd0 tx=0x7fa50c0095e0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.652+0000 7fa513fff640 1 -- 172.21.15.59:0/526010615 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa504060830 con 0x7fa51c15c2f0 2024-06-29T15:57:14.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.844+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa51c0b1f10 con 0x7fa51c15c2f0 2024-06-29T15:57:14.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.845+0000 7fa513fff640 1 -- 172.21.15.59:0/526010615 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v214) ==== 93+0+10 (secure 0 0 0) 0x7fa504064b30 con 0x7fa51c15c2f0 2024-06-29T15:57:14.846 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:57:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4fc076420 msgr2=0x7fa4fc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4fc076420 0x7fa4fc078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa50c005fd0 tx=0x7fa50c0095e0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 msgr2=0x7fa51c140070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 0x7fa51c140070 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fa50400d6c0 tx=0x7fa50400db80 comp rx=0 tx=0).stop 2024-06-29T15:57:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 shutdown_connections 2024-06-29T15:57:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fa4fc076420 0x7fa4fc078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa51c06e560 0x7fa51c1437f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa51c15c2f0 0x7fa51c140070 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 --2- 172.21.15.59:0/526010615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa51c0b3740 0x7fa51c15e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 >> 172.21.15.59:0/526010615 conn(0x7fa51c06c950 msgr2=0x7fa51c1414e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 shutdown_connections 2024-06-29T15:57:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.847+0000 7fa5219ee640 1 -- 172.21.15.59:0/526010615 wait complete. 2024-06-29T15:57:14.859 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-06-29T15:57:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.976+0000 7fc08d9a1640 1 -- 172.21.15.59:0/3160955799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc088162800 msgr2=0x7fc088162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.976+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/3160955799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc088162800 0x7fc088162c10 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fc080005fd0 tx=0x7fc08002aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.976+0000 7fc08d9a1640 1 -- 172.21.15.59:0/3160955799 shutdown_connections 2024-06-29T15:57:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.976+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/3160955799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc088162800 0x7fc088162c10 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.976+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/3160955799 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0880ce950 0x7fc0881622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.976+0000 7fc08d9a1640 1 -- 172.21.15.59:0/3160955799 >> 172.21.15.59:0/3160955799 conn(0x7fc0880b5610 msgr2=0x7fc0880b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.976+0000 7fc08d9a1640 1 -- 172.21.15.59:0/3160955799 shutdown_connections 2024-06-29T15:57:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.976+0000 7fc08d9a1640 1 -- 172.21.15.59:0/3160955799 wait complete. 2024-06-29T15:57:14.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.978+0000 7fc08d9a1640 1 Processor -- start 2024-06-29T15:57:14.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.978+0000 7fc08d9a1640 1 -- start start 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc08d9a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc08808c190 0x7fc0881530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc08d9a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0880ce950 0x7fc0880ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc08d9a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc088162800 0x7fc0881f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc08d9a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc0880b7fa0 con 0x7fc0880ce950 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc08d9a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0880b7e20 con 0x7fc088162800 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc08d9a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc0880b8120 con 0x7fc08808c190 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0867fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0880ce950 0x7fc0880ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0867fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0880ce950 0x7fc0880ab820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59462/0 (socket says 172.21.15.59:59462) 2024-06-29T15:57:14.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0867fc640 1 -- 172.21.15.59:0/4134405099 learned_addr learned my addr 172.21.15.59:0/4134405099 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc086ffd640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc08808c190 0x7fc0881530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0877fe640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc088162800 0x7fc0881f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0867fc640 1 -- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc08808c190 msgr2=0x7fc0881530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0867fc640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc08808c190 0x7fc0881530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0867fc640 1 -- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc088162800 msgr2=0x7fc0881f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0867fc640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc088162800 0x7fc0881f53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.980+0000 7fc0867fc640 1 -- 172.21.15.59:0/4134405099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0880abd60 con 0x7fc0880ce950 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.981+0000 7fc0867fc640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0880ce950 0x7fc0880ab820 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00d6d0 tx=0x7fc07c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.981+0000 7fc06ffff640 1 -- 172.21.15.59:0/4134405099 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07c017070 con 0x7fc0880ce950 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.981+0000 7fc06ffff640 1 -- 172.21.15.59:0/4134405099 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc07c005330 con 0x7fc0880ce950 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.981+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0880cdbc0 con 0x7fc0880ce950 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.981+0000 7fc06ffff640 1 -- 172.21.15.59:0/4134405099 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07c009820 con 0x7fc0880ce950 2024-06-29T15:57:14.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.981+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0880aad40 con 0x7fc0880ce950 2024-06-29T15:57:14.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.982+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0880d2c40 con 0x7fc0880ce950 2024-06-29T15:57:14.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.983+0000 7fc06ffff640 1 -- 172.21.15.59:0/4134405099 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fc07c0099c0 con 0x7fc0880ce950 2024-06-29T15:57:14.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.983+0000 7fc06ffff640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc074076420 0x7fc074078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.983+0000 7fc06ffff640 1 -- 172.21.15.59:0/4134405099 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7fc07c097ba0 con 0x7fc0880ce950 2024-06-29T15:57:14.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.983+0000 7fc086ffd640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc074076420 0x7fc074078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.984+0000 7fc086ffd640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc074076420 0x7fc074078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc0880b7620 tx=0x7fc078009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:14.988+0000 7fc06ffff640 1 -- 172.21.15.59:0/4134405099 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0880d2c40 con 0x7fc0880ce950 2024-06-29T15:57:15.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:15.253+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) -- 0x7fc0881622c0 con 0x7fc0880ce950 2024-06-29T15:57:16.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.167+0000 7fc06ffff640 1 -- 172.21.15.59:0/4134405099 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3112) ==== 189+0+0 (secure 0 0 0) 0x7fc07c069100 con 0x7fc0880ce950 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.169+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc074076420 msgr2=0x7fc074078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.169+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc074076420 0x7fc074078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc0880b7620 tx=0x7fc078009040 comp rx=0 tx=0).stop 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.169+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0880ce950 msgr2=0x7fc0880ab820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.169+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0880ce950 0x7fc0880ab820 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00d6d0 tx=0x7fc07c00db90 comp rx=0 tx=0).stop 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.170+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 shutdown_connections 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.170+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc074076420 0x7fc074078930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.170+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc088162800 0x7fc0881f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.170+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0880ce950 0x7fc0880ab820 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.170+0000 7fc08d9a1640 1 --2- 172.21.15.59:0/4134405099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc08808c190 0x7fc0881530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.170+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 >> 172.21.15.59:0/4134405099 conn(0x7fc0880b5610 msgr2=0x7fc08806caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.170+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 shutdown_connections 2024-06-29T15:57:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.170+0000 7fc08d9a1640 1 -- 172.21.15.59:0/4134405099 wait complete. 2024-06-29T15:57:16.184 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:57:16.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.309+0000 7fb19b04e640 1 -- 172.21.15.59:0/2572513292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1941458a0 msgr2=0x7fb1940cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.309+0000 7fb19b04e640 1 --2- 172.21.15.59:0/2572513292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1941458a0 0x7fb1940cf870 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fb188009830 tx=0x7fb18802a5c0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.309+0000 7fb19b04e640 1 -- 172.21.15.59:0/2572513292 shutdown_connections 2024-06-29T15:57:16.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.309+0000 7fb19b04e640 1 --2- 172.21.15.59:0/2572513292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1941458a0 0x7fb1940cf870 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.309+0000 7fb19b04e640 1 -- 172.21.15.59:0/2572513292 >> 172.21.15.59:0/2572513292 conn(0x7fb19406c950 msgr2=0x7fb194073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.309+0000 7fb19b04e640 1 -- 172.21.15.59:0/2572513292 shutdown_connections 2024-06-29T15:57:16.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.309+0000 7fb19b04e640 1 -- 172.21.15.59:0/2572513292 wait complete. 2024-06-29T15:57:16.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.311+0000 7fb19b04e640 1 Processor -- start 2024-06-29T15:57:16.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.311+0000 7fb19b04e640 1 -- start start 2024-06-29T15:57:16.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb19b04e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940b3740 0x7fb1940ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb19b04e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b3b50 0x7fb19415da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb19b04e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb19413dfa0 0x7fb19415df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb19b04e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb1940b6320 con 0x7fb19413dfa0 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb19b04e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1940b61a0 con 0x7fb1940b3740 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb19b04e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb1940b64a0 con 0x7fb1940b3b50 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb1995eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb19413dfa0 0x7fb19415df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb1995eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb19413dfa0 0x7fb19415df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59484/0 (socket says 172.21.15.59:59484) 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb1995eb640 1 -- 172.21.15.59:0/733268928 learned_addr learned my addr 172.21.15.59:0/733268928 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb198dea640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940b3740 0x7fb1940ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.312+0000 7fb193fff640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b3b50 0x7fb19415da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1995eb640 1 -- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b3b50 msgr2=0x7fb19415da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1995eb640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b3b50 0x7fb19415da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1995eb640 1 -- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940b3740 msgr2=0x7fb1940ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1995eb640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940b3740 0x7fb1940ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1995eb640 1 -- 172.21.15.59:0/733268928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19406ed30 con 0x7fb19413dfa0 2024-06-29T15:57:16.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb198dea640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940b3740 0x7fb1940ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:16.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1995eb640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb19413dfa0 0x7fb19415df70 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fb18c00cae0 tx=0x7fb18c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:16.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1917fa640 1 -- 172.21.15.59:0/733268928 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18c016020 con 0x7fb19413dfa0 2024-06-29T15:57:16.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1917fa640 1 -- 172.21.15.59:0/733268928 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb18c00ad30 con 0x7fb19413dfa0 2024-06-29T15:57:16.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb19406efd0 con 0x7fb19413dfa0 2024-06-29T15:57:16.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb1917fa640 1 -- 172.21.15.59:0/733268928 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18c012680 con 0x7fb19413dfa0 2024-06-29T15:57:16.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.313+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb19415d510 con 0x7fb19413dfa0 2024-06-29T15:57:16.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.314+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1941458a0 con 0x7fb19413dfa0 2024-06-29T15:57:16.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.315+0000 7fb1917fa640 1 -- 172.21.15.59:0/733268928 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fb18c019020 con 0x7fb19413dfa0 2024-06-29T15:57:16.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.315+0000 7fb1917fa640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb1780763f0 0x7fb178078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.315+0000 7fb1917fa640 1 -- 172.21.15.59:0/733268928 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7fb18c098000 con 0x7fb19413dfa0 2024-06-29T15:57:16.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.316+0000 7fb198dea640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb1780763f0 0x7fb178078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.316+0000 7fb198dea640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb1780763f0 0x7fb178078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb188034000 tx=0x7fb188004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:16.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.320+0000 7fb1917fa640 1 -- 172.21.15.59:0/733268928 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb18c020020 con 0x7fb19413dfa0 2024-06-29T15:57:16.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.511+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fb1940b1f10 con 0x7fb19413dfa0 2024-06-29T15:57:16.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.512+0000 7fb1917fa640 1 -- 172.21.15.59:0/733268928 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v71) ==== 59+0+1998 (secure 0 0 0) 0x7fb18c061670 con 0x7fb19413dfa0 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T15:57:16.513 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:57:16.514 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T15:57:16.515 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T15:57:16.515 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T15:57:16.515 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:57:16.515 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.513+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb1780763f0 msgr2=0x7fb178078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.513+0000 7fb19b04e640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb1780763f0 0x7fb178078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb188034000 tx=0x7fb188004730 comp rx=0 tx=0).stop 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.513+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb19413dfa0 msgr2=0x7fb19415df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.513+0000 7fb19b04e640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb19413dfa0 0x7fb19415df70 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fb18c00cae0 tx=0x7fb18c00a060 comp rx=0 tx=0).stop 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.514+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 shutdown_connections 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.514+0000 7fb19b04e640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb1780763f0 0x7fb178078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.514+0000 7fb19b04e640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb19413dfa0 0x7fb19415df70 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.514+0000 7fb19b04e640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b3b50 0x7fb19415da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.514+0000 7fb19b04e640 1 --2- 172.21.15.59:0/733268928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940b3740 0x7fb1940ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.514+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 >> 172.21.15.59:0/733268928 conn(0x7fb19406c950 msgr2=0x7fb194071610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.514+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 shutdown_connections 2024-06-29T15:57:16.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.514+0000 7fb19b04e640 1 -- 172.21.15.59:0/733268928 wait complete. 2024-06-29T15:57:16.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:57:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.646+0000 7f2ae3168640 1 -- 172.21.15.59:0/3822281081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2adc0d23c0 msgr2=0x7f2adc0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.646+0000 7f2ae3168640 1 --2- 172.21.15.59:0/3822281081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2adc0d23c0 0x7f2adc0b3990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009830 tx=0x7f2acc02a510 comp rx=0 tx=0).stop 2024-06-29T15:57:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.646+0000 7f2ae3168640 1 -- 172.21.15.59:0/3822281081 shutdown_connections 2024-06-29T15:57:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.646+0000 7f2ae3168640 1 --2- 172.21.15.59:0/3822281081 >> v2:172.21.15.153:3300/0 conn(0x7f2adc0b3ed0 0x7f2adc15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.646+0000 7f2ae3168640 1 --2- 172.21.15.59:0/3822281081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2adc0d23c0 0x7f2adc0b3990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.646+0000 7f2ae3168640 1 -- 172.21.15.59:0/3822281081 >> 172.21.15.59:0/3822281081 conn(0x7f2adc0d3ee0 msgr2=0x7f2adc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.646+0000 7f2ae3168640 1 -- 172.21.15.59:0/3822281081 shutdown_connections 2024-06-29T15:57:16.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.646+0000 7f2ae3168640 1 -- 172.21.15.59:0/3822281081 wait complete. 2024-06-29T15:57:16.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.647+0000 7f2ae3168640 1 Processor -- start 2024-06-29T15:57:16.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.648+0000 7f2ae3168640 1 -- start start 2024-06-29T15:57:16.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae3168640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2adc0b3ed0 0x7f2adc1e9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae3168640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2adc0d23c0 0x7f2adc1e9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae3168640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2adc15ee10 0x7f2adc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae3168640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc160240 con 0x7f2adc15ee10 2024-06-29T15:57:16.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae3168640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1600c0 con 0x7f2adc0b3ed0 2024-06-29T15:57:16.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae3168640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2adc1603c0 con 0x7f2adc0d23c0 2024-06-29T15:57:16.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae1705640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2adc15ee10 0x7f2adc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae1705640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2adc15ee10 0x7f2adc0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59488/0 (socket says 172.21.15.59:59488) 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae1705640 1 -- 172.21.15.59:0/896201527 learned_addr learned my addr 172.21.15.59:0/896201527 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2adbfff640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2adc0d23c0 0x7f2adc1e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.649+0000 7f2ae0f04640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2adc0b3ed0 0x7f2adc1e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae1705640 1 -- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2adc0d23c0 msgr2=0x7f2adc1e9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae1705640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2adc0d23c0 0x7f2adc1e9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae1705640 1 -- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2adc0b3ed0 msgr2=0x7f2adc1e9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae1705640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2adc0b3ed0 0x7f2adc1e9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae1705640 1 -- 172.21.15.59:0/896201527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc1e9fd0 con 0x7f2adc15ee10 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae0f04640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2adc0b3ed0 0x7f2adc1e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae1705640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2adc15ee10 0x7f2adc0ce0f0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400cb10 tx=0x7f2ad400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:16.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ad97fa640 1 -- 172.21.15.59:0/896201527 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4016020 con 0x7f2adc15ee10 2024-06-29T15:57:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ad97fa640 1 -- 172.21.15.59:0/896201527 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ad400ad60 con 0x7f2adc15ee10 2024-06-29T15:57:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ad97fa640 1 -- 172.21.15.59:0/896201527 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4012680 con 0x7f2adc15ee10 2024-06-29T15:57:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2adc1ea2b0 con 0x7f2adc15ee10 2024-06-29T15:57:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.650+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2adc1ea870 con 0x7f2adc15ee10 2024-06-29T15:57:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.651+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2adc15e8d0 con 0x7f2adc15ee10 2024-06-29T15:57:16.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.652+0000 7f2ad97fa640 1 -- 172.21.15.59:0/896201527 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f2ad4019020 con 0x7f2adc15ee10 2024-06-29T15:57:16.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.652+0000 7f2ad97fa640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f2ac8076420 0x7f2ac8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.653+0000 7f2ad97fa640 1 -- 172.21.15.59:0/896201527 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7f2ad40980c0 con 0x7f2adc15ee10 2024-06-29T15:57:16.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.653+0000 7f2ae0f04640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f2ac8076420 0x7f2ac8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.653+0000 7f2ae0f04640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f2ac8076420 0x7f2ac8078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2acc034000 tx=0x7f2acc004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.657+0000 7f2ad97fa640 1 -- 172.21.15.59:0/896201527 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ad4020020 con 0x7f2adc15ee10 2024-06-29T15:57:16.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.848+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f2adc0d2950 con 0x7f2adc15ee10 2024-06-29T15:57:16.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.851+0000 7f2ad97fa640 1 -- 172.21.15.59:0/896201527 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v72)=0 v72) ==== 122+0+0 (secure 0 0 0) 0x7f2adc0b42e0 con 0x7f2adc15ee10 2024-06-29T15:57:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.853+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f2ac8076420 msgr2=0x7f2ac8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.853+0000 7f2ae3168640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f2ac8076420 0x7f2ac8078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2acc034000 tx=0x7f2acc004730 comp rx=0 tx=0).stop 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.853+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2adc15ee10 msgr2=0x7f2adc0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.853+0000 7f2ae3168640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2adc15ee10 0x7f2adc0ce0f0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400cb10 tx=0x7f2ad400a060 comp rx=0 tx=0).stop 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.854+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 shutdown_connections 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.854+0000 7f2ae3168640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f2ac8076420 0x7f2ac8078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.854+0000 7f2ae3168640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2adc15ee10 0x7f2adc0ce0f0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.854+0000 7f2ae3168640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2adc0d23c0 0x7f2adc1e9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.854+0000 7f2ae3168640 1 --2- 172.21.15.59:0/896201527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2adc0b3ed0 0x7f2adc1e9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.854+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 >> 172.21.15.59:0/896201527 conn(0x7f2adc0d3ee0 msgr2=0x7f2adc06d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.854+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 shutdown_connections 2024-06-29T15:57:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:16.854+0000 7f2ae3168640 1 -- 172.21.15.59:0/896201527 wait complete. 2024-06-29T15:57:16.866 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:57:16.866 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:16.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:16.959 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:57:16.959 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:16.960 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:16.977 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:57:16.977 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:57:16.978 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:16.996 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:16.996 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:16.996 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:17.034 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:57:17.034 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:57:17.034 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:17.055 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:17.055 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:57:17.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:17.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.192+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/3094489663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1b0003af0 msgr2=0x7fd1b0003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.192+0000 7fd1cb3eb640 1 --2- 172.21.15.59:0/3094489663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1b0003af0 0x7fd1b0003f60 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0001660 tx=0x7fd1b0005830 comp rx=0 tx=0).stop 2024-06-29T15:57:17.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.192+0000 7fd1c9988640 1 -- 172.21.15.59:0/3094489663 reap_dead start 2024-06-29T15:57:17.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.192+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/3094489663 shutdown_connections 2024-06-29T15:57:17.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.192+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/3094489663 >> 172.21.15.59:0/3094489663 conn(0x7fd1c40d45b0 msgr2=0x7fd1c40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:17.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.192+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/3094489663 shutdown_connections 2024-06-29T15:57:17.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.192+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/3094489663 wait complete. 2024-06-29T15:57:17.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.194+0000 7fd1cb3eb640 1 Processor -- start 2024-06-29T15:57:17.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.194+0000 7fd1cb3eb640 1 -- start start 2024-06-29T15:57:17.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1cb3eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b0003af0 0x7fd1c415e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1cb3eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1c415ab00 0x7fd1c41e7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1cb3eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1c41e8010 0x7fd1c4160e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1cb3eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c40b3ff0 con 0x7fd1c415ab00 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1cb3eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c40ce8d0 con 0x7fd1b0003af0 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1cb3eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c40b4170 con 0x7fd1c41e8010 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c8986640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1c415ab00 0x7fd1c41e7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c8986640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1c415ab00 0x7fd1c41e7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59510/0 (socket says 172.21.15.59:59510) 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c8986640 1 -- 172.21.15.59:0/658596432 learned_addr learned my addr 172.21.15.59:0/658596432 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c9187640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b0003af0 0x7fd1c415e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c9988640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1c41e8010 0x7fd1c4160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c8986640 1 -- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1c41e8010 msgr2=0x7fd1c4160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c8986640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1c41e8010 0x7fd1c4160e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c8986640 1 -- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b0003af0 msgr2=0x7fd1c415e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c8986640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b0003af0 0x7fd1c415e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c8986640 1 -- 172.21.15.59:0/658596432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c41e8420 con 0x7fd1c415ab00 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c9988640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1c41e8010 0x7fd1c4160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.195+0000 7fd1c9187640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b0003af0 0x7fd1c415e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.196+0000 7fd1c8986640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1c415ab00 0x7fd1c41e7ad0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e570 tx=0x7fd1b400ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.196+0000 7fd1b9ffb640 1 -- 172.21.15.59:0/658596432 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4015020 con 0x7fd1c415ab00 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.196+0000 7fd1b9ffb640 1 -- 172.21.15.59:0/658596432 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1b400c9e0 con 0x7fd1c415ab00 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.196+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c415ea80 con 0x7fd1c415ab00 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.196+0000 7fd1b9ffb640 1 -- 172.21.15.59:0/658596432 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b400ce10 con 0x7fd1c415ab00 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.196+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c40c8d90 con 0x7fd1c415ab00 2024-06-29T15:57:17.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.197+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c40b8300 con 0x7fd1c415ab00 2024-06-29T15:57:17.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.198+0000 7fd1b9ffb640 1 -- 172.21.15.59:0/658596432 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fd1b4018020 con 0x7fd1c415ab00 2024-06-29T15:57:17.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.198+0000 7fd1b9ffb640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd1a80763f0 0x7fd1a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.198+0000 7fd1b9ffb640 1 -- 172.21.15.59:0/658596432 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7fd1b4098050 con 0x7fd1c415ab00 2024-06-29T15:57:17.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.199+0000 7fd1c9187640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd1a80763f0 0x7fd1a8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.199+0000 7fd1c9187640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd1a80763f0 0x7fd1a8078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0032000 tx=0x7fd1c0001190 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:17.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.203+0000 7fd1b9ffb640 1 -- 172.21.15.59:0/658596432 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1b401f020 con 0x7fd1c415ab00 2024-06-29T15:57:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.430+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1c40cdd90 con 0x7fd1c415ab00 2024-06-29T15:57:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.430+0000 7fd1b9ffb640 1 -- 172.21.15.59:0/658596432 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+3925 (secure 0 0 0) 0x7fd1c41e8420 con 0x7fd1c415ab00 2024-06-29T15:57:17.432 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:17.432 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"btime":"2024-06-29T15:57:11:144777+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3297476433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":3297476433},{"type":"v1","addr":"172.21.15.153:6837","nonce":3297476433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":10669,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/3243744233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3243744233},{"type":"v1","addr":"172.21.15.153:6836","nonce":3243744233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108},{"gid":10974,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3987613682","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3987613682},{"type":"v1","addr":"172.21.15.59:6834","nonce":3987613682}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":10977,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2501188073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2501188073},{"type":"v1","addr":"172.21.15.59:6835","nonce":2501188073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[]} 2024-06-29T15:57:17.432 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-06-29T15:57:17.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd1a80763f0 msgr2=0x7fd1a8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd1a80763f0 0x7fd1a8078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0032000 tx=0x7fd1c0001190 comp rx=0 tx=0).stop 2024-06-29T15:57:17.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1c415ab00 msgr2=0x7fd1c41e7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1c415ab00 0x7fd1c41e7ad0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e570 tx=0x7fd1b400ea30 comp rx=0 tx=0).stop 2024-06-29T15:57:17.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 shutdown_connections 2024-06-29T15:57:17.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd1a80763f0 0x7fd1a8078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd1c41e8010 0x7fd1c4160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1c415ab00 0x7fd1c41e7ad0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 --2- 172.21.15.59:0/658596432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b0003af0 0x7fd1c415e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 >> 172.21.15.59:0/658596432 conn(0x7fd1c40d45b0 msgr2=0x7fd1c40b4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:17.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 shutdown_connections 2024-06-29T15:57:17.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.433+0000 7fd1cb3eb640 1 -- 172.21.15.59:0/658596432 wait complete. 2024-06-29T15:57:17.445 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:57:17.446 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:57:17.446 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:57:17.446 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:57:17.447 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:57:17.447 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:57:17.447 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:57:17.447 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:57:17.447 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:57:17.447 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:57:17.448 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:57:17.448 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:57:17.448 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:57:17.448 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:57:17.448 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:57:17.448 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:57:17.457 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:57:17.457 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:57:17.459 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:57:17.459 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:57:17.460 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:57:17.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:57:17.463 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:57:17.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:57:17.500 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:57:17.501 DEBUG:teuthology.parallel:result is None 2024-06-29T15:57:17.501 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:57:17.502 DEBUG:teuthology.parallel:result is None 2024-06-29T15:57:17.505 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:57:17.505 DEBUG:teuthology.parallel:result is None 2024-06-29T15:57:17.506 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:57:17.507 DEBUG:teuthology.parallel:result is None 2024-06-29T15:57:17.507 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:57:17.590 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:57:17.591 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:57:17.602 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:57:17.602 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T15:57:17.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.631+0000 7f0032a66640 1 -- 172.21.15.59:0/2321573821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f002c0b40a0 msgr2=0x7f002c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.631+0000 7f0032a66640 1 --2- 172.21.15.59:0/2321573821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f002c0b40a0 0x7f002c15e8d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0018009830 tx=0x7f001802a510 comp rx=0 tx=0).stop 2024-06-29T15:57:17.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.631+0000 7f0032a66640 1 -- 172.21.15.59:0/2321573821 shutdown_connections 2024-06-29T15:57:17.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.631+0000 7f0032a66640 1 --2- 172.21.15.59:0/2321573821 >> v2:172.21.15.153:3300/0 conn(0x7f002c15ee10 0x7f002c15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.631+0000 7f0032a66640 1 --2- 172.21.15.59:0/2321573821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f002c0b40a0 0x7f002c15e8d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.631+0000 7f0032a66640 1 -- 172.21.15.59:0/2321573821 >> 172.21.15.59:0/2321573821 conn(0x7f002c0d3ee0 msgr2=0x7f002c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:17.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.631+0000 7f0032a66640 1 -- 172.21.15.59:0/2321573821 shutdown_connections 2024-06-29T15:57:17.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.631+0000 7f0032a66640 1 -- 172.21.15.59:0/2321573821 wait complete. 2024-06-29T15:57:17.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.633+0000 7f0032a66640 1 Processor -- start 2024-06-29T15:57:17.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.633+0000 7f0032a66640 1 -- start start 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f0032a66640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f002c0d23c0 0x7f002c0ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f0032a66640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002c15ee10 0x7f002c1f5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f0032a66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002c0b8bb0 0x7f002c0b8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f0032a66640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f002c160280 con 0x7f002c15ee10 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f0032a66640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f002c160100 con 0x7f002c0b8bb0 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f0032a66640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f002c160400 con 0x7f002c0d23c0 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002c15ee10 0x7f002c1f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002c15ee10 0x7f002c1f5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59536/0 (socket says 172.21.15.59:59536) 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 -- 172.21.15.59:0/3612453120 learned_addr learned my addr 172.21.15.59:0/3612453120 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f0031003640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002c0b8bb0 0x7f002c0b8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f0030802640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f002c0d23c0 0x7f002c0ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 -- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f002c0d23c0 msgr2=0x7f002c0ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f002c0d23c0 0x7f002c0ac8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 -- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002c0b8bb0 msgr2=0x7f002c0b8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002c0b8bb0 0x7f002c0b8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 -- 172.21.15.59:0/3612453120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002c0b9500 con 0x7f002c15ee10 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.635+0000 7f002bfff640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002c15ee10 0x7f002c1f5340 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f001802ab00 tx=0x7f0018033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.636+0000 7f00297fa640 1 -- 172.21.15.59:0/3612453120 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018005c90 con 0x7f002c15ee10 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.636+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f002c0ad2d0 con 0x7f002c15ee10 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.636+0000 7f00297fa640 1 -- 172.21.15.59:0/3612453120 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0018031e50 con 0x7f002c15ee10 2024-06-29T15:57:17.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.636+0000 7f00297fa640 1 -- 172.21.15.59:0/3612453120 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018005260 con 0x7f002c15ee10 2024-06-29T15:57:17.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.636+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f002c0aa750 con 0x7f002c15ee10 2024-06-29T15:57:17.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.637+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f002c15e8d0 con 0x7f002c15ee10 2024-06-29T15:57:17.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.637+0000 7f00297fa640 1 -- 172.21.15.59:0/3612453120 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f0018005400 con 0x7f002c15ee10 2024-06-29T15:57:17.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.638+0000 7f00297fa640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0010076420 0x7f0010078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.638+0000 7f00297fa640 1 -- 172.21.15.59:0/3612453120 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6165+0+0 (secure 0 0 0) 0x7f00180b8b80 con 0x7f002c15ee10 2024-06-29T15:57:17.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.638+0000 7f0030802640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0010076420 0x7f0010078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.639+0000 7f0030802640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0010076420 0x7f0010078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0020007fd0 tx=0x7f0020001370 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:17.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.642+0000 7f00297fa640 1 -- 172.21.15.59:0/3612453120 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f002c0b9500 con 0x7f002c15ee10 2024-06-29T15:57:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:17.846+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f002c0d2950 con 0x7f002c15ee10 2024-06-29T15:57:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.181+0000 7f00297fa640 1 -- 172.21.15.59:0/3612453120 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v216) ==== 97+0+0 (secure 0 0 0) 0x7f001803f030 con 0x7f002c15ee10 2024-06-29T15:57:18.182 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0010076420 msgr2=0x7f0010078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0010076420 0x7f0010078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0020007fd0 tx=0x7f0020001370 comp rx=0 tx=0).stop 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002c15ee10 msgr2=0x7f002c1f5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002c15ee10 0x7f002c1f5340 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f001802ab00 tx=0x7f0018033c20 comp rx=0 tx=0).stop 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 shutdown_connections 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0010076420 0x7f0010078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002c0b8bb0 0x7f002c0b8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002c15ee10 0x7f002c1f5340 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 --2- 172.21.15.59:0/3612453120 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f002c0d23c0 0x7f002c0ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 >> 172.21.15.59:0/3612453120 conn(0x7f002c0d3ee0 msgr2=0x7f002c06ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 shutdown_connections 2024-06-29T15:57:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.184+0000 7f0032a66640 1 -- 172.21.15.59:0/3612453120 wait complete. 2024-06-29T15:57:18.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:57:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.316+0000 7f34aa08f640 1 -- 172.21.15.59:0/1117767580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a40b3740 msgr2=0x7f34a40b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.316+0000 7f34aa08f640 1 --2- 172.21.15.59:0/1117767580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a40b3740 0x7f34a40b3b50 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f34a40b4b50 tx=0x7f349802a560 comp rx=0 tx=0).stop 2024-06-29T15:57:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.316+0000 7f34aa08f640 1 -- 172.21.15.59:0/1117767580 shutdown_connections 2024-06-29T15:57:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.316+0000 7f34aa08f640 1 --2- 172.21.15.59:0/1117767580 >> v2:172.21.15.153:3300/0 conn(0x7f34a415c2f0 0x7f34a415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.316+0000 7f34aa08f640 1 --2- 172.21.15.59:0/1117767580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a40b3740 0x7f34a40b3b50 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.316+0000 7f34aa08f640 1 -- 172.21.15.59:0/1117767580 >> 172.21.15.59:0/1117767580 conn(0x7f34a406c950 msgr2=0x7f34a4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.316+0000 7f34aa08f640 1 -- 172.21.15.59:0/1117767580 shutdown_connections 2024-06-29T15:57:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.316+0000 7f34aa08f640 1 -- 172.21.15.59:0/1117767580 wait complete. 2024-06-29T15:57:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.318+0000 7f34aa08f640 1 Processor -- start 2024-06-29T15:57:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.318+0000 7f34aa08f640 1 -- start start 2024-06-29T15:57:18.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.319+0000 7f34aa08f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34a40b3740 0x7f34a40c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34aa08f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a415c2f0 0x7f34a40c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34aa08f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a414f720 0x7f34a414fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34aa08f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f34a40b5fd0 con 0x7f34a414f720 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34aa08f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34a40b5e50 con 0x7f34a415c2f0 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34aa08f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f34a40b6150 con 0x7f34a40b3740 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a414f720 0x7f34a414fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a414f720 0x7f34a414fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59554/0 (socket says 172.21.15.59:59554) 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 -- 172.21.15.59:0/4109965738 learned_addr learned my addr 172.21.15.59:0/4109965738 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a2ffd640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a415c2f0 0x7f34a40c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a37fe640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34a40b3740 0x7f34a40c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 -- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34a40b3740 msgr2=0x7f34a40c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34a40b3740 0x7f34a40c6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 -- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a415c2f0 msgr2=0x7f34a40c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a415c2f0 0x7f34a40c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 -- 172.21.15.59:0/4109965738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a414e680 con 0x7f34a414f720 2024-06-29T15:57:18.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34a3fff640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a414f720 0x7f34a414fb30 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f349c00e9c0 tx=0x7f349c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:18.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f348bfff640 1 -- 172.21.15.59:0/4109965738 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c004d20 con 0x7f34a414f720 2024-06-29T15:57:18.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f348bfff640 1 -- 172.21.15.59:0/4109965738 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f349c00cac0 con 0x7f34a414f720 2024-06-29T15:57:18.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a414e960 con 0x7f34a414f720 2024-06-29T15:57:18.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.320+0000 7f348bfff640 1 -- 172.21.15.59:0/4109965738 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c005020 con 0x7f34a414f720 2024-06-29T15:57:18.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.321+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a414ef20 con 0x7f34a414f720 2024-06-29T15:57:18.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.322+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a40689f0 con 0x7f34a414f720 2024-06-29T15:57:18.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.322+0000 7f348bfff640 1 -- 172.21.15.59:0/4109965738 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f349c0051c0 con 0x7f34a414f720 2024-06-29T15:57:18.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.323+0000 7f348bfff640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3490076420 0x7f3490078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.323+0000 7f348bfff640 1 -- 172.21.15.59:0/4109965738 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5305+0+0 (secure 0 0 0) 0x7f349c097ff0 con 0x7f34a414f720 2024-06-29T15:57:18.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.323+0000 7f34a37fe640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3490076420 0x7f3490078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.324+0000 7f34a37fe640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3490076420 0x7f3490078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3494006fd0 tx=0x7f3494009060 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:18.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.327+0000 7f348bfff640 1 -- 172.21.15.59:0/4109965738 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f34a40689f0 con 0x7f34a414f720 2024-06-29T15:57:18.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.589+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f34a40b6670 con 0x7f34a414f720 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.590+0000 7f348bfff640 1 -- 172.21.15.59:0/4109965738 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) ==== 80+0+5304 (secure 0 0 0) 0x7f349c016030 con 0x7f34a414f720 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.591 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:57:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:57:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T15:57:18.593 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T15:57:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:57:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:57:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:57:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.594 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T15:57:18.595 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.596 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.597 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.598 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.599 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.600 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.601 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T15:57:18.603 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.604 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.605 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T15:57:18.606 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:57:18.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3490076420 msgr2=0x7f3490078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3490076420 0x7f3490078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3494006fd0 tx=0x7f3494009060 comp rx=0 tx=0).stop 2024-06-29T15:57:18.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a414f720 msgr2=0x7f34a414fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a414f720 0x7f34a414fb30 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f349c00e9c0 tx=0x7f349c00ee80 comp rx=0 tx=0).stop 2024-06-29T15:57:18.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 shutdown_connections 2024-06-29T15:57:18.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3490076420 0x7f3490078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a414f720 0x7f34a414fb30 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a415c2f0 0x7f34a40c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 --2- 172.21.15.59:0/4109965738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f34a40b3740 0x7f34a40c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 >> 172.21.15.59:0/4109965738 conn(0x7f34a406c950 msgr2=0x7f34a4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 shutdown_connections 2024-06-29T15:57:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.592+0000 7f34aa08f640 1 -- 172.21.15.59:0/4109965738 wait complete. 2024-06-29T15:57:18.610 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:18.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.726+0000 7f5dcb93c640 1 -- 172.21.15.59:0/3302836234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc41604d0 msgr2=0x7f5dc413ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.726+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/3302836234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc41604d0 0x7f5dc413ebd0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc005fd0 tx=0x7f5dbc02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.727+0000 7f5dcb93c640 1 -- 172.21.15.59:0/3302836234 shutdown_connections 2024-06-29T15:57:18.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.727+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/3302836234 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc40b1b00 0x7f5dc40b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.727+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/3302836234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc41604d0 0x7f5dc413ebd0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.727+0000 7f5dcb93c640 1 -- 172.21.15.59:0/3302836234 >> 172.21.15.59:0/3302836234 conn(0x7f5dc40b06e0 msgr2=0x7f5dc40b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.727+0000 7f5dcb93c640 1 -- 172.21.15.59:0/3302836234 shutdown_connections 2024-06-29T15:57:18.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.727+0000 7f5dcb93c640 1 -- 172.21.15.59:0/3302836234 wait complete. 2024-06-29T15:57:18.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.728+0000 7f5dcb93c640 1 Processor -- start 2024-06-29T15:57:18.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.728+0000 7f5dcb93c640 1 -- start start 2024-06-29T15:57:18.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dcb93c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc40b1b00 0x7f5dc40a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dcb93c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc4141360 0x7f5dc4155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dcb93c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc41604d0 0x7f5dc41f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dcb93c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc40af9d0 con 0x7f5dc41604d0 2024-06-29T15:57:18.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dcb93c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc40af850 con 0x7f5dc40b1b00 2024-06-29T15:57:18.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dcb93c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5dc40afb50 con 0x7f5dc4141360 2024-06-29T15:57:18.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc41604d0 0x7f5dc41f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc41604d0 0x7f5dc41f2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59568/0 (socket says 172.21.15.59:59568) 2024-06-29T15:57:18.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 -- 172.21.15.59:0/2658132752 learned_addr learned my addr 172.21.15.59:0/2658132752 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:18.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc96d8640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc40b1b00 0x7f5dc40a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc8ed7640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc4141360 0x7f5dc4155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 -- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc4141360 msgr2=0x7f5dc4155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc4141360 0x7f5dc4155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 -- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc40b1b00 msgr2=0x7f5dc40a9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc40b1b00 0x7f5dc40a9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 -- 172.21.15.59:0/2658132752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc40b50c0 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.730+0000 7f5dc9ed9640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc41604d0 0x7f5dc41f2ae0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f5db400c730 tx=0x7f5db400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.731+0000 7f5dc1ffb640 1 -- 172.21.15.59:0/2658132752 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db4003b70 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.731+0000 7f5dc1ffb640 1 -- 172.21.15.59:0/2658132752 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5db4003d10 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.731+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc40b53a0 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.731+0000 7f5dc1ffb640 1 -- 172.21.15.59:0/2658132752 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db4005330 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.731+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc40b5960 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.732+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc40b00e0 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.733+0000 7f5dc1ffb640 1 -- 172.21.15.59:0/2658132752 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f5db40054d0 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.733+0000 7f5dc1ffb640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f5db00763f0 0x7f5db0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.733+0000 7f5dc1ffb640 1 -- 172.21.15.59:0/2658132752 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5305+0+0 (secure 0 0 0) 0x7f5db4097460 con 0x7f5dc41604d0 2024-06-29T15:57:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.733+0000 7f5dc96d8640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f5db00763f0 0x7f5db0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.734+0000 7f5dc96d8640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f5db00763f0 0x7f5db0078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc033000 tx=0x7f5dbc004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:18.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.737+0000 7f5dc1ffb640 1 -- 172.21.15.59:0/2658132752 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5dc40b2450 con 0x7f5dc41604d0 2024-06-29T15:57:18.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.930+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5dc40b4e50 con 0x7f5dc41604d0 2024-06-29T15:57:18.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.931+0000 7f5dc1ffb640 1 -- 172.21.15.59:0/2658132752 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) ==== 74+0+13276 (secure 0 0 0) 0x7f5db4079e20 con 0x7f5dc41604d0 2024-06-29T15:57:18.932 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:18.932 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":218,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:18.190239+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":202,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:18.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f5db00763f0 msgr2=0x7f5db0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f5db00763f0 0x7f5db0078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc033000 tx=0x7f5dbc004730 comp rx=0 tx=0).stop 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc41604d0 msgr2=0x7f5dc41f2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc41604d0 0x7f5dc41f2ae0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f5db400c730 tx=0x7f5db400cb30 comp rx=0 tx=0).stop 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 shutdown_connections 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f5db00763f0 0x7f5db0078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dc41604d0 0x7f5dc41f2ae0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5dc4141360 0x7f5dc4155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 --2- 172.21.15.59:0/2658132752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5dc40b1b00 0x7f5dc40a9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 >> 172.21.15.59:0/2658132752 conn(0x7f5dc40b06e0 msgr2=0x7f5dc415bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 shutdown_connections 2024-06-29T15:57:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:18.933+0000 7f5dcb93c640 1 -- 172.21.15.59:0/2658132752 wait complete. 2024-06-29T15:57:18.946 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:18.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:19.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.066+0000 7fefea7a1640 1 -- 172.21.15.59:0/129716837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefe408c190 msgr2=0x7fefe40cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.066+0000 7fefea7a1640 1 --2- 172.21.15.59:0/129716837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefe408c190 0x7fefe40cf080 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009900 tx=0x7fefd402a610 comp rx=0 tx=0).stop 2024-06-29T15:57:19.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.066+0000 7fefea7a1640 1 -- 172.21.15.59:0/129716837 shutdown_connections 2024-06-29T15:57:19.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.066+0000 7fefea7a1640 1 --2- 172.21.15.59:0/129716837 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefe40ce950 0x7fefe40d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.066+0000 7fefea7a1640 1 --2- 172.21.15.59:0/129716837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefe408c190 0x7fefe40cf080 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.066+0000 7fefea7a1640 1 -- 172.21.15.59:0/129716837 >> 172.21.15.59:0/129716837 conn(0x7fefe40b5610 msgr2=0x7fefe40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.066+0000 7fefea7a1640 1 -- 172.21.15.59:0/129716837 shutdown_connections 2024-06-29T15:57:19.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.066+0000 7fefea7a1640 1 -- 172.21.15.59:0/129716837 wait complete. 2024-06-29T15:57:19.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.068+0000 7fefea7a1640 1 Processor -- start 2024-06-29T15:57:19.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.068+0000 7fefea7a1640 1 -- start start 2024-06-29T15:57:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefea7a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefe408c190 0x7fefe41f8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefea7a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefe40ce950 0x7fefe4159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefea7a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefe40cf5c0 0x7fefe41604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefea7a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fefe40b7f60 con 0x7fefe40ce950 2024-06-29T15:57:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefea7a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fefe40b7de0 con 0x7fefe40cf5c0 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefea7a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fefe40b80e0 con 0x7fefe408c190 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefe40ce950 0x7fefe4159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefe40ce950 0x7fefe4159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59572/0 (socket says 172.21.15.59:59572) 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 -- 172.21.15.59:0/1417125081 learned_addr learned my addr 172.21.15.59:0/1417125081 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe3fff640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefe408c190 0x7fefe41f8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe8d3e640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefe40cf5c0 0x7fefe41604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 -- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefe408c190 msgr2=0x7fefe41f8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefe408c190 0x7fefe41f8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 -- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefe40cf5c0 msgr2=0x7fefe41604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefe40cf5c0 0x7fefe41604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 -- 172.21.15.59:0/1417125081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe41f94c0 con 0x7fefe40ce950 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe37fe640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefe40ce950 0x7fefe4159f00 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fefd800c420 tx=0x7fefd800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.070+0000 7fefe0ff9640 1 -- 172.21.15.59:0/1417125081 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8017070 con 0x7fefe40ce950 2024-06-29T15:57:19.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.071+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe41e9af0 con 0x7fefe40ce950 2024-06-29T15:57:19.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.071+0000 7fefe0ff9640 1 -- 172.21.15.59:0/1417125081 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefd80053c0 con 0x7fefe40ce950 2024-06-29T15:57:19.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.071+0000 7fefe0ff9640 1 -- 172.21.15.59:0/1417125081 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8003870 con 0x7fefe40ce950 2024-06-29T15:57:19.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.071+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe41e7000 con 0x7fefe40ce950 2024-06-29T15:57:19.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.072+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe40cf080 con 0x7fefe40ce950 2024-06-29T15:57:19.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.072+0000 7fefe0ff9640 1 -- 172.21.15.59:0/1417125081 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fefd801a070 con 0x7fefe40ce950 2024-06-29T15:57:19.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.073+0000 7fefe0ff9640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fefd0076420 0x7fefd0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.073+0000 7fefe0ff9640 1 -- 172.21.15.59:0/1417125081 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5305+0+0 (secure 0 0 0) 0x7fefd8096bd0 con 0x7fefe40ce950 2024-06-29T15:57:19.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.073+0000 7fefe3fff640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fefd0076420 0x7fefd0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.074+0000 7fefe3fff640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fefd0076420 0x7fefd0078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fefd4033000 tx=0x7fefd4004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.078+0000 7fefe0ff9640 1 -- 172.21.15.59:0/1417125081 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fefd805fe70 con 0x7fefe40ce950 2024-06-29T15:57:19.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.269+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fefe40cbc00 con 0x7fefe40ce950 2024-06-29T15:57:19.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.270+0000 7fefe0ff9640 1 -- 172.21.15.59:0/1417125081 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v218) ==== 93+0+10 (secure 0 0 0) 0x7fefd8064170 con 0x7fefe40ce950 2024-06-29T15:57:19.272 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:57:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fefd0076420 msgr2=0x7fefd0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fefd0076420 0x7fefd0078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fefd4033000 tx=0x7fefd4004730 comp rx=0 tx=0).stop 2024-06-29T15:57:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefe40ce950 msgr2=0x7fefe4159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefe40ce950 0x7fefe4159f00 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fefd800c420 tx=0x7fefd800c8e0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 shutdown_connections 2024-06-29T15:57:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fefd0076420 0x7fefd0078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefe40cf5c0 0x7fefe41604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefe40ce950 0x7fefe4159f00 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 --2- 172.21.15.59:0/1417125081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fefe408c190 0x7fefe41f8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 >> 172.21.15.59:0/1417125081 conn(0x7fefe40b5610 msgr2=0x7fefe415e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 shutdown_connections 2024-06-29T15:57:19.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.272+0000 7fefea7a1640 1 -- 172.21.15.59:0/1417125081 wait complete. 2024-06-29T15:57:19.285 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:57:19.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.402+0000 7f0351ecc640 1 -- 172.21.15.59:0/2367114725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0334003aa0 msgr2=0x7f0334003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.402+0000 7f0351ecc640 1 --2- 172.21.15.59:0/2367114725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0334003aa0 0x7f0334003f10 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f033c009b50 tx=0x7f0334005830 comp rx=0 tx=0).stop 2024-06-29T15:57:19.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.402+0000 7f034bfff640 1 -- 172.21.15.59:0/2367114725 reap_dead start 2024-06-29T15:57:19.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.403+0000 7f0351ecc640 1 -- 172.21.15.59:0/2367114725 shutdown_connections 2024-06-29T15:57:19.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.403+0000 7f0351ecc640 1 -- 172.21.15.59:0/2367114725 >> 172.21.15.59:0/2367114725 conn(0x7f034c0b6fb0 msgr2=0x7f034c0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.403+0000 7f0351ecc640 1 -- 172.21.15.59:0/2367114725 shutdown_connections 2024-06-29T15:57:19.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.403+0000 7f0351ecc640 1 -- 172.21.15.59:0/2367114725 wait complete. 2024-06-29T15:57:19.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.404+0000 7f0351ecc640 1 Processor -- start 2024-06-29T15:57:19.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.405+0000 7f0351ecc640 1 -- start start 2024-06-29T15:57:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f0351ecc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0334003aa0 0x7f034c1e7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f0351ecc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c151e20 0x7f034c1525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f0351ecc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f034c1512a0 0x7f034c14db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f0351ecc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f034c0b5850 con 0x7f034c1512a0 2024-06-29T15:57:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f0351ecc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f034c0b56d0 con 0x7f0334003aa0 2024-06-29T15:57:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f0351ecc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f034c0b59d0 con 0x7f034c151e20 2024-06-29T15:57:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f034c1512a0 0x7f034c14db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f034c1512a0 0x7f034c14db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59580/0 (socket says 172.21.15.59:59580) 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034bfff640 1 -- 172.21.15.59:0/310470160 learned_addr learned my addr 172.21.15.59:0/310470160 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034affd640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c151e20 0x7f034c1525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034b7fe640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0334003aa0 0x7f034c1e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034bfff640 1 -- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c151e20 msgr2=0x7f034c1525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034bfff640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c151e20 0x7f034c1525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034bfff640 1 -- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0334003aa0 msgr2=0x7f034c1e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034bfff640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0334003aa0 0x7f034c1e7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034bfff640 1 -- 172.21.15.59:0/310470160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c1501e0 con 0x7f034c1512a0 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.406+0000 7f034affd640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c151e20 0x7f034c1525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.407+0000 7f034bfff640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f034c1512a0 0x7f034c14db90 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f0344000f80 tx=0x7f034400eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.407+0000 7f0333fff640 1 -- 172.21.15.59:0/310470160 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034400c910 con 0x7f034c1512a0 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.407+0000 7f0333fff640 1 -- 172.21.15.59:0/310470160 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0344017070 con 0x7f034c1512a0 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.407+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c1504c0 con 0x7f034c1512a0 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.407+0000 7f0333fff640 1 -- 172.21.15.59:0/310470160 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034400ce40 con 0x7f034c1512a0 2024-06-29T15:57:19.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.407+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c150d80 con 0x7f034c1512a0 2024-06-29T15:57:19.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.408+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034c150700 con 0x7f034c1512a0 2024-06-29T15:57:19.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.408+0000 7f0333fff640 1 -- 172.21.15.59:0/310470160 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f034401c460 con 0x7f034c1512a0 2024-06-29T15:57:19.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.409+0000 7f0333fff640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0338076420 0x7f0338078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.409+0000 7f0333fff640 1 -- 172.21.15.59:0/310470160 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5305+0+0 (secure 0 0 0) 0x7f0344097c50 con 0x7f034c1512a0 2024-06-29T15:57:19.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.409+0000 7f034b7fe640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0338076420 0x7f0338078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.410+0000 7f034b7fe640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0338076420 0x7f0338078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f033c035000 tx=0x7f033c02d040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.414+0000 7f0333fff640 1 -- 172.21.15.59:0/310470160 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f034c150700 con 0x7f034c1512a0 2024-06-29T15:57:19.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.607+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f034c0b5bb0 con 0x7f034c1512a0 2024-06-29T15:57:19.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.607+0000 7f0333fff640 1 -- 172.21.15.59:0/310470160 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v218) ==== 84+0+65 (secure 0 0 0) 0x7f03440613f0 con 0x7f034c1512a0 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:57:19.610 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:19.610 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:57:19.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0338076420 msgr2=0x7f0338078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0338076420 0x7f0338078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f033c035000 tx=0x7f033c02d040 comp rx=0 tx=0).stop 2024-06-29T15:57:19.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f034c1512a0 msgr2=0x7f034c14db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f034c1512a0 0x7f034c14db90 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f0344000f80 tx=0x7f034400eb00 comp rx=0 tx=0).stop 2024-06-29T15:57:19.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 shutdown_connections 2024-06-29T15:57:19.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f0338076420 0x7f0338078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f034c1512a0 0x7f034c14db90 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034c151e20 0x7f034c1525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 --2- 172.21.15.59:0/310470160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0334003aa0 0x7f034c1e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 >> 172.21.15.59:0/310470160 conn(0x7f034c0b6fb0 msgr2=0x7f034c15e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 shutdown_connections 2024-06-29T15:57:19.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.609+0000 7f0351ecc640 1 -- 172.21.15.59:0/310470160 wait complete. 2024-06-29T15:57:19.622 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:57:19.623 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:57:19.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.744+0000 7fc83ff68640 1 -- 172.21.15.59:0/2987927013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380745a0 msgr2=0x7fc83815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.744+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2987927013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380745a0 0x7fc83815e680 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fc838071a10 tx=0x7fc83002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.744+0000 7fc83ff68640 1 -- 172.21.15.59:0/2987927013 shutdown_connections 2024-06-29T15:57:19.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.744+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2987927013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8380745a0 0x7fc83815e680 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.744+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2987927013 >> v2:172.21.15.153:3300/0 conn(0x7fc838145760 0x7fc8380d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.744+0000 7fc83ff68640 1 -- 172.21.15.59:0/2987927013 >> 172.21.15.59:0/2987927013 conn(0x7fc8380cd630 msgr2=0x7fc8380cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.744+0000 7fc83ff68640 1 -- 172.21.15.59:0/2987927013 shutdown_connections 2024-06-29T15:57:19.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.744+0000 7fc83ff68640 1 -- 172.21.15.59:0/2987927013 wait complete. 2024-06-29T15:57:19.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.746+0000 7fc83ff68640 1 Processor -- start 2024-06-29T15:57:19.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.746+0000 7fc83ff68640 1 -- start start 2024-06-29T15:57:19.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83ff68640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc838145760 0x7fc83813f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83ff68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc83815ce00 0x7fc8380b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83ff68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b1f90 0x7fc8381f7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83ff68640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc83815ef50 con 0x7fc838145760 2024-06-29T15:57:19.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83ff68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc83815edd0 con 0x7fc83815ce00 2024-06-29T15:57:19.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83ff68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc83815f0d0 con 0x7fc8380b1f90 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83dd04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc838145760 0x7fc83813f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83dd04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc838145760 0x7fc83813f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59598/0 (socket says 172.21.15.59:59598) 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83dd04640 1 -- 172.21.15.59:0/2239968759 learned_addr learned my addr 172.21.15.59:0/2239968759 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83d503640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc83815ce00 0x7fc8380b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83e505640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b1f90 0x7fc8381f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83dd04640 1 -- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b1f90 msgr2=0x7fc8381f7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83dd04640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b1f90 0x7fc8381f7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83dd04640 1 -- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc83815ce00 msgr2=0x7fc8380b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.747+0000 7fc83dd04640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc83815ce00 0x7fc8380b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.748+0000 7fc83dd04640 1 -- 172.21.15.59:0/2239968759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8381f8480 con 0x7fc838145760 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.748+0000 7fc83e505640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b1f90 0x7fc8381f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:19.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.748+0000 7fc83dd04640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc838145760 0x7fc83813f910 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fc834009fd0 tx=0x7fc83400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.748+0000 7fc82e7fc640 1 -- 172.21.15.59:0/2239968759 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83400cc60 con 0x7fc838145760 2024-06-29T15:57:19.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.748+0000 7fc82e7fc640 1 -- 172.21.15.59:0/2239968759 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc83400ce00 con 0x7fc838145760 2024-06-29T15:57:19.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.748+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8381f8760 con 0x7fc838145760 2024-06-29T15:57:19.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.748+0000 7fc82e7fc640 1 -- 172.21.15.59:0/2239968759 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834004280 con 0x7fc838145760 2024-06-29T15:57:19.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.748+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8381f8d20 con 0x7fc838145760 2024-06-29T15:57:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.749+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc83815f660 con 0x7fc838145760 2024-06-29T15:57:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.750+0000 7fc82e7fc640 1 -- 172.21.15.59:0/2239968759 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fc8340035f0 con 0x7fc838145760 2024-06-29T15:57:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.750+0000 7fc82e7fc640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc81c076420 0x7fc81c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.750+0000 7fc82e7fc640 1 -- 172.21.15.59:0/2239968759 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5305+0+0 (secure 0 0 0) 0x7fc834097fd0 con 0x7fc838145760 2024-06-29T15:57:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.750+0000 7fc83d503640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc81c076420 0x7fc81c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.751+0000 7fc83d503640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc81c076420 0x7fc81c078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc838074000 tx=0x7fc828007e90 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.755+0000 7fc82e7fc640 1 -- 172.21.15.59:0/2239968759 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc8340618a0 con 0x7fc838145760 2024-06-29T15:57:19.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:19.946+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc838074810 con 0x7fc838145760 2024-06-29T15:57:20.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.185+0000 7fc82e7fc640 1 -- 172.21.15.59:0/2239968759 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v219) ==== 138+0+0 (secure 0 0 0) 0x7fc8340611a0 con 0x7fc838145760 2024-06-29T15:57:20.186 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T15:57:20.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc81c076420 msgr2=0x7fc81c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc81c076420 0x7fc81c078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc838074000 tx=0x7fc828007e90 comp rx=0 tx=0).stop 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc838145760 msgr2=0x7fc83813f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc838145760 0x7fc83813f910 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fc834009fd0 tx=0x7fc83400ec20 comp rx=0 tx=0).stop 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 shutdown_connections 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc81c076420 0x7fc81c078930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8380b1f90 0x7fc8381f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc83815ce00 0x7fc8380b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 --2- 172.21.15.59:0/2239968759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc838145760 0x7fc83813f910 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.188+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 >> 172.21.15.59:0/2239968759 conn(0x7fc8380cd630 msgr2=0x7fc8380cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.189+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 shutdown_connections 2024-06-29T15:57:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.189+0000 7fc83ff68640 1 -- 172.21.15.59:0/2239968759 wait complete. 2024-06-29T15:57:20.205 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:57:20.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.331+0000 7f7c33681640 1 -- 172.21.15.59:0/3814199688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c14005250 msgr2=0x7f7c1400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.331+0000 7f7c33681640 1 --2- 172.21.15.59:0/3814199688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c14005250 0x7f7c1400b9c0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f7c24001690 tx=0x7f7c14005830 comp rx=0 tx=0).stop 2024-06-29T15:57:20.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.331+0000 7f7c33681640 1 -- 172.21.15.59:0/3814199688 shutdown_connections 2024-06-29T15:57:20.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.331+0000 7f7c33681640 1 --2- 172.21.15.59:0/3814199688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c14005250 0x7f7c1400b9c0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.331+0000 7f7c33681640 1 -- 172.21.15.59:0/3814199688 >> 172.21.15.59:0/3814199688 conn(0x7f7c2c0d04e0 msgr2=0x7f7c2c0d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:20.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.331+0000 7f7c33681640 1 -- 172.21.15.59:0/3814199688 shutdown_connections 2024-06-29T15:57:20.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.331+0000 7f7c33681640 1 -- 172.21.15.59:0/3814199688 wait complete. 2024-06-29T15:57:20.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.333+0000 7f7c33681640 1 Processor -- start 2024-06-29T15:57:20.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.333+0000 7f7c33681640 1 -- start start 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.334+0000 7f7c33681640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c14005250 0x7f7c2c151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.334+0000 7f7c33681640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c2c0cd3b0 0x7f7c2c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.334+0000 7f7c33681640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c2c0b3340 0x7f7c2c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.334+0000 7f7c33681640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b46f0 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.334+0000 7f7c33681640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b4570 con 0x7f7c2c0b3340 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.334+0000 7f7c33681640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b4870 con 0x7f7c14005250 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c2c0cd3b0 0x7f7c2c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c2c0cd3b0 0x7f7c2c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59626/0 (socket says 172.21.15.59:59626) 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 -- 172.21.15.59:0/4033399398 learned_addr learned my addr 172.21.15.59:0/4033399398 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c3141d640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c14005250 0x7f7c2c151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:20.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c31c1e640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c2c0b3340 0x7f7c2c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 -- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c14005250 msgr2=0x7f7c2c151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c14005250 0x7f7c2c151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 -- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c2c0b3340 msgr2=0x7f7c2c0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c2c0b3340 0x7f7c2c0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 -- 172.21.15.59:0/4033399398 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2c0b3c90 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c30c1c640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c2c0cd3b0 0x7f7c2c0cd7c0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f7c20011860 tx=0x7f7c20011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c29ffb640 1 -- 172.21.15.59:0/4033399398 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2001e070 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c29ffb640 1 -- 172.21.15.59:0/4033399398 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c2000fdb0 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c29ffb640 1 -- 172.21.15.59:0/4033399398 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20019690 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.335+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2c0c7830 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.336+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2c0c7df0 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.336+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2c15e420 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.337+0000 7f7c29ffb640 1 -- 172.21.15.59:0/4033399398 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f7c20021070 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.338+0000 7f7c29ffb640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7c18076420 0x7f7c18078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:20.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.338+0000 7f7c29ffb640 1 -- 172.21.15.59:0/4033399398 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 5675+0+0 (secure 0 0 0) 0x7f7c200982a0 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.338+0000 7f7c3141d640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7c18076420 0x7f7c18078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:20.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.339+0000 7f7c3141d640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7c18076420 0x7f7c18078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c001ca0 tx=0x7f7c1c006c50 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:20.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.343+0000 7f7c29ffb640 1 -- 172.21.15.59:0/4033399398 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7c2c0b3c90 con 0x7f7c2c0cd3b0 2024-06-29T15:57:20.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:20.537+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7c2c06cc90 con 0x7f7c2c0cd3b0 2024-06-29T15:57:21.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.188+0000 7f7c29ffb640 1 -- 172.21.15.59:0/4033399398 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v220) ==== 170+0+0 (secure 0 0 0) 0x7f7c20061440 con 0x7f7c2c0cd3b0 2024-06-29T15:57:21.189 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T15:57:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7c18076420 msgr2=0x7f7c18078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7c18076420 0x7f7c18078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c001ca0 tx=0x7f7c1c006c50 comp rx=0 tx=0).stop 2024-06-29T15:57:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c2c0cd3b0 msgr2=0x7f7c2c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c2c0cd3b0 0x7f7c2c0cd7c0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f7c20011860 tx=0x7f7c20011d20 comp rx=0 tx=0).stop 2024-06-29T15:57:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 shutdown_connections 2024-06-29T15:57:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7c18076420 0x7f7c18078930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c2c0b3340 0x7f7c2c0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c2c0cd3b0 0x7f7c2c0cd7c0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 --2- 172.21.15.59:0/4033399398 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7c14005250 0x7f7c2c151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 >> 172.21.15.59:0/4033399398 conn(0x7f7c2c0d04e0 msgr2=0x7f7c2c160820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:21.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 shutdown_connections 2024-06-29T15:57:21.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.191+0000 7f7c33681640 1 -- 172.21.15.59:0/4033399398 wait complete. 2024-06-29T15:57:21.218 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:57:21.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.344+0000 7f420ff73640 1 -- 172.21.15.59:0/3670198088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f420806f8a0 msgr2=0x7f42080cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.344+0000 7f420ff73640 1 --2- 172.21.15.59:0/3670198088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f420806f8a0 0x7f42080cf870 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f41fc005fd0 tx=0x7f41fc02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.344+0000 7f420ff73640 1 -- 172.21.15.59:0/3670198088 shutdown_connections 2024-06-29T15:57:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.344+0000 7f420ff73640 1 --2- 172.21.15.59:0/3670198088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f420806f8a0 0x7f42080cf870 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.344+0000 7f420ff73640 1 --2- 172.21.15.59:0/3670198088 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4208147370 0x7f42080b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.344+0000 7f420ff73640 1 -- 172.21.15.59:0/3670198088 >> 172.21.15.59:0/3670198088 conn(0x7f420806c950 msgr2=0x7f4208073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.344+0000 7f420ff73640 1 -- 172.21.15.59:0/3670198088 shutdown_connections 2024-06-29T15:57:21.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.344+0000 7f420ff73640 1 -- 172.21.15.59:0/3670198088 wait complete. 2024-06-29T15:57:21.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.346+0000 7f420ff73640 1 Processor -- start 2024-06-29T15:57:21.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.346+0000 7f420ff73640 1 -- start start 2024-06-29T15:57:21.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.347+0000 7f420ff73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f420806f8a0 0x7f420815d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.347+0000 7f420ff73640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4208147370 0x7f42080b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.347+0000 7f420ff73640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4208161b00 0x7f42080b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.347+0000 7f420ff73640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f420815eb10 con 0x7f420806f8a0 2024-06-29T15:57:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.347+0000 7f420ff73640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f420815e990 con 0x7f4208161b00 2024-06-29T15:57:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.347+0000 7f420ff73640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f420815ec90 con 0x7f4208147370 2024-06-29T15:57:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f420806f8a0 0x7f420815d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f420806f8a0 0x7f420815d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59638/0 (socket says 172.21.15.59:59638) 2024-06-29T15:57:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 -- 172.21.15.59:0/3913737745 learned_addr learned my addr 172.21.15.59:0/3913737745 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420d50e640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4208147370 0x7f42080b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420e510640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4208161b00 0x7f42080b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 -- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4208147370 msgr2=0x7f42080b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4208147370 0x7f42080b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 -- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4208161b00 msgr2=0x7f42080b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4208161b00 0x7f42080b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 -- 172.21.15.59:0/3913737745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42080ce990 con 0x7f420806f8a0 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420dd0f640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f420806f8a0 0x7f420815d2e0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f41f800ecb0 tx=0x7f41f800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f42067fc640 1 -- 172.21.15.59:0/3913737745 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8003b70 con 0x7f420806f8a0 2024-06-29T15:57:21.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f42067fc640 1 -- 172.21.15.59:0/3913737745 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41f8003d10 con 0x7f420806f8a0 2024-06-29T15:57:21.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42080cec50 con 0x7f420806f8a0 2024-06-29T15:57:21.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f42067fc640 1 -- 172.21.15.59:0/3913737745 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8004280 con 0x7f420806f8a0 2024-06-29T15:57:21.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.348+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420815c5d0 con 0x7f420806f8a0 2024-06-29T15:57:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.349+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42080d1590 con 0x7f420806f8a0 2024-06-29T15:57:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.350+0000 7f42067fc640 1 -- 172.21.15.59:0/3913737745 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f41f8004420 con 0x7f420806f8a0 2024-06-29T15:57:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.351+0000 7f42067fc640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f41f4076420 0x7f41f4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.351+0000 7f42067fc640 1 -- 172.21.15.59:0/3913737745 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(220..220 src has 1..220) ==== 6057+0+0 (secure 0 0 0) 0x7f41f80987d0 con 0x7f420806f8a0 2024-06-29T15:57:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.351+0000 7f420d50e640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f41f4076420 0x7f41f4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.352+0000 7f420d50e640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f41f4076420 0x7f41f4078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41fc033000 tx=0x7f41fc004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:21.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.355+0000 7f42067fc640 1 -- 172.21.15.59:0/3913737745 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f41f8054af0 con 0x7f420806f8a0 2024-06-29T15:57:21.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:21.587+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f420815f220 con 0x7f420806f8a0 2024-06-29T15:57:22.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.203+0000 7f42067fc640 1 -- 172.21.15.59:0/3913737745 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v111) ==== 174+0+0 (secure 0 0 0) 0x7f42080d1590 con 0x7f420806f8a0 2024-06-29T15:57:22.204 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 27 and data pool 28 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.206+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f41f4076420 msgr2=0x7f41f4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.206+0000 7f420ff73640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f41f4076420 0x7f41f4078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41fc033000 tx=0x7f41fc004730 comp rx=0 tx=0).stop 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.206+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f420806f8a0 msgr2=0x7f420815d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.206+0000 7f420ff73640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f420806f8a0 0x7f420815d2e0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f41f800ecb0 tx=0x7f41f800c6a0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.207+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 shutdown_connections 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.207+0000 7f420ff73640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f41f4076420 0x7f41f4078930 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.207+0000 7f420ff73640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4208161b00 0x7f42080b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.207+0000 7f420ff73640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4208147370 0x7f42080b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.207+0000 7f420ff73640 1 --2- 172.21.15.59:0/3913737745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f420806f8a0 0x7f420815d2e0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.207+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 >> 172.21.15.59:0/3913737745 conn(0x7f420806c950 msgr2=0x7f4208073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.207+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 shutdown_connections 2024-06-29T15:57:22.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.207+0000 7f420ff73640 1 -- 172.21.15.59:0/3913737745 wait complete. 2024-06-29T15:57:22.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:22.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.359+0000 7fb5aff24640 1 -- 172.21.15.59:0/1991043212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d2360 msgr2=0x7fb5a80b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.359+0000 7fb5aff24640 1 --2- 172.21.15.59:0/1991043212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d2360 0x7fb5a80b4b70 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fb598005fd0 tx=0x7fb59802aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:22.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.359+0000 7fb5aff24640 1 -- 172.21.15.59:0/1991043212 shutdown_connections 2024-06-29T15:57:22.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.359+0000 7fb5aff24640 1 --2- 172.21.15.59:0/1991043212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d2360 0x7fb5a80b4b70 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.359+0000 7fb5aff24640 1 -- 172.21.15.59:0/1991043212 >> 172.21.15.59:0/1991043212 conn(0x7fb5a8073e80 msgr2=0x7fb5a8074290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.359+0000 7fb5aff24640 1 -- 172.21.15.59:0/1991043212 shutdown_connections 2024-06-29T15:57:22.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.359+0000 7fb5aff24640 1 -- 172.21.15.59:0/1991043212 wait complete. 2024-06-29T15:57:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.361+0000 7fb5aff24640 1 Processor -- start 2024-06-29T15:57:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.361+0000 7fb5aff24640 1 -- start start 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.362+0000 7fb5aff24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d12f0 0x7fb5a815af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.362+0000 7fb5aff24640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5a8145570 0x7fb5a813cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.362+0000 7fb5aff24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a813d450 0x7fb5a81f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.362+0000 7fb5aff24640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a80b6710 con 0x7fb5a80d12f0 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.362+0000 7fb5aff24640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a80b6590 con 0x7fb5a813d450 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.362+0000 7fb5aff24640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a80b6890 con 0x7fb5a8145570 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d12f0 0x7fb5a815af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d12f0 0x7fb5a815af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59648/0 (socket says 172.21.15.59:59648) 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 -- 172.21.15.59:0/325216267 learned_addr learned my addr 172.21.15.59:0/325216267 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5ae4c1640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a813d450 0x7fb5a81f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5ad4bf640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5a8145570 0x7fb5a813cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 -- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5a8145570 msgr2=0x7fb5a813cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5a8145570 0x7fb5a813cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 -- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a813d450 msgr2=0x7fb5a81f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a813d450 0x7fb5a81f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 -- 172.21.15.59:0/325216267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a813dbd0 con 0x7fb5a80d12f0 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5adcc0640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d12f0 0x7fb5a815af20 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000c3f0 tx=0x7fb5a000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5a67fc640 1 -- 172.21.15.59:0/325216267 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0017070 con 0x7fb5a80d12f0 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5a67fc640 1 -- 172.21.15.59:0/325216267 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5a00035f0 con 0x7fb5a80d12f0 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a813deb0 con 0x7fb5a80d12f0 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5a67fc640 1 -- 172.21.15.59:0/325216267 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0003a40 con 0x7fb5a80d12f0 2024-06-29T15:57:22.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.363+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a813e470 con 0x7fb5a80d12f0 2024-06-29T15:57:22.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.365+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a80d03c0 con 0x7fb5a80d12f0 2024-06-29T15:57:22.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.365+0000 7fb5a67fc640 1 -- 172.21.15.59:0/325216267 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fb5a001a070 con 0x7fb5a80d12f0 2024-06-29T15:57:22.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.366+0000 7fb5a67fc640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb5940763f0 0x7fb594078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.366+0000 7fb5a67fc640 1 -- 172.21.15.59:0/325216267 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 6157+0+0 (secure 0 0 0) 0x7fb5a0096ec0 con 0x7fb5a80d12f0 2024-06-29T15:57:22.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.366+0000 7fb5ad4bf640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb5940763f0 0x7fb594078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.366+0000 7fb5ad4bf640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb5940763f0 0x7fb594078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb59c006fd0 tx=0x7fb59c008040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:22.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.370+0000 7fb5a67fc640 1 -- 172.21.15.59:0/325216267 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb5a80d03c0 con 0x7fb5a80d12f0 2024-06-29T15:57:22.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.564+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5a80b4b70 con 0x7fb5a80d12f0 2024-06-29T15:57:22.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.565+0000 7fb5a67fc640 1 -- 172.21.15.59:0/325216267 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) ==== 74+0+17067 (secure 0 0 0) 0x7fb5a00604c0 con 0x7fb5a80d12f0 2024-06-29T15:57:22.567 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:22.567 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":221,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:21.589093+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:22.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb5940763f0 msgr2=0x7fb594078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb5940763f0 0x7fb594078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb59c006fd0 tx=0x7fb59c008040 comp rx=0 tx=0).stop 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d12f0 msgr2=0x7fb5a815af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d12f0 0x7fb5a815af20 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000c3f0 tx=0x7fb5a000c8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 shutdown_connections 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fb5940763f0 0x7fb594078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5a813d450 0x7fb5a81f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb5a8145570 0x7fb5a813cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 --2- 172.21.15.59:0/325216267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5a80d12f0 0x7fb5a815af20 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 >> 172.21.15.59:0/325216267 conn(0x7fb5a8073e80 msgr2=0x7fb5a80b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 shutdown_connections 2024-06-29T15:57:22.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.567+0000 7fb5aff24640 1 -- 172.21.15.59:0/325216267 wait complete. 2024-06-29T15:57:22.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.697+0000 7f7f67026640 1 -- 172.21.15.59:0/2345659455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f600b40a0 msgr2=0x7f7f6015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.697+0000 7f7f67026640 1 --2- 172.21.15.59:0/2345659455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f600b40a0 0x7f7f6015e8d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005fd0 tx=0x7f7f4c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.697+0000 7f7f67026640 1 -- 172.21.15.59:0/2345659455 shutdown_connections 2024-06-29T15:57:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.697+0000 7f7f67026640 1 --2- 172.21.15.59:0/2345659455 >> v2:172.21.15.153:3300/0 conn(0x7f7f6015ee10 0x7f7f6015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.697+0000 7f7f67026640 1 --2- 172.21.15.59:0/2345659455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f600b40a0 0x7f7f6015e8d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.697+0000 7f7f67026640 1 -- 172.21.15.59:0/2345659455 >> 172.21.15.59:0/2345659455 conn(0x7f7f600d3ee0 msgr2=0x7f7f600d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.697+0000 7f7f67026640 1 -- 172.21.15.59:0/2345659455 shutdown_connections 2024-06-29T15:57:22.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.697+0000 7f7f67026640 1 -- 172.21.15.59:0/2345659455 wait complete. 2024-06-29T15:57:22.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.699+0000 7f7f67026640 1 Processor -- start 2024-06-29T15:57:22.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.699+0000 7f7f67026640 1 -- start start 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f67026640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f600d23c0 0x7f7f601526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f67026640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f6015ee10 0x7f7f6015e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f67026640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f60151070 0x7f7f601f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f67026640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60160280 con 0x7f7f600d23c0 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f67026640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60160100 con 0x7f7f6015ee10 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f67026640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7f60160400 con 0x7f7f60151070 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f64dc2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f600d23c0 0x7f7f601526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f64dc2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f600d23c0 0x7f7f601526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45896/0 (socket says 172.21.15.59:45896) 2024-06-29T15:57:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f64dc2640 1 -- 172.21.15.59:0/1549343131 learned_addr learned my addr 172.21.15.59:0/1549343131 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f655c3640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f60151070 0x7f7f601f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.700+0000 7f7f5ffff640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f6015ee10 0x7f7f6015e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f64dc2640 1 -- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f60151070 msgr2=0x7f7f601f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f64dc2640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f60151070 0x7f7f601f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f64dc2640 1 -- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f6015ee10 msgr2=0x7f7f6015e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f64dc2640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f6015ee10 0x7f7f6015e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f64dc2640 1 -- 172.21.15.59:0/1549343131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f60151610 con 0x7f7f600d23c0 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f655c3640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f60151070 0x7f7f601f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f5ffff640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f6015ee10 0x7f7f6015e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:22.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f64dc2640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f600d23c0 0x7f7f601526b0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c370 tx=0x7f7f5400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:22.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f5d7fa640 1 -- 172.21.15.59:0/1549343131 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54017070 con 0x7f7f600d23c0 2024-06-29T15:57:22.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f601518f0 con 0x7f7f600d23c0 2024-06-29T15:57:22.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f5d7fa640 1 -- 172.21.15.59:0/1549343131 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f540053c0 con 0x7f7f600d23c0 2024-06-29T15:57:22.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f5d7fa640 1 -- 172.21.15.59:0/1549343131 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54003850 con 0x7f7f600d23c0 2024-06-29T15:57:22.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.701+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f60151eb0 con 0x7f7f600d23c0 2024-06-29T15:57:22.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.702+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f6015e8d0 con 0x7f7f600d23c0 2024-06-29T15:57:22.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.703+0000 7f7f5d7fa640 1 -- 172.21.15.59:0/1549343131 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f7f5401a070 con 0x7f7f600d23c0 2024-06-29T15:57:22.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.703+0000 7f7f5d7fa640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7f44076420 0x7f7f44078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.704+0000 7f7f5d7fa640 1 -- 172.21.15.59:0/1549343131 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 6157+0+0 (secure 0 0 0) 0x7f7f54096e90 con 0x7f7f600d23c0 2024-06-29T15:57:22.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.704+0000 7f7f5ffff640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7f44076420 0x7f7f44078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.704+0000 7f7f5ffff640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7f44076420 0x7f7f44078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c033000 tx=0x7f7f4c004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:22.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.708+0000 7f7f5d7fa640 1 -- 172.21.15.59:0/1549343131 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7f540604a0 con 0x7f7f600d23c0 2024-06-29T15:57:22.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.901+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f60158310 con 0x7f7f600d23c0 2024-06-29T15:57:22.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.902+0000 7f7f5d7fa640 1 -- 172.21.15.59:0/1549343131 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) ==== 74+0+17067 (secure 0 0 0) 0x7f7f5405fda0 con 0x7f7f600d23c0 2024-06-29T15:57:22.903 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:22.903 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":221,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:21.589093+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:22.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7f44076420 msgr2=0x7f7f44078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7f44076420 0x7f7f44078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c033000 tx=0x7f7f4c004730 comp rx=0 tx=0).stop 2024-06-29T15:57:22.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f600d23c0 msgr2=0x7f7f601526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f600d23c0 0x7f7f601526b0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c370 tx=0x7f7f5400c830 comp rx=0 tx=0).stop 2024-06-29T15:57:22.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 shutdown_connections 2024-06-29T15:57:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7f44076420 0x7f7f44078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f60151070 0x7f7f601f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f6015ee10 0x7f7f6015e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 --2- 172.21.15.59:0/1549343131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f600d23c0 0x7f7f601526b0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 >> 172.21.15.59:0/1549343131 conn(0x7f7f600d3ee0 msgr2=0x7f7f600b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 shutdown_connections 2024-06-29T15:57:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:22.904+0000 7f7f67026640 1 -- 172.21.15.59:0/1549343131 wait complete. 2024-06-29T15:57:22.917 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:57:23.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.037+0000 7f947a544640 1 -- 172.21.15.59:0/2495340309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9474141bd0 msgr2=0x7f947413edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:23.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.037+0000 7f947a544640 1 --2- 172.21.15.59:0/2495340309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9474141bd0 0x7f947413edc0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f9474069670 tx=0x7f946c02aa30 comp rx=0 tx=0).stop 2024-06-29T15:57:23.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.037+0000 7f947a544640 1 -- 172.21.15.59:0/2495340309 shutdown_connections 2024-06-29T15:57:23.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.037+0000 7f947a544640 1 --2- 172.21.15.59:0/2495340309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9474141bd0 0x7f947413edc0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:23.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.037+0000 7f947a544640 1 --2- 172.21.15.59:0/2495340309 >> v2:172.21.15.153:3300/0 conn(0x7f947415ceb0 0x7f94740b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:23.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.037+0000 7f947a544640 1 -- 172.21.15.59:0/2495340309 >> 172.21.15.59:0/2495340309 conn(0x7f94740b6fb0 msgr2=0x7f94740b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:23.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.037+0000 7f947a544640 1 -- 172.21.15.59:0/2495340309 shutdown_connections 2024-06-29T15:57:23.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.037+0000 7f947a544640 1 -- 172.21.15.59:0/2495340309 wait complete. 2024-06-29T15:57:23.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.038+0000 7f947a544640 1 Processor -- start 2024-06-29T15:57:23.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.039+0000 7f947a544640 1 -- start start 2024-06-29T15:57:23.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f947a544640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f947413e9b0 0x7f9474152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:23.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f947a544640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f947415ceb0 0x7f947415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:23.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f947a544640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9474139ac0 0x7f94740cf110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:23.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f947a544640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f94740b29d0 con 0x7f947413e9b0 2024-06-29T15:57:23.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f947a544640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94740b2850 con 0x7f947415ceb0 2024-06-29T15:57:23.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f947a544640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f94740b2b50 con 0x7f9474139ac0 2024-06-29T15:57:23.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9473fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f947413e9b0 0x7f9474152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:23.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9473fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f947413e9b0 0x7f9474152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45912/0 (socket says 172.21.15.59:45912) 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9473fff640 1 -- 172.21.15.59:0/1791256794 learned_addr learned my addr 172.21.15.59:0/1791256794 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f94737fe640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f947415ceb0 0x7f947415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9478ae1640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9474139ac0 0x7f94740cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9473fff640 1 -- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9474139ac0 msgr2=0x7f94740cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9473fff640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9474139ac0 0x7f94740cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9473fff640 1 -- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f947415ceb0 msgr2=0x7f947415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9473fff640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f947415ceb0 0x7f947415ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.040+0000 7f9473fff640 1 -- 172.21.15.59:0/1791256794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947415b340 con 0x7f947413e9b0 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.041+0000 7f94737fe640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f947415ceb0 0x7f947415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.041+0000 7f9478ae1640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9474139ac0 0x7f94740cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.041+0000 7f9473fff640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f947413e9b0 0x7f9474152a70 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f946400c9a0 tx=0x7f946400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:23.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.041+0000 7f9470ff9640 1 -- 172.21.15.59:0/1791256794 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9464007b50 con 0x7f947413e9b0 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.041+0000 7f9470ff9640 1 -- 172.21.15.59:0/1791256794 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9464007de0 con 0x7f947413e9b0 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.041+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947415b610 con 0x7f947413e9b0 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.041+0000 7f9470ff9640 1 -- 172.21.15.59:0/1791256794 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946400f6b0 con 0x7f947413e9b0 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.041+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947415bbd0 con 0x7f947413e9b0 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.042+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9474156a80 con 0x7f947413e9b0 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.043+0000 7f9470ff9640 1 -- 172.21.15.59:0/1791256794 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f946400f850 con 0x7f947413e9b0 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.043+0000 7f9470ff9640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f94600763f0 0x7f9460078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.043+0000 7f9470ff9640 1 -- 172.21.15.59:0/1791256794 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 6157+0+0 (secure 0 0 0) 0x7f94640982c0 con 0x7f947413e9b0 2024-06-29T15:57:23.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.043+0000 7f94737fe640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f94600763f0 0x7f9460078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:23.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.044+0000 7f94737fe640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f94600763f0 0x7f9460078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f947413ab10 tx=0x7f9468009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:23.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.048+0000 7f9470ff9640 1 -- 172.21.15.59:0/1791256794 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9464014030 con 0x7f947413e9b0 2024-06-29T15:57:23.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:23.282+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f94740b3000 con 0x7f947413e9b0 2024-06-29T15:57:24.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.235+0000 7f9470ff9640 1 -- 172.21.15.59:0/1791256794 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v114) ==== 118+0+0 (secure 0 0 0) 0x7f94640617b0 con 0x7f947413e9b0 2024-06-29T15:57:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f94600763f0 msgr2=0x7f9460078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f94600763f0 0x7f9460078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f947413ab10 tx=0x7f9468009040 comp rx=0 tx=0).stop 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f947413e9b0 msgr2=0x7f9474152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f947413e9b0 0x7f9474152a70 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f946400c9a0 tx=0x7f946400ce60 comp rx=0 tx=0).stop 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 shutdown_connections 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f94600763f0 0x7f9460078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9474139ac0 0x7f94740cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f947415ceb0 0x7f947415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 --2- 172.21.15.59:0/1791256794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f947413e9b0 0x7f9474152a70 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 >> 172.21.15.59:0/1791256794 conn(0x7f94740b6fb0 msgr2=0x7f947415e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.238+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 shutdown_connections 2024-06-29T15:57:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.239+0000 7f947a544640 1 -- 172.21.15.59:0/1791256794 wait complete. 2024-06-29T15:57:24.254 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:57:24.254 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:57:24.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.381+0000 7f6a357eb640 1 -- 172.21.15.59:0/2966894376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a300d23c0 msgr2=0x7f6a300b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.381+0000 7f6a357eb640 1 --2- 172.21.15.59:0/2966894376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a300d23c0 0x7f6a300b3990 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f6a20005fd0 tx=0x7f6a2002aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:24.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.381+0000 7f6a357eb640 1 -- 172.21.15.59:0/2966894376 shutdown_connections 2024-06-29T15:57:24.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.381+0000 7f6a357eb640 1 --2- 172.21.15.59:0/2966894376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a300d23c0 0x7f6a300b3990 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.381+0000 7f6a357eb640 1 -- 172.21.15.59:0/2966894376 >> 172.21.15.59:0/2966894376 conn(0x7f6a300d3ee0 msgr2=0x7f6a300d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:24.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.381+0000 7f6a357eb640 1 -- 172.21.15.59:0/2966894376 shutdown_connections 2024-06-29T15:57:24.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.381+0000 7f6a357eb640 1 -- 172.21.15.59:0/2966894376 wait complete. 2024-06-29T15:57:24.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.383+0000 7f6a357eb640 1 Processor -- start 2024-06-29T15:57:24.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.383+0000 7f6a357eb640 1 -- start start 2024-06-29T15:57:24.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.384+0000 7f6a357eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a300b3ed0 0x7f6a3014a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a357eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a300d23c0 0x7f6a300c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a357eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a3015efe0 0x7f6a300c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a357eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6a30160450 con 0x7f6a3015efe0 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a357eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a301602d0 con 0x7f6a300d23c0 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a357eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6a301605d0 con 0x7f6a300b3ed0 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a3015efe0 0x7f6a300c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a3015efe0 0x7f6a300c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45940/0 (socket says 172.21.15.59:45940) 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 -- 172.21.15.59:0/4023720300 learned_addr learned my addr 172.21.15.59:0/4023720300 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2e7fc640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a300d23c0 0x7f6a300c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2effd640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a300b3ed0 0x7f6a3014a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 -- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a300b3ed0 msgr2=0x7f6a3014a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a300b3ed0 0x7f6a3014a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 -- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a300d23c0 msgr2=0x7f6a300c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a300d23c0 0x7f6a300c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 -- 172.21.15.59:0/4023720300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a300c6ff0 con 0x7f6a3015efe0 2024-06-29T15:57:24.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a2f7fe640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a3015efe0 0x7f6a300c6570 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800b410 tx=0x7f6a2800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:24.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a177fe640 1 -- 172.21.15.59:0/4023720300 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28015030 con 0x7f6a3015efe0 2024-06-29T15:57:24.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a177fe640 1 -- 172.21.15.59:0/4023720300 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a28004130 con 0x7f6a3015efe0 2024-06-29T15:57:24.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.385+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a30157d80 con 0x7f6a3015efe0 2024-06-29T15:57:24.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.386+0000 7f6a177fe640 1 -- 172.21.15.59:0/4023720300 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28011630 con 0x7f6a3015efe0 2024-06-29T15:57:24.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.386+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a30158340 con 0x7f6a3015efe0 2024-06-29T15:57:24.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.386+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a300b3990 con 0x7f6a3015efe0 2024-06-29T15:57:24.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.387+0000 7f6a177fe640 1 -- 172.21.15.59:0/4023720300 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f6a28018070 con 0x7f6a3015efe0 2024-06-29T15:57:24.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.388+0000 7f6a177fe640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a1c076420 0x7f6a1c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:24.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.388+0000 7f6a177fe640 1 -- 172.21.15.59:0/4023720300 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f6a28097480 con 0x7f6a3015efe0 2024-06-29T15:57:24.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.388+0000 7f6a2effd640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a1c076420 0x7f6a1c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:24.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.389+0000 7f6a2effd640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a1c076420 0x7f6a1c078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6a20035000 tx=0x7f6a20038040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.392+0000 7f6a177fe640 1 -- 172.21.15.59:0/4023720300 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6a28004580 con 0x7f6a3015efe0 2024-06-29T15:57:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:24.621+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6a300c6d80 con 0x7f6a3015efe0 2024-06-29T15:57:25.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.241+0000 7f6a177fe640 1 -- 172.21.15.59:0/4023720300 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v115) ==== 122+0+0 (secure 0 0 0) 0x7f6a28060440 con 0x7f6a3015efe0 2024-06-29T15:57:25.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a1c076420 msgr2=0x7f6a1c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a1c076420 0x7f6a1c078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6a20035000 tx=0x7f6a20038040 comp rx=0 tx=0).stop 2024-06-29T15:57:25.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a3015efe0 msgr2=0x7f6a300c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a3015efe0 0x7f6a300c6570 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800b410 tx=0x7f6a2800b8d0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 shutdown_connections 2024-06-29T15:57:25.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a1c076420 0x7f6a1c078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a3015efe0 0x7f6a300c6570 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a300d23c0 0x7f6a300c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 --2- 172.21.15.59:0/4023720300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a300b3ed0 0x7f6a3014a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 >> 172.21.15.59:0/4023720300 conn(0x7f6a300d3ee0 msgr2=0x7f6a300ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 shutdown_connections 2024-06-29T15:57:25.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.244+0000 7f6a357eb640 1 -- 172.21.15.59:0/4023720300 wait complete. 2024-06-29T15:57:25.259 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:57:25.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:25.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.383+0000 7ff1c04ac640 1 -- 172.21.15.59:0/1702936940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b813e9b0 msgr2=0x7ff1b813edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.383+0000 7ff1c04ac640 1 --2- 172.21.15.59:0/1702936940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b813e9b0 0x7ff1b813edc0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac005fd0 tx=0x7ff1ac02a540 comp rx=0 tx=0).stop 2024-06-29T15:57:25.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.383+0000 7ff1c04ac640 1 -- 172.21.15.59:0/1702936940 shutdown_connections 2024-06-29T15:57:25.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.383+0000 7ff1c04ac640 1 --2- 172.21.15.59:0/1702936940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b813e9b0 0x7ff1b813edc0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.383+0000 7ff1c04ac640 1 -- 172.21.15.59:0/1702936940 >> 172.21.15.59:0/1702936940 conn(0x7ff1b80b6fb0 msgr2=0x7ff1b80b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.383+0000 7ff1c04ac640 1 -- 172.21.15.59:0/1702936940 shutdown_connections 2024-06-29T15:57:25.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.383+0000 7ff1c04ac640 1 -- 172.21.15.59:0/1702936940 wait complete. 2024-06-29T15:57:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.385+0000 7ff1c04ac640 1 Processor -- start 2024-06-29T15:57:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.385+0000 7ff1c04ac640 1 -- start start 2024-06-29T15:57:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.386+0000 7ff1c04ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b813e9b0 0x7ff1b8153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.386+0000 7ff1c04ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1b81414c0 0x7ff1b812eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.386+0000 7ff1c04ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b815ceb0 0x7ff1b812eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.386+0000 7ff1c04ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b80b2900 con 0x7ff1b815ceb0 2024-06-29T15:57:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.386+0000 7ff1c04ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b80b2780 con 0x7ff1b813e9b0 2024-06-29T15:57:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.386+0000 7ff1c04ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b80b2a80 con 0x7ff1b81414c0 2024-06-29T15:57:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.386+0000 7ff1bea49640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b815ceb0 0x7ff1b812eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bea49640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b815ceb0 0x7ff1b812eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45952/0 (socket says 172.21.15.59:45952) 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bea49640 1 -- 172.21.15.59:0/3683607318 learned_addr learned my addr 172.21.15.59:0/3683607318 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1be248640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b813e9b0 0x7ff1b8153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bda47640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1b81414c0 0x7ff1b812eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bea49640 1 -- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1b81414c0 msgr2=0x7ff1b812eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bea49640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1b81414c0 0x7ff1b812eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bea49640 1 -- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b813e9b0 msgr2=0x7ff1b8153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bea49640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b813e9b0 0x7ff1b8153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bea49640 1 -- 172.21.15.59:0/3683607318 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b8153f80 con 0x7ff1b815ceb0 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1bea49640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b815ceb0 0x7ff1b812eff0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000b4c0 tx=0x7ff1b000b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1b6ffd640 1 -- 172.21.15.59:0/3683607318 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0015020 con 0x7ff1b815ceb0 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1b6ffd640 1 -- 172.21.15.59:0/3683607318 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1b0011940 con 0x7ff1b815ceb0 2024-06-29T15:57:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b813c880 con 0x7ff1b815ceb0 2024-06-29T15:57:25.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1b6ffd640 1 -- 172.21.15.59:0/3683607318 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0011d90 con 0x7ff1b815ceb0 2024-06-29T15:57:25.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.387+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b80ca270 con 0x7ff1b815ceb0 2024-06-29T15:57:25.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.388+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b80c9d00 con 0x7ff1b815ceb0 2024-06-29T15:57:25.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.389+0000 7ff1b6ffd640 1 -- 172.21.15.59:0/3683607318 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7ff1b0018020 con 0x7ff1b815ceb0 2024-06-29T15:57:25.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.390+0000 7ff1b6ffd640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff1a40763f0 0x7ff1a4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.390+0000 7ff1b6ffd640 1 -- 172.21.15.59:0/3683607318 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7ff1b0097aa0 con 0x7ff1b815ceb0 2024-06-29T15:57:25.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.390+0000 7ff1be248640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff1a40763f0 0x7ff1a4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.390+0000 7ff1be248640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff1a40763f0 0x7ff1a4078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8007fd0 tx=0x7ff1a8009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:25.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.394+0000 7ff1b6ffd640 1 -- 172.21.15.59:0/3683607318 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff1b001f020 con 0x7ff1b815ceb0 2024-06-29T15:57:25.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.627+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1b80b2f70 con 0x7ff1b815ceb0 2024-06-29T15:57:25.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.627+0000 7ff1b6ffd640 1 -- 172.21.15.59:0/3683607318 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5242 (secure 0 0 0) 0x7ff1b8153f80 con 0x7ff1b815ceb0 2024-06-29T15:57:25.628 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:25.629 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":115,"btime":"2024-06-29T15:57:25:241476+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":10747,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.589050+0000","modified":"2024-06-29T15:57:24.623107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11487":{"gid":11487,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1734341893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1734341893},{"type":"v1","addr":"172.21.15.59:6834","nonce":1734341893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11487,"qdb_cluster":[11487]},"id":13}]} 2024-06-29T15:57:25.629 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 115 2024-06-29T15:57:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff1a40763f0 msgr2=0x7ff1a4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff1a40763f0 0x7ff1a4078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8007fd0 tx=0x7ff1a8009040 comp rx=0 tx=0).stop 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b815ceb0 msgr2=0x7ff1b812eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b815ceb0 0x7ff1b812eff0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000b4c0 tx=0x7ff1b000b980 comp rx=0 tx=0).stop 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 shutdown_connections 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff1a40763f0 0x7ff1a4078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b815ceb0 0x7ff1b812eff0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1b81414c0 0x7ff1b812eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 --2- 172.21.15.59:0/3683607318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1b813e9b0 0x7ff1b8153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 >> 172.21.15.59:0/3683607318 conn(0x7ff1b80b6fb0 msgr2=0x7ff1b806fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 shutdown_connections 2024-06-29T15:57:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.629+0000 7ff1c04ac640 1 -- 172.21.15.59:0/3683607318 wait complete. 2024-06-29T15:57:25.642 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/1444102187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec15ee10 msgr2=0x7fc7ec15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/1444102187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec15ee10 0x7fc7ec15f9b0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0b4a80 tx=0x7fc7e402a560 comp rx=0 tx=0).stop 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/1444102187 shutdown_connections 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/1444102187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec15ee10 0x7fc7ec15f9b0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/1444102187 >> v2:172.21.15.153:3300/0 conn(0x7fc7ec0b3ed0 0x7fc7ec15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/1444102187 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7ec0d23c0 0x7fc7ec0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/1444102187 >> 172.21.15.59:0/1444102187 conn(0x7fc7ec0d3ee0 msgr2=0x7fc7ec0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/1444102187 shutdown_connections 2024-06-29T15:57:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.758+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/1444102187 wait complete. 2024-06-29T15:57:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.760+0000 7fc7f2bf5640 1 Processor -- start 2024-06-29T15:57:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.760+0000 7fc7f2bf5640 1 -- start start 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f2bf5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec0b3ed0 0x7fc7ec156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f2bf5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7ec0d23c0 0x7fc7ec15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f2bf5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7ec15ee10 0x7fc7ec15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f2bf5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec160070 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f2bf5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec15fef0 con 0x7fc7ec15ee10 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f2bf5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc7ec1601f0 con 0x7fc7ec0d23c0 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec0b3ed0 0x7fc7ec156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec0b3ed0 0x7fc7ec156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45972/0 (socket says 172.21.15.59:45972) 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 -- 172.21.15.59:0/51262934 learned_addr learned my addr 172.21.15.59:0/51262934 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7ebfff640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7ec0d23c0 0x7fc7ec15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f1192640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7ec15ee10 0x7fc7ec15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 -- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7ec0d23c0 msgr2=0x7fc7ec15bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7ec0d23c0 0x7fc7ec15bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 -- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7ec15ee10 msgr2=0x7fc7ec15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7ec15ee10 0x7fc7ec15c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 -- 172.21.15.59:0/51262934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ec1f53d0 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f0991640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec0b3ed0 0x7fc7ec156c40 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00d780 tx=0x7fc7dc00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7e97fa640 1 -- 172.21.15.59:0/51262934 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc017070 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7e97fa640 1 -- 172.21.15.59:0/51262934 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7dc009b10 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7e97fa640 1 -- 172.21.15.59:0/51262934 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc010460 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.762+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ec1f56b0 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.763+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ec1f5c70 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.763+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7ec0b3990 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.764+0000 7fc7e97fa640 1 -- 172.21.15.59:0/51262934 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fc7dc01a070 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.765+0000 7fc7e97fa640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc7cc0763f0 0x7fc7cc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.765+0000 7fc7e97fa640 1 -- 172.21.15.59:0/51262934 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7fc7dc097dc0 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.765+0000 7fc7ebfff640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc7cc0763f0 0x7fc7cc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.766+0000 7fc7ebfff640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc7cc0763f0 0x7fc7cc078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec15ec70 tx=0x7fc7d4009320 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:25.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.769+0000 7fc7e97fa640 1 -- 172.21.15.59:0/51262934 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc7dc0612b0 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.960+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7ec0d2950 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.961+0000 7fc7e97fa640 1 -- 172.21.15.59:0/51262934 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) ==== 74+0+17054 (secure 0 0 0) 0x7fc7dc060bb0 con 0x7fc7ec0b3ed0 2024-06-29T15:57:25.963 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:25.963 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":222,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:23.218941+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc7cc0763f0 msgr2=0x7fc7cc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc7cc0763f0 0x7fc7cc078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec15ec70 tx=0x7fc7d4009320 comp rx=0 tx=0).stop 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec0b3ed0 msgr2=0x7fc7ec156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec0b3ed0 0x7fc7ec156c40 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00d780 tx=0x7fc7dc00dc40 comp rx=0 tx=0).stop 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 shutdown_connections 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc7cc0763f0 0x7fc7cc078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7ec15ee10 0x7fc7ec15c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc7ec0d23c0 0x7fc7ec15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 --2- 172.21.15.59:0/51262934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7ec0b3ed0 0x7fc7ec156c40 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 >> 172.21.15.59:0/51262934 conn(0x7fc7ec0d3ee0 msgr2=0x7fc7ec06cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 shutdown_connections 2024-06-29T15:57:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:25.964+0000 7fc7f2bf5640 1 -- 172.21.15.59:0/51262934 wait complete. 2024-06-29T15:57:25.976 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:57:25.976 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:26.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.094+0000 7f1dc4e22640 1 -- 172.21.15.59:0/3986360814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da8004d90 msgr2=0x7f1da8005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.094+0000 7f1dc4e22640 1 --2- 172.21.15.59:0/3986360814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1da8004d90 0x7f1da8005260 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f1db80016c0 tx=0x7f1da8005830 comp rx=0 tx=0).stop 2024-06-29T15:57:26.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.094+0000 7f1dbed9d640 1 -- 172.21.15.59:0/3986360814 reap_dead start 2024-06-29T15:57:26.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.094+0000 7f1dc4e22640 1 -- 172.21.15.59:0/3986360814 shutdown_connections 2024-06-29T15:57:26.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.094+0000 7f1dc4e22640 1 -- 172.21.15.59:0/3986360814 >> 172.21.15.59:0/3986360814 conn(0x7f1dc00b5610 msgr2=0x7f1dc00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.094+0000 7f1dc4e22640 1 -- 172.21.15.59:0/3986360814 shutdown_connections 2024-06-29T15:57:26.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.094+0000 7f1dc4e22640 1 -- 172.21.15.59:0/3986360814 wait complete. 2024-06-29T15:57:26.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.096+0000 7f1dc4e22640 1 Processor -- start 2024-06-29T15:57:26.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.096+0000 7f1dc4e22640 1 -- start start 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dc4e22640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da8004d90 0x7f1dc015fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dc4e22640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0156ca0 0x7f1dc00cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dc4e22640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc00cc160 0x7f1dc0159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dc4e22640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc00cf200 con 0x7f1dc0156ca0 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dc4e22640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc00cf080 con 0x7f1dc00cc160 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dc4e22640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc00cf380 con 0x7f1da8004d90 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dbdd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0156ca0 0x7f1dc00cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dbdd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0156ca0 0x7f1dc00cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45996/0 (socket says 172.21.15.59:45996) 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dbdd9b640 1 -- 172.21.15.59:0/2107344942 learned_addr learned my addr 172.21.15.59:0/2107344942 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.097+0000 7f1dbe59c640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da8004d90 0x7f1dc015fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dbed9d640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc00cc160 0x7f1dc0159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dbdd9b640 1 -- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da8004d90 msgr2=0x7f1dc015fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dbdd9b640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da8004d90 0x7f1dc015fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dbdd9b640 1 -- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc00cc160 msgr2=0x7f1dc0159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dbdd9b640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc00cc160 0x7f1dc0159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dbdd9b640 1 -- 172.21.15.59:0/2107344942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc00cc570 con 0x7f1dc0156ca0 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dbed9d640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc00cc160 0x7f1dc0159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dbdd9b640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0156ca0 0x7f1dc00cbc20 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f1db400e600 tx=0x7f1db400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1daeffd640 1 -- 172.21.15.59:0/2107344942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4015070 con 0x7f1dc0156ca0 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1daeffd640 1 -- 172.21.15.59:0/2107344942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1db400ca30 con 0x7f1dc0156ca0 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc0161f00 con 0x7f1dc0156ca0 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1daeffd640 1 -- 172.21.15.59:0/2107344942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4011460 con 0x7f1dc0156ca0 2024-06-29T15:57:26.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.098+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc01624c0 con 0x7f1dc0156ca0 2024-06-29T15:57:26.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.099+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc00d1a20 con 0x7f1dc0156ca0 2024-06-29T15:57:26.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.100+0000 7f1daeffd640 1 -- 172.21.15.59:0/2107344942 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f1db4018070 con 0x7f1dc0156ca0 2024-06-29T15:57:26.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.100+0000 7f1daeffd640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1da0076420 0x7f1da0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.100+0000 7f1daeffd640 1 -- 172.21.15.59:0/2107344942 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f1db4098010 con 0x7f1dc0156ca0 2024-06-29T15:57:26.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.100+0000 7f1dbe59c640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1da0076420 0x7f1da0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.101+0000 7f1dbe59c640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1da0076420 0x7f1da0078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1db0005fd0 tx=0x7f1db0009500 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.105+0000 7f1daeffd640 1 -- 172.21.15.59:0/2107344942 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1db40614f0 con 0x7f1dc0156ca0 2024-06-29T15:57:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.292+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1dc00c73f0 con 0x7f1da0076420 2024-06-29T15:57:26.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.297+0000 7f1daeffd640 1 -- 172.21.15.59:0/2107344942 <== mgr.9206 v2:172.21.15.153:6832/2780049427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252498 (secure 0 0 0) 0x7f1db005da70 con 0x7f1da0076420 2024-06-29T15:57:26.301 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:26.303 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1da0076420 msgr2=0x7f1da0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1da0076420 0x7f1da0078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1db0005fd0 tx=0x7f1db0009500 comp rx=0 tx=0).stop 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0156ca0 msgr2=0x7f1dc00cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0156ca0 0x7f1dc00cbc20 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f1db400e600 tx=0x7f1db400eac0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 shutdown_connections 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1da0076420 0x7f1da0078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1dc00cc160 0x7f1dc0159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1dc0156ca0 0x7f1dc00cbc20 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 --2- 172.21.15.59:0/2107344942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1da8004d90 0x7f1dc015fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 >> 172.21.15.59:0/2107344942 conn(0x7f1dc00b5610 msgr2=0x7f1dc00cd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 shutdown_connections 2024-06-29T15:57:26.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.300+0000 7f1dc4e22640 1 -- 172.21.15.59:0/2107344942 wait complete. 2024-06-29T15:57:26.312 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-06-29T15:57:25.765922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":421,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380648,"kb_used_data":22904,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754594072,"statfs":{"total":773094113280,"available":772704329728,"internally_reserved":0,"allocated":23453696,"data_stored":12597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":366202418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.578148"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428245+0000","last_change":"2024-06-29T15:57:22.225592+0000","last_active":"2024-06-29T15:57:23.428245+0000","last_peered":"2024-06-29T15:57:23.428245+0000","last_clean":"2024-06-29T15:57:23.428245+0000","last_became_active":"2024-06-29T15:57:22.225309+0000","last_became_peered":"2024-06-29T15:57:22.225309+0000","last_unstale":"2024-06-29T15:57:23.428245+0000","last_undegraded":"2024-06-29T15:57:23.428245+0000","last_fullsized":"2024-06-29T15:57:23.428245+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:07.857349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559935+0000","last_change":"2024-06-29T15:57:22.231733+0000","last_active":"2024-06-29T15:57:23.559935+0000","last_peered":"2024-06-29T15:57:23.559935+0000","last_clean":"2024-06-29T15:57:23.559935+0000","last_became_active":"2024-06-29T15:57:22.231387+0000","last_became_peered":"2024-06-29T15:57:22.231387+0000","last_unstale":"2024-06-29T15:57:23.559935+0000","last_undegraded":"2024-06-29T15:57:23.559935+0000","last_fullsized":"2024-06-29T15:57:23.559935+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:41.892974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222792+0000","last_change":"2024-06-29T15:57:22.231073+0000","last_active":"2024-06-29T15:57:23.222792+0000","last_peered":"2024-06-29T15:57:23.222792+0000","last_clean":"2024-06-29T15:57:23.222792+0000","last_became_active":"2024-06-29T15:57:22.230749+0000","last_became_peered":"2024-06-29T15:57:22.230749+0000","last_unstale":"2024-06-29T15:57:23.222792+0000","last_undegraded":"2024-06-29T15:57:23.222792+0000","last_fullsized":"2024-06-29T15:57:23.222792+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:23.168668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560239+0000","last_change":"2024-06-29T15:57:22.236022+0000","last_active":"2024-06-29T15:57:23.560239+0000","last_peered":"2024-06-29T15:57:23.560239+0000","last_clean":"2024-06-29T15:57:23.560239+0000","last_became_active":"2024-06-29T15:57:22.235712+0000","last_became_peered":"2024-06-29T15:57:22.235712+0000","last_unstale":"2024-06-29T15:57:23.560239+0000","last_undegraded":"2024-06-29T15:57:23.560239+0000","last_fullsized":"2024-06-29T15:57:23.560239+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:57.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559703+0000","last_change":"2024-06-29T15:57:22.218369+0000","last_active":"2024-06-29T15:57:23.559703+0000","last_peered":"2024-06-29T15:57:23.559703+0000","last_clean":"2024-06-29T15:57:23.559703+0000","last_became_active":"2024-06-29T15:57:22.217584+0000","last_became_peered":"2024-06-29T15:57:22.217584+0000","last_unstale":"2024-06-29T15:57:23.559703+0000","last_undegraded":"2024-06-29T15:57:23.559703+0000","last_fullsized":"2024-06-29T15:57:23.559703+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:59.760761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559458+0000","last_change":"2024-06-29T15:57:22.226496+0000","last_active":"2024-06-29T15:57:23.559458+0000","last_peered":"2024-06-29T15:57:23.559458+0000","last_clean":"2024-06-29T15:57:23.559458+0000","last_became_active":"2024-06-29T15:57:22.226191+0000","last_became_peered":"2024-06-29T15:57:22.226191+0000","last_unstale":"2024-06-29T15:57:23.559458+0000","last_undegraded":"2024-06-29T15:57:23.559458+0000","last_fullsized":"2024-06-29T15:57:23.559458+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:32.644449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559337+0000","last_change":"2024-06-29T15:57:22.223324+0000","last_active":"2024-06-29T15:57:23.559337+0000","last_peered":"2024-06-29T15:57:23.559337+0000","last_clean":"2024-06-29T15:57:23.559337+0000","last_became_active":"2024-06-29T15:57:22.222958+0000","last_became_peered":"2024-06-29T15:57:22.222958+0000","last_unstale":"2024-06-29T15:57:23.559337+0000","last_undegraded":"2024-06-29T15:57:23.559337+0000","last_fullsized":"2024-06-29T15:57:23.559337+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:21.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225761+0000","last_change":"2024-06-29T15:57:22.233275+0000","last_active":"2024-06-29T15:57:23.225761+0000","last_peered":"2024-06-29T15:57:23.225761+0000","last_clean":"2024-06-29T15:57:23.225761+0000","last_became_active":"2024-06-29T15:57:22.230801+0000","last_became_peered":"2024-06-29T15:57:22.230801+0000","last_unstale":"2024-06-29T15:57:23.225761+0000","last_undegraded":"2024-06-29T15:57:23.225761+0000","last_fullsized":"2024-06-29T15:57:23.225761+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:24.134069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225682+0000","last_change":"2024-06-29T15:57:22.235210+0000","last_active":"2024-06-29T15:57:23.225682+0000","last_peered":"2024-06-29T15:57:23.225682+0000","last_clean":"2024-06-29T15:57:23.225682+0000","last_became_active":"2024-06-29T15:57:22.234573+0000","last_became_peered":"2024-06-29T15:57:22.234573+0000","last_unstale":"2024-06-29T15:57:23.225682+0000","last_undegraded":"2024-06-29T15:57:23.225682+0000","last_fullsized":"2024-06-29T15:57:23.225682+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:48.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226530+0000","last_change":"2024-06-29T15:57:22.237316+0000","last_active":"2024-06-29T15:57:23.226530+0000","last_peered":"2024-06-29T15:57:23.226530+0000","last_clean":"2024-06-29T15:57:23.226530+0000","last_became_active":"2024-06-29T15:57:22.236756+0000","last_became_peered":"2024-06-29T15:57:22.236756+0000","last_unstale":"2024-06-29T15:57:23.226530+0000","last_undegraded":"2024-06-29T15:57:23.226530+0000","last_fullsized":"2024-06-29T15:57:23.226530+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.141583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222821+0000","last_change":"2024-06-29T15:57:22.237225+0000","last_active":"2024-06-29T15:57:23.222821+0000","last_peered":"2024-06-29T15:57:23.222821+0000","last_clean":"2024-06-29T15:57:23.222821+0000","last_became_active":"2024-06-29T15:57:22.236621+0000","last_became_peered":"2024-06-29T15:57:22.236621+0000","last_unstale":"2024-06-29T15:57:23.222821+0000","last_undegraded":"2024-06-29T15:57:23.222821+0000","last_fullsized":"2024-06-29T15:57:23.222821+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:26.582705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560108+0000","last_change":"2024-06-29T15:57:22.231838+0000","last_active":"2024-06-29T15:57:23.560108+0000","last_peered":"2024-06-29T15:57:23.560108+0000","last_clean":"2024-06-29T15:57:23.560108+0000","last_became_active":"2024-06-29T15:57:22.231563+0000","last_became_peered":"2024-06-29T15:57:22.231563+0000","last_unstale":"2024-06-29T15:57:23.560108+0000","last_undegraded":"2024-06-29T15:57:23.560108+0000","last_fullsized":"2024-06-29T15:57:23.560108+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:23.251139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559651+0000","last_change":"2024-06-29T15:57:22.237433+0000","last_active":"2024-06-29T15:57:23.559651+0000","last_peered":"2024-06-29T15:57:23.559651+0000","last_clean":"2024-06-29T15:57:23.559651+0000","last_became_active":"2024-06-29T15:57:22.237070+0000","last_became_peered":"2024-06-29T15:57:22.237070+0000","last_unstale":"2024-06-29T15:57:23.559651+0000","last_undegraded":"2024-06-29T15:57:23.559651+0000","last_fullsized":"2024-06-29T15:57:23.559651+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:25.658901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225868+0000","last_change":"2024-06-29T15:57:22.225596+0000","last_active":"2024-06-29T15:57:23.225868+0000","last_peered":"2024-06-29T15:57:23.225868+0000","last_clean":"2024-06-29T15:57:23.225868+0000","last_became_active":"2024-06-29T15:57:22.225154+0000","last_became_peered":"2024-06-29T15:57:22.225154+0000","last_unstale":"2024-06-29T15:57:23.225868+0000","last_undegraded":"2024-06-29T15:57:23.225868+0000","last_fullsized":"2024-06-29T15:57:23.225868+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:01.272853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560187+0000","last_change":"2024-06-29T15:57:22.220754+0000","last_active":"2024-06-29T15:57:23.560187+0000","last_peered":"2024-06-29T15:57:23.560187+0000","last_clean":"2024-06-29T15:57:23.560187+0000","last_became_active":"2024-06-29T15:57:22.220438+0000","last_became_peered":"2024-06-29T15:57:22.220438+0000","last_unstale":"2024-06-29T15:57:23.560187+0000","last_undegraded":"2024-06-29T15:57:23.560187+0000","last_fullsized":"2024-06-29T15:57:23.560187+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:26.314927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222569+0000","last_change":"2024-06-29T15:57:22.238113+0000","last_active":"2024-06-29T15:57:23.222569+0000","last_peered":"2024-06-29T15:57:23.222569+0000","last_clean":"2024-06-29T15:57:23.222569+0000","last_became_active":"2024-06-29T15:57:22.237726+0000","last_became_peered":"2024-06-29T15:57:22.237726+0000","last_unstale":"2024-06-29T15:57:23.222569+0000","last_undegraded":"2024-06-29T15:57:23.222569+0000","last_fullsized":"2024-06-29T15:57:23.222569+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:08:48.416318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225703+0000","last_change":"2024-06-29T15:57:22.235020+0000","last_active":"2024-06-29T15:57:23.225703+0000","last_peered":"2024-06-29T15:57:23.225703+0000","last_clean":"2024-06-29T15:57:23.225703+0000","last_became_active":"2024-06-29T15:57:22.234417+0000","last_became_peered":"2024-06-29T15:57:22.234417+0000","last_unstale":"2024-06-29T15:57:23.225703+0000","last_undegraded":"2024-06-29T15:57:23.225703+0000","last_fullsized":"2024-06-29T15:57:23.225703+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:06.264167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225423+0000","last_change":"2024-06-29T15:57:22.239205+0000","last_active":"2024-06-29T15:57:23.225423+0000","last_peered":"2024-06-29T15:57:23.225423+0000","last_clean":"2024-06-29T15:57:23.225423+0000","last_became_active":"2024-06-29T15:57:22.238830+0000","last_became_peered":"2024-06-29T15:57:22.238830+0000","last_unstale":"2024-06-29T15:57:23.225423+0000","last_undegraded":"2024-06-29T15:57:23.225423+0000","last_fullsized":"2024-06-29T15:57:23.225423+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.228440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226737+0000","last_change":"2024-06-29T15:57:22.237289+0000","last_active":"2024-06-29T15:57:23.226737+0000","last_peered":"2024-06-29T15:57:23.226737+0000","last_clean":"2024-06-29T15:57:23.226737+0000","last_became_active":"2024-06-29T15:57:22.236267+0000","last_became_peered":"2024-06-29T15:57:22.236267+0000","last_unstale":"2024-06-29T15:57:23.226737+0000","last_undegraded":"2024-06-29T15:57:23.226737+0000","last_fullsized":"2024-06-29T15:57:23.226737+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:53:11.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226450+0000","last_change":"2024-06-29T15:57:22.215484+0000","last_active":"2024-06-29T15:57:23.226450+0000","last_peered":"2024-06-29T15:57:23.226450+0000","last_clean":"2024-06-29T15:57:23.226450+0000","last_became_active":"2024-06-29T15:57:22.215179+0000","last_became_peered":"2024-06-29T15:57:22.215179+0000","last_unstale":"2024-06-29T15:57:23.226450+0000","last_undegraded":"2024-06-29T15:57:23.226450+0000","last_fullsized":"2024-06-29T15:57:23.226450+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:29.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226130+0000","last_change":"2024-06-29T15:57:22.224310+0000","last_active":"2024-06-29T15:57:23.226130+0000","last_peered":"2024-06-29T15:57:23.226130+0000","last_clean":"2024-06-29T15:57:23.226130+0000","last_became_active":"2024-06-29T15:57:22.223959+0000","last_became_peered":"2024-06-29T15:57:22.223959+0000","last_unstale":"2024-06-29T15:57:23.226130+0000","last_undegraded":"2024-06-29T15:57:23.226130+0000","last_fullsized":"2024-06-29T15:57:23.226130+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:13.811309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225893+0000","last_change":"2024-06-29T15:57:22.222075+0000","last_active":"2024-06-29T15:57:23.225893+0000","last_peered":"2024-06-29T15:57:23.225893+0000","last_clean":"2024-06-29T15:57:23.225893+0000","last_became_active":"2024-06-29T15:57:22.221767+0000","last_became_peered":"2024-06-29T15:57:22.221767+0000","last_unstale":"2024-06-29T15:57:23.225893+0000","last_undegraded":"2024-06-29T15:57:23.225893+0000","last_fullsized":"2024-06-29T15:57:23.225893+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:13.762887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559791+0000","last_change":"2024-06-29T15:57:22.232432+0000","last_active":"2024-06-29T15:57:23.559791+0000","last_peered":"2024-06-29T15:57:23.559791+0000","last_clean":"2024-06-29T15:57:23.559791+0000","last_became_active":"2024-06-29T15:57:22.232102+0000","last_became_peered":"2024-06-29T15:57:22.232102+0000","last_unstale":"2024-06-29T15:57:23.559791+0000","last_undegraded":"2024-06-29T15:57:23.559791+0000","last_fullsized":"2024-06-29T15:57:23.559791+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:41.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559347+0000","last_change":"2024-06-29T15:57:22.223814+0000","last_active":"2024-06-29T15:57:23.559347+0000","last_peered":"2024-06-29T15:57:23.559347+0000","last_clean":"2024-06-29T15:57:23.559347+0000","last_became_active":"2024-06-29T15:57:22.223464+0000","last_became_peered":"2024-06-29T15:57:22.223464+0000","last_unstale":"2024-06-29T15:57:23.559347+0000","last_undegraded":"2024-06-29T15:57:23.559347+0000","last_fullsized":"2024-06-29T15:57:23.559347+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:40.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225812+0000","last_change":"2024-06-29T15:57:22.234902+0000","last_active":"2024-06-29T15:57:23.225812+0000","last_peered":"2024-06-29T15:57:23.225812+0000","last_clean":"2024-06-29T15:57:23.225812+0000","last_became_active":"2024-06-29T15:57:22.233244+0000","last_became_peered":"2024-06-29T15:57:22.233244+0000","last_unstale":"2024-06-29T15:57:23.225812+0000","last_undegraded":"2024-06-29T15:57:23.225812+0000","last_fullsized":"2024-06-29T15:57:23.225812+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:58.786612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428127+0000","last_change":"2024-06-29T15:57:22.225280+0000","last_active":"2024-06-29T15:57:23.428127+0000","last_peered":"2024-06-29T15:57:23.428127+0000","last_clean":"2024-06-29T15:57:23.428127+0000","last_became_active":"2024-06-29T15:57:22.224986+0000","last_became_peered":"2024-06-29T15:57:22.224986+0000","last_unstale":"2024-06-29T15:57:23.428127+0000","last_undegraded":"2024-06-29T15:57:23.428127+0000","last_fullsized":"2024-06-29T15:57:23.428127+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:07.391769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222931+0000","last_change":"2024-06-29T15:57:22.236939+0000","last_active":"2024-06-29T15:57:23.222931+0000","last_peered":"2024-06-29T15:57:23.222931+0000","last_clean":"2024-06-29T15:57:23.222931+0000","last_became_active":"2024-06-29T15:57:22.236129+0000","last_became_peered":"2024-06-29T15:57:22.236129+0000","last_unstale":"2024-06-29T15:57:23.222931+0000","last_undegraded":"2024-06-29T15:57:23.222931+0000","last_fullsized":"2024-06-29T15:57:23.222931+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:13.274461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560090+0000","last_change":"2024-06-29T15:57:22.227845+0000","last_active":"2024-06-29T15:57:23.560090+0000","last_peered":"2024-06-29T15:57:23.560090+0000","last_clean":"2024-06-29T15:57:23.560090+0000","last_became_active":"2024-06-29T15:57:22.227549+0000","last_became_peered":"2024-06-29T15:57:22.227549+0000","last_unstale":"2024-06-29T15:57:23.560090+0000","last_undegraded":"2024-06-29T15:57:23.560090+0000","last_fullsized":"2024-06-29T15:57:23.560090+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:22.735783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225452+0000","last_change":"2024-06-29T15:57:22.227876+0000","last_active":"2024-06-29T15:57:23.225452+0000","last_peered":"2024-06-29T15:57:23.225452+0000","last_clean":"2024-06-29T15:57:23.225452+0000","last_became_active":"2024-06-29T15:57:22.227591+0000","last_became_peered":"2024-06-29T15:57:22.227591+0000","last_unstale":"2024-06-29T15:57:23.225452+0000","last_undegraded":"2024-06-29T15:57:23.225452+0000","last_fullsized":"2024-06-29T15:57:23.225452+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:45.600754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559566+0000","last_change":"2024-06-29T15:57:22.236151+0000","last_active":"2024-06-29T15:57:23.559566+0000","last_peered":"2024-06-29T15:57:23.559566+0000","last_clean":"2024-06-29T15:57:23.559566+0000","last_became_active":"2024-06-29T15:57:22.235409+0000","last_became_peered":"2024-06-29T15:57:22.235409+0000","last_unstale":"2024-06-29T15:57:23.559566+0000","last_undegraded":"2024-06-29T15:57:23.559566+0000","last_fullsized":"2024-06-29T15:57:23.559566+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:56.149438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559526+0000","last_change":"2024-06-29T15:57:22.239789+0000","last_active":"2024-06-29T15:57:23.559526+0000","last_peered":"2024-06-29T15:57:23.559526+0000","last_clean":"2024-06-29T15:57:23.559526+0000","last_became_active":"2024-06-29T15:57:22.239438+0000","last_became_peered":"2024-06-29T15:57:22.239438+0000","last_unstale":"2024-06-29T15:57:23.559526+0000","last_undegraded":"2024-06-29T15:57:23.559526+0000","last_fullsized":"2024-06-29T15:57:23.559526+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:08.408484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559240+0000","last_change":"2024-06-29T15:57:22.224277+0000","last_active":"2024-06-29T15:57:23.559240+0000","last_peered":"2024-06-29T15:57:23.559240+0000","last_clean":"2024-06-29T15:57:23.559240+0000","last_became_active":"2024-06-29T15:57:22.223975+0000","last_became_peered":"2024-06-29T15:57:22.223975+0000","last_unstale":"2024-06-29T15:57:23.559240+0000","last_undegraded":"2024-06-29T15:57:23.559240+0000","last_fullsized":"2024-06-29T15:57:23.559240+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:46.705935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559522+0000","last_change":"2024-06-29T15:57:22.228934+0000","last_active":"2024-06-29T15:57:23.559522+0000","last_peered":"2024-06-29T15:57:23.559522+0000","last_clean":"2024-06-29T15:57:23.559522+0000","last_became_active":"2024-06-29T15:57:22.228603+0000","last_became_peered":"2024-06-29T15:57:22.228603+0000","last_unstale":"2024-06-29T15:57:23.559522+0000","last_undegraded":"2024-06-29T15:57:23.559522+0000","last_fullsized":"2024-06-29T15:57:23.559522+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:28.391534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"221'4","reported_seq":18,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560047+0000","last_change":"2024-06-29T15:57:21.218674+0000","last_active":"2024-06-29T15:57:23.560047+0000","last_peered":"2024-06-29T15:57:23.560047+0000","last_clean":"2024-06-29T15:57:23.560047+0000","last_became_active":"2024-06-29T15:57:21.218397+0000","last_became_peered":"2024-06-29T15:57:21.218397+0000","last_unstale":"2024-06-29T15:57:23.560047+0000","last_undegraded":"2024-06-29T15:57:23.560047+0000","last_fullsized":"2024-06-29T15:57:23.560047+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.766844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.427746+0000","last_change":"2024-06-29T15:57:22.224449+0000","last_active":"2024-06-29T15:57:23.427746+0000","last_peered":"2024-06-29T15:57:23.427746+0000","last_clean":"2024-06-29T15:57:23.427746+0000","last_became_active":"2024-06-29T15:57:22.224124+0000","last_became_peered":"2024-06-29T15:57:22.224124+0000","last_unstale":"2024-06-29T15:57:23.427746+0000","last_undegraded":"2024-06-29T15:57:23.427746+0000","last_fullsized":"2024-06-29T15:57:23.427746+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:05.649826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428176+0000","last_change":"2024-06-29T15:57:21.213288+0000","last_active":"2024-06-29T15:57:23.428176+0000","last_peered":"2024-06-29T15:57:23.428176+0000","last_clean":"2024-06-29T15:57:23.428176+0000","last_became_active":"2024-06-29T15:57:21.212954+0000","last_became_peered":"2024-06-29T15:57:21.212954+0000","last_unstale":"2024-06-29T15:57:23.428176+0000","last_undegraded":"2024-06-29T15:57:23.428176+0000","last_fullsized":"2024-06-29T15:57:23.428176+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:24.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225996+0000","last_change":"2024-06-29T15:57:22.213237+0000","last_active":"2024-06-29T15:57:23.225996+0000","last_peered":"2024-06-29T15:57:23.225996+0000","last_clean":"2024-06-29T15:57:23.225996+0000","last_became_active":"2024-06-29T15:57:22.212908+0000","last_became_peered":"2024-06-29T15:57:22.212908+0000","last_unstale":"2024-06-29T15:57:23.225996+0000","last_undegraded":"2024-06-29T15:57:23.225996+0000","last_fullsized":"2024-06-29T15:57:23.225996+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:31.343924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560234+0000","last_change":"2024-06-29T15:57:21.210602+0000","last_active":"2024-06-29T15:57:23.560234+0000","last_peered":"2024-06-29T15:57:23.560234+0000","last_clean":"2024-06-29T15:57:23.560234+0000","last_became_active":"2024-06-29T15:57:21.210296+0000","last_became_peered":"2024-06-29T15:57:21.210296+0000","last_unstale":"2024-06-29T15:57:23.560234+0000","last_undegraded":"2024-06-29T15:57:23.560234+0000","last_fullsized":"2024-06-29T15:57:23.560234+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:07.128352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559868+0000","last_change":"2024-06-29T15:57:22.226181+0000","last_active":"2024-06-29T15:57:23.559868+0000","last_peered":"2024-06-29T15:57:23.559868+0000","last_clean":"2024-06-29T15:57:23.559868+0000","last_became_active":"2024-06-29T15:57:22.225855+0000","last_became_peered":"2024-06-29T15:57:22.225855+0000","last_unstale":"2024-06-29T15:57:23.559868+0000","last_undegraded":"2024-06-29T15:57:23.559868+0000","last_fullsized":"2024-06-29T15:57:23.559868+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:37.067580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225538+0000","last_change":"2024-06-29T15:57:21.210643+0000","last_active":"2024-06-29T15:57:23.225538+0000","last_peered":"2024-06-29T15:57:23.225538+0000","last_clean":"2024-06-29T15:57:23.225538+0000","last_became_active":"2024-06-29T15:57:21.209805+0000","last_became_peered":"2024-06-29T15:57:21.209805+0000","last_unstale":"2024-06-29T15:57:23.225538+0000","last_undegraded":"2024-06-29T15:57:23.225538+0000","last_fullsized":"2024-06-29T15:57:23.225538+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:27.618049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559673+0000","last_change":"2024-06-29T15:57:22.235721+0000","last_active":"2024-06-29T15:57:23.559673+0000","last_peered":"2024-06-29T15:57:23.559673+0000","last_clean":"2024-06-29T15:57:23.559673+0000","last_became_active":"2024-06-29T15:57:22.235380+0000","last_became_peered":"2024-06-29T15:57:22.235380+0000","last_unstale":"2024-06-29T15:57:23.559673+0000","last_undegraded":"2024-06-29T15:57:23.559673+0000","last_fullsized":"2024-06-29T15:57:23.559673+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:38.144376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559711+0000","last_change":"2024-06-29T15:57:21.211113+0000","last_active":"2024-06-29T15:57:23.559711+0000","last_peered":"2024-06-29T15:57:23.559711+0000","last_clean":"2024-06-29T15:57:23.559711+0000","last_became_active":"2024-06-29T15:57:21.210817+0000","last_became_peered":"2024-06-29T15:57:21.210817+0000","last_unstale":"2024-06-29T15:57:23.559711+0000","last_undegraded":"2024-06-29T15:57:23.559711+0000","last_fullsized":"2024-06-29T15:57:23.559711+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:26.176884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560046+0000","last_change":"2024-06-29T15:57:22.214168+0000","last_active":"2024-06-29T15:57:23.560046+0000","last_peered":"2024-06-29T15:57:23.560046+0000","last_clean":"2024-06-29T15:57:23.560046+0000","last_became_active":"2024-06-29T15:57:22.213780+0000","last_became_peered":"2024-06-29T15:57:22.213780+0000","last_unstale":"2024-06-29T15:57:23.560046+0000","last_undegraded":"2024-06-29T15:57:23.560046+0000","last_fullsized":"2024-06-29T15:57:23.560046+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:01.904725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559645+0000","last_change":"2024-06-29T15:57:21.212034+0000","last_active":"2024-06-29T15:57:23.559645+0000","last_peered":"2024-06-29T15:57:23.559645+0000","last_clean":"2024-06-29T15:57:23.559645+0000","last_became_active":"2024-06-29T15:57:21.211762+0000","last_became_peered":"2024-06-29T15:57:21.211762+0000","last_unstale":"2024-06-29T15:57:23.559645+0000","last_undegraded":"2024-06-29T15:57:23.559645+0000","last_fullsized":"2024-06-29T15:57:23.559645+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:36.775853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225820+0000","last_change":"2024-06-29T15:57:22.234743+0000","last_active":"2024-06-29T15:57:23.225820+0000","last_peered":"2024-06-29T15:57:23.225820+0000","last_clean":"2024-06-29T15:57:23.225820+0000","last_became_active":"2024-06-29T15:57:22.233091+0000","last_became_peered":"2024-06-29T15:57:22.233091+0000","last_unstale":"2024-06-29T15:57:23.225820+0000","last_undegraded":"2024-06-29T15:57:23.225820+0000","last_fullsized":"2024-06-29T15:57:23.225820+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:46.131681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222578+0000","last_change":"2024-06-29T15:57:21.208896+0000","last_active":"2024-06-29T15:57:23.222578+0000","last_peered":"2024-06-29T15:57:23.222578+0000","last_clean":"2024-06-29T15:57:23.222578+0000","last_became_active":"2024-06-29T15:57:21.208555+0000","last_became_peered":"2024-06-29T15:57:21.208555+0000","last_unstale":"2024-06-29T15:57:23.222578+0000","last_undegraded":"2024-06-29T15:57:23.222578+0000","last_fullsized":"2024-06-29T15:57:23.222578+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559436+0000","last_change":"2024-06-29T15:57:22.235312+0000","last_active":"2024-06-29T15:57:23.559436+0000","last_peered":"2024-06-29T15:57:23.559436+0000","last_clean":"2024-06-29T15:57:23.559436+0000","last_became_active":"2024-06-29T15:57:22.234828+0000","last_became_peered":"2024-06-29T15:57:22.234828+0000","last_unstale":"2024-06-29T15:57:23.559436+0000","last_undegraded":"2024-06-29T15:57:23.559436+0000","last_fullsized":"2024-06-29T15:57:23.559436+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:39.661714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559507+0000","last_change":"2024-06-29T15:57:21.218814+0000","last_active":"2024-06-29T15:57:23.559507+0000","last_peered":"2024-06-29T15:57:23.559507+0000","last_clean":"2024-06-29T15:57:23.559507+0000","last_became_active":"2024-06-29T15:57:21.218495+0000","last_became_peered":"2024-06-29T15:57:21.218495+0000","last_unstale":"2024-06-29T15:57:23.559507+0000","last_undegraded":"2024-06-29T15:57:23.559507+0000","last_fullsized":"2024-06-29T15:57:23.559507+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:04.680014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222556+0000","last_change":"2024-06-29T15:57:22.219192+0000","last_active":"2024-06-29T15:57:23.222556+0000","last_peered":"2024-06-29T15:57:23.222556+0000","last_clean":"2024-06-29T15:57:23.222556+0000","last_became_active":"2024-06-29T15:57:22.218844+0000","last_became_peered":"2024-06-29T15:57:22.218844+0000","last_unstale":"2024-06-29T15:57:23.222556+0000","last_undegraded":"2024-06-29T15:57:23.222556+0000","last_fullsized":"2024-06-29T15:57:23.222556+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:50.717283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559891+0000","last_change":"2024-06-29T15:57:21.214536+0000","last_active":"2024-06-29T15:57:23.559891+0000","last_peered":"2024-06-29T15:57:23.559891+0000","last_clean":"2024-06-29T15:57:23.559891+0000","last_became_active":"2024-06-29T15:57:21.214227+0000","last_became_peered":"2024-06-29T15:57:21.214227+0000","last_unstale":"2024-06-29T15:57:23.559891+0000","last_undegraded":"2024-06-29T15:57:23.559891+0000","last_fullsized":"2024-06-29T15:57:23.559891+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:25.839522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225514+0000","last_change":"2024-06-29T15:57:22.231011+0000","last_active":"2024-06-29T15:57:23.225514+0000","last_peered":"2024-06-29T15:57:23.225514+0000","last_clean":"2024-06-29T15:57:23.225514+0000","last_became_active":"2024-06-29T15:57:22.230675+0000","last_became_peered":"2024-06-29T15:57:22.230675+0000","last_unstale":"2024-06-29T15:57:23.225514+0000","last_undegraded":"2024-06-29T15:57:23.225514+0000","last_fullsized":"2024-06-29T15:57:23.225514+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:09.478769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"221'3","reported_seq":17,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225700+0000","last_change":"2024-06-29T15:57:21.210270+0000","last_active":"2024-06-29T15:57:23.225700+0000","last_peered":"2024-06-29T15:57:23.225700+0000","last_clean":"2024-06-29T15:57:23.225700+0000","last_became_active":"2024-06-29T15:57:21.209731+0000","last_became_peered":"2024-06-29T15:57:21.209731+0000","last_unstale":"2024-06-29T15:57:23.225700+0000","last_undegraded":"2024-06-29T15:57:23.225700+0000","last_fullsized":"2024-06-29T15:57:23.225700+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:18.182233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559925+0000","last_change":"2024-06-29T15:57:22.224600+0000","last_active":"2024-06-29T15:57:23.559925+0000","last_peered":"2024-06-29T15:57:23.559925+0000","last_clean":"2024-06-29T15:57:23.559925+0000","last_became_active":"2024-06-29T15:57:22.224260+0000","last_became_peered":"2024-06-29T15:57:22.224260+0000","last_unstale":"2024-06-29T15:57:23.559925+0000","last_undegraded":"2024-06-29T15:57:23.559925+0000","last_fullsized":"2024-06-29T15:57:23.559925+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:21.773370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"221'6","reported_seq":20,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560020+0000","last_change":"2024-06-29T15:57:21.217755+0000","last_active":"2024-06-29T15:57:23.560020+0000","last_peered":"2024-06-29T15:57:23.560020+0000","last_clean":"2024-06-29T15:57:23.560020+0000","last_became_active":"2024-06-29T15:57:21.216983+0000","last_became_peered":"2024-06-29T15:57:21.216983+0000","last_unstale":"2024-06-29T15:57:23.560020+0000","last_undegraded":"2024-06-29T15:57:23.560020+0000","last_fullsized":"2024-06-29T15:57:23.560020+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:31.889343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560016+0000","last_change":"2024-06-29T15:57:22.234802+0000","last_active":"2024-06-29T15:57:23.560016+0000","last_peered":"2024-06-29T15:57:23.560016+0000","last_clean":"2024-06-29T15:57:23.560016+0000","last_became_active":"2024-06-29T15:57:22.234492+0000","last_became_peered":"2024-06-29T15:57:22.234492+0000","last_unstale":"2024-06-29T15:57:23.560016+0000","last_undegraded":"2024-06-29T15:57:23.560016+0000","last_fullsized":"2024-06-29T15:57:23.560016+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:37.928014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559833+0000","last_change":"2024-06-29T15:57:21.215892+0000","last_active":"2024-06-29T15:57:23.559833+0000","last_peered":"2024-06-29T15:57:23.559833+0000","last_clean":"2024-06-29T15:57:23.559833+0000","last_became_active":"2024-06-29T15:57:21.215250+0000","last_became_peered":"2024-06-29T15:57:21.215250+0000","last_unstale":"2024-06-29T15:57:23.559833+0000","last_undegraded":"2024-06-29T15:57:23.559833+0000","last_fullsized":"2024-06-29T15:57:23.559833+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:23.234622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222962+0000","last_change":"2024-06-29T15:57:22.241881+0000","last_active":"2024-06-29T15:57:23.222962+0000","last_peered":"2024-06-29T15:57:23.222962+0000","last_clean":"2024-06-29T15:57:23.222962+0000","last_became_active":"2024-06-29T15:57:22.241549+0000","last_became_peered":"2024-06-29T15:57:22.241549+0000","last_unstale":"2024-06-29T15:57:23.222962+0000","last_undegraded":"2024-06-29T15:57:23.222962+0000","last_fullsized":"2024-06-29T15:57:23.222962+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:25.352328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222741+0000","last_change":"2024-06-29T15:57:21.203410+0000","last_active":"2024-06-29T15:57:23.222741+0000","last_peered":"2024-06-29T15:57:23.222741+0000","last_clean":"2024-06-29T15:57:23.222741+0000","last_became_active":"2024-06-29T15:57:21.203034+0000","last_became_peered":"2024-06-29T15:57:21.203034+0000","last_unstale":"2024-06-29T15:57:23.222741+0000","last_undegraded":"2024-06-29T15:57:23.222741+0000","last_fullsized":"2024-06-29T15:57:23.222741+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:52.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225471+0000","last_change":"2024-06-29T15:57:22.228484+0000","last_active":"2024-06-29T15:57:23.225471+0000","last_peered":"2024-06-29T15:57:23.225471+0000","last_clean":"2024-06-29T15:57:23.225471+0000","last_became_active":"2024-06-29T15:57:22.227880+0000","last_became_peered":"2024-06-29T15:57:22.227880+0000","last_unstale":"2024-06-29T15:57:23.225471+0000","last_undegraded":"2024-06-29T15:57:23.225471+0000","last_fullsized":"2024-06-29T15:57:23.225471+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:21.749071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225100+0000","last_change":"2024-06-29T15:57:21.215032+0000","last_active":"2024-06-29T15:57:23.225100+0000","last_peered":"2024-06-29T15:57:23.225100+0000","last_clean":"2024-06-29T15:57:23.225100+0000","last_became_active":"2024-06-29T15:57:21.214557+0000","last_became_peered":"2024-06-29T15:57:21.214557+0000","last_unstale":"2024-06-29T15:57:23.225100+0000","last_undegraded":"2024-06-29T15:57:23.225100+0000","last_fullsized":"2024-06-29T15:57:23.225100+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560171+0000","last_change":"2024-06-29T15:57:22.223112+0000","last_active":"2024-06-29T15:57:23.560171+0000","last_peered":"2024-06-29T15:57:23.560171+0000","last_clean":"2024-06-29T15:57:23.560171+0000","last_became_active":"2024-06-29T15:57:22.222632+0000","last_became_peered":"2024-06-29T15:57:22.222632+0000","last_unstale":"2024-06-29T15:57:23.560171+0000","last_undegraded":"2024-06-29T15:57:23.560171+0000","last_fullsized":"2024-06-29T15:57:23.560171+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:05.408776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"221'8","reported_seq":22,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559228+0000","last_change":"2024-06-29T15:57:21.216156+0000","last_active":"2024-06-29T15:57:23.559228+0000","last_peered":"2024-06-29T15:57:23.559228+0000","last_clean":"2024-06-29T15:57:23.559228+0000","last_became_active":"2024-06-29T15:57:21.214088+0000","last_became_peered":"2024-06-29T15:57:21.214088+0000","last_unstale":"2024-06-29T15:57:23.559228+0000","last_undegraded":"2024-06-29T15:57:23.559228+0000","last_fullsized":"2024-06-29T15:57:23.559228+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:30.314243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225778+0000","last_change":"2024-06-29T15:57:22.238347+0000","last_active":"2024-06-29T15:57:23.225778+0000","last_peered":"2024-06-29T15:57:23.225778+0000","last_clean":"2024-06-29T15:57:23.225778+0000","last_became_active":"2024-06-29T15:57:22.236896+0000","last_became_peered":"2024-06-29T15:57:22.236896+0000","last_unstale":"2024-06-29T15:57:23.225778+0000","last_undegraded":"2024-06-29T15:57:23.225778+0000","last_fullsized":"2024-06-29T15:57:23.225778+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:39.739194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428213+0000","last_change":"2024-06-29T15:57:21.212718+0000","last_active":"2024-06-29T15:57:23.428213+0000","last_peered":"2024-06-29T15:57:23.428213+0000","last_clean":"2024-06-29T15:57:23.428213+0000","last_became_active":"2024-06-29T15:57:21.212406+0000","last_became_peered":"2024-06-29T15:57:21.212406+0000","last_unstale":"2024-06-29T15:57:23.428213+0000","last_undegraded":"2024-06-29T15:57:23.428213+0000","last_fullsized":"2024-06-29T15:57:23.428213+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:00.180622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428033+0000","last_change":"2024-06-29T15:57:22.225793+0000","last_active":"2024-06-29T15:57:23.428033+0000","last_peered":"2024-06-29T15:57:23.428033+0000","last_clean":"2024-06-29T15:57:23.428033+0000","last_became_active":"2024-06-29T15:57:22.224806+0000","last_became_peered":"2024-06-29T15:57:22.224806+0000","last_unstale":"2024-06-29T15:57:23.428033+0000","last_undegraded":"2024-06-29T15:57:23.428033+0000","last_fullsized":"2024-06-29T15:57:23.428033+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:03.667477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428289+0000","last_change":"2024-06-29T15:57:21.216179+0000","last_active":"2024-06-29T15:57:23.428289+0000","last_peered":"2024-06-29T15:57:23.428289+0000","last_clean":"2024-06-29T15:57:23.428289+0000","last_became_active":"2024-06-29T15:57:21.215876+0000","last_became_peered":"2024-06-29T15:57:21.215876+0000","last_unstale":"2024-06-29T15:57:23.428289+0000","last_undegraded":"2024-06-29T15:57:23.428289+0000","last_fullsized":"2024-06-29T15:57:23.428289+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:37.942186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428078+0000","last_change":"2024-06-29T15:57:22.232727+0000","last_active":"2024-06-29T15:57:23.428078+0000","last_peered":"2024-06-29T15:57:23.428078+0000","last_clean":"2024-06-29T15:57:23.428078+0000","last_became_active":"2024-06-29T15:57:22.232392+0000","last_became_peered":"2024-06-29T15:57:22.232392+0000","last_unstale":"2024-06-29T15:57:23.428078+0000","last_undegraded":"2024-06-29T15:57:23.428078+0000","last_fullsized":"2024-06-29T15:57:23.428078+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:54.805984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560256+0000","last_change":"2024-06-29T15:57:21.210528+0000","last_active":"2024-06-29T15:57:23.560256+0000","last_peered":"2024-06-29T15:57:23.560256+0000","last_clean":"2024-06-29T15:57:23.560256+0000","last_became_active":"2024-06-29T15:57:21.208691+0000","last_became_peered":"2024-06-29T15:57:21.208691+0000","last_unstale":"2024-06-29T15:57:23.560256+0000","last_undegraded":"2024-06-29T15:57:23.560256+0000","last_fullsized":"2024-06-29T15:57:23.560256+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.762590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226207+0000","last_change":"2024-06-29T15:57:22.217630+0000","last_active":"2024-06-29T15:57:23.226207+0000","last_peered":"2024-06-29T15:57:23.226207+0000","last_clean":"2024-06-29T15:57:23.226207+0000","last_became_active":"2024-06-29T15:57:22.217303+0000","last_became_peered":"2024-06-29T15:57:22.217303+0000","last_unstale":"2024-06-29T15:57:23.226207+0000","last_undegraded":"2024-06-29T15:57:23.226207+0000","last_fullsized":"2024-06-29T15:57:23.226207+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:05.554814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560152+0000","last_change":"2024-06-29T15:57:21.213193+0000","last_active":"2024-06-29T15:57:23.560152+0000","last_peered":"2024-06-29T15:57:23.560152+0000","last_clean":"2024-06-29T15:57:23.560152+0000","last_became_active":"2024-06-29T15:57:21.212838+0000","last_became_peered":"2024-06-29T15:57:21.212838+0000","last_unstale":"2024-06-29T15:57:23.560152+0000","last_undegraded":"2024-06-29T15:57:23.560152+0000","last_fullsized":"2024-06-29T15:57:23.560152+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:54.574391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560023+0000","last_change":"2024-06-29T15:57:22.231883+0000","last_active":"2024-06-29T15:57:23.560023+0000","last_peered":"2024-06-29T15:57:23.560023+0000","last_clean":"2024-06-29T15:57:23.560023+0000","last_became_active":"2024-06-29T15:57:22.231588+0000","last_became_peered":"2024-06-29T15:57:22.231588+0000","last_unstale":"2024-06-29T15:57:23.560023+0000","last_undegraded":"2024-06-29T15:57:23.560023+0000","last_fullsized":"2024-06-29T15:57:23.560023+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:17.705760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559988+0000","last_change":"2024-06-29T15:57:21.215358+0000","last_active":"2024-06-29T15:57:23.559988+0000","last_peered":"2024-06-29T15:57:23.559988+0000","last_clean":"2024-06-29T15:57:23.559988+0000","last_became_active":"2024-06-29T15:57:21.213979+0000","last_became_peered":"2024-06-29T15:57:21.213979+0000","last_unstale":"2024-06-29T15:57:23.559988+0000","last_undegraded":"2024-06-29T15:57:23.559988+0000","last_fullsized":"2024-06-29T15:57:23.559988+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:42.446135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559969+0000","last_change":"2024-06-29T15:57:22.238829+0000","last_active":"2024-06-29T15:57:23.559969+0000","last_peered":"2024-06-29T15:57:23.559969+0000","last_clean":"2024-06-29T15:57:23.559969+0000","last_became_active":"2024-06-29T15:57:22.238520+0000","last_became_peered":"2024-06-29T15:57:22.238520+0000","last_unstale":"2024-06-29T15:57:23.559969+0000","last_undegraded":"2024-06-29T15:57:23.559969+0000","last_fullsized":"2024-06-29T15:57:23.559969+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:22.127384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559664+0000","last_change":"2024-06-29T15:57:21.208592+0000","last_active":"2024-06-29T15:57:23.559664+0000","last_peered":"2024-06-29T15:57:23.559664+0000","last_clean":"2024-06-29T15:57:23.559664+0000","last_became_active":"2024-06-29T15:57:21.208156+0000","last_became_peered":"2024-06-29T15:57:21.208156+0000","last_unstale":"2024-06-29T15:57:23.559664+0000","last_undegraded":"2024-06-29T15:57:23.559664+0000","last_fullsized":"2024-06-29T15:57:23.559664+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:20.325402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225614+0000","last_change":"2024-06-29T15:57:22.228211+0000","last_active":"2024-06-29T15:57:23.225614+0000","last_peered":"2024-06-29T15:57:23.225614+0000","last_clean":"2024-06-29T15:57:23.225614+0000","last_became_active":"2024-06-29T15:57:22.227519+0000","last_became_peered":"2024-06-29T15:57:22.227519+0000","last_unstale":"2024-06-29T15:57:23.225614+0000","last_undegraded":"2024-06-29T15:57:23.225614+0000","last_fullsized":"2024-06-29T15:57:23.225614+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:16:35.999038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560306+0000","last_change":"2024-06-29T15:57:21.212085+0000","last_active":"2024-06-29T15:57:23.560306+0000","last_peered":"2024-06-29T15:57:23.560306+0000","last_clean":"2024-06-29T15:57:23.560306+0000","last_became_active":"2024-06-29T15:57:21.211731+0000","last_became_peered":"2024-06-29T15:57:21.211731+0000","last_unstale":"2024-06-29T15:57:23.560306+0000","last_undegraded":"2024-06-29T15:57:23.560306+0000","last_fullsized":"2024-06-29T15:57:23.560306+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:36.061054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226814+0000","last_change":"2024-06-29T15:57:22.213920+0000","last_active":"2024-06-29T15:57:23.226814+0000","last_peered":"2024-06-29T15:57:23.226814+0000","last_clean":"2024-06-29T15:57:23.226814+0000","last_became_active":"2024-06-29T15:57:22.213583+0000","last_became_peered":"2024-06-29T15:57:22.213583+0000","last_unstale":"2024-06-29T15:57:23.226814+0000","last_undegraded":"2024-06-29T15:57:23.226814+0000","last_fullsized":"2024-06-29T15:57:23.226814+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:31.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.427663+0000","last_change":"2024-06-29T15:57:21.211203+0000","last_active":"2024-06-29T15:57:23.427663+0000","last_peered":"2024-06-29T15:57:23.427663+0000","last_clean":"2024-06-29T15:57:23.427663+0000","last_became_active":"2024-06-29T15:57:21.210898+0000","last_became_peered":"2024-06-29T15:57:21.210898+0000","last_unstale":"2024-06-29T15:57:23.427663+0000","last_undegraded":"2024-06-29T15:57:23.427663+0000","last_fullsized":"2024-06-29T15:57:23.427663+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:17.270875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225961+0000","last_change":"2024-06-29T15:57:22.224294+0000","last_active":"2024-06-29T15:57:23.225961+0000","last_peered":"2024-06-29T15:57:23.225961+0000","last_clean":"2024-06-29T15:57:23.225961+0000","last_became_active":"2024-06-29T15:57:22.223922+0000","last_became_peered":"2024-06-29T15:57:22.223922+0000","last_unstale":"2024-06-29T15:57:23.225961+0000","last_undegraded":"2024-06-29T15:57:23.225961+0000","last_fullsized":"2024-06-29T15:57:23.225961+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:46.725456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225876+0000","last_change":"2024-06-29T15:57:21.212745+0000","last_active":"2024-06-29T15:57:23.225876+0000","last_peered":"2024-06-29T15:57:23.225876+0000","last_clean":"2024-06-29T15:57:23.225876+0000","last_became_active":"2024-06-29T15:57:21.212335+0000","last_became_peered":"2024-06-29T15:57:21.212335+0000","last_unstale":"2024-06-29T15:57:23.225876+0000","last_undegraded":"2024-06-29T15:57:23.225876+0000","last_fullsized":"2024-06-29T15:57:23.225876+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:07.049637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222567+0000","last_change":"2024-06-29T15:57:22.223140+0000","last_active":"2024-06-29T15:57:23.222567+0000","last_peered":"2024-06-29T15:57:23.222567+0000","last_clean":"2024-06-29T15:57:23.222567+0000","last_became_active":"2024-06-29T15:57:22.222839+0000","last_became_peered":"2024-06-29T15:57:22.222839+0000","last_unstale":"2024-06-29T15:57:23.222567+0000","last_undegraded":"2024-06-29T15:57:23.222567+0000","last_fullsized":"2024-06-29T15:57:23.222567+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:17.520475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226569+0000","last_change":"2024-06-29T15:57:21.203822+0000","last_active":"2024-06-29T15:57:23.226569+0000","last_peered":"2024-06-29T15:57:23.226569+0000","last_clean":"2024-06-29T15:57:23.226569+0000","last_became_active":"2024-06-29T15:57:21.203517+0000","last_became_peered":"2024-06-29T15:57:21.203517+0000","last_unstale":"2024-06-29T15:57:23.226569+0000","last_undegraded":"2024-06-29T15:57:23.226569+0000","last_fullsized":"2024-06-29T15:57:23.226569+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:05.810997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225198+0000","last_change":"2024-06-29T15:57:22.233919+0000","last_active":"2024-06-29T15:57:23.225198+0000","last_peered":"2024-06-29T15:57:23.225198+0000","last_clean":"2024-06-29T15:57:23.225198+0000","last_became_active":"2024-06-29T15:57:22.231028+0000","last_became_peered":"2024-06-29T15:57:22.231028+0000","last_unstale":"2024-06-29T15:57:23.225198+0000","last_undegraded":"2024-06-29T15:57:23.225198+0000","last_fullsized":"2024-06-29T15:57:23.225198+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:34:24.910914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559644+0000","last_change":"2024-06-29T15:57:21.217039+0000","last_active":"2024-06-29T15:57:23.559644+0000","last_peered":"2024-06-29T15:57:23.559644+0000","last_clean":"2024-06-29T15:57:23.559644+0000","last_became_active":"2024-06-29T15:57:21.216542+0000","last_became_peered":"2024-06-29T15:57:21.216542+0000","last_unstale":"2024-06-29T15:57:23.559644+0000","last_undegraded":"2024-06-29T15:57:23.559644+0000","last_fullsized":"2024-06-29T15:57:23.559644+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:21.729257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428068+0000","last_change":"2024-06-29T15:57:22.227969+0000","last_active":"2024-06-29T15:57:23.428068+0000","last_peered":"2024-06-29T15:57:23.428068+0000","last_clean":"2024-06-29T15:57:23.428068+0000","last_became_active":"2024-06-29T15:57:22.227579+0000","last_became_peered":"2024-06-29T15:57:22.227579+0000","last_unstale":"2024-06-29T15:57:23.428068+0000","last_undegraded":"2024-06-29T15:57:23.428068+0000","last_fullsized":"2024-06-29T15:57:23.428068+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.427896+0000","last_change":"2024-06-29T15:57:21.216399+0000","last_active":"2024-06-29T15:57:23.427896+0000","last_peered":"2024-06-29T15:57:23.427896+0000","last_clean":"2024-06-29T15:57:23.427896+0000","last_became_active":"2024-06-29T15:57:21.216104+0000","last_became_peered":"2024-06-29T15:57:21.216104+0000","last_unstale":"2024-06-29T15:57:23.427896+0000","last_undegraded":"2024-06-29T15:57:23.427896+0000","last_fullsized":"2024-06-29T15:57:23.427896+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226376+0000","last_change":"2024-06-29T15:57:22.229230+0000","last_active":"2024-06-29T15:57:23.226376+0000","last_peered":"2024-06-29T15:57:23.226376+0000","last_clean":"2024-06-29T15:57:23.226376+0000","last_became_active":"2024-06-29T15:57:22.228914+0000","last_became_peered":"2024-06-29T15:57:22.228914+0000","last_unstale":"2024-06-29T15:57:23.226376+0000","last_undegraded":"2024-06-29T15:57:23.226376+0000","last_fullsized":"2024-06-29T15:57:23.226376+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:09.918042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226274+0000","last_change":"2024-06-29T15:57:21.213113+0000","last_active":"2024-06-29T15:57:23.226274+0000","last_peered":"2024-06-29T15:57:23.226274+0000","last_clean":"2024-06-29T15:57:23.226274+0000","last_became_active":"2024-06-29T15:57:21.212819+0000","last_became_peered":"2024-06-29T15:57:21.212819+0000","last_unstale":"2024-06-29T15:57:23.226274+0000","last_undegraded":"2024-06-29T15:57:23.226274+0000","last_fullsized":"2024-06-29T15:57:23.226274+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:59.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559925+0000","last_change":"2024-06-29T15:57:22.244396+0000","last_active":"2024-06-29T15:57:23.559925+0000","last_peered":"2024-06-29T15:57:23.559925+0000","last_clean":"2024-06-29T15:57:23.559925+0000","last_became_active":"2024-06-29T15:57:22.244044+0000","last_became_peered":"2024-06-29T15:57:22.244044+0000","last_unstale":"2024-06-29T15:57:23.559925+0000","last_undegraded":"2024-06-29T15:57:23.559925+0000","last_fullsized":"2024-06-29T15:57:23.559925+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.981276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225484+0000","last_change":"2024-06-29T15:57:21.210563+0000","last_active":"2024-06-29T15:57:23.225484+0000","last_peered":"2024-06-29T15:57:23.225484+0000","last_clean":"2024-06-29T15:57:23.225484+0000","last_became_active":"2024-06-29T15:57:21.210243+0000","last_became_peered":"2024-06-29T15:57:21.210243+0000","last_unstale":"2024-06-29T15:57:23.225484+0000","last_undegraded":"2024-06-29T15:57:23.225484+0000","last_fullsized":"2024-06-29T15:57:23.225484+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.900240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559946+0000","last_change":"2024-06-29T15:57:22.237722+0000","last_active":"2024-06-29T15:57:23.559946+0000","last_peered":"2024-06-29T15:57:23.559946+0000","last_clean":"2024-06-29T15:57:23.559946+0000","last_became_active":"2024-06-29T15:57:22.237372+0000","last_became_peered":"2024-06-29T15:57:22.237372+0000","last_unstale":"2024-06-29T15:57:23.559946+0000","last_undegraded":"2024-06-29T15:57:23.559946+0000","last_fullsized":"2024-06-29T15:57:23.559946+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:24.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428173+0000","last_change":"2024-06-29T15:57:21.212837+0000","last_active":"2024-06-29T15:57:23.428173+0000","last_peered":"2024-06-29T15:57:23.428173+0000","last_clean":"2024-06-29T15:57:23.428173+0000","last_became_active":"2024-06-29T15:57:21.212491+0000","last_became_peered":"2024-06-29T15:57:21.212491+0000","last_unstale":"2024-06-29T15:57:23.428173+0000","last_undegraded":"2024-06-29T15:57:23.428173+0000","last_fullsized":"2024-06-29T15:57:23.428173+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:35:53.641775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222886+0000","last_change":"2024-06-29T15:57:22.223950+0000","last_active":"2024-06-29T15:57:23.222886+0000","last_peered":"2024-06-29T15:57:23.222886+0000","last_clean":"2024-06-29T15:57:23.222886+0000","last_became_active":"2024-06-29T15:57:22.223622+0000","last_became_peered":"2024-06-29T15:57:22.223622+0000","last_unstale":"2024-06-29T15:57:23.222886+0000","last_undegraded":"2024-06-29T15:57:23.222886+0000","last_fullsized":"2024-06-29T15:57:23.222886+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.556171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559353+0000","last_change":"2024-06-29T15:57:21.215589+0000","last_active":"2024-06-29T15:57:23.559353+0000","last_peered":"2024-06-29T15:57:23.559353+0000","last_clean":"2024-06-29T15:57:23.559353+0000","last_became_active":"2024-06-29T15:57:21.213719+0000","last_became_peered":"2024-06-29T15:57:21.213719+0000","last_unstale":"2024-06-29T15:57:23.559353+0000","last_undegraded":"2024-06-29T15:57:23.559353+0000","last_fullsized":"2024-06-29T15:57:23.559353+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:32.413502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428127+0000","last_change":"2024-06-29T15:57:22.230794+0000","last_active":"2024-06-29T15:57:23.428127+0000","last_peered":"2024-06-29T15:57:23.428127+0000","last_clean":"2024-06-29T15:57:23.428127+0000","last_became_active":"2024-06-29T15:57:22.230474+0000","last_became_peered":"2024-06-29T15:57:22.230474+0000","last_unstale":"2024-06-29T15:57:23.428127+0000","last_undegraded":"2024-06-29T15:57:23.428127+0000","last_fullsized":"2024-06-29T15:57:23.428127+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:43.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225354+0000","last_change":"2024-06-29T15:57:21.209163+0000","last_active":"2024-06-29T15:57:23.225354+0000","last_peered":"2024-06-29T15:57:23.225354+0000","last_clean":"2024-06-29T15:57:23.225354+0000","last_became_active":"2024-06-29T15:57:21.208825+0000","last_became_peered":"2024-06-29T15:57:21.208825+0000","last_unstale":"2024-06-29T15:57:23.225354+0000","last_undegraded":"2024-06-29T15:57:23.225354+0000","last_fullsized":"2024-06-29T15:57:23.225354+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:13.641268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"210'233","reported_seq":1101,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559198+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:57:23.559198+0000","last_peered":"2024-06-29T15:57:23.559198+0000","last_clean":"2024-06-29T15:57:23.559198+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:57:23.559198+0000","last_undegraded":"2024-06-29T15:57:23.559198+0000","last_fullsized":"2024-06-29T15:57:23.559198+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543287,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52532,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319308,"statfs":{"total":96636764160,"available":96582971392,"internally_reserved":0,"allocated":3379200,"data_stored":2015142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543288,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42236,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329604,"statfs":{"total":96636764160,"available":96593514496,"internally_reserved":0,"allocated":2797568,"data_stored":1432034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543287,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2789376,"data_stored":1429896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543286,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51948,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319892,"statfs":{"total":96636764160,"available":96583569408,"internally_reserved":0,"allocated":2785280,"data_stored":1430422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510584,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47784,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324056,"statfs":{"total":96636764160,"available":96587833344,"internally_reserved":0,"allocated":2781184,"data_stored":1423218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575992,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47776,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324064,"statfs":{"total":96636764160,"available":96587841536,"internally_reserved":0,"allocated":2772992,"data_stored":1423182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510583,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47776,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324064,"statfs":{"total":96636764160,"available":96587841536,"internally_reserved":0,"allocated":2768896,"data_stored":1423092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510585,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42800,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329040,"statfs":{"total":96636764160,"available":96592936960,"internally_reserved":0,"allocated":3379200,"data_stored":2020298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:26.317 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.469+0000 7fc865e17640 1 -- 172.21.15.59:0/3955123501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 msgr2=0x7fc8600b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.469+0000 7fc865e17640 1 --2- 172.21.15.59:0/3955123501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 0x7fc8600b1b00 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fc850005fd0 tx=0x7fc85002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.469+0000 7fc865e17640 1 -- 172.21.15.59:0/3955123501 shutdown_connections 2024-06-29T15:57:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.469+0000 7fc865e17640 1 --2- 172.21.15.59:0/3955123501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc86006ea10 0x7fc8600cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.469+0000 7fc865e17640 1 --2- 172.21.15.59:0/3955123501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 0x7fc8600b1b00 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.469+0000 7fc865e17640 1 -- 172.21.15.59:0/3955123501 >> 172.21.15.59:0/3955123501 conn(0x7fc8600b06e0 msgr2=0x7fc8600b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.469+0000 7fc865e17640 1 -- 172.21.15.59:0/3955123501 shutdown_connections 2024-06-29T15:57:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.469+0000 7fc865e17640 1 -- 172.21.15.59:0/3955123501 wait complete. 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.471+0000 7fc865e17640 1 Processor -- start 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.471+0000 7fc865e17640 1 -- start start 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.472+0000 7fc865e17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc86006ea10 0x7fc8600ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.472+0000 7fc865e17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2040 0x7fc8601f5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.472+0000 7fc865e17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 0x7fc860140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.472+0000 7fc865e17640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600cfaa0 con 0x7fc860144470 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.472+0000 7fc865e17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600cf920 con 0x7fc86006ea10 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.472+0000 7fc865e17640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc8600cfc20 con 0x7fc8600b2040 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 0x7fc860140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 0x7fc860140860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46018/0 (socket says 172.21.15.59:46018) 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 -- 172.21.15.59:0/2242629820 learned_addr learned my addr 172.21.15.59:0/2242629820 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85effd640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2040 0x7fc8601f5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85f7fe640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc86006ea10 0x7fc8600ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 -- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2040 msgr2=0x7fc8601f5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2040 0x7fc8601f5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 -- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc86006ea10 msgr2=0x7fc8600ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc86006ea10 0x7fc8600ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 -- 172.21.15.59:0/2242629820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8601f5bb0 con 0x7fc860144470 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc85ffff640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 0x7fc860140860 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fc85800e9c0 tx=0x7fc85800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc847fff640 1 -- 172.21.15.59:0/2242629820 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858004d20 con 0x7fc860144470 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc847fff640 1 -- 172.21.15.59:0/2242629820 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc85800cb20 con 0x7fc860144470 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8600af9f0 con 0x7fc860144470 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc847fff640 1 -- 172.21.15.59:0/2242629820 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858005020 con 0x7fc860144470 2024-06-29T15:57:26.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.473+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8600add10 con 0x7fc860144470 2024-06-29T15:57:26.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.475+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8600d0140 con 0x7fc860144470 2024-06-29T15:57:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.475+0000 7fc847fff640 1 -- 172.21.15.59:0/2242629820 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fc8580051c0 con 0x7fc860144470 2024-06-29T15:57:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.476+0000 7fc847fff640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc84c0763f0 0x7fc84c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.476+0000 7fc847fff640 1 -- 172.21.15.59:0/2242629820 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7fc858098490 con 0x7fc860144470 2024-06-29T15:57:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.476+0000 7fc85f7fe640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc84c0763f0 0x7fc84c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.477+0000 7fc85f7fe640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc84c0763f0 0x7fc84c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc850033000 tx=0x7fc850004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.481+0000 7fc847fff640 1 -- 172.21.15.59:0/2242629820 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc8601f5bb0 con 0x7fc860144470 2024-06-29T15:57:26.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.669+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc860050500 con 0x7fc84c0763f0 2024-06-29T15:57:26.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.673+0000 7fc847fff640 1 -- 172.21.15.59:0/2242629820 <== mgr.9206 v2:172.21.15.153:6832/2780049427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252498 (secure 0 0 0) 0x7fc860050500 con 0x7fc84c0763f0 2024-06-29T15:57:26.676 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:26.678 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:57:26.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc84c0763f0 msgr2=0x7fc84c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc84c0763f0 0x7fc84c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc850033000 tx=0x7fc850004730 comp rx=0 tx=0).stop 2024-06-29T15:57:26.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 msgr2=0x7fc860140860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 0x7fc860140860 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fc85800e9c0 tx=0x7fc85800ee80 comp rx=0 tx=0).stop 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 shutdown_connections 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc84c0763f0 0x7fc84c078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc860144470 0x7fc860140860 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600b2040 0x7fc8601f5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 --2- 172.21.15.59:0/2242629820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc86006ea10 0x7fc8600ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 >> 172.21.15.59:0/2242629820 conn(0x7fc8600b06e0 msgr2=0x7fc86015d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 shutdown_connections 2024-06-29T15:57:26.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.676+0000 7fc865e17640 1 -- 172.21.15.59:0/2242629820 wait complete. 2024-06-29T15:57:26.688 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-06-29T15:57:25.765922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":421,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380648,"kb_used_data":22904,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754594072,"statfs":{"total":773094113280,"available":772704329728,"internally_reserved":0,"allocated":23453696,"data_stored":12597284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":366202418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.578148"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428245+0000","last_change":"2024-06-29T15:57:22.225592+0000","last_active":"2024-06-29T15:57:23.428245+0000","last_peered":"2024-06-29T15:57:23.428245+0000","last_clean":"2024-06-29T15:57:23.428245+0000","last_became_active":"2024-06-29T15:57:22.225309+0000","last_became_peered":"2024-06-29T15:57:22.225309+0000","last_unstale":"2024-06-29T15:57:23.428245+0000","last_undegraded":"2024-06-29T15:57:23.428245+0000","last_fullsized":"2024-06-29T15:57:23.428245+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:07.857349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559935+0000","last_change":"2024-06-29T15:57:22.231733+0000","last_active":"2024-06-29T15:57:23.559935+0000","last_peered":"2024-06-29T15:57:23.559935+0000","last_clean":"2024-06-29T15:57:23.559935+0000","last_became_active":"2024-06-29T15:57:22.231387+0000","last_became_peered":"2024-06-29T15:57:22.231387+0000","last_unstale":"2024-06-29T15:57:23.559935+0000","last_undegraded":"2024-06-29T15:57:23.559935+0000","last_fullsized":"2024-06-29T15:57:23.559935+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:41.892974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222792+0000","last_change":"2024-06-29T15:57:22.231073+0000","last_active":"2024-06-29T15:57:23.222792+0000","last_peered":"2024-06-29T15:57:23.222792+0000","last_clean":"2024-06-29T15:57:23.222792+0000","last_became_active":"2024-06-29T15:57:22.230749+0000","last_became_peered":"2024-06-29T15:57:22.230749+0000","last_unstale":"2024-06-29T15:57:23.222792+0000","last_undegraded":"2024-06-29T15:57:23.222792+0000","last_fullsized":"2024-06-29T15:57:23.222792+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:23.168668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560239+0000","last_change":"2024-06-29T15:57:22.236022+0000","last_active":"2024-06-29T15:57:23.560239+0000","last_peered":"2024-06-29T15:57:23.560239+0000","last_clean":"2024-06-29T15:57:23.560239+0000","last_became_active":"2024-06-29T15:57:22.235712+0000","last_became_peered":"2024-06-29T15:57:22.235712+0000","last_unstale":"2024-06-29T15:57:23.560239+0000","last_undegraded":"2024-06-29T15:57:23.560239+0000","last_fullsized":"2024-06-29T15:57:23.560239+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:57.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559703+0000","last_change":"2024-06-29T15:57:22.218369+0000","last_active":"2024-06-29T15:57:23.559703+0000","last_peered":"2024-06-29T15:57:23.559703+0000","last_clean":"2024-06-29T15:57:23.559703+0000","last_became_active":"2024-06-29T15:57:22.217584+0000","last_became_peered":"2024-06-29T15:57:22.217584+0000","last_unstale":"2024-06-29T15:57:23.559703+0000","last_undegraded":"2024-06-29T15:57:23.559703+0000","last_fullsized":"2024-06-29T15:57:23.559703+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:59.760761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559458+0000","last_change":"2024-06-29T15:57:22.226496+0000","last_active":"2024-06-29T15:57:23.559458+0000","last_peered":"2024-06-29T15:57:23.559458+0000","last_clean":"2024-06-29T15:57:23.559458+0000","last_became_active":"2024-06-29T15:57:22.226191+0000","last_became_peered":"2024-06-29T15:57:22.226191+0000","last_unstale":"2024-06-29T15:57:23.559458+0000","last_undegraded":"2024-06-29T15:57:23.559458+0000","last_fullsized":"2024-06-29T15:57:23.559458+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:32.644449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559337+0000","last_change":"2024-06-29T15:57:22.223324+0000","last_active":"2024-06-29T15:57:23.559337+0000","last_peered":"2024-06-29T15:57:23.559337+0000","last_clean":"2024-06-29T15:57:23.559337+0000","last_became_active":"2024-06-29T15:57:22.222958+0000","last_became_peered":"2024-06-29T15:57:22.222958+0000","last_unstale":"2024-06-29T15:57:23.559337+0000","last_undegraded":"2024-06-29T15:57:23.559337+0000","last_fullsized":"2024-06-29T15:57:23.559337+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:21.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225761+0000","last_change":"2024-06-29T15:57:22.233275+0000","last_active":"2024-06-29T15:57:23.225761+0000","last_peered":"2024-06-29T15:57:23.225761+0000","last_clean":"2024-06-29T15:57:23.225761+0000","last_became_active":"2024-06-29T15:57:22.230801+0000","last_became_peered":"2024-06-29T15:57:22.230801+0000","last_unstale":"2024-06-29T15:57:23.225761+0000","last_undegraded":"2024-06-29T15:57:23.225761+0000","last_fullsized":"2024-06-29T15:57:23.225761+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:24.134069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225682+0000","last_change":"2024-06-29T15:57:22.235210+0000","last_active":"2024-06-29T15:57:23.225682+0000","last_peered":"2024-06-29T15:57:23.225682+0000","last_clean":"2024-06-29T15:57:23.225682+0000","last_became_active":"2024-06-29T15:57:22.234573+0000","last_became_peered":"2024-06-29T15:57:22.234573+0000","last_unstale":"2024-06-29T15:57:23.225682+0000","last_undegraded":"2024-06-29T15:57:23.225682+0000","last_fullsized":"2024-06-29T15:57:23.225682+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:48.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226530+0000","last_change":"2024-06-29T15:57:22.237316+0000","last_active":"2024-06-29T15:57:23.226530+0000","last_peered":"2024-06-29T15:57:23.226530+0000","last_clean":"2024-06-29T15:57:23.226530+0000","last_became_active":"2024-06-29T15:57:22.236756+0000","last_became_peered":"2024-06-29T15:57:22.236756+0000","last_unstale":"2024-06-29T15:57:23.226530+0000","last_undegraded":"2024-06-29T15:57:23.226530+0000","last_fullsized":"2024-06-29T15:57:23.226530+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.141583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222821+0000","last_change":"2024-06-29T15:57:22.237225+0000","last_active":"2024-06-29T15:57:23.222821+0000","last_peered":"2024-06-29T15:57:23.222821+0000","last_clean":"2024-06-29T15:57:23.222821+0000","last_became_active":"2024-06-29T15:57:22.236621+0000","last_became_peered":"2024-06-29T15:57:22.236621+0000","last_unstale":"2024-06-29T15:57:23.222821+0000","last_undegraded":"2024-06-29T15:57:23.222821+0000","last_fullsized":"2024-06-29T15:57:23.222821+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:26.582705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560108+0000","last_change":"2024-06-29T15:57:22.231838+0000","last_active":"2024-06-29T15:57:23.560108+0000","last_peered":"2024-06-29T15:57:23.560108+0000","last_clean":"2024-06-29T15:57:23.560108+0000","last_became_active":"2024-06-29T15:57:22.231563+0000","last_became_peered":"2024-06-29T15:57:22.231563+0000","last_unstale":"2024-06-29T15:57:23.560108+0000","last_undegraded":"2024-06-29T15:57:23.560108+0000","last_fullsized":"2024-06-29T15:57:23.560108+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:23.251139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559651+0000","last_change":"2024-06-29T15:57:22.237433+0000","last_active":"2024-06-29T15:57:23.559651+0000","last_peered":"2024-06-29T15:57:23.559651+0000","last_clean":"2024-06-29T15:57:23.559651+0000","last_became_active":"2024-06-29T15:57:22.237070+0000","last_became_peered":"2024-06-29T15:57:22.237070+0000","last_unstale":"2024-06-29T15:57:23.559651+0000","last_undegraded":"2024-06-29T15:57:23.559651+0000","last_fullsized":"2024-06-29T15:57:23.559651+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:25.658901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225868+0000","last_change":"2024-06-29T15:57:22.225596+0000","last_active":"2024-06-29T15:57:23.225868+0000","last_peered":"2024-06-29T15:57:23.225868+0000","last_clean":"2024-06-29T15:57:23.225868+0000","last_became_active":"2024-06-29T15:57:22.225154+0000","last_became_peered":"2024-06-29T15:57:22.225154+0000","last_unstale":"2024-06-29T15:57:23.225868+0000","last_undegraded":"2024-06-29T15:57:23.225868+0000","last_fullsized":"2024-06-29T15:57:23.225868+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:01.272853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560187+0000","last_change":"2024-06-29T15:57:22.220754+0000","last_active":"2024-06-29T15:57:23.560187+0000","last_peered":"2024-06-29T15:57:23.560187+0000","last_clean":"2024-06-29T15:57:23.560187+0000","last_became_active":"2024-06-29T15:57:22.220438+0000","last_became_peered":"2024-06-29T15:57:22.220438+0000","last_unstale":"2024-06-29T15:57:23.560187+0000","last_undegraded":"2024-06-29T15:57:23.560187+0000","last_fullsized":"2024-06-29T15:57:23.560187+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:26.314927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222569+0000","last_change":"2024-06-29T15:57:22.238113+0000","last_active":"2024-06-29T15:57:23.222569+0000","last_peered":"2024-06-29T15:57:23.222569+0000","last_clean":"2024-06-29T15:57:23.222569+0000","last_became_active":"2024-06-29T15:57:22.237726+0000","last_became_peered":"2024-06-29T15:57:22.237726+0000","last_unstale":"2024-06-29T15:57:23.222569+0000","last_undegraded":"2024-06-29T15:57:23.222569+0000","last_fullsized":"2024-06-29T15:57:23.222569+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:08:48.416318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225703+0000","last_change":"2024-06-29T15:57:22.235020+0000","last_active":"2024-06-29T15:57:23.225703+0000","last_peered":"2024-06-29T15:57:23.225703+0000","last_clean":"2024-06-29T15:57:23.225703+0000","last_became_active":"2024-06-29T15:57:22.234417+0000","last_became_peered":"2024-06-29T15:57:22.234417+0000","last_unstale":"2024-06-29T15:57:23.225703+0000","last_undegraded":"2024-06-29T15:57:23.225703+0000","last_fullsized":"2024-06-29T15:57:23.225703+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:06.264167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225423+0000","last_change":"2024-06-29T15:57:22.239205+0000","last_active":"2024-06-29T15:57:23.225423+0000","last_peered":"2024-06-29T15:57:23.225423+0000","last_clean":"2024-06-29T15:57:23.225423+0000","last_became_active":"2024-06-29T15:57:22.238830+0000","last_became_peered":"2024-06-29T15:57:22.238830+0000","last_unstale":"2024-06-29T15:57:23.225423+0000","last_undegraded":"2024-06-29T15:57:23.225423+0000","last_fullsized":"2024-06-29T15:57:23.225423+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.228440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226737+0000","last_change":"2024-06-29T15:57:22.237289+0000","last_active":"2024-06-29T15:57:23.226737+0000","last_peered":"2024-06-29T15:57:23.226737+0000","last_clean":"2024-06-29T15:57:23.226737+0000","last_became_active":"2024-06-29T15:57:22.236267+0000","last_became_peered":"2024-06-29T15:57:22.236267+0000","last_unstale":"2024-06-29T15:57:23.226737+0000","last_undegraded":"2024-06-29T15:57:23.226737+0000","last_fullsized":"2024-06-29T15:57:23.226737+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:53:11.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226450+0000","last_change":"2024-06-29T15:57:22.215484+0000","last_active":"2024-06-29T15:57:23.226450+0000","last_peered":"2024-06-29T15:57:23.226450+0000","last_clean":"2024-06-29T15:57:23.226450+0000","last_became_active":"2024-06-29T15:57:22.215179+0000","last_became_peered":"2024-06-29T15:57:22.215179+0000","last_unstale":"2024-06-29T15:57:23.226450+0000","last_undegraded":"2024-06-29T15:57:23.226450+0000","last_fullsized":"2024-06-29T15:57:23.226450+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:29.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226130+0000","last_change":"2024-06-29T15:57:22.224310+0000","last_active":"2024-06-29T15:57:23.226130+0000","last_peered":"2024-06-29T15:57:23.226130+0000","last_clean":"2024-06-29T15:57:23.226130+0000","last_became_active":"2024-06-29T15:57:22.223959+0000","last_became_peered":"2024-06-29T15:57:22.223959+0000","last_unstale":"2024-06-29T15:57:23.226130+0000","last_undegraded":"2024-06-29T15:57:23.226130+0000","last_fullsized":"2024-06-29T15:57:23.226130+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:13.811309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225893+0000","last_change":"2024-06-29T15:57:22.222075+0000","last_active":"2024-06-29T15:57:23.225893+0000","last_peered":"2024-06-29T15:57:23.225893+0000","last_clean":"2024-06-29T15:57:23.225893+0000","last_became_active":"2024-06-29T15:57:22.221767+0000","last_became_peered":"2024-06-29T15:57:22.221767+0000","last_unstale":"2024-06-29T15:57:23.225893+0000","last_undegraded":"2024-06-29T15:57:23.225893+0000","last_fullsized":"2024-06-29T15:57:23.225893+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:13.762887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559791+0000","last_change":"2024-06-29T15:57:22.232432+0000","last_active":"2024-06-29T15:57:23.559791+0000","last_peered":"2024-06-29T15:57:23.559791+0000","last_clean":"2024-06-29T15:57:23.559791+0000","last_became_active":"2024-06-29T15:57:22.232102+0000","last_became_peered":"2024-06-29T15:57:22.232102+0000","last_unstale":"2024-06-29T15:57:23.559791+0000","last_undegraded":"2024-06-29T15:57:23.559791+0000","last_fullsized":"2024-06-29T15:57:23.559791+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:41.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559347+0000","last_change":"2024-06-29T15:57:22.223814+0000","last_active":"2024-06-29T15:57:23.559347+0000","last_peered":"2024-06-29T15:57:23.559347+0000","last_clean":"2024-06-29T15:57:23.559347+0000","last_became_active":"2024-06-29T15:57:22.223464+0000","last_became_peered":"2024-06-29T15:57:22.223464+0000","last_unstale":"2024-06-29T15:57:23.559347+0000","last_undegraded":"2024-06-29T15:57:23.559347+0000","last_fullsized":"2024-06-29T15:57:23.559347+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:40.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225812+0000","last_change":"2024-06-29T15:57:22.234902+0000","last_active":"2024-06-29T15:57:23.225812+0000","last_peered":"2024-06-29T15:57:23.225812+0000","last_clean":"2024-06-29T15:57:23.225812+0000","last_became_active":"2024-06-29T15:57:22.233244+0000","last_became_peered":"2024-06-29T15:57:22.233244+0000","last_unstale":"2024-06-29T15:57:23.225812+0000","last_undegraded":"2024-06-29T15:57:23.225812+0000","last_fullsized":"2024-06-29T15:57:23.225812+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:58.786612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428127+0000","last_change":"2024-06-29T15:57:22.225280+0000","last_active":"2024-06-29T15:57:23.428127+0000","last_peered":"2024-06-29T15:57:23.428127+0000","last_clean":"2024-06-29T15:57:23.428127+0000","last_became_active":"2024-06-29T15:57:22.224986+0000","last_became_peered":"2024-06-29T15:57:22.224986+0000","last_unstale":"2024-06-29T15:57:23.428127+0000","last_undegraded":"2024-06-29T15:57:23.428127+0000","last_fullsized":"2024-06-29T15:57:23.428127+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:07.391769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222931+0000","last_change":"2024-06-29T15:57:22.236939+0000","last_active":"2024-06-29T15:57:23.222931+0000","last_peered":"2024-06-29T15:57:23.222931+0000","last_clean":"2024-06-29T15:57:23.222931+0000","last_became_active":"2024-06-29T15:57:22.236129+0000","last_became_peered":"2024-06-29T15:57:22.236129+0000","last_unstale":"2024-06-29T15:57:23.222931+0000","last_undegraded":"2024-06-29T15:57:23.222931+0000","last_fullsized":"2024-06-29T15:57:23.222931+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:13.274461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560090+0000","last_change":"2024-06-29T15:57:22.227845+0000","last_active":"2024-06-29T15:57:23.560090+0000","last_peered":"2024-06-29T15:57:23.560090+0000","last_clean":"2024-06-29T15:57:23.560090+0000","last_became_active":"2024-06-29T15:57:22.227549+0000","last_became_peered":"2024-06-29T15:57:22.227549+0000","last_unstale":"2024-06-29T15:57:23.560090+0000","last_undegraded":"2024-06-29T15:57:23.560090+0000","last_fullsized":"2024-06-29T15:57:23.560090+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:22.735783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225452+0000","last_change":"2024-06-29T15:57:22.227876+0000","last_active":"2024-06-29T15:57:23.225452+0000","last_peered":"2024-06-29T15:57:23.225452+0000","last_clean":"2024-06-29T15:57:23.225452+0000","last_became_active":"2024-06-29T15:57:22.227591+0000","last_became_peered":"2024-06-29T15:57:22.227591+0000","last_unstale":"2024-06-29T15:57:23.225452+0000","last_undegraded":"2024-06-29T15:57:23.225452+0000","last_fullsized":"2024-06-29T15:57:23.225452+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:45.600754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559566+0000","last_change":"2024-06-29T15:57:22.236151+0000","last_active":"2024-06-29T15:57:23.559566+0000","last_peered":"2024-06-29T15:57:23.559566+0000","last_clean":"2024-06-29T15:57:23.559566+0000","last_became_active":"2024-06-29T15:57:22.235409+0000","last_became_peered":"2024-06-29T15:57:22.235409+0000","last_unstale":"2024-06-29T15:57:23.559566+0000","last_undegraded":"2024-06-29T15:57:23.559566+0000","last_fullsized":"2024-06-29T15:57:23.559566+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:56.149438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559526+0000","last_change":"2024-06-29T15:57:22.239789+0000","last_active":"2024-06-29T15:57:23.559526+0000","last_peered":"2024-06-29T15:57:23.559526+0000","last_clean":"2024-06-29T15:57:23.559526+0000","last_became_active":"2024-06-29T15:57:22.239438+0000","last_became_peered":"2024-06-29T15:57:22.239438+0000","last_unstale":"2024-06-29T15:57:23.559526+0000","last_undegraded":"2024-06-29T15:57:23.559526+0000","last_fullsized":"2024-06-29T15:57:23.559526+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:08.408484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559240+0000","last_change":"2024-06-29T15:57:22.224277+0000","last_active":"2024-06-29T15:57:23.559240+0000","last_peered":"2024-06-29T15:57:23.559240+0000","last_clean":"2024-06-29T15:57:23.559240+0000","last_became_active":"2024-06-29T15:57:22.223975+0000","last_became_peered":"2024-06-29T15:57:22.223975+0000","last_unstale":"2024-06-29T15:57:23.559240+0000","last_undegraded":"2024-06-29T15:57:23.559240+0000","last_fullsized":"2024-06-29T15:57:23.559240+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:46.705935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559522+0000","last_change":"2024-06-29T15:57:22.228934+0000","last_active":"2024-06-29T15:57:23.559522+0000","last_peered":"2024-06-29T15:57:23.559522+0000","last_clean":"2024-06-29T15:57:23.559522+0000","last_became_active":"2024-06-29T15:57:22.228603+0000","last_became_peered":"2024-06-29T15:57:22.228603+0000","last_unstale":"2024-06-29T15:57:23.559522+0000","last_undegraded":"2024-06-29T15:57:23.559522+0000","last_fullsized":"2024-06-29T15:57:23.559522+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:28.391534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"221'4","reported_seq":18,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560047+0000","last_change":"2024-06-29T15:57:21.218674+0000","last_active":"2024-06-29T15:57:23.560047+0000","last_peered":"2024-06-29T15:57:23.560047+0000","last_clean":"2024-06-29T15:57:23.560047+0000","last_became_active":"2024-06-29T15:57:21.218397+0000","last_became_peered":"2024-06-29T15:57:21.218397+0000","last_unstale":"2024-06-29T15:57:23.560047+0000","last_undegraded":"2024-06-29T15:57:23.560047+0000","last_fullsized":"2024-06-29T15:57:23.560047+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.766844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.427746+0000","last_change":"2024-06-29T15:57:22.224449+0000","last_active":"2024-06-29T15:57:23.427746+0000","last_peered":"2024-06-29T15:57:23.427746+0000","last_clean":"2024-06-29T15:57:23.427746+0000","last_became_active":"2024-06-29T15:57:22.224124+0000","last_became_peered":"2024-06-29T15:57:22.224124+0000","last_unstale":"2024-06-29T15:57:23.427746+0000","last_undegraded":"2024-06-29T15:57:23.427746+0000","last_fullsized":"2024-06-29T15:57:23.427746+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:05.649826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428176+0000","last_change":"2024-06-29T15:57:21.213288+0000","last_active":"2024-06-29T15:57:23.428176+0000","last_peered":"2024-06-29T15:57:23.428176+0000","last_clean":"2024-06-29T15:57:23.428176+0000","last_became_active":"2024-06-29T15:57:21.212954+0000","last_became_peered":"2024-06-29T15:57:21.212954+0000","last_unstale":"2024-06-29T15:57:23.428176+0000","last_undegraded":"2024-06-29T15:57:23.428176+0000","last_fullsized":"2024-06-29T15:57:23.428176+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:24.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225996+0000","last_change":"2024-06-29T15:57:22.213237+0000","last_active":"2024-06-29T15:57:23.225996+0000","last_peered":"2024-06-29T15:57:23.225996+0000","last_clean":"2024-06-29T15:57:23.225996+0000","last_became_active":"2024-06-29T15:57:22.212908+0000","last_became_peered":"2024-06-29T15:57:22.212908+0000","last_unstale":"2024-06-29T15:57:23.225996+0000","last_undegraded":"2024-06-29T15:57:23.225996+0000","last_fullsized":"2024-06-29T15:57:23.225996+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:31.343924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560234+0000","last_change":"2024-06-29T15:57:21.210602+0000","last_active":"2024-06-29T15:57:23.560234+0000","last_peered":"2024-06-29T15:57:23.560234+0000","last_clean":"2024-06-29T15:57:23.560234+0000","last_became_active":"2024-06-29T15:57:21.210296+0000","last_became_peered":"2024-06-29T15:57:21.210296+0000","last_unstale":"2024-06-29T15:57:23.560234+0000","last_undegraded":"2024-06-29T15:57:23.560234+0000","last_fullsized":"2024-06-29T15:57:23.560234+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:07.128352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559868+0000","last_change":"2024-06-29T15:57:22.226181+0000","last_active":"2024-06-29T15:57:23.559868+0000","last_peered":"2024-06-29T15:57:23.559868+0000","last_clean":"2024-06-29T15:57:23.559868+0000","last_became_active":"2024-06-29T15:57:22.225855+0000","last_became_peered":"2024-06-29T15:57:22.225855+0000","last_unstale":"2024-06-29T15:57:23.559868+0000","last_undegraded":"2024-06-29T15:57:23.559868+0000","last_fullsized":"2024-06-29T15:57:23.559868+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:37.067580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225538+0000","last_change":"2024-06-29T15:57:21.210643+0000","last_active":"2024-06-29T15:57:23.225538+0000","last_peered":"2024-06-29T15:57:23.225538+0000","last_clean":"2024-06-29T15:57:23.225538+0000","last_became_active":"2024-06-29T15:57:21.209805+0000","last_became_peered":"2024-06-29T15:57:21.209805+0000","last_unstale":"2024-06-29T15:57:23.225538+0000","last_undegraded":"2024-06-29T15:57:23.225538+0000","last_fullsized":"2024-06-29T15:57:23.225538+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:27.618049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559673+0000","last_change":"2024-06-29T15:57:22.235721+0000","last_active":"2024-06-29T15:57:23.559673+0000","last_peered":"2024-06-29T15:57:23.559673+0000","last_clean":"2024-06-29T15:57:23.559673+0000","last_became_active":"2024-06-29T15:57:22.235380+0000","last_became_peered":"2024-06-29T15:57:22.235380+0000","last_unstale":"2024-06-29T15:57:23.559673+0000","last_undegraded":"2024-06-29T15:57:23.559673+0000","last_fullsized":"2024-06-29T15:57:23.559673+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:38.144376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559711+0000","last_change":"2024-06-29T15:57:21.211113+0000","last_active":"2024-06-29T15:57:23.559711+0000","last_peered":"2024-06-29T15:57:23.559711+0000","last_clean":"2024-06-29T15:57:23.559711+0000","last_became_active":"2024-06-29T15:57:21.210817+0000","last_became_peered":"2024-06-29T15:57:21.210817+0000","last_unstale":"2024-06-29T15:57:23.559711+0000","last_undegraded":"2024-06-29T15:57:23.559711+0000","last_fullsized":"2024-06-29T15:57:23.559711+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:26.176884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560046+0000","last_change":"2024-06-29T15:57:22.214168+0000","last_active":"2024-06-29T15:57:23.560046+0000","last_peered":"2024-06-29T15:57:23.560046+0000","last_clean":"2024-06-29T15:57:23.560046+0000","last_became_active":"2024-06-29T15:57:22.213780+0000","last_became_peered":"2024-06-29T15:57:22.213780+0000","last_unstale":"2024-06-29T15:57:23.560046+0000","last_undegraded":"2024-06-29T15:57:23.560046+0000","last_fullsized":"2024-06-29T15:57:23.560046+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:01.904725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559645+0000","last_change":"2024-06-29T15:57:21.212034+0000","last_active":"2024-06-29T15:57:23.559645+0000","last_peered":"2024-06-29T15:57:23.559645+0000","last_clean":"2024-06-29T15:57:23.559645+0000","last_became_active":"2024-06-29T15:57:21.211762+0000","last_became_peered":"2024-06-29T15:57:21.211762+0000","last_unstale":"2024-06-29T15:57:23.559645+0000","last_undegraded":"2024-06-29T15:57:23.559645+0000","last_fullsized":"2024-06-29T15:57:23.559645+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:36.775853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225820+0000","last_change":"2024-06-29T15:57:22.234743+0000","last_active":"2024-06-29T15:57:23.225820+0000","last_peered":"2024-06-29T15:57:23.225820+0000","last_clean":"2024-06-29T15:57:23.225820+0000","last_became_active":"2024-06-29T15:57:22.233091+0000","last_became_peered":"2024-06-29T15:57:22.233091+0000","last_unstale":"2024-06-29T15:57:23.225820+0000","last_undegraded":"2024-06-29T15:57:23.225820+0000","last_fullsized":"2024-06-29T15:57:23.225820+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:46.131681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222578+0000","last_change":"2024-06-29T15:57:21.208896+0000","last_active":"2024-06-29T15:57:23.222578+0000","last_peered":"2024-06-29T15:57:23.222578+0000","last_clean":"2024-06-29T15:57:23.222578+0000","last_became_active":"2024-06-29T15:57:21.208555+0000","last_became_peered":"2024-06-29T15:57:21.208555+0000","last_unstale":"2024-06-29T15:57:23.222578+0000","last_undegraded":"2024-06-29T15:57:23.222578+0000","last_fullsized":"2024-06-29T15:57:23.222578+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559436+0000","last_change":"2024-06-29T15:57:22.235312+0000","last_active":"2024-06-29T15:57:23.559436+0000","last_peered":"2024-06-29T15:57:23.559436+0000","last_clean":"2024-06-29T15:57:23.559436+0000","last_became_active":"2024-06-29T15:57:22.234828+0000","last_became_peered":"2024-06-29T15:57:22.234828+0000","last_unstale":"2024-06-29T15:57:23.559436+0000","last_undegraded":"2024-06-29T15:57:23.559436+0000","last_fullsized":"2024-06-29T15:57:23.559436+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:39.661714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559507+0000","last_change":"2024-06-29T15:57:21.218814+0000","last_active":"2024-06-29T15:57:23.559507+0000","last_peered":"2024-06-29T15:57:23.559507+0000","last_clean":"2024-06-29T15:57:23.559507+0000","last_became_active":"2024-06-29T15:57:21.218495+0000","last_became_peered":"2024-06-29T15:57:21.218495+0000","last_unstale":"2024-06-29T15:57:23.559507+0000","last_undegraded":"2024-06-29T15:57:23.559507+0000","last_fullsized":"2024-06-29T15:57:23.559507+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:04.680014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222556+0000","last_change":"2024-06-29T15:57:22.219192+0000","last_active":"2024-06-29T15:57:23.222556+0000","last_peered":"2024-06-29T15:57:23.222556+0000","last_clean":"2024-06-29T15:57:23.222556+0000","last_became_active":"2024-06-29T15:57:22.218844+0000","last_became_peered":"2024-06-29T15:57:22.218844+0000","last_unstale":"2024-06-29T15:57:23.222556+0000","last_undegraded":"2024-06-29T15:57:23.222556+0000","last_fullsized":"2024-06-29T15:57:23.222556+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:50.717283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559891+0000","last_change":"2024-06-29T15:57:21.214536+0000","last_active":"2024-06-29T15:57:23.559891+0000","last_peered":"2024-06-29T15:57:23.559891+0000","last_clean":"2024-06-29T15:57:23.559891+0000","last_became_active":"2024-06-29T15:57:21.214227+0000","last_became_peered":"2024-06-29T15:57:21.214227+0000","last_unstale":"2024-06-29T15:57:23.559891+0000","last_undegraded":"2024-06-29T15:57:23.559891+0000","last_fullsized":"2024-06-29T15:57:23.559891+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:25.839522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225514+0000","last_change":"2024-06-29T15:57:22.231011+0000","last_active":"2024-06-29T15:57:23.225514+0000","last_peered":"2024-06-29T15:57:23.225514+0000","last_clean":"2024-06-29T15:57:23.225514+0000","last_became_active":"2024-06-29T15:57:22.230675+0000","last_became_peered":"2024-06-29T15:57:22.230675+0000","last_unstale":"2024-06-29T15:57:23.225514+0000","last_undegraded":"2024-06-29T15:57:23.225514+0000","last_fullsized":"2024-06-29T15:57:23.225514+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:09.478769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"221'3","reported_seq":17,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225700+0000","last_change":"2024-06-29T15:57:21.210270+0000","last_active":"2024-06-29T15:57:23.225700+0000","last_peered":"2024-06-29T15:57:23.225700+0000","last_clean":"2024-06-29T15:57:23.225700+0000","last_became_active":"2024-06-29T15:57:21.209731+0000","last_became_peered":"2024-06-29T15:57:21.209731+0000","last_unstale":"2024-06-29T15:57:23.225700+0000","last_undegraded":"2024-06-29T15:57:23.225700+0000","last_fullsized":"2024-06-29T15:57:23.225700+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:18.182233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559925+0000","last_change":"2024-06-29T15:57:22.224600+0000","last_active":"2024-06-29T15:57:23.559925+0000","last_peered":"2024-06-29T15:57:23.559925+0000","last_clean":"2024-06-29T15:57:23.559925+0000","last_became_active":"2024-06-29T15:57:22.224260+0000","last_became_peered":"2024-06-29T15:57:22.224260+0000","last_unstale":"2024-06-29T15:57:23.559925+0000","last_undegraded":"2024-06-29T15:57:23.559925+0000","last_fullsized":"2024-06-29T15:57:23.559925+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:21.773370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"221'6","reported_seq":20,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560020+0000","last_change":"2024-06-29T15:57:21.217755+0000","last_active":"2024-06-29T15:57:23.560020+0000","last_peered":"2024-06-29T15:57:23.560020+0000","last_clean":"2024-06-29T15:57:23.560020+0000","last_became_active":"2024-06-29T15:57:21.216983+0000","last_became_peered":"2024-06-29T15:57:21.216983+0000","last_unstale":"2024-06-29T15:57:23.560020+0000","last_undegraded":"2024-06-29T15:57:23.560020+0000","last_fullsized":"2024-06-29T15:57:23.560020+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:31.889343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560016+0000","last_change":"2024-06-29T15:57:22.234802+0000","last_active":"2024-06-29T15:57:23.560016+0000","last_peered":"2024-06-29T15:57:23.560016+0000","last_clean":"2024-06-29T15:57:23.560016+0000","last_became_active":"2024-06-29T15:57:22.234492+0000","last_became_peered":"2024-06-29T15:57:22.234492+0000","last_unstale":"2024-06-29T15:57:23.560016+0000","last_undegraded":"2024-06-29T15:57:23.560016+0000","last_fullsized":"2024-06-29T15:57:23.560016+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:37.928014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559833+0000","last_change":"2024-06-29T15:57:21.215892+0000","last_active":"2024-06-29T15:57:23.559833+0000","last_peered":"2024-06-29T15:57:23.559833+0000","last_clean":"2024-06-29T15:57:23.559833+0000","last_became_active":"2024-06-29T15:57:21.215250+0000","last_became_peered":"2024-06-29T15:57:21.215250+0000","last_unstale":"2024-06-29T15:57:23.559833+0000","last_undegraded":"2024-06-29T15:57:23.559833+0000","last_fullsized":"2024-06-29T15:57:23.559833+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:23.234622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222962+0000","last_change":"2024-06-29T15:57:22.241881+0000","last_active":"2024-06-29T15:57:23.222962+0000","last_peered":"2024-06-29T15:57:23.222962+0000","last_clean":"2024-06-29T15:57:23.222962+0000","last_became_active":"2024-06-29T15:57:22.241549+0000","last_became_peered":"2024-06-29T15:57:22.241549+0000","last_unstale":"2024-06-29T15:57:23.222962+0000","last_undegraded":"2024-06-29T15:57:23.222962+0000","last_fullsized":"2024-06-29T15:57:23.222962+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:25.352328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222741+0000","last_change":"2024-06-29T15:57:21.203410+0000","last_active":"2024-06-29T15:57:23.222741+0000","last_peered":"2024-06-29T15:57:23.222741+0000","last_clean":"2024-06-29T15:57:23.222741+0000","last_became_active":"2024-06-29T15:57:21.203034+0000","last_became_peered":"2024-06-29T15:57:21.203034+0000","last_unstale":"2024-06-29T15:57:23.222741+0000","last_undegraded":"2024-06-29T15:57:23.222741+0000","last_fullsized":"2024-06-29T15:57:23.222741+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:52.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225471+0000","last_change":"2024-06-29T15:57:22.228484+0000","last_active":"2024-06-29T15:57:23.225471+0000","last_peered":"2024-06-29T15:57:23.225471+0000","last_clean":"2024-06-29T15:57:23.225471+0000","last_became_active":"2024-06-29T15:57:22.227880+0000","last_became_peered":"2024-06-29T15:57:22.227880+0000","last_unstale":"2024-06-29T15:57:23.225471+0000","last_undegraded":"2024-06-29T15:57:23.225471+0000","last_fullsized":"2024-06-29T15:57:23.225471+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:21.749071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225100+0000","last_change":"2024-06-29T15:57:21.215032+0000","last_active":"2024-06-29T15:57:23.225100+0000","last_peered":"2024-06-29T15:57:23.225100+0000","last_clean":"2024-06-29T15:57:23.225100+0000","last_became_active":"2024-06-29T15:57:21.214557+0000","last_became_peered":"2024-06-29T15:57:21.214557+0000","last_unstale":"2024-06-29T15:57:23.225100+0000","last_undegraded":"2024-06-29T15:57:23.225100+0000","last_fullsized":"2024-06-29T15:57:23.225100+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560171+0000","last_change":"2024-06-29T15:57:22.223112+0000","last_active":"2024-06-29T15:57:23.560171+0000","last_peered":"2024-06-29T15:57:23.560171+0000","last_clean":"2024-06-29T15:57:23.560171+0000","last_became_active":"2024-06-29T15:57:22.222632+0000","last_became_peered":"2024-06-29T15:57:22.222632+0000","last_unstale":"2024-06-29T15:57:23.560171+0000","last_undegraded":"2024-06-29T15:57:23.560171+0000","last_fullsized":"2024-06-29T15:57:23.560171+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:05.408776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"221'8","reported_seq":22,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559228+0000","last_change":"2024-06-29T15:57:21.216156+0000","last_active":"2024-06-29T15:57:23.559228+0000","last_peered":"2024-06-29T15:57:23.559228+0000","last_clean":"2024-06-29T15:57:23.559228+0000","last_became_active":"2024-06-29T15:57:21.214088+0000","last_became_peered":"2024-06-29T15:57:21.214088+0000","last_unstale":"2024-06-29T15:57:23.559228+0000","last_undegraded":"2024-06-29T15:57:23.559228+0000","last_fullsized":"2024-06-29T15:57:23.559228+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:30.314243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225778+0000","last_change":"2024-06-29T15:57:22.238347+0000","last_active":"2024-06-29T15:57:23.225778+0000","last_peered":"2024-06-29T15:57:23.225778+0000","last_clean":"2024-06-29T15:57:23.225778+0000","last_became_active":"2024-06-29T15:57:22.236896+0000","last_became_peered":"2024-06-29T15:57:22.236896+0000","last_unstale":"2024-06-29T15:57:23.225778+0000","last_undegraded":"2024-06-29T15:57:23.225778+0000","last_fullsized":"2024-06-29T15:57:23.225778+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:39.739194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428213+0000","last_change":"2024-06-29T15:57:21.212718+0000","last_active":"2024-06-29T15:57:23.428213+0000","last_peered":"2024-06-29T15:57:23.428213+0000","last_clean":"2024-06-29T15:57:23.428213+0000","last_became_active":"2024-06-29T15:57:21.212406+0000","last_became_peered":"2024-06-29T15:57:21.212406+0000","last_unstale":"2024-06-29T15:57:23.428213+0000","last_undegraded":"2024-06-29T15:57:23.428213+0000","last_fullsized":"2024-06-29T15:57:23.428213+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:00.180622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428033+0000","last_change":"2024-06-29T15:57:22.225793+0000","last_active":"2024-06-29T15:57:23.428033+0000","last_peered":"2024-06-29T15:57:23.428033+0000","last_clean":"2024-06-29T15:57:23.428033+0000","last_became_active":"2024-06-29T15:57:22.224806+0000","last_became_peered":"2024-06-29T15:57:22.224806+0000","last_unstale":"2024-06-29T15:57:23.428033+0000","last_undegraded":"2024-06-29T15:57:23.428033+0000","last_fullsized":"2024-06-29T15:57:23.428033+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:03.667477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428289+0000","last_change":"2024-06-29T15:57:21.216179+0000","last_active":"2024-06-29T15:57:23.428289+0000","last_peered":"2024-06-29T15:57:23.428289+0000","last_clean":"2024-06-29T15:57:23.428289+0000","last_became_active":"2024-06-29T15:57:21.215876+0000","last_became_peered":"2024-06-29T15:57:21.215876+0000","last_unstale":"2024-06-29T15:57:23.428289+0000","last_undegraded":"2024-06-29T15:57:23.428289+0000","last_fullsized":"2024-06-29T15:57:23.428289+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:37.942186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428078+0000","last_change":"2024-06-29T15:57:22.232727+0000","last_active":"2024-06-29T15:57:23.428078+0000","last_peered":"2024-06-29T15:57:23.428078+0000","last_clean":"2024-06-29T15:57:23.428078+0000","last_became_active":"2024-06-29T15:57:22.232392+0000","last_became_peered":"2024-06-29T15:57:22.232392+0000","last_unstale":"2024-06-29T15:57:23.428078+0000","last_undegraded":"2024-06-29T15:57:23.428078+0000","last_fullsized":"2024-06-29T15:57:23.428078+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:54.805984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560256+0000","last_change":"2024-06-29T15:57:21.210528+0000","last_active":"2024-06-29T15:57:23.560256+0000","last_peered":"2024-06-29T15:57:23.560256+0000","last_clean":"2024-06-29T15:57:23.560256+0000","last_became_active":"2024-06-29T15:57:21.208691+0000","last_became_peered":"2024-06-29T15:57:21.208691+0000","last_unstale":"2024-06-29T15:57:23.560256+0000","last_undegraded":"2024-06-29T15:57:23.560256+0000","last_fullsized":"2024-06-29T15:57:23.560256+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.762590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226207+0000","last_change":"2024-06-29T15:57:22.217630+0000","last_active":"2024-06-29T15:57:23.226207+0000","last_peered":"2024-06-29T15:57:23.226207+0000","last_clean":"2024-06-29T15:57:23.226207+0000","last_became_active":"2024-06-29T15:57:22.217303+0000","last_became_peered":"2024-06-29T15:57:22.217303+0000","last_unstale":"2024-06-29T15:57:23.226207+0000","last_undegraded":"2024-06-29T15:57:23.226207+0000","last_fullsized":"2024-06-29T15:57:23.226207+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:05.554814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560152+0000","last_change":"2024-06-29T15:57:21.213193+0000","last_active":"2024-06-29T15:57:23.560152+0000","last_peered":"2024-06-29T15:57:23.560152+0000","last_clean":"2024-06-29T15:57:23.560152+0000","last_became_active":"2024-06-29T15:57:21.212838+0000","last_became_peered":"2024-06-29T15:57:21.212838+0000","last_unstale":"2024-06-29T15:57:23.560152+0000","last_undegraded":"2024-06-29T15:57:23.560152+0000","last_fullsized":"2024-06-29T15:57:23.560152+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:54.574391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560023+0000","last_change":"2024-06-29T15:57:22.231883+0000","last_active":"2024-06-29T15:57:23.560023+0000","last_peered":"2024-06-29T15:57:23.560023+0000","last_clean":"2024-06-29T15:57:23.560023+0000","last_became_active":"2024-06-29T15:57:22.231588+0000","last_became_peered":"2024-06-29T15:57:22.231588+0000","last_unstale":"2024-06-29T15:57:23.560023+0000","last_undegraded":"2024-06-29T15:57:23.560023+0000","last_fullsized":"2024-06-29T15:57:23.560023+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:17.705760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559988+0000","last_change":"2024-06-29T15:57:21.215358+0000","last_active":"2024-06-29T15:57:23.559988+0000","last_peered":"2024-06-29T15:57:23.559988+0000","last_clean":"2024-06-29T15:57:23.559988+0000","last_became_active":"2024-06-29T15:57:21.213979+0000","last_became_peered":"2024-06-29T15:57:21.213979+0000","last_unstale":"2024-06-29T15:57:23.559988+0000","last_undegraded":"2024-06-29T15:57:23.559988+0000","last_fullsized":"2024-06-29T15:57:23.559988+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:42.446135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559969+0000","last_change":"2024-06-29T15:57:22.238829+0000","last_active":"2024-06-29T15:57:23.559969+0000","last_peered":"2024-06-29T15:57:23.559969+0000","last_clean":"2024-06-29T15:57:23.559969+0000","last_became_active":"2024-06-29T15:57:22.238520+0000","last_became_peered":"2024-06-29T15:57:22.238520+0000","last_unstale":"2024-06-29T15:57:23.559969+0000","last_undegraded":"2024-06-29T15:57:23.559969+0000","last_fullsized":"2024-06-29T15:57:23.559969+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:22.127384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559664+0000","last_change":"2024-06-29T15:57:21.208592+0000","last_active":"2024-06-29T15:57:23.559664+0000","last_peered":"2024-06-29T15:57:23.559664+0000","last_clean":"2024-06-29T15:57:23.559664+0000","last_became_active":"2024-06-29T15:57:21.208156+0000","last_became_peered":"2024-06-29T15:57:21.208156+0000","last_unstale":"2024-06-29T15:57:23.559664+0000","last_undegraded":"2024-06-29T15:57:23.559664+0000","last_fullsized":"2024-06-29T15:57:23.559664+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:20.325402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225614+0000","last_change":"2024-06-29T15:57:22.228211+0000","last_active":"2024-06-29T15:57:23.225614+0000","last_peered":"2024-06-29T15:57:23.225614+0000","last_clean":"2024-06-29T15:57:23.225614+0000","last_became_active":"2024-06-29T15:57:22.227519+0000","last_became_peered":"2024-06-29T15:57:22.227519+0000","last_unstale":"2024-06-29T15:57:23.225614+0000","last_undegraded":"2024-06-29T15:57:23.225614+0000","last_fullsized":"2024-06-29T15:57:23.225614+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:16:35.999038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.560306+0000","last_change":"2024-06-29T15:57:21.212085+0000","last_active":"2024-06-29T15:57:23.560306+0000","last_peered":"2024-06-29T15:57:23.560306+0000","last_clean":"2024-06-29T15:57:23.560306+0000","last_became_active":"2024-06-29T15:57:21.211731+0000","last_became_peered":"2024-06-29T15:57:21.211731+0000","last_unstale":"2024-06-29T15:57:23.560306+0000","last_undegraded":"2024-06-29T15:57:23.560306+0000","last_fullsized":"2024-06-29T15:57:23.560306+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:36.061054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226814+0000","last_change":"2024-06-29T15:57:22.213920+0000","last_active":"2024-06-29T15:57:23.226814+0000","last_peered":"2024-06-29T15:57:23.226814+0000","last_clean":"2024-06-29T15:57:23.226814+0000","last_became_active":"2024-06-29T15:57:22.213583+0000","last_became_peered":"2024-06-29T15:57:22.213583+0000","last_unstale":"2024-06-29T15:57:23.226814+0000","last_undegraded":"2024-06-29T15:57:23.226814+0000","last_fullsized":"2024-06-29T15:57:23.226814+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:31.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.427663+0000","last_change":"2024-06-29T15:57:21.211203+0000","last_active":"2024-06-29T15:57:23.427663+0000","last_peered":"2024-06-29T15:57:23.427663+0000","last_clean":"2024-06-29T15:57:23.427663+0000","last_became_active":"2024-06-29T15:57:21.210898+0000","last_became_peered":"2024-06-29T15:57:21.210898+0000","last_unstale":"2024-06-29T15:57:23.427663+0000","last_undegraded":"2024-06-29T15:57:23.427663+0000","last_fullsized":"2024-06-29T15:57:23.427663+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:17.270875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225961+0000","last_change":"2024-06-29T15:57:22.224294+0000","last_active":"2024-06-29T15:57:23.225961+0000","last_peered":"2024-06-29T15:57:23.225961+0000","last_clean":"2024-06-29T15:57:23.225961+0000","last_became_active":"2024-06-29T15:57:22.223922+0000","last_became_peered":"2024-06-29T15:57:22.223922+0000","last_unstale":"2024-06-29T15:57:23.225961+0000","last_undegraded":"2024-06-29T15:57:23.225961+0000","last_fullsized":"2024-06-29T15:57:23.225961+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:46.725456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"221'2","reported_seq":16,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225876+0000","last_change":"2024-06-29T15:57:21.212745+0000","last_active":"2024-06-29T15:57:23.225876+0000","last_peered":"2024-06-29T15:57:23.225876+0000","last_clean":"2024-06-29T15:57:23.225876+0000","last_became_active":"2024-06-29T15:57:21.212335+0000","last_became_peered":"2024-06-29T15:57:21.212335+0000","last_unstale":"2024-06-29T15:57:23.225876+0000","last_undegraded":"2024-06-29T15:57:23.225876+0000","last_fullsized":"2024-06-29T15:57:23.225876+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:07.049637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222567+0000","last_change":"2024-06-29T15:57:22.223140+0000","last_active":"2024-06-29T15:57:23.222567+0000","last_peered":"2024-06-29T15:57:23.222567+0000","last_clean":"2024-06-29T15:57:23.222567+0000","last_became_active":"2024-06-29T15:57:22.222839+0000","last_became_peered":"2024-06-29T15:57:22.222839+0000","last_unstale":"2024-06-29T15:57:23.222567+0000","last_undegraded":"2024-06-29T15:57:23.222567+0000","last_fullsized":"2024-06-29T15:57:23.222567+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:17.520475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226569+0000","last_change":"2024-06-29T15:57:21.203822+0000","last_active":"2024-06-29T15:57:23.226569+0000","last_peered":"2024-06-29T15:57:23.226569+0000","last_clean":"2024-06-29T15:57:23.226569+0000","last_became_active":"2024-06-29T15:57:21.203517+0000","last_became_peered":"2024-06-29T15:57:21.203517+0000","last_unstale":"2024-06-29T15:57:23.226569+0000","last_undegraded":"2024-06-29T15:57:23.226569+0000","last_fullsized":"2024-06-29T15:57:23.226569+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:05.810997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225198+0000","last_change":"2024-06-29T15:57:22.233919+0000","last_active":"2024-06-29T15:57:23.225198+0000","last_peered":"2024-06-29T15:57:23.225198+0000","last_clean":"2024-06-29T15:57:23.225198+0000","last_became_active":"2024-06-29T15:57:22.231028+0000","last_became_peered":"2024-06-29T15:57:22.231028+0000","last_unstale":"2024-06-29T15:57:23.225198+0000","last_undegraded":"2024-06-29T15:57:23.225198+0000","last_fullsized":"2024-06-29T15:57:23.225198+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:34:24.910914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559644+0000","last_change":"2024-06-29T15:57:21.217039+0000","last_active":"2024-06-29T15:57:23.559644+0000","last_peered":"2024-06-29T15:57:23.559644+0000","last_clean":"2024-06-29T15:57:23.559644+0000","last_became_active":"2024-06-29T15:57:21.216542+0000","last_became_peered":"2024-06-29T15:57:21.216542+0000","last_unstale":"2024-06-29T15:57:23.559644+0000","last_undegraded":"2024-06-29T15:57:23.559644+0000","last_fullsized":"2024-06-29T15:57:23.559644+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:21.729257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428068+0000","last_change":"2024-06-29T15:57:22.227969+0000","last_active":"2024-06-29T15:57:23.428068+0000","last_peered":"2024-06-29T15:57:23.428068+0000","last_clean":"2024-06-29T15:57:23.428068+0000","last_became_active":"2024-06-29T15:57:22.227579+0000","last_became_peered":"2024-06-29T15:57:22.227579+0000","last_unstale":"2024-06-29T15:57:23.428068+0000","last_undegraded":"2024-06-29T15:57:23.428068+0000","last_fullsized":"2024-06-29T15:57:23.428068+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.427896+0000","last_change":"2024-06-29T15:57:21.216399+0000","last_active":"2024-06-29T15:57:23.427896+0000","last_peered":"2024-06-29T15:57:23.427896+0000","last_clean":"2024-06-29T15:57:23.427896+0000","last_became_active":"2024-06-29T15:57:21.216104+0000","last_became_peered":"2024-06-29T15:57:21.216104+0000","last_unstale":"2024-06-29T15:57:23.427896+0000","last_undegraded":"2024-06-29T15:57:23.427896+0000","last_fullsized":"2024-06-29T15:57:23.427896+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226376+0000","last_change":"2024-06-29T15:57:22.229230+0000","last_active":"2024-06-29T15:57:23.226376+0000","last_peered":"2024-06-29T15:57:23.226376+0000","last_clean":"2024-06-29T15:57:23.226376+0000","last_became_active":"2024-06-29T15:57:22.228914+0000","last_became_peered":"2024-06-29T15:57:22.228914+0000","last_unstale":"2024-06-29T15:57:23.226376+0000","last_undegraded":"2024-06-29T15:57:23.226376+0000","last_fullsized":"2024-06-29T15:57:23.226376+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:09.918042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.226274+0000","last_change":"2024-06-29T15:57:21.213113+0000","last_active":"2024-06-29T15:57:23.226274+0000","last_peered":"2024-06-29T15:57:23.226274+0000","last_clean":"2024-06-29T15:57:23.226274+0000","last_became_active":"2024-06-29T15:57:21.212819+0000","last_became_peered":"2024-06-29T15:57:21.212819+0000","last_unstale":"2024-06-29T15:57:23.226274+0000","last_undegraded":"2024-06-29T15:57:23.226274+0000","last_fullsized":"2024-06-29T15:57:23.226274+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:59.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559925+0000","last_change":"2024-06-29T15:57:22.244396+0000","last_active":"2024-06-29T15:57:23.559925+0000","last_peered":"2024-06-29T15:57:23.559925+0000","last_clean":"2024-06-29T15:57:23.559925+0000","last_became_active":"2024-06-29T15:57:22.244044+0000","last_became_peered":"2024-06-29T15:57:22.244044+0000","last_unstale":"2024-06-29T15:57:23.559925+0000","last_undegraded":"2024-06-29T15:57:23.559925+0000","last_fullsized":"2024-06-29T15:57:23.559925+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.981276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225484+0000","last_change":"2024-06-29T15:57:21.210563+0000","last_active":"2024-06-29T15:57:23.225484+0000","last_peered":"2024-06-29T15:57:23.225484+0000","last_clean":"2024-06-29T15:57:23.225484+0000","last_became_active":"2024-06-29T15:57:21.210243+0000","last_became_peered":"2024-06-29T15:57:21.210243+0000","last_unstale":"2024-06-29T15:57:23.225484+0000","last_undegraded":"2024-06-29T15:57:23.225484+0000","last_fullsized":"2024-06-29T15:57:23.225484+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.900240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559946+0000","last_change":"2024-06-29T15:57:22.237722+0000","last_active":"2024-06-29T15:57:23.559946+0000","last_peered":"2024-06-29T15:57:23.559946+0000","last_clean":"2024-06-29T15:57:23.559946+0000","last_became_active":"2024-06-29T15:57:22.237372+0000","last_became_peered":"2024-06-29T15:57:22.237372+0000","last_unstale":"2024-06-29T15:57:23.559946+0000","last_undegraded":"2024-06-29T15:57:23.559946+0000","last_fullsized":"2024-06-29T15:57:23.559946+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:24.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428173+0000","last_change":"2024-06-29T15:57:21.212837+0000","last_active":"2024-06-29T15:57:23.428173+0000","last_peered":"2024-06-29T15:57:23.428173+0000","last_clean":"2024-06-29T15:57:23.428173+0000","last_became_active":"2024-06-29T15:57:21.212491+0000","last_became_peered":"2024-06-29T15:57:21.212491+0000","last_unstale":"2024-06-29T15:57:23.428173+0000","last_undegraded":"2024-06-29T15:57:23.428173+0000","last_fullsized":"2024-06-29T15:57:23.428173+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:35:53.641775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.222886+0000","last_change":"2024-06-29T15:57:22.223950+0000","last_active":"2024-06-29T15:57:23.222886+0000","last_peered":"2024-06-29T15:57:23.222886+0000","last_clean":"2024-06-29T15:57:23.222886+0000","last_became_active":"2024-06-29T15:57:22.223622+0000","last_became_peered":"2024-06-29T15:57:22.223622+0000","last_unstale":"2024-06-29T15:57:23.222886+0000","last_undegraded":"2024-06-29T15:57:23.222886+0000","last_fullsized":"2024-06-29T15:57:23.222886+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.556171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559353+0000","last_change":"2024-06-29T15:57:21.215589+0000","last_active":"2024-06-29T15:57:23.559353+0000","last_peered":"2024-06-29T15:57:23.559353+0000","last_clean":"2024-06-29T15:57:23.559353+0000","last_became_active":"2024-06-29T15:57:21.213719+0000","last_became_peered":"2024-06-29T15:57:21.213719+0000","last_unstale":"2024-06-29T15:57:23.559353+0000","last_undegraded":"2024-06-29T15:57:23.559353+0000","last_fullsized":"2024-06-29T15:57:23.559353+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:32.413502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":12,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.428127+0000","last_change":"2024-06-29T15:57:22.230794+0000","last_active":"2024-06-29T15:57:23.428127+0000","last_peered":"2024-06-29T15:57:23.428127+0000","last_clean":"2024-06-29T15:57:23.428127+0000","last_became_active":"2024-06-29T15:57:22.230474+0000","last_became_peered":"2024-06-29T15:57:22.230474+0000","last_unstale":"2024-06-29T15:57:23.428127+0000","last_undegraded":"2024-06-29T15:57:23.428127+0000","last_fullsized":"2024-06-29T15:57:23.428127+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:43.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"221'1","reported_seq":15,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.225354+0000","last_change":"2024-06-29T15:57:21.209163+0000","last_active":"2024-06-29T15:57:23.225354+0000","last_peered":"2024-06-29T15:57:23.225354+0000","last_clean":"2024-06-29T15:57:23.225354+0000","last_became_active":"2024-06-29T15:57:21.208825+0000","last_became_peered":"2024-06-29T15:57:21.208825+0000","last_unstale":"2024-06-29T15:57:23.225354+0000","last_undegraded":"2024-06-29T15:57:23.225354+0000","last_fullsized":"2024-06-29T15:57:23.225354+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:13.641268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"210'233","reported_seq":1101,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.559198+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:57:23.559198+0000","last_peered":"2024-06-29T15:57:23.559198+0000","last_clean":"2024-06-29T15:57:23.559198+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:57:23.559198+0000","last_undegraded":"2024-06-29T15:57:23.559198+0000","last_fullsized":"2024-06-29T15:57:23.559198+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543287,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52532,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319308,"statfs":{"total":96636764160,"available":96582971392,"internally_reserved":0,"allocated":3379200,"data_stored":2015142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543288,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42236,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329604,"statfs":{"total":96636764160,"available":96593514496,"internally_reserved":0,"allocated":2797568,"data_stored":1432034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543287,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2789376,"data_stored":1429896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543286,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51948,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319892,"statfs":{"total":96636764160,"available":96583569408,"internally_reserved":0,"allocated":2785280,"data_stored":1430422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510584,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47784,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324056,"statfs":{"total":96636764160,"available":96587833344,"internally_reserved":0,"allocated":2781184,"data_stored":1423218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575992,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47776,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324064,"statfs":{"total":96636764160,"available":96587841536,"internally_reserved":0,"allocated":2772992,"data_stored":1423182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510583,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47776,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324064,"statfs":{"total":96636764160,"available":96587841536,"internally_reserved":0,"allocated":2768896,"data_stored":1423092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510585,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42800,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329040,"statfs":{"total":96636764160,"available":96592936960,"internally_reserved":0,"allocated":3379200,"data_stored":2020298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:26.694 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:57:26.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:57:26.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 -- 172.21.15.59:0/3735639477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c074320 msgr2=0x7fbb3c0b5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 --2- 172.21.15.59:0/3735639477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c074320 0x7fbb3c0b5500 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c074a90 tx=0x7fbb2c02a980 comp rx=0 tx=0).stop 2024-06-29T15:57:26.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 -- 172.21.15.59:0/3735639477 shutdown_connections 2024-06-29T15:57:26.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 --2- 172.21.15.59:0/3735639477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c074320 0x7fbb3c0b5500 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 --2- 172.21.15.59:0/3735639477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb3c15ec90 0x7fbb3c073de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 --2- 172.21.15.59:0/3735639477 >> v2:172.21.15.153:3300/0 conn(0x7fbb3c15bb50 0x7fbb3c15e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 -- 172.21.15.59:0/3735639477 >> 172.21.15.59:0/3735639477 conn(0x7fbb3c0cd870 msgr2=0x7fbb3c0cdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 -- 172.21.15.59:0/3735639477 shutdown_connections 2024-06-29T15:57:26.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.852+0000 7fbb415fb640 1 -- 172.21.15.59:0/3735639477 wait complete. 2024-06-29T15:57:26.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.854+0000 7fbb415fb640 1 Processor -- start 2024-06-29T15:57:26.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.854+0000 7fbb415fb640 1 -- start start 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.855+0000 7fbb415fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb3c074320 0x7fbb3c1344e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.855+0000 7fbb415fb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb3c15bb50 0x7fbb3c138270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.855+0000 7fbb415fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c15ec90 0x7fbb3c1387b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.855+0000 7fbb415fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c15c0e0 con 0x7fbb3c15ec90 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.855+0000 7fbb415fb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c15bf60 con 0x7fbb3c074320 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.855+0000 7fbb415fb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3c0b5a40 con 0x7fbb3c15bb50 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c15ec90 0x7fbb3c1387b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c15ec90 0x7fbb3c1387b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46022/0 (socket says 172.21.15.59:46022) 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 -- 172.21.15.59:0/1241734838 learned_addr learned my addr 172.21.15.59:0/1241734838 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3a7fc640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb3c15bb50 0x7fbb3c138270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3affd640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb3c074320 0x7fbb3c1344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 -- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb3c15bb50 msgr2=0x7fbb3c138270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb3c15bb50 0x7fbb3c138270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 -- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb3c074320 msgr2=0x7fbb3c1344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb3c074320 0x7fbb3c1344e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 -- 172.21.15.59:0/1241734838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c1337d0 con 0x7fbb3c15ec90 2024-06-29T15:57:26.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb3b7fe640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c15ec90 0x7fbb3c1387b0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009d60 tx=0x7fbb2c003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb337fe640 1 -- 172.21.15.59:0/1241734838 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c034040 con 0x7fbb3c15ec90 2024-06-29T15:57:26.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb337fe640 1 -- 172.21.15.59:0/1241734838 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb2c0046e0 con 0x7fbb3c15ec90 2024-06-29T15:57:26.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3c1f6850 con 0x7fbb3c15ec90 2024-06-29T15:57:26.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb337fe640 1 -- 172.21.15.59:0/1241734838 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c0456e0 con 0x7fbb3c15ec90 2024-06-29T15:57:26.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.856+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb3c1f6e10 con 0x7fbb3c15ec90 2024-06-29T15:57:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.857+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3c0b6000 con 0x7fbb3c15ec90 2024-06-29T15:57:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.858+0000 7fbb337fe640 1 -- 172.21.15.59:0/1241734838 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fbb2c037020 con 0x7fbb3c15ec90 2024-06-29T15:57:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.859+0000 7fbb337fe640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fbb1c076420 0x7fbb1c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.859+0000 7fbb337fe640 1 -- 172.21.15.59:0/1241734838 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7fbb2c0b93c0 con 0x7fbb3c15ec90 2024-06-29T15:57:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.859+0000 7fbb3affd640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fbb1c076420 0x7fbb1c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.859+0000 7fbb3affd640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fbb1c076420 0x7fbb1c078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400aa10 tx=0x7fbb34005ef0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:26.863+0000 7fbb337fe640 1 -- 172.21.15.59:0/1241734838 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb2c082400 con 0x7fbb3c15ec90 2024-06-29T15:57:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.125+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fbb3c0cde90 con 0x7fbb3c15ec90 2024-06-29T15:57:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.128+0000 7fbb337fe640 1 -- 172.21.15.59:0/1241734838 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v51) ==== 190+0+147 (secure 0 0 0) 0x7fbb2c086700 con 0x7fbb3c15ec90 2024-06-29T15:57:27.129 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T15:57:27.129 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T15:57:27.130 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T15:57:27.130 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T15:57:27.130 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:57:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.130+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fbb1c076420 msgr2=0x7fbb1c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.130+0000 7fbb415fb640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fbb1c076420 0x7fbb1c078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400aa10 tx=0x7fbb34005ef0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.130+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c15ec90 msgr2=0x7fbb3c1387b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.130+0000 7fbb415fb640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c15ec90 0x7fbb3c1387b0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009d60 tx=0x7fbb2c003f70 comp rx=0 tx=0).stop 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.131+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 shutdown_connections 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.131+0000 7fbb415fb640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fbb1c076420 0x7fbb1c078930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.131+0000 7fbb415fb640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb3c15ec90 0x7fbb3c1387b0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.131+0000 7fbb415fb640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbb3c15bb50 0x7fbb3c138270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.131+0000 7fbb415fb640 1 --2- 172.21.15.59:0/1241734838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb3c074320 0x7fbb3c1344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.131+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 >> 172.21.15.59:0/1241734838 conn(0x7fbb3c0cd870 msgr2=0x7fbb3c139250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.131+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 shutdown_connections 2024-06-29T15:57:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.131+0000 7fbb415fb640 1 -- 172.21.15.59:0/1241734838 wait complete. 2024-06-29T15:57:27.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.260+0000 7fef3de88640 1 -- 172.21.15.59:0/2970683464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef38162770 msgr2=0x7fef380ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.260+0000 7fef3de88640 1 --2- 172.21.15.59:0/2970683464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef38162770 0x7fef380ce550 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fef3808c200 tx=0x7fef2802a4b0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.260+0000 7fef3de88640 1 -- 172.21.15.59:0/2970683464 shutdown_connections 2024-06-29T15:57:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.260+0000 7fef3de88640 1 --2- 172.21.15.59:0/2970683464 >> v2:172.21.15.153:3300/0 conn(0x7fef38162190 0x7fef380b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.260+0000 7fef3de88640 1 --2- 172.21.15.59:0/2970683464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef38162770 0x7fef380ce550 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.260+0000 7fef3de88640 1 -- 172.21.15.59:0/2970683464 >> 172.21.15.59:0/2970683464 conn(0x7fef380d45b0 msgr2=0x7fef380d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.260+0000 7fef3de88640 1 -- 172.21.15.59:0/2970683464 shutdown_connections 2024-06-29T15:57:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.260+0000 7fef3de88640 1 -- 172.21.15.59:0/2970683464 wait complete. 2024-06-29T15:57:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.262+0000 7fef3de88640 1 Processor -- start 2024-06-29T15:57:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.262+0000 7fef3de88640 1 -- start start 2024-06-29T15:57:27.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef3de88640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3815e0a0 0x7fef381f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef3de88640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef38162770 0x7fef3815fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef3de88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef381619b0 0x7fef38161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef3de88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fef380b4230 con 0x7fef381619b0 2024-06-29T15:57:27.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef3de88640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fef380b40b0 con 0x7fef38162770 2024-06-29T15:57:27.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef3de88640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fef380b43b0 con 0x7fef3815e0a0 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef381619b0 0x7fef38161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef381619b0 0x7fef38161dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46046/0 (socket says 172.21.15.59:46046) 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef37fff640 1 -- 172.21.15.59:0/3625728556 learned_addr learned my addr 172.21.15.59:0/3625728556 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef36ffd640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef38162770 0x7fef3815fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.263+0000 7fef377fe640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3815e0a0 0x7fef381f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef37fff640 1 -- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3815e0a0 msgr2=0x7fef381f5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef37fff640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3815e0a0 0x7fef381f5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef37fff640 1 -- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef38162770 msgr2=0x7fef3815fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef37fff640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef38162770 0x7fef3815fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef37fff640 1 -- 172.21.15.59:0/3625728556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef381f5890 con 0x7fef381619b0 2024-06-29T15:57:27.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef37fff640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef381619b0 0x7fef38161dc0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fef3000cab0 tx=0x7fef3000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef1ffff640 1 -- 172.21.15.59:0/3625728556 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30007df0 con 0x7fef381619b0 2024-06-29T15:57:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef1ffff640 1 -- 172.21.15.59:0/3625728556 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef30003a80 con 0x7fef381619b0 2024-06-29T15:57:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef38162300 con 0x7fef381619b0 2024-06-29T15:57:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef1ffff640 1 -- 172.21.15.59:0/3625728556 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30004020 con 0x7fef381619b0 2024-06-29T15:57:27.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.264+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3815b510 con 0x7fef381619b0 2024-06-29T15:57:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.265+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef380b48d0 con 0x7fef381619b0 2024-06-29T15:57:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.266+0000 7fef1ffff640 1 -- 172.21.15.59:0/3625728556 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fef300041c0 con 0x7fef381619b0 2024-06-29T15:57:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.266+0000 7fef1ffff640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fef240763f0 0x7fef24078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.266+0000 7fef1ffff640 1 -- 172.21.15.59:0/3625728556 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7fef300982e0 con 0x7fef381619b0 2024-06-29T15:57:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.267+0000 7fef377fe640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fef240763f0 0x7fef24078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.267+0000 7fef377fe640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fef240763f0 0x7fef24078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef28033000 tx=0x7fef28004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:27.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.271+0000 7fef1ffff640 1 -- 172.21.15.59:0/3625728556 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fef38162300 con 0x7fef381619b0 2024-06-29T15:57:27.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.499+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef3806c380 con 0x7fef381619b0 2024-06-29T15:57:27.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.500+0000 7fef1ffff640 1 -- 172.21.15.59:0/3625728556 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5242 (secure 0 0 0) 0x7fef381f5890 con 0x7fef381619b0 2024-06-29T15:57:27.501 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:27.501 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":115,"btime":"2024-06-29T15:57:25:241476+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":10747,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.589050+0000","modified":"2024-06-29T15:57:24.623107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11487":{"gid":11487,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1734341893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1734341893},{"type":"v1","addr":"172.21.15.59:6834","nonce":1734341893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11487,"qdb_cluster":[11487]},"id":13}]} 2024-06-29T15:57:27.501 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 115 2024-06-29T15:57:27.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fef240763f0 msgr2=0x7fef24078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fef240763f0 0x7fef24078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef28033000 tx=0x7fef28004730 comp rx=0 tx=0).stop 2024-06-29T15:57:27.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef381619b0 msgr2=0x7fef38161dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef381619b0 0x7fef38161dc0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fef3000cab0 tx=0x7fef3000cf70 comp rx=0 tx=0).stop 2024-06-29T15:57:27.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 shutdown_connections 2024-06-29T15:57:27.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fef240763f0 0x7fef24078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef381619b0 0x7fef38161dc0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef38162770 0x7fef3815fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 --2- 172.21.15.59:0/3625728556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef3815e0a0 0x7fef381f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 >> 172.21.15.59:0/3625728556 conn(0x7fef380d45b0 msgr2=0x7fef3815bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:27.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 shutdown_connections 2024-06-29T15:57:27.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:27.502+0000 7fef3de88640 1 -- 172.21.15.59:0/3625728556 wait complete. 2024-06-29T15:57:27.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:57:21.589050+0000', 'modified': '2024-06-29T15:57:24.623107+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11487': {'gid': 11487, 'name': 'a', 'rank': 0, 'incarnation': 111, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6834/1734341893', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1734341893}, {'type': 'v1', 'addr': '172.21.15.59:6834', 'nonce': 1734341893}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11487, 'qdb_cluster': [11487]} 2024-06-29T15:57:27.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:57:27.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T15:57:28.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.042+0000 7f40145bd640 1 -- 172.21.15.59:0/2229458850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ff4004800 msgr2=0x7f3ff4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.042+0000 7f40145bd640 1 --2- 172.21.15.59:0/2229458850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ff4004800 0x7f3ff4004c10 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f4000009b50 tx=0x7f3ff4005830 comp rx=0 tx=0).stop 2024-06-29T15:57:28.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.043+0000 7f4012b5a640 1 -- 172.21.15.59:0/2229458850 reap_dead start 2024-06-29T15:57:28.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.043+0000 7f40145bd640 1 -- 172.21.15.59:0/2229458850 shutdown_connections 2024-06-29T15:57:28.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.043+0000 7f40145bd640 1 -- 172.21.15.59:0/2229458850 >> 172.21.15.59:0/2229458850 conn(0x7f400c06c950 msgr2=0x7f400c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.043+0000 7f40145bd640 1 -- 172.21.15.59:0/2229458850 shutdown_connections 2024-06-29T15:57:28.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.043+0000 7f40145bd640 1 -- 172.21.15.59:0/2229458850 wait complete. 2024-06-29T15:57:28.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.044+0000 7f40145bd640 1 Processor -- start 2024-06-29T15:57:28.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.045+0000 7f40145bd640 1 -- start start 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f40145bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ff4004800 0x7f400c1e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f40145bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c1e88a0 0x7f400c0a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f40145bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f400c150770 0x7f400c0a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f40145bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f400c0b5c10 con 0x7f400c150770 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f40145bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f400c0b5a90 con 0x7f400c1e88a0 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f40145bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f400c0b5d90 con 0x7f3ff4004800 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f400c150770 0x7f400c0a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f400c150770 0x7f400c0a99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46070/0 (socket says 172.21.15.59:46070) 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012b5a640 1 -- 172.21.15.59:0/1464523506 learned_addr learned my addr 172.21.15.59:0/1464523506 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012359640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ff4004800 0x7f400c1e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4011b58640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c1e88a0 0x7f400c0a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012b5a640 1 -- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ff4004800 msgr2=0x7f400c1e8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012b5a640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ff4004800 0x7f400c1e8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012b5a640 1 -- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c1e88a0 msgr2=0x7f400c0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012b5a640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c1e88a0 0x7f400c0a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4012b5a640 1 -- 172.21.15.59:0/1464523506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400c0a9f20 con 0x7f400c150770 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.046+0000 7f4011b58640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c1e88a0 0x7f400c0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.047+0000 7f4012b5a640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f400c150770 0x7f400c0a99e0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f400c0b5030 tx=0x7f4004011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.047+0000 7f400affd640 1 -- 172.21.15.59:0/1464523506 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004004230 con 0x7f400c150770 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.047+0000 7f400affd640 1 -- 172.21.15.59:0/1464523506 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4004016070 con 0x7f400c150770 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.047+0000 7f400affd640 1 -- 172.21.15.59:0/1464523506 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400400fa90 con 0x7f400c150770 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.047+0000 7f40145bd640 1 -- 172.21.15.59:0/1464523506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400c1f8420 con 0x7f400c150770 2024-06-29T15:57:28.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.047+0000 7f40145bd640 1 -- 172.21.15.59:0/1464523506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f400c1f89e0 con 0x7f400c150770 2024-06-29T15:57:28.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.049+0000 7f400affd640 1 -- 172.21.15.59:0/1464523506 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f400400fc30 con 0x7f400c150770 2024-06-29T15:57:28.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.049+0000 7f400affd640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3ff80763f0 0x7f3ff8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.050+0000 7f400affd640 1 -- 172.21.15.59:0/1464523506 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f4004098300 con 0x7f400c150770 2024-06-29T15:57:28.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.050+0000 7f4012359640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3ff80763f0 0x7f3ff8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.050+0000 7f401b47b740 1 Processor -- start 2024-06-29T15:57:28.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.050+0000 7f401b47b740 1 -- start start 2024-06-29T15:57:28.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.050+0000 7f4012359640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3ff80763f0 0x7f3ff8078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc001be0 tx=0x7f3ffc006040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.051+0000 7f401b47b740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x564444814270 0x564444c006e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.051+0000 7f401b47b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 0x564444be43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f401b47b740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447f3590 0x564444be48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f401b47b740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x564444b7cbc0 con 0x5644447f3590 2024-06-29T15:57:28.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f401b47b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564444b2b630 con 0x5644447eb3b0 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f401b47b740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x564444b1abd0 con 0x564444814270 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447f3590 0x564444be48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447f3590 0x564444be48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46080/0 (socket says 172.21.15.59:46080) 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 -- 172.21.15.59:0/4158679991 learned_addr learned my addr 172.21.15.59:0/4158679991 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012359640 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x564444814270 0x564444c006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4011b58640 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 0x564444be43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 -- 172.21.15.59:0/4158679991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x564444814270 msgr2=0x564444c006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x564444814270 0x564444c006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 -- 172.21.15.59:0/4158679991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 msgr2=0x564444be43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 0x564444be43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 -- 172.21.15.59:0/4158679991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564444b17ca0 con 0x5644447f3590 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f4012b5a640 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447f3590 0x564444be48e0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f4004096550 tx=0x7f4004060ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d9355d371ea46 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f3ff3fff640 1 -- 172.21.15.59:0/4158679991 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004094070 con 0x5644447f3590 2024-06-29T15:57:28.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f3ff3fff640 1 -- 172.21.15.59:0/4158679991 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4004061290 con 0x5644447f3590 2024-06-29T15:57:28.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.052+0000 7f3ff3fff640 1 -- 172.21.15.59:0/4158679991 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004065640 con 0x5644447f3590 2024-06-29T15:57:28.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 -- 172.21.15.59:0/4158679991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447f3590 msgr2=0x564444be48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447f3590 0x564444be48e0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f4004096550 tx=0x7f4004060ca0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 -- 172.21.15.59:0/4158679991 shutdown_connections 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447f3590 0x564444be48e0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 0x564444be43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 --2- 172.21.15.59:0/4158679991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x564444814270 0x564444c006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 -- 172.21.15.59:0/4158679991 >> 172.21.15.59:0/4158679991 conn(0x5644448083c0 msgr2=0x564444bb7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 -- 172.21.15.59:0/4158679991 shutdown_connections 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.053+0000 7f401b47b740 1 -- 172.21.15.59:0/4158679991 wait complete. 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.054+0000 7f401b47b740 1 -- start start 2024-06-29T15:57:28.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.054+0000 7f401b47b740 1 Processor -- start 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f401b47b740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447eb3b0 0x564444bdc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f401b47b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447f3590 0x564444bf69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f401b47b740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444c12110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f401b47b740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x564444ab8ae0 con 0x564444814270 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f401b47b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564444b1ec10 con 0x5644447f3590 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f401b47b740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x56444488e620 con 0x5644447eb3b0 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f401b47b740 10 client.0 _ll_register_callbacks cb 0x564444ac5480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444c12110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444c12110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46084/0 (socket says 172.21.15.59:46084) 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012b5a640 1 -- 172.21.15.59:0/3040638699 learned_addr learned my addr 172.21.15.59:0/3040638699 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012359640 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447eb3b0 0x564444bdc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4011b58640 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447f3590 0x564444bf69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012b5a640 1 -- 172.21.15.59:0/3040638699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447eb3b0 msgr2=0x564444bdc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012b5a640 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447eb3b0 0x564444bdc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012b5a640 1 -- 172.21.15.59:0/3040638699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447f3590 msgr2=0x564444bf69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012b5a640 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447f3590 0x564444bf69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.056+0000 7f4012b5a640 1 -- 172.21.15.59:0/3040638699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56444487f850 con 0x564444814270 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f4012b5a640 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444c12110 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f40040965a0 tx=0x7f400405d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb95a6066d756c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f3ff17fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f3ff17fa640 1 -- 172.21.15.59:0/3040638699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004094070 con 0x564444814270 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f3ff17fa640 1 -- 172.21.15.59:0/3040638699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f400405cc00 con 0x564444814270 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 1 -- 172.21.15.59:0/3040638699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4004061290 con 0x564444814270 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f3ff17fa640 1 -- 172.21.15.59:0/3040638699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f4004065640 con 0x564444814270 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 10 client.11625 fetch_fsmap learned FSMap version 115 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 1 -- 172.21.15.59:0/3040638699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) -- 0x5644448f66b0 con 0x564444814270 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f3ff17fa640 1 -- 172.21.15.59:0/3040638699 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4004061290 con 0x564444814270 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f3ff17fa640 1 -- 172.21.15.59:0/3040638699 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 115) ==== 3502+0+0 (secure 0 0 0) 0x7f4004023030 con 0x564444814270 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 10 client.11625 fetch_fsmap finished waiting for FSMap version 115 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 10 client.11625 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 20 client.11625 populate_metadata read hostname 'smithi059' 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] conn(0x5644447d2e40 0x564444be35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 4 client.11625 mds_command: new command op to 11487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f401b47b740 1 -- 172.21.15.59:0/3040638699 --> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564444b2d580 con 0x5644447d2e40 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.057+0000 7f4012359640 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] conn(0x5644447d2e40 0x564444be35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.058+0000 7f3ff17fa640 10 client.11625 ms_handle_connect on v2:172.21.15.59:6832/1734341893 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.058+0000 7f4012359640 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] conn(0x5644447d2e40 0x564444be35b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b39529a5bd70c162 server_cookie=8fb8ff6eb631c068 in_seq=0 out_seq=0 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.058+0000 7f3ff17fa640 1 -- 172.21.15.59:0/3040638699 <== mds.0 v2:172.21.15.59:6832/1734341893 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564444b2d580 con 0x5644447d2e40 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.058+0000 7f3ff17fa640 10 client.11625 handle_command_reply: tid=0 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.058+0000 7f401b47b740 1 client.11625 shutdown 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 -- 172.21.15.59:0/3040638699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 msgr2=0x564444c12110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444c12110 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f40040965a0 tx=0x7f400405d7d0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 -- 172.21.15.59:0/3040638699 shutdown_connections 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] conn(0x5644447d2e40 0x564444be35b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444c12110 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f3ff17fa640 0 client.11625 ms_handle_reset on v2:172.21.15.59:6832/1734341893 2024-06-29T15:57:28.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447f3590 0x564444bf69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 --2- 172.21.15.59:0/3040638699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447eb3b0 0x564444bdc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 -- 172.21.15.59:0/3040638699 >> 172.21.15.59:0/3040638699 conn(0x5644448083c0 msgr2=0x564444c01310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 -- 172.21.15.59:0/3040638699 shutdown_connections 2024-06-29T15:57:28.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 1 -- 172.21.15.59:0/3040638699 wait complete. 2024-06-29T15:57:28.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.059+0000 7f401b47b740 20 client.11625 trim_cache size 0 max 16384 2024-06-29T15:57:28.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.078+0000 7f401b47b740 1 Processor -- start 2024-06-29T15:57:28.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.078+0000 7f401b47b740 1 -- start start 2024-06-29T15:57:28.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.079+0000 7f401b47b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 0x564444c21d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f401b47b740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 0x564444bdbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f401b47b740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444be3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f401b47b740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f400c200d70 con 0x564444814270 2024-06-29T15:57:28.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f401b47b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f400c2000d0 con 0x5644447eb3b0 2024-06-29T15:57:28.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f401b47b740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f400c200720 con 0x5644447f3590 2024-06-29T15:57:28.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444be3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444be3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46088/0 (socket says 172.21.15.59:46088) 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 -- 172.21.15.59:0/1127263809 learned_addr learned my addr 172.21.15.59:0/1127263809 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012359640 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 0x564444c21d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4011b58640 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 0x564444bdbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 -- 172.21.15.59:0/1127263809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 msgr2=0x564444bdbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 0x564444bdbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 -- 172.21.15.59:0/1127263809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 msgr2=0x564444c21d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 0x564444c21d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 -- 172.21.15.59:0/1127263809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564444bbb1e0 con 0x564444814270 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f4012b5a640 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444be3580 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f400405da50 tx=0x7f4004004520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e4074791a40425 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f3ff37fe640 1 -- 172.21.15.59:0/1127263809 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004094040 con 0x564444814270 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f3ff37fe640 1 -- 172.21.15.59:0/1127263809 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4004061290 con 0x564444814270 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.080+0000 7f3ff37fe640 1 -- 172.21.15.59:0/1127263809 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400405ce50 con 0x564444814270 2024-06-29T15:57:28.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 -- 172.21.15.59:0/1127263809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 msgr2=0x564444be3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444be3580 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f400405da50 tx=0x7f4004004520 comp rx=0 tx=0).stop 2024-06-29T15:57:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 -- 172.21.15.59:0/1127263809 shutdown_connections 2024-06-29T15:57:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x564444814270 0x564444be3580 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 0x564444bdbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 --2- 172.21.15.59:0/1127263809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5644447eb3b0 0x564444c21d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 -- 172.21.15.59:0/1127263809 >> 172.21.15.59:0/1127263809 conn(0x5644448083c0 msgr2=0x564444bf57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 -- 172.21.15.59:0/1127263809 shutdown_connections 2024-06-29T15:57:28.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.081+0000 7f401b47b740 1 -- 172.21.15.59:0/1127263809 wait complete. 2024-06-29T15:57:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.082+0000 7f401b47b740 1 -- start start 2024-06-29T15:57:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.082+0000 7f401b47b740 1 Processor -- start 2024-06-29T15:57:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.083+0000 7f401b47b740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447eb3b0 0x564444c7b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.083+0000 7f401b47b740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 0x564444bb51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.083+0000 7f401b47b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564444814270 0x564444bb5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.083+0000 7f401b47b740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x564444ab8ae0 con 0x5644447eb3b0 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.083+0000 7f401b47b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564444b1ec10 con 0x564444814270 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.083+0000 7f401b47b740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x564444bf6f10 con 0x5644447f3590 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.083+0000 7f401b47b740 10 client.0 _ll_register_callbacks cb 0x7f4004092070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447eb3b0 0x564444c7b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447eb3b0 0x564444c7b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46102/0 (socket says 172.21.15.59:46102) 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 -- 172.21.15.59:0/2238090655 learned_addr learned my addr 172.21.15.59:0/2238090655 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4011b58640 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 0x564444bb51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012b5a640 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564444814270 0x564444bb5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 -- 172.21.15.59:0/2238090655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 msgr2=0x564444bb51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 0x564444bb51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 -- 172.21.15.59:0/2238090655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564444814270 msgr2=0x564444bb5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564444814270 0x564444bb5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 -- 172.21.15.59:0/2238090655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564444bda4d0 con 0x5644447eb3b0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f4012359640 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447eb3b0 0x564444c7b470 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x564444bf7260 tx=0x7f3ffc014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b518fdb82323bce server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f3ff17fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f3ff17fa640 1 -- 172.21.15.59:0/2238090655 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc012500 con 0x5644447eb3b0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f3ff17fa640 1 -- 172.21.15.59:0/2238090655 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f3ffc021cb0 con 0x5644447eb3b0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f401b47b740 1 -- 172.21.15.59:0/2238090655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4004061290 con 0x5644447eb3b0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.084+0000 7f3ff17fa640 1 -- 172.21.15.59:0/2238090655 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f3ffc039050 con 0x5644447eb3b0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f401b47b740 10 client.11631 fetch_fsmap learned FSMap version 115 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f401b47b740 1 -- 172.21.15.59:0/2238090655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) -- 0x564444bda6e0 con 0x5644447eb3b0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f3ff17fa640 1 -- 172.21.15.59:0/2238090655 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4004061290 con 0x5644447eb3b0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f3ff17fa640 1 -- 172.21.15.59:0/2238090655 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 115) ==== 3502+0+0 (secure 0 0 0) 0x7f3ffc03c020 con 0x5644447eb3b0 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f401b47b740 10 client.11631 fetch_fsmap finished waiting for FSMap version 115 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f401b47b740 10 client.11631 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f401b47b740 20 client.11631 populate_metadata read hostname 'smithi059' 2024-06-29T15:57:28.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f401b47b740 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] conn(0x564444c20e60 0x564444be1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f401b47b740 4 client.11631 mds_command: new command op to 11487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f401b47b740 1 -- 172.21.15.59:0/2238090655 --> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] -- command(tid 0: {"prefix": "status"}) -- 0x564444b29020 con 0x564444c20e60 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.085+0000 7f4011b58640 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] conn(0x564444c20e60 0x564444be1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f3ff17fa640 10 client.11631 ms_handle_connect on v2:172.21.15.59:6832/1734341893 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f4011b58640 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] conn(0x564444c20e60 0x564444be1440 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b02c4ab8170c8316 server_cookie=f3f0506ffca45cbb in_seq=0 out_seq=0 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f3ff17fa640 1 -- 172.21.15.59:0/2238090655 <== mds.0 v2:172.21.15.59:6832/1734341893 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x564444b29020 con 0x564444c20e60 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f3ff17fa640 10 client.11631 handle_command_reply: tid=0 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 client.11631 shutdown 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 -- 172.21.15.59:0/2238090655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447eb3b0 msgr2=0x564444c7b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447eb3b0 0x564444c7b470 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x564444bf7260 tx=0x7f3ffc014070 comp rx=0 tx=0).stop 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 -- 172.21.15.59:0/2238090655 shutdown_connections 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564444814270 0x564444bb5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5644447f3590 0x564444bb51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.59:6832/1734341893,v1:172.21.15.59:6834/1734341893] conn(0x564444c20e60 0x564444be1440 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 --2- 172.21.15.59:0/2238090655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5644447eb3b0 0x564444c7b470 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f3ff17fa640 0 client.11631 ms_handle_reset on v2:172.21.15.59:6832/1734341893 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 -- 172.21.15.59:0/2238090655 >> 172.21.15.59:0/2238090655 conn(0x5644448083c0 msgr2=0x564444bf67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 -- 172.21.15.59:0/2238090655 shutdown_connections 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 1 -- 172.21.15.59:0/2238090655 wait complete. 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.086+0000 7f401b47b740 20 client.11631 trim_cache size 0 max 16384 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 11487, 2024-06-29T15:57:28.092 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:57:28.093 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:57:28.093 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T15:57:28.093 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8822569920000003, 2024-06-29T15:57:28.093 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 115, 2024-06-29T15:57:28.093 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 222, 2024-06-29T15:57:28.093 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 222, 2024-06-29T15:57:28.093 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.494339896 2024-06-29T15:57:28.093 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 -- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3ff80763f0 msgr2=0x7f3ff8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3ff80763f0 0x7f3ff8078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc001be0 tx=0x7f3ffc006040 comp rx=0 tx=0).stop 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 -- 172.21.15.59:0/1464523506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f400c150770 msgr2=0x7f400c0a99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f400c150770 0x7f400c0a99e0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f400c0b5030 tx=0x7f4004011b10 comp rx=0 tx=0).stop 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 -- 172.21.15.59:0/1464523506 shutdown_connections 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f3ff80763f0 0x7f3ff8078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f400c150770 0x7f400c0a99e0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f400c1e88a0 0x7f400c0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 --2- 172.21.15.59:0/1464523506 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ff4004800 0x7f400c1e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 -- 172.21.15.59:0/1464523506 >> 172.21.15.59:0/1464523506 conn(0x7f400c06c950 msgr2=0x7f400c143200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 -- 172.21.15.59:0/1464523506 shutdown_connections 2024-06-29T15:57:28.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.087+0000 7f40145bd640 1 -- 172.21.15.59:0/1464523506 wait complete. 2024-06-29T15:57:28.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:57:28.184 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T15:57:28.184+0000 7fde07262640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:57:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.216+0000 7f9a7c44e640 1 -- 172.21.15.59:0/1027662304 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a7415e0a0 msgr2=0x7f9a740ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.216+0000 7f9a7c44e640 1 --2- 172.21.15.59:0/1027662304 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a7415e0a0 0x7f9a740ce550 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009be0 tx=0x7f9a6c02a820 comp rx=0 tx=0).stop 2024-06-29T15:57:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.216+0000 7f9a7c44e640 1 -- 172.21.15.59:0/1027662304 shutdown_connections 2024-06-29T15:57:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.216+0000 7f9a7c44e640 1 --2- 172.21.15.59:0/1027662304 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a7415e0a0 0x7f9a740ce550 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.216+0000 7f9a7c44e640 1 -- 172.21.15.59:0/1027662304 >> 172.21.15.59:0/1027662304 conn(0x7f9a740d45b0 msgr2=0x7f9a740d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.216+0000 7f9a7c44e640 1 -- 172.21.15.59:0/1027662304 shutdown_connections 2024-06-29T15:57:28.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.216+0000 7f9a7c44e640 1 -- 172.21.15.59:0/1027662304 wait complete. 2024-06-29T15:57:28.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.218+0000 7f9a7c44e640 1 Processor -- start 2024-06-29T15:57:28.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.218+0000 7f9a7c44e640 1 -- start start 2024-06-29T15:57:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a7c44e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a7415e0a0 0x7f9a74159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a7c44e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a74162770 0x7f9a741f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a7c44e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a74151490 0x7f9a741f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a7c44e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9a740b4540 con 0x7f9a74162770 2024-06-29T15:57:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a7c44e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a740b43c0 con 0x7f9a7415e0a0 2024-06-29T15:57:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a7c44e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9a740b46c0 con 0x7f9a74151490 2024-06-29T15:57:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a799e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a74162770 0x7f9a741f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a799e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a74162770 0x7f9a741f7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46114/0 (socket says 172.21.15.59:46114) 2024-06-29T15:57:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.219+0000 7f9a799e9640 1 -- 172.21.15.59:0/3887803413 learned_addr learned my addr 172.21.15.59:0/3887803413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a7a9eb640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a74151490 0x7f9a741f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a7a1ea640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a7415e0a0 0x7f9a74159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a799e9640 1 -- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a74151490 msgr2=0x7f9a741f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a799e9640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a74151490 0x7f9a741f8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a799e9640 1 -- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a7415e0a0 msgr2=0x7f9a74159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a799e9640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a7415e0a0 0x7f9a74159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a799e9640 1 -- 172.21.15.59:0/3887803413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7415a230 con 0x7f9a74162770 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a799e9640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a74162770 0x7f9a741f7be0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400c390 tx=0x7f9a6400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a6affd640 1 -- 172.21.15.59:0/3887803413 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64017070 con 0x7f9a74162770 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7415a540 con 0x7f9a74162770 2024-06-29T15:57:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a6affd640 1 -- 172.21.15.59:0/3887803413 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a640053c0 con 0x7f9a74162770 2024-06-29T15:57:28.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a6affd640 1 -- 172.21.15.59:0/3887803413 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64003850 con 0x7f9a74162770 2024-06-29T15:57:28.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.220+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a7415ab00 con 0x7f9a74162770 2024-06-29T15:57:28.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.221+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a740b4b70 con 0x7f9a74162770 2024-06-29T15:57:28.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.222+0000 7f9a6affd640 1 -- 172.21.15.59:0/3887803413 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f9a6401a070 con 0x7f9a74162770 2024-06-29T15:57:28.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.223+0000 7f9a6affd640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9a580763f0 0x7f9a58078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.223+0000 7f9a6affd640 1 -- 172.21.15.59:0/3887803413 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f9a64097070 con 0x7f9a74162770 2024-06-29T15:57:28.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.223+0000 7f9a7a1ea640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9a580763f0 0x7f9a58078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.223+0000 7f9a7a1ea640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9a580763f0 0x7f9a58078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a70001e70 tx=0x7f9a70009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.227+0000 7f9a6affd640 1 -- 172.21.15.59:0/3887803413 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a640605f0 con 0x7f9a74162770 2024-06-29T15:57:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.456+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9a740b2b60 con 0x7f9a74162770 2024-06-29T15:57:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.457+0000 7f9a6affd640 1 -- 172.21.15.59:0/3887803413 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v115) ==== 53+0+73 (secure 0 0 0) 0x7f9a6405fef0 con 0x7f9a74162770 2024-06-29T15:57:28.458 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9a580763f0 msgr2=0x7f9a58078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9a580763f0 0x7f9a58078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a70001e70 tx=0x7f9a70009040 comp rx=0 tx=0).stop 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a74162770 msgr2=0x7f9a741f7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a74162770 0x7f9a741f7be0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400c390 tx=0x7f9a6400c850 comp rx=0 tx=0).stop 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 shutdown_connections 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9a580763f0 0x7f9a58078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a74151490 0x7f9a741f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a74162770 0x7f9a741f7be0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 --2- 172.21.15.59:0/3887803413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a7415e0a0 0x7f9a74159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 >> 172.21.15.59:0/3887803413 conn(0x7f9a740d45b0 msgr2=0x7f9a74152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 shutdown_connections 2024-06-29T15:57:28.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28.459+0000 7f9a7c44e640 1 -- 172.21.15.59:0/3887803413 wait complete. 2024-06-29T15:57:28.470 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:57:28.471 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:57:28.471 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T15:57:28.471 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:57:28.471 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:28.471 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:57:28.471 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:57:28.471 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:57:28.471 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:57:28.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.472 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.488 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 37002sec preferred_lft 37002sec 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:57:28.489 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T15:57:28.490 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.491 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:57:28.552 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T15:57:28.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.554 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T15:57:28.610 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T15:57:28.610 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:57:28.611 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T15:57:28.611 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T15:57:28.611 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:57:28.611 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:57:28.611 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T15:57:28.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T15:57:28.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T15:57:28.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T15:57:28.611 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T15:57:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:57:28.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:57:28.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.740 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T15:57:28.757 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:57:28.757 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:57:28.757 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:57:28.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.758 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T15:57:28.824 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:57:28.824 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:28.880 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:57:28.880 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:57:28.880 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:28.938 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T15:57:28.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.939 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:57:29.034 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:57:29.034 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:57:29.034 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:57:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:57:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:57:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:57:29.045 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:57:29.046 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:57:29.050 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:57:29.050 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:57:29.050 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:57:29.051 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:57:29.052 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:57:29.053 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:57:29.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:29.053 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:29.080 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:57:29.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:57:29.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:57:29.182 INFO:teuthology.orchestra.run.smithi059.stdout:client11640 2024-06-29T15:57:29.182 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:29.182 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:29.199 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-06-29T15:57:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.360+0000 7f730ef0a640 1 -- 172.21.15.59:0/977896862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72f0005330 msgr2=0x7f72f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.360+0000 7f730ef0a640 1 --2- 172.21.15.59:0/977896862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72f0005330 0x7f72f000b9c0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c290 tx=0x7f72f0005830 comp rx=0 tx=0).stop 2024-06-29T15:57:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.360+0000 7f730d4a7640 1 -- 172.21.15.59:0/977896862 reap_dead start 2024-06-29T15:57:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.360+0000 7f730ef0a640 1 -- 172.21.15.59:0/977896862 shutdown_connections 2024-06-29T15:57:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.360+0000 7f730ef0a640 1 -- 172.21.15.59:0/977896862 >> 172.21.15.59:0/977896862 conn(0x7f73080b7690 msgr2=0x7f7308073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.360+0000 7f730ef0a640 1 -- 172.21.15.59:0/977896862 shutdown_connections 2024-06-29T15:57:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.360+0000 7f730ef0a640 1 -- 172.21.15.59:0/977896862 wait complete. 2024-06-29T15:57:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.362+0000 7f730ef0a640 1 Processor -- start 2024-06-29T15:57:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.362+0000 7f730ef0a640 1 -- start start 2024-06-29T15:57:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730ef0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72f0005330 0x7f730806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730ef0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f730806f870 0x7f73081f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730ef0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73080d0940 0x7f73080d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730ef0a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f73081598b0 con 0x7f73080d0940 2024-06-29T15:57:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730ef0a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73080b10d0 con 0x7f730806f870 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730ef0a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7308159a30 con 0x7f72f0005330 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73080d0940 0x7f73080d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73080d0940 0x7f73080d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46128/0 (socket says 172.21.15.59:46128) 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 -- 172.21.15.59:0/3617595229 learned_addr learned my addr 172.21.15.59:0/3617595229 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f7307fff640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f730806f870 0x7f73081f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730cca6640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72f0005330 0x7f730806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 -- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72f0005330 msgr2=0x7f730806e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72f0005330 0x7f730806e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 -- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f730806f870 msgr2=0x7f73081f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f730806f870 0x7f73081f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:29.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 -- 172.21.15.59:0/3617595229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73080d1290 con 0x7f73080d0940 2024-06-29T15:57:29.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f730d4a7640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73080d0940 0x7f73080d0d50 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c3b0 tx=0x7f72f8036c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:29.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.364+0000 7f73057fa640 1 -- 172.21.15.59:0/3617595229 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8031070 con 0x7f73080d0940 2024-06-29T15:57:29.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.365+0000 7f73057fa640 1 -- 172.21.15.59:0/3617595229 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72f8000ed0 con 0x7f73080d0940 2024-06-29T15:57:29.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.365+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730806ed80 con 0x7f73080d0940 2024-06-29T15:57:29.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.365+0000 7f73057fa640 1 -- 172.21.15.59:0/3617595229 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8006c30 con 0x7f73080d0940 2024-06-29T15:57:29.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.365+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73080adb00 con 0x7f73080d0940 2024-06-29T15:57:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.366+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73080b4e10 con 0x7f73080d0940 2024-06-29T15:57:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.366+0000 7f73057fa640 1 -- 172.21.15.59:0/3617595229 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f72f8036d10 con 0x7f73080d0940 2024-06-29T15:57:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.367+0000 7f73057fa640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f72f4076420 0x7f72f4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.367+0000 7f73057fa640 1 -- 172.21.15.59:0/3617595229 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f72f80b7570 con 0x7f73080d0940 2024-06-29T15:57:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.367+0000 7f730cca6640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f72f4076420 0x7f72f4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.368+0000 7f730cca6640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f72f4076420 0x7f72f4078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7300000f80 tx=0x7f7300006ef0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.371+0000 7f73057fa640 1 -- 172.21.15.59:0/3617595229 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73080d1290 con 0x7f73080d0940 2024-06-29T15:57:29.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:29.601+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f730806eb10 con 0x7f73080d0940 2024-06-29T15:57:30.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.261+0000 7f73057fa640 1 -- 172.21.15.59:0/3617595229 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v116) ==== 136+0+0 (secure 0 0 0) 0x7f72f8080530 con 0x7f73080d0940 2024-06-29T15:57:30.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f72f4076420 msgr2=0x7f72f4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f72f4076420 0x7f72f4078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7300000f80 tx=0x7f7300006ef0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73080d0940 msgr2=0x7f73080d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73080d0940 0x7f73080d0d50 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c3b0 tx=0x7f72f8036c20 comp rx=0 tx=0).stop 2024-06-29T15:57:30.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 shutdown_connections 2024-06-29T15:57:30.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f72f4076420 0x7f72f4078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73080d0940 0x7f73080d0d50 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f730806f870 0x7f73081f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 --2- 172.21.15.59:0/3617595229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f72f0005330 0x7f730806e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 >> 172.21.15.59:0/3617595229 conn(0x7f73080b7690 msgr2=0x7f7308068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 shutdown_connections 2024-06-29T15:57:30.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.264+0000 7f730ef0a640 1 -- 172.21.15.59:0/3617595229 wait complete. 2024-06-29T15:57:30.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:30.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.402+0000 7f93a2963640 1 -- 172.21.15.59:0/1463932965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3b50 msgr2=0x7f939c166ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.402+0000 7f93a2963640 1 --2- 172.21.15.59:0/1463932965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3b50 0x7f939c166ca0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f9394005fd0 tx=0x7f939402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.402+0000 7f93a2963640 1 -- 172.21.15.59:0/1463932965 shutdown_connections 2024-06-29T15:57:30.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.402+0000 7f93a2963640 1 --2- 172.21.15.59:0/1463932965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3b50 0x7f939c166ca0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.402+0000 7f93a2963640 1 -- 172.21.15.59:0/1463932965 >> 172.21.15.59:0/1463932965 conn(0x7f939c06c950 msgr2=0x7f939c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.402+0000 7f93a2963640 1 -- 172.21.15.59:0/1463932965 shutdown_connections 2024-06-29T15:57:30.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.402+0000 7f93a2963640 1 -- 172.21.15.59:0/1463932965 wait complete. 2024-06-29T15:57:30.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.404+0000 7f93a2963640 1 Processor -- start 2024-06-29T15:57:30.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.404+0000 7f93a2963640 1 -- start start 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f93a2963640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3740 0x7f939c140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f93a2963640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f939c0b3b50 0x7f939c15db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f93a2963640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c140cf0 0x7f939c15e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f93a2963640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f939c167570 con 0x7f939c0b3740 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f93a2963640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1673f0 con 0x7f939c140cf0 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f93a2963640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f939c1676f0 con 0x7f939c0b3b50 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f939bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3740 0x7f939c140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f939bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3740 0x7f939c140410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46142/0 (socket says 172.21.15.59:46142) 2024-06-29T15:57:30.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.405+0000 7f939bfff640 1 -- 172.21.15.59:0/2961587736 learned_addr learned my addr 172.21.15.59:0/2961587736 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f93a0f00640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c140cf0 0x7f939c15e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f939b7fe640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f939c0b3b50 0x7f939c15db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f939bfff640 1 -- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f939c0b3b50 msgr2=0x7f939c15db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f939bfff640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f939c0b3b50 0x7f939c15db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f939bfff640 1 -- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c140cf0 msgr2=0x7f939c15e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f939bfff640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c140cf0 0x7f939c15e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f939bfff640 1 -- 172.21.15.59:0/2961587736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c0cef90 con 0x7f939c0b3740 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f939bfff640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3740 0x7f939c140410 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f938c00b420 tx=0x7f938c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f9398ff9640 1 -- 172.21.15.59:0/2961587736 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c011830 con 0x7f939c0b3740 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f9398ff9640 1 -- 172.21.15.59:0/2961587736 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f938c016070 con 0x7f939c0b3740 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939c1434d0 con 0x7f939c0b3740 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f9398ff9640 1 -- 172.21.15.59:0/2961587736 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c011d60 con 0x7f939c0b3740 2024-06-29T15:57:30.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.406+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939c143a90 con 0x7f939c0b3740 2024-06-29T15:57:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.407+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f939c1458a0 con 0x7f939c0b3740 2024-06-29T15:57:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.408+0000 7f9398ff9640 1 -- 172.21.15.59:0/2961587736 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f938c01b460 con 0x7f939c0b3740 2024-06-29T15:57:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.409+0000 7f9398ff9640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f93880763f0 0x7f9388078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.409+0000 7f9398ff9640 1 -- 172.21.15.59:0/2961587736 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f938c097650 con 0x7f939c0b3740 2024-06-29T15:57:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.409+0000 7f939b7fe640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f93880763f0 0x7f9388078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.410+0000 7f939b7fe640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f93880763f0 0x7f9388078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9390000f60 tx=0x7f9390009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:30.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.413+0000 7f9398ff9640 1 -- 172.21.15.59:0/2961587736 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f938c020020 con 0x7f939c0b3740 2024-06-29T15:57:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.607+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f939c0d1f60 con 0x7f939c0b3740 2024-06-29T15:57:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.608+0000 7f9398ff9640 1 -- 172.21.15.59:0/2961587736 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) ==== 74+0+17054 (secure 0 0 0) 0x7f938c060c40 con 0x7f939c0b3740 2024-06-29T15:57:30.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:30.609 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":222,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:23.218941+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f93880763f0 msgr2=0x7f9388078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f93880763f0 0x7f9388078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9390000f60 tx=0x7f9390009040 comp rx=0 tx=0).stop 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3740 msgr2=0x7f939c140410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3740 0x7f939c140410 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f938c00b420 tx=0x7f938c00b8e0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 shutdown_connections 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f93880763f0 0x7f9388078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c140cf0 0x7f939c15e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f939c0b3b50 0x7f939c15db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 --2- 172.21.15.59:0/2961587736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f939c0b3740 0x7f939c140410 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 >> 172.21.15.59:0/2961587736 conn(0x7f939c06c950 msgr2=0x7f939c15ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 shutdown_connections 2024-06-29T15:57:30.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.610+0000 7f93a2963640 1 -- 172.21.15.59:0/2961587736 wait complete. 2024-06-29T15:57:30.623 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:57:19.955891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '221', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:57:20.545869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:30.623 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:30.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.744+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/900164244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 msgr2=0x7f7fa8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.744+0000 7f7fc6fbd640 1 --2- 172.21.15.59:0/900164244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 0x7f7fa8003f30 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00b5480 tx=0x7f7fa8005830 comp rx=0 tx=0).stop 2024-06-29T15:57:30.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.744+0000 7f7fc555a640 1 -- 172.21.15.59:0/900164244 reap_dead start 2024-06-29T15:57:30.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.744+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/900164244 shutdown_connections 2024-06-29T15:57:30.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.744+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/900164244 >> 172.21.15.59:0/900164244 conn(0x7f7fc00d45b0 msgr2=0x7f7fc00d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.744+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/900164244 shutdown_connections 2024-06-29T15:57:30.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.744+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/900164244 wait complete. 2024-06-29T15:57:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.745+0000 7f7fc6fbd640 1 Processor -- start 2024-06-29T15:57:30.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.746+0000 7f7fc6fbd640 1 -- start start 2024-06-29T15:57:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc6fbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 0x7f7fc014d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc6fbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc00c8880 0x7f7fc00c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc6fbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fc0161760 0x7f7fc015ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc6fbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc00ce6d0 con 0x7f7fa8003ac0 2024-06-29T15:57:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc6fbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc00ce550 con 0x7f7fc00c8880 2024-06-29T15:57:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc6fbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7fc00ce850 con 0x7f7fc0161760 2024-06-29T15:57:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc4d59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 0x7f7fc014d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc4d59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 0x7f7fc014d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46164/0 (socket says 172.21.15.59:46164) 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc4d59640 1 -- 172.21.15.59:0/701864778 learned_addr learned my addr 172.21.15.59:0/701864778 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc555a640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fc0161760 0x7f7fc015ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc4d59640 1 -- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fc0161760 msgr2=0x7f7fc015ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fbffff640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc00c8880 0x7f7fc00c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc4d59640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fc0161760 0x7f7fc015ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc4d59640 1 -- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc00c8880 msgr2=0x7f7fc00c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc4d59640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc00c8880 0x7f7fc00c7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc4d59640 1 -- 172.21.15.59:0/701864778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc015e920 con 0x7f7fa8003ac0 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.747+0000 7f7fc555a640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fc0161760 0x7f7fc015ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.748+0000 7f7fc4d59640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 0x7f7fc014d020 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000f80 tx=0x7f7fb8004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.748+0000 7f7fbd7fa640 1 -- 172.21.15.59:0/701864778 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8042070 con 0x7f7fa8003ac0 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.748+0000 7f7fbd7fa640 1 -- 172.21.15.59:0/701864778 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fb803a070 con 0x7f7fa8003ac0 2024-06-29T15:57:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.748+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc0160c80 con 0x7f7fa8003ac0 2024-06-29T15:57:30.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.748+0000 7f7fbd7fa640 1 -- 172.21.15.59:0/701864778 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8004e40 con 0x7f7fa8003ac0 2024-06-29T15:57:30.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.748+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc015f180 con 0x7f7fa8003ac0 2024-06-29T15:57:30.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.749+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc00b38e0 con 0x7f7fa8003ac0 2024-06-29T15:57:30.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.750+0000 7f7fbd7fa640 1 -- 172.21.15.59:0/701864778 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f7fb8045070 con 0x7f7fa8003ac0 2024-06-29T15:57:30.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.750+0000 7f7fbd7fa640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7fac0763f0 0x7f7fac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.750+0000 7f7fbd7fa640 1 -- 172.21.15.59:0/701864778 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f7fb80b78d0 con 0x7f7fa8003ac0 2024-06-29T15:57:30.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.751+0000 7f7fbffff640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7fac0763f0 0x7f7fac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.751+0000 7f7fbffff640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7fac0763f0 0x7f7fac078900 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00b7b20 tx=0x7f7fb4005f90 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:30.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.755+0000 7f7fbd7fa640 1 -- 172.21.15.59:0/701864778 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7fb8080e30 con 0x7f7fa8003ac0 2024-06-29T15:57:30.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.949+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7fc00d0710 con 0x7f7fa8003ac0 2024-06-29T15:57:30.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.949+0000 7f7fbd7fa640 1 -- 172.21.15.59:0/701864778 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v222) ==== 93+0+10 (secure 0 0 0) 0x7f7fb8080730 con 0x7f7fa8003ac0 2024-06-29T15:57:30.950 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:57:30.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7fac0763f0 msgr2=0x7f7fac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7fac0763f0 0x7f7fac078900 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00b7b20 tx=0x7f7fb4005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:30.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 msgr2=0x7f7fc014d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 0x7f7fc014d020 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000f80 tx=0x7f7fb8004c50 comp rx=0 tx=0).stop 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 shutdown_connections 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7fac0763f0 0x7f7fac078900 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7fc0161760 0x7f7fc015ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fc00c8880 0x7f7fc00c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 --2- 172.21.15.59:0/701864778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fa8003ac0 0x7f7fc014d020 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 >> 172.21.15.59:0/701864778 conn(0x7f7fc00d45b0 msgr2=0x7f7fc015ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 shutdown_connections 2024-06-29T15:57:30.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:30.951+0000 7f7fc6fbd640 1 -- 172.21.15.59:0/701864778 wait complete. 2024-06-29T15:57:30.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.085+0000 7f0549ba1640 1 -- 172.21.15.59:0/188918153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f052c005150 msgr2=0x7f052c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.085+0000 7f0549ba1640 1 --2- 172.21.15.59:0/188918153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f052c005150 0x7f052c00b9c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f054413f300 tx=0x7f052c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.085+0000 7f0543fff640 1 -- 172.21.15.59:0/188918153 reap_dead start 2024-06-29T15:57:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.085+0000 7f0549ba1640 1 -- 172.21.15.59:0/188918153 shutdown_connections 2024-06-29T15:57:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.085+0000 7f0549ba1640 1 -- 172.21.15.59:0/188918153 >> 172.21.15.59:0/188918153 conn(0x7f05440b6fb0 msgr2=0x7f05440b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.085+0000 7f0549ba1640 1 -- 172.21.15.59:0/188918153 shutdown_connections 2024-06-29T15:57:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.085+0000 7f0549ba1640 1 -- 172.21.15.59:0/188918153 wait complete. 2024-06-29T15:57:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.086+0000 7f0549ba1640 1 Processor -- start 2024-06-29T15:57:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.087+0000 7f0549ba1640 1 -- start start 2024-06-29T15:57:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0549ba1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05441320b0 0x7f0544153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0549ba1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05441544c0 0x7f0544158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0549ba1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0544159120 0x7f054415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0549ba1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f054413ef40 con 0x7f05441544c0 2024-06-29T15:57:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0549ba1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f054413edc0 con 0x7f05441320b0 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0549ba1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f054413f0c0 con 0x7f0544159120 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0542ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05441544c0 0x7f0544158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0542ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05441544c0 0x7f0544158be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46180/0 (socket says 172.21.15.59:46180) 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0542ffd640 1 -- 172.21.15.59:0/2765059307 learned_addr learned my addr 172.21.15.59:0/2765059307 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f05437fe640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05441320b0 0x7f0544153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0543fff640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0544159120 0x7f054415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0542ffd640 1 -- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0544159120 msgr2=0x7f054415ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0542ffd640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0544159120 0x7f054415ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0542ffd640 1 -- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05441320b0 msgr2=0x7f0544153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0542ffd640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05441320b0 0x7f0544153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0542ffd640 1 -- 172.21.15.59:0/2765059307 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0544159530 con 0x7f05441544c0 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.088+0000 7f0543fff640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0544159120 0x7f054415ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.089+0000 7f0542ffd640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05441544c0 0x7f0544158be0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f05380066b0 tx=0x7f0538003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.089+0000 7f052bfff640 1 -- 172.21.15.59:0/2765059307 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538015070 con 0x7f05441544c0 2024-06-29T15:57:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.089+0000 7f052bfff640 1 -- 172.21.15.59:0/2765059307 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f053800f9a0 con 0x7f05441544c0 2024-06-29T15:57:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.089+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0544156660 con 0x7f05441544c0 2024-06-29T15:57:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.089+0000 7f052bfff640 1 -- 172.21.15.59:0/2765059307 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053800fdf0 con 0x7f05441544c0 2024-06-29T15:57:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.089+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0544154dd0 con 0x7f05441544c0 2024-06-29T15:57:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.090+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f054406a7c0 con 0x7f05441544c0 2024-06-29T15:57:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.091+0000 7f052bfff640 1 -- 172.21.15.59:0/2765059307 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f0538011430 con 0x7f05441544c0 2024-06-29T15:57:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.091+0000 7f052bfff640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f05300763f0 0x7f0530078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.092+0000 7f052bfff640 1 -- 172.21.15.59:0/2765059307 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f0538098900 con 0x7f05441544c0 2024-06-29T15:57:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.092+0000 7f05437fe640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f05300763f0 0x7f0530078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.092+0000 7f05437fe640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f05300763f0 0x7f0530078900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0544153540 tx=0x7f0534005f90 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.096+0000 7f052bfff640 1 -- 172.21.15.59:0/2765059307 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f054406a7c0 con 0x7f05441544c0 2024-06-29T15:57:31.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.286+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0544154fa0 con 0x7f05441544c0 2024-06-29T15:57:31.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.286+0000 7f052bfff640 1 -- 172.21.15.59:0/2765059307 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v222) ==== 104+0+11 (secure 0 0 0) 0x7f0544159530 con 0x7f05441544c0 2024-06-29T15:57:31.287 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:57:31.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f05300763f0 msgr2=0x7f0530078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f05300763f0 0x7f0530078900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0544153540 tx=0x7f0534005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:31.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05441544c0 msgr2=0x7f0544158be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05441544c0 0x7f0544158be0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f05380066b0 tx=0x7f0538003aa0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 shutdown_connections 2024-06-29T15:57:31.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f05300763f0 0x7f0530078900 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0544159120 0x7f054415ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05441544c0 0x7f0544158be0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 --2- 172.21.15.59:0/2765059307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05441320b0 0x7f0544153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 >> 172.21.15.59:0/2765059307 conn(0x7f05440b6fb0 msgr2=0x7f05440ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 shutdown_connections 2024-06-29T15:57:31.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.288+0000 7f0549ba1640 1 -- 172.21.15.59:0/2765059307 wait complete. 2024-06-29T15:57:31.300 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:31.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.420+0000 7ff4373e4640 1 -- 172.21.15.59:0/3354360763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162190 msgr2=0x7ff4300b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.420+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3354360763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162190 0x7ff4300b38e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7ff428005fd0 tx=0x7ff42802a810 comp rx=0 tx=0).stop 2024-06-29T15:57:31.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.420+0000 7ff4373e4640 1 -- 172.21.15.59:0/3354360763 shutdown_connections 2024-06-29T15:57:31.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.420+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3354360763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162190 0x7ff4300b38e0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.420+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3354360763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff430162770 0x7ff4300ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.420+0000 7ff4373e4640 1 -- 172.21.15.59:0/3354360763 >> 172.21.15.59:0/3354360763 conn(0x7ff4300d45b0 msgr2=0x7ff4300d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.420+0000 7ff4373e4640 1 -- 172.21.15.59:0/3354360763 shutdown_connections 2024-06-29T15:57:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.420+0000 7ff4373e4640 1 -- 172.21.15.59:0/3354360763 wait complete. 2024-06-29T15:57:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.422+0000 7ff4373e4640 1 Processor -- start 2024-06-29T15:57:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.422+0000 7ff4373e4640 1 -- start start 2024-06-29T15:57:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff4373e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e0a0 0x7ff430150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff4373e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162770 0x7ff4301566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff4373e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4301511a0 0x7ff43015b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff4373e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff4300b42f0 con 0x7ff430162770 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff4373e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4300b4170 con 0x7ff4301511a0 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff4373e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff4300b4470 con 0x7ff43015e0a0 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff43497f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162770 0x7ff4301566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff43497f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162770 0x7ff4301566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46198/0 (socket says 172.21.15.59:46198) 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff43497f640 1 -- 172.21.15.59:0/3904504356 learned_addr learned my addr 172.21.15.59:0/3904504356 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff435180640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e0a0 0x7ff430150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.423+0000 7ff435981640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4301511a0 0x7ff43015b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff43497f640 1 -- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e0a0 msgr2=0x7ff430150a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff43497f640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e0a0 0x7ff430150a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff43497f640 1 -- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4301511a0 msgr2=0x7ff43015b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff43497f640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4301511a0 0x7ff43015b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff43497f640 1 -- 172.21.15.59:0/3904504356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff430153940 con 0x7ff430162770 2024-06-29T15:57:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff43497f640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162770 0x7ff4301566a0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7ff42000c3c0 tx=0x7ff42000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff425ffb640 1 -- 172.21.15.59:0/3904504356 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420011790 con 0x7ff430162770 2024-06-29T15:57:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff425ffb640 1 -- 172.21.15.59:0/3904504356 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff420017070 con 0x7ff430162770 2024-06-29T15:57:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff430156be0 con 0x7ff430162770 2024-06-29T15:57:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff425ffb640 1 -- 172.21.15.59:0/3904504356 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420011d00 con 0x7ff430162770 2024-06-29T15:57:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.424+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff430153f00 con 0x7ff430162770 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.425+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff430069ea0 con 0x7ff430162770 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.426+0000 7ff425ffb640 1 -- 172.21.15.59:0/3904504356 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7ff42001c460 con 0x7ff430162770 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.426+0000 7ff425ffb640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff4140763f0 0x7ff414078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.427+0000 7ff425ffb640 1 -- 172.21.15.59:0/3904504356 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7ff420097850 con 0x7ff430162770 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.427+0000 7ff435180640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff4140763f0 0x7ff414078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.427+0000 7ff435180640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff4140763f0 0x7ff414078900 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00bb50 tx=0x7ff42c00b2e0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.431+0000 7ff425ffb640 1 -- 172.21.15.59:0/3904504356 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff430069ea0 con 0x7ff430162770 2024-06-29T15:57:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.619+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff430150fc0 con 0x7ff430162770 2024-06-29T15:57:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.619+0000 7ff425ffb640 1 -- 172.21.15.59:0/3904504356 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v222) ==== 100+0+11 (secure 0 0 0) 0x7ff430150fc0 con 0x7ff430162770 2024-06-29T15:57:31.621 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:57:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff4140763f0 msgr2=0x7ff414078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff4140763f0 0x7ff414078900 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00bb50 tx=0x7ff42c00b2e0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162770 msgr2=0x7ff4301566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162770 0x7ff4301566a0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7ff42000c3c0 tx=0x7ff42000c880 comp rx=0 tx=0).stop 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 shutdown_connections 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff4140763f0 0x7ff414078900 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4301511a0 0x7ff43015b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff430162770 0x7ff4301566a0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 --2- 172.21.15.59:0/3904504356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff43015e0a0 0x7ff430150a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 >> 172.21.15.59:0/3904504356 conn(0x7ff4300d45b0 msgr2=0x7ff4301530c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 shutdown_connections 2024-06-29T15:57:31.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.622+0000 7ff4373e4640 1 -- 172.21.15.59:0/3904504356 wait complete. 2024-06-29T15:57:31.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:57:31.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.751+0000 7f03cfacc640 1 -- 172.21.15.59:0/82848326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b0004d90 msgr2=0x7f03b0005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.751+0000 7f03cfacc640 1 --2- 172.21.15.59:0/82848326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b0004d90 0x7f03b0005260 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009ed0 tx=0x7f03b0005830 comp rx=0 tx=0).stop 2024-06-29T15:57:31.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.751+0000 7f03ce069640 1 -- 172.21.15.59:0/82848326 reap_dead start 2024-06-29T15:57:31.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.751+0000 7f03cfacc640 1 -- 172.21.15.59:0/82848326 shutdown_connections 2024-06-29T15:57:31.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.751+0000 7f03cfacc640 1 -- 172.21.15.59:0/82848326 >> 172.21.15.59:0/82848326 conn(0x7f03c8073e50 msgr2=0x7f03c80b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.751+0000 7f03cfacc640 1 -- 172.21.15.59:0/82848326 shutdown_connections 2024-06-29T15:57:31.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.751+0000 7f03cfacc640 1 -- 172.21.15.59:0/82848326 wait complete. 2024-06-29T15:57:31.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.753+0000 7f03cfacc640 1 Processor -- start 2024-06-29T15:57:31.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.753+0000 7f03cfacc640 1 -- start start 2024-06-29T15:57:31.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03cfacc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b0004d90 0x7f03c8159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03cfacc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c8159f70 0x7f03c813a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03cfacc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03c813a5f0 0x7f03c8135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03cfacc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f03c8163d40 con 0x7f03c813a5f0 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03cfacc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03c8163bc0 con 0x7f03c8159f70 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03cfacc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f03c8163ec0 con 0x7f03b0004d90 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03c813a5f0 0x7f03c8135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03c813a5f0 0x7f03c8135ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46214/0 (socket says 172.21.15.59:46214) 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 -- 172.21.15.59:0/1083396754 learned_addr learned my addr 172.21.15.59:0/1083396754 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03cd067640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c8159f70 0x7f03c813a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03cd868640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b0004d90 0x7f03c8159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 -- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b0004d90 msgr2=0x7f03c8159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b0004d90 0x7f03c8159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 -- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c8159f70 msgr2=0x7f03c813a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c8159f70 0x7f03c813a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 -- 172.21.15.59:0/1083396754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c813aa00 con 0x7f03c813a5f0 2024-06-29T15:57:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.755+0000 7f03ce069640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03c813a5f0 0x7f03c8135ff0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f03c0036040 tx=0x7f03c0034c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.756+0000 7f03c67fc640 1 -- 172.21.15.59:0/1083396754 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c0043070 con 0x7f03c813a5f0 2024-06-29T15:57:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.756+0000 7f03c67fc640 1 -- 172.21.15.59:0/1083396754 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03c0034d30 con 0x7f03c813a5f0 2024-06-29T15:57:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.756+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c81370a0 con 0x7f03c813a5f0 2024-06-29T15:57:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.756+0000 7f03c67fc640 1 -- 172.21.15.59:0/1083396754 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c0038040 con 0x7f03c813a5f0 2024-06-29T15:57:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.756+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c8137660 con 0x7f03c813a5f0 2024-06-29T15:57:31.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.757+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c81443e0 con 0x7f03c813a5f0 2024-06-29T15:57:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.757+0000 7f03c67fc640 1 -- 172.21.15.59:0/1083396754 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f03c0046070 con 0x7f03c813a5f0 2024-06-29T15:57:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.758+0000 7f03c67fc640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f03b4076420 0x7f03b4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.758+0000 7f03c67fc640 1 -- 172.21.15.59:0/1083396754 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f03c00b7c20 con 0x7f03c813a5f0 2024-06-29T15:57:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.758+0000 7f03cd868640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f03b4076420 0x7f03b4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.759+0000 7f03cd868640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f03b4076420 0x7f03b4078930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03b8001050 tx=0x7f03b8006910 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.762+0000 7f03c67fc640 1 -- 172.21.15.59:0/1083396754 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f03c813aa00 con 0x7f03c813a5f0 2024-06-29T15:57:31.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.955+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f03c8159d20 con 0x7f03c813a5f0 2024-06-29T15:57:31.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.955+0000 7f03c67fc640 1 -- 172.21.15.59:0/1083396754 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v222) ==== 84+0+209 (secure 0 0 0) 0x7f03c00892a0 con 0x7f03c813a5f0 2024-06-29T15:57:31.956 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 27, 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": "cephfs_metadata" 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:31.957 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 28, 2024-06-29T15:57:31.958 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": "cephfs_data" 2024-06-29T15:57:31.958 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:31.958 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:57:31.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f03b4076420 msgr2=0x7f03b4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f03b4076420 0x7f03b4078930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03b8001050 tx=0x7f03b8006910 comp rx=0 tx=0).stop 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03c813a5f0 msgr2=0x7f03c8135ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03c813a5f0 0x7f03c8135ff0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f03c0036040 tx=0x7f03c0034c40 comp rx=0 tx=0).stop 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 shutdown_connections 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f03b4076420 0x7f03b4078930 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03c813a5f0 0x7f03c8135ff0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03c8159f70 0x7f03c813a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 --2- 172.21.15.59:0/1083396754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b0004d90 0x7f03c8159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.957+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 >> 172.21.15.59:0/1083396754 conn(0x7f03c8073e50 msgr2=0x7f03c8143030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.958+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 shutdown_connections 2024-06-29T15:57:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:31.958+0000 7f03cfacc640 1 -- 172.21.15.59:0/1083396754 wait complete. 2024-06-29T15:57:31.970 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-06-29T15:57:31.970 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-06-29T15:57:32.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.091+0000 7f9e7e34a640 1 -- 172.21.15.59:0/643031811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 msgr2=0x7f9e7815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.091+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/643031811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 0x7f9e7815b340 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009830 tx=0x7f9e6802a510 comp rx=0 tx=0).stop 2024-06-29T15:57:32.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.091+0000 7f9e7e34a640 1 -- 172.21.15.59:0/643031811 shutdown_connections 2024-06-29T15:57:32.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.091+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/643031811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e78072150 0x7f9e78072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.091+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/643031811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 0x7f9e7815b340 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.091+0000 7f9e7e34a640 1 -- 172.21.15.59:0/643031811 >> 172.21.15.59:0/643031811 conn(0x7f9e7806cb30 msgr2=0x7f9e7806cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.091+0000 7f9e7e34a640 1 -- 172.21.15.59:0/643031811 shutdown_connections 2024-06-29T15:57:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.091+0000 7f9e7e34a640 1 -- 172.21.15.59:0/643031811 wait complete. 2024-06-29T15:57:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.093+0000 7f9e7e34a640 1 Processor -- start 2024-06-29T15:57:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.093+0000 7f9e7e34a640 1 -- start start 2024-06-29T15:57:32.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7e34a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e78072150 0x7f9e78153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7e34a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e780b3410 0x7f9e78149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7e34a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 0x7f9e78166000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7e34a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7806f300 con 0x7f9e780cf310 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7e34a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7806f180 con 0x7f9e78072150 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7e34a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9e7806f480 con 0x7f9e780b3410 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7c8e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 0x7f9e78166000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7c8e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 0x7f9e78166000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46232/0 (socket says 172.21.15.59:46232) 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e7c8e7640 1 -- 172.21.15.59:0/2049309507 learned_addr learned my addr 172.21.15.59:0/2049309507 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e77fff640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e78072150 0x7f9e78153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.094+0000 7f9e6ffff640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e780b3410 0x7f9e78149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e7c8e7640 1 -- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e780b3410 msgr2=0x7f9e78149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e7c8e7640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e780b3410 0x7f9e78149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e7c8e7640 1 -- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e78072150 msgr2=0x7f9e78153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e7c8e7640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e78072150 0x7f9e78153560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e7c8e7640 1 -- 172.21.15.59:0/2049309507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e78153db0 con 0x7f9e780cf310 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e6ffff640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e780b3410 0x7f9e78149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e7c8e7640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 0x7f9e78166000 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000e9c0 tx=0x7f9e7000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e757fa640 1 -- 172.21.15.59:0/2049309507 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70004d20 con 0x7f9e780cf310 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e780ca450 con 0x7f9e780cf310 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e757fa640 1 -- 172.21.15.59:0/2049309507 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e7000cb20 con 0x7f9e780cf310 2024-06-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e757fa640 1 -- 172.21.15.59:0/2049309507 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70005020 con 0x7f9e780cf310 2024-06-29T15:57:32.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.095+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e780caa10 con 0x7f9e780cf310 2024-06-29T15:57:32.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.096+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e781563c0 con 0x7f9e780cf310 2024-06-29T15:57:32.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.097+0000 7f9e757fa640 1 -- 172.21.15.59:0/2049309507 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f9e700051c0 con 0x7f9e780cf310 2024-06-29T15:57:32.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.097+0000 7f9e757fa640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9e580763f0 0x7f9e58078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.097+0000 7f9e757fa640 1 -- 172.21.15.59:0/2049309507 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6157+0+0 (secure 0 0 0) 0x7f9e700984f0 con 0x7f9e780cf310 2024-06-29T15:57:32.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.097+0000 7f9e77fff640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9e580763f0 0x7f9e58078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.098+0000 7f9e77fff640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9e580763f0 0x7f9e58078900 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9e68034000 tx=0x7f9e68004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:32.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.101+0000 7f9e757fa640 1 -- 172.21.15.59:0/2049309507 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9e781563c0 con 0x7f9e780cf310 2024-06-29T15:57:32.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:32.296+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) -- 0x7f9e7815b340 con 0x7f9e780cf310 2024-06-29T15:57:33.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.278+0000 7f9e757fa640 1 -- 172.21.15.59:0/2049309507 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v223) ==== 144+0+0 (secure 0 0 0) 0x7f9e70016030 con 0x7f9e780cf310 2024-06-29T15:57:33.280 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'backup_fs_metadata' created 2024-06-29T15:57:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.283+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9e580763f0 msgr2=0x7f9e58078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.283+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9e580763f0 0x7f9e58078900 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9e68034000 tx=0x7f9e68004730 comp rx=0 tx=0).stop 2024-06-29T15:57:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.283+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 msgr2=0x7f9e78166000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.283+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 0x7f9e78166000 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000e9c0 tx=0x7f9e7000ee80 comp rx=0 tx=0).stop 2024-06-29T15:57:33.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.285+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 shutdown_connections 2024-06-29T15:57:33.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.285+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9e580763f0 0x7f9e58078900 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.285+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780cf310 0x7f9e78166000 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.285+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9e780b3410 0x7f9e78149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.285+0000 7f9e7e34a640 1 --2- 172.21.15.59:0/2049309507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e78072150 0x7f9e78153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.285+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 >> 172.21.15.59:0/2049309507 conn(0x7f9e7806cb30 msgr2=0x7f9e780ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:33.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.286+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 shutdown_connections 2024-06-29T15:57:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.287+0000 7f9e7e34a640 1 -- 172.21.15.59:0/2049309507 wait complete. 2024-06-29T15:57:33.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.434+0000 7f48b30d2640 1 -- 172.21.15.59:0/1648647499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0cebf0 msgr2=0x7f48ac0cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.434+0000 7f48b30d2640 1 --2- 172.21.15.59:0/1648647499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0cebf0 0x7f48ac0cf000 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f48a0005fd0 tx=0x7f48a002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.434+0000 7f48b30d2640 1 -- 172.21.15.59:0/1648647499 shutdown_connections 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.434+0000 7f48b30d2640 1 --2- 172.21.15.59:0/1648647499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0cebf0 0x7f48ac0cf000 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.434+0000 7f48b30d2640 1 --2- 172.21.15.59:0/1648647499 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48ac0ce510 0x7f48ac15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.434+0000 7f48b30d2640 1 -- 172.21.15.59:0/1648647499 >> 172.21.15.59:0/1648647499 conn(0x7f48ac0d3e40 msgr2=0x7f48ac0d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.434+0000 7f48b30d2640 1 -- 172.21.15.59:0/1648647499 shutdown_connections 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.434+0000 7f48b30d2640 1 -- 172.21.15.59:0/1648647499 wait complete. 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.436+0000 7f48b30d2640 1 Processor -- start 2024-06-29T15:57:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.436+0000 7f48b30d2640 1 -- start start 2024-06-29T15:57:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b30d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0ce510 0x7f48ac1f4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b30d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48ac0cebf0 0x7f48ac1571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b30d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48ac161980 0x7f48ac0c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b30d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac0b2b90 con 0x7f48ac0ce510 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b30d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac0b2a10 con 0x7f48ac161980 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b30d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f48ac0b4bc0 con 0x7f48ac0cebf0 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b0e6e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0ce510 0x7f48ac1f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b0e6e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0ce510 0x7f48ac1f4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44600/0 (socket says 172.21.15.59:44600) 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b0e6e640 1 -- 172.21.15.59:0/2484037462 learned_addr learned my addr 172.21.15.59:0/2484037462 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48b166f640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48ac161980 0x7f48ac0c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.437+0000 7f48abfff640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48ac0cebf0 0x7f48ac1571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48b0e6e640 1 -- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48ac0cebf0 msgr2=0x7f48ac1571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48b0e6e640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48ac0cebf0 0x7f48ac1571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48b0e6e640 1 -- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48ac161980 msgr2=0x7f48ac0c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48b0e6e640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48ac161980 0x7f48ac0c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48b0e6e640 1 -- 172.21.15.59:0/2484037462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ac1598d0 con 0x7f48ac0ce510 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48b0e6e640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0ce510 0x7f48ac1f4970 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f489c00d7a0 tx=0x7f489c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48a97fa640 1 -- 172.21.15.59:0/2484037462 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c017070 con 0x7f48ac0ce510 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ac159dd0 con 0x7f48ac0ce510 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48a97fa640 1 -- 172.21.15.59:0/2484037462 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f489c009d20 con 0x7f48ac0ce510 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48a97fa640 1 -- 172.21.15.59:0/2484037462 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c004280 con 0x7f48ac0ce510 2024-06-29T15:57:33.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.438+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ac15a390 con 0x7f48ac0ce510 2024-06-29T15:57:33.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.439+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48ac15dc10 con 0x7f48ac0ce510 2024-06-29T15:57:33.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.440+0000 7f48a97fa640 1 -- 172.21.15.59:0/2484037462 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f489c005020 con 0x7f48ac0ce510 2024-06-29T15:57:33.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.441+0000 7f48a97fa640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f48980763f0 0x7f4898078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:33.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.441+0000 7f48a97fa640 1 -- 172.21.15.59:0/2484037462 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 6530+0+0 (secure 0 0 0) 0x7f489c0985e0 con 0x7f48ac0ce510 2024-06-29T15:57:33.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.441+0000 7f48abfff640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f48980763f0 0x7f4898078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:33.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.442+0000 7f48abfff640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f48980763f0 0x7f4898078900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f48a0033000 tx=0x7f48a0004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:33.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.445+0000 7f48a97fa640 1 -- 172.21.15.59:0/2484037462 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f489c020020 con 0x7f48ac0ce510 2024-06-29T15:57:33.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:33.639+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f48ac0cde40 con 0x7f48ac0ce510 2024-06-29T15:57:34.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.285+0000 7f48a97fa640 1 -- 172.21.15.59:0/2484037462 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v224) ==== 176+0+0 (secure 0 0 0) 0x7f489c061300 con 0x7f48ac0ce510 2024-06-29T15:57:34.286 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'backup_fs_data' created 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.288+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f48980763f0 msgr2=0x7f4898078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.288+0000 7f48b30d2640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f48980763f0 0x7f4898078900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f48a0033000 tx=0x7f48a0004730 comp rx=0 tx=0).stop 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.288+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0ce510 msgr2=0x7f48ac1f4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.288+0000 7f48b30d2640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0ce510 0x7f48ac1f4970 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f489c00d7a0 tx=0x7f489c00dc60 comp rx=0 tx=0).stop 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.288+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 shutdown_connections 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.288+0000 7f48b30d2640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f48980763f0 0x7f4898078900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.289+0000 7f48b30d2640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48ac161980 0x7f48ac0c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.289+0000 7f48b30d2640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48ac0cebf0 0x7f48ac1571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.289+0000 7f48b30d2640 1 --2- 172.21.15.59:0/2484037462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48ac0ce510 0x7f48ac1f4970 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.289+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 >> 172.21.15.59:0/2484037462 conn(0x7f48ac0d3e40 msgr2=0x7f48ac06da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:34.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.289+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 shutdown_connections 2024-06-29T15:57:34.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.289+0000 7f48b30d2640 1 -- 172.21.15.59:0/2484037462 wait complete. 2024-06-29T15:57:34.313 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-06-29T15:57:34.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.438+0000 7f6a70dd2640 1 -- 172.21.15.59:0/814749316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c15ee10 msgr2=0x7f6a6c15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.438+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/814749316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c15ee10 0x7f6a6c15f9b0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f6a64005fd0 tx=0x7f6a6402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.438+0000 7f6a70dd2640 1 -- 172.21.15.59:0/814749316 shutdown_connections 2024-06-29T15:57:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.438+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/814749316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c15ee10 0x7f6a6c15f9b0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.438+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/814749316 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a6c0b40a0 0x7f6a6c15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.438+0000 7f6a70dd2640 1 -- 172.21.15.59:0/814749316 >> 172.21.15.59:0/814749316 conn(0x7f6a6c0d3ee0 msgr2=0x7f6a6c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.438+0000 7f6a70dd2640 1 -- 172.21.15.59:0/814749316 shutdown_connections 2024-06-29T15:57:34.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.438+0000 7f6a70dd2640 1 -- 172.21.15.59:0/814749316 wait complete. 2024-06-29T15:57:34.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.440+0000 7f6a70dd2640 1 Processor -- start 2024-06-29T15:57:34.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.440+0000 7f6a70dd2640 1 -- start start 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a70dd2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c0d23c0 0x7f6a6c15c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a70dd2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a6c15ee10 0x7f6a6c14eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a70dd2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c0ca130 0x7f6a6c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a70dd2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6c160280 con 0x7f6a6c0ca130 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a70dd2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6c160100 con 0x7f6a6c0d23c0 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a70dd2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6c160400 con 0x7f6a6c15ee10 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c0ca130 0x7f6a6c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c0ca130 0x7f6a6c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44618/0 (socket says 172.21.15.59:44618) 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 -- 172.21.15.59:0/1291698005 learned_addr learned my addr 172.21.15.59:0/1291698005 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:34.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a69d9b640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a6c15ee10 0x7f6a6c14eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6a59c640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c0d23c0 0x7f6a6c15c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 -- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a6c15ee10 msgr2=0x7f6a6c14eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a6c15ee10 0x7f6a6c14eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 -- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c0d23c0 msgr2=0x7f6a6c15c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c0d23c0 0x7f6a6c15c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 -- 172.21.15.59:0/1291698005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6c0caa80 con 0x7f6a6c0ca130 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.442+0000 7f6a6ad9d640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c0ca130 0x7f6a6c0ca540 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009cd0 tx=0x7f6a640039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.443+0000 7f6a62ffd640 1 -- 172.21.15.59:0/1291698005 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64031990 con 0x7f6a6c0ca130 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.443+0000 7f6a62ffd640 1 -- 172.21.15.59:0/1291698005 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a64031c20 con 0x7f6a6c0ca130 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.443+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6c14f430 con 0x7f6a6c0ca130 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.443+0000 7f6a62ffd640 1 -- 172.21.15.59:0/1291698005 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64033290 con 0x7f6a6c0ca130 2024-06-29T15:57:34.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.443+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a6c15d410 con 0x7f6a6c0ca130 2024-06-29T15:57:34.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.444+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6c0b8d00 con 0x7f6a6c0ca130 2024-06-29T15:57:34.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.444+0000 7f6a62ffd640 1 -- 172.21.15.59:0/1291698005 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f6a64033430 con 0x7f6a6c0ca130 2024-06-29T15:57:34.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.445+0000 7f6a62ffd640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a4c0763f0 0x7f6a4c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.445+0000 7f6a62ffd640 1 -- 172.21.15.59:0/1291698005 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(224..224 src has 1..224) ==== 6915+0+0 (secure 0 0 0) 0x7f6a640b99a0 con 0x7f6a6c0ca130 2024-06-29T15:57:34.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.445+0000 7f6a6a59c640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a4c0763f0 0x7f6a4c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.446+0000 7f6a6a59c640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a4c0763f0 0x7f6a4c078900 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c000f60 tx=0x7f6a5c009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:34.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.449+0000 7f6a62ffd640 1 -- 172.21.15.59:0/1291698005 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6a6c0b8d00 con 0x7f6a6c0ca130 2024-06-29T15:57:34.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:34.677+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) -- 0x7f6a6c0c79f0 con 0x7f6a6c0ca130 2024-06-29T15:57:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.286+0000 7f6a62ffd640 1 -- 172.21.15.59:0/1291698005 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 29 and data pool 30 v117) ==== 183+0+0 (secure 0 0 0) 0x7f6a64051020 con 0x7f6a6c0ca130 2024-06-29T15:57:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 29 and data pool 30 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a4c0763f0 msgr2=0x7f6a4c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a4c0763f0 0x7f6a4c078900 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c000f60 tx=0x7f6a5c009040 comp rx=0 tx=0).stop 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c0ca130 msgr2=0x7f6a6c0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c0ca130 0x7f6a6c0ca540 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009cd0 tx=0x7f6a640039f0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 shutdown_connections 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6a4c0763f0 0x7f6a4c078900 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6c0ca130 0x7f6a6c0ca540 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6a6c15ee10 0x7f6a6c14eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 --2- 172.21.15.59:0/1291698005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a6c0d23c0 0x7f6a6c15c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 >> 172.21.15.59:0/1291698005 conn(0x7f6a6c0d3ee0 msgr2=0x7f6a6c0b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 shutdown_connections 2024-06-29T15:57:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.289+0000 7f6a70dd2640 1 -- 172.21.15.59:0/1291698005 wait complete. 2024-06-29T15:57:35.307 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:35.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.430+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1319512710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f408c190 msgr2=0x7fc2f40cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.430+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1319512710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f408c190 0x7fc2f40cf080 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4005fd0 tx=0x7fc2e402a5a0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.430+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1319512710 shutdown_connections 2024-06-29T15:57:35.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.430+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1319512710 >> v2:172.21.15.153:3300/0 conn(0x7fc2f40ce950 0x7fc2f40d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.430+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1319512710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f408c190 0x7fc2f40cf080 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.430+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1319512710 >> 172.21.15.59:0/1319512710 conn(0x7fc2f40b5610 msgr2=0x7fc2f40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:35.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.430+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1319512710 shutdown_connections 2024-06-29T15:57:35.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.430+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1319512710 wait complete. 2024-06-29T15:57:35.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.432+0000 7fc2f9d2e640 1 Processor -- start 2024-06-29T15:57:35.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.432+0000 7fc2f9d2e640 1 -- start start 2024-06-29T15:57:35.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f9d2e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f408c190 0x7fc2f40af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f9d2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f40ce950 0x7fc2f40aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f9d2e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f40cf5c0 0x7fc2f40aae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f9d2e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f40b7f60 con 0x7fc2f40ce950 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f9d2e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f40b7de0 con 0x7fc2f408c190 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f9d2e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f40b80e0 con 0x7fc2f40cf5c0 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f40ce950 0x7fc2f40aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f40ce950 0x7fc2f40aa8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44644/0 (socket says 172.21.15.59:44644) 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f2ffd640 1 -- 172.21.15.59:0/1671179238 learned_addr learned my addr 172.21.15.59:0/1671179238 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f37fe640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f408c190 0x7fc2f40af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.433+0000 7fc2f3fff640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f40cf5c0 0x7fc2f40aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2f2ffd640 1 -- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f40cf5c0 msgr2=0x7fc2f40aae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2f2ffd640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f40cf5c0 0x7fc2f40aae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2f2ffd640 1 -- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f408c190 msgr2=0x7fc2f40af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2f2ffd640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f408c190 0x7fc2f40af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2f2ffd640 1 -- 172.21.15.59:0/1671179238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f40adc60 con 0x7fc2f40ce950 2024-06-29T15:57:35.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2f2ffd640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f40ce950 0x7fc2f40aa8d0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c370 tx=0x7fc2e800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:35.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2dbfff640 1 -- 172.21.15.59:0/1671179238 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8017070 con 0x7fc2f40ce950 2024-06-29T15:57:35.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2dbfff640 1 -- 172.21.15.59:0/1671179238 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2e80053c0 con 0x7fc2f40ce950 2024-06-29T15:57:35.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f40adf40 con 0x7fc2f40ce950 2024-06-29T15:57:35.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2dbfff640 1 -- 172.21.15.59:0/1671179238 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8003850 con 0x7fc2f40ce950 2024-06-29T15:57:35.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.434+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f40ae500 con 0x7fc2f40ce950 2024-06-29T15:57:35.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.435+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f40cf080 con 0x7fc2f40ce950 2024-06-29T15:57:35.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.436+0000 7fc2dbfff640 1 -- 172.21.15.59:0/1671179238 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fc2e801a070 con 0x7fc2f40ce950 2024-06-29T15:57:35.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.436+0000 7fc2dbfff640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc2e0076420 0x7fc2e0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.437+0000 7fc2dbfff640 1 -- 172.21.15.59:0/1671179238 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 7021+0+0 (secure 0 0 0) 0x7fc2e8097050 con 0x7fc2f40ce950 2024-06-29T15:57:35.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.437+0000 7fc2f37fe640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc2e0076420 0x7fc2e0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.437+0000 7fc2f37fe640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc2e0076420 0x7fc2e0078930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4033000 tx=0x7fc2e4004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:35.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.441+0000 7fc2dbfff640 1 -- 172.21.15.59:0/1671179238 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2e805fdc0 con 0x7fc2f40ce950 2024-06-29T15:57:35.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.637+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2f40cc0d0 con 0x7fc2f40ce950 2024-06-29T15:57:35.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.638+0000 7fc2dbfff640 1 -- 172.21.15.59:0/1671179238 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) ==== 74+0+20789 (secure 0 0 0) 0x7fc2e8068400 con 0x7fc2f40ce950 2024-06-29T15:57:35.639 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:35.640 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":225,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:34.679120+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:35.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc2e0076420 msgr2=0x7fc2e0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc2e0076420 0x7fc2e0078930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4033000 tx=0x7fc2e4004730 comp rx=0 tx=0).stop 2024-06-29T15:57:35.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f40ce950 msgr2=0x7fc2f40aa8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f40ce950 0x7fc2f40aa8d0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c370 tx=0x7fc2e800c830 comp rx=0 tx=0).stop 2024-06-29T15:57:35.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 shutdown_connections 2024-06-29T15:57:35.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fc2e0076420 0x7fc2e0078930 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2f40cf5c0 0x7fc2f40aae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2f40ce950 0x7fc2f40aa8d0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 --2- 172.21.15.59:0/1671179238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f408c190 0x7fc2f40af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 >> 172.21.15.59:0/1671179238 conn(0x7fc2f40b5610 msgr2=0x7fc2f4068420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:35.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 shutdown_connections 2024-06-29T15:57:35.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.640+0000 7fc2f9d2e640 1 -- 172.21.15.59:0/1671179238 wait complete. 2024-06-29T15:57:35.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:35.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.770+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3139548619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0cf5c0 msgr2=0x7fdccc0d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.770+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3139548619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0cf5c0 0x7fdccc0d1a20 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0005fd0 tx=0x7fdcc002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.770+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3139548619 shutdown_connections 2024-06-29T15:57:35.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.770+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3139548619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0cf5c0 0x7fdccc0d1a20 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.770+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3139548619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdccc08c190 0x7fdccc0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.770+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3139548619 >> 172.21.15.59:0/3139548619 conn(0x7fdccc0b5610 msgr2=0x7fdccc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:35.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.770+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3139548619 shutdown_connections 2024-06-29T15:57:35.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.770+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3139548619 wait complete. 2024-06-29T15:57:35.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.772+0000 7fdcd23ef640 1 Processor -- start 2024-06-29T15:57:35.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.772+0000 7fdcd23ef640 1 -- start start 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.773+0000 7fdcd23ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdccc08c190 0x7fdccc1e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.773+0000 7fdcd23ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0ce950 0x7fdccc1f87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.773+0000 7fdcd23ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdccc0cf5c0 0x7fdccc1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.773+0000 7fdcd23ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc0b7fa0 con 0x7fdccc0ce950 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.773+0000 7fdcd23ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc0b7e20 con 0x7fdccc08c190 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.773+0000 7fdcd23ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdccc0b8120 con 0x7fdccc0cf5c0 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0ce950 0x7fdccc1f87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0ce950 0x7fdccc1f87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44658/0 (socket says 172.21.15.59:44658) 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 -- 172.21.15.59:0/3210075904 learned_addr learned my addr 172.21.15.59:0/3210075904 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccbfff640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdccc08c190 0x7fdccc1e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdcd098c640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdccc0cf5c0 0x7fdccc1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 -- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdccc0cf5c0 msgr2=0x7fdccc1f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdccc0cf5c0 0x7fdccc1f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 -- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdccc08c190 msgr2=0x7fdccc1e71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdccc08c190 0x7fdccc1e71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 -- 172.21.15.59:0/3210075904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdccc1e7c60 con 0x7fdccc0ce950 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdccb7fe640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0ce950 0x7fdccc1f87b0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0034040 tx=0x7fdcc00035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdcc8ff9640 1 -- 172.21.15.59:0/3210075904 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc003e030 con 0x7fdccc0ce950 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdcc8ff9640 1 -- 172.21.15.59:0/3210075904 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcc0003ae0 con 0x7fdccc0ce950 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdccc1e7f40 con 0x7fdccc0ce950 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdcc8ff9640 1 -- 172.21.15.59:0/3210075904 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0035040 con 0x7fdccc0ce950 2024-06-29T15:57:35.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.774+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdccc1e8500 con 0x7fdccc0ce950 2024-06-29T15:57:35.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.775+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdccc0cf080 con 0x7fdccc0ce950 2024-06-29T15:57:35.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.776+0000 7fdcc8ff9640 1 -- 172.21.15.59:0/3210075904 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fdcc0041030 con 0x7fdccc0ce950 2024-06-29T15:57:35.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.777+0000 7fdcc8ff9640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fdcb8076420 0x7fdcb8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.777+0000 7fdcc8ff9640 1 -- 172.21.15.59:0/3210075904 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 7021+0+0 (secure 0 0 0) 0x7fdcc00b8750 con 0x7fdccc0ce950 2024-06-29T15:57:35.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.777+0000 7fdccbfff640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fdcb8076420 0x7fdcb8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.778+0000 7fdccbfff640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fdcb8076420 0x7fdcb8078930 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009bb0 tx=0x7fdcbc005f90 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:35.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.780+0000 7fdcc8ff9640 1 -- 172.21.15.59:0/3210075904 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdcc0081270 con 0x7fdccc0ce950 2024-06-29T15:57:35.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.971+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdccc06d950 con 0x7fdccc0ce950 2024-06-29T15:57:35.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.972+0000 7fdcc8ff9640 1 -- 172.21.15.59:0/3210075904 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) ==== 74+0+20789 (secure 0 0 0) 0x7fdcc0085570 con 0x7fdccc0ce950 2024-06-29T15:57:35.973 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:35.973 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":225,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:34.679120+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:35.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fdcb8076420 msgr2=0x7fdcb8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fdcb8076420 0x7fdcb8078930 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009bb0 tx=0x7fdcbc005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:35.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0ce950 msgr2=0x7fdccc1f87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0ce950 0x7fdccc1f87b0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0034040 tx=0x7fdcc00035f0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 shutdown_connections 2024-06-29T15:57:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fdcb8076420 0x7fdcb8078930 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdccc0cf5c0 0x7fdccc1f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdccc0ce950 0x7fdccc1f87b0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 --2- 172.21.15.59:0/3210075904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdccc08c190 0x7fdccc1e71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 >> 172.21.15.59:0/3210075904 conn(0x7fdccc0b5610 msgr2=0x7fdccc0c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 shutdown_connections 2024-06-29T15:57:35.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:35.974+0000 7fdcd23ef640 1 -- 172.21.15.59:0/3210075904 wait complete. 2024-06-29T15:57:35.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-06-29T15:57:36.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.107+0000 7ff72d081640 1 -- 172.21.15.59:0/3798098280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72815e0a0 msgr2=0x7ff7280b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:36.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.107+0000 7ff72d081640 1 --2- 172.21.15.59:0/3798098280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72815e0a0 0x7ff7280b38e0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff720009830 tx=0x7ff72002a470 comp rx=0 tx=0).stop 2024-06-29T15:57:36.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.107+0000 7ff72d081640 1 -- 172.21.15.59:0/3798098280 shutdown_connections 2024-06-29T15:57:36.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.107+0000 7ff72d081640 1 --2- 172.21.15.59:0/3798098280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff72815e0a0 0x7ff7280b38e0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.107+0000 7ff72d081640 1 --2- 172.21.15.59:0/3798098280 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff728162190 0x7ff7280ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.107+0000 7ff72d081640 1 -- 172.21.15.59:0/3798098280 >> 172.21.15.59:0/3798098280 conn(0x7ff7280d45b0 msgr2=0x7ff7280d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:36.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.107+0000 7ff72d081640 1 -- 172.21.15.59:0/3798098280 shutdown_connections 2024-06-29T15:57:36.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.107+0000 7ff72d081640 1 -- 172.21.15.59:0/3798098280 wait complete. 2024-06-29T15:57:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.109+0000 7ff72d081640 1 Processor -- start 2024-06-29T15:57:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.109+0000 7ff72d081640 1 -- start start 2024-06-29T15:57:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.110+0000 7ff72d081640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff72815e0a0 0x7ff7281f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.110+0000 7ff72d081640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff728162770 0x7ff72815a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.110+0000 7ff72d081640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7281f7e30 0x7ff7281f8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.110+0000 7ff72d081640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff7280b43a0 con 0x7ff72815e0a0 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.110+0000 7ff72d081640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7280b4220 con 0x7ff728162770 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.110+0000 7ff72d081640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff7280b4520 con 0x7ff7281f7e30 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff72815e0a0 0x7ff7281f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff72815e0a0 0x7ff7281f4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44674/0 (socket says 172.21.15.59:44674) 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 -- 172.21.15.59:0/1985225408 learned_addr learned my addr 172.21.15.59:0/1985225408 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff72659c640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff728162770 0x7ff72815a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff72759e640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7281f7e30 0x7ff7281f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 -- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7281f7e30 msgr2=0x7ff7281f8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7281f7e30 0x7ff7281f8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 -- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff728162770 msgr2=0x7ff72815a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff728162770 0x7ff72815a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 -- 172.21.15.59:0/1985225408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7281f8780 con 0x7ff72815e0a0 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff72759e640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7281f7e30 0x7ff7281f8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff726d9d640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff72815e0a0 0x7ff7281f4140 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7ff71800cbc0 tx=0x7ff71800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff71f7fe640 1 -- 172.21.15.59:0/1985225408 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718016020 con 0x7ff72815e0a0 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff71f7fe640 1 -- 172.21.15.59:0/1985225408 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff71800ad90 con 0x7ff72815e0a0 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff71f7fe640 1 -- 172.21.15.59:0/1985225408 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718012680 con 0x7ff72815e0a0 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7281f8c00 con 0x7ff72815e0a0 2024-06-29T15:57:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.111+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff72815b1f0 con 0x7ff72815e0a0 2024-06-29T15:57:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.112+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7281536c0 con 0x7ff72815e0a0 2024-06-29T15:57:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.113+0000 7ff71f7fe640 1 -- 172.21.15.59:0/1985225408 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7ff718019020 con 0x7ff72815e0a0 2024-06-29T15:57:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.114+0000 7ff71f7fe640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff7080763f0 0x7ff708078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.114+0000 7ff71f7fe640 1 -- 172.21.15.59:0/1985225408 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 7021+0+0 (secure 0 0 0) 0x7ff7180985e0 con 0x7ff72815e0a0 2024-06-29T15:57:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.114+0000 7ff72659c640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff7080763f0 0x7ff708078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.114+0000 7ff72659c640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff7080763f0 0x7ff708078900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff72815c220 tx=0x7ff710005f90 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.118+0000 7ff71f7fe640 1 -- 172.21.15.59:0/1985225408 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff7281536c0 con 0x7ff72815e0a0 2024-06-29T15:57:36.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:36.360+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff7280b6360 con 0x7ff72815e0a0 2024-06-29T15:57:37.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.313+0000 7ff71f7fe640 1 -- 172.21.15.59:0/1985225408 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v120) ==== 121+0+0 (secure 0 0 0) 0x7ff718020020 con 0x7ff72815e0a0 2024-06-29T15:57:37.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff7080763f0 msgr2=0x7ff708078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff7080763f0 0x7ff708078900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff72815c220 tx=0x7ff710005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff72815e0a0 msgr2=0x7ff7281f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff72815e0a0 0x7ff7281f4140 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7ff71800cbc0 tx=0x7ff71800a060 comp rx=0 tx=0).stop 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 shutdown_connections 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7ff7080763f0 0x7ff708078900 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff7281f7e30 0x7ff7281f8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff728162770 0x7ff72815a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 --2- 172.21.15.59:0/1985225408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff72815e0a0 0x7ff7281f4140 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 >> 172.21.15.59:0/1985225408 conn(0x7ff7280d45b0 msgr2=0x7ff7280b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 shutdown_connections 2024-06-29T15:57:37.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.316+0000 7ff72d081640 1 -- 172.21.15.59:0/1985225408 wait complete. 2024-06-29T15:57:37.331 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:57:37.331 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-06-29T15:57:37.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.459+0000 7f0155acc640 1 -- 172.21.15.59:0/4165401424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2270 msgr2=0x7f015015bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.459+0000 7f0155acc640 1 --2- 172.21.15.59:0/4165401424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2270 0x7f015015bc20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0140009830 tx=0x7f014002a510 comp rx=0 tx=0).stop 2024-06-29T15:57:37.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.459+0000 7f0155acc640 1 -- 172.21.15.59:0/4165401424 shutdown_connections 2024-06-29T15:57:37.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.459+0000 7f0155acc640 1 --2- 172.21.15.59:0/4165401424 >> v2:172.21.15.153:3300/0 conn(0x7f01500b2fa0 0x7f01500d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.459+0000 7f0155acc640 1 --2- 172.21.15.59:0/4165401424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2270 0x7f015015bc20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.459+0000 7f0155acc640 1 -- 172.21.15.59:0/4165401424 >> 172.21.15.59:0/4165401424 conn(0x7f0150073e10 msgr2=0x7f0150074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:37.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.459+0000 7f0155acc640 1 -- 172.21.15.59:0/4165401424 shutdown_connections 2024-06-29T15:57:37.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.459+0000 7f0155acc640 1 -- 172.21.15.59:0/4165401424 wait complete. 2024-06-29T15:57:37.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.460+0000 7f0155acc640 1 Processor -- start 2024-06-29T15:57:37.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.461+0000 7f0155acc640 1 -- start start 2024-06-29T15:57:37.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f0155acc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01500b2270 0x7f015014d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f0155acc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2fa0 0x7f01500aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f0155acc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500d1a60 0x7f015015e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f0155acc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f01500b6cc0 con 0x7f01500d1a60 2024-06-29T15:57:37.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f0155acc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01500b6b40 con 0x7f01500b2270 2024-06-29T15:57:37.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f0155acc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f01500cfe90 con 0x7f01500b2fa0 2024-06-29T15:57:37.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f014ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500d1a60 0x7f015015e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f014ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500d1a60 0x7f015015e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44676/0 (socket says 172.21.15.59:44676) 2024-06-29T15:57:37.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f014ffff640 1 -- 172.21.15.59:0/2117217766 learned_addr learned my addr 172.21.15.59:0/2117217766 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:37.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f014f7fe640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01500b2270 0x7f015014d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f014effd640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2fa0 0x7f01500aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f014ffff640 1 -- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2fa0 msgr2=0x7f01500aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f014ffff640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2fa0 0x7f01500aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.462+0000 7f014ffff640 1 -- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01500b2270 msgr2=0x7f015014d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f014ffff640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01500b2270 0x7f015014d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f014ffff640 1 -- 172.21.15.59:0/2117217766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01500ab3e0 con 0x7f01500d1a60 2024-06-29T15:57:37.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f014effd640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2fa0 0x7f01500aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:37.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f014ffff640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500d1a60 0x7f015015e410 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f014800cb10 tx=0x7f014800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:37.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f0137fff640 1 -- 172.21.15.59:0/2117217766 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148016020 con 0x7f01500d1a60 2024-06-29T15:57:37.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f0137fff640 1 -- 172.21.15.59:0/2117217766 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f014800ad80 con 0x7f01500d1a60 2024-06-29T15:57:37.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01500ab100 con 0x7f01500d1a60 2024-06-29T15:57:37.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f0137fff640 1 -- 172.21.15.59:0/2117217766 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148012680 con 0x7f01500d1a60 2024-06-29T15:57:37.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.463+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01500abbe0 con 0x7f01500d1a60 2024-06-29T15:57:37.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.464+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01500d0340 con 0x7f01500d1a60 2024-06-29T15:57:37.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.465+0000 7f0137fff640 1 -- 172.21.15.59:0/2117217766 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f0148019020 con 0x7f01500d1a60 2024-06-29T15:57:37.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.465+0000 7f0137fff640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f013c0763f0 0x7f013c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.465+0000 7f0137fff640 1 -- 172.21.15.59:0/2117217766 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f0148098370 con 0x7f01500d1a60 2024-06-29T15:57:37.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.465+0000 7f014f7fe640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f013c0763f0 0x7f013c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.466+0000 7f014f7fe640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f013c0763f0 0x7f013c078900 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0140034000 tx=0x7f0140004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:37.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.470+0000 7f0137fff640 1 -- 172.21.15.59:0/2117217766 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0148020020 con 0x7f01500d1a60 2024-06-29T15:57:37.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:37.699+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f01500b08d0 con 0x7f01500d1a60 2024-06-29T15:57:38.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.320+0000 7f0137fff640 1 -- 172.21.15.59:0/2117217766 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v121) ==== 125+0+0 (secure 0 0 0) 0x7f01480616a0 con 0x7f01500d1a60 2024-06-29T15:57:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f013c0763f0 msgr2=0x7f013c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f013c0763f0 0x7f013c078900 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0140034000 tx=0x7f0140004730 comp rx=0 tx=0).stop 2024-06-29T15:57:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500d1a60 msgr2=0x7f015015e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500d1a60 0x7f015015e410 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f014800cb10 tx=0x7f014800a060 comp rx=0 tx=0).stop 2024-06-29T15:57:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 shutdown_connections 2024-06-29T15:57:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f013c0763f0 0x7f013c078900 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500d1a60 0x7f015015e410 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01500b2fa0 0x7f01500aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 --2- 172.21.15.59:0/2117217766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01500b2270 0x7f015014d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 >> 172.21.15.59:0/2117217766 conn(0x7f0150073e10 msgr2=0x7f01501592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 shutdown_connections 2024-06-29T15:57:38.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.323+0000 7f0155acc640 1 -- 172.21.15.59:0/2117217766 wait complete. 2024-06-29T15:57:38.337 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-06-29T15:57:38.337 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.459+0000 7fd983b7f640 1 -- 172.21.15.59:0/3061754372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 msgr2=0x7fd97c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.459+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3061754372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 0x7fd97c0cf080 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fd970005fd0 tx=0x7fd97002a4f0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.459+0000 7fd983b7f640 1 -- 172.21.15.59:0/3061754372 shutdown_connections 2024-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.459+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3061754372 >> v2:172.21.15.153:3300/0 conn(0x7fd97c0cf5c0 0x7fd97c0d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.459+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3061754372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 0x7fd97c0cf080 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.459+0000 7fd983b7f640 1 -- 172.21.15.59:0/3061754372 >> 172.21.15.59:0/3061754372 conn(0x7fd97c0b5610 msgr2=0x7fd97c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.460+0000 7fd983b7f640 1 -- 172.21.15.59:0/3061754372 shutdown_connections 2024-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.460+0000 7fd983b7f640 1 -- 172.21.15.59:0/3061754372 wait complete. 2024-06-29T15:57:38.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.461+0000 7fd983b7f640 1 Processor -- start 2024-06-29T15:57:38.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.461+0000 7fd983b7f640 1 -- start start 2024-06-29T15:57:38.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd983b7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 0x7fd97c1588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd983b7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd97c0ce950 0x7fd97c15dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd983b7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd97c0cf5c0 0x7fd97c154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd983b7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c0b7f60 con 0x7fd97c08c190 2024-06-29T15:57:38.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd983b7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c0b7de0 con 0x7fd97c0ce950 2024-06-29T15:57:38.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd983b7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd97c0b80e0 con 0x7fd97c0cf5c0 2024-06-29T15:57:38.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 0x7fd97c1588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 0x7fd97c1588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44698/0 (socket says 172.21.15.59:44698) 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 -- 172.21.15.59:0/3012987822 learned_addr learned my addr 172.21.15.59:0/3012987822 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98111a640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd97c0ce950 0x7fd97c15dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98211c640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd97c0cf5c0 0x7fd97c154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 -- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd97c0cf5c0 msgr2=0x7fd97c154c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd97c0cf5c0 0x7fd97c154c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 -- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd97c0ce950 msgr2=0x7fd97c15dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd97c0ce950 0x7fd97c15dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 -- 172.21.15.59:0/3012987822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97c158df0 con 0x7fd97c08c190 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98111a640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd97c0ce950 0x7fd97c15dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd98191b640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 0x7fd97c1588b0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fd970000c00 tx=0x7fd970003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.463+0000 7fd96e7fc640 1 -- 172.21.15.59:0/3012987822 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97003e030 con 0x7fd97c08c190 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.464+0000 7fd96e7fc640 1 -- 172.21.15.59:0/3012987822 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd970003e60 con 0x7fd97c08c190 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.464+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97c0ccdb0 con 0x7fd97c08c190 2024-06-29T15:57:38.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.464+0000 7fd96e7fc640 1 -- 172.21.15.59:0/3012987822 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970035040 con 0x7fd97c08c190 2024-06-29T15:57:38.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.464+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd97c0cd370 con 0x7fd97c08c190 2024-06-29T15:57:38.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.465+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd97c0cf080 con 0x7fd97c08c190 2024-06-29T15:57:38.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.465+0000 7fd96e7fc640 1 -- 172.21.15.59:0/3012987822 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fd970041030 con 0x7fd97c08c190 2024-06-29T15:57:38.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.466+0000 7fd96e7fc640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd960076420 0x7fd960078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.466+0000 7fd96e7fc640 1 -- 172.21.15.59:0/3012987822 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7fd9700b8720 con 0x7fd97c08c190 2024-06-29T15:57:38.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.466+0000 7fd98111a640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd960076420 0x7fd960078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.467+0000 7fd98111a640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd960076420 0x7fd960078930 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd97c15da50 tx=0x7fd978008040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.470+0000 7fd96e7fc640 1 -- 172.21.15.59:0/3012987822 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd9700818b0 con 0x7fd97c08c190 2024-06-29T15:57:38.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.701+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd97c15d7a0 con 0x7fd97c08c190 2024-06-29T15:57:38.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.702+0000 7fd96e7fc640 1 -- 172.21.15.59:0/3012987822 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6567 (secure 0 0 0) 0x7fd9700811b0 con 0x7fd97c08c190 2024-06-29T15:57:38.703 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:38.704 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":121,"btime":"2024-06-29T15:57:38:315315+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.589050+0000","modified":"2024-06-29T15:57:24.623107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11487":{"gid":11487,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1734341893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1734341893},{"type":"v1","addr":"172.21.15.59:6834","nonce":1734341893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11487,"qdb_cluster":[11487]},"id":13},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:57:38.704 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 121 2024-06-29T15:57:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd960076420 msgr2=0x7fd960078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd960076420 0x7fd960078930 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd97c15da50 tx=0x7fd978008040 comp rx=0 tx=0).stop 2024-06-29T15:57:38.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 msgr2=0x7fd97c1588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 0x7fd97c1588b0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fd970000c00 tx=0x7fd970003970 comp rx=0 tx=0).stop 2024-06-29T15:57:38.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 shutdown_connections 2024-06-29T15:57:38.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd960076420 0x7fd960078930 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd97c0cf5c0 0x7fd97c154c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd97c0ce950 0x7fd97c15dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 --2- 172.21.15.59:0/3012987822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd97c08c190 0x7fd97c1588b0 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 >> 172.21.15.59:0/3012987822 conn(0x7fd97c0b5610 msgr2=0x7fd97c0b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 shutdown_connections 2024-06-29T15:57:38.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.704+0000 7fd983b7f640 1 -- 172.21.15.59:0/3012987822 wait complete. 2024-06-29T15:57:38.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:38.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.834+0000 7f499becc640 1 -- 172.21.15.59:0/1559579007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f499408c190 msgr2=0x7f49940cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.834+0000 7f499becc640 1 --2- 172.21.15.59:0/1559579007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f499408c190 0x7f49940cf080 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f4984005fd0 tx=0x7f498402aa70 comp rx=0 tx=0).stop 2024-06-29T15:57:38.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.834+0000 7f499becc640 1 -- 172.21.15.59:0/1559579007 shutdown_connections 2024-06-29T15:57:38.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.834+0000 7f499becc640 1 --2- 172.21.15.59:0/1559579007 >> v2:172.21.15.153:3300/0 conn(0x7f49940ce950 0x7f49940d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.834+0000 7f499becc640 1 --2- 172.21.15.59:0/1559579007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f499408c190 0x7f49940cf080 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.834+0000 7f499becc640 1 -- 172.21.15.59:0/1559579007 >> 172.21.15.59:0/1559579007 conn(0x7f49940b5610 msgr2=0x7f49940b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.834+0000 7f499becc640 1 -- 172.21.15.59:0/1559579007 shutdown_connections 2024-06-29T15:57:38.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.834+0000 7f499becc640 1 -- 172.21.15.59:0/1559579007 wait complete. 2024-06-29T15:57:38.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.836+0000 7f499becc640 1 Processor -- start 2024-06-29T15:57:38.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.836+0000 7f499becc640 1 -- start start 2024-06-29T15:57:38.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f499becc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f499408c190 0x7f499415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f499becc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940ce950 0x7f499414af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f499becc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49940cf5c0 0x7f499414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f499becc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f49940b7f60 con 0x7f49940ce950 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f499becc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f49940b7de0 con 0x7f499408c190 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f499becc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f49940b80e0 con 0x7f49940cf5c0 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f4999467640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940ce950 0x7f499414af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f4999467640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940ce950 0x7f499414af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44712/0 (socket says 172.21.15.59:44712) 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f4999467640 1 -- 172.21.15.59:0/3209136162 learned_addr learned my addr 172.21.15.59:0/3209136162 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f499a469640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49940cf5c0 0x7f499414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.837+0000 7f4999c68640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f499408c190 0x7f499415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f4999467640 1 -- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49940cf5c0 msgr2=0x7f499414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f4999467640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49940cf5c0 0x7f499414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f4999467640 1 -- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f499408c190 msgr2=0x7f499415ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f4999467640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f499408c190 0x7f499415ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f4999467640 1 -- 172.21.15.59:0/3209136162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499415b380 con 0x7f49940ce950 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f499a469640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49940cf5c0 0x7f499414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f4999c68640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f499408c190 0x7f499415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f4999467640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940ce950 0x7f499414af80 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f498800c370 tx=0x7f498800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f49927fc640 1 -- 172.21.15.59:0/3209136162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988017070 con 0x7f49940ce950 2024-06-29T15:57:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f49927fc640 1 -- 172.21.15.59:0/3209136162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49880053c0 con 0x7f49940ce950 2024-06-29T15:57:38.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f49927fc640 1 -- 172.21.15.59:0/3209136162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988003850 con 0x7f49940ce950 2024-06-29T15:57:38.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49941490b0 con 0x7f49940ce950 2024-06-29T15:57:38.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.838+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4994149670 con 0x7f49940ce950 2024-06-29T15:57:38.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.839+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49940cf080 con 0x7f49940ce950 2024-06-29T15:57:38.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.840+0000 7f49927fc640 1 -- 172.21.15.59:0/3209136162 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f498801a070 con 0x7f49940ce950 2024-06-29T15:57:38.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.840+0000 7f49927fc640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f4980076420 0x7f4980078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.840+0000 7f49927fc640 1 -- 172.21.15.59:0/3209136162 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f49880971a0 con 0x7f49940ce950 2024-06-29T15:57:38.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.841+0000 7f4999c68640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f4980076420 0x7f4980078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.841+0000 7f4999c68640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f4980076420 0x7f4980078930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4984035000 tx=0x7f4984004ab0 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:38.845+0000 7f49927fc640 1 -- 172.21.15.59:0/3209136162 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f49880604d0 con 0x7f49940ce950 2024-06-29T15:57:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.038+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49940b6370 con 0x7f49940ce950 2024-06-29T15:57:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.039+0000 7f49927fc640 1 -- 172.21.15.59:0/3209136162 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) ==== 74+0+20776 (secure 0 0 0) 0x7f498805fdd0 con 0x7f49940ce950 2024-06-29T15:57:39.040 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:39.041 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":226,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:36.230341+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f4980076420 msgr2=0x7f4980078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f4980076420 0x7f4980078930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4984035000 tx=0x7f4984004ab0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940ce950 msgr2=0x7f499414af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940ce950 0x7f499414af80 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f498800c370 tx=0x7f498800c830 comp rx=0 tx=0).stop 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 shutdown_connections 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f4980076420 0x7f4980078930 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f49940cf5c0 0x7f499414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940ce950 0x7f499414af80 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 --2- 172.21.15.59:0/3209136162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f499408c190 0x7f499415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 >> 172.21.15.59:0/3209136162 conn(0x7f49940b5610 msgr2=0x7f49940c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 shutdown_connections 2024-06-29T15:57:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.042+0000 7f499becc640 1 -- 172.21.15.59:0/3209136162 wait complete. 2024-06-29T15:57:39.054 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:57:39.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:39.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.170+0000 7f26e9ff5640 1 -- 172.21.15.59:0/2334110500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40b3ed0 msgr2=0x7f26e415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.170+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/2334110500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40b3ed0 0x7f26e415e8d0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f26d8009830 tx=0x7f26d802a510 comp rx=0 tx=0).stop 2024-06-29T15:57:39.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.171+0000 7f26e9ff5640 1 -- 172.21.15.59:0/2334110500 shutdown_connections 2024-06-29T15:57:39.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.171+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/2334110500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40b3ed0 0x7f26e415e8d0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.171+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/2334110500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26e40d23c0 0x7f26e40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.171+0000 7f26e9ff5640 1 -- 172.21.15.59:0/2334110500 >> 172.21.15.59:0/2334110500 conn(0x7f26e40d3ee0 msgr2=0x7f26e40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.171+0000 7f26e9ff5640 1 -- 172.21.15.59:0/2334110500 shutdown_connections 2024-06-29T15:57:39.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.171+0000 7f26e9ff5640 1 -- 172.21.15.59:0/2334110500 wait complete. 2024-06-29T15:57:39.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.172+0000 7f26e9ff5640 1 Processor -- start 2024-06-29T15:57:39.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.173+0000 7f26e9ff5640 1 -- start start 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e9ff5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26e40b3ed0 0x7f26e41f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e9ff5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40d23c0 0x7f26e415d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e9ff5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e415ee10 0x7f26e415d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e9ff5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f26e4160240 con 0x7f26e40d23c0 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e9ff5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f26e41600c0 con 0x7f26e415ee10 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e9ff5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f26e41603c0 con 0x7f26e40b3ed0 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40d23c0 0x7f26e415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40d23c0 0x7f26e415d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44732/0 (socket says 172.21.15.59:44732) 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 -- 172.21.15.59:0/1793700501 learned_addr learned my addr 172.21.15.59:0/1793700501 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e37fe640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26e40b3ed0 0x7f26e41f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e3fff640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e415ee10 0x7f26e415d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 -- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26e40b3ed0 msgr2=0x7f26e41f4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26e40b3ed0 0x7f26e41f4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 -- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e415ee10 msgr2=0x7f26e415d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e415ee10 0x7f26e415d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 -- 172.21.15.59:0/1793700501 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e415deb0 con 0x7f26e40d23c0 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.174+0000 7f26e2ffd640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40d23c0 0x7f26e415d130 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000c00 tx=0x7f26d8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.175+0000 7f26cbfff640 1 -- 172.21.15.59:0/1793700501 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d8031d60 con 0x7f26e40d23c0 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.175+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26e415e190 con 0x7f26e40d23c0 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.175+0000 7f26cbfff640 1 -- 172.21.15.59:0/1793700501 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26d8033e10 con 0x7f26e40d23c0 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.175+0000 7f26cbfff640 1 -- 172.21.15.59:0/1793700501 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d80332b0 con 0x7f26e40d23c0 2024-06-29T15:57:39.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.175+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26e41e8c50 con 0x7f26e40d23c0 2024-06-29T15:57:39.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.176+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26e4158e50 con 0x7f26e40d23c0 2024-06-29T15:57:39.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.177+0000 7f26cbfff640 1 -- 172.21.15.59:0/1793700501 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f26d8033450 con 0x7f26e40d23c0 2024-06-29T15:57:39.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.177+0000 7f26cbfff640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f26d00763f0 0x7f26d0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.177+0000 7f26cbfff640 1 -- 172.21.15.59:0/1793700501 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f26d80b92f0 con 0x7f26e40d23c0 2024-06-29T15:57:39.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.177+0000 7f26e37fe640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f26d00763f0 0x7f26d0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.178+0000 7f26e37fe640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f26d00763f0 0x7f26d0078900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f26e40d2e40 tx=0x7f26d4005f90 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.181+0000 7f26cbfff640 1 -- 172.21.15.59:0/1793700501 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f26d803f030 con 0x7f26e40d23c0 2024-06-29T15:57:39.187 INFO:tasks.ceph.mds.b.smithi153.stderr:2024-06-29T15:57:39.187+0000 7f57f2ec5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:57:39.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.371+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f26e415dc40 con 0x7f26d00763f0 2024-06-29T15:57:39.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.380+0000 7f26cbfff640 1 -- 172.21.15.59:0/1793700501 <== mgr.9206 v2:172.21.15.153:6832/2780049427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+479293 (secure 0 0 0) 0x7f26e415dc40 con 0x7f26d00763f0 2024-06-29T15:57:39.386 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:39.389 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:57:39.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f26d00763f0 msgr2=0x7f26d0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f26d00763f0 0x7f26d0078900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f26e40d2e40 tx=0x7f26d4005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:39.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40d23c0 msgr2=0x7f26e415d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40d23c0 0x7f26e415d130 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000c00 tx=0x7f26d8033c20 comp rx=0 tx=0).stop 2024-06-29T15:57:39.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 shutdown_connections 2024-06-29T15:57:39.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f26d00763f0 0x7f26d0078900 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f26e415ee10 0x7f26e415d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e40d23c0 0x7f26e415d130 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 --2- 172.21.15.59:0/1793700501 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f26e40b3ed0 0x7f26e41f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 >> 172.21.15.59:0/1793700501 conn(0x7f26e40d3ee0 msgr2=0x7f26e4157280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 shutdown_connections 2024-06-29T15:57:39.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.384+0000 7f26e9ff5640 1 -- 172.21.15.59:0/1793700501 wait complete. 2024-06-29T15:57:39.398 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-06-29T15:57:37.769244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594021,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":424,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380760,"kb_used_data":23024,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593960,"statfs":{"total":773094113280,"available":772704215040,"internally_reserved":0,"allocated":23576576,"data_stored":12669500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":366202418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1243,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.488644"},"pg_stats":[{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"210'233","reported_seq":1103,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288044+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:57:33.288044+0000","last_peered":"2024-06-29T15:57:33.288044+0000","last_clean":"2024-06-29T15:57:33.288044+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:57:33.288044+0000","last_undegraded":"2024-06-29T15:57:33.288044+0000","last_fullsized":"2024-06-29T15:57:33.288044+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283944+0000","last_change":"2024-06-29T15:57:21.209163+0000","last_active":"2024-06-29T15:57:33.283944+0000","last_peered":"2024-06-29T15:57:33.283944+0000","last_clean":"2024-06-29T15:57:33.283944+0000","last_became_active":"2024-06-29T15:57:21.208825+0000","last_became_peered":"2024-06-29T15:57:21.208825+0000","last_unstale":"2024-06-29T15:57:33.283944+0000","last_undegraded":"2024-06-29T15:57:33.283944+0000","last_fullsized":"2024-06-29T15:57:33.283944+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:13.641268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290795+0000","last_change":"2024-06-29T15:57:22.230794+0000","last_active":"2024-06-29T15:57:33.290795+0000","last_peered":"2024-06-29T15:57:33.290795+0000","last_clean":"2024-06-29T15:57:33.290795+0000","last_became_active":"2024-06-29T15:57:22.230474+0000","last_became_peered":"2024-06-29T15:57:22.230474+0000","last_unstale":"2024-06-29T15:57:33.290795+0000","last_undegraded":"2024-06-29T15:57:33.290795+0000","last_fullsized":"2024-06-29T15:57:33.290795+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:43.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287008+0000","last_change":"2024-06-29T15:57:33.287008+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287008+0000","last_undegraded":"2024-06-29T15:57:33.287008+0000","last_fullsized":"2024-06-29T15:57:33.287008+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286721+0000","last_change":"2024-06-29T15:57:21.215589+0000","last_active":"2024-06-29T15:57:33.286721+0000","last_peered":"2024-06-29T15:57:33.286721+0000","last_clean":"2024-06-29T15:57:33.286721+0000","last_became_active":"2024-06-29T15:57:21.213719+0000","last_became_peered":"2024-06-29T15:57:21.213719+0000","last_unstale":"2024-06-29T15:57:33.286721+0000","last_undegraded":"2024-06-29T15:57:33.286721+0000","last_fullsized":"2024-06-29T15:57:33.286721+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:32.413502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283532+0000","last_change":"2024-06-29T15:57:22.223950+0000","last_active":"2024-06-29T15:57:33.283532+0000","last_peered":"2024-06-29T15:57:33.283532+0000","last_clean":"2024-06-29T15:57:33.283532+0000","last_became_active":"2024-06-29T15:57:22.223622+0000","last_became_peered":"2024-06-29T15:57:22.223622+0000","last_unstale":"2024-06-29T15:57:33.283532+0000","last_undegraded":"2024-06-29T15:57:33.283532+0000","last_fullsized":"2024-06-29T15:57:33.283532+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.556171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286422+0000","last_change":"2024-06-29T15:57:33.286422+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286422+0000","last_undegraded":"2024-06-29T15:57:33.286422+0000","last_fullsized":"2024-06-29T15:57:33.286422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290938+0000","last_change":"2024-06-29T15:57:21.212837+0000","last_active":"2024-06-29T15:57:33.290938+0000","last_peered":"2024-06-29T15:57:33.290938+0000","last_clean":"2024-06-29T15:57:33.290938+0000","last_became_active":"2024-06-29T15:57:21.212491+0000","last_became_peered":"2024-06-29T15:57:21.212491+0000","last_unstale":"2024-06-29T15:57:33.290938+0000","last_undegraded":"2024-06-29T15:57:33.290938+0000","last_fullsized":"2024-06-29T15:57:33.290938+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:35:53.641775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.292001+0000","last_change":"2024-06-29T15:57:22.237722+0000","last_active":"2024-06-29T15:57:33.292001+0000","last_peered":"2024-06-29T15:57:33.292001+0000","last_clean":"2024-06-29T15:57:33.292001+0000","last_became_active":"2024-06-29T15:57:22.237372+0000","last_became_peered":"2024-06-29T15:57:22.237372+0000","last_unstale":"2024-06-29T15:57:33.292001+0000","last_undegraded":"2024-06-29T15:57:33.292001+0000","last_fullsized":"2024-06-29T15:57:33.292001+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:24.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283133+0000","last_change":"2024-06-29T15:57:33.283133+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283133+0000","last_undegraded":"2024-06-29T15:57:33.283133+0000","last_fullsized":"2024-06-29T15:57:33.283133+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284094+0000","last_change":"2024-06-29T15:57:21.210563+0000","last_active":"2024-06-29T15:57:33.284094+0000","last_peered":"2024-06-29T15:57:33.284094+0000","last_clean":"2024-06-29T15:57:33.284094+0000","last_became_active":"2024-06-29T15:57:21.210243+0000","last_became_peered":"2024-06-29T15:57:21.210243+0000","last_unstale":"2024-06-29T15:57:33.284094+0000","last_undegraded":"2024-06-29T15:57:33.284094+0000","last_fullsized":"2024-06-29T15:57:33.284094+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.900240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291290+0000","last_change":"2024-06-29T15:57:22.244396+0000","last_active":"2024-06-29T15:57:33.291290+0000","last_peered":"2024-06-29T15:57:33.291290+0000","last_clean":"2024-06-29T15:57:33.291290+0000","last_became_active":"2024-06-29T15:57:22.244044+0000","last_became_peered":"2024-06-29T15:57:22.244044+0000","last_unstale":"2024-06-29T15:57:33.291290+0000","last_undegraded":"2024-06-29T15:57:33.291290+0000","last_fullsized":"2024-06-29T15:57:33.291290+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.981276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283374+0000","last_change":"2024-06-29T15:57:33.283374+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283374+0000","last_undegraded":"2024-06-29T15:57:33.283374+0000","last_fullsized":"2024-06-29T15:57:33.283374+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285468+0000","last_change":"2024-06-29T15:57:21.213113+0000","last_active":"2024-06-29T15:57:33.285468+0000","last_peered":"2024-06-29T15:57:33.285468+0000","last_clean":"2024-06-29T15:57:33.285468+0000","last_became_active":"2024-06-29T15:57:21.212819+0000","last_became_peered":"2024-06-29T15:57:21.212819+0000","last_unstale":"2024-06-29T15:57:33.285468+0000","last_undegraded":"2024-06-29T15:57:33.285468+0000","last_fullsized":"2024-06-29T15:57:33.285468+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:59.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285691+0000","last_change":"2024-06-29T15:57:22.229230+0000","last_active":"2024-06-29T15:57:33.285691+0000","last_peered":"2024-06-29T15:57:33.285691+0000","last_clean":"2024-06-29T15:57:33.285691+0000","last_became_active":"2024-06-29T15:57:22.228914+0000","last_became_peered":"2024-06-29T15:57:22.228914+0000","last_unstale":"2024-06-29T15:57:33.285691+0000","last_undegraded":"2024-06-29T15:57:33.285691+0000","last_fullsized":"2024-06-29T15:57:33.285691+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:09.918042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.285594+0000","last_change":"2024-06-29T15:57:33.285594+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.285594+0000","last_undegraded":"2024-06-29T15:57:33.285594+0000","last_fullsized":"2024-06-29T15:57:33.285594+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290684+0000","last_change":"2024-06-29T15:57:21.216399+0000","last_active":"2024-06-29T15:57:33.290684+0000","last_peered":"2024-06-29T15:57:33.290684+0000","last_clean":"2024-06-29T15:57:33.290684+0000","last_became_active":"2024-06-29T15:57:21.216104+0000","last_became_peered":"2024-06-29T15:57:21.216104+0000","last_unstale":"2024-06-29T15:57:33.290684+0000","last_undegraded":"2024-06-29T15:57:33.290684+0000","last_fullsized":"2024-06-29T15:57:33.290684+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290475+0000","last_change":"2024-06-29T15:57:22.227969+0000","last_active":"2024-06-29T15:57:33.290475+0000","last_peered":"2024-06-29T15:57:33.290475+0000","last_clean":"2024-06-29T15:57:33.290475+0000","last_became_active":"2024-06-29T15:57:22.227579+0000","last_became_peered":"2024-06-29T15:57:22.227579+0000","last_unstale":"2024-06-29T15:57:33.290475+0000","last_undegraded":"2024-06-29T15:57:33.290475+0000","last_fullsized":"2024-06-29T15:57:33.290475+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287295+0000","last_change":"2024-06-29T15:57:33.287295+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287295+0000","last_undegraded":"2024-06-29T15:57:33.287295+0000","last_fullsized":"2024-06-29T15:57:33.287295+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286605+0000","last_change":"2024-06-29T15:57:21.217039+0000","last_active":"2024-06-29T15:57:33.286605+0000","last_peered":"2024-06-29T15:57:33.286605+0000","last_clean":"2024-06-29T15:57:33.286605+0000","last_became_active":"2024-06-29T15:57:21.216542+0000","last_became_peered":"2024-06-29T15:57:21.216542+0000","last_unstale":"2024-06-29T15:57:33.286605+0000","last_undegraded":"2024-06-29T15:57:33.286605+0000","last_fullsized":"2024-06-29T15:57:33.286605+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:21.729257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283680+0000","last_change":"2024-06-29T15:57:22.233919+0000","last_active":"2024-06-29T15:57:33.283680+0000","last_peered":"2024-06-29T15:57:33.283680+0000","last_clean":"2024-06-29T15:57:33.283680+0000","last_became_active":"2024-06-29T15:57:22.231028+0000","last_became_peered":"2024-06-29T15:57:22.231028+0000","last_unstale":"2024-06-29T15:57:33.283680+0000","last_undegraded":"2024-06-29T15:57:33.283680+0000","last_fullsized":"2024-06-29T15:57:33.283680+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:34:24.910914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283097+0000","last_change":"2024-06-29T15:57:33.283097+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283097+0000","last_undegraded":"2024-06-29T15:57:33.283097+0000","last_fullsized":"2024-06-29T15:57:33.283097+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"222'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286084+0000","last_change":"2024-06-29T15:57:21.203822+0000","last_active":"2024-06-29T15:57:33.286084+0000","last_peered":"2024-06-29T15:57:33.286084+0000","last_clean":"2024-06-29T15:57:33.286084+0000","last_became_active":"2024-06-29T15:57:21.203517+0000","last_became_peered":"2024-06-29T15:57:21.203517+0000","last_unstale":"2024-06-29T15:57:33.286084+0000","last_undegraded":"2024-06-29T15:57:33.286084+0000","last_fullsized":"2024-06-29T15:57:33.286084+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:05.810997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.282967+0000","last_change":"2024-06-29T15:57:22.223140+0000","last_active":"2024-06-29T15:57:33.282967+0000","last_peered":"2024-06-29T15:57:33.282967+0000","last_clean":"2024-06-29T15:57:33.282967+0000","last_became_active":"2024-06-29T15:57:22.222839+0000","last_became_peered":"2024-06-29T15:57:22.222839+0000","last_unstale":"2024-06-29T15:57:33.282967+0000","last_undegraded":"2024-06-29T15:57:33.282967+0000","last_fullsized":"2024-06-29T15:57:33.282967+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:17.520475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283134+0000","last_change":"2024-06-29T15:57:33.283134+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283134+0000","last_undegraded":"2024-06-29T15:57:33.283134+0000","last_fullsized":"2024-06-29T15:57:33.283134+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284126+0000","last_change":"2024-06-29T15:57:21.212745+0000","last_active":"2024-06-29T15:57:33.284126+0000","last_peered":"2024-06-29T15:57:33.284126+0000","last_clean":"2024-06-29T15:57:33.284126+0000","last_became_active":"2024-06-29T15:57:21.212335+0000","last_became_peered":"2024-06-29T15:57:21.212335+0000","last_unstale":"2024-06-29T15:57:33.284126+0000","last_undegraded":"2024-06-29T15:57:33.284126+0000","last_fullsized":"2024-06-29T15:57:33.284126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:07.049637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285779+0000","last_change":"2024-06-29T15:57:22.224294+0000","last_active":"2024-06-29T15:57:33.285779+0000","last_peered":"2024-06-29T15:57:33.285779+0000","last_clean":"2024-06-29T15:57:33.285779+0000","last_became_active":"2024-06-29T15:57:22.223922+0000","last_became_peered":"2024-06-29T15:57:22.223922+0000","last_unstale":"2024-06-29T15:57:33.285779+0000","last_undegraded":"2024-06-29T15:57:33.285779+0000","last_fullsized":"2024-06-29T15:57:33.285779+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:46.725456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286716+0000","last_change":"2024-06-29T15:57:33.286716+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286716+0000","last_undegraded":"2024-06-29T15:57:33.286716+0000","last_fullsized":"2024-06-29T15:57:33.286716+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290020+0000","last_change":"2024-06-29T15:57:21.211203+0000","last_active":"2024-06-29T15:57:33.290020+0000","last_peered":"2024-06-29T15:57:33.290020+0000","last_clean":"2024-06-29T15:57:33.290020+0000","last_became_active":"2024-06-29T15:57:21.210898+0000","last_became_peered":"2024-06-29T15:57:21.210898+0000","last_unstale":"2024-06-29T15:57:33.290020+0000","last_undegraded":"2024-06-29T15:57:33.290020+0000","last_fullsized":"2024-06-29T15:57:33.290020+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:17.270875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286704+0000","last_change":"2024-06-29T15:57:22.213920+0000","last_active":"2024-06-29T15:57:33.286704+0000","last_peered":"2024-06-29T15:57:33.286704+0000","last_clean":"2024-06-29T15:57:33.286704+0000","last_became_active":"2024-06-29T15:57:22.213583+0000","last_became_peered":"2024-06-29T15:57:22.213583+0000","last_unstale":"2024-06-29T15:57:33.286704+0000","last_undegraded":"2024-06-29T15:57:33.286704+0000","last_fullsized":"2024-06-29T15:57:33.286704+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:31.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.285625+0000","last_change":"2024-06-29T15:57:33.285625+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.285625+0000","last_undegraded":"2024-06-29T15:57:33.285625+0000","last_fullsized":"2024-06-29T15:57:33.285625+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287942+0000","last_change":"2024-06-29T15:57:21.212085+0000","last_active":"2024-06-29T15:57:33.287942+0000","last_peered":"2024-06-29T15:57:33.287942+0000","last_clean":"2024-06-29T15:57:33.287942+0000","last_became_active":"2024-06-29T15:57:21.211731+0000","last_became_peered":"2024-06-29T15:57:21.211731+0000","last_unstale":"2024-06-29T15:57:33.287942+0000","last_undegraded":"2024-06-29T15:57:33.287942+0000","last_fullsized":"2024-06-29T15:57:33.287942+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:36.061054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284429+0000","last_change":"2024-06-29T15:57:22.228211+0000","last_active":"2024-06-29T15:57:33.284429+0000","last_peered":"2024-06-29T15:57:33.284429+0000","last_clean":"2024-06-29T15:57:33.284429+0000","last_became_active":"2024-06-29T15:57:22.227519+0000","last_became_peered":"2024-06-29T15:57:22.227519+0000","last_unstale":"2024-06-29T15:57:33.284429+0000","last_undegraded":"2024-06-29T15:57:33.284429+0000","last_fullsized":"2024-06-29T15:57:33.284429+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:16:35.999038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288476+0000","last_change":"2024-06-29T15:57:33.288476+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288476+0000","last_undegraded":"2024-06-29T15:57:33.288476+0000","last_fullsized":"2024-06-29T15:57:33.288476+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289681+0000","last_change":"2024-06-29T15:57:21.208592+0000","last_active":"2024-06-29T15:57:33.289681+0000","last_peered":"2024-06-29T15:57:33.289681+0000","last_clean":"2024-06-29T15:57:33.289681+0000","last_became_active":"2024-06-29T15:57:21.208156+0000","last_became_peered":"2024-06-29T15:57:21.208156+0000","last_unstale":"2024-06-29T15:57:33.289681+0000","last_undegraded":"2024-06-29T15:57:33.289681+0000","last_fullsized":"2024-06-29T15:57:33.289681+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:20.325402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291188+0000","last_change":"2024-06-29T15:57:22.238829+0000","last_active":"2024-06-29T15:57:33.291188+0000","last_peered":"2024-06-29T15:57:33.291188+0000","last_clean":"2024-06-29T15:57:33.291188+0000","last_became_active":"2024-06-29T15:57:22.238520+0000","last_became_peered":"2024-06-29T15:57:22.238520+0000","last_unstale":"2024-06-29T15:57:33.291188+0000","last_undegraded":"2024-06-29T15:57:33.291188+0000","last_fullsized":"2024-06-29T15:57:33.291188+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:22.127384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.284230+0000","last_change":"2024-06-29T15:57:33.284230+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.284230+0000","last_undegraded":"2024-06-29T15:57:33.284230+0000","last_fullsized":"2024-06-29T15:57:33.284230+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287695+0000","last_change":"2024-06-29T15:57:21.215358+0000","last_active":"2024-06-29T15:57:33.287695+0000","last_peered":"2024-06-29T15:57:33.287695+0000","last_clean":"2024-06-29T15:57:33.287695+0000","last_became_active":"2024-06-29T15:57:21.213979+0000","last_became_peered":"2024-06-29T15:57:21.213979+0000","last_unstale":"2024-06-29T15:57:33.287695+0000","last_undegraded":"2024-06-29T15:57:33.287695+0000","last_fullsized":"2024-06-29T15:57:33.287695+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:42.446135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289780+0000","last_change":"2024-06-29T15:57:22.231883+0000","last_active":"2024-06-29T15:57:33.289780+0000","last_peered":"2024-06-29T15:57:33.289780+0000","last_clean":"2024-06-29T15:57:33.289780+0000","last_became_active":"2024-06-29T15:57:22.231588+0000","last_became_peered":"2024-06-29T15:57:22.231588+0000","last_unstale":"2024-06-29T15:57:33.289780+0000","last_undegraded":"2024-06-29T15:57:33.289780+0000","last_fullsized":"2024-06-29T15:57:33.289780+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:17.705760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283133+0000","last_change":"2024-06-29T15:57:33.283133+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283133+0000","last_undegraded":"2024-06-29T15:57:33.283133+0000","last_fullsized":"2024-06-29T15:57:33.283133+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287693+0000","last_change":"2024-06-29T15:57:21.213193+0000","last_active":"2024-06-29T15:57:33.287693+0000","last_peered":"2024-06-29T15:57:33.287693+0000","last_clean":"2024-06-29T15:57:33.287693+0000","last_became_active":"2024-06-29T15:57:21.212838+0000","last_became_peered":"2024-06-29T15:57:21.212838+0000","last_unstale":"2024-06-29T15:57:33.287693+0000","last_undegraded":"2024-06-29T15:57:33.287693+0000","last_fullsized":"2024-06-29T15:57:33.287693+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:54.574391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285342+0000","last_change":"2024-06-29T15:57:22.217630+0000","last_active":"2024-06-29T15:57:33.285342+0000","last_peered":"2024-06-29T15:57:33.285342+0000","last_clean":"2024-06-29T15:57:33.285342+0000","last_became_active":"2024-06-29T15:57:22.217303+0000","last_became_peered":"2024-06-29T15:57:22.217303+0000","last_unstale":"2024-06-29T15:57:33.285342+0000","last_undegraded":"2024-06-29T15:57:33.285342+0000","last_fullsized":"2024-06-29T15:57:33.285342+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:05.554814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286544+0000","last_change":"2024-06-29T15:57:33.286544+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286544+0000","last_undegraded":"2024-06-29T15:57:33.286544+0000","last_fullsized":"2024-06-29T15:57:33.286544+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290097+0000","last_change":"2024-06-29T15:57:21.210528+0000","last_active":"2024-06-29T15:57:33.290097+0000","last_peered":"2024-06-29T15:57:33.290097+0000","last_clean":"2024-06-29T15:57:33.290097+0000","last_became_active":"2024-06-29T15:57:21.208691+0000","last_became_peered":"2024-06-29T15:57:21.208691+0000","last_unstale":"2024-06-29T15:57:33.290097+0000","last_undegraded":"2024-06-29T15:57:33.290097+0000","last_fullsized":"2024-06-29T15:57:33.290097+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.762590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290667+0000","last_change":"2024-06-29T15:57:22.232727+0000","last_active":"2024-06-29T15:57:33.290667+0000","last_peered":"2024-06-29T15:57:33.290667+0000","last_clean":"2024-06-29T15:57:33.290667+0000","last_became_active":"2024-06-29T15:57:22.232392+0000","last_became_peered":"2024-06-29T15:57:22.232392+0000","last_unstale":"2024-06-29T15:57:33.290667+0000","last_undegraded":"2024-06-29T15:57:33.290667+0000","last_fullsized":"2024-06-29T15:57:33.290667+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:54.805984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283760+0000","last_change":"2024-06-29T15:57:33.283760+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283760+0000","last_undegraded":"2024-06-29T15:57:33.283760+0000","last_fullsized":"2024-06-29T15:57:33.283760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291169+0000","last_change":"2024-06-29T15:57:21.216179+0000","last_active":"2024-06-29T15:57:33.291169+0000","last_peered":"2024-06-29T15:57:33.291169+0000","last_clean":"2024-06-29T15:57:33.291169+0000","last_became_active":"2024-06-29T15:57:21.215876+0000","last_became_peered":"2024-06-29T15:57:21.215876+0000","last_unstale":"2024-06-29T15:57:33.291169+0000","last_undegraded":"2024-06-29T15:57:33.291169+0000","last_fullsized":"2024-06-29T15:57:33.291169+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:37.942186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290682+0000","last_change":"2024-06-29T15:57:22.225793+0000","last_active":"2024-06-29T15:57:33.290682+0000","last_peered":"2024-06-29T15:57:33.290682+0000","last_clean":"2024-06-29T15:57:33.290682+0000","last_became_active":"2024-06-29T15:57:22.224806+0000","last_became_peered":"2024-06-29T15:57:22.224806+0000","last_unstale":"2024-06-29T15:57:33.290682+0000","last_undegraded":"2024-06-29T15:57:33.290682+0000","last_fullsized":"2024-06-29T15:57:33.290682+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:03.667477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286140+0000","last_change":"2024-06-29T15:57:33.286140+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286140+0000","last_undegraded":"2024-06-29T15:57:33.286140+0000","last_fullsized":"2024-06-29T15:57:33.286140+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291389+0000","last_change":"2024-06-29T15:57:21.212718+0000","last_active":"2024-06-29T15:57:33.291389+0000","last_peered":"2024-06-29T15:57:33.291389+0000","last_clean":"2024-06-29T15:57:33.291389+0000","last_became_active":"2024-06-29T15:57:21.212406+0000","last_became_peered":"2024-06-29T15:57:21.212406+0000","last_unstale":"2024-06-29T15:57:33.291389+0000","last_undegraded":"2024-06-29T15:57:33.291389+0000","last_fullsized":"2024-06-29T15:57:33.291389+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:00.180622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285664+0000","last_change":"2024-06-29T15:57:22.238347+0000","last_active":"2024-06-29T15:57:33.285664+0000","last_peered":"2024-06-29T15:57:33.285664+0000","last_clean":"2024-06-29T15:57:33.285664+0000","last_became_active":"2024-06-29T15:57:22.236896+0000","last_became_peered":"2024-06-29T15:57:22.236896+0000","last_unstale":"2024-06-29T15:57:33.285664+0000","last_undegraded":"2024-06-29T15:57:33.285664+0000","last_fullsized":"2024-06-29T15:57:33.285664+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:39.739194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.285610+0000","last_change":"2024-06-29T15:57:33.285610+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.285610+0000","last_undegraded":"2024-06-29T15:57:33.285610+0000","last_fullsized":"2024-06-29T15:57:33.285610+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"221'8","reported_seq":24,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286557+0000","last_change":"2024-06-29T15:57:21.216156+0000","last_active":"2024-06-29T15:57:33.286557+0000","last_peered":"2024-06-29T15:57:33.286557+0000","last_clean":"2024-06-29T15:57:33.286557+0000","last_became_active":"2024-06-29T15:57:21.214088+0000","last_became_peered":"2024-06-29T15:57:21.214088+0000","last_unstale":"2024-06-29T15:57:33.286557+0000","last_undegraded":"2024-06-29T15:57:33.286557+0000","last_fullsized":"2024-06-29T15:57:33.286557+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:30.314243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291583+0000","last_change":"2024-06-29T15:57:22.223112+0000","last_active":"2024-06-29T15:57:33.291583+0000","last_peered":"2024-06-29T15:57:33.291583+0000","last_clean":"2024-06-29T15:57:33.291583+0000","last_became_active":"2024-06-29T15:57:22.222632+0000","last_became_peered":"2024-06-29T15:57:22.222632+0000","last_unstale":"2024-06-29T15:57:33.291583+0000","last_undegraded":"2024-06-29T15:57:33.291583+0000","last_fullsized":"2024-06-29T15:57:33.291583+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:05.408776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287648+0000","last_change":"2024-06-29T15:57:33.287648+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287648+0000","last_undegraded":"2024-06-29T15:57:33.287648+0000","last_fullsized":"2024-06-29T15:57:33.287648+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283547+0000","last_change":"2024-06-29T15:57:21.215032+0000","last_active":"2024-06-29T15:57:33.283547+0000","last_peered":"2024-06-29T15:57:33.283547+0000","last_clean":"2024-06-29T15:57:33.283547+0000","last_became_active":"2024-06-29T15:57:21.214557+0000","last_became_peered":"2024-06-29T15:57:21.214557+0000","last_unstale":"2024-06-29T15:57:33.283547+0000","last_undegraded":"2024-06-29T15:57:33.283547+0000","last_fullsized":"2024-06-29T15:57:33.283547+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284364+0000","last_change":"2024-06-29T15:57:22.228484+0000","last_active":"2024-06-29T15:57:33.284364+0000","last_peered":"2024-06-29T15:57:33.284364+0000","last_clean":"2024-06-29T15:57:33.284364+0000","last_became_active":"2024-06-29T15:57:22.227880+0000","last_became_peered":"2024-06-29T15:57:22.227880+0000","last_unstale":"2024-06-29T15:57:33.284364+0000","last_undegraded":"2024-06-29T15:57:33.284364+0000","last_fullsized":"2024-06-29T15:57:33.284364+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:21.749071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287403+0000","last_change":"2024-06-29T15:57:33.287403+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287403+0000","last_undegraded":"2024-06-29T15:57:33.287403+0000","last_fullsized":"2024-06-29T15:57:33.287403+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283118+0000","last_change":"2024-06-29T15:57:21.203410+0000","last_active":"2024-06-29T15:57:33.283118+0000","last_peered":"2024-06-29T15:57:33.283118+0000","last_clean":"2024-06-29T15:57:33.283118+0000","last_became_active":"2024-06-29T15:57:21.203034+0000","last_became_peered":"2024-06-29T15:57:21.203034+0000","last_unstale":"2024-06-29T15:57:33.283118+0000","last_undegraded":"2024-06-29T15:57:33.283118+0000","last_fullsized":"2024-06-29T15:57:33.283118+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:52.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283462+0000","last_change":"2024-06-29T15:57:22.241881+0000","last_active":"2024-06-29T15:57:33.283462+0000","last_peered":"2024-06-29T15:57:33.283462+0000","last_clean":"2024-06-29T15:57:33.283462+0000","last_became_active":"2024-06-29T15:57:22.241549+0000","last_became_peered":"2024-06-29T15:57:22.241549+0000","last_unstale":"2024-06-29T15:57:33.283462+0000","last_undegraded":"2024-06-29T15:57:33.283462+0000","last_fullsized":"2024-06-29T15:57:33.283462+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:25.352328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288277+0000","last_change":"2024-06-29T15:57:33.288277+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288277+0000","last_undegraded":"2024-06-29T15:57:33.288277+0000","last_fullsized":"2024-06-29T15:57:33.288277+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289594+0000","last_change":"2024-06-29T15:57:21.215892+0000","last_active":"2024-06-29T15:57:33.289594+0000","last_peered":"2024-06-29T15:57:33.289594+0000","last_clean":"2024-06-29T15:57:33.289594+0000","last_became_active":"2024-06-29T15:57:21.215250+0000","last_became_peered":"2024-06-29T15:57:21.215250+0000","last_unstale":"2024-06-29T15:57:33.289594+0000","last_undegraded":"2024-06-29T15:57:33.289594+0000","last_fullsized":"2024-06-29T15:57:33.289594+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:23.234622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289678+0000","last_change":"2024-06-29T15:57:22.234802+0000","last_active":"2024-06-29T15:57:33.289678+0000","last_peered":"2024-06-29T15:57:33.289678+0000","last_clean":"2024-06-29T15:57:33.289678+0000","last_became_active":"2024-06-29T15:57:22.234492+0000","last_became_peered":"2024-06-29T15:57:22.234492+0000","last_unstale":"2024-06-29T15:57:33.289678+0000","last_undegraded":"2024-06-29T15:57:33.289678+0000","last_fullsized":"2024-06-29T15:57:33.289678+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:37.928014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286760+0000","last_change":"2024-06-29T15:57:33.286760+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286760+0000","last_undegraded":"2024-06-29T15:57:33.286760+0000","last_fullsized":"2024-06-29T15:57:33.286760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"222'8","reported_seq":24,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291177+0000","last_change":"2024-06-29T15:57:21.217755+0000","last_active":"2024-06-29T15:57:33.291177+0000","last_peered":"2024-06-29T15:57:33.291177+0000","last_clean":"2024-06-29T15:57:33.291177+0000","last_became_active":"2024-06-29T15:57:21.216983+0000","last_became_peered":"2024-06-29T15:57:21.216983+0000","last_unstale":"2024-06-29T15:57:33.291177+0000","last_undegraded":"2024-06-29T15:57:33.291177+0000","last_fullsized":"2024-06-29T15:57:33.291177+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:31.889343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287603+0000","last_change":"2024-06-29T15:57:22.224600+0000","last_active":"2024-06-29T15:57:33.287603+0000","last_peered":"2024-06-29T15:57:33.287603+0000","last_clean":"2024-06-29T15:57:33.287603+0000","last_became_active":"2024-06-29T15:57:22.224260+0000","last_became_peered":"2024-06-29T15:57:22.224260+0000","last_unstale":"2024-06-29T15:57:33.287603+0000","last_undegraded":"2024-06-29T15:57:33.287603+0000","last_fullsized":"2024-06-29T15:57:33.287603+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:21.773370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.282975+0000","last_change":"2024-06-29T15:57:33.282975+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.282975+0000","last_undegraded":"2024-06-29T15:57:33.282975+0000","last_fullsized":"2024-06-29T15:57:33.282975+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"221'3","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284539+0000","last_change":"2024-06-29T15:57:21.210270+0000","last_active":"2024-06-29T15:57:33.284539+0000","last_peered":"2024-06-29T15:57:33.284539+0000","last_clean":"2024-06-29T15:57:33.284539+0000","last_became_active":"2024-06-29T15:57:21.209731+0000","last_became_peered":"2024-06-29T15:57:21.209731+0000","last_unstale":"2024-06-29T15:57:33.284539+0000","last_undegraded":"2024-06-29T15:57:33.284539+0000","last_fullsized":"2024-06-29T15:57:33.284539+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:18.182233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283812+0000","last_change":"2024-06-29T15:57:22.231011+0000","last_active":"2024-06-29T15:57:33.283812+0000","last_peered":"2024-06-29T15:57:33.283812+0000","last_clean":"2024-06-29T15:57:33.283812+0000","last_became_active":"2024-06-29T15:57:22.230675+0000","last_became_peered":"2024-06-29T15:57:22.230675+0000","last_unstale":"2024-06-29T15:57:33.283812+0000","last_undegraded":"2024-06-29T15:57:33.283812+0000","last_fullsized":"2024-06-29T15:57:33.283812+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:09.478769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.284342+0000","last_change":"2024-06-29T15:57:33.284342+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.284342+0000","last_undegraded":"2024-06-29T15:57:33.284342+0000","last_fullsized":"2024-06-29T15:57:33.284342+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.292126+0000","last_change":"2024-06-29T15:57:21.214536+0000","last_active":"2024-06-29T15:57:33.292126+0000","last_peered":"2024-06-29T15:57:33.292126+0000","last_clean":"2024-06-29T15:57:33.292126+0000","last_became_active":"2024-06-29T15:57:21.214227+0000","last_became_peered":"2024-06-29T15:57:21.214227+0000","last_unstale":"2024-06-29T15:57:33.292126+0000","last_undegraded":"2024-06-29T15:57:33.292126+0000","last_fullsized":"2024-06-29T15:57:33.292126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:25.839522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.282963+0000","last_change":"2024-06-29T15:57:22.219192+0000","last_active":"2024-06-29T15:57:33.282963+0000","last_peered":"2024-06-29T15:57:33.282963+0000","last_clean":"2024-06-29T15:57:33.282963+0000","last_became_active":"2024-06-29T15:57:22.218844+0000","last_became_peered":"2024-06-29T15:57:22.218844+0000","last_unstale":"2024-06-29T15:57:33.282963+0000","last_undegraded":"2024-06-29T15:57:33.282963+0000","last_fullsized":"2024-06-29T15:57:33.282963+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:50.717283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283773+0000","last_change":"2024-06-29T15:57:33.283773+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283773+0000","last_undegraded":"2024-06-29T15:57:33.283773+0000","last_fullsized":"2024-06-29T15:57:33.283773+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288108+0000","last_change":"2024-06-29T15:57:21.218814+0000","last_active":"2024-06-29T15:57:33.288108+0000","last_peered":"2024-06-29T15:57:33.288108+0000","last_clean":"2024-06-29T15:57:33.288108+0000","last_became_active":"2024-06-29T15:57:21.218495+0000","last_became_peered":"2024-06-29T15:57:21.218495+0000","last_unstale":"2024-06-29T15:57:33.288108+0000","last_undegraded":"2024-06-29T15:57:33.288108+0000","last_fullsized":"2024-06-29T15:57:33.288108+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:04.680014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286900+0000","last_change":"2024-06-29T15:57:22.235312+0000","last_active":"2024-06-29T15:57:33.286900+0000","last_peered":"2024-06-29T15:57:33.286900+0000","last_clean":"2024-06-29T15:57:33.286900+0000","last_became_active":"2024-06-29T15:57:22.234828+0000","last_became_peered":"2024-06-29T15:57:22.234828+0000","last_unstale":"2024-06-29T15:57:33.286900+0000","last_undegraded":"2024-06-29T15:57:33.286900+0000","last_fullsized":"2024-06-29T15:57:33.286900+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:39.661714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288150+0000","last_change":"2024-06-29T15:57:33.288150+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288150+0000","last_undegraded":"2024-06-29T15:57:33.288150+0000","last_fullsized":"2024-06-29T15:57:33.288150+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283149+0000","last_change":"2024-06-29T15:57:33.283149+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283149+0000","last_undegraded":"2024-06-29T15:57:33.283149+0000","last_fullsized":"2024-06-29T15:57:33.283149+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.282789+0000","last_change":"2024-06-29T15:57:21.208896+0000","last_active":"2024-06-29T15:57:33.282789+0000","last_peered":"2024-06-29T15:57:33.282789+0000","last_clean":"2024-06-29T15:57:33.282789+0000","last_became_active":"2024-06-29T15:57:21.208555+0000","last_became_peered":"2024-06-29T15:57:21.208555+0000","last_unstale":"2024-06-29T15:57:33.282789+0000","last_undegraded":"2024-06-29T15:57:33.282789+0000","last_fullsized":"2024-06-29T15:57:33.282789+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283991+0000","last_change":"2024-06-29T15:57:22.234743+0000","last_active":"2024-06-29T15:57:33.283991+0000","last_peered":"2024-06-29T15:57:33.283991+0000","last_clean":"2024-06-29T15:57:33.283991+0000","last_became_active":"2024-06-29T15:57:22.233091+0000","last_became_peered":"2024-06-29T15:57:22.233091+0000","last_unstale":"2024-06-29T15:57:33.283991+0000","last_undegraded":"2024-06-29T15:57:33.283991+0000","last_fullsized":"2024-06-29T15:57:33.283991+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:46.131681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.285686+0000","last_change":"2024-06-29T15:57:33.285686+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.285686+0000","last_undegraded":"2024-06-29T15:57:33.285686+0000","last_fullsized":"2024-06-29T15:57:33.285686+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288695+0000","last_change":"2024-06-29T15:57:21.212034+0000","last_active":"2024-06-29T15:57:33.288695+0000","last_peered":"2024-06-29T15:57:33.288695+0000","last_clean":"2024-06-29T15:57:33.288695+0000","last_became_active":"2024-06-29T15:57:21.211762+0000","last_became_peered":"2024-06-29T15:57:21.211762+0000","last_unstale":"2024-06-29T15:57:33.288695+0000","last_undegraded":"2024-06-29T15:57:33.288695+0000","last_fullsized":"2024-06-29T15:57:33.288695+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:36.775853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287907+0000","last_change":"2024-06-29T15:57:22.214168+0000","last_active":"2024-06-29T15:57:33.287907+0000","last_peered":"2024-06-29T15:57:33.287907+0000","last_clean":"2024-06-29T15:57:33.287907+0000","last_became_active":"2024-06-29T15:57:22.213780+0000","last_became_peered":"2024-06-29T15:57:22.213780+0000","last_unstale":"2024-06-29T15:57:33.287907+0000","last_undegraded":"2024-06-29T15:57:33.287907+0000","last_fullsized":"2024-06-29T15:57:33.287907+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:01.904725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289552+0000","last_change":"2024-06-29T15:57:21.211113+0000","last_active":"2024-06-29T15:57:33.289552+0000","last_peered":"2024-06-29T15:57:33.289552+0000","last_clean":"2024-06-29T15:57:33.289552+0000","last_became_active":"2024-06-29T15:57:21.210817+0000","last_became_peered":"2024-06-29T15:57:21.210817+0000","last_unstale":"2024-06-29T15:57:33.289552+0000","last_undegraded":"2024-06-29T15:57:33.289552+0000","last_fullsized":"2024-06-29T15:57:33.289552+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:26.176884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287567+0000","last_change":"2024-06-29T15:57:22.235721+0000","last_active":"2024-06-29T15:57:33.287567+0000","last_peered":"2024-06-29T15:57:33.287567+0000","last_clean":"2024-06-29T15:57:33.287567+0000","last_became_active":"2024-06-29T15:57:22.235380+0000","last_became_peered":"2024-06-29T15:57:22.235380+0000","last_unstale":"2024-06-29T15:57:33.287567+0000","last_undegraded":"2024-06-29T15:57:33.287567+0000","last_fullsized":"2024-06-29T15:57:33.287567+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:38.144376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288700+0000","last_change":"2024-06-29T15:57:33.288700+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288700+0000","last_undegraded":"2024-06-29T15:57:33.288700+0000","last_fullsized":"2024-06-29T15:57:33.288700+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284087+0000","last_change":"2024-06-29T15:57:21.210643+0000","last_active":"2024-06-29T15:57:33.284087+0000","last_peered":"2024-06-29T15:57:33.284087+0000","last_clean":"2024-06-29T15:57:33.284087+0000","last_became_active":"2024-06-29T15:57:21.209805+0000","last_became_peered":"2024-06-29T15:57:21.209805+0000","last_unstale":"2024-06-29T15:57:33.284087+0000","last_undegraded":"2024-06-29T15:57:33.284087+0000","last_fullsized":"2024-06-29T15:57:33.284087+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:27.618049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287540+0000","last_change":"2024-06-29T15:57:22.226181+0000","last_active":"2024-06-29T15:57:33.287540+0000","last_peered":"2024-06-29T15:57:33.287540+0000","last_clean":"2024-06-29T15:57:33.287540+0000","last_became_active":"2024-06-29T15:57:22.225855+0000","last_became_peered":"2024-06-29T15:57:22.225855+0000","last_unstale":"2024-06-29T15:57:33.287540+0000","last_undegraded":"2024-06-29T15:57:33.287540+0000","last_fullsized":"2024-06-29T15:57:33.287540+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:37.067580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287076+0000","last_change":"2024-06-29T15:57:33.287076+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287076+0000","last_undegraded":"2024-06-29T15:57:33.287076+0000","last_fullsized":"2024-06-29T15:57:33.287076+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290044+0000","last_change":"2024-06-29T15:57:21.210602+0000","last_active":"2024-06-29T15:57:33.290044+0000","last_peered":"2024-06-29T15:57:33.290044+0000","last_clean":"2024-06-29T15:57:33.290044+0000","last_became_active":"2024-06-29T15:57:21.210296+0000","last_became_peered":"2024-06-29T15:57:21.210296+0000","last_unstale":"2024-06-29T15:57:33.290044+0000","last_undegraded":"2024-06-29T15:57:33.290044+0000","last_fullsized":"2024-06-29T15:57:33.290044+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:07.128352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285263+0000","last_change":"2024-06-29T15:57:22.213237+0000","last_active":"2024-06-29T15:57:33.285263+0000","last_peered":"2024-06-29T15:57:33.285263+0000","last_clean":"2024-06-29T15:57:33.285263+0000","last_became_active":"2024-06-29T15:57:22.212908+0000","last_became_peered":"2024-06-29T15:57:22.212908+0000","last_unstale":"2024-06-29T15:57:33.285263+0000","last_undegraded":"2024-06-29T15:57:33.285263+0000","last_fullsized":"2024-06-29T15:57:33.285263+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:31.343924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286084+0000","last_change":"2024-06-29T15:57:33.286084+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286084+0000","last_undegraded":"2024-06-29T15:57:33.286084+0000","last_fullsized":"2024-06-29T15:57:33.286084+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291011+0000","last_change":"2024-06-29T15:57:21.213288+0000","last_active":"2024-06-29T15:57:33.291011+0000","last_peered":"2024-06-29T15:57:33.291011+0000","last_clean":"2024-06-29T15:57:33.291011+0000","last_became_active":"2024-06-29T15:57:21.212954+0000","last_became_peered":"2024-06-29T15:57:21.212954+0000","last_unstale":"2024-06-29T15:57:33.291011+0000","last_undegraded":"2024-06-29T15:57:33.291011+0000","last_fullsized":"2024-06-29T15:57:33.291011+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:24.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290567+0000","last_change":"2024-06-29T15:57:22.224449+0000","last_active":"2024-06-29T15:57:33.290567+0000","last_peered":"2024-06-29T15:57:33.290567+0000","last_clean":"2024-06-29T15:57:33.290567+0000","last_became_active":"2024-06-29T15:57:22.224124+0000","last_became_peered":"2024-06-29T15:57:22.224124+0000","last_unstale":"2024-06-29T15:57:33.290567+0000","last_undegraded":"2024-06-29T15:57:33.290567+0000","last_fullsized":"2024-06-29T15:57:33.290567+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:05.649826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288470+0000","last_change":"2024-06-29T15:57:33.288470+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288470+0000","last_undegraded":"2024-06-29T15:57:33.288470+0000","last_fullsized":"2024-06-29T15:57:33.288470+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"221'4","reported_seq":20,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286813+0000","last_change":"2024-06-29T15:57:21.218674+0000","last_active":"2024-06-29T15:57:33.286813+0000","last_peered":"2024-06-29T15:57:33.286813+0000","last_clean":"2024-06-29T15:57:33.286813+0000","last_became_active":"2024-06-29T15:57:21.218397+0000","last_became_peered":"2024-06-29T15:57:21.218397+0000","last_unstale":"2024-06-29T15:57:33.286813+0000","last_undegraded":"2024-06-29T15:57:33.286813+0000","last_fullsized":"2024-06-29T15:57:33.286813+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.766844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289540+0000","last_change":"2024-06-29T15:57:22.228934+0000","last_active":"2024-06-29T15:57:33.289540+0000","last_peered":"2024-06-29T15:57:33.289540+0000","last_clean":"2024-06-29T15:57:33.289540+0000","last_became_active":"2024-06-29T15:57:22.228603+0000","last_became_peered":"2024-06-29T15:57:22.228603+0000","last_unstale":"2024-06-29T15:57:33.289540+0000","last_undegraded":"2024-06-29T15:57:33.289540+0000","last_fullsized":"2024-06-29T15:57:33.289540+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:28.391534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287273+0000","last_change":"2024-06-29T15:57:33.287273+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287273+0000","last_undegraded":"2024-06-29T15:57:33.287273+0000","last_fullsized":"2024-06-29T15:57:33.287273+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287779+0000","last_change":"2024-06-29T15:57:22.224277+0000","last_active":"2024-06-29T15:57:33.287779+0000","last_peered":"2024-06-29T15:57:33.287779+0000","last_clean":"2024-06-29T15:57:33.287779+0000","last_became_active":"2024-06-29T15:57:22.223975+0000","last_became_peered":"2024-06-29T15:57:22.223975+0000","last_unstale":"2024-06-29T15:57:33.287779+0000","last_undegraded":"2024-06-29T15:57:33.287779+0000","last_fullsized":"2024-06-29T15:57:33.287779+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:46.705935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288808+0000","last_change":"2024-06-29T15:57:22.239789+0000","last_active":"2024-06-29T15:57:33.288808+0000","last_peered":"2024-06-29T15:57:33.288808+0000","last_clean":"2024-06-29T15:57:33.288808+0000","last_became_active":"2024-06-29T15:57:22.239438+0000","last_became_peered":"2024-06-29T15:57:22.239438+0000","last_unstale":"2024-06-29T15:57:33.288808+0000","last_undegraded":"2024-06-29T15:57:33.288808+0000","last_fullsized":"2024-06-29T15:57:33.288808+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:08.408484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287651+0000","last_change":"2024-06-29T15:57:22.236151+0000","last_active":"2024-06-29T15:57:33.287651+0000","last_peered":"2024-06-29T15:57:33.287651+0000","last_clean":"2024-06-29T15:57:33.287651+0000","last_became_active":"2024-06-29T15:57:22.235409+0000","last_became_peered":"2024-06-29T15:57:22.235409+0000","last_unstale":"2024-06-29T15:57:33.287651+0000","last_undegraded":"2024-06-29T15:57:33.287651+0000","last_fullsized":"2024-06-29T15:57:33.287651+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:56.149438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283690+0000","last_change":"2024-06-29T15:57:22.227876+0000","last_active":"2024-06-29T15:57:33.283690+0000","last_peered":"2024-06-29T15:57:33.283690+0000","last_clean":"2024-06-29T15:57:33.283690+0000","last_became_active":"2024-06-29T15:57:22.227591+0000","last_became_peered":"2024-06-29T15:57:22.227591+0000","last_unstale":"2024-06-29T15:57:33.283690+0000","last_undegraded":"2024-06-29T15:57:33.283690+0000","last_fullsized":"2024-06-29T15:57:33.283690+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:45.600754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287210+0000","last_change":"2024-06-29T15:57:22.227845+0000","last_active":"2024-06-29T15:57:33.287210+0000","last_peered":"2024-06-29T15:57:33.287210+0000","last_clean":"2024-06-29T15:57:33.287210+0000","last_became_active":"2024-06-29T15:57:22.227549+0000","last_became_peered":"2024-06-29T15:57:22.227549+0000","last_unstale":"2024-06-29T15:57:33.287210+0000","last_undegraded":"2024-06-29T15:57:33.287210+0000","last_fullsized":"2024-06-29T15:57:33.287210+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:22.735783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283689+0000","last_change":"2024-06-29T15:57:22.236939+0000","last_active":"2024-06-29T15:57:33.283689+0000","last_peered":"2024-06-29T15:57:33.283689+0000","last_clean":"2024-06-29T15:57:33.283689+0000","last_became_active":"2024-06-29T15:57:22.236129+0000","last_became_peered":"2024-06-29T15:57:22.236129+0000","last_unstale":"2024-06-29T15:57:33.283689+0000","last_undegraded":"2024-06-29T15:57:33.283689+0000","last_fullsized":"2024-06-29T15:57:33.283689+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:13.274461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290581+0000","last_change":"2024-06-29T15:57:22.225280+0000","last_active":"2024-06-29T15:57:33.290581+0000","last_peered":"2024-06-29T15:57:33.290581+0000","last_clean":"2024-06-29T15:57:33.290581+0000","last_became_active":"2024-06-29T15:57:22.224986+0000","last_became_peered":"2024-06-29T15:57:22.224986+0000","last_unstale":"2024-06-29T15:57:33.290581+0000","last_undegraded":"2024-06-29T15:57:33.290581+0000","last_fullsized":"2024-06-29T15:57:33.290581+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:07.391769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283859+0000","last_change":"2024-06-29T15:57:22.234902+0000","last_active":"2024-06-29T15:57:33.283859+0000","last_peered":"2024-06-29T15:57:33.283859+0000","last_clean":"2024-06-29T15:57:33.283859+0000","last_became_active":"2024-06-29T15:57:22.233244+0000","last_became_peered":"2024-06-29T15:57:22.233244+0000","last_unstale":"2024-06-29T15:57:33.283859+0000","last_undegraded":"2024-06-29T15:57:33.283859+0000","last_fullsized":"2024-06-29T15:57:33.283859+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:58.786612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286548+0000","last_change":"2024-06-29T15:57:22.223814+0000","last_active":"2024-06-29T15:57:33.286548+0000","last_peered":"2024-06-29T15:57:33.286548+0000","last_clean":"2024-06-29T15:57:33.286548+0000","last_became_active":"2024-06-29T15:57:22.223464+0000","last_became_peered":"2024-06-29T15:57:22.223464+0000","last_unstale":"2024-06-29T15:57:33.286548+0000","last_undegraded":"2024-06-29T15:57:33.286548+0000","last_fullsized":"2024-06-29T15:57:33.286548+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:40.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289362+0000","last_change":"2024-06-29T15:57:22.232432+0000","last_active":"2024-06-29T15:57:33.289362+0000","last_peered":"2024-06-29T15:57:33.289362+0000","last_clean":"2024-06-29T15:57:33.289362+0000","last_became_active":"2024-06-29T15:57:22.232102+0000","last_became_peered":"2024-06-29T15:57:22.232102+0000","last_unstale":"2024-06-29T15:57:33.289362+0000","last_undegraded":"2024-06-29T15:57:33.289362+0000","last_fullsized":"2024-06-29T15:57:33.289362+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:41.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285155+0000","last_change":"2024-06-29T15:57:22.222075+0000","last_active":"2024-06-29T15:57:33.285155+0000","last_peered":"2024-06-29T15:57:33.285155+0000","last_clean":"2024-06-29T15:57:33.285155+0000","last_became_active":"2024-06-29T15:57:22.221767+0000","last_became_peered":"2024-06-29T15:57:22.221767+0000","last_unstale":"2024-06-29T15:57:33.285155+0000","last_undegraded":"2024-06-29T15:57:33.285155+0000","last_fullsized":"2024-06-29T15:57:33.285155+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:13.762887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285142+0000","last_change":"2024-06-29T15:57:22.224310+0000","last_active":"2024-06-29T15:57:33.285142+0000","last_peered":"2024-06-29T15:57:33.285142+0000","last_clean":"2024-06-29T15:57:33.285142+0000","last_became_active":"2024-06-29T15:57:22.223959+0000","last_became_peered":"2024-06-29T15:57:22.223959+0000","last_unstale":"2024-06-29T15:57:33.285142+0000","last_undegraded":"2024-06-29T15:57:33.285142+0000","last_fullsized":"2024-06-29T15:57:33.285142+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:13.811309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285573+0000","last_change":"2024-06-29T15:57:22.215484+0000","last_active":"2024-06-29T15:57:33.285573+0000","last_peered":"2024-06-29T15:57:33.285573+0000","last_clean":"2024-06-29T15:57:33.285573+0000","last_became_active":"2024-06-29T15:57:22.215179+0000","last_became_peered":"2024-06-29T15:57:22.215179+0000","last_unstale":"2024-06-29T15:57:33.285573+0000","last_undegraded":"2024-06-29T15:57:33.285573+0000","last_fullsized":"2024-06-29T15:57:33.285573+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:29.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286139+0000","last_change":"2024-06-29T15:57:22.237289+0000","last_active":"2024-06-29T15:57:33.286139+0000","last_peered":"2024-06-29T15:57:33.286139+0000","last_clean":"2024-06-29T15:57:33.286139+0000","last_became_active":"2024-06-29T15:57:22.236267+0000","last_became_peered":"2024-06-29T15:57:22.236267+0000","last_unstale":"2024-06-29T15:57:33.286139+0000","last_undegraded":"2024-06-29T15:57:33.286139+0000","last_fullsized":"2024-06-29T15:57:33.286139+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:53:11.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284251+0000","last_change":"2024-06-29T15:57:22.239205+0000","last_active":"2024-06-29T15:57:33.284251+0000","last_peered":"2024-06-29T15:57:33.284251+0000","last_clean":"2024-06-29T15:57:33.284251+0000","last_became_active":"2024-06-29T15:57:22.238830+0000","last_became_peered":"2024-06-29T15:57:22.238830+0000","last_unstale":"2024-06-29T15:57:33.284251+0000","last_undegraded":"2024-06-29T15:57:33.284251+0000","last_fullsized":"2024-06-29T15:57:33.284251+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.228440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284276+0000","last_change":"2024-06-29T15:57:22.235020+0000","last_active":"2024-06-29T15:57:33.284276+0000","last_peered":"2024-06-29T15:57:33.284276+0000","last_clean":"2024-06-29T15:57:33.284276+0000","last_became_active":"2024-06-29T15:57:22.234417+0000","last_became_peered":"2024-06-29T15:57:22.234417+0000","last_unstale":"2024-06-29T15:57:33.284276+0000","last_undegraded":"2024-06-29T15:57:33.284276+0000","last_fullsized":"2024-06-29T15:57:33.284276+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:06.264167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.282629+0000","last_change":"2024-06-29T15:57:22.238113+0000","last_active":"2024-06-29T15:57:33.282629+0000","last_peered":"2024-06-29T15:57:33.282629+0000","last_clean":"2024-06-29T15:57:33.282629+0000","last_became_active":"2024-06-29T15:57:22.237726+0000","last_became_peered":"2024-06-29T15:57:22.237726+0000","last_unstale":"2024-06-29T15:57:33.282629+0000","last_undegraded":"2024-06-29T15:57:33.282629+0000","last_fullsized":"2024-06-29T15:57:33.282629+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:08:48.416318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287733+0000","last_change":"2024-06-29T15:57:22.220754+0000","last_active":"2024-06-29T15:57:33.287733+0000","last_peered":"2024-06-29T15:57:33.287733+0000","last_clean":"2024-06-29T15:57:33.287733+0000","last_became_active":"2024-06-29T15:57:22.220438+0000","last_became_peered":"2024-06-29T15:57:22.220438+0000","last_unstale":"2024-06-29T15:57:33.287733+0000","last_undegraded":"2024-06-29T15:57:33.287733+0000","last_fullsized":"2024-06-29T15:57:33.287733+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:26.314927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284889+0000","last_change":"2024-06-29T15:57:22.225596+0000","last_active":"2024-06-29T15:57:33.284889+0000","last_peered":"2024-06-29T15:57:33.284889+0000","last_clean":"2024-06-29T15:57:33.284889+0000","last_became_active":"2024-06-29T15:57:22.225154+0000","last_became_peered":"2024-06-29T15:57:22.225154+0000","last_unstale":"2024-06-29T15:57:33.284889+0000","last_undegraded":"2024-06-29T15:57:33.284889+0000","last_fullsized":"2024-06-29T15:57:33.284889+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:01.272853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289427+0000","last_change":"2024-06-29T15:57:22.237433+0000","last_active":"2024-06-29T15:57:33.289427+0000","last_peered":"2024-06-29T15:57:33.289427+0000","last_clean":"2024-06-29T15:57:33.289427+0000","last_became_active":"2024-06-29T15:57:22.237070+0000","last_became_peered":"2024-06-29T15:57:22.237070+0000","last_unstale":"2024-06-29T15:57:33.289427+0000","last_undegraded":"2024-06-29T15:57:33.289427+0000","last_fullsized":"2024-06-29T15:57:33.289427+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:25.658901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287084+0000","last_change":"2024-06-29T15:57:22.231838+0000","last_active":"2024-06-29T15:57:33.287084+0000","last_peered":"2024-06-29T15:57:33.287084+0000","last_clean":"2024-06-29T15:57:33.287084+0000","last_became_active":"2024-06-29T15:57:22.231563+0000","last_became_peered":"2024-06-29T15:57:22.231563+0000","last_unstale":"2024-06-29T15:57:33.287084+0000","last_undegraded":"2024-06-29T15:57:33.287084+0000","last_fullsized":"2024-06-29T15:57:33.287084+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:23.251139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283290+0000","last_change":"2024-06-29T15:57:22.237225+0000","last_active":"2024-06-29T15:57:33.283290+0000","last_peered":"2024-06-29T15:57:33.283290+0000","last_clean":"2024-06-29T15:57:33.283290+0000","last_became_active":"2024-06-29T15:57:22.236621+0000","last_became_peered":"2024-06-29T15:57:22.236621+0000","last_unstale":"2024-06-29T15:57:33.283290+0000","last_undegraded":"2024-06-29T15:57:33.283290+0000","last_fullsized":"2024-06-29T15:57:33.283290+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:26.582705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285984+0000","last_change":"2024-06-29T15:57:22.237316+0000","last_active":"2024-06-29T15:57:33.285984+0000","last_peered":"2024-06-29T15:57:33.285984+0000","last_clean":"2024-06-29T15:57:33.285984+0000","last_became_active":"2024-06-29T15:57:22.236756+0000","last_became_peered":"2024-06-29T15:57:22.236756+0000","last_unstale":"2024-06-29T15:57:33.285984+0000","last_undegraded":"2024-06-29T15:57:33.285984+0000","last_fullsized":"2024-06-29T15:57:33.285984+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.141583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283723+0000","last_change":"2024-06-29T15:57:22.235210+0000","last_active":"2024-06-29T15:57:33.283723+0000","last_peered":"2024-06-29T15:57:33.283723+0000","last_clean":"2024-06-29T15:57:33.283723+0000","last_became_active":"2024-06-29T15:57:22.234573+0000","last_became_peered":"2024-06-29T15:57:22.234573+0000","last_unstale":"2024-06-29T15:57:33.283723+0000","last_undegraded":"2024-06-29T15:57:33.283723+0000","last_fullsized":"2024-06-29T15:57:33.283723+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:48.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285269+0000","last_change":"2024-06-29T15:57:22.233275+0000","last_active":"2024-06-29T15:57:33.285269+0000","last_peered":"2024-06-29T15:57:33.285269+0000","last_clean":"2024-06-29T15:57:33.285269+0000","last_became_active":"2024-06-29T15:57:22.230801+0000","last_became_peered":"2024-06-29T15:57:22.230801+0000","last_unstale":"2024-06-29T15:57:33.285269+0000","last_undegraded":"2024-06-29T15:57:33.285269+0000","last_fullsized":"2024-06-29T15:57:33.285269+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:24.134069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286822+0000","last_change":"2024-06-29T15:57:22.223324+0000","last_active":"2024-06-29T15:57:33.286822+0000","last_peered":"2024-06-29T15:57:33.286822+0000","last_clean":"2024-06-29T15:57:33.286822+0000","last_became_active":"2024-06-29T15:57:22.222958+0000","last_became_peered":"2024-06-29T15:57:22.222958+0000","last_unstale":"2024-06-29T15:57:33.286822+0000","last_undegraded":"2024-06-29T15:57:33.286822+0000","last_fullsized":"2024-06-29T15:57:33.286822+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:21.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288844+0000","last_change":"2024-06-29T15:57:22.226496+0000","last_active":"2024-06-29T15:57:33.288844+0000","last_peered":"2024-06-29T15:57:33.288844+0000","last_clean":"2024-06-29T15:57:33.288844+0000","last_became_active":"2024-06-29T15:57:22.226191+0000","last_became_peered":"2024-06-29T15:57:22.226191+0000","last_unstale":"2024-06-29T15:57:33.288844+0000","last_undegraded":"2024-06-29T15:57:33.288844+0000","last_fullsized":"2024-06-29T15:57:33.288844+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:32.644449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287426+0000","last_change":"2024-06-29T15:57:22.218369+0000","last_active":"2024-06-29T15:57:33.287426+0000","last_peered":"2024-06-29T15:57:33.287426+0000","last_clean":"2024-06-29T15:57:33.287426+0000","last_became_active":"2024-06-29T15:57:22.217584+0000","last_became_peered":"2024-06-29T15:57:22.217584+0000","last_unstale":"2024-06-29T15:57:33.287426+0000","last_undegraded":"2024-06-29T15:57:33.287426+0000","last_fullsized":"2024-06-29T15:57:33.287426+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:59.760761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291286+0000","last_change":"2024-06-29T15:57:22.236022+0000","last_active":"2024-06-29T15:57:33.291286+0000","last_peered":"2024-06-29T15:57:33.291286+0000","last_clean":"2024-06-29T15:57:33.291286+0000","last_became_active":"2024-06-29T15:57:22.235712+0000","last_became_peered":"2024-06-29T15:57:22.235712+0000","last_unstale":"2024-06-29T15:57:33.291286+0000","last_undegraded":"2024-06-29T15:57:33.291286+0000","last_fullsized":"2024-06-29T15:57:33.291286+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:57.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283276+0000","last_change":"2024-06-29T15:57:22.231073+0000","last_active":"2024-06-29T15:57:33.283276+0000","last_peered":"2024-06-29T15:57:33.283276+0000","last_clean":"2024-06-29T15:57:33.283276+0000","last_became_active":"2024-06-29T15:57:22.230749+0000","last_became_peered":"2024-06-29T15:57:22.230749+0000","last_unstale":"2024-06-29T15:57:33.283276+0000","last_undegraded":"2024-06-29T15:57:33.283276+0000","last_fullsized":"2024-06-29T15:57:33.283276+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:23.168668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289947+0000","last_change":"2024-06-29T15:57:22.231733+0000","last_active":"2024-06-29T15:57:33.289947+0000","last_peered":"2024-06-29T15:57:33.289947+0000","last_clean":"2024-06-29T15:57:33.289947+0000","last_became_active":"2024-06-29T15:57:22.231387+0000","last_became_peered":"2024-06-29T15:57:22.231387+0000","last_unstale":"2024-06-29T15:57:33.289947+0000","last_undegraded":"2024-06-29T15:57:33.289947+0000","last_fullsized":"2024-06-29T15:57:33.289947+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:41.892974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291260+0000","last_change":"2024-06-29T15:57:22.225592+0000","last_active":"2024-06-29T15:57:33.291260+0000","last_peered":"2024-06-29T15:57:33.291260+0000","last_clean":"2024-06-29T15:57:33.291260+0000","last_became_active":"2024-06-29T15:57:22.225309+0000","last_became_peered":"2024-06-29T15:57:22.225309+0000","last_unstale":"2024-06-29T15:57:33.291260+0000","last_undegraded":"2024-06-29T15:57:33.291260+0000","last_fullsized":"2024-06-29T15:57:33.291260+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:07.857349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543289,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52552,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319288,"statfs":{"total":96636764160,"available":96582950912,"internally_reserved":0,"allocated":3403776,"data_stored":2030256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543290,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42244,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329596,"statfs":{"total":96636764160,"available":96593506304,"internally_reserved":0,"allocated":2809856,"data_stored":1440400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543289,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47804,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324036,"statfs":{"total":96636764160,"available":96587812864,"internally_reserved":0,"allocated":2801664,"data_stored":1437019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543288,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51948,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319892,"statfs":{"total":96636764160,"available":96583569408,"internally_reserved":0,"allocated":2785280,"data_stored":1430422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510586,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47808,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324032,"statfs":{"total":96636764160,"available":96587808768,"internally_reserved":0,"allocated":2805760,"data_stored":1437089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47800,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324040,"statfs":{"total":96636764160,"available":96587816960,"internally_reserved":0,"allocated":2797568,"data_stored":1437053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510585,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47792,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324048,"statfs":{"total":96636764160,"available":96587825152,"internally_reserved":0,"allocated":2781184,"data_stored":1429840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510587,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42812,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329028,"statfs":{"total":96636764160,"available":96592924672,"internally_reserved":0,"allocated":3391488,"data_stored":2027421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:39.407 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.555+0000 7f6c6a681640 1 -- 172.21.15.59:0/780267041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c6415efe0 msgr2=0x7f6c6415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.555+0000 7f6c6a681640 1 --2- 172.21.15.59:0/780267041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c6415efe0 0x7f6c6415f9b0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f6c640d15d0 tx=0x7f6c5c02aac0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.555+0000 7f6c6a681640 1 -- 172.21.15.59:0/780267041 shutdown_connections 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.555+0000 7f6c6a681640 1 --2- 172.21.15.59:0/780267041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c6415efe0 0x7f6c6415f9b0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.555+0000 7f6c6a681640 1 --2- 172.21.15.59:0/780267041 >> v2:172.21.15.153:3300/0 conn(0x7f6c640d23c0 0x7f6c640b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.555+0000 7f6c6a681640 1 -- 172.21.15.59:0/780267041 >> 172.21.15.59:0/780267041 conn(0x7f6c640d3ee0 msgr2=0x7f6c640d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.556+0000 7f6c6a681640 1 -- 172.21.15.59:0/780267041 shutdown_connections 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.556+0000 7f6c6a681640 1 -- 172.21.15.59:0/780267041 wait complete. 2024-06-29T15:57:39.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.557+0000 7f6c6a681640 1 Processor -- start 2024-06-29T15:57:39.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.558+0000 7f6c6a681640 1 -- start start 2024-06-29T15:57:39.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c6a681640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c640b3ed0 0x7f6c6415c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c6a681640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c640d23c0 0x7f6c6415b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c6a681640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c6415efe0 0x7f6c641e9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c6a681640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6c64160280 con 0x7f6c640b3ed0 2024-06-29T15:57:39.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c6a681640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c64160100 con 0x7f6c6415efe0 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c6a681640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6c64160400 con 0x7f6c640d23c0 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c63fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c640b3ed0 0x7f6c6415c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c63fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c640b3ed0 0x7f6c6415c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44756/0 (socket says 172.21.15.59:44756) 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c63fff640 1 -- 172.21.15.59:0/3010255741 learned_addr learned my addr 172.21.15.59:0/3010255741 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c63fff640 1 -- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c640d23c0 msgr2=0x7f6c6415b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c637fe640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c640d23c0 0x7f6c6415b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c68c1e640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c6415efe0 0x7f6c641e9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c63fff640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c640d23c0 0x7f6c6415b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c63fff640 1 -- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c6415efe0 msgr2=0x7f6c641e9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c63fff640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c6415efe0 0x7f6c641e9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.559+0000 7f6c63fff640 1 -- 172.21.15.59:0/3010255741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6415ce00 con 0x7f6c640b3ed0 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.560+0000 7f6c63fff640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c640b3ed0 0x7f6c6415c8c0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400e740 tx=0x7f6c5400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.560+0000 7f6c60ff9640 1 -- 172.21.15.59:0/3010255741 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5400ccc0 con 0x7f6c640b3ed0 2024-06-29T15:57:39.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.560+0000 7f6c60ff9640 1 -- 172.21.15.59:0/3010255741 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c5400ee50 con 0x7f6c640b3ed0 2024-06-29T15:57:39.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.560+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c641eb760 con 0x7f6c640b3ed0 2024-06-29T15:57:39.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.560+0000 7f6c60ff9640 1 -- 172.21.15.59:0/3010255741 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54004290 con 0x7f6c640b3ed0 2024-06-29T15:57:39.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.560+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c6415a250 con 0x7f6c640b3ed0 2024-06-29T15:57:39.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.561+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c640b3990 con 0x7f6c640b3ed0 2024-06-29T15:57:39.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.561+0000 7f6c60ff9640 1 -- 172.21.15.59:0/3010255741 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f6c540035f0 con 0x7f6c640b3ed0 2024-06-29T15:57:39.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.562+0000 7f6c60ff9640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6c500763f0 0x7f6c50078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.562+0000 7f6c60ff9640 1 -- 172.21.15.59:0/3010255741 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f6c54098810 con 0x7f6c640b3ed0 2024-06-29T15:57:39.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.562+0000 7f6c637fe640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6c500763f0 0x7f6c50078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.563+0000 7f6c637fe640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6c500763f0 0x7f6c50078900 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c6415e730 tx=0x7f6c58009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.566+0000 7f6c60ff9640 1 -- 172.21.15.59:0/3010255741 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6c54061a30 con 0x7f6c640b3ed0 2024-06-29T15:57:39.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.755+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6c6406cda0 con 0x7f6c500763f0 2024-06-29T15:57:39.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.763+0000 7f6c60ff9640 1 -- 172.21.15.59:0/3010255741 <== mgr.9206 v2:172.21.15.153:6832/2780049427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+479293 (secure 0 0 0) 0x7f6c6406cda0 con 0x7f6c500763f0 2024-06-29T15:57:39.768 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:57:39.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.766+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6c500763f0 msgr2=0x7f6c50078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.766+0000 7f6c6a681640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6c500763f0 0x7f6c50078900 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c6415e730 tx=0x7f6c58009040 comp rx=0 tx=0).stop 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.766+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c640b3ed0 msgr2=0x7f6c6415c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.766+0000 7f6c6a681640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c640b3ed0 0x7f6c6415c8c0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400e740 tx=0x7f6c5400ec00 comp rx=0 tx=0).stop 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.766+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 shutdown_connections 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.767+0000 7f6c6a681640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f6c500763f0 0x7f6c50078900 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.767+0000 7f6c6a681640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c6415efe0 0x7f6c641e9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.767+0000 7f6c6a681640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c640d23c0 0x7f6c6415b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.767+0000 7f6c6a681640 1 --2- 172.21.15.59:0/3010255741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c640b3ed0 0x7f6c6415c8c0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.767+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 >> 172.21.15.59:0/3010255741 conn(0x7f6c640d3ee0 msgr2=0x7f6c6406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.767+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 shutdown_connections 2024-06-29T15:57:39.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.767+0000 7f6c6a681640 1 -- 172.21.15.59:0/3010255741 wait complete. 2024-06-29T15:57:39.779 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-06-29T15:57:37.769244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594021,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":424,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380760,"kb_used_data":23024,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593960,"statfs":{"total":773094113280,"available":772704215040,"internally_reserved":0,"allocated":23576576,"data_stored":12669500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":366202418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1243,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.488644"},"pg_stats":[{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"210'233","reported_seq":1103,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288044+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:57:33.288044+0000","last_peered":"2024-06-29T15:57:33.288044+0000","last_clean":"2024-06-29T15:57:33.288044+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:57:33.288044+0000","last_undegraded":"2024-06-29T15:57:33.288044+0000","last_fullsized":"2024-06-29T15:57:33.288044+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283944+0000","last_change":"2024-06-29T15:57:21.209163+0000","last_active":"2024-06-29T15:57:33.283944+0000","last_peered":"2024-06-29T15:57:33.283944+0000","last_clean":"2024-06-29T15:57:33.283944+0000","last_became_active":"2024-06-29T15:57:21.208825+0000","last_became_peered":"2024-06-29T15:57:21.208825+0000","last_unstale":"2024-06-29T15:57:33.283944+0000","last_undegraded":"2024-06-29T15:57:33.283944+0000","last_fullsized":"2024-06-29T15:57:33.283944+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:13.641268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290795+0000","last_change":"2024-06-29T15:57:22.230794+0000","last_active":"2024-06-29T15:57:33.290795+0000","last_peered":"2024-06-29T15:57:33.290795+0000","last_clean":"2024-06-29T15:57:33.290795+0000","last_became_active":"2024-06-29T15:57:22.230474+0000","last_became_peered":"2024-06-29T15:57:22.230474+0000","last_unstale":"2024-06-29T15:57:33.290795+0000","last_undegraded":"2024-06-29T15:57:33.290795+0000","last_fullsized":"2024-06-29T15:57:33.290795+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:43.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287008+0000","last_change":"2024-06-29T15:57:33.287008+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287008+0000","last_undegraded":"2024-06-29T15:57:33.287008+0000","last_fullsized":"2024-06-29T15:57:33.287008+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286721+0000","last_change":"2024-06-29T15:57:21.215589+0000","last_active":"2024-06-29T15:57:33.286721+0000","last_peered":"2024-06-29T15:57:33.286721+0000","last_clean":"2024-06-29T15:57:33.286721+0000","last_became_active":"2024-06-29T15:57:21.213719+0000","last_became_peered":"2024-06-29T15:57:21.213719+0000","last_unstale":"2024-06-29T15:57:33.286721+0000","last_undegraded":"2024-06-29T15:57:33.286721+0000","last_fullsized":"2024-06-29T15:57:33.286721+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:32.413502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283532+0000","last_change":"2024-06-29T15:57:22.223950+0000","last_active":"2024-06-29T15:57:33.283532+0000","last_peered":"2024-06-29T15:57:33.283532+0000","last_clean":"2024-06-29T15:57:33.283532+0000","last_became_active":"2024-06-29T15:57:22.223622+0000","last_became_peered":"2024-06-29T15:57:22.223622+0000","last_unstale":"2024-06-29T15:57:33.283532+0000","last_undegraded":"2024-06-29T15:57:33.283532+0000","last_fullsized":"2024-06-29T15:57:33.283532+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.556171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286422+0000","last_change":"2024-06-29T15:57:33.286422+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286422+0000","last_undegraded":"2024-06-29T15:57:33.286422+0000","last_fullsized":"2024-06-29T15:57:33.286422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290938+0000","last_change":"2024-06-29T15:57:21.212837+0000","last_active":"2024-06-29T15:57:33.290938+0000","last_peered":"2024-06-29T15:57:33.290938+0000","last_clean":"2024-06-29T15:57:33.290938+0000","last_became_active":"2024-06-29T15:57:21.212491+0000","last_became_peered":"2024-06-29T15:57:21.212491+0000","last_unstale":"2024-06-29T15:57:33.290938+0000","last_undegraded":"2024-06-29T15:57:33.290938+0000","last_fullsized":"2024-06-29T15:57:33.290938+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:35:53.641775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.292001+0000","last_change":"2024-06-29T15:57:22.237722+0000","last_active":"2024-06-29T15:57:33.292001+0000","last_peered":"2024-06-29T15:57:33.292001+0000","last_clean":"2024-06-29T15:57:33.292001+0000","last_became_active":"2024-06-29T15:57:22.237372+0000","last_became_peered":"2024-06-29T15:57:22.237372+0000","last_unstale":"2024-06-29T15:57:33.292001+0000","last_undegraded":"2024-06-29T15:57:33.292001+0000","last_fullsized":"2024-06-29T15:57:33.292001+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:24.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283133+0000","last_change":"2024-06-29T15:57:33.283133+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283133+0000","last_undegraded":"2024-06-29T15:57:33.283133+0000","last_fullsized":"2024-06-29T15:57:33.283133+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284094+0000","last_change":"2024-06-29T15:57:21.210563+0000","last_active":"2024-06-29T15:57:33.284094+0000","last_peered":"2024-06-29T15:57:33.284094+0000","last_clean":"2024-06-29T15:57:33.284094+0000","last_became_active":"2024-06-29T15:57:21.210243+0000","last_became_peered":"2024-06-29T15:57:21.210243+0000","last_unstale":"2024-06-29T15:57:33.284094+0000","last_undegraded":"2024-06-29T15:57:33.284094+0000","last_fullsized":"2024-06-29T15:57:33.284094+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.900240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291290+0000","last_change":"2024-06-29T15:57:22.244396+0000","last_active":"2024-06-29T15:57:33.291290+0000","last_peered":"2024-06-29T15:57:33.291290+0000","last_clean":"2024-06-29T15:57:33.291290+0000","last_became_active":"2024-06-29T15:57:22.244044+0000","last_became_peered":"2024-06-29T15:57:22.244044+0000","last_unstale":"2024-06-29T15:57:33.291290+0000","last_undegraded":"2024-06-29T15:57:33.291290+0000","last_fullsized":"2024-06-29T15:57:33.291290+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.981276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283374+0000","last_change":"2024-06-29T15:57:33.283374+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283374+0000","last_undegraded":"2024-06-29T15:57:33.283374+0000","last_fullsized":"2024-06-29T15:57:33.283374+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285468+0000","last_change":"2024-06-29T15:57:21.213113+0000","last_active":"2024-06-29T15:57:33.285468+0000","last_peered":"2024-06-29T15:57:33.285468+0000","last_clean":"2024-06-29T15:57:33.285468+0000","last_became_active":"2024-06-29T15:57:21.212819+0000","last_became_peered":"2024-06-29T15:57:21.212819+0000","last_unstale":"2024-06-29T15:57:33.285468+0000","last_undegraded":"2024-06-29T15:57:33.285468+0000","last_fullsized":"2024-06-29T15:57:33.285468+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:59.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285691+0000","last_change":"2024-06-29T15:57:22.229230+0000","last_active":"2024-06-29T15:57:33.285691+0000","last_peered":"2024-06-29T15:57:33.285691+0000","last_clean":"2024-06-29T15:57:33.285691+0000","last_became_active":"2024-06-29T15:57:22.228914+0000","last_became_peered":"2024-06-29T15:57:22.228914+0000","last_unstale":"2024-06-29T15:57:33.285691+0000","last_undegraded":"2024-06-29T15:57:33.285691+0000","last_fullsized":"2024-06-29T15:57:33.285691+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:09.918042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.285594+0000","last_change":"2024-06-29T15:57:33.285594+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.285594+0000","last_undegraded":"2024-06-29T15:57:33.285594+0000","last_fullsized":"2024-06-29T15:57:33.285594+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290684+0000","last_change":"2024-06-29T15:57:21.216399+0000","last_active":"2024-06-29T15:57:33.290684+0000","last_peered":"2024-06-29T15:57:33.290684+0000","last_clean":"2024-06-29T15:57:33.290684+0000","last_became_active":"2024-06-29T15:57:21.216104+0000","last_became_peered":"2024-06-29T15:57:21.216104+0000","last_unstale":"2024-06-29T15:57:33.290684+0000","last_undegraded":"2024-06-29T15:57:33.290684+0000","last_fullsized":"2024-06-29T15:57:33.290684+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290475+0000","last_change":"2024-06-29T15:57:22.227969+0000","last_active":"2024-06-29T15:57:33.290475+0000","last_peered":"2024-06-29T15:57:33.290475+0000","last_clean":"2024-06-29T15:57:33.290475+0000","last_became_active":"2024-06-29T15:57:22.227579+0000","last_became_peered":"2024-06-29T15:57:22.227579+0000","last_unstale":"2024-06-29T15:57:33.290475+0000","last_undegraded":"2024-06-29T15:57:33.290475+0000","last_fullsized":"2024-06-29T15:57:33.290475+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287295+0000","last_change":"2024-06-29T15:57:33.287295+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287295+0000","last_undegraded":"2024-06-29T15:57:33.287295+0000","last_fullsized":"2024-06-29T15:57:33.287295+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286605+0000","last_change":"2024-06-29T15:57:21.217039+0000","last_active":"2024-06-29T15:57:33.286605+0000","last_peered":"2024-06-29T15:57:33.286605+0000","last_clean":"2024-06-29T15:57:33.286605+0000","last_became_active":"2024-06-29T15:57:21.216542+0000","last_became_peered":"2024-06-29T15:57:21.216542+0000","last_unstale":"2024-06-29T15:57:33.286605+0000","last_undegraded":"2024-06-29T15:57:33.286605+0000","last_fullsized":"2024-06-29T15:57:33.286605+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:21.729257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283680+0000","last_change":"2024-06-29T15:57:22.233919+0000","last_active":"2024-06-29T15:57:33.283680+0000","last_peered":"2024-06-29T15:57:33.283680+0000","last_clean":"2024-06-29T15:57:33.283680+0000","last_became_active":"2024-06-29T15:57:22.231028+0000","last_became_peered":"2024-06-29T15:57:22.231028+0000","last_unstale":"2024-06-29T15:57:33.283680+0000","last_undegraded":"2024-06-29T15:57:33.283680+0000","last_fullsized":"2024-06-29T15:57:33.283680+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:34:24.910914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283097+0000","last_change":"2024-06-29T15:57:33.283097+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283097+0000","last_undegraded":"2024-06-29T15:57:33.283097+0000","last_fullsized":"2024-06-29T15:57:33.283097+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"222'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286084+0000","last_change":"2024-06-29T15:57:21.203822+0000","last_active":"2024-06-29T15:57:33.286084+0000","last_peered":"2024-06-29T15:57:33.286084+0000","last_clean":"2024-06-29T15:57:33.286084+0000","last_became_active":"2024-06-29T15:57:21.203517+0000","last_became_peered":"2024-06-29T15:57:21.203517+0000","last_unstale":"2024-06-29T15:57:33.286084+0000","last_undegraded":"2024-06-29T15:57:33.286084+0000","last_fullsized":"2024-06-29T15:57:33.286084+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:05.810997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.282967+0000","last_change":"2024-06-29T15:57:22.223140+0000","last_active":"2024-06-29T15:57:33.282967+0000","last_peered":"2024-06-29T15:57:33.282967+0000","last_clean":"2024-06-29T15:57:33.282967+0000","last_became_active":"2024-06-29T15:57:22.222839+0000","last_became_peered":"2024-06-29T15:57:22.222839+0000","last_unstale":"2024-06-29T15:57:33.282967+0000","last_undegraded":"2024-06-29T15:57:33.282967+0000","last_fullsized":"2024-06-29T15:57:33.282967+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:17.520475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283134+0000","last_change":"2024-06-29T15:57:33.283134+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283134+0000","last_undegraded":"2024-06-29T15:57:33.283134+0000","last_fullsized":"2024-06-29T15:57:33.283134+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284126+0000","last_change":"2024-06-29T15:57:21.212745+0000","last_active":"2024-06-29T15:57:33.284126+0000","last_peered":"2024-06-29T15:57:33.284126+0000","last_clean":"2024-06-29T15:57:33.284126+0000","last_became_active":"2024-06-29T15:57:21.212335+0000","last_became_peered":"2024-06-29T15:57:21.212335+0000","last_unstale":"2024-06-29T15:57:33.284126+0000","last_undegraded":"2024-06-29T15:57:33.284126+0000","last_fullsized":"2024-06-29T15:57:33.284126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:07.049637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285779+0000","last_change":"2024-06-29T15:57:22.224294+0000","last_active":"2024-06-29T15:57:33.285779+0000","last_peered":"2024-06-29T15:57:33.285779+0000","last_clean":"2024-06-29T15:57:33.285779+0000","last_became_active":"2024-06-29T15:57:22.223922+0000","last_became_peered":"2024-06-29T15:57:22.223922+0000","last_unstale":"2024-06-29T15:57:33.285779+0000","last_undegraded":"2024-06-29T15:57:33.285779+0000","last_fullsized":"2024-06-29T15:57:33.285779+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:46.725456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286716+0000","last_change":"2024-06-29T15:57:33.286716+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286716+0000","last_undegraded":"2024-06-29T15:57:33.286716+0000","last_fullsized":"2024-06-29T15:57:33.286716+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290020+0000","last_change":"2024-06-29T15:57:21.211203+0000","last_active":"2024-06-29T15:57:33.290020+0000","last_peered":"2024-06-29T15:57:33.290020+0000","last_clean":"2024-06-29T15:57:33.290020+0000","last_became_active":"2024-06-29T15:57:21.210898+0000","last_became_peered":"2024-06-29T15:57:21.210898+0000","last_unstale":"2024-06-29T15:57:33.290020+0000","last_undegraded":"2024-06-29T15:57:33.290020+0000","last_fullsized":"2024-06-29T15:57:33.290020+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:17.270875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286704+0000","last_change":"2024-06-29T15:57:22.213920+0000","last_active":"2024-06-29T15:57:33.286704+0000","last_peered":"2024-06-29T15:57:33.286704+0000","last_clean":"2024-06-29T15:57:33.286704+0000","last_became_active":"2024-06-29T15:57:22.213583+0000","last_became_peered":"2024-06-29T15:57:22.213583+0000","last_unstale":"2024-06-29T15:57:33.286704+0000","last_undegraded":"2024-06-29T15:57:33.286704+0000","last_fullsized":"2024-06-29T15:57:33.286704+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:31.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.285625+0000","last_change":"2024-06-29T15:57:33.285625+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.285625+0000","last_undegraded":"2024-06-29T15:57:33.285625+0000","last_fullsized":"2024-06-29T15:57:33.285625+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287942+0000","last_change":"2024-06-29T15:57:21.212085+0000","last_active":"2024-06-29T15:57:33.287942+0000","last_peered":"2024-06-29T15:57:33.287942+0000","last_clean":"2024-06-29T15:57:33.287942+0000","last_became_active":"2024-06-29T15:57:21.211731+0000","last_became_peered":"2024-06-29T15:57:21.211731+0000","last_unstale":"2024-06-29T15:57:33.287942+0000","last_undegraded":"2024-06-29T15:57:33.287942+0000","last_fullsized":"2024-06-29T15:57:33.287942+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:36.061054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284429+0000","last_change":"2024-06-29T15:57:22.228211+0000","last_active":"2024-06-29T15:57:33.284429+0000","last_peered":"2024-06-29T15:57:33.284429+0000","last_clean":"2024-06-29T15:57:33.284429+0000","last_became_active":"2024-06-29T15:57:22.227519+0000","last_became_peered":"2024-06-29T15:57:22.227519+0000","last_unstale":"2024-06-29T15:57:33.284429+0000","last_undegraded":"2024-06-29T15:57:33.284429+0000","last_fullsized":"2024-06-29T15:57:33.284429+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:16:35.999038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288476+0000","last_change":"2024-06-29T15:57:33.288476+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288476+0000","last_undegraded":"2024-06-29T15:57:33.288476+0000","last_fullsized":"2024-06-29T15:57:33.288476+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289681+0000","last_change":"2024-06-29T15:57:21.208592+0000","last_active":"2024-06-29T15:57:33.289681+0000","last_peered":"2024-06-29T15:57:33.289681+0000","last_clean":"2024-06-29T15:57:33.289681+0000","last_became_active":"2024-06-29T15:57:21.208156+0000","last_became_peered":"2024-06-29T15:57:21.208156+0000","last_unstale":"2024-06-29T15:57:33.289681+0000","last_undegraded":"2024-06-29T15:57:33.289681+0000","last_fullsized":"2024-06-29T15:57:33.289681+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:20.325402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291188+0000","last_change":"2024-06-29T15:57:22.238829+0000","last_active":"2024-06-29T15:57:33.291188+0000","last_peered":"2024-06-29T15:57:33.291188+0000","last_clean":"2024-06-29T15:57:33.291188+0000","last_became_active":"2024-06-29T15:57:22.238520+0000","last_became_peered":"2024-06-29T15:57:22.238520+0000","last_unstale":"2024-06-29T15:57:33.291188+0000","last_undegraded":"2024-06-29T15:57:33.291188+0000","last_fullsized":"2024-06-29T15:57:33.291188+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:22.127384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.284230+0000","last_change":"2024-06-29T15:57:33.284230+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.284230+0000","last_undegraded":"2024-06-29T15:57:33.284230+0000","last_fullsized":"2024-06-29T15:57:33.284230+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287695+0000","last_change":"2024-06-29T15:57:21.215358+0000","last_active":"2024-06-29T15:57:33.287695+0000","last_peered":"2024-06-29T15:57:33.287695+0000","last_clean":"2024-06-29T15:57:33.287695+0000","last_became_active":"2024-06-29T15:57:21.213979+0000","last_became_peered":"2024-06-29T15:57:21.213979+0000","last_unstale":"2024-06-29T15:57:33.287695+0000","last_undegraded":"2024-06-29T15:57:33.287695+0000","last_fullsized":"2024-06-29T15:57:33.287695+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:42.446135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289780+0000","last_change":"2024-06-29T15:57:22.231883+0000","last_active":"2024-06-29T15:57:33.289780+0000","last_peered":"2024-06-29T15:57:33.289780+0000","last_clean":"2024-06-29T15:57:33.289780+0000","last_became_active":"2024-06-29T15:57:22.231588+0000","last_became_peered":"2024-06-29T15:57:22.231588+0000","last_unstale":"2024-06-29T15:57:33.289780+0000","last_undegraded":"2024-06-29T15:57:33.289780+0000","last_fullsized":"2024-06-29T15:57:33.289780+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:17.705760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283133+0000","last_change":"2024-06-29T15:57:33.283133+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283133+0000","last_undegraded":"2024-06-29T15:57:33.283133+0000","last_fullsized":"2024-06-29T15:57:33.283133+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287693+0000","last_change":"2024-06-29T15:57:21.213193+0000","last_active":"2024-06-29T15:57:33.287693+0000","last_peered":"2024-06-29T15:57:33.287693+0000","last_clean":"2024-06-29T15:57:33.287693+0000","last_became_active":"2024-06-29T15:57:21.212838+0000","last_became_peered":"2024-06-29T15:57:21.212838+0000","last_unstale":"2024-06-29T15:57:33.287693+0000","last_undegraded":"2024-06-29T15:57:33.287693+0000","last_fullsized":"2024-06-29T15:57:33.287693+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:54.574391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285342+0000","last_change":"2024-06-29T15:57:22.217630+0000","last_active":"2024-06-29T15:57:33.285342+0000","last_peered":"2024-06-29T15:57:33.285342+0000","last_clean":"2024-06-29T15:57:33.285342+0000","last_became_active":"2024-06-29T15:57:22.217303+0000","last_became_peered":"2024-06-29T15:57:22.217303+0000","last_unstale":"2024-06-29T15:57:33.285342+0000","last_undegraded":"2024-06-29T15:57:33.285342+0000","last_fullsized":"2024-06-29T15:57:33.285342+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:05.554814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286544+0000","last_change":"2024-06-29T15:57:33.286544+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286544+0000","last_undegraded":"2024-06-29T15:57:33.286544+0000","last_fullsized":"2024-06-29T15:57:33.286544+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290097+0000","last_change":"2024-06-29T15:57:21.210528+0000","last_active":"2024-06-29T15:57:33.290097+0000","last_peered":"2024-06-29T15:57:33.290097+0000","last_clean":"2024-06-29T15:57:33.290097+0000","last_became_active":"2024-06-29T15:57:21.208691+0000","last_became_peered":"2024-06-29T15:57:21.208691+0000","last_unstale":"2024-06-29T15:57:33.290097+0000","last_undegraded":"2024-06-29T15:57:33.290097+0000","last_fullsized":"2024-06-29T15:57:33.290097+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.762590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290667+0000","last_change":"2024-06-29T15:57:22.232727+0000","last_active":"2024-06-29T15:57:33.290667+0000","last_peered":"2024-06-29T15:57:33.290667+0000","last_clean":"2024-06-29T15:57:33.290667+0000","last_became_active":"2024-06-29T15:57:22.232392+0000","last_became_peered":"2024-06-29T15:57:22.232392+0000","last_unstale":"2024-06-29T15:57:33.290667+0000","last_undegraded":"2024-06-29T15:57:33.290667+0000","last_fullsized":"2024-06-29T15:57:33.290667+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:54.805984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283760+0000","last_change":"2024-06-29T15:57:33.283760+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283760+0000","last_undegraded":"2024-06-29T15:57:33.283760+0000","last_fullsized":"2024-06-29T15:57:33.283760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291169+0000","last_change":"2024-06-29T15:57:21.216179+0000","last_active":"2024-06-29T15:57:33.291169+0000","last_peered":"2024-06-29T15:57:33.291169+0000","last_clean":"2024-06-29T15:57:33.291169+0000","last_became_active":"2024-06-29T15:57:21.215876+0000","last_became_peered":"2024-06-29T15:57:21.215876+0000","last_unstale":"2024-06-29T15:57:33.291169+0000","last_undegraded":"2024-06-29T15:57:33.291169+0000","last_fullsized":"2024-06-29T15:57:33.291169+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:37.942186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290682+0000","last_change":"2024-06-29T15:57:22.225793+0000","last_active":"2024-06-29T15:57:33.290682+0000","last_peered":"2024-06-29T15:57:33.290682+0000","last_clean":"2024-06-29T15:57:33.290682+0000","last_became_active":"2024-06-29T15:57:22.224806+0000","last_became_peered":"2024-06-29T15:57:22.224806+0000","last_unstale":"2024-06-29T15:57:33.290682+0000","last_undegraded":"2024-06-29T15:57:33.290682+0000","last_fullsized":"2024-06-29T15:57:33.290682+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:03.667477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286140+0000","last_change":"2024-06-29T15:57:33.286140+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286140+0000","last_undegraded":"2024-06-29T15:57:33.286140+0000","last_fullsized":"2024-06-29T15:57:33.286140+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291389+0000","last_change":"2024-06-29T15:57:21.212718+0000","last_active":"2024-06-29T15:57:33.291389+0000","last_peered":"2024-06-29T15:57:33.291389+0000","last_clean":"2024-06-29T15:57:33.291389+0000","last_became_active":"2024-06-29T15:57:21.212406+0000","last_became_peered":"2024-06-29T15:57:21.212406+0000","last_unstale":"2024-06-29T15:57:33.291389+0000","last_undegraded":"2024-06-29T15:57:33.291389+0000","last_fullsized":"2024-06-29T15:57:33.291389+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:00.180622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285664+0000","last_change":"2024-06-29T15:57:22.238347+0000","last_active":"2024-06-29T15:57:33.285664+0000","last_peered":"2024-06-29T15:57:33.285664+0000","last_clean":"2024-06-29T15:57:33.285664+0000","last_became_active":"2024-06-29T15:57:22.236896+0000","last_became_peered":"2024-06-29T15:57:22.236896+0000","last_unstale":"2024-06-29T15:57:33.285664+0000","last_undegraded":"2024-06-29T15:57:33.285664+0000","last_fullsized":"2024-06-29T15:57:33.285664+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:39.739194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.285610+0000","last_change":"2024-06-29T15:57:33.285610+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.285610+0000","last_undegraded":"2024-06-29T15:57:33.285610+0000","last_fullsized":"2024-06-29T15:57:33.285610+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"221'8","reported_seq":24,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286557+0000","last_change":"2024-06-29T15:57:21.216156+0000","last_active":"2024-06-29T15:57:33.286557+0000","last_peered":"2024-06-29T15:57:33.286557+0000","last_clean":"2024-06-29T15:57:33.286557+0000","last_became_active":"2024-06-29T15:57:21.214088+0000","last_became_peered":"2024-06-29T15:57:21.214088+0000","last_unstale":"2024-06-29T15:57:33.286557+0000","last_undegraded":"2024-06-29T15:57:33.286557+0000","last_fullsized":"2024-06-29T15:57:33.286557+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:30.314243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291583+0000","last_change":"2024-06-29T15:57:22.223112+0000","last_active":"2024-06-29T15:57:33.291583+0000","last_peered":"2024-06-29T15:57:33.291583+0000","last_clean":"2024-06-29T15:57:33.291583+0000","last_became_active":"2024-06-29T15:57:22.222632+0000","last_became_peered":"2024-06-29T15:57:22.222632+0000","last_unstale":"2024-06-29T15:57:33.291583+0000","last_undegraded":"2024-06-29T15:57:33.291583+0000","last_fullsized":"2024-06-29T15:57:33.291583+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:05.408776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287648+0000","last_change":"2024-06-29T15:57:33.287648+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287648+0000","last_undegraded":"2024-06-29T15:57:33.287648+0000","last_fullsized":"2024-06-29T15:57:33.287648+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283547+0000","last_change":"2024-06-29T15:57:21.215032+0000","last_active":"2024-06-29T15:57:33.283547+0000","last_peered":"2024-06-29T15:57:33.283547+0000","last_clean":"2024-06-29T15:57:33.283547+0000","last_became_active":"2024-06-29T15:57:21.214557+0000","last_became_peered":"2024-06-29T15:57:21.214557+0000","last_unstale":"2024-06-29T15:57:33.283547+0000","last_undegraded":"2024-06-29T15:57:33.283547+0000","last_fullsized":"2024-06-29T15:57:33.283547+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284364+0000","last_change":"2024-06-29T15:57:22.228484+0000","last_active":"2024-06-29T15:57:33.284364+0000","last_peered":"2024-06-29T15:57:33.284364+0000","last_clean":"2024-06-29T15:57:33.284364+0000","last_became_active":"2024-06-29T15:57:22.227880+0000","last_became_peered":"2024-06-29T15:57:22.227880+0000","last_unstale":"2024-06-29T15:57:33.284364+0000","last_undegraded":"2024-06-29T15:57:33.284364+0000","last_fullsized":"2024-06-29T15:57:33.284364+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:21.749071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287403+0000","last_change":"2024-06-29T15:57:33.287403+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287403+0000","last_undegraded":"2024-06-29T15:57:33.287403+0000","last_fullsized":"2024-06-29T15:57:33.287403+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283118+0000","last_change":"2024-06-29T15:57:21.203410+0000","last_active":"2024-06-29T15:57:33.283118+0000","last_peered":"2024-06-29T15:57:33.283118+0000","last_clean":"2024-06-29T15:57:33.283118+0000","last_became_active":"2024-06-29T15:57:21.203034+0000","last_became_peered":"2024-06-29T15:57:21.203034+0000","last_unstale":"2024-06-29T15:57:33.283118+0000","last_undegraded":"2024-06-29T15:57:33.283118+0000","last_fullsized":"2024-06-29T15:57:33.283118+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:52.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283462+0000","last_change":"2024-06-29T15:57:22.241881+0000","last_active":"2024-06-29T15:57:33.283462+0000","last_peered":"2024-06-29T15:57:33.283462+0000","last_clean":"2024-06-29T15:57:33.283462+0000","last_became_active":"2024-06-29T15:57:22.241549+0000","last_became_peered":"2024-06-29T15:57:22.241549+0000","last_unstale":"2024-06-29T15:57:33.283462+0000","last_undegraded":"2024-06-29T15:57:33.283462+0000","last_fullsized":"2024-06-29T15:57:33.283462+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:25.352328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288277+0000","last_change":"2024-06-29T15:57:33.288277+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288277+0000","last_undegraded":"2024-06-29T15:57:33.288277+0000","last_fullsized":"2024-06-29T15:57:33.288277+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289594+0000","last_change":"2024-06-29T15:57:21.215892+0000","last_active":"2024-06-29T15:57:33.289594+0000","last_peered":"2024-06-29T15:57:33.289594+0000","last_clean":"2024-06-29T15:57:33.289594+0000","last_became_active":"2024-06-29T15:57:21.215250+0000","last_became_peered":"2024-06-29T15:57:21.215250+0000","last_unstale":"2024-06-29T15:57:33.289594+0000","last_undegraded":"2024-06-29T15:57:33.289594+0000","last_fullsized":"2024-06-29T15:57:33.289594+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:23.234622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289678+0000","last_change":"2024-06-29T15:57:22.234802+0000","last_active":"2024-06-29T15:57:33.289678+0000","last_peered":"2024-06-29T15:57:33.289678+0000","last_clean":"2024-06-29T15:57:33.289678+0000","last_became_active":"2024-06-29T15:57:22.234492+0000","last_became_peered":"2024-06-29T15:57:22.234492+0000","last_unstale":"2024-06-29T15:57:33.289678+0000","last_undegraded":"2024-06-29T15:57:33.289678+0000","last_fullsized":"2024-06-29T15:57:33.289678+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:37.928014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286760+0000","last_change":"2024-06-29T15:57:33.286760+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286760+0000","last_undegraded":"2024-06-29T15:57:33.286760+0000","last_fullsized":"2024-06-29T15:57:33.286760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"222'8","reported_seq":24,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291177+0000","last_change":"2024-06-29T15:57:21.217755+0000","last_active":"2024-06-29T15:57:33.291177+0000","last_peered":"2024-06-29T15:57:33.291177+0000","last_clean":"2024-06-29T15:57:33.291177+0000","last_became_active":"2024-06-29T15:57:21.216983+0000","last_became_peered":"2024-06-29T15:57:21.216983+0000","last_unstale":"2024-06-29T15:57:33.291177+0000","last_undegraded":"2024-06-29T15:57:33.291177+0000","last_fullsized":"2024-06-29T15:57:33.291177+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:31.889343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287603+0000","last_change":"2024-06-29T15:57:22.224600+0000","last_active":"2024-06-29T15:57:33.287603+0000","last_peered":"2024-06-29T15:57:33.287603+0000","last_clean":"2024-06-29T15:57:33.287603+0000","last_became_active":"2024-06-29T15:57:22.224260+0000","last_became_peered":"2024-06-29T15:57:22.224260+0000","last_unstale":"2024-06-29T15:57:33.287603+0000","last_undegraded":"2024-06-29T15:57:33.287603+0000","last_fullsized":"2024-06-29T15:57:33.287603+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:21.773370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.282975+0000","last_change":"2024-06-29T15:57:33.282975+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.282975+0000","last_undegraded":"2024-06-29T15:57:33.282975+0000","last_fullsized":"2024-06-29T15:57:33.282975+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"221'3","reported_seq":19,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284539+0000","last_change":"2024-06-29T15:57:21.210270+0000","last_active":"2024-06-29T15:57:33.284539+0000","last_peered":"2024-06-29T15:57:33.284539+0000","last_clean":"2024-06-29T15:57:33.284539+0000","last_became_active":"2024-06-29T15:57:21.209731+0000","last_became_peered":"2024-06-29T15:57:21.209731+0000","last_unstale":"2024-06-29T15:57:33.284539+0000","last_undegraded":"2024-06-29T15:57:33.284539+0000","last_fullsized":"2024-06-29T15:57:33.284539+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:18.182233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283812+0000","last_change":"2024-06-29T15:57:22.231011+0000","last_active":"2024-06-29T15:57:33.283812+0000","last_peered":"2024-06-29T15:57:33.283812+0000","last_clean":"2024-06-29T15:57:33.283812+0000","last_became_active":"2024-06-29T15:57:22.230675+0000","last_became_peered":"2024-06-29T15:57:22.230675+0000","last_unstale":"2024-06-29T15:57:33.283812+0000","last_undegraded":"2024-06-29T15:57:33.283812+0000","last_fullsized":"2024-06-29T15:57:33.283812+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:09.478769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.284342+0000","last_change":"2024-06-29T15:57:33.284342+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.284342+0000","last_undegraded":"2024-06-29T15:57:33.284342+0000","last_fullsized":"2024-06-29T15:57:33.284342+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.292126+0000","last_change":"2024-06-29T15:57:21.214536+0000","last_active":"2024-06-29T15:57:33.292126+0000","last_peered":"2024-06-29T15:57:33.292126+0000","last_clean":"2024-06-29T15:57:33.292126+0000","last_became_active":"2024-06-29T15:57:21.214227+0000","last_became_peered":"2024-06-29T15:57:21.214227+0000","last_unstale":"2024-06-29T15:57:33.292126+0000","last_undegraded":"2024-06-29T15:57:33.292126+0000","last_fullsized":"2024-06-29T15:57:33.292126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:25.839522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.282963+0000","last_change":"2024-06-29T15:57:22.219192+0000","last_active":"2024-06-29T15:57:33.282963+0000","last_peered":"2024-06-29T15:57:33.282963+0000","last_clean":"2024-06-29T15:57:33.282963+0000","last_became_active":"2024-06-29T15:57:22.218844+0000","last_became_peered":"2024-06-29T15:57:22.218844+0000","last_unstale":"2024-06-29T15:57:33.282963+0000","last_undegraded":"2024-06-29T15:57:33.282963+0000","last_fullsized":"2024-06-29T15:57:33.282963+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:50.717283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283773+0000","last_change":"2024-06-29T15:57:33.283773+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283773+0000","last_undegraded":"2024-06-29T15:57:33.283773+0000","last_fullsized":"2024-06-29T15:57:33.283773+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288108+0000","last_change":"2024-06-29T15:57:21.218814+0000","last_active":"2024-06-29T15:57:33.288108+0000","last_peered":"2024-06-29T15:57:33.288108+0000","last_clean":"2024-06-29T15:57:33.288108+0000","last_became_active":"2024-06-29T15:57:21.218495+0000","last_became_peered":"2024-06-29T15:57:21.218495+0000","last_unstale":"2024-06-29T15:57:33.288108+0000","last_undegraded":"2024-06-29T15:57:33.288108+0000","last_fullsized":"2024-06-29T15:57:33.288108+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:04.680014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286900+0000","last_change":"2024-06-29T15:57:22.235312+0000","last_active":"2024-06-29T15:57:33.286900+0000","last_peered":"2024-06-29T15:57:33.286900+0000","last_clean":"2024-06-29T15:57:33.286900+0000","last_became_active":"2024-06-29T15:57:22.234828+0000","last_became_peered":"2024-06-29T15:57:22.234828+0000","last_unstale":"2024-06-29T15:57:33.286900+0000","last_undegraded":"2024-06-29T15:57:33.286900+0000","last_fullsized":"2024-06-29T15:57:33.286900+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:39.661714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288150+0000","last_change":"2024-06-29T15:57:33.288150+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288150+0000","last_undegraded":"2024-06-29T15:57:33.288150+0000","last_fullsized":"2024-06-29T15:57:33.288150+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.283149+0000","last_change":"2024-06-29T15:57:33.283149+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.283149+0000","last_undegraded":"2024-06-29T15:57:33.283149+0000","last_fullsized":"2024-06-29T15:57:33.283149+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.282789+0000","last_change":"2024-06-29T15:57:21.208896+0000","last_active":"2024-06-29T15:57:33.282789+0000","last_peered":"2024-06-29T15:57:33.282789+0000","last_clean":"2024-06-29T15:57:33.282789+0000","last_became_active":"2024-06-29T15:57:21.208555+0000","last_became_peered":"2024-06-29T15:57:21.208555+0000","last_unstale":"2024-06-29T15:57:33.282789+0000","last_undegraded":"2024-06-29T15:57:33.282789+0000","last_fullsized":"2024-06-29T15:57:33.282789+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283991+0000","last_change":"2024-06-29T15:57:22.234743+0000","last_active":"2024-06-29T15:57:33.283991+0000","last_peered":"2024-06-29T15:57:33.283991+0000","last_clean":"2024-06-29T15:57:33.283991+0000","last_became_active":"2024-06-29T15:57:22.233091+0000","last_became_peered":"2024-06-29T15:57:22.233091+0000","last_unstale":"2024-06-29T15:57:33.283991+0000","last_undegraded":"2024-06-29T15:57:33.283991+0000","last_fullsized":"2024-06-29T15:57:33.283991+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:46.131681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.285686+0000","last_change":"2024-06-29T15:57:33.285686+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.285686+0000","last_undegraded":"2024-06-29T15:57:33.285686+0000","last_fullsized":"2024-06-29T15:57:33.285686+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288695+0000","last_change":"2024-06-29T15:57:21.212034+0000","last_active":"2024-06-29T15:57:33.288695+0000","last_peered":"2024-06-29T15:57:33.288695+0000","last_clean":"2024-06-29T15:57:33.288695+0000","last_became_active":"2024-06-29T15:57:21.211762+0000","last_became_peered":"2024-06-29T15:57:21.211762+0000","last_unstale":"2024-06-29T15:57:33.288695+0000","last_undegraded":"2024-06-29T15:57:33.288695+0000","last_fullsized":"2024-06-29T15:57:33.288695+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:36.775853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287907+0000","last_change":"2024-06-29T15:57:22.214168+0000","last_active":"2024-06-29T15:57:33.287907+0000","last_peered":"2024-06-29T15:57:33.287907+0000","last_clean":"2024-06-29T15:57:33.287907+0000","last_became_active":"2024-06-29T15:57:22.213780+0000","last_became_peered":"2024-06-29T15:57:22.213780+0000","last_unstale":"2024-06-29T15:57:33.287907+0000","last_undegraded":"2024-06-29T15:57:33.287907+0000","last_fullsized":"2024-06-29T15:57:33.287907+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:01.904725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289552+0000","last_change":"2024-06-29T15:57:21.211113+0000","last_active":"2024-06-29T15:57:33.289552+0000","last_peered":"2024-06-29T15:57:33.289552+0000","last_clean":"2024-06-29T15:57:33.289552+0000","last_became_active":"2024-06-29T15:57:21.210817+0000","last_became_peered":"2024-06-29T15:57:21.210817+0000","last_unstale":"2024-06-29T15:57:33.289552+0000","last_undegraded":"2024-06-29T15:57:33.289552+0000","last_fullsized":"2024-06-29T15:57:33.289552+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:26.176884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287567+0000","last_change":"2024-06-29T15:57:22.235721+0000","last_active":"2024-06-29T15:57:33.287567+0000","last_peered":"2024-06-29T15:57:33.287567+0000","last_clean":"2024-06-29T15:57:33.287567+0000","last_became_active":"2024-06-29T15:57:22.235380+0000","last_became_peered":"2024-06-29T15:57:22.235380+0000","last_unstale":"2024-06-29T15:57:33.287567+0000","last_undegraded":"2024-06-29T15:57:33.287567+0000","last_fullsized":"2024-06-29T15:57:33.287567+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:38.144376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288700+0000","last_change":"2024-06-29T15:57:33.288700+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288700+0000","last_undegraded":"2024-06-29T15:57:33.288700+0000","last_fullsized":"2024-06-29T15:57:33.288700+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284087+0000","last_change":"2024-06-29T15:57:21.210643+0000","last_active":"2024-06-29T15:57:33.284087+0000","last_peered":"2024-06-29T15:57:33.284087+0000","last_clean":"2024-06-29T15:57:33.284087+0000","last_became_active":"2024-06-29T15:57:21.209805+0000","last_became_peered":"2024-06-29T15:57:21.209805+0000","last_unstale":"2024-06-29T15:57:33.284087+0000","last_undegraded":"2024-06-29T15:57:33.284087+0000","last_fullsized":"2024-06-29T15:57:33.284087+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:27.618049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287540+0000","last_change":"2024-06-29T15:57:22.226181+0000","last_active":"2024-06-29T15:57:33.287540+0000","last_peered":"2024-06-29T15:57:33.287540+0000","last_clean":"2024-06-29T15:57:33.287540+0000","last_became_active":"2024-06-29T15:57:22.225855+0000","last_became_peered":"2024-06-29T15:57:22.225855+0000","last_unstale":"2024-06-29T15:57:33.287540+0000","last_undegraded":"2024-06-29T15:57:33.287540+0000","last_fullsized":"2024-06-29T15:57:33.287540+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:37.067580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287076+0000","last_change":"2024-06-29T15:57:33.287076+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287076+0000","last_undegraded":"2024-06-29T15:57:33.287076+0000","last_fullsized":"2024-06-29T15:57:33.287076+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"221'2","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290044+0000","last_change":"2024-06-29T15:57:21.210602+0000","last_active":"2024-06-29T15:57:33.290044+0000","last_peered":"2024-06-29T15:57:33.290044+0000","last_clean":"2024-06-29T15:57:33.290044+0000","last_became_active":"2024-06-29T15:57:21.210296+0000","last_became_peered":"2024-06-29T15:57:21.210296+0000","last_unstale":"2024-06-29T15:57:33.290044+0000","last_undegraded":"2024-06-29T15:57:33.290044+0000","last_fullsized":"2024-06-29T15:57:33.290044+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:07.128352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285263+0000","last_change":"2024-06-29T15:57:22.213237+0000","last_active":"2024-06-29T15:57:33.285263+0000","last_peered":"2024-06-29T15:57:33.285263+0000","last_clean":"2024-06-29T15:57:33.285263+0000","last_became_active":"2024-06-29T15:57:22.212908+0000","last_became_peered":"2024-06-29T15:57:22.212908+0000","last_unstale":"2024-06-29T15:57:33.285263+0000","last_undegraded":"2024-06-29T15:57:33.285263+0000","last_fullsized":"2024-06-29T15:57:33.285263+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:31.343924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.286084+0000","last_change":"2024-06-29T15:57:33.286084+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.286084+0000","last_undegraded":"2024-06-29T15:57:33.286084+0000","last_fullsized":"2024-06-29T15:57:33.286084+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"221'1","reported_seq":17,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291011+0000","last_change":"2024-06-29T15:57:21.213288+0000","last_active":"2024-06-29T15:57:33.291011+0000","last_peered":"2024-06-29T15:57:33.291011+0000","last_clean":"2024-06-29T15:57:33.291011+0000","last_became_active":"2024-06-29T15:57:21.212954+0000","last_became_peered":"2024-06-29T15:57:21.212954+0000","last_unstale":"2024-06-29T15:57:33.291011+0000","last_undegraded":"2024-06-29T15:57:33.291011+0000","last_fullsized":"2024-06-29T15:57:33.291011+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:24.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290567+0000","last_change":"2024-06-29T15:57:22.224449+0000","last_active":"2024-06-29T15:57:33.290567+0000","last_peered":"2024-06-29T15:57:33.290567+0000","last_clean":"2024-06-29T15:57:33.290567+0000","last_became_active":"2024-06-29T15:57:22.224124+0000","last_became_peered":"2024-06-29T15:57:22.224124+0000","last_unstale":"2024-06-29T15:57:33.290567+0000","last_undegraded":"2024-06-29T15:57:33.290567+0000","last_fullsized":"2024-06-29T15:57:33.290567+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:05.649826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.288470+0000","last_change":"2024-06-29T15:57:33.288470+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.288470+0000","last_undegraded":"2024-06-29T15:57:33.288470+0000","last_fullsized":"2024-06-29T15:57:33.288470+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"221'4","reported_seq":20,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286813+0000","last_change":"2024-06-29T15:57:21.218674+0000","last_active":"2024-06-29T15:57:33.286813+0000","last_peered":"2024-06-29T15:57:33.286813+0000","last_clean":"2024-06-29T15:57:33.286813+0000","last_became_active":"2024-06-29T15:57:21.218397+0000","last_became_peered":"2024-06-29T15:57:21.218397+0000","last_unstale":"2024-06-29T15:57:33.286813+0000","last_undegraded":"2024-06-29T15:57:33.286813+0000","last_fullsized":"2024-06-29T15:57:33.286813+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.766844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289540+0000","last_change":"2024-06-29T15:57:22.228934+0000","last_active":"2024-06-29T15:57:33.289540+0000","last_peered":"2024-06-29T15:57:33.289540+0000","last_clean":"2024-06-29T15:57:33.289540+0000","last_became_active":"2024-06-29T15:57:22.228603+0000","last_became_peered":"2024-06-29T15:57:22.228603+0000","last_unstale":"2024-06-29T15:57:33.289540+0000","last_undegraded":"2024-06-29T15:57:33.289540+0000","last_fullsized":"2024-06-29T15:57:33.289540+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:28.391534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":223,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.287273+0000","last_change":"2024-06-29T15:57:33.287273+0000","last_active":"2024-06-29T15:57:33.271564+0000","last_peered":"2024-06-29T15:57:33.271564+0000","last_clean":"2024-06-29T15:57:33.271564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.287273+0000","last_undegraded":"2024-06-29T15:57:33.287273+0000","last_fullsized":"2024-06-29T15:57:33.287273+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287779+0000","last_change":"2024-06-29T15:57:22.224277+0000","last_active":"2024-06-29T15:57:33.287779+0000","last_peered":"2024-06-29T15:57:33.287779+0000","last_clean":"2024-06-29T15:57:33.287779+0000","last_became_active":"2024-06-29T15:57:22.223975+0000","last_became_peered":"2024-06-29T15:57:22.223975+0000","last_unstale":"2024-06-29T15:57:33.287779+0000","last_undegraded":"2024-06-29T15:57:33.287779+0000","last_fullsized":"2024-06-29T15:57:33.287779+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:46.705935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288808+0000","last_change":"2024-06-29T15:57:22.239789+0000","last_active":"2024-06-29T15:57:33.288808+0000","last_peered":"2024-06-29T15:57:33.288808+0000","last_clean":"2024-06-29T15:57:33.288808+0000","last_became_active":"2024-06-29T15:57:22.239438+0000","last_became_peered":"2024-06-29T15:57:22.239438+0000","last_unstale":"2024-06-29T15:57:33.288808+0000","last_undegraded":"2024-06-29T15:57:33.288808+0000","last_fullsized":"2024-06-29T15:57:33.288808+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:08.408484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287651+0000","last_change":"2024-06-29T15:57:22.236151+0000","last_active":"2024-06-29T15:57:33.287651+0000","last_peered":"2024-06-29T15:57:33.287651+0000","last_clean":"2024-06-29T15:57:33.287651+0000","last_became_active":"2024-06-29T15:57:22.235409+0000","last_became_peered":"2024-06-29T15:57:22.235409+0000","last_unstale":"2024-06-29T15:57:33.287651+0000","last_undegraded":"2024-06-29T15:57:33.287651+0000","last_fullsized":"2024-06-29T15:57:33.287651+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:56.149438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283690+0000","last_change":"2024-06-29T15:57:22.227876+0000","last_active":"2024-06-29T15:57:33.283690+0000","last_peered":"2024-06-29T15:57:33.283690+0000","last_clean":"2024-06-29T15:57:33.283690+0000","last_became_active":"2024-06-29T15:57:22.227591+0000","last_became_peered":"2024-06-29T15:57:22.227591+0000","last_unstale":"2024-06-29T15:57:33.283690+0000","last_undegraded":"2024-06-29T15:57:33.283690+0000","last_fullsized":"2024-06-29T15:57:33.283690+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:45.600754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287210+0000","last_change":"2024-06-29T15:57:22.227845+0000","last_active":"2024-06-29T15:57:33.287210+0000","last_peered":"2024-06-29T15:57:33.287210+0000","last_clean":"2024-06-29T15:57:33.287210+0000","last_became_active":"2024-06-29T15:57:22.227549+0000","last_became_peered":"2024-06-29T15:57:22.227549+0000","last_unstale":"2024-06-29T15:57:33.287210+0000","last_undegraded":"2024-06-29T15:57:33.287210+0000","last_fullsized":"2024-06-29T15:57:33.287210+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:22.735783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283689+0000","last_change":"2024-06-29T15:57:22.236939+0000","last_active":"2024-06-29T15:57:33.283689+0000","last_peered":"2024-06-29T15:57:33.283689+0000","last_clean":"2024-06-29T15:57:33.283689+0000","last_became_active":"2024-06-29T15:57:22.236129+0000","last_became_peered":"2024-06-29T15:57:22.236129+0000","last_unstale":"2024-06-29T15:57:33.283689+0000","last_undegraded":"2024-06-29T15:57:33.283689+0000","last_fullsized":"2024-06-29T15:57:33.283689+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:13.274461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.290581+0000","last_change":"2024-06-29T15:57:22.225280+0000","last_active":"2024-06-29T15:57:33.290581+0000","last_peered":"2024-06-29T15:57:33.290581+0000","last_clean":"2024-06-29T15:57:33.290581+0000","last_became_active":"2024-06-29T15:57:22.224986+0000","last_became_peered":"2024-06-29T15:57:22.224986+0000","last_unstale":"2024-06-29T15:57:33.290581+0000","last_undegraded":"2024-06-29T15:57:33.290581+0000","last_fullsized":"2024-06-29T15:57:33.290581+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:07.391769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283859+0000","last_change":"2024-06-29T15:57:22.234902+0000","last_active":"2024-06-29T15:57:33.283859+0000","last_peered":"2024-06-29T15:57:33.283859+0000","last_clean":"2024-06-29T15:57:33.283859+0000","last_became_active":"2024-06-29T15:57:22.233244+0000","last_became_peered":"2024-06-29T15:57:22.233244+0000","last_unstale":"2024-06-29T15:57:33.283859+0000","last_undegraded":"2024-06-29T15:57:33.283859+0000","last_fullsized":"2024-06-29T15:57:33.283859+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:58.786612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286548+0000","last_change":"2024-06-29T15:57:22.223814+0000","last_active":"2024-06-29T15:57:33.286548+0000","last_peered":"2024-06-29T15:57:33.286548+0000","last_clean":"2024-06-29T15:57:33.286548+0000","last_became_active":"2024-06-29T15:57:22.223464+0000","last_became_peered":"2024-06-29T15:57:22.223464+0000","last_unstale":"2024-06-29T15:57:33.286548+0000","last_undegraded":"2024-06-29T15:57:33.286548+0000","last_fullsized":"2024-06-29T15:57:33.286548+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:40.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289362+0000","last_change":"2024-06-29T15:57:22.232432+0000","last_active":"2024-06-29T15:57:33.289362+0000","last_peered":"2024-06-29T15:57:33.289362+0000","last_clean":"2024-06-29T15:57:33.289362+0000","last_became_active":"2024-06-29T15:57:22.232102+0000","last_became_peered":"2024-06-29T15:57:22.232102+0000","last_unstale":"2024-06-29T15:57:33.289362+0000","last_undegraded":"2024-06-29T15:57:33.289362+0000","last_fullsized":"2024-06-29T15:57:33.289362+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:41.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285155+0000","last_change":"2024-06-29T15:57:22.222075+0000","last_active":"2024-06-29T15:57:33.285155+0000","last_peered":"2024-06-29T15:57:33.285155+0000","last_clean":"2024-06-29T15:57:33.285155+0000","last_became_active":"2024-06-29T15:57:22.221767+0000","last_became_peered":"2024-06-29T15:57:22.221767+0000","last_unstale":"2024-06-29T15:57:33.285155+0000","last_undegraded":"2024-06-29T15:57:33.285155+0000","last_fullsized":"2024-06-29T15:57:33.285155+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:13.762887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285142+0000","last_change":"2024-06-29T15:57:22.224310+0000","last_active":"2024-06-29T15:57:33.285142+0000","last_peered":"2024-06-29T15:57:33.285142+0000","last_clean":"2024-06-29T15:57:33.285142+0000","last_became_active":"2024-06-29T15:57:22.223959+0000","last_became_peered":"2024-06-29T15:57:22.223959+0000","last_unstale":"2024-06-29T15:57:33.285142+0000","last_undegraded":"2024-06-29T15:57:33.285142+0000","last_fullsized":"2024-06-29T15:57:33.285142+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:13.811309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285573+0000","last_change":"2024-06-29T15:57:22.215484+0000","last_active":"2024-06-29T15:57:33.285573+0000","last_peered":"2024-06-29T15:57:33.285573+0000","last_clean":"2024-06-29T15:57:33.285573+0000","last_became_active":"2024-06-29T15:57:22.215179+0000","last_became_peered":"2024-06-29T15:57:22.215179+0000","last_unstale":"2024-06-29T15:57:33.285573+0000","last_undegraded":"2024-06-29T15:57:33.285573+0000","last_fullsized":"2024-06-29T15:57:33.285573+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:29.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286139+0000","last_change":"2024-06-29T15:57:22.237289+0000","last_active":"2024-06-29T15:57:33.286139+0000","last_peered":"2024-06-29T15:57:33.286139+0000","last_clean":"2024-06-29T15:57:33.286139+0000","last_became_active":"2024-06-29T15:57:22.236267+0000","last_became_peered":"2024-06-29T15:57:22.236267+0000","last_unstale":"2024-06-29T15:57:33.286139+0000","last_undegraded":"2024-06-29T15:57:33.286139+0000","last_fullsized":"2024-06-29T15:57:33.286139+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:53:11.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284251+0000","last_change":"2024-06-29T15:57:22.239205+0000","last_active":"2024-06-29T15:57:33.284251+0000","last_peered":"2024-06-29T15:57:33.284251+0000","last_clean":"2024-06-29T15:57:33.284251+0000","last_became_active":"2024-06-29T15:57:22.238830+0000","last_became_peered":"2024-06-29T15:57:22.238830+0000","last_unstale":"2024-06-29T15:57:33.284251+0000","last_undegraded":"2024-06-29T15:57:33.284251+0000","last_fullsized":"2024-06-29T15:57:33.284251+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.228440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284276+0000","last_change":"2024-06-29T15:57:22.235020+0000","last_active":"2024-06-29T15:57:33.284276+0000","last_peered":"2024-06-29T15:57:33.284276+0000","last_clean":"2024-06-29T15:57:33.284276+0000","last_became_active":"2024-06-29T15:57:22.234417+0000","last_became_peered":"2024-06-29T15:57:22.234417+0000","last_unstale":"2024-06-29T15:57:33.284276+0000","last_undegraded":"2024-06-29T15:57:33.284276+0000","last_fullsized":"2024-06-29T15:57:33.284276+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:06.264167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.282629+0000","last_change":"2024-06-29T15:57:22.238113+0000","last_active":"2024-06-29T15:57:33.282629+0000","last_peered":"2024-06-29T15:57:33.282629+0000","last_clean":"2024-06-29T15:57:33.282629+0000","last_became_active":"2024-06-29T15:57:22.237726+0000","last_became_peered":"2024-06-29T15:57:22.237726+0000","last_unstale":"2024-06-29T15:57:33.282629+0000","last_undegraded":"2024-06-29T15:57:33.282629+0000","last_fullsized":"2024-06-29T15:57:33.282629+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:08:48.416318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287733+0000","last_change":"2024-06-29T15:57:22.220754+0000","last_active":"2024-06-29T15:57:33.287733+0000","last_peered":"2024-06-29T15:57:33.287733+0000","last_clean":"2024-06-29T15:57:33.287733+0000","last_became_active":"2024-06-29T15:57:22.220438+0000","last_became_peered":"2024-06-29T15:57:22.220438+0000","last_unstale":"2024-06-29T15:57:33.287733+0000","last_undegraded":"2024-06-29T15:57:33.287733+0000","last_fullsized":"2024-06-29T15:57:33.287733+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:26.314927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.284889+0000","last_change":"2024-06-29T15:57:22.225596+0000","last_active":"2024-06-29T15:57:33.284889+0000","last_peered":"2024-06-29T15:57:33.284889+0000","last_clean":"2024-06-29T15:57:33.284889+0000","last_became_active":"2024-06-29T15:57:22.225154+0000","last_became_peered":"2024-06-29T15:57:22.225154+0000","last_unstale":"2024-06-29T15:57:33.284889+0000","last_undegraded":"2024-06-29T15:57:33.284889+0000","last_fullsized":"2024-06-29T15:57:33.284889+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:01.272853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289427+0000","last_change":"2024-06-29T15:57:22.237433+0000","last_active":"2024-06-29T15:57:33.289427+0000","last_peered":"2024-06-29T15:57:33.289427+0000","last_clean":"2024-06-29T15:57:33.289427+0000","last_became_active":"2024-06-29T15:57:22.237070+0000","last_became_peered":"2024-06-29T15:57:22.237070+0000","last_unstale":"2024-06-29T15:57:33.289427+0000","last_undegraded":"2024-06-29T15:57:33.289427+0000","last_fullsized":"2024-06-29T15:57:33.289427+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:25.658901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287084+0000","last_change":"2024-06-29T15:57:22.231838+0000","last_active":"2024-06-29T15:57:33.287084+0000","last_peered":"2024-06-29T15:57:33.287084+0000","last_clean":"2024-06-29T15:57:33.287084+0000","last_became_active":"2024-06-29T15:57:22.231563+0000","last_became_peered":"2024-06-29T15:57:22.231563+0000","last_unstale":"2024-06-29T15:57:33.287084+0000","last_undegraded":"2024-06-29T15:57:33.287084+0000","last_fullsized":"2024-06-29T15:57:33.287084+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:23.251139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283290+0000","last_change":"2024-06-29T15:57:22.237225+0000","last_active":"2024-06-29T15:57:33.283290+0000","last_peered":"2024-06-29T15:57:33.283290+0000","last_clean":"2024-06-29T15:57:33.283290+0000","last_became_active":"2024-06-29T15:57:22.236621+0000","last_became_peered":"2024-06-29T15:57:22.236621+0000","last_unstale":"2024-06-29T15:57:33.283290+0000","last_undegraded":"2024-06-29T15:57:33.283290+0000","last_fullsized":"2024-06-29T15:57:33.283290+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:26.582705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285984+0000","last_change":"2024-06-29T15:57:22.237316+0000","last_active":"2024-06-29T15:57:33.285984+0000","last_peered":"2024-06-29T15:57:33.285984+0000","last_clean":"2024-06-29T15:57:33.285984+0000","last_became_active":"2024-06-29T15:57:22.236756+0000","last_became_peered":"2024-06-29T15:57:22.236756+0000","last_unstale":"2024-06-29T15:57:33.285984+0000","last_undegraded":"2024-06-29T15:57:33.285984+0000","last_fullsized":"2024-06-29T15:57:33.285984+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.141583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283723+0000","last_change":"2024-06-29T15:57:22.235210+0000","last_active":"2024-06-29T15:57:33.283723+0000","last_peered":"2024-06-29T15:57:33.283723+0000","last_clean":"2024-06-29T15:57:33.283723+0000","last_became_active":"2024-06-29T15:57:22.234573+0000","last_became_peered":"2024-06-29T15:57:22.234573+0000","last_unstale":"2024-06-29T15:57:33.283723+0000","last_undegraded":"2024-06-29T15:57:33.283723+0000","last_fullsized":"2024-06-29T15:57:33.283723+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:48.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.285269+0000","last_change":"2024-06-29T15:57:22.233275+0000","last_active":"2024-06-29T15:57:33.285269+0000","last_peered":"2024-06-29T15:57:33.285269+0000","last_clean":"2024-06-29T15:57:33.285269+0000","last_became_active":"2024-06-29T15:57:22.230801+0000","last_became_peered":"2024-06-29T15:57:22.230801+0000","last_unstale":"2024-06-29T15:57:33.285269+0000","last_undegraded":"2024-06-29T15:57:33.285269+0000","last_fullsized":"2024-06-29T15:57:33.285269+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:24.134069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.286822+0000","last_change":"2024-06-29T15:57:22.223324+0000","last_active":"2024-06-29T15:57:33.286822+0000","last_peered":"2024-06-29T15:57:33.286822+0000","last_clean":"2024-06-29T15:57:33.286822+0000","last_became_active":"2024-06-29T15:57:22.222958+0000","last_became_peered":"2024-06-29T15:57:22.222958+0000","last_unstale":"2024-06-29T15:57:33.286822+0000","last_undegraded":"2024-06-29T15:57:33.286822+0000","last_fullsized":"2024-06-29T15:57:33.286822+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:21.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.288844+0000","last_change":"2024-06-29T15:57:22.226496+0000","last_active":"2024-06-29T15:57:33.288844+0000","last_peered":"2024-06-29T15:57:33.288844+0000","last_clean":"2024-06-29T15:57:33.288844+0000","last_became_active":"2024-06-29T15:57:22.226191+0000","last_became_peered":"2024-06-29T15:57:22.226191+0000","last_unstale":"2024-06-29T15:57:33.288844+0000","last_undegraded":"2024-06-29T15:57:33.288844+0000","last_fullsized":"2024-06-29T15:57:33.288844+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:32.644449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.287426+0000","last_change":"2024-06-29T15:57:22.218369+0000","last_active":"2024-06-29T15:57:33.287426+0000","last_peered":"2024-06-29T15:57:33.287426+0000","last_clean":"2024-06-29T15:57:33.287426+0000","last_became_active":"2024-06-29T15:57:22.217584+0000","last_became_peered":"2024-06-29T15:57:22.217584+0000","last_unstale":"2024-06-29T15:57:33.287426+0000","last_undegraded":"2024-06-29T15:57:33.287426+0000","last_fullsized":"2024-06-29T15:57:33.287426+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:59.760761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291286+0000","last_change":"2024-06-29T15:57:22.236022+0000","last_active":"2024-06-29T15:57:33.291286+0000","last_peered":"2024-06-29T15:57:33.291286+0000","last_clean":"2024-06-29T15:57:33.291286+0000","last_became_active":"2024-06-29T15:57:22.235712+0000","last_became_peered":"2024-06-29T15:57:22.235712+0000","last_unstale":"2024-06-29T15:57:33.291286+0000","last_undegraded":"2024-06-29T15:57:33.291286+0000","last_fullsized":"2024-06-29T15:57:33.291286+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:57.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.283276+0000","last_change":"2024-06-29T15:57:22.231073+0000","last_active":"2024-06-29T15:57:33.283276+0000","last_peered":"2024-06-29T15:57:33.283276+0000","last_clean":"2024-06-29T15:57:33.283276+0000","last_became_active":"2024-06-29T15:57:22.230749+0000","last_became_peered":"2024-06-29T15:57:22.230749+0000","last_unstale":"2024-06-29T15:57:33.283276+0000","last_undegraded":"2024-06-29T15:57:33.283276+0000","last_fullsized":"2024-06-29T15:57:33.283276+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:23.168668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.289947+0000","last_change":"2024-06-29T15:57:22.231733+0000","last_active":"2024-06-29T15:57:33.289947+0000","last_peered":"2024-06-29T15:57:33.289947+0000","last_clean":"2024-06-29T15:57:33.289947+0000","last_became_active":"2024-06-29T15:57:22.231387+0000","last_became_peered":"2024-06-29T15:57:22.231387+0000","last_unstale":"2024-06-29T15:57:33.289947+0000","last_undegraded":"2024-06-29T15:57:33.289947+0000","last_fullsized":"2024-06-29T15:57:33.289947+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:41.892974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-29T15:57:34.280660+0000","last_change":"2024-06-29T15:57:34.280660+0000","last_active":"2024-06-29T15:57:34.280660+0000","last_peered":"2024-06-29T15:57:34.280660+0000","last_clean":"2024-06-29T15:57:34.280660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:34.280660+0000","last_undegraded":"2024-06-29T15:57:34.280660+0000","last_fullsized":"2024-06-29T15:57:34.280660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.291260+0000","last_change":"2024-06-29T15:57:22.225592+0000","last_active":"2024-06-29T15:57:33.291260+0000","last_peered":"2024-06-29T15:57:33.291260+0000","last_clean":"2024-06-29T15:57:33.291260+0000","last_became_active":"2024-06-29T15:57:22.225309+0000","last_became_peered":"2024-06-29T15:57:22.225309+0000","last_unstale":"2024-06-29T15:57:33.291260+0000","last_undegraded":"2024-06-29T15:57:33.291260+0000","last_fullsized":"2024-06-29T15:57:33.291260+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:07.857349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543289,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52552,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319288,"statfs":{"total":96636764160,"available":96582950912,"internally_reserved":0,"allocated":3403776,"data_stored":2030256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543290,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42244,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329596,"statfs":{"total":96636764160,"available":96593506304,"internally_reserved":0,"allocated":2809856,"data_stored":1440400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543289,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47804,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324036,"statfs":{"total":96636764160,"available":96587812864,"internally_reserved":0,"allocated":2801664,"data_stored":1437019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543288,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51948,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319892,"statfs":{"total":96636764160,"available":96583569408,"internally_reserved":0,"allocated":2785280,"data_stored":1430422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510586,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47808,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324032,"statfs":{"total":96636764160,"available":96587808768,"internally_reserved":0,"allocated":2805760,"data_stored":1437089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47800,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324040,"statfs":{"total":96636764160,"available":96587816960,"internally_reserved":0,"allocated":2797568,"data_stored":1437053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510585,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47792,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324048,"statfs":{"total":96636764160,"available":96587825152,"internally_reserved":0,"allocated":2781184,"data_stored":1429840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510587,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42812,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329028,"statfs":{"total":96636764160,"available":96592924672,"internally_reserved":0,"allocated":3391488,"data_stored":2027421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:39.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.937+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3332114455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbd00cf5c0 msgr2=0x7fcbd00d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.937+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3332114455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbd00cf5c0 0x7fcbd00d1a20 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8009830 tx=0x7fcbb802a510 comp rx=0 tx=0).stop 2024-06-29T15:57:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.937+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3332114455 shutdown_connections 2024-06-29T15:57:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.937+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3332114455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbd00cf5c0 0x7fcbd00d1a20 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.937+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3332114455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbd008c190 0x7fcbd00cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.937+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3332114455 >> 172.21.15.59:0/3332114455 conn(0x7fcbd00b5610 msgr2=0x7fcbd00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.937+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3332114455 shutdown_connections 2024-06-29T15:57:39.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.937+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3332114455 wait complete. 2024-06-29T15:57:39.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.939+0000 7fcbd5f54640 1 Processor -- start 2024-06-29T15:57:39.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.939+0000 7fcbd5f54640 1 -- start start 2024-06-29T15:57:39.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbd5f54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbd008c190 0x7fcbd015d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbd5f54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd00ce950 0x7fcbd00aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbd5f54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbd00cf5c0 0x7fcbd00ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbd5f54640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd00b7fa0 con 0x7fcbd00ce950 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbd5f54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd00b7e20 con 0x7fcbd008c190 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbd5f54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd00b8120 con 0x7fcbd00cf5c0 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbc6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd00ce950 0x7fcbd00aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbc6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd00ce950 0x7fcbd00aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44768/0 (socket says 172.21.15.59:44768) 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbc6ffd640 1 -- 172.21.15.59:0/3972397873 learned_addr learned my addr 172.21.15.59:0/3972397873 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbcf7fe640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbd008c190 0x7fcbd015d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.940+0000 7fcbcffff640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbd00cf5c0 0x7fcbd00ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbc6ffd640 1 -- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbd00cf5c0 msgr2=0x7fcbd00ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbc6ffd640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbd00cf5c0 0x7fcbd00ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbc6ffd640 1 -- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbd008c190 msgr2=0x7fcbd015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbc6ffd640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbd008c190 0x7fcbd015d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbc6ffd640 1 -- 172.21.15.59:0/3972397873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd00ab1e0 con 0x7fcbd00ce950 2024-06-29T15:57:39.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbc6ffd640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd00ce950 0x7fcbd00aa4f0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8009950 tx=0x7fcbb8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbccff9640 1 -- 172.21.15.59:0/3972397873 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8031d60 con 0x7fcbd00ce950 2024-06-29T15:57:39.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbccff9640 1 -- 172.21.15.59:0/3972397873 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbb8033e10 con 0x7fcbd00ce950 2024-06-29T15:57:39.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd00ab4c0 con 0x7fcbd00ce950 2024-06-29T15:57:39.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbccff9640 1 -- 172.21.15.59:0/3972397873 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8005260 con 0x7fcbd00ce950 2024-06-29T15:57:39.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.941+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd00aba80 con 0x7fcbd00ce950 2024-06-29T15:57:39.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.942+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd00aaf50 con 0x7fcbd00ce950 2024-06-29T15:57:39.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.943+0000 7fcbccff9640 1 -- 172.21.15.59:0/3972397873 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fcbb8005400 con 0x7fcbd00ce950 2024-06-29T15:57:39.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.943+0000 7fcbccff9640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fcbb00763f0 0x7fcbb0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.944+0000 7fcbccff9640 1 -- 172.21.15.59:0/3972397873 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7fcbb80b8f50 con 0x7fcbd00ce950 2024-06-29T15:57:39.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.944+0000 7fcbcf7fe640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fcbb00763f0 0x7fcbb0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.944+0000 7fcbcf7fe640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fcbb00763f0 0x7fcbb0078900 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00095f0 tx=0x7fcbc0005f90 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:39.948+0000 7fcbccff9640 1 -- 172.21.15.59:0/3972397873 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcbb803f030 con 0x7fcbd00ce950 2024-06-29T15:57:40.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.135+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcbd00c9b10 con 0x7fcbb00763f0 2024-06-29T15:57:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.144+0000 7fcbccff9640 1 -- 172.21.15.59:0/3972397873 <== mgr.9206 v2:172.21.15.153:6832/2780049427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491475 (secure 0 0 0) 0x7fcbd00c9b10 con 0x7fcbb00763f0 2024-06-29T15:57:40.150 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:40.153 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fcbb00763f0 msgr2=0x7fcbb0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fcbb00763f0 0x7fcbb0078900 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00095f0 tx=0x7fcbc0005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd00ce950 msgr2=0x7fcbd00aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd00ce950 0x7fcbd00aa4f0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8009950 tx=0x7fcbb8033c20 comp rx=0 tx=0).stop 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 shutdown_connections 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fcbb00763f0 0x7fcbb0078900 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcbd00cf5c0 0x7fcbd00ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd00ce950 0x7fcbd00aa4f0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 --2- 172.21.15.59:0/3972397873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbd008c190 0x7fcbd015d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 >> 172.21.15.59:0/3972397873 conn(0x7fcbd00b5610 msgr2=0x7fcbd00c8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 shutdown_connections 2024-06-29T15:57:40.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.148+0000 7fcbd5f54640 1 -- 172.21.15.59:0/3972397873 wait complete. 2024-06-29T15:57:40.160 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-06-29T15:57:39.769670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":469,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":356,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381160,"kb_used_data":23400,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593560,"statfs":{"total":773094113280,"available":772703805440,"internally_reserved":0,"allocated":23961600,"data_stored":12874902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":366202418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001764"},"pg_stats":[{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331581+0000","last_change":"2024-06-29T15:57:35.257416+0000","last_active":"2024-06-29T15:57:36.331581+0000","last_peered":"2024-06-29T15:57:36.331581+0000","last_clean":"2024-06-29T15:57:36.331581+0000","last_became_active":"2024-06-29T15:57:35.256867+0000","last_became_peered":"2024-06-29T15:57:35.256867+0000","last_unstale":"2024-06-29T15:57:36.331581+0000","last_undegraded":"2024-06-29T15:57:36.331581+0000","last_fullsized":"2024-06-29T15:57:36.331581+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:30:33.761276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"210'233","reported_seq":1109,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311738+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:57:36.311738+0000","last_peered":"2024-06-29T15:57:36.311738+0000","last_clean":"2024-06-29T15:57:36.311738+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:57:36.311738+0000","last_undegraded":"2024-06-29T15:57:36.311738+0000","last_fullsized":"2024-06-29T15:57:36.311738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331626+0000","last_change":"2024-06-29T15:57:21.209163+0000","last_active":"2024-06-29T15:57:36.331626+0000","last_peered":"2024-06-29T15:57:36.331626+0000","last_clean":"2024-06-29T15:57:36.331626+0000","last_became_active":"2024-06-29T15:57:21.208825+0000","last_became_peered":"2024-06-29T15:57:21.208825+0000","last_unstale":"2024-06-29T15:57:36.331626+0000","last_undegraded":"2024-06-29T15:57:36.331626+0000","last_fullsized":"2024-06-29T15:57:36.331626+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:13.641268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237943+0000","last_change":"2024-06-29T15:57:22.230794+0000","last_active":"2024-06-29T15:57:36.237943+0000","last_peered":"2024-06-29T15:57:36.237943+0000","last_clean":"2024-06-29T15:57:36.237943+0000","last_became_active":"2024-06-29T15:57:22.230474+0000","last_became_peered":"2024-06-29T15:57:22.230474+0000","last_unstale":"2024-06-29T15:57:36.237943+0000","last_undegraded":"2024-06-29T15:57:36.237943+0000","last_fullsized":"2024-06-29T15:57:36.237943+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:43.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312052+0000","last_change":"2024-06-29T15:57:34.312155+0000","last_active":"2024-06-29T15:57:36.312052+0000","last_peered":"2024-06-29T15:57:36.312052+0000","last_clean":"2024-06-29T15:57:36.312052+0000","last_became_active":"2024-06-29T15:57:34.311830+0000","last_became_peered":"2024-06-29T15:57:34.311830+0000","last_unstale":"2024-06-29T15:57:36.312052+0000","last_undegraded":"2024-06-29T15:57:36.312052+0000","last_fullsized":"2024-06-29T15:57:36.312052+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:58:13.468981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312504+0000","last_change":"2024-06-29T15:57:35.255842+0000","last_active":"2024-06-29T15:57:36.312504+0000","last_peered":"2024-06-29T15:57:36.312504+0000","last_clean":"2024-06-29T15:57:36.312504+0000","last_became_active":"2024-06-29T15:57:35.255195+0000","last_became_peered":"2024-06-29T15:57:35.255195+0000","last_unstale":"2024-06-29T15:57:36.312504+0000","last_undegraded":"2024-06-29T15:57:36.312504+0000","last_fullsized":"2024-06-29T15:57:36.312504+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:38.905359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331919+0000","last_change":"2024-06-29T15:57:21.215589+0000","last_active":"2024-06-29T15:57:36.331919+0000","last_peered":"2024-06-29T15:57:36.331919+0000","last_clean":"2024-06-29T15:57:36.331919+0000","last_became_active":"2024-06-29T15:57:21.213719+0000","last_became_peered":"2024-06-29T15:57:21.213719+0000","last_unstale":"2024-06-29T15:57:36.331919+0000","last_undegraded":"2024-06-29T15:57:36.331919+0000","last_fullsized":"2024-06-29T15:57:36.331919+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:32.413502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236935+0000","last_change":"2024-06-29T15:57:22.223950+0000","last_active":"2024-06-29T15:57:36.236935+0000","last_peered":"2024-06-29T15:57:36.236935+0000","last_clean":"2024-06-29T15:57:36.236935+0000","last_became_active":"2024-06-29T15:57:22.223622+0000","last_became_peered":"2024-06-29T15:57:22.223622+0000","last_unstale":"2024-06-29T15:57:36.236935+0000","last_undegraded":"2024-06-29T15:57:36.236935+0000","last_fullsized":"2024-06-29T15:57:36.236935+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.556171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"226'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236795+0000","last_change":"2024-06-29T15:57:34.307024+0000","last_active":"2024-06-29T15:57:36.236795+0000","last_peered":"2024-06-29T15:57:36.236795+0000","last_clean":"2024-06-29T15:57:36.236795+0000","last_became_active":"2024-06-29T15:57:34.306637+0000","last_became_peered":"2024-06-29T15:57:34.306637+0000","last_unstale":"2024-06-29T15:57:36.236795+0000","last_undegraded":"2024-06-29T15:57:36.236795+0000","last_fullsized":"2024-06-29T15:57:36.236795+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:30.362344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329816+0000","last_change":"2024-06-29T15:57:35.254053+0000","last_active":"2024-06-29T15:57:36.329816+0000","last_peered":"2024-06-29T15:57:36.329816+0000","last_clean":"2024-06-29T15:57:36.329816+0000","last_became_active":"2024-06-29T15:57:35.253616+0000","last_became_peered":"2024-06-29T15:57:35.253616+0000","last_unstale":"2024-06-29T15:57:36.329816+0000","last_undegraded":"2024-06-29T15:57:36.329816+0000","last_fullsized":"2024-06-29T15:57:36.329816+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:13.830138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238919+0000","last_change":"2024-06-29T15:57:21.212837+0000","last_active":"2024-06-29T15:57:36.238919+0000","last_peered":"2024-06-29T15:57:36.238919+0000","last_clean":"2024-06-29T15:57:36.238919+0000","last_became_active":"2024-06-29T15:57:21.212491+0000","last_became_peered":"2024-06-29T15:57:21.212491+0000","last_unstale":"2024-06-29T15:57:36.238919+0000","last_undegraded":"2024-06-29T15:57:36.238919+0000","last_fullsized":"2024-06-29T15:57:36.238919+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:35:53.641775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312497+0000","last_change":"2024-06-29T15:57:22.237722+0000","last_active":"2024-06-29T15:57:36.312497+0000","last_peered":"2024-06-29T15:57:36.312497+0000","last_clean":"2024-06-29T15:57:36.312497+0000","last_became_active":"2024-06-29T15:57:22.237372+0000","last_became_peered":"2024-06-29T15:57:22.237372+0000","last_unstale":"2024-06-29T15:57:36.312497+0000","last_undegraded":"2024-06-29T15:57:36.312497+0000","last_fullsized":"2024-06-29T15:57:36.312497+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:24.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239300+0000","last_change":"2024-06-29T15:57:34.309853+0000","last_active":"2024-06-29T15:57:36.239300+0000","last_peered":"2024-06-29T15:57:36.239300+0000","last_clean":"2024-06-29T15:57:36.239300+0000","last_became_active":"2024-06-29T15:57:34.309541+0000","last_became_peered":"2024-06-29T15:57:34.309541+0000","last_unstale":"2024-06-29T15:57:36.239300+0000","last_undegraded":"2024-06-29T15:57:36.239300+0000","last_fullsized":"2024-06-29T15:57:36.239300+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:17:45.264114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331328+0000","last_change":"2024-06-29T15:57:35.255572+0000","last_active":"2024-06-29T15:57:36.331328+0000","last_peered":"2024-06-29T15:57:36.331328+0000","last_clean":"2024-06-29T15:57:36.331328+0000","last_became_active":"2024-06-29T15:57:35.255216+0000","last_became_peered":"2024-06-29T15:57:35.255216+0000","last_unstale":"2024-06-29T15:57:36.331328+0000","last_undegraded":"2024-06-29T15:57:36.331328+0000","last_fullsized":"2024-06-29T15:57:36.331328+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:59.819495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331738+0000","last_change":"2024-06-29T15:57:21.210563+0000","last_active":"2024-06-29T15:57:36.331738+0000","last_peered":"2024-06-29T15:57:36.331738+0000","last_clean":"2024-06-29T15:57:36.331738+0000","last_became_active":"2024-06-29T15:57:21.210243+0000","last_became_peered":"2024-06-29T15:57:21.210243+0000","last_unstale":"2024-06-29T15:57:36.331738+0000","last_undegraded":"2024-06-29T15:57:36.331738+0000","last_fullsized":"2024-06-29T15:57:36.331738+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.900240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312769+0000","last_change":"2024-06-29T15:57:22.244396+0000","last_active":"2024-06-29T15:57:36.312769+0000","last_peered":"2024-06-29T15:57:36.312769+0000","last_clean":"2024-06-29T15:57:36.312769+0000","last_became_active":"2024-06-29T15:57:22.244044+0000","last_became_peered":"2024-06-29T15:57:22.244044+0000","last_unstale":"2024-06-29T15:57:36.312769+0000","last_undegraded":"2024-06-29T15:57:36.312769+0000","last_fullsized":"2024-06-29T15:57:36.312769+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.981276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239376+0000","last_change":"2024-06-29T15:57:34.314135+0000","last_active":"2024-06-29T15:57:36.239376+0000","last_peered":"2024-06-29T15:57:36.239376+0000","last_clean":"2024-06-29T15:57:36.239376+0000","last_became_active":"2024-06-29T15:57:34.313741+0000","last_became_peered":"2024-06-29T15:57:34.313741+0000","last_unstale":"2024-06-29T15:57:36.239376+0000","last_undegraded":"2024-06-29T15:57:36.239376+0000","last_fullsized":"2024-06-29T15:57:36.239376+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:12:29.995937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238236+0000","last_change":"2024-06-29T15:57:35.251995+0000","last_active":"2024-06-29T15:57:36.238236+0000","last_peered":"2024-06-29T15:57:36.238236+0000","last_clean":"2024-06-29T15:57:36.238236+0000","last_became_active":"2024-06-29T15:57:35.251543+0000","last_became_peered":"2024-06-29T15:57:35.251543+0000","last_unstale":"2024-06-29T15:57:36.238236+0000","last_undegraded":"2024-06-29T15:57:36.238236+0000","last_fullsized":"2024-06-29T15:57:36.238236+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:30.799799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240227+0000","last_change":"2024-06-29T15:57:21.213113+0000","last_active":"2024-06-29T15:57:36.240227+0000","last_peered":"2024-06-29T15:57:36.240227+0000","last_clean":"2024-06-29T15:57:36.240227+0000","last_became_active":"2024-06-29T15:57:21.212819+0000","last_became_peered":"2024-06-29T15:57:21.212819+0000","last_unstale":"2024-06-29T15:57:36.240227+0000","last_undegraded":"2024-06-29T15:57:36.240227+0000","last_fullsized":"2024-06-29T15:57:36.240227+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:59.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239776+0000","last_change":"2024-06-29T15:57:22.229230+0000","last_active":"2024-06-29T15:57:36.239776+0000","last_peered":"2024-06-29T15:57:36.239776+0000","last_clean":"2024-06-29T15:57:36.239776+0000","last_became_active":"2024-06-29T15:57:22.228914+0000","last_became_peered":"2024-06-29T15:57:22.228914+0000","last_unstale":"2024-06-29T15:57:36.239776+0000","last_undegraded":"2024-06-29T15:57:36.239776+0000","last_fullsized":"2024-06-29T15:57:36.239776+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:09.918042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236907+0000","last_change":"2024-06-29T15:57:34.307100+0000","last_active":"2024-06-29T15:57:36.236907+0000","last_peered":"2024-06-29T15:57:36.236907+0000","last_clean":"2024-06-29T15:57:36.236907+0000","last_became_active":"2024-06-29T15:57:34.306718+0000","last_became_peered":"2024-06-29T15:57:34.306718+0000","last_unstale":"2024-06-29T15:57:36.236907+0000","last_undegraded":"2024-06-29T15:57:36.236907+0000","last_fullsized":"2024-06-29T15:57:36.236907+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:55.770944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331213+0000","last_change":"2024-06-29T15:57:35.249106+0000","last_active":"2024-06-29T15:57:36.331213+0000","last_peered":"2024-06-29T15:57:36.331213+0000","last_clean":"2024-06-29T15:57:36.331213+0000","last_became_active":"2024-06-29T15:57:35.248469+0000","last_became_peered":"2024-06-29T15:57:35.248469+0000","last_unstale":"2024-06-29T15:57:36.331213+0000","last_undegraded":"2024-06-29T15:57:36.331213+0000","last_fullsized":"2024-06-29T15:57:36.331213+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:21:47.954549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240435+0000","last_change":"2024-06-29T15:57:21.216399+0000","last_active":"2024-06-29T15:57:36.240435+0000","last_peered":"2024-06-29T15:57:36.240435+0000","last_clean":"2024-06-29T15:57:36.240435+0000","last_became_active":"2024-06-29T15:57:21.216104+0000","last_became_peered":"2024-06-29T15:57:21.216104+0000","last_unstale":"2024-06-29T15:57:36.240435+0000","last_undegraded":"2024-06-29T15:57:36.240435+0000","last_fullsized":"2024-06-29T15:57:36.240435+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239349+0000","last_change":"2024-06-29T15:57:22.227969+0000","last_active":"2024-06-29T15:57:36.239349+0000","last_peered":"2024-06-29T15:57:36.239349+0000","last_clean":"2024-06-29T15:57:36.239349+0000","last_became_active":"2024-06-29T15:57:22.227579+0000","last_became_peered":"2024-06-29T15:57:22.227579+0000","last_unstale":"2024-06-29T15:57:36.239349+0000","last_undegraded":"2024-06-29T15:57:36.239349+0000","last_fullsized":"2024-06-29T15:57:36.239349+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329273+0000","last_change":"2024-06-29T15:57:34.306550+0000","last_active":"2024-06-29T15:57:36.329273+0000","last_peered":"2024-06-29T15:57:36.329273+0000","last_clean":"2024-06-29T15:57:36.329273+0000","last_became_active":"2024-06-29T15:57:34.306222+0000","last_became_peered":"2024-06-29T15:57:34.306222+0000","last_unstale":"2024-06-29T15:57:36.329273+0000","last_undegraded":"2024-06-29T15:57:36.329273+0000","last_fullsized":"2024-06-29T15:57:36.329273+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:57.908667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330924+0000","last_change":"2024-06-29T15:57:35.257493+0000","last_active":"2024-06-29T15:57:36.330924+0000","last_peered":"2024-06-29T15:57:36.330924+0000","last_clean":"2024-06-29T15:57:36.330924+0000","last_became_active":"2024-06-29T15:57:35.256381+0000","last_became_peered":"2024-06-29T15:57:35.256381+0000","last_unstale":"2024-06-29T15:57:36.330924+0000","last_undegraded":"2024-06-29T15:57:36.330924+0000","last_fullsized":"2024-06-29T15:57:36.330924+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:13.192616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330024+0000","last_change":"2024-06-29T15:57:21.217039+0000","last_active":"2024-06-29T15:57:36.330024+0000","last_peered":"2024-06-29T15:57:36.330024+0000","last_clean":"2024-06-29T15:57:36.330024+0000","last_became_active":"2024-06-29T15:57:21.216542+0000","last_became_peered":"2024-06-29T15:57:21.216542+0000","last_unstale":"2024-06-29T15:57:36.330024+0000","last_undegraded":"2024-06-29T15:57:36.330024+0000","last_fullsized":"2024-06-29T15:57:36.330024+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:21.729257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331424+0000","last_change":"2024-06-29T15:57:22.233919+0000","last_active":"2024-06-29T15:57:36.331424+0000","last_peered":"2024-06-29T15:57:36.331424+0000","last_clean":"2024-06-29T15:57:36.331424+0000","last_became_active":"2024-06-29T15:57:22.231028+0000","last_became_peered":"2024-06-29T15:57:22.231028+0000","last_unstale":"2024-06-29T15:57:36.331424+0000","last_undegraded":"2024-06-29T15:57:36.331424+0000","last_fullsized":"2024-06-29T15:57:36.331424+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:34:24.910914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239552+0000","last_change":"2024-06-29T15:57:34.312110+0000","last_active":"2024-06-29T15:57:36.239552+0000","last_peered":"2024-06-29T15:57:36.239552+0000","last_clean":"2024-06-29T15:57:36.239552+0000","last_became_active":"2024-06-29T15:57:34.311789+0000","last_became_peered":"2024-06-29T15:57:34.311789+0000","last_unstale":"2024-06-29T15:57:36.239552+0000","last_undegraded":"2024-06-29T15:57:36.239552+0000","last_fullsized":"2024-06-29T15:57:36.239552+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:37.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239893+0000","last_change":"2024-06-29T15:57:35.256330+0000","last_active":"2024-06-29T15:57:36.239893+0000","last_peered":"2024-06-29T15:57:36.239893+0000","last_clean":"2024-06-29T15:57:36.239893+0000","last_became_active":"2024-06-29T15:57:35.256005+0000","last_became_peered":"2024-06-29T15:57:35.256005+0000","last_unstale":"2024-06-29T15:57:36.239893+0000","last_undegraded":"2024-06-29T15:57:36.239893+0000","last_fullsized":"2024-06-29T15:57:36.239893+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:03.579383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.6","version":"222'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238934+0000","last_change":"2024-06-29T15:57:21.203822+0000","last_active":"2024-06-29T15:57:36.238934+0000","last_peered":"2024-06-29T15:57:36.238934+0000","last_clean":"2024-06-29T15:57:36.238934+0000","last_became_active":"2024-06-29T15:57:21.203517+0000","last_became_peered":"2024-06-29T15:57:21.203517+0000","last_unstale":"2024-06-29T15:57:36.238934+0000","last_undegraded":"2024-06-29T15:57:36.238934+0000","last_fullsized":"2024-06-29T15:57:36.238934+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:05.810997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236238+0000","last_change":"2024-06-29T15:57:22.223140+0000","last_active":"2024-06-29T15:57:36.236238+0000","last_peered":"2024-06-29T15:57:36.236238+0000","last_clean":"2024-06-29T15:57:36.236238+0000","last_became_active":"2024-06-29T15:57:22.222839+0000","last_became_peered":"2024-06-29T15:57:22.222839+0000","last_unstale":"2024-06-29T15:57:36.236238+0000","last_undegraded":"2024-06-29T15:57:36.236238+0000","last_fullsized":"2024-06-29T15:57:36.236238+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:17.520475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238524+0000","last_change":"2024-06-29T15:57:34.308352+0000","last_active":"2024-06-29T15:57:36.238524+0000","last_peered":"2024-06-29T15:57:36.238524+0000","last_clean":"2024-06-29T15:57:36.238524+0000","last_became_active":"2024-06-29T15:57:34.308040+0000","last_became_peered":"2024-06-29T15:57:34.308040+0000","last_unstale":"2024-06-29T15:57:36.238524+0000","last_undegraded":"2024-06-29T15:57:36.238524+0000","last_fullsized":"2024-06-29T15:57:36.238524+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:34.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329855+0000","last_change":"2024-06-29T15:57:35.251788+0000","last_active":"2024-06-29T15:57:36.329855+0000","last_peered":"2024-06-29T15:57:36.329855+0000","last_clean":"2024-06-29T15:57:36.329855+0000","last_became_active":"2024-06-29T15:57:35.251451+0000","last_became_peered":"2024-06-29T15:57:35.251451+0000","last_unstale":"2024-06-29T15:57:36.329855+0000","last_undegraded":"2024-06-29T15:57:36.329855+0000","last_fullsized":"2024-06-29T15:57:36.329855+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:14.366892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330636+0000","last_change":"2024-06-29T15:57:21.212745+0000","last_active":"2024-06-29T15:57:36.330636+0000","last_peered":"2024-06-29T15:57:36.330636+0000","last_clean":"2024-06-29T15:57:36.330636+0000","last_became_active":"2024-06-29T15:57:21.212335+0000","last_became_peered":"2024-06-29T15:57:21.212335+0000","last_unstale":"2024-06-29T15:57:36.330636+0000","last_undegraded":"2024-06-29T15:57:36.330636+0000","last_fullsized":"2024-06-29T15:57:36.330636+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:07.049637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239385+0000","last_change":"2024-06-29T15:57:22.224294+0000","last_active":"2024-06-29T15:57:36.239385+0000","last_peered":"2024-06-29T15:57:36.239385+0000","last_clean":"2024-06-29T15:57:36.239385+0000","last_became_active":"2024-06-29T15:57:22.223922+0000","last_became_peered":"2024-06-29T15:57:22.223922+0000","last_unstale":"2024-06-29T15:57:36.239385+0000","last_undegraded":"2024-06-29T15:57:36.239385+0000","last_fullsized":"2024-06-29T15:57:36.239385+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:46.725456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311940+0000","last_change":"2024-06-29T15:57:34.308042+0000","last_active":"2024-06-29T15:57:36.311940+0000","last_peered":"2024-06-29T15:57:36.311940+0000","last_clean":"2024-06-29T15:57:36.311940+0000","last_became_active":"2024-06-29T15:57:34.307706+0000","last_became_peered":"2024-06-29T15:57:34.307706+0000","last_unstale":"2024-06-29T15:57:36.311940+0000","last_undegraded":"2024-06-29T15:57:36.311940+0000","last_fullsized":"2024-06-29T15:57:36.311940+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:24.276099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312202+0000","last_change":"2024-06-29T15:57:35.254882+0000","last_active":"2024-06-29T15:57:36.312202+0000","last_peered":"2024-06-29T15:57:36.312202+0000","last_clean":"2024-06-29T15:57:36.312202+0000","last_became_active":"2024-06-29T15:57:35.254543+0000","last_became_peered":"2024-06-29T15:57:35.254543+0000","last_unstale":"2024-06-29T15:57:36.312202+0000","last_undegraded":"2024-06-29T15:57:36.312202+0000","last_fullsized":"2024-06-29T15:57:36.312202+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:03:35.376526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237536+0000","last_change":"2024-06-29T15:57:21.211203+0000","last_active":"2024-06-29T15:57:36.237536+0000","last_peered":"2024-06-29T15:57:36.237536+0000","last_clean":"2024-06-29T15:57:36.237536+0000","last_became_active":"2024-06-29T15:57:21.210898+0000","last_became_peered":"2024-06-29T15:57:21.210898+0000","last_unstale":"2024-06-29T15:57:36.237536+0000","last_undegraded":"2024-06-29T15:57:36.237536+0000","last_fullsized":"2024-06-29T15:57:36.237536+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:17.270875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240196+0000","last_change":"2024-06-29T15:57:22.213920+0000","last_active":"2024-06-29T15:57:36.240196+0000","last_peered":"2024-06-29T15:57:36.240196+0000","last_clean":"2024-06-29T15:57:36.240196+0000","last_became_active":"2024-06-29T15:57:22.213583+0000","last_became_peered":"2024-06-29T15:57:22.213583+0000","last_unstale":"2024-06-29T15:57:36.240196+0000","last_undegraded":"2024-06-29T15:57:36.240196+0000","last_fullsized":"2024-06-29T15:57:36.240196+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:31.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331622+0000","last_change":"2024-06-29T15:57:34.308946+0000","last_active":"2024-06-29T15:57:36.331622+0000","last_peered":"2024-06-29T15:57:36.331622+0000","last_clean":"2024-06-29T15:57:36.331622+0000","last_became_active":"2024-06-29T15:57:34.308411+0000","last_became_peered":"2024-06-29T15:57:34.308411+0000","last_unstale":"2024-06-29T15:57:36.331622+0000","last_undegraded":"2024-06-29T15:57:36.331622+0000","last_fullsized":"2024-06-29T15:57:36.331622+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:13.989668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330052+0000","last_change":"2024-06-29T15:57:35.259846+0000","last_active":"2024-06-29T15:57:36.330052+0000","last_peered":"2024-06-29T15:57:36.330052+0000","last_clean":"2024-06-29T15:57:36.330052+0000","last_became_active":"2024-06-29T15:57:35.259493+0000","last_became_peered":"2024-06-29T15:57:35.259493+0000","last_unstale":"2024-06-29T15:57:36.330052+0000","last_undegraded":"2024-06-29T15:57:36.330052+0000","last_fullsized":"2024-06-29T15:57:36.330052+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:15.812204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329676+0000","last_change":"2024-06-29T15:57:21.212085+0000","last_active":"2024-06-29T15:57:36.329676+0000","last_peered":"2024-06-29T15:57:36.329676+0000","last_clean":"2024-06-29T15:57:36.329676+0000","last_became_active":"2024-06-29T15:57:21.211731+0000","last_became_peered":"2024-06-29T15:57:21.211731+0000","last_unstale":"2024-06-29T15:57:36.329676+0000","last_undegraded":"2024-06-29T15:57:36.329676+0000","last_fullsized":"2024-06-29T15:57:36.329676+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:36.061054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330288+0000","last_change":"2024-06-29T15:57:22.228211+0000","last_active":"2024-06-29T15:57:36.330288+0000","last_peered":"2024-06-29T15:57:36.330288+0000","last_clean":"2024-06-29T15:57:36.330288+0000","last_became_active":"2024-06-29T15:57:22.227519+0000","last_became_peered":"2024-06-29T15:57:22.227519+0000","last_unstale":"2024-06-29T15:57:36.330288+0000","last_undegraded":"2024-06-29T15:57:36.330288+0000","last_fullsized":"2024-06-29T15:57:36.330288+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:16:35.999038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331724+0000","last_change":"2024-06-29T15:57:34.314930+0000","last_active":"2024-06-29T15:57:36.331724+0000","last_peered":"2024-06-29T15:57:36.331724+0000","last_clean":"2024-06-29T15:57:36.331724+0000","last_became_active":"2024-06-29T15:57:34.314549+0000","last_became_peered":"2024-06-29T15:57:34.314549+0000","last_unstale":"2024-06-29T15:57:36.331724+0000","last_undegraded":"2024-06-29T15:57:36.331724+0000","last_fullsized":"2024-06-29T15:57:36.331724+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:02.172151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239694+0000","last_change":"2024-06-29T15:57:35.257200+0000","last_active":"2024-06-29T15:57:36.239694+0000","last_peered":"2024-06-29T15:57:36.239694+0000","last_clean":"2024-06-29T15:57:36.239694+0000","last_became_active":"2024-06-29T15:57:35.256810+0000","last_became_peered":"2024-06-29T15:57:35.256810+0000","last_unstale":"2024-06-29T15:57:36.239694+0000","last_undegraded":"2024-06-29T15:57:36.239694+0000","last_fullsized":"2024-06-29T15:57:36.239694+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:42.431331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330636+0000","last_change":"2024-06-29T15:57:21.208592+0000","last_active":"2024-06-29T15:57:36.330636+0000","last_peered":"2024-06-29T15:57:36.330636+0000","last_clean":"2024-06-29T15:57:36.330636+0000","last_became_active":"2024-06-29T15:57:21.208156+0000","last_became_peered":"2024-06-29T15:57:21.208156+0000","last_unstale":"2024-06-29T15:57:36.330636+0000","last_undegraded":"2024-06-29T15:57:36.330636+0000","last_fullsized":"2024-06-29T15:57:36.330636+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:20.325402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312669+0000","last_change":"2024-06-29T15:57:22.238829+0000","last_active":"2024-06-29T15:57:36.312669+0000","last_peered":"2024-06-29T15:57:36.312669+0000","last_clean":"2024-06-29T15:57:36.312669+0000","last_became_active":"2024-06-29T15:57:22.238520+0000","last_became_peered":"2024-06-29T15:57:22.238520+0000","last_unstale":"2024-06-29T15:57:36.312669+0000","last_undegraded":"2024-06-29T15:57:36.312669+0000","last_fullsized":"2024-06-29T15:57:36.312669+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:22.127384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239235+0000","last_change":"2024-06-29T15:57:34.308672+0000","last_active":"2024-06-29T15:57:36.239235+0000","last_peered":"2024-06-29T15:57:36.239235+0000","last_clean":"2024-06-29T15:57:36.239235+0000","last_became_active":"2024-06-29T15:57:34.307347+0000","last_became_peered":"2024-06-29T15:57:34.307347+0000","last_unstale":"2024-06-29T15:57:36.239235+0000","last_undegraded":"2024-06-29T15:57:36.239235+0000","last_fullsized":"2024-06-29T15:57:36.239235+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:53:19.725300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332378+0000","last_change":"2024-06-29T15:57:35.248756+0000","last_active":"2024-06-29T15:57:36.332378+0000","last_peered":"2024-06-29T15:57:36.332378+0000","last_clean":"2024-06-29T15:57:36.332378+0000","last_became_active":"2024-06-29T15:57:35.248281+0000","last_became_peered":"2024-06-29T15:57:35.248281+0000","last_unstale":"2024-06-29T15:57:36.332378+0000","last_undegraded":"2024-06-29T15:57:36.332378+0000","last_fullsized":"2024-06-29T15:57:36.332378+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:50.250829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331396+0000","last_change":"2024-06-29T15:57:21.215358+0000","last_active":"2024-06-29T15:57:36.331396+0000","last_peered":"2024-06-29T15:57:36.331396+0000","last_clean":"2024-06-29T15:57:36.331396+0000","last_became_active":"2024-06-29T15:57:21.213979+0000","last_became_peered":"2024-06-29T15:57:21.213979+0000","last_unstale":"2024-06-29T15:57:36.331396+0000","last_undegraded":"2024-06-29T15:57:36.331396+0000","last_fullsized":"2024-06-29T15:57:36.331396+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:42.446135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331975+0000","last_change":"2024-06-29T15:57:22.231883+0000","last_active":"2024-06-29T15:57:36.331975+0000","last_peered":"2024-06-29T15:57:36.331975+0000","last_clean":"2024-06-29T15:57:36.331975+0000","last_became_active":"2024-06-29T15:57:22.231588+0000","last_became_peered":"2024-06-29T15:57:22.231588+0000","last_unstale":"2024-06-29T15:57:36.331975+0000","last_undegraded":"2024-06-29T15:57:36.331975+0000","last_fullsized":"2024-06-29T15:57:36.331975+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:17.705760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238496+0000","last_change":"2024-06-29T15:57:34.307517+0000","last_active":"2024-06-29T15:57:36.238496+0000","last_peered":"2024-06-29T15:57:36.238496+0000","last_clean":"2024-06-29T15:57:36.238496+0000","last_became_active":"2024-06-29T15:57:34.307213+0000","last_became_peered":"2024-06-29T15:57:34.307213+0000","last_unstale":"2024-06-29T15:57:36.238496+0000","last_undegraded":"2024-06-29T15:57:36.238496+0000","last_fullsized":"2024-06-29T15:57:36.238496+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:02.055681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311679+0000","last_change":"2024-06-29T15:57:35.262345+0000","last_active":"2024-06-29T15:57:36.311679+0000","last_peered":"2024-06-29T15:57:36.311679+0000","last_clean":"2024-06-29T15:57:36.311679+0000","last_became_active":"2024-06-29T15:57:35.262052+0000","last_became_peered":"2024-06-29T15:57:35.262052+0000","last_unstale":"2024-06-29T15:57:36.311679+0000","last_undegraded":"2024-06-29T15:57:36.311679+0000","last_fullsized":"2024-06-29T15:57:36.311679+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:54.683205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330760+0000","last_change":"2024-06-29T15:57:21.213193+0000","last_active":"2024-06-29T15:57:36.330760+0000","last_peered":"2024-06-29T15:57:36.330760+0000","last_clean":"2024-06-29T15:57:36.330760+0000","last_became_active":"2024-06-29T15:57:21.212838+0000","last_became_peered":"2024-06-29T15:57:21.212838+0000","last_unstale":"2024-06-29T15:57:36.330760+0000","last_undegraded":"2024-06-29T15:57:36.330760+0000","last_fullsized":"2024-06-29T15:57:36.330760+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:54.574391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240083+0000","last_change":"2024-06-29T15:57:22.217630+0000","last_active":"2024-06-29T15:57:36.240083+0000","last_peered":"2024-06-29T15:57:36.240083+0000","last_clean":"2024-06-29T15:57:36.240083+0000","last_became_active":"2024-06-29T15:57:22.217303+0000","last_became_peered":"2024-06-29T15:57:22.217303+0000","last_unstale":"2024-06-29T15:57:36.240083+0000","last_undegraded":"2024-06-29T15:57:36.240083+0000","last_fullsized":"2024-06-29T15:57:36.240083+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:05.554814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236277+0000","last_change":"2024-06-29T15:57:34.305621+0000","last_active":"2024-06-29T15:57:36.236277+0000","last_peered":"2024-06-29T15:57:36.236277+0000","last_clean":"2024-06-29T15:57:36.236277+0000","last_became_active":"2024-06-29T15:57:34.305274+0000","last_became_peered":"2024-06-29T15:57:34.305274+0000","last_unstale":"2024-06-29T15:57:36.236277+0000","last_undegraded":"2024-06-29T15:57:36.236277+0000","last_fullsized":"2024-06-29T15:57:36.236277+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:58.216624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238128+0000","last_change":"2024-06-29T15:57:35.258213+0000","last_active":"2024-06-29T15:57:36.238128+0000","last_peered":"2024-06-29T15:57:36.238128+0000","last_clean":"2024-06-29T15:57:36.238128+0000","last_became_active":"2024-06-29T15:57:35.257901+0000","last_became_peered":"2024-06-29T15:57:35.257901+0000","last_unstale":"2024-06-29T15:57:36.238128+0000","last_undegraded":"2024-06-29T15:57:36.238128+0000","last_fullsized":"2024-06-29T15:57:36.238128+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:34.564002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330032+0000","last_change":"2024-06-29T15:57:21.210528+0000","last_active":"2024-06-29T15:57:36.330032+0000","last_peered":"2024-06-29T15:57:36.330032+0000","last_clean":"2024-06-29T15:57:36.330032+0000","last_became_active":"2024-06-29T15:57:21.208691+0000","last_became_peered":"2024-06-29T15:57:21.208691+0000","last_unstale":"2024-06-29T15:57:36.330032+0000","last_undegraded":"2024-06-29T15:57:36.330032+0000","last_fullsized":"2024-06-29T15:57:36.330032+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.762590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240196+0000","last_change":"2024-06-29T15:57:22.232727+0000","last_active":"2024-06-29T15:57:36.240196+0000","last_peered":"2024-06-29T15:57:36.240196+0000","last_clean":"2024-06-29T15:57:36.240196+0000","last_became_active":"2024-06-29T15:57:22.232392+0000","last_became_peered":"2024-06-29T15:57:22.232392+0000","last_unstale":"2024-06-29T15:57:36.240196+0000","last_undegraded":"2024-06-29T15:57:36.240196+0000","last_fullsized":"2024-06-29T15:57:36.240196+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:54.805984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239217+0000","last_change":"2024-06-29T15:57:34.308385+0000","last_active":"2024-06-29T15:57:36.239217+0000","last_peered":"2024-06-29T15:57:36.239217+0000","last_clean":"2024-06-29T15:57:36.239217+0000","last_became_active":"2024-06-29T15:57:34.308070+0000","last_became_peered":"2024-06-29T15:57:34.308070+0000","last_unstale":"2024-06-29T15:57:36.239217+0000","last_undegraded":"2024-06-29T15:57:36.239217+0000","last_fullsized":"2024-06-29T15:57:36.239217+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:11.812167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331467+0000","last_change":"2024-06-29T15:57:35.256356+0000","last_active":"2024-06-29T15:57:36.331467+0000","last_peered":"2024-06-29T15:57:36.331467+0000","last_clean":"2024-06-29T15:57:36.331467+0000","last_became_active":"2024-06-29T15:57:35.255846+0000","last_became_peered":"2024-06-29T15:57:35.255846+0000","last_unstale":"2024-06-29T15:57:36.331467+0000","last_undegraded":"2024-06-29T15:57:36.331467+0000","last_fullsized":"2024-06-29T15:57:36.331467+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:41.823081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.e","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238583+0000","last_change":"2024-06-29T15:57:21.216179+0000","last_active":"2024-06-29T15:57:36.238583+0000","last_peered":"2024-06-29T15:57:36.238583+0000","last_clean":"2024-06-29T15:57:36.238583+0000","last_became_active":"2024-06-29T15:57:21.215876+0000","last_became_peered":"2024-06-29T15:57:21.215876+0000","last_unstale":"2024-06-29T15:57:36.238583+0000","last_undegraded":"2024-06-29T15:57:36.238583+0000","last_fullsized":"2024-06-29T15:57:36.238583+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:37.942186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238721+0000","last_change":"2024-06-29T15:57:22.225793+0000","last_active":"2024-06-29T15:57:36.238721+0000","last_peered":"2024-06-29T15:57:36.238721+0000","last_clean":"2024-06-29T15:57:36.238721+0000","last_became_active":"2024-06-29T15:57:22.224806+0000","last_became_peered":"2024-06-29T15:57:22.224806+0000","last_unstale":"2024-06-29T15:57:36.238721+0000","last_undegraded":"2024-06-29T15:57:36.238721+0000","last_fullsized":"2024-06-29T15:57:36.238721+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:03.667477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329189+0000","last_change":"2024-06-29T15:57:34.311124+0000","last_active":"2024-06-29T15:57:36.329189+0000","last_peered":"2024-06-29T15:57:36.329189+0000","last_clean":"2024-06-29T15:57:36.329189+0000","last_became_active":"2024-06-29T15:57:34.310775+0000","last_became_peered":"2024-06-29T15:57:34.310775+0000","last_unstale":"2024-06-29T15:57:36.329189+0000","last_undegraded":"2024-06-29T15:57:36.329189+0000","last_fullsized":"2024-06-29T15:57:36.329189+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:50.229318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331196+0000","last_change":"2024-06-29T15:57:35.259909+0000","last_active":"2024-06-29T15:57:36.331196+0000","last_peered":"2024-06-29T15:57:36.331196+0000","last_clean":"2024-06-29T15:57:36.331196+0000","last_became_active":"2024-06-29T15:57:35.259564+0000","last_became_peered":"2024-06-29T15:57:35.259564+0000","last_unstale":"2024-06-29T15:57:36.331196+0000","last_undegraded":"2024-06-29T15:57:36.331196+0000","last_fullsized":"2024-06-29T15:57:36.331196+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:18.743864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.f","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239899+0000","last_change":"2024-06-29T15:57:21.212718+0000","last_active":"2024-06-29T15:57:36.239899+0000","last_peered":"2024-06-29T15:57:36.239899+0000","last_clean":"2024-06-29T15:57:36.239899+0000","last_became_active":"2024-06-29T15:57:21.212406+0000","last_became_peered":"2024-06-29T15:57:21.212406+0000","last_unstale":"2024-06-29T15:57:36.239899+0000","last_undegraded":"2024-06-29T15:57:36.239899+0000","last_fullsized":"2024-06-29T15:57:36.239899+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:00.180622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239223+0000","last_change":"2024-06-29T15:57:22.238347+0000","last_active":"2024-06-29T15:57:36.239223+0000","last_peered":"2024-06-29T15:57:36.239223+0000","last_clean":"2024-06-29T15:57:36.239223+0000","last_became_active":"2024-06-29T15:57:22.236896+0000","last_became_peered":"2024-06-29T15:57:22.236896+0000","last_unstale":"2024-06-29T15:57:36.239223+0000","last_undegraded":"2024-06-29T15:57:36.239223+0000","last_fullsized":"2024-06-29T15:57:36.239223+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:39.739194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236003+0000","last_change":"2024-06-29T15:57:34.308954+0000","last_active":"2024-06-29T15:57:36.236003+0000","last_peered":"2024-06-29T15:57:36.236003+0000","last_clean":"2024-06-29T15:57:36.236003+0000","last_became_active":"2024-06-29T15:57:34.308595+0000","last_became_peered":"2024-06-29T15:57:34.308595+0000","last_unstale":"2024-06-29T15:57:36.236003+0000","last_undegraded":"2024-06-29T15:57:36.236003+0000","last_fullsized":"2024-06-29T15:57:36.236003+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:06:10.871555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236850+0000","last_change":"2024-06-29T15:57:35.250645+0000","last_active":"2024-06-29T15:57:36.236850+0000","last_peered":"2024-06-29T15:57:36.236850+0000","last_clean":"2024-06-29T15:57:36.236850+0000","last_became_active":"2024-06-29T15:57:35.250314+0000","last_became_peered":"2024-06-29T15:57:35.250314+0000","last_unstale":"2024-06-29T15:57:36.236850+0000","last_undegraded":"2024-06-29T15:57:36.236850+0000","last_fullsized":"2024-06-29T15:57:36.236850+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:06.391987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.10","version":"221'8","reported_seq":30,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329441+0000","last_change":"2024-06-29T15:57:21.216156+0000","last_active":"2024-06-29T15:57:36.329441+0000","last_peered":"2024-06-29T15:57:36.329441+0000","last_clean":"2024-06-29T15:57:36.329441+0000","last_became_active":"2024-06-29T15:57:21.214088+0000","last_became_peered":"2024-06-29T15:57:21.214088+0000","last_unstale":"2024-06-29T15:57:36.329441+0000","last_undegraded":"2024-06-29T15:57:36.329441+0000","last_fullsized":"2024-06-29T15:57:36.329441+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:30.314243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312937+0000","last_change":"2024-06-29T15:57:22.223112+0000","last_active":"2024-06-29T15:57:36.312937+0000","last_peered":"2024-06-29T15:57:36.312937+0000","last_clean":"2024-06-29T15:57:36.312937+0000","last_became_active":"2024-06-29T15:57:22.222632+0000","last_became_peered":"2024-06-29T15:57:22.222632+0000","last_unstale":"2024-06-29T15:57:36.312937+0000","last_undegraded":"2024-06-29T15:57:36.312937+0000","last_fullsized":"2024-06-29T15:57:36.312937+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:05.408776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329903+0000","last_change":"2024-06-29T15:57:34.310931+0000","last_active":"2024-06-29T15:57:36.329903+0000","last_peered":"2024-06-29T15:57:36.329903+0000","last_clean":"2024-06-29T15:57:36.329903+0000","last_became_active":"2024-06-29T15:57:34.310598+0000","last_became_peered":"2024-06-29T15:57:34.310598+0000","last_unstale":"2024-06-29T15:57:36.329903+0000","last_undegraded":"2024-06-29T15:57:36.329903+0000","last_fullsized":"2024-06-29T15:57:36.329903+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:51.824136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236659+0000","last_change":"2024-06-29T15:57:35.250035+0000","last_active":"2024-06-29T15:57:36.236659+0000","last_peered":"2024-06-29T15:57:36.236659+0000","last_clean":"2024-06-29T15:57:36.236659+0000","last_became_active":"2024-06-29T15:57:35.249711+0000","last_became_peered":"2024-06-29T15:57:35.249711+0000","last_unstale":"2024-06-29T15:57:36.236659+0000","last_undegraded":"2024-06-29T15:57:36.236659+0000","last_fullsized":"2024-06-29T15:57:36.236659+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:13.347781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329612+0000","last_change":"2024-06-29T15:57:21.215032+0000","last_active":"2024-06-29T15:57:36.329612+0000","last_peered":"2024-06-29T15:57:36.329612+0000","last_clean":"2024-06-29T15:57:36.329612+0000","last_became_active":"2024-06-29T15:57:21.214557+0000","last_became_peered":"2024-06-29T15:57:21.214557+0000","last_unstale":"2024-06-29T15:57:36.329612+0000","last_undegraded":"2024-06-29T15:57:36.329612+0000","last_fullsized":"2024-06-29T15:57:36.329612+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330185+0000","last_change":"2024-06-29T15:57:22.228484+0000","last_active":"2024-06-29T15:57:36.330185+0000","last_peered":"2024-06-29T15:57:36.330185+0000","last_clean":"2024-06-29T15:57:36.330185+0000","last_became_active":"2024-06-29T15:57:22.227880+0000","last_became_peered":"2024-06-29T15:57:22.227880+0000","last_unstale":"2024-06-29T15:57:36.330185+0000","last_undegraded":"2024-06-29T15:57:36.330185+0000","last_fullsized":"2024-06-29T15:57:36.330185+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:21.749071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312893+0000","last_change":"2024-06-29T15:57:34.309910+0000","last_active":"2024-06-29T15:57:36.312893+0000","last_peered":"2024-06-29T15:57:36.312893+0000","last_clean":"2024-06-29T15:57:36.312893+0000","last_became_active":"2024-06-29T15:57:34.309509+0000","last_became_peered":"2024-06-29T15:57:34.309509+0000","last_unstale":"2024-06-29T15:57:36.312893+0000","last_undegraded":"2024-06-29T15:57:36.312893+0000","last_fullsized":"2024-06-29T15:57:36.312893+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:05.013352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329983+0000","last_change":"2024-06-29T15:57:35.254963+0000","last_active":"2024-06-29T15:57:36.329983+0000","last_peered":"2024-06-29T15:57:36.329983+0000","last_clean":"2024-06-29T15:57:36.329983+0000","last_became_active":"2024-06-29T15:57:35.254059+0000","last_became_peered":"2024-06-29T15:57:35.254059+0000","last_unstale":"2024-06-29T15:57:36.329983+0000","last_undegraded":"2024-06-29T15:57:36.329983+0000","last_fullsized":"2024-06-29T15:57:36.329983+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:42.786307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236445+0000","last_change":"2024-06-29T15:57:21.203410+0000","last_active":"2024-06-29T15:57:36.236445+0000","last_peered":"2024-06-29T15:57:36.236445+0000","last_clean":"2024-06-29T15:57:36.236445+0000","last_became_active":"2024-06-29T15:57:21.203034+0000","last_became_peered":"2024-06-29T15:57:21.203034+0000","last_unstale":"2024-06-29T15:57:36.236445+0000","last_undegraded":"2024-06-29T15:57:36.236445+0000","last_fullsized":"2024-06-29T15:57:36.236445+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:52.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237205+0000","last_change":"2024-06-29T15:57:22.241881+0000","last_active":"2024-06-29T15:57:36.237205+0000","last_peered":"2024-06-29T15:57:36.237205+0000","last_clean":"2024-06-29T15:57:36.237205+0000","last_became_active":"2024-06-29T15:57:22.241549+0000","last_became_peered":"2024-06-29T15:57:22.241549+0000","last_unstale":"2024-06-29T15:57:36.237205+0000","last_undegraded":"2024-06-29T15:57:36.237205+0000","last_fullsized":"2024-06-29T15:57:36.237205+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:25.352328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"225'6","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240234+0000","last_change":"2024-06-29T15:57:34.309744+0000","last_active":"2024-06-29T15:57:36.240234+0000","last_peered":"2024-06-29T15:57:36.240234+0000","last_clean":"2024-06-29T15:57:36.240234+0000","last_became_active":"2024-06-29T15:57:34.309408+0000","last_became_peered":"2024-06-29T15:57:34.309408+0000","last_unstale":"2024-06-29T15:57:36.240234+0000","last_undegraded":"2024-06-29T15:57:36.240234+0000","last_fullsized":"2024-06-29T15:57:36.240234+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:20.219821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332388+0000","last_change":"2024-06-29T15:57:35.248181+0000","last_active":"2024-06-29T15:57:36.332388+0000","last_peered":"2024-06-29T15:57:36.332388+0000","last_clean":"2024-06-29T15:57:36.332388+0000","last_became_active":"2024-06-29T15:57:35.247656+0000","last_became_peered":"2024-06-29T15:57:35.247656+0000","last_unstale":"2024-06-29T15:57:36.332388+0000","last_undegraded":"2024-06-29T15:57:36.332388+0000","last_fullsized":"2024-06-29T15:57:36.332388+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:26.486217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331412+0000","last_change":"2024-06-29T15:57:21.215892+0000","last_active":"2024-06-29T15:57:36.331412+0000","last_peered":"2024-06-29T15:57:36.331412+0000","last_clean":"2024-06-29T15:57:36.331412+0000","last_became_active":"2024-06-29T15:57:21.215250+0000","last_became_peered":"2024-06-29T15:57:21.215250+0000","last_unstale":"2024-06-29T15:57:36.331412+0000","last_undegraded":"2024-06-29T15:57:36.331412+0000","last_fullsized":"2024-06-29T15:57:36.331412+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:23.234622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331871+0000","last_change":"2024-06-29T15:57:22.234802+0000","last_active":"2024-06-29T15:57:36.331871+0000","last_peered":"2024-06-29T15:57:36.331871+0000","last_clean":"2024-06-29T15:57:36.331871+0000","last_became_active":"2024-06-29T15:57:22.234492+0000","last_became_peered":"2024-06-29T15:57:22.234492+0000","last_unstale":"2024-06-29T15:57:36.331871+0000","last_undegraded":"2024-06-29T15:57:36.331871+0000","last_fullsized":"2024-06-29T15:57:36.331871+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:37.928014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"225'3","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312419+0000","last_change":"2024-06-29T15:57:34.308776+0000","last_active":"2024-06-29T15:57:36.312419+0000","last_peered":"2024-06-29T15:57:36.312419+0000","last_clean":"2024-06-29T15:57:36.312419+0000","last_became_active":"2024-06-29T15:57:34.308478+0000","last_became_peered":"2024-06-29T15:57:34.308478+0000","last_unstale":"2024-06-29T15:57:36.312419+0000","last_undegraded":"2024-06-29T15:57:36.312419+0000","last_fullsized":"2024-06-29T15:57:36.312419+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:14.899188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237305+0000","last_change":"2024-06-29T15:57:35.251086+0000","last_active":"2024-06-29T15:57:36.237305+0000","last_peered":"2024-06-29T15:57:36.237305+0000","last_clean":"2024-06-29T15:57:36.237305+0000","last_became_active":"2024-06-29T15:57:35.249526+0000","last_became_peered":"2024-06-29T15:57:35.249526+0000","last_unstale":"2024-06-29T15:57:36.237305+0000","last_undegraded":"2024-06-29T15:57:36.237305+0000","last_fullsized":"2024-06-29T15:57:36.237305+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:22.660265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.14","version":"222'8","reported_seq":30,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312307+0000","last_change":"2024-06-29T15:57:21.217755+0000","last_active":"2024-06-29T15:57:36.312307+0000","last_peered":"2024-06-29T15:57:36.312307+0000","last_clean":"2024-06-29T15:57:36.312307+0000","last_became_active":"2024-06-29T15:57:21.216983+0000","last_became_peered":"2024-06-29T15:57:21.216983+0000","last_unstale":"2024-06-29T15:57:36.312307+0000","last_undegraded":"2024-06-29T15:57:36.312307+0000","last_fullsized":"2024-06-29T15:57:36.312307+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:31.889343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331251+0000","last_change":"2024-06-29T15:57:22.224600+0000","last_active":"2024-06-29T15:57:36.331251+0000","last_peered":"2024-06-29T15:57:36.331251+0000","last_clean":"2024-06-29T15:57:36.331251+0000","last_became_active":"2024-06-29T15:57:22.224260+0000","last_became_peered":"2024-06-29T15:57:22.224260+0000","last_unstale":"2024-06-29T15:57:36.331251+0000","last_undegraded":"2024-06-29T15:57:36.331251+0000","last_fullsized":"2024-06-29T15:57:36.331251+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:21.773370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331316+0000","last_change":"2024-06-29T15:57:34.311602+0000","last_active":"2024-06-29T15:57:36.331316+0000","last_peered":"2024-06-29T15:57:36.331316+0000","last_clean":"2024-06-29T15:57:36.331316+0000","last_became_active":"2024-06-29T15:57:34.311208+0000","last_became_peered":"2024-06-29T15:57:34.311208+0000","last_unstale":"2024-06-29T15:57:36.331316+0000","last_undegraded":"2024-06-29T15:57:36.331316+0000","last_fullsized":"2024-06-29T15:57:36.331316+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:22.313530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329523+0000","last_change":"2024-06-29T15:57:35.257800+0000","last_active":"2024-06-29T15:57:36.329523+0000","last_peered":"2024-06-29T15:57:36.329523+0000","last_clean":"2024-06-29T15:57:36.329523+0000","last_became_active":"2024-06-29T15:57:35.254594+0000","last_became_peered":"2024-06-29T15:57:35.254594+0000","last_unstale":"2024-06-29T15:57:36.329523+0000","last_undegraded":"2024-06-29T15:57:36.329523+0000","last_fullsized":"2024-06-29T15:57:36.329523+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:10.058134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"221'3","reported_seq":25,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330966+0000","last_change":"2024-06-29T15:57:21.210270+0000","last_active":"2024-06-29T15:57:36.330966+0000","last_peered":"2024-06-29T15:57:36.330966+0000","last_clean":"2024-06-29T15:57:36.330966+0000","last_became_active":"2024-06-29T15:57:21.209731+0000","last_became_peered":"2024-06-29T15:57:21.209731+0000","last_unstale":"2024-06-29T15:57:36.330966+0000","last_undegraded":"2024-06-29T15:57:36.330966+0000","last_fullsized":"2024-06-29T15:57:36.330966+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:18.182233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331525+0000","last_change":"2024-06-29T15:57:22.231011+0000","last_active":"2024-06-29T15:57:36.331525+0000","last_peered":"2024-06-29T15:57:36.331525+0000","last_clean":"2024-06-29T15:57:36.331525+0000","last_became_active":"2024-06-29T15:57:22.230675+0000","last_became_peered":"2024-06-29T15:57:22.230675+0000","last_unstale":"2024-06-29T15:57:36.331525+0000","last_undegraded":"2024-06-29T15:57:36.331525+0000","last_fullsized":"2024-06-29T15:57:36.331525+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:09.478769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239084+0000","last_change":"2024-06-29T15:57:34.311119+0000","last_active":"2024-06-29T15:57:36.239084+0000","last_peered":"2024-06-29T15:57:36.239084+0000","last_clean":"2024-06-29T15:57:36.239084+0000","last_became_active":"2024-06-29T15:57:34.310820+0000","last_became_peered":"2024-06-29T15:57:34.310820+0000","last_unstale":"2024-06-29T15:57:36.239084+0000","last_undegraded":"2024-06-29T15:57:36.239084+0000","last_fullsized":"2024-06-29T15:57:36.239084+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:12:01.172748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236419+0000","last_change":"2024-06-29T15:57:35.252881+0000","last_active":"2024-06-29T15:57:36.236419+0000","last_peered":"2024-06-29T15:57:36.236419+0000","last_clean":"2024-06-29T15:57:36.236419+0000","last_became_active":"2024-06-29T15:57:35.252555+0000","last_became_peered":"2024-06-29T15:57:35.252555+0000","last_unstale":"2024-06-29T15:57:36.236419+0000","last_undegraded":"2024-06-29T15:57:36.236419+0000","last_fullsized":"2024-06-29T15:57:36.236419+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:13.830138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312629+0000","last_change":"2024-06-29T15:57:21.214536+0000","last_active":"2024-06-29T15:57:36.312629+0000","last_peered":"2024-06-29T15:57:36.312629+0000","last_clean":"2024-06-29T15:57:36.312629+0000","last_became_active":"2024-06-29T15:57:21.214227+0000","last_became_peered":"2024-06-29T15:57:21.214227+0000","last_unstale":"2024-06-29T15:57:36.312629+0000","last_undegraded":"2024-06-29T15:57:36.312629+0000","last_fullsized":"2024-06-29T15:57:36.312629+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:25.839522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236127+0000","last_change":"2024-06-29T15:57:22.219192+0000","last_active":"2024-06-29T15:57:36.236127+0000","last_peered":"2024-06-29T15:57:36.236127+0000","last_clean":"2024-06-29T15:57:36.236127+0000","last_became_active":"2024-06-29T15:57:22.218844+0000","last_became_peered":"2024-06-29T15:57:22.218844+0000","last_unstale":"2024-06-29T15:57:36.236127+0000","last_undegraded":"2024-06-29T15:57:36.236127+0000","last_fullsized":"2024-06-29T15:57:36.236127+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:50.717283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"225'8","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238266+0000","last_change":"2024-06-29T15:57:34.309117+0000","last_active":"2024-06-29T15:57:36.238266+0000","last_peered":"2024-06-29T15:57:36.238266+0000","last_clean":"2024-06-29T15:57:36.238266+0000","last_became_active":"2024-06-29T15:57:34.308792+0000","last_became_peered":"2024-06-29T15:57:34.308792+0000","last_unstale":"2024-06-29T15:57:36.238266+0000","last_undegraded":"2024-06-29T15:57:36.238266+0000","last_fullsized":"2024-06-29T15:57:36.238266+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:31:47.175085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330362+0000","last_change":"2024-06-29T15:57:35.256824+0000","last_active":"2024-06-29T15:57:36.330362+0000","last_peered":"2024-06-29T15:57:36.330362+0000","last_clean":"2024-06-29T15:57:36.330362+0000","last_became_active":"2024-06-29T15:57:35.255771+0000","last_became_peered":"2024-06-29T15:57:35.255771+0000","last_unstale":"2024-06-29T15:57:36.330362+0000","last_undegraded":"2024-06-29T15:57:36.330362+0000","last_fullsized":"2024-06-29T15:57:36.330362+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:51.718094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331403+0000","last_change":"2024-06-29T15:57:21.218814+0000","last_active":"2024-06-29T15:57:36.331403+0000","last_peered":"2024-06-29T15:57:36.331403+0000","last_clean":"2024-06-29T15:57:36.331403+0000","last_became_active":"2024-06-29T15:57:21.218495+0000","last_became_peered":"2024-06-29T15:57:21.218495+0000","last_unstale":"2024-06-29T15:57:36.331403+0000","last_undegraded":"2024-06-29T15:57:36.331403+0000","last_fullsized":"2024-06-29T15:57:36.331403+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:04.680014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329332+0000","last_change":"2024-06-29T15:57:22.235312+0000","last_active":"2024-06-29T15:57:36.329332+0000","last_peered":"2024-06-29T15:57:36.329332+0000","last_clean":"2024-06-29T15:57:36.329332+0000","last_became_active":"2024-06-29T15:57:22.234828+0000","last_became_peered":"2024-06-29T15:57:22.234828+0000","last_unstale":"2024-06-29T15:57:36.329332+0000","last_undegraded":"2024-06-29T15:57:36.329332+0000","last_fullsized":"2024-06-29T15:57:36.329332+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:39.661714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237083+0000","last_change":"2024-06-29T15:57:34.306092+0000","last_active":"2024-06-29T15:57:36.237083+0000","last_peered":"2024-06-29T15:57:36.237083+0000","last_clean":"2024-06-29T15:57:36.237083+0000","last_became_active":"2024-06-29T15:57:34.305717+0000","last_became_peered":"2024-06-29T15:57:34.305717+0000","last_unstale":"2024-06-29T15:57:36.237083+0000","last_undegraded":"2024-06-29T15:57:36.237083+0000","last_fullsized":"2024-06-29T15:57:36.237083+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:12.354679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329622+0000","last_change":"2024-06-29T15:57:35.244422+0000","last_active":"2024-06-29T15:57:36.329622+0000","last_peered":"2024-06-29T15:57:36.329622+0000","last_clean":"2024-06-29T15:57:36.329622+0000","last_became_active":"2024-06-29T15:57:35.244085+0000","last_became_peered":"2024-06-29T15:57:35.244085+0000","last_unstale":"2024-06-29T15:57:36.329622+0000","last_undegraded":"2024-06-29T15:57:36.329622+0000","last_fullsized":"2024-06-29T15:57:36.329622+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:31.011742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238696+0000","last_change":"2024-06-29T15:57:34.312096+0000","last_active":"2024-06-29T15:57:36.238696+0000","last_peered":"2024-06-29T15:57:36.238696+0000","last_clean":"2024-06-29T15:57:36.238696+0000","last_became_active":"2024-06-29T15:57:34.311757+0000","last_became_peered":"2024-06-29T15:57:34.311757+0000","last_unstale":"2024-06-29T15:57:36.238696+0000","last_undegraded":"2024-06-29T15:57:36.238696+0000","last_fullsized":"2024-06-29T15:57:36.238696+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:51.898399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236052+0000","last_change":"2024-06-29T15:57:21.208896+0000","last_active":"2024-06-29T15:57:36.236052+0000","last_peered":"2024-06-29T15:57:36.236052+0000","last_clean":"2024-06-29T15:57:36.236052+0000","last_became_active":"2024-06-29T15:57:21.208555+0000","last_became_peered":"2024-06-29T15:57:21.208555+0000","last_unstale":"2024-06-29T15:57:36.236052+0000","last_undegraded":"2024-06-29T15:57:36.236052+0000","last_fullsized":"2024-06-29T15:57:36.236052+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330531+0000","last_change":"2024-06-29T15:57:22.234743+0000","last_active":"2024-06-29T15:57:36.330531+0000","last_peered":"2024-06-29T15:57:36.330531+0000","last_clean":"2024-06-29T15:57:36.330531+0000","last_became_active":"2024-06-29T15:57:22.233091+0000","last_became_peered":"2024-06-29T15:57:22.233091+0000","last_unstale":"2024-06-29T15:57:36.330531+0000","last_undegraded":"2024-06-29T15:57:36.330531+0000","last_fullsized":"2024-06-29T15:57:36.330531+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:46.131681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331747+0000","last_change":"2024-06-29T15:57:35.247871+0000","last_active":"2024-06-29T15:57:36.331747+0000","last_peered":"2024-06-29T15:57:36.331747+0000","last_clean":"2024-06-29T15:57:36.331747+0000","last_became_active":"2024-06-29T15:57:35.247551+0000","last_became_peered":"2024-06-29T15:57:35.247551+0000","last_unstale":"2024-06-29T15:57:36.331747+0000","last_undegraded":"2024-06-29T15:57:36.331747+0000","last_fullsized":"2024-06-29T15:57:36.331747+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:38:28.226246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"225'4","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330982+0000","last_change":"2024-06-29T15:57:34.310706+0000","last_active":"2024-06-29T15:57:36.330982+0000","last_peered":"2024-06-29T15:57:36.330982+0000","last_clean":"2024-06-29T15:57:36.330982+0000","last_became_active":"2024-06-29T15:57:34.310360+0000","last_became_peered":"2024-06-29T15:57:34.310360+0000","last_unstale":"2024-06-29T15:57:36.330982+0000","last_undegraded":"2024-06-29T15:57:36.330982+0000","last_fullsized":"2024-06-29T15:57:36.330982+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:38.399869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311825+0000","last_change":"2024-06-29T15:57:21.212034+0000","last_active":"2024-06-29T15:57:36.311825+0000","last_peered":"2024-06-29T15:57:36.311825+0000","last_clean":"2024-06-29T15:57:36.311825+0000","last_became_active":"2024-06-29T15:57:21.211762+0000","last_became_peered":"2024-06-29T15:57:21.211762+0000","last_unstale":"2024-06-29T15:57:36.311825+0000","last_undegraded":"2024-06-29T15:57:36.311825+0000","last_fullsized":"2024-06-29T15:57:36.311825+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:36.775853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330016+0000","last_change":"2024-06-29T15:57:22.214168+0000","last_active":"2024-06-29T15:57:36.330016+0000","last_peered":"2024-06-29T15:57:36.330016+0000","last_clean":"2024-06-29T15:57:36.330016+0000","last_became_active":"2024-06-29T15:57:22.213780+0000","last_became_peered":"2024-06-29T15:57:22.213780+0000","last_unstale":"2024-06-29T15:57:36.330016+0000","last_undegraded":"2024-06-29T15:57:36.330016+0000","last_fullsized":"2024-06-29T15:57:36.330016+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:01.904725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238570+0000","last_change":"2024-06-29T15:57:35.257408+0000","last_active":"2024-06-29T15:57:36.238570+0000","last_peered":"2024-06-29T15:57:36.238570+0000","last_clean":"2024-06-29T15:57:36.238570+0000","last_became_active":"2024-06-29T15:57:35.257065+0000","last_became_peered":"2024-06-29T15:57:35.257065+0000","last_unstale":"2024-06-29T15:57:36.238570+0000","last_undegraded":"2024-06-29T15:57:36.238570+0000","last_fullsized":"2024-06-29T15:57:36.238570+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:32.986678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1b","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311976+0000","last_change":"2024-06-29T15:57:21.211113+0000","last_active":"2024-06-29T15:57:36.311976+0000","last_peered":"2024-06-29T15:57:36.311976+0000","last_clean":"2024-06-29T15:57:36.311976+0000","last_became_active":"2024-06-29T15:57:21.210817+0000","last_became_peered":"2024-06-29T15:57:21.210817+0000","last_unstale":"2024-06-29T15:57:36.311976+0000","last_undegraded":"2024-06-29T15:57:36.311976+0000","last_fullsized":"2024-06-29T15:57:36.311976+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:26.176884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329924+0000","last_change":"2024-06-29T15:57:22.235721+0000","last_active":"2024-06-29T15:57:36.329924+0000","last_peered":"2024-06-29T15:57:36.329924+0000","last_clean":"2024-06-29T15:57:36.329924+0000","last_became_active":"2024-06-29T15:57:22.235380+0000","last_became_peered":"2024-06-29T15:57:22.235380+0000","last_unstale":"2024-06-29T15:57:36.329924+0000","last_undegraded":"2024-06-29T15:57:36.329924+0000","last_fullsized":"2024-06-29T15:57:36.329924+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:38.144376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237819+0000","last_change":"2024-06-29T15:57:34.310685+0000","last_active":"2024-06-29T15:57:36.237819+0000","last_peered":"2024-06-29T15:57:36.237819+0000","last_clean":"2024-06-29T15:57:36.237819+0000","last_became_active":"2024-06-29T15:57:34.310377+0000","last_became_peered":"2024-06-29T15:57:34.310377+0000","last_unstale":"2024-06-29T15:57:36.237819+0000","last_undegraded":"2024-06-29T15:57:36.237819+0000","last_fullsized":"2024-06-29T15:57:36.237819+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:36.326091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237218+0000","last_change":"2024-06-29T15:57:35.251476+0000","last_active":"2024-06-29T15:57:36.237218+0000","last_peered":"2024-06-29T15:57:36.237218+0000","last_clean":"2024-06-29T15:57:36.237218+0000","last_became_active":"2024-06-29T15:57:35.250496+0000","last_became_peered":"2024-06-29T15:57:35.250496+0000","last_unstale":"2024-06-29T15:57:36.237218+0000","last_undegraded":"2024-06-29T15:57:36.237218+0000","last_fullsized":"2024-06-29T15:57:36.237218+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:52.738461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330242+0000","last_change":"2024-06-29T15:57:21.210643+0000","last_active":"2024-06-29T15:57:36.330242+0000","last_peered":"2024-06-29T15:57:36.330242+0000","last_clean":"2024-06-29T15:57:36.330242+0000","last_became_active":"2024-06-29T15:57:21.209805+0000","last_became_peered":"2024-06-29T15:57:21.209805+0000","last_unstale":"2024-06-29T15:57:36.330242+0000","last_undegraded":"2024-06-29T15:57:36.330242+0000","last_fullsized":"2024-06-29T15:57:36.330242+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:27.618049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331248+0000","last_change":"2024-06-29T15:57:22.226181+0000","last_active":"2024-06-29T15:57:36.331248+0000","last_peered":"2024-06-29T15:57:36.331248+0000","last_clean":"2024-06-29T15:57:36.331248+0000","last_became_active":"2024-06-29T15:57:22.225855+0000","last_became_peered":"2024-06-29T15:57:22.225855+0000","last_unstale":"2024-06-29T15:57:36.331248+0000","last_undegraded":"2024-06-29T15:57:36.331248+0000","last_fullsized":"2024-06-29T15:57:36.331248+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:37.067580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311945+0000","last_change":"2024-06-29T15:57:34.314461+0000","last_active":"2024-06-29T15:57:36.311945+0000","last_peered":"2024-06-29T15:57:36.311945+0000","last_clean":"2024-06-29T15:57:36.311945+0000","last_became_active":"2024-06-29T15:57:34.314139+0000","last_became_peered":"2024-06-29T15:57:34.314139+0000","last_unstale":"2024-06-29T15:57:36.311945+0000","last_undegraded":"2024-06-29T15:57:36.311945+0000","last_fullsized":"2024-06-29T15:57:36.311945+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:09.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330356+0000","last_change":"2024-06-29T15:57:35.246277+0000","last_active":"2024-06-29T15:57:36.330356+0000","last_peered":"2024-06-29T15:57:36.330356+0000","last_clean":"2024-06-29T15:57:36.330356+0000","last_became_active":"2024-06-29T15:57:35.245894+0000","last_became_peered":"2024-06-29T15:57:35.245894+0000","last_unstale":"2024-06-29T15:57:36.330356+0000","last_undegraded":"2024-06-29T15:57:36.330356+0000","last_fullsized":"2024-06-29T15:57:36.330356+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:16.673478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1d","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329758+0000","last_change":"2024-06-29T15:57:21.210602+0000","last_active":"2024-06-29T15:57:36.329758+0000","last_peered":"2024-06-29T15:57:36.329758+0000","last_clean":"2024-06-29T15:57:36.329758+0000","last_became_active":"2024-06-29T15:57:21.210296+0000","last_became_peered":"2024-06-29T15:57:21.210296+0000","last_unstale":"2024-06-29T15:57:36.329758+0000","last_undegraded":"2024-06-29T15:57:36.329758+0000","last_fullsized":"2024-06-29T15:57:36.329758+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:07.128352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238476+0000","last_change":"2024-06-29T15:57:22.213237+0000","last_active":"2024-06-29T15:57:36.238476+0000","last_peered":"2024-06-29T15:57:36.238476+0000","last_clean":"2024-06-29T15:57:36.238476+0000","last_became_active":"2024-06-29T15:57:22.212908+0000","last_became_peered":"2024-06-29T15:57:22.212908+0000","last_unstale":"2024-06-29T15:57:36.238476+0000","last_undegraded":"2024-06-29T15:57:36.238476+0000","last_fullsized":"2024-06-29T15:57:36.238476+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:31.343924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236505+0000","last_change":"2024-06-29T15:57:34.313977+0000","last_active":"2024-06-29T15:57:36.236505+0000","last_peered":"2024-06-29T15:57:36.236505+0000","last_clean":"2024-06-29T15:57:36.236505+0000","last_became_active":"2024-06-29T15:57:34.313702+0000","last_became_peered":"2024-06-29T15:57:34.313702+0000","last_unstale":"2024-06-29T15:57:36.236505+0000","last_undegraded":"2024-06-29T15:57:36.236505+0000","last_fullsized":"2024-06-29T15:57:36.236505+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:12.668316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329526+0000","last_change":"2024-06-29T15:57:35.242307+0000","last_active":"2024-06-29T15:57:36.329526+0000","last_peered":"2024-06-29T15:57:36.329526+0000","last_clean":"2024-06-29T15:57:36.329526+0000","last_became_active":"2024-06-29T15:57:35.241559+0000","last_became_peered":"2024-06-29T15:57:35.241559+0000","last_unstale":"2024-06-29T15:57:36.329526+0000","last_undegraded":"2024-06-29T15:57:36.329526+0000","last_fullsized":"2024-06-29T15:57:36.329526+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:37.228107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1e","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238472+0000","last_change":"2024-06-29T15:57:21.213288+0000","last_active":"2024-06-29T15:57:36.238472+0000","last_peered":"2024-06-29T15:57:36.238472+0000","last_clean":"2024-06-29T15:57:36.238472+0000","last_became_active":"2024-06-29T15:57:21.212954+0000","last_became_peered":"2024-06-29T15:57:21.212954+0000","last_unstale":"2024-06-29T15:57:36.238472+0000","last_undegraded":"2024-06-29T15:57:36.238472+0000","last_fullsized":"2024-06-29T15:57:36.238472+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:24.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237336+0000","last_change":"2024-06-29T15:57:22.224449+0000","last_active":"2024-06-29T15:57:36.237336+0000","last_peered":"2024-06-29T15:57:36.237336+0000","last_clean":"2024-06-29T15:57:36.237336+0000","last_became_active":"2024-06-29T15:57:22.224124+0000","last_became_peered":"2024-06-29T15:57:22.224124+0000","last_unstale":"2024-06-29T15:57:36.237336+0000","last_undegraded":"2024-06-29T15:57:36.237336+0000","last_fullsized":"2024-06-29T15:57:36.237336+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:05.649826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330422+0000","last_change":"2024-06-29T15:57:34.304015+0000","last_active":"2024-06-29T15:57:36.330422+0000","last_peered":"2024-06-29T15:57:36.330422+0000","last_clean":"2024-06-29T15:57:36.330422+0000","last_became_active":"2024-06-29T15:57:34.303686+0000","last_became_peered":"2024-06-29T15:57:34.303686+0000","last_unstale":"2024-06-29T15:57:36.330422+0000","last_undegraded":"2024-06-29T15:57:36.330422+0000","last_fullsized":"2024-06-29T15:57:36.330422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:48.678300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236409+0000","last_change":"2024-06-29T15:57:35.244368+0000","last_active":"2024-06-29T15:57:36.236409+0000","last_peered":"2024-06-29T15:57:36.236409+0000","last_clean":"2024-06-29T15:57:36.236409+0000","last_became_active":"2024-06-29T15:57:35.243599+0000","last_became_peered":"2024-06-29T15:57:35.243599+0000","last_unstale":"2024-06-29T15:57:36.236409+0000","last_undegraded":"2024-06-29T15:57:36.236409+0000","last_fullsized":"2024-06-29T15:57:36.236409+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:10.058134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1f","version":"221'4","reported_seq":26,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330083+0000","last_change":"2024-06-29T15:57:21.218674+0000","last_active":"2024-06-29T15:57:36.330083+0000","last_peered":"2024-06-29T15:57:36.330083+0000","last_clean":"2024-06-29T15:57:36.330083+0000","last_became_active":"2024-06-29T15:57:21.218397+0000","last_became_peered":"2024-06-29T15:57:21.218397+0000","last_unstale":"2024-06-29T15:57:36.330083+0000","last_undegraded":"2024-06-29T15:57:36.330083+0000","last_fullsized":"2024-06-29T15:57:36.330083+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.766844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330985+0000","last_change":"2024-06-29T15:57:22.228934+0000","last_active":"2024-06-29T15:57:36.330985+0000","last_peered":"2024-06-29T15:57:36.330985+0000","last_clean":"2024-06-29T15:57:36.330985+0000","last_became_active":"2024-06-29T15:57:22.228603+0000","last_became_peered":"2024-06-29T15:57:22.228603+0000","last_unstale":"2024-06-29T15:57:36.330985+0000","last_undegraded":"2024-06-29T15:57:36.330985+0000","last_fullsized":"2024-06-29T15:57:36.330985+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:28.391534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329161+0000","last_change":"2024-06-29T15:57:34.310834+0000","last_active":"2024-06-29T15:57:36.329161+0000","last_peered":"2024-06-29T15:57:36.329161+0000","last_clean":"2024-06-29T15:57:36.329161+0000","last_became_active":"2024-06-29T15:57:34.310527+0000","last_became_peered":"2024-06-29T15:57:34.310527+0000","last_unstale":"2024-06-29T15:57:36.329161+0000","last_undegraded":"2024-06-29T15:57:36.329161+0000","last_fullsized":"2024-06-29T15:57:36.329161+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:36.219011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239563+0000","last_change":"2024-06-29T15:57:35.256077+0000","last_active":"2024-06-29T15:57:36.239563+0000","last_peered":"2024-06-29T15:57:36.239563+0000","last_clean":"2024-06-29T15:57:36.239563+0000","last_became_active":"2024-06-29T15:57:35.255699+0000","last_became_peered":"2024-06-29T15:57:35.255699+0000","last_unstale":"2024-06-29T15:57:36.239563+0000","last_undegraded":"2024-06-29T15:57:36.239563+0000","last_fullsized":"2024-06-29T15:57:36.239563+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:30.089154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311494+0000","last_change":"2024-06-29T15:57:22.224277+0000","last_active":"2024-06-29T15:57:36.311494+0000","last_peered":"2024-06-29T15:57:36.311494+0000","last_clean":"2024-06-29T15:57:36.311494+0000","last_became_active":"2024-06-29T15:57:22.223975+0000","last_became_peered":"2024-06-29T15:57:22.223975+0000","last_unstale":"2024-06-29T15:57:36.311494+0000","last_undegraded":"2024-06-29T15:57:36.311494+0000","last_fullsized":"2024-06-29T15:57:36.311494+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:46.705935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329941+0000","last_change":"2024-06-29T15:57:35.257997+0000","last_active":"2024-06-29T15:57:36.329941+0000","last_peered":"2024-06-29T15:57:36.329941+0000","last_clean":"2024-06-29T15:57:36.329941+0000","last_became_active":"2024-06-29T15:57:35.254612+0000","last_became_peered":"2024-06-29T15:57:35.254612+0000","last_unstale":"2024-06-29T15:57:36.329941+0000","last_undegraded":"2024-06-29T15:57:36.329941+0000","last_fullsized":"2024-06-29T15:57:36.329941+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:13.347781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311649+0000","last_change":"2024-06-29T15:57:22.239789+0000","last_active":"2024-06-29T15:57:36.311649+0000","last_peered":"2024-06-29T15:57:36.311649+0000","last_clean":"2024-06-29T15:57:36.311649+0000","last_became_active":"2024-06-29T15:57:22.239438+0000","last_became_peered":"2024-06-29T15:57:22.239438+0000","last_unstale":"2024-06-29T15:57:36.311649+0000","last_undegraded":"2024-06-29T15:57:36.311649+0000","last_fullsized":"2024-06-29T15:57:36.311649+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:08.408484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237238+0000","last_change":"2024-06-29T15:57:35.251148+0000","last_active":"2024-06-29T15:57:36.237238+0000","last_peered":"2024-06-29T15:57:36.237238+0000","last_clean":"2024-06-29T15:57:36.237238+0000","last_became_active":"2024-06-29T15:57:35.250806+0000","last_became_peered":"2024-06-29T15:57:35.250806+0000","last_unstale":"2024-06-29T15:57:36.237238+0000","last_undegraded":"2024-06-29T15:57:36.237238+0000","last_fullsized":"2024-06-29T15:57:36.237238+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:54.987013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329513+0000","last_change":"2024-06-29T15:57:22.236151+0000","last_active":"2024-06-29T15:57:36.329513+0000","last_peered":"2024-06-29T15:57:36.329513+0000","last_clean":"2024-06-29T15:57:36.329513+0000","last_became_active":"2024-06-29T15:57:22.235409+0000","last_became_peered":"2024-06-29T15:57:22.235409+0000","last_unstale":"2024-06-29T15:57:36.329513+0000","last_undegraded":"2024-06-29T15:57:36.329513+0000","last_fullsized":"2024-06-29T15:57:36.329513+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:56.149438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329621+0000","last_change":"2024-06-29T15:57:35.255181+0000","last_active":"2024-06-29T15:57:36.329621+0000","last_peered":"2024-06-29T15:57:36.329621+0000","last_clean":"2024-06-29T15:57:36.329621+0000","last_became_active":"2024-06-29T15:57:35.254686+0000","last_became_peered":"2024-06-29T15:57:35.254686+0000","last_unstale":"2024-06-29T15:57:36.329621+0000","last_undegraded":"2024-06-29T15:57:36.329621+0000","last_fullsized":"2024-06-29T15:57:36.329621+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:52.833232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329737+0000","last_change":"2024-06-29T15:57:22.227876+0000","last_active":"2024-06-29T15:57:36.329737+0000","last_peered":"2024-06-29T15:57:36.329737+0000","last_clean":"2024-06-29T15:57:36.329737+0000","last_became_active":"2024-06-29T15:57:22.227591+0000","last_became_peered":"2024-06-29T15:57:22.227591+0000","last_unstale":"2024-06-29T15:57:36.329737+0000","last_undegraded":"2024-06-29T15:57:36.329737+0000","last_fullsized":"2024-06-29T15:57:36.329737+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:45.600754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332187+0000","last_change":"2024-06-29T15:57:35.257955+0000","last_active":"2024-06-29T15:57:36.332187+0000","last_peered":"2024-06-29T15:57:36.332187+0000","last_clean":"2024-06-29T15:57:36.332187+0000","last_became_active":"2024-06-29T15:57:35.256657+0000","last_became_peered":"2024-06-29T15:57:35.256657+0000","last_unstale":"2024-06-29T15:57:36.332187+0000","last_undegraded":"2024-06-29T15:57:36.332187+0000","last_fullsized":"2024-06-29T15:57:36.332187+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:57.126833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330655+0000","last_change":"2024-06-29T15:57:22.227845+0000","last_active":"2024-06-29T15:57:36.330655+0000","last_peered":"2024-06-29T15:57:36.330655+0000","last_clean":"2024-06-29T15:57:36.330655+0000","last_became_active":"2024-06-29T15:57:22.227549+0000","last_became_peered":"2024-06-29T15:57:22.227549+0000","last_unstale":"2024-06-29T15:57:36.330655+0000","last_undegraded":"2024-06-29T15:57:36.330655+0000","last_fullsized":"2024-06-29T15:57:36.330655+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:22.735783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312767+0000","last_change":"2024-06-29T15:57:35.258694+0000","last_active":"2024-06-29T15:57:36.312767+0000","last_peered":"2024-06-29T15:57:36.312767+0000","last_clean":"2024-06-29T15:57:36.312767+0000","last_became_active":"2024-06-29T15:57:35.258369+0000","last_became_peered":"2024-06-29T15:57:35.258369+0000","last_unstale":"2024-06-29T15:57:36.312767+0000","last_undegraded":"2024-06-29T15:57:36.312767+0000","last_fullsized":"2024-06-29T15:57:36.312767+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:30.546092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237188+0000","last_change":"2024-06-29T15:57:22.236939+0000","last_active":"2024-06-29T15:57:36.237188+0000","last_peered":"2024-06-29T15:57:36.237188+0000","last_clean":"2024-06-29T15:57:36.237188+0000","last_became_active":"2024-06-29T15:57:22.236129+0000","last_became_peered":"2024-06-29T15:57:22.236129+0000","last_unstale":"2024-06-29T15:57:36.237188+0000","last_undegraded":"2024-06-29T15:57:36.237188+0000","last_fullsized":"2024-06-29T15:57:36.237188+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:13.274461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239151+0000","last_change":"2024-06-29T15:57:35.248969+0000","last_active":"2024-06-29T15:57:36.239151+0000","last_peered":"2024-06-29T15:57:36.239151+0000","last_clean":"2024-06-29T15:57:36.239151+0000","last_became_active":"2024-06-29T15:57:35.248617+0000","last_became_peered":"2024-06-29T15:57:35.248617+0000","last_unstale":"2024-06-29T15:57:36.239151+0000","last_undegraded":"2024-06-29T15:57:36.239151+0000","last_fullsized":"2024-06-29T15:57:36.239151+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:23.344350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238364+0000","last_change":"2024-06-29T15:57:22.225280+0000","last_active":"2024-06-29T15:57:36.238364+0000","last_peered":"2024-06-29T15:57:36.238364+0000","last_clean":"2024-06-29T15:57:36.238364+0000","last_became_active":"2024-06-29T15:57:22.224986+0000","last_became_peered":"2024-06-29T15:57:22.224986+0000","last_unstale":"2024-06-29T15:57:36.238364+0000","last_undegraded":"2024-06-29T15:57:36.238364+0000","last_fullsized":"2024-06-29T15:57:36.238364+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:07.391769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329941+0000","last_change":"2024-06-29T15:57:35.249995+0000","last_active":"2024-06-29T15:57:36.329941+0000","last_peered":"2024-06-29T15:57:36.329941+0000","last_clean":"2024-06-29T15:57:36.329941+0000","last_became_active":"2024-06-29T15:57:35.249082+0000","last_became_peered":"2024-06-29T15:57:35.249082+0000","last_unstale":"2024-06-29T15:57:36.329941+0000","last_undegraded":"2024-06-29T15:57:36.329941+0000","last_fullsized":"2024-06-29T15:57:36.329941+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:18.648852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330431+0000","last_change":"2024-06-29T15:57:22.234902+0000","last_active":"2024-06-29T15:57:36.330431+0000","last_peered":"2024-06-29T15:57:36.330431+0000","last_clean":"2024-06-29T15:57:36.330431+0000","last_became_active":"2024-06-29T15:57:22.233244+0000","last_became_peered":"2024-06-29T15:57:22.233244+0000","last_unstale":"2024-06-29T15:57:36.330431+0000","last_undegraded":"2024-06-29T15:57:36.330431+0000","last_fullsized":"2024-06-29T15:57:36.330431+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:58.786612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236210+0000","last_change":"2024-06-29T15:57:35.243088+0000","last_active":"2024-06-29T15:57:36.236210+0000","last_peered":"2024-06-29T15:57:36.236210+0000","last_clean":"2024-06-29T15:57:36.236210+0000","last_became_active":"2024-06-29T15:57:35.242768+0000","last_became_peered":"2024-06-29T15:57:35.242768+0000","last_unstale":"2024-06-29T15:57:36.236210+0000","last_undegraded":"2024-06-29T15:57:36.236210+0000","last_fullsized":"2024-06-29T15:57:36.236210+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:43.817141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329319+0000","last_change":"2024-06-29T15:57:22.223814+0000","last_active":"2024-06-29T15:57:36.329319+0000","last_peered":"2024-06-29T15:57:36.329319+0000","last_clean":"2024-06-29T15:57:36.329319+0000","last_became_active":"2024-06-29T15:57:22.223464+0000","last_became_peered":"2024-06-29T15:57:22.223464+0000","last_unstale":"2024-06-29T15:57:36.329319+0000","last_undegraded":"2024-06-29T15:57:36.329319+0000","last_fullsized":"2024-06-29T15:57:36.329319+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:40.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331060+0000","last_change":"2024-06-29T15:57:35.256428+0000","last_active":"2024-06-29T15:57:36.331060+0000","last_peered":"2024-06-29T15:57:36.331060+0000","last_clean":"2024-06-29T15:57:36.331060+0000","last_became_active":"2024-06-29T15:57:35.255357+0000","last_became_peered":"2024-06-29T15:57:35.255357+0000","last_unstale":"2024-06-29T15:57:36.331060+0000","last_undegraded":"2024-06-29T15:57:36.331060+0000","last_fullsized":"2024-06-29T15:57:36.331060+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:10:51.973923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331341+0000","last_change":"2024-06-29T15:57:22.232432+0000","last_active":"2024-06-29T15:57:36.331341+0000","last_peered":"2024-06-29T15:57:36.331341+0000","last_clean":"2024-06-29T15:57:36.331341+0000","last_became_active":"2024-06-29T15:57:22.232102+0000","last_became_peered":"2024-06-29T15:57:22.232102+0000","last_unstale":"2024-06-29T15:57:36.331341+0000","last_undegraded":"2024-06-29T15:57:36.331341+0000","last_fullsized":"2024-06-29T15:57:36.331341+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:41.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311586+0000","last_change":"2024-06-29T15:57:35.259401+0000","last_active":"2024-06-29T15:57:36.311586+0000","last_peered":"2024-06-29T15:57:36.311586+0000","last_clean":"2024-06-29T15:57:36.311586+0000","last_became_active":"2024-06-29T15:57:35.259002+0000","last_became_peered":"2024-06-29T15:57:35.259002+0000","last_unstale":"2024-06-29T15:57:36.311586+0000","last_undegraded":"2024-06-29T15:57:36.311586+0000","last_fullsized":"2024-06-29T15:57:36.311586+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:34.435763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238215+0000","last_change":"2024-06-29T15:57:22.222075+0000","last_active":"2024-06-29T15:57:36.238215+0000","last_peered":"2024-06-29T15:57:36.238215+0000","last_clean":"2024-06-29T15:57:36.238215+0000","last_became_active":"2024-06-29T15:57:22.221767+0000","last_became_peered":"2024-06-29T15:57:22.221767+0000","last_unstale":"2024-06-29T15:57:36.238215+0000","last_undegraded":"2024-06-29T15:57:36.238215+0000","last_fullsized":"2024-06-29T15:57:36.238215+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:13.762887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329358+0000","last_change":"2024-06-29T15:57:35.249923+0000","last_active":"2024-06-29T15:57:36.329358+0000","last_peered":"2024-06-29T15:57:36.329358+0000","last_clean":"2024-06-29T15:57:36.329358+0000","last_became_active":"2024-06-29T15:57:35.249561+0000","last_became_peered":"2024-06-29T15:57:35.249561+0000","last_unstale":"2024-06-29T15:57:36.329358+0000","last_undegraded":"2024-06-29T15:57:36.329358+0000","last_fullsized":"2024-06-29T15:57:36.329358+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:58.917763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239963+0000","last_change":"2024-06-29T15:57:22.224310+0000","last_active":"2024-06-29T15:57:36.239963+0000","last_peered":"2024-06-29T15:57:36.239963+0000","last_clean":"2024-06-29T15:57:36.239963+0000","last_became_active":"2024-06-29T15:57:22.223959+0000","last_became_peered":"2024-06-29T15:57:22.223959+0000","last_unstale":"2024-06-29T15:57:36.239963+0000","last_undegraded":"2024-06-29T15:57:36.239963+0000","last_fullsized":"2024-06-29T15:57:36.239963+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:13.811309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332185+0000","last_change":"2024-06-29T15:57:35.258233+0000","last_active":"2024-06-29T15:57:36.332185+0000","last_peered":"2024-06-29T15:57:36.332185+0000","last_clean":"2024-06-29T15:57:36.332185+0000","last_became_active":"2024-06-29T15:57:35.256283+0000","last_became_peered":"2024-06-29T15:57:35.256283+0000","last_unstale":"2024-06-29T15:57:36.332185+0000","last_undegraded":"2024-06-29T15:57:36.332185+0000","last_fullsized":"2024-06-29T15:57:36.332185+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:49.167991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239662+0000","last_change":"2024-06-29T15:57:22.215484+0000","last_active":"2024-06-29T15:57:36.239662+0000","last_peered":"2024-06-29T15:57:36.239662+0000","last_clean":"2024-06-29T15:57:36.239662+0000","last_became_active":"2024-06-29T15:57:22.215179+0000","last_became_peered":"2024-06-29T15:57:22.215179+0000","last_unstale":"2024-06-29T15:57:36.239662+0000","last_undegraded":"2024-06-29T15:57:36.239662+0000","last_fullsized":"2024-06-29T15:57:36.239662+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:29.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239331+0000","last_change":"2024-06-29T15:57:35.258740+0000","last_active":"2024-06-29T15:57:36.239331+0000","last_peered":"2024-06-29T15:57:36.239331+0000","last_clean":"2024-06-29T15:57:36.239331+0000","last_became_active":"2024-06-29T15:57:35.258400+0000","last_became_peered":"2024-06-29T15:57:35.258400+0000","last_unstale":"2024-06-29T15:57:36.239331+0000","last_undegraded":"2024-06-29T15:57:36.239331+0000","last_fullsized":"2024-06-29T15:57:36.239331+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:33.378531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238877+0000","last_change":"2024-06-29T15:57:22.237289+0000","last_active":"2024-06-29T15:57:36.238877+0000","last_peered":"2024-06-29T15:57:36.238877+0000","last_clean":"2024-06-29T15:57:36.238877+0000","last_became_active":"2024-06-29T15:57:22.236267+0000","last_became_peered":"2024-06-29T15:57:22.236267+0000","last_unstale":"2024-06-29T15:57:36.238877+0000","last_undegraded":"2024-06-29T15:57:36.238877+0000","last_fullsized":"2024-06-29T15:57:36.238877+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:53:11.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240048+0000","last_change":"2024-06-29T15:57:35.243220+0000","last_active":"2024-06-29T15:57:36.240048+0000","last_peered":"2024-06-29T15:57:36.240048+0000","last_clean":"2024-06-29T15:57:36.240048+0000","last_became_active":"2024-06-29T15:57:35.242914+0000","last_became_peered":"2024-06-29T15:57:35.242914+0000","last_unstale":"2024-06-29T15:57:36.240048+0000","last_undegraded":"2024-06-29T15:57:36.240048+0000","last_fullsized":"2024-06-29T15:57:36.240048+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:16.068033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330080+0000","last_change":"2024-06-29T15:57:22.239205+0000","last_active":"2024-06-29T15:57:36.330080+0000","last_peered":"2024-06-29T15:57:36.330080+0000","last_clean":"2024-06-29T15:57:36.330080+0000","last_became_active":"2024-06-29T15:57:22.238830+0000","last_became_peered":"2024-06-29T15:57:22.238830+0000","last_unstale":"2024-06-29T15:57:36.330080+0000","last_undegraded":"2024-06-29T15:57:36.330080+0000","last_fullsized":"2024-06-29T15:57:36.330080+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.228440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312824+0000","last_change":"2024-06-29T15:57:35.256144+0000","last_active":"2024-06-29T15:57:36.312824+0000","last_peered":"2024-06-29T15:57:36.312824+0000","last_clean":"2024-06-29T15:57:36.312824+0000","last_became_active":"2024-06-29T15:57:35.255541+0000","last_became_peered":"2024-06-29T15:57:35.255541+0000","last_unstale":"2024-06-29T15:57:36.312824+0000","last_undegraded":"2024-06-29T15:57:36.312824+0000","last_fullsized":"2024-06-29T15:57:36.312824+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:23:28.979863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330285+0000","last_change":"2024-06-29T15:57:22.235020+0000","last_active":"2024-06-29T15:57:36.330285+0000","last_peered":"2024-06-29T15:57:36.330285+0000","last_clean":"2024-06-29T15:57:36.330285+0000","last_became_active":"2024-06-29T15:57:22.234417+0000","last_became_peered":"2024-06-29T15:57:22.234417+0000","last_unstale":"2024-06-29T15:57:36.330285+0000","last_undegraded":"2024-06-29T15:57:36.330285+0000","last_fullsized":"2024-06-29T15:57:36.330285+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:06.264167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237895+0000","last_change":"2024-06-29T15:57:35.249930+0000","last_active":"2024-06-29T15:57:36.237895+0000","last_peered":"2024-06-29T15:57:36.237895+0000","last_clean":"2024-06-29T15:57:36.237895+0000","last_became_active":"2024-06-29T15:57:35.249298+0000","last_became_peered":"2024-06-29T15:57:35.249298+0000","last_unstale":"2024-06-29T15:57:36.237895+0000","last_undegraded":"2024-06-29T15:57:36.237895+0000","last_fullsized":"2024-06-29T15:57:36.237895+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.903736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.235930+0000","last_change":"2024-06-29T15:57:22.238113+0000","last_active":"2024-06-29T15:57:36.235930+0000","last_peered":"2024-06-29T15:57:36.235930+0000","last_clean":"2024-06-29T15:57:36.235930+0000","last_became_active":"2024-06-29T15:57:22.237726+0000","last_became_peered":"2024-06-29T15:57:22.237726+0000","last_unstale":"2024-06-29T15:57:36.235930+0000","last_undegraded":"2024-06-29T15:57:36.235930+0000","last_fullsized":"2024-06-29T15:57:36.235930+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:08:48.416318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239184+0000","last_change":"2024-06-29T15:57:35.247761+0000","last_active":"2024-06-29T15:57:36.239184+0000","last_peered":"2024-06-29T15:57:36.239184+0000","last_clean":"2024-06-29T15:57:36.239184+0000","last_became_active":"2024-06-29T15:57:35.247325+0000","last_became_peered":"2024-06-29T15:57:35.247325+0000","last_unstale":"2024-06-29T15:57:36.239184+0000","last_undegraded":"2024-06-29T15:57:36.239184+0000","last_fullsized":"2024-06-29T15:57:36.239184+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:55:34.536510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329410+0000","last_change":"2024-06-29T15:57:22.220754+0000","last_active":"2024-06-29T15:57:36.329410+0000","last_peered":"2024-06-29T15:57:36.329410+0000","last_clean":"2024-06-29T15:57:36.329410+0000","last_became_active":"2024-06-29T15:57:22.220438+0000","last_became_peered":"2024-06-29T15:57:22.220438+0000","last_unstale":"2024-06-29T15:57:36.329410+0000","last_undegraded":"2024-06-29T15:57:36.329410+0000","last_fullsized":"2024-06-29T15:57:36.329410+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:26.314927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237379+0000","last_change":"2024-06-29T15:57:35.241980+0000","last_active":"2024-06-29T15:57:36.237379+0000","last_peered":"2024-06-29T15:57:36.237379+0000","last_clean":"2024-06-29T15:57:36.237379+0000","last_became_active":"2024-06-29T15:57:35.241607+0000","last_became_peered":"2024-06-29T15:57:35.241607+0000","last_unstale":"2024-06-29T15:57:36.237379+0000","last_undegraded":"2024-06-29T15:57:36.237379+0000","last_fullsized":"2024-06-29T15:57:36.237379+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:51.022053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238091+0000","last_change":"2024-06-29T15:57:22.225596+0000","last_active":"2024-06-29T15:57:36.238091+0000","last_peered":"2024-06-29T15:57:36.238091+0000","last_clean":"2024-06-29T15:57:36.238091+0000","last_became_active":"2024-06-29T15:57:22.225154+0000","last_became_peered":"2024-06-29T15:57:22.225154+0000","last_unstale":"2024-06-29T15:57:36.238091+0000","last_undegraded":"2024-06-29T15:57:36.238091+0000","last_fullsized":"2024-06-29T15:57:36.238091+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:01.272853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236100+0000","last_change":"2024-06-29T15:57:35.246784+0000","last_active":"2024-06-29T15:57:36.236100+0000","last_peered":"2024-06-29T15:57:36.236100+0000","last_clean":"2024-06-29T15:57:36.236100+0000","last_became_active":"2024-06-29T15:57:35.246467+0000","last_became_peered":"2024-06-29T15:57:35.246467+0000","last_unstale":"2024-06-29T15:57:36.236100+0000","last_undegraded":"2024-06-29T15:57:36.236100+0000","last_fullsized":"2024-06-29T15:57:36.236100+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:46.600461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311897+0000","last_change":"2024-06-29T15:57:22.237433+0000","last_active":"2024-06-29T15:57:36.311897+0000","last_peered":"2024-06-29T15:57:36.311897+0000","last_clean":"2024-06-29T15:57:36.311897+0000","last_became_active":"2024-06-29T15:57:22.237070+0000","last_became_peered":"2024-06-29T15:57:22.237070+0000","last_unstale":"2024-06-29T15:57:36.311897+0000","last_undegraded":"2024-06-29T15:57:36.311897+0000","last_fullsized":"2024-06-29T15:57:36.311897+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:25.658901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332066+0000","last_change":"2024-06-29T15:57:35.251241+0000","last_active":"2024-06-29T15:57:36.332066+0000","last_peered":"2024-06-29T15:57:36.332066+0000","last_clean":"2024-06-29T15:57:36.332066+0000","last_became_active":"2024-06-29T15:57:35.248566+0000","last_became_peered":"2024-06-29T15:57:35.248566+0000","last_unstale":"2024-06-29T15:57:36.332066+0000","last_undegraded":"2024-06-29T15:57:36.332066+0000","last_fullsized":"2024-06-29T15:57:36.332066+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:52.934381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330550+0000","last_change":"2024-06-29T15:57:22.231838+0000","last_active":"2024-06-29T15:57:36.330550+0000","last_peered":"2024-06-29T15:57:36.330550+0000","last_clean":"2024-06-29T15:57:36.330550+0000","last_became_active":"2024-06-29T15:57:22.231563+0000","last_became_peered":"2024-06-29T15:57:22.231563+0000","last_unstale":"2024-06-29T15:57:36.330550+0000","last_undegraded":"2024-06-29T15:57:36.330550+0000","last_fullsized":"2024-06-29T15:57:36.330550+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:23.251139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330914+0000","last_change":"2024-06-29T15:57:35.251166+0000","last_active":"2024-06-29T15:57:36.330914+0000","last_peered":"2024-06-29T15:57:36.330914+0000","last_clean":"2024-06-29T15:57:36.330914+0000","last_became_active":"2024-06-29T15:57:35.249802+0000","last_became_peered":"2024-06-29T15:57:35.249802+0000","last_unstale":"2024-06-29T15:57:36.330914+0000","last_undegraded":"2024-06-29T15:57:36.330914+0000","last_fullsized":"2024-06-29T15:57:36.330914+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:26.856925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237097+0000","last_change":"2024-06-29T15:57:22.237225+0000","last_active":"2024-06-29T15:57:36.237097+0000","last_peered":"2024-06-29T15:57:36.237097+0000","last_clean":"2024-06-29T15:57:36.237097+0000","last_became_active":"2024-06-29T15:57:22.236621+0000","last_became_peered":"2024-06-29T15:57:22.236621+0000","last_unstale":"2024-06-29T15:57:36.237097+0000","last_undegraded":"2024-06-29T15:57:36.237097+0000","last_fullsized":"2024-06-29T15:57:36.237097+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:26.582705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238326+0000","last_change":"2024-06-29T15:57:35.252791+0000","last_active":"2024-06-29T15:57:36.238326+0000","last_peered":"2024-06-29T15:57:36.238326+0000","last_clean":"2024-06-29T15:57:36.238326+0000","last_became_active":"2024-06-29T15:57:35.252002+0000","last_became_peered":"2024-06-29T15:57:35.252002+0000","last_unstale":"2024-06-29T15:57:36.238326+0000","last_undegraded":"2024-06-29T15:57:36.238326+0000","last_fullsized":"2024-06-29T15:57:36.238326+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:32.593283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238829+0000","last_change":"2024-06-29T15:57:22.237316+0000","last_active":"2024-06-29T15:57:36.238829+0000","last_peered":"2024-06-29T15:57:36.238829+0000","last_clean":"2024-06-29T15:57:36.238829+0000","last_became_active":"2024-06-29T15:57:22.236756+0000","last_became_peered":"2024-06-29T15:57:22.236756+0000","last_unstale":"2024-06-29T15:57:36.238829+0000","last_undegraded":"2024-06-29T15:57:36.238829+0000","last_fullsized":"2024-06-29T15:57:36.238829+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.141583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329947+0000","last_change":"2024-06-29T15:57:35.254538+0000","last_active":"2024-06-29T15:57:36.329947+0000","last_peered":"2024-06-29T15:57:36.329947+0000","last_clean":"2024-06-29T15:57:36.329947+0000","last_became_active":"2024-06-29T15:57:35.254193+0000","last_became_peered":"2024-06-29T15:57:35.254193+0000","last_unstale":"2024-06-29T15:57:36.329947+0000","last_undegraded":"2024-06-29T15:57:36.329947+0000","last_fullsized":"2024-06-29T15:57:36.329947+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:23.012081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330101+0000","last_change":"2024-06-29T15:57:22.235210+0000","last_active":"2024-06-29T15:57:36.330101+0000","last_peered":"2024-06-29T15:57:36.330101+0000","last_clean":"2024-06-29T15:57:36.330101+0000","last_became_active":"2024-06-29T15:57:22.234573+0000","last_became_peered":"2024-06-29T15:57:22.234573+0000","last_unstale":"2024-06-29T15:57:36.330101+0000","last_undegraded":"2024-06-29T15:57:36.330101+0000","last_fullsized":"2024-06-29T15:57:36.330101+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:48.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239771+0000","last_change":"2024-06-29T15:57:35.252893+0000","last_active":"2024-06-29T15:57:36.239771+0000","last_peered":"2024-06-29T15:57:36.239771+0000","last_clean":"2024-06-29T15:57:36.239771+0000","last_became_active":"2024-06-29T15:57:35.252527+0000","last_became_peered":"2024-06-29T15:57:35.252527+0000","last_unstale":"2024-06-29T15:57:36.239771+0000","last_undegraded":"2024-06-29T15:57:36.239771+0000","last_fullsized":"2024-06-29T15:57:36.239771+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:02.911105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239077+0000","last_change":"2024-06-29T15:57:22.233275+0000","last_active":"2024-06-29T15:57:36.239077+0000","last_peered":"2024-06-29T15:57:36.239077+0000","last_clean":"2024-06-29T15:57:36.239077+0000","last_became_active":"2024-06-29T15:57:22.230801+0000","last_became_peered":"2024-06-29T15:57:22.230801+0000","last_unstale":"2024-06-29T15:57:36.239077+0000","last_undegraded":"2024-06-29T15:57:36.239077+0000","last_fullsized":"2024-06-29T15:57:36.239077+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:24.134069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311790+0000","last_change":"2024-06-29T15:57:35.257258+0000","last_active":"2024-06-29T15:57:36.311790+0000","last_peered":"2024-06-29T15:57:36.311790+0000","last_clean":"2024-06-29T15:57:36.311790+0000","last_became_active":"2024-06-29T15:57:35.256939+0000","last_became_peered":"2024-06-29T15:57:35.256939+0000","last_unstale":"2024-06-29T15:57:36.311790+0000","last_undegraded":"2024-06-29T15:57:36.311790+0000","last_fullsized":"2024-06-29T15:57:36.311790+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:40:53.758735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329683+0000","last_change":"2024-06-29T15:57:22.223324+0000","last_active":"2024-06-29T15:57:36.329683+0000","last_peered":"2024-06-29T15:57:36.329683+0000","last_clean":"2024-06-29T15:57:36.329683+0000","last_became_active":"2024-06-29T15:57:22.222958+0000","last_became_peered":"2024-06-29T15:57:22.222958+0000","last_unstale":"2024-06-29T15:57:36.329683+0000","last_undegraded":"2024-06-29T15:57:36.329683+0000","last_fullsized":"2024-06-29T15:57:36.329683+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:21.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237061+0000","last_change":"2024-06-29T15:57:35.244095+0000","last_active":"2024-06-29T15:57:36.237061+0000","last_peered":"2024-06-29T15:57:36.237061+0000","last_clean":"2024-06-29T15:57:36.237061+0000","last_became_active":"2024-06-29T15:57:35.243771+0000","last_became_peered":"2024-06-29T15:57:35.243771+0000","last_unstale":"2024-06-29T15:57:36.237061+0000","last_undegraded":"2024-06-29T15:57:36.237061+0000","last_fullsized":"2024-06-29T15:57:36.237061+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:55.022325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312052+0000","last_change":"2024-06-29T15:57:22.226496+0000","last_active":"2024-06-29T15:57:36.312052+0000","last_peered":"2024-06-29T15:57:36.312052+0000","last_clean":"2024-06-29T15:57:36.312052+0000","last_became_active":"2024-06-29T15:57:22.226191+0000","last_became_peered":"2024-06-29T15:57:22.226191+0000","last_unstale":"2024-06-29T15:57:36.312052+0000","last_undegraded":"2024-06-29T15:57:36.312052+0000","last_fullsized":"2024-06-29T15:57:36.312052+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:32.644449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330874+0000","last_change":"2024-06-29T15:57:35.258649+0000","last_active":"2024-06-29T15:57:36.330874+0000","last_peered":"2024-06-29T15:57:36.330874+0000","last_clean":"2024-06-29T15:57:36.330874+0000","last_became_active":"2024-06-29T15:57:35.258325+0000","last_became_peered":"2024-06-29T15:57:35.258325+0000","last_unstale":"2024-06-29T15:57:36.330874+0000","last_undegraded":"2024-06-29T15:57:36.330874+0000","last_fullsized":"2024-06-29T15:57:36.330874+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:52.146056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331112+0000","last_change":"2024-06-29T15:57:22.218369+0000","last_active":"2024-06-29T15:57:36.331112+0000","last_peered":"2024-06-29T15:57:36.331112+0000","last_clean":"2024-06-29T15:57:36.331112+0000","last_became_active":"2024-06-29T15:57:22.217584+0000","last_became_peered":"2024-06-29T15:57:22.217584+0000","last_unstale":"2024-06-29T15:57:36.331112+0000","last_undegraded":"2024-06-29T15:57:36.331112+0000","last_fullsized":"2024-06-29T15:57:36.331112+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:59.760761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330020+0000","last_change":"2024-06-29T15:57:35.250646+0000","last_active":"2024-06-29T15:57:36.330020+0000","last_peered":"2024-06-29T15:57:36.330020+0000","last_clean":"2024-06-29T15:57:36.330020+0000","last_became_active":"2024-06-29T15:57:35.250307+0000","last_became_peered":"2024-06-29T15:57:35.250307+0000","last_unstale":"2024-06-29T15:57:36.330020+0000","last_undegraded":"2024-06-29T15:57:36.330020+0000","last_fullsized":"2024-06-29T15:57:36.330020+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:06.391987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312266+0000","last_change":"2024-06-29T15:57:22.236022+0000","last_active":"2024-06-29T15:57:36.312266+0000","last_peered":"2024-06-29T15:57:36.312266+0000","last_clean":"2024-06-29T15:57:36.312266+0000","last_became_active":"2024-06-29T15:57:22.235712+0000","last_became_peered":"2024-06-29T15:57:22.235712+0000","last_unstale":"2024-06-29T15:57:36.312266+0000","last_undegraded":"2024-06-29T15:57:36.312266+0000","last_fullsized":"2024-06-29T15:57:36.312266+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:57.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312823+0000","last_change":"2024-06-29T15:57:35.256845+0000","last_active":"2024-06-29T15:57:36.312823+0000","last_peered":"2024-06-29T15:57:36.312823+0000","last_clean":"2024-06-29T15:57:36.312823+0000","last_became_active":"2024-06-29T15:57:35.256516+0000","last_became_peered":"2024-06-29T15:57:35.256516+0000","last_unstale":"2024-06-29T15:57:36.312823+0000","last_undegraded":"2024-06-29T15:57:36.312823+0000","last_fullsized":"2024-06-29T15:57:36.312823+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.581911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237077+0000","last_change":"2024-06-29T15:57:22.231073+0000","last_active":"2024-06-29T15:57:36.237077+0000","last_peered":"2024-06-29T15:57:36.237077+0000","last_clean":"2024-06-29T15:57:36.237077+0000","last_became_active":"2024-06-29T15:57:22.230749+0000","last_became_peered":"2024-06-29T15:57:22.230749+0000","last_unstale":"2024-06-29T15:57:36.237077+0000","last_undegraded":"2024-06-29T15:57:36.237077+0000","last_fullsized":"2024-06-29T15:57:36.237077+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:23.168668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329839+0000","last_change":"2024-06-29T15:57:35.252089+0000","last_active":"2024-06-29T15:57:36.329839+0000","last_peered":"2024-06-29T15:57:36.329839+0000","last_clean":"2024-06-29T15:57:36.329839+0000","last_became_active":"2024-06-29T15:57:35.251775+0000","last_became_peered":"2024-06-29T15:57:35.251775+0000","last_unstale":"2024-06-29T15:57:36.329839+0000","last_undegraded":"2024-06-29T15:57:36.329839+0000","last_fullsized":"2024-06-29T15:57:36.329839+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:26.905348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332427+0000","last_change":"2024-06-29T15:57:22.231733+0000","last_active":"2024-06-29T15:57:36.332427+0000","last_peered":"2024-06-29T15:57:36.332427+0000","last_clean":"2024-06-29T15:57:36.332427+0000","last_became_active":"2024-06-29T15:57:22.231387+0000","last_became_peered":"2024-06-29T15:57:22.231387+0000","last_unstale":"2024-06-29T15:57:36.332427+0000","last_undegraded":"2024-06-29T15:57:36.332427+0000","last_fullsized":"2024-06-29T15:57:36.332427+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:41.892974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329977+0000","last_change":"2024-06-29T15:57:35.255683+0000","last_active":"2024-06-29T15:57:36.329977+0000","last_peered":"2024-06-29T15:57:36.329977+0000","last_clean":"2024-06-29T15:57:36.329977+0000","last_became_active":"2024-06-29T15:57:35.255328+0000","last_became_peered":"2024-06-29T15:57:35.255328+0000","last_unstale":"2024-06-29T15:57:36.329977+0000","last_undegraded":"2024-06-29T15:57:36.329977+0000","last_fullsized":"2024-06-29T15:57:36.329977+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:35.404663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239593+0000","last_change":"2024-06-29T15:57:22.225592+0000","last_active":"2024-06-29T15:57:36.239593+0000","last_peered":"2024-06-29T15:57:36.239593+0000","last_clean":"2024-06-29T15:57:36.239593+0000","last_became_active":"2024-06-29T15:57:22.225309+0000","last_became_peered":"2024-06-29T15:57:22.225309+0000","last_unstale":"2024-06-29T15:57:36.239593+0000","last_undegraded":"2024-06-29T15:57:36.239593+0000","last_fullsized":"2024-06-29T15:57:36.239593+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:07.857349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543290,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52596,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319244,"statfs":{"total":96636764160,"available":96582905856,"internally_reserved":0,"allocated":3448832,"data_stored":2054220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543291,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42312,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329528,"statfs":{"total":96636764160,"available":96593436672,"internally_reserved":0,"allocated":2871296,"data_stored":1465430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543290,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47864,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323976,"statfs":{"total":96636764160,"available":96587751424,"internally_reserved":0,"allocated":2854912,"data_stored":1461993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543289,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52004,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319836,"statfs":{"total":96636764160,"available":96583512064,"internally_reserved":0,"allocated":2838528,"data_stored":1460927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510587,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47856,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323984,"statfs":{"total":96636764160,"available":96587759616,"internally_reserved":0,"allocated":2846720,"data_stored":1460403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47844,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323996,"statfs":{"total":96636764160,"available":96587771904,"internally_reserved":0,"allocated":2842624,"data_stored":1460961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510586,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47832,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324008,"statfs":{"total":96636764160,"available":96587784192,"internally_reserved":0,"allocated":2830336,"data_stored":1460255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510588,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42852,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328988,"statfs":{"total":96636764160,"available":96592883712,"internally_reserved":0,"allocated":3428352,"data_stored":2050713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:40.167 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-29T15:57:40.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.316+0000 7f64039f0640 1 -- 172.21.15.59:0/3463740015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc15a400 msgr2=0x7f63fc0cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.316+0000 7f64039f0640 1 --2- 172.21.15.59:0/3463740015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc15a400 0x7f63fc0cf120 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009830 tx=0x7f63ec02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.316+0000 7f64039f0640 1 -- 172.21.15.59:0/3463740015 shutdown_connections 2024-06-29T15:57:40.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.316+0000 7f64039f0640 1 --2- 172.21.15.59:0/3463740015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc15a400 0x7f63fc0cf120 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.316+0000 7f64039f0640 1 -- 172.21.15.59:0/3463740015 >> 172.21.15.59:0/3463740015 conn(0x7f63fc0b24c0 msgr2=0x7f63fc0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:40.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.317+0000 7f64039f0640 1 -- 172.21.15.59:0/3463740015 shutdown_connections 2024-06-29T15:57:40.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.317+0000 7f64039f0640 1 -- 172.21.15.59:0/3463740015 wait complete. 2024-06-29T15:57:40.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.318+0000 7f64039f0640 1 Processor -- start 2024-06-29T15:57:40.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.318+0000 7f64039f0640 1 -- start start 2024-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f64039f0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc06c1e0 0x7f63fc1533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f64039f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63fc073f30 0x7f63fc151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f64039f0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63fc15a400 0x7f63fc152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f64039f0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc073d20 con 0x7f63fc06c1e0 2024-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f64039f0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc0b43a0 con 0x7f63fc15a400 2024-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f64039f0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f63fc06c5f0 con 0x7f63fc073f30 2024-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc06c1e0 0x7f63fc1533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc06c1e0 0x7f63fc1533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44790/0 (socket says 172.21.15.59:44790) 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 -- 172.21.15.59:0/1551010799 learned_addr learned my addr 172.21.15.59:0/1551010799 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f6400f8b640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63fc073f30 0x7f63fc151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f6401f8d640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63fc15a400 0x7f63fc152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 -- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63fc073f30 msgr2=0x7f63fc151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63fc073f30 0x7f63fc151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 -- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63fc15a400 msgr2=0x7f63fc152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63fc15a400 0x7f63fc152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 -- 172.21.15.59:0/1551010799 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63fc153930 con 0x7f63fc06c1e0 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f6401f8d640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63fc15a400 0x7f63fc152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f6400f8b640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63fc073f30 0x7f63fc151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f640178c640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc06c1e0 0x7f63fc1533f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f63f800c400 tx=0x7f63f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.320+0000 7f63f1ffb640 1 -- 172.21.15.59:0/1551010799 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8017070 con 0x7f63fc06c1e0 2024-06-29T15:57:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.321+0000 7f63f1ffb640 1 -- 172.21.15.59:0/1551010799 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63f80053c0 con 0x7f63fc06c1e0 2024-06-29T15:57:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.321+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63fc15aa90 con 0x7f63fc06c1e0 2024-06-29T15:57:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.321+0000 7f63f1ffb640 1 -- 172.21.15.59:0/1551010799 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8003830 con 0x7f63fc06c1e0 2024-06-29T15:57:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.321+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63fc13c230 con 0x7f63fc06c1e0 2024-06-29T15:57:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.322+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63fc0cf120 con 0x7f63fc06c1e0 2024-06-29T15:57:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.323+0000 7f63f1ffb640 1 -- 172.21.15.59:0/1551010799 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f63f801a070 con 0x7f63fc06c1e0 2024-06-29T15:57:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.323+0000 7f63f1ffb640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f63e0076420 0x7f63e0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.323+0000 7f63f1ffb640 1 -- 172.21.15.59:0/1551010799 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f63f80971e0 con 0x7f63fc06c1e0 2024-06-29T15:57:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.323+0000 7f6400f8b640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f63e0076420 0x7f63e0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.324+0000 7f6400f8b640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f63e0076420 0x7f63e0078930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f63ec034000 tx=0x7f63ec004730 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.330+0000 7f63f1ffb640 1 -- 172.21.15.59:0/1551010799 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f63fc153930 con 0x7f63fc06c1e0 2024-06-29T15:57:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.601+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f63fc155c10 con 0x7f63fc06c1e0 2024-06-29T15:57:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.602+0000 7f63f1ffb640 1 -- 172.21.15.59:0/1551010799 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1168 (secure 0 0 0) 0x7f63f8060510 con 0x7f63fc06c1e0 2024-06-29T15:57:40.603 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:40.603 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5771,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":226,"num_osds":8,"num_up_osds":8,"osd_up_since":1719670901,"num_in_osds":8,"osd_in_since":1719670892,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":193}],"num_pgs":193,"num_pools":5,"num_objects":46,"data_bytes":596431,"bytes_used":390307840,"bytes_avail":772703805440,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":121,"btime":"2024-06-29T15:57:38:315315+0000","id":13,"up":1,"in":1,"max":1,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"b","status":"up:active","gid":10747},{"filesystem_id":13,"rank":0,"name":"a","status":"up:active","gid":11487}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-06-29T15:56:51.851811+0000","services":{}},"progress_events":{}} 2024-06-29T15:57:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f63e0076420 msgr2=0x7f63e0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f63e0076420 0x7f63e0078930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f63ec034000 tx=0x7f63ec004730 comp rx=0 tx=0).stop 2024-06-29T15:57:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc06c1e0 msgr2=0x7f63fc1533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc06c1e0 0x7f63fc1533f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f63f800c400 tx=0x7f63f800c8c0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 shutdown_connections 2024-06-29T15:57:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f63e0076420 0x7f63e0078930 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63fc15a400 0x7f63fc152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63fc073f30 0x7f63fc151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 --2- 172.21.15.59:0/1551010799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63fc06c1e0 0x7f63fc1533f0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 >> 172.21.15.59:0/1551010799 conn(0x7f63fc0b24c0 msgr2=0x7f63fc0ce710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 shutdown_connections 2024-06-29T15:57:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.604+0000 7f64039f0640 1 -- 172.21.15.59:0/1551010799 wait complete. 2024-06-29T15:57:40.616 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-06-29T15:57:40.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:40.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.733+0000 7faa52dbd640 1 -- 172.21.15.59:0/2947701676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3b50 msgr2=0x7faa4c0b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.733+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2947701676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3b50 0x7faa4c0b5880 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7faa44009830 tx=0x7faa4402a530 comp rx=0 tx=0).stop 2024-06-29T15:57:40.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.733+0000 7faa52dbd640 1 -- 172.21.15.59:0/2947701676 shutdown_connections 2024-06-29T15:57:40.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.733+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2947701676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3b50 0x7faa4c0b5880 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.733+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2947701676 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c1458a0 0x7faa4c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.733+0000 7faa52dbd640 1 -- 172.21.15.59:0/2947701676 >> 172.21.15.59:0/2947701676 conn(0x7faa4c06c950 msgr2=0x7faa4c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:40.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.733+0000 7faa52dbd640 1 -- 172.21.15.59:0/2947701676 shutdown_connections 2024-06-29T15:57:40.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.733+0000 7faa52dbd640 1 -- 172.21.15.59:0/2947701676 wait complete. 2024-06-29T15:57:40.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.735+0000 7faa52dbd640 1 Processor -- start 2024-06-29T15:57:40.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.735+0000 7faa52dbd640 1 -- start start 2024-06-29T15:57:40.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa52dbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3740 0x7faa4c0c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:40.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa52dbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c0b3b50 0x7faa4c1f0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa52dbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c0c7ac0 0x7faa4c0c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa52dbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0b6150 con 0x7faa4c0b3740 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa52dbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0b5fd0 con 0x7faa4c0c7ac0 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa52dbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c0b62d0 con 0x7faa4c0b3b50 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa50b59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3740 0x7faa4c0c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa50b59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3740 0x7faa4c0c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44808/0 (socket says 172.21.15.59:44808) 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.736+0000 7faa50b59640 1 -- 172.21.15.59:0/2876730855 learned_addr learned my addr 172.21.15.59:0/2876730855 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa4bfff640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c0b3b50 0x7faa4c1f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa5135a640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c0c7ac0 0x7faa4c0c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa50b59640 1 -- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c0b3b50 msgr2=0x7faa4c1f0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa50b59640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c0b3b50 0x7faa4c1f0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa50b59640 1 -- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c0c7ac0 msgr2=0x7faa4c0c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa50b59640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c0c7ac0 0x7faa4c0c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa50b59640 1 -- 172.21.15.59:0/2876730855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c0c87f0 con 0x7faa4c0b3740 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa4bfff640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c0b3b50 0x7faa4c1f0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa50b59640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3740 0x7faa4c0c9330 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00d770 tx=0x7faa3c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa497fa640 1 -- 172.21.15.59:0/2876730855 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c017070 con 0x7faa4c0b3740 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa497fa640 1 -- 172.21.15.59:0/2876730855 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa3c009d20 con 0x7faa4c0b3740 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c0cb150 con 0x7faa4c0b3740 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa497fa640 1 -- 172.21.15.59:0/2876730855 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c0106c0 con 0x7faa4c0b3740 2024-06-29T15:57:40.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.737+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c0c9900 con 0x7faa4c0b3740 2024-06-29T15:57:40.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.738+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c0c9b60 con 0x7faa4c0b3740 2024-06-29T15:57:40.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.739+0000 7faa497fa640 1 -- 172.21.15.59:0/2876730855 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7faa3c01a070 con 0x7faa4c0b3740 2024-06-29T15:57:40.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.740+0000 7faa497fa640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7faa2c076420 0x7faa2c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:40.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.740+0000 7faa497fa640 1 -- 172.21.15.59:0/2876730855 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7faa3c097f50 con 0x7faa4c0b3740 2024-06-29T15:57:40.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.740+0000 7faa4bfff640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7faa2c076420 0x7faa2c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:40.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.740+0000 7faa4bfff640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7faa2c076420 0x7faa2c078930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faa4c15eeb0 tx=0x7faa34009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:40.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.744+0000 7faa497fa640 1 -- 172.21.15.59:0/2876730855 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faa3c061170 con 0x7faa4c0b3740 2024-06-29T15:57:40.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.931+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa4c1f0540 con 0x7faa2c076420 2024-06-29T15:57:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.940+0000 7faa497fa640 1 -- 172.21.15.59:0/2876730855 <== mgr.9206 v2:172.21.15.153:6832/2780049427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491475 (secure 0 0 0) 0x7faa4c1f0540 con 0x7faa2c076420 2024-06-29T15:57:40.945 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:40.948 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:57:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7faa2c076420 msgr2=0x7faa2c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7faa2c076420 0x7faa2c078930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faa4c15eeb0 tx=0x7faa34009040 comp rx=0 tx=0).stop 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3740 msgr2=0x7faa4c0c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3740 0x7faa4c0c9330 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00d770 tx=0x7faa3c00dc30 comp rx=0 tx=0).stop 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 shutdown_connections 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7faa2c076420 0x7faa2c078930 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa4c0c7ac0 0x7faa4c0c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faa4c0b3b50 0x7faa4c1f0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 --2- 172.21.15.59:0/2876730855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa4c0b3740 0x7faa4c0c9330 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 >> 172.21.15.59:0/2876730855 conn(0x7faa4c06c950 msgr2=0x7faa4c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 shutdown_connections 2024-06-29T15:57:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:40.944+0000 7faa52dbd640 1 -- 172.21.15.59:0/2876730855 wait complete. 2024-06-29T15:57:40.956 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-06-29T15:57:39.769670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":469,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":356,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381160,"kb_used_data":23400,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593560,"statfs":{"total":773094113280,"available":772703805440,"internally_reserved":0,"allocated":23961600,"data_stored":12874902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":366202418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001764"},"pg_stats":[{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331581+0000","last_change":"2024-06-29T15:57:35.257416+0000","last_active":"2024-06-29T15:57:36.331581+0000","last_peered":"2024-06-29T15:57:36.331581+0000","last_clean":"2024-06-29T15:57:36.331581+0000","last_became_active":"2024-06-29T15:57:35.256867+0000","last_became_peered":"2024-06-29T15:57:35.256867+0000","last_unstale":"2024-06-29T15:57:36.331581+0000","last_undegraded":"2024-06-29T15:57:36.331581+0000","last_fullsized":"2024-06-29T15:57:36.331581+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:30:33.761276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"210'233","reported_seq":1109,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311738+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:57:36.311738+0000","last_peered":"2024-06-29T15:57:36.311738+0000","last_clean":"2024-06-29T15:57:36.311738+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:57:36.311738+0000","last_undegraded":"2024-06-29T15:57:36.311738+0000","last_fullsized":"2024-06-29T15:57:36.311738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331626+0000","last_change":"2024-06-29T15:57:21.209163+0000","last_active":"2024-06-29T15:57:36.331626+0000","last_peered":"2024-06-29T15:57:36.331626+0000","last_clean":"2024-06-29T15:57:36.331626+0000","last_became_active":"2024-06-29T15:57:21.208825+0000","last_became_peered":"2024-06-29T15:57:21.208825+0000","last_unstale":"2024-06-29T15:57:36.331626+0000","last_undegraded":"2024-06-29T15:57:36.331626+0000","last_fullsized":"2024-06-29T15:57:36.331626+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:13.641268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237943+0000","last_change":"2024-06-29T15:57:22.230794+0000","last_active":"2024-06-29T15:57:36.237943+0000","last_peered":"2024-06-29T15:57:36.237943+0000","last_clean":"2024-06-29T15:57:36.237943+0000","last_became_active":"2024-06-29T15:57:22.230474+0000","last_became_peered":"2024-06-29T15:57:22.230474+0000","last_unstale":"2024-06-29T15:57:36.237943+0000","last_undegraded":"2024-06-29T15:57:36.237943+0000","last_fullsized":"2024-06-29T15:57:36.237943+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:43.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312052+0000","last_change":"2024-06-29T15:57:34.312155+0000","last_active":"2024-06-29T15:57:36.312052+0000","last_peered":"2024-06-29T15:57:36.312052+0000","last_clean":"2024-06-29T15:57:36.312052+0000","last_became_active":"2024-06-29T15:57:34.311830+0000","last_became_peered":"2024-06-29T15:57:34.311830+0000","last_unstale":"2024-06-29T15:57:36.312052+0000","last_undegraded":"2024-06-29T15:57:36.312052+0000","last_fullsized":"2024-06-29T15:57:36.312052+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:58:13.468981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312504+0000","last_change":"2024-06-29T15:57:35.255842+0000","last_active":"2024-06-29T15:57:36.312504+0000","last_peered":"2024-06-29T15:57:36.312504+0000","last_clean":"2024-06-29T15:57:36.312504+0000","last_became_active":"2024-06-29T15:57:35.255195+0000","last_became_peered":"2024-06-29T15:57:35.255195+0000","last_unstale":"2024-06-29T15:57:36.312504+0000","last_undegraded":"2024-06-29T15:57:36.312504+0000","last_fullsized":"2024-06-29T15:57:36.312504+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:38.905359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331919+0000","last_change":"2024-06-29T15:57:21.215589+0000","last_active":"2024-06-29T15:57:36.331919+0000","last_peered":"2024-06-29T15:57:36.331919+0000","last_clean":"2024-06-29T15:57:36.331919+0000","last_became_active":"2024-06-29T15:57:21.213719+0000","last_became_peered":"2024-06-29T15:57:21.213719+0000","last_unstale":"2024-06-29T15:57:36.331919+0000","last_undegraded":"2024-06-29T15:57:36.331919+0000","last_fullsized":"2024-06-29T15:57:36.331919+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:32.413502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236935+0000","last_change":"2024-06-29T15:57:22.223950+0000","last_active":"2024-06-29T15:57:36.236935+0000","last_peered":"2024-06-29T15:57:36.236935+0000","last_clean":"2024-06-29T15:57:36.236935+0000","last_became_active":"2024-06-29T15:57:22.223622+0000","last_became_peered":"2024-06-29T15:57:22.223622+0000","last_unstale":"2024-06-29T15:57:36.236935+0000","last_undegraded":"2024-06-29T15:57:36.236935+0000","last_fullsized":"2024-06-29T15:57:36.236935+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.556171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"226'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236795+0000","last_change":"2024-06-29T15:57:34.307024+0000","last_active":"2024-06-29T15:57:36.236795+0000","last_peered":"2024-06-29T15:57:36.236795+0000","last_clean":"2024-06-29T15:57:36.236795+0000","last_became_active":"2024-06-29T15:57:34.306637+0000","last_became_peered":"2024-06-29T15:57:34.306637+0000","last_unstale":"2024-06-29T15:57:36.236795+0000","last_undegraded":"2024-06-29T15:57:36.236795+0000","last_fullsized":"2024-06-29T15:57:36.236795+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:30.362344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329816+0000","last_change":"2024-06-29T15:57:35.254053+0000","last_active":"2024-06-29T15:57:36.329816+0000","last_peered":"2024-06-29T15:57:36.329816+0000","last_clean":"2024-06-29T15:57:36.329816+0000","last_became_active":"2024-06-29T15:57:35.253616+0000","last_became_peered":"2024-06-29T15:57:35.253616+0000","last_unstale":"2024-06-29T15:57:36.329816+0000","last_undegraded":"2024-06-29T15:57:36.329816+0000","last_fullsized":"2024-06-29T15:57:36.329816+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:13.830138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238919+0000","last_change":"2024-06-29T15:57:21.212837+0000","last_active":"2024-06-29T15:57:36.238919+0000","last_peered":"2024-06-29T15:57:36.238919+0000","last_clean":"2024-06-29T15:57:36.238919+0000","last_became_active":"2024-06-29T15:57:21.212491+0000","last_became_peered":"2024-06-29T15:57:21.212491+0000","last_unstale":"2024-06-29T15:57:36.238919+0000","last_undegraded":"2024-06-29T15:57:36.238919+0000","last_fullsized":"2024-06-29T15:57:36.238919+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:35:53.641775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312497+0000","last_change":"2024-06-29T15:57:22.237722+0000","last_active":"2024-06-29T15:57:36.312497+0000","last_peered":"2024-06-29T15:57:36.312497+0000","last_clean":"2024-06-29T15:57:36.312497+0000","last_became_active":"2024-06-29T15:57:22.237372+0000","last_became_peered":"2024-06-29T15:57:22.237372+0000","last_unstale":"2024-06-29T15:57:36.312497+0000","last_undegraded":"2024-06-29T15:57:36.312497+0000","last_fullsized":"2024-06-29T15:57:36.312497+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:24.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239300+0000","last_change":"2024-06-29T15:57:34.309853+0000","last_active":"2024-06-29T15:57:36.239300+0000","last_peered":"2024-06-29T15:57:36.239300+0000","last_clean":"2024-06-29T15:57:36.239300+0000","last_became_active":"2024-06-29T15:57:34.309541+0000","last_became_peered":"2024-06-29T15:57:34.309541+0000","last_unstale":"2024-06-29T15:57:36.239300+0000","last_undegraded":"2024-06-29T15:57:36.239300+0000","last_fullsized":"2024-06-29T15:57:36.239300+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:17:45.264114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331328+0000","last_change":"2024-06-29T15:57:35.255572+0000","last_active":"2024-06-29T15:57:36.331328+0000","last_peered":"2024-06-29T15:57:36.331328+0000","last_clean":"2024-06-29T15:57:36.331328+0000","last_became_active":"2024-06-29T15:57:35.255216+0000","last_became_peered":"2024-06-29T15:57:35.255216+0000","last_unstale":"2024-06-29T15:57:36.331328+0000","last_undegraded":"2024-06-29T15:57:36.331328+0000","last_fullsized":"2024-06-29T15:57:36.331328+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:59.819495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331738+0000","last_change":"2024-06-29T15:57:21.210563+0000","last_active":"2024-06-29T15:57:36.331738+0000","last_peered":"2024-06-29T15:57:36.331738+0000","last_clean":"2024-06-29T15:57:36.331738+0000","last_became_active":"2024-06-29T15:57:21.210243+0000","last_became_peered":"2024-06-29T15:57:21.210243+0000","last_unstale":"2024-06-29T15:57:36.331738+0000","last_undegraded":"2024-06-29T15:57:36.331738+0000","last_fullsized":"2024-06-29T15:57:36.331738+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.900240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312769+0000","last_change":"2024-06-29T15:57:22.244396+0000","last_active":"2024-06-29T15:57:36.312769+0000","last_peered":"2024-06-29T15:57:36.312769+0000","last_clean":"2024-06-29T15:57:36.312769+0000","last_became_active":"2024-06-29T15:57:22.244044+0000","last_became_peered":"2024-06-29T15:57:22.244044+0000","last_unstale":"2024-06-29T15:57:36.312769+0000","last_undegraded":"2024-06-29T15:57:36.312769+0000","last_fullsized":"2024-06-29T15:57:36.312769+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.981276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239376+0000","last_change":"2024-06-29T15:57:34.314135+0000","last_active":"2024-06-29T15:57:36.239376+0000","last_peered":"2024-06-29T15:57:36.239376+0000","last_clean":"2024-06-29T15:57:36.239376+0000","last_became_active":"2024-06-29T15:57:34.313741+0000","last_became_peered":"2024-06-29T15:57:34.313741+0000","last_unstale":"2024-06-29T15:57:36.239376+0000","last_undegraded":"2024-06-29T15:57:36.239376+0000","last_fullsized":"2024-06-29T15:57:36.239376+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:12:29.995937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238236+0000","last_change":"2024-06-29T15:57:35.251995+0000","last_active":"2024-06-29T15:57:36.238236+0000","last_peered":"2024-06-29T15:57:36.238236+0000","last_clean":"2024-06-29T15:57:36.238236+0000","last_became_active":"2024-06-29T15:57:35.251543+0000","last_became_peered":"2024-06-29T15:57:35.251543+0000","last_unstale":"2024-06-29T15:57:36.238236+0000","last_undegraded":"2024-06-29T15:57:36.238236+0000","last_fullsized":"2024-06-29T15:57:36.238236+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:30.799799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240227+0000","last_change":"2024-06-29T15:57:21.213113+0000","last_active":"2024-06-29T15:57:36.240227+0000","last_peered":"2024-06-29T15:57:36.240227+0000","last_clean":"2024-06-29T15:57:36.240227+0000","last_became_active":"2024-06-29T15:57:21.212819+0000","last_became_peered":"2024-06-29T15:57:21.212819+0000","last_unstale":"2024-06-29T15:57:36.240227+0000","last_undegraded":"2024-06-29T15:57:36.240227+0000","last_fullsized":"2024-06-29T15:57:36.240227+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:59.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239776+0000","last_change":"2024-06-29T15:57:22.229230+0000","last_active":"2024-06-29T15:57:36.239776+0000","last_peered":"2024-06-29T15:57:36.239776+0000","last_clean":"2024-06-29T15:57:36.239776+0000","last_became_active":"2024-06-29T15:57:22.228914+0000","last_became_peered":"2024-06-29T15:57:22.228914+0000","last_unstale":"2024-06-29T15:57:36.239776+0000","last_undegraded":"2024-06-29T15:57:36.239776+0000","last_fullsized":"2024-06-29T15:57:36.239776+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:09.918042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236907+0000","last_change":"2024-06-29T15:57:34.307100+0000","last_active":"2024-06-29T15:57:36.236907+0000","last_peered":"2024-06-29T15:57:36.236907+0000","last_clean":"2024-06-29T15:57:36.236907+0000","last_became_active":"2024-06-29T15:57:34.306718+0000","last_became_peered":"2024-06-29T15:57:34.306718+0000","last_unstale":"2024-06-29T15:57:36.236907+0000","last_undegraded":"2024-06-29T15:57:36.236907+0000","last_fullsized":"2024-06-29T15:57:36.236907+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:55.770944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331213+0000","last_change":"2024-06-29T15:57:35.249106+0000","last_active":"2024-06-29T15:57:36.331213+0000","last_peered":"2024-06-29T15:57:36.331213+0000","last_clean":"2024-06-29T15:57:36.331213+0000","last_became_active":"2024-06-29T15:57:35.248469+0000","last_became_peered":"2024-06-29T15:57:35.248469+0000","last_unstale":"2024-06-29T15:57:36.331213+0000","last_undegraded":"2024-06-29T15:57:36.331213+0000","last_fullsized":"2024-06-29T15:57:36.331213+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:21:47.954549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240435+0000","last_change":"2024-06-29T15:57:21.216399+0000","last_active":"2024-06-29T15:57:36.240435+0000","last_peered":"2024-06-29T15:57:36.240435+0000","last_clean":"2024-06-29T15:57:36.240435+0000","last_became_active":"2024-06-29T15:57:21.216104+0000","last_became_peered":"2024-06-29T15:57:21.216104+0000","last_unstale":"2024-06-29T15:57:36.240435+0000","last_undegraded":"2024-06-29T15:57:36.240435+0000","last_fullsized":"2024-06-29T15:57:36.240435+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239349+0000","last_change":"2024-06-29T15:57:22.227969+0000","last_active":"2024-06-29T15:57:36.239349+0000","last_peered":"2024-06-29T15:57:36.239349+0000","last_clean":"2024-06-29T15:57:36.239349+0000","last_became_active":"2024-06-29T15:57:22.227579+0000","last_became_peered":"2024-06-29T15:57:22.227579+0000","last_unstale":"2024-06-29T15:57:36.239349+0000","last_undegraded":"2024-06-29T15:57:36.239349+0000","last_fullsized":"2024-06-29T15:57:36.239349+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329273+0000","last_change":"2024-06-29T15:57:34.306550+0000","last_active":"2024-06-29T15:57:36.329273+0000","last_peered":"2024-06-29T15:57:36.329273+0000","last_clean":"2024-06-29T15:57:36.329273+0000","last_became_active":"2024-06-29T15:57:34.306222+0000","last_became_peered":"2024-06-29T15:57:34.306222+0000","last_unstale":"2024-06-29T15:57:36.329273+0000","last_undegraded":"2024-06-29T15:57:36.329273+0000","last_fullsized":"2024-06-29T15:57:36.329273+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:57.908667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330924+0000","last_change":"2024-06-29T15:57:35.257493+0000","last_active":"2024-06-29T15:57:36.330924+0000","last_peered":"2024-06-29T15:57:36.330924+0000","last_clean":"2024-06-29T15:57:36.330924+0000","last_became_active":"2024-06-29T15:57:35.256381+0000","last_became_peered":"2024-06-29T15:57:35.256381+0000","last_unstale":"2024-06-29T15:57:36.330924+0000","last_undegraded":"2024-06-29T15:57:36.330924+0000","last_fullsized":"2024-06-29T15:57:36.330924+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:13.192616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330024+0000","last_change":"2024-06-29T15:57:21.217039+0000","last_active":"2024-06-29T15:57:36.330024+0000","last_peered":"2024-06-29T15:57:36.330024+0000","last_clean":"2024-06-29T15:57:36.330024+0000","last_became_active":"2024-06-29T15:57:21.216542+0000","last_became_peered":"2024-06-29T15:57:21.216542+0000","last_unstale":"2024-06-29T15:57:36.330024+0000","last_undegraded":"2024-06-29T15:57:36.330024+0000","last_fullsized":"2024-06-29T15:57:36.330024+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:21.729257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331424+0000","last_change":"2024-06-29T15:57:22.233919+0000","last_active":"2024-06-29T15:57:36.331424+0000","last_peered":"2024-06-29T15:57:36.331424+0000","last_clean":"2024-06-29T15:57:36.331424+0000","last_became_active":"2024-06-29T15:57:22.231028+0000","last_became_peered":"2024-06-29T15:57:22.231028+0000","last_unstale":"2024-06-29T15:57:36.331424+0000","last_undegraded":"2024-06-29T15:57:36.331424+0000","last_fullsized":"2024-06-29T15:57:36.331424+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:34:24.910914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239552+0000","last_change":"2024-06-29T15:57:34.312110+0000","last_active":"2024-06-29T15:57:36.239552+0000","last_peered":"2024-06-29T15:57:36.239552+0000","last_clean":"2024-06-29T15:57:36.239552+0000","last_became_active":"2024-06-29T15:57:34.311789+0000","last_became_peered":"2024-06-29T15:57:34.311789+0000","last_unstale":"2024-06-29T15:57:36.239552+0000","last_undegraded":"2024-06-29T15:57:36.239552+0000","last_fullsized":"2024-06-29T15:57:36.239552+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:37.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239893+0000","last_change":"2024-06-29T15:57:35.256330+0000","last_active":"2024-06-29T15:57:36.239893+0000","last_peered":"2024-06-29T15:57:36.239893+0000","last_clean":"2024-06-29T15:57:36.239893+0000","last_became_active":"2024-06-29T15:57:35.256005+0000","last_became_peered":"2024-06-29T15:57:35.256005+0000","last_unstale":"2024-06-29T15:57:36.239893+0000","last_undegraded":"2024-06-29T15:57:36.239893+0000","last_fullsized":"2024-06-29T15:57:36.239893+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:03.579383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.6","version":"222'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238934+0000","last_change":"2024-06-29T15:57:21.203822+0000","last_active":"2024-06-29T15:57:36.238934+0000","last_peered":"2024-06-29T15:57:36.238934+0000","last_clean":"2024-06-29T15:57:36.238934+0000","last_became_active":"2024-06-29T15:57:21.203517+0000","last_became_peered":"2024-06-29T15:57:21.203517+0000","last_unstale":"2024-06-29T15:57:36.238934+0000","last_undegraded":"2024-06-29T15:57:36.238934+0000","last_fullsized":"2024-06-29T15:57:36.238934+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:05.810997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236238+0000","last_change":"2024-06-29T15:57:22.223140+0000","last_active":"2024-06-29T15:57:36.236238+0000","last_peered":"2024-06-29T15:57:36.236238+0000","last_clean":"2024-06-29T15:57:36.236238+0000","last_became_active":"2024-06-29T15:57:22.222839+0000","last_became_peered":"2024-06-29T15:57:22.222839+0000","last_unstale":"2024-06-29T15:57:36.236238+0000","last_undegraded":"2024-06-29T15:57:36.236238+0000","last_fullsized":"2024-06-29T15:57:36.236238+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:17.520475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238524+0000","last_change":"2024-06-29T15:57:34.308352+0000","last_active":"2024-06-29T15:57:36.238524+0000","last_peered":"2024-06-29T15:57:36.238524+0000","last_clean":"2024-06-29T15:57:36.238524+0000","last_became_active":"2024-06-29T15:57:34.308040+0000","last_became_peered":"2024-06-29T15:57:34.308040+0000","last_unstale":"2024-06-29T15:57:36.238524+0000","last_undegraded":"2024-06-29T15:57:36.238524+0000","last_fullsized":"2024-06-29T15:57:36.238524+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:34.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329855+0000","last_change":"2024-06-29T15:57:35.251788+0000","last_active":"2024-06-29T15:57:36.329855+0000","last_peered":"2024-06-29T15:57:36.329855+0000","last_clean":"2024-06-29T15:57:36.329855+0000","last_became_active":"2024-06-29T15:57:35.251451+0000","last_became_peered":"2024-06-29T15:57:35.251451+0000","last_unstale":"2024-06-29T15:57:36.329855+0000","last_undegraded":"2024-06-29T15:57:36.329855+0000","last_fullsized":"2024-06-29T15:57:36.329855+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:14.366892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330636+0000","last_change":"2024-06-29T15:57:21.212745+0000","last_active":"2024-06-29T15:57:36.330636+0000","last_peered":"2024-06-29T15:57:36.330636+0000","last_clean":"2024-06-29T15:57:36.330636+0000","last_became_active":"2024-06-29T15:57:21.212335+0000","last_became_peered":"2024-06-29T15:57:21.212335+0000","last_unstale":"2024-06-29T15:57:36.330636+0000","last_undegraded":"2024-06-29T15:57:36.330636+0000","last_fullsized":"2024-06-29T15:57:36.330636+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:07.049637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239385+0000","last_change":"2024-06-29T15:57:22.224294+0000","last_active":"2024-06-29T15:57:36.239385+0000","last_peered":"2024-06-29T15:57:36.239385+0000","last_clean":"2024-06-29T15:57:36.239385+0000","last_became_active":"2024-06-29T15:57:22.223922+0000","last_became_peered":"2024-06-29T15:57:22.223922+0000","last_unstale":"2024-06-29T15:57:36.239385+0000","last_undegraded":"2024-06-29T15:57:36.239385+0000","last_fullsized":"2024-06-29T15:57:36.239385+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:46.725456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311940+0000","last_change":"2024-06-29T15:57:34.308042+0000","last_active":"2024-06-29T15:57:36.311940+0000","last_peered":"2024-06-29T15:57:36.311940+0000","last_clean":"2024-06-29T15:57:36.311940+0000","last_became_active":"2024-06-29T15:57:34.307706+0000","last_became_peered":"2024-06-29T15:57:34.307706+0000","last_unstale":"2024-06-29T15:57:36.311940+0000","last_undegraded":"2024-06-29T15:57:36.311940+0000","last_fullsized":"2024-06-29T15:57:36.311940+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:24.276099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312202+0000","last_change":"2024-06-29T15:57:35.254882+0000","last_active":"2024-06-29T15:57:36.312202+0000","last_peered":"2024-06-29T15:57:36.312202+0000","last_clean":"2024-06-29T15:57:36.312202+0000","last_became_active":"2024-06-29T15:57:35.254543+0000","last_became_peered":"2024-06-29T15:57:35.254543+0000","last_unstale":"2024-06-29T15:57:36.312202+0000","last_undegraded":"2024-06-29T15:57:36.312202+0000","last_fullsized":"2024-06-29T15:57:36.312202+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:03:35.376526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237536+0000","last_change":"2024-06-29T15:57:21.211203+0000","last_active":"2024-06-29T15:57:36.237536+0000","last_peered":"2024-06-29T15:57:36.237536+0000","last_clean":"2024-06-29T15:57:36.237536+0000","last_became_active":"2024-06-29T15:57:21.210898+0000","last_became_peered":"2024-06-29T15:57:21.210898+0000","last_unstale":"2024-06-29T15:57:36.237536+0000","last_undegraded":"2024-06-29T15:57:36.237536+0000","last_fullsized":"2024-06-29T15:57:36.237536+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:17.270875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240196+0000","last_change":"2024-06-29T15:57:22.213920+0000","last_active":"2024-06-29T15:57:36.240196+0000","last_peered":"2024-06-29T15:57:36.240196+0000","last_clean":"2024-06-29T15:57:36.240196+0000","last_became_active":"2024-06-29T15:57:22.213583+0000","last_became_peered":"2024-06-29T15:57:22.213583+0000","last_unstale":"2024-06-29T15:57:36.240196+0000","last_undegraded":"2024-06-29T15:57:36.240196+0000","last_fullsized":"2024-06-29T15:57:36.240196+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:31.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331622+0000","last_change":"2024-06-29T15:57:34.308946+0000","last_active":"2024-06-29T15:57:36.331622+0000","last_peered":"2024-06-29T15:57:36.331622+0000","last_clean":"2024-06-29T15:57:36.331622+0000","last_became_active":"2024-06-29T15:57:34.308411+0000","last_became_peered":"2024-06-29T15:57:34.308411+0000","last_unstale":"2024-06-29T15:57:36.331622+0000","last_undegraded":"2024-06-29T15:57:36.331622+0000","last_fullsized":"2024-06-29T15:57:36.331622+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:13.989668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330052+0000","last_change":"2024-06-29T15:57:35.259846+0000","last_active":"2024-06-29T15:57:36.330052+0000","last_peered":"2024-06-29T15:57:36.330052+0000","last_clean":"2024-06-29T15:57:36.330052+0000","last_became_active":"2024-06-29T15:57:35.259493+0000","last_became_peered":"2024-06-29T15:57:35.259493+0000","last_unstale":"2024-06-29T15:57:36.330052+0000","last_undegraded":"2024-06-29T15:57:36.330052+0000","last_fullsized":"2024-06-29T15:57:36.330052+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:15.812204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329676+0000","last_change":"2024-06-29T15:57:21.212085+0000","last_active":"2024-06-29T15:57:36.329676+0000","last_peered":"2024-06-29T15:57:36.329676+0000","last_clean":"2024-06-29T15:57:36.329676+0000","last_became_active":"2024-06-29T15:57:21.211731+0000","last_became_peered":"2024-06-29T15:57:21.211731+0000","last_unstale":"2024-06-29T15:57:36.329676+0000","last_undegraded":"2024-06-29T15:57:36.329676+0000","last_fullsized":"2024-06-29T15:57:36.329676+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:36.061054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330288+0000","last_change":"2024-06-29T15:57:22.228211+0000","last_active":"2024-06-29T15:57:36.330288+0000","last_peered":"2024-06-29T15:57:36.330288+0000","last_clean":"2024-06-29T15:57:36.330288+0000","last_became_active":"2024-06-29T15:57:22.227519+0000","last_became_peered":"2024-06-29T15:57:22.227519+0000","last_unstale":"2024-06-29T15:57:36.330288+0000","last_undegraded":"2024-06-29T15:57:36.330288+0000","last_fullsized":"2024-06-29T15:57:36.330288+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:16:35.999038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331724+0000","last_change":"2024-06-29T15:57:34.314930+0000","last_active":"2024-06-29T15:57:36.331724+0000","last_peered":"2024-06-29T15:57:36.331724+0000","last_clean":"2024-06-29T15:57:36.331724+0000","last_became_active":"2024-06-29T15:57:34.314549+0000","last_became_peered":"2024-06-29T15:57:34.314549+0000","last_unstale":"2024-06-29T15:57:36.331724+0000","last_undegraded":"2024-06-29T15:57:36.331724+0000","last_fullsized":"2024-06-29T15:57:36.331724+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:02.172151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239694+0000","last_change":"2024-06-29T15:57:35.257200+0000","last_active":"2024-06-29T15:57:36.239694+0000","last_peered":"2024-06-29T15:57:36.239694+0000","last_clean":"2024-06-29T15:57:36.239694+0000","last_became_active":"2024-06-29T15:57:35.256810+0000","last_became_peered":"2024-06-29T15:57:35.256810+0000","last_unstale":"2024-06-29T15:57:36.239694+0000","last_undegraded":"2024-06-29T15:57:36.239694+0000","last_fullsized":"2024-06-29T15:57:36.239694+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:42.431331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330636+0000","last_change":"2024-06-29T15:57:21.208592+0000","last_active":"2024-06-29T15:57:36.330636+0000","last_peered":"2024-06-29T15:57:36.330636+0000","last_clean":"2024-06-29T15:57:36.330636+0000","last_became_active":"2024-06-29T15:57:21.208156+0000","last_became_peered":"2024-06-29T15:57:21.208156+0000","last_unstale":"2024-06-29T15:57:36.330636+0000","last_undegraded":"2024-06-29T15:57:36.330636+0000","last_fullsized":"2024-06-29T15:57:36.330636+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:20.325402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312669+0000","last_change":"2024-06-29T15:57:22.238829+0000","last_active":"2024-06-29T15:57:36.312669+0000","last_peered":"2024-06-29T15:57:36.312669+0000","last_clean":"2024-06-29T15:57:36.312669+0000","last_became_active":"2024-06-29T15:57:22.238520+0000","last_became_peered":"2024-06-29T15:57:22.238520+0000","last_unstale":"2024-06-29T15:57:36.312669+0000","last_undegraded":"2024-06-29T15:57:36.312669+0000","last_fullsized":"2024-06-29T15:57:36.312669+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:22.127384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239235+0000","last_change":"2024-06-29T15:57:34.308672+0000","last_active":"2024-06-29T15:57:36.239235+0000","last_peered":"2024-06-29T15:57:36.239235+0000","last_clean":"2024-06-29T15:57:36.239235+0000","last_became_active":"2024-06-29T15:57:34.307347+0000","last_became_peered":"2024-06-29T15:57:34.307347+0000","last_unstale":"2024-06-29T15:57:36.239235+0000","last_undegraded":"2024-06-29T15:57:36.239235+0000","last_fullsized":"2024-06-29T15:57:36.239235+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:53:19.725300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332378+0000","last_change":"2024-06-29T15:57:35.248756+0000","last_active":"2024-06-29T15:57:36.332378+0000","last_peered":"2024-06-29T15:57:36.332378+0000","last_clean":"2024-06-29T15:57:36.332378+0000","last_became_active":"2024-06-29T15:57:35.248281+0000","last_became_peered":"2024-06-29T15:57:35.248281+0000","last_unstale":"2024-06-29T15:57:36.332378+0000","last_undegraded":"2024-06-29T15:57:36.332378+0000","last_fullsized":"2024-06-29T15:57:36.332378+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:50.250829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331396+0000","last_change":"2024-06-29T15:57:21.215358+0000","last_active":"2024-06-29T15:57:36.331396+0000","last_peered":"2024-06-29T15:57:36.331396+0000","last_clean":"2024-06-29T15:57:36.331396+0000","last_became_active":"2024-06-29T15:57:21.213979+0000","last_became_peered":"2024-06-29T15:57:21.213979+0000","last_unstale":"2024-06-29T15:57:36.331396+0000","last_undegraded":"2024-06-29T15:57:36.331396+0000","last_fullsized":"2024-06-29T15:57:36.331396+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:42.446135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331975+0000","last_change":"2024-06-29T15:57:22.231883+0000","last_active":"2024-06-29T15:57:36.331975+0000","last_peered":"2024-06-29T15:57:36.331975+0000","last_clean":"2024-06-29T15:57:36.331975+0000","last_became_active":"2024-06-29T15:57:22.231588+0000","last_became_peered":"2024-06-29T15:57:22.231588+0000","last_unstale":"2024-06-29T15:57:36.331975+0000","last_undegraded":"2024-06-29T15:57:36.331975+0000","last_fullsized":"2024-06-29T15:57:36.331975+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:17.705760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238496+0000","last_change":"2024-06-29T15:57:34.307517+0000","last_active":"2024-06-29T15:57:36.238496+0000","last_peered":"2024-06-29T15:57:36.238496+0000","last_clean":"2024-06-29T15:57:36.238496+0000","last_became_active":"2024-06-29T15:57:34.307213+0000","last_became_peered":"2024-06-29T15:57:34.307213+0000","last_unstale":"2024-06-29T15:57:36.238496+0000","last_undegraded":"2024-06-29T15:57:36.238496+0000","last_fullsized":"2024-06-29T15:57:36.238496+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:02.055681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311679+0000","last_change":"2024-06-29T15:57:35.262345+0000","last_active":"2024-06-29T15:57:36.311679+0000","last_peered":"2024-06-29T15:57:36.311679+0000","last_clean":"2024-06-29T15:57:36.311679+0000","last_became_active":"2024-06-29T15:57:35.262052+0000","last_became_peered":"2024-06-29T15:57:35.262052+0000","last_unstale":"2024-06-29T15:57:36.311679+0000","last_undegraded":"2024-06-29T15:57:36.311679+0000","last_fullsized":"2024-06-29T15:57:36.311679+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:54.683205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330760+0000","last_change":"2024-06-29T15:57:21.213193+0000","last_active":"2024-06-29T15:57:36.330760+0000","last_peered":"2024-06-29T15:57:36.330760+0000","last_clean":"2024-06-29T15:57:36.330760+0000","last_became_active":"2024-06-29T15:57:21.212838+0000","last_became_peered":"2024-06-29T15:57:21.212838+0000","last_unstale":"2024-06-29T15:57:36.330760+0000","last_undegraded":"2024-06-29T15:57:36.330760+0000","last_fullsized":"2024-06-29T15:57:36.330760+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:54.574391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240083+0000","last_change":"2024-06-29T15:57:22.217630+0000","last_active":"2024-06-29T15:57:36.240083+0000","last_peered":"2024-06-29T15:57:36.240083+0000","last_clean":"2024-06-29T15:57:36.240083+0000","last_became_active":"2024-06-29T15:57:22.217303+0000","last_became_peered":"2024-06-29T15:57:22.217303+0000","last_unstale":"2024-06-29T15:57:36.240083+0000","last_undegraded":"2024-06-29T15:57:36.240083+0000","last_fullsized":"2024-06-29T15:57:36.240083+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:05.554814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236277+0000","last_change":"2024-06-29T15:57:34.305621+0000","last_active":"2024-06-29T15:57:36.236277+0000","last_peered":"2024-06-29T15:57:36.236277+0000","last_clean":"2024-06-29T15:57:36.236277+0000","last_became_active":"2024-06-29T15:57:34.305274+0000","last_became_peered":"2024-06-29T15:57:34.305274+0000","last_unstale":"2024-06-29T15:57:36.236277+0000","last_undegraded":"2024-06-29T15:57:36.236277+0000","last_fullsized":"2024-06-29T15:57:36.236277+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:58.216624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238128+0000","last_change":"2024-06-29T15:57:35.258213+0000","last_active":"2024-06-29T15:57:36.238128+0000","last_peered":"2024-06-29T15:57:36.238128+0000","last_clean":"2024-06-29T15:57:36.238128+0000","last_became_active":"2024-06-29T15:57:35.257901+0000","last_became_peered":"2024-06-29T15:57:35.257901+0000","last_unstale":"2024-06-29T15:57:36.238128+0000","last_undegraded":"2024-06-29T15:57:36.238128+0000","last_fullsized":"2024-06-29T15:57:36.238128+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:34.564002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330032+0000","last_change":"2024-06-29T15:57:21.210528+0000","last_active":"2024-06-29T15:57:36.330032+0000","last_peered":"2024-06-29T15:57:36.330032+0000","last_clean":"2024-06-29T15:57:36.330032+0000","last_became_active":"2024-06-29T15:57:21.208691+0000","last_became_peered":"2024-06-29T15:57:21.208691+0000","last_unstale":"2024-06-29T15:57:36.330032+0000","last_undegraded":"2024-06-29T15:57:36.330032+0000","last_fullsized":"2024-06-29T15:57:36.330032+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.762590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240196+0000","last_change":"2024-06-29T15:57:22.232727+0000","last_active":"2024-06-29T15:57:36.240196+0000","last_peered":"2024-06-29T15:57:36.240196+0000","last_clean":"2024-06-29T15:57:36.240196+0000","last_became_active":"2024-06-29T15:57:22.232392+0000","last_became_peered":"2024-06-29T15:57:22.232392+0000","last_unstale":"2024-06-29T15:57:36.240196+0000","last_undegraded":"2024-06-29T15:57:36.240196+0000","last_fullsized":"2024-06-29T15:57:36.240196+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:54.805984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239217+0000","last_change":"2024-06-29T15:57:34.308385+0000","last_active":"2024-06-29T15:57:36.239217+0000","last_peered":"2024-06-29T15:57:36.239217+0000","last_clean":"2024-06-29T15:57:36.239217+0000","last_became_active":"2024-06-29T15:57:34.308070+0000","last_became_peered":"2024-06-29T15:57:34.308070+0000","last_unstale":"2024-06-29T15:57:36.239217+0000","last_undegraded":"2024-06-29T15:57:36.239217+0000","last_fullsized":"2024-06-29T15:57:36.239217+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:11.812167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331467+0000","last_change":"2024-06-29T15:57:35.256356+0000","last_active":"2024-06-29T15:57:36.331467+0000","last_peered":"2024-06-29T15:57:36.331467+0000","last_clean":"2024-06-29T15:57:36.331467+0000","last_became_active":"2024-06-29T15:57:35.255846+0000","last_became_peered":"2024-06-29T15:57:35.255846+0000","last_unstale":"2024-06-29T15:57:36.331467+0000","last_undegraded":"2024-06-29T15:57:36.331467+0000","last_fullsized":"2024-06-29T15:57:36.331467+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:41.823081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.e","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238583+0000","last_change":"2024-06-29T15:57:21.216179+0000","last_active":"2024-06-29T15:57:36.238583+0000","last_peered":"2024-06-29T15:57:36.238583+0000","last_clean":"2024-06-29T15:57:36.238583+0000","last_became_active":"2024-06-29T15:57:21.215876+0000","last_became_peered":"2024-06-29T15:57:21.215876+0000","last_unstale":"2024-06-29T15:57:36.238583+0000","last_undegraded":"2024-06-29T15:57:36.238583+0000","last_fullsized":"2024-06-29T15:57:36.238583+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:37.942186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238721+0000","last_change":"2024-06-29T15:57:22.225793+0000","last_active":"2024-06-29T15:57:36.238721+0000","last_peered":"2024-06-29T15:57:36.238721+0000","last_clean":"2024-06-29T15:57:36.238721+0000","last_became_active":"2024-06-29T15:57:22.224806+0000","last_became_peered":"2024-06-29T15:57:22.224806+0000","last_unstale":"2024-06-29T15:57:36.238721+0000","last_undegraded":"2024-06-29T15:57:36.238721+0000","last_fullsized":"2024-06-29T15:57:36.238721+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:03.667477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329189+0000","last_change":"2024-06-29T15:57:34.311124+0000","last_active":"2024-06-29T15:57:36.329189+0000","last_peered":"2024-06-29T15:57:36.329189+0000","last_clean":"2024-06-29T15:57:36.329189+0000","last_became_active":"2024-06-29T15:57:34.310775+0000","last_became_peered":"2024-06-29T15:57:34.310775+0000","last_unstale":"2024-06-29T15:57:36.329189+0000","last_undegraded":"2024-06-29T15:57:36.329189+0000","last_fullsized":"2024-06-29T15:57:36.329189+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:50.229318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331196+0000","last_change":"2024-06-29T15:57:35.259909+0000","last_active":"2024-06-29T15:57:36.331196+0000","last_peered":"2024-06-29T15:57:36.331196+0000","last_clean":"2024-06-29T15:57:36.331196+0000","last_became_active":"2024-06-29T15:57:35.259564+0000","last_became_peered":"2024-06-29T15:57:35.259564+0000","last_unstale":"2024-06-29T15:57:36.331196+0000","last_undegraded":"2024-06-29T15:57:36.331196+0000","last_fullsized":"2024-06-29T15:57:36.331196+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:18.743864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.f","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239899+0000","last_change":"2024-06-29T15:57:21.212718+0000","last_active":"2024-06-29T15:57:36.239899+0000","last_peered":"2024-06-29T15:57:36.239899+0000","last_clean":"2024-06-29T15:57:36.239899+0000","last_became_active":"2024-06-29T15:57:21.212406+0000","last_became_peered":"2024-06-29T15:57:21.212406+0000","last_unstale":"2024-06-29T15:57:36.239899+0000","last_undegraded":"2024-06-29T15:57:36.239899+0000","last_fullsized":"2024-06-29T15:57:36.239899+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:00.180622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239223+0000","last_change":"2024-06-29T15:57:22.238347+0000","last_active":"2024-06-29T15:57:36.239223+0000","last_peered":"2024-06-29T15:57:36.239223+0000","last_clean":"2024-06-29T15:57:36.239223+0000","last_became_active":"2024-06-29T15:57:22.236896+0000","last_became_peered":"2024-06-29T15:57:22.236896+0000","last_unstale":"2024-06-29T15:57:36.239223+0000","last_undegraded":"2024-06-29T15:57:36.239223+0000","last_fullsized":"2024-06-29T15:57:36.239223+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:39.739194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236003+0000","last_change":"2024-06-29T15:57:34.308954+0000","last_active":"2024-06-29T15:57:36.236003+0000","last_peered":"2024-06-29T15:57:36.236003+0000","last_clean":"2024-06-29T15:57:36.236003+0000","last_became_active":"2024-06-29T15:57:34.308595+0000","last_became_peered":"2024-06-29T15:57:34.308595+0000","last_unstale":"2024-06-29T15:57:36.236003+0000","last_undegraded":"2024-06-29T15:57:36.236003+0000","last_fullsized":"2024-06-29T15:57:36.236003+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:06:10.871555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236850+0000","last_change":"2024-06-29T15:57:35.250645+0000","last_active":"2024-06-29T15:57:36.236850+0000","last_peered":"2024-06-29T15:57:36.236850+0000","last_clean":"2024-06-29T15:57:36.236850+0000","last_became_active":"2024-06-29T15:57:35.250314+0000","last_became_peered":"2024-06-29T15:57:35.250314+0000","last_unstale":"2024-06-29T15:57:36.236850+0000","last_undegraded":"2024-06-29T15:57:36.236850+0000","last_fullsized":"2024-06-29T15:57:36.236850+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:06.391987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.10","version":"221'8","reported_seq":30,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329441+0000","last_change":"2024-06-29T15:57:21.216156+0000","last_active":"2024-06-29T15:57:36.329441+0000","last_peered":"2024-06-29T15:57:36.329441+0000","last_clean":"2024-06-29T15:57:36.329441+0000","last_became_active":"2024-06-29T15:57:21.214088+0000","last_became_peered":"2024-06-29T15:57:21.214088+0000","last_unstale":"2024-06-29T15:57:36.329441+0000","last_undegraded":"2024-06-29T15:57:36.329441+0000","last_fullsized":"2024-06-29T15:57:36.329441+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:30.314243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312937+0000","last_change":"2024-06-29T15:57:22.223112+0000","last_active":"2024-06-29T15:57:36.312937+0000","last_peered":"2024-06-29T15:57:36.312937+0000","last_clean":"2024-06-29T15:57:36.312937+0000","last_became_active":"2024-06-29T15:57:22.222632+0000","last_became_peered":"2024-06-29T15:57:22.222632+0000","last_unstale":"2024-06-29T15:57:36.312937+0000","last_undegraded":"2024-06-29T15:57:36.312937+0000","last_fullsized":"2024-06-29T15:57:36.312937+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:05.408776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329903+0000","last_change":"2024-06-29T15:57:34.310931+0000","last_active":"2024-06-29T15:57:36.329903+0000","last_peered":"2024-06-29T15:57:36.329903+0000","last_clean":"2024-06-29T15:57:36.329903+0000","last_became_active":"2024-06-29T15:57:34.310598+0000","last_became_peered":"2024-06-29T15:57:34.310598+0000","last_unstale":"2024-06-29T15:57:36.329903+0000","last_undegraded":"2024-06-29T15:57:36.329903+0000","last_fullsized":"2024-06-29T15:57:36.329903+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:51.824136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236659+0000","last_change":"2024-06-29T15:57:35.250035+0000","last_active":"2024-06-29T15:57:36.236659+0000","last_peered":"2024-06-29T15:57:36.236659+0000","last_clean":"2024-06-29T15:57:36.236659+0000","last_became_active":"2024-06-29T15:57:35.249711+0000","last_became_peered":"2024-06-29T15:57:35.249711+0000","last_unstale":"2024-06-29T15:57:36.236659+0000","last_undegraded":"2024-06-29T15:57:36.236659+0000","last_fullsized":"2024-06-29T15:57:36.236659+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:13.347781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329612+0000","last_change":"2024-06-29T15:57:21.215032+0000","last_active":"2024-06-29T15:57:36.329612+0000","last_peered":"2024-06-29T15:57:36.329612+0000","last_clean":"2024-06-29T15:57:36.329612+0000","last_became_active":"2024-06-29T15:57:21.214557+0000","last_became_peered":"2024-06-29T15:57:21.214557+0000","last_unstale":"2024-06-29T15:57:36.329612+0000","last_undegraded":"2024-06-29T15:57:36.329612+0000","last_fullsized":"2024-06-29T15:57:36.329612+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330185+0000","last_change":"2024-06-29T15:57:22.228484+0000","last_active":"2024-06-29T15:57:36.330185+0000","last_peered":"2024-06-29T15:57:36.330185+0000","last_clean":"2024-06-29T15:57:36.330185+0000","last_became_active":"2024-06-29T15:57:22.227880+0000","last_became_peered":"2024-06-29T15:57:22.227880+0000","last_unstale":"2024-06-29T15:57:36.330185+0000","last_undegraded":"2024-06-29T15:57:36.330185+0000","last_fullsized":"2024-06-29T15:57:36.330185+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:21.749071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312893+0000","last_change":"2024-06-29T15:57:34.309910+0000","last_active":"2024-06-29T15:57:36.312893+0000","last_peered":"2024-06-29T15:57:36.312893+0000","last_clean":"2024-06-29T15:57:36.312893+0000","last_became_active":"2024-06-29T15:57:34.309509+0000","last_became_peered":"2024-06-29T15:57:34.309509+0000","last_unstale":"2024-06-29T15:57:36.312893+0000","last_undegraded":"2024-06-29T15:57:36.312893+0000","last_fullsized":"2024-06-29T15:57:36.312893+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:05.013352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329983+0000","last_change":"2024-06-29T15:57:35.254963+0000","last_active":"2024-06-29T15:57:36.329983+0000","last_peered":"2024-06-29T15:57:36.329983+0000","last_clean":"2024-06-29T15:57:36.329983+0000","last_became_active":"2024-06-29T15:57:35.254059+0000","last_became_peered":"2024-06-29T15:57:35.254059+0000","last_unstale":"2024-06-29T15:57:36.329983+0000","last_undegraded":"2024-06-29T15:57:36.329983+0000","last_fullsized":"2024-06-29T15:57:36.329983+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:42.786307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236445+0000","last_change":"2024-06-29T15:57:21.203410+0000","last_active":"2024-06-29T15:57:36.236445+0000","last_peered":"2024-06-29T15:57:36.236445+0000","last_clean":"2024-06-29T15:57:36.236445+0000","last_became_active":"2024-06-29T15:57:21.203034+0000","last_became_peered":"2024-06-29T15:57:21.203034+0000","last_unstale":"2024-06-29T15:57:36.236445+0000","last_undegraded":"2024-06-29T15:57:36.236445+0000","last_fullsized":"2024-06-29T15:57:36.236445+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:52.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237205+0000","last_change":"2024-06-29T15:57:22.241881+0000","last_active":"2024-06-29T15:57:36.237205+0000","last_peered":"2024-06-29T15:57:36.237205+0000","last_clean":"2024-06-29T15:57:36.237205+0000","last_became_active":"2024-06-29T15:57:22.241549+0000","last_became_peered":"2024-06-29T15:57:22.241549+0000","last_unstale":"2024-06-29T15:57:36.237205+0000","last_undegraded":"2024-06-29T15:57:36.237205+0000","last_fullsized":"2024-06-29T15:57:36.237205+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:25.352328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"225'6","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240234+0000","last_change":"2024-06-29T15:57:34.309744+0000","last_active":"2024-06-29T15:57:36.240234+0000","last_peered":"2024-06-29T15:57:36.240234+0000","last_clean":"2024-06-29T15:57:36.240234+0000","last_became_active":"2024-06-29T15:57:34.309408+0000","last_became_peered":"2024-06-29T15:57:34.309408+0000","last_unstale":"2024-06-29T15:57:36.240234+0000","last_undegraded":"2024-06-29T15:57:36.240234+0000","last_fullsized":"2024-06-29T15:57:36.240234+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:20.219821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332388+0000","last_change":"2024-06-29T15:57:35.248181+0000","last_active":"2024-06-29T15:57:36.332388+0000","last_peered":"2024-06-29T15:57:36.332388+0000","last_clean":"2024-06-29T15:57:36.332388+0000","last_became_active":"2024-06-29T15:57:35.247656+0000","last_became_peered":"2024-06-29T15:57:35.247656+0000","last_unstale":"2024-06-29T15:57:36.332388+0000","last_undegraded":"2024-06-29T15:57:36.332388+0000","last_fullsized":"2024-06-29T15:57:36.332388+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:26.486217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331412+0000","last_change":"2024-06-29T15:57:21.215892+0000","last_active":"2024-06-29T15:57:36.331412+0000","last_peered":"2024-06-29T15:57:36.331412+0000","last_clean":"2024-06-29T15:57:36.331412+0000","last_became_active":"2024-06-29T15:57:21.215250+0000","last_became_peered":"2024-06-29T15:57:21.215250+0000","last_unstale":"2024-06-29T15:57:36.331412+0000","last_undegraded":"2024-06-29T15:57:36.331412+0000","last_fullsized":"2024-06-29T15:57:36.331412+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:23.234622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331871+0000","last_change":"2024-06-29T15:57:22.234802+0000","last_active":"2024-06-29T15:57:36.331871+0000","last_peered":"2024-06-29T15:57:36.331871+0000","last_clean":"2024-06-29T15:57:36.331871+0000","last_became_active":"2024-06-29T15:57:22.234492+0000","last_became_peered":"2024-06-29T15:57:22.234492+0000","last_unstale":"2024-06-29T15:57:36.331871+0000","last_undegraded":"2024-06-29T15:57:36.331871+0000","last_fullsized":"2024-06-29T15:57:36.331871+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:37.928014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"225'3","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312419+0000","last_change":"2024-06-29T15:57:34.308776+0000","last_active":"2024-06-29T15:57:36.312419+0000","last_peered":"2024-06-29T15:57:36.312419+0000","last_clean":"2024-06-29T15:57:36.312419+0000","last_became_active":"2024-06-29T15:57:34.308478+0000","last_became_peered":"2024-06-29T15:57:34.308478+0000","last_unstale":"2024-06-29T15:57:36.312419+0000","last_undegraded":"2024-06-29T15:57:36.312419+0000","last_fullsized":"2024-06-29T15:57:36.312419+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:14.899188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237305+0000","last_change":"2024-06-29T15:57:35.251086+0000","last_active":"2024-06-29T15:57:36.237305+0000","last_peered":"2024-06-29T15:57:36.237305+0000","last_clean":"2024-06-29T15:57:36.237305+0000","last_became_active":"2024-06-29T15:57:35.249526+0000","last_became_peered":"2024-06-29T15:57:35.249526+0000","last_unstale":"2024-06-29T15:57:36.237305+0000","last_undegraded":"2024-06-29T15:57:36.237305+0000","last_fullsized":"2024-06-29T15:57:36.237305+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:22.660265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.14","version":"222'8","reported_seq":30,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312307+0000","last_change":"2024-06-29T15:57:21.217755+0000","last_active":"2024-06-29T15:57:36.312307+0000","last_peered":"2024-06-29T15:57:36.312307+0000","last_clean":"2024-06-29T15:57:36.312307+0000","last_became_active":"2024-06-29T15:57:21.216983+0000","last_became_peered":"2024-06-29T15:57:21.216983+0000","last_unstale":"2024-06-29T15:57:36.312307+0000","last_undegraded":"2024-06-29T15:57:36.312307+0000","last_fullsized":"2024-06-29T15:57:36.312307+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:31.889343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331251+0000","last_change":"2024-06-29T15:57:22.224600+0000","last_active":"2024-06-29T15:57:36.331251+0000","last_peered":"2024-06-29T15:57:36.331251+0000","last_clean":"2024-06-29T15:57:36.331251+0000","last_became_active":"2024-06-29T15:57:22.224260+0000","last_became_peered":"2024-06-29T15:57:22.224260+0000","last_unstale":"2024-06-29T15:57:36.331251+0000","last_undegraded":"2024-06-29T15:57:36.331251+0000","last_fullsized":"2024-06-29T15:57:36.331251+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:21.773370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331316+0000","last_change":"2024-06-29T15:57:34.311602+0000","last_active":"2024-06-29T15:57:36.331316+0000","last_peered":"2024-06-29T15:57:36.331316+0000","last_clean":"2024-06-29T15:57:36.331316+0000","last_became_active":"2024-06-29T15:57:34.311208+0000","last_became_peered":"2024-06-29T15:57:34.311208+0000","last_unstale":"2024-06-29T15:57:36.331316+0000","last_undegraded":"2024-06-29T15:57:36.331316+0000","last_fullsized":"2024-06-29T15:57:36.331316+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:22.313530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329523+0000","last_change":"2024-06-29T15:57:35.257800+0000","last_active":"2024-06-29T15:57:36.329523+0000","last_peered":"2024-06-29T15:57:36.329523+0000","last_clean":"2024-06-29T15:57:36.329523+0000","last_became_active":"2024-06-29T15:57:35.254594+0000","last_became_peered":"2024-06-29T15:57:35.254594+0000","last_unstale":"2024-06-29T15:57:36.329523+0000","last_undegraded":"2024-06-29T15:57:36.329523+0000","last_fullsized":"2024-06-29T15:57:36.329523+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:10.058134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"221'3","reported_seq":25,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330966+0000","last_change":"2024-06-29T15:57:21.210270+0000","last_active":"2024-06-29T15:57:36.330966+0000","last_peered":"2024-06-29T15:57:36.330966+0000","last_clean":"2024-06-29T15:57:36.330966+0000","last_became_active":"2024-06-29T15:57:21.209731+0000","last_became_peered":"2024-06-29T15:57:21.209731+0000","last_unstale":"2024-06-29T15:57:36.330966+0000","last_undegraded":"2024-06-29T15:57:36.330966+0000","last_fullsized":"2024-06-29T15:57:36.330966+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:18.182233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331525+0000","last_change":"2024-06-29T15:57:22.231011+0000","last_active":"2024-06-29T15:57:36.331525+0000","last_peered":"2024-06-29T15:57:36.331525+0000","last_clean":"2024-06-29T15:57:36.331525+0000","last_became_active":"2024-06-29T15:57:22.230675+0000","last_became_peered":"2024-06-29T15:57:22.230675+0000","last_unstale":"2024-06-29T15:57:36.331525+0000","last_undegraded":"2024-06-29T15:57:36.331525+0000","last_fullsized":"2024-06-29T15:57:36.331525+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:09.478769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239084+0000","last_change":"2024-06-29T15:57:34.311119+0000","last_active":"2024-06-29T15:57:36.239084+0000","last_peered":"2024-06-29T15:57:36.239084+0000","last_clean":"2024-06-29T15:57:36.239084+0000","last_became_active":"2024-06-29T15:57:34.310820+0000","last_became_peered":"2024-06-29T15:57:34.310820+0000","last_unstale":"2024-06-29T15:57:36.239084+0000","last_undegraded":"2024-06-29T15:57:36.239084+0000","last_fullsized":"2024-06-29T15:57:36.239084+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:12:01.172748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236419+0000","last_change":"2024-06-29T15:57:35.252881+0000","last_active":"2024-06-29T15:57:36.236419+0000","last_peered":"2024-06-29T15:57:36.236419+0000","last_clean":"2024-06-29T15:57:36.236419+0000","last_became_active":"2024-06-29T15:57:35.252555+0000","last_became_peered":"2024-06-29T15:57:35.252555+0000","last_unstale":"2024-06-29T15:57:36.236419+0000","last_undegraded":"2024-06-29T15:57:36.236419+0000","last_fullsized":"2024-06-29T15:57:36.236419+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:13.830138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312629+0000","last_change":"2024-06-29T15:57:21.214536+0000","last_active":"2024-06-29T15:57:36.312629+0000","last_peered":"2024-06-29T15:57:36.312629+0000","last_clean":"2024-06-29T15:57:36.312629+0000","last_became_active":"2024-06-29T15:57:21.214227+0000","last_became_peered":"2024-06-29T15:57:21.214227+0000","last_unstale":"2024-06-29T15:57:36.312629+0000","last_undegraded":"2024-06-29T15:57:36.312629+0000","last_fullsized":"2024-06-29T15:57:36.312629+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:25.839522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236127+0000","last_change":"2024-06-29T15:57:22.219192+0000","last_active":"2024-06-29T15:57:36.236127+0000","last_peered":"2024-06-29T15:57:36.236127+0000","last_clean":"2024-06-29T15:57:36.236127+0000","last_became_active":"2024-06-29T15:57:22.218844+0000","last_became_peered":"2024-06-29T15:57:22.218844+0000","last_unstale":"2024-06-29T15:57:36.236127+0000","last_undegraded":"2024-06-29T15:57:36.236127+0000","last_fullsized":"2024-06-29T15:57:36.236127+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:50.717283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"225'8","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238266+0000","last_change":"2024-06-29T15:57:34.309117+0000","last_active":"2024-06-29T15:57:36.238266+0000","last_peered":"2024-06-29T15:57:36.238266+0000","last_clean":"2024-06-29T15:57:36.238266+0000","last_became_active":"2024-06-29T15:57:34.308792+0000","last_became_peered":"2024-06-29T15:57:34.308792+0000","last_unstale":"2024-06-29T15:57:36.238266+0000","last_undegraded":"2024-06-29T15:57:36.238266+0000","last_fullsized":"2024-06-29T15:57:36.238266+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:31:47.175085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330362+0000","last_change":"2024-06-29T15:57:35.256824+0000","last_active":"2024-06-29T15:57:36.330362+0000","last_peered":"2024-06-29T15:57:36.330362+0000","last_clean":"2024-06-29T15:57:36.330362+0000","last_became_active":"2024-06-29T15:57:35.255771+0000","last_became_peered":"2024-06-29T15:57:35.255771+0000","last_unstale":"2024-06-29T15:57:36.330362+0000","last_undegraded":"2024-06-29T15:57:36.330362+0000","last_fullsized":"2024-06-29T15:57:36.330362+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:51.718094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331403+0000","last_change":"2024-06-29T15:57:21.218814+0000","last_active":"2024-06-29T15:57:36.331403+0000","last_peered":"2024-06-29T15:57:36.331403+0000","last_clean":"2024-06-29T15:57:36.331403+0000","last_became_active":"2024-06-29T15:57:21.218495+0000","last_became_peered":"2024-06-29T15:57:21.218495+0000","last_unstale":"2024-06-29T15:57:36.331403+0000","last_undegraded":"2024-06-29T15:57:36.331403+0000","last_fullsized":"2024-06-29T15:57:36.331403+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:04.680014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329332+0000","last_change":"2024-06-29T15:57:22.235312+0000","last_active":"2024-06-29T15:57:36.329332+0000","last_peered":"2024-06-29T15:57:36.329332+0000","last_clean":"2024-06-29T15:57:36.329332+0000","last_became_active":"2024-06-29T15:57:22.234828+0000","last_became_peered":"2024-06-29T15:57:22.234828+0000","last_unstale":"2024-06-29T15:57:36.329332+0000","last_undegraded":"2024-06-29T15:57:36.329332+0000","last_fullsized":"2024-06-29T15:57:36.329332+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:39.661714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237083+0000","last_change":"2024-06-29T15:57:34.306092+0000","last_active":"2024-06-29T15:57:36.237083+0000","last_peered":"2024-06-29T15:57:36.237083+0000","last_clean":"2024-06-29T15:57:36.237083+0000","last_became_active":"2024-06-29T15:57:34.305717+0000","last_became_peered":"2024-06-29T15:57:34.305717+0000","last_unstale":"2024-06-29T15:57:36.237083+0000","last_undegraded":"2024-06-29T15:57:36.237083+0000","last_fullsized":"2024-06-29T15:57:36.237083+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:12.354679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329622+0000","last_change":"2024-06-29T15:57:35.244422+0000","last_active":"2024-06-29T15:57:36.329622+0000","last_peered":"2024-06-29T15:57:36.329622+0000","last_clean":"2024-06-29T15:57:36.329622+0000","last_became_active":"2024-06-29T15:57:35.244085+0000","last_became_peered":"2024-06-29T15:57:35.244085+0000","last_unstale":"2024-06-29T15:57:36.329622+0000","last_undegraded":"2024-06-29T15:57:36.329622+0000","last_fullsized":"2024-06-29T15:57:36.329622+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:31.011742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238696+0000","last_change":"2024-06-29T15:57:34.312096+0000","last_active":"2024-06-29T15:57:36.238696+0000","last_peered":"2024-06-29T15:57:36.238696+0000","last_clean":"2024-06-29T15:57:36.238696+0000","last_became_active":"2024-06-29T15:57:34.311757+0000","last_became_peered":"2024-06-29T15:57:34.311757+0000","last_unstale":"2024-06-29T15:57:36.238696+0000","last_undegraded":"2024-06-29T15:57:36.238696+0000","last_fullsized":"2024-06-29T15:57:36.238696+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:51.898399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236052+0000","last_change":"2024-06-29T15:57:21.208896+0000","last_active":"2024-06-29T15:57:36.236052+0000","last_peered":"2024-06-29T15:57:36.236052+0000","last_clean":"2024-06-29T15:57:36.236052+0000","last_became_active":"2024-06-29T15:57:21.208555+0000","last_became_peered":"2024-06-29T15:57:21.208555+0000","last_unstale":"2024-06-29T15:57:36.236052+0000","last_undegraded":"2024-06-29T15:57:36.236052+0000","last_fullsized":"2024-06-29T15:57:36.236052+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330531+0000","last_change":"2024-06-29T15:57:22.234743+0000","last_active":"2024-06-29T15:57:36.330531+0000","last_peered":"2024-06-29T15:57:36.330531+0000","last_clean":"2024-06-29T15:57:36.330531+0000","last_became_active":"2024-06-29T15:57:22.233091+0000","last_became_peered":"2024-06-29T15:57:22.233091+0000","last_unstale":"2024-06-29T15:57:36.330531+0000","last_undegraded":"2024-06-29T15:57:36.330531+0000","last_fullsized":"2024-06-29T15:57:36.330531+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:46.131681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331747+0000","last_change":"2024-06-29T15:57:35.247871+0000","last_active":"2024-06-29T15:57:36.331747+0000","last_peered":"2024-06-29T15:57:36.331747+0000","last_clean":"2024-06-29T15:57:36.331747+0000","last_became_active":"2024-06-29T15:57:35.247551+0000","last_became_peered":"2024-06-29T15:57:35.247551+0000","last_unstale":"2024-06-29T15:57:36.331747+0000","last_undegraded":"2024-06-29T15:57:36.331747+0000","last_fullsized":"2024-06-29T15:57:36.331747+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:38:28.226246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"225'4","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330982+0000","last_change":"2024-06-29T15:57:34.310706+0000","last_active":"2024-06-29T15:57:36.330982+0000","last_peered":"2024-06-29T15:57:36.330982+0000","last_clean":"2024-06-29T15:57:36.330982+0000","last_became_active":"2024-06-29T15:57:34.310360+0000","last_became_peered":"2024-06-29T15:57:34.310360+0000","last_unstale":"2024-06-29T15:57:36.330982+0000","last_undegraded":"2024-06-29T15:57:36.330982+0000","last_fullsized":"2024-06-29T15:57:36.330982+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:38.399869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311825+0000","last_change":"2024-06-29T15:57:21.212034+0000","last_active":"2024-06-29T15:57:36.311825+0000","last_peered":"2024-06-29T15:57:36.311825+0000","last_clean":"2024-06-29T15:57:36.311825+0000","last_became_active":"2024-06-29T15:57:21.211762+0000","last_became_peered":"2024-06-29T15:57:21.211762+0000","last_unstale":"2024-06-29T15:57:36.311825+0000","last_undegraded":"2024-06-29T15:57:36.311825+0000","last_fullsized":"2024-06-29T15:57:36.311825+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:36.775853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330016+0000","last_change":"2024-06-29T15:57:22.214168+0000","last_active":"2024-06-29T15:57:36.330016+0000","last_peered":"2024-06-29T15:57:36.330016+0000","last_clean":"2024-06-29T15:57:36.330016+0000","last_became_active":"2024-06-29T15:57:22.213780+0000","last_became_peered":"2024-06-29T15:57:22.213780+0000","last_unstale":"2024-06-29T15:57:36.330016+0000","last_undegraded":"2024-06-29T15:57:36.330016+0000","last_fullsized":"2024-06-29T15:57:36.330016+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:01.904725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238570+0000","last_change":"2024-06-29T15:57:35.257408+0000","last_active":"2024-06-29T15:57:36.238570+0000","last_peered":"2024-06-29T15:57:36.238570+0000","last_clean":"2024-06-29T15:57:36.238570+0000","last_became_active":"2024-06-29T15:57:35.257065+0000","last_became_peered":"2024-06-29T15:57:35.257065+0000","last_unstale":"2024-06-29T15:57:36.238570+0000","last_undegraded":"2024-06-29T15:57:36.238570+0000","last_fullsized":"2024-06-29T15:57:36.238570+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:32.986678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1b","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311976+0000","last_change":"2024-06-29T15:57:21.211113+0000","last_active":"2024-06-29T15:57:36.311976+0000","last_peered":"2024-06-29T15:57:36.311976+0000","last_clean":"2024-06-29T15:57:36.311976+0000","last_became_active":"2024-06-29T15:57:21.210817+0000","last_became_peered":"2024-06-29T15:57:21.210817+0000","last_unstale":"2024-06-29T15:57:36.311976+0000","last_undegraded":"2024-06-29T15:57:36.311976+0000","last_fullsized":"2024-06-29T15:57:36.311976+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:26.176884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329924+0000","last_change":"2024-06-29T15:57:22.235721+0000","last_active":"2024-06-29T15:57:36.329924+0000","last_peered":"2024-06-29T15:57:36.329924+0000","last_clean":"2024-06-29T15:57:36.329924+0000","last_became_active":"2024-06-29T15:57:22.235380+0000","last_became_peered":"2024-06-29T15:57:22.235380+0000","last_unstale":"2024-06-29T15:57:36.329924+0000","last_undegraded":"2024-06-29T15:57:36.329924+0000","last_fullsized":"2024-06-29T15:57:36.329924+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:38.144376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237819+0000","last_change":"2024-06-29T15:57:34.310685+0000","last_active":"2024-06-29T15:57:36.237819+0000","last_peered":"2024-06-29T15:57:36.237819+0000","last_clean":"2024-06-29T15:57:36.237819+0000","last_became_active":"2024-06-29T15:57:34.310377+0000","last_became_peered":"2024-06-29T15:57:34.310377+0000","last_unstale":"2024-06-29T15:57:36.237819+0000","last_undegraded":"2024-06-29T15:57:36.237819+0000","last_fullsized":"2024-06-29T15:57:36.237819+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:36.326091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237218+0000","last_change":"2024-06-29T15:57:35.251476+0000","last_active":"2024-06-29T15:57:36.237218+0000","last_peered":"2024-06-29T15:57:36.237218+0000","last_clean":"2024-06-29T15:57:36.237218+0000","last_became_active":"2024-06-29T15:57:35.250496+0000","last_became_peered":"2024-06-29T15:57:35.250496+0000","last_unstale":"2024-06-29T15:57:36.237218+0000","last_undegraded":"2024-06-29T15:57:36.237218+0000","last_fullsized":"2024-06-29T15:57:36.237218+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:52.738461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330242+0000","last_change":"2024-06-29T15:57:21.210643+0000","last_active":"2024-06-29T15:57:36.330242+0000","last_peered":"2024-06-29T15:57:36.330242+0000","last_clean":"2024-06-29T15:57:36.330242+0000","last_became_active":"2024-06-29T15:57:21.209805+0000","last_became_peered":"2024-06-29T15:57:21.209805+0000","last_unstale":"2024-06-29T15:57:36.330242+0000","last_undegraded":"2024-06-29T15:57:36.330242+0000","last_fullsized":"2024-06-29T15:57:36.330242+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:27.618049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331248+0000","last_change":"2024-06-29T15:57:22.226181+0000","last_active":"2024-06-29T15:57:36.331248+0000","last_peered":"2024-06-29T15:57:36.331248+0000","last_clean":"2024-06-29T15:57:36.331248+0000","last_became_active":"2024-06-29T15:57:22.225855+0000","last_became_peered":"2024-06-29T15:57:22.225855+0000","last_unstale":"2024-06-29T15:57:36.331248+0000","last_undegraded":"2024-06-29T15:57:36.331248+0000","last_fullsized":"2024-06-29T15:57:36.331248+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:37.067580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311945+0000","last_change":"2024-06-29T15:57:34.314461+0000","last_active":"2024-06-29T15:57:36.311945+0000","last_peered":"2024-06-29T15:57:36.311945+0000","last_clean":"2024-06-29T15:57:36.311945+0000","last_became_active":"2024-06-29T15:57:34.314139+0000","last_became_peered":"2024-06-29T15:57:34.314139+0000","last_unstale":"2024-06-29T15:57:36.311945+0000","last_undegraded":"2024-06-29T15:57:36.311945+0000","last_fullsized":"2024-06-29T15:57:36.311945+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:09.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330356+0000","last_change":"2024-06-29T15:57:35.246277+0000","last_active":"2024-06-29T15:57:36.330356+0000","last_peered":"2024-06-29T15:57:36.330356+0000","last_clean":"2024-06-29T15:57:36.330356+0000","last_became_active":"2024-06-29T15:57:35.245894+0000","last_became_peered":"2024-06-29T15:57:35.245894+0000","last_unstale":"2024-06-29T15:57:36.330356+0000","last_undegraded":"2024-06-29T15:57:36.330356+0000","last_fullsized":"2024-06-29T15:57:36.330356+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:16.673478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1d","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329758+0000","last_change":"2024-06-29T15:57:21.210602+0000","last_active":"2024-06-29T15:57:36.329758+0000","last_peered":"2024-06-29T15:57:36.329758+0000","last_clean":"2024-06-29T15:57:36.329758+0000","last_became_active":"2024-06-29T15:57:21.210296+0000","last_became_peered":"2024-06-29T15:57:21.210296+0000","last_unstale":"2024-06-29T15:57:36.329758+0000","last_undegraded":"2024-06-29T15:57:36.329758+0000","last_fullsized":"2024-06-29T15:57:36.329758+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:07.128352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238476+0000","last_change":"2024-06-29T15:57:22.213237+0000","last_active":"2024-06-29T15:57:36.238476+0000","last_peered":"2024-06-29T15:57:36.238476+0000","last_clean":"2024-06-29T15:57:36.238476+0000","last_became_active":"2024-06-29T15:57:22.212908+0000","last_became_peered":"2024-06-29T15:57:22.212908+0000","last_unstale":"2024-06-29T15:57:36.238476+0000","last_undegraded":"2024-06-29T15:57:36.238476+0000","last_fullsized":"2024-06-29T15:57:36.238476+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:31.343924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236505+0000","last_change":"2024-06-29T15:57:34.313977+0000","last_active":"2024-06-29T15:57:36.236505+0000","last_peered":"2024-06-29T15:57:36.236505+0000","last_clean":"2024-06-29T15:57:36.236505+0000","last_became_active":"2024-06-29T15:57:34.313702+0000","last_became_peered":"2024-06-29T15:57:34.313702+0000","last_unstale":"2024-06-29T15:57:36.236505+0000","last_undegraded":"2024-06-29T15:57:36.236505+0000","last_fullsized":"2024-06-29T15:57:36.236505+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:12.668316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329526+0000","last_change":"2024-06-29T15:57:35.242307+0000","last_active":"2024-06-29T15:57:36.329526+0000","last_peered":"2024-06-29T15:57:36.329526+0000","last_clean":"2024-06-29T15:57:36.329526+0000","last_became_active":"2024-06-29T15:57:35.241559+0000","last_became_peered":"2024-06-29T15:57:35.241559+0000","last_unstale":"2024-06-29T15:57:36.329526+0000","last_undegraded":"2024-06-29T15:57:36.329526+0000","last_fullsized":"2024-06-29T15:57:36.329526+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:37.228107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1e","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238472+0000","last_change":"2024-06-29T15:57:21.213288+0000","last_active":"2024-06-29T15:57:36.238472+0000","last_peered":"2024-06-29T15:57:36.238472+0000","last_clean":"2024-06-29T15:57:36.238472+0000","last_became_active":"2024-06-29T15:57:21.212954+0000","last_became_peered":"2024-06-29T15:57:21.212954+0000","last_unstale":"2024-06-29T15:57:36.238472+0000","last_undegraded":"2024-06-29T15:57:36.238472+0000","last_fullsized":"2024-06-29T15:57:36.238472+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:24.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237336+0000","last_change":"2024-06-29T15:57:22.224449+0000","last_active":"2024-06-29T15:57:36.237336+0000","last_peered":"2024-06-29T15:57:36.237336+0000","last_clean":"2024-06-29T15:57:36.237336+0000","last_became_active":"2024-06-29T15:57:22.224124+0000","last_became_peered":"2024-06-29T15:57:22.224124+0000","last_unstale":"2024-06-29T15:57:36.237336+0000","last_undegraded":"2024-06-29T15:57:36.237336+0000","last_fullsized":"2024-06-29T15:57:36.237336+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:05.649826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330422+0000","last_change":"2024-06-29T15:57:34.304015+0000","last_active":"2024-06-29T15:57:36.330422+0000","last_peered":"2024-06-29T15:57:36.330422+0000","last_clean":"2024-06-29T15:57:36.330422+0000","last_became_active":"2024-06-29T15:57:34.303686+0000","last_became_peered":"2024-06-29T15:57:34.303686+0000","last_unstale":"2024-06-29T15:57:36.330422+0000","last_undegraded":"2024-06-29T15:57:36.330422+0000","last_fullsized":"2024-06-29T15:57:36.330422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:48.678300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236409+0000","last_change":"2024-06-29T15:57:35.244368+0000","last_active":"2024-06-29T15:57:36.236409+0000","last_peered":"2024-06-29T15:57:36.236409+0000","last_clean":"2024-06-29T15:57:36.236409+0000","last_became_active":"2024-06-29T15:57:35.243599+0000","last_became_peered":"2024-06-29T15:57:35.243599+0000","last_unstale":"2024-06-29T15:57:36.236409+0000","last_undegraded":"2024-06-29T15:57:36.236409+0000","last_fullsized":"2024-06-29T15:57:36.236409+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:10.058134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1f","version":"221'4","reported_seq":26,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330083+0000","last_change":"2024-06-29T15:57:21.218674+0000","last_active":"2024-06-29T15:57:36.330083+0000","last_peered":"2024-06-29T15:57:36.330083+0000","last_clean":"2024-06-29T15:57:36.330083+0000","last_became_active":"2024-06-29T15:57:21.218397+0000","last_became_peered":"2024-06-29T15:57:21.218397+0000","last_unstale":"2024-06-29T15:57:36.330083+0000","last_undegraded":"2024-06-29T15:57:36.330083+0000","last_fullsized":"2024-06-29T15:57:36.330083+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.766844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330985+0000","last_change":"2024-06-29T15:57:22.228934+0000","last_active":"2024-06-29T15:57:36.330985+0000","last_peered":"2024-06-29T15:57:36.330985+0000","last_clean":"2024-06-29T15:57:36.330985+0000","last_became_active":"2024-06-29T15:57:22.228603+0000","last_became_peered":"2024-06-29T15:57:22.228603+0000","last_unstale":"2024-06-29T15:57:36.330985+0000","last_undegraded":"2024-06-29T15:57:36.330985+0000","last_fullsized":"2024-06-29T15:57:36.330985+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:28.391534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329161+0000","last_change":"2024-06-29T15:57:34.310834+0000","last_active":"2024-06-29T15:57:36.329161+0000","last_peered":"2024-06-29T15:57:36.329161+0000","last_clean":"2024-06-29T15:57:36.329161+0000","last_became_active":"2024-06-29T15:57:34.310527+0000","last_became_peered":"2024-06-29T15:57:34.310527+0000","last_unstale":"2024-06-29T15:57:36.329161+0000","last_undegraded":"2024-06-29T15:57:36.329161+0000","last_fullsized":"2024-06-29T15:57:36.329161+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:36.219011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239563+0000","last_change":"2024-06-29T15:57:35.256077+0000","last_active":"2024-06-29T15:57:36.239563+0000","last_peered":"2024-06-29T15:57:36.239563+0000","last_clean":"2024-06-29T15:57:36.239563+0000","last_became_active":"2024-06-29T15:57:35.255699+0000","last_became_peered":"2024-06-29T15:57:35.255699+0000","last_unstale":"2024-06-29T15:57:36.239563+0000","last_undegraded":"2024-06-29T15:57:36.239563+0000","last_fullsized":"2024-06-29T15:57:36.239563+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:30.089154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311494+0000","last_change":"2024-06-29T15:57:22.224277+0000","last_active":"2024-06-29T15:57:36.311494+0000","last_peered":"2024-06-29T15:57:36.311494+0000","last_clean":"2024-06-29T15:57:36.311494+0000","last_became_active":"2024-06-29T15:57:22.223975+0000","last_became_peered":"2024-06-29T15:57:22.223975+0000","last_unstale":"2024-06-29T15:57:36.311494+0000","last_undegraded":"2024-06-29T15:57:36.311494+0000","last_fullsized":"2024-06-29T15:57:36.311494+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:46.705935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329941+0000","last_change":"2024-06-29T15:57:35.257997+0000","last_active":"2024-06-29T15:57:36.329941+0000","last_peered":"2024-06-29T15:57:36.329941+0000","last_clean":"2024-06-29T15:57:36.329941+0000","last_became_active":"2024-06-29T15:57:35.254612+0000","last_became_peered":"2024-06-29T15:57:35.254612+0000","last_unstale":"2024-06-29T15:57:36.329941+0000","last_undegraded":"2024-06-29T15:57:36.329941+0000","last_fullsized":"2024-06-29T15:57:36.329941+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:13.347781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311649+0000","last_change":"2024-06-29T15:57:22.239789+0000","last_active":"2024-06-29T15:57:36.311649+0000","last_peered":"2024-06-29T15:57:36.311649+0000","last_clean":"2024-06-29T15:57:36.311649+0000","last_became_active":"2024-06-29T15:57:22.239438+0000","last_became_peered":"2024-06-29T15:57:22.239438+0000","last_unstale":"2024-06-29T15:57:36.311649+0000","last_undegraded":"2024-06-29T15:57:36.311649+0000","last_fullsized":"2024-06-29T15:57:36.311649+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:08.408484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237238+0000","last_change":"2024-06-29T15:57:35.251148+0000","last_active":"2024-06-29T15:57:36.237238+0000","last_peered":"2024-06-29T15:57:36.237238+0000","last_clean":"2024-06-29T15:57:36.237238+0000","last_became_active":"2024-06-29T15:57:35.250806+0000","last_became_peered":"2024-06-29T15:57:35.250806+0000","last_unstale":"2024-06-29T15:57:36.237238+0000","last_undegraded":"2024-06-29T15:57:36.237238+0000","last_fullsized":"2024-06-29T15:57:36.237238+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:54.987013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329513+0000","last_change":"2024-06-29T15:57:22.236151+0000","last_active":"2024-06-29T15:57:36.329513+0000","last_peered":"2024-06-29T15:57:36.329513+0000","last_clean":"2024-06-29T15:57:36.329513+0000","last_became_active":"2024-06-29T15:57:22.235409+0000","last_became_peered":"2024-06-29T15:57:22.235409+0000","last_unstale":"2024-06-29T15:57:36.329513+0000","last_undegraded":"2024-06-29T15:57:36.329513+0000","last_fullsized":"2024-06-29T15:57:36.329513+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:56.149438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329621+0000","last_change":"2024-06-29T15:57:35.255181+0000","last_active":"2024-06-29T15:57:36.329621+0000","last_peered":"2024-06-29T15:57:36.329621+0000","last_clean":"2024-06-29T15:57:36.329621+0000","last_became_active":"2024-06-29T15:57:35.254686+0000","last_became_peered":"2024-06-29T15:57:35.254686+0000","last_unstale":"2024-06-29T15:57:36.329621+0000","last_undegraded":"2024-06-29T15:57:36.329621+0000","last_fullsized":"2024-06-29T15:57:36.329621+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:52.833232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329737+0000","last_change":"2024-06-29T15:57:22.227876+0000","last_active":"2024-06-29T15:57:36.329737+0000","last_peered":"2024-06-29T15:57:36.329737+0000","last_clean":"2024-06-29T15:57:36.329737+0000","last_became_active":"2024-06-29T15:57:22.227591+0000","last_became_peered":"2024-06-29T15:57:22.227591+0000","last_unstale":"2024-06-29T15:57:36.329737+0000","last_undegraded":"2024-06-29T15:57:36.329737+0000","last_fullsized":"2024-06-29T15:57:36.329737+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:45.600754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332187+0000","last_change":"2024-06-29T15:57:35.257955+0000","last_active":"2024-06-29T15:57:36.332187+0000","last_peered":"2024-06-29T15:57:36.332187+0000","last_clean":"2024-06-29T15:57:36.332187+0000","last_became_active":"2024-06-29T15:57:35.256657+0000","last_became_peered":"2024-06-29T15:57:35.256657+0000","last_unstale":"2024-06-29T15:57:36.332187+0000","last_undegraded":"2024-06-29T15:57:36.332187+0000","last_fullsized":"2024-06-29T15:57:36.332187+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:57.126833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330655+0000","last_change":"2024-06-29T15:57:22.227845+0000","last_active":"2024-06-29T15:57:36.330655+0000","last_peered":"2024-06-29T15:57:36.330655+0000","last_clean":"2024-06-29T15:57:36.330655+0000","last_became_active":"2024-06-29T15:57:22.227549+0000","last_became_peered":"2024-06-29T15:57:22.227549+0000","last_unstale":"2024-06-29T15:57:36.330655+0000","last_undegraded":"2024-06-29T15:57:36.330655+0000","last_fullsized":"2024-06-29T15:57:36.330655+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:22.735783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312767+0000","last_change":"2024-06-29T15:57:35.258694+0000","last_active":"2024-06-29T15:57:36.312767+0000","last_peered":"2024-06-29T15:57:36.312767+0000","last_clean":"2024-06-29T15:57:36.312767+0000","last_became_active":"2024-06-29T15:57:35.258369+0000","last_became_peered":"2024-06-29T15:57:35.258369+0000","last_unstale":"2024-06-29T15:57:36.312767+0000","last_undegraded":"2024-06-29T15:57:36.312767+0000","last_fullsized":"2024-06-29T15:57:36.312767+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:30.546092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237188+0000","last_change":"2024-06-29T15:57:22.236939+0000","last_active":"2024-06-29T15:57:36.237188+0000","last_peered":"2024-06-29T15:57:36.237188+0000","last_clean":"2024-06-29T15:57:36.237188+0000","last_became_active":"2024-06-29T15:57:22.236129+0000","last_became_peered":"2024-06-29T15:57:22.236129+0000","last_unstale":"2024-06-29T15:57:36.237188+0000","last_undegraded":"2024-06-29T15:57:36.237188+0000","last_fullsized":"2024-06-29T15:57:36.237188+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:13.274461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239151+0000","last_change":"2024-06-29T15:57:35.248969+0000","last_active":"2024-06-29T15:57:36.239151+0000","last_peered":"2024-06-29T15:57:36.239151+0000","last_clean":"2024-06-29T15:57:36.239151+0000","last_became_active":"2024-06-29T15:57:35.248617+0000","last_became_peered":"2024-06-29T15:57:35.248617+0000","last_unstale":"2024-06-29T15:57:36.239151+0000","last_undegraded":"2024-06-29T15:57:36.239151+0000","last_fullsized":"2024-06-29T15:57:36.239151+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:23.344350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238364+0000","last_change":"2024-06-29T15:57:22.225280+0000","last_active":"2024-06-29T15:57:36.238364+0000","last_peered":"2024-06-29T15:57:36.238364+0000","last_clean":"2024-06-29T15:57:36.238364+0000","last_became_active":"2024-06-29T15:57:22.224986+0000","last_became_peered":"2024-06-29T15:57:22.224986+0000","last_unstale":"2024-06-29T15:57:36.238364+0000","last_undegraded":"2024-06-29T15:57:36.238364+0000","last_fullsized":"2024-06-29T15:57:36.238364+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:07.391769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329941+0000","last_change":"2024-06-29T15:57:35.249995+0000","last_active":"2024-06-29T15:57:36.329941+0000","last_peered":"2024-06-29T15:57:36.329941+0000","last_clean":"2024-06-29T15:57:36.329941+0000","last_became_active":"2024-06-29T15:57:35.249082+0000","last_became_peered":"2024-06-29T15:57:35.249082+0000","last_unstale":"2024-06-29T15:57:36.329941+0000","last_undegraded":"2024-06-29T15:57:36.329941+0000","last_fullsized":"2024-06-29T15:57:36.329941+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:18.648852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330431+0000","last_change":"2024-06-29T15:57:22.234902+0000","last_active":"2024-06-29T15:57:36.330431+0000","last_peered":"2024-06-29T15:57:36.330431+0000","last_clean":"2024-06-29T15:57:36.330431+0000","last_became_active":"2024-06-29T15:57:22.233244+0000","last_became_peered":"2024-06-29T15:57:22.233244+0000","last_unstale":"2024-06-29T15:57:36.330431+0000","last_undegraded":"2024-06-29T15:57:36.330431+0000","last_fullsized":"2024-06-29T15:57:36.330431+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:58.786612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236210+0000","last_change":"2024-06-29T15:57:35.243088+0000","last_active":"2024-06-29T15:57:36.236210+0000","last_peered":"2024-06-29T15:57:36.236210+0000","last_clean":"2024-06-29T15:57:36.236210+0000","last_became_active":"2024-06-29T15:57:35.242768+0000","last_became_peered":"2024-06-29T15:57:35.242768+0000","last_unstale":"2024-06-29T15:57:36.236210+0000","last_undegraded":"2024-06-29T15:57:36.236210+0000","last_fullsized":"2024-06-29T15:57:36.236210+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:43.817141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329319+0000","last_change":"2024-06-29T15:57:22.223814+0000","last_active":"2024-06-29T15:57:36.329319+0000","last_peered":"2024-06-29T15:57:36.329319+0000","last_clean":"2024-06-29T15:57:36.329319+0000","last_became_active":"2024-06-29T15:57:22.223464+0000","last_became_peered":"2024-06-29T15:57:22.223464+0000","last_unstale":"2024-06-29T15:57:36.329319+0000","last_undegraded":"2024-06-29T15:57:36.329319+0000","last_fullsized":"2024-06-29T15:57:36.329319+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:40.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331060+0000","last_change":"2024-06-29T15:57:35.256428+0000","last_active":"2024-06-29T15:57:36.331060+0000","last_peered":"2024-06-29T15:57:36.331060+0000","last_clean":"2024-06-29T15:57:36.331060+0000","last_became_active":"2024-06-29T15:57:35.255357+0000","last_became_peered":"2024-06-29T15:57:35.255357+0000","last_unstale":"2024-06-29T15:57:36.331060+0000","last_undegraded":"2024-06-29T15:57:36.331060+0000","last_fullsized":"2024-06-29T15:57:36.331060+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:10:51.973923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331341+0000","last_change":"2024-06-29T15:57:22.232432+0000","last_active":"2024-06-29T15:57:36.331341+0000","last_peered":"2024-06-29T15:57:36.331341+0000","last_clean":"2024-06-29T15:57:36.331341+0000","last_became_active":"2024-06-29T15:57:22.232102+0000","last_became_peered":"2024-06-29T15:57:22.232102+0000","last_unstale":"2024-06-29T15:57:36.331341+0000","last_undegraded":"2024-06-29T15:57:36.331341+0000","last_fullsized":"2024-06-29T15:57:36.331341+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:41.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311586+0000","last_change":"2024-06-29T15:57:35.259401+0000","last_active":"2024-06-29T15:57:36.311586+0000","last_peered":"2024-06-29T15:57:36.311586+0000","last_clean":"2024-06-29T15:57:36.311586+0000","last_became_active":"2024-06-29T15:57:35.259002+0000","last_became_peered":"2024-06-29T15:57:35.259002+0000","last_unstale":"2024-06-29T15:57:36.311586+0000","last_undegraded":"2024-06-29T15:57:36.311586+0000","last_fullsized":"2024-06-29T15:57:36.311586+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:34.435763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238215+0000","last_change":"2024-06-29T15:57:22.222075+0000","last_active":"2024-06-29T15:57:36.238215+0000","last_peered":"2024-06-29T15:57:36.238215+0000","last_clean":"2024-06-29T15:57:36.238215+0000","last_became_active":"2024-06-29T15:57:22.221767+0000","last_became_peered":"2024-06-29T15:57:22.221767+0000","last_unstale":"2024-06-29T15:57:36.238215+0000","last_undegraded":"2024-06-29T15:57:36.238215+0000","last_fullsized":"2024-06-29T15:57:36.238215+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:13.762887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329358+0000","last_change":"2024-06-29T15:57:35.249923+0000","last_active":"2024-06-29T15:57:36.329358+0000","last_peered":"2024-06-29T15:57:36.329358+0000","last_clean":"2024-06-29T15:57:36.329358+0000","last_became_active":"2024-06-29T15:57:35.249561+0000","last_became_peered":"2024-06-29T15:57:35.249561+0000","last_unstale":"2024-06-29T15:57:36.329358+0000","last_undegraded":"2024-06-29T15:57:36.329358+0000","last_fullsized":"2024-06-29T15:57:36.329358+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:58.917763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239963+0000","last_change":"2024-06-29T15:57:22.224310+0000","last_active":"2024-06-29T15:57:36.239963+0000","last_peered":"2024-06-29T15:57:36.239963+0000","last_clean":"2024-06-29T15:57:36.239963+0000","last_became_active":"2024-06-29T15:57:22.223959+0000","last_became_peered":"2024-06-29T15:57:22.223959+0000","last_unstale":"2024-06-29T15:57:36.239963+0000","last_undegraded":"2024-06-29T15:57:36.239963+0000","last_fullsized":"2024-06-29T15:57:36.239963+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:13.811309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332185+0000","last_change":"2024-06-29T15:57:35.258233+0000","last_active":"2024-06-29T15:57:36.332185+0000","last_peered":"2024-06-29T15:57:36.332185+0000","last_clean":"2024-06-29T15:57:36.332185+0000","last_became_active":"2024-06-29T15:57:35.256283+0000","last_became_peered":"2024-06-29T15:57:35.256283+0000","last_unstale":"2024-06-29T15:57:36.332185+0000","last_undegraded":"2024-06-29T15:57:36.332185+0000","last_fullsized":"2024-06-29T15:57:36.332185+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:49.167991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239662+0000","last_change":"2024-06-29T15:57:22.215484+0000","last_active":"2024-06-29T15:57:36.239662+0000","last_peered":"2024-06-29T15:57:36.239662+0000","last_clean":"2024-06-29T15:57:36.239662+0000","last_became_active":"2024-06-29T15:57:22.215179+0000","last_became_peered":"2024-06-29T15:57:22.215179+0000","last_unstale":"2024-06-29T15:57:36.239662+0000","last_undegraded":"2024-06-29T15:57:36.239662+0000","last_fullsized":"2024-06-29T15:57:36.239662+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:29.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239331+0000","last_change":"2024-06-29T15:57:35.258740+0000","last_active":"2024-06-29T15:57:36.239331+0000","last_peered":"2024-06-29T15:57:36.239331+0000","last_clean":"2024-06-29T15:57:36.239331+0000","last_became_active":"2024-06-29T15:57:35.258400+0000","last_became_peered":"2024-06-29T15:57:35.258400+0000","last_unstale":"2024-06-29T15:57:36.239331+0000","last_undegraded":"2024-06-29T15:57:36.239331+0000","last_fullsized":"2024-06-29T15:57:36.239331+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:33.378531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238877+0000","last_change":"2024-06-29T15:57:22.237289+0000","last_active":"2024-06-29T15:57:36.238877+0000","last_peered":"2024-06-29T15:57:36.238877+0000","last_clean":"2024-06-29T15:57:36.238877+0000","last_became_active":"2024-06-29T15:57:22.236267+0000","last_became_peered":"2024-06-29T15:57:22.236267+0000","last_unstale":"2024-06-29T15:57:36.238877+0000","last_undegraded":"2024-06-29T15:57:36.238877+0000","last_fullsized":"2024-06-29T15:57:36.238877+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:53:11.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240048+0000","last_change":"2024-06-29T15:57:35.243220+0000","last_active":"2024-06-29T15:57:36.240048+0000","last_peered":"2024-06-29T15:57:36.240048+0000","last_clean":"2024-06-29T15:57:36.240048+0000","last_became_active":"2024-06-29T15:57:35.242914+0000","last_became_peered":"2024-06-29T15:57:35.242914+0000","last_unstale":"2024-06-29T15:57:36.240048+0000","last_undegraded":"2024-06-29T15:57:36.240048+0000","last_fullsized":"2024-06-29T15:57:36.240048+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:16.068033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330080+0000","last_change":"2024-06-29T15:57:22.239205+0000","last_active":"2024-06-29T15:57:36.330080+0000","last_peered":"2024-06-29T15:57:36.330080+0000","last_clean":"2024-06-29T15:57:36.330080+0000","last_became_active":"2024-06-29T15:57:22.238830+0000","last_became_peered":"2024-06-29T15:57:22.238830+0000","last_unstale":"2024-06-29T15:57:36.330080+0000","last_undegraded":"2024-06-29T15:57:36.330080+0000","last_fullsized":"2024-06-29T15:57:36.330080+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.228440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312824+0000","last_change":"2024-06-29T15:57:35.256144+0000","last_active":"2024-06-29T15:57:36.312824+0000","last_peered":"2024-06-29T15:57:36.312824+0000","last_clean":"2024-06-29T15:57:36.312824+0000","last_became_active":"2024-06-29T15:57:35.255541+0000","last_became_peered":"2024-06-29T15:57:35.255541+0000","last_unstale":"2024-06-29T15:57:36.312824+0000","last_undegraded":"2024-06-29T15:57:36.312824+0000","last_fullsized":"2024-06-29T15:57:36.312824+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:23:28.979863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330285+0000","last_change":"2024-06-29T15:57:22.235020+0000","last_active":"2024-06-29T15:57:36.330285+0000","last_peered":"2024-06-29T15:57:36.330285+0000","last_clean":"2024-06-29T15:57:36.330285+0000","last_became_active":"2024-06-29T15:57:22.234417+0000","last_became_peered":"2024-06-29T15:57:22.234417+0000","last_unstale":"2024-06-29T15:57:36.330285+0000","last_undegraded":"2024-06-29T15:57:36.330285+0000","last_fullsized":"2024-06-29T15:57:36.330285+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:06.264167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237895+0000","last_change":"2024-06-29T15:57:35.249930+0000","last_active":"2024-06-29T15:57:36.237895+0000","last_peered":"2024-06-29T15:57:36.237895+0000","last_clean":"2024-06-29T15:57:36.237895+0000","last_became_active":"2024-06-29T15:57:35.249298+0000","last_became_peered":"2024-06-29T15:57:35.249298+0000","last_unstale":"2024-06-29T15:57:36.237895+0000","last_undegraded":"2024-06-29T15:57:36.237895+0000","last_fullsized":"2024-06-29T15:57:36.237895+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.903736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.235930+0000","last_change":"2024-06-29T15:57:22.238113+0000","last_active":"2024-06-29T15:57:36.235930+0000","last_peered":"2024-06-29T15:57:36.235930+0000","last_clean":"2024-06-29T15:57:36.235930+0000","last_became_active":"2024-06-29T15:57:22.237726+0000","last_became_peered":"2024-06-29T15:57:22.237726+0000","last_unstale":"2024-06-29T15:57:36.235930+0000","last_undegraded":"2024-06-29T15:57:36.235930+0000","last_fullsized":"2024-06-29T15:57:36.235930+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:08:48.416318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239184+0000","last_change":"2024-06-29T15:57:35.247761+0000","last_active":"2024-06-29T15:57:36.239184+0000","last_peered":"2024-06-29T15:57:36.239184+0000","last_clean":"2024-06-29T15:57:36.239184+0000","last_became_active":"2024-06-29T15:57:35.247325+0000","last_became_peered":"2024-06-29T15:57:35.247325+0000","last_unstale":"2024-06-29T15:57:36.239184+0000","last_undegraded":"2024-06-29T15:57:36.239184+0000","last_fullsized":"2024-06-29T15:57:36.239184+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:55:34.536510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329410+0000","last_change":"2024-06-29T15:57:22.220754+0000","last_active":"2024-06-29T15:57:36.329410+0000","last_peered":"2024-06-29T15:57:36.329410+0000","last_clean":"2024-06-29T15:57:36.329410+0000","last_became_active":"2024-06-29T15:57:22.220438+0000","last_became_peered":"2024-06-29T15:57:22.220438+0000","last_unstale":"2024-06-29T15:57:36.329410+0000","last_undegraded":"2024-06-29T15:57:36.329410+0000","last_fullsized":"2024-06-29T15:57:36.329410+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:26.314927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237379+0000","last_change":"2024-06-29T15:57:35.241980+0000","last_active":"2024-06-29T15:57:36.237379+0000","last_peered":"2024-06-29T15:57:36.237379+0000","last_clean":"2024-06-29T15:57:36.237379+0000","last_became_active":"2024-06-29T15:57:35.241607+0000","last_became_peered":"2024-06-29T15:57:35.241607+0000","last_unstale":"2024-06-29T15:57:36.237379+0000","last_undegraded":"2024-06-29T15:57:36.237379+0000","last_fullsized":"2024-06-29T15:57:36.237379+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:51.022053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238091+0000","last_change":"2024-06-29T15:57:22.225596+0000","last_active":"2024-06-29T15:57:36.238091+0000","last_peered":"2024-06-29T15:57:36.238091+0000","last_clean":"2024-06-29T15:57:36.238091+0000","last_became_active":"2024-06-29T15:57:22.225154+0000","last_became_peered":"2024-06-29T15:57:22.225154+0000","last_unstale":"2024-06-29T15:57:36.238091+0000","last_undegraded":"2024-06-29T15:57:36.238091+0000","last_fullsized":"2024-06-29T15:57:36.238091+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:01.272853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236100+0000","last_change":"2024-06-29T15:57:35.246784+0000","last_active":"2024-06-29T15:57:36.236100+0000","last_peered":"2024-06-29T15:57:36.236100+0000","last_clean":"2024-06-29T15:57:36.236100+0000","last_became_active":"2024-06-29T15:57:35.246467+0000","last_became_peered":"2024-06-29T15:57:35.246467+0000","last_unstale":"2024-06-29T15:57:36.236100+0000","last_undegraded":"2024-06-29T15:57:36.236100+0000","last_fullsized":"2024-06-29T15:57:36.236100+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:46.600461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311897+0000","last_change":"2024-06-29T15:57:22.237433+0000","last_active":"2024-06-29T15:57:36.311897+0000","last_peered":"2024-06-29T15:57:36.311897+0000","last_clean":"2024-06-29T15:57:36.311897+0000","last_became_active":"2024-06-29T15:57:22.237070+0000","last_became_peered":"2024-06-29T15:57:22.237070+0000","last_unstale":"2024-06-29T15:57:36.311897+0000","last_undegraded":"2024-06-29T15:57:36.311897+0000","last_fullsized":"2024-06-29T15:57:36.311897+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:25.658901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332066+0000","last_change":"2024-06-29T15:57:35.251241+0000","last_active":"2024-06-29T15:57:36.332066+0000","last_peered":"2024-06-29T15:57:36.332066+0000","last_clean":"2024-06-29T15:57:36.332066+0000","last_became_active":"2024-06-29T15:57:35.248566+0000","last_became_peered":"2024-06-29T15:57:35.248566+0000","last_unstale":"2024-06-29T15:57:36.332066+0000","last_undegraded":"2024-06-29T15:57:36.332066+0000","last_fullsized":"2024-06-29T15:57:36.332066+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:52.934381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330550+0000","last_change":"2024-06-29T15:57:22.231838+0000","last_active":"2024-06-29T15:57:36.330550+0000","last_peered":"2024-06-29T15:57:36.330550+0000","last_clean":"2024-06-29T15:57:36.330550+0000","last_became_active":"2024-06-29T15:57:22.231563+0000","last_became_peered":"2024-06-29T15:57:22.231563+0000","last_unstale":"2024-06-29T15:57:36.330550+0000","last_undegraded":"2024-06-29T15:57:36.330550+0000","last_fullsized":"2024-06-29T15:57:36.330550+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:23.251139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330914+0000","last_change":"2024-06-29T15:57:35.251166+0000","last_active":"2024-06-29T15:57:36.330914+0000","last_peered":"2024-06-29T15:57:36.330914+0000","last_clean":"2024-06-29T15:57:36.330914+0000","last_became_active":"2024-06-29T15:57:35.249802+0000","last_became_peered":"2024-06-29T15:57:35.249802+0000","last_unstale":"2024-06-29T15:57:36.330914+0000","last_undegraded":"2024-06-29T15:57:36.330914+0000","last_fullsized":"2024-06-29T15:57:36.330914+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:26.856925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237097+0000","last_change":"2024-06-29T15:57:22.237225+0000","last_active":"2024-06-29T15:57:36.237097+0000","last_peered":"2024-06-29T15:57:36.237097+0000","last_clean":"2024-06-29T15:57:36.237097+0000","last_became_active":"2024-06-29T15:57:22.236621+0000","last_became_peered":"2024-06-29T15:57:22.236621+0000","last_unstale":"2024-06-29T15:57:36.237097+0000","last_undegraded":"2024-06-29T15:57:36.237097+0000","last_fullsized":"2024-06-29T15:57:36.237097+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:26.582705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238326+0000","last_change":"2024-06-29T15:57:35.252791+0000","last_active":"2024-06-29T15:57:36.238326+0000","last_peered":"2024-06-29T15:57:36.238326+0000","last_clean":"2024-06-29T15:57:36.238326+0000","last_became_active":"2024-06-29T15:57:35.252002+0000","last_became_peered":"2024-06-29T15:57:35.252002+0000","last_unstale":"2024-06-29T15:57:36.238326+0000","last_undegraded":"2024-06-29T15:57:36.238326+0000","last_fullsized":"2024-06-29T15:57:36.238326+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:32.593283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238829+0000","last_change":"2024-06-29T15:57:22.237316+0000","last_active":"2024-06-29T15:57:36.238829+0000","last_peered":"2024-06-29T15:57:36.238829+0000","last_clean":"2024-06-29T15:57:36.238829+0000","last_became_active":"2024-06-29T15:57:22.236756+0000","last_became_peered":"2024-06-29T15:57:22.236756+0000","last_unstale":"2024-06-29T15:57:36.238829+0000","last_undegraded":"2024-06-29T15:57:36.238829+0000","last_fullsized":"2024-06-29T15:57:36.238829+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.141583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329947+0000","last_change":"2024-06-29T15:57:35.254538+0000","last_active":"2024-06-29T15:57:36.329947+0000","last_peered":"2024-06-29T15:57:36.329947+0000","last_clean":"2024-06-29T15:57:36.329947+0000","last_became_active":"2024-06-29T15:57:35.254193+0000","last_became_peered":"2024-06-29T15:57:35.254193+0000","last_unstale":"2024-06-29T15:57:36.329947+0000","last_undegraded":"2024-06-29T15:57:36.329947+0000","last_fullsized":"2024-06-29T15:57:36.329947+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:23.012081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330101+0000","last_change":"2024-06-29T15:57:22.235210+0000","last_active":"2024-06-29T15:57:36.330101+0000","last_peered":"2024-06-29T15:57:36.330101+0000","last_clean":"2024-06-29T15:57:36.330101+0000","last_became_active":"2024-06-29T15:57:22.234573+0000","last_became_peered":"2024-06-29T15:57:22.234573+0000","last_unstale":"2024-06-29T15:57:36.330101+0000","last_undegraded":"2024-06-29T15:57:36.330101+0000","last_fullsized":"2024-06-29T15:57:36.330101+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:48.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239771+0000","last_change":"2024-06-29T15:57:35.252893+0000","last_active":"2024-06-29T15:57:36.239771+0000","last_peered":"2024-06-29T15:57:36.239771+0000","last_clean":"2024-06-29T15:57:36.239771+0000","last_became_active":"2024-06-29T15:57:35.252527+0000","last_became_peered":"2024-06-29T15:57:35.252527+0000","last_unstale":"2024-06-29T15:57:36.239771+0000","last_undegraded":"2024-06-29T15:57:36.239771+0000","last_fullsized":"2024-06-29T15:57:36.239771+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:02.911105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239077+0000","last_change":"2024-06-29T15:57:22.233275+0000","last_active":"2024-06-29T15:57:36.239077+0000","last_peered":"2024-06-29T15:57:36.239077+0000","last_clean":"2024-06-29T15:57:36.239077+0000","last_became_active":"2024-06-29T15:57:22.230801+0000","last_became_peered":"2024-06-29T15:57:22.230801+0000","last_unstale":"2024-06-29T15:57:36.239077+0000","last_undegraded":"2024-06-29T15:57:36.239077+0000","last_fullsized":"2024-06-29T15:57:36.239077+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:24.134069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311790+0000","last_change":"2024-06-29T15:57:35.257258+0000","last_active":"2024-06-29T15:57:36.311790+0000","last_peered":"2024-06-29T15:57:36.311790+0000","last_clean":"2024-06-29T15:57:36.311790+0000","last_became_active":"2024-06-29T15:57:35.256939+0000","last_became_peered":"2024-06-29T15:57:35.256939+0000","last_unstale":"2024-06-29T15:57:36.311790+0000","last_undegraded":"2024-06-29T15:57:36.311790+0000","last_fullsized":"2024-06-29T15:57:36.311790+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:40:53.758735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329683+0000","last_change":"2024-06-29T15:57:22.223324+0000","last_active":"2024-06-29T15:57:36.329683+0000","last_peered":"2024-06-29T15:57:36.329683+0000","last_clean":"2024-06-29T15:57:36.329683+0000","last_became_active":"2024-06-29T15:57:22.222958+0000","last_became_peered":"2024-06-29T15:57:22.222958+0000","last_unstale":"2024-06-29T15:57:36.329683+0000","last_undegraded":"2024-06-29T15:57:36.329683+0000","last_fullsized":"2024-06-29T15:57:36.329683+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:21.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237061+0000","last_change":"2024-06-29T15:57:35.244095+0000","last_active":"2024-06-29T15:57:36.237061+0000","last_peered":"2024-06-29T15:57:36.237061+0000","last_clean":"2024-06-29T15:57:36.237061+0000","last_became_active":"2024-06-29T15:57:35.243771+0000","last_became_peered":"2024-06-29T15:57:35.243771+0000","last_unstale":"2024-06-29T15:57:36.237061+0000","last_undegraded":"2024-06-29T15:57:36.237061+0000","last_fullsized":"2024-06-29T15:57:36.237061+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:55.022325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312052+0000","last_change":"2024-06-29T15:57:22.226496+0000","last_active":"2024-06-29T15:57:36.312052+0000","last_peered":"2024-06-29T15:57:36.312052+0000","last_clean":"2024-06-29T15:57:36.312052+0000","last_became_active":"2024-06-29T15:57:22.226191+0000","last_became_peered":"2024-06-29T15:57:22.226191+0000","last_unstale":"2024-06-29T15:57:36.312052+0000","last_undegraded":"2024-06-29T15:57:36.312052+0000","last_fullsized":"2024-06-29T15:57:36.312052+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:32.644449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330874+0000","last_change":"2024-06-29T15:57:35.258649+0000","last_active":"2024-06-29T15:57:36.330874+0000","last_peered":"2024-06-29T15:57:36.330874+0000","last_clean":"2024-06-29T15:57:36.330874+0000","last_became_active":"2024-06-29T15:57:35.258325+0000","last_became_peered":"2024-06-29T15:57:35.258325+0000","last_unstale":"2024-06-29T15:57:36.330874+0000","last_undegraded":"2024-06-29T15:57:36.330874+0000","last_fullsized":"2024-06-29T15:57:36.330874+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:52.146056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331112+0000","last_change":"2024-06-29T15:57:22.218369+0000","last_active":"2024-06-29T15:57:36.331112+0000","last_peered":"2024-06-29T15:57:36.331112+0000","last_clean":"2024-06-29T15:57:36.331112+0000","last_became_active":"2024-06-29T15:57:22.217584+0000","last_became_peered":"2024-06-29T15:57:22.217584+0000","last_unstale":"2024-06-29T15:57:36.331112+0000","last_undegraded":"2024-06-29T15:57:36.331112+0000","last_fullsized":"2024-06-29T15:57:36.331112+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:59.760761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330020+0000","last_change":"2024-06-29T15:57:35.250646+0000","last_active":"2024-06-29T15:57:36.330020+0000","last_peered":"2024-06-29T15:57:36.330020+0000","last_clean":"2024-06-29T15:57:36.330020+0000","last_became_active":"2024-06-29T15:57:35.250307+0000","last_became_peered":"2024-06-29T15:57:35.250307+0000","last_unstale":"2024-06-29T15:57:36.330020+0000","last_undegraded":"2024-06-29T15:57:36.330020+0000","last_fullsized":"2024-06-29T15:57:36.330020+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:06.391987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312266+0000","last_change":"2024-06-29T15:57:22.236022+0000","last_active":"2024-06-29T15:57:36.312266+0000","last_peered":"2024-06-29T15:57:36.312266+0000","last_clean":"2024-06-29T15:57:36.312266+0000","last_became_active":"2024-06-29T15:57:22.235712+0000","last_became_peered":"2024-06-29T15:57:22.235712+0000","last_unstale":"2024-06-29T15:57:36.312266+0000","last_undegraded":"2024-06-29T15:57:36.312266+0000","last_fullsized":"2024-06-29T15:57:36.312266+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:57.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312823+0000","last_change":"2024-06-29T15:57:35.256845+0000","last_active":"2024-06-29T15:57:36.312823+0000","last_peered":"2024-06-29T15:57:36.312823+0000","last_clean":"2024-06-29T15:57:36.312823+0000","last_became_active":"2024-06-29T15:57:35.256516+0000","last_became_peered":"2024-06-29T15:57:35.256516+0000","last_unstale":"2024-06-29T15:57:36.312823+0000","last_undegraded":"2024-06-29T15:57:36.312823+0000","last_fullsized":"2024-06-29T15:57:36.312823+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.581911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237077+0000","last_change":"2024-06-29T15:57:22.231073+0000","last_active":"2024-06-29T15:57:36.237077+0000","last_peered":"2024-06-29T15:57:36.237077+0000","last_clean":"2024-06-29T15:57:36.237077+0000","last_became_active":"2024-06-29T15:57:22.230749+0000","last_became_peered":"2024-06-29T15:57:22.230749+0000","last_unstale":"2024-06-29T15:57:36.237077+0000","last_undegraded":"2024-06-29T15:57:36.237077+0000","last_fullsized":"2024-06-29T15:57:36.237077+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:23.168668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329839+0000","last_change":"2024-06-29T15:57:35.252089+0000","last_active":"2024-06-29T15:57:36.329839+0000","last_peered":"2024-06-29T15:57:36.329839+0000","last_clean":"2024-06-29T15:57:36.329839+0000","last_became_active":"2024-06-29T15:57:35.251775+0000","last_became_peered":"2024-06-29T15:57:35.251775+0000","last_unstale":"2024-06-29T15:57:36.329839+0000","last_undegraded":"2024-06-29T15:57:36.329839+0000","last_fullsized":"2024-06-29T15:57:36.329839+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:26.905348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332427+0000","last_change":"2024-06-29T15:57:22.231733+0000","last_active":"2024-06-29T15:57:36.332427+0000","last_peered":"2024-06-29T15:57:36.332427+0000","last_clean":"2024-06-29T15:57:36.332427+0000","last_became_active":"2024-06-29T15:57:22.231387+0000","last_became_peered":"2024-06-29T15:57:22.231387+0000","last_unstale":"2024-06-29T15:57:36.332427+0000","last_undegraded":"2024-06-29T15:57:36.332427+0000","last_fullsized":"2024-06-29T15:57:36.332427+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:41.892974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329977+0000","last_change":"2024-06-29T15:57:35.255683+0000","last_active":"2024-06-29T15:57:36.329977+0000","last_peered":"2024-06-29T15:57:36.329977+0000","last_clean":"2024-06-29T15:57:36.329977+0000","last_became_active":"2024-06-29T15:57:35.255328+0000","last_became_peered":"2024-06-29T15:57:35.255328+0000","last_unstale":"2024-06-29T15:57:36.329977+0000","last_undegraded":"2024-06-29T15:57:36.329977+0000","last_fullsized":"2024-06-29T15:57:36.329977+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:35.404663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239593+0000","last_change":"2024-06-29T15:57:22.225592+0000","last_active":"2024-06-29T15:57:36.239593+0000","last_peered":"2024-06-29T15:57:36.239593+0000","last_clean":"2024-06-29T15:57:36.239593+0000","last_became_active":"2024-06-29T15:57:22.225309+0000","last_became_peered":"2024-06-29T15:57:22.225309+0000","last_unstale":"2024-06-29T15:57:36.239593+0000","last_undegraded":"2024-06-29T15:57:36.239593+0000","last_fullsized":"2024-06-29T15:57:36.239593+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:07.857349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543290,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52596,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319244,"statfs":{"total":96636764160,"available":96582905856,"internally_reserved":0,"allocated":3448832,"data_stored":2054220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543291,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42312,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329528,"statfs":{"total":96636764160,"available":96593436672,"internally_reserved":0,"allocated":2871296,"data_stored":1465430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543290,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47864,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323976,"statfs":{"total":96636764160,"available":96587751424,"internally_reserved":0,"allocated":2854912,"data_stored":1461993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543289,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52004,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319836,"statfs":{"total":96636764160,"available":96583512064,"internally_reserved":0,"allocated":2838528,"data_stored":1460927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510587,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47856,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323984,"statfs":{"total":96636764160,"available":96587759616,"internally_reserved":0,"allocated":2846720,"data_stored":1460403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47844,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323996,"statfs":{"total":96636764160,"available":96587771904,"internally_reserved":0,"allocated":2842624,"data_stored":1460961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510586,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47832,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324008,"statfs":{"total":96636764160,"available":96587784192,"internally_reserved":0,"allocated":2830336,"data_stored":1460255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510588,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42852,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328988,"statfs":{"total":96636764160,"available":96592883712,"internally_reserved":0,"allocated":3428352,"data_stored":2050713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:43.156 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:57:43.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:57:43.964 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.096+0000 7f9ca15b2640 1 -- 172.21.15.59:0/264519110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 msgr2=0x7f9c84004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.096+0000 7f9ca15b2640 1 --2- 172.21.15.59:0/264519110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 0x7f9c84004c10 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f9c900016c0 tx=0x7f9c84005830 comp rx=0 tx=0).stop 2024-06-29T15:57:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.096+0000 7f9ca15b2640 1 -- 172.21.15.59:0/264519110 shutdown_connections 2024-06-29T15:57:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.096+0000 7f9ca15b2640 1 --2- 172.21.15.59:0/264519110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 0x7f9c84004c10 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.096+0000 7f9ca15b2640 1 -- 172.21.15.59:0/264519110 >> 172.21.15.59:0/264519110 conn(0x7f9c9c0d45b0 msgr2=0x7f9c9c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.096+0000 7f9ca15b2640 1 -- 172.21.15.59:0/264519110 shutdown_connections 2024-06-29T15:57:44.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.096+0000 7f9ca15b2640 1 -- 172.21.15.59:0/264519110 wait complete. 2024-06-29T15:57:44.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.098+0000 7f9ca15b2640 1 Processor -- start 2024-06-29T15:57:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.098+0000 7f9ca15b2640 1 -- start start 2024-06-29T15:57:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9ca15b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 0x7f9c9c0b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9ca15b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c9c0ccd40 0x7f9c9c15af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9ca15b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c9c1f7c90 0x7f9c9c1f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9ca15b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c161a80 con 0x7f9c84004800 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9ca15b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c0ce7c0 con 0x7f9c9c0ccd40 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9ca15b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9c161c00 con 0x7f9c9c1f7c90 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9c9affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 0x7f9c9c0b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9c9affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 0x7f9c9c0b22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54584/0 (socket says 172.21.15.59:54584) 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.099+0000 7f9c9affd640 1 -- 172.21.15.59:0/3398796845 learned_addr learned my addr 172.21.15.59:0/3398796845 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9b7fe640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c9c1f7c90 0x7f9c9c1f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9a7fc640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c9c0ccd40 0x7f9c9c15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9affd640 1 -- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c9c1f7c90 msgr2=0x7f9c9c1f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9affd640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c9c1f7c90 0x7f9c9c1f80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9affd640 1 -- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c9c0ccd40 msgr2=0x7f9c9c15af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9affd640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c9c0ccd40 0x7f9c9c15af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9affd640 1 -- 172.21.15.59:0/3398796845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9c1f85e0 con 0x7f9c84004800 2024-06-29T15:57:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9a7fc640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c9c0ccd40 0x7f9c9c15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c9affd640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 0x7f9c9c0b22f0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c006600 tx=0x7f9c8c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c837fe640 1 -- 172.21.15.59:0/3398796845 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c015070 con 0x7f9c84004800 2024-06-29T15:57:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c837fe640 1 -- 172.21.15.59:0/3398796845 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c8c011920 con 0x7f9c84004800 2024-06-29T15:57:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c9c0cd420 con 0x7f9c84004800 2024-06-29T15:57:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9c837fe640 1 -- 172.21.15.59:0/3398796845 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c011d70 con 0x7f9c84004800 2024-06-29T15:57:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.100+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9c0cd9e0 con 0x7f9c84004800 2024-06-29T15:57:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.101+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c9c0cdc20 con 0x7f9c84004800 2024-06-29T15:57:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.102+0000 7f9c837fe640 1 -- 172.21.15.59:0/3398796845 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f9c8c018070 con 0x7f9c84004800 2024-06-29T15:57:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.103+0000 7f9c837fe640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9c880763f0 0x7f9c88078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.103+0000 7f9c837fe640 1 -- 172.21.15.59:0/3398796845 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f9c8c098b70 con 0x7f9c84004800 2024-06-29T15:57:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.103+0000 7f9c9a7fc640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9c880763f0 0x7f9c88078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.103+0000 7f9c9a7fc640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9c880763f0 0x7f9c88078900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c90032000 tx=0x7f9c90001190 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.107+0000 7f9c837fe640 1 -- 172.21.15.59:0/3398796845 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9c9c1f85e0 con 0x7f9c84004800 2024-06-29T15:57:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.297+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9c9c162160 con 0x7f9c880763f0 2024-06-29T15:57:44.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.306+0000 7f9c837fe640 1 -- 172.21.15.59:0/3398796845 <== mgr.9206 v2:172.21.15.153:6832/2780049427 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491363 (secure 0 0 0) 0x7f9c900baf80 con 0x7f9c880763f0 2024-06-29T15:57:44.311 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:44.314 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:57:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.309+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9c880763f0 msgr2=0x7f9c88078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.309+0000 7f9ca15b2640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9c880763f0 0x7f9c88078900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c90032000 tx=0x7f9c90001190 comp rx=0 tx=0).stop 2024-06-29T15:57:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.309+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 msgr2=0x7f9c9c0b22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.309+0000 7f9ca15b2640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 0x7f9c9c0b22f0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c006600 tx=0x7f9c8c0039c0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.309+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 shutdown_connections 2024-06-29T15:57:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.309+0000 7f9ca15b2640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f9c880763f0 0x7f9c88078900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.310+0000 7f9ca15b2640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9c9c1f7c90 0x7f9c9c1f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.310+0000 7f9ca15b2640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c9c0ccd40 0x7f9c9c15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.310+0000 7f9ca15b2640 1 --2- 172.21.15.59:0/3398796845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c84004800 0x7f9c9c0b22f0 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.310+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 >> 172.21.15.59:0/3398796845 conn(0x7f9c9c0d45b0 msgr2=0x7f9c9c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.310+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 shutdown_connections 2024-06-29T15:57:44.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.310+0000 7f9ca15b2640 1 -- 172.21.15.59:0/3398796845 wait complete. 2024-06-29T15:57:44.322 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-06-29T15:57:43.770900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":469,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381160,"kb_used_data":23400,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754593560,"statfs":{"total":773094113280,"available":772703805440,"internally_reserved":0,"allocated":23961600,"data_stored":12874902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":366202418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.538214"},"pg_stats":[{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331581+0000","last_change":"2024-06-29T15:57:35.257416+0000","last_active":"2024-06-29T15:57:36.331581+0000","last_peered":"2024-06-29T15:57:36.331581+0000","last_clean":"2024-06-29T15:57:36.331581+0000","last_became_active":"2024-06-29T15:57:35.256867+0000","last_became_peered":"2024-06-29T15:57:35.256867+0000","last_unstale":"2024-06-29T15:57:36.331581+0000","last_undegraded":"2024-06-29T15:57:36.331581+0000","last_fullsized":"2024-06-29T15:57:36.331581+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:30:33.761276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"210'233","reported_seq":1109,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311738+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:57:36.311738+0000","last_peered":"2024-06-29T15:57:36.311738+0000","last_clean":"2024-06-29T15:57:36.311738+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:57:36.311738+0000","last_undegraded":"2024-06-29T15:57:36.311738+0000","last_fullsized":"2024-06-29T15:57:36.311738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331626+0000","last_change":"2024-06-29T15:57:21.209163+0000","last_active":"2024-06-29T15:57:36.331626+0000","last_peered":"2024-06-29T15:57:36.331626+0000","last_clean":"2024-06-29T15:57:36.331626+0000","last_became_active":"2024-06-29T15:57:21.208825+0000","last_became_peered":"2024-06-29T15:57:21.208825+0000","last_unstale":"2024-06-29T15:57:36.331626+0000","last_undegraded":"2024-06-29T15:57:36.331626+0000","last_fullsized":"2024-06-29T15:57:36.331626+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:13.641268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237943+0000","last_change":"2024-06-29T15:57:22.230794+0000","last_active":"2024-06-29T15:57:36.237943+0000","last_peered":"2024-06-29T15:57:36.237943+0000","last_clean":"2024-06-29T15:57:36.237943+0000","last_became_active":"2024-06-29T15:57:22.230474+0000","last_became_peered":"2024-06-29T15:57:22.230474+0000","last_unstale":"2024-06-29T15:57:36.237943+0000","last_undegraded":"2024-06-29T15:57:36.237943+0000","last_fullsized":"2024-06-29T15:57:36.237943+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:43.098132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312052+0000","last_change":"2024-06-29T15:57:34.312155+0000","last_active":"2024-06-29T15:57:36.312052+0000","last_peered":"2024-06-29T15:57:36.312052+0000","last_clean":"2024-06-29T15:57:36.312052+0000","last_became_active":"2024-06-29T15:57:34.311830+0000","last_became_peered":"2024-06-29T15:57:34.311830+0000","last_unstale":"2024-06-29T15:57:36.312052+0000","last_undegraded":"2024-06-29T15:57:36.312052+0000","last_fullsized":"2024-06-29T15:57:36.312052+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:58:13.468981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312504+0000","last_change":"2024-06-29T15:57:35.255842+0000","last_active":"2024-06-29T15:57:36.312504+0000","last_peered":"2024-06-29T15:57:36.312504+0000","last_clean":"2024-06-29T15:57:36.312504+0000","last_became_active":"2024-06-29T15:57:35.255195+0000","last_became_peered":"2024-06-29T15:57:35.255195+0000","last_unstale":"2024-06-29T15:57:36.312504+0000","last_undegraded":"2024-06-29T15:57:36.312504+0000","last_fullsized":"2024-06-29T15:57:36.312504+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:38.905359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331919+0000","last_change":"2024-06-29T15:57:21.215589+0000","last_active":"2024-06-29T15:57:36.331919+0000","last_peered":"2024-06-29T15:57:36.331919+0000","last_clean":"2024-06-29T15:57:36.331919+0000","last_became_active":"2024-06-29T15:57:21.213719+0000","last_became_peered":"2024-06-29T15:57:21.213719+0000","last_unstale":"2024-06-29T15:57:36.331919+0000","last_undegraded":"2024-06-29T15:57:36.331919+0000","last_fullsized":"2024-06-29T15:57:36.331919+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:20:32.413502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236935+0000","last_change":"2024-06-29T15:57:22.223950+0000","last_active":"2024-06-29T15:57:36.236935+0000","last_peered":"2024-06-29T15:57:36.236935+0000","last_clean":"2024-06-29T15:57:36.236935+0000","last_became_active":"2024-06-29T15:57:22.223622+0000","last_became_peered":"2024-06-29T15:57:22.223622+0000","last_unstale":"2024-06-29T15:57:36.236935+0000","last_undegraded":"2024-06-29T15:57:36.236935+0000","last_fullsized":"2024-06-29T15:57:36.236935+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.556171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"226'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236795+0000","last_change":"2024-06-29T15:57:34.307024+0000","last_active":"2024-06-29T15:57:36.236795+0000","last_peered":"2024-06-29T15:57:36.236795+0000","last_clean":"2024-06-29T15:57:36.236795+0000","last_became_active":"2024-06-29T15:57:34.306637+0000","last_became_peered":"2024-06-29T15:57:34.306637+0000","last_unstale":"2024-06-29T15:57:36.236795+0000","last_undegraded":"2024-06-29T15:57:36.236795+0000","last_fullsized":"2024-06-29T15:57:36.236795+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:30.362344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329816+0000","last_change":"2024-06-29T15:57:35.254053+0000","last_active":"2024-06-29T15:57:36.329816+0000","last_peered":"2024-06-29T15:57:36.329816+0000","last_clean":"2024-06-29T15:57:36.329816+0000","last_became_active":"2024-06-29T15:57:35.253616+0000","last_became_peered":"2024-06-29T15:57:35.253616+0000","last_unstale":"2024-06-29T15:57:36.329816+0000","last_undegraded":"2024-06-29T15:57:36.329816+0000","last_fullsized":"2024-06-29T15:57:36.329816+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:13.830138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238919+0000","last_change":"2024-06-29T15:57:21.212837+0000","last_active":"2024-06-29T15:57:36.238919+0000","last_peered":"2024-06-29T15:57:36.238919+0000","last_clean":"2024-06-29T15:57:36.238919+0000","last_became_active":"2024-06-29T15:57:21.212491+0000","last_became_peered":"2024-06-29T15:57:21.212491+0000","last_unstale":"2024-06-29T15:57:36.238919+0000","last_undegraded":"2024-06-29T15:57:36.238919+0000","last_fullsized":"2024-06-29T15:57:36.238919+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:35:53.641775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312497+0000","last_change":"2024-06-29T15:57:22.237722+0000","last_active":"2024-06-29T15:57:36.312497+0000","last_peered":"2024-06-29T15:57:36.312497+0000","last_clean":"2024-06-29T15:57:36.312497+0000","last_became_active":"2024-06-29T15:57:22.237372+0000","last_became_peered":"2024-06-29T15:57:22.237372+0000","last_unstale":"2024-06-29T15:57:36.312497+0000","last_undegraded":"2024-06-29T15:57:36.312497+0000","last_fullsized":"2024-06-29T15:57:36.312497+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:19:24.651383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239300+0000","last_change":"2024-06-29T15:57:34.309853+0000","last_active":"2024-06-29T15:57:36.239300+0000","last_peered":"2024-06-29T15:57:36.239300+0000","last_clean":"2024-06-29T15:57:36.239300+0000","last_became_active":"2024-06-29T15:57:34.309541+0000","last_became_peered":"2024-06-29T15:57:34.309541+0000","last_unstale":"2024-06-29T15:57:36.239300+0000","last_undegraded":"2024-06-29T15:57:36.239300+0000","last_fullsized":"2024-06-29T15:57:36.239300+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:17:45.264114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331328+0000","last_change":"2024-06-29T15:57:35.255572+0000","last_active":"2024-06-29T15:57:36.331328+0000","last_peered":"2024-06-29T15:57:36.331328+0000","last_clean":"2024-06-29T15:57:36.331328+0000","last_became_active":"2024-06-29T15:57:35.255216+0000","last_became_peered":"2024-06-29T15:57:35.255216+0000","last_unstale":"2024-06-29T15:57:36.331328+0000","last_undegraded":"2024-06-29T15:57:36.331328+0000","last_fullsized":"2024-06-29T15:57:36.331328+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:59.819495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331738+0000","last_change":"2024-06-29T15:57:21.210563+0000","last_active":"2024-06-29T15:57:36.331738+0000","last_peered":"2024-06-29T15:57:36.331738+0000","last_clean":"2024-06-29T15:57:36.331738+0000","last_became_active":"2024-06-29T15:57:21.210243+0000","last_became_peered":"2024-06-29T15:57:21.210243+0000","last_unstale":"2024-06-29T15:57:36.331738+0000","last_undegraded":"2024-06-29T15:57:36.331738+0000","last_fullsized":"2024-06-29T15:57:36.331738+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.900240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312769+0000","last_change":"2024-06-29T15:57:22.244396+0000","last_active":"2024-06-29T15:57:36.312769+0000","last_peered":"2024-06-29T15:57:36.312769+0000","last_clean":"2024-06-29T15:57:36.312769+0000","last_became_active":"2024-06-29T15:57:22.244044+0000","last_became_peered":"2024-06-29T15:57:22.244044+0000","last_unstale":"2024-06-29T15:57:36.312769+0000","last_undegraded":"2024-06-29T15:57:36.312769+0000","last_fullsized":"2024-06-29T15:57:36.312769+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:07.981276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239376+0000","last_change":"2024-06-29T15:57:34.314135+0000","last_active":"2024-06-29T15:57:36.239376+0000","last_peered":"2024-06-29T15:57:36.239376+0000","last_clean":"2024-06-29T15:57:36.239376+0000","last_became_active":"2024-06-29T15:57:34.313741+0000","last_became_peered":"2024-06-29T15:57:34.313741+0000","last_unstale":"2024-06-29T15:57:36.239376+0000","last_undegraded":"2024-06-29T15:57:36.239376+0000","last_fullsized":"2024-06-29T15:57:36.239376+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:12:29.995937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238236+0000","last_change":"2024-06-29T15:57:35.251995+0000","last_active":"2024-06-29T15:57:36.238236+0000","last_peered":"2024-06-29T15:57:36.238236+0000","last_clean":"2024-06-29T15:57:36.238236+0000","last_became_active":"2024-06-29T15:57:35.251543+0000","last_became_peered":"2024-06-29T15:57:35.251543+0000","last_unstale":"2024-06-29T15:57:36.238236+0000","last_undegraded":"2024-06-29T15:57:36.238236+0000","last_fullsized":"2024-06-29T15:57:36.238236+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:30.799799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240227+0000","last_change":"2024-06-29T15:57:21.213113+0000","last_active":"2024-06-29T15:57:36.240227+0000","last_peered":"2024-06-29T15:57:36.240227+0000","last_clean":"2024-06-29T15:57:36.240227+0000","last_became_active":"2024-06-29T15:57:21.212819+0000","last_became_peered":"2024-06-29T15:57:21.212819+0000","last_unstale":"2024-06-29T15:57:36.240227+0000","last_undegraded":"2024-06-29T15:57:36.240227+0000","last_fullsized":"2024-06-29T15:57:36.240227+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:59.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239776+0000","last_change":"2024-06-29T15:57:22.229230+0000","last_active":"2024-06-29T15:57:36.239776+0000","last_peered":"2024-06-29T15:57:36.239776+0000","last_clean":"2024-06-29T15:57:36.239776+0000","last_became_active":"2024-06-29T15:57:22.228914+0000","last_became_peered":"2024-06-29T15:57:22.228914+0000","last_unstale":"2024-06-29T15:57:36.239776+0000","last_undegraded":"2024-06-29T15:57:36.239776+0000","last_fullsized":"2024-06-29T15:57:36.239776+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:09.918042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236907+0000","last_change":"2024-06-29T15:57:34.307100+0000","last_active":"2024-06-29T15:57:36.236907+0000","last_peered":"2024-06-29T15:57:36.236907+0000","last_clean":"2024-06-29T15:57:36.236907+0000","last_became_active":"2024-06-29T15:57:34.306718+0000","last_became_peered":"2024-06-29T15:57:34.306718+0000","last_unstale":"2024-06-29T15:57:36.236907+0000","last_undegraded":"2024-06-29T15:57:36.236907+0000","last_fullsized":"2024-06-29T15:57:36.236907+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:55.770944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331213+0000","last_change":"2024-06-29T15:57:35.249106+0000","last_active":"2024-06-29T15:57:36.331213+0000","last_peered":"2024-06-29T15:57:36.331213+0000","last_clean":"2024-06-29T15:57:36.331213+0000","last_became_active":"2024-06-29T15:57:35.248469+0000","last_became_peered":"2024-06-29T15:57:35.248469+0000","last_unstale":"2024-06-29T15:57:36.331213+0000","last_undegraded":"2024-06-29T15:57:36.331213+0000","last_fullsized":"2024-06-29T15:57:36.331213+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:21:47.954549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240435+0000","last_change":"2024-06-29T15:57:21.216399+0000","last_active":"2024-06-29T15:57:36.240435+0000","last_peered":"2024-06-29T15:57:36.240435+0000","last_clean":"2024-06-29T15:57:36.240435+0000","last_became_active":"2024-06-29T15:57:21.216104+0000","last_became_peered":"2024-06-29T15:57:21.216104+0000","last_unstale":"2024-06-29T15:57:36.240435+0000","last_undegraded":"2024-06-29T15:57:36.240435+0000","last_fullsized":"2024-06-29T15:57:36.240435+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.222061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239349+0000","last_change":"2024-06-29T15:57:22.227969+0000","last_active":"2024-06-29T15:57:36.239349+0000","last_peered":"2024-06-29T15:57:36.239349+0000","last_clean":"2024-06-29T15:57:36.239349+0000","last_became_active":"2024-06-29T15:57:22.227579+0000","last_became_peered":"2024-06-29T15:57:22.227579+0000","last_unstale":"2024-06-29T15:57:36.239349+0000","last_undegraded":"2024-06-29T15:57:36.239349+0000","last_fullsized":"2024-06-29T15:57:36.239349+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329273+0000","last_change":"2024-06-29T15:57:34.306550+0000","last_active":"2024-06-29T15:57:36.329273+0000","last_peered":"2024-06-29T15:57:36.329273+0000","last_clean":"2024-06-29T15:57:36.329273+0000","last_became_active":"2024-06-29T15:57:34.306222+0000","last_became_peered":"2024-06-29T15:57:34.306222+0000","last_unstale":"2024-06-29T15:57:36.329273+0000","last_undegraded":"2024-06-29T15:57:36.329273+0000","last_fullsized":"2024-06-29T15:57:36.329273+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:57.908667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330924+0000","last_change":"2024-06-29T15:57:35.257493+0000","last_active":"2024-06-29T15:57:36.330924+0000","last_peered":"2024-06-29T15:57:36.330924+0000","last_clean":"2024-06-29T15:57:36.330924+0000","last_became_active":"2024-06-29T15:57:35.256381+0000","last_became_peered":"2024-06-29T15:57:35.256381+0000","last_unstale":"2024-06-29T15:57:36.330924+0000","last_undegraded":"2024-06-29T15:57:36.330924+0000","last_fullsized":"2024-06-29T15:57:36.330924+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:13.192616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330024+0000","last_change":"2024-06-29T15:57:21.217039+0000","last_active":"2024-06-29T15:57:36.330024+0000","last_peered":"2024-06-29T15:57:36.330024+0000","last_clean":"2024-06-29T15:57:36.330024+0000","last_became_active":"2024-06-29T15:57:21.216542+0000","last_became_peered":"2024-06-29T15:57:21.216542+0000","last_unstale":"2024-06-29T15:57:36.330024+0000","last_undegraded":"2024-06-29T15:57:36.330024+0000","last_fullsized":"2024-06-29T15:57:36.330024+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:21.729257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331424+0000","last_change":"2024-06-29T15:57:22.233919+0000","last_active":"2024-06-29T15:57:36.331424+0000","last_peered":"2024-06-29T15:57:36.331424+0000","last_clean":"2024-06-29T15:57:36.331424+0000","last_became_active":"2024-06-29T15:57:22.231028+0000","last_became_peered":"2024-06-29T15:57:22.231028+0000","last_unstale":"2024-06-29T15:57:36.331424+0000","last_undegraded":"2024-06-29T15:57:36.331424+0000","last_fullsized":"2024-06-29T15:57:36.331424+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:34:24.910914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239552+0000","last_change":"2024-06-29T15:57:34.312110+0000","last_active":"2024-06-29T15:57:36.239552+0000","last_peered":"2024-06-29T15:57:36.239552+0000","last_clean":"2024-06-29T15:57:36.239552+0000","last_became_active":"2024-06-29T15:57:34.311789+0000","last_became_peered":"2024-06-29T15:57:34.311789+0000","last_unstale":"2024-06-29T15:57:36.239552+0000","last_undegraded":"2024-06-29T15:57:36.239552+0000","last_fullsized":"2024-06-29T15:57:36.239552+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:37.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239893+0000","last_change":"2024-06-29T15:57:35.256330+0000","last_active":"2024-06-29T15:57:36.239893+0000","last_peered":"2024-06-29T15:57:36.239893+0000","last_clean":"2024-06-29T15:57:36.239893+0000","last_became_active":"2024-06-29T15:57:35.256005+0000","last_became_peered":"2024-06-29T15:57:35.256005+0000","last_unstale":"2024-06-29T15:57:36.239893+0000","last_undegraded":"2024-06-29T15:57:36.239893+0000","last_fullsized":"2024-06-29T15:57:36.239893+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:03.579383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.6","version":"222'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238934+0000","last_change":"2024-06-29T15:57:21.203822+0000","last_active":"2024-06-29T15:57:36.238934+0000","last_peered":"2024-06-29T15:57:36.238934+0000","last_clean":"2024-06-29T15:57:36.238934+0000","last_became_active":"2024-06-29T15:57:21.203517+0000","last_became_peered":"2024-06-29T15:57:21.203517+0000","last_unstale":"2024-06-29T15:57:36.238934+0000","last_undegraded":"2024-06-29T15:57:36.238934+0000","last_fullsized":"2024-06-29T15:57:36.238934+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:05.810997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236238+0000","last_change":"2024-06-29T15:57:22.223140+0000","last_active":"2024-06-29T15:57:36.236238+0000","last_peered":"2024-06-29T15:57:36.236238+0000","last_clean":"2024-06-29T15:57:36.236238+0000","last_became_active":"2024-06-29T15:57:22.222839+0000","last_became_peered":"2024-06-29T15:57:22.222839+0000","last_unstale":"2024-06-29T15:57:36.236238+0000","last_undegraded":"2024-06-29T15:57:36.236238+0000","last_fullsized":"2024-06-29T15:57:36.236238+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:17.520475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238524+0000","last_change":"2024-06-29T15:57:34.308352+0000","last_active":"2024-06-29T15:57:36.238524+0000","last_peered":"2024-06-29T15:57:36.238524+0000","last_clean":"2024-06-29T15:57:36.238524+0000","last_became_active":"2024-06-29T15:57:34.308040+0000","last_became_peered":"2024-06-29T15:57:34.308040+0000","last_unstale":"2024-06-29T15:57:36.238524+0000","last_undegraded":"2024-06-29T15:57:36.238524+0000","last_fullsized":"2024-06-29T15:57:36.238524+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:34.472225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329855+0000","last_change":"2024-06-29T15:57:35.251788+0000","last_active":"2024-06-29T15:57:36.329855+0000","last_peered":"2024-06-29T15:57:36.329855+0000","last_clean":"2024-06-29T15:57:36.329855+0000","last_became_active":"2024-06-29T15:57:35.251451+0000","last_became_peered":"2024-06-29T15:57:35.251451+0000","last_unstale":"2024-06-29T15:57:36.329855+0000","last_undegraded":"2024-06-29T15:57:36.329855+0000","last_fullsized":"2024-06-29T15:57:36.329855+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:14.366892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330636+0000","last_change":"2024-06-29T15:57:21.212745+0000","last_active":"2024-06-29T15:57:36.330636+0000","last_peered":"2024-06-29T15:57:36.330636+0000","last_clean":"2024-06-29T15:57:36.330636+0000","last_became_active":"2024-06-29T15:57:21.212335+0000","last_became_peered":"2024-06-29T15:57:21.212335+0000","last_unstale":"2024-06-29T15:57:36.330636+0000","last_undegraded":"2024-06-29T15:57:36.330636+0000","last_fullsized":"2024-06-29T15:57:36.330636+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:07.049637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239385+0000","last_change":"2024-06-29T15:57:22.224294+0000","last_active":"2024-06-29T15:57:36.239385+0000","last_peered":"2024-06-29T15:57:36.239385+0000","last_clean":"2024-06-29T15:57:36.239385+0000","last_became_active":"2024-06-29T15:57:22.223922+0000","last_became_peered":"2024-06-29T15:57:22.223922+0000","last_unstale":"2024-06-29T15:57:36.239385+0000","last_undegraded":"2024-06-29T15:57:36.239385+0000","last_fullsized":"2024-06-29T15:57:36.239385+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:46.725456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311940+0000","last_change":"2024-06-29T15:57:34.308042+0000","last_active":"2024-06-29T15:57:36.311940+0000","last_peered":"2024-06-29T15:57:36.311940+0000","last_clean":"2024-06-29T15:57:36.311940+0000","last_became_active":"2024-06-29T15:57:34.307706+0000","last_became_peered":"2024-06-29T15:57:34.307706+0000","last_unstale":"2024-06-29T15:57:36.311940+0000","last_undegraded":"2024-06-29T15:57:36.311940+0000","last_fullsized":"2024-06-29T15:57:36.311940+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:24.276099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312202+0000","last_change":"2024-06-29T15:57:35.254882+0000","last_active":"2024-06-29T15:57:36.312202+0000","last_peered":"2024-06-29T15:57:36.312202+0000","last_clean":"2024-06-29T15:57:36.312202+0000","last_became_active":"2024-06-29T15:57:35.254543+0000","last_became_peered":"2024-06-29T15:57:35.254543+0000","last_unstale":"2024-06-29T15:57:36.312202+0000","last_undegraded":"2024-06-29T15:57:36.312202+0000","last_fullsized":"2024-06-29T15:57:36.312202+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:03:35.376526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237536+0000","last_change":"2024-06-29T15:57:21.211203+0000","last_active":"2024-06-29T15:57:36.237536+0000","last_peered":"2024-06-29T15:57:36.237536+0000","last_clean":"2024-06-29T15:57:36.237536+0000","last_became_active":"2024-06-29T15:57:21.210898+0000","last_became_peered":"2024-06-29T15:57:21.210898+0000","last_unstale":"2024-06-29T15:57:36.237536+0000","last_undegraded":"2024-06-29T15:57:36.237536+0000","last_fullsized":"2024-06-29T15:57:36.237536+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:17.270875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240196+0000","last_change":"2024-06-29T15:57:22.213920+0000","last_active":"2024-06-29T15:57:36.240196+0000","last_peered":"2024-06-29T15:57:36.240196+0000","last_clean":"2024-06-29T15:57:36.240196+0000","last_became_active":"2024-06-29T15:57:22.213583+0000","last_became_peered":"2024-06-29T15:57:22.213583+0000","last_unstale":"2024-06-29T15:57:36.240196+0000","last_undegraded":"2024-06-29T15:57:36.240196+0000","last_fullsized":"2024-06-29T15:57:36.240196+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:31.756785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331622+0000","last_change":"2024-06-29T15:57:34.308946+0000","last_active":"2024-06-29T15:57:36.331622+0000","last_peered":"2024-06-29T15:57:36.331622+0000","last_clean":"2024-06-29T15:57:36.331622+0000","last_became_active":"2024-06-29T15:57:34.308411+0000","last_became_peered":"2024-06-29T15:57:34.308411+0000","last_unstale":"2024-06-29T15:57:36.331622+0000","last_undegraded":"2024-06-29T15:57:36.331622+0000","last_fullsized":"2024-06-29T15:57:36.331622+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:13.989668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330052+0000","last_change":"2024-06-29T15:57:35.259846+0000","last_active":"2024-06-29T15:57:36.330052+0000","last_peered":"2024-06-29T15:57:36.330052+0000","last_clean":"2024-06-29T15:57:36.330052+0000","last_became_active":"2024-06-29T15:57:35.259493+0000","last_became_peered":"2024-06-29T15:57:35.259493+0000","last_unstale":"2024-06-29T15:57:36.330052+0000","last_undegraded":"2024-06-29T15:57:36.330052+0000","last_fullsized":"2024-06-29T15:57:36.330052+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:15.812204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329676+0000","last_change":"2024-06-29T15:57:21.212085+0000","last_active":"2024-06-29T15:57:36.329676+0000","last_peered":"2024-06-29T15:57:36.329676+0000","last_clean":"2024-06-29T15:57:36.329676+0000","last_became_active":"2024-06-29T15:57:21.211731+0000","last_became_peered":"2024-06-29T15:57:21.211731+0000","last_unstale":"2024-06-29T15:57:36.329676+0000","last_undegraded":"2024-06-29T15:57:36.329676+0000","last_fullsized":"2024-06-29T15:57:36.329676+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:36.061054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330288+0000","last_change":"2024-06-29T15:57:22.228211+0000","last_active":"2024-06-29T15:57:36.330288+0000","last_peered":"2024-06-29T15:57:36.330288+0000","last_clean":"2024-06-29T15:57:36.330288+0000","last_became_active":"2024-06-29T15:57:22.227519+0000","last_became_peered":"2024-06-29T15:57:22.227519+0000","last_unstale":"2024-06-29T15:57:36.330288+0000","last_undegraded":"2024-06-29T15:57:36.330288+0000","last_fullsized":"2024-06-29T15:57:36.330288+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:16:35.999038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331724+0000","last_change":"2024-06-29T15:57:34.314930+0000","last_active":"2024-06-29T15:57:36.331724+0000","last_peered":"2024-06-29T15:57:36.331724+0000","last_clean":"2024-06-29T15:57:36.331724+0000","last_became_active":"2024-06-29T15:57:34.314549+0000","last_became_peered":"2024-06-29T15:57:34.314549+0000","last_unstale":"2024-06-29T15:57:36.331724+0000","last_undegraded":"2024-06-29T15:57:36.331724+0000","last_fullsized":"2024-06-29T15:57:36.331724+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:02.172151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239694+0000","last_change":"2024-06-29T15:57:35.257200+0000","last_active":"2024-06-29T15:57:36.239694+0000","last_peered":"2024-06-29T15:57:36.239694+0000","last_clean":"2024-06-29T15:57:36.239694+0000","last_became_active":"2024-06-29T15:57:35.256810+0000","last_became_peered":"2024-06-29T15:57:35.256810+0000","last_unstale":"2024-06-29T15:57:36.239694+0000","last_undegraded":"2024-06-29T15:57:36.239694+0000","last_fullsized":"2024-06-29T15:57:36.239694+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:42.431331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330636+0000","last_change":"2024-06-29T15:57:21.208592+0000","last_active":"2024-06-29T15:57:36.330636+0000","last_peered":"2024-06-29T15:57:36.330636+0000","last_clean":"2024-06-29T15:57:36.330636+0000","last_became_active":"2024-06-29T15:57:21.208156+0000","last_became_peered":"2024-06-29T15:57:21.208156+0000","last_unstale":"2024-06-29T15:57:36.330636+0000","last_undegraded":"2024-06-29T15:57:36.330636+0000","last_fullsized":"2024-06-29T15:57:36.330636+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:20.325402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312669+0000","last_change":"2024-06-29T15:57:22.238829+0000","last_active":"2024-06-29T15:57:36.312669+0000","last_peered":"2024-06-29T15:57:36.312669+0000","last_clean":"2024-06-29T15:57:36.312669+0000","last_became_active":"2024-06-29T15:57:22.238520+0000","last_became_peered":"2024-06-29T15:57:22.238520+0000","last_unstale":"2024-06-29T15:57:36.312669+0000","last_undegraded":"2024-06-29T15:57:36.312669+0000","last_fullsized":"2024-06-29T15:57:36.312669+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:22.127384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239235+0000","last_change":"2024-06-29T15:57:34.308672+0000","last_active":"2024-06-29T15:57:36.239235+0000","last_peered":"2024-06-29T15:57:36.239235+0000","last_clean":"2024-06-29T15:57:36.239235+0000","last_became_active":"2024-06-29T15:57:34.307347+0000","last_became_peered":"2024-06-29T15:57:34.307347+0000","last_unstale":"2024-06-29T15:57:36.239235+0000","last_undegraded":"2024-06-29T15:57:36.239235+0000","last_fullsized":"2024-06-29T15:57:36.239235+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:53:19.725300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332378+0000","last_change":"2024-06-29T15:57:35.248756+0000","last_active":"2024-06-29T15:57:36.332378+0000","last_peered":"2024-06-29T15:57:36.332378+0000","last_clean":"2024-06-29T15:57:36.332378+0000","last_became_active":"2024-06-29T15:57:35.248281+0000","last_became_peered":"2024-06-29T15:57:35.248281+0000","last_unstale":"2024-06-29T15:57:36.332378+0000","last_undegraded":"2024-06-29T15:57:36.332378+0000","last_fullsized":"2024-06-29T15:57:36.332378+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:50.250829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331396+0000","last_change":"2024-06-29T15:57:21.215358+0000","last_active":"2024-06-29T15:57:36.331396+0000","last_peered":"2024-06-29T15:57:36.331396+0000","last_clean":"2024-06-29T15:57:36.331396+0000","last_became_active":"2024-06-29T15:57:21.213979+0000","last_became_peered":"2024-06-29T15:57:21.213979+0000","last_unstale":"2024-06-29T15:57:36.331396+0000","last_undegraded":"2024-06-29T15:57:36.331396+0000","last_fullsized":"2024-06-29T15:57:36.331396+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:42.446135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331975+0000","last_change":"2024-06-29T15:57:22.231883+0000","last_active":"2024-06-29T15:57:36.331975+0000","last_peered":"2024-06-29T15:57:36.331975+0000","last_clean":"2024-06-29T15:57:36.331975+0000","last_became_active":"2024-06-29T15:57:22.231588+0000","last_became_peered":"2024-06-29T15:57:22.231588+0000","last_unstale":"2024-06-29T15:57:36.331975+0000","last_undegraded":"2024-06-29T15:57:36.331975+0000","last_fullsized":"2024-06-29T15:57:36.331975+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:17.705760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238496+0000","last_change":"2024-06-29T15:57:34.307517+0000","last_active":"2024-06-29T15:57:36.238496+0000","last_peered":"2024-06-29T15:57:36.238496+0000","last_clean":"2024-06-29T15:57:36.238496+0000","last_became_active":"2024-06-29T15:57:34.307213+0000","last_became_peered":"2024-06-29T15:57:34.307213+0000","last_unstale":"2024-06-29T15:57:36.238496+0000","last_undegraded":"2024-06-29T15:57:36.238496+0000","last_fullsized":"2024-06-29T15:57:36.238496+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:02.055681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311679+0000","last_change":"2024-06-29T15:57:35.262345+0000","last_active":"2024-06-29T15:57:36.311679+0000","last_peered":"2024-06-29T15:57:36.311679+0000","last_clean":"2024-06-29T15:57:36.311679+0000","last_became_active":"2024-06-29T15:57:35.262052+0000","last_became_peered":"2024-06-29T15:57:35.262052+0000","last_unstale":"2024-06-29T15:57:36.311679+0000","last_undegraded":"2024-06-29T15:57:36.311679+0000","last_fullsized":"2024-06-29T15:57:36.311679+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:54.683205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330760+0000","last_change":"2024-06-29T15:57:21.213193+0000","last_active":"2024-06-29T15:57:36.330760+0000","last_peered":"2024-06-29T15:57:36.330760+0000","last_clean":"2024-06-29T15:57:36.330760+0000","last_became_active":"2024-06-29T15:57:21.212838+0000","last_became_peered":"2024-06-29T15:57:21.212838+0000","last_unstale":"2024-06-29T15:57:36.330760+0000","last_undegraded":"2024-06-29T15:57:36.330760+0000","last_fullsized":"2024-06-29T15:57:36.330760+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:54.574391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240083+0000","last_change":"2024-06-29T15:57:22.217630+0000","last_active":"2024-06-29T15:57:36.240083+0000","last_peered":"2024-06-29T15:57:36.240083+0000","last_clean":"2024-06-29T15:57:36.240083+0000","last_became_active":"2024-06-29T15:57:22.217303+0000","last_became_peered":"2024-06-29T15:57:22.217303+0000","last_unstale":"2024-06-29T15:57:36.240083+0000","last_undegraded":"2024-06-29T15:57:36.240083+0000","last_fullsized":"2024-06-29T15:57:36.240083+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:05.554814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236277+0000","last_change":"2024-06-29T15:57:34.305621+0000","last_active":"2024-06-29T15:57:36.236277+0000","last_peered":"2024-06-29T15:57:36.236277+0000","last_clean":"2024-06-29T15:57:36.236277+0000","last_became_active":"2024-06-29T15:57:34.305274+0000","last_became_peered":"2024-06-29T15:57:34.305274+0000","last_unstale":"2024-06-29T15:57:36.236277+0000","last_undegraded":"2024-06-29T15:57:36.236277+0000","last_fullsized":"2024-06-29T15:57:36.236277+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:58.216624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238128+0000","last_change":"2024-06-29T15:57:35.258213+0000","last_active":"2024-06-29T15:57:36.238128+0000","last_peered":"2024-06-29T15:57:36.238128+0000","last_clean":"2024-06-29T15:57:36.238128+0000","last_became_active":"2024-06-29T15:57:35.257901+0000","last_became_peered":"2024-06-29T15:57:35.257901+0000","last_unstale":"2024-06-29T15:57:36.238128+0000","last_undegraded":"2024-06-29T15:57:36.238128+0000","last_fullsized":"2024-06-29T15:57:36.238128+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:34.564002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330032+0000","last_change":"2024-06-29T15:57:21.210528+0000","last_active":"2024-06-29T15:57:36.330032+0000","last_peered":"2024-06-29T15:57:36.330032+0000","last_clean":"2024-06-29T15:57:36.330032+0000","last_became_active":"2024-06-29T15:57:21.208691+0000","last_became_peered":"2024-06-29T15:57:21.208691+0000","last_unstale":"2024-06-29T15:57:36.330032+0000","last_undegraded":"2024-06-29T15:57:36.330032+0000","last_fullsized":"2024-06-29T15:57:36.330032+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:21.762590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240196+0000","last_change":"2024-06-29T15:57:22.232727+0000","last_active":"2024-06-29T15:57:36.240196+0000","last_peered":"2024-06-29T15:57:36.240196+0000","last_clean":"2024-06-29T15:57:36.240196+0000","last_became_active":"2024-06-29T15:57:22.232392+0000","last_became_peered":"2024-06-29T15:57:22.232392+0000","last_unstale":"2024-06-29T15:57:36.240196+0000","last_undegraded":"2024-06-29T15:57:36.240196+0000","last_fullsized":"2024-06-29T15:57:36.240196+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:54.805984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239217+0000","last_change":"2024-06-29T15:57:34.308385+0000","last_active":"2024-06-29T15:57:36.239217+0000","last_peered":"2024-06-29T15:57:36.239217+0000","last_clean":"2024-06-29T15:57:36.239217+0000","last_became_active":"2024-06-29T15:57:34.308070+0000","last_became_peered":"2024-06-29T15:57:34.308070+0000","last_unstale":"2024-06-29T15:57:36.239217+0000","last_undegraded":"2024-06-29T15:57:36.239217+0000","last_fullsized":"2024-06-29T15:57:36.239217+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:11.812167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331467+0000","last_change":"2024-06-29T15:57:35.256356+0000","last_active":"2024-06-29T15:57:36.331467+0000","last_peered":"2024-06-29T15:57:36.331467+0000","last_clean":"2024-06-29T15:57:36.331467+0000","last_became_active":"2024-06-29T15:57:35.255846+0000","last_became_peered":"2024-06-29T15:57:35.255846+0000","last_unstale":"2024-06-29T15:57:36.331467+0000","last_undegraded":"2024-06-29T15:57:36.331467+0000","last_fullsized":"2024-06-29T15:57:36.331467+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:41.823081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.e","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238583+0000","last_change":"2024-06-29T15:57:21.216179+0000","last_active":"2024-06-29T15:57:36.238583+0000","last_peered":"2024-06-29T15:57:36.238583+0000","last_clean":"2024-06-29T15:57:36.238583+0000","last_became_active":"2024-06-29T15:57:21.215876+0000","last_became_peered":"2024-06-29T15:57:21.215876+0000","last_unstale":"2024-06-29T15:57:36.238583+0000","last_undegraded":"2024-06-29T15:57:36.238583+0000","last_fullsized":"2024-06-29T15:57:36.238583+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:37.942186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238721+0000","last_change":"2024-06-29T15:57:22.225793+0000","last_active":"2024-06-29T15:57:36.238721+0000","last_peered":"2024-06-29T15:57:36.238721+0000","last_clean":"2024-06-29T15:57:36.238721+0000","last_became_active":"2024-06-29T15:57:22.224806+0000","last_became_peered":"2024-06-29T15:57:22.224806+0000","last_unstale":"2024-06-29T15:57:36.238721+0000","last_undegraded":"2024-06-29T15:57:36.238721+0000","last_fullsized":"2024-06-29T15:57:36.238721+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:03.667477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329189+0000","last_change":"2024-06-29T15:57:34.311124+0000","last_active":"2024-06-29T15:57:36.329189+0000","last_peered":"2024-06-29T15:57:36.329189+0000","last_clean":"2024-06-29T15:57:36.329189+0000","last_became_active":"2024-06-29T15:57:34.310775+0000","last_became_peered":"2024-06-29T15:57:34.310775+0000","last_unstale":"2024-06-29T15:57:36.329189+0000","last_undegraded":"2024-06-29T15:57:36.329189+0000","last_fullsized":"2024-06-29T15:57:36.329189+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:50.229318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331196+0000","last_change":"2024-06-29T15:57:35.259909+0000","last_active":"2024-06-29T15:57:36.331196+0000","last_peered":"2024-06-29T15:57:36.331196+0000","last_clean":"2024-06-29T15:57:36.331196+0000","last_became_active":"2024-06-29T15:57:35.259564+0000","last_became_peered":"2024-06-29T15:57:35.259564+0000","last_unstale":"2024-06-29T15:57:36.331196+0000","last_undegraded":"2024-06-29T15:57:36.331196+0000","last_fullsized":"2024-06-29T15:57:36.331196+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:18.743864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.f","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239899+0000","last_change":"2024-06-29T15:57:21.212718+0000","last_active":"2024-06-29T15:57:36.239899+0000","last_peered":"2024-06-29T15:57:36.239899+0000","last_clean":"2024-06-29T15:57:36.239899+0000","last_became_active":"2024-06-29T15:57:21.212406+0000","last_became_peered":"2024-06-29T15:57:21.212406+0000","last_unstale":"2024-06-29T15:57:36.239899+0000","last_undegraded":"2024-06-29T15:57:36.239899+0000","last_fullsized":"2024-06-29T15:57:36.239899+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:00.180622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239223+0000","last_change":"2024-06-29T15:57:22.238347+0000","last_active":"2024-06-29T15:57:36.239223+0000","last_peered":"2024-06-29T15:57:36.239223+0000","last_clean":"2024-06-29T15:57:36.239223+0000","last_became_active":"2024-06-29T15:57:22.236896+0000","last_became_peered":"2024-06-29T15:57:22.236896+0000","last_unstale":"2024-06-29T15:57:36.239223+0000","last_undegraded":"2024-06-29T15:57:36.239223+0000","last_fullsized":"2024-06-29T15:57:36.239223+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:39.739194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236003+0000","last_change":"2024-06-29T15:57:34.308954+0000","last_active":"2024-06-29T15:57:36.236003+0000","last_peered":"2024-06-29T15:57:36.236003+0000","last_clean":"2024-06-29T15:57:36.236003+0000","last_became_active":"2024-06-29T15:57:34.308595+0000","last_became_peered":"2024-06-29T15:57:34.308595+0000","last_unstale":"2024-06-29T15:57:36.236003+0000","last_undegraded":"2024-06-29T15:57:36.236003+0000","last_fullsized":"2024-06-29T15:57:36.236003+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:06:10.871555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236850+0000","last_change":"2024-06-29T15:57:35.250645+0000","last_active":"2024-06-29T15:57:36.236850+0000","last_peered":"2024-06-29T15:57:36.236850+0000","last_clean":"2024-06-29T15:57:36.236850+0000","last_became_active":"2024-06-29T15:57:35.250314+0000","last_became_peered":"2024-06-29T15:57:35.250314+0000","last_unstale":"2024-06-29T15:57:36.236850+0000","last_undegraded":"2024-06-29T15:57:36.236850+0000","last_fullsized":"2024-06-29T15:57:36.236850+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:06.391987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.10","version":"221'8","reported_seq":30,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329441+0000","last_change":"2024-06-29T15:57:21.216156+0000","last_active":"2024-06-29T15:57:36.329441+0000","last_peered":"2024-06-29T15:57:36.329441+0000","last_clean":"2024-06-29T15:57:36.329441+0000","last_became_active":"2024-06-29T15:57:21.214088+0000","last_became_peered":"2024-06-29T15:57:21.214088+0000","last_unstale":"2024-06-29T15:57:36.329441+0000","last_undegraded":"2024-06-29T15:57:36.329441+0000","last_fullsized":"2024-06-29T15:57:36.329441+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:30.314243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312937+0000","last_change":"2024-06-29T15:57:22.223112+0000","last_active":"2024-06-29T15:57:36.312937+0000","last_peered":"2024-06-29T15:57:36.312937+0000","last_clean":"2024-06-29T15:57:36.312937+0000","last_became_active":"2024-06-29T15:57:22.222632+0000","last_became_peered":"2024-06-29T15:57:22.222632+0000","last_unstale":"2024-06-29T15:57:36.312937+0000","last_undegraded":"2024-06-29T15:57:36.312937+0000","last_fullsized":"2024-06-29T15:57:36.312937+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:05.408776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329903+0000","last_change":"2024-06-29T15:57:34.310931+0000","last_active":"2024-06-29T15:57:36.329903+0000","last_peered":"2024-06-29T15:57:36.329903+0000","last_clean":"2024-06-29T15:57:36.329903+0000","last_became_active":"2024-06-29T15:57:34.310598+0000","last_became_peered":"2024-06-29T15:57:34.310598+0000","last_unstale":"2024-06-29T15:57:36.329903+0000","last_undegraded":"2024-06-29T15:57:36.329903+0000","last_fullsized":"2024-06-29T15:57:36.329903+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:51.824136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236659+0000","last_change":"2024-06-29T15:57:35.250035+0000","last_active":"2024-06-29T15:57:36.236659+0000","last_peered":"2024-06-29T15:57:36.236659+0000","last_clean":"2024-06-29T15:57:36.236659+0000","last_became_active":"2024-06-29T15:57:35.249711+0000","last_became_peered":"2024-06-29T15:57:35.249711+0000","last_unstale":"2024-06-29T15:57:36.236659+0000","last_undegraded":"2024-06-29T15:57:36.236659+0000","last_fullsized":"2024-06-29T15:57:36.236659+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:13.347781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329612+0000","last_change":"2024-06-29T15:57:21.215032+0000","last_active":"2024-06-29T15:57:36.329612+0000","last_peered":"2024-06-29T15:57:36.329612+0000","last_clean":"2024-06-29T15:57:36.329612+0000","last_became_active":"2024-06-29T15:57:21.214557+0000","last_became_peered":"2024-06-29T15:57:21.214557+0000","last_unstale":"2024-06-29T15:57:36.329612+0000","last_undegraded":"2024-06-29T15:57:36.329612+0000","last_fullsized":"2024-06-29T15:57:36.329612+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330185+0000","last_change":"2024-06-29T15:57:22.228484+0000","last_active":"2024-06-29T15:57:36.330185+0000","last_peered":"2024-06-29T15:57:36.330185+0000","last_clean":"2024-06-29T15:57:36.330185+0000","last_became_active":"2024-06-29T15:57:22.227880+0000","last_became_peered":"2024-06-29T15:57:22.227880+0000","last_unstale":"2024-06-29T15:57:36.330185+0000","last_undegraded":"2024-06-29T15:57:36.330185+0000","last_fullsized":"2024-06-29T15:57:36.330185+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:21.749071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312893+0000","last_change":"2024-06-29T15:57:34.309910+0000","last_active":"2024-06-29T15:57:36.312893+0000","last_peered":"2024-06-29T15:57:36.312893+0000","last_clean":"2024-06-29T15:57:36.312893+0000","last_became_active":"2024-06-29T15:57:34.309509+0000","last_became_peered":"2024-06-29T15:57:34.309509+0000","last_unstale":"2024-06-29T15:57:36.312893+0000","last_undegraded":"2024-06-29T15:57:36.312893+0000","last_fullsized":"2024-06-29T15:57:36.312893+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:05.013352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329983+0000","last_change":"2024-06-29T15:57:35.254963+0000","last_active":"2024-06-29T15:57:36.329983+0000","last_peered":"2024-06-29T15:57:36.329983+0000","last_clean":"2024-06-29T15:57:36.329983+0000","last_became_active":"2024-06-29T15:57:35.254059+0000","last_became_peered":"2024-06-29T15:57:35.254059+0000","last_unstale":"2024-06-29T15:57:36.329983+0000","last_undegraded":"2024-06-29T15:57:36.329983+0000","last_fullsized":"2024-06-29T15:57:36.329983+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:42.786307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236445+0000","last_change":"2024-06-29T15:57:21.203410+0000","last_active":"2024-06-29T15:57:36.236445+0000","last_peered":"2024-06-29T15:57:36.236445+0000","last_clean":"2024-06-29T15:57:36.236445+0000","last_became_active":"2024-06-29T15:57:21.203034+0000","last_became_peered":"2024-06-29T15:57:21.203034+0000","last_unstale":"2024-06-29T15:57:36.236445+0000","last_undegraded":"2024-06-29T15:57:36.236445+0000","last_fullsized":"2024-06-29T15:57:36.236445+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:52.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237205+0000","last_change":"2024-06-29T15:57:22.241881+0000","last_active":"2024-06-29T15:57:36.237205+0000","last_peered":"2024-06-29T15:57:36.237205+0000","last_clean":"2024-06-29T15:57:36.237205+0000","last_became_active":"2024-06-29T15:57:22.241549+0000","last_became_peered":"2024-06-29T15:57:22.241549+0000","last_unstale":"2024-06-29T15:57:36.237205+0000","last_undegraded":"2024-06-29T15:57:36.237205+0000","last_fullsized":"2024-06-29T15:57:36.237205+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:25.352328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"225'6","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240234+0000","last_change":"2024-06-29T15:57:34.309744+0000","last_active":"2024-06-29T15:57:36.240234+0000","last_peered":"2024-06-29T15:57:36.240234+0000","last_clean":"2024-06-29T15:57:36.240234+0000","last_became_active":"2024-06-29T15:57:34.309408+0000","last_became_peered":"2024-06-29T15:57:34.309408+0000","last_unstale":"2024-06-29T15:57:36.240234+0000","last_undegraded":"2024-06-29T15:57:36.240234+0000","last_fullsized":"2024-06-29T15:57:36.240234+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:20.219821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332388+0000","last_change":"2024-06-29T15:57:35.248181+0000","last_active":"2024-06-29T15:57:36.332388+0000","last_peered":"2024-06-29T15:57:36.332388+0000","last_clean":"2024-06-29T15:57:36.332388+0000","last_became_active":"2024-06-29T15:57:35.247656+0000","last_became_peered":"2024-06-29T15:57:35.247656+0000","last_unstale":"2024-06-29T15:57:36.332388+0000","last_undegraded":"2024-06-29T15:57:36.332388+0000","last_fullsized":"2024-06-29T15:57:36.332388+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:26.486217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331412+0000","last_change":"2024-06-29T15:57:21.215892+0000","last_active":"2024-06-29T15:57:36.331412+0000","last_peered":"2024-06-29T15:57:36.331412+0000","last_clean":"2024-06-29T15:57:36.331412+0000","last_became_active":"2024-06-29T15:57:21.215250+0000","last_became_peered":"2024-06-29T15:57:21.215250+0000","last_unstale":"2024-06-29T15:57:36.331412+0000","last_undegraded":"2024-06-29T15:57:36.331412+0000","last_fullsized":"2024-06-29T15:57:36.331412+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:23.234622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331871+0000","last_change":"2024-06-29T15:57:22.234802+0000","last_active":"2024-06-29T15:57:36.331871+0000","last_peered":"2024-06-29T15:57:36.331871+0000","last_clean":"2024-06-29T15:57:36.331871+0000","last_became_active":"2024-06-29T15:57:22.234492+0000","last_became_peered":"2024-06-29T15:57:22.234492+0000","last_unstale":"2024-06-29T15:57:36.331871+0000","last_undegraded":"2024-06-29T15:57:36.331871+0000","last_fullsized":"2024-06-29T15:57:36.331871+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:37.928014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"225'3","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312419+0000","last_change":"2024-06-29T15:57:34.308776+0000","last_active":"2024-06-29T15:57:36.312419+0000","last_peered":"2024-06-29T15:57:36.312419+0000","last_clean":"2024-06-29T15:57:36.312419+0000","last_became_active":"2024-06-29T15:57:34.308478+0000","last_became_peered":"2024-06-29T15:57:34.308478+0000","last_unstale":"2024-06-29T15:57:36.312419+0000","last_undegraded":"2024-06-29T15:57:36.312419+0000","last_fullsized":"2024-06-29T15:57:36.312419+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:14.899188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237305+0000","last_change":"2024-06-29T15:57:35.251086+0000","last_active":"2024-06-29T15:57:36.237305+0000","last_peered":"2024-06-29T15:57:36.237305+0000","last_clean":"2024-06-29T15:57:36.237305+0000","last_became_active":"2024-06-29T15:57:35.249526+0000","last_became_peered":"2024-06-29T15:57:35.249526+0000","last_unstale":"2024-06-29T15:57:36.237305+0000","last_undegraded":"2024-06-29T15:57:36.237305+0000","last_fullsized":"2024-06-29T15:57:36.237305+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:22.660265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.14","version":"222'8","reported_seq":30,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312307+0000","last_change":"2024-06-29T15:57:21.217755+0000","last_active":"2024-06-29T15:57:36.312307+0000","last_peered":"2024-06-29T15:57:36.312307+0000","last_clean":"2024-06-29T15:57:36.312307+0000","last_became_active":"2024-06-29T15:57:21.216983+0000","last_became_peered":"2024-06-29T15:57:21.216983+0000","last_unstale":"2024-06-29T15:57:36.312307+0000","last_undegraded":"2024-06-29T15:57:36.312307+0000","last_fullsized":"2024-06-29T15:57:36.312307+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:31.889343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331251+0000","last_change":"2024-06-29T15:57:22.224600+0000","last_active":"2024-06-29T15:57:36.331251+0000","last_peered":"2024-06-29T15:57:36.331251+0000","last_clean":"2024-06-29T15:57:36.331251+0000","last_became_active":"2024-06-29T15:57:22.224260+0000","last_became_peered":"2024-06-29T15:57:22.224260+0000","last_unstale":"2024-06-29T15:57:36.331251+0000","last_undegraded":"2024-06-29T15:57:36.331251+0000","last_fullsized":"2024-06-29T15:57:36.331251+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:21.773370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331316+0000","last_change":"2024-06-29T15:57:34.311602+0000","last_active":"2024-06-29T15:57:36.331316+0000","last_peered":"2024-06-29T15:57:36.331316+0000","last_clean":"2024-06-29T15:57:36.331316+0000","last_became_active":"2024-06-29T15:57:34.311208+0000","last_became_peered":"2024-06-29T15:57:34.311208+0000","last_unstale":"2024-06-29T15:57:36.331316+0000","last_undegraded":"2024-06-29T15:57:36.331316+0000","last_fullsized":"2024-06-29T15:57:36.331316+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:22.313530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329523+0000","last_change":"2024-06-29T15:57:35.257800+0000","last_active":"2024-06-29T15:57:36.329523+0000","last_peered":"2024-06-29T15:57:36.329523+0000","last_clean":"2024-06-29T15:57:36.329523+0000","last_became_active":"2024-06-29T15:57:35.254594+0000","last_became_peered":"2024-06-29T15:57:35.254594+0000","last_unstale":"2024-06-29T15:57:36.329523+0000","last_undegraded":"2024-06-29T15:57:36.329523+0000","last_fullsized":"2024-06-29T15:57:36.329523+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:10.058134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"221'3","reported_seq":25,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330966+0000","last_change":"2024-06-29T15:57:21.210270+0000","last_active":"2024-06-29T15:57:36.330966+0000","last_peered":"2024-06-29T15:57:36.330966+0000","last_clean":"2024-06-29T15:57:36.330966+0000","last_became_active":"2024-06-29T15:57:21.209731+0000","last_became_peered":"2024-06-29T15:57:21.209731+0000","last_unstale":"2024-06-29T15:57:36.330966+0000","last_undegraded":"2024-06-29T15:57:36.330966+0000","last_fullsized":"2024-06-29T15:57:36.330966+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:18.182233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331525+0000","last_change":"2024-06-29T15:57:22.231011+0000","last_active":"2024-06-29T15:57:36.331525+0000","last_peered":"2024-06-29T15:57:36.331525+0000","last_clean":"2024-06-29T15:57:36.331525+0000","last_became_active":"2024-06-29T15:57:22.230675+0000","last_became_peered":"2024-06-29T15:57:22.230675+0000","last_unstale":"2024-06-29T15:57:36.331525+0000","last_undegraded":"2024-06-29T15:57:36.331525+0000","last_fullsized":"2024-06-29T15:57:36.331525+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:09.478769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239084+0000","last_change":"2024-06-29T15:57:34.311119+0000","last_active":"2024-06-29T15:57:36.239084+0000","last_peered":"2024-06-29T15:57:36.239084+0000","last_clean":"2024-06-29T15:57:36.239084+0000","last_became_active":"2024-06-29T15:57:34.310820+0000","last_became_peered":"2024-06-29T15:57:34.310820+0000","last_unstale":"2024-06-29T15:57:36.239084+0000","last_undegraded":"2024-06-29T15:57:36.239084+0000","last_fullsized":"2024-06-29T15:57:36.239084+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:12:01.172748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236419+0000","last_change":"2024-06-29T15:57:35.252881+0000","last_active":"2024-06-29T15:57:36.236419+0000","last_peered":"2024-06-29T15:57:36.236419+0000","last_clean":"2024-06-29T15:57:36.236419+0000","last_became_active":"2024-06-29T15:57:35.252555+0000","last_became_peered":"2024-06-29T15:57:35.252555+0000","last_unstale":"2024-06-29T15:57:36.236419+0000","last_undegraded":"2024-06-29T15:57:36.236419+0000","last_fullsized":"2024-06-29T15:57:36.236419+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:13.830138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312629+0000","last_change":"2024-06-29T15:57:21.214536+0000","last_active":"2024-06-29T15:57:36.312629+0000","last_peered":"2024-06-29T15:57:36.312629+0000","last_clean":"2024-06-29T15:57:36.312629+0000","last_became_active":"2024-06-29T15:57:21.214227+0000","last_became_peered":"2024-06-29T15:57:21.214227+0000","last_unstale":"2024-06-29T15:57:36.312629+0000","last_undegraded":"2024-06-29T15:57:36.312629+0000","last_fullsized":"2024-06-29T15:57:36.312629+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:25.839522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236127+0000","last_change":"2024-06-29T15:57:22.219192+0000","last_active":"2024-06-29T15:57:36.236127+0000","last_peered":"2024-06-29T15:57:36.236127+0000","last_clean":"2024-06-29T15:57:36.236127+0000","last_became_active":"2024-06-29T15:57:22.218844+0000","last_became_peered":"2024-06-29T15:57:22.218844+0000","last_unstale":"2024-06-29T15:57:36.236127+0000","last_undegraded":"2024-06-29T15:57:36.236127+0000","last_fullsized":"2024-06-29T15:57:36.236127+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:50.717283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"225'8","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238266+0000","last_change":"2024-06-29T15:57:34.309117+0000","last_active":"2024-06-29T15:57:36.238266+0000","last_peered":"2024-06-29T15:57:36.238266+0000","last_clean":"2024-06-29T15:57:36.238266+0000","last_became_active":"2024-06-29T15:57:34.308792+0000","last_became_peered":"2024-06-29T15:57:34.308792+0000","last_unstale":"2024-06-29T15:57:36.238266+0000","last_undegraded":"2024-06-29T15:57:36.238266+0000","last_fullsized":"2024-06-29T15:57:36.238266+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:31:47.175085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330362+0000","last_change":"2024-06-29T15:57:35.256824+0000","last_active":"2024-06-29T15:57:36.330362+0000","last_peered":"2024-06-29T15:57:36.330362+0000","last_clean":"2024-06-29T15:57:36.330362+0000","last_became_active":"2024-06-29T15:57:35.255771+0000","last_became_peered":"2024-06-29T15:57:35.255771+0000","last_unstale":"2024-06-29T15:57:36.330362+0000","last_undegraded":"2024-06-29T15:57:36.330362+0000","last_fullsized":"2024-06-29T15:57:36.330362+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:51.718094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331403+0000","last_change":"2024-06-29T15:57:21.218814+0000","last_active":"2024-06-29T15:57:36.331403+0000","last_peered":"2024-06-29T15:57:36.331403+0000","last_clean":"2024-06-29T15:57:36.331403+0000","last_became_active":"2024-06-29T15:57:21.218495+0000","last_became_peered":"2024-06-29T15:57:21.218495+0000","last_unstale":"2024-06-29T15:57:36.331403+0000","last_undegraded":"2024-06-29T15:57:36.331403+0000","last_fullsized":"2024-06-29T15:57:36.331403+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:04.680014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329332+0000","last_change":"2024-06-29T15:57:22.235312+0000","last_active":"2024-06-29T15:57:36.329332+0000","last_peered":"2024-06-29T15:57:36.329332+0000","last_clean":"2024-06-29T15:57:36.329332+0000","last_became_active":"2024-06-29T15:57:22.234828+0000","last_became_peered":"2024-06-29T15:57:22.234828+0000","last_unstale":"2024-06-29T15:57:36.329332+0000","last_undegraded":"2024-06-29T15:57:36.329332+0000","last_fullsized":"2024-06-29T15:57:36.329332+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:39.661714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237083+0000","last_change":"2024-06-29T15:57:34.306092+0000","last_active":"2024-06-29T15:57:36.237083+0000","last_peered":"2024-06-29T15:57:36.237083+0000","last_clean":"2024-06-29T15:57:36.237083+0000","last_became_active":"2024-06-29T15:57:34.305717+0000","last_became_peered":"2024-06-29T15:57:34.305717+0000","last_unstale":"2024-06-29T15:57:36.237083+0000","last_undegraded":"2024-06-29T15:57:36.237083+0000","last_fullsized":"2024-06-29T15:57:36.237083+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:12.354679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329622+0000","last_change":"2024-06-29T15:57:35.244422+0000","last_active":"2024-06-29T15:57:36.329622+0000","last_peered":"2024-06-29T15:57:36.329622+0000","last_clean":"2024-06-29T15:57:36.329622+0000","last_became_active":"2024-06-29T15:57:35.244085+0000","last_became_peered":"2024-06-29T15:57:35.244085+0000","last_unstale":"2024-06-29T15:57:36.329622+0000","last_undegraded":"2024-06-29T15:57:36.329622+0000","last_fullsized":"2024-06-29T15:57:36.329622+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:31.011742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238696+0000","last_change":"2024-06-29T15:57:34.312096+0000","last_active":"2024-06-29T15:57:36.238696+0000","last_peered":"2024-06-29T15:57:36.238696+0000","last_clean":"2024-06-29T15:57:36.238696+0000","last_became_active":"2024-06-29T15:57:34.311757+0000","last_became_peered":"2024-06-29T15:57:34.311757+0000","last_unstale":"2024-06-29T15:57:36.238696+0000","last_undegraded":"2024-06-29T15:57:36.238696+0000","last_fullsized":"2024-06-29T15:57:36.238696+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:51.898399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236052+0000","last_change":"2024-06-29T15:57:21.208896+0000","last_active":"2024-06-29T15:57:36.236052+0000","last_peered":"2024-06-29T15:57:36.236052+0000","last_clean":"2024-06-29T15:57:36.236052+0000","last_became_active":"2024-06-29T15:57:21.208555+0000","last_became_peered":"2024-06-29T15:57:21.208555+0000","last_unstale":"2024-06-29T15:57:36.236052+0000","last_undegraded":"2024-06-29T15:57:36.236052+0000","last_fullsized":"2024-06-29T15:57:36.236052+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:46.722681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330531+0000","last_change":"2024-06-29T15:57:22.234743+0000","last_active":"2024-06-29T15:57:36.330531+0000","last_peered":"2024-06-29T15:57:36.330531+0000","last_clean":"2024-06-29T15:57:36.330531+0000","last_became_active":"2024-06-29T15:57:22.233091+0000","last_became_peered":"2024-06-29T15:57:22.233091+0000","last_unstale":"2024-06-29T15:57:36.330531+0000","last_undegraded":"2024-06-29T15:57:36.330531+0000","last_fullsized":"2024-06-29T15:57:36.330531+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:46.131681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331747+0000","last_change":"2024-06-29T15:57:35.247871+0000","last_active":"2024-06-29T15:57:36.331747+0000","last_peered":"2024-06-29T15:57:36.331747+0000","last_clean":"2024-06-29T15:57:36.331747+0000","last_became_active":"2024-06-29T15:57:35.247551+0000","last_became_peered":"2024-06-29T15:57:35.247551+0000","last_unstale":"2024-06-29T15:57:36.331747+0000","last_undegraded":"2024-06-29T15:57:36.331747+0000","last_fullsized":"2024-06-29T15:57:36.331747+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:38:28.226246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"225'4","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330982+0000","last_change":"2024-06-29T15:57:34.310706+0000","last_active":"2024-06-29T15:57:36.330982+0000","last_peered":"2024-06-29T15:57:36.330982+0000","last_clean":"2024-06-29T15:57:36.330982+0000","last_became_active":"2024-06-29T15:57:34.310360+0000","last_became_peered":"2024-06-29T15:57:34.310360+0000","last_unstale":"2024-06-29T15:57:36.330982+0000","last_undegraded":"2024-06-29T15:57:36.330982+0000","last_fullsized":"2024-06-29T15:57:36.330982+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:38.399869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311825+0000","last_change":"2024-06-29T15:57:21.212034+0000","last_active":"2024-06-29T15:57:36.311825+0000","last_peered":"2024-06-29T15:57:36.311825+0000","last_clean":"2024-06-29T15:57:36.311825+0000","last_became_active":"2024-06-29T15:57:21.211762+0000","last_became_peered":"2024-06-29T15:57:21.211762+0000","last_unstale":"2024-06-29T15:57:36.311825+0000","last_undegraded":"2024-06-29T15:57:36.311825+0000","last_fullsized":"2024-06-29T15:57:36.311825+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:36.775853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330016+0000","last_change":"2024-06-29T15:57:22.214168+0000","last_active":"2024-06-29T15:57:36.330016+0000","last_peered":"2024-06-29T15:57:36.330016+0000","last_clean":"2024-06-29T15:57:36.330016+0000","last_became_active":"2024-06-29T15:57:22.213780+0000","last_became_peered":"2024-06-29T15:57:22.213780+0000","last_unstale":"2024-06-29T15:57:36.330016+0000","last_undegraded":"2024-06-29T15:57:36.330016+0000","last_fullsized":"2024-06-29T15:57:36.330016+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:15:01.904725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238570+0000","last_change":"2024-06-29T15:57:35.257408+0000","last_active":"2024-06-29T15:57:36.238570+0000","last_peered":"2024-06-29T15:57:36.238570+0000","last_clean":"2024-06-29T15:57:36.238570+0000","last_became_active":"2024-06-29T15:57:35.257065+0000","last_became_peered":"2024-06-29T15:57:35.257065+0000","last_unstale":"2024-06-29T15:57:36.238570+0000","last_undegraded":"2024-06-29T15:57:36.238570+0000","last_fullsized":"2024-06-29T15:57:36.238570+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:32.986678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1b","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311976+0000","last_change":"2024-06-29T15:57:21.211113+0000","last_active":"2024-06-29T15:57:36.311976+0000","last_peered":"2024-06-29T15:57:36.311976+0000","last_clean":"2024-06-29T15:57:36.311976+0000","last_became_active":"2024-06-29T15:57:21.210817+0000","last_became_peered":"2024-06-29T15:57:21.210817+0000","last_unstale":"2024-06-29T15:57:36.311976+0000","last_undegraded":"2024-06-29T15:57:36.311976+0000","last_fullsized":"2024-06-29T15:57:36.311976+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:26.176884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329924+0000","last_change":"2024-06-29T15:57:22.235721+0000","last_active":"2024-06-29T15:57:36.329924+0000","last_peered":"2024-06-29T15:57:36.329924+0000","last_clean":"2024-06-29T15:57:36.329924+0000","last_became_active":"2024-06-29T15:57:22.235380+0000","last_became_peered":"2024-06-29T15:57:22.235380+0000","last_unstale":"2024-06-29T15:57:36.329924+0000","last_undegraded":"2024-06-29T15:57:36.329924+0000","last_fullsized":"2024-06-29T15:57:36.329924+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:38.144376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"225'2","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237819+0000","last_change":"2024-06-29T15:57:34.310685+0000","last_active":"2024-06-29T15:57:36.237819+0000","last_peered":"2024-06-29T15:57:36.237819+0000","last_clean":"2024-06-29T15:57:36.237819+0000","last_became_active":"2024-06-29T15:57:34.310377+0000","last_became_peered":"2024-06-29T15:57:34.310377+0000","last_unstale":"2024-06-29T15:57:36.237819+0000","last_undegraded":"2024-06-29T15:57:36.237819+0000","last_fullsized":"2024-06-29T15:57:36.237819+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:36.326091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237218+0000","last_change":"2024-06-29T15:57:35.251476+0000","last_active":"2024-06-29T15:57:36.237218+0000","last_peered":"2024-06-29T15:57:36.237218+0000","last_clean":"2024-06-29T15:57:36.237218+0000","last_became_active":"2024-06-29T15:57:35.250496+0000","last_became_peered":"2024-06-29T15:57:35.250496+0000","last_unstale":"2024-06-29T15:57:36.237218+0000","last_undegraded":"2024-06-29T15:57:36.237218+0000","last_fullsized":"2024-06-29T15:57:36.237218+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:52.738461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330242+0000","last_change":"2024-06-29T15:57:21.210643+0000","last_active":"2024-06-29T15:57:36.330242+0000","last_peered":"2024-06-29T15:57:36.330242+0000","last_clean":"2024-06-29T15:57:36.330242+0000","last_became_active":"2024-06-29T15:57:21.209805+0000","last_became_peered":"2024-06-29T15:57:21.209805+0000","last_unstale":"2024-06-29T15:57:36.330242+0000","last_undegraded":"2024-06-29T15:57:36.330242+0000","last_fullsized":"2024-06-29T15:57:36.330242+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:27.618049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331248+0000","last_change":"2024-06-29T15:57:22.226181+0000","last_active":"2024-06-29T15:57:36.331248+0000","last_peered":"2024-06-29T15:57:36.331248+0000","last_clean":"2024-06-29T15:57:36.331248+0000","last_became_active":"2024-06-29T15:57:22.225855+0000","last_became_peered":"2024-06-29T15:57:22.225855+0000","last_unstale":"2024-06-29T15:57:36.331248+0000","last_undegraded":"2024-06-29T15:57:36.331248+0000","last_fullsized":"2024-06-29T15:57:36.331248+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:37.067580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311945+0000","last_change":"2024-06-29T15:57:34.314461+0000","last_active":"2024-06-29T15:57:36.311945+0000","last_peered":"2024-06-29T15:57:36.311945+0000","last_clean":"2024-06-29T15:57:36.311945+0000","last_became_active":"2024-06-29T15:57:34.314139+0000","last_became_peered":"2024-06-29T15:57:34.314139+0000","last_unstale":"2024-06-29T15:57:36.311945+0000","last_undegraded":"2024-06-29T15:57:36.311945+0000","last_fullsized":"2024-06-29T15:57:36.311945+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:09.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330356+0000","last_change":"2024-06-29T15:57:35.246277+0000","last_active":"2024-06-29T15:57:36.330356+0000","last_peered":"2024-06-29T15:57:36.330356+0000","last_clean":"2024-06-29T15:57:36.330356+0000","last_became_active":"2024-06-29T15:57:35.245894+0000","last_became_peered":"2024-06-29T15:57:35.245894+0000","last_unstale":"2024-06-29T15:57:36.330356+0000","last_undegraded":"2024-06-29T15:57:36.330356+0000","last_fullsized":"2024-06-29T15:57:36.330356+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:16.673478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1d","version":"221'2","reported_seq":24,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329758+0000","last_change":"2024-06-29T15:57:21.210602+0000","last_active":"2024-06-29T15:57:36.329758+0000","last_peered":"2024-06-29T15:57:36.329758+0000","last_clean":"2024-06-29T15:57:36.329758+0000","last_became_active":"2024-06-29T15:57:21.210296+0000","last_became_peered":"2024-06-29T15:57:21.210296+0000","last_unstale":"2024-06-29T15:57:36.329758+0000","last_undegraded":"2024-06-29T15:57:36.329758+0000","last_fullsized":"2024-06-29T15:57:36.329758+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:07.128352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238476+0000","last_change":"2024-06-29T15:57:22.213237+0000","last_active":"2024-06-29T15:57:36.238476+0000","last_peered":"2024-06-29T15:57:36.238476+0000","last_clean":"2024-06-29T15:57:36.238476+0000","last_became_active":"2024-06-29T15:57:22.212908+0000","last_became_peered":"2024-06-29T15:57:22.212908+0000","last_unstale":"2024-06-29T15:57:36.238476+0000","last_undegraded":"2024-06-29T15:57:36.238476+0000","last_fullsized":"2024-06-29T15:57:36.238476+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:31.343924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"225'1","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236505+0000","last_change":"2024-06-29T15:57:34.313977+0000","last_active":"2024-06-29T15:57:36.236505+0000","last_peered":"2024-06-29T15:57:36.236505+0000","last_clean":"2024-06-29T15:57:36.236505+0000","last_became_active":"2024-06-29T15:57:34.313702+0000","last_became_peered":"2024-06-29T15:57:34.313702+0000","last_unstale":"2024-06-29T15:57:36.236505+0000","last_undegraded":"2024-06-29T15:57:36.236505+0000","last_fullsized":"2024-06-29T15:57:36.236505+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:25:12.668316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329526+0000","last_change":"2024-06-29T15:57:35.242307+0000","last_active":"2024-06-29T15:57:36.329526+0000","last_peered":"2024-06-29T15:57:36.329526+0000","last_clean":"2024-06-29T15:57:36.329526+0000","last_became_active":"2024-06-29T15:57:35.241559+0000","last_became_peered":"2024-06-29T15:57:35.241559+0000","last_unstale":"2024-06-29T15:57:36.329526+0000","last_undegraded":"2024-06-29T15:57:36.329526+0000","last_fullsized":"2024-06-29T15:57:36.329526+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:37.228107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1e","version":"221'1","reported_seq":23,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238472+0000","last_change":"2024-06-29T15:57:21.213288+0000","last_active":"2024-06-29T15:57:36.238472+0000","last_peered":"2024-06-29T15:57:36.238472+0000","last_clean":"2024-06-29T15:57:36.238472+0000","last_became_active":"2024-06-29T15:57:21.212954+0000","last_became_peered":"2024-06-29T15:57:21.212954+0000","last_unstale":"2024-06-29T15:57:36.238472+0000","last_undegraded":"2024-06-29T15:57:36.238472+0000","last_fullsized":"2024-06-29T15:57:36.238472+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:24.708849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237336+0000","last_change":"2024-06-29T15:57:22.224449+0000","last_active":"2024-06-29T15:57:36.237336+0000","last_peered":"2024-06-29T15:57:36.237336+0000","last_clean":"2024-06-29T15:57:36.237336+0000","last_became_active":"2024-06-29T15:57:22.224124+0000","last_became_peered":"2024-06-29T15:57:22.224124+0000","last_unstale":"2024-06-29T15:57:36.237336+0000","last_undegraded":"2024-06-29T15:57:36.237336+0000","last_fullsized":"2024-06-29T15:57:36.237336+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:05.649826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330422+0000","last_change":"2024-06-29T15:57:34.304015+0000","last_active":"2024-06-29T15:57:36.330422+0000","last_peered":"2024-06-29T15:57:36.330422+0000","last_clean":"2024-06-29T15:57:36.330422+0000","last_became_active":"2024-06-29T15:57:34.303686+0000","last_became_peered":"2024-06-29T15:57:34.303686+0000","last_unstale":"2024-06-29T15:57:36.330422+0000","last_undegraded":"2024-06-29T15:57:36.330422+0000","last_fullsized":"2024-06-29T15:57:36.330422+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:48.678300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236409+0000","last_change":"2024-06-29T15:57:35.244368+0000","last_active":"2024-06-29T15:57:36.236409+0000","last_peered":"2024-06-29T15:57:36.236409+0000","last_clean":"2024-06-29T15:57:36.236409+0000","last_became_active":"2024-06-29T15:57:35.243599+0000","last_became_peered":"2024-06-29T15:57:35.243599+0000","last_unstale":"2024-06-29T15:57:36.236409+0000","last_undegraded":"2024-06-29T15:57:36.236409+0000","last_fullsized":"2024-06-29T15:57:36.236409+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:10.058134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1f","version":"221'4","reported_seq":26,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330083+0000","last_change":"2024-06-29T15:57:21.218674+0000","last_active":"2024-06-29T15:57:36.330083+0000","last_peered":"2024-06-29T15:57:36.330083+0000","last_clean":"2024-06-29T15:57:36.330083+0000","last_became_active":"2024-06-29T15:57:21.218397+0000","last_became_peered":"2024-06-29T15:57:21.218397+0000","last_unstale":"2024-06-29T15:57:36.330083+0000","last_undegraded":"2024-06-29T15:57:36.330083+0000","last_fullsized":"2024-06-29T15:57:36.330083+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":219,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.180095+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.180095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.766844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330985+0000","last_change":"2024-06-29T15:57:22.228934+0000","last_active":"2024-06-29T15:57:36.330985+0000","last_peered":"2024-06-29T15:57:36.330985+0000","last_clean":"2024-06-29T15:57:36.330985+0000","last_became_active":"2024-06-29T15:57:22.228603+0000","last_became_peered":"2024-06-29T15:57:22.228603+0000","last_unstale":"2024-06-29T15:57:36.330985+0000","last_undegraded":"2024-06-29T15:57:36.330985+0000","last_fullsized":"2024-06-29T15:57:36.330985+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:28.391534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329161+0000","last_change":"2024-06-29T15:57:34.310834+0000","last_active":"2024-06-29T15:57:36.329161+0000","last_peered":"2024-06-29T15:57:36.329161+0000","last_clean":"2024-06-29T15:57:36.329161+0000","last_became_active":"2024-06-29T15:57:34.310527+0000","last_became_peered":"2024-06-29T15:57:34.310527+0000","last_unstale":"2024-06-29T15:57:36.329161+0000","last_undegraded":"2024-06-29T15:57:36.329161+0000","last_fullsized":"2024-06-29T15:57:36.329161+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":223,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.271564+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.271564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:36.219011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239563+0000","last_change":"2024-06-29T15:57:35.256077+0000","last_active":"2024-06-29T15:57:36.239563+0000","last_peered":"2024-06-29T15:57:36.239563+0000","last_clean":"2024-06-29T15:57:36.239563+0000","last_became_active":"2024-06-29T15:57:35.255699+0000","last_became_peered":"2024-06-29T15:57:35.255699+0000","last_unstale":"2024-06-29T15:57:36.239563+0000","last_undegraded":"2024-06-29T15:57:36.239563+0000","last_fullsized":"2024-06-29T15:57:36.239563+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:30.089154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311494+0000","last_change":"2024-06-29T15:57:22.224277+0000","last_active":"2024-06-29T15:57:36.311494+0000","last_peered":"2024-06-29T15:57:36.311494+0000","last_clean":"2024-06-29T15:57:36.311494+0000","last_became_active":"2024-06-29T15:57:22.223975+0000","last_became_peered":"2024-06-29T15:57:22.223975+0000","last_unstale":"2024-06-29T15:57:36.311494+0000","last_undegraded":"2024-06-29T15:57:36.311494+0000","last_fullsized":"2024-06-29T15:57:36.311494+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:46.705935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329941+0000","last_change":"2024-06-29T15:57:35.257997+0000","last_active":"2024-06-29T15:57:36.329941+0000","last_peered":"2024-06-29T15:57:36.329941+0000","last_clean":"2024-06-29T15:57:36.329941+0000","last_became_active":"2024-06-29T15:57:35.254612+0000","last_became_peered":"2024-06-29T15:57:35.254612+0000","last_unstale":"2024-06-29T15:57:36.329941+0000","last_undegraded":"2024-06-29T15:57:36.329941+0000","last_fullsized":"2024-06-29T15:57:36.329941+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:13.347781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311649+0000","last_change":"2024-06-29T15:57:22.239789+0000","last_active":"2024-06-29T15:57:36.311649+0000","last_peered":"2024-06-29T15:57:36.311649+0000","last_clean":"2024-06-29T15:57:36.311649+0000","last_became_active":"2024-06-29T15:57:22.239438+0000","last_became_peered":"2024-06-29T15:57:22.239438+0000","last_unstale":"2024-06-29T15:57:36.311649+0000","last_undegraded":"2024-06-29T15:57:36.311649+0000","last_fullsized":"2024-06-29T15:57:36.311649+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:08.408484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237238+0000","last_change":"2024-06-29T15:57:35.251148+0000","last_active":"2024-06-29T15:57:36.237238+0000","last_peered":"2024-06-29T15:57:36.237238+0000","last_clean":"2024-06-29T15:57:36.237238+0000","last_became_active":"2024-06-29T15:57:35.250806+0000","last_became_peered":"2024-06-29T15:57:35.250806+0000","last_unstale":"2024-06-29T15:57:36.237238+0000","last_undegraded":"2024-06-29T15:57:36.237238+0000","last_fullsized":"2024-06-29T15:57:36.237238+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:54.987013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329513+0000","last_change":"2024-06-29T15:57:22.236151+0000","last_active":"2024-06-29T15:57:36.329513+0000","last_peered":"2024-06-29T15:57:36.329513+0000","last_clean":"2024-06-29T15:57:36.329513+0000","last_became_active":"2024-06-29T15:57:22.235409+0000","last_became_peered":"2024-06-29T15:57:22.235409+0000","last_unstale":"2024-06-29T15:57:36.329513+0000","last_undegraded":"2024-06-29T15:57:36.329513+0000","last_fullsized":"2024-06-29T15:57:36.329513+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:56.149438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329621+0000","last_change":"2024-06-29T15:57:35.255181+0000","last_active":"2024-06-29T15:57:36.329621+0000","last_peered":"2024-06-29T15:57:36.329621+0000","last_clean":"2024-06-29T15:57:36.329621+0000","last_became_active":"2024-06-29T15:57:35.254686+0000","last_became_peered":"2024-06-29T15:57:35.254686+0000","last_unstale":"2024-06-29T15:57:36.329621+0000","last_undegraded":"2024-06-29T15:57:36.329621+0000","last_fullsized":"2024-06-29T15:57:36.329621+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:52.833232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329737+0000","last_change":"2024-06-29T15:57:22.227876+0000","last_active":"2024-06-29T15:57:36.329737+0000","last_peered":"2024-06-29T15:57:36.329737+0000","last_clean":"2024-06-29T15:57:36.329737+0000","last_became_active":"2024-06-29T15:57:22.227591+0000","last_became_peered":"2024-06-29T15:57:22.227591+0000","last_unstale":"2024-06-29T15:57:36.329737+0000","last_undegraded":"2024-06-29T15:57:36.329737+0000","last_fullsized":"2024-06-29T15:57:36.329737+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:45.600754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332187+0000","last_change":"2024-06-29T15:57:35.257955+0000","last_active":"2024-06-29T15:57:36.332187+0000","last_peered":"2024-06-29T15:57:36.332187+0000","last_clean":"2024-06-29T15:57:36.332187+0000","last_became_active":"2024-06-29T15:57:35.256657+0000","last_became_peered":"2024-06-29T15:57:35.256657+0000","last_unstale":"2024-06-29T15:57:36.332187+0000","last_undegraded":"2024-06-29T15:57:36.332187+0000","last_fullsized":"2024-06-29T15:57:36.332187+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:57.126833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330655+0000","last_change":"2024-06-29T15:57:22.227845+0000","last_active":"2024-06-29T15:57:36.330655+0000","last_peered":"2024-06-29T15:57:36.330655+0000","last_clean":"2024-06-29T15:57:36.330655+0000","last_became_active":"2024-06-29T15:57:22.227549+0000","last_became_peered":"2024-06-29T15:57:22.227549+0000","last_unstale":"2024-06-29T15:57:36.330655+0000","last_undegraded":"2024-06-29T15:57:36.330655+0000","last_fullsized":"2024-06-29T15:57:36.330655+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:22.735783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312767+0000","last_change":"2024-06-29T15:57:35.258694+0000","last_active":"2024-06-29T15:57:36.312767+0000","last_peered":"2024-06-29T15:57:36.312767+0000","last_clean":"2024-06-29T15:57:36.312767+0000","last_became_active":"2024-06-29T15:57:35.258369+0000","last_became_peered":"2024-06-29T15:57:35.258369+0000","last_unstale":"2024-06-29T15:57:36.312767+0000","last_undegraded":"2024-06-29T15:57:36.312767+0000","last_fullsized":"2024-06-29T15:57:36.312767+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:30.546092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237188+0000","last_change":"2024-06-29T15:57:22.236939+0000","last_active":"2024-06-29T15:57:36.237188+0000","last_peered":"2024-06-29T15:57:36.237188+0000","last_clean":"2024-06-29T15:57:36.237188+0000","last_became_active":"2024-06-29T15:57:22.236129+0000","last_became_peered":"2024-06-29T15:57:22.236129+0000","last_unstale":"2024-06-29T15:57:36.237188+0000","last_undegraded":"2024-06-29T15:57:36.237188+0000","last_fullsized":"2024-06-29T15:57:36.237188+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:13.274461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239151+0000","last_change":"2024-06-29T15:57:35.248969+0000","last_active":"2024-06-29T15:57:36.239151+0000","last_peered":"2024-06-29T15:57:36.239151+0000","last_clean":"2024-06-29T15:57:36.239151+0000","last_became_active":"2024-06-29T15:57:35.248617+0000","last_became_peered":"2024-06-29T15:57:35.248617+0000","last_unstale":"2024-06-29T15:57:36.239151+0000","last_undegraded":"2024-06-29T15:57:36.239151+0000","last_fullsized":"2024-06-29T15:57:36.239151+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:23.344350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238364+0000","last_change":"2024-06-29T15:57:22.225280+0000","last_active":"2024-06-29T15:57:36.238364+0000","last_peered":"2024-06-29T15:57:36.238364+0000","last_clean":"2024-06-29T15:57:36.238364+0000","last_became_active":"2024-06-29T15:57:22.224986+0000","last_became_peered":"2024-06-29T15:57:22.224986+0000","last_unstale":"2024-06-29T15:57:36.238364+0000","last_undegraded":"2024-06-29T15:57:36.238364+0000","last_fullsized":"2024-06-29T15:57:36.238364+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:07.391769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329941+0000","last_change":"2024-06-29T15:57:35.249995+0000","last_active":"2024-06-29T15:57:36.329941+0000","last_peered":"2024-06-29T15:57:36.329941+0000","last_clean":"2024-06-29T15:57:36.329941+0000","last_became_active":"2024-06-29T15:57:35.249082+0000","last_became_peered":"2024-06-29T15:57:35.249082+0000","last_unstale":"2024-06-29T15:57:36.329941+0000","last_undegraded":"2024-06-29T15:57:36.329941+0000","last_fullsized":"2024-06-29T15:57:36.329941+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:18.648852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330431+0000","last_change":"2024-06-29T15:57:22.234902+0000","last_active":"2024-06-29T15:57:36.330431+0000","last_peered":"2024-06-29T15:57:36.330431+0000","last_clean":"2024-06-29T15:57:36.330431+0000","last_became_active":"2024-06-29T15:57:22.233244+0000","last_became_peered":"2024-06-29T15:57:22.233244+0000","last_unstale":"2024-06-29T15:57:36.330431+0000","last_undegraded":"2024-06-29T15:57:36.330431+0000","last_fullsized":"2024-06-29T15:57:36.330431+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:58.786612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236210+0000","last_change":"2024-06-29T15:57:35.243088+0000","last_active":"2024-06-29T15:57:36.236210+0000","last_peered":"2024-06-29T15:57:36.236210+0000","last_clean":"2024-06-29T15:57:36.236210+0000","last_became_active":"2024-06-29T15:57:35.242768+0000","last_became_peered":"2024-06-29T15:57:35.242768+0000","last_unstale":"2024-06-29T15:57:36.236210+0000","last_undegraded":"2024-06-29T15:57:36.236210+0000","last_fullsized":"2024-06-29T15:57:36.236210+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:43.817141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329319+0000","last_change":"2024-06-29T15:57:22.223814+0000","last_active":"2024-06-29T15:57:36.329319+0000","last_peered":"2024-06-29T15:57:36.329319+0000","last_clean":"2024-06-29T15:57:36.329319+0000","last_became_active":"2024-06-29T15:57:22.223464+0000","last_became_peered":"2024-06-29T15:57:22.223464+0000","last_unstale":"2024-06-29T15:57:36.329319+0000","last_undegraded":"2024-06-29T15:57:36.329319+0000","last_fullsized":"2024-06-29T15:57:36.329319+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:55:40.015999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331060+0000","last_change":"2024-06-29T15:57:35.256428+0000","last_active":"2024-06-29T15:57:36.331060+0000","last_peered":"2024-06-29T15:57:36.331060+0000","last_clean":"2024-06-29T15:57:36.331060+0000","last_became_active":"2024-06-29T15:57:35.255357+0000","last_became_peered":"2024-06-29T15:57:35.255357+0000","last_unstale":"2024-06-29T15:57:36.331060+0000","last_undegraded":"2024-06-29T15:57:36.331060+0000","last_fullsized":"2024-06-29T15:57:36.331060+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:10:51.973923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331341+0000","last_change":"2024-06-29T15:57:22.232432+0000","last_active":"2024-06-29T15:57:36.331341+0000","last_peered":"2024-06-29T15:57:36.331341+0000","last_clean":"2024-06-29T15:57:36.331341+0000","last_became_active":"2024-06-29T15:57:22.232102+0000","last_became_peered":"2024-06-29T15:57:22.232102+0000","last_unstale":"2024-06-29T15:57:36.331341+0000","last_undegraded":"2024-06-29T15:57:36.331341+0000","last_fullsized":"2024-06-29T15:57:36.331341+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:41.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311586+0000","last_change":"2024-06-29T15:57:35.259401+0000","last_active":"2024-06-29T15:57:36.311586+0000","last_peered":"2024-06-29T15:57:36.311586+0000","last_clean":"2024-06-29T15:57:36.311586+0000","last_became_active":"2024-06-29T15:57:35.259002+0000","last_became_peered":"2024-06-29T15:57:35.259002+0000","last_unstale":"2024-06-29T15:57:36.311586+0000","last_undegraded":"2024-06-29T15:57:36.311586+0000","last_fullsized":"2024-06-29T15:57:36.311586+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:34.435763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238215+0000","last_change":"2024-06-29T15:57:22.222075+0000","last_active":"2024-06-29T15:57:36.238215+0000","last_peered":"2024-06-29T15:57:36.238215+0000","last_clean":"2024-06-29T15:57:36.238215+0000","last_became_active":"2024-06-29T15:57:22.221767+0000","last_became_peered":"2024-06-29T15:57:22.221767+0000","last_unstale":"2024-06-29T15:57:36.238215+0000","last_undegraded":"2024-06-29T15:57:36.238215+0000","last_fullsized":"2024-06-29T15:57:36.238215+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:13.762887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329358+0000","last_change":"2024-06-29T15:57:35.249923+0000","last_active":"2024-06-29T15:57:36.329358+0000","last_peered":"2024-06-29T15:57:36.329358+0000","last_clean":"2024-06-29T15:57:36.329358+0000","last_became_active":"2024-06-29T15:57:35.249561+0000","last_became_peered":"2024-06-29T15:57:35.249561+0000","last_unstale":"2024-06-29T15:57:36.329358+0000","last_undegraded":"2024-06-29T15:57:36.329358+0000","last_fullsized":"2024-06-29T15:57:36.329358+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:58.917763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239963+0000","last_change":"2024-06-29T15:57:22.224310+0000","last_active":"2024-06-29T15:57:36.239963+0000","last_peered":"2024-06-29T15:57:36.239963+0000","last_clean":"2024-06-29T15:57:36.239963+0000","last_became_active":"2024-06-29T15:57:22.223959+0000","last_became_peered":"2024-06-29T15:57:22.223959+0000","last_unstale":"2024-06-29T15:57:36.239963+0000","last_undegraded":"2024-06-29T15:57:36.239963+0000","last_fullsized":"2024-06-29T15:57:36.239963+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:13.811309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332185+0000","last_change":"2024-06-29T15:57:35.258233+0000","last_active":"2024-06-29T15:57:36.332185+0000","last_peered":"2024-06-29T15:57:36.332185+0000","last_clean":"2024-06-29T15:57:36.332185+0000","last_became_active":"2024-06-29T15:57:35.256283+0000","last_became_peered":"2024-06-29T15:57:35.256283+0000","last_unstale":"2024-06-29T15:57:36.332185+0000","last_undegraded":"2024-06-29T15:57:36.332185+0000","last_fullsized":"2024-06-29T15:57:36.332185+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:49.167991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239662+0000","last_change":"2024-06-29T15:57:22.215484+0000","last_active":"2024-06-29T15:57:36.239662+0000","last_peered":"2024-06-29T15:57:36.239662+0000","last_clean":"2024-06-29T15:57:36.239662+0000","last_became_active":"2024-06-29T15:57:22.215179+0000","last_became_peered":"2024-06-29T15:57:22.215179+0000","last_unstale":"2024-06-29T15:57:36.239662+0000","last_undegraded":"2024-06-29T15:57:36.239662+0000","last_fullsized":"2024-06-29T15:57:36.239662+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:26:29.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239331+0000","last_change":"2024-06-29T15:57:35.258740+0000","last_active":"2024-06-29T15:57:36.239331+0000","last_peered":"2024-06-29T15:57:36.239331+0000","last_clean":"2024-06-29T15:57:36.239331+0000","last_became_active":"2024-06-29T15:57:35.258400+0000","last_became_peered":"2024-06-29T15:57:35.258400+0000","last_unstale":"2024-06-29T15:57:36.239331+0000","last_undegraded":"2024-06-29T15:57:36.239331+0000","last_fullsized":"2024-06-29T15:57:36.239331+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:33.378531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238877+0000","last_change":"2024-06-29T15:57:22.237289+0000","last_active":"2024-06-29T15:57:36.238877+0000","last_peered":"2024-06-29T15:57:36.238877+0000","last_clean":"2024-06-29T15:57:36.238877+0000","last_became_active":"2024-06-29T15:57:22.236267+0000","last_became_peered":"2024-06-29T15:57:22.236267+0000","last_unstale":"2024-06-29T15:57:36.238877+0000","last_undegraded":"2024-06-29T15:57:36.238877+0000","last_fullsized":"2024-06-29T15:57:36.238877+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:53:11.467204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.240048+0000","last_change":"2024-06-29T15:57:35.243220+0000","last_active":"2024-06-29T15:57:36.240048+0000","last_peered":"2024-06-29T15:57:36.240048+0000","last_clean":"2024-06-29T15:57:36.240048+0000","last_became_active":"2024-06-29T15:57:35.242914+0000","last_became_peered":"2024-06-29T15:57:35.242914+0000","last_unstale":"2024-06-29T15:57:36.240048+0000","last_undegraded":"2024-06-29T15:57:36.240048+0000","last_fullsized":"2024-06-29T15:57:36.240048+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:17:16.068033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330080+0000","last_change":"2024-06-29T15:57:22.239205+0000","last_active":"2024-06-29T15:57:36.330080+0000","last_peered":"2024-06-29T15:57:36.330080+0000","last_clean":"2024-06-29T15:57:36.330080+0000","last_became_active":"2024-06-29T15:57:22.238830+0000","last_became_peered":"2024-06-29T15:57:22.238830+0000","last_unstale":"2024-06-29T15:57:36.330080+0000","last_undegraded":"2024-06-29T15:57:36.330080+0000","last_fullsized":"2024-06-29T15:57:36.330080+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.228440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312824+0000","last_change":"2024-06-29T15:57:35.256144+0000","last_active":"2024-06-29T15:57:36.312824+0000","last_peered":"2024-06-29T15:57:36.312824+0000","last_clean":"2024-06-29T15:57:36.312824+0000","last_became_active":"2024-06-29T15:57:35.255541+0000","last_became_peered":"2024-06-29T15:57:35.255541+0000","last_unstale":"2024-06-29T15:57:36.312824+0000","last_undegraded":"2024-06-29T15:57:36.312824+0000","last_fullsized":"2024-06-29T15:57:36.312824+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:23:28.979863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330285+0000","last_change":"2024-06-29T15:57:22.235020+0000","last_active":"2024-06-29T15:57:36.330285+0000","last_peered":"2024-06-29T15:57:36.330285+0000","last_clean":"2024-06-29T15:57:36.330285+0000","last_became_active":"2024-06-29T15:57:22.234417+0000","last_became_peered":"2024-06-29T15:57:22.234417+0000","last_unstale":"2024-06-29T15:57:36.330285+0000","last_undegraded":"2024-06-29T15:57:36.330285+0000","last_fullsized":"2024-06-29T15:57:36.330285+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:06.264167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237895+0000","last_change":"2024-06-29T15:57:35.249930+0000","last_active":"2024-06-29T15:57:36.237895+0000","last_peered":"2024-06-29T15:57:36.237895+0000","last_clean":"2024-06-29T15:57:36.237895+0000","last_became_active":"2024-06-29T15:57:35.249298+0000","last_became_peered":"2024-06-29T15:57:35.249298+0000","last_unstale":"2024-06-29T15:57:36.237895+0000","last_undegraded":"2024-06-29T15:57:36.237895+0000","last_fullsized":"2024-06-29T15:57:36.237895+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.903736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.235930+0000","last_change":"2024-06-29T15:57:22.238113+0000","last_active":"2024-06-29T15:57:36.235930+0000","last_peered":"2024-06-29T15:57:36.235930+0000","last_clean":"2024-06-29T15:57:36.235930+0000","last_became_active":"2024-06-29T15:57:22.237726+0000","last_became_peered":"2024-06-29T15:57:22.237726+0000","last_unstale":"2024-06-29T15:57:36.235930+0000","last_undegraded":"2024-06-29T15:57:36.235930+0000","last_fullsized":"2024-06-29T15:57:36.235930+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:08:48.416318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239184+0000","last_change":"2024-06-29T15:57:35.247761+0000","last_active":"2024-06-29T15:57:36.239184+0000","last_peered":"2024-06-29T15:57:36.239184+0000","last_clean":"2024-06-29T15:57:36.239184+0000","last_became_active":"2024-06-29T15:57:35.247325+0000","last_became_peered":"2024-06-29T15:57:35.247325+0000","last_unstale":"2024-06-29T15:57:36.239184+0000","last_undegraded":"2024-06-29T15:57:36.239184+0000","last_fullsized":"2024-06-29T15:57:36.239184+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:55:34.536510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329410+0000","last_change":"2024-06-29T15:57:22.220754+0000","last_active":"2024-06-29T15:57:36.329410+0000","last_peered":"2024-06-29T15:57:36.329410+0000","last_clean":"2024-06-29T15:57:36.329410+0000","last_became_active":"2024-06-29T15:57:22.220438+0000","last_became_peered":"2024-06-29T15:57:22.220438+0000","last_unstale":"2024-06-29T15:57:36.329410+0000","last_undegraded":"2024-06-29T15:57:36.329410+0000","last_fullsized":"2024-06-29T15:57:36.329410+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:26.314927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237379+0000","last_change":"2024-06-29T15:57:35.241980+0000","last_active":"2024-06-29T15:57:36.237379+0000","last_peered":"2024-06-29T15:57:36.237379+0000","last_clean":"2024-06-29T15:57:36.237379+0000","last_became_active":"2024-06-29T15:57:35.241607+0000","last_became_peered":"2024-06-29T15:57:35.241607+0000","last_unstale":"2024-06-29T15:57:36.237379+0000","last_undegraded":"2024-06-29T15:57:36.237379+0000","last_fullsized":"2024-06-29T15:57:36.237379+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:51.022053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238091+0000","last_change":"2024-06-29T15:57:22.225596+0000","last_active":"2024-06-29T15:57:36.238091+0000","last_peered":"2024-06-29T15:57:36.238091+0000","last_clean":"2024-06-29T15:57:36.238091+0000","last_became_active":"2024-06-29T15:57:22.225154+0000","last_became_peered":"2024-06-29T15:57:22.225154+0000","last_unstale":"2024-06-29T15:57:36.238091+0000","last_undegraded":"2024-06-29T15:57:36.238091+0000","last_fullsized":"2024-06-29T15:57:36.238091+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:01.272853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.236100+0000","last_change":"2024-06-29T15:57:35.246784+0000","last_active":"2024-06-29T15:57:36.236100+0000","last_peered":"2024-06-29T15:57:36.236100+0000","last_clean":"2024-06-29T15:57:36.236100+0000","last_became_active":"2024-06-29T15:57:35.246467+0000","last_became_peered":"2024-06-29T15:57:35.246467+0000","last_unstale":"2024-06-29T15:57:36.236100+0000","last_undegraded":"2024-06-29T15:57:36.236100+0000","last_fullsized":"2024-06-29T15:57:36.236100+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:46.600461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311897+0000","last_change":"2024-06-29T15:57:22.237433+0000","last_active":"2024-06-29T15:57:36.311897+0000","last_peered":"2024-06-29T15:57:36.311897+0000","last_clean":"2024-06-29T15:57:36.311897+0000","last_became_active":"2024-06-29T15:57:22.237070+0000","last_became_peered":"2024-06-29T15:57:22.237070+0000","last_unstale":"2024-06-29T15:57:36.311897+0000","last_undegraded":"2024-06-29T15:57:36.311897+0000","last_fullsized":"2024-06-29T15:57:36.311897+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:42:25.658901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332066+0000","last_change":"2024-06-29T15:57:35.251241+0000","last_active":"2024-06-29T15:57:36.332066+0000","last_peered":"2024-06-29T15:57:36.332066+0000","last_clean":"2024-06-29T15:57:36.332066+0000","last_became_active":"2024-06-29T15:57:35.248566+0000","last_became_peered":"2024-06-29T15:57:35.248566+0000","last_unstale":"2024-06-29T15:57:36.332066+0000","last_undegraded":"2024-06-29T15:57:36.332066+0000","last_fullsized":"2024-06-29T15:57:36.332066+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:52.934381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330550+0000","last_change":"2024-06-29T15:57:22.231838+0000","last_active":"2024-06-29T15:57:36.330550+0000","last_peered":"2024-06-29T15:57:36.330550+0000","last_clean":"2024-06-29T15:57:36.330550+0000","last_became_active":"2024-06-29T15:57:22.231563+0000","last_became_peered":"2024-06-29T15:57:22.231563+0000","last_unstale":"2024-06-29T15:57:36.330550+0000","last_undegraded":"2024-06-29T15:57:36.330550+0000","last_fullsized":"2024-06-29T15:57:36.330550+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:23.251139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330914+0000","last_change":"2024-06-29T15:57:35.251166+0000","last_active":"2024-06-29T15:57:36.330914+0000","last_peered":"2024-06-29T15:57:36.330914+0000","last_clean":"2024-06-29T15:57:36.330914+0000","last_became_active":"2024-06-29T15:57:35.249802+0000","last_became_peered":"2024-06-29T15:57:35.249802+0000","last_unstale":"2024-06-29T15:57:36.330914+0000","last_undegraded":"2024-06-29T15:57:36.330914+0000","last_fullsized":"2024-06-29T15:57:36.330914+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:26.856925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237097+0000","last_change":"2024-06-29T15:57:22.237225+0000","last_active":"2024-06-29T15:57:36.237097+0000","last_peered":"2024-06-29T15:57:36.237097+0000","last_clean":"2024-06-29T15:57:36.237097+0000","last_became_active":"2024-06-29T15:57:22.236621+0000","last_became_peered":"2024-06-29T15:57:22.236621+0000","last_unstale":"2024-06-29T15:57:36.237097+0000","last_undegraded":"2024-06-29T15:57:36.237097+0000","last_fullsized":"2024-06-29T15:57:36.237097+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:26.582705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238326+0000","last_change":"2024-06-29T15:57:35.252791+0000","last_active":"2024-06-29T15:57:36.238326+0000","last_peered":"2024-06-29T15:57:36.238326+0000","last_clean":"2024-06-29T15:57:36.238326+0000","last_became_active":"2024-06-29T15:57:35.252002+0000","last_became_peered":"2024-06-29T15:57:35.252002+0000","last_unstale":"2024-06-29T15:57:36.238326+0000","last_undegraded":"2024-06-29T15:57:36.238326+0000","last_fullsized":"2024-06-29T15:57:36.238326+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:32.593283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.238829+0000","last_change":"2024-06-29T15:57:22.237316+0000","last_active":"2024-06-29T15:57:36.238829+0000","last_peered":"2024-06-29T15:57:36.238829+0000","last_clean":"2024-06-29T15:57:36.238829+0000","last_became_active":"2024-06-29T15:57:22.236756+0000","last_became_peered":"2024-06-29T15:57:22.236756+0000","last_unstale":"2024-06-29T15:57:36.238829+0000","last_undegraded":"2024-06-29T15:57:36.238829+0000","last_fullsized":"2024-06-29T15:57:36.238829+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:06.141583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329947+0000","last_change":"2024-06-29T15:57:35.254538+0000","last_active":"2024-06-29T15:57:36.329947+0000","last_peered":"2024-06-29T15:57:36.329947+0000","last_clean":"2024-06-29T15:57:36.329947+0000","last_became_active":"2024-06-29T15:57:35.254193+0000","last_became_peered":"2024-06-29T15:57:35.254193+0000","last_unstale":"2024-06-29T15:57:36.329947+0000","last_undegraded":"2024-06-29T15:57:36.329947+0000","last_fullsized":"2024-06-29T15:57:36.329947+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:00:23.012081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330101+0000","last_change":"2024-06-29T15:57:22.235210+0000","last_active":"2024-06-29T15:57:36.330101+0000","last_peered":"2024-06-29T15:57:36.330101+0000","last_clean":"2024-06-29T15:57:36.330101+0000","last_became_active":"2024-06-29T15:57:22.234573+0000","last_became_peered":"2024-06-29T15:57:22.234573+0000","last_unstale":"2024-06-29T15:57:36.330101+0000","last_undegraded":"2024-06-29T15:57:36.330101+0000","last_fullsized":"2024-06-29T15:57:36.330101+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:48.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239771+0000","last_change":"2024-06-29T15:57:35.252893+0000","last_active":"2024-06-29T15:57:36.239771+0000","last_peered":"2024-06-29T15:57:36.239771+0000","last_clean":"2024-06-29T15:57:36.239771+0000","last_became_active":"2024-06-29T15:57:35.252527+0000","last_became_peered":"2024-06-29T15:57:35.252527+0000","last_unstale":"2024-06-29T15:57:36.239771+0000","last_undegraded":"2024-06-29T15:57:36.239771+0000","last_fullsized":"2024-06-29T15:57:36.239771+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:02.911105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239077+0000","last_change":"2024-06-29T15:57:22.233275+0000","last_active":"2024-06-29T15:57:36.239077+0000","last_peered":"2024-06-29T15:57:36.239077+0000","last_clean":"2024-06-29T15:57:36.239077+0000","last_became_active":"2024-06-29T15:57:22.230801+0000","last_became_peered":"2024-06-29T15:57:22.230801+0000","last_unstale":"2024-06-29T15:57:36.239077+0000","last_undegraded":"2024-06-29T15:57:36.239077+0000","last_fullsized":"2024-06-29T15:57:36.239077+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:24.134069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.311790+0000","last_change":"2024-06-29T15:57:35.257258+0000","last_active":"2024-06-29T15:57:36.311790+0000","last_peered":"2024-06-29T15:57:36.311790+0000","last_clean":"2024-06-29T15:57:36.311790+0000","last_became_active":"2024-06-29T15:57:35.256939+0000","last_became_peered":"2024-06-29T15:57:35.256939+0000","last_unstale":"2024-06-29T15:57:36.311790+0000","last_undegraded":"2024-06-29T15:57:36.311790+0000","last_fullsized":"2024-06-29T15:57:36.311790+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:40:53.758735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329683+0000","last_change":"2024-06-29T15:57:22.223324+0000","last_active":"2024-06-29T15:57:36.329683+0000","last_peered":"2024-06-29T15:57:36.329683+0000","last_clean":"2024-06-29T15:57:36.329683+0000","last_became_active":"2024-06-29T15:57:22.222958+0000","last_became_peered":"2024-06-29T15:57:22.222958+0000","last_unstale":"2024-06-29T15:57:36.329683+0000","last_undegraded":"2024-06-29T15:57:36.329683+0000","last_fullsized":"2024-06-29T15:57:36.329683+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:21.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237061+0000","last_change":"2024-06-29T15:57:35.244095+0000","last_active":"2024-06-29T15:57:36.237061+0000","last_peered":"2024-06-29T15:57:36.237061+0000","last_clean":"2024-06-29T15:57:36.237061+0000","last_became_active":"2024-06-29T15:57:35.243771+0000","last_became_peered":"2024-06-29T15:57:35.243771+0000","last_unstale":"2024-06-29T15:57:36.237061+0000","last_undegraded":"2024-06-29T15:57:36.237061+0000","last_fullsized":"2024-06-29T15:57:36.237061+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:55.022325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312052+0000","last_change":"2024-06-29T15:57:22.226496+0000","last_active":"2024-06-29T15:57:36.312052+0000","last_peered":"2024-06-29T15:57:36.312052+0000","last_clean":"2024-06-29T15:57:36.312052+0000","last_became_active":"2024-06-29T15:57:22.226191+0000","last_became_peered":"2024-06-29T15:57:22.226191+0000","last_unstale":"2024-06-29T15:57:36.312052+0000","last_undegraded":"2024-06-29T15:57:36.312052+0000","last_fullsized":"2024-06-29T15:57:36.312052+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:32.644449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330874+0000","last_change":"2024-06-29T15:57:35.258649+0000","last_active":"2024-06-29T15:57:36.330874+0000","last_peered":"2024-06-29T15:57:36.330874+0000","last_clean":"2024-06-29T15:57:36.330874+0000","last_became_active":"2024-06-29T15:57:35.258325+0000","last_became_peered":"2024-06-29T15:57:35.258325+0000","last_unstale":"2024-06-29T15:57:36.330874+0000","last_undegraded":"2024-06-29T15:57:36.330874+0000","last_fullsized":"2024-06-29T15:57:36.330874+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:52.146056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.331112+0000","last_change":"2024-06-29T15:57:22.218369+0000","last_active":"2024-06-29T15:57:36.331112+0000","last_peered":"2024-06-29T15:57:36.331112+0000","last_clean":"2024-06-29T15:57:36.331112+0000","last_became_active":"2024-06-29T15:57:22.217584+0000","last_became_peered":"2024-06-29T15:57:22.217584+0000","last_unstale":"2024-06-29T15:57:36.331112+0000","last_undegraded":"2024-06-29T15:57:36.331112+0000","last_fullsized":"2024-06-29T15:57:36.331112+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:59.760761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.330020+0000","last_change":"2024-06-29T15:57:35.250646+0000","last_active":"2024-06-29T15:57:36.330020+0000","last_peered":"2024-06-29T15:57:36.330020+0000","last_clean":"2024-06-29T15:57:36.330020+0000","last_became_active":"2024-06-29T15:57:35.250307+0000","last_became_peered":"2024-06-29T15:57:35.250307+0000","last_unstale":"2024-06-29T15:57:36.330020+0000","last_undegraded":"2024-06-29T15:57:36.330020+0000","last_fullsized":"2024-06-29T15:57:36.330020+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:06.391987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312266+0000","last_change":"2024-06-29T15:57:22.236022+0000","last_active":"2024-06-29T15:57:36.312266+0000","last_peered":"2024-06-29T15:57:36.312266+0000","last_clean":"2024-06-29T15:57:36.312266+0000","last_became_active":"2024-06-29T15:57:22.235712+0000","last_became_peered":"2024-06-29T15:57:22.235712+0000","last_unstale":"2024-06-29T15:57:36.312266+0000","last_undegraded":"2024-06-29T15:57:36.312266+0000","last_fullsized":"2024-06-29T15:57:36.312266+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:57.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.312823+0000","last_change":"2024-06-29T15:57:35.256845+0000","last_active":"2024-06-29T15:57:36.312823+0000","last_peered":"2024-06-29T15:57:36.312823+0000","last_clean":"2024-06-29T15:57:36.312823+0000","last_became_active":"2024-06-29T15:57:35.256516+0000","last_became_peered":"2024-06-29T15:57:35.256516+0000","last_unstale":"2024-06-29T15:57:36.312823+0000","last_undegraded":"2024-06-29T15:57:36.312823+0000","last_fullsized":"2024-06-29T15:57:36.312823+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.581911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.237077+0000","last_change":"2024-06-29T15:57:22.231073+0000","last_active":"2024-06-29T15:57:36.237077+0000","last_peered":"2024-06-29T15:57:36.237077+0000","last_clean":"2024-06-29T15:57:36.237077+0000","last_became_active":"2024-06-29T15:57:22.230749+0000","last_became_peered":"2024-06-29T15:57:22.230749+0000","last_unstale":"2024-06-29T15:57:36.237077+0000","last_undegraded":"2024-06-29T15:57:36.237077+0000","last_fullsized":"2024-06-29T15:57:36.237077+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:23.168668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329839+0000","last_change":"2024-06-29T15:57:35.252089+0000","last_active":"2024-06-29T15:57:36.329839+0000","last_peered":"2024-06-29T15:57:36.329839+0000","last_clean":"2024-06-29T15:57:36.329839+0000","last_became_active":"2024-06-29T15:57:35.251775+0000","last_became_peered":"2024-06-29T15:57:35.251775+0000","last_unstale":"2024-06-29T15:57:36.329839+0000","last_undegraded":"2024-06-29T15:57:36.329839+0000","last_fullsized":"2024-06-29T15:57:36.329839+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:26.905348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.332427+0000","last_change":"2024-06-29T15:57:22.231733+0000","last_active":"2024-06-29T15:57:36.332427+0000","last_peered":"2024-06-29T15:57:36.332427+0000","last_clean":"2024-06-29T15:57:36.332427+0000","last_became_active":"2024-06-29T15:57:22.231387+0000","last_became_peered":"2024-06-29T15:57:22.231387+0000","last_unstale":"2024-06-29T15:57:36.332427+0000","last_undegraded":"2024-06-29T15:57:36.332427+0000","last_fullsized":"2024-06-29T15:57:36.332427+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:41.892974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.329977+0000","last_change":"2024-06-29T15:57:35.255683+0000","last_active":"2024-06-29T15:57:36.329977+0000","last_peered":"2024-06-29T15:57:36.329977+0000","last_clean":"2024-06-29T15:57:36.329977+0000","last_became_active":"2024-06-29T15:57:35.255328+0000","last_became_peered":"2024-06-29T15:57:35.255328+0000","last_unstale":"2024-06-29T15:57:36.329977+0000","last_undegraded":"2024-06-29T15:57:36.329977+0000","last_fullsized":"2024-06-29T15:57:36.329977+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:34.280660+0000","last_clean_scrub_stamp":"2024-06-29T15:57:34.280660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:35.404663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":20,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.239593+0000","last_change":"2024-06-29T15:57:22.225592+0000","last_active":"2024-06-29T15:57:36.239593+0000","last_peered":"2024-06-29T15:57:36.239593+0000","last_clean":"2024-06-29T15:57:36.239593+0000","last_became_active":"2024-06-29T15:57:22.225309+0000","last_became_peered":"2024-06-29T15:57:22.225309+0000","last_unstale":"2024-06-29T15:57:36.239593+0000","last_undegraded":"2024-06-29T15:57:36.239593+0000","last_fullsized":"2024-06-29T15:57:36.239593+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.186622+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.186622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:07.857349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543291,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52596,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319244,"statfs":{"total":96636764160,"available":96582905856,"internally_reserved":0,"allocated":3448832,"data_stored":2054220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543292,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42312,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329528,"statfs":{"total":96636764160,"available":96593436672,"internally_reserved":0,"allocated":2871296,"data_stored":1465430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543291,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47864,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323976,"statfs":{"total":96636764160,"available":96587751424,"internally_reserved":0,"allocated":2854912,"data_stored":1461993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543290,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52004,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319836,"statfs":{"total":96636764160,"available":96583512064,"internally_reserved":0,"allocated":2838528,"data_stored":1460927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510588,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47856,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323984,"statfs":{"total":96636764160,"available":96587759616,"internally_reserved":0,"allocated":2846720,"data_stored":1460403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575996,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47844,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323996,"statfs":{"total":96636764160,"available":96587771904,"internally_reserved":0,"allocated":2842624,"data_stored":1460961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510587,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47832,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324008,"statfs":{"total":96636764160,"available":96587784192,"internally_reserved":0,"allocated":2830336,"data_stored":1460255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510589,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42852,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328988,"statfs":{"total":96636764160,"available":96592883712,"internally_reserved":0,"allocated":3428352,"data_stored":2050713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:44.329 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:57:44.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:44.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.479+0000 7f62af410640 1 -- 172.21.15.59:0/977563700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62a80b2270 msgr2=0x7f62a815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.479+0000 7f62af410640 1 --2- 172.21.15.59:0/977563700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62a80b2270 0x7f62a815bc20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f62a4005fd0 tx=0x7f62a402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.479+0000 7f62af410640 1 -- 172.21.15.59:0/977563700 shutdown_connections 2024-06-29T15:57:44.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.479+0000 7f62af410640 1 --2- 172.21.15.59:0/977563700 >> v2:172.21.15.153:3300/0 conn(0x7f62a80d1c30 0x7f62a80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.479+0000 7f62af410640 1 --2- 172.21.15.59:0/977563700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62a80b2270 0x7f62a815bc20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.479+0000 7f62af410640 1 -- 172.21.15.59:0/977563700 >> 172.21.15.59:0/977563700 conn(0x7f62a8073e10 msgr2=0x7f62a8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.479+0000 7f62af410640 1 -- 172.21.15.59:0/977563700 shutdown_connections 2024-06-29T15:57:44.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.479+0000 7f62af410640 1 -- 172.21.15.59:0/977563700 wait complete. 2024-06-29T15:57:44.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.481+0000 7f62af410640 1 Processor -- start 2024-06-29T15:57:44.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.481+0000 7f62af410640 1 -- start start 2024-06-29T15:57:44.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62af410640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80b2270 0x7f62a814c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62af410640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62a80b2fa0 0x7f62a80b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62af410640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a80d1c30 0x7f62a80cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62af410640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f62a80b6400 con 0x7f62a80b2270 2024-06-29T15:57:44.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62af410640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62a80b6280 con 0x7f62a80d1c30 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62af410640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f62a80b6580 con 0x7f62a80b2fa0 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad1ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80b2270 0x7f62a814c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad1ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80b2270 0x7f62a814c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54588/0 (socket says 172.21.15.59:54588) 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad1ac640 1 -- 172.21.15.59:0/2037136455 learned_addr learned my addr 172.21.15.59:0/2037136455 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad9ad640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a80d1c30 0x7f62a80cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ac9ab640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62a80b2fa0 0x7f62a80b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad1ac640 1 -- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62a80b2fa0 msgr2=0x7f62a80b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad1ac640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62a80b2fa0 0x7f62a80b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad1ac640 1 -- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a80d1c30 msgr2=0x7f62a80cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad1ac640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a80d1c30 0x7f62a80cfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad1ac640 1 -- 172.21.15.59:0/2037136455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a80b0a70 con 0x7f62a80b2270 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.482+0000 7f62ad9ad640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a80d1c30 0x7f62a80cfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:44.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.483+0000 7f62ad1ac640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80b2270 0x7f62a814c350 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f62a4034040 tx=0x7f62a40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.483+0000 7f629dffb640 1 -- 172.21.15.59:0/2037136455 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a403e030 con 0x7f62a80b2270 2024-06-29T15:57:44.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.483+0000 7f629dffb640 1 -- 172.21.15.59:0/2037136455 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62a4003ae0 con 0x7f62a80b2270 2024-06-29T15:57:44.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.483+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a80ab0e0 con 0x7f62a80b2270 2024-06-29T15:57:44.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.483+0000 7f629dffb640 1 -- 172.21.15.59:0/2037136455 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4035040 con 0x7f62a80b2270 2024-06-29T15:57:44.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.483+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a80ae1e0 con 0x7f62a80b2270 2024-06-29T15:57:44.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.484+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a80b6ac0 con 0x7f62a80b2270 2024-06-29T15:57:44.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.485+0000 7f629dffb640 1 -- 172.21.15.59:0/2037136455 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f62a4041030 con 0x7f62a80b2270 2024-06-29T15:57:44.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.485+0000 7f629dffb640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f628c0763f0 0x7f628c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.485+0000 7f629dffb640 1 -- 172.21.15.59:0/2037136455 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f62a40b8470 con 0x7f62a80b2270 2024-06-29T15:57:44.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.486+0000 7f62ac9ab640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f628c0763f0 0x7f628c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.486+0000 7f62ac9ab640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f628c0763f0 0x7f628c078900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f62a81f6070 tx=0x7f6298009040 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.490+0000 7f629dffb640 1 -- 172.21.15.59:0/2037136455 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f62a40815f0 con 0x7f62a80b2270 2024-06-29T15:57:44.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.717+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62a80b5c40 con 0x7f62a80b2270 2024-06-29T15:57:44.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.718+0000 7f629dffb640 1 -- 172.21.15.59:0/2037136455 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6567 (secure 0 0 0) 0x7f62a4080ef0 con 0x7f62a80b2270 2024-06-29T15:57:44.719 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:44.719 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":121,"btime":"2024-06-29T15:57:38:315315+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.589050+0000","modified":"2024-06-29T15:57:24.623107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11487":{"gid":11487,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1734341893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1734341893},{"type":"v1","addr":"172.21.15.59:6834","nonce":1734341893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11487,"qdb_cluster":[11487]},"id":13},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:57:44.719 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 121 2024-06-29T15:57:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f628c0763f0 msgr2=0x7f628c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f628c0763f0 0x7f628c078900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f62a81f6070 tx=0x7f6298009040 comp rx=0 tx=0).stop 2024-06-29T15:57:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80b2270 msgr2=0x7f62a814c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80b2270 0x7f62a814c350 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f62a4034040 tx=0x7f62a40035f0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 shutdown_connections 2024-06-29T15:57:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f628c0763f0 0x7f628c078900 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62a80d1c30 0x7f62a80cfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f62a80b2fa0 0x7f62a80b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 --2- 172.21.15.59:0/2037136455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80b2270 0x7f62a814c350 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 >> 172.21.15.59:0/2037136455 conn(0x7f62a8073e10 msgr2=0x7f62a815b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 shutdown_connections 2024-06-29T15:57:44.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:44.720+0000 7f62af410640 1 -- 172.21.15.59:0/2037136455 wait complete. 2024-06-29T15:57:45.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:57:34.679058+0000', 'modified': '2024-06-29T15:57:37.701097+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10747}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10747': {'gid': 10747, 'name': 'b', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.153:6837/601671949', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6835', 'nonce': 601671949}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 601671949}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10747, 'qdb_cluster': [10747]} 2024-06-29T15:57:45.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:57:45.134 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T15:57:45.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.255+0000 7f283db03640 1 -- 172.21.15.59:0/2322924153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2838162190 msgr2=0x7f28380b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.255+0000 7f283db03640 1 --2- 172.21.15.59:0/2322924153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2838162190 0x7f28380b38e0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f2830005fd0 tx=0x7f283002a810 comp rx=0 tx=0).stop 2024-06-29T15:57:45.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.256+0000 7f283db03640 1 -- 172.21.15.59:0/2322924153 shutdown_connections 2024-06-29T15:57:45.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.256+0000 7f283db03640 1 --2- 172.21.15.59:0/2322924153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2838162190 0x7f28380b38e0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.256+0000 7f283db03640 1 --2- 172.21.15.59:0/2322924153 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2838162770 0x7f28380ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.256+0000 7f283db03640 1 -- 172.21.15.59:0/2322924153 >> 172.21.15.59:0/2322924153 conn(0x7f28380d45b0 msgr2=0x7f28380d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.256+0000 7f283db03640 1 -- 172.21.15.59:0/2322924153 shutdown_connections 2024-06-29T15:57:45.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.256+0000 7f283db03640 1 -- 172.21.15.59:0/2322924153 wait complete. 2024-06-29T15:57:45.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.257+0000 7f283db03640 1 Processor -- start 2024-06-29T15:57:45.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.258+0000 7f283db03640 1 -- start start 2024-06-29T15:57:45.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f283db03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f283815e0a0 0x7f28381f7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f283db03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2838162770 0x7f28381f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f283db03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2838155890 0x7f28380cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f283db03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f28380b42f0 con 0x7f2838162770 2024-06-29T15:57:45.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f283db03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28380b4170 con 0x7f2838155890 2024-06-29T15:57:45.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f283db03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f28380b4470 con 0x7f283815e0a0 2024-06-29T15:57:45.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f282effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2838162770 0x7f28381f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f282effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2838162770 0x7f28381f5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54602/0 (socket says 172.21.15.59:54602) 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f282effd640 1 -- 172.21.15.59:0/1659902664 learned_addr learned my addr 172.21.15.59:0/1659902664 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f28377fe640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f283815e0a0 0x7f28381f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f2837fff640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2838155890 0x7f28380cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f282effd640 1 -- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f283815e0a0 msgr2=0x7f28381f7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f282effd640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f283815e0a0 0x7f28381f7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f282effd640 1 -- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2838155890 msgr2=0x7f28380cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f282effd640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2838155890 0x7f28380cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.259+0000 7f282effd640 1 -- 172.21.15.59:0/1659902664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2838156070 con 0x7f2838162770 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.260+0000 7f282effd640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2838162770 0x7f28381f5550 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f282000c3c0 tx=0x7f282000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.260+0000 7f2834ff9640 1 -- 172.21.15.59:0/1659902664 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28200052d0 con 0x7f2838162770 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.260+0000 7f2834ff9640 1 -- 172.21.15.59:0/1659902664 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2820007020 con 0x7f2838162770 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.260+0000 7f283db03640 1 -- 172.21.15.59:0/1659902664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2838156f70 con 0x7f2838162770 2024-06-29T15:57:45.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.260+0000 7f2834ff9640 1 -- 172.21.15.59:0/1659902664 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2820007470 con 0x7f2838162770 2024-06-29T15:57:45.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.260+0000 7f283db03640 1 -- 172.21.15.59:0/1659902664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28381574f0 con 0x7f2838162770 2024-06-29T15:57:45.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.262+0000 7f2834ff9640 1 -- 172.21.15.59:0/1659902664 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f28200067d0 con 0x7f2838162770 2024-06-29T15:57:45.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.262+0000 7f2834ff9640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28180763f0 0x7f2818078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.262+0000 7f2834ff9640 1 -- 172.21.15.59:0/1659902664 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f2820097170 con 0x7f2838162770 2024-06-29T15:57:45.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.263+0000 7f28377fe640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28180763f0 0x7f2818078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.263+0000 7f28449c3740 1 Processor -- start 2024-06-29T15:57:45.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.263+0000 7f28449c3740 1 -- start start 2024-06-29T15:57:45.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.263+0000 7f28377fe640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28180763f0 0x7f2818078900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2828009530 tx=0x7f2828009260 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f28449c3740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f28449c3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 0x5592397a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f28449c3740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x5592397a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f28449c3740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x559239739bc0 con 0x5592393b0590 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f28449c3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5592396e8630 con 0x5592393a83b0 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f28449c3740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5592396d7bd0 con 0x5592393d1270 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x5592397a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x5592397a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54604/0 (socket says 172.21.15.59:54604) 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 -- 172.21.15.59:0/780104290 learned_addr learned my addr 172.21.15.59:0/780104290 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f28377fe640 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f282effd640 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 0x5592397a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 -- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 msgr2=0x5592397bd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 -- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 msgr2=0x5592397a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 0x5592397a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 -- 172.21.15.59:0/780104290 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592396d4ca0 con 0x5592393b0590 2024-06-29T15:57:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f282effd640 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 0x5592397a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f2837fff640 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x5592397a18e0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f2830009cd0 tx=0x7f2830004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f250bd255cc323 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f282d7fa640 1 -- 172.21.15.59:0/780104290 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830037040 con 0x5592393b0590 2024-06-29T15:57:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f282d7fa640 1 -- 172.21.15.59:0/780104290 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2830032be0 con 0x5592393b0590 2024-06-29T15:57:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.265+0000 7f282d7fa640 1 -- 172.21.15.59:0/780104290 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830046430 con 0x5592393b0590 2024-06-29T15:57:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 -- 172.21.15.59:0/780104290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 msgr2=0x5592397a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x5592397a18e0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f2830009cd0 tx=0x7f2830004a40 comp rx=0 tx=0).stop 2024-06-29T15:57:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 -- 172.21.15.59:0/780104290 shutdown_connections 2024-06-29T15:57:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x5592397a18e0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 0x5592397a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 --2- 172.21.15.59:0/780104290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 -- 172.21.15.59:0/780104290 >> 172.21.15.59:0/780104290 conn(0x5592393c53c0 msgr2=0x559239774f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 -- 172.21.15.59:0/780104290 shutdown_connections 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.266+0000 7f28449c3740 1 -- 172.21.15.59:0/780104290 wait complete. 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.267+0000 7f28449c3740 1 -- start start 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.267+0000 7f28449c3740 1 Processor -- start 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28449c3740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397bce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28449c3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x5592397b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28449c3740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28449c3740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x559239675ae0 con 0x5592393a83b0 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28449c3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5592396dbc10 con 0x5592393b0590 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28449c3740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55923944b620 con 0x5592393d1270 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28449c3740 10 client.0 _ll_register_callbacks cb 0x559239682480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:57:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397bce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397bce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54620/0 (socket says 172.21.15.59:54620) 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 -- 172.21.15.59:0/3989049038 learned_addr learned my addr 172.21.15.59:0/3989049038 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f282effd640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x5592397b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f2837fff640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 -- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 msgr2=0x5592397cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 -- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 msgr2=0x5592397b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x5592397b68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 -- 172.21.15.59:0/3989049038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55923943c850 con 0x5592393a83b0 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f282effd640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x5592397b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f2837fff640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:45.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28377fe640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397bce50 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f282801dd50 tx=0x7f2828004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13299e84d62636c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28167fc640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.269+0000 7f28167fc640 1 -- 172.21.15.59:0/3989049038 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282800eac0 con 0x5592393a83b0 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28167fc640 1 -- 172.21.15.59:0/3989049038 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f2828009040 con 0x5592393a83b0 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 1 -- 172.21.15.59:0/3989049038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2830032be0 con 0x5592393a83b0 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28167fc640 1 -- 172.21.15.59:0/3989049038 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7f282802f050 con 0x5592393a83b0 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 10 client.11796 fetch_fsmap learned FSMap version 121 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 1 -- 172.21.15.59:0/3989049038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) -- 0x5592394b36b0 con 0x5592393a83b0 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28167fc640 1 -- 172.21.15.59:0/3989049038 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2830032be0 con 0x5592393a83b0 2024-06-29T15:57:45.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28167fc640 1 -- 172.21.15.59:0/3989049038 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 121) ==== 4149+0+0 (secure 0 0 0) 0x7f282800ec60 con 0x5592393a83b0 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 10 client.11796 fetch_fsmap finished waiting for FSMap version 121 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 10 client.11796 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 20 client.11796 populate_metadata read hostname 'smithi059' 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] conn(0x55923938fe40 0x5592397d97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 4 client.11796 mds_command: new command op to 10747 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f28449c3740 1 -- 172.21.15.59:0/3989049038 --> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592396ea580 con 0x55923938fe40 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.270+0000 7f282effd640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] conn(0x55923938fe40 0x5592397d97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.271+0000 7f282effd640 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] conn(0x55923938fe40 0x5592397d97e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b6250bbf9bc4a8e server_cookie=58c234b5091d3cf6 in_seq=0 out_seq=0 2024-06-29T15:57:45.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.271+0000 7f28167fc640 10 client.11796 ms_handle_connect on v2:172.21.15.153:6835/601671949 2024-06-29T15:57:45.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28167fc640 1 -- 172.21.15.59:0/3989049038 <== mds.0 v2:172.21.15.153:6835/601671949 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5592396ea580 con 0x55923938fe40 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28167fc640 10 client.11796 handle_command_reply: tid=0 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 client.11796 shutdown 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 -- 172.21.15.59:0/3989049038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 msgr2=0x5592397bce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397bce50 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f282801dd50 tx=0x7f2828004730 comp rx=0 tx=0).stop 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 -- 172.21.15.59:0/3989049038 shutdown_connections 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] conn(0x55923938fe40 0x5592397d97e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x5592397b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28167fc640 0 client.11796 ms_handle_reset on v2:172.21.15.153:6835/601671949 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 --2- 172.21.15.59:0/3989049038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397bce50 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 -- 172.21.15.59:0/3989049038 >> 172.21.15.59:0/3989049038 conn(0x5592393c53c0 msgr2=0x55923979dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 -- 172.21.15.59:0/3989049038 shutdown_connections 2024-06-29T15:57:45.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 1 -- 172.21.15.59:0/3989049038 wait complete. 2024-06-29T15:57:45.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.272+0000 7f28449c3740 20 client.11796 trim_cache size 0 max 16384 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.291+0000 7f28449c3740 1 Processor -- start 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.291+0000 7f28449c3740 1 -- start start 2024-06-29T15:57:45.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f28449c3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 0x559239773760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f28449c3740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x559239770dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f28449c3740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x559239771300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f28449c3740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2838204bb0 con 0x5592393b0590 2024-06-29T15:57:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f28449c3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2838202630 con 0x5592393a83b0 2024-06-29T15:57:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f28449c3740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2838202c80 con 0x5592393d1270 2024-06-29T15:57:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x559239770dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x559239770dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54634/0 (socket says 172.21.15.59:54634) 2024-06-29T15:57:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 -- 172.21.15.59:0/592055897 learned_addr learned my addr 172.21.15.59:0/592055897 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 -- 172.21.15.59:0/592055897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 msgr2=0x559239771300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 --2- 172.21.15.59:0/592055897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x559239771300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 -- 172.21.15.59:0/592055897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 msgr2=0x559239773760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 --2- 172.21.15.59:0/592055897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 0x559239773760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 -- 172.21.15.59:0/592055897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592397be580 con 0x5592393b0590 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.293+0000 7f282effd640 1 --2- 172.21.15.59:0/592055897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x559239770dc0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f282000c390 tx=0x7f2820000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc41144c387256f server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f282cff9640 1 -- 172.21.15.59:0/592055897 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282007e070 con 0x5592393b0590 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f282cff9640 1 -- 172.21.15.59:0/592055897 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28200641e0 con 0x5592393b0590 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f282cff9640 1 -- 172.21.15.59:0/592055897 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2820068c00 con 0x5592393b0590 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 -- 172.21.15.59:0/592055897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 msgr2=0x559239770dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 --2- 172.21.15.59:0/592055897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x559239770dc0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f282000c390 tx=0x7f2820000ed0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 -- 172.21.15.59:0/592055897 shutdown_connections 2024-06-29T15:57:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 --2- 172.21.15.59:0/592055897 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x559239771300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 --2- 172.21.15.59:0/592055897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393b0590 0x559239770dc0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 --2- 172.21.15.59:0/592055897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393a83b0 0x559239773760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 -- 172.21.15.59:0/592055897 >> 172.21.15.59:0/592055897 conn(0x5592393c53c0 msgr2=0x5592397d5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 -- 172.21.15.59:0/592055897 shutdown_connections 2024-06-29T15:57:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.294+0000 7f28449c3740 1 -- 172.21.15.59:0/592055897 wait complete. 2024-06-29T15:57:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.295+0000 7f28449c3740 1 -- start start 2024-06-29T15:57:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.295+0000 7f28449c3740 1 Processor -- start 2024-06-29T15:57:45.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28449c3740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397cc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28449c3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x559239833530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28449c3740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28449c3740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5592397a0be0 con 0x5592393a83b0 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28449c3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5592397b6e00 con 0x5592393b0590 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28449c3740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5592397b4ec0 con 0x5592393d1270 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28449c3740 10 client.0 _ll_register_callbacks cb 0x7f282801d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397cc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54640/0 (socket says 172.21.15.59:54640) 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 -- 172.21.15.59:0/641976853 learned_addr learned my addr 172.21.15.59:0/641976853 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f282effd640 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x559239833530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f2837fff640 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 -- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 msgr2=0x5592397bc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 -- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 msgr2=0x559239833530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x559239833530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 -- 172.21.15.59:0/641976853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559239778ef0 con 0x5592393a83b0 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f2837fff640 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28377fe640 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397cc090 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f2828011980 tx=0x7f28280119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed4a8a9fbc9897b server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28167fc640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28167fc640 1 -- 172.21.15.59:0/641976853 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828038060 con 0x5592393a83b0 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28167fc640 1 -- 172.21.15.59:0/641976853 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f282800ecd0 con 0x5592393a83b0 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.297+0000 7f28449c3740 1 -- 172.21.15.59:0/641976853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28200641e0 con 0x5592393a83b0 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28167fc640 1 -- 172.21.15.59:0/641976853 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) ==== 24+0+0 (secure 0 0 0) 0x7f2828021070 con 0x5592393a83b0 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28449c3740 10 client.11802 fetch_fsmap learned FSMap version 121 2024-06-29T15:57:45.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28449c3740 1 -- 172.21.15.59:0/641976853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) -- 0x559239833a70 con 0x5592393a83b0 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28167fc640 1 -- 172.21.15.59:0/641976853 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28200641e0 con 0x5592393a83b0 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28167fc640 1 -- 172.21.15.59:0/641976853 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 121) ==== 4149+0+0 (secure 0 0 0) 0x7f2828004660 con 0x5592393a83b0 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28449c3740 10 client.11802 fetch_fsmap finished waiting for FSMap version 121 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28449c3740 10 client.11802 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28449c3740 20 client.11802 populate_metadata read hostname 'smithi059' 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28449c3740 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] conn(0x559239771be0 0x55923979bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28449c3740 4 client.11802 mds_command: new command op to 10747 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f28449c3740 1 -- 172.21.15.59:0/641976853 --> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] -- command(tid 0: {"prefix": "status"}) -- 0x5592396e6020 con 0x559239771be0 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.298+0000 7f282effd640 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] conn(0x559239771be0 0x55923979bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28167fc640 10 client.11802 ms_handle_connect on v2:172.21.15.153:6835/601671949 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f282effd640 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] conn(0x559239771be0 0x55923979bfd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cda68950aa6b4634 server_cookie=89eba826f52ce94a in_seq=0 out_seq=0 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28167fc640 1 -- 172.21.15.59:0/641976853 <== mds.0 v2:172.21.15.153:6835/601671949 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5592396e6020 con 0x559239771be0 2024-06-29T15:57:45.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28167fc640 10 client.11802 handle_command_reply: tid=0 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 client.11802 shutdown 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 -- 172.21.15.59:0/641976853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 msgr2=0x5592397cc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397cc090 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f2828011980 tx=0x7f28280119b0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 -- 172.21.15.59:0/641976853 shutdown_connections 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:6835/601671949,v1:172.21.15.153:6837/601671949] conn(0x559239771be0 0x55923979bfd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5592393d1270 0x5592397bc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592393b0590 0x559239833530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28167fc640 0 client.11802 ms_handle_reset on v2:172.21.15.153:6835/601671949 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 --2- 172.21.15.59:0/641976853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5592393a83b0 0x5592397cc090 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 -- 172.21.15.59:0/641976853 >> 172.21.15.59:0/641976853 conn(0x5592393c53c0 msgr2=0x55923979ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 -- 172.21.15.59:0/641976853 shutdown_connections 2024-06-29T15:57:45.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 1 -- 172.21.15.59:0/641976853 wait complete. 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.299+0000 7f28449c3740 20 client.11802 trim_cache size 0 max 16384 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 10747, 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "backup_fs", 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 10.011497538, 2024-06-29T15:57:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 121, 2024-06-29T15:57:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 226, 2024-06-29T15:57:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 226, 2024-06-29T15:57:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 27.696540055 2024-06-29T15:57:45.308 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 -- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28180763f0 msgr2=0x7f2818078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28180763f0 0x7f2818078900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2828009530 tx=0x7f2828009260 comp rx=0 tx=0).stop 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 -- 172.21.15.59:0/1659902664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2838162770 msgr2=0x7f28381f5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2838162770 0x7f28381f5550 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f282000c3c0 tx=0x7f282000c880 comp rx=0 tx=0).stop 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 -- 172.21.15.59:0/1659902664 shutdown_connections 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f28180763f0 0x7f2818078900 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2838155890 0x7f28380cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2838162770 0x7f28381f5550 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 --2- 172.21.15.59:0/1659902664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f283815e0a0 0x7f28381f7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 -- 172.21.15.59:0/1659902664 >> 172.21.15.59:0/1659902664 conn(0x7f28380d45b0 msgr2=0x7f283806da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 -- 172.21.15.59:0/1659902664 shutdown_connections 2024-06-29T15:57:45.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.300+0000 7f283db03640 1 -- 172.21.15.59:0/1659902664 wait complete. 2024-06-29T15:57:45.312 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:57:45.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.431+0000 7fd90a466640 1 -- 172.21.15.59:0/735742633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd90415e290 msgr2=0x7fd9040ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.431+0000 7fd90a466640 1 --2- 172.21.15.59:0/735742633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd90415e290 0x7fd9040ce110 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009860 tx=0x7fd8f402a530 comp rx=0 tx=0).stop 2024-06-29T15:57:45.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.431+0000 7fd90a466640 1 -- 172.21.15.59:0/735742633 shutdown_connections 2024-06-29T15:57:45.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.431+0000 7fd90a466640 1 --2- 172.21.15.59:0/735742633 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd90415e290 0x7fd9040ce110 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.431+0000 7fd90a466640 1 -- 172.21.15.59:0/735742633 >> 172.21.15.59:0/735742633 conn(0x7fd9040d3ee0 msgr2=0x7fd9040d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.431+0000 7fd90a466640 1 -- 172.21.15.59:0/735742633 shutdown_connections 2024-06-29T15:57:45.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.431+0000 7fd90a466640 1 -- 172.21.15.59:0/735742633 wait complete. 2024-06-29T15:57:45.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.433+0000 7fd90a466640 1 Processor -- start 2024-06-29T15:57:45.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.433+0000 7fd90a466640 1 -- start start 2024-06-29T15:57:45.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.434+0000 7fd90a466640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9040ce650 0x7fd9040cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.434+0000 7fd90a466640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd90415e290 0x7fd904153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.434+0000 7fd90a466640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd90415f630 0x7fd9041539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.434+0000 7fd90a466640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd9040b7f10 con 0x7fd9040ce650 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.434+0000 7fd90a466640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd9040b7d90 con 0x7fd90415f630 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.434+0000 7fd90a466640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd9040b8090 con 0x7fd90415e290 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9040ce650 0x7fd9040cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9040ce650 0x7fd9040cd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54654/0 (socket says 172.21.15.59:54654) 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 -- 172.21.15.59:0/1609273664 learned_addr learned my addr 172.21.15.59:0/1609273664 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd9037fe640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd90415e290 0x7fd904153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd908a03640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd90415f630 0x7fd9041539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 -- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd90415e290 msgr2=0x7fd904153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd90415e290 0x7fd904153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 -- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd90415f630 msgr2=0x7fd9041539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd90415f630 0x7fd9041539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 -- 172.21.15.59:0/1609273664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9041553d0 con 0x7fd9040ce650 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd908a03640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd90415f630 0x7fd9041539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd903fff640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9040ce650 0x7fd9040cd900 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fd8f402ab20 tx=0x7fd8f4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd900ff9640 1 -- 172.21.15.59:0/1609273664 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4005c90 con 0x7fd9040ce650 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd900ff9640 1 -- 172.21.15.59:0/1609273664 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8f4005020 con 0x7fd9040ce650 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9041558d0 con 0x7fd9040ce650 2024-06-29T15:57:45.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd900ff9640 1 -- 172.21.15.59:0/1609273664 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4005410 con 0x7fd9040ce650 2024-06-29T15:57:45.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.435+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90415eb50 con 0x7fd9040ce650 2024-06-29T15:57:45.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.436+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9040ce110 con 0x7fd9040ce650 2024-06-29T15:57:45.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.437+0000 7fd900ff9640 1 -- 172.21.15.59:0/1609273664 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fd8f40317c0 con 0x7fd9040ce650 2024-06-29T15:57:45.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.438+0000 7fd900ff9640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd8f00763f0 0x7fd8f0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.438+0000 7fd900ff9640 1 -- 172.21.15.59:0/1609273664 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7fd8f40b8e90 con 0x7fd9040ce650 2024-06-29T15:57:45.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.438+0000 7fd9037fe640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd8f00763f0 0x7fd8f0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.439+0000 7fd9037fe640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd8f00763f0 0x7fd8f0078900 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8001220 tx=0x7fd8f8001770 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.442+0000 7fd900ff9640 1 -- 172.21.15.59:0/1609273664 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd8f403f030 con 0x7fd9040ce650 2024-06-29T15:57:45.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.659+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fd9041e6ca0 con 0x7fd8f00763f0 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.662+0000 7fd900ff9640 1 -- 172.21.15.59:0/1609273664 <== mgr.9206 v2:172.21.15.153:6832/2780049427 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+82 (secure 0 0 0) 0x7fd9041e6ca0 con 0x7fd8f00763f0 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "backup_fs" 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:57:45.662 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:57:45.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd8f00763f0 msgr2=0x7fd8f0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd8f00763f0 0x7fd8f0078900 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8001220 tx=0x7fd8f8001770 comp rx=0 tx=0).stop 2024-06-29T15:57:45.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9040ce650 msgr2=0x7fd9040cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9040ce650 0x7fd9040cd900 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fd8f402ab20 tx=0x7fd8f4003c10 comp rx=0 tx=0).stop 2024-06-29T15:57:45.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 shutdown_connections 2024-06-29T15:57:45.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7fd8f00763f0 0x7fd8f0078900 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd90415f630 0x7fd9041539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd90415e290 0x7fd904153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 --2- 172.21.15.59:0/1609273664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9040ce650 0x7fd9040cd900 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 >> 172.21.15.59:0/1609273664 conn(0x7fd9040d3ee0 msgr2=0x7fd904151f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 shutdown_connections 2024-06-29T15:57:45.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.663+0000 7fd90a466640 1 -- 172.21.15.59:0/1609273664 wait complete. 2024-06-29T15:57:45.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:57:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.795+0000 7f1d6ce72640 1 -- 172.21.15.59:0/3552575391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d54004d90 msgr2=0x7f1d54005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.795+0000 7f1d6ce72640 1 --2- 172.21.15.59:0/3552575391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d54004d90 0x7f1d54005260 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f1d680b5090 tx=0x7f1d54005830 comp rx=0 tx=0).stop 2024-06-29T15:57:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.795+0000 7f1d66d9d640 1 -- 172.21.15.59:0/3552575391 reap_dead start 2024-06-29T15:57:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.795+0000 7f1d6ce72640 1 -- 172.21.15.59:0/3552575391 shutdown_connections 2024-06-29T15:57:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.795+0000 7f1d6ce72640 1 -- 172.21.15.59:0/3552575391 >> 172.21.15.59:0/3552575391 conn(0x7f1d680b24c0 msgr2=0x7f1d680cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.795+0000 7f1d6ce72640 1 -- 172.21.15.59:0/3552575391 shutdown_connections 2024-06-29T15:57:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.795+0000 7f1d6ce72640 1 -- 172.21.15.59:0/3552575391 wait complete. 2024-06-29T15:57:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.797+0000 7f1d6ce72640 1 Processor -- start 2024-06-29T15:57:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.797+0000 7f1d6ce72640 1 -- start start 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d6ce72640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d54004d90 0x7f1d6813f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d6ce72640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d68137ee0 0x7f1d681f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d6ce72640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d68140960 0x7f1d68140d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d6ce72640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1d680ce6e0 con 0x7f1d68137ee0 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d6ce72640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d68142f30 con 0x7f1d54004d90 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d6ce72640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1d680ce860 con 0x7f1d68140960 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d65d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d68137ee0 0x7f1d681f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d65d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d68137ee0 0x7f1d681f5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54676/0 (socket says 172.21.15.59:54676) 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d65d9b640 1 -- 172.21.15.59:0/1346563735 learned_addr learned my addr 172.21.15.59:0/1346563735 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d66d9d640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d68140960 0x7f1d68140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.798+0000 7f1d6659c640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d54004d90 0x7f1d6813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d65d9b640 1 -- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d68140960 msgr2=0x7f1d68140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d65d9b640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d68140960 0x7f1d68140d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d65d9b640 1 -- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d54004d90 msgr2=0x7f1d6813f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d65d9b640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d54004d90 0x7f1d6813f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d65d9b640 1 -- 172.21.15.59:0/1346563735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d681412b0 con 0x7f1d68137ee0 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d65d9b640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d68137ee0 0x7f1d681f5da0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f1d50006680 tx=0x7f1d500066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d5effd640 1 -- 172.21.15.59:0/1346563735 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d50015070 con 0x7f1d68137ee0 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d5effd640 1 -- 172.21.15.59:0/1346563735 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d50011900 con 0x7f1d68137ee0 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d6813fe00 con 0x7f1d68137ee0 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d5effd640 1 -- 172.21.15.59:0/1346563735 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d50011d50 con 0x7f1d68137ee0 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.799+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d68135190 con 0x7f1d68137ee0 2024-06-29T15:57:45.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.800+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d6813fb70 con 0x7f1d68137ee0 2024-06-29T15:57:45.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.801+0000 7f1d5effd640 1 -- 172.21.15.59:0/1346563735 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f1d50018070 con 0x7f1d68137ee0 2024-06-29T15:57:45.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.802+0000 7f1d5effd640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1d480763f0 0x7f1d48078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.802+0000 7f1d5effd640 1 -- 172.21.15.59:0/1346563735 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f1d50098ae0 con 0x7f1d68137ee0 2024-06-29T15:57:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.802+0000 7f1d6659c640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1d480763f0 0x7f1d48078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.802+0000 7f1d6659c640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1d480763f0 0x7f1d48078900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005fd0 tx=0x7f1d58009500 comp rx=0 tx=0).ready entity=mgr.9206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:45.806+0000 7f1d5effd640 1 -- 172.21.15.59:0/1346563735 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d681412b0 con 0x7f1d68137ee0 2024-06-29T15:57:46.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.049+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f1d6813de80 con 0x7f1d68137ee0 2024-06-29T15:57:46.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.842+0000 7f1d5effd640 1 -- 172.21.15.59:0/1346563735 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v129) ==== 92+0+0 (secure 0 0 0) 0x7f1d6813de80 con 0x7f1d68137ee0 2024-06-29T15:57:46.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1d480763f0 msgr2=0x7f1d48078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1d480763f0 0x7f1d48078900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005fd0 tx=0x7f1d58009500 comp rx=0 tx=0).stop 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d68137ee0 msgr2=0x7f1d681f5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d68137ee0 0x7f1d681f5da0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f1d50006680 tx=0x7f1d500066b0 comp rx=0 tx=0).stop 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 shutdown_connections 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f1d480763f0 0x7f1d48078900 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005fd0 tx=0x7f1d58009500 comp rx=0 tx=0).stop 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d68140960 0x7f1d68140d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d68137ee0 0x7f1d681f5da0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 --2- 172.21.15.59:0/1346563735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d54004d90 0x7f1d6813f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.846+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 >> 172.21.15.59:0/1346563735 conn(0x7f1d680b24c0 msgr2=0x7f1d681441c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.847+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 shutdown_connections 2024-06-29T15:57:46.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.847+0000 7f1d6ce72640 1 -- 172.21.15.59:0/1346563735 wait complete. 2024-06-29T15:57:46.863 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:57:46.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.993+0000 7f31c492b640 1 -- 172.21.15.59:0/2880529972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 msgr2=0x7f31c00b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:46.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.993+0000 7f31c492b640 1 --2- 172.21.15.59:0/2880529972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 0x7f31c00b5b30 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005fd0 tx=0x7f31b802aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:46.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.994+0000 7f31c492b640 1 -- 172.21.15.59:0/2880529972 shutdown_connections 2024-06-29T15:57:46.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.994+0000 7f31c492b640 1 --2- 172.21.15.59:0/2880529972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 0x7f31c00b5b30 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:46.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.994+0000 7f31c492b640 1 --2- 172.21.15.59:0/2880529972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31c00b2fa0 0x7f31c00d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:46.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.994+0000 7f31c492b640 1 -- 172.21.15.59:0/2880529972 >> 172.21.15.59:0/2880529972 conn(0x7f31c0073e10 msgr2=0x7f31c0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:46.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.994+0000 7f31c492b640 1 -- 172.21.15.59:0/2880529972 shutdown_connections 2024-06-29T15:57:46.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.994+0000 7f31c492b640 1 -- 172.21.15.59:0/2880529972 wait complete. 2024-06-29T15:57:46.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.996+0000 7f31c492b640 1 Processor -- start 2024-06-29T15:57:46.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.996+0000 7f31c492b640 1 -- start start 2024-06-29T15:57:46.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31c492b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00b2270 0x7f31c01f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:46.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31c492b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31c00b2fa0 0x7f31c00aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:46.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31c492b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 0x7f31c0154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:46.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31c492b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f31c00b6400 con 0x7f31c00d1a60 2024-06-29T15:57:46.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31c492b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31c00b6280 con 0x7f31c00b2270 2024-06-29T15:57:46.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31c492b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f31c00b6580 con 0x7f31c00b2fa0 2024-06-29T15:57:46.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31bed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 0x7f31c0154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31bed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 0x7f31c0154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54710/0 (socket says 172.21.15.59:54710) 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31bed9d640 1 -- 172.21.15.59:0/63258046 learned_addr learned my addr 172.21.15.59:0/63258046 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31bdd9b640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31c00b2fa0 0x7f31c00aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.997+0000 7f31be59c640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00b2270 0x7f31c01f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31bed9d640 1 -- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31c00b2fa0 msgr2=0x7f31c00aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31bed9d640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31c00b2fa0 0x7f31c00aaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31bed9d640 1 -- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00b2270 msgr2=0x7f31c01f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31bed9d640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00b2270 0x7f31c01f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31bed9d640 1 -- 172.21.15.59:0/63258046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c015d5d0 con 0x7f31c00d1a60 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31be59c640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00b2270 0x7f31c01f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31bdd9b640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31c00b2fa0 0x7f31c00aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31bed9d640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 0x7f31c0154440 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f31b8006010 tx=0x7f31b8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31b6ffd640 1 -- 172.21.15.59:0/63258046 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8044070 con 0x7f31c00d1a60 2024-06-29T15:57:47.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31b6ffd640 1 -- 172.21.15.59:0/63258046 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31b8004040 con 0x7f31c00d1a60 2024-06-29T15:57:47.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31b6ffd640 1 -- 172.21.15.59:0/63258046 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8004430 con 0x7f31c00d1a60 2024-06-29T15:57:47.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c015d890 con 0x7f31c00d1a60 2024-06-29T15:57:47.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.998+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c015de50 con 0x7f31c00d1a60 2024-06-29T15:57:47.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:46.999+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c00ab0a0 con 0x7f31c00d1a60 2024-06-29T15:57:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.000+0000 7f31b6ffd640 1 -- 172.21.15.59:0/63258046 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 129) ==== 98320+0+0 (secure 0 0 0) 0x7f31b8047070 con 0x7f31c00d1a60 2024-06-29T15:57:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.000+0000 7f31b6ffd640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f31a0076440 0x7f31a0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.001+0000 7f31b6ffd640 1 -- 172.21.15.59:0/63258046 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f31b80b8c90 con 0x7f31c00d1a60 2024-06-29T15:57:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.001+0000 7f31be59c640 1 -- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f31a0076440 msgr2=0x7f31a0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2780049427 2024-06-29T15:57:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.001+0000 7f31be59c640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f31a0076440 0x7f31a0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:47.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.005+0000 7f31b6ffd640 1 -- 172.21.15.59:0/63258046 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f31b8032030 con 0x7f31c00d1a60 2024-06-29T15:57:47.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.200+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f31c0154980 con 0x7f31c00d1a60 2024-06-29T15:57:47.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.202+0000 7f31be59c640 1 -- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f31a0076440 msgr2=0x7f31a0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2780049427 2024-06-29T15:57:47.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.202+0000 7f31be59c640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f31a0076440 0x7f31a0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:47.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.203+0000 7f31b6ffd640 1 -- 172.21.15.59:0/63258046 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v73)=0 v73) ==== 138+0+0 (secure 0 0 0) 0x7f31c0154980 con 0x7f31c00d1a60 2024-06-29T15:57:47.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f31a0076440 msgr2=0x7f31a0078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:47.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f31a0076440 0x7f31a0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 msgr2=0x7f31c0154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 0x7f31c0154440 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f31b8006010 tx=0x7f31b8003ce0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 shutdown_connections 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f31a0076440 0x7f31a0078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31c00d1a60 0x7f31c0154440 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31c00b2fa0 0x7f31c00aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 --2- 172.21.15.59:0/63258046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00b2270 0x7f31c01f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.205+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 >> 172.21.15.59:0/63258046 conn(0x7f31c0073e10 msgr2=0x7f31c0070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.206+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 shutdown_connections 2024-06-29T15:57:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.206+0000 7f31c492b640 1 -- 172.21.15.59:0/63258046 wait complete. 2024-06-29T15:57:47.218 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:57:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.339+0000 7f2565711640 1 -- 172.21.15.59:0/4102979097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f256006e3a0 msgr2=0x7f256015c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.339+0000 7f2565711640 1 --2- 172.21.15.59:0/4102979097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f256006e3a0 0x7f256015c730 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f2558005fd0 tx=0x7f255802a840 comp rx=0 tx=0).stop 2024-06-29T15:57:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.339+0000 7f2565711640 1 -- 172.21.15.59:0/4102979097 shutdown_connections 2024-06-29T15:57:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.339+0000 7f2565711640 1 --2- 172.21.15.59:0/4102979097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f256006e3a0 0x7f256015c730 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.339+0000 7f2565711640 1 --2- 172.21.15.59:0/4102979097 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2560144970 0x7f256015f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.340+0000 7f2565711640 1 -- 172.21.15.59:0/4102979097 >> 172.21.15.59:0/4102979097 conn(0x7f25600b2ab0 msgr2=0x7f25600b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:47.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.340+0000 7f2565711640 1 -- 172.21.15.59:0/4102979097 shutdown_connections 2024-06-29T15:57:47.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.340+0000 7f2565711640 1 -- 172.21.15.59:0/4102979097 wait complete. 2024-06-29T15:57:47.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.342+0000 7f2565711640 1 Processor -- start 2024-06-29T15:57:47.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.342+0000 7f2565711640 1 -- start start 2024-06-29T15:57:47.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f2565711640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25600cfa00 0x7f256013d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f2565711640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2560144970 0x7f25601f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f2565711640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25601f66d0 0x7f256013bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f2565711640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f256015d250 con 0x7f2560144970 2024-06-29T15:57:47.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f2565711640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f256015d0d0 con 0x7f25601f66d0 2024-06-29T15:57:47.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f2565711640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f256015d3d0 con 0x7f25600cfa00 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2560144970 0x7f25601f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2560144970 0x7f25601f6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54720/0 (socket says 172.21.15.59:54720) 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255e7fc640 1 -- 172.21.15.59:0/2703540693 learned_addr learned my addr 172.21.15.59:0/2703540693 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25600cfa00 0x7f256013d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255f7fe640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25601f66d0 0x7f256013bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255effd640 1 -- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25601f66d0 msgr2=0x7f256013bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255effd640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25601f66d0 0x7f256013bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255effd640 1 -- 172.21.15.59:0/2703540693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2560144970 msgr2=0x7f25601f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255effd640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2560144970 0x7f25601f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255effd640 1 -- 172.21.15.59:0/2703540693 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2560138a20 con 0x7f25600cfa00 2024-06-29T15:57:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.343+0000 7f255e7fc640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2560144970 0x7f25601f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:47.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.344+0000 7f255effd640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25600cfa00 0x7f256013d070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f255000ea20 tx=0x7f255000eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:47.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.344+0000 7f25477fe640 1 -- 172.21.15.59:0/2703540693 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255000ccc0 con 0x7f25600cfa00 2024-06-29T15:57:47.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.344+0000 7f25477fe640 1 -- 172.21.15.59:0/2703540693 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f255000ce60 con 0x7f25600cfa00 2024-06-29T15:57:47.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.344+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2560138d00 con 0x7f25600cfa00 2024-06-29T15:57:47.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.344+0000 7f25477fe640 1 -- 172.21.15.59:0/2703540693 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2550004280 con 0x7f25600cfa00 2024-06-29T15:57:47.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.344+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25601392c0 con 0x7f25600cfa00 2024-06-29T15:57:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.345+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f256015d8f0 con 0x7f25600cfa00 2024-06-29T15:57:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.346+0000 7f25477fe640 1 -- 172.21.15.59:0/2703540693 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 129) ==== 98320+0+0 (secure 0 0 0) 0x7f25500035f0 con 0x7f25600cfa00 2024-06-29T15:57:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.346+0000 7f25477fe640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f254c076440 0x7f254c078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.346+0000 7f255e7fc640 1 -- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f254c076440 msgr2=0x7f254c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2780049427 2024-06-29T15:57:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.346+0000 7f25477fe640 1 -- 172.21.15.59:0/2703540693 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f25500988f0 con 0x7f25600cfa00 2024-06-29T15:57:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.346+0000 7f255e7fc640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f254c076440 0x7f254c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:47.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.351+0000 7f25477fe640 1 -- 172.21.15.59:0/2703540693 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2550061310 con 0x7f25600cfa00 2024-06-29T15:57:47.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.543+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f25600cf7e0 con 0x7f25600cfa00 2024-06-29T15:57:47.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.547+0000 7f255e7fc640 1 -- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f254c076440 msgr2=0x7f254c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2780049427 2024-06-29T15:57:47.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.547+0000 7f255e7fc640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f254c076440 0x7f254c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.549+0000 7f25477fe640 1 -- 172.21.15.59:0/2703540693 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v74)=0 v74) ==== 133+0+0 (secure 0 0 0) 0x7f2550065610 con 0x7f25600cfa00 2024-06-29T15:57:47.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f254c076440 msgr2=0x7f254c078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:47.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f254c076440 0x7f254c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25600cfa00 msgr2=0x7f256013d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25600cfa00 0x7f256013d070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f255000ea20 tx=0x7f255000eee0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 shutdown_connections 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f254c076440 0x7f254c078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25601f66d0 0x7f256013bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2560144970 0x7f25601f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 --2- 172.21.15.59:0/2703540693 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25600cfa00 0x7f256013d070 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 >> 172.21.15.59:0/2703540693 conn(0x7f25600b2ab0 msgr2=0x7f256015b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 shutdown_connections 2024-06-29T15:57:47.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.551+0000 7f2565711640 1 -- 172.21.15.59:0/2703540693 wait complete. 2024-06-29T15:57:47.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:47.564 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-06-29T15:57:47.584 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:47.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.748+0000 7f7655b7a640 1 -- 172.21.15.59:0/2295898030 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76501423c0 msgr2=0x7f76500b56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.748+0000 7f7655b7a640 1 --2- 172.21.15.59:0/2295898030 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76501423c0 0x7f76500b56e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7640009890 tx=0x7f764002a560 comp rx=0 tx=0).stop 2024-06-29T15:57:47.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.748+0000 7f7655b7a640 1 -- 172.21.15.59:0/2295898030 shutdown_connections 2024-06-29T15:57:47.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.748+0000 7f7655b7a640 1 --2- 172.21.15.59:0/2295898030 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76501423c0 0x7f76500b56e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.748+0000 7f7655b7a640 1 -- 172.21.15.59:0/2295898030 >> 172.21.15.59:0/2295898030 conn(0x7f7650073e10 msgr2=0x7f7650074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:47.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.748+0000 7f7655b7a640 1 -- 172.21.15.59:0/2295898030 shutdown_connections 2024-06-29T15:57:47.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.748+0000 7f7655b7a640 1 -- 172.21.15.59:0/2295898030 wait complete. 2024-06-29T15:57:47.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.749+0000 7f7655b7a640 1 Processor -- start 2024-06-29T15:57:47.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.750+0000 7f7655b7a640 1 -- start start 2024-06-29T15:57:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f7655b7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76500cec20 0x7f76500a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f7655b7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76501423c0 0x7f765012f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f7655b7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765012fd20 0x7f7650130130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f7655b7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f76500d01a0 con 0x7f765012fd20 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f7655b7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76500d0020 con 0x7f76501423c0 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f7655b7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f76500d0320 con 0x7f76500cec20 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f764ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765012fd20 0x7f7650130130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f764ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765012fd20 0x7f7650130130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54736/0 (socket says 172.21.15.59:54736) 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f764ffff640 1 -- 172.21.15.59:0/325258156 learned_addr learned my addr 172.21.15.59:0/325258156 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f764f7fe640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76500cec20 0x7f76500a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.751+0000 7f7646ffd640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76501423c0 0x7f765012f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764ffff640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76500cec20 msgr2=0x7f76500a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764ffff640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76500cec20 0x7f76500a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764ffff640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76501423c0 msgr2=0x7f765012f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764ffff640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76501423c0 0x7f765012f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764ffff640 1 -- 172.21.15.59:0/325258156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7650130670 con 0x7f765012fd20 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764ffff640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765012fd20 0x7f7650130130 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f764800c730 tx=0x7f764800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648003b70 con 0x7f765012fd20 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7648003d10 con 0x7f765012fd20 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76501e6bf0 con 0x7f765012fd20 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764800aab0 con 0x7f765012fd20 2024-06-29T15:57:47.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.752+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765012ed00 con 0x7f765012fd20 2024-06-29T15:57:47.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.753+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76500b56e0 con 0x7f765012fd20 2024-06-29T15:57:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.754+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 129) ==== 98320+0+0 (secure 0 0 0) 0x7f764800ac50 con 0x7f765012fd20 2024-06-29T15:57:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.755+0000 7f764cff9640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 0x7f7630078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.755+0000 7f764f7fe640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 msgr2=0x7f7630078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2780049427 2024-06-29T15:57:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.755+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 7021+0+0 (secure 0 0 0) 0x7f7648097a80 con 0x7f765012fd20 2024-06-29T15:57:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.755+0000 7f764f7fe640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 0x7f7630078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:47.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.760+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76480607f0 con 0x7f765012fd20 2024-06-29T15:57:47.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.956+0000 7f764f7fe640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 msgr2=0x7f7630078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2780049427 2024-06-29T15:57:47.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.956+0000 7f764f7fe640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 0x7f7630078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:47.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:47.983+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 --> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f765015e7d0 con 0x7f7630076470 2024-06-29T15:57:48.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:48.357+0000 7f764f7fe640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 msgr2=0x7f7630078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2780049427 2024-06-29T15:57:48.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:48.357+0000 7f764f7fe640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 0x7f7630078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:57:49.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:49.158+0000 7f764f7fe640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 msgr2=0x7f7630078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2780049427 2024-06-29T15:57:49.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:49.158+0000 7f764f7fe640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 0x7f7630078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:57:50.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:50.467+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 130) ==== 97929+0+0 (secure 0 0 0) 0x7f764805ca80 con 0x7f765012fd20 2024-06-29T15:57:50.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:50.468+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 msgr2=0x7f7630078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:50.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:50.468+0000 7f764cff9640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2780049427,v1:172.21.15.153:6833/2780049427] conn(0x7f7630076470 0x7f7630078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.709+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.709+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.709+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.709+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.709+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.710+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.710+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.710+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.710+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.711 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.710+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a7365640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.722 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:50.723 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:50.721+0000 7f83a435f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.471+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 131) ==== 98056+0+0 (secure 0 0 0) 0x7f7648095a50 con 0x7f765012fd20 2024-06-29T15:57:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.472+0000 7f764cff9640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f763003ad60 0x7f7630083160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.472+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 --> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f765015e7d0 con 0x7f763003ad60 2024-06-29T15:57:51.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.474+0000 7f764f7fe640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f763003ad60 0x7f7630083160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.475+0000 7f764f7fe640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f763003ad60 0x7f7630083160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7640034000 tx=0x7f7640004730 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:51.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.543+0000 7f764cff9640 1 -- 172.21.15.59:0/325258156 <== mgr.11005 v2:172.21.15.153:6832/2014343395 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+53 (secure 0 0 0) 0x7f765015e7d0 con 0x7f763003ad60 2024-06-29T15:57:51.544 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f763003ad60 msgr2=0x7f7630083160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f7655b7a640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f763003ad60 0x7f7630083160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7640034000 tx=0x7f7640004730 comp rx=0 tx=0).stop 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765012fd20 msgr2=0x7f7650130130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f7655b7a640 1 --2- 172.21.15.59:0/325258156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f765012fd20 0x7f7650130130 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f764800c730 tx=0x7f764800cbf0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f764ffff640 1 -- 172.21.15.59:0/325258156 reap_dead start 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 shutdown_connections 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 >> 172.21.15.59:0/325258156 conn(0x7f7650073e10 msgr2=0x7f765015ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 shutdown_connections 2024-06-29T15:57:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.546+0000 7f7655b7a640 1 -- 172.21.15.59:0/325258156 wait complete. 2024-06-29T15:57:51.559 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-06-29T15:57:51.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:51.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.684+0000 7f5cc13bd640 1 -- 172.21.15.59:0/1891616903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ca40052a0 msgr2=0x7f5ca400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.684+0000 7f5cc13bd640 1 --2- 172.21.15.59:0/1891616903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ca40052a0 0x7f5ca400b9c0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400c610 tx=0x7f5ca4005830 comp rx=0 tx=0).stop 2024-06-29T15:57:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.684+0000 7f5cc13bd640 1 -- 172.21.15.59:0/1891616903 shutdown_connections 2024-06-29T15:57:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.684+0000 7f5cc13bd640 1 --2- 172.21.15.59:0/1891616903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ca40052a0 0x7f5ca400b9c0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.684+0000 7f5cc13bd640 1 -- 172.21.15.59:0/1891616903 >> 172.21.15.59:0/1891616903 conn(0x7f5cbc0b5590 msgr2=0x7f5cbc0b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.684+0000 7f5cc13bd640 1 -- 172.21.15.59:0/1891616903 shutdown_connections 2024-06-29T15:57:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.685+0000 7f5cc13bd640 1 -- 172.21.15.59:0/1891616903 wait complete. 2024-06-29T15:57:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.686+0000 7f5cc13bd640 1 Processor -- start 2024-06-29T15:57:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.686+0000 7f5cc13bd640 1 -- start start 2024-06-29T15:57:51.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cc13bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ca40052a0 0x7f5cbc14d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cc13bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cbc15b680 0x7f5cbc15ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cc13bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cbc0cc840 0x7f5cbc0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cc13bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc0b78f0 con 0x7f5cbc15b680 2024-06-29T15:57:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cc13bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc0b7770 con 0x7f5ca40052a0 2024-06-29T15:57:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cc13bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5cbc0d3380 con 0x7f5cbc0cc840 2024-06-29T15:57:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cbc15b680 0x7f5cbc15ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cbc15b680 0x7f5cbc15ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54750/0 (socket says 172.21.15.59:54750) 2024-06-29T15:57:51.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 -- 172.21.15.59:0/2467946988 learned_addr learned my addr 172.21.15.59:0/2467946988 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:51.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cbb7fe640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cbc0cc840 0x7f5cbc0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cbaffd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ca40052a0 0x7f5cbc14d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 -- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cbc0cc840 msgr2=0x7f5cbc0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cbc0cc840 0x7f5cbc0ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 -- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ca40052a0 msgr2=0x7f5cbc14d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ca40052a0 0x7f5cbc14d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 -- 172.21.15.59:0/2467946988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cbc14d630 con 0x7f5cbc15b680 2024-06-29T15:57:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.688+0000 7f5cba7fc640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cbc15b680 0x7f5cbc15ba90 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0011ab0 tx=0x7f5cb0011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.689+0000 7f5ca37fe640 1 -- 172.21.15.59:0/2467946988 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb001e070 con 0x7f5cbc15b680 2024-06-29T15:57:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.689+0000 7f5ca37fe640 1 -- 172.21.15.59:0/2467946988 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cb000fc70 con 0x7f5cbc15b680 2024-06-29T15:57:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.689+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cbc15bfd0 con 0x7f5cbc15b680 2024-06-29T15:57:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.689+0000 7f5ca37fe640 1 -- 172.21.15.59:0/2467946988 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb0019410 con 0x7f5cbc15b680 2024-06-29T15:57:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.689+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cbc0cba80 con 0x7f5cbc15b680 2024-06-29T15:57:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.690+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cbc161380 con 0x7f5cbc15b680 2024-06-29T15:57:51.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.691+0000 7f5ca37fe640 1 -- 172.21.15.59:0/2467946988 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 131) ==== 98056+0+0 (secure 0 0 0) 0x7f5cb0021070 con 0x7f5cbc15b680 2024-06-29T15:57:51.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.691+0000 7f5ca37fe640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f5ca8076170 0x7f5ca8078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.691+0000 7f5ca37fe640 1 -- 172.21.15.59:0/2467946988 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f5cb0098340 con 0x7f5cbc15b680 2024-06-29T15:57:51.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.691+0000 7f5cbaffd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f5ca8076170 0x7f5ca8078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.692+0000 7f5cbaffd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f5ca8076170 0x7f5ca8078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5cac000c00 tx=0x7f5cac006b10 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:51.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.696+0000 7f5ca37fe640 1 -- 172.21.15.59:0/2467946988 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5cbc15bfd0 con 0x7f5cbc15b680 2024-06-29T15:57:51.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.916+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 --> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7f5cbc0ca2e0 con 0x7f5ca8076170 2024-06-29T15:57:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.965+0000 7f5ca37fe640 1 -- 172.21.15.59:0/2467946988 <== mgr.11005 v2:172.21.15.153:6832/2014343395 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f5cbc0ca2e0 con 0x7f5ca8076170 2024-06-29T15:57:51.966 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-06-29T15:57:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f5ca8076170 msgr2=0x7f5ca8078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f5ca8076170 0x7f5ca8078680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5cac000c00 tx=0x7f5cac006b10 comp rx=0 tx=0).stop 2024-06-29T15:57:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cbc15b680 msgr2=0x7f5cbc15ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cbc15b680 0x7f5cbc15ba90 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0011ab0 tx=0x7f5cb0011eb0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 shutdown_connections 2024-06-29T15:57:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f5ca8076170 0x7f5ca8078680 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5cbc0cc840 0x7f5cbc0ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5cbc15b680 0x7f5cbc15ba90 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 --2- 172.21.15.59:0/2467946988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ca40052a0 0x7f5cbc14d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 >> 172.21.15.59:0/2467946988 conn(0x7f5cbc0b5590 msgr2=0x7f5cbc06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 shutdown_connections 2024-06-29T15:57:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:51.967+0000 7f5cc13bd640 1 -- 172.21.15.59:0/2467946988 wait complete. 2024-06-29T15:57:51.979 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-06-29T15:57:51.980 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.103+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3450759695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd806ea10 msgr2=0x7f8fd80cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.103+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3450759695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd806ea10 0x7f8fd80cef90 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8074350 tx=0x7f8fd002aa30 comp rx=0 tx=0).stop 2024-06-29T15:57:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.103+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3450759695 shutdown_connections 2024-06-29T15:57:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.103+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3450759695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd806ea10 0x7f8fd80cef90 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.103+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3450759695 >> v2:172.21.15.153:3300/0 conn(0x7f8fd8143840 0x7f8fd806e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.103+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3450759695 >> 172.21.15.59:0/3450759695 conn(0x7f8fd80b2ab0 msgr2=0x7f8fd80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.104+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3450759695 shutdown_connections 2024-06-29T15:57:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.104+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3450759695 wait complete. 2024-06-29T15:57:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.105+0000 7f8fdf5b2640 1 Processor -- start 2024-06-29T15:57:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.105+0000 7f8fdf5b2640 1 -- start start 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdf5b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd806ea10 0x7f8fd80ad250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdf5b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd80d0130 0x7f8fd81596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdf5b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd8143840 0x7f8fd815d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdf5b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd80cf650 con 0x7f8fd80d0130 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdf5b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd80cf4d0 con 0x7f8fd806ea10 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdf5b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd80cf7d0 con 0x7f8fd8143840 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd80d0130 0x7f8fd81596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd80d0130 0x7f8fd81596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54764/0 (socket says 172.21.15.59:54764) 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdd34e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd806ea10 0x7f8fd80ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 -- 172.21.15.59:0/3488539214 learned_addr learned my addr 172.21.15.59:0/3488539214 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fddb4f640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd8143840 0x7f8fd815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 -- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd8143840 msgr2=0x7f8fd815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd8143840 0x7f8fd815d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 -- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd806ea10 msgr2=0x7f8fd80ad250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd806ea10 0x7f8fd80ad250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 -- 172.21.15.59:0/3488539214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd81f5c20 con 0x7f8fd80d0130 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdd34e640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd806ea10 0x7f8fd80ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.107+0000 7f8fdcb4d640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd80d0130 0x7f8fd81596e0 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800e7c0 tx=0x7f8fc800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.108+0000 7f8fcdffb640 1 -- 172.21.15.59:0/3488539214 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc800ccc0 con 0x7f8fd80d0130 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.108+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd81f6120 con 0x7f8fd80d0130 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.108+0000 7f8fcdffb640 1 -- 172.21.15.59:0/3488539214 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fc800ce80 con 0x7f8fd80d0130 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.108+0000 7f8fcdffb640 1 -- 172.21.15.59:0/3488539214 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8005280 con 0x7f8fd80d0130 2024-06-29T15:57:52.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.108+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd81f66e0 con 0x7f8fd80d0130 2024-06-29T15:57:52.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.109+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd80cfbf0 con 0x7f8fd80d0130 2024-06-29T15:57:52.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.110+0000 7f8fcdffb640 1 -- 172.21.15.59:0/3488539214 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 131) ==== 98056+0+0 (secure 0 0 0) 0x7f8fc8005420 con 0x7f8fd80d0130 2024-06-29T15:57:52.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.110+0000 7f8fcdffb640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8fbc076170 0x7f8fbc078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.110+0000 7f8fcdffb640 1 -- 172.21.15.59:0/3488539214 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f8fc80982a0 con 0x7f8fd80d0130 2024-06-29T15:57:52.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.110+0000 7f8fdd34e640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8fbc076170 0x7f8fbc078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.111+0000 7f8fdd34e640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8fbc076170 0x7f8fbc078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4000f60 tx=0x7f8fd4009040 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.115+0000 7f8fcdffb640 1 -- 172.21.15.59:0/3488539214 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8fc80611e0 con 0x7f8fd80d0130 2024-06-29T15:57:52.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.334+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 --> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) -- 0x7f8fd81f5eb0 con 0x7f8fbc076170 2024-06-29T15:57:52.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.391+0000 7f8fcdffb640 1 -- 172.21.15.59:0/3488539214 <== mgr.11005 v2:172.21.15.153:6832/2014343395 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+55 (secure 0 0 0) 0x7f8fd81f5eb0 con 0x7f8fbc076170 2024-06-29T15:57:52.392 INFO:teuthology.orchestra.run.smithi059.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-06-29T15:57:52.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8fbc076170 msgr2=0x7f8fbc078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8fbc076170 0x7f8fbc078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4000f60 tx=0x7f8fd4009040 comp rx=0 tx=0).stop 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd80d0130 msgr2=0x7f8fd81596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd80d0130 0x7f8fd81596e0 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800e7c0 tx=0x7f8fc800ec80 comp rx=0 tx=0).stop 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 shutdown_connections 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8fbc076170 0x7f8fbc078680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8fd8143840 0x7f8fd815d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd80d0130 0x7f8fd81596e0 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 --2- 172.21.15.59:0/3488539214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fd806ea10 0x7f8fd80ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 >> 172.21.15.59:0/3488539214 conn(0x7f8fd80b2ab0 msgr2=0x7f8fd80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 shutdown_connections 2024-06-29T15:57:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.393+0000 7f8fdf5b2640 1 -- 172.21.15.59:0/3488539214 wait complete. 2024-06-29T15:57:52.406 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-06-29T15:57:52.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-06-29T15:57:52.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.531+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/706353175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815f8c0 msgr2=0x7fb6d80cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.531+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/706353175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815f8c0 0x7fb6d80cf860 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc005fd0 tx=0x7fb6cc02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.531+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/706353175 shutdown_connections 2024-06-29T15:57:52.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.531+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/706353175 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d80cfda0 0x7fb6d806cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.531+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/706353175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815f8c0 0x7fb6d80cf860 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.531+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/706353175 >> 172.21.15.59:0/706353175 conn(0x7fb6d80b2fe0 msgr2=0x7fb6d80cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.531+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/706353175 shutdown_connections 2024-06-29T15:57:52.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.531+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/706353175 wait complete. 2024-06-29T15:57:52.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.533+0000 7fb6dd5e1640 1 Processor -- start 2024-06-29T15:57:52.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.533+0000 7fb6dd5e1640 1 -- start start 2024-06-29T15:57:52.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.534+0000 7fb6dd5e1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d80cfda0 0x7fb6d815ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.534+0000 7fb6dd5e1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d815f8c0 0x7fb6d815e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.534+0000 7fb6dd5e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815b690 0x7fb6d815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.534+0000 7fb6dd5e1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d806da30 con 0x7fb6d80cfda0 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.534+0000 7fb6dd5e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d806d8b0 con 0x7fb6d815b690 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.534+0000 7fb6dd5e1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d806dbb0 con 0x7fb6d815f8c0 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d80cfda0 0x7fb6d815ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d80cfda0 0x7fb6d815ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54776/0 (socket says 172.21.15.59:54776) 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 -- 172.21.15.59:0/241685813 learned_addr learned my addr 172.21.15.59:0/241685813 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d77fe640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815b690 0x7fb6d815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d67fc640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d815f8c0 0x7fb6d815e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 -- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d815f8c0 msgr2=0x7fb6d815e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d815f8c0 0x7fb6d815e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 -- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815b690 msgr2=0x7fb6d815ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815b690 0x7fb6d815ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 -- 172.21.15.59:0/241685813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d81e5e00 con 0x7fb6d80cfda0 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d77fe640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815b690 0x7fb6d815ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6d6ffd640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d80cfda0 0x7fb6d815ae70 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800c3a0 tx=0x7fb6c800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6bf7fe640 1 -- 172.21.15.59:0/241685813 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80052d0 con 0x7fb6d80cfda0 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6bf7fe640 1 -- 172.21.15.59:0/241685813 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6c8007020 con 0x7fb6d80cfda0 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d81e60e0 con 0x7fb6d80cfda0 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6bf7fe640 1 -- 172.21.15.59:0/241685813 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8007470 con 0x7fb6d80cfda0 2024-06-29T15:57:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.535+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d81e66a0 con 0x7fb6d80cfda0 2024-06-29T15:57:52.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.536+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d806e0d0 con 0x7fb6d80cfda0 2024-06-29T15:57:52.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.537+0000 7fb6bf7fe640 1 -- 172.21.15.59:0/241685813 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7fb6c80067d0 con 0x7fb6d80cfda0 2024-06-29T15:57:52.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.538+0000 7fb6bf7fe640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb6c4076560 0x7fb6c4078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.538+0000 7fb6bf7fe640 1 -- 172.21.15.59:0/241685813 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fb6c8097650 con 0x7fb6d80cfda0 2024-06-29T15:57:52.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.538+0000 7fb6d67fc640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb6c4076560 0x7fb6c4078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.538+0000 7fb6d67fc640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb6c4076560 0x7fb6c4078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc033000 tx=0x7fb6cc004730 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.542+0000 7fb6bf7fe640 1 -- 172.21.15.59:0/241685813 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb6d80d01b0 con 0x7fb6d80cfda0 2024-06-29T15:57:52.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.754+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 --> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6d81e8720 con 0x7fb6c4076560 2024-06-29T15:57:52.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.761+0000 7fb6bf7fe640 1 -- 172.21.15.59:0/241685813 <== mgr.11005 v2:172.21.15.153:6832/2014343395 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+88 (secure 0 0 0) 0x7fb6d81e8720 con 0x7fb6c4076560 2024-06-29T15:57:52.762 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:52.762 INFO:teuthology.orchestra.run.smithi059.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb6c4076560 msgr2=0x7fb6c4078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb6c4076560 0x7fb6c4078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc033000 tx=0x7fb6cc004730 comp rx=0 tx=0).stop 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d80cfda0 msgr2=0x7fb6d815ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d80cfda0 0x7fb6d815ae70 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800c3a0 tx=0x7fb6c800c7a0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 shutdown_connections 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb6c4076560 0x7fb6c4078a70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d815b690 0x7fb6d815ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d815f8c0 0x7fb6d815e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 --2- 172.21.15.59:0/241685813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6d80cfda0 0x7fb6d815ae70 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 >> 172.21.15.59:0/241685813 conn(0x7fb6d80b2fe0 msgr2=0x7fb6d813e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 shutdown_connections 2024-06-29T15:57:52.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.763+0000 7fb6dd5e1640 1 -- 172.21.15.59:0/241685813 wait complete. 2024-06-29T15:57:52.776 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-06-29T15:57:52.776 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-06-29T15:57:52.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.897+0000 7fc307610640 1 -- 172.21.15.59:0/3883053639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000b2980 msgr2=0x7fc3000d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.897+0000 7fc307610640 1 --2- 172.21.15.59:0/3883053639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000b2980 0x7fc3000d27a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fc3000b3d60 tx=0x7fc2f002a4a0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.897+0000 7fc307610640 1 -- 172.21.15.59:0/3883053639 shutdown_connections 2024-06-29T15:57:52.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.897+0000 7fc307610640 1 --2- 172.21.15.59:0/3883053639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000b2980 0x7fc3000d27a0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.897+0000 7fc307610640 1 --2- 172.21.15.59:0/3883053639 >> v2:172.21.15.153:3300/0 conn(0x7fc3000d1c30 0x7fc3000b6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.897+0000 7fc307610640 1 -- 172.21.15.59:0/3883053639 >> 172.21.15.59:0/3883053639 conn(0x7fc3000d0f90 msgr2=0x7fc3000d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.897+0000 7fc307610640 1 -- 172.21.15.59:0/3883053639 shutdown_connections 2024-06-29T15:57:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.897+0000 7fc307610640 1 -- 172.21.15.59:0/3883053639 wait complete. 2024-06-29T15:57:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.899+0000 7fc307610640 1 Processor -- start 2024-06-29T15:57:52.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.899+0000 7fc307610640 1 -- start start 2024-06-29T15:57:52.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc307610640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3000b2980 0x7fc3000cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc307610640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000d1c30 0x7fc30014a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc307610640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3000d2ce0 0x7fc30014aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc307610640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc3000b4780 con 0x7fc3000d1c30 2024-06-29T15:57:52.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc307610640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc3000d3300 con 0x7fc3000d2ce0 2024-06-29T15:57:52.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc307610640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc3000b4900 con 0x7fc3000b2980 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc304bab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000d1c30 0x7fc30014a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc304bab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000d1c30 0x7fc30014a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46310/0 (socket says 172.21.15.59:46310) 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc304bab640 1 -- 172.21.15.59:0/2093302541 learned_addr learned my addr 172.21.15.59:0/2093302541 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc3053ac640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3000b2980 0x7fc3000cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc305bad640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3000d2ce0 0x7fc30014aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc304bab640 1 -- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3000b2980 msgr2=0x7fc3000cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc304bab640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3000b2980 0x7fc3000cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc304bab640 1 -- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3000d2ce0 msgr2=0x7fc30014aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc304bab640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3000d2ce0 0x7fc30014aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.900+0000 7fc304bab640 1 -- 172.21.15.59:0/2093302541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30014d400 con 0x7fc3000d1c30 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.901+0000 7fc304bab640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000d1c30 0x7fc30014a540 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc3000b65f0 tx=0x7fc2f0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.901+0000 7fc2f5ffb640 1 -- 172.21.15.59:0/2093302541 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0005c90 con 0x7fc3000d1c30 2024-06-29T15:57:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.901+0000 7fc2f5ffb640 1 -- 172.21.15.59:0/2093302541 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2f0031e50 con 0x7fc3000d1c30 2024-06-29T15:57:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.901+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30014d6e0 con 0x7fc3000d1c30 2024-06-29T15:57:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.901+0000 7fc2f5ffb640 1 -- 172.21.15.59:0/2093302541 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0005260 con 0x7fc3000d1c30 2024-06-29T15:57:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.901+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30014dca0 con 0x7fc3000d1c30 2024-06-29T15:57:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.902+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30014e080 con 0x7fc3000d1c30 2024-06-29T15:57:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.903+0000 7fc2f5ffb640 1 -- 172.21.15.59:0/2093302541 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7fc2f0005400 con 0x7fc3000d1c30 2024-06-29T15:57:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.903+0000 7fc2f5ffb640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc2e4076560 0x7fc2e4078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.903+0000 7fc2f5ffb640 1 -- 172.21.15.59:0/2093302541 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fc2f00b93c0 con 0x7fc3000d1c30 2024-06-29T15:57:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.903+0000 7fc3053ac640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc2e4076560 0x7fc2e4078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.904+0000 7fc3053ac640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc2e4076560 0x7fc2e4078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009b30 tx=0x7fc2fc005f90 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:52.908+0000 7fc2f5ffb640 1 -- 172.21.15.59:0/2093302541 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc30014e080 con 0x7fc3000d1c30 2024-06-29T15:57:53.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.122+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 --> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3000b4db0 con 0x7fc2e4076560 2024-06-29T15:57:53.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.129+0000 7fc2f5ffb640 1 -- 172.21.15.59:0/2093302541 <== mgr.11005 v2:172.21.15.153:6832/2014343395 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+353 (secure 0 0 0) 0x7fc3000b4db0 con 0x7fc2e4076560 2024-06-29T15:57:53.130 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:53.130 INFO:teuthology.orchestra.run.smithi059.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:57:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-06-29T15:57:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc2e4076560 msgr2=0x7fc2e4078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc2e4076560 0x7fc2e4078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009b30 tx=0x7fc2fc005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000d1c30 msgr2=0x7fc30014a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000d1c30 0x7fc30014a540 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc3000b65f0 tx=0x7fc2f0033c20 comp rx=0 tx=0).stop 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 shutdown_connections 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc2e4076560 0x7fc2e4078a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3000d2ce0 0x7fc30014aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000d1c30 0x7fc30014a540 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 --2- 172.21.15.59:0/2093302541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3000b2980 0x7fc3000cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.131+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 >> 172.21.15.59:0/2093302541 conn(0x7fc3000d0f90 msgr2=0x7fc30015aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.132+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 shutdown_connections 2024-06-29T15:57:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.132+0000 7fc307610640 1 -- 172.21.15.59:0/2093302541 wait complete. 2024-06-29T15:57:53.144 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:57:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-06-29T15:57:53.144 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:53.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.268+0000 7f7cb07eb640 1 -- 172.21.15.59:0/1411516930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80b3740 msgr2=0x7f7ca80b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.268+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/1411516930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80b3740 0x7f7ca80b3b50 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009830 tx=0x7f7c9c02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.268+0000 7f7cb07eb640 1 -- 172.21.15.59:0/1411516930 shutdown_connections 2024-06-29T15:57:53.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.268+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/1411516930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca815c2f0 0x7f7ca815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.268+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/1411516930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80b3740 0x7f7ca80b3b50 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.268+0000 7f7cb07eb640 1 -- 172.21.15.59:0/1411516930 >> 172.21.15.59:0/1411516930 conn(0x7f7ca806c950 msgr2=0x7f7ca8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:53.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.268+0000 7f7cb07eb640 1 -- 172.21.15.59:0/1411516930 shutdown_connections 2024-06-29T15:57:53.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.268+0000 7f7cb07eb640 1 -- 172.21.15.59:0/1411516930 wait complete. 2024-06-29T15:57:53.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.270+0000 7f7cb07eb640 1 Processor -- start 2024-06-29T15:57:53.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.270+0000 7f7cb07eb640 1 -- start start 2024-06-29T15:57:53.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cb07eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80b3740 0x7f7ca81f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cb07eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca815c2f0 0x7f7ca8140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cb07eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca8141390 0x7f7ca80aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cb07eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca80b5fd0 con 0x7f7ca815c2f0 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cb07eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca80b5e50 con 0x7f7ca80b3740 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cb07eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca80b6150 con 0x7f7ca8141390 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca815c2f0 0x7f7ca8140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca815c2f0 0x7f7ca8140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46314/0 (socket says 172.21.15.59:46314) 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 -- 172.21.15.59:0/354728918 learned_addr learned my addr 172.21.15.59:0/354728918 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7caed88640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca8141390 0x7f7ca80aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cae587640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80b3740 0x7f7ca81f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 -- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca8141390 msgr2=0x7f7ca80aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca8141390 0x7f7ca80aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 -- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80b3740 msgr2=0x7f7ca81f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80b3740 0x7f7ca81f2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 -- 172.21.15.59:0/354728918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca81417a0 con 0x7f7ca815c2f0 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7cadd86640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca815c2f0 0x7f7ca8140e50 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009950 tx=0x7f7c9c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.272+0000 7f7ca6ffd640 1 -- 172.21.15.59:0/354728918 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c005c90 con 0x7f7ca815c2f0 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.273+0000 7f7ca6ffd640 1 -- 172.21.15.59:0/354728918 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c9c005020 con 0x7f7ca815c2f0 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.273+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca8143200 con 0x7f7ca815c2f0 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.273+0000 7f7ca6ffd640 1 -- 172.21.15.59:0/354728918 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c005450 con 0x7f7ca815c2f0 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.273+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca81437c0 con 0x7f7ca815c2f0 2024-06-29T15:57:53.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.274+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca80b6670 con 0x7f7ca815c2f0 2024-06-29T15:57:53.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.275+0000 7f7ca6ffd640 1 -- 172.21.15.59:0/354728918 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7f7c9c0317c0 con 0x7f7ca815c2f0 2024-06-29T15:57:53.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.276+0000 7f7ca6ffd640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f7c94076590 0x7f7c94078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.276+0000 7f7ca6ffd640 1 -- 172.21.15.59:0/354728918 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f7c9c0b9570 con 0x7f7ca815c2f0 2024-06-29T15:57:53.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.276+0000 7f7cae587640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f7c94076590 0x7f7c94078aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:53.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.276+0000 7f7cae587640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f7c94076590 0x7f7c94078aa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800f290 tx=0x7f7c980020d0 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:53.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.279+0000 7f7ca6ffd640 1 -- 172.21.15.59:0/354728918 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7ca81417a0 con 0x7f7ca815c2f0 2024-06-29T15:57:53.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.493+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 --> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) -- 0x7f7ca8142f90 con 0x7f7c94076590 2024-06-29T15:57:53.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.556+0000 7f7ca6ffd640 1 -- 172.21.15.59:0/354728918 <== mgr.11005 v2:172.21.15.153:6832/2014343395 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f7ca8142f90 con 0x7f7c94076590 2024-06-29T15:57:53.557 INFO:teuthology.orchestra.run.smithi059.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f7c94076590 msgr2=0x7f7c94078aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f7c94076590 0x7f7c94078aa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800f290 tx=0x7f7c980020d0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca815c2f0 msgr2=0x7f7ca8140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca815c2f0 0x7f7ca8140e50 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009950 tx=0x7f7c9c003c10 comp rx=0 tx=0).stop 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 shutdown_connections 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f7c94076590 0x7f7c94078aa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ca8141390 0x7f7ca80aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ca815c2f0 0x7f7ca8140e50 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 --2- 172.21.15.59:0/354728918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80b3740 0x7f7ca81f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 >> 172.21.15.59:0/354728918 conn(0x7f7ca806c950 msgr2=0x7f7ca8070650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:53.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 shutdown_connections 2024-06-29T15:57:53.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.558+0000 7f7cb07eb640 1 -- 172.21.15.59:0/354728918 wait complete. 2024-06-29T15:57:53.570 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-06-29T15:57:53.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:53.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.692+0000 7fa68e3ba640 1 -- 172.21.15.59:0/839773424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa688162600 msgr2=0x7fa6880b3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.692+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/839773424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa688162600 0x7fa6880b3ed0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fa680005fd0 tx=0x7fa68002aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:53.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.692+0000 7fa68e3ba640 1 -- 172.21.15.59:0/839773424 shutdown_connections 2024-06-29T15:57:53.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.692+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/839773424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa688162600 0x7fa6880b3ed0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.693+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/839773424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa68815dff0 0x7fa6880d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.693+0000 7fa68e3ba640 1 -- 172.21.15.59:0/839773424 >> 172.21.15.59:0/839773424 conn(0x7fa6880d3e60 msgr2=0x7fa6880d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:53.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.693+0000 7fa68e3ba640 1 -- 172.21.15.59:0/839773424 shutdown_connections 2024-06-29T15:57:53.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.693+0000 7fa68e3ba640 1 -- 172.21.15.59:0/839773424 wait complete. 2024-06-29T15:57:53.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.694+0000 7fa68e3ba640 1 Processor -- start 2024-06-29T15:57:53.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.695+0000 7fa68e3ba640 1 -- start start 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa68e3ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa68815dff0 0x7fa6881f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa68e3ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa688161ae0 0x7fa6880cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa68e3ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688162600 0x7fa68815f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa68e3ba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa6880b4850 con 0x7fa68815dff0 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa68e3ba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6880b46d0 con 0x7fa688162600 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa68e3ba640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa6880b49d0 con 0x7fa688161ae0 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa68815dff0 0x7fa6881f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa68815dff0 0x7fa6881f7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46324/0 (socket says 172.21.15.59:46324) 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 -- 172.21.15.59:0/2316521368 learned_addr learned my addr 172.21.15.59:0/2316521368 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:53.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa68c957640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688162600 0x7fa68815f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa6877fe640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa688161ae0 0x7fa6880cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 -- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa688161ae0 msgr2=0x7fa6880cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa688161ae0 0x7fa6880cbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 -- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688162600 msgr2=0x7fa68815f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688162600 0x7fa68815f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 -- 172.21.15.59:0/2316521368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6881f8460 con 0x7fa68815dff0 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.696+0000 7fa687fff640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa68815dff0 0x7fa6881f7f20 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7fa67800d750 tx=0x7fa67800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.697+0000 7fa684ff9640 1 -- 172.21.15.59:0/2316521368 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678017070 con 0x7fa68815dff0 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.697+0000 7fa684ff9640 1 -- 172.21.15.59:0/2316521368 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa678004310 con 0x7fa68815dff0 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.697+0000 7fa684ff9640 1 -- 172.21.15.59:0/2316521368 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67800de60 con 0x7fa68815dff0 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.697+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6881f50f0 con 0x7fa68815dff0 2024-06-29T15:57:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.697+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6880cc990 con 0x7fa68815dff0 2024-06-29T15:57:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.698+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6880b4ef0 con 0x7fa68815dff0 2024-06-29T15:57:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.698+0000 7fa684ff9640 1 -- 172.21.15.59:0/2316521368 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7fa678009c30 con 0x7fa68815dff0 2024-06-29T15:57:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.699+0000 7fa684ff9640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fa674076560 0x7fa674078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.699+0000 7fa684ff9640 1 -- 172.21.15.59:0/2316521368 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fa678098720 con 0x7fa68815dff0 2024-06-29T15:57:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.699+0000 7fa6877fe640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fa674076560 0x7fa674078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.700+0000 7fa6877fe640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fa674076560 0x7fa674078a70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa6880aae80 tx=0x7fa67c009040 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:53.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.703+0000 7fa684ff9640 1 -- 172.21.15.59:0/2316521368 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa678020020 con 0x7fa68815dff0 2024-06-29T15:57:53.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.921+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 --> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7fa6880b2c20 con 0x7fa674076560 2024-06-29T15:57:53.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.984+0000 7fa684ff9640 1 -- 172.21.15.59:0/2316521368 <== mgr.11005 v2:172.21.15.153:6832/2014343395 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+61 (secure 0 0 0) 0x7fa6880b2c20 con 0x7fa674076560 2024-06-29T15:57:53.985 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-06-29T15:57:53.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fa674076560 msgr2=0x7fa674078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fa674076560 0x7fa674078a70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa6880aae80 tx=0x7fa67c009040 comp rx=0 tx=0).stop 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa68815dff0 msgr2=0x7fa6881f7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa68815dff0 0x7fa6881f7f20 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7fa67800d750 tx=0x7fa67800dc10 comp rx=0 tx=0).stop 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 shutdown_connections 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fa674076560 0x7fa674078a70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa688162600 0x7fa68815f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa688161ae0 0x7fa6880cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 --2- 172.21.15.59:0/2316521368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa68815dff0 0x7fa6881f7f20 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 >> 172.21.15.59:0/2316521368 conn(0x7fa6880d3e60 msgr2=0x7fa68815ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 shutdown_connections 2024-06-29T15:57:53.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:53.986+0000 7fa68e3ba640 1 -- 172.21.15.59:0/2316521368 wait complete. 2024-06-29T15:57:53.998 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-06-29T15:57:53.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:54.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.122+0000 7f988a4f6640 1 -- 172.21.15.59:0/524165766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f986c004d40 msgr2=0x7f986c0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.122+0000 7f988a4f6640 1 --2- 172.21.15.59:0/524165766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f986c004d40 0x7f986c0051f0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f98840b3490 tx=0x7f986c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:54.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.122+0000 7f9888a93640 1 -- 172.21.15.59:0/524165766 reap_dead start 2024-06-29T15:57:54.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.122+0000 7f988a4f6640 1 -- 172.21.15.59:0/524165766 shutdown_connections 2024-06-29T15:57:54.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.122+0000 7f988a4f6640 1 -- 172.21.15.59:0/524165766 >> 172.21.15.59:0/524165766 conn(0x7f98840d3e60 msgr2=0x7f98840d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.122+0000 7f988a4f6640 1 -- 172.21.15.59:0/524165766 shutdown_connections 2024-06-29T15:57:54.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.122+0000 7f988a4f6640 1 -- 172.21.15.59:0/524165766 wait complete. 2024-06-29T15:57:54.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.124+0000 7f988a4f6640 1 Processor -- start 2024-06-29T15:57:54.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.124+0000 7f988a4f6640 1 -- start start 2024-06-29T15:57:54.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f988a4f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98841f47b0 0x7f98840aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f988a4f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988406cbe0 0x7f98840b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f988a4f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98840ac570 0x7f98840ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f988a4f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f98840b45e0 con 0x7f98841f47b0 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f988a4f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98840d2aa0 con 0x7f98840ac570 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f988a4f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f98840b4760 con 0x7f988406cbe0 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f9883fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98841f47b0 0x7f98840aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f9883fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98841f47b0 0x7f98840aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46342/0 (socket says 172.21.15.59:46342) 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f9883fff640 1 -- 172.21.15.59:0/25714927 learned_addr learned my addr 172.21.15.59:0/25714927 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f98837fe640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988406cbe0 0x7f98840b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.125+0000 7f9888a93640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98840ac570 0x7f98840ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9883fff640 1 -- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988406cbe0 msgr2=0x7f98840b1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9883fff640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988406cbe0 0x7f98840b1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9883fff640 1 -- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98840ac570 msgr2=0x7f98840ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9883fff640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98840ac570 0x7f98840ac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9883fff640 1 -- 172.21.15.59:0/25714927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98840acec0 con 0x7f98841f47b0 2024-06-29T15:57:54.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9883fff640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98841f47b0 0x7f98840aae00 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f9874011ae0 tx=0x7f9874009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:54.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9880ff9640 1 -- 172.21.15.59:0/25714927 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987401e070 con 0x7f98841f47b0 2024-06-29T15:57:54.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9880ff9640 1 -- 172.21.15.59:0/25714927 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f987400fe70 con 0x7f98841f47b0 2024-06-29T15:57:54.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98840ab4d0 con 0x7f98841f47b0 2024-06-29T15:57:54.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f9880ff9640 1 -- 172.21.15.59:0/25714927 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874004290 con 0x7f98841f47b0 2024-06-29T15:57:54.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.126+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98840ccf10 con 0x7f98841f47b0 2024-06-29T15:57:54.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.127+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98840b4cc0 con 0x7f98841f47b0 2024-06-29T15:57:54.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.128+0000 7f9880ff9640 1 -- 172.21.15.59:0/25714927 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7f9874021070 con 0x7f98841f47b0 2024-06-29T15:57:54.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.128+0000 7f9880ff9640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f9870076560 0x7f9870078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.128+0000 7f9880ff9640 1 -- 172.21.15.59:0/25714927 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f9874098d30 con 0x7f98841f47b0 2024-06-29T15:57:54.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.129+0000 7f98837fe640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f9870076560 0x7f9870078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.129+0000 7f98837fe640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f9870076560 0x7f9870078a70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9878000c00 tx=0x7f9878006b70 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:54.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.133+0000 7f9880ff9640 1 -- 172.21.15.59:0/25714927 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f98840acec0 con 0x7f98841f47b0 2024-06-29T15:57:54.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.351+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 --> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7f98840b2400 con 0x7f9870076560 2024-06-29T15:57:54.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.418+0000 7f9880ff9640 1 -- 172.21.15.59:0/25714927 <== mgr.11005 v2:172.21.15.153:6832/2014343395 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+57 (secure 0 0 0) 0x7f98840b2400 con 0x7f9870076560 2024-06-29T15:57:54.419 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-06-29T15:57:54.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f9870076560 msgr2=0x7f9870078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f9870076560 0x7f9870078a70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9878000c00 tx=0x7f9878006b70 comp rx=0 tx=0).stop 2024-06-29T15:57:54.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98841f47b0 msgr2=0x7f98840aae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98841f47b0 0x7f98840aae00 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f9874011ae0 tx=0x7f9874009e90 comp rx=0 tx=0).stop 2024-06-29T15:57:54.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 shutdown_connections 2024-06-29T15:57:54.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f9870076560 0x7f9870078a70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98840ac570 0x7f98840ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f988406cbe0 0x7f98840b1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 --2- 172.21.15.59:0/25714927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98841f47b0 0x7f98840aae00 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 >> 172.21.15.59:0/25714927 conn(0x7f98840d3e60 msgr2=0x7f98840b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 shutdown_connections 2024-06-29T15:57:54.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.420+0000 7f988a4f6640 1 -- 172.21.15.59:0/25714927 wait complete. 2024-06-29T15:57:54.432 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-06-29T15:57:54.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:54.433 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-06-29T15:57:54.450 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:57:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.613+0000 7f4e16a64640 1 -- 172.21.15.59:0/1322111807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 msgr2=0x7f4df80040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.613+0000 7f4e16a64640 1 --2- 172.21.15.59:0/1322111807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 0x7f4df80040d0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c590 tx=0x7f4df8005830 comp rx=0 tx=0).stop 2024-06-29T15:57:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.613+0000 7f4e15001640 1 -- 172.21.15.59:0/1322111807 reap_dead start 2024-06-29T15:57:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.613+0000 7f4e16a64640 1 -- 172.21.15.59:0/1322111807 shutdown_connections 2024-06-29T15:57:54.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.613+0000 7f4e16a64640 1 -- 172.21.15.59:0/1322111807 >> 172.21.15.59:0/1322111807 conn(0x7f4e1006c950 msgr2=0x7f4e10073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.613+0000 7f4e16a64640 1 -- 172.21.15.59:0/1322111807 shutdown_connections 2024-06-29T15:57:54.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.613+0000 7f4e16a64640 1 -- 172.21.15.59:0/1322111807 wait complete. 2024-06-29T15:57:54.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.615+0000 7f4e16a64640 1 Processor -- start 2024-06-29T15:57:54.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.615+0000 7f4e16a64640 1 -- start start 2024-06-29T15:57:54.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e16a64640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 0x7f4e1013ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e16a64640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e1013f450 0x7f4e1015f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e16a64640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e1015bb40 0x7f4e1015f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e16a64640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4e10165020 con 0x7f4df8003c80 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e16a64640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e10164ea0 con 0x7f4e1013f450 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e16a64640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4e101651a0 con 0x7f4e1015bb40 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e14800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 0x7f4e1013ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e14800640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 0x7f4e1013ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46350/0 (socket says 172.21.15.59:46350) 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e14800640 1 -- 172.21.15.59:0/3424021038 learned_addr learned my addr 172.21.15.59:0/3424021038 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e0ffff640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e1013f450 0x7f4e1015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.616+0000 7f4e15001640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e1015bb40 0x7f4e1015f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e14800640 1 -- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e1015bb40 msgr2=0x7f4e1015f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e14800640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e1015bb40 0x7f4e1015f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e14800640 1 -- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e1013f450 msgr2=0x7f4e1015f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e14800640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e1013f450 0x7f4e1015f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e14800640 1 -- 172.21.15.59:0/3424021038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e101f47d0 con 0x7f4df8003c80 2024-06-29T15:57:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e14800640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 0x7f4e1013ef10 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c590 tx=0x7f4e08004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e0d7fa640 1 -- 172.21.15.59:0/3424021038 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0802b960 con 0x7f4df8003c80 2024-06-29T15:57:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e0d7fa640 1 -- 172.21.15.59:0/3424021038 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e0802bb00 con 0x7f4df8003c80 2024-06-29T15:57:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e101f4ab0 con 0x7f4df8003c80 2024-06-29T15:57:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e0d7fa640 1 -- 172.21.15.59:0/3424021038 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e080060c0 con 0x7f4df8003c80 2024-06-29T15:57:54.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.617+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e101f5070 con 0x7f4df8003c80 2024-06-29T15:57:54.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.618+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e101458a0 con 0x7f4df8003c80 2024-06-29T15:57:54.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.619+0000 7f4e0d7fa640 1 -- 172.21.15.59:0/3424021038 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 133) ==== 98479+0+0 (secure 0 0 0) 0x7f4e08006260 con 0x7f4df8003c80 2024-06-29T15:57:54.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.619+0000 7f4e0d7fa640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f4dfc076620 0x7f4dfc078b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.619+0000 7f4e0d7fa640 1 -- 172.21.15.59:0/3424021038 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f4e080b8e80 con 0x7f4df8003c80 2024-06-29T15:57:54.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.620+0000 7f4e0ffff640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f4dfc076620 0x7f4dfc078b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.620+0000 7f4e0ffff640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f4dfc076620 0x7f4dfc078b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e1013ec60 tx=0x7f4e04006b10 comp rx=0 tx=0).ready entity=mgr.11005 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:54.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.624+0000 7f4e0d7fa640 1 -- 172.21.15.59:0/3424021038 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4e08046060 con 0x7f4df8003c80 2024-06-29T15:57:54.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:54.864+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f4e100cf2b0 con 0x7f4df8003c80 2024-06-29T15:57:55.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.532+0000 7f4e0d7fa640 1 -- 172.21.15.59:0/3424021038 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v134) ==== 93+0+0 (secure 0 0 0) 0x7f4e08081ed0 con 0x7f4df8003c80 2024-06-29T15:57:55.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.535+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f4dfc076620 msgr2=0x7f4dfc078b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.535+0000 7f4e16a64640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f4dfc076620 0x7f4dfc078b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e1013ec60 tx=0x7f4e04006b10 comp rx=0 tx=0).stop 2024-06-29T15:57:55.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.535+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 msgr2=0x7f4e1013ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.535+0000 7f4e16a64640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 0x7f4e1013ef10 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c590 tx=0x7f4e08004bb0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.536+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 shutdown_connections 2024-06-29T15:57:55.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.536+0000 7f4e16a64640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f4dfc076620 0x7f4dfc078b30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.536+0000 7f4e16a64640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e1015bb40 0x7f4e1015f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.536+0000 7f4e16a64640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e1013f450 0x7f4e1015f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.536+0000 7f4e16a64640 1 --2- 172.21.15.59:0/3424021038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df8003c80 0x7f4e1013ef10 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.536+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 >> 172.21.15.59:0/3424021038 conn(0x7f4e1006c950 msgr2=0x7f4e1013ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:55.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.536+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 shutdown_connections 2024-06-29T15:57:55.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.536+0000 7f4e16a64640 1 -- 172.21.15.59:0/3424021038 wait complete. 2024-06-29T15:57:55.554 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:57:55.554 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:55.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:55.639 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:57:55.639 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:55.639 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:55.639 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:55.656 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:55.656 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:55.713 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.871+0000 7f8d039a9640 1 -- 172.21.15.59:0/460149557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cfc0b5320 msgr2=0x7f8cfc0b5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.871+0000 7f8d039a9640 1 --2- 172.21.15.59:0/460149557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cfc0b5320 0x7f8cfc0b5730 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4005fd0 tx=0x7f8cf402a810 comp rx=0 tx=0).stop 2024-06-29T15:57:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.872+0000 7f8d039a9640 1 -- 172.21.15.59:0/460149557 shutdown_connections 2024-06-29T15:57:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.872+0000 7f8d039a9640 1 --2- 172.21.15.59:0/460149557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cfc0b5320 0x7f8cfc0b5730 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.872+0000 7f8d039a9640 1 --2- 172.21.15.59:0/460149557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cfc06eb40 0x7f8cfc1443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.872+0000 7f8d039a9640 1 -- 172.21.15.59:0/460149557 >> 172.21.15.59:0/460149557 conn(0x7f8cfc073e50 msgr2=0x7f8cfc0b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.872+0000 7f8d039a9640 1 -- 172.21.15.59:0/460149557 shutdown_connections 2024-06-29T15:57:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.872+0000 7f8d039a9640 1 -- 172.21.15.59:0/460149557 wait complete. 2024-06-29T15:57:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.873+0000 7f8d039a9640 1 Processor -- start 2024-06-29T15:57:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.874+0000 7f8d039a9640 1 -- start start 2024-06-29T15:57:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d039a9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cfc06eb40 0x7f8cfc135030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d039a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cfc071620 0x7f8cfc13e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d039a9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cfc0b5320 0x7f8cfc0afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d039a9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc0cf520 con 0x7f8cfc071620 2024-06-29T15:57:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d039a9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc0cf3a0 con 0x7f8cfc06eb40 2024-06-29T15:57:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d039a9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8cfc0cf6a0 con 0x7f8cfc0b5320 2024-06-29T15:57:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d00f44640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cfc071620 0x7f8cfc13e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d00f44640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cfc071620 0x7f8cfc13e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46380/0 (socket says 172.21.15.59:46380) 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d00f44640 1 -- 172.21.15.59:0/1639982081 learned_addr learned my addr 172.21.15.59:0/1639982081 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d01745640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cfc06eb40 0x7f8cfc135030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d01f46640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cfc0b5320 0x7f8cfc0afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d00f44640 1 -- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cfc0b5320 msgr2=0x7f8cfc0afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d00f44640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cfc0b5320 0x7f8cfc0afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d00f44640 1 -- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cfc06eb40 msgr2=0x7f8cfc135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d00f44640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cfc06eb40 0x7f8cfc135030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.875+0000 7f8d00f44640 1 -- 172.21.15.59:0/1639982081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cfc0c5a70 con 0x7f8cfc071620 2024-06-29T15:57:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.876+0000 7f8d00f44640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cfc071620 0x7f8cfc13e3b0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000e790 tx=0x7f8cf000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.876+0000 7f8cedffb640 1 -- 172.21.15.59:0/1639982081 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf000eea0 con 0x7f8cfc071620 2024-06-29T15:57:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.876+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cfc0c5ee0 con 0x7f8cfc071620 2024-06-29T15:57:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.876+0000 7f8cedffb640 1 -- 172.21.15.59:0/1639982081 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8cf000cdb0 con 0x7f8cfc071620 2024-06-29T15:57:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.876+0000 7f8cedffb640 1 -- 172.21.15.59:0/1639982081 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0010690 con 0x7f8cfc071620 2024-06-29T15:57:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.876+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cfc0c64a0 con 0x7f8cfc071620 2024-06-29T15:57:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.877+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cfc0c5c40 con 0x7f8cfc071620 2024-06-29T15:57:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.878+0000 7f8cedffb640 1 -- 172.21.15.59:0/1639982081 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f8cf0010830 con 0x7f8cfc071620 2024-06-29T15:57:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.878+0000 7f8cedffb640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8ce00765d0 0x7f8ce0078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.879+0000 7f8cedffb640 1 -- 172.21.15.59:0/1639982081 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f8cf0098cf0 con 0x7f8cfc071620 2024-06-29T15:57:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.879+0000 7f8d01745640 1 -- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8ce00765d0 msgr2=0x7f8ce0078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.879+0000 7f8d01745640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8ce00765d0 0x7f8ce0078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:55.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:55.883+0000 7f8cedffb640 1 -- 172.21.15.59:0/1639982081 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8cfc0c5c40 con 0x7f8cfc071620 2024-06-29T15:57:56.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.075+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8cfc1443e0 con 0x7f8cfc071620 2024-06-29T15:57:56.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.076+0000 7f8cedffb640 1 -- 172.21.15.59:0/1639982081 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) ==== 74+0+21283 (secure 0 0 0) 0x7f8cf0061410 con 0x7f8cfc071620 2024-06-29T15:57:56.078 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:56.078 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":227,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:50.460468+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2050849752":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/239127507":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2051017624":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/4093149557":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/572137829":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2737762390":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6833/2780049427":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2780049427":"2024-06-30T15:57:50.460403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:56.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.078+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8ce00765d0 msgr2=0x7f8ce0078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:56.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.078+0000 7f8d039a9640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8ce00765d0 0x7f8ce0078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cfc071620 msgr2=0x7f8cfc13e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cfc071620 0x7f8cfc13e3b0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000e790 tx=0x7f8cf000ec50 comp rx=0 tx=0).stop 2024-06-29T15:57:56.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 shutdown_connections 2024-06-29T15:57:56.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f8ce00765d0 0x7f8ce0078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cfc0b5320 0x7f8cfc0afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cfc071620 0x7f8cfc13e3b0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 --2- 172.21.15.59:0/1639982081 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cfc06eb40 0x7f8cfc135030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 >> 172.21.15.59:0/1639982081 conn(0x7f8cfc073e50 msgr2=0x7f8cfc15b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 shutdown_connections 2024-06-29T15:57:56.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.079+0000 7f8d039a9640 1 -- 172.21.15.59:0/1639982081 wait complete. 2024-06-29T15:57:56.092 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:57:19.955891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '221', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:57:20.545869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:57:32.305142+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:57:33.647895+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:56.092 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.213+0000 7fd907a81640 1 -- 172.21.15.59:0/1241736235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000d1c30 msgr2=0x7fd900166a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.213+0000 7fd907a81640 1 --2- 172.21.15.59:0/1241736235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000d1c30 0x7fd900166a20 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8005fd0 tx=0x7fd8f802a940 comp rx=0 tx=0).stop 2024-06-29T15:57:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.214+0000 7fd907a81640 1 -- 172.21.15.59:0/1241736235 shutdown_connections 2024-06-29T15:57:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.214+0000 7fd907a81640 1 --2- 172.21.15.59:0/1241736235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000d1c30 0x7fd900166a20 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.214+0000 7fd907a81640 1 -- 172.21.15.59:0/1241736235 >> 172.21.15.59:0/1241736235 conn(0x7fd900073e10 msgr2=0x7fd900074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.214+0000 7fd907a81640 1 -- 172.21.15.59:0/1241736235 shutdown_connections 2024-06-29T15:57:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.214+0000 7fd907a81640 1 -- 172.21.15.59:0/1241736235 wait complete. 2024-06-29T15:57:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.215+0000 7fd907a81640 1 Processor -- start 2024-06-29T15:57:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.216+0000 7fd907a81640 1 -- start start 2024-06-29T15:57:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd907a81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9000b2270 0x7fd90014b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd907a81640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000b2fa0 0x7fd90014d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd907a81640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9000d1c30 0x7fd90015a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd907a81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd9001670e0 con 0x7fd9000b2270 2024-06-29T15:57:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd907a81640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd900166f60 con 0x7fd9000b2fa0 2024-06-29T15:57:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd907a81640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd900167260 con 0x7fd9000d1c30 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90581d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9000b2270 0x7fd90014b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90581d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9000b2270 0x7fd90014b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46392/0 (socket says 172.21.15.59:46392) 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90581d640 1 -- 172.21.15.59:0/799656935 learned_addr learned my addr 172.21.15.59:0/799656935 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90501c640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000b2fa0 0x7fd90014d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90601e640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9000d1c30 0x7fd90015a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90581d640 1 -- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9000d1c30 msgr2=0x7fd90015a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90581d640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9000d1c30 0x7fd90015a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90581d640 1 -- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000b2fa0 msgr2=0x7fd90014d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90581d640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000b2fa0 0x7fd90014d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90581d640 1 -- 172.21.15.59:0/799656935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90015e860 con 0x7fd9000b2270 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.217+0000 7fd90501c640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000b2fa0 0x7fd90014d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.218+0000 7fd90581d640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9000b2270 0x7fd90014b4d0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00c460 tx=0x7fd8fc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.218+0000 7fd8f27fc640 1 -- 172.21.15.59:0/799656935 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc0052d0 con 0x7fd9000b2270 2024-06-29T15:57:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.218+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd90015eb40 con 0x7fd9000b2270 2024-06-29T15:57:56.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.218+0000 7fd8f27fc640 1 -- 172.21.15.59:0/799656935 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8fc005490 con 0x7fd9000b2270 2024-06-29T15:57:56.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.218+0000 7fd8f27fc640 1 -- 172.21.15.59:0/799656935 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc012870 con 0x7fd9000b2270 2024-06-29T15:57:56.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.218+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90015f100 con 0x7fd9000b2270 2024-06-29T15:57:56.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.219+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd90015e400 con 0x7fd9000b2270 2024-06-29T15:57:56.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.220+0000 7fd8f27fc640 1 -- 172.21.15.59:0/799656935 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7fd8fc012a10 con 0x7fd9000b2270 2024-06-29T15:57:56.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.220+0000 7fd8f27fc640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fd8e40765d0 0x7fd8e4078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.220+0000 7fd90501c640 1 -- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fd8e40765d0 msgr2=0x7fd8e4078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:56.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.220+0000 7fd8f27fc640 1 -- 172.21.15.59:0/799656935 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fd8fc097c70 con 0x7fd9000b2270 2024-06-29T15:57:56.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.220+0000 7fd90501c640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fd8e40765d0 0x7fd8e4078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:56.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.225+0000 7fd8f27fc640 1 -- 172.21.15.59:0/799656935 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd8fc0605c0 con 0x7fd9000b2270 2024-06-29T15:57:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.415+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd90015bc20 con 0x7fd9000b2270 2024-06-29T15:57:56.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.416+0000 7fd8f27fc640 1 -- 172.21.15.59:0/799656935 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v227) ==== 93+0+10 (secure 0 0 0) 0x7fd8fc0648c0 con 0x7fd9000b2270 2024-06-29T15:57:56.417 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:57:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fd8e40765d0 msgr2=0x7fd8e4078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fd8e40765d0 0x7fd8e4078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9000b2270 msgr2=0x7fd90014b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9000b2270 0x7fd90014b4d0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00c460 tx=0x7fd8fc00c920 comp rx=0 tx=0).stop 2024-06-29T15:57:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 shutdown_connections 2024-06-29T15:57:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fd8e40765d0 0x7fd8e4078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9000d1c30 0x7fd90015a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9000b2fa0 0x7fd90014d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 --2- 172.21.15.59:0/799656935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9000b2270 0x7fd90014b4d0 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 >> 172.21.15.59:0/799656935 conn(0x7fd900073e10 msgr2=0x7fd900071040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 shutdown_connections 2024-06-29T15:57:56.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.418+0000 7fd907a81640 1 -- 172.21.15.59:0/799656935 wait complete. 2024-06-29T15:57:56.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:56.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.554+0000 7fc7236ef640 1 -- 172.21.15.59:0/4225533614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 msgr2=0x7fc71c15bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.554+0000 7fc7236ef640 1 --2- 172.21.15.59:0/4225533614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 0x7fc71c15bcd0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc718009830 tx=0x7fc71802a530 comp rx=0 tx=0).stop 2024-06-29T15:57:56.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.554+0000 7fc7236ef640 1 -- 172.21.15.59:0/4225533614 shutdown_connections 2024-06-29T15:57:56.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.554+0000 7fc7236ef640 1 --2- 172.21.15.59:0/4225533614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 0x7fc71c15bcd0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.554+0000 7fc7236ef640 1 -- 172.21.15.59:0/4225533614 >> 172.21.15.59:0/4225533614 conn(0x7fc71c073e50 msgr2=0x7fc71c0b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.554+0000 7fc7236ef640 1 -- 172.21.15.59:0/4225533614 shutdown_connections 2024-06-29T15:57:56.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.554+0000 7fc7236ef640 1 -- 172.21.15.59:0/4225533614 wait complete. 2024-06-29T15:57:56.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.556+0000 7fc7236ef640 1 Processor -- start 2024-06-29T15:57:56.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.556+0000 7fc7236ef640 1 -- start start 2024-06-29T15:57:56.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc7236ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc71c06eb40 0x7fc71c1652a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc7236ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 0x7fc71c140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc7236ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc71c0b5320 0x7fc71c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc7236ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c1661d0 con 0x7fc71c0b5320 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc7236ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c166050 con 0x7fc71c071620 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc7236ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc71c166350 con 0x7fc71c06eb40 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc721c8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc71c0b5320 0x7fc71c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc72148b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc71c06eb40 0x7fc71c1652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc720c8a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 0x7fc71c140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc721c8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc71c0b5320 0x7fc71c06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46402/0 (socket says 172.21.15.59:46402) 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.557+0000 7fc721c8c640 1 -- 172.21.15.59:0/3356894720 learned_addr learned my addr 172.21.15.59:0/3356894720 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc720c8a640 1 -- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc71c06eb40 msgr2=0x7fc71c1652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc720c8a640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc71c06eb40 0x7fc71c1652a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc720c8a640 1 -- 172.21.15.59:0/3356894720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc71c0b5320 msgr2=0x7fc71c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc720c8a640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc71c0b5320 0x7fc71c06ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc720c8a640 1 -- 172.21.15.59:0/3356894720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71c0a9540 con 0x7fc71c071620 2024-06-29T15:57:56.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc72148b640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc71c06eb40 0x7fc71c1652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:56.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc721c8c640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc71c0b5320 0x7fc71c06ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:56.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc720c8a640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 0x7fc71c140780 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00c3c0 tx=0x7fc70c00c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:56.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc711ffb640 1 -- 172.21.15.59:0/3356894720 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c017070 con 0x7fc71c071620 2024-06-29T15:57:56.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc711ffb640 1 -- 172.21.15.59:0/3356894720 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc70c0035f0 con 0x7fc71c071620 2024-06-29T15:57:56.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc71c0a9820 con 0x7fc71c071620 2024-06-29T15:57:56.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc711ffb640 1 -- 172.21.15.59:0/3356894720 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70c003a80 con 0x7fc71c071620 2024-06-29T15:57:56.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.558+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc71c0a9de0 con 0x7fc71c071620 2024-06-29T15:57:56.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.560+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc71c15bcd0 con 0x7fc71c071620 2024-06-29T15:57:56.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.560+0000 7fc711ffb640 1 -- 172.21.15.59:0/3356894720 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7fc70c01a070 con 0x7fc71c071620 2024-06-29T15:57:56.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.561+0000 7fc711ffb640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc7000765d0 0x7fc700078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.561+0000 7fc72148b640 1 -- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc7000765d0 msgr2=0x7fc700078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:56.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.561+0000 7fc711ffb640 1 -- 172.21.15.59:0/3356894720 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fc70c097580 con 0x7fc71c071620 2024-06-29T15:57:56.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.561+0000 7fc72148b640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc7000765d0 0x7fc700078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:56.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.570+0000 7fc711ffb640 1 -- 172.21.15.59:0/3356894720 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc71c15c210 con 0x7fc71c071620 2024-06-29T15:57:56.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.761+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc71c0cec90 con 0x7fc71c071620 2024-06-29T15:57:56.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.762+0000 7fc711ffb640 1 -- 172.21.15.59:0/3356894720 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v227) ==== 104+0+11 (secure 0 0 0) 0x7fc70c05fe30 con 0x7fc71c071620 2024-06-29T15:57:56.763 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:57:56.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.762+0000 7fc72148b640 1 -- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc7000765d0 msgr2=0x7fc700078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:56.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.762+0000 7fc72148b640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc7000765d0 0x7fc700078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:56.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc7000765d0 msgr2=0x7fc700078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:56.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc7000765d0 0x7fc700078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 msgr2=0x7fc71c140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 0x7fc71c140780 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00c3c0 tx=0x7fc70c00c7c0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 shutdown_connections 2024-06-29T15:57:56.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fc7000765d0 0x7fc700078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc71c0b5320 0x7fc71c06ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc71c071620 0x7fc71c140780 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 --2- 172.21.15.59:0/3356894720 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc71c06eb40 0x7fc71c1652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 >> 172.21.15.59:0/3356894720 conn(0x7fc71c073e50 msgr2=0x7fc71c15cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 shutdown_connections 2024-06-29T15:57:56.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.764+0000 7fc7236ef640 1 -- 172.21.15.59:0/3356894720 wait complete. 2024-06-29T15:57:56.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.899+0000 7fb7a43e1640 1 -- 172.21.15.59:0/272842650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c06ebe0 msgr2=0x7fb79c0cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.899+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/272842650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c06ebe0 0x7fb79c0cef90 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0753d0 tx=0x7fb79402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.899+0000 7fb7a43e1640 1 -- 172.21.15.59:0/272842650 shutdown_connections 2024-06-29T15:57:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.899+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/272842650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c06ebe0 0x7fb79c0cef90 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.899+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/272842650 >> v2:172.21.15.153:3300/0 conn(0x7fb79c0d0130 0x7fb79c143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.899+0000 7fb7a43e1640 1 -- 172.21.15.59:0/272842650 >> 172.21.15.59:0/272842650 conn(0x7fb79c0b2ab0 msgr2=0x7fb79c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.899+0000 7fb7a43e1640 1 -- 172.21.15.59:0/272842650 shutdown_connections 2024-06-29T15:57:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.899+0000 7fb7a43e1640 1 -- 172.21.15.59:0/272842650 wait complete. 2024-06-29T15:57:56.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.901+0000 7fb7a43e1640 1 Processor -- start 2024-06-29T15:57:56.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.901+0000 7fb7a43e1640 1 -- start start 2024-06-29T15:57:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a43e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb79c06ebe0 0x7fb79c13d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a43e1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb79c0b4ee0 0x7fb79c15bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a43e1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c0d0130 0x7fb79c142ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a43e1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb79c0cf650 con 0x7fb79c0d0130 2024-06-29T15:57:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a43e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb79c0cf4d0 con 0x7fb79c06ebe0 2024-06-29T15:57:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a43e1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb79c0cf7d0 con 0x7fb79c0b4ee0 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a297e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c0d0130 0x7fb79c142ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a297e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c0d0130 0x7fb79c142ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46414/0 (socket says 172.21.15.59:46414) 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a297e640 1 -- 172.21.15.59:0/1468385054 learned_addr learned my addr 172.21.15.59:0/1468385054 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a217d640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb79c06ebe0 0x7fb79c13d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a197c640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb79c0b4ee0 0x7fb79c15bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a297e640 1 -- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb79c0b4ee0 msgr2=0x7fb79c15bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a297e640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb79c0b4ee0 0x7fb79c15bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a297e640 1 -- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb79c06ebe0 msgr2=0x7fb79c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a297e640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb79c06ebe0 0x7fb79c13d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.902+0000 7fb7a297e640 1 -- 172.21.15.59:0/1468385054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb79c1f1540 con 0x7fb79c0d0130 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.903+0000 7fb7a297e640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c0d0130 0x7fb79c142ad0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fb79402aea0 tx=0x7fb7940316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.903+0000 7fb792ffd640 1 -- 172.21.15.59:0/1468385054 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794031da0 con 0x7fb79c0d0130 2024-06-29T15:57:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.903+0000 7fb792ffd640 1 -- 172.21.15.59:0/1468385054 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb794003a60 con 0x7fb79c0d0130 2024-06-29T15:57:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.903+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb79c0a9e50 con 0x7fb79c0d0130 2024-06-29T15:57:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.903+0000 7fb792ffd640 1 -- 172.21.15.59:0/1468385054 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7940330d0 con 0x7fb79c0d0130 2024-06-29T15:57:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.903+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb79c0aa410 con 0x7fb79c0d0130 2024-06-29T15:57:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.904+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb79c073a40 con 0x7fb79c0d0130 2024-06-29T15:57:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.905+0000 7fb792ffd640 1 -- 172.21.15.59:0/1468385054 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7fb794033270 con 0x7fb79c0d0130 2024-06-29T15:57:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.905+0000 7fb792ffd640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb7800765a0 0x7fb780078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.905+0000 7fb7a217d640 1 -- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb7800765a0 msgr2=0x7fb780078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.905+0000 7fb792ffd640 1 -- 172.21.15.59:0/1468385054 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fb7940b95e0 con 0x7fb79c0d0130 2024-06-29T15:57:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.905+0000 7fb7a217d640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb7800765a0 0x7fb780078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:56.910+0000 7fb792ffd640 1 -- 172.21.15.59:0/1468385054 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb79c073a40 con 0x7fb79c0d0130 2024-06-29T15:57:57.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.101+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb79c0b3110 con 0x7fb79c0d0130 2024-06-29T15:57:57.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.102+0000 7fb792ffd640 1 -- 172.21.15.59:0/1468385054 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v227) ==== 100+0+11 (secure 0 0 0) 0x7fb79403f030 con 0x7fb79c0d0130 2024-06-29T15:57:57.103 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:57:57.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb7800765a0 msgr2=0x7fb780078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb7800765a0 0x7fb780078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c0d0130 msgr2=0x7fb79c142ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c0d0130 0x7fb79c142ad0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fb79402aea0 tx=0x7fb7940316f0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 shutdown_connections 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fb7800765a0 0x7fb780078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb79c0d0130 0x7fb79c142ad0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb79c0b4ee0 0x7fb79c15bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 --2- 172.21.15.59:0/1468385054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb79c06ebe0 0x7fb79c13d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 >> 172.21.15.59:0/1468385054 conn(0x7fb79c0b2ab0 msgr2=0x7fb79c15a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 shutdown_connections 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.104+0000 7fb7a43e1640 1 -- 172.21.15.59:0/1468385054 wait complete. 2024-06-29T15:57:57.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:57:57.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.237+0000 7f0049c47640 1 -- 172.21.15.59:0/2886243016 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 msgr2=0x7f004415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.237+0000 7f0049c47640 1 --2- 172.21.15.59:0/2886243016 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 0x7f004415bc20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0034009830 tx=0x7f003402a4a0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.237+0000 7f0049c47640 1 -- 172.21.15.59:0/2886243016 shutdown_connections 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.237+0000 7f0049c47640 1 --2- 172.21.15.59:0/2886243016 >> v2:172.21.15.153:3300/0 conn(0x7f00440b2fa0 0x7f00440d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.237+0000 7f0049c47640 1 --2- 172.21.15.59:0/2886243016 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 0x7f004415bc20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.237+0000 7f0049c47640 1 -- 172.21.15.59:0/2886243016 >> 172.21.15.59:0/2886243016 conn(0x7f0044073e10 msgr2=0x7f0044074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.237+0000 7f0049c47640 1 -- 172.21.15.59:0/2886243016 shutdown_connections 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.237+0000 7f0049c47640 1 -- 172.21.15.59:0/2886243016 wait complete. 2024-06-29T15:57:57.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.239+0000 7f0049c47640 1 Processor -- start 2024-06-29T15:57:57.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.239+0000 7f0049c47640 1 -- start start 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.240+0000 7f0049c47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 0x7f0044155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.240+0000 7f0049c47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00440b2fa0 0x7f00440afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.240+0000 7f0049c47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00440d1a60 0x7f0044159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.240+0000 7f0049c47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f00440b6cc0 con 0x7f00440d1a60 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.240+0000 7f0049c47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00440b6b40 con 0x7f00440b2fa0 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.240+0000 7f0049c47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f00440cfe90 con 0x7f00440b2270 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f0043fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00440d1a60 0x7f0044159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f00437fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 0x7f0044155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f0043fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00440d1a60 0x7f0044159e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46428/0 (socket says 172.21.15.59:46428) 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f0043fff640 1 -- 172.21.15.59:0/588512997 learned_addr learned my addr 172.21.15.59:0/588512997 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f003affd640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00440b2fa0 0x7f00440afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f00437fe640 1 -- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00440b2fa0 msgr2=0x7f00440afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f00437fe640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00440b2fa0 0x7f00440afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f00437fe640 1 -- 172.21.15.59:0/588512997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00440d1a60 msgr2=0x7f0044159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f00437fe640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00440d1a60 0x7f0044159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f00437fe640 1 -- 172.21.15.59:0/588512997 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00440b06f0 con 0x7f00440b2270 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f0043fff640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00440d1a60 0x7f0044159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f003affd640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00440b2fa0 0x7f00440afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f00437fe640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 0x7f0044155810 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0034031000 tx=0x7f0034003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.241+0000 7f0040ff9640 1 -- 172.21.15.59:0/588512997 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034005c90 con 0x7f00440b2270 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.242+0000 7f0040ff9640 1 -- 172.21.15.59:0/588512997 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0034005020 con 0x7f00440b2270 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.242+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00440b09d0 con 0x7f00440b2270 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.242+0000 7f0040ff9640 1 -- 172.21.15.59:0/588512997 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034005450 con 0x7f00440b2270 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.242+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f00440ac070 con 0x7f00440b2270 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.243+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00440d03c0 con 0x7f00440b2270 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.243+0000 7f0040ff9640 1 -- 172.21.15.59:0/588512997 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f00340317c0 con 0x7f00440b2270 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.244+0000 7f0040ff9640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f00240765d0 0x7f0024078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.244+0000 7f0040ff9640 1 -- 172.21.15.59:0/588512997 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f00340b9220 con 0x7f00440b2270 2024-06-29T15:57:57.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.244+0000 7f003affd640 1 -- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f00240765d0 msgr2=0x7f0024078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:57.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.244+0000 7f003affd640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f00240765d0 0x7f0024078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:57.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.248+0000 7f0040ff9640 1 -- 172.21.15.59:0/588512997 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f003403f030 con 0x7f00440b2270 2024-06-29T15:57:57.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.439+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f0044157cb0 con 0x7f00440b2270 2024-06-29T15:57:57.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.439+0000 7f0040ff9640 1 -- 172.21.15.59:0/588512997 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v227) ==== 107+0+11 (secure 0 0 0) 0x7f003408a090 con 0x7f00440b2270 2024-06-29T15:57:57.440 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.441+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f00240765d0 msgr2=0x7f0024078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.441+0000 7f0049c47640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f00240765d0 0x7f0024078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.441+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 msgr2=0x7f0044155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.441+0000 7f0049c47640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 0x7f0044155810 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0034031000 tx=0x7f0034003c10 comp rx=0 tx=0).stop 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.442+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 shutdown_connections 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.442+0000 7f0049c47640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f00240765d0 0x7f0024078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.442+0000 7f0049c47640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00440d1a60 0x7f0044159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.442+0000 7f0049c47640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00440b2fa0 0x7f00440afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.442+0000 7f0049c47640 1 --2- 172.21.15.59:0/588512997 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00440b2270 0x7f0044155810 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.442+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 >> 172.21.15.59:0/588512997 conn(0x7f0044073e10 msgr2=0x7f00440d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.442+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 shutdown_connections 2024-06-29T15:57:57.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.442+0000 7f0049c47640 1 -- 172.21.15.59:0/588512997 wait complete. 2024-06-29T15:57:57.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:57:57.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.576+0000 7fe56d7bd640 1 -- 172.21.15.59:0/1249464621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 msgr2=0x7fe56815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.576+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/1249464621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 0x7fe56815f9b0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fe560005fd0 tx=0x7fe56002aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:57.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.576+0000 7fe56d7bd640 1 -- 172.21.15.59:0/1249464621 shutdown_connections 2024-06-29T15:57:57.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.576+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/1249464621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 0x7fe56815f9b0 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.576+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/1249464621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5680d23c0 0x7fe5680b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.576+0000 7fe56d7bd640 1 -- 172.21.15.59:0/1249464621 >> 172.21.15.59:0/1249464621 conn(0x7fe5680d3ee0 msgr2=0x7fe5680d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.576+0000 7fe56d7bd640 1 -- 172.21.15.59:0/1249464621 shutdown_connections 2024-06-29T15:57:57.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.576+0000 7fe56d7bd640 1 -- 172.21.15.59:0/1249464621 wait complete. 2024-06-29T15:57:57.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.578+0000 7fe56d7bd640 1 Processor -- start 2024-06-29T15:57:57.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.578+0000 7fe56d7bd640 1 -- start start 2024-06-29T15:57:57.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.579+0000 7fe56d7bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5680b3ed0 0x7fe56815d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.579+0000 7fe56d7bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5680d23c0 0x7fe568148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.579+0000 7fe56d7bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 0x7fe568145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.579+0000 7fe56d7bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe568160280 con 0x7fe56815efe0 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.579+0000 7fe56d7bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe568160100 con 0x7fe5680d23c0 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.579+0000 7fe56d7bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe568160400 con 0x7fe5680b3ed0 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe5677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 0x7fe568145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe566ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5680b3ed0 0x7fe56815d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe5677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 0x7fe568145600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46454/0 (socket says 172.21.15.59:46454) 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe566ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5680b3ed0 0x7fe56815d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:59434/0 (socket says 172.21.15.59:59434) 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe5667fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5680d23c0 0x7fe568148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe5677fe640 1 -- 172.21.15.59:0/4221838731 learned_addr learned my addr 172.21.15.59:0/4221838731 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe566ffd640 1 -- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5680d23c0 msgr2=0x7fe568148d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe566ffd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5680d23c0 0x7fe568148d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe566ffd640 1 -- 172.21.15.59:0/4221838731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 msgr2=0x7fe568145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe566ffd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 0x7fe568145600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe566ffd640 1 -- 172.21.15.59:0/4221838731 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56815de00 con 0x7fe5680b3ed0 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe5667fc640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5680d23c0 0x7fe568148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe5677fe640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 0x7fe568145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe566ffd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5680b3ed0 0x7fe56815d490 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe55800e6d0 tx=0x7fe55800eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe54f7fe640 1 -- 172.21.15.59:0/4221838731 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558004220 con 0x7fe5680b3ed0 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe54f7fe640 1 -- 172.21.15.59:0/4221838731 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5580043c0 con 0x7fe5680b3ed0 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe56815e0e0 con 0x7fe5680b3ed0 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe54f7fe640 1 -- 172.21.15.59:0/4221838731 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55800cb10 con 0x7fe5680b3ed0 2024-06-29T15:57:57.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.580+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5681eb1c0 con 0x7fe5680b3ed0 2024-06-29T15:57:57.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.581+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5680b3990 con 0x7fe5680b3ed0 2024-06-29T15:57:57.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.582+0000 7fe54f7fe640 1 -- 172.21.15.59:0/4221838731 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7fe55800ccb0 con 0x7fe5680b3ed0 2024-06-29T15:57:57.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.583+0000 7fe54f7fe640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe5540765a0 0x7fe554078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.583+0000 7fe5667fc640 1 -- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe5540765a0 msgr2=0x7fe554078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:57.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.583+0000 7fe54f7fe640 1 -- 172.21.15.59:0/4221838731 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fe558098a90 con 0x7fe5680b3ed0 2024-06-29T15:57:57.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.583+0000 7fe5667fc640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe5540765a0 0x7fe554078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:57.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.587+0000 7fe54f7fe640 1 -- 172.21.15.59:0/4221838731 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe558061230 con 0x7fe5680b3ed0 2024-06-29T15:57:57.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.781+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7fe5680d2950 con 0x7fe5680b3ed0 2024-06-29T15:57:57.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.781+0000 7fe54f7fe640 1 -- 172.21.15.59:0/4221838731 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v227) ==== 103+0+11 (secure 0 0 0) 0x7fe558065530 con 0x7fe5680b3ed0 2024-06-29T15:57:57.782 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:57:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe5540765a0 msgr2=0x7fe554078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe5540765a0 0x7fe554078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5680b3ed0 msgr2=0x7fe56815d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5680b3ed0 0x7fe56815d490 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe55800e6d0 tx=0x7fe55800eb90 comp rx=0 tx=0).stop 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 shutdown_connections 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe5540765a0 0x7fe554078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe56815efe0 0x7fe568145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5680d23c0 0x7fe568148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 --2- 172.21.15.59:0/4221838731 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5680b3ed0 0x7fe56815d490 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.783+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 >> 172.21.15.59:0/4221838731 conn(0x7fe5680d3ee0 msgr2=0x7fe568157500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.784+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 shutdown_connections 2024-06-29T15:57:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.784+0000 7fe56d7bd640 1 -- 172.21.15.59:0/4221838731 wait complete. 2024-06-29T15:57:57.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:57:57.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.916+0000 7fec58054640 1 -- 172.21.15.59:0/3558664178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 msgr2=0x7fec3c004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.916+0000 7fec58054640 1 --2- 172.21.15.59:0/3558664178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 0x7fec3c004c70 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fec5008c4f0 tx=0x7fec3c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:57.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.916+0000 7fec565f1640 1 -- 172.21.15.59:0/3558664178 reap_dead start 2024-06-29T15:57:57.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.916+0000 7fec58054640 1 -- 172.21.15.59:0/3558664178 shutdown_connections 2024-06-29T15:57:57.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.916+0000 7fec58054640 1 -- 172.21.15.59:0/3558664178 >> 172.21.15.59:0/3558664178 conn(0x7fec500d3ee0 msgr2=0x7fec500d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.916+0000 7fec58054640 1 -- 172.21.15.59:0/3558664178 shutdown_connections 2024-06-29T15:57:57.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.916+0000 7fec58054640 1 -- 172.21.15.59:0/3558664178 wait complete. 2024-06-29T15:57:57.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.918+0000 7fec58054640 1 Processor -- start 2024-06-29T15:57:57.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.918+0000 7fec58054640 1 -- start start 2024-06-29T15:57:57.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec58054640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 0x7fec50153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec58054640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec500cab70 0x7fec500caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec58054640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec500cb4c0 0x7fec500cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec58054640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fec5015e8d0 con 0x7fec3c004800 2024-06-29T15:57:57.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec58054640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fec500b42b0 con 0x7fec500cab70 2024-06-29T15:57:57.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec58054640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fec5015ea50 con 0x7fec500cb4c0 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec55df0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 0x7fec50153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec55df0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 0x7fec50153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46474/0 (socket says 172.21.15.59:46474) 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec55df0640 1 -- 172.21.15.59:0/4293248174 learned_addr learned my addr 172.21.15.59:0/4293248174 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec555ef640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec500cab70 0x7fec500caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.919+0000 7fec565f1640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec500cb4c0 0x7fec500cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec55df0640 1 -- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec500cb4c0 msgr2=0x7fec500cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec55df0640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec500cb4c0 0x7fec500cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec55df0640 1 -- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec500cab70 msgr2=0x7fec500caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec55df0640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec500cab70 0x7fec500caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec55df0640 1 -- 172.21.15.59:0/4293248174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec500cda60 con 0x7fec3c004800 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec55df0640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 0x7fec50153900 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fec4c006600 tx=0x7fec4c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec427fc640 1 -- 172.21.15.59:0/4293248174 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c015070 con 0x7fec3c004800 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec427fc640 1 -- 172.21.15.59:0/4293248174 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec4c00f960 con 0x7fec3c004800 2024-06-29T15:57:57.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec500cbc20 con 0x7fec3c004800 2024-06-29T15:57:57.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec427fc640 1 -- 172.21.15.59:0/4293248174 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c00fd70 con 0x7fec3c004800 2024-06-29T15:57:57.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.920+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec500cc1e0 con 0x7fec3c004800 2024-06-29T15:57:57.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.921+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec500b3990 con 0x7fec3c004800 2024-06-29T15:57:57.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.922+0000 7fec427fc640 1 -- 172.21.15.59:0/4293248174 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7fec4c011430 con 0x7fec3c004800 2024-06-29T15:57:57.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.922+0000 7fec427fc640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fec340765a0 0x7fec34078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.922+0000 7fec555ef640 1 -- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fec340765a0 msgr2=0x7fec34078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:57.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.922+0000 7fec427fc640 1 -- 172.21.15.59:0/4293248174 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fec4c099060 con 0x7fec3c004800 2024-06-29T15:57:57.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.922+0000 7fec555ef640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fec340765a0 0x7fec34078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:57.931+0000 7fec427fc640 1 -- 172.21.15.59:0/4293248174 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fec4c061850 con 0x7fec3c004800 2024-06-29T15:57:58.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.123+0000 7fec555ef640 1 -- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fec340765a0 msgr2=0x7fec34078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:58.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.123+0000 7fec555ef640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fec340765a0 0x7fec34078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:58.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.135+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fec500d2950 con 0x7fec3c004800 2024-06-29T15:57:58.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.135+0000 7fec427fc640 1 -- 172.21.15.59:0/4293248174 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v227) ==== 80+0+476 (secure 0 0 0) 0x7fec4c065b50 con 0x7fec3c004800 2024-06-29T15:57:58.136 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2050849752 2024-06-30T15:57:50.460403+0000 2024-06-29T15:57:58.137 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/239127507 2024-06-30T15:57:50.460403+0000 2024-06-29T15:57:58.137 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2051017624 2024-06-30T15:57:50.460403+0000 2024-06-29T15:57:58.137 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/4093149557 2024-06-30T15:57:50.460403+0000 2024-06-29T15:57:58.137 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/572137829 2024-06-30T15:57:50.460403+0000 2024-06-29T15:57:58.137 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2737762390 2024-06-30T15:57:50.460403+0000 2024-06-29T15:57:58.137 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/2780049427 2024-06-30T15:57:50.460403+0000 2024-06-29T15:57:58.137 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/2780049427 2024-06-30T15:57:50.460403+0000 2024-06-29T15:57:58.137 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fec340765a0 msgr2=0x7fec34078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fec340765a0 0x7fec34078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 msgr2=0x7fec50153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 0x7fec50153900 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fec4c006600 tx=0x7fec4c0039c0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 shutdown_connections 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fec340765a0 0x7fec34078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fec500cb4c0 0x7fec500cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec500cab70 0x7fec500caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 --2- 172.21.15.59:0/4293248174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec3c004800 0x7fec50153900 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 >> 172.21.15.59:0/4293248174 conn(0x7fec500d3ee0 msgr2=0x7fec50160b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 shutdown_connections 2024-06-29T15:57:58.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.138+0000 7fec58054640 1 -- 172.21.15.59:0/4293248174 wait complete. 2024-06-29T15:57:58.150 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:57:58.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:58.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:58.179 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T15:57:58.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:58.180 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:57:58.237 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:58.237 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:57:58.237 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:58.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:58.253 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:58.253 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:57:58.253 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:58.293 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:57:58.293 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:58.293 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:57:58.293 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:58.309 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:58.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 -- 172.21.15.59:0/1611932187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 msgr2=0x7f11c40b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 --2- 172.21.15.59:0/1611932187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 0x7f11c40b3990 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f11c40d04f0 tx=0x7f11c002a560 comp rx=0 tx=0).stop 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 -- 172.21.15.59:0/1611932187 shutdown_connections 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 --2- 172.21.15.59:0/1611932187 >> v2:172.21.15.153:3300/0 conn(0x7f11c415ee10 0x7f11c415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 --2- 172.21.15.59:0/1611932187 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c40b3ed0 0x7f11c415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 --2- 172.21.15.59:0/1611932187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 0x7f11c40b3990 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 -- 172.21.15.59:0/1611932187 >> 172.21.15.59:0/1611932187 conn(0x7f11c40d3ee0 msgr2=0x7f11c40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 -- 172.21.15.59:0/1611932187 shutdown_connections 2024-06-29T15:57:58.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.450+0000 7f11cbca5640 1 -- 172.21.15.59:0/1611932187 wait complete. 2024-06-29T15:57:58.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.452+0000 7f11cbca5640 1 Processor -- start 2024-06-29T15:57:58.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.452+0000 7f11cbca5640 1 -- start start 2024-06-29T15:57:58.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.453+0000 7f11cbca5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c40b3ed0 0x7f11c415cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.453+0000 7f11cbca5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 0x7f11c41f86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.453+0000 7f11cbca5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c415ee10 0x7f11c41f8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.453+0000 7f11cbca5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f11c4160070 con 0x7f11c40d23c0 2024-06-29T15:57:58.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.453+0000 7f11cbca5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11c415fef0 con 0x7f11c40b3ed0 2024-06-29T15:57:58.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.453+0000 7f11cbca5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11c41601f0 con 0x7f11c415ee10 2024-06-29T15:57:58.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 0x7f11c41f86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 0x7f11c41f86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46486/0 (socket says 172.21.15.59:46486) 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 -- 172.21.15.59:0/2679766642 learned_addr learned my addr 172.21.15.59:0/2679766642 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9a41640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c40b3ed0 0x7f11c415cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11ca242640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c415ee10 0x7f11c41f8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 -- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c415ee10 msgr2=0x7f11c41f8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c415ee10 0x7f11c41f8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 -- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c40b3ed0 msgr2=0x7f11c415cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c40b3ed0 0x7f11c415cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 -- 172.21.15.59:0/2679766642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c415da70 con 0x7f11c40d23c0 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11c9240640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 0x7f11c41f86e0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f11b400e740 tx=0x7f11b400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11ba7fc640 1 -- 172.21.15.59:0/2679766642 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b400ccc0 con 0x7f11c40d23c0 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11ba7fc640 1 -- 172.21.15.59:0/2679766642 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11b400ee50 con 0x7f11c40d23c0 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11ba7fc640 1 -- 172.21.15.59:0/2679766642 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b4010670 con 0x7f11c40d23c0 2024-06-29T15:57:58.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c415dd50 con 0x7f11c40d23c0 2024-06-29T15:57:58.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.454+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c415e310 con 0x7f11c40d23c0 2024-06-29T15:57:58.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.455+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c415d7e0 con 0x7f11c40d23c0 2024-06-29T15:57:58.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.456+0000 7f11ba7fc640 1 -- 172.21.15.59:0/2679766642 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f11b4010810 con 0x7f11c40d23c0 2024-06-29T15:57:58.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.457+0000 7f11ba7fc640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f11a80765a0 0x7f11a8078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.457+0000 7f11ba7fc640 1 -- 172.21.15.59:0/2679766642 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f11b4098b50 con 0x7f11c40d23c0 2024-06-29T15:57:58.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.457+0000 7f11c9a41640 1 -- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f11a80765a0 msgr2=0x7f11a8078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:58.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.457+0000 7f11c9a41640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f11a80765a0 0x7f11a8078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:58.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.466+0000 7f11ba7fc640 1 -- 172.21.15.59:0/2679766642 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11b4061380 con 0x7f11c40d23c0 2024-06-29T15:57:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.658+0000 7f11c9a41640 1 -- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f11a80765a0 msgr2=0x7f11a8078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:58.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.658+0000 7f11c9a41640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f11a80765a0 0x7f11a8078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:58.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.695+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11c40b3990 con 0x7f11c40d23c0 2024-06-29T15:57:58.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.695+0000 7f11ba7fc640 1 -- 172.21.15.59:0/2679766642 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6567 (secure 0 0 0) 0x7f11b4096e20 con 0x7f11c40d23c0 2024-06-29T15:57:58.695 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:58.696 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":121,"btime":"2024-06-29T15:57:38:315315+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.589050+0000","modified":"2024-06-29T15:57:24.623107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11487":{"gid":11487,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1734341893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1734341893},{"type":"v1","addr":"172.21.15.59:6834","nonce":1734341893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11487,"qdb_cluster":[11487]},"id":13},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:57:58.696 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 121 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f11a80765a0 msgr2=0x7f11a8078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f11a80765a0 0x7f11a8078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 msgr2=0x7f11c41f86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 0x7f11c41f86e0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f11b400e740 tx=0x7f11b400ec00 comp rx=0 tx=0).stop 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 shutdown_connections 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f11a80765a0 0x7f11a8078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11c415ee10 0x7f11c41f8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11c40d23c0 0x7f11c41f86e0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 --2- 172.21.15.59:0/2679766642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c40b3ed0 0x7f11c415cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 >> 172.21.15.59:0/2679766642 conn(0x7f11c40d3ee0 msgr2=0x7f11c406d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 shutdown_connections 2024-06-29T15:57:58.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.698+0000 7f11cbca5640 1 -- 172.21.15.59:0/2679766642 wait complete. 2024-06-29T15:57:58.709 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:58.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.827+0000 7fe1a919e640 1 -- 172.21.15.59:0/3168810105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a4143840 msgr2=0x7fe1a406e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.827+0000 7fe1a919e640 1 --2- 172.21.15.59:0/3168810105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a4143840 0x7fe1a406e4d0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fe18c005fd0 tx=0x7fe18c02a540 comp rx=0 tx=0).stop 2024-06-29T15:57:58.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.827+0000 7fe1a919e640 1 -- 172.21.15.59:0/3168810105 shutdown_connections 2024-06-29T15:57:58.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.827+0000 7fe1a919e640 1 --2- 172.21.15.59:0/3168810105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a406ea10 0x7fe1a40cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.827+0000 7fe1a919e640 1 --2- 172.21.15.59:0/3168810105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a4143840 0x7fe1a406e4d0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.828+0000 7fe1a919e640 1 -- 172.21.15.59:0/3168810105 >> 172.21.15.59:0/3168810105 conn(0x7fe1a40b2ab0 msgr2=0x7fe1a40b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.828+0000 7fe1a919e640 1 -- 172.21.15.59:0/3168810105 shutdown_connections 2024-06-29T15:57:58.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.828+0000 7fe1a919e640 1 -- 172.21.15.59:0/3168810105 wait complete. 2024-06-29T15:57:58.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.831+0000 7fe1a919e640 1 Processor -- start 2024-06-29T15:57:58.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.831+0000 7fe1a919e640 1 -- start start 2024-06-29T15:57:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe1a919e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a406ea10 0x7fe1a415ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe1a919e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a40d0130 0x7fe1a415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe1a919e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a4143840 0x7fe1a41f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe1a919e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a40cf650 con 0x7fe1a40d0130 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe1a919e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a40cf4d0 con 0x7fe1a406ea10 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe1a919e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe1a40cf7d0 con 0x7fe1a4143840 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe19bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a40d0130 0x7fe1a415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe19bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a40d0130 0x7fe1a415ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46506/0 (socket says 172.21.15.59:46506) 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe19bfff640 1 -- 172.21.15.59:0/62969036 learned_addr learned my addr 172.21.15.59:0/62969036 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe1a359e640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a4143840 0x7fe1a41f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.833+0000 7fe1a2d9d640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a406ea10 0x7fe1a415ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19bfff640 1 -- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a4143840 msgr2=0x7fe1a41f4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19bfff640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a4143840 0x7fe1a41f4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19bfff640 1 -- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a406ea10 msgr2=0x7fe1a415ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19bfff640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a406ea10 0x7fe1a415ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19bfff640 1 -- 172.21.15.59:0/62969036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a415d2e0 con 0x7fe1a40d0130 2024-06-29T15:57:58.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe1a2d9d640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a406ea10 0x7fe1a415ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:58.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe1a359e640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a4143840 0x7fe1a41f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:58.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19bfff640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a40d0130 0x7fe1a415ecd0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fe18c034040 tx=0x7fe18c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:58.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19b7fe640 1 -- 172.21.15.59:0/62969036 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c03e030 con 0x7fe1a40d0130 2024-06-29T15:57:58.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19b7fe640 1 -- 172.21.15.59:0/62969036 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe18c003d90 con 0x7fe1a40d0130 2024-06-29T15:57:58.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a415d5c0 con 0x7fe1a40d0130 2024-06-29T15:57:58.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe19b7fe640 1 -- 172.21.15.59:0/62969036 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c035040 con 0x7fe1a40d0130 2024-06-29T15:57:58.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.834+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1a415b500 con 0x7fe1a40d0130 2024-06-29T15:57:58.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.835+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a40cfc50 con 0x7fe1a40d0130 2024-06-29T15:57:58.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.837+0000 7fe19b7fe640 1 -- 172.21.15.59:0/62969036 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7fe18c041030 con 0x7fe1a40d0130 2024-06-29T15:57:58.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.837+0000 7fe19b7fe640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe1840765d0 0x7fe184078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.838+0000 7fe1a2d9d640 1 -- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe1840765d0 msgr2=0x7fe184078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.838+0000 7fe19b7fe640 1 -- 172.21.15.59:0/62969036 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7fe18c0b8db0 con 0x7fe1a40d0130 2024-06-29T15:57:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.838+0000 7fe1a2d9d640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe1840765d0 0x7fe184078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:58.841+0000 7fe19b7fe640 1 -- 172.21.15.59:0/62969036 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe18c081770 con 0x7fe1a40d0130 2024-06-29T15:57:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.036+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1a406f7c0 con 0x7fe1a40d0130 2024-06-29T15:57:59.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.037+0000 7fe19b7fe640 1 -- 172.21.15.59:0/62969036 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) ==== 74+0+21283 (secure 0 0 0) 0x7fe18c085a70 con 0x7fe1a40d0130 2024-06-29T15:57:59.038 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:57:59.038 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":227,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:50.460468+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2050849752":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/239127507":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2051017624":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/4093149557":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/572137829":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2737762390":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6833/2780049427":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2780049427":"2024-06-30T15:57:50.460403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.039+0000 7fe1a2d9d640 1 -- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe1840765d0 msgr2=0x7fe184078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.039+0000 7fe1a2d9d640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe1840765d0 0x7fe184078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe1840765d0 msgr2=0x7fe184078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe1840765d0 0x7fe184078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a40d0130 msgr2=0x7fe1a415ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a40d0130 0x7fe1a415ecd0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fe18c034040 tx=0x7fe18c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 shutdown_connections 2024-06-29T15:57:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7fe1840765d0 0x7fe184078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe1a4143840 0x7fe1a41f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1a40d0130 0x7fe1a415ecd0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 --2- 172.21.15.59:0/62969036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1a406ea10 0x7fe1a415ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 >> 172.21.15.59:0/62969036 conn(0x7fe1a40b2ab0 msgr2=0x7fe1a41412c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 shutdown_connections 2024-06-29T15:57:59.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.040+0000 7fe1a919e640 1 -- 172.21.15.59:0/62969036 wait complete. 2024-06-29T15:57:59.051 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:57:19.955891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '221', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:57:20.545869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:57:32.305142+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:57:33.647895+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:59.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:59.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.174+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2064318324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84003c80 msgr2=0x7f1a840040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.174+0000 7f1a9cc8f640 1 --2- 172.21.15.59:0/2064318324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84003c80 0x7f1a840040d0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f1a88001830 tx=0x7f1a84005830 comp rx=0 tx=0).stop 2024-06-29T15:57:59.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.174+0000 7f1a96d9d640 1 -- 172.21.15.59:0/2064318324 reap_dead start 2024-06-29T15:57:59.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.174+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2064318324 shutdown_connections 2024-06-29T15:57:59.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.174+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2064318324 >> 172.21.15.59:0/2064318324 conn(0x7f1a98072d80 msgr2=0x7f1a98073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.174+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2064318324 shutdown_connections 2024-06-29T15:57:59.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.174+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2064318324 wait complete. 2024-06-29T15:57:59.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.176+0000 7f1a9cc8f640 1 Processor -- start 2024-06-29T15:57:59.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.176+0000 7f1a9cc8f640 1 -- start start 2024-06-29T15:57:59.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.177+0000 7f1a9cc8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a84003c80 0x7f1a98135720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.177+0000 7f1a9cc8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a98159600 0x7f1a98132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.177+0000 7f1a9cc8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a98133030 0x7f1a98133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a9cc8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9815c130 con 0x7f1a98133030 2024-06-29T15:57:59.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a9cc8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9815bfb0 con 0x7f1a84003c80 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a9cc8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9815b660 con 0x7f1a98159600 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a8ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a98159600 0x7f1a98132af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a8ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a98159600 0x7f1a98132af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:59532/0 (socket says 172.21.15.59:59532) 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a8ffff640 1 -- 172.21.15.59:0/2211679974 learned_addr learned my addr 172.21.15.59:0/2211679974 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a96d9d640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a98133030 0x7f1a98133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a9659c640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a84003c80 0x7f1a98135720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a96d9d640 1 -- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a98159600 msgr2=0x7f1a98132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a96d9d640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a98159600 0x7f1a98132af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a96d9d640 1 -- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a84003c80 msgr2=0x7f1a98135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a96d9d640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a84003c80 0x7f1a98135720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a96d9d640 1 -- 172.21.15.59:0/2211679974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9815a0c0 con 0x7f1a98133030 2024-06-29T15:57:59.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a8ffff640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a98159600 0x7f1a98132af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:59.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.178+0000 7f1a96d9d640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a98133030 0x7f1a98133440 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f1a980b4df0 tx=0x7f1a90006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.179+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80009720 con 0x7f1a98133030 2024-06-29T15:57:59.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.179+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a90018070 con 0x7f1a98133030 2024-06-29T15:57:59.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.179+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9000fd60 con 0x7f1a98133030 2024-06-29T15:57:59.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.179+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a9815a3a0 con 0x7f1a98133030 2024-06-29T15:57:59.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.179+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a9815a960 con 0x7f1a98133030 2024-06-29T15:57:59.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.180+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a98159e30 con 0x7f1a98133030 2024-06-29T15:57:59.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.181+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f1a9001d460 con 0x7f1a98133030 2024-06-29T15:57:59.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.182+0000 7f1a8effd640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f1a780765d0 0x7f1a78078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.182+0000 7f1a9659c640 1 -- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f1a780765d0 msgr2=0x7f1a78078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2014343395 2024-06-29T15:57:59.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.182+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 7365+0+0 (secure 0 0 0) 0x7f1a900987c0 con 0x7f1a98133030 2024-06-29T15:57:59.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.182+0000 7f1a9659c640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f1a780765d0 0x7f1a78078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:59.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.189+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 135) ==== 97912+0+0 (secure 0 0 0) 0x7f1a9005d280 con 0x7f1a98133030 2024-06-29T15:57:59.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.190+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f1a780765d0 msgr2=0x7f1a78078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.190+0000 7f1a8effd640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f1a780765d0 0x7f1a78078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.195+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a9006ab20 con 0x7f1a98133030 2024-06-29T15:57:59.250 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.250 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.250 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.250 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.251 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.251 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.251 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.251 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.251 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.251 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.249+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.261 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.261 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.261 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.262 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.262 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251ef31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.262 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.262 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.262 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.262 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.262 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:57:59.261+0000 7f251bf2b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:59.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.396+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1a98140760 con 0x7f1a98133030 2024-06-29T15:57:59.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.397+0000 7f1a8effd640 1 -- 172.21.15.59:0/2211679974 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v228) ==== 93+0+10 (secure 0 0 0) 0x7f1a9006a420 con 0x7f1a98133030 2024-06-29T15:57:59.397 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:57:59.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a98133030 msgr2=0x7f1a98133440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a98133030 0x7f1a98133440 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f1a980b4df0 tx=0x7f1a90006660 comp rx=0 tx=0).stop 2024-06-29T15:57:59.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 shutdown_connections 2024-06-29T15:57:59.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:6832/2014343395,v1:172.21.15.153:6833/2014343395] conn(0x7f1a780765d0 0x7f1a78078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a98133030 0x7f1a98133440 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1a98159600 0x7f1a98132af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 --2- 172.21.15.59:0/2211679974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a84003c80 0x7f1a98135720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 >> 172.21.15.59:0/2211679974 conn(0x7f1a98072d80 msgr2=0x7f1a9815c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 shutdown_connections 2024-06-29T15:57:59.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.399+0000 7f1a9cc8f640 1 -- 172.21.15.59:0/2211679974 wait complete. 2024-06-29T15:57:59.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:59.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.534+0000 7f2316489640 1 -- 172.21.15.59:0/3698641691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 msgr2=0x7f231015a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.534+0000 7f2316489640 1 --2- 172.21.15.59:0/3698641691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 0x7f231015a400 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f23100cdfd0 tx=0x7f230002a5f0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.534+0000 7f2316489640 1 -- 172.21.15.59:0/3698641691 shutdown_connections 2024-06-29T15:57:59.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.534+0000 7f2316489640 1 --2- 172.21.15.59:0/3698641691 >> v2:172.21.15.153:3300/0 conn(0x7f231006c1e0 0x7f231006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.534+0000 7f2316489640 1 --2- 172.21.15.59:0/3698641691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 0x7f231015a400 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.534+0000 7f2316489640 1 -- 172.21.15.59:0/3698641691 >> 172.21.15.59:0/3698641691 conn(0x7f23100b24c0 msgr2=0x7f23100cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.534+0000 7f2316489640 1 -- 172.21.15.59:0/3698641691 shutdown_connections 2024-06-29T15:57:59.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.534+0000 7f2316489640 1 -- 172.21.15.59:0/3698641691 wait complete. 2024-06-29T15:57:59.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.536+0000 7f2316489640 1 Processor -- start 2024-06-29T15:57:59.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.536+0000 7f2316489640 1 -- start start 2024-06-29T15:57:59.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2316489640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f231006c1e0 0x7f2310143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2316489640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 0x7f2310148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2316489640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23100cf120 0x7f2310148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2316489640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f23100cce40 con 0x7f2310073f30 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2316489640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23100cccc0 con 0x7f23100cf120 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2316489640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f23100ccfc0 con 0x7f231006c1e0 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2307fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 0x7f2310148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2307fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 0x7f2310148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46548/0 (socket says 172.21.15.59:46548) 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2307fff640 1 -- 172.21.15.59:0/216524092 learned_addr learned my addr 172.21.15.59:0/216524092 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f2314a26640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23100cf120 0x7f2310148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.537+0000 7f230ffff640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f231006c1e0 0x7f2310143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f2307fff640 1 -- 172.21.15.59:0/216524092 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f231006c1e0 msgr2=0x7f2310143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f2307fff640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f231006c1e0 0x7f2310143540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f2307fff640 1 -- 172.21.15.59:0/216524092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23100cf120 msgr2=0x7f2310148b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f2307fff640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23100cf120 0x7f2310148b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f2307fff640 1 -- 172.21.15.59:0/216524092 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2310154be0 con 0x7f2310073f30 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f2307fff640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 0x7f2310148610 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c3d0 tx=0x7f22f800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f230d7fa640 1 -- 172.21.15.59:0/216524092 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8017070 con 0x7f2310073f30 2024-06-29T15:57:59.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f230d7fa640 1 -- 172.21.15.59:0/216524092 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22f80053c0 con 0x7f2310073f30 2024-06-29T15:57:59.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f230d7fa640 1 -- 172.21.15.59:0/216524092 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8003850 con 0x7f2310073f30 2024-06-29T15:57:59.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2310154ec0 con 0x7f2310073f30 2024-06-29T15:57:59.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.538+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2310155480 con 0x7f2310073f30 2024-06-29T15:57:59.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.539+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2310155860 con 0x7f2310073f30 2024-06-29T15:57:59.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.540+0000 7f230d7fa640 1 -- 172.21.15.59:0/216524092 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 135) ==== 97912+0+0 (secure 0 0 0) 0x7f22f801a070 con 0x7f2310073f30 2024-06-29T15:57:59.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.540+0000 7f230d7fa640 1 -- 172.21.15.59:0/216524092 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7f22f8096d90 con 0x7f2310073f30 2024-06-29T15:57:59.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.545+0000 7f230d7fa640 1 -- 172.21.15.59:0/216524092 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2310155860 con 0x7f2310073f30 2024-06-29T15:57:59.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.734+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f231006c5f0 con 0x7f2310073f30 2024-06-29T15:57:59.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.735+0000 7f230d7fa640 1 -- 172.21.15.59:0/216524092 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v228) ==== 104+0+11 (secure 0 0 0) 0x7f22f805fca0 con 0x7f2310073f30 2024-06-29T15:57:59.735 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.736+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 msgr2=0x7f2310148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.736+0000 7f2316489640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 0x7f2310148610 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c3d0 tx=0x7f22f800c890 comp rx=0 tx=0).stop 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.737+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 shutdown_connections 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.737+0000 7f2316489640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23100cf120 0x7f2310148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.737+0000 7f2316489640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2310073f30 0x7f2310148610 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.737+0000 7f2316489640 1 --2- 172.21.15.59:0/216524092 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f231006c1e0 0x7f2310143540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.737+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 >> 172.21.15.59:0/216524092 conn(0x7f23100b24c0 msgr2=0x7f23100ce710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.737+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 shutdown_connections 2024-06-29T15:57:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.737+0000 7f2316489640 1 -- 172.21.15.59:0/216524092 wait complete. 2024-06-29T15:57:59.748 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:59.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.869+0000 7fe861a47640 1 -- 172.21.15.59:0/3784294704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c15e290 msgr2=0x7fe85c0ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.869+0000 7fe861a47640 1 --2- 172.21.15.59:0/3784294704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c15e290 0x7fe85c0ce110 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fe84c005fd0 tx=0x7fe84c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.869+0000 7fe861a47640 1 -- 172.21.15.59:0/3784294704 shutdown_connections 2024-06-29T15:57:59.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.869+0000 7fe861a47640 1 --2- 172.21.15.59:0/3784294704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe85c0b40a0 0x7fe85c0b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.869+0000 7fe861a47640 1 --2- 172.21.15.59:0/3784294704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c15e290 0x7fe85c0ce110 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.869+0000 7fe861a47640 1 -- 172.21.15.59:0/3784294704 >> 172.21.15.59:0/3784294704 conn(0x7fe85c0d3ee0 msgr2=0x7fe85c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.869+0000 7fe861a47640 1 -- 172.21.15.59:0/3784294704 shutdown_connections 2024-06-29T15:57:59.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.869+0000 7fe861a47640 1 -- 172.21.15.59:0/3784294704 wait complete. 2024-06-29T15:57:59.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.870+0000 7fe861a47640 1 Processor -- start 2024-06-29T15:57:59.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.871+0000 7fe861a47640 1 -- start start 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe861a47640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe85c0ce650 0x7fe85c150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe861a47640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe85c15e290 0x7fe85c158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe861a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c155520 0x7fe85c1f8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe861a47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c0b7cc0 con 0x7fe85c155520 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe861a47640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c0b7b40 con 0x7fe85c0ce650 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe861a47640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe85c0b7e40 con 0x7fe85c15e290 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c155520 0x7fe85c1f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c155520 0x7fe85c1f8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46562/0 (socket says 172.21.15.59:46562) 2024-06-29T15:57:59.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 -- 172.21.15.59:0/15446300 learned_addr learned my addr 172.21.15.59:0/15446300 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe853fff640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe85c15e290 0x7fe85c158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85affd640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe85c0ce650 0x7fe85c150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 -- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe85c15e290 msgr2=0x7fe85c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe85c15e290 0x7fe85c158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 -- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe85c0ce650 msgr2=0x7fe85c150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe85c0ce650 0x7fe85c150fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 -- 172.21.15.59:0/15446300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c153d80 con 0x7fe85c155520 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe853fff640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe85c15e290 0x7fe85c158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.872+0000 7fe85b7fe640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c155520 0x7fe85c1f8960 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fe85400da10 tx=0x7fe85400ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.873+0000 7fe858ff9640 1 -- 172.21.15.59:0/15446300 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854017070 con 0x7fe85c155520 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.873+0000 7fe858ff9640 1 -- 172.21.15.59:0/15446300 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe854003970 con 0x7fe85c155520 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.873+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85c155e80 con 0x7fe85c155520 2024-06-29T15:57:59.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.873+0000 7fe858ff9640 1 -- 172.21.15.59:0/15446300 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85400be60 con 0x7fe85c155520 2024-06-29T15:57:59.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.873+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe85c1f8440 con 0x7fe85c155520 2024-06-29T15:57:59.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.874+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe85c0d0190 con 0x7fe85c155520 2024-06-29T15:57:59.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.875+0000 7fe858ff9640 1 -- 172.21.15.59:0/15446300 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 135) ==== 97912+0+0 (secure 0 0 0) 0x7fe854010460 con 0x7fe85c155520 2024-06-29T15:57:59.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.875+0000 7fe858ff9640 1 -- 172.21.15.59:0/15446300 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7fe854097dc0 con 0x7fe85c155520 2024-06-29T15:57:59.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:57:59.880+0000 7fe858ff9640 1 -- 172.21.15.59:0/15446300 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe85c0d0190 con 0x7fe85c155520 2024-06-29T15:58:00.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.068+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe85c06c5f0 con 0x7fe85c155520 2024-06-29T15:58:00.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.069+0000 7fe858ff9640 1 -- 172.21.15.59:0/15446300 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v228) ==== 100+0+11 (secure 0 0 0) 0x7fe854061320 con 0x7fe85c155520 2024-06-29T15:58:00.069 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c155520 msgr2=0x7fe85c1f8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c155520 0x7fe85c1f8960 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fe85400da10 tx=0x7fe85400ded0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 shutdown_connections 2024-06-29T15:58:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe85c155520 0x7fe85c1f8960 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe85c15e290 0x7fe85c158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 --2- 172.21.15.59:0/15446300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe85c0ce650 0x7fe85c150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 >> 172.21.15.59:0/15446300 conn(0x7fe85c0d3ee0 msgr2=0x7fe85c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 shutdown_connections 2024-06-29T15:58:00.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.071+0000 7fe861a47640 1 -- 172.21.15.59:0/15446300 wait complete. 2024-06-29T15:58:00.082 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:00.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.205+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/2001731677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac073f30 msgr2=0x7fd9ac15a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.205+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/2001731677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac073f30 0x7fd9ac15a400 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd99c005fd0 tx=0x7fd99c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.205+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/2001731677 shutdown_connections 2024-06-29T15:58:00.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.205+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/2001731677 >> v2:172.21.15.153:3300/0 conn(0x7fd9ac06c1e0 0x7fd9ac06c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.205+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/2001731677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac073f30 0x7fd9ac15a400 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.205+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/2001731677 >> 172.21.15.59:0/2001731677 conn(0x7fd9ac0b24c0 msgr2=0x7fd9ac0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.205+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/2001731677 shutdown_connections 2024-06-29T15:58:00.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.205+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/2001731677 wait complete. 2024-06-29T15:58:00.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.207+0000 7fd9b3aa5640 1 Processor -- start 2024-06-29T15:58:00.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.207+0000 7fd9b3aa5640 1 -- start start 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.208+0000 7fd9b3aa5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac06c1e0 0x7fd9ac13aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.208+0000 7fd9b3aa5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac073f30 0x7fd9ac1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b3aa5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0cf120 0x7fd9ac150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b3aa5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac0cce40 con 0x7fd9ac073f30 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b3aa5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac0cccc0 con 0x7fd9ac0cf120 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b3aa5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd9ac0ccfc0 con 0x7fd9ac06c1e0 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac073f30 0x7fd9ac1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac073f30 0x7fd9ac1f4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46566/0 (socket says 172.21.15.59:46566) 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1841640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac06c1e0 0x7fd9ac13aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 -- 172.21.15.59:0/363170995 learned_addr learned my addr 172.21.15.59:0/363170995 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b2042640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0cf120 0x7fd9ac150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 -- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac06c1e0 msgr2=0x7fd9ac13aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac06c1e0 0x7fd9ac13aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 -- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0cf120 msgr2=0x7fd9ac150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0cf120 0x7fd9ac150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 -- 172.21.15.59:0/363170995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac152f10 con 0x7fd9ac073f30 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b1040640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac073f30 0x7fd9ac1f4420 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000c370 tx=0x7fd9a000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9aa7fc640 1 -- 172.21.15.59:0/363170995 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0006890 con 0x7fd9ac073f30 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ac13b950 con 0x7fd9ac073f30 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9aa7fc640 1 -- 172.21.15.59:0/363170995 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9a0006b20 con 0x7fd9ac073f30 2024-06-29T15:58:00.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9aa7fc640 1 -- 172.21.15.59:0/363170995 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0007020 con 0x7fd9ac073f30 2024-06-29T15:58:00.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.209+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ac1531d0 con 0x7fd9ac073f30 2024-06-29T15:58:00.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.210+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9ac13b6c0 con 0x7fd9ac073f30 2024-06-29T15:58:00.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.211+0000 7fd9aa7fc640 1 -- 172.21.15.59:0/363170995 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7fd9a0019020 con 0x7fd9ac073f30 2024-06-29T15:58:00.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.212+0000 7fd9aa7fc640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd998076120 0x7fd998078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.212+0000 7fd9aa7fc640 1 -- 172.21.15.59:0/363170995 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7fd9a00972f0 con 0x7fd9ac073f30 2024-06-29T15:58:00.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.212+0000 7fd9b1841640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd998076120 0x7fd998078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.213+0000 7fd9b1841640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd998076120 0x7fd998078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd99c034000 tx=0x7fd99c004ab0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.216+0000 7fd9aa7fc640 1 -- 172.21.15.59:0/363170995 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd9a005ffe0 con 0x7fd9ac073f30 2024-06-29T15:58:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.407+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7fd9ac06c5f0 con 0x7fd9ac073f30 2024-06-29T15:58:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.408+0000 7fd9aa7fc640 1 -- 172.21.15.59:0/363170995 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v228) ==== 107+0+11 (secure 0 0 0) 0x7fd9a00642e0 con 0x7fd9ac073f30 2024-06-29T15:58:00.408 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:58:00.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd998076120 msgr2=0x7fd998078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd998076120 0x7fd998078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd99c034000 tx=0x7fd99c004ab0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac073f30 msgr2=0x7fd9ac1f4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac073f30 0x7fd9ac1f4420 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000c370 tx=0x7fd9a000c830 comp rx=0 tx=0).stop 2024-06-29T15:58:00.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 shutdown_connections 2024-06-29T15:58:00.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd998076120 0x7fd998078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0cf120 0x7fd9ac150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac073f30 0x7fd9ac1f4420 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 --2- 172.21.15.59:0/363170995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac06c1e0 0x7fd9ac13aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 >> 172.21.15.59:0/363170995 conn(0x7fd9ac0b24c0 msgr2=0x7fd9ac15e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 shutdown_connections 2024-06-29T15:58:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.410+0000 7fd9b3aa5640 1 -- 172.21.15.59:0/363170995 wait complete. 2024-06-29T15:58:00.421 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:00.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.543+0000 7f7f117a1640 1 -- 172.21.15.59:0/3047210286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef4004800 msgr2=0x7f7ef4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.543+0000 7f7f117a1640 1 --2- 172.21.15.59:0/3047210286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef4004800 0x7f7ef4004c10 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f7f00001690 tx=0x7f7ef4005830 comp rx=0 tx=0).stop 2024-06-29T15:58:00.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.543+0000 7f7f117a1640 1 -- 172.21.15.59:0/3047210286 shutdown_connections 2024-06-29T15:58:00.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.543+0000 7f7f117a1640 1 --2- 172.21.15.59:0/3047210286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef4004800 0x7f7ef4004c10 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.543+0000 7f7f117a1640 1 -- 172.21.15.59:0/3047210286 >> 172.21.15.59:0/3047210286 conn(0x7f7f0c0b5610 msgr2=0x7f7f0c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.543+0000 7f7f117a1640 1 -- 172.21.15.59:0/3047210286 shutdown_connections 2024-06-29T15:58:00.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.543+0000 7f7f117a1640 1 -- 172.21.15.59:0/3047210286 wait complete. 2024-06-29T15:58:00.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.545+0000 7f7f117a1640 1 Processor -- start 2024-06-29T15:58:00.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.545+0000 7f7f117a1640 1 -- start start 2024-06-29T15:58:00.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f117a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ef4004800 0x7f7f0c152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f117a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f0c1f6010 0x7f7f0c15edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f117a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0c152b80 0x7f7f0c15f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f117a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c0d1a20 con 0x7f7f0c1f6010 2024-06-29T15:58:00.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f117a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c0cf3a0 con 0x7f7f0c152b80 2024-06-29T15:58:00.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f117a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0c0d1ba0 con 0x7f7ef4004800 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f0c1f6010 0x7f7f0c15edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f0c1f6010 0x7f7f0c15edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46590/0 (socket says 172.21.15.59:46590) 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 -- 172.21.15.59:0/1465085043 learned_addr learned my addr 172.21.15.59:0/1465085043 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0b7fe640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0c152b80 0x7f7f0c15f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0affd640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ef4004800 0x7f7f0c152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 -- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ef4004800 msgr2=0x7f7f0c152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ef4004800 0x7f7f0c152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 -- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0c152b80 msgr2=0x7f7f0c15f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0c152b80 0x7f7f0c15f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 -- 172.21.15.59:0/1465085043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c1f5250 con 0x7f7f0c1f6010 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0b7fe640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0c152b80 0x7f7f0c15f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0affd640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ef4004800 0x7f7f0c152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7f0a7fc640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f0c1f6010 0x7f7f0c15edc0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f7f0002a8e0 tx=0x7f7f000063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.547+0000 7f7ef37fe640 1 -- 172.21.15.59:0/1465085043 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f00003a00 con 0x7f7f0c1f6010 2024-06-29T15:58:00.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.548+0000 7f7ef37fe640 1 -- 172.21.15.59:0/1465085043 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f00039070 con 0x7f7f0c1f6010 2024-06-29T15:58:00.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.548+0000 7f7ef37fe640 1 -- 172.21.15.59:0/1465085043 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f00033420 con 0x7f7f0c1f6010 2024-06-29T15:58:00.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.548+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0c1f5530 con 0x7f7f0c1f6010 2024-06-29T15:58:00.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.548+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0c1f5af0 con 0x7f7f0c1f6010 2024-06-29T15:58:00.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.549+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0c0d2100 con 0x7f7f0c1f6010 2024-06-29T15:58:00.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.549+0000 7f7ef37fe640 1 -- 172.21.15.59:0/1465085043 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7f7f00044020 con 0x7f7f0c1f6010 2024-06-29T15:58:00.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.550+0000 7f7ef37fe640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7ef8076120 0x7f7ef8078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.550+0000 7f7ef37fe640 1 -- 172.21.15.59:0/1465085043 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7f7f000b7540 con 0x7f7f0c1f6010 2024-06-29T15:58:00.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.550+0000 7f7f0affd640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7ef8076120 0x7f7ef8078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.550+0000 7f7f0affd640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7ef8076120 0x7f7ef8078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0d0500 tx=0x7f7efc005f90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.554+0000 7f7ef37fe640 1 -- 172.21.15.59:0/1465085043 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7f00080910 con 0x7f7f0c1f6010 2024-06-29T15:58:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.745+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f7f0c15bd30 con 0x7f7f0c1f6010 2024-06-29T15:58:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.745+0000 7f7ef37fe640 1 -- 172.21.15.59:0/1465085043 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v228) ==== 103+0+11 (secure 0 0 0) 0x7f7f00080210 con 0x7f7f0c1f6010 2024-06-29T15:58:00.746 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.747+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7ef8076120 msgr2=0x7f7ef8078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.747+0000 7f7f117a1640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7ef8076120 0x7f7ef8078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0d0500 tx=0x7f7efc005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.747+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f0c1f6010 msgr2=0x7f7f0c15edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.747+0000 7f7f117a1640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f0c1f6010 0x7f7f0c15edc0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f7f0002a8e0 tx=0x7f7f000063a0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.748+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 shutdown_connections 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.748+0000 7f7f117a1640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7ef8076120 0x7f7ef8078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.748+0000 7f7f117a1640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0c152b80 0x7f7f0c15f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.748+0000 7f7f117a1640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f0c1f6010 0x7f7f0c15edc0 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.748+0000 7f7f117a1640 1 --2- 172.21.15.59:0/1465085043 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7ef4004800 0x7f7f0c152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.748+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 >> 172.21.15.59:0/1465085043 conn(0x7f7f0c0b5610 msgr2=0x7f7f0c0b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.748+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 shutdown_connections 2024-06-29T15:58:00.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.748+0000 7f7f117a1640 1 -- 172.21.15.59:0/1465085043 wait complete. 2024-06-29T15:58:00.759 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:00.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.876+0000 7fba2b768640 1 -- 172.21.15.59:0/3375623594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 msgr2=0x7fba240b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.876+0000 7fba2b768640 1 --2- 172.21.15.59:0/3375623594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 0x7fba240b3990 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fba20009830 tx=0x7fba2002a510 comp rx=0 tx=0).stop 2024-06-29T15:58:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.876+0000 7fba2b768640 1 -- 172.21.15.59:0/3375623594 shutdown_connections 2024-06-29T15:58:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.876+0000 7fba2b768640 1 --2- 172.21.15.59:0/3375623594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 0x7fba240b3990 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.876+0000 7fba2b768640 1 -- 172.21.15.59:0/3375623594 >> 172.21.15.59:0/3375623594 conn(0x7fba240d3ee0 msgr2=0x7fba240d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.877+0000 7fba2b768640 1 -- 172.21.15.59:0/3375623594 shutdown_connections 2024-06-29T15:58:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.877+0000 7fba2b768640 1 -- 172.21.15.59:0/3375623594 wait complete. 2024-06-29T15:58:00.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.878+0000 7fba2b768640 1 Processor -- start 2024-06-29T15:58:00.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.878+0000 7fba2b768640 1 -- start start 2024-06-29T15:58:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba2b768640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba240b3ed0 0x7fba24152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba2b768640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 0x7fba241534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba2b768640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba2415efe0 0x7fba24153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba2b768640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fba24160450 con 0x7fba240d23c0 2024-06-29T15:58:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba2b768640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba241602d0 con 0x7fba240b3ed0 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba2b768640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fba241605d0 con 0x7fba2415efe0 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 0x7fba241534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 0x7fba241534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46600/0 (socket says 172.21.15.59:46600) 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 -- 172.21.15.59:0/41760402 learned_addr learned my addr 172.21.15.59:0/41760402 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba29504640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba240b3ed0 0x7fba24152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba29d05640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba2415efe0 0x7fba24153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 -- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba2415efe0 msgr2=0x7fba24153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba2415efe0 0x7fba24153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 -- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba240b3ed0 msgr2=0x7fba24152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba240b3ed0 0x7fba24152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 -- 172.21.15.59:0/41760402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba24154000 con 0x7fba240d23c0 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba29d05640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba2415efe0 0x7fba24153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba28d03640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 0x7fba241534c0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fba1400b3c0 tx=0x7fba1400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.880+0000 7fba19ffb640 1 -- 172.21.15.59:0/41760402 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14011830 con 0x7fba240d23c0 2024-06-29T15:58:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.881+0000 7fba19ffb640 1 -- 172.21.15.59:0/41760402 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba14016070 con 0x7fba240d23c0 2024-06-29T15:58:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.881+0000 7fba19ffb640 1 -- 172.21.15.59:0/41760402 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14011d60 con 0x7fba240d23c0 2024-06-29T15:58:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.881+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba24157df0 con 0x7fba240d23c0 2024-06-29T15:58:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.881+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba24157b10 con 0x7fba240d23c0 2024-06-29T15:58:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.882+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba24157880 con 0x7fba240d23c0 2024-06-29T15:58:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.882+0000 7fba19ffb640 1 -- 172.21.15.59:0/41760402 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7fba1401b460 con 0x7fba240d23c0 2024-06-29T15:58:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.883+0000 7fba19ffb640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fba08076120 0x7fba08078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.883+0000 7fba19ffb640 1 -- 172.21.15.59:0/41760402 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7fba140977e0 con 0x7fba240d23c0 2024-06-29T15:58:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.883+0000 7fba29504640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fba08076120 0x7fba08078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.884+0000 7fba29504640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fba08076120 0x7fba08078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fba20034000 tx=0x7fba20004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:00.887+0000 7fba19ffb640 1 -- 172.21.15.59:0/41760402 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fba14020020 con 0x7fba240d23c0 2024-06-29T15:58:01.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.116+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba240b3990 con 0x7fba240d23c0 2024-06-29T15:58:01.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.117+0000 7fba19ffb640 1 -- 172.21.15.59:0/41760402 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6567 (secure 0 0 0) 0x7fba24154000 con 0x7fba240d23c0 2024-06-29T15:58:01.117 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:01.117 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":121,"btime":"2024-06-29T15:57:38:315315+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.589050+0000","modified":"2024-06-29T15:57:24.623107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11487":{"gid":11487,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1734341893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1734341893},{"type":"v1","addr":"172.21.15.59:6834","nonce":1734341893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11487,"qdb_cluster":[11487]},"id":13},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:58:01.117 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 121 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fba08076120 msgr2=0x7fba08078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fba08076120 0x7fba08078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fba20034000 tx=0x7fba20004730 comp rx=0 tx=0).stop 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 msgr2=0x7fba241534c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 0x7fba241534c0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fba1400b3c0 tx=0x7fba1400b880 comp rx=0 tx=0).stop 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 shutdown_connections 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fba08076120 0x7fba08078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba2415efe0 0x7fba24153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba240d23c0 0x7fba241534c0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 --2- 172.21.15.59:0/41760402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba240b3ed0 0x7fba24152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 >> 172.21.15.59:0/41760402 conn(0x7fba240d3ee0 msgr2=0x7fba2406d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 shutdown_connections 2024-06-29T15:58:01.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.119+0000 7fba2b768640 1 -- 172.21.15.59:0/41760402 wait complete. 2024-06-29T15:58:01.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:01.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.251+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2800639886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c980d23c0 msgr2=0x7f6c980b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.251+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2800639886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c980d23c0 0x7f6c980b3990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009830 tx=0x7f6c9002a510 comp rx=0 tx=0).stop 2024-06-29T15:58:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.251+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2800639886 shutdown_connections 2024-06-29T15:58:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.251+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2800639886 >> v2:172.21.15.153:3300/0 conn(0x7f6c9815efe0 0x7f6c9815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.251+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2800639886 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c980d23c0 0x7f6c980b3990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.251+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2800639886 >> 172.21.15.59:0/2800639886 conn(0x7f6c980d3ee0 msgr2=0x7f6c980d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.251+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2800639886 shutdown_connections 2024-06-29T15:58:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.251+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2800639886 wait complete. 2024-06-29T15:58:01.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.253+0000 7f6c9fdeb640 1 Processor -- start 2024-06-29T15:58:01.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.253+0000 7f6c9fdeb640 1 -- start start 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.254+0000 7f6c9fdeb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c980b3ed0 0x7f6c9815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9fdeb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c980d23c0 0x7f6c9815cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9fdeb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c9815efe0 0x7f6c9815d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9fdeb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6c98160280 con 0x7f6c980b3ed0 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9fdeb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c98160100 con 0x7f6c9815efe0 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9fdeb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6c98160400 con 0x7f6c980d23c0 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c980b3ed0 0x7f6c9815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c980b3ed0 0x7f6c9815a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46602/0 (socket says 172.21.15.59:46602) 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 -- 172.21.15.59:0/2413887320 learned_addr learned my addr 172.21.15.59:0/2413887320 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9e388640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c9815efe0 0x7f6c9815d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9d386640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c980d23c0 0x7f6c9815cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 -- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c980d23c0 msgr2=0x7f6c9815cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c980d23c0 0x7f6c9815cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 -- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c9815efe0 msgr2=0x7f6c9815d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c9815efe0 0x7f6c9815d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 -- 172.21.15.59:0/2413887320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9815aad0 con 0x7f6c980b3ed0 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9e388640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c9815efe0 0x7f6c9815d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9db87640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c980b3ed0 0x7f6c9815a590 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009950 tx=0x7f6c90033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c967fc640 1 -- 172.21.15.59:0/2413887320 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90005c90 con 0x7f6c980b3ed0 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c967fc640 1 -- 172.21.15.59:0/2413887320 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c90031e50 con 0x7f6c980b3ed0 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9815daa0 con 0x7f6c980b3ed0 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.255+0000 7f6c967fc640 1 -- 172.21.15.59:0/2413887320 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90005260 con 0x7f6c980b3ed0 2024-06-29T15:58:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.256+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9815e060 con 0x7f6c980b3ed0 2024-06-29T15:58:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.257+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c980b3990 con 0x7f6c980b3ed0 2024-06-29T15:58:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.258+0000 7f6c967fc640 1 -- 172.21.15.59:0/2413887320 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7f6c90005400 con 0x7f6c980b3ed0 2024-06-29T15:58:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.258+0000 7f6c967fc640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c84076120 0x7f6c84078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.258+0000 7f6c967fc640 1 -- 172.21.15.59:0/2413887320 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7f6c900b90b0 con 0x7f6c980b3ed0 2024-06-29T15:58:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.258+0000 7f6c9d386640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c84076120 0x7f6c84078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.259+0000 7f6c9d386640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c84076120 0x7f6c84078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c9806ce80 tx=0x7f6c8c009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:01.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.262+0000 7f6c967fc640 1 -- 172.21.15.59:0/2413887320 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6c9003f030 con 0x7f6c980b3ed0 2024-06-29T15:58:01.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.451+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c980d2950 con 0x7f6c980b3ed0 2024-06-29T15:58:01.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.452+0000 7f6c967fc640 1 -- 172.21.15.59:0/2413887320 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) ==== 74+0+21978 (secure 0 0 0) 0x7f6c900b8020 con 0x7f6c980b3ed0 2024-06-29T15:58:01.453 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:01.453 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":228,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:59.178408+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1969822965":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/813560021":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2732252046":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/240322135":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/215704132":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/1238730780":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/3658145560":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2050849752":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/239127507":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2051017624":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/743055221":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/4093149557":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/572137829":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2642354773":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2737762390":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2780049427":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2780049427":"2024-06-30T15:57:50.460403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c84076120 msgr2=0x7f6c84078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c84076120 0x7f6c84078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c9806ce80 tx=0x7f6c8c009040 comp rx=0 tx=0).stop 2024-06-29T15:58:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c980b3ed0 msgr2=0x7f6c9815a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c980b3ed0 0x7f6c9815a590 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009950 tx=0x7f6c90033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 shutdown_connections 2024-06-29T15:58:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c84076120 0x7f6c84078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c9815efe0 0x7f6c9815d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c980d23c0 0x7f6c9815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 --2- 172.21.15.59:0/2413887320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c980b3ed0 0x7f6c9815a590 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 >> 172.21.15.59:0/2413887320 conn(0x7f6c980d3ee0 msgr2=0x7f6c98160940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:01.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 shutdown_connections 2024-06-29T15:58:01.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.455+0000 7f6c9fdeb640 1 -- 172.21.15.59:0/2413887320 wait complete. 2024-06-29T15:58:01.466 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:58:01.466 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.585+0000 7fd43a154640 1 -- 172.21.15.59:0/1521694241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd43408c190 msgr2=0x7fd4340cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.585+0000 7fd43a154640 1 --2- 172.21.15.59:0/1521694241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd43408c190 0x7fd4340cf080 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fd4340cfbc0 tx=0x7fd42402a560 comp rx=0 tx=0).stop 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.585+0000 7fd43a154640 1 -- 172.21.15.59:0/1521694241 shutdown_connections 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.585+0000 7fd43a154640 1 --2- 172.21.15.59:0/1521694241 >> v2:172.21.15.153:3300/0 conn(0x7fd4340ce950 0x7fd4340d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.585+0000 7fd43a154640 1 --2- 172.21.15.59:0/1521694241 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4340cf5c0 0x7fd4340d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.585+0000 7fd43a154640 1 --2- 172.21.15.59:0/1521694241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd43408c190 0x7fd4340cf080 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.585+0000 7fd43a154640 1 -- 172.21.15.59:0/1521694241 >> 172.21.15.59:0/1521694241 conn(0x7fd4340b5610 msgr2=0x7fd4340b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.586+0000 7fd43a154640 1 -- 172.21.15.59:0/1521694241 shutdown_connections 2024-06-29T15:58:01.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.586+0000 7fd43a154640 1 -- 172.21.15.59:0/1521694241 wait complete. 2024-06-29T15:58:01.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.587+0000 7fd43a154640 1 Processor -- start 2024-06-29T15:58:01.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.588+0000 7fd43a154640 1 -- start start 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd43a154640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd43408c190 0x7fd4341f9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd43a154640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4340ce950 0x7fd43415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd43a154640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4340cf5c0 0x7fd43406d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd43a154640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd434162440 con 0x7fd4340ce950 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd43a154640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341622c0 con 0x7fd43408c190 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd43a154640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd4341625c0 con 0x7fd4340cf5c0 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4340ce950 0x7fd43415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4340ce950 0x7fd43415c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46618/0 (socket says 172.21.15.59:46618) 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 -- 172.21.15.59:0/3231611396 learned_addr learned my addr 172.21.15.59:0/3231611396 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:01.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd433fff640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4340cf5c0 0x7fd43406d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd4337fe640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd43408c190 0x7fd4341f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 -- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4340cf5c0 msgr2=0x7fd43406d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4340cf5c0 0x7fd43406d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 -- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd43408c190 msgr2=0x7fd4341f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd43408c190 0x7fd4341f9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 -- 172.21.15.59:0/3231611396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4340aa4f0 con 0x7fd4340ce950 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.589+0000 7fd432ffd640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4340ce950 0x7fd43415c020 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7fd42800d740 tx=0x7fd42800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.590+0000 7fd41bfff640 1 -- 172.21.15.59:0/3231611396 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd428017070 con 0x7fd4340ce950 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.590+0000 7fd41bfff640 1 -- 172.21.15.59:0/3231611396 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd428009d20 con 0x7fd4340ce950 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.590+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4340ad050 con 0x7fd4340ce950 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.590+0000 7fd41bfff640 1 -- 172.21.15.59:0/3231611396 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4280106c0 con 0x7fd4340ce950 2024-06-29T15:58:01.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.590+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4340aa790 con 0x7fd4340ce950 2024-06-29T15:58:01.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.591+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4340cf080 con 0x7fd4340ce950 2024-06-29T15:58:01.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.591+0000 7fd41bfff640 1 -- 172.21.15.59:0/3231611396 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7fd42801a070 con 0x7fd4340ce950 2024-06-29T15:58:01.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.592+0000 7fd41bfff640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd420076480 0x7fd420078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.592+0000 7fd41bfff640 1 -- 172.21.15.59:0/3231611396 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7fd428098470 con 0x7fd4340ce950 2024-06-29T15:58:01.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.592+0000 7fd4337fe640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd420076480 0x7fd420078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.593+0000 7fd4337fe640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd420076480 0x7fd420078990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd424033000 tx=0x7fd424004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:01.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.596+0000 7fd41bfff640 1 -- 172.21.15.59:0/3231611396 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd428060b20 con 0x7fd4340ce950 2024-06-29T15:58:01.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.824+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4340acde0 con 0x7fd4340ce950 2024-06-29T15:58:01.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.825+0000 7fd41bfff640 1 -- 172.21.15.59:0/3231611396 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6567 (secure 0 0 0) 0x7fd428064e20 con 0x7fd4340ce950 2024-06-29T15:58:01.825 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:01.825 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":121,"btime":"2024-06-29T15:57:38:315315+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.589050+0000","modified":"2024-06-29T15:57:24.623107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11487":{"gid":11487,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1734341893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1734341893},{"type":"v1","addr":"172.21.15.59:6834","nonce":1734341893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11487,"qdb_cluster":[11487]},"id":13},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:58:01.825 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 121 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd420076480 msgr2=0x7fd420078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd420076480 0x7fd420078990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd424033000 tx=0x7fd424004730 comp rx=0 tx=0).stop 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4340ce950 msgr2=0x7fd43415c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4340ce950 0x7fd43415c020 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7fd42800d740 tx=0x7fd42800dc00 comp rx=0 tx=0).stop 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 shutdown_connections 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd420076480 0x7fd420078990 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd4340cf5c0 0x7fd43406d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4340ce950 0x7fd43415c020 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 --2- 172.21.15.59:0/3231611396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd43408c190 0x7fd4341f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 >> 172.21.15.59:0/3231611396 conn(0x7fd4340b5610 msgr2=0x7fd4340c9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:01.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 shutdown_connections 2024-06-29T15:58:01.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.827+0000 7fd43a154640 1 -- 172.21.15.59:0/3231611396 wait complete. 2024-06-29T15:58:01.838 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.956+0000 7fcfe36ac640 1 -- 172.21.15.59:0/2667908409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc15e0a0 msgr2=0x7fcfdc161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.956+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/2667908409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc15e0a0 0x7fcfdc161a80 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc08c200 tx=0x7fcfcc02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.956+0000 7fcfe36ac640 1 -- 172.21.15.59:0/2667908409 shutdown_connections 2024-06-29T15:58:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.956+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/2667908409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc15e0a0 0x7fcfdc161a80 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.956+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/2667908409 >> v2:172.21.15.153:3300/0 conn(0x7fcfdc162770 0x7fcfdc0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.956+0000 7fcfe36ac640 1 -- 172.21.15.59:0/2667908409 >> 172.21.15.59:0/2667908409 conn(0x7fcfdc0d45b0 msgr2=0x7fcfdc0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.956+0000 7fcfe36ac640 1 -- 172.21.15.59:0/2667908409 shutdown_connections 2024-06-29T15:58:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.956+0000 7fcfe36ac640 1 -- 172.21.15.59:0/2667908409 wait complete. 2024-06-29T15:58:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.958+0000 7fcfe36ac640 1 Processor -- start 2024-06-29T15:58:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.958+0000 7fcfe36ac640 1 -- start start 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe36ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfdc15e0a0 0x7fcfdc1608f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe36ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc161fc0 0x7fcfdc149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe36ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfdc162770 0x7fcfdc14a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe36ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc0d2d80 con 0x7fcfdc161fc0 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe36ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc0b4b00 con 0x7fcfdc15e0a0 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe36ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcfdc0d2f00 con 0x7fcfdc162770 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe0c47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc161fc0 0x7fcfdc149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe0c47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc161fc0 0x7fcfdc149ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46644/0 (socket says 172.21.15.59:46644) 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe0c47640 1 -- 172.21.15.59:0/935554590 learned_addr learned my addr 172.21.15.59:0/935554590 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe0c47640 1 -- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfdc162770 msgr2=0x7fcfdc14a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe0c47640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfdc162770 0x7fcfdc14a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe0c47640 1 -- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfdc15e0a0 msgr2=0x7fcfdc1608f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe0c47640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfdc15e0a0 0x7fcfdc1608f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.959+0000 7fcfe0c47640 1 -- 172.21.15.59:0/935554590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc1f4e40 con 0x7fcfdc161fc0 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.960+0000 7fcfe0c47640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc161fc0 0x7fcfdc149ac0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0098f0 tx=0x7fcfcc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.960+0000 7fcfd1ffb640 1 -- 172.21.15.59:0/935554590 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc031d60 con 0x7fcfdc161fc0 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.960+0000 7fcfd1ffb640 1 -- 172.21.15.59:0/935554590 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfcc033e10 con 0x7fcfdc161fc0 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.960+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc0cb750 con 0x7fcfdc161fc0 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.960+0000 7fcfd1ffb640 1 -- 172.21.15.59:0/935554590 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc005260 con 0x7fcfdc161fc0 2024-06-29T15:58:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.960+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc0cbd10 con 0x7fcfdc161fc0 2024-06-29T15:58:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.961+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfdc0d3420 con 0x7fcfdc161fc0 2024-06-29T15:58:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.962+0000 7fcfd1ffb640 1 -- 172.21.15.59:0/935554590 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7fcfcc005400 con 0x7fcfdc161fc0 2024-06-29T15:58:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.962+0000 7fcfd1ffb640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc00764b0 0x7fcfc00789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.962+0000 7fcfd1ffb640 1 -- 172.21.15.59:0/935554590 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7fcfcc0b94c0 con 0x7fcfdc161fc0 2024-06-29T15:58:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.962+0000 7fcfe1448640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc00764b0 0x7fcfc00789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.963+0000 7fcfe1448640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc00764b0 0x7fcfc00789c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80058c0 tx=0x7fcfd8005850 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:01.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:01.966+0000 7fcfd1ffb640 1 -- 172.21.15.59:0/935554590 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcfcc03f030 con 0x7fcfdc161fc0 2024-06-29T15:58:02.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.197+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfdc06cd90 con 0x7fcfdc161fc0 2024-06-29T15:58:02.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.198+0000 7fcfd1ffb640 1 -- 172.21.15.59:0/935554590 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6567 (secure 0 0 0) 0x7fcfcc082240 con 0x7fcfdc161fc0 2024-06-29T15:58:02.198 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:02.198 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":121,"btime":"2024-06-29T15:57:38:315315+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.589050+0000","modified":"2024-06-29T15:57:24.623107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11487":{"gid":11487,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/1734341893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1734341893},{"type":"v1","addr":"172.21.15.59:6834","nonce":1734341893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11487,"qdb_cluster":[11487]},"id":13},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:58:02.199 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 121 2024-06-29T15:58:02.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc00764b0 msgr2=0x7fcfc00789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc00764b0 0x7fcfc00789c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80058c0 tx=0x7fcfd8005850 comp rx=0 tx=0).stop 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc161fc0 msgr2=0x7fcfdc149ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc161fc0 0x7fcfdc149ac0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0098f0 tx=0x7fcfcc033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 shutdown_connections 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc00764b0 0x7fcfc00789c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfdc162770 0x7fcfdc14a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfdc161fc0 0x7fcfdc149ac0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 --2- 172.21.15.59:0/935554590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfdc15e0a0 0x7fcfdc1608f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 >> 172.21.15.59:0/935554590 conn(0x7fcfdc0d45b0 msgr2=0x7fcfdc06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 shutdown_connections 2024-06-29T15:58:02.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.200+0000 7fcfe36ac640 1 -- 172.21.15.59:0/935554590 wait complete. 2024-06-29T15:58:02.211 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:02.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.331+0000 7f14267cf640 1 -- 172.21.15.59:0/3835437329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14200ce950 msgr2=0x7f14200d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.331+0000 7f14267cf640 1 --2- 172.21.15.59:0/3835437329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14200ce950 0x7f14200d2130 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1418005fd0 tx=0x7f141802a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.331+0000 7f14267cf640 1 -- 172.21.15.59:0/3835437329 shutdown_connections 2024-06-29T15:58:02.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.331+0000 7f14267cf640 1 --2- 172.21.15.59:0/3835437329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14200ce950 0x7f14200d2130 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.331+0000 7f14267cf640 1 --2- 172.21.15.59:0/3835437329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142008c190 0x7f14200cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.331+0000 7f14267cf640 1 -- 172.21.15.59:0/3835437329 >> 172.21.15.59:0/3835437329 conn(0x7f14200b5610 msgr2=0x7f14200b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:02.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.332+0000 7f14267cf640 1 -- 172.21.15.59:0/3835437329 shutdown_connections 2024-06-29T15:58:02.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.332+0000 7f14267cf640 1 -- 172.21.15.59:0/3835437329 wait complete. 2024-06-29T15:58:02.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.333+0000 7f14267cf640 1 Processor -- start 2024-06-29T15:58:02.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.334+0000 7f14267cf640 1 -- start start 2024-06-29T15:58:02.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f14267cf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142008c190 0x7f14200b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f14267cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14200ce950 0x7f142014ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f14267cf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14200cf5c0 0x7f142014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f14267cf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f14200b7fa0 con 0x7f14200cf5c0 2024-06-29T15:58:02.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f14267cf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14200b7e20 con 0x7f14200ce950 2024-06-29T15:58:02.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f14267cf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f14200b8120 con 0x7f142008c190 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f1424d6c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14200cf5c0 0x7f142014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f1424d6c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14200cf5c0 0x7f142014ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46650/0 (socket says 172.21.15.59:46650) 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f1424d6c640 1 -- 172.21.15.59:0/3159134624 learned_addr learned my addr 172.21.15.59:0/3159134624 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f141ffff640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142008c190 0x7f14200b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f141f7fe640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14200ce950 0x7f142014ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f1424d6c640 1 -- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142008c190 msgr2=0x7f14200b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f1424d6c640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142008c190 0x7f14200b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f1424d6c640 1 -- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14200ce950 msgr2=0x7f142014ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.335+0000 7f1424d6c640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14200ce950 0x7f142014ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.336+0000 7f1424d6c640 1 -- 172.21.15.59:0/3159134624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142014b790 con 0x7f14200cf5c0 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.336+0000 7f1424d6c640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14200cf5c0 0x7f142014ccb0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f141802aea0 tx=0x7f1418033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.336+0000 7f141cff9640 1 -- 172.21.15.59:0/3159134624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418031990 con 0x7f14200cf5c0 2024-06-29T15:58:02.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.336+0000 7f141cff9640 1 -- 172.21.15.59:0/3159134624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1418031c20 con 0x7f14200cf5c0 2024-06-29T15:58:02.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.336+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142014ba70 con 0x7f14200cf5c0 2024-06-29T15:58:02.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.336+0000 7f141cff9640 1 -- 172.21.15.59:0/3159134624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14180330d0 con 0x7f14200cf5c0 2024-06-29T15:58:02.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.336+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f142014c030 con 0x7f14200cf5c0 2024-06-29T15:58:02.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.337+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f142014b500 con 0x7f14200cf5c0 2024-06-29T15:58:02.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.338+0000 7f141cff9640 1 -- 172.21.15.59:0/3159134624 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7f1418033270 con 0x7f14200cf5c0 2024-06-29T15:58:02.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.338+0000 7f141cff9640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1400076480 0x7f1400078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.338+0000 7f141cff9640 1 -- 172.21.15.59:0/3159134624 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7f14180b98e0 con 0x7f14200cf5c0 2024-06-29T15:58:02.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.339+0000 7f141ffff640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1400076480 0x7f1400078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.339+0000 7f141ffff640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1400076480 0x7f1400078990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1410009ac0 tx=0x7f1410005f90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:02.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.343+0000 7f141cff9640 1 -- 172.21.15.59:0/3159134624 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f141803f030 con 0x7f14200cf5c0 2024-06-29T15:58:02.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.535+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14200cf080 con 0x7f14200cf5c0 2024-06-29T15:58:02.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.537+0000 7f141cff9640 1 -- 172.21.15.59:0/3159134624 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) ==== 74+0+21978 (secure 0 0 0) 0x7f14180825e0 con 0x7f14200cf5c0 2024-06-29T15:58:02.537 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:02.537 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":228,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:57:59.178408+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.955891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"221","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.545869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1969822965":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/813560021":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2732252046":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/240322135":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/215704132":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/1238730780":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/3658145560":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2050849752":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/239127507":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2051017624":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/743055221":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/4093149557":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/572137829":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2642354773":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2737762390":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2780049427":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2780049427":"2024-06-30T15:57:50.460403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:02.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1400076480 msgr2=0x7f1400078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1400076480 0x7f1400078990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1410009ac0 tx=0x7f1410005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:02.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14200cf5c0 msgr2=0x7f142014ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14200cf5c0 0x7f142014ccb0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f141802aea0 tx=0x7f1418033f70 comp rx=0 tx=0).stop 2024-06-29T15:58:02.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 shutdown_connections 2024-06-29T15:58:02.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1400076480 0x7f1400078990 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14200cf5c0 0x7f142014ccb0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14200ce950 0x7f142014ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 --2- 172.21.15.59:0/3159134624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f142008c190 0x7f14200b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 >> 172.21.15.59:0/3159134624 conn(0x7f14200b5610 msgr2=0x7f14200c7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:02.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 shutdown_connections 2024-06-29T15:58:02.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.539+0000 7f14267cf640 1 -- 172.21.15.59:0/3159134624 wait complete. 2024-06-29T15:58:02.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:58:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.670+0000 7f6753923640 1 -- 172.21.15.59:0/538623325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6738005380 msgr2=0x7f673800ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.670+0000 7f6753923640 1 --2- 172.21.15.59:0/538623325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6738005380 0x7f673800ba40 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f6744005fd0 tx=0x7f6738005830 comp rx=0 tx=0).stop 2024-06-29T15:58:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.670+0000 7f6753923640 1 -- 172.21.15.59:0/538623325 shutdown_connections 2024-06-29T15:58:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.670+0000 7f6753923640 1 --2- 172.21.15.59:0/538623325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6738005380 0x7f673800ba40 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.670+0000 7f6753923640 1 -- 172.21.15.59:0/538623325 >> 172.21.15.59:0/538623325 conn(0x7f674c0b5610 msgr2=0x7f674c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.670+0000 7f6753923640 1 -- 172.21.15.59:0/538623325 shutdown_connections 2024-06-29T15:58:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.670+0000 7f6753923640 1 -- 172.21.15.59:0/538623325 wait complete. 2024-06-29T15:58:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.672+0000 7f6753923640 1 Processor -- start 2024-06-29T15:58:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.672+0000 7f6753923640 1 -- start start 2024-06-29T15:58:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.673+0000 7f6753923640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6738005380 0x7f674c15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.673+0000 7f6753923640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f674c0cd5c0 0x7f674c0cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.673+0000 7f6753923640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f674c0ccd30 0x7f674c1e8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.673+0000 7f6753923640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0cf620 con 0x7f674c0ccd30 2024-06-29T15:58:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.673+0000 7f6753923640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0cf4a0 con 0x7f6738005380 2024-06-29T15:58:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.673+0000 7f6753923640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f674c0cf7a0 con 0x7f674c0cd5c0 2024-06-29T15:58:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f674c0ccd30 0x7f674c1e8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f674c0ccd30 0x7f674c1e8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46664/0 (socket says 172.21.15.59:46664) 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 -- 172.21.15.59:0/489491413 learned_addr learned my addr 172.21.15.59:0/489491413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f67516bf640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6738005380 0x7f674c15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6750ebe640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f674c0cd5c0 0x7f674c0cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 -- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f674c0cd5c0 msgr2=0x7f674c0cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f674c0cd5c0 0x7f674c0cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 -- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6738005380 msgr2=0x7f674c15bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6738005380 0x7f674c15bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 -- 172.21.15.59:0/489491413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c0cd140 con 0x7f674c0ccd30 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6751ec0640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f674c0ccd30 0x7f674c1e8fb0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f6744009e20 tx=0x7f67440042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6741ffb640 1 -- 172.21.15.59:0/489491413 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744009200 con 0x7f674c0ccd30 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6741ffb640 1 -- 172.21.15.59:0/489491413 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6744042070 con 0x7f674c0ccd30 2024-06-29T15:58:02.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c0b3d80 con 0x7f674c0ccd30 2024-06-29T15:58:02.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.674+0000 7f6741ffb640 1 -- 172.21.15.59:0/489491413 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744009ad0 con 0x7f674c0ccd30 2024-06-29T15:58:02.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.675+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c0b4340 con 0x7f674c0ccd30 2024-06-29T15:58:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.676+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674c1625f0 con 0x7f674c0ccd30 2024-06-29T15:58:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.676+0000 7f6741ffb640 1 -- 172.21.15.59:0/489491413 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7f6744047460 con 0x7f674c0ccd30 2024-06-29T15:58:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.677+0000 7f6741ffb640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6730076480 0x7f6730078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.677+0000 7f6741ffb640 1 -- 172.21.15.59:0/489491413 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 7838+0+0 (secure 0 0 0) 0x7f67440b95d0 con 0x7f674c0ccd30 2024-06-29T15:58:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.677+0000 7f67516bf640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6730076480 0x7f6730078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.678+0000 7f67516bf640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6730076480 0x7f6730078990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6748001920 tx=0x7f6748006040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:02.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.681+0000 7f6741ffb640 1 -- 172.21.15.59:0/489491413 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f674c0cd140 con 0x7f674c0ccd30 2024-06-29T15:58:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.917+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f674c06dd80 con 0x7f674c0ccd30 2024-06-29T15:58:02.917 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:58:02.917+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:58:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.919+0000 7f6741ffb640 1 -- 172.21.15.59:0/489491413 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v122) ==== 161+0+0 (secure 0 0 0) 0x7f674408a3b0 con 0x7f674c0ccd30 2024-06-29T15:58:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:58:02.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6730076480 msgr2=0x7f6730078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6730076480 0x7f6730078990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6748001920 tx=0x7f6748006040 comp rx=0 tx=0).stop 2024-06-29T15:58:02.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f674c0ccd30 msgr2=0x7f674c1e8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f674c0ccd30 0x7f674c1e8fb0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f6744009e20 tx=0x7f67440042d0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 shutdown_connections 2024-06-29T15:58:02.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6730076480 0x7f6730078990 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f674c0ccd30 0x7f674c1e8fb0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f674c0cd5c0 0x7f674c0cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 --2- 172.21.15.59:0/489491413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6738005380 0x7f674c15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.922+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 >> 172.21.15.59:0/489491413 conn(0x7f674c0b5610 msgr2=0x7f674c0b7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:02.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.923+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 shutdown_connections 2024-06-29T15:58:02.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:02.923+0000 7f6753923640 1 -- 172.21.15.59:0/489491413 wait complete. 2024-06-29T15:58:02.926 INFO:tasks.ceph.mds.a.smithi059.stderr: -4797> 2024-06-29T15:57:28.184+0000 7fde07262640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:02.936 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:58:02.982 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:58:03.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.059+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/3982584480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd40cf310 msgr2=0x7f6bd415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.059+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/3982584480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd40cf310 0x7f6bd415b340 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4071920 tx=0x7f6bc802b890 comp rx=0 tx=0).stop 2024-06-29T15:58:03.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.059+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/3982584480 shutdown_connections 2024-06-29T15:58:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.059+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/3982584480 >> v2:172.21.15.153:3300/0 conn(0x7f6bd40b3410 0x7f6bd40b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.059+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/3982584480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd40cf310 0x7f6bd415b340 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.059+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/3982584480 >> 172.21.15.59:0/3982584480 conn(0x7f6bd406cb30 msgr2=0x7f6bd406cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.059+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/3982584480 shutdown_connections 2024-06-29T15:58:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.059+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/3982584480 wait complete. 2024-06-29T15:58:03.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.061+0000 7f6bdb4a7640 1 Processor -- start 2024-06-29T15:58:03.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.061+0000 7f6bdb4a7640 1 -- start start 2024-06-29T15:58:03.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bdb4a7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bd40b3410 0x7f6bd41524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:03.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bdb4a7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd40cf310 0x7f6bd4158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bdb4a7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd415deb0 0x7f6bd4153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bdb4a7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd4073160 con 0x7f6bd415deb0 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bdb4a7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd40b4dd0 con 0x7f6bd40cf310 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bdb4a7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd40732e0 con 0x7f6bd40b3410 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bd9a44640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd415deb0 0x7f6bd4153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bd9a44640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd415deb0 0x7f6bd4153b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56632/0 (socket says 172.21.15.59:56632) 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bd9a44640 1 -- 172.21.15.59:0/114066173 learned_addr learned my addr 172.21.15.59:0/114066173 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bd8a42640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd40cf310 0x7f6bd4158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.062+0000 7f6bd9243640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bd40b3410 0x7f6bd41524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bd9a44640 1 -- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bd40b3410 msgr2=0x7f6bd41524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bd9a44640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bd40b3410 0x7f6bd41524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bd9a44640 1 -- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd40cf310 msgr2=0x7f6bd4158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bd9a44640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd40cf310 0x7f6bd4158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bd9a44640 1 -- 172.21.15.59:0/114066173 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd4150f40 con 0x7f6bd415deb0 2024-06-29T15:58:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bd9a44640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd415deb0 0x7f6bd4153b60 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c700 tx=0x7f6bcc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bc5ffb640 1 -- 172.21.15.59:0/114066173 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc003b70 con 0x7f6bd415deb0 2024-06-29T15:58:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bc5ffb640 1 -- 172.21.15.59:0/114066173 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bcc003d10 con 0x7f6bd415deb0 2024-06-29T15:58:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd4151440 con 0x7f6bd415deb0 2024-06-29T15:58:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bc5ffb640 1 -- 172.21.15.59:0/114066173 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00aa50 con 0x7f6bd415deb0 2024-06-29T15:58:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.063+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd4151a00 con 0x7f6bd415deb0 2024-06-29T15:58:03.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.064+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd40740a0 con 0x7f6bd415deb0 2024-06-29T15:58:03.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.065+0000 7f6bc5ffb640 1 -- 172.21.15.59:0/114066173 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7f6bcc00abf0 con 0x7f6bd415deb0 2024-06-29T15:58:03.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.065+0000 7f6bc5ffb640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6bb80764b0 0x7f6bb80789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:03.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.066+0000 7f6bc5ffb640 1 -- 172.21.15.59:0/114066173 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 7924+0+0 (secure 0 0 0) 0x7f6bcc098060 con 0x7f6bd415deb0 2024-06-29T15:58:03.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.066+0000 7f6bd9243640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6bb80764b0 0x7f6bb80789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:03.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.066+0000 7f6bd9243640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6bb80764b0 0x7f6bb80789c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8035000 tx=0x7f6bc8004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:03.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.070+0000 7f6bc5ffb640 1 -- 172.21.15.59:0/114066173 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6bd40740a0 con 0x7f6bd415deb0 2024-06-29T15:58:03.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.301+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6bd4151190 con 0x7f6bd415deb0 2024-06-29T15:58:03.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.925+0000 7f6bc5ffb640 1 -- 172.21.15.59:0/114066173 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v123) ==== 104+0+0 (secure 0 0 0) 0x7f6bd40b3d60 con 0x7f6bd415deb0 2024-06-29T15:58:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6bb80764b0 msgr2=0x7f6bb80789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6bb80764b0 0x7f6bb80789c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8035000 tx=0x7f6bc8004730 comp rx=0 tx=0).stop 2024-06-29T15:58:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd415deb0 msgr2=0x7f6bd4153b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd415deb0 0x7f6bd4153b60 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c700 tx=0x7f6bcc00cbc0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 shutdown_connections 2024-06-29T15:58:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6bb80764b0 0x7f6bb80789c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bd415deb0 0x7f6bd4153b60 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6bd40cf310 0x7f6bd4158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 --2- 172.21.15.59:0/114066173 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6bd40b3410 0x7f6bd41524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.928+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 >> 172.21.15.59:0/114066173 conn(0x7f6bd406cb30 msgr2=0x7f6bd4155b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.929+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 shutdown_connections 2024-06-29T15:58:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:03.929+0000 7f6bdb4a7640 1 -- 172.21.15.59:0/114066173 wait complete. 2024-06-29T15:58:03.943 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:58:04.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.070+0000 7f4e3429e640 1 -- 172.21.15.59:0/36136594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 msgr2=0x7f4e2c15b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.070+0000 7f4e3429e640 1 --2- 172.21.15.59:0/36136594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 0x7f4e2c15b340 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c073980 tx=0x7f4e1c02bb50 comp rx=0 tx=0).stop 2024-06-29T15:58:04.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.070+0000 7f4e3429e640 1 -- 172.21.15.59:0/36136594 shutdown_connections 2024-06-29T15:58:04.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.070+0000 7f4e3429e640 1 --2- 172.21.15.59:0/36136594 >> v2:172.21.15.153:3300/0 conn(0x7f4e2c072150 0x7f4e2c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.070+0000 7f4e3429e640 1 --2- 172.21.15.59:0/36136594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 0x7f4e2c15b340 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.070+0000 7f4e3429e640 1 -- 172.21.15.59:0/36136594 >> 172.21.15.59:0/36136594 conn(0x7f4e2c06cb30 msgr2=0x7f4e2c06cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:04.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.070+0000 7f4e3429e640 1 -- 172.21.15.59:0/36136594 shutdown_connections 2024-06-29T15:58:04.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.070+0000 7f4e3429e640 1 -- 172.21.15.59:0/36136594 wait complete. 2024-06-29T15:58:04.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.072+0000 7f4e3429e640 1 Processor -- start 2024-06-29T15:58:04.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.072+0000 7f4e3429e640 1 -- start start 2024-06-29T15:58:04.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3429e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e2c072150 0x7f4e2c14aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3429e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e2c0b3410 0x7f4e2c14db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3429e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 0x7f4e2c14e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3429e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c06f300 con 0x7f4e2c0cf310 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3429e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c06f180 con 0x7f4e2c0b3410 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3429e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4e2c06f480 con 0x7f4e2c072150 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3283b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 0x7f4e2c14e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3283b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 0x7f4e2c14e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56654/0 (socket says 172.21.15.59:56654) 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.073+0000 7f4e3283b640 1 -- 172.21.15.59:0/4191787603 learned_addr learned my addr 172.21.15.59:0/4191787603 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e31839640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e2c0b3410 0x7f4e2c14db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3203a640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e2c072150 0x7f4e2c14aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3283b640 1 -- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e2c072150 msgr2=0x7f4e2c14aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3283b640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e2c072150 0x7f4e2c14aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3283b640 1 -- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e2c0b3410 msgr2=0x7f4e2c14db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3283b640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e2c0b3410 0x7f4e2c14db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3283b640 1 -- 172.21.15.59:0/4191787603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2c14afc0 con 0x7f4e2c0cf310 2024-06-29T15:58:04.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3283b640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 0x7f4e2c14e050 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400c9a0 tx=0x7f4e2400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:04.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e2affd640 1 -- 172.21.15.59:0/4191787603 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24007bb0 con 0x7f4e2c0cf310 2024-06-29T15:58:04.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e2affd640 1 -- 172.21.15.59:0/4191787603 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e2400f450 con 0x7f4e2c0cf310 2024-06-29T15:58:04.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e2affd640 1 -- 172.21.15.59:0/4191787603 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24007e60 con 0x7f4e2c0cf310 2024-06-29T15:58:04.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2c1e6170 con 0x7f4e2c0cf310 2024-06-29T15:58:04.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.074+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2c1e6730 con 0x7f4e2c0cf310 2024-06-29T15:58:04.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.075+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e2c15b340 con 0x7f4e2c0cf310 2024-06-29T15:58:04.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.076+0000 7f4e2affd640 1 -- 172.21.15.59:0/4191787603 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4e2400f8a0 con 0x7f4e2c0cf310 2024-06-29T15:58:04.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.076+0000 7f4e2affd640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e18076510 0x7f4e18078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.077+0000 7f4e2affd640 1 -- 172.21.15.59:0/4191787603 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 7924+0+0 (secure 0 0 0) 0x7f4e24098cf0 con 0x7f4e2c0cf310 2024-06-29T15:58:04.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.077+0000 7f4e3203a640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e18076510 0x7f4e18078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.077+0000 7f4e3203a640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e18076510 0x7f4e18078a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c035000 tx=0x7f4e1c004ab0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:04.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.081+0000 7f4e2affd640 1 -- 172.21.15.59:0/4191787603 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4e2409d070 con 0x7f4e2c0cf310 2024-06-29T15:58:04.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.274+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4e2c1e5f00 con 0x7f4e2c0cf310 2024-06-29T15:58:04.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.927+0000 7f4e2affd640 1 -- 172.21.15.59:0/4191787603 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v230) ==== 181+0+0 (secure 0 0 0) 0x7f4e2c14afc0 con 0x7f4e2c0cf310 2024-06-29T15:58:04.927 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.932+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e18076510 msgr2=0x7f4e18078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.932+0000 7f4e3429e640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e18076510 0x7f4e18078a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c035000 tx=0x7f4e1c004ab0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 msgr2=0x7f4e2c14e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 0x7f4e2c14e050 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400c9a0 tx=0x7f4e2400ce60 comp rx=0 tx=0).stop 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 shutdown_connections 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e18076510 0x7f4e18078a20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e2c0cf310 0x7f4e2c14e050 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e2c0b3410 0x7f4e2c14db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 --2- 172.21.15.59:0/4191787603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4e2c072150 0x7f4e2c14aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 >> 172.21.15.59:0/4191787603 conn(0x7f4e2c06cb30 msgr2=0x7f4e2c0ce8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 shutdown_connections 2024-06-29T15:58:04.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:04.933+0000 7f4e3429e640 1 -- 172.21.15.59:0/4191787603 wait complete. 2024-06-29T15:58:04.947 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:58:05.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.076+0000 7f2129278640 1 -- 172.21.15.59:0/3492742194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240b0e30 msgr2=0x7f21241596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.076+0000 7f2129278640 1 --2- 172.21.15.59:0/3492742194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240b0e30 0x7f21241596a0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f210c005fd0 tx=0x7f210c02a4f0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.076+0000 7f2129278640 1 -- 172.21.15.59:0/3492742194 shutdown_connections 2024-06-29T15:58:05.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.076+0000 7f2129278640 1 --2- 172.21.15.59:0/3492742194 >> v2:172.21.15.153:3300/0 conn(0x7f2124160090 0x7f21240b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.076+0000 7f2129278640 1 --2- 172.21.15.59:0/3492742194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240b0e30 0x7f21241596a0 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.076+0000 7f2129278640 1 -- 172.21.15.59:0/3492742194 >> 172.21.15.59:0/3492742194 conn(0x7f21240b7690 msgr2=0x7f2124073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:05.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.076+0000 7f2129278640 1 -- 172.21.15.59:0/3492742194 shutdown_connections 2024-06-29T15:58:05.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.076+0000 7f2129278640 1 -- 172.21.15.59:0/3492742194 wait complete. 2024-06-29T15:58:05.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.078+0000 7f2129278640 1 Processor -- start 2024-06-29T15:58:05.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.078+0000 7f2129278640 1 -- start start 2024-06-29T15:58:05.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f2129278640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21240ce2c0 0x7f21240b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f2129278640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2124160090 0x7f212414b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f2129278640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240d0f30 0x7f212414f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f2129278640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f21240b54e0 con 0x7f21240d0f30 2024-06-29T15:58:05.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f2129278640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f21240b5360 con 0x7f21240ce2c0 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f2129278640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f21240b5660 con 0x7f2124160090 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f21237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240d0f30 0x7f212414f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f21237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240d0f30 0x7f212414f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56668/0 (socket says 172.21.15.59:56668) 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f21237fe640 1 -- 172.21.15.59:0/3648124286 learned_addr learned my addr 172.21.15.59:0/3648124286 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.079+0000 7f2122ffd640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21240ce2c0 0x7f21240b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f21227fc640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2124160090 0x7f212414b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f21237fe640 1 -- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2124160090 msgr2=0x7f212414b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f21237fe640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2124160090 0x7f212414b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f21237fe640 1 -- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21240ce2c0 msgr2=0x7f21240b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f21237fe640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21240ce2c0 0x7f21240b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f21237fe640 1 -- 172.21.15.59:0/3648124286 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21240b15b0 con 0x7f21240d0f30 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f21237fe640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240d0f30 0x7f212414f490 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f211c00b440 tx=0x7f211c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f211b7fe640 1 -- 172.21.15.59:0/3648124286 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c015020 con 0x7f21240d0f30 2024-06-29T15:58:05.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f211b7fe640 1 -- 172.21.15.59:0/3648124286 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f211c011920 con 0x7f21240d0f30 2024-06-29T15:58:05.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21240b1890 con 0x7f21240d0f30 2024-06-29T15:58:05.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f211b7fe640 1 -- 172.21.15.59:0/3648124286 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c011d70 con 0x7f21240d0f30 2024-06-29T15:58:05.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.080+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21240b1e50 con 0x7f21240d0f30 2024-06-29T15:58:05.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.081+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21240d1720 con 0x7f21240d0f30 2024-06-29T15:58:05.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.082+0000 7f211b7fe640 1 -- 172.21.15.59:0/3648124286 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f211c018020 con 0x7f21240d0f30 2024-06-29T15:58:05.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.082+0000 7f211b7fe640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2104076510 0x7f2104078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.082+0000 7f211b7fe640 1 -- 172.21.15.59:0/3648124286 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 7486+0+0 (secure 0 0 0) 0x7f211c098240 con 0x7f21240d0f30 2024-06-29T15:58:05.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.083+0000 7f2122ffd640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2104076510 0x7f2104078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.083+0000 7f2122ffd640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2104076510 0x7f2104078a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2114009f60 tx=0x7f21140094f0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:05.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.087+0000 7f211b7fe640 1 -- 172.21.15.59:0/3648124286 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f211c01f020 con 0x7f21240d0f30 2024-06-29T15:58:05.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.282+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f21241596a0 con 0x7f21240d0f30 2024-06-29T15:58:05.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.936+0000 7f211b7fe640 1 -- 172.21.15.59:0/3648124286 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v231) ==== 169+0+0 (secure 0 0 0) 0x7f211c061150 con 0x7f21240d0f30 2024-06-29T15:58:05.936 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T15:58:05.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.941+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2104076510 msgr2=0x7f2104078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.941+0000 7f2129278640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2104076510 0x7f2104078a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2114009f60 tx=0x7f21140094f0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.941+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240d0f30 msgr2=0x7f212414f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.941+0000 7f2129278640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240d0f30 0x7f212414f490 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f211c00b440 tx=0x7f211c00b840 comp rx=0 tx=0).stop 2024-06-29T15:58:05.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.942+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 shutdown_connections 2024-06-29T15:58:05.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.942+0000 7f2129278640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2104076510 0x7f2104078a20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.942+0000 7f2129278640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21240d0f30 0x7f212414f490 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.942+0000 7f2129278640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2124160090 0x7f212414b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.942+0000 7f2129278640 1 --2- 172.21.15.59:0/3648124286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21240ce2c0 0x7f21240b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.942+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 >> 172.21.15.59:0/3648124286 conn(0x7f21240b7690 msgr2=0x7f2124068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:05.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.942+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 shutdown_connections 2024-06-29T15:58:05.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:05.942+0000 7f2129278640 1 -- 172.21.15.59:0/3648124286 wait complete. 2024-06-29T15:58:05.957 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:06.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.082+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3768156481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4eb40b2270 msgr2=0x7f4eb415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.082+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3768156481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4eb40b2270 0x7f4eb415bc20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009830 tx=0x7f4eac02a510 comp rx=0 tx=0).stop 2024-06-29T15:58:06.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.082+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3768156481 shutdown_connections 2024-06-29T15:58:06.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.082+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3768156481 >> v2:172.21.15.153:3300/0 conn(0x7f4eb40b2fa0 0x7f4eb40d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.082+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3768156481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4eb40b2270 0x7f4eb415bc20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.082+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3768156481 >> 172.21.15.59:0/3768156481 conn(0x7f4eb4073e10 msgr2=0x7f4eb4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.082+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3768156481 shutdown_connections 2024-06-29T15:58:06.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.082+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3768156481 wait complete. 2024-06-29T15:58:06.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.084+0000 7f4eb8b2a640 1 Processor -- start 2024-06-29T15:58:06.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.084+0000 7f4eb8b2a640 1 -- start start 2024-06-29T15:58:06.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb8b2a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4eb40b2270 0x7f4eb4147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb8b2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4eb40b2fa0 0x7f4eb41566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb8b2a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eb40d1a60 0x7f4eb4156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb8b2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb40b6cc0 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb8b2a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb40b6b40 con 0x7f4eb40d1a60 2024-06-29T15:58:06.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb8b2a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4eb40cfe90 con 0x7f4eb40b2270 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4eb40b2fa0 0x7f4eb41566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4eb40b2fa0 0x7f4eb41566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56680/0 (socket says 172.21.15.59:56680) 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb1d9b640 1 -- 172.21.15.59:0/3426630272 learned_addr learned my addr 172.21.15.59:0/3426630272 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb259c640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4eb40b2270 0x7f4eb4147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.085+0000 7f4eb2d9d640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eb40d1a60 0x7f4eb4156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eb1d9b640 1 -- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4eb40b2270 msgr2=0x7f4eb4147160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eb1d9b640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4eb40b2270 0x7f4eb4147160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eb1d9b640 1 -- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eb40d1a60 msgr2=0x7f4eb4156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eb1d9b640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eb40d1a60 0x7f4eb4156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eb1d9b640 1 -- 172.21.15.59:0/3426630272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb4147cd0 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eb1d9b640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4eb40b2fa0 0x7f4eb41566a0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e9f0 tx=0x7f4e9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eaaffd640 1 -- 172.21.15.59:0/3426630272 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00cc60 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eaaffd640 1 -- 172.21.15.59:0/3426630272 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e9c00ce00 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb4147fb0 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eaaffd640 1 -- 172.21.15.59:0/3426630272 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c004280 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.086+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb41453b0 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.087+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eb40b4fe0 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.088+0000 7f4eaaffd640 1 -- 172.21.15.59:0/3426630272 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4e9c0035f0 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.088+0000 7f4eaaffd640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e94076510 0x7f4e94078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.089+0000 7f4eaaffd640 1 -- 172.21.15.59:0/3426630272 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7f4e9c098a80 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.089+0000 7f4eb259c640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e94076510 0x7f4e94078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.089+0000 7f4eb259c640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e94076510 0x7f4e94078a20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4eac034000 tx=0x7f4eac004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.093+0000 7f4eaaffd640 1 -- 172.21.15.59:0/3426630272 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4eb40b4fe0 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.289+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4eb40d0340 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.290+0000 7f4eaaffd640 1 -- 172.21.15.59:0/3426630272 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) ==== 74+0+18330 (secure 0 0 0) 0x7f4e9c061340 con 0x7f4eb40b2fa0 2024-06-29T15:58:06.290 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:06.290 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":231,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:05.928387+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6832/1734341893":"2024-06-30T15:58:02.917822+0000","172.21.15.153:0/1969822965":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/813560021":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2732252046":"2024-06-30T15:57:59.178318+0000","172.21.15.59:6834/1734341893":"2024-06-30T15:58:02.917822+0000","172.21.15.153:0/240322135":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/215704132":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/1238730780":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/3658145560":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2050849752":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/239127507":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2051017624":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/743055221":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/4093149557":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/572137829":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2642354773":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2737762390":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2780049427":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2780049427":"2024-06-30T15:57:50.460403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e94076510 msgr2=0x7f4e94078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e94076510 0x7f4e94078a20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4eac034000 tx=0x7f4eac004730 comp rx=0 tx=0).stop 2024-06-29T15:58:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4eb40b2fa0 msgr2=0x7f4eb41566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4eb40b2fa0 0x7f4eb41566a0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e9f0 tx=0x7f4e9c00eeb0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 shutdown_connections 2024-06-29T15:58:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4e94076510 0x7f4e94078a20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eb40d1a60 0x7f4eb4156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4eb40b2fa0 0x7f4eb41566a0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 --2- 172.21.15.59:0/3426630272 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4eb40b2270 0x7f4eb4147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 >> 172.21.15.59:0/3426630272 conn(0x7f4eb4073e10 msgr2=0x7f4eb40d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 shutdown_connections 2024-06-29T15:58:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.292+0000 7f4eb8b2a640 1 -- 172.21.15.59:0/3426630272 wait complete. 2024-06-29T15:58:06.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:57:32.305142+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:57:33.647895+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:06.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:06.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.430+0000 7f7955f7f640 1 -- 172.21.15.59:0/825139294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79500b40a0 msgr2=0x7f795015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.430+0000 7f7955f7f640 1 --2- 172.21.15.59:0/825139294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79500b40a0 0x7f795015e8d0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f7944009830 tx=0x7f794402a500 comp rx=0 tx=0).stop 2024-06-29T15:58:06.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.430+0000 7f7955f7f640 1 -- 172.21.15.59:0/825139294 shutdown_connections 2024-06-29T15:58:06.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.430+0000 7f7955f7f640 1 --2- 172.21.15.59:0/825139294 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f795015ee10 0x7f795015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.430+0000 7f7955f7f640 1 --2- 172.21.15.59:0/825139294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79500b40a0 0x7f795015e8d0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.431+0000 7f7955f7f640 1 -- 172.21.15.59:0/825139294 >> 172.21.15.59:0/825139294 conn(0x7f79500d3ee0 msgr2=0x7f79500d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.431+0000 7f7955f7f640 1 -- 172.21.15.59:0/825139294 shutdown_connections 2024-06-29T15:58:06.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.431+0000 7f7955f7f640 1 -- 172.21.15.59:0/825139294 wait complete. 2024-06-29T15:58:06.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.432+0000 7f7955f7f640 1 Processor -- start 2024-06-29T15:58:06.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.433+0000 7f7955f7f640 1 -- start start 2024-06-29T15:58:06.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f7955f7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79500d23c0 0x7f795014f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f7955f7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f795015ee10 0x7f795015d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f7955f7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f795015d9a0 0x7f79501608d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f7955f7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7950160280 con 0x7f795015d9a0 2024-06-29T15:58:06.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f7955f7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7950160100 con 0x7f79500d23c0 2024-06-29T15:58:06.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f7955f7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7950160400 con 0x7f795015ee10 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f795015d9a0 0x7f79501608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f795015d9a0 0x7f79501608d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56700/0 (socket says 172.21.15.59:56700) 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794ffff640 1 -- 172.21.15.59:0/273706106 learned_addr learned my addr 172.21.15.59:0/273706106 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794f7fe640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79500d23c0 0x7f795014f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794effd640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f795015ee10 0x7f795015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794ffff640 1 -- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f795015ee10 msgr2=0x7f795015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794ffff640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f795015ee10 0x7f795015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794ffff640 1 -- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79500d23c0 msgr2=0x7f795014f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794ffff640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79500d23c0 0x7f795014f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.434+0000 7f794ffff640 1 -- 172.21.15.59:0/273706106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79500c7840 con 0x7f795015d9a0 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.435+0000 7f794ffff640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f795015d9a0 0x7f79501608d0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f794800e990 tx=0x7f794800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.435+0000 7f7937fff640 1 -- 172.21.15.59:0/273706106 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948004d20 con 0x7f795015d9a0 2024-06-29T15:58:06.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.435+0000 7f7937fff640 1 -- 172.21.15.59:0/273706106 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f794800caa0 con 0x7f795015d9a0 2024-06-29T15:58:06.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.435+0000 7f7937fff640 1 -- 172.21.15.59:0/273706106 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948005020 con 0x7f795015d9a0 2024-06-29T15:58:06.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.435+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79500c7b20 con 0x7f795015d9a0 2024-06-29T15:58:06.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.435+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79500c80e0 con 0x7f795015d9a0 2024-06-29T15:58:06.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.436+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f795015e8d0 con 0x7f795015d9a0 2024-06-29T15:58:06.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.437+0000 7f7937fff640 1 -- 172.21.15.59:0/273706106 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f79480051c0 con 0x7f795015d9a0 2024-06-29T15:58:06.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.437+0000 7f7937fff640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f793c076510 0x7f793c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.437+0000 7f7937fff640 1 -- 172.21.15.59:0/273706106 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7f7948098b70 con 0x7f795015d9a0 2024-06-29T15:58:06.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.437+0000 7f794f7fe640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f793c076510 0x7f793c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.438+0000 7f794f7fe640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f793c076510 0x7f793c078a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7940007fd0 tx=0x7f7940009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.442+0000 7f7937fff640 1 -- 172.21.15.59:0/273706106 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7948016030 con 0x7f795015d9a0 2024-06-29T15:58:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.634+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79500d2950 con 0x7f795015d9a0 2024-06-29T15:58:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.635+0000 7f7937fff640 1 -- 172.21.15.59:0/273706106 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v231) ==== 93+0+10 (secure 0 0 0) 0x7f7948061a80 con 0x7f795015d9a0 2024-06-29T15:58:06.635 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f793c076510 msgr2=0x7f793c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f793c076510 0x7f793c078a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7940007fd0 tx=0x7f7940009040 comp rx=0 tx=0).stop 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f795015d9a0 msgr2=0x7f79501608d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f795015d9a0 0x7f79501608d0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f794800e990 tx=0x7f794800ee50 comp rx=0 tx=0).stop 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 shutdown_connections 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f793c076510 0x7f793c078a20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f795015d9a0 0x7f79501608d0 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f795015ee10 0x7f795015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 --2- 172.21.15.59:0/273706106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79500d23c0 0x7f795014f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 >> 172.21.15.59:0/273706106 conn(0x7f79500d3ee0 msgr2=0x7f79500d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 shutdown_connections 2024-06-29T15:58:06.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.637+0000 7f7955f7f640 1 -- 172.21.15.59:0/273706106 wait complete. 2024-06-29T15:58:06.650 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:06.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.770+0000 7fb4425f0640 1 -- 172.21.15.59:0/2881798813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 msgr2=0x7fb43c0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.770+0000 7fb4425f0640 1 --2- 172.21.15.59:0/2881798813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 0x7fb43c0b3990 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fb42c005fd0 tx=0x7fb42c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.770+0000 7fb4425f0640 1 -- 172.21.15.59:0/2881798813 shutdown_connections 2024-06-29T15:58:06.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.770+0000 7fb4425f0640 1 --2- 172.21.15.59:0/2881798813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 0x7fb43c0b3990 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.770+0000 7fb4425f0640 1 -- 172.21.15.59:0/2881798813 >> 172.21.15.59:0/2881798813 conn(0x7fb43c0d3ee0 msgr2=0x7fb43c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.770+0000 7fb4425f0640 1 -- 172.21.15.59:0/2881798813 shutdown_connections 2024-06-29T15:58:06.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.770+0000 7fb4425f0640 1 -- 172.21.15.59:0/2881798813 wait complete. 2024-06-29T15:58:06.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.772+0000 7fb4425f0640 1 Processor -- start 2024-06-29T15:58:06.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.772+0000 7fb4425f0640 1 -- start start 2024-06-29T15:58:06.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb4425f0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb43c0b3ed0 0x7fb43c0cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb4425f0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 0x7fb43c1523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb4425f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb43c15efe0 0x7fb43c152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb4425f0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c160450 con 0x7fb43c0d23c0 2024-06-29T15:58:06.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb4425f0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c1602d0 con 0x7fb43c0b3ed0 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb4425f0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb43c1605d0 con 0x7fb43c15efe0 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 0x7fb43c1523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 0x7fb43c1523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56722/0 (socket says 172.21.15.59:56722) 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 -- 172.21.15.59:0/567041944 learned_addr learned my addr 172.21.15.59:0/567041944 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43bfff640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb43c0b3ed0 0x7fb43c0cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb440b8d640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb43c15efe0 0x7fb43c152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 -- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb43c15efe0 msgr2=0x7fb43c152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb43c15efe0 0x7fb43c152920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 -- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb43c0b3ed0 msgr2=0x7fb43c0cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb43c0b3ed0 0x7fb43c0cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 -- 172.21.15.59:0/567041944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c1400f0 con 0x7fb43c0d23c0 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43bfff640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb43c0b3ed0 0x7fb43c0cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:06.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb43b7fe640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 0x7fb43c1523e0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fb43000c390 tx=0x7fb43000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.774+0000 7fb438ff9640 1 -- 172.21.15.59:0/567041944 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430017070 con 0x7fb43c0d23c0 2024-06-29T15:58:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.775+0000 7fb438ff9640 1 -- 172.21.15.59:0/567041944 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4300035f0 con 0x7fb43c0d23c0 2024-06-29T15:58:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.775+0000 7fb438ff9640 1 -- 172.21.15.59:0/567041944 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430003a20 con 0x7fb43c0d23c0 2024-06-29T15:58:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.775+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43c1403d0 con 0x7fb43c0d23c0 2024-06-29T15:58:06.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.775+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43c140990 con 0x7fb43c0d23c0 2024-06-29T15:58:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.776+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43c13fe60 con 0x7fb43c0d23c0 2024-06-29T15:58:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.776+0000 7fb438ff9640 1 -- 172.21.15.59:0/567041944 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fb43001a070 con 0x7fb43c0d23c0 2024-06-29T15:58:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.777+0000 7fb438ff9640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb428076540 0x7fb428078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.777+0000 7fb438ff9640 1 -- 172.21.15.59:0/567041944 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7fb430097490 con 0x7fb43c0d23c0 2024-06-29T15:58:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.777+0000 7fb43bfff640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb428076540 0x7fb428078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.778+0000 7fb43bfff640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb428076540 0x7fb428078a50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb42c033000 tx=0x7fb42c004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.781+0000 7fb438ff9640 1 -- 172.21.15.59:0/567041944 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb43c0b42e0 con 0x7fb43c0d23c0 2024-06-29T15:58:06.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.972+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7fb43c0b3990 con 0x7fb43c0d23c0 2024-06-29T15:58:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.972+0000 7fb438ff9640 1 -- 172.21.15.59:0/567041944 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v231) ==== 107+0+11 (secure 0 0 0) 0x7fb4300604b0 con 0x7fb43c0d23c0 2024-06-29T15:58:06.972 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:58:06.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb428076540 msgr2=0x7fb428078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb428076540 0x7fb428078a50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb42c033000 tx=0x7fb42c004730 comp rx=0 tx=0).stop 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 msgr2=0x7fb43c1523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 0x7fb43c1523e0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fb43000c390 tx=0x7fb43000c850 comp rx=0 tx=0).stop 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 shutdown_connections 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb428076540 0x7fb428078a50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb43c15efe0 0x7fb43c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb43c0d23c0 0x7fb43c1523e0 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 --2- 172.21.15.59:0/567041944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb43c0b3ed0 0x7fb43c0cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 >> 172.21.15.59:0/567041944 conn(0x7fb43c0d3ee0 msgr2=0x7fb43c06d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 shutdown_connections 2024-06-29T15:58:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:06.974+0000 7fb4425f0640 1 -- 172.21.15.59:0/567041944 wait complete. 2024-06-29T15:58:06.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.107+0000 7f29ce37a640 1 -- 172.21.15.59:0/1030087203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29b0003ac0 msgr2=0x7f29b0003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.107+0000 7f29ce37a640 1 --2- 172.21.15.59:0/1030087203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29b0003ac0 0x7f29b0003f30 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009b50 tx=0x7f29b0005830 comp rx=0 tx=0).stop 2024-06-29T15:58:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.107+0000 7f29ce37a640 1 -- 172.21.15.59:0/1030087203 shutdown_connections 2024-06-29T15:58:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.107+0000 7f29ce37a640 1 --2- 172.21.15.59:0/1030087203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29b0003ac0 0x7f29b0003f30 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.107+0000 7f29ce37a640 1 -- 172.21.15.59:0/1030087203 >> 172.21.15.59:0/1030087203 conn(0x7f29c80b24c0 msgr2=0x7f29c80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.107+0000 7f29ce37a640 1 -- 172.21.15.59:0/1030087203 shutdown_connections 2024-06-29T15:58:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.107+0000 7f29ce37a640 1 -- 172.21.15.59:0/1030087203 wait complete. 2024-06-29T15:58:07.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.109+0000 7f29ce37a640 1 Processor -- start 2024-06-29T15:58:07.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.109+0000 7f29ce37a640 1 -- start start 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29ce37a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b0003ac0 0x7f29c80b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29ce37a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c813d3a0 0x7f29c81f8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29ce37a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c81f8e40 0x7f29c81404e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29ce37a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f29c80cd250 con 0x7f29c81f8e40 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29ce37a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29c80cd0d0 con 0x7f29c813d3a0 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29ce37a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f29c80cd3d0 con 0x7f29b0003ac0 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29cc917640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c81f8e40 0x7f29c81404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29cc917640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c81f8e40 0x7f29c81404e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56736/0 (socket says 172.21.15.59:56736) 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29cc917640 1 -- 172.21.15.59:0/190968024 learned_addr learned my addr 172.21.15.59:0/190968024 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:07.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29c77fe640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c813d3a0 0x7f29c81f8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.110+0000 7f29c7fff640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b0003ac0 0x7f29c80b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29cc917640 1 -- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b0003ac0 msgr2=0x7f29c80b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29cc917640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b0003ac0 0x7f29c80b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29cc917640 1 -- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c813d3a0 msgr2=0x7f29c81f8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29cc917640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c813d3a0 0x7f29c81f8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29cc917640 1 -- 172.21.15.59:0/190968024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c81f9250 con 0x7f29c81f8e40 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29cc917640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c81f8e40 0x7f29c81404e0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f29c80b4ff0 tx=0x7f29c00019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29c4ff9640 1 -- 172.21.15.59:0/190968024 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0011870 con 0x7f29c81f8e40 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29c4ff9640 1 -- 172.21.15.59:0/190968024 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29c0017070 con 0x7f29c81f8e40 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c80b47e0 con 0x7f29c81f8e40 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29c4ff9640 1 -- 172.21.15.59:0/190968024 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0011dc0 con 0x7f29c81f8e40 2024-06-29T15:58:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.111+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c80adb00 con 0x7f29c81f8e40 2024-06-29T15:58:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.112+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c806c6d0 con 0x7f29c81f8e40 2024-06-29T15:58:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.113+0000 7f29c4ff9640 1 -- 172.21.15.59:0/190968024 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f29c001c460 con 0x7f29c81f8e40 2024-06-29T15:58:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.113+0000 7f29c4ff9640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29b4076510 0x7f29b4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.113+0000 7f29c4ff9640 1 -- 172.21.15.59:0/190968024 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7f29c00989c0 con 0x7f29c81f8e40 2024-06-29T15:58:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.113+0000 7f29c7fff640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29b4076510 0x7f29b4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.114+0000 7f29c7fff640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29b4076510 0x7f29b4078a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29b8035000 tx=0x7f29b80046c0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.118+0000 7f29c4ff9640 1 -- 172.21.15.59:0/190968024 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f29c009ae20 con 0x7f29c81f8e40 2024-06-29T15:58:07.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.313+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f29c8152280 con 0x7f29c81f8e40 2024-06-29T15:58:07.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.314+0000 7f29c4ff9640 1 -- 172.21.15.59:0/190968024 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v231) ==== 103+0+11 (secure 0 0 0) 0x7f29c81f9250 con 0x7f29c81f8e40 2024-06-29T15:58:07.314 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29b4076510 msgr2=0x7f29b4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29b4076510 0x7f29b4078a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29b8035000 tx=0x7f29b80046c0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c81f8e40 msgr2=0x7f29c81404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c81f8e40 0x7f29c81404e0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f29c80b4ff0 tx=0x7f29c00019b0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 shutdown_connections 2024-06-29T15:58:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29b4076510 0x7f29b4078a20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c81f8e40 0x7f29c81404e0 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c813d3a0 0x7f29c81f8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 --2- 172.21.15.59:0/190968024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29b0003ac0 0x7f29c80b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 >> 172.21.15.59:0/190968024 conn(0x7f29c80b24c0 msgr2=0x7f29c815f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 shutdown_connections 2024-06-29T15:58:07.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.316+0000 7f29ce37a640 1 -- 172.21.15.59:0/190968024 wait complete. 2024-06-29T15:58:07.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:07.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.445+0000 7fd8611b2640 1 -- 172.21.15.59:0/514039601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd848004800 msgr2=0x7fd848004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.445+0000 7fd8611b2640 1 --2- 172.21.15.59:0/514039601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd848004800 0x7fd848004c70 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0b7620 tx=0x7fd848005830 comp rx=0 tx=0).stop 2024-06-29T15:58:07.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.445+0000 7fd85b59e640 1 -- 172.21.15.59:0/514039601 reap_dead start 2024-06-29T15:58:07.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.445+0000 7fd8611b2640 1 -- 172.21.15.59:0/514039601 shutdown_connections 2024-06-29T15:58:07.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.445+0000 7fd8611b2640 1 -- 172.21.15.59:0/514039601 >> 172.21.15.59:0/514039601 conn(0x7fd85c0b5610 msgr2=0x7fd85c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:07.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.446+0000 7fd8611b2640 1 -- 172.21.15.59:0/514039601 shutdown_connections 2024-06-29T15:58:07.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.446+0000 7fd8611b2640 1 -- 172.21.15.59:0/514039601 wait complete. 2024-06-29T15:58:07.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.448+0000 7fd8611b2640 1 Processor -- start 2024-06-29T15:58:07.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.448+0000 7fd8611b2640 1 -- start start 2024-06-29T15:58:07.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd8611b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd848004800 0x7fd85c157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd8611b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd85c14a380 0x7fd85c15d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd8611b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd85c1583f0 0x7fd85c15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd8611b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c162e70 con 0x7fd85c14a380 2024-06-29T15:58:07.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd8611b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c162cf0 con 0x7fd848004800 2024-06-29T15:58:07.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd8611b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd85c162ff0 con 0x7fd85c1583f0 2024-06-29T15:58:07.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd85c14a380 0x7fd85c15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd85c14a380 0x7fd85c15d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56756/0 (socket says 172.21.15.59:56756) 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85a59c640 1 -- 172.21.15.59:0/3181255556 learned_addr learned my addr 172.21.15.59:0/3181255556 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85ad9d640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd848004800 0x7fd85c157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85b59e640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd85c1583f0 0x7fd85c15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85a59c640 1 -- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd85c1583f0 msgr2=0x7fd85c15ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85a59c640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd85c1583f0 0x7fd85c15ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85a59c640 1 -- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd848004800 msgr2=0x7fd85c157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85a59c640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd848004800 0x7fd85c157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.449+0000 7fd85a59c640 1 -- 172.21.15.59:0/3181255556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85c0cd030 con 0x7fd85c14a380 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.450+0000 7fd85a59c640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd85c14a380 0x7fd85c15d990 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7fd84400c3d0 tx=0x7fd844006c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.450+0000 7fd8537fe640 1 -- 172.21.15.59:0/3181255556 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd844003a20 con 0x7fd85c14a380 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.450+0000 7fd8537fe640 1 -- 172.21.15.59:0/3181255556 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd844003cf0 con 0x7fd85c14a380 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.450+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd85c0cd310 con 0x7fd85c14a380 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.450+0000 7fd8537fe640 1 -- 172.21.15.59:0/3181255556 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84402e6a0 con 0x7fd85c14a380 2024-06-29T15:58:07.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.450+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd85c0cd8d0 con 0x7fd85c14a380 2024-06-29T15:58:07.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.451+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd85c0ccda0 con 0x7fd85c14a380 2024-06-29T15:58:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.452+0000 7fd8537fe640 1 -- 172.21.15.59:0/3181255556 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fd844035020 con 0x7fd85c14a380 2024-06-29T15:58:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.452+0000 7fd8537fe640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd83c076510 0x7fd83c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.452+0000 7fd8537fe640 1 -- 172.21.15.59:0/3181255556 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7fd8440b8420 con 0x7fd85c14a380 2024-06-29T15:58:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.453+0000 7fd85ad9d640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd83c076510 0x7fd83c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.453+0000 7fd85ad9d640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd83c076510 0x7fd83c078a20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd84c000f80 tx=0x7fd84c006e90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:07.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.457+0000 7fd8537fe640 1 -- 172.21.15.59:0/3181255556 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd844047030 con 0x7fd85c14a380 2024-06-29T15:58:07.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.687+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd85c0cf080 con 0x7fd85c14a380 2024-06-29T15:58:07.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.688+0000 7fd8537fe640 1 -- 172.21.15.59:0/3181255556 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5245 (secure 0 0 0) 0x7fd84403e070 con 0x7fd85c14a380 2024-06-29T15:58:07.688 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:07.688 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":123,"btime":"2024-06-29T15:58:03:919935+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":12132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2086282920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2086282920},{"type":"v1","addr":"172.21.15.59:6834","nonce":2086282920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:58:07.688 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 123 2024-06-29T15:58:07.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd83c076510 msgr2=0x7fd83c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd83c076510 0x7fd83c078a20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd84c000f80 tx=0x7fd84c006e90 comp rx=0 tx=0).stop 2024-06-29T15:58:07.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd85c14a380 msgr2=0x7fd85c15d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd85c14a380 0x7fd85c15d990 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7fd84400c3d0 tx=0x7fd844006c70 comp rx=0 tx=0).stop 2024-06-29T15:58:07.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 shutdown_connections 2024-06-29T15:58:07.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd83c076510 0x7fd83c078a20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd85c1583f0 0x7fd85c15ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd85c14a380 0x7fd85c15d990 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 --2- 172.21.15.59:0/3181255556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd848004800 0x7fd85c157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 >> 172.21.15.59:0/3181255556 conn(0x7fd85c0b5610 msgr2=0x7fd85c06d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:07.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 shutdown_connections 2024-06-29T15:58:07.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.690+0000 7fd8611b2640 1 -- 172.21.15.59:0/3181255556 wait complete. 2024-06-29T15:58:07.701 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:07.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.818+0000 7f11ed0ac640 1 -- 172.21.15.59:0/2745664487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 msgr2=0x7f11d4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.818+0000 7f11ed0ac640 1 --2- 172.21.15.59:0/2745664487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 0x7f11d4004800 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009b50 tx=0x7f11d4005830 comp rx=0 tx=0).stop 2024-06-29T15:58:07.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.818+0000 7f11e759e640 1 -- 172.21.15.59:0/2745664487 reap_dead start 2024-06-29T15:58:07.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.819+0000 7f11ed0ac640 1 -- 172.21.15.59:0/2745664487 shutdown_connections 2024-06-29T15:58:07.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.819+0000 7f11ed0ac640 1 -- 172.21.15.59:0/2745664487 >> 172.21.15.59:0/2745664487 conn(0x7f11e80b24c0 msgr2=0x7f11e80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:07.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.819+0000 7f11ed0ac640 1 -- 172.21.15.59:0/2745664487 shutdown_connections 2024-06-29T15:58:07.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.819+0000 7f11ed0ac640 1 -- 172.21.15.59:0/2745664487 wait complete. 2024-06-29T15:58:07.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.821+0000 7f11ed0ac640 1 Processor -- start 2024-06-29T15:58:07.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.821+0000 7f11ed0ac640 1 -- start start 2024-06-29T15:58:07.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11ed0ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 0x7f11e81594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11ed0ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e80adbc0 0x7f11e80adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11ed0ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81e4eb0 0x7f11e81e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11ed0ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f11e80733f0 con 0x7f11d4004040 2024-06-29T15:58:07.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11ed0ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8073270 con 0x7f11e81e4eb0 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11ed0ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11e8073570 con 0x7f11e80adbc0 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11e6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 0x7f11e81594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11e6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 0x7f11e81594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56774/0 (socket says 172.21.15.59:56774) 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11e6d9d640 1 -- 172.21.15.59:0/1633421407 learned_addr learned my addr 172.21.15.59:0/1633421407 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11e659c640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e80adbc0 0x7f11e80adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.822+0000 7f11e759e640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81e4eb0 0x7f11e81e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11e6d9d640 1 -- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e80adbc0 msgr2=0x7f11e80adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11e6d9d640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e80adbc0 0x7f11e80adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11e6d9d640 1 -- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81e4eb0 msgr2=0x7f11e81e52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11e6d9d640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81e4eb0 0x7f11e81e52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11e6d9d640 1 -- 172.21.15.59:0/1633421407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e81e5800 con 0x7f11d4004040 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11e759e640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81e4eb0 0x7f11e81e52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11e659c640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e80adbc0 0x7f11e80adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11e6d9d640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 0x7f11e81594d0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f11d8006340 tx=0x7f11d8006740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:07.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11df7fe640 1 -- 172.21.15.59:0/1633421407 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d800fb40 con 0x7f11d4004040 2024-06-29T15:58:07.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e80ae8f0 con 0x7f11d4004040 2024-06-29T15:58:07.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11df7fe640 1 -- 172.21.15.59:0/1633421407 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11d8016020 con 0x7f11d4004040 2024-06-29T15:58:07.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11df7fe640 1 -- 172.21.15.59:0/1633421407 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8011430 con 0x7f11d4004040 2024-06-29T15:58:07.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.823+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e80aeeb0 con 0x7f11d4004040 2024-06-29T15:58:07.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.824+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e815f240 con 0x7f11d4004040 2024-06-29T15:58:07.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.825+0000 7f11df7fe640 1 -- 172.21.15.59:0/1633421407 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f11d80115d0 con 0x7f11d4004040 2024-06-29T15:58:07.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.825+0000 7f11df7fe640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f11c8076540 0x7f11c8078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.825+0000 7f11df7fe640 1 -- 172.21.15.59:0/1633421407 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7f11d8099070 con 0x7f11d4004040 2024-06-29T15:58:07.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.826+0000 7f11e659c640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f11c8076540 0x7f11c8078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.826+0000 7f11e659c640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f11c8076540 0x7f11c8078a50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f11d0035000 tx=0x7f11d00046c0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:07.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:07.830+0000 7f11df7fe640 1 -- 172.21.15.59:0/1633421407 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11d8020020 con 0x7f11d4004040 2024-06-29T15:58:08.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.025+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11e815cd40 con 0x7f11d4004040 2024-06-29T15:58:08.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.026+0000 7f11df7fe640 1 -- 172.21.15.59:0/1633421407 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) ==== 74+0+18330 (secure 0 0 0) 0x7f11e81e5800 con 0x7f11d4004040 2024-06-29T15:58:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:08.027 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":231,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:05.928387+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6832/1734341893":"2024-06-30T15:58:02.917822+0000","172.21.15.153:0/1969822965":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/813560021":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2732252046":"2024-06-30T15:57:59.178318+0000","172.21.15.59:6834/1734341893":"2024-06-30T15:58:02.917822+0000","172.21.15.153:0/240322135":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/215704132":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/1238730780":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/3658145560":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2050849752":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/239127507":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2051017624":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/743055221":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/4093149557":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/572137829":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2642354773":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2737762390":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2780049427":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2780049427":"2024-06-30T15:57:50.460403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:08.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f11c8076540 msgr2=0x7f11c8078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f11c8076540 0x7f11c8078a50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f11d0035000 tx=0x7f11d00046c0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 msgr2=0x7f11e81594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 0x7f11e81594d0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f11d8006340 tx=0x7f11d8006740 comp rx=0 tx=0).stop 2024-06-29T15:58:08.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 shutdown_connections 2024-06-29T15:58:08.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f11c8076540 0x7f11c8078a50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e81e4eb0 0x7f11e81e52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11e80adbc0 0x7f11e80adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 --2- 172.21.15.59:0/1633421407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11d4004040 0x7f11e81594d0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 >> 172.21.15.59:0/1633421407 conn(0x7f11e80b24c0 msgr2=0x7f11e815a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:08.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 shutdown_connections 2024-06-29T15:58:08.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.029+0000 7f11ed0ac640 1 -- 172.21.15.59:0/1633421407 wait complete. 2024-06-29T15:58:08.040 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-06-29T15:58:08.040 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:08.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.159+0000 7f2fdbc64640 1 -- 172.21.15.59:0/236494011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fd415ceb0 msgr2=0x7f2fd40b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.159+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/236494011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fd415ceb0 0x7f2fd40b56d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc005fd0 tx=0x7f2fcc02aa10 comp rx=0 tx=0).stop 2024-06-29T15:58:08.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.159+0000 7f2fdbc64640 1 -- 172.21.15.59:0/236494011 shutdown_connections 2024-06-29T15:58:08.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.159+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/236494011 >> v2:172.21.15.153:3300/0 conn(0x7f2fd4141bd0 0x7f2fd413edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.159+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/236494011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fd415ceb0 0x7f2fd40b56d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.159+0000 7f2fdbc64640 1 -- 172.21.15.59:0/236494011 >> 172.21.15.59:0/236494011 conn(0x7f2fd40b6fb0 msgr2=0x7f2fd40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:08.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.159+0000 7f2fdbc64640 1 -- 172.21.15.59:0/236494011 shutdown_connections 2024-06-29T15:58:08.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.159+0000 7f2fdbc64640 1 -- 172.21.15.59:0/236494011 wait complete. 2024-06-29T15:58:08.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.161+0000 7f2fdbc64640 1 Processor -- start 2024-06-29T15:58:08.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.161+0000 7f2fdbc64640 1 -- start start 2024-06-29T15:58:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fdbc64640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fd413e9b0 0x7f2fd41518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fdbc64640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fd415ceb0 0x7f2fd4154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fdbc64640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fd41314f0 0x7f2fd4154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fdbc64640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd40b2a80 con 0x7f2fd413e9b0 2024-06-29T15:58:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fdbc64640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd40b2900 con 0x7f2fd41314f0 2024-06-29T15:58:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fdbc64640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2fd40b2c00 con 0x7f2fd415ceb0 2024-06-29T15:58:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fd9a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fd413e9b0 0x7f2fd41518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fd9a00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fd413e9b0 0x7f2fd41518f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56778/0 (socket says 172.21.15.59:56778) 2024-06-29T15:58:08.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fd9a00640 1 -- 172.21.15.59:0/3962577199 learned_addr learned my addr 172.21.15.59:0/3962577199 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:08.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fd91ff640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fd415ceb0 0x7f2fd4154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.162+0000 7f2fda201640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fd41314f0 0x7f2fd4154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd9a00640 1 -- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fd415ceb0 msgr2=0x7f2fd4154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd9a00640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fd415ceb0 0x7f2fd4154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd9a00640 1 -- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fd41314f0 msgr2=0x7f2fd4154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd9a00640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fd41314f0 0x7f2fd4154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd9a00640 1 -- 172.21.15.59:0/3962577199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd412ec00 con 0x7f2fd413e9b0 2024-06-29T15:58:08.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd9a00640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fd413e9b0 0x7f2fd41518f0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc006010 tx=0x7f2fcc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:08.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd27fc640 1 -- 172.21.15.59:0/3962577199 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc033050 con 0x7f2fd413e9b0 2024-06-29T15:58:08.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd27fc640 1 -- 172.21.15.59:0/3962577199 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fcc004250 con 0x7f2fd413e9b0 2024-06-29T15:58:08.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd412eee0 con 0x7f2fd413e9b0 2024-06-29T15:58:08.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fd27fc640 1 -- 172.21.15.59:0/3962577199 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc004680 con 0x7f2fd413e9b0 2024-06-29T15:58:08.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.163+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd412f4a0 con 0x7f2fd413e9b0 2024-06-29T15:58:08.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.164+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd413edc0 con 0x7f2fd413e9b0 2024-06-29T15:58:08.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.165+0000 7f2fd27fc640 1 -- 172.21.15.59:0/3962577199 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f2fcc036070 con 0x7f2fd413e9b0 2024-06-29T15:58:08.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.165+0000 7f2fd27fc640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2fc0076510 0x7f2fc0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.165+0000 7f2fd27fc640 1 -- 172.21.15.59:0/3962577199 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7f2fcc0b9220 con 0x7f2fd413e9b0 2024-06-29T15:58:08.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.165+0000 7f2fd91ff640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2fc0076510 0x7f2fc0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.166+0000 7f2fd91ff640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2fc0076510 0x7f2fc0078a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4131b60 tx=0x7f2fc8009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:08.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.170+0000 7f2fd27fc640 1 -- 172.21.15.59:0/3962577199 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2fcc004820 con 0x7f2fd413e9b0 2024-06-29T15:58:08.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.398+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fd40cf690 con 0x7f2fd413e9b0 2024-06-29T15:58:08.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.398+0000 7f2fd27fc640 1 -- 172.21.15.59:0/3962577199 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5245 (secure 0 0 0) 0x7f2fcc081af0 con 0x7f2fd413e9b0 2024-06-29T15:58:08.398 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:08.399 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":123,"btime":"2024-06-29T15:58:03:919935+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":12132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2086282920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2086282920},{"type":"v1","addr":"172.21.15.59:6834","nonce":2086282920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:58:08.399 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 123 2024-06-29T15:58:08.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2fc0076510 msgr2=0x7f2fc0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2fc0076510 0x7f2fc0078a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4131b60 tx=0x7f2fc8009040 comp rx=0 tx=0).stop 2024-06-29T15:58:08.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fd413e9b0 msgr2=0x7f2fd41518f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fd413e9b0 0x7f2fd41518f0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc006010 tx=0x7f2fcc003f10 comp rx=0 tx=0).stop 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 shutdown_connections 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2fc0076510 0x7f2fc0078a20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2fd41314f0 0x7f2fd4154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2fd415ceb0 0x7f2fd4154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 --2- 172.21.15.59:0/3962577199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fd413e9b0 0x7f2fd41518f0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 >> 172.21.15.59:0/3962577199 conn(0x7f2fd40b6fb0 msgr2=0x7f2fd415d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 shutdown_connections 2024-06-29T15:58:08.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.400+0000 7f2fdbc64640 1 -- 172.21.15.59:0/3962577199 wait complete. 2024-06-29T15:58:08.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:08.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 -- 172.21.15.59:0/2425767362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 msgr2=0x7f79b413edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/2425767362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 0x7f79b413edc0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f79b4073f30 tx=0x7f79a802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 -- 172.21.15.59:0/2425767362 shutdown_connections 2024-06-29T15:58:08.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/2425767362 >> v2:172.21.15.153:3300/0 conn(0x7f79b41414c0 0x7f79b41418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/2425767362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 0x7f79b413edc0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/2425767362 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b415ceb0 0x7f79b40b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 -- 172.21.15.59:0/2425767362 >> 172.21.15.59:0/2425767362 conn(0x7f79b40b6fb0 msgr2=0x7f79b40b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:08.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 -- 172.21.15.59:0/2425767362 shutdown_connections 2024-06-29T15:58:08.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.529+0000 7f79ba3d2640 1 -- 172.21.15.59:0/2425767362 wait complete. 2024-06-29T15:58:08.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.531+0000 7f79ba3d2640 1 Processor -- start 2024-06-29T15:58:08.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.531+0000 7f79ba3d2640 1 -- start start 2024-06-29T15:58:08.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.532+0000 7f79ba3d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 0x7f79b4152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.532+0000 7f79ba3d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b41414c0 0x7f79b406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.532+0000 7f79ba3d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b415ceb0 0x7f79b4158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.532+0000 7f79ba3d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f79b40b28c0 con 0x7f79b413e9b0 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.532+0000 7f79ba3d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79b40b2740 con 0x7f79b415ceb0 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.532+0000 7f79ba3d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f79b40b2a40 con 0x7f79b41414c0 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 0x7f79b4152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 0x7f79b4152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56794/0 (socket says 172.21.15.59:56794) 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 -- 172.21.15.59:0/1289722562 learned_addr learned my addr 172.21.15.59:0/1289722562 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b37fe640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b41414c0 0x7f79b406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b896f640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b415ceb0 0x7f79b4158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 -- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b41414c0 msgr2=0x7f79b406ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b41414c0 0x7f79b406ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 -- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b415ceb0 msgr2=0x7f79b4158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b415ceb0 0x7f79b4158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 -- 172.21.15.59:0/1289722562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b406f700 con 0x7f79b413e9b0 2024-06-29T15:58:08.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b3fff640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 0x7f79b4152f10 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f79a400d700 tx=0x7f79a400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:08.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b0ff9640 1 -- 172.21.15.59:0/1289722562 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4017070 con 0x7f79b413e9b0 2024-06-29T15:58:08.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b0ff9640 1 -- 172.21.15.59:0/1289722562 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79a4005310 con 0x7f79b413e9b0 2024-06-29T15:58:08.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79b0ff9640 1 -- 172.21.15.59:0/1289722562 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4009820 con 0x7f79b413e9b0 2024-06-29T15:58:08.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b4157970 con 0x7f79b413e9b0 2024-06-29T15:58:08.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.533+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b4158990 con 0x7f79b413e9b0 2024-06-29T15:58:08.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.534+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b40b2f60 con 0x7f79b413e9b0 2024-06-29T15:58:08.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.535+0000 7f79b0ff9640 1 -- 172.21.15.59:0/1289722562 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f79a40099c0 con 0x7f79b413e9b0 2024-06-29T15:58:08.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.536+0000 7f79b0ff9640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f79a0076510 0x7f79a0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.536+0000 7f79b0ff9640 1 -- 172.21.15.59:0/1289722562 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7f79a40982d0 con 0x7f79b413e9b0 2024-06-29T15:58:08.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.536+0000 7f79b37fe640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f79a0076510 0x7f79a0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.536+0000 7f79b37fe640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f79a0076510 0x7f79a0078a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79a8033000 tx=0x7f79a8004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:08.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.540+0000 7f79b0ff9640 1 -- 172.21.15.59:0/1289722562 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f79a4060b10 con 0x7f79b413e9b0 2024-06-29T15:58:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.771+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79b40cdf60 con 0x7f79b413e9b0 2024-06-29T15:58:08.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.772+0000 7f79b0ff9640 1 -- 172.21.15.59:0/1289722562 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5245 (secure 0 0 0) 0x7f79a40691e0 con 0x7f79b413e9b0 2024-06-29T15:58:08.772 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:08.772 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":123,"btime":"2024-06-29T15:58:03:919935+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":12132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2086282920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2086282920},{"type":"v1","addr":"172.21.15.59:6834","nonce":2086282920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:34.679058+0000","modified":"2024-06-29T15:57:37.701097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10747":{"gid":10747,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6837/601671949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":601671949},{"type":"v1","addr":"172.21.15.153:6837","nonce":601671949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10747,"qdb_cluster":[10747]},"id":14}]} 2024-06-29T15:58:08.772 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 123 2024-06-29T15:58:08.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f79a0076510 msgr2=0x7f79a0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f79a0076510 0x7f79a0078a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79a8033000 tx=0x7f79a8004730 comp rx=0 tx=0).stop 2024-06-29T15:58:08.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 msgr2=0x7f79b4152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 0x7f79b4152f10 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f79a400d700 tx=0x7f79a400dbc0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 shutdown_connections 2024-06-29T15:58:08.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f79a0076510 0x7f79a0078a20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79b415ceb0 0x7f79b4158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f79b41414c0 0x7f79b406ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 --2- 172.21.15.59:0/1289722562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79b413e9b0 0x7f79b4152f10 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 >> 172.21.15.59:0/1289722562 conn(0x7f79b40b6fb0 msgr2=0x7f79b413e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:08.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 shutdown_connections 2024-06-29T15:58:08.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.774+0000 7f79ba3d2640 1 -- 172.21.15.59:0/1289722562 wait complete. 2024-06-29T15:58:08.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:08.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.903+0000 7ff510bba640 1 -- 172.21.15.59:0/2539238740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c162190 msgr2=0x7ff50c0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.903+0000 7ff510bba640 1 --2- 172.21.15.59:0/2539238740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c162190 0x7ff50c0b38e0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009830 tx=0x7ff4fc02a470 comp rx=0 tx=0).stop 2024-06-29T15:58:08.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.903+0000 7ff510bba640 1 -- 172.21.15.59:0/2539238740 shutdown_connections 2024-06-29T15:58:08.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.903+0000 7ff510bba640 1 --2- 172.21.15.59:0/2539238740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c162190 0x7ff50c0b38e0 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.903+0000 7ff510bba640 1 --2- 172.21.15.59:0/2539238740 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff50c162770 0x7ff50c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.903+0000 7ff510bba640 1 -- 172.21.15.59:0/2539238740 >> 172.21.15.59:0/2539238740 conn(0x7ff50c0d45b0 msgr2=0x7ff50c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.903+0000 7ff510bba640 1 -- 172.21.15.59:0/2539238740 shutdown_connections 2024-06-29T15:58:08.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.903+0000 7ff510bba640 1 -- 172.21.15.59:0/2539238740 wait complete. 2024-06-29T15:58:08.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.905+0000 7ff510bba640 1 Processor -- start 2024-06-29T15:58:08.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.905+0000 7ff510bba640 1 -- start start 2024-06-29T15:58:08.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff510bba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c15e0a0 0x7ff50c155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff510bba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff50c162770 0x7ff50c1f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff510bba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff50c1551c0 0x7ff50c1538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff510bba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c0b43a0 con 0x7ff50c15e0a0 2024-06-29T15:58:08.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff510bba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c0b4220 con 0x7ff50c162770 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff510bba640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff50c0b4520 con 0x7ff50c1551c0 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff50a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c15e0a0 0x7ff50c155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff50a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c15e0a0 0x7ff50c155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56810/0 (socket says 172.21.15.59:56810) 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.906+0000 7ff50a59c640 1 -- 172.21.15.59:0/1233134413 learned_addr learned my addr 172.21.15.59:0/1233134413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff50ad9d640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff50c1551c0 0x7ff50c1538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff509d9b640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff50c162770 0x7ff50c1f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff50a59c640 1 -- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff50c1551c0 msgr2=0x7ff50c1538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff50a59c640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff50c1551c0 0x7ff50c1538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff50a59c640 1 -- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff50c162770 msgr2=0x7ff50c1f3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff50a59c640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff50c162770 0x7ff50c1f3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff50a59c640 1 -- 172.21.15.59:0/1233134413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50c1f4660 con 0x7ff50c15e0a0 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff50a59c640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c15e0a0 0x7ff50c155830 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff50400ea50 tx=0x7ff50400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:08.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff502ffd640 1 -- 172.21.15.59:0/1233134413 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50400cd60 con 0x7ff50c15e0a0 2024-06-29T15:58:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff502ffd640 1 -- 172.21.15.59:0/1233134413 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff504011470 con 0x7ff50c15e0a0 2024-06-29T15:58:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50c1f4940 con 0x7ff50c15e0a0 2024-06-29T15:58:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff502ffd640 1 -- 172.21.15.59:0/1233134413 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5040118a0 con 0x7ff50c15e0a0 2024-06-29T15:58:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.907+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff50c1f4f00 con 0x7ff50c15e0a0 2024-06-29T15:58:08.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.908+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff50c0b49d0 con 0x7ff50c15e0a0 2024-06-29T15:58:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.909+0000 7ff502ffd640 1 -- 172.21.15.59:0/1233134413 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7ff504011a40 con 0x7ff50c15e0a0 2024-06-29T15:58:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.909+0000 7ff502ffd640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff4ec076540 0x7ff4ec078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.910+0000 7ff502ffd640 1 -- 172.21.15.59:0/1233134413 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7ff504098cf0 con 0x7ff50c15e0a0 2024-06-29T15:58:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.910+0000 7ff509d9b640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff4ec076540 0x7ff4ec078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.910+0000 7ff509d9b640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff4ec076540 0x7ff4ec078a50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4007fd0 tx=0x7ff4f4009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:08.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:08.914+0000 7ff502ffd640 1 -- 172.21.15.59:0/1233134413 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff50409d070 con 0x7ff50c15e0a0 2024-06-29T15:58:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.105+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff50c1e7a60 con 0x7ff50c15e0a0 2024-06-29T15:58:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.106+0000 7ff502ffd640 1 -- 172.21.15.59:0/1233134413 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) ==== 74+0+18330 (secure 0 0 0) 0x7ff504016030 con 0x7ff50c15e0a0 2024-06-29T15:58:09.106 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:09.107 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":231,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:05.928387+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.305142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"225","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:33.647895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6832/1734341893":"2024-06-30T15:58:02.917822+0000","172.21.15.153:0/1969822965":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/813560021":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2732252046":"2024-06-30T15:57:59.178318+0000","172.21.15.59:6834/1734341893":"2024-06-30T15:58:02.917822+0000","172.21.15.153:0/240322135":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/215704132":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/1238730780":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/3658145560":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2050849752":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/239127507":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2051017624":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/743055221":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/4093149557":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/572137829":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2642354773":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2737762390":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2780049427":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2780049427":"2024-06-30T15:57:50.460403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff4ec076540 msgr2=0x7ff4ec078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff4ec076540 0x7ff4ec078a50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4007fd0 tx=0x7ff4f4009040 comp rx=0 tx=0).stop 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c15e0a0 msgr2=0x7ff50c155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c15e0a0 0x7ff50c155830 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff50400ea50 tx=0x7ff50400ef10 comp rx=0 tx=0).stop 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 shutdown_connections 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff4ec076540 0x7ff4ec078a50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff50c1551c0 0x7ff50c1538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff50c162770 0x7ff50c1f3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 --2- 172.21.15.59:0/1233134413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff50c15e0a0 0x7ff50c155830 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 >> 172.21.15.59:0/1233134413 conn(0x7ff50c0d45b0 msgr2=0x7ff50c06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 shutdown_connections 2024-06-29T15:58:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.108+0000 7ff510bba640 1 -- 172.21.15.59:0/1233134413 wait complete. 2024-06-29T15:58:09.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-06-29T15:58:09.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.239+0000 7f355a09e640 1 -- 172.21.15.59:0/3516563593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 msgr2=0x7f353c004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.239+0000 7f355a09e640 1 --2- 172.21.15.59:0/3516563593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 0x7f353c004c70 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f35480017a0 tx=0x7f353c005830 comp rx=0 tx=0).stop 2024-06-29T15:58:09.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.239+0000 7f3553fff640 1 -- 172.21.15.59:0/3516563593 reap_dead start 2024-06-29T15:58:09.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.239+0000 7f355a09e640 1 -- 172.21.15.59:0/3516563593 shutdown_connections 2024-06-29T15:58:09.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.239+0000 7f355a09e640 1 -- 172.21.15.59:0/3516563593 >> 172.21.15.59:0/3516563593 conn(0x7f35540b6fb0 msgr2=0x7f35540b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:09.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.239+0000 7f355a09e640 1 -- 172.21.15.59:0/3516563593 shutdown_connections 2024-06-29T15:58:09.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.239+0000 7f355a09e640 1 -- 172.21.15.59:0/3516563593 wait complete. 2024-06-29T15:58:09.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.241+0000 7f355a09e640 1 Processor -- start 2024-06-29T15:58:09.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.241+0000 7f355a09e640 1 -- start start 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f355a09e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 0x7f35540b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f355a09e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f35541f5bf0 0x7f35541f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f355a09e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f355413d9a0 0x7f355413ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f355a09e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f35540b5850 con 0x7f353c004800 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f355a09e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f35540b56d0 con 0x7f355413d9a0 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f355a09e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f35540b59d0 con 0x7f35541f5bf0 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 0x7f35540b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 0x7f35540b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56826/0 (socket says 172.21.15.59:56826) 2024-06-29T15:58:09.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 -- 172.21.15.59:0/947625967 learned_addr learned my addr 172.21.15.59:0/947625967 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f3553fff640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f355413d9a0 0x7f355413ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f3552ffd640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f35541f5bf0 0x7f35541f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 -- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f35541f5bf0 msgr2=0x7f35541f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f35541f5bf0 0x7f35541f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 -- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f355413d9a0 msgr2=0x7f355413ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f355413d9a0 0x7f355413ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 -- 172.21.15.59:0/947625967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35541f6540 con 0x7f353c004800 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f3552ffd640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f35541f5bf0 0x7f35541f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.243+0000 7f35537fe640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 0x7f35540b2c90 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f3544011ab0 tx=0x7f3544011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.244+0000 7f353bfff640 1 -- 172.21.15.59:0/947625967 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354401e070 con 0x7f353c004800 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.244+0000 7f353bfff640 1 -- 172.21.15.59:0/947625967 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f354400fe70 con 0x7f353c004800 2024-06-29T15:58:09.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.244+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35540b31d0 con 0x7f353c004800 2024-06-29T15:58:09.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.244+0000 7f353bfff640 1 -- 172.21.15.59:0/947625967 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35440042b0 con 0x7f353c004800 2024-06-29T15:58:09.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.244+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35541506c0 con 0x7f353c004800 2024-06-29T15:58:09.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.245+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f355415f6c0 con 0x7f353c004800 2024-06-29T15:58:09.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.245+0000 7f353bfff640 1 -- 172.21.15.59:0/947625967 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f3544021070 con 0x7f353c004800 2024-06-29T15:58:09.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.246+0000 7f353bfff640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3540076540 0x7f3540078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.246+0000 7f353bfff640 1 -- 172.21.15.59:0/947625967 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 7072+0+0 (secure 0 0 0) 0x7f3544098bb0 con 0x7f353c004800 2024-06-29T15:58:09.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.246+0000 7f3552ffd640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3540076540 0x7f3540078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.247+0000 7f3552ffd640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3540076540 0x7f3540078a50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3548032000 tx=0x7f3548001360 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:09.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.250+0000 7f353bfff640 1 -- 172.21.15.59:0/947625967 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f355415f6c0 con 0x7f353c004800 2024-06-29T15:58:09.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.479+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) -- 0x7f355413f060 con 0x7f353c004800 2024-06-29T15:58:09.479 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:58:09.479+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:58:09.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.481+0000 7f353bfff640 1 -- 172.21.15.59:0/947625967 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v124) ==== 167+0+0 (secure 0 0 0) 0x7f35541f6540 con 0x7f353c004800 2024-06-29T15:58:09.481 INFO:teuthology.orchestra.run.smithi059.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3540076540 msgr2=0x7f3540078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3540076540 0x7f3540078a50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3548032000 tx=0x7f3548001360 comp rx=0 tx=0).stop 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 msgr2=0x7f35540b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 0x7f35540b2c90 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f3544011ab0 tx=0x7f3544011f70 comp rx=0 tx=0).stop 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 shutdown_connections 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3540076540 0x7f3540078a50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f355413d9a0 0x7f355413ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f35541f5bf0 0x7f35541f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 --2- 172.21.15.59:0/947625967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f353c004800 0x7f35540b2c90 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.483+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 >> 172.21.15.59:0/947625967 conn(0x7f35540b6fb0 msgr2=0x7f35540711b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.484+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 shutdown_connections 2024-06-29T15:58:09.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.484+0000 7f355a09e640 1 -- 172.21.15.59:0/947625967 wait complete. 2024-06-29T15:58:09.486 INFO:tasks.ceph.mds.b.smithi153.stderr: -2004> 2024-06-29T15:57:39.187+0000 7f57f2ec5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:09.495 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-06-29T15:58:09.541 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T15:58:09.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.615+0000 7efdd3c8f640 1 -- 172.21.15.59:0/1346149966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb4003f90 msgr2=0x7efdb4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.615+0000 7efdd3c8f640 1 --2- 172.21.15.59:0/1346149966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdb4003f90 0x7efdb4004800 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7efdc0001740 tx=0x7efdb4005830 comp rx=0 tx=0).stop 2024-06-29T15:58:09.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.616+0000 7efdd222c640 1 -- 172.21.15.59:0/1346149966 reap_dead start 2024-06-29T15:58:09.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.616+0000 7efdd3c8f640 1 -- 172.21.15.59:0/1346149966 shutdown_connections 2024-06-29T15:58:09.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.616+0000 7efdd3c8f640 1 -- 172.21.15.59:0/1346149966 >> 172.21.15.59:0/1346149966 conn(0x7efdcc0d4050 msgr2=0x7efdcc0b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:09.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.616+0000 7efdd3c8f640 1 -- 172.21.15.59:0/1346149966 shutdown_connections 2024-06-29T15:58:09.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.616+0000 7efdd3c8f640 1 -- 172.21.15.59:0/1346149966 wait complete. 2024-06-29T15:58:09.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.617+0000 7efdd3c8f640 1 Processor -- start 2024-06-29T15:58:09.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.618+0000 7efdd3c8f640 1 -- start start 2024-06-29T15:58:09.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd3c8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4003f90 0x7efdcc0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd3c8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0cb2d0 0x7efdcc160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd3c8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdcc0cb6e0 0x7efdcc1607d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd3c8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc0b7120 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd3c8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc0b6fa0 con 0x7efdb4003f90 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd3c8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efdcc0b72a0 con 0x7efdcc0cb6e0 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd122a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0cb2d0 0x7efdcc160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd122a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0cb2d0 0x7efdcc160290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56858/0 (socket says 172.21.15.59:56858) 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd122a640 1 -- 172.21.15.59:0/2156581282 learned_addr learned my addr 172.21.15.59:0/2156581282 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd222c640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdcc0cb6e0 0x7efdcc1607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd1a2b640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4003f90 0x7efdcc0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd122a640 1 -- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdcc0cb6e0 msgr2=0x7efdcc1607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd122a640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdcc0cb6e0 0x7efdcc1607d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd122a640 1 -- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4003f90 msgr2=0x7efdcc0ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd122a640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4003f90 0x7efdcc0ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.619+0000 7efdd122a640 1 -- 172.21.15.59:0/2156581282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc0cbf60 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.620+0000 7efdd122a640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0cb2d0 0x7efdcc160290 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7efdc002a9d0 tx=0x7efdc0006170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:09.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.620+0000 7efdca7fc640 1 -- 172.21.15.59:0/2156581282 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0042070 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.620+0000 7efdca7fc640 1 -- 172.21.15.59:0/2156581282 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efdc00037f0 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.620+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc15b6e0 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.620+0000 7efdca7fc640 1 -- 172.21.15.59:0/2156581282 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0037040 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.620+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc14a1a0 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.621+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdcc15dd20 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.621+0000 7efdca7fc640 1 -- 172.21.15.59:0/2156581282 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7efdc0045070 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.622+0000 7efdca7fc640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7efdb8076510 0x7efdb8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.622+0000 7efdca7fc640 1 -- 172.21.15.59:0/2156581282 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(232..232 src has 1..232) ==== 7158+0+0 (secure 0 0 0) 0x7efdc00b7e60 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.622+0000 7efdd1a2b640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7efdb8076510 0x7efdb8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.623+0000 7efdd1a2b640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7efdb8076510 0x7efdb8078a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efdbc005fd0 tx=0x7efdbc009500 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:09.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.627+0000 7efdca7fc640 1 -- 172.21.15.59:0/2156581282 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efdc0006260 con 0x7efdcc0cb2d0 2024-06-29T15:58:09.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:09.856+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) -- 0x7efdcc0cbb60 con 0x7efdcc0cb2d0 2024-06-29T15:58:10.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.485+0000 7efdca7fc640 1 -- 172.21.15.59:0/2156581282 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v125) ==== 107+0+0 (secure 0 0 0) 0x7efdc0080870 con 0x7efdcc0cb2d0 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7efdb8076510 msgr2=0x7efdb8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7efdb8076510 0x7efdb8078a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efdbc005fd0 tx=0x7efdbc009500 comp rx=0 tx=0).stop 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0cb2d0 msgr2=0x7efdcc160290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0cb2d0 0x7efdcc160290 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7efdc002a9d0 tx=0x7efdc0006170 comp rx=0 tx=0).stop 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 shutdown_connections 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7efdb8076510 0x7efdb8078a20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efdcc0cb6e0 0x7efdcc1607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0cb2d0 0x7efdcc160290 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 --2- 172.21.15.59:0/2156581282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efdb4003f90 0x7efdcc0ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.489+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 >> 172.21.15.59:0/2156581282 conn(0x7efdcc0d4050 msgr2=0x7efdcc0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.490+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 shutdown_connections 2024-06-29T15:58:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.490+0000 7efdd3c8f640 1 -- 172.21.15.59:0/2156581282 wait complete. 2024-06-29T15:58:10.503 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-06-29T15:58:10.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.633+0000 7f0d57f7a640 1 -- 172.21.15.59:0/571389042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d500cec20 msgr2=0x7f0d500d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.633+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/571389042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d500cec20 0x7f0d500d1790 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f0d44005fd0 tx=0x7f0d4402a4f0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.633+0000 7f0d57f7a640 1 -- 172.21.15.59:0/571389042 shutdown_connections 2024-06-29T15:58:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.633+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/571389042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d500cec20 0x7f0d500d1790 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.633+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/571389042 >> v2:172.21.15.153:3300/0 conn(0x7f0d501423c0 0x7f0d500b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.633+0000 7f0d57f7a640 1 -- 172.21.15.59:0/571389042 >> 172.21.15.59:0/571389042 conn(0x7f0d50073e10 msgr2=0x7f0d50074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.633+0000 7f0d57f7a640 1 -- 172.21.15.59:0/571389042 shutdown_connections 2024-06-29T15:58:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.633+0000 7f0d57f7a640 1 -- 172.21.15.59:0/571389042 wait complete. 2024-06-29T15:58:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.635+0000 7f0d57f7a640 1 Processor -- start 2024-06-29T15:58:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.635+0000 7f0d57f7a640 1 -- start start 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.636+0000 7f0d57f7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d500cec20 0x7f0d5015b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.636+0000 7f0d57f7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d500d1cd0 0x7f0d500b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.636+0000 7f0d57f7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d501423c0 0x7f0d500b6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.636+0000 7f0d57f7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0d500b2bf0 con 0x7f0d501423c0 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.636+0000 7f0d57f7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5015c4d0 con 0x7f0d500d1cd0 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.636+0000 7f0d57f7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0d500b2d70 con 0x7f0d500cec20 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d501423c0 0x7f0d500b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d501423c0 0x7f0d500b6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56872/0 (socket says 172.21.15.59:56872) 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 -- 172.21.15.59:0/3533661675 learned_addr learned my addr 172.21.15.59:0/3533661675 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d55d16640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d500cec20 0x7f0d5015b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d55515640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d500d1cd0 0x7f0d500b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 -- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d500cec20 msgr2=0x7f0d5015b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d500cec20 0x7f0d5015b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 -- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d500d1cd0 msgr2=0x7f0d500b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d500d1cd0 0x7f0d500b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 -- 172.21.15.59:0/3533661675 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d501f4d70 con 0x7f0d501423c0 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d56517640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d501423c0 0x7f0d500b6660 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f0d4000b9f0 tx=0x7f0d4000beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d4e7fc640 1 -- 172.21.15.59:0/3533661675 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d40019070 con 0x7f0d501423c0 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d4e7fc640 1 -- 172.21.15.59:0/3533661675 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d4000cc60 con 0x7f0d501423c0 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d501f5050 con 0x7f0d501423c0 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d4e7fc640 1 -- 172.21.15.59:0/3533661675 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d400146a0 con 0x7f0d501423c0 2024-06-29T15:58:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.637+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d501f5610 con 0x7f0d501423c0 2024-06-29T15:58:10.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.638+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d500cdf30 con 0x7f0d501423c0 2024-06-29T15:58:10.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.639+0000 7f0d4e7fc640 1 -- 172.21.15.59:0/3533661675 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f0d4001c070 con 0x7f0d501423c0 2024-06-29T15:58:10.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.640+0000 7f0d4e7fc640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0d3c076540 0x7f0d3c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.640+0000 7f0d4e7fc640 1 -- 172.21.15.59:0/3533661675 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(232..232 src has 1..232) ==== 7158+0+0 (secure 0 0 0) 0x7f0d40097f60 con 0x7f0d501423c0 2024-06-29T15:58:10.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.640+0000 7f0d55d16640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0d3c076540 0x7f0d3c078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.641+0000 7f0d55d16640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0d3c076540 0x7f0d3c078a50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0d480098e0 tx=0x7f0d48009060 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:10.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.644+0000 7f0d4e7fc640 1 -- 172.21.15.59:0/3533661675 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0d500cdf30 con 0x7f0d501423c0 2024-06-29T15:58:10.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:10.834+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0d500d1790 con 0x7f0d501423c0 2024-06-29T15:58:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.487+0000 7f0d4e7fc640 1 -- 172.21.15.59:0/3533661675 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v233) ==== 190+0+0 (secure 0 0 0) 0x7f0d40060830 con 0x7f0d501423c0 2024-06-29T15:58:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'backup_fs_metadata' removed 2024-06-29T15:58:11.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0d3c076540 msgr2=0x7f0d3c078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0d3c076540 0x7f0d3c078a50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0d480098e0 tx=0x7f0d48009060 comp rx=0 tx=0).stop 2024-06-29T15:58:11.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d501423c0 msgr2=0x7f0d500b6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d501423c0 0x7f0d500b6660 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f0d4000b9f0 tx=0x7f0d4000beb0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 shutdown_connections 2024-06-29T15:58:11.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0d3c076540 0x7f0d3c078a50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d501423c0 0x7f0d500b6660 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d500d1cd0 0x7f0d500b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 --2- 172.21.15.59:0/3533661675 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d500cec20 0x7f0d5015b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 >> 172.21.15.59:0/3533661675 conn(0x7f0d50073e10 msgr2=0x7f0d50074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:11.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 shutdown_connections 2024-06-29T15:58:11.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.491+0000 7f0d57f7a640 1 -- 172.21.15.59:0/3533661675 wait complete. 2024-06-29T15:58:11.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-06-29T15:58:11.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.634+0000 7f13ee038640 1 -- 172.21.15.59:0/1022537734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13e815f1e0 msgr2=0x7f13e8073e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.634+0000 7f13ee038640 1 --2- 172.21.15.59:0/1022537734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13e815f1e0 0x7f13e8073e90 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009830 tx=0x7f13dc02a510 comp rx=0 tx=0).stop 2024-06-29T15:58:11.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.634+0000 7f13ee038640 1 -- 172.21.15.59:0/1022537734 shutdown_connections 2024-06-29T15:58:11.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.634+0000 7f13ee038640 1 --2- 172.21.15.59:0/1022537734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13e815f1e0 0x7f13e8073e90 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.634+0000 7f13ee038640 1 --2- 172.21.15.59:0/1022537734 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13e81440a0 0x7f13e8143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.634+0000 7f13ee038640 1 -- 172.21.15.59:0/1022537734 >> 172.21.15.59:0/1022537734 conn(0x7f13e80cd630 msgr2=0x7f13e80cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:11.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.634+0000 7f13ee038640 1 -- 172.21.15.59:0/1022537734 shutdown_connections 2024-06-29T15:58:11.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.634+0000 7f13ee038640 1 -- 172.21.15.59:0/1022537734 wait complete. 2024-06-29T15:58:11.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.636+0000 7f13ee038640 1 Processor -- start 2024-06-29T15:58:11.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.636+0000 7f13ee038640 1 -- start start 2024-06-29T15:58:11.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.637+0000 7f13ee038640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13e80743d0 0x7f13e813e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.637+0000 7f13ee038640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e81440a0 0x7f13e815a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.637+0000 7f13ee038640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13e815f1e0 0x7f13e815d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.637+0000 7f13ee038640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f13e80cf140 con 0x7f13e81440a0 2024-06-29T15:58:11.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.637+0000 7f13ee038640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13e80b4e00 con 0x7f13e815f1e0 2024-06-29T15:58:11.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.637+0000 7f13ee038640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f13e80cf2c0 con 0x7f13e80743d0 2024-06-29T15:58:11.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e81440a0 0x7f13e815a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13e80743d0 0x7f13e813e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e81440a0 0x7f13e815a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56884/0 (socket says 172.21.15.59:56884) 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 -- 172.21.15.59:0/858125792 learned_addr learned my addr 172.21.15.59:0/858125792 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e7fff640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13e815f1e0 0x7f13e815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 -- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13e80743d0 msgr2=0x7f13e813e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13e80743d0 0x7f13e813e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 -- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13e815f1e0 msgr2=0x7f13e815d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13e815f1e0 0x7f13e815d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 -- 172.21.15.59:0/858125792 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e813ea10 con 0x7f13e81440a0 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13e6ffd640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e81440a0 0x7f13e815a570 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009950 tx=0x7f13dc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13cffff640 1 -- 172.21.15.59:0/858125792 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc031d60 con 0x7f13e81440a0 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13cffff640 1 -- 172.21.15.59:0/858125792 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13dc033e10 con 0x7f13e81440a0 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13cffff640 1 -- 172.21.15.59:0/858125792 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc005260 con 0x7f13e81440a0 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.638+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e813ecf0 con 0x7f13e81440a0 2024-06-29T15:58:11.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.639+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e813f2b0 con 0x7f13e81440a0 2024-06-29T15:58:11.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.640+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e813e780 con 0x7f13e81440a0 2024-06-29T15:58:11.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.640+0000 7f13cffff640 1 -- 172.21.15.59:0/858125792 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f13dc005400 con 0x7f13e81440a0 2024-06-29T15:58:11.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.641+0000 7f13cffff640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f13d4076510 0x7f13d4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.641+0000 7f13cffff640 1 -- 172.21.15.59:0/858125792 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 6714+0+0 (secure 0 0 0) 0x7f13dc0b9090 con 0x7f13e81440a0 2024-06-29T15:58:11.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.641+0000 7f13e77fe640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f13d4076510 0x7f13d4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.642+0000 7f13e77fe640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f13d4076510 0x7f13d4078a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009b30 tx=0x7f13d8005f90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:11.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.645+0000 7f13cffff640 1 -- 172.21.15.59:0/858125792 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f13dc03f030 con 0x7f13e81440a0 2024-06-29T15:58:11.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:11.837+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f13e8143a00 con 0x7f13e81440a0 2024-06-29T15:58:12.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.493+0000 7f13cffff640 1 -- 172.21.15.59:0/858125792 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v234) ==== 178+0+0 (secure 0 0 0) 0x7f13dc08a1d0 con 0x7f13e81440a0 2024-06-29T15:58:12.494 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'backup_fs_data' removed 2024-06-29T15:58:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f13d4076510 msgr2=0x7f13d4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f13d4076510 0x7f13d4078a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009b30 tx=0x7f13d8005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e81440a0 msgr2=0x7f13e815a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e81440a0 0x7f13e815a570 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009950 tx=0x7f13dc033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 shutdown_connections 2024-06-29T15:58:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f13d4076510 0x7f13d4078a20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13e815f1e0 0x7f13e815d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13e81440a0 0x7f13e815a570 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 --2- 172.21.15.59:0/858125792 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f13e80743d0 0x7f13e813e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 >> 172.21.15.59:0/858125792 conn(0x7f13e80cd630 msgr2=0x7f13e80b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 shutdown_connections 2024-06-29T15:58:12.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.497+0000 7f13ee038640 1 -- 172.21.15.59:0/858125792 wait complete. 2024-06-29T15:58:12.515 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:58:12.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.641+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1050891065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f815e0a0 msgr2=0x7fc0f8161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.641+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1050891065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f815e0a0 0x7fc0f8161a80 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fc0f808c200 tx=0x7fc0e802a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.641+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1050891065 shutdown_connections 2024-06-29T15:58:12.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.641+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1050891065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f815e0a0 0x7fc0f8161a80 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.641+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1050891065 >> v2:172.21.15.153:3300/0 conn(0x7fc0f8162770 0x7fc0f80ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.641+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1050891065 >> 172.21.15.59:0/1050891065 conn(0x7fc0f80d45b0 msgr2=0x7fc0f80d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:12.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.641+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1050891065 shutdown_connections 2024-06-29T15:58:12.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.641+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1050891065 wait complete. 2024-06-29T15:58:12.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.643+0000 7fc0ffc74640 1 Processor -- start 2024-06-29T15:58:12.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.643+0000 7fc0ffc74640 1 -- start start 2024-06-29T15:58:12.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0ffc74640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0f815e0a0 0x7fc0f81f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0ffc74640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0f8161fc0 0x7fc0f81f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0ffc74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f8162770 0x7fc0f81f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0ffc74640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f80d2d80 con 0x7fc0f8162770 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0ffc74640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f80b4b00 con 0x7fc0f815e0a0 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0ffc74640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f80d2f00 con 0x7fc0f8161fc0 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0fe211640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f8162770 0x7fc0f81f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0fe211640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f8162770 0x7fc0f81f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56904/0 (socket says 172.21.15.59:56904) 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0fe211640 1 -- 172.21.15.59:0/1559668455 learned_addr learned my addr 172.21.15.59:0/1559668455 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0fda10640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0f815e0a0 0x7fc0f81f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.644+0000 7fc0fd20f640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0f8161fc0 0x7fc0f81f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0fe211640 1 -- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0f8161fc0 msgr2=0x7fc0f81f7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0fe211640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0f8161fc0 0x7fc0f81f7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0fe211640 1 -- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0f815e0a0 msgr2=0x7fc0f81f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0fe211640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0f815e0a0 0x7fc0f81f1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0fe211640 1 -- 172.21.15.59:0/1559668455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f80c8d30 con 0x7fc0f8162770 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0fd20f640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0f8161fc0 0x7fc0f81f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0fe211640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f8162770 0x7fc0f81f84c0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7fc0f000c9f0 tx=0x7fc0f000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0ee7fc640 1 -- 172.21.15.59:0/1559668455 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0016020 con 0x7fc0f8162770 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0ee7fc640 1 -- 172.21.15.59:0/1559668455 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0f000ac60 con 0x7fc0f8162770 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f80c9010 con 0x7fc0f8162770 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0ee7fc640 1 -- 172.21.15.59:0/1559668455 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0012680 con 0x7fc0f8162770 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.645+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f80c95d0 con 0x7fc0f8162770 2024-06-29T15:58:12.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.646+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f8161ba0 con 0x7fc0f8162770 2024-06-29T15:58:12.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.647+0000 7fc0ee7fc640 1 -- 172.21.15.59:0/1559668455 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fc0f0019020 con 0x7fc0f8162770 2024-06-29T15:58:12.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.647+0000 7fc0ee7fc640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc0dc076510 0x7fc0dc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:12.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.647+0000 7fc0ee7fc640 1 -- 172.21.15.59:0/1559668455 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7fc0f00982f0 con 0x7fc0f8162770 2024-06-29T15:58:12.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.648+0000 7fc0fda10640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc0dc076510 0x7fc0dc078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:12.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.648+0000 7fc0fda10640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc0dc076510 0x7fc0dc078a20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc0f400bb50 tx=0x7fc0f400b2e0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:12.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.652+0000 7fc0ee7fc640 1 -- 172.21.15.59:0/1559668455 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0f0020020 con 0x7fc0f8162770 2024-06-29T15:58:12.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.848+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fc0f80d3420 con 0x7fc0f8162770 2024-06-29T15:58:12.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.852+0000 7fc0ee7fc640 1 -- 172.21.15.59:0/1559668455 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v75) ==== 71+0+0 (secure 0 0 0) 0x7fc0f81623d0 con 0x7fc0f8162770 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.854+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc0dc076510 msgr2=0x7fc0dc078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.854+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc0dc076510 0x7fc0dc078a20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc0f400bb50 tx=0x7fc0f400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.854+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f8162770 msgr2=0x7fc0f81f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.854+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f8162770 0x7fc0f81f84c0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7fc0f000c9f0 tx=0x7fc0f000ceb0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.855+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 shutdown_connections 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.855+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc0dc076510 0x7fc0dc078a20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.855+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0f8162770 0x7fc0f81f84c0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.855+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0f8161fc0 0x7fc0f81f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.855+0000 7fc0ffc74640 1 --2- 172.21.15.59:0/1559668455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0f815e0a0 0x7fc0f81f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.855+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 >> 172.21.15.59:0/1559668455 conn(0x7fc0f80d45b0 msgr2=0x7fc0f80b7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:12.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.855+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 shutdown_connections 2024-06-29T15:58:12.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.855+0000 7fc0ffc74640 1 -- 172.21.15.59:0/1559668455 wait complete. 2024-06-29T15:58:12.866 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-06-29T15:58:12.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.984+0000 7f632b2ef640 1 -- 172.21.15.59:0/1611810116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63240b3740 msgr2=0x7f63240b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.984+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1611810116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63240b3740 0x7f63240b3b50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6314009830 tx=0x7f631402a510 comp rx=0 tx=0).stop 2024-06-29T15:58:12.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.984+0000 7f632b2ef640 1 -- 172.21.15.59:0/1611810116 shutdown_connections 2024-06-29T15:58:12.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.984+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1611810116 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63240b3740 0x7f63240b3b50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.984+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1611810116 >> v2:172.21.15.153:3300/0 conn(0x7f63241458a0 0x7f63240cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.984+0000 7f632b2ef640 1 -- 172.21.15.59:0/1611810116 >> 172.21.15.59:0/1611810116 conn(0x7f632406c950 msgr2=0x7f6324073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:12.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.984+0000 7f632b2ef640 1 -- 172.21.15.59:0/1611810116 shutdown_connections 2024-06-29T15:58:12.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.984+0000 7f632b2ef640 1 -- 172.21.15.59:0/1611810116 wait complete. 2024-06-29T15:58:12.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.986+0000 7f632b2ef640 1 Processor -- start 2024-06-29T15:58:12.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.986+0000 7f632b2ef640 1 -- start start 2024-06-29T15:58:12.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632b2ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63240b3740 0x7f63240aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:12.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632b2ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f632415c2f0 0x7f63240b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:12.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632b2ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240b5840 0x7f63240b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:12.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632b2ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6324160020 con 0x7f63240b5840 2024-06-29T15:58:12.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632b2ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f632415cbb0 con 0x7f632415c2f0 2024-06-29T15:58:12.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632b2ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f63241601a0 con 0x7f63240b3740 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632988c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240b5840 0x7f63240b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632988c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240b5840 0x7f63240b5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44066/0 (socket says 172.21.15.59:44066) 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632988c640 1 -- 172.21.15.59:0/1515555105 learned_addr learned my addr 172.21.15.59:0/1515555105 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632888a640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f632415c2f0 0x7f63240b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.987+0000 7f632908b640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63240b3740 0x7f63240aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f632988c640 1 -- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63240b3740 msgr2=0x7f63240aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f632988c640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63240b3740 0x7f63240aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f632988c640 1 -- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f632415c2f0 msgr2=0x7f63240b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f632988c640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f632415c2f0 0x7f63240b1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f632988c640 1 -- 172.21.15.59:0/1515555105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63240b6190 con 0x7f63240b5840 2024-06-29T15:58:12.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f632988c640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240b5840 0x7f63240b5c50 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f631c00cb10 tx=0x7f631c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:12.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f6319ffb640 1 -- 172.21.15.59:0/1515555105 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c016020 con 0x7f63240b5840 2024-06-29T15:58:12.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f6319ffb640 1 -- 172.21.15.59:0/1515555105 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f631c00ad80 con 0x7f63240b5840 2024-06-29T15:58:12.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63240aadd0 con 0x7f63240b5840 2024-06-29T15:58:12.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f6319ffb640 1 -- 172.21.15.59:0/1515555105 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c012680 con 0x7f63240b5840 2024-06-29T15:58:12.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.988+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63240ab390 con 0x7f63240b5840 2024-06-29T15:58:12.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.989+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63241606c0 con 0x7f63240b5840 2024-06-29T15:58:12.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.990+0000 7f6319ffb640 1 -- 172.21.15.59:0/1515555105 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f631c019020 con 0x7f63240b5840 2024-06-29T15:58:12.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.990+0000 7f6319ffb640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6308076540 0x7f6308078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:12.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.990+0000 7f6319ffb640 1 -- 172.21.15.59:0/1515555105 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7f631c098240 con 0x7f63240b5840 2024-06-29T15:58:12.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.991+0000 7f632908b640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6308076540 0x7f6308078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:12.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.991+0000 7f632908b640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6308076540 0x7f6308078a50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f63200095f0 tx=0x7f6320005f90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:12.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:12.995+0000 7f6319ffb640 1 -- 172.21.15.59:0/1515555105 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f631c020020 con 0x7f63240b5840 2024-06-29T15:58:13.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:58:13.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:58:13.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.266+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) -- 0x7f63240b1f10 con 0x7f63240b5840 2024-06-29T15:58:13.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.495+0000 7f6319ffb640 1 -- 172.21.15.59:0/1515555105 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3165) ==== 186+0+0 (secure 0 0 0) 0x7f63240b6190 con 0x7f63240b5840 2024-06-29T15:58:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6308076540 msgr2=0x7f6308078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6308076540 0x7f6308078a50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f63200095f0 tx=0x7f6320005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240b5840 msgr2=0x7f63240b5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240b5840 0x7f63240b5c50 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f631c00cb10 tx=0x7f631c00a060 comp rx=0 tx=0).stop 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 shutdown_connections 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6308076540 0x7f6308078a50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240b5840 0x7f63240b5c50 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f632415c2f0 0x7f63240b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 --2- 172.21.15.59:0/1515555105 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f63240b3740 0x7f63240aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 >> 172.21.15.59:0/1515555105 conn(0x7f632406c950 msgr2=0x7f63241413e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 shutdown_connections 2024-06-29T15:58:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.497+0000 7f632b2ef640 1 -- 172.21.15.59:0/1515555105 wait complete. 2024-06-29T15:58:13.510 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-06-29T15:58:13.510 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-06-29T15:58:13.510 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-06-29T15:58:13.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:13.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.629+0000 7f5284fcf640 1 -- 172.21.15.59:0/2107501370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800b2fa0 msgr2=0x7f52800d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.629+0000 7f5284fcf640 1 --2- 172.21.15.59:0/2107501370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800b2fa0 0x7f52800d1520 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f52800cf310 tx=0x7f526802a530 comp rx=0 tx=0).stop 2024-06-29T15:58:13.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.629+0000 7f5284fcf640 1 -- 172.21.15.59:0/2107501370 shutdown_connections 2024-06-29T15:58:13.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.629+0000 7f5284fcf640 1 --2- 172.21.15.59:0/2107501370 >> v2:172.21.15.153:3300/0 conn(0x7f52800d1a60 0x7f52800b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.629+0000 7f5284fcf640 1 --2- 172.21.15.59:0/2107501370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800b2fa0 0x7f52800d1520 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.629+0000 7f5284fcf640 1 -- 172.21.15.59:0/2107501370 >> 172.21.15.59:0/2107501370 conn(0x7f5280073e10 msgr2=0x7f5280074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.629+0000 7f5284fcf640 1 -- 172.21.15.59:0/2107501370 shutdown_connections 2024-06-29T15:58:13.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.629+0000 7f5284fcf640 1 -- 172.21.15.59:0/2107501370 wait complete. 2024-06-29T15:58:13.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.631+0000 7f5284fcf640 1 Processor -- start 2024-06-29T15:58:13.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.631+0000 7f5284fcf640 1 -- start start 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.632+0000 7f5284fcf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52800b2270 0x7f52800aa380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.632+0000 7f5284fcf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52800b2fa0 0x7f52801efce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f5284fcf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800d1a60 0x7f52800ab570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f5284fcf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f52800b6400 con 0x7f52800d1a60 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f5284fcf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52800b6280 con 0x7f52800b2fa0 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f5284fcf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f52800b6580 con 0x7f52800b2270 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800d1a60 0x7f52800ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800d1a60 0x7f52800ab570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44084/0 (socket says 172.21.15.59:44084) 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 -- 172.21.15.59:0/845389995 learned_addr learned my addr 172.21.15.59:0/845389995 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:13.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527dd9b640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52800b2fa0 0x7f52801efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527e59c640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52800b2270 0x7f52800aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 -- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52800b2270 msgr2=0x7f52800aa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52800b2270 0x7f52800aa380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 -- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52800b2fa0 msgr2=0x7f52801efce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52800b2fa0 0x7f52801efce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 -- 172.21.15.59:0/845389995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528015c820 con 0x7f52800d1a60 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f527ed9d640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800d1a60 0x7f52800ab570 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f527800c9a0 tx=0x7f527800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f5276ffd640 1 -- 172.21.15.59:0/845389995 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278007b50 con 0x7f52800d1a60 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f5276ffd640 1 -- 172.21.15.59:0/845389995 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5278007e00 con 0x7f52800d1a60 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f5276ffd640 1 -- 172.21.15.59:0/845389995 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527800f6b0 con 0x7f52800d1a60 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.633+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52801f0220 con 0x7f52800d1a60 2024-06-29T15:58:13.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.634+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f528015cdc0 con 0x7f52800d1a60 2024-06-29T15:58:13.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.634+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52800a9750 con 0x7f52800d1a60 2024-06-29T15:58:13.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.635+0000 7f5276ffd640 1 -- 172.21.15.59:0/845389995 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f527800f850 con 0x7f52800d1a60 2024-06-29T15:58:13.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.636+0000 7f5276ffd640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5260076540 0x7f5260078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.636+0000 7f5276ffd640 1 -- 172.21.15.59:0/845389995 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7f52780984a0 con 0x7f52800d1a60 2024-06-29T15:58:13.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.636+0000 7f527e59c640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5260076540 0x7f5260078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.637+0000 7f527e59c640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5260076540 0x7f5260078a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5270007fd0 tx=0x7f5270009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:13.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.640+0000 7f5276ffd640 1 -- 172.21.15.59:0/845389995 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5278014030 con 0x7f52800d1a60 2024-06-29T15:58:13.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.830+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52800b0860 con 0x7f52800d1a60 2024-06-29T15:58:13.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.831+0000 7f5276ffd640 1 -- 172.21.15.59:0/845389995 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) ==== 74+0+14738 (secure 0 0 0) 0x7f5278061750 con 0x7f52800d1a60 2024-06-29T15:58:13.831 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:13.831 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":234,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:12.487949+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6837/601671949":"2024-06-30T15:58:09.479797+0000","172.21.15.153:6835/601671949":"2024-06-30T15:58:09.479797+0000","172.21.15.59:6832/1734341893":"2024-06-30T15:58:02.917822+0000","172.21.15.153:0/1969822965":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/813560021":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2732252046":"2024-06-30T15:57:59.178318+0000","172.21.15.59:6834/1734341893":"2024-06-30T15:58:02.917822+0000","172.21.15.153:0/240322135":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/215704132":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/1238730780":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/3658145560":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2050849752":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/239127507":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2051017624":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/743055221":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/4093149557":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/572137829":"2024-06-30T15:57:50.460403+0000","172.21.15.153:0/2642354773":"2024-06-30T15:57:59.178318+0000","172.21.15.153:0/2737762390":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2014343395":"2024-06-30T15:57:59.178318+0000","172.21.15.153:6833/2780049427":"2024-06-30T15:57:50.460403+0000","172.21.15.153:6832/2780049427":"2024-06-30T15:57:50.460403+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:13.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5260076540 msgr2=0x7f5260078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5260076540 0x7f5260078a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5270007fd0 tx=0x7f5270009040 comp rx=0 tx=0).stop 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800d1a60 msgr2=0x7f52800ab570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800d1a60 0x7f52800ab570 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f527800c9a0 tx=0x7f527800ce60 comp rx=0 tx=0).stop 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 shutdown_connections 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5260076540 0x7f5260078a50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52800d1a60 0x7f52800ab570 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52800b2fa0 0x7f52801efce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 --2- 172.21.15.59:0/845389995 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52800b2270 0x7f52800aa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.833+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 >> 172.21.15.59:0/845389995 conn(0x7f5280073e10 msgr2=0x7f5280071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.834+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 shutdown_connections 2024-06-29T15:58:13.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.834+0000 7f5284fcf640 1 -- 172.21.15.59:0/845389995 wait complete. 2024-06-29T15:58:13.845 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:13.846 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 -- 172.21.15.59:0/4115279505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 msgr2=0x7f8cbc0d2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4115279505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 0x7f8cbc0d2670 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0b35d0 tx=0x7f8cb402a970 comp rx=0 tx=0).stop 2024-06-29T15:58:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 -- 172.21.15.59:0/4115279505 shutdown_connections 2024-06-29T15:58:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4115279505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 0x7f8cbc0d2670 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4115279505 >> v2:172.21.15.153:3300/0 conn(0x7f8cbc15e0a0 0x7f8cbc161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4115279505 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cbc162770 0x7f8cbc0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 -- 172.21.15.59:0/4115279505 >> 172.21.15.59:0/4115279505 conn(0x7f8cbc0d45b0 msgr2=0x7f8cbc0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 -- 172.21.15.59:0/4115279505 shutdown_connections 2024-06-29T15:58:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.967+0000 7f8cc0968640 1 -- 172.21.15.59:0/4115279505 wait complete. 2024-06-29T15:58:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.969+0000 7f8cc0968640 1 Processor -- start 2024-06-29T15:58:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.969+0000 7f8cc0968640 1 -- start start 2024-06-29T15:58:13.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cc0968640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cbc15e0a0 0x7f8cbc15a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cc0968640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 0x7f8cbc15f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cc0968640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cbc162770 0x7f8cbc15b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cc0968640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc0d2f40 con 0x7f8cbc161fc0 2024-06-29T15:58:13.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cc0968640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc0d2dc0 con 0x7f8cbc15e0a0 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cc0968640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8cbc0d30c0 con 0x7f8cbc162770 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cb9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 0x7f8cbc15f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cb9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 0x7f8cbc15f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44106/0 (socket says 172.21.15.59:44106) 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cb9d9b640 1 -- 172.21.15.59:0/4265940428 learned_addr learned my addr 172.21.15.59:0/4265940428 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cba59c640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cbc15e0a0 0x7f8cbc15a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cbad9d640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cbc162770 0x7f8cbc15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cb9d9b640 1 -- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cbc162770 msgr2=0x7f8cbc15b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cb9d9b640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cbc162770 0x7f8cbc15b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cb9d9b640 1 -- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cbc15e0a0 msgr2=0x7f8cbc15a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cb9d9b640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cbc15e0a0 0x7f8cbc15a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.970+0000 7f8cb9d9b640 1 -- 172.21.15.59:0/4265940428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc0c9140 con 0x7f8cbc161fc0 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.971+0000 7f8cb9d9b640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 0x7f8cbc15f060 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00058c0 tx=0x7f8cb000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.971+0000 7f8caaffd640 1 -- 172.21.15.59:0/4265940428 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0017070 con 0x7f8cbc161fc0 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.971+0000 7f8caaffd640 1 -- 172.21.15.59:0/4265940428 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8cb00099b0 con 0x7f8cbc161fc0 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.971+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cbc0c9420 con 0x7f8cbc161fc0 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.971+0000 7f8caaffd640 1 -- 172.21.15.59:0/4265940428 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0009de0 con 0x7f8cbc161fc0 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.971+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cbc0c99e0 con 0x7f8cbc161fc0 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.972+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cbc0cdd90 con 0x7f8cbc161fc0 2024-06-29T15:58:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.973+0000 7f8caaffd640 1 -- 172.21.15.59:0/4265940428 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f8cb001a070 con 0x7f8cbc161fc0 2024-06-29T15:58:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.973+0000 7f8caaffd640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c9c076510 0x7f8c9c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.973+0000 7f8caaffd640 1 -- 172.21.15.59:0/4265940428 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7f8cb0097de0 con 0x7f8cbc161fc0 2024-06-29T15:58:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.973+0000 7f8cba59c640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c9c076510 0x7f8c9c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.974+0000 7f8cba59c640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c9c076510 0x7f8c9c078a20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8cac004630 tx=0x7f8cac00b2e0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:13.977+0000 7f8caaffd640 1 -- 172.21.15.59:0/4265940428 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8cb0060a30 con 0x7f8cbc161fc0 2024-06-29T15:58:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.167+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8cbc0b26d0 con 0x7f8cbc161fc0 2024-06-29T15:58:14.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.167+0000 7f8caaffd640 1 -- 172.21.15.59:0/4265940428 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v234) ==== 93+0+10 (secure 0 0 0) 0x7f8cb0064d30 con 0x7f8cbc161fc0 2024-06-29T15:58:14.168 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c9c076510 msgr2=0x7f8c9c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c9c076510 0x7f8c9c078a20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8cac004630 tx=0x7f8cac00b2e0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 msgr2=0x7f8cbc15f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 0x7f8cbc15f060 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00058c0 tx=0x7f8cb000dae0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 shutdown_connections 2024-06-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c9c076510 0x7f8c9c078a20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8cbc162770 0x7f8cbc15b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8cbc161fc0 0x7f8cbc15f060 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 --2- 172.21.15.59:0/4265940428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cbc15e0a0 0x7f8cbc15a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 >> 172.21.15.59:0/4265940428 conn(0x7f8cbc0d45b0 msgr2=0x7f8cbc0ccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 shutdown_connections 2024-06-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.170+0000 7f8cc0968640 1 -- 172.21.15.59:0/4265940428 wait complete. 2024-06-29T15:58:14.181 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-06-29T15:58:14.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.300+0000 7f5afc43f640 1 -- 172.21.15.59:0/1612729102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc004800 msgr2=0x7f5adc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.300+0000 7f5afc43f640 1 --2- 172.21.15.59:0/1612729102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc004800 0x7f5adc004c10 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80016c0 tx=0x7f5adc005830 comp rx=0 tx=0).stop 2024-06-29T15:58:14.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.300+0000 7f5afa9dc640 1 -- 172.21.15.59:0/1612729102 reap_dead start 2024-06-29T15:58:14.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.300+0000 7f5afc43f640 1 -- 172.21.15.59:0/1612729102 shutdown_connections 2024-06-29T15:58:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.300+0000 7f5afc43f640 1 -- 172.21.15.59:0/1612729102 >> 172.21.15.59:0/1612729102 conn(0x7f5af406c950 msgr2=0x7f5af4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.300+0000 7f5afc43f640 1 -- 172.21.15.59:0/1612729102 shutdown_connections 2024-06-29T15:58:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.300+0000 7f5afc43f640 1 -- 172.21.15.59:0/1612729102 wait complete. 2024-06-29T15:58:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.302+0000 7f5afc43f640 1 Processor -- start 2024-06-29T15:58:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.302+0000 7f5afc43f640 1 -- start start 2024-06-29T15:58:14.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5afc43f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5adc004800 0x7f5af40a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5afc43f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af413e5a0 0x7f5af4071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5afc43f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af4071580 0x7f5af41f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5afc43f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5af406df10 con 0x7f5af413e5a0 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5afc43f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5af406dd90 con 0x7f5adc004800 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5afc43f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5af406e090 con 0x7f5af4071580 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5af99da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af413e5a0 0x7f5af4071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5af99da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af413e5a0 0x7f5af4071040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44124/0 (socket says 172.21.15.59:44124) 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5af99da640 1 -- 172.21.15.59:0/325434171 learned_addr learned my addr 172.21.15.59:0/325434171 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5afa1db640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5adc004800 0x7f5af40a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.303+0000 7f5afa9dc640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af4071580 0x7f5af41f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af99da640 1 -- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af4071580 msgr2=0x7f5af41f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af99da640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af4071580 0x7f5af41f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af99da640 1 -- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5adc004800 msgr2=0x7f5af40a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af99da640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5adc004800 0x7f5af40a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af99da640 1 -- 172.21.15.59:0/325434171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af4071990 con 0x7f5af413e5a0 2024-06-29T15:58:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af99da640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af413e5a0 0x7f5af4071040 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80017b0 tx=0x7f5ae80069a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af2ffd640 1 -- 172.21.15.59:0/325434171 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8047070 con 0x7f5af413e5a0 2024-06-29T15:58:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af2ffd640 1 -- 172.21.15.59:0/325434171 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ae80038e0 con 0x7f5af413e5a0 2024-06-29T15:58:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af40a9e80 con 0x7f5af413e5a0 2024-06-29T15:58:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5af2ffd640 1 -- 172.21.15.59:0/325434171 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8003d10 con 0x7f5af413e5a0 2024-06-29T15:58:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.304+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af40aa440 con 0x7f5af413e5a0 2024-06-29T15:58:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.305+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af40a9bf0 con 0x7f5af413e5a0 2024-06-29T15:58:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.306+0000 7f5af2ffd640 1 -- 172.21.15.59:0/325434171 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f5ae804a070 con 0x7f5af413e5a0 2024-06-29T15:58:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.307+0000 7f5af2ffd640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5ae0076540 0x7f5ae0078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.307+0000 7f5af2ffd640 1 -- 172.21.15.59:0/325434171 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7f5ae80bbc50 con 0x7f5af413e5a0 2024-06-29T15:58:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.307+0000 7f5afa1db640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5ae0076540 0x7f5ae0078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.307+0000 7f5afa1db640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5ae0076540 0x7f5ae0078a50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4006fa0 tx=0x7f5ae4006b20 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:14.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.311+0000 7f5af2ffd640 1 -- 172.21.15.59:0/325434171 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5af4071990 con 0x7f5af413e5a0 2024-06-29T15:58:14.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:14.576+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) -- 0x7f5af4141ba0 con 0x7f5af413e5a0 2024-06-29T15:58:15.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.502+0000 7f5af2ffd640 1 -- 172.21.15.59:0/325434171 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3167) ==== 190+0+0 (secure 0 0 0) 0x7f5af4141ba0 con 0x7f5af413e5a0 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.504+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5ae0076540 msgr2=0x7f5ae0078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.504+0000 7f5afc43f640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5ae0076540 0x7f5ae0078a50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4006fa0 tx=0x7f5ae4006b20 comp rx=0 tx=0).stop 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af413e5a0 msgr2=0x7f5af4071040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af413e5a0 0x7f5af4071040 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80017b0 tx=0x7f5ae80069a0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 shutdown_connections 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5ae0076540 0x7f5ae0078a50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af4071580 0x7f5af41f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af413e5a0 0x7f5af4071040 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 --2- 172.21.15.59:0/325434171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5adc004800 0x7f5af40a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 >> 172.21.15.59:0/325434171 conn(0x7f5af406c950 msgr2=0x7f5af41405f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 shutdown_connections 2024-06-29T15:58:15.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.505+0000 7f5afc43f640 1 -- 172.21.15.59:0/325434171 wait complete. 2024-06-29T15:58:15.519 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.645+0000 7f59548a5640 1 -- 172.21.15.59:0/2159042506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f593c004e40 msgr2=0x7f593c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.645+0000 7f59548a5640 1 --2- 172.21.15.59:0/2159042506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f593c004e40 0x7f593c00b9c0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f594800baf0 tx=0x7f593c005830 comp rx=0 tx=0).stop 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.645+0000 7f594ed9d640 1 -- 172.21.15.59:0/2159042506 reap_dead start 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.645+0000 7f59548a5640 1 -- 172.21.15.59:0/2159042506 shutdown_connections 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.645+0000 7f59548a5640 1 -- 172.21.15.59:0/2159042506 >> 172.21.15.59:0/2159042506 conn(0x7f59500b24c0 msgr2=0x7f59500cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.645+0000 7f59548a5640 1 -- 172.21.15.59:0/2159042506 shutdown_connections 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.645+0000 7f59548a5640 1 -- 172.21.15.59:0/2159042506 wait complete. 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.647+0000 7f59548a5640 1 Processor -- start 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.647+0000 7f59548a5640 1 -- start start 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.648+0000 7f59548a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c004e40 0x7f5950146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.648+0000 7f59548a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f595014e8a0 0x7f5950146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.648+0000 7f59548a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59501526b0 0x7f595014fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.648+0000 7f59548a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f595006ca70 con 0x7f59501526b0 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.648+0000 7f59548a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f595006c8f0 con 0x7f593c004e40 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.648+0000 7f59548a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f59500cccc0 con 0x7f595014e8a0 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59501526b0 0x7f595014fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59501526b0 0x7f595014fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44146/0 (socket says 172.21.15.59:44146) 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 -- 172.21.15.59:0/1390916842 learned_addr learned my addr 172.21.15.59:0/1390916842 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c004e40 0x7f5950146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594dd9b640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f595014e8a0 0x7f5950146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 -- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f595014e8a0 msgr2=0x7f5950146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f595014e8a0 0x7f5950146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 -- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c004e40 msgr2=0x7f5950146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c004e40 0x7f5950146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 -- 172.21.15.59:0/1390916842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5950151610 con 0x7f59501526b0 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f594ed9d640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59501526b0 0x7f595014fe20 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f594800bc10 tx=0x7f5948031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f5946ffd640 1 -- 172.21.15.59:0/1390916842 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5948040070 con 0x7f59501526b0 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f5946ffd640 1 -- 172.21.15.59:0/1390916842 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5948004030 con 0x7f59501526b0 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59501518f0 con 0x7f59501526b0 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f5946ffd640 1 -- 172.21.15.59:0/1390916842 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5948035040 con 0x7f59501526b0 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.649+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5950151eb0 con 0x7f59501526b0 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.650+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59500cd250 con 0x7f59501526b0 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.651+0000 7f5946ffd640 1 -- 172.21.15.59:0/1390916842 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f5948038020 con 0x7f59501526b0 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.652+0000 7f5946ffd640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5930076510 0x7f5930078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.652+0000 7f5946ffd640 1 -- 172.21.15.59:0/1390916842 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7f59480b8660 con 0x7f59501526b0 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.652+0000 7f594e59c640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5930076510 0x7f5930078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.653+0000 7f594e59c640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5930076510 0x7f5930078a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5940001330 tx=0x7f5940012040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:15.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.656+0000 7f5946ffd640 1 -- 172.21.15.59:0/1390916842 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f594802c040 con 0x7f59501526b0 2024-06-29T15:58:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.847+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f5950151360 con 0x7f59501526b0 2024-06-29T15:58:15.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.848+0000 7f5946ffd640 1 -- 172.21.15.59:0/1390916842 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v75) ==== 59+0+1998 (secure 0 0 0) 0x7f5948081290 con 0x7f59501526b0 2024-06-29T15:58:15.848 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:58:15.848 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:58:15.848 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:58:15.848 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:58:15.848 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T15:58:15.848 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T15:58:15.848 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:58:15.849 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T15:58:15.850 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:58:15.850 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:58:15.850 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:58:15.850 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T15:58:15.850 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T15:58:15.850 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T15:58:15.850 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:58:15.850 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:58:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5930076510 msgr2=0x7f5930078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5930076510 0x7f5930078a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5940001330 tx=0x7f5940012040 comp rx=0 tx=0).stop 2024-06-29T15:58:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59501526b0 msgr2=0x7f595014fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59501526b0 0x7f595014fe20 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f594800bc10 tx=0x7f5948031f50 comp rx=0 tx=0).stop 2024-06-29T15:58:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 shutdown_connections 2024-06-29T15:58:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5930076510 0x7f5930078a20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59501526b0 0x7f595014fe20 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f595014e8a0 0x7f5950146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 --2- 172.21.15.59:0/1390916842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f593c004e40 0x7f5950146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 >> 172.21.15.59:0/1390916842 conn(0x7f59500b24c0 msgr2=0x7f595006d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 shutdown_connections 2024-06-29T15:58:15.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.850+0000 7f59548a5640 1 -- 172.21.15.59:0/1390916842 wait complete. 2024-06-29T15:58:15.862 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:58:15.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.983+0000 7f6213495640 1 -- 172.21.15.59:0/3264197117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c162800 msgr2=0x7f620c162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.983+0000 7f6213495640 1 --2- 172.21.15.59:0/3264197117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c162800 0x7f620c162c10 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f6204005fd0 tx=0x7f620402aa70 comp rx=0 tx=0).stop 2024-06-29T15:58:15.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.983+0000 7f6213495640 1 -- 172.21.15.59:0/3264197117 shutdown_connections 2024-06-29T15:58:15.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.983+0000 7f6213495640 1 --2- 172.21.15.59:0/3264197117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c162800 0x7f620c162c10 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.983+0000 7f6213495640 1 --2- 172.21.15.59:0/3264197117 >> v2:172.21.15.153:3300/0 conn(0x7f620c0ce950 0x7f620c1622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.983+0000 7f6213495640 1 -- 172.21.15.59:0/3264197117 >> 172.21.15.59:0/3264197117 conn(0x7f620c0b5610 msgr2=0x7f620c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.983+0000 7f6213495640 1 -- 172.21.15.59:0/3264197117 shutdown_connections 2024-06-29T15:58:15.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.983+0000 7f6213495640 1 -- 172.21.15.59:0/3264197117 wait complete. 2024-06-29T15:58:15.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.984+0000 7f6213495640 1 Processor -- start 2024-06-29T15:58:15.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.985+0000 7f6213495640 1 -- start start 2024-06-29T15:58:15.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6213495640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f620c08c190 0x7f620c1f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6213495640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c0ce950 0x7f620c161800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6213495640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c162800 0x7f620c0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6213495640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f620c0b7f60 con 0x7f620c0ce950 2024-06-29T15:58:15.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6213495640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f620c0b7de0 con 0x7f620c162800 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6213495640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f620c0b80e0 con 0x7f620c08c190 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6210a30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c0ce950 0x7f620c161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6210a30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c0ce950 0x7f620c161800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44158/0 (socket says 172.21.15.59:44158) 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6210a30640 1 -- 172.21.15.59:0/4267533266 learned_addr learned my addr 172.21.15.59:0/4267533266 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6211a32640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c162800 0x7f620c0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6211231640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f620c08c190 0x7f620c1f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6210a30640 1 -- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f620c08c190 msgr2=0x7f620c1f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6210a30640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f620c08c190 0x7f620c1f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6210a30640 1 -- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c162800 msgr2=0x7f620c0cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6210a30640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c162800 0x7f620c0cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.986+0000 7f6210a30640 1 -- 172.21.15.59:0/4267533266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620c15f790 con 0x7f620c0ce950 2024-06-29T15:58:15.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.987+0000 7f6210a30640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c0ce950 0x7f620c161800 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00d6d0 tx=0x7f61fc00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:15.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.987+0000 7f6201ffb640 1 -- 172.21.15.59:0/4267533266 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc017070 con 0x7f620c0ce950 2024-06-29T15:58:15.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.987+0000 7f6201ffb640 1 -- 172.21.15.59:0/4267533266 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61fc004310 con 0x7f620c0ce950 2024-06-29T15:58:15.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.987+0000 7f6201ffb640 1 -- 172.21.15.59:0/4267533266 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc009a80 con 0x7f620c0ce950 2024-06-29T15:58:15.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.987+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f620c15fa90 con 0x7f620c0ce950 2024-06-29T15:58:15.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.987+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f620c160050 con 0x7f620c0ce950 2024-06-29T15:58:15.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.988+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f620c0cfc60 con 0x7f620c0ce950 2024-06-29T15:58:15.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.989+0000 7f6201ffb640 1 -- 172.21.15.59:0/4267533266 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f61fc0035f0 con 0x7f620c0ce950 2024-06-29T15:58:15.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.989+0000 7f6201ffb640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f61f0076540 0x7f61f0078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.989+0000 7f6201ffb640 1 -- 172.21.15.59:0/4267533266 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7f61fc098090 con 0x7f620c0ce950 2024-06-29T15:58:15.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.989+0000 7f6211231640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f61f0076540 0x7f61f0078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.990+0000 7f6211231640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f61f0076540 0x7f61f0078a50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6208001720 tx=0x7f6208009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:15.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:15.994+0000 7f6201ffb640 1 -- 172.21.15.59:0/4267533266 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f620c0cfc60 con 0x7f620c0ce950 2024-06-29T15:58:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.188+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f620c1622c0 con 0x7f620c0ce950 2024-06-29T15:58:16.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.191+0000 7f6201ffb640 1 -- 172.21.15.59:0/4267533266 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) ==== 122+0+0 (secure 0 0 0) 0x7f61fc060c50 con 0x7f620c0ce950 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.193+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f61f0076540 msgr2=0x7f61f0078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.193+0000 7f6213495640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f61f0076540 0x7f61f0078a50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6208001720 tx=0x7f6208009040 comp rx=0 tx=0).stop 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.193+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c0ce950 msgr2=0x7f620c161800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.193+0000 7f6213495640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c0ce950 0x7f620c161800 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00d6d0 tx=0x7f61fc00db90 comp rx=0 tx=0).stop 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.194+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 shutdown_connections 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.194+0000 7f6213495640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f61f0076540 0x7f61f0078a50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.194+0000 7f6213495640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f620c162800 0x7f620c0cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.194+0000 7f6213495640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f620c0ce950 0x7f620c161800 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.194+0000 7f6213495640 1 --2- 172.21.15.59:0/4267533266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f620c08c190 0x7f620c1f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.194+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 >> 172.21.15.59:0/4267533266 conn(0x7f620c0b5610 msgr2=0x7f620c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.194+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 shutdown_connections 2024-06-29T15:58:16.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.194+0000 7f6213495640 1 -- 172.21.15.59:0/4267533266 wait complete. 2024-06-29T15:58:16.205 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:58:16.205 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:58:16.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:58:16.294 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:58:16.294 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:58:16.294 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:16.311 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:58:16.311 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:58:16.311 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:16.330 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:58:16.331 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:58:16.331 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:16.366 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:58:16.366 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:58:16.367 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:16.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:58:16.388 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:58:16.388 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.525+0000 7f2c6e763640 1 -- 172.21.15.59:0/1892897697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c54004800 msgr2=0x7f2c54004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.525+0000 7f2c6e763640 1 --2- 172.21.15.59:0/1892897697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c54004800 0x7f2c54004c10 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c2e0 tx=0x7f2c54005830 comp rx=0 tx=0).stop 2024-06-29T15:58:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.526+0000 7f2c6e763640 1 -- 172.21.15.59:0/1892897697 shutdown_connections 2024-06-29T15:58:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.526+0000 7f2c6e763640 1 --2- 172.21.15.59:0/1892897697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c54004800 0x7f2c54004c10 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.526+0000 7f2c6e763640 1 -- 172.21.15.59:0/1892897697 >> 172.21.15.59:0/1892897697 conn(0x7f2c680d45b0 msgr2=0x7f2c680d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.526+0000 7f2c6e763640 1 -- 172.21.15.59:0/1892897697 shutdown_connections 2024-06-29T15:58:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.526+0000 7f2c6e763640 1 -- 172.21.15.59:0/1892897697 wait complete. 2024-06-29T15:58:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.527+0000 7f2c6e763640 1 Processor -- start 2024-06-29T15:58:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.528+0000 7f2c6e763640 1 -- start start 2024-06-29T15:58:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6e763640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c54004800 0x7f2c68148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6e763640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c680c8870 0x7f2c680c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6e763640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c680cd0e0 0x7f2c680cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6e763640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2c68161a80 con 0x7f2c680cd0e0 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6e763640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c680ce7c0 con 0x7f2c54004800 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6e763640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2c68161c00 con 0x7f2c680c8870 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6cd00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c680cd0e0 0x7f2c680cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6cd00640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c680cd0e0 0x7f2c680cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44186/0 (socket says 172.21.15.59:44186) 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6cd00640 1 -- 172.21.15.59:0/4202851880 learned_addr learned my addr 172.21.15.59:0/4202851880 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c67fff640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c54004800 0x7f2c68148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c677fe640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c680c8870 0x7f2c680c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6cd00640 1 -- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c680c8870 msgr2=0x7f2c680c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6cd00640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c680c8870 0x7f2c680c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6cd00640 1 -- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c54004800 msgr2=0x7f2c68148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6cd00640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c54004800 0x7f2c68148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c6cd00640 1 -- 172.21.15.59:0/4202851880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c680cda30 con 0x7f2c680cd0e0 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.529+0000 7f2c677fe640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c680c8870 0x7f2c680c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.530+0000 7f2c6cd00640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c680cd0e0 0x7f2c680cd4f0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f2c680ce310 tx=0x7f2c6000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.530+0000 7f2c64ff9640 1 -- 172.21.15.59:0/4202851880 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60003d80 con 0x7f2c680cd0e0 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.530+0000 7f2c64ff9640 1 -- 172.21.15.59:0/4202851880 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c60004b20 con 0x7f2c680cd0e0 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.530+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c680c9580 con 0x7f2c680cd0e0 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.530+0000 7f2c64ff9640 1 -- 172.21.15.59:0/4202851880 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60004030 con 0x7f2c680cd0e0 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.530+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c680c9b40 con 0x7f2c680cd0e0 2024-06-29T15:58:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.531+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c68162160 con 0x7f2c680cd0e0 2024-06-29T15:58:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.532+0000 7f2c64ff9640 1 -- 172.21.15.59:0/4202851880 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f2c600041d0 con 0x7f2c680cd0e0 2024-06-29T15:58:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.532+0000 7f2c64ff9640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2c48076510 0x7f2c48078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.532+0000 7f2c64ff9640 1 -- 172.21.15.59:0/4202851880 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7f2c60098e60 con 0x7f2c680cd0e0 2024-06-29T15:58:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.532+0000 7f2c67fff640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2c48076510 0x7f2c48078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.533+0000 7f2c67fff640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2c48076510 0x7f2c48078a20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c58006600 tx=0x7f2c58006b70 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.537+0000 7f2c64ff9640 1 -- 172.21.15.59:0/4202851880 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2c680cda30 con 0x7f2c680cd0e0 2024-06-29T15:58:16.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.765+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c680b4700 con 0x7f2c680cd0e0 2024-06-29T15:58:16.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.766+0000 7f2c64ff9640 1 -- 172.21.15.59:0/4202851880 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) ==== 95+0+3923 (secure 0 0 0) 0x7f2c60062070 con 0x7f2c680cd0e0 2024-06-29T15:58:16.766 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:16.766 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":125,"btime":"2024-06-29T15:58:10:481670+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9368,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3873062235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3873062235},{"type":"v1","addr":"172.21.15.59:6835","nonce":3873062235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9371,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2524746356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":2524746356},{"type":"v1","addr":"172.21.15.153:6836","nonce":2524746356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9908,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/1084748397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":1084748397},{"type":"v1","addr":"172.21.15.153:6837","nonce":1084748397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125},{"gid":12132,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2086282920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2086282920},{"type":"v1","addr":"172.21.15.59:6834","nonce":2086282920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[]} 2024-06-29T15:58:16.766 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 125 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2c48076510 msgr2=0x7f2c48078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2c48076510 0x7f2c48078a20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c58006600 tx=0x7f2c58006b70 comp rx=0 tx=0).stop 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c680cd0e0 msgr2=0x7f2c680cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c680cd0e0 0x7f2c680cd4f0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f2c680ce310 tx=0x7f2c6000f790 comp rx=0 tx=0).stop 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 shutdown_connections 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2c48076510 0x7f2c48078a20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c680cd0e0 0x7f2c680cd4f0 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c680c8870 0x7f2c680c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 --2- 172.21.15.59:0/4202851880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c54004800 0x7f2c68148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 >> 172.21.15.59:0/4202851880 conn(0x7f2c680d45b0 msgr2=0x7f2c6815d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 shutdown_connections 2024-06-29T15:58:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.768+0000 7f2c6e763640 1 -- 172.21.15.59:0/4202851880 wait complete. 2024-06-29T15:58:16.780 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:58:16.780 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:58:16.780 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:58:16.780 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:58:16.780 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:58:16.780 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:58:16.781 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:58:16.781 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:58:16.781 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:58:16.781 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:58:16.781 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:58:16.781 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:58:16.781 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:58:16.781 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:58:16.782 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:58:16.782 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:58:16.790 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:58:16.790 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:58:16.791 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:58:16.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:58:16.797 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:58:16.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:58:16.799 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:58:16.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:58:16.834 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:58:16.834 DEBUG:teuthology.parallel:result is None 2024-06-29T15:58:16.835 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:58:16.835 DEBUG:teuthology.parallel:result is None 2024-06-29T15:58:16.841 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:58:16.842 DEBUG:teuthology.parallel:result is None 2024-06-29T15:58:16.843 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:58:16.843 DEBUG:teuthology.parallel:result is None 2024-06-29T15:58:16.843 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:58:16.922 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:58:16.924 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:58:16.935 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:58:16.935 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T15:58:16.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 -- 172.21.15.59:0/198759603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc15ee10 msgr2=0x7f1bdc15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/198759603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc15ee10 0x7f1bdc15f9b0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0d15d0 tx=0x7f1bd402aa30 comp rx=0 tx=0).stop 2024-06-29T15:58:16.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 -- 172.21.15.59:0/198759603 shutdown_connections 2024-06-29T15:58:16.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/198759603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc15ee10 0x7f1bdc15f9b0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/198759603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc0b3ed0 0x7f1bdc15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/198759603 >> v2:172.21.15.153:3300/0 conn(0x7f1bdc0d23c0 0x7f1bdc0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 -- 172.21.15.59:0/198759603 >> 172.21.15.59:0/198759603 conn(0x7f1bdc0d3ee0 msgr2=0x7f1bdc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 -- 172.21.15.59:0/198759603 shutdown_connections 2024-06-29T15:58:16.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.970+0000 7f1be0d0a640 1 -- 172.21.15.59:0/198759603 wait complete. 2024-06-29T15:58:16.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.972+0000 7f1be0d0a640 1 Processor -- start 2024-06-29T15:58:16.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.972+0000 7f1be0d0a640 1 -- start start 2024-06-29T15:58:16.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1be0d0a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc0b3ed0 0x7f1bdc150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1be0d0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0d23c0 0x7f1bdc1e9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1be0d0a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc15ee10 0x7f1bdc1e7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1be0d0a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1bdc160070 con 0x7f1bdc0b3ed0 2024-06-29T15:58:16.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1be0d0a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1bdc15fef0 con 0x7f1bdc0d23c0 2024-06-29T15:58:16.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1be0d0a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1bdc1601f0 con 0x7f1bdc15ee10 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1bda59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc0b3ed0 0x7f1bdc150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1bd9d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0d23c0 0x7f1bdc1e9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1bda59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc0b3ed0 0x7f1bdc150e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44252/0 (socket says 172.21.15.59:44252) 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1bdad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc15ee10 0x7f1bdc1e7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.973+0000 7f1bda59c640 1 -- 172.21.15.59:0/2905462159 learned_addr learned my addr 172.21.15.59:0/2905462159 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bdad9d640 1 -- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0d23c0 msgr2=0x7f1bdc1e9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bdad9d640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0d23c0 0x7f1bdc1e9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bdad9d640 1 -- 172.21.15.59:0/2905462159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc0b3ed0 msgr2=0x7f1bdc150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bdad9d640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc0b3ed0 0x7f1bdc150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bdad9d640 1 -- 172.21.15.59:0/2905462159 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bdc1e6fb0 con 0x7f1bdc15ee10 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bd9d9b640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0d23c0 0x7f1bdc1e9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bda59c640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc0b3ed0 0x7f1bdc150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bdad9d640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc15ee10 0x7f1bdc1e7b70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0cf190 tx=0x7f1bd4003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bcaffd640 1 -- 172.21.15.59:0/2905462159 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4033050 con 0x7f1bdc15ee10 2024-06-29T15:58:16.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bcaffd640 1 -- 172.21.15.59:0/2905462159 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bd4004450 con 0x7f1bdc15ee10 2024-06-29T15:58:16.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1bcaffd640 1 -- 172.21.15.59:0/2905462159 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4004840 con 0x7f1bdc15ee10 2024-06-29T15:58:16.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bdc1e87b0 con 0x7f1bdc15ee10 2024-06-29T15:58:16.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.974+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bdc1e8d70 con 0x7f1bdc15ee10 2024-06-29T15:58:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.975+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bdc1e8520 con 0x7f1bdc15ee10 2024-06-29T15:58:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.976+0000 7f1bcaffd640 1 -- 172.21.15.59:0/2905462159 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f1bd4036070 con 0x7f1bdc15ee10 2024-06-29T15:58:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.977+0000 7f1bcaffd640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1bbc076510 0x7f1bbc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.977+0000 7f1bcaffd640 1 -- 172.21.15.59:0/2905462159 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(234..234 src has 1..234) ==== 6294+0+0 (secure 0 0 0) 0x7f1bd40b9470 con 0x7f1bdc15ee10 2024-06-29T15:58:16.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.977+0000 7f1bda59c640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1bbc076510 0x7f1bbc078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.977+0000 7f1bda59c640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1bbc076510 0x7f1bbc078a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00bfd0 tx=0x7f1bcc00b390 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:16.980+0000 7f1bcaffd640 1 -- 172.21.15.59:0/2905462159 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1bdc1e6fb0 con 0x7f1bdc15ee10 2024-06-29T15:58:17.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.184+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f1bdc0b3990 con 0x7f1bdc15ee10 2024-06-29T15:58:17.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.518+0000 7f1bcaffd640 1 -- 172.21.15.59:0/2905462159 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v236) ==== 97+0+0 (secure 0 0 0) 0x7f1bd40821a0 con 0x7f1bdc15ee10 2024-06-29T15:58:17.518 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T15:58:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1bbc076510 msgr2=0x7f1bbc078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1bbc076510 0x7f1bbc078a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00bfd0 tx=0x7f1bcc00b390 comp rx=0 tx=0).stop 2024-06-29T15:58:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc15ee10 msgr2=0x7f1bdc1e7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc15ee10 0x7f1bdc1e7b70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0cf190 tx=0x7f1bd4003f10 comp rx=0 tx=0).stop 2024-06-29T15:58:17.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 shutdown_connections 2024-06-29T15:58:17.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1bbc076510 0x7f1bbc078a20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1bdc15ee10 0x7f1bdc1e7b70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1bdc0d23c0 0x7f1bdc1e9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 --2- 172.21.15.59:0/2905462159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1bdc0b3ed0 0x7f1bdc150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 >> 172.21.15.59:0/2905462159 conn(0x7f1bdc0d3ee0 msgr2=0x7f1bdc06c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 shutdown_connections 2024-06-29T15:58:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.520+0000 7f1be0d0a640 1 -- 172.21.15.59:0/2905462159 wait complete. 2024-06-29T15:58:17.534 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:58:17.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.654+0000 7f1174e8f640 1 -- 172.21.15.59:0/1786188348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 msgr2=0x7f1158004cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.654+0000 7f1174e8f640 1 --2- 172.21.15.59:0/1786188348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 0x7f1158004cb0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f1164001860 tx=0x7f1158005830 comp rx=0 tx=0).stop 2024-06-29T15:58:17.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.654+0000 7f116ed9d640 1 -- 172.21.15.59:0/1786188348 reap_dead start 2024-06-29T15:58:17.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.654+0000 7f1174e8f640 1 -- 172.21.15.59:0/1786188348 shutdown_connections 2024-06-29T15:58:17.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.654+0000 7f1174e8f640 1 -- 172.21.15.59:0/1786188348 >> 172.21.15.59:0/1786188348 conn(0x7f11700b5610 msgr2=0x7f11700b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:17.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.654+0000 7f1174e8f640 1 -- 172.21.15.59:0/1786188348 shutdown_connections 2024-06-29T15:58:17.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.654+0000 7f1174e8f640 1 -- 172.21.15.59:0/1786188348 wait complete. 2024-06-29T15:58:17.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.656+0000 7f1174e8f640 1 Processor -- start 2024-06-29T15:58:17.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.656+0000 7f1174e8f640 1 -- start start 2024-06-29T15:58:17.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f1174e8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 0x7f117015d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:17.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f1174e8f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f117015dba0 0x7f11700b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:17.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f1174e8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11700acad0 0x7f11700acee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:17.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f1174e8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f11700cf200 con 0x7f1158004800 2024-06-29T15:58:17.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f1174e8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11700cf080 con 0x7f11700acad0 2024-06-29T15:58:17.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f1174e8f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11700cf380 con 0x7f117015dba0 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f116e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 0x7f117015d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f116e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 0x7f117015d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44274/0 (socket says 172.21.15.59:44274) 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f116e59c640 1 -- 172.21.15.59:0/3988895702 learned_addr learned my addr 172.21.15.59:0/3988895702 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f116dd9b640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f117015dba0 0x7f11700b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.657+0000 7f116ed9d640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11700acad0 0x7f11700acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f116e59c640 1 -- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f117015dba0 msgr2=0x7f11700b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f116e59c640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f117015dba0 0x7f11700b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f116e59c640 1 -- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11700acad0 msgr2=0x7f11700acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f116e59c640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11700acad0 0x7f11700acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f116e59c640 1 -- 172.21.15.59:0/3988895702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11700ad420 con 0x7f1158004800 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f116e59c640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 0x7f117015d560 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f1160011ab0 tx=0x7f1160011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f115effd640 1 -- 172.21.15.59:0/3988895702 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116001e070 con 0x7f1158004800 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f115effd640 1 -- 172.21.15.59:0/3988895702 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1160004460 con 0x7f1158004800 2024-06-29T15:58:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f115effd640 1 -- 172.21.15.59:0/3988895702 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116000fc60 con 0x7f1158004800 2024-06-29T15:58:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11700aa590 con 0x7f1158004800 2024-06-29T15:58:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.658+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11700aab50 con 0x7f1158004800 2024-06-29T15:58:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.659+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11700d1a20 con 0x7f1158004800 2024-06-29T15:58:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.660+0000 7f115effd640 1 -- 172.21.15.59:0/3988895702 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f1160021070 con 0x7f1158004800 2024-06-29T15:58:17.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.660+0000 7f115effd640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1150076510 0x7f1150078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:17.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.661+0000 7f115effd640 1 -- 172.21.15.59:0/3988895702 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 5305+0+0 (secure 0 0 0) 0x7f11600984e0 con 0x7f1158004800 2024-06-29T15:58:17.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.661+0000 7f116dd9b640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1150076510 0x7f1150078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:17.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.661+0000 7f116dd9b640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1150076510 0x7f1150078a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1164032000 tx=0x7f1164001360 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:17.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.665+0000 7f115effd640 1 -- 172.21.15.59:0/3988895702 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11700ad420 con 0x7f1158004800 2024-06-29T15:58:17.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.933+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f11701f4770 con 0x7f1158004800 2024-06-29T15:58:17.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.933+0000 7f115effd640 1 -- 172.21.15.59:0/3988895702 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) ==== 80+0+5304 (secure 0 0 0) 0x7f1160061400 con 0x7f1158004800 2024-06-29T15:58:17.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.934 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.935 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.937 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T15:58:17.938 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.939 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.940 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.941 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T15:58:17.942 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:58:17.943 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T15:58:17.944 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T15:58:17.945 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:58:17.946 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T15:58:17.947 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.935+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1150076510 msgr2=0x7f1150078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.935+0000 7f1174e8f640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1150076510 0x7f1150078a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1164032000 tx=0x7f1164001360 comp rx=0 tx=0).stop 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.935+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 msgr2=0x7f117015d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.935+0000 7f1174e8f640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 0x7f117015d560 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f1160011ab0 tx=0x7f1160011f70 comp rx=0 tx=0).stop 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.936+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 shutdown_connections 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.936+0000 7f1174e8f640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1150076510 0x7f1150078a20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.936+0000 7f1174e8f640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11700acad0 0x7f11700acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.936+0000 7f1174e8f640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f117015dba0 0x7f11700b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.936+0000 7f1174e8f640 1 --2- 172.21.15.59:0/3988895702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1158004800 0x7f117015d560 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.936+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 >> 172.21.15.59:0/3988895702 conn(0x7f11700b5610 msgr2=0x7f11700b9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:17.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.936+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 shutdown_connections 2024-06-29T15:58:17.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:17.936+0000 7f1174e8f640 1 -- 172.21.15.59:0/3988895702 wait complete. 2024-06-29T15:58:17.950 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:18.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.068+0000 7f23aa518640 1 -- 172.21.15.59:0/4277018327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a408c190 msgr2=0x7f23a40cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.068+0000 7f23aa518640 1 --2- 172.21.15.59:0/4277018327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a408c190 0x7f23a40cf080 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2394009860 tx=0x7f239402a4f0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.068+0000 7f23aa518640 1 -- 172.21.15.59:0/4277018327 shutdown_connections 2024-06-29T15:58:18.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.068+0000 7f23aa518640 1 --2- 172.21.15.59:0/4277018327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a408c190 0x7f23a40cf080 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.068+0000 7f23aa518640 1 -- 172.21.15.59:0/4277018327 >> 172.21.15.59:0/4277018327 conn(0x7f23a40b5610 msgr2=0x7f23a40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:18.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.068+0000 7f23aa518640 1 -- 172.21.15.59:0/4277018327 shutdown_connections 2024-06-29T15:58:18.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.068+0000 7f23aa518640 1 -- 172.21.15.59:0/4277018327 wait complete. 2024-06-29T15:58:18.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.070+0000 7f23aa518640 1 Processor -- start 2024-06-29T15:58:18.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.070+0000 7f23aa518640 1 -- start start 2024-06-29T15:58:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23aa518640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23a408c190 0x7f23a4153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23aa518640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a40ce950 0x7f23a40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23aa518640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40cf5c0 0x7f23a40cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23aa518640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f23a4169690 con 0x7f23a40ce950 2024-06-29T15:58:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23aa518640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23a4169510 con 0x7f23a40cf5c0 2024-06-29T15:58:18.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23aa518640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f23a4169810 con 0x7f23a408c190 2024-06-29T15:58:18.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a40ce950 0x7f23a40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a40ce950 0x7f23a40cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44290/0 (socket says 172.21.15.59:44290) 2024-06-29T15:58:18.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 -- 172.21.15.59:0/1444863838 learned_addr learned my addr 172.21.15.59:0/1444863838 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:18.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a3fff640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23a408c190 0x7f23a4153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a8ab5640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40cf5c0 0x7f23a40cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 -- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23a408c190 msgr2=0x7f23a4153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23a408c190 0x7f23a4153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 -- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40cf5c0 msgr2=0x7f23a40cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40cf5c0 0x7f23a40cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 -- 172.21.15.59:0/1444863838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23a40b4db0 con 0x7f23a40ce950 2024-06-29T15:58:18.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a3fff640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23a408c190 0x7f23a4153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:18.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a8ab5640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40cf5c0 0x7f23a40cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:18.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.072+0000 7f23a37fe640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a40ce950 0x7f23a40cae00 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c500 tx=0x7f238c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:18.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.073+0000 7f23a0ff9640 1 -- 172.21.15.59:0/1444863838 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c011790 con 0x7f23a40ce950 2024-06-29T15:58:18.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.073+0000 7f23a0ff9640 1 -- 172.21.15.59:0/1444863838 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f238c017070 con 0x7f23a40ce950 2024-06-29T15:58:18.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.073+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23a415d7b0 con 0x7f23a40ce950 2024-06-29T15:58:18.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.073+0000 7f23a0ff9640 1 -- 172.21.15.59:0/1444863838 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c011ca0 con 0x7f23a40ce950 2024-06-29T15:58:18.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.073+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23a415dd70 con 0x7f23a40ce950 2024-06-29T15:58:18.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.074+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23a40cf080 con 0x7f23a40ce950 2024-06-29T15:58:18.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.074+0000 7f23a0ff9640 1 -- 172.21.15.59:0/1444863838 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f238c01c460 con 0x7f23a40ce950 2024-06-29T15:58:18.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.075+0000 7f23a0ff9640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2384076510 0x7f2384078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.075+0000 7f23a0ff9640 1 -- 172.21.15.59:0/1444863838 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 5305+0+0 (secure 0 0 0) 0x7f238c0977e0 con 0x7f23a40ce950 2024-06-29T15:58:18.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.075+0000 7f23a3fff640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2384076510 0x7f2384078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.076+0000 7f23a3fff640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2384076510 0x7f2384078a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2394034000 tx=0x7f2394004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:18.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.079+0000 7f23a0ff9640 1 -- 172.21.15.59:0/1444863838 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f238c0610d0 con 0x7f23a40ce950 2024-06-29T15:58:18.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.273+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23a40cdc40 con 0x7f23a40ce950 2024-06-29T15:58:18.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.273+0000 7f23a0ff9640 1 -- 172.21.15.59:0/1444863838 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) ==== 74+0+13276 (secure 0 0 0) 0x7f238c0690c0 con 0x7f23a40ce950 2024-06-29T15:58:18.273 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:18.274 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":238,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:17.522785+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.275+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2384076510 msgr2=0x7f2384078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.275+0000 7f23aa518640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2384076510 0x7f2384078a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2394034000 tx=0x7f2394004730 comp rx=0 tx=0).stop 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.275+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a40ce950 msgr2=0x7f23a40cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.275+0000 7f23aa518640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a40ce950 0x7f23a40cae00 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c500 tx=0x7f238c00c9c0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.275+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 shutdown_connections 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.276+0000 7f23aa518640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f2384076510 0x7f2384078a20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.276+0000 7f23aa518640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23a40cf5c0 0x7f23a40cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.276+0000 7f23aa518640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a40ce950 0x7f23a40cae00 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.276+0000 7f23aa518640 1 --2- 172.21.15.59:0/1444863838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23a408c190 0x7f23a4153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.276+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 >> 172.21.15.59:0/1444863838 conn(0x7f23a40b5610 msgr2=0x7f23a4161100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.276+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 shutdown_connections 2024-06-29T15:58:18.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.276+0000 7f23aa518640 1 -- 172.21.15.59:0/1444863838 wait complete. 2024-06-29T15:58:18.287 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:18.287 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 -- 172.21.15.59:0/1858742362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80815ee10 msgr2=0x7fd80815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 --2- 172.21.15.59:0/1858742362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80815ee10 0x7fd80815f9b0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fd8080b4a80 tx=0x7fd80002aa30 comp rx=0 tx=0).stop 2024-06-29T15:58:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 -- 172.21.15.59:0/1858742362 shutdown_connections 2024-06-29T15:58:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 --2- 172.21.15.59:0/1858742362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd80815ee10 0x7fd80815f9b0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 --2- 172.21.15.59:0/1858742362 >> v2:172.21.15.153:3300/0 conn(0x7fd8080b3ed0 0x7fd80815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 --2- 172.21.15.59:0/1858742362 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8080d23c0 0x7fd8080b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 -- 172.21.15.59:0/1858742362 >> 172.21.15.59:0/1858742362 conn(0x7fd8080d3ee0 msgr2=0x7fd8080d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 -- 172.21.15.59:0/1858742362 shutdown_connections 2024-06-29T15:58:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.408+0000 7fd80cae4640 1 -- 172.21.15.59:0/1858742362 wait complete. 2024-06-29T15:58:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.410+0000 7fd80cae4640 1 Processor -- start 2024-06-29T15:58:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.410+0000 7fd80cae4640 1 -- start start 2024-06-29T15:58:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80cae4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8080b3ed0 0x7fd808159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80cae4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8080d23c0 0x7fd808149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80cae4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd80815ee10 0x7fd80815af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80cae4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd808160070 con 0x7fd8080b3ed0 2024-06-29T15:58:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80cae4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd80815fef0 con 0x7fd80815ee10 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80cae4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd8081601f0 con 0x7fd8080d23c0 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8080b3ed0 0x7fd808159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8080b3ed0 0x7fd808159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44316/0 (socket says 172.21.15.59:44316) 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 -- 172.21.15.59:0/3013500619 learned_addr learned my addr 172.21.15.59:0/3013500619 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd805d9b640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8080d23c0 0x7fd808149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd806d9d640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd80815ee10 0x7fd80815af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 -- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8080d23c0 msgr2=0x7fd808149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8080d23c0 0x7fd808149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 -- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd80815ee10 msgr2=0x7fd80815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd80815ee10 0x7fd80815af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 -- 172.21.15.59:0/3013500619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8081599a0 con 0x7fd8080b3ed0 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.412+0000 7fd80659c640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8080b3ed0 0x7fd808159460 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800d750 tx=0x7fd7f800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.413+0000 7fd7f6ffd640 1 -- 172.21.15.59:0/3013500619 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8017070 con 0x7fd8080b3ed0 2024-06-29T15:58:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.413+0000 7fd7f6ffd640 1 -- 172.21.15.59:0/3013500619 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7f8009b20 con 0x7fd8080b3ed0 2024-06-29T15:58:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.413+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd808159eb0 con 0x7fd8080b3ed0 2024-06-29T15:58:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.413+0000 7fd7f6ffd640 1 -- 172.21.15.59:0/3013500619 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8010460 con 0x7fd8080b3ed0 2024-06-29T15:58:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.413+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80815a470 con 0x7fd8080b3ed0 2024-06-29T15:58:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.414+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8080b3990 con 0x7fd8080b3ed0 2024-06-29T15:58:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.415+0000 7fd7f6ffd640 1 -- 172.21.15.59:0/3013500619 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fd7f801a070 con 0x7fd8080b3ed0 2024-06-29T15:58:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.415+0000 7fd7f6ffd640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd7e8076510 0x7fd7e8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.415+0000 7fd7f6ffd640 1 -- 172.21.15.59:0/3013500619 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 5305+0+0 (secure 0 0 0) 0x7fd7f8097b70 con 0x7fd8080b3ed0 2024-06-29T15:58:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.416+0000 7fd805d9b640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd7e8076510 0x7fd7e8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.416+0000 7fd805d9b640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd7e8076510 0x7fd7e8078a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8080b99a0 tx=0x7fd7fc00b260 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:18.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.420+0000 7fd7f6ffd640 1 -- 172.21.15.59:0/3013500619 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd7f8060ba0 con 0x7fd8080b3ed0 2024-06-29T15:58:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.617+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd80815fae0 con 0x7fd8080b3ed0 2024-06-29T15:58:18.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.618+0000 7fd7f6ffd640 1 -- 172.21.15.59:0/3013500619 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v238) ==== 93+0+10 (secure 0 0 0) 0x7fd7f8064ea0 con 0x7fd8080b3ed0 2024-06-29T15:58:18.618 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:58:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd7e8076510 msgr2=0x7fd7e8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd7e8076510 0x7fd7e8078a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8080b99a0 tx=0x7fd7fc00b260 comp rx=0 tx=0).stop 2024-06-29T15:58:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8080b3ed0 msgr2=0x7fd808159460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8080b3ed0 0x7fd808159460 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800d750 tx=0x7fd7f800dc10 comp rx=0 tx=0).stop 2024-06-29T15:58:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 shutdown_connections 2024-06-29T15:58:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd7e8076510 0x7fd7e8078a20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd80815ee10 0x7fd80815af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd8080d23c0 0x7fd808149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 --2- 172.21.15.59:0/3013500619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8080b3ed0 0x7fd808159460 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 >> 172.21.15.59:0/3013500619 conn(0x7fd8080d3ee0 msgr2=0x7fd80806c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 shutdown_connections 2024-06-29T15:58:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.620+0000 7fd80cae4640 1 -- 172.21.15.59:0/3013500619 wait complete. 2024-06-29T15:58:18.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:58:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.751+0000 7fa908fef640 1 -- 172.21.15.59:0/3401130839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f0004d40 msgr2=0x7fa8f0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.751+0000 7fa908fef640 1 --2- 172.21.15.59:0/3401130839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f0004d40 0x7fa8f0005150 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009b50 tx=0x7fa8f0005830 comp rx=0 tx=0).stop 2024-06-29T15:58:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.751+0000 7fa902d9d640 1 -- 172.21.15.59:0/3401130839 reap_dead start 2024-06-29T15:58:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.751+0000 7fa908fef640 1 -- 172.21.15.59:0/3401130839 shutdown_connections 2024-06-29T15:58:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.751+0000 7fa908fef640 1 -- 172.21.15.59:0/3401130839 >> 172.21.15.59:0/3401130839 conn(0x7fa9040b5610 msgr2=0x7fa9040b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.751+0000 7fa908fef640 1 -- 172.21.15.59:0/3401130839 shutdown_connections 2024-06-29T15:58:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.751+0000 7fa908fef640 1 -- 172.21.15.59:0/3401130839 wait complete. 2024-06-29T15:58:18.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.753+0000 7fa908fef640 1 Processor -- start 2024-06-29T15:58:18.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.753+0000 7fa908fef640 1 -- start start 2024-06-29T15:58:18.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa908fef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f0004d40 0x7fa90415c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa908fef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa90415f7d0 0x7fa9040cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa908fef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa904073c30 0x7fa90415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa908fef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa904162650 con 0x7fa904073c30 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa908fef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041624d0 con 0x7fa8f0004d40 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa908fef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa9040b7bd0 con 0x7fa90415f7d0 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa902d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa904073c30 0x7fa90415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa902d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa904073c30 0x7fa90415d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44342/0 (socket says 172.21.15.59:44342) 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa902d9d640 1 -- 172.21.15.59:0/2878430240 learned_addr learned my addr 172.21.15.59:0/2878430240 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa901d9b640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa90415f7d0 0x7fa9040cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.754+0000 7fa90259c640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f0004d40 0x7fa90415c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa902d9d640 1 -- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa90415f7d0 msgr2=0x7fa9040cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa902d9d640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa90415f7d0 0x7fa9040cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa902d9d640 1 -- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f0004d40 msgr2=0x7fa90415c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa902d9d640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f0004d40 0x7fa90415c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa902d9d640 1 -- 172.21.15.59:0/2878430240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90415edc0 con 0x7fa904073c30 2024-06-29T15:58:18.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa901d9b640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa90415f7d0 0x7fa9040cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:18.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa902d9d640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa904073c30 0x7fa90415d7e0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009c70 tx=0x7fa8fc033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:18.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa8faffd640 1 -- 172.21.15.59:0/2878430240 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc038040 con 0x7fa904073c30 2024-06-29T15:58:18.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa8faffd640 1 -- 172.21.15.59:0/2878430240 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8fc004020 con 0x7fa904073c30 2024-06-29T15:58:18.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa8faffd640 1 -- 172.21.15.59:0/2878430240 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc004470 con 0x7fa904073c30 2024-06-29T15:58:18.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90415f0a0 con 0x7fa904073c30 2024-06-29T15:58:18.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.755+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9040ccda0 con 0x7fa904073c30 2024-06-29T15:58:18.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.756+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9040c7e80 con 0x7fa904073c30 2024-06-29T15:58:18.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.757+0000 7fa8faffd640 1 -- 172.21.15.59:0/2878430240 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fa8fc044030 con 0x7fa904073c30 2024-06-29T15:58:18.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.757+0000 7fa8faffd640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa8e4076540 0x7fa8e4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.757+0000 7fa8faffd640 1 -- 172.21.15.59:0/2878430240 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 5305+0+0 (secure 0 0 0) 0x7fa8fc0b7c60 con 0x7fa904073c30 2024-06-29T15:58:18.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.758+0000 7fa90259c640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa8e4076540 0x7fa8e4078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.758+0000 7fa90259c640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa8e4076540 0x7fa8e4078a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4001c00 tx=0x7fa8f4001fc0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.762+0000 7fa8faffd640 1 -- 172.21.15.59:0/2878430240 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa9040c7e80 con 0x7fa904073c30 2024-06-29T15:58:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.958+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa904162c10 con 0x7fa904073c30 2024-06-29T15:58:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.958+0000 7fa8faffd640 1 -- 172.21.15.59:0/2878430240 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v238) ==== 84+0+65 (secure 0 0 0) 0x7fa8fc04b020 con 0x7fa904073c30 2024-06-29T15:58:18.959 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:18.959 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:58:18.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:18.959 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:58:18.959 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:58:18.959 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:18.959 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:58:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa8e4076540 msgr2=0x7fa8e4078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa8e4076540 0x7fa8e4078a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4001c00 tx=0x7fa8f4001fc0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa904073c30 msgr2=0x7fa90415d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa904073c30 0x7fa90415d7e0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009c70 tx=0x7fa8fc033670 comp rx=0 tx=0).stop 2024-06-29T15:58:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 shutdown_connections 2024-06-29T15:58:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa8e4076540 0x7fa8e4078a50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa904073c30 0x7fa90415d7e0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa90415f7d0 0x7fa9040cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 --2- 172.21.15.59:0/2878430240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8f0004d40 0x7fa90415c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 >> 172.21.15.59:0/2878430240 conn(0x7fa9040b5610 msgr2=0x7fa9040c7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 shutdown_connections 2024-06-29T15:58:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:18.961+0000 7fa908fef640 1 -- 172.21.15.59:0/2878430240 wait complete. 2024-06-29T15:58:18.972 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:58:18.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:58:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.095+0000 7f1f525d4640 1 -- 172.21.15.59:0/3620488665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c0b3ed0 msgr2=0x7f1f4c0b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.095+0000 7f1f525d4640 1 --2- 172.21.15.59:0/3620488665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c0b3ed0 0x7f1f4c0b42e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1f44005fd0 tx=0x7f1f4402aa90 comp rx=0 tx=0).stop 2024-06-29T15:58:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.095+0000 7f1f525d4640 1 -- 172.21.15.59:0/3620488665 shutdown_connections 2024-06-29T15:58:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.095+0000 7f1f525d4640 1 --2- 172.21.15.59:0/3620488665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c0b3ed0 0x7f1f4c0b42e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.095+0000 7f1f525d4640 1 --2- 172.21.15.59:0/3620488665 >> v2:172.21.15.153:3300/0 conn(0x7f1f4c0d2780 0x7f1f4c1621f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.095+0000 7f1f525d4640 1 -- 172.21.15.59:0/3620488665 >> 172.21.15.59:0/3620488665 conn(0x7f1f4c0d3e60 msgr2=0x7f1f4c0d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.095+0000 7f1f525d4640 1 -- 172.21.15.59:0/3620488665 shutdown_connections 2024-06-29T15:58:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.095+0000 7f1f525d4640 1 -- 172.21.15.59:0/3620488665 wait complete. 2024-06-29T15:58:19.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.096+0000 7f1f525d4640 1 Processor -- start 2024-06-29T15:58:19.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.097+0000 7f1f525d4640 1 -- start start 2024-06-29T15:58:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f525d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f4c0b3ed0 0x7f1f4c15a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f525d4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4c0d2780 0x7f1f4c0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f525d4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c15dff0 0x7f1f4c0aa430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f525d4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4c0b4820 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f525d4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4c162940 con 0x7f1f4c0d2780 2024-06-29T15:58:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f525d4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1f4c0b49a0 con 0x7f1f4c15dff0 2024-06-29T15:58:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f4bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f4c0b3ed0 0x7f1f4c15a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f4bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f4c0b3ed0 0x7f1f4c15a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44350/0 (socket says 172.21.15.59:44350) 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f4bfff640 1 -- 172.21.15.59:0/2048645190 learned_addr learned my addr 172.21.15.59:0/2048645190 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f50b71640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c15dff0 0x7f1f4c0aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f4b7fe640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4c0d2780 0x7f1f4c0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f4bfff640 1 -- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c15dff0 msgr2=0x7f1f4c0aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f4bfff640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c15dff0 0x7f1f4c0aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f4bfff640 1 -- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4c0d2780 msgr2=0x7f1f4c0cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.098+0000 7f1f4bfff640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4c0d2780 0x7f1f4c0cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f4bfff640 1 -- 172.21.15.59:0/2048645190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4c0cd190 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f50b71640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c15dff0 0x7f1f4c0aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f4b7fe640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4c0d2780 0x7f1f4c0cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f4bfff640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f4c0b3ed0 0x7f1f4c15a1e0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c3a0 tx=0x7f1f3c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f48ff9640 1 -- 172.21.15.59:0/2048645190 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c017070 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f48ff9640 1 -- 172.21.15.59:0/2048645190 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f3c0053c0 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4c0cd690 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f48ff9640 1 -- 172.21.15.59:0/2048645190 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c003850 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.099+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f4c0cdc50 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.100+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f4c0b4e90 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.101+0000 7f1f48ff9640 1 -- 172.21.15.59:0/2048645190 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f1f3c01a070 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.101+0000 7f1f48ff9640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1f38076510 0x7f1f38078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.101+0000 7f1f48ff9640 1 -- 172.21.15.59:0/2048645190 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 5305+0+0 (secure 0 0 0) 0x7f1f3c096d60 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.102+0000 7f1f4b7fe640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1f38076510 0x7f1f38078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.102+0000 7f1f4b7fe640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1f38076510 0x7f1f38078a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c15d800 tx=0x7f1f40005f90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.106+0000 7f1f48ff9640 1 -- 172.21.15.59:0/2048645190 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1f4c0d2b90 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.295+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1f4c0d1080 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.524+0000 7f1f48ff9640 1 -- 172.21.15.59:0/2048645190 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v239) ==== 138+0+0 (secure 0 0 0) 0x7f1f3c0605c0 con 0x7f1f4c0b3ed0 2024-06-29T15:58:19.524 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T15:58:19.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.526+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1f38076510 msgr2=0x7f1f38078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.526+0000 7f1f525d4640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1f38076510 0x7f1f38078a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c15d800 tx=0x7f1f40005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:19.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.526+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f4c0b3ed0 msgr2=0x7f1f4c15a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.526+0000 7f1f525d4640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f4c0b3ed0 0x7f1f4c15a1e0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c3a0 tx=0x7f1f3c00c860 comp rx=0 tx=0).stop 2024-06-29T15:58:19.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.526+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 shutdown_connections 2024-06-29T15:58:19.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.527+0000 7f1f525d4640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1f38076510 0x7f1f38078a20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.527+0000 7f1f525d4640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f4c15dff0 0x7f1f4c0aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.527+0000 7f1f525d4640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4c0d2780 0x7f1f4c0cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.527+0000 7f1f525d4640 1 --2- 172.21.15.59:0/2048645190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f4c0b3ed0 0x7f1f4c15a1e0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.527+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 >> 172.21.15.59:0/2048645190 conn(0x7f1f4c0d3e60 msgr2=0x7f1f4c0b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:19.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.527+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 shutdown_connections 2024-06-29T15:58:19.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.527+0000 7f1f525d4640 1 -- 172.21.15.59:0/2048645190 wait complete. 2024-06-29T15:58:19.542 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:58:19.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.668+0000 7f4f81103640 1 -- 172.21.15.59:0/2971252129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f64004d40 msgr2=0x7f4f64005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.668+0000 7f4f81103640 1 --2- 172.21.15.59:0/2971252129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f64004d40 0x7f4f64005150 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009ed0 tx=0x7f4f64005830 comp rx=0 tx=0).stop 2024-06-29T15:58:19.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.668+0000 7f4f7b59e640 1 -- 172.21.15.59:0/2971252129 reap_dead start 2024-06-29T15:58:19.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.669+0000 7f4f81103640 1 -- 172.21.15.59:0/2971252129 shutdown_connections 2024-06-29T15:58:19.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.669+0000 7f4f81103640 1 -- 172.21.15.59:0/2971252129 >> 172.21.15.59:0/2971252129 conn(0x7f4f7c0d3e40 msgr2=0x7f4f7c0d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:19.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.669+0000 7f4f81103640 1 -- 172.21.15.59:0/2971252129 shutdown_connections 2024-06-29T15:58:19.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.669+0000 7f4f81103640 1 -- 172.21.15.59:0/2971252129 wait complete. 2024-06-29T15:58:19.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.671+0000 7f4f81103640 1 Processor -- start 2024-06-29T15:58:19.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.671+0000 7f4f81103640 1 -- start start 2024-06-29T15:58:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f81103640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f64004d40 0x7f4f7c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f81103640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f7c0ccad0 0x7f4f7c06d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f81103640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c0ccee0 0x7f4f7c06d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f81103640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c0b3fc0 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f81103640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c0b2c20 con 0x7f4f7c0ccee0 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f81103640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c0b4140 con 0x7f4f64004d40 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f7a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f7c0ccad0 0x7f4f7c06d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f7a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f7c0ccad0 0x7f4f7c06d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44360/0 (socket says 172.21.15.59:44360) 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f7a59c640 1 -- 172.21.15.59:0/1979214559 learned_addr learned my addr 172.21.15.59:0/1979214559 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f7ad9d640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f64004d40 0x7f4f7c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.672+0000 7f4f7b59e640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c0ccee0 0x7f4f7c06d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f7a59c640 1 -- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f64004d40 msgr2=0x7f4f7c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f7a59c640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f64004d40 0x7f4f7c15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f7a59c640 1 -- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c0ccee0 msgr2=0x7f4f7c06d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f7a59c640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c0ccee0 0x7f4f7c06d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f7a59c640 1 -- 172.21.15.59:0/1979214559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c0cd640 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f7a59c640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f7c0ccad0 0x7f4f7c06d1f0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f4f70011b10 tx=0x7f4f70009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f637fe640 1 -- 172.21.15.59:0/1979214559 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000fce0 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f637fe640 1 -- 172.21.15.59:0/1979214559 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f7000fe80 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f637fe640 1 -- 172.21.15.59:0/1979214559 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70004280 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7c1f5cf0 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.673+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f7c1f62b0 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.674+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f7c0cd3b0 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.675+0000 7f4f637fe640 1 -- 172.21.15.59:0/1979214559 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4f700035f0 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.676+0000 7f4f637fe640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f68076510 0x7f4f68078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.676+0000 7f4f637fe640 1 -- 172.21.15.59:0/1979214559 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 5675+0+0 (secure 0 0 0) 0x7f4f70098880 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.676+0000 7f4f7ad9d640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f68076510 0x7f4f68078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.677+0000 7f4f7ad9d640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f68076510 0x7f4f68078a20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c001820 tx=0x7f4f6c0012d0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:19.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.681+0000 7f4f637fe640 1 -- 172.21.15.59:0/1979214559 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f70061c60 con 0x7f4f7c0ccad0 2024-06-29T15:58:19.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:19.877+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4f7c0cf000 con 0x7f4f7c0ccad0 2024-06-29T15:58:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.536+0000 7f4f637fe640 1 -- 172.21.15.59:0/1979214559 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v240) ==== 170+0+0 (secure 0 0 0) 0x7f4f70061560 con 0x7f4f7c0ccad0 2024-06-29T15:58:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T15:58:20.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f68076510 msgr2=0x7f4f68078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:20.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f68076510 0x7f4f68078a20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c001820 tx=0x7f4f6c0012d0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f7c0ccad0 msgr2=0x7f4f7c06d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:20.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f7c0ccad0 0x7f4f7c06d1f0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f4f70011b10 tx=0x7f4f70009f10 comp rx=0 tx=0).stop 2024-06-29T15:58:20.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 shutdown_connections 2024-06-29T15:58:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f68076510 0x7f4f68078a20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c0ccee0 0x7f4f7c06d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f7c0ccad0 0x7f4f7c06d1f0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 --2- 172.21.15.59:0/1979214559 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f64004d40 0x7f4f7c15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 >> 172.21.15.59:0/1979214559 conn(0x7f4f7c0d3e40 msgr2=0x7f4f7c0b89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 shutdown_connections 2024-06-29T15:58:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.539+0000 7f4f81103640 1 -- 172.21.15.59:0/1979214559 wait complete. 2024-06-29T15:58:20.567 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:58:20.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.693+0000 7f80a0cc3640 1 -- 172.21.15.59:0/4127240704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0ce950 msgr2=0x7f809c0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:20.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.693+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/4127240704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0ce950 0x7f809c0d2130 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f8094009830 tx=0x7f809402a510 comp rx=0 tx=0).stop 2024-06-29T15:58:20.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.693+0000 7f80a0cc3640 1 -- 172.21.15.59:0/4127240704 shutdown_connections 2024-06-29T15:58:20.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.693+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/4127240704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0ce950 0x7f809c0d2130 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.693+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/4127240704 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f809c08c190 0x7f809c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.693+0000 7f80a0cc3640 1 -- 172.21.15.59:0/4127240704 >> 172.21.15.59:0/4127240704 conn(0x7f809c0b5610 msgr2=0x7f809c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:20.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.693+0000 7f80a0cc3640 1 -- 172.21.15.59:0/4127240704 shutdown_connections 2024-06-29T15:58:20.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.693+0000 7f80a0cc3640 1 -- 172.21.15.59:0/4127240704 wait complete. 2024-06-29T15:58:20.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.695+0000 7f80a0cc3640 1 Processor -- start 2024-06-29T15:58:20.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.695+0000 7f80a0cc3640 1 -- start start 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f80a0cc3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f809c08c190 0x7f809c0cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f80a0cc3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f809c0ce950 0x7f809c14d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f80a0cc3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0cf5c0 0x7f809c15b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f80a0cc3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f809c0b7fa0 con 0x7f809c0cf5c0 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f80a0cc3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f809c0b7e20 con 0x7f809c0ce950 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f80a0cc3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f809c0b8120 con 0x7f809c08c190 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f809ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0cf5c0 0x7f809c15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f809ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0cf5c0 0x7f809c15b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44370/0 (socket says 172.21.15.59:44370) 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f809ad9d640 1 -- 172.21.15.59:0/1884488178 learned_addr learned my addr 172.21.15.59:0/1884488178 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f809a59c640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f809c08c190 0x7f809c0cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.696+0000 7f8099d9b640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f809c0ce950 0x7f809c14d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f809ad9d640 1 -- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f809c08c190 msgr2=0x7f809c0cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f809ad9d640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f809c08c190 0x7f809c0cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f809ad9d640 1 -- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f809c0ce950 msgr2=0x7f809c14d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f809ad9d640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f809c0ce950 0x7f809c14d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f809ad9d640 1 -- 172.21.15.59:0/1884488178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809c14d590 con 0x7f809c0cf5c0 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f809ad9d640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0cf5c0 0x7f809c15b140 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f8094009920 tx=0x7f8094033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f808affd640 1 -- 172.21.15.59:0/1884488178 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094031d60 con 0x7f809c0cf5c0 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f808affd640 1 -- 172.21.15.59:0/1884488178 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8094033e10 con 0x7f809c0cf5c0 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809c0cb620 con 0x7f809c0cf5c0 2024-06-29T15:58:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f808affd640 1 -- 172.21.15.59:0/1884488178 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094033310 con 0x7f809c0cf5c0 2024-06-29T15:58:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.697+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f809c0cbbe0 con 0x7f809c0cf5c0 2024-06-29T15:58:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.698+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f809c0cf080 con 0x7f809c0cf5c0 2024-06-29T15:58:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.699+0000 7f808affd640 1 -- 172.21.15.59:0/1884488178 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f80940334b0 con 0x7f809c0cf5c0 2024-06-29T15:58:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.699+0000 7f808affd640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f807c076510 0x7f807c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.700+0000 7f808affd640 1 -- 172.21.15.59:0/1884488178 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(240..240 src has 1..240) ==== 6057+0+0 (secure 0 0 0) 0x7f80940b9060 con 0x7f809c0cf5c0 2024-06-29T15:58:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.700+0000 7f809a59c640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f807c076510 0x7f807c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:20.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.700+0000 7f809a59c640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f807c076510 0x7f807c078a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f808c005fd0 tx=0x7f808c0095e0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:20.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.704+0000 7f808affd640 1 -- 172.21.15.59:0/1884488178 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f809403f030 con 0x7f809c0cf5c0 2024-06-29T15:58:20.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:20.935+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f809c0cb3b0 con 0x7f809c0cf5c0 2024-06-29T15:58:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.555+0000 7f808affd640 1 -- 172.21.15.59:0/1884488178 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v128) ==== 174+0+0 (secure 0 0 0) 0x7f8094031770 con 0x7f809c0cf5c0 2024-06-29T15:58:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 31 and data pool 32 2024-06-29T15:58:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.561+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f807c076510 msgr2=0x7f807c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.561+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f807c076510 0x7f807c078a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f808c005fd0 tx=0x7f808c0095e0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.561+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0cf5c0 msgr2=0x7f809c15b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.561+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0cf5c0 0x7f809c15b140 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f8094009920 tx=0x7f8094033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.562+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 shutdown_connections 2024-06-29T15:58:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.562+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f807c076510 0x7f807c078a20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.562+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f809c0cf5c0 0x7f809c15b140 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.562+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f809c0ce950 0x7f809c14d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.563+0000 7f80a0cc3640 1 --2- 172.21.15.59:0/1884488178 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f809c08c190 0x7f809c0cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.563+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 >> 172.21.15.59:0/1884488178 conn(0x7f809c0b5610 msgr2=0x7f809c15e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.564+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 shutdown_connections 2024-06-29T15:58:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.564+0000 7f80a0cc3640 1 -- 172.21.15.59:0/1884488178 wait complete. 2024-06-29T15:58:21.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.713+0000 7ff9d2466640 1 -- 172.21.15.59:0/404981261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc162190 msgr2=0x7ff9cc0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.713+0000 7ff9d2466640 1 --2- 172.21.15.59:0/404981261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc162190 0x7ff9cc0b38e0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc08c200 tx=0x7ff9c402a820 comp rx=0 tx=0).stop 2024-06-29T15:58:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.713+0000 7ff9d2466640 1 -- 172.21.15.59:0/404981261 shutdown_connections 2024-06-29T15:58:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.713+0000 7ff9d2466640 1 --2- 172.21.15.59:0/404981261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc162190 0x7ff9cc0b38e0 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.713+0000 7ff9d2466640 1 --2- 172.21.15.59:0/404981261 >> v2:172.21.15.153:3300/0 conn(0x7ff9cc162770 0x7ff9cc0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.713+0000 7ff9d2466640 1 -- 172.21.15.59:0/404981261 >> 172.21.15.59:0/404981261 conn(0x7ff9cc0d45b0 msgr2=0x7ff9cc0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.713+0000 7ff9d2466640 1 -- 172.21.15.59:0/404981261 shutdown_connections 2024-06-29T15:58:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.714+0000 7ff9d2466640 1 -- 172.21.15.59:0/404981261 wait complete. 2024-06-29T15:58:21.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.715+0000 7ff9d2466640 1 Processor -- start 2024-06-29T15:58:21.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.715+0000 7ff9d2466640 1 -- start start 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d2466640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc15e0a0 0x7ff9cc151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d2466640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc162770 0x7ff9cc0c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d2466640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc1564e0 0x7ff9cc0c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d2466640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc0b41b0 con 0x7ff9cc1564e0 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d2466640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc0b4030 con 0x7ff9cc15e0a0 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d2466640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff9cc0b4330 con 0x7ff9cc162770 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc1564e0 0x7ff9cc0c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc1564e0 0x7ff9cc0c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44376/0 (socket says 172.21.15.59:44376) 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 -- 172.21.15.59:0/2005835377 learned_addr learned my addr 172.21.15.59:0/2005835377 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9cbfff640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc15e0a0 0x7ff9cc151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9c3fff640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc162770 0x7ff9cc0c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 -- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc162770 msgr2=0x7ff9cc0c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc162770 0x7ff9cc0c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 -- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc15e0a0 msgr2=0x7ff9cc151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc15e0a0 0x7ff9cc151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 -- 172.21.15.59:0/2005835377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc0c7a90 con 0x7ff9cc1564e0 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9cbfff640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc15e0a0 0x7ff9cc151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.717+0000 7ff9d0a03640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc1564e0 0x7ff9cc0c9880 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7ff9c402ae10 tx=0x7ff9c4032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.718+0000 7ff9c97fa640 1 -- 172.21.15.59:0/2005835377 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4037040 con 0x7ff9cc1564e0 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.718+0000 7ff9c97fa640 1 -- 172.21.15.59:0/2005835377 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9c4042070 con 0x7ff9cc1564e0 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.718+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9cc1f5220 con 0x7ff9cc1564e0 2024-06-29T15:58:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.718+0000 7ff9c97fa640 1 -- 172.21.15.59:0/2005835377 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4003e60 con 0x7ff9cc1564e0 2024-06-29T15:58:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.718+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9cc1f4c60 con 0x7ff9cc1564e0 2024-06-29T15:58:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.719+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9cc0c7c60 con 0x7ff9cc1564e0 2024-06-29T15:58:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.720+0000 7ff9c97fa640 1 -- 172.21.15.59:0/2005835377 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7ff9c40042a0 con 0x7ff9cc1564e0 2024-06-29T15:58:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.720+0000 7ff9c97fa640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff9ac076510 0x7ff9ac078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.720+0000 7ff9c97fa640 1 -- 172.21.15.59:0/2005835377 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(241..241 src has 1..241) ==== 6157+0+0 (secure 0 0 0) 0x7ff9c40b8be0 con 0x7ff9cc1564e0 2024-06-29T15:58:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.720+0000 7ff9cbfff640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff9ac076510 0x7ff9ac078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.721+0000 7ff9cbfff640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff9ac076510 0x7ff9ac078a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009b80 tx=0x7ff9bc005f90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:21.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.725+0000 7ff9c97fa640 1 -- 172.21.15.59:0/2005835377 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff9c4081f10 con 0x7ff9cc1564e0 2024-06-29T15:58:21.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.917+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9cc1f4710 con 0x7ff9cc1564e0 2024-06-29T15:58:21.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.918+0000 7ff9c97fa640 1 -- 172.21.15.59:0/2005835377 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v241) ==== 74+0+17067 (secure 0 0 0) 0x7ff9c4081810 con 0x7ff9cc1564e0 2024-06-29T15:58:21.918 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":241,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:20.935741+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:21.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff9ac076510 msgr2=0x7ff9ac078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff9ac076510 0x7ff9ac078a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009b80 tx=0x7ff9bc005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:21.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc1564e0 msgr2=0x7ff9cc0c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc1564e0 0x7ff9cc0c9880 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7ff9c402ae10 tx=0x7ff9c4032c60 comp rx=0 tx=0).stop 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 shutdown_connections 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff9ac076510 0x7ff9ac078a20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc1564e0 0x7ff9cc0c9880 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc162770 0x7ff9cc0c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 --2- 172.21.15.59:0/2005835377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc15e0a0 0x7ff9cc151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 >> 172.21.15.59:0/2005835377 conn(0x7ff9cc0d45b0 msgr2=0x7ff9cc0ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 shutdown_connections 2024-06-29T15:58:21.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:21.920+0000 7ff9d2466640 1 -- 172.21.15.59:0/2005835377 wait complete. 2024-06-29T15:58:21.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.050+0000 7fdb83812640 1 -- 172.21.15.59:0/2717991106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c15f5a0 msgr2=0x7fdb7c15eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.050+0000 7fdb83812640 1 --2- 172.21.15.59:0/2717991106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c15f5a0 0x7fdb7c15eed0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c005fd0 tx=0x7fdb6c02aa90 comp rx=0 tx=0).stop 2024-06-29T15:58:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.050+0000 7fdb83812640 1 -- 172.21.15.59:0/2717991106 shutdown_connections 2024-06-29T15:58:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.050+0000 7fdb83812640 1 --2- 172.21.15.59:0/2717991106 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb7c0b1b00 0x7fdb7c0b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.050+0000 7fdb83812640 1 --2- 172.21.15.59:0/2717991106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c15f5a0 0x7fdb7c15eed0 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.050+0000 7fdb83812640 1 -- 172.21.15.59:0/2717991106 >> 172.21.15.59:0/2717991106 conn(0x7fdb7c0b06e0 msgr2=0x7fdb7c0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:22.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.050+0000 7fdb83812640 1 -- 172.21.15.59:0/2717991106 shutdown_connections 2024-06-29T15:58:22.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.050+0000 7fdb83812640 1 -- 172.21.15.59:0/2717991106 wait complete. 2024-06-29T15:58:22.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.051+0000 7fdb83812640 1 Processor -- start 2024-06-29T15:58:22.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.052+0000 7fdb83812640 1 -- start start 2024-06-29T15:58:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb83812640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb7c0b1b00 0x7fdb7c0ac080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb83812640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c144350 0x7fdb7c15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb83812640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb7c15f5a0 0x7fdb7c1f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb83812640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c160e50 con 0x7fdb7c144350 2024-06-29T15:58:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb83812640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c160cd0 con 0x7fdb7c15f5a0 2024-06-29T15:58:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb83812640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7c160fd0 con 0x7fdb7c0b1b00 2024-06-29T15:58:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb80dad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c144350 0x7fdb7c15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb80dad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c144350 0x7fdb7c15d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44404/0 (socket says 172.21.15.59:44404) 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb80dad640 1 -- 172.21.15.59:0/2325998496 learned_addr learned my addr 172.21.15.59:0/2325998496 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb815ae640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb7c0b1b00 0x7fdb7c0ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb81daf640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb7c15f5a0 0x7fdb7c1f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb80dad640 1 -- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb7c0b1b00 msgr2=0x7fdb7c0ac080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb80dad640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb7c0b1b00 0x7fdb7c0ac080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb80dad640 1 -- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb7c15f5a0 msgr2=0x7fdb7c1f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb80dad640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb7c15f5a0 0x7fdb7c1f4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.053+0000 7fdb80dad640 1 -- 172.21.15.59:0/2325998496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c1f58c0 con 0x7fdb7c144350 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.054+0000 7fdb80dad640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c144350 0x7fdb7c15d8f0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000c3d0 tx=0x7fdb7000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.054+0000 7fdb79ffb640 1 -- 172.21.15.59:0/2325998496 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70017070 con 0x7fdb7c144350 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.054+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7c1f5ba0 con 0x7fdb7c144350 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.054+0000 7fdb79ffb640 1 -- 172.21.15.59:0/2325998496 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb700053c0 con 0x7fdb7c144350 2024-06-29T15:58:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.054+0000 7fdb79ffb640 1 -- 172.21.15.59:0/2325998496 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70003850 con 0x7fdb7c144350 2024-06-29T15:58:22.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.054+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb7c0c5d60 con 0x7fdb7c144350 2024-06-29T15:58:22.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.055+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb7c1614d0 con 0x7fdb7c144350 2024-06-29T15:58:22.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.056+0000 7fdb79ffb640 1 -- 172.21.15.59:0/2325998496 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fdb7001a070 con 0x7fdb7c144350 2024-06-29T15:58:22.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.056+0000 7fdb79ffb640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdb68076510 0x7fdb68078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.056+0000 7fdb79ffb640 1 -- 172.21.15.59:0/2325998496 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(241..241 src has 1..241) ==== 6157+0+0 (secure 0 0 0) 0x7fdb70097000 con 0x7fdb7c144350 2024-06-29T15:58:22.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.056+0000 7fdb815ae640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdb68076510 0x7fdb68078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.057+0000 7fdb815ae640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdb68076510 0x7fdb68078a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c035000 tx=0x7fdb6c038040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:22.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.061+0000 7fdb79ffb640 1 -- 172.21.15.59:0/2325998496 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdb7c15f9b0 con 0x7fdb7c144350 2024-06-29T15:58:22.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.256+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb7c0b4960 con 0x7fdb7c144350 2024-06-29T15:58:22.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.257+0000 7fdb79ffb640 1 -- 172.21.15.59:0/2325998496 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v241) ==== 74+0+17067 (secure 0 0 0) 0x7fdb7005fed0 con 0x7fdb7c144350 2024-06-29T15:58:22.258 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:22.258 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":241,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:20.935741+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:22.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdb68076510 msgr2=0x7fdb68078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdb68076510 0x7fdb68078a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c035000 tx=0x7fdb6c038040 comp rx=0 tx=0).stop 2024-06-29T15:58:22.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c144350 msgr2=0x7fdb7c15d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c144350 0x7fdb7c15d8f0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000c3d0 tx=0x7fdb7000c890 comp rx=0 tx=0).stop 2024-06-29T15:58:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 shutdown_connections 2024-06-29T15:58:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdb68076510 0x7fdb68078a20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb7c15f5a0 0x7fdb7c1f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb7c144350 0x7fdb7c15d8f0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 --2- 172.21.15.59:0/2325998496 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb7c0b1b00 0x7fdb7c0ac080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.259+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 >> 172.21.15.59:0/2325998496 conn(0x7fdb7c0b06e0 msgr2=0x7fdb7c0b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.260+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 shutdown_connections 2024-06-29T15:58:22.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.260+0000 7fdb83812640 1 -- 172.21.15.59:0/2325998496 wait complete. 2024-06-29T15:58:22.271 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:58:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.392+0000 7fa33cff0640 1 -- 172.21.15.59:0/1447474148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33815ceb0 msgr2=0x7fa3380b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.392+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1447474148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33815ceb0 0x7fa3380b56d0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fa328005fd0 tx=0x7fa32802a970 comp rx=0 tx=0).stop 2024-06-29T15:58:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.392+0000 7fa33cff0640 1 -- 172.21.15.59:0/1447474148 shutdown_connections 2024-06-29T15:58:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.392+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1447474148 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa338141bd0 0x7fa33813edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.392+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1447474148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33815ceb0 0x7fa3380b56d0 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.392+0000 7fa33cff0640 1 -- 172.21.15.59:0/1447474148 >> 172.21.15.59:0/1447474148 conn(0x7fa3380b6fb0 msgr2=0x7fa3380b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.392+0000 7fa33cff0640 1 -- 172.21.15.59:0/1447474148 shutdown_connections 2024-06-29T15:58:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.392+0000 7fa33cff0640 1 -- 172.21.15.59:0/1447474148 wait complete. 2024-06-29T15:58:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.394+0000 7fa33cff0640 1 Processor -- start 2024-06-29T15:58:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.394+0000 7fa33cff0640 1 -- start start 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33cff0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33813e9b0 0x7fa33806f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33cff0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa33815ceb0 0x7fa33813acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33cff0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa338152f70 0x7fa338136e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33cff0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa3380b2a00 con 0x7fa33813e9b0 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33cff0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa3380b2880 con 0x7fa338152f70 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33cff0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa3380b2b80 con 0x7fa33815ceb0 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33813e9b0 0x7fa33806f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33813e9b0 0x7fa33806f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44432/0 (socket says 172.21.15.59:44432) 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 -- 172.21.15.59:0/1362452484 learned_addr learned my addr 172.21.15.59:0/1362452484 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa336d9d640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa338152f70 0x7fa338136e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 -- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa33815ceb0 msgr2=0x7fa33813acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa335d9b640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa33815ceb0 0x7fa33813acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa33815ceb0 0x7fa33813acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 -- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa338152f70 msgr2=0x7fa338136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa338152f70 0x7fa338136e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 -- 172.21.15.59:0/1362452484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33813b210 con 0x7fa33813e9b0 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa336d9d640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa338152f70 0x7fa338136e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa335d9b640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa33815ceb0 0x7fa33813acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33659c640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33813e9b0 0x7fa33806f710 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fa328006010 tx=0x7fa328003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa32effd640 1 -- 172.21.15.59:0/1362452484 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328034050 con 0x7fa33813e9b0 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa32effd640 1 -- 172.21.15.59:0/1362452484 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa328004690 con 0x7fa33813e9b0 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33813b720 con 0x7fa33813e9b0 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.396+0000 7fa32effd640 1 -- 172.21.15.59:0/1362452484 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3280436a0 con 0x7fa33813e9b0 2024-06-29T15:58:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.397+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa33806ef10 con 0x7fa33813e9b0 2024-06-29T15:58:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.397+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa33813edc0 con 0x7fa33813e9b0 2024-06-29T15:58:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.398+0000 7fa32effd640 1 -- 172.21.15.59:0/1362452484 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fa328037020 con 0x7fa33813e9b0 2024-06-29T15:58:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.399+0000 7fa32effd640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa318076540 0x7fa318078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.399+0000 7fa32effd640 1 -- 172.21.15.59:0/1362452484 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(241..241 src has 1..241) ==== 6157+0+0 (secure 0 0 0) 0x7fa3280b9260 con 0x7fa33813e9b0 2024-06-29T15:58:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.399+0000 7fa335d9b640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa318076540 0x7fa318078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.400+0000 7fa335d9b640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa318076540 0x7fa318078a50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa33815c7c0 tx=0x7fa320008040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.403+0000 7fa32effd640 1 -- 172.21.15.59:0/1362452484 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa328082590 con 0x7fa33813e9b0 2024-06-29T15:58:22.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:22.636+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa33815c7f0 con 0x7fa33813e9b0 2024-06-29T15:58:23.523 INFO:tasks.ceph.mds.d.smithi153.stderr:2024-06-29T15:58:23.523+0000 7efc5f023640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:23.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.565+0000 7fa32effd640 1 -- 172.21.15.59:0/1362452484 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v131) ==== 118+0+0 (secure 0 0 0) 0x7fa328081e90 con 0x7fa33813e9b0 2024-06-29T15:58:23.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa318076540 msgr2=0x7fa318078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa318076540 0x7fa318078a50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa33815c7c0 tx=0x7fa320008040 comp rx=0 tx=0).stop 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33813e9b0 msgr2=0x7fa33806f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33813e9b0 0x7fa33806f710 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fa328006010 tx=0x7fa328003f90 comp rx=0 tx=0).stop 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 shutdown_connections 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa318076540 0x7fa318078a50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa338152f70 0x7fa338136e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa33815ceb0 0x7fa33813acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 --2- 172.21.15.59:0/1362452484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa33813e9b0 0x7fa33806f710 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 >> 172.21.15.59:0/1362452484 conn(0x7fa3380b6fb0 msgr2=0x7fa33815ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 shutdown_connections 2024-06-29T15:58:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.568+0000 7fa33cff0640 1 -- 172.21.15.59:0/1362452484 wait complete. 2024-06-29T15:58:23.583 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:58:23.583 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:58:23.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.711+0000 7f062b09e640 1 -- 172.21.15.59:0/1452299299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 msgr2=0x7f06240b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.711+0000 7f062b09e640 1 --2- 172.21.15.59:0/1452299299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 0x7f06240b3b50 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f060c005fd0 tx=0x7f060c02aa90 comp rx=0 tx=0).stop 2024-06-29T15:58:23.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.711+0000 7f062b09e640 1 -- 172.21.15.59:0/1452299299 shutdown_connections 2024-06-29T15:58:23.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.711+0000 7f062b09e640 1 --2- 172.21.15.59:0/1452299299 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f062415c2f0 0x7f062415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.711+0000 7f062b09e640 1 --2- 172.21.15.59:0/1452299299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 0x7f06240b3b50 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.711+0000 7f062b09e640 1 -- 172.21.15.59:0/1452299299 >> 172.21.15.59:0/1452299299 conn(0x7f062406c950 msgr2=0x7f0624073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:23.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.711+0000 7f062b09e640 1 -- 172.21.15.59:0/1452299299 shutdown_connections 2024-06-29T15:58:23.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.711+0000 7f062b09e640 1 -- 172.21.15.59:0/1452299299 wait complete. 2024-06-29T15:58:23.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.713+0000 7f062b09e640 1 Processor -- start 2024-06-29T15:58:23.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.713+0000 7f062b09e640 1 -- start start 2024-06-29T15:58:23.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f062b09e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 0x7f06240ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f062b09e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f062415c2f0 0x7f06240ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f062b09e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06240d1900 0x7f06240d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f062b09e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f06240b5f80 con 0x7f06240b3740 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f062b09e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06240b5e00 con 0x7f06240d1900 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f062b09e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f06240b6100 con 0x7f062415c2f0 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f0628e3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 0x7f06240ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f0628e3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 0x7f06240ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35866/0 (socket says 172.21.15.59:35866) 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.714+0000 7f0628e3a640 1 -- 172.21.15.59:0/356571164 learned_addr learned my addr 172.21.15.59:0/356571164 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f062963b640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06240d1900 0x7f06240d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f0623fff640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f062415c2f0 0x7f06240ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f0628e3a640 1 -- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f062415c2f0 msgr2=0x7f06240ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f0628e3a640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f062415c2f0 0x7f06240ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f0628e3a640 1 -- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06240d1900 msgr2=0x7f06240d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f0628e3a640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06240d1900 0x7f06240d1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f0628e3a640 1 -- 172.21.15.59:0/356571164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06240d2250 con 0x7f06240b3740 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f0623fff640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f062415c2f0 0x7f06240ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f062963b640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06240d1900 0x7f06240d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f0628e3a640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 0x7f06240ae2e0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f061400c3d0 tx=0x7f061400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f06217fa640 1 -- 172.21.15.59:0/356571164 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614017070 con 0x7f06240b3740 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f06217fa640 1 -- 172.21.15.59:0/356571164 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06140053c0 con 0x7f06240b3740 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06240b09e0 con 0x7f06240b3740 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f06217fa640 1 -- 172.21.15.59:0/356571164 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614003850 con 0x7f06240b3740 2024-06-29T15:58:23.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.715+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06240b0fa0 con 0x7f06240b3740 2024-06-29T15:58:23.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.717+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06240b0740 con 0x7f06240b3740 2024-06-29T15:58:23.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.717+0000 7f06217fa640 1 -- 172.21.15.59:0/356571164 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f061401a070 con 0x7f06240b3740 2024-06-29T15:58:23.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.718+0000 7f06217fa640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0604076540 0x7f0604078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.718+0000 7f06217fa640 1 -- 172.21.15.59:0/356571164 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f0614097070 con 0x7f06240b3740 2024-06-29T15:58:23.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.719+0000 7f0623fff640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0604076540 0x7f0604078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.720+0000 7f0623fff640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0604076540 0x7f0604078a50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f060c035000 tx=0x7f060c038040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:23.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.723+0000 7f06217fa640 1 -- 172.21.15.59:0/356571164 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f06240b0740 con 0x7f06240b3740 2024-06-29T15:58:23.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:23.956+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f06240b6620 con 0x7f06240b3740 2024-06-29T15:58:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.567+0000 7f06217fa640 1 -- 172.21.15.59:0/356571164 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v132) ==== 122+0+0 (secure 0 0 0) 0x7f06240d2250 con 0x7f06240b3740 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.569+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0604076540 msgr2=0x7f0604078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.569+0000 7f062b09e640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0604076540 0x7f0604078a50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f060c035000 tx=0x7f060c038040 comp rx=0 tx=0).stop 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.569+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 msgr2=0x7f06240ae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.569+0000 7f062b09e640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 0x7f06240ae2e0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f061400c3d0 tx=0x7f061400c890 comp rx=0 tx=0).stop 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.570+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 shutdown_connections 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.570+0000 7f062b09e640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f0604076540 0x7f0604078a50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.570+0000 7f062b09e640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06240d1900 0x7f06240d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.570+0000 7f062b09e640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f062415c2f0 0x7f06240ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.570+0000 7f062b09e640 1 --2- 172.21.15.59:0/356571164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06240b3740 0x7f06240ae2e0 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.570+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 >> 172.21.15.59:0/356571164 conn(0x7f062406c950 msgr2=0x7f0624073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.570+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 shutdown_connections 2024-06-29T15:58:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.570+0000 7f062b09e640 1 -- 172.21.15.59:0/356571164 wait complete. 2024-06-29T15:58:24.584 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:58:24.584 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:24.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.709+0000 7fba07a95640 1 -- 172.21.15.59:0/286195690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000b2fa0 msgr2=0x7fba000d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.709+0000 7fba07a95640 1 --2- 172.21.15.59:0/286195690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000b2fa0 0x7fba000d1520 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009830 tx=0x7fb9f402a510 comp rx=0 tx=0).stop 2024-06-29T15:58:24.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.709+0000 7fba07a95640 1 -- 172.21.15.59:0/286195690 shutdown_connections 2024-06-29T15:58:24.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.709+0000 7fba07a95640 1 --2- 172.21.15.59:0/286195690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000b2fa0 0x7fba000d1520 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.709+0000 7fba07a95640 1 -- 172.21.15.59:0/286195690 >> 172.21.15.59:0/286195690 conn(0x7fba00073e10 msgr2=0x7fba00074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:24.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.709+0000 7fba07a95640 1 -- 172.21.15.59:0/286195690 shutdown_connections 2024-06-29T15:58:24.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.709+0000 7fba07a95640 1 -- 172.21.15.59:0/286195690 wait complete. 2024-06-29T15:58:24.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.711+0000 7fba07a95640 1 Processor -- start 2024-06-29T15:58:24.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.711+0000 7fba07a95640 1 -- start start 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.712+0000 7fba07a95640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba000b2270 0x7fba00151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba07a95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba000b2fa0 0x7fba00153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba07a95640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000d1a60 0x7fba00154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba07a95640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fba000b65d0 con 0x7fba000d1a60 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba07a95640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba000b6450 con 0x7fba000b2fa0 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba07a95640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fba000b6750 con 0x7fba000b2270 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000d1a60 0x7fba00154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000d1a60 0x7fba00154210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35894/0 (socket says 172.21.15.59:35894) 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 -- 172.21.15.59:0/3716028612 learned_addr learned my addr 172.21.15.59:0/3716028612 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba05030640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba000b2fa0 0x7fba00153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba05831640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba000b2270 0x7fba00151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 -- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba000b2270 msgr2=0x7fba00151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba000b2270 0x7fba00151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 -- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba000b2fa0 msgr2=0x7fba00153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba000b2fa0 0x7fba00153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 -- 172.21.15.59:0/3716028612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0015cb50 con 0x7fba000d1a60 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba06032640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000d1a60 0x7fba00154210 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000b9a0 tx=0x7fb9f000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fb9fe7fc640 1 -- 172.21.15.59:0/3716028612 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0019070 con 0x7fba000d1a60 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fb9fe7fc640 1 -- 172.21.15.59:0/3716028612 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9f000cc90 con 0x7fba000d1a60 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0015ce30 con 0x7fba000d1a60 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fb9fe7fc640 1 -- 172.21.15.59:0/3716028612 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f00146a0 con 0x7fba000d1a60 2024-06-29T15:58:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.713+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0015d3f0 con 0x7fba000d1a60 2024-06-29T15:58:24.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.714+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba00069ca0 con 0x7fba000d1a60 2024-06-29T15:58:24.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.715+0000 7fb9fe7fc640 1 -- 172.21.15.59:0/3716028612 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fb9f001c070 con 0x7fba000d1a60 2024-06-29T15:58:24.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.716+0000 7fb9fe7fc640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb9ec076510 0x7fb9ec078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:24.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.716+0000 7fb9fe7fc640 1 -- 172.21.15.59:0/3716028612 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fb9f0097a30 con 0x7fba000d1a60 2024-06-29T15:58:24.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.716+0000 7fba05831640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb9ec076510 0x7fb9ec078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:24.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.716+0000 7fba05831640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb9ec076510 0x7fb9ec078a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8000f60 tx=0x7fb9f8009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:24.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.720+0000 7fb9fe7fc640 1 -- 172.21.15.59:0/3716028612 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb9f0060e70 con 0x7fba000d1a60 2024-06-29T15:58:24.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.947+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba000b6cf0 con 0x7fba000d1a60 2024-06-29T15:58:24.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.948+0000 7fb9fe7fc640 1 -- 172.21.15.59:0/3716028612 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+5243 (secure 0 0 0) 0x7fb9f0060770 con 0x7fba000d1a60 2024-06-29T15:58:24.948 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:24.948 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":132,"btime":"2024-06-29T15:58:24:565730+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126},{"gid":12303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:20.935697+0000","modified":"2024-06-29T15:58:23.956772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/177893499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":177893499},{"type":"v1","addr":"172.21.15.153:6837","nonce":177893499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12318,"qdb_cluster":[12318]},"id":15}]} 2024-06-29T15:58:24.949 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 132 2024-06-29T15:58:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb9ec076510 msgr2=0x7fb9ec078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb9ec076510 0x7fb9ec078a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8000f60 tx=0x7fb9f8009040 comp rx=0 tx=0).stop 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000d1a60 msgr2=0x7fba00154210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000d1a60 0x7fba00154210 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000b9a0 tx=0x7fb9f000be60 comp rx=0 tx=0).stop 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 shutdown_connections 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fb9ec076510 0x7fb9ec078a20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba000d1a60 0x7fba00154210 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba000b2fa0 0x7fba00153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 --2- 172.21.15.59:0/3716028612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba000b2270 0x7fba00151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 >> 172.21.15.59:0/3716028612 conn(0x7fba00073e10 msgr2=0x7fba00070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 shutdown_connections 2024-06-29T15:58:24.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:24.950+0000 7fba07a95640 1 -- 172.21.15.59:0/3716028612 wait complete. 2024-06-29T15:58:24.962 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:25.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.080+0000 7f911c7a1640 1 -- 172.21.15.59:0/3868454646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114162190 msgr2=0x7f91140ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.080+0000 7f911c7a1640 1 --2- 172.21.15.59:0/3868454646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114162190 0x7f91140ce550 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f91140b29a0 tx=0x7f910402a4b0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.080+0000 7f911c7a1640 1 -- 172.21.15.59:0/3868454646 shutdown_connections 2024-06-29T15:58:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.080+0000 7f911c7a1640 1 --2- 172.21.15.59:0/3868454646 >> v2:172.21.15.153:3300/0 conn(0x7f911415e0a0 0x7f91140b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.080+0000 7f911c7a1640 1 --2- 172.21.15.59:0/3868454646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9114162190 0x7f91140ce550 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.080+0000 7f911c7a1640 1 -- 172.21.15.59:0/3868454646 >> 172.21.15.59:0/3868454646 conn(0x7f91140d45b0 msgr2=0x7f91140d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.080+0000 7f911c7a1640 1 -- 172.21.15.59:0/3868454646 shutdown_connections 2024-06-29T15:58:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.080+0000 7f911c7a1640 1 -- 172.21.15.59:0/3868454646 wait complete. 2024-06-29T15:58:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.082+0000 7f911c7a1640 1 Processor -- start 2024-06-29T15:58:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.082+0000 7f911c7a1640 1 -- start start 2024-06-29T15:58:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911c7a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911415e0a0 0x7f91140c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911c7a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9114162770 0x7f91140c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911c7a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9114152be0 0x7f91141523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911c7a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f91140b4270 con 0x7f911415e0a0 2024-06-29T15:58:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911c7a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f91140b40f0 con 0x7f9114152be0 2024-06-29T15:58:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911c7a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f91140b43f0 con 0x7f9114162770 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911a53d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911415e0a0 0x7f91140c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911a53d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911415e0a0 0x7f91140c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35902/0 (socket says 172.21.15.59:35902) 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911a53d640 1 -- 172.21.15.59:0/1881291599 learned_addr learned my addr 172.21.15.59:0/1881291599 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911a53d640 1 -- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9114162770 msgr2=0x7f91140c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f9119d3c640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9114162770 0x7f91140c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911ad3e640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9114152be0 0x7f91141523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911a53d640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9114162770 0x7f91140c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911a53d640 1 -- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9114152be0 msgr2=0x7f91141523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911a53d640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9114152be0 0x7f91141523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.083+0000 7f911a53d640 1 -- 172.21.15.59:0/1881291599 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91140c9ab0 con 0x7f911415e0a0 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.084+0000 7f911a53d640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911415e0a0 0x7f91140c8f80 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f911000b3e0 tx=0x7f911000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.084+0000 7f910affd640 1 -- 172.21.15.59:0/1881291599 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9110015020 con 0x7f911415e0a0 2024-06-29T15:58:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.084+0000 7f910affd640 1 -- 172.21.15.59:0/1881291599 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9110011940 con 0x7f911415e0a0 2024-06-29T15:58:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.084+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91140c97d0 con 0x7f911415e0a0 2024-06-29T15:58:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.084+0000 7f910affd640 1 -- 172.21.15.59:0/1881291599 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9110011d90 con 0x7f911415e0a0 2024-06-29T15:58:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.084+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91140c76b0 con 0x7f911415e0a0 2024-06-29T15:58:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.085+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91140b4990 con 0x7f911415e0a0 2024-06-29T15:58:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.086+0000 7f910affd640 1 -- 172.21.15.59:0/1881291599 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f9110018020 con 0x7f911415e0a0 2024-06-29T15:58:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.086+0000 7f910affd640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f90f8076510 0x7f90f8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.086+0000 7f910affd640 1 -- 172.21.15.59:0/1881291599 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f9110097dd0 con 0x7f911415e0a0 2024-06-29T15:58:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.087+0000 7f9119d3c640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f90f8076510 0x7f90f8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.087+0000 7f9119d3c640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f90f8076510 0x7f90f8078a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9104033000 tx=0x7f9104004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.091+0000 7f910affd640 1 -- 172.21.15.59:0/1881291599 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f911001f020 con 0x7f911415e0a0 2024-06-29T15:58:25.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.285+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91140cdd70 con 0x7f911415e0a0 2024-06-29T15:58:25.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.286+0000 7f910affd640 1 -- 172.21.15.59:0/1881291599 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) ==== 74+0+17054 (secure 0 0 0) 0x7f9110099090 con 0x7f911415e0a0 2024-06-29T15:58:25.287 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:25.287 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":242,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:22.538542+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.288+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f90f8076510 msgr2=0x7f90f8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f90f8076510 0x7f90f8078a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9104033000 tx=0x7f9104004730 comp rx=0 tx=0).stop 2024-06-29T15:58:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911415e0a0 msgr2=0x7f91140c8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911415e0a0 0x7f91140c8f80 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f911000b3e0 tx=0x7f911000b8a0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 shutdown_connections 2024-06-29T15:58:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f90f8076510 0x7f90f8078a20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9114152be0 0x7f91141523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9114162770 0x7f91140c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 --2- 172.21.15.59:0/1881291599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f911415e0a0 0x7f91140c8f80 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 >> 172.21.15.59:0/1881291599 conn(0x7f91140d45b0 msgr2=0x7f911415c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 shutdown_connections 2024-06-29T15:58:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.289+0000 7f911c7a1640 1 -- 172.21.15.59:0/1881291599 wait complete. 2024-06-29T15:58:25.300 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:58:25.300 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.418+0000 7fcdcf99f640 1 -- 172.21.15.59:0/943152742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc815ee10 msgr2=0x7fcdc815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.418+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/943152742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc815ee10 0x7fcdc815f9b0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0009830 tx=0x7fcdc002a510 comp rx=0 tx=0).stop 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.418+0000 7fcdcf99f640 1 -- 172.21.15.59:0/943152742 shutdown_connections 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.418+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/943152742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc815ee10 0x7fcdc815f9b0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.418+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/943152742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdc80b40a0 0x7fcdc815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.418+0000 7fcdcf99f640 1 -- 172.21.15.59:0/943152742 >> 172.21.15.59:0/943152742 conn(0x7fcdc80d3ee0 msgr2=0x7fcdc80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.419+0000 7fcdcf99f640 1 -- 172.21.15.59:0/943152742 shutdown_connections 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.419+0000 7fcdcf99f640 1 -- 172.21.15.59:0/943152742 wait complete. 2024-06-29T15:58:25.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.420+0000 7fcdcf99f640 1 Processor -- start 2024-06-29T15:58:25.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.420+0000 7fcdcf99f640 1 -- start start 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcf99f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdc80d23c0 0x7fcdc81563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcf99f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc815ee10 0x7fcdc80ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcf99f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdc80ca5b0 0x7fcdc814eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcf99f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc8160280 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcf99f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc8160100 con 0x7fcdc815ee10 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcf99f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc8160400 con 0x7fcdc80d23c0 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdc80ca5b0 0x7fcdc814eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdc80ca5b0 0x7fcdc814eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35908/0 (socket says 172.21.15.59:35908) 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 -- 172.21.15.59:0/1594364298 learned_addr learned my addr 172.21.15.59:0/1594364298 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdccf3a640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc815ee10 0x7fcdc80ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcd73b640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdc80d23c0 0x7fcdc81563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 -- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdc80d23c0 msgr2=0x7fcdc81563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdc80d23c0 0x7fcdc81563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 -- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc815ee10 msgr2=0x7fcdc80ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc815ee10 0x7fcdc80ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 -- 172.21.15.59:0/1594364298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc80cd4d0 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.422+0000 7fcdcdf3c640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdc80ca5b0 0x7fcdc814eae0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fcdc002ab00 tx=0x7fcdc0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.423+0000 7fcdc5ffb640 1 -- 172.21.15.59:0/1594364298 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0005c90 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.423+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc8156910 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.423+0000 7fcdc5ffb640 1 -- 172.21.15.59:0/1594364298 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdc0031e50 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.423+0000 7fcdc5ffb640 1 -- 172.21.15.59:0/1594364298 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0005220 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.423+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc8156ec0 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.424+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc815e8d0 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.425+0000 7fcdc5ffb640 1 -- 172.21.15.59:0/1594364298 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fcdc00053c0 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.425+0000 7fcdc5ffb640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcdb4076510 0x7fcdb4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.425+0000 7fcdc5ffb640 1 -- 172.21.15.59:0/1594364298 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fcdc00b8d50 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.425+0000 7fcdcd73b640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcdb4076510 0x7fcdb4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.426+0000 7fcdcd73b640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcdb4076510 0x7fcdb4078a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcdb80017a0 tx=0x7fcdb8001730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.430+0000 7fcdc5ffb640 1 -- 172.21.15.59:0/1594364298 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcdc00bc030 con 0x7fcdc80ca5b0 2024-06-29T15:58:25.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.619+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 --> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcdc8158260 con 0x7fcdb4076510 2024-06-29T15:58:25.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.624+0000 7fcdc5ffb640 1 -- 172.21.15.59:0/1594364298 <== mgr.9710 v2:172.21.15.153:6832/1831136380 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7fcdc8158260 con 0x7fcdb4076510 2024-06-29T15:58:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:25.628 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcdb4076510 msgr2=0x7fcdb4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcdb4076510 0x7fcdb4078a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcdb80017a0 tx=0x7fcdb8001730 comp rx=0 tx=0).stop 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdc80ca5b0 msgr2=0x7fcdc814eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdc80ca5b0 0x7fcdc814eae0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fcdc002ab00 tx=0x7fcdc0033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 shutdown_connections 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcdb4076510 0x7fcdb4078a20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdc80ca5b0 0x7fcdc814eae0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcdc815ee10 0x7fcdc80ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 --2- 172.21.15.59:0/1594364298 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcdc80d23c0 0x7fcdc81563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 >> 172.21.15.59:0/1594364298 conn(0x7fcdc80d3ee0 msgr2=0x7fcdc806c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 shutdown_connections 2024-06-29T15:58:25.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.627+0000 7fcdcf99f640 1 -- 172.21.15.59:0/1594364298 wait complete. 2024-06-29T15:58:25.639 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-06-29T15:58:25.204876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":446,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":286,"ondisk_log_size":286,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":422996,"kb_used_data":25000,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754551724,"statfs":{"total":773094113280,"available":772660965376,"internally_reserved":0,"allocated":25600000,"data_stored":13771284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":407424562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.676214"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799510+0000","last_change":"2024-06-29T15:58:21.556650+0000","last_active":"2024-06-29T15:58:22.799510+0000","last_peered":"2024-06-29T15:58:22.799510+0000","last_clean":"2024-06-29T15:58:22.799510+0000","last_became_active":"2024-06-29T15:58:21.556321+0000","last_became_peered":"2024-06-29T15:58:21.556321+0000","last_unstale":"2024-06-29T15:58:22.799510+0000","last_undegraded":"2024-06-29T15:58:22.799510+0000","last_fullsized":"2024-06-29T15:58:22.799510+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:40:17.694204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719577+0000","last_change":"2024-06-29T15:58:21.553068+0000","last_active":"2024-06-29T15:58:23.719577+0000","last_peered":"2024-06-29T15:58:23.719577+0000","last_clean":"2024-06-29T15:58:23.719577+0000","last_became_active":"2024-06-29T15:58:21.552631+0000","last_became_peered":"2024-06-29T15:58:21.552631+0000","last_unstale":"2024-06-29T15:58:23.719577+0000","last_undegraded":"2024-06-29T15:58:23.719577+0000","last_fullsized":"2024-06-29T15:58:23.719577+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:18.380041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542110+0000","last_change":"2024-06-29T15:58:21.553826+0000","last_active":"2024-06-29T15:58:22.542110+0000","last_peered":"2024-06-29T15:58:22.542110+0000","last_clean":"2024-06-29T15:58:22.542110+0000","last_became_active":"2024-06-29T15:58:21.553476+0000","last_became_peered":"2024-06-29T15:58:21.553476+0000","last_unstale":"2024-06-29T15:58:22.542110+0000","last_undegraded":"2024-06-29T15:58:22.542110+0000","last_fullsized":"2024-06-29T15:58:22.542110+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:09:27.869044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542064+0000","last_change":"2024-06-29T15:58:21.556754+0000","last_active":"2024-06-29T15:58:22.542064+0000","last_peered":"2024-06-29T15:58:22.542064+0000","last_clean":"2024-06-29T15:58:22.542064+0000","last_became_active":"2024-06-29T15:58:21.556399+0000","last_became_peered":"2024-06-29T15:58:21.556399+0000","last_unstale":"2024-06-29T15:58:22.542064+0000","last_undegraded":"2024-06-29T15:58:22.542064+0000","last_fullsized":"2024-06-29T15:58:22.542064+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:53:06.570549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547499+0000","last_change":"2024-06-29T15:58:21.558986+0000","last_active":"2024-06-29T15:58:22.547499+0000","last_peered":"2024-06-29T15:58:22.547499+0000","last_clean":"2024-06-29T15:58:22.547499+0000","last_became_active":"2024-06-29T15:58:21.558669+0000","last_became_peered":"2024-06-29T15:58:21.558669+0000","last_unstale":"2024-06-29T15:58:22.547499+0000","last_undegraded":"2024-06-29T15:58:22.547499+0000","last_fullsized":"2024-06-29T15:58:22.547499+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:07.747196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799086+0000","last_change":"2024-06-29T15:58:21.558888+0000","last_active":"2024-06-29T15:58:22.799086+0000","last_peered":"2024-06-29T15:58:22.799086+0000","last_clean":"2024-06-29T15:58:22.799086+0000","last_became_active":"2024-06-29T15:58:21.558578+0000","last_became_peered":"2024-06-29T15:58:21.558578+0000","last_unstale":"2024-06-29T15:58:22.799086+0000","last_undegraded":"2024-06-29T15:58:22.799086+0000","last_fullsized":"2024-06-29T15:58:22.799086+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:30.612798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548295+0000","last_change":"2024-06-29T15:58:21.552220+0000","last_active":"2024-06-29T15:58:22.548295+0000","last_peered":"2024-06-29T15:58:22.548295+0000","last_clean":"2024-06-29T15:58:22.548295+0000","last_became_active":"2024-06-29T15:58:21.551911+0000","last_became_peered":"2024-06-29T15:58:21.551911+0000","last_unstale":"2024-06-29T15:58:22.548295+0000","last_undegraded":"2024-06-29T15:58:22.548295+0000","last_fullsized":"2024-06-29T15:58:22.548295+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:20.005949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547458+0000","last_change":"2024-06-29T15:58:21.558631+0000","last_active":"2024-06-29T15:58:22.547458+0000","last_peered":"2024-06-29T15:58:22.547458+0000","last_clean":"2024-06-29T15:58:22.547458+0000","last_became_active":"2024-06-29T15:58:21.558300+0000","last_became_peered":"2024-06-29T15:58:21.558300+0000","last_unstale":"2024-06-29T15:58:22.547458+0000","last_undegraded":"2024-06-29T15:58:22.547458+0000","last_fullsized":"2024-06-29T15:58:22.547458+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:46.044647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799608+0000","last_change":"2024-06-29T15:58:21.560160+0000","last_active":"2024-06-29T15:58:22.799608+0000","last_peered":"2024-06-29T15:58:22.799608+0000","last_clean":"2024-06-29T15:58:22.799608+0000","last_became_active":"2024-06-29T15:58:21.559735+0000","last_became_peered":"2024-06-29T15:58:21.559735+0000","last_unstale":"2024-06-29T15:58:22.799608+0000","last_undegraded":"2024-06-29T15:58:22.799608+0000","last_fullsized":"2024-06-29T15:58:22.799608+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:33.429459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542216+0000","last_change":"2024-06-29T15:58:21.559778+0000","last_active":"2024-06-29T15:58:22.542216+0000","last_peered":"2024-06-29T15:58:22.542216+0000","last_clean":"2024-06-29T15:58:22.542216+0000","last_became_active":"2024-06-29T15:58:21.559442+0000","last_became_peered":"2024-06-29T15:58:21.559442+0000","last_unstale":"2024-06-29T15:58:22.542216+0000","last_undegraded":"2024-06-29T15:58:22.542216+0000","last_fullsized":"2024-06-29T15:58:22.542216+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:14.817280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542143+0000","last_change":"2024-06-29T15:58:21.560672+0000","last_active":"2024-06-29T15:58:22.542143+0000","last_peered":"2024-06-29T15:58:22.542143+0000","last_clean":"2024-06-29T15:58:22.542143+0000","last_became_active":"2024-06-29T15:58:21.560333+0000","last_became_peered":"2024-06-29T15:58:21.560333+0000","last_unstale":"2024-06-29T15:58:22.542143+0000","last_undegraded":"2024-06-29T15:58:22.542143+0000","last_fullsized":"2024-06-29T15:58:22.542143+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:16.662908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799391+0000","last_change":"2024-06-29T15:58:21.557848+0000","last_active":"2024-06-29T15:58:22.799391+0000","last_peered":"2024-06-29T15:58:22.799391+0000","last_clean":"2024-06-29T15:58:22.799391+0000","last_became_active":"2024-06-29T15:58:21.557516+0000","last_became_peered":"2024-06-29T15:58:21.557516+0000","last_unstale":"2024-06-29T15:58:22.799391+0000","last_undegraded":"2024-06-29T15:58:22.799391+0000","last_fullsized":"2024-06-29T15:58:22.799391+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:26.084704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597951+0000","last_change":"2024-06-29T15:58:21.557388+0000","last_active":"2024-06-29T15:58:22.597951+0000","last_peered":"2024-06-29T15:58:22.597951+0000","last_clean":"2024-06-29T15:58:22.597951+0000","last_became_active":"2024-06-29T15:58:21.557098+0000","last_became_peered":"2024-06-29T15:58:21.557098+0000","last_unstale":"2024-06-29T15:58:22.597951+0000","last_undegraded":"2024-06-29T15:58:22.597951+0000","last_fullsized":"2024-06-29T15:58:22.597951+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:27.148409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719514+0000","last_change":"2024-06-29T15:58:21.567227+0000","last_active":"2024-06-29T15:58:23.719514+0000","last_peered":"2024-06-29T15:58:23.719514+0000","last_clean":"2024-06-29T15:58:23.719514+0000","last_became_active":"2024-06-29T15:58:21.566195+0000","last_became_peered":"2024-06-29T15:58:21.566195+0000","last_unstale":"2024-06-29T15:58:23.719514+0000","last_undegraded":"2024-06-29T15:58:23.719514+0000","last_fullsized":"2024-06-29T15:58:23.719514+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:49.425921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618140+0000","last_change":"2024-06-29T15:58:21.562321+0000","last_active":"2024-06-29T15:58:22.618140+0000","last_peered":"2024-06-29T15:58:22.618140+0000","last_clean":"2024-06-29T15:58:22.618140+0000","last_became_active":"2024-06-29T15:58:21.561988+0000","last_became_peered":"2024-06-29T15:58:21.561988+0000","last_unstale":"2024-06-29T15:58:22.618140+0000","last_undegraded":"2024-06-29T15:58:22.618140+0000","last_fullsized":"2024-06-29T15:58:22.618140+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:27.579111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541807+0000","last_change":"2024-06-29T15:58:21.561992+0000","last_active":"2024-06-29T15:58:22.541807+0000","last_peered":"2024-06-29T15:58:22.541807+0000","last_clean":"2024-06-29T15:58:22.541807+0000","last_became_active":"2024-06-29T15:58:21.561586+0000","last_became_peered":"2024-06-29T15:58:21.561586+0000","last_unstale":"2024-06-29T15:58:22.541807+0000","last_undegraded":"2024-06-29T15:58:22.541807+0000","last_fullsized":"2024-06-29T15:58:22.541807+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:57.843559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598246+0000","last_change":"2024-06-29T15:58:21.559279+0000","last_active":"2024-06-29T15:58:22.598246+0000","last_peered":"2024-06-29T15:58:22.598246+0000","last_clean":"2024-06-29T15:58:22.598246+0000","last_became_active":"2024-06-29T15:58:21.559033+0000","last_became_peered":"2024-06-29T15:58:21.559033+0000","last_unstale":"2024-06-29T15:58:22.598246+0000","last_undegraded":"2024-06-29T15:58:22.598246+0000","last_fullsized":"2024-06-29T15:58:22.598246+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:19.231351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542358+0000","last_change":"2024-06-29T15:58:21.561849+0000","last_active":"2024-06-29T15:58:22.542358+0000","last_peered":"2024-06-29T15:58:22.542358+0000","last_clean":"2024-06-29T15:58:22.542358+0000","last_became_active":"2024-06-29T15:58:21.561051+0000","last_became_peered":"2024-06-29T15:58:21.561051+0000","last_unstale":"2024-06-29T15:58:22.542358+0000","last_undegraded":"2024-06-29T15:58:22.542358+0000","last_fullsized":"2024-06-29T15:58:22.542358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:05:18.245403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618086+0000","last_change":"2024-06-29T15:58:21.561160+0000","last_active":"2024-06-29T15:58:22.618086+0000","last_peered":"2024-06-29T15:58:22.618086+0000","last_clean":"2024-06-29T15:58:22.618086+0000","last_became_active":"2024-06-29T15:58:21.560810+0000","last_became_peered":"2024-06-29T15:58:21.560810+0000","last_unstale":"2024-06-29T15:58:22.618086+0000","last_undegraded":"2024-06-29T15:58:22.618086+0000","last_fullsized":"2024-06-29T15:58:22.618086+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:02.548677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719830+0000","last_change":"2024-06-29T15:58:21.561325+0000","last_active":"2024-06-29T15:58:23.719830+0000","last_peered":"2024-06-29T15:58:23.719830+0000","last_clean":"2024-06-29T15:58:23.719830+0000","last_became_active":"2024-06-29T15:58:21.561021+0000","last_became_peered":"2024-06-29T15:58:21.561021+0000","last_unstale":"2024-06-29T15:58:23.719830+0000","last_undegraded":"2024-06-29T15:58:23.719830+0000","last_fullsized":"2024-06-29T15:58:23.719830+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:25:28.950204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548555+0000","last_change":"2024-06-29T15:58:21.561369+0000","last_active":"2024-06-29T15:58:22.548555+0000","last_peered":"2024-06-29T15:58:22.548555+0000","last_clean":"2024-06-29T15:58:22.548555+0000","last_became_active":"2024-06-29T15:58:21.561021+0000","last_became_peered":"2024-06-29T15:58:21.561021+0000","last_unstale":"2024-06-29T15:58:22.548555+0000","last_undegraded":"2024-06-29T15:58:22.548555+0000","last_fullsized":"2024-06-29T15:58:22.548555+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:35.412665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597914+0000","last_change":"2024-06-29T15:58:21.555481+0000","last_active":"2024-06-29T15:58:22.597914+0000","last_peered":"2024-06-29T15:58:22.597914+0000","last_clean":"2024-06-29T15:58:22.597914+0000","last_became_active":"2024-06-29T15:58:21.553482+0000","last_became_peered":"2024-06-29T15:58:21.553482+0000","last_unstale":"2024-06-29T15:58:22.597914+0000","last_undegraded":"2024-06-29T15:58:22.597914+0000","last_fullsized":"2024-06-29T15:58:22.597914+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:08.904938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618204+0000","last_change":"2024-06-29T15:58:21.560413+0000","last_active":"2024-06-29T15:58:22.618204+0000","last_peered":"2024-06-29T15:58:22.618204+0000","last_clean":"2024-06-29T15:58:22.618204+0000","last_became_active":"2024-06-29T15:58:21.560114+0000","last_became_peered":"2024-06-29T15:58:21.560114+0000","last_unstale":"2024-06-29T15:58:22.618204+0000","last_undegraded":"2024-06-29T15:58:22.618204+0000","last_fullsized":"2024-06-29T15:58:22.618204+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:43:35.054888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597748+0000","last_change":"2024-06-29T15:58:21.562168+0000","last_active":"2024-06-29T15:58:22.597748+0000","last_peered":"2024-06-29T15:58:22.597748+0000","last_clean":"2024-06-29T15:58:22.597748+0000","last_became_active":"2024-06-29T15:58:21.555306+0000","last_became_peered":"2024-06-29T15:58:21.555306+0000","last_unstale":"2024-06-29T15:58:22.597748+0000","last_undegraded":"2024-06-29T15:58:22.597748+0000","last_fullsized":"2024-06-29T15:58:22.597748+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:40.193793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719887+0000","last_change":"2024-06-29T15:58:21.559133+0000","last_active":"2024-06-29T15:58:23.719887+0000","last_peered":"2024-06-29T15:58:23.719887+0000","last_clean":"2024-06-29T15:58:23.719887+0000","last_became_active":"2024-06-29T15:58:21.558649+0000","last_became_peered":"2024-06-29T15:58:21.558649+0000","last_unstale":"2024-06-29T15:58:23.719887+0000","last_undegraded":"2024-06-29T15:58:23.719887+0000","last_fullsized":"2024-06-29T15:58:23.719887+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:28.201954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548124+0000","last_change":"2024-06-29T15:58:21.561677+0000","last_active":"2024-06-29T15:58:22.548124+0000","last_peered":"2024-06-29T15:58:22.548124+0000","last_clean":"2024-06-29T15:58:22.548124+0000","last_became_active":"2024-06-29T15:58:21.561367+0000","last_became_peered":"2024-06-29T15:58:21.561367+0000","last_unstale":"2024-06-29T15:58:22.548124+0000","last_undegraded":"2024-06-29T15:58:22.548124+0000","last_fullsized":"2024-06-29T15:58:22.548124+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:23:28.067754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547690+0000","last_change":"2024-06-29T15:58:21.552197+0000","last_active":"2024-06-29T15:58:22.547690+0000","last_peered":"2024-06-29T15:58:22.547690+0000","last_clean":"2024-06-29T15:58:22.547690+0000","last_became_active":"2024-06-29T15:58:21.551846+0000","last_became_peered":"2024-06-29T15:58:21.551846+0000","last_unstale":"2024-06-29T15:58:22.547690+0000","last_undegraded":"2024-06-29T15:58:22.547690+0000","last_fullsized":"2024-06-29T15:58:22.547690+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:31.983161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799428+0000","last_change":"2024-06-29T15:58:21.560349+0000","last_active":"2024-06-29T15:58:22.799428+0000","last_peered":"2024-06-29T15:58:22.799428+0000","last_clean":"2024-06-29T15:58:22.799428+0000","last_became_active":"2024-06-29T15:58:21.559976+0000","last_became_peered":"2024-06-29T15:58:21.559976+0000","last_unstale":"2024-06-29T15:58:22.799428+0000","last_undegraded":"2024-06-29T15:58:22.799428+0000","last_fullsized":"2024-06-29T15:58:22.799428+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:35:34.378626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720218+0000","last_change":"2024-06-29T15:58:21.567137+0000","last_active":"2024-06-29T15:58:23.720218+0000","last_peered":"2024-06-29T15:58:23.720218+0000","last_clean":"2024-06-29T15:58:23.720218+0000","last_became_active":"2024-06-29T15:58:21.566816+0000","last_became_peered":"2024-06-29T15:58:21.566816+0000","last_unstale":"2024-06-29T15:58:23.720218+0000","last_undegraded":"2024-06-29T15:58:23.720218+0000","last_fullsized":"2024-06-29T15:58:23.720218+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:36.096648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719685+0000","last_change":"2024-06-29T15:58:21.561708+0000","last_active":"2024-06-29T15:58:23.719685+0000","last_peered":"2024-06-29T15:58:23.719685+0000","last_clean":"2024-06-29T15:58:23.719685+0000","last_became_active":"2024-06-29T15:58:21.561360+0000","last_became_peered":"2024-06-29T15:58:21.561360+0000","last_unstale":"2024-06-29T15:58:23.719685+0000","last_undegraded":"2024-06-29T15:58:23.719685+0000","last_fullsized":"2024-06-29T15:58:23.719685+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:17.841782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719243+0000","last_change":"2024-06-29T15:58:21.560378+0000","last_active":"2024-06-29T15:58:23.719243+0000","last_peered":"2024-06-29T15:58:23.719243+0000","last_clean":"2024-06-29T15:58:23.719243+0000","last_became_active":"2024-06-29T15:58:21.560075+0000","last_became_peered":"2024-06-29T15:58:21.560075+0000","last_unstale":"2024-06-29T15:58:23.719243+0000","last_undegraded":"2024-06-29T15:58:23.719243+0000","last_fullsized":"2024-06-29T15:58:23.719243+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:16.886076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799079+0000","last_change":"2024-06-29T15:58:21.561504+0000","last_active":"2024-06-29T15:58:22.799079+0000","last_peered":"2024-06-29T15:58:22.799079+0000","last_clean":"2024-06-29T15:58:22.799079+0000","last_became_active":"2024-06-29T15:58:21.560810+0000","last_became_peered":"2024-06-29T15:58:21.560810+0000","last_unstale":"2024-06-29T15:58:22.799079+0000","last_undegraded":"2024-06-29T15:58:22.799079+0000","last_fullsized":"2024-06-29T15:58:22.799079+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:53:36.634654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719125+0000","last_change":"2024-06-29T15:58:21.562794+0000","last_active":"2024-06-29T15:58:23.719125+0000","last_peered":"2024-06-29T15:58:23.719125+0000","last_clean":"2024-06-29T15:58:23.719125+0000","last_became_active":"2024-06-29T15:58:21.562011+0000","last_became_peered":"2024-06-29T15:58:21.562011+0000","last_unstale":"2024-06-29T15:58:23.719125+0000","last_undegraded":"2024-06-29T15:58:23.719125+0000","last_fullsized":"2024-06-29T15:58:23.719125+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:09.874394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"241'4","reported_seq":18,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799470+0000","last_change":"2024-06-29T15:58:20.549867+0000","last_active":"2024-06-29T15:58:22.799470+0000","last_peered":"2024-06-29T15:58:22.799470+0000","last_clean":"2024-06-29T15:58:22.799470+0000","last_became_active":"2024-06-29T15:58:20.549532+0000","last_became_peered":"2024-06-29T15:58:20.549532+0000","last_unstale":"2024-06-29T15:58:22.799470+0000","last_undegraded":"2024-06-29T15:58:22.799470+0000","last_fullsized":"2024-06-29T15:58:22.799470+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:11.359524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618083+0000","last_change":"2024-06-29T15:58:21.562279+0000","last_active":"2024-06-29T15:58:22.618083+0000","last_peered":"2024-06-29T15:58:22.618083+0000","last_clean":"2024-06-29T15:58:22.618083+0000","last_became_active":"2024-06-29T15:58:21.561962+0000","last_became_peered":"2024-06-29T15:58:21.561962+0000","last_unstale":"2024-06-29T15:58:22.618083+0000","last_undegraded":"2024-06-29T15:58:22.618083+0000","last_fullsized":"2024-06-29T15:58:22.618083+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:43:25.515942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598043+0000","last_change":"2024-06-29T15:58:20.547115+0000","last_active":"2024-06-29T15:58:22.598043+0000","last_peered":"2024-06-29T15:58:22.598043+0000","last_clean":"2024-06-29T15:58:22.598043+0000","last_became_active":"2024-06-29T15:58:20.546778+0000","last_became_peered":"2024-06-29T15:58:20.546778+0000","last_unstale":"2024-06-29T15:58:22.598043+0000","last_undegraded":"2024-06-29T15:58:22.598043+0000","last_fullsized":"2024-06-29T15:58:22.598043+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:05.970706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799428+0000","last_change":"2024-06-29T15:58:21.562748+0000","last_active":"2024-06-29T15:58:22.799428+0000","last_peered":"2024-06-29T15:58:22.799428+0000","last_clean":"2024-06-29T15:58:22.799428+0000","last_became_active":"2024-06-29T15:58:21.562417+0000","last_became_peered":"2024-06-29T15:58:21.562417+0000","last_unstale":"2024-06-29T15:58:22.799428+0000","last_undegraded":"2024-06-29T15:58:22.799428+0000","last_fullsized":"2024-06-29T15:58:22.799428+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:27.973848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720413+0000","last_change":"2024-06-29T15:58:20.568004+0000","last_active":"2024-06-29T15:58:23.720413+0000","last_peered":"2024-06-29T15:58:23.720413+0000","last_clean":"2024-06-29T15:58:23.720413+0000","last_became_active":"2024-06-29T15:58:20.567638+0000","last_became_peered":"2024-06-29T15:58:20.567638+0000","last_unstale":"2024-06-29T15:58:23.720413+0000","last_undegraded":"2024-06-29T15:58:23.720413+0000","last_fullsized":"2024-06-29T15:58:23.720413+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:19.772821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720384+0000","last_change":"2024-06-29T15:58:21.566347+0000","last_active":"2024-06-29T15:58:23.720384+0000","last_peered":"2024-06-29T15:58:23.720384+0000","last_clean":"2024-06-29T15:58:23.720384+0000","last_became_active":"2024-06-29T15:58:21.565888+0000","last_became_peered":"2024-06-29T15:58:21.565888+0000","last_unstale":"2024-06-29T15:58:23.720384+0000","last_undegraded":"2024-06-29T15:58:23.720384+0000","last_fullsized":"2024-06-29T15:58:23.720384+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:28.461061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618013+0000","last_change":"2024-06-29T15:58:20.556653+0000","last_active":"2024-06-29T15:58:22.618013+0000","last_peered":"2024-06-29T15:58:22.618013+0000","last_clean":"2024-06-29T15:58:22.618013+0000","last_became_active":"2024-06-29T15:58:20.555367+0000","last_became_peered":"2024-06-29T15:58:20.555367+0000","last_unstale":"2024-06-29T15:58:22.618013+0000","last_undegraded":"2024-06-29T15:58:22.618013+0000","last_fullsized":"2024-06-29T15:58:22.618013+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:31.173223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548678+0000","last_change":"2024-06-29T15:58:21.562456+0000","last_active":"2024-06-29T15:58:22.548678+0000","last_peered":"2024-06-29T15:58:22.548678+0000","last_clean":"2024-06-29T15:58:22.548678+0000","last_became_active":"2024-06-29T15:58:21.562024+0000","last_became_peered":"2024-06-29T15:58:21.562024+0000","last_unstale":"2024-06-29T15:58:22.548678+0000","last_undegraded":"2024-06-29T15:58:22.548678+0000","last_fullsized":"2024-06-29T15:58:22.548678+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:36.495502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719745+0000","last_change":"2024-06-29T15:58:20.558360+0000","last_active":"2024-06-29T15:58:23.719745+0000","last_peered":"2024-06-29T15:58:23.719745+0000","last_clean":"2024-06-29T15:58:23.719745+0000","last_became_active":"2024-06-29T15:58:20.558053+0000","last_became_peered":"2024-06-29T15:58:20.558053+0000","last_unstale":"2024-06-29T15:58:23.719745+0000","last_undegraded":"2024-06-29T15:58:23.719745+0000","last_fullsized":"2024-06-29T15:58:23.719745+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:40.258636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542198+0000","last_change":"2024-06-29T15:58:21.568464+0000","last_active":"2024-06-29T15:58:22.542198+0000","last_peered":"2024-06-29T15:58:22.542198+0000","last_clean":"2024-06-29T15:58:22.542198+0000","last_became_active":"2024-06-29T15:58:21.568114+0000","last_became_peered":"2024-06-29T15:58:21.568114+0000","last_unstale":"2024-06-29T15:58:22.542198+0000","last_undegraded":"2024-06-29T15:58:22.542198+0000","last_fullsized":"2024-06-29T15:58:22.542198+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:09.625056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541676+0000","last_change":"2024-06-29T15:58:20.563140+0000","last_active":"2024-06-29T15:58:22.541676+0000","last_peered":"2024-06-29T15:58:22.541676+0000","last_clean":"2024-06-29T15:58:22.541676+0000","last_became_active":"2024-06-29T15:58:20.562823+0000","last_became_peered":"2024-06-29T15:58:20.562823+0000","last_unstale":"2024-06-29T15:58:22.541676+0000","last_undegraded":"2024-06-29T15:58:22.541676+0000","last_fullsized":"2024-06-29T15:58:22.541676+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:24.191012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548052+0000","last_change":"2024-06-29T15:58:21.556649+0000","last_active":"2024-06-29T15:58:22.548052+0000","last_peered":"2024-06-29T15:58:22.548052+0000","last_clean":"2024-06-29T15:58:22.548052+0000","last_became_active":"2024-06-29T15:58:21.556297+0000","last_became_peered":"2024-06-29T15:58:21.556297+0000","last_unstale":"2024-06-29T15:58:22.548052+0000","last_undegraded":"2024-06-29T15:58:22.548052+0000","last_fullsized":"2024-06-29T15:58:22.548052+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:07.319988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799150+0000","last_change":"2024-06-29T15:58:20.555706+0000","last_active":"2024-06-29T15:58:22.799150+0000","last_peered":"2024-06-29T15:58:22.799150+0000","last_clean":"2024-06-29T15:58:22.799150+0000","last_became_active":"2024-06-29T15:58:20.555380+0000","last_became_peered":"2024-06-29T15:58:20.555380+0000","last_unstale":"2024-06-29T15:58:22.799150+0000","last_undegraded":"2024-06-29T15:58:22.799150+0000","last_fullsized":"2024-06-29T15:58:22.799150+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:03:17.347006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548358+0000","last_change":"2024-06-29T15:58:21.554565+0000","last_active":"2024-06-29T15:58:22.548358+0000","last_peered":"2024-06-29T15:58:22.548358+0000","last_clean":"2024-06-29T15:58:22.548358+0000","last_became_active":"2024-06-29T15:58:21.554255+0000","last_became_peered":"2024-06-29T15:58:21.554255+0000","last_unstale":"2024-06-29T15:58:22.548358+0000","last_undegraded":"2024-06-29T15:58:22.548358+0000","last_fullsized":"2024-06-29T15:58:22.548358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:14.470919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720200+0000","last_change":"2024-06-29T15:58:20.562271+0000","last_active":"2024-06-29T15:58:23.720200+0000","last_peered":"2024-06-29T15:58:23.720200+0000","last_clean":"2024-06-29T15:58:23.720200+0000","last_became_active":"2024-06-29T15:58:20.561940+0000","last_became_peered":"2024-06-29T15:58:20.561940+0000","last_unstale":"2024-06-29T15:58:23.720200+0000","last_undegraded":"2024-06-29T15:58:23.720200+0000","last_fullsized":"2024-06-29T15:58:23.720200+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:16.036109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597833+0000","last_change":"2024-06-29T15:58:21.548845+0000","last_active":"2024-06-29T15:58:22.597833+0000","last_peered":"2024-06-29T15:58:22.597833+0000","last_clean":"2024-06-29T15:58:22.597833+0000","last_became_active":"2024-06-29T15:58:21.548448+0000","last_became_peered":"2024-06-29T15:58:21.548448+0000","last_unstale":"2024-06-29T15:58:22.597833+0000","last_undegraded":"2024-06-29T15:58:22.597833+0000","last_fullsized":"2024-06-29T15:58:22.597833+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:38.983134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542265+0000","last_change":"2024-06-29T15:58:20.562116+0000","last_active":"2024-06-29T15:58:22.542265+0000","last_peered":"2024-06-29T15:58:22.542265+0000","last_clean":"2024-06-29T15:58:22.542265+0000","last_became_active":"2024-06-29T15:58:20.561801+0000","last_became_peered":"2024-06-29T15:58:20.561801+0000","last_unstale":"2024-06-29T15:58:22.542265+0000","last_undegraded":"2024-06-29T15:58:22.542265+0000","last_fullsized":"2024-06-29T15:58:22.542265+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:26.721883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542073+0000","last_change":"2024-06-29T15:58:21.560085+0000","last_active":"2024-06-29T15:58:22.542073+0000","last_peered":"2024-06-29T15:58:22.542073+0000","last_clean":"2024-06-29T15:58:22.542073+0000","last_became_active":"2024-06-29T15:58:21.559716+0000","last_became_peered":"2024-06-29T15:58:21.559716+0000","last_unstale":"2024-06-29T15:58:22.542073+0000","last_undegraded":"2024-06-29T15:58:22.542073+0000","last_fullsized":"2024-06-29T15:58:22.542073+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:15:37.579793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"241'3","reported_seq":17,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547764+0000","last_change":"2024-06-29T15:58:20.553528+0000","last_active":"2024-06-29T15:58:22.547764+0000","last_peered":"2024-06-29T15:58:22.547764+0000","last_clean":"2024-06-29T15:58:22.547764+0000","last_became_active":"2024-06-29T15:58:20.553185+0000","last_became_peered":"2024-06-29T15:58:20.553185+0000","last_unstale":"2024-06-29T15:58:22.547764+0000","last_undegraded":"2024-06-29T15:58:22.547764+0000","last_fullsized":"2024-06-29T15:58:22.547764+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:26.807742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548371+0000","last_change":"2024-06-29T15:58:21.553783+0000","last_active":"2024-06-29T15:58:22.548371+0000","last_peered":"2024-06-29T15:58:22.548371+0000","last_clean":"2024-06-29T15:58:22.548371+0000","last_became_active":"2024-06-29T15:58:21.553278+0000","last_became_peered":"2024-06-29T15:58:21.553278+0000","last_unstale":"2024-06-29T15:58:22.548371+0000","last_undegraded":"2024-06-29T15:58:22.548371+0000","last_fullsized":"2024-06-29T15:58:22.548371+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:39.179054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"241'6","reported_seq":20,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799462+0000","last_change":"2024-06-29T15:58:20.557583+0000","last_active":"2024-06-29T15:58:22.799462+0000","last_peered":"2024-06-29T15:58:22.799462+0000","last_clean":"2024-06-29T15:58:22.799462+0000","last_became_active":"2024-06-29T15:58:20.557274+0000","last_became_peered":"2024-06-29T15:58:20.557274+0000","last_unstale":"2024-06-29T15:58:22.799462+0000","last_undegraded":"2024-06-29T15:58:22.799462+0000","last_fullsized":"2024-06-29T15:58:22.799462+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:38.435050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719655+0000","last_change":"2024-06-29T15:58:21.559479+0000","last_active":"2024-06-29T15:58:23.719655+0000","last_peered":"2024-06-29T15:58:23.719655+0000","last_clean":"2024-06-29T15:58:23.719655+0000","last_became_active":"2024-06-29T15:58:21.559128+0000","last_became_peered":"2024-06-29T15:58:21.559128+0000","last_unstale":"2024-06-29T15:58:23.719655+0000","last_undegraded":"2024-06-29T15:58:23.719655+0000","last_fullsized":"2024-06-29T15:58:23.719655+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:28.321473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618061+0000","last_change":"2024-06-29T15:58:20.561589+0000","last_active":"2024-06-29T15:58:22.618061+0000","last_peered":"2024-06-29T15:58:22.618061+0000","last_clean":"2024-06-29T15:58:22.618061+0000","last_became_active":"2024-06-29T15:58:20.561251+0000","last_became_peered":"2024-06-29T15:58:20.561251+0000","last_unstale":"2024-06-29T15:58:22.618061+0000","last_undegraded":"2024-06-29T15:58:22.618061+0000","last_fullsized":"2024-06-29T15:58:22.618061+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:09.657256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548626+0000","last_change":"2024-06-29T15:58:21.560367+0000","last_active":"2024-06-29T15:58:22.548626+0000","last_peered":"2024-06-29T15:58:22.548626+0000","last_clean":"2024-06-29T15:58:22.548626+0000","last_became_active":"2024-06-29T15:58:21.560056+0000","last_became_peered":"2024-06-29T15:58:21.560056+0000","last_unstale":"2024-06-29T15:58:22.548626+0000","last_undegraded":"2024-06-29T15:58:22.548626+0000","last_fullsized":"2024-06-29T15:58:22.548626+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:43.371506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547696+0000","last_change":"2024-06-29T15:58:20.552449+0000","last_active":"2024-06-29T15:58:22.547696+0000","last_peered":"2024-06-29T15:58:22.547696+0000","last_clean":"2024-06-29T15:58:22.547696+0000","last_became_active":"2024-06-29T15:58:20.551202+0000","last_became_peered":"2024-06-29T15:58:20.551202+0000","last_unstale":"2024-06-29T15:58:22.547696+0000","last_undegraded":"2024-06-29T15:58:22.547696+0000","last_fullsized":"2024-06-29T15:58:22.547696+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:35.829408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618192+0000","last_change":"2024-06-29T15:58:21.566336+0000","last_active":"2024-06-29T15:58:22.618192+0000","last_peered":"2024-06-29T15:58:22.618192+0000","last_clean":"2024-06-29T15:58:22.618192+0000","last_became_active":"2024-06-29T15:58:21.566003+0000","last_became_peered":"2024-06-29T15:58:21.566003+0000","last_unstale":"2024-06-29T15:58:22.618192+0000","last_undegraded":"2024-06-29T15:58:22.618192+0000","last_fullsized":"2024-06-29T15:58:22.618192+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:46:35.306688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547153+0000","last_change":"2024-06-29T15:58:20.557295+0000","last_active":"2024-06-29T15:58:22.547153+0000","last_peered":"2024-06-29T15:58:22.547153+0000","last_clean":"2024-06-29T15:58:22.547153+0000","last_became_active":"2024-06-29T15:58:20.556868+0000","last_became_peered":"2024-06-29T15:58:20.556868+0000","last_unstale":"2024-06-29T15:58:22.547153+0000","last_undegraded":"2024-06-29T15:58:22.547153+0000","last_fullsized":"2024-06-29T15:58:22.547153+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:31.226218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547857+0000","last_change":"2024-06-29T15:58:21.565109+0000","last_active":"2024-06-29T15:58:22.547857+0000","last_peered":"2024-06-29T15:58:22.547857+0000","last_clean":"2024-06-29T15:58:22.547857+0000","last_became_active":"2024-06-29T15:58:21.564764+0000","last_became_peered":"2024-06-29T15:58:21.564764+0000","last_unstale":"2024-06-29T15:58:22.547857+0000","last_undegraded":"2024-06-29T15:58:22.547857+0000","last_fullsized":"2024-06-29T15:58:22.547857+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:43:24.997613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"241'8","reported_seq":22,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719193+0000","last_change":"2024-06-29T15:58:20.562144+0000","last_active":"2024-06-29T15:58:23.719193+0000","last_peered":"2024-06-29T15:58:23.719193+0000","last_clean":"2024-06-29T15:58:23.719193+0000","last_became_active":"2024-06-29T15:58:20.561879+0000","last_became_peered":"2024-06-29T15:58:20.561879+0000","last_unstale":"2024-06-29T15:58:23.719193+0000","last_undegraded":"2024-06-29T15:58:23.719193+0000","last_fullsized":"2024-06-29T15:58:23.719193+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:28.961624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547186+0000","last_change":"2024-06-29T15:58:21.546206+0000","last_active":"2024-06-29T15:58:22.547186+0000","last_peered":"2024-06-29T15:58:22.547186+0000","last_clean":"2024-06-29T15:58:22.547186+0000","last_became_active":"2024-06-29T15:58:21.545742+0000","last_became_peered":"2024-06-29T15:58:21.545742+0000","last_unstale":"2024-06-29T15:58:22.547186+0000","last_undegraded":"2024-06-29T15:58:22.547186+0000","last_fullsized":"2024-06-29T15:58:22.547186+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:57.523444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799610+0000","last_change":"2024-06-29T15:58:20.555827+0000","last_active":"2024-06-29T15:58:22.799610+0000","last_peered":"2024-06-29T15:58:22.799610+0000","last_clean":"2024-06-29T15:58:22.799610+0000","last_became_active":"2024-06-29T15:58:20.555506+0000","last_became_peered":"2024-06-29T15:58:20.555506+0000","last_unstale":"2024-06-29T15:58:22.799610+0000","last_undegraded":"2024-06-29T15:58:22.799610+0000","last_fullsized":"2024-06-29T15:58:22.799610+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:33:20.721321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547069+0000","last_change":"2024-06-29T15:58:21.558057+0000","last_active":"2024-06-29T15:58:22.547069+0000","last_peered":"2024-06-29T15:58:22.547069+0000","last_clean":"2024-06-29T15:58:22.547069+0000","last_became_active":"2024-06-29T15:58:21.557750+0000","last_became_peered":"2024-06-29T15:58:21.557750+0000","last_unstale":"2024-06-29T15:58:22.547069+0000","last_undegraded":"2024-06-29T15:58:22.547069+0000","last_fullsized":"2024-06-29T15:58:22.547069+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:35.875818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719642+0000","last_change":"2024-06-29T15:58:20.561995+0000","last_active":"2024-06-29T15:58:23.719642+0000","last_peered":"2024-06-29T15:58:23.719642+0000","last_clean":"2024-06-29T15:58:23.719642+0000","last_became_active":"2024-06-29T15:58:20.560719+0000","last_became_peered":"2024-06-29T15:58:20.560719+0000","last_unstale":"2024-06-29T15:58:23.719642+0000","last_undegraded":"2024-06-29T15:58:23.719642+0000","last_fullsized":"2024-06-29T15:58:23.719642+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:59.752024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719119+0000","last_change":"2024-06-29T15:58:21.552894+0000","last_active":"2024-06-29T15:58:23.719119+0000","last_peered":"2024-06-29T15:58:23.719119+0000","last_clean":"2024-06-29T15:58:23.719119+0000","last_became_active":"2024-06-29T15:58:21.552487+0000","last_became_peered":"2024-06-29T15:58:21.552487+0000","last_unstale":"2024-06-29T15:58:23.719119+0000","last_undegraded":"2024-06-29T15:58:23.719119+0000","last_fullsized":"2024-06-29T15:58:23.719119+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:49.309450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598156+0000","last_change":"2024-06-29T15:58:20.544642+0000","last_active":"2024-06-29T15:58:22.598156+0000","last_peered":"2024-06-29T15:58:22.598156+0000","last_clean":"2024-06-29T15:58:22.598156+0000","last_became_active":"2024-06-29T15:58:20.544356+0000","last_became_peered":"2024-06-29T15:58:20.544356+0000","last_unstale":"2024-06-29T15:58:22.598156+0000","last_undegraded":"2024-06-29T15:58:22.598156+0000","last_fullsized":"2024-06-29T15:58:22.598156+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:38.143805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542024+0000","last_change":"2024-06-29T15:58:21.560349+0000","last_active":"2024-06-29T15:58:22.542024+0000","last_peered":"2024-06-29T15:58:22.542024+0000","last_clean":"2024-06-29T15:58:22.542024+0000","last_became_active":"2024-06-29T15:58:21.559987+0000","last_became_peered":"2024-06-29T15:58:21.559987+0000","last_unstale":"2024-06-29T15:58:22.542024+0000","last_undegraded":"2024-06-29T15:58:22.542024+0000","last_fullsized":"2024-06-29T15:58:22.542024+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:20.317519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.617864+0000","last_change":"2024-06-29T15:58:20.551598+0000","last_active":"2024-06-29T15:58:22.617864+0000","last_peered":"2024-06-29T15:58:22.617864+0000","last_clean":"2024-06-29T15:58:22.617864+0000","last_became_active":"2024-06-29T15:58:20.551266+0000","last_became_peered":"2024-06-29T15:58:20.551266+0000","last_unstale":"2024-06-29T15:58:22.617864+0000","last_undegraded":"2024-06-29T15:58:22.617864+0000","last_fullsized":"2024-06-29T15:58:22.617864+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:56:15.731511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541950+0000","last_change":"2024-06-29T15:58:21.562779+0000","last_active":"2024-06-29T15:58:22.541950+0000","last_peered":"2024-06-29T15:58:22.541950+0000","last_clean":"2024-06-29T15:58:22.541950+0000","last_became_active":"2024-06-29T15:58:21.562503+0000","last_became_peered":"2024-06-29T15:58:21.562503+0000","last_unstale":"2024-06-29T15:58:22.541950+0000","last_undegraded":"2024-06-29T15:58:22.541950+0000","last_fullsized":"2024-06-29T15:58:22.541950+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:28.461061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541921+0000","last_change":"2024-06-29T15:58:20.559588+0000","last_active":"2024-06-29T15:58:22.541921+0000","last_peered":"2024-06-29T15:58:22.541921+0000","last_clean":"2024-06-29T15:58:22.541921+0000","last_became_active":"2024-06-29T15:58:20.559245+0000","last_became_peered":"2024-06-29T15:58:20.559245+0000","last_unstale":"2024-06-29T15:58:22.541921+0000","last_undegraded":"2024-06-29T15:58:22.541921+0000","last_fullsized":"2024-06-29T15:58:22.541921+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:40.258636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719349+0000","last_change":"2024-06-29T15:58:21.553384+0000","last_active":"2024-06-29T15:58:23.719349+0000","last_peered":"2024-06-29T15:58:23.719349+0000","last_clean":"2024-06-29T15:58:23.719349+0000","last_became_active":"2024-06-29T15:58:21.551240+0000","last_became_peered":"2024-06-29T15:58:21.551240+0000","last_unstale":"2024-06-29T15:58:23.719349+0000","last_undegraded":"2024-06-29T15:58:23.719349+0000","last_fullsized":"2024-06-29T15:58:23.719349+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:38.040891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597927+0000","last_change":"2024-06-29T15:58:20.557247+0000","last_active":"2024-06-29T15:58:22.597927+0000","last_peered":"2024-06-29T15:58:22.597927+0000","last_clean":"2024-06-29T15:58:22.597927+0000","last_became_active":"2024-06-29T15:58:20.556889+0000","last_became_peered":"2024-06-29T15:58:20.556889+0000","last_unstale":"2024-06-29T15:58:22.597927+0000","last_undegraded":"2024-06-29T15:58:22.597927+0000","last_fullsized":"2024-06-29T15:58:22.597927+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:48.301087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598280+0000","last_change":"2024-06-29T15:58:21.558641+0000","last_active":"2024-06-29T15:58:22.598280+0000","last_peered":"2024-06-29T15:58:22.598280+0000","last_clean":"2024-06-29T15:58:22.598280+0000","last_became_active":"2024-06-29T15:58:21.558377+0000","last_became_peered":"2024-06-29T15:58:21.558377+0000","last_unstale":"2024-06-29T15:58:22.598280+0000","last_undegraded":"2024-06-29T15:58:22.598280+0000","last_fullsized":"2024-06-29T15:58:22.598280+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:09.589023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597747+0000","last_change":"2024-06-29T15:58:20.543880+0000","last_active":"2024-06-29T15:58:22.597747+0000","last_peered":"2024-06-29T15:58:22.597747+0000","last_clean":"2024-06-29T15:58:22.597747+0000","last_became_active":"2024-06-29T15:58:20.543365+0000","last_became_peered":"2024-06-29T15:58:20.543365+0000","last_unstale":"2024-06-29T15:58:22.597747+0000","last_undegraded":"2024-06-29T15:58:22.597747+0000","last_fullsized":"2024-06-29T15:58:22.597747+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:18:31.509521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598176+0000","last_change":"2024-06-29T15:58:21.555113+0000","last_active":"2024-06-29T15:58:22.598176+0000","last_peered":"2024-06-29T15:58:22.598176+0000","last_clean":"2024-06-29T15:58:22.598176+0000","last_became_active":"2024-06-29T15:58:21.553362+0000","last_became_peered":"2024-06-29T15:58:21.553362+0000","last_unstale":"2024-06-29T15:58:22.598176+0000","last_undegraded":"2024-06-29T15:58:22.598176+0000","last_fullsized":"2024-06-29T15:58:22.598176+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:18.837956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.718819+0000","last_change":"2024-06-29T15:58:20.551665+0000","last_active":"2024-06-29T15:58:23.718819+0000","last_peered":"2024-06-29T15:58:23.718819+0000","last_clean":"2024-06-29T15:58:23.718819+0000","last_became_active":"2024-06-29T15:58:20.551347+0000","last_became_peered":"2024-06-29T15:58:20.551347+0000","last_unstale":"2024-06-29T15:58:23.718819+0000","last_undegraded":"2024-06-29T15:58:23.718819+0000","last_fullsized":"2024-06-29T15:58:23.718819+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:22:40.223323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548008+0000","last_change":"2024-06-29T15:58:21.561571+0000","last_active":"2024-06-29T15:58:22.548008+0000","last_peered":"2024-06-29T15:58:22.548008+0000","last_clean":"2024-06-29T15:58:22.548008+0000","last_became_active":"2024-06-29T15:58:21.561042+0000","last_became_peered":"2024-06-29T15:58:21.561042+0000","last_unstale":"2024-06-29T15:58:22.548008+0000","last_undegraded":"2024-06-29T15:58:22.548008+0000","last_fullsized":"2024-06-29T15:58:22.548008+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:34.199222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542229+0000","last_change":"2024-06-29T15:58:20.563978+0000","last_active":"2024-06-29T15:58:22.542229+0000","last_peered":"2024-06-29T15:58:22.542229+0000","last_clean":"2024-06-29T15:58:22.542229+0000","last_became_active":"2024-06-29T15:58:20.563680+0000","last_became_peered":"2024-06-29T15:58:20.563680+0000","last_unstale":"2024-06-29T15:58:22.542229+0000","last_undegraded":"2024-06-29T15:58:22.542229+0000","last_fullsized":"2024-06-29T15:58:22.542229+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:27.903522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618056+0000","last_change":"2024-06-29T15:58:21.565805+0000","last_active":"2024-06-29T15:58:22.618056+0000","last_peered":"2024-06-29T15:58:22.618056+0000","last_clean":"2024-06-29T15:58:22.618056+0000","last_became_active":"2024-06-29T15:58:21.564982+0000","last_became_peered":"2024-06-29T15:58:21.564982+0000","last_unstale":"2024-06-29T15:58:22.618056+0000","last_undegraded":"2024-06-29T15:58:22.618056+0000","last_fullsized":"2024-06-29T15:58:22.618056+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:18.217838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"242'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548253+0000","last_change":"2024-06-29T15:58:20.554719+0000","last_active":"2024-06-29T15:58:22.548253+0000","last_peered":"2024-06-29T15:58:22.548253+0000","last_clean":"2024-06-29T15:58:22.548253+0000","last_became_active":"2024-06-29T15:58:20.554423+0000","last_became_peered":"2024-06-29T15:58:20.554423+0000","last_unstale":"2024-06-29T15:58:22.548253+0000","last_undegraded":"2024-06-29T15:58:22.548253+0000","last_fullsized":"2024-06-29T15:58:22.548253+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:36.112729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541842+0000","last_change":"2024-06-29T15:58:21.559427+0000","last_active":"2024-06-29T15:58:22.541842+0000","last_peered":"2024-06-29T15:58:22.541842+0000","last_clean":"2024-06-29T15:58:22.541842+0000","last_became_active":"2024-06-29T15:58:21.558942+0000","last_became_peered":"2024-06-29T15:58:21.558942+0000","last_unstale":"2024-06-29T15:58:22.541842+0000","last_undegraded":"2024-06-29T15:58:22.541842+0000","last_fullsized":"2024-06-29T15:58:22.541842+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:36.096648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719798+0000","last_change":"2024-06-29T15:58:20.559651+0000","last_active":"2024-06-29T15:58:23.719798+0000","last_peered":"2024-06-29T15:58:23.719798+0000","last_clean":"2024-06-29T15:58:23.719798+0000","last_became_active":"2024-06-29T15:58:20.559323+0000","last_became_peered":"2024-06-29T15:58:20.559323+0000","last_unstale":"2024-06-29T15:58:23.719798+0000","last_undegraded":"2024-06-29T15:58:23.719798+0000","last_fullsized":"2024-06-29T15:58:23.719798+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:18:36.258363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547295+0000","last_change":"2024-06-29T15:58:21.545630+0000","last_active":"2024-06-29T15:58:22.547295+0000","last_peered":"2024-06-29T15:58:22.547295+0000","last_clean":"2024-06-29T15:58:22.547295+0000","last_became_active":"2024-06-29T15:58:21.545301+0000","last_became_peered":"2024-06-29T15:58:21.545301+0000","last_unstale":"2024-06-29T15:58:22.547295+0000","last_undegraded":"2024-06-29T15:58:22.547295+0000","last_fullsized":"2024-06-29T15:58:22.547295+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:36.336990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720428+0000","last_change":"2024-06-29T15:58:20.556695+0000","last_active":"2024-06-29T15:58:23.720428+0000","last_peered":"2024-06-29T15:58:23.720428+0000","last_clean":"2024-06-29T15:58:23.720428+0000","last_became_active":"2024-06-29T15:58:20.556053+0000","last_became_peered":"2024-06-29T15:58:20.556053+0000","last_unstale":"2024-06-29T15:58:23.720428+0000","last_undegraded":"2024-06-29T15:58:23.720428+0000","last_fullsized":"2024-06-29T15:58:23.720428+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:15.325464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720205+0000","last_change":"2024-06-29T15:58:21.565888+0000","last_active":"2024-06-29T15:58:23.720205+0000","last_peered":"2024-06-29T15:58:23.720205+0000","last_clean":"2024-06-29T15:58:23.720205+0000","last_became_active":"2024-06-29T15:58:21.565418+0000","last_became_peered":"2024-06-29T15:58:21.565418+0000","last_unstale":"2024-06-29T15:58:23.720205+0000","last_undegraded":"2024-06-29T15:58:23.720205+0000","last_fullsized":"2024-06-29T15:58:23.720205+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:57.843559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597925+0000","last_change":"2024-06-29T15:58:20.557670+0000","last_active":"2024-06-29T15:58:22.597925+0000","last_peered":"2024-06-29T15:58:22.597925+0000","last_clean":"2024-06-29T15:58:22.597925+0000","last_became_active":"2024-06-29T15:58:20.557366+0000","last_became_peered":"2024-06-29T15:58:20.557366+0000","last_unstale":"2024-06-29T15:58:22.597925+0000","last_undegraded":"2024-06-29T15:58:22.597925+0000","last_fullsized":"2024-06-29T15:58:22.597925+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:08.866031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799266+0000","last_change":"2024-06-29T15:58:21.562317+0000","last_active":"2024-06-29T15:58:22.799266+0000","last_peered":"2024-06-29T15:58:22.799266+0000","last_clean":"2024-06-29T15:58:22.799266+0000","last_became_active":"2024-06-29T15:58:21.561968+0000","last_became_peered":"2024-06-29T15:58:21.561968+0000","last_unstale":"2024-06-29T15:58:22.799266+0000","last_undegraded":"2024-06-29T15:58:22.799266+0000","last_fullsized":"2024-06-29T15:58:22.799266+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.632416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547844+0000","last_change":"2024-06-29T15:58:20.547790+0000","last_active":"2024-06-29T15:58:22.547844+0000","last_peered":"2024-06-29T15:58:22.547844+0000","last_clean":"2024-06-29T15:58:22.547844+0000","last_became_active":"2024-06-29T15:58:20.547481+0000","last_became_peered":"2024-06-29T15:58:20.547481+0000","last_unstale":"2024-06-29T15:58:22.547844+0000","last_undegraded":"2024-06-29T15:58:22.547844+0000","last_fullsized":"2024-06-29T15:58:22.547844+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:32.913616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719588+0000","last_change":"2024-06-29T15:58:21.553783+0000","last_active":"2024-06-29T15:58:23.719588+0000","last_peered":"2024-06-29T15:58:23.719588+0000","last_clean":"2024-06-29T15:58:23.719588+0000","last_became_active":"2024-06-29T15:58:21.553390+0000","last_became_peered":"2024-06-29T15:58:21.553390+0000","last_unstale":"2024-06-29T15:58:23.719588+0000","last_undegraded":"2024-06-29T15:58:23.719588+0000","last_fullsized":"2024-06-29T15:58:23.719588+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:01.841016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541769+0000","last_change":"2024-06-29T15:58:20.561399+0000","last_active":"2024-06-29T15:58:22.541769+0000","last_peered":"2024-06-29T15:58:22.541769+0000","last_clean":"2024-06-29T15:58:22.541769+0000","last_became_active":"2024-06-29T15:58:20.560716+0000","last_became_peered":"2024-06-29T15:58:20.560716+0000","last_unstale":"2024-06-29T15:58:22.541769+0000","last_undegraded":"2024-06-29T15:58:22.541769+0000","last_fullsized":"2024-06-29T15:58:22.541769+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:42:19.927414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598004+0000","last_change":"2024-06-29T15:58:21.563096+0000","last_active":"2024-06-29T15:58:22.598004+0000","last_peered":"2024-06-29T15:58:22.598004+0000","last_clean":"2024-06-29T15:58:22.598004+0000","last_became_active":"2024-06-29T15:58:21.562778+0000","last_became_peered":"2024-06-29T15:58:21.562778+0000","last_unstale":"2024-06-29T15:58:22.598004+0000","last_undegraded":"2024-06-29T15:58:22.598004+0000","last_fullsized":"2024-06-29T15:58:22.598004+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:20.808675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597580+0000","last_change":"2024-06-29T15:58:20.554308+0000","last_active":"2024-06-29T15:58:22.597580+0000","last_peered":"2024-06-29T15:58:22.597580+0000","last_clean":"2024-06-29T15:58:22.597580+0000","last_became_active":"2024-06-29T15:58:20.553976+0000","last_became_peered":"2024-06-29T15:58:20.553976+0000","last_unstale":"2024-06-29T15:58:22.597580+0000","last_undegraded":"2024-06-29T15:58:22.597580+0000","last_fullsized":"2024-06-29T15:58:22.597580+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:13:16.241344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799135+0000","last_change":"2024-06-29T15:58:21.550432+0000","last_active":"2024-06-29T15:58:22.799135+0000","last_peered":"2024-06-29T15:58:22.799135+0000","last_clean":"2024-06-29T15:58:22.799135+0000","last_became_active":"2024-06-29T15:58:21.550075+0000","last_became_peered":"2024-06-29T15:58:21.550075+0000","last_unstale":"2024-06-29T15:58:22.799135+0000","last_undegraded":"2024-06-29T15:58:22.799135+0000","last_fullsized":"2024-06-29T15:58:22.799135+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:24.427535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548196+0000","last_change":"2024-06-29T15:58:20.561371+0000","last_active":"2024-06-29T15:58:22.548196+0000","last_peered":"2024-06-29T15:58:22.548196+0000","last_clean":"2024-06-29T15:58:22.548196+0000","last_became_active":"2024-06-29T15:58:20.561026+0000","last_became_peered":"2024-06-29T15:58:20.561026+0000","last_unstale":"2024-06-29T15:58:22.548196+0000","last_undegraded":"2024-06-29T15:58:22.548196+0000","last_fullsized":"2024-06-29T15:58:22.548196+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:07.861501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"228'247","reported_seq":1190,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.617778+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:58:22.617778+0000","last_peered":"2024-06-29T15:58:22.617778+0000","last_clean":"2024-06-29T15:58:22.617778+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:58:22.617778+0000","last_undegraded":"2024-06-29T15:58:22.617778+0000","last_fullsized":"2024-06-29T15:58:22.617778+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543298,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52792,"kb_used_data":3560,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319048,"statfs":{"total":96636764160,"available":96582705152,"internally_reserved":0,"allocated":3645440,"data_stored":2163708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543299,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60528,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311312,"statfs":{"total":96636764160,"available":96574783488,"internally_reserved":0,"allocated":3051520,"data_stored":1573328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543298,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70248,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301592,"statfs":{"total":96636764160,"available":96564830208,"internally_reserved":0,"allocated":3043328,"data_stored":1573216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543297,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52204,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319636,"statfs":{"total":96636764160,"available":96583307264,"internally_reserved":0,"allocated":3047424,"data_stored":1573286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510595,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48064,"kb_used_data":2996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323776,"statfs":{"total":96636764160,"available":96587546624,"internally_reserved":0,"allocated":3067904,"data_stored":1575466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48052,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323788,"statfs":{"total":96636764160,"available":96587558912,"internally_reserved":0,"allocated":3051520,"data_stored":1573868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510594,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48044,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323796,"statfs":{"total":96636764160,"available":96587567104,"internally_reserved":0,"allocated":3043328,"data_stored":1573216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510596,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43064,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328776,"statfs":{"total":96636764160,"available":96592666624,"internally_reserved":0,"allocated":3649536,"data_stored":2165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:58:25.643 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.797+0000 7fa0c9812640 1 -- 172.21.15.59:0/3275207955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 msgr2=0x7fa0c4161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.797+0000 7fa0c9812640 1 --2- 172.21.15.59:0/3275207955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 0x7fa0c4161a80 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009830 tx=0x7fa0b802a470 comp rx=0 tx=0).stop 2024-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.797+0000 7fa0c9812640 1 -- 172.21.15.59:0/3275207955 shutdown_connections 2024-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.797+0000 7fa0c9812640 1 --2- 172.21.15.59:0/3275207955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 0x7fa0c4161a80 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.797+0000 7fa0c9812640 1 --2- 172.21.15.59:0/3275207955 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0c4162770 0x7fa0c40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.797+0000 7fa0c9812640 1 -- 172.21.15.59:0/3275207955 >> 172.21.15.59:0/3275207955 conn(0x7fa0c40d45b0 msgr2=0x7fa0c40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.797+0000 7fa0c9812640 1 -- 172.21.15.59:0/3275207955 shutdown_connections 2024-06-29T15:58:25.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.797+0000 7fa0c9812640 1 -- 172.21.15.59:0/3275207955 wait complete. 2024-06-29T15:58:25.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.799+0000 7fa0c9812640 1 Processor -- start 2024-06-29T15:58:25.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.799+0000 7fa0c9812640 1 -- start start 2024-06-29T15:58:25.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c9812640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 0x7fa0c40cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c9812640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c4161fc0 0x7fa0c40c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c9812640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0c4162770 0x7fa0c4160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c9812640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c40d2d80 con 0x7fa0c415e0a0 2024-06-29T15:58:25.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c9812640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c40b4b00 con 0x7fa0c4161fc0 2024-06-29T15:58:25.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c9812640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c40d2f00 con 0x7fa0c4162770 2024-06-29T15:58:25.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 0x7fa0c40cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 0x7fa0c40cab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35922/0 (socket says 172.21.15.59:35922) 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c2ffd640 1 -- 172.21.15.59:0/1202330900 learned_addr learned my addr 172.21.15.59:0/1202330900 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c2ffd640 1 -- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0c4162770 msgr2=0x7fa0c4160b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c2ffd640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0c4162770 0x7fa0c4160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c2ffd640 1 -- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c4161fc0 msgr2=0x7fa0c40c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c2ffd640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c4161fc0 0x7fa0c40c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.800+0000 7fa0c2ffd640 1 -- 172.21.15.59:0/1202330900 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c40c99b0 con 0x7fa0c415e0a0 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.801+0000 7fa0c2ffd640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 0x7fa0c40cab30 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400c970 tx=0x7fa0b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.801+0000 7fa0abfff640 1 -- 172.21.15.59:0/1202330900 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4007bb0 con 0x7fa0c415e0a0 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.801+0000 7fa0abfff640 1 -- 172.21.15.59:0/1202330900 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0b4007e40 con 0x7fa0c415e0a0 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.801+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c40cc610 con 0x7fa0c415e0a0 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.801+0000 7fa0abfff640 1 -- 172.21.15.59:0/1202330900 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b400f690 con 0x7fa0c415e0a0 2024-06-29T15:58:25.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.801+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c40ccbd0 con 0x7fa0c415e0a0 2024-06-29T15:58:25.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.802+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c40d33d0 con 0x7fa0c415e0a0 2024-06-29T15:58:25.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.803+0000 7fa0abfff640 1 -- 172.21.15.59:0/1202330900 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fa0b400f830 con 0x7fa0c415e0a0 2024-06-29T15:58:25.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.803+0000 7fa0abfff640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa0b0076510 0x7fa0b0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.804+0000 7fa0abfff640 1 -- 172.21.15.59:0/1202330900 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fa0b4098610 con 0x7fa0c415e0a0 2024-06-29T15:58:25.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.804+0000 7fa0c27fc640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa0b0076510 0x7fa0b0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.804+0000 7fa0c27fc640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa0b0076510 0x7fa0b0078a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8034000 tx=0x7fa0b80046c0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.808+0000 7fa0abfff640 1 -- 172.21.15.59:0/1202330900 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa0b4014030 con 0x7fa0c415e0a0 2024-06-29T15:58:25.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.994+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 --> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0c40ccfb0 con 0x7fa0b0076510 2024-06-29T15:58:25.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:25.999+0000 7fa0abfff640 1 -- 172.21.15.59:0/1202330900 <== mgr.9710 v2:172.21.15.153:6832/1831136380 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7fa0c40ccfb0 con 0x7fa0b0076510 2024-06-29T15:58:26.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:26.006 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:58:26.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa0b0076510 msgr2=0x7fa0b0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa0b0076510 0x7fa0b0078a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8034000 tx=0x7fa0b80046c0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 msgr2=0x7fa0c40cab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 0x7fa0c40cab30 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400c970 tx=0x7fa0b400ce30 comp rx=0 tx=0).stop 2024-06-29T15:58:26.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 shutdown_connections 2024-06-29T15:58:26.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa0b0076510 0x7fa0b0078a20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0c4162770 0x7fa0c4160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0c4161fc0 0x7fa0c40c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 --2- 172.21.15.59:0/1202330900 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c415e0a0 0x7fa0c40cab30 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 >> 172.21.15.59:0/1202330900 conn(0x7fa0c40d45b0 msgr2=0x7fa0c40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:26.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 shutdown_connections 2024-06-29T15:58:26.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.003+0000 7fa0c9812640 1 -- 172.21.15.59:0/1202330900 wait complete. 2024-06-29T15:58:26.014 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-06-29T15:58:25.204876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":446,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":286,"ondisk_log_size":286,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":422996,"kb_used_data":25000,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754551724,"statfs":{"total":773094113280,"available":772660965376,"internally_reserved":0,"allocated":25600000,"data_stored":13771284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":407424562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.676214"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799510+0000","last_change":"2024-06-29T15:58:21.556650+0000","last_active":"2024-06-29T15:58:22.799510+0000","last_peered":"2024-06-29T15:58:22.799510+0000","last_clean":"2024-06-29T15:58:22.799510+0000","last_became_active":"2024-06-29T15:58:21.556321+0000","last_became_peered":"2024-06-29T15:58:21.556321+0000","last_unstale":"2024-06-29T15:58:22.799510+0000","last_undegraded":"2024-06-29T15:58:22.799510+0000","last_fullsized":"2024-06-29T15:58:22.799510+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:40:17.694204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719577+0000","last_change":"2024-06-29T15:58:21.553068+0000","last_active":"2024-06-29T15:58:23.719577+0000","last_peered":"2024-06-29T15:58:23.719577+0000","last_clean":"2024-06-29T15:58:23.719577+0000","last_became_active":"2024-06-29T15:58:21.552631+0000","last_became_peered":"2024-06-29T15:58:21.552631+0000","last_unstale":"2024-06-29T15:58:23.719577+0000","last_undegraded":"2024-06-29T15:58:23.719577+0000","last_fullsized":"2024-06-29T15:58:23.719577+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:18.380041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542110+0000","last_change":"2024-06-29T15:58:21.553826+0000","last_active":"2024-06-29T15:58:22.542110+0000","last_peered":"2024-06-29T15:58:22.542110+0000","last_clean":"2024-06-29T15:58:22.542110+0000","last_became_active":"2024-06-29T15:58:21.553476+0000","last_became_peered":"2024-06-29T15:58:21.553476+0000","last_unstale":"2024-06-29T15:58:22.542110+0000","last_undegraded":"2024-06-29T15:58:22.542110+0000","last_fullsized":"2024-06-29T15:58:22.542110+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:09:27.869044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542064+0000","last_change":"2024-06-29T15:58:21.556754+0000","last_active":"2024-06-29T15:58:22.542064+0000","last_peered":"2024-06-29T15:58:22.542064+0000","last_clean":"2024-06-29T15:58:22.542064+0000","last_became_active":"2024-06-29T15:58:21.556399+0000","last_became_peered":"2024-06-29T15:58:21.556399+0000","last_unstale":"2024-06-29T15:58:22.542064+0000","last_undegraded":"2024-06-29T15:58:22.542064+0000","last_fullsized":"2024-06-29T15:58:22.542064+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:53:06.570549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547499+0000","last_change":"2024-06-29T15:58:21.558986+0000","last_active":"2024-06-29T15:58:22.547499+0000","last_peered":"2024-06-29T15:58:22.547499+0000","last_clean":"2024-06-29T15:58:22.547499+0000","last_became_active":"2024-06-29T15:58:21.558669+0000","last_became_peered":"2024-06-29T15:58:21.558669+0000","last_unstale":"2024-06-29T15:58:22.547499+0000","last_undegraded":"2024-06-29T15:58:22.547499+0000","last_fullsized":"2024-06-29T15:58:22.547499+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:07.747196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799086+0000","last_change":"2024-06-29T15:58:21.558888+0000","last_active":"2024-06-29T15:58:22.799086+0000","last_peered":"2024-06-29T15:58:22.799086+0000","last_clean":"2024-06-29T15:58:22.799086+0000","last_became_active":"2024-06-29T15:58:21.558578+0000","last_became_peered":"2024-06-29T15:58:21.558578+0000","last_unstale":"2024-06-29T15:58:22.799086+0000","last_undegraded":"2024-06-29T15:58:22.799086+0000","last_fullsized":"2024-06-29T15:58:22.799086+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:30.612798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548295+0000","last_change":"2024-06-29T15:58:21.552220+0000","last_active":"2024-06-29T15:58:22.548295+0000","last_peered":"2024-06-29T15:58:22.548295+0000","last_clean":"2024-06-29T15:58:22.548295+0000","last_became_active":"2024-06-29T15:58:21.551911+0000","last_became_peered":"2024-06-29T15:58:21.551911+0000","last_unstale":"2024-06-29T15:58:22.548295+0000","last_undegraded":"2024-06-29T15:58:22.548295+0000","last_fullsized":"2024-06-29T15:58:22.548295+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:20.005949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547458+0000","last_change":"2024-06-29T15:58:21.558631+0000","last_active":"2024-06-29T15:58:22.547458+0000","last_peered":"2024-06-29T15:58:22.547458+0000","last_clean":"2024-06-29T15:58:22.547458+0000","last_became_active":"2024-06-29T15:58:21.558300+0000","last_became_peered":"2024-06-29T15:58:21.558300+0000","last_unstale":"2024-06-29T15:58:22.547458+0000","last_undegraded":"2024-06-29T15:58:22.547458+0000","last_fullsized":"2024-06-29T15:58:22.547458+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:46.044647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799608+0000","last_change":"2024-06-29T15:58:21.560160+0000","last_active":"2024-06-29T15:58:22.799608+0000","last_peered":"2024-06-29T15:58:22.799608+0000","last_clean":"2024-06-29T15:58:22.799608+0000","last_became_active":"2024-06-29T15:58:21.559735+0000","last_became_peered":"2024-06-29T15:58:21.559735+0000","last_unstale":"2024-06-29T15:58:22.799608+0000","last_undegraded":"2024-06-29T15:58:22.799608+0000","last_fullsized":"2024-06-29T15:58:22.799608+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:33.429459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542216+0000","last_change":"2024-06-29T15:58:21.559778+0000","last_active":"2024-06-29T15:58:22.542216+0000","last_peered":"2024-06-29T15:58:22.542216+0000","last_clean":"2024-06-29T15:58:22.542216+0000","last_became_active":"2024-06-29T15:58:21.559442+0000","last_became_peered":"2024-06-29T15:58:21.559442+0000","last_unstale":"2024-06-29T15:58:22.542216+0000","last_undegraded":"2024-06-29T15:58:22.542216+0000","last_fullsized":"2024-06-29T15:58:22.542216+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:14.817280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542143+0000","last_change":"2024-06-29T15:58:21.560672+0000","last_active":"2024-06-29T15:58:22.542143+0000","last_peered":"2024-06-29T15:58:22.542143+0000","last_clean":"2024-06-29T15:58:22.542143+0000","last_became_active":"2024-06-29T15:58:21.560333+0000","last_became_peered":"2024-06-29T15:58:21.560333+0000","last_unstale":"2024-06-29T15:58:22.542143+0000","last_undegraded":"2024-06-29T15:58:22.542143+0000","last_fullsized":"2024-06-29T15:58:22.542143+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:16.662908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799391+0000","last_change":"2024-06-29T15:58:21.557848+0000","last_active":"2024-06-29T15:58:22.799391+0000","last_peered":"2024-06-29T15:58:22.799391+0000","last_clean":"2024-06-29T15:58:22.799391+0000","last_became_active":"2024-06-29T15:58:21.557516+0000","last_became_peered":"2024-06-29T15:58:21.557516+0000","last_unstale":"2024-06-29T15:58:22.799391+0000","last_undegraded":"2024-06-29T15:58:22.799391+0000","last_fullsized":"2024-06-29T15:58:22.799391+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:26.084704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597951+0000","last_change":"2024-06-29T15:58:21.557388+0000","last_active":"2024-06-29T15:58:22.597951+0000","last_peered":"2024-06-29T15:58:22.597951+0000","last_clean":"2024-06-29T15:58:22.597951+0000","last_became_active":"2024-06-29T15:58:21.557098+0000","last_became_peered":"2024-06-29T15:58:21.557098+0000","last_unstale":"2024-06-29T15:58:22.597951+0000","last_undegraded":"2024-06-29T15:58:22.597951+0000","last_fullsized":"2024-06-29T15:58:22.597951+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:27.148409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719514+0000","last_change":"2024-06-29T15:58:21.567227+0000","last_active":"2024-06-29T15:58:23.719514+0000","last_peered":"2024-06-29T15:58:23.719514+0000","last_clean":"2024-06-29T15:58:23.719514+0000","last_became_active":"2024-06-29T15:58:21.566195+0000","last_became_peered":"2024-06-29T15:58:21.566195+0000","last_unstale":"2024-06-29T15:58:23.719514+0000","last_undegraded":"2024-06-29T15:58:23.719514+0000","last_fullsized":"2024-06-29T15:58:23.719514+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:49.425921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618140+0000","last_change":"2024-06-29T15:58:21.562321+0000","last_active":"2024-06-29T15:58:22.618140+0000","last_peered":"2024-06-29T15:58:22.618140+0000","last_clean":"2024-06-29T15:58:22.618140+0000","last_became_active":"2024-06-29T15:58:21.561988+0000","last_became_peered":"2024-06-29T15:58:21.561988+0000","last_unstale":"2024-06-29T15:58:22.618140+0000","last_undegraded":"2024-06-29T15:58:22.618140+0000","last_fullsized":"2024-06-29T15:58:22.618140+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:27.579111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541807+0000","last_change":"2024-06-29T15:58:21.561992+0000","last_active":"2024-06-29T15:58:22.541807+0000","last_peered":"2024-06-29T15:58:22.541807+0000","last_clean":"2024-06-29T15:58:22.541807+0000","last_became_active":"2024-06-29T15:58:21.561586+0000","last_became_peered":"2024-06-29T15:58:21.561586+0000","last_unstale":"2024-06-29T15:58:22.541807+0000","last_undegraded":"2024-06-29T15:58:22.541807+0000","last_fullsized":"2024-06-29T15:58:22.541807+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:57.843559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598246+0000","last_change":"2024-06-29T15:58:21.559279+0000","last_active":"2024-06-29T15:58:22.598246+0000","last_peered":"2024-06-29T15:58:22.598246+0000","last_clean":"2024-06-29T15:58:22.598246+0000","last_became_active":"2024-06-29T15:58:21.559033+0000","last_became_peered":"2024-06-29T15:58:21.559033+0000","last_unstale":"2024-06-29T15:58:22.598246+0000","last_undegraded":"2024-06-29T15:58:22.598246+0000","last_fullsized":"2024-06-29T15:58:22.598246+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:19.231351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542358+0000","last_change":"2024-06-29T15:58:21.561849+0000","last_active":"2024-06-29T15:58:22.542358+0000","last_peered":"2024-06-29T15:58:22.542358+0000","last_clean":"2024-06-29T15:58:22.542358+0000","last_became_active":"2024-06-29T15:58:21.561051+0000","last_became_peered":"2024-06-29T15:58:21.561051+0000","last_unstale":"2024-06-29T15:58:22.542358+0000","last_undegraded":"2024-06-29T15:58:22.542358+0000","last_fullsized":"2024-06-29T15:58:22.542358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:05:18.245403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618086+0000","last_change":"2024-06-29T15:58:21.561160+0000","last_active":"2024-06-29T15:58:22.618086+0000","last_peered":"2024-06-29T15:58:22.618086+0000","last_clean":"2024-06-29T15:58:22.618086+0000","last_became_active":"2024-06-29T15:58:21.560810+0000","last_became_peered":"2024-06-29T15:58:21.560810+0000","last_unstale":"2024-06-29T15:58:22.618086+0000","last_undegraded":"2024-06-29T15:58:22.618086+0000","last_fullsized":"2024-06-29T15:58:22.618086+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:02.548677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719830+0000","last_change":"2024-06-29T15:58:21.561325+0000","last_active":"2024-06-29T15:58:23.719830+0000","last_peered":"2024-06-29T15:58:23.719830+0000","last_clean":"2024-06-29T15:58:23.719830+0000","last_became_active":"2024-06-29T15:58:21.561021+0000","last_became_peered":"2024-06-29T15:58:21.561021+0000","last_unstale":"2024-06-29T15:58:23.719830+0000","last_undegraded":"2024-06-29T15:58:23.719830+0000","last_fullsized":"2024-06-29T15:58:23.719830+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:25:28.950204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548555+0000","last_change":"2024-06-29T15:58:21.561369+0000","last_active":"2024-06-29T15:58:22.548555+0000","last_peered":"2024-06-29T15:58:22.548555+0000","last_clean":"2024-06-29T15:58:22.548555+0000","last_became_active":"2024-06-29T15:58:21.561021+0000","last_became_peered":"2024-06-29T15:58:21.561021+0000","last_unstale":"2024-06-29T15:58:22.548555+0000","last_undegraded":"2024-06-29T15:58:22.548555+0000","last_fullsized":"2024-06-29T15:58:22.548555+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:35.412665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597914+0000","last_change":"2024-06-29T15:58:21.555481+0000","last_active":"2024-06-29T15:58:22.597914+0000","last_peered":"2024-06-29T15:58:22.597914+0000","last_clean":"2024-06-29T15:58:22.597914+0000","last_became_active":"2024-06-29T15:58:21.553482+0000","last_became_peered":"2024-06-29T15:58:21.553482+0000","last_unstale":"2024-06-29T15:58:22.597914+0000","last_undegraded":"2024-06-29T15:58:22.597914+0000","last_fullsized":"2024-06-29T15:58:22.597914+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:08.904938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618204+0000","last_change":"2024-06-29T15:58:21.560413+0000","last_active":"2024-06-29T15:58:22.618204+0000","last_peered":"2024-06-29T15:58:22.618204+0000","last_clean":"2024-06-29T15:58:22.618204+0000","last_became_active":"2024-06-29T15:58:21.560114+0000","last_became_peered":"2024-06-29T15:58:21.560114+0000","last_unstale":"2024-06-29T15:58:22.618204+0000","last_undegraded":"2024-06-29T15:58:22.618204+0000","last_fullsized":"2024-06-29T15:58:22.618204+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:43:35.054888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597748+0000","last_change":"2024-06-29T15:58:21.562168+0000","last_active":"2024-06-29T15:58:22.597748+0000","last_peered":"2024-06-29T15:58:22.597748+0000","last_clean":"2024-06-29T15:58:22.597748+0000","last_became_active":"2024-06-29T15:58:21.555306+0000","last_became_peered":"2024-06-29T15:58:21.555306+0000","last_unstale":"2024-06-29T15:58:22.597748+0000","last_undegraded":"2024-06-29T15:58:22.597748+0000","last_fullsized":"2024-06-29T15:58:22.597748+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:40.193793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719887+0000","last_change":"2024-06-29T15:58:21.559133+0000","last_active":"2024-06-29T15:58:23.719887+0000","last_peered":"2024-06-29T15:58:23.719887+0000","last_clean":"2024-06-29T15:58:23.719887+0000","last_became_active":"2024-06-29T15:58:21.558649+0000","last_became_peered":"2024-06-29T15:58:21.558649+0000","last_unstale":"2024-06-29T15:58:23.719887+0000","last_undegraded":"2024-06-29T15:58:23.719887+0000","last_fullsized":"2024-06-29T15:58:23.719887+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:28.201954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548124+0000","last_change":"2024-06-29T15:58:21.561677+0000","last_active":"2024-06-29T15:58:22.548124+0000","last_peered":"2024-06-29T15:58:22.548124+0000","last_clean":"2024-06-29T15:58:22.548124+0000","last_became_active":"2024-06-29T15:58:21.561367+0000","last_became_peered":"2024-06-29T15:58:21.561367+0000","last_unstale":"2024-06-29T15:58:22.548124+0000","last_undegraded":"2024-06-29T15:58:22.548124+0000","last_fullsized":"2024-06-29T15:58:22.548124+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:23:28.067754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547690+0000","last_change":"2024-06-29T15:58:21.552197+0000","last_active":"2024-06-29T15:58:22.547690+0000","last_peered":"2024-06-29T15:58:22.547690+0000","last_clean":"2024-06-29T15:58:22.547690+0000","last_became_active":"2024-06-29T15:58:21.551846+0000","last_became_peered":"2024-06-29T15:58:21.551846+0000","last_unstale":"2024-06-29T15:58:22.547690+0000","last_undegraded":"2024-06-29T15:58:22.547690+0000","last_fullsized":"2024-06-29T15:58:22.547690+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:31.983161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799428+0000","last_change":"2024-06-29T15:58:21.560349+0000","last_active":"2024-06-29T15:58:22.799428+0000","last_peered":"2024-06-29T15:58:22.799428+0000","last_clean":"2024-06-29T15:58:22.799428+0000","last_became_active":"2024-06-29T15:58:21.559976+0000","last_became_peered":"2024-06-29T15:58:21.559976+0000","last_unstale":"2024-06-29T15:58:22.799428+0000","last_undegraded":"2024-06-29T15:58:22.799428+0000","last_fullsized":"2024-06-29T15:58:22.799428+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:35:34.378626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720218+0000","last_change":"2024-06-29T15:58:21.567137+0000","last_active":"2024-06-29T15:58:23.720218+0000","last_peered":"2024-06-29T15:58:23.720218+0000","last_clean":"2024-06-29T15:58:23.720218+0000","last_became_active":"2024-06-29T15:58:21.566816+0000","last_became_peered":"2024-06-29T15:58:21.566816+0000","last_unstale":"2024-06-29T15:58:23.720218+0000","last_undegraded":"2024-06-29T15:58:23.720218+0000","last_fullsized":"2024-06-29T15:58:23.720218+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:36.096648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719685+0000","last_change":"2024-06-29T15:58:21.561708+0000","last_active":"2024-06-29T15:58:23.719685+0000","last_peered":"2024-06-29T15:58:23.719685+0000","last_clean":"2024-06-29T15:58:23.719685+0000","last_became_active":"2024-06-29T15:58:21.561360+0000","last_became_peered":"2024-06-29T15:58:21.561360+0000","last_unstale":"2024-06-29T15:58:23.719685+0000","last_undegraded":"2024-06-29T15:58:23.719685+0000","last_fullsized":"2024-06-29T15:58:23.719685+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:17.841782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719243+0000","last_change":"2024-06-29T15:58:21.560378+0000","last_active":"2024-06-29T15:58:23.719243+0000","last_peered":"2024-06-29T15:58:23.719243+0000","last_clean":"2024-06-29T15:58:23.719243+0000","last_became_active":"2024-06-29T15:58:21.560075+0000","last_became_peered":"2024-06-29T15:58:21.560075+0000","last_unstale":"2024-06-29T15:58:23.719243+0000","last_undegraded":"2024-06-29T15:58:23.719243+0000","last_fullsized":"2024-06-29T15:58:23.719243+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:16.886076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799079+0000","last_change":"2024-06-29T15:58:21.561504+0000","last_active":"2024-06-29T15:58:22.799079+0000","last_peered":"2024-06-29T15:58:22.799079+0000","last_clean":"2024-06-29T15:58:22.799079+0000","last_became_active":"2024-06-29T15:58:21.560810+0000","last_became_peered":"2024-06-29T15:58:21.560810+0000","last_unstale":"2024-06-29T15:58:22.799079+0000","last_undegraded":"2024-06-29T15:58:22.799079+0000","last_fullsized":"2024-06-29T15:58:22.799079+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:53:36.634654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719125+0000","last_change":"2024-06-29T15:58:21.562794+0000","last_active":"2024-06-29T15:58:23.719125+0000","last_peered":"2024-06-29T15:58:23.719125+0000","last_clean":"2024-06-29T15:58:23.719125+0000","last_became_active":"2024-06-29T15:58:21.562011+0000","last_became_peered":"2024-06-29T15:58:21.562011+0000","last_unstale":"2024-06-29T15:58:23.719125+0000","last_undegraded":"2024-06-29T15:58:23.719125+0000","last_fullsized":"2024-06-29T15:58:23.719125+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:09.874394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"241'4","reported_seq":18,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799470+0000","last_change":"2024-06-29T15:58:20.549867+0000","last_active":"2024-06-29T15:58:22.799470+0000","last_peered":"2024-06-29T15:58:22.799470+0000","last_clean":"2024-06-29T15:58:22.799470+0000","last_became_active":"2024-06-29T15:58:20.549532+0000","last_became_peered":"2024-06-29T15:58:20.549532+0000","last_unstale":"2024-06-29T15:58:22.799470+0000","last_undegraded":"2024-06-29T15:58:22.799470+0000","last_fullsized":"2024-06-29T15:58:22.799470+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:11.359524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618083+0000","last_change":"2024-06-29T15:58:21.562279+0000","last_active":"2024-06-29T15:58:22.618083+0000","last_peered":"2024-06-29T15:58:22.618083+0000","last_clean":"2024-06-29T15:58:22.618083+0000","last_became_active":"2024-06-29T15:58:21.561962+0000","last_became_peered":"2024-06-29T15:58:21.561962+0000","last_unstale":"2024-06-29T15:58:22.618083+0000","last_undegraded":"2024-06-29T15:58:22.618083+0000","last_fullsized":"2024-06-29T15:58:22.618083+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:43:25.515942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598043+0000","last_change":"2024-06-29T15:58:20.547115+0000","last_active":"2024-06-29T15:58:22.598043+0000","last_peered":"2024-06-29T15:58:22.598043+0000","last_clean":"2024-06-29T15:58:22.598043+0000","last_became_active":"2024-06-29T15:58:20.546778+0000","last_became_peered":"2024-06-29T15:58:20.546778+0000","last_unstale":"2024-06-29T15:58:22.598043+0000","last_undegraded":"2024-06-29T15:58:22.598043+0000","last_fullsized":"2024-06-29T15:58:22.598043+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:05.970706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799428+0000","last_change":"2024-06-29T15:58:21.562748+0000","last_active":"2024-06-29T15:58:22.799428+0000","last_peered":"2024-06-29T15:58:22.799428+0000","last_clean":"2024-06-29T15:58:22.799428+0000","last_became_active":"2024-06-29T15:58:21.562417+0000","last_became_peered":"2024-06-29T15:58:21.562417+0000","last_unstale":"2024-06-29T15:58:22.799428+0000","last_undegraded":"2024-06-29T15:58:22.799428+0000","last_fullsized":"2024-06-29T15:58:22.799428+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:27.973848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720413+0000","last_change":"2024-06-29T15:58:20.568004+0000","last_active":"2024-06-29T15:58:23.720413+0000","last_peered":"2024-06-29T15:58:23.720413+0000","last_clean":"2024-06-29T15:58:23.720413+0000","last_became_active":"2024-06-29T15:58:20.567638+0000","last_became_peered":"2024-06-29T15:58:20.567638+0000","last_unstale":"2024-06-29T15:58:23.720413+0000","last_undegraded":"2024-06-29T15:58:23.720413+0000","last_fullsized":"2024-06-29T15:58:23.720413+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:19.772821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720384+0000","last_change":"2024-06-29T15:58:21.566347+0000","last_active":"2024-06-29T15:58:23.720384+0000","last_peered":"2024-06-29T15:58:23.720384+0000","last_clean":"2024-06-29T15:58:23.720384+0000","last_became_active":"2024-06-29T15:58:21.565888+0000","last_became_peered":"2024-06-29T15:58:21.565888+0000","last_unstale":"2024-06-29T15:58:23.720384+0000","last_undegraded":"2024-06-29T15:58:23.720384+0000","last_fullsized":"2024-06-29T15:58:23.720384+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:28.461061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618013+0000","last_change":"2024-06-29T15:58:20.556653+0000","last_active":"2024-06-29T15:58:22.618013+0000","last_peered":"2024-06-29T15:58:22.618013+0000","last_clean":"2024-06-29T15:58:22.618013+0000","last_became_active":"2024-06-29T15:58:20.555367+0000","last_became_peered":"2024-06-29T15:58:20.555367+0000","last_unstale":"2024-06-29T15:58:22.618013+0000","last_undegraded":"2024-06-29T15:58:22.618013+0000","last_fullsized":"2024-06-29T15:58:22.618013+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:31.173223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548678+0000","last_change":"2024-06-29T15:58:21.562456+0000","last_active":"2024-06-29T15:58:22.548678+0000","last_peered":"2024-06-29T15:58:22.548678+0000","last_clean":"2024-06-29T15:58:22.548678+0000","last_became_active":"2024-06-29T15:58:21.562024+0000","last_became_peered":"2024-06-29T15:58:21.562024+0000","last_unstale":"2024-06-29T15:58:22.548678+0000","last_undegraded":"2024-06-29T15:58:22.548678+0000","last_fullsized":"2024-06-29T15:58:22.548678+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:36.495502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719745+0000","last_change":"2024-06-29T15:58:20.558360+0000","last_active":"2024-06-29T15:58:23.719745+0000","last_peered":"2024-06-29T15:58:23.719745+0000","last_clean":"2024-06-29T15:58:23.719745+0000","last_became_active":"2024-06-29T15:58:20.558053+0000","last_became_peered":"2024-06-29T15:58:20.558053+0000","last_unstale":"2024-06-29T15:58:23.719745+0000","last_undegraded":"2024-06-29T15:58:23.719745+0000","last_fullsized":"2024-06-29T15:58:23.719745+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:40.258636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542198+0000","last_change":"2024-06-29T15:58:21.568464+0000","last_active":"2024-06-29T15:58:22.542198+0000","last_peered":"2024-06-29T15:58:22.542198+0000","last_clean":"2024-06-29T15:58:22.542198+0000","last_became_active":"2024-06-29T15:58:21.568114+0000","last_became_peered":"2024-06-29T15:58:21.568114+0000","last_unstale":"2024-06-29T15:58:22.542198+0000","last_undegraded":"2024-06-29T15:58:22.542198+0000","last_fullsized":"2024-06-29T15:58:22.542198+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:09.625056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541676+0000","last_change":"2024-06-29T15:58:20.563140+0000","last_active":"2024-06-29T15:58:22.541676+0000","last_peered":"2024-06-29T15:58:22.541676+0000","last_clean":"2024-06-29T15:58:22.541676+0000","last_became_active":"2024-06-29T15:58:20.562823+0000","last_became_peered":"2024-06-29T15:58:20.562823+0000","last_unstale":"2024-06-29T15:58:22.541676+0000","last_undegraded":"2024-06-29T15:58:22.541676+0000","last_fullsized":"2024-06-29T15:58:22.541676+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:24.191012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548052+0000","last_change":"2024-06-29T15:58:21.556649+0000","last_active":"2024-06-29T15:58:22.548052+0000","last_peered":"2024-06-29T15:58:22.548052+0000","last_clean":"2024-06-29T15:58:22.548052+0000","last_became_active":"2024-06-29T15:58:21.556297+0000","last_became_peered":"2024-06-29T15:58:21.556297+0000","last_unstale":"2024-06-29T15:58:22.548052+0000","last_undegraded":"2024-06-29T15:58:22.548052+0000","last_fullsized":"2024-06-29T15:58:22.548052+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:07.319988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799150+0000","last_change":"2024-06-29T15:58:20.555706+0000","last_active":"2024-06-29T15:58:22.799150+0000","last_peered":"2024-06-29T15:58:22.799150+0000","last_clean":"2024-06-29T15:58:22.799150+0000","last_became_active":"2024-06-29T15:58:20.555380+0000","last_became_peered":"2024-06-29T15:58:20.555380+0000","last_unstale":"2024-06-29T15:58:22.799150+0000","last_undegraded":"2024-06-29T15:58:22.799150+0000","last_fullsized":"2024-06-29T15:58:22.799150+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:03:17.347006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548358+0000","last_change":"2024-06-29T15:58:21.554565+0000","last_active":"2024-06-29T15:58:22.548358+0000","last_peered":"2024-06-29T15:58:22.548358+0000","last_clean":"2024-06-29T15:58:22.548358+0000","last_became_active":"2024-06-29T15:58:21.554255+0000","last_became_peered":"2024-06-29T15:58:21.554255+0000","last_unstale":"2024-06-29T15:58:22.548358+0000","last_undegraded":"2024-06-29T15:58:22.548358+0000","last_fullsized":"2024-06-29T15:58:22.548358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:14.470919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720200+0000","last_change":"2024-06-29T15:58:20.562271+0000","last_active":"2024-06-29T15:58:23.720200+0000","last_peered":"2024-06-29T15:58:23.720200+0000","last_clean":"2024-06-29T15:58:23.720200+0000","last_became_active":"2024-06-29T15:58:20.561940+0000","last_became_peered":"2024-06-29T15:58:20.561940+0000","last_unstale":"2024-06-29T15:58:23.720200+0000","last_undegraded":"2024-06-29T15:58:23.720200+0000","last_fullsized":"2024-06-29T15:58:23.720200+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:16.036109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597833+0000","last_change":"2024-06-29T15:58:21.548845+0000","last_active":"2024-06-29T15:58:22.597833+0000","last_peered":"2024-06-29T15:58:22.597833+0000","last_clean":"2024-06-29T15:58:22.597833+0000","last_became_active":"2024-06-29T15:58:21.548448+0000","last_became_peered":"2024-06-29T15:58:21.548448+0000","last_unstale":"2024-06-29T15:58:22.597833+0000","last_undegraded":"2024-06-29T15:58:22.597833+0000","last_fullsized":"2024-06-29T15:58:22.597833+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:38.983134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542265+0000","last_change":"2024-06-29T15:58:20.562116+0000","last_active":"2024-06-29T15:58:22.542265+0000","last_peered":"2024-06-29T15:58:22.542265+0000","last_clean":"2024-06-29T15:58:22.542265+0000","last_became_active":"2024-06-29T15:58:20.561801+0000","last_became_peered":"2024-06-29T15:58:20.561801+0000","last_unstale":"2024-06-29T15:58:22.542265+0000","last_undegraded":"2024-06-29T15:58:22.542265+0000","last_fullsized":"2024-06-29T15:58:22.542265+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:26.721883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542073+0000","last_change":"2024-06-29T15:58:21.560085+0000","last_active":"2024-06-29T15:58:22.542073+0000","last_peered":"2024-06-29T15:58:22.542073+0000","last_clean":"2024-06-29T15:58:22.542073+0000","last_became_active":"2024-06-29T15:58:21.559716+0000","last_became_peered":"2024-06-29T15:58:21.559716+0000","last_unstale":"2024-06-29T15:58:22.542073+0000","last_undegraded":"2024-06-29T15:58:22.542073+0000","last_fullsized":"2024-06-29T15:58:22.542073+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:15:37.579793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"241'3","reported_seq":17,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547764+0000","last_change":"2024-06-29T15:58:20.553528+0000","last_active":"2024-06-29T15:58:22.547764+0000","last_peered":"2024-06-29T15:58:22.547764+0000","last_clean":"2024-06-29T15:58:22.547764+0000","last_became_active":"2024-06-29T15:58:20.553185+0000","last_became_peered":"2024-06-29T15:58:20.553185+0000","last_unstale":"2024-06-29T15:58:22.547764+0000","last_undegraded":"2024-06-29T15:58:22.547764+0000","last_fullsized":"2024-06-29T15:58:22.547764+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:26.807742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548371+0000","last_change":"2024-06-29T15:58:21.553783+0000","last_active":"2024-06-29T15:58:22.548371+0000","last_peered":"2024-06-29T15:58:22.548371+0000","last_clean":"2024-06-29T15:58:22.548371+0000","last_became_active":"2024-06-29T15:58:21.553278+0000","last_became_peered":"2024-06-29T15:58:21.553278+0000","last_unstale":"2024-06-29T15:58:22.548371+0000","last_undegraded":"2024-06-29T15:58:22.548371+0000","last_fullsized":"2024-06-29T15:58:22.548371+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:39.179054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"241'6","reported_seq":20,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799462+0000","last_change":"2024-06-29T15:58:20.557583+0000","last_active":"2024-06-29T15:58:22.799462+0000","last_peered":"2024-06-29T15:58:22.799462+0000","last_clean":"2024-06-29T15:58:22.799462+0000","last_became_active":"2024-06-29T15:58:20.557274+0000","last_became_peered":"2024-06-29T15:58:20.557274+0000","last_unstale":"2024-06-29T15:58:22.799462+0000","last_undegraded":"2024-06-29T15:58:22.799462+0000","last_fullsized":"2024-06-29T15:58:22.799462+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:38.435050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719655+0000","last_change":"2024-06-29T15:58:21.559479+0000","last_active":"2024-06-29T15:58:23.719655+0000","last_peered":"2024-06-29T15:58:23.719655+0000","last_clean":"2024-06-29T15:58:23.719655+0000","last_became_active":"2024-06-29T15:58:21.559128+0000","last_became_peered":"2024-06-29T15:58:21.559128+0000","last_unstale":"2024-06-29T15:58:23.719655+0000","last_undegraded":"2024-06-29T15:58:23.719655+0000","last_fullsized":"2024-06-29T15:58:23.719655+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:28.321473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618061+0000","last_change":"2024-06-29T15:58:20.561589+0000","last_active":"2024-06-29T15:58:22.618061+0000","last_peered":"2024-06-29T15:58:22.618061+0000","last_clean":"2024-06-29T15:58:22.618061+0000","last_became_active":"2024-06-29T15:58:20.561251+0000","last_became_peered":"2024-06-29T15:58:20.561251+0000","last_unstale":"2024-06-29T15:58:22.618061+0000","last_undegraded":"2024-06-29T15:58:22.618061+0000","last_fullsized":"2024-06-29T15:58:22.618061+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:09.657256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548626+0000","last_change":"2024-06-29T15:58:21.560367+0000","last_active":"2024-06-29T15:58:22.548626+0000","last_peered":"2024-06-29T15:58:22.548626+0000","last_clean":"2024-06-29T15:58:22.548626+0000","last_became_active":"2024-06-29T15:58:21.560056+0000","last_became_peered":"2024-06-29T15:58:21.560056+0000","last_unstale":"2024-06-29T15:58:22.548626+0000","last_undegraded":"2024-06-29T15:58:22.548626+0000","last_fullsized":"2024-06-29T15:58:22.548626+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:43.371506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547696+0000","last_change":"2024-06-29T15:58:20.552449+0000","last_active":"2024-06-29T15:58:22.547696+0000","last_peered":"2024-06-29T15:58:22.547696+0000","last_clean":"2024-06-29T15:58:22.547696+0000","last_became_active":"2024-06-29T15:58:20.551202+0000","last_became_peered":"2024-06-29T15:58:20.551202+0000","last_unstale":"2024-06-29T15:58:22.547696+0000","last_undegraded":"2024-06-29T15:58:22.547696+0000","last_fullsized":"2024-06-29T15:58:22.547696+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:35.829408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618192+0000","last_change":"2024-06-29T15:58:21.566336+0000","last_active":"2024-06-29T15:58:22.618192+0000","last_peered":"2024-06-29T15:58:22.618192+0000","last_clean":"2024-06-29T15:58:22.618192+0000","last_became_active":"2024-06-29T15:58:21.566003+0000","last_became_peered":"2024-06-29T15:58:21.566003+0000","last_unstale":"2024-06-29T15:58:22.618192+0000","last_undegraded":"2024-06-29T15:58:22.618192+0000","last_fullsized":"2024-06-29T15:58:22.618192+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:46:35.306688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547153+0000","last_change":"2024-06-29T15:58:20.557295+0000","last_active":"2024-06-29T15:58:22.547153+0000","last_peered":"2024-06-29T15:58:22.547153+0000","last_clean":"2024-06-29T15:58:22.547153+0000","last_became_active":"2024-06-29T15:58:20.556868+0000","last_became_peered":"2024-06-29T15:58:20.556868+0000","last_unstale":"2024-06-29T15:58:22.547153+0000","last_undegraded":"2024-06-29T15:58:22.547153+0000","last_fullsized":"2024-06-29T15:58:22.547153+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:31.226218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547857+0000","last_change":"2024-06-29T15:58:21.565109+0000","last_active":"2024-06-29T15:58:22.547857+0000","last_peered":"2024-06-29T15:58:22.547857+0000","last_clean":"2024-06-29T15:58:22.547857+0000","last_became_active":"2024-06-29T15:58:21.564764+0000","last_became_peered":"2024-06-29T15:58:21.564764+0000","last_unstale":"2024-06-29T15:58:22.547857+0000","last_undegraded":"2024-06-29T15:58:22.547857+0000","last_fullsized":"2024-06-29T15:58:22.547857+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:43:24.997613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"241'8","reported_seq":22,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719193+0000","last_change":"2024-06-29T15:58:20.562144+0000","last_active":"2024-06-29T15:58:23.719193+0000","last_peered":"2024-06-29T15:58:23.719193+0000","last_clean":"2024-06-29T15:58:23.719193+0000","last_became_active":"2024-06-29T15:58:20.561879+0000","last_became_peered":"2024-06-29T15:58:20.561879+0000","last_unstale":"2024-06-29T15:58:23.719193+0000","last_undegraded":"2024-06-29T15:58:23.719193+0000","last_fullsized":"2024-06-29T15:58:23.719193+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:28.961624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547186+0000","last_change":"2024-06-29T15:58:21.546206+0000","last_active":"2024-06-29T15:58:22.547186+0000","last_peered":"2024-06-29T15:58:22.547186+0000","last_clean":"2024-06-29T15:58:22.547186+0000","last_became_active":"2024-06-29T15:58:21.545742+0000","last_became_peered":"2024-06-29T15:58:21.545742+0000","last_unstale":"2024-06-29T15:58:22.547186+0000","last_undegraded":"2024-06-29T15:58:22.547186+0000","last_fullsized":"2024-06-29T15:58:22.547186+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:57.523444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799610+0000","last_change":"2024-06-29T15:58:20.555827+0000","last_active":"2024-06-29T15:58:22.799610+0000","last_peered":"2024-06-29T15:58:22.799610+0000","last_clean":"2024-06-29T15:58:22.799610+0000","last_became_active":"2024-06-29T15:58:20.555506+0000","last_became_peered":"2024-06-29T15:58:20.555506+0000","last_unstale":"2024-06-29T15:58:22.799610+0000","last_undegraded":"2024-06-29T15:58:22.799610+0000","last_fullsized":"2024-06-29T15:58:22.799610+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:33:20.721321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547069+0000","last_change":"2024-06-29T15:58:21.558057+0000","last_active":"2024-06-29T15:58:22.547069+0000","last_peered":"2024-06-29T15:58:22.547069+0000","last_clean":"2024-06-29T15:58:22.547069+0000","last_became_active":"2024-06-29T15:58:21.557750+0000","last_became_peered":"2024-06-29T15:58:21.557750+0000","last_unstale":"2024-06-29T15:58:22.547069+0000","last_undegraded":"2024-06-29T15:58:22.547069+0000","last_fullsized":"2024-06-29T15:58:22.547069+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:35.875818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719642+0000","last_change":"2024-06-29T15:58:20.561995+0000","last_active":"2024-06-29T15:58:23.719642+0000","last_peered":"2024-06-29T15:58:23.719642+0000","last_clean":"2024-06-29T15:58:23.719642+0000","last_became_active":"2024-06-29T15:58:20.560719+0000","last_became_peered":"2024-06-29T15:58:20.560719+0000","last_unstale":"2024-06-29T15:58:23.719642+0000","last_undegraded":"2024-06-29T15:58:23.719642+0000","last_fullsized":"2024-06-29T15:58:23.719642+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:59.752024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719119+0000","last_change":"2024-06-29T15:58:21.552894+0000","last_active":"2024-06-29T15:58:23.719119+0000","last_peered":"2024-06-29T15:58:23.719119+0000","last_clean":"2024-06-29T15:58:23.719119+0000","last_became_active":"2024-06-29T15:58:21.552487+0000","last_became_peered":"2024-06-29T15:58:21.552487+0000","last_unstale":"2024-06-29T15:58:23.719119+0000","last_undegraded":"2024-06-29T15:58:23.719119+0000","last_fullsized":"2024-06-29T15:58:23.719119+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:49.309450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598156+0000","last_change":"2024-06-29T15:58:20.544642+0000","last_active":"2024-06-29T15:58:22.598156+0000","last_peered":"2024-06-29T15:58:22.598156+0000","last_clean":"2024-06-29T15:58:22.598156+0000","last_became_active":"2024-06-29T15:58:20.544356+0000","last_became_peered":"2024-06-29T15:58:20.544356+0000","last_unstale":"2024-06-29T15:58:22.598156+0000","last_undegraded":"2024-06-29T15:58:22.598156+0000","last_fullsized":"2024-06-29T15:58:22.598156+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:38.143805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542024+0000","last_change":"2024-06-29T15:58:21.560349+0000","last_active":"2024-06-29T15:58:22.542024+0000","last_peered":"2024-06-29T15:58:22.542024+0000","last_clean":"2024-06-29T15:58:22.542024+0000","last_became_active":"2024-06-29T15:58:21.559987+0000","last_became_peered":"2024-06-29T15:58:21.559987+0000","last_unstale":"2024-06-29T15:58:22.542024+0000","last_undegraded":"2024-06-29T15:58:22.542024+0000","last_fullsized":"2024-06-29T15:58:22.542024+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:20.317519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.617864+0000","last_change":"2024-06-29T15:58:20.551598+0000","last_active":"2024-06-29T15:58:22.617864+0000","last_peered":"2024-06-29T15:58:22.617864+0000","last_clean":"2024-06-29T15:58:22.617864+0000","last_became_active":"2024-06-29T15:58:20.551266+0000","last_became_peered":"2024-06-29T15:58:20.551266+0000","last_unstale":"2024-06-29T15:58:22.617864+0000","last_undegraded":"2024-06-29T15:58:22.617864+0000","last_fullsized":"2024-06-29T15:58:22.617864+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:56:15.731511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541950+0000","last_change":"2024-06-29T15:58:21.562779+0000","last_active":"2024-06-29T15:58:22.541950+0000","last_peered":"2024-06-29T15:58:22.541950+0000","last_clean":"2024-06-29T15:58:22.541950+0000","last_became_active":"2024-06-29T15:58:21.562503+0000","last_became_peered":"2024-06-29T15:58:21.562503+0000","last_unstale":"2024-06-29T15:58:22.541950+0000","last_undegraded":"2024-06-29T15:58:22.541950+0000","last_fullsized":"2024-06-29T15:58:22.541950+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:28.461061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541921+0000","last_change":"2024-06-29T15:58:20.559588+0000","last_active":"2024-06-29T15:58:22.541921+0000","last_peered":"2024-06-29T15:58:22.541921+0000","last_clean":"2024-06-29T15:58:22.541921+0000","last_became_active":"2024-06-29T15:58:20.559245+0000","last_became_peered":"2024-06-29T15:58:20.559245+0000","last_unstale":"2024-06-29T15:58:22.541921+0000","last_undegraded":"2024-06-29T15:58:22.541921+0000","last_fullsized":"2024-06-29T15:58:22.541921+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:40.258636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719349+0000","last_change":"2024-06-29T15:58:21.553384+0000","last_active":"2024-06-29T15:58:23.719349+0000","last_peered":"2024-06-29T15:58:23.719349+0000","last_clean":"2024-06-29T15:58:23.719349+0000","last_became_active":"2024-06-29T15:58:21.551240+0000","last_became_peered":"2024-06-29T15:58:21.551240+0000","last_unstale":"2024-06-29T15:58:23.719349+0000","last_undegraded":"2024-06-29T15:58:23.719349+0000","last_fullsized":"2024-06-29T15:58:23.719349+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:38.040891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597927+0000","last_change":"2024-06-29T15:58:20.557247+0000","last_active":"2024-06-29T15:58:22.597927+0000","last_peered":"2024-06-29T15:58:22.597927+0000","last_clean":"2024-06-29T15:58:22.597927+0000","last_became_active":"2024-06-29T15:58:20.556889+0000","last_became_peered":"2024-06-29T15:58:20.556889+0000","last_unstale":"2024-06-29T15:58:22.597927+0000","last_undegraded":"2024-06-29T15:58:22.597927+0000","last_fullsized":"2024-06-29T15:58:22.597927+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:48.301087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598280+0000","last_change":"2024-06-29T15:58:21.558641+0000","last_active":"2024-06-29T15:58:22.598280+0000","last_peered":"2024-06-29T15:58:22.598280+0000","last_clean":"2024-06-29T15:58:22.598280+0000","last_became_active":"2024-06-29T15:58:21.558377+0000","last_became_peered":"2024-06-29T15:58:21.558377+0000","last_unstale":"2024-06-29T15:58:22.598280+0000","last_undegraded":"2024-06-29T15:58:22.598280+0000","last_fullsized":"2024-06-29T15:58:22.598280+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:09.589023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597747+0000","last_change":"2024-06-29T15:58:20.543880+0000","last_active":"2024-06-29T15:58:22.597747+0000","last_peered":"2024-06-29T15:58:22.597747+0000","last_clean":"2024-06-29T15:58:22.597747+0000","last_became_active":"2024-06-29T15:58:20.543365+0000","last_became_peered":"2024-06-29T15:58:20.543365+0000","last_unstale":"2024-06-29T15:58:22.597747+0000","last_undegraded":"2024-06-29T15:58:22.597747+0000","last_fullsized":"2024-06-29T15:58:22.597747+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:18:31.509521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598176+0000","last_change":"2024-06-29T15:58:21.555113+0000","last_active":"2024-06-29T15:58:22.598176+0000","last_peered":"2024-06-29T15:58:22.598176+0000","last_clean":"2024-06-29T15:58:22.598176+0000","last_became_active":"2024-06-29T15:58:21.553362+0000","last_became_peered":"2024-06-29T15:58:21.553362+0000","last_unstale":"2024-06-29T15:58:22.598176+0000","last_undegraded":"2024-06-29T15:58:22.598176+0000","last_fullsized":"2024-06-29T15:58:22.598176+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:18.837956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.718819+0000","last_change":"2024-06-29T15:58:20.551665+0000","last_active":"2024-06-29T15:58:23.718819+0000","last_peered":"2024-06-29T15:58:23.718819+0000","last_clean":"2024-06-29T15:58:23.718819+0000","last_became_active":"2024-06-29T15:58:20.551347+0000","last_became_peered":"2024-06-29T15:58:20.551347+0000","last_unstale":"2024-06-29T15:58:23.718819+0000","last_undegraded":"2024-06-29T15:58:23.718819+0000","last_fullsized":"2024-06-29T15:58:23.718819+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:22:40.223323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548008+0000","last_change":"2024-06-29T15:58:21.561571+0000","last_active":"2024-06-29T15:58:22.548008+0000","last_peered":"2024-06-29T15:58:22.548008+0000","last_clean":"2024-06-29T15:58:22.548008+0000","last_became_active":"2024-06-29T15:58:21.561042+0000","last_became_peered":"2024-06-29T15:58:21.561042+0000","last_unstale":"2024-06-29T15:58:22.548008+0000","last_undegraded":"2024-06-29T15:58:22.548008+0000","last_fullsized":"2024-06-29T15:58:22.548008+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:34.199222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"241'2","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.542229+0000","last_change":"2024-06-29T15:58:20.563978+0000","last_active":"2024-06-29T15:58:22.542229+0000","last_peered":"2024-06-29T15:58:22.542229+0000","last_clean":"2024-06-29T15:58:22.542229+0000","last_became_active":"2024-06-29T15:58:20.563680+0000","last_became_peered":"2024-06-29T15:58:20.563680+0000","last_unstale":"2024-06-29T15:58:22.542229+0000","last_undegraded":"2024-06-29T15:58:22.542229+0000","last_fullsized":"2024-06-29T15:58:22.542229+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:27.903522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.618056+0000","last_change":"2024-06-29T15:58:21.565805+0000","last_active":"2024-06-29T15:58:22.618056+0000","last_peered":"2024-06-29T15:58:22.618056+0000","last_clean":"2024-06-29T15:58:22.618056+0000","last_became_active":"2024-06-29T15:58:21.564982+0000","last_became_peered":"2024-06-29T15:58:21.564982+0000","last_unstale":"2024-06-29T15:58:22.618056+0000","last_undegraded":"2024-06-29T15:58:22.618056+0000","last_fullsized":"2024-06-29T15:58:22.618056+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:18.217838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"242'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548253+0000","last_change":"2024-06-29T15:58:20.554719+0000","last_active":"2024-06-29T15:58:22.548253+0000","last_peered":"2024-06-29T15:58:22.548253+0000","last_clean":"2024-06-29T15:58:22.548253+0000","last_became_active":"2024-06-29T15:58:20.554423+0000","last_became_peered":"2024-06-29T15:58:20.554423+0000","last_unstale":"2024-06-29T15:58:22.548253+0000","last_undegraded":"2024-06-29T15:58:22.548253+0000","last_fullsized":"2024-06-29T15:58:22.548253+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:36.112729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541842+0000","last_change":"2024-06-29T15:58:21.559427+0000","last_active":"2024-06-29T15:58:22.541842+0000","last_peered":"2024-06-29T15:58:22.541842+0000","last_clean":"2024-06-29T15:58:22.541842+0000","last_became_active":"2024-06-29T15:58:21.558942+0000","last_became_peered":"2024-06-29T15:58:21.558942+0000","last_unstale":"2024-06-29T15:58:22.541842+0000","last_undegraded":"2024-06-29T15:58:22.541842+0000","last_fullsized":"2024-06-29T15:58:22.541842+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:36.096648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719798+0000","last_change":"2024-06-29T15:58:20.559651+0000","last_active":"2024-06-29T15:58:23.719798+0000","last_peered":"2024-06-29T15:58:23.719798+0000","last_clean":"2024-06-29T15:58:23.719798+0000","last_became_active":"2024-06-29T15:58:20.559323+0000","last_became_peered":"2024-06-29T15:58:20.559323+0000","last_unstale":"2024-06-29T15:58:23.719798+0000","last_undegraded":"2024-06-29T15:58:23.719798+0000","last_fullsized":"2024-06-29T15:58:23.719798+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:18:36.258363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547295+0000","last_change":"2024-06-29T15:58:21.545630+0000","last_active":"2024-06-29T15:58:22.547295+0000","last_peered":"2024-06-29T15:58:22.547295+0000","last_clean":"2024-06-29T15:58:22.547295+0000","last_became_active":"2024-06-29T15:58:21.545301+0000","last_became_peered":"2024-06-29T15:58:21.545301+0000","last_unstale":"2024-06-29T15:58:22.547295+0000","last_undegraded":"2024-06-29T15:58:22.547295+0000","last_fullsized":"2024-06-29T15:58:22.547295+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:36.336990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720428+0000","last_change":"2024-06-29T15:58:20.556695+0000","last_active":"2024-06-29T15:58:23.720428+0000","last_peered":"2024-06-29T15:58:23.720428+0000","last_clean":"2024-06-29T15:58:23.720428+0000","last_became_active":"2024-06-29T15:58:20.556053+0000","last_became_peered":"2024-06-29T15:58:20.556053+0000","last_unstale":"2024-06-29T15:58:23.720428+0000","last_undegraded":"2024-06-29T15:58:23.720428+0000","last_fullsized":"2024-06-29T15:58:23.720428+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:15.325464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.720205+0000","last_change":"2024-06-29T15:58:21.565888+0000","last_active":"2024-06-29T15:58:23.720205+0000","last_peered":"2024-06-29T15:58:23.720205+0000","last_clean":"2024-06-29T15:58:23.720205+0000","last_became_active":"2024-06-29T15:58:21.565418+0000","last_became_peered":"2024-06-29T15:58:21.565418+0000","last_unstale":"2024-06-29T15:58:23.720205+0000","last_undegraded":"2024-06-29T15:58:23.720205+0000","last_fullsized":"2024-06-29T15:58:23.720205+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:57.843559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597925+0000","last_change":"2024-06-29T15:58:20.557670+0000","last_active":"2024-06-29T15:58:22.597925+0000","last_peered":"2024-06-29T15:58:22.597925+0000","last_clean":"2024-06-29T15:58:22.597925+0000","last_became_active":"2024-06-29T15:58:20.557366+0000","last_became_peered":"2024-06-29T15:58:20.557366+0000","last_unstale":"2024-06-29T15:58:22.597925+0000","last_undegraded":"2024-06-29T15:58:22.597925+0000","last_fullsized":"2024-06-29T15:58:22.597925+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:08.866031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799266+0000","last_change":"2024-06-29T15:58:21.562317+0000","last_active":"2024-06-29T15:58:22.799266+0000","last_peered":"2024-06-29T15:58:22.799266+0000","last_clean":"2024-06-29T15:58:22.799266+0000","last_became_active":"2024-06-29T15:58:21.561968+0000","last_became_peered":"2024-06-29T15:58:21.561968+0000","last_unstale":"2024-06-29T15:58:22.799266+0000","last_undegraded":"2024-06-29T15:58:22.799266+0000","last_fullsized":"2024-06-29T15:58:22.799266+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.632416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.547844+0000","last_change":"2024-06-29T15:58:20.547790+0000","last_active":"2024-06-29T15:58:22.547844+0000","last_peered":"2024-06-29T15:58:22.547844+0000","last_clean":"2024-06-29T15:58:22.547844+0000","last_became_active":"2024-06-29T15:58:20.547481+0000","last_became_peered":"2024-06-29T15:58:20.547481+0000","last_unstale":"2024-06-29T15:58:22.547844+0000","last_undegraded":"2024-06-29T15:58:22.547844+0000","last_fullsized":"2024-06-29T15:58:22.547844+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:32.913616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:23.719588+0000","last_change":"2024-06-29T15:58:21.553783+0000","last_active":"2024-06-29T15:58:23.719588+0000","last_peered":"2024-06-29T15:58:23.719588+0000","last_clean":"2024-06-29T15:58:23.719588+0000","last_became_active":"2024-06-29T15:58:21.553390+0000","last_became_peered":"2024-06-29T15:58:21.553390+0000","last_unstale":"2024-06-29T15:58:23.719588+0000","last_undegraded":"2024-06-29T15:58:23.719588+0000","last_fullsized":"2024-06-29T15:58:23.719588+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:01.841016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.541769+0000","last_change":"2024-06-29T15:58:20.561399+0000","last_active":"2024-06-29T15:58:22.541769+0000","last_peered":"2024-06-29T15:58:22.541769+0000","last_clean":"2024-06-29T15:58:22.541769+0000","last_became_active":"2024-06-29T15:58:20.560716+0000","last_became_peered":"2024-06-29T15:58:20.560716+0000","last_unstale":"2024-06-29T15:58:22.541769+0000","last_undegraded":"2024-06-29T15:58:22.541769+0000","last_fullsized":"2024-06-29T15:58:22.541769+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:42:19.927414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.598004+0000","last_change":"2024-06-29T15:58:21.563096+0000","last_active":"2024-06-29T15:58:22.598004+0000","last_peered":"2024-06-29T15:58:22.598004+0000","last_clean":"2024-06-29T15:58:22.598004+0000","last_became_active":"2024-06-29T15:58:21.562778+0000","last_became_peered":"2024-06-29T15:58:21.562778+0000","last_unstale":"2024-06-29T15:58:22.598004+0000","last_undegraded":"2024-06-29T15:58:22.598004+0000","last_fullsized":"2024-06-29T15:58:22.598004+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:20.808675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.597580+0000","last_change":"2024-06-29T15:58:20.554308+0000","last_active":"2024-06-29T15:58:22.597580+0000","last_peered":"2024-06-29T15:58:22.597580+0000","last_clean":"2024-06-29T15:58:22.597580+0000","last_became_active":"2024-06-29T15:58:20.553976+0000","last_became_peered":"2024-06-29T15:58:20.553976+0000","last_unstale":"2024-06-29T15:58:22.597580+0000","last_undegraded":"2024-06-29T15:58:22.597580+0000","last_fullsized":"2024-06-29T15:58:22.597580+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:13:16.241344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.799135+0000","last_change":"2024-06-29T15:58:21.550432+0000","last_active":"2024-06-29T15:58:22.799135+0000","last_peered":"2024-06-29T15:58:22.799135+0000","last_clean":"2024-06-29T15:58:22.799135+0000","last_became_active":"2024-06-29T15:58:21.550075+0000","last_became_peered":"2024-06-29T15:58:21.550075+0000","last_unstale":"2024-06-29T15:58:22.799135+0000","last_undegraded":"2024-06-29T15:58:22.799135+0000","last_fullsized":"2024-06-29T15:58:22.799135+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:24.427535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"241'1","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.548196+0000","last_change":"2024-06-29T15:58:20.561371+0000","last_active":"2024-06-29T15:58:22.548196+0000","last_peered":"2024-06-29T15:58:22.548196+0000","last_clean":"2024-06-29T15:58:22.548196+0000","last_became_active":"2024-06-29T15:58:20.561026+0000","last_became_peered":"2024-06-29T15:58:20.561026+0000","last_unstale":"2024-06-29T15:58:22.548196+0000","last_undegraded":"2024-06-29T15:58:22.548196+0000","last_fullsized":"2024-06-29T15:58:22.548196+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:07.861501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"228'247","reported_seq":1190,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.617778+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:58:22.617778+0000","last_peered":"2024-06-29T15:58:22.617778+0000","last_clean":"2024-06-29T15:58:22.617778+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:58:22.617778+0000","last_undegraded":"2024-06-29T15:58:22.617778+0000","last_fullsized":"2024-06-29T15:58:22.617778+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543298,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52792,"kb_used_data":3560,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319048,"statfs":{"total":96636764160,"available":96582705152,"internally_reserved":0,"allocated":3645440,"data_stored":2163708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543299,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60528,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311312,"statfs":{"total":96636764160,"available":96574783488,"internally_reserved":0,"allocated":3051520,"data_stored":1573328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543298,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70248,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301592,"statfs":{"total":96636764160,"available":96564830208,"internally_reserved":0,"allocated":3043328,"data_stored":1573216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543297,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52204,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319636,"statfs":{"total":96636764160,"available":96583307264,"internally_reserved":0,"allocated":3047424,"data_stored":1573286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510595,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48064,"kb_used_data":2996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323776,"statfs":{"total":96636764160,"available":96587546624,"internally_reserved":0,"allocated":3067904,"data_stored":1575466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48052,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323788,"statfs":{"total":96636764160,"available":96587558912,"internally_reserved":0,"allocated":3051520,"data_stored":1573868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510594,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48044,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323796,"statfs":{"total":96636764160,"available":96587567104,"internally_reserved":0,"allocated":3043328,"data_stored":1573216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510596,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43064,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94328776,"statfs":{"total":96636764160,"available":96592666624,"internally_reserved":0,"allocated":3649536,"data_stored":2165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:58:26.018 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:58:26.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:58:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.181+0000 7f73633ef640 1 -- 172.21.15.59:0/2857066744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c071e00 msgr2=0x7f735c072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.181+0000 7f73633ef640 1 --2- 172.21.15.59:0/2857066744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c071e00 0x7f735c072210 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f735c06fa20 tx=0x7f735402aa30 comp rx=0 tx=0).stop 2024-06-29T15:58:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.181+0000 7f73633ef640 1 -- 172.21.15.59:0/2857066744 shutdown_connections 2024-06-29T15:58:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.181+0000 7f73633ef640 1 --2- 172.21.15.59:0/2857066744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c071e00 0x7f735c072210 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.181+0000 7f73633ef640 1 --2- 172.21.15.59:0/2857066744 >> v2:172.21.15.153:3300/0 conn(0x7f735c0b43e0 0x7f735c0b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.181+0000 7f73633ef640 1 -- 172.21.15.59:0/2857066744 >> 172.21.15.59:0/2857066744 conn(0x7f735c0ccb70 msgr2=0x7f735c0ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.181+0000 7f73633ef640 1 -- 172.21.15.59:0/2857066744 shutdown_connections 2024-06-29T15:58:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.181+0000 7f73633ef640 1 -- 172.21.15.59:0/2857066744 wait complete. 2024-06-29T15:58:26.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.183+0000 7f73633ef640 1 Processor -- start 2024-06-29T15:58:26.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.183+0000 7f73633ef640 1 -- start start 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f73633ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c06d790 0x7f735c143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f73633ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c071e00 0x7f735c140c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f73633ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c0b43e0 0x7f735c14f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f73633ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f735c0b53f0 con 0x7f735c0b43e0 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f73633ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f735c0b5270 con 0x7f735c06d790 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f73633ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f735c06fef0 con 0x7f735c071e00 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c0b43e0 0x7f735c14f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c0b43e0 0x7f735c14f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35928/0 (socket says 172.21.15.59:35928) 2024-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 -- 172.21.15.59:0/2885353159 learned_addr learned my addr 172.21.15.59:0/2885353159 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736118b640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c06d790 0x7f735c143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736098a640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c071e00 0x7f735c140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 -- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c071e00 msgr2=0x7f735c140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c071e00 0x7f735c140c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 -- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c06d790 msgr2=0x7f735c143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c06d790 0x7f735c143610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 -- 172.21.15.59:0/2885353159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735c143b50 con 0x7f735c0b43e0 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736098a640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c071e00 0x7f735c140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.184+0000 7f736198c640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c0b43e0 0x7f735c14f3b0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f7354003f90 tx=0x7f7354004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.185+0000 7f7351ffb640 1 -- 172.21.15.59:0/2885353159 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354033050 con 0x7f735c0b43e0 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.185+0000 7f7351ffb640 1 -- 172.21.15.59:0/2885353159 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7354004690 con 0x7f735c0b43e0 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.185+0000 7f7351ffb640 1 -- 172.21.15.59:0/2885353159 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73540386b0 con 0x7f735c0b43e0 2024-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.185+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f735c144060 con 0x7f735c0b43e0 2024-06-29T15:58:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.185+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f735c0ada30 con 0x7f735c0b43e0 2024-06-29T15:58:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.186+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f735c143dd0 con 0x7f735c0b43e0 2024-06-29T15:58:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.187+0000 7f7351ffb640 1 -- 172.21.15.59:0/2885353159 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f7354036070 con 0x7f735c0b43e0 2024-06-29T15:58:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.187+0000 7f7351ffb640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7340076540 0x7f7340078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.187+0000 7f7351ffb640 1 -- 172.21.15.59:0/2885353159 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f73540b9320 con 0x7f735c0b43e0 2024-06-29T15:58:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.188+0000 7f736118b640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7340076540 0x7f7340078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.188+0000 7f736118b640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7340076540 0x7f7340078a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7358006fd0 tx=0x7f7358009060 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:26.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.192+0000 7f7351ffb640 1 -- 172.21.15.59:0/2885353159 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73540825c0 con 0x7f735c0b43e0 2024-06-29T15:58:26.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.461+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f735c072210 con 0x7f735c0b43e0 2024-06-29T15:58:26.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.464+0000 7f7351ffb640 1 -- 172.21.15.59:0/2885353159 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v53) ==== 190+0+147 (secure 0 0 0) 0x7f7354081ec0 con 0x7f735c0b43e0 2024-06-29T15:58:26.464 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T15:58:26.464 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T15:58:26.464 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T15:58:26.464 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T15:58:26.464 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:58:26.464 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T15:58:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7340076540 msgr2=0x7f7340078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7340076540 0x7f7340078a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7358006fd0 tx=0x7f7358009060 comp rx=0 tx=0).stop 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c0b43e0 msgr2=0x7f735c14f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c0b43e0 0x7f735c14f3b0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f7354003f90 tx=0x7f7354004040 comp rx=0 tx=0).stop 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 shutdown_connections 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f7340076540 0x7f7340078a50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f735c0b43e0 0x7f735c14f3b0 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f735c071e00 0x7f735c140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 --2- 172.21.15.59:0/2885353159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f735c06d790 0x7f735c143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 >> 172.21.15.59:0/2885353159 conn(0x7f735c0ccb70 msgr2=0x7f735c15a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 shutdown_connections 2024-06-29T15:58:26.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.466+0000 7f73633ef640 1 -- 172.21.15.59:0/2885353159 wait complete. 2024-06-29T15:58:26.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.596+0000 7f8c801bd640 1 -- 172.21.15.59:0/2646533341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7815e0a0 msgr2=0x7f8c780ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.596+0000 7f8c801bd640 1 --2- 172.21.15.59:0/2646533341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7815e0a0 0x7f8c780ce550 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f8c68005fd0 tx=0x7f8c6802aaa0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.596+0000 7f8c801bd640 1 -- 172.21.15.59:0/2646533341 shutdown_connections 2024-06-29T15:58:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.596+0000 7f8c801bd640 1 --2- 172.21.15.59:0/2646533341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7815e0a0 0x7f8c780ce550 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.596+0000 7f8c801bd640 1 -- 172.21.15.59:0/2646533341 >> 172.21.15.59:0/2646533341 conn(0x7f8c780d45b0 msgr2=0x7f8c780d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.596+0000 7f8c801bd640 1 -- 172.21.15.59:0/2646533341 shutdown_connections 2024-06-29T15:58:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.596+0000 7f8c801bd640 1 -- 172.21.15.59:0/2646533341 wait complete. 2024-06-29T15:58:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.598+0000 7f8c801bd640 1 Processor -- start 2024-06-29T15:58:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.598+0000 7f8c801bd640 1 -- start start 2024-06-29T15:58:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c801bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7815e0a0 0x7f8c7815af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c801bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c78162770 0x7f8c780c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c801bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c78159ad0 0x7f8c781418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c801bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8c780b4470 con 0x7f8c78159ad0 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c801bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c780b42f0 con 0x7f8c7815e0a0 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c801bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8c780b45f0 con 0x7f8c78162770 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c7e75a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c78159ad0 0x7f8c781418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c7e75a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c78159ad0 0x7f8c781418e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35942/0 (socket says 172.21.15.59:35942) 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.599+0000 7f8c7e75a640 1 -- 172.21.15.59:0/1056825902 learned_addr learned my addr 172.21.15.59:0/1056825902 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c7df59640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7815e0a0 0x7f8c7815af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c7d758640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c78162770 0x7f8c780c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c7e75a640 1 -- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c78162770 msgr2=0x7f8c780c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c7e75a640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c78162770 0x7f8c780c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c7e75a640 1 -- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7815e0a0 msgr2=0x7f8c7815af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c7e75a640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7815e0a0 0x7f8c7815af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c7e75a640 1 -- 172.21.15.59:0/1056825902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7815b750 con 0x7f8c78159ad0 2024-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c7e75a640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c78159ad0 0x7f8c781418e0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009d90 tx=0x7f8c68003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c767fc640 1 -- 172.21.15.59:0/1056825902 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68044070 con 0x7f8c78159ad0 2024-06-29T15:58:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c767fc640 1 -- 172.21.15.59:0/1056825902 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c68004250 con 0x7f8c78159ad0 2024-06-29T15:58:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7815ba30 con 0x7f8c78159ad0 2024-06-29T15:58:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c767fc640 1 -- 172.21.15.59:0/1056825902 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68004660 con 0x7f8c78159ad0 2024-06-29T15:58:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.600+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c78140f60 con 0x7f8c78159ad0 2024-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.601+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c781411c0 con 0x7f8c78159ad0 2024-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.602+0000 7f8c767fc640 1 -- 172.21.15.59:0/1056825902 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f8c68047070 con 0x7f8c78159ad0 2024-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.603+0000 7f8c767fc640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c64076540 0x7f8c64078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.603+0000 7f8c767fc640 1 -- 172.21.15.59:0/1056825902 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f8c680b8c10 con 0x7f8c78159ad0 2024-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.603+0000 7f8c7df59640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c64076540 0x7f8c64078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.604+0000 7f8c7df59640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c64076540 0x7f8c64078a50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c70001720 tx=0x7f8c70009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:26.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.607+0000 7f8c767fc640 1 -- 172.21.15.59:0/1056825902 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c68032030 con 0x7f8c78159ad0 2024-06-29T15:58:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.836+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c780b4b10 con 0x7f8c78159ad0 2024-06-29T15:58:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.836+0000 7f8c767fc640 1 -- 172.21.15.59:0/1056825902 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+5243 (secure 0 0 0) 0x7f8c68081980 con 0x7f8c78159ad0 2024-06-29T15:58:26.837 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":132,"btime":"2024-06-29T15:58:24:565730+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126},{"gid":12303,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:20.935697+0000","modified":"2024-06-29T15:58:23.956772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/177893499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":177893499},{"type":"v1","addr":"172.21.15.153:6837","nonce":177893499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12318,"qdb_cluster":[12318]},"id":15}]} 2024-06-29T15:58:26.837 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 132 2024-06-29T15:58:26.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.838+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c64076540 msgr2=0x7f8c64078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.838+0000 7f8c801bd640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c64076540 0x7f8c64078a50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c70001720 tx=0x7f8c70009040 comp rx=0 tx=0).stop 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.838+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c78159ad0 msgr2=0x7f8c781418e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.838+0000 7f8c801bd640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c78159ad0 0x7f8c781418e0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009d90 tx=0x7f8c68003d00 comp rx=0 tx=0).stop 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.839+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 shutdown_connections 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.839+0000 7f8c801bd640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f8c64076540 0x7f8c64078a50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.839+0000 7f8c801bd640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c78159ad0 0x7f8c781418e0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.839+0000 7f8c801bd640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8c78162770 0x7f8c780c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.839+0000 7f8c801bd640 1 --2- 172.21.15.59:0/1056825902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c7815e0a0 0x7f8c7815af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.839+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 >> 172.21.15.59:0/1056825902 conn(0x7f8c780d45b0 msgr2=0x7f8c780684a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.839+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 shutdown_connections 2024-06-29T15:58:26.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:26.839+0000 7f8c801bd640 1 -- 172.21.15.59:0/1056825902 wait complete. 2024-06-29T15:58:27.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 132, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:58:20.935697+0000', 'modified': '2024-06-29T15:58:23.956772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12318': {'gid': 12318, 'name': 'd', 'rank': 0, 'incarnation': 128, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6837/177893499', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6834', 'nonce': 177893499}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 177893499}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12318, 'qdb_cluster': [12318]} 2024-06-29T15:58:27.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:58:27.253 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-06-29T15:58:27.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.375+0000 7fbef40c1640 1 -- 172.21.15.59:0/1354593167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed8003ac0 msgr2=0x7fbed8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.375+0000 7fbef40c1640 1 --2- 172.21.15.59:0/1354593167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed8003ac0 0x7fbed8003f30 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c260 tx=0x7fbed8005830 comp rx=0 tx=0).stop 2024-06-29T15:58:27.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.375+0000 7fbef40c1640 1 -- 172.21.15.59:0/1354593167 shutdown_connections 2024-06-29T15:58:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.375+0000 7fbef40c1640 1 --2- 172.21.15.59:0/1354593167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed8003ac0 0x7fbed8003f30 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.375+0000 7fbef40c1640 1 -- 172.21.15.59:0/1354593167 >> 172.21.15.59:0/1354593167 conn(0x7fbeec0d3ee0 msgr2=0x7fbeec0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.375+0000 7fbef40c1640 1 -- 172.21.15.59:0/1354593167 shutdown_connections 2024-06-29T15:58:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.375+0000 7fbef40c1640 1 -- 172.21.15.59:0/1354593167 wait complete. 2024-06-29T15:58:27.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.377+0000 7fbef40c1640 1 Processor -- start 2024-06-29T15:58:27.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.377+0000 7fbef40c1640 1 -- start start 2024-06-29T15:58:27.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.378+0000 7fbef40c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed8003ac0 0x7fbeec152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.378+0000 7fbef40c1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeec1f5710 0x7fbeec1f2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.378+0000 7fbef40c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec150270 0x7fbeec0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.378+0000 7fbef40c1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec15ec60 con 0x7fbeec150270 2024-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.378+0000 7fbef40c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec15eae0 con 0x7fbed8003ac0 2024-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.378+0000 7fbef40c1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbeec0b3990 con 0x7fbeec1f5710 2024-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec150270 0x7fbeec0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec150270 0x7fbeec0cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35952/0 (socket says 172.21.15.59:35952) 2024-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 -- 172.21.15.59:0/3587630268 learned_addr learned my addr 172.21.15.59:0/3587630268 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef165c640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeec1f5710 0x7fbeec1f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed8003ac0 0x7fbeec152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 -- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeec1f5710 msgr2=0x7fbeec1f2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeec1f5710 0x7fbeec1f2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 -- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed8003ac0 msgr2=0x7fbeec152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed8003ac0 0x7fbeec152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 -- 172.21.15.59:0/3587630268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeec152d70 con 0x7fbeec150270 2024-06-29T15:58:27.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef265e640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec150270 0x7fbeec0cc160 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fbeec0b5540 tx=0x7fbee40019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbee27fc640 1 -- 172.21.15.59:0/3587630268 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee4011870 con 0x7fbeec150270 2024-06-29T15:58:27.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbee27fc640 1 -- 172.21.15.59:0/3587630268 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbee4017070 con 0x7fbeec150270 2024-06-29T15:58:27.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef40c1640 1 -- 172.21.15.59:0/3587630268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeec0ca4f0 con 0x7fbeec150270 2024-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbee27fc640 1 -- 172.21.15.59:0/3587630268 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee4011dc0 con 0x7fbeec150270 2024-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.379+0000 7fbef40c1640 1 -- 172.21.15.59:0/3587630268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeec151d50 con 0x7fbeec150270 2024-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.381+0000 7fbee27fc640 1 -- 172.21.15.59:0/3587630268 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fbee401c460 con 0x7fbeec150270 2024-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.382+0000 7fbee27fc640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbed0076510 0x7fbed0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.382+0000 7fbee27fc640 1 -- 172.21.15.59:0/3587630268 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fbee40987a0 con 0x7fbeec150270 2024-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.382+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbed0076510 0x7fbed0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.382+0000 7fbefaf80740 1 Processor -- start 2024-06-29T15:58:27.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.383+0000 7fbefaf80740 1 -- start start 2024-06-29T15:58:27.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.383+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbed0076510 0x7fbed0078a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbee8033000 tx=0x7fbee802d000 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c84d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c8313a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 0x55b40c8318e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbefaf80740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55b40c7c9bc0 con 0x55b40c4383b0 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbefaf80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b40c778630 con 0x55b40c461270 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbefaf80740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55b40c767bd0 con 0x55b40c440590 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef165c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c8313a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef165c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c8313a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35956/0 (socket says 172.21.15.59:35956) 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef165c640 1 -- 172.21.15.59:0/1347352101 learned_addr learned my addr 172.21.15.59:0/1347352101 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef265e640 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 0x55b40c8318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c84d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef165c640 1 -- 172.21.15.59:0/1347352101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 msgr2=0x55b40c8318e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef165c640 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 0x55b40c8318e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef165c640 1 -- 172.21.15.59:0/1347352101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 msgr2=0x55b40c84d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef165c640 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c84d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.384+0000 7fbef165c640 1 -- 172.21.15.59:0/1347352101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b40c764ca0 con 0x55b40c4383b0 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.385+0000 7fbef165c640 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c8313a0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fbedc003720 tx=0x7fbedc003be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e395f17c83ee3f12 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.385+0000 7fbecf7fe640 1 -- 172.21.15.59:0/1347352101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc00bb90 con 0x55b40c4383b0 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.385+0000 7fbecf7fe640 1 -- 172.21.15.59:0/1347352101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbedc00bd30 con 0x55b40c4383b0 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.385+0000 7fbecf7fe640 1 -- 172.21.15.59:0/1347352101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc005220 con 0x55b40c4383b0 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 -- 172.21.15.59:0/1347352101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 msgr2=0x55b40c8313a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c8313a0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fbedc003720 tx=0x7fbedc003be0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 -- 172.21.15.59:0/1347352101 shutdown_connections 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 0x55b40c8318e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c8313a0 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1347352101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c84d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 -- 172.21.15.59:0/1347352101 >> 172.21.15.59:0/1347352101 conn(0x55b40c4553c0 msgr2=0x55b40c804f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 -- 172.21.15.59:0/1347352101 shutdown_connections 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.386+0000 7fbefaf80740 1 -- 172.21.15.59:0/1347352101 wait complete. 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.387+0000 7fbefaf80740 1 -- start start 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.387+0000 7fbefaf80740 1 Processor -- start 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c829800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 0x55b40c85b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c85b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbefaf80740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55b40c705ae0 con 0x55b40c4383b0 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbefaf80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b40c76bc10 con 0x55b40c461270 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbefaf80740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55b40c4db620 con 0x55b40c440590 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbefaf80740 10 client.0 _ll_register_callbacks cb 0x55b40c712480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbef1e5d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c829800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbef1e5d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c829800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35962/0 (socket says 172.21.15.59:35962) 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbef1e5d640 1 -- 172.21.15.59:0/1209872458 learned_addr learned my addr 172.21.15.59:0/1209872458 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbef265e640 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c85b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.388+0000 7fbef165c640 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 0x55b40c85b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbef1e5d640 1 -- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 msgr2=0x55b40c85b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 0x55b40c85b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbef1e5d640 1 -- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 msgr2=0x55b40c85b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c85b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbef1e5d640 1 -- 172.21.15.59:0/1209872458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b40c4cc850 con 0x55b40c4383b0 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbef265e640 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c85b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c829800 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fbee8006440 tx=0x7fbee80067a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7f9fb2e99f31e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbeccff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbeccff9640 1 -- 172.21.15.59:0/1209872458 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8031040 con 0x55b40c4383b0 2024-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbeccff9640 1 -- 172.21.15.59:0/1209872458 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fbee8003cf0 con 0x55b40c4383b0 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbefaf80740 1 -- 172.21.15.59:0/1209872458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbedc00bd30 con 0x55b40c4383b0 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbeccff9640 1 -- 172.21.15.59:0/1209872458 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7fbee8054070 con 0x55b40c4383b0 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbefaf80740 10 client.12447 fetch_fsmap learned FSMap version 132 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbefaf80740 1 -- 172.21.15.59:0/1209872458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=2+}) -- 0x55b40c5436b0 con 0x55b40c4383b0 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbeccff9640 1 -- 172.21.15.59:0/1209872458 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbedc00bd30 con 0x55b40c4383b0 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbeccff9640 1 -- 172.21.15.59:0/1209872458 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 132) ==== 3502+0+0 (secure 0 0 0) 0x7fbee803f070 con 0x55b40c4383b0 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbefaf80740 10 client.12447 fetch_fsmap finished waiting for FSMap version 132 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbefaf80740 10 client.12447 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbefaf80740 20 client.12447 populate_metadata read hostname 'smithi059' 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.389+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] conn(0x55b40c41fe40 0x55b40c85dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.390+0000 7fbefaf80740 4 client.12447 mds_command: new command op to 12318 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.390+0000 7fbefaf80740 1 -- 172.21.15.59:0/1209872458 --> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b40c77a580 con 0x55b40c41fe40 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.390+0000 7fbef165c640 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] conn(0x55b40c41fe40 0x55b40c85dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.390+0000 7fbef165c640 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] conn(0x55b40c41fe40 0x55b40c85dcd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=240d2b45a8f3bade server_cookie=e453e03cec018354 in_seq=0 out_seq=0 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.390+0000 7fbeccff9640 10 client.12447 ms_handle_connect on v2:172.21.15.153:6834/177893499 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.391+0000 7fbeccff9640 1 -- 172.21.15.59:0/1209872458 <== mds.0 v2:172.21.15.153:6834/177893499 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b40c77a580 con 0x55b40c41fe40 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.391+0000 7fbeccff9640 10 client.12447 handle_command_reply: tid=0 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.391+0000 7fbefaf80740 1 client.12447 shutdown 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 -- 172.21.15.59:0/1209872458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 msgr2=0x55b40c829800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c829800 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fbee8006440 tx=0x7fbee80067a0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 -- 172.21.15.59:0/1209872458 shutdown_connections 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] conn(0x55b40c41fe40 0x55b40c85dcd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c461270 0x55b40c85b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c440590 0x55b40c85b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbeccff9640 0 client.12447 ms_handle_reset on v2:172.21.15.153:6834/177893499 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1209872458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c4383b0 0x55b40c829800 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 -- 172.21.15.59:0/1209872458 >> 172.21.15.59:0/1209872458 conn(0x55b40c4553c0 msgr2=0x55b40c82dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 -- 172.21.15.59:0/1209872458 shutdown_connections 2024-06-29T15:58:27.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 1 -- 172.21.15.59:0/1209872458 wait complete. 2024-06-29T15:58:27.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.392+0000 7fbefaf80740 20 client.12447 trim_cache size 0 max 16384 2024-06-29T15:58:27.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.411+0000 7fbefaf80740 1 Processor -- start 2024-06-29T15:58:27.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.411+0000 7fbefaf80740 1 -- start start 2024-06-29T15:58:27.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 0x55b40c870950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c800b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c4383b0 0x55b40c801080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbefaf80740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec200cc0 con 0x55b40c41fe40 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbefaf80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1f8cb0 con 0x55b40c4553c0 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbefaf80740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbeec200650 con 0x55b40c4383b0 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c800b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c800b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35966/0 (socket says 172.21.15.59:35966) 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 -- 172.21.15.59:0/1593558195 learned_addr learned my addr 172.21.15.59:0/1593558195 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 0x55b40c870950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 -- 172.21.15.59:0/1593558195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c4383b0 msgr2=0x55b40c801080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c4383b0 0x55b40c801080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 -- 172.21.15.59:0/1593558195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 msgr2=0x55b40c870950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 0x55b40c870950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 -- 172.21.15.59:0/1593558195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b40c809170 con 0x55b40c41fe40 2024-06-29T15:58:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 0x55b40c870950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbef165c640 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c800b40 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fbedc003720 tx=0x7fbedc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62afa7c569a1fb6a server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbeceffd640 1 -- 172.21.15.59:0/1593558195 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc005850 con 0x55b40c41fe40 2024-06-29T15:58:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.413+0000 7fbeceffd640 1 -- 172.21.15.59:0/1593558195 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbedc005020 con 0x55b40c41fe40 2024-06-29T15:58:27.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbeceffd640 1 -- 172.21.15.59:0/1593558195 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc005450 con 0x55b40c41fe40 2024-06-29T15:58:27.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 -- 172.21.15.59:0/1593558195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 msgr2=0x55b40c800b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c800b40 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fbedc003720 tx=0x7fbedc004730 comp rx=0 tx=0).stop 2024-06-29T15:58:27.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 -- 172.21.15.59:0/1593558195 shutdown_connections 2024-06-29T15:58:27.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c4383b0 0x55b40c801080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c800b40 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1593558195 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 0x55b40c870950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 -- 172.21.15.59:0/1593558195 >> 172.21.15.59:0/1593558195 conn(0x55b40c842ec0 msgr2=0x55b40c84c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 -- 172.21.15.59:0/1593558195 shutdown_connections 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.414+0000 7fbefaf80740 1 -- 172.21.15.59:0/1593558195 wait complete. 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.415+0000 7fbefaf80740 1 -- start start 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.415+0000 7fbefaf80740 1 Processor -- start 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c8663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c4383b0 0x55b40c800b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbefaf80740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 0x55b40c801080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbefaf80740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55b40c831170 con 0x55b40c41fe40 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbefaf80740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b40c82fa70 con 0x55b40c4553c0 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbefaf80740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55b40c84af90 con 0x55b40c4383b0 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbefaf80740 10 client.0 _ll_register_callbacks cb 0x7fbed80036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c8663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c8663c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35980/0 (socket says 172.21.15.59:35980) 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 -- 172.21.15.59:0/1843185099 learned_addr learned my addr 172.21.15.59:0/1843185099 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 -- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c4383b0 msgr2=0x55b40c800b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c4383b0 0x55b40c800b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 -- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 msgr2=0x55b40c801080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 0x55b40c801080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 -- 172.21.15.59:0/1843185099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b40c870680 con 0x55b40c41fe40 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbef1e5d640 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c8663c0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7fbee802b020 tx=0x7fbee8003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e9a51913083679 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbeccff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbeccff9640 1 -- 172.21.15.59:0/1843185099 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8031040 con 0x55b40c41fe40 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbefaf80740 1 -- 172.21.15.59:0/1843185099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbedc005020 con 0x55b40c41fe40 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.417+0000 7fbeccff9640 1 -- 172.21.15.59:0/1843185099 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fbee8037e00 con 0x55b40c41fe40 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbeccff9640 1 -- 172.21.15.59:0/1843185099 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7fbee804e870 con 0x55b40c41fe40 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbefaf80740 10 client.12453 fetch_fsmap learned FSMap version 132 2024-06-29T15:58:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbefaf80740 1 -- 172.21.15.59:0/1843185099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=2+}) -- 0x55b40c865ea0 con 0x55b40c41fe40 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbeccff9640 1 -- 172.21.15.59:0/1843185099 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbedc005020 con 0x55b40c41fe40 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbeccff9640 1 -- 172.21.15.59:0/1843185099 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 132) ==== 3502+0+0 (secure 0 0 0) 0x7fbee803f070 con 0x55b40c41fe40 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbefaf80740 10 client.12453 fetch_fsmap finished waiting for FSMap version 132 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbefaf80740 10 client.12453 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbefaf80740 20 client.12453 populate_metadata read hostname 'smithi059' 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] conn(0x55b40c461270 0x55b40c8c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbefaf80740 4 client.12453 mds_command: new command op to 12318 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbefaf80740 1 -- 172.21.15.59:0/1843185099 --> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] -- command(tid 0: {"prefix": "status"}) -- 0x55b40c776020 con 0x55b40c461270 2024-06-29T15:58:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.418+0000 7fbef165c640 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] conn(0x55b40c461270 0x55b40c8c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbeccff9640 10 client.12453 ms_handle_connect on v2:172.21.15.153:6834/177893499 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbef165c640 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] conn(0x55b40c461270 0x55b40c8c92b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde5f5a3112f8256 server_cookie=e179fc5c1969c003 in_seq=0 out_seq=0 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbeccff9640 1 -- 172.21.15.59:0/1843185099 <== mds.0 v2:172.21.15.153:6834/177893499 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x55b40c776020 con 0x55b40c461270 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbeccff9640 10 client.12453 handle_command_reply: tid=0 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 client.12453 shutdown 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 -- 172.21.15.59:0/1843185099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 msgr2=0x55b40c8663c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c8663c0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7fbee802b020 tx=0x7fbee8003bf0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 -- 172.21.15.59:0/1843185099 shutdown_connections 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:6834/177893499,v1:172.21.15.153:6837/177893499] conn(0x55b40c461270 0x55b40c8c92b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b40c4553c0 0x55b40c801080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55b40c4383b0 0x55b40c800b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbeccff9640 0 client.12453 ms_handle_reset on v2:172.21.15.153:6834/177893499 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 --2- 172.21.15.59:0/1843185099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b40c41fe40 0x55b40c8663c0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 -- 172.21.15.59:0/1843185099 >> 172.21.15.59:0/1843185099 conn(0x55b40c842ec0 msgr2=0x55b40c84d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 -- 172.21.15.59:0/1843185099 shutdown_connections 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 1 -- 172.21.15.59:0/1843185099 wait complete. 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.419+0000 7fbefaf80740 20 client.12453 trim_cache size 0 max 16384 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 12318, 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T15:58:27.424 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8640848280000002, 2024-06-29T15:58:27.425 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 132, 2024-06-29T15:58:27.425 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 242, 2024-06-29T15:58:27.425 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 242, 2024-06-29T15:58:27.425 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.482426565000001 2024-06-29T15:58:27.425 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 -- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbed0076510 msgr2=0x7fbed0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbed0076510 0x7fbed0078a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbee8033000 tx=0x7fbee802d000 comp rx=0 tx=0).stop 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 -- 172.21.15.59:0/3587630268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec150270 msgr2=0x7fbeec0cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec150270 0x7fbeec0cc160 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fbeec0b5540 tx=0x7fbee40019b0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 -- 172.21.15.59:0/3587630268 shutdown_connections 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbed0076510 0x7fbed0078a20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec150270 0x7fbeec0cc160 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbeec1f5710 0x7fbeec1f2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 --2- 172.21.15.59:0/3587630268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed8003ac0 0x7fbeec152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 -- 172.21.15.59:0/3587630268 >> 172.21.15.59:0/3587630268 conn(0x7fbeec0d3ee0 msgr2=0x7fbeec160280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 -- 172.21.15.59:0/3587630268 shutdown_connections 2024-06-29T15:58:27.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.420+0000 7fbef40c1640 1 -- 172.21.15.59:0/3587630268 wait complete. 2024-06-29T15:58:27.432 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:58:27.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.550+0000 7fe8edbeb640 1 -- 172.21.15.59:0/3466115810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80cccc0 msgr2=0x7fe8e8073430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.550+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/3466115810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80cccc0 0x7fe8e8073430 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8068d50 tx=0x7fe8dc02a530 comp rx=0 tx=0).stop 2024-06-29T15:58:27.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.550+0000 7fe8edbeb640 1 -- 172.21.15.59:0/3466115810 shutdown_connections 2024-06-29T15:58:27.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.550+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/3466115810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80cccc0 0x7fe8e8073430 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.550+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/3466115810 >> v2:172.21.15.153:3300/0 conn(0x7fe8e81599a0 0x7fe8e80cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.550+0000 7fe8edbeb640 1 -- 172.21.15.59:0/3466115810 >> 172.21.15.59:0/3466115810 conn(0x7fe8e806cef0 msgr2=0x7fe8e806d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.550+0000 7fe8edbeb640 1 -- 172.21.15.59:0/3466115810 shutdown_connections 2024-06-29T15:58:27.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.550+0000 7fe8edbeb640 1 -- 172.21.15.59:0/3466115810 wait complete. 2024-06-29T15:58:27.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.551+0000 7fe8edbeb640 1 Processor -- start 2024-06-29T15:58:27.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.552+0000 7fe8edbeb640 1 -- start start 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8edbeb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8e8073970 0x7fe8e80c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8edbeb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8e80cccc0 0x7fe8e8146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8edbeb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e81599a0 0x7fe8e815a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8edbeb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e815c820 con 0x7fe8e81599a0 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8edbeb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e815c6a0 con 0x7fe8e8073970 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8edbeb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e8160170 con 0x7fe8e80cccc0 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e81599a0 0x7fe8e815a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e81599a0 0x7fe8e815a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36000/0 (socket says 172.21.15.59:36000) 2024-06-29T15:58:27.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e7fff640 1 -- 172.21.15.59:0/1102248119 learned_addr learned my addr 172.21.15.59:0/1102248119 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e7fff640 1 -- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8e80cccc0 msgr2=0x7fe8e8146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e77fe640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8e8073970 0x7fe8e80c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e7fff640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8e80cccc0 0x7fe8e8146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e7fff640 1 -- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8e8073970 msgr2=0x7fe8e80c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e7fff640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8e8073970 0x7fe8e80c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.553+0000 7fe8e7fff640 1 -- 172.21.15.59:0/1102248119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e8156cc0 con 0x7fe8e81599a0 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.554+0000 7fe8e7fff640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e81599a0 0x7fe8e815a000 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000ca80 tx=0x7fe8e000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.554+0000 7fe8cffff640 1 -- 172.21.15.59:0/1102248119 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0016020 con 0x7fe8e81599a0 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.554+0000 7fe8cffff640 1 -- 172.21.15.59:0/1102248119 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8e000ad20 con 0x7fe8e81599a0 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.554+0000 7fe8cffff640 1 -- 172.21.15.59:0/1102248119 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0012680 con 0x7fe8e81599a0 2024-06-29T15:58:27.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.554+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e814a850 con 0x7fe8e81599a0 2024-06-29T15:58:27.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.554+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e80ab830 con 0x7fe8e81599a0 2024-06-29T15:58:27.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.555+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e8160640 con 0x7fe8e81599a0 2024-06-29T15:58:27.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.556+0000 7fe8cffff640 1 -- 172.21.15.59:0/1102248119 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fe8e0019020 con 0x7fe8e81599a0 2024-06-29T15:58:27.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.556+0000 7fe8cffff640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fe8d4076540 0x7fe8d4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.556+0000 7fe8cffff640 1 -- 172.21.15.59:0/1102248119 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fe8e00983c0 con 0x7fe8e81599a0 2024-06-29T15:58:27.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.556+0000 7fe8e77fe640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fe8d4076540 0x7fe8d4078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.557+0000 7fe8e77fe640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fe8d4076540 0x7fe8d4078a50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe8d80058c0 tx=0x7fe8d8005850 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.561+0000 7fe8cffff640 1 -- 172.21.15.59:0/1102248119 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe8e0020020 con 0x7fe8e81599a0 2024-06-29T15:58:27.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.794+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe8e80b0610 con 0x7fe8e81599a0 2024-06-29T15:58:27.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.795+0000 7fe8cffff640 1 -- 172.21.15.59:0/1102248119 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v132) ==== 53+0+73 (secure 0 0 0) 0x7fe8e8156cc0 con 0x7fe8e81599a0 2024-06-29T15:58:27.795 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:58:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fe8d4076540 msgr2=0x7fe8d4078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fe8d4076540 0x7fe8d4078a50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe8d80058c0 tx=0x7fe8d8005850 comp rx=0 tx=0).stop 2024-06-29T15:58:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e81599a0 msgr2=0x7fe8e815a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e81599a0 0x7fe8e815a000 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000ca80 tx=0x7fe8e000cf40 comp rx=0 tx=0).stop 2024-06-29T15:58:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 shutdown_connections 2024-06-29T15:58:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fe8d4076540 0x7fe8d4078a50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e81599a0 0x7fe8e815a000 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8e80cccc0 0x7fe8e8146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 --2- 172.21.15.59:0/1102248119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8e8073970 0x7fe8e80c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 >> 172.21.15.59:0/1102248119 conn(0x7fe8e806cef0 msgr2=0x7fe8e80b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 shutdown_connections 2024-06-29T15:58:27.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:27.797+0000 7fe8edbeb640 1 -- 172.21.15.59:0/1102248119 wait complete. 2024-06-29T15:58:27.808 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:58:27.808 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:58:27.808 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T15:58:27.808 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:58:27.809 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:27.809 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:58:27.809 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:58:27.809 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:58:27.809 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:58:27.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:27.809 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T15:58:27.826 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:58:27.826 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:58:27.826 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:58:27.826 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T15:58:27.827 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 36943sec preferred_lft 36943sec 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:58:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:58:27.829 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T15:58:27.829 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:27.829 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:58:27.829 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:58:27.829 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T15:58:27.829 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:27.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:27.829 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:58:27.891 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T15:58:27.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:27.892 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T15:58:27.951 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T15:58:27.951 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:58:27.951 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T15:58:27.951 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T15:58:27.951 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:58:27.952 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:58:27.952 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T15:58:27.952 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T15:58:27.952 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T15:58:27.952 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T15:58:27.952 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T15:58:28.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:58:28.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:58:28.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:28.082 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T15:58:28.098 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:58:28.098 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:58:28.098 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:58:28.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:28.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T15:58:28.165 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:58:28.165 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:28.222 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:58:28.222 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:58:28.222 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:28.278 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T15:58:28.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:28.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:58:28.374 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:58:28.374 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:58:28.374 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:58:28.385 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:58:28.385 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:58:28.385 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:58:28.385 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:58:28.385 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T15:58:28.393 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:58:28.393 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:58:28.393 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:58:28.393 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:58:28.393 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:58:28.394 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:58:28.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:28.394 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:28.420 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:58:28.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:58:28.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:58:28.526 INFO:teuthology.orchestra.run.smithi059.stdout:client12465 2024-06-29T15:58:28.526 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:58:28.527 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:28.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-06-29T15:58:28.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.707+0000 7f5025d68640 1 -- 172.21.15.59:0/2508586723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f500c004d40 msgr2=0x7f500c0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.707+0000 7f5025d68640 1 --2- 172.21.15.59:0/2508586723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f500c004d40 0x7f500c0051f0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f5018001dc0 tx=0x7f500c005830 comp rx=0 tx=0).stop 2024-06-29T15:58:28.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.707+0000 7f501ffff640 1 -- 172.21.15.59:0/2508586723 reap_dead start 2024-06-29T15:58:28.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.707+0000 7f5025d68640 1 -- 172.21.15.59:0/2508586723 shutdown_connections 2024-06-29T15:58:28.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.707+0000 7f5025d68640 1 -- 172.21.15.59:0/2508586723 >> 172.21.15.59:0/2508586723 conn(0x7f50200b7690 msgr2=0x7f5020073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:28.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.707+0000 7f5025d68640 1 -- 172.21.15.59:0/2508586723 shutdown_connections 2024-06-29T15:58:28.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.707+0000 7f5025d68640 1 -- 172.21.15.59:0/2508586723 wait complete. 2024-06-29T15:58:28.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.709+0000 7f5025d68640 1 Processor -- start 2024-06-29T15:58:28.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.709+0000 7f5025d68640 1 -- start start 2024-06-29T15:58:28.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5025d68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f500c004d40 0x7f5020156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5025d68640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5020154260 0x7f5020154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5025d68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50201582d0 0x7f50201586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5025d68640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f50200b51d0 con 0x7f5020154260 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5025d68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50200b5050 con 0x7f50201582d0 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5025d68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f50200b5350 con 0x7f500c004d40 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5016ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5020154260 0x7f5020154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5016ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5020154260 0x7f5020154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36014/0 (socket says 172.21.15.59:36014) 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f5016ffd640 1 -- 172.21.15.59:0/2482538593 learned_addr learned my addr 172.21.15.59:0/2482538593 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f501ffff640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50201582d0 0x7f50201586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.710+0000 7f501f7fe640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f500c004d40 0x7f5020156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f5016ffd640 1 -- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f500c004d40 msgr2=0x7f5020156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f5016ffd640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f500c004d40 0x7f5020156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f5016ffd640 1 -- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50201582d0 msgr2=0x7f50201586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f5016ffd640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50201582d0 0x7f50201586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f5016ffd640 1 -- 172.21.15.59:0/2482538593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5020158c20 con 0x7f5020154260 2024-06-29T15:58:28.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f501f7fe640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f500c004d40 0x7f5020156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:28.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f5016ffd640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5020154260 0x7f5020154670 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f50080066b0 tx=0x7f5008003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:28.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f501cff9640 1 -- 172.21.15.59:0/2482538593 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500800f800 con 0x7f5020154260 2024-06-29T15:58:28.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f501cff9640 1 -- 172.21.15.59:0/2482538593 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5008016020 con 0x7f5020154260 2024-06-29T15:58:28.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5020156af0 con 0x7f5020154260 2024-06-29T15:58:28.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f501cff9640 1 -- 172.21.15.59:0/2482538593 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500800fc60 con 0x7f5020154260 2024-06-29T15:58:28.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.711+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50201570b0 con 0x7f5020154260 2024-06-29T15:58:28.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.712+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50200b58e0 con 0x7f5020154260 2024-06-29T15:58:28.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.713+0000 7f501cff9640 1 -- 172.21.15.59:0/2482538593 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f5008011430 con 0x7f5020154260 2024-06-29T15:58:28.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.713+0000 7f501cff9640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5000076510 0x7f5000078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.713+0000 7f501cff9640 1 -- 172.21.15.59:0/2482538593 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f5008098970 con 0x7f5020154260 2024-06-29T15:58:28.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.714+0000 7f501f7fe640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5000076510 0x7f5000078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.714+0000 7f501f7fe640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5000076510 0x7f5000078a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5010009c50 tx=0x7f5010005f90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:28.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.718+0000 7f501cff9640 1 -- 172.21.15.59:0/2482538593 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5008020020 con 0x7f5020154260 2024-06-29T15:58:28.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:28.954+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f5020156880 con 0x7f5020154260 2024-06-29T15:58:29.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.589+0000 7f501cff9640 1 -- 172.21.15.59:0/2482538593 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v133) ==== 136+0+0 (secure 0 0 0) 0x7f5020158c20 con 0x7f5020154260 2024-06-29T15:58:29.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5000076510 msgr2=0x7f5000078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5000076510 0x7f5000078a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5010009c50 tx=0x7f5010005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:29.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5020154260 msgr2=0x7f5020154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5020154260 0x7f5020154670 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f50080066b0 tx=0x7f5008003a80 comp rx=0 tx=0).stop 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 shutdown_connections 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5000076510 0x7f5000078a20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50201582d0 0x7f50201586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5020154260 0x7f5020154670 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 --2- 172.21.15.59:0/2482538593 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f500c004d40 0x7f5020156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 >> 172.21.15.59:0/2482538593 conn(0x7f50200b7690 msgr2=0x7f50200b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 shutdown_connections 2024-06-29T15:58:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.593+0000 7f5025d68640 1 -- 172.21.15.59:0/2482538593 wait complete. 2024-06-29T15:58:29.607 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.728+0000 7fd2d2247640 1 -- 172.21.15.59:0/1216718181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc144470 msgr2=0x7fd2cc0b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.728+0000 7fd2d2247640 1 --2- 172.21.15.59:0/1216718181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc144470 0x7fd2cc0b1b00 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc072450 tx=0x7fd2bc02a530 comp rx=0 tx=0).stop 2024-06-29T15:58:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.728+0000 7fd2d2247640 1 -- 172.21.15.59:0/1216718181 shutdown_connections 2024-06-29T15:58:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.728+0000 7fd2d2247640 1 --2- 172.21.15.59:0/1216718181 >> v2:172.21.15.153:3300/0 conn(0x7fd2cc0b2040 0x7fd2cc06e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.728+0000 7fd2d2247640 1 --2- 172.21.15.59:0/1216718181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc144470 0x7fd2cc0b1b00 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.728+0000 7fd2d2247640 1 -- 172.21.15.59:0/1216718181 >> 172.21.15.59:0/1216718181 conn(0x7fd2cc0b06e0 msgr2=0x7fd2cc0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.728+0000 7fd2d2247640 1 -- 172.21.15.59:0/1216718181 shutdown_connections 2024-06-29T15:58:29.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.728+0000 7fd2d2247640 1 -- 172.21.15.59:0/1216718181 wait complete. 2024-06-29T15:58:29.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.730+0000 7fd2d2247640 1 Processor -- start 2024-06-29T15:58:29.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.730+0000 7fd2d2247640 1 -- start start 2024-06-29T15:58:29.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.731+0000 7fd2d2247640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2cc06e840 0x7fd2cc13da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.731+0000 7fd2d2247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc0b2040 0x7fd2cc13c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.731+0000 7fd2d2247640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2cc144470 0x7fd2cc0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.731+0000 7fd2d2247640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc0d0360 con 0x7fd2cc0b2040 2024-06-29T15:58:29.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.731+0000 7fd2d2247640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc0d01e0 con 0x7fd2cc06e840 2024-06-29T15:58:29.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.731+0000 7fd2d2247640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd2cc15ee80 con 0x7fd2cc144470 2024-06-29T15:58:29.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc0b2040 0x7fd2cc13c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc0b2040 0x7fd2cc13c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36042/0 (socket says 172.21.15.59:36042) 2024-06-29T15:58:29.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 -- 172.21.15.59:0/2166856358 learned_addr learned my addr 172.21.15.59:0/2166856358 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2cb7fe640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2cc06e840 0x7fd2cc13da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2cbfff640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2cc144470 0x7fd2cc0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 -- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2cc144470 msgr2=0x7fd2cc0b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2cc144470 0x7fd2cc0b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 -- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2cc06e840 msgr2=0x7fd2cc13da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2cc06e840 0x7fd2cc13da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 -- 172.21.15.59:0/2166856358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc1f2890 con 0x7fd2cc0b2040 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2cbfff640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2cc144470 0x7fd2cc0b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2cb7fe640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2cc06e840 0x7fd2cc13da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2caffd640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc0b2040 0x7fd2cc13c0d0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000c940 tx=0x7fd2c000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2b3fff640 1 -- 172.21.15.59:0/2166856358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0004230 con 0x7fd2cc0b2040 2024-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2b3fff640 1 -- 172.21.15.59:0/2166856358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2c00079b0 con 0x7fd2cc0b2040 2024-06-29T15:58:29.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2cc13e340 con 0x7fd2cc0b2040 2024-06-29T15:58:29.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2b3fff640 1 -- 172.21.15.59:0/2166856358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c00044c0 con 0x7fd2cc0b2040 2024-06-29T15:58:29.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.732+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2cc13e900 con 0x7fd2cc0b2040 2024-06-29T15:58:29.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.734+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2cc13e0b0 con 0x7fd2cc0b2040 2024-06-29T15:58:29.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.734+0000 7fd2b3fff640 1 -- 172.21.15.59:0/2166856358 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fd2c00035f0 con 0x7fd2cc0b2040 2024-06-29T15:58:29.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.734+0000 7fd2b3fff640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd2b8076510 0x7fd2b8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.735+0000 7fd2b3fff640 1 -- 172.21.15.59:0/2166856358 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fd2c0098260 con 0x7fd2cc0b2040 2024-06-29T15:58:29.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.735+0000 7fd2cb7fe640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd2b8076510 0x7fd2b8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.735+0000 7fd2cb7fe640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd2b8076510 0x7fd2b8078a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc034000 tx=0x7fd2bc004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:29.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.739+0000 7fd2b3fff640 1 -- 172.21.15.59:0/2166856358 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd2c0014070 con 0x7fd2cc0b2040 2024-06-29T15:58:29.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.932+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2cc15f340 con 0x7fd2cc0b2040 2024-06-29T15:58:29.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.933+0000 7fd2b3fff640 1 -- 172.21.15.59:0/2166856358 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) ==== 74+0+17054 (secure 0 0 0) 0x7fd2c0061580 con 0x7fd2cc0b2040 2024-06-29T15:58:29.935 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:29.935 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":242,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:22.538542+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:29.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd2b8076510 msgr2=0x7fd2b8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd2b8076510 0x7fd2b8078a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc034000 tx=0x7fd2bc004730 comp rx=0 tx=0).stop 2024-06-29T15:58:29.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc0b2040 msgr2=0x7fd2cc13c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc0b2040 0x7fd2cc13c0d0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000c940 tx=0x7fd2c000ce00 comp rx=0 tx=0).stop 2024-06-29T15:58:29.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 shutdown_connections 2024-06-29T15:58:29.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fd2b8076510 0x7fd2b8078a20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2cc144470 0x7fd2cc0b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2cc0b2040 0x7fd2cc13c0d0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 --2- 172.21.15.59:0/2166856358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2cc06e840 0x7fd2cc13da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 >> 172.21.15.59:0/2166856358 conn(0x7fd2cc0b06e0 msgr2=0x7fd2cc141360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:29.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 shutdown_connections 2024-06-29T15:58:29.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:29.936+0000 7fd2d2247640 1 -- 172.21.15.59:0/2166856358 wait complete. 2024-06-29T15:58:29.948 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:58:19.303469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '241', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:58:19.884648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:29.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:30.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.068+0000 7f98ac827640 1 -- 172.21.15.59:0/378983523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80ce950 msgr2=0x7f98a80d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.068+0000 7f98ac827640 1 --2- 172.21.15.59:0/378983523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80ce950 0x7f98a80d2130 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f98a0005fd0 tx=0x7f98a002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.068+0000 7f98ac827640 1 -- 172.21.15.59:0/378983523 shutdown_connections 2024-06-29T15:58:30.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.068+0000 7f98ac827640 1 --2- 172.21.15.59:0/378983523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80ce950 0x7f98a80d2130 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.068+0000 7f98ac827640 1 --2- 172.21.15.59:0/378983523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98a808c190 0x7f98a80cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.068+0000 7f98ac827640 1 -- 172.21.15.59:0/378983523 >> 172.21.15.59:0/378983523 conn(0x7f98a80b5610 msgr2=0x7f98a80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:30.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.068+0000 7f98ac827640 1 -- 172.21.15.59:0/378983523 shutdown_connections 2024-06-29T15:58:30.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.068+0000 7f98ac827640 1 -- 172.21.15.59:0/378983523 wait complete. 2024-06-29T15:58:30.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.070+0000 7f98ac827640 1 Processor -- start 2024-06-29T15:58:30.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.070+0000 7f98ac827640 1 -- start start 2024-06-29T15:58:30.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98ac827640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a808c190 0x7f98a80cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98ac827640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98a80ce950 0x7f98a814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98ac827640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98a80cf5c0 0x7f98a80ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98ac827640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f98a80b7fa0 con 0x7f98a808c190 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98ac827640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98a80b7e20 con 0x7f98a80cf5c0 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98ac827640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f98a80b8120 con 0x7f98a80ce950 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a808c190 0x7f98a80cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a808c190 0x7f98a80cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36052/0 (socket says 172.21.15.59:36052) 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 -- 172.21.15.59:0/911712628 learned_addr learned my addr 172.21.15.59:0/911712628 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 -- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98a80ce950 msgr2=0x7f98a814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98a80ce950 0x7f98a814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 -- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98a80cf5c0 msgr2=0x7f98a80ac4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a6d9d640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98a80cf5c0 0x7f98a80ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98a80cf5c0 0x7f98a80ac4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 -- 172.21.15.59:0/911712628 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a80cbec0 con 0x7f98a808c190 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98a659c640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a808c190 0x7f98a80cb5e0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f989800d740 tx=0x7f989800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f9896ffd640 1 -- 172.21.15.59:0/911712628 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9898017070 con 0x7f98a808c190 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f9896ffd640 1 -- 172.21.15.59:0/911712628 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9898009d20 con 0x7f98a808c190 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.072+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a80cc1a0 con 0x7f98a808c190 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.073+0000 7f9896ffd640 1 -- 172.21.15.59:0/911712628 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98980106c0 con 0x7f98a808c190 2024-06-29T15:58:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.073+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a80aa740 con 0x7f98a808c190 2024-06-29T15:58:30.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.073+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98a80cf080 con 0x7f98a808c190 2024-06-29T15:58:30.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.074+0000 7f9896ffd640 1 -- 172.21.15.59:0/911712628 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f989801a070 con 0x7f98a808c190 2024-06-29T15:58:30.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.075+0000 7f9896ffd640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9888076540 0x7f9888078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.075+0000 7f9896ffd640 1 -- 172.21.15.59:0/911712628 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f9898098520 con 0x7f98a808c190 2024-06-29T15:58:30.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.075+0000 7f98a5d9b640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9888076540 0x7f9888078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.076+0000 7f98a5d9b640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9888076540 0x7f9888078a50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f989c0046b0 tx=0x7f989c007a70 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:30.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.078+0000 7f9896ffd640 1 -- 172.21.15.59:0/911712628 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9898061720 con 0x7f98a808c190 2024-06-29T15:58:30.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.275+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f98a81e7190 con 0x7f98a808c190 2024-06-29T15:58:30.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.275+0000 7f9896ffd640 1 -- 172.21.15.59:0/911712628 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v242) ==== 93+0+10 (secure 0 0 0) 0x7f9898061020 con 0x7f98a808c190 2024-06-29T15:58:30.276 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:58:30.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9888076540 msgr2=0x7f9888078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9888076540 0x7f9888078a50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f989c0046b0 tx=0x7f989c007a70 comp rx=0 tx=0).stop 2024-06-29T15:58:30.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a808c190 msgr2=0x7f98a80cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a808c190 0x7f98a80cb5e0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f989800d740 tx=0x7f989800dc00 comp rx=0 tx=0).stop 2024-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 shutdown_connections 2024-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9888076540 0x7f9888078a50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98a80cf5c0 0x7f98a80ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98a80ce950 0x7f98a814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 --2- 172.21.15.59:0/911712628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a808c190 0x7f98a80cb5e0 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 >> 172.21.15.59:0/911712628 conn(0x7f98a80b5610 msgr2=0x7f98a80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 shutdown_connections 2024-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.277+0000 7f98ac827640 1 -- 172.21.15.59:0/911712628 wait complete. 2024-06-29T15:58:30.289 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:58:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.410+0000 7f6cb96ba640 1 -- 172.21.15.59:0/3288520903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 msgr2=0x7f6ca000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.410+0000 7f6cb96ba640 1 --2- 172.21.15.59:0/3288520903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 0x7f6ca000b9c0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f6cb40d3730 tx=0x7f6ca0005830 comp rx=0 tx=0).stop 2024-06-29T15:58:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.410+0000 7f6cb37fe640 1 -- 172.21.15.59:0/3288520903 reap_dead start 2024-06-29T15:58:30.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.410+0000 7f6cb96ba640 1 -- 172.21.15.59:0/3288520903 shutdown_connections 2024-06-29T15:58:30.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.410+0000 7f6cb96ba640 1 -- 172.21.15.59:0/3288520903 >> 172.21.15.59:0/3288520903 conn(0x7f6cb40d45b0 msgr2=0x7f6cb40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:30.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.410+0000 7f6cb96ba640 1 -- 172.21.15.59:0/3288520903 shutdown_connections 2024-06-29T15:58:30.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.410+0000 7f6cb96ba640 1 -- 172.21.15.59:0/3288520903 wait complete. 2024-06-29T15:58:30.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.412+0000 7f6cb96ba640 1 Processor -- start 2024-06-29T15:58:30.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.412+0000 7f6cb96ba640 1 -- start start 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.413+0000 7f6cb96ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 0x7f6cb40c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.413+0000 7f6cb96ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cb4158e40 0x7f6cb4159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.413+0000 7f6cb96ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cb4156eb0 0x7f6cb41572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.413+0000 7f6cb96ba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb40ce6d0 con 0x7f6ca00051c0 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.413+0000 7f6cb96ba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb40ce550 con 0x7f6cb4156eb0 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.413+0000 7f6cb96ba640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6cb40ce850 con 0x7f6cb4158e40 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 0x7f6cb40c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 0x7f6cb40c6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36074/0 (socket says 172.21.15.59:36074) 2024-06-29T15:58:30.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 -- 172.21.15.59:0/787549805 learned_addr learned my addr 172.21.15.59:0/787549805 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb37fe640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cb4156eb0 0x7f6cb41572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6caa7fc640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cb4158e40 0x7f6cb4159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 -- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cb4158e40 msgr2=0x7f6cb4159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cb4158e40 0x7f6cb4159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 -- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cb4156eb0 msgr2=0x7f6cb41572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cb4156eb0 0x7f6cb41572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 -- 172.21.15.59:0/787549805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb4157800 con 0x7f6ca00051c0 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb2ffd640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 0x7f6cb40c6850 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4004ed0 tx=0x7f6ca40015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cabfff640 1 -- 172.21.15.59:0/787549805 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4015020 con 0x7f6ca00051c0 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cabfff640 1 -- 172.21.15.59:0/787549805 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ca4011960 con 0x7f6ca00051c0 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb4159b70 con 0x7f6ca00051c0 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cabfff640 1 -- 172.21.15.59:0/787549805 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4011d90 con 0x7f6ca00051c0 2024-06-29T15:58:30.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.414+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb415f340 con 0x7f6ca00051c0 2024-06-29T15:58:30.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.415+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb415a090 con 0x7f6ca00051c0 2024-06-29T15:58:30.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.416+0000 7f6cabfff640 1 -- 172.21.15.59:0/787549805 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f6ca4018020 con 0x7f6ca00051c0 2024-06-29T15:58:30.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.417+0000 7f6cabfff640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c94076510 0x7f6c94078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.417+0000 7f6cabfff640 1 -- 172.21.15.59:0/787549805 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f6ca4098d70 con 0x7f6ca00051c0 2024-06-29T15:58:30.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.417+0000 7f6caa7fc640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c94076510 0x7f6c94078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.418+0000 7f6caa7fc640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c94076510 0x7f6c94078a20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6cb41536e0 tx=0x7f6c9c005f90 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:30.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.421+0000 7f6cabfff640 1 -- 172.21.15.59:0/787549805 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6cb415a090 con 0x7f6ca00051c0 2024-06-29T15:58:30.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.611+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6cb40b38e0 con 0x7f6ca00051c0 2024-06-29T15:58:30.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.612+0000 7f6cabfff640 1 -- 172.21.15.59:0/787549805 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v242) ==== 104+0+11 (secure 0 0 0) 0x7f6ca401f020 con 0x7f6ca00051c0 2024-06-29T15:58:30.613 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:58:30.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c94076510 msgr2=0x7f6c94078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c94076510 0x7f6c94078a20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6cb41536e0 tx=0x7f6c9c005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:30.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 msgr2=0x7f6cb40c6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 0x7f6cb40c6850 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4004ed0 tx=0x7f6ca40015d0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 shutdown_connections 2024-06-29T15:58:30.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f6c94076510 0x7f6c94078a20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cb4156eb0 0x7f6cb41572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6cb4158e40 0x7f6cb4159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 --2- 172.21.15.59:0/787549805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ca00051c0 0x7f6cb40c6850 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 >> 172.21.15.59:0/787549805 conn(0x7f6cb40d45b0 msgr2=0x7f6cb4068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:30.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 shutdown_connections 2024-06-29T15:58:30.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.614+0000 7f6cb96ba640 1 -- 172.21.15.59:0/787549805 wait complete. 2024-06-29T15:58:30.626 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:58:30.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.747+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1258952941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d84144470 msgr2=0x7f1d840b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.747+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1258952941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d84144470 0x7f1d840b1b00 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f1d84072450 tx=0x7f1d7c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.747+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1258952941 shutdown_connections 2024-06-29T15:58:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.747+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1258952941 >> v2:172.21.15.153:3300/0 conn(0x7f1d840b2040 0x7f1d8406e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.747+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1258952941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d84144470 0x7f1d840b1b00 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.747+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1258952941 >> 172.21.15.59:0/1258952941 conn(0x7f1d840b06e0 msgr2=0x7f1d840b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.747+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1258952941 shutdown_connections 2024-06-29T15:58:30.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.747+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1258952941 wait complete. 2024-06-29T15:58:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.749+0000 7f1d8ac4d640 1 Processor -- start 2024-06-29T15:58:30.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.749+0000 7f1d8ac4d640 1 -- start start 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.750+0000 7f1d8ac4d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d8406e840 0x7f1d841544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.750+0000 7f1d8ac4d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d840b2040 0x7f1d840a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d8ac4d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d84144470 0x7f1d841330d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d8ac4d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1d840d0360 con 0x7f1d8406e840 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d8ac4d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d840d01e0 con 0x7f1d84144470 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d8ac4d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8415ee80 con 0x7f1d840b2040 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d8406e840 0x7f1d841544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d8406e840 0x7f1d841544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36098/0 (socket says 172.21.15.59:36098) 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 -- 172.21.15.59:0/1176941432 learned_addr learned my addr 172.21.15.59:0/1176941432 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:30.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d83fff640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d840b2040 0x7f1d840a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d891ea640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d84144470 0x7f1d841330d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 -- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d840b2040 msgr2=0x7f1d840a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d840b2040 0x7f1d840a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 -- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d84144470 msgr2=0x7f1d841330d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d84144470 0x7f1d841330d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 -- 172.21.15.59:0/1176941432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d84131290 con 0x7f1d8406e840 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d83fff640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d840b2040 0x7f1d840a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d891ea640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d84144470 0x7f1d841330d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d889e9640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d8406e840 0x7f1d841544e0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c02aea0 tx=0x7f1d7c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d817fa640 1 -- 172.21.15.59:0/1176941432 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c031b90 con 0x7f1d8406e840 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d817fa640 1 -- 172.21.15.59:0/1176941432 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d7c031e20 con 0x7f1d8406e840 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d84131570 con 0x7f1d8406e840 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.751+0000 7f1d817fa640 1 -- 172.21.15.59:0/1176941432 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0332d0 con 0x7f1d8406e840 2024-06-29T15:58:30.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.752+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d84131b30 con 0x7f1d8406e840 2024-06-29T15:58:30.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.752+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d8415f340 con 0x7f1d8406e840 2024-06-29T15:58:30.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.753+0000 7f1d817fa640 1 -- 172.21.15.59:0/1176941432 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f1d7c033470 con 0x7f1d8406e840 2024-06-29T15:58:30.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.754+0000 7f1d817fa640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1d70076540 0x7f1d70078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.754+0000 7f1d817fa640 1 -- 172.21.15.59:0/1176941432 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f1d7c0b93f0 con 0x7f1d8406e840 2024-06-29T15:58:30.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.754+0000 7f1d83fff640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1d70076540 0x7f1d70078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.755+0000 7f1d83fff640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1d70076540 0x7f1d70078a50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d84130f10 tx=0x7f1d78005f10 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:30.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.758+0000 7f1d817fa640 1 -- 172.21.15.59:0/1176941432 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d7c03f030 con 0x7f1d8406e840 2024-06-29T15:58:30.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.947+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1d8406d870 con 0x7f1d8406e840 2024-06-29T15:58:30.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.948+0000 7f1d817fa640 1 -- 172.21.15.59:0/1176941432 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v242) ==== 100+0+11 (secure 0 0 0) 0x7f1d7c082690 con 0x7f1d8406e840 2024-06-29T15:58:30.949 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1d70076540 msgr2=0x7f1d70078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1d70076540 0x7f1d70078a50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d84130f10 tx=0x7f1d78005f10 comp rx=0 tx=0).stop 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d8406e840 msgr2=0x7f1d841544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d8406e840 0x7f1d841544e0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c02aea0 tx=0x7f1d7c003970 comp rx=0 tx=0).stop 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 shutdown_connections 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f1d70076540 0x7f1d70078a50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d84144470 0x7f1d841330d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1d840b2040 0x7f1d840a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 --2- 172.21.15.59:0/1176941432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d8406e840 0x7f1d841544e0 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 >> 172.21.15.59:0/1176941432 conn(0x7f1d840b06e0 msgr2=0x7f1d84141230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:30.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 shutdown_connections 2024-06-29T15:58:30.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:30.950+0000 7f1d8ac4d640 1 -- 172.21.15.59:0/1176941432 wait complete. 2024-06-29T15:58:30.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:58:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.079+0000 7fdd5a911640 1 -- 172.21.15.59:0/3960514150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 msgr2=0x7fdd5415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.079+0000 7fdd5a911640 1 --2- 172.21.15.59:0/3960514150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 0x7fdd5415e8d0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fdd540d15d0 tx=0x7fdd4802a530 comp rx=0 tx=0).stop 2024-06-29T15:58:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.079+0000 7fdd5a911640 1 -- 172.21.15.59:0/3960514150 shutdown_connections 2024-06-29T15:58:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.079+0000 7fdd5a911640 1 --2- 172.21.15.59:0/3960514150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 0x7fdd5415e8d0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.079+0000 7fdd5a911640 1 --2- 172.21.15.59:0/3960514150 >> v2:172.21.15.153:3300/0 conn(0x7fdd540d23c0 0x7fdd540b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.079+0000 7fdd5a911640 1 -- 172.21.15.59:0/3960514150 >> 172.21.15.59:0/3960514150 conn(0x7fdd540d3ee0 msgr2=0x7fdd540d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.079+0000 7fdd5a911640 1 -- 172.21.15.59:0/3960514150 shutdown_connections 2024-06-29T15:58:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.079+0000 7fdd5a911640 1 -- 172.21.15.59:0/3960514150 wait complete. 2024-06-29T15:58:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.081+0000 7fdd5a911640 1 Processor -- start 2024-06-29T15:58:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.081+0000 7fdd5a911640 1 -- start start 2024-06-29T15:58:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd5a911640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 0x7fdd540c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd5a911640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d23c0 0x7fdd541e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd5a911640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd5415ee10 0x7fdd540c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd5a911640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdd54160240 con 0x7fdd540b3ed0 2024-06-29T15:58:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd5a911640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541600c0 con 0x7fdd5415ee10 2024-06-29T15:58:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd5a911640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd541603c0 con 0x7fdd540d23c0 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 0x7fdd540c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 0x7fdd540c69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36130/0 (socket says 172.21.15.59:36130) 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 -- 172.21.15.59:0/636163323 learned_addr learned my addr 172.21.15.59:0/636163323 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd537fe640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d23c0 0x7fdd541e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd58eae640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd5415ee10 0x7fdd540c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 -- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d23c0 msgr2=0x7fdd541e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d23c0 0x7fdd541e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 -- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd5415ee10 msgr2=0x7fdd540c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd5415ee10 0x7fdd540c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 -- 172.21.15.59:0/636163323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd540c8520 con 0x7fdd540b3ed0 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd53fff640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 0x7fdd540c69e0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009fd0 tx=0x7fdd4400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:31.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.083+0000 7fdd50ff9640 1 -- 172.21.15.59:0/636163323 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4400ccc0 con 0x7fdd540b3ed0 2024-06-29T15:58:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.084+0000 7fdd50ff9640 1 -- 172.21.15.59:0/636163323 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd4400ee70 con 0x7fdd540b3ed0 2024-06-29T15:58:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.084+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd540c6f20 con 0x7fdd540b3ed0 2024-06-29T15:58:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.084+0000 7fdd50ff9640 1 -- 172.21.15.59:0/636163323 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4400ce60 con 0x7fdd540b3ed0 2024-06-29T15:58:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.084+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd540c9430 con 0x7fdd540b3ed0 2024-06-29T15:58:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.084+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd5415e8d0 con 0x7fdd540b3ed0 2024-06-29T15:58:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.086+0000 7fdd50ff9640 1 -- 172.21.15.59:0/636163323 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fdd44004280 con 0x7fdd540b3ed0 2024-06-29T15:58:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.086+0000 7fdd50ff9640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdd40076540 0x7fdd40078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.086+0000 7fdd50ff9640 1 -- 172.21.15.59:0/636163323 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7fdd44098720 con 0x7fdd540b3ed0 2024-06-29T15:58:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.086+0000 7fdd537fe640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdd40076540 0x7fdd40078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.087+0000 7fdd537fe640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdd40076540 0x7fdd40078a50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdd48034000 tx=0x7fdd48004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.090+0000 7fdd50ff9640 1 -- 172.21.15.59:0/636163323 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd44061a50 con 0x7fdd540b3ed0 2024-06-29T15:58:31.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.282+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdd540d2950 con 0x7fdd540b3ed0 2024-06-29T15:58:31.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.283+0000 7fdd50ff9640 1 -- 172.21.15.59:0/636163323 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v242) ==== 84+0+209 (secure 0 0 0) 0x7fdd44061350 con 0x7fdd540b3ed0 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 31, 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": "cephfs_metadata" 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:31.284 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:31.285 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 32, 2024-06-29T15:58:31.285 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": "cephfs_data" 2024-06-29T15:58:31.285 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:31.285 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.284+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdd40076540 msgr2=0x7fdd40078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.284+0000 7fdd5a911640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdd40076540 0x7fdd40078a50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdd48034000 tx=0x7fdd48004730 comp rx=0 tx=0).stop 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.284+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 msgr2=0x7fdd540c69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.284+0000 7fdd5a911640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 0x7fdd540c69e0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009fd0 tx=0x7fdd4400ec20 comp rx=0 tx=0).stop 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.285+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 shutdown_connections 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.285+0000 7fdd5a911640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fdd40076540 0x7fdd40078a50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.285+0000 7fdd5a911640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd5415ee10 0x7fdd540c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.285+0000 7fdd5a911640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd540d23c0 0x7fdd541e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.285+0000 7fdd5a911640 1 --2- 172.21.15.59:0/636163323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd540b3ed0 0x7fdd540c69e0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.285+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 >> 172.21.15.59:0/636163323 conn(0x7fdd540d3ee0 msgr2=0x7fdd5406d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:31.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.285+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 shutdown_connections 2024-06-29T15:58:31.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.285+0000 7fdd5a911640 1 -- 172.21.15.59:0/636163323 wait complete. 2024-06-29T15:58:31.297 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-06-29T15:58:31.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-06-29T15:58:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.419+0000 7f9d72d35640 1 -- 172.21.15.59:0/2277337873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d5c003c10 msgr2=0x7f9d5c004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.419+0000 7f9d72d35640 1 --2- 172.21.15.59:0/2277337873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d5c003c10 0x7f9d5c004060 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009b50 tx=0x7f9d5c005830 comp rx=0 tx=0).stop 2024-06-29T15:58:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.419+0000 7f9d712d2640 1 -- 172.21.15.59:0/2277337873 reap_dead start 2024-06-29T15:58:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.419+0000 7f9d72d35640 1 -- 172.21.15.59:0/2277337873 shutdown_connections 2024-06-29T15:58:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.419+0000 7f9d72d35640 1 -- 172.21.15.59:0/2277337873 >> 172.21.15.59:0/2277337873 conn(0x7f9d6c073e10 msgr2=0x7f9d6c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.419+0000 7f9d72d35640 1 -- 172.21.15.59:0/2277337873 shutdown_connections 2024-06-29T15:58:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.419+0000 7f9d72d35640 1 -- 172.21.15.59:0/2277337873 wait complete. 2024-06-29T15:58:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.421+0000 7f9d72d35640 1 Processor -- start 2024-06-29T15:58:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.421+0000 7f9d72d35640 1 -- start start 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d72d35640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c003c10 0x7f9d6c0abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d72d35640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d6c0ab100 0x7f9d6c0ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d72d35640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d6c0b2e60 0x7f9d6c0b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d72d35640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c0b5a70 con 0x7f9d6c0ab100 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d72d35640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c0b58f0 con 0x7f9d5c003c10 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d72d35640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9d6c15c480 con 0x7f9d6c0b2e60 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d6c0ab100 0x7f9d6c0ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d6c0ab100 0x7f9d6c0ac510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36156/0 (socket says 172.21.15.59:36156) 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.422+0000 7f9d6bfff640 1 -- 172.21.15.59:0/2069010557 learned_addr learned my addr 172.21.15.59:0/2069010557 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d70ad1640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c003c10 0x7f9d6c0abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d712d2640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d6c0b2e60 0x7f9d6c0b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d6bfff640 1 -- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d6c0b2e60 msgr2=0x7f9d6c0b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d6bfff640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d6c0b2e60 0x7f9d6c0b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d6bfff640 1 -- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c003c10 msgr2=0x7f9d6c0abfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d6bfff640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c003c10 0x7f9d6c0abfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d6bfff640 1 -- 172.21.15.59:0/2069010557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c0b37b0 con 0x7f9d6c0ab100 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d6bfff640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d6c0ab100 0x7f9d6c0ac510 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c073dc0 tx=0x7f9d58001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d697fa640 1 -- 172.21.15.59:0/2069010557 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58003e30 con 0x7f9d6c0ab100 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d697fa640 1 -- 172.21.15.59:0/2069010557 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d58016020 con 0x7f9d6c0ab100 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d697fa640 1 -- 172.21.15.59:0/2069010557 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d580118a0 con 0x7f9d6c0ab100 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c1f3480 con 0x7f9d6c0ab100 2024-06-29T15:58:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.423+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6c0aa620 con 0x7f9d6c0ab100 2024-06-29T15:58:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.424+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6c0d01a0 con 0x7f9d6c0ab100 2024-06-29T15:58:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.425+0000 7f9d697fa640 1 -- 172.21.15.59:0/2069010557 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f9d58011a40 con 0x7f9d6c0ab100 2024-06-29T15:58:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.426+0000 7f9d697fa640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9d50076540 0x7f9d50078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.426+0000 7f9d697fa640 1 -- 172.21.15.59:0/2069010557 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6157+0+0 (secure 0 0 0) 0x7f9d580988f0 con 0x7f9d6c0ab100 2024-06-29T15:58:31.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.426+0000 7f9d70ad1640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9d50076540 0x7f9d50078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.426+0000 7f9d70ad1640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9d50076540 0x7f9d50078a50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d60035000 tx=0x7f9d6002d040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:31.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.430+0000 7f9d697fa640 1 -- 172.21.15.59:0/2069010557 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9d58020020 con 0x7f9d6c0ab100 2024-06-29T15:58:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:31.621+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) -- 0x7f9d6c0b0f50 con 0x7f9d6c0ab100 2024-06-29T15:58:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.605+0000 7f9d697fa640 1 -- 172.21.15.59:0/2069010557 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v243) ==== 144+0+0 (secure 0 0 0) 0x7f9d6c0b37b0 con 0x7f9d6c0ab100 2024-06-29T15:58:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'backup_fs_metadata' created 2024-06-29T15:58:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9d50076540 msgr2=0x7f9d50078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9d50076540 0x7f9d50078a50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d60035000 tx=0x7f9d6002d040 comp rx=0 tx=0).stop 2024-06-29T15:58:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d6c0ab100 msgr2=0x7f9d6c0ac510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d6c0ab100 0x7f9d6c0ac510 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c073dc0 tx=0x7f9d58001800 comp rx=0 tx=0).stop 2024-06-29T15:58:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 shutdown_connections 2024-06-29T15:58:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f9d50076540 0x7f9d50078a50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9d6c0b2e60 0x7f9d6c0b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d6c0ab100 0x7f9d6c0ac510 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 --2- 172.21.15.59:0/2069010557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c003c10 0x7f9d6c0abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 >> 172.21.15.59:0/2069010557 conn(0x7f9d6c073e10 msgr2=0x7f9d6c15e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:32.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 shutdown_connections 2024-06-29T15:58:32.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.609+0000 7f9d72d35640 1 -- 172.21.15.59:0/2069010557 wait complete. 2024-06-29T15:58:32.629 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:58:32.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 -- 172.21.15.59:0/3848565037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30013db70 msgr2=0x7fc3000cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 --2- 172.21.15.59:0/3848565037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30013db70 0x7fc3000cf170 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fc3000b5d40 tx=0x7fc2f002aa30 comp rx=0 tx=0).stop 2024-06-29T15:58:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 -- 172.21.15.59:0/3848565037 shutdown_connections 2024-06-29T15:58:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 --2- 172.21.15.59:0/3848565037 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc300140840 0x7fc30015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 --2- 172.21.15.59:0/3848565037 >> v2:172.21.15.153:3300/0 conn(0x7fc3000cf6b0 0x7fc300140300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 --2- 172.21.15.59:0/3848565037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30013db70 0x7fc3000cf170 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 -- 172.21.15.59:0/3848565037 >> 172.21.15.59:0/3848565037 conn(0x7fc300070f10 msgr2=0x7fc300071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 -- 172.21.15.59:0/3848565037 shutdown_connections 2024-06-29T15:58:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.760+0000 7fc3085be640 1 -- 172.21.15.59:0/3848565037 wait complete. 2024-06-29T15:58:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.762+0000 7fc3085be640 1 Processor -- start 2024-06-29T15:58:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.762+0000 7fc3085be640 1 -- start start 2024-06-29T15:58:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.763+0000 7fc3085be640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000cf6b0 0x7fc30013cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.763+0000 7fc3085be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30013db70 0x7fc3001f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.763+0000 7fc3085be640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc300140840 0x7fc300159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc3085be640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc30015f080 con 0x7fc3000cf6b0 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc3085be640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc30015ef00 con 0x7fc30013db70 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc3085be640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc30015f200 con 0x7fc300140840 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000cf6b0 0x7fc30013cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000cf6b0 0x7fc30013cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50078/0 (socket says 172.21.15.59:50078) 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 -- 172.21.15.59:0/1082565857 learned_addr learned my addr 172.21.15.59:0/1082565857 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc306b5b640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc300140840 0x7fc300159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc305b59640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30013db70 0x7fc3001f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 -- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc300140840 msgr2=0x7fc300159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc300140840 0x7fc300159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 -- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30013db70 msgr2=0x7fc3001f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30013db70 0x7fc3001f4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 -- 172.21.15.59:0/1082565857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3001f49c0 con 0x7fc3000cf6b0 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc306b5b640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc300140840 0x7fc300159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc30635a640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000cf6b0 0x7fc30013cfb0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009d60 tx=0x7fc2f0003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc2feffd640 1 -- 172.21.15.59:0/1082565857 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0033050 con 0x7fc3000cf6b0 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc2feffd640 1 -- 172.21.15.59:0/1082565857 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2f00045f0 con 0x7fc3000cf6b0 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc300154650 con 0x7fc3000cf6b0 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc2feffd640 1 -- 172.21.15.59:0/1082565857 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0038450 con 0x7fc3000cf6b0 2024-06-29T15:58:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.764+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30015dbe0 con 0x7fc3000cf6b0 2024-06-29T15:58:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.765+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3000cf170 con 0x7fc3000cf6b0 2024-06-29T15:58:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.766+0000 7fc2feffd640 1 -- 172.21.15.59:0/1082565857 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fc2f0036070 con 0x7fc3000cf6b0 2024-06-29T15:58:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.767+0000 7fc2feffd640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc2ec076510 0x7fc2ec078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.767+0000 7fc2feffd640 1 -- 172.21.15.59:0/1082565857 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(243..243 src has 1..243) ==== 6530+0+0 (secure 0 0 0) 0x7fc2f00b9750 con 0x7fc3000cf6b0 2024-06-29T15:58:32.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.767+0000 7fc305b59640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc2ec076510 0x7fc2ec078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:32.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.768+0000 7fc305b59640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc2ec076510 0x7fc2ec078a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc300153f80 tx=0x7fc2f4005b50 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.771+0000 7fc2feffd640 1 -- 172.21.15.59:0/1082565857 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2f00829e0 con 0x7fc3000cf6b0 2024-06-29T15:58:32.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:32.963+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc3001554e0 con 0x7fc3000cf6b0 2024-06-29T15:58:33.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.612+0000 7fc2feffd640 1 -- 172.21.15.59:0/1082565857 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v244) ==== 176+0+0 (secure 0 0 0) 0x7fc2f00822e0 con 0x7fc3000cf6b0 2024-06-29T15:58:33.613 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'backup_fs_data' created 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc2ec076510 msgr2=0x7fc2ec078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc2ec076510 0x7fc2ec078a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc300153f80 tx=0x7fc2f4005b50 comp rx=0 tx=0).stop 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000cf6b0 msgr2=0x7fc30013cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000cf6b0 0x7fc30013cfb0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009d60 tx=0x7fc2f0003f90 comp rx=0 tx=0).stop 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 shutdown_connections 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fc2ec076510 0x7fc2ec078a20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc300140840 0x7fc300159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc30013db70 0x7fc3001f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 --2- 172.21.15.59:0/1082565857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3000cf6b0 0x7fc30013cfb0 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 >> 172.21.15.59:0/1082565857 conn(0x7fc300070f10 msgr2=0x7fc3000b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:33.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 shutdown_connections 2024-06-29T15:58:33.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.615+0000 7fc3085be640 1 -- 172.21.15.59:0/1082565857 wait complete. 2024-06-29T15:58:33.641 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-06-29T15:58:33.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.769+0000 7f315aa9e640 1 -- 172.21.15.59:0/3832160969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31540d1a60 msgr2=0x7f31540b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.769+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3832160969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31540d1a60 0x7f31540b5b30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f314c005fd0 tx=0x7f314c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.769+0000 7f315aa9e640 1 -- 172.21.15.59:0/3832160969 shutdown_connections 2024-06-29T15:58:33.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.769+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3832160969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31540d1a60 0x7f31540b5b30 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.769+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3832160969 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31540b2fa0 0x7f31540d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.769+0000 7f315aa9e640 1 -- 172.21.15.59:0/3832160969 >> 172.21.15.59:0/3832160969 conn(0x7f3154073e10 msgr2=0x7f3154074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:33.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.769+0000 7f315aa9e640 1 -- 172.21.15.59:0/3832160969 shutdown_connections 2024-06-29T15:58:33.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.769+0000 7f315aa9e640 1 -- 172.21.15.59:0/3832160969 wait complete. 2024-06-29T15:58:33.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.771+0000 7f315aa9e640 1 Processor -- start 2024-06-29T15:58:33.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.771+0000 7f315aa9e640 1 -- start start 2024-06-29T15:58:33.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315aa9e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31540b2270 0x7f3154146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:33.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315aa9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31540b2fa0 0x7f315415e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:33.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315aa9e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31540d1a60 0x7f3154142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:33.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315aa9e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f31540b6400 con 0x7f31540b2270 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315aa9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31540b6280 con 0x7f31540b2fa0 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315aa9e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f31540b6580 con 0x7f31540d1a60 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315883a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31540b2270 0x7f3154146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315883a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31540b2270 0x7f3154146450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50084/0 (socket says 172.21.15.59:50084) 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.772+0000 7f315883a640 1 -- 172.21.15.59:0/3521085413 learned_addr learned my addr 172.21.15.59:0/3521085413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f3153fff640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31540b2fa0 0x7f315415e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315903b640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31540d1a60 0x7f3154142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315883a640 1 -- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31540d1a60 msgr2=0x7f3154142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315883a640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31540d1a60 0x7f3154142de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315883a640 1 -- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31540b2fa0 msgr2=0x7f315415e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315883a640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31540b2fa0 0x7f315415e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315883a640 1 -- 172.21.15.59:0/3521085413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315415e5a0 con 0x7f31540b2270 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315903b640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31540d1a60 0x7f3154142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315883a640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31540b2270 0x7f3154146450 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f314800c370 tx=0x7f314800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f31517fa640 1 -- 172.21.15.59:0/3521085413 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31480052d0 con 0x7f31540b2270 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315415e870 con 0x7f31540b2270 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f31517fa640 1 -- 172.21.15.59:0/3521085413 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3148007020 con 0x7f31540b2270 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f31517fa640 1 -- 172.21.15.59:0/3521085413 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3148007470 con 0x7f31540b2270 2024-06-29T15:58:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.773+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315415ee30 con 0x7f31540b2270 2024-06-29T15:58:33.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.774+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31540b6aa0 con 0x7f31540b2270 2024-06-29T15:58:33.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.775+0000 7f31517fa640 1 -- 172.21.15.59:0/3521085413 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f31480067d0 con 0x7f31540b2270 2024-06-29T15:58:33.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.776+0000 7f31517fa640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3138076510 0x7f3138078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:33.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.776+0000 7f31517fa640 1 -- 172.21.15.59:0/3521085413 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(244..244 src has 1..244) ==== 6915+0+0 (secure 0 0 0) 0x7f31480975d0 con 0x7f31540b2270 2024-06-29T15:58:33.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.776+0000 7f3153fff640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3138076510 0x7f3138078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:33.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.777+0000 7f3153fff640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3138076510 0x7f3138078a20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f31400096c0 tx=0x7f3140009260 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:33.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:33.780+0000 7f31517fa640 1 -- 172.21.15.59:0/3521085413 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f314805ffe0 con 0x7f31540b2270 2024-06-29T15:58:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.010+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) -- 0x7f315415bda0 con 0x7f31540b2270 2024-06-29T15:58:34.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.613+0000 7f31517fa640 1 -- 172.21.15.59:0/3521085413 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 33 and data pool 34 v134) ==== 183+0+0 (secure 0 0 0) 0x7f31480642e0 con 0x7f31540b2270 2024-06-29T15:58:34.615 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 33 and data pool 34 2024-06-29T15:58:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3138076510 msgr2=0x7f3138078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3138076510 0x7f3138078a20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f31400096c0 tx=0x7f3140009260 comp rx=0 tx=0).stop 2024-06-29T15:58:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31540b2270 msgr2=0x7f3154146450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31540b2270 0x7f3154146450 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f314800c370 tx=0x7f314800c830 comp rx=0 tx=0).stop 2024-06-29T15:58:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 shutdown_connections 2024-06-29T15:58:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3138076510 0x7f3138078a20 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f31400096c0 tx=0x7f3140009260 comp rx=0 tx=0).stop 2024-06-29T15:58:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31540d1a60 0x7f3154142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31540b2fa0 0x7f315415e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 --2- 172.21.15.59:0/3521085413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31540b2270 0x7f3154146450 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.618+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 >> 172.21.15.59:0/3521085413 conn(0x7f3154073e10 msgr2=0x7f31540b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.619+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 shutdown_connections 2024-06-29T15:58:34.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.619+0000 7f315aa9e640 1 -- 172.21.15.59:0/3521085413 wait complete. 2024-06-29T15:58:34.658 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:34.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.780+0000 7f4280570640 1 -- 172.21.15.59:0/681243368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42780b3740 msgr2=0x7f42780b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.780+0000 7f4280570640 1 --2- 172.21.15.59:0/681243368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42780b3740 0x7f42780b3b50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f426c009830 tx=0x7f426c02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.780+0000 7f4280570640 1 -- 172.21.15.59:0/681243368 shutdown_connections 2024-06-29T15:58:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.780+0000 7f4280570640 1 --2- 172.21.15.59:0/681243368 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f42780b3740 0x7f42780b3b50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.780+0000 7f4280570640 1 --2- 172.21.15.59:0/681243368 >> v2:172.21.15.153:3300/0 conn(0x7f42781458a0 0x7f42780cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.780+0000 7f4280570640 1 -- 172.21.15.59:0/681243368 >> 172.21.15.59:0/681243368 conn(0x7f427806c950 msgr2=0x7f4278073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.780+0000 7f4280570640 1 -- 172.21.15.59:0/681243368 shutdown_connections 2024-06-29T15:58:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.780+0000 7f4280570640 1 -- 172.21.15.59:0/681243368 wait complete. 2024-06-29T15:58:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.782+0000 7f4280570640 1 Processor -- start 2024-06-29T15:58:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.782+0000 7f4280570640 1 -- start start 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.783+0000 7f4280570640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42780b3740 0x7f4278138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.783+0000 7f4280570640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f427815c2f0 0x7f427815b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.783+0000 7f4280570640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f427813d020 0x7f427813d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.783+0000 7f4280570640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4278160020 con 0x7f427815c2f0 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.783+0000 7f4280570640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f427815cbb0 con 0x7f42780b3740 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.783+0000 7f4280570640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f42781601a0 con 0x7f427813d020 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f427815c2f0 0x7f427815b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f427815c2f0 0x7f427815b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50090/0 (socket says 172.21.15.59:50090) 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 -- 172.21.15.59:0/2227450910 learned_addr learned my addr 172.21.15.59:0/2227450910 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427e30c640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42780b3740 0x7f4278138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427eb0d640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f427813d020 0x7f427813d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 -- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f427813d020 msgr2=0x7f427813d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f427813d020 0x7f427813d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 -- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42780b3740 msgr2=0x7f4278138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42780b3740 0x7f4278138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 -- 172.21.15.59:0/2227450910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427806eec0 con 0x7f427815c2f0 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427e30c640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42780b3740 0x7f4278138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f427db0b640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f427815c2f0 0x7f427815b810 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f426c02aa90 tx=0x7f426c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f4276ffd640 1 -- 172.21.15.59:0/2227450910 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c005c90 con 0x7f427815c2f0 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f4276ffd640 1 -- 172.21.15.59:0/2227450910 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f426c005020 con 0x7f427815c2f0 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427806f1a0 con 0x7f427815c2f0 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.784+0000 7f4276ffd640 1 -- 172.21.15.59:0/2227450910 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c005450 con 0x7f427815c2f0 2024-06-29T15:58:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.785+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427806f760 con 0x7f427815c2f0 2024-06-29T15:58:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.785+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42781606c0 con 0x7f427815c2f0 2024-06-29T15:58:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.786+0000 7f4276ffd640 1 -- 172.21.15.59:0/2227450910 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f426c0317c0 con 0x7f427815c2f0 2024-06-29T15:58:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.787+0000 7f4276ffd640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4264076510 0x7f4264078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.787+0000 7f4276ffd640 1 -- 172.21.15.59:0/2227450910 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 7021+0+0 (secure 0 0 0) 0x7f426c0b90a0 con 0x7f427815c2f0 2024-06-29T15:58:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.787+0000 7f427e30c640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4264076510 0x7f4264078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.788+0000 7f427e30c640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4264076510 0x7f4264078a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f42680099a0 tx=0x7f4268009500 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.791+0000 7f4276ffd640 1 -- 172.21.15.59:0/2227450910 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f426c03f030 con 0x7f427815c2f0 2024-06-29T15:58:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.985+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f427806fd90 con 0x7f427815c2f0 2024-06-29T15:58:34.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.987+0000 7f4276ffd640 1 -- 172.21.15.59:0/2227450910 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v245) ==== 74+0+20793 (secure 0 0 0) 0x7f426c08a0b0 con 0x7f427815c2f0 2024-06-29T15:58:34.988 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:34.988 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":245,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:34.012139+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4264076510 msgr2=0x7f4264078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4264076510 0x7f4264078a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f42680099a0 tx=0x7f4268009500 comp rx=0 tx=0).stop 2024-06-29T15:58:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f427815c2f0 msgr2=0x7f427815b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f427815c2f0 0x7f427815b810 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f426c02aa90 tx=0x7f426c003c10 comp rx=0 tx=0).stop 2024-06-29T15:58:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 shutdown_connections 2024-06-29T15:58:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4264076510 0x7f4264078a20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f427813d020 0x7f427813d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f427815c2f0 0x7f427815b810 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 --2- 172.21.15.59:0/2227450910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42780b3740 0x7f4278138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 >> 172.21.15.59:0/2227450910 conn(0x7f427806c950 msgr2=0x7f42780d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 shutdown_connections 2024-06-29T15:58:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:34.989+0000 7f4280570640 1 -- 172.21.15.59:0/2227450910 wait complete. 2024-06-29T15:58:35.002 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.118+0000 7fbbaebf5640 1 -- 172.21.15.59:0/1114604743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba8162190 msgr2=0x7fbba80b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.118+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/1114604743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba8162190 0x7fbba80b38e0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fbba80d3710 tx=0x7fbba002a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.118+0000 7fbbaebf5640 1 -- 172.21.15.59:0/1114604743 shutdown_connections 2024-06-29T15:58:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.118+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/1114604743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba8162190 0x7fbba80b38e0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.118+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/1114604743 >> v2:172.21.15.153:3300/0 conn(0x7fbba8162770 0x7fbba80ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.118+0000 7fbbaebf5640 1 -- 172.21.15.59:0/1114604743 >> 172.21.15.59:0/1114604743 conn(0x7fbba80d45b0 msgr2=0x7fbba80d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.118+0000 7fbbaebf5640 1 -- 172.21.15.59:0/1114604743 shutdown_connections 2024-06-29T15:58:35.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.118+0000 7fbbaebf5640 1 -- 172.21.15.59:0/1114604743 wait complete. 2024-06-29T15:58:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.120+0000 7fbbaebf5640 1 Processor -- start 2024-06-29T15:58:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.120+0000 7fbbaebf5640 1 -- start start 2024-06-29T15:58:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbaebf5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba815e0a0 0x7fbba80cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbaebf5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbba8162770 0x7fbba8151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbaebf5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba80cab30 0x7fbba8150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbaebf5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbba80b4270 con 0x7fbba80cab30 2024-06-29T15:58:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbaebf5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbba80b40f0 con 0x7fbba8162770 2024-06-29T15:58:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbaebf5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbba80b43f0 con 0x7fbba815e0a0 2024-06-29T15:58:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba80cab30 0x7fbba8150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba80cab30 0x7fbba8150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50096/0 (socket says 172.21.15.59:50096) 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 -- 172.21.15.59:0/3099713478 learned_addr learned my addr 172.21.15.59:0/3099713478 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbac991640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba815e0a0 0x7fbba80cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbba7fff640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbba8162770 0x7fbba8151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 -- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba815e0a0 msgr2=0x7fbba80cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba815e0a0 0x7fbba80cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 -- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbba8162770 msgr2=0x7fbba8151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbba8162770 0x7fbba8151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 -- 172.21.15.59:0/3099713478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba80cb280 con 0x7fbba80cab30 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbba7fff640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbba8162770 0x7fbba8151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbbad192640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba80cab30 0x7fbba8150bc0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fbba0034040 tx=0x7fbba0003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.122+0000 7fbba57fa640 1 -- 172.21.15.59:0/3099713478 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba003e030 con 0x7fbba80cab30 2024-06-29T15:58:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.123+0000 7fbba57fa640 1 -- 172.21.15.59:0/3099713478 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbba0003e50 con 0x7fbba80cab30 2024-06-29T15:58:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.123+0000 7fbba57fa640 1 -- 172.21.15.59:0/3099713478 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba0035040 con 0x7fbba80cab30 2024-06-29T15:58:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.123+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba80cb560 con 0x7fbba80cab30 2024-06-29T15:58:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.123+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba80cbb20 con 0x7fbba80cab30 2024-06-29T15:58:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.124+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba80b4910 con 0x7fbba80cab30 2024-06-29T15:58:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.124+0000 7fbba57fa640 1 -- 172.21.15.59:0/3099713478 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fbba0041030 con 0x7fbba80cab30 2024-06-29T15:58:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.125+0000 7fbba57fa640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbb88076510 0x7fbb88078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.125+0000 7fbba57fa640 1 -- 172.21.15.59:0/3099713478 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 7021+0+0 (secure 0 0 0) 0x7fbba00b8970 con 0x7fbba80cab30 2024-06-29T15:58:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.125+0000 7fbbac991640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbb88076510 0x7fbb88078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.126+0000 7fbbac991640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbb88076510 0x7fbb88078a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbb900058c0 tx=0x7fbb90005f60 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.129+0000 7fbba57fa640 1 -- 172.21.15.59:0/3099713478 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbba00812b0 con 0x7fbba80cab30 2024-06-29T15:58:35.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.325+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbba80cafd0 con 0x7fbba80cab30 2024-06-29T15:58:35.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.326+0000 7fbba57fa640 1 -- 172.21.15.59:0/3099713478 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v245) ==== 74+0+20793 (secure 0 0 0) 0x7fbba00855b0 con 0x7fbba80cab30 2024-06-29T15:58:35.327 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:35.327 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":245,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:34.012139+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbb88076510 msgr2=0x7fbb88078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbb88076510 0x7fbb88078a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbb900058c0 tx=0x7fbb90005f60 comp rx=0 tx=0).stop 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba80cab30 msgr2=0x7fbba8150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba80cab30 0x7fbba8150bc0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fbba0034040 tx=0x7fbba0003960 comp rx=0 tx=0).stop 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 shutdown_connections 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbb88076510 0x7fbb88078a20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba80cab30 0x7fbba8150bc0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbba8162770 0x7fbba8151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 --2- 172.21.15.59:0/3099713478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbba815e0a0 0x7fbba80cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 >> 172.21.15.59:0/3099713478 conn(0x7fbba80d45b0 msgr2=0x7fbba8152e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 shutdown_connections 2024-06-29T15:58:35.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.328+0000 7fbbaebf5640 1 -- 172.21.15.59:0/3099713478 wait complete. 2024-06-29T15:58:35.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-06-29T15:58:35.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.460+0000 7f77ee2ef640 1 -- 172.21.15.59:0/3952321329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2270 msgr2=0x7f77e815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.460+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/3952321329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2270 0x7f77e815bc20 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f77e8074c50 tx=0x7f77d802a530 comp rx=0 tx=0).stop 2024-06-29T15:58:35.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.460+0000 7f77ee2ef640 1 -- 172.21.15.59:0/3952321329 shutdown_connections 2024-06-29T15:58:35.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.460+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/3952321329 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77e80b2fa0 0x7f77e80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.460+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/3952321329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2270 0x7f77e815bc20 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.460+0000 7f77ee2ef640 1 -- 172.21.15.59:0/3952321329 >> 172.21.15.59:0/3952321329 conn(0x7f77e8073e10 msgr2=0x7f77e8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.460+0000 7f77ee2ef640 1 -- 172.21.15.59:0/3952321329 shutdown_connections 2024-06-29T15:58:35.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.460+0000 7f77ee2ef640 1 -- 172.21.15.59:0/3952321329 wait complete. 2024-06-29T15:58:35.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.462+0000 7f77ee2ef640 1 Processor -- start 2024-06-29T15:58:35.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.462+0000 7f77ee2ef640 1 -- start start 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77ee2ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77e80b2270 0x7f77e815e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77ee2ef640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2fa0 0x7f77e8145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77ee2ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77e80d1a60 0x7f77e814a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77ee2ef640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f77e80b6cc0 con 0x7f77e80b2fa0 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77ee2ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f77e80b6b40 con 0x7f77e80b2270 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77ee2ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f77e80cfe90 con 0x7f77e80d1a60 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2fa0 0x7f77e8145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2fa0 0x7f77e8145190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50120/0 (socket says 172.21.15.59:50120) 2024-06-29T15:58:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 -- 172.21.15.59:0/1993100342 learned_addr learned my addr 172.21.15.59:0/1993100342 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e7fff640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77e80b2270 0x7f77e815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77ec88c640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77e80d1a60 0x7f77e814a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 -- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77e80d1a60 msgr2=0x7f77e814a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77e80d1a60 0x7f77e814a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 -- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77e80b2270 msgr2=0x7f77e815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77e80b2270 0x7f77e815e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 -- 172.21.15.59:0/1993100342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e815e660 con 0x7f77e80b2fa0 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.464+0000 7f77e77fe640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2fa0 0x7f77e8145190 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00c9a0 tx=0x7f77dc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.465+0000 7f77e4ff9640 1 -- 172.21.15.59:0/1993100342 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc007bb0 con 0x7f77e80b2fa0 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.465+0000 7f77e4ff9640 1 -- 172.21.15.59:0/1993100342 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77dc007e40 con 0x7f77e80b2fa0 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.465+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e815d080 con 0x7f77e80b2fa0 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.465+0000 7f77e4ff9640 1 -- 172.21.15.59:0/1993100342 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00f690 con 0x7f77e80b2fa0 2024-06-29T15:58:35.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.465+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e815d640 con 0x7f77e80b2fa0 2024-06-29T15:58:35.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.466+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e815cdf0 con 0x7f77e80b2fa0 2024-06-29T15:58:35.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.467+0000 7f77e4ff9640 1 -- 172.21.15.59:0/1993100342 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f77dc00f830 con 0x7f77e80b2fa0 2024-06-29T15:58:35.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.467+0000 7f77e4ff9640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f77d4076510 0x7f77d4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.467+0000 7f77e4ff9640 1 -- 172.21.15.59:0/1993100342 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 7021+0+0 (secure 0 0 0) 0x7f77dc098870 con 0x7f77e80b2fa0 2024-06-29T15:58:35.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.467+0000 7f77e7fff640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f77d4076510 0x7f77d4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.468+0000 7f77e7fff640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f77d4076510 0x7f77d4078a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f77d8033000 tx=0x7f77d8004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.472+0000 7f77e4ff9640 1 -- 172.21.15.59:0/1993100342 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f77dc014030 con 0x7f77e80b2fa0 2024-06-29T15:58:35.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:35.713+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f77e80b2680 con 0x7f77e80b2fa0 2024-06-29T15:58:36.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.639+0000 7f77e4ff9640 1 -- 172.21.15.59:0/1993100342 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v136) ==== 121+0+0 (secure 0 0 0) 0x7f77e80b2680 con 0x7f77e80b2fa0 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f77d4076510 msgr2=0x7f77d4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f77d4076510 0x7f77d4078a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f77d8033000 tx=0x7f77d8004730 comp rx=0 tx=0).stop 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2fa0 msgr2=0x7f77e8145190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2fa0 0x7f77e8145190 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00c9a0 tx=0x7f77dc00ce60 comp rx=0 tx=0).stop 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 shutdown_connections 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f77d4076510 0x7f77d4078a20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77e80d1a60 0x7f77e814a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77e80b2fa0 0x7f77e8145190 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 --2- 172.21.15.59:0/1993100342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77e80b2270 0x7f77e815e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 >> 172.21.15.59:0/1993100342 conn(0x7f77e8073e10 msgr2=0x7f77e80d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 shutdown_connections 2024-06-29T15:58:36.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.642+0000 7f77ee2ef640 1 -- 172.21.15.59:0/1993100342 wait complete. 2024-06-29T15:58:36.657 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:58:36.658 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-06-29T15:58:36.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.787+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/4172994010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd80b8be0 msgr2=0x7fcfd81629d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.787+0000 7fcfdd4ac640 1 --2- 172.21.15.59:0/4172994010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd80b8be0 0x7fcfd81629d0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80b78c0 tx=0x7fcfd002a610 comp rx=0 tx=0).stop 2024-06-29T15:58:36.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.787+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/4172994010 shutdown_connections 2024-06-29T15:58:36.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.787+0000 7fcfdd4ac640 1 --2- 172.21.15.59:0/4172994010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd80b8be0 0x7fcfd81629d0 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.787+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/4172994010 >> 172.21.15.59:0/4172994010 conn(0x7fcfd80b5610 msgr2=0x7fcfd80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.787+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/4172994010 shutdown_connections 2024-06-29T15:58:36.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.787+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/4172994010 wait complete. 2024-06-29T15:58:36.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.789+0000 7fcfdd4ac640 1 Processor -- start 2024-06-29T15:58:36.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.789+0000 7fcfdd4ac640 1 -- start start 2024-06-29T15:58:36.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfdd4ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd808c190 0x7fcfd81592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfdd4ac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd80b8be0 0x7fcfd8155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfdd4ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd80ce950 0x7fcfd81561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfdd4ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8169690 con 0x7fcfd808c190 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfdd4ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8169510 con 0x7fcfd80ce950 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfdd4ac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd8169810 con 0x7fcfd80b8be0 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd808c190 0x7fcfd81592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfd6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd808c190 0x7fcfd81592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50146/0 (socket says 172.21.15.59:50146) 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.790+0000 7fcfd6ffd640 1 -- 172.21.15.59:0/3052803541 learned_addr learned my addr 172.21.15.59:0/3052803541 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd77fe640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd80ce950 0x7fcfd81561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd67fc640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd80b8be0 0x7fcfd8155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd6ffd640 1 -- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd80b8be0 msgr2=0x7fcfd8155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd6ffd640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd80b8be0 0x7fcfd8155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd6ffd640 1 -- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd80ce950 msgr2=0x7fcfd81561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd6ffd640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd80ce950 0x7fcfd81561e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd6ffd640 1 -- 172.21.15.59:0/3052803541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd8157dd0 con 0x7fcfd808c190 2024-06-29T15:58:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd67fc640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd80b8be0 0x7fcfd8155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfd6ffd640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd808c190 0x7fcfd81592e0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800c780 tx=0x7fcfc800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfbf7fe640 1 -- 172.21.15.59:0/3052803541 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8003b70 con 0x7fcfd808c190 2024-06-29T15:58:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfbf7fe640 1 -- 172.21.15.59:0/3052803541 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfc8003d10 con 0x7fcfd808c190 2024-06-29T15:58:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd8158240 con 0x7fcfd808c190 2024-06-29T15:58:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfbf7fe640 1 -- 172.21.15.59:0/3052803541 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc800aab0 con 0x7fcfd808c190 2024-06-29T15:58:36.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.791+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd8158800 con 0x7fcfd808c190 2024-06-29T15:58:36.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.792+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd80cf080 con 0x7fcfd808c190 2024-06-29T15:58:36.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.793+0000 7fcfbf7fe640 1 -- 172.21.15.59:0/3052803541 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fcfc800ac50 con 0x7fcfd808c190 2024-06-29T15:58:36.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.794+0000 7fcfbf7fe640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc4076540 0x7fcfc4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.794+0000 7fcfbf7fe640 1 -- 172.21.15.59:0/3052803541 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7fcfc8097fe0 con 0x7fcfd808c190 2024-06-29T15:58:36.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.794+0000 7fcfd67fc640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc4076540 0x7fcfc4078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.794+0000 7fcfd67fc640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc4076540 0x7fcfc4078a50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8161c30 tx=0x7fcfcc009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:36.798+0000 7fcfbf7fe640 1 -- 172.21.15.59:0/3052803541 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcfc8060f90 con 0x7fcfd808c190 2024-06-29T15:58:37.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.029+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fcfd80cdf50 con 0x7fcfd808c190 2024-06-29T15:58:37.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.649+0000 7fcfbf7fe640 1 -- 172.21.15.59:0/3052803541 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v137) ==== 125+0+0 (secure 0 0 0) 0x7fcfc8060890 con 0x7fcfd808c190 2024-06-29T15:58:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc4076540 msgr2=0x7fcfc4078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc4076540 0x7fcfc4078a50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8161c30 tx=0x7fcfcc009040 comp rx=0 tx=0).stop 2024-06-29T15:58:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd808c190 msgr2=0x7fcfd81592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd808c190 0x7fcfd81592e0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800c780 tx=0x7fcfc800cc40 comp rx=0 tx=0).stop 2024-06-29T15:58:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 shutdown_connections 2024-06-29T15:58:37.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fcfc4076540 0x7fcfc4078a50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfd80ce950 0x7fcfd81561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcfd80b8be0 0x7fcfd8155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 --2- 172.21.15.59:0/3052803541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd808c190 0x7fcfd81592e0 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 >> 172.21.15.59:0/3052803541 conn(0x7fcfd80b5610 msgr2=0x7fcfd806d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:37.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 shutdown_connections 2024-06-29T15:58:37.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.652+0000 7fcfdd4ac640 1 -- 172.21.15.59:0/3052803541 wait complete. 2024-06-29T15:58:37.667 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-06-29T15:58:37.668 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.791+0000 7f3bb645e640 1 -- 172.21.15.59:0/1013195220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb015ee10 msgr2=0x7f3bb015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.792+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1013195220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb015ee10 0x7f3bb015f9b0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8005fd0 tx=0x7f3ba802aa90 comp rx=0 tx=0).stop 2024-06-29T15:58:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.792+0000 7f3bb645e640 1 -- 172.21.15.59:0/1013195220 shutdown_connections 2024-06-29T15:58:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.792+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1013195220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb015ee10 0x7f3bb015f9b0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.792+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1013195220 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bb00b40a0 0x7f3bb015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.792+0000 7f3bb645e640 1 -- 172.21.15.59:0/1013195220 >> 172.21.15.59:0/1013195220 conn(0x7f3bb00d3ee0 msgr2=0x7f3bb00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.792+0000 7f3bb645e640 1 -- 172.21.15.59:0/1013195220 shutdown_connections 2024-06-29T15:58:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.792+0000 7f3bb645e640 1 -- 172.21.15.59:0/1013195220 wait complete. 2024-06-29T15:58:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.793+0000 7f3bb645e640 1 Processor -- start 2024-06-29T15:58:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.794+0000 7f3bb645e640 1 -- start start 2024-06-29T15:58:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb645e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bb00d23c0 0x7f3bb014f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb645e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bb015ee10 0x7f3bb0159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb645e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb0151a80 0x7f3bb0151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb645e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb0160280 con 0x7f3bb0151a80 2024-06-29T15:58:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb645e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb0160100 con 0x7f3bb00d23c0 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb645e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb0160400 con 0x7f3bb015ee10 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb49fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb0151a80 0x7f3bb0151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb49fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb0151a80 0x7f3bb0151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50162/0 (socket says 172.21.15.59:50162) 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb49fb640 1 -- 172.21.15.59:0/1834208034 learned_addr learned my addr 172.21.15.59:0/1834208034 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3baf7fe640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bb015ee10 0x7f3bb0159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3baffff640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bb00d23c0 0x7f3bb014f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb49fb640 1 -- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bb015ee10 msgr2=0x7f3bb0159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb49fb640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bb015ee10 0x7f3bb0159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb49fb640 1 -- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bb00d23c0 msgr2=0x7f3bb014f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.795+0000 7f3bb49fb640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bb00d23c0 0x7f3bb014f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.796+0000 7f3bb49fb640 1 -- 172.21.15.59:0/1834208034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb015a290 con 0x7f3bb0151a80 2024-06-29T15:58:37.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.796+0000 7f3baffff640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bb00d23c0 0x7f3bb014f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:37.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.796+0000 7f3bb49fb640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb0151a80 0x7f3bb0151e90 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009d90 tx=0x7f3ba8003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:37.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.796+0000 7f3bacff9640 1 -- 172.21.15.59:0/1834208034 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8044070 con 0x7f3bb0151a80 2024-06-29T15:58:37.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.796+0000 7f3bacff9640 1 -- 172.21.15.59:0/1834208034 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ba8004250 con 0x7f3bb0151a80 2024-06-29T15:58:37.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.796+0000 7f3bacff9640 1 -- 172.21.15.59:0/1834208034 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8004660 con 0x7f3bb0151a80 2024-06-29T15:58:37.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.796+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb01523d0 con 0x7f3bb0151a80 2024-06-29T15:58:37.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.796+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb0159fb0 con 0x7f3bb0151a80 2024-06-29T15:58:37.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.797+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bb014f910 con 0x7f3bb0151a80 2024-06-29T15:58:37.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.798+0000 7f3bacff9640 1 -- 172.21.15.59:0/1834208034 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f3ba8047070 con 0x7f3bb0151a80 2024-06-29T15:58:37.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.798+0000 7f3bacff9640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3b9c076510 0x7f3b9c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:37.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.798+0000 7f3bacff9640 1 -- 172.21.15.59:0/1834208034 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f3ba80b92d0 con 0x7f3bb0151a80 2024-06-29T15:58:37.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.799+0000 7f3baffff640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3b9c076510 0x7f3b9c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:37.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.799+0000 7f3baffff640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3b9c076510 0x7f3b9c078a20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3bb00d2e40 tx=0x7f3ba0009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:37.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:37.803+0000 7f3bacff9640 1 -- 172.21.15.59:0/1834208034 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3bb014f910 con 0x7f3bb0151a80 2024-06-29T15:58:38.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.038+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bb015e8d0 con 0x7f3bb0151a80 2024-06-29T15:58:38.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.039+0000 7f3bacff9640 1 -- 172.21.15.59:0/1834208034 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6568 (secure 0 0 0) 0x7f3bb01523d0 con 0x7f3bb0151a80 2024-06-29T15:58:38.040 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:38.040 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":137,"btime":"2024-06-29T15:58:37:645635+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:20.935697+0000","modified":"2024-06-29T15:58:23.956772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/177893499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":177893499},{"type":"v1","addr":"172.21.15.153:6837","nonce":177893499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12318,"qdb_cluster":[12318]},"id":15},{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:58:38.040 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 137 2024-06-29T15:58:38.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3b9c076510 msgr2=0x7f3b9c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3b9c076510 0x7f3b9c078a20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3bb00d2e40 tx=0x7f3ba0009040 comp rx=0 tx=0).stop 2024-06-29T15:58:38.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb0151a80 msgr2=0x7f3bb0151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb0151a80 0x7f3bb0151e90 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009d90 tx=0x7f3ba8003d00 comp rx=0 tx=0).stop 2024-06-29T15:58:38.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 shutdown_connections 2024-06-29T15:58:38.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f3b9c076510 0x7f3b9c078a20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bb0151a80 0x7f3bb0151e90 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3bb015ee10 0x7f3bb0159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 --2- 172.21.15.59:0/1834208034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3bb00d23c0 0x7f3bb014f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 >> 172.21.15.59:0/1834208034 conn(0x7f3bb00d3ee0 msgr2=0x7f3bb00b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 shutdown_connections 2024-06-29T15:58:38.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.041+0000 7f3bb645e640 1 -- 172.21.15.59:0/1834208034 wait complete. 2024-06-29T15:58:38.053 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:38.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.169+0000 7f4c5c2df640 1 -- 172.21.15.59:0/398213982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 msgr2=0x7f4c540d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.169+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/398213982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 0x7f4c540d1a20 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009830 tx=0x7f4c4402a510 comp rx=0 tx=0).stop 2024-06-29T15:58:38.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.170+0000 7f4c5c2df640 1 -- 172.21.15.59:0/398213982 shutdown_connections 2024-06-29T15:58:38.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.170+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/398213982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 0x7f4c540d1a20 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.170+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/398213982 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c5408c190 0x7f4c540cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.170+0000 7f4c5c2df640 1 -- 172.21.15.59:0/398213982 >> 172.21.15.59:0/398213982 conn(0x7f4c540b5610 msgr2=0x7f4c540b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.170+0000 7f4c5c2df640 1 -- 172.21.15.59:0/398213982 shutdown_connections 2024-06-29T15:58:38.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.170+0000 7f4c5c2df640 1 -- 172.21.15.59:0/398213982 wait complete. 2024-06-29T15:58:38.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.171+0000 7f4c5c2df640 1 Processor -- start 2024-06-29T15:58:38.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.171+0000 7f4c5c2df640 1 -- start start 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5c2df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c5408c190 0x7f4c54151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5c2df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c540ce950 0x7f4c54161880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5c2df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 0x7f4c541f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5c2df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4c540b7fa0 con 0x7f4c540cf5c0 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5c2df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c540b7e20 con 0x7f4c5408c190 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5c2df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4c540b8120 con 0x7f4c540ce950 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 0x7f4c541f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 0x7f4c541f8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50172/0 (socket says 172.21.15.59:50172) 2024-06-29T15:58:38.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 -- 172.21.15.59:0/893507482 learned_addr learned my addr 172.21.15.59:0/893507482 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a07b640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c5408c190 0x7f4c54151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5987a640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c540ce950 0x7f4c54161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 -- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c540ce950 msgr2=0x7f4c54161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c540ce950 0x7f4c54161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 -- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c5408c190 msgr2=0x7f4c54151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c5408c190 0x7f4c54151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 -- 172.21.15.59:0/893507482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c54152d80 con 0x7f4c540cf5c0 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.173+0000 7f4c5a87c640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 0x7f4c541f8280 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00cab0 tx=0x7f4c4c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.174+0000 7f4c4affd640 1 -- 172.21.15.59:0/893507482 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c016020 con 0x7f4c540cf5c0 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.174+0000 7f4c4affd640 1 -- 172.21.15.59:0/893507482 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c4c00ad40 con 0x7f4c540cf5c0 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.174+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c540cb710 con 0x7f4c540cf5c0 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.174+0000 7f4c4affd640 1 -- 172.21.15.59:0/893507482 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c012680 con 0x7f4c540cf5c0 2024-06-29T15:58:38.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.174+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c54153300 con 0x7f4c540cf5c0 2024-06-29T15:58:38.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.175+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c540cf080 con 0x7f4c540cf5c0 2024-06-29T15:58:38.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.175+0000 7f4c4affd640 1 -- 172.21.15.59:0/893507482 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4c4c019020 con 0x7f4c540cf5c0 2024-06-29T15:58:38.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.176+0000 7f4c4affd640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4c38076510 0x7f4c38078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.176+0000 7f4c4affd640 1 -- 172.21.15.59:0/893507482 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f4c4c0983b0 con 0x7f4c540cf5c0 2024-06-29T15:58:38.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.176+0000 7f4c5a07b640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4c38076510 0x7f4c38078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.177+0000 7f4c5a07b640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4c38076510 0x7f4c38078a20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000bb50 tx=0x7f4c5000b2e0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:38.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.181+0000 7f4c4affd640 1 -- 172.21.15.59:0/893507482 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4c4c020030 con 0x7f4c540cf5c0 2024-06-29T15:58:38.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.371+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c54074600 con 0x7f4c540cf5c0 2024-06-29T15:58:38.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.372+0000 7f4c4affd640 1 -- 172.21.15.59:0/893507482 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) ==== 74+0+20780 (secure 0 0 0) 0x7f4c4c069400 con 0x7f4c540cf5c0 2024-06-29T15:58:38.374 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:38.374 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":246,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:35.615155+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.374+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4c38076510 msgr2=0x7f4c38078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.374+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4c38076510 0x7f4c38078a20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000bb50 tx=0x7f4c5000b2e0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 msgr2=0x7f4c541f8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 0x7f4c541f8280 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00cab0 tx=0x7f4c4c00cf70 comp rx=0 tx=0).stop 2024-06-29T15:58:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 shutdown_connections 2024-06-29T15:58:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4c38076510 0x7f4c38078a20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c540cf5c0 0x7f4c541f8280 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4c540ce950 0x7f4c54161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 --2- 172.21.15.59:0/893507482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c5408c190 0x7f4c54151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 >> 172.21.15.59:0/893507482 conn(0x7f4c540b5610 msgr2=0x7f4c540c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 shutdown_connections 2024-06-29T15:58:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.375+0000 7f4c5c2df640 1 -- 172.21.15.59:0/893507482 wait complete. 2024-06-29T15:58:38.387 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:58:38.388 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:58:38.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.504+0000 7f5b37489640 1 -- 172.21.15.59:0/2189510507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 msgr2=0x7f5b300b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.504+0000 7f5b37489640 1 --2- 172.21.15.59:0/2189510507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 0x7f5b300b2bd0 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f5b28005fd0 tx=0x7f5b2802ab40 comp rx=0 tx=0).stop 2024-06-29T15:58:38.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.504+0000 7f5b37489640 1 -- 172.21.15.59:0/2189510507 shutdown_connections 2024-06-29T15:58:38.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.504+0000 7f5b37489640 1 --2- 172.21.15.59:0/2189510507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 0x7f5b300b2bd0 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.504+0000 7f5b37489640 1 -- 172.21.15.59:0/2189510507 >> 172.21.15.59:0/2189510507 conn(0x7f5b300d3ec0 msgr2=0x7f5b300d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.504+0000 7f5b37489640 1 -- 172.21.15.59:0/2189510507 shutdown_connections 2024-06-29T15:58:38.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.504+0000 7f5b37489640 1 -- 172.21.15.59:0/2189510507 wait complete. 2024-06-29T15:58:38.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.506+0000 7f5b37489640 1 Processor -- start 2024-06-29T15:58:38.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.506+0000 7f5b37489640 1 -- start start 2024-06-29T15:58:38.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.507+0000 7f5b37489640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b300b27c0 0x7f5b3015edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.507+0000 7f5b37489640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 0x7f5b3014e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.507+0000 7f5b37489640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b300cd840 0x7f5b30156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.507+0000 7f5b37489640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301694c0 con 0x7f5b300d32b0 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.507+0000 7f5b37489640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b30169340 con 0x7f5b300b27c0 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.507+0000 7f5b37489640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5b30169640 con 0x7f5b300cd840 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.507+0000 7f5b34a24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 0x7f5b3014e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b35225640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b300b27c0 0x7f5b3015edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b34a24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 0x7f5b3014e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50188/0 (socket says 172.21.15.59:50188) 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b34a24640 1 -- 172.21.15.59:0/1197251391 learned_addr learned my addr 172.21.15.59:0/1197251391 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b35a26640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b300cd840 0x7f5b30156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b34a24640 1 -- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b300cd840 msgr2=0x7f5b30156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b34a24640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b300cd840 0x7f5b30156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b34a24640 1 -- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b300b27c0 msgr2=0x7f5b3015edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b34a24640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b300b27c0 0x7f5b3015edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b34a24640 1 -- 172.21.15.59:0/1197251391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b30155b10 con 0x7f5b300d32b0 2024-06-29T15:58:38.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b34a24640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 0x7f5b3014e100 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000c400 tx=0x7f5b2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:38.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b25ffb640 1 -- 172.21.15.59:0/1197251391 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20017070 con 0x7f5b300d32b0 2024-06-29T15:58:38.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b25ffb640 1 -- 172.21.15.59:0/1197251391 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b200053c0 con 0x7f5b300d32b0 2024-06-29T15:58:38.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b30155df0 con 0x7f5b300d32b0 2024-06-29T15:58:38.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.508+0000 7f5b25ffb640 1 -- 172.21.15.59:0/1197251391 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20003850 con 0x7f5b300d32b0 2024-06-29T15:58:38.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.509+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b301563b0 con 0x7f5b300d32b0 2024-06-29T15:58:38.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.510+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b300b2bd0 con 0x7f5b300d32b0 2024-06-29T15:58:38.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.510+0000 7f5b25ffb640 1 -- 172.21.15.59:0/1197251391 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f5b2001a070 con 0x7f5b300d32b0 2024-06-29T15:58:38.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.511+0000 7f5b25ffb640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5b14076540 0x7f5b14078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.511+0000 7f5b25ffb640 1 -- 172.21.15.59:0/1197251391 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f5b20097480 con 0x7f5b300d32b0 2024-06-29T15:58:38.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.511+0000 7f5b35225640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5b14076540 0x7f5b14078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.511+0000 7f5b35225640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5b14076540 0x7f5b14078a50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c000f60 tx=0x7f5b2c009040 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:38.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.515+0000 7f5b25ffb640 1 -- 172.21.15.59:0/1197251391 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5b2005feb0 con 0x7f5b300d32b0 2024-06-29T15:58:38.522 INFO:tasks.ceph.mds.c.smithi059.stderr:2024-06-29T15:58:38.521+0000 7f98c9af1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.708+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 --> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5b301558c0 con 0x7f5b14076540 2024-06-29T15:58:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.716+0000 7f5b25ffb640 1 -- 172.21.15.59:0/1197251391 <== mgr.9710 v2:172.21.15.153:6832/1831136380 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491413 (secure 0 0 0) 0x7f5b301558c0 con 0x7f5b14076540 2024-06-29T15:58:38.722 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:58:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.720+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5b14076540 msgr2=0x7f5b14078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.720+0000 7f5b37489640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5b14076540 0x7f5b14078a50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c000f60 tx=0x7f5b2c009040 comp rx=0 tx=0).stop 2024-06-29T15:58:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.720+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 msgr2=0x7f5b3014e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 0x7f5b3014e100 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000c400 tx=0x7f5b2000c8c0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 shutdown_connections 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f5b14076540 0x7f5b14078a50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5b300cd840 0x7f5b30156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b300d32b0 0x7f5b3014e100 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 --2- 172.21.15.59:0/1197251391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b300b27c0 0x7f5b3015edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 >> 172.21.15.59:0/1197251391 conn(0x7f5b300d3ec0 msgr2=0x7f5b3006d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 shutdown_connections 2024-06-29T15:58:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.721+0000 7f5b37489640 1 -- 172.21.15.59:0/1197251391 wait complete. 2024-06-29T15:58:38.733 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-06-29T15:58:37.208753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":492,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":449672,"kb_used_data":25300,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754525048,"statfs":{"total":773094113280,"available":772633649152,"internally_reserved":0,"allocated":25907200,"data_stored":13935967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":434425394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.601083"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.641640+0000","last_change":"2024-06-29T15:58:34.638268+0000","last_active":"2024-06-29T15:58:34.641640+0000","last_peered":"2024-06-29T15:58:34.641640+0000","last_clean":"2024-06-29T15:58:34.641640+0000","last_became_active":"2024-06-29T15:58:34.637938+0000","last_became_peered":"2024-06-29T15:58:34.637938+0000","last_unstale":"2024-06-29T15:58:34.641640+0000","last_undegraded":"2024-06-29T15:58:34.641640+0000","last_fullsized":"2024-06-29T15:58:34.641640+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:43.776145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.648895+0000","last_change":"2024-06-29T15:58:33.632076+0000","last_active":"2024-06-29T15:58:34.648895+0000","last_peered":"2024-06-29T15:58:34.648895+0000","last_clean":"2024-06-29T15:58:34.648895+0000","last_became_active":"2024-06-29T15:58:33.631677+0000","last_became_peered":"2024-06-29T15:58:33.631677+0000","last_unstale":"2024-06-29T15:58:34.648895+0000","last_undegraded":"2024-06-29T15:58:34.648895+0000","last_fullsized":"2024-06-29T15:58:34.648895+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:52:51.054347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629994+0000","last_change":"2024-06-29T15:58:21.556650+0000","last_active":"2024-06-29T15:58:34.629994+0000","last_peered":"2024-06-29T15:58:34.629994+0000","last_clean":"2024-06-29T15:58:34.629994+0000","last_became_active":"2024-06-29T15:58:21.556321+0000","last_became_peered":"2024-06-29T15:58:21.556321+0000","last_unstale":"2024-06-29T15:58:34.629994+0000","last_undegraded":"2024-06-29T15:58:34.629994+0000","last_fullsized":"2024-06-29T15:58:34.629994+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:40:17.694204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.639846+0000","last_change":"2024-06-29T15:58:34.635358+0000","last_active":"2024-06-29T15:58:34.639846+0000","last_peered":"2024-06-29T15:58:34.639846+0000","last_clean":"2024-06-29T15:58:34.639846+0000","last_became_active":"2024-06-29T15:58:34.635052+0000","last_became_peered":"2024-06-29T15:58:34.635052+0000","last_unstale":"2024-06-29T15:58:34.639846+0000","last_undegraded":"2024-06-29T15:58:34.639846+0000","last_fullsized":"2024-06-29T15:58:34.639846+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:13.804732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"245'4","reported_seq":16,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.657305+0000","last_change":"2024-06-29T15:58:33.637644+0000","last_active":"2024-06-29T15:58:34.657305+0000","last_peered":"2024-06-29T15:58:34.657305+0000","last_clean":"2024-06-29T15:58:34.657305+0000","last_became_active":"2024-06-29T15:58:33.636791+0000","last_became_peered":"2024-06-29T15:58:33.636791+0000","last_unstale":"2024-06-29T15:58:34.657305+0000","last_undegraded":"2024-06-29T15:58:34.657305+0000","last_fullsized":"2024-06-29T15:58:34.657305+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:29.799985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631686+0000","last_change":"2024-06-29T15:58:21.553068+0000","last_active":"2024-06-29T15:58:34.631686+0000","last_peered":"2024-06-29T15:58:34.631686+0000","last_clean":"2024-06-29T15:58:34.631686+0000","last_became_active":"2024-06-29T15:58:21.552631+0000","last_became_peered":"2024-06-29T15:58:21.552631+0000","last_unstale":"2024-06-29T15:58:34.631686+0000","last_undegraded":"2024-06-29T15:58:34.631686+0000","last_fullsized":"2024-06-29T15:58:34.631686+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:18.380041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659317+0000","last_change":"2024-06-29T15:58:34.656858+0000","last_active":"2024-06-29T15:58:34.659317+0000","last_peered":"2024-06-29T15:58:34.659317+0000","last_clean":"2024-06-29T15:58:34.659317+0000","last_became_active":"2024-06-29T15:58:34.656529+0000","last_became_peered":"2024-06-29T15:58:34.656529+0000","last_unstale":"2024-06-29T15:58:34.659317+0000","last_undegraded":"2024-06-29T15:58:34.659317+0000","last_fullsized":"2024-06-29T15:58:34.659317+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:23:41.149736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.652787+0000","last_change":"2024-06-29T15:58:33.635736+0000","last_active":"2024-06-29T15:58:34.652787+0000","last_peered":"2024-06-29T15:58:34.652787+0000","last_clean":"2024-06-29T15:58:34.652787+0000","last_became_active":"2024-06-29T15:58:33.635382+0000","last_became_peered":"2024-06-29T15:58:33.635382+0000","last_unstale":"2024-06-29T15:58:34.652787+0000","last_undegraded":"2024-06-29T15:58:34.652787+0000","last_fullsized":"2024-06-29T15:58:34.652787+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:51.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619707+0000","last_change":"2024-06-29T15:58:21.553826+0000","last_active":"2024-06-29T15:58:34.619707+0000","last_peered":"2024-06-29T15:58:34.619707+0000","last_clean":"2024-06-29T15:58:34.619707+0000","last_became_active":"2024-06-29T15:58:21.553476+0000","last_became_peered":"2024-06-29T15:58:21.553476+0000","last_unstale":"2024-06-29T15:58:34.619707+0000","last_undegraded":"2024-06-29T15:58:34.619707+0000","last_fullsized":"2024-06-29T15:58:34.619707+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:09:27.869044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.657050+0000","last_change":"2024-06-29T15:58:34.655115+0000","last_active":"2024-06-29T15:58:34.657050+0000","last_peered":"2024-06-29T15:58:34.657050+0000","last_clean":"2024-06-29T15:58:34.657050+0000","last_became_active":"2024-06-29T15:58:34.654817+0000","last_became_peered":"2024-06-29T15:58:34.654817+0000","last_unstale":"2024-06-29T15:58:34.657050+0000","last_undegraded":"2024-06-29T15:58:34.657050+0000","last_fullsized":"2024-06-29T15:58:34.657050+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:12.560261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.661986+0000","last_change":"2024-06-29T15:58:33.643604+0000","last_active":"2024-06-29T15:58:34.661986+0000","last_peered":"2024-06-29T15:58:34.661986+0000","last_clean":"2024-06-29T15:58:34.661986+0000","last_became_active":"2024-06-29T15:58:33.643295+0000","last_became_peered":"2024-06-29T15:58:33.643295+0000","last_unstale":"2024-06-29T15:58:34.661986+0000","last_undegraded":"2024-06-29T15:58:34.661986+0000","last_fullsized":"2024-06-29T15:58:34.661986+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:45.601470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619152+0000","last_change":"2024-06-29T15:58:21.556754+0000","last_active":"2024-06-29T15:58:34.619152+0000","last_peered":"2024-06-29T15:58:34.619152+0000","last_clean":"2024-06-29T15:58:34.619152+0000","last_became_active":"2024-06-29T15:58:21.556399+0000","last_became_peered":"2024-06-29T15:58:21.556399+0000","last_unstale":"2024-06-29T15:58:34.619152+0000","last_undegraded":"2024-06-29T15:58:34.619152+0000","last_fullsized":"2024-06-29T15:58:34.619152+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:53:06.570549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.642247+0000","last_change":"2024-06-29T15:58:34.637540+0000","last_active":"2024-06-29T15:58:34.642247+0000","last_peered":"2024-06-29T15:58:34.642247+0000","last_clean":"2024-06-29T15:58:34.642247+0000","last_became_active":"2024-06-29T15:58:34.637206+0000","last_became_peered":"2024-06-29T15:58:34.637206+0000","last_unstale":"2024-06-29T15:58:34.642247+0000","last_undegraded":"2024-06-29T15:58:34.642247+0000","last_fullsized":"2024-06-29T15:58:34.642247+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:54.848045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.656343+0000","last_change":"2024-06-29T15:58:33.646871+0000","last_active":"2024-06-29T15:58:34.656343+0000","last_peered":"2024-06-29T15:58:34.656343+0000","last_clean":"2024-06-29T15:58:34.656343+0000","last_became_active":"2024-06-29T15:58:33.646553+0000","last_became_peered":"2024-06-29T15:58:33.646553+0000","last_unstale":"2024-06-29T15:58:34.656343+0000","last_undegraded":"2024-06-29T15:58:34.656343+0000","last_fullsized":"2024-06-29T15:58:34.656343+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:05:30.320148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621972+0000","last_change":"2024-06-29T15:58:21.558986+0000","last_active":"2024-06-29T15:58:34.621972+0000","last_peered":"2024-06-29T15:58:34.621972+0000","last_clean":"2024-06-29T15:58:34.621972+0000","last_became_active":"2024-06-29T15:58:21.558669+0000","last_became_peered":"2024-06-29T15:58:21.558669+0000","last_unstale":"2024-06-29T15:58:34.621972+0000","last_undegraded":"2024-06-29T15:58:34.621972+0000","last_fullsized":"2024-06-29T15:58:34.621972+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:07.747196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658916+0000","last_change":"2024-06-29T15:58:34.643734+0000","last_active":"2024-06-29T15:58:34.658916+0000","last_peered":"2024-06-29T15:58:34.658916+0000","last_clean":"2024-06-29T15:58:34.658916+0000","last_became_active":"2024-06-29T15:58:34.643426+0000","last_became_peered":"2024-06-29T15:58:34.643426+0000","last_unstale":"2024-06-29T15:58:34.658916+0000","last_undegraded":"2024-06-29T15:58:34.658916+0000","last_fullsized":"2024-06-29T15:58:34.658916+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:55.258468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.647516+0000","last_change":"2024-06-29T15:58:33.638695+0000","last_active":"2024-06-29T15:58:34.647516+0000","last_peered":"2024-06-29T15:58:34.647516+0000","last_clean":"2024-06-29T15:58:34.647516+0000","last_became_active":"2024-06-29T15:58:33.638335+0000","last_became_peered":"2024-06-29T15:58:33.638335+0000","last_unstale":"2024-06-29T15:58:34.647516+0000","last_undegraded":"2024-06-29T15:58:34.647516+0000","last_fullsized":"2024-06-29T15:58:34.647516+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:46.273967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630986+0000","last_change":"2024-06-29T15:58:21.558888+0000","last_active":"2024-06-29T15:58:34.630986+0000","last_peered":"2024-06-29T15:58:34.630986+0000","last_clean":"2024-06-29T15:58:34.630986+0000","last_became_active":"2024-06-29T15:58:21.558578+0000","last_became_peered":"2024-06-29T15:58:21.558578+0000","last_unstale":"2024-06-29T15:58:34.630986+0000","last_undegraded":"2024-06-29T15:58:34.630986+0000","last_fullsized":"2024-06-29T15:58:34.630986+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:30.612798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651912+0000","last_change":"2024-06-29T15:58:34.642968+0000","last_active":"2024-06-29T15:58:34.651912+0000","last_peered":"2024-06-29T15:58:34.651912+0000","last_clean":"2024-06-29T15:58:34.651912+0000","last_became_active":"2024-06-29T15:58:34.642120+0000","last_became_peered":"2024-06-29T15:58:34.642120+0000","last_unstale":"2024-06-29T15:58:34.651912+0000","last_undegraded":"2024-06-29T15:58:34.651912+0000","last_fullsized":"2024-06-29T15:58:34.651912+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:07.025026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618528+0000","last_change":"2024-06-29T15:58:33.636596+0000","last_active":"2024-06-29T15:58:34.618528+0000","last_peered":"2024-06-29T15:58:34.618528+0000","last_clean":"2024-06-29T15:58:34.618528+0000","last_became_active":"2024-06-29T15:58:33.636251+0000","last_became_peered":"2024-06-29T15:58:33.636251+0000","last_unstale":"2024-06-29T15:58:34.618528+0000","last_undegraded":"2024-06-29T15:58:34.618528+0000","last_fullsized":"2024-06-29T15:58:34.618528+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:50.037512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630803+0000","last_change":"2024-06-29T15:58:21.552220+0000","last_active":"2024-06-29T15:58:34.630803+0000","last_peered":"2024-06-29T15:58:34.630803+0000","last_clean":"2024-06-29T15:58:34.630803+0000","last_became_active":"2024-06-29T15:58:21.551911+0000","last_became_peered":"2024-06-29T15:58:21.551911+0000","last_unstale":"2024-06-29T15:58:34.630803+0000","last_undegraded":"2024-06-29T15:58:34.630803+0000","last_fullsized":"2024-06-29T15:58:34.630803+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:20.005949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662613+0000","last_change":"2024-06-29T15:58:34.662000+0000","last_active":"2024-06-29T15:58:34.662613+0000","last_peered":"2024-06-29T15:58:34.662613+0000","last_clean":"2024-06-29T15:58:34.662613+0000","last_became_active":"2024-06-29T15:58:34.661659+0000","last_became_peered":"2024-06-29T15:58:34.661659+0000","last_unstale":"2024-06-29T15:58:34.662613+0000","last_undegraded":"2024-06-29T15:58:34.662613+0000","last_fullsized":"2024-06-29T15:58:34.662613+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:07:15.822268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621645+0000","last_change":"2024-06-29T15:58:33.623668+0000","last_active":"2024-06-29T15:58:34.621645+0000","last_peered":"2024-06-29T15:58:34.621645+0000","last_clean":"2024-06-29T15:58:34.621645+0000","last_became_active":"2024-06-29T15:58:33.623328+0000","last_became_peered":"2024-06-29T15:58:33.623328+0000","last_unstale":"2024-06-29T15:58:34.621645+0000","last_undegraded":"2024-06-29T15:58:34.621645+0000","last_fullsized":"2024-06-29T15:58:34.621645+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:47.691623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622335+0000","last_change":"2024-06-29T15:58:21.558631+0000","last_active":"2024-06-29T15:58:34.622335+0000","last_peered":"2024-06-29T15:58:34.622335+0000","last_clean":"2024-06-29T15:58:34.622335+0000","last_became_active":"2024-06-29T15:58:21.558300+0000","last_became_peered":"2024-06-29T15:58:21.558300+0000","last_unstale":"2024-06-29T15:58:34.622335+0000","last_undegraded":"2024-06-29T15:58:34.622335+0000","last_fullsized":"2024-06-29T15:58:34.622335+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:46.044647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.656459+0000","last_change":"2024-06-29T15:58:34.649980+0000","last_active":"2024-06-29T15:58:34.656459+0000","last_peered":"2024-06-29T15:58:34.656459+0000","last_clean":"2024-06-29T15:58:34.656459+0000","last_became_active":"2024-06-29T15:58:34.649594+0000","last_became_peered":"2024-06-29T15:58:34.649594+0000","last_unstale":"2024-06-29T15:58:34.656459+0000","last_undegraded":"2024-06-29T15:58:34.656459+0000","last_fullsized":"2024-06-29T15:58:34.656459+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:29.253507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622685+0000","last_change":"2024-06-29T15:58:33.629024+0000","last_active":"2024-06-29T15:58:34.622685+0000","last_peered":"2024-06-29T15:58:34.622685+0000","last_clean":"2024-06-29T15:58:34.622685+0000","last_became_active":"2024-06-29T15:58:33.622998+0000","last_became_peered":"2024-06-29T15:58:33.622998+0000","last_unstale":"2024-06-29T15:58:34.622685+0000","last_undegraded":"2024-06-29T15:58:34.622685+0000","last_fullsized":"2024-06-29T15:58:34.622685+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:01.088268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630286+0000","last_change":"2024-06-29T15:58:21.560160+0000","last_active":"2024-06-29T15:58:34.630286+0000","last_peered":"2024-06-29T15:58:34.630286+0000","last_clean":"2024-06-29T15:58:34.630286+0000","last_became_active":"2024-06-29T15:58:21.559735+0000","last_became_peered":"2024-06-29T15:58:21.559735+0000","last_unstale":"2024-06-29T15:58:34.630286+0000","last_undegraded":"2024-06-29T15:58:34.630286+0000","last_fullsized":"2024-06-29T15:58:34.630286+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:33.429459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.652072+0000","last_change":"2024-06-29T15:58:34.643328+0000","last_active":"2024-06-29T15:58:34.652072+0000","last_peered":"2024-06-29T15:58:34.652072+0000","last_clean":"2024-06-29T15:58:34.652072+0000","last_became_active":"2024-06-29T15:58:34.639700+0000","last_became_peered":"2024-06-29T15:58:34.639700+0000","last_unstale":"2024-06-29T15:58:34.652072+0000","last_undegraded":"2024-06-29T15:58:34.652072+0000","last_fullsized":"2024-06-29T15:58:34.652072+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:56.133749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629630+0000","last_change":"2024-06-29T15:58:33.641952+0000","last_active":"2024-06-29T15:58:34.629630+0000","last_peered":"2024-06-29T15:58:34.629630+0000","last_clean":"2024-06-29T15:58:34.629630+0000","last_became_active":"2024-06-29T15:58:33.640950+0000","last_became_peered":"2024-06-29T15:58:33.640950+0000","last_unstale":"2024-06-29T15:58:34.629630+0000","last_undegraded":"2024-06-29T15:58:34.629630+0000","last_fullsized":"2024-06-29T15:58:34.629630+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:20:47.201871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619463+0000","last_change":"2024-06-29T15:58:21.559778+0000","last_active":"2024-06-29T15:58:34.619463+0000","last_peered":"2024-06-29T15:58:34.619463+0000","last_clean":"2024-06-29T15:58:34.619463+0000","last_became_active":"2024-06-29T15:58:21.559442+0000","last_became_peered":"2024-06-29T15:58:21.559442+0000","last_unstale":"2024-06-29T15:58:34.619463+0000","last_undegraded":"2024-06-29T15:58:34.619463+0000","last_fullsized":"2024-06-29T15:58:34.619463+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:14.817280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632388+0000","last_change":"2024-06-29T15:58:34.631762+0000","last_active":"2024-06-29T15:58:34.632388+0000","last_peered":"2024-06-29T15:58:34.632388+0000","last_clean":"2024-06-29T15:58:34.632388+0000","last_became_active":"2024-06-29T15:58:34.631450+0000","last_became_peered":"2024-06-29T15:58:34.631450+0000","last_unstale":"2024-06-29T15:58:34.632388+0000","last_undegraded":"2024-06-29T15:58:34.632388+0000","last_fullsized":"2024-06-29T15:58:34.632388+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:42.958627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"245'6","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662647+0000","last_change":"2024-06-29T15:58:33.636484+0000","last_active":"2024-06-29T15:58:34.662647+0000","last_peered":"2024-06-29T15:58:34.662647+0000","last_clean":"2024-06-29T15:58:34.662647+0000","last_became_active":"2024-06-29T15:58:33.635723+0000","last_became_peered":"2024-06-29T15:58:33.635723+0000","last_unstale":"2024-06-29T15:58:34.662647+0000","last_undegraded":"2024-06-29T15:58:34.662647+0000","last_fullsized":"2024-06-29T15:58:34.662647+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:24.142073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619842+0000","last_change":"2024-06-29T15:58:21.560672+0000","last_active":"2024-06-29T15:58:34.619842+0000","last_peered":"2024-06-29T15:58:34.619842+0000","last_clean":"2024-06-29T15:58:34.619842+0000","last_became_active":"2024-06-29T15:58:21.560333+0000","last_became_peered":"2024-06-29T15:58:21.560333+0000","last_unstale":"2024-06-29T15:58:34.619842+0000","last_undegraded":"2024-06-29T15:58:34.619842+0000","last_fullsized":"2024-06-29T15:58:34.619842+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:16.662908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651377+0000","last_change":"2024-06-29T15:58:34.650497+0000","last_active":"2024-06-29T15:58:34.651377+0000","last_peered":"2024-06-29T15:58:34.651377+0000","last_clean":"2024-06-29T15:58:34.651377+0000","last_became_active":"2024-06-29T15:58:34.650188+0000","last_became_peered":"2024-06-29T15:58:34.650188+0000","last_unstale":"2024-06-29T15:58:34.651377+0000","last_undegraded":"2024-06-29T15:58:34.651377+0000","last_fullsized":"2024-06-29T15:58:34.651377+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:56:51.215044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"245'3","reported_seq":15,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.660442+0000","last_change":"2024-06-29T15:58:33.641361+0000","last_active":"2024-06-29T15:58:34.660442+0000","last_peered":"2024-06-29T15:58:34.660442+0000","last_clean":"2024-06-29T15:58:34.660442+0000","last_became_active":"2024-06-29T15:58:33.641006+0000","last_became_peered":"2024-06-29T15:58:33.641006+0000","last_unstale":"2024-06-29T15:58:34.660442+0000","last_undegraded":"2024-06-29T15:58:34.660442+0000","last_fullsized":"2024-06-29T15:58:34.660442+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:47:15.436543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629008+0000","last_change":"2024-06-29T15:58:21.557848+0000","last_active":"2024-06-29T15:58:34.629008+0000","last_peered":"2024-06-29T15:58:34.629008+0000","last_clean":"2024-06-29T15:58:34.629008+0000","last_became_active":"2024-06-29T15:58:21.557516+0000","last_became_peered":"2024-06-29T15:58:21.557516+0000","last_unstale":"2024-06-29T15:58:34.629008+0000","last_undegraded":"2024-06-29T15:58:34.629008+0000","last_fullsized":"2024-06-29T15:58:34.629008+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:26.084704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646917+0000","last_change":"2024-06-29T15:58:34.645224+0000","last_active":"2024-06-29T15:58:34.646917+0000","last_peered":"2024-06-29T15:58:34.646917+0000","last_clean":"2024-06-29T15:58:34.646917+0000","last_became_active":"2024-06-29T15:58:34.644926+0000","last_became_peered":"2024-06-29T15:58:34.644926+0000","last_unstale":"2024-06-29T15:58:34.646917+0000","last_undegraded":"2024-06-29T15:58:34.646917+0000","last_fullsized":"2024-06-29T15:58:34.646917+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:12.560261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630668+0000","last_change":"2024-06-29T15:58:33.636773+0000","last_active":"2024-06-29T15:58:34.630668+0000","last_peered":"2024-06-29T15:58:34.630668+0000","last_clean":"2024-06-29T15:58:34.630668+0000","last_became_active":"2024-06-29T15:58:33.636476+0000","last_became_peered":"2024-06-29T15:58:33.636476+0000","last_unstale":"2024-06-29T15:58:34.630668+0000","last_undegraded":"2024-06-29T15:58:34.630668+0000","last_fullsized":"2024-06-29T15:58:34.630668+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:24.260348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627751+0000","last_change":"2024-06-29T15:58:21.557388+0000","last_active":"2024-06-29T15:58:34.627751+0000","last_peered":"2024-06-29T15:58:34.627751+0000","last_clean":"2024-06-29T15:58:34.627751+0000","last_became_active":"2024-06-29T15:58:21.557098+0000","last_became_peered":"2024-06-29T15:58:21.557098+0000","last_unstale":"2024-06-29T15:58:34.627751+0000","last_undegraded":"2024-06-29T15:58:34.627751+0000","last_fullsized":"2024-06-29T15:58:34.627751+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:27.148409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625828+0000","last_change":"2024-06-29T15:58:34.624723+0000","last_active":"2024-06-29T15:58:34.625828+0000","last_peered":"2024-06-29T15:58:34.625828+0000","last_clean":"2024-06-29T15:58:34.625828+0000","last_became_active":"2024-06-29T15:58:34.624331+0000","last_became_peered":"2024-06-29T15:58:34.624331+0000","last_unstale":"2024-06-29T15:58:34.625828+0000","last_undegraded":"2024-06-29T15:58:34.625828+0000","last_fullsized":"2024-06-29T15:58:34.625828+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:23:41.149736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649825+0000","last_change":"2024-06-29T15:58:33.638685+0000","last_active":"2024-06-29T15:58:34.649825+0000","last_peered":"2024-06-29T15:58:34.649825+0000","last_clean":"2024-06-29T15:58:34.649825+0000","last_became_active":"2024-06-29T15:58:33.638401+0000","last_became_peered":"2024-06-29T15:58:33.638401+0000","last_unstale":"2024-06-29T15:58:34.649825+0000","last_undegraded":"2024-06-29T15:58:34.649825+0000","last_fullsized":"2024-06-29T15:58:34.649825+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:11:50.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628010+0000","last_change":"2024-06-29T15:58:21.567227+0000","last_active":"2024-06-29T15:58:34.628010+0000","last_peered":"2024-06-29T15:58:34.628010+0000","last_clean":"2024-06-29T15:58:34.628010+0000","last_became_active":"2024-06-29T15:58:21.566195+0000","last_became_peered":"2024-06-29T15:58:21.566195+0000","last_unstale":"2024-06-29T15:58:34.628010+0000","last_undegraded":"2024-06-29T15:58:34.628010+0000","last_fullsized":"2024-06-29T15:58:34.628010+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:49.425921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651066+0000","last_change":"2024-06-29T15:58:34.649146+0000","last_active":"2024-06-29T15:58:34.651066+0000","last_peered":"2024-06-29T15:58:34.651066+0000","last_clean":"2024-06-29T15:58:34.651066+0000","last_became_active":"2024-06-29T15:58:34.648790+0000","last_became_peered":"2024-06-29T15:58:34.648790+0000","last_unstale":"2024-06-29T15:58:34.651066+0000","last_undegraded":"2024-06-29T15:58:34.651066+0000","last_fullsized":"2024-06-29T15:58:34.651066+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:48.494647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"245'8","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662440+0000","last_change":"2024-06-29T15:58:33.634549+0000","last_active":"2024-06-29T15:58:34.662440+0000","last_peered":"2024-06-29T15:58:34.662440+0000","last_clean":"2024-06-29T15:58:34.662440+0000","last_became_active":"2024-06-29T15:58:33.634233+0000","last_became_peered":"2024-06-29T15:58:33.634233+0000","last_unstale":"2024-06-29T15:58:34.662440+0000","last_undegraded":"2024-06-29T15:58:34.662440+0000","last_fullsized":"2024-06-29T15:58:34.662440+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:50:41.829436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628404+0000","last_change":"2024-06-29T15:58:21.562321+0000","last_active":"2024-06-29T15:58:34.628404+0000","last_peered":"2024-06-29T15:58:34.628404+0000","last_clean":"2024-06-29T15:58:34.628404+0000","last_became_active":"2024-06-29T15:58:21.561988+0000","last_became_peered":"2024-06-29T15:58:21.561988+0000","last_unstale":"2024-06-29T15:58:34.628404+0000","last_undegraded":"2024-06-29T15:58:34.628404+0000","last_fullsized":"2024-06-29T15:58:34.628404+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:27.579111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.635269+0000","last_change":"2024-06-29T15:58:34.633069+0000","last_active":"2024-06-29T15:58:34.635269+0000","last_peered":"2024-06-29T15:58:34.635269+0000","last_clean":"2024-06-29T15:58:34.635269+0000","last_became_active":"2024-06-29T15:58:34.631861+0000","last_became_peered":"2024-06-29T15:58:34.631861+0000","last_unstale":"2024-06-29T15:58:34.635269+0000","last_undegraded":"2024-06-29T15:58:34.635269+0000","last_fullsized":"2024-06-29T15:58:34.635269+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:33.087931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.624539+0000","last_change":"2024-06-29T15:58:33.633007+0000","last_active":"2024-06-29T15:58:34.624539+0000","last_peered":"2024-06-29T15:58:34.624539+0000","last_clean":"2024-06-29T15:58:34.624539+0000","last_became_active":"2024-06-29T15:58:33.632662+0000","last_became_peered":"2024-06-29T15:58:33.632662+0000","last_unstale":"2024-06-29T15:58:34.624539+0000","last_undegraded":"2024-06-29T15:58:34.624539+0000","last_fullsized":"2024-06-29T15:58:34.624539+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:48:00.558325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618608+0000","last_change":"2024-06-29T15:58:21.561992+0000","last_active":"2024-06-29T15:58:34.618608+0000","last_peered":"2024-06-29T15:58:34.618608+0000","last_clean":"2024-06-29T15:58:34.618608+0000","last_became_active":"2024-06-29T15:58:21.561586+0000","last_became_peered":"2024-06-29T15:58:21.561586+0000","last_unstale":"2024-06-29T15:58:34.618608+0000","last_undegraded":"2024-06-29T15:58:34.618608+0000","last_fullsized":"2024-06-29T15:58:34.618608+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:57.843559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.638627+0000","last_change":"2024-06-29T15:58:34.636779+0000","last_active":"2024-06-29T15:58:34.638627+0000","last_peered":"2024-06-29T15:58:34.638627+0000","last_clean":"2024-06-29T15:58:34.638627+0000","last_became_active":"2024-06-29T15:58:34.636451+0000","last_became_peered":"2024-06-29T15:58:34.636451+0000","last_unstale":"2024-06-29T15:58:34.638627+0000","last_undegraded":"2024-06-29T15:58:34.638627+0000","last_fullsized":"2024-06-29T15:58:34.638627+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:30.807221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.652506+0000","last_change":"2024-06-29T15:58:33.634810+0000","last_active":"2024-06-29T15:58:34.652506+0000","last_peered":"2024-06-29T15:58:34.652506+0000","last_clean":"2024-06-29T15:58:34.652506+0000","last_became_active":"2024-06-29T15:58:33.634402+0000","last_became_peered":"2024-06-29T15:58:33.634402+0000","last_unstale":"2024-06-29T15:58:34.652506+0000","last_undegraded":"2024-06-29T15:58:34.652506+0000","last_fullsized":"2024-06-29T15:58:34.652506+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:45.601470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629846+0000","last_change":"2024-06-29T15:58:21.559279+0000","last_active":"2024-06-29T15:58:34.629846+0000","last_peered":"2024-06-29T15:58:34.629846+0000","last_clean":"2024-06-29T15:58:34.629846+0000","last_became_active":"2024-06-29T15:58:21.559033+0000","last_became_peered":"2024-06-29T15:58:21.559033+0000","last_unstale":"2024-06-29T15:58:34.629846+0000","last_undegraded":"2024-06-29T15:58:34.629846+0000","last_fullsized":"2024-06-29T15:58:34.629846+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:19.231351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.641812+0000","last_change":"2024-06-29T15:58:34.640422+0000","last_active":"2024-06-29T15:58:34.641812+0000","last_peered":"2024-06-29T15:58:34.641812+0000","last_clean":"2024-06-29T15:58:34.641812+0000","last_became_active":"2024-06-29T15:58:34.640011+0000","last_became_peered":"2024-06-29T15:58:34.640011+0000","last_unstale":"2024-06-29T15:58:34.641812+0000","last_undegraded":"2024-06-29T15:58:34.641812+0000","last_fullsized":"2024-06-29T15:58:34.641812+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:43.776145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.657794+0000","last_change":"2024-06-29T15:58:33.639974+0000","last_active":"2024-06-29T15:58:34.657794+0000","last_peered":"2024-06-29T15:58:34.657794+0000","last_clean":"2024-06-29T15:58:34.657794+0000","last_became_active":"2024-06-29T15:58:33.639562+0000","last_became_peered":"2024-06-29T15:58:33.639562+0000","last_unstale":"2024-06-29T15:58:34.657794+0000","last_undegraded":"2024-06-29T15:58:34.657794+0000","last_fullsized":"2024-06-29T15:58:34.657794+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:14:59.418907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619590+0000","last_change":"2024-06-29T15:58:21.561849+0000","last_active":"2024-06-29T15:58:34.619590+0000","last_peered":"2024-06-29T15:58:34.619590+0000","last_clean":"2024-06-29T15:58:34.619590+0000","last_became_active":"2024-06-29T15:58:21.561051+0000","last_became_peered":"2024-06-29T15:58:21.561051+0000","last_unstale":"2024-06-29T15:58:34.619590+0000","last_undegraded":"2024-06-29T15:58:34.619590+0000","last_fullsized":"2024-06-29T15:58:34.619590+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:05:18.245403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646565+0000","last_change":"2024-06-29T15:58:34.644167+0000","last_active":"2024-06-29T15:58:34.646565+0000","last_peered":"2024-06-29T15:58:34.646565+0000","last_clean":"2024-06-29T15:58:34.646565+0000","last_became_active":"2024-06-29T15:58:34.643506+0000","last_became_peered":"2024-06-29T15:58:34.643506+0000","last_unstale":"2024-06-29T15:58:34.646565+0000","last_undegraded":"2024-06-29T15:58:34.646565+0000","last_fullsized":"2024-06-29T15:58:34.646565+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:44:43.831068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632537+0000","last_change":"2024-06-29T15:58:33.634722+0000","last_active":"2024-06-29T15:58:34.632537+0000","last_peered":"2024-06-29T15:58:34.632537+0000","last_clean":"2024-06-29T15:58:34.632537+0000","last_became_active":"2024-06-29T15:58:33.634357+0000","last_became_peered":"2024-06-29T15:58:33.634357+0000","last_unstale":"2024-06-29T15:58:34.632537+0000","last_undegraded":"2024-06-29T15:58:34.632537+0000","last_fullsized":"2024-06-29T15:58:34.632537+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:46.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630892+0000","last_change":"2024-06-29T15:58:21.561160+0000","last_active":"2024-06-29T15:58:34.630892+0000","last_peered":"2024-06-29T15:58:34.630892+0000","last_clean":"2024-06-29T15:58:34.630892+0000","last_became_active":"2024-06-29T15:58:21.560810+0000","last_became_peered":"2024-06-29T15:58:21.560810+0000","last_unstale":"2024-06-29T15:58:34.630892+0000","last_undegraded":"2024-06-29T15:58:34.630892+0000","last_fullsized":"2024-06-29T15:58:34.630892+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:02.548677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651228+0000","last_change":"2024-06-29T15:58:34.640872+0000","last_active":"2024-06-29T15:58:34.651228+0000","last_peered":"2024-06-29T15:58:34.651228+0000","last_clean":"2024-06-29T15:58:34.651228+0000","last_became_active":"2024-06-29T15:58:34.639856+0000","last_became_peered":"2024-06-29T15:58:34.639856+0000","last_unstale":"2024-06-29T15:58:34.651228+0000","last_undegraded":"2024-06-29T15:58:34.651228+0000","last_fullsized":"2024-06-29T15:58:34.651228+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:56.937582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.661611+0000","last_change":"2024-06-29T15:58:33.645366+0000","last_active":"2024-06-29T15:58:34.661611+0000","last_peered":"2024-06-29T15:58:34.661611+0000","last_clean":"2024-06-29T15:58:34.661611+0000","last_became_active":"2024-06-29T15:58:33.645140+0000","last_became_peered":"2024-06-29T15:58:33.645140+0000","last_unstale":"2024-06-29T15:58:34.661611+0000","last_undegraded":"2024-06-29T15:58:34.661611+0000","last_fullsized":"2024-06-29T15:58:34.661611+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:21.929648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632861+0000","last_change":"2024-06-29T15:58:21.561325+0000","last_active":"2024-06-29T15:58:34.632861+0000","last_peered":"2024-06-29T15:58:34.632861+0000","last_clean":"2024-06-29T15:58:34.632861+0000","last_became_active":"2024-06-29T15:58:21.561021+0000","last_became_peered":"2024-06-29T15:58:21.561021+0000","last_unstale":"2024-06-29T15:58:34.632861+0000","last_undegraded":"2024-06-29T15:58:34.632861+0000","last_fullsized":"2024-06-29T15:58:34.632861+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:25:28.950204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.641302+0000","last_change":"2024-06-29T15:58:34.639508+0000","last_active":"2024-06-29T15:58:34.641302+0000","last_peered":"2024-06-29T15:58:34.641302+0000","last_clean":"2024-06-29T15:58:34.641302+0000","last_became_active":"2024-06-29T15:58:34.639235+0000","last_became_peered":"2024-06-29T15:58:34.639235+0000","last_unstale":"2024-06-29T15:58:34.641302+0000","last_undegraded":"2024-06-29T15:58:34.641302+0000","last_fullsized":"2024-06-29T15:58:34.641302+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:47.106695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618776+0000","last_change":"2024-06-29T15:58:33.643134+0000","last_active":"2024-06-29T15:58:34.618776+0000","last_peered":"2024-06-29T15:58:34.618776+0000","last_clean":"2024-06-29T15:58:34.618776+0000","last_became_active":"2024-06-29T15:58:33.642765+0000","last_became_peered":"2024-06-29T15:58:33.642765+0000","last_unstale":"2024-06-29T15:58:34.618776+0000","last_undegraded":"2024-06-29T15:58:34.618776+0000","last_fullsized":"2024-06-29T15:58:34.618776+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:50.465474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626713+0000","last_change":"2024-06-29T15:58:21.561369+0000","last_active":"2024-06-29T15:58:34.626713+0000","last_peered":"2024-06-29T15:58:34.626713+0000","last_clean":"2024-06-29T15:58:34.626713+0000","last_became_active":"2024-06-29T15:58:21.561021+0000","last_became_peered":"2024-06-29T15:58:21.561021+0000","last_unstale":"2024-06-29T15:58:34.626713+0000","last_undegraded":"2024-06-29T15:58:34.626713+0000","last_fullsized":"2024-06-29T15:58:34.626713+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:35.412665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659284+0000","last_change":"2024-06-29T15:58:34.655983+0000","last_active":"2024-06-29T15:58:34.659284+0000","last_peered":"2024-06-29T15:58:34.659284+0000","last_clean":"2024-06-29T15:58:34.659284+0000","last_became_active":"2024-06-29T15:58:34.655665+0000","last_became_peered":"2024-06-29T15:58:34.655665+0000","last_unstale":"2024-06-29T15:58:34.659284+0000","last_undegraded":"2024-06-29T15:58:34.659284+0000","last_fullsized":"2024-06-29T15:58:34.659284+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:47.281204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.657655+0000","last_change":"2024-06-29T15:58:33.643317+0000","last_active":"2024-06-29T15:58:34.657655+0000","last_peered":"2024-06-29T15:58:34.657655+0000","last_clean":"2024-06-29T15:58:34.657655+0000","last_became_active":"2024-06-29T15:58:33.643011+0000","last_became_peered":"2024-06-29T15:58:33.643011+0000","last_unstale":"2024-06-29T15:58:34.657655+0000","last_undegraded":"2024-06-29T15:58:34.657655+0000","last_fullsized":"2024-06-29T15:58:34.657655+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:11.512035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628018+0000","last_change":"2024-06-29T15:58:21.555481+0000","last_active":"2024-06-29T15:58:34.628018+0000","last_peered":"2024-06-29T15:58:34.628018+0000","last_clean":"2024-06-29T15:58:34.628018+0000","last_became_active":"2024-06-29T15:58:21.553482+0000","last_became_peered":"2024-06-29T15:58:21.553482+0000","last_unstale":"2024-06-29T15:58:34.628018+0000","last_undegraded":"2024-06-29T15:58:34.628018+0000","last_fullsized":"2024-06-29T15:58:34.628018+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:08.904938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659015+0000","last_change":"2024-06-29T15:58:34.647483+0000","last_active":"2024-06-29T15:58:34.659015+0000","last_peered":"2024-06-29T15:58:34.659015+0000","last_clean":"2024-06-29T15:58:34.659015+0000","last_became_active":"2024-06-29T15:58:34.647153+0000","last_became_peered":"2024-06-29T15:58:34.647153+0000","last_unstale":"2024-06-29T15:58:34.659015+0000","last_undegraded":"2024-06-29T15:58:34.659015+0000","last_fullsized":"2024-06-29T15:58:34.659015+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:51.044749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625254+0000","last_change":"2024-06-29T15:58:33.632232+0000","last_active":"2024-06-29T15:58:34.625254+0000","last_peered":"2024-06-29T15:58:34.625254+0000","last_clean":"2024-06-29T15:58:34.625254+0000","last_became_active":"2024-06-29T15:58:33.631892+0000","last_became_peered":"2024-06-29T15:58:33.631892+0000","last_unstale":"2024-06-29T15:58:34.625254+0000","last_undegraded":"2024-06-29T15:58:34.625254+0000","last_fullsized":"2024-06-29T15:58:34.625254+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:32.556085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628518+0000","last_change":"2024-06-29T15:58:21.560413+0000","last_active":"2024-06-29T15:58:34.628518+0000","last_peered":"2024-06-29T15:58:34.628518+0000","last_clean":"2024-06-29T15:58:34.628518+0000","last_became_active":"2024-06-29T15:58:21.560114+0000","last_became_peered":"2024-06-29T15:58:21.560114+0000","last_unstale":"2024-06-29T15:58:34.628518+0000","last_undegraded":"2024-06-29T15:58:34.628518+0000","last_fullsized":"2024-06-29T15:58:34.628518+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:43:35.054888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640650+0000","last_change":"2024-06-29T15:58:34.639095+0000","last_active":"2024-06-29T15:58:34.640650+0000","last_peered":"2024-06-29T15:58:34.640650+0000","last_clean":"2024-06-29T15:58:34.640650+0000","last_became_active":"2024-06-29T15:58:34.638745+0000","last_became_peered":"2024-06-29T15:58:34.638745+0000","last_unstale":"2024-06-29T15:58:34.640650+0000","last_undegraded":"2024-06-29T15:58:34.640650+0000","last_fullsized":"2024-06-29T15:58:34.640650+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:59:06.483477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631469+0000","last_change":"2024-06-29T15:58:33.646999+0000","last_active":"2024-06-29T15:58:34.631469+0000","last_peered":"2024-06-29T15:58:34.631469+0000","last_clean":"2024-06-29T15:58:34.631469+0000","last_became_active":"2024-06-29T15:58:33.646536+0000","last_became_peered":"2024-06-29T15:58:33.646536+0000","last_unstale":"2024-06-29T15:58:34.631469+0000","last_undegraded":"2024-06-29T15:58:34.631469+0000","last_fullsized":"2024-06-29T15:58:34.631469+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:07:21.845128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627655+0000","last_change":"2024-06-29T15:58:21.562168+0000","last_active":"2024-06-29T15:58:34.627655+0000","last_peered":"2024-06-29T15:58:34.627655+0000","last_clean":"2024-06-29T15:58:34.627655+0000","last_became_active":"2024-06-29T15:58:21.555306+0000","last_became_peered":"2024-06-29T15:58:21.555306+0000","last_unstale":"2024-06-29T15:58:34.627655+0000","last_undegraded":"2024-06-29T15:58:34.627655+0000","last_fullsized":"2024-06-29T15:58:34.627655+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:40.193793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649795+0000","last_change":"2024-06-29T15:58:34.649447+0000","last_active":"2024-06-29T15:58:34.649795+0000","last_peered":"2024-06-29T15:58:34.649795+0000","last_clean":"2024-06-29T15:58:34.649795+0000","last_became_active":"2024-06-29T15:58:34.649170+0000","last_became_peered":"2024-06-29T15:58:34.649170+0000","last_unstale":"2024-06-29T15:58:34.649795+0000","last_undegraded":"2024-06-29T15:58:34.649795+0000","last_fullsized":"2024-06-29T15:58:34.649795+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:09.189841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631513+0000","last_change":"2024-06-29T15:58:33.635574+0000","last_active":"2024-06-29T15:58:34.631513+0000","last_peered":"2024-06-29T15:58:34.631513+0000","last_clean":"2024-06-29T15:58:34.631513+0000","last_became_active":"2024-06-29T15:58:33.635259+0000","last_became_peered":"2024-06-29T15:58:33.635259+0000","last_unstale":"2024-06-29T15:58:34.631513+0000","last_undegraded":"2024-06-29T15:58:34.631513+0000","last_fullsized":"2024-06-29T15:58:34.631513+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:41.951390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632590+0000","last_change":"2024-06-29T15:58:21.559133+0000","last_active":"2024-06-29T15:58:34.632590+0000","last_peered":"2024-06-29T15:58:34.632590+0000","last_clean":"2024-06-29T15:58:34.632590+0000","last_became_active":"2024-06-29T15:58:21.558649+0000","last_became_peered":"2024-06-29T15:58:21.558649+0000","last_unstale":"2024-06-29T15:58:34.632590+0000","last_undegraded":"2024-06-29T15:58:34.632590+0000","last_fullsized":"2024-06-29T15:58:34.632590+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:28.201954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646713+0000","last_change":"2024-06-29T15:58:34.645080+0000","last_active":"2024-06-29T15:58:34.646713+0000","last_peered":"2024-06-29T15:58:34.646713+0000","last_clean":"2024-06-29T15:58:34.646713+0000","last_became_active":"2024-06-29T15:58:34.644481+0000","last_became_peered":"2024-06-29T15:58:34.644481+0000","last_unstale":"2024-06-29T15:58:34.646713+0000","last_undegraded":"2024-06-29T15:58:34.646713+0000","last_fullsized":"2024-06-29T15:58:34.646713+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:55.000504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.660160+0000","last_change":"2024-06-29T15:58:33.633026+0000","last_active":"2024-06-29T15:58:34.660160+0000","last_peered":"2024-06-29T15:58:34.660160+0000","last_clean":"2024-06-29T15:58:34.660160+0000","last_became_active":"2024-06-29T15:58:33.632699+0000","last_became_peered":"2024-06-29T15:58:33.632699+0000","last_unstale":"2024-06-29T15:58:34.660160+0000","last_undegraded":"2024-06-29T15:58:34.660160+0000","last_fullsized":"2024-06-29T15:58:34.660160+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:41.572497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627434+0000","last_change":"2024-06-29T15:58:21.561677+0000","last_active":"2024-06-29T15:58:34.627434+0000","last_peered":"2024-06-29T15:58:34.627434+0000","last_clean":"2024-06-29T15:58:34.627434+0000","last_became_active":"2024-06-29T15:58:21.561367+0000","last_became_peered":"2024-06-29T15:58:21.561367+0000","last_unstale":"2024-06-29T15:58:34.627434+0000","last_undegraded":"2024-06-29T15:58:34.627434+0000","last_fullsized":"2024-06-29T15:58:34.627434+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:23:28.067754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.643403+0000","last_change":"2024-06-29T15:58:34.641174+0000","last_active":"2024-06-29T15:58:34.643403+0000","last_peered":"2024-06-29T15:58:34.643403+0000","last_clean":"2024-06-29T15:58:34.643403+0000","last_became_active":"2024-06-29T15:58:34.640804+0000","last_became_peered":"2024-06-29T15:58:34.640804+0000","last_unstale":"2024-06-29T15:58:34.643403+0000","last_undegraded":"2024-06-29T15:58:34.643403+0000","last_fullsized":"2024-06-29T15:58:34.643403+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:38:32.263425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626725+0000","last_change":"2024-06-29T15:58:33.643166+0000","last_active":"2024-06-29T15:58:34.626725+0000","last_peered":"2024-06-29T15:58:34.626725+0000","last_clean":"2024-06-29T15:58:34.626725+0000","last_became_active":"2024-06-29T15:58:33.642827+0000","last_became_peered":"2024-06-29T15:58:33.642827+0000","last_unstale":"2024-06-29T15:58:34.626725+0000","last_undegraded":"2024-06-29T15:58:34.626725+0000","last_fullsized":"2024-06-29T15:58:34.626725+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:24.209579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622358+0000","last_change":"2024-06-29T15:58:21.552197+0000","last_active":"2024-06-29T15:58:34.622358+0000","last_peered":"2024-06-29T15:58:34.622358+0000","last_clean":"2024-06-29T15:58:34.622358+0000","last_became_active":"2024-06-29T15:58:21.551846+0000","last_became_peered":"2024-06-29T15:58:21.551846+0000","last_unstale":"2024-06-29T15:58:34.622358+0000","last_undegraded":"2024-06-29T15:58:34.622358+0000","last_fullsized":"2024-06-29T15:58:34.622358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:31.983161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.647181+0000","last_change":"2024-06-29T15:58:34.643697+0000","last_active":"2024-06-29T15:58:34.647181+0000","last_peered":"2024-06-29T15:58:34.647181+0000","last_clean":"2024-06-29T15:58:34.647181+0000","last_became_active":"2024-06-29T15:58:34.641630+0000","last_became_peered":"2024-06-29T15:58:34.641630+0000","last_unstale":"2024-06-29T15:58:34.647181+0000","last_undegraded":"2024-06-29T15:58:34.647181+0000","last_fullsized":"2024-06-29T15:58:34.647181+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:48.481424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631486+0000","last_change":"2024-06-29T15:58:33.638350+0000","last_active":"2024-06-29T15:58:34.631486+0000","last_peered":"2024-06-29T15:58:34.631486+0000","last_clean":"2024-06-29T15:58:34.631486+0000","last_became_active":"2024-06-29T15:58:33.638072+0000","last_became_peered":"2024-06-29T15:58:33.638072+0000","last_unstale":"2024-06-29T15:58:34.631486+0000","last_undegraded":"2024-06-29T15:58:34.631486+0000","last_fullsized":"2024-06-29T15:58:34.631486+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:26.233478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629137+0000","last_change":"2024-06-29T15:58:21.560349+0000","last_active":"2024-06-29T15:58:34.629137+0000","last_peered":"2024-06-29T15:58:34.629137+0000","last_clean":"2024-06-29T15:58:34.629137+0000","last_became_active":"2024-06-29T15:58:21.559976+0000","last_became_peered":"2024-06-29T15:58:21.559976+0000","last_unstale":"2024-06-29T15:58:34.629137+0000","last_undegraded":"2024-06-29T15:58:34.629137+0000","last_fullsized":"2024-06-29T15:58:34.629137+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:35:34.378626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659030+0000","last_change":"2024-06-29T15:58:34.642890+0000","last_active":"2024-06-29T15:58:34.659030+0000","last_peered":"2024-06-29T15:58:34.659030+0000","last_clean":"2024-06-29T15:58:34.659030+0000","last_became_active":"2024-06-29T15:58:34.642520+0000","last_became_peered":"2024-06-29T15:58:34.642520+0000","last_unstale":"2024-06-29T15:58:34.659030+0000","last_undegraded":"2024-06-29T15:58:34.659030+0000","last_fullsized":"2024-06-29T15:58:34.659030+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:21.951845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631238+0000","last_change":"2024-06-29T15:58:33.627943+0000","last_active":"2024-06-29T15:58:34.631238+0000","last_peered":"2024-06-29T15:58:34.631238+0000","last_clean":"2024-06-29T15:58:34.631238+0000","last_became_active":"2024-06-29T15:58:33.626589+0000","last_became_peered":"2024-06-29T15:58:33.626589+0000","last_unstale":"2024-06-29T15:58:34.631238+0000","last_undegraded":"2024-06-29T15:58:34.631238+0000","last_fullsized":"2024-06-29T15:58:34.631238+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:01.898801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625701+0000","last_change":"2024-06-29T15:58:21.567137+0000","last_active":"2024-06-29T15:58:34.625701+0000","last_peered":"2024-06-29T15:58:34.625701+0000","last_clean":"2024-06-29T15:58:34.625701+0000","last_became_active":"2024-06-29T15:58:21.566816+0000","last_became_peered":"2024-06-29T15:58:21.566816+0000","last_unstale":"2024-06-29T15:58:34.625701+0000","last_undegraded":"2024-06-29T15:58:34.625701+0000","last_fullsized":"2024-06-29T15:58:34.625701+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:36.096648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.644310+0000","last_change":"2024-06-29T15:58:34.634052+0000","last_active":"2024-06-29T15:58:34.644310+0000","last_peered":"2024-06-29T15:58:34.644310+0000","last_clean":"2024-06-29T15:58:34.644310+0000","last_became_active":"2024-06-29T15:58:34.633713+0000","last_became_peered":"2024-06-29T15:58:34.633713+0000","last_unstale":"2024-06-29T15:58:34.644310+0000","last_undegraded":"2024-06-29T15:58:34.644310+0000","last_fullsized":"2024-06-29T15:58:34.644310+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:12.235244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.633820+0000","last_change":"2024-06-29T15:58:33.625957+0000","last_active":"2024-06-29T15:58:34.633820+0000","last_peered":"2024-06-29T15:58:34.633820+0000","last_clean":"2024-06-29T15:58:34.633820+0000","last_became_active":"2024-06-29T15:58:33.624954+0000","last_became_peered":"2024-06-29T15:58:33.624954+0000","last_unstale":"2024-06-29T15:58:34.633820+0000","last_undegraded":"2024-06-29T15:58:34.633820+0000","last_fullsized":"2024-06-29T15:58:34.633820+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:16.253720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.633012+0000","last_change":"2024-06-29T15:58:21.561708+0000","last_active":"2024-06-29T15:58:34.633012+0000","last_peered":"2024-06-29T15:58:34.633012+0000","last_clean":"2024-06-29T15:58:34.633012+0000","last_became_active":"2024-06-29T15:58:21.561360+0000","last_became_peered":"2024-06-29T15:58:21.561360+0000","last_unstale":"2024-06-29T15:58:34.633012+0000","last_undegraded":"2024-06-29T15:58:34.633012+0000","last_fullsized":"2024-06-29T15:58:34.633012+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:17.841782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640391+0000","last_change":"2024-06-29T15:58:34.636562+0000","last_active":"2024-06-29T15:58:34.640391+0000","last_peered":"2024-06-29T15:58:34.640391+0000","last_clean":"2024-06-29T15:58:34.640391+0000","last_became_active":"2024-06-29T15:58:34.636282+0000","last_became_peered":"2024-06-29T15:58:34.636282+0000","last_unstale":"2024-06-29T15:58:34.640391+0000","last_undegraded":"2024-06-29T15:58:34.640391+0000","last_fullsized":"2024-06-29T15:58:34.640391+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:05.349103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629501+0000","last_change":"2024-06-29T15:58:33.636425+0000","last_active":"2024-06-29T15:58:34.629501+0000","last_peered":"2024-06-29T15:58:34.629501+0000","last_clean":"2024-06-29T15:58:34.629501+0000","last_became_active":"2024-06-29T15:58:33.635983+0000","last_became_peered":"2024-06-29T15:58:33.635983+0000","last_unstale":"2024-06-29T15:58:34.629501+0000","last_undegraded":"2024-06-29T15:58:34.629501+0000","last_fullsized":"2024-06-29T15:58:34.629501+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:40.750750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630894+0000","last_change":"2024-06-29T15:58:21.560378+0000","last_active":"2024-06-29T15:58:34.630894+0000","last_peered":"2024-06-29T15:58:34.630894+0000","last_clean":"2024-06-29T15:58:34.630894+0000","last_became_active":"2024-06-29T15:58:21.560075+0000","last_became_peered":"2024-06-29T15:58:21.560075+0000","last_unstale":"2024-06-29T15:58:34.630894+0000","last_undegraded":"2024-06-29T15:58:34.630894+0000","last_fullsized":"2024-06-29T15:58:34.630894+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:16.886076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658855+0000","last_change":"2024-06-29T15:58:34.648158+0000","last_active":"2024-06-29T15:58:34.658855+0000","last_peered":"2024-06-29T15:58:34.658855+0000","last_clean":"2024-06-29T15:58:34.658855+0000","last_became_active":"2024-06-29T15:58:34.647788+0000","last_became_peered":"2024-06-29T15:58:34.647788+0000","last_unstale":"2024-06-29T15:58:34.658855+0000","last_undegraded":"2024-06-29T15:58:34.658855+0000","last_fullsized":"2024-06-29T15:58:34.658855+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:51.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630739+0000","last_change":"2024-06-29T15:58:33.634531+0000","last_active":"2024-06-29T15:58:34.630739+0000","last_peered":"2024-06-29T15:58:34.630739+0000","last_clean":"2024-06-29T15:58:34.630739+0000","last_became_active":"2024-06-29T15:58:33.634126+0000","last_became_peered":"2024-06-29T15:58:33.634126+0000","last_unstale":"2024-06-29T15:58:34.630739+0000","last_undegraded":"2024-06-29T15:58:34.630739+0000","last_fullsized":"2024-06-29T15:58:34.630739+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:31.650912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625820+0000","last_change":"2024-06-29T15:58:21.561504+0000","last_active":"2024-06-29T15:58:34.625820+0000","last_peered":"2024-06-29T15:58:34.625820+0000","last_clean":"2024-06-29T15:58:34.625820+0000","last_became_active":"2024-06-29T15:58:21.560810+0000","last_became_peered":"2024-06-29T15:58:21.560810+0000","last_unstale":"2024-06-29T15:58:34.625820+0000","last_undegraded":"2024-06-29T15:58:34.625820+0000","last_fullsized":"2024-06-29T15:58:34.625820+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:53:36.634654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640285+0000","last_change":"2024-06-29T15:58:34.639130+0000","last_active":"2024-06-29T15:58:34.640285+0000","last_peered":"2024-06-29T15:58:34.640285+0000","last_clean":"2024-06-29T15:58:34.640285+0000","last_became_active":"2024-06-29T15:58:34.638792+0000","last_became_peered":"2024-06-29T15:58:34.638792+0000","last_unstale":"2024-06-29T15:58:34.640285+0000","last_undegraded":"2024-06-29T15:58:34.640285+0000","last_fullsized":"2024-06-29T15:58:34.640285+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:54.624785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629861+0000","last_change":"2024-06-29T15:58:21.562794+0000","last_active":"2024-06-29T15:58:34.629861+0000","last_peered":"2024-06-29T15:58:34.629861+0000","last_clean":"2024-06-29T15:58:34.629861+0000","last_became_active":"2024-06-29T15:58:21.562011+0000","last_became_peered":"2024-06-29T15:58:21.562011+0000","last_unstale":"2024-06-29T15:58:34.629861+0000","last_undegraded":"2024-06-29T15:58:34.629861+0000","last_fullsized":"2024-06-29T15:58:34.629861+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:09.874394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"241'4","reported_seq":24,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630137+0000","last_change":"2024-06-29T15:58:20.549867+0000","last_active":"2024-06-29T15:58:34.630137+0000","last_peered":"2024-06-29T15:58:34.630137+0000","last_clean":"2024-06-29T15:58:34.630137+0000","last_became_active":"2024-06-29T15:58:20.549532+0000","last_became_peered":"2024-06-29T15:58:20.549532+0000","last_unstale":"2024-06-29T15:58:34.630137+0000","last_undegraded":"2024-06-29T15:58:34.630137+0000","last_fullsized":"2024-06-29T15:58:34.630137+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:11.359524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658942+0000","last_change":"2024-06-29T15:58:34.646233+0000","last_active":"2024-06-29T15:58:34.658942+0000","last_peered":"2024-06-29T15:58:34.658942+0000","last_clean":"2024-06-29T15:58:34.658942+0000","last_became_active":"2024-06-29T15:58:34.645874+0000","last_became_peered":"2024-06-29T15:58:34.645874+0000","last_unstale":"2024-06-29T15:58:34.658942+0000","last_undegraded":"2024-06-29T15:58:34.658942+0000","last_fullsized":"2024-06-29T15:58:34.658942+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:30.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630456+0000","last_change":"2024-06-29T15:58:21.562279+0000","last_active":"2024-06-29T15:58:34.630456+0000","last_peered":"2024-06-29T15:58:34.630456+0000","last_clean":"2024-06-29T15:58:34.630456+0000","last_became_active":"2024-06-29T15:58:21.561962+0000","last_became_peered":"2024-06-29T15:58:21.561962+0000","last_unstale":"2024-06-29T15:58:34.630456+0000","last_undegraded":"2024-06-29T15:58:34.630456+0000","last_fullsized":"2024-06-29T15:58:34.630456+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:43:25.515942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629756+0000","last_change":"2024-06-29T15:58:20.547115+0000","last_active":"2024-06-29T15:58:34.629756+0000","last_peered":"2024-06-29T15:58:34.629756+0000","last_clean":"2024-06-29T15:58:34.629756+0000","last_became_active":"2024-06-29T15:58:20.546778+0000","last_became_peered":"2024-06-29T15:58:20.546778+0000","last_unstale":"2024-06-29T15:58:34.629756+0000","last_undegraded":"2024-06-29T15:58:34.629756+0000","last_fullsized":"2024-06-29T15:58:34.629756+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:05.970706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659082+0000","last_change":"2024-06-29T15:58:34.647340+0000","last_active":"2024-06-29T15:58:34.659082+0000","last_peered":"2024-06-29T15:58:34.659082+0000","last_clean":"2024-06-29T15:58:34.659082+0000","last_became_active":"2024-06-29T15:58:34.646885+0000","last_became_peered":"2024-06-29T15:58:34.646885+0000","last_unstale":"2024-06-29T15:58:34.659082+0000","last_undegraded":"2024-06-29T15:58:34.659082+0000","last_fullsized":"2024-06-29T15:58:34.659082+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:16.000133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629521+0000","last_change":"2024-06-29T15:58:21.562748+0000","last_active":"2024-06-29T15:58:34.629521+0000","last_peered":"2024-06-29T15:58:34.629521+0000","last_clean":"2024-06-29T15:58:34.629521+0000","last_became_active":"2024-06-29T15:58:21.562417+0000","last_became_peered":"2024-06-29T15:58:21.562417+0000","last_unstale":"2024-06-29T15:58:34.629521+0000","last_undegraded":"2024-06-29T15:58:34.629521+0000","last_fullsized":"2024-06-29T15:58:34.629521+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:27.973848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627211+0000","last_change":"2024-06-29T15:58:20.568004+0000","last_active":"2024-06-29T15:58:34.627211+0000","last_peered":"2024-06-29T15:58:34.627211+0000","last_clean":"2024-06-29T15:58:34.627211+0000","last_became_active":"2024-06-29T15:58:20.567638+0000","last_became_peered":"2024-06-29T15:58:20.567638+0000","last_unstale":"2024-06-29T15:58:34.627211+0000","last_undegraded":"2024-06-29T15:58:34.627211+0000","last_fullsized":"2024-06-29T15:58:34.627211+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:19.772821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658924+0000","last_change":"2024-06-29T15:58:34.646747+0000","last_active":"2024-06-29T15:58:34.658924+0000","last_peered":"2024-06-29T15:58:34.658924+0000","last_clean":"2024-06-29T15:58:34.658924+0000","last_became_active":"2024-06-29T15:58:34.646397+0000","last_became_peered":"2024-06-29T15:58:34.646397+0000","last_unstale":"2024-06-29T15:58:34.658924+0000","last_undegraded":"2024-06-29T15:58:34.658924+0000","last_fullsized":"2024-06-29T15:58:34.658924+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:34.731318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626902+0000","last_change":"2024-06-29T15:58:21.566347+0000","last_active":"2024-06-29T15:58:34.626902+0000","last_peered":"2024-06-29T15:58:34.626902+0000","last_clean":"2024-06-29T15:58:34.626902+0000","last_became_active":"2024-06-29T15:58:21.565888+0000","last_became_peered":"2024-06-29T15:58:21.565888+0000","last_unstale":"2024-06-29T15:58:34.626902+0000","last_undegraded":"2024-06-29T15:58:34.626902+0000","last_fullsized":"2024-06-29T15:58:34.626902+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:28.461061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628877+0000","last_change":"2024-06-29T15:58:20.556653+0000","last_active":"2024-06-29T15:58:34.628877+0000","last_peered":"2024-06-29T15:58:34.628877+0000","last_clean":"2024-06-29T15:58:34.628877+0000","last_became_active":"2024-06-29T15:58:20.555367+0000","last_became_peered":"2024-06-29T15:58:20.555367+0000","last_unstale":"2024-06-29T15:58:34.628877+0000","last_undegraded":"2024-06-29T15:58:34.628877+0000","last_fullsized":"2024-06-29T15:58:34.628877+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:31.173223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649284+0000","last_change":"2024-06-29T15:58:34.647891+0000","last_active":"2024-06-29T15:58:34.649284+0000","last_peered":"2024-06-29T15:58:34.649284+0000","last_clean":"2024-06-29T15:58:34.649284+0000","last_became_active":"2024-06-29T15:58:34.647354+0000","last_became_peered":"2024-06-29T15:58:34.647354+0000","last_unstale":"2024-06-29T15:58:34.649284+0000","last_undegraded":"2024-06-29T15:58:34.649284+0000","last_fullsized":"2024-06-29T15:58:34.649284+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:11.281394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.633083+0000","last_change":"2024-06-29T15:58:21.562456+0000","last_active":"2024-06-29T15:58:34.633083+0000","last_peered":"2024-06-29T15:58:34.633083+0000","last_clean":"2024-06-29T15:58:34.633083+0000","last_became_active":"2024-06-29T15:58:21.562024+0000","last_became_peered":"2024-06-29T15:58:21.562024+0000","last_unstale":"2024-06-29T15:58:34.633083+0000","last_undegraded":"2024-06-29T15:58:34.633083+0000","last_fullsized":"2024-06-29T15:58:34.633083+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:36.495502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632827+0000","last_change":"2024-06-29T15:58:20.558360+0000","last_active":"2024-06-29T15:58:34.632827+0000","last_peered":"2024-06-29T15:58:34.632827+0000","last_clean":"2024-06-29T15:58:34.632827+0000","last_became_active":"2024-06-29T15:58:20.558053+0000","last_became_peered":"2024-06-29T15:58:20.558053+0000","last_unstale":"2024-06-29T15:58:34.632827+0000","last_undegraded":"2024-06-29T15:58:34.632827+0000","last_fullsized":"2024-06-29T15:58:34.632827+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:40.258636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.647182+0000","last_change":"2024-06-29T15:58:34.646442+0000","last_active":"2024-06-29T15:58:34.647182+0000","last_peered":"2024-06-29T15:58:34.647182+0000","last_clean":"2024-06-29T15:58:34.647182+0000","last_became_active":"2024-06-29T15:58:34.645545+0000","last_became_peered":"2024-06-29T15:58:34.645545+0000","last_unstale":"2024-06-29T15:58:34.647182+0000","last_undegraded":"2024-06-29T15:58:34.647182+0000","last_fullsized":"2024-06-29T15:58:34.647182+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:33.180128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619310+0000","last_change":"2024-06-29T15:58:21.568464+0000","last_active":"2024-06-29T15:58:34.619310+0000","last_peered":"2024-06-29T15:58:34.619310+0000","last_clean":"2024-06-29T15:58:34.619310+0000","last_became_active":"2024-06-29T15:58:21.568114+0000","last_became_peered":"2024-06-29T15:58:21.568114+0000","last_unstale":"2024-06-29T15:58:34.619310+0000","last_undegraded":"2024-06-29T15:58:34.619310+0000","last_fullsized":"2024-06-29T15:58:34.619310+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:09.625056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618713+0000","last_change":"2024-06-29T15:58:20.563140+0000","last_active":"2024-06-29T15:58:34.618713+0000","last_peered":"2024-06-29T15:58:34.618713+0000","last_clean":"2024-06-29T15:58:34.618713+0000","last_became_active":"2024-06-29T15:58:20.562823+0000","last_became_peered":"2024-06-29T15:58:20.562823+0000","last_unstale":"2024-06-29T15:58:34.618713+0000","last_undegraded":"2024-06-29T15:58:34.618713+0000","last_fullsized":"2024-06-29T15:58:34.618713+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:24.191012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646965+0000","last_change":"2024-06-29T15:58:34.644576+0000","last_active":"2024-06-29T15:58:34.646965+0000","last_peered":"2024-06-29T15:58:34.646965+0000","last_clean":"2024-06-29T15:58:34.646965+0000","last_became_active":"2024-06-29T15:58:34.644165+0000","last_became_peered":"2024-06-29T15:58:34.644165+0000","last_unstale":"2024-06-29T15:58:34.646965+0000","last_undegraded":"2024-06-29T15:58:34.646965+0000","last_fullsized":"2024-06-29T15:58:34.646965+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:29.253507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622811+0000","last_change":"2024-06-29T15:58:21.556649+0000","last_active":"2024-06-29T15:58:34.622811+0000","last_peered":"2024-06-29T15:58:34.622811+0000","last_clean":"2024-06-29T15:58:34.622811+0000","last_became_active":"2024-06-29T15:58:21.556297+0000","last_became_peered":"2024-06-29T15:58:21.556297+0000","last_unstale":"2024-06-29T15:58:34.622811+0000","last_undegraded":"2024-06-29T15:58:34.622811+0000","last_fullsized":"2024-06-29T15:58:34.622811+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:07.319988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626017+0000","last_change":"2024-06-29T15:58:20.555706+0000","last_active":"2024-06-29T15:58:34.626017+0000","last_peered":"2024-06-29T15:58:34.626017+0000","last_clean":"2024-06-29T15:58:34.626017+0000","last_became_active":"2024-06-29T15:58:20.555380+0000","last_became_peered":"2024-06-29T15:58:20.555380+0000","last_unstale":"2024-06-29T15:58:34.626017+0000","last_undegraded":"2024-06-29T15:58:34.626017+0000","last_fullsized":"2024-06-29T15:58:34.626017+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:03:17.347006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659569+0000","last_change":"2024-06-29T15:58:34.659095+0000","last_active":"2024-06-29T15:58:34.659569+0000","last_peered":"2024-06-29T15:58:34.659569+0000","last_clean":"2024-06-29T15:58:34.659569+0000","last_became_active":"2024-06-29T15:58:34.658482+0000","last_became_peered":"2024-06-29T15:58:34.658482+0000","last_unstale":"2024-06-29T15:58:34.659569+0000","last_undegraded":"2024-06-29T15:58:34.659569+0000","last_fullsized":"2024-06-29T15:58:34.659569+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:25.614229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629077+0000","last_change":"2024-06-29T15:58:21.554565+0000","last_active":"2024-06-29T15:58:34.629077+0000","last_peered":"2024-06-29T15:58:34.629077+0000","last_clean":"2024-06-29T15:58:34.629077+0000","last_became_active":"2024-06-29T15:58:21.554255+0000","last_became_peered":"2024-06-29T15:58:21.554255+0000","last_unstale":"2024-06-29T15:58:34.629077+0000","last_undegraded":"2024-06-29T15:58:34.629077+0000","last_fullsized":"2024-06-29T15:58:34.629077+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:14.470919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626079+0000","last_change":"2024-06-29T15:58:20.562271+0000","last_active":"2024-06-29T15:58:34.626079+0000","last_peered":"2024-06-29T15:58:34.626079+0000","last_clean":"2024-06-29T15:58:34.626079+0000","last_became_active":"2024-06-29T15:58:20.561940+0000","last_became_peered":"2024-06-29T15:58:20.561940+0000","last_unstale":"2024-06-29T15:58:34.626079+0000","last_undegraded":"2024-06-29T15:58:34.626079+0000","last_fullsized":"2024-06-29T15:58:34.626079+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:16.036109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.639400+0000","last_change":"2024-06-29T15:58:34.638123+0000","last_active":"2024-06-29T15:58:34.639400+0000","last_peered":"2024-06-29T15:58:34.639400+0000","last_clean":"2024-06-29T15:58:34.639400+0000","last_became_active":"2024-06-29T15:58:34.637789+0000","last_became_peered":"2024-06-29T15:58:34.637789+0000","last_unstale":"2024-06-29T15:58:34.639400+0000","last_undegraded":"2024-06-29T15:58:34.639400+0000","last_fullsized":"2024-06-29T15:58:34.639400+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:52.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628202+0000","last_change":"2024-06-29T15:58:21.548845+0000","last_active":"2024-06-29T15:58:34.628202+0000","last_peered":"2024-06-29T15:58:34.628202+0000","last_clean":"2024-06-29T15:58:34.628202+0000","last_became_active":"2024-06-29T15:58:21.548448+0000","last_became_peered":"2024-06-29T15:58:21.548448+0000","last_unstale":"2024-06-29T15:58:34.628202+0000","last_undegraded":"2024-06-29T15:58:34.628202+0000","last_fullsized":"2024-06-29T15:58:34.628202+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:38.983134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619705+0000","last_change":"2024-06-29T15:58:20.562116+0000","last_active":"2024-06-29T15:58:34.619705+0000","last_peered":"2024-06-29T15:58:34.619705+0000","last_clean":"2024-06-29T15:58:34.619705+0000","last_became_active":"2024-06-29T15:58:20.561801+0000","last_became_peered":"2024-06-29T15:58:20.561801+0000","last_unstale":"2024-06-29T15:58:34.619705+0000","last_undegraded":"2024-06-29T15:58:34.619705+0000","last_fullsized":"2024-06-29T15:58:34.619705+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:26.721883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640304+0000","last_change":"2024-06-29T15:58:34.637931+0000","last_active":"2024-06-29T15:58:34.640304+0000","last_peered":"2024-06-29T15:58:34.640304+0000","last_clean":"2024-06-29T15:58:34.640304+0000","last_became_active":"2024-06-29T15:58:34.637342+0000","last_became_peered":"2024-06-29T15:58:34.637342+0000","last_unstale":"2024-06-29T15:58:34.640304+0000","last_undegraded":"2024-06-29T15:58:34.640304+0000","last_fullsized":"2024-06-29T15:58:34.640304+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:25.149310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618903+0000","last_change":"2024-06-29T15:58:21.560085+0000","last_active":"2024-06-29T15:58:34.618903+0000","last_peered":"2024-06-29T15:58:34.618903+0000","last_clean":"2024-06-29T15:58:34.618903+0000","last_became_active":"2024-06-29T15:58:21.559716+0000","last_became_peered":"2024-06-29T15:58:21.559716+0000","last_unstale":"2024-06-29T15:58:34.618903+0000","last_undegraded":"2024-06-29T15:58:34.618903+0000","last_fullsized":"2024-06-29T15:58:34.618903+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:15:37.579793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"241'3","reported_seq":23,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622530+0000","last_change":"2024-06-29T15:58:20.553528+0000","last_active":"2024-06-29T15:58:34.622530+0000","last_peered":"2024-06-29T15:58:34.622530+0000","last_clean":"2024-06-29T15:58:34.622530+0000","last_became_active":"2024-06-29T15:58:20.553185+0000","last_became_peered":"2024-06-29T15:58:20.553185+0000","last_unstale":"2024-06-29T15:58:34.622530+0000","last_undegraded":"2024-06-29T15:58:34.622530+0000","last_fullsized":"2024-06-29T15:58:34.622530+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:26.807742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659056+0000","last_change":"2024-06-29T15:58:34.643180+0000","last_active":"2024-06-29T15:58:34.659056+0000","last_peered":"2024-06-29T15:58:34.659056+0000","last_clean":"2024-06-29T15:58:34.659056+0000","last_became_active":"2024-06-29T15:58:34.642846+0000","last_became_peered":"2024-06-29T15:58:34.642846+0000","last_unstale":"2024-06-29T15:58:34.659056+0000","last_undegraded":"2024-06-29T15:58:34.659056+0000","last_fullsized":"2024-06-29T15:58:34.659056+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:24:27.526406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626571+0000","last_change":"2024-06-29T15:58:21.553783+0000","last_active":"2024-06-29T15:58:34.626571+0000","last_peered":"2024-06-29T15:58:34.626571+0000","last_clean":"2024-06-29T15:58:34.626571+0000","last_became_active":"2024-06-29T15:58:21.553278+0000","last_became_peered":"2024-06-29T15:58:21.553278+0000","last_unstale":"2024-06-29T15:58:34.626571+0000","last_undegraded":"2024-06-29T15:58:34.626571+0000","last_fullsized":"2024-06-29T15:58:34.626571+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:39.179054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"242'8","reported_seq":28,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629263+0000","last_change":"2024-06-29T15:58:20.557583+0000","last_active":"2024-06-29T15:58:34.629263+0000","last_peered":"2024-06-29T15:58:34.629263+0000","last_clean":"2024-06-29T15:58:34.629263+0000","last_became_active":"2024-06-29T15:58:20.557274+0000","last_became_peered":"2024-06-29T15:58:20.557274+0000","last_unstale":"2024-06-29T15:58:34.629263+0000","last_undegraded":"2024-06-29T15:58:34.629263+0000","last_fullsized":"2024-06-29T15:58:34.629263+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:38.435050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649709+0000","last_change":"2024-06-29T15:58:34.642355+0000","last_active":"2024-06-29T15:58:34.649709+0000","last_peered":"2024-06-29T15:58:34.649709+0000","last_clean":"2024-06-29T15:58:34.649709+0000","last_became_active":"2024-06-29T15:58:34.641999+0000","last_became_peered":"2024-06-29T15:58:34.641999+0000","last_unstale":"2024-06-29T15:58:34.649709+0000","last_undegraded":"2024-06-29T15:58:34.649709+0000","last_fullsized":"2024-06-29T15:58:34.649709+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:44:43.831068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632735+0000","last_change":"2024-06-29T15:58:21.559479+0000","last_active":"2024-06-29T15:58:34.632735+0000","last_peered":"2024-06-29T15:58:34.632735+0000","last_clean":"2024-06-29T15:58:34.632735+0000","last_became_active":"2024-06-29T15:58:21.559128+0000","last_became_peered":"2024-06-29T15:58:21.559128+0000","last_unstale":"2024-06-29T15:58:34.632735+0000","last_undegraded":"2024-06-29T15:58:34.632735+0000","last_fullsized":"2024-06-29T15:58:34.632735+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:28.321473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630359+0000","last_change":"2024-06-29T15:58:20.561589+0000","last_active":"2024-06-29T15:58:34.630359+0000","last_peered":"2024-06-29T15:58:34.630359+0000","last_clean":"2024-06-29T15:58:34.630359+0000","last_became_active":"2024-06-29T15:58:20.561251+0000","last_became_peered":"2024-06-29T15:58:20.561251+0000","last_unstale":"2024-06-29T15:58:34.630359+0000","last_undegraded":"2024-06-29T15:58:34.630359+0000","last_fullsized":"2024-06-29T15:58:34.630359+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:09.657256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640355+0000","last_change":"2024-06-29T15:58:34.637948+0000","last_active":"2024-06-29T15:58:34.640355+0000","last_peered":"2024-06-29T15:58:34.640355+0000","last_clean":"2024-06-29T15:58:34.640355+0000","last_became_active":"2024-06-29T15:58:34.637190+0000","last_became_peered":"2024-06-29T15:58:34.637190+0000","last_unstale":"2024-06-29T15:58:34.640355+0000","last_undegraded":"2024-06-29T15:58:34.640355+0000","last_fullsized":"2024-06-29T15:58:34.640355+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:15.234940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632345+0000","last_change":"2024-06-29T15:58:21.560367+0000","last_active":"2024-06-29T15:58:34.632345+0000","last_peered":"2024-06-29T15:58:34.632345+0000","last_clean":"2024-06-29T15:58:34.632345+0000","last_became_active":"2024-06-29T15:58:21.560056+0000","last_became_peered":"2024-06-29T15:58:21.560056+0000","last_unstale":"2024-06-29T15:58:34.632345+0000","last_undegraded":"2024-06-29T15:58:34.632345+0000","last_fullsized":"2024-06-29T15:58:34.632345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:43.371506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621895+0000","last_change":"2024-06-29T15:58:20.552449+0000","last_active":"2024-06-29T15:58:34.621895+0000","last_peered":"2024-06-29T15:58:34.621895+0000","last_clean":"2024-06-29T15:58:34.621895+0000","last_became_active":"2024-06-29T15:58:20.551202+0000","last_became_peered":"2024-06-29T15:58:20.551202+0000","last_unstale":"2024-06-29T15:58:34.621895+0000","last_undegraded":"2024-06-29T15:58:34.621895+0000","last_fullsized":"2024-06-29T15:58:34.621895+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:35.829408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646661+0000","last_change":"2024-06-29T15:58:34.644777+0000","last_active":"2024-06-29T15:58:34.646661+0000","last_peered":"2024-06-29T15:58:34.646661+0000","last_clean":"2024-06-29T15:58:34.646661+0000","last_became_active":"2024-06-29T15:58:34.644047+0000","last_became_peered":"2024-06-29T15:58:34.644047+0000","last_unstale":"2024-06-29T15:58:34.646661+0000","last_undegraded":"2024-06-29T15:58:34.646661+0000","last_fullsized":"2024-06-29T15:58:34.646661+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:57.533609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630674+0000","last_change":"2024-06-29T15:58:21.566336+0000","last_active":"2024-06-29T15:58:34.630674+0000","last_peered":"2024-06-29T15:58:34.630674+0000","last_clean":"2024-06-29T15:58:34.630674+0000","last_became_active":"2024-06-29T15:58:21.566003+0000","last_became_peered":"2024-06-29T15:58:21.566003+0000","last_unstale":"2024-06-29T15:58:34.630674+0000","last_undegraded":"2024-06-29T15:58:34.630674+0000","last_fullsized":"2024-06-29T15:58:34.630674+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:46:35.306688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621874+0000","last_change":"2024-06-29T15:58:20.557295+0000","last_active":"2024-06-29T15:58:34.621874+0000","last_peered":"2024-06-29T15:58:34.621874+0000","last_clean":"2024-06-29T15:58:34.621874+0000","last_became_active":"2024-06-29T15:58:20.556868+0000","last_became_peered":"2024-06-29T15:58:20.556868+0000","last_unstale":"2024-06-29T15:58:34.621874+0000","last_undegraded":"2024-06-29T15:58:34.621874+0000","last_fullsized":"2024-06-29T15:58:34.621874+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:31.226218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.638849+0000","last_change":"2024-06-29T15:58:34.635098+0000","last_active":"2024-06-29T15:58:34.638849+0000","last_peered":"2024-06-29T15:58:34.638849+0000","last_clean":"2024-06-29T15:58:34.638849+0000","last_became_active":"2024-06-29T15:58:34.634799+0000","last_became_peered":"2024-06-29T15:58:34.634799+0000","last_unstale":"2024-06-29T15:58:34.638849+0000","last_undegraded":"2024-06-29T15:58:34.638849+0000","last_fullsized":"2024-06-29T15:58:34.638849+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:15:09.575512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622802+0000","last_change":"2024-06-29T15:58:21.565109+0000","last_active":"2024-06-29T15:58:34.622802+0000","last_peered":"2024-06-29T15:58:34.622802+0000","last_clean":"2024-06-29T15:58:34.622802+0000","last_became_active":"2024-06-29T15:58:21.564764+0000","last_became_peered":"2024-06-29T15:58:21.564764+0000","last_unstale":"2024-06-29T15:58:34.622802+0000","last_undegraded":"2024-06-29T15:58:34.622802+0000","last_fullsized":"2024-06-29T15:58:34.622802+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:43:24.997613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"241'8","reported_seq":28,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629994+0000","last_change":"2024-06-29T15:58:20.562144+0000","last_active":"2024-06-29T15:58:34.629994+0000","last_peered":"2024-06-29T15:58:34.629994+0000","last_clean":"2024-06-29T15:58:34.629994+0000","last_became_active":"2024-06-29T15:58:20.561879+0000","last_became_peered":"2024-06-29T15:58:20.561879+0000","last_unstale":"2024-06-29T15:58:34.629994+0000","last_undegraded":"2024-06-29T15:58:34.629994+0000","last_fullsized":"2024-06-29T15:58:34.629994+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:28.961624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658882+0000","last_change":"2024-06-29T15:58:34.653641+0000","last_active":"2024-06-29T15:58:34.658882+0000","last_peered":"2024-06-29T15:58:34.658882+0000","last_clean":"2024-06-29T15:58:34.658882+0000","last_became_active":"2024-06-29T15:58:34.647800+0000","last_became_peered":"2024-06-29T15:58:34.647800+0000","last_unstale":"2024-06-29T15:58:34.658882+0000","last_undegraded":"2024-06-29T15:58:34.658882+0000","last_fullsized":"2024-06-29T15:58:34.658882+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:11:51.300578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622216+0000","last_change":"2024-06-29T15:58:21.546206+0000","last_active":"2024-06-29T15:58:34.622216+0000","last_peered":"2024-06-29T15:58:34.622216+0000","last_clean":"2024-06-29T15:58:34.622216+0000","last_became_active":"2024-06-29T15:58:21.545742+0000","last_became_peered":"2024-06-29T15:58:21.545742+0000","last_unstale":"2024-06-29T15:58:34.622216+0000","last_undegraded":"2024-06-29T15:58:34.622216+0000","last_fullsized":"2024-06-29T15:58:34.622216+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:57.523444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630822+0000","last_change":"2024-06-29T15:58:20.555827+0000","last_active":"2024-06-29T15:58:34.630822+0000","last_peered":"2024-06-29T15:58:34.630822+0000","last_clean":"2024-06-29T15:58:34.630822+0000","last_became_active":"2024-06-29T15:58:20.555506+0000","last_became_peered":"2024-06-29T15:58:20.555506+0000","last_unstale":"2024-06-29T15:58:34.630822+0000","last_undegraded":"2024-06-29T15:58:34.630822+0000","last_fullsized":"2024-06-29T15:58:34.630822+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:33:20.721321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651461+0000","last_change":"2024-06-29T15:58:34.642428+0000","last_active":"2024-06-29T15:58:34.651461+0000","last_peered":"2024-06-29T15:58:34.651461+0000","last_clean":"2024-06-29T15:58:34.651461+0000","last_became_active":"2024-06-29T15:58:34.641271+0000","last_became_peered":"2024-06-29T15:58:34.641271+0000","last_unstale":"2024-06-29T15:58:34.651461+0000","last_undegraded":"2024-06-29T15:58:34.651461+0000","last_fullsized":"2024-06-29T15:58:34.651461+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:52:52.231569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621719+0000","last_change":"2024-06-29T15:58:21.558057+0000","last_active":"2024-06-29T15:58:34.621719+0000","last_peered":"2024-06-29T15:58:34.621719+0000","last_clean":"2024-06-29T15:58:34.621719+0000","last_became_active":"2024-06-29T15:58:21.557750+0000","last_became_peered":"2024-06-29T15:58:21.557750+0000","last_unstale":"2024-06-29T15:58:34.621719+0000","last_undegraded":"2024-06-29T15:58:34.621719+0000","last_fullsized":"2024-06-29T15:58:34.621719+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:35.875818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631966+0000","last_change":"2024-06-29T15:58:20.561995+0000","last_active":"2024-06-29T15:58:34.631966+0000","last_peered":"2024-06-29T15:58:34.631966+0000","last_clean":"2024-06-29T15:58:34.631966+0000","last_became_active":"2024-06-29T15:58:20.560719+0000","last_became_peered":"2024-06-29T15:58:20.560719+0000","last_unstale":"2024-06-29T15:58:34.631966+0000","last_undegraded":"2024-06-29T15:58:34.631966+0000","last_fullsized":"2024-06-29T15:58:34.631966+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:59.752024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.644470+0000","last_change":"2024-06-29T15:58:34.643196+0000","last_active":"2024-06-29T15:58:34.644470+0000","last_peered":"2024-06-29T15:58:34.644470+0000","last_clean":"2024-06-29T15:58:34.644470+0000","last_became_active":"2024-06-29T15:58:34.642891+0000","last_became_peered":"2024-06-29T15:58:34.642891+0000","last_unstale":"2024-06-29T15:58:34.644470+0000","last_undegraded":"2024-06-29T15:58:34.644470+0000","last_fullsized":"2024-06-29T15:58:34.644470+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:24.611850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631308+0000","last_change":"2024-06-29T15:58:21.552894+0000","last_active":"2024-06-29T15:58:34.631308+0000","last_peered":"2024-06-29T15:58:34.631308+0000","last_clean":"2024-06-29T15:58:34.631308+0000","last_became_active":"2024-06-29T15:58:21.552487+0000","last_became_peered":"2024-06-29T15:58:21.552487+0000","last_unstale":"2024-06-29T15:58:34.631308+0000","last_undegraded":"2024-06-29T15:58:34.631308+0000","last_fullsized":"2024-06-29T15:58:34.631308+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:49.309450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629783+0000","last_change":"2024-06-29T15:58:20.544642+0000","last_active":"2024-06-29T15:58:34.629783+0000","last_peered":"2024-06-29T15:58:34.629783+0000","last_clean":"2024-06-29T15:58:34.629783+0000","last_became_active":"2024-06-29T15:58:20.544356+0000","last_became_peered":"2024-06-29T15:58:20.544356+0000","last_unstale":"2024-06-29T15:58:34.629783+0000","last_undegraded":"2024-06-29T15:58:34.629783+0000","last_fullsized":"2024-06-29T15:58:34.629783+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:38.143805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.660730+0000","last_change":"2024-06-29T15:58:34.658866+0000","last_active":"2024-06-29T15:58:34.660730+0000","last_peered":"2024-06-29T15:58:34.660730+0000","last_clean":"2024-06-29T15:58:34.660730+0000","last_became_active":"2024-06-29T15:58:34.658422+0000","last_became_peered":"2024-06-29T15:58:34.658422+0000","last_unstale":"2024-06-29T15:58:34.660730+0000","last_undegraded":"2024-06-29T15:58:34.660730+0000","last_fullsized":"2024-06-29T15:58:34.660730+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:33.087931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.620595+0000","last_change":"2024-06-29T15:58:21.560349+0000","last_active":"2024-06-29T15:58:34.620595+0000","last_peered":"2024-06-29T15:58:34.620595+0000","last_clean":"2024-06-29T15:58:34.620595+0000","last_became_active":"2024-06-29T15:58:21.559987+0000","last_became_peered":"2024-06-29T15:58:21.559987+0000","last_unstale":"2024-06-29T15:58:34.620595+0000","last_undegraded":"2024-06-29T15:58:34.620595+0000","last_fullsized":"2024-06-29T15:58:34.620595+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:20.317519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.634434+0000","last_change":"2024-06-29T15:58:20.551598+0000","last_active":"2024-06-29T15:58:34.634434+0000","last_peered":"2024-06-29T15:58:34.634434+0000","last_clean":"2024-06-29T15:58:34.634434+0000","last_became_active":"2024-06-29T15:58:20.551266+0000","last_became_peered":"2024-06-29T15:58:20.551266+0000","last_unstale":"2024-06-29T15:58:34.634434+0000","last_undegraded":"2024-06-29T15:58:34.634434+0000","last_fullsized":"2024-06-29T15:58:34.634434+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:56:15.731511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625949+0000","last_change":"2024-06-29T15:58:34.625489+0000","last_active":"2024-06-29T15:58:34.625949+0000","last_peered":"2024-06-29T15:58:34.625949+0000","last_clean":"2024-06-29T15:58:34.625949+0000","last_became_active":"2024-06-29T15:58:34.625171+0000","last_became_peered":"2024-06-29T15:58:34.625171+0000","last_unstale":"2024-06-29T15:58:34.625949+0000","last_undegraded":"2024-06-29T15:58:34.625949+0000","last_fullsized":"2024-06-29T15:58:34.625949+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:59.352335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619148+0000","last_change":"2024-06-29T15:58:21.562779+0000","last_active":"2024-06-29T15:58:34.619148+0000","last_peered":"2024-06-29T15:58:34.619148+0000","last_clean":"2024-06-29T15:58:34.619148+0000","last_became_active":"2024-06-29T15:58:21.562503+0000","last_became_peered":"2024-06-29T15:58:21.562503+0000","last_unstale":"2024-06-29T15:58:34.619148+0000","last_undegraded":"2024-06-29T15:58:34.619148+0000","last_fullsized":"2024-06-29T15:58:34.619148+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:28.461061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.620737+0000","last_change":"2024-06-29T15:58:20.559588+0000","last_active":"2024-06-29T15:58:34.620737+0000","last_peered":"2024-06-29T15:58:34.620737+0000","last_clean":"2024-06-29T15:58:34.620737+0000","last_became_active":"2024-06-29T15:58:20.559245+0000","last_became_peered":"2024-06-29T15:58:20.559245+0000","last_unstale":"2024-06-29T15:58:34.620737+0000","last_undegraded":"2024-06-29T15:58:34.620737+0000","last_fullsized":"2024-06-29T15:58:34.620737+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:40.258636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649102+0000","last_change":"2024-06-29T15:58:34.648358+0000","last_active":"2024-06-29T15:58:34.649102+0000","last_peered":"2024-06-29T15:58:34.649102+0000","last_clean":"2024-06-29T15:58:34.649102+0000","last_became_active":"2024-06-29T15:58:34.648049+0000","last_became_peered":"2024-06-29T15:58:34.648049+0000","last_unstale":"2024-06-29T15:58:34.649102+0000","last_undegraded":"2024-06-29T15:58:34.649102+0000","last_fullsized":"2024-06-29T15:58:34.649102+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:13.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629190+0000","last_change":"2024-06-29T15:58:21.553384+0000","last_active":"2024-06-29T15:58:34.629190+0000","last_peered":"2024-06-29T15:58:34.629190+0000","last_clean":"2024-06-29T15:58:34.629190+0000","last_became_active":"2024-06-29T15:58:21.551240+0000","last_became_peered":"2024-06-29T15:58:21.551240+0000","last_unstale":"2024-06-29T15:58:34.629190+0000","last_undegraded":"2024-06-29T15:58:34.629190+0000","last_fullsized":"2024-06-29T15:58:34.629190+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:38.040891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628442+0000","last_change":"2024-06-29T15:58:20.557247+0000","last_active":"2024-06-29T15:58:34.628442+0000","last_peered":"2024-06-29T15:58:34.628442+0000","last_clean":"2024-06-29T15:58:34.628442+0000","last_became_active":"2024-06-29T15:58:20.556889+0000","last_became_peered":"2024-06-29T15:58:20.556889+0000","last_unstale":"2024-06-29T15:58:34.628442+0000","last_undegraded":"2024-06-29T15:58:34.628442+0000","last_fullsized":"2024-06-29T15:58:34.628442+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:48.301087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.645452+0000","last_change":"2024-06-29T15:58:34.641442+0000","last_active":"2024-06-29T15:58:34.645452+0000","last_peered":"2024-06-29T15:58:34.645452+0000","last_clean":"2024-06-29T15:58:34.645452+0000","last_became_active":"2024-06-29T15:58:34.640887+0000","last_became_peered":"2024-06-29T15:58:34.640887+0000","last_unstale":"2024-06-29T15:58:34.645452+0000","last_undegraded":"2024-06-29T15:58:34.645452+0000","last_fullsized":"2024-06-29T15:58:34.645452+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:52:03.341429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629393+0000","last_change":"2024-06-29T15:58:21.558641+0000","last_active":"2024-06-29T15:58:34.629393+0000","last_peered":"2024-06-29T15:58:34.629393+0000","last_clean":"2024-06-29T15:58:34.629393+0000","last_became_active":"2024-06-29T15:58:21.558377+0000","last_became_peered":"2024-06-29T15:58:21.558377+0000","last_unstale":"2024-06-29T15:58:34.629393+0000","last_undegraded":"2024-06-29T15:58:34.629393+0000","last_fullsized":"2024-06-29T15:58:34.629393+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:09.589023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628333+0000","last_change":"2024-06-29T15:58:20.543880+0000","last_active":"2024-06-29T15:58:34.628333+0000","last_peered":"2024-06-29T15:58:34.628333+0000","last_clean":"2024-06-29T15:58:34.628333+0000","last_became_active":"2024-06-29T15:58:20.543365+0000","last_became_peered":"2024-06-29T15:58:20.543365+0000","last_unstale":"2024-06-29T15:58:34.628333+0000","last_undegraded":"2024-06-29T15:58:34.628333+0000","last_fullsized":"2024-06-29T15:58:34.628333+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:18:31.509521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.641843+0000","last_change":"2024-06-29T15:58:34.640187+0000","last_active":"2024-06-29T15:58:34.641843+0000","last_peered":"2024-06-29T15:58:34.641843+0000","last_clean":"2024-06-29T15:58:34.641843+0000","last_became_active":"2024-06-29T15:58:34.639854+0000","last_became_peered":"2024-06-29T15:58:34.639854+0000","last_unstale":"2024-06-29T15:58:34.641843+0000","last_undegraded":"2024-06-29T15:58:34.641843+0000","last_fullsized":"2024-06-29T15:58:34.641843+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:17.260957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629667+0000","last_change":"2024-06-29T15:58:21.555113+0000","last_active":"2024-06-29T15:58:34.629667+0000","last_peered":"2024-06-29T15:58:34.629667+0000","last_clean":"2024-06-29T15:58:34.629667+0000","last_became_active":"2024-06-29T15:58:21.553362+0000","last_became_peered":"2024-06-29T15:58:21.553362+0000","last_unstale":"2024-06-29T15:58:34.629667+0000","last_undegraded":"2024-06-29T15:58:34.629667+0000","last_fullsized":"2024-06-29T15:58:34.629667+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:18.837956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.623429+0000","last_change":"2024-06-29T15:58:20.551665+0000","last_active":"2024-06-29T15:58:34.623429+0000","last_peered":"2024-06-29T15:58:34.623429+0000","last_clean":"2024-06-29T15:58:34.623429+0000","last_became_active":"2024-06-29T15:58:20.551347+0000","last_became_peered":"2024-06-29T15:58:20.551347+0000","last_unstale":"2024-06-29T15:58:34.623429+0000","last_undegraded":"2024-06-29T15:58:34.623429+0000","last_fullsized":"2024-06-29T15:58:34.623429+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:22:40.223323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646126+0000","last_change":"2024-06-29T15:58:34.643477+0000","last_active":"2024-06-29T15:58:34.646126+0000","last_peered":"2024-06-29T15:58:34.646126+0000","last_clean":"2024-06-29T15:58:34.646126+0000","last_became_active":"2024-06-29T15:58:34.643144+0000","last_became_peered":"2024-06-29T15:58:34.643144+0000","last_unstale":"2024-06-29T15:58:34.646126+0000","last_undegraded":"2024-06-29T15:58:34.646126+0000","last_fullsized":"2024-06-29T15:58:34.646126+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:18.918717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628924+0000","last_change":"2024-06-29T15:58:21.561571+0000","last_active":"2024-06-29T15:58:34.628924+0000","last_peered":"2024-06-29T15:58:34.628924+0000","last_clean":"2024-06-29T15:58:34.628924+0000","last_became_active":"2024-06-29T15:58:21.561042+0000","last_became_peered":"2024-06-29T15:58:21.561042+0000","last_unstale":"2024-06-29T15:58:34.628924+0000","last_undegraded":"2024-06-29T15:58:34.628924+0000","last_fullsized":"2024-06-29T15:58:34.628924+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:34.199222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.620265+0000","last_change":"2024-06-29T15:58:20.563978+0000","last_active":"2024-06-29T15:58:34.620265+0000","last_peered":"2024-06-29T15:58:34.620265+0000","last_clean":"2024-06-29T15:58:34.620265+0000","last_became_active":"2024-06-29T15:58:20.563680+0000","last_became_peered":"2024-06-29T15:58:20.563680+0000","last_unstale":"2024-06-29T15:58:34.620265+0000","last_undegraded":"2024-06-29T15:58:34.620265+0000","last_fullsized":"2024-06-29T15:58:34.620265+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:27.903522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.638298+0000","last_change":"2024-06-29T15:58:34.634993+0000","last_active":"2024-06-29T15:58:34.638298+0000","last_peered":"2024-06-29T15:58:34.638298+0000","last_clean":"2024-06-29T15:58:34.638298+0000","last_became_active":"2024-06-29T15:58:34.634658+0000","last_became_peered":"2024-06-29T15:58:34.634658+0000","last_unstale":"2024-06-29T15:58:34.638298+0000","last_undegraded":"2024-06-29T15:58:34.638298+0000","last_fullsized":"2024-06-29T15:58:34.638298+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:27.552901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628290+0000","last_change":"2024-06-29T15:58:21.565805+0000","last_active":"2024-06-29T15:58:34.628290+0000","last_peered":"2024-06-29T15:58:34.628290+0000","last_clean":"2024-06-29T15:58:34.628290+0000","last_became_active":"2024-06-29T15:58:21.564982+0000","last_became_peered":"2024-06-29T15:58:21.564982+0000","last_unstale":"2024-06-29T15:58:34.628290+0000","last_undegraded":"2024-06-29T15:58:34.628290+0000","last_fullsized":"2024-06-29T15:58:34.628290+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:18.217838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"242'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622735+0000","last_change":"2024-06-29T15:58:20.554719+0000","last_active":"2024-06-29T15:58:34.622735+0000","last_peered":"2024-06-29T15:58:34.622735+0000","last_clean":"2024-06-29T15:58:34.622735+0000","last_became_active":"2024-06-29T15:58:20.554423+0000","last_became_peered":"2024-06-29T15:58:20.554423+0000","last_unstale":"2024-06-29T15:58:34.622735+0000","last_undegraded":"2024-06-29T15:58:34.622735+0000","last_fullsized":"2024-06-29T15:58:34.622735+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:36.112729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.655025+0000","last_change":"2024-06-29T15:58:34.648450+0000","last_active":"2024-06-29T15:58:34.655025+0000","last_peered":"2024-06-29T15:58:34.655025+0000","last_clean":"2024-06-29T15:58:34.655025+0000","last_became_active":"2024-06-29T15:58:34.648075+0000","last_became_peered":"2024-06-29T15:58:34.648075+0000","last_unstale":"2024-06-29T15:58:34.655025+0000","last_undegraded":"2024-06-29T15:58:34.655025+0000","last_fullsized":"2024-06-29T15:58:34.655025+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:30.699858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618892+0000","last_change":"2024-06-29T15:58:21.559427+0000","last_active":"2024-06-29T15:58:34.618892+0000","last_peered":"2024-06-29T15:58:34.618892+0000","last_clean":"2024-06-29T15:58:34.618892+0000","last_became_active":"2024-06-29T15:58:21.558942+0000","last_became_peered":"2024-06-29T15:58:21.558942+0000","last_unstale":"2024-06-29T15:58:34.618892+0000","last_undegraded":"2024-06-29T15:58:34.618892+0000","last_fullsized":"2024-06-29T15:58:34.618892+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:36.096648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626387+0000","last_change":"2024-06-29T15:58:20.559651+0000","last_active":"2024-06-29T15:58:34.626387+0000","last_peered":"2024-06-29T15:58:34.626387+0000","last_clean":"2024-06-29T15:58:34.626387+0000","last_became_active":"2024-06-29T15:58:20.559323+0000","last_became_peered":"2024-06-29T15:58:20.559323+0000","last_unstale":"2024-06-29T15:58:34.626387+0000","last_undegraded":"2024-06-29T15:58:34.626387+0000","last_fullsized":"2024-06-29T15:58:34.626387+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:18:36.258363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.637600+0000","last_change":"2024-06-29T15:58:34.636916+0000","last_active":"2024-06-29T15:58:34.637600+0000","last_peered":"2024-06-29T15:58:34.637600+0000","last_clean":"2024-06-29T15:58:34.637600+0000","last_became_active":"2024-06-29T15:58:34.636611+0000","last_became_peered":"2024-06-29T15:58:34.636611+0000","last_unstale":"2024-06-29T15:58:34.637600+0000","last_undegraded":"2024-06-29T15:58:34.637600+0000","last_fullsized":"2024-06-29T15:58:34.637600+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:21.903198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621677+0000","last_change":"2024-06-29T15:58:21.545630+0000","last_active":"2024-06-29T15:58:34.621677+0000","last_peered":"2024-06-29T15:58:34.621677+0000","last_clean":"2024-06-29T15:58:34.621677+0000","last_became_active":"2024-06-29T15:58:21.545301+0000","last_became_peered":"2024-06-29T15:58:21.545301+0000","last_unstale":"2024-06-29T15:58:34.621677+0000","last_undegraded":"2024-06-29T15:58:34.621677+0000","last_fullsized":"2024-06-29T15:58:34.621677+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:36.336990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627349+0000","last_change":"2024-06-29T15:58:20.556695+0000","last_active":"2024-06-29T15:58:34.627349+0000","last_peered":"2024-06-29T15:58:34.627349+0000","last_clean":"2024-06-29T15:58:34.627349+0000","last_became_active":"2024-06-29T15:58:20.556053+0000","last_became_peered":"2024-06-29T15:58:20.556053+0000","last_unstale":"2024-06-29T15:58:34.627349+0000","last_undegraded":"2024-06-29T15:58:34.627349+0000","last_fullsized":"2024-06-29T15:58:34.627349+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:15.325464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658955+0000","last_change":"2024-06-29T15:58:34.648233+0000","last_active":"2024-06-29T15:58:34.658955+0000","last_peered":"2024-06-29T15:58:34.658955+0000","last_clean":"2024-06-29T15:58:34.658955+0000","last_became_active":"2024-06-29T15:58:34.647411+0000","last_became_peered":"2024-06-29T15:58:34.647411+0000","last_unstale":"2024-06-29T15:58:34.658955+0000","last_undegraded":"2024-06-29T15:58:34.658955+0000","last_fullsized":"2024-06-29T15:58:34.658955+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:12.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626776+0000","last_change":"2024-06-29T15:58:21.565888+0000","last_active":"2024-06-29T15:58:34.626776+0000","last_peered":"2024-06-29T15:58:34.626776+0000","last_clean":"2024-06-29T15:58:34.626776+0000","last_became_active":"2024-06-29T15:58:21.565418+0000","last_became_peered":"2024-06-29T15:58:21.565418+0000","last_unstale":"2024-06-29T15:58:34.626776+0000","last_undegraded":"2024-06-29T15:58:34.626776+0000","last_fullsized":"2024-06-29T15:58:34.626776+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:57.843559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628122+0000","last_change":"2024-06-29T15:58:20.557670+0000","last_active":"2024-06-29T15:58:34.628122+0000","last_peered":"2024-06-29T15:58:34.628122+0000","last_clean":"2024-06-29T15:58:34.628122+0000","last_became_active":"2024-06-29T15:58:20.557366+0000","last_became_peered":"2024-06-29T15:58:20.557366+0000","last_unstale":"2024-06-29T15:58:34.628122+0000","last_undegraded":"2024-06-29T15:58:34.628122+0000","last_fullsized":"2024-06-29T15:58:34.628122+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:08.866031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662407+0000","last_change":"2024-06-29T15:58:34.662002+0000","last_active":"2024-06-29T15:58:34.662407+0000","last_peered":"2024-06-29T15:58:34.662407+0000","last_clean":"2024-06-29T15:58:34.662407+0000","last_became_active":"2024-06-29T15:58:34.661589+0000","last_became_peered":"2024-06-29T15:58:34.661589+0000","last_unstale":"2024-06-29T15:58:34.662407+0000","last_undegraded":"2024-06-29T15:58:34.662407+0000","last_fullsized":"2024-06-29T15:58:34.662407+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:30.075819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626174+0000","last_change":"2024-06-29T15:58:21.562317+0000","last_active":"2024-06-29T15:58:34.626174+0000","last_peered":"2024-06-29T15:58:34.626174+0000","last_clean":"2024-06-29T15:58:34.626174+0000","last_became_active":"2024-06-29T15:58:21.561968+0000","last_became_peered":"2024-06-29T15:58:21.561968+0000","last_unstale":"2024-06-29T15:58:34.626174+0000","last_undegraded":"2024-06-29T15:58:34.626174+0000","last_fullsized":"2024-06-29T15:58:34.626174+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.632416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622286+0000","last_change":"2024-06-29T15:58:20.547790+0000","last_active":"2024-06-29T15:58:34.622286+0000","last_peered":"2024-06-29T15:58:34.622286+0000","last_clean":"2024-06-29T15:58:34.622286+0000","last_became_active":"2024-06-29T15:58:20.547481+0000","last_became_peered":"2024-06-29T15:58:20.547481+0000","last_unstale":"2024-06-29T15:58:34.622286+0000","last_undegraded":"2024-06-29T15:58:34.622286+0000","last_fullsized":"2024-06-29T15:58:34.622286+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:32.913616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.652099+0000","last_change":"2024-06-29T15:58:34.649154+0000","last_active":"2024-06-29T15:58:34.652099+0000","last_peered":"2024-06-29T15:58:34.652099+0000","last_clean":"2024-06-29T15:58:34.652099+0000","last_became_active":"2024-06-29T15:58:34.648835+0000","last_became_peered":"2024-06-29T15:58:34.648835+0000","last_unstale":"2024-06-29T15:58:34.652099+0000","last_undegraded":"2024-06-29T15:58:34.652099+0000","last_fullsized":"2024-06-29T15:58:34.652099+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:12.584396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631820+0000","last_change":"2024-06-29T15:58:21.553783+0000","last_active":"2024-06-29T15:58:34.631820+0000","last_peered":"2024-06-29T15:58:34.631820+0000","last_clean":"2024-06-29T15:58:34.631820+0000","last_became_active":"2024-06-29T15:58:21.553390+0000","last_became_peered":"2024-06-29T15:58:21.553390+0000","last_unstale":"2024-06-29T15:58:34.631820+0000","last_undegraded":"2024-06-29T15:58:34.631820+0000","last_fullsized":"2024-06-29T15:58:34.631820+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:01.841016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618448+0000","last_change":"2024-06-29T15:58:20.561399+0000","last_active":"2024-06-29T15:58:34.618448+0000","last_peered":"2024-06-29T15:58:34.618448+0000","last_clean":"2024-06-29T15:58:34.618448+0000","last_became_active":"2024-06-29T15:58:20.560716+0000","last_became_peered":"2024-06-29T15:58:20.560716+0000","last_unstale":"2024-06-29T15:58:34.618448+0000","last_undegraded":"2024-06-29T15:58:34.618448+0000","last_fullsized":"2024-06-29T15:58:34.618448+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:42:19.927414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662632+0000","last_change":"2024-06-29T15:58:34.662010+0000","last_active":"2024-06-29T15:58:34.662632+0000","last_peered":"2024-06-29T15:58:34.662632+0000","last_clean":"2024-06-29T15:58:34.662632+0000","last_became_active":"2024-06-29T15:58:34.661637+0000","last_became_peered":"2024-06-29T15:58:34.661637+0000","last_unstale":"2024-06-29T15:58:34.662632+0000","last_undegraded":"2024-06-29T15:58:34.662632+0000","last_fullsized":"2024-06-29T15:58:34.662632+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:10.132577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629876+0000","last_change":"2024-06-29T15:58:21.563096+0000","last_active":"2024-06-29T15:58:34.629876+0000","last_peered":"2024-06-29T15:58:34.629876+0000","last_clean":"2024-06-29T15:58:34.629876+0000","last_became_active":"2024-06-29T15:58:21.562778+0000","last_became_peered":"2024-06-29T15:58:21.562778+0000","last_unstale":"2024-06-29T15:58:34.629876+0000","last_undegraded":"2024-06-29T15:58:34.629876+0000","last_fullsized":"2024-06-29T15:58:34.629876+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:20.808675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627706+0000","last_change":"2024-06-29T15:58:20.554308+0000","last_active":"2024-06-29T15:58:34.627706+0000","last_peered":"2024-06-29T15:58:34.627706+0000","last_clean":"2024-06-29T15:58:34.627706+0000","last_became_active":"2024-06-29T15:58:20.553976+0000","last_became_peered":"2024-06-29T15:58:20.553976+0000","last_unstale":"2024-06-29T15:58:34.627706+0000","last_undegraded":"2024-06-29T15:58:34.627706+0000","last_fullsized":"2024-06-29T15:58:34.627706+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:13:16.241344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651880+0000","last_change":"2024-06-29T15:58:34.642709+0000","last_active":"2024-06-29T15:58:34.651880+0000","last_peered":"2024-06-29T15:58:34.651880+0000","last_clean":"2024-06-29T15:58:34.651880+0000","last_became_active":"2024-06-29T15:58:34.641698+0000","last_became_peered":"2024-06-29T15:58:34.641698+0000","last_unstale":"2024-06-29T15:58:34.651880+0000","last_undegraded":"2024-06-29T15:58:34.651880+0000","last_fullsized":"2024-06-29T15:58:34.651880+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:22.592796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.624166+0000","last_change":"2024-06-29T15:58:21.550432+0000","last_active":"2024-06-29T15:58:34.624166+0000","last_peered":"2024-06-29T15:58:34.624166+0000","last_clean":"2024-06-29T15:58:34.624166+0000","last_became_active":"2024-06-29T15:58:21.550075+0000","last_became_peered":"2024-06-29T15:58:21.550075+0000","last_unstale":"2024-06-29T15:58:34.624166+0000","last_undegraded":"2024-06-29T15:58:34.624166+0000","last_fullsized":"2024-06-29T15:58:34.624166+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:24.427535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627721+0000","last_change":"2024-06-29T15:58:20.561371+0000","last_active":"2024-06-29T15:58:34.627721+0000","last_peered":"2024-06-29T15:58:34.627721+0000","last_clean":"2024-06-29T15:58:34.627721+0000","last_became_active":"2024-06-29T15:58:20.561026+0000","last_became_peered":"2024-06-29T15:58:20.561026+0000","last_unstale":"2024-06-29T15:58:34.627721+0000","last_undegraded":"2024-06-29T15:58:34.627721+0000","last_fullsized":"2024-06-29T15:58:34.627721+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:07.861501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"228'247","reported_seq":1196,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627406+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:58:34.627406+0000","last_peered":"2024-06-29T15:58:34.627406+0000","last_clean":"2024-06-29T15:58:34.627406+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:58:34.627406+0000","last_undegraded":"2024-06-29T15:58:34.627406+0000","last_fullsized":"2024-06-29T15:58:34.627406+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543300,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52812,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3670016,"data_stored":2178415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543301,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64728,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307112,"statfs":{"total":96636764160,"available":96570482688,"internally_reserved":0,"allocated":3092480,"data_stored":1596162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543300,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70276,"kb_used_data":2996,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301564,"statfs":{"total":96636764160,"available":96564801536,"internally_reserved":0,"allocated":3067904,"data_stored":1587923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543299,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3096576,"data_stored":1597698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510597,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48104,"kb_used_data":3036,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323736,"statfs":{"total":96636764160,"available":96587505664,"internally_reserved":0,"allocated":3108864,"data_stored":1599599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576005,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48108,"kb_used_data":3036,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323732,"statfs":{"total":96636764160,"available":96587501568,"internally_reserved":0,"allocated":3108864,"data_stored":1598884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510596,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3088384,"data_stored":1596140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510598,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61140,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310700,"statfs":{"total":96636764160,"available":96574156800,"internally_reserved":0,"allocated":3674112,"data_stored":2181146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:58:38.741 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:58:38.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.890+0000 7f48e0dbe640 1 -- 172.21.15.59:0/342720604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48c8004800 msgr2=0x7f48c8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.890+0000 7f48e0dbe640 1 --2- 172.21.15.59:0/342720604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48c8004800 0x7f48c8004c10 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0d2cd0 tx=0x7f48c8005830 comp rx=0 tx=0).stop 2024-06-29T15:58:38.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.890+0000 7f48e0dbe640 1 -- 172.21.15.59:0/342720604 shutdown_connections 2024-06-29T15:58:38.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.890+0000 7f48e0dbe640 1 --2- 172.21.15.59:0/342720604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48c8004800 0x7f48c8004c10 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.890+0000 7f48e0dbe640 1 -- 172.21.15.59:0/342720604 >> 172.21.15.59:0/342720604 conn(0x7f48dc0d3ee0 msgr2=0x7f48dc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.890+0000 7f48e0dbe640 1 -- 172.21.15.59:0/342720604 shutdown_connections 2024-06-29T15:58:38.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.890+0000 7f48e0dbe640 1 -- 172.21.15.59:0/342720604 wait complete. 2024-06-29T15:58:38.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.892+0000 7f48e0dbe640 1 Processor -- start 2024-06-29T15:58:38.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.892+0000 7f48e0dbe640 1 -- start start 2024-06-29T15:58:38.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48e0dbe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c8004800 0x7f48dc1e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48e0dbe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48dc1e7ef0 0x7f48dc1e8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48e0dbe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc152850 0x7f48dc152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48e0dbe640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc0b3d20 con 0x7f48dc152850 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48e0dbe640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc0b3ba0 con 0x7f48c8004800 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48e0dbe640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc15e8d0 con 0x7f48dc1e7ef0 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48dad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc152850 0x7f48dc152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48dad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc152850 0x7f48dc152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50220/0 (socket says 172.21.15.59:50220) 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.893+0000 7f48dad9d640 1 -- 172.21.15.59:0/3929512574 learned_addr learned my addr 172.21.15.59:0/3929512574 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48da59c640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c8004800 0x7f48dc1e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48d9d9b640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48dc1e7ef0 0x7f48dc1e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48dad9d640 1 -- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48dc1e7ef0 msgr2=0x7f48dc1e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48dad9d640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48dc1e7ef0 0x7f48dc1e8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48dad9d640 1 -- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c8004800 msgr2=0x7f48dc1e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48dad9d640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c8004800 0x7f48dc1e79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48dad9d640 1 -- 172.21.15.59:0/3929512574 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc0c6e70 con 0x7f48dc152850 2024-06-29T15:58:38.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48dad9d640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc152850 0x7f48dc152c60 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0c75c0 tx=0x7f48cc001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:38.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48d2ffd640 1 -- 172.21.15.59:0/3929512574 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc011640 con 0x7f48dc152850 2024-06-29T15:58:38.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48d2ffd640 1 -- 172.21.15.59:0/3929512574 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48cc0117e0 con 0x7f48dc152850 2024-06-29T15:58:38.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48d2ffd640 1 -- 172.21.15.59:0/3929512574 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc011c30 con 0x7f48dc152850 2024-06-29T15:58:38.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48dc1e8840 con 0x7f48dc152850 2024-06-29T15:58:38.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.894+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48dc1e6ed0 con 0x7f48dc152850 2024-06-29T15:58:38.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.895+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48dc154090 con 0x7f48dc152850 2024-06-29T15:58:38.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.896+0000 7f48d2ffd640 1 -- 172.21.15.59:0/3929512574 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f48cc01a020 con 0x7f48dc152850 2024-06-29T15:58:38.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.896+0000 7f48d2ffd640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f48bc076540 0x7f48bc078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.897+0000 7f48d2ffd640 1 -- 172.21.15.59:0/3929512574 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f48cc098230 con 0x7f48dc152850 2024-06-29T15:58:38.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.897+0000 7f48da59c640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f48bc076540 0x7f48bc078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.897+0000 7f48da59c640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f48bc076540 0x7f48bc078a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0d2e40 tx=0x7f48d4009500 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:38.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:38.901+0000 7f48d2ffd640 1 -- 172.21.15.59:0/3929512574 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f48cc060bb0 con 0x7f48dc152850 2024-06-29T15:58:39.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.087+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 --> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f48dc0d2950 con 0x7f48bc076540 2024-06-29T15:58:39.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.096+0000 7f48d2ffd640 1 -- 172.21.15.59:0/3929512574 <== mgr.9710 v2:172.21.15.153:6832/1831136380 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491413 (secure 0 0 0) 0x7f48dc0d2950 con 0x7f48bc076540 2024-06-29T15:58:39.101 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:39.104 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:58:39.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f48bc076540 msgr2=0x7f48bc078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f48bc076540 0x7f48bc078a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0d2e40 tx=0x7f48d4009500 comp rx=0 tx=0).stop 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc152850 msgr2=0x7f48dc152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc152850 0x7f48dc152c60 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0c75c0 tx=0x7f48cc001630 comp rx=0 tx=0).stop 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 shutdown_connections 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f48bc076540 0x7f48bc078a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48dc152850 0x7f48dc152c60 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48dc1e7ef0 0x7f48dc1e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 --2- 172.21.15.59:0/3929512574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48c8004800 0x7f48dc1e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 >> 172.21.15.59:0/3929512574 conn(0x7f48dc0d3ee0 msgr2=0x7f48dc0ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.099+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 shutdown_connections 2024-06-29T15:58:39.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.100+0000 7f48e0dbe640 1 -- 172.21.15.59:0/3929512574 wait complete. 2024-06-29T15:58:39.112 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-06-29T15:58:37.208753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":492,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":299,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":449672,"kb_used_data":25300,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754525048,"statfs":{"total":773094113280,"available":772633649152,"internally_reserved":0,"allocated":25907200,"data_stored":13935967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":434425394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.601083"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.641640+0000","last_change":"2024-06-29T15:58:34.638268+0000","last_active":"2024-06-29T15:58:34.641640+0000","last_peered":"2024-06-29T15:58:34.641640+0000","last_clean":"2024-06-29T15:58:34.641640+0000","last_became_active":"2024-06-29T15:58:34.637938+0000","last_became_peered":"2024-06-29T15:58:34.637938+0000","last_unstale":"2024-06-29T15:58:34.641640+0000","last_undegraded":"2024-06-29T15:58:34.641640+0000","last_fullsized":"2024-06-29T15:58:34.641640+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:43.776145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.648895+0000","last_change":"2024-06-29T15:58:33.632076+0000","last_active":"2024-06-29T15:58:34.648895+0000","last_peered":"2024-06-29T15:58:34.648895+0000","last_clean":"2024-06-29T15:58:34.648895+0000","last_became_active":"2024-06-29T15:58:33.631677+0000","last_became_peered":"2024-06-29T15:58:33.631677+0000","last_unstale":"2024-06-29T15:58:34.648895+0000","last_undegraded":"2024-06-29T15:58:34.648895+0000","last_fullsized":"2024-06-29T15:58:34.648895+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:52:51.054347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629994+0000","last_change":"2024-06-29T15:58:21.556650+0000","last_active":"2024-06-29T15:58:34.629994+0000","last_peered":"2024-06-29T15:58:34.629994+0000","last_clean":"2024-06-29T15:58:34.629994+0000","last_became_active":"2024-06-29T15:58:21.556321+0000","last_became_peered":"2024-06-29T15:58:21.556321+0000","last_unstale":"2024-06-29T15:58:34.629994+0000","last_undegraded":"2024-06-29T15:58:34.629994+0000","last_fullsized":"2024-06-29T15:58:34.629994+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:40:17.694204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.639846+0000","last_change":"2024-06-29T15:58:34.635358+0000","last_active":"2024-06-29T15:58:34.639846+0000","last_peered":"2024-06-29T15:58:34.639846+0000","last_clean":"2024-06-29T15:58:34.639846+0000","last_became_active":"2024-06-29T15:58:34.635052+0000","last_became_peered":"2024-06-29T15:58:34.635052+0000","last_unstale":"2024-06-29T15:58:34.639846+0000","last_undegraded":"2024-06-29T15:58:34.639846+0000","last_fullsized":"2024-06-29T15:58:34.639846+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:13.804732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"245'4","reported_seq":16,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.657305+0000","last_change":"2024-06-29T15:58:33.637644+0000","last_active":"2024-06-29T15:58:34.657305+0000","last_peered":"2024-06-29T15:58:34.657305+0000","last_clean":"2024-06-29T15:58:34.657305+0000","last_became_active":"2024-06-29T15:58:33.636791+0000","last_became_peered":"2024-06-29T15:58:33.636791+0000","last_unstale":"2024-06-29T15:58:34.657305+0000","last_undegraded":"2024-06-29T15:58:34.657305+0000","last_fullsized":"2024-06-29T15:58:34.657305+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:29.799985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631686+0000","last_change":"2024-06-29T15:58:21.553068+0000","last_active":"2024-06-29T15:58:34.631686+0000","last_peered":"2024-06-29T15:58:34.631686+0000","last_clean":"2024-06-29T15:58:34.631686+0000","last_became_active":"2024-06-29T15:58:21.552631+0000","last_became_peered":"2024-06-29T15:58:21.552631+0000","last_unstale":"2024-06-29T15:58:34.631686+0000","last_undegraded":"2024-06-29T15:58:34.631686+0000","last_fullsized":"2024-06-29T15:58:34.631686+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:18.380041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659317+0000","last_change":"2024-06-29T15:58:34.656858+0000","last_active":"2024-06-29T15:58:34.659317+0000","last_peered":"2024-06-29T15:58:34.659317+0000","last_clean":"2024-06-29T15:58:34.659317+0000","last_became_active":"2024-06-29T15:58:34.656529+0000","last_became_peered":"2024-06-29T15:58:34.656529+0000","last_unstale":"2024-06-29T15:58:34.659317+0000","last_undegraded":"2024-06-29T15:58:34.659317+0000","last_fullsized":"2024-06-29T15:58:34.659317+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:23:41.149736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.652787+0000","last_change":"2024-06-29T15:58:33.635736+0000","last_active":"2024-06-29T15:58:34.652787+0000","last_peered":"2024-06-29T15:58:34.652787+0000","last_clean":"2024-06-29T15:58:34.652787+0000","last_became_active":"2024-06-29T15:58:33.635382+0000","last_became_peered":"2024-06-29T15:58:33.635382+0000","last_unstale":"2024-06-29T15:58:34.652787+0000","last_undegraded":"2024-06-29T15:58:34.652787+0000","last_fullsized":"2024-06-29T15:58:34.652787+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:51.972517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619707+0000","last_change":"2024-06-29T15:58:21.553826+0000","last_active":"2024-06-29T15:58:34.619707+0000","last_peered":"2024-06-29T15:58:34.619707+0000","last_clean":"2024-06-29T15:58:34.619707+0000","last_became_active":"2024-06-29T15:58:21.553476+0000","last_became_peered":"2024-06-29T15:58:21.553476+0000","last_unstale":"2024-06-29T15:58:34.619707+0000","last_undegraded":"2024-06-29T15:58:34.619707+0000","last_fullsized":"2024-06-29T15:58:34.619707+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:09:27.869044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.657050+0000","last_change":"2024-06-29T15:58:34.655115+0000","last_active":"2024-06-29T15:58:34.657050+0000","last_peered":"2024-06-29T15:58:34.657050+0000","last_clean":"2024-06-29T15:58:34.657050+0000","last_became_active":"2024-06-29T15:58:34.654817+0000","last_became_peered":"2024-06-29T15:58:34.654817+0000","last_unstale":"2024-06-29T15:58:34.657050+0000","last_undegraded":"2024-06-29T15:58:34.657050+0000","last_fullsized":"2024-06-29T15:58:34.657050+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:12.560261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.661986+0000","last_change":"2024-06-29T15:58:33.643604+0000","last_active":"2024-06-29T15:58:34.661986+0000","last_peered":"2024-06-29T15:58:34.661986+0000","last_clean":"2024-06-29T15:58:34.661986+0000","last_became_active":"2024-06-29T15:58:33.643295+0000","last_became_peered":"2024-06-29T15:58:33.643295+0000","last_unstale":"2024-06-29T15:58:34.661986+0000","last_undegraded":"2024-06-29T15:58:34.661986+0000","last_fullsized":"2024-06-29T15:58:34.661986+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:45.601470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619152+0000","last_change":"2024-06-29T15:58:21.556754+0000","last_active":"2024-06-29T15:58:34.619152+0000","last_peered":"2024-06-29T15:58:34.619152+0000","last_clean":"2024-06-29T15:58:34.619152+0000","last_became_active":"2024-06-29T15:58:21.556399+0000","last_became_peered":"2024-06-29T15:58:21.556399+0000","last_unstale":"2024-06-29T15:58:34.619152+0000","last_undegraded":"2024-06-29T15:58:34.619152+0000","last_fullsized":"2024-06-29T15:58:34.619152+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:53:06.570549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.642247+0000","last_change":"2024-06-29T15:58:34.637540+0000","last_active":"2024-06-29T15:58:34.642247+0000","last_peered":"2024-06-29T15:58:34.642247+0000","last_clean":"2024-06-29T15:58:34.642247+0000","last_became_active":"2024-06-29T15:58:34.637206+0000","last_became_peered":"2024-06-29T15:58:34.637206+0000","last_unstale":"2024-06-29T15:58:34.642247+0000","last_undegraded":"2024-06-29T15:58:34.642247+0000","last_fullsized":"2024-06-29T15:58:34.642247+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:50:54.848045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.656343+0000","last_change":"2024-06-29T15:58:33.646871+0000","last_active":"2024-06-29T15:58:34.656343+0000","last_peered":"2024-06-29T15:58:34.656343+0000","last_clean":"2024-06-29T15:58:34.656343+0000","last_became_active":"2024-06-29T15:58:33.646553+0000","last_became_peered":"2024-06-29T15:58:33.646553+0000","last_unstale":"2024-06-29T15:58:34.656343+0000","last_undegraded":"2024-06-29T15:58:34.656343+0000","last_fullsized":"2024-06-29T15:58:34.656343+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:05:30.320148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621972+0000","last_change":"2024-06-29T15:58:21.558986+0000","last_active":"2024-06-29T15:58:34.621972+0000","last_peered":"2024-06-29T15:58:34.621972+0000","last_clean":"2024-06-29T15:58:34.621972+0000","last_became_active":"2024-06-29T15:58:21.558669+0000","last_became_peered":"2024-06-29T15:58:21.558669+0000","last_unstale":"2024-06-29T15:58:34.621972+0000","last_undegraded":"2024-06-29T15:58:34.621972+0000","last_fullsized":"2024-06-29T15:58:34.621972+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:07.747196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658916+0000","last_change":"2024-06-29T15:58:34.643734+0000","last_active":"2024-06-29T15:58:34.658916+0000","last_peered":"2024-06-29T15:58:34.658916+0000","last_clean":"2024-06-29T15:58:34.658916+0000","last_became_active":"2024-06-29T15:58:34.643426+0000","last_became_peered":"2024-06-29T15:58:34.643426+0000","last_unstale":"2024-06-29T15:58:34.658916+0000","last_undegraded":"2024-06-29T15:58:34.658916+0000","last_fullsized":"2024-06-29T15:58:34.658916+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:55.258468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.647516+0000","last_change":"2024-06-29T15:58:33.638695+0000","last_active":"2024-06-29T15:58:34.647516+0000","last_peered":"2024-06-29T15:58:34.647516+0000","last_clean":"2024-06-29T15:58:34.647516+0000","last_became_active":"2024-06-29T15:58:33.638335+0000","last_became_peered":"2024-06-29T15:58:33.638335+0000","last_unstale":"2024-06-29T15:58:34.647516+0000","last_undegraded":"2024-06-29T15:58:34.647516+0000","last_fullsized":"2024-06-29T15:58:34.647516+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:46.273967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630986+0000","last_change":"2024-06-29T15:58:21.558888+0000","last_active":"2024-06-29T15:58:34.630986+0000","last_peered":"2024-06-29T15:58:34.630986+0000","last_clean":"2024-06-29T15:58:34.630986+0000","last_became_active":"2024-06-29T15:58:21.558578+0000","last_became_peered":"2024-06-29T15:58:21.558578+0000","last_unstale":"2024-06-29T15:58:34.630986+0000","last_undegraded":"2024-06-29T15:58:34.630986+0000","last_fullsized":"2024-06-29T15:58:34.630986+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:30.612798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651912+0000","last_change":"2024-06-29T15:58:34.642968+0000","last_active":"2024-06-29T15:58:34.651912+0000","last_peered":"2024-06-29T15:58:34.651912+0000","last_clean":"2024-06-29T15:58:34.651912+0000","last_became_active":"2024-06-29T15:58:34.642120+0000","last_became_peered":"2024-06-29T15:58:34.642120+0000","last_unstale":"2024-06-29T15:58:34.651912+0000","last_undegraded":"2024-06-29T15:58:34.651912+0000","last_fullsized":"2024-06-29T15:58:34.651912+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:07.025026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618528+0000","last_change":"2024-06-29T15:58:33.636596+0000","last_active":"2024-06-29T15:58:34.618528+0000","last_peered":"2024-06-29T15:58:34.618528+0000","last_clean":"2024-06-29T15:58:34.618528+0000","last_became_active":"2024-06-29T15:58:33.636251+0000","last_became_peered":"2024-06-29T15:58:33.636251+0000","last_unstale":"2024-06-29T15:58:34.618528+0000","last_undegraded":"2024-06-29T15:58:34.618528+0000","last_fullsized":"2024-06-29T15:58:34.618528+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:50.037512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630803+0000","last_change":"2024-06-29T15:58:21.552220+0000","last_active":"2024-06-29T15:58:34.630803+0000","last_peered":"2024-06-29T15:58:34.630803+0000","last_clean":"2024-06-29T15:58:34.630803+0000","last_became_active":"2024-06-29T15:58:21.551911+0000","last_became_peered":"2024-06-29T15:58:21.551911+0000","last_unstale":"2024-06-29T15:58:34.630803+0000","last_undegraded":"2024-06-29T15:58:34.630803+0000","last_fullsized":"2024-06-29T15:58:34.630803+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:20.005949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662613+0000","last_change":"2024-06-29T15:58:34.662000+0000","last_active":"2024-06-29T15:58:34.662613+0000","last_peered":"2024-06-29T15:58:34.662613+0000","last_clean":"2024-06-29T15:58:34.662613+0000","last_became_active":"2024-06-29T15:58:34.661659+0000","last_became_peered":"2024-06-29T15:58:34.661659+0000","last_unstale":"2024-06-29T15:58:34.662613+0000","last_undegraded":"2024-06-29T15:58:34.662613+0000","last_fullsized":"2024-06-29T15:58:34.662613+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:07:15.822268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621645+0000","last_change":"2024-06-29T15:58:33.623668+0000","last_active":"2024-06-29T15:58:34.621645+0000","last_peered":"2024-06-29T15:58:34.621645+0000","last_clean":"2024-06-29T15:58:34.621645+0000","last_became_active":"2024-06-29T15:58:33.623328+0000","last_became_peered":"2024-06-29T15:58:33.623328+0000","last_unstale":"2024-06-29T15:58:34.621645+0000","last_undegraded":"2024-06-29T15:58:34.621645+0000","last_fullsized":"2024-06-29T15:58:34.621645+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:47.691623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622335+0000","last_change":"2024-06-29T15:58:21.558631+0000","last_active":"2024-06-29T15:58:34.622335+0000","last_peered":"2024-06-29T15:58:34.622335+0000","last_clean":"2024-06-29T15:58:34.622335+0000","last_became_active":"2024-06-29T15:58:21.558300+0000","last_became_peered":"2024-06-29T15:58:21.558300+0000","last_unstale":"2024-06-29T15:58:34.622335+0000","last_undegraded":"2024-06-29T15:58:34.622335+0000","last_fullsized":"2024-06-29T15:58:34.622335+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:46.044647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.656459+0000","last_change":"2024-06-29T15:58:34.649980+0000","last_active":"2024-06-29T15:58:34.656459+0000","last_peered":"2024-06-29T15:58:34.656459+0000","last_clean":"2024-06-29T15:58:34.656459+0000","last_became_active":"2024-06-29T15:58:34.649594+0000","last_became_peered":"2024-06-29T15:58:34.649594+0000","last_unstale":"2024-06-29T15:58:34.656459+0000","last_undegraded":"2024-06-29T15:58:34.656459+0000","last_fullsized":"2024-06-29T15:58:34.656459+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:29.253507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622685+0000","last_change":"2024-06-29T15:58:33.629024+0000","last_active":"2024-06-29T15:58:34.622685+0000","last_peered":"2024-06-29T15:58:34.622685+0000","last_clean":"2024-06-29T15:58:34.622685+0000","last_became_active":"2024-06-29T15:58:33.622998+0000","last_became_peered":"2024-06-29T15:58:33.622998+0000","last_unstale":"2024-06-29T15:58:34.622685+0000","last_undegraded":"2024-06-29T15:58:34.622685+0000","last_fullsized":"2024-06-29T15:58:34.622685+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:01.088268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630286+0000","last_change":"2024-06-29T15:58:21.560160+0000","last_active":"2024-06-29T15:58:34.630286+0000","last_peered":"2024-06-29T15:58:34.630286+0000","last_clean":"2024-06-29T15:58:34.630286+0000","last_became_active":"2024-06-29T15:58:21.559735+0000","last_became_peered":"2024-06-29T15:58:21.559735+0000","last_unstale":"2024-06-29T15:58:34.630286+0000","last_undegraded":"2024-06-29T15:58:34.630286+0000","last_fullsized":"2024-06-29T15:58:34.630286+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:33.429459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.652072+0000","last_change":"2024-06-29T15:58:34.643328+0000","last_active":"2024-06-29T15:58:34.652072+0000","last_peered":"2024-06-29T15:58:34.652072+0000","last_clean":"2024-06-29T15:58:34.652072+0000","last_became_active":"2024-06-29T15:58:34.639700+0000","last_became_peered":"2024-06-29T15:58:34.639700+0000","last_unstale":"2024-06-29T15:58:34.652072+0000","last_undegraded":"2024-06-29T15:58:34.652072+0000","last_fullsized":"2024-06-29T15:58:34.652072+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:56.133749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629630+0000","last_change":"2024-06-29T15:58:33.641952+0000","last_active":"2024-06-29T15:58:34.629630+0000","last_peered":"2024-06-29T15:58:34.629630+0000","last_clean":"2024-06-29T15:58:34.629630+0000","last_became_active":"2024-06-29T15:58:33.640950+0000","last_became_peered":"2024-06-29T15:58:33.640950+0000","last_unstale":"2024-06-29T15:58:34.629630+0000","last_undegraded":"2024-06-29T15:58:34.629630+0000","last_fullsized":"2024-06-29T15:58:34.629630+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:20:47.201871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619463+0000","last_change":"2024-06-29T15:58:21.559778+0000","last_active":"2024-06-29T15:58:34.619463+0000","last_peered":"2024-06-29T15:58:34.619463+0000","last_clean":"2024-06-29T15:58:34.619463+0000","last_became_active":"2024-06-29T15:58:21.559442+0000","last_became_peered":"2024-06-29T15:58:21.559442+0000","last_unstale":"2024-06-29T15:58:34.619463+0000","last_undegraded":"2024-06-29T15:58:34.619463+0000","last_fullsized":"2024-06-29T15:58:34.619463+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:45:14.817280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632388+0000","last_change":"2024-06-29T15:58:34.631762+0000","last_active":"2024-06-29T15:58:34.632388+0000","last_peered":"2024-06-29T15:58:34.632388+0000","last_clean":"2024-06-29T15:58:34.632388+0000","last_became_active":"2024-06-29T15:58:34.631450+0000","last_became_peered":"2024-06-29T15:58:34.631450+0000","last_unstale":"2024-06-29T15:58:34.632388+0000","last_undegraded":"2024-06-29T15:58:34.632388+0000","last_fullsized":"2024-06-29T15:58:34.632388+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:42.958627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"245'6","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662647+0000","last_change":"2024-06-29T15:58:33.636484+0000","last_active":"2024-06-29T15:58:34.662647+0000","last_peered":"2024-06-29T15:58:34.662647+0000","last_clean":"2024-06-29T15:58:34.662647+0000","last_became_active":"2024-06-29T15:58:33.635723+0000","last_became_peered":"2024-06-29T15:58:33.635723+0000","last_unstale":"2024-06-29T15:58:34.662647+0000","last_undegraded":"2024-06-29T15:58:34.662647+0000","last_fullsized":"2024-06-29T15:58:34.662647+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:24.142073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619842+0000","last_change":"2024-06-29T15:58:21.560672+0000","last_active":"2024-06-29T15:58:34.619842+0000","last_peered":"2024-06-29T15:58:34.619842+0000","last_clean":"2024-06-29T15:58:34.619842+0000","last_became_active":"2024-06-29T15:58:21.560333+0000","last_became_peered":"2024-06-29T15:58:21.560333+0000","last_unstale":"2024-06-29T15:58:34.619842+0000","last_undegraded":"2024-06-29T15:58:34.619842+0000","last_fullsized":"2024-06-29T15:58:34.619842+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:16.662908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651377+0000","last_change":"2024-06-29T15:58:34.650497+0000","last_active":"2024-06-29T15:58:34.651377+0000","last_peered":"2024-06-29T15:58:34.651377+0000","last_clean":"2024-06-29T15:58:34.651377+0000","last_became_active":"2024-06-29T15:58:34.650188+0000","last_became_peered":"2024-06-29T15:58:34.650188+0000","last_unstale":"2024-06-29T15:58:34.651377+0000","last_undegraded":"2024-06-29T15:58:34.651377+0000","last_fullsized":"2024-06-29T15:58:34.651377+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:56:51.215044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"245'3","reported_seq":15,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.660442+0000","last_change":"2024-06-29T15:58:33.641361+0000","last_active":"2024-06-29T15:58:34.660442+0000","last_peered":"2024-06-29T15:58:34.660442+0000","last_clean":"2024-06-29T15:58:34.660442+0000","last_became_active":"2024-06-29T15:58:33.641006+0000","last_became_peered":"2024-06-29T15:58:33.641006+0000","last_unstale":"2024-06-29T15:58:34.660442+0000","last_undegraded":"2024-06-29T15:58:34.660442+0000","last_fullsized":"2024-06-29T15:58:34.660442+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:47:15.436543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629008+0000","last_change":"2024-06-29T15:58:21.557848+0000","last_active":"2024-06-29T15:58:34.629008+0000","last_peered":"2024-06-29T15:58:34.629008+0000","last_clean":"2024-06-29T15:58:34.629008+0000","last_became_active":"2024-06-29T15:58:21.557516+0000","last_became_peered":"2024-06-29T15:58:21.557516+0000","last_unstale":"2024-06-29T15:58:34.629008+0000","last_undegraded":"2024-06-29T15:58:34.629008+0000","last_fullsized":"2024-06-29T15:58:34.629008+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:26.084704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646917+0000","last_change":"2024-06-29T15:58:34.645224+0000","last_active":"2024-06-29T15:58:34.646917+0000","last_peered":"2024-06-29T15:58:34.646917+0000","last_clean":"2024-06-29T15:58:34.646917+0000","last_became_active":"2024-06-29T15:58:34.644926+0000","last_became_peered":"2024-06-29T15:58:34.644926+0000","last_unstale":"2024-06-29T15:58:34.646917+0000","last_undegraded":"2024-06-29T15:58:34.646917+0000","last_fullsized":"2024-06-29T15:58:34.646917+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:12.560261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630668+0000","last_change":"2024-06-29T15:58:33.636773+0000","last_active":"2024-06-29T15:58:34.630668+0000","last_peered":"2024-06-29T15:58:34.630668+0000","last_clean":"2024-06-29T15:58:34.630668+0000","last_became_active":"2024-06-29T15:58:33.636476+0000","last_became_peered":"2024-06-29T15:58:33.636476+0000","last_unstale":"2024-06-29T15:58:34.630668+0000","last_undegraded":"2024-06-29T15:58:34.630668+0000","last_fullsized":"2024-06-29T15:58:34.630668+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:24.260348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627751+0000","last_change":"2024-06-29T15:58:21.557388+0000","last_active":"2024-06-29T15:58:34.627751+0000","last_peered":"2024-06-29T15:58:34.627751+0000","last_clean":"2024-06-29T15:58:34.627751+0000","last_became_active":"2024-06-29T15:58:21.557098+0000","last_became_peered":"2024-06-29T15:58:21.557098+0000","last_unstale":"2024-06-29T15:58:34.627751+0000","last_undegraded":"2024-06-29T15:58:34.627751+0000","last_fullsized":"2024-06-29T15:58:34.627751+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:27.148409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625828+0000","last_change":"2024-06-29T15:58:34.624723+0000","last_active":"2024-06-29T15:58:34.625828+0000","last_peered":"2024-06-29T15:58:34.625828+0000","last_clean":"2024-06-29T15:58:34.625828+0000","last_became_active":"2024-06-29T15:58:34.624331+0000","last_became_peered":"2024-06-29T15:58:34.624331+0000","last_unstale":"2024-06-29T15:58:34.625828+0000","last_undegraded":"2024-06-29T15:58:34.625828+0000","last_fullsized":"2024-06-29T15:58:34.625828+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:23:41.149736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649825+0000","last_change":"2024-06-29T15:58:33.638685+0000","last_active":"2024-06-29T15:58:34.649825+0000","last_peered":"2024-06-29T15:58:34.649825+0000","last_clean":"2024-06-29T15:58:34.649825+0000","last_became_active":"2024-06-29T15:58:33.638401+0000","last_became_peered":"2024-06-29T15:58:33.638401+0000","last_unstale":"2024-06-29T15:58:34.649825+0000","last_undegraded":"2024-06-29T15:58:34.649825+0000","last_fullsized":"2024-06-29T15:58:34.649825+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:11:50.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628010+0000","last_change":"2024-06-29T15:58:21.567227+0000","last_active":"2024-06-29T15:58:34.628010+0000","last_peered":"2024-06-29T15:58:34.628010+0000","last_clean":"2024-06-29T15:58:34.628010+0000","last_became_active":"2024-06-29T15:58:21.566195+0000","last_became_peered":"2024-06-29T15:58:21.566195+0000","last_unstale":"2024-06-29T15:58:34.628010+0000","last_undegraded":"2024-06-29T15:58:34.628010+0000","last_fullsized":"2024-06-29T15:58:34.628010+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:49.425921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651066+0000","last_change":"2024-06-29T15:58:34.649146+0000","last_active":"2024-06-29T15:58:34.651066+0000","last_peered":"2024-06-29T15:58:34.651066+0000","last_clean":"2024-06-29T15:58:34.651066+0000","last_became_active":"2024-06-29T15:58:34.648790+0000","last_became_peered":"2024-06-29T15:58:34.648790+0000","last_unstale":"2024-06-29T15:58:34.651066+0000","last_undegraded":"2024-06-29T15:58:34.651066+0000","last_fullsized":"2024-06-29T15:58:34.651066+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:48.494647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"245'8","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662440+0000","last_change":"2024-06-29T15:58:33.634549+0000","last_active":"2024-06-29T15:58:34.662440+0000","last_peered":"2024-06-29T15:58:34.662440+0000","last_clean":"2024-06-29T15:58:34.662440+0000","last_became_active":"2024-06-29T15:58:33.634233+0000","last_became_peered":"2024-06-29T15:58:33.634233+0000","last_unstale":"2024-06-29T15:58:34.662440+0000","last_undegraded":"2024-06-29T15:58:34.662440+0000","last_fullsized":"2024-06-29T15:58:34.662440+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:50:41.829436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628404+0000","last_change":"2024-06-29T15:58:21.562321+0000","last_active":"2024-06-29T15:58:34.628404+0000","last_peered":"2024-06-29T15:58:34.628404+0000","last_clean":"2024-06-29T15:58:34.628404+0000","last_became_active":"2024-06-29T15:58:21.561988+0000","last_became_peered":"2024-06-29T15:58:21.561988+0000","last_unstale":"2024-06-29T15:58:34.628404+0000","last_undegraded":"2024-06-29T15:58:34.628404+0000","last_fullsized":"2024-06-29T15:58:34.628404+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:27.579111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.635269+0000","last_change":"2024-06-29T15:58:34.633069+0000","last_active":"2024-06-29T15:58:34.635269+0000","last_peered":"2024-06-29T15:58:34.635269+0000","last_clean":"2024-06-29T15:58:34.635269+0000","last_became_active":"2024-06-29T15:58:34.631861+0000","last_became_peered":"2024-06-29T15:58:34.631861+0000","last_unstale":"2024-06-29T15:58:34.635269+0000","last_undegraded":"2024-06-29T15:58:34.635269+0000","last_fullsized":"2024-06-29T15:58:34.635269+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:33.087931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.624539+0000","last_change":"2024-06-29T15:58:33.633007+0000","last_active":"2024-06-29T15:58:34.624539+0000","last_peered":"2024-06-29T15:58:34.624539+0000","last_clean":"2024-06-29T15:58:34.624539+0000","last_became_active":"2024-06-29T15:58:33.632662+0000","last_became_peered":"2024-06-29T15:58:33.632662+0000","last_unstale":"2024-06-29T15:58:34.624539+0000","last_undegraded":"2024-06-29T15:58:34.624539+0000","last_fullsized":"2024-06-29T15:58:34.624539+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:48:00.558325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618608+0000","last_change":"2024-06-29T15:58:21.561992+0000","last_active":"2024-06-29T15:58:34.618608+0000","last_peered":"2024-06-29T15:58:34.618608+0000","last_clean":"2024-06-29T15:58:34.618608+0000","last_became_active":"2024-06-29T15:58:21.561586+0000","last_became_peered":"2024-06-29T15:58:21.561586+0000","last_unstale":"2024-06-29T15:58:34.618608+0000","last_undegraded":"2024-06-29T15:58:34.618608+0000","last_fullsized":"2024-06-29T15:58:34.618608+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:57.843559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.638627+0000","last_change":"2024-06-29T15:58:34.636779+0000","last_active":"2024-06-29T15:58:34.638627+0000","last_peered":"2024-06-29T15:58:34.638627+0000","last_clean":"2024-06-29T15:58:34.638627+0000","last_became_active":"2024-06-29T15:58:34.636451+0000","last_became_peered":"2024-06-29T15:58:34.636451+0000","last_unstale":"2024-06-29T15:58:34.638627+0000","last_undegraded":"2024-06-29T15:58:34.638627+0000","last_fullsized":"2024-06-29T15:58:34.638627+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:30.807221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.652506+0000","last_change":"2024-06-29T15:58:33.634810+0000","last_active":"2024-06-29T15:58:34.652506+0000","last_peered":"2024-06-29T15:58:34.652506+0000","last_clean":"2024-06-29T15:58:34.652506+0000","last_became_active":"2024-06-29T15:58:33.634402+0000","last_became_peered":"2024-06-29T15:58:33.634402+0000","last_unstale":"2024-06-29T15:58:34.652506+0000","last_undegraded":"2024-06-29T15:58:34.652506+0000","last_fullsized":"2024-06-29T15:58:34.652506+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:45.601470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629846+0000","last_change":"2024-06-29T15:58:21.559279+0000","last_active":"2024-06-29T15:58:34.629846+0000","last_peered":"2024-06-29T15:58:34.629846+0000","last_clean":"2024-06-29T15:58:34.629846+0000","last_became_active":"2024-06-29T15:58:21.559033+0000","last_became_peered":"2024-06-29T15:58:21.559033+0000","last_unstale":"2024-06-29T15:58:34.629846+0000","last_undegraded":"2024-06-29T15:58:34.629846+0000","last_fullsized":"2024-06-29T15:58:34.629846+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:19.231351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.641812+0000","last_change":"2024-06-29T15:58:34.640422+0000","last_active":"2024-06-29T15:58:34.641812+0000","last_peered":"2024-06-29T15:58:34.641812+0000","last_clean":"2024-06-29T15:58:34.641812+0000","last_became_active":"2024-06-29T15:58:34.640011+0000","last_became_peered":"2024-06-29T15:58:34.640011+0000","last_unstale":"2024-06-29T15:58:34.641812+0000","last_undegraded":"2024-06-29T15:58:34.641812+0000","last_fullsized":"2024-06-29T15:58:34.641812+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:43.776145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.657794+0000","last_change":"2024-06-29T15:58:33.639974+0000","last_active":"2024-06-29T15:58:34.657794+0000","last_peered":"2024-06-29T15:58:34.657794+0000","last_clean":"2024-06-29T15:58:34.657794+0000","last_became_active":"2024-06-29T15:58:33.639562+0000","last_became_peered":"2024-06-29T15:58:33.639562+0000","last_unstale":"2024-06-29T15:58:34.657794+0000","last_undegraded":"2024-06-29T15:58:34.657794+0000","last_fullsized":"2024-06-29T15:58:34.657794+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:14:59.418907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619590+0000","last_change":"2024-06-29T15:58:21.561849+0000","last_active":"2024-06-29T15:58:34.619590+0000","last_peered":"2024-06-29T15:58:34.619590+0000","last_clean":"2024-06-29T15:58:34.619590+0000","last_became_active":"2024-06-29T15:58:21.561051+0000","last_became_peered":"2024-06-29T15:58:21.561051+0000","last_unstale":"2024-06-29T15:58:34.619590+0000","last_undegraded":"2024-06-29T15:58:34.619590+0000","last_fullsized":"2024-06-29T15:58:34.619590+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:05:18.245403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646565+0000","last_change":"2024-06-29T15:58:34.644167+0000","last_active":"2024-06-29T15:58:34.646565+0000","last_peered":"2024-06-29T15:58:34.646565+0000","last_clean":"2024-06-29T15:58:34.646565+0000","last_became_active":"2024-06-29T15:58:34.643506+0000","last_became_peered":"2024-06-29T15:58:34.643506+0000","last_unstale":"2024-06-29T15:58:34.646565+0000","last_undegraded":"2024-06-29T15:58:34.646565+0000","last_fullsized":"2024-06-29T15:58:34.646565+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:44:43.831068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632537+0000","last_change":"2024-06-29T15:58:33.634722+0000","last_active":"2024-06-29T15:58:34.632537+0000","last_peered":"2024-06-29T15:58:34.632537+0000","last_clean":"2024-06-29T15:58:34.632537+0000","last_became_active":"2024-06-29T15:58:33.634357+0000","last_became_peered":"2024-06-29T15:58:33.634357+0000","last_unstale":"2024-06-29T15:58:34.632537+0000","last_undegraded":"2024-06-29T15:58:34.632537+0000","last_fullsized":"2024-06-29T15:58:34.632537+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:46.099458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630892+0000","last_change":"2024-06-29T15:58:21.561160+0000","last_active":"2024-06-29T15:58:34.630892+0000","last_peered":"2024-06-29T15:58:34.630892+0000","last_clean":"2024-06-29T15:58:34.630892+0000","last_became_active":"2024-06-29T15:58:21.560810+0000","last_became_peered":"2024-06-29T15:58:21.560810+0000","last_unstale":"2024-06-29T15:58:34.630892+0000","last_undegraded":"2024-06-29T15:58:34.630892+0000","last_fullsized":"2024-06-29T15:58:34.630892+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:02.548677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651228+0000","last_change":"2024-06-29T15:58:34.640872+0000","last_active":"2024-06-29T15:58:34.651228+0000","last_peered":"2024-06-29T15:58:34.651228+0000","last_clean":"2024-06-29T15:58:34.651228+0000","last_became_active":"2024-06-29T15:58:34.639856+0000","last_became_peered":"2024-06-29T15:58:34.639856+0000","last_unstale":"2024-06-29T15:58:34.651228+0000","last_undegraded":"2024-06-29T15:58:34.651228+0000","last_fullsized":"2024-06-29T15:58:34.651228+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:56.937582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"245'1","reported_seq":13,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.661611+0000","last_change":"2024-06-29T15:58:33.645366+0000","last_active":"2024-06-29T15:58:34.661611+0000","last_peered":"2024-06-29T15:58:34.661611+0000","last_clean":"2024-06-29T15:58:34.661611+0000","last_became_active":"2024-06-29T15:58:33.645140+0000","last_became_peered":"2024-06-29T15:58:33.645140+0000","last_unstale":"2024-06-29T15:58:34.661611+0000","last_undegraded":"2024-06-29T15:58:34.661611+0000","last_fullsized":"2024-06-29T15:58:34.661611+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:21.929648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632861+0000","last_change":"2024-06-29T15:58:21.561325+0000","last_active":"2024-06-29T15:58:34.632861+0000","last_peered":"2024-06-29T15:58:34.632861+0000","last_clean":"2024-06-29T15:58:34.632861+0000","last_became_active":"2024-06-29T15:58:21.561021+0000","last_became_peered":"2024-06-29T15:58:21.561021+0000","last_unstale":"2024-06-29T15:58:34.632861+0000","last_undegraded":"2024-06-29T15:58:34.632861+0000","last_fullsized":"2024-06-29T15:58:34.632861+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:25:28.950204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.641302+0000","last_change":"2024-06-29T15:58:34.639508+0000","last_active":"2024-06-29T15:58:34.641302+0000","last_peered":"2024-06-29T15:58:34.641302+0000","last_clean":"2024-06-29T15:58:34.641302+0000","last_became_active":"2024-06-29T15:58:34.639235+0000","last_became_peered":"2024-06-29T15:58:34.639235+0000","last_unstale":"2024-06-29T15:58:34.641302+0000","last_undegraded":"2024-06-29T15:58:34.641302+0000","last_fullsized":"2024-06-29T15:58:34.641302+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:47.106695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618776+0000","last_change":"2024-06-29T15:58:33.643134+0000","last_active":"2024-06-29T15:58:34.618776+0000","last_peered":"2024-06-29T15:58:34.618776+0000","last_clean":"2024-06-29T15:58:34.618776+0000","last_became_active":"2024-06-29T15:58:33.642765+0000","last_became_peered":"2024-06-29T15:58:33.642765+0000","last_unstale":"2024-06-29T15:58:34.618776+0000","last_undegraded":"2024-06-29T15:58:34.618776+0000","last_fullsized":"2024-06-29T15:58:34.618776+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:50.465474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626713+0000","last_change":"2024-06-29T15:58:21.561369+0000","last_active":"2024-06-29T15:58:34.626713+0000","last_peered":"2024-06-29T15:58:34.626713+0000","last_clean":"2024-06-29T15:58:34.626713+0000","last_became_active":"2024-06-29T15:58:21.561021+0000","last_became_peered":"2024-06-29T15:58:21.561021+0000","last_unstale":"2024-06-29T15:58:34.626713+0000","last_undegraded":"2024-06-29T15:58:34.626713+0000","last_fullsized":"2024-06-29T15:58:34.626713+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:35.412665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659284+0000","last_change":"2024-06-29T15:58:34.655983+0000","last_active":"2024-06-29T15:58:34.659284+0000","last_peered":"2024-06-29T15:58:34.659284+0000","last_clean":"2024-06-29T15:58:34.659284+0000","last_became_active":"2024-06-29T15:58:34.655665+0000","last_became_peered":"2024-06-29T15:58:34.655665+0000","last_unstale":"2024-06-29T15:58:34.659284+0000","last_undegraded":"2024-06-29T15:58:34.659284+0000","last_fullsized":"2024-06-29T15:58:34.659284+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:47.281204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.657655+0000","last_change":"2024-06-29T15:58:33.643317+0000","last_active":"2024-06-29T15:58:34.657655+0000","last_peered":"2024-06-29T15:58:34.657655+0000","last_clean":"2024-06-29T15:58:34.657655+0000","last_became_active":"2024-06-29T15:58:33.643011+0000","last_became_peered":"2024-06-29T15:58:33.643011+0000","last_unstale":"2024-06-29T15:58:34.657655+0000","last_undegraded":"2024-06-29T15:58:34.657655+0000","last_fullsized":"2024-06-29T15:58:34.657655+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:11.512035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628018+0000","last_change":"2024-06-29T15:58:21.555481+0000","last_active":"2024-06-29T15:58:34.628018+0000","last_peered":"2024-06-29T15:58:34.628018+0000","last_clean":"2024-06-29T15:58:34.628018+0000","last_became_active":"2024-06-29T15:58:21.553482+0000","last_became_peered":"2024-06-29T15:58:21.553482+0000","last_unstale":"2024-06-29T15:58:34.628018+0000","last_undegraded":"2024-06-29T15:58:34.628018+0000","last_fullsized":"2024-06-29T15:58:34.628018+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:08.904938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659015+0000","last_change":"2024-06-29T15:58:34.647483+0000","last_active":"2024-06-29T15:58:34.659015+0000","last_peered":"2024-06-29T15:58:34.659015+0000","last_clean":"2024-06-29T15:58:34.659015+0000","last_became_active":"2024-06-29T15:58:34.647153+0000","last_became_peered":"2024-06-29T15:58:34.647153+0000","last_unstale":"2024-06-29T15:58:34.659015+0000","last_undegraded":"2024-06-29T15:58:34.659015+0000","last_fullsized":"2024-06-29T15:58:34.659015+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:51.044749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625254+0000","last_change":"2024-06-29T15:58:33.632232+0000","last_active":"2024-06-29T15:58:34.625254+0000","last_peered":"2024-06-29T15:58:34.625254+0000","last_clean":"2024-06-29T15:58:34.625254+0000","last_became_active":"2024-06-29T15:58:33.631892+0000","last_became_peered":"2024-06-29T15:58:33.631892+0000","last_unstale":"2024-06-29T15:58:34.625254+0000","last_undegraded":"2024-06-29T15:58:34.625254+0000","last_fullsized":"2024-06-29T15:58:34.625254+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:32.556085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628518+0000","last_change":"2024-06-29T15:58:21.560413+0000","last_active":"2024-06-29T15:58:34.628518+0000","last_peered":"2024-06-29T15:58:34.628518+0000","last_clean":"2024-06-29T15:58:34.628518+0000","last_became_active":"2024-06-29T15:58:21.560114+0000","last_became_peered":"2024-06-29T15:58:21.560114+0000","last_unstale":"2024-06-29T15:58:34.628518+0000","last_undegraded":"2024-06-29T15:58:34.628518+0000","last_fullsized":"2024-06-29T15:58:34.628518+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:43:35.054888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640650+0000","last_change":"2024-06-29T15:58:34.639095+0000","last_active":"2024-06-29T15:58:34.640650+0000","last_peered":"2024-06-29T15:58:34.640650+0000","last_clean":"2024-06-29T15:58:34.640650+0000","last_became_active":"2024-06-29T15:58:34.638745+0000","last_became_peered":"2024-06-29T15:58:34.638745+0000","last_unstale":"2024-06-29T15:58:34.640650+0000","last_undegraded":"2024-06-29T15:58:34.640650+0000","last_fullsized":"2024-06-29T15:58:34.640650+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:59:06.483477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631469+0000","last_change":"2024-06-29T15:58:33.646999+0000","last_active":"2024-06-29T15:58:34.631469+0000","last_peered":"2024-06-29T15:58:34.631469+0000","last_clean":"2024-06-29T15:58:34.631469+0000","last_became_active":"2024-06-29T15:58:33.646536+0000","last_became_peered":"2024-06-29T15:58:33.646536+0000","last_unstale":"2024-06-29T15:58:34.631469+0000","last_undegraded":"2024-06-29T15:58:34.631469+0000","last_fullsized":"2024-06-29T15:58:34.631469+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:07:21.845128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627655+0000","last_change":"2024-06-29T15:58:21.562168+0000","last_active":"2024-06-29T15:58:34.627655+0000","last_peered":"2024-06-29T15:58:34.627655+0000","last_clean":"2024-06-29T15:58:34.627655+0000","last_became_active":"2024-06-29T15:58:21.555306+0000","last_became_peered":"2024-06-29T15:58:21.555306+0000","last_unstale":"2024-06-29T15:58:34.627655+0000","last_undegraded":"2024-06-29T15:58:34.627655+0000","last_fullsized":"2024-06-29T15:58:34.627655+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:40.193793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649795+0000","last_change":"2024-06-29T15:58:34.649447+0000","last_active":"2024-06-29T15:58:34.649795+0000","last_peered":"2024-06-29T15:58:34.649795+0000","last_clean":"2024-06-29T15:58:34.649795+0000","last_became_active":"2024-06-29T15:58:34.649170+0000","last_became_peered":"2024-06-29T15:58:34.649170+0000","last_unstale":"2024-06-29T15:58:34.649795+0000","last_undegraded":"2024-06-29T15:58:34.649795+0000","last_fullsized":"2024-06-29T15:58:34.649795+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:09.189841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631513+0000","last_change":"2024-06-29T15:58:33.635574+0000","last_active":"2024-06-29T15:58:34.631513+0000","last_peered":"2024-06-29T15:58:34.631513+0000","last_clean":"2024-06-29T15:58:34.631513+0000","last_became_active":"2024-06-29T15:58:33.635259+0000","last_became_peered":"2024-06-29T15:58:33.635259+0000","last_unstale":"2024-06-29T15:58:34.631513+0000","last_undegraded":"2024-06-29T15:58:34.631513+0000","last_fullsized":"2024-06-29T15:58:34.631513+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:41.951390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632590+0000","last_change":"2024-06-29T15:58:21.559133+0000","last_active":"2024-06-29T15:58:34.632590+0000","last_peered":"2024-06-29T15:58:34.632590+0000","last_clean":"2024-06-29T15:58:34.632590+0000","last_became_active":"2024-06-29T15:58:21.558649+0000","last_became_peered":"2024-06-29T15:58:21.558649+0000","last_unstale":"2024-06-29T15:58:34.632590+0000","last_undegraded":"2024-06-29T15:58:34.632590+0000","last_fullsized":"2024-06-29T15:58:34.632590+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:28.201954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646713+0000","last_change":"2024-06-29T15:58:34.645080+0000","last_active":"2024-06-29T15:58:34.646713+0000","last_peered":"2024-06-29T15:58:34.646713+0000","last_clean":"2024-06-29T15:58:34.646713+0000","last_became_active":"2024-06-29T15:58:34.644481+0000","last_became_peered":"2024-06-29T15:58:34.644481+0000","last_unstale":"2024-06-29T15:58:34.646713+0000","last_undegraded":"2024-06-29T15:58:34.646713+0000","last_fullsized":"2024-06-29T15:58:34.646713+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:55.000504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"245'2","reported_seq":14,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.660160+0000","last_change":"2024-06-29T15:58:33.633026+0000","last_active":"2024-06-29T15:58:34.660160+0000","last_peered":"2024-06-29T15:58:34.660160+0000","last_clean":"2024-06-29T15:58:34.660160+0000","last_became_active":"2024-06-29T15:58:33.632699+0000","last_became_peered":"2024-06-29T15:58:33.632699+0000","last_unstale":"2024-06-29T15:58:34.660160+0000","last_undegraded":"2024-06-29T15:58:34.660160+0000","last_fullsized":"2024-06-29T15:58:34.660160+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:41.572497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627434+0000","last_change":"2024-06-29T15:58:21.561677+0000","last_active":"2024-06-29T15:58:34.627434+0000","last_peered":"2024-06-29T15:58:34.627434+0000","last_clean":"2024-06-29T15:58:34.627434+0000","last_became_active":"2024-06-29T15:58:21.561367+0000","last_became_peered":"2024-06-29T15:58:21.561367+0000","last_unstale":"2024-06-29T15:58:34.627434+0000","last_undegraded":"2024-06-29T15:58:34.627434+0000","last_fullsized":"2024-06-29T15:58:34.627434+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:23:28.067754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.643403+0000","last_change":"2024-06-29T15:58:34.641174+0000","last_active":"2024-06-29T15:58:34.643403+0000","last_peered":"2024-06-29T15:58:34.643403+0000","last_clean":"2024-06-29T15:58:34.643403+0000","last_became_active":"2024-06-29T15:58:34.640804+0000","last_became_peered":"2024-06-29T15:58:34.640804+0000","last_unstale":"2024-06-29T15:58:34.643403+0000","last_undegraded":"2024-06-29T15:58:34.643403+0000","last_fullsized":"2024-06-29T15:58:34.643403+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:38:32.263425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626725+0000","last_change":"2024-06-29T15:58:33.643166+0000","last_active":"2024-06-29T15:58:34.626725+0000","last_peered":"2024-06-29T15:58:34.626725+0000","last_clean":"2024-06-29T15:58:34.626725+0000","last_became_active":"2024-06-29T15:58:33.642827+0000","last_became_peered":"2024-06-29T15:58:33.642827+0000","last_unstale":"2024-06-29T15:58:34.626725+0000","last_undegraded":"2024-06-29T15:58:34.626725+0000","last_fullsized":"2024-06-29T15:58:34.626725+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:24.209579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622358+0000","last_change":"2024-06-29T15:58:21.552197+0000","last_active":"2024-06-29T15:58:34.622358+0000","last_peered":"2024-06-29T15:58:34.622358+0000","last_clean":"2024-06-29T15:58:34.622358+0000","last_became_active":"2024-06-29T15:58:21.551846+0000","last_became_peered":"2024-06-29T15:58:21.551846+0000","last_unstale":"2024-06-29T15:58:34.622358+0000","last_undegraded":"2024-06-29T15:58:34.622358+0000","last_fullsized":"2024-06-29T15:58:34.622358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:31.983161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.647181+0000","last_change":"2024-06-29T15:58:34.643697+0000","last_active":"2024-06-29T15:58:34.647181+0000","last_peered":"2024-06-29T15:58:34.647181+0000","last_clean":"2024-06-29T15:58:34.647181+0000","last_became_active":"2024-06-29T15:58:34.641630+0000","last_became_peered":"2024-06-29T15:58:34.641630+0000","last_unstale":"2024-06-29T15:58:34.647181+0000","last_undegraded":"2024-06-29T15:58:34.647181+0000","last_fullsized":"2024-06-29T15:58:34.647181+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:46:48.481424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631486+0000","last_change":"2024-06-29T15:58:33.638350+0000","last_active":"2024-06-29T15:58:34.631486+0000","last_peered":"2024-06-29T15:58:34.631486+0000","last_clean":"2024-06-29T15:58:34.631486+0000","last_became_active":"2024-06-29T15:58:33.638072+0000","last_became_peered":"2024-06-29T15:58:33.638072+0000","last_unstale":"2024-06-29T15:58:34.631486+0000","last_undegraded":"2024-06-29T15:58:34.631486+0000","last_fullsized":"2024-06-29T15:58:34.631486+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:26.233478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629137+0000","last_change":"2024-06-29T15:58:21.560349+0000","last_active":"2024-06-29T15:58:34.629137+0000","last_peered":"2024-06-29T15:58:34.629137+0000","last_clean":"2024-06-29T15:58:34.629137+0000","last_became_active":"2024-06-29T15:58:21.559976+0000","last_became_peered":"2024-06-29T15:58:21.559976+0000","last_unstale":"2024-06-29T15:58:34.629137+0000","last_undegraded":"2024-06-29T15:58:34.629137+0000","last_fullsized":"2024-06-29T15:58:34.629137+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:35:34.378626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659030+0000","last_change":"2024-06-29T15:58:34.642890+0000","last_active":"2024-06-29T15:58:34.659030+0000","last_peered":"2024-06-29T15:58:34.659030+0000","last_clean":"2024-06-29T15:58:34.659030+0000","last_became_active":"2024-06-29T15:58:34.642520+0000","last_became_peered":"2024-06-29T15:58:34.642520+0000","last_unstale":"2024-06-29T15:58:34.659030+0000","last_undegraded":"2024-06-29T15:58:34.659030+0000","last_fullsized":"2024-06-29T15:58:34.659030+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:21.951845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631238+0000","last_change":"2024-06-29T15:58:33.627943+0000","last_active":"2024-06-29T15:58:34.631238+0000","last_peered":"2024-06-29T15:58:34.631238+0000","last_clean":"2024-06-29T15:58:34.631238+0000","last_became_active":"2024-06-29T15:58:33.626589+0000","last_became_peered":"2024-06-29T15:58:33.626589+0000","last_unstale":"2024-06-29T15:58:34.631238+0000","last_undegraded":"2024-06-29T15:58:34.631238+0000","last_fullsized":"2024-06-29T15:58:34.631238+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:01.898801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625701+0000","last_change":"2024-06-29T15:58:21.567137+0000","last_active":"2024-06-29T15:58:34.625701+0000","last_peered":"2024-06-29T15:58:34.625701+0000","last_clean":"2024-06-29T15:58:34.625701+0000","last_became_active":"2024-06-29T15:58:21.566816+0000","last_became_peered":"2024-06-29T15:58:21.566816+0000","last_unstale":"2024-06-29T15:58:34.625701+0000","last_undegraded":"2024-06-29T15:58:34.625701+0000","last_fullsized":"2024-06-29T15:58:34.625701+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:36.096648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.644310+0000","last_change":"2024-06-29T15:58:34.634052+0000","last_active":"2024-06-29T15:58:34.644310+0000","last_peered":"2024-06-29T15:58:34.644310+0000","last_clean":"2024-06-29T15:58:34.644310+0000","last_became_active":"2024-06-29T15:58:34.633713+0000","last_became_peered":"2024-06-29T15:58:34.633713+0000","last_unstale":"2024-06-29T15:58:34.644310+0000","last_undegraded":"2024-06-29T15:58:34.644310+0000","last_fullsized":"2024-06-29T15:58:34.644310+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:12.235244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.633820+0000","last_change":"2024-06-29T15:58:33.625957+0000","last_active":"2024-06-29T15:58:34.633820+0000","last_peered":"2024-06-29T15:58:34.633820+0000","last_clean":"2024-06-29T15:58:34.633820+0000","last_became_active":"2024-06-29T15:58:33.624954+0000","last_became_peered":"2024-06-29T15:58:33.624954+0000","last_unstale":"2024-06-29T15:58:34.633820+0000","last_undegraded":"2024-06-29T15:58:34.633820+0000","last_fullsized":"2024-06-29T15:58:34.633820+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:16.253720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.633012+0000","last_change":"2024-06-29T15:58:21.561708+0000","last_active":"2024-06-29T15:58:34.633012+0000","last_peered":"2024-06-29T15:58:34.633012+0000","last_clean":"2024-06-29T15:58:34.633012+0000","last_became_active":"2024-06-29T15:58:21.561360+0000","last_became_peered":"2024-06-29T15:58:21.561360+0000","last_unstale":"2024-06-29T15:58:34.633012+0000","last_undegraded":"2024-06-29T15:58:34.633012+0000","last_fullsized":"2024-06-29T15:58:34.633012+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:17.841782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640391+0000","last_change":"2024-06-29T15:58:34.636562+0000","last_active":"2024-06-29T15:58:34.640391+0000","last_peered":"2024-06-29T15:58:34.640391+0000","last_clean":"2024-06-29T15:58:34.640391+0000","last_became_active":"2024-06-29T15:58:34.636282+0000","last_became_peered":"2024-06-29T15:58:34.636282+0000","last_unstale":"2024-06-29T15:58:34.640391+0000","last_undegraded":"2024-06-29T15:58:34.640391+0000","last_fullsized":"2024-06-29T15:58:34.640391+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:05.349103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629501+0000","last_change":"2024-06-29T15:58:33.636425+0000","last_active":"2024-06-29T15:58:34.629501+0000","last_peered":"2024-06-29T15:58:34.629501+0000","last_clean":"2024-06-29T15:58:34.629501+0000","last_became_active":"2024-06-29T15:58:33.635983+0000","last_became_peered":"2024-06-29T15:58:33.635983+0000","last_unstale":"2024-06-29T15:58:34.629501+0000","last_undegraded":"2024-06-29T15:58:34.629501+0000","last_fullsized":"2024-06-29T15:58:34.629501+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:40.750750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630894+0000","last_change":"2024-06-29T15:58:21.560378+0000","last_active":"2024-06-29T15:58:34.630894+0000","last_peered":"2024-06-29T15:58:34.630894+0000","last_clean":"2024-06-29T15:58:34.630894+0000","last_became_active":"2024-06-29T15:58:21.560075+0000","last_became_peered":"2024-06-29T15:58:21.560075+0000","last_unstale":"2024-06-29T15:58:34.630894+0000","last_undegraded":"2024-06-29T15:58:34.630894+0000","last_fullsized":"2024-06-29T15:58:34.630894+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:16.886076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658855+0000","last_change":"2024-06-29T15:58:34.648158+0000","last_active":"2024-06-29T15:58:34.658855+0000","last_peered":"2024-06-29T15:58:34.658855+0000","last_clean":"2024-06-29T15:58:34.658855+0000","last_became_active":"2024-06-29T15:58:34.647788+0000","last_became_peered":"2024-06-29T15:58:34.647788+0000","last_unstale":"2024-06-29T15:58:34.658855+0000","last_undegraded":"2024-06-29T15:58:34.658855+0000","last_fullsized":"2024-06-29T15:58:34.658855+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:51.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630739+0000","last_change":"2024-06-29T15:58:33.634531+0000","last_active":"2024-06-29T15:58:34.630739+0000","last_peered":"2024-06-29T15:58:34.630739+0000","last_clean":"2024-06-29T15:58:34.630739+0000","last_became_active":"2024-06-29T15:58:33.634126+0000","last_became_peered":"2024-06-29T15:58:33.634126+0000","last_unstale":"2024-06-29T15:58:34.630739+0000","last_undegraded":"2024-06-29T15:58:34.630739+0000","last_fullsized":"2024-06-29T15:58:34.630739+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:32.600079+0000","last_clean_scrub_stamp":"2024-06-29T15:58:32.600079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:31.650912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625820+0000","last_change":"2024-06-29T15:58:21.561504+0000","last_active":"2024-06-29T15:58:34.625820+0000","last_peered":"2024-06-29T15:58:34.625820+0000","last_clean":"2024-06-29T15:58:34.625820+0000","last_became_active":"2024-06-29T15:58:21.560810+0000","last_became_peered":"2024-06-29T15:58:21.560810+0000","last_unstale":"2024-06-29T15:58:34.625820+0000","last_undegraded":"2024-06-29T15:58:34.625820+0000","last_fullsized":"2024-06-29T15:58:34.625820+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:53:36.634654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640285+0000","last_change":"2024-06-29T15:58:34.639130+0000","last_active":"2024-06-29T15:58:34.640285+0000","last_peered":"2024-06-29T15:58:34.640285+0000","last_clean":"2024-06-29T15:58:34.640285+0000","last_became_active":"2024-06-29T15:58:34.638792+0000","last_became_peered":"2024-06-29T15:58:34.638792+0000","last_unstale":"2024-06-29T15:58:34.640285+0000","last_undegraded":"2024-06-29T15:58:34.640285+0000","last_fullsized":"2024-06-29T15:58:34.640285+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:51:54.624785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629861+0000","last_change":"2024-06-29T15:58:21.562794+0000","last_active":"2024-06-29T15:58:34.629861+0000","last_peered":"2024-06-29T15:58:34.629861+0000","last_clean":"2024-06-29T15:58:34.629861+0000","last_became_active":"2024-06-29T15:58:21.562011+0000","last_became_peered":"2024-06-29T15:58:21.562011+0000","last_unstale":"2024-06-29T15:58:34.629861+0000","last_undegraded":"2024-06-29T15:58:34.629861+0000","last_fullsized":"2024-06-29T15:58:34.629861+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:09.874394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"241'4","reported_seq":24,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630137+0000","last_change":"2024-06-29T15:58:20.549867+0000","last_active":"2024-06-29T15:58:34.630137+0000","last_peered":"2024-06-29T15:58:34.630137+0000","last_clean":"2024-06-29T15:58:34.630137+0000","last_became_active":"2024-06-29T15:58:20.549532+0000","last_became_peered":"2024-06-29T15:58:20.549532+0000","last_unstale":"2024-06-29T15:58:34.630137+0000","last_undegraded":"2024-06-29T15:58:34.630137+0000","last_fullsized":"2024-06-29T15:58:34.630137+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:11.359524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658942+0000","last_change":"2024-06-29T15:58:34.646233+0000","last_active":"2024-06-29T15:58:34.658942+0000","last_peered":"2024-06-29T15:58:34.658942+0000","last_clean":"2024-06-29T15:58:34.658942+0000","last_became_active":"2024-06-29T15:58:34.645874+0000","last_became_peered":"2024-06-29T15:58:34.645874+0000","last_unstale":"2024-06-29T15:58:34.658942+0000","last_undegraded":"2024-06-29T15:58:34.658942+0000","last_fullsized":"2024-06-29T15:58:34.658942+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:30.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630456+0000","last_change":"2024-06-29T15:58:21.562279+0000","last_active":"2024-06-29T15:58:34.630456+0000","last_peered":"2024-06-29T15:58:34.630456+0000","last_clean":"2024-06-29T15:58:34.630456+0000","last_became_active":"2024-06-29T15:58:21.561962+0000","last_became_peered":"2024-06-29T15:58:21.561962+0000","last_unstale":"2024-06-29T15:58:34.630456+0000","last_undegraded":"2024-06-29T15:58:34.630456+0000","last_fullsized":"2024-06-29T15:58:34.630456+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:43:25.515942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629756+0000","last_change":"2024-06-29T15:58:20.547115+0000","last_active":"2024-06-29T15:58:34.629756+0000","last_peered":"2024-06-29T15:58:34.629756+0000","last_clean":"2024-06-29T15:58:34.629756+0000","last_became_active":"2024-06-29T15:58:20.546778+0000","last_became_peered":"2024-06-29T15:58:20.546778+0000","last_unstale":"2024-06-29T15:58:34.629756+0000","last_undegraded":"2024-06-29T15:58:34.629756+0000","last_fullsized":"2024-06-29T15:58:34.629756+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:05.970706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659082+0000","last_change":"2024-06-29T15:58:34.647340+0000","last_active":"2024-06-29T15:58:34.659082+0000","last_peered":"2024-06-29T15:58:34.659082+0000","last_clean":"2024-06-29T15:58:34.659082+0000","last_became_active":"2024-06-29T15:58:34.646885+0000","last_became_peered":"2024-06-29T15:58:34.646885+0000","last_unstale":"2024-06-29T15:58:34.659082+0000","last_undegraded":"2024-06-29T15:58:34.659082+0000","last_fullsized":"2024-06-29T15:58:34.659082+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:10:16.000133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629521+0000","last_change":"2024-06-29T15:58:21.562748+0000","last_active":"2024-06-29T15:58:34.629521+0000","last_peered":"2024-06-29T15:58:34.629521+0000","last_clean":"2024-06-29T15:58:34.629521+0000","last_became_active":"2024-06-29T15:58:21.562417+0000","last_became_peered":"2024-06-29T15:58:21.562417+0000","last_unstale":"2024-06-29T15:58:34.629521+0000","last_undegraded":"2024-06-29T15:58:34.629521+0000","last_fullsized":"2024-06-29T15:58:34.629521+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:00:27.973848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627211+0000","last_change":"2024-06-29T15:58:20.568004+0000","last_active":"2024-06-29T15:58:34.627211+0000","last_peered":"2024-06-29T15:58:34.627211+0000","last_clean":"2024-06-29T15:58:34.627211+0000","last_became_active":"2024-06-29T15:58:20.567638+0000","last_became_peered":"2024-06-29T15:58:20.567638+0000","last_unstale":"2024-06-29T15:58:34.627211+0000","last_undegraded":"2024-06-29T15:58:34.627211+0000","last_fullsized":"2024-06-29T15:58:34.627211+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:19.772821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658924+0000","last_change":"2024-06-29T15:58:34.646747+0000","last_active":"2024-06-29T15:58:34.658924+0000","last_peered":"2024-06-29T15:58:34.658924+0000","last_clean":"2024-06-29T15:58:34.658924+0000","last_became_active":"2024-06-29T15:58:34.646397+0000","last_became_peered":"2024-06-29T15:58:34.646397+0000","last_unstale":"2024-06-29T15:58:34.658924+0000","last_undegraded":"2024-06-29T15:58:34.658924+0000","last_fullsized":"2024-06-29T15:58:34.658924+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:34.731318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626902+0000","last_change":"2024-06-29T15:58:21.566347+0000","last_active":"2024-06-29T15:58:34.626902+0000","last_peered":"2024-06-29T15:58:34.626902+0000","last_clean":"2024-06-29T15:58:34.626902+0000","last_became_active":"2024-06-29T15:58:21.565888+0000","last_became_peered":"2024-06-29T15:58:21.565888+0000","last_unstale":"2024-06-29T15:58:34.626902+0000","last_undegraded":"2024-06-29T15:58:34.626902+0000","last_fullsized":"2024-06-29T15:58:34.626902+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:28.461061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628877+0000","last_change":"2024-06-29T15:58:20.556653+0000","last_active":"2024-06-29T15:58:34.628877+0000","last_peered":"2024-06-29T15:58:34.628877+0000","last_clean":"2024-06-29T15:58:34.628877+0000","last_became_active":"2024-06-29T15:58:20.555367+0000","last_became_peered":"2024-06-29T15:58:20.555367+0000","last_unstale":"2024-06-29T15:58:34.628877+0000","last_undegraded":"2024-06-29T15:58:34.628877+0000","last_fullsized":"2024-06-29T15:58:34.628877+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:31.173223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649284+0000","last_change":"2024-06-29T15:58:34.647891+0000","last_active":"2024-06-29T15:58:34.649284+0000","last_peered":"2024-06-29T15:58:34.649284+0000","last_clean":"2024-06-29T15:58:34.649284+0000","last_became_active":"2024-06-29T15:58:34.647354+0000","last_became_peered":"2024-06-29T15:58:34.647354+0000","last_unstale":"2024-06-29T15:58:34.649284+0000","last_undegraded":"2024-06-29T15:58:34.649284+0000","last_fullsized":"2024-06-29T15:58:34.649284+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:11.281394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.633083+0000","last_change":"2024-06-29T15:58:21.562456+0000","last_active":"2024-06-29T15:58:34.633083+0000","last_peered":"2024-06-29T15:58:34.633083+0000","last_clean":"2024-06-29T15:58:34.633083+0000","last_became_active":"2024-06-29T15:58:21.562024+0000","last_became_peered":"2024-06-29T15:58:21.562024+0000","last_unstale":"2024-06-29T15:58:34.633083+0000","last_undegraded":"2024-06-29T15:58:34.633083+0000","last_fullsized":"2024-06-29T15:58:34.633083+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:36.495502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632827+0000","last_change":"2024-06-29T15:58:20.558360+0000","last_active":"2024-06-29T15:58:34.632827+0000","last_peered":"2024-06-29T15:58:34.632827+0000","last_clean":"2024-06-29T15:58:34.632827+0000","last_became_active":"2024-06-29T15:58:20.558053+0000","last_became_peered":"2024-06-29T15:58:20.558053+0000","last_unstale":"2024-06-29T15:58:34.632827+0000","last_undegraded":"2024-06-29T15:58:34.632827+0000","last_fullsized":"2024-06-29T15:58:34.632827+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:40.258636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.647182+0000","last_change":"2024-06-29T15:58:34.646442+0000","last_active":"2024-06-29T15:58:34.647182+0000","last_peered":"2024-06-29T15:58:34.647182+0000","last_clean":"2024-06-29T15:58:34.647182+0000","last_became_active":"2024-06-29T15:58:34.645545+0000","last_became_peered":"2024-06-29T15:58:34.645545+0000","last_unstale":"2024-06-29T15:58:34.647182+0000","last_undegraded":"2024-06-29T15:58:34.647182+0000","last_fullsized":"2024-06-29T15:58:34.647182+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:33.180128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619310+0000","last_change":"2024-06-29T15:58:21.568464+0000","last_active":"2024-06-29T15:58:34.619310+0000","last_peered":"2024-06-29T15:58:34.619310+0000","last_clean":"2024-06-29T15:58:34.619310+0000","last_became_active":"2024-06-29T15:58:21.568114+0000","last_became_peered":"2024-06-29T15:58:21.568114+0000","last_unstale":"2024-06-29T15:58:34.619310+0000","last_undegraded":"2024-06-29T15:58:34.619310+0000","last_fullsized":"2024-06-29T15:58:34.619310+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:09.625056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618713+0000","last_change":"2024-06-29T15:58:20.563140+0000","last_active":"2024-06-29T15:58:34.618713+0000","last_peered":"2024-06-29T15:58:34.618713+0000","last_clean":"2024-06-29T15:58:34.618713+0000","last_became_active":"2024-06-29T15:58:20.562823+0000","last_became_peered":"2024-06-29T15:58:20.562823+0000","last_unstale":"2024-06-29T15:58:34.618713+0000","last_undegraded":"2024-06-29T15:58:34.618713+0000","last_fullsized":"2024-06-29T15:58:34.618713+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:24.191012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646965+0000","last_change":"2024-06-29T15:58:34.644576+0000","last_active":"2024-06-29T15:58:34.646965+0000","last_peered":"2024-06-29T15:58:34.646965+0000","last_clean":"2024-06-29T15:58:34.646965+0000","last_became_active":"2024-06-29T15:58:34.644165+0000","last_became_peered":"2024-06-29T15:58:34.644165+0000","last_unstale":"2024-06-29T15:58:34.646965+0000","last_undegraded":"2024-06-29T15:58:34.646965+0000","last_fullsized":"2024-06-29T15:58:34.646965+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:29.253507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622811+0000","last_change":"2024-06-29T15:58:21.556649+0000","last_active":"2024-06-29T15:58:34.622811+0000","last_peered":"2024-06-29T15:58:34.622811+0000","last_clean":"2024-06-29T15:58:34.622811+0000","last_became_active":"2024-06-29T15:58:21.556297+0000","last_became_peered":"2024-06-29T15:58:21.556297+0000","last_unstale":"2024-06-29T15:58:34.622811+0000","last_undegraded":"2024-06-29T15:58:34.622811+0000","last_fullsized":"2024-06-29T15:58:34.622811+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:07.319988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626017+0000","last_change":"2024-06-29T15:58:20.555706+0000","last_active":"2024-06-29T15:58:34.626017+0000","last_peered":"2024-06-29T15:58:34.626017+0000","last_clean":"2024-06-29T15:58:34.626017+0000","last_became_active":"2024-06-29T15:58:20.555380+0000","last_became_peered":"2024-06-29T15:58:20.555380+0000","last_unstale":"2024-06-29T15:58:34.626017+0000","last_undegraded":"2024-06-29T15:58:34.626017+0000","last_fullsized":"2024-06-29T15:58:34.626017+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:03:17.347006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659569+0000","last_change":"2024-06-29T15:58:34.659095+0000","last_active":"2024-06-29T15:58:34.659569+0000","last_peered":"2024-06-29T15:58:34.659569+0000","last_clean":"2024-06-29T15:58:34.659569+0000","last_became_active":"2024-06-29T15:58:34.658482+0000","last_became_peered":"2024-06-29T15:58:34.658482+0000","last_unstale":"2024-06-29T15:58:34.659569+0000","last_undegraded":"2024-06-29T15:58:34.659569+0000","last_fullsized":"2024-06-29T15:58:34.659569+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:25.614229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629077+0000","last_change":"2024-06-29T15:58:21.554565+0000","last_active":"2024-06-29T15:58:34.629077+0000","last_peered":"2024-06-29T15:58:34.629077+0000","last_clean":"2024-06-29T15:58:34.629077+0000","last_became_active":"2024-06-29T15:58:21.554255+0000","last_became_peered":"2024-06-29T15:58:21.554255+0000","last_unstale":"2024-06-29T15:58:34.629077+0000","last_undegraded":"2024-06-29T15:58:34.629077+0000","last_fullsized":"2024-06-29T15:58:34.629077+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:14.470919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626079+0000","last_change":"2024-06-29T15:58:20.562271+0000","last_active":"2024-06-29T15:58:34.626079+0000","last_peered":"2024-06-29T15:58:34.626079+0000","last_clean":"2024-06-29T15:58:34.626079+0000","last_became_active":"2024-06-29T15:58:20.561940+0000","last_became_peered":"2024-06-29T15:58:20.561940+0000","last_unstale":"2024-06-29T15:58:34.626079+0000","last_undegraded":"2024-06-29T15:58:34.626079+0000","last_fullsized":"2024-06-29T15:58:34.626079+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:16.036109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.639400+0000","last_change":"2024-06-29T15:58:34.638123+0000","last_active":"2024-06-29T15:58:34.639400+0000","last_peered":"2024-06-29T15:58:34.639400+0000","last_clean":"2024-06-29T15:58:34.639400+0000","last_became_active":"2024-06-29T15:58:34.637789+0000","last_became_peered":"2024-06-29T15:58:34.637789+0000","last_unstale":"2024-06-29T15:58:34.639400+0000","last_undegraded":"2024-06-29T15:58:34.639400+0000","last_fullsized":"2024-06-29T15:58:34.639400+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:52.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628202+0000","last_change":"2024-06-29T15:58:21.548845+0000","last_active":"2024-06-29T15:58:34.628202+0000","last_peered":"2024-06-29T15:58:34.628202+0000","last_clean":"2024-06-29T15:58:34.628202+0000","last_became_active":"2024-06-29T15:58:21.548448+0000","last_became_peered":"2024-06-29T15:58:21.548448+0000","last_unstale":"2024-06-29T15:58:34.628202+0000","last_undegraded":"2024-06-29T15:58:34.628202+0000","last_fullsized":"2024-06-29T15:58:34.628202+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:38.983134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619705+0000","last_change":"2024-06-29T15:58:20.562116+0000","last_active":"2024-06-29T15:58:34.619705+0000","last_peered":"2024-06-29T15:58:34.619705+0000","last_clean":"2024-06-29T15:58:34.619705+0000","last_became_active":"2024-06-29T15:58:20.561801+0000","last_became_peered":"2024-06-29T15:58:20.561801+0000","last_unstale":"2024-06-29T15:58:34.619705+0000","last_undegraded":"2024-06-29T15:58:34.619705+0000","last_fullsized":"2024-06-29T15:58:34.619705+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:26.721883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640304+0000","last_change":"2024-06-29T15:58:34.637931+0000","last_active":"2024-06-29T15:58:34.640304+0000","last_peered":"2024-06-29T15:58:34.640304+0000","last_clean":"2024-06-29T15:58:34.640304+0000","last_became_active":"2024-06-29T15:58:34.637342+0000","last_became_peered":"2024-06-29T15:58:34.637342+0000","last_unstale":"2024-06-29T15:58:34.640304+0000","last_undegraded":"2024-06-29T15:58:34.640304+0000","last_fullsized":"2024-06-29T15:58:34.640304+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:25.149310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618903+0000","last_change":"2024-06-29T15:58:21.560085+0000","last_active":"2024-06-29T15:58:34.618903+0000","last_peered":"2024-06-29T15:58:34.618903+0000","last_clean":"2024-06-29T15:58:34.618903+0000","last_became_active":"2024-06-29T15:58:21.559716+0000","last_became_peered":"2024-06-29T15:58:21.559716+0000","last_unstale":"2024-06-29T15:58:34.618903+0000","last_undegraded":"2024-06-29T15:58:34.618903+0000","last_fullsized":"2024-06-29T15:58:34.618903+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:15:37.579793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"241'3","reported_seq":23,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622530+0000","last_change":"2024-06-29T15:58:20.553528+0000","last_active":"2024-06-29T15:58:34.622530+0000","last_peered":"2024-06-29T15:58:34.622530+0000","last_clean":"2024-06-29T15:58:34.622530+0000","last_became_active":"2024-06-29T15:58:20.553185+0000","last_became_peered":"2024-06-29T15:58:20.553185+0000","last_unstale":"2024-06-29T15:58:34.622530+0000","last_undegraded":"2024-06-29T15:58:34.622530+0000","last_fullsized":"2024-06-29T15:58:34.622530+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:26.807742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.659056+0000","last_change":"2024-06-29T15:58:34.643180+0000","last_active":"2024-06-29T15:58:34.659056+0000","last_peered":"2024-06-29T15:58:34.659056+0000","last_clean":"2024-06-29T15:58:34.659056+0000","last_became_active":"2024-06-29T15:58:34.642846+0000","last_became_peered":"2024-06-29T15:58:34.642846+0000","last_unstale":"2024-06-29T15:58:34.659056+0000","last_undegraded":"2024-06-29T15:58:34.659056+0000","last_fullsized":"2024-06-29T15:58:34.659056+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:24:27.526406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626571+0000","last_change":"2024-06-29T15:58:21.553783+0000","last_active":"2024-06-29T15:58:34.626571+0000","last_peered":"2024-06-29T15:58:34.626571+0000","last_clean":"2024-06-29T15:58:34.626571+0000","last_became_active":"2024-06-29T15:58:21.553278+0000","last_became_peered":"2024-06-29T15:58:21.553278+0000","last_unstale":"2024-06-29T15:58:34.626571+0000","last_undegraded":"2024-06-29T15:58:34.626571+0000","last_fullsized":"2024-06-29T15:58:34.626571+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:39.179054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"242'8","reported_seq":28,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629263+0000","last_change":"2024-06-29T15:58:20.557583+0000","last_active":"2024-06-29T15:58:34.629263+0000","last_peered":"2024-06-29T15:58:34.629263+0000","last_clean":"2024-06-29T15:58:34.629263+0000","last_became_active":"2024-06-29T15:58:20.557274+0000","last_became_peered":"2024-06-29T15:58:20.557274+0000","last_unstale":"2024-06-29T15:58:34.629263+0000","last_undegraded":"2024-06-29T15:58:34.629263+0000","last_fullsized":"2024-06-29T15:58:34.629263+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:38.435050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649709+0000","last_change":"2024-06-29T15:58:34.642355+0000","last_active":"2024-06-29T15:58:34.649709+0000","last_peered":"2024-06-29T15:58:34.649709+0000","last_clean":"2024-06-29T15:58:34.649709+0000","last_became_active":"2024-06-29T15:58:34.641999+0000","last_became_peered":"2024-06-29T15:58:34.641999+0000","last_unstale":"2024-06-29T15:58:34.649709+0000","last_undegraded":"2024-06-29T15:58:34.649709+0000","last_fullsized":"2024-06-29T15:58:34.649709+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:44:43.831068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632735+0000","last_change":"2024-06-29T15:58:21.559479+0000","last_active":"2024-06-29T15:58:34.632735+0000","last_peered":"2024-06-29T15:58:34.632735+0000","last_clean":"2024-06-29T15:58:34.632735+0000","last_became_active":"2024-06-29T15:58:21.559128+0000","last_became_peered":"2024-06-29T15:58:21.559128+0000","last_unstale":"2024-06-29T15:58:34.632735+0000","last_undegraded":"2024-06-29T15:58:34.632735+0000","last_fullsized":"2024-06-29T15:58:34.632735+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:28.321473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630359+0000","last_change":"2024-06-29T15:58:20.561589+0000","last_active":"2024-06-29T15:58:34.630359+0000","last_peered":"2024-06-29T15:58:34.630359+0000","last_clean":"2024-06-29T15:58:34.630359+0000","last_became_active":"2024-06-29T15:58:20.561251+0000","last_became_peered":"2024-06-29T15:58:20.561251+0000","last_unstale":"2024-06-29T15:58:34.630359+0000","last_undegraded":"2024-06-29T15:58:34.630359+0000","last_fullsized":"2024-06-29T15:58:34.630359+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:09.657256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.640355+0000","last_change":"2024-06-29T15:58:34.637948+0000","last_active":"2024-06-29T15:58:34.640355+0000","last_peered":"2024-06-29T15:58:34.640355+0000","last_clean":"2024-06-29T15:58:34.640355+0000","last_became_active":"2024-06-29T15:58:34.637190+0000","last_became_peered":"2024-06-29T15:58:34.637190+0000","last_unstale":"2024-06-29T15:58:34.640355+0000","last_undegraded":"2024-06-29T15:58:34.640355+0000","last_fullsized":"2024-06-29T15:58:34.640355+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:15.234940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.632345+0000","last_change":"2024-06-29T15:58:21.560367+0000","last_active":"2024-06-29T15:58:34.632345+0000","last_peered":"2024-06-29T15:58:34.632345+0000","last_clean":"2024-06-29T15:58:34.632345+0000","last_became_active":"2024-06-29T15:58:21.560056+0000","last_became_peered":"2024-06-29T15:58:21.560056+0000","last_unstale":"2024-06-29T15:58:34.632345+0000","last_undegraded":"2024-06-29T15:58:34.632345+0000","last_fullsized":"2024-06-29T15:58:34.632345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:43.371506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621895+0000","last_change":"2024-06-29T15:58:20.552449+0000","last_active":"2024-06-29T15:58:34.621895+0000","last_peered":"2024-06-29T15:58:34.621895+0000","last_clean":"2024-06-29T15:58:34.621895+0000","last_became_active":"2024-06-29T15:58:20.551202+0000","last_became_peered":"2024-06-29T15:58:20.551202+0000","last_unstale":"2024-06-29T15:58:34.621895+0000","last_undegraded":"2024-06-29T15:58:34.621895+0000","last_fullsized":"2024-06-29T15:58:34.621895+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:35.829408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646661+0000","last_change":"2024-06-29T15:58:34.644777+0000","last_active":"2024-06-29T15:58:34.646661+0000","last_peered":"2024-06-29T15:58:34.646661+0000","last_clean":"2024-06-29T15:58:34.646661+0000","last_became_active":"2024-06-29T15:58:34.644047+0000","last_became_peered":"2024-06-29T15:58:34.644047+0000","last_unstale":"2024-06-29T15:58:34.646661+0000","last_undegraded":"2024-06-29T15:58:34.646661+0000","last_fullsized":"2024-06-29T15:58:34.646661+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:57.533609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630674+0000","last_change":"2024-06-29T15:58:21.566336+0000","last_active":"2024-06-29T15:58:34.630674+0000","last_peered":"2024-06-29T15:58:34.630674+0000","last_clean":"2024-06-29T15:58:34.630674+0000","last_became_active":"2024-06-29T15:58:21.566003+0000","last_became_peered":"2024-06-29T15:58:21.566003+0000","last_unstale":"2024-06-29T15:58:34.630674+0000","last_undegraded":"2024-06-29T15:58:34.630674+0000","last_fullsized":"2024-06-29T15:58:34.630674+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:46:35.306688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621874+0000","last_change":"2024-06-29T15:58:20.557295+0000","last_active":"2024-06-29T15:58:34.621874+0000","last_peered":"2024-06-29T15:58:34.621874+0000","last_clean":"2024-06-29T15:58:34.621874+0000","last_became_active":"2024-06-29T15:58:20.556868+0000","last_became_peered":"2024-06-29T15:58:20.556868+0000","last_unstale":"2024-06-29T15:58:34.621874+0000","last_undegraded":"2024-06-29T15:58:34.621874+0000","last_fullsized":"2024-06-29T15:58:34.621874+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:31.226218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.638849+0000","last_change":"2024-06-29T15:58:34.635098+0000","last_active":"2024-06-29T15:58:34.638849+0000","last_peered":"2024-06-29T15:58:34.638849+0000","last_clean":"2024-06-29T15:58:34.638849+0000","last_became_active":"2024-06-29T15:58:34.634799+0000","last_became_peered":"2024-06-29T15:58:34.634799+0000","last_unstale":"2024-06-29T15:58:34.638849+0000","last_undegraded":"2024-06-29T15:58:34.638849+0000","last_fullsized":"2024-06-29T15:58:34.638849+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:15:09.575512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622802+0000","last_change":"2024-06-29T15:58:21.565109+0000","last_active":"2024-06-29T15:58:34.622802+0000","last_peered":"2024-06-29T15:58:34.622802+0000","last_clean":"2024-06-29T15:58:34.622802+0000","last_became_active":"2024-06-29T15:58:21.564764+0000","last_became_peered":"2024-06-29T15:58:21.564764+0000","last_unstale":"2024-06-29T15:58:34.622802+0000","last_undegraded":"2024-06-29T15:58:34.622802+0000","last_fullsized":"2024-06-29T15:58:34.622802+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:43:24.997613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"241'8","reported_seq":28,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629994+0000","last_change":"2024-06-29T15:58:20.562144+0000","last_active":"2024-06-29T15:58:34.629994+0000","last_peered":"2024-06-29T15:58:34.629994+0000","last_clean":"2024-06-29T15:58:34.629994+0000","last_became_active":"2024-06-29T15:58:20.561879+0000","last_became_peered":"2024-06-29T15:58:20.561879+0000","last_unstale":"2024-06-29T15:58:34.629994+0000","last_undegraded":"2024-06-29T15:58:34.629994+0000","last_fullsized":"2024-06-29T15:58:34.629994+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:28.961624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658882+0000","last_change":"2024-06-29T15:58:34.653641+0000","last_active":"2024-06-29T15:58:34.658882+0000","last_peered":"2024-06-29T15:58:34.658882+0000","last_clean":"2024-06-29T15:58:34.658882+0000","last_became_active":"2024-06-29T15:58:34.647800+0000","last_became_peered":"2024-06-29T15:58:34.647800+0000","last_unstale":"2024-06-29T15:58:34.658882+0000","last_undegraded":"2024-06-29T15:58:34.658882+0000","last_fullsized":"2024-06-29T15:58:34.658882+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:11:51.300578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622216+0000","last_change":"2024-06-29T15:58:21.546206+0000","last_active":"2024-06-29T15:58:34.622216+0000","last_peered":"2024-06-29T15:58:34.622216+0000","last_clean":"2024-06-29T15:58:34.622216+0000","last_became_active":"2024-06-29T15:58:21.545742+0000","last_became_peered":"2024-06-29T15:58:21.545742+0000","last_unstale":"2024-06-29T15:58:34.622216+0000","last_undegraded":"2024-06-29T15:58:34.622216+0000","last_fullsized":"2024-06-29T15:58:34.622216+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:55:57.523444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.630822+0000","last_change":"2024-06-29T15:58:20.555827+0000","last_active":"2024-06-29T15:58:34.630822+0000","last_peered":"2024-06-29T15:58:34.630822+0000","last_clean":"2024-06-29T15:58:34.630822+0000","last_became_active":"2024-06-29T15:58:20.555506+0000","last_became_peered":"2024-06-29T15:58:20.555506+0000","last_unstale":"2024-06-29T15:58:34.630822+0000","last_undegraded":"2024-06-29T15:58:34.630822+0000","last_fullsized":"2024-06-29T15:58:34.630822+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:33:20.721321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651461+0000","last_change":"2024-06-29T15:58:34.642428+0000","last_active":"2024-06-29T15:58:34.651461+0000","last_peered":"2024-06-29T15:58:34.651461+0000","last_clean":"2024-06-29T15:58:34.651461+0000","last_became_active":"2024-06-29T15:58:34.641271+0000","last_became_peered":"2024-06-29T15:58:34.641271+0000","last_unstale":"2024-06-29T15:58:34.651461+0000","last_undegraded":"2024-06-29T15:58:34.651461+0000","last_fullsized":"2024-06-29T15:58:34.651461+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:52:52.231569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621719+0000","last_change":"2024-06-29T15:58:21.558057+0000","last_active":"2024-06-29T15:58:34.621719+0000","last_peered":"2024-06-29T15:58:34.621719+0000","last_clean":"2024-06-29T15:58:34.621719+0000","last_became_active":"2024-06-29T15:58:21.557750+0000","last_became_peered":"2024-06-29T15:58:21.557750+0000","last_unstale":"2024-06-29T15:58:34.621719+0000","last_undegraded":"2024-06-29T15:58:34.621719+0000","last_fullsized":"2024-06-29T15:58:34.621719+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:35.875818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631966+0000","last_change":"2024-06-29T15:58:20.561995+0000","last_active":"2024-06-29T15:58:34.631966+0000","last_peered":"2024-06-29T15:58:34.631966+0000","last_clean":"2024-06-29T15:58:34.631966+0000","last_became_active":"2024-06-29T15:58:20.560719+0000","last_became_peered":"2024-06-29T15:58:20.560719+0000","last_unstale":"2024-06-29T15:58:34.631966+0000","last_undegraded":"2024-06-29T15:58:34.631966+0000","last_fullsized":"2024-06-29T15:58:34.631966+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:59.752024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.644470+0000","last_change":"2024-06-29T15:58:34.643196+0000","last_active":"2024-06-29T15:58:34.644470+0000","last_peered":"2024-06-29T15:58:34.644470+0000","last_clean":"2024-06-29T15:58:34.644470+0000","last_became_active":"2024-06-29T15:58:34.642891+0000","last_became_peered":"2024-06-29T15:58:34.642891+0000","last_unstale":"2024-06-29T15:58:34.644470+0000","last_undegraded":"2024-06-29T15:58:34.644470+0000","last_fullsized":"2024-06-29T15:58:34.644470+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:24.611850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631308+0000","last_change":"2024-06-29T15:58:21.552894+0000","last_active":"2024-06-29T15:58:34.631308+0000","last_peered":"2024-06-29T15:58:34.631308+0000","last_clean":"2024-06-29T15:58:34.631308+0000","last_became_active":"2024-06-29T15:58:21.552487+0000","last_became_peered":"2024-06-29T15:58:21.552487+0000","last_unstale":"2024-06-29T15:58:34.631308+0000","last_undegraded":"2024-06-29T15:58:34.631308+0000","last_fullsized":"2024-06-29T15:58:34.631308+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:49.309450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629783+0000","last_change":"2024-06-29T15:58:20.544642+0000","last_active":"2024-06-29T15:58:34.629783+0000","last_peered":"2024-06-29T15:58:34.629783+0000","last_clean":"2024-06-29T15:58:34.629783+0000","last_became_active":"2024-06-29T15:58:20.544356+0000","last_became_peered":"2024-06-29T15:58:20.544356+0000","last_unstale":"2024-06-29T15:58:34.629783+0000","last_undegraded":"2024-06-29T15:58:34.629783+0000","last_fullsized":"2024-06-29T15:58:34.629783+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:38.143805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.660730+0000","last_change":"2024-06-29T15:58:34.658866+0000","last_active":"2024-06-29T15:58:34.660730+0000","last_peered":"2024-06-29T15:58:34.660730+0000","last_clean":"2024-06-29T15:58:34.660730+0000","last_became_active":"2024-06-29T15:58:34.658422+0000","last_became_peered":"2024-06-29T15:58:34.658422+0000","last_unstale":"2024-06-29T15:58:34.660730+0000","last_undegraded":"2024-06-29T15:58:34.660730+0000","last_fullsized":"2024-06-29T15:58:34.660730+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:33.087931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.620595+0000","last_change":"2024-06-29T15:58:21.560349+0000","last_active":"2024-06-29T15:58:34.620595+0000","last_peered":"2024-06-29T15:58:34.620595+0000","last_clean":"2024-06-29T15:58:34.620595+0000","last_became_active":"2024-06-29T15:58:21.559987+0000","last_became_peered":"2024-06-29T15:58:21.559987+0000","last_unstale":"2024-06-29T15:58:34.620595+0000","last_undegraded":"2024-06-29T15:58:34.620595+0000","last_fullsized":"2024-06-29T15:58:34.620595+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:20.317519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.634434+0000","last_change":"2024-06-29T15:58:20.551598+0000","last_active":"2024-06-29T15:58:34.634434+0000","last_peered":"2024-06-29T15:58:34.634434+0000","last_clean":"2024-06-29T15:58:34.634434+0000","last_became_active":"2024-06-29T15:58:20.551266+0000","last_became_peered":"2024-06-29T15:58:20.551266+0000","last_unstale":"2024-06-29T15:58:34.634434+0000","last_undegraded":"2024-06-29T15:58:34.634434+0000","last_fullsized":"2024-06-29T15:58:34.634434+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:56:15.731511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.625949+0000","last_change":"2024-06-29T15:58:34.625489+0000","last_active":"2024-06-29T15:58:34.625949+0000","last_peered":"2024-06-29T15:58:34.625949+0000","last_clean":"2024-06-29T15:58:34.625949+0000","last_became_active":"2024-06-29T15:58:34.625171+0000","last_became_peered":"2024-06-29T15:58:34.625171+0000","last_unstale":"2024-06-29T15:58:34.625949+0000","last_undegraded":"2024-06-29T15:58:34.625949+0000","last_fullsized":"2024-06-29T15:58:34.625949+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:59.352335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.619148+0000","last_change":"2024-06-29T15:58:21.562779+0000","last_active":"2024-06-29T15:58:34.619148+0000","last_peered":"2024-06-29T15:58:34.619148+0000","last_clean":"2024-06-29T15:58:34.619148+0000","last_became_active":"2024-06-29T15:58:21.562503+0000","last_became_peered":"2024-06-29T15:58:21.562503+0000","last_unstale":"2024-06-29T15:58:34.619148+0000","last_undegraded":"2024-06-29T15:58:34.619148+0000","last_fullsized":"2024-06-29T15:58:34.619148+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:07:28.461061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.620737+0000","last_change":"2024-06-29T15:58:20.559588+0000","last_active":"2024-06-29T15:58:34.620737+0000","last_peered":"2024-06-29T15:58:34.620737+0000","last_clean":"2024-06-29T15:58:34.620737+0000","last_became_active":"2024-06-29T15:58:20.559245+0000","last_became_peered":"2024-06-29T15:58:20.559245+0000","last_unstale":"2024-06-29T15:58:34.620737+0000","last_undegraded":"2024-06-29T15:58:34.620737+0000","last_fullsized":"2024-06-29T15:58:34.620737+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:40.258636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.649102+0000","last_change":"2024-06-29T15:58:34.648358+0000","last_active":"2024-06-29T15:58:34.649102+0000","last_peered":"2024-06-29T15:58:34.649102+0000","last_clean":"2024-06-29T15:58:34.649102+0000","last_became_active":"2024-06-29T15:58:34.648049+0000","last_became_peered":"2024-06-29T15:58:34.648049+0000","last_unstale":"2024-06-29T15:58:34.649102+0000","last_undegraded":"2024-06-29T15:58:34.649102+0000","last_fullsized":"2024-06-29T15:58:34.649102+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:13.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629190+0000","last_change":"2024-06-29T15:58:21.553384+0000","last_active":"2024-06-29T15:58:34.629190+0000","last_peered":"2024-06-29T15:58:34.629190+0000","last_clean":"2024-06-29T15:58:34.629190+0000","last_became_active":"2024-06-29T15:58:21.551240+0000","last_became_peered":"2024-06-29T15:58:21.551240+0000","last_unstale":"2024-06-29T15:58:34.629190+0000","last_undegraded":"2024-06-29T15:58:34.629190+0000","last_fullsized":"2024-06-29T15:58:34.629190+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:07:38.040891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628442+0000","last_change":"2024-06-29T15:58:20.557247+0000","last_active":"2024-06-29T15:58:34.628442+0000","last_peered":"2024-06-29T15:58:34.628442+0000","last_clean":"2024-06-29T15:58:34.628442+0000","last_became_active":"2024-06-29T15:58:20.556889+0000","last_became_peered":"2024-06-29T15:58:20.556889+0000","last_unstale":"2024-06-29T15:58:34.628442+0000","last_undegraded":"2024-06-29T15:58:34.628442+0000","last_fullsized":"2024-06-29T15:58:34.628442+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:48.301087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.645452+0000","last_change":"2024-06-29T15:58:34.641442+0000","last_active":"2024-06-29T15:58:34.645452+0000","last_peered":"2024-06-29T15:58:34.645452+0000","last_clean":"2024-06-29T15:58:34.645452+0000","last_became_active":"2024-06-29T15:58:34.640887+0000","last_became_peered":"2024-06-29T15:58:34.640887+0000","last_unstale":"2024-06-29T15:58:34.645452+0000","last_undegraded":"2024-06-29T15:58:34.645452+0000","last_fullsized":"2024-06-29T15:58:34.645452+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:52:03.341429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629393+0000","last_change":"2024-06-29T15:58:21.558641+0000","last_active":"2024-06-29T15:58:34.629393+0000","last_peered":"2024-06-29T15:58:34.629393+0000","last_clean":"2024-06-29T15:58:34.629393+0000","last_became_active":"2024-06-29T15:58:21.558377+0000","last_became_peered":"2024-06-29T15:58:21.558377+0000","last_unstale":"2024-06-29T15:58:34.629393+0000","last_undegraded":"2024-06-29T15:58:34.629393+0000","last_fullsized":"2024-06-29T15:58:34.629393+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:09.589023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628333+0000","last_change":"2024-06-29T15:58:20.543880+0000","last_active":"2024-06-29T15:58:34.628333+0000","last_peered":"2024-06-29T15:58:34.628333+0000","last_clean":"2024-06-29T15:58:34.628333+0000","last_became_active":"2024-06-29T15:58:20.543365+0000","last_became_peered":"2024-06-29T15:58:20.543365+0000","last_unstale":"2024-06-29T15:58:34.628333+0000","last_undegraded":"2024-06-29T15:58:34.628333+0000","last_fullsized":"2024-06-29T15:58:34.628333+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:18:31.509521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.641843+0000","last_change":"2024-06-29T15:58:34.640187+0000","last_active":"2024-06-29T15:58:34.641843+0000","last_peered":"2024-06-29T15:58:34.641843+0000","last_clean":"2024-06-29T15:58:34.641843+0000","last_became_active":"2024-06-29T15:58:34.639854+0000","last_became_peered":"2024-06-29T15:58:34.639854+0000","last_unstale":"2024-06-29T15:58:34.641843+0000","last_undegraded":"2024-06-29T15:58:34.641843+0000","last_fullsized":"2024-06-29T15:58:34.641843+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:17.260957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629667+0000","last_change":"2024-06-29T15:58:21.555113+0000","last_active":"2024-06-29T15:58:34.629667+0000","last_peered":"2024-06-29T15:58:34.629667+0000","last_clean":"2024-06-29T15:58:34.629667+0000","last_became_active":"2024-06-29T15:58:21.553362+0000","last_became_peered":"2024-06-29T15:58:21.553362+0000","last_unstale":"2024-06-29T15:58:34.629667+0000","last_undegraded":"2024-06-29T15:58:34.629667+0000","last_fullsized":"2024-06-29T15:58:34.629667+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:18.837956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.623429+0000","last_change":"2024-06-29T15:58:20.551665+0000","last_active":"2024-06-29T15:58:34.623429+0000","last_peered":"2024-06-29T15:58:34.623429+0000","last_clean":"2024-06-29T15:58:34.623429+0000","last_became_active":"2024-06-29T15:58:20.551347+0000","last_became_peered":"2024-06-29T15:58:20.551347+0000","last_unstale":"2024-06-29T15:58:34.623429+0000","last_undegraded":"2024-06-29T15:58:34.623429+0000","last_fullsized":"2024-06-29T15:58:34.623429+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:22:40.223323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.646126+0000","last_change":"2024-06-29T15:58:34.643477+0000","last_active":"2024-06-29T15:58:34.646126+0000","last_peered":"2024-06-29T15:58:34.646126+0000","last_clean":"2024-06-29T15:58:34.646126+0000","last_became_active":"2024-06-29T15:58:34.643144+0000","last_became_peered":"2024-06-29T15:58:34.643144+0000","last_unstale":"2024-06-29T15:58:34.646126+0000","last_undegraded":"2024-06-29T15:58:34.646126+0000","last_fullsized":"2024-06-29T15:58:34.646126+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:18.918717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628924+0000","last_change":"2024-06-29T15:58:21.561571+0000","last_active":"2024-06-29T15:58:34.628924+0000","last_peered":"2024-06-29T15:58:34.628924+0000","last_clean":"2024-06-29T15:58:34.628924+0000","last_became_active":"2024-06-29T15:58:21.561042+0000","last_became_peered":"2024-06-29T15:58:21.561042+0000","last_unstale":"2024-06-29T15:58:34.628924+0000","last_undegraded":"2024-06-29T15:58:34.628924+0000","last_fullsized":"2024-06-29T15:58:34.628924+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:22:34.199222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"241'2","reported_seq":22,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.620265+0000","last_change":"2024-06-29T15:58:20.563978+0000","last_active":"2024-06-29T15:58:34.620265+0000","last_peered":"2024-06-29T15:58:34.620265+0000","last_clean":"2024-06-29T15:58:34.620265+0000","last_became_active":"2024-06-29T15:58:20.563680+0000","last_became_peered":"2024-06-29T15:58:20.563680+0000","last_unstale":"2024-06-29T15:58:34.620265+0000","last_undegraded":"2024-06-29T15:58:34.620265+0000","last_fullsized":"2024-06-29T15:58:34.620265+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:00:27.903522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.638298+0000","last_change":"2024-06-29T15:58:34.634993+0000","last_active":"2024-06-29T15:58:34.638298+0000","last_peered":"2024-06-29T15:58:34.638298+0000","last_clean":"2024-06-29T15:58:34.638298+0000","last_became_active":"2024-06-29T15:58:34.634658+0000","last_became_peered":"2024-06-29T15:58:34.634658+0000","last_unstale":"2024-06-29T15:58:34.638298+0000","last_undegraded":"2024-06-29T15:58:34.638298+0000","last_fullsized":"2024-06-29T15:58:34.638298+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:27.552901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628290+0000","last_change":"2024-06-29T15:58:21.565805+0000","last_active":"2024-06-29T15:58:34.628290+0000","last_peered":"2024-06-29T15:58:34.628290+0000","last_clean":"2024-06-29T15:58:34.628290+0000","last_became_active":"2024-06-29T15:58:21.564982+0000","last_became_peered":"2024-06-29T15:58:21.564982+0000","last_unstale":"2024-06-29T15:58:34.628290+0000","last_undegraded":"2024-06-29T15:58:34.628290+0000","last_fullsized":"2024-06-29T15:58:34.628290+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:18.217838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"242'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622735+0000","last_change":"2024-06-29T15:58:20.554719+0000","last_active":"2024-06-29T15:58:34.622735+0000","last_peered":"2024-06-29T15:58:34.622735+0000","last_clean":"2024-06-29T15:58:34.622735+0000","last_became_active":"2024-06-29T15:58:20.554423+0000","last_became_peered":"2024-06-29T15:58:20.554423+0000","last_unstale":"2024-06-29T15:58:34.622735+0000","last_undegraded":"2024-06-29T15:58:34.622735+0000","last_fullsized":"2024-06-29T15:58:34.622735+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:36.112729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.655025+0000","last_change":"2024-06-29T15:58:34.648450+0000","last_active":"2024-06-29T15:58:34.655025+0000","last_peered":"2024-06-29T15:58:34.655025+0000","last_clean":"2024-06-29T15:58:34.655025+0000","last_became_active":"2024-06-29T15:58:34.648075+0000","last_became_peered":"2024-06-29T15:58:34.648075+0000","last_unstale":"2024-06-29T15:58:34.655025+0000","last_undegraded":"2024-06-29T15:58:34.655025+0000","last_fullsized":"2024-06-29T15:58:34.655025+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:47:30.699858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618892+0000","last_change":"2024-06-29T15:58:21.559427+0000","last_active":"2024-06-29T15:58:34.618892+0000","last_peered":"2024-06-29T15:58:34.618892+0000","last_clean":"2024-06-29T15:58:34.618892+0000","last_became_active":"2024-06-29T15:58:21.558942+0000","last_became_peered":"2024-06-29T15:58:21.558942+0000","last_unstale":"2024-06-29T15:58:34.618892+0000","last_undegraded":"2024-06-29T15:58:34.618892+0000","last_fullsized":"2024-06-29T15:58:34.618892+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:36.096648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626387+0000","last_change":"2024-06-29T15:58:20.559651+0000","last_active":"2024-06-29T15:58:34.626387+0000","last_peered":"2024-06-29T15:58:34.626387+0000","last_clean":"2024-06-29T15:58:34.626387+0000","last_became_active":"2024-06-29T15:58:20.559323+0000","last_became_peered":"2024-06-29T15:58:20.559323+0000","last_unstale":"2024-06-29T15:58:34.626387+0000","last_undegraded":"2024-06-29T15:58:34.626387+0000","last_fullsized":"2024-06-29T15:58:34.626387+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:18:36.258363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.637600+0000","last_change":"2024-06-29T15:58:34.636916+0000","last_active":"2024-06-29T15:58:34.637600+0000","last_peered":"2024-06-29T15:58:34.637600+0000","last_clean":"2024-06-29T15:58:34.637600+0000","last_became_active":"2024-06-29T15:58:34.636611+0000","last_became_peered":"2024-06-29T15:58:34.636611+0000","last_unstale":"2024-06-29T15:58:34.637600+0000","last_undegraded":"2024-06-29T15:58:34.637600+0000","last_fullsized":"2024-06-29T15:58:34.637600+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:21.903198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.621677+0000","last_change":"2024-06-29T15:58:21.545630+0000","last_active":"2024-06-29T15:58:34.621677+0000","last_peered":"2024-06-29T15:58:34.621677+0000","last_clean":"2024-06-29T15:58:34.621677+0000","last_became_active":"2024-06-29T15:58:21.545301+0000","last_became_peered":"2024-06-29T15:58:21.545301+0000","last_unstale":"2024-06-29T15:58:34.621677+0000","last_undegraded":"2024-06-29T15:58:34.621677+0000","last_fullsized":"2024-06-29T15:58:34.621677+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:36.336990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627349+0000","last_change":"2024-06-29T15:58:20.556695+0000","last_active":"2024-06-29T15:58:34.627349+0000","last_peered":"2024-06-29T15:58:34.627349+0000","last_clean":"2024-06-29T15:58:34.627349+0000","last_became_active":"2024-06-29T15:58:20.556053+0000","last_became_peered":"2024-06-29T15:58:20.556053+0000","last_unstale":"2024-06-29T15:58:34.627349+0000","last_undegraded":"2024-06-29T15:58:34.627349+0000","last_fullsized":"2024-06-29T15:58:34.627349+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:15.325464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.658955+0000","last_change":"2024-06-29T15:58:34.648233+0000","last_active":"2024-06-29T15:58:34.658955+0000","last_peered":"2024-06-29T15:58:34.658955+0000","last_clean":"2024-06-29T15:58:34.658955+0000","last_became_active":"2024-06-29T15:58:34.647411+0000","last_became_peered":"2024-06-29T15:58:34.647411+0000","last_unstale":"2024-06-29T15:58:34.658955+0000","last_undegraded":"2024-06-29T15:58:34.658955+0000","last_fullsized":"2024-06-29T15:58:34.658955+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:12.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626776+0000","last_change":"2024-06-29T15:58:21.565888+0000","last_active":"2024-06-29T15:58:34.626776+0000","last_peered":"2024-06-29T15:58:34.626776+0000","last_clean":"2024-06-29T15:58:34.626776+0000","last_became_active":"2024-06-29T15:58:21.565418+0000","last_became_peered":"2024-06-29T15:58:21.565418+0000","last_unstale":"2024-06-29T15:58:34.626776+0000","last_undegraded":"2024-06-29T15:58:34.626776+0000","last_fullsized":"2024-06-29T15:58:34.626776+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:57.843559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.628122+0000","last_change":"2024-06-29T15:58:20.557670+0000","last_active":"2024-06-29T15:58:34.628122+0000","last_peered":"2024-06-29T15:58:34.628122+0000","last_clean":"2024-06-29T15:58:34.628122+0000","last_became_active":"2024-06-29T15:58:20.557366+0000","last_became_peered":"2024-06-29T15:58:20.557366+0000","last_unstale":"2024-06-29T15:58:34.628122+0000","last_undegraded":"2024-06-29T15:58:34.628122+0000","last_fullsized":"2024-06-29T15:58:34.628122+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:48:08.866031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662407+0000","last_change":"2024-06-29T15:58:34.662002+0000","last_active":"2024-06-29T15:58:34.662407+0000","last_peered":"2024-06-29T15:58:34.662407+0000","last_clean":"2024-06-29T15:58:34.662407+0000","last_became_active":"2024-06-29T15:58:34.661589+0000","last_became_peered":"2024-06-29T15:58:34.661589+0000","last_unstale":"2024-06-29T15:58:34.662407+0000","last_undegraded":"2024-06-29T15:58:34.662407+0000","last_fullsized":"2024-06-29T15:58:34.662407+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:30.075819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.626174+0000","last_change":"2024-06-29T15:58:21.562317+0000","last_active":"2024-06-29T15:58:34.626174+0000","last_peered":"2024-06-29T15:58:34.626174+0000","last_clean":"2024-06-29T15:58:34.626174+0000","last_became_active":"2024-06-29T15:58:21.561968+0000","last_became_peered":"2024-06-29T15:58:21.561968+0000","last_unstale":"2024-06-29T15:58:34.626174+0000","last_undegraded":"2024-06-29T15:58:34.626174+0000","last_fullsized":"2024-06-29T15:58:34.626174+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.632416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.622286+0000","last_change":"2024-06-29T15:58:20.547790+0000","last_active":"2024-06-29T15:58:34.622286+0000","last_peered":"2024-06-29T15:58:34.622286+0000","last_clean":"2024-06-29T15:58:34.622286+0000","last_became_active":"2024-06-29T15:58:20.547481+0000","last_became_peered":"2024-06-29T15:58:20.547481+0000","last_unstale":"2024-06-29T15:58:34.622286+0000","last_undegraded":"2024-06-29T15:58:34.622286+0000","last_fullsized":"2024-06-29T15:58:34.622286+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:32.913616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.652099+0000","last_change":"2024-06-29T15:58:34.649154+0000","last_active":"2024-06-29T15:58:34.652099+0000","last_peered":"2024-06-29T15:58:34.652099+0000","last_clean":"2024-06-29T15:58:34.652099+0000","last_became_active":"2024-06-29T15:58:34.648835+0000","last_became_peered":"2024-06-29T15:58:34.648835+0000","last_unstale":"2024-06-29T15:58:34.652099+0000","last_undegraded":"2024-06-29T15:58:34.652099+0000","last_fullsized":"2024-06-29T15:58:34.652099+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:51:12.584396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.631820+0000","last_change":"2024-06-29T15:58:21.553783+0000","last_active":"2024-06-29T15:58:34.631820+0000","last_peered":"2024-06-29T15:58:34.631820+0000","last_clean":"2024-06-29T15:58:34.631820+0000","last_became_active":"2024-06-29T15:58:21.553390+0000","last_became_peered":"2024-06-29T15:58:21.553390+0000","last_unstale":"2024-06-29T15:58:34.631820+0000","last_undegraded":"2024-06-29T15:58:34.631820+0000","last_fullsized":"2024-06-29T15:58:34.631820+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:01.841016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.618448+0000","last_change":"2024-06-29T15:58:20.561399+0000","last_active":"2024-06-29T15:58:34.618448+0000","last_peered":"2024-06-29T15:58:34.618448+0000","last_clean":"2024-06-29T15:58:34.618448+0000","last_became_active":"2024-06-29T15:58:20.560716+0000","last_became_peered":"2024-06-29T15:58:20.560716+0000","last_unstale":"2024-06-29T15:58:34.618448+0000","last_undegraded":"2024-06-29T15:58:34.618448+0000","last_fullsized":"2024-06-29T15:58:34.618448+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:42:19.927414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.662632+0000","last_change":"2024-06-29T15:58:34.662010+0000","last_active":"2024-06-29T15:58:34.662632+0000","last_peered":"2024-06-29T15:58:34.662632+0000","last_clean":"2024-06-29T15:58:34.662632+0000","last_became_active":"2024-06-29T15:58:34.661637+0000","last_became_peered":"2024-06-29T15:58:34.661637+0000","last_unstale":"2024-06-29T15:58:34.662632+0000","last_undegraded":"2024-06-29T15:58:34.662632+0000","last_fullsized":"2024-06-29T15:58:34.662632+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:10.132577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.629876+0000","last_change":"2024-06-29T15:58:21.563096+0000","last_active":"2024-06-29T15:58:34.629876+0000","last_peered":"2024-06-29T15:58:34.629876+0000","last_clean":"2024-06-29T15:58:34.629876+0000","last_became_active":"2024-06-29T15:58:21.562778+0000","last_became_peered":"2024-06-29T15:58:21.562778+0000","last_unstale":"2024-06-29T15:58:34.629876+0000","last_undegraded":"2024-06-29T15:58:34.629876+0000","last_fullsized":"2024-06-29T15:58:34.629876+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:52:20.808675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":20,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627706+0000","last_change":"2024-06-29T15:58:20.554308+0000","last_active":"2024-06-29T15:58:34.627706+0000","last_peered":"2024-06-29T15:58:34.627706+0000","last_clean":"2024-06-29T15:58:34.627706+0000","last_became_active":"2024-06-29T15:58:20.553976+0000","last_became_peered":"2024-06-29T15:58:20.553976+0000","last_unstale":"2024-06-29T15:58:34.627706+0000","last_undegraded":"2024-06-29T15:58:34.627706+0000","last_fullsized":"2024-06-29T15:58:34.627706+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:13:16.241344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.651880+0000","last_change":"2024-06-29T15:58:34.642709+0000","last_active":"2024-06-29T15:58:34.651880+0000","last_peered":"2024-06-29T15:58:34.651880+0000","last_clean":"2024-06-29T15:58:34.651880+0000","last_became_active":"2024-06-29T15:58:34.641698+0000","last_became_peered":"2024-06-29T15:58:34.641698+0000","last_unstale":"2024-06-29T15:58:34.651880+0000","last_undegraded":"2024-06-29T15:58:34.651880+0000","last_fullsized":"2024-06-29T15:58:34.651880+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:33.607315+0000","last_clean_scrub_stamp":"2024-06-29T15:58:33.607315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:22.592796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":18,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.624166+0000","last_change":"2024-06-29T15:58:21.550432+0000","last_active":"2024-06-29T15:58:34.624166+0000","last_peered":"2024-06-29T15:58:34.624166+0000","last_clean":"2024-06-29T15:58:34.624166+0000","last_became_active":"2024-06-29T15:58:21.550075+0000","last_became_peered":"2024-06-29T15:58:21.550075+0000","last_unstale":"2024-06-29T15:58:34.624166+0000","last_undegraded":"2024-06-29T15:58:34.624166+0000","last_fullsized":"2024-06-29T15:58:34.624166+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:20.525334+0000","last_clean_scrub_stamp":"2024-06-29T15:58:20.525334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:24.427535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"241'1","reported_seq":21,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627721+0000","last_change":"2024-06-29T15:58:20.561371+0000","last_active":"2024-06-29T15:58:34.627721+0000","last_peered":"2024-06-29T15:58:34.627721+0000","last_clean":"2024-06-29T15:58:34.627721+0000","last_became_active":"2024-06-29T15:58:20.561026+0000","last_became_peered":"2024-06-29T15:58:20.561026+0000","last_unstale":"2024-06-29T15:58:34.627721+0000","last_undegraded":"2024-06-29T15:58:34.627721+0000","last_fullsized":"2024-06-29T15:58:34.627721+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:19.516971+0000","last_clean_scrub_stamp":"2024-06-29T15:58:19.516971+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:47:07.861501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"228'247","reported_seq":1196,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-06-29T15:58:34.627406+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:58:34.627406+0000","last_peered":"2024-06-29T15:58:34.627406+0000","last_clean":"2024-06-29T15:58:34.627406+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:58:34.627406+0000","last_undegraded":"2024-06-29T15:58:34.627406+0000","last_fullsized":"2024-06-29T15:58:34.627406+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543300,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52812,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3670016,"data_stored":2178415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543301,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64728,"kb_used_data":3020,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307112,"statfs":{"total":96636764160,"available":96570482688,"internally_reserved":0,"allocated":3092480,"data_stored":1596162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543300,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70276,"kb_used_data":2996,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301564,"statfs":{"total":96636764160,"available":96564801536,"internally_reserved":0,"allocated":3067904,"data_stored":1587923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543299,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3024,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3096576,"data_stored":1597698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510597,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48104,"kb_used_data":3036,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323736,"statfs":{"total":96636764160,"available":96587505664,"internally_reserved":0,"allocated":3108864,"data_stored":1599599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576005,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48108,"kb_used_data":3036,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323732,"statfs":{"total":96636764160,"available":96587501568,"internally_reserved":0,"allocated":3108864,"data_stored":1598884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510596,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52252,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319588,"statfs":{"total":96636764160,"available":96583258112,"internally_reserved":0,"allocated":3088384,"data_stored":1596140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510598,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61140,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310700,"statfs":{"total":96636764160,"available":96574156800,"internally_reserved":0,"allocated":3674112,"data_stored":2181146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:58:39.119 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:58:39.119 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:39.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.268+0000 7f29cb438640 1 -- 172.21.15.59:0/180011135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 msgr2=0x7f29c415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.268+0000 7f29cb438640 1 --2- 172.21.15.59:0/180011135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 0x7f29c415f9b0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f29c40d15d0 tx=0x7f29bc02aac0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.268+0000 7f29cb438640 1 -- 172.21.15.59:0/180011135 shutdown_connections 2024-06-29T15:58:39.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.268+0000 7f29cb438640 1 --2- 172.21.15.59:0/180011135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 0x7f29c415f9b0 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.268+0000 7f29cb438640 1 --2- 172.21.15.59:0/180011135 >> v2:172.21.15.153:3300/0 conn(0x7f29c40d23c0 0x7f29c40b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.268+0000 7f29cb438640 1 -- 172.21.15.59:0/180011135 >> 172.21.15.59:0/180011135 conn(0x7f29c40d3ee0 msgr2=0x7f29c40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:39.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.269+0000 7f29cb438640 1 -- 172.21.15.59:0/180011135 shutdown_connections 2024-06-29T15:58:39.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.269+0000 7f29cb438640 1 -- 172.21.15.59:0/180011135 wait complete. 2024-06-29T15:58:39.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.270+0000 7f29cb438640 1 Processor -- start 2024-06-29T15:58:39.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.270+0000 7f29cb438640 1 -- start start 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29cb438640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c40b3ed0 0x7f29c40c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29cb438640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c40d23c0 0x7f29c415dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29cb438640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 0x7f29c41572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29cb438640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f29c4160280 con 0x7f29c415efe0 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29cb438640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29c4160100 con 0x7f29c40b3ed0 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29cb438640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f29c4160400 con 0x7f29c40d23c0 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 0x7f29c41572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 0x7f29c41572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50234/0 (socket says 172.21.15.59:50234) 2024-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 -- 172.21.15.59:0/272164970 learned_addr learned my addr 172.21.15.59:0/272164970 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c89d3640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c40d23c0 0x7f29c415dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c91d4640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c40b3ed0 0x7f29c40c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 -- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c40d23c0 msgr2=0x7f29c415dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c40d23c0 0x7f29c415dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 -- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c40b3ed0 msgr2=0x7f29c40c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c40b3ed0 0x7f29c40c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 -- 172.21.15.59:0/272164970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c4159340 con 0x7f29c415efe0 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c89d3640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c40d23c0 0x7f29c415dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c91d4640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c40b3ed0 0x7f29c40c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29c99d5640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 0x7f29c41572c0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f29bc02b0b0 tx=0x7f29bc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29b9ffb640 1 -- 172.21.15.59:0/272164970 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc004250 con 0x7f29c415efe0 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.272+0000 7f29b9ffb640 1 -- 172.21.15.59:0/272164970 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29bc0043f0 con 0x7f29c415efe0 2024-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.273+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c4159620 con 0x7f29c415efe0 2024-06-29T15:58:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.273+0000 7f29b9ffb640 1 -- 172.21.15.59:0/272164970 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc004820 con 0x7f29c415efe0 2024-06-29T15:58:39.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.273+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c4159be0 con 0x7f29c415efe0 2024-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.274+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c41590b0 con 0x7f29c415efe0 2024-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.275+0000 7f29b9ffb640 1 -- 172.21.15.59:0/272164970 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f29bc049430 con 0x7f29c415efe0 2024-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.275+0000 7f29b9ffb640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29a8076510 0x7f29a8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.275+0000 7f29b9ffb640 1 -- 172.21.15.59:0/272164970 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f29bc0b8f00 con 0x7f29c415efe0 2024-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.275+0000 7f29c91d4640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29a8076510 0x7f29a8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.276+0000 7f29c91d4640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29a8076510 0x7f29a8078a20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f29c40d2e40 tx=0x7f29c0005dd0 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:39.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.280+0000 7f29b9ffb640 1 -- 172.21.15.59:0/272164970 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f29bc032030 con 0x7f29c415efe0 2024-06-29T15:58:39.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.508+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29c4158e50 con 0x7f29c415efe0 2024-06-29T15:58:39.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.509+0000 7f29b9ffb640 1 -- 172.21.15.59:0/272164970 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6568 (secure 0 0 0) 0x7f29bc081f60 con 0x7f29c415efe0 2024-06-29T15:58:39.510 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:39.510 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":137,"btime":"2024-06-29T15:58:37:645635+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:20.935697+0000","modified":"2024-06-29T15:58:23.956772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/177893499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":177893499},{"type":"v1","addr":"172.21.15.153:6837","nonce":177893499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12318,"qdb_cluster":[12318]},"id":15},{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:58:39.511 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 137 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.511+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29a8076510 msgr2=0x7f29a8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.511+0000 7f29cb438640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29a8076510 0x7f29a8078a20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f29c40d2e40 tx=0x7f29c0005dd0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.511+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 msgr2=0x7f29c41572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.511+0000 7f29cb438640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 0x7f29c41572c0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f29bc02b0b0 tx=0x7f29bc003f10 comp rx=0 tx=0).stop 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.512+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 shutdown_connections 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.512+0000 7f29cb438640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f29a8076510 0x7f29a8078a20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.512+0000 7f29cb438640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29c415efe0 0x7f29c41572c0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.512+0000 7f29cb438640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f29c40d23c0 0x7f29c415dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.512+0000 7f29cb438640 1 --2- 172.21.15.59:0/272164970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29c40b3ed0 0x7f29c40c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.512+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 >> 172.21.15.59:0/272164970 conn(0x7f29c40d3ee0 msgr2=0x7f29c406d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:39.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.512+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 shutdown_connections 2024-06-29T15:58:39.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:39.512+0000 7f29cb438640 1 -- 172.21.15.59:0/272164970 wait complete. 2024-06-29T15:58:39.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 137, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:58:34.012091+0000', 'modified': '2024-06-29T15:58:37.031172+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12303}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12303': {'gid': 12303, 'name': 'c', 'rank': 0, 'incarnation': 134, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.59:6835/1525995706', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1525995706}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1525995706}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12303, 'qdb_cluster': [12303]} 2024-06-29T15:58:39.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:58:39.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-06-29T15:58:40.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.047+0000 7f450cc7f640 1 -- 172.21.15.59:0/1997744700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450813e9b0 msgr2=0x7f450813edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.047+0000 7f450cc7f640 1 --2- 172.21.15.59:0/1997744700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450813e9b0 0x7f450813edc0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009830 tx=0x7f44fc02a480 comp rx=0 tx=0).stop 2024-06-29T15:58:40.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.048+0000 7f450cc7f640 1 -- 172.21.15.59:0/1997744700 shutdown_connections 2024-06-29T15:58:40.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.048+0000 7f450cc7f640 1 --2- 172.21.15.59:0/1997744700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450813e9b0 0x7f450813edc0 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.048+0000 7f450cc7f640 1 --2- 172.21.15.59:0/1997744700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450815ceb0 0x7f45080b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.048+0000 7f450cc7f640 1 -- 172.21.15.59:0/1997744700 >> 172.21.15.59:0/1997744700 conn(0x7f45080b6fb0 msgr2=0x7f45080b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.048+0000 7f450cc7f640 1 -- 172.21.15.59:0/1997744700 shutdown_connections 2024-06-29T15:58:40.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.048+0000 7f450cc7f640 1 -- 172.21.15.59:0/1997744700 wait complete. 2024-06-29T15:58:40.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.050+0000 7f450cc7f640 1 Processor -- start 2024-06-29T15:58:40.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.050+0000 7f450cc7f640 1 -- start start 2024-06-29T15:58:40.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f450cc7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450813e9b0 0x7f450813b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f450cc7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45081414c0 0x7f450813d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f450cc7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450815ceb0 0x7f450813d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f450cc7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f45080b2ab0 con 0x7f450815ceb0 2024-06-29T15:58:40.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f450cc7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45080b2930 con 0x7f45081414c0 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f450cc7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45080b2c30 con 0x7f450813e9b0 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4506d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450815ceb0 0x7f450813d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4506d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450815ceb0 0x7f450813d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50244/0 (socket says 172.21.15.59:50244) 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4506d9d640 1 -- 172.21.15.59:0/3092599859 learned_addr learned my addr 172.21.15.59:0/3092599859 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f450659c640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450813e9b0 0x7f450813b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4505d9b640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45081414c0 0x7f450813d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4506d9d640 1 -- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450813e9b0 msgr2=0x7f450813b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4506d9d640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450813e9b0 0x7f450813b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4506d9d640 1 -- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45081414c0 msgr2=0x7f450813d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4506d9d640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45081414c0 0x7f450813d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.051+0000 7f4506d9d640 1 -- 172.21.15.59:0/3092599859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45080c8a10 con 0x7f450815ceb0 2024-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.052+0000 7f4506d9d640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450815ceb0 0x7f450813d620 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f450000c390 tx=0x7f450000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.052+0000 7f44f6ffd640 1 -- 172.21.15.59:0/3092599859 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500017070 con 0x7f450815ceb0 2024-06-29T15:58:40.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.052+0000 7f44f6ffd640 1 -- 172.21.15.59:0/3092599859 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45000035f0 con 0x7f450815ceb0 2024-06-29T15:58:40.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.052+0000 7f450cc7f640 1 -- 172.21.15.59:0/3092599859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45080c8cf0 con 0x7f450815ceb0 2024-06-29T15:58:40.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.052+0000 7f44f6ffd640 1 -- 172.21.15.59:0/3092599859 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500003a40 con 0x7f450815ceb0 2024-06-29T15:58:40.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.052+0000 7f450cc7f640 1 -- 172.21.15.59:0/3092599859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45080c92b0 con 0x7f450815ceb0 2024-06-29T15:58:40.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.054+0000 7f44f6ffd640 1 -- 172.21.15.59:0/3092599859 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f450001a070 con 0x7f450815ceb0 2024-06-29T15:58:40.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.054+0000 7f44f6ffd640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f44e8076510 0x7f44e8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.055+0000 7f44f6ffd640 1 -- 172.21.15.59:0/3092599859 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f4500097410 con 0x7f450815ceb0 2024-06-29T15:58:40.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.055+0000 7f450659c640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f44e8076510 0x7f44e8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.055+0000 7f4513c10740 1 Processor -- start 2024-06-29T15:58:40.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.055+0000 7f450659c640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f44e8076510 0x7f44e8078a20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f44f8005c40 tx=0x7f44f800a370 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.056+0000 7f4513c10740 1 -- start start 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4513c10740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0addc3270 0x55d0ae1af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4513c10740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0add9a3b0 0x55d0ae1933a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4513c10740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 0x55d0ae1938e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4513c10740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d0ae12bbc0 con 0x55d0add9a3b0 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4513c10740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d0ae0da630 con 0x55d0adda2590 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4513c10740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d0ae0c9bd0 con 0x55d0addc3270 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4505d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0add9a3b0 0x55d0ae1933a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4505d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0add9a3b0 0x55d0ae1933a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50254/0 (socket says 172.21.15.59:50254) 2024-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4505d9b640 1 -- 172.21.15.59:0/1286525284 learned_addr learned my addr 172.21.15.59:0/1286525284 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f450659c640 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0addc3270 0x55d0ae1af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4506d9d640 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 0x55d0ae1938e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4505d9b640 1 -- 172.21.15.59:0/1286525284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0addc3270 msgr2=0x55d0ae1af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4505d9b640 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0addc3270 0x55d0ae1af6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4505d9b640 1 -- 172.21.15.59:0/1286525284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 msgr2=0x55d0ae1938e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4505d9b640 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 0x55d0ae1938e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.057+0000 7f4505d9b640 1 -- 172.21.15.59:0/1286525284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0ae0c6ca0 con 0x55d0add9a3b0 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.058+0000 7f4505d9b640 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0add9a3b0 0x55d0ae1933a0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009830 tx=0x7f44fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1074f21019ac7b28 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.058+0000 7f44e7fff640 1 -- 172.21.15.59:0/1286525284 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44fc005020 con 0x55d0add9a3b0 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.058+0000 7f44e7fff640 1 -- 172.21.15.59:0/1286525284 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44fc040070 con 0x55d0add9a3b0 2024-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.058+0000 7f44e7fff640 1 -- 172.21.15.59:0/1286525284 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0054f0 con 0x55d0add9a3b0 2024-06-29T15:58:40.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.058+0000 7f4513c10740 1 -- 172.21.15.59:0/1286525284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0add9a3b0 msgr2=0x55d0ae1933a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.058+0000 7f4513c10740 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0add9a3b0 0x55d0ae1933a0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009830 tx=0x7f44fc0046c0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.059+0000 7f4513c10740 1 -- 172.21.15.59:0/1286525284 shutdown_connections 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.059+0000 7f4513c10740 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 0x55d0ae1938e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.059+0000 7f4513c10740 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0add9a3b0 0x55d0ae1933a0 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.059+0000 7f4513c10740 1 --2- 172.21.15.59:0/1286525284 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0addc3270 0x55d0ae1af6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.059+0000 7f4513c10740 1 -- 172.21.15.59:0/1286525284 >> 172.21.15.59:0/1286525284 conn(0x55d0addb73c0 msgr2=0x55d0ae166f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.059+0000 7f4513c10740 1 -- 172.21.15.59:0/1286525284 shutdown_connections 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.059+0000 7f4513c10740 1 -- 172.21.15.59:0/1286525284 wait complete. 2024-06-29T15:58:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.060+0000 7f4513c10740 1 -- start start 2024-06-29T15:58:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.060+0000 7f4513c10740 1 Processor -- start 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4513c10740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 0x55d0ae192b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4513c10740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 0x55d0ae18b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4513c10740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4513c10740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d0ae067ae0 con 0x55d0addc3270 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4513c10740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d0ae0cdc10 con 0x55d0add9a3b0 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4513c10740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d0ade3d620 con 0x55d0adda2590 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4513c10740 10 client.0 _ll_register_callbacks cb 0x55d0ae074480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4506d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4506d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50258/0 (socket says 172.21.15.59:50258) 2024-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4506d9d640 1 -- 172.21.15.59:0/371135435 learned_addr learned my addr 172.21.15.59:0/371135435 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f450659c640 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 0x55d0ae192b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.061+0000 7f4505d9b640 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 0x55d0ae18b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4506d9d640 1 -- 172.21.15.59:0/371135435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 msgr2=0x55d0ae18b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4506d9d640 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 0x55d0ae18b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4506d9d640 1 -- 172.21.15.59:0/371135435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 msgr2=0x55d0ae192b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4506d9d640 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 0x55d0ae192b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4506d9d640 1 -- 172.21.15.59:0/371135435 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0ade2e850 con 0x55d0addc3270 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4506d9d640 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a7e30 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f450005fce0 tx=0x7f4500003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb511be726c50cd server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f44e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f44e57fa640 1 -- 172.21.15.59:0/371135435 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450005f1b0 con 0x55d0addc3270 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f44e57fa640 1 -- 172.21.15.59:0/371135435 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f4500066300 con 0x55d0addc3270 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4513c10740 1 -- 172.21.15.59:0/371135435 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44fc040070 con 0x55d0addc3270 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f44e57fa640 1 -- 172.21.15.59:0/371135435 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7f450007e050 con 0x55d0addc3270 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4513c10740 10 client.12609 fetch_fsmap learned FSMap version 137 2024-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f4513c10740 1 -- 172.21.15.59:0/371135435 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=2+}) -- 0x55d0adea56b0 con 0x55d0addc3270 2024-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f44e57fa640 1 -- 172.21.15.59:0/371135435 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44fc040070 con 0x55d0addc3270 2024-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.062+0000 7f44e57fa640 1 -- 172.21.15.59:0/371135435 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 137) ==== 4149+0+0 (secure 0 0 0) 0x7f450005b6c0 con 0x55d0addc3270 2024-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f4513c10740 10 client.12609 fetch_fsmap finished waiting for FSMap version 137 2024-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f4513c10740 10 client.12609 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f4513c10740 20 client.12609 populate_metadata read hostname 'smithi059' 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f4513c10740 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] conn(0x55d0add81e40 0x55d0ae1a6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f4513c10740 4 client.12609 mds_command: new command op to 12303 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f4513c10740 1 -- 172.21.15.59:0/371135435 --> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0ae0dc580 con 0x55d0add81e40 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f450659c640 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] conn(0x55d0add81e40 0x55d0ae1a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f44e57fa640 10 client.12609 ms_handle_connect on v2:172.21.15.59:6834/1525995706 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.063+0000 7f450659c640 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] conn(0x55d0add81e40 0x55d0ae1a6270 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cb5e0f03eb03140 server_cookie=b75505a9007be718 in_seq=0 out_seq=0 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f44e57fa640 1 -- 172.21.15.59:0/371135435 <== mds.0 v2:172.21.15.59:6834/1525995706 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d0ae0dc580 con 0x55d0add81e40 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f44e57fa640 10 client.12609 handle_command_reply: tid=0 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 client.12609 shutdown 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 -- 172.21.15.59:0/371135435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 msgr2=0x55d0ae1a7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a7e30 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f450005fce0 tx=0x7f4500003f50 comp rx=0 tx=0).stop 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 -- 172.21.15.59:0/371135435 shutdown_connections 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] conn(0x55d0add81e40 0x55d0ae1a6270 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a7e30 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 0x55d0ae18b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f44e57fa640 0 client.12609 ms_handle_reset on v2:172.21.15.59:6834/1525995706 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 --2- 172.21.15.59:0/371135435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 0x55d0ae192b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 -- 172.21.15.59:0/371135435 >> 172.21.15.59:0/371135435 conn(0x55d0addb73c0 msgr2=0x55d0ae18fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 -- 172.21.15.59:0/371135435 shutdown_connections 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 1 -- 172.21.15.59:0/371135435 wait complete. 2024-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.064+0000 7f4513c10740 20 client.12609 trim_cache size 0 max 16384 2024-06-29T15:58:40.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.084+0000 7f4513c10740 1 Processor -- start 2024-06-29T15:58:40.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.084+0000 7f4513c10740 1 -- start start 2024-06-29T15:58:40.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4513c10740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0add9a3b0 0x55d0ae1925d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4513c10740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 0x55d0ae192b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4513c10740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4513c10740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4508202990 con 0x55d0addc3270 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4513c10740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45080cac00 con 0x55d0adda2590 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4513c10740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4508200a00 con 0x55d0add9a3b0 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4506d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4506d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50270/0 (socket says 172.21.15.59:50270) 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4506d9d640 1 -- 172.21.15.59:0/398494572 learned_addr learned my addr 172.21.15.59:0/398494572 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4505d9b640 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 0x55d0ae192b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f450659c640 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0add9a3b0 0x55d0ae1925d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4506d9d640 1 -- 172.21.15.59:0/398494572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0add9a3b0 msgr2=0x55d0ae1925d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:40.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.085+0000 7f4506d9d640 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0add9a3b0 0x55d0ae1925d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.086+0000 7f4506d9d640 1 -- 172.21.15.59:0/398494572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 msgr2=0x55d0ae192b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.086+0000 7f4506d9d640 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 0x55d0ae192b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.086+0000 7f4506d9d640 1 -- 172.21.15.59:0/398494572 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0ae190440 con 0x55d0addc3270 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.086+0000 7f4506d9d640 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a6250 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f450005c340 tx=0x7f4500000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d821a849b2e195e server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.086+0000 7f44e77fe640 1 -- 172.21.15.59:0/398494572 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450006d070 con 0x55d0addc3270 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.086+0000 7f44e77fe640 1 -- 172.21.15.59:0/398494572 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4500077070 con 0x55d0addc3270 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.086+0000 7f44e77fe640 1 -- 172.21.15.59:0/398494572 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450005c3a0 con 0x55d0addc3270 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 -- 172.21.15.59:0/398494572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 msgr2=0x55d0ae1a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a6250 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f450005c340 tx=0x7f4500000ed0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 -- 172.21.15.59:0/398494572 shutdown_connections 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1a6250 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0adda2590 0x55d0ae192b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 --2- 172.21.15.59:0/398494572 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0add9a3b0 0x55d0ae1925d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 -- 172.21.15.59:0/398494572 >> 172.21.15.59:0/398494572 conn(0x55d0addb73c0 msgr2=0x55d0ae15f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 -- 172.21.15.59:0/398494572 shutdown_connections 2024-06-29T15:58:40.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 -- 172.21.15.59:0/398494572 wait complete. 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.087+0000 7f4513c10740 1 -- start start 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.088+0000 7f4513c10740 1 Processor -- start 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4513c10740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 0x55d0ae18e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4513c10740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 0x55d0ae22d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4513c10740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1925d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4513c10740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x55d0ae1a7950 con 0x55d0addc3270 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4513c10740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d0ae1a72f0 con 0x55d0add9a3b0 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4513c10740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55d0ae1c1870 con 0x55d0adda2590 2024-06-29T15:58:40.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4513c10740 10 client.0 _ll_register_callbacks cb 0x7f4500001130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4506d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1925d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4506d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1925d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50278/0 (socket says 172.21.15.59:50278) 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4506d9d640 1 -- 172.21.15.59:0/3535107443 learned_addr learned my addr 172.21.15.59:0/3535107443 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f450659c640 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 0x55d0ae18e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.089+0000 7f4505d9b640 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 0x55d0ae22d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4506d9d640 1 -- 172.21.15.59:0/3535107443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 msgr2=0x55d0ae22d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4506d9d640 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 0x55d0ae22d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4506d9d640 1 -- 172.21.15.59:0/3535107443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 msgr2=0x55d0ae18e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4506d9d640 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 0x55d0ae18e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4506d9d640 1 -- 172.21.15.59:0/3535107443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0ae1664f0 con 0x55d0addc3270 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4506d9d640 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1925d0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f450005bdd0 tx=0x7f4500097030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d1719f8fb0d748 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f44e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f44e57fa640 1 -- 172.21.15.59:0/3535107443 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450006d070 con 0x55d0addc3270 2024-06-29T15:58:40.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 1 -- 172.21.15.59:0/3535107443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4500077070 con 0x55d0addc3270 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f44e57fa640 1 -- 172.21.15.59:0/3535107443 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f450005f1b0 con 0x55d0addc3270 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f44e57fa640 1 -- 172.21.15.59:0/3535107443 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7f4500087050 con 0x55d0addc3270 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 10 client.12615 fetch_fsmap learned FSMap version 137 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 1 -- 172.21.15.59:0/3535107443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=2+}) -- 0x55d0ae1c63d0 con 0x55d0addc3270 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f44e57fa640 1 -- 172.21.15.59:0/3535107443 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4500077070 con 0x55d0addc3270 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f44e57fa640 1 -- 172.21.15.59:0/3535107443 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 137) ==== 4149+0+0 (secure 0 0 0) 0x7f450008a080 con 0x55d0addc3270 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 10 client.12615 fetch_fsmap finished waiting for FSMap version 137 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 10 client.12615 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 20 client.12615 populate_metadata read hostname 'smithi059' 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] conn(0x55d0ae230190 0x55d0ae2305a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 4 client.12615 mds_command: new command op to 12303 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.090+0000 7f4513c10740 1 -- 172.21.15.59:0/3535107443 --> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] -- command(tid 0: {"prefix": "status"}) -- 0x55d0ae1a97a0 con 0x55d0ae230190 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.091+0000 7f450659c640 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] conn(0x55d0ae230190 0x55d0ae2305a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.091+0000 7f44e57fa640 10 client.12615 ms_handle_connect on v2:172.21.15.59:6834/1525995706 2024-06-29T15:58:40.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.091+0000 7f450659c640 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] conn(0x55d0ae230190 0x55d0ae2305a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa3bdc1bf6f3c6e8 server_cookie=13e306b2ec351c in_seq=0 out_seq=0 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.091+0000 7f44e57fa640 1 -- 172.21.15.59:0/3535107443 <== mds.0 v2:172.21.15.59:6834/1525995706 1 ==== command_reply(tid 0: 0 ) ==== 8+0+338 (crc 0 0 0) 0x55d0ae1a97a0 con 0x55d0ae230190 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.091+0000 7f44e57fa640 10 client.12615 handle_command_reply: tid=0 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.091+0000 7f4513c10740 1 client.12615 shutdown 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.091+0000 7f4513c10740 1 -- 172.21.15.59:0/3535107443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 msgr2=0x55d0ae1925d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.091+0000 7f4513c10740 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1925d0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f450005bdd0 tx=0x7f4500097030 comp rx=0 tx=0).stop 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 1 -- 172.21.15.59:0/3535107443 shutdown_connections 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.59:6834/1525995706,v1:172.21.15.59:6835/1525995706] conn(0x55d0ae230190 0x55d0ae2305a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d0addc3270 0x55d0ae1925d0 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55d0adda2590 0x55d0ae22d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 1 --2- 172.21.15.59:0/3535107443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d0add9a3b0 0x55d0ae18e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f44e57fa640 0 client.12615 ms_handle_reset on v2:172.21.15.59:6834/1525995706 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 1 -- 172.21.15.59:0/3535107443 >> 172.21.15.59:0/3535107443 conn(0x55d0addb73c0 msgr2=0x55d0ae1af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 1 -- 172.21.15.59:0/3535107443 shutdown_connections 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 1 -- 172.21.15.59:0/3535107443 wait complete. 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f4513c10740 20 client.12615 trim_cache size 0 max 16384 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 12303, 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:58:40.098 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "backup_fs", 2024-06-29T15:58:40.099 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.4770018059999996, 2024-06-29T15:58:40.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 137, 2024-06-29T15:58:40.099 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 246, 2024-06-29T15:58:40.099 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 246, 2024-06-29T15:58:40.099 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 23.166160173000002 2024-06-29T15:58:40.099 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:58:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 -- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f44e8076510 msgr2=0x7f44e8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f44e8076510 0x7f44e8078a20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f44f8005c40 tx=0x7f44f800a370 comp rx=0 tx=0).stop 2024-06-29T15:58:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 -- 172.21.15.59:0/3092599859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450815ceb0 msgr2=0x7f450813d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450815ceb0 0x7f450813d620 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f450000c390 tx=0x7f450000c850 comp rx=0 tx=0).stop 2024-06-29T15:58:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 -- 172.21.15.59:0/3092599859 shutdown_connections 2024-06-29T15:58:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f44e8076510 0x7f44e8078a20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f450815ceb0 0x7f450813d620 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45081414c0 0x7f450813d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 --2- 172.21.15.59:0/3092599859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f450813e9b0 0x7f450813b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 -- 172.21.15.59:0/3092599859 >> 172.21.15.59:0/3092599859 conn(0x7f45080b6fb0 msgr2=0x7f45080ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 -- 172.21.15.59:0/3092599859 shutdown_connections 2024-06-29T15:58:40.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.092+0000 7f450cc7f640 1 -- 172.21.15.59:0/3092599859 wait complete. 2024-06-29T15:58:40.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:58:40.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.222+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1205432782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf080d23c0 msgr2=0x7fbf080b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.222+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1205432782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf080d23c0 0x7fbf080b3990 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fbf08161d40 tx=0x7fbef802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.222+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1205432782 shutdown_connections 2024-06-29T15:58:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.222+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1205432782 >> v2:172.21.15.153:3300/0 conn(0x7fbf0815efe0 0x7fbf0815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.222+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1205432782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf080d23c0 0x7fbf080b3990 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.222+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1205432782 >> 172.21.15.59:0/1205432782 conn(0x7fbf080d3ee0 msgr2=0x7fbf080d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.222+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1205432782 shutdown_connections 2024-06-29T15:58:40.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.222+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1205432782 wait complete. 2024-06-29T15:58:40.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.224+0000 7fbf0d1e7640 1 Processor -- start 2024-06-29T15:58:40.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.225+0000 7fbf0d1e7640 1 -- start start 2024-06-29T15:58:40.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0d1e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf080b3ed0 0x7fbf08153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0d1e7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf080d23c0 0x7fbf081f5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0d1e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf0815efe0 0x7fbf081e8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0d1e7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fbf08160280 con 0x7fbf0815efe0 2024-06-29T15:58:40.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0d1e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf08160100 con 0x7fbf080b3ed0 2024-06-29T15:58:40.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0d1e7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbf08160400 con 0x7fbf080d23c0 2024-06-29T15:58:40.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf0815efe0 0x7fbf081e8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf06d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf080b3ed0 0x7fbf08153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf0815efe0 0x7fbf081e8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50292/0 (socket says 172.21.15.59:50292) 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0759e640 1 -- 172.21.15.59:0/1872519699 learned_addr learned my addr 172.21.15.59:0/1872519699 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0659c640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf080d23c0 0x7fbf081f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0759e640 1 -- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf080d23c0 msgr2=0x7fbf081f5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0759e640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf080d23c0 0x7fbf081f5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0759e640 1 -- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf080b3ed0 msgr2=0x7fbf08153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0759e640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf080b3ed0 0x7fbf08153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.226+0000 7fbf0759e640 1 -- 172.21.15.59:0/1872519699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf08152da0 con 0x7fbf0815efe0 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.227+0000 7fbf0759e640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf0815efe0 0x7fbf081e8990 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000e960 tx=0x7fbf0000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.227+0000 7fbeef7fe640 1 -- 172.21.15.59:0/1872519699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf00004d20 con 0x7fbf0815efe0 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.227+0000 7fbeef7fe640 1 -- 172.21.15.59:0/1872519699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf0000cac0 con 0x7fbf0815efe0 2024-06-29T15:58:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.227+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf08153080 con 0x7fbf0815efe0 2024-06-29T15:58:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.227+0000 7fbeef7fe640 1 -- 172.21.15.59:0/1872519699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf00005020 con 0x7fbf0815efe0 2024-06-29T15:58:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.227+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf08153640 con 0x7fbf0815efe0 2024-06-29T15:58:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.228+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf080b3990 con 0x7fbf0815efe0 2024-06-29T15:58:40.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.229+0000 7fbeef7fe640 1 -- 172.21.15.59:0/1872519699 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fbf000051c0 con 0x7fbf0815efe0 2024-06-29T15:58:40.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.229+0000 7fbeef7fe640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbef4076510 0x7fbef4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.229+0000 7fbeef7fe640 1 -- 172.21.15.59:0/1872519699 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7fbf000989c0 con 0x7fbf0815efe0 2024-06-29T15:58:40.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.229+0000 7fbf06d9d640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbef4076510 0x7fbef4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.230+0000 7fbf06d9d640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbef4076510 0x7fbef4078a20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbef8033000 tx=0x7fbef8004730 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.234+0000 7fbeef7fe640 1 -- 172.21.15.59:0/1872519699 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbf0009ae20 con 0x7fbf0815efe0 2024-06-29T15:58:40.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.453+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 --> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fbf08152b30 con 0x7fbef4076510 2024-06-29T15:58:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.455+0000 7fbeef7fe640 1 -- 172.21.15.59:0/1872519699 <== mgr.9710 v2:172.21.15.153:6832/1831136380 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+82 (secure 0 0 0) 0x7fbf08152b30 con 0x7fbef4076510 2024-06-29T15:58:40.456 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:58:40.456 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:40.456 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T15:58:40.456 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:58:40.456 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:58:40.456 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "backup_fs" 2024-06-29T15:58:40.456 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:58:40.456 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:58:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbef4076510 msgr2=0x7fbef4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbef4076510 0x7fbef4078a20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbef8033000 tx=0x7fbef8004730 comp rx=0 tx=0).stop 2024-06-29T15:58:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf0815efe0 msgr2=0x7fbf081e8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf0815efe0 0x7fbf081e8990 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000e960 tx=0x7fbf0000ee20 comp rx=0 tx=0).stop 2024-06-29T15:58:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 shutdown_connections 2024-06-29T15:58:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fbef4076510 0x7fbef4078a20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf0815efe0 0x7fbf081e8990 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbf080d23c0 0x7fbf081f5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 --2- 172.21.15.59:0/1872519699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf080b3ed0 0x7fbf08153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 >> 172.21.15.59:0/1872519699 conn(0x7fbf080d3ee0 msgr2=0x7fbf080d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 shutdown_connections 2024-06-29T15:58:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.457+0000 7fbf0d1e7640 1 -- 172.21.15.59:0/1872519699 wait complete. 2024-06-29T15:58:40.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:58:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 -- 172.21.15.59:0/1548958749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45480ce950 msgr2=0x7f45480d1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 --2- 172.21.15.59:0/1548958749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45480ce950 0x7f45480d1f60 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f45480b64f0 tx=0x7f454002aa30 comp rx=0 tx=0).stop 2024-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 -- 172.21.15.59:0/1548958749 shutdown_connections 2024-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 --2- 172.21.15.59:0/1548958749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45480ce950 0x7f45480d1f60 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 --2- 172.21.15.59:0/1548958749 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45480cf5c0 0x7f45480d1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 --2- 172.21.15.59:0/1548958749 >> v2:172.21.15.153:3300/0 conn(0x7f454808c190 0x7f45480cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 -- 172.21.15.59:0/1548958749 >> 172.21.15.59:0/1548958749 conn(0x7f45480b5610 msgr2=0x7f45480b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 -- 172.21.15.59:0/1548958749 shutdown_connections 2024-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.589+0000 7f454dba1640 1 -- 172.21.15.59:0/1548958749 wait complete. 2024-06-29T15:58:40.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.591+0000 7f454dba1640 1 Processor -- start 2024-06-29T15:58:40.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.591+0000 7f454dba1640 1 -- start start 2024-06-29T15:58:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.592+0000 7f454dba1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f454808c190 0x7f4548160ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.592+0000 7f454dba1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45480ce950 0x7f45480cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.592+0000 7f454dba1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45480cf5c0 0x7f4548157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.592+0000 7f454dba1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4548162440 con 0x7f454808c190 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.592+0000 7f454dba1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45481622c0 con 0x7f45480cf5c0 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.592+0000 7f454dba1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45481625c0 con 0x7f45480ce950 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f454808c190 0x7f4548160ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f454808c190 0x7f4548160ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50302/0 (socket says 172.21.15.59:50302) 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 -- 172.21.15.59:0/1775827229 learned_addr learned my addr 172.21.15.59:0/1775827229 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f4547fff640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45480cf5c0 0x7f4548157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f453effd640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45480ce950 0x7f45480cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 -- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45480ce950 msgr2=0x7f45480cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45480ce950 0x7f45480cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 -- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45480cf5c0 msgr2=0x7f4548157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45480cf5c0 0x7f4548157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 -- 172.21.15.59:0/1775827229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45480cabd0 con 0x7f454808c190 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f45477fe640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f454808c190 0x7f4548160ae0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f453800d6b0 tx=0x7f453800db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f4544ff9640 1 -- 172.21.15.59:0/1775827229 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538017070 con 0x7f454808c190 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f4544ff9640 1 -- 172.21.15.59:0/1775827229 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4538004310 con 0x7f454808c190 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45480cae70 con 0x7f454808c190 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f4544ff9640 1 -- 172.21.15.59:0/1775827229 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4538005280 con 0x7f454808c190 2024-06-29T15:58:40.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.593+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45480cb430 con 0x7f454808c190 2024-06-29T15:58:40.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.595+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45480d1c40 con 0x7f454808c190 2024-06-29T15:58:40.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.595+0000 7f4544ff9640 1 -- 172.21.15.59:0/1775827229 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4538005420 con 0x7f454808c190 2024-06-29T15:58:40.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.595+0000 7f4544ff9640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4528076540 0x7f4528078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:40.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.596+0000 7f4544ff9640 1 -- 172.21.15.59:0/1775827229 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f45380983e0 con 0x7f454808c190 2024-06-29T15:58:40.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.596+0000 7f453effd640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4528076540 0x7f4528078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:40.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.596+0000 7f453effd640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4528076540 0x7f4528078a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4530005fd0 tx=0x7f4530005850 comp rx=0 tx=0).ready entity=mgr.9710 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.600+0000 7f4544ff9640 1 -- 172.21.15.59:0/1775827229 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f45480d1c40 con 0x7f454808c190 2024-06-29T15:58:40.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:40.840+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f45480cf080 con 0x7f454808c190 2024-06-29T15:58:41.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.664+0000 7f4544ff9640 1 -- 172.21.15.59:0/1775827229 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v139) ==== 92+0+0 (secure 0 0 0) 0x7f4538020020 con 0x7f454808c190 2024-06-29T15:58:41.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.667+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4528076540 msgr2=0x7f4528078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:41.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.667+0000 7f454dba1640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4528076540 0x7f4528078a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4530005fd0 tx=0x7f4530005850 comp rx=0 tx=0).stop 2024-06-29T15:58:41.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.668+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f454808c190 msgr2=0x7f4548160ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:41.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.668+0000 7f454dba1640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f454808c190 0x7f4548160ae0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f453800d6b0 tx=0x7f453800db70 comp rx=0 tx=0).stop 2024-06-29T15:58:41.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.668+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 shutdown_connections 2024-06-29T15:58:41.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.668+0000 7f454dba1640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4528076540 0x7f4528078a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.668+0000 7f454dba1640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45480cf5c0 0x7f4548157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.668+0000 7f454dba1640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45480ce950 0x7f45480cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.668+0000 7f454dba1640 1 --2- 172.21.15.59:0/1775827229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f454808c190 0x7f4548160ae0 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.668+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 >> 172.21.15.59:0/1775827229 conn(0x7f45480b5610 msgr2=0x7f45480b89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:41.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.670+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 shutdown_connections 2024-06-29T15:58:41.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.670+0000 7f454dba1640 1 -- 172.21.15.59:0/1775827229 wait complete. 2024-06-29T15:58:41.686 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:58:41.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.817+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2348915375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2fa0 msgr2=0x7ff2d00d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:41.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.817+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2348915375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2fa0 0x7ff2d00d1520 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4005fd0 tx=0x7ff2c402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.817+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2348915375 shutdown_connections 2024-06-29T15:58:41.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.817+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2348915375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2fa0 0x7ff2d00d1520 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.817+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2348915375 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2d00b2270 0x7ff2d015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.817+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2348915375 >> 172.21.15.59:0/2348915375 conn(0x7ff2d0073e10 msgr2=0x7ff2d0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:41.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.817+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2348915375 shutdown_connections 2024-06-29T15:58:41.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.817+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2348915375 wait complete. 2024-06-29T15:58:41.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.819+0000 7ff2d7fc7640 1 Processor -- start 2024-06-29T15:58:41.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.819+0000 7ff2d7fc7640 1 -- start start 2024-06-29T15:58:41.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d7fc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2270 0x7ff2d006f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:41.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d7fc7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2d00b2fa0 0x7ff2d00af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:41.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d7fc7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2d00d1a60 0x7ff2d00b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:41.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d7fc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d00b6cc0 con 0x7ff2d00b2270 2024-06-29T15:58:41.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d7fc7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d00b6b40 con 0x7ff2d00d1a60 2024-06-29T15:58:41.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d7fc7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff2d00cfe90 con 0x7ff2d00b2fa0 2024-06-29T15:58:41.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2270 0x7ff2d006f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2270 0x7ff2d006f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50324/0 (socket says 172.21.15.59:50324) 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 -- 172.21.15.59:0/2533363009 learned_addr learned my addr 172.21.15.59:0/2533363009 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5562640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2d00b2fa0 0x7ff2d00af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d6564640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2d00d1a60 0x7ff2d00b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 -- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2d00b2fa0 msgr2=0x7ff2d00af5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2d00b2fa0 0x7ff2d00af5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 -- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2d00d1a60 msgr2=0x7ff2d00b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2d00d1a60 0x7ff2d00b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 -- 172.21.15.59:0/2533363009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d01f5050 con 0x7ff2d00b2270 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.821+0000 7ff2d5d63640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2270 0x7ff2d006f700 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800e7c0 tx=0x7ff2c800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.822+0000 7ff2ce7fc640 1 -- 172.21.15.59:0/2533363009 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800ccc0 con 0x7ff2d00b2270 2024-06-29T15:58:41.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.822+0000 7ff2ce7fc640 1 -- 172.21.15.59:0/2533363009 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2c8010430 con 0x7ff2d00b2270 2024-06-29T15:58:41.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.822+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d01f5330 con 0x7ff2d00b2270 2024-06-29T15:58:41.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.822+0000 7ff2ce7fc640 1 -- 172.21.15.59:0/2533363009 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8010860 con 0x7ff2d00b2270 2024-06-29T15:58:41.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.822+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d01f58f0 con 0x7ff2d00b2270 2024-06-29T15:58:41.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.823+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d00d0340 con 0x7ff2d00b2270 2024-06-29T15:58:41.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.824+0000 7ff2ce7fc640 1 -- 172.21.15.59:0/2533363009 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 139) ==== 98422+0+0 (secure 0 0 0) 0x7ff2c8010a00 con 0x7ff2d00b2270 2024-06-29T15:58:41.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.824+0000 7ff2ce7fc640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff2bc076590 0x7ff2bc078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:41.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.824+0000 7ff2ce7fc640 1 -- 172.21.15.59:0/2533363009 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7ff2c80992e0 con 0x7ff2d00b2270 2024-06-29T15:58:41.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.824+0000 7ff2d5562640 1 -- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff2bc076590 msgr2=0x7ff2bc078aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1831136380 2024-06-29T15:58:41.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.824+0000 7ff2d5562640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff2bc076590 0x7ff2bc078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:41.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:41.832+0000 7ff2ce7fc640 1 -- 172.21.15.59:0/2533363009 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff2c8061cb0 con 0x7ff2d00b2270 2024-06-29T15:58:42.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.026+0000 7ff2d5562640 1 -- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff2bc076590 msgr2=0x7ff2bc078aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1831136380 2024-06-29T15:58:42.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.026+0000 7ff2d5562640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff2bc076590 0x7ff2bc078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:42.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.026+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7ff2d00d1730 con 0x7ff2d00b2270 2024-06-29T15:58:42.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.030+0000 7ff2ce7fc640 1 -- 172.21.15.59:0/2533363009 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) ==== 138+0+0 (secure 0 0 0) 0x7ff2c8065fb0 con 0x7ff2d00b2270 2024-06-29T15:58:42.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff2bc076590 msgr2=0x7ff2bc078aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:42.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff2bc076590 0x7ff2bc078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2270 msgr2=0x7ff2d006f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:42.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2270 0x7ff2d006f700 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800e7c0 tx=0x7ff2c800ec80 comp rx=0 tx=0).stop 2024-06-29T15:58:42.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 shutdown_connections 2024-06-29T15:58:42.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7ff2bc076590 0x7ff2bc078aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2d00d1a60 0x7ff2d00b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2d00b2fa0 0x7ff2d00af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 --2- 172.21.15.59:0/2533363009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d00b2270 0x7ff2d006f700 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 >> 172.21.15.59:0/2533363009 conn(0x7ff2d0073e10 msgr2=0x7ff2d0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:42.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 shutdown_connections 2024-06-29T15:58:42.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.034+0000 7ff2d7fc7640 1 -- 172.21.15.59:0/2533363009 wait complete. 2024-06-29T15:58:42.051 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:58:42.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.171+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/2529015029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2270 msgr2=0x7f4f2815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:42.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.171+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/2529015029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2270 0x7f4f2815bc20 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c005fd0 tx=0x7f4f1c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.171+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/2529015029 shutdown_connections 2024-06-29T15:58:42.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.171+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/2529015029 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280d1c30 0x7f4f280b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.171+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/2529015029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2270 0x7f4f2815bc20 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.171+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/2529015029 >> 172.21.15.59:0/2529015029 conn(0x7f4f28073e10 msgr2=0x7f4f28074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:42.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.172+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/2529015029 shutdown_connections 2024-06-29T15:58:42.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.172+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/2529015029 wait complete. 2024-06-29T15:58:42.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.173+0000 7f4f2eeb3640 1 Processor -- start 2024-06-29T15:58:42.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.173+0000 7f4f2eeb3640 1 -- start start 2024-06-29T15:58:42.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f2eeb3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280b2270 0x7f4f28146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:42.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f2eeb3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2fa0 0x7f4f2815e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:42.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f2eeb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f280d1c30 0x7f4f2815e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:42.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f2eeb3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4f280b6400 con 0x7f4f280b2fa0 2024-06-29T15:58:42.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f2eeb3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f280b6280 con 0x7f4f280d1c30 2024-06-29T15:58:42.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f2eeb3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4f280b6580 con 0x7f4f280b2270 2024-06-29T15:58:42.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2fa0 0x7f4f2815e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2fa0 0x7f4f2815e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50338/0 (socket says 172.21.15.59:50338) 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f27fff640 1 -- 172.21.15.59:0/1147591745 learned_addr learned my addr 172.21.15.59:0/1147591745 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f2d450640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f280d1c30 0x7f4f2815e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f2cc4f640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280b2270 0x7f4f28146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f27fff640 1 -- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280b2270 msgr2=0x7f4f28146e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f27fff640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280b2270 0x7f4f28146e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f27fff640 1 -- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f280d1c30 msgr2=0x7f4f2815e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f27fff640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f280d1c30 0x7f4f2815e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.175+0000 7f4f27fff640 1 -- 172.21.15.59:0/1147591745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2806ea80 con 0x7f4f280b2fa0 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.176+0000 7f4f27fff640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2fa0 0x7f4f2815e170 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400c370 tx=0x7f4f1400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.176+0000 7f4f257fa640 1 -- 172.21.15.59:0/1147591745 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f140052d0 con 0x7f4f280b2fa0 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.176+0000 7f4f257fa640 1 -- 172.21.15.59:0/1147591745 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f14007020 con 0x7f4f280b2fa0 2024-06-29T15:58:42.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.176+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f2806ed60 con 0x7f4f280b2fa0 2024-06-29T15:58:42.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.176+0000 7f4f257fa640 1 -- 172.21.15.59:0/1147591745 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f14007470 con 0x7f4f280b2fa0 2024-06-29T15:58:42.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.176+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f2806f320 con 0x7f4f280b2fa0 2024-06-29T15:58:42.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.177+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f2806e7f0 con 0x7f4f280b2fa0 2024-06-29T15:58:42.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.178+0000 7f4f257fa640 1 -- 172.21.15.59:0/1147591745 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 139) ==== 98422+0+0 (secure 0 0 0) 0x7f4f140067d0 con 0x7f4f280b2fa0 2024-06-29T15:58:42.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.178+0000 7f4f257fa640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f0c076560 0x7f4f0c078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:42.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.179+0000 7f4f2cc4f640 1 -- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f0c076560 msgr2=0x7f4f0c078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1831136380 2024-06-29T15:58:42.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.179+0000 7f4f257fa640 1 -- 172.21.15.59:0/1147591745 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7f4f14097450 con 0x7f4f280b2fa0 2024-06-29T15:58:42.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.179+0000 7f4f2cc4f640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f0c076560 0x7f4f0c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:42.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.183+0000 7f4f257fa640 1 -- 172.21.15.59:0/1147591745 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f14060040 con 0x7f4f280b2fa0 2024-06-29T15:58:42.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.376+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f4f28069ca0 con 0x7f4f280b2fa0 2024-06-29T15:58:42.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.379+0000 7f4f257fa640 1 -- 172.21.15.59:0/1147591745 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) ==== 133+0+0 (secure 0 0 0) 0x7f4f28069ca0 con 0x7f4f280b2fa0 2024-06-29T15:58:42.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.380+0000 7f4f2cc4f640 1 -- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f0c076560 msgr2=0x7f4f0c078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1831136380 2024-06-29T15:58:42.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.380+0000 7f4f2cc4f640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f0c076560 0x7f4f0c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:42.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.381+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f0c076560 msgr2=0x7f4f0c078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:42.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.381+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f0c076560 0x7f4f0c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.381+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2fa0 msgr2=0x7f4f2815e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:42.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.381+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2fa0 0x7f4f2815e170 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400c370 tx=0x7f4f1400c830 comp rx=0 tx=0).stop 2024-06-29T15:58:42.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.382+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 shutdown_connections 2024-06-29T15:58:42.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.382+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7f4f0c076560 0x7f4f0c078a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.382+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f280d1c30 0x7f4f2815e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.382+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f280b2fa0 0x7f4f2815e170 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.382+0000 7f4f2eeb3640 1 --2- 172.21.15.59:0/1147591745 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f280b2270 0x7f4f28146e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.382+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 >> 172.21.15.59:0/1147591745 conn(0x7f4f28073e10 msgr2=0x7f4f280b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:42.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.382+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 shutdown_connections 2024-06-29T15:58:42.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.382+0000 7f4f2eeb3640 1 -- 172.21.15.59:0/1147591745 wait complete. 2024-06-29T15:58:42.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:42.395 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-06-29T15:58:42.414 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-06-29T15:58:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.576+0000 7fa3b2743640 1 -- 172.21.15.59:0/3518687973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3ac0d23c0 msgr2=0x7fa3ac0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.576+0000 7fa3b2743640 1 --2- 172.21.15.59:0/3518687973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3ac0d23c0 0x7fa3ac0b3990 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fa39c005fd0 tx=0x7fa39c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.576+0000 7fa3b2743640 1 -- 172.21.15.59:0/3518687973 shutdown_connections 2024-06-29T15:58:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.576+0000 7fa3b2743640 1 --2- 172.21.15.59:0/3518687973 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3ac0b3ed0 0x7fa3ac15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.576+0000 7fa3b2743640 1 --2- 172.21.15.59:0/3518687973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3ac0d23c0 0x7fa3ac0b3990 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.576+0000 7fa3b2743640 1 -- 172.21.15.59:0/3518687973 >> 172.21.15.59:0/3518687973 conn(0x7fa3ac0d3ee0 msgr2=0x7fa3ac0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.576+0000 7fa3b2743640 1 -- 172.21.15.59:0/3518687973 shutdown_connections 2024-06-29T15:58:42.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.576+0000 7fa3b2743640 1 -- 172.21.15.59:0/3518687973 wait complete. 2024-06-29T15:58:42.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.578+0000 7fa3b2743640 1 Processor -- start 2024-06-29T15:58:42.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.578+0000 7fa3b2743640 1 -- start start 2024-06-29T15:58:42.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3b2743640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3ac0b3ed0 0x7fa3ac0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:42.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3b2743640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3ac0d23c0 0x7fa3ac1e8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:42.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3b2743640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3ac15ee10 0x7fa3ac0b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:42.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3b2743640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ac160240 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3b2743640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ac1600c0 con 0x7fa3ac0b3ed0 2024-06-29T15:58:42.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3b2743640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ac1603c0 con 0x7fa3ac15ee10 2024-06-29T15:58:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3ab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3ac0d23c0 0x7fa3ac1e8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3ab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3ac0d23c0 0x7fa3ac1e8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50352/0 (socket says 172.21.15.59:50352) 2024-06-29T15:58:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.579+0000 7fa3ab7fe640 1 -- 172.21.15.59:0/2070128853 learned_addr learned my addr 172.21.15.59:0/2070128853 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3abfff640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3ac0b3ed0 0x7fa3ac0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3b0ce0640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3ac15ee10 0x7fa3ac0b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3ab7fe640 1 -- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3ac15ee10 msgr2=0x7fa3ac0b8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3ab7fe640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa3ac15ee10 0x7fa3ac0b8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3ab7fe640 1 -- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3ac0b3ed0 msgr2=0x7fa3ac0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3ab7fe640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3ac0b3ed0 0x7fa3ac0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3ab7fe640 1 -- 172.21.15.59:0/2070128853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ac1e9610 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3ab7fe640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3ac0d23c0 0x7fa3ac1e8b50 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000e790 tx=0x7fa3a000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000eea0 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3a000cd50 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3b2743640 1 -- 172.21.15.59:0/2070128853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ac159100 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0010690 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.580+0000 7fa3b2743640 1 -- 172.21.15.59:0/2070128853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3ac1e9090 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.581+0000 7fa3b2743640 1 -- 172.21.15.59:0/2070128853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3ac15e8d0 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.582+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 139) ==== 98422+0+0 (secure 0 0 0) 0x7fa3a0010830 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.583+0000 7fa3a8ff9640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 0x7fa398078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.583+0000 7fa3abfff640 1 -- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 msgr2=0x7fa398078aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1831136380 2024-06-29T15:58:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.583+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 7021+0+0 (secure 0 0 0) 0x7fa3a00993c0 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.583+0000 7fa3abfff640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 0x7fa398078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:42.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.591+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa3a0061cb0 con 0x7fa3ac0d23c0 2024-06-29T15:58:42.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.784+0000 7fa3abfff640 1 -- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 msgr2=0x7fa398078aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1831136380 2024-06-29T15:58:42.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.784+0000 7fa3abfff640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 0x7fa398078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:42.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:42.810+0000 7fa3b2743640 1 -- 172.21.15.59:0/2070128853 --> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7fa3ac1e9360 con 0x7fa398076590 2024-06-29T15:58:43.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:43.185+0000 7fa3abfff640 1 -- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 msgr2=0x7fa398078aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1831136380 2024-06-29T15:58:43.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:43.185+0000 7fa3abfff640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 0x7fa398078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:58:43.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:58:43.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:58:43.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:43.986+0000 7fa3abfff640 1 -- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 msgr2=0x7fa398078aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1831136380 2024-06-29T15:58:43.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:43.986+0000 7fa3abfff640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 0x7fa398078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:58:45.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:45.325+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 140) ==== 97929+0+0 (secure 0 0 0) 0x7fa3a005df40 con 0x7fa3ac0d23c0 2024-06-29T15:58:45.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:45.325+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 msgr2=0x7fa398078aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:45.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:45.325+0000 7fa3a8ff9640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/1831136380,v1:172.21.15.153:6833/1831136380] conn(0x7fa398076590 0x7fa398078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.561 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.560+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.560+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.560+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.560+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.560+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.561+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.561+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.561+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.561+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.562 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.561+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.573 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.573 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.573 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.573 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.574 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f881e810640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.574 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.574 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.574 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.574 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:45.574 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:45.572+0000 7f8821816640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:46.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.328+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 141) ==== 98056+0+0 (secure 0 0 0) 0x7fa3a0060f10 con 0x7fa3ac0d23c0 2024-06-29T15:58:46.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.328+0000 7fa3a8ff9640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa39803b020 0x7fa398083200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.328+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 --> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7fa3a000e1a0 con 0x7fa39803b020 2024-06-29T15:58:46.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.330+0000 7fa3abfff640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa39803b020 0x7fa398083200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.334+0000 7fa3abfff640 1 --2- 172.21.15.59:0/2070128853 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa39803b020 0x7fa398083200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa39c033000 tx=0x7fa39c004730 comp rx=0 tx=0).ready entity=mgr.11704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:46.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.336+0000 7fa3a8ff9640 1 -- 172.21.15.59:0/2070128853 <== mgr.11704 v2:172.21.15.153:6832/2715634623 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) ==== 79+0+0 (secure 0 0 0) 0x7fa3a000e1a0 con 0x7fa39803b020 2024-06-29T15:58:46.337 INFO:teuthology.orchestra.run.smithi059.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-06-29T15:58:46.337 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:46.336+0000 7f8828824640 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-06-29T15:58:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-29T15:58:46.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:46.345 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-06-29T15:58:46.365 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:58:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.533+0000 7f0c51e22640 1 -- 172.21.15.59:0/1669403481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c34003a70 msgr2=0x7f0c34003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.533+0000 7f0c51e22640 1 --2- 172.21.15.59:0/1669403481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c34003a70 0x7f0c34003ee0 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009d00 tx=0x7f0c34005830 comp rx=0 tx=0).stop 2024-06-29T15:58:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.533+0000 7f0c4bfff640 1 -- 172.21.15.59:0/1669403481 reap_dead start 2024-06-29T15:58:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.533+0000 7f0c51e22640 1 -- 172.21.15.59:0/1669403481 shutdown_connections 2024-06-29T15:58:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.533+0000 7f0c51e22640 1 -- 172.21.15.59:0/1669403481 >> 172.21.15.59:0/1669403481 conn(0x7f0c4c0b6fb0 msgr2=0x7f0c4c0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.533+0000 7f0c51e22640 1 -- 172.21.15.59:0/1669403481 shutdown_connections 2024-06-29T15:58:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.533+0000 7f0c51e22640 1 -- 172.21.15.59:0/1669403481 wait complete. 2024-06-29T15:58:46.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.535+0000 7f0c51e22640 1 Processor -- start 2024-06-29T15:58:46.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.535+0000 7f0c51e22640 1 -- start start 2024-06-29T15:58:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c51e22640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c34003a70 0x7f0c4c1f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c51e22640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c4c154970 0x7f0c4c1577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c51e22640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c4c15a920 0x7f0c4c158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c51e22640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4c13ef40 con 0x7f0c4c15a920 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c51e22640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4c13edc0 con 0x7f0c4c154970 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c51e22640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4c13f0c0 con 0x7f0c34003a70 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c4bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c4c15a920 0x7f0c4c158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c4bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c4c15a920 0x7f0c4c158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52046/0 (socket says 172.21.15.59:52046) 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c4bfff640 1 -- 172.21.15.59:0/52139629 learned_addr learned my addr 172.21.15.59:0/52139629 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c4b7fe640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c34003a70 0x7f0c4c1f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.536+0000 7f0c4affd640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c4c154970 0x7f0c4c1577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c4bfff640 1 -- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c34003a70 msgr2=0x7f0c4c1f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c4bfff640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c34003a70 0x7f0c4c1f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c4bfff640 1 -- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c4c154970 msgr2=0x7f0c4c1577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c4bfff640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c4c154970 0x7f0c4c1577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c4bfff640 1 -- 172.21.15.59:0/52139629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4c1f35c0 con 0x7f0c4c15a920 2024-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c4b7fe640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c34003a70 0x7f0c4c1f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c4affd640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c4c154970 0x7f0c4c1577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c4bfff640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c4c15a920 0x7f0c4c158460 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c13f300 tx=0x7f0c4400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c33fff640 1 -- 172.21.15.59:0/52139629 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c44004d20 con 0x7f0c4c15a920 2024-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c33fff640 1 -- 172.21.15.59:0/52139629 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c4400fe50 con 0x7f0c4c15a920 2024-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c4c156680 con 0x7f0c4c15a920 2024-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c33fff640 1 -- 172.21.15.59:0/52139629 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c44003bd0 con 0x7f0c4c15a920 2024-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.537+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c4c156fb0 con 0x7f0c4c15a920 2024-06-29T15:58:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.538+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c4c0b56d0 con 0x7f0c4c15a920 2024-06-29T15:58:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.539+0000 7f0c33fff640 1 -- 172.21.15.59:0/52139629 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 141) ==== 98056+0+0 (secure 0 0 0) 0x7f0c44004030 con 0x7f0c4c15a920 2024-06-29T15:58:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.540+0000 7f0c33fff640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f0c380761a0 0x7f0c380786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.540+0000 7f0c33fff640 1 -- 172.21.15.59:0/52139629 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7f0c44098e80 con 0x7f0c4c15a920 2024-06-29T15:58:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.540+0000 7f0c4b7fe640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f0c380761a0 0x7f0c380786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.541+0000 7f0c4b7fe640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f0c380761a0 0x7f0c380786b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c035000 tx=0x7f0c3c0046c0 comp rx=0 tx=0).ready entity=mgr.11704 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:46.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.545+0000 7f0c33fff640 1 -- 172.21.15.59:0/52139629 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0c4401f060 con 0x7f0c4c15a920 2024-06-29T15:58:46.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:46.786+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f0c4c13e770 con 0x7f0c4c15a920 2024-06-29T15:58:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.372+0000 7f0c33fff640 1 -- 172.21.15.59:0/52139629 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v142) ==== 93+0+0 (secure 0 0 0) 0x7f0c4c1f35c0 con 0x7f0c4c15a920 2024-06-29T15:58:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.375+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f0c380761a0 msgr2=0x7f0c380786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.375+0000 7f0c51e22640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f0c380761a0 0x7f0c380786b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c035000 tx=0x7f0c3c0046c0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.375+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c4c15a920 msgr2=0x7f0c4c158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.375+0000 7f0c51e22640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c4c15a920 0x7f0c4c158460 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c13f300 tx=0x7f0c4400f790 comp rx=0 tx=0).stop 2024-06-29T15:58:47.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.376+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 shutdown_connections 2024-06-29T15:58:47.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.376+0000 7f0c51e22640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f0c380761a0 0x7f0c380786b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.376+0000 7f0c51e22640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c4c15a920 0x7f0c4c158460 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.376+0000 7f0c51e22640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c4c154970 0x7f0c4c1577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.376+0000 7f0c51e22640 1 --2- 172.21.15.59:0/52139629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c34003a70 0x7f0c4c1f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.376+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 >> 172.21.15.59:0/52139629 conn(0x7f0c4c0b6fb0 msgr2=0x7f0c4c13dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.377+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 shutdown_connections 2024-06-29T15:58:47.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.377+0000 7f0c51e22640 1 -- 172.21.15.59:0/52139629 wait complete. 2024-06-29T15:58:47.395 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:58:47.395 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:58:47.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:58:47.484 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:58:47.484 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:58:47.484 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:58:47.484 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:47.501 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:58:47.501 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:47.557 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.716+0000 7fe39b43c640 1 -- 172.21.15.59:0/3240487717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39406da80 msgr2=0x7fe3940cd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.716+0000 7fe39b43c640 1 --2- 172.21.15.59:0/3240487717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39406da80 0x7fe3940cd990 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fe38c005fd0 tx=0x7fe38c02a840 comp rx=0 tx=0).stop 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.716+0000 7fe39b43c640 1 -- 172.21.15.59:0/3240487717 shutdown_connections 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.716+0000 7fe39b43c640 1 --2- 172.21.15.59:0/3240487717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39406da80 0x7fe3940cd990 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.716+0000 7fe39b43c640 1 -- 172.21.15.59:0/3240487717 >> 172.21.15.59:0/3240487717 conn(0x7fe3940b2ab0 msgr2=0x7fe3940b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.716+0000 7fe39b43c640 1 -- 172.21.15.59:0/3240487717 shutdown_connections 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.716+0000 7fe39b43c640 1 -- 172.21.15.59:0/3240487717 wait complete. 2024-06-29T15:58:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.717+0000 7fe39b43c640 1 Processor -- start 2024-06-29T15:58:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.718+0000 7fe39b43c640 1 -- start start 2024-06-29T15:58:47.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe39b43c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe39406da80 0x7fe39406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe39b43c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3941448e0 0x7fe39406cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe39b43c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3941439d0 0x7fe3940b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe39b43c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe394166f20 con 0x7fe3941448e0 2024-06-29T15:58:47.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe39b43c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe394166da0 con 0x7fe39406da80 2024-06-29T15:58:47.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe39b43c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe3941670a0 con 0x7fe3941439d0 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3989d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3941448e0 0x7fe39406cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3991d8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe39406da80 0x7fe39406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3991d8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe39406da80 0x7fe39406ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:35710/0 (socket says 172.21.15.59:35710) 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3991d8640 1 -- 172.21.15.59:0/4281475981 learned_addr learned my addr 172.21.15.59:0/4281475981 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3999d9640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3941439d0 0x7fe3940b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3989d7640 1 -- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3941439d0 msgr2=0x7fe3940b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3989d7640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3941439d0 0x7fe3940b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3989d7640 1 -- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe39406da80 msgr2=0x7fe39406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3989d7640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe39406da80 0x7fe39406ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.719+0000 7fe3989d7640 1 -- 172.21.15.59:0/4281475981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3940b4070 con 0x7fe3941448e0 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.720+0000 7fe3991d8640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe39406da80 0x7fe39406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.720+0000 7fe3989d7640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3941448e0 0x7fe39406cfb0 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fe38800c3f0 tx=0x7fe38800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:47.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.720+0000 7fe385ffb640 1 -- 172.21.15.59:0/4281475981 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe388017070 con 0x7fe3941448e0 2024-06-29T15:58:47.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.720+0000 7fe385ffb640 1 -- 172.21.15.59:0/4281475981 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3880053c0 con 0x7fe3941448e0 2024-06-29T15:58:47.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.720+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3940b4350 con 0x7fe3941448e0 2024-06-29T15:58:47.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.720+0000 7fe385ffb640 1 -- 172.21.15.59:0/4281475981 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe388003850 con 0x7fe3941448e0 2024-06-29T15:58:47.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.720+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3940b4910 con 0x7fe3941448e0 2024-06-29T15:58:47.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.721+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3940b3de0 con 0x7fe3941448e0 2024-06-29T15:58:47.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.722+0000 7fe385ffb640 1 -- 172.21.15.59:0/4281475981 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7fe38801a070 con 0x7fe3941448e0 2024-06-29T15:58:47.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.723+0000 7fe385ffb640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fe378076510 0x7fe378078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.723+0000 7fe3991d8640 1 -- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fe378076510 msgr2=0x7fe378078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:47.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.723+0000 7fe385ffb640 1 -- 172.21.15.59:0/4281475981 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7fe3880974d0 con 0x7fe3941448e0 2024-06-29T15:58:47.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.723+0000 7fe3991d8640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fe378076510 0x7fe378078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:47.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.727+0000 7fe385ffb640 1 -- 172.21.15.59:0/4281475981 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe38805fde0 con 0x7fe3941448e0 2024-06-29T15:58:47.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.923+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe39406f750 con 0x7fe3941448e0 2024-06-29T15:58:47.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.924+0000 7fe3991d8640 1 -- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fe378076510 msgr2=0x7fe378078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:47.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.924+0000 7fe3991d8640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fe378076510 0x7fe378078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:47.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.925+0000 7fe385ffb640 1 -- 172.21.15.59:0/4281475981 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) ==== 74+0+21414 (secure 0 0 0) 0x7fe39406f750 con 0x7fe3941448e0 2024-06-29T15:58:47.926 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:47.926 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":247,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:45.316883+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/877026286":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3972699112":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1636302558":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1944500160":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3355566242":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6832/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/2494131047":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1461412440":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1259499350":"2024-06-30T15:58:45.316819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fe378076510 msgr2=0x7fe378078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fe378076510 0x7fe378078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3941448e0 msgr2=0x7fe39406cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3941448e0 0x7fe39406cfb0 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fe38800c3f0 tx=0x7fe38800c8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 shutdown_connections 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fe378076510 0x7fe378078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe3941439d0 0x7fe3940b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3941448e0 0x7fe39406cfb0 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 --2- 172.21.15.59:0/4281475981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe39406da80 0x7fe39406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 >> 172.21.15.59:0/4281475981 conn(0x7fe3940b2ab0 msgr2=0x7fe39415fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 shutdown_connections 2024-06-29T15:58:47.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:47.927+0000 7fe39b43c640 1 -- 172.21.15.59:0/4281475981 wait complete. 2024-06-29T15:58:47.940 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:58:19.303469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '241', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:58:19.884648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:58:31.630427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:58:32.972384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:47.941 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.062+0000 7f36ccf7f640 1 -- 172.21.15.59:0/1508012866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80d23c0 msgr2=0x7f36c80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.062+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/1508012866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80d23c0 0x7f36c80b3990 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009830 tx=0x7f36b802a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.063+0000 7f36ccf7f640 1 -- 172.21.15.59:0/1508012866 shutdown_connections 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.063+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/1508012866 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36c815efe0 0x7f36c815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.063+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/1508012866 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80d23c0 0x7f36c80b3990 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.063+0000 7f36ccf7f640 1 -- 172.21.15.59:0/1508012866 >> 172.21.15.59:0/1508012866 conn(0x7f36c80d3ee0 msgr2=0x7f36c80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.063+0000 7f36ccf7f640 1 -- 172.21.15.59:0/1508012866 shutdown_connections 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.063+0000 7f36ccf7f640 1 -- 172.21.15.59:0/1508012866 wait complete. 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.065+0000 7f36ccf7f640 1 Processor -- start 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.065+0000 7f36ccf7f640 1 -- start start 2024-06-29T15:58:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36ccf7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80b3ed0 0x7f36c81e79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36ccf7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36c80d23c0 0x7f36c80c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36ccf7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815efe0 0x7f36c81e94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36ccf7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f36c8160280 con 0x7f36c815efe0 2024-06-29T15:58:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36ccf7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f36c8160100 con 0x7f36c80b3ed0 2024-06-29T15:58:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36ccf7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f36c8160400 con 0x7f36c80d23c0 2024-06-29T15:58:48.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36c6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815efe0 0x7f36c81e94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36c659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80b3ed0 0x7f36c81e79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36c6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815efe0 0x7f36c81e94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52094/0 (socket says 172.21.15.59:52094) 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36c5d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36c80d23c0 0x7f36c80c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36c6d9d640 1 -- 172.21.15.59:0/475387391 learned_addr learned my addr 172.21.15.59:0/475387391 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.066+0000 7f36c6d9d640 1 -- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36c80d23c0 msgr2=0x7f36c80c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36c6d9d640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36c80d23c0 0x7f36c80c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36c6d9d640 1 -- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80b3ed0 msgr2=0x7f36c81e79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36c6d9d640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80b3ed0 0x7f36c81e79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36c6d9d640 1 -- 172.21.15.59:0/475387391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c81e8420 con 0x7f36c815efe0 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36c659c640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80b3ed0 0x7f36c81e79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36c6d9d640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815efe0 0x7f36c81e94c0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f36c000ec20 tx=0x7f36c000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36b6ffd640 1 -- 172.21.15.59:0/475387391 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c0003b70 con 0x7f36c815efe0 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36b6ffd640 1 -- 172.21.15.59:0/475387391 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36c0003d10 con 0x7f36c815efe0 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c81e8700 con 0x7f36c815efe0 2024-06-29T15:58:48.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36b6ffd640 1 -- 172.21.15.59:0/475387391 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c0004280 con 0x7f36c815efe0 2024-06-29T15:58:48.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.067+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c81e8cc0 con 0x7f36c815efe0 2024-06-29T15:58:48.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.068+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36c81e8190 con 0x7f36c815efe0 2024-06-29T15:58:48.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.069+0000 7f36b6ffd640 1 -- 172.21.15.59:0/475387391 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7f36c0004420 con 0x7f36c815efe0 2024-06-29T15:58:48.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.069+0000 7f36b6ffd640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f36a8076510 0x7f36a8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.070+0000 7f36c659c640 1 -- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f36a8076510 msgr2=0x7f36a8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:48.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.070+0000 7f36b6ffd640 1 -- 172.21.15.59:0/475387391 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7f36c0098ec0 con 0x7f36c815efe0 2024-06-29T15:58:48.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.070+0000 7f36c659c640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f36a8076510 0x7f36a8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:48.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.078+0000 7f36b6ffd640 1 -- 172.21.15.59:0/475387391 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f36c0061610 con 0x7f36c815efe0 2024-06-29T15:58:48.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.269+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36c80d2950 con 0x7f36c815efe0 2024-06-29T15:58:48.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.269+0000 7f36b6ffd640 1 -- 172.21.15.59:0/475387391 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v247) ==== 93+0+10 (secure 0 0 0) 0x7f36c0065910 con 0x7f36c815efe0 2024-06-29T15:58:48.270 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:58:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.271+0000 7f36c659c640 1 -- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f36a8076510 msgr2=0x7f36a8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.271+0000 7f36c659c640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f36a8076510 0x7f36a8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f36a8076510 msgr2=0x7f36a8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f36a8076510 0x7f36a8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815efe0 msgr2=0x7f36c81e94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815efe0 0x7f36c81e94c0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f36c000ec20 tx=0x7f36c000c6a0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 shutdown_connections 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f36a8076510 0x7f36a8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815efe0 0x7f36c81e94c0 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36c80d23c0 0x7f36c80c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 --2- 172.21.15.59:0/475387391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c80b3ed0 0x7f36c81e79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 >> 172.21.15.59:0/475387391 conn(0x7f36c80d3ee0 msgr2=0x7f36c80ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 shutdown_connections 2024-06-29T15:58:48.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.272+0000 7f36ccf7f640 1 -- 172.21.15.59:0/475387391 wait complete. 2024-06-29T15:58:48.284 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:58:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.409+0000 7f3de8aac640 1 -- 172.21.15.59:0/1608999269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 msgr2=0x7f3de40d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.409+0000 7f3de8aac640 1 --2- 172.21.15.59:0/1608999269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 0x7f3de40d1520 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc005fd0 tx=0x7f3dcc02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.409+0000 7f3de8aac640 1 -- 172.21.15.59:0/1608999269 shutdown_connections 2024-06-29T15:58:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.409+0000 7f3de8aac640 1 --2- 172.21.15.59:0/1608999269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 0x7f3de40d1520 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.409+0000 7f3de8aac640 1 --2- 172.21.15.59:0/1608999269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de40b2270 0x7f3de415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.409+0000 7f3de8aac640 1 -- 172.21.15.59:0/1608999269 >> 172.21.15.59:0/1608999269 conn(0x7f3de4073e10 msgr2=0x7f3de4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.409+0000 7f3de8aac640 1 -- 172.21.15.59:0/1608999269 shutdown_connections 2024-06-29T15:58:48.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.409+0000 7f3de8aac640 1 -- 172.21.15.59:0/1608999269 wait complete. 2024-06-29T15:58:48.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.411+0000 7f3de8aac640 1 Processor -- start 2024-06-29T15:58:48.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.411+0000 7f3de8aac640 1 -- start start 2024-06-29T15:58:48.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de8aac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40b2270 0x7f3de406db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de8aac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 0x7f3de4149390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de8aac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de40d1a60 0x7f3de406f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de8aac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3de40b6cc0 con 0x7f3de40b2fa0 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de8aac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3de40b6b40 con 0x7f3de40b2270 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de8aac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3de40cfe90 con 0x7f3de40d1a60 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 0x7f3de4149390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 0x7f3de4149390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52114/0 (socket says 172.21.15.59:52114) 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de1d9b640 1 -- 172.21.15.59:0/3190824477 learned_addr learned my addr 172.21.15.59:0/3190824477 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de2d9d640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de40d1a60 0x7f3de406f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.412+0000 7f3de259c640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40b2270 0x7f3de406db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3de1d9b640 1 -- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de40d1a60 msgr2=0x7f3de406f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3de1d9b640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de40d1a60 0x7f3de406f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3de1d9b640 1 -- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40b2270 msgr2=0x7f3de406db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3de1d9b640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40b2270 0x7f3de406db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3de1d9b640 1 -- 172.21.15.59:0/3190824477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de406e3d0 con 0x7f3de40b2fa0 2024-06-29T15:58:48.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3de1d9b640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 0x7f3de4149390 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc000c00 tx=0x7f3dcc0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:48.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3ddaffd640 1 -- 172.21.15.59:0/3190824477 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc03e030 con 0x7f3de40b2fa0 2024-06-29T15:58:48.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3ddaffd640 1 -- 172.21.15.59:0/3190824477 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dcc003ae0 con 0x7f3de40b2fa0 2024-06-29T15:58:48.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de406e6b0 con 0x7f3de40b2fa0 2024-06-29T15:58:48.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3ddaffd640 1 -- 172.21.15.59:0/3190824477 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc035040 con 0x7f3de40b2fa0 2024-06-29T15:58:48.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.413+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de406ec70 con 0x7f3de40b2fa0 2024-06-29T15:58:48.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.414+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de40d0340 con 0x7f3de40b2fa0 2024-06-29T15:58:48.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.415+0000 7f3ddaffd640 1 -- 172.21.15.59:0/3190824477 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7f3dcc041030 con 0x7f3de40b2fa0 2024-06-29T15:58:48.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.416+0000 7f3ddaffd640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f3dc4076510 0x7f3dc4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.416+0000 7f3de259c640 1 -- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f3dc4076510 msgr2=0x7f3dc4078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:48.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.416+0000 7f3ddaffd640 1 -- 172.21.15.59:0/3190824477 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7f3dcc0b8710 con 0x7f3de40b2fa0 2024-06-29T15:58:48.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.416+0000 7f3de259c640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f3dc4076510 0x7f3dc4078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:48.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.420+0000 7f3ddaffd640 1 -- 172.21.15.59:0/3190824477 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3dcc080f30 con 0x7f3de40b2fa0 2024-06-29T15:58:48.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.612+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3de4155ae0 con 0x7f3de40b2fa0 2024-06-29T15:58:48.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.612+0000 7f3ddaffd640 1 -- 172.21.15.59:0/3190824477 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v247) ==== 104+0+11 (secure 0 0 0) 0x7f3dcc085230 con 0x7f3de40b2fa0 2024-06-29T15:58:48.613 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f3dc4076510 msgr2=0x7f3dc4078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f3dc4076510 0x7f3dc4078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 msgr2=0x7f3de4149390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 0x7f3de4149390 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc000c00 tx=0x7f3dcc0035f0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 shutdown_connections 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f3dc4076510 0x7f3dc4078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3de40d1a60 0x7f3de406f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3de40b2fa0 0x7f3de4149390 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 --2- 172.21.15.59:0/3190824477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3de40b2270 0x7f3de406db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.614+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 >> 172.21.15.59:0/3190824477 conn(0x7f3de4073e10 msgr2=0x7f3de415ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.615+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 shutdown_connections 2024-06-29T15:58:48.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.615+0000 7f3de8aac640 1 -- 172.21.15.59:0/3190824477 wait complete. 2024-06-29T15:58:48.628 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:58:48.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.749+0000 7fa43197a640 1 -- 172.21.15.59:0/1168567393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0d32b0 msgr2=0x7fa42c0b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.749+0000 7fa43197a640 1 --2- 172.21.15.59:0/1168567393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0d32b0 0x7fa42c0b2bd0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa424005fd0 tx=0x7fa42402a940 comp rx=0 tx=0).stop 2024-06-29T15:58:48.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.749+0000 7fa43197a640 1 -- 172.21.15.59:0/1168567393 shutdown_connections 2024-06-29T15:58:48.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.749+0000 7fa43197a640 1 --2- 172.21.15.59:0/1168567393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c0d32b0 0x7fa42c0b2bd0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.749+0000 7fa43197a640 1 -- 172.21.15.59:0/1168567393 >> 172.21.15.59:0/1168567393 conn(0x7fa42c0d3ec0 msgr2=0x7fa42c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.749+0000 7fa43197a640 1 -- 172.21.15.59:0/1168567393 shutdown_connections 2024-06-29T15:58:48.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.749+0000 7fa43197a640 1 -- 172.21.15.59:0/1168567393 wait complete. 2024-06-29T15:58:48.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.751+0000 7fa43197a640 1 Processor -- start 2024-06-29T15:58:48.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.751+0000 7fa43197a640 1 -- start start 2024-06-29T15:58:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa43197a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0b27c0 0x7fa42c1537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa43197a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c0d32b0 0x7fa42c15c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa43197a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c153fa0 0x7fa42c15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa43197a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c1694c0 con 0x7fa42c0b27c0 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa43197a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c169340 con 0x7fa42c153fa0 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa43197a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa42c169640 con 0x7fa42c0d32b0 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa42affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0b27c0 0x7fa42c1537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa42affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0b27c0 0x7fa42c1537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52118/0 (socket says 172.21.15.59:52118) 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa42affd640 1 -- 172.21.15.59:0/3225407119 learned_addr learned my addr 172.21.15.59:0/3225407119 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa42b7fe640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c153fa0 0x7fa42c15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.752+0000 7fa423fff640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c0d32b0 0x7fa42c15c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa42affd640 1 -- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c0d32b0 msgr2=0x7fa42c15c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa42affd640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c0d32b0 0x7fa42c15c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa42affd640 1 -- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c153fa0 msgr2=0x7fa42c15ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa42affd640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c153fa0 0x7fa42c15ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa42affd640 1 -- 172.21.15.59:0/3225407119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c1618f0 con 0x7fa42c0b27c0 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa42affd640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0b27c0 0x7fa42c1537a0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00c3f0 tx=0x7fa41c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa428ff9640 1 -- 172.21.15.59:0/3225407119 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c017070 con 0x7fa42c0b27c0 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa428ff9640 1 -- 172.21.15.59:0/3225407119 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa41c0035f0 con 0x7fa42c0b27c0 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa428ff9640 1 -- 172.21.15.59:0/3225407119 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c003a40 con 0x7fa42c0b27c0 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c158b10 con 0x7fa42c0b27c0 2024-06-29T15:58:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.753+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c1590d0 con 0x7fa42c0b27c0 2024-06-29T15:58:48.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.754+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c0b2bd0 con 0x7fa42c0b27c0 2024-06-29T15:58:48.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.755+0000 7fa428ff9640 1 -- 172.21.15.59:0/3225407119 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7fa41c01a070 con 0x7fa42c0b27c0 2024-06-29T15:58:48.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.756+0000 7fa428ff9640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa40c076540 0x7fa40c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.756+0000 7fa423fff640 1 -- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa40c076540 msgr2=0x7fa40c078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:48.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.756+0000 7fa428ff9640 1 -- 172.21.15.59:0/3225407119 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7fa41c097530 con 0x7fa42c0b27c0 2024-06-29T15:58:48.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.756+0000 7fa423fff640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa40c076540 0x7fa40c078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:48.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.760+0000 7fa428ff9640 1 -- 172.21.15.59:0/3225407119 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa41c05fe10 con 0x7fa42c0b27c0 2024-06-29T15:58:48.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.957+0000 7fa423fff640 1 -- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa40c076540 msgr2=0x7fa40c078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:48.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.957+0000 7fa423fff640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa40c076540 0x7fa40c078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:48.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.979+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa42c15b400 con 0x7fa42c0b27c0 2024-06-29T15:58:48.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.980+0000 7fa428ff9640 1 -- 172.21.15.59:0/3225407119 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v247) ==== 100+0+11 (secure 0 0 0) 0x7fa41c064110 con 0x7fa42c0b27c0 2024-06-29T15:58:48.981 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:48.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa40c076540 msgr2=0x7fa40c078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:48.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa40c076540 0x7fa40c078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0b27c0 msgr2=0x7fa42c1537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0b27c0 0x7fa42c1537a0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00c3f0 tx=0x7fa41c00c8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 shutdown_connections 2024-06-29T15:58:48.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fa40c076540 0x7fa40c078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c153fa0 0x7fa42c15ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa42c0d32b0 0x7fa42c15c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 --2- 172.21.15.59:0/3225407119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa42c0b27c0 0x7fa42c1537a0 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 >> 172.21.15.59:0/3225407119 conn(0x7fa42c0d3ec0 msgr2=0x7fa42c06cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 shutdown_connections 2024-06-29T15:58:48.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:48.982+0000 7fa43197a640 1 -- 172.21.15.59:0/3225407119 wait complete. 2024-06-29T15:58:48.995 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:49.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.116+0000 7f24e7a74640 1 -- 172.21.15.59:0/547437344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00d0130 msgr2=0x7f24e0143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.116+0000 7f24e7a74640 1 --2- 172.21.15.59:0/547437344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00d0130 0x7f24e0143840 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f24e0144290 tx=0x7f24dc02a530 comp rx=0 tx=0).stop 2024-06-29T15:58:49.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.116+0000 7f24e7a74640 1 -- 172.21.15.59:0/547437344 shutdown_connections 2024-06-29T15:58:49.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.116+0000 7f24e7a74640 1 --2- 172.21.15.59:0/547437344 >> v2:172.21.15.153:3300/0 conn(0x7f24e006ebe0 0x7f24e00cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.116+0000 7f24e7a74640 1 --2- 172.21.15.59:0/547437344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00d0130 0x7f24e0143840 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.116+0000 7f24e7a74640 1 -- 172.21.15.59:0/547437344 >> 172.21.15.59:0/547437344 conn(0x7f24e00b2ab0 msgr2=0x7f24e00b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.116+0000 7f24e7a74640 1 -- 172.21.15.59:0/547437344 shutdown_connections 2024-06-29T15:58:49.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.116+0000 7f24e7a74640 1 -- 172.21.15.59:0/547437344 wait complete. 2024-06-29T15:58:49.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.118+0000 7f24e7a74640 1 Processor -- start 2024-06-29T15:58:49.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.118+0000 7f24e7a74640 1 -- start start 2024-06-29T15:58:49.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e7a74640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24e006ebe0 0x7f24e013e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e7a74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00b4ee0 0x7f24e00ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e7a74640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24e00d0130 0x7f24e00acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e7a74640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f24e00cf650 con 0x7f24e00b4ee0 2024-06-29T15:58:49.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e7a74640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f24e00cf4d0 con 0x7f24e006ebe0 2024-06-29T15:58:49.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e7a74640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f24e00cf7d0 con 0x7f24e00d0130 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e500f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00b4ee0 0x7f24e00ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e5810640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24e006ebe0 0x7f24e013e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e500f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00b4ee0 0x7f24e00ac860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52140/0 (socket says 172.21.15.59:52140) 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e500f640 1 -- 172.21.15.59:0/759352114 learned_addr learned my addr 172.21.15.59:0/759352114 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e6011640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24e00d0130 0x7f24e00acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e500f640 1 -- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24e00d0130 msgr2=0x7f24e00acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e500f640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24e00d0130 0x7f24e00acda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e500f640 1 -- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24e006ebe0 msgr2=0x7f24e013e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e500f640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24e006ebe0 0x7f24e013e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.120+0000 7f24e500f640 1 -- 172.21.15.59:0/759352114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e013eec0 con 0x7f24e00b4ee0 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.121+0000 7f24e500f640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00b4ee0 0x7f24e00ac860 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f24d400c3a0 tx=0x7f24d400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.121+0000 7f24d27fc640 1 -- 172.21.15.59:0/759352114 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d4016070 con 0x7f24e00b4ee0 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.121+0000 7f24d27fc640 1 -- 172.21.15.59:0/759352114 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24d4006b80 con 0x7f24e00b4ee0 2024-06-29T15:58:49.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.121+0000 7f24d27fc640 1 -- 172.21.15.59:0/759352114 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d4007020 con 0x7f24e00b4ee0 2024-06-29T15:58:49.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.121+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e00a9640 con 0x7f24e00b4ee0 2024-06-29T15:58:49.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.121+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e00a9c00 con 0x7f24e00b4ee0 2024-06-29T15:58:49.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.122+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e00ce2b0 con 0x7f24e00b4ee0 2024-06-29T15:58:49.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.123+0000 7f24d27fc640 1 -- 172.21.15.59:0/759352114 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7f24d40071c0 con 0x7f24e00b4ee0 2024-06-29T15:58:49.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.124+0000 7f24d27fc640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f24c4076510 0x7f24c4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.124+0000 7f24e5810640 1 -- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f24c4076510 msgr2=0x7f24c4078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:49.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.124+0000 7f24d27fc640 1 -- 172.21.15.59:0/759352114 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7f24d4099c50 con 0x7f24e00b4ee0 2024-06-29T15:58:49.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.124+0000 7f24e5810640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f24c4076510 0x7f24c4078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:49.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.129+0000 7f24d27fc640 1 -- 172.21.15.59:0/759352114 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f24e00ce2b0 con 0x7f24e00b4ee0 2024-06-29T15:58:49.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.323+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f24e00cfcf0 con 0x7f24e00b4ee0 2024-06-29T15:58:49.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.323+0000 7f24d27fc640 1 -- 172.21.15.59:0/759352114 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v247) ==== 107+0+11 (secure 0 0 0) 0x7f24d409f020 con 0x7f24e00b4ee0 2024-06-29T15:58:49.324 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f24c4076510 msgr2=0x7f24c4078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f24c4076510 0x7f24c4078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00b4ee0 msgr2=0x7f24e00ac860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00b4ee0 0x7f24e00ac860 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f24d400c3a0 tx=0x7f24d400c860 comp rx=0 tx=0).stop 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 shutdown_connections 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f24c4076510 0x7f24c4078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f24e00d0130 0x7f24e00acda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e00b4ee0 0x7f24e00ac860 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 --2- 172.21.15.59:0/759352114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24e006ebe0 0x7f24e013e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 >> 172.21.15.59:0/759352114 conn(0x7f24e00b2ab0 msgr2=0x7f24e006e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 shutdown_connections 2024-06-29T15:58:49.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.326+0000 7f24e7a74640 1 -- 172.21.15.59:0/759352114 wait complete. 2024-06-29T15:58:49.338 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.459+0000 7f67651f5640 1 -- 172.21.15.59:0/3250039927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 msgr2=0x7f676006e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.459+0000 7f67651f5640 1 --2- 172.21.15.59:0/3250039927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 0x7f676006e4d0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f6748009830 tx=0x7f674802a510 comp rx=0 tx=0).stop 2024-06-29T15:58:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.459+0000 7f67651f5640 1 -- 172.21.15.59:0/3250039927 shutdown_connections 2024-06-29T15:58:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.459+0000 7f67651f5640 1 --2- 172.21.15.59:0/3250039927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 0x7f676006e4d0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.459+0000 7f67651f5640 1 --2- 172.21.15.59:0/3250039927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67600d0130 0x7f6760143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.459+0000 7f67651f5640 1 -- 172.21.15.59:0/3250039927 >> 172.21.15.59:0/3250039927 conn(0x7f67600b2ab0 msgr2=0x7f67600b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.459+0000 7f67651f5640 1 -- 172.21.15.59:0/3250039927 shutdown_connections 2024-06-29T15:58:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.459+0000 7f67651f5640 1 -- 172.21.15.59:0/3250039927 wait complete. 2024-06-29T15:58:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.461+0000 7f67651f5640 1 Processor -- start 2024-06-29T15:58:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.461+0000 7f67651f5640 1 -- start start 2024-06-29T15:58:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f67651f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676006ea10 0x7f6760152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f67651f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 0x7f6760153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f67651f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67600d0130 0x7f676015c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f67651f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f676015ed80 con 0x7f67600b4ee0 2024-06-29T15:58:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f67651f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f676006f030 con 0x7f676006ea10 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f67651f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f676015ef00 con 0x7f67600d0130 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f6757fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 0x7f6760153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f6757fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 0x7f6760153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52148/0 (socket says 172.21.15.59:52148) 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f6757fff640 1 -- 172.21.15.59:0/4028461844 learned_addr learned my addr 172.21.15.59:0/4028461844 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f675f59e640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67600d0130 0x7f676015c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f675ed9d640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676006ea10 0x7f6760152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f6757fff640 1 -- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67600d0130 msgr2=0x7f676015c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f6757fff640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67600d0130 0x7f676015c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f6757fff640 1 -- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676006ea10 msgr2=0x7f6760152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f6757fff640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676006ea10 0x7f6760152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f6757fff640 1 -- 172.21.15.59:0/4028461844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6760154e10 con 0x7f67600b4ee0 2024-06-29T15:58:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.463+0000 7f675ed9d640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676006ea10 0x7f6760152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:49.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.464+0000 7f6757fff640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 0x7f6760153100 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f6748009950 tx=0x7f6748033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.464+0000 7f67577fe640 1 -- 172.21.15.59:0/4028461844 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748005c90 con 0x7f67600b4ee0 2024-06-29T15:58:49.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.464+0000 7f67577fe640 1 -- 172.21.15.59:0/4028461844 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6748031e50 con 0x7f67600b4ee0 2024-06-29T15:58:49.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.464+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67601550f0 con 0x7f67600b4ee0 2024-06-29T15:58:49.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.464+0000 7f67577fe640 1 -- 172.21.15.59:0/4028461844 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67480332f0 con 0x7f67600b4ee0 2024-06-29T15:58:49.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.464+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67601556b0 con 0x7f67600b4ee0 2024-06-29T15:58:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.465+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f676015f3a0 con 0x7f67600b4ee0 2024-06-29T15:58:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.466+0000 7f67577fe640 1 -- 172.21.15.59:0/4028461844 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7f6748033490 con 0x7f67600b4ee0 2024-06-29T15:58:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.466+0000 7f67577fe640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f6740076510 0x7f6740078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.466+0000 7f67577fe640 1 -- 172.21.15.59:0/4028461844 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7f67480b9330 con 0x7f67600b4ee0 2024-06-29T15:58:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.466+0000 7f675ed9d640 1 -- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f6740076510 msgr2=0x7f6740078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.466+0000 7f675ed9d640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f6740076510 0x7f6740078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:49.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.471+0000 7f67577fe640 1 -- 172.21.15.59:0/4028461844 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f674804a030 con 0x7f67600b4ee0 2024-06-29T15:58:49.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.663+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f676006f4e0 con 0x7f67600b4ee0 2024-06-29T15:58:49.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.664+0000 7f67577fe640 1 -- 172.21.15.59:0/4028461844 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v247) ==== 103+0+11 (secure 0 0 0) 0x7f674803f030 con 0x7f67600b4ee0 2024-06-29T15:58:49.665 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:49.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f6740076510 msgr2=0x7f6740078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:49.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f6740076510 0x7f6740078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 msgr2=0x7f6760153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 0x7f6760153100 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f6748009950 tx=0x7f6748033c90 comp rx=0 tx=0).stop 2024-06-29T15:58:49.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 shutdown_connections 2024-06-29T15:58:49.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7f6740076510 0x7f6740078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67600d0130 0x7f676015c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67600b4ee0 0x7f6760153100 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 --2- 172.21.15.59:0/4028461844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676006ea10 0x7f6760152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 >> 172.21.15.59:0/4028461844 conn(0x7f67600b2ab0 msgr2=0x7f67600b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 shutdown_connections 2024-06-29T15:58:49.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.666+0000 7f67651f5640 1 -- 172.21.15.59:0/4028461844 wait complete. 2024-06-29T15:58:49.682 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:58:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.800+0000 7fda47bc7640 1 -- 172.21.15.59:0/3983803582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda400b5320 msgr2=0x7fda400b5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.800+0000 7fda47bc7640 1 --2- 172.21.15.59:0/3983803582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda400b5320 0x7fda400b5730 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fda38005fd0 tx=0x7fda3802a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.800+0000 7fda47bc7640 1 -- 172.21.15.59:0/3983803582 shutdown_connections 2024-06-29T15:58:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.800+0000 7fda47bc7640 1 --2- 172.21.15.59:0/3983803582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda400b5320 0x7fda400b5730 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.800+0000 7fda47bc7640 1 -- 172.21.15.59:0/3983803582 >> 172.21.15.59:0/3983803582 conn(0x7fda40073e50 msgr2=0x7fda400b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.800+0000 7fda47bc7640 1 -- 172.21.15.59:0/3983803582 shutdown_connections 2024-06-29T15:58:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.800+0000 7fda47bc7640 1 -- 172.21.15.59:0/3983803582 wait complete. 2024-06-29T15:58:49.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.802+0000 7fda47bc7640 1 Processor -- start 2024-06-29T15:58:49.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.802+0000 7fda47bc7640 1 -- start start 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.803+0000 7fda47bc7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4006eb40 0x7fda4015a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.803+0000 7fda47bc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda40071620 0x7fda4013c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.803+0000 7fda47bc7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda400b5320 0x7fda401f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.803+0000 7fda47bc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fda401661d0 con 0x7fda40071620 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.803+0000 7fda47bc7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda40166050 con 0x7fda400b5320 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.803+0000 7fda47bc7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda40166350 con 0x7fda4006eb40 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda40071620 0x7fda4013c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda40071620 0x7fda4013c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52174/0 (socket says 172.21.15.59:52174) 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 -- 172.21.15.59:0/538091451 learned_addr learned my addr 172.21.15.59:0/538091451 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda46164640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda400b5320 0x7fda401f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45963640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4006eb40 0x7fda4015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 -- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4006eb40 msgr2=0x7fda4015a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4006eb40 0x7fda4015a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 -- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda400b5320 msgr2=0x7fda401f4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda400b5320 0x7fda401f4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 -- 172.21.15.59:0/538091451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4013c160 con 0x7fda40071620 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda45162640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda40071620 0x7fda4013c920 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00c400 tx=0x7fda3c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda327fc640 1 -- 172.21.15.59:0/538091451 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c011790 con 0x7fda40071620 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda327fc640 1 -- 172.21.15.59:0/538091451 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda3c017070 con 0x7fda40071620 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda40143690 con 0x7fda40071620 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.804+0000 7fda327fc640 1 -- 172.21.15.59:0/538091451 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c011ce0 con 0x7fda40071620 2024-06-29T15:58:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.805+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4013c400 con 0x7fda40071620 2024-06-29T15:58:49.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.806+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda400b5730 con 0x7fda40071620 2024-06-29T15:58:49.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.807+0000 7fda327fc640 1 -- 172.21.15.59:0/538091451 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7fda3c01c460 con 0x7fda40071620 2024-06-29T15:58:49.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.808+0000 7fda327fc640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fda24076510 0x7fda24078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.808+0000 7fda327fc640 1 -- 172.21.15.59:0/538091451 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7fda3c097f40 con 0x7fda40071620 2024-06-29T15:58:49.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.808+0000 7fda45963640 1 -- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fda24076510 msgr2=0x7fda24078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:49.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.808+0000 7fda45963640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fda24076510 0x7fda24078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:49.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:49.812+0000 7fda327fc640 1 -- 172.21.15.59:0/538091451 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda40070300 con 0x7fda40071620 2024-06-29T15:58:50.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.009+0000 7fda45963640 1 -- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fda24076510 msgr2=0x7fda24078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:50.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.009+0000 7fda45963640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fda24076510 0x7fda24078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:50.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.018+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fda40143360 con 0x7fda40071620 2024-06-29T15:58:50.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.019+0000 7fda327fc640 1 -- 172.21.15.59:0/538091451 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v247) ==== 81+0+595 (secure 0 0 0) 0x7fda3c060860 con 0x7fda40071620 2024-06-29T15:58:50.020 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/877026286 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.020 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3972699112 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.020 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1636302558 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.020 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1944500160 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.020 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3355566242 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.021 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/1831136380 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.021 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/1831136380 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.021 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2494131047 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.021 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1461412440 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.021 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1259499350 2024-06-30T15:58:45.316819+0000 2024-06-29T15:58:50.021 INFO:teuthology.orchestra.run.smithi059.stderr:listed 10 entries 2024-06-29T15:58:50.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fda24076510 msgr2=0x7fda24078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:50.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fda24076510 0x7fda24078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda40071620 msgr2=0x7fda4013c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda40071620 0x7fda4013c920 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00c400 tx=0x7fda3c00c8c0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 shutdown_connections 2024-06-29T15:58:50.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fda24076510 0x7fda24078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda400b5320 0x7fda401f4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda40071620 0x7fda4013c920 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 --2- 172.21.15.59:0/538091451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4006eb40 0x7fda4015a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 >> 172.21.15.59:0/538091451 conn(0x7fda40073e50 msgr2=0x7fda400b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 shutdown_connections 2024-06-29T15:58:50.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.021+0000 7fda47bc7640 1 -- 172.21.15.59:0/538091451 wait complete. 2024-06-29T15:58:50.033 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:58:50.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:50.033 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:50.063 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T15:58:50.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:50.063 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:58:50.120 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:58:50.120 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:58:50.120 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:50.136 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:58:50.136 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:58:50.136 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:58:50.136 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:50.176 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:58:50.176 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:58:50.176 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:58:50.176 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:50.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:58:50.194 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:50.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.333+0000 7fb71511c640 1 -- 172.21.15.59:0/2860445813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 msgr2=0x7fb7100b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.333+0000 7fb71511c640 1 --2- 172.21.15.59:0/2860445813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 0x7fb7100b3990 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7fb71008c4f0 tx=0x7fb70002a5f0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.333+0000 7fb71511c640 1 -- 172.21.15.59:0/2860445813 shutdown_connections 2024-06-29T15:58:50.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.333+0000 7fb71511c640 1 --2- 172.21.15.59:0/2860445813 >> v2:172.21.15.153:3300/0 conn(0x7fb7100b3ed0 0x7fb71015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.333+0000 7fb71511c640 1 --2- 172.21.15.59:0/2860445813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 0x7fb7100b3990 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.333+0000 7fb71511c640 1 -- 172.21.15.59:0/2860445813 >> 172.21.15.59:0/2860445813 conn(0x7fb7100d3ee0 msgr2=0x7fb7100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.333+0000 7fb71511c640 1 -- 172.21.15.59:0/2860445813 shutdown_connections 2024-06-29T15:58:50.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.333+0000 7fb71511c640 1 -- 172.21.15.59:0/2860445813 wait complete. 2024-06-29T15:58:50.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.335+0000 7fb71511c640 1 Processor -- start 2024-06-29T15:58:50.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.335+0000 7fb71511c640 1 -- start start 2024-06-29T15:58:50.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb71511c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7100b3ed0 0x7fb71014f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb71511c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 0x7fb710151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb71511c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71015ee10 0x7fb710154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb71511c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb710160240 con 0x7fb7100d23c0 2024-06-29T15:58:50.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb71511c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb7101600c0 con 0x7fb7100b3ed0 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb71511c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb7101603c0 con 0x7fb71015ee10 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 0x7fb710151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 0x7fb710151400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52186/0 (socket says 172.21.15.59:52186) 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 -- 172.21.15.59:0/1282575049 learned_addr learned my addr 172.21.15.59:0/1282575049 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70ed9d640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7100b3ed0 0x7fb71014f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70f59e640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71015ee10 0x7fb710154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 -- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71015ee10 msgr2=0x7fb710154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71015ee10 0x7fb710154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 -- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7100b3ed0 msgr2=0x7fb71014f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7100b3ed0 0x7fb71014f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 -- 172.21.15.59:0/1282575049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7101e77c0 con 0x7fb7100d23c0 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.337+0000 7fb70e59c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 0x7fb710151400 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7fb704009fd0 tx=0x7fb70400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.338+0000 7fb6f77fe640 1 -- 172.21.15.59:0/1282575049 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70400ccc0 con 0x7fb7100d23c0 2024-06-29T15:58:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.338+0000 7fb6f77fe640 1 -- 172.21.15.59:0/1282575049 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb70400ee70 con 0x7fb7100d23c0 2024-06-29T15:58:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.338+0000 7fb6f77fe640 1 -- 172.21.15.59:0/1282575049 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70400ce60 con 0x7fb7100d23c0 2024-06-29T15:58:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.338+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb710151fe0 con 0x7fb7100d23c0 2024-06-29T15:58:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.338+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7101525a0 con 0x7fb7100d23c0 2024-06-29T15:58:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.339+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb710151d50 con 0x7fb7100d23c0 2024-06-29T15:58:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.341+0000 7fb6f77fe640 1 -- 172.21.15.59:0/1282575049 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7fb704005260 con 0x7fb7100d23c0 2024-06-29T15:58:50.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.341+0000 7fb6f77fe640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fb6fc076510 0x7fb6fc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.341+0000 7fb70ed9d640 1 -- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fb6fc076510 msgr2=0x7fb6fc078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:50.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.341+0000 7fb6f77fe640 1 -- 172.21.15.59:0/1282575049 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7fb704098ee0 con 0x7fb7100d23c0 2024-06-29T15:58:50.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.341+0000 7fb70ed9d640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fb6fc076510 0x7fb6fc078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:50.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.345+0000 7fb6f77fe640 1 -- 172.21.15.59:0/1282575049 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7101e77c0 con 0x7fb7100d23c0 2024-06-29T15:58:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.543+0000 7fb70ed9d640 1 -- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fb6fc076510 msgr2=0x7fb6fc078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.543+0000 7fb70ed9d640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fb6fc076510 0x7fb6fc078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.578+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb71015e8d0 con 0x7fb7100d23c0 2024-06-29T15:58:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.579+0000 7fb6f77fe640 1 -- 172.21.15.59:0/1282575049 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6568 (secure 0 0 0) 0x7fb704061700 con 0x7fb7100d23c0 2024-06-29T15:58:50.580 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:50.580 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":137,"btime":"2024-06-29T15:58:37:645635+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:20.935697+0000","modified":"2024-06-29T15:58:23.956772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/177893499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":177893499},{"type":"v1","addr":"172.21.15.153:6837","nonce":177893499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12318,"qdb_cluster":[12318]},"id":15},{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:58:50.580 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 137 2024-06-29T15:58:50.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fb6fc076510 msgr2=0x7fb6fc078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:50.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fb6fc076510 0x7fb6fc078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 msgr2=0x7fb710151400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 0x7fb710151400 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7fb704009fd0 tx=0x7fb70400ec20 comp rx=0 tx=0).stop 2024-06-29T15:58:50.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 shutdown_connections 2024-06-29T15:58:50.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7fb6fc076510 0x7fb6fc078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb71015ee10 0x7fb710154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7100d23c0 0x7fb710151400 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 --2- 172.21.15.59:0/1282575049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7100b3ed0 0x7fb71014f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 >> 172.21.15.59:0/1282575049 conn(0x7fb7100d3ee0 msgr2=0x7fb71015f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 shutdown_connections 2024-06-29T15:58:50.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.582+0000 7fb71511c640 1 -- 172.21.15.59:0/1282575049 wait complete. 2024-06-29T15:58:50.595 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.715+0000 7ff069455640 1 -- 172.21.15.59:0/3975901888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff06415efe0 msgr2=0x7ff06415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.715+0000 7ff069455640 1 --2- 172.21.15.59:0/3975901888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff06415efe0 0x7ff06415f9b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009be0 tx=0x7ff05c02a920 comp rx=0 tx=0).stop 2024-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.716+0000 7ff069455640 1 -- 172.21.15.59:0/3975901888 shutdown_connections 2024-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.716+0000 7ff069455640 1 --2- 172.21.15.59:0/3975901888 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff06415efe0 0x7ff06415f9b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.716+0000 7ff069455640 1 -- 172.21.15.59:0/3975901888 >> 172.21.15.59:0/3975901888 conn(0x7ff0640d3ee0 msgr2=0x7ff0640d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.716+0000 7ff069455640 1 -- 172.21.15.59:0/3975901888 shutdown_connections 2024-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.716+0000 7ff069455640 1 -- 172.21.15.59:0/3975901888 wait complete. 2024-06-29T15:58:50.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.717+0000 7ff069455640 1 Processor -- start 2024-06-29T15:58:50.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.717+0000 7ff069455640 1 -- start start 2024-06-29T15:58:50.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff069455640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0640d23c0 0x7ff0640cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff069455640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06415efe0 0x7ff0640c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff069455640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff064157f80 0x7ff0641f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff069455640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff064160470 con 0x7ff064157f80 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff069455640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0641602f0 con 0x7ff06415efe0 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff069455640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff0641605f0 con 0x7ff0640d23c0 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff064157f80 0x7ff0641f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff062ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0640d23c0 0x7ff0640cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff064157f80 0x7ff0641f5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52194/0 (socket says 172.21.15.59:52194) 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff062ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0640d23c0 0x7ff0640cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:59048/0 (socket says 172.21.15.59:59048) 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 -- 172.21.15.59:0/3788163487 learned_addr learned my addr 172.21.15.59:0/3788163487 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff05bfff640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06415efe0 0x7ff0640c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 -- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0640d23c0 msgr2=0x7ff0640cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0640d23c0 0x7ff0640cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 -- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06415efe0 msgr2=0x7ff0640c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06415efe0 0x7ff0640c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 -- 172.21.15.59:0/3788163487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0640ccd30 con 0x7ff064157f80 2024-06-29T15:58:50.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.719+0000 7ff0637fe640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff064157f80 0x7ff0641f5230 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009be0 tx=0x7ff05c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:50.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.720+0000 7ff05b7fe640 1 -- 172.21.15.59:0/3788163487 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c030c50 con 0x7ff064157f80 2024-06-29T15:58:50.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.720+0000 7ff05b7fe640 1 -- 172.21.15.59:0/3788163487 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff05c03e070 con 0x7ff064157f80 2024-06-29T15:58:50.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.720+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0640cd010 con 0x7ff064157f80 2024-06-29T15:58:50.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.720+0000 7ff05b7fe640 1 -- 172.21.15.59:0/3788163487 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c032330 con 0x7ff064157f80 2024-06-29T15:58:50.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.720+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0641552e0 con 0x7ff064157f80 2024-06-29T15:58:50.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.721+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0640ccaa0 con 0x7ff064157f80 2024-06-29T15:58:50.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.722+0000 7ff05b7fe640 1 -- 172.21.15.59:0/3788163487 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7ff05c0324d0 con 0x7ff064157f80 2024-06-29T15:58:50.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.722+0000 7ff05b7fe640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7ff044076540 0x7ff044078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.722+0000 7ff062ffd640 1 -- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7ff044076540 msgr2=0x7ff044078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:50.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.722+0000 7ff05b7fe640 1 -- 172.21.15.59:0/3788163487 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 7451+0+0 (secure 0 0 0) 0x7ff05c0b9920 con 0x7ff064157f80 2024-06-29T15:58:50.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.722+0000 7ff062ffd640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7ff044076540 0x7ff044078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:50.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.726+0000 7ff05b7fe640 1 -- 172.21.15.59:0/3788163487 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff05c02d070 con 0x7ff064157f80 2024-06-29T15:58:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.922+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff06415f9b0 con 0x7ff064157f80 2024-06-29T15:58:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.924+0000 7ff062ffd640 1 -- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7ff044076540 msgr2=0x7ff044078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2715634623 2024-06-29T15:58:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.924+0000 7ff062ffd640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7ff044076540 0x7ff044078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.924+0000 7ff05b7fe640 1 -- 172.21.15.59:0/3788163487 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) ==== 74+0+21414 (secure 0 0 0) 0x7ff05c082200 con 0x7ff064157f80 2024-06-29T15:58:50.925 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:50.925 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":247,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:45.316883+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/877026286":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3972699112":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1636302558":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1944500160":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3355566242":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6832/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/2494131047":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1461412440":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1259499350":"2024-06-30T15:58:45.316819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:50.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7ff044076540 msgr2=0x7ff044078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:50.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7ff044076540 0x7ff044078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff064157f80 msgr2=0x7ff0641f5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff064157f80 0x7ff0641f5230 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009be0 tx=0x7ff05c003ba0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 shutdown_connections 2024-06-29T15:58:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:6832/2715634623,v1:172.21.15.153:6833/2715634623] conn(0x7ff044076540 0x7ff044078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff064157f80 0x7ff0641f5230 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff06415efe0 0x7ff0640c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 --2- 172.21.15.59:0/3788163487 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0640d23c0 0x7ff0640cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 >> 172.21.15.59:0/3788163487 conn(0x7ff0640d3ee0 msgr2=0x7ff0640b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 shutdown_connections 2024-06-29T15:58:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:50.926+0000 7ff069455640 1 -- 172.21.15.59:0/3788163487 wait complete. 2024-06-29T15:58:50.939 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:58:19.303469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '241', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:58:19.884648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:58:31.630427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:58:32.972384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:50.939 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:51.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.067+0000 7fcd32deb640 1 -- 172.21.15.59:0/2113426984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 msgr2=0x7fcd180040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.067+0000 7fcd32deb640 1 --2- 172.21.15.59:0/2113426984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 0x7fcd180040f0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c001830 tx=0x7fcd18005830 comp rx=0 tx=0).stop 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.067+0000 7fcd32deb640 1 -- 172.21.15.59:0/2113426984 shutdown_connections 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.067+0000 7fcd32deb640 1 --2- 172.21.15.59:0/2113426984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 0x7fcd180040f0 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.067+0000 7fcd32deb640 1 -- 172.21.15.59:0/2113426984 >> 172.21.15.59:0/2113426984 conn(0x7fcd2c0cd4c0 msgr2=0x7fcd2c0cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.067+0000 7fcd32deb640 1 -- 172.21.15.59:0/2113426984 shutdown_connections 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.067+0000 7fcd32deb640 1 -- 172.21.15.59:0/2113426984 wait complete. 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.069+0000 7fcd32deb640 1 Processor -- start 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.069+0000 7fcd32deb640 1 -- start start 2024-06-29T15:58:51.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.070+0000 7fcd32deb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 0x7fcd2c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.070+0000 7fcd32deb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd2c13c710 0x7fcd2c06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.070+0000 7fcd32deb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd2c156030 0x7fcd2c156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.070+0000 7fcd32deb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c1410a0 con 0x7fcd18003ca0 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.070+0000 7fcd32deb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c140f20 con 0x7fcd2c13c710 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.070+0000 7fcd32deb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcd2c141220 con 0x7fcd2c156030 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 0x7fcd2c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 0x7fcd2c06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52218/0 (socket says 172.21.15.59:52218) 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 -- 172.21.15.59:0/1866890838 learned_addr learned my addr 172.21.15.59:0/1866890838 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd31388640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd2c156030 0x7fcd2c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd2bfff640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd2c13c710 0x7fcd2c06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 -- 172.21.15.59:0/1866890838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd2c156030 msgr2=0x7fcd2c156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd2c156030 0x7fcd2c156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 -- 172.21.15.59:0/1866890838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd2c13c710 msgr2=0x7fcd2c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd2c13c710 0x7fcd2c06cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 -- 172.21.15.59:0/1866890838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2c13c4f0 con 0x7fcd18003ca0 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd30b87640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 0x7fcd2c06ca70 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c001950 tx=0x7fcd1c006170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd297fa640 1 -- 172.21.15.59:0/1866890838 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c043070 con 0x7fcd18003ca0 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd297fa640 1 -- 172.21.15.59:0/1866890838 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd1c003af0 con 0x7fcd18003ca0 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2c157c30 con 0x7fcd18003ca0 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd297fa640 1 -- 172.21.15.59:0/1866890838 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c033480 con 0x7fcd18003ca0 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.071+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd2c15a2d0 con 0x7fcd18003ca0 2024-06-29T15:58:51.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.073+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd2c15adb0 con 0x7fcd18003ca0 2024-06-29T15:58:51.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.073+0000 7fcd297fa640 1 -- 172.21.15.59:0/1866890838 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 143) ==== 97912+0+0 (secure 0 0 0) 0x7fcd1c046070 con 0x7fcd18003ca0 2024-06-29T15:58:51.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.074+0000 7fcd297fa640 1 -- 172.21.15.59:0/1866890838 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7fcd1c0b7880 con 0x7fcd18003ca0 2024-06-29T15:58:51.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.078+0000 7fcd297fa640 1 -- 172.21.15.59:0/1866890838 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcd1c006260 con 0x7fcd18003ca0 2024-06-29T15:58:51.081 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.081 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.081 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.081 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.081 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.081 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.082 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.082 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.082 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.082 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.080+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.091 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.091 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.092 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.092 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.092 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea03c3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.092 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.092 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.092 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.092 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.092 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:58:51.091+0000 7fea00c34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:51.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.272+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcd2c073e20 con 0x7fcd18003ca0 2024-06-29T15:58:51.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.273+0000 7fcd297fa640 1 -- 172.21.15.59:0/1866890838 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v248) ==== 93+0+10 (secure 0 0 0) 0x7fcd1c080670 con 0x7fcd18003ca0 2024-06-29T15:58:51.274 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:58:51.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 msgr2=0x7fcd2c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 0x7fcd2c06ca70 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c001950 tx=0x7fcd1c006170 comp rx=0 tx=0).stop 2024-06-29T15:58:51.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 shutdown_connections 2024-06-29T15:58:51.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd2c156030 0x7fcd2c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd2c13c710 0x7fcd2c06cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 --2- 172.21.15.59:0/1866890838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd18003ca0 0x7fcd2c06ca70 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 >> 172.21.15.59:0/1866890838 conn(0x7fcd2c0cd4c0 msgr2=0x7fcd2c0b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 shutdown_connections 2024-06-29T15:58:51.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.275+0000 7fcd32deb640 1 -- 172.21.15.59:0/1866890838 wait complete. 2024-06-29T15:58:51.287 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:58:51.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.408+0000 7f17b40a7640 1 -- 172.21.15.59:0/3627339868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac0d23c0 msgr2=0x7f17ac0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.408+0000 7f17b40a7640 1 --2- 172.21.15.59:0/3627339868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac0d23c0 0x7f17ac0b3990 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7f17a8005fd0 tx=0x7f17a802a670 comp rx=0 tx=0).stop 2024-06-29T15:58:51.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.408+0000 7f17b40a7640 1 -- 172.21.15.59:0/3627339868 shutdown_connections 2024-06-29T15:58:51.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.408+0000 7f17b40a7640 1 --2- 172.21.15.59:0/3627339868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac0d23c0 0x7f17ac0b3990 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.408+0000 7f17b40a7640 1 -- 172.21.15.59:0/3627339868 >> 172.21.15.59:0/3627339868 conn(0x7f17ac0d3ee0 msgr2=0x7f17ac0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.408+0000 7f17b40a7640 1 -- 172.21.15.59:0/3627339868 shutdown_connections 2024-06-29T15:58:51.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.408+0000 7f17b40a7640 1 -- 172.21.15.59:0/3627339868 wait complete. 2024-06-29T15:58:51.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.410+0000 7f17b40a7640 1 Processor -- start 2024-06-29T15:58:51.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.410+0000 7f17b40a7640 1 -- start start 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b40a7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17ac0b3ed0 0x7f17ac14b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b40a7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17ac0d23c0 0x7f17ac157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b40a7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac15efe0 0x7f17ac157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b40a7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac160450 con 0x7f17ac15efe0 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b40a7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac1602d0 con 0x7f17ac0d23c0 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b40a7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f17ac1605d0 con 0x7f17ac0b3ed0 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b2644640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac15efe0 0x7f17ac157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b2644640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac15efe0 0x7f17ac157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52228/0 (socket says 172.21.15.59:52228) 2024-06-29T15:58:51.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b2644640 1 -- 172.21.15.59:0/295058596 learned_addr learned my addr 172.21.15.59:0/295058596 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:51.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b1642640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17ac0d23c0 0x7f17ac157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b1e43640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17ac0b3ed0 0x7f17ac14b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b2644640 1 -- 172.21.15.59:0/295058596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17ac0b3ed0 msgr2=0x7f17ac14b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b2644640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17ac0b3ed0 0x7f17ac14b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b2644640 1 -- 172.21.15.59:0/295058596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17ac0d23c0 msgr2=0x7f17ac157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b2644640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17ac0d23c0 0x7f17ac157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.411+0000 7f17b2644640 1 -- 172.21.15.59:0/295058596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ac0c51b0 con 0x7f17ac15efe0 2024-06-29T15:58:51.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.412+0000 7f17b2644640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac15efe0 0x7f17ac157740 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f17a400c6b0 tx=0x7f17a400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:51.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.412+0000 7f17a27fc640 1 -- 172.21.15.59:0/295058596 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4003b70 con 0x7f17ac15efe0 2024-06-29T15:58:51.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.412+0000 7f17a27fc640 1 -- 172.21.15.59:0/295058596 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17a4003d10 con 0x7f17ac15efe0 2024-06-29T15:58:51.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.412+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ac1554e0 con 0x7f17ac15efe0 2024-06-29T15:58:51.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.412+0000 7f17a27fc640 1 -- 172.21.15.59:0/295058596 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4005330 con 0x7f17ac15efe0 2024-06-29T15:58:51.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.412+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ac155aa0 con 0x7f17ac15efe0 2024-06-29T15:58:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.413+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17ac0c5380 con 0x7f17ac15efe0 2024-06-29T15:58:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.414+0000 7f17a27fc640 1 -- 172.21.15.59:0/295058596 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 143) ==== 97912+0+0 (secure 0 0 0) 0x7f17a40054d0 con 0x7f17ac15efe0 2024-06-29T15:58:51.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.414+0000 7f17a27fc640 1 -- 172.21.15.59:0/295058596 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7f17a4098420 con 0x7f17ac15efe0 2024-06-29T15:58:51.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.418+0000 7f17a27fc640 1 -- 172.21.15.59:0/295058596 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f17a4061950 con 0x7f17ac15efe0 2024-06-29T15:58:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.608+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f17ac0b3990 con 0x7f17ac15efe0 2024-06-29T15:58:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.608+0000 7f17a27fc640 1 -- 172.21.15.59:0/295058596 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v248) ==== 104+0+11 (secure 0 0 0) 0x7f17a4061250 con 0x7f17ac15efe0 2024-06-29T15:58:51.610 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:58:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.610+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac15efe0 msgr2=0x7f17ac157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.610+0000 7f17b40a7640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac15efe0 0x7f17ac157740 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f17a400c6b0 tx=0x7f17a400cb70 comp rx=0 tx=0).stop 2024-06-29T15:58:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.610+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 shutdown_connections 2024-06-29T15:58:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.610+0000 7f17b40a7640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17ac15efe0 0x7f17ac157740 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.610+0000 7f17b40a7640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17ac0d23c0 0x7f17ac157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.610+0000 7f17b40a7640 1 --2- 172.21.15.59:0/295058596 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f17ac0b3ed0 0x7f17ac14b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.610+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 >> 172.21.15.59:0/295058596 conn(0x7f17ac0d3ee0 msgr2=0x7f17ac06c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.610+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 shutdown_connections 2024-06-29T15:58:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.611+0000 7f17b40a7640 1 -- 172.21.15.59:0/295058596 wait complete. 2024-06-29T15:58:51.623 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:58:51.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.743+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1651749767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 msgr2=0x7fd2ec0acfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.743+0000 7fd2f0e3c640 1 --2- 172.21.15.59:0/1651749767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 0x7fd2ec0acfa0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0005fd0 tx=0x7fd2e002a670 comp rx=0 tx=0).stop 2024-06-29T15:58:51.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.743+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1651749767 shutdown_connections 2024-06-29T15:58:51.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.743+0000 7fd2f0e3c640 1 --2- 172.21.15.59:0/1651749767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 0x7fd2ec0acfa0 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.743+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1651749767 >> 172.21.15.59:0/1651749767 conn(0x7fd2ec0b1ea0 msgr2=0x7fd2ec0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.743+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1651749767 shutdown_connections 2024-06-29T15:58:51.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.743+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1651749767 wait complete. 2024-06-29T15:58:51.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.745+0000 7fd2f0e3c640 1 Processor -- start 2024-06-29T15:58:51.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.745+0000 7fd2f0e3c640 1 -- start start 2024-06-29T15:58:51.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2f0e3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec0ad4e0 0x7fd2ec14e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2f0e3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ec0cb490 0x7fd2ec1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2f0e3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 0x7fd2ec1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2f0e3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec15bd80 con 0x7fd2ec0cc060 2024-06-29T15:58:51.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2f0e3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec15bc00 con 0x7fd2ec0ad4e0 2024-06-29T15:58:51.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2f0e3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ec15bf00 con 0x7fd2ec0cb490 2024-06-29T15:58:51.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ead9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 0x7fd2ec1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ead9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 0x7fd2ec1559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52248/0 (socket says 172.21.15.59:52248) 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ead9d640 1 -- 172.21.15.59:0/1171508666 learned_addr learned my addr 172.21.15.59:0/1171508666 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ea59c640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec0ad4e0 0x7fd2ec14e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2e9d9b640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ec0cb490 0x7fd2ec1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ead9d640 1 -- 172.21.15.59:0/1171508666 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ec0cb490 msgr2=0x7fd2ec1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ead9d640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ec0cb490 0x7fd2ec1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ead9d640 1 -- 172.21.15.59:0/1171508666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec0ad4e0 msgr2=0x7fd2ec14e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ead9d640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec0ad4e0 0x7fd2ec14e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.746+0000 7fd2ead9d640 1 -- 172.21.15.59:0/1171508666 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec15d180 con 0x7fd2ec0cc060 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.747+0000 7fd2ead9d640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 0x7fd2ec1559e0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4002230 tx=0x7fd2e400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.747+0000 7fd2daffd640 1 -- 172.21.15.59:0/1171508666 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4003b70 con 0x7fd2ec0cc060 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.747+0000 7fd2daffd640 1 -- 172.21.15.59:0/1171508666 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2e4003d10 con 0x7fd2ec0cc060 2024-06-29T15:58:51.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.747+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ec15d460 con 0x7fd2ec0cc060 2024-06-29T15:58:51.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.747+0000 7fd2daffd640 1 -- 172.21.15.59:0/1171508666 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4005330 con 0x7fd2ec0cc060 2024-06-29T15:58:51.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.747+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ec09c180 con 0x7fd2ec0cc060 2024-06-29T15:58:51.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.748+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2ec15d980 con 0x7fd2ec0cc060 2024-06-29T15:58:51.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.748+0000 7fd2daffd640 1 -- 172.21.15.59:0/1171508666 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 143) ==== 97912+0+0 (secure 0 0 0) 0x7fd2e40054d0 con 0x7fd2ec0cc060 2024-06-29T15:58:51.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.749+0000 7fd2daffd640 1 -- 172.21.15.59:0/1171508666 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7fd2e4097720 con 0x7fd2ec0cc060 2024-06-29T15:58:51.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.753+0000 7fd2daffd640 1 -- 172.21.15.59:0/1171508666 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd2e4060600 con 0x7fd2ec0cc060 2024-06-29T15:58:51.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.943+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd2ec0acfa0 con 0x7fd2ec0cc060 2024-06-29T15:58:51.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.944+0000 7fd2daffd640 1 -- 172.21.15.59:0/1171508666 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v248) ==== 100+0+11 (secure 0 0 0) 0x7fd2e4064900 con 0x7fd2ec0cc060 2024-06-29T15:58:51.945 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 msgr2=0x7fd2ec1559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 0x7fd2ec1559e0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4002230 tx=0x7fd2e400cb90 comp rx=0 tx=0).stop 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 shutdown_connections 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0cc060 0x7fd2ec1559e0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ec0cb490 0x7fd2ec1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 --2- 172.21.15.59:0/1171508666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ec0ad4e0 0x7fd2ec14e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 >> 172.21.15.59:0/1171508666 conn(0x7fd2ec0b1ea0 msgr2=0x7fd2ec0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 shutdown_connections 2024-06-29T15:58:51.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:51.946+0000 7fd2f0e3c640 1 -- 172.21.15.59:0/1171508666 wait complete. 2024-06-29T15:58:51.958 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.081+0000 7f0783466640 1 -- 172.21.15.59:0/3736683323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c08c190 msgr2=0x7f077c0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.081+0000 7f0783466640 1 --2- 172.21.15.59:0/3736683323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c08c190 0x7f077c0cf080 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f0778009860 tx=0x7f077802a560 comp rx=0 tx=0).stop 2024-06-29T15:58:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.081+0000 7f0783466640 1 -- 172.21.15.59:0/3736683323 shutdown_connections 2024-06-29T15:58:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.081+0000 7f0783466640 1 --2- 172.21.15.59:0/3736683323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c08c190 0x7f077c0cf080 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.081+0000 7f0783466640 1 -- 172.21.15.59:0/3736683323 >> 172.21.15.59:0/3736683323 conn(0x7f077c0b5610 msgr2=0x7f077c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.081+0000 7f0783466640 1 -- 172.21.15.59:0/3736683323 shutdown_connections 2024-06-29T15:58:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.081+0000 7f0783466640 1 -- 172.21.15.59:0/3736683323 wait complete. 2024-06-29T15:58:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.083+0000 7f0783466640 1 Processor -- start 2024-06-29T15:58:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.083+0000 7f0783466640 1 -- start start 2024-06-29T15:58:52.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.084+0000 7f0783466640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f077c08c190 0x7f077c14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.084+0000 7f0783466640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c0ce950 0x7f077c160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.084+0000 7f0783466640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c0cf5c0 0x7f077c0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.084+0000 7f0783466640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f077c169690 con 0x7f077c0cf5c0 2024-06-29T15:58:52.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.084+0000 7f0783466640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f077c169510 con 0x7f077c0ce950 2024-06-29T15:58:52.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.084+0000 7f0783466640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f077c169810 con 0x7f077c08c190 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.084+0000 7f0781a03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c0cf5c0 0x7f077c0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.084+0000 7f0781a03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c0cf5c0 0x7f077c0ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52266/0 (socket says 172.21.15.59:52266) 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0781a03640 1 -- 172.21.15.59:0/1514001251 learned_addr learned my addr 172.21.15.59:0/1514001251 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0781202640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f077c08c190 0x7f077c14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0780a01640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c0ce950 0x7f077c160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0781a03640 1 -- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f077c08c190 msgr2=0x7f077c14ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0781a03640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f077c08c190 0x7f077c14ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0781a03640 1 -- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c0ce950 msgr2=0x7f077c160970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0781a03640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c0ce950 0x7f077c160970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0781a03640 1 -- 172.21.15.59:0/1514001251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077c0cbb50 con 0x7f077c0cf5c0 2024-06-29T15:58:52.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0781a03640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c0cf5c0 0x7f077c0ccfc0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f077400c700 tx=0x7f077400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:52.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0771ffb640 1 -- 172.21.15.59:0/1514001251 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774003b70 con 0x7f077c0cf5c0 2024-06-29T15:58:52.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0771ffb640 1 -- 172.21.15.59:0/1514001251 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0774003d10 con 0x7f077c0cf5c0 2024-06-29T15:58:52.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f077c0cbe30 con 0x7f077c0cf5c0 2024-06-29T15:58:52.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0771ffb640 1 -- 172.21.15.59:0/1514001251 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077400aab0 con 0x7f077c0cf5c0 2024-06-29T15:58:52.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.085+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077c0cc3f0 con 0x7f077c0cf5c0 2024-06-29T15:58:52.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.087+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f077c0cf080 con 0x7f077c0cf5c0 2024-06-29T15:58:52.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.087+0000 7f0771ffb640 1 -- 172.21.15.59:0/1514001251 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7f077400ac50 con 0x7f077c0cf5c0 2024-06-29T15:58:52.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.088+0000 7f0771ffb640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0760076120 0x7f0760078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.088+0000 7f0771ffb640 1 -- 172.21.15.59:0/1514001251 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7f0774097930 con 0x7f077c0cf5c0 2024-06-29T15:58:52.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.088+0000 7f0781202640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0760076120 0x7f0760078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.089+0000 7f0781202640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0760076120 0x7f0760078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0778034000 tx=0x7f0778004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:52.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.093+0000 7f0771ffb640 1 -- 172.21.15.59:0/1514001251 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0774060680 con 0x7f077c0cf5c0 2024-06-29T15:58:52.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.284+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f077c0ca940 con 0x7f077c0cf5c0 2024-06-29T15:58:52.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.284+0000 7f0771ffb640 1 -- 172.21.15.59:0/1514001251 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v248) ==== 107+0+11 (secure 0 0 0) 0x7f0774064980 con 0x7f077c0cf5c0 2024-06-29T15:58:52.285 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.286+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0760076120 msgr2=0x7f0760078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.286+0000 7f0783466640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0760076120 0x7f0760078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0778034000 tx=0x7f0778004730 comp rx=0 tx=0).stop 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.286+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c0cf5c0 msgr2=0x7f077c0ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.286+0000 7f0783466640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c0cf5c0 0x7f077c0ccfc0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f077400c700 tx=0x7f077400cbc0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.287+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 shutdown_connections 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.287+0000 7f0783466640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0760076120 0x7f0760078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.287+0000 7f0783466640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f077c0cf5c0 0x7f077c0ccfc0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.287+0000 7f0783466640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077c0ce950 0x7f077c160970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.287+0000 7f0783466640 1 --2- 172.21.15.59:0/1514001251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f077c08c190 0x7f077c14ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.287+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 >> 172.21.15.59:0/1514001251 conn(0x7f077c0b5610 msgr2=0x7f077c157ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.287+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 shutdown_connections 2024-06-29T15:58:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.287+0000 7f0783466640 1 -- 172.21.15.59:0/1514001251 wait complete. 2024-06-29T15:58:52.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:52.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.420+0000 7f3dc3a81640 1 -- 172.21.15.59:0/2288223402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3da8004800 msgr2=0x7f3da8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.420+0000 7f3dc3a81640 1 --2- 172.21.15.59:0/2288223402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3da8004800 0x7f3da8004c10 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009b20 tx=0x7f3da8005830 comp rx=0 tx=0).stop 2024-06-29T15:58:52.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.420+0000 7f3dc3a81640 1 -- 172.21.15.59:0/2288223402 shutdown_connections 2024-06-29T15:58:52.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.420+0000 7f3dc3a81640 1 --2- 172.21.15.59:0/2288223402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3da8004800 0x7f3da8004c10 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.420+0000 7f3dc3a81640 1 -- 172.21.15.59:0/2288223402 >> 172.21.15.59:0/2288223402 conn(0x7f3dbc0b5610 msgr2=0x7f3dbc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:52.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.420+0000 7f3dc3a81640 1 -- 172.21.15.59:0/2288223402 shutdown_connections 2024-06-29T15:58:52.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.420+0000 7f3dc3a81640 1 -- 172.21.15.59:0/2288223402 wait complete. 2024-06-29T15:58:52.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.422+0000 7f3dc3a81640 1 Processor -- start 2024-06-29T15:58:52.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.422+0000 7f3dc3a81640 1 -- start start 2024-06-29T15:58:52.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc3a81640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3da8004800 0x7f3dbc154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc3a81640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3dbc0cb3d0 0x7f3dbc0c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc3a81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dbc0cd960 0x7f3dbc0c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc3a81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc162d90 con 0x7f3dbc0cd960 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc3a81640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc162c10 con 0x7f3da8004800 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc3a81640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3dbc162f10 con 0x7f3dbc0cb3d0 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc201e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dbc0cd960 0x7f3dbc0c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc201e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dbc0cd960 0x7f3dbc0c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52276/0 (socket says 172.21.15.59:52276) 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.423+0000 7f3dc201e640 1 -- 172.21.15.59:0/3446659554 learned_addr learned my addr 172.21.15.59:0/3446659554 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc101c640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3dbc0cb3d0 0x7f3dbc0c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc181d640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3da8004800 0x7f3dbc154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc201e640 1 -- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3dbc0cb3d0 msgr2=0x7f3dbc0c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc201e640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3dbc0cb3d0 0x7f3dbc0c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc201e640 1 -- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3da8004800 msgr2=0x7f3dbc154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc201e640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3da8004800 0x7f3dbc154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc201e640 1 -- 172.21.15.59:0/3446659554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dbc154570 con 0x7f3dbc0cd960 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc181d640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3da8004800 0x7f3dbc154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc201e640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dbc0cd960 0x7f3dbc0c8570 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f3db4000f80 tx=0x7f3db400eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3db27fc640 1 -- 172.21.15.59:0/3446659554 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db400c8f0 con 0x7f3dbc0cd960 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3db27fc640 1 -- 172.21.15.59:0/3446659554 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3db4017070 con 0x7f3dbc0cd960 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dbc0caf40 con 0x7f3dbc0cd960 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3db27fc640 1 -- 172.21.15.59:0/3446659554 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db400ce40 con 0x7f3dbc0cd960 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.424+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dbc161ce0 con 0x7f3dbc0cd960 2024-06-29T15:58:52.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.425+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dbc0cacb0 con 0x7f3dbc0cd960 2024-06-29T15:58:52.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.426+0000 7f3db27fc640 1 -- 172.21.15.59:0/3446659554 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7f3db401c460 con 0x7f3dbc0cd960 2024-06-29T15:58:52.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.426+0000 7f3db27fc640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3da0076120 0x7f3da0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.427+0000 7f3db27fc640 1 -- 172.21.15.59:0/3446659554 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7f3db4097ea0 con 0x7f3dbc0cd960 2024-06-29T15:58:52.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.427+0000 7f3dc181d640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3da0076120 0x7f3da0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.427+0000 7f3dc181d640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3da0076120 0x7f3da0078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3db8000c00 tx=0x7f3db80067e0 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:52.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.431+0000 7f3db27fc640 1 -- 172.21.15.59:0/3446659554 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3dbc154570 con 0x7f3dbc0cd960 2024-06-29T15:58:52.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.623+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f3dbc0cf240 con 0x7f3dbc0cd960 2024-06-29T15:58:52.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.623+0000 7f3db27fc640 1 -- 172.21.15.59:0/3446659554 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v248) ==== 103+0+11 (secure 0 0 0) 0x7f3db4061240 con 0x7f3dbc0cd960 2024-06-29T15:58:52.624 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:52.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3da0076120 msgr2=0x7f3da0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3da0076120 0x7f3da0078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3db8000c00 tx=0x7f3db80067e0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dbc0cd960 msgr2=0x7f3dbc0c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dbc0cd960 0x7f3dbc0c8570 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f3db4000f80 tx=0x7f3db400eb00 comp rx=0 tx=0).stop 2024-06-29T15:58:52.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 shutdown_connections 2024-06-29T15:58:52.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3da0076120 0x7f3da0078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3dbc0cd960 0x7f3dbc0c8570 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3dbc0cb3d0 0x7f3dbc0c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 --2- 172.21.15.59:0/3446659554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3da8004800 0x7f3dbc154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 >> 172.21.15.59:0/3446659554 conn(0x7f3dbc0b5610 msgr2=0x7f3dbc06d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:52.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 shutdown_connections 2024-06-29T15:58:52.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.626+0000 7f3dc3a81640 1 -- 172.21.15.59:0/3446659554 wait complete. 2024-06-29T15:58:52.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:52.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 -- 172.21.15.59:0/11715451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800b2270 msgr2=0x7f0f8015bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 --2- 172.21.15.59:0/11715451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800b2270 0x7f0f8015bc20 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f0f800b2d10 tx=0x7f0f7002a560 comp rx=0 tx=0).stop 2024-06-29T15:58:52.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 -- 172.21.15.59:0/11715451 shutdown_connections 2024-06-29T15:58:52.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 --2- 172.21.15.59:0/11715451 >> v2:172.21.15.153:3300/0 conn(0x7f0f800d1a60 0x7f0f800cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 --2- 172.21.15.59:0/11715451 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f800b2fa0 0x7f0f800d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 --2- 172.21.15.59:0/11715451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800b2270 0x7f0f8015bc20 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 -- 172.21.15.59:0/11715451 >> 172.21.15.59:0/11715451 conn(0x7f0f80073e10 msgr2=0x7f0f80074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:52.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 -- 172.21.15.59:0/11715451 shutdown_connections 2024-06-29T15:58:52.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.754+0000 7f0f86855640 1 -- 172.21.15.59:0/11715451 wait complete. 2024-06-29T15:58:52.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.756+0000 7f0f86855640 1 Processor -- start 2024-06-29T15:58:52.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.756+0000 7f0f86855640 1 -- start start 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f86855640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f800b2270 0x7f0f80149390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f86855640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f800b2fa0 0x7f0f800aa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f86855640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800d1a60 0x7f0f800b5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f86855640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0f800cfe40 con 0x7f0f800d1a60 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f86855640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f800cfcc0 con 0x7f0f800b2270 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f86855640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0f800cffc0 con 0x7f0f800b2fa0 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800d1a60 0x7f0f800b5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800d1a60 0x7f0f800b5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34696/0 (socket says 172.21.15.59:34696) 2024-06-29T15:58:52.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 -- 172.21.15.59:0/750520877 learned_addr learned my addr 172.21.15.59:0/750520877 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f7f7fe640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f800b2fa0 0x7f0f800aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f7ffff640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f800b2270 0x7f0f80149390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 -- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f800b2fa0 msgr2=0x7f0f800aa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f800b2fa0 0x7f0f800aa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 -- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f800b2270 msgr2=0x7f0f80149390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f800b2270 0x7f0f80149390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 -- 172.21.15.59:0/750520877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8015f020 con 0x7f0f800d1a60 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f7ffff640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f800b2270 0x7f0f80149390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f84df2640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800d1a60 0x7f0f800b5aa0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800e960 tx=0x7f0f7800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f7cff9640 1 -- 172.21.15.59:0/750520877 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78004d20 con 0x7f0f800d1a60 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f7cff9640 1 -- 172.21.15.59:0/750520877 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f7800cac0 con 0x7f0f800d1a60 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.758+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f800a9640 con 0x7f0f800d1a60 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.759+0000 7f0f7cff9640 1 -- 172.21.15.59:0/750520877 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78005020 con 0x7f0f800d1a60 2024-06-29T15:58:52.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.759+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f800a9c00 con 0x7f0f800d1a60 2024-06-29T15:58:52.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.760+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f800d0450 con 0x7f0f800d1a60 2024-06-29T15:58:52.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.760+0000 7f0f7cff9640 1 -- 172.21.15.59:0/750520877 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7f0f780051c0 con 0x7f0f800d1a60 2024-06-29T15:58:52.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.761+0000 7f0f7cff9640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0f6c076150 0x7f0f6c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.761+0000 7f0f7cff9640 1 -- 172.21.15.59:0/750520877 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7f0f780983e0 con 0x7f0f800d1a60 2024-06-29T15:58:52.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.761+0000 7f0f7ffff640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0f6c076150 0x7f0f6c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.761+0000 7f0f7ffff640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0f6c076150 0x7f0f6c078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0f70033000 tx=0x7f0f70004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:52.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.765+0000 7f0f7cff9640 1 -- 172.21.15.59:0/750520877 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f78016030 con 0x7f0f800d1a60 2024-06-29T15:58:52.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.993+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f800b6670 con 0x7f0f800d1a60 2024-06-29T15:58:52.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.994+0000 7f0f7cff9640 1 -- 172.21.15.59:0/750520877 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6568 (secure 0 0 0) 0x7f0f78061770 con 0x7f0f800d1a60 2024-06-29T15:58:52.995 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:52.995 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":137,"btime":"2024-06-29T15:58:37:645635+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:20.935697+0000","modified":"2024-06-29T15:58:23.956772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/177893499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":177893499},{"type":"v1","addr":"172.21.15.153:6837","nonce":177893499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12318,"qdb_cluster":[12318]},"id":15},{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:58:52.995 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 137 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.995+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0f6c076150 msgr2=0x7f0f6c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.995+0000 7f0f86855640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0f6c076150 0x7f0f6c078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0f70033000 tx=0x7f0f70004730 comp rx=0 tx=0).stop 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.995+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800d1a60 msgr2=0x7f0f800b5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.995+0000 7f0f86855640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800d1a60 0x7f0f800b5aa0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800e960 tx=0x7f0f7800ee20 comp rx=0 tx=0).stop 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.996+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 shutdown_connections 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.996+0000 7f0f86855640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0f6c076150 0x7f0f6c078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.996+0000 7f0f86855640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f800d1a60 0x7f0f800b5aa0 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.996+0000 7f0f86855640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f800b2fa0 0x7f0f800aa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.996+0000 7f0f86855640 1 --2- 172.21.15.59:0/750520877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f800b2270 0x7f0f80149390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.996+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 >> 172.21.15.59:0/750520877 conn(0x7f0f80073e10 msgr2=0x7f0f80158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:52.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.996+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 shutdown_connections 2024-06-29T15:58:52.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:52.996+0000 7f0f86855640 1 -- 172.21.15.59:0/750520877 wait complete. 2024-06-29T15:58:53.008 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:53.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.127+0000 7fd968218640 1 -- 172.21.15.59:0/3936418170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 msgr2=0x7fd9601622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.127+0000 7fd968218640 1 --2- 172.21.15.59:0/3936418170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 0x7fd9601622c0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fd9600b78c0 tx=0x7fd95002a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.127+0000 7fd968218640 1 -- 172.21.15.59:0/3936418170 shutdown_connections 2024-06-29T15:58:53.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.127+0000 7fd968218640 1 --2- 172.21.15.59:0/3936418170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 0x7fd9601622c0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.127+0000 7fd968218640 1 -- 172.21.15.59:0/3936418170 >> 172.21.15.59:0/3936418170 conn(0x7fd9600b5610 msgr2=0x7fd9600b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:53.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.127+0000 7fd968218640 1 -- 172.21.15.59:0/3936418170 shutdown_connections 2024-06-29T15:58:53.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.127+0000 7fd968218640 1 -- 172.21.15.59:0/3936418170 wait complete. 2024-06-29T15:58:53.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.128+0000 7fd968218640 1 Processor -- start 2024-06-29T15:58:53.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.129+0000 7fd968218640 1 -- start start 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd968218640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd96008c190 0x7fd960152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd968218640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 0x7fd96015f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd968218640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd960162800 0x7fd960161410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd968218640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd960163110 con 0x7fd9600ce950 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd968218640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd960162f90 con 0x7fd960162800 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd968218640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd9600b82e0 con 0x7fd96008c190 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 0x7fd96015f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 0x7fd96015f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34702/0 (socket says 172.21.15.59:34702) 2024-06-29T15:58:53.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 -- 172.21.15.59:0/3323333863 learned_addr learned my addr 172.21.15.59:0/3323333863 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd965fb4640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd96008c190 0x7fd960152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9667b5640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd960162800 0x7fd960161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 -- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd96008c190 msgr2=0x7fd960152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd96008c190 0x7fd960152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 -- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd960162800 msgr2=0x7fd960161410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd960162800 0x7fd960161410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 -- 172.21.15.59:0/3323333863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96015abf0 con 0x7fd9600ce950 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9667b5640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd960162800 0x7fd960161410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.130+0000 7fd9657b3640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 0x7fd96015f3c0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fd95002aa90 tx=0x7fd950033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.131+0000 7fd9567fc640 1 -- 172.21.15.59:0/3323333863 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950031d60 con 0x7fd9600ce950 2024-06-29T15:58:53.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.131+0000 7fd9567fc640 1 -- 172.21.15.59:0/3323333863 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd950033e10 con 0x7fd9600ce950 2024-06-29T15:58:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.131+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96015aed0 con 0x7fd9600ce950 2024-06-29T15:58:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.131+0000 7fd9567fc640 1 -- 172.21.15.59:0/3323333863 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950005260 con 0x7fd9600ce950 2024-06-29T15:58:53.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.131+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96015b880 con 0x7fd9600ce950 2024-06-29T15:58:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.132+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9600b6370 con 0x7fd9600ce950 2024-06-29T15:58:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.132+0000 7fd9567fc640 1 -- 172.21.15.59:0/3323333863 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7fd950005400 con 0x7fd9600ce950 2024-06-29T15:58:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.133+0000 7fd9567fc640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd944076150 0x7fd944078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.133+0000 7fd9567fc640 1 -- 172.21.15.59:0/3323333863 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7fd9500b8e30 con 0x7fd9600ce950 2024-06-29T15:58:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.134+0000 7fd965fb4640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd944076150 0x7fd944078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.134+0000 7fd965fb4640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd944076150 0x7fd944078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd95c006fd0 tx=0x7fd95c009060 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:53.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.138+0000 7fd9567fc640 1 -- 172.21.15.59:0/3323333863 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd9600b6370 con 0x7fd9600ce950 2024-06-29T15:58:53.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.330+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9601622c0 con 0x7fd9600ce950 2024-06-29T15:58:53.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.332+0000 7fd9567fc640 1 -- 172.21.15.59:0/3323333863 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) ==== 74+0+22048 (secure 0 0 0) 0x7fd95003f030 con 0x7fd9600ce950 2024-06-29T15:58:53.333 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:53.333 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":248,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:51.014825+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6832/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2374956971":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2562681960":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/221125627":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3082371830":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1656484964":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/402968645":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1556406125":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/877026286":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3972699112":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1636302558":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1944500160":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1892969816":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3355566242":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6832/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/2494131047":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1461412440":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1259499350":"2024-06-30T15:58:45.316819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd944076150 msgr2=0x7fd944078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd944076150 0x7fd944078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd95c006fd0 tx=0x7fd95c009060 comp rx=0 tx=0).stop 2024-06-29T15:58:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 msgr2=0x7fd96015f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 0x7fd96015f3c0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fd95002aa90 tx=0x7fd950033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 shutdown_connections 2024-06-29T15:58:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd944076150 0x7fd944078660 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd960162800 0x7fd960161410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9600ce950 0x7fd96015f3c0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 --2- 172.21.15.59:0/3323333863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd96008c190 0x7fd960152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 >> 172.21.15.59:0/3323333863 conn(0x7fd9600b5610 msgr2=0x7fd9600b8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:53.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 shutdown_connections 2024-06-29T15:58:53.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.334+0000 7fd968218640 1 -- 172.21.15.59:0/3323333863 wait complete. 2024-06-29T15:58:53.346 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:58:53.347 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.469+0000 7efd7892a640 1 -- 172.21.15.59:0/1407767570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 msgr2=0x7efd7415f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.469+0000 7efd7892a640 1 --2- 172.21.15.59:0/1407767570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 0x7efd7415f240 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009830 tx=0x7efd6c02a510 comp rx=0 tx=0).stop 2024-06-29T15:58:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.469+0000 7efd7892a640 1 -- 172.21.15.59:0/1407767570 shutdown_connections 2024-06-29T15:58:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.469+0000 7efd7892a640 1 --2- 172.21.15.59:0/1407767570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd741444c0 0x7efd7406d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.469+0000 7efd7892a640 1 --2- 172.21.15.59:0/1407767570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 0x7efd7415f240 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.469+0000 7efd7892a640 1 -- 172.21.15.59:0/1407767570 >> 172.21.15.59:0/1407767570 conn(0x7efd740b24c0 msgr2=0x7efd740cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.469+0000 7efd7892a640 1 -- 172.21.15.59:0/1407767570 shutdown_connections 2024-06-29T15:58:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.469+0000 7efd7892a640 1 -- 172.21.15.59:0/1407767570 wait complete. 2024-06-29T15:58:53.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.471+0000 7efd7892a640 1 Processor -- start 2024-06-29T15:58:53.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.471+0000 7efd7892a640 1 -- start start 2024-06-29T15:58:53.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7892a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7406d830 0x7efd7415a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7892a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 0x7efd741f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7892a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd741444c0 0x7efd74140ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7892a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efd7415b9b0 con 0x7efd741444c0 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7892a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd74074300 con 0x7efd740cd9b0 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7892a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efd7415bb30 con 0x7efd7406d830 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7406d830 0x7efd7415a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd72d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd741444c0 0x7efd74140ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7406d830 0x7efd7415a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:48464/0 (socket says 172.21.15.59:48464) 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd7259c640 1 -- 172.21.15.59:0/3853355217 learned_addr learned my addr 172.21.15.59:0/3853355217 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:53.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd72d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd741444c0 0x7efd74140ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34706/0 (socket says 172.21.15.59:34706) 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.472+0000 7efd71d9b640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 0x7efd741f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd7259c640 1 -- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 msgr2=0x7efd741f8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd7259c640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 0x7efd741f8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd7259c640 1 -- 172.21.15.59:0/3853355217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd741444c0 msgr2=0x7efd74140ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd7259c640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd741444c0 0x7efd74140ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd7259c640 1 -- 172.21.15.59:0/3853355217 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7415ab60 con 0x7efd7406d830 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd71d9b640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 0x7efd741f8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd7259c640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7406d830 0x7efd7415a620 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009950 tx=0x7efd6c033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd6affd640 1 -- 172.21.15.59:0/3853355217 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c031d60 con 0x7efd7406d830 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd6affd640 1 -- 172.21.15.59:0/3853355217 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd6c033e10 con 0x7efd7406d830 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd7413b220 con 0x7efd7406d830 2024-06-29T15:58:53.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd6affd640 1 -- 172.21.15.59:0/3853355217 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c005260 con 0x7efd7406d830 2024-06-29T15:58:53.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.473+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd74138cc0 con 0x7efd7406d830 2024-06-29T15:58:53.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.475+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd7415bf20 con 0x7efd7406d830 2024-06-29T15:58:53.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.475+0000 7efd6affd640 1 -- 172.21.15.59:0/3853355217 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7efd6c005400 con 0x7efd7406d830 2024-06-29T15:58:53.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.476+0000 7efd6affd640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7efd54076480 0x7efd54078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.476+0000 7efd6affd640 1 -- 172.21.15.59:0/3853355217 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7efd6c0b9350 con 0x7efd7406d830 2024-06-29T15:58:53.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.476+0000 7efd71d9b640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7efd54076480 0x7efd54078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.476+0000 7efd71d9b640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7efd54076480 0x7efd54078990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0097a0 tx=0x7efd5c009260 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:53.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.480+0000 7efd6affd640 1 -- 172.21.15.59:0/3853355217 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd7415ab60 con 0x7efd7406d830 2024-06-29T15:58:53.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.715+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd740689b0 con 0x7efd7406d830 2024-06-29T15:58:53.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.716+0000 7efd6affd640 1 -- 172.21.15.59:0/3853355217 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6568 (secure 0 0 0) 0x7efd6c03f030 con 0x7efd7406d830 2024-06-29T15:58:53.717 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:53.717 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":137,"btime":"2024-06-29T15:58:37:645635+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:20.935697+0000","modified":"2024-06-29T15:58:23.956772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/177893499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":177893499},{"type":"v1","addr":"172.21.15.153:6837","nonce":177893499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12318,"qdb_cluster":[12318]},"id":15},{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:58:53.717 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 137 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7efd54076480 msgr2=0x7efd54078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7efd54076480 0x7efd54078990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0097a0 tx=0x7efd5c009260 comp rx=0 tx=0).stop 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7406d830 msgr2=0x7efd7415a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7406d830 0x7efd7415a620 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009950 tx=0x7efd6c033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 shutdown_connections 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7efd54076480 0x7efd54078990 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd741444c0 0x7efd74140ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd740cd9b0 0x7efd741f8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 --2- 172.21.15.59:0/3853355217 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd7406d830 0x7efd7415a620 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 >> 172.21.15.59:0/3853355217 conn(0x7efd740b24c0 msgr2=0x7efd740731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:53.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 shutdown_connections 2024-06-29T15:58:53.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.718+0000 7efd7892a640 1 -- 172.21.15.59:0/3853355217 wait complete. 2024-06-29T15:58:53.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:53.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.847+0000 7f8243303640 1 -- 172.21.15.59:0/2658748007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c0d23c0 msgr2=0x7f823c0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.847+0000 7f8243303640 1 --2- 172.21.15.59:0/2658748007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c0d23c0 0x7f823c0b3990 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f8238005fd0 tx=0x7f823802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.847+0000 7f8243303640 1 -- 172.21.15.59:0/2658748007 shutdown_connections 2024-06-29T15:58:53.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.847+0000 7f8243303640 1 --2- 172.21.15.59:0/2658748007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c0d23c0 0x7f823c0b3990 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.847+0000 7f8243303640 1 -- 172.21.15.59:0/2658748007 >> 172.21.15.59:0/2658748007 conn(0x7f823c0d3ee0 msgr2=0x7f823c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:53.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.847+0000 7f8243303640 1 -- 172.21.15.59:0/2658748007 shutdown_connections 2024-06-29T15:58:53.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.847+0000 7f8243303640 1 -- 172.21.15.59:0/2658748007 wait complete. 2024-06-29T15:58:53.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.849+0000 7f8243303640 1 Processor -- start 2024-06-29T15:58:53.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.849+0000 7f8243303640 1 -- start start 2024-06-29T15:58:53.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f8243303640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c0b3ed0 0x7f823c14c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f8243303640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f823c0d23c0 0x7f823c14b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f8243303640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c15efe0 0x7f823c14cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f8243303640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f823c160450 con 0x7f823c15efe0 2024-06-29T15:58:53.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f8243303640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f823c1602d0 con 0x7f823c0b3ed0 2024-06-29T15:58:53.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f8243303640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f823c1605d0 con 0x7f823c0d23c0 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f82418a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c15efe0 0x7f823c14cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f82418a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c15efe0 0x7f823c14cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34730/0 (socket says 172.21.15.59:34730) 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f82418a0640 1 -- 172.21.15.59:0/2476003374 learned_addr learned my addr 172.21.15.59:0/2476003374 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f824109f640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c0b3ed0 0x7f823c14c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.850+0000 7f824089e640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f823c0d23c0 0x7f823c14b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f82418a0640 1 -- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f823c0d23c0 msgr2=0x7f823c14b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f82418a0640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f823c0d23c0 0x7f823c14b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f82418a0640 1 -- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c0b3ed0 msgr2=0x7f823c14c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f82418a0640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c0b3ed0 0x7f823c14c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f82418a0640 1 -- 172.21.15.59:0/2476003374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823c14d3c0 con 0x7f823c15efe0 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f82418a0640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c15efe0 0x7f823c14cac0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f823400b960 tx=0x7f823400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f8231ffb640 1 -- 172.21.15.59:0/2476003374 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8234019070 con 0x7f823c15efe0 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f8231ffb640 1 -- 172.21.15.59:0/2476003374 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f823400cc10 con 0x7f823c15efe0 2024-06-29T15:58:53.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823c14d6a0 con 0x7f823c15efe0 2024-06-29T15:58:53.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f8231ffb640 1 -- 172.21.15.59:0/2476003374 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8234014440 con 0x7f823c15efe0 2024-06-29T15:58:53.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.851+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f823c14dc60 con 0x7f823c15efe0 2024-06-29T15:58:53.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.852+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f823c0b3990 con 0x7f823c15efe0 2024-06-29T15:58:53.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.853+0000 7f8231ffb640 1 -- 172.21.15.59:0/2476003374 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7f823401d3f0 con 0x7f823c15efe0 2024-06-29T15:58:53.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.853+0000 7f8231ffb640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f82200764b0 0x7f82200789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.853+0000 7f8231ffb640 1 -- 172.21.15.59:0/2476003374 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7f8234097d70 con 0x7f823c15efe0 2024-06-29T15:58:53.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.854+0000 7f824109f640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f82200764b0 0x7f82200789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.854+0000 7f824109f640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f82200764b0 0x7f82200789c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8238033000 tx=0x7f8238004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:53.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:53.858+0000 7f8231ffb640 1 -- 172.21.15.59:0/2476003374 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8234060630 con 0x7f823c15efe0 2024-06-29T15:58:54.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.089+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f823c0c7370 con 0x7f823c15efe0 2024-06-29T15:58:54.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.090+0000 7f8231ffb640 1 -- 172.21.15.59:0/2476003374 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6568 (secure 0 0 0) 0x7f8234064930 con 0x7f823c15efe0 2024-06-29T15:58:54.090 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:54.091 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":137,"btime":"2024-06-29T15:58:37:645635+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:20.935697+0000","modified":"2024-06-29T15:58:23.956772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6837/177893499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":177893499},{"type":"v1","addr":"172.21.15.153:6837","nonce":177893499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12318,"qdb_cluster":[12318]},"id":15},{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:58:54.091 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 137 2024-06-29T15:58:54.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f82200764b0 msgr2=0x7f82200789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f82200764b0 0x7f82200789c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8238033000 tx=0x7f8238004730 comp rx=0 tx=0).stop 2024-06-29T15:58:54.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c15efe0 msgr2=0x7f823c14cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c15efe0 0x7f823c14cac0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f823400b960 tx=0x7f823400be20 comp rx=0 tx=0).stop 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 shutdown_connections 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f82200764b0 0x7f82200789c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f823c15efe0 0x7f823c14cac0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f823c0d23c0 0x7f823c14b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 --2- 172.21.15.59:0/2476003374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f823c0b3ed0 0x7f823c14c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 >> 172.21.15.59:0/2476003374 conn(0x7f823c0d3ee0 msgr2=0x7f823c0b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 shutdown_connections 2024-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.092+0000 7f8243303640 1 -- 172.21.15.59:0/2476003374 wait complete. 2024-06-29T15:58:54.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.220+0000 7f757850a640 1 -- 172.21.15.59:0/2814043085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 msgr2=0x7f75700cedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.220+0000 7f757850a640 1 --2- 172.21.15.59:0/2814043085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 0x7f75700cedf0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f757008c120 tx=0x7f756402a5f0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.220+0000 7f757850a640 1 -- 172.21.15.59:0/2814043085 shutdown_connections 2024-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.220+0000 7f757850a640 1 --2- 172.21.15.59:0/2814043085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 0x7f75700cedf0 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.220+0000 7f757850a640 1 --2- 172.21.15.59:0/2814043085 >> v2:172.21.15.153:3300/0 conn(0x7f75700b27c0 0x7f75700b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.220+0000 7f757850a640 1 -- 172.21.15.59:0/2814043085 >> 172.21.15.59:0/2814043085 conn(0x7f75700d3ec0 msgr2=0x7f75700d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.220+0000 7f757850a640 1 -- 172.21.15.59:0/2814043085 shutdown_connections 2024-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.220+0000 7f757850a640 1 -- 172.21.15.59:0/2814043085 wait complete. 2024-06-29T15:58:54.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.222+0000 7f757850a640 1 Processor -- start 2024-06-29T15:58:54.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.222+0000 7f757850a640 1 -- start start 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f757850a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75700b27c0 0x7f757014cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f757850a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 0x7f75700c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f757850a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75700d22a0 0x7f75700c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f757850a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f75700d2dc0 con 0x7f75700ce9e0 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f757850a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75700b4490 con 0x7f75700b27c0 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f757850a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f75700d2f40 con 0x7f75700d22a0 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 0x7f75700c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 0x7f75700c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34744/0 (socket says 172.21.15.59:34744) 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 -- 172.21.15.59:0/779020163 learned_addr learned my addr 172.21.15.59:0/779020163 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f75762a6640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75700b27c0 0x7f757014cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7576aa7640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75700d22a0 0x7f75700c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 -- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75700d22a0 msgr2=0x7f75700c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75700d22a0 0x7f75700c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 -- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75700b27c0 msgr2=0x7f757014cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75700b27c0 0x7f757014cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 -- 172.21.15.59:0/779020163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757014d480 con 0x7f75700ce9e0 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7576aa7640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75700d22a0 0x7f75700c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f7575aa5640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 0x7f75700c75d0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f75640099c0 tx=0x7f75640039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.224+0000 7f756effd640 1 -- 172.21.15.59:0/779020163 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756403e030 con 0x7f75700ce9e0 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.225+0000 7f756effd640 1 -- 172.21.15.59:0/779020163 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75640047a0 con 0x7f75700ce9e0 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.225+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7570157420 con 0x7f75700ce9e0 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.225+0000 7f756effd640 1 -- 172.21.15.59:0/779020163 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7564035040 con 0x7f75700ce9e0 2024-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.225+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75701579e0 con 0x7f75700ce9e0 2024-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.225+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75700d3430 con 0x7f75700ce9e0 2024-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.226+0000 7f756effd640 1 -- 172.21.15.59:0/779020163 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7f7564041030 con 0x7f75700ce9e0 2024-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.227+0000 7f756effd640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f755c0764b0 0x7f755c0789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.227+0000 7f756effd640 1 -- 172.21.15.59:0/779020163 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7f75640b8d60 con 0x7f75700ce9e0 2024-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.227+0000 7f75762a6640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f755c0764b0 0x7f755c0789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.228+0000 7f75762a6640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f755c0764b0 0x7f755c0789c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7560005fd0 tx=0x7f75600095e0 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.231+0000 7f756effd640 1 -- 172.21.15.59:0/779020163 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f75640815d0 con 0x7f75700ce9e0 2024-06-29T15:58:54.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.423+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75701571b0 con 0x7f75700ce9e0 2024-06-29T15:58:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.425+0000 7f756effd640 1 -- 172.21.15.59:0/779020163 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) ==== 74+0+22048 (secure 0 0 0) 0x7f75640858d0 con 0x7f75700ce9e0 2024-06-29T15:58:54.426 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:54.426 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":248,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:51.014825+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:19.303469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"241","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:19.884648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6832/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2374956971":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2562681960":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/221125627":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3082371830":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1656484964":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/402968645":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1556406125":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/877026286":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3972699112":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1636302558":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1944500160":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1892969816":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3355566242":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6832/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/2494131047":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1461412440":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1259499350":"2024-06-30T15:58:45.316819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:54.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f755c0764b0 msgr2=0x7f755c0789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f755c0764b0 0x7f755c0789c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7560005fd0 tx=0x7f75600095e0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 msgr2=0x7f75700c75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 0x7f75700c75d0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f75640099c0 tx=0x7f75640039e0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 shutdown_connections 2024-06-29T15:58:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f755c0764b0 0x7f755c0789c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f75700d22a0 0x7f75700c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75700ce9e0 0x7f75700c75d0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 --2- 172.21.15.59:0/779020163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75700b27c0 0x7f757014cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 >> 172.21.15.59:0/779020163 conn(0x7f75700d3ec0 msgr2=0x7f75700d26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 shutdown_connections 2024-06-29T15:58:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.427+0000 7f757850a640 1 -- 172.21.15.59:0/779020163 wait complete. 2024-06-29T15:58:54.439 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:58:54.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.557+0000 7f821e118640 1 -- 172.21.15.59:0/1511068550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 msgr2=0x7f8218162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.557+0000 7f821e118640 1 --2- 172.21.15.59:0/1511068550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 0x7f8218162c10 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f8210005fd0 tx=0x7f821002aa90 comp rx=0 tx=0).stop 2024-06-29T15:58:54.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.557+0000 7f821e118640 1 -- 172.21.15.59:0/1511068550 shutdown_connections 2024-06-29T15:58:54.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.557+0000 7f821e118640 1 --2- 172.21.15.59:0/1511068550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 0x7f8218162c10 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.557+0000 7f821e118640 1 --2- 172.21.15.59:0/1511068550 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82180ce950 0x7f82181622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.557+0000 7f821e118640 1 -- 172.21.15.59:0/1511068550 >> 172.21.15.59:0/1511068550 conn(0x7f82180b5610 msgr2=0x7f82180b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.557+0000 7f821e118640 1 -- 172.21.15.59:0/1511068550 shutdown_connections 2024-06-29T15:58:54.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.557+0000 7f821e118640 1 -- 172.21.15.59:0/1511068550 wait complete. 2024-06-29T15:58:54.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.559+0000 7f821e118640 1 Processor -- start 2024-06-29T15:58:54.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.559+0000 7f821e118640 1 -- start start 2024-06-29T15:58:54.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.560+0000 7f821e118640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821808c190 0x7f8218152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.560+0000 7f821e118640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82180ce950 0x7f8218155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.560+0000 7f821e118640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 0x7f821815c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.560+0000 7f821e118640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f82180b7fa0 con 0x7f8218162800 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.560+0000 7f821e118640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82180b7e20 con 0x7f821808c190 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.560+0000 7f821e118640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f82180b8120 con 0x7f82180ce950 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.560+0000 7f8217fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 0x7f821815c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8217fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 0x7f821815c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34772/0 (socket says 172.21.15.59:34772) 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8217fff640 1 -- 172.21.15.59:0/531398789 learned_addr learned my addr 172.21.15.59:0/531398789 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f82177fe640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821808c190 0x7f8218152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f820effd640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82180ce950 0x7f8218155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8217fff640 1 -- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82180ce950 msgr2=0x7f8218155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8217fff640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82180ce950 0x7f8218155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8217fff640 1 -- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821808c190 msgr2=0x7f8218152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8217fff640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821808c190 0x7f8218152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8217fff640 1 -- 172.21.15.59:0/531398789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82181560b0 con 0x7f8218162800 2024-06-29T15:58:54.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f82177fe640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821808c190 0x7f8218152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:54.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8217fff640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 0x7f821815c4c0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f8210009d90 tx=0x7f8210003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8214ff9640 1 -- 172.21.15.59:0/531398789 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210044070 con 0x7f8218162800 2024-06-29T15:58:54.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8214ff9640 1 -- 172.21.15.59:0/531398789 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8210004250 con 0x7f8218162800 2024-06-29T15:58:54.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8218157cc0 con 0x7f8218162800 2024-06-29T15:58:54.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f8214ff9640 1 -- 172.21.15.59:0/531398789 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210004660 con 0x7f8218162800 2024-06-29T15:58:54.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.561+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821815f1d0 con 0x7f8218162800 2024-06-29T15:58:54.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.562+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82181622c0 con 0x7f8218162800 2024-06-29T15:58:54.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.563+0000 7f8214ff9640 1 -- 172.21.15.59:0/531398789 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7f8210047070 con 0x7f8218162800 2024-06-29T15:58:54.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.564+0000 7f8214ff9640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f81f8076480 0x7f81f8078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.564+0000 7f8214ff9640 1 -- 172.21.15.59:0/531398789 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 7881+0+0 (secure 0 0 0) 0x7f82100b9290 con 0x7f8218162800 2024-06-29T15:58:54.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.564+0000 7f82177fe640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f81f8076480 0x7f81f8078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.564+0000 7f82177fe640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f81f8076480 0x7f81f8078990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8208007fd0 tx=0x7f8208009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.568+0000 7f8214ff9640 1 -- 172.21.15.59:0/531398789 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8218157cc0 con 0x7f8218162800 2024-06-29T15:58:54.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.798+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f82180ca1c0 con 0x7f8218162800 2024-06-29T15:58:54.799 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:58:54.799+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:58:54.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.801+0000 7f8214ff9640 1 -- 172.21.15.59:0/531398789 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v138) ==== 161+0+0 (secure 0 0 0) 0x7f8210032030 con 0x7f8218162800 2024-06-29T15:58:54.802 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:58:54.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f81f8076480 msgr2=0x7f81f8078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f81f8076480 0x7f81f8078990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8208007fd0 tx=0x7f8208009040 comp rx=0 tx=0).stop 2024-06-29T15:58:54.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 msgr2=0x7f821815c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 0x7f821815c4c0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f8210009d90 tx=0x7f8210003ce0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 shutdown_connections 2024-06-29T15:58:54.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f81f8076480 0x7f81f8078990 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8218162800 0x7f821815c4c0 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82180ce950 0x7f8218155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 --2- 172.21.15.59:0/531398789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f821808c190 0x7f8218152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 >> 172.21.15.59:0/531398789 conn(0x7f82180b5610 msgr2=0x7f82180c8010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 shutdown_connections 2024-06-29T15:58:54.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.803+0000 7f821e118640 1 -- 172.21.15.59:0/531398789 wait complete. 2024-06-29T15:58:54.809 INFO:tasks.ceph.mds.d.smithi153.stderr: -4613> 2024-06-29T15:58:23.523+0000 7efc5f023640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:54.816 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:58:54.865 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:58:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.935+0000 7fca253b9640 1 -- 172.21.15.59:0/2872480399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200b2980 msgr2=0x7fca200d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.935+0000 7fca253b9640 1 --2- 172.21.15.59:0/2872480399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200b2980 0x7fca200d27a0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7fca08009830 tx=0x7fca0802a480 comp rx=0 tx=0).stop 2024-06-29T15:58:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.936+0000 7fca253b9640 1 -- 172.21.15.59:0/2872480399 shutdown_connections 2024-06-29T15:58:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.936+0000 7fca253b9640 1 --2- 172.21.15.59:0/2872480399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200b2980 0x7fca200d27a0 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.936+0000 7fca253b9640 1 --2- 172.21.15.59:0/2872480399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca200d1c30 0x7fca200b6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.936+0000 7fca253b9640 1 -- 172.21.15.59:0/2872480399 >> 172.21.15.59:0/2872480399 conn(0x7fca200d0f90 msgr2=0x7fca200d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.936+0000 7fca253b9640 1 -- 172.21.15.59:0/2872480399 shutdown_connections 2024-06-29T15:58:54.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.936+0000 7fca253b9640 1 -- 172.21.15.59:0/2872480399 wait complete. 2024-06-29T15:58:54.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.937+0000 7fca253b9640 1 Processor -- start 2024-06-29T15:58:54.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.937+0000 7fca253b9640 1 -- start start 2024-06-29T15:58:54.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca253b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca200b2980 0x7fca200cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca253b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200d1c30 0x7fca2014fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca253b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca200d2ce0 0x7fca200c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca253b9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fca200b4780 con 0x7fca200d1c30 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca253b9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fca200d3300 con 0x7fca200d2ce0 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca253b9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fca200b4900 con 0x7fca200b2980 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200d1c30 0x7fca2014fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200d1c30 0x7fca2014fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34790/0 (socket says 172.21.15.59:34790) 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 -- 172.21.15.59:0/1877119670 learned_addr learned my addr 172.21.15.59:0/1877119670 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca1effd640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca200b2980 0x7fca200cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca1f7fe640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca200d2ce0 0x7fca200c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 -- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca200b2980 msgr2=0x7fca200cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca200b2980 0x7fca200cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 -- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca200d2ce0 msgr2=0x7fca200c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca200d2ce0 0x7fca200c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 -- 172.21.15.59:0/1877119670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2015af30 con 0x7fca200d1c30 2024-06-29T15:58:54.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.939+0000 7fca17fff640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200d1c30 0x7fca2014fca0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7fca08009950 tx=0x7fca08033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.940+0000 7fca177fe640 1 -- 172.21.15.59:0/1877119670 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08031d60 con 0x7fca200d1c30 2024-06-29T15:58:54.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.940+0000 7fca177fe640 1 -- 172.21.15.59:0/1877119670 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca08033e10 con 0x7fca200d1c30 2024-06-29T15:58:54.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.940+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2015b210 con 0x7fca200d1c30 2024-06-29T15:58:54.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.940+0000 7fca177fe640 1 -- 172.21.15.59:0/1877119670 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08033310 con 0x7fca200d1c30 2024-06-29T15:58:54.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.940+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca2015b7d0 con 0x7fca200d1c30 2024-06-29T15:58:54.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.941+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca200d3f50 con 0x7fca200d1c30 2024-06-29T15:58:54.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.941+0000 7fca177fe640 1 -- 172.21.15.59:0/1877119670 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7fca080334b0 con 0x7fca200d1c30 2024-06-29T15:58:54.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.942+0000 7fca177fe640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fca00076480 0x7fca00078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.942+0000 7fca177fe640 1 -- 172.21.15.59:0/1877119670 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 7967+0+0 (secure 0 0 0) 0x7fca080b9790 con 0x7fca200d1c30 2024-06-29T15:58:54.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.942+0000 7fca1effd640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fca00076480 0x7fca00078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.943+0000 7fca1effd640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fca00076480 0x7fca00078990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca10009b30 tx=0x7fca10005f90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:54.946+0000 7fca177fe640 1 -- 172.21.15.59:0/1877119670 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fca200d3f50 con 0x7fca200d1c30 2024-06-29T15:58:55.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.185+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fca200d37c0 con 0x7fca200d1c30 2024-06-29T15:58:55.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.810+0000 7fca177fe640 1 -- 172.21.15.59:0/1877119670 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v139) ==== 104+0+0 (secure 0 0 0) 0x7fca08088020 con 0x7fca200d1c30 2024-06-29T15:58:55.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.814+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fca00076480 msgr2=0x7fca00078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.814+0000 7fca253b9640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fca00076480 0x7fca00078990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fca10009b30 tx=0x7fca10005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:55.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.814+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200d1c30 msgr2=0x7fca2014fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.814+0000 7fca253b9640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200d1c30 0x7fca2014fca0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7fca08009950 tx=0x7fca08033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:55.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.815+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 shutdown_connections 2024-06-29T15:58:55.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.815+0000 7fca253b9640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fca00076480 0x7fca00078990 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.815+0000 7fca253b9640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca200d2ce0 0x7fca200c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.815+0000 7fca253b9640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca200d1c30 0x7fca2014fca0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.815+0000 7fca253b9640 1 --2- 172.21.15.59:0/1877119670 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fca200b2980 0x7fca200cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.815+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 >> 172.21.15.59:0/1877119670 conn(0x7fca200d0f90 msgr2=0x7fca200b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:55.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.815+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 shutdown_connections 2024-06-29T15:58:55.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.815+0000 7fca253b9640 1 -- 172.21.15.59:0/1877119670 wait complete. 2024-06-29T15:58:55.831 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:58:55.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.959+0000 7fd746881640 1 -- 172.21.15.59:0/3855160973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400cec20 msgr2=0x7fd7400cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.959+0000 7fd746881640 1 --2- 172.21.15.59:0/3855160973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400cec20 0x7fd7400cf700 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7fd738005fd0 tx=0x7fd73802a870 comp rx=0 tx=0).stop 2024-06-29T15:58:55.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.959+0000 7fd746881640 1 -- 172.21.15.59:0/3855160973 shutdown_connections 2024-06-29T15:58:55.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.959+0000 7fd746881640 1 --2- 172.21.15.59:0/3855160973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7400cec20 0x7fd7400cf700 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.959+0000 7fd746881640 1 --2- 172.21.15.59:0/3855160973 >> v2:172.21.15.153:3300/0 conn(0x7fd7400d1ea0 0x7fd7400b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.959+0000 7fd746881640 1 -- 172.21.15.59:0/3855160973 >> 172.21.15.59:0/3855160973 conn(0x7fd740073e10 msgr2=0x7fd740074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:55.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.959+0000 7fd746881640 1 -- 172.21.15.59:0/3855160973 shutdown_connections 2024-06-29T15:58:55.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.960+0000 7fd746881640 1 -- 172.21.15.59:0/3855160973 wait complete. 2024-06-29T15:58:55.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.961+0000 7fd746881640 1 Processor -- start 2024-06-29T15:58:55.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.962+0000 7fd746881640 1 -- start start 2024-06-29T15:58:55.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd746881640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400cec20 0x7fd7401439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd746881640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7401423c0 0x7fd740145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd746881640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7401571c0 0x7fd7400b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd746881640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd7400cfc40 con 0x7fd7401571c0 2024-06-29T15:58:55.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd746881640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd74015c4d0 con 0x7fd7401423c0 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd746881640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd7400cfdc0 con 0x7fd7400cec20 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd744e1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7401571c0 0x7fd7400b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd744e1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7401571c0 0x7fd7400b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34806/0 (socket says 172.21.15.59:34806) 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd744e1e640 1 -- 172.21.15.59:0/1778429012 learned_addr learned my addr 172.21.15.59:0/1778429012 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd73f7fe640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7401423c0 0x7fd740145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd73ffff640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400cec20 0x7fd7401439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd744e1e640 1 -- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400cec20 msgr2=0x7fd7401439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd744e1e640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400cec20 0x7fd7401439a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd744e1e640 1 -- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7401423c0 msgr2=0x7fd740145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd744e1e640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7401423c0 0x7fd740145540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd744e1e640 1 -- 172.21.15.59:0/1778429012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7401441f0 con 0x7fd7401571c0 2024-06-29T15:58:55.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd73f7fe640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7401423c0 0x7fd740145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.963+0000 7fd73ffff640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400cec20 0x7fd7401439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.964+0000 7fd744e1e640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7401571c0 0x7fd7400b37b0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7fd738005fd0 tx=0x7fd738033ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.964+0000 7fd73cff9640 1 -- 172.21.15.59:0/1778429012 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738042020 con 0x7fd7401571c0 2024-06-29T15:58:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.964+0000 7fd73cff9640 1 -- 172.21.15.59:0/1778429012 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd738033db0 con 0x7fd7401571c0 2024-06-29T15:58:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.964+0000 7fd73cff9640 1 -- 172.21.15.59:0/1778429012 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738037040 con 0x7fd7401571c0 2024-06-29T15:58:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.964+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7401444d0 con 0x7fd7401571c0 2024-06-29T15:58:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.964+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd740153710 con 0x7fd7401571c0 2024-06-29T15:58:55.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.965+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7400d02c0 con 0x7fd7401571c0 2024-06-29T15:58:55.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.966+0000 7fd73cff9640 1 -- 172.21.15.59:0/1778429012 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fd738045020 con 0x7fd7401571c0 2024-06-29T15:58:55.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.966+0000 7fd73cff9640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd72c076510 0x7fd72c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.966+0000 7fd73cff9640 1 -- 172.21.15.59:0/1778429012 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 7967+0+0 (secure 0 0 0) 0x7fd7380b9580 con 0x7fd7401571c0 2024-06-29T15:58:55.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.967+0000 7fd73ffff640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd72c076510 0x7fd72c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.967+0000 7fd73ffff640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd72c076510 0x7fd72c078a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd730000f60 tx=0x7fd730009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:55.971+0000 7fd73cff9640 1 -- 172.21.15.59:0/1778429012 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd73804c030 con 0x7fd7401571c0 2024-06-29T15:58:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.161+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd740143f80 con 0x7fd7401571c0 2024-06-29T15:58:56.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.811+0000 7fd73cff9640 1 -- 172.21.15.59:0/1778429012 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v250) ==== 181+0+0 (secure 0 0 0) 0x7fd738082110 con 0x7fd7401571c0 2024-06-29T15:58:56.813 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:58:56.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd72c076510 msgr2=0x7fd72c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd72c076510 0x7fd72c078a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd730000f60 tx=0x7fd730009040 comp rx=0 tx=0).stop 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7401571c0 msgr2=0x7fd7400b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7401571c0 0x7fd7400b37b0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7fd738005fd0 tx=0x7fd738033ad0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 shutdown_connections 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd72c076510 0x7fd72c078a20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7401571c0 0x7fd7400b37b0 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7401423c0 0x7fd740145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 --2- 172.21.15.59:0/1778429012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd7400cec20 0x7fd7401439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.814+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 >> 172.21.15.59:0/1778429012 conn(0x7fd740073e10 msgr2=0x7fd74013fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.815+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 shutdown_connections 2024-06-29T15:58:56.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.815+0000 7fd746881640 1 -- 172.21.15.59:0/1778429012 wait complete. 2024-06-29T15:58:56.831 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:58:56.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.960+0000 7f2e14d24640 1 -- 172.21.15.59:0/1161197619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 msgr2=0x7f2dfc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.960+0000 7f2e14d24640 1 --2- 172.21.15.59:0/1161197619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 0x7f2dfc003e70 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c290 tx=0x7f2dfc005830 comp rx=0 tx=0).stop 2024-06-29T15:58:56.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.960+0000 7f2e0ed9d640 1 -- 172.21.15.59:0/1161197619 reap_dead start 2024-06-29T15:58:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.960+0000 7f2e14d24640 1 -- 172.21.15.59:0/1161197619 shutdown_connections 2024-06-29T15:58:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.960+0000 7f2e14d24640 1 -- 172.21.15.59:0/1161197619 >> 172.21.15.59:0/1161197619 conn(0x7f2e100b7690 msgr2=0x7f2e10073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.960+0000 7f2e14d24640 1 -- 172.21.15.59:0/1161197619 shutdown_connections 2024-06-29T15:58:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.960+0000 7f2e14d24640 1 -- 172.21.15.59:0/1161197619 wait complete. 2024-06-29T15:58:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.962+0000 7f2e14d24640 1 Processor -- start 2024-06-29T15:58:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.962+0000 7f2e14d24640 1 -- start start 2024-06-29T15:58:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e14d24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 0x7f2e10154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e14d24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e100b3b80 0x7f2e100b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e14d24640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e100aaaf0 0x7f2e100aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e14d24640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2e100b4f70 con 0x7f2dfc003a00 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e14d24640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e100b4df0 con 0x7f2e100b3b80 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e14d24640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2e100b50f0 con 0x7f2e100aaaf0 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e0e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 0x7f2e10154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e0e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 0x7f2e10154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34818/0 (socket says 172.21.15.59:34818) 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.963+0000 7f2e0e59c640 1 -- 172.21.15.59:0/67541336 learned_addr learned my addr 172.21.15.59:0/67541336 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0ed9d640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e100aaaf0 0x7f2e100aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0dd9b640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e100b3b80 0x7f2e100b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0e59c640 1 -- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e100aaaf0 msgr2=0x7f2e100aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0e59c640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e100aaaf0 0x7f2e100aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0e59c640 1 -- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e100b3b80 msgr2=0x7f2e100b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0e59c640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e100b3b80 0x7f2e100b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0e59c640 1 -- 172.21.15.59:0/67541336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e100ab440 con 0x7f2dfc003a00 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0dd9b640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e100b3b80 0x7f2e100b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e0e59c640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 0x7f2e10154550 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c380 tx=0x7f2e00006ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e06ffd640 1 -- 172.21.15.59:0/67541336 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00031040 con 0x7f2dfc003a00 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e06ffd640 1 -- 172.21.15.59:0/67541336 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e000035f0 con 0x7f2dfc003a00 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e100b2170 con 0x7f2dfc003a00 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e06ffd640 1 -- 172.21.15.59:0/67541336 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00003a40 con 0x7f2dfc003a00 2024-06-29T15:58:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.964+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e100b2730 con 0x7f2dfc003a00 2024-06-29T15:58:56.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.965+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e101596a0 con 0x7f2dfc003a00 2024-06-29T15:58:56.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.966+0000 7f2e06ffd640 1 -- 172.21.15.59:0/67541336 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f2e00034070 con 0x7f2dfc003a00 2024-06-29T15:58:56.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.967+0000 7f2e06ffd640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2df0076510 0x7f2df0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.967+0000 7f2e06ffd640 1 -- 172.21.15.59:0/67541336 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 7529+0+0 (secure 0 0 0) 0x7f2e000b86c0 con 0x7f2dfc003a00 2024-06-29T15:58:56.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.967+0000 7f2e0dd9b640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2df0076510 0x7f2df0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.968+0000 7f2e0dd9b640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2df0076510 0x7f2df0078a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2df8005eb0 tx=0x7f2df8005f90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:56.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:56.971+0000 7f2e06ffd640 1 -- 172.21.15.59:0/67541336 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2e100ab440 con 0x7f2dfc003a00 2024-06-29T15:58:57.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.164+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2e100d1f60 con 0x7f2dfc003a00 2024-06-29T15:58:57.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.817+0000 7f2e06ffd640 1 -- 172.21.15.59:0/67541336 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v251) ==== 169+0+0 (secure 0 0 0) 0x7f2e000813e0 con 0x7f2dfc003a00 2024-06-29T15:58:57.818 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T15:58:57.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.820+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2df0076510 msgr2=0x7f2df0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.820+0000 7f2e14d24640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2df0076510 0x7f2df0078a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2df8005eb0 tx=0x7f2df8005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:57.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.820+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 msgr2=0x7f2e10154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.820+0000 7f2e14d24640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 0x7f2e10154550 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c380 tx=0x7f2e00006ce0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.822+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 shutdown_connections 2024-06-29T15:58:57.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.822+0000 7f2e14d24640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2df0076510 0x7f2df0078a20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.822+0000 7f2e14d24640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2e100aaaf0 0x7f2e100aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.822+0000 7f2e14d24640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e100b3b80 0x7f2e100b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.822+0000 7f2e14d24640 1 --2- 172.21.15.59:0/67541336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc003a00 0x7f2e10154550 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.822+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 >> 172.21.15.59:0/67541336 conn(0x7f2e100b7690 msgr2=0x7f2e1015ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.823+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 shutdown_connections 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.823+0000 7f2e14d24640 1 -- 172.21.15.59:0/67541336 wait complete. 2024-06-29T15:58:57.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:57.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.969+0000 7fe56464e640 1 -- 172.21.15.59:0/3940872123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 msgr2=0x7fe548003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.969+0000 7fe56464e640 1 --2- 172.21.15.59:0/3940872123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 0x7fe548003f10 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7fe55800baf0 tx=0x7fe548005830 comp rx=0 tx=0).stop 2024-06-29T15:58:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.969+0000 7fe562beb640 1 -- 172.21.15.59:0/3940872123 reap_dead start 2024-06-29T15:58:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.969+0000 7fe56464e640 1 -- 172.21.15.59:0/3940872123 shutdown_connections 2024-06-29T15:58:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.969+0000 7fe56464e640 1 -- 172.21.15.59:0/3940872123 >> 172.21.15.59:0/3940872123 conn(0x7fe55c0d3ee0 msgr2=0x7fe55c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.969+0000 7fe56464e640 1 -- 172.21.15.59:0/3940872123 shutdown_connections 2024-06-29T15:58:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.969+0000 7fe56464e640 1 -- 172.21.15.59:0/3940872123 wait complete. 2024-06-29T15:58:57.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.971+0000 7fe56464e640 1 Processor -- start 2024-06-29T15:58:57.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.971+0000 7fe56464e640 1 -- start start 2024-06-29T15:58:57.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe56464e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 0x7fe55c1eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe56464e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe55c15ace0 0x7fe55c14aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe56464e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe55c15cab0 0x7fe55c15cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe56464e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c0b3d20 con 0x7fe548003aa0 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe56464e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c0b3ba0 con 0x7fe55c15ace0 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe56464e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe55c1600c0 con 0x7fe55c15cab0 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe5623ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 0x7fe55c1eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe5623ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 0x7fe55c1eb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34836/0 (socket says 172.21.15.59:34836) 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe5623ea640 1 -- 172.21.15.59:0/443279979 learned_addr learned my addr 172.21.15.59:0/443279979 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe562beb640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe55c15cab0 0x7fe55c15cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.972+0000 7fe561be9640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe55c15ace0 0x7fe55c14aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe5623ea640 1 -- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe55c15cab0 msgr2=0x7fe55c15cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe5623ea640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe55c15cab0 0x7fe55c15cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe5623ea640 1 -- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe55c15ace0 msgr2=0x7fe55c14aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe5623ea640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe55c15ace0 0x7fe55c14aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe5623ea640 1 -- 172.21.15.59:0/443279979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55c14b410 con 0x7fe548003aa0 2024-06-29T15:58:57.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe5623ea640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 0x7fe55c1eb560 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fe55800bc10 tx=0x7fe558031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:57.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe552ffd640 1 -- 172.21.15.59:0/443279979 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558040070 con 0x7fe548003aa0 2024-06-29T15:58:57.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe552ffd640 1 -- 172.21.15.59:0/443279979 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe558005310 con 0x7fe548003aa0 2024-06-29T15:58:57.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55c15d400 con 0x7fe548003aa0 2024-06-29T15:58:57.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe552ffd640 1 -- 172.21.15.59:0/443279979 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558035040 con 0x7fe548003aa0 2024-06-29T15:58:57.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.973+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe55c15b660 con 0x7fe548003aa0 2024-06-29T15:58:57.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.974+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe55c160d90 con 0x7fe548003aa0 2024-06-29T15:58:57.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.975+0000 7fe552ffd640 1 -- 172.21.15.59:0/443279979 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fe558038020 con 0x7fe548003aa0 2024-06-29T15:58:57.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.975+0000 7fe552ffd640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fe540076510 0x7fe540078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.976+0000 7fe552ffd640 1 -- 172.21.15.59:0/443279979 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7fe5580b8a40 con 0x7fe548003aa0 2024-06-29T15:58:57.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.976+0000 7fe561be9640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fe540076510 0x7fe540078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.976+0000 7fe561be9640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fe540076510 0x7fe540078a20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe55c0b7dc0 tx=0x7fe54c006d70 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:57.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:57.980+0000 7fe552ffd640 1 -- 172.21.15.59:0/443279979 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe55c15d400 con 0x7fe548003aa0 2024-06-29T15:58:58.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.170+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe55c0d2950 con 0x7fe548003aa0 2024-06-29T15:58:58.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.171+0000 7fe552ffd640 1 -- 172.21.15.59:0/443279979 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) ==== 74+0+18400 (secure 0 0 0) 0x7fe55804a060 con 0x7fe548003aa0 2024-06-29T15:58:58.172 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:58.173 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":251,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:57.813620+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6834/177893499":"2024-06-30T15:58:54.800279+0000","172.21.15.153:6832/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2374956971":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2562681960":"2024-06-30T15:58:51.014782+0000","172.21.15.153:6837/177893499":"2024-06-30T15:58:54.800279+0000","172.21.15.153:0/221125627":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3082371830":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1656484964":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/402968645":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1556406125":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/877026286":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3972699112":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1636302558":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1944500160":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1892969816":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3355566242":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6832/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/2494131047":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1461412440":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1259499350":"2024-06-30T15:58:45.316819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fe540076510 msgr2=0x7fe540078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fe540076510 0x7fe540078a20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe55c0b7dc0 tx=0x7fe54c006d70 comp rx=0 tx=0).stop 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 msgr2=0x7fe55c1eb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 0x7fe55c1eb560 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fe55800bc10 tx=0x7fe558031f90 comp rx=0 tx=0).stop 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 shutdown_connections 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fe540076510 0x7fe540078a20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe55c15cab0 0x7fe55c15cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe55c15ace0 0x7fe55c14aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 --2- 172.21.15.59:0/443279979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe548003aa0 0x7fe55c1eb560 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 >> 172.21.15.59:0/443279979 conn(0x7fe55c0d3ee0 msgr2=0x7fe55c06d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 shutdown_connections 2024-06-29T15:58:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.173+0000 7fe56464e640 1 -- 172.21.15.59:0/443279979 wait complete. 2024-06-29T15:58:58.185 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:58:31.630427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '245', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:58:32.972384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:58.186 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:58.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.305+0000 7f0eb6610640 1 -- 172.21.15.59:0/2952739889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 msgr2=0x7f0eb00b7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.305+0000 7f0eb6610640 1 --2- 172.21.15.59:0/2952739889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 0x7f0eb00b7800 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8005fd0 tx=0x7f0ea802aa90 comp rx=0 tx=0).stop 2024-06-29T15:58:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.305+0000 7f0eb6610640 1 -- 172.21.15.59:0/2952739889 shutdown_connections 2024-06-29T15:58:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.305+0000 7f0eb6610640 1 --2- 172.21.15.59:0/2952739889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 0x7f0eb00b7800 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.305+0000 7f0eb6610640 1 --2- 172.21.15.59:0/2952739889 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb00b3990 0x7f0eb00b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.305+0000 7f0eb6610640 1 -- 172.21.15.59:0/2952739889 >> 172.21.15.59:0/2952739889 conn(0x7f0eb00d3ee0 msgr2=0x7f0eb00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.305+0000 7f0eb6610640 1 -- 172.21.15.59:0/2952739889 shutdown_connections 2024-06-29T15:58:58.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.305+0000 7f0eb6610640 1 -- 172.21.15.59:0/2952739889 wait complete. 2024-06-29T15:58:58.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.307+0000 7f0eb6610640 1 Processor -- start 2024-06-29T15:58:58.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.307+0000 7f0eb6610640 1 -- start start 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eb6610640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb00b3990 0x7f0eb01f8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eb6610640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 0x7f0eb015b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eb6610640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb015e290 0x7f0eb0154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eb6610640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb00b7d40 con 0x7f0eb00b73f0 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eb6610640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb00cea30 con 0x7f0eb015e290 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eb6610640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb00b7ec0 con 0x7f0eb00b3990 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 0x7f0eb015b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 0x7f0eb015b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34842/0 (socket says 172.21.15.59:34842) 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 -- 172.21.15.59:0/1925722453 learned_addr learned my addr 172.21.15.59:0/1925722453 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eb4bad640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb015e290 0x7f0eb0154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaffff640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb00b3990 0x7f0eb01f8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 -- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb00b3990 msgr2=0x7f0eb01f8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb00b3990 0x7f0eb01f8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 -- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb015e290 msgr2=0x7f0eb0154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb015e290 0x7f0eb0154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 -- 172.21.15.59:0/1925722453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb01f9090 con 0x7f0eb00b73f0 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eaf7fe640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 0x7f0eb015b9d0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400e7f0 tx=0x7f0ea400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eacff9640 1 -- 172.21.15.59:0/1925722453 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400eea0 con 0x7f0eb00b73f0 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eacff9640 1 -- 172.21.15.59:0/1925722453 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ea400cdb0 con 0x7f0eb00b73f0 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb00aa7e0 con 0x7f0eb00b73f0 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.309+0000 7f0eacff9640 1 -- 172.21.15.59:0/1925722453 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4010690 con 0x7f0eb00b73f0 2024-06-29T15:58:58.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.310+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb01f5040 con 0x7f0eb00b73f0 2024-06-29T15:58:58.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.311+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb00aa550 con 0x7f0eb00b73f0 2024-06-29T15:58:58.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.311+0000 7f0eacff9640 1 -- 172.21.15.59:0/1925722453 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f0ea4010830 con 0x7f0eb00b73f0 2024-06-29T15:58:58.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.312+0000 7f0eacff9640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0e9c076510 0x7f0e9c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.312+0000 7f0eacff9640 1 -- 172.21.15.59:0/1925722453 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7f0ea4098b00 con 0x7f0eb00b73f0 2024-06-29T15:58:58.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.312+0000 7f0eaffff640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0e9c076510 0x7f0e9c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.313+0000 7f0eaffff640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0e9c076510 0x7f0e9c078a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0001720 tx=0x7f0ea0009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:58.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.316+0000 7f0eacff9640 1 -- 172.21.15.59:0/1925722453 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0eb01f9090 con 0x7f0eb00b73f0 2024-06-29T15:58:58.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.506+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0eb00b8420 con 0x7f0eb00b73f0 2024-06-29T15:58:58.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.507+0000 7f0eacff9640 1 -- 172.21.15.59:0/1925722453 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v251) ==== 93+0+10 (secure 0 0 0) 0x7f0ea40613a0 con 0x7f0eb00b73f0 2024-06-29T15:58:58.508 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:58:58.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0e9c076510 msgr2=0x7f0e9c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0e9c076510 0x7f0e9c078a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0001720 tx=0x7f0ea0009040 comp rx=0 tx=0).stop 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 msgr2=0x7f0eb015b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 0x7f0eb015b9d0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400e7f0 tx=0x7f0ea400ecb0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 shutdown_connections 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0e9c076510 0x7f0e9c078a20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eb015e290 0x7f0eb0154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0eb00b73f0 0x7f0eb015b9d0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 --2- 172.21.15.59:0/1925722453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0eb00b3990 0x7f0eb01f8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 >> 172.21.15.59:0/1925722453 conn(0x7f0eb00d3ee0 msgr2=0x7f0eb00ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 shutdown_connections 2024-06-29T15:58:58.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.509+0000 7f0eb6610640 1 -- 172.21.15.59:0/1925722453 wait complete. 2024-06-29T15:58:58.521 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:58.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.642+0000 7fb1f47ba640 1 -- 172.21.15.59:0/1962313921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec0d23c0 msgr2=0x7fb1ec0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.642+0000 7fb1f47ba640 1 --2- 172.21.15.59:0/1962313921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec0d23c0 0x7fb1ec0b3990 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009830 tx=0x7fb1e802a5c0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.642+0000 7fb1f47ba640 1 -- 172.21.15.59:0/1962313921 shutdown_connections 2024-06-29T15:58:58.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.642+0000 7fb1f47ba640 1 --2- 172.21.15.59:0/1962313921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec0d23c0 0x7fb1ec0b3990 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.643+0000 7fb1f47ba640 1 -- 172.21.15.59:0/1962313921 >> 172.21.15.59:0/1962313921 conn(0x7fb1ec0d3ee0 msgr2=0x7fb1ec0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:58.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.643+0000 7fb1f47ba640 1 -- 172.21.15.59:0/1962313921 shutdown_connections 2024-06-29T15:58:58.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.643+0000 7fb1f47ba640 1 -- 172.21.15.59:0/1962313921 wait complete. 2024-06-29T15:58:58.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.644+0000 7fb1f47ba640 1 Processor -- start 2024-06-29T15:58:58.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.645+0000 7fb1f47ba640 1 -- start start 2024-06-29T15:58:58.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f47ba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1ec0b3ed0 0x7fb1ec14b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f47ba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec0d23c0 0x7fb1ec157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f47ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec15efe0 0x7fb1ec157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f47ba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec160450 con 0x7fb1ec15efe0 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f47ba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec1602d0 con 0x7fb1ec0d23c0 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f47ba640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ec1605d0 con 0x7fb1ec0b3ed0 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f2d57640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec15efe0 0x7fb1ec157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f2d57640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec15efe0 0x7fb1ec157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34850/0 (socket says 172.21.15.59:34850) 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f2d57640 1 -- 172.21.15.59:0/2605639482 learned_addr learned my addr 172.21.15.59:0/2605639482 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f2556640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1ec0b3ed0 0x7fb1ec14b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.646+0000 7fb1f1d55640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec0d23c0 0x7fb1ec157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1f2d57640 1 -- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1ec0b3ed0 msgr2=0x7fb1ec14b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1f2d57640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1ec0b3ed0 0x7fb1ec14b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1f2d57640 1 -- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec0d23c0 msgr2=0x7fb1ec157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1f2d57640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec0d23c0 0x7fb1ec157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1f2d57640 1 -- 172.21.15.59:0/2605639482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec0c51b0 con 0x7fb1ec15efe0 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1f2d57640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec15efe0 0x7fb1ec157740 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400c6b0 tx=0x7fb1e400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1e2ffd640 1 -- 172.21.15.59:0/2605639482 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4003b70 con 0x7fb1ec15efe0 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1e2ffd640 1 -- 172.21.15.59:0/2605639482 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1e4003d10 con 0x7fb1ec15efe0 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec1554e0 con 0x7fb1ec15efe0 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1e2ffd640 1 -- 172.21.15.59:0/2605639482 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4005330 con 0x7fb1ec15efe0 2024-06-29T15:58:58.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.647+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec155aa0 con 0x7fb1ec15efe0 2024-06-29T15:58:58.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.648+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1ec0c5380 con 0x7fb1ec15efe0 2024-06-29T15:58:58.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.649+0000 7fb1e2ffd640 1 -- 172.21.15.59:0/2605639482 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fb1e40054d0 con 0x7fb1ec15efe0 2024-06-29T15:58:58.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.650+0000 7fb1e2ffd640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb1d0076510 0x7fb1d0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.650+0000 7fb1e2ffd640 1 -- 172.21.15.59:0/2605639482 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7fb1e4097f50 con 0x7fb1ec15efe0 2024-06-29T15:58:58.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.650+0000 7fb1f2556640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb1d0076510 0x7fb1d0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.651+0000 7fb1f2556640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb1d0076510 0x7fb1d0078a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8034000 tx=0x7fb1e8004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:58.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.654+0000 7fb1e2ffd640 1 -- 172.21.15.59:0/2605639482 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb1e4060f50 con 0x7fb1ec15efe0 2024-06-29T15:58:58.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.845+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7fb1ec157f50 con 0x7fb1ec15efe0 2024-06-29T15:58:58.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.846+0000 7fb1e2ffd640 1 -- 172.21.15.59:0/2605639482 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v251) ==== 107+0+11 (secure 0 0 0) 0x7fb1e4060850 con 0x7fb1ec15efe0 2024-06-29T15:58:58.847 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb1d0076510 msgr2=0x7fb1d0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb1d0076510 0x7fb1d0078a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8034000 tx=0x7fb1e8004730 comp rx=0 tx=0).stop 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec15efe0 msgr2=0x7fb1ec157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec15efe0 0x7fb1ec157740 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400c6b0 tx=0x7fb1e400cb70 comp rx=0 tx=0).stop 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 shutdown_connections 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb1d0076510 0x7fb1d0078a20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1ec15efe0 0x7fb1ec157740 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1ec0d23c0 0x7fb1ec157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 --2- 172.21.15.59:0/2605639482 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1ec0b3ed0 0x7fb1ec14b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 >> 172.21.15.59:0/2605639482 conn(0x7fb1ec0d3ee0 msgr2=0x7fb1ec06c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:58.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 shutdown_connections 2024-06-29T15:58:58.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.848+0000 7fb1f47ba640 1 -- 172.21.15.59:0/2605639482 wait complete. 2024-06-29T15:58:58.860 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:58.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.980+0000 7f554e35b640 1 -- 172.21.15.59:0/2594906039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 msgr2=0x7f55480b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.980+0000 7f554e35b640 1 --2- 172.21.15.59:0/2594906039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 0x7f55480b5880 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f5548074b10 tx=0x7f554002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.980+0000 7f554e35b640 1 -- 172.21.15.59:0/2594906039 shutdown_connections 2024-06-29T15:58:58.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.980+0000 7f554e35b640 1 --2- 172.21.15.59:0/2594906039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 0x7f55480b5880 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.980+0000 7f554e35b640 1 --2- 172.21.15.59:0/2594906039 >> v2:172.21.15.153:3300/0 conn(0x7f55481458a0 0x7f55480cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.980+0000 7f554e35b640 1 -- 172.21.15.59:0/2594906039 >> 172.21.15.59:0/2594906039 conn(0x7f554806c950 msgr2=0x7f5548073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:58.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.980+0000 7f554e35b640 1 -- 172.21.15.59:0/2594906039 shutdown_connections 2024-06-29T15:58:58.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.980+0000 7f554e35b640 1 -- 172.21.15.59:0/2594906039 wait complete. 2024-06-29T15:58:58.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.982+0000 7f554e35b640 1 Processor -- start 2024-06-29T15:58:58.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.982+0000 7f554e35b640 1 -- start start 2024-06-29T15:58:58.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.983+0000 7f554e35b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55480b3740 0x7f55481f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.983+0000 7f554e35b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 0x7f554806f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.983+0000 7f554e35b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55480d1ae0 0x7f55480d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.983+0000 7f554e35b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f55480b6150 con 0x7f55480b3b50 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.983+0000 7f554e35b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55480b5fd0 con 0x7f55480d1ae0 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.983+0000 7f554e35b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f55480b62d0 con 0x7f55480b3740 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.983+0000 7f55477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 0x7f554806f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f55477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 0x7f554806f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34862/0 (socket says 172.21.15.59:34862) 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f55477fe640 1 -- 172.21.15.59:0/184593290 learned_addr learned my addr 172.21.15.59:0/184593290 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f5547fff640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55480b3740 0x7f55481f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f554c8f8640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55480d1ae0 0x7f55480d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f55477fe640 1 -- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55480b3740 msgr2=0x7f55481f5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f55477fe640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55480b3740 0x7f55481f5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f55477fe640 1 -- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55480d1ae0 msgr2=0x7f55480d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f55477fe640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55480d1ae0 0x7f55480d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f55477fe640 1 -- 172.21.15.59:0/184593290 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554806f830 con 0x7f55480b3b50 2024-06-29T15:58:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f55477fe640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 0x7f554806f2f0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f553c00c3a0 tx=0x7f553c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f5544ff9640 1 -- 172.21.15.59:0/184593290 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c017070 con 0x7f55480b3b50 2024-06-29T15:58:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f5544ff9640 1 -- 172.21.15.59:0/184593290 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f553c0035f0 con 0x7f55480b3b50 2024-06-29T15:58:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55481f8450 con 0x7f55480b3b50 2024-06-29T15:58:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f5544ff9640 1 -- 172.21.15.59:0/184593290 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c003a40 con 0x7f55480b3b50 2024-06-29T15:58:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.984+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55481f8a10 con 0x7f55480b3b50 2024-06-29T15:58:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.985+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55481458a0 con 0x7f55480b3b50 2024-06-29T15:58:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.986+0000 7f5544ff9640 1 -- 172.21.15.59:0/184593290 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f553c01a070 con 0x7f55480b3b50 2024-06-29T15:58:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.987+0000 7f5544ff9640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5534076540 0x7f5534078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.987+0000 7f5544ff9640 1 -- 172.21.15.59:0/184593290 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7f553c097870 con 0x7f55480b3b50 2024-06-29T15:58:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.987+0000 7f5547fff640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5534076540 0x7f5534078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.988+0000 7f5547fff640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5534076540 0x7f5534078a50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f553800bb50 tx=0x7f553800b2e0 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:58.990+0000 7f5544ff9640 1 -- 172.21.15.59:0/184593290 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f553c021020 con 0x7f55480b3b50 2024-06-29T15:58:59.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.179+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f55480d25d0 con 0x7f55480b3b50 2024-06-29T15:58:59.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.179+0000 7f5544ff9640 1 -- 172.21.15.59:0/184593290 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v251) ==== 103+0+11 (secure 0 0 0) 0x7f553c060210 con 0x7f55480b3b50 2024-06-29T15:58:59.181 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:58:59.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5534076540 msgr2=0x7f5534078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5534076540 0x7f5534078a50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f553800bb50 tx=0x7f553800b2e0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 msgr2=0x7f554806f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 0x7f554806f2f0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f553c00c3a0 tx=0x7f553c00c860 comp rx=0 tx=0).stop 2024-06-29T15:58:59.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 shutdown_connections 2024-06-29T15:58:59.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5534076540 0x7f5534078a50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55480d1ae0 0x7f55480d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55480b3b50 0x7f554806f2f0 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 --2- 172.21.15.59:0/184593290 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55480b3740 0x7f55481f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 >> 172.21.15.59:0/184593290 conn(0x7f554806c950 msgr2=0x7f55480708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:59.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 shutdown_connections 2024-06-29T15:58:59.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.182+0000 7f554e35b640 1 -- 172.21.15.59:0/184593290 wait complete. 2024-06-29T15:58:59.195 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:59.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.313+0000 7f4659a95640 1 -- 172.21.15.59:0/3300784438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c004040 msgr2=0x7f463c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.313+0000 7f4659a95640 1 --2- 172.21.15.59:0/3300784438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c004040 0x7f463c004800 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f465413f300 tx=0x7f463c005830 comp rx=0 tx=0).stop 2024-06-29T15:58:59.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.313+0000 7f4653fff640 1 -- 172.21.15.59:0/3300784438 reap_dead start 2024-06-29T15:58:59.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.313+0000 7f4659a95640 1 -- 172.21.15.59:0/3300784438 shutdown_connections 2024-06-29T15:58:59.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.313+0000 7f4659a95640 1 -- 172.21.15.59:0/3300784438 >> 172.21.15.59:0/3300784438 conn(0x7f46540b6fb0 msgr2=0x7f46540b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:59.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.313+0000 7f4659a95640 1 -- 172.21.15.59:0/3300784438 shutdown_connections 2024-06-29T15:58:59.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.313+0000 7f4659a95640 1 -- 172.21.15.59:0/3300784438 wait complete. 2024-06-29T15:58:59.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.314+0000 7f4659a95640 1 Processor -- start 2024-06-29T15:58:59.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.315+0000 7f4659a95640 1 -- start start 2024-06-29T15:58:59.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4659a95640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463c004040 0x7f4654153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4659a95640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465406e570 0x7f465413d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4659a95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4654070de0 0x7f46540711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4659a95640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f46540b2530 con 0x7f465406e570 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4659a95640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46540b5a70 con 0x7f4654070de0 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4659a95640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46540b26b0 con 0x7f463c004040 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4652ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465406e570 0x7f465413d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4652ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465406e570 0x7f465413d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34894/0 (socket says 172.21.15.59:34894) 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4652ffd640 1 -- 172.21.15.59:0/3182795024 learned_addr learned my addr 172.21.15.59:0/3182795024 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4653fff640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4654070de0 0x7f46540711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f46537fe640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463c004040 0x7f4654153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4652ffd640 1 -- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463c004040 msgr2=0x7f4654153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4652ffd640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463c004040 0x7f4654153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4652ffd640 1 -- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4654070de0 msgr2=0x7f46540711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4652ffd640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4654070de0 0x7f46540711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.316+0000 7f4652ffd640 1 -- 172.21.15.59:0/3182795024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4654071730 con 0x7f465406e570 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.317+0000 7f4652ffd640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465406e570 0x7f465413d0a0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f4648009a70 tx=0x7f46480336d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:59.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.317+0000 7f463bfff640 1 -- 172.21.15.59:0/3182795024 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648038040 con 0x7f465406e570 2024-06-29T15:58:59.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.317+0000 7f463bfff640 1 -- 172.21.15.59:0/3182795024 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f464802b7e0 con 0x7f465406e570 2024-06-29T15:58:59.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.317+0000 7f463bfff640 1 -- 172.21.15.59:0/3182795024 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648033b20 con 0x7f465406e570 2024-06-29T15:58:59.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.317+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465406ed50 con 0x7f465406e570 2024-06-29T15:58:59.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.317+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4654151130 con 0x7f465406e570 2024-06-29T15:58:59.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.318+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f465406eac0 con 0x7f465406e570 2024-06-29T15:58:59.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.319+0000 7f463bfff640 1 -- 172.21.15.59:0/3182795024 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f4648045020 con 0x7f465406e570 2024-06-29T15:58:59.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.319+0000 7f463bfff640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f4640076510 0x7f4640078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.319+0000 7f463bfff640 1 -- 172.21.15.59:0/3182795024 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7f46480b88a0 con 0x7f465406e570 2024-06-29T15:58:59.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.320+0000 7f46537fe640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f4640076510 0x7f4640078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.320+0000 7f46537fe640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f4640076510 0x7f4640078a20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4644009bb0 tx=0x7f4644005f90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:59.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.324+0000 7f463bfff640 1 -- 172.21.15.59:0/3182795024 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f464804c020 con 0x7f465406e570 2024-06-29T15:58:59.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.555+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f465413edc0 con 0x7f465406e570 2024-06-29T15:58:59.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.555+0000 7f463bfff640 1 -- 172.21.15.59:0/3182795024 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5246 (secure 0 0 0) 0x7f4654071730 con 0x7f465406e570 2024-06-29T15:58:59.556 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:59.557 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":139,"btime":"2024-06-29T15:58:55:802987+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":10358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/555027351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":555027351},{"type":"v1","addr":"172.21.15.153:6837","nonce":555027351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:58:59.557 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 139 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f4640076510 msgr2=0x7f4640078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f4640076510 0x7f4640078a20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4644009bb0 tx=0x7f4644005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465406e570 msgr2=0x7f465413d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465406e570 0x7f465413d0a0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f4648009a70 tx=0x7f46480336d0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 shutdown_connections 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f4640076510 0x7f4640078a20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4654070de0 0x7f46540711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f465406e570 0x7f465413d0a0 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 --2- 172.21.15.59:0/3182795024 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f463c004040 0x7f4654153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 >> 172.21.15.59:0/3182795024 conn(0x7f46540b6fb0 msgr2=0x7f46540b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:59.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 shutdown_connections 2024-06-29T15:58:59.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.557+0000 7f4659a95640 1 -- 172.21.15.59:0/3182795024 wait complete. 2024-06-29T15:58:59.569 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:59.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.686+0000 7f1e3a943640 1 -- 172.21.15.59:0/403928610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e34162800 msgr2=0x7f1e34162c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.686+0000 7f1e3a943640 1 --2- 172.21.15.59:0/403928610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e34162800 0x7f1e34162c10 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c005fd0 tx=0x7f1e2c02aa70 comp rx=0 tx=0).stop 2024-06-29T15:58:59.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.686+0000 7f1e3a943640 1 -- 172.21.15.59:0/403928610 shutdown_connections 2024-06-29T15:58:59.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.686+0000 7f1e3a943640 1 --2- 172.21.15.59:0/403928610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e34162800 0x7f1e34162c10 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.686+0000 7f1e3a943640 1 --2- 172.21.15.59:0/403928610 >> v2:172.21.15.153:3300/0 conn(0x7f1e340ce950 0x7f1e341622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.686+0000 7f1e3a943640 1 -- 172.21.15.59:0/403928610 >> 172.21.15.59:0/403928610 conn(0x7f1e340b5610 msgr2=0x7f1e340b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:59.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.686+0000 7f1e3a943640 1 -- 172.21.15.59:0/403928610 shutdown_connections 2024-06-29T15:58:59.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.686+0000 7f1e3a943640 1 -- 172.21.15.59:0/403928610 wait complete. 2024-06-29T15:58:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.688+0000 7f1e3a943640 1 Processor -- start 2024-06-29T15:58:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.688+0000 7f1e3a943640 1 -- start start 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e3a943640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e3408c190 0x7f1e340aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e3a943640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e340ce950 0x7f1e340aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e3a943640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34162800 0x7f1e340cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e3a943640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1e340b7f60 con 0x7f1e340ce950 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e3a943640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e340b7de0 con 0x7f1e34162800 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e3a943640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1e340b80e0 con 0x7f1e3408c190 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e340ce950 0x7f1e340aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e340ce950 0x7f1e340aaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34906/0 (socket says 172.21.15.59:34906) 2024-06-29T15:58:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 -- 172.21.15.59:0/4180696905 learned_addr learned my addr 172.21.15.59:0/4180696905 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e33fff640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e3408c190 0x7f1e340aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e38ee0640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34162800 0x7f1e340cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 -- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e3408c190 msgr2=0x7f1e340aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e3408c190 0x7f1e340aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 -- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34162800 msgr2=0x7f1e340cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34162800 0x7f1e340cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 -- 172.21.15.59:0/4180696905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e340cc070 con 0x7f1e340ce950 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.690+0000 7f1e337fe640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e340ce950 0x7f1e340aaa30 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f1e28009fd0 tx=0x7f1e2800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.691+0000 7f1e30ff9640 1 -- 172.21.15.59:0/4180696905 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2800cc60 con 0x7f1e340ce950 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.691+0000 7f1e30ff9640 1 -- 172.21.15.59:0/4180696905 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e2800ce00 con 0x7f1e340ce950 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.691+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e340cc350 con 0x7f1e340ce950 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.691+0000 7f1e30ff9640 1 -- 172.21.15.59:0/4180696905 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28004280 con 0x7f1e340ce950 2024-06-29T15:58:59.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.691+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e340cc910 con 0x7f1e340ce950 2024-06-29T15:58:59.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.692+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e341622c0 con 0x7f1e340ce950 2024-06-29T15:58:59.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.692+0000 7f1e30ff9640 1 -- 172.21.15.59:0/4180696905 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f1e280035f0 con 0x7f1e340ce950 2024-06-29T15:58:59.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.693+0000 7f1e30ff9640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1e20076540 0x7f1e20078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.693+0000 7f1e30ff9640 1 -- 172.21.15.59:0/4180696905 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7f1e28098970 con 0x7f1e340ce950 2024-06-29T15:58:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.693+0000 7f1e33fff640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1e20076540 0x7f1e20078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.694+0000 7f1e33fff640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1e20076540 0x7f1e20078a50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e24001720 tx=0x7f1e24009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:59.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.697+0000 7f1e30ff9640 1 -- 172.21.15.59:0/4180696905 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1e28061200 con 0x7f1e340ce950 2024-06-29T15:58:59.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.893+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e340c7a50 con 0x7f1e340ce950 2024-06-29T15:58:59.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.894+0000 7f1e30ff9640 1 -- 172.21.15.59:0/4180696905 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) ==== 74+0+18400 (secure 0 0 0) 0x7f1e28065500 con 0x7f1e340ce950 2024-06-29T15:58:59.895 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:58:59.896 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":251,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:57.813620+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6834/177893499":"2024-06-30T15:58:54.800279+0000","172.21.15.153:6832/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2374956971":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2562681960":"2024-06-30T15:58:51.014782+0000","172.21.15.153:6837/177893499":"2024-06-30T15:58:54.800279+0000","172.21.15.153:0/221125627":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3082371830":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1656484964":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/402968645":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1556406125":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/877026286":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3972699112":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1636302558":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1944500160":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1892969816":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3355566242":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6832/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/2494131047":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1461412440":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1259499350":"2024-06-30T15:58:45.316819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:59.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.896+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1e20076540 msgr2=0x7f1e20078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.896+0000 7f1e3a943640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1e20076540 0x7f1e20078a50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e24001720 tx=0x7f1e24009040 comp rx=0 tx=0).stop 2024-06-29T15:58:59.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e340ce950 msgr2=0x7f1e340aaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e340ce950 0x7f1e340aaa30 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f1e28009fd0 tx=0x7f1e2800ec20 comp rx=0 tx=0).stop 2024-06-29T15:58:59.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 shutdown_connections 2024-06-29T15:58:59.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1e20076540 0x7f1e20078a50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34162800 0x7f1e340cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e340ce950 0x7f1e340aaa30 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 --2- 172.21.15.59:0/4180696905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1e3408c190 0x7f1e340aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 >> 172.21.15.59:0/4180696905 conn(0x7f1e340b5610 msgr2=0x7f1e3415d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:59.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 shutdown_connections 2024-06-29T15:58:59.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:58:59.897+0000 7f1e3a943640 1 -- 172.21.15.59:0/4180696905 wait complete. 2024-06-29T15:58:59.909 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-06-29T15:58:59.909 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 -- 172.21.15.59:0/2550299992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0b3ed0 msgr2=0x7ff80c15e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 --2- 172.21.15.59:0/2550299992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0b3ed0 0x7ff80c15e8d0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0d15d0 tx=0x7ff80002a560 comp rx=0 tx=0).stop 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 -- 172.21.15.59:0/2550299992 shutdown_connections 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 --2- 172.21.15.59:0/2550299992 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff80c15ee10 0x7ff80c15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 --2- 172.21.15.59:0/2550299992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0b3ed0 0x7ff80c15e8d0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 --2- 172.21.15.59:0/2550299992 >> v2:172.21.15.153:3300/0 conn(0x7ff80c0d23c0 0x7ff80c0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 -- 172.21.15.59:0/2550299992 >> 172.21.15.59:0/2550299992 conn(0x7ff80c0d3ee0 msgr2=0x7ff80c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 -- 172.21.15.59:0/2550299992 shutdown_connections 2024-06-29T15:59:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.026+0000 7ff811bbd640 1 -- 172.21.15.59:0/2550299992 wait complete. 2024-06-29T15:59:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.028+0000 7ff811bbd640 1 Processor -- start 2024-06-29T15:59:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.028+0000 7ff811bbd640 1 -- start start 2024-06-29T15:59:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff811bbd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff80c0b3ed0 0x7ff80c1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff811bbd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0d23c0 0x7ff80c15b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff811bbd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff80c15ee10 0x7ff80c0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff811bbd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c160070 con 0x7ff80c0d23c0 2024-06-29T15:59:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff811bbd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c15fef0 con 0x7ff80c0b3ed0 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff811bbd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff80c1601f0 con 0x7ff80c15ee10 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0d23c0 0x7ff80c15b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0d23c0 0x7ff80c15b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34926/0 (socket says 172.21.15.59:34926) 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 -- 172.21.15.59:0/56123906 learned_addr learned my addr 172.21.15.59:0/56123906 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80b7fe640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff80c0b3ed0 0x7ff80c1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80bfff640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff80c15ee10 0x7ff80c0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 -- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff80c15ee10 msgr2=0x7ff80c0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff80c15ee10 0x7ff80c0cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 -- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff80c0b3ed0 msgr2=0x7ff80c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff80c0b3ed0 0x7ff80c1598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 -- 172.21.15.59:0/56123906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c14c430 con 0x7ff80c0d23c0 2024-06-29T15:59:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80bfff640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff80c15ee10 0x7ff80c0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff80affd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0d23c0 0x7ff80c15b810 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7ff8000099c0 tx=0x7ff8000039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff7f3fff640 1 -- 172.21.15.59:0/56123906 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80003e030 con 0x7ff80c0d23c0 2024-06-29T15:59:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff7f3fff640 1 -- 172.21.15.59:0/56123906 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8000047a0 con 0x7ff80c0d23c0 2024-06-29T15:59:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff7f3fff640 1 -- 172.21.15.59:0/56123906 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff800035040 con 0x7ff80c0d23c0 2024-06-29T15:59:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.030+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c0cbc60 con 0x7ff80c0d23c0 2024-06-29T15:59:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.031+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c0cc1e0 con 0x7ff80c0d23c0 2024-06-29T15:59:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.031+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff80c06d910 con 0x7ff80c0d23c0 2024-06-29T15:59:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.032+0000 7ff7f3fff640 1 -- 172.21.15.59:0/56123906 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7ff800041030 con 0x7ff80c0d23c0 2024-06-29T15:59:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.033+0000 7ff7f3fff640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff7f8076510 0x7ff7f8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.033+0000 7ff7f3fff640 1 -- 172.21.15.59:0/56123906 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7ff8000b88d0 con 0x7ff80c0d23c0 2024-06-29T15:59:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.033+0000 7ff80b7fe640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff7f8076510 0x7ff7f8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.034+0000 7ff80b7fe640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff7f8076510 0x7ff7f8078a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0d2e40 tx=0x7ff7fc00b260 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:00.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.037+0000 7ff7f3fff640 1 -- 172.21.15.59:0/56123906 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff80c06d910 con 0x7ff80c0d23c0 2024-06-29T15:59:00.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.266+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff80c0b3990 con 0x7ff80c0d23c0 2024-06-29T15:59:00.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.267+0000 7ff7f3fff640 1 -- 172.21.15.59:0/56123906 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5246 (secure 0 0 0) 0x7ff800081230 con 0x7ff80c0d23c0 2024-06-29T15:59:00.268 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:00.268 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":139,"btime":"2024-06-29T15:58:55:802987+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":10358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/555027351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":555027351},{"type":"v1","addr":"172.21.15.153:6837","nonce":555027351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:59:00.268 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 139 2024-06-29T15:59:00.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff7f8076510 msgr2=0x7ff7f8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff7f8076510 0x7ff7f8078a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0d2e40 tx=0x7ff7fc00b260 comp rx=0 tx=0).stop 2024-06-29T15:59:00.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0d23c0 msgr2=0x7ff80c15b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0d23c0 0x7ff80c15b810 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7ff8000099c0 tx=0x7ff8000039e0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 shutdown_connections 2024-06-29T15:59:00.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff7f8076510 0x7ff7f8078a20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff80c15ee10 0x7ff80c0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff80c0d23c0 0x7ff80c15b810 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 --2- 172.21.15.59:0/56123906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff80c0b3ed0 0x7ff80c1598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 >> 172.21.15.59:0/56123906 conn(0x7ff80c0d3ee0 msgr2=0x7ff80c160730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:00.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 shutdown_connections 2024-06-29T15:59:00.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.269+0000 7ff811bbd640 1 -- 172.21.15.59:0/56123906 wait complete. 2024-06-29T15:59:00.281 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:00.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.398+0000 7fd09e519640 1 -- 172.21.15.59:0/1261053078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080003ac0 msgr2=0x7fd080003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.398+0000 7fd09e519640 1 --2- 172.21.15.59:0/1261053078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080003ac0 0x7fd080003f50 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fd0880016c0 tx=0x7fd080005830 comp rx=0 tx=0).stop 2024-06-29T15:59:00.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.398+0000 7fd09e519640 1 -- 172.21.15.59:0/1261053078 shutdown_connections 2024-06-29T15:59:00.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.398+0000 7fd09e519640 1 --2- 172.21.15.59:0/1261053078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd080003ac0 0x7fd080003f50 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.398+0000 7fd09e519640 1 -- 172.21.15.59:0/1261053078 >> 172.21.15.59:0/1261053078 conn(0x7fd0980b5610 msgr2=0x7fd0980b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:00.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.398+0000 7fd09e519640 1 -- 172.21.15.59:0/1261053078 shutdown_connections 2024-06-29T15:59:00.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.398+0000 7fd09e519640 1 -- 172.21.15.59:0/1261053078 wait complete. 2024-06-29T15:59:00.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.400+0000 7fd09e519640 1 Processor -- start 2024-06-29T15:59:00.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.400+0000 7fd09e519640 1 -- start start 2024-06-29T15:59:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd09e519640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd080003ac0 0x7fd098154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd09e519640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd09815b7a0 0x7fd09815a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd09e519640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd09815af20 0x7fd0981f8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd09e519640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd0980cf680 con 0x7fd09815b7a0 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd09e519640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd0980cf500 con 0x7fd09815af20 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd09e519640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd0980cf800 con 0x7fd080003ac0 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd0977fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd09815b7a0 0x7fd09815a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd0977fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd09815b7a0 0x7fd09815a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34946/0 (socket says 172.21.15.59:34946) 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd0977fe640 1 -- 172.21.15.59:0/3856918701 learned_addr learned my addr 172.21.15.59:0/3856918701 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd097fff640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd080003ac0 0x7fd098154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.401+0000 7fd09cab6640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd09815af20 0x7fd0981f8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd0977fe640 1 -- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd080003ac0 msgr2=0x7fd098154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd0977fe640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd080003ac0 0x7fd098154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd0977fe640 1 -- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd09815af20 msgr2=0x7fd0981f8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd0977fe640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd09815af20 0x7fd0981f8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd0977fe640 1 -- 172.21.15.59:0/3856918701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09815b330 con 0x7fd09815b7a0 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd09cab6640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd09815af20 0x7fd0981f8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd0977fe640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd09815b7a0 0x7fd09815a9e0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fd08c006430 tx=0x7fd08c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd094ff9640 1 -- 172.21.15.59:0/3856918701 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c015070 con 0x7fd09815b7a0 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd094ff9640 1 -- 172.21.15.59:0/3856918701 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd08c011840 con 0x7fd09815b7a0 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd094ff9640 1 -- 172.21.15.59:0/3856918701 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c011cd0 con 0x7fd09815b7a0 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09815a570 con 0x7fd09815b7a0 2024-06-29T15:59:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.402+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0980ccec0 con 0x7fd09815b7a0 2024-06-29T15:59:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.403+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0980ccc30 con 0x7fd09815b7a0 2024-06-29T15:59:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.404+0000 7fd094ff9640 1 -- 172.21.15.59:0/3856918701 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fd08c018070 con 0x7fd09815b7a0 2024-06-29T15:59:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.405+0000 7fd094ff9640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd084076540 0x7fd084078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.405+0000 7fd094ff9640 1 -- 172.21.15.59:0/3856918701 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7fd08c098dc0 con 0x7fd09815b7a0 2024-06-29T15:59:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.405+0000 7fd097fff640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd084076540 0x7fd084078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.406+0000 7fd097fff640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd084076540 0x7fd084078a50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd088032000 tx=0x7fd08802c040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:00.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.409+0000 7fd094ff9640 1 -- 172.21.15.59:0/3856918701 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd09815b330 con 0x7fd09815b7a0 2024-06-29T15:59:00.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.638+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd098162650 con 0x7fd09815b7a0 2024-06-29T15:59:00.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.638+0000 7fd094ff9640 1 -- 172.21.15.59:0/3856918701 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5246 (secure 0 0 0) 0x7fd08c0617b0 con 0x7fd09815b7a0 2024-06-29T15:59:00.640 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:00.640 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":139,"btime":"2024-06-29T15:58:55:802987+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":10358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/555027351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":555027351},{"type":"v1","addr":"172.21.15.153:6837","nonce":555027351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:34.012091+0000","modified":"2024-06-29T15:58:37.031172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6835/1525995706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1525995706},{"type":"v1","addr":"172.21.15.59:6835","nonce":1525995706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12303,"qdb_cluster":[12303]},"id":16}]} 2024-06-29T15:59:00.640 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 139 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd084076540 msgr2=0x7fd084078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd084076540 0x7fd084078a50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd088032000 tx=0x7fd08802c040 comp rx=0 tx=0).stop 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd09815b7a0 msgr2=0x7fd09815a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd09815b7a0 0x7fd09815a9e0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fd08c006430 tx=0x7fd08c0039c0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 shutdown_connections 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd084076540 0x7fd084078a50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd09815af20 0x7fd0981f8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd09815b7a0 0x7fd09815a9e0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 --2- 172.21.15.59:0/3856918701 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd080003ac0 0x7fd098154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 >> 172.21.15.59:0/3856918701 conn(0x7fd0980b5610 msgr2=0x7fd0980c8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:00.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 shutdown_connections 2024-06-29T15:59:00.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.641+0000 7fd09e519640 1 -- 172.21.15.59:0/3856918701 wait complete. 2024-06-29T15:59:00.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:00.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 -- 172.21.15.59:0/2242798266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab41458a0 msgr2=0x7ffab40cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 --2- 172.21.15.59:0/2242798266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab41458a0 0x7ffab40cf870 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7ffab40d0730 tx=0x7ffaa402aa30 comp rx=0 tx=0).stop 2024-06-29T15:59:00.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 -- 172.21.15.59:0/2242798266 shutdown_connections 2024-06-29T15:59:00.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 --2- 172.21.15.59:0/2242798266 >> v2:172.21.15.153:3300/0 conn(0x7ffab415c2f0 0x7ffab415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 --2- 172.21.15.59:0/2242798266 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffab40b3740 0x7ffab40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 --2- 172.21.15.59:0/2242798266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab41458a0 0x7ffab40cf870 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 -- 172.21.15.59:0/2242798266 >> 172.21.15.59:0/2242798266 conn(0x7ffab406c950 msgr2=0x7ffab4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:00.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 -- 172.21.15.59:0/2242798266 shutdown_connections 2024-06-29T15:59:00.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.769+0000 7ffabc45e640 1 -- 172.21.15.59:0/2242798266 wait complete. 2024-06-29T15:59:00.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.771+0000 7ffabc45e640 1 Processor -- start 2024-06-29T15:59:00.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.771+0000 7ffabc45e640 1 -- start start 2024-06-29T15:59:00.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffabc45e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffab40b3740 0x7ffab415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffabc45e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab415c2f0 0x7ffab40b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffabc45e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffab4070550 0x7ffab4070960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffabc45e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ffab415fe40 con 0x7ffab415c2f0 2024-06-29T15:59:00.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffabc45e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffab415fcc0 con 0x7ffab4070550 2024-06-29T15:59:00.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffabc45e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffab415ffc0 con 0x7ffab40b3740 2024-06-29T15:59:00.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffab99f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab415c2f0 0x7ffab40b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffab99f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab415c2f0 0x7ffab40b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34976/0 (socket says 172.21.15.59:34976) 2024-06-29T15:59:00.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffab99f9640 1 -- 172.21.15.59:0/236555174 learned_addr learned my addr 172.21.15.59:0/236555174 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:00.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffaba1fa640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffab40b3740 0x7ffab415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.772+0000 7ffaba9fb640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffab4070550 0x7ffab4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab99f9640 1 -- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffab40b3740 msgr2=0x7ffab415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab99f9640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffab40b3740 0x7ffab415b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab99f9640 1 -- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffab4070550 msgr2=0x7ffab4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab99f9640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffab4070550 0x7ffab4070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab99f9640 1 -- 172.21.15.59:0/236555174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab4070ea0 con 0x7ffab415c2f0 2024-06-29T15:59:00.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab99f9640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab415c2f0 0x7ffab40b0cd0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d680 tx=0x7ffaa800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:00.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab2ffd640 1 -- 172.21.15.59:0/236555174 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8017070 con 0x7ffab415c2f0 2024-06-29T15:59:00.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab2ffd640 1 -- 172.21.15.59:0/236555174 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffaa8004310 con 0x7ffab415c2f0 2024-06-29T15:59:00.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab413a9d0 con 0x7ffab415c2f0 2024-06-29T15:59:00.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffab2ffd640 1 -- 172.21.15.59:0/236555174 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8005280 con 0x7ffab415c2f0 2024-06-29T15:59:00.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.773+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab413af90 con 0x7ffab415c2f0 2024-06-29T15:59:00.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.774+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab413a740 con 0x7ffab415c2f0 2024-06-29T15:59:00.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.776+0000 7ffab2ffd640 1 -- 172.21.15.59:0/236555174 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7ffaa8005420 con 0x7ffab415c2f0 2024-06-29T15:59:00.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.776+0000 7ffab2ffd640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffaa0076510 0x7ffaa0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.776+0000 7ffab2ffd640 1 -- 172.21.15.59:0/236555174 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7ffaa8098900 con 0x7ffab415c2f0 2024-06-29T15:59:00.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.776+0000 7ffaba1fa640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffaa0076510 0x7ffaa0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.777+0000 7ffaba1fa640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffaa0076510 0x7ffaa0078a20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4035000 tx=0x7ffaa403b040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:00.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.780+0000 7ffab2ffd640 1 -- 172.21.15.59:0/236555174 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffaa8020020 con 0x7ffab415c2f0 2024-06-29T15:59:00.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.973+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffab41458a0 con 0x7ffab415c2f0 2024-06-29T15:59:00.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.974+0000 7ffab2ffd640 1 -- 172.21.15.59:0/236555174 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v251) ==== 74+0+18400 (secure 0 0 0) 0x7ffab4070ea0 con 0x7ffab415c2f0 2024-06-29T15:59:00.975 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:00.975 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":251,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:58:57.813620+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:31.630427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"245","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:32.972384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6834/177893499":"2024-06-30T15:58:54.800279+0000","172.21.15.153:6832/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2374956971":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2562681960":"2024-06-30T15:58:51.014782+0000","172.21.15.153:6837/177893499":"2024-06-30T15:58:54.800279+0000","172.21.15.153:0/221125627":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3082371830":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1656484964":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/402968645":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1556406125":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/877026286":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3972699112":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1636302558":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1944500160":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1892969816":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3355566242":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6832/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/2494131047":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1461412440":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1259499350":"2024-06-30T15:58:45.316819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffaa0076510 msgr2=0x7ffaa0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffaa0076510 0x7ffaa0078a20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4035000 tx=0x7ffaa403b040 comp rx=0 tx=0).stop 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab415c2f0 msgr2=0x7ffab40b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab415c2f0 0x7ffab40b0cd0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d680 tx=0x7ffaa800db40 comp rx=0 tx=0).stop 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 shutdown_connections 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffaa0076510 0x7ffaa0078a20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffab4070550 0x7ffab4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffab415c2f0 0x7ffab40b0cd0 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 --2- 172.21.15.59:0/236555174 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffab40b3740 0x7ffab415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 >> 172.21.15.59:0/236555174 conn(0x7ffab406c950 msgr2=0x7ffab40d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 shutdown_connections 2024-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:00.976+0000 7ffabc45e640 1 -- 172.21.15.59:0/236555174 wait complete. 2024-06-29T15:59:00.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-06-29T15:59:01.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.106+0000 7ffa2b881640 1 -- 172.21.15.59:0/1374692982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa24141bd0 msgr2=0x7ffa2413edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.106+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1374692982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa24141bd0 0x7ffa2413edc0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7ffa24069670 tx=0x7ffa1402a980 comp rx=0 tx=0).stop 2024-06-29T15:59:01.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.106+0000 7ffa2b881640 1 -- 172.21.15.59:0/1374692982 shutdown_connections 2024-06-29T15:59:01.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.106+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1374692982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa24141bd0 0x7ffa2413edc0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.106+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1374692982 >> v2:172.21.15.153:3300/0 conn(0x7ffa2415ceb0 0x7ffa240b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.106+0000 7ffa2b881640 1 -- 172.21.15.59:0/1374692982 >> 172.21.15.59:0/1374692982 conn(0x7ffa240b6fb0 msgr2=0x7ffa240b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:01.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.106+0000 7ffa2b881640 1 -- 172.21.15.59:0/1374692982 shutdown_connections 2024-06-29T15:59:01.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.106+0000 7ffa2b881640 1 -- 172.21.15.59:0/1374692982 wait complete. 2024-06-29T15:59:01.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.108+0000 7ffa2b881640 1 Processor -- start 2024-06-29T15:59:01.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.108+0000 7ffa2b881640 1 -- start start 2024-06-29T15:59:01.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa2b881640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2413e9b0 0x7ffa241f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa2b881640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2415ceb0 0x7ffa241f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa2b881640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa241f5a70 0x7ffa241f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa2b881640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ffa240b29d0 con 0x7ffa241f5a70 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa2b881640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa240b2850 con 0x7ffa2415ceb0 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa2b881640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffa240b2b50 con 0x7ffa2413e9b0 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa29e1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa241f5a70 0x7ffa241f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa29e1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa241f5a70 0x7ffa241f5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34996/0 (socket says 172.21.15.59:34996) 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa29e1e640 1 -- 172.21.15.59:0/1633143763 learned_addr learned my addr 172.21.15.59:0/1633143763 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa2961d640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2413e9b0 0x7ffa241f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.109+0000 7ffa28e1c640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2415ceb0 0x7ffa241f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa29e1e640 1 -- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2413e9b0 msgr2=0x7ffa241f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa29e1e640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2413e9b0 0x7ffa241f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa29e1e640 1 -- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2415ceb0 msgr2=0x7ffa241f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa29e1e640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2415ceb0 0x7ffa241f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa29e1e640 1 -- 172.21.15.59:0/1633143763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa241f2320 con 0x7ffa241f5a70 2024-06-29T15:59:01.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa29e1e640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa241f5a70 0x7ffa241f5e80 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7ffa14033040 tx=0x7ffa14003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:01.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa21ffb640 1 -- 172.21.15.59:0/1633143763 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14034040 con 0x7ffa241f5a70 2024-06-29T15:59:01.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa21ffb640 1 -- 172.21.15.59:0/1633143763 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa14004540 con 0x7ffa241f5a70 2024-06-29T15:59:01.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa241f63c0 con 0x7ffa241f5a70 2024-06-29T15:59:01.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa21ffb640 1 -- 172.21.15.59:0/1633143763 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14045480 con 0x7ffa241f5a70 2024-06-29T15:59:01.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.110+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa240c6b80 con 0x7ffa241f5a70 2024-06-29T15:59:01.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.111+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa240b3000 con 0x7ffa241f5a70 2024-06-29T15:59:01.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.112+0000 7ffa21ffb640 1 -- 172.21.15.59:0/1633143763 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7ffa14037020 con 0x7ffa241f5a70 2024-06-29T15:59:01.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.113+0000 7ffa21ffb640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffa10076510 0x7ffa10078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.113+0000 7ffa21ffb640 1 -- 172.21.15.59:0/1633143763 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 7115+0+0 (secure 0 0 0) 0x7ffa140b9a30 con 0x7ffa241f5a70 2024-06-29T15:59:01.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.113+0000 7ffa2961d640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffa10076510 0x7ffa10078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.114+0000 7ffa2961d640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffa10076510 0x7ffa10078a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c004630 tx=0x7ffa1c00b2e0 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:01.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.117+0000 7ffa21ffb640 1 -- 172.21.15.59:0/1633143763 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa241f63c0 con 0x7ffa241f5a70 2024-06-29T15:59:01.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.348+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) -- 0x7ffa24156800 con 0x7ffa241f5a70 2024-06-29T15:59:01.349 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:59:01.348+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:59:01.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.350+0000 7ffa21ffb640 1 -- 172.21.15.59:0/1633143763 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v140) ==== 167+0+0 (secure 0 0 0) 0x7ffa140823d0 con 0x7ffa241f5a70 2024-06-29T15:59:01.351 INFO:teuthology.orchestra.run.smithi059.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:59:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffa10076510 msgr2=0x7ffa10078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffa10076510 0x7ffa10078a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c004630 tx=0x7ffa1c00b2e0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa241f5a70 msgr2=0x7ffa241f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa241f5a70 0x7ffa241f5e80 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7ffa14033040 tx=0x7ffa14003f10 comp rx=0 tx=0).stop 2024-06-29T15:59:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 shutdown_connections 2024-06-29T15:59:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ffa10076510 0x7ffa10078a20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa241f5a70 0x7ffa241f5e80 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa2415ceb0 0x7ffa241f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 --2- 172.21.15.59:0/1633143763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa2413e9b0 0x7ffa241f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 >> 172.21.15.59:0/1633143763 conn(0x7ffa240b6fb0 msgr2=0x7ffa2413e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 shutdown_connections 2024-06-29T15:59:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.353+0000 7ffa2b881640 1 -- 172.21.15.59:0/1633143763 wait complete. 2024-06-29T15:59:01.355 INFO:tasks.ceph.mds.c.smithi059.stderr: -1685> 2024-06-29T15:58:38.521+0000 7f98c9af1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:59:01.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-06-29T15:59:01.409 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:59:01.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.488+0000 7fb48a0df640 1 -- 172.21.15.59:0/2341332307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4840b5cb0 msgr2=0x7fb48415f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.488+0000 7fb48a0df640 1 --2- 172.21.15.59:0/2341332307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4840b5cb0 0x7fb48415f7f0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fb48415d670 tx=0x7fb47802a510 comp rx=0 tx=0).stop 2024-06-29T15:59:01.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.488+0000 7fb48a0df640 1 -- 172.21.15.59:0/2341332307 shutdown_connections 2024-06-29T15:59:01.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.488+0000 7fb48a0df640 1 --2- 172.21.15.59:0/2341332307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4840b5cb0 0x7fb48415f7f0 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.488+0000 7fb48a0df640 1 -- 172.21.15.59:0/2341332307 >> 172.21.15.59:0/2341332307 conn(0x7fb4840b6ed0 msgr2=0x7fb4840b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:01.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.488+0000 7fb48a0df640 1 -- 172.21.15.59:0/2341332307 shutdown_connections 2024-06-29T15:59:01.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.488+0000 7fb48a0df640 1 -- 172.21.15.59:0/2341332307 wait complete. 2024-06-29T15:59:01.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.490+0000 7fb48a0df640 1 Processor -- start 2024-06-29T15:59:01.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.490+0000 7fb48a0df640 1 -- start start 2024-06-29T15:59:01.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.491+0000 7fb48a0df640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4840b5cb0 0x7fb48413c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.491+0000 7fb48a0df640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb484145650 0x7fb4841447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.491+0000 7fb48a0df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb48415fd30 0x7fb484070d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.491+0000 7fb48a0df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb48415e560 con 0x7fb48415fd30 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.491+0000 7fb48a0df640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb48415e3e0 con 0x7fb4840b5cb0 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.491+0000 7fb48a0df640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb48415e6e0 con 0x7fb484145650 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.491+0000 7fb483fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb48415fd30 0x7fb484070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.491+0000 7fb483fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb48415fd30 0x7fb484070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35032/0 (socket says 172.21.15.59:35032) 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb483fff640 1 -- 172.21.15.59:0/3618863877 learned_addr learned my addr 172.21.15.59:0/3618863877 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb482ffd640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb484145650 0x7fb4841447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb4837fe640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4840b5cb0 0x7fb48413c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb483fff640 1 -- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb484145650 msgr2=0x7fb4841447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb483fff640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb484145650 0x7fb4841447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb483fff640 1 -- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4840b5cb0 msgr2=0x7fb48413c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb483fff640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4840b5cb0 0x7fb48413c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb483fff640 1 -- 172.21.15.59:0/3618863877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb48413ca60 con 0x7fb48415fd30 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb4837fe640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4840b5cb0 0x7fb48413c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb483fff640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb48415fd30 0x7fb484070d20 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00c6e0 tx=0x7fb47c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb46bfff640 1 -- 172.21.15.59:0/3618863877 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c003b70 con 0x7fb48415fd30 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb46bfff640 1 -- 172.21.15.59:0/3618863877 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb47c003d10 con 0x7fb48415fd30 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb484142e40 con 0x7fb48415fd30 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb46bfff640 1 -- 172.21.15.59:0/3618863877 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c005330 con 0x7fb48415fd30 2024-06-29T15:59:01.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.492+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb484143400 con 0x7fb48415fd30 2024-06-29T15:59:01.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.494+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb48415ebd0 con 0x7fb48415fd30 2024-06-29T15:59:01.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.494+0000 7fb46bfff640 1 -- 172.21.15.59:0/3618863877 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fb47c0054d0 con 0x7fb48415fd30 2024-06-29T15:59:01.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.495+0000 7fb46bfff640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb470076510 0x7fb470078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.495+0000 7fb46bfff640 1 -- 172.21.15.59:0/3618863877 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 7201+0+0 (secure 0 0 0) 0x7fb47c097ff0 con 0x7fb48415fd30 2024-06-29T15:59:01.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.495+0000 7fb4837fe640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb470076510 0x7fb470078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.496+0000 7fb4837fe640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb470076510 0x7fb470078a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb474000f60 tx=0x7fb474009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:01.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.500+0000 7fb46bfff640 1 -- 172.21.15.59:0/3618863877 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb484160140 con 0x7fb48415fd30 2024-06-29T15:59:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:01.731+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) -- 0x7fb4840ceaa0 con 0x7fb48415fd30 2024-06-29T15:59:02.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.351+0000 7fb46bfff640 1 -- 172.21.15.59:0/3618863877 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v141) ==== 107+0+0 (secure 0 0 0) 0x7fb47c060f90 con 0x7fb48415fd30 2024-06-29T15:59:02.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb470076510 msgr2=0x7fb470078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb470076510 0x7fb470078a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb474000f60 tx=0x7fb474009040 comp rx=0 tx=0).stop 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb48415fd30 msgr2=0x7fb484070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb48415fd30 0x7fb484070d20 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00c6e0 tx=0x7fb47c00cba0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 shutdown_connections 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb470076510 0x7fb470078a20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb48415fd30 0x7fb484070d20 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb484145650 0x7fb4841447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 --2- 172.21.15.59:0/3618863877 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4840b5cb0 0x7fb48413c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.354+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 >> 172.21.15.59:0/3618863877 conn(0x7fb4840b6ed0 msgr2=0x7fb48415c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.355+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 shutdown_connections 2024-06-29T15:59:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.355+0000 7fb48a0df640 1 -- 172.21.15.59:0/3618863877 wait complete. 2024-06-29T15:59:02.369 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-06-29T15:59:02.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.536+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3986037825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc130003a30 msgr2=0x7fc130003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.536+0000 7fc14f4e4640 1 --2- 172.21.15.59:0/3986037825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc130003a30 0x7fc130003ea0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fc138009ed0 tx=0x7fc130005830 comp rx=0 tx=0).stop 2024-06-29T15:59:02.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.536+0000 7fc14da81640 1 -- 172.21.15.59:0/3986037825 reap_dead start 2024-06-29T15:59:02.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.536+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3986037825 shutdown_connections 2024-06-29T15:59:02.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.536+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3986037825 >> 172.21.15.59:0/3986037825 conn(0x7fc148073e10 msgr2=0x7fc148074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:02.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.536+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3986037825 shutdown_connections 2024-06-29T15:59:02.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.536+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3986037825 wait complete. 2024-06-29T15:59:02.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.538+0000 7fc14f4e4640 1 Processor -- start 2024-06-29T15:59:02.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.538+0000 7fc14f4e4640 1 -- start start 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14f4e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc130003a30 0x7fc14815b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14f4e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1481f4750 0x7fc148144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14f4e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc14806e8d0 0x7fc14806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14f4e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc1480b5a70 con 0x7fc14806e8d0 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14f4e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1480b58f0 con 0x7fc130003a30 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14f4e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc14815c480 con 0x7fc1481f4750 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14da81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc14806e8d0 0x7fc14806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14da81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc14806e8d0 0x7fc14806ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35048/0 (socket says 172.21.15.59:35048) 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.539+0000 7fc14da81640 1 -- 172.21.15.59:0/3696647927 learned_addr learned my addr 172.21.15.59:0/3696647927 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14d280640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc130003a30 0x7fc14815b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14ca7f640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1481f4750 0x7fc148144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14da81640 1 -- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1481f4750 msgr2=0x7fc148144df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14da81640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1481f4750 0x7fc148144df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14da81640 1 -- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc130003a30 msgr2=0x7fc14815b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14da81640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc130003a30 0x7fc14815b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14da81640 1 -- 172.21.15.59:0/3696647927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc148143d50 con 0x7fc14806e8d0 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14da81640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc14806e8d0 0x7fc14806ece0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fc14806ceb0 tx=0x7fc14000e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc145ffb640 1 -- 172.21.15.59:0/3696647927 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1400107d0 con 0x7fc14806e8d0 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc145ffb640 1 -- 172.21.15.59:0/3696647927 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc140016070 con 0x7fc14806e8d0 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc148144030 con 0x7fc14806e8d0 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc145ffb640 1 -- 172.21.15.59:0/3696647927 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140010d00 con 0x7fc14806e8d0 2024-06-29T15:59:02.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.540+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1481445f0 con 0x7fc14806e8d0 2024-06-29T15:59:02.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.542+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1480d01a0 con 0x7fc14806e8d0 2024-06-29T15:59:02.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.542+0000 7fc145ffb640 1 -- 172.21.15.59:0/3696647927 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fc14001b460 con 0x7fc14806e8d0 2024-06-29T15:59:02.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.543+0000 7fc145ffb640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fc134076510 0x7fc134078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.543+0000 7fc145ffb640 1 -- 172.21.15.59:0/3696647927 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 7201+0+0 (secure 0 0 0) 0x7fc1400984e0 con 0x7fc14806e8d0 2024-06-29T15:59:02.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.543+0000 7fc14d280640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fc134076510 0x7fc134078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.544+0000 7fc14d280640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fc134076510 0x7fc134078a20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc138035000 tx=0x7fc13802d040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:02.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.548+0000 7fc145ffb640 1 -- 172.21.15.59:0/3696647927 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc140060d10 con 0x7fc14806e8d0 2024-06-29T15:59:02.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:02.740+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc1480aec10 con 0x7fc14806e8d0 2024-06-29T15:59:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.353+0000 7fc145ffb640 1 -- 172.21.15.59:0/3696647927 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v253) ==== 190+0+0 (secure 0 0 0) 0x7fc1400693e0 con 0x7fc14806e8d0 2024-06-29T15:59:03.354 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'backup_fs_metadata' removed 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.357+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fc134076510 msgr2=0x7fc134078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.357+0000 7fc14f4e4640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fc134076510 0x7fc134078a20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc138035000 tx=0x7fc13802d040 comp rx=0 tx=0).stop 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.357+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc14806e8d0 msgr2=0x7fc14806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.357+0000 7fc14f4e4640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc14806e8d0 0x7fc14806ece0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fc14806ceb0 tx=0x7fc14000e980 comp rx=0 tx=0).stop 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.358+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 shutdown_connections 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.358+0000 7fc14f4e4640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fc134076510 0x7fc134078a20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.358+0000 7fc14f4e4640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc14806e8d0 0x7fc14806ece0 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.358+0000 7fc14f4e4640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc1481f4750 0x7fc148144df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.358+0000 7fc14f4e4640 1 --2- 172.21.15.59:0/3696647927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc130003a30 0x7fc14815b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.358+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 >> 172.21.15.59:0/3696647927 conn(0x7fc148073e10 msgr2=0x7fc148074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:03.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.358+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 shutdown_connections 2024-06-29T15:59:03.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.358+0000 7fc14f4e4640 1 -- 172.21.15.59:0/3696647927 wait complete. 2024-06-29T15:59:03.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-06-29T15:59:03.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.497+0000 7f8ec0238640 1 -- 172.21.15.59:0/1435977424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80b2980 msgr2=0x7f8eb80d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.497+0000 7f8ec0238640 1 --2- 172.21.15.59:0/1435977424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80b2980 0x7f8eb80d27a0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f8eac005fd0 tx=0x7f8eac02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.497+0000 7f8ec0238640 1 -- 172.21.15.59:0/1435977424 shutdown_connections 2024-06-29T15:59:03.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.497+0000 7f8ec0238640 1 --2- 172.21.15.59:0/1435977424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80b2980 0x7f8eb80d27a0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.497+0000 7f8ec0238640 1 --2- 172.21.15.59:0/1435977424 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb80d1c30 0x7f8eb80b6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.497+0000 7f8ec0238640 1 -- 172.21.15.59:0/1435977424 >> 172.21.15.59:0/1435977424 conn(0x7f8eb80d0f90 msgr2=0x7f8eb80d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:03.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.497+0000 7f8ec0238640 1 -- 172.21.15.59:0/1435977424 shutdown_connections 2024-06-29T15:59:03.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.497+0000 7f8ec0238640 1 -- 172.21.15.59:0/1435977424 wait complete. 2024-06-29T15:59:03.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.499+0000 7f8ec0238640 1 Processor -- start 2024-06-29T15:59:03.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.499+0000 7f8ec0238640 1 -- start start 2024-06-29T15:59:03.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ec0238640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80b2980 0x7f8eb81f5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:03.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ec0238640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb80d1c30 0x7f8eb8150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:03.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ec0238640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80d2ce0 0x7f8eb81f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:03.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ec0238640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb80b4780 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ec0238640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb80d3300 con 0x7f8eb80b2980 2024-06-29T15:59:03.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ec0238640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8eb80b4900 con 0x7f8eb80d1c30 2024-06-29T15:59:03.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ebe7d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80d2ce0 0x7f8eb81f89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:03.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ebe7d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80d2ce0 0x7f8eb81f89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46898/0 (socket says 172.21.15.59:46898) 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ebe7d5640 1 -- 172.21.15.59:0/3372109004 learned_addr learned my addr 172.21.15.59:0/3372109004 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ebdfd4640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80b2980 0x7f8eb81f5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.500+0000 7f8ebd7d3640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb80d1c30 0x7f8eb8150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8ebe7d5640 1 -- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb80d1c30 msgr2=0x7f8eb8150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8ebe7d5640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb80d1c30 0x7f8eb8150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8ebe7d5640 1 -- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80b2980 msgr2=0x7f8eb81f5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8ebe7d5640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80b2980 0x7f8eb81f5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8ebe7d5640 1 -- 172.21.15.59:0/3372109004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb81e6e50 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8ebe7d5640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80d2ce0 0x7f8eb81f89a0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000b410 tx=0x7f8eb000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8eb67fc640 1 -- 172.21.15.59:0/3372109004 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0015030 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8eb67fc640 1 -- 172.21.15.59:0/3372109004 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8eb0004020 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb81e7130 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8eb67fc640 1 -- 172.21.15.59:0/3372109004 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0004470 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.501+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb81e76f0 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.502+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eb80d2ad0 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.503+0000 7f8eb67fc640 1 -- 172.21.15.59:0/3372109004 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f8eb0018070 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.503+0000 7f8eb67fc640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8ea4076510 0x7f8ea4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:03.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.503+0000 7f8eb67fc640 1 -- 172.21.15.59:0/3372109004 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 6757+0+0 (secure 0 0 0) 0x7f8eb0097b30 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.504+0000 7f8ebdfd4640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8ea4076510 0x7f8ea4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:03.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.504+0000 7f8ebdfd4640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8ea4076510 0x7f8ea4078a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8005fd0 tx=0x7f8ea80097d0 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:03.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.508+0000 7f8eb67fc640 1 -- 172.21.15.59:0/3372109004 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8eb80d2ad0 con 0x7f8eb80d2ce0 2024-06-29T15:59:03.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:03.706+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8eb80d27a0 con 0x7f8eb80d2ce0 2024-06-29T15:59:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.355+0000 7f8eb67fc640 1 -- 172.21.15.59:0/3372109004 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v254) ==== 178+0+0 (secure 0 0 0) 0x7f8eb0060d10 con 0x7f8eb80d2ce0 2024-06-29T15:59:04.356 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'backup_fs_data' removed 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8ea4076510 msgr2=0x7f8ea4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8ea4076510 0x7f8ea4078a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8005fd0 tx=0x7f8ea80097d0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80d2ce0 msgr2=0x7f8eb81f89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80d2ce0 0x7f8eb81f89a0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000b410 tx=0x7f8eb000b8d0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 shutdown_connections 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8ea4076510 0x7f8ea4078a20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8eb80d2ce0 0x7f8eb81f89a0 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8eb80d1c30 0x7f8eb8150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 --2- 172.21.15.59:0/3372109004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80b2980 0x7f8eb81f5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.357+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 >> 172.21.15.59:0/3372109004 conn(0x7f8eb80d0f90 msgr2=0x7f8eb80b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.358+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 shutdown_connections 2024-06-29T15:59:04.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.358+0000 7f8ec0238640 1 -- 172.21.15.59:0/3372109004 wait complete. 2024-06-29T15:59:04.375 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:59:04.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.500+0000 7fd8930e4640 1 -- 172.21.15.59:0/1420310087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c0b3990 msgr2=0x7fd88c0b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.500+0000 7fd8930e4640 1 --2- 172.21.15.59:0/1420310087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c0b3990 0x7fd88c0b3da0 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fd878009860 tx=0x7fd87802a540 comp rx=0 tx=0).stop 2024-06-29T15:59:04.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.500+0000 7fd8930e4640 1 -- 172.21.15.59:0/1420310087 shutdown_connections 2024-06-29T15:59:04.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.500+0000 7fd8930e4640 1 --2- 172.21.15.59:0/1420310087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c0b3990 0x7fd88c0b3da0 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.500+0000 7fd8930e4640 1 -- 172.21.15.59:0/1420310087 >> 172.21.15.59:0/1420310087 conn(0x7fd88c0d3ee0 msgr2=0x7fd88c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.500+0000 7fd8930e4640 1 -- 172.21.15.59:0/1420310087 shutdown_connections 2024-06-29T15:59:04.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.500+0000 7fd8930e4640 1 -- 172.21.15.59:0/1420310087 wait complete. 2024-06-29T15:59:04.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.502+0000 7fd8930e4640 1 Processor -- start 2024-06-29T15:59:04.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.502+0000 7fd8930e4640 1 -- start start 2024-06-29T15:59:04.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd8930e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd88c0b3990 0x7fd88c0ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd8930e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88c0b73f0 0x7fd88c1e9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd8930e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c15e290 0x7fd88c160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd8930e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c0b7f10 con 0x7fd88c15e290 2024-06-29T15:59:04.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd8930e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c0cea30 con 0x7fd88c0b73f0 2024-06-29T15:59:04.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd8930e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd88c0b8090 con 0x7fd88c0b3990 2024-06-29T15:59:04.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c15e290 0x7fd88c160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c15e290 0x7fd88c160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46924/0 (socket says 172.21.15.59:46924) 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 -- 172.21.15.59:0/1273097310 learned_addr learned my addr 172.21.15.59:0/1273097310 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd890e80640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd88c0b3990 0x7fd88c0ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd88bfff640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88c0b73f0 0x7fd88c1e9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 -- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd88c0b3990 msgr2=0x7fd88c0ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd88c0b3990 0x7fd88c0ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 -- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88c0b73f0 msgr2=0x7fd88c1e9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88c0b73f0 0x7fd88c1e9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 -- 172.21.15.59:0/1273097310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c1e7050 con 0x7fd88c15e290 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.504+0000 7fd891681640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c15e290 0x7fd88c160fb0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fd88400b9e0 tx=0x7fd88400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.505+0000 7fd8897fa640 1 -- 172.21.15.59:0/1273097310 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884019070 con 0x7fd88c15e290 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.505+0000 7fd8897fa640 1 -- 172.21.15.59:0/1273097310 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd88400cc10 con 0x7fd88c15e290 2024-06-29T15:59:04.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.505+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88c1e7330 con 0x7fd88c15e290 2024-06-29T15:59:04.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.505+0000 7fd8897fa640 1 -- 172.21.15.59:0/1273097310 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8840146a0 con 0x7fd88c15e290 2024-06-29T15:59:04.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.505+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88c1e78f0 con 0x7fd88c15e290 2024-06-29T15:59:04.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.506+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd88c0b3da0 con 0x7fd88c15e290 2024-06-29T15:59:04.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.506+0000 7fd8897fa640 1 -- 172.21.15.59:0/1273097310 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fd88401c070 con 0x7fd88c15e290 2024-06-29T15:59:04.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.507+0000 7fd8897fa640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd870076510 0x7fd870078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.507+0000 7fd8897fa640 1 -- 172.21.15.59:0/1273097310 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7fd884097be0 con 0x7fd88c15e290 2024-06-29T15:59:04.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.507+0000 7fd890e80640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd870076510 0x7fd870078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.508+0000 7fd890e80640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd870076510 0x7fd870078a20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd880001720 tx=0x7fd880007e90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:04.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.512+0000 7fd8897fa640 1 -- 172.21.15.59:0/1273097310 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd884060880 con 0x7fd88c15e290 2024-06-29T15:59:04.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.702+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fd88c0ce0c0 con 0x7fd88c15e290 2024-06-29T15:59:04.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.707+0000 7fd8897fa640 1 -- 172.21.15.59:0/1273097310 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v79) ==== 71+0+0 (secure 0 0 0) 0x7fd884064b80 con 0x7fd88c15e290 2024-06-29T15:59:04.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd870076510 msgr2=0x7fd870078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd870076510 0x7fd870078a20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd880001720 tx=0x7fd880007e90 comp rx=0 tx=0).stop 2024-06-29T15:59:04.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c15e290 msgr2=0x7fd88c160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c15e290 0x7fd88c160fb0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fd88400b9e0 tx=0x7fd88400bea0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 shutdown_connections 2024-06-29T15:59:04.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fd870076510 0x7fd870078a20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd88c15e290 0x7fd88c160fb0 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd88c0b73f0 0x7fd88c1e9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 --2- 172.21.15.59:0/1273097310 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd88c0b3990 0x7fd88c0ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 >> 172.21.15.59:0/1273097310 conn(0x7fd88c0d3ee0 msgr2=0x7fd88c0cd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 shutdown_connections 2024-06-29T15:59:04.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.709+0000 7fd8930e4640 1 -- 172.21.15.59:0/1273097310 wait complete. 2024-06-29T15:59:04.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-06-29T15:59:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.844+0000 7fae31464640 1 -- 172.21.15.59:0/1223106009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0b2270 msgr2=0x7fae2c15bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.844+0000 7fae31464640 1 --2- 172.21.15.59:0/1223106009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0b2270 0x7fae2c15bc20 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fae2c0cf310 tx=0x7fae1c02a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.844+0000 7fae31464640 1 -- 172.21.15.59:0/1223106009 shutdown_connections 2024-06-29T15:59:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.844+0000 7fae31464640 1 --2- 172.21.15.59:0/1223106009 >> v2:172.21.15.153:3300/0 conn(0x7fae2c0d1c30 0x7fae2c0b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.844+0000 7fae31464640 1 --2- 172.21.15.59:0/1223106009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0b2270 0x7fae2c15bc20 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.844+0000 7fae31464640 1 -- 172.21.15.59:0/1223106009 >> 172.21.15.59:0/1223106009 conn(0x7fae2c073e10 msgr2=0x7fae2c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.845+0000 7fae31464640 1 -- 172.21.15.59:0/1223106009 shutdown_connections 2024-06-29T15:59:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.845+0000 7fae31464640 1 -- 172.21.15.59:0/1223106009 wait complete. 2024-06-29T15:59:04.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.846+0000 7fae31464640 1 Processor -- start 2024-06-29T15:59:04.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.846+0000 7fae31464640 1 -- start start 2024-06-29T15:59:04.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae31464640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae2c0b2270 0x7fae2c0aa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae31464640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae2c0b2fa0 0x7fae2c0aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae31464640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0d1c30 0x7fae2c15d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae31464640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c0b6400 con 0x7fae2c0d1c30 2024-06-29T15:59:04.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae31464640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c0b6280 con 0x7fae2c0b2fa0 2024-06-29T15:59:04.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae31464640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fae2c0b6580 con 0x7fae2c0b2270 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0d1c30 0x7fae2c15d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0d1c30 0x7fae2c15d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46934/0 (socket says 172.21.15.59:46934) 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 -- 172.21.15.59:0/1021116191 learned_addr learned my addr 172.21.15.59:0/1021116191 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2affd640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae2c0b2270 0x7fae2c0aa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae23fff640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae2c0b2fa0 0x7fae2c0aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 -- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae2c0b2270 msgr2=0x7fae2c0aa7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae2c0b2270 0x7fae2c0aa7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 -- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae2c0b2fa0 msgr2=0x7fae2c0aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae2c0b2fa0 0x7fae2c0aad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 -- 172.21.15.59:0/1021116191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2c1544d0 con 0x7fae2c0d1c30 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae23fff640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae2c0b2fa0 0x7fae2c0aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae2b7fe640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0d1c30 0x7fae2c15d450 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fae2400e990 tx=0x7fae2400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae237fe640 1 -- 172.21.15.59:0/1021116191 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24004d20 con 0x7fae2c0d1c30 2024-06-29T15:59:04.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae237fe640 1 -- 172.21.15.59:0/1021116191 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae2400cb00 con 0x7fae2c0d1c30 2024-06-29T15:59:04.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2c0a9a10 con 0x7fae2c0d1c30 2024-06-29T15:59:04.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.848+0000 7fae237fe640 1 -- 172.21.15.59:0/1021116191 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24005020 con 0x7fae2c0d1c30 2024-06-29T15:59:04.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.849+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2c0a9fd0 con 0x7fae2c0d1c30 2024-06-29T15:59:04.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.849+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2c15eb40 con 0x7fae2c0d1c30 2024-06-29T15:59:04.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.850+0000 7fae237fe640 1 -- 172.21.15.59:0/1021116191 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fae240051c0 con 0x7fae2c0d1c30 2024-06-29T15:59:04.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.851+0000 7fae237fe640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fae0c076510 0x7fae0c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.851+0000 7fae237fe640 1 -- 172.21.15.59:0/1021116191 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7fae240987f0 con 0x7fae2c0d1c30 2024-06-29T15:59:04.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.851+0000 7fae2affd640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fae0c076510 0x7fae0c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.852+0000 7fae2affd640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fae0c076510 0x7fae0c078a20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fae1c033000 tx=0x7fae1c004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:04.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:04.855+0000 7fae237fe640 1 -- 172.21.15.59:0/1021116191 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fae24016030 con 0x7fae2c0d1c30 2024-06-29T15:59:05.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.120+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) -- 0x7fae2c0b6aa0 con 0x7fae2c0d1c30 2024-06-29T15:59:05.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.365+0000 7fae237fe640 1 -- 172.21.15.59:0/1021116191 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3214) ==== 187+0+0 (secure 0 0 0) 0x7fae24096e20 con 0x7fae2c0d1c30 2024-06-29T15:59:05.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fae0c076510 msgr2=0x7fae0c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fae0c076510 0x7fae0c078a20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fae1c033000 tx=0x7fae1c004730 comp rx=0 tx=0).stop 2024-06-29T15:59:05.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0d1c30 msgr2=0x7fae2c15d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0d1c30 0x7fae2c15d450 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fae2400e990 tx=0x7fae2400ee50 comp rx=0 tx=0).stop 2024-06-29T15:59:05.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 shutdown_connections 2024-06-29T15:59:05.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fae0c076510 0x7fae0c078a20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae2c0d1c30 0x7fae2c15d450 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae2c0b2fa0 0x7fae2c0aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 --2- 172.21.15.59:0/1021116191 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fae2c0b2270 0x7fae2c0aa7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 >> 172.21.15.59:0/1021116191 conn(0x7fae2c073e10 msgr2=0x7fae2c0b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:05.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 shutdown_connections 2024-06-29T15:59:05.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.367+0000 7fae31464640 1 -- 172.21.15.59:0/1021116191 wait complete. 2024-06-29T15:59:05.380 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-06-29T15:59:05.380 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-06-29T15:59:05.380 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-06-29T15:59:05.381 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:05.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.499+0000 7f287bda1640 1 -- 172.21.15.59:0/2476420988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 msgr2=0x7f2860004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.499+0000 7f287bda1640 1 --2- 172.21.15.59:0/2476420988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 0x7f2860004c10 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f2864001690 tx=0x7f2860005830 comp rx=0 tx=0).stop 2024-06-29T15:59:05.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.499+0000 7f287bda1640 1 -- 172.21.15.59:0/2476420988 shutdown_connections 2024-06-29T15:59:05.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.499+0000 7f287bda1640 1 --2- 172.21.15.59:0/2476420988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 0x7f2860004c10 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.499+0000 7f287bda1640 1 -- 172.21.15.59:0/2476420988 >> 172.21.15.59:0/2476420988 conn(0x7f28740d3ee0 msgr2=0x7f28740d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:05.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.499+0000 7f287bda1640 1 -- 172.21.15.59:0/2476420988 shutdown_connections 2024-06-29T15:59:05.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.499+0000 7f287bda1640 1 -- 172.21.15.59:0/2476420988 wait complete. 2024-06-29T15:59:05.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.501+0000 7f287bda1640 1 Processor -- start 2024-06-29T15:59:05.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.501+0000 7f287bda1640 1 -- start start 2024-06-29T15:59:05.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f287bda1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 0x7f287414fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f287bda1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2874159a90 0x7f28741f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f287bda1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f287415c710 0x7f287415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f287bda1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f28740b3d20 con 0x7f2860004800 2024-06-29T15:59:05.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f287bda1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28740b3ba0 con 0x7f287415c710 2024-06-29T15:59:05.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f287bda1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f287415e8d0 con 0x7f2874159a90 2024-06-29T15:59:05.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f2879b3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 0x7f287414fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f2879b3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 0x7f287414fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46942/0 (socket says 172.21.15.59:46942) 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.502+0000 7f2879b3d640 1 -- 172.21.15.59:0/2564530742 learned_addr learned my addr 172.21.15.59:0/2564530742 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f287a33e640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f287415c710 0x7f287415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f287933c640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2874159a90 0x7f28741f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f2879b3d640 1 -- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2874159a90 msgr2=0x7f28741f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f2879b3d640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2874159a90 0x7f28741f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f2879b3d640 1 -- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f287415c710 msgr2=0x7f287415cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f2879b3d640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f287415c710 0x7f287415cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f2879b3d640 1 -- 172.21.15.59:0/2564530742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2874158ff0 con 0x7f2860004800 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f287933c640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2874159a90 0x7f28741f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f287a33e640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f287415c710 0x7f287415cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f2879b3d640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 0x7f287414fe20 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f2870011a10 tx=0x7f2870011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:05.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f286a7fc640 1 -- 172.21.15.59:0/2564530742 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287001e070 con 0x7f2860004800 2024-06-29T15:59:05.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f286a7fc640 1 -- 172.21.15.59:0/2564530742 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f287000fb50 con 0x7f2860004800 2024-06-29T15:59:05.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2874150740 con 0x7f2860004800 2024-06-29T15:59:05.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f286a7fc640 1 -- 172.21.15.59:0/2564530742 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870019410 con 0x7f2860004800 2024-06-29T15:59:05.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.503+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28741f62e0 con 0x7f2860004800 2024-06-29T15:59:05.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.504+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f287415f9b0 con 0x7f2860004800 2024-06-29T15:59:05.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.505+0000 7f286a7fc640 1 -- 172.21.15.59:0/2564530742 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f2870021070 con 0x7f2860004800 2024-06-29T15:59:05.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.505+0000 7f286a7fc640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2858076510 0x7f2858078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.506+0000 7f286a7fc640 1 -- 172.21.15.59:0/2564530742 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7f2870098670 con 0x7f2860004800 2024-06-29T15:59:05.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.506+0000 7f287933c640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2858076510 0x7f2858078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.506+0000 7f287933c640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2858076510 0x7f2858078a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2864032000 tx=0x7f2864001190 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:05.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.510+0000 7f286a7fc640 1 -- 172.21.15.59:0/2564530742 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2870061200 con 0x7f2860004800 2024-06-29T15:59:05.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.704+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28740d2950 con 0x7f2860004800 2024-06-29T15:59:05.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.705+0000 7f286a7fc640 1 -- 172.21.15.59:0/2564530742 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) ==== 74+0+14804 (secure 0 0 0) 0x7f2870065500 con 0x7f2860004800 2024-06-29T15:59:05.706 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:05.707 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":254,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:04.354940+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/1525995706":"2024-06-30T15:59:01.349755+0000","172.21.15.153:6834/177893499":"2024-06-30T15:58:54.800279+0000","172.21.15.153:6832/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2374956971":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/2562681960":"2024-06-30T15:58:51.014782+0000","172.21.15.153:6837/177893499":"2024-06-30T15:58:54.800279+0000","172.21.15.153:0/221125627":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3082371830":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1656484964":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/402968645":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1556406125":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/877026286":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/3972699112":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1636302558":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1944500160":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1892969816":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/3355566242":"2024-06-30T15:58:45.316819+0000","172.21.15.59:6835/1525995706":"2024-06-30T15:59:01.349755+0000","172.21.15.153:6833/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6832/1831136380":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/2494131047":"2024-06-30T15:58:45.316819+0000","172.21.15.153:0/1461412440":"2024-06-30T15:58:45.316819+0000","172.21.15.153:6833/2715634623":"2024-06-30T15:58:51.014782+0000","172.21.15.153:0/1259499350":"2024-06-30T15:58:45.316819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:05.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2858076510 msgr2=0x7f2858078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2858076510 0x7f2858078a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2864032000 tx=0x7f2864001190 comp rx=0 tx=0).stop 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 msgr2=0x7f287414fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 0x7f287414fe20 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f2870011a10 tx=0x7f2870011ed0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 shutdown_connections 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2858076510 0x7f2858078a20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f287415c710 0x7f287415cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2874159a90 0x7f28741f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 --2- 172.21.15.59:0/2564530742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2860004800 0x7f287414fe20 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 >> 172.21.15.59:0/2564530742 conn(0x7f28740d3ee0 msgr2=0x7f28740d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.708+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 shutdown_connections 2024-06-29T15:59:05.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.709+0000 7f287bda1640 1 -- 172.21.15.59:0/2564530742 wait complete. 2024-06-29T15:59:05.720 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:05.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:05.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.840+0000 7f3e0688f640 1 -- 172.21.15.59:0/3966320128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000d02f0 msgr2=0x7f3e000cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.840+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3966320128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000d02f0 0x7f3e000cd280 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f3e000cf980 tx=0x7f3df402a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.840+0000 7f3e0688f640 1 -- 172.21.15.59:0/3966320128 shutdown_connections 2024-06-29T15:59:05.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.840+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3966320128 >> v2:172.21.15.153:3300/0 conn(0x7f3e000cd7c0 0x7f3e0006ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.840+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3966320128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000d02f0 0x7f3e000cd280 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.840+0000 7f3e0688f640 1 -- 172.21.15.59:0/3966320128 >> 172.21.15.59:0/3966320128 conn(0x7f3e000b2ab0 msgr2=0x7f3e000b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:05.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.840+0000 7f3e0688f640 1 -- 172.21.15.59:0/3966320128 shutdown_connections 2024-06-29T15:59:05.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.840+0000 7f3e0688f640 1 -- 172.21.15.59:0/3966320128 wait complete. 2024-06-29T15:59:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.842+0000 7f3e0688f640 1 Processor -- start 2024-06-29T15:59:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.842+0000 7f3e0688f640 1 -- start start 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e0688f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000cd7c0 0x7f3e001419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e0688f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e001448e0 0x7f3e00159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e0688f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0013f3f0 0x7f3e000b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e0688f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0006d3c0 con 0x7f3e000cd7c0 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e0688f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0006d240 con 0x7f3e001448e0 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e0688f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0006d540 con 0x7f3e0013f3f0 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000cd7c0 0x7f3e001419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000cd7c0 0x7f3e001419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46952/0 (socket says 172.21.15.59:46952) 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 -- 172.21.15.59:0/3785080835 learned_addr learned my addr 172.21.15.59:0/3785080835 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dff7fe640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e001448e0 0x7f3e00159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e04e2c640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0013f3f0 0x7f3e000b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 -- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0013f3f0 msgr2=0x7f3e000b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0013f3f0 0x7f3e000b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 -- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e001448e0 msgr2=0x7f3e00159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e001448e0 0x7f3e00159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 -- 172.21.15.59:0/3785080835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e00159ac0 con 0x7f3e000cd7c0 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e04e2c640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0013f3f0 0x7f3e000b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfffff640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000cd7c0 0x7f3e001419c0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f3df000c400 tx=0x7f3df000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfcff9640 1 -- 172.21.15.59:0/3785080835 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0017070 con 0x7f3e000cd7c0 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfcff9640 1 -- 172.21.15.59:0/3785080835 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3df00053c0 con 0x7f3e000cd7c0 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e00140400 con 0x7f3e000cd7c0 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3dfcff9640 1 -- 172.21.15.59:0/3785080835 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df0003850 con 0x7f3e000cd7c0 2024-06-29T15:59:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.844+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e0013fe40 con 0x7f3e000cd7c0 2024-06-29T15:59:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.845+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e0006da60 con 0x7f3e000cd7c0 2024-06-29T15:59:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.846+0000 7f3dfcff9640 1 -- 172.21.15.59:0/3785080835 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f3df001a070 con 0x7f3e000cd7c0 2024-06-29T15:59:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.847+0000 7f3dfcff9640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3dec076540 0x7f3dec078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.847+0000 7f3dfcff9640 1 -- 172.21.15.59:0/3785080835 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7f3df00970a0 con 0x7f3e000cd7c0 2024-06-29T15:59:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.847+0000 7f3dff7fe640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3dec076540 0x7f3dec078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.848+0000 7f3dff7fe640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3dec076540 0x7f3dec078a50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3df4033000 tx=0x7f3df4004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:05.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:05.852+0000 7f3dfcff9640 1 -- 172.21.15.59:0/3785080835 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3e00159ac0 con 0x7f3e000cd7c0 2024-06-29T15:59:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.040+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e000b2010 con 0x7f3e000cd7c0 2024-06-29T15:59:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.040+0000 7f3dfcff9640 1 -- 172.21.15.59:0/3785080835 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v254) ==== 93+0+10 (secure 0 0 0) 0x7f3df005fe10 con 0x7f3e000cd7c0 2024-06-29T15:59:06.042 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:59:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3dec076540 msgr2=0x7f3dec078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3dec076540 0x7f3dec078a50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3df4033000 tx=0x7f3df4004730 comp rx=0 tx=0).stop 2024-06-29T15:59:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000cd7c0 msgr2=0x7f3e001419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000cd7c0 0x7f3e001419c0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f3df000c400 tx=0x7f3df000c8c0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 shutdown_connections 2024-06-29T15:59:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3dec076540 0x7f3dec078a50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3e0013f3f0 0x7f3e000b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e001448e0 0x7f3e00159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 --2- 172.21.15.59:0/3785080835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e000cd7c0 0x7f3e001419c0 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 >> 172.21.15.59:0/3785080835 conn(0x7f3e000b2ab0 msgr2=0x7f3e00068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 shutdown_connections 2024-06-29T15:59:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.043+0000 7f3e0688f640 1 -- 172.21.15.59:0/3785080835 wait complete. 2024-06-29T15:59:06.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-06-29T15:59:06.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.172+0000 7fcbc0e95640 1 -- 172.21.15.59:0/2128902565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcba40053a0 msgr2=0x7fcba400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.172+0000 7fcbc0e95640 1 --2- 172.21.15.59:0/2128902565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcba40053a0 0x7fcba400ba20 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4005fd0 tx=0x7fcba4005830 comp rx=0 tx=0).stop 2024-06-29T15:59:06.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.172+0000 7fcbc0e95640 1 -- 172.21.15.59:0/2128902565 shutdown_connections 2024-06-29T15:59:06.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.172+0000 7fcbc0e95640 1 --2- 172.21.15.59:0/2128902565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcba40053a0 0x7fcba400ba20 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.172+0000 7fcbc0e95640 1 -- 172.21.15.59:0/2128902565 >> 172.21.15.59:0/2128902565 conn(0x7fcbbc0b6fb0 msgr2=0x7fcbbc0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:06.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.172+0000 7fcbc0e95640 1 -- 172.21.15.59:0/2128902565 shutdown_connections 2024-06-29T15:59:06.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.172+0000 7fcbc0e95640 1 -- 172.21.15.59:0/2128902565 wait complete. 2024-06-29T15:59:06.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.174+0000 7fcbc0e95640 1 Processor -- start 2024-06-29T15:59:06.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.174+0000 7fcbc0e95640 1 -- start start 2024-06-29T15:59:06.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.175+0000 7fcbc0e95640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcba40053a0 0x7fcbbc155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.175+0000 7fcbc0e95640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbbc133970 0x7fcbbc152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.175+0000 7fcbc0e95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc15e570 0x7fcbbc15e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.175+0000 7fcbc0e95640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc0b5850 con 0x7fcbbc133970 2024-06-29T15:59:06.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.175+0000 7fcbc0e95640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc0b56d0 con 0x7fcbbc15e570 2024-06-29T15:59:06.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.175+0000 7fcbc0e95640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcbbc0b59d0 con 0x7fcba40053a0 2024-06-29T15:59:06.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbbc133970 0x7fcbbc152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbbc133970 0x7fcbbc152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46962/0 (socket says 172.21.15.59:46962) 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 -- 172.21.15.59:0/1752324523 learned_addr learned my addr 172.21.15.59:0/1752324523 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbba59c640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcba40053a0 0x7fcbbc155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbbad9d640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc15e570 0x7fcbbc15e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 -- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcba40053a0 msgr2=0x7fcbbc155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcba40053a0 0x7fcbbc155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 -- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc15e570 msgr2=0x7fcbbc15e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc15e570 0x7fcbbc15e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 -- 172.21.15.59:0/1752324523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbbc15eec0 con 0x7fcbbc133970 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbb9d9b640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbbc133970 0x7fcbbc152700 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0004ed0 tx=0x7fcbb0001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbaaffd640 1 -- 172.21.15.59:0/1752324523 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0015020 con 0x7fcbbc133970 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbaaffd640 1 -- 172.21.15.59:0/1752324523 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbb0011960 con 0x7fcbbc133970 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbbc1530c0 con 0x7fcbbc133970 2024-06-29T15:59:06.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbaaffd640 1 -- 172.21.15.59:0/1752324523 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0011d90 con 0x7fcbbc133970 2024-06-29T15:59:06.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.176+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbbc153680 con 0x7fcbbc133970 2024-06-29T15:59:06.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.177+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbbc1415a0 con 0x7fcbbc133970 2024-06-29T15:59:06.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.178+0000 7fcbaaffd640 1 -- 172.21.15.59:0/1752324523 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fcbb0018020 con 0x7fcbbc133970 2024-06-29T15:59:06.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.179+0000 7fcbaaffd640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcb9c076510 0x7fcb9c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.179+0000 7fcbaaffd640 1 -- 172.21.15.59:0/1752324523 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7fcbb00987f0 con 0x7fcbbc133970 2024-06-29T15:59:06.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.179+0000 7fcbba59c640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcb9c076510 0x7fcb9c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.179+0000 7fcbba59c640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcb9c076510 0x7fcb9c078a20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0018b0 tx=0x7fcbac006720 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:06.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.183+0000 7fcbaaffd640 1 -- 172.21.15.59:0/1752324523 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcbb001f020 con 0x7fcbbc133970 2024-06-29T15:59:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:06.448+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) -- 0x7fcbbc15a6e0 con 0x7fcbbc133970 2024-06-29T15:59:07.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.374+0000 7fcbaaffd640 1 -- 172.21.15.59:0/1752324523 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3216) ==== 158+0+0 (secure 0 0 0) 0x7fcbbc15eec0 con 0x7fcbbc133970 2024-06-29T15:59:07.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcb9c076510 msgr2=0x7fcb9c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcb9c076510 0x7fcb9c078a20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0018b0 tx=0x7fcbac006720 comp rx=0 tx=0).stop 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbbc133970 msgr2=0x7fcbbc152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbbc133970 0x7fcbbc152700 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0004ed0 tx=0x7fcbb0001630 comp rx=0 tx=0).stop 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 shutdown_connections 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcb9c076510 0x7fcb9c078a20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbbc15e570 0x7fcbbc15e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbbc133970 0x7fcbbc152700 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.377+0000 7fcbc0e95640 1 --2- 172.21.15.59:0/1752324523 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcba40053a0 0x7fcbbc155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.378+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 >> 172.21.15.59:0/1752324523 conn(0x7fcbbc0b6fb0 msgr2=0x7fcbbc0b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.378+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 shutdown_connections 2024-06-29T15:59:07.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.378+0000 7fcbc0e95640 1 -- 172.21.15.59:0/1752324523 wait complete. 2024-06-29T15:59:07.392 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:59:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.517+0000 7fdd46924640 1 -- 172.21.15.59:0/4001605055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd400ce950 msgr2=0x7fdd401622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.517+0000 7fdd46924640 1 --2- 172.21.15.59:0/4001605055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd400ce950 0x7fdd401622c0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fdd34005fd0 tx=0x7fdd3402a4f0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.517+0000 7fdd46924640 1 -- 172.21.15.59:0/4001605055 shutdown_connections 2024-06-29T15:59:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.517+0000 7fdd46924640 1 --2- 172.21.15.59:0/4001605055 >> v2:172.21.15.153:3300/0 conn(0x7fdd40162800 0x7fdd40162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.517+0000 7fdd46924640 1 --2- 172.21.15.59:0/4001605055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd400ce950 0x7fdd401622c0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.517+0000 7fdd46924640 1 -- 172.21.15.59:0/4001605055 >> 172.21.15.59:0/4001605055 conn(0x7fdd400b5610 msgr2=0x7fdd400b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.517+0000 7fdd46924640 1 -- 172.21.15.59:0/4001605055 shutdown_connections 2024-06-29T15:59:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.517+0000 7fdd46924640 1 -- 172.21.15.59:0/4001605055 wait complete. 2024-06-29T15:59:07.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.519+0000 7fdd46924640 1 Processor -- start 2024-06-29T15:59:07.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.519+0000 7fdd46924640 1 -- start start 2024-06-29T15:59:07.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd46924640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd4008c190 0x7fdd4015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd46924640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd400ce950 0x7fdd4015edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd46924640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd40162800 0x7fdd400ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd46924640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdd400b7f60 con 0x7fdd4008c190 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd46924640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd400b7de0 con 0x7fdd400ce950 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd46924640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd400b80e0 con 0x7fdd40162800 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd3ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd4008c190 0x7fdd4015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd3ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd4008c190 0x7fdd4015b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46986/0 (socket says 172.21.15.59:46986) 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd3ffff640 1 -- 172.21.15.59:0/3893999440 learned_addr learned my addr 172.21.15.59:0/3893999440 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.520+0000 7fdd3f7fe640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd400ce950 0x7fdd4015edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd44ec1640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd40162800 0x7fdd400ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3ffff640 1 -- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd40162800 msgr2=0x7fdd400ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3ffff640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd40162800 0x7fdd400ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3ffff640 1 -- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd400ce950 msgr2=0x7fdd4015edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3ffff640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd400ce950 0x7fdd4015edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3ffff640 1 -- 172.21.15.59:0/3893999440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4015fab0 con 0x7fdd4008c190 2024-06-29T15:59:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd44ec1640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd40162800 0x7fdd400ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3ffff640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd4008c190 0x7fdd4015b330 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b420 tx=0x7fdd3000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3cff9640 1 -- 172.21.15.59:0/3893999440 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30011830 con 0x7fdd4008c190 2024-06-29T15:59:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3cff9640 1 -- 172.21.15.59:0/3893999440 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd30016070 con 0x7fdd4008c190 2024-06-29T15:59:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4015fd90 con 0x7fdd4008c190 2024-06-29T15:59:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd3cff9640 1 -- 172.21.15.59:0/3893999440 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30011d60 con 0x7fdd4008c190 2024-06-29T15:59:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.521+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd401605b0 con 0x7fdd4008c190 2024-06-29T15:59:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.522+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd401622c0 con 0x7fdd4008c190 2024-06-29T15:59:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.523+0000 7fdd3cff9640 1 -- 172.21.15.59:0/3893999440 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fdd3001b460 con 0x7fdd4008c190 2024-06-29T15:59:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.524+0000 7fdd3cff9640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fdd2c076540 0x7fdd2c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.524+0000 7fdd3cff9640 1 -- 172.21.15.59:0/3893999440 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7fdd30097b40 con 0x7fdd4008c190 2024-06-29T15:59:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.524+0000 7fdd3f7fe640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fdd2c076540 0x7fdd2c078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.525+0000 7fdd3f7fe640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fdd2c076540 0x7fdd2c078a50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdd34033000 tx=0x7fdd34004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:07.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.528+0000 7fdd3cff9640 1 -- 172.21.15.59:0/3893999440 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd30060e30 con 0x7fdd4008c190 2024-06-29T15:59:07.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.718+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fdd40075bb0 con 0x7fdd4008c190 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.719+0000 7fdd3cff9640 1 -- 172.21.15.59:0/3893999440 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v79) ==== 59+0+1998 (secure 0 0 0) 0x7fdd30060730 con 0x7fdd4008c190 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T15:59:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T15:59:07.721 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T15:59:07.722 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T15:59:07.722 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:59:07.722 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fdd2c076540 msgr2=0x7fdd2c078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fdd2c076540 0x7fdd2c078a50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdd34033000 tx=0x7fdd34004730 comp rx=0 tx=0).stop 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd4008c190 msgr2=0x7fdd4015b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd4008c190 0x7fdd4015b330 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b420 tx=0x7fdd3000b8e0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 shutdown_connections 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fdd2c076540 0x7fdd2c078a50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd40162800 0x7fdd400ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd400ce950 0x7fdd4015edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 --2- 172.21.15.59:0/3893999440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd4008c190 0x7fdd4015b330 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 >> 172.21.15.59:0/3893999440 conn(0x7fdd400b5610 msgr2=0x7fdd400c7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 shutdown_connections 2024-06-29T15:59:07.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.721+0000 7fdd46924640 1 -- 172.21.15.59:0/3893999440 wait complete. 2024-06-29T15:59:07.733 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:59:07.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.852+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/2291419029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00d23c0 msgr2=0x7f5ad00b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.852+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/2291419029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00d23c0 0x7f5ad00b3990 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009830 tx=0x7f5ac002a510 comp rx=0 tx=0).stop 2024-06-29T15:59:07.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.852+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/2291419029 shutdown_connections 2024-06-29T15:59:07.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.852+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/2291419029 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ad015efe0 0x7f5ad015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.852+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/2291419029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00d23c0 0x7f5ad00b3990 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.852+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/2291419029 >> 172.21.15.59:0/2291419029 conn(0x7f5ad00d3ee0 msgr2=0x7f5ad00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:07.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.852+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/2291419029 shutdown_connections 2024-06-29T15:59:07.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.852+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/2291419029 wait complete. 2024-06-29T15:59:07.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.854+0000 7f5ad5f7a640 1 Processor -- start 2024-06-29T15:59:07.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.854+0000 7f5ad5f7a640 1 -- start start 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.855+0000 7f5ad5f7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00b3ed0 0x7f5ad01546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.855+0000 7f5ad5f7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad00d23c0 0x7f5ad00c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.855+0000 7f5ad5f7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ad015efe0 0x7f5ad01e8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.855+0000 7f5ad5f7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0160280 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.855+0000 7f5ad5f7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad0160100 con 0x7f5ad00d23c0 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.855+0000 7f5ad5f7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad0160400 con 0x7f5ad015efe0 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00b3ed0 0x7f5ad01546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00b3ed0 0x7f5ad01546d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47014/0 (socket says 172.21.15.59:47014) 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 -- 172.21.15.59:0/3797762112 learned_addr learned my addr 172.21.15.59:0/3797762112 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acffff640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ad015efe0 0x7f5ad01e8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5aceffd640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad00d23c0 0x7f5ad00c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 -- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ad015efe0 msgr2=0x7f5ad01e8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ad015efe0 0x7f5ad01e8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 -- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad00d23c0 msgr2=0x7f5ad00c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad00d23c0 0x7f5ad00c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 -- 172.21.15.59:0/3797762112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad0154c10 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5acf7fe640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00b3ed0 0x7f5ad01546d0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f5ac002ab00 tx=0x7f5ac0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5ab7fff640 1 -- 172.21.15.59:0/3797762112 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0031d60 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5ab7fff640 1 -- 172.21.15.59:0/3797762112 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ac0033e10 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5ab7fff640 1 -- 172.21.15.59:0/3797762112 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0005260 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad0156030 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.856+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad01562d0 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.857+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad00b3990 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.858+0000 7f5ab7fff640 1 -- 172.21.15.59:0/3797762112 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f5ac0005400 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.859+0000 7f5ab7fff640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5abc076510 0x7f5abc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.859+0000 7f5ab7fff640 1 -- 172.21.15.59:0/3797762112 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7f5ac00b91d0 con 0x7f5ad00b3ed0 2024-06-29T15:59:07.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.859+0000 7f5aceffd640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5abc076510 0x7f5abc078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.860+0000 7f5aceffd640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5abc076510 0x7f5abc078a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4007fd0 tx=0x7f5ac4009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:07.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:07.863+0000 7f5ab7fff640 1 -- 172.21.15.59:0/3797762112 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ac003f030 con 0x7f5ad00b3ed0 2024-06-29T15:59:08.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.053+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5ad00d2950 con 0x7f5ad00b3ed0 2024-06-29T15:59:08.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.057+0000 7f5ab7fff640 1 -- 172.21.15.59:0/3797762112 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v80)=0 v80) ==== 122+0+0 (secure 0 0 0) 0x7f5ac0082480 con 0x7f5ad00b3ed0 2024-06-29T15:59:08.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5abc076510 msgr2=0x7f5abc078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5abc076510 0x7f5abc078a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4007fd0 tx=0x7f5ac4009040 comp rx=0 tx=0).stop 2024-06-29T15:59:08.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00b3ed0 msgr2=0x7f5ad01546d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00b3ed0 0x7f5ad01546d0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f5ac002ab00 tx=0x7f5ac0033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 shutdown_connections 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f5abc076510 0x7f5abc078a20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ad015efe0 0x7f5ad01e8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ad00d23c0 0x7f5ad00c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 --2- 172.21.15.59:0/3797762112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ad00b3ed0 0x7f5ad01546d0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 >> 172.21.15.59:0/3797762112 conn(0x7f5ad00d3ee0 msgr2=0x7f5ad00ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 shutdown_connections 2024-06-29T15:59:08.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.059+0000 7f5ad5f7a640 1 -- 172.21.15.59:0/3797762112 wait complete. 2024-06-29T15:59:08.071 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:59:08.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:08.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:08.147 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:59:08.147 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:08.147 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:08.165 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:08.165 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:59:08.165 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:08.205 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:08.206 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:08.206 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:08.221 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:08.222 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:59:08.222 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:08.263 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:08.263 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:59:08.263 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:08.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.380+0000 7f480ae4e640 1 -- 172.21.15.59:0/3103050279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2270 msgr2=0x7f480415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.380+0000 7f480ae4e640 1 --2- 172.21.15.59:0/3103050279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2270 0x7f480415bc20 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009830 tx=0x7f47f802a5c0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.380+0000 7f480ae4e640 1 -- 172.21.15.59:0/3103050279 shutdown_connections 2024-06-29T15:59:08.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.380+0000 7f480ae4e640 1 --2- 172.21.15.59:0/3103050279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2270 0x7f480415bc20 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.380+0000 7f480ae4e640 1 -- 172.21.15.59:0/3103050279 >> 172.21.15.59:0/3103050279 conn(0x7f4804073e10 msgr2=0x7f4804074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:08.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.380+0000 7f480ae4e640 1 -- 172.21.15.59:0/3103050279 shutdown_connections 2024-06-29T15:59:08.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.380+0000 7f480ae4e640 1 -- 172.21.15.59:0/3103050279 wait complete. 2024-06-29T15:59:08.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.382+0000 7f480ae4e640 1 Processor -- start 2024-06-29T15:59:08.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.382+0000 7f480ae4e640 1 -- start start 2024-06-29T15:59:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f480ae4e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040b2270 0x7f480415e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f480ae4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2fa0 0x7f48041f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f480ae4e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040d1c30 0x7f4804147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f480ae4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f48040b65d0 con 0x7f48040b2fa0 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f480ae4e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f48040b6450 con 0x7f48040b2270 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f480ae4e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f48040b6750 con 0x7f48040d1c30 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f4803fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2fa0 0x7f48041f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f4803fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2fa0 0x7f48041f3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47028/0 (socket says 172.21.15.59:47028) 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f4803fff640 1 -- 172.21.15.59:0/2352759957 learned_addr learned my addr 172.21.15.59:0/2352759957 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f4808bea640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040b2270 0x7f480415e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.383+0000 7f48093eb640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040d1c30 0x7f4804147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f4803fff640 1 -- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040d1c30 msgr2=0x7f4804147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f4803fff640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040d1c30 0x7f4804147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f4803fff640 1 -- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040b2270 msgr2=0x7f480415e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f4803fff640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040b2270 0x7f480415e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f4803fff640 1 -- 172.21.15.59:0/2352759957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480415d910 con 0x7f48040b2fa0 2024-06-29T15:59:08.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f4803fff640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2fa0 0x7f48041f3960 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f47f000c3d0 tx=0x7f47f000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:08.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f48017fa640 1 -- 172.21.15.59:0/2352759957 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0017070 con 0x7f48040b2fa0 2024-06-29T15:59:08.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f48017fa640 1 -- 172.21.15.59:0/2352759957 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47f00053c0 con 0x7f48040b2fa0 2024-06-29T15:59:08.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480415dbf0 con 0x7f48040b2fa0 2024-06-29T15:59:08.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f48017fa640 1 -- 172.21.15.59:0/2352759957 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0003850 con 0x7f48040b2fa0 2024-06-29T15:59:08.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.384+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f480415e1b0 con 0x7f48040b2fa0 2024-06-29T15:59:08.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.385+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48040b6c70 con 0x7f48040b2fa0 2024-06-29T15:59:08.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.386+0000 7f48017fa640 1 -- 172.21.15.59:0/2352759957 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f47f001a070 con 0x7f48040b2fa0 2024-06-29T15:59:08.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.386+0000 7f48017fa640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47e8076540 0x7f47e8078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.387+0000 7f48017fa640 1 -- 172.21.15.59:0/2352759957 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7f47f00971b0 con 0x7f48040b2fa0 2024-06-29T15:59:08.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.387+0000 7f4808bea640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47e8076540 0x7f47e8078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.387+0000 7f4808bea640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47e8076540 0x7f47e8078a50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f47f8034000 tx=0x7f47f8004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:08.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.391+0000 7f48017fa640 1 -- 172.21.15.59:0/2352759957 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47f005fe80 con 0x7f48040b2fa0 2024-06-29T15:59:08.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.620+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f480415be30 con 0x7f48040b2fa0 2024-06-29T15:59:08.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.620+0000 7f48017fa640 1 -- 172.21.15.59:0/2352759957 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+3922 (secure 0 0 0) 0x7f47f0064180 con 0x7f48040b2fa0 2024-06-29T15:59:08.621 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:08.621 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":141,"btime":"2024-06-29T15:59:02:351525+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9971,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/2599560908","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2599560908},{"type":"v1","addr":"172.21.15.153:6836","nonce":2599560908}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127},{"gid":10358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/555027351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":555027351},{"type":"v1","addr":"172.21.15.153:6837","nonce":555027351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":12300,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/4120337310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":4120337310},{"type":"v1","addr":"172.21.15.59:6833","nonce":4120337310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126},{"gid":13020,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2634186730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2634186730},{"type":"v1","addr":"172.21.15.59:6835","nonce":2634186730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[]} 2024-06-29T15:59:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 141 2024-06-29T15:59:08.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47e8076540 msgr2=0x7f47e8078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47e8076540 0x7f47e8078a50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f47f8034000 tx=0x7f47f8004730 comp rx=0 tx=0).stop 2024-06-29T15:59:08.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2fa0 msgr2=0x7f48041f3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2fa0 0x7f48041f3960 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f47f000c3d0 tx=0x7f47f000c890 comp rx=0 tx=0).stop 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 shutdown_connections 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47e8076540 0x7f47e8078a50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f48040d1c30 0x7f4804147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040b2fa0 0x7f48041f3960 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 --2- 172.21.15.59:0/2352759957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48040b2270 0x7f480415e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 >> 172.21.15.59:0/2352759957 conn(0x7f4804073e10 msgr2=0x7f4804070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 shutdown_connections 2024-06-29T15:59:08.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.622+0000 7f480ae4e640 1 -- 172.21.15.59:0/2352759957 wait complete. 2024-06-29T15:59:08.635 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:59:08.635 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:59:08.635 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:59:08.635 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:08.635 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:59:08.635 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:59:08.636 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:59:08.636 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:08.636 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:59:08.636 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:59:08.636 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:59:08.636 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:08.636 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:59:08.636 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:59:08.637 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:59:08.637 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:08.645 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:59:08.645 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:59:08.648 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:59:08.648 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:59:08.649 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:59:08.649 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:59:08.650 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:59:08.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:59:08.690 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:59:08.690 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:08.691 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:59:08.691 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:08.692 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:59:08.692 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:08.694 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:59:08.694 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:08.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:59:08.779 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:59:08.779 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:59:08.784 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:59:08.784 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T15:59:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.818+0000 7fde16865640 1 -- 172.21.15.59:0/1363137765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b3990 msgr2=0x7fde100b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.818+0000 7fde16865640 1 --2- 172.21.15.59:0/1363137765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b3990 0x7fde100b3da0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fde04009860 tx=0x7fde0402a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.818+0000 7fde16865640 1 -- 172.21.15.59:0/1363137765 shutdown_connections 2024-06-29T15:59:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.818+0000 7fde16865640 1 --2- 172.21.15.59:0/1363137765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b3990 0x7fde100b3da0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.818+0000 7fde16865640 1 -- 172.21.15.59:0/1363137765 >> 172.21.15.59:0/1363137765 conn(0x7fde100d3ee0 msgr2=0x7fde100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.818+0000 7fde16865640 1 -- 172.21.15.59:0/1363137765 shutdown_connections 2024-06-29T15:59:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.818+0000 7fde16865640 1 -- 172.21.15.59:0/1363137765 wait complete. 2024-06-29T15:59:08.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.819+0000 7fde16865640 1 Processor -- start 2024-06-29T15:59:08.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.820+0000 7fde16865640 1 -- start start 2024-06-29T15:59:08.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde16865640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde100b3990 0x7fde101e7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde16865640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b73f0 0x7fde10149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde16865640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde1015e290 0x7fde101e8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde16865640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fde100b7f10 con 0x7fde100b73f0 2024-06-29T15:59:08.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde16865640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde100cea30 con 0x7fde100b3990 2024-06-29T15:59:08.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde16865640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fde100b8090 con 0x7fde1015e290 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b73f0 0x7fde10149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b73f0 0x7fde10149e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47064/0 (socket says 172.21.15.59:47064) 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0f7fe640 1 -- 172.21.15.59:0/1257159603 learned_addr learned my addr 172.21.15.59:0/1257159603 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0ffff640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde100b3990 0x7fde101e7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde14e02640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde1015e290 0x7fde101e8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0f7fe640 1 -- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde1015e290 msgr2=0x7fde101e8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0f7fe640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde1015e290 0x7fde101e8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0f7fe640 1 -- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde100b3990 msgr2=0x7fde101e7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0f7fe640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde100b3990 0x7fde101e7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.821+0000 7fde0f7fe640 1 -- 172.21.15.59:0/1257159603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde101e69e0 con 0x7fde100b73f0 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.822+0000 7fde0f7fe640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b73f0 0x7fde10149e30 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fde0402abe0 tx=0x7fde04033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.822+0000 7fde0cff9640 1 -- 172.21.15.59:0/1257159603 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04031b60 con 0x7fde100b73f0 2024-06-29T15:59:08.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.822+0000 7fde0cff9640 1 -- 172.21.15.59:0/1257159603 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde04031df0 con 0x7fde100b73f0 2024-06-29T15:59:08.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.822+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde101e6cc0 con 0x7fde100b73f0 2024-06-29T15:59:08.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.822+0000 7fde0cff9640 1 -- 172.21.15.59:0/1257159603 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04005280 con 0x7fde100b73f0 2024-06-29T15:59:08.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.822+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde101e7280 con 0x7fde100b73f0 2024-06-29T15:59:08.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.823+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde101e9180 con 0x7fde100b73f0 2024-06-29T15:59:08.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.824+0000 7fde0cff9640 1 -- 172.21.15.59:0/1257159603 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fde04005420 con 0x7fde100b73f0 2024-06-29T15:59:08.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.824+0000 7fde0cff9640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fddfc076510 0x7fddfc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.824+0000 7fde0cff9640 1 -- 172.21.15.59:0/1257159603 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6337+0+0 (secure 0 0 0) 0x7fde040b8ff0 con 0x7fde100b73f0 2024-06-29T15:59:08.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.824+0000 7fde0ffff640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fddfc076510 0x7fddfc078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.825+0000 7fde0ffff640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fddfc076510 0x7fddfc078a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde00000f60 tx=0x7fde00009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:08.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:08.829+0000 7fde0cff9640 1 -- 172.21.15.59:0/1257159603 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fde0403f030 con 0x7fde100b73f0 2024-06-29T15:59:08.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-06-29T15:59:08.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-06-29T15:59:08.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-06-29T15:59:09.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.036+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fde100ce5b0 con 0x7fde100b73f0 2024-06-29T15:59:09.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.392+0000 7fde0cff9640 1 -- 172.21.15.59:0/1257159603 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v256) ==== 97+0+0 (secure 0 0 0) 0x7fde04031770 con 0x7fde100b73f0 2024-06-29T15:59:09.393 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fddfc076510 msgr2=0x7fddfc078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fddfc076510 0x7fddfc078a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde00000f60 tx=0x7fde00009040 comp rx=0 tx=0).stop 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b73f0 msgr2=0x7fde10149e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b73f0 0x7fde10149e30 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fde0402abe0 tx=0x7fde04033c10 comp rx=0 tx=0).stop 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 shutdown_connections 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fddfc076510 0x7fddfc078a20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fde1015e290 0x7fde101e8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde100b73f0 0x7fde10149e30 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 --2- 172.21.15.59:0/1257159603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde100b3990 0x7fde101e7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 >> 172.21.15.59:0/1257159603 conn(0x7fde100d3ee0 msgr2=0x7fde100b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:09.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 shutdown_connections 2024-06-29T15:59:09.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.395+0000 7fde16865640 1 -- 172.21.15.59:0/1257159603 wait complete. 2024-06-29T15:59:09.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:59:09.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.527+0000 7f60d37eb640 1 -- 172.21.15.59:0/252686879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 msgr2=0x7f60cc166ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.527+0000 7f60d37eb640 1 --2- 172.21.15.59:0/252686879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 0x7f60cc166ca0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f60c4009830 tx=0x7f60c402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.527+0000 7f60d37eb640 1 -- 172.21.15.59:0/252686879 shutdown_connections 2024-06-29T15:59:09.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.527+0000 7f60d37eb640 1 --2- 172.21.15.59:0/252686879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 0x7f60cc166ca0 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.527+0000 7f60d37eb640 1 -- 172.21.15.59:0/252686879 >> 172.21.15.59:0/252686879 conn(0x7f60cc06c950 msgr2=0x7f60cc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:09.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.527+0000 7f60d37eb640 1 -- 172.21.15.59:0/252686879 shutdown_connections 2024-06-29T15:59:09.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.527+0000 7f60d37eb640 1 -- 172.21.15.59:0/252686879 wait complete. 2024-06-29T15:59:09.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.529+0000 7f60d37eb640 1 Processor -- start 2024-06-29T15:59:09.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.529+0000 7f60d37eb640 1 -- start start 2024-06-29T15:59:09.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d37eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60cc0b3740 0x7f60cc15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d37eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 0x7f60cc13ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d37eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60cc144570 0x7f60cc144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d37eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f60cc167570 con 0x7f60cc0b3b50 2024-06-29T15:59:09.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d37eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60cc1673f0 con 0x7f60cc0b3740 2024-06-29T15:59:09.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d37eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f60cc1676f0 con 0x7f60cc144570 2024-06-29T15:59:09.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d0d86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 0x7f60cc13ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d0d86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 0x7f60cc13ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47072/0 (socket says 172.21.15.59:47072) 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d0d86640 1 -- 172.21.15.59:0/2287497777 learned_addr learned my addr 172.21.15.59:0/2287497777 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d1d88640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60cc144570 0x7f60cc144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d1587640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60cc0b3740 0x7f60cc15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d0d86640 1 -- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60cc144570 msgr2=0x7f60cc144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d0d86640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60cc144570 0x7f60cc144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.530+0000 7f60d0d86640 1 -- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60cc0b3740 msgr2=0x7f60cc15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.531+0000 7f60d0d86640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60cc0b3740 0x7f60cc15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.531+0000 7f60d0d86640 1 -- 172.21.15.59:0/2287497777 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60cc1434d0 con 0x7f60cc0b3b50 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.531+0000 7f60d0d86640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 0x7f60cc13ea70 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f60c000c3d0 tx=0x7f60c000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.531+0000 7f60c9ffb640 1 -- 172.21.15.59:0/2287497777 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0017070 con 0x7f60cc0b3b50 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.531+0000 7f60c9ffb640 1 -- 172.21.15.59:0/2287497777 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60c00053c0 con 0x7f60cc0b3b50 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.531+0000 7f60c9ffb640 1 -- 172.21.15.59:0/2287497777 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0003850 con 0x7f60cc0b3b50 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.531+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60cc1437b0 con 0x7f60cc0b3b50 2024-06-29T15:59:09.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.531+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60cc143d70 con 0x7f60cc0b3b50 2024-06-29T15:59:09.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.532+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60cc1458a0 con 0x7f60cc0b3b50 2024-06-29T15:59:09.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.533+0000 7f60c9ffb640 1 -- 172.21.15.59:0/2287497777 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f60c001a070 con 0x7f60cc0b3b50 2024-06-29T15:59:09.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.533+0000 7f60c9ffb640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f60b8076510 0x7f60b8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.534+0000 7f60c9ffb640 1 -- 172.21.15.59:0/2287497777 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5305+0+0 (secure 0 0 0) 0x7f60c0096d10 con 0x7f60cc0b3b50 2024-06-29T15:59:09.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.534+0000 7f60d1587640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f60b8076510 0x7f60b8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.534+0000 7f60d1587640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f60b8076510 0x7f60b8078a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0012e0 tx=0x7f60bc009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:09.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.538+0000 7f60c9ffb640 1 -- 172.21.15.59:0/2287497777 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f60c005fe90 con 0x7f60cc0b3b50 2024-06-29T15:59:09.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.801+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f60cc0b1f10 con 0x7f60cc0b3b50 2024-06-29T15:59:09.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.802+0000 7f60c9ffb640 1 -- 172.21.15.59:0/2287497777 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v53) ==== 80+0+5304 (secure 0 0 0) 0x7f60c0064190 con 0x7f60cc0b3b50 2024-06-29T15:59:09.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.803 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.804 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T15:59:09.805 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:09.806 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.807 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:09.808 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.809 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T15:59:09.810 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T15:59:09.810 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.810 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:09.810 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:09.810 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.810 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.810 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.810 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.811 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T15:59:09.811 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T15:59:09.811 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.811 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:09.811 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:09.811 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.812 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.812 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.812 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.812 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T15:59:09.812 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T15:59:09.813 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.813 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:09.813 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:09.813 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.813 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.813 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.813 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.813 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.814 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T15:59:09.815 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T15:59:09.815 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.815 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:09.815 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:59:09.815 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.816 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T15:59:09.817 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:59:09.818 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T15:59:09.819 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:59:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f60b8076510 msgr2=0x7f60b8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f60b8076510 0x7f60b8078a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0012e0 tx=0x7f60bc009040 comp rx=0 tx=0).stop 2024-06-29T15:59:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 msgr2=0x7f60cc13ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 0x7f60cc13ea70 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f60c000c3d0 tx=0x7f60c000c890 comp rx=0 tx=0).stop 2024-06-29T15:59:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 shutdown_connections 2024-06-29T15:59:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f60b8076510 0x7f60b8078a20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60cc144570 0x7f60cc144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60cc0b3b50 0x7f60cc13ea70 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 --2- 172.21.15.59:0/2287497777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60cc0b3740 0x7f60cc15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 >> 172.21.15.59:0/2287497777 conn(0x7f60cc06c950 msgr2=0x7f60cc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 shutdown_connections 2024-06-29T15:59:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.804+0000 7f60d37eb640 1 -- 172.21.15.59:0/2287497777 wait complete. 2024-06-29T15:59:09.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.940+0000 7f70421f5640 1 -- 172.21.15.59:0/829517022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7024004d90 msgr2=0x7f7024005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.940+0000 7f70421f5640 1 --2- 172.21.15.59:0/829517022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7024004d90 0x7f7024005260 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f7034001660 tx=0x7f7024005830 comp rx=0 tx=0).stop 2024-06-29T15:59:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.940+0000 7f703bfff640 1 -- 172.21.15.59:0/829517022 reap_dead start 2024-06-29T15:59:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.940+0000 7f70421f5640 1 -- 172.21.15.59:0/829517022 shutdown_connections 2024-06-29T15:59:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.940+0000 7f70421f5640 1 -- 172.21.15.59:0/829517022 >> 172.21.15.59:0/829517022 conn(0x7f703c0d3ee0 msgr2=0x7f703c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:09.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.940+0000 7f70421f5640 1 -- 172.21.15.59:0/829517022 shutdown_connections 2024-06-29T15:59:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.940+0000 7f70421f5640 1 -- 172.21.15.59:0/829517022 wait complete. 2024-06-29T15:59:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.942+0000 7f70421f5640 1 Processor -- start 2024-06-29T15:59:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.942+0000 7f70421f5640 1 -- start start 2024-06-29T15:59:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f70421f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024004d90 0x7f703c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f70421f5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f703c14dde0 0x7f703c06c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f70421f5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f703c14e1f0 0x7f703c06c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f70421f5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f703c0b3d20 con 0x7f703c14e1f0 2024-06-29T15:59:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f70421f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f703c0b3ba0 con 0x7f7024004d90 2024-06-29T15:59:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f70421f5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f703c1600c0 con 0x7f703c14dde0 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f703bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f703c14e1f0 0x7f703c06c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f703bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f703c14e1f0 0x7f703c06c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47098/0 (socket says 172.21.15.59:47098) 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f703bfff640 1 -- 172.21.15.59:0/1973471222 learned_addr learned my addr 172.21.15.59:0/1973471222 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f703affd640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f703c14dde0 0x7f703c06c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.943+0000 7f703b7fe640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024004d90 0x7f703c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f703bfff640 1 -- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f703c14dde0 msgr2=0x7f703c06c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f703bfff640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f703c14dde0 0x7f703c06c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f703bfff640 1 -- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024004d90 msgr2=0x7f703c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f703bfff640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024004d90 0x7f703c15c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f703bfff640 1 -- 172.21.15.59:0/1973471222 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703c0c75b0 con 0x7f703c14e1f0 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f703bfff640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f703c14e1f0 0x7f703c06c720 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f7034001750 tx=0x7f70340062e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:09.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f7023fff640 1 -- 172.21.15.59:0/1973471222 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70340037f0 con 0x7f703c14e1f0 2024-06-29T15:59:09.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f7023fff640 1 -- 172.21.15.59:0/1973471222 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7034003a80 con 0x7f703c14e1f0 2024-06-29T15:59:09.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703c0c7ab0 con 0x7f703c14e1f0 2024-06-29T15:59:09.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f7023fff640 1 -- 172.21.15.59:0/1973471222 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703403d430 con 0x7f703c14e1f0 2024-06-29T15:59:09.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.944+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f703c0c8070 con 0x7f703c14e1f0 2024-06-29T15:59:09.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.945+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703c0c7820 con 0x7f703c14e1f0 2024-06-29T15:59:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.946+0000 7f7023fff640 1 -- 172.21.15.59:0/1973471222 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f7034030050 con 0x7f703c14e1f0 2024-06-29T15:59:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.946+0000 7f7023fff640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7028076510 0x7f7028078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.947+0000 7f7023fff640 1 -- 172.21.15.59:0/1973471222 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5305+0+0 (secure 0 0 0) 0x7f70340b6d50 con 0x7f703c14e1f0 2024-06-29T15:59:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.947+0000 7f703b7fe640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7028076510 0x7f7028078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.948+0000 7f703b7fe640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7028076510 0x7f7028078a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f703c0d2e40 tx=0x7f702c009500 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:09.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:09.951+0000 7f7023fff640 1 -- 172.21.15.59:0/1973471222 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f70340ba030 con 0x7f703c14e1f0 2024-06-29T15:59:10.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.142+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f703c160de0 con 0x7f703c14e1f0 2024-06-29T15:59:10.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.143+0000 7f7023fff640 1 -- 172.21.15.59:0/1973471222 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) ==== 74+0+13276 (secure 0 0 0) 0x7f703407fe00 con 0x7f703c14e1f0 2024-06-29T15:59:10.144 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:10.144 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":258,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:09.401635+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":244,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7028076510 msgr2=0x7f7028078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7028076510 0x7f7028078a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f703c0d2e40 tx=0x7f702c009500 comp rx=0 tx=0).stop 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f703c14e1f0 msgr2=0x7f703c06c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f703c14e1f0 0x7f703c06c720 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f7034001750 tx=0x7f70340062e0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 shutdown_connections 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7028076510 0x7f7028078a20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f703c14e1f0 0x7f703c06c720 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f703c14dde0 0x7f703c06c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 --2- 172.21.15.59:0/1973471222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7024004d90 0x7f703c15c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 >> 172.21.15.59:0/1973471222 conn(0x7f703c0d3ee0 msgr2=0x7f703c157e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 shutdown_connections 2024-06-29T15:59:10.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.145+0000 7f70421f5640 1 -- 172.21.15.59:0/1973471222 wait complete. 2024-06-29T15:59:10.158 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:10.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.282+0000 7f0fe963c640 1 -- 172.21.15.59:0/1846386776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe415c2f0 msgr2=0x7f0fe415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.282+0000 7f0fe963c640 1 --2- 172.21.15.59:0/1846386776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe415c2f0 0x7f0fe415c700 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f0fe40b1c40 tx=0x7f0fdc02a530 comp rx=0 tx=0).stop 2024-06-29T15:59:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.282+0000 7f0fe963c640 1 -- 172.21.15.59:0/1846386776 shutdown_connections 2024-06-29T15:59:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.282+0000 7f0fe963c640 1 --2- 172.21.15.59:0/1846386776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe415c2f0 0x7f0fe415c700 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.282+0000 7f0fe963c640 1 --2- 172.21.15.59:0/1846386776 >> v2:172.21.15.153:3300/0 conn(0x7f0fe40b3740 0x7f0fe40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.282+0000 7f0fe963c640 1 -- 172.21.15.59:0/1846386776 >> 172.21.15.59:0/1846386776 conn(0x7f0fe406c950 msgr2=0x7f0fe4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.282+0000 7f0fe963c640 1 -- 172.21.15.59:0/1846386776 shutdown_connections 2024-06-29T15:59:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.282+0000 7f0fe963c640 1 -- 172.21.15.59:0/1846386776 wait complete. 2024-06-29T15:59:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.284+0000 7f0fe963c640 1 Processor -- start 2024-06-29T15:59:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.284+0000 7f0fe963c640 1 -- start start 2024-06-29T15:59:10.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe963c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe40b3740 0x7f0fe40a9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe963c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe415c2f0 0x7f0fe41f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe963c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe40aa020 0x7f0fe41f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe963c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe40b5fd0 con 0x7f0fe40b3740 2024-06-29T15:59:10.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe963c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe40b5e50 con 0x7f0fe415c2f0 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe963c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe40b6150 con 0x7f0fe40aa020 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe40b3740 0x7f0fe40a9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe40b3740 0x7f0fe40a9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47126/0 (socket says 172.21.15.59:47126) 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe2ffd640 1 -- 172.21.15.59:0/2716390729 learned_addr learned my addr 172.21.15.59:0/2716390729 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe37fe640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe40aa020 0x7f0fe41f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.285+0000 7f0fe27fc640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe415c2f0 0x7f0fe41f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fe2ffd640 1 -- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe40aa020 msgr2=0x7f0fe41f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fe2ffd640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe40aa020 0x7f0fe41f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fe2ffd640 1 -- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe415c2f0 msgr2=0x7f0fe41f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fe2ffd640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe415c2f0 0x7f0fe41f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fe2ffd640 1 -- 172.21.15.59:0/2716390729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe40aa800 con 0x7f0fe40b3740 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fe2ffd640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe40b3740 0x7f0fe40a9920 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400c7a0 tx=0x7f0fd400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fdb7fe640 1 -- 172.21.15.59:0/2716390729 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4017070 con 0x7f0fe40b3740 2024-06-29T15:59:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fdb7fe640 1 -- 172.21.15.59:0/2716390729 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fd400ce50 con 0x7f0fe40b3740 2024-06-29T15:59:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe40aaae0 con 0x7f0fe40b3740 2024-06-29T15:59:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fdb7fe640 1 -- 172.21.15.59:0/2716390729 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd40052d0 con 0x7f0fe40b3740 2024-06-29T15:59:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.286+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe40ab0a0 con 0x7f0fe40b3740 2024-06-29T15:59:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.287+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe40b6670 con 0x7f0fe40b3740 2024-06-29T15:59:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.288+0000 7f0fdb7fe640 1 -- 172.21.15.59:0/2716390729 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f0fd401a070 con 0x7f0fe40b3740 2024-06-29T15:59:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.288+0000 7f0fdb7fe640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0fc4076510 0x7f0fc4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.289+0000 7f0fdb7fe640 1 -- 172.21.15.59:0/2716390729 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5305+0+0 (secure 0 0 0) 0x7f0fd4097180 con 0x7f0fe40b3740 2024-06-29T15:59:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.289+0000 7f0fe27fc640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0fc4076510 0x7f0fc4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.290+0000 7f0fe27fc640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0fc4076510 0x7f0fc4078a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0fe41f01e0 tx=0x7f0fcc005f60 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.293+0000 7f0fdb7fe640 1 -- 172.21.15.59:0/2716390729 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0fd4060190 con 0x7f0fe40b3740 2024-06-29T15:59:10.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.484+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0fe415f770 con 0x7f0fe40b3740 2024-06-29T15:59:10.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.485+0000 7f0fdb7fe640 1 -- 172.21.15.59:0/2716390729 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v258) ==== 93+0+10 (secure 0 0 0) 0x7f0fd4064490 con 0x7f0fe40b3740 2024-06-29T15:59:10.486 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0fc4076510 msgr2=0x7f0fc4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0fc4076510 0x7f0fc4078a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0fe41f01e0 tx=0x7f0fcc005f60 comp rx=0 tx=0).stop 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe40b3740 msgr2=0x7f0fe40a9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe40b3740 0x7f0fe40a9920 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400c7a0 tx=0x7f0fd400cc60 comp rx=0 tx=0).stop 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 shutdown_connections 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0fc4076510 0x7f0fc4078a20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0fe40aa020 0x7f0fe41f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fe415c2f0 0x7f0fe41f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 --2- 172.21.15.59:0/2716390729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe40b3740 0x7f0fe40a9920 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 >> 172.21.15.59:0/2716390729 conn(0x7f0fe406c950 msgr2=0x7f0fe4160050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 shutdown_connections 2024-06-29T15:59:10.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.487+0000 7f0fe963c640 1 -- 172.21.15.59:0/2716390729 wait complete. 2024-06-29T15:59:10.500 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:59:10.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.616+0000 7fb09b2d6640 1 -- 172.21.15.59:0/4058814059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb09406ea10 msgr2=0x7fb0940cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.616+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/4058814059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb09406ea10 0x7fb0940cef90 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7fb08c005fd0 tx=0x7fb08c02a970 comp rx=0 tx=0).stop 2024-06-29T15:59:10.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.616+0000 7fb09b2d6640 1 -- 172.21.15.59:0/4058814059 shutdown_connections 2024-06-29T15:59:10.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.616+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/4058814059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb09406ea10 0x7fb0940cef90 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.616+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/4058814059 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb094143840 0x7fb09406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.616+0000 7fb09b2d6640 1 -- 172.21.15.59:0/4058814059 >> 172.21.15.59:0/4058814059 conn(0x7fb0940b2ab0 msgr2=0x7fb0940b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.617+0000 7fb09b2d6640 1 -- 172.21.15.59:0/4058814059 shutdown_connections 2024-06-29T15:59:10.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.617+0000 7fb09b2d6640 1 -- 172.21.15.59:0/4058814059 wait complete. 2024-06-29T15:59:10.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.618+0000 7fb09b2d6640 1 Processor -- start 2024-06-29T15:59:10.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.618+0000 7fb09b2d6640 1 -- start start 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb09b2d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb09406ea10 0x7fb0941552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb09b2d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0940d0130 0x7fb09415d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb09b2d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb094143840 0x7fb0941f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb09b2d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb0940cf650 con 0x7fb094143840 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb09b2d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb0940cf4d0 con 0x7fb0940d0130 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb09b2d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb0940cf7d0 con 0x7fb09406ea10 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb094143840 0x7fb0941f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb094143840 0x7fb0941f38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47142/0 (socket says 172.21.15.59:47142) 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 -- 172.21.15.59:0/3250451957 learned_addr learned my addr 172.21.15.59:0/3250451957 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099072640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb09406ea10 0x7fb0941552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb098871640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0940d0130 0x7fb09415d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 -- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb09406ea10 msgr2=0x7fb0941552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb09406ea10 0x7fb0941552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 -- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0940d0130 msgr2=0x7fb09415d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0940d0130 0x7fb09415d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 -- 172.21.15.59:0/3250451957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09415da00 con 0x7fb094143840 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.620+0000 7fb099873640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb094143840 0x7fb0941f38a0 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fb08c006010 tx=0x7fb08c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.621+0000 7fb089ffb640 1 -- 172.21.15.59:0/3250451957 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c034070 con 0x7fb094143840 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.621+0000 7fb089ffb640 1 -- 172.21.15.59:0/3250451957 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb08c004640 con 0x7fb094143840 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.621+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09415c420 con 0x7fb094143840 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.621+0000 7fb089ffb640 1 -- 172.21.15.59:0/3250451957 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c0436a0 con 0x7fb094143840 2024-06-29T15:59:10.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.621+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb09415c9e0 con 0x7fb094143840 2024-06-29T15:59:10.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.622+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0940cfc20 con 0x7fb094143840 2024-06-29T15:59:10.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.623+0000 7fb089ffb640 1 -- 172.21.15.59:0/3250451957 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fb08c037020 con 0x7fb094143840 2024-06-29T15:59:10.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.623+0000 7fb089ffb640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb078076510 0x7fb078078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.623+0000 7fb089ffb640 1 -- 172.21.15.59:0/3250451957 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5305+0+0 (secure 0 0 0) 0x7fb08c0b8dd0 con 0x7fb094143840 2024-06-29T15:59:10.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.623+0000 7fb099072640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb078076510 0x7fb078078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.624+0000 7fb099072640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb078076510 0x7fb078078a20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb090006fd0 tx=0x7fb090009060 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.627+0000 7fb089ffb640 1 -- 172.21.15.59:0/3250451957 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb08c081e10 con 0x7fb094143840 2024-06-29T15:59:10.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.820+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb09415d0a0 con 0x7fb094143840 2024-06-29T15:59:10.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.821+0000 7fb089ffb640 1 -- 172.21.15.59:0/3250451957 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v258) ==== 84+0+65 (secure 0 0 0) 0x7fb08c086110 con 0x7fb094143840 2024-06-29T15:59:10.822 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:10.822 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:59:10.822 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:10.822 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:59:10.822 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:59:10.822 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:10.822 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb078076510 msgr2=0x7fb078078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb078076510 0x7fb078078a20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb090006fd0 tx=0x7fb090009060 comp rx=0 tx=0).stop 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb094143840 msgr2=0x7fb0941f38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb094143840 0x7fb0941f38a0 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fb08c006010 tx=0x7fb08c004040 comp rx=0 tx=0).stop 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 shutdown_connections 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fb078076510 0x7fb078078a20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb094143840 0x7fb0941f38a0 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0940d0130 0x7fb09415d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 --2- 172.21.15.59:0/3250451957 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb09406ea10 0x7fb0941552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 >> 172.21.15.59:0/3250451957 conn(0x7fb0940b2ab0 msgr2=0x7fb09415ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 shutdown_connections 2024-06-29T15:59:10.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.823+0000 7fb09b2d6640 1 -- 172.21.15.59:0/3250451957 wait complete. 2024-06-29T15:59:10.835 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:59:10.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:59:10.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.956+0000 7f2f70b68640 1 -- 172.21.15.59:0/2679696046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f54003a00 msgr2=0x7f2f54003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.956+0000 7f2f70b68640 1 --2- 172.21.15.59:0/2679696046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f54003a00 0x7f2f54003e70 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009ed0 tx=0x7f2f54005830 comp rx=0 tx=0).stop 2024-06-29T15:59:10.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.956+0000 7f2f6ad9d640 1 -- 172.21.15.59:0/2679696046 reap_dead start 2024-06-29T15:59:10.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.956+0000 7f2f70b68640 1 -- 172.21.15.59:0/2679696046 shutdown_connections 2024-06-29T15:59:10.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.956+0000 7f2f70b68640 1 -- 172.21.15.59:0/2679696046 >> 172.21.15.59:0/2679696046 conn(0x7f2f6c0d0f90 msgr2=0x7f2f6c0d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.957+0000 7f2f70b68640 1 -- 172.21.15.59:0/2679696046 shutdown_connections 2024-06-29T15:59:10.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.957+0000 7f2f70b68640 1 -- 172.21.15.59:0/2679696046 wait complete. 2024-06-29T15:59:10.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.959+0000 7f2f70b68640 1 Processor -- start 2024-06-29T15:59:10.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.959+0000 7f2f70b68640 1 -- start start 2024-06-29T15:59:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f70b68640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f54003a00 0x7f2f6c1f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f70b68640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f6c1f1e40 0x7f2f6c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f70b68640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f6c15d6f0 0x7f2f6c15f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f70b68640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2f6c0b41f0 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f70b68640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2f6c0b4070 con 0x7f2f6c1f1e40 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f70b68640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2f6c0b4370 con 0x7f2f54003a00 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f6c15d6f0 0x7f2f6c15f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f6c15d6f0 0x7f2f6c15f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47166/0 (socket says 172.21.15.59:47166) 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6ad9d640 1 -- 172.21.15.59:0/1479518846 learned_addr learned my addr 172.21.15.59:0/1479518846 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6a59c640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f54003a00 0x7f2f6c1f89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f69d9b640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f6c1f1e40 0x7f2f6c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6ad9d640 1 -- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f54003a00 msgr2=0x7f2f6c1f89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6ad9d640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f54003a00 0x7f2f6c1f89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6ad9d640 1 -- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f6c1f1e40 msgr2=0x7f2f6c15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6ad9d640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f6c1f1e40 0x7f2f6c15d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.960+0000 7f2f6ad9d640 1 -- 172.21.15.59:0/1479518846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f6c15db00 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.961+0000 7f2f6ad9d640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f6c15d6f0 0x7f2f6c15f540 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0ce290 tx=0x7f2f64011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.961+0000 7f2f5affd640 1 -- 172.21.15.59:0/1479518846 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64004230 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.961+0000 7f2f5affd640 1 -- 172.21.15.59:0/1479518846 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f64016070 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.961+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f6c0c9100 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.961+0000 7f2f5affd640 1 -- 172.21.15.59:0/1479518846 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64005280 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.961+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f6c1f51d0 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.962+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f6c0cd850 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.963+0000 7f2f5affd640 1 -- 172.21.15.59:0/1479518846 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f2f6400f830 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.963+0000 7f2f5affd640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2f4c076510 0x7f2f4c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.963+0000 7f2f5affd640 1 -- 172.21.15.59:0/1479518846 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5305+0+0 (secure 0 0 0) 0x7f2f640983f0 con 0x7f2f6c15d6f0 2024-06-29T15:59:10.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.963+0000 7f2f6a59c640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2f4c076510 0x7f2f4c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.964+0000 7f2f6a59c640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2f4c076510 0x7f2f4c078a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009ed0 tx=0x7f2f5c004ab0 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:10.968+0000 7f2f5affd640 1 -- 172.21.15.59:0/1479518846 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2f64021020 con 0x7f2f6c15d6f0 2024-06-29T15:59:11.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.160+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2f6c075b40 con 0x7f2f6c15d6f0 2024-06-29T15:59:11.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.402+0000 7f2f5affd640 1 -- 172.21.15.59:0/1479518846 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v259) ==== 138+0+0 (secure 0 0 0) 0x7f2f6c15db00 con 0x7f2f6c15d6f0 2024-06-29T15:59:11.403 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T15:59:11.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2f4c076510 msgr2=0x7f2f4c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2f4c076510 0x7f2f4c078a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009ed0 tx=0x7f2f5c004ab0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f6c15d6f0 msgr2=0x7f2f6c15f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f6c15d6f0 0x7f2f6c15f540 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0ce290 tx=0x7f2f64011b10 comp rx=0 tx=0).stop 2024-06-29T15:59:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 shutdown_connections 2024-06-29T15:59:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f2f4c076510 0x7f2f4c078a20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f6c15d6f0 0x7f2f6c15f540 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f6c1f1e40 0x7f2f6c15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 --2- 172.21.15.59:0/1479518846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2f54003a00 0x7f2f6c1f89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 >> 172.21.15.59:0/1479518846 conn(0x7f2f6c0d0f90 msgr2=0x7f2f6c15a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 shutdown_connections 2024-06-29T15:59:11.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.405+0000 7f2f70b68640 1 -- 172.21.15.59:0/1479518846 wait complete. 2024-06-29T15:59:11.421 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:59:11.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.547+0000 7fac8e2d6640 1 -- 172.21.15.59:0/3536410236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac70004d40 msgr2=0x7fac70005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.547+0000 7fac8e2d6640 1 --2- 172.21.15.59:0/3536410236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac70004d40 0x7fac70005150 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7fac88073ea0 tx=0x7fac70005830 comp rx=0 tx=0).stop 2024-06-29T15:59:11.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.547+0000 7fac8c873640 1 -- 172.21.15.59:0/3536410236 reap_dead start 2024-06-29T15:59:11.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.547+0000 7fac8e2d6640 1 -- 172.21.15.59:0/3536410236 shutdown_connections 2024-06-29T15:59:11.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.547+0000 7fac8e2d6640 1 -- 172.21.15.59:0/3536410236 >> 172.21.15.59:0/3536410236 conn(0x7fac88073ef0 msgr2=0x7fac88074300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:11.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.547+0000 7fac8e2d6640 1 -- 172.21.15.59:0/3536410236 shutdown_connections 2024-06-29T15:59:11.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.547+0000 7fac8e2d6640 1 -- 172.21.15.59:0/3536410236 wait complete. 2024-06-29T15:59:11.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.549+0000 7fac8e2d6640 1 Processor -- start 2024-06-29T15:59:11.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.549+0000 7fac8e2d6640 1 -- start start 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.550+0000 7fac8e2d6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac70004d40 0x7fac8815e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.550+0000 7fac8e2d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac880b0370 0x7fac880b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.550+0000 7fac8e2d6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8815b030 0x7fac8815b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.550+0000 7fac8e2d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fac881453d0 con 0x7fac880b0370 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.550+0000 7fac8e2d6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac880d20e0 con 0x7fac70004d40 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac8e2d6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fac88145550 con 0x7fac8815b030 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac880b0370 0x7fac880b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac880b0370 0x7fac880b0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47186/0 (socket says 172.21.15.59:47186) 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 -- 172.21.15.59:0/1592897968 learned_addr learned my addr 172.21.15.59:0/1592897968 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac8c873640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8815b030 0x7fac8815b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac87fff640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac70004d40 0x7fac8815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 -- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8815b030 msgr2=0x7fac8815b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8815b030 0x7fac8815b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 -- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac70004d40 msgr2=0x7fac8815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac70004d40 0x7fac8815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 -- 172.21.15.59:0/1592897968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8815f180 con 0x7fac880b0370 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac87fff640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac70004d40 0x7fac8815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac8c873640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8815b030 0x7fac8815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac877fe640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac880b0370 0x7fac880b0780 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fac7c011ae0 tx=0x7fac7c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac84ff9640 1 -- 172.21.15.59:0/1592897968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c00fe60 con 0x7fac880b0370 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac84ff9640 1 -- 172.21.15.59:0/1592897968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac7c005020 con 0x7fac880b0370 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8806dd60 con 0x7fac880b0370 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac84ff9640 1 -- 172.21.15.59:0/1592897968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c005470 con 0x7fac880b0370 2024-06-29T15:59:11.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.551+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac8806e320 con 0x7fac880b0370 2024-06-29T15:59:11.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.552+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac8806fd90 con 0x7fac880b0370 2024-06-29T15:59:11.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.553+0000 7fac84ff9640 1 -- 172.21.15.59:0/1592897968 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fac7c0047a0 con 0x7fac880b0370 2024-06-29T15:59:11.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.554+0000 7fac84ff9640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fac74076510 0x7fac74078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:11.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.554+0000 7fac84ff9640 1 -- 172.21.15.59:0/1592897968 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 5675+0+0 (secure 0 0 0) 0x7fac7c0988f0 con 0x7fac880b0370 2024-06-29T15:59:11.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.554+0000 7fac87fff640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fac74076510 0x7fac74078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:11.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.555+0000 7fac87fff640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fac74076510 0x7fac74078a20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fac78009c60 tx=0x7fac78005f90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:11.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.558+0000 7fac84ff9640 1 -- 172.21.15.59:0/1592897968 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fac7c061e00 con 0x7fac880b0370 2024-06-29T15:59:11.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:11.751+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fac880cdd80 con 0x7fac880b0370 2024-06-29T15:59:12.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.405+0000 7fac84ff9640 1 -- 172.21.15.59:0/1592897968 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v260) ==== 170+0+0 (secure 0 0 0) 0x7fac7c061700 con 0x7fac880b0370 2024-06-29T15:59:12.406 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.408+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fac74076510 msgr2=0x7fac74078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.408+0000 7fac8e2d6640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fac74076510 0x7fac74078a20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fac78009c60 tx=0x7fac78005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.408+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac880b0370 msgr2=0x7fac880b0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.408+0000 7fac8e2d6640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac880b0370 0x7fac880b0780 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fac7c011ae0 tx=0x7fac7c00f3c0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.409+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 shutdown_connections 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.409+0000 7fac8e2d6640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fac74076510 0x7fac74078a20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.409+0000 7fac8e2d6640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8815b030 0x7fac8815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.409+0000 7fac8e2d6640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac880b0370 0x7fac880b0780 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.409+0000 7fac8e2d6640 1 --2- 172.21.15.59:0/1592897968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac70004d40 0x7fac8815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.409+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 >> 172.21.15.59:0/1592897968 conn(0x7fac88073ef0 msgr2=0x7fac880b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:12.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.409+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 shutdown_connections 2024-06-29T15:59:12.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.409+0000 7fac8e2d6640 1 -- 172.21.15.59:0/1592897968 wait complete. 2024-06-29T15:59:12.439 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:59:12.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.564+0000 7fda98d03640 1 -- 172.21.15.59:0/2430268132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 msgr2=0x7fda9413edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.564+0000 7fda98d03640 1 --2- 172.21.15.59:0/2430268132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 0x7fda9413edc0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fda94073da0 tx=0x7fda7c02a4d0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.564+0000 7fda98d03640 1 -- 172.21.15.59:0/2430268132 shutdown_connections 2024-06-29T15:59:12.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.564+0000 7fda98d03640 1 --2- 172.21.15.59:0/2430268132 >> v2:172.21.15.153:3300/0 conn(0x7fda941414c0 0x7fda941418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.564+0000 7fda98d03640 1 --2- 172.21.15.59:0/2430268132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 0x7fda9413edc0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.564+0000 7fda98d03640 1 -- 172.21.15.59:0/2430268132 >> 172.21.15.59:0/2430268132 conn(0x7fda940b6fb0 msgr2=0x7fda940b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:12.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.564+0000 7fda98d03640 1 -- 172.21.15.59:0/2430268132 shutdown_connections 2024-06-29T15:59:12.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.564+0000 7fda98d03640 1 -- 172.21.15.59:0/2430268132 wait complete. 2024-06-29T15:59:12.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.566+0000 7fda98d03640 1 Processor -- start 2024-06-29T15:59:12.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.566+0000 7fda98d03640 1 -- start start 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.567+0000 7fda98d03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 0x7fda94152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.567+0000 7fda98d03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda941414c0 0x7fda94070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.567+0000 7fda98d03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda9415ceb0 0x7fda94070890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.567+0000 7fda98d03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fda940b29d0 con 0x7fda9413e9b0 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.567+0000 7fda98d03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda940b2850 con 0x7fda941414c0 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.567+0000 7fda98d03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda940b2b50 con 0x7fda9415ceb0 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.567+0000 7fda9259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 0x7fda94152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda9259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 0x7fda94152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47198/0 (socket says 172.21.15.59:47198) 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda9259c640 1 -- 172.21.15.59:0/1311992876 learned_addr learned my addr 172.21.15.59:0/1311992876 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda92d9d640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda9415ceb0 0x7fda94070890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda91d9b640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda941414c0 0x7fda94070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda9259c640 1 -- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda9415ceb0 msgr2=0x7fda94070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda9259c640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda9415ceb0 0x7fda94070890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda9259c640 1 -- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda941414c0 msgr2=0x7fda94070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda9259c640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda941414c0 0x7fda94070350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda9259c640 1 -- 172.21.15.59:0/1311992876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda94139ae0 con 0x7fda9413e9b0 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda91d9b640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda941414c0 0x7fda94070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda9259c640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 0x7fda94152590 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00b410 tx=0x7fda8c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda8affd640 1 -- 172.21.15.59:0/1311992876 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c015020 con 0x7fda9413e9b0 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda8affd640 1 -- 172.21.15.59:0/1311992876 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda8c011940 con 0x7fda9413e9b0 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda94152ad0 con 0x7fda9413e9b0 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda8affd640 1 -- 172.21.15.59:0/1311992876 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c011d90 con 0x7fda9413e9b0 2024-06-29T15:59:12.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.568+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9413a0a0 con 0x7fda9413e9b0 2024-06-29T15:59:12.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.569+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda940b3000 con 0x7fda9413e9b0 2024-06-29T15:59:12.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.571+0000 7fda8affd640 1 -- 172.21.15.59:0/1311992876 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fda8c018020 con 0x7fda9413e9b0 2024-06-29T15:59:12.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.571+0000 7fda8affd640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fda74076540 0x7fda74078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:12.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.571+0000 7fda8affd640 1 -- 172.21.15.59:0/1311992876 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 6057+0+0 (secure 0 0 0) 0x7fda8c097fe0 con 0x7fda9413e9b0 2024-06-29T15:59:12.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.571+0000 7fda91d9b640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fda74076540 0x7fda74078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.572+0000 7fda91d9b640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fda74076540 0x7fda74078a50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda7c033000 tx=0x7fda7c004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:12.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.575+0000 7fda8affd640 1 -- 172.21.15.59:0/1311992876 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda8c01f020 con 0x7fda9413e9b0 2024-06-29T15:59:12.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:12.806+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fda94135890 con 0x7fda9413e9b0 2024-06-29T15:59:13.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.428+0000 7fda8affd640 1 -- 172.21.15.59:0/1311992876 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 35 and data pool 36 v144) ==== 174+0+0 (secure 0 0 0) 0x7fda8c0615a0 con 0x7fda9413e9b0 2024-06-29T15:59:13.429 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 35 and data pool 36 2024-06-29T15:59:13.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.440+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fda74076540 msgr2=0x7fda74078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.440+0000 7fda98d03640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fda74076540 0x7fda74078a50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda7c033000 tx=0x7fda7c004730 comp rx=0 tx=0).stop 2024-06-29T15:59:13.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.440+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 msgr2=0x7fda94152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.440+0000 7fda98d03640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 0x7fda94152590 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00b410 tx=0x7fda8c00b8d0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.442+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 shutdown_connections 2024-06-29T15:59:13.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.442+0000 7fda98d03640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fda74076540 0x7fda74078a50 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda7c033000 tx=0x7fda7c004730 comp rx=0 tx=0).stop 2024-06-29T15:59:13.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.442+0000 7fda98d03640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda9415ceb0 0x7fda94070890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.442+0000 7fda98d03640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda941414c0 0x7fda94070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.442+0000 7fda98d03640 1 --2- 172.21.15.59:0/1311992876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda9413e9b0 0x7fda94152590 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.442+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 >> 172.21.15.59:0/1311992876 conn(0x7fda940b6fb0 msgr2=0x7fda9415df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.442+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 shutdown_connections 2024-06-29T15:59:13.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.443+0000 7fda98d03640 1 -- 172.21.15.59:0/1311992876 wait complete. 2024-06-29T15:59:13.465 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:13.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.585+0000 7f00cbcb2640 1 -- 172.21.15.59:0/2620798973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00b0004850 msgr2=0x7f00b0004d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.585+0000 7f00cbcb2640 1 --2- 172.21.15.59:0/2620798973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00b0004850 0x7f00b0004d20 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f00c4162c00 tx=0x7f00b0005830 comp rx=0 tx=0).stop 2024-06-29T15:59:13.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.585+0000 7f00ca24f640 1 -- 172.21.15.59:0/2620798973 reap_dead start 2024-06-29T15:59:13.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.585+0000 7f00cbcb2640 1 -- 172.21.15.59:0/2620798973 shutdown_connections 2024-06-29T15:59:13.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.585+0000 7f00cbcb2640 1 -- 172.21.15.59:0/2620798973 >> 172.21.15.59:0/2620798973 conn(0x7f00c406c950 msgr2=0x7f00c4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.585+0000 7f00cbcb2640 1 -- 172.21.15.59:0/2620798973 shutdown_connections 2024-06-29T15:59:13.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.585+0000 7f00cbcb2640 1 -- 172.21.15.59:0/2620798973 wait complete. 2024-06-29T15:59:13.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.587+0000 7f00cbcb2640 1 Processor -- start 2024-06-29T15:59:13.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.587+0000 7f00cbcb2640 1 -- start start 2024-06-29T15:59:13.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:59:13.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.588+0000 7f00cbcb2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00b0004850 0x7f00c415d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00cbcb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c40cd460 0x7f00c40cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00cbcb2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c40b5ba0 0x7f00c40b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00cbcb2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4162fe0 con 0x7f00c40cd460 2024-06-29T15:59:13.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00cbcb2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4162e60 con 0x7f00c40b5ba0 2024-06-29T15:59:13.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00cbcb2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f00c4163160 con 0x7f00b0004850 2024-06-29T15:59:13.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c40cd460 0x7f00c40cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c40cd460 0x7f00c40cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43382/0 (socket says 172.21.15.59:43382) 2024-06-29T15:59:13.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 -- 172.21.15.59:0/1768673287 learned_addr learned my addr 172.21.15.59:0/1768673287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c9a4e640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00b0004850 0x7f00c415d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00ca24f640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c40b5ba0 0x7f00c40b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 -- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00b0004850 msgr2=0x7f00c415d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00b0004850 0x7f00c415d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 -- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c40b5ba0 msgr2=0x7f00c40b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c40b5ba0 0x7f00c40b5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 -- 172.21.15.59:0/1768673287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c40b64f0 con 0x7f00c40cd460 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00c924d640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c40cd460 0x7f00c40cd870 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f00b402a9f0 tx=0x7f00b4006170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00ba7fc640 1 -- 172.21.15.59:0/1768673287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4003a00 con 0x7f00c40cd460 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00ba7fc640 1 -- 172.21.15.59:0/1768673287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00b4003c90 con 0x7f00c40cd460 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c40cf1f0 con 0x7f00c40cd460 2024-06-29T15:59:13.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00ba7fc640 1 -- 172.21.15.59:0/1768673287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b40336e0 con 0x7f00c40cd460 2024-06-29T15:59:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.589+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c40ce150 con 0x7f00c40cd460 2024-06-29T15:59:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.590+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c40cdec0 con 0x7f00c40cd460 2024-06-29T15:59:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.591+0000 7f00ba7fc640 1 -- 172.21.15.59:0/1768673287 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f00b4045020 con 0x7f00c40cd460 2024-06-29T15:59:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.592+0000 7f00ba7fc640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f00a8076510 0x7f00a8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.592+0000 7f00ba7fc640 1 -- 172.21.15.59:0/1768673287 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(261..261 src has 1..261) ==== 6157+0+0 (secure 0 0 0) 0x7f00b40b7d80 con 0x7f00c40cd460 2024-06-29T15:59:13.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:59:13.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.592+0000 7f00c9a4e640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f00a8076510 0x7f00a8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.593+0000 7f00c9a4e640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f00a8076510 0x7f00a8078a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009bb0 tx=0x7f00c0005f90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.597+0000 7f00ba7fc640 1 -- 172.21.15.59:0/1768673287 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f00c40b64f0 con 0x7f00c40cd460 2024-06-29T15:59:13.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.790+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00c40cf870 con 0x7f00c40cd460 2024-06-29T15:59:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.791+0000 7f00ba7fc640 1 -- 172.21.15.59:0/1768673287 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) ==== 74+0+17003 (secure 0 0 0) 0x7f00b4006260 con 0x7f00c40cd460 2024-06-29T15:59:13.792 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:13.792 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":261,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:12.808162+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:11.169128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:11.760290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:13.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f00a8076510 msgr2=0x7f00a8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f00a8076510 0x7f00a8078a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009bb0 tx=0x7f00c0005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c40cd460 msgr2=0x7f00c40cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c40cd460 0x7f00c40cd870 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f00b402a9f0 tx=0x7f00b4006170 comp rx=0 tx=0).stop 2024-06-29T15:59:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 shutdown_connections 2024-06-29T15:59:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f00a8076510 0x7f00a8078a20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00c40b5ba0 0x7f00c40b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00c40cd460 0x7f00c40cd870 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 --2- 172.21.15.59:0/1768673287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f00b0004850 0x7f00c415d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 >> 172.21.15.59:0/1768673287 conn(0x7f00c406c950 msgr2=0x7f00c4161f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 shutdown_connections 2024-06-29T15:59:13.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.793+0000 7f00cbcb2640 1 -- 172.21.15.59:0/1768673287 wait complete. 2024-06-29T15:59:13.806 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:13.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.924+0000 7f0604264640 1 -- 172.21.15.59:0/153072322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 msgr2=0x7f05fc0b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.924+0000 7f0604264640 1 --2- 172.21.15.59:0/153072322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 0x7f05fc0b4a90 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f05fc069640 tx=0x7f05f002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.924+0000 7f0604264640 1 -- 172.21.15.59:0/153072322 shutdown_connections 2024-06-29T15:59:13.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.924+0000 7f0604264640 1 --2- 172.21.15.59:0/153072322 >> v2:172.21.15.153:3300/0 conn(0x7f05fc0b4fd0 0x7f05fc0b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.924+0000 7f0604264640 1 --2- 172.21.15.59:0/153072322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 0x7f05fc0b4a90 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.924+0000 7f0604264640 1 -- 172.21.15.59:0/153072322 >> 172.21.15.59:0/153072322 conn(0x7f05fc0b6fb0 msgr2=0x7f05fc0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.924+0000 7f0604264640 1 -- 172.21.15.59:0/153072322 shutdown_connections 2024-06-29T15:59:13.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.924+0000 7f0604264640 1 -- 172.21.15.59:0/153072322 wait complete. 2024-06-29T15:59:13.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.926+0000 7f0604264640 1 Processor -- start 2024-06-29T15:59:13.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.926+0000 7f0604264640 1 -- start start 2024-06-29T15:59:13.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f0604264640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05fc0b4fd0 0x7f05fc1403b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f0604264640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 0x7f05fc1f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f0604264640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05fc15e540 0x7f05fc1f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f0604264640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc0b64b0 con 0x7f05fc141590 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f0604264640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc0b6330 con 0x7f05fc0b4fd0 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f0604264640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f05fc0b6630 con 0x7f05fc15e540 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f06017ff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 0x7f05fc1f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f06017ff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 0x7f05fc1f5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43386/0 (socket says 172.21.15.59:43386) 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f06017ff640 1 -- 172.21.15.59:0/3019201516 learned_addr learned my addr 172.21.15.59:0/3019201516 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f0602000640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05fc0b4fd0 0x7f05fc1403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.927+0000 7f0602801640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05fc15e540 0x7f05fc1f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f06017ff640 1 -- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05fc15e540 msgr2=0x7f05fc1f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f06017ff640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05fc15e540 0x7f05fc1f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f06017ff640 1 -- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05fc0b4fd0 msgr2=0x7f05fc1403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f06017ff640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05fc0b4fd0 0x7f05fc1403b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f06017ff640 1 -- 172.21.15.59:0/3019201516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc155920 con 0x7f05fc141590 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f06017ff640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 0x7f05fc1f5430 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f05f0034040 tx=0x7f05f0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f05fa7fc640 1 -- 172.21.15.59:0/3019201516 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f003e030 con 0x7f05fc141590 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f05fa7fc640 1 -- 172.21.15.59:0/3019201516 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05f0003e00 con 0x7f05fc141590 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05fc155c00 con 0x7f05fc141590 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f05fa7fc640 1 -- 172.21.15.59:0/3019201516 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0035040 con 0x7f05fc141590 2024-06-29T15:59:13.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.928+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05fc1561c0 con 0x7f05fc141590 2024-06-29T15:59:13.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.929+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05fc0b5db0 con 0x7f05fc141590 2024-06-29T15:59:13.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.930+0000 7f05fa7fc640 1 -- 172.21.15.59:0/3019201516 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f05f0041030 con 0x7f05fc141590 2024-06-29T15:59:13.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.931+0000 7f05fa7fc640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f05e8076540 0x7f05e8078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.931+0000 7f05fa7fc640 1 -- 172.21.15.59:0/3019201516 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(261..261 src has 1..261) ==== 6157+0+0 (secure 0 0 0) 0x7f05f00b8480 con 0x7f05fc141590 2024-06-29T15:59:13.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.931+0000 7f0602000640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f05e8076540 0x7f05e8078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.932+0000 7f0602000640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f05e8076540 0x7f05e8078a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f05f400b450 tx=0x7f05f400d040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:13.935+0000 7f05fa7fc640 1 -- 172.21.15.59:0/3019201516 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f05f0081230 con 0x7f05fc141590 2024-06-29T15:59:14.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.129+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05fc06fb70 con 0x7f05fc141590 2024-06-29T15:59:14.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.130+0000 7f05fa7fc640 1 -- 172.21.15.59:0/3019201516 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v261) ==== 74+0+17003 (secure 0 0 0) 0x7f05f0085530 con 0x7f05fc141590 2024-06-29T15:59:14.131 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:14.131 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":261,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:12.808162+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:11.169128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:11.760290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:14.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f05e8076540 msgr2=0x7f05e8078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f05e8076540 0x7f05e8078a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f05f400b450 tx=0x7f05f400d040 comp rx=0 tx=0).stop 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 msgr2=0x7f05fc1f5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 0x7f05fc1f5430 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f05f0034040 tx=0x7f05f0003910 comp rx=0 tx=0).stop 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 shutdown_connections 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f05e8076540 0x7f05e8078a50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05fc15e540 0x7f05fc1f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05fc141590 0x7f05fc1f5430 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 --2- 172.21.15.59:0/3019201516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05fc0b4fd0 0x7f05fc1403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 >> 172.21.15.59:0/3019201516 conn(0x7f05fc0b6fb0 msgr2=0x7f05fc0b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 shutdown_connections 2024-06-29T15:59:14.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.132+0000 7f0604264640 1 -- 172.21.15.59:0/3019201516 wait complete. 2024-06-29T15:59:14.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:59:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.266+0000 7f47cb865640 1 -- 172.21.15.59:0/4160346986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c41444c0 msgr2=0x7f47c406d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.266+0000 7f47cb865640 1 --2- 172.21.15.59:0/4160346986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c41444c0 0x7f47c406d2f0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f47c4070d30 tx=0x7f47b402a560 comp rx=0 tx=0).stop 2024-06-29T15:59:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.266+0000 7f47cb865640 1 -- 172.21.15.59:0/4160346986 shutdown_connections 2024-06-29T15:59:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.266+0000 7f47cb865640 1 --2- 172.21.15.59:0/4160346986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c41444c0 0x7f47c406d2f0 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.266+0000 7f47cb865640 1 --2- 172.21.15.59:0/4160346986 >> v2:172.21.15.153:3300/0 conn(0x7f47c40cd9b0 0x7f47c415f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.266+0000 7f47cb865640 1 -- 172.21.15.59:0/4160346986 >> 172.21.15.59:0/4160346986 conn(0x7f47c40b24c0 msgr2=0x7f47c40cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.266+0000 7f47cb865640 1 -- 172.21.15.59:0/4160346986 shutdown_connections 2024-06-29T15:59:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.266+0000 7f47cb865640 1 -- 172.21.15.59:0/4160346986 wait complete. 2024-06-29T15:59:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.268+0000 7f47cb865640 1 Processor -- start 2024-06-29T15:59:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.268+0000 7f47cb865640 1 -- start start 2024-06-29T15:59:14.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47cb865640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c406d830 0x7f47c415a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47cb865640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c40cd9b0 0x7f47c4145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47cb865640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c41444c0 0x7f47c41459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47cb865640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f47c415b9b0 con 0x7f47c406d830 2024-06-29T15:59:14.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47cb865640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f47c41448d0 con 0x7f47c40cd9b0 2024-06-29T15:59:14.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47cb865640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f47c415bb30 con 0x7f47c41444c0 2024-06-29T15:59:14.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47c9601640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c406d830 0x7f47c415a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47c9601640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c406d830 0x7f47c415a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43412/0 (socket says 172.21.15.59:43412) 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47c9601640 1 -- 172.21.15.59:0/3672154597 learned_addr learned my addr 172.21.15.59:0/3672154597 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47c8e00640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c40cd9b0 0x7f47c4145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.269+0000 7f47c9e02640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c41444c0 0x7f47c41459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47c9601640 1 -- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c41444c0 msgr2=0x7f47c41459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47c9601640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c41444c0 0x7f47c41459c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47c9601640 1 -- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c40cd9b0 msgr2=0x7f47c4145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47c9601640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c40cd9b0 0x7f47c4145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47c9601640 1 -- 172.21.15.59:0/3672154597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c4146220 con 0x7f47c406d830 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47c9601640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c406d830 0x7f47c415a450 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f47c000d6d0 tx=0x7f47c000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47b9ffb640 1 -- 172.21.15.59:0/3672154597 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0017070 con 0x7f47c406d830 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47b9ffb640 1 -- 172.21.15.59:0/3672154597 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47c0005330 con 0x7f47c406d830 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c4146500 con 0x7f47c406d830 2024-06-29T15:59:14.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47b9ffb640 1 -- 172.21.15.59:0/3672154597 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0009820 con 0x7f47c406d830 2024-06-29T15:59:14.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.270+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c413dca0 con 0x7f47c406d830 2024-06-29T15:59:14.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.271+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c415bff0 con 0x7f47c406d830 2024-06-29T15:59:14.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.272+0000 7f47b9ffb640 1 -- 172.21.15.59:0/3672154597 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f47c0009bd0 con 0x7f47c406d830 2024-06-29T15:59:14.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.273+0000 7f47b9ffb640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47a8076510 0x7f47a8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.273+0000 7f47b9ffb640 1 -- 172.21.15.59:0/3672154597 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(261..261 src has 1..261) ==== 6157+0+0 (secure 0 0 0) 0x7f47c0098150 con 0x7f47c406d830 2024-06-29T15:59:14.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.273+0000 7f47c8e00640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47a8076510 0x7f47a8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.274+0000 7f47c8e00640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47a8076510 0x7f47a8078a20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f47b4033000 tx=0x7f47b4004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:14.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.277+0000 7f47b9ffb640 1 -- 172.21.15.59:0/3672154597 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47c0060d90 con 0x7f47c406d830 2024-06-29T15:59:14.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:14.513+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f47c406ca80 con 0x7f47c406d830 2024-06-29T15:59:15.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.454+0000 7f47b9ffb640 1 -- 172.21.15.59:0/3672154597 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v147) ==== 118+0+0 (secure 0 0 0) 0x7f47c00693d0 con 0x7f47c406d830 2024-06-29T15:59:15.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47a8076510 msgr2=0x7f47a8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47a8076510 0x7f47a8078a20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f47b4033000 tx=0x7f47b4004730 comp rx=0 tx=0).stop 2024-06-29T15:59:15.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c406d830 msgr2=0x7f47c415a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c406d830 0x7f47c415a450 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f47c000d6d0 tx=0x7f47c000db90 comp rx=0 tx=0).stop 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 shutdown_connections 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f47a8076510 0x7f47a8078a20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f47c41444c0 0x7f47c41459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c40cd9b0 0x7f47c4145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 --2- 172.21.15.59:0/3672154597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47c406d830 0x7f47c415a450 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 >> 172.21.15.59:0/3672154597 conn(0x7f47c40b24c0 msgr2=0x7f47c415e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 shutdown_connections 2024-06-29T15:59:15.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.457+0000 7f47cb865640 1 -- 172.21.15.59:0/3672154597 wait complete. 2024-06-29T15:59:15.472 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:59:15.473 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:59:15.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-06-29T15:59:15.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-06-29T15:59:15.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-06-29T15:59:15.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.601+0000 7f0a91568640 1 -- 172.21.15.59:0/1351600633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a78003ac0 msgr2=0x7f0a78003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.601+0000 7f0a91568640 1 --2- 172.21.15.59:0/1351600633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a78003ac0 0x7f0a78003f30 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c001690 tx=0x7f0a78005830 comp rx=0 tx=0).stop 2024-06-29T15:59:15.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.601+0000 7f0a91568640 1 -- 172.21.15.59:0/1351600633 shutdown_connections 2024-06-29T15:59:15.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.601+0000 7f0a91568640 1 --2- 172.21.15.59:0/1351600633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a78003ac0 0x7f0a78003f30 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.601+0000 7f0a91568640 1 -- 172.21.15.59:0/1351600633 >> 172.21.15.59:0/1351600633 conn(0x7f0a8c0d3ee0 msgr2=0x7f0a8c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.601+0000 7f0a91568640 1 -- 172.21.15.59:0/1351600633 shutdown_connections 2024-06-29T15:59:15.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.601+0000 7f0a91568640 1 -- 172.21.15.59:0/1351600633 wait complete. 2024-06-29T15:59:15.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.603+0000 7f0a91568640 1 Processor -- start 2024-06-29T15:59:15.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.603+0000 7f0a91568640 1 -- start start 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.604+0000 7f0a91568640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a78003ac0 0x7f0a8c155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.604+0000 7f0a91568640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c15dbc0 0x7f0a8c0c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.604+0000 7f0a91568640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a8c0c9d00 0x7f0a8c0cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.604+0000 7f0a91568640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c15ed10 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.604+0000 7f0a91568640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c15eb90 con 0x7f0a8c15dbc0 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.604+0000 7f0a91568640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c0b3990 con 0x7f0a78003ac0 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a8c0c9d00 0x7f0a8c0cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a8c0c9d00 0x7f0a8c0cac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43432/0 (socket says 172.21.15.59:43432) 2024-06-29T15:59:15.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 -- 172.21.15.59:0/2621911408 learned_addr learned my addr 172.21.15.59:0/2621911408 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a83fff640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c15dbc0 0x7f0a8c0c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8affd640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a78003ac0 0x7f0a8c155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 -- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a78003ac0 msgr2=0x7f0a8c155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a78003ac0 0x7f0a8c155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 -- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c15dbc0 msgr2=0x7f0a8c0c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c15dbc0 0x7f0a8c0c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 -- 172.21.15.59:0/2621911408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c0ca110 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a8b7fe640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a8c0c9d00 0x7f0a8c0cac70 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c75c0 tx=0x7f0a8400e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a837fe640 1 -- 172.21.15.59:0/2621911408 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84004b20 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a837fe640 1 -- 172.21.15.59:0/2621911408 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a84015070 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c0cc8c0 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a837fe640 1 -- 172.21.15.59:0/2621911408 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84003970 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.605+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c0cd190 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.606+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c0b3ef0 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.607+0000 7f0a837fe640 1 -- 172.21.15.59:0/2621911408 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f0a84003b10 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.608+0000 7f0a837fe640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0a6c076510 0x7f0a6c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.608+0000 7f0a837fe640 1 -- 172.21.15.59:0/2621911408 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f0a84098530 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.608+0000 7f0a8affd640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0a6c076510 0x7f0a6c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.609+0000 7f0a8affd640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0a6c076510 0x7f0a6c078a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c032000 tx=0x7f0a7c02c040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.612+0000 7f0a837fe640 1 -- 172.21.15.59:0/2621911408 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0a8c0ca110 con 0x7f0a8c0c9d00 2024-06-29T15:59:15.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:15.845+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0a8c0d2950 con 0x7f0a8c0c9d00 2024-06-29T15:59:16.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.458+0000 7f0a837fe640 1 -- 172.21.15.59:0/2621911408 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v148) ==== 122+0+0 (secure 0 0 0) 0x7f0a84061320 con 0x7f0a8c0c9d00 2024-06-29T15:59:16.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0a6c076510 msgr2=0x7f0a6c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0a6c076510 0x7f0a6c078a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c032000 tx=0x7f0a7c02c040 comp rx=0 tx=0).stop 2024-06-29T15:59:16.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a8c0c9d00 msgr2=0x7f0a8c0cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a8c0c9d00 0x7f0a8c0cac70 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c75c0 tx=0x7f0a8400e350 comp rx=0 tx=0).stop 2024-06-29T15:59:16.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 shutdown_connections 2024-06-29T15:59:16.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f0a6c076510 0x7f0a6c078a20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a8c0c9d00 0x7f0a8c0cac70 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c15dbc0 0x7f0a8c0c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 --2- 172.21.15.59:0/2621911408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0a78003ac0 0x7f0a8c155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 >> 172.21.15.59:0/2621911408 conn(0x7f0a8c0d3ee0 msgr2=0x7f0a8c161520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:16.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 shutdown_connections 2024-06-29T15:59:16.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.460+0000 7f0a91568640 1 -- 172.21.15.59:0/2621911408 wait complete. 2024-06-29T15:59:16.475 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:59:16.475 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:16.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 -- 172.21.15.59:0/1103320130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00b3ed0 msgr2=0x7f73a015e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1103320130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00b3ed0 0x7f73a015e8d0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f73a00d04f0 tx=0x7f739002aa30 comp rx=0 tx=0).stop 2024-06-29T15:59:16.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 -- 172.21.15.59:0/1103320130 shutdown_connections 2024-06-29T15:59:16.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1103320130 >> v2:172.21.15.153:3300/0 conn(0x7f73a015ee10 0x7f73a015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1103320130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00b3ed0 0x7f73a015e8d0 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1103320130 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73a00d23c0 0x7f73a00b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 -- 172.21.15.59:0/1103320130 >> 172.21.15.59:0/1103320130 conn(0x7f73a00d3ee0 msgr2=0x7f73a00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:16.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 -- 172.21.15.59:0/1103320130 shutdown_connections 2024-06-29T15:59:16.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.598+0000 7f73a6e09640 1 -- 172.21.15.59:0/1103320130 wait complete. 2024-06-29T15:59:16.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.600+0000 7f73a6e09640 1 Processor -- start 2024-06-29T15:59:16.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.600+0000 7f73a6e09640 1 -- start start 2024-06-29T15:59:16.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.601+0000 7f73a6e09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00b3ed0 0x7f73a00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.601+0000 7f73a6e09640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00d23c0 0x7f73a015b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f73a6e09640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73a015ee10 0x7f73a0160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f73a6e09640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f73a0160070 con 0x7f73a00d23c0 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f73a6e09640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73a015fef0 con 0x7f73a00b3ed0 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f73a6e09640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f73a01601f0 con 0x7f73a015ee10 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00d23c0 0x7f73a015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00d23c0 0x7f73a015b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43442/0 (socket says 172.21.15.59:43442) 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 -- 172.21.15.59:0/1303781949 learned_addr learned my addr 172.21.15.59:0/1303781949 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f73a4ba5640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00b3ed0 0x7f73a00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f73a53a6640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73a015ee10 0x7f73a0160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 -- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73a015ee10 msgr2=0x7f73a0160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73a015ee10 0x7f73a0160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 -- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00b3ed0 msgr2=0x7f73a00cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00b3ed0 0x7f73a00cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 -- 172.21.15.59:0/1303781949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a015e530 con 0x7f73a00d23c0 2024-06-29T15:59:16.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739ffff640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00d23c0 0x7f73a015b380 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f7390009d60 tx=0x7f7390003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:16.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739d7fa640 1 -- 172.21.15.59:0/1303781949 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390033050 con 0x7f73a00d23c0 2024-06-29T15:59:16.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739d7fa640 1 -- 172.21.15.59:0/1303781949 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73900044d0 con 0x7f73a00d23c0 2024-06-29T15:59:16.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f739d7fa640 1 -- 172.21.15.59:0/1303781949 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390038450 con 0x7f73a00d23c0 2024-06-29T15:59:16.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.602+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a0158fa0 con 0x7f73a00d23c0 2024-06-29T15:59:16.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.603+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a015df70 con 0x7f73a00d23c0 2024-06-29T15:59:16.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.604+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a00b3990 con 0x7f73a00d23c0 2024-06-29T15:59:16.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.604+0000 7f739d7fa640 1 -- 172.21.15.59:0/1303781949 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f7390036070 con 0x7f73a00d23c0 2024-06-29T15:59:16.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.605+0000 7f739d7fa640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7380076510 0x7f7380078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.605+0000 7f739d7fa640 1 -- 172.21.15.59:0/1303781949 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f73900b9260 con 0x7f73a00d23c0 2024-06-29T15:59:16.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.605+0000 7f73a4ba5640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7380076510 0x7f7380078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.606+0000 7f73a4ba5640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7380076510 0x7f7380078a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f73a00d2e40 tx=0x7f738800b260 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:16.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.609+0000 7f739d7fa640 1 -- 172.21.15.59:0/1303781949 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7390081f40 con 0x7f73a00d23c0 2024-06-29T15:59:16.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.839+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73a00d2950 con 0x7f73a00d23c0 2024-06-29T15:59:16.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.839+0000 7f739d7fa640 1 -- 172.21.15.59:0/1303781949 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5247 (secure 0 0 0) 0x7f7390086240 con 0x7f73a00d23c0 2024-06-29T15:59:16.841 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:16.841 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":148,"btime":"2024-06-29T15:59:16:455546+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1010984928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1010984928},{"type":"v1","addr":"172.21.15.153:6836","nonce":1010984928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":10475,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2101409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2101409892},{"type":"v1","addr":"172.21.15.153:6837","nonce":2101409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":12130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1208243017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1208243017},{"type":"v1","addr":"172.21.15.59:6835","nonce":1208243017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:12.808119+0000","modified":"2024-06-29T15:59:15.847810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13110":{"gid":13110,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/2985893835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2985893835},{"type":"v1","addr":"172.21.15.59:6833","nonce":2985893835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13110,"qdb_cluster":[13110]},"id":17}]} 2024-06-29T15:59:16.841 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 148 2024-06-29T15:59:16.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.841+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7380076510 msgr2=0x7f7380078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.841+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7380076510 0x7f7380078a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f73a00d2e40 tx=0x7f738800b260 comp rx=0 tx=0).stop 2024-06-29T15:59:16.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.841+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00d23c0 msgr2=0x7f73a015b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.841+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00d23c0 0x7f73a015b380 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f7390009d60 tx=0x7f7390003f10 comp rx=0 tx=0).stop 2024-06-29T15:59:16.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.842+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 shutdown_connections 2024-06-29T15:59:16.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.842+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f7380076510 0x7f7380078a20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.842+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73a015ee10 0x7f73a0160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.842+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73a00d23c0 0x7f73a015b380 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.842+0000 7f73a6e09640 1 --2- 172.21.15.59:0/1303781949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00b3ed0 0x7f73a00cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.842+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 >> 172.21.15.59:0/1303781949 conn(0x7f73a00d3ee0 msgr2=0x7f73a006cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:16.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.842+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 shutdown_connections 2024-06-29T15:59:16.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.842+0000 7f73a6e09640 1 -- 172.21.15.59:0/1303781949 wait complete. 2024-06-29T15:59:16.854 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:16.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.971+0000 7f3cea5d2640 1 -- 172.21.15.59:0/3613210495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ce40ce950 msgr2=0x7f3ce41622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.971+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/3613210495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ce40ce950 0x7f3ce41622c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009900 tx=0x7f3cd802a610 comp rx=0 tx=0).stop 2024-06-29T15:59:16.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.971+0000 7f3cea5d2640 1 -- 172.21.15.59:0/3613210495 shutdown_connections 2024-06-29T15:59:16.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.971+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/3613210495 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ce4162800 0x7f3ce4162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.971+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/3613210495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ce40ce950 0x7f3ce41622c0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.971+0000 7f3cea5d2640 1 -- 172.21.15.59:0/3613210495 >> 172.21.15.59:0/3613210495 conn(0x7f3ce40b5610 msgr2=0x7f3ce40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:16.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.971+0000 7f3cea5d2640 1 -- 172.21.15.59:0/3613210495 shutdown_connections 2024-06-29T15:59:16.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.971+0000 7f3cea5d2640 1 -- 172.21.15.59:0/3613210495 wait complete. 2024-06-29T15:59:16.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.973+0000 7f3cea5d2640 1 Processor -- start 2024-06-29T15:59:16.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.973+0000 7f3cea5d2640 1 -- start start 2024-06-29T15:59:16.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3cea5d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ce408c190 0x7f3ce4154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3cea5d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ce40ce950 0x7f3ce40cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3cea5d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ce4162800 0x7f3ce4146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3cea5d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce40b7f60 con 0x7f3ce40ce950 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3cea5d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce40b7de0 con 0x7f3ce408c190 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3cea5d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce40b80e0 con 0x7f3ce4162800 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ce40ce950 0x7f3ce40cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3ce37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ce40ce950 0x7f3ce40cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43446/0 (socket says 172.21.15.59:43446) 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3ce37fe640 1 -- 172.21.15.59:0/2455143621 learned_addr learned my addr 172.21.15.59:0/2455143621 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3ce8b6f640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ce4162800 0x7f3ce4146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.974+0000 7f3ce3fff640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ce408c190 0x7f3ce4154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce37fe640 1 -- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ce4162800 msgr2=0x7f3ce4146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce37fe640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ce4162800 0x7f3ce4146d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce37fe640 1 -- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ce408c190 msgr2=0x7f3ce4154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce37fe640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ce408c190 0x7f3ce4154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce37fe640 1 -- 172.21.15.59:0/2455143621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce4160a30 con 0x7f3ce40ce950 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce37fe640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ce40ce950 0x7f3ce40cb3d0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8034040 tx=0x7f3cd80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce0ff9640 1 -- 172.21.15.59:0/2455143621 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd803e030 con 0x7f3ce40ce950 2024-06-29T15:59:16.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce0ff9640 1 -- 172.21.15.59:0/2455143621 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cd8003ae0 con 0x7f3ce40ce950 2024-06-29T15:59:16.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce4160d10 con 0x7f3ce40ce950 2024-06-29T15:59:16.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3ce0ff9640 1 -- 172.21.15.59:0/2455143621 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8035040 con 0x7f3ce40ce950 2024-06-29T15:59:16.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.975+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce41612d0 con 0x7f3ce40ce950 2024-06-29T15:59:16.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.976+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce41622c0 con 0x7f3ce40ce950 2024-06-29T15:59:16.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.977+0000 7f3ce0ff9640 1 -- 172.21.15.59:0/2455143621 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f3cd8041030 con 0x7f3ce40ce950 2024-06-29T15:59:16.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.977+0000 7f3ce0ff9640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3cd0076510 0x7f3cd0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.978+0000 7f3ce0ff9640 1 -- 172.21.15.59:0/2455143621 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f3cd80b81a0 con 0x7f3ce40ce950 2024-06-29T15:59:16.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.978+0000 7f3ce3fff640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3cd0076510 0x7f3cd0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.978+0000 7f3ce3fff640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3cd0076510 0x7f3cd0078a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40b7620 tx=0x7f3cd4009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:16.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:16.982+0000 7f3ce0ff9640 1 -- 172.21.15.59:0/2455143621 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3cd8080ed0 con 0x7f3ce40ce950 2024-06-29T15:59:17.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.174+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ce406c430 con 0x7f3ce40ce950 2024-06-29T15:59:17.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.176+0000 7f3ce0ff9640 1 -- 172.21.15.59:0/2455143621 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) ==== 74+0+16990 (secure 0 0 0) 0x7f3cd80851d0 con 0x7f3ce40ce950 2024-06-29T15:59:17.177 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:17.177 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":262,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:14.419180+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:11.169128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:11.760290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:17.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3cd0076510 msgr2=0x7f3cd0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3cd0076510 0x7f3cd0078a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40b7620 tx=0x7f3cd4009040 comp rx=0 tx=0).stop 2024-06-29T15:59:17.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ce40ce950 msgr2=0x7f3ce40cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ce40ce950 0x7f3ce40cb3d0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8034040 tx=0x7f3cd80035f0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 shutdown_connections 2024-06-29T15:59:17.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f3cd0076510 0x7f3cd0078a20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3ce4162800 0x7f3ce4146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ce40ce950 0x7f3ce40cb3d0 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 --2- 172.21.15.59:0/2455143621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ce408c190 0x7f3ce4154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 >> 172.21.15.59:0/2455143621 conn(0x7f3ce40b5610 msgr2=0x7f3ce40c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:17.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 shutdown_connections 2024-06-29T15:59:17.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.178+0000 7f3cea5d2640 1 -- 172.21.15.59:0/2455143621 wait complete. 2024-06-29T15:59:17.191 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:59:17.193 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:59:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.309+0000 7f114b7eb640 1 -- 172.21.15.59:0/2061866934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144161fc0 msgr2=0x7f11440d2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.309+0000 7f114b7eb640 1 --2- 172.21.15.59:0/2061866934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144161fc0 0x7f11440d2670 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f11440b35d0 tx=0x7f113c02a970 comp rx=0 tx=0).stop 2024-06-29T15:59:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.309+0000 7f114b7eb640 1 -- 172.21.15.59:0/2061866934 shutdown_connections 2024-06-29T15:59:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.309+0000 7f114b7eb640 1 --2- 172.21.15.59:0/2061866934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144161fc0 0x7f11440d2670 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.309+0000 7f114b7eb640 1 --2- 172.21.15.59:0/2061866934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f114415e0a0 0x7f1144161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.310+0000 7f114b7eb640 1 --2- 172.21.15.59:0/2061866934 >> v2:172.21.15.153:3300/0 conn(0x7f1144162770 0x7f11440ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.310+0000 7f114b7eb640 1 -- 172.21.15.59:0/2061866934 >> 172.21.15.59:0/2061866934 conn(0x7f11440d45b0 msgr2=0x7f11440d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.310+0000 7f114b7eb640 1 -- 172.21.15.59:0/2061866934 shutdown_connections 2024-06-29T15:59:17.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.310+0000 7f114b7eb640 1 -- 172.21.15.59:0/2061866934 wait complete. 2024-06-29T15:59:17.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.311+0000 7f114b7eb640 1 Processor -- start 2024-06-29T15:59:17.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.311+0000 7f114b7eb640 1 -- start start 2024-06-29T15:59:17.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f114b7eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114415e0a0 0x7f1144153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f114b7eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1144161fc0 0x7f1144156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f114b7eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144162770 0x7f11441f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f114b7eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f11440d2f40 con 0x7f1144162770 2024-06-29T15:59:17.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f114b7eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11440d2dc0 con 0x7f114415e0a0 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f114b7eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11440d30c0 con 0x7f1144161fc0 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144162770 0x7f11441f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144162770 0x7f11441f81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43462/0 (socket says 172.21.15.59:43462) 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 -- 172.21.15.59:0/3027695976 learned_addr learned my addr 172.21.15.59:0/3027695976 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149587640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114415e0a0 0x7f1144153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1148d86640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1144161fc0 0x7f1144156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 -- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1144161fc0 msgr2=0x7f1144156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1144161fc0 0x7f1144156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 -- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114415e0a0 msgr2=0x7f1144153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114415e0a0 0x7f1144153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 -- 172.21.15.59:0/3027695976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1144156fb0 con 0x7f1144162770 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.313+0000 7f1149d88640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144162770 0x7f11441f81c0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f113c005fd0 tx=0x7f113c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:17.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.314+0000 7f1139ffb640 1 -- 172.21.15.59:0/3027695976 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c034050 con 0x7f1144162770 2024-06-29T15:59:17.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.314+0000 7f1139ffb640 1 -- 172.21.15.59:0/3027695976 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f113c004550 con 0x7f1144162770 2024-06-29T15:59:17.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.314+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1144154fc0 con 0x7f1144162770 2024-06-29T15:59:17.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.314+0000 7f1139ffb640 1 -- 172.21.15.59:0/3027695976 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c043440 con 0x7f1144162770 2024-06-29T15:59:17.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.314+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1144157eb0 con 0x7f1144162770 2024-06-29T15:59:17.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.315+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11440b9400 con 0x7f1144162770 2024-06-29T15:59:17.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.316+0000 7f1139ffb640 1 -- 172.21.15.59:0/3027695976 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f113c037020 con 0x7f1144162770 2024-06-29T15:59:17.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.316+0000 7f1139ffb640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1128076540 0x7f1128078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.317+0000 7f1139ffb640 1 -- 172.21.15.59:0/3027695976 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f113c0b9140 con 0x7f1144162770 2024-06-29T15:59:17.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.317+0000 7f1149587640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1128076540 0x7f1128078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.317+0000 7f1149587640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1128076540 0x7f1128078a50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1140005fd0 tx=0x7f114000a250 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:17.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.321+0000 7f1139ffb640 1 -- 172.21.15.59:0/3027695976 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11441623d0 con 0x7f1144162770 2024-06-29T15:59:17.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.511+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 --> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1144075b40 con 0x7f1128076540 2024-06-29T15:59:17.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.516+0000 7f1139ffb640 1 -- 172.21.15.59:0/3027695976 <== mgr.11797 v2:172.21.15.153:6832/3574471609 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252494 (secure 0 0 0) 0x7f1144075b40 con 0x7f1128076540 2024-06-29T15:59:17.520 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:17.521 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:59:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1128076540 msgr2=0x7f1128078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1128076540 0x7f1128078a50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1140005fd0 tx=0x7f114000a250 comp rx=0 tx=0).stop 2024-06-29T15:59:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144162770 msgr2=0x7f11441f81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144162770 0x7f11441f81c0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f113c005fd0 tx=0x7f113c003f90 comp rx=0 tx=0).stop 2024-06-29T15:59:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 shutdown_connections 2024-06-29T15:59:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1128076540 0x7f1128078a50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1144162770 0x7f11441f81c0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1144161fc0 0x7f1144156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 --2- 172.21.15.59:0/3027695976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f114415e0a0 0x7f1144153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 >> 172.21.15.59:0/3027695976 conn(0x7f11440d45b0 msgr2=0x7f11440b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 shutdown_connections 2024-06-29T15:59:17.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.519+0000 7f114b7eb640 1 -- 172.21.15.59:0/3027695976 wait complete. 2024-06-29T15:59:17.531 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-06-29T15:59:17.038978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1668,"num_read_kb":4792,"num_write":466,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":298,"ondisk_log_size":298,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":447280,"kb_used_data":27048,"kb_used_omap":12,"kb_used_meta":420083,"kb_avail":754527440,"statfs":{"total":773094113280,"available":772636098560,"internally_reserved":0,"allocated":27697152,"data_stored":14922356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":430165554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.633931"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424021+0000","last_change":"2024-06-29T15:59:13.439581+0000","last_active":"2024-06-29T15:59:14.424021+0000","last_peered":"2024-06-29T15:59:14.424021+0000","last_clean":"2024-06-29T15:59:14.424021+0000","last_became_active":"2024-06-29T15:59:13.439283+0000","last_became_peered":"2024-06-29T15:59:13.439283+0000","last_unstale":"2024-06-29T15:59:14.424021+0000","last_undegraded":"2024-06-29T15:59:14.424021+0000","last_fullsized":"2024-06-29T15:59:14.424021+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:02.699526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424016+0000","last_change":"2024-06-29T15:59:13.430811+0000","last_active":"2024-06-29T15:59:14.424016+0000","last_peered":"2024-06-29T15:59:14.424016+0000","last_clean":"2024-06-29T15:59:14.424016+0000","last_became_active":"2024-06-29T15:59:13.429734+0000","last_became_peered":"2024-06-29T15:59:13.429734+0000","last_unstale":"2024-06-29T15:59:14.424016+0000","last_undegraded":"2024-06-29T15:59:14.424016+0000","last_fullsized":"2024-06-29T15:59:14.424016+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:24.631972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445791+0000","last_change":"2024-06-29T15:59:13.451333+0000","last_active":"2024-06-29T15:59:14.445791+0000","last_peered":"2024-06-29T15:59:14.445791+0000","last_clean":"2024-06-29T15:59:14.445791+0000","last_became_active":"2024-06-29T15:59:13.451007+0000","last_became_peered":"2024-06-29T15:59:13.451007+0000","last_unstale":"2024-06-29T15:59:14.445791+0000","last_undegraded":"2024-06-29T15:59:14.445791+0000","last_fullsized":"2024-06-29T15:59:14.445791+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:40.892308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446813+0000","last_change":"2024-06-29T15:59:13.440536+0000","last_active":"2024-06-29T15:59:14.446813+0000","last_peered":"2024-06-29T15:59:14.446813+0000","last_clean":"2024-06-29T15:59:14.446813+0000","last_became_active":"2024-06-29T15:59:13.440220+0000","last_became_peered":"2024-06-29T15:59:13.440220+0000","last_unstale":"2024-06-29T15:59:14.446813+0000","last_undegraded":"2024-06-29T15:59:14.446813+0000","last_fullsized":"2024-06-29T15:59:14.446813+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:21.792076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425066+0000","last_change":"2024-06-29T15:59:13.443124+0000","last_active":"2024-06-29T15:59:14.425066+0000","last_peered":"2024-06-29T15:59:14.425066+0000","last_clean":"2024-06-29T15:59:14.425066+0000","last_became_active":"2024-06-29T15:59:13.442775+0000","last_became_peered":"2024-06-29T15:59:13.442775+0000","last_unstale":"2024-06-29T15:59:14.425066+0000","last_undegraded":"2024-06-29T15:59:14.425066+0000","last_fullsized":"2024-06-29T15:59:14.425066+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:52.400423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425003+0000","last_change":"2024-06-29T15:59:13.455594+0000","last_active":"2024-06-29T15:59:14.425003+0000","last_peered":"2024-06-29T15:59:14.425003+0000","last_clean":"2024-06-29T15:59:14.425003+0000","last_became_active":"2024-06-29T15:59:13.454176+0000","last_became_peered":"2024-06-29T15:59:13.454176+0000","last_unstale":"2024-06-29T15:59:14.425003+0000","last_undegraded":"2024-06-29T15:59:14.425003+0000","last_fullsized":"2024-06-29T15:59:14.425003+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:36.307861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444718+0000","last_change":"2024-06-29T15:59:13.439926+0000","last_active":"2024-06-29T15:59:14.444718+0000","last_peered":"2024-06-29T15:59:14.444718+0000","last_clean":"2024-06-29T15:59:14.444718+0000","last_became_active":"2024-06-29T15:59:13.439588+0000","last_became_peered":"2024-06-29T15:59:13.439588+0000","last_unstale":"2024-06-29T15:59:14.444718+0000","last_undegraded":"2024-06-29T15:59:14.444718+0000","last_fullsized":"2024-06-29T15:59:14.444718+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:19.549486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446404+0000","last_change":"2024-06-29T15:59:13.428209+0000","last_active":"2024-06-29T15:59:14.446404+0000","last_peered":"2024-06-29T15:59:14.446404+0000","last_clean":"2024-06-29T15:59:14.446404+0000","last_became_active":"2024-06-29T15:59:13.427874+0000","last_became_peered":"2024-06-29T15:59:13.427874+0000","last_unstale":"2024-06-29T15:59:14.446404+0000","last_undegraded":"2024-06-29T15:59:14.446404+0000","last_fullsized":"2024-06-29T15:59:14.446404+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:05:58.293345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445637+0000","last_change":"2024-06-29T15:59:13.445452+0000","last_active":"2024-06-29T15:59:14.445637+0000","last_peered":"2024-06-29T15:59:14.445637+0000","last_clean":"2024-06-29T15:59:14.445637+0000","last_became_active":"2024-06-29T15:59:13.445076+0000","last_became_peered":"2024-06-29T15:59:13.445076+0000","last_unstale":"2024-06-29T15:59:14.445637+0000","last_undegraded":"2024-06-29T15:59:14.445637+0000","last_fullsized":"2024-06-29T15:59:14.445637+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:38.149677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445308+0000","last_change":"2024-06-29T15:59:13.439287+0000","last_active":"2024-06-29T15:59:14.445308+0000","last_peered":"2024-06-29T15:59:14.445308+0000","last_clean":"2024-06-29T15:59:14.445308+0000","last_became_active":"2024-06-29T15:59:13.438730+0000","last_became_peered":"2024-06-29T15:59:13.438730+0000","last_unstale":"2024-06-29T15:59:14.445308+0000","last_undegraded":"2024-06-29T15:59:14.445308+0000","last_fullsized":"2024-06-29T15:59:14.445308+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:10.673096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446558+0000","last_change":"2024-06-29T15:59:13.437106+0000","last_active":"2024-06-29T15:59:14.446558+0000","last_peered":"2024-06-29T15:59:14.446558+0000","last_clean":"2024-06-29T15:59:14.446558+0000","last_became_active":"2024-06-29T15:59:13.436750+0000","last_became_peered":"2024-06-29T15:59:13.436750+0000","last_unstale":"2024-06-29T15:59:14.446558+0000","last_undegraded":"2024-06-29T15:59:14.446558+0000","last_fullsized":"2024-06-29T15:59:14.446558+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:29:51.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424010+0000","last_change":"2024-06-29T15:59:13.442944+0000","last_active":"2024-06-29T15:59:14.424010+0000","last_peered":"2024-06-29T15:59:14.424010+0000","last_clean":"2024-06-29T15:59:14.424010+0000","last_became_active":"2024-06-29T15:59:13.442097+0000","last_became_peered":"2024-06-29T15:59:13.442097+0000","last_unstale":"2024-06-29T15:59:14.424010+0000","last_undegraded":"2024-06-29T15:59:14.424010+0000","last_fullsized":"2024-06-29T15:59:14.424010+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:57.668260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445626+0000","last_change":"2024-06-29T15:59:13.435581+0000","last_active":"2024-06-29T15:59:14.445626+0000","last_peered":"2024-06-29T15:59:14.445626+0000","last_clean":"2024-06-29T15:59:14.445626+0000","last_became_active":"2024-06-29T15:59:13.435261+0000","last_became_peered":"2024-06-29T15:59:13.435261+0000","last_unstale":"2024-06-29T15:59:14.445626+0000","last_undegraded":"2024-06-29T15:59:14.445626+0000","last_fullsized":"2024-06-29T15:59:14.445626+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:53:30.858386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445176+0000","last_change":"2024-06-29T15:59:13.448375+0000","last_active":"2024-06-29T15:59:14.445176+0000","last_peered":"2024-06-29T15:59:14.445176+0000","last_clean":"2024-06-29T15:59:14.445176+0000","last_became_active":"2024-06-29T15:59:13.447766+0000","last_became_peered":"2024-06-29T15:59:13.447766+0000","last_unstale":"2024-06-29T15:59:14.445176+0000","last_undegraded":"2024-06-29T15:59:14.445176+0000","last_fullsized":"2024-06-29T15:59:14.445176+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:25:01.384822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445368+0000","last_change":"2024-06-29T15:59:13.433667+0000","last_active":"2024-06-29T15:59:14.445368+0000","last_peered":"2024-06-29T15:59:14.445368+0000","last_clean":"2024-06-29T15:59:14.445368+0000","last_became_active":"2024-06-29T15:59:13.433355+0000","last_became_peered":"2024-06-29T15:59:13.433355+0000","last_unstale":"2024-06-29T15:59:14.445368+0000","last_undegraded":"2024-06-29T15:59:14.445368+0000","last_fullsized":"2024-06-29T15:59:14.445368+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:27.495662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444753+0000","last_change":"2024-06-29T15:59:13.434557+0000","last_active":"2024-06-29T15:59:14.444753+0000","last_peered":"2024-06-29T15:59:14.444753+0000","last_clean":"2024-06-29T15:59:14.444753+0000","last_became_active":"2024-06-29T15:59:13.434208+0000","last_became_peered":"2024-06-29T15:59:13.434208+0000","last_unstale":"2024-06-29T15:59:14.444753+0000","last_undegraded":"2024-06-29T15:59:14.444753+0000","last_fullsized":"2024-06-29T15:59:14.444753+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:40.892308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445293+0000","last_change":"2024-06-29T15:59:13.442005+0000","last_active":"2024-06-29T15:59:14.445293+0000","last_peered":"2024-06-29T15:59:14.445293+0000","last_clean":"2024-06-29T15:59:14.445293+0000","last_became_active":"2024-06-29T15:59:13.441467+0000","last_became_peered":"2024-06-29T15:59:13.441467+0000","last_unstale":"2024-06-29T15:59:14.445293+0000","last_undegraded":"2024-06-29T15:59:14.445293+0000","last_fullsized":"2024-06-29T15:59:14.445293+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:44.145906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446278+0000","last_change":"2024-06-29T15:59:13.452999+0000","last_active":"2024-06-29T15:59:14.446278+0000","last_peered":"2024-06-29T15:59:14.446278+0000","last_clean":"2024-06-29T15:59:14.446278+0000","last_became_active":"2024-06-29T15:59:13.451855+0000","last_became_peered":"2024-06-29T15:59:13.451855+0000","last_unstale":"2024-06-29T15:59:14.446278+0000","last_undegraded":"2024-06-29T15:59:14.446278+0000","last_fullsized":"2024-06-29T15:59:14.446278+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:03.408653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445031+0000","last_change":"2024-06-29T15:59:13.441103+0000","last_active":"2024-06-29T15:59:14.445031+0000","last_peered":"2024-06-29T15:59:14.445031+0000","last_clean":"2024-06-29T15:59:14.445031+0000","last_became_active":"2024-06-29T15:59:13.440750+0000","last_became_peered":"2024-06-29T15:59:13.440750+0000","last_unstale":"2024-06-29T15:59:14.445031+0000","last_undegraded":"2024-06-29T15:59:14.445031+0000","last_fullsized":"2024-06-29T15:59:14.445031+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:15:48.372315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445163+0000","last_change":"2024-06-29T15:59:13.435152+0000","last_active":"2024-06-29T15:59:14.445163+0000","last_peered":"2024-06-29T15:59:14.445163+0000","last_clean":"2024-06-29T15:59:14.445163+0000","last_became_active":"2024-06-29T15:59:13.434649+0000","last_became_peered":"2024-06-29T15:59:13.434649+0000","last_unstale":"2024-06-29T15:59:14.445163+0000","last_undegraded":"2024-06-29T15:59:14.445163+0000","last_fullsized":"2024-06-29T15:59:14.445163+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:52.601535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445771+0000","last_change":"2024-06-29T15:59:13.439978+0000","last_active":"2024-06-29T15:59:14.445771+0000","last_peered":"2024-06-29T15:59:14.445771+0000","last_clean":"2024-06-29T15:59:14.445771+0000","last_became_active":"2024-06-29T15:59:13.439632+0000","last_became_peered":"2024-06-29T15:59:13.439632+0000","last_unstale":"2024-06-29T15:59:14.445771+0000","last_undegraded":"2024-06-29T15:59:14.445771+0000","last_fullsized":"2024-06-29T15:59:14.445771+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:16.101361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447543+0000","last_change":"2024-06-29T15:59:13.451368+0000","last_active":"2024-06-29T15:59:14.447543+0000","last_peered":"2024-06-29T15:59:14.447543+0000","last_clean":"2024-06-29T15:59:14.447543+0000","last_became_active":"2024-06-29T15:59:13.451076+0000","last_became_peered":"2024-06-29T15:59:13.451076+0000","last_unstale":"2024-06-29T15:59:14.447543+0000","last_undegraded":"2024-06-29T15:59:14.447543+0000","last_fullsized":"2024-06-29T15:59:14.447543+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:04.885401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444606+0000","last_change":"2024-06-29T15:59:13.451673+0000","last_active":"2024-06-29T15:59:14.444606+0000","last_peered":"2024-06-29T15:59:14.444606+0000","last_clean":"2024-06-29T15:59:14.444606+0000","last_became_active":"2024-06-29T15:59:13.451341+0000","last_became_peered":"2024-06-29T15:59:13.451341+0000","last_unstale":"2024-06-29T15:59:14.444606+0000","last_undegraded":"2024-06-29T15:59:14.444606+0000","last_fullsized":"2024-06-29T15:59:14.444606+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:49.789073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447170+0000","last_change":"2024-06-29T15:59:13.448480+0000","last_active":"2024-06-29T15:59:14.447170+0000","last_peered":"2024-06-29T15:59:14.447170+0000","last_clean":"2024-06-29T15:59:14.447170+0000","last_became_active":"2024-06-29T15:59:13.448061+0000","last_became_peered":"2024-06-29T15:59:13.448061+0000","last_unstale":"2024-06-29T15:59:14.447170+0000","last_undegraded":"2024-06-29T15:59:14.447170+0000","last_fullsized":"2024-06-29T15:59:14.447170+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:58.902537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424212+0000","last_change":"2024-06-29T15:59:13.441128+0000","last_active":"2024-06-29T15:59:14.424212+0000","last_peered":"2024-06-29T15:59:14.424212+0000","last_clean":"2024-06-29T15:59:14.424212+0000","last_became_active":"2024-06-29T15:59:13.440804+0000","last_became_peered":"2024-06-29T15:59:13.440804+0000","last_unstale":"2024-06-29T15:59:14.424212+0000","last_undegraded":"2024-06-29T15:59:14.424212+0000","last_fullsized":"2024-06-29T15:59:14.424212+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:01:44.110046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425386+0000","last_change":"2024-06-29T15:59:13.442744+0000","last_active":"2024-06-29T15:59:14.425386+0000","last_peered":"2024-06-29T15:59:14.425386+0000","last_clean":"2024-06-29T15:59:14.425386+0000","last_became_active":"2024-06-29T15:59:13.442242+0000","last_became_peered":"2024-06-29T15:59:13.442242+0000","last_unstale":"2024-06-29T15:59:14.425386+0000","last_undegraded":"2024-06-29T15:59:14.425386+0000","last_fullsized":"2024-06-29T15:59:14.425386+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:16.306319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425350+0000","last_change":"2024-06-29T15:59:13.432309+0000","last_active":"2024-06-29T15:59:14.425350+0000","last_peered":"2024-06-29T15:59:14.425350+0000","last_clean":"2024-06-29T15:59:14.425350+0000","last_became_active":"2024-06-29T15:59:13.431982+0000","last_became_peered":"2024-06-29T15:59:13.431982+0000","last_unstale":"2024-06-29T15:59:14.425350+0000","last_undegraded":"2024-06-29T15:59:14.425350+0000","last_fullsized":"2024-06-29T15:59:14.425350+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:22.491582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445804+0000","last_change":"2024-06-29T15:59:13.440493+0000","last_active":"2024-06-29T15:59:14.445804+0000","last_peered":"2024-06-29T15:59:14.445804+0000","last_clean":"2024-06-29T15:59:14.445804+0000","last_became_active":"2024-06-29T15:59:13.440062+0000","last_became_peered":"2024-06-29T15:59:13.440062+0000","last_unstale":"2024-06-29T15:59:14.445804+0000","last_undegraded":"2024-06-29T15:59:14.445804+0000","last_fullsized":"2024-06-29T15:59:14.445804+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:19.852632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423490+0000","last_change":"2024-06-29T15:59:13.441504+0000","last_active":"2024-06-29T15:59:14.423490+0000","last_peered":"2024-06-29T15:59:14.423490+0000","last_clean":"2024-06-29T15:59:14.423490+0000","last_became_active":"2024-06-29T15:59:13.441166+0000","last_became_peered":"2024-06-29T15:59:13.441166+0000","last_unstale":"2024-06-29T15:59:14.423490+0000","last_undegraded":"2024-06-29T15:59:14.423490+0000","last_fullsized":"2024-06-29T15:59:14.423490+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:12.360125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447413+0000","last_change":"2024-06-29T15:59:13.451137+0000","last_active":"2024-06-29T15:59:14.447413+0000","last_peered":"2024-06-29T15:59:14.447413+0000","last_clean":"2024-06-29T15:59:14.447413+0000","last_became_active":"2024-06-29T15:59:13.450774+0000","last_became_peered":"2024-06-29T15:59:13.450774+0000","last_unstale":"2024-06-29T15:59:14.447413+0000","last_undegraded":"2024-06-29T15:59:14.447413+0000","last_fullsized":"2024-06-29T15:59:14.447413+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:36.576595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447043+0000","last_change":"2024-06-29T15:59:13.433269+0000","last_active":"2024-06-29T15:59:14.447043+0000","last_peered":"2024-06-29T15:59:14.447043+0000","last_clean":"2024-06-29T15:59:14.447043+0000","last_became_active":"2024-06-29T15:59:13.432928+0000","last_became_peered":"2024-06-29T15:59:13.432928+0000","last_unstale":"2024-06-29T15:59:14.447043+0000","last_undegraded":"2024-06-29T15:59:14.447043+0000","last_fullsized":"2024-06-29T15:59:14.447043+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:54.031742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447222+0000","last_change":"2024-06-29T15:59:13.444722+0000","last_active":"2024-06-29T15:59:14.447222+0000","last_peered":"2024-06-29T15:59:14.447222+0000","last_clean":"2024-06-29T15:59:14.447222+0000","last_became_active":"2024-06-29T15:59:13.444368+0000","last_became_peered":"2024-06-29T15:59:13.444368+0000","last_unstale":"2024-06-29T15:59:14.447222+0000","last_undegraded":"2024-06-29T15:59:14.447222+0000","last_fullsized":"2024-06-29T15:59:14.447222+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:13.604106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"261'4","reported_seq":18,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445707+0000","last_change":"2024-06-29T15:59:12.430097+0000","last_active":"2024-06-29T15:59:14.445707+0000","last_peered":"2024-06-29T15:59:14.445707+0000","last_clean":"2024-06-29T15:59:14.445707+0000","last_became_active":"2024-06-29T15:59:12.429756+0000","last_became_peered":"2024-06-29T15:59:12.429756+0000","last_unstale":"2024-06-29T15:59:14.445707+0000","last_undegraded":"2024-06-29T15:59:14.445707+0000","last_fullsized":"2024-06-29T15:59:14.445707+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:24:33.588310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444854+0000","last_change":"2024-06-29T15:59:13.444128+0000","last_active":"2024-06-29T15:59:14.444854+0000","last_peered":"2024-06-29T15:59:14.444854+0000","last_clean":"2024-06-29T15:59:14.444854+0000","last_became_active":"2024-06-29T15:59:13.443772+0000","last_became_peered":"2024-06-29T15:59:13.443772+0000","last_unstale":"2024-06-29T15:59:14.444854+0000","last_undegraded":"2024-06-29T15:59:14.444854+0000","last_fullsized":"2024-06-29T15:59:14.444854+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:51.953825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446186+0000","last_change":"2024-06-29T15:59:12.433422+0000","last_active":"2024-06-29T15:59:14.446186+0000","last_peered":"2024-06-29T15:59:14.446186+0000","last_clean":"2024-06-29T15:59:14.446186+0000","last_became_active":"2024-06-29T15:59:12.433082+0000","last_became_peered":"2024-06-29T15:59:12.433082+0000","last_unstale":"2024-06-29T15:59:14.446186+0000","last_undegraded":"2024-06-29T15:59:14.446186+0000","last_fullsized":"2024-06-29T15:59:14.446186+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:29:16.769680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423519+0000","last_change":"2024-06-29T15:59:13.434180+0000","last_active":"2024-06-29T15:59:14.423519+0000","last_peered":"2024-06-29T15:59:14.423519+0000","last_clean":"2024-06-29T15:59:14.423519+0000","last_became_active":"2024-06-29T15:59:13.433819+0000","last_became_peered":"2024-06-29T15:59:13.433819+0000","last_unstale":"2024-06-29T15:59:14.423519+0000","last_undegraded":"2024-06-29T15:59:14.423519+0000","last_fullsized":"2024-06-29T15:59:14.423519+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:02.544403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446612+0000","last_change":"2024-06-29T15:59:12.426690+0000","last_active":"2024-06-29T15:59:14.446612+0000","last_peered":"2024-06-29T15:59:14.446612+0000","last_clean":"2024-06-29T15:59:14.446612+0000","last_became_active":"2024-06-29T15:59:12.426323+0000","last_became_peered":"2024-06-29T15:59:12.426323+0000","last_unstale":"2024-06-29T15:59:14.446612+0000","last_undegraded":"2024-06-29T15:59:14.446612+0000","last_fullsized":"2024-06-29T15:59:14.446612+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:29.016484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445285+0000","last_change":"2024-06-29T15:59:13.445359+0000","last_active":"2024-06-29T15:59:14.445285+0000","last_peered":"2024-06-29T15:59:14.445285+0000","last_clean":"2024-06-29T15:59:14.445285+0000","last_became_active":"2024-06-29T15:59:13.445038+0000","last_became_peered":"2024-06-29T15:59:13.445038+0000","last_unstale":"2024-06-29T15:59:14.445285+0000","last_undegraded":"2024-06-29T15:59:14.445285+0000","last_fullsized":"2024-06-29T15:59:14.445285+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:21.466654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446909+0000","last_change":"2024-06-29T15:59:12.425762+0000","last_active":"2024-06-29T15:59:14.446909+0000","last_peered":"2024-06-29T15:59:14.446909+0000","last_clean":"2024-06-29T15:59:14.446909+0000","last_became_active":"2024-06-29T15:59:12.425398+0000","last_became_peered":"2024-06-29T15:59:12.425398+0000","last_unstale":"2024-06-29T15:59:14.446909+0000","last_undegraded":"2024-06-29T15:59:14.446909+0000","last_fullsized":"2024-06-29T15:59:14.446909+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:39.315919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446976+0000","last_change":"2024-06-29T15:59:13.443730+0000","last_active":"2024-06-29T15:59:14.446976+0000","last_peered":"2024-06-29T15:59:14.446976+0000","last_clean":"2024-06-29T15:59:14.446976+0000","last_became_active":"2024-06-29T15:59:13.443366+0000","last_became_peered":"2024-06-29T15:59:13.443366+0000","last_unstale":"2024-06-29T15:59:14.446976+0000","last_undegraded":"2024-06-29T15:59:14.446976+0000","last_fullsized":"2024-06-29T15:59:14.446976+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:01.147927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423683+0000","last_change":"2024-06-29T15:59:12.424684+0000","last_active":"2024-06-29T15:59:14.423683+0000","last_peered":"2024-06-29T15:59:14.423683+0000","last_clean":"2024-06-29T15:59:14.423683+0000","last_became_active":"2024-06-29T15:59:12.424373+0000","last_became_peered":"2024-06-29T15:59:12.424373+0000","last_unstale":"2024-06-29T15:59:14.423683+0000","last_undegraded":"2024-06-29T15:59:14.423683+0000","last_fullsized":"2024-06-29T15:59:14.423683+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:20:20.878132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446832+0000","last_change":"2024-06-29T15:59:13.445812+0000","last_active":"2024-06-29T15:59:14.446832+0000","last_peered":"2024-06-29T15:59:14.446832+0000","last_clean":"2024-06-29T15:59:14.446832+0000","last_became_active":"2024-06-29T15:59:13.445480+0000","last_became_peered":"2024-06-29T15:59:13.445480+0000","last_unstale":"2024-06-29T15:59:14.446832+0000","last_undegraded":"2024-06-29T15:59:14.446832+0000","last_fullsized":"2024-06-29T15:59:14.446832+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:25.308244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425074+0000","last_change":"2024-06-29T15:59:12.429752+0000","last_active":"2024-06-29T15:59:14.425074+0000","last_peered":"2024-06-29T15:59:14.425074+0000","last_clean":"2024-06-29T15:59:14.425074+0000","last_became_active":"2024-06-29T15:59:12.429448+0000","last_became_peered":"2024-06-29T15:59:12.429448+0000","last_unstale":"2024-06-29T15:59:14.425074+0000","last_undegraded":"2024-06-29T15:59:14.425074+0000","last_fullsized":"2024-06-29T15:59:14.425074+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:07:53.612197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446063+0000","last_change":"2024-06-29T15:59:13.449838+0000","last_active":"2024-06-29T15:59:14.446063+0000","last_peered":"2024-06-29T15:59:14.446063+0000","last_clean":"2024-06-29T15:59:14.446063+0000","last_became_active":"2024-06-29T15:59:13.449502+0000","last_became_peered":"2024-06-29T15:59:13.449502+0000","last_unstale":"2024-06-29T15:59:14.446063+0000","last_undegraded":"2024-06-29T15:59:14.446063+0000","last_fullsized":"2024-06-29T15:59:14.446063+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:41:09.572988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423449+0000","last_change":"2024-06-29T15:59:12.434931+0000","last_active":"2024-06-29T15:59:14.423449+0000","last_peered":"2024-06-29T15:59:14.423449+0000","last_clean":"2024-06-29T15:59:14.423449+0000","last_became_active":"2024-06-29T15:59:12.434666+0000","last_became_peered":"2024-06-29T15:59:12.434666+0000","last_unstale":"2024-06-29T15:59:14.423449+0000","last_undegraded":"2024-06-29T15:59:14.423449+0000","last_fullsized":"2024-06-29T15:59:14.423449+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:12.597232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445511+0000","last_change":"2024-06-29T15:59:13.441752+0000","last_active":"2024-06-29T15:59:14.445511+0000","last_peered":"2024-06-29T15:59:14.445511+0000","last_clean":"2024-06-29T15:59:14.445511+0000","last_became_active":"2024-06-29T15:59:13.441020+0000","last_became_peered":"2024-06-29T15:59:13.441020+0000","last_unstale":"2024-06-29T15:59:14.445511+0000","last_undegraded":"2024-06-29T15:59:14.445511+0000","last_fullsized":"2024-06-29T15:59:14.445511+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:03.408653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446805+0000","last_change":"2024-06-29T15:59:12.423520+0000","last_active":"2024-06-29T15:59:14.446805+0000","last_peered":"2024-06-29T15:59:14.446805+0000","last_clean":"2024-06-29T15:59:14.446805+0000","last_became_active":"2024-06-29T15:59:12.422629+0000","last_became_peered":"2024-06-29T15:59:12.422629+0000","last_unstale":"2024-06-29T15:59:14.446805+0000","last_undegraded":"2024-06-29T15:59:14.446805+0000","last_fullsized":"2024-06-29T15:59:14.446805+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:43.555280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446583+0000","last_change":"2024-06-29T15:59:13.443197+0000","last_active":"2024-06-29T15:59:14.446583+0000","last_peered":"2024-06-29T15:59:14.446583+0000","last_clean":"2024-06-29T15:59:14.446583+0000","last_became_active":"2024-06-29T15:59:13.442608+0000","last_became_peered":"2024-06-29T15:59:13.442608+0000","last_unstale":"2024-06-29T15:59:14.446583+0000","last_undegraded":"2024-06-29T15:59:14.446583+0000","last_fullsized":"2024-06-29T15:59:14.446583+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:28.989892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445575+0000","last_change":"2024-06-29T15:59:12.419441+0000","last_active":"2024-06-29T15:59:14.445575+0000","last_peered":"2024-06-29T15:59:14.445575+0000","last_clean":"2024-06-29T15:59:14.445575+0000","last_became_active":"2024-06-29T15:59:12.418790+0000","last_became_peered":"2024-06-29T15:59:12.418790+0000","last_unstale":"2024-06-29T15:59:14.445575+0000","last_undegraded":"2024-06-29T15:59:14.445575+0000","last_fullsized":"2024-06-29T15:59:14.445575+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:58.191899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424916+0000","last_change":"2024-06-29T15:59:13.430965+0000","last_active":"2024-06-29T15:59:14.424916+0000","last_peered":"2024-06-29T15:59:14.424916+0000","last_clean":"2024-06-29T15:59:14.424916+0000","last_became_active":"2024-06-29T15:59:13.430613+0000","last_became_peered":"2024-06-29T15:59:13.430613+0000","last_unstale":"2024-06-29T15:59:14.424916+0000","last_undegraded":"2024-06-29T15:59:14.424916+0000","last_fullsized":"2024-06-29T15:59:14.424916+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:03.795372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423929+0000","last_change":"2024-06-29T15:59:12.434416+0000","last_active":"2024-06-29T15:59:14.423929+0000","last_peered":"2024-06-29T15:59:14.423929+0000","last_clean":"2024-06-29T15:59:14.423929+0000","last_became_active":"2024-06-29T15:59:12.433987+0000","last_became_peered":"2024-06-29T15:59:12.433987+0000","last_unstale":"2024-06-29T15:59:14.423929+0000","last_undegraded":"2024-06-29T15:59:14.423929+0000","last_fullsized":"2024-06-29T15:59:14.423929+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:22.796587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445820+0000","last_change":"2024-06-29T15:59:13.439223+0000","last_active":"2024-06-29T15:59:14.445820+0000","last_peered":"2024-06-29T15:59:14.445820+0000","last_clean":"2024-06-29T15:59:14.445820+0000","last_became_active":"2024-06-29T15:59:13.438880+0000","last_became_peered":"2024-06-29T15:59:13.438880+0000","last_unstale":"2024-06-29T15:59:14.445820+0000","last_undegraded":"2024-06-29T15:59:14.445820+0000","last_fullsized":"2024-06-29T15:59:14.445820+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:38.149137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"261'3","reported_seq":17,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445346+0000","last_change":"2024-06-29T15:59:12.430464+0000","last_active":"2024-06-29T15:59:14.445346+0000","last_peered":"2024-06-29T15:59:14.445346+0000","last_clean":"2024-06-29T15:59:14.445346+0000","last_became_active":"2024-06-29T15:59:12.429623+0000","last_became_peered":"2024-06-29T15:59:12.429623+0000","last_unstale":"2024-06-29T15:59:14.445346+0000","last_undegraded":"2024-06-29T15:59:14.445346+0000","last_fullsized":"2024-06-29T15:59:14.445346+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:12.338007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425157+0000","last_change":"2024-06-29T15:59:13.431492+0000","last_active":"2024-06-29T15:59:14.425157+0000","last_peered":"2024-06-29T15:59:14.425157+0000","last_clean":"2024-06-29T15:59:14.425157+0000","last_became_active":"2024-06-29T15:59:13.430766+0000","last_became_peered":"2024-06-29T15:59:13.430766+0000","last_unstale":"2024-06-29T15:59:14.425157+0000","last_undegraded":"2024-06-29T15:59:14.425157+0000","last_fullsized":"2024-06-29T15:59:14.425157+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:54:28.513438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"261'6","reported_seq":20,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445836+0000","last_change":"2024-06-29T15:59:12.428517+0000","last_active":"2024-06-29T15:59:14.445836+0000","last_peered":"2024-06-29T15:59:14.445836+0000","last_clean":"2024-06-29T15:59:14.445836+0000","last_became_active":"2024-06-29T15:59:12.428205+0000","last_became_peered":"2024-06-29T15:59:12.428205+0000","last_unstale":"2024-06-29T15:59:14.445836+0000","last_undegraded":"2024-06-29T15:59:14.445836+0000","last_fullsized":"2024-06-29T15:59:14.445836+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:31.673730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424163+0000","last_change":"2024-06-29T15:59:13.442621+0000","last_active":"2024-06-29T15:59:14.424163+0000","last_peered":"2024-06-29T15:59:14.424163+0000","last_clean":"2024-06-29T15:59:14.424163+0000","last_became_active":"2024-06-29T15:59:13.440600+0000","last_became_peered":"2024-06-29T15:59:13.440600+0000","last_unstale":"2024-06-29T15:59:14.424163+0000","last_undegraded":"2024-06-29T15:59:14.424163+0000","last_fullsized":"2024-06-29T15:59:14.424163+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:29:18.070825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424357+0000","last_change":"2024-06-29T15:59:12.432250+0000","last_active":"2024-06-29T15:59:14.424357+0000","last_peered":"2024-06-29T15:59:14.424357+0000","last_clean":"2024-06-29T15:59:14.424357+0000","last_became_active":"2024-06-29T15:59:12.431884+0000","last_became_peered":"2024-06-29T15:59:12.431884+0000","last_unstale":"2024-06-29T15:59:14.424357+0000","last_undegraded":"2024-06-29T15:59:14.424357+0000","last_fullsized":"2024-06-29T15:59:14.424357+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:54.812185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445819+0000","last_change":"2024-06-29T15:59:13.445906+0000","last_active":"2024-06-29T15:59:14.445819+0000","last_peered":"2024-06-29T15:59:14.445819+0000","last_clean":"2024-06-29T15:59:14.445819+0000","last_became_active":"2024-06-29T15:59:13.445559+0000","last_became_peered":"2024-06-29T15:59:13.445559+0000","last_unstale":"2024-06-29T15:59:14.445819+0000","last_undegraded":"2024-06-29T15:59:14.445819+0000","last_fullsized":"2024-06-29T15:59:14.445819+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:15:48.372315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423586+0000","last_change":"2024-06-29T15:59:12.425764+0000","last_active":"2024-06-29T15:59:14.423586+0000","last_peered":"2024-06-29T15:59:14.423586+0000","last_clean":"2024-06-29T15:59:14.423586+0000","last_became_active":"2024-06-29T15:59:12.425468+0000","last_became_peered":"2024-06-29T15:59:12.425468+0000","last_unstale":"2024-06-29T15:59:14.423586+0000","last_undegraded":"2024-06-29T15:59:14.423586+0000","last_fullsized":"2024-06-29T15:59:14.423586+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:38:57.726737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445356+0000","last_change":"2024-06-29T15:59:13.440480+0000","last_active":"2024-06-29T15:59:14.445356+0000","last_peered":"2024-06-29T15:59:14.445356+0000","last_clean":"2024-06-29T15:59:14.445356+0000","last_became_active":"2024-06-29T15:59:13.440132+0000","last_became_peered":"2024-06-29T15:59:13.440132+0000","last_unstale":"2024-06-29T15:59:14.445356+0000","last_undegraded":"2024-06-29T15:59:14.445356+0000","last_fullsized":"2024-06-29T15:59:14.445356+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:54.031742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424784+0000","last_change":"2024-06-29T15:59:12.418805+0000","last_active":"2024-06-29T15:59:14.424784+0000","last_peered":"2024-06-29T15:59:14.424784+0000","last_clean":"2024-06-29T15:59:14.424784+0000","last_became_active":"2024-06-29T15:59:12.418425+0000","last_became_peered":"2024-06-29T15:59:12.418425+0000","last_unstale":"2024-06-29T15:59:14.424784+0000","last_undegraded":"2024-06-29T15:59:14.424784+0000","last_fullsized":"2024-06-29T15:59:14.424784+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:27.279490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445535+0000","last_change":"2024-06-29T15:59:13.429156+0000","last_active":"2024-06-29T15:59:14.445535+0000","last_peered":"2024-06-29T15:59:14.445535+0000","last_clean":"2024-06-29T15:59:14.445535+0000","last_became_active":"2024-06-29T15:59:13.428794+0000","last_became_peered":"2024-06-29T15:59:13.428794+0000","last_unstale":"2024-06-29T15:59:14.445535+0000","last_undegraded":"2024-06-29T15:59:14.445535+0000","last_fullsized":"2024-06-29T15:59:14.445535+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:05:58.293345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"261'8","reported_seq":22,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444424+0000","last_change":"2024-06-29T15:59:12.432460+0000","last_active":"2024-06-29T15:59:14.444424+0000","last_peered":"2024-06-29T15:59:14.444424+0000","last_clean":"2024-06-29T15:59:14.444424+0000","last_became_active":"2024-06-29T15:59:12.432034+0000","last_became_peered":"2024-06-29T15:59:12.432034+0000","last_unstale":"2024-06-29T15:59:14.444424+0000","last_undegraded":"2024-06-29T15:59:14.444424+0000","last_fullsized":"2024-06-29T15:59:14.444424+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:10.834888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424926+0000","last_change":"2024-06-29T15:59:13.441638+0000","last_active":"2024-06-29T15:59:14.424926+0000","last_peered":"2024-06-29T15:59:14.424926+0000","last_clean":"2024-06-29T15:59:14.424926+0000","last_became_active":"2024-06-29T15:59:13.441291+0000","last_became_peered":"2024-06-29T15:59:13.441291+0000","last_unstale":"2024-06-29T15:59:14.424926+0000","last_undegraded":"2024-06-29T15:59:14.424926+0000","last_fullsized":"2024-06-29T15:59:14.424926+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:36.430274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446565+0000","last_change":"2024-06-29T15:59:12.435454+0000","last_active":"2024-06-29T15:59:14.446565+0000","last_peered":"2024-06-29T15:59:14.446565+0000","last_clean":"2024-06-29T15:59:14.446565+0000","last_became_active":"2024-06-29T15:59:12.435094+0000","last_became_peered":"2024-06-29T15:59:12.435094+0000","last_unstale":"2024-06-29T15:59:14.446565+0000","last_undegraded":"2024-06-29T15:59:14.446565+0000","last_fullsized":"2024-06-29T15:59:14.446565+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:18.001944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447429+0000","last_change":"2024-06-29T15:59:13.450966+0000","last_active":"2024-06-29T15:59:14.447429+0000","last_peered":"2024-06-29T15:59:14.447429+0000","last_clean":"2024-06-29T15:59:14.447429+0000","last_became_active":"2024-06-29T15:59:13.450503+0000","last_became_peered":"2024-06-29T15:59:13.450503+0000","last_unstale":"2024-06-29T15:59:14.447429+0000","last_undegraded":"2024-06-29T15:59:14.447429+0000","last_fullsized":"2024-06-29T15:59:14.447429+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:55.819059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425427+0000","last_change":"2024-06-29T15:59:12.428269+0000","last_active":"2024-06-29T15:59:14.425427+0000","last_peered":"2024-06-29T15:59:14.425427+0000","last_clean":"2024-06-29T15:59:14.425427+0000","last_became_active":"2024-06-29T15:59:12.427965+0000","last_became_peered":"2024-06-29T15:59:12.427965+0000","last_unstale":"2024-06-29T15:59:14.425427+0000","last_undegraded":"2024-06-29T15:59:14.425427+0000","last_fullsized":"2024-06-29T15:59:14.425427+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:47.922506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425023+0000","last_change":"2024-06-29T15:59:13.442572+0000","last_active":"2024-06-29T15:59:14.425023+0000","last_peered":"2024-06-29T15:59:14.425023+0000","last_clean":"2024-06-29T15:59:14.425023+0000","last_became_active":"2024-06-29T15:59:13.442144+0000","last_became_peered":"2024-06-29T15:59:13.442144+0000","last_unstale":"2024-06-29T15:59:14.425023+0000","last_undegraded":"2024-06-29T15:59:14.425023+0000","last_fullsized":"2024-06-29T15:59:14.425023+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:44.074095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446713+0000","last_change":"2024-06-29T15:59:12.428771+0000","last_active":"2024-06-29T15:59:14.446713+0000","last_peered":"2024-06-29T15:59:14.446713+0000","last_clean":"2024-06-29T15:59:14.446713+0000","last_became_active":"2024-06-29T15:59:12.428438+0000","last_became_peered":"2024-06-29T15:59:12.428438+0000","last_unstale":"2024-06-29T15:59:14.446713+0000","last_undegraded":"2024-06-29T15:59:14.446713+0000","last_fullsized":"2024-06-29T15:59:14.446713+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:37.835105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446014+0000","last_change":"2024-06-29T15:59:13.446641+0000","last_active":"2024-06-29T15:59:14.446014+0000","last_peered":"2024-06-29T15:59:14.446014+0000","last_clean":"2024-06-29T15:59:14.446014+0000","last_became_active":"2024-06-29T15:59:13.446331+0000","last_became_peered":"2024-06-29T15:59:13.446331+0000","last_unstale":"2024-06-29T15:59:14.446014+0000","last_undegraded":"2024-06-29T15:59:14.446014+0000","last_fullsized":"2024-06-29T15:59:14.446014+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:44.145906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423689+0000","last_change":"2024-06-29T15:59:12.432453+0000","last_active":"2024-06-29T15:59:14.423689+0000","last_peered":"2024-06-29T15:59:14.423689+0000","last_clean":"2024-06-29T15:59:14.423689+0000","last_became_active":"2024-06-29T15:59:12.432117+0000","last_became_peered":"2024-06-29T15:59:12.432117+0000","last_unstale":"2024-06-29T15:59:14.423689+0000","last_undegraded":"2024-06-29T15:59:14.423689+0000","last_fullsized":"2024-06-29T15:59:14.423689+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:38.151140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447246+0000","last_change":"2024-06-29T15:59:13.448769+0000","last_active":"2024-06-29T15:59:14.447246+0000","last_peered":"2024-06-29T15:59:14.447246+0000","last_clean":"2024-06-29T15:59:14.447246+0000","last_became_active":"2024-06-29T15:59:13.440030+0000","last_became_peered":"2024-06-29T15:59:13.440030+0000","last_unstale":"2024-06-29T15:59:14.447246+0000","last_undegraded":"2024-06-29T15:59:14.447246+0000","last_fullsized":"2024-06-29T15:59:14.447246+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:45:28.360140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424490+0000","last_change":"2024-06-29T15:59:12.431185+0000","last_active":"2024-06-29T15:59:14.424490+0000","last_peered":"2024-06-29T15:59:14.424490+0000","last_clean":"2024-06-29T15:59:14.424490+0000","last_became_active":"2024-06-29T15:59:12.430731+0000","last_became_peered":"2024-06-29T15:59:12.430731+0000","last_unstale":"2024-06-29T15:59:14.424490+0000","last_undegraded":"2024-06-29T15:59:14.424490+0000","last_fullsized":"2024-06-29T15:59:14.424490+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:03.878528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446985+0000","last_change":"2024-06-29T15:59:13.440696+0000","last_active":"2024-06-29T15:59:14.446985+0000","last_peered":"2024-06-29T15:59:14.446985+0000","last_clean":"2024-06-29T15:59:14.446985+0000","last_became_active":"2024-06-29T15:59:13.440115+0000","last_became_peered":"2024-06-29T15:59:13.440115+0000","last_unstale":"2024-06-29T15:59:14.446985+0000","last_undegraded":"2024-06-29T15:59:14.446985+0000","last_fullsized":"2024-06-29T15:59:14.446985+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:13.486031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446027+0000","last_change":"2024-06-29T15:59:12.422447+0000","last_active":"2024-06-29T15:59:14.446027+0000","last_peered":"2024-06-29T15:59:14.446027+0000","last_clean":"2024-06-29T15:59:14.446027+0000","last_became_active":"2024-06-29T15:59:12.422069+0000","last_became_peered":"2024-06-29T15:59:12.422069+0000","last_unstale":"2024-06-29T15:59:14.446027+0000","last_undegraded":"2024-06-29T15:59:14.446027+0000","last_fullsized":"2024-06-29T15:59:14.446027+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:35.205755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445740+0000","last_change":"2024-06-29T15:59:13.439675+0000","last_active":"2024-06-29T15:59:14.445740+0000","last_peered":"2024-06-29T15:59:14.445740+0000","last_clean":"2024-06-29T15:59:14.445740+0000","last_became_active":"2024-06-29T15:59:13.439283+0000","last_became_peered":"2024-06-29T15:59:13.439283+0000","last_unstale":"2024-06-29T15:59:14.445740+0000","last_undegraded":"2024-06-29T15:59:14.445740+0000","last_fullsized":"2024-06-29T15:59:14.445740+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:28.989892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447472+0000","last_change":"2024-06-29T15:59:12.431843+0000","last_active":"2024-06-29T15:59:14.447472+0000","last_peered":"2024-06-29T15:59:14.447472+0000","last_clean":"2024-06-29T15:59:14.447472+0000","last_became_active":"2024-06-29T15:59:12.431515+0000","last_became_peered":"2024-06-29T15:59:12.431515+0000","last_unstale":"2024-06-29T15:59:14.447472+0000","last_undegraded":"2024-06-29T15:59:14.447472+0000","last_fullsized":"2024-06-29T15:59:14.447472+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:20.413656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425451+0000","last_change":"2024-06-29T15:59:13.453749+0000","last_active":"2024-06-29T15:59:14.425451+0000","last_peered":"2024-06-29T15:59:14.425451+0000","last_clean":"2024-06-29T15:59:14.425451+0000","last_became_active":"2024-06-29T15:59:13.453411+0000","last_became_peered":"2024-06-29T15:59:13.453411+0000","last_unstale":"2024-06-29T15:59:14.425451+0000","last_undegraded":"2024-06-29T15:59:14.425451+0000","last_fullsized":"2024-06-29T15:59:14.425451+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:20.511200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423277+0000","last_change":"2024-06-29T15:59:12.433349+0000","last_active":"2024-06-29T15:59:14.423277+0000","last_peered":"2024-06-29T15:59:14.423277+0000","last_clean":"2024-06-29T15:59:14.423277+0000","last_became_active":"2024-06-29T15:59:12.433012+0000","last_became_peered":"2024-06-29T15:59:12.433012+0000","last_unstale":"2024-06-29T15:59:14.423277+0000","last_undegraded":"2024-06-29T15:59:14.423277+0000","last_fullsized":"2024-06-29T15:59:14.423277+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:45:27.353266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446746+0000","last_change":"2024-06-29T15:59:13.442901+0000","last_active":"2024-06-29T15:59:14.446746+0000","last_peered":"2024-06-29T15:59:14.446746+0000","last_clean":"2024-06-29T15:59:14.446746+0000","last_became_active":"2024-06-29T15:59:13.442190+0000","last_became_peered":"2024-06-29T15:59:13.442190+0000","last_unstale":"2024-06-29T15:59:14.446746+0000","last_undegraded":"2024-06-29T15:59:14.446746+0000","last_fullsized":"2024-06-29T15:59:14.446746+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:27.217163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425029+0000","last_change":"2024-06-29T15:59:12.432352+0000","last_active":"2024-06-29T15:59:14.425029+0000","last_peered":"2024-06-29T15:59:14.425029+0000","last_clean":"2024-06-29T15:59:14.425029+0000","last_became_active":"2024-06-29T15:59:12.432010+0000","last_became_peered":"2024-06-29T15:59:12.432010+0000","last_unstale":"2024-06-29T15:59:14.425029+0000","last_undegraded":"2024-06-29T15:59:14.425029+0000","last_fullsized":"2024-06-29T15:59:14.425029+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:08:01.817435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444965+0000","last_change":"2024-06-29T15:59:13.441950+0000","last_active":"2024-06-29T15:59:14.444965+0000","last_peered":"2024-06-29T15:59:14.444965+0000","last_clean":"2024-06-29T15:59:14.444965+0000","last_became_active":"2024-06-29T15:59:13.441383+0000","last_became_peered":"2024-06-29T15:59:13.441383+0000","last_unstale":"2024-06-29T15:59:14.444965+0000","last_undegraded":"2024-06-29T15:59:14.444965+0000","last_fullsized":"2024-06-29T15:59:14.444965+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:38.149137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"262'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424058+0000","last_change":"2024-06-29T15:59:12.426468+0000","last_active":"2024-06-29T15:59:14.424058+0000","last_peered":"2024-06-29T15:59:14.424058+0000","last_clean":"2024-06-29T15:59:14.424058+0000","last_became_active":"2024-06-29T15:59:12.426154+0000","last_became_peered":"2024-06-29T15:59:12.426154+0000","last_unstale":"2024-06-29T15:59:14.424058+0000","last_undegraded":"2024-06-29T15:59:14.424058+0000","last_fullsized":"2024-06-29T15:59:14.424058+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:29:41.390283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445626+0000","last_change":"2024-06-29T15:59:13.438549+0000","last_active":"2024-06-29T15:59:14.445626+0000","last_peered":"2024-06-29T15:59:14.445626+0000","last_clean":"2024-06-29T15:59:14.445626+0000","last_became_active":"2024-06-29T15:59:13.436240+0000","last_became_peered":"2024-06-29T15:59:13.436240+0000","last_unstale":"2024-06-29T15:59:14.445626+0000","last_undegraded":"2024-06-29T15:59:14.445626+0000","last_fullsized":"2024-06-29T15:59:14.445626+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:31.776557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446810+0000","last_change":"2024-06-29T15:59:12.427504+0000","last_active":"2024-06-29T15:59:14.446810+0000","last_peered":"2024-06-29T15:59:14.446810+0000","last_clean":"2024-06-29T15:59:14.446810+0000","last_became_active":"2024-06-29T15:59:12.427176+0000","last_became_peered":"2024-06-29T15:59:12.427176+0000","last_unstale":"2024-06-29T15:59:14.446810+0000","last_undegraded":"2024-06-29T15:59:14.446810+0000","last_fullsized":"2024-06-29T15:59:14.446810+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:57.940152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445234+0000","last_change":"2024-06-29T15:59:13.452546+0000","last_active":"2024-06-29T15:59:14.445234+0000","last_peered":"2024-06-29T15:59:14.445234+0000","last_clean":"2024-06-29T15:59:14.445234+0000","last_became_active":"2024-06-29T15:59:13.452195+0000","last_became_peered":"2024-06-29T15:59:13.452195+0000","last_unstale":"2024-06-29T15:59:14.445234+0000","last_undegraded":"2024-06-29T15:59:14.445234+0000","last_fullsized":"2024-06-29T15:59:14.445234+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:39:56.354268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424205+0000","last_change":"2024-06-29T15:59:12.429915+0000","last_active":"2024-06-29T15:59:14.424205+0000","last_peered":"2024-06-29T15:59:14.424205+0000","last_clean":"2024-06-29T15:59:14.424205+0000","last_became_active":"2024-06-29T15:59:12.429549+0000","last_became_peered":"2024-06-29T15:59:12.429549+0000","last_unstale":"2024-06-29T15:59:14.424205+0000","last_undegraded":"2024-06-29T15:59:14.424205+0000","last_fullsized":"2024-06-29T15:59:14.424205+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:29.873597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445703+0000","last_change":"2024-06-29T15:59:13.454876+0000","last_active":"2024-06-29T15:59:14.445703+0000","last_peered":"2024-06-29T15:59:14.445703+0000","last_clean":"2024-06-29T15:59:14.445703+0000","last_became_active":"2024-06-29T15:59:13.454527+0000","last_became_peered":"2024-06-29T15:59:13.454527+0000","last_unstale":"2024-06-29T15:59:14.445703+0000","last_undegraded":"2024-06-29T15:59:14.445703+0000","last_fullsized":"2024-06-29T15:59:14.445703+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:33.002806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424617+0000","last_change":"2024-06-29T15:59:12.431710+0000","last_active":"2024-06-29T15:59:14.424617+0000","last_peered":"2024-06-29T15:59:14.424617+0000","last_clean":"2024-06-29T15:59:14.424617+0000","last_became_active":"2024-06-29T15:59:12.431161+0000","last_became_peered":"2024-06-29T15:59:12.431161+0000","last_unstale":"2024-06-29T15:59:14.424617+0000","last_undegraded":"2024-06-29T15:59:14.424617+0000","last_fullsized":"2024-06-29T15:59:14.424617+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:09.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445958+0000","last_change":"2024-06-29T15:59:13.440886+0000","last_active":"2024-06-29T15:59:14.445958+0000","last_peered":"2024-06-29T15:59:14.445958+0000","last_clean":"2024-06-29T15:59:14.445958+0000","last_became_active":"2024-06-29T15:59:13.440093+0000","last_became_peered":"2024-06-29T15:59:13.440093+0000","last_unstale":"2024-06-29T15:59:14.445958+0000","last_undegraded":"2024-06-29T15:59:14.445958+0000","last_fullsized":"2024-06-29T15:59:14.445958+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:52.601535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446136+0000","last_change":"2024-06-29T15:59:12.421669+0000","last_active":"2024-06-29T15:59:14.446136+0000","last_peered":"2024-06-29T15:59:14.446136+0000","last_clean":"2024-06-29T15:59:14.446136+0000","last_became_active":"2024-06-29T15:59:12.421352+0000","last_became_peered":"2024-06-29T15:59:12.421352+0000","last_unstale":"2024-06-29T15:59:14.446136+0000","last_undegraded":"2024-06-29T15:59:14.446136+0000","last_fullsized":"2024-06-29T15:59:14.446136+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:25:34.895975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425494+0000","last_change":"2024-06-29T15:59:13.438245+0000","last_active":"2024-06-29T15:59:14.425494+0000","last_peered":"2024-06-29T15:59:14.425494+0000","last_clean":"2024-06-29T15:59:14.425494+0000","last_became_active":"2024-06-29T15:59:13.437939+0000","last_became_peered":"2024-06-29T15:59:13.437939+0000","last_unstale":"2024-06-29T15:59:14.425494+0000","last_undegraded":"2024-06-29T15:59:14.425494+0000","last_fullsized":"2024-06-29T15:59:14.425494+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:47:17.963488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445054+0000","last_change":"2024-06-29T15:59:12.421008+0000","last_active":"2024-06-29T15:59:14.445054+0000","last_peered":"2024-06-29T15:59:14.445054+0000","last_clean":"2024-06-29T15:59:14.445054+0000","last_became_active":"2024-06-29T15:59:12.420647+0000","last_became_peered":"2024-06-29T15:59:12.420647+0000","last_unstale":"2024-06-29T15:59:14.445054+0000","last_undegraded":"2024-06-29T15:59:14.445054+0000","last_fullsized":"2024-06-29T15:59:14.445054+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:14.862006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447322+0000","last_change":"2024-06-29T15:59:13.445068+0000","last_active":"2024-06-29T15:59:14.447322+0000","last_peered":"2024-06-29T15:59:14.447322+0000","last_clean":"2024-06-29T15:59:14.447322+0000","last_became_active":"2024-06-29T15:59:13.444753+0000","last_became_peered":"2024-06-29T15:59:13.444753+0000","last_unstale":"2024-06-29T15:59:14.447322+0000","last_undegraded":"2024-06-29T15:59:14.447322+0000","last_fullsized":"2024-06-29T15:59:14.447322+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:40:42.567648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423401+0000","last_change":"2024-06-29T15:59:12.429568+0000","last_active":"2024-06-29T15:59:14.423401+0000","last_peered":"2024-06-29T15:59:14.423401+0000","last_clean":"2024-06-29T15:59:14.423401+0000","last_became_active":"2024-06-29T15:59:12.429235+0000","last_became_peered":"2024-06-29T15:59:12.429235+0000","last_unstale":"2024-06-29T15:59:14.423401+0000","last_undegraded":"2024-06-29T15:59:14.423401+0000","last_fullsized":"2024-06-29T15:59:14.423401+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:35.569722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424920+0000","last_change":"2024-06-29T15:59:13.438450+0000","last_active":"2024-06-29T15:59:14.424920+0000","last_peered":"2024-06-29T15:59:14.424920+0000","last_clean":"2024-06-29T15:59:14.424920+0000","last_became_active":"2024-06-29T15:59:13.438058+0000","last_became_peered":"2024-06-29T15:59:13.438058+0000","last_unstale":"2024-06-29T15:59:14.424920+0000","last_undegraded":"2024-06-29T15:59:14.424920+0000","last_fullsized":"2024-06-29T15:59:14.424920+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:47:55.240583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"248'259","reported_seq":1271,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444427+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:59:14.444427+0000","last_peered":"2024-06-29T15:59:14.444427+0000","last_clean":"2024-06-29T15:59:14.444427+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:59:14.444427+0000","last_undegraded":"2024-06-29T15:59:14.444427+0000","last_fullsized":"2024-06-29T15:59:14.444427+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1668,"num_read_kb":4792,"num_write":421,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1668,"num_read_kb":4792,"num_write":421,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":259,"ondisk_log_size":259,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543308,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53064,"kb_used_data":3828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318776,"statfs":{"total":96636764160,"available":96582426624,"internally_reserved":0,"allocated":3919872,"data_stored":2309732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543309,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48300,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323540,"statfs":{"total":96636764160,"available":96587304960,"internally_reserved":0,"allocated":3305472,"data_stored":1717100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543308,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70516,"kb_used_data":3236,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301324,"statfs":{"total":96636764160,"available":96564555776,"internally_reserved":0,"allocated":3313664,"data_stored":1717156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543307,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52476,"kb_used_data":3244,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319364,"statfs":{"total":96636764160,"available":96583028736,"internally_reserved":0,"allocated":3321856,"data_stored":1718802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510605,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52468,"kb_used_data":3236,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319372,"statfs":{"total":96636764160,"available":96583036928,"internally_reserved":0,"allocated":3313664,"data_stored":1717192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576013,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52460,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319380,"statfs":{"total":96636764160,"available":96583045120,"internally_reserved":0,"allocated":3305472,"data_stored":1717100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510604,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52468,"kb_used_data":3232,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319372,"statfs":{"total":96636764160,"available":96583036928,"internally_reserved":0,"allocated":3309568,"data_stored":1717190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510606,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65528,"kb_used_data":3816,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306312,"statfs":{"total":96636764160,"available":96569663488,"internally_reserved":0,"allocated":3907584,"data_stored":2308084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:59:17.535 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:59:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.688+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/4158707451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80b2fa0 msgr2=0x7fa0b80d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.688+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/4158707451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80b2fa0 0x7fa0b80d1520 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80cf310 tx=0x7fa0a002a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.688+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/4158707451 shutdown_connections 2024-06-29T15:59:17.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.688+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/4158707451 >> v2:172.21.15.153:3300/0 conn(0x7fa0b80d1a60 0x7fa0b80b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.688+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/4158707451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80b2fa0 0x7fa0b80d1520 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.688+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/4158707451 >> 172.21.15.59:0/4158707451 conn(0x7fa0b8073e10 msgr2=0x7fa0b8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:17.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.688+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/4158707451 shutdown_connections 2024-06-29T15:59:17.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.688+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/4158707451 wait complete. 2024-06-29T15:59:17.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.690+0000 7fa0bd6a4640 1 Processor -- start 2024-06-29T15:59:17.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.690+0000 7fa0bd6a4640 1 -- start start 2024-06-29T15:59:17.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.691+0000 7fa0bd6a4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b80b2270 0x7fa0b8144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.691+0000 7fa0bd6a4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0b80b2fa0 0x7fa0b8146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.691+0000 7fa0bd6a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80d1a60 0x7fa0b8146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.691+0000 7fa0bd6a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80b6400 con 0x7fa0b80d1a60 2024-06-29T15:59:17.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.691+0000 7fa0bd6a4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80b6280 con 0x7fa0b80b2270 2024-06-29T15:59:17.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.691+0000 7fa0bd6a4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b80b6580 con 0x7fa0b80b2fa0 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80d1a60 0x7fa0b8146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80d1a60 0x7fa0b8146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43470/0 (socket says 172.21.15.59:43470) 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 -- 172.21.15.59:0/3181426689 learned_addr learned my addr 172.21.15.59:0/3181426689 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b67fc640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0b80b2fa0 0x7fa0b8146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b6ffd640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b80b2270 0x7fa0b8144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 -- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0b80b2fa0 msgr2=0x7fa0b8146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0b80b2fa0 0x7fa0b8146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 -- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b80b2270 msgr2=0x7fa0b8144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b80b2270 0x7fa0b8144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 -- 172.21.15.59:0/3181426689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b81453b0 con 0x7fa0b80d1a60 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b6ffd640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b80b2270 0x7fa0b8144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0b77fe640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80d1a60 0x7fa0b8146990 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800d660 tx=0x7fa0a800db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0af7fe640 1 -- 172.21.15.59:0/3181426689 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8017070 con 0x7fa0b80d1a60 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0af7fe640 1 -- 172.21.15.59:0/3181426689 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0a8005310 con 0x7fa0b80d1a60 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b8145690 con 0x7fa0b80d1a60 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0af7fe640 1 -- 172.21.15.59:0/3181426689 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8009820 con 0x7fa0b80d1a60 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.692+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b8145c50 con 0x7fa0b80d1a60 2024-06-29T15:59:17.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.693+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b8145120 con 0x7fa0b80d1a60 2024-06-29T15:59:17.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.694+0000 7fa0af7fe640 1 -- 172.21.15.59:0/3181426689 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fa0a80099c0 con 0x7fa0b80d1a60 2024-06-29T15:59:17.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.695+0000 7fa0af7fe640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa098076540 0x7fa098078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.695+0000 7fa0af7fe640 1 -- 172.21.15.59:0/3181426689 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7fa0a8097f50 con 0x7fa0b80d1a60 2024-06-29T15:59:17.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.695+0000 7fa0b6ffd640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa098076540 0x7fa098078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.696+0000 7fa0b6ffd640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa098076540 0x7fa098078a50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001d10 tx=0x7fa0b0009040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:17.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.699+0000 7fa0af7fe640 1 -- 172.21.15.59:0/3181426689 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa0a8061170 con 0x7fa0b80d1a60 2024-06-29T15:59:17.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.886+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 --> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0b80b6aa0 con 0x7fa098076540 2024-06-29T15:59:17.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.891+0000 7fa0af7fe640 1 -- 172.21.15.59:0/3181426689 <== mgr.11797 v2:172.21.15.153:6832/3574471609 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252494 (secure 0 0 0) 0x7fa0b80b6aa0 con 0x7fa098076540 2024-06-29T15:59:17.894 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:17.895 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:59:17.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa098076540 msgr2=0x7fa098078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa098076540 0x7fa098078a50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001d10 tx=0x7fa0b0009040 comp rx=0 tx=0).stop 2024-06-29T15:59:17.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80d1a60 msgr2=0x7fa0b8146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80d1a60 0x7fa0b8146990 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800d660 tx=0x7fa0a800db20 comp rx=0 tx=0).stop 2024-06-29T15:59:17.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 shutdown_connections 2024-06-29T15:59:17.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa098076540 0x7fa098078a50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b80d1a60 0x7fa0b8146990 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa0b80b2fa0 0x7fa0b8146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 --2- 172.21.15.59:0/3181426689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b80b2270 0x7fa0b8144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 >> 172.21.15.59:0/3181426689 conn(0x7fa0b8073e10 msgr2=0x7fa0b815b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:17.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 shutdown_connections 2024-06-29T15:59:17.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:17.894+0000 7fa0bd6a4640 1 -- 172.21.15.59:0/3181426689 wait complete. 2024-06-29T15:59:17.907 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-06-29T15:59:17.038978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1668,"num_read_kb":4792,"num_write":466,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":298,"ondisk_log_size":298,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":447280,"kb_used_data":27048,"kb_used_omap":12,"kb_used_meta":420083,"kb_avail":754527440,"statfs":{"total":773094113280,"available":772636098560,"internally_reserved":0,"allocated":27697152,"data_stored":14922356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":430165554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.633931"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424021+0000","last_change":"2024-06-29T15:59:13.439581+0000","last_active":"2024-06-29T15:59:14.424021+0000","last_peered":"2024-06-29T15:59:14.424021+0000","last_clean":"2024-06-29T15:59:14.424021+0000","last_became_active":"2024-06-29T15:59:13.439283+0000","last_became_peered":"2024-06-29T15:59:13.439283+0000","last_unstale":"2024-06-29T15:59:14.424021+0000","last_undegraded":"2024-06-29T15:59:14.424021+0000","last_fullsized":"2024-06-29T15:59:14.424021+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:02.699526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424016+0000","last_change":"2024-06-29T15:59:13.430811+0000","last_active":"2024-06-29T15:59:14.424016+0000","last_peered":"2024-06-29T15:59:14.424016+0000","last_clean":"2024-06-29T15:59:14.424016+0000","last_became_active":"2024-06-29T15:59:13.429734+0000","last_became_peered":"2024-06-29T15:59:13.429734+0000","last_unstale":"2024-06-29T15:59:14.424016+0000","last_undegraded":"2024-06-29T15:59:14.424016+0000","last_fullsized":"2024-06-29T15:59:14.424016+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:24.631972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445791+0000","last_change":"2024-06-29T15:59:13.451333+0000","last_active":"2024-06-29T15:59:14.445791+0000","last_peered":"2024-06-29T15:59:14.445791+0000","last_clean":"2024-06-29T15:59:14.445791+0000","last_became_active":"2024-06-29T15:59:13.451007+0000","last_became_peered":"2024-06-29T15:59:13.451007+0000","last_unstale":"2024-06-29T15:59:14.445791+0000","last_undegraded":"2024-06-29T15:59:14.445791+0000","last_fullsized":"2024-06-29T15:59:14.445791+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:40.892308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446813+0000","last_change":"2024-06-29T15:59:13.440536+0000","last_active":"2024-06-29T15:59:14.446813+0000","last_peered":"2024-06-29T15:59:14.446813+0000","last_clean":"2024-06-29T15:59:14.446813+0000","last_became_active":"2024-06-29T15:59:13.440220+0000","last_became_peered":"2024-06-29T15:59:13.440220+0000","last_unstale":"2024-06-29T15:59:14.446813+0000","last_undegraded":"2024-06-29T15:59:14.446813+0000","last_fullsized":"2024-06-29T15:59:14.446813+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:21.792076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425066+0000","last_change":"2024-06-29T15:59:13.443124+0000","last_active":"2024-06-29T15:59:14.425066+0000","last_peered":"2024-06-29T15:59:14.425066+0000","last_clean":"2024-06-29T15:59:14.425066+0000","last_became_active":"2024-06-29T15:59:13.442775+0000","last_became_peered":"2024-06-29T15:59:13.442775+0000","last_unstale":"2024-06-29T15:59:14.425066+0000","last_undegraded":"2024-06-29T15:59:14.425066+0000","last_fullsized":"2024-06-29T15:59:14.425066+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:52.400423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425003+0000","last_change":"2024-06-29T15:59:13.455594+0000","last_active":"2024-06-29T15:59:14.425003+0000","last_peered":"2024-06-29T15:59:14.425003+0000","last_clean":"2024-06-29T15:59:14.425003+0000","last_became_active":"2024-06-29T15:59:13.454176+0000","last_became_peered":"2024-06-29T15:59:13.454176+0000","last_unstale":"2024-06-29T15:59:14.425003+0000","last_undegraded":"2024-06-29T15:59:14.425003+0000","last_fullsized":"2024-06-29T15:59:14.425003+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:36.307861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444718+0000","last_change":"2024-06-29T15:59:13.439926+0000","last_active":"2024-06-29T15:59:14.444718+0000","last_peered":"2024-06-29T15:59:14.444718+0000","last_clean":"2024-06-29T15:59:14.444718+0000","last_became_active":"2024-06-29T15:59:13.439588+0000","last_became_peered":"2024-06-29T15:59:13.439588+0000","last_unstale":"2024-06-29T15:59:14.444718+0000","last_undegraded":"2024-06-29T15:59:14.444718+0000","last_fullsized":"2024-06-29T15:59:14.444718+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:23:19.549486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446404+0000","last_change":"2024-06-29T15:59:13.428209+0000","last_active":"2024-06-29T15:59:14.446404+0000","last_peered":"2024-06-29T15:59:14.446404+0000","last_clean":"2024-06-29T15:59:14.446404+0000","last_became_active":"2024-06-29T15:59:13.427874+0000","last_became_peered":"2024-06-29T15:59:13.427874+0000","last_unstale":"2024-06-29T15:59:14.446404+0000","last_undegraded":"2024-06-29T15:59:14.446404+0000","last_fullsized":"2024-06-29T15:59:14.446404+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:05:58.293345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445637+0000","last_change":"2024-06-29T15:59:13.445452+0000","last_active":"2024-06-29T15:59:14.445637+0000","last_peered":"2024-06-29T15:59:14.445637+0000","last_clean":"2024-06-29T15:59:14.445637+0000","last_became_active":"2024-06-29T15:59:13.445076+0000","last_became_peered":"2024-06-29T15:59:13.445076+0000","last_unstale":"2024-06-29T15:59:14.445637+0000","last_undegraded":"2024-06-29T15:59:14.445637+0000","last_fullsized":"2024-06-29T15:59:14.445637+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:41:38.149677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445308+0000","last_change":"2024-06-29T15:59:13.439287+0000","last_active":"2024-06-29T15:59:14.445308+0000","last_peered":"2024-06-29T15:59:14.445308+0000","last_clean":"2024-06-29T15:59:14.445308+0000","last_became_active":"2024-06-29T15:59:13.438730+0000","last_became_peered":"2024-06-29T15:59:13.438730+0000","last_unstale":"2024-06-29T15:59:14.445308+0000","last_undegraded":"2024-06-29T15:59:14.445308+0000","last_fullsized":"2024-06-29T15:59:14.445308+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:31:10.673096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446558+0000","last_change":"2024-06-29T15:59:13.437106+0000","last_active":"2024-06-29T15:59:14.446558+0000","last_peered":"2024-06-29T15:59:14.446558+0000","last_clean":"2024-06-29T15:59:14.446558+0000","last_became_active":"2024-06-29T15:59:13.436750+0000","last_became_peered":"2024-06-29T15:59:13.436750+0000","last_unstale":"2024-06-29T15:59:14.446558+0000","last_undegraded":"2024-06-29T15:59:14.446558+0000","last_fullsized":"2024-06-29T15:59:14.446558+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:29:51.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424010+0000","last_change":"2024-06-29T15:59:13.442944+0000","last_active":"2024-06-29T15:59:14.424010+0000","last_peered":"2024-06-29T15:59:14.424010+0000","last_clean":"2024-06-29T15:59:14.424010+0000","last_became_active":"2024-06-29T15:59:13.442097+0000","last_became_peered":"2024-06-29T15:59:13.442097+0000","last_unstale":"2024-06-29T15:59:14.424010+0000","last_undegraded":"2024-06-29T15:59:14.424010+0000","last_fullsized":"2024-06-29T15:59:14.424010+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:54:57.668260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445626+0000","last_change":"2024-06-29T15:59:13.435581+0000","last_active":"2024-06-29T15:59:14.445626+0000","last_peered":"2024-06-29T15:59:14.445626+0000","last_clean":"2024-06-29T15:59:14.445626+0000","last_became_active":"2024-06-29T15:59:13.435261+0000","last_became_peered":"2024-06-29T15:59:13.435261+0000","last_unstale":"2024-06-29T15:59:14.445626+0000","last_undegraded":"2024-06-29T15:59:14.445626+0000","last_fullsized":"2024-06-29T15:59:14.445626+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:53:30.858386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445176+0000","last_change":"2024-06-29T15:59:13.448375+0000","last_active":"2024-06-29T15:59:14.445176+0000","last_peered":"2024-06-29T15:59:14.445176+0000","last_clean":"2024-06-29T15:59:14.445176+0000","last_became_active":"2024-06-29T15:59:13.447766+0000","last_became_peered":"2024-06-29T15:59:13.447766+0000","last_unstale":"2024-06-29T15:59:14.445176+0000","last_undegraded":"2024-06-29T15:59:14.445176+0000","last_fullsized":"2024-06-29T15:59:14.445176+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:25:01.384822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445368+0000","last_change":"2024-06-29T15:59:13.433667+0000","last_active":"2024-06-29T15:59:14.445368+0000","last_peered":"2024-06-29T15:59:14.445368+0000","last_clean":"2024-06-29T15:59:14.445368+0000","last_became_active":"2024-06-29T15:59:13.433355+0000","last_became_peered":"2024-06-29T15:59:13.433355+0000","last_unstale":"2024-06-29T15:59:14.445368+0000","last_undegraded":"2024-06-29T15:59:14.445368+0000","last_fullsized":"2024-06-29T15:59:14.445368+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:27.495662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444753+0000","last_change":"2024-06-29T15:59:13.434557+0000","last_active":"2024-06-29T15:59:14.444753+0000","last_peered":"2024-06-29T15:59:14.444753+0000","last_clean":"2024-06-29T15:59:14.444753+0000","last_became_active":"2024-06-29T15:59:13.434208+0000","last_became_peered":"2024-06-29T15:59:13.434208+0000","last_unstale":"2024-06-29T15:59:14.444753+0000","last_undegraded":"2024-06-29T15:59:14.444753+0000","last_fullsized":"2024-06-29T15:59:14.444753+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:17:40.892308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445293+0000","last_change":"2024-06-29T15:59:13.442005+0000","last_active":"2024-06-29T15:59:14.445293+0000","last_peered":"2024-06-29T15:59:14.445293+0000","last_clean":"2024-06-29T15:59:14.445293+0000","last_became_active":"2024-06-29T15:59:13.441467+0000","last_became_peered":"2024-06-29T15:59:13.441467+0000","last_unstale":"2024-06-29T15:59:14.445293+0000","last_undegraded":"2024-06-29T15:59:14.445293+0000","last_fullsized":"2024-06-29T15:59:14.445293+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:44.145906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446278+0000","last_change":"2024-06-29T15:59:13.452999+0000","last_active":"2024-06-29T15:59:14.446278+0000","last_peered":"2024-06-29T15:59:14.446278+0000","last_clean":"2024-06-29T15:59:14.446278+0000","last_became_active":"2024-06-29T15:59:13.451855+0000","last_became_peered":"2024-06-29T15:59:13.451855+0000","last_unstale":"2024-06-29T15:59:14.446278+0000","last_undegraded":"2024-06-29T15:59:14.446278+0000","last_fullsized":"2024-06-29T15:59:14.446278+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:03.408653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445031+0000","last_change":"2024-06-29T15:59:13.441103+0000","last_active":"2024-06-29T15:59:14.445031+0000","last_peered":"2024-06-29T15:59:14.445031+0000","last_clean":"2024-06-29T15:59:14.445031+0000","last_became_active":"2024-06-29T15:59:13.440750+0000","last_became_peered":"2024-06-29T15:59:13.440750+0000","last_unstale":"2024-06-29T15:59:14.445031+0000","last_undegraded":"2024-06-29T15:59:14.445031+0000","last_fullsized":"2024-06-29T15:59:14.445031+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:15:48.372315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445163+0000","last_change":"2024-06-29T15:59:13.435152+0000","last_active":"2024-06-29T15:59:14.445163+0000","last_peered":"2024-06-29T15:59:14.445163+0000","last_clean":"2024-06-29T15:59:14.445163+0000","last_became_active":"2024-06-29T15:59:13.434649+0000","last_became_peered":"2024-06-29T15:59:13.434649+0000","last_unstale":"2024-06-29T15:59:14.445163+0000","last_undegraded":"2024-06-29T15:59:14.445163+0000","last_fullsized":"2024-06-29T15:59:14.445163+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:52.601535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445771+0000","last_change":"2024-06-29T15:59:13.439978+0000","last_active":"2024-06-29T15:59:14.445771+0000","last_peered":"2024-06-29T15:59:14.445771+0000","last_clean":"2024-06-29T15:59:14.445771+0000","last_became_active":"2024-06-29T15:59:13.439632+0000","last_became_peered":"2024-06-29T15:59:13.439632+0000","last_unstale":"2024-06-29T15:59:14.445771+0000","last_undegraded":"2024-06-29T15:59:14.445771+0000","last_fullsized":"2024-06-29T15:59:14.445771+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:16.101361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447543+0000","last_change":"2024-06-29T15:59:13.451368+0000","last_active":"2024-06-29T15:59:14.447543+0000","last_peered":"2024-06-29T15:59:14.447543+0000","last_clean":"2024-06-29T15:59:14.447543+0000","last_became_active":"2024-06-29T15:59:13.451076+0000","last_became_peered":"2024-06-29T15:59:13.451076+0000","last_unstale":"2024-06-29T15:59:14.447543+0000","last_undegraded":"2024-06-29T15:59:14.447543+0000","last_fullsized":"2024-06-29T15:59:14.447543+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:04.885401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444606+0000","last_change":"2024-06-29T15:59:13.451673+0000","last_active":"2024-06-29T15:59:14.444606+0000","last_peered":"2024-06-29T15:59:14.444606+0000","last_clean":"2024-06-29T15:59:14.444606+0000","last_became_active":"2024-06-29T15:59:13.451341+0000","last_became_peered":"2024-06-29T15:59:13.451341+0000","last_unstale":"2024-06-29T15:59:14.444606+0000","last_undegraded":"2024-06-29T15:59:14.444606+0000","last_fullsized":"2024-06-29T15:59:14.444606+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:49.789073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447170+0000","last_change":"2024-06-29T15:59:13.448480+0000","last_active":"2024-06-29T15:59:14.447170+0000","last_peered":"2024-06-29T15:59:14.447170+0000","last_clean":"2024-06-29T15:59:14.447170+0000","last_became_active":"2024-06-29T15:59:13.448061+0000","last_became_peered":"2024-06-29T15:59:13.448061+0000","last_unstale":"2024-06-29T15:59:14.447170+0000","last_undegraded":"2024-06-29T15:59:14.447170+0000","last_fullsized":"2024-06-29T15:59:14.447170+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:41:58.902537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424212+0000","last_change":"2024-06-29T15:59:13.441128+0000","last_active":"2024-06-29T15:59:14.424212+0000","last_peered":"2024-06-29T15:59:14.424212+0000","last_clean":"2024-06-29T15:59:14.424212+0000","last_became_active":"2024-06-29T15:59:13.440804+0000","last_became_peered":"2024-06-29T15:59:13.440804+0000","last_unstale":"2024-06-29T15:59:14.424212+0000","last_undegraded":"2024-06-29T15:59:14.424212+0000","last_fullsized":"2024-06-29T15:59:14.424212+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:01:44.110046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425386+0000","last_change":"2024-06-29T15:59:13.442744+0000","last_active":"2024-06-29T15:59:14.425386+0000","last_peered":"2024-06-29T15:59:14.425386+0000","last_clean":"2024-06-29T15:59:14.425386+0000","last_became_active":"2024-06-29T15:59:13.442242+0000","last_became_peered":"2024-06-29T15:59:13.442242+0000","last_unstale":"2024-06-29T15:59:14.425386+0000","last_undegraded":"2024-06-29T15:59:14.425386+0000","last_fullsized":"2024-06-29T15:59:14.425386+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:16.306319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425350+0000","last_change":"2024-06-29T15:59:13.432309+0000","last_active":"2024-06-29T15:59:14.425350+0000","last_peered":"2024-06-29T15:59:14.425350+0000","last_clean":"2024-06-29T15:59:14.425350+0000","last_became_active":"2024-06-29T15:59:13.431982+0000","last_became_peered":"2024-06-29T15:59:13.431982+0000","last_unstale":"2024-06-29T15:59:14.425350+0000","last_undegraded":"2024-06-29T15:59:14.425350+0000","last_fullsized":"2024-06-29T15:59:14.425350+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:22.491582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445804+0000","last_change":"2024-06-29T15:59:13.440493+0000","last_active":"2024-06-29T15:59:14.445804+0000","last_peered":"2024-06-29T15:59:14.445804+0000","last_clean":"2024-06-29T15:59:14.445804+0000","last_became_active":"2024-06-29T15:59:13.440062+0000","last_became_peered":"2024-06-29T15:59:13.440062+0000","last_unstale":"2024-06-29T15:59:14.445804+0000","last_undegraded":"2024-06-29T15:59:14.445804+0000","last_fullsized":"2024-06-29T15:59:14.445804+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:19.852632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423490+0000","last_change":"2024-06-29T15:59:13.441504+0000","last_active":"2024-06-29T15:59:14.423490+0000","last_peered":"2024-06-29T15:59:14.423490+0000","last_clean":"2024-06-29T15:59:14.423490+0000","last_became_active":"2024-06-29T15:59:13.441166+0000","last_became_peered":"2024-06-29T15:59:13.441166+0000","last_unstale":"2024-06-29T15:59:14.423490+0000","last_undegraded":"2024-06-29T15:59:14.423490+0000","last_fullsized":"2024-06-29T15:59:14.423490+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:12.360125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447413+0000","last_change":"2024-06-29T15:59:13.451137+0000","last_active":"2024-06-29T15:59:14.447413+0000","last_peered":"2024-06-29T15:59:14.447413+0000","last_clean":"2024-06-29T15:59:14.447413+0000","last_became_active":"2024-06-29T15:59:13.450774+0000","last_became_peered":"2024-06-29T15:59:13.450774+0000","last_unstale":"2024-06-29T15:59:14.447413+0000","last_undegraded":"2024-06-29T15:59:14.447413+0000","last_fullsized":"2024-06-29T15:59:14.447413+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:36.576595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447043+0000","last_change":"2024-06-29T15:59:13.433269+0000","last_active":"2024-06-29T15:59:14.447043+0000","last_peered":"2024-06-29T15:59:14.447043+0000","last_clean":"2024-06-29T15:59:14.447043+0000","last_became_active":"2024-06-29T15:59:13.432928+0000","last_became_peered":"2024-06-29T15:59:13.432928+0000","last_unstale":"2024-06-29T15:59:14.447043+0000","last_undegraded":"2024-06-29T15:59:14.447043+0000","last_fullsized":"2024-06-29T15:59:14.447043+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:54.031742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447222+0000","last_change":"2024-06-29T15:59:13.444722+0000","last_active":"2024-06-29T15:59:14.447222+0000","last_peered":"2024-06-29T15:59:14.447222+0000","last_clean":"2024-06-29T15:59:14.447222+0000","last_became_active":"2024-06-29T15:59:13.444368+0000","last_became_peered":"2024-06-29T15:59:13.444368+0000","last_unstale":"2024-06-29T15:59:14.447222+0000","last_undegraded":"2024-06-29T15:59:14.447222+0000","last_fullsized":"2024-06-29T15:59:14.447222+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:13.604106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"261'4","reported_seq":18,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445707+0000","last_change":"2024-06-29T15:59:12.430097+0000","last_active":"2024-06-29T15:59:14.445707+0000","last_peered":"2024-06-29T15:59:14.445707+0000","last_clean":"2024-06-29T15:59:14.445707+0000","last_became_active":"2024-06-29T15:59:12.429756+0000","last_became_peered":"2024-06-29T15:59:12.429756+0000","last_unstale":"2024-06-29T15:59:14.445707+0000","last_undegraded":"2024-06-29T15:59:14.445707+0000","last_fullsized":"2024-06-29T15:59:14.445707+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:24:33.588310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444854+0000","last_change":"2024-06-29T15:59:13.444128+0000","last_active":"2024-06-29T15:59:14.444854+0000","last_peered":"2024-06-29T15:59:14.444854+0000","last_clean":"2024-06-29T15:59:14.444854+0000","last_became_active":"2024-06-29T15:59:13.443772+0000","last_became_peered":"2024-06-29T15:59:13.443772+0000","last_unstale":"2024-06-29T15:59:14.444854+0000","last_undegraded":"2024-06-29T15:59:14.444854+0000","last_fullsized":"2024-06-29T15:59:14.444854+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:51.953825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446186+0000","last_change":"2024-06-29T15:59:12.433422+0000","last_active":"2024-06-29T15:59:14.446186+0000","last_peered":"2024-06-29T15:59:14.446186+0000","last_clean":"2024-06-29T15:59:14.446186+0000","last_became_active":"2024-06-29T15:59:12.433082+0000","last_became_peered":"2024-06-29T15:59:12.433082+0000","last_unstale":"2024-06-29T15:59:14.446186+0000","last_undegraded":"2024-06-29T15:59:14.446186+0000","last_fullsized":"2024-06-29T15:59:14.446186+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:29:16.769680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423519+0000","last_change":"2024-06-29T15:59:13.434180+0000","last_active":"2024-06-29T15:59:14.423519+0000","last_peered":"2024-06-29T15:59:14.423519+0000","last_clean":"2024-06-29T15:59:14.423519+0000","last_became_active":"2024-06-29T15:59:13.433819+0000","last_became_peered":"2024-06-29T15:59:13.433819+0000","last_unstale":"2024-06-29T15:59:14.423519+0000","last_undegraded":"2024-06-29T15:59:14.423519+0000","last_fullsized":"2024-06-29T15:59:14.423519+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:02.544403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446612+0000","last_change":"2024-06-29T15:59:12.426690+0000","last_active":"2024-06-29T15:59:14.446612+0000","last_peered":"2024-06-29T15:59:14.446612+0000","last_clean":"2024-06-29T15:59:14.446612+0000","last_became_active":"2024-06-29T15:59:12.426323+0000","last_became_peered":"2024-06-29T15:59:12.426323+0000","last_unstale":"2024-06-29T15:59:14.446612+0000","last_undegraded":"2024-06-29T15:59:14.446612+0000","last_fullsized":"2024-06-29T15:59:14.446612+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:29.016484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445285+0000","last_change":"2024-06-29T15:59:13.445359+0000","last_active":"2024-06-29T15:59:14.445285+0000","last_peered":"2024-06-29T15:59:14.445285+0000","last_clean":"2024-06-29T15:59:14.445285+0000","last_became_active":"2024-06-29T15:59:13.445038+0000","last_became_peered":"2024-06-29T15:59:13.445038+0000","last_unstale":"2024-06-29T15:59:14.445285+0000","last_undegraded":"2024-06-29T15:59:14.445285+0000","last_fullsized":"2024-06-29T15:59:14.445285+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:21.466654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446909+0000","last_change":"2024-06-29T15:59:12.425762+0000","last_active":"2024-06-29T15:59:14.446909+0000","last_peered":"2024-06-29T15:59:14.446909+0000","last_clean":"2024-06-29T15:59:14.446909+0000","last_became_active":"2024-06-29T15:59:12.425398+0000","last_became_peered":"2024-06-29T15:59:12.425398+0000","last_unstale":"2024-06-29T15:59:14.446909+0000","last_undegraded":"2024-06-29T15:59:14.446909+0000","last_fullsized":"2024-06-29T15:59:14.446909+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:39.315919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446976+0000","last_change":"2024-06-29T15:59:13.443730+0000","last_active":"2024-06-29T15:59:14.446976+0000","last_peered":"2024-06-29T15:59:14.446976+0000","last_clean":"2024-06-29T15:59:14.446976+0000","last_became_active":"2024-06-29T15:59:13.443366+0000","last_became_peered":"2024-06-29T15:59:13.443366+0000","last_unstale":"2024-06-29T15:59:14.446976+0000","last_undegraded":"2024-06-29T15:59:14.446976+0000","last_fullsized":"2024-06-29T15:59:14.446976+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:01.147927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423683+0000","last_change":"2024-06-29T15:59:12.424684+0000","last_active":"2024-06-29T15:59:14.423683+0000","last_peered":"2024-06-29T15:59:14.423683+0000","last_clean":"2024-06-29T15:59:14.423683+0000","last_became_active":"2024-06-29T15:59:12.424373+0000","last_became_peered":"2024-06-29T15:59:12.424373+0000","last_unstale":"2024-06-29T15:59:14.423683+0000","last_undegraded":"2024-06-29T15:59:14.423683+0000","last_fullsized":"2024-06-29T15:59:14.423683+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:20:20.878132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446832+0000","last_change":"2024-06-29T15:59:13.445812+0000","last_active":"2024-06-29T15:59:14.446832+0000","last_peered":"2024-06-29T15:59:14.446832+0000","last_clean":"2024-06-29T15:59:14.446832+0000","last_became_active":"2024-06-29T15:59:13.445480+0000","last_became_peered":"2024-06-29T15:59:13.445480+0000","last_unstale":"2024-06-29T15:59:14.446832+0000","last_undegraded":"2024-06-29T15:59:14.446832+0000","last_fullsized":"2024-06-29T15:59:14.446832+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:25.308244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425074+0000","last_change":"2024-06-29T15:59:12.429752+0000","last_active":"2024-06-29T15:59:14.425074+0000","last_peered":"2024-06-29T15:59:14.425074+0000","last_clean":"2024-06-29T15:59:14.425074+0000","last_became_active":"2024-06-29T15:59:12.429448+0000","last_became_peered":"2024-06-29T15:59:12.429448+0000","last_unstale":"2024-06-29T15:59:14.425074+0000","last_undegraded":"2024-06-29T15:59:14.425074+0000","last_fullsized":"2024-06-29T15:59:14.425074+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:07:53.612197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446063+0000","last_change":"2024-06-29T15:59:13.449838+0000","last_active":"2024-06-29T15:59:14.446063+0000","last_peered":"2024-06-29T15:59:14.446063+0000","last_clean":"2024-06-29T15:59:14.446063+0000","last_became_active":"2024-06-29T15:59:13.449502+0000","last_became_peered":"2024-06-29T15:59:13.449502+0000","last_unstale":"2024-06-29T15:59:14.446063+0000","last_undegraded":"2024-06-29T15:59:14.446063+0000","last_fullsized":"2024-06-29T15:59:14.446063+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:41:09.572988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423449+0000","last_change":"2024-06-29T15:59:12.434931+0000","last_active":"2024-06-29T15:59:14.423449+0000","last_peered":"2024-06-29T15:59:14.423449+0000","last_clean":"2024-06-29T15:59:14.423449+0000","last_became_active":"2024-06-29T15:59:12.434666+0000","last_became_peered":"2024-06-29T15:59:12.434666+0000","last_unstale":"2024-06-29T15:59:14.423449+0000","last_undegraded":"2024-06-29T15:59:14.423449+0000","last_fullsized":"2024-06-29T15:59:14.423449+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:12.597232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445511+0000","last_change":"2024-06-29T15:59:13.441752+0000","last_active":"2024-06-29T15:59:14.445511+0000","last_peered":"2024-06-29T15:59:14.445511+0000","last_clean":"2024-06-29T15:59:14.445511+0000","last_became_active":"2024-06-29T15:59:13.441020+0000","last_became_peered":"2024-06-29T15:59:13.441020+0000","last_unstale":"2024-06-29T15:59:14.445511+0000","last_undegraded":"2024-06-29T15:59:14.445511+0000","last_fullsized":"2024-06-29T15:59:14.445511+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:03.408653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446805+0000","last_change":"2024-06-29T15:59:12.423520+0000","last_active":"2024-06-29T15:59:14.446805+0000","last_peered":"2024-06-29T15:59:14.446805+0000","last_clean":"2024-06-29T15:59:14.446805+0000","last_became_active":"2024-06-29T15:59:12.422629+0000","last_became_peered":"2024-06-29T15:59:12.422629+0000","last_unstale":"2024-06-29T15:59:14.446805+0000","last_undegraded":"2024-06-29T15:59:14.446805+0000","last_fullsized":"2024-06-29T15:59:14.446805+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:43.555280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446583+0000","last_change":"2024-06-29T15:59:13.443197+0000","last_active":"2024-06-29T15:59:14.446583+0000","last_peered":"2024-06-29T15:59:14.446583+0000","last_clean":"2024-06-29T15:59:14.446583+0000","last_became_active":"2024-06-29T15:59:13.442608+0000","last_became_peered":"2024-06-29T15:59:13.442608+0000","last_unstale":"2024-06-29T15:59:14.446583+0000","last_undegraded":"2024-06-29T15:59:14.446583+0000","last_fullsized":"2024-06-29T15:59:14.446583+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:28.989892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445575+0000","last_change":"2024-06-29T15:59:12.419441+0000","last_active":"2024-06-29T15:59:14.445575+0000","last_peered":"2024-06-29T15:59:14.445575+0000","last_clean":"2024-06-29T15:59:14.445575+0000","last_became_active":"2024-06-29T15:59:12.418790+0000","last_became_peered":"2024-06-29T15:59:12.418790+0000","last_unstale":"2024-06-29T15:59:14.445575+0000","last_undegraded":"2024-06-29T15:59:14.445575+0000","last_fullsized":"2024-06-29T15:59:14.445575+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:58.191899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424916+0000","last_change":"2024-06-29T15:59:13.430965+0000","last_active":"2024-06-29T15:59:14.424916+0000","last_peered":"2024-06-29T15:59:14.424916+0000","last_clean":"2024-06-29T15:59:14.424916+0000","last_became_active":"2024-06-29T15:59:13.430613+0000","last_became_peered":"2024-06-29T15:59:13.430613+0000","last_unstale":"2024-06-29T15:59:14.424916+0000","last_undegraded":"2024-06-29T15:59:14.424916+0000","last_fullsized":"2024-06-29T15:59:14.424916+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:03.795372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423929+0000","last_change":"2024-06-29T15:59:12.434416+0000","last_active":"2024-06-29T15:59:14.423929+0000","last_peered":"2024-06-29T15:59:14.423929+0000","last_clean":"2024-06-29T15:59:14.423929+0000","last_became_active":"2024-06-29T15:59:12.433987+0000","last_became_peered":"2024-06-29T15:59:12.433987+0000","last_unstale":"2024-06-29T15:59:14.423929+0000","last_undegraded":"2024-06-29T15:59:14.423929+0000","last_fullsized":"2024-06-29T15:59:14.423929+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:22.796587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445820+0000","last_change":"2024-06-29T15:59:13.439223+0000","last_active":"2024-06-29T15:59:14.445820+0000","last_peered":"2024-06-29T15:59:14.445820+0000","last_clean":"2024-06-29T15:59:14.445820+0000","last_became_active":"2024-06-29T15:59:13.438880+0000","last_became_peered":"2024-06-29T15:59:13.438880+0000","last_unstale":"2024-06-29T15:59:14.445820+0000","last_undegraded":"2024-06-29T15:59:14.445820+0000","last_fullsized":"2024-06-29T15:59:14.445820+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:38.149137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"261'3","reported_seq":17,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445346+0000","last_change":"2024-06-29T15:59:12.430464+0000","last_active":"2024-06-29T15:59:14.445346+0000","last_peered":"2024-06-29T15:59:14.445346+0000","last_clean":"2024-06-29T15:59:14.445346+0000","last_became_active":"2024-06-29T15:59:12.429623+0000","last_became_peered":"2024-06-29T15:59:12.429623+0000","last_unstale":"2024-06-29T15:59:14.445346+0000","last_undegraded":"2024-06-29T15:59:14.445346+0000","last_fullsized":"2024-06-29T15:59:14.445346+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:12.338007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425157+0000","last_change":"2024-06-29T15:59:13.431492+0000","last_active":"2024-06-29T15:59:14.425157+0000","last_peered":"2024-06-29T15:59:14.425157+0000","last_clean":"2024-06-29T15:59:14.425157+0000","last_became_active":"2024-06-29T15:59:13.430766+0000","last_became_peered":"2024-06-29T15:59:13.430766+0000","last_unstale":"2024-06-29T15:59:14.425157+0000","last_undegraded":"2024-06-29T15:59:14.425157+0000","last_fullsized":"2024-06-29T15:59:14.425157+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:54:28.513438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"261'6","reported_seq":20,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445836+0000","last_change":"2024-06-29T15:59:12.428517+0000","last_active":"2024-06-29T15:59:14.445836+0000","last_peered":"2024-06-29T15:59:14.445836+0000","last_clean":"2024-06-29T15:59:14.445836+0000","last_became_active":"2024-06-29T15:59:12.428205+0000","last_became_peered":"2024-06-29T15:59:12.428205+0000","last_unstale":"2024-06-29T15:59:14.445836+0000","last_undegraded":"2024-06-29T15:59:14.445836+0000","last_fullsized":"2024-06-29T15:59:14.445836+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:31.673730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424163+0000","last_change":"2024-06-29T15:59:13.442621+0000","last_active":"2024-06-29T15:59:14.424163+0000","last_peered":"2024-06-29T15:59:14.424163+0000","last_clean":"2024-06-29T15:59:14.424163+0000","last_became_active":"2024-06-29T15:59:13.440600+0000","last_became_peered":"2024-06-29T15:59:13.440600+0000","last_unstale":"2024-06-29T15:59:14.424163+0000","last_undegraded":"2024-06-29T15:59:14.424163+0000","last_fullsized":"2024-06-29T15:59:14.424163+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:29:18.070825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424357+0000","last_change":"2024-06-29T15:59:12.432250+0000","last_active":"2024-06-29T15:59:14.424357+0000","last_peered":"2024-06-29T15:59:14.424357+0000","last_clean":"2024-06-29T15:59:14.424357+0000","last_became_active":"2024-06-29T15:59:12.431884+0000","last_became_peered":"2024-06-29T15:59:12.431884+0000","last_unstale":"2024-06-29T15:59:14.424357+0000","last_undegraded":"2024-06-29T15:59:14.424357+0000","last_fullsized":"2024-06-29T15:59:14.424357+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:54.812185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445819+0000","last_change":"2024-06-29T15:59:13.445906+0000","last_active":"2024-06-29T15:59:14.445819+0000","last_peered":"2024-06-29T15:59:14.445819+0000","last_clean":"2024-06-29T15:59:14.445819+0000","last_became_active":"2024-06-29T15:59:13.445559+0000","last_became_peered":"2024-06-29T15:59:13.445559+0000","last_unstale":"2024-06-29T15:59:14.445819+0000","last_undegraded":"2024-06-29T15:59:14.445819+0000","last_fullsized":"2024-06-29T15:59:14.445819+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:15:48.372315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423586+0000","last_change":"2024-06-29T15:59:12.425764+0000","last_active":"2024-06-29T15:59:14.423586+0000","last_peered":"2024-06-29T15:59:14.423586+0000","last_clean":"2024-06-29T15:59:14.423586+0000","last_became_active":"2024-06-29T15:59:12.425468+0000","last_became_peered":"2024-06-29T15:59:12.425468+0000","last_unstale":"2024-06-29T15:59:14.423586+0000","last_undegraded":"2024-06-29T15:59:14.423586+0000","last_fullsized":"2024-06-29T15:59:14.423586+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:38:57.726737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445356+0000","last_change":"2024-06-29T15:59:13.440480+0000","last_active":"2024-06-29T15:59:14.445356+0000","last_peered":"2024-06-29T15:59:14.445356+0000","last_clean":"2024-06-29T15:59:14.445356+0000","last_became_active":"2024-06-29T15:59:13.440132+0000","last_became_peered":"2024-06-29T15:59:13.440132+0000","last_unstale":"2024-06-29T15:59:14.445356+0000","last_undegraded":"2024-06-29T15:59:14.445356+0000","last_fullsized":"2024-06-29T15:59:14.445356+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:54.031742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424784+0000","last_change":"2024-06-29T15:59:12.418805+0000","last_active":"2024-06-29T15:59:14.424784+0000","last_peered":"2024-06-29T15:59:14.424784+0000","last_clean":"2024-06-29T15:59:14.424784+0000","last_became_active":"2024-06-29T15:59:12.418425+0000","last_became_peered":"2024-06-29T15:59:12.418425+0000","last_unstale":"2024-06-29T15:59:14.424784+0000","last_undegraded":"2024-06-29T15:59:14.424784+0000","last_fullsized":"2024-06-29T15:59:14.424784+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:27.279490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445535+0000","last_change":"2024-06-29T15:59:13.429156+0000","last_active":"2024-06-29T15:59:14.445535+0000","last_peered":"2024-06-29T15:59:14.445535+0000","last_clean":"2024-06-29T15:59:14.445535+0000","last_became_active":"2024-06-29T15:59:13.428794+0000","last_became_peered":"2024-06-29T15:59:13.428794+0000","last_unstale":"2024-06-29T15:59:14.445535+0000","last_undegraded":"2024-06-29T15:59:14.445535+0000","last_fullsized":"2024-06-29T15:59:14.445535+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:05:58.293345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"261'8","reported_seq":22,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444424+0000","last_change":"2024-06-29T15:59:12.432460+0000","last_active":"2024-06-29T15:59:14.444424+0000","last_peered":"2024-06-29T15:59:14.444424+0000","last_clean":"2024-06-29T15:59:14.444424+0000","last_became_active":"2024-06-29T15:59:12.432034+0000","last_became_peered":"2024-06-29T15:59:12.432034+0000","last_unstale":"2024-06-29T15:59:14.444424+0000","last_undegraded":"2024-06-29T15:59:14.444424+0000","last_fullsized":"2024-06-29T15:59:14.444424+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:10.834888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424926+0000","last_change":"2024-06-29T15:59:13.441638+0000","last_active":"2024-06-29T15:59:14.424926+0000","last_peered":"2024-06-29T15:59:14.424926+0000","last_clean":"2024-06-29T15:59:14.424926+0000","last_became_active":"2024-06-29T15:59:13.441291+0000","last_became_peered":"2024-06-29T15:59:13.441291+0000","last_unstale":"2024-06-29T15:59:14.424926+0000","last_undegraded":"2024-06-29T15:59:14.424926+0000","last_fullsized":"2024-06-29T15:59:14.424926+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:36.430274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446565+0000","last_change":"2024-06-29T15:59:12.435454+0000","last_active":"2024-06-29T15:59:14.446565+0000","last_peered":"2024-06-29T15:59:14.446565+0000","last_clean":"2024-06-29T15:59:14.446565+0000","last_became_active":"2024-06-29T15:59:12.435094+0000","last_became_peered":"2024-06-29T15:59:12.435094+0000","last_unstale":"2024-06-29T15:59:14.446565+0000","last_undegraded":"2024-06-29T15:59:14.446565+0000","last_fullsized":"2024-06-29T15:59:14.446565+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:18.001944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447429+0000","last_change":"2024-06-29T15:59:13.450966+0000","last_active":"2024-06-29T15:59:14.447429+0000","last_peered":"2024-06-29T15:59:14.447429+0000","last_clean":"2024-06-29T15:59:14.447429+0000","last_became_active":"2024-06-29T15:59:13.450503+0000","last_became_peered":"2024-06-29T15:59:13.450503+0000","last_unstale":"2024-06-29T15:59:14.447429+0000","last_undegraded":"2024-06-29T15:59:14.447429+0000","last_fullsized":"2024-06-29T15:59:14.447429+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:55.819059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425427+0000","last_change":"2024-06-29T15:59:12.428269+0000","last_active":"2024-06-29T15:59:14.425427+0000","last_peered":"2024-06-29T15:59:14.425427+0000","last_clean":"2024-06-29T15:59:14.425427+0000","last_became_active":"2024-06-29T15:59:12.427965+0000","last_became_peered":"2024-06-29T15:59:12.427965+0000","last_unstale":"2024-06-29T15:59:14.425427+0000","last_undegraded":"2024-06-29T15:59:14.425427+0000","last_fullsized":"2024-06-29T15:59:14.425427+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:47.922506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425023+0000","last_change":"2024-06-29T15:59:13.442572+0000","last_active":"2024-06-29T15:59:14.425023+0000","last_peered":"2024-06-29T15:59:14.425023+0000","last_clean":"2024-06-29T15:59:14.425023+0000","last_became_active":"2024-06-29T15:59:13.442144+0000","last_became_peered":"2024-06-29T15:59:13.442144+0000","last_unstale":"2024-06-29T15:59:14.425023+0000","last_undegraded":"2024-06-29T15:59:14.425023+0000","last_fullsized":"2024-06-29T15:59:14.425023+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:44.074095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"261'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446713+0000","last_change":"2024-06-29T15:59:12.428771+0000","last_active":"2024-06-29T15:59:14.446713+0000","last_peered":"2024-06-29T15:59:14.446713+0000","last_clean":"2024-06-29T15:59:14.446713+0000","last_became_active":"2024-06-29T15:59:12.428438+0000","last_became_peered":"2024-06-29T15:59:12.428438+0000","last_unstale":"2024-06-29T15:59:14.446713+0000","last_undegraded":"2024-06-29T15:59:14.446713+0000","last_fullsized":"2024-06-29T15:59:14.446713+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:37.835105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446014+0000","last_change":"2024-06-29T15:59:13.446641+0000","last_active":"2024-06-29T15:59:14.446014+0000","last_peered":"2024-06-29T15:59:14.446014+0000","last_clean":"2024-06-29T15:59:14.446014+0000","last_became_active":"2024-06-29T15:59:13.446331+0000","last_became_peered":"2024-06-29T15:59:13.446331+0000","last_unstale":"2024-06-29T15:59:14.446014+0000","last_undegraded":"2024-06-29T15:59:14.446014+0000","last_fullsized":"2024-06-29T15:59:14.446014+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:44.145906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423689+0000","last_change":"2024-06-29T15:59:12.432453+0000","last_active":"2024-06-29T15:59:14.423689+0000","last_peered":"2024-06-29T15:59:14.423689+0000","last_clean":"2024-06-29T15:59:14.423689+0000","last_became_active":"2024-06-29T15:59:12.432117+0000","last_became_peered":"2024-06-29T15:59:12.432117+0000","last_unstale":"2024-06-29T15:59:14.423689+0000","last_undegraded":"2024-06-29T15:59:14.423689+0000","last_fullsized":"2024-06-29T15:59:14.423689+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:38.151140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447246+0000","last_change":"2024-06-29T15:59:13.448769+0000","last_active":"2024-06-29T15:59:14.447246+0000","last_peered":"2024-06-29T15:59:14.447246+0000","last_clean":"2024-06-29T15:59:14.447246+0000","last_became_active":"2024-06-29T15:59:13.440030+0000","last_became_peered":"2024-06-29T15:59:13.440030+0000","last_unstale":"2024-06-29T15:59:14.447246+0000","last_undegraded":"2024-06-29T15:59:14.447246+0000","last_fullsized":"2024-06-29T15:59:14.447246+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:45:28.360140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424490+0000","last_change":"2024-06-29T15:59:12.431185+0000","last_active":"2024-06-29T15:59:14.424490+0000","last_peered":"2024-06-29T15:59:14.424490+0000","last_clean":"2024-06-29T15:59:14.424490+0000","last_became_active":"2024-06-29T15:59:12.430731+0000","last_became_peered":"2024-06-29T15:59:12.430731+0000","last_unstale":"2024-06-29T15:59:14.424490+0000","last_undegraded":"2024-06-29T15:59:14.424490+0000","last_fullsized":"2024-06-29T15:59:14.424490+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:03.878528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446985+0000","last_change":"2024-06-29T15:59:13.440696+0000","last_active":"2024-06-29T15:59:14.446985+0000","last_peered":"2024-06-29T15:59:14.446985+0000","last_clean":"2024-06-29T15:59:14.446985+0000","last_became_active":"2024-06-29T15:59:13.440115+0000","last_became_peered":"2024-06-29T15:59:13.440115+0000","last_unstale":"2024-06-29T15:59:14.446985+0000","last_undegraded":"2024-06-29T15:59:14.446985+0000","last_fullsized":"2024-06-29T15:59:14.446985+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:13.486031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446027+0000","last_change":"2024-06-29T15:59:12.422447+0000","last_active":"2024-06-29T15:59:14.446027+0000","last_peered":"2024-06-29T15:59:14.446027+0000","last_clean":"2024-06-29T15:59:14.446027+0000","last_became_active":"2024-06-29T15:59:12.422069+0000","last_became_peered":"2024-06-29T15:59:12.422069+0000","last_unstale":"2024-06-29T15:59:14.446027+0000","last_undegraded":"2024-06-29T15:59:14.446027+0000","last_fullsized":"2024-06-29T15:59:14.446027+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:35.205755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445740+0000","last_change":"2024-06-29T15:59:13.439675+0000","last_active":"2024-06-29T15:59:14.445740+0000","last_peered":"2024-06-29T15:59:14.445740+0000","last_clean":"2024-06-29T15:59:14.445740+0000","last_became_active":"2024-06-29T15:59:13.439283+0000","last_became_peered":"2024-06-29T15:59:13.439283+0000","last_unstale":"2024-06-29T15:59:14.445740+0000","last_undegraded":"2024-06-29T15:59:14.445740+0000","last_fullsized":"2024-06-29T15:59:14.445740+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:28.989892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447472+0000","last_change":"2024-06-29T15:59:12.431843+0000","last_active":"2024-06-29T15:59:14.447472+0000","last_peered":"2024-06-29T15:59:14.447472+0000","last_clean":"2024-06-29T15:59:14.447472+0000","last_became_active":"2024-06-29T15:59:12.431515+0000","last_became_peered":"2024-06-29T15:59:12.431515+0000","last_unstale":"2024-06-29T15:59:14.447472+0000","last_undegraded":"2024-06-29T15:59:14.447472+0000","last_fullsized":"2024-06-29T15:59:14.447472+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:41:20.413656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425451+0000","last_change":"2024-06-29T15:59:13.453749+0000","last_active":"2024-06-29T15:59:14.425451+0000","last_peered":"2024-06-29T15:59:14.425451+0000","last_clean":"2024-06-29T15:59:14.425451+0000","last_became_active":"2024-06-29T15:59:13.453411+0000","last_became_peered":"2024-06-29T15:59:13.453411+0000","last_unstale":"2024-06-29T15:59:14.425451+0000","last_undegraded":"2024-06-29T15:59:14.425451+0000","last_fullsized":"2024-06-29T15:59:14.425451+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:20.511200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423277+0000","last_change":"2024-06-29T15:59:12.433349+0000","last_active":"2024-06-29T15:59:14.423277+0000","last_peered":"2024-06-29T15:59:14.423277+0000","last_clean":"2024-06-29T15:59:14.423277+0000","last_became_active":"2024-06-29T15:59:12.433012+0000","last_became_peered":"2024-06-29T15:59:12.433012+0000","last_unstale":"2024-06-29T15:59:14.423277+0000","last_undegraded":"2024-06-29T15:59:14.423277+0000","last_fullsized":"2024-06-29T15:59:14.423277+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:45:27.353266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446746+0000","last_change":"2024-06-29T15:59:13.442901+0000","last_active":"2024-06-29T15:59:14.446746+0000","last_peered":"2024-06-29T15:59:14.446746+0000","last_clean":"2024-06-29T15:59:14.446746+0000","last_became_active":"2024-06-29T15:59:13.442190+0000","last_became_peered":"2024-06-29T15:59:13.442190+0000","last_unstale":"2024-06-29T15:59:14.446746+0000","last_undegraded":"2024-06-29T15:59:14.446746+0000","last_fullsized":"2024-06-29T15:59:14.446746+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:27.217163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"261'2","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425029+0000","last_change":"2024-06-29T15:59:12.432352+0000","last_active":"2024-06-29T15:59:14.425029+0000","last_peered":"2024-06-29T15:59:14.425029+0000","last_clean":"2024-06-29T15:59:14.425029+0000","last_became_active":"2024-06-29T15:59:12.432010+0000","last_became_peered":"2024-06-29T15:59:12.432010+0000","last_unstale":"2024-06-29T15:59:14.425029+0000","last_undegraded":"2024-06-29T15:59:14.425029+0000","last_fullsized":"2024-06-29T15:59:14.425029+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:08:01.817435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444965+0000","last_change":"2024-06-29T15:59:13.441950+0000","last_active":"2024-06-29T15:59:14.444965+0000","last_peered":"2024-06-29T15:59:14.444965+0000","last_clean":"2024-06-29T15:59:14.444965+0000","last_became_active":"2024-06-29T15:59:13.441383+0000","last_became_peered":"2024-06-29T15:59:13.441383+0000","last_unstale":"2024-06-29T15:59:14.444965+0000","last_undegraded":"2024-06-29T15:59:14.444965+0000","last_fullsized":"2024-06-29T15:59:14.444965+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:38.149137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"262'1","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424058+0000","last_change":"2024-06-29T15:59:12.426468+0000","last_active":"2024-06-29T15:59:14.424058+0000","last_peered":"2024-06-29T15:59:14.424058+0000","last_clean":"2024-06-29T15:59:14.424058+0000","last_became_active":"2024-06-29T15:59:12.426154+0000","last_became_peered":"2024-06-29T15:59:12.426154+0000","last_unstale":"2024-06-29T15:59:14.424058+0000","last_undegraded":"2024-06-29T15:59:14.424058+0000","last_fullsized":"2024-06-29T15:59:14.424058+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:29:41.390283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445626+0000","last_change":"2024-06-29T15:59:13.438549+0000","last_active":"2024-06-29T15:59:14.445626+0000","last_peered":"2024-06-29T15:59:14.445626+0000","last_clean":"2024-06-29T15:59:14.445626+0000","last_became_active":"2024-06-29T15:59:13.436240+0000","last_became_peered":"2024-06-29T15:59:13.436240+0000","last_unstale":"2024-06-29T15:59:14.445626+0000","last_undegraded":"2024-06-29T15:59:14.445626+0000","last_fullsized":"2024-06-29T15:59:14.445626+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:31.776557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446810+0000","last_change":"2024-06-29T15:59:12.427504+0000","last_active":"2024-06-29T15:59:14.446810+0000","last_peered":"2024-06-29T15:59:14.446810+0000","last_clean":"2024-06-29T15:59:14.446810+0000","last_became_active":"2024-06-29T15:59:12.427176+0000","last_became_peered":"2024-06-29T15:59:12.427176+0000","last_unstale":"2024-06-29T15:59:14.446810+0000","last_undegraded":"2024-06-29T15:59:14.446810+0000","last_fullsized":"2024-06-29T15:59:14.446810+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:57.940152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445234+0000","last_change":"2024-06-29T15:59:13.452546+0000","last_active":"2024-06-29T15:59:14.445234+0000","last_peered":"2024-06-29T15:59:14.445234+0000","last_clean":"2024-06-29T15:59:14.445234+0000","last_became_active":"2024-06-29T15:59:13.452195+0000","last_became_peered":"2024-06-29T15:59:13.452195+0000","last_unstale":"2024-06-29T15:59:14.445234+0000","last_undegraded":"2024-06-29T15:59:14.445234+0000","last_fullsized":"2024-06-29T15:59:14.445234+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:39:56.354268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424205+0000","last_change":"2024-06-29T15:59:12.429915+0000","last_active":"2024-06-29T15:59:14.424205+0000","last_peered":"2024-06-29T15:59:14.424205+0000","last_clean":"2024-06-29T15:59:14.424205+0000","last_became_active":"2024-06-29T15:59:12.429549+0000","last_became_peered":"2024-06-29T15:59:12.429549+0000","last_unstale":"2024-06-29T15:59:14.424205+0000","last_undegraded":"2024-06-29T15:59:14.424205+0000","last_fullsized":"2024-06-29T15:59:14.424205+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:29.873597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445703+0000","last_change":"2024-06-29T15:59:13.454876+0000","last_active":"2024-06-29T15:59:14.445703+0000","last_peered":"2024-06-29T15:59:14.445703+0000","last_clean":"2024-06-29T15:59:14.445703+0000","last_became_active":"2024-06-29T15:59:13.454527+0000","last_became_peered":"2024-06-29T15:59:13.454527+0000","last_unstale":"2024-06-29T15:59:14.445703+0000","last_undegraded":"2024-06-29T15:59:14.445703+0000","last_fullsized":"2024-06-29T15:59:14.445703+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:33.002806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424617+0000","last_change":"2024-06-29T15:59:12.431710+0000","last_active":"2024-06-29T15:59:14.424617+0000","last_peered":"2024-06-29T15:59:14.424617+0000","last_clean":"2024-06-29T15:59:14.424617+0000","last_became_active":"2024-06-29T15:59:12.431161+0000","last_became_peered":"2024-06-29T15:59:12.431161+0000","last_unstale":"2024-06-29T15:59:14.424617+0000","last_undegraded":"2024-06-29T15:59:14.424617+0000","last_fullsized":"2024-06-29T15:59:14.424617+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:09.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445958+0000","last_change":"2024-06-29T15:59:13.440886+0000","last_active":"2024-06-29T15:59:14.445958+0000","last_peered":"2024-06-29T15:59:14.445958+0000","last_clean":"2024-06-29T15:59:14.445958+0000","last_became_active":"2024-06-29T15:59:13.440093+0000","last_became_peered":"2024-06-29T15:59:13.440093+0000","last_unstale":"2024-06-29T15:59:14.445958+0000","last_undegraded":"2024-06-29T15:59:14.445958+0000","last_fullsized":"2024-06-29T15:59:14.445958+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:52.601535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.446136+0000","last_change":"2024-06-29T15:59:12.421669+0000","last_active":"2024-06-29T15:59:14.446136+0000","last_peered":"2024-06-29T15:59:14.446136+0000","last_clean":"2024-06-29T15:59:14.446136+0000","last_became_active":"2024-06-29T15:59:12.421352+0000","last_became_peered":"2024-06-29T15:59:12.421352+0000","last_unstale":"2024-06-29T15:59:14.446136+0000","last_undegraded":"2024-06-29T15:59:14.446136+0000","last_fullsized":"2024-06-29T15:59:14.446136+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:25:34.895975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.425494+0000","last_change":"2024-06-29T15:59:13.438245+0000","last_active":"2024-06-29T15:59:14.425494+0000","last_peered":"2024-06-29T15:59:14.425494+0000","last_clean":"2024-06-29T15:59:14.425494+0000","last_became_active":"2024-06-29T15:59:13.437939+0000","last_became_peered":"2024-06-29T15:59:13.437939+0000","last_unstale":"2024-06-29T15:59:14.425494+0000","last_undegraded":"2024-06-29T15:59:14.425494+0000","last_fullsized":"2024-06-29T15:59:14.425494+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:47:17.963488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.445054+0000","last_change":"2024-06-29T15:59:12.421008+0000","last_active":"2024-06-29T15:59:14.445054+0000","last_peered":"2024-06-29T15:59:14.445054+0000","last_clean":"2024-06-29T15:59:14.445054+0000","last_became_active":"2024-06-29T15:59:12.420647+0000","last_became_peered":"2024-06-29T15:59:12.420647+0000","last_unstale":"2024-06-29T15:59:14.445054+0000","last_undegraded":"2024-06-29T15:59:14.445054+0000","last_fullsized":"2024-06-29T15:59:14.445054+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:21:14.862006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.447322+0000","last_change":"2024-06-29T15:59:13.445068+0000","last_active":"2024-06-29T15:59:14.447322+0000","last_peered":"2024-06-29T15:59:14.447322+0000","last_clean":"2024-06-29T15:59:14.447322+0000","last_became_active":"2024-06-29T15:59:13.444753+0000","last_became_peered":"2024-06-29T15:59:13.444753+0000","last_unstale":"2024-06-29T15:59:14.447322+0000","last_undegraded":"2024-06-29T15:59:14.447322+0000","last_fullsized":"2024-06-29T15:59:14.447322+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:40:42.567648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.423401+0000","last_change":"2024-06-29T15:59:12.429568+0000","last_active":"2024-06-29T15:59:14.423401+0000","last_peered":"2024-06-29T15:59:14.423401+0000","last_clean":"2024-06-29T15:59:14.423401+0000","last_became_active":"2024-06-29T15:59:12.429235+0000","last_became_peered":"2024-06-29T15:59:12.429235+0000","last_unstale":"2024-06-29T15:59:14.423401+0000","last_undegraded":"2024-06-29T15:59:14.423401+0000","last_fullsized":"2024-06-29T15:59:14.423401+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:11.397245+0000","last_clean_scrub_stamp":"2024-06-29T15:59:11.397245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:35.569722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.424920+0000","last_change":"2024-06-29T15:59:13.438450+0000","last_active":"2024-06-29T15:59:14.424920+0000","last_peered":"2024-06-29T15:59:14.424920+0000","last_clean":"2024-06-29T15:59:14.424920+0000","last_became_active":"2024-06-29T15:59:13.438058+0000","last_became_peered":"2024-06-29T15:59:13.438058+0000","last_unstale":"2024-06-29T15:59:14.424920+0000","last_undegraded":"2024-06-29T15:59:14.424920+0000","last_fullsized":"2024-06-29T15:59:14.424920+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:12.404118+0000","last_clean_scrub_stamp":"2024-06-29T15:59:12.404118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:47:55.240583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"248'259","reported_seq":1271,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-06-29T15:59:14.444427+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:59:14.444427+0000","last_peered":"2024-06-29T15:59:14.444427+0000","last_clean":"2024-06-29T15:59:14.444427+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:59:14.444427+0000","last_undegraded":"2024-06-29T15:59:14.444427+0000","last_fullsized":"2024-06-29T15:59:14.444427+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1668,"num_read_kb":4792,"num_write":421,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1668,"num_read_kb":4792,"num_write":421,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":259,"ondisk_log_size":259,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543308,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53064,"kb_used_data":3828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318776,"statfs":{"total":96636764160,"available":96582426624,"internally_reserved":0,"allocated":3919872,"data_stored":2309732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543309,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48300,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323540,"statfs":{"total":96636764160,"available":96587304960,"internally_reserved":0,"allocated":3305472,"data_stored":1717100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543308,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70516,"kb_used_data":3236,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301324,"statfs":{"total":96636764160,"available":96564555776,"internally_reserved":0,"allocated":3313664,"data_stored":1717156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543307,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52476,"kb_used_data":3244,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319364,"statfs":{"total":96636764160,"available":96583028736,"internally_reserved":0,"allocated":3321856,"data_stored":1718802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510605,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52468,"kb_used_data":3236,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319372,"statfs":{"total":96636764160,"available":96583036928,"internally_reserved":0,"allocated":3313664,"data_stored":1717192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576013,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52460,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319380,"statfs":{"total":96636764160,"available":96583045120,"internally_reserved":0,"allocated":3305472,"data_stored":1717100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510604,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52468,"kb_used_data":3232,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319372,"statfs":{"total":96636764160,"available":96583036928,"internally_reserved":0,"allocated":3309568,"data_stored":1717190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510606,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65528,"kb_used_data":3816,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306312,"statfs":{"total":96636764160,"available":96569663488,"internally_reserved":0,"allocated":3907584,"data_stored":2308084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:59:17.911 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:59:17.911 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:59:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.074+0000 7f531585b640 1 -- 172.21.15.59:0/319443249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 msgr2=0x7f52f8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.074+0000 7f531585b640 1 --2- 172.21.15.59:0/319443249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 0x7f52f8003e10 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f530000c7b0 tx=0x7f52f8005830 comp rx=0 tx=0).stop 2024-06-29T15:59:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.074+0000 7f530f7fe640 1 -- 172.21.15.59:0/319443249 reap_dead start 2024-06-29T15:59:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.074+0000 7f531585b640 1 -- 172.21.15.59:0/319443249 shutdown_connections 2024-06-29T15:59:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.074+0000 7f531585b640 1 -- 172.21.15.59:0/319443249 >> 172.21.15.59:0/319443249 conn(0x7f53100d40c0 msgr2=0x7f53100b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.074+0000 7f531585b640 1 -- 172.21.15.59:0/319443249 shutdown_connections 2024-06-29T15:59:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.074+0000 7f531585b640 1 -- 172.21.15.59:0/319443249 wait complete. 2024-06-29T15:59:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.076+0000 7f531585b640 1 Processor -- start 2024-06-29T15:59:18.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.076+0000 7f531585b640 1 -- start start 2024-06-29T15:59:18.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f531585b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 0x7f53101f4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f531585b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f531014c820 0x7f5310148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f531585b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5310148d50 0x7f5310149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f531585b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f53100d26a0 con 0x7f52f80039e0 2024-06-29T15:59:18.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f531585b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53100b4790 con 0x7f531014c820 2024-06-29T15:59:18.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f531585b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f53100d2820 con 0x7f5310148d50 2024-06-29T15:59:18.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f530effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 0x7f53101f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f530effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 0x7f53101f4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43478/0 (socket says 172.21.15.59:43478) 2024-06-29T15:59:18.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f530effd640 1 -- 172.21.15.59:0/1084905985 learned_addr learned my addr 172.21.15.59:0/1084905985 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:18.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f530e7fc640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f531014c820 0x7f5310148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.077+0000 7f530f7fe640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5310148d50 0x7f5310149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f530effd640 1 -- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5310148d50 msgr2=0x7f5310149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f530effd640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5310148d50 0x7f5310149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f530effd640 1 -- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f531014c820 msgr2=0x7f5310148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f530effd640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f531014c820 0x7f5310148810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f530effd640 1 -- 172.21.15.59:0/1084905985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53101616f0 con 0x7f52f80039e0 2024-06-29T15:59:18.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f530e7fc640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f531014c820 0x7f5310148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:18.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f530effd640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 0x7f53101f4890 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f530000c8d0 tx=0x7f53000012e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:18.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f52f7fff640 1 -- 172.21.15.59:0/1084905985 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300004b40 con 0x7f52f80039e0 2024-06-29T15:59:18.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f52f7fff640 1 -- 172.21.15.59:0/1084905985 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5300004ce0 con 0x7f52f80039e0 2024-06-29T15:59:18.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5310158460 con 0x7f52f80039e0 2024-06-29T15:59:18.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f52f7fff640 1 -- 172.21.15.59:0/1084905985 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530002bc20 con 0x7f52f80039e0 2024-06-29T15:59:18.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.078+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53100cb250 con 0x7f52f80039e0 2024-06-29T15:59:18.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.079+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53100d2d80 con 0x7f52f80039e0 2024-06-29T15:59:18.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.080+0000 7f52f7fff640 1 -- 172.21.15.59:0/1084905985 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f5300006030 con 0x7f52f80039e0 2024-06-29T15:59:18.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.080+0000 7f52f7fff640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f52fc076540 0x7f52fc078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.081+0000 7f52f7fff640 1 -- 172.21.15.59:0/1084905985 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f53000b82e0 con 0x7f52f80039e0 2024-06-29T15:59:18.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.081+0000 7f530e7fc640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f52fc076540 0x7f52fc078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.081+0000 7f530e7fc640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f52fc076540 0x7f52fc078a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f531014ce90 tx=0x7f5304005f90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:18.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.085+0000 7f52f7fff640 1 -- 172.21.15.59:0/1084905985 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f53100b4e50 con 0x7f52f80039e0 2024-06-29T15:59:18.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.348+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f53100c71a0 con 0x7f52f80039e0 2024-06-29T15:59:18.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.351+0000 7f52f7fff640 1 -- 172.21.15.59:0/1084905985 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v54) ==== 190+0+147 (secure 0 0 0) 0x7f5300081690 con 0x7f52f80039e0 2024-06-29T15:59:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T15:59:18.354 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T15:59:18.355 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T15:59:18.355 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T15:59:18.355 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.353+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f52fc076540 msgr2=0x7f52fc078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.353+0000 7f531585b640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f52fc076540 0x7f52fc078a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f531014ce90 tx=0x7f5304005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.353+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 msgr2=0x7f53101f4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.353+0000 7f531585b640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 0x7f53101f4890 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f530000c8d0 tx=0x7f53000012e0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.354+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 shutdown_connections 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.354+0000 7f531585b640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f52fc076540 0x7f52fc078a50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.354+0000 7f531585b640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5310148d50 0x7f5310149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.354+0000 7f531585b640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f531014c820 0x7f5310148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.354+0000 7f531585b640 1 --2- 172.21.15.59:0/1084905985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52f80039e0 0x7f53101f4890 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.354+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 >> 172.21.15.59:0/1084905985 conn(0x7f53100d40c0 msgr2=0x7f53100b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:18.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.354+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 shutdown_connections 2024-06-29T15:59:18.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.354+0000 7f531585b640 1 -- 172.21.15.59:0/1084905985 wait complete. 2024-06-29T15:59:18.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.484+0000 7f82a790b640 1 -- 172.21.15.59:0/334657576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f828c004040 msgr2=0x7f828c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.484+0000 7f82a790b640 1 --2- 172.21.15.59:0/334657576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f828c004040 0x7f828c004800 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f829000c4a0 tx=0x7f828c005830 comp rx=0 tx=0).stop 2024-06-29T15:59:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.484+0000 7f82a5ea8640 1 -- 172.21.15.59:0/334657576 reap_dead start 2024-06-29T15:59:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.484+0000 7f82a790b640 1 -- 172.21.15.59:0/334657576 shutdown_connections 2024-06-29T15:59:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.484+0000 7f82a790b640 1 -- 172.21.15.59:0/334657576 >> 172.21.15.59:0/334657576 conn(0x7f82a00d3ee0 msgr2=0x7f82a00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.484+0000 7f82a790b640 1 -- 172.21.15.59:0/334657576 shutdown_connections 2024-06-29T15:59:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.484+0000 7f82a790b640 1 -- 172.21.15.59:0/334657576 wait complete. 2024-06-29T15:59:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.486+0000 7f82a790b640 1 Processor -- start 2024-06-29T15:59:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.486+0000 7f82a790b640 1 -- start start 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.487+0000 7f82a790b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f828c004040 0x7f82a01e6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.487+0000 7f82a790b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a0159750 0x7f82a01f8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.487+0000 7f82a790b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a01f8b60 0x7f82a01538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.487+0000 7f82a790b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f82a00b3d20 con 0x7f82a0159750 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.487+0000 7f82a790b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82a00b3ba0 con 0x7f828c004040 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.487+0000 7f82a790b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f82a015fef0 con 0x7f82a01f8b60 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a0159750 0x7f82a01f8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a0159750 0x7f82a01f8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43496/0 (socket says 172.21.15.59:43496) 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 -- 172.21.15.59:0/1584930815 learned_addr learned my addr 172.21.15.59:0/1584930815 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a5ea8640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a01f8b60 0x7f82a01538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a56a7640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f828c004040 0x7f82a01e6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 -- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a01f8b60 msgr2=0x7f82a01538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a01f8b60 0x7f82a01538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 -- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f828c004040 msgr2=0x7f82a01e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f828c004040 0x7f82a01e6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 -- 172.21.15.59:0/1584930815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a01f8f70 con 0x7f82a0159750 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a56a7640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f828c004040 0x7f82a01e6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a4ea6640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a0159750 0x7f82a01f8620 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f8290000c00 tx=0x7f8290006eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f8295ffb640 1 -- 172.21.15.59:0/1584930815 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290006570 con 0x7f82a0159750 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f8295ffb640 1 -- 172.21.15.59:0/1584930815 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f829002b670 con 0x7f82a0159750 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a015a0a0 con 0x7f82a0159750 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.488+0000 7f8295ffb640 1 -- 172.21.15.59:0/1584930815 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829002bac0 con 0x7f82a0159750 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.489+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a015a660 con 0x7f82a0159750 2024-06-29T15:59:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.490+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a0142f00 con 0x7f82a0159750 2024-06-29T15:59:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.491+0000 7f8295ffb640 1 -- 172.21.15.59:0/1584930815 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f82900035f0 con 0x7f82a0159750 2024-06-29T15:59:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.491+0000 7f8295ffb640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8284076510 0x7f8284078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.491+0000 7f8295ffb640 1 -- 172.21.15.59:0/1584930815 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f82900b7f40 con 0x7f82a0159750 2024-06-29T15:59:18.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.491+0000 7f82a56a7640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8284076510 0x7f8284078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.492+0000 7f82a56a7640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8284076510 0x7f8284078a20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82a00d2e40 tx=0x7f829c005f90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:18.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.495+0000 7f8295ffb640 1 -- 172.21.15.59:0/1584930815 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f82a01f8f70 con 0x7f82a0159750 2024-06-29T15:59:18.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.730+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82a015f9b0 con 0x7f82a0159750 2024-06-29T15:59:18.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.731+0000 7f8295ffb640 1 -- 172.21.15.59:0/1584930815 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5247 (secure 0 0 0) 0x7f82a00b42e0 con 0x7f82a0159750 2024-06-29T15:59:18.732 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:18.732 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":148,"btime":"2024-06-29T15:59:16:455546+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1010984928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1010984928},{"type":"v1","addr":"172.21.15.153:6836","nonce":1010984928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":10475,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2101409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2101409892},{"type":"v1","addr":"172.21.15.153:6837","nonce":2101409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":12130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1208243017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1208243017},{"type":"v1","addr":"172.21.15.59:6835","nonce":1208243017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:12.808119+0000","modified":"2024-06-29T15:59:15.847810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13110":{"gid":13110,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/2985893835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2985893835},{"type":"v1","addr":"172.21.15.59:6833","nonce":2985893835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13110,"qdb_cluster":[13110]},"id":17}]} 2024-06-29T15:59:18.732 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 148 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8284076510 msgr2=0x7f8284078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8284076510 0x7f8284078a20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82a00d2e40 tx=0x7f829c005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a0159750 msgr2=0x7f82a01f8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a0159750 0x7f82a01f8620 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f8290000c00 tx=0x7f8290006eb0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 shutdown_connections 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f8284076510 0x7f8284078a20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82a01f8b60 0x7f82a01538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82a0159750 0x7f82a01f8620 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.733+0000 7f82a790b640 1 --2- 172.21.15.59:0/1584930815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f828c004040 0x7f82a01e6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.734+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 >> 172.21.15.59:0/1584930815 conn(0x7f82a00d3ee0 msgr2=0x7f82a00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.734+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 shutdown_connections 2024-06-29T15:59:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:18.734+0000 7f82a790b640 1 -- 172.21.15.59:0/1584930815 wait complete. 2024-06-29T15:59:19.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 148, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:59:12.808119+0000', 'modified': '2024-06-29T15:59:15.847810+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13110}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13110': {'gid': 13110, 'name': 'a', 'rank': 0, 'incarnation': 144, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/2985893835', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 2985893835}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 2985893835}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [36], 'metadata_pool': 35, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 13110, 'qdb_cluster': [13110]} 2024-06-29T15:59:19.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:59:19.149 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T15:59:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.271+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/631286063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf9c0053a0 msgr2=0x7fcf9c00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.271+0000 7fcfb9b9e640 1 --2- 172.21.15.59:0/631286063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf9c0053a0 0x7fcf9c00ba20 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fcfac001d90 tx=0x7fcf9c005830 comp rx=0 tx=0).stop 2024-06-29T15:59:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.271+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/631286063 shutdown_connections 2024-06-29T15:59:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.271+0000 7fcfb9b9e640 1 --2- 172.21.15.59:0/631286063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf9c0053a0 0x7fcf9c00ba20 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.271+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/631286063 >> 172.21.15.59:0/631286063 conn(0x7fcfb40d45b0 msgr2=0x7fcfb40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.271+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/631286063 shutdown_connections 2024-06-29T15:59:19.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.271+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/631286063 wait complete. 2024-06-29T15:59:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.273+0000 7fcfb9b9e640 1 Processor -- start 2024-06-29T15:59:19.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.273+0000 7fcfb9b9e640 1 -- start start 2024-06-29T15:59:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb9b9e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf9c0053a0 0x7fcfb40c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb9b9e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfb414de30 0x7fcfb406d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb9b9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb414e240 0x7fcfb406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb9b9e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb4161a80 con 0x7fcfb414de30 2024-06-29T15:59:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb9b9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb40ce7c0 con 0x7fcfb414e240 2024-06-29T15:59:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb9b9e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb4161c00 con 0x7fcf9c0053a0 2024-06-29T15:59:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfb414de30 0x7fcfb406d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfb414de30 0x7fcfb406d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43512/0 (socket says 172.21.15.59:43512) 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/3117876821 learned_addr learned my addr 172.21.15.59:0/3117876821 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf9c0053a0 0x7fcfb40c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb3fff640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb414e240 0x7fcfb406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf9c0053a0 msgr2=0x7fcfb40c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf9c0053a0 0x7fcfb40c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb414e240 msgr2=0x7fcfb406d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb414e240 0x7fcfb406d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/3117876821 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb40c8dc0 con 0x7fcfb414de30 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfb414de30 0x7fcfb406d310 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8011ae0 tx=0x7fcfa8009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.275+0000 7fcf9bfff640 1 -- 172.21.15.59:0/3117876821 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa801e070 con 0x7fcfb414de30 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.276+0000 7fcf9bfff640 1 -- 172.21.15.59:0/3117876821 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfa8004480 con 0x7fcfb414de30 2024-06-29T15:59:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.276+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/3117876821 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb40c90a0 con 0x7fcfb414de30 2024-06-29T15:59:19.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.276+0000 7fcf9bfff640 1 -- 172.21.15.59:0/3117876821 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800fc60 con 0x7fcfb414de30 2024-06-29T15:59:19.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.276+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/3117876821 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb40cbe60 con 0x7fcfb414de30 2024-06-29T15:59:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.277+0000 7fcf9bfff640 1 -- 172.21.15.59:0/3117876821 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fcfa8021070 con 0x7fcfb414de30 2024-06-29T15:59:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.278+0000 7fcf9bfff640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcfa0076540 0x7fcfa0078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.278+0000 7fcf9bfff640 1 -- 172.21.15.59:0/3117876821 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7fcfa8098950 con 0x7fcfb414de30 2024-06-29T15:59:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.278+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcfa0076540 0x7fcfa0078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.279+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcfa0076540 0x7fcfa0078a50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4000f60 tx=0x7fcfa4000ed0 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.279+0000 7fcfc0a5b740 1 Processor -- start 2024-06-29T15:59:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.279+0000 7fcfc0a5b740 1 -- start start 2024-06-29T15:59:19.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.280+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8ade86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.280+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adcc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.280+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.280+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555f8ad64bc0 con 0x555f8a9d33b0 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.280+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555f8ad13630 con 0x555f8a9fc270 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555f8ad02bd0 con 0x555f8a9db590 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adcc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adcc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43522/0 (socket says 172.21.15.59:43522) 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/1768937269 learned_addr learned my addr 172.21.15.59:0/1768937269 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb3fff640 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8ade86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 msgr2=0x555f8adcc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 msgr2=0x555f8ade86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8ade86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/1768937269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f8acffca0 con 0x555f8a9d33b0 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb3fff640 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adcc3a0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fcfa805db80 tx=0x7fcfa8061490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b214ed3031fb46 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcf997fa640 1 -- 172.21.15.59:0/1768937269 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa807a020 con 0x555f8a9d33b0 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcf997fa640 1 -- 172.21.15.59:0/1768937269 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfa806a340 con 0x555f8a9d33b0 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.281+0000 7fcf997fa640 1 -- 172.21.15.59:0/1768937269 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8096d90 con 0x555f8a9d33b0 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1768937269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 msgr2=0x555f8adcc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adcc3a0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fcfa805db80 tx=0x7fcfa8061490 comp rx=0 tx=0).stop 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1768937269 shutdown_connections 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adcc3a0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1768937269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8ade86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1768937269 >> 172.21.15.59:0/1768937269 conn(0x555f8a9f03c0 msgr2=0x555f8ad9ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:19.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1768937269 shutdown_connections 2024-06-29T15:59:19.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.282+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1768937269 wait complete. 2024-06-29T15:59:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.283+0000 7fcfc0a5b740 1 -- start start 2024-06-29T15:59:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.283+0000 7fcfc0a5b740 1 Processor -- start 2024-06-29T15:59:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 0x555f8adc3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8adfa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adfaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555f8aca0ae0 con 0x555f8a9db590 2024-06-29T15:59:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555f8ad06c10 con 0x555f8a9fc270 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555f8aa76620 con 0x555f8a9d33b0 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 10 client.0 _ll_register_callbacks cb 0x555f8acad480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8adfa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8adfa580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43536/0 (socket says 172.21.15.59:43536) 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/3761247460 learned_addr learned my addr 172.21.15.59:0/3761247460 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 0x555f8adc3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb3fff640 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adfaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/3761247460 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 msgr2=0x555f8adc3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 0x555f8adc3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/3761247460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 msgr2=0x555f8adfaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adfaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/3761247460 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f8aa67850 con 0x555f8a9db590 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8adfa580 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8010010 tx=0x7fcfa8060a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9a9cc198ec07d4 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcf927fc640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcf927fc640 1 -- 172.21.15.59:0/3761247460 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8094060 con 0x555f8a9db590 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcf927fc640 1 -- 172.21.15.59:0/3761247460 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7fcfa8081360 con 0x555f8a9db590 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/3761247460 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa806a340 con 0x555f8a9db590 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcf927fc640 1 -- 172.21.15.59:0/3761247460 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=148) ==== 24+0+0 (secure 0 0 0) 0x7fcfa806f050 con 0x555f8a9db590 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 10 client.13248 fetch_fsmap learned FSMap version 148 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.285+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/3761247460 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=148,monmap=2+}) -- 0x555f8aade6b0 con 0x555f8a9db590 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcf927fc640 1 -- 172.21.15.59:0/3761247460 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfa806a340 con 0x555f8a9db590 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcf927fc640 1 -- 172.21.15.59:0/3761247460 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 148) ==== 3502+0+0 (secure 0 0 0) 0x7fcfa80962b0 con 0x555f8a9db590 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcfc0a5b740 10 client.13248 fetch_fsmap finished waiting for FSMap version 148 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcfc0a5b740 10 client.13248 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcfc0a5b740 20 client.13248 populate_metadata read hostname 'smithi059' 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] conn(0x555f8a9bae40 0x555f8adcbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcfc0a5b740 4 client.13248 mds_command: new command op to 13110 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/3761247460 --> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f8ad15580 con 0x555f8a9bae40 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] conn(0x555f8a9bae40 0x555f8adcbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcf927fc640 10 client.13248 ms_handle_connect on v2:172.21.15.59:6832/2985893835 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.286+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] conn(0x555f8a9bae40 0x555f8adcbb30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c0cf5475d47067 server_cookie=d82cf4047de04508 in_seq=0 out_seq=0 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcf927fc640 1 -- 172.21.15.59:0/3761247460 <== mds.0 v2:172.21.15.59:6832/2985893835 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555f8ad15580 con 0x555f8a9bae40 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcf927fc640 10 client.13248 handle_command_reply: tid=0 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 client.13248 shutdown 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/3761247460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 msgr2=0x555f8adfa580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8adfa580 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8010010 tx=0x7fcfa8060a20 comp rx=0 tx=0).stop 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/3761247460 shutdown_connections 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] conn(0x555f8a9bae40 0x555f8adcbb30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adfaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8adfa580 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/3761247460 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 0x555f8adc3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/3761247460 >> 172.21.15.59:0/3761247460 conn(0x555f8a9f03c0 msgr2=0x555f8adc8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:19.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcf927fc640 0 client.13248 ms_handle_reset on v2:172.21.15.59:6832/2985893835 2024-06-29T15:59:19.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/3761247460 shutdown_connections 2024-06-29T15:59:19.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.287+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/3761247460 wait complete. 2024-06-29T15:59:19.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.288+0000 7fcfc0a5b740 20 client.13248 trim_cache size 0 max 16384 2024-06-29T15:59:19.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.307+0000 7fcfc0a5b740 1 Processor -- start 2024-06-29T15:59:19.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.307+0000 7fcfc0a5b740 1 -- start start 2024-06-29T15:59:19.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.308+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adc9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb4201050 con 0x555f8a9d33b0 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb4200530 con 0x555f8a9fc270 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb4200a30 con 0x555f8a9db590 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adc9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adc9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43552/0 (socket says 172.21.15.59:43552) 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 -- 172.21.15.59:0/1720749623 learned_addr learned my addr 172.21.15.59:0/1720749623 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb3fff640 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 -- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 msgr2=0x555f8adcca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 -- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 msgr2=0x555f8adccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:19.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 -- 172.21.15.59:0/1720749623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f8adddb40 con 0x555f8a9d33b0 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb3fff640 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adc9f40 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4005b20 tx=0x7fcfa4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497f2bcad22c9a05 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcf98ff9640 1 -- 172.21.15.59:0/1720749623 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa402e030 con 0x555f8a9d33b0 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcf98ff9640 1 -- 172.21.15.59:0/1720749623 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfa4005190 con 0x555f8a9d33b0 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.309+0000 7fcf98ff9640 1 -- 172.21.15.59:0/1720749623 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4016070 con 0x555f8a9d33b0 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1720749623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 msgr2=0x555f8adc9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adc9f40 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4005b20 tx=0x7fcfa4013000 comp rx=0 tx=0).stop 2024-06-29T15:59:19.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1720749623 shutdown_connections 2024-06-29T15:59:19.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9db590 0x555f8adcca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1720749623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9d33b0 0x555f8adc9f40 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1720749623 >> 172.21.15.59:0/1720749623 conn(0x555f8a9f03c0 msgr2=0x555f8addec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:19.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1720749623 shutdown_connections 2024-06-29T15:59:19.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.310+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1720749623 wait complete. 2024-06-29T15:59:19.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.311+0000 7fcfc0a5b740 1 -- start start 2024-06-29T15:59:19.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.311+0000 7fcfc0a5b740 1 Processor -- start 2024-06-29T15:59:19.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 0x555f8ada2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8ae617c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfc0a5b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adf9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x555f8aca0ae0 con 0x555f8a9db590 2024-06-29T15:59:19.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555f8aa76620 con 0x555f8a9fc270 2024-06-29T15:59:19.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfc0a5b740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x555f8ad06c10 con 0x555f8a9d33b0 2024-06-29T15:59:19.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfc0a5b740 10 client.0 _ll_register_callbacks cb 0x7fcfa8060d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:59:19.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8ae617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8ae617c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43560/0 (socket says 172.21.15.59:43560) 2024-06-29T15:59:19.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/1702280902 learned_addr learned my addr 172.21.15.59:0/1702280902 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:19.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 0x555f8ada2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 msgr2=0x555f8ada2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:19.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 0x555f8ada2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 msgr2=0x555f8adf9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:19.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb3fff640 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adf9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adf9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 -- 172.21.15.59:0/1702280902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f8adc9fc0 con 0x555f8a9db590 2024-06-29T15:59:19.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb3fff640 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adf9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:19.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfb2ffd640 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8ae617c0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fcfa805e800 tx=0x7fcfa8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0b8658b716cb4f server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:19.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcf927fc640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:59:19.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcf927fc640 1 -- 172.21.15.59:0/1702280902 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa807a040 con 0x555f8a9db590 2024-06-29T15:59:19.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcf927fc640 1 -- 172.21.15.59:0/1702280902 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7fcfa80962b0 con 0x555f8a9db590 2024-06-29T15:59:19.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.313+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1702280902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa4005190 con 0x555f8a9db590 2024-06-29T15:59:19.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcf927fc640 1 -- 172.21.15.59:0/1702280902 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=148) ==== 24+0+0 (secure 0 0 0) 0x7fcfa805dbe0 con 0x555f8a9db590 2024-06-29T15:59:19.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfc0a5b740 10 client.13254 fetch_fsmap learned FSMap version 148 2024-06-29T15:59:19.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1702280902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=148,monmap=2+}) -- 0x555f8ae64e00 con 0x555f8a9db590 2024-06-29T15:59:19.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcf927fc640 1 -- 172.21.15.59:0/1702280902 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfa4005190 con 0x555f8a9db590 2024-06-29T15:59:19.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcf927fc640 1 -- 172.21.15.59:0/1702280902 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 148) ==== 3502+0+0 (secure 0 0 0) 0x7fcfa8060260 con 0x555f8a9db590 2024-06-29T15:59:19.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfc0a5b740 10 client.13254 fetch_fsmap finished waiting for FSMap version 148 2024-06-29T15:59:19.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfc0a5b740 10 client.13254 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:59:19.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfc0a5b740 20 client.13254 populate_metadata read hostname 'smithi059' 2024-06-29T15:59:19.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] conn(0x555f8adc8a60 0x555f8ada0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfc0a5b740 4 client.13254 mds_command: new command op to 13110 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:59:19.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1702280902 --> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] -- command(tid 0: {"prefix": "status"}) -- 0x555f8ade2e40 con 0x555f8adc8a60 2024-06-29T15:59:19.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.314+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] conn(0x555f8adc8a60 0x555f8ada0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcf927fc640 10 client.13254 ms_handle_connect on v2:172.21.15.59:6832/2985893835 2024-06-29T15:59:19.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfb37fe640 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] conn(0x555f8adc8a60 0x555f8ada0cd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4f1de3b12a63b71 server_cookie=9833a940ee8d9774 in_seq=0 out_seq=0 2024-06-29T15:59:19.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcf927fc640 1 -- 172.21.15.59:0/1702280902 <== mds.0 v2:172.21.15.59:6832/2985893835 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x555f8ade2e40 con 0x555f8adc8a60 2024-06-29T15:59:19.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcf927fc640 10 client.13254 handle_command_reply: tid=0 2024-06-29T15:59:19.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 client.13254 shutdown 2024-06-29T15:59:19.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1702280902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 msgr2=0x555f8ae617c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8ae617c0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fcfa805e800 tx=0x7fcfa8009f90 comp rx=0 tx=0).stop 2024-06-29T15:59:19.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1702280902 shutdown_connections 2024-06-29T15:59:19.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.59:6832/2985893835,v1:172.21.15.59:6833/2985893835] conn(0x555f8adc8a60 0x555f8ada0cd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555f8a9fc270 0x555f8adf9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x555f8a9db590 0x555f8ae617c0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 --2- 172.21.15.59:0/1702280902 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x555f8a9d33b0 0x555f8ada2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1702280902 >> 172.21.15.59:0/1702280902 conn(0x555f8a9f03c0 msgr2=0x555f8ade7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:19.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcf927fc640 0 client.13254 ms_handle_reset on v2:172.21.15.59:6832/2985893835 2024-06-29T15:59:19.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1702280902 shutdown_connections 2024-06-29T15:59:19.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 1 -- 172.21.15.59:0/1702280902 wait complete. 2024-06-29T15:59:19.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.315+0000 7fcfc0a5b740 20 client.13254 trim_cache size 0 max 16384 2024-06-29T15:59:19.331 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:59:19.331 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:59:19.331 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:59:19.331 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 13110, 2024-06-29T15:59:19.331 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:59:19.331 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:59:19.332 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T15:59:19.332 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8858701189999998, 2024-06-29T15:59:19.332 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 148, 2024-06-29T15:59:19.332 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 262, 2024-06-29T15:59:19.332 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 262, 2024-06-29T15:59:19.332 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.535310172000001 2024-06-29T15:59:19.332 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:59:19.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcfa0076540 msgr2=0x7fcfa0078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcfa0076540 0x7fcfa0078a50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4000f60 tx=0x7fcfa4000ed0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/3117876821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfb414de30 msgr2=0x7fcfb406d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfb414de30 0x7fcfb406d310 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8011ae0 tx=0x7fcfa8009e40 comp rx=0 tx=0).stop 2024-06-29T15:59:19.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/3117876821 shutdown_connections 2024-06-29T15:59:19.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fcfa0076540 0x7fcfa0078a50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcfb414e240 0x7fcfb406d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfb414de30 0x7fcfb406d310 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 --2- 172.21.15.59:0/3117876821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcf9c0053a0 0x7fcfb40c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/3117876821 >> 172.21.15.59:0/3117876821 conn(0x7fcfb40d45b0 msgr2=0x7fcfb40b79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/3117876821 shutdown_connections 2024-06-29T15:59:19.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.316+0000 7fcfb9b9e640 1 -- 172.21.15.59:0/3117876821 wait complete. 2024-06-29T15:59:19.335 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:59:19.392 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-06-29T15:59:19.391+0000 7fe62b548640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:59:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.453+0000 7ff447f7f640 1 -- 172.21.15.59:0/176069257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 msgr2=0x7ff4400d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.453+0000 7ff447f7f640 1 --2- 172.21.15.59:0/176069257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 0x7ff4400d2130 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7ff438005fd0 tx=0x7ff43802aa70 comp rx=0 tx=0).stop 2024-06-29T15:59:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.453+0000 7ff447f7f640 1 -- 172.21.15.59:0/176069257 shutdown_connections 2024-06-29T15:59:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.453+0000 7ff447f7f640 1 --2- 172.21.15.59:0/176069257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 0x7ff4400d2130 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.453+0000 7ff447f7f640 1 --2- 172.21.15.59:0/176069257 >> v2:172.21.15.153:3300/0 conn(0x7ff44008c190 0x7ff4400cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.453+0000 7ff447f7f640 1 -- 172.21.15.59:0/176069257 >> 172.21.15.59:0/176069257 conn(0x7ff4400b5610 msgr2=0x7ff4400b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.453+0000 7ff447f7f640 1 -- 172.21.15.59:0/176069257 shutdown_connections 2024-06-29T15:59:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.453+0000 7ff447f7f640 1 -- 172.21.15.59:0/176069257 wait complete. 2024-06-29T15:59:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.455+0000 7ff447f7f640 1 Processor -- start 2024-06-29T15:59:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.455+0000 7ff447f7f640 1 -- start start 2024-06-29T15:59:19.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff447f7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff44008c190 0x7ff440158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff447f7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 0x7ff44015bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff447f7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4400cf5c0 0x7ff440156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff447f7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff4400b7f60 con 0x7ff4400ce950 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff447f7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff4400b7de0 con 0x7ff44008c190 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff447f7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff4400b80e0 con 0x7ff4400cf5c0 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44551a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 0x7ff44015bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44551a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 0x7ff44015bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43578/0 (socket says 172.21.15.59:43578) 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44551a640 1 -- 172.21.15.59:0/1544318094 learned_addr learned my addr 172.21.15.59:0/1544318094 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44651c640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4400cf5c0 0x7ff440156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff445d1b640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff44008c190 0x7ff440158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44551a640 1 -- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4400cf5c0 msgr2=0x7ff440156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44551a640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4400cf5c0 0x7ff440156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44551a640 1 -- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff44008c190 msgr2=0x7ff440158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44551a640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff44008c190 0x7ff440158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff44551a640 1 -- 172.21.15.59:0/1544318094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44015ac30 con 0x7ff4400ce950 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.456+0000 7ff445d1b640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff44008c190 0x7ff440158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.457+0000 7ff44551a640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 0x7ff44015bcd0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7ff43000c370 tx=0x7ff43000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.457+0000 7ff4367fc640 1 -- 172.21.15.59:0/1544318094 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430017070 con 0x7ff4400ce950 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.457+0000 7ff4367fc640 1 -- 172.21.15.59:0/1544318094 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4300053c0 con 0x7ff4400ce950 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.457+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff44015af10 con 0x7ff4400ce950 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.457+0000 7ff4367fc640 1 -- 172.21.15.59:0/1544318094 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430003850 con 0x7ff4400ce950 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.457+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff44015b4d0 con 0x7ff4400ce950 2024-06-29T15:59:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.458+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4400cf080 con 0x7ff4400ce950 2024-06-29T15:59:19.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.459+0000 7ff4367fc640 1 -- 172.21.15.59:0/1544318094 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7ff43001a070 con 0x7ff4400ce950 2024-06-29T15:59:19.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.459+0000 7ff4367fc640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff424076540 0x7ff424078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:19.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.459+0000 7ff4367fc640 1 -- 172.21.15.59:0/1544318094 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7ff430097020 con 0x7ff4400ce950 2024-06-29T15:59:19.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.460+0000 7ff445d1b640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff424076540 0x7ff424078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:19.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.460+0000 7ff445d1b640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff424076540 0x7ff424078a50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009ab0 tx=0x7ff43c005f90 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:19.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.464+0000 7ff4367fc640 1 -- 172.21.15.59:0/1544318094 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff430060530 con 0x7ff4400ce950 2024-06-29T15:59:19.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.695+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff44015d610 con 0x7ff4400ce950 2024-06-29T15:59:19.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.695+0000 7ff4367fc640 1 -- 172.21.15.59:0/1544318094 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v148) ==== 53+0+73 (secure 0 0 0) 0x7ff43005fe30 con 0x7ff4400ce950 2024-06-29T15:59:19.696 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:59:19.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff424076540 msgr2=0x7ff424078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff424076540 0x7ff424078a50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009ab0 tx=0x7ff43c005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:19.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 msgr2=0x7ff44015bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:19.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 0x7ff44015bcd0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7ff43000c370 tx=0x7ff43000c830 comp rx=0 tx=0).stop 2024-06-29T15:59:19.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 shutdown_connections 2024-06-29T15:59:19.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7ff424076540 0x7ff424078a50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff4400cf5c0 0x7ff440156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4400ce950 0x7ff44015bcd0 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 --2- 172.21.15.59:0/1544318094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff44008c190 0x7ff440158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:19.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 >> 172.21.15.59:0/1544318094 conn(0x7ff4400b5610 msgr2=0x7ff4400c9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:19.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 shutdown_connections 2024-06-29T15:59:19.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19.697+0000 7ff447f7f640 1 -- 172.21.15.59:0/1544318094 wait complete. 2024-06-29T15:59:19.709 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:59:19.709 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:59:19.709 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T15:59:19.709 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:59:19.710 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:19.710 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:59:19.710 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:59:19.710 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:59:19.710 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:19.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:19.710 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T15:59:19.727 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 36891sec preferred_lft 36891sec 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T15:59:19.728 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T15:59:19.729 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:19.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:19.730 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:59:19.790 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T15:59:19.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:19.792 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T15:59:19.849 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T15:59:19.850 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:59:19.850 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T15:59:19.850 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T15:59:19.850 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:59:19.850 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:59:19.850 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T15:59:19.850 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T15:59:19.850 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T15:59:19.850 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T15:59:19.851 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T15:59:19.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:59:19.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:59:19.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:19.980 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T15:59:19.998 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:59:19.999 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:59:19.999 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:19.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:19.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T15:59:20.064 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:59:20.065 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:20.120 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:20.121 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:59:20.121 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:20.177 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T15:59:20.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:20.178 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:59:20.271 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:59:20.272 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:59:20.272 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:59:20.282 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:59:20.282 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:59:20.282 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:59:20.282 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:59:20.283 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:59:20.288 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:59:20.288 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:59:20.289 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:59:20.289 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:59:20.289 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:59:20.289 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T15:59:20.289 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T15:59:20.289 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T15:59:20.291 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:59:20.291 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:59:20.291 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:59:20.291 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:59:20.291 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:59:20.292 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:59:20.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:20.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:20.322 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:59:20.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:59:20.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:59:20.426 INFO:teuthology.orchestra.run.smithi059.stdout:client13266 2024-06-29T15:59:20.426 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:20.426 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:20.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:59:20.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.601+0000 7f150999e640 1 -- 172.21.15.59:0/650876932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15041458a0 msgr2=0x7f15040cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:20.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.601+0000 7f150999e640 1 --2- 172.21.15.59:0/650876932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15041458a0 0x7f15040cf870 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f15040b1c40 tx=0x7f14fc02a560 comp rx=0 tx=0).stop 2024-06-29T15:59:20.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.601+0000 7f150999e640 1 -- 172.21.15.59:0/650876932 shutdown_connections 2024-06-29T15:59:20.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.601+0000 7f150999e640 1 --2- 172.21.15.59:0/650876932 >> v2:172.21.15.153:3300/0 conn(0x7f15040b3740 0x7f15040b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.601+0000 7f150999e640 1 --2- 172.21.15.59:0/650876932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15041458a0 0x7f15040cf870 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.601+0000 7f150999e640 1 -- 172.21.15.59:0/650876932 >> 172.21.15.59:0/650876932 conn(0x7f150406c950 msgr2=0x7f1504073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:20.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.601+0000 7f150999e640 1 -- 172.21.15.59:0/650876932 shutdown_connections 2024-06-29T15:59:20.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.601+0000 7f150999e640 1 -- 172.21.15.59:0/650876932 wait complete. 2024-06-29T15:59:20.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.602+0000 7f150999e640 1 Processor -- start 2024-06-29T15:59:20.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.603+0000 7f150999e640 1 -- start start 2024-06-29T15:59:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f150999e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15040b3740 0x7f1504150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f150999e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150415c2f0 0x7f15041506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f150999e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150414e0f0 0x7f1504150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:20.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f150999e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1504160020 con 0x7f150415c2f0 2024-06-29T15:59:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f150999e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f150415cbb0 con 0x7f15040b3740 2024-06-29T15:59:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f150999e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f15041601a0 con 0x7f150414e0f0 2024-06-29T15:59:20.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f15027fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150415c2f0 0x7f15041506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:20.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f15027fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150415c2f0 0x7f15041506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43594/0 (socket says 172.21.15.59:43594) 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f15027fc640 1 -- 172.21.15.59:0/2341413336 learned_addr learned my addr 172.21.15.59:0/2341413336 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f1502ffd640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15040b3740 0x7f1504150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f15037fe640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150414e0f0 0x7f1504150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f15027fc640 1 -- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150414e0f0 msgr2=0x7f1504150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.604+0000 7f15027fc640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150414e0f0 0x7f1504150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f15027fc640 1 -- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15040b3740 msgr2=0x7f1504150030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f15027fc640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15040b3740 0x7f1504150030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f15027fc640 1 -- 172.21.15.59:0/2341413336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15040c6b30 con 0x7f150415c2f0 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f1502ffd640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15040b3740 0x7f1504150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:20.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f15037fe640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150414e0f0 0x7f1504150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f15027fc640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150415c2f0 0x7f15041506a0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00d700 tx=0x7f14ec00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f14fbfff640 1 -- 172.21.15.59:0/2341413336 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec017070 con 0x7f150415c2f0 2024-06-29T15:59:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f14fbfff640 1 -- 172.21.15.59:0/2341413336 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14ec005310 con 0x7f150415c2f0 2024-06-29T15:59:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15040c6e10 con 0x7f150415c2f0 2024-06-29T15:59:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f14fbfff640 1 -- 172.21.15.59:0/2341413336 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec009820 con 0x7f150415c2f0 2024-06-29T15:59:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.605+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1504151120 con 0x7f150415c2f0 2024-06-29T15:59:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.606+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1504151400 con 0x7f150415c2f0 2024-06-29T15:59:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.607+0000 7f14fbfff640 1 -- 172.21.15.59:0/2341413336 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f14ec0099c0 con 0x7f150415c2f0 2024-06-29T15:59:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.607+0000 7f14fbfff640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f14e4076510 0x7f14e4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.607+0000 7f14fbfff640 1 -- 172.21.15.59:0/2341413336 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f14ec097fc0 con 0x7f150415c2f0 2024-06-29T15:59:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.608+0000 7f1502ffd640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f14e4076510 0x7f14e4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.608+0000 7f1502ffd640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f14e4076510 0x7f14e4078a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f14fc033000 tx=0x7f14fc004730 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:20.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.612+0000 7f14fbfff640 1 -- 172.21.15.59:0/2341413336 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f14ec061270 con 0x7f150415c2f0 2024-06-29T15:59:20.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.826+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 --> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f15041606c0 con 0x7f14e4076510 2024-06-29T15:59:20.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.829+0000 7f14fbfff640 1 -- 172.21.15.59:0/2341413336 <== mgr.11797 v2:172.21.15.153:6832/3574471609 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f15041606c0 con 0x7f14e4076510 2024-06-29T15:59:20.829 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:59:20.829 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:20.830 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T15:59:20.830 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:20.830 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f14e4076510 msgr2=0x7f14e4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f14e4076510 0x7f14e4078a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f14fc033000 tx=0x7f14fc004730 comp rx=0 tx=0).stop 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150415c2f0 msgr2=0x7f15041506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150415c2f0 0x7f15041506a0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00d700 tx=0x7f14ec00dbc0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 shutdown_connections 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f14e4076510 0x7f14e4078a20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f150414e0f0 0x7f1504150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150415c2f0 0x7f15041506a0 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 --2- 172.21.15.59:0/2341413336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15040b3740 0x7f1504150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 >> 172.21.15.59:0/2341413336 conn(0x7f150406c950 msgr2=0x7f1504141df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:20.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 shutdown_connections 2024-06-29T15:59:20.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.831+0000 7f150999e640 1 -- 172.21.15.59:0/2341413336 wait complete. 2024-06-29T15:59:20.843 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:59:20.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 -- 172.21.15.59:0/2016959081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f0144470 msgr2=0x7f08f00b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:20.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 --2- 172.21.15.59:0/2016959081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f0144470 0x7f08f00b1b00 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f08f0072450 tx=0x7f08e002aa30 comp rx=0 tx=0).stop 2024-06-29T15:59:20.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 -- 172.21.15.59:0/2016959081 shutdown_connections 2024-06-29T15:59:20.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 --2- 172.21.15.59:0/2016959081 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08f006e840 0x7f08f015e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 --2- 172.21.15.59:0/2016959081 >> v2:172.21.15.153:3300/0 conn(0x7f08f00b2040 0x7f08f006e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 --2- 172.21.15.59:0/2016959081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f0144470 0x7f08f00b1b00 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 -- 172.21.15.59:0/2016959081 >> 172.21.15.59:0/2016959081 conn(0x7f08f00b06e0 msgr2=0x7f08f00b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:20.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 -- 172.21.15.59:0/2016959081 shutdown_connections 2024-06-29T15:59:20.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.962+0000 7f08f7eba640 1 -- 172.21.15.59:0/2016959081 wait complete. 2024-06-29T15:59:20.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.964+0000 7f08f7eba640 1 Processor -- start 2024-06-29T15:59:20.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.964+0000 7f08f7eba640 1 -- start start 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f7eba640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f006e840 0x7f08f01f8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f7eba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f00b2040 0x7f08f0140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f7eba640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08f0144470 0x7f08f0070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f7eba640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f08f015ee30 con 0x7f08f00b2040 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f7eba640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08f015ecb0 con 0x7f08f006e840 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f7eba640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f08f015efb0 con 0x7f08f0144470 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5455640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f00b2040 0x7f08f0140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5455640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f00b2040 0x7f08f0140430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43612/0 (socket says 172.21.15.59:43612) 2024-06-29T15:59:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5455640 1 -- 172.21.15.59:0/4217584595 learned_addr learned my addr 172.21.15.59:0/4217584595 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5c56640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f006e840 0x7f08f01f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f6457640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08f0144470 0x7f08f0070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5455640 1 -- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08f0144470 msgr2=0x7f08f0070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5455640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08f0144470 0x7f08f0070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5455640 1 -- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f006e840 msgr2=0x7f08f01f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5455640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f006e840 0x7f08f01f8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.965+0000 7f08f5455640 1 -- 172.21.15.59:0/4217584595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f01f8c30 con 0x7f08f00b2040 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.966+0000 7f08f5455640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f00b2040 0x7f08f0140430 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f08e400c970 tx=0x7f08e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.966+0000 7f08ee7fc640 1 -- 172.21.15.59:0/4217584595 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4007b50 con 0x7f08f00b2040 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.966+0000 7f08ee7fc640 1 -- 172.21.15.59:0/4217584595 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08e4007de0 con 0x7f08f00b2040 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.966+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f01f8f10 con 0x7f08f00b2040 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.966+0000 7f08ee7fc640 1 -- 172.21.15.59:0/4217584595 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e400f6b0 con 0x7f08f00b2040 2024-06-29T15:59:20.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.966+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f00a9b30 con 0x7f08f00b2040 2024-06-29T15:59:20.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.967+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08f015f470 con 0x7f08f00b2040 2024-06-29T15:59:20.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.968+0000 7f08ee7fc640 1 -- 172.21.15.59:0/4217584595 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f08e400f850 con 0x7f08f00b2040 2024-06-29T15:59:20.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.968+0000 7f08ee7fc640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f08dc076510 0x7f08dc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:20.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.968+0000 7f08ee7fc640 1 -- 172.21.15.59:0/4217584595 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f08e4098450 con 0x7f08f00b2040 2024-06-29T15:59:20.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.969+0000 7f08f5c56640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f08dc076510 0x7f08dc078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:20.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.969+0000 7f08f5c56640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f08dc076510 0x7f08dc078a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f08e0035000 tx=0x7f08e003b040 comp rx=0 tx=0).ready entity=mgr.11797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:20.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:20.973+0000 7f08ee7fc640 1 -- 172.21.15.59:0/4217584595 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f08e4014030 con 0x7f08f00b2040 2024-06-29T15:59:21.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.213+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f08f01f89c0 con 0x7f08f00b2040 2024-06-29T15:59:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.482+0000 7f08ee7fc640 1 -- 172.21.15.59:0/4217584595 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v147) ==== 92+0+0 (secure 0 0 0) 0x7f08e40610a0 con 0x7f08f00b2040 2024-06-29T15:59:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.484+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f08dc076510 msgr2=0x7f08dc078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.484+0000 7f08f7eba640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f08dc076510 0x7f08dc078a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f08e0035000 tx=0x7f08e003b040 comp rx=0 tx=0).stop 2024-06-29T15:59:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.484+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f00b2040 msgr2=0x7f08f0140430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.484+0000 7f08f7eba640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f00b2040 0x7f08f0140430 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f08e400c970 tx=0x7f08e400ce30 comp rx=0 tx=0).stop 2024-06-29T15:59:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.485+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 shutdown_connections 2024-06-29T15:59:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.485+0000 7f08f7eba640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f08dc076510 0x7f08dc078a20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.485+0000 7f08f7eba640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08f0144470 0x7f08f0070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.485+0000 7f08f7eba640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08f00b2040 0x7f08f0140430 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.485+0000 7f08f7eba640 1 --2- 172.21.15.59:0/4217584595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08f006e840 0x7f08f01f8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.485+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 >> 172.21.15.59:0/4217584595 conn(0x7f08f00b06e0 msgr2=0x7f08f0143b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.485+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 shutdown_connections 2024-06-29T15:59:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.486+0000 7f08f7eba640 1 -- 172.21.15.59:0/4217584595 wait complete. 2024-06-29T15:59:21.500 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:59:21.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.624+0000 7f921a4b3640 1 -- 172.21.15.59:0/1719419010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b3ed0 msgr2=0x7f921415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.624+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1719419010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b3ed0 0x7f921415e8d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9208009830 tx=0x7f920802a510 comp rx=0 tx=0).stop 2024-06-29T15:59:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.624+0000 7f921a4b3640 1 -- 172.21.15.59:0/1719419010 shutdown_connections 2024-06-29T15:59:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.624+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1719419010 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b3ed0 0x7f921415e8d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.624+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1719419010 >> v2:172.21.15.153:3300/0 conn(0x7f92140d23c0 0x7f92140b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.624+0000 7f921a4b3640 1 -- 172.21.15.59:0/1719419010 >> 172.21.15.59:0/1719419010 conn(0x7f92140d3ee0 msgr2=0x7f92140d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.624+0000 7f921a4b3640 1 -- 172.21.15.59:0/1719419010 shutdown_connections 2024-06-29T15:59:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.624+0000 7f921a4b3640 1 -- 172.21.15.59:0/1719419010 wait complete. 2024-06-29T15:59:21.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.626+0000 7f921a4b3640 1 Processor -- start 2024-06-29T15:59:21.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.626+0000 7f921a4b3640 1 -- start start 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f921a4b3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b3ed0 0x7f92141f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f921a4b3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140d23c0 0x7f9214158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f921a4b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921415ee10 0x7f92141ea710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f921a4b3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9214160240 con 0x7f92140d23c0 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f921a4b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92141600c0 con 0x7f921415ee10 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f921a4b3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f92141603c0 con 0x7f92140b3ed0 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140d23c0 0x7f9214158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140d23c0 0x7f9214158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43636/0 (socket says 172.21.15.59:43636) 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 -- 172.21.15.59:0/1535587263 learned_addr learned my addr 172.21.15.59:0/1535587263 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f9213fff640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b3ed0 0x7f92141f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f9218a50640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921415ee10 0x7f92141ea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 -- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b3ed0 msgr2=0x7f92141f50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b3ed0 0x7f92141f50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 -- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921415ee10 msgr2=0x7f92141ea710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921415ee10 0x7f92141ea710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 -- 172.21.15.59:0/1535587263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9214159a90 con 0x7f92140d23c0 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f9218a50640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921415ee10 0x7f92141ea710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.628+0000 7f92137fe640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140d23c0 0x7f9214158eb0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f9208009950 tx=0x7f9208033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.629+0000 7f9210ff9640 1 -- 172.21.15.59:0/1535587263 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9208031d60 con 0x7f92140d23c0 2024-06-29T15:59:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.629+0000 7f9210ff9640 1 -- 172.21.15.59:0/1535587263 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9208033e10 con 0x7f92140d23c0 2024-06-29T15:59:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.629+0000 7f9210ff9640 1 -- 172.21.15.59:0/1535587263 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92080332b0 con 0x7f92140d23c0 2024-06-29T15:59:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.629+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9214159d70 con 0x7f92140d23c0 2024-06-29T15:59:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.629+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921415ad10 con 0x7f92140d23c0 2024-06-29T15:59:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.630+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92141597e0 con 0x7f92140d23c0 2024-06-29T15:59:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.630+0000 7f9210ff9640 1 -- 172.21.15.59:0/1535587263 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 147) ==== 98422+0+0 (secure 0 0 0) 0x7f9208033450 con 0x7f92140d23c0 2024-06-29T15:59:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.631+0000 7f9210ff9640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f9200076560 0x7f9200078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.631+0000 7f9213fff640 1 -- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f9200076560 msgr2=0x7f9200078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3574471609 2024-06-29T15:59:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.631+0000 7f9210ff9640 1 -- 172.21.15.59:0/1535587263 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f92080b8e70 con 0x7f92140d23c0 2024-06-29T15:59:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.631+0000 7f9213fff640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f9200076560 0x7f9200078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:21.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.636+0000 7f9210ff9640 1 -- 172.21.15.59:0/1535587263 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f920804a030 con 0x7f92140d23c0 2024-06-29T15:59:21.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.832+0000 7f9213fff640 1 -- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f9200076560 msgr2=0x7f9200078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3574471609 2024-06-29T15:59:21.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.833+0000 7f9213fff640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f9200076560 0x7f9200078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:21.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.850+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f921415e8d0 con 0x7f92140d23c0 2024-06-29T15:59:21.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.854+0000 7f9210ff9640 1 -- 172.21.15.59:0/1535587263 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v81)=0 v81) ==== 138+0+0 (secure 0 0 0) 0x7f920803f030 con 0x7f92140d23c0 2024-06-29T15:59:21.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f9200076560 msgr2=0x7f9200078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:21.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f9200076560 0x7f9200078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140d23c0 msgr2=0x7f9214158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140d23c0 0x7f9214158eb0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f9208009950 tx=0x7f9208033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:21.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 shutdown_connections 2024-06-29T15:59:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f9200076560 0x7f9200078a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921415ee10 0x7f92141ea710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92140d23c0 0x7f9214158eb0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 --2- 172.21.15.59:0/1535587263 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92140b3ed0 0x7f92141f50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 >> 172.21.15.59:0/1535587263 conn(0x7f92140d3ee0 msgr2=0x7f921406d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 shutdown_connections 2024-06-29T15:59:21.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.856+0000 7f921a4b3640 1 -- 172.21.15.59:0/1535587263 wait complete. 2024-06-29T15:59:21.869 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:59:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.991+0000 7f1c15a74640 1 -- 172.21.15.59:0/2769925715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100d1a60 msgr2=0x7f1c100b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.991+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2769925715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100d1a60 0x7f1c100b5b30 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f1c100b2680 tx=0x7f1c0802aac0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.991+0000 7f1c15a74640 1 -- 172.21.15.59:0/2769925715 shutdown_connections 2024-06-29T15:59:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.991+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2769925715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100d1a60 0x7f1c100b5b30 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.991+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2769925715 >> v2:172.21.15.153:3300/0 conn(0x7f1c100b2fa0 0x7f1c100d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.991+0000 7f1c15a74640 1 -- 172.21.15.59:0/2769925715 >> 172.21.15.59:0/2769925715 conn(0x7f1c10073e10 msgr2=0x7f1c10074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.991+0000 7f1c15a74640 1 -- 172.21.15.59:0/2769925715 shutdown_connections 2024-06-29T15:59:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.991+0000 7f1c15a74640 1 -- 172.21.15.59:0/2769925715 wait complete. 2024-06-29T15:59:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.993+0000 7f1c15a74640 1 Processor -- start 2024-06-29T15:59:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.993+0000 7f1c15a74640 1 -- start start 2024-06-29T15:59:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.994+0000 7f1c15a74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100b2270 0x7f1c101539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.994+0000 7f1c15a74640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c100b2fa0 0x7f1c100aca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.994+0000 7f1c15a74640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c100d1a60 0x7f1c100b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.994+0000 7f1c15a74640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1c100b6400 con 0x7f1c100b2270 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.994+0000 7f1c15a74640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c100b6280 con 0x7f1c100b2fa0 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.994+0000 7f1c15a74640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1c100b6580 con 0x7f1c100d1a60 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.994+0000 7f1c0f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100b2270 0x7f1c101539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100b2270 0x7f1c101539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43648/0 (socket says 172.21.15.59:43648) 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0f7fe640 1 -- 172.21.15.59:0/2254096609 learned_addr learned my addr 172.21.15.59:0/2254096609 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0effd640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c100b2fa0 0x7f1c100aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0ffff640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c100d1a60 0x7f1c100b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0f7fe640 1 -- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c100d1a60 msgr2=0x7f1c100b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0f7fe640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c100d1a60 0x7f1c100b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0f7fe640 1 -- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c100b2fa0 msgr2=0x7f1c100aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0f7fe640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c100b2fa0 0x7f1c100aca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0f7fe640 1 -- 172.21.15.59:0/2254096609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c100ad510 con 0x7f1c100b2270 2024-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c0f7fe640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100b2270 0x7f1c101539f0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000c3c0 tx=0x7f1c0000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1bf7fff640 1 -- 172.21.15.59:0/2254096609 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00017070 con 0x7f1c100b2270 2024-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1bf7fff640 1 -- 172.21.15.59:0/2254096609 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c000035f0 con 0x7f1c100b2270 2024-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c100aa3f0 con 0x7f1c100b2270 2024-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1bf7fff640 1 -- 172.21.15.59:0/2254096609 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00003a80 con 0x7f1c100b2270 2024-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.995+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c100aa9b0 con 0x7f1c100b2270 2024-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.997+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c100b4fe0 con 0x7f1c100b2270 2024-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.997+0000 7f1bf7fff640 1 -- 172.21.15.59:0/2254096609 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 147) ==== 98422+0+0 (secure 0 0 0) 0x7f1c0001a070 con 0x7f1c100b2270 2024-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.998+0000 7f1bf7fff640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1bfc076560 0x7f1bfc078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.998+0000 7f1c0effd640 1 -- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1bfc076560 msgr2=0x7f1bfc078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3574471609 2024-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.998+0000 7f1bf7fff640 1 -- 172.21.15.59:0/2254096609 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7f1c00097080 con 0x7f1c100b2270 2024-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:21.998+0000 7f1c0effd640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1bfc076560 0x7f1bfc078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.002+0000 7f1bf7fff640 1 -- 172.21.15.59:0/2254096609 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c100b4fe0 con 0x7f1c100b2270 2024-06-29T15:59:22.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.194+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f1c10070070 con 0x7f1c100b2270 2024-06-29T15:59:22.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.198+0000 7f1bf7fff640 1 -- 172.21.15.59:0/2254096609 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v82)=0 v82) ==== 133+0+0 (secure 0 0 0) 0x7f1c10070070 con 0x7f1c100b2270 2024-06-29T15:59:22.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.199+0000 7f1c0effd640 1 -- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1bfc076560 msgr2=0x7f1bfc078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3574471609 2024-06-29T15:59:22.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.199+0000 7f1c0effd640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1bfc076560 0x7f1bfc078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:22.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1bfc076560 msgr2=0x7f1bfc078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1bfc076560 0x7f1bfc078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100b2270 msgr2=0x7f1c101539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100b2270 0x7f1c101539f0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000c3c0 tx=0x7f1c0000c880 comp rx=0 tx=0).stop 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 shutdown_connections 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7f1bfc076560 0x7f1bfc078a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c100d1a60 0x7f1c100b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c100b2fa0 0x7f1c100aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 --2- 172.21.15.59:0/2254096609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c100b2270 0x7f1c101539f0 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 >> 172.21.15.59:0/2254096609 conn(0x7f1c10073e10 msgr2=0x7f1c1015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 shutdown_connections 2024-06-29T15:59:22.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.200+0000 7f1c15a74640 1 -- 172.21.15.59:0/2254096609 wait complete. 2024-06-29T15:59:22.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:22.213 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T15:59:22.234 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-06-29T15:59:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.395+0000 7fa333bd4640 1 -- 172.21.15.59:0/3033269938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 msgr2=0x7fa32c0b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.395+0000 7fa333bd4640 1 --2- 172.21.15.59:0/3033269938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 0x7fa32c0b5b30 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0b2680 tx=0x7fa32402aac0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.395+0000 7fa333bd4640 1 -- 172.21.15.59:0/3033269938 shutdown_connections 2024-06-29T15:59:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.395+0000 7fa333bd4640 1 --2- 172.21.15.59:0/3033269938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 0x7fa32c0b5b30 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.395+0000 7fa333bd4640 1 --2- 172.21.15.59:0/3033269938 >> v2:172.21.15.153:3300/0 conn(0x7fa32c0b2fa0 0x7fa32c0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.395+0000 7fa333bd4640 1 -- 172.21.15.59:0/3033269938 >> 172.21.15.59:0/3033269938 conn(0x7fa32c073e10 msgr2=0x7fa32c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.395+0000 7fa333bd4640 1 -- 172.21.15.59:0/3033269938 shutdown_connections 2024-06-29T15:59:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.395+0000 7fa333bd4640 1 -- 172.21.15.59:0/3033269938 wait complete. 2024-06-29T15:59:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.398+0000 7fa333bd4640 1 Processor -- start 2024-06-29T15:59:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.398+0000 7fa333bd4640 1 -- start start 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.400+0000 7fa333bd4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa32c0b2270 0x7fa32c1f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.400+0000 7fa333bd4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa32c0b2fa0 0x7fa32c15dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa333bd4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 0x7fa32c15a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa333bd4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c0b6400 con 0x7fa32c0d1a60 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa333bd4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c0b6280 con 0x7fa32c0b2fa0 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa333bd4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa32c0b6580 con 0x7fa32c0b2270 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 0x7fa32c15a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 0x7fa32c15a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43668/0 (socket says 172.21.15.59:43668) 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 -- 172.21.15.59:0/2686384454 learned_addr learned my addr 172.21.15.59:0/2686384454 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa33116f640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa32c0b2fa0 0x7fa32c15dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa331970640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa32c0b2270 0x7fa32c1f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa32c0b2270 msgr2=0x7fa32c1f5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa32c0b2270 0x7fa32c1f5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa32c0b2fa0 msgr2=0x7fa32c15dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa32c0b2fa0 0x7fa32c15dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 -- 172.21.15.59:0/2686384454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32c15e860 con 0x7fa32c0d1a60 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.401+0000 7fa332171640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 0x7fa32c15a2f0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fa324009d60 tx=0x7fa324003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.402+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324044070 con 0x7fa32c0d1a60 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.402+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa324003e00 con 0x7fa32c0d1a60 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.402+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa32c15eb40 con 0x7fa32c0d1a60 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.402+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3240044b0 con 0x7fa32c0d1a60 2024-06-29T15:59:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.402+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa32c15f100 con 0x7fa32c0d1a60 2024-06-29T15:59:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.403+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa32c0b6aa0 con 0x7fa32c0d1a60 2024-06-29T15:59:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.404+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 147) ==== 98422+0+0 (secure 0 0 0) 0x7fa324004650 con 0x7fa32c0d1a60 2024-06-29T15:59:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.404+0000 7fa31e7fc640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 0x7fa310078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.404+0000 7fa331970640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 msgr2=0x7fa310078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3574471609 2024-06-29T15:59:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.405+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 6157+0+0 (secure 0 0 0) 0x7fa3240b84c0 con 0x7fa32c0d1a60 2024-06-29T15:59:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.405+0000 7fa331970640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 0x7fa310078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:22.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.409+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa324032030 con 0x7fa32c0d1a60 2024-06-29T15:59:22.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.606+0000 7fa331970640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 msgr2=0x7fa310078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3574471609 2024-06-29T15:59:22.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.606+0000 7fa331970640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 0x7fa310078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:22.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:22.625+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 --> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fa32c15e580 con 0x7fa310076560 2024-06-29T15:59:23.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:23.008+0000 7fa331970640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 msgr2=0x7fa310078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3574471609 2024-06-29T15:59:23.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:23.008+0000 7fa331970640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 0x7fa310078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:59:23.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:23.809+0000 7fa331970640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 msgr2=0x7fa310078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3574471609 2024-06-29T15:59:23.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:23.809+0000 7fa331970640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 0x7fa310078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:59:25.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:25.124+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 148) ==== 97929+0+0 (secure 0 0 0) 0x7fa32407d4c0 con 0x7fa32c0d1a60 2024-06-29T15:59:25.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:25.125+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 msgr2=0x7fa310078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:25.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:25.125+0000 7fa31e7fc640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/3574471609,v1:172.21.15.153:6833/3574471609] conn(0x7fa310076560 0x7fa310078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.276 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f02616e9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.276 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f02616e9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.276 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f025e6e3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.276 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f02616e9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.276 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f02616e9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.277 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f02616e9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.277 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f025e6e3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.277 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f025e6e3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.277 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f025e6e3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:25.277 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:25.275+0000 7f025e6e3640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:26.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.129+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 149) ==== 98056+0+0 (secure 0 0 0) 0x7fa324081490 con 0x7fa32c0d1a60 2024-06-29T15:59:26.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.129+0000 7fa31e7fc640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fa31003b190 0x7fa310080b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.129+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 --> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fa32403ae50 con 0x7fa31003b190 2024-06-29T15:59:26.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.133+0000 7fa331970640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fa31003b190 0x7fa310080b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.135+0000 7fa331970640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fa31003b190 0x7fa310080b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa3280011c0 tx=0x7fa328007e30 comp rx=0 tx=0).ready entity=mgr.12274 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.201+0000 7fa31e7fc640 1 -- 172.21.15.59:0/2686384454 <== mgr.12274 v2:172.21.15.153:6832/1744808854 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa32403ae50 con 0x7fa31003b190 2024-06-29T15:59:26.201 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path snap_test_dir1 2024-06-29T15:59:26.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fa31003b190 msgr2=0x7fa310080b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa333bd4640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fa31003b190 0x7fa310080b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa3280011c0 tx=0x7fa328007e30 comp rx=0 tx=0).stop 2024-06-29T15:59:26.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 msgr2=0x7fa32c15a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa333bd4640 1 --2- 172.21.15.59:0/2686384454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa32c0d1a60 0x7fa32c15a2f0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fa324009d60 tx=0x7fa324003c10 comp rx=0 tx=0).stop 2024-06-29T15:59:26.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa332171640 1 -- 172.21.15.59:0/2686384454 reap_dead start 2024-06-29T15:59:26.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 shutdown_connections 2024-06-29T15:59:26.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 >> 172.21.15.59:0/2686384454 conn(0x7fa32c073e10 msgr2=0x7fa32c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 shutdown_connections 2024-06-29T15:59:26.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.203+0000 7fa333bd4640 1 -- 172.21.15.59:0/2686384454 wait complete. 2024-06-29T15:59:26.216 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T15:59:26.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-06-29T15:59:26.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.340+0000 7f8a4cd31640 1 -- 172.21.15.59:0/992121289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480d1ea0 msgr2=0x7f8a480b56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.340+0000 7f8a4cd31640 1 --2- 172.21.15.59:0/992121289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480d1ea0 0x7f8a480b56e0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009860 tx=0x7f8a3c02a4b0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.340+0000 7f8a4cd31640 1 -- 172.21.15.59:0/992121289 shutdown_connections 2024-06-29T15:59:26.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.340+0000 7f8a4cd31640 1 --2- 172.21.15.59:0/992121289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480d1ea0 0x7f8a480b56e0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.340+0000 7f8a4cd31640 1 -- 172.21.15.59:0/992121289 >> 172.21.15.59:0/992121289 conn(0x7f8a48073e10 msgr2=0x7f8a48074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.340+0000 7f8a4cd31640 1 -- 172.21.15.59:0/992121289 shutdown_connections 2024-06-29T15:59:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.340+0000 7f8a4cd31640 1 -- 172.21.15.59:0/992121289 wait complete. 2024-06-29T15:59:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.342+0000 7f8a4cd31640 1 Processor -- start 2024-06-29T15:59:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.342+0000 7f8a4cd31640 1 -- start start 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.343+0000 7f8a4cd31640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a480cec20 0x7f8a48156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.343+0000 7f8a4cd31640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a481423c0 0x7f8a480cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.343+0000 7f8a4cd31640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480cab70 0x7f8a4806e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.343+0000 7f8a4cd31640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480cff90 con 0x7f8a480cab70 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.343+0000 7f8a4cd31640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480cfe10 con 0x7f8a480cec20 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.343+0000 7f8a4cd31640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8a480d0110 con 0x7f8a481423c0 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480cab70 0x7f8a4806e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480cab70 0x7f8a4806e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44300/0 (socket says 172.21.15.59:44300) 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 -- 172.21.15.59:0/2071421304 learned_addr learned my addr 172.21.15.59:0/2071421304 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a4659c640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a480cec20 0x7f8a48156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a45d9b640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a481423c0 0x7f8a480cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 -- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a481423c0 msgr2=0x7f8a480cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a481423c0 0x7f8a480cc670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 -- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a480cec20 msgr2=0x7f8a48156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a480cec20 0x7f8a48156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 -- 172.21.15.59:0/2071421304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a481566d0 con 0x7f8a480cab70 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a4659c640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a480cec20 0x7f8a48156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a46d9d640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480cab70 0x7f8a4806e140 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000ca80 tx=0x7f8a4000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a36ffd640 1 -- 172.21.15.59:0/2071421304 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40016020 con 0x7f8a480cab70 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a36ffd640 1 -- 172.21.15.59:0/2071421304 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a4000ad20 con 0x7f8a480cab70 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a481f5850 con 0x7f8a480cab70 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a36ffd640 1 -- 172.21.15.59:0/2071421304 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40012680 con 0x7f8a480cab70 2024-06-29T15:59:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.344+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a481f5290 con 0x7f8a480cab70 2024-06-29T15:59:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.346+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a480b56e0 con 0x7f8a480cab70 2024-06-29T15:59:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.346+0000 7f8a36ffd640 1 -- 172.21.15.59:0/2071421304 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 149) ==== 98056+0+0 (secure 0 0 0) 0x7f8a40019020 con 0x7f8a480cab70 2024-06-29T15:59:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.347+0000 7f8a36ffd640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8a28076170 0x7f8a28078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.347+0000 7f8a36ffd640 1 -- 172.21.15.59:0/2071421304 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f8a40097e60 con 0x7f8a480cab70 2024-06-29T15:59:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.347+0000 7f8a4659c640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8a28076170 0x7f8a28078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.347+0000 7f8a4659c640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8a28076170 0x7f8a28078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a38006fd0 tx=0x7f8a38008040 comp rx=0 tx=0).ready entity=mgr.12274 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:26.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.351+0000 7f8a36ffd640 1 -- 172.21.15.59:0/2071421304 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8a40020020 con 0x7f8a480cab70 2024-06-29T15:59:26.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.565+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 --> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) -- 0x7f8a480cf950 con 0x7f8a28076170 2024-06-29T15:59:26.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.620+0000 7f8a36ffd640 1 -- 172.21.15.59:0/2071421304 <== mgr.12274 v2:172.21.15.153:6832/1744808854 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+39 (secure 0 0 0) 0x7f8a480cf950 con 0x7f8a28076170 2024-06-29T15:59:26.620 INFO:teuthology.orchestra.run.smithi059.stdout:Retention added to path snap_test_dir1 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.622+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8a28076170 msgr2=0x7f8a28078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.622+0000 7f8a4cd31640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8a28076170 0x7f8a28078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a38006fd0 tx=0x7f8a38008040 comp rx=0 tx=0).stop 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.622+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480cab70 msgr2=0x7f8a4806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.622+0000 7f8a4cd31640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480cab70 0x7f8a4806e140 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000ca80 tx=0x7f8a4000cf40 comp rx=0 tx=0).stop 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.623+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 shutdown_connections 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.623+0000 7f8a4cd31640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8a28076170 0x7f8a28078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.623+0000 7f8a4cd31640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a480cab70 0x7f8a4806e140 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.623+0000 7f8a4cd31640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a481423c0 0x7f8a480cc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.623+0000 7f8a4cd31640 1 --2- 172.21.15.59:0/2071421304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a480cec20 0x7f8a48156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.623+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 >> 172.21.15.59:0/2071421304 conn(0x7f8a48073e10 msgr2=0x7f8a4806fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.623+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 shutdown_connections 2024-06-29T15:59:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.623+0000 7f8a4cd31640 1 -- 172.21.15.59:0/2071421304 wait complete. 2024-06-29T15:59:26.634 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-06-29T15:59:26.634 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-16_00_00 in ~34s... 2024-06-29T15:59:26.634 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-06-29T15:59:26.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-06-29T15:59:26.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.755+0000 7fef264f6640 1 -- 172.21.15.59:0/2764233300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef200ce950 msgr2=0x7fef201622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.755+0000 7fef264f6640 1 --2- 172.21.15.59:0/2764233300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef200ce950 0x7fef201622c0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fef14009860 tx=0x7fef1402a540 comp rx=0 tx=0).stop 2024-06-29T15:59:26.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.755+0000 7fef264f6640 1 -- 172.21.15.59:0/2764233300 shutdown_connections 2024-06-29T15:59:26.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.755+0000 7fef264f6640 1 --2- 172.21.15.59:0/2764233300 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef20162800 0x7fef20162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.755+0000 7fef264f6640 1 --2- 172.21.15.59:0/2764233300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef200ce950 0x7fef201622c0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.755+0000 7fef264f6640 1 -- 172.21.15.59:0/2764233300 >> 172.21.15.59:0/2764233300 conn(0x7fef200b5610 msgr2=0x7fef200b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.755+0000 7fef264f6640 1 -- 172.21.15.59:0/2764233300 shutdown_connections 2024-06-29T15:59:26.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.755+0000 7fef264f6640 1 -- 172.21.15.59:0/2764233300 wait complete. 2024-06-29T15:59:26.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.757+0000 7fef264f6640 1 Processor -- start 2024-06-29T15:59:26.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.757+0000 7fef264f6640 1 -- start start 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef264f6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef2008c190 0x7fef2015edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef264f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef200ce950 0x7fef2015ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef264f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef20162800 0x7fef200cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef264f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fef200b7f60 con 0x7fef20162800 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef264f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fef200b7de0 con 0x7fef200ce950 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef264f6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fef200b80e0 con 0x7fef2008c190 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef20162800 0x7fef200cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef20162800 0x7fef200cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44302/0 (socket says 172.21.15.59:44302) 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 -- 172.21.15.59:0/4250949655 learned_addr learned my addr 172.21.15.59:0/4250949655 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:26.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef1f7fe640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef200ce950 0x7fef2015ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef1ffff640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef2008c190 0x7fef2015edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 -- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef2008c190 msgr2=0x7fef2015edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef2008c190 0x7fef2015edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 -- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef200ce950 msgr2=0x7fef2015ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef200ce950 0x7fef2015ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 -- 172.21.15.59:0/4250949655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef200cb110 con 0x7fef20162800 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.759+0000 7fef24a93640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef20162800 0x7fef200cabd0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7fef1800ec20 tx=0x7fef1800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.760+0000 7fef1cff9640 1 -- 172.21.15.59:0/4250949655 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18003b70 con 0x7fef20162800 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.760+0000 7fef1cff9640 1 -- 172.21.15.59:0/4250949655 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef18003d10 con 0x7fef20162800 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.760+0000 7fef1cff9640 1 -- 172.21.15.59:0/4250949655 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18004280 con 0x7fef20162800 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.760+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef200cb620 con 0x7fef20162800 2024-06-29T15:59:26.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.760+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef200cc120 con 0x7fef20162800 2024-06-29T15:59:26.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.761+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef200cb390 con 0x7fef20162800 2024-06-29T15:59:26.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.762+0000 7fef1cff9640 1 -- 172.21.15.59:0/4250949655 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 149) ==== 98056+0+0 (secure 0 0 0) 0x7fef18004420 con 0x7fef20162800 2024-06-29T15:59:26.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.762+0000 7fef1cff9640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fef0c076170 0x7fef0c078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.762+0000 7fef1cff9640 1 -- 172.21.15.59:0/4250949655 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7fef18098190 con 0x7fef20162800 2024-06-29T15:59:26.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.762+0000 7fef1ffff640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fef0c076170 0x7fef0c078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.763+0000 7fef1ffff640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fef0c076170 0x7fef0c078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef100046b0 tx=0x7fef10007eb0 comp rx=0 tx=0).ready entity=mgr.12274 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:26.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.767+0000 7fef1cff9640 1 -- 172.21.15.59:0/4250949655 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fef18061470 con 0x7fef20162800 2024-06-29T15:59:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.979+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 --> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef201622c0 con 0x7fef0c076170 2024-06-29T15:59:26.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.985+0000 7fef1cff9640 1 -- 172.21.15.59:0/4250949655 <== mgr.12274 v2:172.21.15.153:6832/1744808854 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+72 (secure 0 0 0) 0x7fef201622c0 con 0x7fef0c076170 2024-06-29T15:59:26.986 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:26.986 INFO:teuthology.orchestra.run.smithi059.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-06-29T15:59:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fef0c076170 msgr2=0x7fef0c078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fef0c076170 0x7fef0c078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef100046b0 tx=0x7fef10007eb0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef20162800 msgr2=0x7fef200cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef20162800 0x7fef200cabd0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7fef1800ec20 tx=0x7fef1800c6a0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 shutdown_connections 2024-06-29T15:59:26.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fef0c076170 0x7fef0c078680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef20162800 0x7fef200cabd0 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef200ce950 0x7fef2015ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 --2- 172.21.15.59:0/4250949655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fef2008c190 0x7fef2015edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 >> 172.21.15.59:0/4250949655 conn(0x7fef200b5610 msgr2=0x7fef200cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 shutdown_connections 2024-06-29T15:59:26.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:26.987+0000 7fef264f6640 1 -- 172.21.15.59:0/4250949655 wait complete. 2024-06-29T15:59:26.999 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-06-29T15:59:26.999 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-06-29T15:59:27.000 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T15:59:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.120+0000 7f5ab9d63640 1 -- 172.21.15.59:0/3479389323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2270 msgr2=0x7f5ab415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.120+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/3479389323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2270 0x7f5ab415bc20 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40cf310 tx=0x7f5aa402a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.120+0000 7f5ab9d63640 1 -- 172.21.15.59:0/3479389323 shutdown_connections 2024-06-29T15:59:27.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.120+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/3479389323 >> v2:172.21.15.153:3300/0 conn(0x7f5ab40d1c30 0x7f5ab40b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.120+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/3479389323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2270 0x7f5ab415bc20 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.120+0000 7f5ab9d63640 1 -- 172.21.15.59:0/3479389323 >> 172.21.15.59:0/3479389323 conn(0x7f5ab4073e10 msgr2=0x7f5ab4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.120+0000 7f5ab9d63640 1 -- 172.21.15.59:0/3479389323 shutdown_connections 2024-06-29T15:59:27.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.120+0000 7f5ab9d63640 1 -- 172.21.15.59:0/3479389323 wait complete. 2024-06-29T15:59:27.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.122+0000 7f5ab9d63640 1 Processor -- start 2024-06-29T15:59:27.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.122+0000 7f5ab9d63640 1 -- start start 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab9d63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab40b2270 0x7f5ab40b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab9d63640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2fa0 0x7f5ab4145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab9d63640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ab40d1c30 0x7f5ab4145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab9d63640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab40b6400 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab9d63640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab40b6280 con 0x7f5ab40b2270 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab9d63640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab40b6580 con 0x7f5ab40d1c30 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2fa0 0x7f5ab4145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2fa0 0x7f5ab4145a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44318/0 (socket says 172.21.15.59:44318) 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 -- 172.21.15.59:0/243296384 learned_addr learned my addr 172.21.15.59:0/243296384 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab37fe640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab40b2270 0x7f5ab40b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab3fff640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ab40d1c30 0x7f5ab4145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 -- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ab40d1c30 msgr2=0x7f5ab4145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ab40d1c30 0x7f5ab4145fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 -- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab40b2270 msgr2=0x7f5ab40b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab40b2270 0x7f5ab40b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 -- 172.21.15.59:0/243296384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab415e9d0 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab3fff640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ab40d1c30 0x7f5ab4145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.124+0000 7f5ab2ffd640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2fa0 0x7f5ab4145a90 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800c3d0 tx=0x7f5aa800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.125+0000 7f5a9bfff640 1 -- 172.21.15.59:0/243296384 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8017070 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.125+0000 7f5a9bfff640 1 -- 172.21.15.59:0/243296384 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5aa80053c0 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.125+0000 7f5a9bfff640 1 -- 172.21.15.59:0/243296384 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8003850 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.125+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab415ecb0 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.125+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab4145290 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.126+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab40b6aa0 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.127+0000 7f5a9bfff640 1 -- 172.21.15.59:0/243296384 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 149) ==== 98056+0+0 (secure 0 0 0) 0x7f5aa801a070 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.127+0000 7f5a9bfff640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f5aa00761a0 0x7f5aa00786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.127+0000 7f5ab37fe640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f5aa00761a0 0x7f5aa00786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.128+0000 7f5a9bfff640 1 -- 172.21.15.59:0/243296384 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f5aa805bfb0 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.128+0000 7f5ab37fe640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f5aa00761a0 0x7f5aa00786b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4033000 tx=0x7f5aa4004730 comp rx=0 tx=0).ready entity=mgr.12274 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.132+0000 7f5a9bfff640 1 -- 172.21.15.59:0/243296384 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5aa805fd20 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.181+0000 7f5a9bfff640 1 -- 172.21.15.59:0/243296384 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 150) ==== 98321+0+0 (secure 0 0 0) 0x7f5aa805ff80 con 0x7f5ab40b2fa0 2024-06-29T15:59:27.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.350+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 --> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f5ab41578d0 con 0x7f5aa00761a0 2024-06-29T15:59:27.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.416+0000 7f5a9bfff640 1 -- 172.21.15.59:0/243296384 <== mgr.12274 v2:172.21.15.153:6832/1744808854 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f5ab41578d0 con 0x7f5aa00761a0 2024-06-29T15:59:27.416 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path snap_test_dir1 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f5aa00761a0 msgr2=0x7f5aa00786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f5aa00761a0 0x7f5aa00786b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4033000 tx=0x7f5aa4004730 comp rx=0 tx=0).stop 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2fa0 msgr2=0x7f5ab4145a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2fa0 0x7f5ab4145a90 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800c3d0 tx=0x7f5aa800c890 comp rx=0 tx=0).stop 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 shutdown_connections 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f5aa00761a0 0x7f5aa00786b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ab40d1c30 0x7f5ab4145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ab40b2fa0 0x7f5ab4145a90 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.418+0000 7f5ab9d63640 1 --2- 172.21.15.59:0/243296384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab40b2270 0x7f5ab40b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.419+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 >> 172.21.15.59:0/243296384 conn(0x7f5ab4073e10 msgr2=0x7f5ab415b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.419+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 shutdown_connections 2024-06-29T15:59:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.419+0000 7f5ab9d63640 1 -- 172.21.15.59:0/243296384 wait complete. 2024-06-29T15:59:27.430 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T15:59:27.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:27.430 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:59:27.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:27.447 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T15:59:27.506 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:59:27.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.666+0000 7f7d2998c640 1 -- 172.21.15.59:0/3289721960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 msgr2=0x7f7d24164c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.666+0000 7f7d2998c640 1 --2- 172.21.15.59:0/3289721960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 0x7f7d24164c30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d14005fd0 tx=0x7f7d1402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.666+0000 7f7d2998c640 1 -- 172.21.15.59:0/3289721960 shutdown_connections 2024-06-29T15:59:27.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.666+0000 7f7d2998c640 1 --2- 172.21.15.59:0/3289721960 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 0x7f7d24164c30 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.666+0000 7f7d2998c640 1 --2- 172.21.15.59:0/3289721960 >> v2:172.21.15.153:3300/0 conn(0x7f7d2406d2f0 0x7f7d2406d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.666+0000 7f7d2998c640 1 -- 172.21.15.59:0/3289721960 >> 172.21.15.59:0/3289721960 conn(0x7f7d240b24c0 msgr2=0x7f7d240cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.666+0000 7f7d2998c640 1 -- 172.21.15.59:0/3289721960 shutdown_connections 2024-06-29T15:59:27.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.666+0000 7f7d2998c640 1 -- 172.21.15.59:0/3289721960 wait complete. 2024-06-29T15:59:27.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.668+0000 7f7d2998c640 1 Processor -- start 2024-06-29T15:59:27.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.668+0000 7f7d2998c640 1 -- start start 2024-06-29T15:59:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d2998c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d2406d2f0 0x7f7d24140400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d2998c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d240cd9b0 0x7f7d24158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d2998c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 0x7f7d2406c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d2998c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7d241448f0 con 0x7f7d2406d2f0 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d2998c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7d24144770 con 0x7f7d240cd9b0 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d2998c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7d24165170 con 0x7f7d24162810 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d22ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d2406d2f0 0x7f7d24140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d22ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d2406d2f0 0x7f7d24140400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44334/0 (socket says 172.21.15.59:44334) 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d22ffd640 1 -- 172.21.15.59:0/2590332725 learned_addr learned my addr 172.21.15.59:0/2590332725 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.669+0000 7f7d227fc640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d240cd9b0 0x7f7d24158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d237fe640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 0x7f7d2406c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d227fc640 1 -- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 msgr2=0x7f7d2406c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d227fc640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 0x7f7d2406c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d227fc640 1 -- 172.21.15.59:0/2590332725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d2406d2f0 msgr2=0x7f7d24140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d227fc640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d2406d2f0 0x7f7d24140400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d227fc640 1 -- 172.21.15.59:0/2590332725 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d24158a40 con 0x7f7d240cd9b0 2024-06-29T15:59:27.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d22ffd640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d2406d2f0 0x7f7d24140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:27.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d237fe640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 0x7f7d2406c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:27.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d227fc640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d240cd9b0 0x7f7d24158500 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00e9f0 tx=0x7f7d0c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:27.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d1bfff640 1 -- 172.21.15.59:0/2590332725 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c00ccc0 con 0x7f7d240cd9b0 2024-06-29T15:59:27.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d1bfff640 1 -- 172.21.15.59:0/2590332725 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d0c004020 con 0x7f7d240cd9b0 2024-06-29T15:59:27.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d24138e30 con 0x7f7d240cd9b0 2024-06-29T15:59:27.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d1bfff640 1 -- 172.21.15.59:0/2590332725 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c00ce60 con 0x7f7d240cd9b0 2024-06-29T15:59:27.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.670+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d24138590 con 0x7f7d240cd9b0 2024-06-29T15:59:27.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.672+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d2406d700 con 0x7f7d240cd9b0 2024-06-29T15:59:27.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.672+0000 7f7d1bfff640 1 -- 172.21.15.59:0/2590332725 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 150) ==== 98321+0+0 (secure 0 0 0) 0x7f7d0c0035f0 con 0x7f7d240cd9b0 2024-06-29T15:59:27.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.673+0000 7f7d1bfff640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f7d04076440 0x7f7d04078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.673+0000 7f7d1bfff640 1 -- 172.21.15.59:0/2590332725 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f7d0c098570 con 0x7f7d240cd9b0 2024-06-29T15:59:27.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.673+0000 7f7d22ffd640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f7d04076440 0x7f7d04078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.674+0000 7f7d22ffd640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f7d04076440 0x7f7d04078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c007fd0 tx=0x7f7d1c007f40 comp rx=0 tx=0).ready entity=mgr.12274 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:27.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.678+0000 7f7d1bfff640 1 -- 172.21.15.59:0/2590332725 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7d24158a40 con 0x7f7d240cd9b0 2024-06-29T15:59:27.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:27.917+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f7d24159c90 con 0x7f7d240cd9b0 2024-06-29T15:59:28.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.187+0000 7f7d1bfff640 1 -- 172.21.15.59:0/2590332725 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f7d0c009310 con 0x7f7d240cd9b0 2024-06-29T15:59:28.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.190+0000 7f7d1bfff640 1 -- 172.21.15.59:0/2590332725 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v151) ==== 93+0+0 (secure 0 0 0) 0x7f7d0c06e320 con 0x7f7d240cd9b0 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f7d04076440 msgr2=0x7f7d04078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f7d04076440 0x7f7d04078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c007fd0 tx=0x7f7d1c007f40 comp rx=0 tx=0).stop 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d240cd9b0 msgr2=0x7f7d24158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d240cd9b0 0x7f7d24158500 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00e9f0 tx=0x7f7d0c00eeb0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 shutdown_connections 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f7d04076440 0x7f7d04078950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7d24162810 0x7f7d2406c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d240cd9b0 0x7f7d24158500 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 --2- 172.21.15.59:0/2590332725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d2406d2f0 0x7f7d24140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 >> 172.21.15.59:0/2590332725 conn(0x7f7d240b24c0 msgr2=0x7f7d240b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 shutdown_connections 2024-06-29T15:59:28.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.193+0000 7f7d2998c640 1 -- 172.21.15.59:0/2590332725 wait complete. 2024-06-29T15:59:28.206 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:59:28.206 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:28.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:28.293 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:59:28.293 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:59:28.293 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:28.293 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:28.309 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:28.309 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:28.367 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:28.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.526+0000 7fdd3e963640 1 -- 172.21.15.59:0/1564363053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380d0130 msgr2=0x7fdd38143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.526+0000 7fdd3e963640 1 --2- 172.21.15.59:0/1564363053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380d0130 0x7fdd38143840 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009830 tx=0x7fdd2802a5c0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.527+0000 7fdd3e963640 1 -- 172.21.15.59:0/1564363053 shutdown_connections 2024-06-29T15:59:28.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.527+0000 7fdd3e963640 1 --2- 172.21.15.59:0/1564363053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380d0130 0x7fdd38143840 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.527+0000 7fdd3e963640 1 -- 172.21.15.59:0/1564363053 >> 172.21.15.59:0/1564363053 conn(0x7fdd380b2ab0 msgr2=0x7fdd380b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.527+0000 7fdd3e963640 1 -- 172.21.15.59:0/1564363053 shutdown_connections 2024-06-29T15:59:28.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.527+0000 7fdd3e963640 1 -- 172.21.15.59:0/1564363053 wait complete. 2024-06-29T15:59:28.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.528+0000 7fdd3e963640 1 Processor -- start 2024-06-29T15:59:28.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.529+0000 7fdd3e963640 1 -- start start 2024-06-29T15:59:28.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd3e963640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd3806ebe0 0x7fdd38158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd3e963640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380b4ee0 0x7fdd381f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd3e963640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd380d0130 0x7fdd381e9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd3e963640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdd380cfa30 con 0x7fdd3806ebe0 2024-06-29T15:59:28.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd3e963640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd380cf8b0 con 0x7fdd380b4ee0 2024-06-29T15:59:28.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd3e963640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd380cfbb0 con 0x7fdd380d0130 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd3806ebe0 0x7fdd38158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd3806ebe0 0x7fdd38158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44366/0 (socket says 172.21.15.59:44366) 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd37fff640 1 -- 172.21.15.59:0/340244972 learned_addr learned my addr 172.21.15.59:0/340244972 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd377fe640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380b4ee0 0x7fdd381f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.530+0000 7fdd3cf00640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd380d0130 0x7fdd381e9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd37fff640 1 -- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd380d0130 msgr2=0x7fdd381e9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd37fff640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd380d0130 0x7fdd381e9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd37fff640 1 -- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380b4ee0 msgr2=0x7fdd381f2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd37fff640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380b4ee0 0x7fdd381f2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd37fff640 1 -- 172.21.15.59:0/340244972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd381e8b80 con 0x7fdd3806ebe0 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd37fff640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd3806ebe0 0x7fdd38158d80 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009950 tx=0x7fdd28033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd34ff9640 1 -- 172.21.15.59:0/340244972 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28005c90 con 0x7fdd3806ebe0 2024-06-29T15:59:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd34ff9640 1 -- 172.21.15.59:0/340244972 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd28031e50 con 0x7fdd3806ebe0 2024-06-29T15:59:28.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd381e8e60 con 0x7fdd3806ebe0 2024-06-29T15:59:28.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd34ff9640 1 -- 172.21.15.59:0/340244972 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28005260 con 0x7fdd3806ebe0 2024-06-29T15:59:28.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.531+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd381e9420 con 0x7fdd3806ebe0 2024-06-29T15:59:28.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.532+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd38069ca0 con 0x7fdd3806ebe0 2024-06-29T15:59:28.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.533+0000 7fdd34ff9640 1 -- 172.21.15.59:0/340244972 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7fdd28005400 con 0x7fdd3806ebe0 2024-06-29T15:59:28.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.534+0000 7fdd34ff9640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fdd240763f0 0x7fdd24078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.534+0000 7fdd377fe640 1 -- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fdd240763f0 msgr2=0x7fdd24078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:28.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.534+0000 7fdd34ff9640 1 -- 172.21.15.59:0/340244972 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7fdd280b9160 con 0x7fdd3806ebe0 2024-06-29T15:59:28.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.534+0000 7fdd377fe640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fdd240763f0 0x7fdd24078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:28.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.542+0000 7fdd34ff9640 1 -- 172.21.15.59:0/340244972 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd38069ca0 con 0x7fdd3806ebe0 2024-06-29T15:59:28.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.734+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd38143840 con 0x7fdd3806ebe0 2024-06-29T15:59:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.735+0000 7fdd377fe640 1 -- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fdd240763f0 msgr2=0x7fdd24078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.735+0000 7fdd377fe640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fdd240763f0 0x7fdd24078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.736+0000 7fdd34ff9640 1 -- 172.21.15.59:0/340244972 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v263) ==== 74+0+17625 (secure 0 0 0) 0x7fdd2803f030 con 0x7fdd3806ebe0 2024-06-29T15:59:28.736 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:28.736 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":263,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:25.117344+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:11.169128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:11.760290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2546767818":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6833/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6832/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2003505066":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2250305531":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1734622549":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1337010763":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2869442511":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/3703209251":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2872771609":"2024-06-30T15:59:25.117274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:28.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fdd240763f0 msgr2=0x7fdd24078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:28.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fdd240763f0 0x7fdd24078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd3806ebe0 msgr2=0x7fdd38158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd3806ebe0 0x7fdd38158d80 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009950 tx=0x7fdd28033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:28.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 shutdown_connections 2024-06-29T15:59:28.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7fdd240763f0 0x7fdd24078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd380d0130 0x7fdd381e9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380b4ee0 0x7fdd381f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 --2- 172.21.15.59:0/340244972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd3806ebe0 0x7fdd38158d80 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 >> 172.21.15.59:0/340244972 conn(0x7fdd380b2ab0 msgr2=0x7fdd380cef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 shutdown_connections 2024-06-29T15:59:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.738+0000 7fdd3e963640 1 -- 172.21.15.59:0/340244972 wait complete. 2024-06-29T15:59:28.751 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:59:11.169128+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '261', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:59:11.760290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:28.752 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.873+0000 7f554001f640 1 -- 172.21.15.59:0/2173437826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2fa0 msgr2=0x7f55380d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.873+0000 7f554001f640 1 --2- 172.21.15.59:0/2173437826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2fa0 0x7f55380d1520 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f5538074c50 tx=0x7f552802a560 comp rx=0 tx=0).stop 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.874+0000 7f554001f640 1 -- 172.21.15.59:0/2173437826 shutdown_connections 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.874+0000 7f554001f640 1 --2- 172.21.15.59:0/2173437826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55380d1a60 0x7f55380cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.874+0000 7f554001f640 1 --2- 172.21.15.59:0/2173437826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2fa0 0x7f55380d1520 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.874+0000 7f554001f640 1 --2- 172.21.15.59:0/2173437826 >> v2:172.21.15.153:3300/0 conn(0x7f55380b2270 0x7f553815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.874+0000 7f554001f640 1 -- 172.21.15.59:0/2173437826 >> 172.21.15.59:0/2173437826 conn(0x7f5538073e10 msgr2=0x7f5538074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.874+0000 7f554001f640 1 -- 172.21.15.59:0/2173437826 shutdown_connections 2024-06-29T15:59:28.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.874+0000 7f554001f640 1 -- 172.21.15.59:0/2173437826 wait complete. 2024-06-29T15:59:28.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.875+0000 7f554001f640 1 Processor -- start 2024-06-29T15:59:28.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.876+0000 7f554001f640 1 -- start start 2024-06-29T15:59:28.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f554001f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2270 0x7f55381f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f554001f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55380b2fa0 0x7f55381523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f554001f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55380d1a60 0x7f5538153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f554001f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f55380cfe40 con 0x7f55380b2270 2024-06-29T15:59:28.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f554001f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55380cfcc0 con 0x7f55380b2fa0 2024-06-29T15:59:28.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f554001f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f55380cffc0 con 0x7f55380d1a60 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553ddbb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2270 0x7f55381f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553ddbb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2270 0x7f55381f33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44394/0 (socket says 172.21.15.59:44394) 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553ddbb640 1 -- 172.21.15.59:0/1970227274 learned_addr learned my addr 172.21.15.59:0/1970227274 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553d5ba640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55380b2fa0 0x7f55381523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553e5bc640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55380d1a60 0x7f5538153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553ddbb640 1 -- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55380d1a60 msgr2=0x7f5538153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553ddbb640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55380d1a60 0x7f5538153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553ddbb640 1 -- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55380b2fa0 msgr2=0x7f55381523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553ddbb640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55380b2fa0 0x7f55381523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553ddbb640 1 -- 172.21.15.59:0/1970227274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55381f38e0 con 0x7f55380b2270 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.877+0000 7f553d5ba640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55380b2fa0 0x7f55381523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.878+0000 7f553ddbb640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2270 0x7f55381f33a0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f553400e7c0 tx=0x7f553400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:28.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.878+0000 7f552e7fc640 1 -- 172.21.15.59:0/1970227274 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553400ccc0 con 0x7f55380b2270 2024-06-29T15:59:28.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.878+0000 7f552e7fc640 1 -- 172.21.15.59:0/1970227274 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5534010430 con 0x7f55380b2270 2024-06-29T15:59:28.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.878+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55380aa530 con 0x7f55380b2270 2024-06-29T15:59:28.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.878+0000 7f552e7fc640 1 -- 172.21.15.59:0/1970227274 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534010860 con 0x7f55380b2270 2024-06-29T15:59:28.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.878+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55380aaaf0 con 0x7f55380b2270 2024-06-29T15:59:28.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.879+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55380d0450 con 0x7f55380b2270 2024-06-29T15:59:28.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.880+0000 7f552e7fc640 1 -- 172.21.15.59:0/1970227274 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f5534010a00 con 0x7f55380b2270 2024-06-29T15:59:28.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.880+0000 7f552e7fc640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f551c076420 0x7f551c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.881+0000 7f553d5ba640 1 -- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f551c076420 msgr2=0x7f551c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:28.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.881+0000 7f552e7fc640 1 -- 172.21.15.59:0/1970227274 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f55340985f0 con 0x7f55380b2270 2024-06-29T15:59:28.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.881+0000 7f553d5ba640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f551c076420 0x7f551c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:28.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:28.885+0000 7f552e7fc640 1 -- 172.21.15.59:0/1970227274 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f55381f38e0 con 0x7f55380b2270 2024-06-29T15:59:29.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.076+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f55381557b0 con 0x7f55380b2270 2024-06-29T15:59:29.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.077+0000 7f552e7fc640 1 -- 172.21.15.59:0/1970227274 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v263) ==== 93+0+10 (secure 0 0 0) 0x7f5534061370 con 0x7f55380b2270 2024-06-29T15:59:29.077 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:59:29.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f551c076420 msgr2=0x7f551c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f551c076420 0x7f551c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2270 msgr2=0x7f55381f33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2270 0x7f55381f33a0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f553400e7c0 tx=0x7f553400ec80 comp rx=0 tx=0).stop 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 shutdown_connections 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f551c076420 0x7f551c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55380d1a60 0x7f5538153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55380b2fa0 0x7f55381523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 --2- 172.21.15.59:0/1970227274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55380b2270 0x7f55381f33a0 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 >> 172.21.15.59:0/1970227274 conn(0x7f5538073e10 msgr2=0x7f5538074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.079+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 shutdown_connections 2024-06-29T15:59:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.080+0000 7f554001f640 1 -- 172.21.15.59:0/1970227274 wait complete. 2024-06-29T15:59:29.091 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:59:29.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.217+0000 7f96e80ce640 1 -- 172.21.15.59:0/967183814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96e00b2fa0 msgr2=0x7f96e00d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.217+0000 7f96e80ce640 1 --2- 172.21.15.59:0/967183814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96e00b2fa0 0x7f96e00d1520 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f96dc009830 tx=0x7f96dc02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.217+0000 7f96e80ce640 1 -- 172.21.15.59:0/967183814 shutdown_connections 2024-06-29T15:59:29.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.217+0000 7f96e80ce640 1 --2- 172.21.15.59:0/967183814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96e00b2fa0 0x7f96e00d1520 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.217+0000 7f96e80ce640 1 -- 172.21.15.59:0/967183814 >> 172.21.15.59:0/967183814 conn(0x7f96e0073e10 msgr2=0x7f96e0074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.217+0000 7f96e80ce640 1 -- 172.21.15.59:0/967183814 shutdown_connections 2024-06-29T15:59:29.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.217+0000 7f96e80ce640 1 -- 172.21.15.59:0/967183814 wait complete. 2024-06-29T15:59:29.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.219+0000 7f96e80ce640 1 Processor -- start 2024-06-29T15:59:29.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.219+0000 7f96e80ce640 1 -- start start 2024-06-29T15:59:29.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e80ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96e00b2270 0x7f96e00ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e80ce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96e00b2fa0 0x7f96e00b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e80ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96e00d1a60 0x7f96e0156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e80ce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00b65d0 con 0x7f96e00b2270 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e80ce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00b6450 con 0x7f96e00d1a60 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e80ce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f96e00b6750 con 0x7f96e00b2fa0 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e5e6a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96e00b2270 0x7f96e00ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e5e6a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96e00b2270 0x7f96e00ac030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44408/0 (socket says 172.21.15.59:44408) 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e5e6a640 1 -- 172.21.15.59:0/3481638968 learned_addr learned my addr 172.21.15.59:0/3481638968 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e5669640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96e00b2fa0 0x7f96e00b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.220+0000 7f96e666b640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96e00d1a60 0x7f96e0156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96e5e6a640 1 -- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96e00b2fa0 msgr2=0x7f96e00b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96e5e6a640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96e00b2fa0 0x7f96e00b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96e5e6a640 1 -- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96e00d1a60 msgr2=0x7f96e0156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96e5e6a640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96e00d1a60 0x7f96e0156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96e5e6a640 1 -- 172.21.15.59:0/3481638968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e01f5fb0 con 0x7f96e00b2270 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96e5e6a640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96e00b2270 0x7f96e00ac030 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f96d000c460 tx=0x7f96d000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96d67fc640 1 -- 172.21.15.59:0/3481638968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0016020 con 0x7f96e00b2270 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96d67fc640 1 -- 172.21.15.59:0/3481638968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96d0012a00 con 0x7f96e00b2270 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e01f6290 con 0x7f96e00b2270 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96d67fc640 1 -- 172.21.15.59:0/3481638968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0012e50 con 0x7f96e00b2270 2024-06-29T15:59:29.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.221+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e01f6850 con 0x7f96e00b2270 2024-06-29T15:59:29.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.222+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e00b6c70 con 0x7f96e00b2270 2024-06-29T15:59:29.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.223+0000 7f96d67fc640 1 -- 172.21.15.59:0/3481638968 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f96d0019020 con 0x7f96e00b2270 2024-06-29T15:59:29.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.224+0000 7f96d67fc640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f96c40765c0 0x7f96c4078ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.224+0000 7f96e5669640 1 -- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f96c40765c0 msgr2=0x7f96c4078ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:29.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.224+0000 7f96d67fc640 1 -- 172.21.15.59:0/3481638968 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f96d00977a0 con 0x7f96e00b2270 2024-06-29T15:59:29.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.224+0000 7f96e5669640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f96c40765c0 0x7f96c4078ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:29.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.228+0000 7f96d67fc640 1 -- 172.21.15.59:0/3481638968 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f96d0020030 con 0x7f96e00b2270 2024-06-29T15:59:29.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.423+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f96e00ceb80 con 0x7f96e00b2270 2024-06-29T15:59:29.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.423+0000 7f96d67fc640 1 -- 172.21.15.59:0/3481638968 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v263) ==== 104+0+11 (secure 0 0 0) 0x7f96d0060630 con 0x7f96e00b2270 2024-06-29T15:59:29.423 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:59:29.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f96c40765c0 msgr2=0x7f96c4078ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:29.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f96c40765c0 0x7f96c4078ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96e00b2270 msgr2=0x7f96e00ac030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96e00b2270 0x7f96e00ac030 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f96d000c460 tx=0x7f96d000c920 comp rx=0 tx=0).stop 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 shutdown_connections 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f96c40765c0 0x7f96c4078ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96e00d1a60 0x7f96e0156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f96e00b2fa0 0x7f96e00b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 --2- 172.21.15.59:0/3481638968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96e00b2270 0x7f96e00ac030 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.425+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 >> 172.21.15.59:0/3481638968 conn(0x7f96e0073e10 msgr2=0x7f96e0155420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.426+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 shutdown_connections 2024-06-29T15:59:29.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.426+0000 7f96e80ce640 1 -- 172.21.15.59:0/3481638968 wait complete. 2024-06-29T15:59:29.438 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:59:29.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.562+0000 7f4de1cce640 1 -- 172.21.15.59:0/11664082 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ddc15efe0 msgr2=0x7f4ddc15f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.562+0000 7f4de1cce640 1 --2- 172.21.15.59:0/11664082 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ddc15efe0 0x7f4ddc15f9b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4005fd0 tx=0x7f4dd402a840 comp rx=0 tx=0).stop 2024-06-29T15:59:29.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.562+0000 7f4de1cce640 1 -- 172.21.15.59:0/11664082 shutdown_connections 2024-06-29T15:59:29.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.562+0000 7f4de1cce640 1 --2- 172.21.15.59:0/11664082 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ddc15efe0 0x7f4ddc15f9b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.562+0000 7f4de1cce640 1 --2- 172.21.15.59:0/11664082 >> v2:172.21.15.153:3300/0 conn(0x7f4ddc0d23c0 0x7f4ddc0b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.562+0000 7f4de1cce640 1 -- 172.21.15.59:0/11664082 >> 172.21.15.59:0/11664082 conn(0x7f4ddc0d3ee0 msgr2=0x7f4ddc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.562+0000 7f4de1cce640 1 -- 172.21.15.59:0/11664082 shutdown_connections 2024-06-29T15:59:29.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.562+0000 7f4de1cce640 1 -- 172.21.15.59:0/11664082 wait complete. 2024-06-29T15:59:29.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.564+0000 7f4de1cce640 1 Processor -- start 2024-06-29T15:59:29.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.564+0000 7f4de1cce640 1 -- start start 2024-06-29T15:59:29.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4de1cce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ddc0b3ed0 0x7f4ddc1eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4de1cce640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ddc0d23c0 0x7f4ddc1f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4de1cce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc15efe0 0x7f4ddc15c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4de1cce640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4ddc160280 con 0x7f4ddc15efe0 2024-06-29T15:59:29.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4de1cce640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ddc160100 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4de1cce640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4ddc160400 con 0x7f4ddc0d23c0 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4ddbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc15efe0 0x7f4ddc15c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4ddbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc15efe0 0x7f4ddc15c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44416/0 (socket says 172.21.15.59:44416) 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4ddbfff640 1 -- 172.21.15.59:0/1368529894 learned_addr learned my addr 172.21.15.59:0/1368529894 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4ddb7fe640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ddc0b3ed0 0x7f4ddc1eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.565+0000 7f4ddaffd640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ddc0d23c0 0x7f4ddc1f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4ddb7fe640 1 -- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ddc0d23c0 msgr2=0x7f4ddc1f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4ddb7fe640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ddc0d23c0 0x7f4ddc1f59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4ddb7fe640 1 -- 172.21.15.59:0/1368529894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc15efe0 msgr2=0x7f4ddc15c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4ddb7fe640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc15efe0 0x7f4ddc15c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4ddb7fe640 1 -- 172.21.15.59:0/1368529894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ddc1f6750 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4ddbfff640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc15efe0 0x7f4ddc15c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4ddb7fe640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ddc0b3ed0 0x7f4ddc1eabc0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00e9f0 tx=0x7f4dcc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4dc3fff640 1 -- 172.21.15.59:0/1368529894 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00ccc0 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4dc3fff640 1 -- 172.21.15.59:0/1368529894 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4dcc00ce60 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ddc1f8ed0 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4dc3fff640 1 -- 172.21.15.59:0/1368529894 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc010690 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.566+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ddc1f9490 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.567+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ddc0b3990 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.568+0000 7f4dc3fff640 1 -- 172.21.15.59:0/1368529894 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f4dcc010830 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.568+0000 7f4dc3fff640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f4dc8076420 0x7f4dc8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.569+0000 7f4ddaffd640 1 -- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f4dc8076420 msgr2=0x7f4dc8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.569+0000 7f4dc3fff640 1 -- 172.21.15.59:0/1368529894 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f4dcc098460 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.569+0000 7f4ddaffd640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f4dc8076420 0x7f4dc8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:29.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.573+0000 7f4dc3fff640 1 -- 172.21.15.59:0/1368529894 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4dcc0611e0 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.767+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4ddc0d2950 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.768+0000 7f4dc3fff640 1 -- 172.21.15.59:0/1368529894 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v263) ==== 100+0+11 (secure 0 0 0) 0x7f4dcc0654e0 con 0x7f4ddc0b3ed0 2024-06-29T15:59:29.768 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:59:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4ddaffd640 1 -- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f4dc8076420 msgr2=0x7f4dc8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4ddaffd640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f4dc8076420 0x7f4dc8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f4dc8076420 msgr2=0x7f4dc8078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f4dc8076420 0x7f4dc8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ddc0b3ed0 msgr2=0x7f4ddc1eabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ddc0b3ed0 0x7f4ddc1eabc0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00e9f0 tx=0x7f4dcc00eeb0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 shutdown_connections 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f4dc8076420 0x7f4dc8078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc15efe0 0x7f4ddc15c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ddc0d23c0 0x7f4ddc1f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 --2- 172.21.15.59:0/1368529894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ddc0b3ed0 0x7f4ddc1eabc0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 >> 172.21.15.59:0/1368529894 conn(0x7f4ddc0d3ee0 msgr2=0x7f4ddc0d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 shutdown_connections 2024-06-29T15:59:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.770+0000 7f4de1cce640 1 -- 172.21.15.59:0/1368529894 wait complete. 2024-06-29T15:59:29.782 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:59:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.903+0000 7f2dca290640 1 -- 172.21.15.59:0/1655860868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40b2fa0 msgr2=0x7f2dc40d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.903+0000 7f2dca290640 1 --2- 172.21.15.59:0/1655860868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40b2fa0 0x7f2dc40d1520 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f2dc40cf310 tx=0x7f2db802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.903+0000 7f2dca290640 1 -- 172.21.15.59:0/1655860868 shutdown_connections 2024-06-29T15:59:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.903+0000 7f2dca290640 1 --2- 172.21.15.59:0/1655860868 >> v2:172.21.15.153:3300/0 conn(0x7f2dc40d1a60 0x7f2dc40b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.903+0000 7f2dca290640 1 --2- 172.21.15.59:0/1655860868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40b2fa0 0x7f2dc40d1520 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.903+0000 7f2dca290640 1 -- 172.21.15.59:0/1655860868 >> 172.21.15.59:0/1655860868 conn(0x7f2dc4073e10 msgr2=0x7f2dc4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.903+0000 7f2dca290640 1 -- 172.21.15.59:0/1655860868 shutdown_connections 2024-06-29T15:59:29.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.903+0000 7f2dca290640 1 -- 172.21.15.59:0/1655860868 wait complete. 2024-06-29T15:59:29.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.905+0000 7f2dca290640 1 Processor -- start 2024-06-29T15:59:29.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.905+0000 7f2dca290640 1 -- start start 2024-06-29T15:59:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dca290640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dc40b2270 0x7f2dc4144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dca290640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dc40b2fa0 0x7f2dc4146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dca290640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40d1a60 0x7f2dc4146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dca290640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc40b6400 con 0x7f2dc40d1a60 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dca290640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc40b6280 con 0x7f2dc40b2270 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dca290640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc40b6580 con 0x7f2dc40b2fa0 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dc882d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40d1a60 0x7f2dc4146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dc882d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40d1a60 0x7f2dc4146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44428/0 (socket says 172.21.15.59:44428) 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dc882d640 1 -- 172.21.15.59:0/207904073 learned_addr learned my addr 172.21.15.59:0/207904073 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dc3fff640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dc40b2270 0x7f2dc4144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.906+0000 7f2dc37fe640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dc40b2fa0 0x7f2dc4146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc882d640 1 -- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dc40b2fa0 msgr2=0x7f2dc4146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc882d640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dc40b2fa0 0x7f2dc4146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc882d640 1 -- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dc40b2270 msgr2=0x7f2dc4144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc882d640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dc40b2270 0x7f2dc4144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc882d640 1 -- 172.21.15.59:0/207904073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc41453b0 con 0x7f2dc40d1a60 2024-06-29T15:59:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc37fe640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dc40b2fa0 0x7f2dc4146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc882d640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40d1a60 0x7f2dc4146990 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00ea80 tx=0x7f2dbc00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc0ff9640 1 -- 172.21.15.59:0/207904073 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc004b20 con 0x7f2dc40d1a60 2024-06-29T15:59:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc0ff9640 1 -- 172.21.15.59:0/207904073 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2dbc004ce0 con 0x7f2dc40d1a60 2024-06-29T15:59:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc4145690 con 0x7f2dc40d1a60 2024-06-29T15:59:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dc0ff9640 1 -- 172.21.15.59:0/207904073 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc005280 con 0x7f2dc40d1a60 2024-06-29T15:59:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.907+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc4145c50 con 0x7f2dc40d1a60 2024-06-29T15:59:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.908+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc4145120 con 0x7f2dc40d1a60 2024-06-29T15:59:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.909+0000 7f2dc0ff9640 1 -- 172.21.15.59:0/207904073 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f2dbc005420 con 0x7f2dc40d1a60 2024-06-29T15:59:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.909+0000 7f2dc0ff9640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f2db0076420 0x7f2db0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.909+0000 7f2dc3fff640 1 -- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f2db0076420 msgr2=0x7f2db0078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.910+0000 7f2dc0ff9640 1 -- 172.21.15.59:0/207904073 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f2dbc098a60 con 0x7f2dc40d1a60 2024-06-29T15:59:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.910+0000 7f2dc3fff640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f2db0076420 0x7f2db0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:29.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:29.918+0000 7f2dc0ff9640 1 -- 172.21.15.59:0/207904073 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2dbc061c80 con 0x7f2dc40d1a60 2024-06-29T15:59:30.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.111+0000 7f2dc3fff640 1 -- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f2db0076420 msgr2=0x7f2db0078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:30.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.111+0000 7f2dc3fff640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f2db0076420 0x7f2db0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:30.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.123+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2dc40b6aa0 con 0x7f2dc40d1a60 2024-06-29T15:59:30.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.123+0000 7f2dc0ff9640 1 -- 172.21.15.59:0/207904073 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v263) ==== 81+0+596 (secure 0 0 0) 0x7f2dbc061580 con 0x7f2dc40d1a60 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2546767818 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/3574471609 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/3574471609 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2003505066 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2250305531 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1734622549 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1337010763 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2869442511 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3703209251 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.124 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2872771609 2024-06-30T15:59:25.117274+0000 2024-06-29T15:59:30.125 INFO:teuthology.orchestra.run.smithi059.stderr:listed 10 entries 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f2db0076420 msgr2=0x7f2db0078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f2db0076420 0x7f2db0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40d1a60 msgr2=0x7f2dc4146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40d1a60 0x7f2dc4146990 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00ea80 tx=0x7f2dbc00ef40 comp rx=0 tx=0).stop 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 shutdown_connections 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f2db0076420 0x7f2db0078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dc40d1a60 0x7f2dc4146990 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2dc40b2fa0 0x7f2dc4146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 --2- 172.21.15.59:0/207904073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dc40b2270 0x7f2dc4144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 >> 172.21.15.59:0/207904073 conn(0x7f2dc4073e10 msgr2=0x7f2dc415b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:30.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 shutdown_connections 2024-06-29T15:59:30.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.126+0000 7f2dca290640 1 -- 172.21.15.59:0/207904073 wait complete. 2024-06-29T15:59:30.138 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:59:30.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:30.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:30.168 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T15:59:30.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:30.169 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:59:30.225 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:59:30.225 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:59:30.225 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:30.242 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:30.242 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:59:30.242 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:30.242 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:30.282 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:30.282 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:59:30.282 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:59:30.282 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:30.298 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:30.298 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:30.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.440+0000 7f8d7b970640 1 -- 172.21.15.59:0/607974774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d740b40a0 msgr2=0x7f8d7415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.440+0000 7f8d7b970640 1 --2- 172.21.15.59:0/607974774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d740b40a0 0x7f8d7415e8d0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f8d64005fd0 tx=0x7f8d6402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.440+0000 7f8d7b970640 1 -- 172.21.15.59:0/607974774 shutdown_connections 2024-06-29T15:59:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.440+0000 7f8d7b970640 1 --2- 172.21.15.59:0/607974774 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d7415ee10 0x7f8d7415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.440+0000 7f8d7b970640 1 --2- 172.21.15.59:0/607974774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d740b40a0 0x7f8d7415e8d0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.440+0000 7f8d7b970640 1 -- 172.21.15.59:0/607974774 >> 172.21.15.59:0/607974774 conn(0x7f8d740d3ee0 msgr2=0x7f8d740d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.440+0000 7f8d7b970640 1 -- 172.21.15.59:0/607974774 shutdown_connections 2024-06-29T15:59:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.440+0000 7f8d7b970640 1 -- 172.21.15.59:0/607974774 wait complete. 2024-06-29T15:59:30.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.442+0000 7f8d7b970640 1 Processor -- start 2024-06-29T15:59:30.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.442+0000 7f8d7b970640 1 -- start start 2024-06-29T15:59:30.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d7b970640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d740d23c0 0x7f8d741f8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d7b970640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7415ee10 0x7f8d741479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d7b970640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d741608d0 0x7f8d74160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d7b970640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74160280 con 0x7f8d741608d0 2024-06-29T15:59:30.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d7b970640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8d74160100 con 0x7f8d7415ee10 2024-06-29T15:59:30.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d7b970640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8d74160400 con 0x7f8d740d23c0 2024-06-29T15:59:30.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d79f0d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d741608d0 0x7f8d74160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d7970c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d740d23c0 0x7f8d741f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d78f0b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7415ee10 0x7f8d741479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d79f0d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d741608d0 0x7f8d74160ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44442/0 (socket says 172.21.15.59:44442) 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d79f0d640 1 -- 172.21.15.59:0/3566203008 learned_addr learned my addr 172.21.15.59:0/3566203008 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.444+0000 7f8d79f0d640 1 -- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d740d23c0 msgr2=0x7f8d741f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d79f0d640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d740d23c0 0x7f8d741f8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d79f0d640 1 -- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7415ee10 msgr2=0x7f8d741479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d79f0d640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7415ee10 0x7f8d741479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d79f0d640 1 -- 172.21.15.59:0/3566203008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d740ca0b0 con 0x7f8d741608d0 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d78f0b640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7415ee10 0x7f8d741479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d79f0d640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d741608d0 0x7f8d74160ce0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00ea70 tx=0x7f8d6c00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d69ffb640 1 -- 172.21.15.59:0/3566203008 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c00cde0 con 0x7f8d741608d0 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d74161220 con 0x7f8d741608d0 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d69ffb640 1 -- 172.21.15.59:0/3566203008 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d6c004c10 con 0x7f8d741608d0 2024-06-29T15:59:30.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d69ffb640 1 -- 172.21.15.59:0/3566203008 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c005020 con 0x7f8d741608d0 2024-06-29T15:59:30.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.445+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d7414abe0 con 0x7f8d741608d0 2024-06-29T15:59:30.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.446+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d7414a670 con 0x7f8d741608d0 2024-06-29T15:59:30.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.447+0000 7f8d69ffb640 1 -- 172.21.15.59:0/3566203008 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f8d6c0051c0 con 0x7f8d741608d0 2024-06-29T15:59:30.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.448+0000 7f8d69ffb640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8d580763f0 0x7f8d58078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.448+0000 7f8d7970c640 1 -- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8d580763f0 msgr2=0x7f8d58078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:30.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.448+0000 7f8d69ffb640 1 -- 172.21.15.59:0/3566203008 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f8d6c0988c0 con 0x7f8d741608d0 2024-06-29T15:59:30.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.448+0000 7f8d7970c640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8d580763f0 0x7f8d58078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:30.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.457+0000 7f8d69ffb640 1 -- 172.21.15.59:0/3566203008 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8d74161220 con 0x7f8d741608d0 2024-06-29T15:59:30.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.649+0000 7f8d7970c640 1 -- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8d580763f0 msgr2=0x7f8d58078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:30.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.649+0000 7f8d7970c640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8d580763f0 0x7f8d58078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:30.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.689+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d7415e8d0 con 0x7f8d741608d0 2024-06-29T15:59:30.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.689+0000 7f8d69ffb640 1 -- 172.21.15.59:0/3566203008 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5247 (secure 0 0 0) 0x7f8d6c061650 con 0x7f8d741608d0 2024-06-29T15:59:30.689 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:30.689 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":148,"btime":"2024-06-29T15:59:16:455546+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1010984928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1010984928},{"type":"v1","addr":"172.21.15.153:6836","nonce":1010984928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":10475,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2101409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2101409892},{"type":"v1","addr":"172.21.15.153:6837","nonce":2101409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":12130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1208243017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1208243017},{"type":"v1","addr":"172.21.15.59:6835","nonce":1208243017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:12.808119+0000","modified":"2024-06-29T15:59:15.847810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13110":{"gid":13110,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/2985893835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2985893835},{"type":"v1","addr":"172.21.15.59:6833","nonce":2985893835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13110,"qdb_cluster":[13110]},"id":17}]} 2024-06-29T15:59:30.690 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 148 2024-06-29T15:59:30.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8d580763f0 msgr2=0x7f8d58078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8d580763f0 0x7f8d58078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d741608d0 msgr2=0x7f8d74160ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d741608d0 0x7f8d74160ce0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00ea70 tx=0x7f8d6c00ef30 comp rx=0 tx=0).stop 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 shutdown_connections 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f8d580763f0 0x7f8d58078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d741608d0 0x7f8d74160ce0 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d7415ee10 0x7f8d741479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.691+0000 7f8d7b970640 1 --2- 172.21.15.59:0/3566203008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8d740d23c0 0x7f8d741f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.692+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 >> 172.21.15.59:0/3566203008 conn(0x7f8d740d3ee0 msgr2=0x7f8d740b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:30.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.692+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 shutdown_connections 2024-06-29T15:59:30.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.692+0000 7f8d7b970640 1 -- 172.21.15.59:0/3566203008 wait complete. 2024-06-29T15:59:30.704 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:30.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.823+0000 7f181619e640 1 -- 172.21.15.59:0/3408646565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100d23c0 msgr2=0x7f18100b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.823+0000 7f181619e640 1 --2- 172.21.15.59:0/3408646565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100d23c0 0x7f18100b3990 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f1800005fd0 tx=0x7f180002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.823+0000 7f181619e640 1 -- 172.21.15.59:0/3408646565 shutdown_connections 2024-06-29T15:59:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.823+0000 7f181619e640 1 --2- 172.21.15.59:0/3408646565 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18100b3ed0 0x7f181015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.823+0000 7f181619e640 1 --2- 172.21.15.59:0/3408646565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100d23c0 0x7f18100b3990 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.823+0000 7f181619e640 1 -- 172.21.15.59:0/3408646565 >> 172.21.15.59:0/3408646565 conn(0x7f18100d3ee0 msgr2=0x7f18100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.823+0000 7f181619e640 1 -- 172.21.15.59:0/3408646565 shutdown_connections 2024-06-29T15:59:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.823+0000 7f181619e640 1 -- 172.21.15.59:0/3408646565 wait complete. 2024-06-29T15:59:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.825+0000 7f181619e640 1 Processor -- start 2024-06-29T15:59:30.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.825+0000 7f181619e640 1 -- start start 2024-06-29T15:59:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.826+0000 7f181619e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100b3ed0 0x7f18101f8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.826+0000 7f181619e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18100d23c0 0x7f1810154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.826+0000 7f181619e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181015ee10 0x7f18101f8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.826+0000 7f181619e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f1810160240 con 0x7f18100b3ed0 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.826+0000 7f181619e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f18101600c0 con 0x7f18100d23c0 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.826+0000 7f181619e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f18101603c0 con 0x7f181015ee10 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.826+0000 7f180f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100b3ed0 0x7f18101f8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100b3ed0 0x7f18101f8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44460/0 (socket says 172.21.15.59:44460) 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180f7fe640 1 -- 172.21.15.59:0/3616705478 learned_addr learned my addr 172.21.15.59:0/3616705478 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180ffff640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181015ee10 0x7f18101f8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180effd640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18100d23c0 0x7f1810154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180f7fe640 1 -- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181015ee10 msgr2=0x7f18101f8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180f7fe640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181015ee10 0x7f18101f8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180f7fe640 1 -- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18100d23c0 msgr2=0x7f1810154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180f7fe640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18100d23c0 0x7f1810154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180f7fe640 1 -- 172.21.15.59:0/3616705478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1810155290 con 0x7f18100b3ed0 2024-06-29T15:59:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180effd640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18100d23c0 0x7f1810154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f180f7fe640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100b3ed0 0x7f18101f8420 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f1800034040 tx=0x7f18000035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f1807fff640 1 -- 172.21.15.59:0/3616705478 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180003e030 con 0x7f18100b3ed0 2024-06-29T15:59:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f1807fff640 1 -- 172.21.15.59:0/3616705478 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1800003ae0 con 0x7f18100b3ed0 2024-06-29T15:59:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1810155570 con 0x7f18100b3ed0 2024-06-29T15:59:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f1807fff640 1 -- 172.21.15.59:0/3616705478 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800035040 con 0x7f18100b3ed0 2024-06-29T15:59:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.827+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1810155b30 con 0x7f18100b3ed0 2024-06-29T15:59:30.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.829+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f181015e8d0 con 0x7f18100b3ed0 2024-06-29T15:59:30.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.829+0000 7f1807fff640 1 -- 172.21.15.59:0/3616705478 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f1800041030 con 0x7f18100b3ed0 2024-06-29T15:59:30.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.830+0000 7f1807fff640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f17f00763f0 0x7f17f0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.830+0000 7f1807fff640 1 -- 172.21.15.59:0/3616705478 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f18000b81e0 con 0x7f18100b3ed0 2024-06-29T15:59:30.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.830+0000 7f180effd640 1 -- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f17f00763f0 msgr2=0x7f17f0078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:30.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.830+0000 7f180effd640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f17f00763f0 0x7f17f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:30.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:30.839+0000 7f1807fff640 1 -- 172.21.15.59:0/3616705478 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1800080e50 con 0x7f18100b3ed0 2024-06-29T15:59:31.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.031+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18100d2950 con 0x7f18100b3ed0 2024-06-29T15:59:31.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.031+0000 7f180effd640 1 -- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f17f00763f0 msgr2=0x7f17f0078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:31.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.031+0000 7f180effd640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f17f00763f0 0x7f17f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:31.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.033+0000 7f1807fff640 1 -- 172.21.15.59:0/3616705478 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v263) ==== 74+0+17625 (secure 0 0 0) 0x7f1800085150 con 0x7f18100b3ed0 2024-06-29T15:59:31.033 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:31.033 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":263,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:25.117344+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:11.169128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:11.760290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2546767818":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6833/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6832/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2003505066":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2250305531":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1734622549":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1337010763":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2869442511":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/3703209251":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2872771609":"2024-06-30T15:59:25.117274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:31.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f17f00763f0 msgr2=0x7f17f0078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:31.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f17f00763f0 0x7f17f0078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100b3ed0 msgr2=0x7f18101f8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100b3ed0 0x7f18101f8420 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f1800034040 tx=0x7f18000035f0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 shutdown_connections 2024-06-29T15:59:31.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f17f00763f0 0x7f17f0078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f181015ee10 0x7f18101f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18100d23c0 0x7f1810154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 --2- 172.21.15.59:0/3616705478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18100b3ed0 0x7f18101f8420 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 >> 172.21.15.59:0/3616705478 conn(0x7f18100d3ee0 msgr2=0x7f181006c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:31.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 shutdown_connections 2024-06-29T15:59:31.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.035+0000 7f181619e640 1 -- 172.21.15.59:0/3616705478 wait complete. 2024-06-29T15:59:31.047 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:59:11.169128+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '261', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:59:11.760290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:31.047 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:31.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.170+0000 7f18c67d2640 1 -- 172.21.15.59:0/580654268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 msgr2=0x7f18c006e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.170+0000 7f18c67d2640 1 --2- 172.21.15.59:0/580654268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 0x7f18c006e4d0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f18c0144290 tx=0x7f18b402a560 comp rx=0 tx=0).stop 2024-06-29T15:59:31.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.170+0000 7f18c67d2640 1 -- 172.21.15.59:0/580654268 shutdown_connections 2024-06-29T15:59:31.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.170+0000 7f18c67d2640 1 --2- 172.21.15.59:0/580654268 >> v2:172.21.15.153:3300/0 conn(0x7f18c006ea10 0x7f18c00cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.170+0000 7f18c67d2640 1 --2- 172.21.15.59:0/580654268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 0x7f18c006e4d0 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.170+0000 7f18c67d2640 1 -- 172.21.15.59:0/580654268 >> 172.21.15.59:0/580654268 conn(0x7f18c00b2ab0 msgr2=0x7f18c00b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.170+0000 7f18c67d2640 1 -- 172.21.15.59:0/580654268 shutdown_connections 2024-06-29T15:59:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.170+0000 7f18c67d2640 1 -- 172.21.15.59:0/580654268 wait complete. 2024-06-29T15:59:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.172+0000 7f18c67d2640 1 Processor -- start 2024-06-29T15:59:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.172+0000 7f18c67d2640 1 -- start start 2024-06-29T15:59:31.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c67d2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c006ea10 0x7f18c0154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c67d2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c00d0130 0x7f18c013c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c67d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 0x7f18c015d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c67d2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f18c00cf650 con 0x7f18c0143840 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c67d2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f18c00cf4d0 con 0x7f18c00d0130 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c67d2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f18c00cf7d0 con 0x7f18c006ea10 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c4d6f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 0x7f18c015d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c4d6f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 0x7f18c015d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44486/0 (socket says 172.21.15.59:44486) 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18c4d6f640 1 -- 172.21.15.59:0/2028089212 learned_addr learned my addr 172.21.15.59:0/2028089212 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18bf7fe640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c00d0130 0x7f18c013c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.173+0000 7f18bffff640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c006ea10 0x7f18c0154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18c4d6f640 1 -- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c006ea10 msgr2=0x7f18c0154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18c4d6f640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c006ea10 0x7f18c0154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18c4d6f640 1 -- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c00d0130 msgr2=0x7f18c013c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18c4d6f640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c00d0130 0x7f18c013c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18c4d6f640 1 -- 172.21.15.59:0/2028089212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c013d0d0 con 0x7f18c0143840 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18c4d6f640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 0x7f18c015d300 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f18b800ea70 tx=0x7f18b800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18bcff9640 1 -- 172.21.15.59:0/2028089212 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b800ce40 con 0x7f18c0143840 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18bcff9640 1 -- 172.21.15.59:0/2028089212 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18b8004c10 con 0x7f18c0143840 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c013d3b0 con 0x7f18c0143840 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18bcff9640 1 -- 172.21.15.59:0/2028089212 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8005020 con 0x7f18c0143840 2024-06-29T15:59:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.174+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c0155120 con 0x7f18c0143840 2024-06-29T15:59:31.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.175+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c00cfbf0 con 0x7f18c0143840 2024-06-29T15:59:31.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.176+0000 7f18bcff9640 1 -- 172.21.15.59:0/2028089212 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f18b80051c0 con 0x7f18c0143840 2024-06-29T15:59:31.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.177+0000 7f18bcff9640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f18ac0763f0 0x7f18ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.177+0000 7f18bffff640 1 -- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f18ac0763f0 msgr2=0x7f18ac078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:31.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.177+0000 7f18bcff9640 1 -- 172.21.15.59:0/2028089212 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f18b8098a90 con 0x7f18c0143840 2024-06-29T15:59:31.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.177+0000 7f18bffff640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f18ac0763f0 0x7f18ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:31.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.181+0000 7f18bcff9640 1 -- 172.21.15.59:0/2028089212 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f18b80616e0 con 0x7f18c0143840 2024-06-29T15:59:31.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.372+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18c006e6e0 con 0x7f18c0143840 2024-06-29T15:59:31.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.372+0000 7f18bcff9640 1 -- 172.21.15.59:0/2028089212 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v263) ==== 93+0+10 (secure 0 0 0) 0x7f18b80659e0 con 0x7f18c0143840 2024-06-29T15:59:31.373 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:59:31.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f18ac0763f0 msgr2=0x7f18ac078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:31.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f18ac0763f0 0x7f18ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 msgr2=0x7f18c015d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 0x7f18c015d300 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f18b800ea70 tx=0x7f18b800ef30 comp rx=0 tx=0).stop 2024-06-29T15:59:31.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 shutdown_connections 2024-06-29T15:59:31.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f18ac0763f0 0x7f18ac078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18c0143840 0x7f18c015d300 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c00d0130 0x7f18c013c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 --2- 172.21.15.59:0/2028089212 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c006ea10 0x7f18c0154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.374+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 >> 172.21.15.59:0/2028089212 conn(0x7f18c00b2ab0 msgr2=0x7f18c00b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:31.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.375+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 shutdown_connections 2024-06-29T15:59:31.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.375+0000 7f18c67d2640 1 -- 172.21.15.59:0/2028089212 wait complete. 2024-06-29T15:59:31.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:59:31.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.509+0000 7f5a17f94640 1 -- 172.21.15.59:0/2195737781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1015ee10 msgr2=0x7f5a1015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.509+0000 7f5a17f94640 1 --2- 172.21.15.59:0/2195737781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1015ee10 0x7f5a1015f9b0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f5a08005fd0 tx=0x7f5a0802a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.509+0000 7f5a17f94640 1 -- 172.21.15.59:0/2195737781 shutdown_connections 2024-06-29T15:59:31.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.509+0000 7f5a17f94640 1 --2- 172.21.15.59:0/2195737781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1015ee10 0x7f5a1015f9b0 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.509+0000 7f5a17f94640 1 --2- 172.21.15.59:0/2195737781 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a100b40a0 0x7f5a1015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.509+0000 7f5a17f94640 1 -- 172.21.15.59:0/2195737781 >> 172.21.15.59:0/2195737781 conn(0x7f5a100d3ee0 msgr2=0x7f5a100d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:31.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.509+0000 7f5a17f94640 1 -- 172.21.15.59:0/2195737781 shutdown_connections 2024-06-29T15:59:31.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.509+0000 7f5a17f94640 1 -- 172.21.15.59:0/2195737781 wait complete. 2024-06-29T15:59:31.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.511+0000 7f5a17f94640 1 Processor -- start 2024-06-29T15:59:31.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.511+0000 7f5a17f94640 1 -- start start 2024-06-29T15:59:31.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a17f94640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a100d23c0 0x7f5a101ea170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a17f94640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a1015ee10 0x7f5a10145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a17f94640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a10145840 0x7f5a101f5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a17f94640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5a10160280 con 0x7f5a100d23c0 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a17f94640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5a10160100 con 0x7f5a10145840 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a17f94640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5a10160400 con 0x7f5a1015ee10 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a15d30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a100d23c0 0x7f5a101ea170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a15d30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a100d23c0 0x7f5a101ea170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44516/0 (socket says 172.21.15.59:44516) 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a15d30640 1 -- 172.21.15.59:0/3644927453 learned_addr learned my addr 172.21.15.59:0/3644927453 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a1552f640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a1015ee10 0x7f5a10145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a16531640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a10145840 0x7f5a101f5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a15d30640 1 -- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a1015ee10 msgr2=0x7f5a10145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a15d30640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a1015ee10 0x7f5a10145300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a15d30640 1 -- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a10145840 msgr2=0x7f5a101f5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a15d30640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a10145840 0x7f5a101f5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.512+0000 7f5a15d30640 1 -- 172.21.15.59:0/3644927453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a101f51a0 con 0x7f5a100d23c0 2024-06-29T15:59:31.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.513+0000 7f5a15d30640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a100d23c0 0x7f5a101ea170 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00c390 tx=0x7f5a0c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:31.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.513+0000 7f5a067fc640 1 -- 172.21.15.59:0/3644927453 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c017070 con 0x7f5a100d23c0 2024-06-29T15:59:31.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.513+0000 7f5a067fc640 1 -- 172.21.15.59:0/3644927453 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a0c0035f0 con 0x7f5a100d23c0 2024-06-29T15:59:31.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.513+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a10146050 con 0x7f5a100d23c0 2024-06-29T15:59:31.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.513+0000 7f5a067fc640 1 -- 172.21.15.59:0/3644927453 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c003a60 con 0x7f5a100d23c0 2024-06-29T15:59:31.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.513+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a101ea6b0 con 0x7f5a100d23c0 2024-06-29T15:59:31.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.514+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a100b3760 con 0x7f5a100d23c0 2024-06-29T15:59:31.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.516+0000 7f5a067fc640 1 -- 172.21.15.59:0/3644927453 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f5a0c01a070 con 0x7f5a100d23c0 2024-06-29T15:59:31.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.516+0000 7f5a067fc640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f59f4076420 0x7f59f4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.516+0000 7f5a067fc640 1 -- 172.21.15.59:0/3644927453 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 6587+0+0 (secure 0 0 0) 0x7f5a0c096ec0 con 0x7f5a100d23c0 2024-06-29T15:59:31.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.516+0000 7f5a1552f640 1 -- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f59f4076420 msgr2=0x7f59f4078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/1744808854 2024-06-29T15:59:31.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.516+0000 7f5a1552f640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f59f4076420 0x7f59f4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:31.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.521+0000 7f5a067fc640 1 -- 172.21.15.59:0/3644927453 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5a100b3760 con 0x7f5a100d23c0 2024-06-29T15:59:31.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.712+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5a1015e8d0 con 0x7f5a100d23c0 2024-06-29T15:59:31.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.713+0000 7f5a067fc640 1 -- 172.21.15.59:0/3644927453 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v263) ==== 104+0+11 (secure 0 0 0) 0x7f5a0c05fcc0 con 0x7f5a100d23c0 2024-06-29T15:59:31.713 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T15:59:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f59f4076420 msgr2=0x7f59f4078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f59f4076420 0x7f59f4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a100d23c0 msgr2=0x7f5a101ea170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a100d23c0 0x7f5a101ea170 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00c390 tx=0x7f5a0c00c850 comp rx=0 tx=0).stop 2024-06-29T15:59:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 shutdown_connections 2024-06-29T15:59:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:6832/1744808854,v1:172.21.15.153:6833/1744808854] conn(0x7f59f4076420 0x7f59f4078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5a10145840 0x7f5a101f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5a1015ee10 0x7f5a10145300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 --2- 172.21.15.59:0/3644927453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a100d23c0 0x7f5a101ea170 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 >> 172.21.15.59:0/3644927453 conn(0x7f5a100d3ee0 msgr2=0x7f5a10157fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 shutdown_connections 2024-06-29T15:59:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.715+0000 7f5a17f94640 1 -- 172.21.15.59:0/3644927453 wait complete. 2024-06-29T15:59:31.726 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:59:31.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.855+0000 7ff698be0640 1 -- 172.21.15.59:0/4097928399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 msgr2=0x7ff6940b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.855+0000 7ff698be0640 1 --2- 172.21.15.59:0/4097928399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 0x7ff6940b5b30 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7ff68c005fd0 tx=0x7ff68c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.855+0000 7ff698be0640 1 -- 172.21.15.59:0/4097928399 shutdown_connections 2024-06-29T15:59:31.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.855+0000 7ff698be0640 1 --2- 172.21.15.59:0/4097928399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 0x7ff6940b5b30 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.855+0000 7ff698be0640 1 --2- 172.21.15.59:0/4097928399 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6940b2270 0x7ff69415bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.855+0000 7ff698be0640 1 -- 172.21.15.59:0/4097928399 >> 172.21.15.59:0/4097928399 conn(0x7ff694073e10 msgr2=0x7ff694074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:31.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.855+0000 7ff698be0640 1 -- 172.21.15.59:0/4097928399 shutdown_connections 2024-06-29T15:59:31.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.855+0000 7ff698be0640 1 -- 172.21.15.59:0/4097928399 wait complete. 2024-06-29T15:59:31.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.857+0000 7ff698be0640 1 Processor -- start 2024-06-29T15:59:31.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.857+0000 7ff698be0640 1 -- start start 2024-06-29T15:59:31.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.858+0000 7ff698be0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6940b2270 0x7ff69415e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.858+0000 7ff698be0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6940b2fa0 0x7ff6941450d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.858+0000 7ff698be0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 0x7ff6940ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.858+0000 7ff698be0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff6940b6400 con 0x7ff6940d1c30 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.858+0000 7ff698be0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6940b6280 con 0x7ff6940b2270 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.858+0000 7ff698be0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff6940b6580 con 0x7ff6940b2fa0 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 0x7ff6940ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 0x7ff6940ac510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44526/0 (socket says 172.21.15.59:44526) 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 -- 172.21.15.59:0/2664733385 learned_addr learned my addr 172.21.15.59:0/2664733385 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff69259c640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6940b2270 0x7ff69415e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff68bfff640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6940b2fa0 0x7ff6941450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 -- 172.21.15.59:0/2664733385 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6940b2fa0 msgr2=0x7ff6941450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6940b2fa0 0x7ff6941450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 -- 172.21.15.59:0/2664733385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6940b2270 msgr2=0x7ff69415e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6940b2270 0x7ff69415e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 -- 172.21.15.59:0/2664733385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69415d900 con 0x7ff6940d1c30 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff692d9d640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 0x7ff6940ac510 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009cd0 tx=0x7ff68c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff68affd640 1 -- 172.21.15.59:0/2664733385 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c031990 con 0x7ff6940d1c30 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff68affd640 1 -- 172.21.15.59:0/2664733385 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff68c031c20 con 0x7ff6940d1c30 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.859+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69415dbe0 con 0x7ff6940d1c30 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.860+0000 7ff68affd640 1 -- 172.21.15.59:0/2664733385 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0330d0 con 0x7ff6940d1c30 2024-06-29T15:59:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.860+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff69415e1a0 con 0x7ff6940d1c30 2024-06-29T15:59:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.861+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6940b6ac0 con 0x7ff6940d1c30 2024-06-29T15:59:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.861+0000 7ff68affd640 1 -- 172.21.15.59:0/2664733385 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 152) ==== 97912+0+0 (secure 0 0 0) 0x7ff68c033270 con 0x7ff6940d1c30 2024-06-29T15:59:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.862+0000 7ff68affd640 1 -- 172.21.15.59:0/2664733385 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 6931+0+0 (secure 0 0 0) 0x7ff68c0b8960 con 0x7ff6940d1c30 2024-06-29T15:59:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:31.866+0000 7ff68affd640 1 -- 172.21.15.59:0/2664733385 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff68c03f030 con 0x7ff6940d1c30 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5c21c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5c21c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5c21c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5c21c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5c21c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5f222640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5f222640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5f222640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5f222640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:31.905 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:31.903+0000 7fed5f222640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.059+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff694155520 con 0x7ff6940d1c30 2024-06-29T15:59:32.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.060+0000 7ff68affd640 1 -- 172.21.15.59:0/2664733385 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v264) ==== 100+0+11 (secure 0 0 0) 0x7ff68c081bf0 con 0x7ff6940d1c30 2024-06-29T15:59:32.060 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T15:59:32.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 msgr2=0x7ff6940ac510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 0x7ff6940ac510 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009cd0 tx=0x7ff68c031700 comp rx=0 tx=0).stop 2024-06-29T15:59:32.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 shutdown_connections 2024-06-29T15:59:32.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6940d1c30 0x7ff6940ac510 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff6940b2fa0 0x7ff6941450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 --2- 172.21.15.59:0/2664733385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6940b2270 0x7ff69415e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 >> 172.21.15.59:0/2664733385 conn(0x7ff694073e10 msgr2=0x7ff69406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:32.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 shutdown_connections 2024-06-29T15:59:32.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.062+0000 7ff698be0640 1 -- 172.21.15.59:0/2664733385 wait complete. 2024-06-29T15:59:32.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:32.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.193+0000 7f3a41d3c640 1 -- 172.21.15.59:0/2785423465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 msgr2=0x7f3a3c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.193+0000 7f3a41d3c640 1 --2- 172.21.15.59:0/2785423465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 0x7f3a3c0ce550 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c005fd0 tx=0x7f3a2c02a540 comp rx=0 tx=0).stop 2024-06-29T15:59:32.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.193+0000 7f3a41d3c640 1 -- 172.21.15.59:0/2785423465 shutdown_connections 2024-06-29T15:59:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.193+0000 7f3a41d3c640 1 --2- 172.21.15.59:0/2785423465 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a3c15e0a0 0x7f3a3c161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.193+0000 7f3a41d3c640 1 --2- 172.21.15.59:0/2785423465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 0x7f3a3c0ce550 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.193+0000 7f3a41d3c640 1 -- 172.21.15.59:0/2785423465 >> 172.21.15.59:0/2785423465 conn(0x7f3a3c0d45b0 msgr2=0x7f3a3c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.193+0000 7f3a41d3c640 1 -- 172.21.15.59:0/2785423465 shutdown_connections 2024-06-29T15:59:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.193+0000 7f3a41d3c640 1 -- 172.21.15.59:0/2785423465 wait complete. 2024-06-29T15:59:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.195+0000 7f3a41d3c640 1 Processor -- start 2024-06-29T15:59:32.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.195+0000 7f3a41d3c640 1 -- start start 2024-06-29T15:59:32.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.196+0000 7f3a41d3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a3c15e0a0 0x7f3a3c1f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.196+0000 7f3a41d3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a3c161fc0 0x7f3a3c157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.196+0000 7f3a41d3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 0x7f3a3c1530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.196+0000 7f3a41d3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c0d2d80 con 0x7f3a3c162770 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.196+0000 7f3a41d3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c0b4b00 con 0x7f3a3c161fc0 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.196+0000 7f3a41d3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3c0d2f00 con 0x7f3a3c15e0a0 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 0x7f3a3c1530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 0x7f3a3c1530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44540/0 (socket says 172.21.15.59:44540) 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 -- 172.21.15.59:0/4021765101 learned_addr learned my addr 172.21.15.59:0/4021765101 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3b7fe640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a3c15e0a0 0x7f3a3c1f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3affd640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a3c161fc0 0x7f3a3c157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 -- 172.21.15.59:0/4021765101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a3c15e0a0 msgr2=0x7f3a3c1f54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a3c15e0a0 0x7f3a3c1f54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 -- 172.21.15.59:0/4021765101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a3c161fc0 msgr2=0x7f3a3c157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a3c161fc0 0x7f3a3c157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 -- 172.21.15.59:0/4021765101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3c157c10 con 0x7f3a3c162770 2024-06-29T15:59:32.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a3bfff640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 0x7f3a3c1530f0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400c730 tx=0x7f3a3400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:32.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a23fff640 1 -- 172.21.15.59:0/4021765101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a34003b90 con 0x7f3a3c162770 2024-06-29T15:59:32.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.197+0000 7f3a23fff640 1 -- 172.21.15.59:0/4021765101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a34003d50 con 0x7f3a3c162770 2024-06-29T15:59:32.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.198+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3c158080 con 0x7f3a3c162770 2024-06-29T15:59:32.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.198+0000 7f3a23fff640 1 -- 172.21.15.59:0/4021765101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a34005330 con 0x7f3a3c162770 2024-06-29T15:59:32.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.198+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3c1f4cb0 con 0x7f3a3c162770 2024-06-29T15:59:32.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.199+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3c0d3390 con 0x7f3a3c162770 2024-06-29T15:59:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.206+0000 7f3a23fff640 1 -- 172.21.15.59:0/4021765101 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 152) ==== 97912+0+0 (secure 0 0 0) 0x7f3a340056e0 con 0x7f3a3c162770 2024-06-29T15:59:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.206+0000 7f3a23fff640 1 -- 172.21.15.59:0/4021765101 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 6931+0+0 (secure 0 0 0) 0x7f3a34098040 con 0x7f3a3c162770 2024-06-29T15:59:32.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.209+0000 7f3a23fff640 1 -- 172.21.15.59:0/4021765101 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3a3c1623d0 con 0x7f3a3c162770 2024-06-29T15:59:32.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.437+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a3c06c310 con 0x7f3a3c162770 2024-06-29T15:59:32.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.438+0000 7f3a23fff640 1 -- 172.21.15.59:0/4021765101 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5247 (secure 0 0 0) 0x7f3a340619c0 con 0x7f3a3c162770 2024-06-29T15:59:32.438 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:32.438 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":148,"btime":"2024-06-29T15:59:16:455546+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1010984928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1010984928},{"type":"v1","addr":"172.21.15.153:6836","nonce":1010984928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":10475,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2101409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2101409892},{"type":"v1","addr":"172.21.15.153:6837","nonce":2101409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":12130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1208243017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1208243017},{"type":"v1","addr":"172.21.15.59:6835","nonce":1208243017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:12.808119+0000","modified":"2024-06-29T15:59:15.847810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13110":{"gid":13110,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/2985893835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2985893835},{"type":"v1","addr":"172.21.15.59:6833","nonce":2985893835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13110,"qdb_cluster":[13110]},"id":17}]} 2024-06-29T15:59:32.438 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 148 2024-06-29T15:59:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.440+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 msgr2=0x7f3a3c1530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.440+0000 7f3a41d3c640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 0x7f3a3c1530f0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400c730 tx=0x7f3a3400cbf0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.440+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 shutdown_connections 2024-06-29T15:59:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.440+0000 7f3a41d3c640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a3c162770 0x7f3a3c1530f0 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.440+0000 7f3a41d3c640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a3c161fc0 0x7f3a3c157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.440+0000 7f3a41d3c640 1 --2- 172.21.15.59:0/4021765101 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3a3c15e0a0 0x7f3a3c1f54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.440+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 >> 172.21.15.59:0/4021765101 conn(0x7f3a3c0d45b0 msgr2=0x7f3a3c0b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.441+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 shutdown_connections 2024-06-29T15:59:32.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.441+0000 7f3a41d3c640 1 -- 172.21.15.59:0/4021765101 wait complete. 2024-06-29T15:59:32.452 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:32.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.570+0000 7f03b6f82640 1 -- 172.21.15.59:0/2500337356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b00cd9b0 msgr2=0x7f03b015f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.570+0000 7f03b6f82640 1 --2- 172.21.15.59:0/2500337356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b00cd9b0 0x7f03b015f240 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f03a0005fd0 tx=0x7f03a002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.570+0000 7f03b6f82640 1 -- 172.21.15.59:0/2500337356 shutdown_connections 2024-06-29T15:59:32.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.570+0000 7f03b6f82640 1 --2- 172.21.15.59:0/2500337356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b00cd9b0 0x7f03b015f240 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.570+0000 7f03b6f82640 1 -- 172.21.15.59:0/2500337356 >> 172.21.15.59:0/2500337356 conn(0x7f03b00b24c0 msgr2=0x7f03b00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:32.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.570+0000 7f03b6f82640 1 -- 172.21.15.59:0/2500337356 shutdown_connections 2024-06-29T15:59:32.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.570+0000 7f03b6f82640 1 -- 172.21.15.59:0/2500337356 wait complete. 2024-06-29T15:59:32.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.572+0000 7f03b6f82640 1 Processor -- start 2024-06-29T15:59:32.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.572+0000 7f03b6f82640 1 -- start start 2024-06-29T15:59:32.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b6f82640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b00cd9b0 0x7f03b013a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b6f82640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03b01444c0 0x7f03b01e6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b6f82640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b01f3090 0x7f03b013f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b6f82640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f03b0073d10 con 0x7f03b01f3090 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b6f82640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03b0073b90 con 0x7f03b01444c0 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b6f82640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f03b0073e90 con 0x7f03b00cd9b0 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b551f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b01f3090 0x7f03b013f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b551f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b01f3090 0x7f03b013f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44548/0 (socket says 172.21.15.59:44548) 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b551f640 1 -- 172.21.15.59:0/402481911 learned_addr learned my addr 172.21.15.59:0/402481911 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03affff640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03b01444c0 0x7f03b01e6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.573+0000 7f03b4d1e640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b00cd9b0 0x7f03b013a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03b551f640 1 -- 172.21.15.59:0/402481911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b00cd9b0 msgr2=0x7f03b013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03b551f640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b00cd9b0 0x7f03b013a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03b551f640 1 -- 172.21.15.59:0/402481911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03b01444c0 msgr2=0x7f03b01e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03b551f640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03b01444c0 0x7f03b01e6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03b551f640 1 -- 172.21.15.59:0/402481911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b01e5dc0 con 0x7f03b01f3090 2024-06-29T15:59:32.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03b551f640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b01f3090 0x7f03b013f610 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f03a800b410 tx=0x7f03a800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:32.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03ad7fa640 1 -- 172.21.15.59:0/402481911 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a8015030 con 0x7f03b01f3090 2024-06-29T15:59:32.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03ad7fa640 1 -- 172.21.15.59:0/402481911 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03a8004130 con 0x7f03b01f3090 2024-06-29T15:59:32.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b01e60a0 con 0x7f03b01f3090 2024-06-29T15:59:32.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03ad7fa640 1 -- 172.21.15.59:0/402481911 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a8011630 con 0x7f03b01f3090 2024-06-29T15:59:32.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.574+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b01e6660 con 0x7f03b01f3090 2024-06-29T15:59:32.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.575+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b015f240 con 0x7f03b01f3090 2024-06-29T15:59:32.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.576+0000 7f03ad7fa640 1 -- 172.21.15.59:0/402481911 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 152) ==== 97912+0+0 (secure 0 0 0) 0x7f03a8018070 con 0x7f03b01f3090 2024-06-29T15:59:32.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.577+0000 7f03ad7fa640 1 -- 172.21.15.59:0/402481911 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 6931+0+0 (secure 0 0 0) 0x7f03a8097260 con 0x7f03b01f3090 2024-06-29T15:59:32.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.581+0000 7f03ad7fa640 1 -- 172.21.15.59:0/402481911 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f03a8060380 con 0x7f03b01f3090 2024-06-29T15:59:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.773+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03b006c9d0 con 0x7f03b01f3090 2024-06-29T15:59:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.774+0000 7f03ad7fa640 1 -- 172.21.15.59:0/402481911 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) ==== 74+0+18133 (secure 0 0 0) 0x7f03a8064680 con 0x7f03b01f3090 2024-06-29T15:59:32.774 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:32.774 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":264,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:31.842603+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:11.169128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:11.760290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/376097765":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/3823156403":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/3656223405":"2024-06-30T15:59:31.842564+0000","172.21.15.153:6832/1744808854":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2017664585":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/623754277":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2546767818":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6833/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6832/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2003505066":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6833/1744808854":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2217399912":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2250305531":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1734622549":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1337010763":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2869442511":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/3703209251":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2872771609":"2024-06-30T15:59:25.117274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b01f3090 msgr2=0x7f03b013f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b01f3090 0x7f03b013f610 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f03a800b410 tx=0x7f03a800b8d0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 shutdown_connections 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03b01f3090 0x7f03b013f610 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03b01444c0 0x7f03b01e6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 --2- 172.21.15.59:0/402481911 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03b00cd9b0 0x7f03b013a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 >> 172.21.15.59:0/402481911 conn(0x7f03b00b24c0 msgr2=0x7f03b015ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 shutdown_connections 2024-06-29T15:59:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.777+0000 7f03b6f82640 1 -- 172.21.15.59:0/402481911 wait complete. 2024-06-29T15:59:32.788 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:59:32.788 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:32.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.911+0000 7f61bfe66640 1 -- 172.21.15.59:0/1576866347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 msgr2=0x7f61b80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.911+0000 7f61bfe66640 1 --2- 172.21.15.59:0/1576866347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 0x7f61b80b3990 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f61b0005fd0 tx=0x7f61b002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.911+0000 7f61bfe66640 1 -- 172.21.15.59:0/1576866347 shutdown_connections 2024-06-29T15:59:32.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.911+0000 7f61bfe66640 1 --2- 172.21.15.59:0/1576866347 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b815efe0 0x7f61b815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.911+0000 7f61bfe66640 1 --2- 172.21.15.59:0/1576866347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 0x7f61b80b3990 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.911+0000 7f61bfe66640 1 -- 172.21.15.59:0/1576866347 >> 172.21.15.59:0/1576866347 conn(0x7f61b80d3ee0 msgr2=0x7f61b80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:32.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.911+0000 7f61bfe66640 1 -- 172.21.15.59:0/1576866347 shutdown_connections 2024-06-29T15:59:32.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.911+0000 7f61bfe66640 1 -- 172.21.15.59:0/1576866347 wait complete. 2024-06-29T15:59:32.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.913+0000 7f61bfe66640 1 Processor -- start 2024-06-29T15:59:32.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.913+0000 7f61bfe66640 1 -- start start 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bfe66640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80b3ed0 0x7f61b80ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bfe66640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 0x7f61b81e7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bfe66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b815efe0 0x7f61b81e8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bfe66640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8160280 con 0x7f61b80d23c0 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bfe66640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8160100 con 0x7f61b815efe0 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bfe66640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f61b8160400 con 0x7f61b80b3ed0 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bd401640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 0x7f61b81e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bd401640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 0x7f61b81e7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53774/0 (socket says 172.21.15.59:53774) 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bd401640 1 -- 172.21.15.59:0/3066251780 learned_addr learned my addr 172.21.15.59:0/3066251780 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bdc02640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80b3ed0 0x7f61b80ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61be403640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b815efe0 0x7f61b81e8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bd401640 1 -- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80b3ed0 msgr2=0x7f61b80ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bd401640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80b3ed0 0x7f61b80ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bd401640 1 -- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b815efe0 msgr2=0x7f61b81e8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bd401640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b815efe0 0x7f61b81e8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bd401640 1 -- 172.21.15.59:0/3066251780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b81477c0 con 0x7f61b80d23c0 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.914+0000 7f61bdc02640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80b3ed0 0x7f61b80ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.915+0000 7f61bd401640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 0x7f61b81e7470 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c3a0 tx=0x7f61ac00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.915+0000 7f61b67fc640 1 -- 172.21.15.59:0/3066251780 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac017070 con 0x7f61b80d23c0 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.915+0000 7f61b67fc640 1 -- 172.21.15.59:0/3066251780 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61ac0053c0 con 0x7f61b80d23c0 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.915+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b81e7f30 con 0x7f61b80d23c0 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.915+0000 7f61b67fc640 1 -- 172.21.15.59:0/3066251780 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac003850 con 0x7f61b80d23c0 2024-06-29T15:59:32.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.915+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b81e84f0 con 0x7f61b80d23c0 2024-06-29T15:59:32.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.916+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b81e7ca0 con 0x7f61b80d23c0 2024-06-29T15:59:32.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.917+0000 7f61b67fc640 1 -- 172.21.15.59:0/3066251780 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 153) ==== 98039+0+0 (secure 0 0 0) 0x7f61ac01a070 con 0x7f61b80d23c0 2024-06-29T15:59:32.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.917+0000 7f61b67fc640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f61a4076120 0x7f61a4078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.917+0000 7f61b67fc640 1 -- 172.21.15.59:0/3066251780 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 6931+0+0 (secure 0 0 0) 0x7f61ac096ce0 con 0x7f61b80d23c0 2024-06-29T15:59:32.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.917+0000 7f61bdc02640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f61a4076120 0x7f61a4078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.918+0000 7f61bdc02640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f61a4076120 0x7f61a4078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61b0033000 tx=0x7f61b0004730 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:32.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:32.922+0000 7f61b67fc640 1 -- 172.21.15.59:0/3066251780 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f61ac09b070 con 0x7f61b80d23c0 2024-06-29T15:59:33.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.152+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61b80b3990 con 0x7f61b80d23c0 2024-06-29T15:59:33.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.152+0000 7f61b67fc640 1 -- 172.21.15.59:0/3066251780 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5247 (secure 0 0 0) 0x7f61b80b42e0 con 0x7f61b80d23c0 2024-06-29T15:59:33.152 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:33.152 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":148,"btime":"2024-06-29T15:59:16:455546+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1010984928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1010984928},{"type":"v1","addr":"172.21.15.153:6836","nonce":1010984928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":10475,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2101409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2101409892},{"type":"v1","addr":"172.21.15.153:6837","nonce":2101409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":12130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1208243017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1208243017},{"type":"v1","addr":"172.21.15.59:6835","nonce":1208243017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:12.808119+0000","modified":"2024-06-29T15:59:15.847810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13110":{"gid":13110,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/2985893835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2985893835},{"type":"v1","addr":"172.21.15.59:6833","nonce":2985893835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13110,"qdb_cluster":[13110]},"id":17}]} 2024-06-29T15:59:33.153 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 148 2024-06-29T15:59:33.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f61a4076120 msgr2=0x7f61a4078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f61a4076120 0x7f61a4078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61b0033000 tx=0x7f61b0004730 comp rx=0 tx=0).stop 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 msgr2=0x7f61b81e7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 0x7f61b81e7470 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c3a0 tx=0x7f61ac00c860 comp rx=0 tx=0).stop 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 shutdown_connections 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f61a4076120 0x7f61a4078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b815efe0 0x7f61b81e8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61b80d23c0 0x7f61b81e7470 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 --2- 172.21.15.59:0/3066251780 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80b3ed0 0x7f61b80ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 >> 172.21.15.59:0/3066251780 conn(0x7f61b80d3ee0 msgr2=0x7f61b80d2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.154+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 shutdown_connections 2024-06-29T15:59:33.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.155+0000 7f61bfe66640 1 -- 172.21.15.59:0/3066251780 wait complete. 2024-06-29T15:59:33.166 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.284+0000 7fdb6ace7640 1 -- 172.21.15.59:0/1545571747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 msgr2=0x7fdb641622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.284+0000 7fdb6ace7640 1 --2- 172.21.15.59:0/1545571747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 0x7fdb641622c0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7fdb640b78c0 tx=0x7fdb4c02a840 comp rx=0 tx=0).stop 2024-06-29T15:59:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.284+0000 7fdb6ace7640 1 -- 172.21.15.59:0/1545571747 shutdown_connections 2024-06-29T15:59:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.284+0000 7fdb6ace7640 1 --2- 172.21.15.59:0/1545571747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 0x7fdb641622c0 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.284+0000 7fdb6ace7640 1 -- 172.21.15.59:0/1545571747 >> 172.21.15.59:0/1545571747 conn(0x7fdb640b5610 msgr2=0x7fdb640b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.284+0000 7fdb6ace7640 1 -- 172.21.15.59:0/1545571747 shutdown_connections 2024-06-29T15:59:33.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.284+0000 7fdb6ace7640 1 -- 172.21.15.59:0/1545571747 wait complete. 2024-06-29T15:59:33.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.285+0000 7fdb6ace7640 1 Processor -- start 2024-06-29T15:59:33.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.286+0000 7fdb6ace7640 1 -- start start 2024-06-29T15:59:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb6ace7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6408c190 0x7fdb641f8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb6ace7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 0x7fdb6414c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb6ace7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb64162800 0x7fdb640c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb6ace7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdb640b82e0 con 0x7fdb640ce950 2024-06-29T15:59:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb6ace7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64162ff0 con 0x7fdb64162800 2024-06-29T15:59:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb6ace7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdb640b8460 con 0x7fdb6408c190 2024-06-29T15:59:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 0x7fdb6414c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb63fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 0x7fdb6414c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53784/0 (socket says 172.21.15.59:53784) 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb63fff640 1 -- 172.21.15.59:0/765072706 learned_addr learned my addr 172.21.15.59:0/765072706 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb69284640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb64162800 0x7fdb640c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb68a83640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6408c190 0x7fdb641f8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb63fff640 1 -- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6408c190 msgr2=0x7fdb641f8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb63fff640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6408c190 0x7fdb641f8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb63fff640 1 -- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb64162800 msgr2=0x7fdb640c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb63fff640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb64162800 0x7fdb640c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.287+0000 7fdb63fff640 1 -- 172.21.15.59:0/765072706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb641616b0 con 0x7fdb640ce950 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.288+0000 7fdb63fff640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 0x7fdb6414c660 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009b30 tx=0x7fdb4c033f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.288+0000 7fdb617fa640 1 -- 172.21.15.59:0/765072706 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c042020 con 0x7fdb640ce950 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.288+0000 7fdb617fa640 1 -- 172.21.15.59:0/765072706 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb4c004330 con 0x7fdb640ce950 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.288+0000 7fdb617fa640 1 -- 172.21.15.59:0/765072706 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c037040 con 0x7fdb640ce950 2024-06-29T15:59:33.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.288+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb64161990 con 0x7fdb640ce950 2024-06-29T15:59:33.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.288+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb6414cf90 con 0x7fdb640ce950 2024-06-29T15:59:33.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.289+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb641622c0 con 0x7fdb640ce950 2024-06-29T15:59:33.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.290+0000 7fdb617fa640 1 -- 172.21.15.59:0/765072706 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 153) ==== 98039+0+0 (secure 0 0 0) 0x7fdb4c045020 con 0x7fdb640ce950 2024-06-29T15:59:33.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.290+0000 7fdb617fa640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fdb44076150 0x7fdb44078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.290+0000 7fdb617fa640 1 -- 172.21.15.59:0/765072706 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 6931+0+0 (secure 0 0 0) 0x7fdb4c0b8910 con 0x7fdb640ce950 2024-06-29T15:59:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.290+0000 7fdb68a83640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fdb44076150 0x7fdb44078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.291+0000 7fdb68a83640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fdb44076150 0x7fdb44078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdb54007fd0 tx=0x7fdb54009040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:33.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.295+0000 7fdb617fa640 1 -- 172.21.15.59:0/765072706 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdb4c04c030 con 0x7fdb640ce950 2024-06-29T15:59:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.524+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb640683b0 con 0x7fdb640ce950 2024-06-29T15:59:33.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.524+0000 7fdb617fa640 1 -- 172.21.15.59:0/765072706 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5247 (secure 0 0 0) 0x7fdb4c02f070 con 0x7fdb640ce950 2024-06-29T15:59:33.524 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:33.524 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":148,"btime":"2024-06-29T15:59:16:455546+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1010984928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1010984928},{"type":"v1","addr":"172.21.15.153:6836","nonce":1010984928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":10475,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2101409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2101409892},{"type":"v1","addr":"172.21.15.153:6837","nonce":2101409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":12130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1208243017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1208243017},{"type":"v1","addr":"172.21.15.59:6835","nonce":1208243017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:12.808119+0000","modified":"2024-06-29T15:59:15.847810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13110":{"gid":13110,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/2985893835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2985893835},{"type":"v1","addr":"172.21.15.59:6833","nonce":2985893835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13110,"qdb_cluster":[13110]},"id":17}]} 2024-06-29T15:59:33.525 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 148 2024-06-29T15:59:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fdb44076150 msgr2=0x7fdb44078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fdb44076150 0x7fdb44078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdb54007fd0 tx=0x7fdb54009040 comp rx=0 tx=0).stop 2024-06-29T15:59:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 msgr2=0x7fdb6414c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 0x7fdb6414c660 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009b30 tx=0x7fdb4c033f00 comp rx=0 tx=0).stop 2024-06-29T15:59:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 shutdown_connections 2024-06-29T15:59:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fdb44076150 0x7fdb44078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb64162800 0x7fdb640c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb640ce950 0x7fdb6414c660 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 --2- 172.21.15.59:0/765072706 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdb6408c190 0x7fdb641f8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 >> 172.21.15.59:0/765072706 conn(0x7fdb640b5610 msgr2=0x7fdb6415e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 shutdown_connections 2024-06-29T15:59:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.526+0000 7fdb6ace7640 1 -- 172.21.15.59:0/765072706 wait complete. 2024-06-29T15:59:33.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:33.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.654+0000 7f5d75a90640 1 -- 172.21.15.59:0/3575912700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7015efe0 msgr2=0x7f5d7015f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.654+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3575912700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7015efe0 0x7f5d7015f9b0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f5d700d15d0 tx=0x7f5d6802aac0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.655+0000 7f5d75a90640 1 -- 172.21.15.59:0/3575912700 shutdown_connections 2024-06-29T15:59:33.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.655+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3575912700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7015efe0 0x7f5d7015f9b0 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.655+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3575912700 >> v2:172.21.15.153:3300/0 conn(0x7f5d700d23c0 0x7f5d700b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.655+0000 7f5d75a90640 1 -- 172.21.15.59:0/3575912700 >> 172.21.15.59:0/3575912700 conn(0x7f5d700d3ee0 msgr2=0x7f5d700d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.655+0000 7f5d75a90640 1 -- 172.21.15.59:0/3575912700 shutdown_connections 2024-06-29T15:59:33.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.655+0000 7f5d75a90640 1 -- 172.21.15.59:0/3575912700 wait complete. 2024-06-29T15:59:33.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.656+0000 7f5d75a90640 1 Processor -- start 2024-06-29T15:59:33.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.657+0000 7f5d75a90640 1 -- start start 2024-06-29T15:59:33.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d75a90640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d700b3ed0 0x7f5d70159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d75a90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d700d23c0 0x7f5d7014cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d75a90640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d7015efe0 0x7f5d700c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d75a90640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70160280 con 0x7f5d700d23c0 2024-06-29T15:59:33.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d75a90640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70160100 con 0x7f5d700b3ed0 2024-06-29T15:59:33.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d75a90640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5d70160400 con 0x7f5d7015efe0 2024-06-29T15:59:33.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d700d23c0 0x7f5d7014cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d700d23c0 0x7f5d7014cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53808/0 (socket says 172.21.15.59:53808) 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6effd640 1 -- 172.21.15.59:0/3249470582 learned_addr learned my addr 172.21.15.59:0/3249470582 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6f7fe640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d700b3ed0 0x7f5d70159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6ffff640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d7015efe0 0x7f5d700c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6effd640 1 -- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d7015efe0 msgr2=0x7f5d700c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6effd640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d7015efe0 0x7f5d700c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6effd640 1 -- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d700b3ed0 msgr2=0x7f5d70159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6effd640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d700b3ed0 0x7f5d70159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.658+0000 7f5d6effd640 1 -- 172.21.15.59:0/3249470582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7015a800 con 0x7f5d700d23c0 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.659+0000 7f5d6effd640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d700d23c0 0x7f5d7014cd30 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400c400 tx=0x7f5d6400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.659+0000 7f5d57fff640 1 -- 172.21.15.59:0/3249470582 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64017070 con 0x7f5d700d23c0 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.659+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7015aae0 con 0x7f5d700d23c0 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.659+0000 7f5d57fff640 1 -- 172.21.15.59:0/3249470582 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d640053c0 con 0x7f5d700d23c0 2024-06-29T15:59:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.659+0000 7f5d57fff640 1 -- 172.21.15.59:0/3249470582 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64003830 con 0x7f5d700d23c0 2024-06-29T15:59:33.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.659+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d7015b0a0 con 0x7f5d700d23c0 2024-06-29T15:59:33.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.660+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7015a570 con 0x7f5d700d23c0 2024-06-29T15:59:33.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.661+0000 7f5d57fff640 1 -- 172.21.15.59:0/3249470582 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 153) ==== 98039+0+0 (secure 0 0 0) 0x7f5d6401a070 con 0x7f5d700d23c0 2024-06-29T15:59:33.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.661+0000 7f5d57fff640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5d5c076150 0x7f5d5c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.661+0000 7f5d57fff640 1 -- 172.21.15.59:0/3249470582 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 6931+0+0 (secure 0 0 0) 0x7f5d64096c80 con 0x7f5d700d23c0 2024-06-29T15:59:33.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.661+0000 7f5d6f7fe640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5d5c076150 0x7f5d5c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.662+0000 7f5d6f7fe640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5d5c076150 0x7f5d5c078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d700d2e40 tx=0x7f5d60005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.665+0000 7f5d57fff640 1 -- 172.21.15.59:0/3249470582 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d700b42e0 con 0x7f5d700d23c0 2024-06-29T15:59:33.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.856+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d700b3990 con 0x7f5d700d23c0 2024-06-29T15:59:33.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.857+0000 7f5d57fff640 1 -- 172.21.15.59:0/3249470582 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) ==== 74+0+18133 (secure 0 0 0) 0x7f5d640683c0 con 0x7f5d700d23c0 2024-06-29T15:59:33.857 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:33.857 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":264,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:31.842603+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:11.169128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"261","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:11.760290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/376097765":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/3823156403":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/3656223405":"2024-06-30T15:59:31.842564+0000","172.21.15.153:6832/1744808854":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2017664585":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/623754277":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2546767818":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6833/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6832/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2003505066":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6833/1744808854":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2217399912":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2250305531":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1734622549":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1337010763":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2869442511":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/3703209251":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2872771609":"2024-06-30T15:59:25.117274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5d5c076150 msgr2=0x7f5d5c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5d5c076150 0x7f5d5c078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d700d2e40 tx=0x7f5d60005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d700d23c0 msgr2=0x7f5d7014cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d700d23c0 0x7f5d7014cd30 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400c400 tx=0x7f5d6400c8c0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 shutdown_connections 2024-06-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5d5c076150 0x7f5d5c078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5d7015efe0 0x7f5d700c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d700d23c0 0x7f5d7014cd30 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 --2- 172.21.15.59:0/3249470582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d700b3ed0 0x7f5d70159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 >> 172.21.15.59:0/3249470582 conn(0x7f5d700d3ee0 msgr2=0x7f5d7006c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 shutdown_connections 2024-06-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.859+0000 7f5d75a90640 1 -- 172.21.15.59:0/3249470582 wait complete. 2024-06-29T15:59:33.870 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:59:33.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.990+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/1873553589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e40b3740 msgr2=0x7fc8e40b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.990+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/1873553589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e40b3740 0x7fc8e40b3b50 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4074b10 tx=0x7fc8cc02a560 comp rx=0 tx=0).stop 2024-06-29T15:59:33.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.990+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/1873553589 shutdown_connections 2024-06-29T15:59:33.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.990+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/1873553589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e40b3740 0x7fc8e40b3b50 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.990+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/1873553589 >> v2:172.21.15.153:3300/0 conn(0x7fc8e41458a0 0x7fc8e40cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.990+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/1873553589 >> 172.21.15.59:0/1873553589 conn(0x7fc8e406c950 msgr2=0x7fc8e4073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.990+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/1873553589 shutdown_connections 2024-06-29T15:59:33.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.990+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/1873553589 wait complete. 2024-06-29T15:59:33.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.992+0000 7fc8e9e9c640 1 Processor -- start 2024-06-29T15:59:33.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.992+0000 7fc8e9e9c640 1 -- start start 2024-06-29T15:59:33.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e9e9c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8e40b3740 0x7fc8e40b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e9e9c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8e415c2f0 0x7fc8e406db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e9e9c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e41eb7b0 0x7fc8e41ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e9e9c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e4160020 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e9e9c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e415cb90 con 0x7fc8e40b3740 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e9e9c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e41601a0 con 0x7fc8e415c2f0 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e41eb7b0 0x7fc8e41ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e41eb7b0 0x7fc8e41ebbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53832/0 (socket says 172.21.15.59:53832) 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e3fff640 1 -- 172.21.15.59:0/4293227612 learned_addr learned my addr 172.21.15.59:0/4293227612 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8e37fe640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8e40b3740 0x7fc8e40b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.993+0000 7fc8daffd640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8e415c2f0 0x7fc8e406db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e3fff640 1 -- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8e415c2f0 msgr2=0x7fc8e406db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e3fff640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8e415c2f0 0x7fc8e406db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e3fff640 1 -- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8e40b3740 msgr2=0x7fc8e40b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e3fff640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8e40b3740 0x7fc8e40b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e3fff640 1 -- 172.21.15.59:0/4293227612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e406e790 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e3fff640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e41eb7b0 0x7fc8e41ebbc0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00ba10 tx=0x7fc8dc00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:33.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e0ff9640 1 -- 172.21.15.59:0/4293227612 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc019070 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e0ff9640 1 -- 172.21.15.59:0/4293227612 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8dc00cc30 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e406ea70 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e0ff9640 1 -- 172.21.15.59:0/4293227612 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc0146a0 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.994+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e406f030 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.995+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e41606e0 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.996+0000 7fc8e0ff9640 1 -- 172.21.15.59:0/4293227612 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 153) ==== 98039+0+0 (secure 0 0 0) 0x7fc8dc01c070 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.996+0000 7fc8e0ff9640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc8c4076120 0x7fc8c4078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.997+0000 7fc8e0ff9640 1 -- 172.21.15.59:0/4293227612 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 6931+0+0 (secure 0 0 0) 0x7fc8dc097610 con 0x7fc8e41eb7b0 2024-06-29T15:59:33.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.997+0000 7fc8e37fe640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc8c4076120 0x7fc8c4078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:33.997+0000 7fc8e37fe640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc8c4076120 0x7fc8c4078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4004630 tx=0x7fc8d400b2e0 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:34.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.001+0000 7fc8e0ff9640 1 -- 172.21.15.59:0/4293227612 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc8dc00ce20 con 0x7fc8e41eb7b0 2024-06-29T15:59:34.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.210+0000 7fc8e0ff9640 1 -- 172.21.15.59:0/4293227612 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 154) ==== 98247+0+0 (secure 0 0 0) 0x7fc8dc05c910 con 0x7fc8e41eb7b0 2024-06-29T15:59:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.235+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc8e406e520 con 0x7fc8e41eb7b0 2024-06-29T15:59:34.235 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T15:59:34.235+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:59:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.238+0000 7fc8e0ff9640 1 -- 172.21.15.59:0/4293227612 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v149) ==== 161+0+0 (secure 0 0 0) 0x7fc8dc06c3d0 con 0x7fc8e41eb7b0 2024-06-29T15:59:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:59:34.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.240+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc8c4076120 msgr2=0x7fc8c4078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.240+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc8c4076120 0x7fc8c4078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4004630 tx=0x7fc8d400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e41eb7b0 msgr2=0x7fc8e41ebbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e41eb7b0 0x7fc8e41ebbc0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00ba10 tx=0x7fc8dc00bed0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 shutdown_connections 2024-06-29T15:59:34.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc8c4076120 0x7fc8c4078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8e41eb7b0 0x7fc8e41ebbc0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8e415c2f0 0x7fc8e406db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 --2- 172.21.15.59:0/4293227612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8e40b3740 0x7fc8e40b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 >> 172.21.15.59:0/4293227612 conn(0x7fc8e406c950 msgr2=0x7fc8e4070670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:34.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 shutdown_connections 2024-06-29T15:59:34.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.241+0000 7fc8e9e9c640 1 -- 172.21.15.59:0/4293227612 wait complete. 2024-06-29T15:59:34.244 INFO:tasks.ceph.mds.a.smithi059.stderr: -4185> 2024-06-29T15:59:19.391+0000 7fe62b548640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:59:34.254 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:59:34.302 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:59:34.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.378+0000 7fc8b94e4640 1 -- 172.21.15.59:0/621394287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b40b0e30 msgr2=0x7fc8b40b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.378+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/621394287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b40b0e30 0x7fc8b40b4940 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac005fd0 tx=0x7fc8ac02a820 comp rx=0 tx=0).stop 2024-06-29T15:59:34.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.378+0000 7fc8b94e4640 1 -- 172.21.15.59:0/621394287 shutdown_connections 2024-06-29T15:59:34.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.378+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/621394287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b40b0e30 0x7fc8b40b4940 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.378+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/621394287 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8b40ce2c0 0x7fc8b41596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.378+0000 7fc8b94e4640 1 -- 172.21.15.59:0/621394287 >> 172.21.15.59:0/621394287 conn(0x7fc8b40b7690 msgr2=0x7fc8b4073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:34.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.379+0000 7fc8b94e4640 1 -- 172.21.15.59:0/621394287 shutdown_connections 2024-06-29T15:59:34.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.379+0000 7fc8b94e4640 1 -- 172.21.15.59:0/621394287 wait complete. 2024-06-29T15:59:34.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.380+0000 7fc8b94e4640 1 Processor -- start 2024-06-29T15:59:34.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.381+0000 7fc8b94e4640 1 -- start start 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b94e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8b40ce2c0 0x7fc8b40b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b94e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8b4160090 0x7fc8b414a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b94e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b41583a0 0x7fc8b41587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b94e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b40b5470 con 0x7fc8b41583a0 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b94e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b40b52f0 con 0x7fc8b4160090 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b94e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b40b55f0 con 0x7fc8b40ce2c0 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b41583a0 0x7fc8b41587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b41583a0 0x7fc8b41587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53868/0 (socket says 172.21.15.59:53868) 2024-06-29T15:59:34.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 -- 172.21.15.59:0/2420491972 learned_addr learned my addr 172.21.15.59:0/2420491972 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b2ffd640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8b40ce2c0 0x7fc8b40b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8aa7fc640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8b4160090 0x7fc8b414a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 -- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8b40ce2c0 msgr2=0x7fc8b40b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8b40ce2c0 0x7fc8b40b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 -- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8b4160090 msgr2=0x7fc8b414a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8b4160090 0x7fc8b414a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 -- 172.21.15.59:0/2420491972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b4158cf0 con 0x7fc8b41583a0 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b2ffd640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8b40ce2c0 0x7fc8b40b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8aa7fc640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8b4160090 0x7fc8b414a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.382+0000 7fc8b37fe640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b41583a0 0x7fc8b41587b0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac035040 tx=0x7fc8ac032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.383+0000 7fc8abfff640 1 -- 172.21.15.59:0/2420491972 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac037040 con 0x7fc8b41583a0 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.383+0000 7fc8abfff640 1 -- 172.21.15.59:0/2420491972 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8ac042070 con 0x7fc8b41583a0 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.383+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b40b1e60 con 0x7fc8b41583a0 2024-06-29T15:59:34.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.383+0000 7fc8abfff640 1 -- 172.21.15.59:0/2420491972 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac003e60 con 0x7fc8b41583a0 2024-06-29T15:59:34.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.383+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b40b2e70 con 0x7fc8b41583a0 2024-06-29T15:59:34.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.384+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8b40b1bd0 con 0x7fc8b41583a0 2024-06-29T15:59:34.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.384+0000 7fc8abfff640 1 -- 172.21.15.59:0/2420491972 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 154) ==== 98247+0+0 (secure 0 0 0) 0x7fc8ac004280 con 0x7fc8b41583a0 2024-06-29T15:59:34.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.385+0000 7fc8abfff640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc894076360 0x7fc894078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.385+0000 7fc8abfff640 1 -- 172.21.15.59:0/2420491972 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(265..265 src has 1..265) ==== 7017+0+0 (secure 0 0 0) 0x7fc8ac0b8c40 con 0x7fc8b41583a0 2024-06-29T15:59:34.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.385+0000 7fc8b2ffd640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc894076360 0x7fc894078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.386+0000 7fc8b2ffd640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc894076360 0x7fc894078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4009ab0 tx=0x7fc8a4005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:34.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.389+0000 7fc8abfff640 1 -- 172.21.15.59:0/2420491972 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc8b4158cf0 con 0x7fc8b41583a0 2024-06-29T15:59:34.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:34.623+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc8b40b5b10 con 0x7fc8b41583a0 2024-06-29T15:59:35.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.240+0000 7fc8abfff640 1 -- 172.21.15.59:0/2420491972 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v150) ==== 104+0+0 (secure 0 0 0) 0x7fc8ac081e00 con 0x7fc8b41583a0 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc894076360 msgr2=0x7fc894078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc894076360 0x7fc894078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4009ab0 tx=0x7fc8a4005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b41583a0 msgr2=0x7fc8b41587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b41583a0 0x7fc8b41587b0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac035040 tx=0x7fc8ac032c60 comp rx=0 tx=0).stop 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 shutdown_connections 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc894076360 0x7fc894078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8b41583a0 0x7fc8b41587b0 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8b4160090 0x7fc8b414a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 --2- 172.21.15.59:0/2420491972 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8b40ce2c0 0x7fc8b40b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 >> 172.21.15.59:0/2420491972 conn(0x7fc8b40b7690 msgr2=0x7fc8b406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 shutdown_connections 2024-06-29T15:59:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.243+0000 7fc8b94e4640 1 -- 172.21.15.59:0/2420491972 wait complete. 2024-06-29T15:59:35.258 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:59:35.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.386+0000 7fe225135640 1 -- 172.21.15.59:0/972005307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 msgr2=0x7fe2200b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.386+0000 7fe225135640 1 --2- 172.21.15.59:0/972005307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 0x7fe2200b4070 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7fe2200b65f0 tx=0x7fe21802a980 comp rx=0 tx=0).stop 2024-06-29T15:59:35.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.386+0000 7fe225135640 1 -- 172.21.15.59:0/972005307 shutdown_connections 2024-06-29T15:59:35.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.386+0000 7fe225135640 1 --2- 172.21.15.59:0/972005307 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 0x7fe2200b4070 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.386+0000 7fe225135640 1 --2- 172.21.15.59:0/972005307 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2200b2980 0x7fe2200d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.386+0000 7fe225135640 1 --2- 172.21.15.59:0/972005307 >> v2:172.21.15.153:3300/0 conn(0x7fe2200d1c30 0x7fe2200b6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.386+0000 7fe225135640 1 -- 172.21.15.59:0/972005307 >> 172.21.15.59:0/972005307 conn(0x7fe2200d0f90 msgr2=0x7fe2200d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:35.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.386+0000 7fe225135640 1 -- 172.21.15.59:0/972005307 shutdown_connections 2024-06-29T15:59:35.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.387+0000 7fe225135640 1 -- 172.21.15.59:0/972005307 wait complete. 2024-06-29T15:59:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.389+0000 7fe225135640 1 Processor -- start 2024-06-29T15:59:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.389+0000 7fe225135640 1 -- start start 2024-06-29T15:59:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe225135640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2200b2980 0x7fe2201f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe225135640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200d1c30 0x7fe2201607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe225135640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 0x7fe220156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe225135640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe2200b45b0 con 0x7fe2200d2ce0 2024-06-29T15:59:35.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe225135640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe2200d3300 con 0x7fe2200d1c30 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe225135640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe2200b4730 con 0x7fe2200b2980 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21f59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 0x7fe220156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21f59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 0x7fe220156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53888/0 (socket says 172.21.15.59:53888) 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21f59e640 1 -- 172.21.15.59:0/843940843 learned_addr learned my addr 172.21.15.59:0/843940843 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21e59c640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200d1c30 0x7fe2201607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21ed9d640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2200b2980 0x7fe2201f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21f59e640 1 -- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2200b2980 msgr2=0x7fe2201f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21f59e640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2200b2980 0x7fe2201f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21f59e640 1 -- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200d1c30 msgr2=0x7fe2201607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21f59e640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200d1c30 0x7fe2201607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21f59e640 1 -- 172.21.15.59:0/843940843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe220155660 con 0x7fe2200d2ce0 2024-06-29T15:59:35.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.390+0000 7fe21ed9d640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2200b2980 0x7fe2201f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:35.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.391+0000 7fe21f59e640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 0x7fe220156960 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7fe218009d60 tx=0x7fe218003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:35.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.391+0000 7fe2077fe640 1 -- 172.21.15.59:0/843940843 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218034040 con 0x7fe2200d2ce0 2024-06-29T15:59:35.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.391+0000 7fe2077fe640 1 -- 172.21.15.59:0/843940843 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe218004560 con 0x7fe2200d2ce0 2024-06-29T15:59:35.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.391+0000 7fe2077fe640 1 -- 172.21.15.59:0/843940843 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218045480 con 0x7fe2200d2ce0 2024-06-29T15:59:35.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.391+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe220155940 con 0x7fe2200d2ce0 2024-06-29T15:59:35.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.391+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe220155f00 con 0x7fe2200d2ce0 2024-06-29T15:59:35.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.392+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2200b4c50 con 0x7fe2200d2ce0 2024-06-29T15:59:35.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.393+0000 7fe2077fe640 1 -- 172.21.15.59:0/843940843 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 154) ==== 98247+0+0 (secure 0 0 0) 0x7fe218037020 con 0x7fe2200d2ce0 2024-06-29T15:59:35.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.393+0000 7fe2077fe640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe20c076360 0x7fe20c078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.393+0000 7fe2077fe640 1 -- 172.21.15.59:0/843940843 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(265..265 src has 1..265) ==== 7017+0+0 (secure 0 0 0) 0x7fe2180b94e0 con 0x7fe2200d2ce0 2024-06-29T15:59:35.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.394+0000 7fe21ed9d640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe20c076360 0x7fe20c078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.394+0000 7fe21ed9d640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe20c076360 0x7fe20c078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe210009ab0 tx=0x7fe210005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:35.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.398+0000 7fe2077fe640 1 -- 172.21.15.59:0/843940843 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe218082930 con 0x7fe2200d2ce0 2024-06-29T15:59:35.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:35.589+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe2200cdaa0 con 0x7fe2200d2ce0 2024-06-29T15:59:36.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.245+0000 7fe2077fe640 1 -- 172.21.15.59:0/843940843 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v266) ==== 181+0+0 (secure 0 0 0) 0x7fe218082230 con 0x7fe2200d2ce0 2024-06-29T15:59:36.245 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:59:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.252+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe20c076360 msgr2=0x7fe20c078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.252+0000 7fe225135640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe20c076360 0x7fe20c078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe210009ab0 tx=0x7fe210005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.252+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 msgr2=0x7fe220156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.252+0000 7fe2077fe640 1 -- 172.21.15.59:0/843940843 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fe21807e4c0 con 0x7fe2200d2ce0 2024-06-29T15:59:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.252+0000 7fe225135640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 0x7fe220156960 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7fe218009d60 tx=0x7fe218003f90 comp rx=0 tx=0).stop 2024-06-29T15:59:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.253+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 shutdown_connections 2024-06-29T15:59:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.253+0000 7fe225135640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe20c076360 0x7fe20c078870 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe210009ab0 tx=0x7fe210005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.253+0000 7fe225135640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2200d2ce0 0x7fe220156960 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.253+0000 7fe225135640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200d1c30 0x7fe2201607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.253+0000 7fe225135640 1 --2- 172.21.15.59:0/843940843 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe2200b2980 0x7fe2201f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.253+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 >> 172.21.15.59:0/843940843 conn(0x7fe2200d0f90 msgr2=0x7fe2200d3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.254+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 shutdown_connections 2024-06-29T15:59:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.254+0000 7fe225135640 1 -- 172.21.15.59:0/843940843 wait complete. 2024-06-29T15:59:36.268 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:59:36.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.401+0000 7f971e25e640 1 -- 172.21.15.59:0/1571456004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9704004800 msgr2=0x7f9704004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.401+0000 7f971e25e640 1 --2- 172.21.15.59:0/1571456004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9704004800 0x7f9704004c10 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f970000bb20 tx=0x7f9704005830 comp rx=0 tx=0).stop 2024-06-29T15:59:36.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.401+0000 7f9717fff640 1 -- 172.21.15.59:0/1571456004 reap_dead start 2024-06-29T15:59:36.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.402+0000 7f971e25e640 1 -- 172.21.15.59:0/1571456004 shutdown_connections 2024-06-29T15:59:36.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.402+0000 7f971e25e640 1 -- 172.21.15.59:0/1571456004 >> 172.21.15.59:0/1571456004 conn(0x7f971806cb30 msgr2=0x7f971806cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:36.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.402+0000 7f971e25e640 1 -- 172.21.15.59:0/1571456004 shutdown_connections 2024-06-29T15:59:36.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.402+0000 7f971e25e640 1 -- 172.21.15.59:0/1571456004 wait complete. 2024-06-29T15:59:36.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.403+0000 7f971e25e640 1 Processor -- start 2024-06-29T15:59:36.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.404+0000 7f971e25e640 1 -- start start 2024-06-29T15:59:36.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f971e25e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9704004800 0x7f9718145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:36.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f971e25e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f971813f6a0 0x7f9718145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:36.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f971e25e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9718141100 0x7f9718152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:36.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f971e25e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97180732e0 con 0x7f9718141100 2024-06-29T15:59:36.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f971e25e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9718073160 con 0x7f9704004800 2024-06-29T15:59:36.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f971e25e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9718073460 con 0x7f971813f6a0 2024-06-29T15:59:36.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9718141100 0x7f9718152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9718141100 0x7f9718152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53908/0 (socket says 172.21.15.59:53908) 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 -- 172.21.15.59:0/3616593311 learned_addr learned my addr 172.21.15.59:0/3616593311 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f97177fe640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9704004800 0x7f9718145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f970effd640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f971813f6a0 0x7f9718145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 -- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f971813f6a0 msgr2=0x7f9718145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f971813f6a0 0x7f9718145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 -- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9704004800 msgr2=0x7f9718145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9704004800 0x7f9718145910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 -- 172.21.15.59:0/3616593311 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9718140730 con 0x7f9718141100 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f97177fe640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9704004800 0x7f9718145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f970effd640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f971813f6a0 0x7f9718145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.405+0000 7f9717fff640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9718141100 0x7f9718152110 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f9710001330 tx=0x7f9710006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:36.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.406+0000 7f9714ff9640 1 -- 172.21.15.59:0/3616593311 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971000fc30 con 0x7f9718141100 2024-06-29T15:59:36.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.406+0000 7f9714ff9640 1 -- 172.21.15.59:0/3616593311 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9710019070 con 0x7f9718141100 2024-06-29T15:59:36.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.406+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97181549c0 con 0x7f9718141100 2024-06-29T15:59:36.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.406+0000 7f9714ff9640 1 -- 172.21.15.59:0/3616593311 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710011690 con 0x7f9718141100 2024-06-29T15:59:36.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.406+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9718153010 con 0x7f9718141100 2024-06-29T15:59:36.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.407+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9718152e10 con 0x7f9718141100 2024-06-29T15:59:36.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.407+0000 7f9714ff9640 1 -- 172.21.15.59:0/3616593311 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f9710011830 con 0x7f9718141100 2024-06-29T15:59:36.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.408+0000 7f9714ff9640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f96f80763f0 0x7f96f8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:36.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.408+0000 7f9714ff9640 1 -- 172.21.15.59:0/3616593311 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 6579+0+0 (secure 0 0 0) 0x7f9710098390 con 0x7f9718141100 2024-06-29T15:59:36.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.408+0000 7f97177fe640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f96f80763f0 0x7f96f8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:36.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.409+0000 7f97177fe640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f96f80763f0 0x7f96f8078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f97180cfbe0 tx=0x7f970800b2e0 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:36.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.412+0000 7f9714ff9640 1 -- 172.21.15.59:0/3616593311 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f97180b3d60 con 0x7f9718141100 2024-06-29T15:59:36.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:36.604+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f97180b3820 con 0x7f9718141100 2024-06-29T15:59:37.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.252+0000 7f9714ff9640 1 -- 172.21.15.59:0/3616593311 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v267) ==== 169+0+0 (secure 0 0 0) 0x7f9710061630 con 0x7f9718141100 2024-06-29T15:59:37.252 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T15:59:37.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f96f80763f0 msgr2=0x7f96f8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f96f80763f0 0x7f96f8078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f97180cfbe0 tx=0x7f970800b2e0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9718141100 msgr2=0x7f9718152110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9718141100 0x7f9718152110 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f9710001330 tx=0x7f9710006720 comp rx=0 tx=0).stop 2024-06-29T15:59:37.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 shutdown_connections 2024-06-29T15:59:37.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f96f80763f0 0x7f96f8078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9718141100 0x7f9718152110 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f971813f6a0 0x7f9718145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 --2- 172.21.15.59:0/3616593311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9704004800 0x7f9718145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 >> 172.21.15.59:0/3616593311 conn(0x7f971806cb30 msgr2=0x7f971814e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.256+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 shutdown_connections 2024-06-29T15:59:37.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.257+0000 7f971e25e640 1 -- 172.21.15.59:0/3616593311 wait complete. 2024-06-29T15:59:37.271 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:59:37.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.398+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3022089665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 msgr2=0x7f0c0400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.398+0000 7f0c1d2e4640 1 --2- 172.21.15.59:0/3022089665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 0x7f0c0400b9c0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f0c180b55a0 tx=0x7f0c04005830 comp rx=0 tx=0).stop 2024-06-29T15:59:37.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.398+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3022089665 shutdown_connections 2024-06-29T15:59:37.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.398+0000 7f0c1d2e4640 1 --2- 172.21.15.59:0/3022089665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 0x7f0c0400b9c0 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.398+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3022089665 >> 172.21.15.59:0/3022089665 conn(0x7f0c1806c950 msgr2=0x7f0c18073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.398+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3022089665 shutdown_connections 2024-06-29T15:59:37.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.398+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3022089665 wait complete. 2024-06-29T15:59:37.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.400+0000 7f0c1d2e4640 1 Processor -- start 2024-06-29T15:59:37.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.400+0000 7f0c1d2e4640 1 -- start start 2024-06-29T15:59:37.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c1d2e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 0x7f0c18139490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c1d2e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c18137e90 0x7f0c181382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c1d2e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c181387e0 0x7f0c18143980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c1d2e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1815c700 con 0x7f0c04005350 2024-06-29T15:59:37.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c1d2e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0c180b3fd0 con 0x7f0c18137e90 2024-06-29T15:59:37.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c1d2e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1815c880 con 0x7f0c181387e0 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c16ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 0x7f0c18139490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c16ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 0x7f0c18139490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53928/0 (socket says 172.21.15.59:53928) 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c16ffd640 1 -- 172.21.15.59:0/3249989702 learned_addr learned my addr 172.21.15.59:0/3249989702 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c16ffd640 1 -- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c181387e0 msgr2=0x7f0c18143980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c167fc640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c18137e90 0x7f0c181382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c177fe640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c181387e0 0x7f0c18143980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c16ffd640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c181387e0 0x7f0c18143980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c16ffd640 1 -- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c18137e90 msgr2=0x7f0c181382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.401+0000 7f0c16ffd640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c18137e90 0x7f0c181382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c16ffd640 1 -- 172.21.15.59:0/3249989702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c181595b0 con 0x7f0c04005350 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c167fc640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c18137e90 0x7f0c181382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c177fe640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c181387e0 0x7f0c18143980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c16ffd640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 0x7f0c18139490 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f0c10003f00 tx=0x7f0c10001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c0f7fe640 1 -- 172.21.15.59:0/3249989702 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c10011640 con 0x7f0c04005350 2024-06-29T15:59:37.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c0f7fe640 1 -- 172.21.15.59:0/3249989702 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c10016020 con 0x7f0c04005350 2024-06-29T15:59:37.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1806f1d0 con 0x7f0c04005350 2024-06-29T15:59:37.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c0f7fe640 1 -- 172.21.15.59:0/3249989702 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c10011a80 con 0x7f0c04005350 2024-06-29T15:59:37.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.402+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1806f9e0 con 0x7f0c04005350 2024-06-29T15:59:37.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.403+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c18159780 con 0x7f0c04005350 2024-06-29T15:59:37.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.404+0000 7f0c0f7fe640 1 -- 172.21.15.59:0/3249989702 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f0c10011c20 con 0x7f0c04005350 2024-06-29T15:59:37.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.404+0000 7f0c0f7fe640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f0bf8076420 0x7f0bf8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.404+0000 7f0c0f7fe640 1 -- 172.21.15.59:0/3249989702 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7f0c100987b0 con 0x7f0c04005350 2024-06-29T15:59:37.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.404+0000 7f0c167fc640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f0bf8076420 0x7f0bf8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.405+0000 7f0c167fc640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f0bf8076420 0x7f0bf8078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c00005e00 tx=0x7f0c00005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.409+0000 7f0c0f7fe640 1 -- 172.21.15.59:0/3249989702 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0c10020020 con 0x7f0c04005350 2024-06-29T15:59:37.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.603+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f0c180cf8c0 con 0x7f0c04005350 2024-06-29T15:59:37.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.607+0000 7f0c0f7fe640 1 -- 172.21.15.59:0/3249989702 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v83) ==== 71+0+0 (secure 0 0 0) 0x7f0c10061c90 con 0x7f0c04005350 2024-06-29T15:59:37.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f0bf8076420 msgr2=0x7f0bf8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f0bf8076420 0x7f0bf8078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c00005e00 tx=0x7f0c00005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:37.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 msgr2=0x7f0c18139490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 0x7f0c18139490 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f0c10003f00 tx=0x7f0c10001630 comp rx=0 tx=0).stop 2024-06-29T15:59:37.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 shutdown_connections 2024-06-29T15:59:37.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f0bf8076420 0x7f0bf8078930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c181387e0 0x7f0c18143980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c18137e90 0x7f0c181382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 --2- 172.21.15.59:0/3249989702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c04005350 0x7f0c18139490 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 >> 172.21.15.59:0/3249989702 conn(0x7f0c1806c950 msgr2=0x7f0c18071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 shutdown_connections 2024-06-29T15:59:37.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.610+0000 7f0c1d2e4640 1 -- 172.21.15.59:0/3249989702 wait complete. 2024-06-29T15:59:37.621 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-06-29T15:59:37.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.738+0000 7fd3beef6640 1 -- 172.21.15.59:0/2069852472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b81458a0 msgr2=0x7fd3b80cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.738+0000 7fd3beef6640 1 --2- 172.21.15.59:0/2069852472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b81458a0 0x7fd3b80cf870 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009830 tx=0x7fd3b002a5c0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.738+0000 7fd3beef6640 1 -- 172.21.15.59:0/2069852472 shutdown_connections 2024-06-29T15:59:37.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.738+0000 7fd3beef6640 1 --2- 172.21.15.59:0/2069852472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b81458a0 0x7fd3b80cf870 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.738+0000 7fd3beef6640 1 -- 172.21.15.59:0/2069852472 >> 172.21.15.59:0/2069852472 conn(0x7fd3b806c950 msgr2=0x7fd3b8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.738+0000 7fd3beef6640 1 -- 172.21.15.59:0/2069852472 shutdown_connections 2024-06-29T15:59:37.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.739+0000 7fd3beef6640 1 -- 172.21.15.59:0/2069852472 wait complete. 2024-06-29T15:59:37.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.741+0000 7fd3beef6640 1 Processor -- start 2024-06-29T15:59:37.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.741+0000 7fd3beef6640 1 -- start start 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.743+0000 7fd3beef6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b80b3740 0x7fd3b813e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.743+0000 7fd3beef6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3b80b3b50 0x7fd3b814eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.743+0000 7fd3beef6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3b813ab50 0x7fd3b814c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.744+0000 7fd3beef6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b80b6320 con 0x7fd3b80b3740 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.744+0000 7fd3beef6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b80b61a0 con 0x7fd3b80b3b50 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.744+0000 7fd3beef6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b80b64a0 con 0x7fd3b813ab50 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.744+0000 7fd3b7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3b80b3b50 0x7fd3b814eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.744+0000 7fd3b7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3b80b3b50 0x7fd3b814eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:48260/0 (socket says 172.21.15.59:48260) 2024-06-29T15:59:37.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.744+0000 7fd3b7fff640 1 -- 172.21.15.59:0/3104601592 learned_addr learned my addr 172.21.15.59:0/3104601592 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.744+0000 7fd3b7fff640 1 -- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3b813ab50 msgr2=0x7fd3b814c0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.744+0000 7fd3b7fff640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3b813ab50 0x7fd3b814c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3b7fff640 1 -- 172.21.15.59:0/3104601592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b80b3740 msgr2=0x7fd3b813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3bcc92640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b80b3740 0x7fd3b813e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3b7fff640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b80b3740 0x7fd3b813e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3b7fff640 1 -- 172.21.15.59:0/3104601592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b814f090 con 0x7fd3b80b3b50 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3bcc92640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b80b3740 0x7fd3b813e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3b7fff640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3b80b3b50 0x7fd3b814eb50 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800c7b0 tx=0x7fd3a800cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3b57fa640 1 -- 172.21.15.59:0/3104601592 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8003b70 con 0x7fd3b80b3b50 2024-06-29T15:59:37.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3b57fa640 1 -- 172.21.15.59:0/3104601592 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3a8003d10 con 0x7fd3b80b3b50 2024-06-29T15:59:37.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3b57fa640 1 -- 172.21.15.59:0/3104601592 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a800aab0 con 0x7fd3b80b3b50 2024-06-29T15:59:37.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.745+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b813d2d0 con 0x7fd3b80b3b50 2024-06-29T15:59:37.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.746+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b813d890 con 0x7fd3b80b3b50 2024-06-29T15:59:37.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.747+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b81458a0 con 0x7fd3b80b3b50 2024-06-29T15:59:37.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.748+0000 7fd3b57fa640 1 -- 172.21.15.59:0/3104601592 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fd3a800ac50 con 0x7fd3b80b3b50 2024-06-29T15:59:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.748+0000 7fd3b57fa640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd3980763f0 0x7fd398078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.748+0000 7fd3b57fa640 1 -- 172.21.15.59:0/3104601592 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7fd3a8097950 con 0x7fd3b80b3b50 2024-06-29T15:59:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.749+0000 7fd3bcc92640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd3980763f0 0x7fd398078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.749+0000 7fd3bcc92640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd3980763f0 0x7fd398078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0034000 tx=0x7fd3b0004730 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:37.753+0000 7fd3b57fa640 1 -- 172.21.15.59:0/3104601592 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd3a8060fd0 con 0x7fd3b80b3b50 2024-06-29T15:59:38.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.022+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) -- 0x7fd3b813d030 con 0x7fd3b80b3b50 2024-06-29T15:59:38.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.257+0000 7fd3b57fa640 1 -- 172.21.15.59:0/3104601592 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3245) ==== 155+0+0 (secure 0 0 0) 0x7fd3a80608d0 con 0x7fd3b80b3b50 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd3980763f0 msgr2=0x7fd398078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd3980763f0 0x7fd398078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0034000 tx=0x7fd3b0004730 comp rx=0 tx=0).stop 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3b80b3b50 msgr2=0x7fd3b814eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3b80b3b50 0x7fd3b814eb50 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800c7b0 tx=0x7fd3a800cc70 comp rx=0 tx=0).stop 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 shutdown_connections 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd3980763f0 0x7fd398078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd3b813ab50 0x7fd3b814c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3b80b3b50 0x7fd3b814eb50 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.259+0000 7fd3beef6640 1 --2- 172.21.15.59:0/3104601592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3b80b3740 0x7fd3b813e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.260+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 >> 172.21.15.59:0/3104601592 conn(0x7fd3b806c950 msgr2=0x7fd3b815f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:38.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.260+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 shutdown_connections 2024-06-29T15:59:38.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.260+0000 7fd3beef6640 1 -- 172.21.15.59:0/3104601592 wait complete. 2024-06-29T15:59:38.271 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-06-29T15:59:38.271 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol schedule creation succeeds for default subvolgroup. 2024-06-29T15:59:38.271 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T15:59:38.272 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:38.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.391+0000 7faaac12b640 1 -- 172.21.15.59:0/2213641475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 msgr2=0x7faa9000cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.391+0000 7faaac12b640 1 --2- 172.21.15.59:0/2213641475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 0x7faa9000cad0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7faa9c001dc0 tx=0x7faa90006940 comp rx=0 tx=0).stop 2024-06-29T15:59:38.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.391+0000 7faaac12b640 1 -- 172.21.15.59:0/2213641475 shutdown_connections 2024-06-29T15:59:38.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.391+0000 7faaac12b640 1 --2- 172.21.15.59:0/2213641475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 0x7faa9000cad0 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.391+0000 7faaac12b640 1 -- 172.21.15.59:0/2213641475 >> 172.21.15.59:0/2213641475 conn(0x7faaa40b24c0 msgr2=0x7faaa40cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:38.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.391+0000 7faaac12b640 1 -- 172.21.15.59:0/2213641475 shutdown_connections 2024-06-29T15:59:38.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.391+0000 7faaac12b640 1 -- 172.21.15.59:0/2213641475 wait complete. 2024-06-29T15:59:38.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.393+0000 7faaac12b640 1 Processor -- start 2024-06-29T15:59:38.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.393+0000 7faaac12b640 1 -- start start 2024-06-29T15:59:38.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaac12b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 0x7faaa414e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaac12b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaa41510f0 0x7faaa41f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaac12b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaa4145130 0x7faaa4152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaac12b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7faaa40b3c10 con 0x7faa90006460 2024-06-29T15:59:38.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaac12b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faaa40700b0 con 0x7faaa41510f0 2024-06-29T15:59:38.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaac12b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7faaa40b3d90 con 0x7faaa4145130 2024-06-29T15:59:38.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaa9ec7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 0x7faaa414e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaa9ec7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 0x7faaa414e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53974/0 (socket says 172.21.15.59:53974) 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaa9ec7640 1 -- 172.21.15.59:0/828091968 learned_addr learned my addr 172.21.15.59:0/828091968 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.394+0000 7faaa96c6640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaa41510f0 0x7faaa41f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaaa6c8640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaa4145130 0x7faaa4152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaa9ec7640 1 -- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaa4145130 msgr2=0x7faaa4152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaa9ec7640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaa4145130 0x7faaa4152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaa9ec7640 1 -- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaa41510f0 msgr2=0x7faaa41f0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaa9ec7640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaa41510f0 0x7faaa41f0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaa9ec7640 1 -- 172.21.15.59:0/828091968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa41f1430 con 0x7faa90006460 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaa96c6640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaa41510f0 0x7faaa41f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaa9ec7640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 0x7faaa414e9e0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7faaa000e640 tx=0x7faaa000eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:38.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faa9a7fc640 1 -- 172.21.15.59:0/828091968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa0015070 con 0x7faa90006460 2024-06-29T15:59:38.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faa9a7fc640 1 -- 172.21.15.59:0/828091968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faaa00109a0 con 0x7faa90006460 2024-06-29T15:59:38.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaa41518d0 con 0x7faa90006460 2024-06-29T15:59:38.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faa9a7fc640 1 -- 172.21.15.59:0/828091968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa0010df0 con 0x7faa90006460 2024-06-29T15:59:38.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.395+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaa4151e90 con 0x7faa90006460 2024-06-29T15:59:38.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.396+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaa4156400 con 0x7faa90006460 2024-06-29T15:59:38.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.397+0000 7faa9a7fc640 1 -- 172.21.15.59:0/828091968 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7faaa0018070 con 0x7faa90006460 2024-06-29T15:59:38.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.398+0000 7faa9a7fc640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7faa880763f0 0x7faa88078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.398+0000 7faa9a7fc640 1 -- 172.21.15.59:0/828091968 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7faaa0097e40 con 0x7faa90006460 2024-06-29T15:59:38.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.398+0000 7faaa96c6640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7faa880763f0 0x7faa88078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.398+0000 7faaa96c6640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7faa880763f0 0x7faa88078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faaa40cec10 tx=0x7faa94005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:38.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.402+0000 7faa9a7fc640 1 -- 172.21.15.59:0/828091968 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faaa0060c30 con 0x7faa90006460 2024-06-29T15:59:38.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.594+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faaa415bdc0 con 0x7faa90006460 2024-06-29T15:59:38.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.595+0000 7faa9a7fc640 1 -- 172.21.15.59:0/828091968 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v267) ==== 74+0+14549 (secure 0 0 0) 0x7faaa0064f30 con 0x7faa90006460 2024-06-29T15:59:38.594 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:38.595 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":267,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:37.246541+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6833/2985893835":"2024-06-30T15:59:34.235674+0000","172.21.15.59:6832/2985893835":"2024-06-30T15:59:34.235674+0000","172.21.15.153:0/376097765":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/3823156403":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/3656223405":"2024-06-30T15:59:31.842564+0000","172.21.15.153:6832/1744808854":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2017664585":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/623754277":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2546767818":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6833/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6832/3574471609":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2003505066":"2024-06-30T15:59:25.117274+0000","172.21.15.153:6833/1744808854":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2217399912":"2024-06-30T15:59:31.842564+0000","172.21.15.153:0/2250305531":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1734622549":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/1337010763":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2869442511":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/3703209251":"2024-06-30T15:59:25.117274+0000","172.21.15.153:0/2872771609":"2024-06-30T15:59:25.117274+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:38.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7faa880763f0 msgr2=0x7faa88078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7faa880763f0 0x7faa88078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faaa40cec10 tx=0x7faa94005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:38.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 msgr2=0x7faaa414e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 0x7faaa414e9e0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7faaa000e640 tx=0x7faaa000eb00 comp rx=0 tx=0).stop 2024-06-29T15:59:38.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 shutdown_connections 2024-06-29T15:59:38.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7faa880763f0 0x7faa88078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7faaa4145130 0x7faaa4152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaa41510f0 0x7faaa41f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 --2- 172.21.15.59:0/828091968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa90006460 0x7faaa414e9e0 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 >> 172.21.15.59:0/828091968 conn(0x7faaa40b24c0 msgr2=0x7faaa4072e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:38.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 shutdown_connections 2024-06-29T15:59:38.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.597+0000 7faaac12b640 1 -- 172.21.15.59:0/828091968 wait complete. 2024-06-29T15:59:38.608 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:38.608 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 -- 172.21.15.59:0/3440545777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 msgr2=0x7fb5781418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 --2- 172.21.15.59:0/3440545777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 0x7fb5781418d0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fb578069670 tx=0x7fb57002a980 comp rx=0 tx=0).stop 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 -- 172.21.15.59:0/3440545777 shutdown_connections 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 --2- 172.21.15.59:0/3440545777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 0x7fb5781418d0 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 --2- 172.21.15.59:0/3440545777 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb57813e9b0 0x7fb57813edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 --2- 172.21.15.59:0/3440545777 >> v2:172.21.15.153:3300/0 conn(0x7fb57815ceb0 0x7fb5780b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 -- 172.21.15.59:0/3440545777 >> 172.21.15.59:0/3440545777 conn(0x7fb5780b6fb0 msgr2=0x7fb5780b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 -- 172.21.15.59:0/3440545777 shutdown_connections 2024-06-29T15:59:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.729+0000 7fb57e622640 1 -- 172.21.15.59:0/3440545777 wait complete. 2024-06-29T15:59:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.730+0000 7fb57e622640 1 Processor -- start 2024-06-29T15:59:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.731+0000 7fb57e622640 1 -- start start 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb57e622640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb57813e9b0 0x7fb57813d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb57e622640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 0x7fb578134130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb57e622640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb57815ceb0 0x7fb578155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb57e622640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb5780b28c0 con 0x7fb5781414c0 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb57e622640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5780b2740 con 0x7fb57815ceb0 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb57e622640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb5780b2a40 con 0x7fb57813e9b0 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb5777fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 0x7fb578134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb5777fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 0x7fb578134130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53996/0 (socket says 172.21.15.59:53996) 2024-06-29T15:59:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb5777fe640 1 -- 172.21.15.59:0/3963471624 learned_addr learned my addr 172.21.15.59:0/3963471624 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb577fff640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb57813e9b0 0x7fb57813d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.732+0000 7fb57cbbf640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb57815ceb0 0x7fb578155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb5777fe640 1 -- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb57813e9b0 msgr2=0x7fb57813d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb5777fe640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb57813e9b0 0x7fb57813d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb5777fe640 1 -- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb57815ceb0 msgr2=0x7fb578155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb5777fe640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb57815ceb0 0x7fb578155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb5777fe640 1 -- 172.21.15.59:0/3963471624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57813d700 con 0x7fb5781414c0 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb5777fe640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 0x7fb578134130 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00d660 tx=0x7fb56c00db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb574ff9640 1 -- 172.21.15.59:0/3963471624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c017070 con 0x7fb5781414c0 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb574ff9640 1 -- 172.21.15.59:0/3963471624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb56c009ab0 con 0x7fb5781414c0 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5781f4830 con 0x7fb5781414c0 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb574ff9640 1 -- 172.21.15.59:0/3963471624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c010460 con 0x7fb5781414c0 2024-06-29T15:59:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.733+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5781f4df0 con 0x7fb5781414c0 2024-06-29T15:59:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.734+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb578050470 con 0x7fb5781414c0 2024-06-29T15:59:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.735+0000 7fb574ff9640 1 -- 172.21.15.59:0/3963471624 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fb56c01a070 con 0x7fb5781414c0 2024-06-29T15:59:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.735+0000 7fb574ff9640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb5640763f0 0x7fb564078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.735+0000 7fb574ff9640 1 -- 172.21.15.59:0/3963471624 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7fb56c097b10 con 0x7fb5781414c0 2024-06-29T15:59:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.736+0000 7fb577fff640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb5640763f0 0x7fb564078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.736+0000 7fb577fff640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb5640763f0 0x7fb564078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb568009b30 tx=0x7fb568005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:38.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.740+0000 7fb574ff9640 1 -- 172.21.15.59:0/3963471624 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb578050470 con 0x7fb5781414c0 2024-06-29T15:59:38.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.931+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb5780b2f60 con 0x7fb5781414c0 2024-06-29T15:59:38.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.931+0000 7fb574ff9640 1 -- 172.21.15.59:0/3963471624 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v267) ==== 93+0+10 (secure 0 0 0) 0x7fb57813d700 con 0x7fb5781414c0 2024-06-29T15:59:38.931 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:59:38.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb5640763f0 msgr2=0x7fb564078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb5640763f0 0x7fb564078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb568009b30 tx=0x7fb568005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:38.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 msgr2=0x7fb578134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 0x7fb578134130 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00d660 tx=0x7fb56c00db20 comp rx=0 tx=0).stop 2024-06-29T15:59:38.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 shutdown_connections 2024-06-29T15:59:38.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb5640763f0 0x7fb564078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb57815ceb0 0x7fb578155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5781414c0 0x7fb578134130 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 --2- 172.21.15.59:0/3963471624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb57813e9b0 0x7fb57813d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 >> 172.21.15.59:0/3963471624 conn(0x7fb5780b6fb0 msgr2=0x7fb5780b3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:38.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 shutdown_connections 2024-06-29T15:59:38.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:38.933+0000 7fb57e622640 1 -- 172.21.15.59:0/3963471624 wait complete. 2024-06-29T15:59:38.944 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-06-29T15:59:39.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.062+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/1571571028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0b3740 msgr2=0x7f9fac0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.062+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/1571571028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0b3740 0x7f9fac0b3b50 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0005fd0 tx=0x7f9fa002a530 comp rx=0 tx=0).stop 2024-06-29T15:59:39.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.062+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/1571571028 shutdown_connections 2024-06-29T15:59:39.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.062+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/1571571028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0b3740 0x7f9fac0b3b50 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.062+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/1571571028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fac1458a0 0x7f9fac0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.062+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/1571571028 >> 172.21.15.59:0/1571571028 conn(0x7f9fac06c950 msgr2=0x7f9fac073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:39.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.062+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/1571571028 shutdown_connections 2024-06-29T15:59:39.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.062+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/1571571028 wait complete. 2024-06-29T15:59:39.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.064+0000 7f9fb1ab3640 1 Processor -- start 2024-06-29T15:59:39.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.064+0000 7f9fb1ab3640 1 -- start start 2024-06-29T15:59:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.065+0000 7f9fb1ab3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fac0b3740 0x7f9fac15b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.065+0000 7f9fb1ab3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fac15c2f0 0x7f9fac06f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.065+0000 7f9fb1ab3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0af980 0x7f9fac0afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.065+0000 7f9fb1ab3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac1601a0 con 0x7f9fac0af980 2024-06-29T15:59:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.065+0000 7f9fb1ab3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac160020 con 0x7f9fac0b3740 2024-06-29T15:59:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.065+0000 7f9fb1ab3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9fac160320 con 0x7f9fac15c2f0 2024-06-29T15:59:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0af980 0x7f9fac0afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0af980 0x7f9fac0afd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54012/0 (socket says 172.21.15.59:54012) 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 -- 172.21.15.59:0/3276676762 learned_addr learned my addr 172.21.15.59:0/3276676762 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fab7fe640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fac0b3740 0x7f9fac15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9faaffd640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fac15c2f0 0x7f9fac06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 -- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fac15c2f0 msgr2=0x7f9fac06f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fac15c2f0 0x7f9fac06f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 -- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fac0b3740 msgr2=0x7f9fac15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fac0b3740 0x7f9fac15b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 -- 172.21.15.59:0/3276676762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fac0b02d0 con 0x7f9fac0af980 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9faaffd640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fac15c2f0 0x7f9fac06f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fabfff640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0af980 0x7f9fac0afd90 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400ba10 tx=0x7f9fa400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:39.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9f93fff640 1 -- 172.21.15.59:0/3276676762 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4019070 con 0x7f9fac0af980 2024-06-29T15:59:39.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9f93fff640 1 -- 172.21.15.59:0/3276676762 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fa400cbd0 con 0x7f9fac0af980 2024-06-29T15:59:39.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fac1f8670 con 0x7f9fac0af980 2024-06-29T15:59:39.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9f93fff640 1 -- 172.21.15.59:0/3276676762 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4014440 con 0x7f9fac0af980 2024-06-29T15:59:39.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.066+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fac1f8c30 con 0x7f9fac0af980 2024-06-29T15:59:39.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.068+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fac1458a0 con 0x7f9fac0af980 2024-06-29T15:59:39.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.068+0000 7f9f93fff640 1 -- 172.21.15.59:0/3276676762 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f9fa401c070 con 0x7f9fac0af980 2024-06-29T15:59:39.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.069+0000 7f9f93fff640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9f980763f0 0x7f9f98078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.069+0000 7f9f93fff640 1 -- 172.21.15.59:0/3276676762 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7f9fa4097720 con 0x7f9fac0af980 2024-06-29T15:59:39.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.069+0000 7f9fab7fe640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9f980763f0 0x7f9f98078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.070+0000 7f9fab7fe640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9f980763f0 0x7f9f98078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0058c0 tx=0x7f9f9c005d80 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.073+0000 7f9f93fff640 1 -- 172.21.15.59:0/3276676762 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9fac0b02d0 con 0x7f9fac0af980 2024-06-29T15:59:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:39.338+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) -- 0x7f9fac0b1f10 con 0x7f9fac0af980 2024-06-29T15:59:40.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.262+0000 7f9f93fff640 1 -- 172.21.15.59:0/3276676762 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3247) ==== 204+0+0 (secure 0 0 0) 0x7f9fa4060d10 con 0x7f9fac0af980 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9f980763f0 msgr2=0x7f9f98078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9f980763f0 0x7f9f98078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0058c0 tx=0x7f9f9c005d80 comp rx=0 tx=0).stop 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0af980 msgr2=0x7f9fac0afd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0af980 0x7f9fac0afd90 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400ba10 tx=0x7f9fa400bed0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 shutdown_connections 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9f980763f0 0x7f9f98078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0af980 0x7f9fac0afd90 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fac15c2f0 0x7f9fac06f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 --2- 172.21.15.59:0/3276676762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fac0b3740 0x7f9fac15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 >> 172.21.15.59:0/3276676762 conn(0x7f9fac06c950 msgr2=0x7f9fac073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:40.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 shutdown_connections 2024-06-29T15:59:40.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.265+0000 7f9fb1ab3640 1 -- 172.21.15.59:0/3276676762 wait complete. 2024-06-29T15:59:40.278 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:59:40.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.403+0000 7f7606f03640 1 -- 172.21.15.59:0/2401196776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76000d23c0 msgr2=0x7f76000b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.403+0000 7f7606f03640 1 --2- 172.21.15.59:0/2401196776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76000d23c0 0x7f76000b3990 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009830 tx=0x7f75f402a510 comp rx=0 tx=0).stop 2024-06-29T15:59:40.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.403+0000 7f7606f03640 1 -- 172.21.15.59:0/2401196776 shutdown_connections 2024-06-29T15:59:40.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.403+0000 7f7606f03640 1 --2- 172.21.15.59:0/2401196776 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f760015efe0 0x7f760015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.403+0000 7f7606f03640 1 --2- 172.21.15.59:0/2401196776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76000d23c0 0x7f76000b3990 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.403+0000 7f7606f03640 1 -- 172.21.15.59:0/2401196776 >> 172.21.15.59:0/2401196776 conn(0x7f76000d3ee0 msgr2=0x7f76000d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:40.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.403+0000 7f7606f03640 1 -- 172.21.15.59:0/2401196776 shutdown_connections 2024-06-29T15:59:40.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.403+0000 7f7606f03640 1 -- 172.21.15.59:0/2401196776 wait complete. 2024-06-29T15:59:40.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.405+0000 7f7606f03640 1 Processor -- start 2024-06-29T15:59:40.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.405+0000 7f7606f03640 1 -- start start 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f7606f03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76000b3ed0 0x7f7600151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f7606f03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76000d23c0 0x7f7600150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f7606f03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760015efe0 0x7f76001ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f7606f03640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7600160280 con 0x7f760015efe0 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f7606f03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7600160100 con 0x7f76000b3ed0 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f7606f03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7600160400 con 0x7f76000d23c0 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760015efe0 0x7f76001ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760015efe0 0x7f76001ea3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54034/0 (socket says 172.21.15.59:54034) 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 -- 172.21.15.59:0/2826816625 learned_addr learned my addr 172.21.15.59:0/2826816625 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f7604c9f640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76000b3ed0 0x7f7600151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f75fffff640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76000d23c0 0x7f7600150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 -- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76000d23c0 msgr2=0x7f7600150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76000d23c0 0x7f7600150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 -- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76000b3ed0 msgr2=0x7f7600151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76000b3ed0 0x7f7600151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 -- 172.21.15.59:0/2826816625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7600156810 con 0x7f760015efe0 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.407+0000 7f76054a0640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760015efe0 0x7f76001ea3d0 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f75f800e9c0 tx=0x7f75f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.408+0000 7f75fd7fa640 1 -- 172.21.15.59:0/2826816625 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f8004d20 con 0x7f760015efe0 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.408+0000 7f75fd7fa640 1 -- 172.21.15.59:0/2826816625 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75f800cb20 con 0x7f760015efe0 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.408+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7600156af0 con 0x7f760015efe0 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.408+0000 7f75fd7fa640 1 -- 172.21.15.59:0/2826816625 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f8005020 con 0x7f760015efe0 2024-06-29T15:59:40.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.408+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76001570b0 con 0x7f760015efe0 2024-06-29T15:59:40.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.409+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7600152540 con 0x7f760015efe0 2024-06-29T15:59:40.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.410+0000 7f75fd7fa640 1 -- 172.21.15.59:0/2826816625 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f75f80051c0 con 0x7f760015efe0 2024-06-29T15:59:40.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.410+0000 7f75fd7fa640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f75e40763f0 0x7f75e4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.410+0000 7f75fd7fa640 1 -- 172.21.15.59:0/2826816625 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7f75f80983f0 con 0x7f760015efe0 2024-06-29T15:59:40.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.411+0000 7f7604c9f640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f75e40763f0 0x7f75e4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.411+0000 7f7604c9f640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f75e40763f0 0x7f75e4078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75f4034000 tx=0x7f75f4004730 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:40.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.415+0000 7f75fd7fa640 1 -- 172.21.15.59:0/2826816625 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f75f8016030 con 0x7f760015efe0 2024-06-29T15:59:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.607+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f76000b3990 con 0x7f760015efe0 2024-06-29T15:59:40.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.607+0000 7f75fd7fa640 1 -- 172.21.15.59:0/2826816625 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v83) ==== 59+0+1998 (secure 0 0 0) 0x7f75f8061310 con 0x7f760015efe0 2024-06-29T15:59:40.607 INFO:teuthology.orchestra.run.smithi059.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:59:40.607 INFO:teuthology.orchestra.run.smithi059.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:59:40.607 INFO:teuthology.orchestra.run.smithi059.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4928.900892 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5125.045752 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 8241.731977 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 5077.580857 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5137.605013 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 6364.727893 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 4577.627219 2024-06-29T15:59:40.608 INFO:teuthology.orchestra.run.smithi059.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 3954.939388 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds 20/20 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced debug_ms 1/1 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_frag true 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds dev mds_verify_scatter true 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced client_mount_timeout 600 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_client 20/20 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced debug_ms 1/1 2024-06-29T15:59:40.609 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:59:40.610 INFO:teuthology.orchestra.run.smithi059.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:59:40.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f75e40763f0 msgr2=0x7f75e4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f75e40763f0 0x7f75e4078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75f4034000 tx=0x7f75f4004730 comp rx=0 tx=0).stop 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760015efe0 msgr2=0x7f76001ea3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760015efe0 0x7f76001ea3d0 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f75f800e9c0 tx=0x7f75f800ee80 comp rx=0 tx=0).stop 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 shutdown_connections 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f75e40763f0 0x7f75e4078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760015efe0 0x7f76001ea3d0 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f76000d23c0 0x7f7600150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 --2- 172.21.15.59:0/2826816625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76000b3ed0 0x7f7600151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 >> 172.21.15.59:0/2826816625 conn(0x7f76000d3ee0 msgr2=0x7f760006cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 shutdown_connections 2024-06-29T15:59:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.609+0000 7f7606f03640 1 -- 172.21.15.59:0/2826816625 wait complete. 2024-06-29T15:59:40.620 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:59:40.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.741+0000 7f458e9b9640 1 -- 172.21.15.59:0/2344263384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2270 msgr2=0x7f458815bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.741+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2344263384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2270 0x7f458815bc20 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f45880cf310 tx=0x7f457802a560 comp rx=0 tx=0).stop 2024-06-29T15:59:40.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.741+0000 7f458e9b9640 1 -- 172.21.15.59:0/2344263384 shutdown_connections 2024-06-29T15:59:40.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.741+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2344263384 >> v2:172.21.15.153:3300/0 conn(0x7f45880d1c30 0x7f45880b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.741+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2344263384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2270 0x7f458815bc20 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.741+0000 7f458e9b9640 1 -- 172.21.15.59:0/2344263384 >> 172.21.15.59:0/2344263384 conn(0x7f4588073e10 msgr2=0x7f4588074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:40.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.741+0000 7f458e9b9640 1 -- 172.21.15.59:0/2344263384 shutdown_connections 2024-06-29T15:59:40.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.741+0000 7f458e9b9640 1 -- 172.21.15.59:0/2344263384 wait complete. 2024-06-29T15:59:40.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.743+0000 7f458e9b9640 1 Processor -- start 2024-06-29T15:59:40.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.743+0000 7f458e9b9640 1 -- start start 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.744+0000 7f458e9b9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45880b2270 0x7f4588146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.744+0000 7f458e9b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2fa0 0x7f45880aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f458e9b9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45880d1c30 0x7f458815aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f458e9b9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f45880b6400 con 0x7f45880b2fa0 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f458e9b9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45880b6280 con 0x7f45880d1c30 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f458e9b9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45880b6580 con 0x7f45880b2270 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2fa0 0x7f45880aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2fa0 0x7f45880aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54048/0 (socket says 172.21.15.59:54048) 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 -- 172.21.15.59:0/2669878525 learned_addr learned my addr 172.21.15.59:0/2669878525 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f4587fff640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45880b2270 0x7f4588146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f458cf56640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45880d1c30 0x7f458815aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 -- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45880b2270 msgr2=0x7f4588146bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45880b2270 0x7f4588146bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 -- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45880d1c30 msgr2=0x7f458815aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45880d1c30 0x7f458815aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 -- 172.21.15.59:0/2669878525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4588147130 con 0x7f45880b2fa0 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f45877fe640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2fa0 0x7f45880aa4f0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f457c00b3f0 tx=0x7f457c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f4584ff9640 1 -- 172.21.15.59:0/2669878525 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c015020 con 0x7f45880b2fa0 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f4584ff9640 1 -- 172.21.15.59:0/2669878525 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f457c011920 con 0x7f45880b2fa0 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45881567a0 con 0x7f45880b2fa0 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f4584ff9640 1 -- 172.21.15.59:0/2669878525 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c011d70 con 0x7f45880b2fa0 2024-06-29T15:59:40.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.745+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45880ab340 con 0x7f45880b2fa0 2024-06-29T15:59:40.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.746+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45880aad80 con 0x7f45880b2fa0 2024-06-29T15:59:40.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.747+0000 7f4584ff9640 1 -- 172.21.15.59:0/2669878525 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f457c018020 con 0x7f45880b2fa0 2024-06-29T15:59:40.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.748+0000 7f4584ff9640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4574076420 0x7f4574078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.748+0000 7f4584ff9640 1 -- 172.21.15.59:0/2669878525 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7f457c097910 con 0x7f45880b2fa0 2024-06-29T15:59:40.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.748+0000 7f4587fff640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4574076420 0x7f4574078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.749+0000 7f4587fff640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4574076420 0x7f4574078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4578033000 tx=0x7f4578004730 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:40.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.752+0000 7f4584ff9640 1 -- 172.21.15.59:0/2669878525 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f457c01f020 con 0x7f45880b2fa0 2024-06-29T15:59:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.942+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f45881593d0 con 0x7f45880b2fa0 2024-06-29T15:59:40.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.948+0000 7f4584ff9640 1 -- 172.21.15.59:0/2669878525 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v84)=0 v84) ==== 122+0+0 (secure 0 0 0) 0x7f457c060940 con 0x7f45880b2fa0 2024-06-29T15:59:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4574076420 msgr2=0x7f4574078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4574076420 0x7f4574078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4578033000 tx=0x7f4578004730 comp rx=0 tx=0).stop 2024-06-29T15:59:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2fa0 msgr2=0x7f45880aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2fa0 0x7f45880aa4f0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f457c00b3f0 tx=0x7f457c00b8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 shutdown_connections 2024-06-29T15:59:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4574076420 0x7f4574078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45880d1c30 0x7f458815aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45880b2fa0 0x7f45880aa4f0 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 --2- 172.21.15.59:0/2669878525 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45880b2270 0x7f4588146bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 >> 172.21.15.59:0/2669878525 conn(0x7f4588073e10 msgr2=0x7f45880b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 shutdown_connections 2024-06-29T15:59:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:40.950+0000 7f458e9b9640 1 -- 172.21.15.59:0/2669878525 wait complete. 2024-06-29T15:59:40.961 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:59:40.961 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:41.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:41.058 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:59:41.059 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:41.059 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:41.075 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:41.075 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:59:41.075 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:41.116 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:41.117 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:41.117 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:41.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:41.133 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T15:59:41.133 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:41.174 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:41.174 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:59:41.174 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.292+0000 7facf76e7640 1 -- 172.21.15.59:0/2134493199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf00cf4d0 msgr2=0x7facf00cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.292+0000 7facf76e7640 1 --2- 172.21.15.59:0/2134493199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf00cf4d0 0x7facf00cce70 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7face0005fd0 tx=0x7face002a820 comp rx=0 tx=0).stop 2024-06-29T15:59:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.292+0000 7facf76e7640 1 -- 172.21.15.59:0/2134493199 shutdown_connections 2024-06-29T15:59:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.292+0000 7facf76e7640 1 --2- 172.21.15.59:0/2134493199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facf0073830 0x7facf00b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.292+0000 7facf76e7640 1 --2- 172.21.15.59:0/2134493199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf00cf4d0 0x7facf00cce70 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.292+0000 7facf76e7640 1 -- 172.21.15.59:0/2134493199 >> 172.21.15.59:0/2134493199 conn(0x7facf00b1be0 msgr2=0x7facf00b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.292+0000 7facf76e7640 1 -- 172.21.15.59:0/2134493199 shutdown_connections 2024-06-29T15:59:41.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.292+0000 7facf76e7640 1 -- 172.21.15.59:0/2134493199 wait complete. 2024-06-29T15:59:41.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.294+0000 7facf76e7640 1 Processor -- start 2024-06-29T15:59:41.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.294+0000 7facf76e7640 1 -- start start 2024-06-29T15:59:41.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf76e7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facf0073830 0x7facf00afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf76e7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf00b2ee0 0x7facf01f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf76e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facf00cf4d0 0x7facf01f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf76e7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7facf015faa0 con 0x7facf00cf4d0 2024-06-29T15:59:41.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf76e7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7facf015f920 con 0x7facf00b2ee0 2024-06-29T15:59:41.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf76e7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7facf015fc20 con 0x7facf0073830 2024-06-29T15:59:41.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facf00cf4d0 0x7facf01f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facf00cf4d0 0x7facf01f5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54050/0 (socket says 172.21.15.59:54050) 2024-06-29T15:59:41.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 -- 172.21.15.59:0/2189192811 learned_addr learned my addr 172.21.15.59:0/2189192811 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:41.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf4c82640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf00b2ee0 0x7facf01f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5483640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facf0073830 0x7facf00afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 -- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facf0073830 msgr2=0x7facf00afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facf0073830 0x7facf00afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 -- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf00b2ee0 msgr2=0x7facf01f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf00b2ee0 0x7facf01f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 -- 172.21.15.59:0/2189192811 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf015e1a0 con 0x7facf00cf4d0 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facf5c84640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facf00cf4d0 0x7facf01f5320 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7face800e890 tx=0x7face800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.296+0000 7facedffb640 1 -- 172.21.15.59:0/2189192811 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face800ccc0 con 0x7facf00cf4d0 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.297+0000 7facedffb640 1 -- 172.21.15.59:0/2189192811 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7face800ce60 con 0x7facf00cf4d0 2024-06-29T15:59:41.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.297+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf00b0780 con 0x7facf00cf4d0 2024-06-29T15:59:41.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.297+0000 7facedffb640 1 -- 172.21.15.59:0/2189192811 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8004270 con 0x7facf00cf4d0 2024-06-29T15:59:41.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.297+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf00b0d40 con 0x7facf00cf4d0 2024-06-29T15:59:41.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.298+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf00b1820 con 0x7facf00cf4d0 2024-06-29T15:59:41.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.299+0000 7facedffb640 1 -- 172.21.15.59:0/2189192811 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7face80035f0 con 0x7facf00cf4d0 2024-06-29T15:59:41.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.299+0000 7facedffb640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7facdc0763f0 0x7facdc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.299+0000 7facedffb640 1 -- 172.21.15.59:0/2189192811 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7face80983a0 con 0x7facf00cf4d0 2024-06-29T15:59:41.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.299+0000 7facf5483640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7facdc0763f0 0x7facdc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.300+0000 7facf5483640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7facdc0763f0 0x7facdc078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7face0034000 tx=0x7face0004ab0 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:41.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.303+0000 7facedffb640 1 -- 172.21.15.59:0/2189192811 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7face8061240 con 0x7facf00cf4d0 2024-06-29T15:59:41.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.534+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facf00b32f0 con 0x7facf00cf4d0 2024-06-29T15:59:41.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.535+0000 7facedffb640 1 -- 172.21.15.59:0/2189192811 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+3926 (secure 0 0 0) 0x7face8065540 con 0x7facf00cf4d0 2024-06-29T15:59:41.536 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:41.536 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":150,"btime":"2024-06-29T15:59:35:239184+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6836/1010984928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1010984928},{"type":"v1","addr":"172.21.15.153:6836","nonce":1010984928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":10475,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/2101409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6835","nonce":2101409892},{"type":"v1","addr":"172.21.15.153:6837","nonce":2101409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":12130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1208243017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1208243017},{"type":"v1","addr":"172.21.15.59:6835","nonce":1208243017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":143},{"gid":13530,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3094290503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3094290503},{"type":"v1","addr":"172.21.15.59:6833","nonce":3094290503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":150}],"filesystems":[]} 2024-06-29T15:59:41.536 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 150 2024-06-29T15:59:41.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7facdc0763f0 msgr2=0x7facdc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7facdc0763f0 0x7facdc078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7face0034000 tx=0x7face0004ab0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facf00cf4d0 msgr2=0x7facf01f5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facf00cf4d0 0x7facf01f5320 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7face800e890 tx=0x7face800ed50 comp rx=0 tx=0).stop 2024-06-29T15:59:41.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 shutdown_connections 2024-06-29T15:59:41.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7facdc0763f0 0x7facdc078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7facf00cf4d0 0x7facf01f5320 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facf00b2ee0 0x7facf01f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 --2- 172.21.15.59:0/2189192811 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7facf0073830 0x7facf00afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 >> 172.21.15.59:0/2189192811 conn(0x7facf00b1be0 msgr2=0x7facf00ceac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:41.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 shutdown_connections 2024-06-29T15:59:41.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.537+0000 7facf76e7640 1 -- 172.21.15.59:0/2189192811 wait complete. 2024-06-29T15:59:41.548 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:59:41.548 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:59:41.549 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:59:41.549 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:41.549 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:59:41.549 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:59:41.549 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:59:41.549 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:41.549 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:59:41.549 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:59:41.550 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:59:41.550 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:41.550 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:59:41.550 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:59:41.550 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:59:41.550 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:41.558 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:59:41.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:59:41.560 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:59:41.560 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:59:41.562 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:59:41.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:59:41.563 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:59:41.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:59:41.602 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:59:41.603 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:41.604 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:59:41.604 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:41.605 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:59:41.605 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:41.606 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:59:41.606 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:41.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:59:41.695 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-06-29T15:59:41.696 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T15:59:41.697 INFO:tasks.ceph.mds.b.smithi153.stdout:starting mds.b at 2024-06-29T15:59:41.698 INFO:tasks.ceph.mds.d.smithi153.stdout:starting mds.d at 2024-06-29T15:59:41.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.731+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/4164162161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80b40a0 msgr2=0x7f5ff80b73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.731+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/4164162161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80b40a0 0x7f5ff80b73f0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0005fd0 tx=0x7f5ff002a900 comp rx=0 tx=0).stop 2024-06-29T15:59:41.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.731+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/4164162161 shutdown_connections 2024-06-29T15:59:41.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.731+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/4164162161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80b40a0 0x7f5ff80b73f0 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.731+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/4164162161 >> v2:172.21.15.153:3300/0 conn(0x7f5ff815e290 0x7f5ff80ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.731+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/4164162161 >> 172.21.15.59:0/4164162161 conn(0x7f5ff80d3ee0 msgr2=0x7f5ff80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:41.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.731+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/4164162161 shutdown_connections 2024-06-29T15:59:41.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.731+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/4164162161 wait complete. 2024-06-29T15:59:41.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.733+0000 7f5ffd7f0640 1 Processor -- start 2024-06-29T15:59:41.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.733+0000 7f5ffd7f0640 1 -- start start 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.734+0000 7f5ffd7f0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80ce650 0x7f5ff8155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.734+0000 7f5ffd7f0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ff815e290 0x7f5ff81f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.734+0000 7f5ffd7f0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff81f5290 0x7f5ff81f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.734+0000 7f5ffd7f0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff80b7cc0 con 0x7f5ff80ce650 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.734+0000 7f5ffd7f0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff80b7b40 con 0x7f5ff815e290 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.734+0000 7f5ffd7f0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5ff80b7e40 con 0x7f5ff81f5290 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80ce650 0x7f5ff8155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80ce650 0x7f5ff8155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54112/0 (socket says 172.21.15.59:54112) 2024-06-29T15:59:41.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 -- 172.21.15.59:0/2721950476 learned_addr learned my addr 172.21.15.59:0/2721950476 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff67fc640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ff815e290 0x7f5ff81f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff77fe640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff81f5290 0x7f5ff81f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 -- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff81f5290 msgr2=0x7f5ff81f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff81f5290 0x7f5ff81f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 -- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ff815e290 msgr2=0x7f5ff81f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ff815e290 0x7f5ff81f4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 -- 172.21.15.59:0/2721950476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff815f960 con 0x7f5ff80ce650 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff67fc640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ff815e290 0x7f5ff81f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ff6ffd640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80ce650 0x7f5ff8155fc0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f5fe800d740 tx=0x7f5fe800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/2721950476 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe8017070 con 0x7f5ff80ce650 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/2721950476 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fe8009d20 con 0x7f5ff80ce650 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff8158b40 con 0x7f5ff80ce650 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.735+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/2721950476 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe80106c0 con 0x7f5ff80ce650 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.736+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ff815fc00 con 0x7f5ff80ce650 2024-06-29T15:59:41.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.737+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff80b8b20 con 0x7f5ff80ce650 2024-06-29T15:59:41.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.737+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/2721950476 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f5fe801a070 con 0x7f5ff80ce650 2024-06-29T15:59:41.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.738+0000 7f5fdf7fe640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5fe40763f0 0x7f5fe4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.738+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/2721950476 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6165+0+0 (secure 0 0 0) 0x7f5fe8097cb0 con 0x7f5ff80ce650 2024-06-29T15:59:41.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.738+0000 7f5ff67fc640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5fe40763f0 0x7f5fe4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.739+0000 7f5ff67fc640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5fe40763f0 0x7f5fe4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80b4370 tx=0x7f5fec009040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.742+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/2721950476 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5fe8061220 con 0x7f5ff80ce650 2024-06-29T15:59:41.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:41.946+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5ff80b9950 con 0x7f5ff80ce650 2024-06-29T15:59:42.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.281+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/2721950476 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v269) ==== 97+0+0 (secure 0 0 0) 0x7f5fe8060b20 con 0x7f5ff80ce650 2024-06-29T15:59:42.281 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-06-29T15:59:42.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5fe40763f0 msgr2=0x7f5fe4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5fe40763f0 0x7f5fe4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80b4370 tx=0x7f5fec009040 comp rx=0 tx=0).stop 2024-06-29T15:59:42.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80ce650 msgr2=0x7f5ff8155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80ce650 0x7f5ff8155fc0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f5fe800d740 tx=0x7f5fe800dc00 comp rx=0 tx=0).stop 2024-06-29T15:59:42.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 shutdown_connections 2024-06-29T15:59:42.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f5fe40763f0 0x7f5fe4078900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5ff81f5290 0x7f5ff81f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ff815e290 0x7f5ff81f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 --2- 172.21.15.59:0/2721950476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ff80ce650 0x7f5ff8155fc0 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 >> 172.21.15.59:0/2721950476 conn(0x7f5ff80d3ee0 msgr2=0x7f5ff815b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:42.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 shutdown_connections 2024-06-29T15:59:42.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.284+0000 7f5ffd7f0640 1 -- 172.21.15.59:0/2721950476 wait complete. 2024-06-29T15:59:42.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:59:42.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.416+0000 7f034b864640 1 -- 172.21.15.59:0/1156480529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 msgr2=0x7f03440b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.416+0000 7f034b864640 1 --2- 172.21.15.59:0/1156480529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 0x7f03440b1b00 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f0334005fd0 tx=0x7f033402aa90 comp rx=0 tx=0).stop 2024-06-29T15:59:42.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.416+0000 7f034b864640 1 -- 172.21.15.59:0/1156480529 shutdown_connections 2024-06-29T15:59:42.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.416+0000 7f034b864640 1 --2- 172.21.15.59:0/1156480529 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f03440b2040 0x7f034406e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.416+0000 7f034b864640 1 --2- 172.21.15.59:0/1156480529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 0x7f03440b1b00 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.416+0000 7f034b864640 1 -- 172.21.15.59:0/1156480529 >> 172.21.15.59:0/1156480529 conn(0x7f03440b06e0 msgr2=0x7f03440b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:42.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.416+0000 7f034b864640 1 -- 172.21.15.59:0/1156480529 shutdown_connections 2024-06-29T15:59:42.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.416+0000 7f034b864640 1 -- 172.21.15.59:0/1156480529 wait complete. 2024-06-29T15:59:42.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.418+0000 7f034b864640 1 Processor -- start 2024-06-29T15:59:42.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.418+0000 7f034b864640 1 -- start start 2024-06-29T15:59:42.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.419+0000 7f034b864640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034406e840 0x7f034413f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.419+0000 7f034b864640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03440b2040 0x7f034415a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f034b864640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 0x7f03440a9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f034b864640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f03440d0360 con 0x7f0344144470 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f034b864640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03440d01e0 con 0x7f03440b2040 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f034b864640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f034415ee80 con 0x7f034406e840 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 0x7f03440a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 0x7f03440a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54140/0 (socket says 172.21.15.59:54140) 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 -- 172.21.15.59:0/1990458721 learned_addr learned my addr 172.21.15.59:0/1990458721 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0348dff640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03440b2040 0x7f034415a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349600640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034406e840 0x7f034413f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 -- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034406e840 msgr2=0x7f034413f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034406e840 0x7f034413f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 -- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03440b2040 msgr2=0x7f034415a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03440b2040 0x7f034415a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 -- 172.21.15.59:0/1990458721 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0344159130 con 0x7f0344144470 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0349e01640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 0x7f03440a9860 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f033c00c3c0 tx=0x7f033c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0341ffb640 1 -- 172.21.15.59:0/1990458721 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c017070 con 0x7f0344144470 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.420+0000 7f0341ffb640 1 -- 172.21.15.59:0/1990458721 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f033c0053c0 con 0x7f0344144470 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.421+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0344159410 con 0x7f0344144470 2024-06-29T15:59:42.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.421+0000 7f0341ffb640 1 -- 172.21.15.59:0/1990458721 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c003830 con 0x7f0344144470 2024-06-29T15:59:42.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.421+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03441599d0 con 0x7f0344144470 2024-06-29T15:59:42.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.422+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034415f350 con 0x7f0344144470 2024-06-29T15:59:42.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.422+0000 7f0341ffb640 1 -- 172.21.15.59:0/1990458721 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f033c01a070 con 0x7f0344144470 2024-06-29T15:59:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.423+0000 7f0341ffb640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f03300763f0 0x7f0330078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.423+0000 7f0341ffb640 1 -- 172.21.15.59:0/1990458721 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5305+0+0 (secure 0 0 0) 0x7f033c096b10 con 0x7f0344144470 2024-06-29T15:59:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.423+0000 7f0349600640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f03300763f0 0x7f0330078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.424+0000 7f0349600640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f03300763f0 0x7f0330078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0334035000 tx=0x7f0334038040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.428+0000 7f0341ffb640 1 -- 172.21.15.59:0/1990458721 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f03440b2450 con 0x7f0344144470 2024-06-29T15:59:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.694+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f034415b550 con 0x7f0344144470 2024-06-29T15:59:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.694+0000 7f0341ffb640 1 -- 172.21.15.59:0/1990458721 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v54) ==== 80+0+5304 (secure 0 0 0) 0x7f033c05fd70 con 0x7f0344144470 2024-06-29T15:59:42.694 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmcCXGCxAAWVT/1iAOH8oI26tYsTDd8w==", 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.695 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3YThGBAAXm6qwAceQ82wI2y3lVPX1Q==", 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.696 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmI+CMExAApa0ypZj8qzvMApGQiu9Acg==", 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBms86SIBAAtmb2kLTmF3tZVZ+EyZw+HA==", 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.697 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBLGIBmf6t2BhAAlV7yJuj2Lx4goPqs2gnGag==", 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.698 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBOGIBmxg+cBhAAByh+jeqN3EAt6aSAtbizbA==", 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBRGIBmf23JBRAAJYC/fH+1gDBspAM8gm7oOQ==", 2024-06-29T15:59:42.699 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGIBmpzDJBBAA2uDJruOZ9Fmtee9RxEQZzw==", 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.700 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGIBm1un0DxAAdZ/DP4bsoL9QbcoU4cfmMw==", 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.701 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGIBmdOlDDhAAvD3HLKel8/jDjry5hZ6n6Q==", 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGIBmvjGtDBAA2M2B33ukHWNFaJ9rrSfkgA==", 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:42.702 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:42.703 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.703 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.703 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.703 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-06-29T15:59:42.703 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBjGIBm+v35ChAAquO1+AeuiLaDV+Ds2twCjg==", 2024-06-29T15:59:42.703 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg==", 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.704 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm3asKLRAA9xVyvxVIDnhNdD8xqFDGSA==", 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmzew/KRAAp9qWJk3Ljb4WJYoh6XaX4Q==", 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.705 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBm5WHoMBAAaUYY8UUyhQ35tMlIe1p3DA==", 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-06-29T15:59:42.706 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBGGIBmQO5aKRAASmxbDl+zfO2mkyecllJQMA==", 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.707 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmUk/eBxAALdIAGOec8rXHx6z8XXdJzw==", 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBHGIBmdMxdAxAAkTc0If2tXrjmiPo02WW6Eg==", 2024-06-29T15:59:42.708 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-06-29T15:59:42.709 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-06-29T15:59:42.709 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-06-29T15:59:42.709 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-06-29T15:59:42.709 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.709 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:42.709 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-06-29T15:59:42.709 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f03300763f0 msgr2=0x7f0330078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f03300763f0 0x7f0330078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0334035000 tx=0x7f0334038040 comp rx=0 tx=0).stop 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 msgr2=0x7f03440a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 0x7f03440a9860 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f033c00c3c0 tx=0x7f033c00c880 comp rx=0 tx=0).stop 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 shutdown_connections 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f03300763f0 0x7f0330078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0344144470 0x7f03440a9860 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03440b2040 0x7f034415a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 --2- 172.21.15.59:0/1990458721 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f034406e840 0x7f034413f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 >> 172.21.15.59:0/1990458721 conn(0x7f03440b06e0 msgr2=0x7f034415ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 shutdown_connections 2024-06-29T15:59:42.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.697+0000 7f034b864640 1 -- 172.21.15.59:0/1990458721 wait complete. 2024-06-29T15:59:42.712 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:42.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.831+0000 7f52c85eb640 1 -- 172.21.15.59:0/3820303475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52ac004800 msgr2=0x7f52ac004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.831+0000 7f52c85eb640 1 --2- 172.21.15.59:0/3820303475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52ac004800 0x7f52ac004c10 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f52b0001690 tx=0x7f52ac005830 comp rx=0 tx=0).stop 2024-06-29T15:59:42.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.831+0000 7f52c85eb640 1 -- 172.21.15.59:0/3820303475 shutdown_connections 2024-06-29T15:59:42.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.831+0000 7f52c85eb640 1 --2- 172.21.15.59:0/3820303475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52ac004800 0x7f52ac004c10 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.831+0000 7f52c85eb640 1 -- 172.21.15.59:0/3820303475 >> 172.21.15.59:0/3820303475 conn(0x7f52c00b5610 msgr2=0x7f52c00b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:42.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.831+0000 7f52c85eb640 1 -- 172.21.15.59:0/3820303475 shutdown_connections 2024-06-29T15:59:42.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.831+0000 7f52c85eb640 1 -- 172.21.15.59:0/3820303475 wait complete. 2024-06-29T15:59:42.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.833+0000 7f52c85eb640 1 Processor -- start 2024-06-29T15:59:42.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.833+0000 7f52c85eb640 1 -- start start 2024-06-29T15:59:42.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c85eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52ac004800 0x7f52c0158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c85eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c0154e10 0x7f52c0156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c85eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c0155220 0x7f52c0156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c85eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f52c00d1a20 con 0x7f52c0154e10 2024-06-29T15:59:42.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c85eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f52c00cf3a0 con 0x7f52c0155220 2024-06-29T15:59:42.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c85eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f52c00d1ba0 con 0x7f52ac004800 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c5b86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c0154e10 0x7f52c0156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c5b86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c0154e10 0x7f52c0156440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60978/0 (socket says 172.21.15.59:60978) 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c5b86640 1 -- 172.21.15.59:0/3814146208 learned_addr learned my addr 172.21.15.59:0/3814146208 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c6387640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52ac004800 0x7f52c0158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c6b88640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c0155220 0x7f52c0156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c5b86640 1 -- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52ac004800 msgr2=0x7f52c0158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c5b86640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52ac004800 0x7f52c0158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c5b86640 1 -- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c0155220 msgr2=0x7f52c0156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c5b86640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c0155220 0x7f52c0156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.834+0000 7f52c5b86640 1 -- 172.21.15.59:0/3814146208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c00cacf0 con 0x7f52c0154e10 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.835+0000 7f52c5b86640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c0154e10 0x7f52c0156440 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f52b00017b0 tx=0x7f52b00063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.835+0000 7f52b6ffd640 1 -- 172.21.15.59:0/3814146208 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0042070 con 0x7f52c0154e10 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.835+0000 7f52b6ffd640 1 -- 172.21.15.59:0/3814146208 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52b0039070 con 0x7f52c0154e10 2024-06-29T15:59:42.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.835+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c00cd1c0 con 0x7f52c0154e10 2024-06-29T15:59:42.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.835+0000 7f52b6ffd640 1 -- 172.21.15.59:0/3814146208 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0003d90 con 0x7f52c0154e10 2024-06-29T15:59:42.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.835+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c00cd780 con 0x7f52c0154e10 2024-06-29T15:59:42.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.836+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c00ccf30 con 0x7f52c0154e10 2024-06-29T15:59:42.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.837+0000 7f52b6ffd640 1 -- 172.21.15.59:0/3814146208 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f52b003b070 con 0x7f52c0154e10 2024-06-29T15:59:42.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.837+0000 7f52b6ffd640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f52a40763f0 0x7f52a4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.837+0000 7f52b6ffd640 1 -- 172.21.15.59:0/3814146208 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5305+0+0 (secure 0 0 0) 0x7f52b00b6cf0 con 0x7f52c0154e10 2024-06-29T15:59:42.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.837+0000 7f52c6387640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f52a40763f0 0x7f52a4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.838+0000 7f52c6387640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f52a40763f0 0x7f52a4078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f52c00d0500 tx=0x7f52bc009500 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:42.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:42.841+0000 7f52b6ffd640 1 -- 172.21.15.59:0/3814146208 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f52b00bb070 con 0x7f52c0154e10 2024-06-29T15:59:43.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.036+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52c00d2100 con 0x7f52c0154e10 2024-06-29T15:59:43.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.037+0000 7f52b6ffd640 1 -- 172.21.15.59:0/3814146208 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) ==== 74+0+13276 (secure 0 0 0) 0x7f52b00800b0 con 0x7f52c0154e10 2024-06-29T15:59:43.037 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:43.037 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":271,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:42.293208+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f52a40763f0 msgr2=0x7f52a4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f52a40763f0 0x7f52a4078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f52c00d0500 tx=0x7f52bc009500 comp rx=0 tx=0).stop 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c0154e10 msgr2=0x7f52c0156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c0154e10 0x7f52c0156440 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f52b00017b0 tx=0x7f52b00063a0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 shutdown_connections 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f52a40763f0 0x7f52a4078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c0155220 0x7f52c0156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c0154e10 0x7f52c0156440 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 --2- 172.21.15.59:0/3814146208 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f52ac004800 0x7f52c0158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 >> 172.21.15.59:0/3814146208 conn(0x7f52c00b5610 msgr2=0x7f52c015c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 shutdown_connections 2024-06-29T15:59:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.039+0000 7f52c85eb640 1 -- 172.21.15.59:0/3814146208 wait complete. 2024-06-29T15:59:43.051 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:43.052 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:43.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.174+0000 7f9cefd24640 1 -- 172.21.15.59:0/4202434114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd00040b0 msgr2=0x7f9cd0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.174+0000 7f9cefd24640 1 --2- 172.21.15.59:0/4202434114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cd00040b0 0x7f9cd0004800 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80736a0 tx=0x7f9cd0005830 comp rx=0 tx=0).stop 2024-06-29T15:59:43.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.174+0000 7f9cee2c1640 1 -- 172.21.15.59:0/4202434114 reap_dead start 2024-06-29T15:59:43.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.174+0000 7f9cefd24640 1 -- 172.21.15.59:0/4202434114 shutdown_connections 2024-06-29T15:59:43.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.174+0000 7f9cefd24640 1 -- 172.21.15.59:0/4202434114 >> 172.21.15.59:0/4202434114 conn(0x7f9ce80b24c0 msgr2=0x7f9ce80cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.174+0000 7f9cefd24640 1 -- 172.21.15.59:0/4202434114 shutdown_connections 2024-06-29T15:59:43.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.174+0000 7f9cefd24640 1 -- 172.21.15.59:0/4202434114 wait complete. 2024-06-29T15:59:43.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.176+0000 7f9cefd24640 1 Processor -- start 2024-06-29T15:59:43.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.176+0000 7f9cefd24640 1 -- start start 2024-06-29T15:59:43.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cefd24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd00040b0 0x7f9ce8158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cefd24640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce8153400 0x7f9ce8153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cefd24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce8153d50 0x7f9ce81f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cefd24640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce80cd460 con 0x7f9ce8153d50 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cefd24640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce80cd2e0 con 0x7f9cd00040b0 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cefd24640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9ce80ce6e0 con 0x7f9ce8153400 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cee2c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce8153d50 0x7f9ce81f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cee2c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce8153d50 0x7f9ce81f18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60988/0 (socket says 172.21.15.59:60988) 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cee2c1640 1 -- 172.21.15.59:0/3056152147 learned_addr learned my addr 172.21.15.59:0/3056152147 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9ced2bf640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce8153400 0x7f9ce8153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cedac0640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd00040b0 0x7f9ce8158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cee2c1640 1 -- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce8153400 msgr2=0x7f9ce8153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.177+0000 7f9cee2c1640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce8153400 0x7f9ce8153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9cee2c1640 1 -- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd00040b0 msgr2=0x7f9ce8158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9cee2c1640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd00040b0 0x7f9ce8158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9cee2c1640 1 -- 172.21.15.59:0/3056152147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce813f260 con 0x7f9ce8153d50 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9cedac0640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd00040b0 0x7f9ce8158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:43.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9ced2bf640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce8153400 0x7f9ce8153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:43.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9cee2c1640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce8153d50 0x7f9ce81f18c0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80b51d0 tx=0x7f9cd800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9ce67fc640 1 -- 172.21.15.59:0/3056152147 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd800eea0 con 0x7f9ce8153d50 2024-06-29T15:59:43.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9ce67fc640 1 -- 172.21.15.59:0/3056152147 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cd8018070 con 0x7f9ce8153d50 2024-06-29T15:59:43.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce8141e50 con 0x7f9ce8153d50 2024-06-29T15:59:43.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9ce67fc640 1 -- 172.21.15.59:0/3056152147 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8011460 con 0x7f9ce8153d50 2024-06-29T15:59:43.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.178+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce813ef80 con 0x7f9ce8153d50 2024-06-29T15:59:43.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.179+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce813ecf0 con 0x7f9ce8153d50 2024-06-29T15:59:43.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.180+0000 7f9ce67fc640 1 -- 172.21.15.59:0/3056152147 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f9cd8011600 con 0x7f9ce8153d50 2024-06-29T15:59:43.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.181+0000 7f9ce67fc640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9cd40763f0 0x7f9cd4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.181+0000 7f9ce67fc640 1 -- 172.21.15.59:0/3056152147 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5305+0+0 (secure 0 0 0) 0x7f9cd8097d30 con 0x7f9ce8153d50 2024-06-29T15:59:43.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.181+0000 7f9cedac0640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9cd40763f0 0x7f9cd4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.181+0000 7f9cedac0640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9cd40763f0 0x7f9cd4078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009cd0 tx=0x7f9ce0005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.185+0000 7f9ce67fc640 1 -- 172.21.15.59:0/3056152147 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9ce813f260 con 0x7f9ce8153d50 2024-06-29T15:59:43.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.381+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9ce8142be0 con 0x7f9ce8153d50 2024-06-29T15:59:43.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.381+0000 7f9ce67fc640 1 -- 172.21.15.59:0/3056152147 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v271) ==== 93+0+10 (secure 0 0 0) 0x7f9ce80cd610 con 0x7f9ce8153d50 2024-06-29T15:59:43.381 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9cd40763f0 msgr2=0x7f9cd4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9cd40763f0 0x7f9cd4078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009cd0 tx=0x7f9ce0005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce8153d50 msgr2=0x7f9ce81f18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce8153d50 0x7f9ce81f18c0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80b51d0 tx=0x7f9cd800ec50 comp rx=0 tx=0).stop 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 shutdown_connections 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f9cd40763f0 0x7f9cd4078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ce8153d50 0x7f9ce81f18c0 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9ce8153400 0x7f9ce8153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 --2- 172.21.15.59:0/3056152147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cd00040b0 0x7f9ce8158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.383+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 >> 172.21.15.59:0/3056152147 conn(0x7f9ce80b24c0 msgr2=0x7f9ce815edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.384+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 shutdown_connections 2024-06-29T15:59:43.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.384+0000 7f9cefd24640 1 -- 172.21.15.59:0/3056152147 wait complete. 2024-06-29T15:59:43.395 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:59:43.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.513+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1943533108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc0cd9b0 msgr2=0x7ff9cc15f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.513+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1943533108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc0cd9b0 0x7ff9cc15f240 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc005fd0 tx=0x7ff9bc02aa90 comp rx=0 tx=0).stop 2024-06-29T15:59:43.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.513+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1943533108 shutdown_connections 2024-06-29T15:59:43.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.513+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1943533108 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc06da00 0x7ff9cc073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.513+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1943533108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc0cd9b0 0x7ff9cc15f240 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.513+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1943533108 >> 172.21.15.59:0/1943533108 conn(0x7ff9cc0b24c0 msgr2=0x7ff9cc0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.513+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1943533108 shutdown_connections 2024-06-29T15:59:43.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.513+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1943533108 wait complete. 2024-06-29T15:59:43.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.515+0000 7ff9d18a5640 1 Processor -- start 2024-06-29T15:59:43.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.515+0000 7ff9d18a5640 1 -- start start 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9d18a5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc0cd9b0 0x7ff9cc1e6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9d18a5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc1444c0 0x7ff9cc1e56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9d18a5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc13aac0 0x7ff9cc1e5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9d18a5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc073b40 con 0x7ff9cc13aac0 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9d18a5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc0739c0 con 0x7ff9cc1444c0 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9d18a5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff9cc073cc0 con 0x7ff9cc0cd9b0 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc13aac0 0x7ff9cc1e5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9cb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc13aac0 0x7ff9cc1e5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32770/0 (socket says 172.21.15.59:32770) 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.516+0000 7ff9cb7fe640 1 -- 172.21.15.59:0/1526863221 learned_addr learned my addr 172.21.15.59:0/1526863221 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9caffd640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc0cd9b0 0x7ff9cc1e6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9ca7fc640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc1444c0 0x7ff9cc1e56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9cb7fe640 1 -- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc0cd9b0 msgr2=0x7ff9cc1e6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9cb7fe640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc0cd9b0 0x7ff9cc1e6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9cb7fe640 1 -- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc1444c0 msgr2=0x7ff9cc1e56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9cb7fe640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc1444c0 0x7ff9cc1e56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9cb7fe640 1 -- 172.21.15.59:0/1526863221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc138190 con 0x7ff9cc13aac0 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9caffd640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc0cd9b0 0x7ff9cc1e6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9cb7fe640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc13aac0 0x7ff9cc1e5be0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400d620 tx=0x7ff9c400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9c3fff640 1 -- 172.21.15.59:0/1526863221 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4017070 con 0x7ff9cc13aac0 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9c3fff640 1 -- 172.21.15.59:0/1526863221 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9c4005310 con 0x7ff9cc13aac0 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9cc137bd0 con 0x7ff9cc13aac0 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9c3fff640 1 -- 172.21.15.59:0/1526863221 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40047a0 con 0x7ff9cc13aac0 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.517+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9cc13a2c0 con 0x7ff9cc13aac0 2024-06-29T15:59:43.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.518+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9cc0b5060 con 0x7ff9cc13aac0 2024-06-29T15:59:43.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.519+0000 7ff9c3fff640 1 -- 172.21.15.59:0/1526863221 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7ff9c4004940 con 0x7ff9cc13aac0 2024-06-29T15:59:43.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.519+0000 7ff9c3fff640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7ff9ac076420 0x7ff9ac078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.520+0000 7ff9c3fff640 1 -- 172.21.15.59:0/1526863221 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5305+0+0 (secure 0 0 0) 0x7ff9c40977e0 con 0x7ff9cc13aac0 2024-06-29T15:59:43.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.520+0000 7ff9caffd640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7ff9ac076420 0x7ff9ac078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.520+0000 7ff9caffd640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7ff9ac076420 0x7ff9ac078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc035000 tx=0x7ff9bc038040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.524+0000 7ff9c3fff640 1 -- 172.21.15.59:0/1526863221 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff9cc0b5060 con 0x7ff9cc13aac0 2024-06-29T15:59:43.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:59:43.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:59:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.718+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff9cc074170 con 0x7ff9cc13aac0 2024-06-29T15:59:43.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.718+0000 7ff9c3fff640 1 -- 172.21.15.59:0/1526863221 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v271) ==== 84+0+65 (secure 0 0 0) 0x7ff9c4061010 con 0x7ff9cc13aac0 2024-06-29T15:59:43.718 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:43.718 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:59:43.718 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:43.718 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-06-29T15:59:43.718 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-06-29T15:59:43.718 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:43.719 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:59:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7ff9ac076420 msgr2=0x7ff9ac078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7ff9ac076420 0x7ff9ac078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc035000 tx=0x7ff9bc038040 comp rx=0 tx=0).stop 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc13aac0 msgr2=0x7ff9cc1e5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc13aac0 0x7ff9cc1e5be0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400d620 tx=0x7ff9c400dae0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 shutdown_connections 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7ff9ac076420 0x7ff9ac078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9cc13aac0 0x7ff9cc1e5be0 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9cc1444c0 0x7ff9cc1e56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 --2- 172.21.15.59:0/1526863221 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9cc0cd9b0 0x7ff9cc1e6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 >> 172.21.15.59:0/1526863221 conn(0x7ff9cc0b24c0 msgr2=0x7ff9cc137190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 shutdown_connections 2024-06-29T15:59:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.720+0000 7ff9d18a5640 1 -- 172.21.15.59:0/1526863221 wait complete. 2024-06-29T15:59:43.732 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:59:43.732 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:59:43.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.893+0000 7f4500f11640 1 -- 172.21.15.59:0/1170965573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 msgr2=0x7f44fc1621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.893+0000 7f4500f11640 1 --2- 172.21.15.59:0/1170965573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 0x7f44fc1621f0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0c7270 tx=0x7f44e402a530 comp rx=0 tx=0).stop 2024-06-29T15:59:43.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.893+0000 7f4500f11640 1 -- 172.21.15.59:0/1170965573 shutdown_connections 2024-06-29T15:59:43.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.893+0000 7f4500f11640 1 --2- 172.21.15.59:0/1170965573 >> v2:172.21.15.153:3300/0 conn(0x7f44fc0b3ed0 0x7f44fc0b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.893+0000 7f4500f11640 1 --2- 172.21.15.59:0/1170965573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 0x7f44fc1621f0 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.893+0000 7f4500f11640 1 -- 172.21.15.59:0/1170965573 >> 172.21.15.59:0/1170965573 conn(0x7f44fc0d3e60 msgr2=0x7f44fc0d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.893+0000 7f4500f11640 1 -- 172.21.15.59:0/1170965573 shutdown_connections 2024-06-29T15:59:43.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.893+0000 7f4500f11640 1 -- 172.21.15.59:0/1170965573 wait complete. 2024-06-29T15:59:43.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.895+0000 7f4500f11640 1 Processor -- start 2024-06-29T15:59:43.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.895+0000 7f4500f11640 1 -- start start 2024-06-29T15:59:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f4500f11640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44fc0b3ed0 0x7f44fc1596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f4500f11640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 0x7f44fc1f8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f4500f11640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44fc15dff0 0x7f44fc1f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f4500f11640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc0b4820 con 0x7f44fc0d2780 2024-06-29T15:59:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f4500f11640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc162940 con 0x7f44fc15dff0 2024-06-29T15:59:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f4500f11640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f44fc0b49a0 con 0x7f44fc0b3ed0 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 0x7f44fc1f8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 0x7f44fc1f8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32786/0 (socket says 172.21.15.59:32786) 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 -- 172.21.15.59:0/522026467 learned_addr learned my addr 172.21.15.59:0/522026467 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44fad9d640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44fc15dff0 0x7f44fc1f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44fa59c640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44fc0b3ed0 0x7f44fc1596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 -- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44fc0b3ed0 msgr2=0x7f44fc1596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44fc0b3ed0 0x7f44fc1596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 -- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44fc15dff0 msgr2=0x7f44fc1f8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44fc15dff0 0x7f44fc1f8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 -- 172.21.15.59:0/522026467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc0aa5b0 con 0x7f44fc0d2780 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44fa59c640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44fc0b3ed0 0x7f44fc1596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f9d9b640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 0x7f44fc1f8530 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f44e40098f0 tx=0x7f44e4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.897+0000 7f44f2ffd640 1 -- 172.21.15.59:0/522026467 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4005c90 con 0x7f44fc0d2780 2024-06-29T15:59:43.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.898+0000 7f44f2ffd640 1 -- 172.21.15.59:0/522026467 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44e4031e50 con 0x7f44fc0d2780 2024-06-29T15:59:43.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.898+0000 7f44f2ffd640 1 -- 172.21.15.59:0/522026467 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4005260 con 0x7f44fc0d2780 2024-06-29T15:59:43.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.898+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44fc0cd6d0 con 0x7f44fc0d2780 2024-06-29T15:59:43.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.898+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44fc0aa870 con 0x7f44fc0d2780 2024-06-29T15:59:43.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.899+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44fc0b4e90 con 0x7f44fc0d2780 2024-06-29T15:59:43.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.899+0000 7f44f2ffd640 1 -- 172.21.15.59:0/522026467 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f44e4005400 con 0x7f44fc0d2780 2024-06-29T15:59:43.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.900+0000 7f44f2ffd640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f44dc076420 0x7f44dc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.900+0000 7f44f2ffd640 1 -- 172.21.15.59:0/522026467 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5305+0+0 (secure 0 0 0) 0x7f44e40b8830 con 0x7f44fc0d2780 2024-06-29T15:59:43.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.900+0000 7f44fa59c640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f44dc076420 0x7f44dc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.901+0000 7f44fa59c640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f44dc076420 0x7f44dc078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f44ec001770 tx=0x7f44ec009040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:43.905+0000 7f44f2ffd640 1 -- 172.21.15.59:0/522026467 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f44e403f030 con 0x7f44fc0d2780 2024-06-29T15:59:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.096+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f44fc0b2c20 con 0x7f44fc0d2780 2024-06-29T15:59:44.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.286+0000 7f44f2ffd640 1 -- 172.21.15.59:0/522026467 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v272) ==== 138+0+0 (secure 0 0 0) 0x7f44e4081a10 con 0x7f44fc0d2780 2024-06-29T15:59:44.286 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-06-29T15:59:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f44dc076420 msgr2=0x7f44dc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f44dc076420 0x7f44dc078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f44ec001770 tx=0x7f44ec009040 comp rx=0 tx=0).stop 2024-06-29T15:59:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 msgr2=0x7f44fc1f8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 0x7f44fc1f8530 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f44e40098f0 tx=0x7f44e4033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 shutdown_connections 2024-06-29T15:59:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f44dc076420 0x7f44dc078930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44fc15dff0 0x7f44fc1f8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44fc0d2780 0x7f44fc1f8530 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 --2- 172.21.15.59:0/522026467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f44fc0b3ed0 0x7f44fc1596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 >> 172.21.15.59:0/522026467 conn(0x7f44fc0d3e60 msgr2=0x7f44fc0b81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:44.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 shutdown_connections 2024-06-29T15:59:44.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.289+0000 7f4500f11640 1 -- 172.21.15.59:0/522026467 wait complete. 2024-06-29T15:59:44.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:59:44.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.429+0000 7f309d89c640 1 -- 172.21.15.59:0/1973566486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 msgr2=0x7f309815dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.429+0000 7f309d89c640 1 --2- 172.21.15.59:0/1973566486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 0x7f309815dc10 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f3088005fd0 tx=0x7f308802aa90 comp rx=0 tx=0).stop 2024-06-29T15:59:44.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.429+0000 7f309d89c640 1 -- 172.21.15.59:0/1973566486 shutdown_connections 2024-06-29T15:59:44.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.429+0000 7f309d89c640 1 --2- 172.21.15.59:0/1973566486 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30980cebf0 0x7f30980cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.429+0000 7f309d89c640 1 --2- 172.21.15.59:0/1973566486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 0x7f309815dc10 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.429+0000 7f309d89c640 1 -- 172.21.15.59:0/1973566486 >> 172.21.15.59:0/1973566486 conn(0x7f30980d3e40 msgr2=0x7f30980d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:44.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.429+0000 7f309d89c640 1 -- 172.21.15.59:0/1973566486 shutdown_connections 2024-06-29T15:59:44.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.429+0000 7f309d89c640 1 -- 172.21.15.59:0/1973566486 wait complete. 2024-06-29T15:59:44.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.431+0000 7f309d89c640 1 Processor -- start 2024-06-29T15:59:44.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.431+0000 7f309d89c640 1 -- start start 2024-06-29T15:59:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f309d89c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 0x7f30980c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f309d89c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30980cebf0 0x7f309814ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f309d89c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3098161980 0x7f30981595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f309d89c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f30980b2b90 con 0x7f30980ce510 2024-06-29T15:59:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f309d89c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30980b2a10 con 0x7f30980cebf0 2024-06-29T15:59:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f309d89c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f30980b4bc0 con 0x7f3098161980 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f3096ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 0x7f30980c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f3096ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 0x7f30980c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32800/0 (socket says 172.21.15.59:32800) 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f3096ffd640 1 -- 172.21.15.59:0/2441090239 learned_addr learned my addr 172.21.15.59:0/2441090239 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f30967fc640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30980cebf0 0x7f309814ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.432+0000 7f30977fe640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3098161980 0x7f30981595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f3096ffd640 1 -- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3098161980 msgr2=0x7f30981595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f3096ffd640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3098161980 0x7f30981595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f3096ffd640 1 -- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30980cebf0 msgr2=0x7f309814ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f3096ffd640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30980cebf0 0x7f309814ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f3096ffd640 1 -- 172.21.15.59:0/2441090239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309815a090 con 0x7f30980ce510 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f3096ffd640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 0x7f30980c8190 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f3088000f80 tx=0x7f3088003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f307ffff640 1 -- 172.21.15.59:0/2441090239 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088044070 con 0x7f30980ce510 2024-06-29T15:59:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30980c76b0 con 0x7f30980ce510 2024-06-29T15:59:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f307ffff640 1 -- 172.21.15.59:0/2441090239 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3088003e00 con 0x7f30980ce510 2024-06-29T15:59:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f307ffff640 1 -- 172.21.15.59:0/2441090239 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088004490 con 0x7f30980ce510 2024-06-29T15:59:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.433+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30981571b0 con 0x7f30980ce510 2024-06-29T15:59:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.434+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f309815dc10 con 0x7f30980ce510 2024-06-29T15:59:44.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.435+0000 7f307ffff640 1 -- 172.21.15.59:0/2441090239 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f3088047070 con 0x7f30980ce510 2024-06-29T15:59:44.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.435+0000 7f307ffff640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3084076420 0x7f3084078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:44.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.436+0000 7f307ffff640 1 -- 172.21.15.59:0/2441090239 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5675+0+0 (secure 0 0 0) 0x7f30880b88c0 con 0x7f30980ce510 2024-06-29T15:59:44.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.436+0000 7f30967fc640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3084076420 0x7f3084078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:44.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.436+0000 7f30967fc640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3084076420 0x7f3084078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3098156ed0 tx=0x7f308c005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:44.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.440+0000 7f307ffff640 1 -- 172.21.15.59:0/2441090239 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3088032030 con 0x7f30980ce510 2024-06-29T15:59:44.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:44.635+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f30980b76d0 con 0x7f30980ce510 2024-06-29T15:59:45.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.293+0000 7f307ffff640 1 -- 172.21.15.59:0/2441090239 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v273) ==== 170+0+0 (secure 0 0 0) 0x7f3088081fa0 con 0x7f30980ce510 2024-06-29T15:59:45.293 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-06-29T15:59:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.296+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3084076420 msgr2=0x7f3084078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.296+0000 7f309d89c640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3084076420 0x7f3084078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3098156ed0 tx=0x7f308c005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.296+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 msgr2=0x7f30980c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.296+0000 7f309d89c640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 0x7f30980c8190 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f3088000f80 tx=0x7f3088003c10 comp rx=0 tx=0).stop 2024-06-29T15:59:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.297+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 shutdown_connections 2024-06-29T15:59:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.297+0000 7f309d89c640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3084076420 0x7f3084078930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.297+0000 7f309d89c640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3098161980 0x7f30981595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.297+0000 7f309d89c640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30980cebf0 0x7f309814ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.297+0000 7f309d89c640 1 --2- 172.21.15.59:0/2441090239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30980ce510 0x7f30980c8190 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.297+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 >> 172.21.15.59:0/2441090239 conn(0x7f30980d3e40 msgr2=0x7f30980b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.297+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 shutdown_connections 2024-06-29T15:59:45.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.297+0000 7f309d89c640 1 -- 172.21.15.59:0/2441090239 wait complete. 2024-06-29T15:59:45.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:59:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.455+0000 7f38bd97f640 1 -- 172.21.15.59:0/3203981214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80d0130 msgr2=0x7f38b8143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.455+0000 7f38bd97f640 1 --2- 172.21.15.59:0/3203981214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80d0130 0x7f38b8143840 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f38b80753d0 tx=0x7f38a802a530 comp rx=0 tx=0).stop 2024-06-29T15:59:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.455+0000 7f38bd97f640 1 -- 172.21.15.59:0/3203981214 shutdown_connections 2024-06-29T15:59:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.456+0000 7f38bd97f640 1 --2- 172.21.15.59:0/3203981214 >> v2:172.21.15.153:3300/0 conn(0x7f38b806ebe0 0x7f38b80cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.456+0000 7f38bd97f640 1 --2- 172.21.15.59:0/3203981214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80d0130 0x7f38b8143840 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.456+0000 7f38bd97f640 1 -- 172.21.15.59:0/3203981214 >> 172.21.15.59:0/3203981214 conn(0x7f38b80b2ab0 msgr2=0x7f38b80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.456+0000 7f38bd97f640 1 -- 172.21.15.59:0/3203981214 shutdown_connections 2024-06-29T15:59:45.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.456+0000 7f38bd97f640 1 -- 172.21.15.59:0/3203981214 wait complete. 2024-06-29T15:59:45.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.457+0000 7f38bd97f640 1 Processor -- start 2024-06-29T15:59:45.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.457+0000 7f38bd97f640 1 -- start start 2024-06-29T15:59:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38bd97f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b806ebe0 0x7f38b815a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38bd97f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80b4ee0 0x7f38b80aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38bd97f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38b80d0130 0x7f38b80af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38bd97f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f38b80cf860 con 0x7f38b80b4ee0 2024-06-29T15:59:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38bd97f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f38b80cf6e0 con 0x7f38b806ebe0 2024-06-29T15:59:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38bd97f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f38b80cf9e0 con 0x7f38b80d0130 2024-06-29T15:59:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80b4ee0 0x7f38b80aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80b4ee0 0x7f38b80aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32820/0 (socket says 172.21.15.59:32820) 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 -- 172.21.15.59:0/2528069022 learned_addr learned my addr 172.21.15.59:0/2528069022 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38b6ffd640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b806ebe0 0x7f38b815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38b77fe640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38b80d0130 0x7f38b80af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 -- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38b80d0130 msgr2=0x7f38b80af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38b80d0130 0x7f38b80af2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 -- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b806ebe0 msgr2=0x7f38b815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b806ebe0 0x7f38b815a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 -- 172.21.15.59:0/2528069022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b80ab650 con 0x7f38b80b4ee0 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38affff640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80b4ee0 0x7f38b80aed70 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f38a000b3f0 tx=0x7f38a000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.459+0000 7f38b4ff9640 1 -- 172.21.15.59:0/2528069022 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0015020 con 0x7f38b80b4ee0 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.460+0000 7f38b4ff9640 1 -- 172.21.15.59:0/2528069022 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38a0011920 con 0x7f38b80b4ee0 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.460+0000 7f38b4ff9640 1 -- 172.21.15.59:0/2528069022 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0011d70 con 0x7f38b80b4ee0 2024-06-29T15:59:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.460+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b80ab930 con 0x7f38b80b4ee0 2024-06-29T15:59:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.460+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b80abef0 con 0x7f38b80b4ee0 2024-06-29T15:59:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.461+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b80cef90 con 0x7f38b80b4ee0 2024-06-29T15:59:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.462+0000 7f38b4ff9640 1 -- 172.21.15.59:0/2528069022 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f38a0018020 con 0x7f38b80b4ee0 2024-06-29T15:59:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.462+0000 7f38b4ff9640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38980763f0 0x7f3898078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.462+0000 7f38b4ff9640 1 -- 172.21.15.59:0/2528069022 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6057+0+0 (secure 0 0 0) 0x7f38a00977e0 con 0x7f38b80b4ee0 2024-06-29T15:59:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.463+0000 7f38b6ffd640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38980763f0 0x7f3898078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.463+0000 7f38b6ffd640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38980763f0 0x7f3898078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38a8033000 tx=0x7f38a8004730 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.467+0000 7f38b4ff9640 1 -- 172.21.15.59:0/2528069022 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f38a001f030 con 0x7f38b80b4ee0 2024-06-29T15:59:45.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:45.698+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f38b80b1f90 con 0x7f38b80b4ee0 2024-06-29T15:59:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.307+0000 7f38b4ff9640 1 -- 172.21.15.59:0/2528069022 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 37 and data pool 38 v153) ==== 174+0+0 (secure 0 0 0) 0x7f38a0060e40 con 0x7f38b80b4ee0 2024-06-29T15:59:46.307 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 37 and data pool 38 2024-06-29T15:59:46.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.312+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38980763f0 msgr2=0x7f3898078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.312+0000 7f38bd97f640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38980763f0 0x7f3898078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38a8033000 tx=0x7f38a8004730 comp rx=0 tx=0).stop 2024-06-29T15:59:46.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.312+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80b4ee0 msgr2=0x7f38b80aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.312+0000 7f38bd97f640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80b4ee0 0x7f38b80aed70 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f38a000b3f0 tx=0x7f38a000b8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.318+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 shutdown_connections 2024-06-29T15:59:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.318+0000 7f38bd97f640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38980763f0 0x7f3898078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.318+0000 7f38bd97f640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f38b80d0130 0x7f38b80af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.318+0000 7f38bd97f640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38b80b4ee0 0x7f38b80aed70 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.318+0000 7f38bd97f640 1 --2- 172.21.15.59:0/2528069022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38b806ebe0 0x7f38b815a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.318+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 >> 172.21.15.59:0/2528069022 conn(0x7f38b80b2ab0 msgr2=0x7f38b806ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.321+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 shutdown_connections 2024-06-29T15:59:46.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.323+0000 7f38bd97f640 1 -- 172.21.15.59:0/2528069022 wait complete. 2024-06-29T15:59:46.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:46.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.461+0000 7f07e17bd640 1 -- 172.21.15.59:0/3934923276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc0b3740 msgr2=0x7f07dc0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.461+0000 7f07e17bd640 1 --2- 172.21.15.59:0/3934923276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc0b3740 0x7f07dc0b3b50 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009830 tx=0x7f07c402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.461+0000 7f07e17bd640 1 -- 172.21.15.59:0/3934923276 shutdown_connections 2024-06-29T15:59:46.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.461+0000 7f07e17bd640 1 --2- 172.21.15.59:0/3934923276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc0b3740 0x7f07dc0b3b50 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.461+0000 7f07e17bd640 1 -- 172.21.15.59:0/3934923276 >> 172.21.15.59:0/3934923276 conn(0x7f07dc06c950 msgr2=0x7f07dc073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.461+0000 7f07e17bd640 1 -- 172.21.15.59:0/3934923276 shutdown_connections 2024-06-29T15:59:46.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.461+0000 7f07e17bd640 1 -- 172.21.15.59:0/3934923276 wait complete. 2024-06-29T15:59:46.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.463+0000 7f07e17bd640 1 Processor -- start 2024-06-29T15:59:46.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.463+0000 7f07e17bd640 1 -- start start 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07e17bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07dc0b3740 0x7f07dc0ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07e17bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07dc15c2f0 0x7f07dc1f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07e17bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc141df0 0x7f07dc142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07e17bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc0b6320 con 0x7f07dc141df0 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07e17bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc0b61a0 con 0x7f07dc0b3740 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07e17bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc0b64a0 con 0x7f07dc15c2f0 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc141df0 0x7f07dc142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc141df0 0x7f07dc142200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32846/0 (socket says 172.21.15.59:32846) 2024-06-29T15:59:46.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 -- 172.21.15.59:0/421424655 learned_addr learned my addr 172.21.15.59:0/421424655 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07d27fc640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07dc15c2f0 0x7f07dc1f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07daffd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07dc0b3740 0x7f07dc0ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 -- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07dc15c2f0 msgr2=0x7f07dc1f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07dc15c2f0 0x7f07dc1f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 -- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07dc0b3740 msgr2=0x7f07dc0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07dc0b3740 0x7f07dc0ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 -- 172.21.15.59:0/421424655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc142740 con 0x7f07dc141df0 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.465+0000 7f07db7fe640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc141df0 0x7f07dc142200 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f07d400cb40 tx=0x7f07d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.466+0000 7f07d3fff640 1 -- 172.21.15.59:0/421424655 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4016020 con 0x7f07dc141df0 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.466+0000 7f07d3fff640 1 -- 172.21.15.59:0/421424655 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07d400ad30 con 0x7f07dc141df0 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.466+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07dc0aeea0 con 0x7f07dc141df0 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.466+0000 7f07d3fff640 1 -- 172.21.15.59:0/421424655 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4012680 con 0x7f07dc141df0 2024-06-29T15:59:46.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.466+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07dc0aa7d0 con 0x7f07dc141df0 2024-06-29T15:59:46.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.467+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07dc1458a0 con 0x7f07dc141df0 2024-06-29T15:59:46.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.467+0000 7f07d3fff640 1 -- 172.21.15.59:0/421424655 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f07d4019020 con 0x7f07dc141df0 2024-06-29T15:59:46.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.468+0000 7f07d3fff640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f07bc0763f0 0x7f07bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.468+0000 7f07d3fff640 1 -- 172.21.15.59:0/421424655 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6157+0+0 (secure 0 0 0) 0x7f07d40980e0 con 0x7f07dc141df0 2024-06-29T15:59:46.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.469+0000 7f07daffd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f07bc0763f0 0x7f07bc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.469+0000 7f07daffd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f07bc0763f0 0x7f07bc078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f07cc001720 tx=0x7f07cc009040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.473+0000 7f07d3fff640 1 -- 172.21.15.59:0/421424655 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f07d4020020 con 0x7f07dc141df0 2024-06-29T15:59:46.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.663+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07dc0af140 con 0x7f07dc141df0 2024-06-29T15:59:46.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.664+0000 7f07d3fff640 1 -- 172.21.15.59:0/421424655 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) ==== 74+0+17011 (secure 0 0 0) 0x7f07dc142740 con 0x7f07dc141df0 2024-06-29T15:59:46.664 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:46.665 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":274,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:45.699426+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:44.104733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:44.643605+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f07bc0763f0 msgr2=0x7f07bc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f07bc0763f0 0x7f07bc078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f07cc001720 tx=0x7f07cc009040 comp rx=0 tx=0).stop 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc141df0 msgr2=0x7f07dc142200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc141df0 0x7f07dc142200 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f07d400cb40 tx=0x7f07d400a060 comp rx=0 tx=0).stop 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 shutdown_connections 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f07bc0763f0 0x7f07bc078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07dc141df0 0x7f07dc142200 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07dc15c2f0 0x7f07dc1f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 --2- 172.21.15.59:0/421424655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07dc0b3740 0x7f07dc0ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 >> 172.21.15.59:0/421424655 conn(0x7f07dc06c950 msgr2=0x7f07dc06fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 shutdown_connections 2024-06-29T15:59:46.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.666+0000 7f07e17bd640 1 -- 172.21.15.59:0/421424655 wait complete. 2024-06-29T15:59:46.679 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.796+0000 7f452ea3f640 1 -- 172.21.15.59:0/2756025437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45281425e0 msgr2=0x7f4528142be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.796+0000 7f452ea3f640 1 --2- 172.21.15.59:0/2756025437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45281425e0 0x7f4528142be0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f45280cd850 tx=0x7f451802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.796+0000 7f452ea3f640 1 -- 172.21.15.59:0/2756025437 shutdown_connections 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.796+0000 7f452ea3f640 1 --2- 172.21.15.59:0/2756025437 >> v2:172.21.15.153:3300/0 conn(0x7f452806c270 0x7f452806c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.796+0000 7f452ea3f640 1 --2- 172.21.15.59:0/2756025437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45281425e0 0x7f4528142be0 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.796+0000 7f452ea3f640 1 -- 172.21.15.59:0/2756025437 >> 172.21.15.59:0/2756025437 conn(0x7f45280b24c0 msgr2=0x7f45280cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.796+0000 7f452ea3f640 1 -- 172.21.15.59:0/2756025437 shutdown_connections 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.796+0000 7f452ea3f640 1 -- 172.21.15.59:0/2756025437 wait complete. 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.798+0000 7f452ea3f640 1 Processor -- start 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.798+0000 7f452ea3f640 1 -- start start 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f452ea3f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f452806c270 0x7f452815b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f452ea3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45280cccc0 0x7f452813f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f452ea3f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45281425e0 0x7f452813c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f452ea3f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f45280cea70 con 0x7f45280cccc0 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f452ea3f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45280ce8f0 con 0x7f45281425e0 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f452ea3f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45280cebf0 con 0x7f452806c270 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45280cccc0 0x7f452813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45280cccc0 0x7f452813f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32866/0 (socket says 172.21.15.59:32866) 2024-06-29T15:59:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 -- 172.21.15.59:0/1602106624 learned_addr learned my addr 172.21.15.59:0/1602106624 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f4527fff640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f452806c270 0x7f452815b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f452cfdc640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45281425e0 0x7f452813c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 -- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f452806c270 msgr2=0x7f452815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f452806c270 0x7f452815b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 -- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45281425e0 msgr2=0x7f452813c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45281425e0 0x7f452813c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 -- 172.21.15.59:0/1602106624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45280aa0c0 con 0x7f45280cccc0 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f45277fe640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45280cccc0 0x7f452813f520 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f451c00c3d0 tx=0x7f451c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f4524ff9640 1 -- 172.21.15.59:0/1602106624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c017070 con 0x7f45280cccc0 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f4524ff9640 1 -- 172.21.15.59:0/1602106624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f451c0053c0 con 0x7f45280cccc0 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.800+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4528140310 con 0x7f45280cccc0 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.801+0000 7f4524ff9640 1 -- 172.21.15.59:0/1602106624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c003850 con 0x7f45280cccc0 2024-06-29T15:59:46.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.801+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45281408d0 con 0x7f45280cccc0 2024-06-29T15:59:46.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.802+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45280cf110 con 0x7f45280cccc0 2024-06-29T15:59:46.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.803+0000 7f4524ff9640 1 -- 172.21.15.59:0/1602106624 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f451c01a070 con 0x7f45280cccc0 2024-06-29T15:59:46.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.803+0000 7f4524ff9640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4514076420 0x7f4514078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.803+0000 7f4524ff9640 1 -- 172.21.15.59:0/1602106624 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6157+0+0 (secure 0 0 0) 0x7f451c0971b0 con 0x7f45280cccc0 2024-06-29T15:59:46.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.803+0000 7f4527fff640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4514076420 0x7f4514078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.804+0000 7f4527fff640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4514076420 0x7f4514078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4518033000 tx=0x7f4518004730 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.807+0000 7f4524ff9640 1 -- 172.21.15.59:0/1602106624 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f451c021020 con 0x7f45280cccc0 2024-06-29T15:59:46.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:46.998+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f452815f4c0 con 0x7f45280cccc0 2024-06-29T15:59:46.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.000+0000 7f4524ff9640 1 -- 172.21.15.59:0/1602106624 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) ==== 74+0+17011 (secure 0 0 0) 0x7f45280aa0c0 con 0x7f45280cccc0 2024-06-29T15:59:47.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:47.000 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":274,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:45.699426+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:44.104733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:44.643605+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.001+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4514076420 msgr2=0x7f4514078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4514076420 0x7f4514078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4518033000 tx=0x7f4518004730 comp rx=0 tx=0).stop 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45280cccc0 msgr2=0x7f452813f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45280cccc0 0x7f452813f520 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f451c00c3d0 tx=0x7f451c00c890 comp rx=0 tx=0).stop 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 shutdown_connections 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f4514076420 0x7f4514078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45281425e0 0x7f452813c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45280cccc0 0x7f452813f520 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 --2- 172.21.15.59:0/1602106624 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f452806c270 0x7f452815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 >> 172.21.15.59:0/1602106624 conn(0x7f45280b24c0 msgr2=0x7f45280b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:47.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 shutdown_connections 2024-06-29T15:59:47.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.002+0000 7f452ea3f640 1 -- 172.21.15.59:0/1602106624 wait complete. 2024-06-29T15:59:47.014 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:59:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.134+0000 7f68321c7640 1 -- 172.21.15.59:0/2161660535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c1458a0 msgr2=0x7f682c0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.134+0000 7f68321c7640 1 --2- 172.21.15.59:0/2161660535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c1458a0 0x7f682c0cf870 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f681c005fd0 tx=0x7f681c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.135+0000 7f68321c7640 1 -- 172.21.15.59:0/2161660535 shutdown_connections 2024-06-29T15:59:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.135+0000 7f68321c7640 1 --2- 172.21.15.59:0/2161660535 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f682c0b3740 0x7f682c0b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.135+0000 7f68321c7640 1 --2- 172.21.15.59:0/2161660535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c1458a0 0x7f682c0cf870 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.135+0000 7f68321c7640 1 -- 172.21.15.59:0/2161660535 >> 172.21.15.59:0/2161660535 conn(0x7f682c06c950 msgr2=0x7f682c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.135+0000 7f68321c7640 1 -- 172.21.15.59:0/2161660535 shutdown_connections 2024-06-29T15:59:47.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.135+0000 7f68321c7640 1 -- 172.21.15.59:0/2161660535 wait complete. 2024-06-29T15:59:47.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.136+0000 7f68321c7640 1 Processor -- start 2024-06-29T15:59:47.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.137+0000 7f68321c7640 1 -- start start 2024-06-29T15:59:47.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f68321c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c0b3740 0x7f682c1f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f68321c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f682c15c2f0 0x7f682c0ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f68321c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f682c15abd0 0x7f682c0af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f68321c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f682c160020 con 0x7f682c0b3740 2024-06-29T15:59:47.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f68321c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f682c15cbb0 con 0x7f682c15c2f0 2024-06-29T15:59:47.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f68321c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f682c1601a0 con 0x7f682c15abd0 2024-06-29T15:59:47.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c0b3740 0x7f682c1f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c0b3740 0x7f682c1f6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32882/0 (socket says 172.21.15.59:32882) 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682b7fe640 1 -- 172.21.15.59:0/3679293099 learned_addr learned my addr 172.21.15.59:0/3679293099 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f6822ffd640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f682c15c2f0 0x7f682c0ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682bfff640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f682c15abd0 0x7f682c0af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682b7fe640 1 -- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f682c15abd0 msgr2=0x7f682c0af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682b7fe640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f682c15abd0 0x7f682c0af160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682b7fe640 1 -- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f682c15c2f0 msgr2=0x7f682c0ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682b7fe640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f682c15c2f0 0x7f682c0ac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.138+0000 7f682b7fe640 1 -- 172.21.15.59:0/3679293099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682c0ae3a0 con 0x7f682c0b3740 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.139+0000 7f682b7fe640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c0b3740 0x7f682c1f6d70 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f681c000c00 tx=0x7f681c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.139+0000 7f6828ff9640 1 -- 172.21.15.59:0/3679293099 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c03e030 con 0x7f682c0b3740 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.139+0000 7f6828ff9640 1 -- 172.21.15.59:0/3679293099 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f681c003ca0 con 0x7f682c0b3740 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.139+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f682c0ae680 con 0x7f682c0b3740 2024-06-29T15:59:47.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.139+0000 7f6828ff9640 1 -- 172.21.15.59:0/3679293099 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c035040 con 0x7f682c0b3740 2024-06-29T15:59:47.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.139+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f682c0acb30 con 0x7f682c0b3740 2024-06-29T15:59:47.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.140+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f682c1606c0 con 0x7f682c0b3740 2024-06-29T15:59:47.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.141+0000 7f6828ff9640 1 -- 172.21.15.59:0/3679293099 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f681c041030 con 0x7f682c0b3740 2024-06-29T15:59:47.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.141+0000 7f6828ff9640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f680c0763f0 0x7f680c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.141+0000 7f6828ff9640 1 -- 172.21.15.59:0/3679293099 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6157+0+0 (secure 0 0 0) 0x7f681c0b8080 con 0x7f682c0b3740 2024-06-29T15:59:47.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.141+0000 7f6822ffd640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f680c0763f0 0x7f680c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.142+0000 7f6822ffd640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f680c0763f0 0x7f680c078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f682c0ab5f0 tx=0x7f681400b2e0 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:47.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.146+0000 7f6828ff9640 1 -- 172.21.15.59:0/3679293099 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f681c080f20 con 0x7f682c0b3740 2024-06-29T15:59:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:47.379+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f682c0b1f10 con 0x7f682c0b3740 2024-06-29T15:59:48.281 INFO:tasks.ceph.mds.c.smithi059.stderr:2024-06-29T15:59:48.281+0000 7f67c28aa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:59:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.316+0000 7f6828ff9640 1 -- 172.21.15.59:0/3679293099 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v156) ==== 118+0+0 (secure 0 0 0) 0x7f681c085220 con 0x7f682c0b3740 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.320+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f680c0763f0 msgr2=0x7f680c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.320+0000 7f68321c7640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f680c0763f0 0x7f680c078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f682c0ab5f0 tx=0x7f681400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.320+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c0b3740 msgr2=0x7f682c1f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.320+0000 7f68321c7640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c0b3740 0x7f682c1f6d70 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f681c000c00 tx=0x7f681c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.321+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 shutdown_connections 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.321+0000 7f68321c7640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f680c0763f0 0x7f680c078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.321+0000 7f68321c7640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f682c15abd0 0x7f682c0af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.321+0000 7f68321c7640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f682c15c2f0 0x7f682c0ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.321+0000 7f68321c7640 1 --2- 172.21.15.59:0/3679293099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f682c0b3740 0x7f682c1f6d70 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.321+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 >> 172.21.15.59:0/3679293099 conn(0x7f682c06c950 msgr2=0x7f682c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.321+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 shutdown_connections 2024-06-29T15:59:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.321+0000 7f68321c7640 1 -- 172.21.15.59:0/3679293099 wait complete. 2024-06-29T15:59:48.335 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:59:48.335 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:59:48.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.464+0000 7f71acc74640 1 -- 172.21.15.59:0/1859548588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a81458a0 msgr2=0x7f71a80cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.464+0000 7f71acc74640 1 --2- 172.21.15.59:0/1859548588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a81458a0 0x7f71a80cf870 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f71a80b1c40 tx=0x7f719802a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.464+0000 7f71acc74640 1 -- 172.21.15.59:0/1859548588 shutdown_connections 2024-06-29T15:59:48.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.464+0000 7f71acc74640 1 --2- 172.21.15.59:0/1859548588 >> v2:172.21.15.153:3300/0 conn(0x7f71a80b3740 0x7f71a80b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.464+0000 7f71acc74640 1 --2- 172.21.15.59:0/1859548588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a81458a0 0x7f71a80cf870 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.464+0000 7f71acc74640 1 -- 172.21.15.59:0/1859548588 >> 172.21.15.59:0/1859548588 conn(0x7f71a806c950 msgr2=0x7f71a8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.464+0000 7f71acc74640 1 -- 172.21.15.59:0/1859548588 shutdown_connections 2024-06-29T15:59:48.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.464+0000 7f71acc74640 1 -- 172.21.15.59:0/1859548588 wait complete. 2024-06-29T15:59:48.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.466+0000 7f71acc74640 1 Processor -- start 2024-06-29T15:59:48.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.466+0000 7f71acc74640 1 -- start start 2024-06-29T15:59:48.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71acc74640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71a80b3740 0x7f71a815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71acc74640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71a815c2f0 0x7f71a813b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71acc74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a80b5840 0x7f71a80b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71acc74640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f71a8160020 con 0x7f71a80b5840 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71acc74640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71a815cbb0 con 0x7f71a815c2f0 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71acc74640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f71a81601a0 con 0x7f71a80b3740 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71a6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a80b5840 0x7f71a80b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71a6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a80b5840 0x7f71a80b5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32902/0 (socket says 172.21.15.59:32902) 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71a6d9d640 1 -- 172.21.15.59:0/1468496227 learned_addr learned my addr 172.21.15.59:0/1468496227 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71a5d9b640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71a815c2f0 0x7f71a813b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.467+0000 7f71a659c640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71a80b3740 0x7f71a815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f71a6d9d640 1 -- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71a80b3740 msgr2=0x7f71a815b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f71a6d9d640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71a80b3740 0x7f71a815b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f71a6d9d640 1 -- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71a815c2f0 msgr2=0x7f71a813b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f71a6d9d640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71a815c2f0 0x7f71a813b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f71a6d9d640 1 -- 172.21.15.59:0/1468496227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a814c460 con 0x7f71a80b5840 2024-06-29T15:59:48.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f71a6d9d640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a80b5840 0x7f71a80b5c50 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7f71a000c3c0 tx=0x7f71a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f719effd640 1 -- 172.21.15.59:0/1468496227 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0011790 con 0x7f71a80b5840 2024-06-29T15:59:48.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f719effd640 1 -- 172.21.15.59:0/1468496227 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71a0017070 con 0x7f71a80b5840 2024-06-29T15:59:48.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f719effd640 1 -- 172.21.15.59:0/1468496227 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0011c40 con 0x7f71a80b5840 2024-06-29T15:59:48.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a815b7e0 con 0x7f71a80b5840 2024-06-29T15:59:48.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.468+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a813e030 con 0x7f71a80b5840 2024-06-29T15:59:48.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.469+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a81606c0 con 0x7f71a80b5840 2024-06-29T15:59:48.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.470+0000 7f719effd640 1 -- 172.21.15.59:0/1468496227 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f71a001c460 con 0x7f71a80b5840 2024-06-29T15:59:48.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.470+0000 7f719effd640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f71880763f0 0x7f7188078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.471+0000 7f719effd640 1 -- 172.21.15.59:0/1468496227 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f71a0097700 con 0x7f71a80b5840 2024-06-29T15:59:48.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.471+0000 7f71a659c640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f71880763f0 0x7f7188078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.471+0000 7f71a659c640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f71880763f0 0x7f7188078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7198034000 tx=0x7f7198004ab0 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.475+0000 7f719effd640 1 -- 172.21.15.59:0/1468496227 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f71a0060740 con 0x7f71a80b5840 2024-06-29T15:59:48.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:48.708+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f71a813ddc0 con 0x7f71a80b5840 2024-06-29T15:59:49.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.318+0000 7f719effd640 1 -- 172.21.15.59:0/1468496227 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v157) ==== 122+0+0 (secure 0 0 0) 0x7f71a0064a40 con 0x7f71a80b5840 2024-06-29T15:59:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.320+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f71880763f0 msgr2=0x7f7188078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.320+0000 7f71acc74640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f71880763f0 0x7f7188078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7198034000 tx=0x7f7198004ab0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.320+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a80b5840 msgr2=0x7f71a80b5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.320+0000 7f71acc74640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a80b5840 0x7f71a80b5c50 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7f71a000c3c0 tx=0x7f71a000c880 comp rx=0 tx=0).stop 2024-06-29T15:59:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.321+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 shutdown_connections 2024-06-29T15:59:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.321+0000 7f71acc74640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f71880763f0 0x7f7188078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.321+0000 7f71acc74640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71a80b5840 0x7f71a80b5c50 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.321+0000 7f71acc74640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71a815c2f0 0x7f71a813b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.321+0000 7f71acc74640 1 --2- 172.21.15.59:0/1468496227 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71a80b3740 0x7f71a815b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.321+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 >> 172.21.15.59:0/1468496227 conn(0x7f71a806c950 msgr2=0x7f71a80ced30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.321+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 shutdown_connections 2024-06-29T15:59:49.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.321+0000 7f71acc74640 1 -- 172.21.15.59:0/1468496227 wait complete. 2024-06-29T15:59:49.335 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:59:49.335 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.457+0000 7fead790a640 1 -- 172.21.15.59:0/3123542134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead01458a0 msgr2=0x7fead00cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.457+0000 7fead790a640 1 --2- 172.21.15.59:0/3123542134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead01458a0 0x7fead00cf870 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7feac0005fd0 tx=0x7feac002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.457+0000 7fead790a640 1 -- 172.21.15.59:0/3123542134 shutdown_connections 2024-06-29T15:59:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.457+0000 7fead790a640 1 --2- 172.21.15.59:0/3123542134 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fead00b3740 0x7fead00b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.457+0000 7fead790a640 1 --2- 172.21.15.59:0/3123542134 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead01458a0 0x7fead00cf870 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.457+0000 7fead790a640 1 -- 172.21.15.59:0/3123542134 >> 172.21.15.59:0/3123542134 conn(0x7fead006c950 msgr2=0x7fead0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.457+0000 7fead790a640 1 -- 172.21.15.59:0/3123542134 shutdown_connections 2024-06-29T15:59:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.457+0000 7fead790a640 1 -- 172.21.15.59:0/3123542134 wait complete. 2024-06-29T15:59:49.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.459+0000 7fead790a640 1 Processor -- start 2024-06-29T15:59:49.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.459+0000 7fead790a640 1 -- start start 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead790a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead00b3740 0x7fead00b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead790a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fead015c2f0 0x7fead01f83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead790a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fead01f4e80 0x7fead01f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead790a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fead0160020 con 0x7fead00b3740 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead790a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fead015cbb0 con 0x7fead01f4e80 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead790a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fead01601a0 con 0x7fead015c2f0 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead56a6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead00b3740 0x7fead00b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead56a6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead00b3740 0x7fead00b10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32908/0 (socket says 172.21.15.59:32908) 2024-06-29T15:59:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.460+0000 7fead56a6640 1 -- 172.21.15.59:0/3889634215 learned_addr learned my addr 172.21.15.59:0/3889634215 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead4ea5640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fead015c2f0 0x7fead01f83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead5ea7640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fead01f4e80 0x7fead01f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead56a6640 1 -- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fead015c2f0 msgr2=0x7fead01f83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead56a6640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fead015c2f0 0x7fead01f83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead56a6640 1 -- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fead01f4e80 msgr2=0x7fead01f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead56a6640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fead01f4e80 0x7fead01f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead56a6640 1 -- 172.21.15.59:0/3889634215 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead00ae9b0 con 0x7fead00b3740 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead5ea7640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fead01f4e80 0x7fead01f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead56a6640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead00b3740 0x7fead00b10f0 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7feac0009cd0 tx=0x7feac0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7feacdffb640 1 -- 172.21.15.59:0/3889634215 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0031b90 con 0x7fead00b3740 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7feacdffb640 1 -- 172.21.15.59:0/3889634215 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feac0031e20 con 0x7fead00b3740 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead015b610 con 0x7fead00b3740 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7feacdffb640 1 -- 172.21.15.59:0/3889634215 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac00332d0 con 0x7fead00b3740 2024-06-29T15:59:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.461+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead00aeeb0 con 0x7fead00b3740 2024-06-29T15:59:49.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.462+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead01606c0 con 0x7fead00b3740 2024-06-29T15:59:49.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.463+0000 7feacdffb640 1 -- 172.21.15.59:0/3889634215 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7feac0033470 con 0x7fead00b3740 2024-06-29T15:59:49.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.464+0000 7feacdffb640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7feabc0763f0 0x7feabc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.464+0000 7feacdffb640 1 -- 172.21.15.59:0/3889634215 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7feac00b97c0 con 0x7fead00b3740 2024-06-29T15:59:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.464+0000 7fead4ea5640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7feabc0763f0 0x7feabc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.465+0000 7fead4ea5640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7feabc0763f0 0x7feabc078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feac4005fd0 tx=0x7feac4005850 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:49.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.467+0000 7feacdffb640 1 -- 172.21.15.59:0/3889634215 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feac0051020 con 0x7fead00b3740 2024-06-29T15:59:49.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.696+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fead00b1f10 con 0x7fead00b3740 2024-06-29T15:59:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.697+0000 7feacdffb640 1 -- 172.21.15.59:0/3889634215 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+5244 (secure 0 0 0) 0x7feac003f030 con 0x7fead00b3740 2024-06-29T15:59:49.697 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:49.697 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":157,"btime":"2024-06-29T15:59:49:317195+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1039110220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1039110220},{"type":"v1","addr":"172.21.15.153:6835","nonce":1039110220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":12583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3996608135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3996608135},{"type":"v1","addr":"172.21.15.153:6837","nonce":3996608135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":13620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/952472867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":952472867},{"type":"v1","addr":"172.21.15.59:6834","nonce":952472867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:45.699384+0000","modified":"2024-06-29T15:59:48.709634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13623":{"gid":13623,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3706652401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3706652401},{"type":"v1","addr":"172.21.15.59:6835","nonce":3706652401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13623,"qdb_cluster":[13623]},"id":18}]} 2024-06-29T15:59:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 157 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.699+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7feabc0763f0 msgr2=0x7feabc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.699+0000 7fead790a640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7feabc0763f0 0x7feabc078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feac4005fd0 tx=0x7feac4005850 comp rx=0 tx=0).stop 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.699+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead00b3740 msgr2=0x7fead00b10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.699+0000 7fead790a640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead00b3740 0x7fead00b10f0 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7feac0009cd0 tx=0x7feac0003970 comp rx=0 tx=0).stop 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.700+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 shutdown_connections 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.700+0000 7fead790a640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7feabc0763f0 0x7feabc078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.700+0000 7fead790a640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fead01f4e80 0x7fead01f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.700+0000 7fead790a640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fead015c2f0 0x7fead01f83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.700+0000 7fead790a640 1 --2- 172.21.15.59:0/3889634215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fead00b3740 0x7fead00b10f0 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.700+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 >> 172.21.15.59:0/3889634215 conn(0x7fead006c950 msgr2=0x7fead0070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.700+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 shutdown_connections 2024-06-29T15:59:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.700+0000 7fead790a640 1 -- 172.21.15.59:0/3889634215 wait complete. 2024-06-29T15:59:49.711 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:49.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.828+0000 7fd542e12640 1 -- 172.21.15.59:0/2486894783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c0ce950 msgr2=0x7fd53c1622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.828+0000 7fd542e12640 1 --2- 172.21.15.59:0/2486894783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c0ce950 0x7fd53c1622c0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7fd524005fd0 tx=0x7fd52402a5a0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.828+0000 7fd542e12640 1 -- 172.21.15.59:0/2486894783 shutdown_connections 2024-06-29T15:59:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.828+0000 7fd542e12640 1 --2- 172.21.15.59:0/2486894783 >> v2:172.21.15.153:3300/0 conn(0x7fd53c162800 0x7fd53c162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.828+0000 7fd542e12640 1 --2- 172.21.15.59:0/2486894783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c0ce950 0x7fd53c1622c0 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.828+0000 7fd542e12640 1 -- 172.21.15.59:0/2486894783 >> 172.21.15.59:0/2486894783 conn(0x7fd53c0b5610 msgr2=0x7fd53c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.828+0000 7fd542e12640 1 -- 172.21.15.59:0/2486894783 shutdown_connections 2024-06-29T15:59:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.828+0000 7fd542e12640 1 -- 172.21.15.59:0/2486894783 wait complete. 2024-06-29T15:59:49.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.830+0000 7fd542e12640 1 Processor -- start 2024-06-29T15:59:49.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.830+0000 7fd542e12640 1 -- start start 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.831+0000 7fd542e12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd53c08c190 0x7fd53c1f8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd542e12640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd53c0ce950 0x7fd53c0c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd542e12640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c162800 0x7fd53c0c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd542e12640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c0b7f60 con 0x7fd53c162800 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd542e12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd53c0b7de0 con 0x7fd53c08c190 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd542e12640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd53c0b80e0 con 0x7fd53c0ce950 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c162800 0x7fd53c0c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c162800 0x7fd53c0c7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32924/0 (socket says 172.21.15.59:32924) 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 -- 172.21.15.59:0/2567075618 learned_addr learned my addr 172.21.15.59:0/2567075618 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd540bae640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd53c08c190 0x7fd53c1f8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd53bfff640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd53c0ce950 0x7fd53c0c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 -- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd53c0ce950 msgr2=0x7fd53c0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd53c0ce950 0x7fd53c0c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 -- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd53c08c190 msgr2=0x7fd53c1f8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd53c08c190 0x7fd53c1f8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 -- 172.21.15.59:0/2567075618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53c0aa590 con 0x7fd53c162800 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5413af640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c162800 0x7fd53c0c7240 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7fd53400e990 tx=0x7fd53400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5397fa640 1 -- 172.21.15.59:0/2567075618 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534004d20 con 0x7fd53c162800 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5397fa640 1 -- 172.21.15.59:0/2567075618 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd53400caa0 con 0x7fd53c162800 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd53c1512d0 con 0x7fd53c162800 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.832+0000 7fd5397fa640 1 -- 172.21.15.59:0/2567075618 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534005020 con 0x7fd53c162800 2024-06-29T15:59:49.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.833+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd53c151890 con 0x7fd53c162800 2024-06-29T15:59:49.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.834+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd53c1622c0 con 0x7fd53c162800 2024-06-29T15:59:49.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.834+0000 7fd5397fa640 1 -- 172.21.15.59:0/2567075618 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fd5340051c0 con 0x7fd53c162800 2024-06-29T15:59:49.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.835+0000 7fd5397fa640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd51c0763f0 0x7fd51c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.835+0000 7fd5397fa640 1 -- 172.21.15.59:0/2567075618 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7fd534098460 con 0x7fd53c162800 2024-06-29T15:59:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.835+0000 7fd540bae640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd51c0763f0 0x7fd51c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.836+0000 7fd540bae640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd51c0763f0 0x7fd51c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd52c007fd0 tx=0x7fd52c009040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:49.839+0000 7fd5397fa640 1 -- 172.21.15.59:0/2567075618 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd534016030 con 0x7fd53c162800 2024-06-29T15:59:50.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.029+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd53c06cf20 con 0x7fd53c162800 2024-06-29T15:59:50.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.030+0000 7fd5397fa640 1 -- 172.21.15.59:0/2567075618 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) ==== 74+0+16998 (secure 0 0 0) 0x7fd534061950 con 0x7fd53c162800 2024-06-29T15:59:50.030 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:50.031 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":275,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T15:59:47.304450+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:44.104733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:44.643605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"275","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd51c0763f0 msgr2=0x7fd51c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd51c0763f0 0x7fd51c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd52c007fd0 tx=0x7fd52c009040 comp rx=0 tx=0).stop 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c162800 msgr2=0x7fd53c0c7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c162800 0x7fd53c0c7240 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7fd53400e990 tx=0x7fd53400ee50 comp rx=0 tx=0).stop 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 shutdown_connections 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fd51c0763f0 0x7fd51c078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd53c162800 0x7fd53c0c7240 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd53c0ce950 0x7fd53c0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 --2- 172.21.15.59:0/2567075618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd53c08c190 0x7fd53c1f8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.032+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 >> 172.21.15.59:0/2567075618 conn(0x7fd53c0b5610 msgr2=0x7fd53c15cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.033+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 shutdown_connections 2024-06-29T15:59:50.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.033+0000 7fd542e12640 1 -- 172.21.15.59:0/2567075618 wait complete. 2024-06-29T15:59:50.044 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:59:50.044 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:59:50.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.162+0000 7f051efa5640 1 -- 172.21.15.59:0/1074372045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05180d2db0 msgr2=0x7f05180b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.162+0000 7f051efa5640 1 --2- 172.21.15.59:0/1074372045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05180d2db0 0x7f05180b3b90 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7f05180d4ba0 tx=0x7f051002a790 comp rx=0 tx=0).stop 2024-06-29T15:59:50.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.162+0000 7f051efa5640 1 -- 172.21.15.59:0/1074372045 shutdown_connections 2024-06-29T15:59:50.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.162+0000 7f051efa5640 1 --2- 172.21.15.59:0/1074372045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05180d2db0 0x7f05180b3b90 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.162+0000 7f051efa5640 1 --2- 172.21.15.59:0/1074372045 >> v2:172.21.15.153:3300/0 conn(0x7f05180b2980 0x7f051815dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.163+0000 7f051efa5640 1 -- 172.21.15.59:0/1074372045 >> 172.21.15.59:0/1074372045 conn(0x7f05180d3b20 msgr2=0x7f05180d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.163+0000 7f051efa5640 1 -- 172.21.15.59:0/1074372045 shutdown_connections 2024-06-29T15:59:50.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.163+0000 7f051efa5640 1 -- 172.21.15.59:0/1074372045 wait complete. 2024-06-29T15:59:50.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.164+0000 7f051efa5640 1 Processor -- start 2024-06-29T15:59:50.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.164+0000 7f051efa5640 1 -- start start 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051efa5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05180b2980 0x7f05181f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051efa5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05180cefb0 0x7f0518159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051efa5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0518149130 0x7f0518160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051efa5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f05180b4520 con 0x7f0518149130 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051efa5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05180b43a0 con 0x7f05180b2980 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051efa5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f05180b46a0 con 0x7f05180cefb0 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0518149130 0x7f0518160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0518149130 0x7f0518160720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32942/0 (socket says 172.21.15.59:32942) 2024-06-29T15:59:50.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 -- 172.21.15.59:0/1376724341 learned_addr learned my addr 172.21.15.59:0/1376724341 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051cd41640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05180b2980 0x7f05181f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f0517fff640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05180cefb0 0x7f0518159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 -- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05180cefb0 msgr2=0x7f0518159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05180cefb0 0x7f0518159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 -- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05180b2980 msgr2=0x7f05181f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05180b2980 0x7f05181f5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 -- 172.21.15.59:0/1376724341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0518159cb0 con 0x7f0518149130 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f0517fff640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05180cefb0 0x7f0518159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.166+0000 7f051d542640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0518149130 0x7f0518160720 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f0510009ca0 tx=0x7f0510032f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.167+0000 7f05157fa640 1 -- 172.21.15.59:0/1376724341 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0510041020 con 0x7f0518149130 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.167+0000 7f05157fa640 1 -- 172.21.15.59:0/1376724341 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0510004330 con 0x7f0518149130 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.167+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051815f680 con 0x7f0518149130 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.167+0000 7f05157fa640 1 -- 172.21.15.59:0/1376724341 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0510036040 con 0x7f0518149130 2024-06-29T15:59:50.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.167+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f051815fc40 con 0x7f0518149130 2024-06-29T15:59:50.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.168+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05180b4be0 con 0x7f0518149130 2024-06-29T15:59:50.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.168+0000 7f05157fa640 1 -- 172.21.15.59:0/1376724341 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f0510044020 con 0x7f0518149130 2024-06-29T15:59:50.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.169+0000 7f05157fa640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f04fc0763f0 0x7f04fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.169+0000 7f05157fa640 1 -- 172.21.15.59:0/1376724341 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f05100b8d70 con 0x7f0518149130 2024-06-29T15:59:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.169+0000 7f051cd41640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f04fc0763f0 0x7f04fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.170+0000 7f051cd41640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f04fc0763f0 0x7f04fc078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f050c005fd0 tx=0x7f050c00a250 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:50.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.173+0000 7f05157fa640 1 -- 172.21.15.59:0/1376724341 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f051004b030 con 0x7f0518149130 2024-06-29T15:59:50.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.362+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 --> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f05180b96e0 con 0x7f04fc0763f0 2024-06-29T15:59:50.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.367+0000 7f05157fa640 1 -- 172.21.15.59:0/1376724341 <== mgr.12382 v2:172.21.15.153:6832/3072370296 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252465 (secure 0 0 0) 0x7f05180b96e0 con 0x7f04fc0763f0 2024-06-29T15:59:50.369 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:50.371 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:59:50.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f04fc0763f0 msgr2=0x7f04fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f04fc0763f0 0x7f04fc078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f050c005fd0 tx=0x7f050c00a250 comp rx=0 tx=0).stop 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0518149130 msgr2=0x7f0518160720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0518149130 0x7f0518160720 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f0510009ca0 tx=0x7f0510032f10 comp rx=0 tx=0).stop 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 shutdown_connections 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f04fc0763f0 0x7f04fc078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0518149130 0x7f0518160720 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f05180cefb0 0x7f0518159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 --2- 172.21.15.59:0/1376724341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05180b2980 0x7f05181f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 >> 172.21.15.59:0/1376724341 conn(0x7f05180d3b20 msgr2=0x7f0518161aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 shutdown_connections 2024-06-29T15:59:50.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.370+0000 7f051efa5640 1 -- 172.21.15.59:0/1376724341 wait complete. 2024-06-29T15:59:50.382 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:59:49.863479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1788,"num_read_kb":5144,"num_write":490,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431856,"kb_used_data":28288,"kb_used_omap":12,"kb_used_meta":403443,"kb_avail":754542864,"statfs":{"total":773094113280,"available":772651892736,"internally_reserved":0,"allocated":28966912,"data_stored":15574236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":413126194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.573656"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.306670+0000","last_change":"2024-06-29T15:59:46.305324+0000","last_active":"2024-06-29T15:59:46.306670+0000","last_peered":"2024-06-29T15:59:46.306670+0000","last_clean":"2024-06-29T15:59:46.306670+0000","last_became_active":"2024-06-29T15:59:46.304977+0000","last_became_peered":"2024-06-29T15:59:46.304977+0000","last_unstale":"2024-06-29T15:59:46.306670+0000","last_undegraded":"2024-06-29T15:59:46.306670+0000","last_fullsized":"2024-06-29T15:59:46.306670+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:03.763576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318452+0000","last_change":"2024-06-29T15:59:46.329066+0000","last_active":"2024-06-29T15:59:47.318452+0000","last_peered":"2024-06-29T15:59:47.318452+0000","last_clean":"2024-06-29T15:59:47.318452+0000","last_became_active":"2024-06-29T15:59:46.327443+0000","last_became_peered":"2024-06-29T15:59:46.327443+0000","last_unstale":"2024-06-29T15:59:47.318452+0000","last_undegraded":"2024-06-29T15:59:47.318452+0000","last_fullsized":"2024-06-29T15:59:47.318452+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:30:48.394838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313354+0000","last_change":"2024-06-29T15:59:46.310384+0000","last_active":"2024-06-29T15:59:47.313354+0000","last_peered":"2024-06-29T15:59:47.313354+0000","last_clean":"2024-06-29T15:59:47.313354+0000","last_became_active":"2024-06-29T15:59:46.310034+0000","last_became_peered":"2024-06-29T15:59:46.310034+0000","last_unstale":"2024-06-29T15:59:47.313354+0000","last_undegraded":"2024-06-29T15:59:47.313354+0000","last_fullsized":"2024-06-29T15:59:47.313354+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:08.617490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432183+0000","last_change":"2024-06-29T15:59:46.328214+0000","last_active":"2024-06-29T15:59:47.432183+0000","last_peered":"2024-06-29T15:59:47.432183+0000","last_clean":"2024-06-29T15:59:47.432183+0000","last_became_active":"2024-06-29T15:59:46.327587+0000","last_became_peered":"2024-06-29T15:59:46.327587+0000","last_unstale":"2024-06-29T15:59:47.432183+0000","last_undegraded":"2024-06-29T15:59:47.432183+0000","last_fullsized":"2024-06-29T15:59:47.432183+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:04:36.707743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312612+0000","last_change":"2024-06-29T15:59:46.311208+0000","last_active":"2024-06-29T15:59:46.312612+0000","last_peered":"2024-06-29T15:59:46.312612+0000","last_clean":"2024-06-29T15:59:46.312612+0000","last_became_active":"2024-06-29T15:59:46.310665+0000","last_became_peered":"2024-06-29T15:59:46.310665+0000","last_unstale":"2024-06-29T15:59:46.312612+0000","last_undegraded":"2024-06-29T15:59:46.312612+0000","last_fullsized":"2024-06-29T15:59:46.312612+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:43.538474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.321210+0000","last_change":"2024-06-29T15:59:46.317909+0000","last_active":"2024-06-29T15:59:46.321210+0000","last_peered":"2024-06-29T15:59:46.321210+0000","last_clean":"2024-06-29T15:59:46.321210+0000","last_became_active":"2024-06-29T15:59:46.317563+0000","last_became_peered":"2024-06-29T15:59:46.317563+0000","last_unstale":"2024-06-29T15:59:46.321210+0000","last_undegraded":"2024-06-29T15:59:46.321210+0000","last_fullsized":"2024-06-29T15:59:46.321210+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:45:59.347478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318457+0000","last_change":"2024-06-29T15:59:46.303400+0000","last_active":"2024-06-29T15:59:47.318457+0000","last_peered":"2024-06-29T15:59:47.318457+0000","last_clean":"2024-06-29T15:59:47.318457+0000","last_became_active":"2024-06-29T15:59:46.303091+0000","last_became_peered":"2024-06-29T15:59:46.303091+0000","last_unstale":"2024-06-29T15:59:47.318457+0000","last_undegraded":"2024-06-29T15:59:47.318457+0000","last_fullsized":"2024-06-29T15:59:47.318457+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.680412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312890+0000","last_change":"2024-06-29T15:59:46.311575+0000","last_active":"2024-06-29T15:59:46.312890+0000","last_peered":"2024-06-29T15:59:46.312890+0000","last_clean":"2024-06-29T15:59:46.312890+0000","last_became_active":"2024-06-29T15:59:46.311223+0000","last_became_peered":"2024-06-29T15:59:46.311223+0000","last_unstale":"2024-06-29T15:59:46.312890+0000","last_undegraded":"2024-06-29T15:59:46.312890+0000","last_fullsized":"2024-06-29T15:59:46.312890+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:30:15.280262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312336+0000","last_change":"2024-06-29T15:59:46.307608+0000","last_active":"2024-06-29T15:59:46.312336+0000","last_peered":"2024-06-29T15:59:46.312336+0000","last_clean":"2024-06-29T15:59:46.312336+0000","last_became_active":"2024-06-29T15:59:46.307292+0000","last_became_peered":"2024-06-29T15:59:46.307292+0000","last_unstale":"2024-06-29T15:59:46.312336+0000","last_undegraded":"2024-06-29T15:59:46.312336+0000","last_fullsized":"2024-06-29T15:59:46.312336+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:24.226917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.314807+0000","last_change":"2024-06-29T15:59:46.313902+0000","last_active":"2024-06-29T15:59:46.314807+0000","last_peered":"2024-06-29T15:59:46.314807+0000","last_clean":"2024-06-29T15:59:46.314807+0000","last_became_active":"2024-06-29T15:59:46.313568+0000","last_became_peered":"2024-06-29T15:59:46.313568+0000","last_unstale":"2024-06-29T15:59:46.314807+0000","last_undegraded":"2024-06-29T15:59:46.314807+0000","last_fullsized":"2024-06-29T15:59:46.314807+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:32.104069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.311585+0000","last_change":"2024-06-29T15:59:46.309964+0000","last_active":"2024-06-29T15:59:46.311585+0000","last_peered":"2024-06-29T15:59:46.311585+0000","last_clean":"2024-06-29T15:59:46.311585+0000","last_became_active":"2024-06-29T15:59:46.309603+0000","last_became_peered":"2024-06-29T15:59:46.309603+0000","last_unstale":"2024-06-29T15:59:46.311585+0000","last_undegraded":"2024-06-29T15:59:46.311585+0000","last_fullsized":"2024-06-29T15:59:46.311585+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:28.221179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.321243+0000","last_change":"2024-06-29T15:59:46.319544+0000","last_active":"2024-06-29T15:59:46.321243+0000","last_peered":"2024-06-29T15:59:46.321243+0000","last_clean":"2024-06-29T15:59:46.321243+0000","last_became_active":"2024-06-29T15:59:46.319240+0000","last_became_peered":"2024-06-29T15:59:46.319240+0000","last_unstale":"2024-06-29T15:59:46.321243+0000","last_undegraded":"2024-06-29T15:59:46.321243+0000","last_fullsized":"2024-06-29T15:59:46.321243+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:30.681706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312590+0000","last_change":"2024-06-29T15:59:46.310949+0000","last_active":"2024-06-29T15:59:46.312590+0000","last_peered":"2024-06-29T15:59:46.312590+0000","last_clean":"2024-06-29T15:59:46.312590+0000","last_became_active":"2024-06-29T15:59:46.310312+0000","last_became_peered":"2024-06-29T15:59:46.310312+0000","last_unstale":"2024-06-29T15:59:46.312590+0000","last_undegraded":"2024-06-29T15:59:46.312590+0000","last_fullsized":"2024-06-29T15:59:46.312590+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:21:23.566139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317831+0000","last_change":"2024-06-29T15:59:46.327840+0000","last_active":"2024-06-29T15:59:47.317831+0000","last_peered":"2024-06-29T15:59:47.317831+0000","last_clean":"2024-06-29T15:59:47.317831+0000","last_became_active":"2024-06-29T15:59:46.327540+0000","last_became_peered":"2024-06-29T15:59:46.327540+0000","last_unstale":"2024-06-29T15:59:47.317831+0000","last_undegraded":"2024-06-29T15:59:47.317831+0000","last_fullsized":"2024-06-29T15:59:47.317831+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:25:19.976178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317856+0000","last_change":"2024-06-29T15:59:46.316808+0000","last_active":"2024-06-29T15:59:47.317856+0000","last_peered":"2024-06-29T15:59:47.317856+0000","last_clean":"2024-06-29T15:59:47.317856+0000","last_became_active":"2024-06-29T15:59:46.316475+0000","last_became_peered":"2024-06-29T15:59:46.316475+0000","last_unstale":"2024-06-29T15:59:47.317856+0000","last_undegraded":"2024-06-29T15:59:47.317856+0000","last_fullsized":"2024-06-29T15:59:47.317856+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:21.057883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.316780+0000","last_change":"2024-06-29T15:59:46.315370+0000","last_active":"2024-06-29T15:59:46.316780+0000","last_peered":"2024-06-29T15:59:46.316780+0000","last_clean":"2024-06-29T15:59:46.316780+0000","last_became_active":"2024-06-29T15:59:46.315064+0000","last_became_peered":"2024-06-29T15:59:46.315064+0000","last_unstale":"2024-06-29T15:59:46.316780+0000","last_undegraded":"2024-06-29T15:59:46.316780+0000","last_fullsized":"2024-06-29T15:59:46.316780+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:19.185984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312299+0000","last_change":"2024-06-29T15:59:46.306641+0000","last_active":"2024-06-29T15:59:46.312299+0000","last_peered":"2024-06-29T15:59:46.312299+0000","last_clean":"2024-06-29T15:59:46.312299+0000","last_became_active":"2024-06-29T15:59:46.306302+0000","last_became_peered":"2024-06-29T15:59:46.306302+0000","last_unstale":"2024-06-29T15:59:46.312299+0000","last_undegraded":"2024-06-29T15:59:46.312299+0000","last_fullsized":"2024-06-29T15:59:46.312299+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:06.706794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432500+0000","last_change":"2024-06-29T15:59:46.318108+0000","last_active":"2024-06-29T15:59:47.432500+0000","last_peered":"2024-06-29T15:59:47.432500+0000","last_clean":"2024-06-29T15:59:47.432500+0000","last_became_active":"2024-06-29T15:59:46.316041+0000","last_became_peered":"2024-06-29T15:59:46.316041+0000","last_unstale":"2024-06-29T15:59:47.432500+0000","last_undegraded":"2024-06-29T15:59:47.432500+0000","last_fullsized":"2024-06-29T15:59:47.432500+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:43.209978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313404+0000","last_change":"2024-06-29T15:59:46.308533+0000","last_active":"2024-06-29T15:59:47.313404+0000","last_peered":"2024-06-29T15:59:47.313404+0000","last_clean":"2024-06-29T15:59:47.313404+0000","last_became_active":"2024-06-29T15:59:46.308170+0000","last_became_peered":"2024-06-29T15:59:46.308170+0000","last_unstale":"2024-06-29T15:59:47.313404+0000","last_undegraded":"2024-06-29T15:59:47.313404+0000","last_fullsized":"2024-06-29T15:59:47.313404+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:03.911477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432198+0000","last_change":"2024-06-29T15:59:46.320836+0000","last_active":"2024-06-29T15:59:47.432198+0000","last_peered":"2024-06-29T15:59:47.432198+0000","last_clean":"2024-06-29T15:59:47.432198+0000","last_became_active":"2024-06-29T15:59:46.320363+0000","last_became_peered":"2024-06-29T15:59:46.320363+0000","last_unstale":"2024-06-29T15:59:47.432198+0000","last_undegraded":"2024-06-29T15:59:47.432198+0000","last_fullsized":"2024-06-29T15:59:47.432198+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:55:16.894553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317956+0000","last_change":"2024-06-29T15:59:46.326783+0000","last_active":"2024-06-29T15:59:47.317956+0000","last_peered":"2024-06-29T15:59:47.317956+0000","last_clean":"2024-06-29T15:59:47.317956+0000","last_became_active":"2024-06-29T15:59:46.325921+0000","last_became_peered":"2024-06-29T15:59:46.325921+0000","last_unstale":"2024-06-29T15:59:47.317956+0000","last_undegraded":"2024-06-29T15:59:47.317956+0000","last_fullsized":"2024-06-29T15:59:47.317956+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:56.734569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.431950+0000","last_change":"2024-06-29T15:59:46.327179+0000","last_active":"2024-06-29T15:59:47.431950+0000","last_peered":"2024-06-29T15:59:47.431950+0000","last_clean":"2024-06-29T15:59:47.431950+0000","last_became_active":"2024-06-29T15:59:46.326874+0000","last_became_peered":"2024-06-29T15:59:46.326874+0000","last_unstale":"2024-06-29T15:59:47.431950+0000","last_undegraded":"2024-06-29T15:59:47.431950+0000","last_fullsized":"2024-06-29T15:59:47.431950+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:59:51.387874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.431970+0000","last_change":"2024-06-29T15:59:46.320203+0000","last_active":"2024-06-29T15:59:47.431970+0000","last_peered":"2024-06-29T15:59:47.431970+0000","last_clean":"2024-06-29T15:59:47.431970+0000","last_became_active":"2024-06-29T15:59:46.317266+0000","last_became_peered":"2024-06-29T15:59:46.317266+0000","last_unstale":"2024-06-29T15:59:47.431970+0000","last_undegraded":"2024-06-29T15:59:47.431970+0000","last_fullsized":"2024-06-29T15:59:47.431970+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:17:57.910816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313132+0000","last_change":"2024-06-29T15:59:46.316574+0000","last_active":"2024-06-29T15:59:47.313132+0000","last_peered":"2024-06-29T15:59:47.313132+0000","last_clean":"2024-06-29T15:59:47.313132+0000","last_became_active":"2024-06-29T15:59:46.316263+0000","last_became_peered":"2024-06-29T15:59:46.316263+0000","last_unstale":"2024-06-29T15:59:47.313132+0000","last_undegraded":"2024-06-29T15:59:47.313132+0000","last_fullsized":"2024-06-29T15:59:47.313132+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:18.163385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.326898+0000","last_change":"2024-06-29T15:59:46.325206+0000","last_active":"2024-06-29T15:59:46.326898+0000","last_peered":"2024-06-29T15:59:46.326898+0000","last_clean":"2024-06-29T15:59:46.326898+0000","last_became_active":"2024-06-29T15:59:46.324853+0000","last_became_peered":"2024-06-29T15:59:46.324853+0000","last_unstale":"2024-06-29T15:59:46.326898+0000","last_undegraded":"2024-06-29T15:59:46.326898+0000","last_fullsized":"2024-06-29T15:59:46.326898+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:54.514611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432425+0000","last_change":"2024-06-29T15:59:46.318868+0000","last_active":"2024-06-29T15:59:47.432425+0000","last_peered":"2024-06-29T15:59:47.432425+0000","last_clean":"2024-06-29T15:59:47.432425+0000","last_became_active":"2024-06-29T15:59:46.318528+0000","last_became_peered":"2024-06-29T15:59:46.318528+0000","last_unstale":"2024-06-29T15:59:47.432425+0000","last_undegraded":"2024-06-29T15:59:47.432425+0000","last_fullsized":"2024-06-29T15:59:47.432425+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:44.676408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.309656+0000","last_change":"2024-06-29T15:59:46.307500+0000","last_active":"2024-06-29T15:59:46.309656+0000","last_peered":"2024-06-29T15:59:46.309656+0000","last_clean":"2024-06-29T15:59:46.309656+0000","last_became_active":"2024-06-29T15:59:46.307170+0000","last_became_peered":"2024-06-29T15:59:46.307170+0000","last_unstale":"2024-06-29T15:59:46.309656+0000","last_undegraded":"2024-06-29T15:59:46.309656+0000","last_fullsized":"2024-06-29T15:59:46.309656+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:25.113545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318762+0000","last_change":"2024-06-29T15:59:46.310804+0000","last_active":"2024-06-29T15:59:47.318762+0000","last_peered":"2024-06-29T15:59:47.318762+0000","last_clean":"2024-06-29T15:59:47.318762+0000","last_became_active":"2024-06-29T15:59:46.310482+0000","last_became_peered":"2024-06-29T15:59:46.310482+0000","last_unstale":"2024-06-29T15:59:47.318762+0000","last_undegraded":"2024-06-29T15:59:47.318762+0000","last_fullsized":"2024-06-29T15:59:47.318762+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:31:20.869749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317785+0000","last_change":"2024-06-29T15:59:46.326163+0000","last_active":"2024-06-29T15:59:47.317785+0000","last_peered":"2024-06-29T15:59:47.317785+0000","last_clean":"2024-06-29T15:59:47.317785+0000","last_became_active":"2024-06-29T15:59:46.325843+0000","last_became_peered":"2024-06-29T15:59:46.325843+0000","last_unstale":"2024-06-29T15:59:47.317785+0000","last_undegraded":"2024-06-29T15:59:47.317785+0000","last_fullsized":"2024-06-29T15:59:47.317785+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:13.898105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318424+0000","last_change":"2024-06-29T15:59:46.307124+0000","last_active":"2024-06-29T15:59:47.318424+0000","last_peered":"2024-06-29T15:59:47.318424+0000","last_clean":"2024-06-29T15:59:47.318424+0000","last_became_active":"2024-06-29T15:59:46.306809+0000","last_became_peered":"2024-06-29T15:59:46.306809+0000","last_unstale":"2024-06-29T15:59:47.318424+0000","last_undegraded":"2024-06-29T15:59:47.318424+0000","last_fullsized":"2024-06-29T15:59:47.318424+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:50:09.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.431924+0000","last_change":"2024-06-29T15:59:46.324494+0000","last_active":"2024-06-29T15:59:47.431924+0000","last_peered":"2024-06-29T15:59:47.431924+0000","last_clean":"2024-06-29T15:59:47.431924+0000","last_became_active":"2024-06-29T15:59:46.324191+0000","last_became_peered":"2024-06-29T15:59:46.324191+0000","last_unstale":"2024-06-29T15:59:47.431924+0000","last_undegraded":"2024-06-29T15:59:47.431924+0000","last_fullsized":"2024-06-29T15:59:47.431924+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:28.437703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432476+0000","last_change":"2024-06-29T15:59:46.321593+0000","last_active":"2024-06-29T15:59:47.432476+0000","last_peered":"2024-06-29T15:59:47.432476+0000","last_clean":"2024-06-29T15:59:47.432476+0000","last_became_active":"2024-06-29T15:59:46.321301+0000","last_became_peered":"2024-06-29T15:59:46.321301+0000","last_unstale":"2024-06-29T15:59:47.432476+0000","last_undegraded":"2024-06-29T15:59:47.432476+0000","last_fullsized":"2024-06-29T15:59:47.432476+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:38:37.537548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"274'4","reported_seq":16,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.331088+0000","last_change":"2024-06-29T15:59:45.311920+0000","last_active":"2024-06-29T15:59:46.331088+0000","last_peered":"2024-06-29T15:59:46.331088+0000","last_clean":"2024-06-29T15:59:46.331088+0000","last_became_active":"2024-06-29T15:59:45.311587+0000","last_became_peered":"2024-06-29T15:59:45.311587+0000","last_unstale":"2024-06-29T15:59:46.331088+0000","last_undegraded":"2024-06-29T15:59:46.331088+0000","last_fullsized":"2024-06-29T15:59:46.331088+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:58.332086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312703+0000","last_change":"2024-06-29T15:59:46.309550+0000","last_active":"2024-06-29T15:59:46.312703+0000","last_peered":"2024-06-29T15:59:46.312703+0000","last_clean":"2024-06-29T15:59:46.312703+0000","last_became_active":"2024-06-29T15:59:46.309216+0000","last_became_peered":"2024-06-29T15:59:46.309216+0000","last_unstale":"2024-06-29T15:59:46.312703+0000","last_undegraded":"2024-06-29T15:59:46.312703+0000","last_fullsized":"2024-06-29T15:59:46.312703+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:51.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"274'1","reported_seq":15,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432475+0000","last_change":"2024-06-29T15:59:45.312200+0000","last_active":"2024-06-29T15:59:47.432475+0000","last_peered":"2024-06-29T15:59:47.432475+0000","last_clean":"2024-06-29T15:59:47.432475+0000","last_became_active":"2024-06-29T15:59:45.311885+0000","last_became_peered":"2024-06-29T15:59:45.311885+0000","last_unstale":"2024-06-29T15:59:47.432475+0000","last_undegraded":"2024-06-29T15:59:47.432475+0000","last_fullsized":"2024-06-29T15:59:47.432475+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:55.675973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318603+0000","last_change":"2024-06-29T15:59:46.330485+0000","last_active":"2024-06-29T15:59:47.318603+0000","last_peered":"2024-06-29T15:59:47.318603+0000","last_clean":"2024-06-29T15:59:47.318603+0000","last_became_active":"2024-06-29T15:59:46.328106+0000","last_became_peered":"2024-06-29T15:59:46.328106+0000","last_unstale":"2024-06-29T15:59:47.318603+0000","last_undegraded":"2024-06-29T15:59:47.318603+0000","last_fullsized":"2024-06-29T15:59:47.318603+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:13.205898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"274'2","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313484+0000","last_change":"2024-06-29T15:59:45.302817+0000","last_active":"2024-06-29T15:59:47.313484+0000","last_peered":"2024-06-29T15:59:47.313484+0000","last_clean":"2024-06-29T15:59:47.313484+0000","last_became_active":"2024-06-29T15:59:45.302474+0000","last_became_peered":"2024-06-29T15:59:45.302474+0000","last_unstale":"2024-06-29T15:59:47.313484+0000","last_undegraded":"2024-06-29T15:59:47.313484+0000","last_fullsized":"2024-06-29T15:59:47.313484+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:27.116567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318112+0000","last_change":"2024-06-29T15:59:46.325364+0000","last_active":"2024-06-29T15:59:47.318112+0000","last_peered":"2024-06-29T15:59:47.318112+0000","last_clean":"2024-06-29T15:59:47.318112+0000","last_became_active":"2024-06-29T15:59:46.325002+0000","last_became_peered":"2024-06-29T15:59:46.325002+0000","last_unstale":"2024-06-29T15:59:47.318112+0000","last_undegraded":"2024-06-29T15:59:47.318112+0000","last_fullsized":"2024-06-29T15:59:47.318112+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:30.030322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"274'2","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318053+0000","last_change":"2024-06-29T15:59:45.311877+0000","last_active":"2024-06-29T15:59:47.318053+0000","last_peered":"2024-06-29T15:59:47.318053+0000","last_clean":"2024-06-29T15:59:47.318053+0000","last_became_active":"2024-06-29T15:59:45.311530+0000","last_became_peered":"2024-06-29T15:59:45.311530+0000","last_unstale":"2024-06-29T15:59:47.318053+0000","last_undegraded":"2024-06-29T15:59:47.318053+0000","last_fullsized":"2024-06-29T15:59:47.318053+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:41.372645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.316361+0000","last_change":"2024-06-29T15:59:46.315081+0000","last_active":"2024-06-29T15:59:46.316361+0000","last_peered":"2024-06-29T15:59:46.316361+0000","last_clean":"2024-06-29T15:59:46.316361+0000","last_became_active":"2024-06-29T15:59:46.314785+0000","last_became_peered":"2024-06-29T15:59:46.314785+0000","last_unstale":"2024-06-29T15:59:46.316361+0000","last_undegraded":"2024-06-29T15:59:46.316361+0000","last_fullsized":"2024-06-29T15:59:46.316361+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:39:31.616716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"274'1","reported_seq":13,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.327883+0000","last_change":"2024-06-29T15:59:45.317104+0000","last_active":"2024-06-29T15:59:46.327883+0000","last_peered":"2024-06-29T15:59:46.327883+0000","last_clean":"2024-06-29T15:59:46.327883+0000","last_became_active":"2024-06-29T15:59:45.316754+0000","last_became_peered":"2024-06-29T15:59:45.316754+0000","last_unstale":"2024-06-29T15:59:46.327883+0000","last_undegraded":"2024-06-29T15:59:46.327883+0000","last_fullsized":"2024-06-29T15:59:46.327883+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:12.659077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.311393+0000","last_change":"2024-06-29T15:59:46.310423+0000","last_active":"2024-06-29T15:59:46.311393+0000","last_peered":"2024-06-29T15:59:46.311393+0000","last_clean":"2024-06-29T15:59:46.311393+0000","last_became_active":"2024-06-29T15:59:46.310017+0000","last_became_peered":"2024-06-29T15:59:46.310017+0000","last_unstale":"2024-06-29T15:59:46.311393+0000","last_undegraded":"2024-06-29T15:59:46.311393+0000","last_fullsized":"2024-06-29T15:59:46.311393+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:52.145571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"274'1","reported_seq":15,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313470+0000","last_change":"2024-06-29T15:59:45.304202+0000","last_active":"2024-06-29T15:59:47.313470+0000","last_peered":"2024-06-29T15:59:47.313470+0000","last_clean":"2024-06-29T15:59:47.313470+0000","last_became_active":"2024-06-29T15:59:45.303857+0000","last_became_peered":"2024-06-29T15:59:45.303857+0000","last_unstale":"2024-06-29T15:59:47.313470+0000","last_undegraded":"2024-06-29T15:59:47.313470+0000","last_fullsized":"2024-06-29T15:59:47.313470+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:51:53.509459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432210+0000","last_change":"2024-06-29T15:59:46.317777+0000","last_active":"2024-06-29T15:59:47.432210+0000","last_peered":"2024-06-29T15:59:47.432210+0000","last_clean":"2024-06-29T15:59:47.432210+0000","last_became_active":"2024-06-29T15:59:46.317328+0000","last_became_peered":"2024-06-29T15:59:46.317328+0000","last_unstale":"2024-06-29T15:59:47.432210+0000","last_undegraded":"2024-06-29T15:59:47.432210+0000","last_fullsized":"2024-06-29T15:59:47.432210+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:45.927002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318108+0000","last_change":"2024-06-29T15:59:45.312310+0000","last_active":"2024-06-29T15:59:47.318108+0000","last_peered":"2024-06-29T15:59:47.318108+0000","last_clean":"2024-06-29T15:59:47.318108+0000","last_became_active":"2024-06-29T15:59:45.311790+0000","last_became_peered":"2024-06-29T15:59:45.311790+0000","last_unstale":"2024-06-29T15:59:47.318108+0000","last_undegraded":"2024-06-29T15:59:47.318108+0000","last_fullsized":"2024-06-29T15:59:47.318108+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:06.806536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318012+0000","last_change":"2024-06-29T15:59:46.325400+0000","last_active":"2024-06-29T15:59:47.318012+0000","last_peered":"2024-06-29T15:59:47.318012+0000","last_clean":"2024-06-29T15:59:47.318012+0000","last_became_active":"2024-06-29T15:59:46.323267+0000","last_became_peered":"2024-06-29T15:59:46.323267+0000","last_unstale":"2024-06-29T15:59:47.318012+0000","last_undegraded":"2024-06-29T15:59:47.318012+0000","last_fullsized":"2024-06-29T15:59:47.318012+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:47:34.577970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318022+0000","last_change":"2024-06-29T15:59:45.307319+0000","last_active":"2024-06-29T15:59:47.318022+0000","last_peered":"2024-06-29T15:59:47.318022+0000","last_clean":"2024-06-29T15:59:47.318022+0000","last_became_active":"2024-06-29T15:59:45.306995+0000","last_became_peered":"2024-06-29T15:59:45.306995+0000","last_unstale":"2024-06-29T15:59:47.318022+0000","last_undegraded":"2024-06-29T15:59:47.318022+0000","last_fullsized":"2024-06-29T15:59:47.318022+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:55.816704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432471+0000","last_change":"2024-06-29T15:59:46.315319+0000","last_active":"2024-06-29T15:59:47.432471+0000","last_peered":"2024-06-29T15:59:47.432471+0000","last_clean":"2024-06-29T15:59:47.432471+0000","last_became_active":"2024-06-29T15:59:46.315018+0000","last_became_peered":"2024-06-29T15:59:46.315018+0000","last_unstale":"2024-06-29T15:59:47.432471+0000","last_undegraded":"2024-06-29T15:59:47.432471+0000","last_fullsized":"2024-06-29T15:59:47.432471+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:51:59.720938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.299705+0000","last_change":"2024-06-29T15:59:45.308295+0000","last_active":"2024-06-29T15:59:46.299705+0000","last_peered":"2024-06-29T15:59:46.299705+0000","last_clean":"2024-06-29T15:59:46.299705+0000","last_became_active":"2024-06-29T15:59:45.307913+0000","last_became_peered":"2024-06-29T15:59:45.307913+0000","last_unstale":"2024-06-29T15:59:46.299705+0000","last_undegraded":"2024-06-29T15:59:46.299705+0000","last_fullsized":"2024-06-29T15:59:46.299705+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:38.979304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.328717+0000","last_change":"2024-06-29T15:59:46.324805+0000","last_active":"2024-06-29T15:59:46.328717+0000","last_peered":"2024-06-29T15:59:46.328717+0000","last_clean":"2024-06-29T15:59:46.328717+0000","last_became_active":"2024-06-29T15:59:46.324474+0000","last_became_peered":"2024-06-29T15:59:46.324474+0000","last_unstale":"2024-06-29T15:59:46.328717+0000","last_undegraded":"2024-06-29T15:59:46.328717+0000","last_fullsized":"2024-06-29T15:59:46.328717+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:21.101877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.319012+0000","last_change":"2024-06-29T15:59:45.318294+0000","last_active":"2024-06-29T15:59:47.319012+0000","last_peered":"2024-06-29T15:59:47.319012+0000","last_clean":"2024-06-29T15:59:47.319012+0000","last_became_active":"2024-06-29T15:59:45.317757+0000","last_became_peered":"2024-06-29T15:59:45.317757+0000","last_unstale":"2024-06-29T15:59:47.319012+0000","last_undegraded":"2024-06-29T15:59:47.319012+0000","last_fullsized":"2024-06-29T15:59:47.319012+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:34:20.426052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318752+0000","last_change":"2024-06-29T15:59:46.329429+0000","last_active":"2024-06-29T15:59:47.318752+0000","last_peered":"2024-06-29T15:59:47.318752+0000","last_clean":"2024-06-29T15:59:47.318752+0000","last_became_active":"2024-06-29T15:59:46.326947+0000","last_became_peered":"2024-06-29T15:59:46.326947+0000","last_unstale":"2024-06-29T15:59:47.318752+0000","last_undegraded":"2024-06-29T15:59:47.318752+0000","last_fullsized":"2024-06-29T15:59:47.318752+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:02.906463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"274'3","reported_seq":17,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318120+0000","last_change":"2024-06-29T15:59:45.314428+0000","last_active":"2024-06-29T15:59:47.318120+0000","last_peered":"2024-06-29T15:59:47.318120+0000","last_clean":"2024-06-29T15:59:47.318120+0000","last_became_active":"2024-06-29T15:59:45.313411+0000","last_became_peered":"2024-06-29T15:59:45.313411+0000","last_unstale":"2024-06-29T15:59:47.318120+0000","last_undegraded":"2024-06-29T15:59:47.318120+0000","last_fullsized":"2024-06-29T15:59:47.318120+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:04:17.697812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432630+0000","last_change":"2024-06-29T15:59:46.311288+0000","last_active":"2024-06-29T15:59:47.432630+0000","last_peered":"2024-06-29T15:59:47.432630+0000","last_clean":"2024-06-29T15:59:47.432630+0000","last_became_active":"2024-06-29T15:59:46.310989+0000","last_became_peered":"2024-06-29T15:59:46.310989+0000","last_unstale":"2024-06-29T15:59:47.432630+0000","last_undegraded":"2024-06-29T15:59:47.432630+0000","last_fullsized":"2024-06-29T15:59:47.432630+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:48.933419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"274'6","reported_seq":18,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.334121+0000","last_change":"2024-06-29T15:59:45.305148+0000","last_active":"2024-06-29T15:59:46.334121+0000","last_peered":"2024-06-29T15:59:46.334121+0000","last_clean":"2024-06-29T15:59:46.334121+0000","last_became_active":"2024-06-29T15:59:45.304685+0000","last_became_peered":"2024-06-29T15:59:45.304685+0000","last_unstale":"2024-06-29T15:59:46.334121+0000","last_undegraded":"2024-06-29T15:59:46.334121+0000","last_fullsized":"2024-06-29T15:59:46.334121+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:21:48.981970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.318148+0000","last_change":"2024-06-29T15:59:46.313918+0000","last_active":"2024-06-29T15:59:46.318148+0000","last_peered":"2024-06-29T15:59:46.318148+0000","last_clean":"2024-06-29T15:59:46.318148+0000","last_became_active":"2024-06-29T15:59:46.313597+0000","last_became_peered":"2024-06-29T15:59:46.313597+0000","last_unstale":"2024-06-29T15:59:46.318148+0000","last_undegraded":"2024-06-29T15:59:46.318148+0000","last_fullsized":"2024-06-29T15:59:46.318148+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:15.673811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"274'2","reported_seq":14,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.332235+0000","last_change":"2024-06-29T15:59:45.317748+0000","last_active":"2024-06-29T15:59:46.332235+0000","last_peered":"2024-06-29T15:59:46.332235+0000","last_clean":"2024-06-29T15:59:46.332235+0000","last_became_active":"2024-06-29T15:59:45.317498+0000","last_became_peered":"2024-06-29T15:59:45.317498+0000","last_unstale":"2024-06-29T15:59:46.332235+0000","last_undegraded":"2024-06-29T15:59:46.332235+0000","last_fullsized":"2024-06-29T15:59:46.332235+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:34.763922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.311680+0000","last_change":"2024-06-29T15:59:46.308975+0000","last_active":"2024-06-29T15:59:46.311680+0000","last_peered":"2024-06-29T15:59:46.311680+0000","last_clean":"2024-06-29T15:59:46.311680+0000","last_became_active":"2024-06-29T15:59:46.308657+0000","last_became_peered":"2024-06-29T15:59:46.308657+0000","last_unstale":"2024-06-29T15:59:46.311680+0000","last_undegraded":"2024-06-29T15:59:46.311680+0000","last_fullsized":"2024-06-29T15:59:46.311680+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:17.793455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.302179+0000","last_change":"2024-06-29T15:59:45.313527+0000","last_active":"2024-06-29T15:59:46.302179+0000","last_peered":"2024-06-29T15:59:46.302179+0000","last_clean":"2024-06-29T15:59:46.302179+0000","last_became_active":"2024-06-29T15:59:45.313207+0000","last_became_peered":"2024-06-29T15:59:45.313207+0000","last_unstale":"2024-06-29T15:59:46.302179+0000","last_undegraded":"2024-06-29T15:59:46.302179+0000","last_fullsized":"2024-06-29T15:59:46.302179+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:54.070849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432190+0000","last_change":"2024-06-29T15:59:46.319677+0000","last_active":"2024-06-29T15:59:47.432190+0000","last_peered":"2024-06-29T15:59:47.432190+0000","last_clean":"2024-06-29T15:59:47.432190+0000","last_became_active":"2024-06-29T15:59:46.319294+0000","last_became_peered":"2024-06-29T15:59:46.319294+0000","last_unstale":"2024-06-29T15:59:47.432190+0000","last_undegraded":"2024-06-29T15:59:47.432190+0000","last_fullsized":"2024-06-29T15:59:47.432190+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:38.519216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.304072+0000","last_change":"2024-06-29T15:59:45.317935+0000","last_active":"2024-06-29T15:59:46.304072+0000","last_peered":"2024-06-29T15:59:46.304072+0000","last_clean":"2024-06-29T15:59:46.304072+0000","last_became_active":"2024-06-29T15:59:45.317600+0000","last_became_peered":"2024-06-29T15:59:45.317600+0000","last_unstale":"2024-06-29T15:59:46.304072+0000","last_undegraded":"2024-06-29T15:59:46.304072+0000","last_fullsized":"2024-06-29T15:59:46.304072+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:33.563934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.321918+0000","last_change":"2024-06-29T15:59:46.319714+0000","last_active":"2024-06-29T15:59:46.321918+0000","last_peered":"2024-06-29T15:59:46.321918+0000","last_clean":"2024-06-29T15:59:46.321918+0000","last_became_active":"2024-06-29T15:59:46.319419+0000","last_became_peered":"2024-06-29T15:59:46.319419+0000","last_unstale":"2024-06-29T15:59:46.321918+0000","last_undegraded":"2024-06-29T15:59:46.321918+0000","last_fullsized":"2024-06-29T15:59:46.321918+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:23:45.939879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"274'8","reported_seq":22,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432124+0000","last_change":"2024-06-29T15:59:45.311349+0000","last_active":"2024-06-29T15:59:47.432124+0000","last_peered":"2024-06-29T15:59:47.432124+0000","last_clean":"2024-06-29T15:59:47.432124+0000","last_became_active":"2024-06-29T15:59:45.311028+0000","last_became_peered":"2024-06-29T15:59:45.311028+0000","last_unstale":"2024-06-29T15:59:47.432124+0000","last_undegraded":"2024-06-29T15:59:47.432124+0000","last_fullsized":"2024-06-29T15:59:47.432124+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:14.377837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.329097+0000","last_change":"2024-06-29T15:59:46.328399+0000","last_active":"2024-06-29T15:59:46.329097+0000","last_peered":"2024-06-29T15:59:46.329097+0000","last_clean":"2024-06-29T15:59:46.329097+0000","last_became_active":"2024-06-29T15:59:46.328091+0000","last_became_peered":"2024-06-29T15:59:46.328091+0000","last_unstale":"2024-06-29T15:59:46.329097+0000","last_undegraded":"2024-06-29T15:59:46.329097+0000","last_fullsized":"2024-06-29T15:59:46.329097+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:20:54.768111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"274'1","reported_seq":15,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432748+0000","last_change":"2024-06-29T15:59:45.309069+0000","last_active":"2024-06-29T15:59:47.432748+0000","last_peered":"2024-06-29T15:59:47.432748+0000","last_clean":"2024-06-29T15:59:47.432748+0000","last_became_active":"2024-06-29T15:59:45.308713+0000","last_became_peered":"2024-06-29T15:59:45.308713+0000","last_unstale":"2024-06-29T15:59:47.432748+0000","last_undegraded":"2024-06-29T15:59:47.432748+0000","last_fullsized":"2024-06-29T15:59:47.432748+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:43:31.558256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.320408+0000","last_change":"2024-06-29T15:59:46.318781+0000","last_active":"2024-06-29T15:59:46.320408+0000","last_peered":"2024-06-29T15:59:46.320408+0000","last_clean":"2024-06-29T15:59:46.320408+0000","last_became_active":"2024-06-29T15:59:46.317288+0000","last_became_peered":"2024-06-29T15:59:46.317288+0000","last_unstale":"2024-06-29T15:59:46.320408+0000","last_undegraded":"2024-06-29T15:59:46.320408+0000","last_fullsized":"2024-06-29T15:59:46.320408+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:03.763576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"274'2","reported_seq":14,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.331027+0000","last_change":"2024-06-29T15:59:45.309424+0000","last_active":"2024-06-29T15:59:46.331027+0000","last_peered":"2024-06-29T15:59:46.331027+0000","last_clean":"2024-06-29T15:59:46.331027+0000","last_became_active":"2024-06-29T15:59:45.309045+0000","last_became_peered":"2024-06-29T15:59:45.309045+0000","last_unstale":"2024-06-29T15:59:46.331027+0000","last_undegraded":"2024-06-29T15:59:46.331027+0000","last_fullsized":"2024-06-29T15:59:46.331027+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:21:22.559018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.328909+0000","last_change":"2024-06-29T15:59:46.324839+0000","last_active":"2024-06-29T15:59:46.328909+0000","last_peered":"2024-06-29T15:59:46.328909+0000","last_clean":"2024-06-29T15:59:46.328909+0000","last_became_active":"2024-06-29T15:59:46.324524+0000","last_became_peered":"2024-06-29T15:59:46.324524+0000","last_unstale":"2024-06-29T15:59:46.328909+0000","last_undegraded":"2024-06-29T15:59:46.328909+0000","last_fullsized":"2024-06-29T15:59:46.328909+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:13.898105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"274'1","reported_seq":13,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.332455+0000","last_change":"2024-06-29T15:59:45.307573+0000","last_active":"2024-06-29T15:59:46.332455+0000","last_peered":"2024-06-29T15:59:46.332455+0000","last_clean":"2024-06-29T15:59:46.332455+0000","last_became_active":"2024-06-29T15:59:45.307252+0000","last_became_peered":"2024-06-29T15:59:45.307252+0000","last_unstale":"2024-06-29T15:59:46.332455+0000","last_undegraded":"2024-06-29T15:59:46.332455+0000","last_fullsized":"2024-06-29T15:59:46.332455+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:53.641375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313689+0000","last_change":"2024-06-29T15:59:46.316644+0000","last_active":"2024-06-29T15:59:47.313689+0000","last_peered":"2024-06-29T15:59:47.313689+0000","last_clean":"2024-06-29T15:59:47.313689+0000","last_became_active":"2024-06-29T15:59:46.316331+0000","last_became_peered":"2024-06-29T15:59:46.316331+0000","last_unstale":"2024-06-29T15:59:47.313689+0000","last_undegraded":"2024-06-29T15:59:47.313689+0000","last_fullsized":"2024-06-29T15:59:47.313689+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:55:34.272705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318122+0000","last_change":"2024-06-29T15:59:45.316585+0000","last_active":"2024-06-29T15:59:47.318122+0000","last_peered":"2024-06-29T15:59:47.318122+0000","last_clean":"2024-06-29T15:59:47.318122+0000","last_became_active":"2024-06-29T15:59:45.316221+0000","last_became_peered":"2024-06-29T15:59:45.316221+0000","last_unstale":"2024-06-29T15:59:47.318122+0000","last_undegraded":"2024-06-29T15:59:47.318122+0000","last_fullsized":"2024-06-29T15:59:47.318122+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:23.257182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432007+0000","last_change":"2024-06-29T15:59:46.326671+0000","last_active":"2024-06-29T15:59:47.432007+0000","last_peered":"2024-06-29T15:59:47.432007+0000","last_clean":"2024-06-29T15:59:47.432007+0000","last_became_active":"2024-06-29T15:59:46.326086+0000","last_became_peered":"2024-06-29T15:59:46.326086+0000","last_unstale":"2024-06-29T15:59:47.432007+0000","last_undegraded":"2024-06-29T15:59:47.432007+0000","last_fullsized":"2024-06-29T15:59:47.432007+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:41.570575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"274'2","reported_seq":14,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.324343+0000","last_change":"2024-06-29T15:59:45.313395+0000","last_active":"2024-06-29T15:59:46.324343+0000","last_peered":"2024-06-29T15:59:46.324343+0000","last_clean":"2024-06-29T15:59:46.324343+0000","last_became_active":"2024-06-29T15:59:45.313051+0000","last_became_peered":"2024-06-29T15:59:45.313051+0000","last_unstale":"2024-06-29T15:59:46.324343+0000","last_undegraded":"2024-06-29T15:59:46.324343+0000","last_fullsized":"2024-06-29T15:59:46.324343+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:33.929049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312750+0000","last_change":"2024-06-29T15:59:46.311815+0000","last_active":"2024-06-29T15:59:46.312750+0000","last_peered":"2024-06-29T15:59:46.312750+0000","last_clean":"2024-06-29T15:59:46.312750+0000","last_became_active":"2024-06-29T15:59:46.311496+0000","last_became_peered":"2024-06-29T15:59:46.311496+0000","last_unstale":"2024-06-29T15:59:46.312750+0000","last_undegraded":"2024-06-29T15:59:46.312750+0000","last_fullsized":"2024-06-29T15:59:46.312750+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:56:09.069111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432343+0000","last_change":"2024-06-29T15:59:45.313262+0000","last_active":"2024-06-29T15:59:47.432343+0000","last_peered":"2024-06-29T15:59:47.432343+0000","last_clean":"2024-06-29T15:59:47.432343+0000","last_became_active":"2024-06-29T15:59:45.312937+0000","last_became_peered":"2024-06-29T15:59:45.312937+0000","last_unstale":"2024-06-29T15:59:47.432343+0000","last_undegraded":"2024-06-29T15:59:47.432343+0000","last_fullsized":"2024-06-29T15:59:47.432343+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:50.986287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312556+0000","last_change":"2024-06-29T15:59:46.309967+0000","last_active":"2024-06-29T15:59:46.312556+0000","last_peered":"2024-06-29T15:59:46.312556+0000","last_clean":"2024-06-29T15:59:46.312556+0000","last_became_active":"2024-06-29T15:59:46.309690+0000","last_became_peered":"2024-06-29T15:59:46.309690+0000","last_unstale":"2024-06-29T15:59:46.312556+0000","last_undegraded":"2024-06-29T15:59:46.312556+0000","last_fullsized":"2024-06-29T15:59:46.312556+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:05:14.020915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313207+0000","last_change":"2024-06-29T15:59:45.307624+0000","last_active":"2024-06-29T15:59:47.313207+0000","last_peered":"2024-06-29T15:59:47.313207+0000","last_clean":"2024-06-29T15:59:47.313207+0000","last_became_active":"2024-06-29T15:59:45.307287+0000","last_became_peered":"2024-06-29T15:59:45.307287+0000","last_unstale":"2024-06-29T15:59:47.313207+0000","last_undegraded":"2024-06-29T15:59:47.313207+0000","last_fullsized":"2024-06-29T15:59:47.313207+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:49:12.238348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.314413+0000","last_change":"2024-06-29T15:59:46.310006+0000","last_active":"2024-06-29T15:59:46.314413+0000","last_peered":"2024-06-29T15:59:46.314413+0000","last_clean":"2024-06-29T15:59:46.314413+0000","last_became_active":"2024-06-29T15:59:46.309688+0000","last_became_peered":"2024-06-29T15:59:46.309688+0000","last_unstale":"2024-06-29T15:59:46.314413+0000","last_undegraded":"2024-06-29T15:59:46.314413+0000","last_fullsized":"2024-06-29T15:59:46.314413+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:06.866195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313062+0000","last_change":"2024-06-29T15:59:45.315644+0000","last_active":"2024-06-29T15:59:47.313062+0000","last_peered":"2024-06-29T15:59:47.313062+0000","last_clean":"2024-06-29T15:59:47.313062+0000","last_became_active":"2024-06-29T15:59:45.315324+0000","last_became_peered":"2024-06-29T15:59:45.315324+0000","last_unstale":"2024-06-29T15:59:47.313062+0000","last_undegraded":"2024-06-29T15:59:47.313062+0000","last_fullsized":"2024-06-29T15:59:47.313062+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:08:34.700293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432605+0000","last_change":"2024-06-29T15:59:46.326497+0000","last_active":"2024-06-29T15:59:47.432605+0000","last_peered":"2024-06-29T15:59:47.432605+0000","last_clean":"2024-06-29T15:59:47.432605+0000","last_became_active":"2024-06-29T15:59:46.326121+0000","last_became_peered":"2024-06-29T15:59:46.326121+0000","last_unstale":"2024-06-29T15:59:47.432605+0000","last_undegraded":"2024-06-29T15:59:47.432605+0000","last_fullsized":"2024-06-29T15:59:47.432605+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:29.818673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"274'2","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318337+0000","last_change":"2024-06-29T15:59:45.309596+0000","last_active":"2024-06-29T15:59:47.318337+0000","last_peered":"2024-06-29T15:59:47.318337+0000","last_clean":"2024-06-29T15:59:47.318337+0000","last_became_active":"2024-06-29T15:59:45.309292+0000","last_became_peered":"2024-06-29T15:59:45.309292+0000","last_unstale":"2024-06-29T15:59:47.318337+0000","last_undegraded":"2024-06-29T15:59:47.318337+0000","last_fullsized":"2024-06-29T15:59:47.318337+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:02:27.053946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318470+0000","last_change":"2024-06-29T15:59:46.316443+0000","last_active":"2024-06-29T15:59:47.318470+0000","last_peered":"2024-06-29T15:59:47.318470+0000","last_clean":"2024-06-29T15:59:47.318470+0000","last_became_active":"2024-06-29T15:59:46.316115+0000","last_became_peered":"2024-06-29T15:59:46.316115+0000","last_unstale":"2024-06-29T15:59:47.318470+0000","last_undegraded":"2024-06-29T15:59:47.318470+0000","last_fullsized":"2024-06-29T15:59:47.318470+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:25.056067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"274'1","reported_seq":13,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.310447+0000","last_change":"2024-06-29T15:59:45.312684+0000","last_active":"2024-06-29T15:59:47.310447+0000","last_peered":"2024-06-29T15:59:47.310447+0000","last_clean":"2024-06-29T15:59:47.310447+0000","last_became_active":"2024-06-29T15:59:45.311065+0000","last_became_peered":"2024-06-29T15:59:45.311065+0000","last_unstale":"2024-06-29T15:59:47.310447+0000","last_undegraded":"2024-06-29T15:59:47.310447+0000","last_fullsized":"2024-06-29T15:59:47.310447+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:29.929013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318908+0000","last_change":"2024-06-29T15:59:46.328593+0000","last_active":"2024-06-29T15:59:47.318908+0000","last_peered":"2024-06-29T15:59:47.318908+0000","last_clean":"2024-06-29T15:59:47.318908+0000","last_became_active":"2024-06-29T15:59:46.326388+0000","last_became_peered":"2024-06-29T15:59:46.326388+0000","last_unstale":"2024-06-29T15:59:47.318908+0000","last_undegraded":"2024-06-29T15:59:47.318908+0000","last_fullsized":"2024-06-29T15:59:47.318908+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:11.725084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.306924+0000","last_change":"2024-06-29T15:59:45.312258+0000","last_active":"2024-06-29T15:59:46.306924+0000","last_peered":"2024-06-29T15:59:46.306924+0000","last_clean":"2024-06-29T15:59:46.306924+0000","last_became_active":"2024-06-29T15:59:45.311896+0000","last_became_peered":"2024-06-29T15:59:45.311896+0000","last_unstale":"2024-06-29T15:59:46.306924+0000","last_undegraded":"2024-06-29T15:59:46.306924+0000","last_fullsized":"2024-06-29T15:59:46.306924+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:06.924451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.318018+0000","last_change":"2024-06-29T15:59:46.312607+0000","last_active":"2024-06-29T15:59:46.318018+0000","last_peered":"2024-06-29T15:59:46.318018+0000","last_clean":"2024-06-29T15:59:46.318018+0000","last_became_active":"2024-06-29T15:59:46.312265+0000","last_became_peered":"2024-06-29T15:59:46.312265+0000","last_unstale":"2024-06-29T15:59:46.318018+0000","last_undegraded":"2024-06-29T15:59:46.318018+0000","last_fullsized":"2024-06-29T15:59:46.318018+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:38.688479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.299497+0000","last_change":"2024-06-29T15:59:45.307624+0000","last_active":"2024-06-29T15:59:46.299497+0000","last_peered":"2024-06-29T15:59:46.299497+0000","last_clean":"2024-06-29T15:59:46.299497+0000","last_became_active":"2024-06-29T15:59:45.307318+0000","last_became_peered":"2024-06-29T15:59:45.307318+0000","last_unstale":"2024-06-29T15:59:46.299497+0000","last_undegraded":"2024-06-29T15:59:46.299497+0000","last_fullsized":"2024-06-29T15:59:46.299497+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:50.581353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.318140+0000","last_change":"2024-06-29T15:59:46.316857+0000","last_active":"2024-06-29T15:59:46.318140+0000","last_peered":"2024-06-29T15:59:46.318140+0000","last_clean":"2024-06-29T15:59:46.318140+0000","last_became_active":"2024-06-29T15:59:46.316522+0000","last_became_peered":"2024-06-29T15:59:46.316522+0000","last_unstale":"2024-06-29T15:59:46.318140+0000","last_undegraded":"2024-06-29T15:59:46.318140+0000","last_fullsized":"2024-06-29T15:59:46.318140+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:21:49.989091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.301490+0000","last_change":"2024-06-29T15:59:45.314423+0000","last_active":"2024-06-29T15:59:46.301490+0000","last_peered":"2024-06-29T15:59:46.301490+0000","last_clean":"2024-06-29T15:59:46.301490+0000","last_became_active":"2024-06-29T15:59:45.314090+0000","last_became_peered":"2024-06-29T15:59:45.314090+0000","last_unstale":"2024-06-29T15:59:46.301490+0000","last_undegraded":"2024-06-29T15:59:46.301490+0000","last_fullsized":"2024-06-29T15:59:46.301490+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:42.531353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317766+0000","last_change":"2024-06-29T15:59:46.321869+0000","last_active":"2024-06-29T15:59:47.317766+0000","last_peered":"2024-06-29T15:59:47.317766+0000","last_clean":"2024-06-29T15:59:47.317766+0000","last_became_active":"2024-06-29T15:59:46.321530+0000","last_became_peered":"2024-06-29T15:59:46.321530+0000","last_unstale":"2024-06-29T15:59:47.317766+0000","last_undegraded":"2024-06-29T15:59:47.317766+0000","last_fullsized":"2024-06-29T15:59:47.317766+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:21.101877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317958+0000","last_change":"2024-06-29T15:59:45.315433+0000","last_active":"2024-06-29T15:59:47.317958+0000","last_peered":"2024-06-29T15:59:47.317958+0000","last_clean":"2024-06-29T15:59:47.317958+0000","last_became_active":"2024-06-29T15:59:45.315099+0000","last_became_peered":"2024-06-29T15:59:45.315099+0000","last_unstale":"2024-06-29T15:59:47.317958+0000","last_undegraded":"2024-06-29T15:59:47.317958+0000","last_fullsized":"2024-06-29T15:59:47.317958+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:20.659828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.317962+0000","last_change":"2024-06-29T15:59:46.312511+0000","last_active":"2024-06-29T15:59:46.317962+0000","last_peered":"2024-06-29T15:59:46.317962+0000","last_clean":"2024-06-29T15:59:46.317962+0000","last_became_active":"2024-06-29T15:59:46.312110+0000","last_became_peered":"2024-06-29T15:59:46.312110+0000","last_unstale":"2024-06-29T15:59:46.317962+0000","last_undegraded":"2024-06-29T15:59:46.317962+0000","last_fullsized":"2024-06-29T15:59:46.317962+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:54.659040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.304439+0000","last_change":"2024-06-29T15:59:45.314280+0000","last_active":"2024-06-29T15:59:46.304439+0000","last_peered":"2024-06-29T15:59:46.304439+0000","last_clean":"2024-06-29T15:59:46.304439+0000","last_became_active":"2024-06-29T15:59:45.313968+0000","last_became_peered":"2024-06-29T15:59:45.313968+0000","last_unstale":"2024-06-29T15:59:46.304439+0000","last_undegraded":"2024-06-29T15:59:46.304439+0000","last_fullsized":"2024-06-29T15:59:46.304439+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:49.903052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318857+0000","last_change":"2024-06-29T15:59:46.305937+0000","last_active":"2024-06-29T15:59:47.318857+0000","last_peered":"2024-06-29T15:59:47.318857+0000","last_clean":"2024-06-29T15:59:47.318857+0000","last_became_active":"2024-06-29T15:59:46.305607+0000","last_became_peered":"2024-06-29T15:59:46.305607+0000","last_unstale":"2024-06-29T15:59:47.318857+0000","last_undegraded":"2024-06-29T15:59:47.318857+0000","last_fullsized":"2024-06-29T15:59:47.318857+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:58:02.894952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313182+0000","last_change":"2024-06-29T15:59:45.313038+0000","last_active":"2024-06-29T15:59:47.313182+0000","last_peered":"2024-06-29T15:59:47.313182+0000","last_clean":"2024-06-29T15:59:47.313182+0000","last_became_active":"2024-06-29T15:59:45.312694+0000","last_became_peered":"2024-06-29T15:59:45.312694+0000","last_unstale":"2024-06-29T15:59:47.313182+0000","last_undegraded":"2024-06-29T15:59:47.313182+0000","last_fullsized":"2024-06-29T15:59:47.313182+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:44.236109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.323325+0000","last_change":"2024-06-29T15:59:46.320077+0000","last_active":"2024-06-29T15:59:46.323325+0000","last_peered":"2024-06-29T15:59:46.323325+0000","last_clean":"2024-06-29T15:59:46.323325+0000","last_became_active":"2024-06-29T15:59:46.319184+0000","last_became_peered":"2024-06-29T15:59:46.319184+0000","last_unstale":"2024-06-29T15:59:46.323325+0000","last_undegraded":"2024-06-29T15:59:46.323325+0000","last_fullsized":"2024-06-29T15:59:46.323325+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:10.746800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"264'273","reported_seq":1345,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432093+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:59:47.432093+0000","last_peered":"2024-06-29T15:59:47.432093+0000","last_clean":"2024-06-29T15:59:47.432093+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:59:47.432093+0000","last_undegraded":"2024-06-29T15:59:47.432093+0000","last_fullsized":"2024-06-29T15:59:47.432093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1788,"num_read_kb":5144,"num_write":445,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1788,"num_read_kb":5144,"num_write":445,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543313,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53212,"kb_used_data":3976,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318628,"statfs":{"total":96636764160,"available":96582275072,"internally_reserved":0,"allocated":4071424,"data_stored":2394036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543314,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48472,"kb_used_data":3400,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3481600,"data_stored":1804250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543313,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70660,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301180,"statfs":{"total":96636764160,"available":96564408320,"internally_reserved":0,"allocated":3465216,"data_stored":1796932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543312,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52632,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319208,"statfs":{"total":96636764160,"available":96582868992,"internally_reserved":0,"allocated":3485696,"data_stored":1799252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510610,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":3465216,"data_stored":1796932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52624,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319216,"statfs":{"total":96636764160,"available":96582877184,"internally_reserved":0,"allocated":3477504,"data_stored":1798580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510609,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":3380,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":3461120,"data_stored":1796898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510611,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49032,"kb_used_data":3964,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322808,"statfs":{"total":96636764160,"available":96586555392,"internally_reserved":0,"allocated":4059136,"data_stored":2387356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:59:50.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.539+0000 7f16ee6e3640 1 -- 172.21.15.59:0/999076577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 msgr2=0x7f16e80b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.539+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/999076577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 0x7f16e80b3990 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f16e808c4f0 tx=0x7f16d802a560 comp rx=0 tx=0).stop 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.539+0000 7f16ee6e3640 1 -- 172.21.15.59:0/999076577 shutdown_connections 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.539+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/999076577 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16e815ee10 0x7f16e815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.539+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/999076577 >> v2:172.21.15.153:3300/0 conn(0x7f16e80b3ed0 0x7f16e815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.539+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/999076577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 0x7f16e80b3990 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.539+0000 7f16ee6e3640 1 -- 172.21.15.59:0/999076577 >> 172.21.15.59:0/999076577 conn(0x7f16e80d3ee0 msgr2=0x7f16e80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.540+0000 7f16ee6e3640 1 -- 172.21.15.59:0/999076577 shutdown_connections 2024-06-29T15:59:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.540+0000 7f16ee6e3640 1 -- 172.21.15.59:0/999076577 wait complete. 2024-06-29T15:59:50.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.541+0000 7f16ee6e3640 1 Processor -- start 2024-06-29T15:59:50.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.542+0000 7f16ee6e3640 1 -- start start 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16ee6e3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16e80b3ed0 0x7f16e814f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16ee6e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 0x7f16e81e7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16ee6e3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16e815ee10 0x7f16e81552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16ee6e3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f16e8160070 con 0x7f16e80d23c0 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16ee6e3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f16e815fef0 con 0x7f16e80b3ed0 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16ee6e3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f16e81601f0 con 0x7f16e815ee10 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 0x7f16e81e7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 0x7f16e81e7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32968/0 (socket says 172.21.15.59:32968) 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 -- 172.21.15.59:0/752446665 learned_addr learned my addr 172.21.15.59:0/752446665 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16ecc80640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16e815ee10 0x7f16e81552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e7fff640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16e80b3ed0 0x7f16e814f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 -- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16e815ee10 msgr2=0x7f16e81552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16e815ee10 0x7f16e81552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 -- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16e80b3ed0 msgr2=0x7f16e814f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16e80b3ed0 0x7f16e814f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 -- 172.21.15.59:0/752446665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e8152140 con 0x7f16e80d23c0 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.543+0000 7f16e77fe640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 0x7f16e81e7790 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00e710 tx=0x7f16dc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.544+0000 7f16e4ff9640 1 -- 172.21.15.59:0/752446665 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc00cab0 con 0x7f16e80d23c0 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.544+0000 7f16e4ff9640 1 -- 172.21.15.59:0/752446665 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16dc00cc50 con 0x7f16e80d23c0 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.544+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e8152420 con 0x7f16e80d23c0 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.544+0000 7f16e4ff9640 1 -- 172.21.15.59:0/752446665 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc005020 con 0x7f16e80d23c0 2024-06-29T15:59:50.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.544+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e81529e0 con 0x7f16e80d23c0 2024-06-29T15:59:50.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.545+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16e80b3990 con 0x7f16e80d23c0 2024-06-29T15:59:50.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.545+0000 7f16e4ff9640 1 -- 172.21.15.59:0/752446665 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f16dc0053d0 con 0x7f16e80d23c0 2024-06-29T15:59:50.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.546+0000 7f16e4ff9640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f16d4076420 0x7f16d4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.546+0000 7f16e4ff9640 1 -- 172.21.15.59:0/752446665 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f16dc098620 con 0x7f16e80d23c0 2024-06-29T15:59:50.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.546+0000 7f16e7fff640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f16d4076420 0x7f16d4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.547+0000 7f16e7fff640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f16d4076420 0x7f16d4078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16d8033000 tx=0x7f16d8004730 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:50.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.551+0000 7f16e4ff9640 1 -- 172.21.15.59:0/752446665 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f16dc061b00 con 0x7f16e80d23c0 2024-06-29T15:59:50.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.739+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 --> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f16e8151ed0 con 0x7f16d4076420 2024-06-29T15:59:50.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.744+0000 7f16e4ff9640 1 -- 172.21.15.59:0/752446665 <== mgr.12382 v2:172.21.15.153:6832/3072370296 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252465 (secure 0 0 0) 0x7f16e8151ed0 con 0x7f16d4076420 2024-06-29T15:59:50.746 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T15:59:50.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f16d4076420 msgr2=0x7f16d4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f16d4076420 0x7f16d4078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16d8033000 tx=0x7f16d8004730 comp rx=0 tx=0).stop 2024-06-29T15:59:50.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 msgr2=0x7f16e81e7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 0x7f16e81e7790 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00e710 tx=0x7f16dc00ebd0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 shutdown_connections 2024-06-29T15:59:50.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f16d4076420 0x7f16d4078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16e815ee10 0x7f16e81552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16e80d23c0 0x7f16e81e7790 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 --2- 172.21.15.59:0/752446665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16e80b3ed0 0x7f16e814f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 >> 172.21.15.59:0/752446665 conn(0x7f16e80d3ee0 msgr2=0x7f16e80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 shutdown_connections 2024-06-29T15:59:50.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.747+0000 7f16ee6e3640 1 -- 172.21.15.59:0/752446665 wait complete. 2024-06-29T15:59:50.758 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:59:49.863479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1788,"num_read_kb":5144,"num_write":490,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431856,"kb_used_data":28288,"kb_used_omap":12,"kb_used_meta":403443,"kb_avail":754542864,"statfs":{"total":773094113280,"available":772651892736,"internally_reserved":0,"allocated":28966912,"data_stored":15574236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":413126194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.573656"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.306670+0000","last_change":"2024-06-29T15:59:46.305324+0000","last_active":"2024-06-29T15:59:46.306670+0000","last_peered":"2024-06-29T15:59:46.306670+0000","last_clean":"2024-06-29T15:59:46.306670+0000","last_became_active":"2024-06-29T15:59:46.304977+0000","last_became_peered":"2024-06-29T15:59:46.304977+0000","last_unstale":"2024-06-29T15:59:46.306670+0000","last_undegraded":"2024-06-29T15:59:46.306670+0000","last_fullsized":"2024-06-29T15:59:46.306670+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:03.763576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318452+0000","last_change":"2024-06-29T15:59:46.329066+0000","last_active":"2024-06-29T15:59:47.318452+0000","last_peered":"2024-06-29T15:59:47.318452+0000","last_clean":"2024-06-29T15:59:47.318452+0000","last_became_active":"2024-06-29T15:59:46.327443+0000","last_became_peered":"2024-06-29T15:59:46.327443+0000","last_unstale":"2024-06-29T15:59:47.318452+0000","last_undegraded":"2024-06-29T15:59:47.318452+0000","last_fullsized":"2024-06-29T15:59:47.318452+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:30:48.394838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313354+0000","last_change":"2024-06-29T15:59:46.310384+0000","last_active":"2024-06-29T15:59:47.313354+0000","last_peered":"2024-06-29T15:59:47.313354+0000","last_clean":"2024-06-29T15:59:47.313354+0000","last_became_active":"2024-06-29T15:59:46.310034+0000","last_became_peered":"2024-06-29T15:59:46.310034+0000","last_unstale":"2024-06-29T15:59:47.313354+0000","last_undegraded":"2024-06-29T15:59:47.313354+0000","last_fullsized":"2024-06-29T15:59:47.313354+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:08.617490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432183+0000","last_change":"2024-06-29T15:59:46.328214+0000","last_active":"2024-06-29T15:59:47.432183+0000","last_peered":"2024-06-29T15:59:47.432183+0000","last_clean":"2024-06-29T15:59:47.432183+0000","last_became_active":"2024-06-29T15:59:46.327587+0000","last_became_peered":"2024-06-29T15:59:46.327587+0000","last_unstale":"2024-06-29T15:59:47.432183+0000","last_undegraded":"2024-06-29T15:59:47.432183+0000","last_fullsized":"2024-06-29T15:59:47.432183+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:04:36.707743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312612+0000","last_change":"2024-06-29T15:59:46.311208+0000","last_active":"2024-06-29T15:59:46.312612+0000","last_peered":"2024-06-29T15:59:46.312612+0000","last_clean":"2024-06-29T15:59:46.312612+0000","last_became_active":"2024-06-29T15:59:46.310665+0000","last_became_peered":"2024-06-29T15:59:46.310665+0000","last_unstale":"2024-06-29T15:59:46.312612+0000","last_undegraded":"2024-06-29T15:59:46.312612+0000","last_fullsized":"2024-06-29T15:59:46.312612+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:43.538474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.321210+0000","last_change":"2024-06-29T15:59:46.317909+0000","last_active":"2024-06-29T15:59:46.321210+0000","last_peered":"2024-06-29T15:59:46.321210+0000","last_clean":"2024-06-29T15:59:46.321210+0000","last_became_active":"2024-06-29T15:59:46.317563+0000","last_became_peered":"2024-06-29T15:59:46.317563+0000","last_unstale":"2024-06-29T15:59:46.321210+0000","last_undegraded":"2024-06-29T15:59:46.321210+0000","last_fullsized":"2024-06-29T15:59:46.321210+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:45:59.347478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318457+0000","last_change":"2024-06-29T15:59:46.303400+0000","last_active":"2024-06-29T15:59:47.318457+0000","last_peered":"2024-06-29T15:59:47.318457+0000","last_clean":"2024-06-29T15:59:47.318457+0000","last_became_active":"2024-06-29T15:59:46.303091+0000","last_became_peered":"2024-06-29T15:59:46.303091+0000","last_unstale":"2024-06-29T15:59:47.318457+0000","last_undegraded":"2024-06-29T15:59:47.318457+0000","last_fullsized":"2024-06-29T15:59:47.318457+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:06.680412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312890+0000","last_change":"2024-06-29T15:59:46.311575+0000","last_active":"2024-06-29T15:59:46.312890+0000","last_peered":"2024-06-29T15:59:46.312890+0000","last_clean":"2024-06-29T15:59:46.312890+0000","last_became_active":"2024-06-29T15:59:46.311223+0000","last_became_peered":"2024-06-29T15:59:46.311223+0000","last_unstale":"2024-06-29T15:59:46.312890+0000","last_undegraded":"2024-06-29T15:59:46.312890+0000","last_fullsized":"2024-06-29T15:59:46.312890+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:30:15.280262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312336+0000","last_change":"2024-06-29T15:59:46.307608+0000","last_active":"2024-06-29T15:59:46.312336+0000","last_peered":"2024-06-29T15:59:46.312336+0000","last_clean":"2024-06-29T15:59:46.312336+0000","last_became_active":"2024-06-29T15:59:46.307292+0000","last_became_peered":"2024-06-29T15:59:46.307292+0000","last_unstale":"2024-06-29T15:59:46.312336+0000","last_undegraded":"2024-06-29T15:59:46.312336+0000","last_fullsized":"2024-06-29T15:59:46.312336+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:24.226917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.314807+0000","last_change":"2024-06-29T15:59:46.313902+0000","last_active":"2024-06-29T15:59:46.314807+0000","last_peered":"2024-06-29T15:59:46.314807+0000","last_clean":"2024-06-29T15:59:46.314807+0000","last_became_active":"2024-06-29T15:59:46.313568+0000","last_became_peered":"2024-06-29T15:59:46.313568+0000","last_unstale":"2024-06-29T15:59:46.314807+0000","last_undegraded":"2024-06-29T15:59:46.314807+0000","last_fullsized":"2024-06-29T15:59:46.314807+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:32.104069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.311585+0000","last_change":"2024-06-29T15:59:46.309964+0000","last_active":"2024-06-29T15:59:46.311585+0000","last_peered":"2024-06-29T15:59:46.311585+0000","last_clean":"2024-06-29T15:59:46.311585+0000","last_became_active":"2024-06-29T15:59:46.309603+0000","last_became_peered":"2024-06-29T15:59:46.309603+0000","last_unstale":"2024-06-29T15:59:46.311585+0000","last_undegraded":"2024-06-29T15:59:46.311585+0000","last_fullsized":"2024-06-29T15:59:46.311585+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:28.221179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.321243+0000","last_change":"2024-06-29T15:59:46.319544+0000","last_active":"2024-06-29T15:59:46.321243+0000","last_peered":"2024-06-29T15:59:46.321243+0000","last_clean":"2024-06-29T15:59:46.321243+0000","last_became_active":"2024-06-29T15:59:46.319240+0000","last_became_peered":"2024-06-29T15:59:46.319240+0000","last_unstale":"2024-06-29T15:59:46.321243+0000","last_undegraded":"2024-06-29T15:59:46.321243+0000","last_fullsized":"2024-06-29T15:59:46.321243+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:30.681706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312590+0000","last_change":"2024-06-29T15:59:46.310949+0000","last_active":"2024-06-29T15:59:46.312590+0000","last_peered":"2024-06-29T15:59:46.312590+0000","last_clean":"2024-06-29T15:59:46.312590+0000","last_became_active":"2024-06-29T15:59:46.310312+0000","last_became_peered":"2024-06-29T15:59:46.310312+0000","last_unstale":"2024-06-29T15:59:46.312590+0000","last_undegraded":"2024-06-29T15:59:46.312590+0000","last_fullsized":"2024-06-29T15:59:46.312590+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:21:23.566139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317831+0000","last_change":"2024-06-29T15:59:46.327840+0000","last_active":"2024-06-29T15:59:47.317831+0000","last_peered":"2024-06-29T15:59:47.317831+0000","last_clean":"2024-06-29T15:59:47.317831+0000","last_became_active":"2024-06-29T15:59:46.327540+0000","last_became_peered":"2024-06-29T15:59:46.327540+0000","last_unstale":"2024-06-29T15:59:47.317831+0000","last_undegraded":"2024-06-29T15:59:47.317831+0000","last_fullsized":"2024-06-29T15:59:47.317831+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:25:19.976178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317856+0000","last_change":"2024-06-29T15:59:46.316808+0000","last_active":"2024-06-29T15:59:47.317856+0000","last_peered":"2024-06-29T15:59:47.317856+0000","last_clean":"2024-06-29T15:59:47.317856+0000","last_became_active":"2024-06-29T15:59:46.316475+0000","last_became_peered":"2024-06-29T15:59:46.316475+0000","last_unstale":"2024-06-29T15:59:47.317856+0000","last_undegraded":"2024-06-29T15:59:47.317856+0000","last_fullsized":"2024-06-29T15:59:47.317856+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:21.057883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.316780+0000","last_change":"2024-06-29T15:59:46.315370+0000","last_active":"2024-06-29T15:59:46.316780+0000","last_peered":"2024-06-29T15:59:46.316780+0000","last_clean":"2024-06-29T15:59:46.316780+0000","last_became_active":"2024-06-29T15:59:46.315064+0000","last_became_peered":"2024-06-29T15:59:46.315064+0000","last_unstale":"2024-06-29T15:59:46.316780+0000","last_undegraded":"2024-06-29T15:59:46.316780+0000","last_fullsized":"2024-06-29T15:59:46.316780+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:19.185984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312299+0000","last_change":"2024-06-29T15:59:46.306641+0000","last_active":"2024-06-29T15:59:46.312299+0000","last_peered":"2024-06-29T15:59:46.312299+0000","last_clean":"2024-06-29T15:59:46.312299+0000","last_became_active":"2024-06-29T15:59:46.306302+0000","last_became_peered":"2024-06-29T15:59:46.306302+0000","last_unstale":"2024-06-29T15:59:46.312299+0000","last_undegraded":"2024-06-29T15:59:46.312299+0000","last_fullsized":"2024-06-29T15:59:46.312299+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:06.706794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432500+0000","last_change":"2024-06-29T15:59:46.318108+0000","last_active":"2024-06-29T15:59:47.432500+0000","last_peered":"2024-06-29T15:59:47.432500+0000","last_clean":"2024-06-29T15:59:47.432500+0000","last_became_active":"2024-06-29T15:59:46.316041+0000","last_became_peered":"2024-06-29T15:59:46.316041+0000","last_unstale":"2024-06-29T15:59:47.432500+0000","last_undegraded":"2024-06-29T15:59:47.432500+0000","last_fullsized":"2024-06-29T15:59:47.432500+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:43.209978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313404+0000","last_change":"2024-06-29T15:59:46.308533+0000","last_active":"2024-06-29T15:59:47.313404+0000","last_peered":"2024-06-29T15:59:47.313404+0000","last_clean":"2024-06-29T15:59:47.313404+0000","last_became_active":"2024-06-29T15:59:46.308170+0000","last_became_peered":"2024-06-29T15:59:46.308170+0000","last_unstale":"2024-06-29T15:59:47.313404+0000","last_undegraded":"2024-06-29T15:59:47.313404+0000","last_fullsized":"2024-06-29T15:59:47.313404+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:54:03.911477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432198+0000","last_change":"2024-06-29T15:59:46.320836+0000","last_active":"2024-06-29T15:59:47.432198+0000","last_peered":"2024-06-29T15:59:47.432198+0000","last_clean":"2024-06-29T15:59:47.432198+0000","last_became_active":"2024-06-29T15:59:46.320363+0000","last_became_peered":"2024-06-29T15:59:46.320363+0000","last_unstale":"2024-06-29T15:59:47.432198+0000","last_undegraded":"2024-06-29T15:59:47.432198+0000","last_fullsized":"2024-06-29T15:59:47.432198+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:55:16.894553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317956+0000","last_change":"2024-06-29T15:59:46.326783+0000","last_active":"2024-06-29T15:59:47.317956+0000","last_peered":"2024-06-29T15:59:47.317956+0000","last_clean":"2024-06-29T15:59:47.317956+0000","last_became_active":"2024-06-29T15:59:46.325921+0000","last_became_peered":"2024-06-29T15:59:46.325921+0000","last_unstale":"2024-06-29T15:59:47.317956+0000","last_undegraded":"2024-06-29T15:59:47.317956+0000","last_fullsized":"2024-06-29T15:59:47.317956+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:56.734569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.431950+0000","last_change":"2024-06-29T15:59:46.327179+0000","last_active":"2024-06-29T15:59:47.431950+0000","last_peered":"2024-06-29T15:59:47.431950+0000","last_clean":"2024-06-29T15:59:47.431950+0000","last_became_active":"2024-06-29T15:59:46.326874+0000","last_became_peered":"2024-06-29T15:59:46.326874+0000","last_unstale":"2024-06-29T15:59:47.431950+0000","last_undegraded":"2024-06-29T15:59:47.431950+0000","last_fullsized":"2024-06-29T15:59:47.431950+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:59:51.387874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.431970+0000","last_change":"2024-06-29T15:59:46.320203+0000","last_active":"2024-06-29T15:59:47.431970+0000","last_peered":"2024-06-29T15:59:47.431970+0000","last_clean":"2024-06-29T15:59:47.431970+0000","last_became_active":"2024-06-29T15:59:46.317266+0000","last_became_peered":"2024-06-29T15:59:46.317266+0000","last_unstale":"2024-06-29T15:59:47.431970+0000","last_undegraded":"2024-06-29T15:59:47.431970+0000","last_fullsized":"2024-06-29T15:59:47.431970+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:17:57.910816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313132+0000","last_change":"2024-06-29T15:59:46.316574+0000","last_active":"2024-06-29T15:59:47.313132+0000","last_peered":"2024-06-29T15:59:47.313132+0000","last_clean":"2024-06-29T15:59:47.313132+0000","last_became_active":"2024-06-29T15:59:46.316263+0000","last_became_peered":"2024-06-29T15:59:46.316263+0000","last_unstale":"2024-06-29T15:59:47.313132+0000","last_undegraded":"2024-06-29T15:59:47.313132+0000","last_fullsized":"2024-06-29T15:59:47.313132+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:18.163385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.326898+0000","last_change":"2024-06-29T15:59:46.325206+0000","last_active":"2024-06-29T15:59:46.326898+0000","last_peered":"2024-06-29T15:59:46.326898+0000","last_clean":"2024-06-29T15:59:46.326898+0000","last_became_active":"2024-06-29T15:59:46.324853+0000","last_became_peered":"2024-06-29T15:59:46.324853+0000","last_unstale":"2024-06-29T15:59:46.326898+0000","last_undegraded":"2024-06-29T15:59:46.326898+0000","last_fullsized":"2024-06-29T15:59:46.326898+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:54.514611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432425+0000","last_change":"2024-06-29T15:59:46.318868+0000","last_active":"2024-06-29T15:59:47.432425+0000","last_peered":"2024-06-29T15:59:47.432425+0000","last_clean":"2024-06-29T15:59:47.432425+0000","last_became_active":"2024-06-29T15:59:46.318528+0000","last_became_peered":"2024-06-29T15:59:46.318528+0000","last_unstale":"2024-06-29T15:59:47.432425+0000","last_undegraded":"2024-06-29T15:59:47.432425+0000","last_fullsized":"2024-06-29T15:59:47.432425+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:44.676408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.309656+0000","last_change":"2024-06-29T15:59:46.307500+0000","last_active":"2024-06-29T15:59:46.309656+0000","last_peered":"2024-06-29T15:59:46.309656+0000","last_clean":"2024-06-29T15:59:46.309656+0000","last_became_active":"2024-06-29T15:59:46.307170+0000","last_became_peered":"2024-06-29T15:59:46.307170+0000","last_unstale":"2024-06-29T15:59:46.309656+0000","last_undegraded":"2024-06-29T15:59:46.309656+0000","last_fullsized":"2024-06-29T15:59:46.309656+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:25.113545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318762+0000","last_change":"2024-06-29T15:59:46.310804+0000","last_active":"2024-06-29T15:59:47.318762+0000","last_peered":"2024-06-29T15:59:47.318762+0000","last_clean":"2024-06-29T15:59:47.318762+0000","last_became_active":"2024-06-29T15:59:46.310482+0000","last_became_peered":"2024-06-29T15:59:46.310482+0000","last_unstale":"2024-06-29T15:59:47.318762+0000","last_undegraded":"2024-06-29T15:59:47.318762+0000","last_fullsized":"2024-06-29T15:59:47.318762+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:31:20.869749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317785+0000","last_change":"2024-06-29T15:59:46.326163+0000","last_active":"2024-06-29T15:59:47.317785+0000","last_peered":"2024-06-29T15:59:47.317785+0000","last_clean":"2024-06-29T15:59:47.317785+0000","last_became_active":"2024-06-29T15:59:46.325843+0000","last_became_peered":"2024-06-29T15:59:46.325843+0000","last_unstale":"2024-06-29T15:59:47.317785+0000","last_undegraded":"2024-06-29T15:59:47.317785+0000","last_fullsized":"2024-06-29T15:59:47.317785+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:13.898105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318424+0000","last_change":"2024-06-29T15:59:46.307124+0000","last_active":"2024-06-29T15:59:47.318424+0000","last_peered":"2024-06-29T15:59:47.318424+0000","last_clean":"2024-06-29T15:59:47.318424+0000","last_became_active":"2024-06-29T15:59:46.306809+0000","last_became_peered":"2024-06-29T15:59:46.306809+0000","last_unstale":"2024-06-29T15:59:47.318424+0000","last_undegraded":"2024-06-29T15:59:47.318424+0000","last_fullsized":"2024-06-29T15:59:47.318424+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:50:09.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.431924+0000","last_change":"2024-06-29T15:59:46.324494+0000","last_active":"2024-06-29T15:59:47.431924+0000","last_peered":"2024-06-29T15:59:47.431924+0000","last_clean":"2024-06-29T15:59:47.431924+0000","last_became_active":"2024-06-29T15:59:46.324191+0000","last_became_peered":"2024-06-29T15:59:46.324191+0000","last_unstale":"2024-06-29T15:59:47.431924+0000","last_undegraded":"2024-06-29T15:59:47.431924+0000","last_fullsized":"2024-06-29T15:59:47.431924+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:28.437703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432476+0000","last_change":"2024-06-29T15:59:46.321593+0000","last_active":"2024-06-29T15:59:47.432476+0000","last_peered":"2024-06-29T15:59:47.432476+0000","last_clean":"2024-06-29T15:59:47.432476+0000","last_became_active":"2024-06-29T15:59:46.321301+0000","last_became_peered":"2024-06-29T15:59:46.321301+0000","last_unstale":"2024-06-29T15:59:47.432476+0000","last_undegraded":"2024-06-29T15:59:47.432476+0000","last_fullsized":"2024-06-29T15:59:47.432476+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:38:37.537548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"274'4","reported_seq":16,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.331088+0000","last_change":"2024-06-29T15:59:45.311920+0000","last_active":"2024-06-29T15:59:46.331088+0000","last_peered":"2024-06-29T15:59:46.331088+0000","last_clean":"2024-06-29T15:59:46.331088+0000","last_became_active":"2024-06-29T15:59:45.311587+0000","last_became_peered":"2024-06-29T15:59:45.311587+0000","last_unstale":"2024-06-29T15:59:46.331088+0000","last_undegraded":"2024-06-29T15:59:46.331088+0000","last_fullsized":"2024-06-29T15:59:46.331088+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:27:58.332086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312703+0000","last_change":"2024-06-29T15:59:46.309550+0000","last_active":"2024-06-29T15:59:46.312703+0000","last_peered":"2024-06-29T15:59:46.312703+0000","last_clean":"2024-06-29T15:59:46.312703+0000","last_became_active":"2024-06-29T15:59:46.309216+0000","last_became_peered":"2024-06-29T15:59:46.309216+0000","last_unstale":"2024-06-29T15:59:46.312703+0000","last_undegraded":"2024-06-29T15:59:46.312703+0000","last_fullsized":"2024-06-29T15:59:46.312703+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:51.449872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"274'1","reported_seq":15,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432475+0000","last_change":"2024-06-29T15:59:45.312200+0000","last_active":"2024-06-29T15:59:47.432475+0000","last_peered":"2024-06-29T15:59:47.432475+0000","last_clean":"2024-06-29T15:59:47.432475+0000","last_became_active":"2024-06-29T15:59:45.311885+0000","last_became_peered":"2024-06-29T15:59:45.311885+0000","last_unstale":"2024-06-29T15:59:47.432475+0000","last_undegraded":"2024-06-29T15:59:47.432475+0000","last_fullsized":"2024-06-29T15:59:47.432475+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:55.675973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318603+0000","last_change":"2024-06-29T15:59:46.330485+0000","last_active":"2024-06-29T15:59:47.318603+0000","last_peered":"2024-06-29T15:59:47.318603+0000","last_clean":"2024-06-29T15:59:47.318603+0000","last_became_active":"2024-06-29T15:59:46.328106+0000","last_became_peered":"2024-06-29T15:59:46.328106+0000","last_unstale":"2024-06-29T15:59:47.318603+0000","last_undegraded":"2024-06-29T15:59:47.318603+0000","last_fullsized":"2024-06-29T15:59:47.318603+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:13.205898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"274'2","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313484+0000","last_change":"2024-06-29T15:59:45.302817+0000","last_active":"2024-06-29T15:59:47.313484+0000","last_peered":"2024-06-29T15:59:47.313484+0000","last_clean":"2024-06-29T15:59:47.313484+0000","last_became_active":"2024-06-29T15:59:45.302474+0000","last_became_peered":"2024-06-29T15:59:45.302474+0000","last_unstale":"2024-06-29T15:59:47.313484+0000","last_undegraded":"2024-06-29T15:59:47.313484+0000","last_fullsized":"2024-06-29T15:59:47.313484+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:27.116567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318112+0000","last_change":"2024-06-29T15:59:46.325364+0000","last_active":"2024-06-29T15:59:47.318112+0000","last_peered":"2024-06-29T15:59:47.318112+0000","last_clean":"2024-06-29T15:59:47.318112+0000","last_became_active":"2024-06-29T15:59:46.325002+0000","last_became_peered":"2024-06-29T15:59:46.325002+0000","last_unstale":"2024-06-29T15:59:47.318112+0000","last_undegraded":"2024-06-29T15:59:47.318112+0000","last_fullsized":"2024-06-29T15:59:47.318112+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:30.030322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"274'2","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318053+0000","last_change":"2024-06-29T15:59:45.311877+0000","last_active":"2024-06-29T15:59:47.318053+0000","last_peered":"2024-06-29T15:59:47.318053+0000","last_clean":"2024-06-29T15:59:47.318053+0000","last_became_active":"2024-06-29T15:59:45.311530+0000","last_became_peered":"2024-06-29T15:59:45.311530+0000","last_unstale":"2024-06-29T15:59:47.318053+0000","last_undegraded":"2024-06-29T15:59:47.318053+0000","last_fullsized":"2024-06-29T15:59:47.318053+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:41.372645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.316361+0000","last_change":"2024-06-29T15:59:46.315081+0000","last_active":"2024-06-29T15:59:46.316361+0000","last_peered":"2024-06-29T15:59:46.316361+0000","last_clean":"2024-06-29T15:59:46.316361+0000","last_became_active":"2024-06-29T15:59:46.314785+0000","last_became_peered":"2024-06-29T15:59:46.314785+0000","last_unstale":"2024-06-29T15:59:46.316361+0000","last_undegraded":"2024-06-29T15:59:46.316361+0000","last_fullsized":"2024-06-29T15:59:46.316361+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:39:31.616716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"274'1","reported_seq":13,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.327883+0000","last_change":"2024-06-29T15:59:45.317104+0000","last_active":"2024-06-29T15:59:46.327883+0000","last_peered":"2024-06-29T15:59:46.327883+0000","last_clean":"2024-06-29T15:59:46.327883+0000","last_became_active":"2024-06-29T15:59:45.316754+0000","last_became_peered":"2024-06-29T15:59:45.316754+0000","last_unstale":"2024-06-29T15:59:46.327883+0000","last_undegraded":"2024-06-29T15:59:46.327883+0000","last_fullsized":"2024-06-29T15:59:46.327883+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:12.659077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.311393+0000","last_change":"2024-06-29T15:59:46.310423+0000","last_active":"2024-06-29T15:59:46.311393+0000","last_peered":"2024-06-29T15:59:46.311393+0000","last_clean":"2024-06-29T15:59:46.311393+0000","last_became_active":"2024-06-29T15:59:46.310017+0000","last_became_peered":"2024-06-29T15:59:46.310017+0000","last_unstale":"2024-06-29T15:59:46.311393+0000","last_undegraded":"2024-06-29T15:59:46.311393+0000","last_fullsized":"2024-06-29T15:59:46.311393+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:52.145571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"274'1","reported_seq":15,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313470+0000","last_change":"2024-06-29T15:59:45.304202+0000","last_active":"2024-06-29T15:59:47.313470+0000","last_peered":"2024-06-29T15:59:47.313470+0000","last_clean":"2024-06-29T15:59:47.313470+0000","last_became_active":"2024-06-29T15:59:45.303857+0000","last_became_peered":"2024-06-29T15:59:45.303857+0000","last_unstale":"2024-06-29T15:59:47.313470+0000","last_undegraded":"2024-06-29T15:59:47.313470+0000","last_fullsized":"2024-06-29T15:59:47.313470+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:51:53.509459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432210+0000","last_change":"2024-06-29T15:59:46.317777+0000","last_active":"2024-06-29T15:59:47.432210+0000","last_peered":"2024-06-29T15:59:47.432210+0000","last_clean":"2024-06-29T15:59:47.432210+0000","last_became_active":"2024-06-29T15:59:46.317328+0000","last_became_peered":"2024-06-29T15:59:46.317328+0000","last_unstale":"2024-06-29T15:59:47.432210+0000","last_undegraded":"2024-06-29T15:59:47.432210+0000","last_fullsized":"2024-06-29T15:59:47.432210+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:45.927002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318108+0000","last_change":"2024-06-29T15:59:45.312310+0000","last_active":"2024-06-29T15:59:47.318108+0000","last_peered":"2024-06-29T15:59:47.318108+0000","last_clean":"2024-06-29T15:59:47.318108+0000","last_became_active":"2024-06-29T15:59:45.311790+0000","last_became_peered":"2024-06-29T15:59:45.311790+0000","last_unstale":"2024-06-29T15:59:47.318108+0000","last_undegraded":"2024-06-29T15:59:47.318108+0000","last_fullsized":"2024-06-29T15:59:47.318108+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:24:06.806536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318012+0000","last_change":"2024-06-29T15:59:46.325400+0000","last_active":"2024-06-29T15:59:47.318012+0000","last_peered":"2024-06-29T15:59:47.318012+0000","last_clean":"2024-06-29T15:59:47.318012+0000","last_became_active":"2024-06-29T15:59:46.323267+0000","last_became_peered":"2024-06-29T15:59:46.323267+0000","last_unstale":"2024-06-29T15:59:47.318012+0000","last_undegraded":"2024-06-29T15:59:47.318012+0000","last_fullsized":"2024-06-29T15:59:47.318012+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:47:34.577970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318022+0000","last_change":"2024-06-29T15:59:45.307319+0000","last_active":"2024-06-29T15:59:47.318022+0000","last_peered":"2024-06-29T15:59:47.318022+0000","last_clean":"2024-06-29T15:59:47.318022+0000","last_became_active":"2024-06-29T15:59:45.306995+0000","last_became_peered":"2024-06-29T15:59:45.306995+0000","last_unstale":"2024-06-29T15:59:47.318022+0000","last_undegraded":"2024-06-29T15:59:47.318022+0000","last_fullsized":"2024-06-29T15:59:47.318022+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:55.816704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432471+0000","last_change":"2024-06-29T15:59:46.315319+0000","last_active":"2024-06-29T15:59:47.432471+0000","last_peered":"2024-06-29T15:59:47.432471+0000","last_clean":"2024-06-29T15:59:47.432471+0000","last_became_active":"2024-06-29T15:59:46.315018+0000","last_became_peered":"2024-06-29T15:59:46.315018+0000","last_unstale":"2024-06-29T15:59:47.432471+0000","last_undegraded":"2024-06-29T15:59:47.432471+0000","last_fullsized":"2024-06-29T15:59:47.432471+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:51:59.720938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.299705+0000","last_change":"2024-06-29T15:59:45.308295+0000","last_active":"2024-06-29T15:59:46.299705+0000","last_peered":"2024-06-29T15:59:46.299705+0000","last_clean":"2024-06-29T15:59:46.299705+0000","last_became_active":"2024-06-29T15:59:45.307913+0000","last_became_peered":"2024-06-29T15:59:45.307913+0000","last_unstale":"2024-06-29T15:59:46.299705+0000","last_undegraded":"2024-06-29T15:59:46.299705+0000","last_fullsized":"2024-06-29T15:59:46.299705+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:38.979304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.328717+0000","last_change":"2024-06-29T15:59:46.324805+0000","last_active":"2024-06-29T15:59:46.328717+0000","last_peered":"2024-06-29T15:59:46.328717+0000","last_clean":"2024-06-29T15:59:46.328717+0000","last_became_active":"2024-06-29T15:59:46.324474+0000","last_became_peered":"2024-06-29T15:59:46.324474+0000","last_unstale":"2024-06-29T15:59:46.328717+0000","last_undegraded":"2024-06-29T15:59:46.328717+0000","last_fullsized":"2024-06-29T15:59:46.328717+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:21.101877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.319012+0000","last_change":"2024-06-29T15:59:45.318294+0000","last_active":"2024-06-29T15:59:47.319012+0000","last_peered":"2024-06-29T15:59:47.319012+0000","last_clean":"2024-06-29T15:59:47.319012+0000","last_became_active":"2024-06-29T15:59:45.317757+0000","last_became_peered":"2024-06-29T15:59:45.317757+0000","last_unstale":"2024-06-29T15:59:47.319012+0000","last_undegraded":"2024-06-29T15:59:47.319012+0000","last_fullsized":"2024-06-29T15:59:47.319012+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:34:20.426052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318752+0000","last_change":"2024-06-29T15:59:46.329429+0000","last_active":"2024-06-29T15:59:47.318752+0000","last_peered":"2024-06-29T15:59:47.318752+0000","last_clean":"2024-06-29T15:59:47.318752+0000","last_became_active":"2024-06-29T15:59:46.326947+0000","last_became_peered":"2024-06-29T15:59:46.326947+0000","last_unstale":"2024-06-29T15:59:47.318752+0000","last_undegraded":"2024-06-29T15:59:47.318752+0000","last_fullsized":"2024-06-29T15:59:47.318752+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:02.906463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"274'3","reported_seq":17,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318120+0000","last_change":"2024-06-29T15:59:45.314428+0000","last_active":"2024-06-29T15:59:47.318120+0000","last_peered":"2024-06-29T15:59:47.318120+0000","last_clean":"2024-06-29T15:59:47.318120+0000","last_became_active":"2024-06-29T15:59:45.313411+0000","last_became_peered":"2024-06-29T15:59:45.313411+0000","last_unstale":"2024-06-29T15:59:47.318120+0000","last_undegraded":"2024-06-29T15:59:47.318120+0000","last_fullsized":"2024-06-29T15:59:47.318120+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:04:17.697812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432630+0000","last_change":"2024-06-29T15:59:46.311288+0000","last_active":"2024-06-29T15:59:47.432630+0000","last_peered":"2024-06-29T15:59:47.432630+0000","last_clean":"2024-06-29T15:59:47.432630+0000","last_became_active":"2024-06-29T15:59:46.310989+0000","last_became_peered":"2024-06-29T15:59:46.310989+0000","last_unstale":"2024-06-29T15:59:47.432630+0000","last_undegraded":"2024-06-29T15:59:47.432630+0000","last_fullsized":"2024-06-29T15:59:47.432630+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:48.933419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"274'6","reported_seq":18,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.334121+0000","last_change":"2024-06-29T15:59:45.305148+0000","last_active":"2024-06-29T15:59:46.334121+0000","last_peered":"2024-06-29T15:59:46.334121+0000","last_clean":"2024-06-29T15:59:46.334121+0000","last_became_active":"2024-06-29T15:59:45.304685+0000","last_became_peered":"2024-06-29T15:59:45.304685+0000","last_unstale":"2024-06-29T15:59:46.334121+0000","last_undegraded":"2024-06-29T15:59:46.334121+0000","last_fullsized":"2024-06-29T15:59:46.334121+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:21:48.981970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.318148+0000","last_change":"2024-06-29T15:59:46.313918+0000","last_active":"2024-06-29T15:59:46.318148+0000","last_peered":"2024-06-29T15:59:46.318148+0000","last_clean":"2024-06-29T15:59:46.318148+0000","last_became_active":"2024-06-29T15:59:46.313597+0000","last_became_peered":"2024-06-29T15:59:46.313597+0000","last_unstale":"2024-06-29T15:59:46.318148+0000","last_undegraded":"2024-06-29T15:59:46.318148+0000","last_fullsized":"2024-06-29T15:59:46.318148+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:15.673811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"274'2","reported_seq":14,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.332235+0000","last_change":"2024-06-29T15:59:45.317748+0000","last_active":"2024-06-29T15:59:46.332235+0000","last_peered":"2024-06-29T15:59:46.332235+0000","last_clean":"2024-06-29T15:59:46.332235+0000","last_became_active":"2024-06-29T15:59:45.317498+0000","last_became_peered":"2024-06-29T15:59:45.317498+0000","last_unstale":"2024-06-29T15:59:46.332235+0000","last_undegraded":"2024-06-29T15:59:46.332235+0000","last_fullsized":"2024-06-29T15:59:46.332235+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:34.763922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.311680+0000","last_change":"2024-06-29T15:59:46.308975+0000","last_active":"2024-06-29T15:59:46.311680+0000","last_peered":"2024-06-29T15:59:46.311680+0000","last_clean":"2024-06-29T15:59:46.311680+0000","last_became_active":"2024-06-29T15:59:46.308657+0000","last_became_peered":"2024-06-29T15:59:46.308657+0000","last_unstale":"2024-06-29T15:59:46.311680+0000","last_undegraded":"2024-06-29T15:59:46.311680+0000","last_fullsized":"2024-06-29T15:59:46.311680+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:17.793455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.302179+0000","last_change":"2024-06-29T15:59:45.313527+0000","last_active":"2024-06-29T15:59:46.302179+0000","last_peered":"2024-06-29T15:59:46.302179+0000","last_clean":"2024-06-29T15:59:46.302179+0000","last_became_active":"2024-06-29T15:59:45.313207+0000","last_became_peered":"2024-06-29T15:59:45.313207+0000","last_unstale":"2024-06-29T15:59:46.302179+0000","last_undegraded":"2024-06-29T15:59:46.302179+0000","last_fullsized":"2024-06-29T15:59:46.302179+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:54.070849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432190+0000","last_change":"2024-06-29T15:59:46.319677+0000","last_active":"2024-06-29T15:59:47.432190+0000","last_peered":"2024-06-29T15:59:47.432190+0000","last_clean":"2024-06-29T15:59:47.432190+0000","last_became_active":"2024-06-29T15:59:46.319294+0000","last_became_peered":"2024-06-29T15:59:46.319294+0000","last_unstale":"2024-06-29T15:59:47.432190+0000","last_undegraded":"2024-06-29T15:59:47.432190+0000","last_fullsized":"2024-06-29T15:59:47.432190+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:38.519216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.304072+0000","last_change":"2024-06-29T15:59:45.317935+0000","last_active":"2024-06-29T15:59:46.304072+0000","last_peered":"2024-06-29T15:59:46.304072+0000","last_clean":"2024-06-29T15:59:46.304072+0000","last_became_active":"2024-06-29T15:59:45.317600+0000","last_became_peered":"2024-06-29T15:59:45.317600+0000","last_unstale":"2024-06-29T15:59:46.304072+0000","last_undegraded":"2024-06-29T15:59:46.304072+0000","last_fullsized":"2024-06-29T15:59:46.304072+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:33.563934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.321918+0000","last_change":"2024-06-29T15:59:46.319714+0000","last_active":"2024-06-29T15:59:46.321918+0000","last_peered":"2024-06-29T15:59:46.321918+0000","last_clean":"2024-06-29T15:59:46.321918+0000","last_became_active":"2024-06-29T15:59:46.319419+0000","last_became_peered":"2024-06-29T15:59:46.319419+0000","last_unstale":"2024-06-29T15:59:46.321918+0000","last_undegraded":"2024-06-29T15:59:46.321918+0000","last_fullsized":"2024-06-29T15:59:46.321918+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:23:45.939879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"274'8","reported_seq":22,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432124+0000","last_change":"2024-06-29T15:59:45.311349+0000","last_active":"2024-06-29T15:59:47.432124+0000","last_peered":"2024-06-29T15:59:47.432124+0000","last_clean":"2024-06-29T15:59:47.432124+0000","last_became_active":"2024-06-29T15:59:45.311028+0000","last_became_peered":"2024-06-29T15:59:45.311028+0000","last_unstale":"2024-06-29T15:59:47.432124+0000","last_undegraded":"2024-06-29T15:59:47.432124+0000","last_fullsized":"2024-06-29T15:59:47.432124+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:14.377837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.329097+0000","last_change":"2024-06-29T15:59:46.328399+0000","last_active":"2024-06-29T15:59:46.329097+0000","last_peered":"2024-06-29T15:59:46.329097+0000","last_clean":"2024-06-29T15:59:46.329097+0000","last_became_active":"2024-06-29T15:59:46.328091+0000","last_became_peered":"2024-06-29T15:59:46.328091+0000","last_unstale":"2024-06-29T15:59:46.329097+0000","last_undegraded":"2024-06-29T15:59:46.329097+0000","last_fullsized":"2024-06-29T15:59:46.329097+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:20:54.768111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"274'1","reported_seq":15,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432748+0000","last_change":"2024-06-29T15:59:45.309069+0000","last_active":"2024-06-29T15:59:47.432748+0000","last_peered":"2024-06-29T15:59:47.432748+0000","last_clean":"2024-06-29T15:59:47.432748+0000","last_became_active":"2024-06-29T15:59:45.308713+0000","last_became_peered":"2024-06-29T15:59:45.308713+0000","last_unstale":"2024-06-29T15:59:47.432748+0000","last_undegraded":"2024-06-29T15:59:47.432748+0000","last_fullsized":"2024-06-29T15:59:47.432748+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:43:31.558256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.320408+0000","last_change":"2024-06-29T15:59:46.318781+0000","last_active":"2024-06-29T15:59:46.320408+0000","last_peered":"2024-06-29T15:59:46.320408+0000","last_clean":"2024-06-29T15:59:46.320408+0000","last_became_active":"2024-06-29T15:59:46.317288+0000","last_became_peered":"2024-06-29T15:59:46.317288+0000","last_unstale":"2024-06-29T15:59:46.320408+0000","last_undegraded":"2024-06-29T15:59:46.320408+0000","last_fullsized":"2024-06-29T15:59:46.320408+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:03.763576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"274'2","reported_seq":14,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.331027+0000","last_change":"2024-06-29T15:59:45.309424+0000","last_active":"2024-06-29T15:59:46.331027+0000","last_peered":"2024-06-29T15:59:46.331027+0000","last_clean":"2024-06-29T15:59:46.331027+0000","last_became_active":"2024-06-29T15:59:45.309045+0000","last_became_peered":"2024-06-29T15:59:45.309045+0000","last_unstale":"2024-06-29T15:59:46.331027+0000","last_undegraded":"2024-06-29T15:59:46.331027+0000","last_fullsized":"2024-06-29T15:59:46.331027+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:21:22.559018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.328909+0000","last_change":"2024-06-29T15:59:46.324839+0000","last_active":"2024-06-29T15:59:46.328909+0000","last_peered":"2024-06-29T15:59:46.328909+0000","last_clean":"2024-06-29T15:59:46.328909+0000","last_became_active":"2024-06-29T15:59:46.324524+0000","last_became_peered":"2024-06-29T15:59:46.324524+0000","last_unstale":"2024-06-29T15:59:46.328909+0000","last_undegraded":"2024-06-29T15:59:46.328909+0000","last_fullsized":"2024-06-29T15:59:46.328909+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:13.898105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"274'1","reported_seq":13,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.332455+0000","last_change":"2024-06-29T15:59:45.307573+0000","last_active":"2024-06-29T15:59:46.332455+0000","last_peered":"2024-06-29T15:59:46.332455+0000","last_clean":"2024-06-29T15:59:46.332455+0000","last_became_active":"2024-06-29T15:59:45.307252+0000","last_became_peered":"2024-06-29T15:59:45.307252+0000","last_unstale":"2024-06-29T15:59:46.332455+0000","last_undegraded":"2024-06-29T15:59:46.332455+0000","last_fullsized":"2024-06-29T15:59:46.332455+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:53.641375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313689+0000","last_change":"2024-06-29T15:59:46.316644+0000","last_active":"2024-06-29T15:59:47.313689+0000","last_peered":"2024-06-29T15:59:47.313689+0000","last_clean":"2024-06-29T15:59:47.313689+0000","last_became_active":"2024-06-29T15:59:46.316331+0000","last_became_peered":"2024-06-29T15:59:46.316331+0000","last_unstale":"2024-06-29T15:59:47.313689+0000","last_undegraded":"2024-06-29T15:59:47.313689+0000","last_fullsized":"2024-06-29T15:59:47.313689+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:55:34.272705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318122+0000","last_change":"2024-06-29T15:59:45.316585+0000","last_active":"2024-06-29T15:59:47.318122+0000","last_peered":"2024-06-29T15:59:47.318122+0000","last_clean":"2024-06-29T15:59:47.318122+0000","last_became_active":"2024-06-29T15:59:45.316221+0000","last_became_peered":"2024-06-29T15:59:45.316221+0000","last_unstale":"2024-06-29T15:59:47.318122+0000","last_undegraded":"2024-06-29T15:59:47.318122+0000","last_fullsized":"2024-06-29T15:59:47.318122+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:23.257182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432007+0000","last_change":"2024-06-29T15:59:46.326671+0000","last_active":"2024-06-29T15:59:47.432007+0000","last_peered":"2024-06-29T15:59:47.432007+0000","last_clean":"2024-06-29T15:59:47.432007+0000","last_became_active":"2024-06-29T15:59:46.326086+0000","last_became_peered":"2024-06-29T15:59:46.326086+0000","last_unstale":"2024-06-29T15:59:47.432007+0000","last_undegraded":"2024-06-29T15:59:47.432007+0000","last_fullsized":"2024-06-29T15:59:47.432007+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:41.570575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"274'2","reported_seq":14,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.324343+0000","last_change":"2024-06-29T15:59:45.313395+0000","last_active":"2024-06-29T15:59:46.324343+0000","last_peered":"2024-06-29T15:59:46.324343+0000","last_clean":"2024-06-29T15:59:46.324343+0000","last_became_active":"2024-06-29T15:59:45.313051+0000","last_became_peered":"2024-06-29T15:59:45.313051+0000","last_unstale":"2024-06-29T15:59:46.324343+0000","last_undegraded":"2024-06-29T15:59:46.324343+0000","last_fullsized":"2024-06-29T15:59:46.324343+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:33.929049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312750+0000","last_change":"2024-06-29T15:59:46.311815+0000","last_active":"2024-06-29T15:59:46.312750+0000","last_peered":"2024-06-29T15:59:46.312750+0000","last_clean":"2024-06-29T15:59:46.312750+0000","last_became_active":"2024-06-29T15:59:46.311496+0000","last_became_peered":"2024-06-29T15:59:46.311496+0000","last_unstale":"2024-06-29T15:59:46.312750+0000","last_undegraded":"2024-06-29T15:59:46.312750+0000","last_fullsized":"2024-06-29T15:59:46.312750+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:56:09.069111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432343+0000","last_change":"2024-06-29T15:59:45.313262+0000","last_active":"2024-06-29T15:59:47.432343+0000","last_peered":"2024-06-29T15:59:47.432343+0000","last_clean":"2024-06-29T15:59:47.432343+0000","last_became_active":"2024-06-29T15:59:45.312937+0000","last_became_peered":"2024-06-29T15:59:45.312937+0000","last_unstale":"2024-06-29T15:59:47.432343+0000","last_undegraded":"2024-06-29T15:59:47.432343+0000","last_fullsized":"2024-06-29T15:59:47.432343+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:34:50.986287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.312556+0000","last_change":"2024-06-29T15:59:46.309967+0000","last_active":"2024-06-29T15:59:46.312556+0000","last_peered":"2024-06-29T15:59:46.312556+0000","last_clean":"2024-06-29T15:59:46.312556+0000","last_became_active":"2024-06-29T15:59:46.309690+0000","last_became_peered":"2024-06-29T15:59:46.309690+0000","last_unstale":"2024-06-29T15:59:46.312556+0000","last_undegraded":"2024-06-29T15:59:46.312556+0000","last_fullsized":"2024-06-29T15:59:46.312556+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:05:14.020915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313207+0000","last_change":"2024-06-29T15:59:45.307624+0000","last_active":"2024-06-29T15:59:47.313207+0000","last_peered":"2024-06-29T15:59:47.313207+0000","last_clean":"2024-06-29T15:59:47.313207+0000","last_became_active":"2024-06-29T15:59:45.307287+0000","last_became_peered":"2024-06-29T15:59:45.307287+0000","last_unstale":"2024-06-29T15:59:47.313207+0000","last_undegraded":"2024-06-29T15:59:47.313207+0000","last_fullsized":"2024-06-29T15:59:47.313207+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:49:12.238348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.314413+0000","last_change":"2024-06-29T15:59:46.310006+0000","last_active":"2024-06-29T15:59:46.314413+0000","last_peered":"2024-06-29T15:59:46.314413+0000","last_clean":"2024-06-29T15:59:46.314413+0000","last_became_active":"2024-06-29T15:59:46.309688+0000","last_became_peered":"2024-06-29T15:59:46.309688+0000","last_unstale":"2024-06-29T15:59:46.314413+0000","last_undegraded":"2024-06-29T15:59:46.314413+0000","last_fullsized":"2024-06-29T15:59:46.314413+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:06.866195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313062+0000","last_change":"2024-06-29T15:59:45.315644+0000","last_active":"2024-06-29T15:59:47.313062+0000","last_peered":"2024-06-29T15:59:47.313062+0000","last_clean":"2024-06-29T15:59:47.313062+0000","last_became_active":"2024-06-29T15:59:45.315324+0000","last_became_peered":"2024-06-29T15:59:45.315324+0000","last_unstale":"2024-06-29T15:59:47.313062+0000","last_undegraded":"2024-06-29T15:59:47.313062+0000","last_fullsized":"2024-06-29T15:59:47.313062+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:08:34.700293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432605+0000","last_change":"2024-06-29T15:59:46.326497+0000","last_active":"2024-06-29T15:59:47.432605+0000","last_peered":"2024-06-29T15:59:47.432605+0000","last_clean":"2024-06-29T15:59:47.432605+0000","last_became_active":"2024-06-29T15:59:46.326121+0000","last_became_peered":"2024-06-29T15:59:46.326121+0000","last_unstale":"2024-06-29T15:59:47.432605+0000","last_undegraded":"2024-06-29T15:59:47.432605+0000","last_fullsized":"2024-06-29T15:59:47.432605+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:29.818673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"274'2","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318337+0000","last_change":"2024-06-29T15:59:45.309596+0000","last_active":"2024-06-29T15:59:47.318337+0000","last_peered":"2024-06-29T15:59:47.318337+0000","last_clean":"2024-06-29T15:59:47.318337+0000","last_became_active":"2024-06-29T15:59:45.309292+0000","last_became_peered":"2024-06-29T15:59:45.309292+0000","last_unstale":"2024-06-29T15:59:47.318337+0000","last_undegraded":"2024-06-29T15:59:47.318337+0000","last_fullsized":"2024-06-29T15:59:47.318337+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:02:27.053946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318470+0000","last_change":"2024-06-29T15:59:46.316443+0000","last_active":"2024-06-29T15:59:47.318470+0000","last_peered":"2024-06-29T15:59:47.318470+0000","last_clean":"2024-06-29T15:59:47.318470+0000","last_became_active":"2024-06-29T15:59:46.316115+0000","last_became_peered":"2024-06-29T15:59:46.316115+0000","last_unstale":"2024-06-29T15:59:47.318470+0000","last_undegraded":"2024-06-29T15:59:47.318470+0000","last_fullsized":"2024-06-29T15:59:47.318470+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:25.056067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"274'1","reported_seq":13,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.310447+0000","last_change":"2024-06-29T15:59:45.312684+0000","last_active":"2024-06-29T15:59:47.310447+0000","last_peered":"2024-06-29T15:59:47.310447+0000","last_clean":"2024-06-29T15:59:47.310447+0000","last_became_active":"2024-06-29T15:59:45.311065+0000","last_became_peered":"2024-06-29T15:59:45.311065+0000","last_unstale":"2024-06-29T15:59:47.310447+0000","last_undegraded":"2024-06-29T15:59:47.310447+0000","last_fullsized":"2024-06-29T15:59:47.310447+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:29.929013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318908+0000","last_change":"2024-06-29T15:59:46.328593+0000","last_active":"2024-06-29T15:59:47.318908+0000","last_peered":"2024-06-29T15:59:47.318908+0000","last_clean":"2024-06-29T15:59:47.318908+0000","last_became_active":"2024-06-29T15:59:46.326388+0000","last_became_peered":"2024-06-29T15:59:46.326388+0000","last_unstale":"2024-06-29T15:59:47.318908+0000","last_undegraded":"2024-06-29T15:59:47.318908+0000","last_fullsized":"2024-06-29T15:59:47.318908+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:11.725084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.306924+0000","last_change":"2024-06-29T15:59:45.312258+0000","last_active":"2024-06-29T15:59:46.306924+0000","last_peered":"2024-06-29T15:59:46.306924+0000","last_clean":"2024-06-29T15:59:46.306924+0000","last_became_active":"2024-06-29T15:59:45.311896+0000","last_became_peered":"2024-06-29T15:59:45.311896+0000","last_unstale":"2024-06-29T15:59:46.306924+0000","last_undegraded":"2024-06-29T15:59:46.306924+0000","last_fullsized":"2024-06-29T15:59:46.306924+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:17:06.924451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.318018+0000","last_change":"2024-06-29T15:59:46.312607+0000","last_active":"2024-06-29T15:59:46.318018+0000","last_peered":"2024-06-29T15:59:46.318018+0000","last_clean":"2024-06-29T15:59:46.318018+0000","last_became_active":"2024-06-29T15:59:46.312265+0000","last_became_peered":"2024-06-29T15:59:46.312265+0000","last_unstale":"2024-06-29T15:59:46.318018+0000","last_undegraded":"2024-06-29T15:59:46.318018+0000","last_fullsized":"2024-06-29T15:59:46.318018+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:38.688479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.299497+0000","last_change":"2024-06-29T15:59:45.307624+0000","last_active":"2024-06-29T15:59:46.299497+0000","last_peered":"2024-06-29T15:59:46.299497+0000","last_clean":"2024-06-29T15:59:46.299497+0000","last_became_active":"2024-06-29T15:59:45.307318+0000","last_became_peered":"2024-06-29T15:59:45.307318+0000","last_unstale":"2024-06-29T15:59:46.299497+0000","last_undegraded":"2024-06-29T15:59:46.299497+0000","last_fullsized":"2024-06-29T15:59:46.299497+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:50.581353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.318140+0000","last_change":"2024-06-29T15:59:46.316857+0000","last_active":"2024-06-29T15:59:46.318140+0000","last_peered":"2024-06-29T15:59:46.318140+0000","last_clean":"2024-06-29T15:59:46.318140+0000","last_became_active":"2024-06-29T15:59:46.316522+0000","last_became_peered":"2024-06-29T15:59:46.316522+0000","last_unstale":"2024-06-29T15:59:46.318140+0000","last_undegraded":"2024-06-29T15:59:46.318140+0000","last_fullsized":"2024-06-29T15:59:46.318140+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:21:49.989091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.301490+0000","last_change":"2024-06-29T15:59:45.314423+0000","last_active":"2024-06-29T15:59:46.301490+0000","last_peered":"2024-06-29T15:59:46.301490+0000","last_clean":"2024-06-29T15:59:46.301490+0000","last_became_active":"2024-06-29T15:59:45.314090+0000","last_became_peered":"2024-06-29T15:59:45.314090+0000","last_unstale":"2024-06-29T15:59:46.301490+0000","last_undegraded":"2024-06-29T15:59:46.301490+0000","last_fullsized":"2024-06-29T15:59:46.301490+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:42.531353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317766+0000","last_change":"2024-06-29T15:59:46.321869+0000","last_active":"2024-06-29T15:59:47.317766+0000","last_peered":"2024-06-29T15:59:47.317766+0000","last_clean":"2024-06-29T15:59:47.317766+0000","last_became_active":"2024-06-29T15:59:46.321530+0000","last_became_peered":"2024-06-29T15:59:46.321530+0000","last_unstale":"2024-06-29T15:59:47.317766+0000","last_undegraded":"2024-06-29T15:59:47.317766+0000","last_fullsized":"2024-06-29T15:59:47.317766+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:21.101877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.317958+0000","last_change":"2024-06-29T15:59:45.315433+0000","last_active":"2024-06-29T15:59:47.317958+0000","last_peered":"2024-06-29T15:59:47.317958+0000","last_clean":"2024-06-29T15:59:47.317958+0000","last_became_active":"2024-06-29T15:59:45.315099+0000","last_became_peered":"2024-06-29T15:59:45.315099+0000","last_unstale":"2024-06-29T15:59:47.317958+0000","last_undegraded":"2024-06-29T15:59:47.317958+0000","last_fullsized":"2024-06-29T15:59:47.317958+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:20.659828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.317962+0000","last_change":"2024-06-29T15:59:46.312511+0000","last_active":"2024-06-29T15:59:46.317962+0000","last_peered":"2024-06-29T15:59:46.317962+0000","last_clean":"2024-06-29T15:59:46.317962+0000","last_became_active":"2024-06-29T15:59:46.312110+0000","last_became_peered":"2024-06-29T15:59:46.312110+0000","last_unstale":"2024-06-29T15:59:46.317962+0000","last_undegraded":"2024-06-29T15:59:46.317962+0000","last_fullsized":"2024-06-29T15:59:46.317962+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:54.659040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":12,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.304439+0000","last_change":"2024-06-29T15:59:45.314280+0000","last_active":"2024-06-29T15:59:46.304439+0000","last_peered":"2024-06-29T15:59:46.304439+0000","last_clean":"2024-06-29T15:59:46.304439+0000","last_became_active":"2024-06-29T15:59:45.313968+0000","last_became_peered":"2024-06-29T15:59:45.313968+0000","last_unstale":"2024-06-29T15:59:46.304439+0000","last_undegraded":"2024-06-29T15:59:46.304439+0000","last_fullsized":"2024-06-29T15:59:46.304439+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:49.903052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":12,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.318857+0000","last_change":"2024-06-29T15:59:46.305937+0000","last_active":"2024-06-29T15:59:47.318857+0000","last_peered":"2024-06-29T15:59:47.318857+0000","last_clean":"2024-06-29T15:59:47.318857+0000","last_became_active":"2024-06-29T15:59:46.305607+0000","last_became_peered":"2024-06-29T15:59:46.305607+0000","last_unstale":"2024-06-29T15:59:47.318857+0000","last_undegraded":"2024-06-29T15:59:47.318857+0000","last_fullsized":"2024-06-29T15:59:47.318857+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:58:02.894952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":14,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.313182+0000","last_change":"2024-06-29T15:59:45.313038+0000","last_active":"2024-06-29T15:59:47.313182+0000","last_peered":"2024-06-29T15:59:47.313182+0000","last_clean":"2024-06-29T15:59:47.313182+0000","last_became_active":"2024-06-29T15:59:45.312694+0000","last_became_peered":"2024-06-29T15:59:45.312694+0000","last_unstale":"2024-06-29T15:59:47.313182+0000","last_undegraded":"2024-06-29T15:59:47.313182+0000","last_fullsized":"2024-06-29T15:59:47.313182+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:44.280102+0000","last_clean_scrub_stamp":"2024-06-29T15:59:44.280102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:44.236109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":9,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-29T15:59:46.323325+0000","last_change":"2024-06-29T15:59:46.320077+0000","last_active":"2024-06-29T15:59:46.323325+0000","last_peered":"2024-06-29T15:59:46.323325+0000","last_clean":"2024-06-29T15:59:46.323325+0000","last_became_active":"2024-06-29T15:59:46.319184+0000","last_became_peered":"2024-06-29T15:59:46.319184+0000","last_unstale":"2024-06-29T15:59:46.323325+0000","last_undegraded":"2024-06-29T15:59:46.323325+0000","last_fullsized":"2024-06-29T15:59:46.323325+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:45.287224+0000","last_clean_scrub_stamp":"2024-06-29T15:59:45.287224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:10.746800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"264'273","reported_seq":1345,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T15:59:47.432093+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T15:59:47.432093+0000","last_peered":"2024-06-29T15:59:47.432093+0000","last_clean":"2024-06-29T15:59:47.432093+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T15:59:47.432093+0000","last_undegraded":"2024-06-29T15:59:47.432093+0000","last_fullsized":"2024-06-29T15:59:47.432093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1788,"num_read_kb":5144,"num_write":445,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1788,"num_read_kb":5144,"num_write":445,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543313,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53212,"kb_used_data":3976,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318628,"statfs":{"total":96636764160,"available":96582275072,"internally_reserved":0,"allocated":4071424,"data_stored":2394036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543314,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48472,"kb_used_data":3400,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3481600,"data_stored":1804250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543313,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70660,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301180,"statfs":{"total":96636764160,"available":96564408320,"internally_reserved":0,"allocated":3465216,"data_stored":1796932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543312,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52632,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319208,"statfs":{"total":96636764160,"available":96582868992,"internally_reserved":0,"allocated":3485696,"data_stored":1799252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510610,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":3465216,"data_stored":1796932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52624,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319216,"statfs":{"total":96636764160,"available":96582877184,"internally_reserved":0,"allocated":3477504,"data_stored":1798580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510609,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":3380,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":3461120,"data_stored":1796898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510611,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49032,"kb_used_data":3964,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322808,"statfs":{"total":96636764160,"available":96586555392,"internally_reserved":0,"allocated":4059136,"data_stored":2387356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:59:50.762 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:59:50.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:59:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.925+0000 7f46b1cb3640 1 -- 172.21.15.59:0/3690682435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 msgr2=0x7f4694003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.925+0000 7f46b1cb3640 1 --2- 172.21.15.59:0/3690682435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 0x7f4694003e70 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f469c00c4a0 tx=0x7f4694005830 comp rx=0 tx=0).stop 2024-06-29T15:59:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.925+0000 7f46abfff640 1 -- 172.21.15.59:0/3690682435 reap_dead start 2024-06-29T15:59:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.925+0000 7f46b1cb3640 1 -- 172.21.15.59:0/3690682435 shutdown_connections 2024-06-29T15:59:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.925+0000 7f46b1cb3640 1 -- 172.21.15.59:0/3690682435 >> 172.21.15.59:0/3690682435 conn(0x7f46ac0d3e40 msgr2=0x7f46ac0d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.925+0000 7f46b1cb3640 1 -- 172.21.15.59:0/3690682435 shutdown_connections 2024-06-29T15:59:50.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.925+0000 7f46b1cb3640 1 -- 172.21.15.59:0/3690682435 wait complete. 2024-06-29T15:59:50.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.927+0000 7f46b1cb3640 1 Processor -- start 2024-06-29T15:59:50.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.927+0000 7f46b1cb3640 1 -- start start 2024-06-29T15:59:50.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46b1cb3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 0x7f46ac0c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46b1cb3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ac156f40 0x7f46ac06c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46b1cb3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ac06cd50 0x7f46ac161970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46b1cb3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac0cf000 con 0x7f4694003a00 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46b1cb3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac0b2c20 con 0x7f46ac156f40 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46b1cb3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46ac0cf180 con 0x7f46ac06cd50 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46ab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 0x7f46ac0c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46ab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 0x7f46ac0c7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32974/0 (socket says 172.21.15.59:32974) 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46ab7fe640 1 -- 172.21.15.59:0/2996391432 learned_addr learned my addr 172.21.15.59:0/2996391432 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46abfff640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ac06cd50 0x7f46ac161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.928+0000 7f46aaffd640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ac156f40 0x7f46ac06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f46ab7fe640 1 -- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ac06cd50 msgr2=0x7f46ac161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f46ab7fe640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ac06cd50 0x7f46ac161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f46ab7fe640 1 -- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ac156f40 msgr2=0x7f46ac06c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f46ab7fe640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ac156f40 0x7f46ac06c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f46ab7fe640 1 -- 172.21.15.59:0/2996391432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac06d160 con 0x7f4694003a00 2024-06-29T15:59:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f46ab7fe640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 0x7f46ac0c7950 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f469c00c5c0 tx=0x7f469c006f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f4693fff640 1 -- 172.21.15.59:0/2996391432 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c0065f0 con 0x7f4694003a00 2024-06-29T15:59:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f4693fff640 1 -- 172.21.15.59:0/2996391432 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f469c02b700 con 0x7f4694003a00 2024-06-29T15:59:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ac0c8410 con 0x7f4694003a00 2024-06-29T15:59:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f4693fff640 1 -- 172.21.15.59:0/2996391432 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c02bb10 con 0x7f4694003a00 2024-06-29T15:59:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.929+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ac161170 con 0x7f4694003a00 2024-06-29T15:59:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.930+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ac08ccf0 con 0x7f4694003a00 2024-06-29T15:59:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.931+0000 7f4693fff640 1 -- 172.21.15.59:0/2996391432 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f469c02bcb0 con 0x7f4694003a00 2024-06-29T15:59:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.931+0000 7f4693fff640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f46980763f0 0x7f4698078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.931+0000 7f4693fff640 1 -- 172.21.15.59:0/2996391432 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f469c0b7620 con 0x7f4694003a00 2024-06-29T15:59:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.932+0000 7f46aaffd640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f46980763f0 0x7f4698078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.932+0000 7f46aaffd640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f46980763f0 0x7f4698078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46ac06dd70 tx=0x7f46a0005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:50.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:50.936+0000 7f4693fff640 1 -- 172.21.15.59:0/2996391432 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f46ac08ccf0 con 0x7f4694003a00 2024-06-29T15:59:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.204+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f46ac0b3fc0 con 0x7f4694003a00 2024-06-29T15:59:51.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.207+0000 7f4693fff640 1 -- 172.21.15.59:0/2996391432 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v55) ==== 190+0+147 (secure 0 0 0) 0x7f46ac06d160 con 0x7f4694003a00 2024-06-29T15:59:51.207 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-06-29T15:59:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBHGIBmgmBkJRAA7cfcqGS3NMAuXwJKNUvpUg== 2024-06-29T15:59:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-06-29T15:59:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-06-29T15:59:51.207 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:59:51.208 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f46980763f0 msgr2=0x7f4698078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f46980763f0 0x7f4698078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46ac06dd70 tx=0x7f46a0005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 msgr2=0x7f46ac0c7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 0x7f46ac0c7950 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f469c00c5c0 tx=0x7f469c006f10 comp rx=0 tx=0).stop 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 shutdown_connections 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f46980763f0 0x7f4698078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46ac06cd50 0x7f46ac161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46ac156f40 0x7f46ac06c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 --2- 172.21.15.59:0/2996391432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4694003a00 0x7f46ac0c7950 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 >> 172.21.15.59:0/2996391432 conn(0x7f46ac0d3e40 msgr2=0x7f46ac15ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:51.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 shutdown_connections 2024-06-29T15:59:51.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.209+0000 7f46b1cb3640 1 -- 172.21.15.59:0/2996391432 wait complete. 2024-06-29T15:59:51.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.339+0000 7f3391064640 1 -- 172.21.15.59:0/2228370363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f338c0d23c0 msgr2=0x7f338c0b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.339+0000 7f3391064640 1 --2- 172.21.15.59:0/2228370363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f338c0d23c0 0x7f338c0b3990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f337c009be0 tx=0x7f337c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.339+0000 7f3391064640 1 -- 172.21.15.59:0/2228370363 shutdown_connections 2024-06-29T15:59:51.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.339+0000 7f3391064640 1 --2- 172.21.15.59:0/2228370363 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f338c0d23c0 0x7f338c0b3990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.339+0000 7f3391064640 1 -- 172.21.15.59:0/2228370363 >> 172.21.15.59:0/2228370363 conn(0x7f338c0d3ee0 msgr2=0x7f338c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:51.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.339+0000 7f3391064640 1 -- 172.21.15.59:0/2228370363 shutdown_connections 2024-06-29T15:59:51.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.339+0000 7f3391064640 1 -- 172.21.15.59:0/2228370363 wait complete. 2024-06-29T15:59:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.341+0000 7f3391064640 1 Processor -- start 2024-06-29T15:59:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.341+0000 7f3391064640 1 -- start start 2024-06-29T15:59:51.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f3391064640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f338c0b3ed0 0x7f338c158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f3391064640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f338c0d23c0 0x7f338c1575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f3391064640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f338c15efe0 0x7f338c15da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f3391064640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f338c160450 con 0x7f338c0b3ed0 2024-06-29T15:59:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f3391064640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f338c1602d0 con 0x7f338c15efe0 2024-06-29T15:59:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f3391064640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f338c1605d0 con 0x7f338c0d23c0 2024-06-29T15:59:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f338c0b3ed0 0x7f338c158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f338c0b3ed0 0x7f338c158190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32978/0 (socket says 172.21.15.59:32978) 2024-06-29T15:59:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338ad9d640 1 -- 172.21.15.59:0/1798865841 learned_addr learned my addr 172.21.15.59:0/1798865841 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338a59c640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f338c0d23c0 0x7f338c1575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338b59e640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f338c15efe0 0x7f338c15da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338ad9d640 1 -- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f338c0d23c0 msgr2=0x7f338c1575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338ad9d640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f338c0d23c0 0x7f338c1575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338ad9d640 1 -- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f338c15efe0 msgr2=0x7f338c15da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.343+0000 7f338ad9d640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f338c15efe0 0x7f338c15da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.344+0000 7f338ad9d640 1 -- 172.21.15.59:0/1798865841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338c15df70 con 0x7f338c0b3ed0 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.344+0000 7f338ad9d640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f338c0b3ed0 0x7f338c158190 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f337c009be0 tx=0x7f337c003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.344+0000 7f33737fe640 1 -- 172.21.15.59:0/1798865841 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c042020 con 0x7f338c0b3ed0 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.344+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f338c156ef0 con 0x7f338c0b3ed0 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.344+0000 7f33737fe640 1 -- 172.21.15.59:0/1798865841 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f337c033b80 con 0x7f338c0b3ed0 2024-06-29T15:59:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.344+0000 7f33737fe640 1 -- 172.21.15.59:0/1798865841 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c037040 con 0x7f338c0b3ed0 2024-06-29T15:59:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.344+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f338c1555a0 con 0x7f338c0b3ed0 2024-06-29T15:59:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.345+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f338c0b3990 con 0x7f338c0b3ed0 2024-06-29T15:59:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.346+0000 7f33737fe640 1 -- 172.21.15.59:0/1798865841 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f337c045020 con 0x7f338c0b3ed0 2024-06-29T15:59:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.346+0000 7f33737fe640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f33780763f0 0x7f3378078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.347+0000 7f33737fe640 1 -- 172.21.15.59:0/1798865841 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f337c0b8380 con 0x7f338c0b3ed0 2024-06-29T15:59:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.347+0000 7f338a59c640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f33780763f0 0x7f3378078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.347+0000 7f338a59c640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f33780763f0 0x7f3378078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f338c153860 tx=0x7f3380009040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:51.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.351+0000 7f33737fe640 1 -- 172.21.15.59:0/1798865841 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f337c04c070 con 0x7f338c0b3ed0 2024-06-29T15:59:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.583+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f338c0d2950 con 0x7f338c0b3ed0 2024-06-29T15:59:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.584+0000 7f33737fe640 1 -- 172.21.15.59:0/1798865841 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+5244 (secure 0 0 0) 0x7f338c15df70 con 0x7f338c0b3ed0 2024-06-29T15:59:51.585 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T15:59:51.585 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":157,"btime":"2024-06-29T15:59:49:317195+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1039110220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1039110220},{"type":"v1","addr":"172.21.15.153:6835","nonce":1039110220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":12583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3996608135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3996608135},{"type":"v1","addr":"172.21.15.153:6837","nonce":3996608135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":13620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/952472867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":952472867},{"type":"v1","addr":"172.21.15.59:6834","nonce":952472867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:45.699384+0000","modified":"2024-06-29T15:59:48.709634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13623":{"gid":13623,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3706652401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3706652401},{"type":"v1","addr":"172.21.15.59:6835","nonce":3706652401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13623,"qdb_cluster":[13623]},"id":18}]} 2024-06-29T15:59:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 157 2024-06-29T15:59:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f33780763f0 msgr2=0x7f3378078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f33780763f0 0x7f3378078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f338c153860 tx=0x7f3380009040 comp rx=0 tx=0).stop 2024-06-29T15:59:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f338c0b3ed0 msgr2=0x7f338c158190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f338c0b3ed0 0x7f338c158190 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f337c009be0 tx=0x7f337c003f00 comp rx=0 tx=0).stop 2024-06-29T15:59:51.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 shutdown_connections 2024-06-29T15:59:51.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f33780763f0 0x7f3378078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f338c15efe0 0x7f338c15da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f338c0d23c0 0x7f338c1575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 --2- 172.21.15.59:0/1798865841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f338c0b3ed0 0x7f338c158190 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.586+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 >> 172.21.15.59:0/1798865841 conn(0x7f338c0d3ee0 msgr2=0x7f338c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:51.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.587+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 shutdown_connections 2024-06-29T15:59:51.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:51.587+0000 7f3391064640 1 -- 172.21.15.59:0/1798865841 wait complete. 2024-06-29T15:59:52.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 157, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:59:45.699384+0000', 'modified': '2024-06-29T15:59:48.709634+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13623}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13623': {'gid': 13623, 'name': 'c', 'rank': 0, 'incarnation': 153, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/3706652401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 3706652401}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 3706652401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [38], 'metadata_pool': 37, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 13623, 'qdb_cluster': [13623]} 2024-06-29T15:59:52.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:59:52.000 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-06-29T15:59:52.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.124+0000 7f3883a12640 1 -- 172.21.15.59:0/3473406652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c141bd0 msgr2=0x7f387c13edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.124+0000 7f3883a12640 1 --2- 172.21.15.59:0/3473406652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c141bd0 0x7f387c13edc0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f3874009830 tx=0x7f387402a530 comp rx=0 tx=0).stop 2024-06-29T15:59:52.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.124+0000 7f3883a12640 1 -- 172.21.15.59:0/3473406652 shutdown_connections 2024-06-29T15:59:52.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.124+0000 7f3883a12640 1 --2- 172.21.15.59:0/3473406652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c141bd0 0x7f387c13edc0 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.124+0000 7f3883a12640 1 -- 172.21.15.59:0/3473406652 >> 172.21.15.59:0/3473406652 conn(0x7f387c0b6fb0 msgr2=0x7f387c0b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:52.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.124+0000 7f3883a12640 1 -- 172.21.15.59:0/3473406652 shutdown_connections 2024-06-29T15:59:52.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.124+0000 7f3883a12640 1 -- 172.21.15.59:0/3473406652 wait complete. 2024-06-29T15:59:52.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.126+0000 7f3883a12640 1 Processor -- start 2024-06-29T15:59:52.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.126+0000 7f3883a12640 1 -- start start 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f3883a12640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c13e9b0 0x7f387c0c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f3883a12640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f387c15ceb0 0x7f387c15a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f3883a12640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f387c13c040 0x7f387c15a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f3883a12640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f387c0b2900 con 0x7f387c13e9b0 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f3883a12640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f387c0b2780 con 0x7f387c15ceb0 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f3883a12640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f387c0b2a80 con 0x7f387c13c040 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f38817ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c13e9b0 0x7f387c0c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f38817ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c13e9b0 0x7f387c0c71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33002/0 (socket says 172.21.15.59:33002) 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.127+0000 7f38817ae640 1 -- 172.21.15.59:0/2430985188 learned_addr learned my addr 172.21.15.59:0/2430985188 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f3881faf640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f387c13c040 0x7f387c15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f3880fad640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f387c15ceb0 0x7f387c15a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f38817ae640 1 -- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f387c13c040 msgr2=0x7f387c15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f38817ae640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f387c13c040 0x7f387c15a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f38817ae640 1 -- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f387c15ceb0 msgr2=0x7f387c15a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f38817ae640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f387c15ceb0 0x7f387c15a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f38817ae640 1 -- 172.21.15.59:0/2430985188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387c158270 con 0x7f387c13e9b0 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f38817ae640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c13e9b0 0x7f387c0c71f0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f386c00cb40 tx=0x7f386c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f3879ffb640 1 -- 172.21.15.59:0/2430985188 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c016020 con 0x7f387c13e9b0 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f3879ffb640 1 -- 172.21.15.59:0/2430985188 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f386c00adb0 con 0x7f387c13e9b0 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f3883a12640 1 -- 172.21.15.59:0/2430985188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387c158550 con 0x7f387c13e9b0 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f3879ffb640 1 -- 172.21.15.59:0/2430985188 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c012680 con 0x7f387c13e9b0 2024-06-29T15:59:52.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.128+0000 7f3883a12640 1 -- 172.21.15.59:0/2430985188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f387c15b260 con 0x7f387c13e9b0 2024-06-29T15:59:52.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.130+0000 7f3879ffb640 1 -- 172.21.15.59:0/2430985188 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f386c019020 con 0x7f387c13e9b0 2024-06-29T15:59:52.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.131+0000 7f3879ffb640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38680763f0 0x7f3868078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.131+0000 7f3879ffb640 1 -- 172.21.15.59:0/2430985188 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f386c098230 con 0x7f387c13e9b0 2024-06-29T15:59:52.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.131+0000 7f3880fad640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38680763f0 0x7f3868078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.132+0000 7f388a9ac740 1 Processor -- start 2024-06-29T15:59:52.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.132+0000 7f388a9ac740 1 -- start start 2024-06-29T15:59:52.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.132+0000 7f3880fad640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38680763f0 0x7f3868078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f387c0cdb20 tx=0x7f3870009040 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc5f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc5db3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5db8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f388a9ac740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5587cc573bc0 con 0x5587cc20b270 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f388a9ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5587cc522630 con 0x5587cc1e23b0 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f388a9ac740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5587cc511bd0 con 0x5587cc1ea590 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f38817ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc5f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f38817ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc5f76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33004/0 (socket says 172.21.15.59:33004) 2024-06-29T15:59:52.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f38817ae640 1 -- 172.21.15.59:0/3132063629 learned_addr learned my addr 172.21.15.59:0/3132063629 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:52.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f3880fad640 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc5db3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f3881faf640 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5db8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f38817ae640 1 -- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 msgr2=0x5587cc5db8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:52.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f38817ae640 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5db8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f38817ae640 1 -- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 msgr2=0x5587cc5db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:52.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f38817ae640 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc5db3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f38817ae640 1 -- 172.21.15.59:0/3132063629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587cc50eca0 con 0x5587cc20b270 2024-06-29T15:59:52.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f3881faf640 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5db8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:52.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.133+0000 7f3880fad640 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc5db3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:52.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.134+0000 7f38817ae640 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc5f76e0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f386c05d1c0 tx=0x7f386c060c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c34cb4deb4d7632 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:52.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.134+0000 7f3862ffd640 1 -- 172.21.15.59:0/3132063629 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c07a020 con 0x5587cc20b270 2024-06-29T15:59:52.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.134+0000 7f3862ffd640 1 -- 172.21.15.59:0/3132063629 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f386c065440 con 0x5587cc20b270 2024-06-29T15:59:52.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.134+0000 7f3862ffd640 1 -- 172.21.15.59:0/3132063629 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c069ef0 con 0x5587cc20b270 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.134+0000 7f388a9ac740 1 -- 172.21.15.59:0/3132063629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 msgr2=0x5587cc5f76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.134+0000 7f388a9ac740 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc5f76e0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f386c05d1c0 tx=0x7f386c060c50 comp rx=0 tx=0).stop 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.134+0000 7f388a9ac740 1 -- 172.21.15.59:0/3132063629 shutdown_connections 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.134+0000 7f388a9ac740 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5db8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.135+0000 7f388a9ac740 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc5db3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.135+0000 7f388a9ac740 1 --2- 172.21.15.59:0/3132063629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc5f76e0 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.135+0000 7f388a9ac740 1 -- 172.21.15.59:0/3132063629 >> 172.21.15.59:0/3132063629 conn(0x5587cc1ff3c0 msgr2=0x5587cc5aef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.135+0000 7f388a9ac740 1 -- 172.21.15.59:0/3132063629 shutdown_connections 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.135+0000 7f388a9ac740 1 -- 172.21.15.59:0/3132063629 wait complete. 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.135+0000 7f388a9ac740 1 -- start start 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.136+0000 7f388a9ac740 1 Processor -- start 2024-06-29T15:59:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1e23b0 0x5587cc60d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5d5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f388a9ac740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5587cc4afae0 con 0x5587cc1e23b0 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f388a9ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5587cc515c10 con 0x5587cc20b270 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f388a9ac740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5587cc285620 con 0x5587cc1ea590 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f388a9ac740 10 client.0 _ll_register_callbacks cb 0x5587cc4bc480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f38817ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1e23b0 0x5587cc60d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f38817ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1e23b0 0x5587cc60d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33008/0 (socket says 172.21.15.59:33008) 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f38817ae640 1 -- 172.21.15.59:0/4230192313 learned_addr learned my addr 172.21.15.59:0/4230192313 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f3880fad640 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5d5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f3881faf640 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f38817ae640 1 -- 172.21.15.59:0/4230192313 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 msgr2=0x5587cc5d5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f38817ae640 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5d5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f38817ae640 1 -- 172.21.15.59:0/4230192313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 msgr2=0x5587cc5f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f38817ae640 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.137+0000 7f38817ae640 1 -- 172.21.15.59:0/4230192313 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587cc276850 con 0x5587cc1e23b0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f38817ae640 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1e23b0 0x5587cc60d0a0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f386c05d730 tx=0x7f386c001c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab76971e3f9ec5bd server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f385bfff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f385bfff640 1 -- 172.21.15.59:0/4230192313 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c07e030 con 0x5587cc1e23b0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f385bfff640 1 -- 172.21.15.59:0/4230192313 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f386c090e00 con 0x5587cc1e23b0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 1 -- 172.21.15.59:0/4230192313 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f386c065440 con 0x5587cc1e23b0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f385bfff640 1 -- 172.21.15.59:0/4230192313 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7f386c07a050 con 0x5587cc1e23b0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 10 client.13773 fetch_fsmap learned FSMap version 157 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 1 -- 172.21.15.59:0/4230192313 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=157,monmap=2+}) -- 0x5587cc2ed6b0 con 0x5587cc1e23b0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f385bfff640 1 -- 172.21.15.59:0/4230192313 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f386c065440 con 0x5587cc1e23b0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f385bfff640 1 -- 172.21.15.59:0/4230192313 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 157) ==== 3502+0+0 (secure 0 0 0) 0x7f386c0603a0 con 0x5587cc1e23b0 2024-06-29T15:59:52.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 10 client.13773 fetch_fsmap finished waiting for FSMap version 157 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 10 client.13773 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 20 client.13773 populate_metadata read hostname 'smithi059' 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] conn(0x5587cc1c9e40 0x5587cc5d4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 4 client.13773 mds_command: new command op to 13623 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f388a9ac740 1 -- 172.21.15.59:0/4230192313 --> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587cc524580 con 0x5587cc1c9e40 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.138+0000 7f3880fad640 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] conn(0x5587cc1c9e40 0x5587cc5d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.139+0000 7f385bfff640 10 client.13773 ms_handle_connect on v2:172.21.15.59:6833/3706652401 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.139+0000 7f3880fad640 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] conn(0x5587cc1c9e40 0x5587cc5d4730 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8e3760257262f10 server_cookie=82ddc6b4d0cc1f74 in_seq=0 out_seq=0 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f385bfff640 1 -- 172.21.15.59:0/4230192313 <== mds.0 v2:172.21.15.59:6833/3706652401 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5587cc524580 con 0x5587cc1c9e40 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f385bfff640 10 client.13773 handle_command_reply: tid=0 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 client.13773 shutdown 2024-06-29T15:59:52.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 -- 172.21.15.59:0/4230192313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1e23b0 msgr2=0x5587cc60d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1e23b0 0x5587cc60d0a0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f386c05d730 tx=0x7f386c001c60 comp rx=0 tx=0).stop 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 -- 172.21.15.59:0/4230192313 shutdown_connections 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] conn(0x5587cc1c9e40 0x5587cc5d4730 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc5d5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f385bfff640 0 client.13773 ms_handle_reset on v2:172.21.15.59:6833/3706652401 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 --2- 172.21.15.59:0/4230192313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1e23b0 0x5587cc60d0a0 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 -- 172.21.15.59:0/4230192313 >> 172.21.15.59:0/4230192313 conn(0x5587cc1ff3c0 msgr2=0x5587cc5d7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:52.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 -- 172.21.15.59:0/4230192313 shutdown_connections 2024-06-29T15:59:52.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 1 -- 172.21.15.59:0/4230192313 wait complete. 2024-06-29T15:59:52.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.140+0000 7f388a9ac740 20 client.13773 trim_cache size 0 max 16384 2024-06-29T15:59:52.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.159+0000 7f388a9ac740 1 Processor -- start 2024-06-29T15:59:52.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.159+0000 7f388a9ac740 1 -- start start 2024-06-29T15:59:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1e23b0 0x5587cc5f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1ea590 0x5587cc5f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5d2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f388a9ac740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f387c1fca10 con 0x5587cc1ea590 2024-06-29T15:59:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f388a9ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f387c06f030 con 0x5587cc20b270 2024-06-29T15:59:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f388a9ac740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f387c06f680 con 0x5587cc1e23b0 2024-06-29T15:59:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1ea590 0x5587cc5f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1ea590 0x5587cc5f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33018/0 (socket says 172.21.15.59:33018) 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 -- 172.21.15.59:0/3315830684 learned_addr learned my addr 172.21.15.59:0/3315830684 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3881faf640 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5d2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f38817ae640 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1e23b0 0x5587cc5f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 -- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1e23b0 msgr2=0x5587cc5f76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1e23b0 0x5587cc5f76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 -- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 msgr2=0x5587cc5d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5d2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 -- 172.21.15.59:0/3315830684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587cc60c870 con 0x5587cc1ea590 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3881faf640 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5d2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f38817ae640 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1e23b0 0x5587cc5f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:52.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f3880fad640 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1ea590 0x5587cc5f7c20 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f3870001550 tx=0x7f3870012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d8668d370b2836 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f38627fc640 1 -- 172.21.15.59:0/3315830684 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387000fea0 con 0x5587cc1ea590 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f38627fc640 1 -- 172.21.15.59:0/3315830684 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f387001b040 con 0x5587cc1ea590 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.161+0000 7f38627fc640 1 -- 172.21.15.59:0/3315830684 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870014b10 con 0x5587cc1ea590 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 -- 172.21.15.59:0/3315830684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1ea590 msgr2=0x5587cc5f7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1ea590 0x5587cc5f7c20 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f3870001550 tx=0x7f3870012000 comp rx=0 tx=0).stop 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 -- 172.21.15.59:0/3315830684 shutdown_connections 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc20b270 0x5587cc5d2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc1ea590 0x5587cc5f7c20 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 --2- 172.21.15.59:0/3315830684 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1e23b0 0x5587cc5f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 -- 172.21.15.59:0/3315830684 >> 172.21.15.59:0/3315830684 conn(0x5587cc1ff3c0 msgr2=0x5587cc5ec7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 -- 172.21.15.59:0/3315830684 shutdown_connections 2024-06-29T15:59:52.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.162+0000 7f388a9ac740 1 -- 172.21.15.59:0/3315830684 wait complete. 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.163+0000 7f388a9ac740 1 -- start start 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.163+0000 7f388a9ac740 1 Processor -- start 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc6605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc60b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f388a9ac740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc60bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f388a9ac740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x5587cc4afae0 con 0x5587cc20b270 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f388a9ac740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5587cc285620 con 0x5587cc1e23b0 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f388a9ac740 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x5587cc515c10 con 0x5587cc1ea590 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f388a9ac740 10 client.0 _ll_register_callbacks cb 0x7f386c060fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc60bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc60bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33022/0 (socket says 172.21.15.59:33022) 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 -- 172.21.15.59:0/511771673 learned_addr learned my addr 172.21.15.59:0/511771673 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f38817ae640 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc6605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3880fad640 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc60b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 -- 172.21.15.59:0/511771673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 msgr2=0x5587cc60b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc60b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 -- 172.21.15.59:0/511771673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 msgr2=0x5587cc6605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc6605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 -- 172.21.15.59:0/511771673 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587cc660b10 con 0x5587cc20b270 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f3881faf640 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc60bca0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f387402ab20 tx=0x7f3874004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8028e45df8e70b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f385bfff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f385bfff640 1 -- 172.21.15.59:0/511771673 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874003a30 con 0x5587cc20b270 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f385bfff640 1 -- 172.21.15.59:0/511771673 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f3874003bd0 con 0x5587cc20b270 2024-06-29T15:59:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.165+0000 7f388a9ac740 1 -- 172.21.15.59:0/511771673 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f387001b040 con 0x5587cc20b270 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f385bfff640 1 -- 172.21.15.59:0/511771673 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7f3874003dd0 con 0x5587cc20b270 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f388a9ac740 10 client.13779 fetch_fsmap learned FSMap version 157 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f388a9ac740 1 -- 172.21.15.59:0/511771673 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=157,monmap=2+}) -- 0x5587cc6773d0 con 0x5587cc20b270 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f385bfff640 1 -- 172.21.15.59:0/511771673 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f387001b040 con 0x5587cc20b270 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f385bfff640 1 -- 172.21.15.59:0/511771673 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 157) ==== 3502+0+0 (secure 0 0 0) 0x7f3874052080 con 0x5587cc20b270 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f388a9ac740 10 client.13779 fetch_fsmap finished waiting for FSMap version 157 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f388a9ac740 10 client.13779 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f388a9ac740 20 client.13779 populate_metadata read hostname 'smithi059' 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f388a9ac740 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] conn(0x5587cc5d3fc0 0x5587cc61da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f388a9ac740 4 client.13779 mds_command: new command op to 13623 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f388a9ac740 1 -- 172.21.15.59:0/511771673 --> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] -- command(tid 0: {"prefix": "status"}) -- 0x5587cc520020 con 0x5587cc5d3fc0 2024-06-29T15:59:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f38817ae640 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] conn(0x5587cc5d3fc0 0x5587cc61da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.166+0000 7f385bfff640 10 client.13779 ms_handle_connect on v2:172.21.15.59:6833/3706652401 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f38817ae640 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] conn(0x5587cc5d3fc0 0x5587cc61da20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fedf03fdcb98b16 server_cookie=9feb30c793f4e165 in_seq=0 out_seq=0 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f385bfff640 1 -- 172.21.15.59:0/511771673 <== mds.0 v2:172.21.15.59:6833/3706652401 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x5587cc520020 con 0x5587cc5d3fc0 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f385bfff640 10 client.13779 handle_command_reply: tid=0 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 client.13779 shutdown 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 -- 172.21.15.59:0/511771673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 msgr2=0x5587cc60bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc60bca0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f387402ab20 tx=0x7f3874004610 comp rx=0 tx=0).stop 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 -- 172.21.15.59:0/511771673 shutdown_connections 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5587cc20b270 0x5587cc60bca0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x5587cc1ea590 0x5587cc60b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.59:6833/3706652401,v1:172.21.15.59:6835/3706652401] conn(0x5587cc5d3fc0 0x5587cc61da20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 --2- 172.21.15.59:0/511771673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5587cc1e23b0 0x5587cc6605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 -- 172.21.15.59:0/511771673 >> 172.21.15.59:0/511771673 conn(0x5587cc1ff3c0 msgr2=0x5587cc5ec7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f385bfff640 0 client.13779 ms_handle_reset on v2:172.21.15.59:6833/3706652401 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 -- 172.21.15.59:0/511771673 shutdown_connections 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 1 -- 172.21.15.59:0/511771673 wait complete. 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.167+0000 7f388a9ac740 20 client.13779 trim_cache size 0 max 16384 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "150b8fd8-1bf5-44f0-b8d0-0083a48c0071", 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 13623, 2024-06-29T15:59:52.173 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-06-29T15:59:52.174 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-06-29T15:59:52.174 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-06-29T15:59:52.174 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.8623109700000002, 2024-06-29T15:59:52.174 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 157, 2024-06-29T15:59:52.174 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 275, 2024-06-29T15:59:52.174 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 275, 2024-06-29T15:59:52.174 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.468996832 2024-06-29T15:59:52.174 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 -- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38680763f0 msgr2=0x7f3868078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38680763f0 0x7f3868078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f387c0cdb20 tx=0x7f3870009040 comp rx=0 tx=0).stop 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 -- 172.21.15.59:0/2430985188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c13e9b0 msgr2=0x7f387c0c71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c13e9b0 0x7f387c0c71f0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f386c00cb40 tx=0x7f386c00a060 comp rx=0 tx=0).stop 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 -- 172.21.15.59:0/2430985188 shutdown_connections 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f38680763f0 0x7f3868078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f387c13c040 0x7f387c15a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f387c15ceb0 0x7f387c15a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 --2- 172.21.15.59:0/2430985188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387c13e9b0 0x7f387c0c71f0 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 -- 172.21.15.59:0/2430985188 >> 172.21.15.59:0/2430985188 conn(0x7f387c0b6fb0 msgr2=0x7f387c13dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:52.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 -- 172.21.15.59:0/2430985188 shutdown_connections 2024-06-29T15:59:52.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.168+0000 7f3883a12640 1 -- 172.21.15.59:0/2430985188 wait complete. 2024-06-29T15:59:52.179 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:59:52.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.297+0000 7f06dd481640 1 -- 172.21.15.59:0/4216864345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 msgr2=0x7f06c4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.297+0000 7f06dd481640 1 --2- 172.21.15.59:0/4216864345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 0x7f06c4005150 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f06c8005fd0 tx=0x7f06c4005830 comp rx=0 tx=0).stop 2024-06-29T15:59:52.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.297+0000 7f06d77fe640 1 -- 172.21.15.59:0/4216864345 reap_dead start 2024-06-29T15:59:52.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.298+0000 7f06dd481640 1 -- 172.21.15.59:0/4216864345 shutdown_connections 2024-06-29T15:59:52.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.298+0000 7f06dd481640 1 -- 172.21.15.59:0/4216864345 >> 172.21.15.59:0/4216864345 conn(0x7f06d80d3ee0 msgr2=0x7f06d80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:52.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.298+0000 7f06dd481640 1 -- 172.21.15.59:0/4216864345 shutdown_connections 2024-06-29T15:59:52.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.298+0000 7f06dd481640 1 -- 172.21.15.59:0/4216864345 wait complete. 2024-06-29T15:59:52.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.299+0000 7f06dd481640 1 Processor -- start 2024-06-29T15:59:52.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.300+0000 7f06dd481640 1 -- start start 2024-06-29T15:59:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06dd481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 0x7f06d8158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06dd481640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06d8154fd0 0x7f06d81f8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06dd481640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06d81f8ad0 0x7f06d814e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06dd481640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f06d815fd40 con 0x7f06c4004d40 2024-06-29T15:59:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06dd481640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06d815fbc0 con 0x7f06d8154fd0 2024-06-29T15:59:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06dd481640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f06d815fec0 con 0x7f06d81f8ad0 2024-06-29T15:59:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 0x7f06d8158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 0x7f06d8158c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33042/0 (socket says 172.21.15.59:33042) 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d6ffd640 1 -- 172.21.15.59:0/4271793559 learned_addr learned my addr 172.21.15.59:0/4271793559 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d77fe640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06d81f8ad0 0x7f06d814e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d67fc640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06d8154fd0 0x7f06d81f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d6ffd640 1 -- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06d81f8ad0 msgr2=0x7f06d814e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d6ffd640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06d81f8ad0 0x7f06d814e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d6ffd640 1 -- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06d8154fd0 msgr2=0x7f06d81f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d6ffd640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06d8154fd0 0x7f06d81f8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.301+0000 7f06d6ffd640 1 -- 172.21.15.59:0/4271793559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d81591c0 con 0x7f06c4004d40 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.302+0000 7f06d6ffd640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 0x7f06d8158c80 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f06d0006610 tx=0x7f06d00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.302+0000 7f06cf7fe640 1 -- 172.21.15.59:0/4271793559 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0015070 con 0x7f06c4004d40 2024-06-29T15:59:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.302+0000 7f06cf7fe640 1 -- 172.21.15.59:0/4271793559 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06d000f980 con 0x7f06c4004d40 2024-06-29T15:59:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.302+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d81f8ee0 con 0x7f06c4004d40 2024-06-29T15:59:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.302+0000 7f06cf7fe640 1 -- 172.21.15.59:0/4271793559 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d000fdd0 con 0x7f06c4004d40 2024-06-29T15:59:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.302+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d8148700 con 0x7f06c4004d40 2024-06-29T15:59:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.303+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d8160420 con 0x7f06c4004d40 2024-06-29T15:59:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.304+0000 7f06cf7fe640 1 -- 172.21.15.59:0/4271793559 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f06d0011430 con 0x7f06c4004d40 2024-06-29T15:59:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.304+0000 7f06cf7fe640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f06b80763f0 0x7f06b8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.304+0000 7f06cf7fe640 1 -- 172.21.15.59:0/4271793559 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f06d00989f0 con 0x7f06c4004d40 2024-06-29T15:59:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.304+0000 7f06d67fc640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f06b80763f0 0x7f06b8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.305+0000 7f06d67fc640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f06b80763f0 0x7f06b8078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06d815e710 tx=0x7f06c0005f90 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:52.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.309+0000 7f06cf7fe640 1 -- 172.21.15.59:0/4271793559 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f06d81f8ee0 con 0x7f06c4004d40 2024-06-29T15:59:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.537+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f06d815eaf0 con 0x7f06c4004d40 2024-06-29T15:59:52.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.538+0000 7f06cf7fe640 1 -- 172.21.15.59:0/4271793559 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v157) ==== 53+0+73 (secure 0 0 0) 0x7f06d0061ef0 con 0x7f06c4004d40 2024-06-29T15:59:52.538 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:59:52.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f06b80763f0 msgr2=0x7f06b8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f06b80763f0 0x7f06b8078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06d815e710 tx=0x7f06c0005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:52.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 msgr2=0x7f06d8158c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:52.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 0x7f06d8158c80 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f06d0006610 tx=0x7f06d00039c0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 shutdown_connections 2024-06-29T15:59:52.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f06b80763f0 0x7f06b8078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06d81f8ad0 0x7f06d814e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06d8154fd0 0x7f06d81f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 --2- 172.21.15.59:0/4271793559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06c4004d40 0x7f06d8158c80 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:52.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 >> 172.21.15.59:0/4271793559 conn(0x7f06d80d3ee0 msgr2=0x7f06d806ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:52.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 shutdown_connections 2024-06-29T15:59:52.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52.540+0000 7f06dd481640 1 -- 172.21.15.59:0/4271793559 wait complete. 2024-06-29T15:59:52.551 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:59:52.551 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:59:52.551 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-06-29T15:59:52.551 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:59:52.551 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:52.551 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:59:52.551 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:59:52.551 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:59:52.552 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:52.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:52.552 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-06-29T15:59:52.568 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:59:52.568 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:59:52.568 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-06-29T15:59:52.569 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 36858sec preferred_lft 36858sec 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:59:52.570 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::a81a:9cff:fea4:df57/64 scope link 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether c2:60:c7:24:57:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::c060:c7ff:fe24:57cd/64 scope link 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether b6:87:fa:e6:59:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:59:52.571 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::b487:faff:fee6:5936/64 scope link 2024-06-29T15:59:52.572 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:52.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:52.572 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:59:52.633 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-06-29T15:59:52.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:52.635 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-06-29T15:59:52.691 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-06-29T15:59:52.692 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:59:52.692 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-06-29T15:59:52.692 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-06-29T15:59:52.692 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:59:52.692 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:59:52.692 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-06-29T15:59:52.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-06-29T15:59:52.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-06-29T15:59:52.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-06-29T15:59:52.693 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-06-29T15:59:52.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:59:52.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:59:52.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:52.821 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T15:59:52.837 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:59:52.837 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:59:52.838 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:52.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:52.838 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-06-29T15:59:52.904 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:59:52.904 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:52.959 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:52.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:59:52.960 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:53.015 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-06-29T15:59:53.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:53.016 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:59:53.111 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:59:53.111 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:59:53.111 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:59:53.122 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:59:53.122 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:59:53.122 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:59:53.122 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:59:53.122 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:59:53.128 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-06-29T15:59:53.128 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-06-29T15:59:53.128 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-06-29T15:59:53.128 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.153:3300,172.21.15.153:3301" 2024-06-29T15:59:53.128 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-06-29T15:59:53.129 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790" 2024-06-29T15:59:53.129 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.153:6789,172.21.15.153:6790:/ 2024-06-29T15:59:53.129 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=150b8fd8-1bf5-44f0-b8d0-0083a48c0071" will pass to kernel 2024-06-29T15:59:53.129 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:59:53.129 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:59:53.129 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:59:53.130 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:59:53.130 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:59:53.131 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:59:53.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:53.131 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:53.157 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:59:53.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:59:53.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:59:53.260 INFO:teuthology.orchestra.run.smithi059.stdout:client13788 2024-06-29T15:59:53.260 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:53.261 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:53.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:59:53.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 -- 172.21.15.59:0/1360277985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 msgr2=0x7efe9c0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:53.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 --2- 172.21.15.59:0/1360277985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 0x7efe9c0d1520 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7efe9c074c50 tx=0x7efe8402a560 comp rx=0 tx=0).stop 2024-06-29T15:59:53.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 -- 172.21.15.59:0/1360277985 shutdown_connections 2024-06-29T15:59:53.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 --2- 172.21.15.59:0/1360277985 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe9c0d1a60 0x7efe9c0cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 --2- 172.21.15.59:0/1360277985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 0x7efe9c0d1520 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 --2- 172.21.15.59:0/1360277985 >> v2:172.21.15.153:3300/0 conn(0x7efe9c0b2270 0x7efe9c15bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 -- 172.21.15.59:0/1360277985 >> 172.21.15.59:0/1360277985 conn(0x7efe9c073e10 msgr2=0x7efe9c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:53.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 -- 172.21.15.59:0/1360277985 shutdown_connections 2024-06-29T15:59:53.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.436+0000 7efea1426640 1 -- 172.21.15.59:0/1360277985 wait complete. 2024-06-29T15:59:53.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.438+0000 7efea1426640 1 Processor -- start 2024-06-29T15:59:53.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.438+0000 7efea1426640 1 -- start start 2024-06-29T15:59:53.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.439+0000 7efea1426640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe9c0b2270 0x7efe9c0ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.439+0000 7efea1426640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 0x7efe9c153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.439+0000 7efea1426640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe9c0d1a60 0x7efe9c1573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.439+0000 7efea1426640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c0cfe40 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.439+0000 7efea1426640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c0cfcc0 con 0x7efe9c0d1a60 2024-06-29T15:59:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efea1426640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efe9c0cffc0 con 0x7efe9c0b2270 2024-06-29T15:59:53.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 0x7efe9c153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:53.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 0x7efe9c153d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54364/0 (socket says 172.21.15.59:54364) 2024-06-29T15:59:53.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 -- 172.21.15.59:0/3429753970 learned_addr learned my addr 172.21.15.59:0/3429753970 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:53.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9affd640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe9c0b2270 0x7efe9c0ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:53.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9b7fe640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe9c0d1a60 0x7efe9c1573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:53.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 -- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe9c0b2270 msgr2=0x7efe9c0ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:53.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe9c0b2270 0x7efe9c0ae2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 -- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe9c0d1a60 msgr2=0x7efe9c1573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe9c0d1a60 0x7efe9c1573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 -- 172.21.15.59:0/3429753970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9c0adb00 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe9a7fc640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 0x7efe9c153d60 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7efe84000c00 tx=0x7efe840038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe937fe640 1 -- 172.21.15.59:0/3429753970 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8403e030 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe937fe640 1 -- 172.21.15.59:0/3429753970 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe84003ca0 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efe937fe640 1 -- 172.21.15.59:0/3429753970 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe84035040 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.440+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe9c0adde0 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.441+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe9c15ec40 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.441+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe9c15b2b0 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.442+0000 7efe937fe640 1 -- 172.21.15.59:0/3429753970 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7efe84041030 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.443+0000 7efe937fe640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7efe7c0763f0 0x7efe7c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:53.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.443+0000 7efe937fe640 1 -- 172.21.15.59:0/3429753970 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7efe840b8590 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.443+0000 7efe9affd640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7efe7c0763f0 0x7efe7c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:53.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.444+0000 7efe9affd640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7efe7c0763f0 0x7efe7c078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efe94005fd0 tx=0x7efe94009500 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:53.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.446+0000 7efe937fe640 1 -- 172.21.15.59:0/3429753970 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efe9c15b2b0 con 0x7efe9c0b2fa0 2024-06-29T15:59:53.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.665+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 --> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7efe9c0ad8b0 con 0x7efe7c0763f0 2024-06-29T15:59:53.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.667+0000 7efe937fe640 1 -- 172.21.15.59:0/3429753970 <== mgr.12382 v2:172.21.15.153:6832/3072370296 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7efe9c0ad8b0 con 0x7efe7c0763f0 2024-06-29T15:59:53.667 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-06-29T15:59:53.667 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-06-29T15:59:53.667 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs" 2024-06-29T15:59:53.667 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-06-29T15:59:53.667 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-06-29T15:59:53.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7efe7c0763f0 msgr2=0x7efe7c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:53.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7efe7c0763f0 0x7efe7c078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efe94005fd0 tx=0x7efe94009500 comp rx=0 tx=0).stop 2024-06-29T15:59:53.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 msgr2=0x7efe9c153d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:53.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 0x7efe9c153d60 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7efe84000c00 tx=0x7efe840038a0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 shutdown_connections 2024-06-29T15:59:53.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7efe7c0763f0 0x7efe7c078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe9c0d1a60 0x7efe9c1573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe9c0b2fa0 0x7efe9c153d60 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 --2- 172.21.15.59:0/3429753970 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efe9c0b2270 0x7efe9c0ae2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 >> 172.21.15.59:0/3429753970 conn(0x7efe9c073e10 msgr2=0x7efe9c0707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:53.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 shutdown_connections 2024-06-29T15:59:53.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.669+0000 7efea1426640 1 -- 172.21.15.59:0/3429753970 wait complete. 2024-06-29T15:59:53.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:59:53.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.802+0000 7fc743d2a640 1 -- 172.21.15.59:0/3191850431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc728005350 msgr2=0x7fc72800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:53.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.802+0000 7fc743d2a640 1 --2- 172.21.15.59:0/3191850431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc728005350 0x7fc72800b9c0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0c75c0 tx=0x7fc728005830 comp rx=0 tx=0).stop 2024-06-29T15:59:53.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.802+0000 7fc743d2a640 1 -- 172.21.15.59:0/3191850431 shutdown_connections 2024-06-29T15:59:53.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.802+0000 7fc743d2a640 1 --2- 172.21.15.59:0/3191850431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc728005350 0x7fc72800b9c0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.802+0000 7fc743d2a640 1 -- 172.21.15.59:0/3191850431 >> 172.21.15.59:0/3191850431 conn(0x7fc73c0d3ee0 msgr2=0x7fc73c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:53.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.802+0000 7fc743d2a640 1 -- 172.21.15.59:0/3191850431 shutdown_connections 2024-06-29T15:59:53.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.802+0000 7fc743d2a640 1 -- 172.21.15.59:0/3191850431 wait complete. 2024-06-29T15:59:53.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.804+0000 7fc743d2a640 1 Processor -- start 2024-06-29T15:59:53.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.804+0000 7fc743d2a640 1 -- start start 2024-06-29T15:59:53.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.805+0000 7fc743d2a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc728005350 0x7fc73c0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:53.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.805+0000 7fc743d2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f8d50 0x7fc73c1566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:53.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.805+0000 7fc743d2a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f8140 0x7fc73c1f8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.805+0000 7fc743d2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c15ed10 con 0x7fc73c1f8d50 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.805+0000 7fc743d2a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c15eb90 con 0x7fc728005350 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.805+0000 7fc743d2a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc73c0b3990 con 0x7fc73c1f8140 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc741ac6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc728005350 0x7fc73c0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7412c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f8d50 0x7fc73c1566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc741ac6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc728005350 0x7fc73c0ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.59:48122/0 (socket says 172.21.15.59:48122) 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc741ac6640 1 -- 172.21.15.59:0/1724214691 learned_addr learned my addr 172.21.15.59:0/1724214691 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7422c7640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f8140 0x7fc73c1f8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7412c5640 1 -- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f8140 msgr2=0x7fc73c1f8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7412c5640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f8140 0x7fc73c1f8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7412c5640 1 -- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc728005350 msgr2=0x7fc73c0ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:53.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7412c5640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc728005350 0x7fc73c0ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:53.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7412c5640 1 -- 172.21.15.59:0/1724214691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c1f8a90 con 0x7fc73c1f8d50 2024-06-29T15:59:53.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc741ac6640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc728005350 0x7fc73c0ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:53.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7412c5640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f8d50 0x7fc73c1566f0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fc72c003de0 tx=0x7fc72c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:53.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7327fc640 1 -- 172.21.15.59:0/1724214691 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c011640 con 0x7fc73c1f8d50 2024-06-29T15:59:53.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7327fc640 1 -- 172.21.15.59:0/1724214691 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc72c016020 con 0x7fc73c1f8d50 2024-06-29T15:59:53.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73c15a3a0 con 0x7fc73c1f8d50 2024-06-29T15:59:53.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc7327fc640 1 -- 172.21.15.59:0/1724214691 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c011a00 con 0x7fc73c1f8d50 2024-06-29T15:59:53.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.806+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc73c15aa20 con 0x7fc73c1f8d50 2024-06-29T15:59:53.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.807+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc73c0b3e80 con 0x7fc73c1f8d50 2024-06-29T15:59:53.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.808+0000 7fc7327fc640 1 -- 172.21.15.59:0/1724214691 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fc72c011ba0 con 0x7fc73c1f8d50 2024-06-29T15:59:53.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.809+0000 7fc7327fc640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc7200763f0 0x7fc720078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:53.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.809+0000 7fc7327fc640 1 -- 172.21.15.59:0/1724214691 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7fc72c098af0 con 0x7fc73c1f8d50 2024-06-29T15:59:53.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.809+0000 7fc741ac6640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc7200763f0 0x7fc720078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:53.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.810+0000 7fc741ac6640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc7200763f0 0x7fc720078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc738001ca0 tx=0x7fc738006b20 comp rx=0 tx=0).ready entity=mgr.12382 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:53.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:53.813+0000 7fc7327fc640 1 -- 172.21.15.59:0/1724214691 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc72c020020 con 0x7fc73c1f8d50 2024-06-29T15:59:54.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.052+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fc73c0d2950 con 0x7fc73c1f8d50 2024-06-29T15:59:54.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.344+0000 7fc7327fc640 1 -- 172.21.15.59:0/1724214691 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v156) ==== 92+0+0 (secure 0 0 0) 0x7fc73c1f8a90 con 0x7fc73c1f8d50 2024-06-29T15:59:54.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc7200763f0 msgr2=0x7fc720078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc7200763f0 0x7fc720078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc738001ca0 tx=0x7fc738006b20 comp rx=0 tx=0).stop 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f8d50 msgr2=0x7fc73c1566f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f8d50 0x7fc73c1566f0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fc72c003de0 tx=0x7fc72c001630 comp rx=0 tx=0).stop 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 shutdown_connections 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fc7200763f0 0x7fc720078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc73c1f8140 0x7fc73c1f8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc73c1f8d50 0x7fc73c1566f0 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 --2- 172.21.15.59:0/1724214691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc728005350 0x7fc73c0ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 >> 172.21.15.59:0/1724214691 conn(0x7fc73c0d3ee0 msgr2=0x7fc73c0b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 shutdown_connections 2024-06-29T15:59:54.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.347+0000 7fc743d2a640 1 -- 172.21.15.59:0/1724214691 wait complete. 2024-06-29T15:59:54.361 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:59:54.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.486+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2403210550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d40d23c0 msgr2=0x7fe7d40b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.486+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2403210550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d40d23c0 0x7fe7d40b3990 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fe7d408c4f0 tx=0x7fe7c402a560 comp rx=0 tx=0).stop 2024-06-29T15:59:54.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.486+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2403210550 shutdown_connections 2024-06-29T15:59:54.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.486+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2403210550 >> v2:172.21.15.153:3300/0 conn(0x7fe7d40b3ed0 0x7fe7d415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.486+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2403210550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d40d23c0 0x7fe7d40b3990 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.486+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2403210550 >> 172.21.15.59:0/2403210550 conn(0x7fe7d40d3ee0 msgr2=0x7fe7d40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:54.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.486+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2403210550 shutdown_connections 2024-06-29T15:59:54.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.486+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2403210550 wait complete. 2024-06-29T15:59:54.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.488+0000 7fe7d8d7a640 1 Processor -- start 2024-06-29T15:59:54.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.488+0000 7fe7d8d7a640 1 -- start start 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.489+0000 7fe7d8d7a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d40b3ed0 0x7fe7d415d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.489+0000 7fe7d8d7a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7d40d23c0 0x7fe7d415d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d8d7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d415ee10 0x7fe7d41e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d8d7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4160240 con 0x7fe7d415ee10 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d8d7a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d41600c0 con 0x7fe7d40b3ed0 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d8d7a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d41603c0 con 0x7fe7d40d23c0 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d415ee10 0x7fe7d41e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d40b3ed0 0x7fe7d415d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:54.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d415ee10 0x7fe7d41e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54410/0 (socket says 172.21.15.59:54410) 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 -- 172.21.15.59:0/2913659653 learned_addr learned my addr 172.21.15.59:0/2913659653 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d1d9b640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7d40d23c0 0x7fe7d415d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 -- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7d40d23c0 msgr2=0x7fe7d415d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7d40d23c0 0x7fe7d415d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 -- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d40b3ed0 msgr2=0x7fe7d415d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d40b3ed0 0x7fe7d415d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 -- 172.21.15.59:0/2913659653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d415de90 con 0x7fe7d415ee10 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d259c640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d40b3ed0 0x7fe7d415d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:54.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7d2d9d640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d415ee10 0x7fe7d41e7280 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00be60 tx=0x7fe7cc00bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:54.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.490+0000 7fe7c2ffd640 1 -- 172.21.15.59:0/2913659653 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc019070 con 0x7fe7d415ee10 2024-06-29T15:59:54.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.491+0000 7fe7c2ffd640 1 -- 172.21.15.59:0/2913659653 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7cc014440 con 0x7fe7d415ee10 2024-06-29T15:59:54.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.491+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d415e170 con 0x7fe7d415ee10 2024-06-29T15:59:54.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.491+0000 7fe7c2ffd640 1 -- 172.21.15.59:0/2913659653 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc014890 con 0x7fe7d415ee10 2024-06-29T15:59:54.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.491+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d41e6a80 con 0x7fe7d415ee10 2024-06-29T15:59:54.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.492+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d415e8d0 con 0x7fe7d415ee10 2024-06-29T15:59:54.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.493+0000 7fe7c2ffd640 1 -- 172.21.15.59:0/2913659653 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 156) ==== 98320+0+0 (secure 0 0 0) 0x7fe7cc01c070 con 0x7fe7d415ee10 2024-06-29T15:59:54.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.493+0000 7fe7c2ffd640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe7b4076470 0x7fe7b4078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:54.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.493+0000 7fe7c2ffd640 1 -- 172.21.15.59:0/2913659653 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7fe7cc097a90 con 0x7fe7d415ee10 2024-06-29T15:59:54.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.493+0000 7fe7d259c640 1 -- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe7b4076470 msgr2=0x7fe7b4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3072370296 2024-06-29T15:59:54.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.493+0000 7fe7d259c640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe7b4076470 0x7fe7b4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:54.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.502+0000 7fe7c2ffd640 1 -- 172.21.15.59:0/2913659653 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe7cc060a50 con 0x7fe7d415ee10 2024-06-29T15:59:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.695+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fe7d40d2950 con 0x7fe7d415ee10 2024-06-29T15:59:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.695+0000 7fe7d259c640 1 -- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe7b4076470 msgr2=0x7fe7b4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3072370296 2024-06-29T15:59:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.695+0000 7fe7d259c640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe7b4076470 0x7fe7b4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:54.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.698+0000 7fe7c2ffd640 1 -- 172.21.15.59:0/2913659653 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v85)=0 v85) ==== 138+0+0 (secure 0 0 0) 0x7fe7cc064d50 con 0x7fe7d415ee10 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe7b4076470 msgr2=0x7fe7b4078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe7b4076470 0x7fe7b4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d415ee10 msgr2=0x7fe7d41e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d415ee10 0x7fe7d41e7280 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00be60 tx=0x7fe7cc00bf50 comp rx=0 tx=0).stop 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 shutdown_connections 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fe7b4076470 0x7fe7b4078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7d415ee10 0x7fe7d41e7280 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe7d40d23c0 0x7fe7d415d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 --2- 172.21.15.59:0/2913659653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7d40b3ed0 0x7fe7d415d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.700+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 >> 172.21.15.59:0/2913659653 conn(0x7fe7d40d3ee0 msgr2=0x7fe7d4157470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.701+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 shutdown_connections 2024-06-29T15:59:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.701+0000 7fe7d8d7a640 1 -- 172.21.15.59:0/2913659653 wait complete. 2024-06-29T15:59:54.713 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:59:54.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 -- 172.21.15.59:0/3554428194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2270 msgr2=0x7fb19415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 --2- 172.21.15.59:0/3554428194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2270 0x7fb19415bc20 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7fb1940b2d10 tx=0x7fb18802a560 comp rx=0 tx=0).stop 2024-06-29T15:59:54.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 -- 172.21.15.59:0/3554428194 shutdown_connections 2024-06-29T15:59:54.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 --2- 172.21.15.59:0/3554428194 >> v2:172.21.15.153:3300/0 conn(0x7fb1940d1a60 0x7fb1940cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 --2- 172.21.15.59:0/3554428194 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b2fa0 0x7fb1940d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 --2- 172.21.15.59:0/3554428194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2270 0x7fb19415bc20 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 -- 172.21.15.59:0/3554428194 >> 172.21.15.59:0/3554428194 conn(0x7fb194073e10 msgr2=0x7fb194074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:54.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 -- 172.21.15.59:0/3554428194 shutdown_connections 2024-06-29T15:59:54.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.834+0000 7fb19b33c640 1 -- 172.21.15.59:0/3554428194 wait complete. 2024-06-29T15:59:54.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.836+0000 7fb19b33c640 1 Processor -- start 2024-06-29T15:59:54.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.836+0000 7fb19b33c640 1 -- start start 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.837+0000 7fb19b33c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b2270 0x7fb1940b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.837+0000 7fb19b33c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2fa0 0x7fb19415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.837+0000 7fb19b33c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940d1a60 0x7fb19415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.837+0000 7fb19b33c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb1940cfe40 con 0x7fb1940b2fa0 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.837+0000 7fb19b33c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1940cfcc0 con 0x7fb1940d1a60 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.837+0000 7fb19b33c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb1940cffc0 con 0x7fb1940b2270 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2fa0 0x7fb19415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2fa0 0x7fb19415eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54420/0 (socket says 172.21.15.59:54420) 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 -- 172.21.15.59:0/1030070831 learned_addr learned my addr 172.21.15.59:0/1030070831 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:54.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1990d8640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b2270 0x7fb1940b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1998d9640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940d1a60 0x7fb19415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 -- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b2270 msgr2=0x7fb1940b5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b2270 0x7fb1940b5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 -- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940d1a60 msgr2=0x7fb19415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940d1a60 0x7fb19415eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 -- 172.21.15.59:0/1030070831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1941f3800 con 0x7fb1940b2fa0 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb1988d7640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2fa0 0x7fb19415eab0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fb19000e7c0 tx=0x7fb19000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb185ffb640 1 -- 172.21.15.59:0/1030070831 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19000ccc0 con 0x7fb1940b2fa0 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb185ffb640 1 -- 172.21.15.59:0/1030070831 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb19000ce80 con 0x7fb1940b2fa0 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1940a9860 con 0x7fb1940b2fa0 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.838+0000 7fb185ffb640 1 -- 172.21.15.59:0/1030070831 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190005280 con 0x7fb1940b2fa0 2024-06-29T15:59:54.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.839+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1940a9e20 con 0x7fb1940b2fa0 2024-06-29T15:59:54.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.840+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1940d0450 con 0x7fb1940b2fa0 2024-06-29T15:59:54.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.841+0000 7fb185ffb640 1 -- 172.21.15.59:0/1030070831 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 156) ==== 98320+0+0 (secure 0 0 0) 0x7fb190005420 con 0x7fb1940b2fa0 2024-06-29T15:59:54.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.842+0000 7fb185ffb640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb178076470 0x7fb178078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:54.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.842+0000 7fb185ffb640 1 -- 172.21.15.59:0/1030070831 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7fb1900983f0 con 0x7fb1940b2fa0 2024-06-29T15:59:54.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.842+0000 7fb1990d8640 1 -- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb178076470 msgr2=0x7fb178078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3072370296 2024-06-29T15:59:54.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.842+0000 7fb1990d8640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb178076470 0x7fb178078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:54.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:54.846+0000 7fb185ffb640 1 -- 172.21.15.59:0/1030070831 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb190061290 con 0x7fb1940b2fa0 2024-06-29T15:59:55.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.043+0000 7fb1990d8640 1 -- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb178076470 msgr2=0x7fb178078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3072370296 2024-06-29T15:59:55.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.043+0000 7fb1990d8640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb178076470 0x7fb178078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:55.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.043+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fb1940cf920 con 0x7fb1940b2fa0 2024-06-29T15:59:55.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.047+0000 7fb185ffb640 1 -- 172.21.15.59:0/1030070831 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v86)=0 v86) ==== 133+0+0 (secure 0 0 0) 0x7fb190065590 con 0x7fb1940b2fa0 2024-06-29T15:59:55.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb178076470 msgr2=0x7fb178078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:55.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb178076470 0x7fb178078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2fa0 msgr2=0x7fb19415eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2fa0 0x7fb19415eab0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fb19000e7c0 tx=0x7fb19000ec80 comp rx=0 tx=0).stop 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 shutdown_connections 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7fb178076470 0x7fb178078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1940d1a60 0x7fb19415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1940b2fa0 0x7fb19415eab0 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 --2- 172.21.15.59:0/1030070831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb1940b2270 0x7fb1940b5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 >> 172.21.15.59:0/1030070831 conn(0x7fb194073e10 msgr2=0x7fb194074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 shutdown_connections 2024-06-29T15:59:55.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.049+0000 7fb19b33c640 1 -- 172.21.15.59:0/1030070831 wait complete. 2024-06-29T15:59:55.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:55.063 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8) 2024-06-29T15:59:55.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:55.097 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.subvolume -v 1 volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8) 2024-06-29T15:59:55.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:55.169 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-06-29T15:59:55.225 INFO:teuthology.orchestra.run.smithi059.stdout:cephfs_data 2024-06-29T15:59:55.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:55.251 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8) 2024-06-29T15:59:55.299 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T15:59:55.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/sv01/.meta 2024-06-29T15:59:55.370 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv01 . 1m 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.533+0000 7f3f07226640 1 -- 172.21.15.59:0/2226245007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d0a80 msgr2=0x7f3f000b1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.533+0000 7f3f07226640 1 --2- 172.21.15.59:0/2226245007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d0a80 0x7f3f000b1930 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f3f000738c0 tx=0x7f3ef802aac0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.533+0000 7f3f07226640 1 -- 172.21.15.59:0/2226245007 shutdown_connections 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.533+0000 7f3f07226640 1 --2- 172.21.15.59:0/2226245007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d0a80 0x7f3f000b1930 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.533+0000 7f3f07226640 1 --2- 172.21.15.59:0/2226245007 >> v2:172.21.15.153:3300/0 conn(0x7f3f000b09e0 0x7f3f0015bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.533+0000 7f3f07226640 1 -- 172.21.15.59:0/2226245007 >> 172.21.15.59:0/2226245007 conn(0x7f3f00073e10 msgr2=0x7f3f00074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.533+0000 7f3f07226640 1 -- 172.21.15.59:0/2226245007 shutdown_connections 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.534+0000 7f3f07226640 1 -- 172.21.15.59:0/2226245007 wait complete. 2024-06-29T15:59:55.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.536+0000 7f3f07226640 1 Processor -- start 2024-06-29T15:59:55.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.537+0000 7f3f07226640 1 -- start start 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3f07226640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f000b09e0 0x7f3f00149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3f07226640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d2010 0x7f3f0015ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3f07226640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0015d760 0x7f3f001f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3f07226640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f3f000b1ff0 con 0x7f3f000d2010 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3f07226640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f000b1e70 con 0x7f3f0015d760 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3f07226640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3f000b2170 con 0x7f3f000b09e0 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3efffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d2010 0x7f3f0015ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3efffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d2010 0x7f3f0015ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54438/0 (socket says 172.21.15.59:54438) 2024-06-29T15:59:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3efffff640 1 -- 172.21.15.59:0/1095392224 learned_addr learned my addr 172.21.15.59:0/1095392224 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3f057c3640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0015d760 0x7f3f001f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.539+0000 7f3f04fc2640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f000b09e0 0x7f3f00149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efffff640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f000b09e0 msgr2=0x7f3f00149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efffff640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3f000b09e0 0x7f3f00149640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efffff640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0015d760 msgr2=0x7f3f001f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efffff640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f0015d760 0x7f3f001f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efffff640 1 -- 172.21.15.59:0/1095392224 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0015dd40 con 0x7f3f000d2010 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efffff640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d2010 0x7f3f0015ede0 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00c3a0 tx=0x7f3eec00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec017070 con 0x7f3f000d2010 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3eec0053c0 con 0x7f3f000d2010 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec003850 con 0x7f3f000d2010 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0015e020 con 0x7f3f000d2010 2024-06-29T15:59:55.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.540+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0015e5e0 con 0x7f3f000d2010 2024-06-29T15:59:55.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.541+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f000cf810 con 0x7f3f000d2010 2024-06-29T15:59:55.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.542+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 156) ==== 98320+0+0 (secure 0 0 0) 0x7f3eec01a070 con 0x7f3f000d2010 2024-06-29T15:59:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.543+0000 7f3efd7fa640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 0x7f3ee4078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.543+0000 7f3f04fc2640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 msgr2=0x7f3ee4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3072370296 2024-06-29T15:59:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.543+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6157+0+0 (secure 0 0 0) 0x7f3eec096d50 con 0x7f3f000d2010 2024-06-29T15:59:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.543+0000 7f3f04fc2640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 0x7f3ee4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:55.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.547+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3f000cf810 con 0x7f3f000d2010 2024-06-29T15:59:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.744+0000 7f3f04fc2640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 msgr2=0x7f3ee4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3072370296 2024-06-29T15:59:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.744+0000 7f3f04fc2640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 0x7f3ee4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:55.762+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 --> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv01", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3f000b2640 con 0x7f3ee4076470 2024-06-29T15:59:56.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:56.145+0000 7f3f04fc2640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 msgr2=0x7f3ee4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3072370296 2024-06-29T15:59:56.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:56.145+0000 7f3f04fc2640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 0x7f3ee4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:59:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:56.946+0000 7f3f04fc2640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 msgr2=0x7f3ee4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/3072370296 2024-06-29T15:59:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:56.946+0000 7f3f04fc2640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 0x7f3ee4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:59:57.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:57.987+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 157) ==== 97929+0+0 (secure 0 0 0) 0x7f3eec0950b0 con 0x7f3f000d2010 2024-06-29T15:59:57.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:57.988+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 msgr2=0x7f3ee4078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:57.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:57.988+0000 7f3efd7fa640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/3072370296,v1:172.21.15.153:6833/3072370296] conn(0x7f3ee4076470 0x7f3ee4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f9949560640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f9949560640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f9949560640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f9949560640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f9949560640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f994bd65640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f994bd65640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f994bd65640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f994bd65640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:58.138+0000 7f994bd65640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:58.990+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 158) ==== 98056+0+0 (secure 0 0 0) 0x7f3eec05c020 con 0x7f3f000d2010 2024-06-29T15:59:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:58.990+0000 7f3efd7fa640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f3ee403b050 0x7f3ee4080a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:58.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:58.990+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 --> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv01", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3eec01ee70 con 0x7f3ee403b050 2024-06-29T15:59:58.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:58.991+0000 7f3f04fc2640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f3ee403b050 0x7f3ee4080a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:58.992+0000 7f3f04fc2640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f3ee403b050 0x7f3ee4080a90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009620 tx=0x7f3ef4005f90 comp rx=0 tx=0).ready entity=mgr.10949 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:59.002 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:59.001+0000 7f9953574640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:59.002 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:59.001+0000 7f9953574640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:59.002 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:59.001+0000 7f9953574640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:59.002 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:59.001+0000 7f9953574640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:59.002 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T15:59:59.001+0000 7f9953574640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:59.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.109+0000 7f3efd7fa640 1 -- 172.21.15.59:0/1095392224 <== mgr.10949 v2:172.21.15.153:6832/2469779970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+82 (secure 0 0 0) 0x7f3eec01ee70 con 0x7f3ee403b050 2024-06-29T15:59:59.111 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule set for path /volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8 2024-06-29T15:59:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.111+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f3ee403b050 msgr2=0x7f3ee4080a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.111+0000 7f3f07226640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f3ee403b050 0x7f3ee4080a90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009620 tx=0x7f3ef4005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.111+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d2010 msgr2=0x7f3f0015ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.111+0000 7f3f07226640 1 --2- 172.21.15.59:0/1095392224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f000d2010 0x7f3f0015ede0 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00c3a0 tx=0x7f3eec00c860 comp rx=0 tx=0).stop 2024-06-29T15:59:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.111+0000 7f3f057c3640 1 -- 172.21.15.59:0/1095392224 reap_dead start 2024-06-29T15:59:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.112+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 shutdown_connections 2024-06-29T15:59:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.112+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 >> 172.21.15.59:0/1095392224 conn(0x7f3f00073e10 msgr2=0x7f3f000702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.112+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 shutdown_connections 2024-06-29T15:59:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T15:59:59.112+0000 7f3f07226640 1 -- 172.21.15.59:0/1095392224 wait complete. 2024-06-29T15:59:59.124 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8 2024-06-29T16:00:00.120 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:00:00.119+0000 7f99374fc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:00.120 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:00:00.119+0000 7f99374fc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:00.120 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:00:00.119+0000 7f99374fc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:00.120 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:00:00.119+0000 7f99374fc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:00.120 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:00:00.119+0000 7f99374fc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:13.793 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:00:13.796 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:00:43.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:00:43.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:14.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:14.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:14.125 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv01 2024-06-29T16:01:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.257+0000 7f976565b640 1 -- 172.21.15.59:0/2022548275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 msgr2=0x7f9748005d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.257+0000 7f976565b640 1 --2- 172.21.15.59:0/2022548275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 0x7f9748005d20 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f97540016c0 tx=0x7f9748006940 comp rx=0 tx=0).stop 2024-06-29T16:01:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.257+0000 7f976565b640 1 -- 172.21.15.59:0/2022548275 shutdown_connections 2024-06-29T16:01:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.257+0000 7f976565b640 1 --2- 172.21.15.59:0/2022548275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 0x7f9748005d20 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.257+0000 7f976565b640 1 -- 172.21.15.59:0/2022548275 >> 172.21.15.59:0/2022548275 conn(0x7f97600b24c0 msgr2=0x7f97600cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.257+0000 7f976565b640 1 -- 172.21.15.59:0/2022548275 shutdown_connections 2024-06-29T16:01:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.257+0000 7f976565b640 1 -- 172.21.15.59:0/2022548275 wait complete. 2024-06-29T16:01:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.259+0000 7f976565b640 1 Processor -- start 2024-06-29T16:01:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.259+0000 7f976565b640 1 -- start start 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f976565b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 0x7f97601f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f976565b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97601f3dc0 0x7f97600b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f976565b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97600a94a0 0x7f97600b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f976565b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97600cd050 con 0x7f9748005910 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f976565b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97600cced0 con 0x7f97601f3dc0 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f976565b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f97600cd1d0 con 0x7f97600a94a0 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 0x7f97601f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 0x7f97601f2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56498/0 (socket says 172.21.15.59:56498) 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 -- 172.21.15.59:0/27895306 learned_addr learned my addr 172.21.15.59:0/27895306 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975f7fe640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97600a94a0 0x7f97600b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975e7fc640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97601f3dc0 0x7f97600b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 -- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97600a94a0 msgr2=0x7f97600b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97600a94a0 0x7f97600b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 -- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97601f3dc0 msgr2=0x7f97600b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97601f3dc0 0x7f97600b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 -- 172.21.15.59:0/27895306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9760153430 con 0x7f9748005910 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f975effd640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 0x7f97601f2640 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f9750006340 tx=0x7f9750006800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f97477fe640 1 -- 172.21.15.59:0/27895306 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975000fd40 con 0x7f9748005910 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.261+0000 7f97477fe640 1 -- 172.21.15.59:0/27895306 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9750003e20 con 0x7f9748005910 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.262+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9760153710 con 0x7f9748005910 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.262+0000 7f97477fe640 1 -- 172.21.15.59:0/27895306 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750011670 con 0x7f9748005910 2024-06-29T16:01:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.262+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9760153cd0 con 0x7f9748005910 2024-06-29T16:01:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.263+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9760073670 con 0x7f9748005910 2024-06-29T16:01:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.263+0000 7f97477fe640 1 -- 172.21.15.59:0/27895306 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 160) ==== 98485+0+0 (secure 0 0 0) 0x7f9750011810 con 0x7f9748005910 2024-06-29T16:01:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.264+0000 7f97477fe640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f974c0765f0 0x7f974c078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.264+0000 7f97477fe640 1 -- 172.21.15.59:0/27895306 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(278..278 src has 1..278) ==== 6588+0+0 (secure 0 0 0) 0x7f9750098ac0 con 0x7f9748005910 2024-06-29T16:01:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.264+0000 7f975e7fc640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f974c0765f0 0x7f974c078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.265+0000 7f975e7fc640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f974c0765f0 0x7f974c078b00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9754032000 tx=0x7f975402c040 comp rx=0 tx=0).ready entity=mgr.10949 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.269+0000 7f97477fe640 1 -- 172.21.15.59:0/27895306 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9750015030 con 0x7f9748005910 2024-06-29T16:01:14.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.490+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 --> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7f976014f920 con 0x7f974c0765f0 2024-06-29T16:01:14.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.499+0000 7f97477fe640 1 -- 172.21.15.59:0/27895306 <== mgr.10949 v2:172.21.15.153:6832/2469779970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+60 (secure 0 0 0) 0x7f976014f920 con 0x7f974c0765f0 2024-06-29T16:01:14.499 INFO:teuthology.orchestra.run.smithi059.stdout:/volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8 2024-06-29T16:01:14.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f974c0765f0 msgr2=0x7f974c078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f974c0765f0 0x7f974c078b00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9754032000 tx=0x7f975402c040 comp rx=0 tx=0).stop 2024-06-29T16:01:14.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 msgr2=0x7f97601f2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 0x7f97601f2640 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f9750006340 tx=0x7f9750006800 comp rx=0 tx=0).stop 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 shutdown_connections 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f974c0765f0 0x7f974c078b00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97600a94a0 0x7f97600b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97601f3dc0 0x7f97600b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 --2- 172.21.15.59:0/27895306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748005910 0x7f97601f2640 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 >> 172.21.15.59:0/27895306 conn(0x7f97600b24c0 msgr2=0x7f976006d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 shutdown_connections 2024-06-29T16:01:14.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.501+0000 7f976565b640 1 -- 172.21.15.59:0/27895306 wait complete. 2024-06-29T16:01:14.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:14.513 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8/.snap) 2024-06-29T16:01:14.530 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-16_00_00_UTC 2024-06-29T16:01:14.530 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-16_01_00_UTC 2024-06-29T16:01:14.531 DEBUG:tasks.cephfs.test_snap_schedules:snaps:['scheduled-2024-06-29-16_00_00_UTC', 'scheduled-2024-06-29-16_01_00_UTC'] 2024-06-29T16:01:14.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --subvol sv01 . 1m 2024-06-29T16:01:14.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.694+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/3842496218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc5815ff00 msgr2=0x7fcc5815d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.694+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/3842496218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc5815ff00 0x7fcc5815d980 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7fcc50005fd0 tx=0x7fcc5002a5d0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.694+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/3842496218 shutdown_connections 2024-06-29T16:01:14.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.694+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/3842496218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc5815ff00 0x7fcc5815d980 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.694+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/3842496218 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc58145650 0x7fcc580b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.694+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/3842496218 >> 172.21.15.59:0/3842496218 conn(0x7fcc580b6ed0 msgr2=0x7fcc580b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.694+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/3842496218 shutdown_connections 2024-06-29T16:01:14.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.694+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/3842496218 wait complete. 2024-06-29T16:01:14.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.696+0000 7fcc5d8e4640 1 Processor -- start 2024-06-29T16:01:14.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.696+0000 7fcc5d8e4640 1 -- start start 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.697+0000 7fcc5d8e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc580d1130 0x7fcc58159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.697+0000 7fcc5d8e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc58145650 0x7fcc581f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.697+0000 7fcc5d8e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc581f2690 0x7fcc58070a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.697+0000 7fcc5d8e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5815e3e0 con 0x7fcc580d1130 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.697+0000 7fcc5d8e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5815e260 con 0x7fcc58145650 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.697+0000 7fcc5d8e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5815e560 con 0x7fcc581f2690 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc580d1130 0x7fcc58159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc580d1130 0x7fcc58159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56518/0 (socket says 172.21.15.59:56518) 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 -- 172.21.15.59:0/2805241867 learned_addr learned my addr 172.21.15.59:0/2805241867 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc577fe640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc581f2690 0x7fcc58070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc567fc640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc58145650 0x7fcc581f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 -- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc581f2690 msgr2=0x7fcc58070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc581f2690 0x7fcc58070a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 -- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc58145650 msgr2=0x7fcc581f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc58145650 0x7fcc581f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 -- 172.21.15.59:0/2805241867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc58138020 con 0x7fcc580d1130 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc56ffd640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc580d1130 0x7fcc58159c30 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800eb30 tx=0x7fcc4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc4ffff640 1 -- 172.21.15.59:0/2805241867 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48010430 con 0x7fcc580d1130 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc4ffff640 1 -- 172.21.15.59:0/2805241867 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc480105d0 con 0x7fcc580d1130 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc58138300 con 0x7fcc580d1130 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc4ffff640 1 -- 172.21.15.59:0/2805241867 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48010a20 con 0x7fcc580d1130 2024-06-29T16:01:14.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.698+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc581388c0 con 0x7fcc580d1130 2024-06-29T16:01:14.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.700+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc5815ea80 con 0x7fcc580d1130 2024-06-29T16:01:14.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.700+0000 7fcc4ffff640 1 -- 172.21.15.59:0/2805241867 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 160) ==== 98485+0+0 (secure 0 0 0) 0x7fcc48010bc0 con 0x7fcc580d1130 2024-06-29T16:01:14.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.701+0000 7fcc4ffff640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fcc380765f0 0x7fcc38078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.701+0000 7fcc4ffff640 1 -- 172.21.15.59:0/2805241867 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(278..278 src has 1..278) ==== 6588+0+0 (secure 0 0 0) 0x7fcc48098db0 con 0x7fcc580d1130 2024-06-29T16:01:14.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.701+0000 7fcc567fc640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fcc380765f0 0x7fcc38078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.702+0000 7fcc567fc640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fcc380765f0 0x7fcc38078b00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc5813c710 tx=0x7fcc40009040 comp rx=0 tx=0).ready entity=mgr.10949 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:14.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.705+0000 7fcc4ffff640 1 -- 172.21.15.59:0/2805241867 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc48061820 con 0x7fcc580d1130 2024-06-29T16:01:14.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.919+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 --> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "subvol": "sv01", "path": ".", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fcc580b7660 con 0x7fcc380765f0 2024-06-29T16:01:14.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.994+0000 7fcc4ffff640 1 -- 172.21.15.59:0/2805241867 <== mgr.10949 v2:172.21.15.153:6832/2469779970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+86 (secure 0 0 0) 0x7fcc580b7660 con 0x7fcc380765f0 2024-06-29T16:01:14.995 INFO:teuthology.orchestra.run.smithi059.stdout:Schedule removed for path /volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fcc380765f0 msgr2=0x7fcc38078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fcc380765f0 0x7fcc38078b00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc5813c710 tx=0x7fcc40009040 comp rx=0 tx=0).stop 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc580d1130 msgr2=0x7fcc58159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc580d1130 0x7fcc58159c30 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800eb30 tx=0x7fcc4800c2d0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 shutdown_connections 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fcc380765f0 0x7fcc38078b00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc581f2690 0x7fcc58070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc58145650 0x7fcc581f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 --2- 172.21.15.59:0/2805241867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc580d1130 0x7fcc58159c30 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.996+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 >> 172.21.15.59:0/2805241867 conn(0x7fcc580b6ed0 msgr2=0x7fcc58141140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.997+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 shutdown_connections 2024-06-29T16:01:14.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:14.997+0000 7fcc5d8e4640 1 -- 172.21.15.59:0/2805241867 wait complete. 2024-06-29T16:01:15.008 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8 2024-06-29T16:01:15.008 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv01 2024-06-29T16:01:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.128+0000 7fd6ff212640 1 -- 172.21.15.59:0/777995203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6f80d0120 msgr2=0x7fd6f80cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.128+0000 7fd6ff212640 1 --2- 172.21.15.59:0/777995203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6f80d0120 0x7fd6f80cd280 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009830 tx=0x7fd6e002a510 comp rx=0 tx=0).stop 2024-06-29T16:01:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.128+0000 7fd6ff212640 1 -- 172.21.15.59:0/777995203 shutdown_connections 2024-06-29T16:01:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.128+0000 7fd6ff212640 1 --2- 172.21.15.59:0/777995203 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6f80d0120 0x7fd6f80cd280 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.128+0000 7fd6ff212640 1 --2- 172.21.15.59:0/777995203 >> v2:172.21.15.153:3300/0 conn(0x7fd6f81448e0 0x7fd6f80cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.128+0000 7fd6ff212640 1 -- 172.21.15.59:0/777995203 >> 172.21.15.59:0/777995203 conn(0x7fd6f80b2ab0 msgr2=0x7fd6f80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.128+0000 7fd6ff212640 1 -- 172.21.15.59:0/777995203 shutdown_connections 2024-06-29T16:01:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.129+0000 7fd6ff212640 1 -- 172.21.15.59:0/777995203 wait complete. 2024-06-29T16:01:15.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.130+0000 7fd6ff212640 1 Processor -- start 2024-06-29T16:01:15.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.130+0000 7fd6ff212640 1 -- start start 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6ff212640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6f80cd7c0 0x7fd6f813c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6ff212640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6f80d0120 0x7fd6f80ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6ff212640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6f81448e0 0x7fd6f80addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6ff212640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f815fe30 con 0x7fd6f81448e0 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6ff212640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f806dc90 con 0x7fd6f80cd7c0 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6ff212640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f815ffb0 con 0x7fd6f80d0120 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6f81448e0 0x7fd6f80addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6f81448e0 0x7fd6f80addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56534/0 (socket says 172.21.15.59:56534) 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 -- 172.21.15.59:0/3211289826 learned_addr learned my addr 172.21.15.59:0/3211289826 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fcfae640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6f80cd7c0 0x7fd6f813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6f7fff640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6f80d0120 0x7fd6f80ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 -- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6f80d0120 msgr2=0x7fd6f80ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6f80d0120 0x7fd6f80ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 -- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6f80cd7c0 msgr2=0x7fd6f813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6f80cd7c0 0x7fd6f813c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 -- 172.21.15.59:0/3211289826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f80b3c30 con 0x7fd6f81448e0 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fcfae640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6f80cd7c0 0x7fd6f813c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.132+0000 7fd6fd7af640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6f81448e0 0x7fd6f80addf0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000cad0 tx=0x7fd6f000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.133+0000 7fd6f57fa640 1 -- 172.21.15.59:0/3211289826 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0016020 con 0x7fd6f81448e0 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.133+0000 7fd6f57fa640 1 -- 172.21.15.59:0/3211289826 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6f000ad20 con 0x7fd6f81448e0 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.133+0000 7fd6f57fa640 1 -- 172.21.15.59:0/3211289826 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0012680 con 0x7fd6f81448e0 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.133+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f80b3ed0 con 0x7fd6f81448e0 2024-06-29T16:01:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.133+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f80b4490 con 0x7fd6f81448e0 2024-06-29T16:01:15.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.134+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f81604d0 con 0x7fd6f81448e0 2024-06-29T16:01:15.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.134+0000 7fd6f57fa640 1 -- 172.21.15.59:0/3211289826 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 160) ==== 98485+0+0 (secure 0 0 0) 0x7fd6f0019020 con 0x7fd6f81448e0 2024-06-29T16:01:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.135+0000 7fd6f57fa640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fd6dc0765f0 0x7fd6dc078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.135+0000 7fd6f57fa640 1 -- 172.21.15.59:0/3211289826 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(278..278 src has 1..278) ==== 6588+0+0 (secure 0 0 0) 0x7fd6f0098540 con 0x7fd6f81448e0 2024-06-29T16:01:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.135+0000 7fd6fcfae640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fd6dc0765f0 0x7fd6dc078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.136+0000 7fd6fcfae640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fd6dc0765f0 0x7fd6dc078b00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80095f0 tx=0x7fd6e8005f90 comp rx=0 tx=0).ready entity=mgr.10949 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:15.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.139+0000 7fd6f57fa640 1 -- 172.21.15.59:0/3211289826 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd6f0020020 con 0x7fd6f81448e0 2024-06-29T16:01:15.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.359+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 --> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7fd6f8073a80 con 0x7fd6dc0765f0 2024-06-29T16:01:15.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.365+0000 7fd6f57fa640 1 -- 172.21.15.59:0/3211289826 <== mgr.10949 v2:172.21.15.153:6832/2469779970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+60 (secure 0 0 0) 0x7fd6f8073a80 con 0x7fd6dc0765f0 2024-06-29T16:01:15.365 INFO:teuthology.orchestra.run.smithi059.stdout:/volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fd6dc0765f0 msgr2=0x7fd6dc078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fd6dc0765f0 0x7fd6dc078b00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80095f0 tx=0x7fd6e8005f90 comp rx=0 tx=0).stop 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6f81448e0 msgr2=0x7fd6f80addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6f81448e0 0x7fd6f80addf0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000cad0 tx=0x7fd6f000cf90 comp rx=0 tx=0).stop 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 shutdown_connections 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fd6dc0765f0 0x7fd6dc078b00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6f81448e0 0x7fd6f80addf0 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6f80d0120 0x7fd6f80ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 --2- 172.21.15.59:0/3211289826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6f80cd7c0 0x7fd6f813c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 >> 172.21.15.59:0/3211289826 conn(0x7fd6f80b2ab0 msgr2=0x7fd6f80b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 shutdown_connections 2024-06-29T16:01:15.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.367+0000 7fd6ff212640 1 -- 172.21.15.59:0/3211289826 wait complete. 2024-06-29T16:01:15.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:15.378 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8/.snap) 2024-06-29T16:01:15.394 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-16_00_00_UTC 2024-06-29T16:01:15.397 INFO:teuthology.orchestra.run.smithi059.stdout:scheduled-2024-06-29-16_01_00_UTC 2024-06-29T16:01:15.398 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8/.snap/scheduled-2024-06-29-16_00_00_UTC 2024-06-29T16:01:15.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:15.399 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rmdir volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8/.snap/scheduled-2024-06-29-16_00_00_UTC) 2024-06-29T16:01:15.471 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8/.snap/scheduled-2024-06-29-16_01_00_UTC 2024-06-29T16:01:15.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:15.471 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rmdir volumes/_nogroup/sv01/7221c6e9-7f3e-45e7-aab2-a0fb33ef24b8/.snap/scheduled-2024-06-29-16_01_00_UTC) 2024-06-29T16:01:15.542 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01 2024-06-29T16:01:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.702+0000 7fefff32a640 1 -- 172.21.15.59:0/3784803396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feff815ceb0 msgr2=0x7feff80b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.702+0000 7fefff32a640 1 --2- 172.21.15.59:0/3784803396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feff815ceb0 0x7feff80b56d0 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7feff80b5600 tx=0x7feff002a4a0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.702+0000 7fefff32a640 1 -- 172.21.15.59:0/3784803396 shutdown_connections 2024-06-29T16:01:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.702+0000 7fefff32a640 1 --2- 172.21.15.59:0/3784803396 >> v2:172.21.15.153:3300/0 conn(0x7feff8141bd0 0x7feff813edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.702+0000 7fefff32a640 1 --2- 172.21.15.59:0/3784803396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feff815ceb0 0x7feff80b56d0 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.702+0000 7fefff32a640 1 -- 172.21.15.59:0/3784803396 >> 172.21.15.59:0/3784803396 conn(0x7feff80b6fb0 msgr2=0x7feff80b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.702+0000 7fefff32a640 1 -- 172.21.15.59:0/3784803396 shutdown_connections 2024-06-29T16:01:15.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.702+0000 7fefff32a640 1 -- 172.21.15.59:0/3784803396 wait complete. 2024-06-29T16:01:15.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.704+0000 7fefff32a640 1 Processor -- start 2024-06-29T16:01:15.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.704+0000 7fefff32a640 1 -- start start 2024-06-29T16:01:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7fefff32a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feff813e9b0 0x7feff814a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7fefff32a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff815ceb0 0x7feff813af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7fefff32a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feff81f5e00 0x7feff81f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7fefff32a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7feff80b29d0 con 0x7feff81f5e00 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7fefff32a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feff80b2850 con 0x7feff815ceb0 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7fefff32a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7feff80b2b50 con 0x7feff813e9b0 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7feffd8c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feff81f5e00 0x7feff81f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7feffc8c5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff815ceb0 0x7feff813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7feffd8c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feff81f5e00 0x7feff81f6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56548/0 (socket says 172.21.15.59:56548) 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7feffd8c7640 1 -- 172.21.15.59:0/1787267950 learned_addr learned my addr 172.21.15.59:0/1787267950 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.705+0000 7feffd0c6640 1 --2- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feff813e9b0 0x7feff814a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feffd8c7640 1 -- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feff813e9b0 msgr2=0x7feff814a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feffd8c7640 1 --2- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feff813e9b0 0x7feff814a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feffd8c7640 1 -- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff815ceb0 msgr2=0x7feff813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feffd8c7640 1 --2- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff815ceb0 0x7feff813af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feffd8c7640 1 -- 172.21.15.59:0/1787267950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff813b880 con 0x7feff81f5e00 2024-06-29T16:01:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feffc8c5640 1 --2- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feff815ceb0 0x7feff813af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feffd8c7640 1 --2- 172.21.15.59:0/1787267950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feff81f5e00 0x7feff81f6210 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7fefe800e9c0 tx=0x7fefe800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feff5ffb640 1 -- 172.21.15.59:0/1787267950 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8004d20 con 0x7feff81f5e00 2024-06-29T16:01:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feff5ffb640 1 -- 172.21.15.59:0/1787267950 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefe800cb20 con 0x7feff81f5e00 2024-06-29T16:01:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7fefff32a640 1 -- 172.21.15.59:0/1787267950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff81f6750 con 0x7feff81f5e00 2024-06-29T16:01:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7feff5ffb640 1 -- 172.21.15.59:0/1787267950 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8005020 con 0x7feff81f5e00 2024-06-29T16:01:15.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.706+0000 7fefff32a640 1 -- 172.21.15.59:0/1787267950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff8154410 con 0x7feff81f5e00 2024-06-29T16:01:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.707+0000 7fefff32a640 1 -- 172.21.15.59:0/1787267950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff80b3000 con 0x7feff81f5e00 2024-06-29T16:01:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.708+0000 7feff5ffb640 1 -- 172.21.15.59:0/1787267950 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 160) ==== 98485+0+0 (secure 0 0 0) 0x7fefe80051c0 con 0x7feff81f5e00 2024-06-29T16:01:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.709+0000 7feff5ffb640 1 --2- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fefe40765f0 0x7fefe4078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.709+0000 7feff5ffb640 1 -- 172.21.15.59:0/1787267950 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(278..278 src has 1..278) ==== 6588+0+0 (secure 0 0 0) 0x7fefe8098930 con 0x7feff81f5e00 2024-06-29T16:01:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.709+0000 7feffd0c6640 1 --2- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fefe40765f0 0x7fefe4078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.710+0000 7feffd0c6640 1 --2- 172.21.15.59:0/1787267950 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fefe40765f0 0x7fefe4078b00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feff0034000 tx=0x7feff0004730 comp rx=0 tx=0).ready entity=mgr.10949 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:15.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.713+0000 7feff5ffb640 1 -- 172.21.15.59:0/1787267950 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feff81f6750 con 0x7feff81f5e00 2024-06-29T16:01:15.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.937+0000 7fefff32a640 1 -- 172.21.15.59:0/1787267950 --> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7feff813c050 con 0x7fefe40765f0 2024-06-29T16:01:15.955 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:15.954+0000 7f994655a640 -1 mgr.server reply reply (22) Invalid argument error in rename /volumes/_nogroup/sv01 to /volumes/_nogroup/sv01/.trash/c3e555fe-f51b-43ad-aac6-b43d9c72bb0b 2024-06-29T16:01:15.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:15.955+0000 7feff5ffb640 1 -- 172.21.15.59:0/1787267950 <== mgr.10949 v2:172.21.15.153:6832/2469779970 1 ==== mgr_command_reply(tid 0: -22 error in rename /volumes/_nogroup/sv01 to /volumes/_nogroup/sv01/.trash/c3e555fe-f51b-43ad-aac6-b43d9c72bb0b) ==== 116+0+0 (secure 0 0 0) 0x7feff813c050 con 0x7fefe40765f0 2024-06-29T16:01:15.956 INFO:teuthology.orchestra.run.smithi059.stderr:Error EINVAL: error in rename /volumes/_nogroup/sv01 to /volumes/_nogroup/sv01/.trash/c3e555fe-f51b-43ad-aac6-b43d9c72bb0b 2024-06-29T16:01:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-29T16:01:15.961 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T16:01:16.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.082+0000 7f92a6297640 1 -- 172.21.15.59:0/805085409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a0145610 msgr2=0x7f92a0073e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.082+0000 7f92a6297640 1 --2- 172.21.15.59:0/805085409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a0145610 0x7f92a0073e80 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f9290009830 tx=0x7f929002a5c0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.082+0000 7f92a6297640 1 -- 172.21.15.59:0/805085409 shutdown_connections 2024-06-29T16:01:16.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.082+0000 7f92a6297640 1 --2- 172.21.15.59:0/805085409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a0145610 0x7f92a0073e80 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.082+0000 7f92a6297640 1 -- 172.21.15.59:0/805085409 >> 172.21.15.59:0/805085409 conn(0x7f92a00b24c0 msgr2=0x7f92a00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.082+0000 7f92a6297640 1 -- 172.21.15.59:0/805085409 shutdown_connections 2024-06-29T16:01:16.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.082+0000 7f92a6297640 1 -- 172.21.15.59:0/805085409 wait complete. 2024-06-29T16:01:16.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.084+0000 7f92a6297640 1 Processor -- start 2024-06-29T16:01:16.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.084+0000 7f92a6297640 1 -- start start 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.085+0000 7f92a6297640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a00cccc0 0x7f92a00b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.085+0000 7f92a6297640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92a0145610 0x7f92a015ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.085+0000 7f92a6297640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92a015c320 0x7f92a01f8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.085+0000 7f92a6297640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f92a00ceb40 con 0x7f92a00cccc0 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.085+0000 7f92a6297640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92a00ce9c0 con 0x7f92a0145610 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.085+0000 7f92a6297640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f92a00cecc0 con 0x7f92a015c320 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.085+0000 7f929ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a00cccc0 0x7f92a00b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a00cccc0 0x7f92a00b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56566/0 (socket says 172.21.15.59:56566) 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929ffff640 1 -- 172.21.15.59:0/4093327140 learned_addr learned my addr 172.21.15.59:0/4093327140 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:16.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929f7fe640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92a0145610 0x7f92a015ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f92a4834640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92a015c320 0x7f92a01f8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929ffff640 1 -- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92a015c320 msgr2=0x7f92a01f8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929ffff640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92a015c320 0x7f92a01f8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929ffff640 1 -- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92a0145610 msgr2=0x7f92a015ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929ffff640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92a0145610 0x7f92a015ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929ffff640 1 -- 172.21.15.59:0/4093327140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a015bf50 con 0x7f92a00cccc0 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f92a4834640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92a015c320 0x7f92a01f8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929ffff640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a00cccc0 0x7f92a00b3d40 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f9290009920 tx=0x7f9290033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929cff9640 1 -- 172.21.15.59:0/4093327140 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290031d60 con 0x7f92a00cccc0 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929cff9640 1 -- 172.21.15.59:0/4093327140 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9290005020 con 0x7f92a00cccc0 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f929cff9640 1 -- 172.21.15.59:0/4093327140 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290005470 con 0x7f92a00cccc0 2024-06-29T16:01:16.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a00b46b0 con 0x7f92a00cccc0 2024-06-29T16:01:16.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.086+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a013e010 con 0x7f92a00cccc0 2024-06-29T16:01:16.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.087+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a0073780 con 0x7f92a00cccc0 2024-06-29T16:01:16.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.088+0000 7f929cff9640 1 -- 172.21.15.59:0/4093327140 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 160) ==== 98485+0+0 (secure 0 0 0) 0x7f92900047a0 con 0x7f92a00cccc0 2024-06-29T16:01:16.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.088+0000 7f929cff9640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f928c076620 0x7f928c078b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.089+0000 7f929cff9640 1 -- 172.21.15.59:0/4093327140 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(278..278 src has 1..278) ==== 6588+0+0 (secure 0 0 0) 0x7f92900b9130 con 0x7f92a00cccc0 2024-06-29T16:01:16.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.089+0000 7f929f7fe640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f928c076620 0x7f928c078b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.089+0000 7f929f7fe640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f928c076620 0x7f928c078b30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f92a00b2d20 tx=0x7f9294009040 comp rx=0 tx=0).ready entity=mgr.10949 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:16.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.093+0000 7f929cff9640 1 -- 172.21.15.59:0/4093327140 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f92a0073780 con 0x7f92a00cccc0 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.335+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f92a00cf190 con 0x7f92a00cccc0 2024-06-29T16:01:16.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.416+0000 7f929cff9640 1 -- 172.21.15.59:0/4093327140 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v161) ==== 93+0+0 (secure 0 0 0) 0x7f92a015bf50 con 0x7f92a00cccc0 2024-06-29T16:01:16.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f928c076620 msgr2=0x7f928c078b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f928c076620 0x7f928c078b30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f92a00b2d20 tx=0x7f9294009040 comp rx=0 tx=0).stop 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a00cccc0 msgr2=0x7f92a00b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a00cccc0 0x7f92a00b3d40 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f9290009920 tx=0x7f9290033ce0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 shutdown_connections 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f928c076620 0x7f928c078b30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92a015c320 0x7f92a01f8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92a0145610 0x7f92a015ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 --2- 172.21.15.59:0/4093327140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92a00cccc0 0x7f92a00b3d40 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.419+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 >> 172.21.15.59:0/4093327140 conn(0x7f92a00b24c0 msgr2=0x7f92a006c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.420+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 shutdown_connections 2024-06-29T16:01:16.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.420+0000 7f92a6297640 1 -- 172.21.15.59:0/4093327140 wait complete. 2024-06-29T16:01:16.432 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T16:01:16.433 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T16:01:16.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T16:01:16.530 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T16:01:16.530 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T16:01:16.530 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T16:01:16.531 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:16.547 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T16:01:16.547 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:16.603 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:16.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.762+0000 7fdd3c247640 1 -- 172.21.15.59:0/3419045934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340d23c0 msgr2=0x7fdd340b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.762+0000 7fdd3c247640 1 --2- 172.21.15.59:0/3419045934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340d23c0 0x7fdd340b3990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009860 tx=0x7fdd2802a5a0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.763+0000 7fdd3c247640 1 -- 172.21.15.59:0/3419045934 shutdown_connections 2024-06-29T16:01:16.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.763+0000 7fdd3c247640 1 --2- 172.21.15.59:0/3419045934 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340d23c0 0x7fdd340b3990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.763+0000 7fdd3c247640 1 -- 172.21.15.59:0/3419045934 >> 172.21.15.59:0/3419045934 conn(0x7fdd340d3ee0 msgr2=0x7fdd340d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.763+0000 7fdd3c247640 1 -- 172.21.15.59:0/3419045934 shutdown_connections 2024-06-29T16:01:16.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.763+0000 7fdd3c247640 1 -- 172.21.15.59:0/3419045934 wait complete. 2024-06-29T16:01:16.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.764+0000 7fdd3c247640 1 Processor -- start 2024-06-29T16:01:16.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.765+0000 7fdd3c247640 1 -- start start 2024-06-29T16:01:16.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd3c247640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340b3ed0 0x7fdd3414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd3c247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd340d23c0 0x7fdd34150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd3c247640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd3415efe0 0x7fdd34151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd3c247640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34160450 con 0x7fdd340d23c0 2024-06-29T16:01:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd3c247640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd341602d0 con 0x7fdd3415efe0 2024-06-29T16:01:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd3c247640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd341605d0 con 0x7fdd340b3ed0 2024-06-29T16:01:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd397e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd340d23c0 0x7fdd34150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd397e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd340d23c0 0x7fdd34150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56582/0 (socket says 172.21.15.59:56582) 2024-06-29T16:01:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd39fe3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340b3ed0 0x7fdd3414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd397e2640 1 -- 172.21.15.59:0/2728272867 learned_addr learned my addr 172.21.15.59:0/2728272867 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:16.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd3a7e4640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd3415efe0 0x7fdd34151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd39fe3640 1 -- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd3415efe0 msgr2=0x7fdd34151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd39fe3640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd3415efe0 0x7fdd34151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd39fe3640 1 -- 172.21.15.59:0/2728272867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd340d23c0 msgr2=0x7fdd34150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd39fe3640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd340d23c0 0x7fdd34150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.766+0000 7fdd39fe3640 1 -- 172.21.15.59:0/2728272867 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3414dab0 con 0x7fdd340b3ed0 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.767+0000 7fdd39fe3640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340b3ed0 0x7fdd3414d060 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdd28031000 tx=0x7fdd28033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.767+0000 7fdd267fc640 1 -- 172.21.15.59:0/2728272867 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28031b50 con 0x7fdd340b3ed0 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.767+0000 7fdd267fc640 1 -- 172.21.15.59:0/2728272867 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd28031de0 con 0x7fdd340b3ed0 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.767+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd34151680 con 0x7fdd340b3ed0 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.767+0000 7fdd267fc640 1 -- 172.21.15.59:0/2728272867 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28005280 con 0x7fdd340b3ed0 2024-06-29T16:01:16.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.767+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd34157370 con 0x7fdd340b3ed0 2024-06-29T16:01:16.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.768+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd340b3990 con 0x7fdd340b3ed0 2024-06-29T16:01:16.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.769+0000 7fdd267fc640 1 -- 172.21.15.59:0/2728272867 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7fdd28005420 con 0x7fdd340b3ed0 2024-06-29T16:01:16.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.769+0000 7fdd267fc640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fdd180765a0 0x7fdd18078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.769+0000 7fdd267fc640 1 -- 172.21.15.59:0/2728272867 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(278..278 src has 1..278) ==== 6588+0+0 (secure 0 0 0) 0x7fdd280b9500 con 0x7fdd340b3ed0 2024-06-29T16:01:16.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.769+0000 7fdd397e2640 1 -- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fdd180765a0 msgr2=0x7fdd18078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:16.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.770+0000 7fdd397e2640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fdd180765a0 0x7fdd18078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:16.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.773+0000 7fdd267fc640 1 -- 172.21.15.59:0/2728272867 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd2803f030 con 0x7fdd340b3ed0 2024-06-29T16:01:16.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.965+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd340d2950 con 0x7fdd340b3ed0 2024-06-29T16:01:16.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.966+0000 7fdd267fc640 1 -- 172.21.15.59:0/2728272867 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v278) ==== 74+0+17648 (secure 0 0 0) 0x7fdd28031770 con 0x7fdd340b3ed0 2024-06-29T16:01:16.967 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:16.967 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":278,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T16:00:59.316462+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:44.104733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:44.643605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"275","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.0","mappings":[{"from":7,"to":2}]},{"pgid":"38.12","mappings":[{"from":7,"to":4}]},{"pgid":"38.14","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2302272425":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1710208131":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6833/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6832/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1857217640":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/3489900589":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1124076092":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1986019092":"2024-06-30T15:59:57.978964+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fdd180765a0 msgr2=0x7fdd18078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fdd180765a0 0x7fdd18078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340b3ed0 msgr2=0x7fdd3414d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340b3ed0 0x7fdd3414d060 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdd28031000 tx=0x7fdd28033ba0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 shutdown_connections 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fdd180765a0 0x7fdd18078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd3415efe0 0x7fdd34151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd340d23c0 0x7fdd34150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 --2- 172.21.15.59:0/2728272867 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd340b3ed0 0x7fdd3414d060 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 >> 172.21.15.59:0/2728272867 conn(0x7fdd340d3ee0 msgr2=0x7fdd340d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 shutdown_connections 2024-06-29T16:01:16.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:16.969+0000 7fdd3c247640 1 -- 172.21.15.59:0/2728272867 wait complete. 2024-06-29T16:01:16.981 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:59:44.104733+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:59:44.643605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '275', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T16:01:16.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T16:01:17.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.104+0000 7f600bd7f640 1 -- 172.21.15.59:0/1767624974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60040d23c0 msgr2=0x7f60040b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.104+0000 7f600bd7f640 1 --2- 172.21.15.59:0/1767624974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60040d23c0 0x7f60040b3990 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7f600408c4f0 tx=0x7f600002a530 comp rx=0 tx=0).stop 2024-06-29T16:01:17.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.104+0000 7f600bd7f640 1 -- 172.21.15.59:0/1767624974 shutdown_connections 2024-06-29T16:01:17.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.104+0000 7f600bd7f640 1 --2- 172.21.15.59:0/1767624974 >> v2:172.21.15.153:3300/0 conn(0x7f60040b3ed0 0x7f600415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.104+0000 7f600bd7f640 1 --2- 172.21.15.59:0/1767624974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60040d23c0 0x7f60040b3990 unknown :-1 s=CLOSED pgs=2959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.104+0000 7f600bd7f640 1 -- 172.21.15.59:0/1767624974 >> 172.21.15.59:0/1767624974 conn(0x7f60040d3ee0 msgr2=0x7f60040d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.104+0000 7f600bd7f640 1 -- 172.21.15.59:0/1767624974 shutdown_connections 2024-06-29T16:01:17.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.104+0000 7f600bd7f640 1 -- 172.21.15.59:0/1767624974 wait complete. 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.106+0000 7f600bd7f640 1 Processor -- start 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.106+0000 7f600bd7f640 1 -- start start 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.107+0000 7f600bd7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60040b3ed0 0x7f60041f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.107+0000 7f600bd7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60040d23c0 0x7f6004156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.107+0000 7f600bd7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f600415ee10 0x7f60041f9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.107+0000 7f600bd7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f6004160240 con 0x7f600415ee10 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.107+0000 7f600bd7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60041600c0 con 0x7f60040d23c0 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.107+0000 7f600bd7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f60041603c0 con 0x7f60040b3ed0 2024-06-29T16:01:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f600415ee10 0x7f60041f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f6009b1b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60040b3ed0 0x7f60041f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f600415ee10 0x7f60041f9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56604/0 (socket says 172.21.15.59:56604) 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 -- 172.21.15.59:0/951773371 learned_addr learned my addr 172.21.15.59:0/951773371 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f6009b1b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60040b3ed0 0x7f60041f64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:33784/0 (socket says 172.21.15.59:33784) 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600931a640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60040d23c0 0x7f6004156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 -- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60040b3ed0 msgr2=0x7f60041f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60040b3ed0 0x7f60041f64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 -- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60040d23c0 msgr2=0x7f6004156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60040d23c0 0x7f6004156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 -- 172.21.15.59:0/951773371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6004158eb0 con 0x7f600415ee10 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600a31c640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f600415ee10 0x7f60041f9040 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00c760 tx=0x7f5ffc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f5ff67fc640 1 -- 172.21.15.59:0/951773371 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc003b70 con 0x7f600415ee10 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f5ff67fc640 1 -- 172.21.15.59:0/951773371 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ffc003d30 con 0x7f600415ee10 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600415dbc0 con 0x7f600415ee10 2024-06-29T16:01:17.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f5ff67fc640 1 -- 172.21.15.59:0/951773371 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0052d0 con 0x7f600415ee10 2024-06-29T16:01:17.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.108+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f600415e180 con 0x7f600415ee10 2024-06-29T16:01:17.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.109+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60040ceb60 con 0x7f600415ee10 2024-06-29T16:01:17.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.111+0000 7f5ff67fc640 1 -- 172.21.15.59:0/951773371 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7f5ffc005470 con 0x7f600415ee10 2024-06-29T16:01:17.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.111+0000 7f5ff67fc640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f5fe80765d0 0x7f5fe8078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.112+0000 7f5ff67fc640 1 -- 172.21.15.59:0/951773371 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(278..278 src has 1..278) ==== 6588+0+0 (secure 0 0 0) 0x7f5ffc097b40 con 0x7f600415ee10 2024-06-29T16:01:17.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.112+0000 7f6009b1b640 1 -- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f5fe80765d0 msgr2=0x7f5fe8078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:17.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.112+0000 7f6009b1b640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f5fe80765d0 0x7f5fe8078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:17.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.116+0000 7f5ff67fc640 1 -- 172.21.15.59:0/951773371 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f60040ceb60 con 0x7f600415ee10 2024-06-29T16:01:17.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.308+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6004159080 con 0x7f600415ee10 2024-06-29T16:01:17.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.308+0000 7f5ff67fc640 1 -- 172.21.15.59:0/951773371 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v278) ==== 93+0+10 (secure 0 0 0) 0x7f60040b42e0 con 0x7f600415ee10 2024-06-29T16:01:17.308 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T16:01:17.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f5fe80765d0 msgr2=0x7f5fe8078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:17.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f5fe80765d0 0x7f5fe8078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f600415ee10 msgr2=0x7f60041f9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f600415ee10 0x7f60041f9040 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00c760 tx=0x7f5ffc00cc20 comp rx=0 tx=0).stop 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 shutdown_connections 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f5fe80765d0 0x7f5fe8078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f600415ee10 0x7f60041f9040 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60040d23c0 0x7f6004156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 --2- 172.21.15.59:0/951773371 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f60040b3ed0 0x7f60041f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 >> 172.21.15.59:0/951773371 conn(0x7f60040d3ee0 msgr2=0x7f600406c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 shutdown_connections 2024-06-29T16:01:17.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.310+0000 7f600bd7f640 1 -- 172.21.15.59:0/951773371 wait complete. 2024-06-29T16:01:17.322 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T16:01:17.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.451+0000 7fc4c038c640 1 -- 172.21.15.59:0/3561804971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 msgr2=0x7fc4b815ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.451+0000 7fc4c038c640 1 --2- 172.21.15.59:0/3561804971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 0x7fc4b815ac50 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b402a5d0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.452+0000 7fc4c038c640 1 -- 172.21.15.59:0/3561804971 shutdown_connections 2024-06-29T16:01:17.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.452+0000 7fc4c038c640 1 --2- 172.21.15.59:0/3561804971 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4b80cf080 0x7fc4b806d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.452+0000 7fc4c038c640 1 --2- 172.21.15.59:0/3561804971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 0x7fc4b815ac50 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.452+0000 7fc4c038c640 1 -- 172.21.15.59:0/3561804971 >> 172.21.15.59:0/3561804971 conn(0x7fc4b806f8d0 msgr2=0x7fc4b806fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.452+0000 7fc4c038c640 1 -- 172.21.15.59:0/3561804971 shutdown_connections 2024-06-29T16:01:17.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.452+0000 7fc4c038c640 1 -- 172.21.15.59:0/3561804971 wait complete. 2024-06-29T16:01:17.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.453+0000 7fc4c038c640 1 Processor -- start 2024-06-29T16:01:17.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.454+0000 7fc4c038c640 1 -- start start 2024-06-29T16:01:17.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4c038c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 0x7fc4b8156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4c038c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4b80b5020 0x7fc4b81521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4c038c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4b8152940 0x7fc4b81f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4c038c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b806dba0 con 0x7fc4b80b3a20 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4c038c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b806da20 con 0x7fc4b80b5020 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4c038c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc4b806dd20 con 0x7fc4b8152940 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be128640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 0x7fc4b8156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be128640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 0x7fc4b8156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56622/0 (socket says 172.21.15.59:56622) 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be128640 1 -- 172.21.15.59:0/1376670417 learned_addr learned my addr 172.21.15.59:0/1376670417 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4bd927640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4b80b5020 0x7fc4b81521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be929640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4b8152940 0x7fc4b81f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be128640 1 -- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4b8152940 msgr2=0x7fc4b81f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be128640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4b8152940 0x7fc4b81f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be128640 1 -- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4b80b5020 msgr2=0x7fc4b81521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be128640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4b80b5020 0x7fc4b81521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4be128640 1 -- 172.21.15.59:0/1376670417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b81f5ec0 con 0x7fc4b80b3a20 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.455+0000 7fc4bd927640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4b80b5020 0x7fc4b81521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.456+0000 7fc4be128640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 0x7fc4b8156610 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4000c00 tx=0x7fc4b40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.456+0000 7fc4aeffd640 1 -- 172.21.15.59:0/1376670417 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b403e030 con 0x7fc4b80b3a20 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.456+0000 7fc4aeffd640 1 -- 172.21.15.59:0/1376670417 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4b4003ae0 con 0x7fc4b80b3a20 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.456+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b80ae280 con 0x7fc4b80b3a20 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.456+0000 7fc4aeffd640 1 -- 172.21.15.59:0/1376670417 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4035040 con 0x7fc4b80b3a20 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.456+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b80ae800 con 0x7fc4b80b3a20 2024-06-29T16:01:17.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.457+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b806e1d0 con 0x7fc4b80b3a20 2024-06-29T16:01:17.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.458+0000 7fc4aeffd640 1 -- 172.21.15.59:0/1376670417 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7fc4b4041030 con 0x7fc4b80b3a20 2024-06-29T16:01:17.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.458+0000 7fc4aeffd640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fc49c0765a0 0x7fc49c078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.458+0000 7fc4bd927640 1 -- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fc49c0765a0 msgr2=0x7fc49c078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:17.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.458+0000 7fc4aeffd640 1 -- 172.21.15.59:0/1376670417 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 6644+0+0 (secure 0 0 0) 0x7fc4b40b8410 con 0x7fc4b80b3a20 2024-06-29T16:01:17.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.458+0000 7fc4bd927640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fc49c0765a0 0x7fc49c078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:17.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.463+0000 7fc4aeffd640 1 -- 172.21.15.59:0/1376670417 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc4b4080fa0 con 0x7fc4b80b3a20 2024-06-29T16:01:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.659+0000 7fc4bd927640 1 -- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fc49c0765a0 msgr2=0x7fc49c078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:17.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.659+0000 7fc4bd927640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fc49c0765a0 0x7fc49c078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T16:01:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.662+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc4b80b17a0 con 0x7fc4b80b3a20 2024-06-29T16:01:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.663+0000 7fc4aeffd640 1 -- 172.21.15.59:0/1376670417 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v279) ==== 104+0+11 (secure 0 0 0) 0x7fc4b40852a0 con 0x7fc4b80b3a20 2024-06-29T16:01:17.663 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T16:01:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fc49c0765a0 msgr2=0x7fc49c078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fc49c0765a0 0x7fc49c078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 msgr2=0x7fc4b8156610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 0x7fc4b8156610 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4000c00 tx=0x7fc4b40035f0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 shutdown_connections 2024-06-29T16:01:17.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7fc49c0765a0 0x7fc49c078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc4b8152940 0x7fc4b81f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc4b80b5020 0x7fc4b81521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 --2- 172.21.15.59:0/1376670417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4b80b3a20 0x7fc4b8156610 unknown :-1 s=CLOSED pgs=2962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 >> 172.21.15.59:0/1376670417 conn(0x7fc4b806f8d0 msgr2=0x7fc4b8158690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 shutdown_connections 2024-06-29T16:01:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.665+0000 7fc4c038c640 1 -- 172.21.15.59:0/1376670417 wait complete. 2024-06-29T16:01:17.677 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T16:01:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.800+0000 7f9777295640 1 -- 172.21.15.59:0/1111781903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700d1a60 msgr2=0x7f97700b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.800+0000 7f9777295640 1 --2- 172.21.15.59:0/1111781903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700d1a60 0x7f97700b5b30 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f9768005fd0 tx=0x7f976802aa90 comp rx=0 tx=0).stop 2024-06-29T16:01:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.800+0000 7f9777295640 1 -- 172.21.15.59:0/1111781903 shutdown_connections 2024-06-29T16:01:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.800+0000 7f9777295640 1 --2- 172.21.15.59:0/1111781903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700d1a60 0x7f97700b5b30 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.800+0000 7f9777295640 1 --2- 172.21.15.59:0/1111781903 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97700b2fa0 0x7f97700d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.800+0000 7f9777295640 1 -- 172.21.15.59:0/1111781903 >> 172.21.15.59:0/1111781903 conn(0x7f9770073e10 msgr2=0x7f9770074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.800+0000 7f9777295640 1 -- 172.21.15.59:0/1111781903 shutdown_connections 2024-06-29T16:01:17.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.800+0000 7f9777295640 1 -- 172.21.15.59:0/1111781903 wait complete. 2024-06-29T16:01:17.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.802+0000 7f9777295640 1 Processor -- start 2024-06-29T16:01:17.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.802+0000 7f9777295640 1 -- start start 2024-06-29T16:01:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9777295640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97700b2270 0x7f9770151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9777295640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700b2fa0 0x7f97701f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9777295640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97700d1a60 0x7f977015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9777295640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f97700b6400 con 0x7f97700b2fa0 2024-06-29T16:01:17.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9777295640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97700b6280 con 0x7f97700d1a60 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9777295640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f97700b6580 con 0x7f97700b2270 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9774830640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700b2fa0 0x7f97701f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9775031640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97700b2270 0x7f9770151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9774830640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700b2fa0 0x7f97701f81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56638/0 (socket says 172.21.15.59:56638) 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9774830640 1 -- 172.21.15.59:0/1011183390 learned_addr learned my addr 172.21.15.59:0/1011183390 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.803+0000 7f9775832640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97700d1a60 0x7f977015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9774830640 1 -- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97700b2270 msgr2=0x7f9770151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9774830640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97700b2270 0x7f9770151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9774830640 1 -- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97700d1a60 msgr2=0x7f977015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9774830640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97700d1a60 0x7f977015a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9774830640 1 -- 172.21.15.59:0/1011183390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97701f8e60 con 0x7f97700b2fa0 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9775031640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97700b2270 0x7f9770151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:17.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9775832640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97700d1a60 0x7f977015a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9774830640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700b2fa0 0x7f97701f81a0 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f976000e7c0 tx=0x7f976000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9765ffb640 1 -- 172.21.15.59:0/1011183390 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976000ccc0 con 0x7f97700b2fa0 2024-06-29T16:01:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9765ffb640 1 -- 172.21.15.59:0/1011183390 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9760010430 con 0x7f97700b2fa0 2024-06-29T16:01:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9765ffb640 1 -- 172.21.15.59:0/1011183390 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760010860 con 0x7f97700b2fa0 2024-06-29T16:01:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97701f4dc0 con 0x7f97700b2fa0 2024-06-29T16:01:17.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.804+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97701f5380 con 0x7f97700b2fa0 2024-06-29T16:01:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.805+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97701f4bb0 con 0x7f97700b2fa0 2024-06-29T16:01:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.806+0000 7f9765ffb640 1 -- 172.21.15.59:0/1011183390 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7f9760010a00 con 0x7f97700b2fa0 2024-06-29T16:01:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.807+0000 7f9765ffb640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f97540765a0 0x7f9754078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.807+0000 7f9765ffb640 1 -- 172.21.15.59:0/1011183390 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 6644+0+0 (secure 0 0 0) 0x7f9760098970 con 0x7f97700b2fa0 2024-06-29T16:01:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.807+0000 7f9775031640 1 -- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f97540765a0 msgr2=0x7f9754078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:17.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.807+0000 7f9775031640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f97540765a0 0x7f9754078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:17.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:17.811+0000 7f9765ffb640 1 -- 172.21.15.59:0/1011183390 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9760061470 con 0x7f97700b2fa0 2024-06-29T16:01:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.003+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f97700b6aa0 con 0x7f97700b2fa0 2024-06-29T16:01:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.004+0000 7f9765ffb640 1 -- 172.21.15.59:0/1011183390 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v279) ==== 100+0+11 (secure 0 0 0) 0x7f9760065770 con 0x7f97700b2fa0 2024-06-29T16:01:18.005 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T16:01:18.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f97540765a0 msgr2=0x7f9754078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f97540765a0 0x7f9754078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700b2fa0 msgr2=0x7f97701f81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700b2fa0 0x7f97701f81a0 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f976000e7c0 tx=0x7f976000ec80 comp rx=0 tx=0).stop 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 shutdown_connections 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f97540765a0 0x7f9754078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97700d1a60 0x7f977015a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97700b2fa0 0x7f97701f81a0 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 --2- 172.21.15.59:0/1011183390 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f97700b2270 0x7f9770151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 >> 172.21.15.59:0/1011183390 conn(0x7f9770073e10 msgr2=0x7f9770074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 shutdown_connections 2024-06-29T16:01:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.006+0000 7f9777295640 1 -- 172.21.15.59:0/1011183390 wait complete. 2024-06-29T16:01:18.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T16:01:18.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.138+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3208277153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 msgr2=0x7f0f180d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.138+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3208277153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 0x7f0f180d1520 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f0f180cf310 tx=0x7f0f0c02a530 comp rx=0 tx=0).stop 2024-06-29T16:01:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.138+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3208277153 shutdown_connections 2024-06-29T16:01:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.138+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3208277153 >> v2:172.21.15.153:3300/0 conn(0x7f0f180d1a60 0x7f0f180b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.138+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3208277153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 0x7f0f180d1520 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.138+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3208277153 >> 172.21.15.59:0/3208277153 conn(0x7f0f18073e10 msgr2=0x7f0f18074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.138+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3208277153 shutdown_connections 2024-06-29T16:01:18.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.138+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3208277153 wait complete. 2024-06-29T16:01:18.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.140+0000 7f0f1e7c7640 1 Processor -- start 2024-06-29T16:01:18.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.140+0000 7f0f1e7c7640 1 -- start start 2024-06-29T16:01:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f1e7c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f180b2270 0x7f0f181f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f1e7c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 0x7f0f1815c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f1e7c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f180d1a60 0x7f0f1815e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f1e7c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0f180b6400 con 0x7f0f180b2fa0 2024-06-29T16:01:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f1e7c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f180b6280 con 0x7f0f180d1a60 2024-06-29T16:01:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f1e7c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0f180b6580 con 0x7f0f180b2270 2024-06-29T16:01:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 0x7f0f1815c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 0x7f0f1815c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56656/0 (socket says 172.21.15.59:56656) 2024-06-29T16:01:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f177fe640 1 -- 172.21.15.59:0/3471864625 learned_addr learned my addr 172.21.15.59:0/3471864625 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f1cd64640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f180d1a60 0x7f0f1815e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.141+0000 7f0f17fff640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f180b2270 0x7f0f181f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f177fe640 1 -- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f180b2270 msgr2=0x7f0f181f5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f177fe640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f180b2270 0x7f0f181f5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f177fe640 1 -- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f180d1a60 msgr2=0x7f0f1815e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f177fe640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f180d1a60 0x7f0f1815e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f177fe640 1 -- 172.21.15.59:0/3471864625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1815eee0 con 0x7f0f180b2fa0 2024-06-29T16:01:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f177fe640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 0x7f0f1815c400 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c02ab20 tx=0x7f0f0c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f14ff9640 1 -- 172.21.15.59:0/3471864625 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c005c90 con 0x7f0f180b2fa0 2024-06-29T16:01:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f14ff9640 1 -- 172.21.15.59:0/3471864625 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f0c031e50 con 0x7f0f180b2fa0 2024-06-29T16:01:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f14ff9640 1 -- 172.21.15.59:0/3471864625 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c005260 con 0x7f0f180b2fa0 2024-06-29T16:01:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1815cfc0 con 0x7f0f180b2fa0 2024-06-29T16:01:18.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.142+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f1815d580 con 0x7f0f180b2fa0 2024-06-29T16:01:18.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.143+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f180b4fe0 con 0x7f0f180b2fa0 2024-06-29T16:01:18.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.144+0000 7f0f14ff9640 1 -- 172.21.15.59:0/3471864625 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7f0f0c005400 con 0x7f0f180b2fa0 2024-06-29T16:01:18.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.144+0000 7f0f14ff9640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f0f040765a0 0x7f0f04078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.145+0000 7f0f14ff9640 1 -- 172.21.15.59:0/3471864625 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 6644+0+0 (secure 0 0 0) 0x7f0f0c0b93d0 con 0x7f0f180b2fa0 2024-06-29T16:01:18.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.145+0000 7f0f17fff640 1 -- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f0f040765a0 msgr2=0x7f0f04078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:18.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.145+0000 7f0f17fff640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f0f040765a0 0x7f0f04078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:18.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.148+0000 7f0f14ff9640 1 -- 172.21.15.59:0/3471864625 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f180b4fe0 con 0x7f0f180b2fa0 2024-06-29T16:01:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.346+0000 7f0f17fff640 1 -- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f0f040765a0 msgr2=0x7f0f04078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.346+0000 7f0f17fff640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f0f040765a0 0x7f0f04078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T16:01:18.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.352+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0f180b2d60 con 0x7f0f180b2fa0 2024-06-29T16:01:18.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.353+0000 7f0f14ff9640 1 -- 172.21.15.59:0/3471864625 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v279) ==== 80+0+478 (secure 0 0 0) 0x7f0f180b2d60 con 0x7f0f180b2fa0 2024-06-29T16:01:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/2302272425 2024-06-30T15:59:57.978964+0000 2024-06-29T16:01:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1710208131 2024-06-30T15:59:57.978964+0000 2024-06-29T16:01:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6833/3072370296 2024-06-30T15:59:57.978964+0000 2024-06-29T16:01:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:6832/3072370296 2024-06-30T15:59:57.978964+0000 2024-06-29T16:01:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1857217640 2024-06-30T15:59:57.978964+0000 2024-06-29T16:01:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/3489900589 2024-06-30T15:59:57.978964+0000 2024-06-29T16:01:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1124076092 2024-06-30T15:59:57.978964+0000 2024-06-29T16:01:18.354 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.153:0/1986019092 2024-06-30T15:59:57.978964+0000 2024-06-29T16:01:18.355 INFO:teuthology.orchestra.run.smithi059.stderr:listed 8 entries 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f0f040765a0 msgr2=0x7f0f04078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f0f040765a0 0x7f0f04078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 msgr2=0x7f0f1815c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 0x7f0f1815c400 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c02ab20 tx=0x7f0f0c033c20 comp rx=0 tx=0).stop 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 shutdown_connections 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f0f040765a0 0x7f0f04078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f180d1a60 0x7f0f1815e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f180b2fa0 0x7f0f1815c400 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 --2- 172.21.15.59:0/3471864625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0f180b2270 0x7f0f181f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 >> 172.21.15.59:0/3471864625 conn(0x7f0f18073e10 msgr2=0x7f0f180d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 shutdown_connections 2024-06-29T16:01:18.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.355+0000 7f0f1e7c7640 1 -- 172.21.15.59:0/3471864625 wait complete. 2024-06-29T16:01:18.367 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T16:01:18.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:18.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T16:01:18.411 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T16:01:18.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:18.411 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T16:01:18.470 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T16:01:18.470 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T16:01:18.470 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:18.487 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T16:01:18.488 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T16:01:18.488 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T16:01:18.488 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:18.528 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T16:01:18.528 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T16:01:18.528 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T16:01:18.528 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:18.544 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T16:01:18.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:18.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.687+0000 7f9f8b79a640 1 -- 172.21.15.59:0/580828868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f70004800 msgr2=0x7f9f70004cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.687+0000 7f9f8b79a640 1 --2- 172.21.15.59:0/580828868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f70004800 0x7f9f70004cb0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f9f840b55e0 tx=0x7f9f70005830 comp rx=0 tx=0).stop 2024-06-29T16:01:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.687+0000 7f9f8b79a640 1 -- 172.21.15.59:0/580828868 shutdown_connections 2024-06-29T16:01:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.687+0000 7f9f8b79a640 1 --2- 172.21.15.59:0/580828868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f70004800 0x7f9f70004cb0 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.687+0000 7f9f8b79a640 1 -- 172.21.15.59:0/580828868 >> 172.21.15.59:0/580828868 conn(0x7f9f840d3ee0 msgr2=0x7f9f840d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.687+0000 7f9f8b79a640 1 -- 172.21.15.59:0/580828868 shutdown_connections 2024-06-29T16:01:18.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.687+0000 7f9f8b79a640 1 -- 172.21.15.59:0/580828868 wait complete. 2024-06-29T16:01:18.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.689+0000 7f9f8b79a640 1 Processor -- start 2024-06-29T16:01:18.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.689+0000 7f9f8b79a640 1 -- start start 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f8b79a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f70004800 0x7f9f84158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f8b79a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f840cd770 0x7f9f8414c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f8b79a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f8414cd50 0x7f9f840d0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f8b79a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9f840b4240 con 0x7f9f840cd770 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f8b79a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f840b40c0 con 0x7f9f70004800 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f8b79a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9f840ce110 con 0x7f9f8414cd50 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f88d35640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f840cd770 0x7f9f8414c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f88d35640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f840cd770 0x7f9f8414c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56670/0 (socket says 172.21.15.59:56670) 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f88d35640 1 -- 172.21.15.59:0/835936372 learned_addr learned my addr 172.21.15.59:0/835936372 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:18.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f89536640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f70004800 0x7f9f84158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.690+0000 7f9f89d37640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f8414cd50 0x7f9f840d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f89536640 1 -- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f8414cd50 msgr2=0x7f9f840d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f89536640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f8414cd50 0x7f9f840d0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f89536640 1 -- 172.21.15.59:0/835936372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f840cd770 msgr2=0x7f9f8414c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f89536640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f840cd770 0x7f9f8414c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f89536640 1 -- 172.21.15.59:0/835936372 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f84150ba0 con 0x7f9f70004800 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f88d35640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f840cd770 0x7f9f8414c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f89536640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f70004800 0x7f9f84158610 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f9f80001eb0 tx=0x7f9f8000ea20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f79ffb640 1 -- 172.21.15.59:0/835936372 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80015020 con 0x7f9f70004800 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f79ffb640 1 -- 172.21.15.59:0/835936372 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f80011960 con 0x7f9f70004800 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f84150e80 con 0x7f9f70004800 2024-06-29T16:01:18.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f79ffb640 1 -- 172.21.15.59:0/835936372 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80011db0 con 0x7f9f70004800 2024-06-29T16:01:18.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.691+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f84151440 con 0x7f9f70004800 2024-06-29T16:01:18.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.693+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f840ce670 con 0x7f9f70004800 2024-06-29T16:01:18.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.693+0000 7f9f79ffb640 1 -- 172.21.15.59:0/835936372 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7f9f80018020 con 0x7f9f70004800 2024-06-29T16:01:18.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.693+0000 7f9f79ffb640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f9f680765d0 0x7f9f68078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.694+0000 7f9f88d35640 1 -- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f9f680765d0 msgr2=0x7f9f68078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:18.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.694+0000 7f9f79ffb640 1 -- 172.21.15.59:0/835936372 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 6644+0+0 (secure 0 0 0) 0x7f9f80098590 con 0x7f9f70004800 2024-06-29T16:01:18.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.694+0000 7f9f88d35640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f9f680765d0 0x7f9f68078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.698+0000 7f9f79ffb640 1 -- 172.21.15.59:0/835936372 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f8001f020 con 0x7f9f70004800 2024-06-29T16:01:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.895+0000 7f9f88d35640 1 -- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f9f680765d0 msgr2=0x7f9f68078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.895+0000 7f9f88d35640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f9f680765d0 0x7f9f68078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T16:01:18.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.932+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f840b9a40 con 0x7f9f70004800 2024-06-29T16:01:18.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.932+0000 7f9f79ffb640 1 -- 172.21.15.59:0/835936372 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+5244 (secure 0 0 0) 0x7f9f80061010 con 0x7f9f70004800 2024-06-29T16:01:18.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:18.933 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":157,"btime":"2024-06-29T15:59:49:317195+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1039110220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1039110220},{"type":"v1","addr":"172.21.15.153:6835","nonce":1039110220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":12583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3996608135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3996608135},{"type":"v1","addr":"172.21.15.153:6837","nonce":3996608135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":13620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/952472867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":952472867},{"type":"v1","addr":"172.21.15.59:6834","nonce":952472867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:45.699384+0000","modified":"2024-06-29T15:59:48.709634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13623":{"gid":13623,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3706652401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3706652401},{"type":"v1","addr":"172.21.15.59:6835","nonce":3706652401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13623,"qdb_cluster":[13623]},"id":18}]} 2024-06-29T16:01:18.934 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 157 2024-06-29T16:01:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f9f680765d0 msgr2=0x7f9f68078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f9f680765d0 0x7f9f68078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f70004800 msgr2=0x7f9f84158610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f70004800 0x7f9f84158610 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f9f80001eb0 tx=0x7f9f8000ea20 comp rx=0 tx=0).stop 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 shutdown_connections 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f9f680765d0 0x7f9f68078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9f8414cd50 0x7f9f840d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f840cd770 0x7f9f8414c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 --2- 172.21.15.59:0/835936372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f70004800 0x7f9f84158610 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 >> 172.21.15.59:0/835936372 conn(0x7f9f840d3ee0 msgr2=0x7f9f841564d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 shutdown_connections 2024-06-29T16:01:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:18.936+0000 7f9f8b79a640 1 -- 172.21.15.59:0/835936372 wait complete. 2024-06-29T16:01:18.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:19.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.070+0000 7efcf0370640 1 -- 172.21.15.59:0/2664813187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce8141bd0 msgr2=0x7efce813edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.070+0000 7efcf0370640 1 --2- 172.21.15.59:0/2664813187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce8141bd0 0x7efce813edc0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7efce0005fd0 tx=0x7efce002a8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.070+0000 7efcf0370640 1 -- 172.21.15.59:0/2664813187 shutdown_connections 2024-06-29T16:01:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.070+0000 7efcf0370640 1 --2- 172.21.15.59:0/2664813187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce8141bd0 0x7efce813edc0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.070+0000 7efcf0370640 1 -- 172.21.15.59:0/2664813187 >> 172.21.15.59:0/2664813187 conn(0x7efce80b6fb0 msgr2=0x7efce80b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.070+0000 7efcf0370640 1 -- 172.21.15.59:0/2664813187 shutdown_connections 2024-06-29T16:01:19.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.070+0000 7efcf0370640 1 -- 172.21.15.59:0/2664813187 wait complete. 2024-06-29T16:01:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.072+0000 7efcf0370640 1 Processor -- start 2024-06-29T16:01:19.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.072+0000 7efcf0370640 1 -- start start 2024-06-29T16:01:19.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcf0370640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efce813e9b0 0x7efce815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcf0370640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce815ceb0 0x7efce8138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcf0370640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efce8158cb0 0x7efce81558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcf0370640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efce80b2900 con 0x7efce8158cb0 2024-06-29T16:01:19.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcf0370640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efce80b2780 con 0x7efce815ceb0 2024-06-29T16:01:19.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcf0370640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efce80b2a80 con 0x7efce813e9b0 2024-06-29T16:01:19.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efce8158cb0 0x7efce81558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efce8158cb0 0x7efce81558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56684/0 (socket says 172.21.15.59:56684) 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 -- 172.21.15.59:0/995296563 learned_addr learned my addr 172.21.15.59:0/995296563 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee10c640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efce813e9b0 0x7efce815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efced90b640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce815ceb0 0x7efce8138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 -- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efce813e9b0 msgr2=0x7efce815a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efce813e9b0 0x7efce815a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 -- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce815ceb0 msgr2=0x7efce8138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce815ceb0 0x7efce8138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 -- 172.21.15.59:0/995296563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce815ae10 con 0x7efce8158cb0 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.074+0000 7efcee90d640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efce8158cb0 0x7efce81558d0 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7efce0009cd0 tx=0x7efce0031670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.075+0000 7efcdeffd640 1 -- 172.21.15.59:0/995296563 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce0003e00 con 0x7efce8158cb0 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.075+0000 7efcdeffd640 1 -- 172.21.15.59:0/995296563 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efce0031a80 con 0x7efce8158cb0 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.075+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce815b0f0 con 0x7efce8158cb0 2024-06-29T16:01:19.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.075+0000 7efcdeffd640 1 -- 172.21.15.59:0/995296563 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce00330d0 con 0x7efce8158cb0 2024-06-29T16:01:19.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.075+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce815b6b0 con 0x7efce8158cb0 2024-06-29T16:01:19.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.076+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce80b2f30 con 0x7efce8158cb0 2024-06-29T16:01:19.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.077+0000 7efcdeffd640 1 -- 172.21.15.59:0/995296563 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7efce0033270 con 0x7efce8158cb0 2024-06-29T16:01:19.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.077+0000 7efcdeffd640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7efccc0765a0 0x7efccc078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.077+0000 7efcee10c640 1 -- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7efccc0765a0 msgr2=0x7efccc078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:19.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.077+0000 7efcdeffd640 1 -- 172.21.15.59:0/995296563 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 6644+0+0 (secure 0 0 0) 0x7efce00b9350 con 0x7efce8158cb0 2024-06-29T16:01:19.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.077+0000 7efcee10c640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7efccc0765a0 0x7efccc078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:19.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.082+0000 7efcdeffd640 1 -- 172.21.15.59:0/995296563 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efce003f030 con 0x7efce8158cb0 2024-06-29T16:01:19.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.276+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efce8138870 con 0x7efce8158cb0 2024-06-29T16:01:19.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.277+0000 7efcdeffd640 1 -- 172.21.15.59:0/995296563 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v279) ==== 74+0+17760 (secure 0 0 0) 0x7efce0081dd0 con 0x7efce8158cb0 2024-06-29T16:01:19.278 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:19.278 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":279,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T16:01:17.416556+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:44.104733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:44.643605+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":279,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.0","mappings":[{"from":7,"to":2}]},{"pgid":"38.12","mappings":[{"from":7,"to":4}]},{"pgid":"38.14","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2302272425":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1710208131":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6833/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6832/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1857217640":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/3489900589":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1124076092":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1986019092":"2024-06-30T15:59:57.978964+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":38,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":38,"snaps":[{"begin":2,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:19.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcee10c640 1 -- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7efccc0765a0 msgr2=0x7efccc078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:19.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcee10c640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7efccc0765a0 0x7efccc078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T16:01:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7efccc0765a0 msgr2=0x7efccc078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7efccc0765a0 0x7efccc078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efce8158cb0 msgr2=0x7efce81558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efce8158cb0 0x7efce81558d0 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7efce0009cd0 tx=0x7efce0031670 comp rx=0 tx=0).stop 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 shutdown_connections 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7efccc0765a0 0x7efccc078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efce8158cb0 0x7efce81558d0 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efce815ceb0 0x7efce8138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 --2- 172.21.15.59:0/995296563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efce813e9b0 0x7efce815a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 >> 172.21.15.59:0/995296563 conn(0x7efce80b6fb0 msgr2=0x7efce813dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 shutdown_connections 2024-06-29T16:01:19.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.279+0000 7efcf0370640 1 -- 172.21.15.59:0/995296563 wait complete. 2024-06-29T16:01:19.295 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:21:41.747059+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:59:44.104733+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:59:44.643605+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 279, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T16:01:19.295 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T16:01:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.416+0000 7f871e6e4640 1 -- 172.21.15.59:0/1258075737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815ee10 msgr2=0x7f871815f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.416+0000 7f871e6e4640 1 --2- 172.21.15.59:0/1258075737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815ee10 0x7f871815f9b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f8710005fd0 tx=0x7f871002a8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.416+0000 7f871e6e4640 1 -- 172.21.15.59:0/1258075737 shutdown_connections 2024-06-29T16:01:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.416+0000 7f871e6e4640 1 --2- 172.21.15.59:0/1258075737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815ee10 0x7f871815f9b0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.416+0000 7f871e6e4640 1 --2- 172.21.15.59:0/1258075737 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87180b40a0 0x7f871815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.416+0000 7f871e6e4640 1 -- 172.21.15.59:0/1258075737 >> 172.21.15.59:0/1258075737 conn(0x7f87180d3ee0 msgr2=0x7f87180d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.416+0000 7f871e6e4640 1 -- 172.21.15.59:0/1258075737 shutdown_connections 2024-06-29T16:01:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.416+0000 7f871e6e4640 1 -- 172.21.15.59:0/1258075737 wait complete. 2024-06-29T16:01:19.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.418+0000 7f871e6e4640 1 Processor -- start 2024-06-29T16:01:19.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.418+0000 7f871e6e4640 1 -- start start 2024-06-29T16:01:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f871e6e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87180d23c0 0x7f87181f5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f871e6e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871815ee10 0x7f87180cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f871e6e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815dda0 0x7f871815b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f871e6e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8718160280 con 0x7f87180d23c0 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f871e6e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8718160100 con 0x7f871815dda0 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f871e6e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8718160400 con 0x7f871815ee10 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f8717fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87180d23c0 0x7f87181f5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f8717fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87180d23c0 0x7f87181f5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56696/0 (socket says 172.21.15.59:56696) 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f8717fff640 1 -- 172.21.15.59:0/574578702 learned_addr learned my addr 172.21.15.59:0/574578702 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f871cc81640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815dda0 0x7f871815b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f87177fe640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871815ee10 0x7f87180cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f8717fff640 1 -- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871815ee10 msgr2=0x7f87180cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f8717fff640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871815ee10 0x7f87180cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f8717fff640 1 -- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815dda0 msgr2=0x7f871815b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f8717fff640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815dda0 0x7f871815b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f8717fff640 1 -- 172.21.15.59:0/574578702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87180cc6e0 con 0x7f87180d23c0 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.420+0000 7f871cc81640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815dda0 0x7f871815b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:19.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.421+0000 7f8717fff640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87180d23c0 0x7f87181f5670 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f870800c400 tx=0x7f870800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.421+0000 7f8714ff9640 1 -- 172.21.15.59:0/574578702 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708011790 con 0x7f87180d23c0 2024-06-29T16:01:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.421+0000 7f8714ff9640 1 -- 172.21.15.59:0/574578702 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8708017070 con 0x7f87180d23c0 2024-06-29T16:01:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.421+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871815e480 con 0x7f87180d23c0 2024-06-29T16:01:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.421+0000 7f8714ff9640 1 -- 172.21.15.59:0/574578702 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708011cc0 con 0x7f87180d23c0 2024-06-29T16:01:19.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.421+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87180cb9c0 con 0x7f87180d23c0 2024-06-29T16:01:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.422+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871815e8d0 con 0x7f87180d23c0 2024-06-29T16:01:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.423+0000 7f8714ff9640 1 -- 172.21.15.59:0/574578702 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7f8708005020 con 0x7f87180d23c0 2024-06-29T16:01:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.423+0000 7f8714ff9640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f87040765a0 0x7f8704078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.423+0000 7f8714ff9640 1 -- 172.21.15.59:0/574578702 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 6644+0+0 (secure 0 0 0) 0x7f8708098640 con 0x7f87180d23c0 2024-06-29T16:01:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.424+0000 7f87177fe640 1 -- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f87040765a0 msgr2=0x7f8704078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.424+0000 7f87177fe640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f87040765a0 0x7f8704078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:19.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.427+0000 7f8714ff9640 1 -- 172.21.15.59:0/574578702 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8708061410 con 0x7f87180d23c0 2024-06-29T16:01:19.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.619+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f87180d2950 con 0x7f87180d23c0 2024-06-29T16:01:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.620+0000 7f8714ff9640 1 -- 172.21.15.59:0/574578702 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v279) ==== 93+0+10 (secure 0 0 0) 0x7f8708065710 con 0x7f87180d23c0 2024-06-29T16:01:19.621 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-06-29T16:01:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f87040765a0 msgr2=0x7f8704078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f87040765a0 0x7f8704078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87180d23c0 msgr2=0x7f87181f5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87180d23c0 0x7f87181f5670 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f870800c400 tx=0x7f870800c8c0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 shutdown_connections 2024-06-29T16:01:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f87040765a0 0x7f8704078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f871815dda0 0x7f871815b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f871815ee10 0x7f87180cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 --2- 172.21.15.59:0/574578702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87180d23c0 0x7f87181f5670 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 >> 172.21.15.59:0/574578702 conn(0x7f87180d3ee0 msgr2=0x7f871806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 shutdown_connections 2024-06-29T16:01:19.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.622+0000 7f871e6e4640 1 -- 172.21.15.59:0/574578702 wait complete. 2024-06-29T16:01:19.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 -- 172.21.15.59:0/4169452380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04300d1b10 msgr2=0x7f043015e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 --2- 172.21.15.59:0/4169452380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04300d1b10 0x7f043015e590 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f04300735e0 tx=0x7f041c02a560 comp rx=0 tx=0).stop 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 -- 172.21.15.59:0/4169452380 shutdown_connections 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 --2- 172.21.15.59:0/4169452380 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f043015ead0 0x7f0430161aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 --2- 172.21.15.59:0/4169452380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04300d1b10 0x7f043015e590 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 --2- 172.21.15.59:0/4169452380 >> v2:172.21.15.153:3300/0 conn(0x7f04300d0dc0 0x7f04300d15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 -- 172.21.15.59:0/4169452380 >> 172.21.15.59:0/4169452380 conn(0x7f0430073e10 msgr2=0x7f0430074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 -- 172.21.15.59:0/4169452380 shutdown_connections 2024-06-29T16:01:19.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.755+0000 7f0436dac640 1 -- 172.21.15.59:0/4169452380 wait complete. 2024-06-29T16:01:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.757+0000 7f0436dac640 1 Processor -- start 2024-06-29T16:01:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.757+0000 7f0436dac640 1 -- start start 2024-06-29T16:01:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0436dac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04300d0dc0 0x7f04300cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0436dac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04300d1b10 0x7f04301f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0436dac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043015ead0 0x7f043006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0436dac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0430162160 con 0x7f043015ead0 2024-06-29T16:01:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0436dac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0430161fe0 con 0x7f04300d0dc0 2024-06-29T16:01:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0436dac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f04301622e0 con 0x7f04300d1b10 2024-06-29T16:01:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0435349640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043015ead0 0x7f043006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0434b48640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04300d0dc0 0x7f04300cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0435349640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043015ead0 0x7f043006fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56714/0 (socket says 172.21.15.59:56714) 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0435349640 1 -- 172.21.15.59:0/1883478366 learned_addr learned my addr 172.21.15.59:0/1883478366 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f042ffff640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04300d1b10 0x7f04301f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0434b48640 1 -- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04300d1b10 msgr2=0x7f04301f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0434b48640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04300d1b10 0x7f04301f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0434b48640 1 -- 172.21.15.59:0/1883478366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043015ead0 msgr2=0x7f043006fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0434b48640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043015ead0 0x7f043006fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0434b48640 1 -- 172.21.15.59:0/1883478366 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04301e88a0 con 0x7f04300d0dc0 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.759+0000 7f0435349640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043015ead0 0x7f043006fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.760+0000 7f0434b48640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04300d0dc0 0x7f04300cb580 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f042400e6d0 tx=0x7f042400eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.760+0000 7f042d7fa640 1 -- 172.21.15.59:0/1883478366 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042400cab0 con 0x7f04300d0dc0 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.760+0000 7f042d7fa640 1 -- 172.21.15.59:0/1883478366 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f042400cd40 con 0x7f04300d0dc0 2024-06-29T16:01:19.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.760+0000 7f042d7fa640 1 -- 172.21.15.59:0/1883478366 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424010690 con 0x7f04300d0dc0 2024-06-29T16:01:19.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.760+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04301e7e60 con 0x7f04300d0dc0 2024-06-29T16:01:19.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.760+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04301e85c0 con 0x7f04300d0dc0 2024-06-29T16:01:19.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.761+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0430162830 con 0x7f04300d0dc0 2024-06-29T16:01:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.762+0000 7f042d7fa640 1 -- 172.21.15.59:0/1883478366 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 161) ==== 98468+0+0 (secure 0 0 0) 0x7f0424010830 con 0x7f04300d0dc0 2024-06-29T16:01:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.762+0000 7f042d7fa640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f04140765a0 0x7f0414078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.762+0000 7f042d7fa640 1 -- 172.21.15.59:0/1883478366 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(279..279 src has 1..279) ==== 6644+0+0 (secure 0 0 0) 0x7f0424098930 con 0x7f04300d0dc0 2024-06-29T16:01:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.762+0000 7f042ffff640 1 -- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f04140765a0 msgr2=0x7f0414078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6832/2469779970 2024-06-29T16:01:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.762+0000 7f042ffff640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f04140765a0 0x7f0414078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:19.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.767+0000 7f042d7fa640 1 -- 172.21.15.59:0/1883478366 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f04240613a0 con 0x7f04300d0dc0 2024-06-29T16:01:19.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.958+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f04300d18f0 con 0x7f04300d0dc0 2024-06-29T16:01:19.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.958+0000 7f042d7fa640 1 -- 172.21.15.59:0/1883478366 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v279) ==== 104+0+11 (secure 0 0 0) 0x7f04240656a0 con 0x7f04300d0dc0 2024-06-29T16:01:19.959 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f04140765a0 msgr2=0x7f0414078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f04140765a0 0x7f0414078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04300d0dc0 msgr2=0x7f04300cb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04300d0dc0 0x7f04300cb580 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f042400e6d0 tx=0x7f042400eb90 comp rx=0 tx=0).stop 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 shutdown_connections 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:6832/2469779970,v1:172.21.15.153:6833/2469779970] conn(0x7f04140765a0 0x7f0414078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f043015ead0 0x7f043006fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f04300d1b10 0x7f04301f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 --2- 172.21.15.59:0/1883478366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04300d0dc0 0x7f04300cb580 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 >> 172.21.15.59:0/1883478366 conn(0x7f0430073e10 msgr2=0x7f043015d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 shutdown_connections 2024-06-29T16:01:19.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:19.960+0000 7f0436dac640 1 -- 172.21.15.59:0/1883478366 wait complete. 2024-06-29T16:01:19.973 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T16:01:20.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.093+0000 7ff54323c640 1 -- 172.21.15.59:0/1220960345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0d0130 msgr2=0x7ff53c143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.093+0000 7ff54323c640 1 --2- 172.21.15.59:0/1220960345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0d0130 0x7ff53c143840 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff530005fd0 tx=0x7ff53002a5d0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.093+0000 7ff54323c640 1 -- 172.21.15.59:0/1220960345 shutdown_connections 2024-06-29T16:01:20.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.093+0000 7ff54323c640 1 --2- 172.21.15.59:0/1220960345 >> v2:172.21.15.153:3300/0 conn(0x7ff53c0b4ee0 0x7ff53c06e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.093+0000 7ff54323c640 1 --2- 172.21.15.59:0/1220960345 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0d0130 0x7ff53c143840 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.093+0000 7ff54323c640 1 -- 172.21.15.59:0/1220960345 >> 172.21.15.59:0/1220960345 conn(0x7ff53c0b2ab0 msgr2=0x7ff53c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:20.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.093+0000 7ff54323c640 1 -- 172.21.15.59:0/1220960345 shutdown_connections 2024-06-29T16:01:20.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.093+0000 7ff54323c640 1 -- 172.21.15.59:0/1220960345 wait complete. 2024-06-29T16:01:20.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.096+0000 7ff54323c640 1 Processor -- start 2024-06-29T16:01:20.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.096+0000 7ff54323c640 1 -- start start 2024-06-29T16:01:20.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.099+0000 7ff54323c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c06ea10 0x7ff53c13eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.099+0000 7ff54323c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0b4ee0 0x7ff53c06d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.099+0000 7ff54323c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff53c0d0130 0x7ff53c13fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.099+0000 7ff54323c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c15ed80 con 0x7ff53c0d0130 2024-06-29T16:01:20.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.099+0000 7ff54323c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c06f030 con 0x7ff53c06ea10 2024-06-29T16:01:20.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.099+0000 7ff54323c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff53c15ef00 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.100+0000 7ff53bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0b4ee0 0x7ff53c06d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.100+0000 7ff53bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0b4ee0 0x7ff53c06d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.59:33922/0 (socket says 172.21.15.59:33922) 2024-06-29T16:01:20.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.100+0000 7ff53bfff640 1 -- 172.21.15.59:0/1297655563 learned_addr learned my addr 172.21.15.59:0/1297655563 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:20.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.100+0000 7ff540fd8640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c06ea10 0x7ff53c13eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.100+0000 7ff5417d9640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff53c0d0130 0x7ff53c13fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.101+0000 7ff53bfff640 1 -- 172.21.15.59:0/1297655563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c06ea10 msgr2=0x7ff53c13eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.101+0000 7ff53bfff640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c06ea10 0x7ff53c13eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.101+0000 7ff53bfff640 1 -- 172.21.15.59:0/1297655563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff53c0d0130 msgr2=0x7ff53c13fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.101+0000 7ff53bfff640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff53c0d0130 0x7ff53c13fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.101+0000 7ff53bfff640 1 -- 172.21.15.59:0/1297655563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c1f8d60 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.101+0000 7ff5417d9640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff53c0d0130 0x7ff53c13fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:20.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.102+0000 7ff53bfff640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0b4ee0 0x7ff53c06d340 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff52800e9f0 tx=0x7ff52800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.105+0000 7ff5397fa640 1 -- 172.21.15.59:0/1297655563 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52800cd80 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.105+0000 7ff5397fa640 1 -- 172.21.15.59:0/1297655563 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff528004020 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.105+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53c1f9260 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.105+0000 7ff5397fa640 1 -- 172.21.15.59:0/1297655563 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528004450 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.105+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff53c1f9820 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.106+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53c1f8fd0 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.107+0000 7ff5397fa640 1 -- 172.21.15.59:0/1297655563 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 162) ==== 97912+0+0 (secure 0 0 0) 0x7ff5280035f0 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.107+0000 7ff5397fa640 1 -- 172.21.15.59:0/1297655563 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(280..280 src has 1..280) ==== 7089+0+0 (secure 0 0 0) 0x7ff528097fe0 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.112+0000 7ff5397fa640 1 -- 172.21.15.59:0/1297655563 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff528061820 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.160 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97c5cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97c5cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97c5cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97c5cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97c5cc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97f5d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97f5d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97f5d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97f5d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.161 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-06-29T16:01:20.159+0000 7fd97f5d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:20.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.304+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff53c15f3a0 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.305+0000 7ff5397fa640 1 -- 172.21.15.59:0/1297655563 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v280) ==== 100+0+11 (secure 0 0 0) 0x7ff528061120 con 0x7ff53c0b4ee0 2024-06-29T16:01:20.305 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0b4ee0 msgr2=0x7ff53c06d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0b4ee0 0x7ff53c06d340 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff52800e9f0 tx=0x7ff52800eeb0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 shutdown_connections 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff53c0d0130 0x7ff53c13fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff53c0b4ee0 0x7ff53c06d340 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 --2- 172.21.15.59:0/1297655563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff53c06ea10 0x7ff53c13eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 >> 172.21.15.59:0/1297655563 conn(0x7ff53c0b2ab0 msgr2=0x7ff53c0b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 shutdown_connections 2024-06-29T16:01:20.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.307+0000 7ff54323c640 1 -- 172.21.15.59:0/1297655563 wait complete. 2024-06-29T16:01:20.319 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:20.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.437+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/4025651974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 msgr2=0x7fd9ac0b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.437+0000 7fd9b1bfb640 1 --2- 172.21.15.59:0/4025651974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 0x7fd9ac0b5b30 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4005fd0 tx=0x7fd9a402aa90 comp rx=0 tx=0).stop 2024-06-29T16:01:20.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.437+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/4025651974 shutdown_connections 2024-06-29T16:01:20.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.437+0000 7fd9b1bfb640 1 --2- 172.21.15.59:0/4025651974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 0x7fd9ac0b5b30 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.437+0000 7fd9b1bfb640 1 --2- 172.21.15.59:0/4025651974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac0b2fa0 0x7fd9ac0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.437+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/4025651974 >> 172.21.15.59:0/4025651974 conn(0x7fd9ac073e10 msgr2=0x7fd9ac074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:20.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.437+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/4025651974 shutdown_connections 2024-06-29T16:01:20.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.437+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/4025651974 wait complete. 2024-06-29T16:01:20.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.439+0000 7fd9b1bfb640 1 Processor -- start 2024-06-29T16:01:20.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.439+0000 7fd9b1bfb640 1 -- start start 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9b1bfb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0b2270 0x7fd9ac146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9b1bfb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac0b2fa0 0x7fd9ac1537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9b1bfb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 0x7fd9ac153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9b1bfb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac0b6400 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9b1bfb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac0b6280 con 0x7fd9ac0b2270 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9b1bfb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd9ac0b6580 con 0x7fd9ac0b2fa0 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 0x7fd9ac153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 0x7fd9ac153d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56736/0 (socket says 172.21.15.59:56736) 2024-06-29T16:01:20.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 -- 172.21.15.59:0/3896606831 learned_addr learned my addr 172.21.15.59:0/3896606831 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9aaffd640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac0b2fa0 0x7fd9ac1537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9ab7fe640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0b2270 0x7fd9ac146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 -- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac0b2fa0 msgr2=0x7fd9ac1537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac0b2fa0 0x7fd9ac1537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 -- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0b2270 msgr2=0x7fd9ac146360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0b2270 0x7fd9ac146360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 -- 172.21.15.59:0/3896606831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac15e790 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9ab7fe640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0b2270 0x7fd9ac146360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9aaffd640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac0b2fa0 0x7fd9ac1537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9abfff640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 0x7fd9ac153d20 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009bf0 tx=0x7fd9a4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd993fff640 1 -- 172.21.15.59:0/3896606831 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4044070 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd993fff640 1 -- 172.21.15.59:0/3896606831 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9a4004040 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ac15ea70 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd993fff640 1 -- 172.21.15.59:0/3896606831 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4004450 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.441+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ac152f50 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.442+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9ac0b4fe0 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.443+0000 7fd993fff640 1 -- 172.21.15.59:0/3896606831 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 162) ==== 97912+0+0 (secure 0 0 0) 0x7fd9a4047070 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.444+0000 7fd993fff640 1 -- 172.21.15.59:0/3896606831 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(280..280 src has 1..280) ==== 7089+0+0 (secure 0 0 0) 0x7fd9a40b8690 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.448+0000 7fd993fff640 1 -- 172.21.15.59:0/3896606831 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd9ac0b4fe0 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.677+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9ac0b6aa0 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.678+0000 7fd993fff640 1 -- 172.21.15.59:0/3896606831 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+5244 (secure 0 0 0) 0x7fd9a404e080 con 0x7fd9ac0d1a60 2024-06-29T16:01:20.679 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:20.679 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":157,"btime":"2024-06-29T15:59:49:317195+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1039110220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1039110220},{"type":"v1","addr":"172.21.15.153:6835","nonce":1039110220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":12583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3996608135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3996608135},{"type":"v1","addr":"172.21.15.153:6837","nonce":3996608135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":13620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/952472867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":952472867},{"type":"v1","addr":"172.21.15.59:6834","nonce":952472867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:45.699384+0000","modified":"2024-06-29T15:59:48.709634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13623":{"gid":13623,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3706652401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3706652401},{"type":"v1","addr":"172.21.15.59:6835","nonce":3706652401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13623,"qdb_cluster":[13623]},"id":18}]} 2024-06-29T16:01:20.679 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 157 2024-06-29T16:01:20.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 msgr2=0x7fd9ac153d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 0x7fd9ac153d20 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009bf0 tx=0x7fd9a4003ba0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 shutdown_connections 2024-06-29T16:01:20.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9ac0d1a60 0x7fd9ac153d20 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9ac0b2fa0 0x7fd9ac1537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 --2- 172.21.15.59:0/3896606831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9ac0b2270 0x7fd9ac146360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 >> 172.21.15.59:0/3896606831 conn(0x7fd9ac073e10 msgr2=0x7fd9ac074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:20.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 shutdown_connections 2024-06-29T16:01:20.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.680+0000 7fd9b1bfb640 1 -- 172.21.15.59:0/3896606831 wait complete. 2024-06-29T16:01:20.692 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:20.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.809+0000 7efded864640 1 -- 172.21.15.59:0/70599533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 msgr2=0x7efde80b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.809+0000 7efded864640 1 --2- 172.21.15.59:0/70599533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde80b3b50 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009830 tx=0x7efdd002a510 comp rx=0 tx=0).stop 2024-06-29T16:01:20.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.809+0000 7efded864640 1 -- 172.21.15.59:0/70599533 shutdown_connections 2024-06-29T16:01:20.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.809+0000 7efded864640 1 --2- 172.21.15.59:0/70599533 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde815c2f0 0x7efde815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.809+0000 7efded864640 1 --2- 172.21.15.59:0/70599533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde80b3b50 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.809+0000 7efded864640 1 -- 172.21.15.59:0/70599533 >> 172.21.15.59:0/70599533 conn(0x7efde806c950 msgr2=0x7efde8073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:20.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.809+0000 7efded864640 1 -- 172.21.15.59:0/70599533 shutdown_connections 2024-06-29T16:01:20.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.809+0000 7efded864640 1 -- 172.21.15.59:0/70599533 wait complete. 2024-06-29T16:01:20.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.810+0000 7efded864640 1 Processor -- start 2024-06-29T16:01:20.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.811+0000 7efded864640 1 -- start start 2024-06-29T16:01:20.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efded864640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde8140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efded864640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde815c2f0 0x7efde806e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efded864640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde806ed40 0x7efde806f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efded864640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7efde80b5fd0 con 0x7efde80b3740 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efded864640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efde80b5e50 con 0x7efde815c2f0 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efded864640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efde80b6150 con 0x7efde806ed40 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde8140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde8140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56768/0 (socket says 172.21.15.59:56768) 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde6ffd640 1 -- 172.21.15.59:0/500417797 learned_addr learned my addr 172.21.15.59:0/500417797 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde77fe640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde806ed40 0x7efde806f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde67fc640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde815c2f0 0x7efde806e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde6ffd640 1 -- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde806ed40 msgr2=0x7efde806f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde6ffd640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde806ed40 0x7efde806f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde6ffd640 1 -- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde815c2f0 msgr2=0x7efde806e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde6ffd640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde815c2f0 0x7efde806e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde6ffd640 1 -- 172.21.15.59:0/500417797 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde80af0f0 con 0x7efde80b3740 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde77fe640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde806ed40 0x7efde806f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.812+0000 7efde67fc640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde815c2f0 0x7efde806e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.813+0000 7efde6ffd640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde8140e50 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7efde000c3f0 tx=0x7efde000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.813+0000 7efddffff640 1 -- 172.21.15.59:0/500417797 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0017070 con 0x7efde80b3740 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.813+0000 7efddffff640 1 -- 172.21.15.59:0/500417797 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efde00035f0 con 0x7efde80b3740 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.813+0000 7efddffff640 1 -- 172.21.15.59:0/500417797 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0003a40 con 0x7efde80b3740 2024-06-29T16:01:20.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.813+0000 7efded864640 1 -- 172.21.15.59:0/500417797 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde80adb40 con 0x7efde80b3740 2024-06-29T16:01:20.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.813+0000 7efded864640 1 -- 172.21.15.59:0/500417797 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde806dd20 con 0x7efde80b3740 2024-06-29T16:01:20.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.814+0000 7efded864640 1 -- 172.21.15.59:0/500417797 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde80b6670 con 0x7efde80b3740 2024-06-29T16:01:20.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.814+0000 7efddffff640 1 -- 172.21.15.59:0/500417797 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 162) ==== 97912+0+0 (secure 0 0 0) 0x7efde001a070 con 0x7efde80b3740 2024-06-29T16:01:20.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.815+0000 7efddffff640 1 -- 172.21.15.59:0/500417797 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(280..280 src has 1..280) ==== 7089+0+0 (secure 0 0 0) 0x7efde0096ac0 con 0x7efde80b3740 2024-06-29T16:01:20.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:20.819+0000 7efddffff640 1 -- 172.21.15.59:0/500417797 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efde80adb40 con 0x7efde80b3740 2024-06-29T16:01:21.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.010+0000 7efded864640 1 -- 172.21.15.59:0/500417797 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efde80b1f10 con 0x7efde80b3740 2024-06-29T16:01:21.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.011+0000 7efddffff640 1 -- 172.21.15.59:0/500417797 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v280) ==== 74+0+18413 (secure 0 0 0) 0x7efde0060300 con 0x7efde80b3740 2024-06-29T16:01:21.012 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:21.012 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":280,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T16:01:20.092916+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:44.104733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:44.643605+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":279,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.0","mappings":[{"from":7,"to":2}]},{"pgid":"38.12","mappings":[{"from":7,"to":4}]},{"pgid":"38.14","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3174351846":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/764992950":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3398970025":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3498108787":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2338595954":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/192063323":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3360291833":"2024-06-30T16:01:20.092843+0000","172.21.15.153:6833/2469779970":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2700705605":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2302272425":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1710208131":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6833/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6832/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1857217640":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/2749019736":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3489900589":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6832/2469779970":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/1124076092":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1986019092":"2024-06-30T15:59:57.978964+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":38,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:21.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 -- 172.21.15.59:0/500417797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 msgr2=0x7efde8140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde8140e50 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7efde000c3f0 tx=0x7efde000c8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 -- 172.21.15.59:0/500417797 shutdown_connections 2024-06-29T16:01:21.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efde806ed40 0x7efde806f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde815c2f0 0x7efde806e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 --2- 172.21.15.59:0/500417797 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efde80b3740 0x7efde8140e50 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 -- 172.21.15.59:0/500417797 >> 172.21.15.59:0/500417797 conn(0x7efde806c950 msgr2=0x7efde81415a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:21.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 -- 172.21.15.59:0/500417797 shutdown_connections 2024-06-29T16:01:21.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.013+0000 7efded864640 1 -- 172.21.15.59:0/500417797 wait complete. 2024-06-29T16:01:21.025 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T16:01:21.026 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:21.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.145+0000 7f2c20666640 1 -- 172.21.15.59:0/2135823665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180cf7e0 msgr2=0x7f2c18073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.145+0000 7f2c20666640 1 --2- 172.21.15.59:0/2135823665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180cf7e0 0x7f2c18073370 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f2c08005fd0 tx=0x7f2c0802a670 comp rx=0 tx=0).stop 2024-06-29T16:01:21.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.145+0000 7f2c20666640 1 -- 172.21.15.59:0/2135823665 shutdown_connections 2024-06-29T16:01:21.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.145+0000 7f2c20666640 1 --2- 172.21.15.59:0/2135823665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180cf7e0 0x7f2c18073370 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.145+0000 7f2c20666640 1 -- 172.21.15.59:0/2135823665 >> 172.21.15.59:0/2135823665 conn(0x7f2c1806cef0 msgr2=0x7f2c1806d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:21.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.146+0000 7f2c20666640 1 -- 172.21.15.59:0/2135823665 shutdown_connections 2024-06-29T16:01:21.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.146+0000 7f2c20666640 1 -- 172.21.15.59:0/2135823665 wait complete. 2024-06-29T16:01:21.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.147+0000 7f2c20666640 1 Processor -- start 2024-06-29T16:01:21.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.147+0000 7f2c20666640 1 -- start start 2024-06-29T16:01:21.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c20666640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180738b0 0x7f2c1815b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c20666640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c180b4f30 0x7f2c18131050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c20666640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c180cf7e0 0x7f2c18131950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c20666640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2c180b3bb0 con 0x7f2c180738b0 2024-06-29T16:01:21.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c20666640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c180b3a30 con 0x7f2c180cf7e0 2024-06-29T16:01:21.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c20666640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2c180b3d30 con 0x7f2c180b4f30 2024-06-29T16:01:21.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180738b0 0x7f2c1815b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180738b0 0x7f2c1815b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56784/0 (socket says 172.21.15.59:56784) 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 -- 172.21.15.59:0/306753129 learned_addr learned my addr 172.21.15.59:0/306753129 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1ec03640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c180cf7e0 0x7f2c18131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1dc01640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c180b4f30 0x7f2c18131050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 -- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c180b4f30 msgr2=0x7f2c18131050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c180b4f30 0x7f2c18131050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 -- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c180cf7e0 msgr2=0x7f2c18131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c180cf7e0 0x7f2c18131950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 -- 172.21.15.59:0/306753129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1815a370 con 0x7f2c180738b0 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.149+0000 7f2c1e402640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180738b0 0x7f2c1815b410 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c430 tx=0x7f2c1400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.150+0000 7f2c0effd640 1 -- 172.21.15.59:0/306753129 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c14017070 con 0x7f2c180738b0 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.150+0000 7f2c0effd640 1 -- 172.21.15.59:0/306753129 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c140053c0 con 0x7f2c180738b0 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.150+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c1815a650 con 0x7f2c180738b0 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.150+0000 7f2c0effd640 1 -- 172.21.15.59:0/306753129 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c14003850 con 0x7f2c180738b0 2024-06-29T16:01:21.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.150+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1815ac10 con 0x7f2c180738b0 2024-06-29T16:01:21.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.151+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c1815f120 con 0x7f2c180738b0 2024-06-29T16:01:21.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.151+0000 7f2c0effd640 1 -- 172.21.15.59:0/306753129 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 163) ==== 98039+0+0 (secure 0 0 0) 0x7f2c1401a070 con 0x7f2c180738b0 2024-06-29T16:01:21.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.152+0000 7f2c0effd640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2bfc076150 0x7f2bfc078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.152+0000 7f2c0effd640 1 -- 172.21.15.59:0/306753129 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(280..280 src has 1..280) ==== 7089+0+0 (secure 0 0 0) 0x7f2c14096cd0 con 0x7f2c180738b0 2024-06-29T16:01:21.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.152+0000 7f2c1dc01640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2bfc076150 0x7f2bfc078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.153+0000 7f2c1dc01640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2bfc076150 0x7f2bfc078660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2c08033000 tx=0x7f2c08004730 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:21.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.156+0000 7f2c0effd640 1 -- 172.21.15.59:0/306753129 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2c1405fd20 con 0x7f2c180738b0 2024-06-29T16:01:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.387+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c1815ee40 con 0x7f2c180738b0 2024-06-29T16:01:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.387+0000 7f2c0effd640 1 -- 172.21.15.59:0/306753129 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+5244 (secure 0 0 0) 0x7f2c140683f0 con 0x7f2c180738b0 2024-06-29T16:01:21.388 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:21.388 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":157,"btime":"2024-06-29T15:59:49:317195+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1039110220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1039110220},{"type":"v1","addr":"172.21.15.153:6835","nonce":1039110220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":12583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3996608135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3996608135},{"type":"v1","addr":"172.21.15.153:6837","nonce":3996608135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":13620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/952472867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":952472867},{"type":"v1","addr":"172.21.15.59:6834","nonce":952472867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:45.699384+0000","modified":"2024-06-29T15:59:48.709634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13623":{"gid":13623,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3706652401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3706652401},{"type":"v1","addr":"172.21.15.59:6835","nonce":3706652401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13623,"qdb_cluster":[13623]},"id":18}]} 2024-06-29T16:01:21.389 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 157 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2bfc076150 msgr2=0x7f2bfc078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2bfc076150 0x7f2bfc078660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2c08033000 tx=0x7f2c08004730 comp rx=0 tx=0).stop 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180738b0 msgr2=0x7f2c1815b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180738b0 0x7f2c1815b410 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c430 tx=0x7f2c1400c8f0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 shutdown_connections 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2bfc076150 0x7f2bfc078660 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c180cf7e0 0x7f2c18131950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2c180b4f30 0x7f2c18131050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 --2- 172.21.15.59:0/306753129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c180738b0 0x7f2c1815b410 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 >> 172.21.15.59:0/306753129 conn(0x7f2c1806cef0 msgr2=0x7f2c180af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 shutdown_connections 2024-06-29T16:01:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.389+0000 7f2c20666640 1 -- 172.21.15.59:0/306753129 wait complete. 2024-06-29T16:01:21.402 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:21.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.519+0000 7f2903a4e640 1 -- 172.21.15.59:0/4216341986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28fc08c190 msgr2=0x7f28fc0cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.519+0000 7f2903a4e640 1 --2- 172.21.15.59:0/4216341986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28fc08c190 0x7f28fc0cf080 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009be0 tx=0x7f28ec02a870 comp rx=0 tx=0).stop 2024-06-29T16:01:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.519+0000 7f2903a4e640 1 -- 172.21.15.59:0/4216341986 shutdown_connections 2024-06-29T16:01:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.519+0000 7f2903a4e640 1 --2- 172.21.15.59:0/4216341986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28fc08c190 0x7f28fc0cf080 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.519+0000 7f2903a4e640 1 -- 172.21.15.59:0/4216341986 >> 172.21.15.59:0/4216341986 conn(0x7f28fc0b5610 msgr2=0x7f28fc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.519+0000 7f2903a4e640 1 -- 172.21.15.59:0/4216341986 shutdown_connections 2024-06-29T16:01:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.519+0000 7f2903a4e640 1 -- 172.21.15.59:0/4216341986 wait complete. 2024-06-29T16:01:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.521+0000 7f2903a4e640 1 Processor -- start 2024-06-29T16:01:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.521+0000 7f2903a4e640 1 -- start start 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.522+0000 7f2903a4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28fc08c190 0x7f28fc150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.522+0000 7f2903a4e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28fc0ce950 0x7f28fc14ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.522+0000 7f2903a4e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28fc0cf5c0 0x7f28fc155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.522+0000 7f2903a4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc169690 con 0x7f28fc08c190 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.522+0000 7f2903a4e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc169510 con 0x7f28fc0ce950 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.522+0000 7f2903a4e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f28fc169810 con 0x7f28fc0cf5c0 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28fc08c190 0x7f28fc150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28fc08c190 0x7f28fc150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56794/0 (socket says 172.21.15.59:56794) 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 -- 172.21.15.59:0/2946810931 learned_addr learned my addr 172.21.15.59:0/2946810931 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f2900fe9640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28fc0ce950 0x7f28fc14ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f2901feb640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28fc0cf5c0 0x7f28fc155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 -- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28fc0cf5c0 msgr2=0x7f28fc155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28fc0cf5c0 0x7f28fc155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 -- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28fc0ce950 msgr2=0x7f28fc14ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28fc0ce950 0x7f28fc14ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 -- 172.21.15.59:0/2946810931 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc1510c0 con 0x7f28fc08c190 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f2901feb640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28fc0cf5c0 0x7f28fc155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f29017ea640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28fc08c190 0x7f28fc150750 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009d00 tx=0x7f28ec031f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f28f9ffb640 1 -- 172.21.15.59:0/2946810931 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec03f070 con 0x7f28fc08c190 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f28f9ffb640 1 -- 172.21.15.59:0/2946810931 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28ec003c60 con 0x7f28fc08c190 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc15ba40 con 0x7f28fc08c190 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.523+0000 7f28f9ffb640 1 -- 172.21.15.59:0/2946810931 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec036040 con 0x7f28fc08c190 2024-06-29T16:01:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.524+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc15c000 con 0x7f28fc08c190 2024-06-29T16:01:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.525+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28fc0cf080 con 0x7f28fc08c190 2024-06-29T16:01:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.525+0000 7f28f9ffb640 1 -- 172.21.15.59:0/2946810931 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 163) ==== 98039+0+0 (secure 0 0 0) 0x7f28ec039070 con 0x7f28fc08c190 2024-06-29T16:01:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.526+0000 7f28f9ffb640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28e8076150 0x7f28e8078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.526+0000 7f28f9ffb640 1 -- 172.21.15.59:0/2946810931 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(280..280 src has 1..280) ==== 7089+0+0 (secure 0 0 0) 0x7f28ec0b7ef0 con 0x7f28fc08c190 2024-06-29T16:01:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.526+0000 7f2900fe9640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28e8076150 0x7f28e8078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.527+0000 7f2900fe9640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28e8076150 0x7f28e8078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28f0001eb0 tx=0x7f28f00013b0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.531+0000 7f28f9ffb640 1 -- 172.21.15.59:0/2946810931 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f28ec080fd0 con 0x7f28fc08c190 2024-06-29T16:01:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.762+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28fc06c7a0 con 0x7f28fc08c190 2024-06-29T16:01:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.763+0000 7f28f9ffb640 1 -- 172.21.15.59:0/2946810931 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+5244 (secure 0 0 0) 0x7f28ec0852d0 con 0x7f28fc08c190 2024-06-29T16:01:21.764 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:21.764 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":157,"btime":"2024-06-29T15:59:49:317195+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1039110220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1039110220},{"type":"v1","addr":"172.21.15.153:6835","nonce":1039110220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":12583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3996608135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3996608135},{"type":"v1","addr":"172.21.15.153:6837","nonce":3996608135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":13620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/952472867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":952472867},{"type":"v1","addr":"172.21.15.59:6834","nonce":952472867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:45.699384+0000","modified":"2024-06-29T15:59:48.709634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13623},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13623":{"gid":13623,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/3706652401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3706652401},{"type":"v1","addr":"172.21.15.59:6835","nonce":3706652401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13623,"qdb_cluster":[13623]},"id":18}]} 2024-06-29T16:01:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 157 2024-06-29T16:01:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28e8076150 msgr2=0x7f28e8078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28e8076150 0x7f28e8078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28f0001eb0 tx=0x7f28f00013b0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28fc08c190 msgr2=0x7f28fc150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28fc08c190 0x7f28fc150750 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009d00 tx=0x7f28ec031f30 comp rx=0 tx=0).stop 2024-06-29T16:01:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 shutdown_connections 2024-06-29T16:01:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28e8076150 0x7f28e8078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28fc0cf5c0 0x7f28fc155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28fc0ce950 0x7f28fc14ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 --2- 172.21.15.59:0/2946810931 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28fc08c190 0x7f28fc150750 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 >> 172.21.15.59:0/2946810931 conn(0x7f28fc0b5610 msgr2=0x7f28fc157860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 shutdown_connections 2024-06-29T16:01:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.765+0000 7f2903a4e640 1 -- 172.21.15.59:0/2946810931 wait complete. 2024-06-29T16:01:21.777 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 -- 172.21.15.59:0/3693177697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 msgr2=0x7fb2b413ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/3693177697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 0x7fb2b413ebd0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40b3b20 tx=0x7fb2a402aa30 comp rx=0 tx=0).stop 2024-06-29T16:01:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 -- 172.21.15.59:0/3693177697 shutdown_connections 2024-06-29T16:01:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/3693177697 >> v2:172.21.15.153:3300/0 conn(0x7fb2b4141360 0x7fb2b4141770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/3693177697 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2b40b1b00 0x7fb2b40b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/3693177697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 0x7fb2b413ebd0 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 -- 172.21.15.59:0/3693177697 >> 172.21.15.59:0/3693177697 conn(0x7fb2b40b06e0 msgr2=0x7fb2b40b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 -- 172.21.15.59:0/3693177697 shutdown_connections 2024-06-29T16:01:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.893+0000 7fb2bb17f640 1 -- 172.21.15.59:0/3693177697 wait complete. 2024-06-29T16:01:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.895+0000 7fb2bb17f640 1 Processor -- start 2024-06-29T16:01:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.895+0000 7fb2bb17f640 1 -- start start 2024-06-29T16:01:21.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.896+0000 7fb2bb17f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b40b1b00 0x7fb2b41592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.896+0000 7fb2bb17f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2b4141360 0x7fb2b4155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.896+0000 7fb2bb17f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 0x7fb2b41319d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.896+0000 7fb2bb17f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b40af890 con 0x7fb2b41604d0 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.896+0000 7fb2bb17f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b40af710 con 0x7fb2b40b1b00 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.896+0000 7fb2bb17f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb2b40afa10 con 0x7fb2b4141360 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 0x7fb2b41319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 0x7fb2b41319d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56806/0 (socket says 172.21.15.59:56806) 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 -- 172.21.15.59:0/1380347758 learned_addr learned my addr 172.21.15.59:0/1380347758 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b3fff640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2b4141360 0x7fb2b4155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b8f1b640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b40b1b00 0x7fb2b41592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 -- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2b4141360 msgr2=0x7fb2b4155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2b4141360 0x7fb2b4155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 -- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b40b1b00 msgr2=0x7fb2b41592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b40b1b00 0x7fb2b41592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 -- 172.21.15.59:0/1380347758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b4157420 con 0x7fb2b41604d0 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b971c640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 0x7fb2b41319d0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00c970 tx=0x7fb2ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b17fa640 1 -- 172.21.15.59:0/1380347758 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac004460 con 0x7fb2b41604d0 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b17fa640 1 -- 172.21.15.59:0/1380347758 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2ac0077a0 con 0x7fb2b41604d0 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b4157890 con 0x7fb2b41604d0 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2b17fa640 1 -- 172.21.15.59:0/1380347758 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac007bd0 con 0x7fb2b41604d0 2024-06-29T16:01:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.897+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b4157e50 con 0x7fb2b41604d0 2024-06-29T16:01:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.898+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2b40affa0 con 0x7fb2b41604d0 2024-06-29T16:01:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.899+0000 7fb2b17fa640 1 -- 172.21.15.59:0/1380347758 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 163) ==== 98039+0+0 (secure 0 0 0) 0x7fb2ac0035f0 con 0x7fb2b41604d0 2024-06-29T16:01:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.900+0000 7fb2b17fa640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb294076120 0x7fb294078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.900+0000 7fb2b17fa640 1 -- 172.21.15.59:0/1380347758 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(280..280 src has 1..280) ==== 7089+0+0 (secure 0 0 0) 0x7fb2ac098050 con 0x7fb2b41604d0 2024-06-29T16:01:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.900+0000 7fb2b8f1b640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb294076120 0x7fb294078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.900+0000 7fb2b8f1b640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb294076120 0x7fb294078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4035000 tx=0x7fb2a4004ab0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:21.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:21.904+0000 7fb2b17fa640 1 -- 172.21.15.59:0/1380347758 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb2ac013030 con 0x7fb2b41604d0 2024-06-29T16:01:22.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.099+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2b41575f0 con 0x7fb2b41604d0 2024-06-29T16:01:22.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.101+0000 7fb2b17fa640 1 -- 172.21.15.59:0/1380347758 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v280) ==== 74+0+18413 (secure 0 0 0) 0x7fb2ac061780 con 0x7fb2b41604d0 2024-06-29T16:01:22.103 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:22.103 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":280,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T16:01:20.092916+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:44.104733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:44.643605+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":279,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.0","mappings":[{"from":7,"to":2}]},{"pgid":"38.12","mappings":[{"from":7,"to":4}]},{"pgid":"38.14","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/3174351846":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/764992950":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3398970025":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3498108787":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2338595954":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/192063323":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3360291833":"2024-06-30T16:01:20.092843+0000","172.21.15.153:6833/2469779970":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2700705605":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2302272425":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1710208131":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6833/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6832/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1857217640":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/2749019736":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3489900589":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6832/2469779970":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/1124076092":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1986019092":"2024-06-30T15:59:57.978964+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":38,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb294076120 msgr2=0x7fb294078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb294076120 0x7fb294078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4035000 tx=0x7fb2a4004ab0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 msgr2=0x7fb2b41319d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 0x7fb2b41319d0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00c970 tx=0x7fb2ac00ce30 comp rx=0 tx=0).stop 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 shutdown_connections 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb294076120 0x7fb294078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2b41604d0 0x7fb2b41319d0 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb2b4141360 0x7fb2b4155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 --2- 172.21.15.59:0/1380347758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b40b1b00 0x7fb2b41592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 >> 172.21.15.59:0/1380347758 conn(0x7fb2b40b06e0 msgr2=0x7fb2b415d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 shutdown_connections 2024-06-29T16:01:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.104+0000 7fb2bb17f640 1 -- 172.21.15.59:0/1380347758 wait complete. 2024-06-29T16:01:22.116 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T16:01:22.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.235+0000 7f0b97feb640 1 -- 172.21.15.59:0/3768931842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900ce530 msgr2=0x7f0b900cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.235+0000 7f0b97feb640 1 --2- 172.21.15.59:0/3768931842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900ce530 0x7f0b900cccc0 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009830 tx=0x7f0b8402a480 comp rx=0 tx=0).stop 2024-06-29T16:01:22.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.235+0000 7f0b97feb640 1 -- 172.21.15.59:0/3768931842 shutdown_connections 2024-06-29T16:01:22.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.235+0000 7f0b97feb640 1 --2- 172.21.15.59:0/3768931842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900ce530 0x7f0b900cccc0 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.235+0000 7f0b97feb640 1 -- 172.21.15.59:0/3768931842 >> 172.21.15.59:0/3768931842 conn(0x7f0b900b24c0 msgr2=0x7f0b900cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:22.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.235+0000 7f0b97feb640 1 -- 172.21.15.59:0/3768931842 shutdown_connections 2024-06-29T16:01:22.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.235+0000 7f0b97feb640 1 -- 172.21.15.59:0/3768931842 wait complete. 2024-06-29T16:01:22.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.237+0000 7f0b97feb640 1 Processor -- start 2024-06-29T16:01:22.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.237+0000 7f0b97feb640 1 -- start start 2024-06-29T16:01:22.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b97feb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9006fea0 0x7f0b900ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b97feb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900cd200 0x7f0b90155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b97feb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b900ce530 0x7f0b900a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b97feb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9015e5f0 con 0x7f0b900cd200 2024-06-29T16:01:22.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b97feb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9015e770 con 0x7f0b9006fea0 2024-06-29T16:01:22.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b97feb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9015e8f0 con 0x7f0b900ce530 2024-06-29T16:01:22.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b95586640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900cd200 0x7f0b90155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b95586640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900cd200 0x7f0b90155340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56820/0 (socket says 172.21.15.59:56820) 2024-06-29T16:01:22.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b95586640 1 -- 172.21.15.59:0/1108730366 learned_addr learned my addr 172.21.15.59:0/1108730366 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:22.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b95d87640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9006fea0 0x7f0b900ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.238+0000 7f0b96588640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b900ce530 0x7f0b900a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b95586640 1 -- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b900ce530 msgr2=0x7f0b900a9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b95586640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b900ce530 0x7f0b900a9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b95586640 1 -- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9006fea0 msgr2=0x7f0b900ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b95586640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9006fea0 0x7f0b900ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b95586640 1 -- 172.21.15.59:0/1108730366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b90155110 con 0x7f0b900cd200 2024-06-29T16:01:22.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b95586640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900cd200 0x7f0b90155340 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009950 tx=0x7f0b84033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:22.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b8e7fc640 1 -- 172.21.15.59:0/1108730366 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b84031d60 con 0x7f0b900cd200 2024-06-29T16:01:22.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b8e7fc640 1 -- 172.21.15.59:0/1108730366 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b840330d0 con 0x7f0b900cd200 2024-06-29T16:01:22.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b900cac60 con 0x7f0b900cd200 2024-06-29T16:01:22.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b8e7fc640 1 -- 172.21.15.59:0/1108730366 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b84033520 con 0x7f0b900cd200 2024-06-29T16:01:22.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.239+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b90156190 con 0x7f0b900cd200 2024-06-29T16:01:22.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.240+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b90155c20 con 0x7f0b900cd200 2024-06-29T16:01:22.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.241+0000 7f0b8e7fc640 1 -- 172.21.15.59:0/1108730366 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 163) ==== 98039+0+0 (secure 0 0 0) 0x7f0b840035f0 con 0x7f0b900cd200 2024-06-29T16:01:22.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.241+0000 7f0b8e7fc640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0b7c076150 0x7f0b7c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.242+0000 7f0b8e7fc640 1 -- 172.21.15.59:0/1108730366 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(280..280 src has 1..280) ==== 7089+0+0 (secure 0 0 0) 0x7f0b840b8d80 con 0x7f0b900cd200 2024-06-29T16:01:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.242+0000 7f0b95d87640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0b7c076150 0x7f0b7c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.242+0000 7f0b95d87640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0b7c076150 0x7f0b7c078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0b88001720 tx=0x7f0b88007e90 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:22.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.246+0000 7f0b8e7fc640 1 -- 172.21.15.59:0/1108730366 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b8403f030 con 0x7f0b900cd200 2024-06-29T16:01:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.444+0000 7f0b8e7fc640 1 -- 172.21.15.59:0/1108730366 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 164) ==== 98247+0+0 (secure 0 0 0) 0x7f0b8407e060 con 0x7f0b900cd200 2024-06-29T16:01:22.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.482+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0b900cccc0 con 0x7f0b900cd200 2024-06-29T16:01:22.484 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-06-29T16:01:22.483+0000 7f92f1117640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T16:01:22.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.484+0000 7f0b8e7fc640 1 -- 172.21.15.59:0/1108730366 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v158) ==== 161+0+0 (secure 0 0 0) 0x7f0b840b3520 con 0x7f0b900cd200 2024-06-29T16:01:22.485 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T16:01:22.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0b7c076150 msgr2=0x7f0b7c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0b7c076150 0x7f0b7c078660 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0b88001720 tx=0x7f0b88007e90 comp rx=0 tx=0).stop 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900cd200 msgr2=0x7f0b90155340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900cd200 0x7f0b90155340 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009950 tx=0x7f0b84033d60 comp rx=0 tx=0).stop 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 shutdown_connections 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0b7c076150 0x7f0b7c078660 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0b900ce530 0x7f0b900a9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b900cd200 0x7f0b90155340 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 --2- 172.21.15.59:0/1108730366 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b9006fea0 0x7f0b900ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 >> 172.21.15.59:0/1108730366 conn(0x7f0b900b24c0 msgr2=0x7f0b90072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 shutdown_connections 2024-06-29T16:01:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.487+0000 7f0b97feb640 1 -- 172.21.15.59:0/1108730366 wait complete. 2024-06-29T16:01:22.501 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T16:01:22.554 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-06-29T16:01:22.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.623+0000 7f64384c1640 1 -- 172.21.15.59:0/463060468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64301596a0 msgr2=0x7f6430160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.623+0000 7f64384c1640 1 --2- 172.21.15.59:0/463060468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64301596a0 0x7f6430160090 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6428009c10 tx=0x7f642802a840 comp rx=0 tx=0).stop 2024-06-29T16:01:22.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.623+0000 7f64384c1640 1 -- 172.21.15.59:0/463060468 shutdown_connections 2024-06-29T16:01:22.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.623+0000 7f64384c1640 1 --2- 172.21.15.59:0/463060468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64301596a0 0x7f6430160090 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.623+0000 7f64384c1640 1 -- 172.21.15.59:0/463060468 >> 172.21.15.59:0/463060468 conn(0x7f64300b7690 msgr2=0x7f6430073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:22.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.623+0000 7f64384c1640 1 -- 172.21.15.59:0/463060468 shutdown_connections 2024-06-29T16:01:22.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.623+0000 7f64384c1640 1 -- 172.21.15.59:0/463060468 wait complete. 2024-06-29T16:01:22.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.625+0000 7f64384c1640 1 Processor -- start 2024-06-29T16:01:22.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.625+0000 7f64384c1640 1 -- start start 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.626+0000 7f64384c1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64300ce2c0 0x7f643014b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.626+0000 7f64384c1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64301596a0 0x7f64300ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.626+0000 7f64384c1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f643006db80 0x7f64300aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.626+0000 7f64384c1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f64300b5710 con 0x7f64300ce2c0 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.626+0000 7f64384c1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f64300b5590 con 0x7f643006db80 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.626+0000 7f64384c1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f64300b5890 con 0x7f64301596a0 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.626+0000 7f643625d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64300ce2c0 0x7f643014b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f643625d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64300ce2c0 0x7f643014b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56850/0 (socket says 172.21.15.59:56850) 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f643625d640 1 -- 172.21.15.59:0/2927702158 learned_addr learned my addr 172.21.15.59:0/2927702158 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f6435a5c640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64301596a0 0x7f64300ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f6436a5e640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f643006db80 0x7f64300aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f643625d640 1 -- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64301596a0 msgr2=0x7f64300ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f643625d640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64301596a0 0x7f64300ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f643625d640 1 -- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f643006db80 msgr2=0x7f64300aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f643625d640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f643006db80 0x7f64300aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f643625d640 1 -- 172.21.15.59:0/2927702158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643006e120 con 0x7f64300ce2c0 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f643625d640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64300ce2c0 0x7f643014b3a0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f642400c460 tx=0x7f642400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f6422ffd640 1 -- 172.21.15.59:0/2927702158 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64240052d0 con 0x7f64300ce2c0 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f6422ffd640 1 -- 172.21.15.59:0/2927702158 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6424005490 con 0x7f64300ce2c0 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f6422ffd640 1 -- 172.21.15.59:0/2927702158 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424012870 con 0x7f64300ce2c0 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643006e400 con 0x7f64300ce2c0 2024-06-29T16:01:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.627+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f643006e9c0 con 0x7f64300ce2c0 2024-06-29T16:01:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.629+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6430160090 con 0x7f64300ce2c0 2024-06-29T16:01:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.629+0000 7f6422ffd640 1 -- 172.21.15.59:0/2927702158 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 164) ==== 98247+0+0 (secure 0 0 0) 0x7f6424012a10 con 0x7f64300ce2c0 2024-06-29T16:01:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.630+0000 7f6422ffd640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f6414076390 0x7f64140788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.630+0000 7f6422ffd640 1 -- 172.21.15.59:0/2927702158 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 7175+0+0 (secure 0 0 0) 0x7f6424097840 con 0x7f64300ce2c0 2024-06-29T16:01:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.630+0000 7f6435a5c640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f6414076390 0x7f64140788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.630+0000 7f6435a5c640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f6414076390 0x7f64140788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f642c001360 tx=0x7f642c0012f0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:22.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.634+0000 7f6422ffd640 1 -- 172.21.15.59:0/2927702158 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6424060b60 con 0x7f64300ce2c0 2024-06-29T16:01:22.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:22.864+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f643015f810 con 0x7f64300ce2c0 2024-06-29T16:01:23.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.488+0000 7f6422ffd640 1 -- 172.21.15.59:0/2927702158 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v159) ==== 104+0+0 (secure 0 0 0) 0x7f6424060460 con 0x7f64300ce2c0 2024-06-29T16:01:23.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f6414076390 msgr2=0x7f64140788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f6414076390 0x7f64140788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f642c001360 tx=0x7f642c0012f0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64300ce2c0 msgr2=0x7f643014b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64300ce2c0 0x7f643014b3a0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f642400c460 tx=0x7f642400c920 comp rx=0 tx=0).stop 2024-06-29T16:01:23.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 shutdown_connections 2024-06-29T16:01:23.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f6414076390 0x7f64140788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f643006db80 0x7f64300aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64301596a0 0x7f64300ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 --2- 172.21.15.59:0/2927702158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64300ce2c0 0x7f643014b3a0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 >> 172.21.15.59:0/2927702158 conn(0x7f64300b7690 msgr2=0x7f64301552e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:23.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 shutdown_connections 2024-06-29T16:01:23.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.490+0000 7f64384c1640 1 -- 172.21.15.59:0/2927702158 wait complete. 2024-06-29T16:01:23.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T16:01:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.632+0000 7f0186be0640 1 -- 172.21.15.59:0/2045638844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800736d0 msgr2=0x7f01800cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.632+0000 7f0186be0640 1 --2- 172.21.15.59:0/2045638844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800736d0 0x7f01800cf000 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f0178005fd0 tx=0x7f017802a5d0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.632+0000 7f0186be0640 1 -- 172.21.15.59:0/2045638844 shutdown_connections 2024-06-29T16:01:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.632+0000 7f0186be0640 1 --2- 172.21.15.59:0/2045638844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800736d0 0x7f01800cf000 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.632+0000 7f0186be0640 1 --2- 172.21.15.59:0/2045638844 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f018006e190 0x7f018015bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.632+0000 7f0186be0640 1 -- 172.21.15.59:0/2045638844 >> 172.21.15.59:0/2045638844 conn(0x7f018006fce0 msgr2=0x7f01800700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.632+0000 7f0186be0640 1 -- 172.21.15.59:0/2045638844 shutdown_connections 2024-06-29T16:01:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.632+0000 7f0186be0640 1 -- 172.21.15.59:0/2045638844 wait complete. 2024-06-29T16:01:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.634+0000 7f0186be0640 1 Processor -- start 2024-06-29T16:01:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.634+0000 7f0186be0640 1 -- start start 2024-06-29T16:01:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f0186be0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01800736d0 0x7f01801eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f0186be0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800cca60 0x7f01800af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f0186be0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01800725c0 0x7f01800729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f0186be0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f01800cf6c0 con 0x7f01800cca60 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f0186be0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01800cf540 con 0x7f01800736d0 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f0186be0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f01800cf840 con 0x7f01800725c0 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f017ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800cca60 0x7f01800af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f017ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800cca60 0x7f01800af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44198/0 (socket says 172.21.15.59:44198) 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f017ffff640 1 -- 172.21.15.59:0/556718162 learned_addr learned my addr 172.21.15.59:0/556718162 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f018497c640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01800736d0 0x7f01801eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.635+0000 7f018517d640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01800725c0 0x7f01800729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017ffff640 1 -- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01800725c0 msgr2=0x7f01800729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017ffff640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01800725c0 0x7f01800729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017ffff640 1 -- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01800736d0 msgr2=0x7f01801eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017ffff640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01800736d0 0x7f01801eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017ffff640 1 -- 172.21.15.59:0/556718162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0180072f10 con 0x7f01800cca60 2024-06-29T16:01:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017ffff640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800cca60 0x7f01800af210 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f016800eb30 tx=0x7f016800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017d7fa640 1 -- 172.21.15.59:0/556718162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168010430 con 0x7f01800cca60 2024-06-29T16:01:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017d7fa640 1 -- 172.21.15.59:0/556718162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01680106c0 con 0x7f01800cca60 2024-06-29T16:01:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01800af750 con 0x7f01800cca60 2024-06-29T16:01:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f017d7fa640 1 -- 172.21.15.59:0/556718162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168010b10 con 0x7f01800cca60 2024-06-29T16:01:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.636+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01801f62b0 con 0x7f01800cca60 2024-06-29T16:01:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.637+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01800cfcf0 con 0x7f01800cca60 2024-06-29T16:01:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.638+0000 7f017d7fa640 1 -- 172.21.15.59:0/556718162 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 164) ==== 98247+0+0 (secure 0 0 0) 0x7f0168010cb0 con 0x7f01800cca60 2024-06-29T16:01:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.638+0000 7f017d7fa640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0160076360 0x7f0160078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.639+0000 7f017d7fa640 1 -- 172.21.15.59:0/556718162 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 7175+0+0 (secure 0 0 0) 0x7f0168098d20 con 0x7f01800cca60 2024-06-29T16:01:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.639+0000 7f018497c640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0160076360 0x7f0160078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.640+0000 7f018497c640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0160076360 0x7f0160078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0170001720 tx=0x7f0170009040 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:23.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.643+0000 7f017d7fa640 1 -- 172.21.15.59:0/556718162 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0180072f10 con 0x7f01800cca60 2024-06-29T16:01:23.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:23.832+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f01800af920 con 0x7f01800cca60 2024-06-29T16:01:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.494+0000 7f017d7fa640 1 -- 172.21.15.59:0/556718162 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v282) ==== 181+0+0 (secure 0 0 0) 0x7f01680617e0 con 0x7f01800cca60 2024-06-29T16:01:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-06-29T16:01:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.500+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0160076360 msgr2=0x7f0160078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.500+0000 7f0186be0640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0160076360 0x7f0160078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0170001720 tx=0x7f0170009040 comp rx=0 tx=0).stop 2024-06-29T16:01:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.500+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800cca60 msgr2=0x7f01800af210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.500+0000 7f0186be0640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800cca60 0x7f01800af210 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f016800eb30 tx=0x7f016800c2d0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.502+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 shutdown_connections 2024-06-29T16:01:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.502+0000 7f0186be0640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0160076360 0x7f0160078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.502+0000 7f0186be0640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01800725c0 0x7f01800729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.502+0000 7f0186be0640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800cca60 0x7f01800af210 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.502+0000 7f0186be0640 1 --2- 172.21.15.59:0/556718162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01800736d0 0x7f01801eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.502+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 >> 172.21.15.59:0/556718162 conn(0x7f018006fce0 msgr2=0x7f018014fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.502+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 shutdown_connections 2024-06-29T16:01:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.502+0000 7f0186be0640 1 -- 172.21.15.59:0/556718162 wait complete. 2024-06-29T16:01:24.519 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T16:01:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.643+0000 7f9866735640 1 -- 172.21.15.59:0/950123155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600d1c30 msgr2=0x7f98600b6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.643+0000 7f9866735640 1 --2- 172.21.15.59:0/950123155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600d1c30 0x7f98600b6f70 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f98600b3d60 tx=0x7f985002a4a0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.643+0000 7f9866735640 1 -- 172.21.15.59:0/950123155 shutdown_connections 2024-06-29T16:01:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.643+0000 7f9866735640 1 --2- 172.21.15.59:0/950123155 >> v2:172.21.15.153:3300/0 conn(0x7f98600b2980 0x7f98600d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.643+0000 7f9866735640 1 --2- 172.21.15.59:0/950123155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600d1c30 0x7f98600b6f70 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.643+0000 7f9866735640 1 -- 172.21.15.59:0/950123155 >> 172.21.15.59:0/950123155 conn(0x7f98600d0f90 msgr2=0x7f98600d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.643+0000 7f9866735640 1 -- 172.21.15.59:0/950123155 shutdown_connections 2024-06-29T16:01:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.643+0000 7f9866735640 1 -- 172.21.15.59:0/950123155 wait complete. 2024-06-29T16:01:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.645+0000 7f9866735640 1 Processor -- start 2024-06-29T16:01:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.645+0000 7f9866735640 1 -- start start 2024-06-29T16:01:24.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f9866735640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600b2980 0x7f98601f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:24.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f9866735640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98600d1c30 0x7f98601572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:24.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f9866735640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98600d2ce0 0x7f98601529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:24.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f9866735640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f98600b4780 con 0x7f98600b2980 2024-06-29T16:01:24.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f9866735640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98600d3300 con 0x7f98600d1c30 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f9866735640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f98600b4900 con 0x7f98600d2ce0 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600b2980 0x7f98601f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600b2980 0x7f98601f4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44228/0 (socket says 172.21.15.59:44228) 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985ffff640 1 -- 172.21.15.59:0/1071971643 learned_addr learned my addr 172.21.15.59:0/1071971643 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985f7fe640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98600d1c30 0x7f98601572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f9864cd2640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98600d2ce0 0x7f98601529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985ffff640 1 -- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98600d2ce0 msgr2=0x7f98601529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985ffff640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98600d2ce0 0x7f98601529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985ffff640 1 -- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98600d1c30 msgr2=0x7f98601572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985ffff640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98600d1c30 0x7f98601572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.646+0000 7f985ffff640 1 -- 172.21.15.59:0/1071971643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9860154db0 con 0x7f98600b2980 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.647+0000 7f9864cd2640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98600d2ce0 0x7f98601529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:24.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.647+0000 7f985ffff640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600b2980 0x7f98601f4990 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f985002aa90 tx=0x7f9850033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:24.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.647+0000 7f985cff9640 1 -- 172.21.15.59:0/1071971643 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850031d60 con 0x7f98600b2980 2024-06-29T16:01:24.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.647+0000 7f985cff9640 1 -- 172.21.15.59:0/1071971643 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9850033e10 con 0x7f98600b2980 2024-06-29T16:01:24.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.647+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98601f4ed0 con 0x7f98600b2980 2024-06-29T16:01:24.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.647+0000 7f985cff9640 1 -- 172.21.15.59:0/1071971643 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850005260 con 0x7f98600b2980 2024-06-29T16:01:24.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.647+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98601581e0 con 0x7f98600b2980 2024-06-29T16:01:24.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.648+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98600b4db0 con 0x7f98600b2980 2024-06-29T16:01:24.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.649+0000 7f985cff9640 1 -- 172.21.15.59:0/1071971643 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f9850005400 con 0x7f98600b2980 2024-06-29T16:01:24.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.649+0000 7f985cff9640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f984c076420 0x7f984c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:24.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.649+0000 7f985cff9640 1 -- 172.21.15.59:0/1071971643 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(282..282 src has 1..282) ==== 6737+0+0 (secure 0 0 0) 0x7f98500b8e90 con 0x7f98600b2980 2024-06-29T16:01:24.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.649+0000 7f985f7fe640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f984c076420 0x7f984c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:24.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.650+0000 7f985f7fe640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f984c076420 0x7f984c078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f98600cdc00 tx=0x7f985400b2e0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:24.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.653+0000 7f985cff9640 1 -- 172.21.15.59:0/1071971643 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9860154db0 con 0x7f98600b2980 2024-06-29T16:01:24.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:24.843+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f98600b7290 con 0x7f98600b2980 2024-06-29T16:01:25.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.274+0000 7f985cff9640 1 -- 172.21.15.59:0/1071971643 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v283) ==== 169+0+0 (secure 0 0 0) 0x7f985003f030 con 0x7f98600b2980 2024-06-29T16:01:25.275 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-06-29T16:01:25.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f984c076420 msgr2=0x7f984c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f984c076420 0x7f984c078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f98600cdc00 tx=0x7f985400b2e0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600b2980 msgr2=0x7f98601f4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600b2980 0x7f98601f4990 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f985002aa90 tx=0x7f9850033c20 comp rx=0 tx=0).stop 2024-06-29T16:01:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 shutdown_connections 2024-06-29T16:01:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f984c076420 0x7f984c078930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f98600d2ce0 0x7f98601529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98600d1c30 0x7f98601572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 --2- 172.21.15.59:0/1071971643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98600b2980 0x7f98601f4990 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.278+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 >> 172.21.15.59:0/1071971643 conn(0x7f98600d0f90 msgr2=0x7f9860158600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.279+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 shutdown_connections 2024-06-29T16:01:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.279+0000 7f9866735640 1 -- 172.21.15.59:0/1071971643 wait complete. 2024-06-29T16:01:25.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T16:01:25.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.411+0000 7f27c1dd9640 1 -- 172.21.15.59:0/432344312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27a4004800 msgr2=0x7f27a4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.411+0000 7f27c1dd9640 1 --2- 172.21.15.59:0/432344312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27a4004800 0x7f27a4004c10 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7f27b00016c0 tx=0x7f27a4005830 comp rx=0 tx=0).stop 2024-06-29T16:01:25.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.411+0000 7f27c1dd9640 1 -- 172.21.15.59:0/432344312 shutdown_connections 2024-06-29T16:01:25.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.411+0000 7f27c1dd9640 1 --2- 172.21.15.59:0/432344312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27a4004800 0x7f27a4004c10 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.411+0000 7f27c1dd9640 1 -- 172.21.15.59:0/432344312 >> 172.21.15.59:0/432344312 conn(0x7f27bc0b06e0 msgr2=0x7f27bc0b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.412+0000 7f27c1dd9640 1 -- 172.21.15.59:0/432344312 shutdown_connections 2024-06-29T16:01:25.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.412+0000 7f27c1dd9640 1 -- 172.21.15.59:0/432344312 wait complete. 2024-06-29T16:01:25.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.413+0000 7f27c1dd9640 1 Processor -- start 2024-06-29T16:01:25.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.414+0000 7f27c1dd9640 1 -- start start 2024-06-29T16:01:25.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27c1dd9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27a4004800 0x7f27bc0cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27c1dd9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc0cee90 0x7f27bc1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27c1dd9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27bc159820 0x7f27bc1f5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27c1dd9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f27bc0b1f10 con 0x7f27bc159820 2024-06-29T16:01:25.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27c1dd9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27bc13f050 con 0x7f27bc0cee90 2024-06-29T16:01:25.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27c1dd9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f27bc0b2090 con 0x7f27a4004800 2024-06-29T16:01:25.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27bc159820 0x7f27bc1f5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27bc159820 0x7f27bc1f5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44246/0 (socket says 172.21.15.59:44246) 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bbfff640 1 -- 172.21.15.59:0/1835379875 learned_addr learned my addr 172.21.15.59:0/1835379875 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27baffd640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc0cee90 0x7f27bc1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bb7fe640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27a4004800 0x7f27bc0cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bbfff640 1 -- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27a4004800 msgr2=0x7f27bc0cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bbfff640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27a4004800 0x7f27bc0cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bbfff640 1 -- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc0cee90 msgr2=0x7f27bc1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bbfff640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc0cee90 0x7f27bc1592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.415+0000 7f27bbfff640 1 -- 172.21.15.59:0/1835379875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27bc0b5300 con 0x7f27bc159820 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.416+0000 7f27bbfff640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27bc159820 0x7f27bc1f5530 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0b3b20 tx=0x7f27b400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.416+0000 7f27a3fff640 1 -- 172.21.15.59:0/1835379875 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b400cba0 con 0x7f27bc159820 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.416+0000 7f27a3fff640 1 -- 172.21.15.59:0/1835379875 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27b4017070 con 0x7f27bc159820 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.416+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27bc0b57a0 con 0x7f27bc159820 2024-06-29T16:01:25.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.416+0000 7f27a3fff640 1 -- 172.21.15.59:0/1835379875 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b40116c0 con 0x7f27bc159820 2024-06-29T16:01:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.416+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27bc0b5d60 con 0x7f27bc159820 2024-06-29T16:01:25.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.417+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27bc070390 con 0x7f27bc159820 2024-06-29T16:01:25.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.417+0000 7f27a3fff640 1 -- 172.21.15.59:0/1835379875 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f27b4011860 con 0x7f27bc159820 2024-06-29T16:01:25.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.418+0000 7f27a3fff640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f27a80763f0 0x7f27a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.418+0000 7f27a3fff640 1 -- 172.21.15.59:0/1835379875 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f27b4098090 con 0x7f27bc159820 2024-06-29T16:01:25.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.418+0000 7f27bb7fe640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f27a80763f0 0x7f27a8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.419+0000 7f27bb7fe640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f27a80763f0 0x7f27a8078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f27ac009b90 tx=0x7f27ac005f90 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.423+0000 7f27a3fff640 1 -- 172.21.15.59:0/1835379875 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f27bc070390 con 0x7f27bc159820 2024-06-29T16:01:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.615+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f27bc141770 con 0x7f27bc159820 2024-06-29T16:01:25.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.619+0000 7f27a3fff640 1 -- 172.21.15.59:0/1835379875 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v87) ==== 71+0+0 (secure 0 0 0) 0x7f27b40615c0 con 0x7f27bc159820 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f27a80763f0 msgr2=0x7f27a8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f27a80763f0 0x7f27a8078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f27ac009b90 tx=0x7f27ac005f90 comp rx=0 tx=0).stop 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27bc159820 msgr2=0x7f27bc1f5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27bc159820 0x7f27bc1f5530 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0b3b20 tx=0x7f27b400ed30 comp rx=0 tx=0).stop 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 shutdown_connections 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f27a80763f0 0x7f27a8078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27bc159820 0x7f27bc1f5530 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27bc0cee90 0x7f27bc1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 --2- 172.21.15.59:0/1835379875 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27a4004800 0x7f27bc0cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 >> 172.21.15.59:0/1835379875 conn(0x7f27bc0b06e0 msgr2=0x7f27bc15bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.621+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 shutdown_connections 2024-06-29T16:01:25.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.622+0000 7f27c1dd9640 1 -- 172.21.15.59:0/1835379875 wait complete. 2024-06-29T16:01:25.633 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 -- 172.21.15.59:0/4256693814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 msgr2=0x7f9a2415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/4256693814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 0x7f9a2415bc20 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f9a240b2680 tx=0x7f9a1402a560 comp rx=0 tx=0).stop 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 -- 172.21.15.59:0/4256693814 shutdown_connections 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/4256693814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a240d1a60 0x7f9a240cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/4256693814 >> v2:172.21.15.153:3300/0 conn(0x7f9a240b2fa0 0x7f9a240d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/4256693814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 0x7f9a2415bc20 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 -- 172.21.15.59:0/4256693814 >> 172.21.15.59:0/4256693814 conn(0x7f9a24073e10 msgr2=0x7f9a24074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 -- 172.21.15.59:0/4256693814 shutdown_connections 2024-06-29T16:01:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.750+0000 7f9a28b7f640 1 -- 172.21.15.59:0/4256693814 wait complete. 2024-06-29T16:01:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.752+0000 7f9a28b7f640 1 Processor -- start 2024-06-29T16:01:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.752+0000 7f9a28b7f640 1 -- start start 2024-06-29T16:01:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a28b7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 0x7f9a2415d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a28b7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a240b2fa0 0x7f9a241f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a28b7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a240d1a60 0x7f9a241f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a28b7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9a240cfe40 con 0x7f9a240b2270 2024-06-29T16:01:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a28b7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9a240cfcc0 con 0x7f9a240d1a60 2024-06-29T16:01:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a28b7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9a240cffc0 con 0x7f9a240b2fa0 2024-06-29T16:01:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 0x7f9a2415d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 0x7f9a2415d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44258/0 (socket says 172.21.15.59:44258) 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 -- 172.21.15.59:0/579765384 learned_addr learned my addr 172.21.15.59:0/579765384 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a22d9d640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a240d1a60 0x7f9a241f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a21d9b640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a240b2fa0 0x7f9a241f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 -- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a240b2fa0 msgr2=0x7f9a241f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a240b2fa0 0x7f9a241f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 -- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a240d1a60 msgr2=0x7f9a241f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a240d1a60 0x7f9a241f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 -- 172.21.15.59:0/579765384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2415ed00 con 0x7f9a240b2270 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a21d9b640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a240b2fa0 0x7f9a241f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a22d9d640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a240d1a60 0x7f9a241f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.754+0000 7f9a2259c640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 0x7f9a2415d410 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f9a14000c00 tx=0x7f9a140038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.755+0000 7f9a12ffd640 1 -- 172.21.15.59:0/579765384 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1403e030 con 0x7f9a240b2270 2024-06-29T16:01:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.755+0000 7f9a12ffd640 1 -- 172.21.15.59:0/579765384 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a14003ca0 con 0x7f9a240b2270 2024-06-29T16:01:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.755+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2415dc60 con 0x7f9a240b2270 2024-06-29T16:01:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.755+0000 7f9a12ffd640 1 -- 172.21.15.59:0/579765384 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14035040 con 0x7f9a240b2270 2024-06-29T16:01:25.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.755+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2415e220 con 0x7f9a240b2270 2024-06-29T16:01:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.756+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a240d0450 con 0x7f9a240b2270 2024-06-29T16:01:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.756+0000 7f9a12ffd640 1 -- 172.21.15.59:0/579765384 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f9a14041030 con 0x7f9a240b2270 2024-06-29T16:01:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.757+0000 7f9a12ffd640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9a04076420 0x7f9a04078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.757+0000 7f9a12ffd640 1 -- 172.21.15.59:0/579765384 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f9a140b8290 con 0x7f9a240b2270 2024-06-29T16:01:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.757+0000 7f9a21d9b640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9a04076420 0x7f9a04078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.758+0000 7f9a21d9b640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9a04076420 0x7f9a04078930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9a240cfbd0 tx=0x7f9a18005f90 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:25.761+0000 7f9a12ffd640 1 -- 172.21.15.59:0/579765384 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a14081720 con 0x7f9a240b2270 2024-06-29T16:01:26.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.024+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) -- 0x7f9a240cf920 con 0x7f9a240b2270 2024-06-29T16:01:26.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.497+0000 7f9a12ffd640 1 -- 172.21.15.59:0/579765384 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3317) ==== 201+0+0 (secure 0 0 0) 0x7f9a14081020 con 0x7f9a240b2270 2024-06-29T16:01:26.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9a04076420 msgr2=0x7f9a04078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9a04076420 0x7f9a04078930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9a240cfbd0 tx=0x7f9a18005f90 comp rx=0 tx=0).stop 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 msgr2=0x7f9a2415d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 0x7f9a2415d410 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f9a14000c00 tx=0x7f9a140038a0 comp rx=0 tx=0).stop 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 shutdown_connections 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9a04076420 0x7f9a04078930 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a240d1a60 0x7f9a241f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9a240b2fa0 0x7f9a241f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 --2- 172.21.15.59:0/579765384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a240b2270 0x7f9a2415d410 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.499+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 >> 172.21.15.59:0/579765384 conn(0x7f9a24073e10 msgr2=0x7f9a24074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.500+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 shutdown_connections 2024-06-29T16:01:26.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:26.500+0000 7f9a28b7f640 1 -- 172.21.15.59:0/579765384 wait complete. 2024-06-29T16:01:26.515 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. ... ERROR 2024-06-29T16:01:26.515 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:26.515 INFO:tasks.cephfs_test_runner:====================================================================== 2024-06-29T16:01:26.515 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T16:01:26.515 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-06-29T16:01:26.515 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs/test_snap_schedules.py", line 684, in test_snap_schedule_subvol_and_group_arguments_01 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: self._fs_cmd('subvolume', 'rm', 'cephfs', 'sv01') 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs/test_snap_schedules.py", line 63, in _fs_cmd 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_manager.py", line 1652, in run_cluster_cmd 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/remote.py", line 523, in run 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 455, in run 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: r.wait() 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 161, in wait 2024-06-29T16:01:26.516 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01' 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:Ran 15 tests in 5926.619s 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:====================================================================== 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-06-29T16:01:26.517 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs/test_snap_schedules.py", line 684, in test_snap_schedule_subvol_and_group_arguments_01 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: self._fs_cmd('subvolume', 'rm', 'cephfs', 'sv01') 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs/test_snap_schedules.py", line 63, in _fs_cmd 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_manager.py", line 1652, in run_cluster_cmd 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/remote.py", line 523, in run 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 455, in run 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: r.wait() 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 161, in wait 2024-06-29T16:01:26.518 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-06-29T16:01:26.519 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-06-29T16:01:26.519 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-06-29T16:01:26.519 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01' 2024-06-29T16:01:26.519 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:26.519 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T16:01:26.691 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=729a7f032ddd4e97bf5f1f5d056634a1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T16:01:26.693 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-06-29T16:01:26.701 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-06-29T16:01:26.735 INFO:tasks.kclient:Unmounting kernel clients... 2024-06-29T16:01:26.735 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T16:01:26.736 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:26.758 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T16:01:26.758 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-06-29T16:01:26.758 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:26.818 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057099,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17709 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249424k,nr_inodes=812356,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498852k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T16:01:26.818 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T16:01:26.818 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:26.839 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T16:01:26.839 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-29T16:01:26.839 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:26.898 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18904 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T16:01:26.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:26.899 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T16:01:26.918 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T16:01:26.918 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T16:01:26.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:26.919 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T16:01:26.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:26.990 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T16:01:27.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.062 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2024-06-29T16:01:27.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.161 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-06-29T16:01:27.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.178 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2024-06-29T16:01:27.249 INFO:teuthology.orchestra.run.smithi059.stderr:Cannot find device "ceph-brx" 2024-06-29T16:01:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:01:27.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.251 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-06-29T16:01:27.271 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-06-29T16:01:27.271 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-06-29T16:01:27.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-06-29T16:01:27.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-06-29T16:01:27.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-06-29T16:01:27.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.513 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-06-29T16:01:27.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:27.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-06-29T16:01:27.596 INFO:teuthology.orchestra.run.smithi153.stderr:Cannot find device "ceph-brx" 2024-06-29T16:01:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:01:27.598 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-06-29T16:01:27.607 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-29T16:01:27.608 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:27.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.731+0000 7f2d69864640 1 -- 172.21.15.59:0/3072165168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d6406c1e0 msgr2=0x7f2d6406c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.731+0000 7f2d69864640 1 --2- 172.21.15.59:0/3072165168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d6406c1e0 0x7f2d6406c5f0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c005fd0 tx=0x7f2d5c02b980 comp rx=0 tx=0).stop 2024-06-29T16:01:27.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.731+0000 7f2d69864640 1 -- 172.21.15.59:0/3072165168 shutdown_connections 2024-06-29T16:01:27.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.731+0000 7f2d69864640 1 --2- 172.21.15.59:0/3072165168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d6406c1e0 0x7f2d6406c5f0 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.731+0000 7f2d69864640 1 --2- 172.21.15.59:0/3072165168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d64073f30 0x7f2d6415a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.731+0000 7f2d69864640 1 -- 172.21.15.59:0/3072165168 >> 172.21.15.59:0/3072165168 conn(0x7f2d640b24c0 msgr2=0x7f2d640cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.731+0000 7f2d69864640 1 -- 172.21.15.59:0/3072165168 shutdown_connections 2024-06-29T16:01:27.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.731+0000 7f2d69864640 1 -- 172.21.15.59:0/3072165168 wait complete. 2024-06-29T16:01:27.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.733+0000 7f2d69864640 1 Processor -- start 2024-06-29T16:01:27.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.733+0000 7f2d69864640 1 -- start start 2024-06-29T16:01:27.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d69864640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d6406c1e0 0x7f2d64149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d69864640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d64073f30 0x7f2d641e5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d69864640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640cf120 0x7f2d64155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d69864640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f2d640cce40 con 0x7f2d640cf120 2024-06-29T16:01:27.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d69864640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d640cccc0 con 0x7f2d6406c1e0 2024-06-29T16:01:27.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d69864640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2d640ccfc0 con 0x7f2d64073f30 2024-06-29T16:01:27.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640cf120 0x7f2d64155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640cf120 0x7f2d64155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44280/0 (socket says 172.21.15.59:44280) 2024-06-29T16:01:27.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 -- 172.21.15.59:0/673796905 learned_addr learned my addr 172.21.15.59:0/673796905 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:27.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d62ffd640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d6406c1e0 0x7f2d64149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d627fc640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d64073f30 0x7f2d641e5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 -- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d64073f30 msgr2=0x7f2d641e5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d64073f30 0x7f2d641e5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 -- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d6406c1e0 msgr2=0x7f2d64149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d6406c1e0 0x7f2d64149530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 -- 172.21.15.59:0/673796905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d641e56b0 con 0x7f2d640cf120 2024-06-29T16:01:27.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d62ffd640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d6406c1e0 0x7f2d64149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:27.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d637fe640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640cf120 0x7f2d64155550 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c034040 tx=0x7f2d5c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d5bfff640 1 -- 172.21.15.59:0/673796905 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c035050 con 0x7f2d640cf120 2024-06-29T16:01:27.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d5bfff640 1 -- 172.21.15.59:0/673796905 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d5c004540 con 0x7f2d640cf120 2024-06-29T16:01:27.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d641e5ca0 con 0x7f2d640cf120 2024-06-29T16:01:27.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.735+0000 7f2d5bfff640 1 -- 172.21.15.59:0/673796905 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c045480 con 0x7f2d640cf120 2024-06-29T16:01:27.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.736+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d641e6260 con 0x7f2d640cf120 2024-06-29T16:01:27.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.737+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d6406c5f0 con 0x7f2d640cf120 2024-06-29T16:01:27.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.737+0000 7f2d5bfff640 1 -- 172.21.15.59:0/673796905 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f2d5c038020 con 0x7f2d640cf120 2024-06-29T16:01:27.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.738+0000 7f2d5bfff640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2d440763f0 0x7f2d44078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.738+0000 7f2d5bfff640 1 -- 172.21.15.59:0/673796905 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f2d5c0b91a0 con 0x7f2d640cf120 2024-06-29T16:01:27.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.738+0000 7f2d62ffd640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2d440763f0 0x7f2d44078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.739+0000 7f2d62ffd640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2d440763f0 0x7f2d44078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2d6406f700 tx=0x7f2d54005f10 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.742+0000 7f2d5bfff640 1 -- 172.21.15.59:0/673796905 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d5c082870 con 0x7f2d640cf120 2024-06-29T16:01:27.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.928+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2d64068980 con 0x7f2d440763f0 2024-06-29T16:01:27.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.930+0000 7f2d5bfff640 1 -- 172.21.15.59:0/673796905 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13515 (secure 0 0 0) 0x7f2d64068980 con 0x7f2d440763f0 2024-06-29T16:01:27.931 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:27.931 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T16:01:27.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.931+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2d440763f0 msgr2=0x7f2d44078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.931+0000 7f2d69864640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2d440763f0 0x7f2d44078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2d6406f700 tx=0x7f2d54005f10 comp rx=0 tx=0).stop 2024-06-29T16:01:27.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640cf120 msgr2=0x7f2d64155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640cf120 0x7f2d64155550 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c034040 tx=0x7f2d5c003f10 comp rx=0 tx=0).stop 2024-06-29T16:01:27.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 shutdown_connections 2024-06-29T16:01:27.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f2d440763f0 0x7f2d44078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640cf120 0x7f2d64155550 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d64073f30 0x7f2d641e5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 --2- 172.21.15.59:0/673796905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d6406c1e0 0x7f2d64149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 >> 172.21.15.59:0/673796905 conn(0x7f2d640b24c0 msgr2=0x7f2d640b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 shutdown_connections 2024-06-29T16:01:27.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:27.932+0000 7f2d69864640 1 -- 172.21.15.59:0/673796905 wait complete. 2024-06-29T16:01:27.943 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-06-29T16:01:26.112671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1848,"num_read_kb":5320,"num_write":457,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":432596,"kb_used_data":29028,"kb_used_omap":12,"kb_used_meta":403443,"kb_avail":754542124,"statfs":{"total":773094113280,"available":772651134976,"internally_reserved":0,"allocated":29724672,"data_stored":16020723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":413126194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-803664,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-536,"num_read_kb":-476,"num_write":-444,"num_write_kb":-3000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.998980"},"pg_stats":[{"pgid":"1.0","version":"276'280","reported_seq":1381,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-29T16:01:20.100710+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T16:01:20.100710+0000","last_peered":"2024-06-29T16:01:20.100710+0000","last_clean":"2024-06-29T16:01:20.100710+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T16:01:20.100710+0000","last_undegraded":"2024-06-29T16:01:20.100710+0000","last_fullsized":"2024-06-29T16:01:20.100710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1848,"num_read_kb":5320,"num_write":457,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1848,"num_read_kb":5320,"num_write":457,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543331,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53268,"kb_used_data":4040,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318572,"statfs":{"total":96636764160,"available":96582217728,"internally_reserved":0,"allocated":4136960,"data_stored":2429927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543333,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48564,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323276,"statfs":{"total":96636764160,"available":96587034624,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543332,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70760,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301080,"statfs":{"total":96636764160,"available":96564305920,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543331,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510629,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52712,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319128,"statfs":{"total":96636764160,"available":96582787072,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576037,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510628,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52728,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319112,"statfs":{"total":96636764160,"available":96582770688,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510630,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49132,"kb_used_data":4060,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322708,"statfs":{"total":96636764160,"available":96586452992,"internally_reserved":0,"allocated":4157440,"data_stored":2444599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:27.944 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:28.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.064+0000 7fb8a01eb640 1 -- 172.21.15.59:0/3753691856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb880004800 msgr2=0x7fb880004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.064+0000 7fb8a01eb640 1 --2- 172.21.15.59:0/3753691856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb880004800 0x7fb880004c10 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009b50 tx=0x7fb880005830 comp rx=0 tx=0).stop 2024-06-29T16:01:28.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.064+0000 7fb8a01eb640 1 -- 172.21.15.59:0/3753691856 shutdown_connections 2024-06-29T16:01:28.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.064+0000 7fb8a01eb640 1 --2- 172.21.15.59:0/3753691856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb880004800 0x7fb880004c10 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.064+0000 7fb8a01eb640 1 -- 172.21.15.59:0/3753691856 >> 172.21.15.59:0/3753691856 conn(0x7fb898073e10 msgr2=0x7fb898074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.064+0000 7fb8a01eb640 1 -- 172.21.15.59:0/3753691856 shutdown_connections 2024-06-29T16:01:28.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.064+0000 7fb8a01eb640 1 -- 172.21.15.59:0/3753691856 wait complete. 2024-06-29T16:01:28.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.066+0000 7fb8a01eb640 1 Processor -- start 2024-06-29T16:01:28.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.066+0000 7fb8a01eb640 1 -- start start 2024-06-29T16:01:28.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb8a01eb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb880004800 0x7fb8981507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb8a01eb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb898142bb0 0x7fb898151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb8a01eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb898152440 0x7fb8981f6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb8a01eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fb8980d1730 con 0x7fb898152440 2024-06-29T16:01:28.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb8a01eb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb89815fa60 con 0x7fb898142bb0 2024-06-29T16:01:28.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb8a01eb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb8980d18b0 con 0x7fb880004800 2024-06-29T16:01:28.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb89e788640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb898152440 0x7fb8981f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb89e788640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb898152440 0x7fb8981f6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44302/0 (socket says 172.21.15.59:44302) 2024-06-29T16:01:28.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.067+0000 7fb89e788640 1 -- 172.21.15.59:0/2542764778 learned_addr learned my addr 172.21.15.59:0/2542764778 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:28.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb89d786640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb898142bb0 0x7fb898151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb89df87640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb880004800 0x7fb8981507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb89e788640 1 -- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb880004800 msgr2=0x7fb8981507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb89e788640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb880004800 0x7fb8981507b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb89e788640 1 -- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb898142bb0 msgr2=0x7fb898151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb89e788640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb898142bb0 0x7fb898151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb89e788640 1 -- 172.21.15.59:0/2542764778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb898152850 con 0x7fb898152440 2024-06-29T16:01:28.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb89e788640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb898152440 0x7fb8981f6410 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fb890000c00 tx=0x7fb89000e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb8967fc640 1 -- 172.21.15.59:0/2542764778 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8900107d0 con 0x7fb898152440 2024-06-29T16:01:28.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb8967fc640 1 -- 172.21.15.59:0/2542764778 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb890016070 con 0x7fb898152440 2024-06-29T16:01:28.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb8967fc640 1 -- 172.21.15.59:0/2542764778 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb890010d20 con 0x7fb898152440 2024-06-29T16:01:28.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb898151b10 con 0x7fb898152440 2024-06-29T16:01:28.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.068+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb898151550 con 0x7fb898152440 2024-06-29T16:01:28.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.069+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8980b1b50 con 0x7fb898152440 2024-06-29T16:01:28.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.070+0000 7fb8967fc640 1 -- 172.21.15.59:0/2542764778 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7fb89001b460 con 0x7fb898152440 2024-06-29T16:01:28.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.070+0000 7fb8967fc640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb884076420 0x7fb884078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.071+0000 7fb8967fc640 1 -- 172.21.15.59:0/2542764778 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7fb890097da0 con 0x7fb898152440 2024-06-29T16:01:28.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.071+0000 7fb89df87640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb884076420 0x7fb884078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.071+0000 7fb89df87640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb884076420 0x7fb884078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb888000f80 tx=0x7fb888006ef0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.075+0000 7fb8967fc640 1 -- 172.21.15.59:0/2542764778 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb8980b1b50 con 0x7fb898152440 2024-06-29T16:01:28.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.262+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb89815c000 con 0x7fb884076420 2024-06-29T16:01:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.262+0000 7fb8967fc640 1 -- 172.21.15.59:0/2542764778 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13504 (secure 0 0 0) 0x7fb89815c000 con 0x7fb884076420 2024-06-29T16:01:28.263 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:28.264 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T16:01:28.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb884076420 msgr2=0x7fb884078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb884076420 0x7fb884078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb888000f80 tx=0x7fb888006ef0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb898152440 msgr2=0x7fb8981f6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb898152440 0x7fb8981f6410 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fb890000c00 tx=0x7fb89000e770 comp rx=0 tx=0).stop 2024-06-29T16:01:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 shutdown_connections 2024-06-29T16:01:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fb884076420 0x7fb884078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb898152440 0x7fb8981f6410 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb898142bb0 0x7fb898151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 --2- 172.21.15.59:0/2542764778 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb880004800 0x7fb8981507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 >> 172.21.15.59:0/2542764778 conn(0x7fb898073e10 msgr2=0x7fb8980b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.264+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 shutdown_connections 2024-06-29T16:01:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.265+0000 7fb8a01eb640 1 -- 172.21.15.59:0/2542764778 wait complete. 2024-06-29T16:01:28.276 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-06-29T16:01:28.113103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":432620,"kb_used_data":29052,"kb_used_omap":12,"kb_used_meta":403443,"kb_avail":754542100,"statfs":{"total":773094113280,"available":772651110400,"internally_reserved":0,"allocated":29749248,"data_stored":16035545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":413126194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-803664,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-476,"num_read_kb":-300,"num_write":-432,"num_write_kb":-3000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001419"},"pg_stats":[{"pgid":"1.0","version":"280'287","reported_seq":1410,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-06-29T16:01:25.101516+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T16:01:25.101516+0000","last_peered":"2024-06-29T16:01:25.101516+0000","last_clean":"2024-06-29T16:01:25.101516+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T16:01:25.101516+0000","last_undegraded":"2024-06-29T16:01:25.101516+0000","last_fullsized":"2024-06-29T16:01:25.101516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543332,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53292,"kb_used_data":4064,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318548,"statfs":{"total":96636764160,"available":96582193152,"internally_reserved":0,"allocated":4161536,"data_stored":2444749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543333,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48564,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323276,"statfs":{"total":96636764160,"available":96587034624,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543332,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70760,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301080,"statfs":{"total":96636764160,"available":96564305920,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543331,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510629,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52712,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319128,"statfs":{"total":96636764160,"available":96582787072,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576037,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510628,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52728,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319112,"statfs":{"total":96636764160,"available":96582770688,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510630,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49132,"kb_used_data":4060,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322708,"statfs":{"total":96636764160,"available":96586452992,"internally_reserved":0,"allocated":4157440,"data_stored":2444599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:28.277 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-29T16:01:28.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:28.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.393+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4110508194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cb490 msgr2=0x7f9fdc073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.393+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4110508194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cb490 0x7f9fdc073be0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0b5650 tx=0x7f9fcc02a530 comp rx=0 tx=0).stop 2024-06-29T16:01:28.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.393+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4110508194 shutdown_connections 2024-06-29T16:01:28.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.393+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4110508194 >> v2:172.21.15.153:3300/0 conn(0x7f9fdc0ad6b0 0x7f9fdc15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.393+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4110508194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cb490 0x7f9fdc073be0 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.393+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4110508194 >> 172.21.15.59:0/4110508194 conn(0x7f9fdc0b1ea0 msgr2=0x7f9fdc0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.393+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4110508194 shutdown_connections 2024-06-29T16:01:28.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.393+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4110508194 wait complete. 2024-06-29T16:01:28.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.395+0000 7f9fe29b2640 1 Processor -- start 2024-06-29T16:01:28.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.395+0000 7f9fe29b2640 1 -- start start 2024-06-29T16:01:28.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.396+0000 7f9fe29b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc0cb490 0x7f9fdc14fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.396+0000 7f9fe29b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cc060 0x7f9fdc152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.396+0000 7f9fe29b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc154e80 0x7f9fdc1533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.396+0000 7f9fe29b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc15bb20 con 0x7f9fdc0cc060 2024-06-29T16:01:28.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.396+0000 7f9fe29b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc15b9a0 con 0x7f9fdc154e80 2024-06-29T16:01:28.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.396+0000 7f9fe29b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9fdc15bca0 con 0x7f9fdc0cb490 2024-06-29T16:01:28.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.396+0000 7f9fdb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cc060 0x7f9fdc152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cc060 0x7f9fdc152eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44330/0 (socket says 172.21.15.59:44330) 2024-06-29T16:01:28.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdb7fe640 1 -- 172.21.15.59:0/4249041885 learned_addr learned my addr 172.21.15.59:0/4249041885 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:28.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdbfff640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc0cb490 0x7f9fdc14fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fe0f4f640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc154e80 0x7f9fdc1533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdb7fe640 1 -- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc0cb490 msgr2=0x7f9fdc14fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdb7fe640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc0cb490 0x7f9fdc14fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdb7fe640 1 -- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc154e80 msgr2=0x7f9fdc1533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdb7fe640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc154e80 0x7f9fdc1533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdb7fe640 1 -- 172.21.15.59:0/4249041885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc1553b0 con 0x7f9fdc0cc060 2024-06-29T16:01:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fdb7fe640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cc060 0x7f9fdc152eb0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000c3c0 tx=0x7f9fd000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fd8ff9640 1 -- 172.21.15.59:0/4249041885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0017070 con 0x7f9fdc0cc060 2024-06-29T16:01:28.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fd8ff9640 1 -- 172.21.15.59:0/4249041885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fd00035f0 con 0x7f9fdc0cc060 2024-06-29T16:01:28.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fdc155690 con 0x7f9fdc0cc060 2024-06-29T16:01:28.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fd8ff9640 1 -- 172.21.15.59:0/4249041885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0003a80 con 0x7f9fdc0cc060 2024-06-29T16:01:28.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.397+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fdc155c50 con 0x7f9fdc0cc060 2024-06-29T16:01:28.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.398+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fdc15b140 con 0x7f9fdc0cc060 2024-06-29T16:01:28.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.399+0000 7f9fd8ff9640 1 -- 172.21.15.59:0/4249041885 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f9fd001a070 con 0x7f9fdc0cc060 2024-06-29T16:01:28.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.400+0000 7f9fd8ff9640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9fc8076420 0x7f9fc8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.400+0000 7f9fd8ff9640 1 -- 172.21.15.59:0/4249041885 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f9fd0096dd0 con 0x7f9fdc0cc060 2024-06-29T16:01:28.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.400+0000 7f9fdbfff640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9fc8076420 0x7f9fc8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.400+0000 7f9fdbfff640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9fc8076420 0x7f9fc8078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc034000 tx=0x7f9fcc004730 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.404+0000 7f9fd8ff9640 1 -- 172.21.15.59:0/4249041885 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9fd0060420 con 0x7f9fdc0cc060 2024-06-29T16:01:28.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.593+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9fdc0cc780 con 0x7f9fc8076420 2024-06-29T16:01:28.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.594+0000 7f9fd8ff9640 1 -- 172.21.15.59:0/4249041885 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13504 (secure 0 0 0) 0x7f9fdc0cc780 con 0x7f9fc8076420 2024-06-29T16:01:28.596 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:28.596 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T16:01:28.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9fc8076420 msgr2=0x7f9fc8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9fc8076420 0x7f9fc8078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc034000 tx=0x7f9fcc004730 comp rx=0 tx=0).stop 2024-06-29T16:01:28.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cc060 msgr2=0x7f9fdc152eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cc060 0x7f9fdc152eb0 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000c3c0 tx=0x7f9fd000c880 comp rx=0 tx=0).stop 2024-06-29T16:01:28.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 shutdown_connections 2024-06-29T16:01:28.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9fc8076420 0x7f9fc8078930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fdc154e80 0x7f9fdc1533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fdc0cc060 0x7f9fdc152eb0 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 --2- 172.21.15.59:0/4249041885 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9fdc0cb490 0x7f9fdc14fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 >> 172.21.15.59:0/4249041885 conn(0x7f9fdc0b1ea0 msgr2=0x7f9fdc15a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 shutdown_connections 2024-06-29T16:01:28.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.597+0000 7f9fe29b2640 1 -- 172.21.15.59:0/4249041885 wait complete. 2024-06-29T16:01:28.608 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-06-29T16:01:28.113103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":432620,"kb_used_data":29052,"kb_used_omap":12,"kb_used_meta":403443,"kb_avail":754542100,"statfs":{"total":773094113280,"available":772651110400,"internally_reserved":0,"allocated":29749248,"data_stored":16035545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":413126194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-803664,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-476,"num_read_kb":-300,"num_write":-432,"num_write_kb":-3000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001419"},"pg_stats":[{"pgid":"1.0","version":"280'287","reported_seq":1410,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-06-29T16:01:25.101516+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T16:01:25.101516+0000","last_peered":"2024-06-29T16:01:25.101516+0000","last_clean":"2024-06-29T16:01:25.101516+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T16:01:25.101516+0000","last_undegraded":"2024-06-29T16:01:25.101516+0000","last_fullsized":"2024-06-29T16:01:25.101516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543332,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53292,"kb_used_data":4064,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318548,"statfs":{"total":96636764160,"available":96582193152,"internally_reserved":0,"allocated":4161536,"data_stored":2444749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543333,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48564,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323276,"statfs":{"total":96636764160,"available":96587034624,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543332,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70760,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301080,"statfs":{"total":96636764160,"available":96564305920,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543331,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510629,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52712,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319128,"statfs":{"total":96636764160,"available":96582787072,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576037,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":3480,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":3563520,"data_stored":1854231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510628,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52728,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319112,"statfs":{"total":96636764160,"available":96582770688,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510630,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49132,"kb_used_data":4060,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322708,"statfs":{"total":96636764160,"available":96586452992,"internally_reserved":0,"allocated":4157440,"data_stored":2444599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:28.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.725+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/31778838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340b40a0 msgr2=0x7f7b3415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.725+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/31778838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340b40a0 0x7f7b3415e8d0 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f7b34161d40 tx=0x7f7b2402a530 comp rx=0 tx=0).stop 2024-06-29T16:01:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.725+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/31778838 shutdown_connections 2024-06-29T16:01:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.725+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/31778838 >> v2:172.21.15.153:3300/0 conn(0x7f7b3415ee10 0x7f7b3415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.725+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/31778838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340b40a0 0x7f7b3415e8d0 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.725+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/31778838 >> 172.21.15.59:0/31778838 conn(0x7f7b340d3ee0 msgr2=0x7f7b340d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.725+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/31778838 shutdown_connections 2024-06-29T16:01:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.725+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/31778838 wait complete. 2024-06-29T16:01:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.727+0000 7f7b3bd9a640 1 Processor -- start 2024-06-29T16:01:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.727+0000 7f7b3bd9a640 1 -- start start 2024-06-29T16:01:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3bd9a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b340d23c0 0x7f7b341f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3bd9a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b3415ee10 0x7f7b34144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3bd9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340c9c30 0x7f7b340ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3bd9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7b34160280 con 0x7f7b340c9c30 2024-06-29T16:01:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3bd9a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7b34160100 con 0x7f7b340d23c0 2024-06-29T16:01:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3bd9a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7b34160400 con 0x7f7b3415ee10 2024-06-29T16:01:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340c9c30 0x7f7b340ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340c9c30 0x7f7b340ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44350/0 (socket says 172.21.15.59:44350) 2024-06-29T16:01:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 -- 172.21.15.59:0/1320193759 learned_addr learned my addr 172.21.15.59:0/1320193759 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b39b36640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b340d23c0 0x7f7b341f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b39335640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b3415ee10 0x7f7b34144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 -- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b3415ee10 msgr2=0x7f7b34144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b3415ee10 0x7f7b34144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 -- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b340d23c0 msgr2=0x7f7b341f4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b340d23c0 0x7f7b341f4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 -- 172.21.15.59:0/1320193759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3414fab0 con 0x7f7b340c9c30 2024-06-29T16:01:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b39335640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b3415ee10 0x7f7b34144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3a337640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340c9c30 0x7f7b340ca040 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00ca90 tx=0x7f7b2c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b2a7fc640 1 -- 172.21.15.59:0/1320193759 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c007d30 con 0x7f7b340c9c30 2024-06-29T16:01:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b2a7fc640 1 -- 172.21.15.59:0/1320193759 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b2c00f450 con 0x7f7b340c9c30 2024-06-29T16:01:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3414fd90 con 0x7f7b340c9c30 2024-06-29T16:01:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.729+0000 7f7b2a7fc640 1 -- 172.21.15.59:0/1320193759 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c00f880 con 0x7f7b340c9c30 2024-06-29T16:01:28.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.730+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b34150350 con 0x7f7b340c9c30 2024-06-29T16:01:28.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.730+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3415e8d0 con 0x7f7b340c9c30 2024-06-29T16:01:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.731+0000 7f7b2a7fc640 1 -- 172.21.15.59:0/1320193759 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f7b2c00fa20 con 0x7f7b340c9c30 2024-06-29T16:01:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.732+0000 7f7b2a7fc640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f7b18076420 0x7f7b18078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.732+0000 7f7b2a7fc640 1 -- 172.21.15.59:0/1320193759 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f7b2c0985d0 con 0x7f7b340c9c30 2024-06-29T16:01:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.732+0000 7f7b39b36640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f7b18076420 0x7f7b18078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.733+0000 7f7b39b36640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f7b18076420 0x7f7b18078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7b340d2e40 tx=0x7f7b30009040 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.736+0000 7f7b2a7fc640 1 -- 172.21.15.59:0/1320193759 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7b2c014030 con 0x7f7b340c9c30 2024-06-29T16:01:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.929+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b340d3830 con 0x7f7b340c9c30 2024-06-29T16:01:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.930+0000 7f7b2a7fc640 1 -- 172.21.15.59:0/1320193759 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v283) ==== 74+0+14656 (secure 0 0 0) 0x7f7b2c061a70 con 0x7f7b340c9c30 2024-06-29T16:01:28.931 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:28.931 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":283,"fsid":"150b8fd8-1bf5-44f0-b8d0-0083a48c0071","created":"2024-06-29T14:21:29.249177+0000","modified":"2024-06-29T16:01:25.274417+0000","last_up_change":"2024-06-29T14:21:41.454591+0000","last_in_change":"2024-06-29T14:21:32.552755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:21:41.747059+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f02c84e6-3893-40f1-a178-d9b3cafe2129","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6809","nonce":1075583359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6811","nonce":1075583359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6815","nonce":1075583359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1075583359},{"type":"v1","addr":"172.21.15.59:6813","nonce":1075583359}]},"public_addr":"172.21.15.59:6809/1075583359","cluster_addr":"172.21.15.59:6811/1075583359","heartbeat_back_addr":"172.21.15.59:6815/1075583359","heartbeat_front_addr":"172.21.15.59:6813/1075583359","state":["exists","up"]},{"osd":1,"uuid":"1fd5fe92-f046-4d7f-b0e5-6caecf0c6201","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6825","nonce":1021101403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6827","nonce":1021101403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6831","nonce":1021101403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":1021101403},{"type":"v1","addr":"172.21.15.59:6829","nonce":1021101403}]},"public_addr":"172.21.15.59:6825/1021101403","cluster_addr":"172.21.15.59:6827/1021101403","heartbeat_back_addr":"172.21.15.59:6831/1021101403","heartbeat_front_addr":"172.21.15.59:6829/1021101403","state":["exists","up"]},{"osd":2,"uuid":"e9c69d5d-fbda-4514-a952-731694e1abb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6801","nonce":2051957706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6803","nonce":2051957706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6807","nonce":2051957706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2051957706},{"type":"v1","addr":"172.21.15.59:6805","nonce":2051957706}]},"public_addr":"172.21.15.59:6801/2051957706","cluster_addr":"172.21.15.59:6803/2051957706","heartbeat_back_addr":"172.21.15.59:6807/2051957706","heartbeat_front_addr":"172.21.15.59:6805/2051957706","state":["exists","up"]},{"osd":3,"uuid":"aea314fe-1780-4b91-8f6c-b2b5125046b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6817","nonce":2018446268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6819","nonce":2018446268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6823","nonce":2018446268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2018446268},{"type":"v1","addr":"172.21.15.59:6821","nonce":2018446268}]},"public_addr":"172.21.15.59:6817/2018446268","cluster_addr":"172.21.15.59:6819/2018446268","heartbeat_back_addr":"172.21.15.59:6823/2018446268","heartbeat_front_addr":"172.21.15.59:6821/2018446268","state":["exists","up"]},{"osd":4,"uuid":"5532cdeb-7ff7-4174-bf7a-7f3f90cc4009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":277,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6801","nonce":2158079959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6803","nonce":2158079959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6807","nonce":2158079959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2158079959},{"type":"v1","addr":"172.21.15.153:6805","nonce":2158079959}]},"public_addr":"172.21.15.153:6801/2158079959","cluster_addr":"172.21.15.153:6803/2158079959","heartbeat_back_addr":"172.21.15.153:6807/2158079959","heartbeat_front_addr":"172.21.15.153:6805/2158079959","state":["exists","up"]},{"osd":5,"uuid":"aea8df17-2e12-4559-bd03-8c496f34482c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6825","nonce":4044954575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6827","nonce":4044954575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6831","nonce":4044954575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":4044954575},{"type":"v1","addr":"172.21.15.153:6829","nonce":4044954575}]},"public_addr":"172.21.15.153:6825/4044954575","cluster_addr":"172.21.15.153:6827/4044954575","heartbeat_back_addr":"172.21.15.153:6831/4044954575","heartbeat_front_addr":"172.21.15.153:6829/4044954575","state":["exists","up"]},{"osd":6,"uuid":"737193be-7224-4e5f-9857-bc0349f5a29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6809","nonce":4247178163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6811","nonce":4247178163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6815","nonce":4247178163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4247178163},{"type":"v1","addr":"172.21.15.153:6813","nonce":4247178163}]},"public_addr":"172.21.15.153:6809/4247178163","cluster_addr":"172.21.15.153:6811/4247178163","heartbeat_back_addr":"172.21.15.153:6815/4247178163","heartbeat_front_addr":"172.21.15.153:6813/4247178163","state":["exists","up"]},{"osd":7,"uuid":"5fe7aecb-2572-4365-9ee9-9f272916b7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":273,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6817","nonce":1688984748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6819","nonce":1688984748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6823","nonce":1688984748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1688984748},{"type":"v1","addr":"172.21.15.153:6821","nonce":1688984748}]},"public_addr":"172.21.15.153:6817/1688984748","cluster_addr":"172.21.15.153:6819/1688984748","heartbeat_back_addr":"172.21.15.153:6823/1688984748","heartbeat_front_addr":"172.21.15.153:6821/1688984748","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.699517+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.755511+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.721882+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.733890+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672611+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.800895+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.659541+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:21:37.672437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6833/3706652401":"2024-06-30T16:01:22.484588+0000","172.21.15.153:0/3174351846":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/764992950":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3398970025":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3498108787":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2338595954":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/192063323":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3360291833":"2024-06-30T16:01:20.092843+0000","172.21.15.153:6833/2469779970":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2700705605":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/2302272425":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1710208131":"2024-06-30T15:59:57.978964+0000","172.21.15.59:6835/3706652401":"2024-06-30T16:01:22.484588+0000","172.21.15.153:6833/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6832/3072370296":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1857217640":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/2749019736":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/3489900589":"2024-06-30T15:59:57.978964+0000","172.21.15.153:6832/2469779970":"2024-06-30T16:01:20.092843+0000","172.21.15.153:0/1124076092":"2024-06-30T15:59:57.978964+0000","172.21.15.153:0/1986019092":"2024-06-30T15:59:57.978964+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":38,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f7b18076420 msgr2=0x7f7b18078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f7b18076420 0x7f7b18078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7b340d2e40 tx=0x7f7b30009040 comp rx=0 tx=0).stop 2024-06-29T16:01:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340c9c30 msgr2=0x7f7b340ca040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340c9c30 0x7f7b340ca040 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00ca90 tx=0x7f7b2c00cf50 comp rx=0 tx=0).stop 2024-06-29T16:01:28.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 shutdown_connections 2024-06-29T16:01:28.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f7b18076420 0x7f7b18078930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b340c9c30 0x7f7b340ca040 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7b3415ee10 0x7f7b34144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 --2- 172.21.15.59:0/1320193759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b340d23c0 0x7f7b341f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 >> 172.21.15.59:0/1320193759 conn(0x7f7b340d3ee0 msgr2=0x7f7b340ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 shutdown_connections 2024-06-29T16:01:28.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:28.932+0000 7f7b3bd9a640 1 -- 172.21.15.59:0/1320193759 wait complete. 2024-06-29T16:01:29.945 INFO:tasks.ceph:Scrubbing osd.0 2024-06-29T16:01:29.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.082+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/90684793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9af0003fd0 msgr2=0x7f9af0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.082+0000 7f9b0e3fc640 1 --2- 172.21.15.59:0/90684793 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9af0003fd0 0x7f9af0004800 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7f9afc001eb0 tx=0x7f9af0005830 comp rx=0 tx=0).stop 2024-06-29T16:01:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.082+0000 7f9b0c999640 1 -- 172.21.15.59:0/90684793 reap_dead start 2024-06-29T16:01:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.082+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/90684793 shutdown_connections 2024-06-29T16:01:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.082+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/90684793 >> 172.21.15.59:0/90684793 conn(0x7f9b080d4070 msgr2=0x7f9b080d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.082+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/90684793 shutdown_connections 2024-06-29T16:01:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.082+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/90684793 wait complete. 2024-06-29T16:01:30.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.084+0000 7f9b0e3fc640 1 Processor -- start 2024-06-29T16:01:30.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.084+0000 7f9b0e3fc640 1 -- start start 2024-06-29T16:01:30.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.085+0000 7f9b0e3fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9af0003fd0 0x7f9b0814adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.085+0000 7f9b0e3fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b08153cc0 0x7f9b08156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.085+0000 7f9b0e3fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b081f5120 0x7f9b081607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0e3fc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f9b080b2fd0 con 0x7f9b081f5120 2024-06-29T16:01:30.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0e3fc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b080b2e50 con 0x7f9b08153cc0 2024-06-29T16:01:30.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0e3fc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9b080b3150 con 0x7f9af0003fd0 2024-06-29T16:01:30.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b081f5120 0x7f9b081607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b081f5120 0x7f9b081607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44358/0 (socket says 172.21.15.59:44358) 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 -- 172.21.15.59:0/3076285346 learned_addr learned my addr 172.21.15.59:0/3076285346 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b077fe640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b08153cc0 0x7f9b08156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b07fff640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9af0003fd0 0x7f9b0814adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 -- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9af0003fd0 msgr2=0x7f9b0814adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f9af0003fd0 0x7f9b0814adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 -- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b08153cc0 msgr2=0x7f9b08156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b08153cc0 0x7f9b08156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 -- 172.21.15.59:0/3076285346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b080cc830 con 0x7f9b081f5120 2024-06-29T16:01:30.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0c999640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b081f5120 0x7f9b081607a0 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f9b00005fd0 tx=0x7f9b00011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00004b20 con 0x7f9b081f5120 2024-06-29T16:01:30.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b00004cc0 con 0x7f9b081f5120 2024-06-29T16:01:30.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b080ccd30 con 0x7f9b081f5120 2024-06-29T16:01:30.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.086+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00003bd0 con 0x7f9b081f5120 2024-06-29T16:01:30.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.087+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b08155b50 con 0x7f9b081f5120 2024-06-29T16:01:30.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.087+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b080b4920 con 0x7f9b081f5120 2024-06-29T16:01:30.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.088+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f9b00004030 con 0x7f9b081f5120 2024-06-29T16:01:30.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.089+0000 7f9b04ff9640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9af40763f0 0x7f9af4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.089+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f9b00098d90 con 0x7f9b081f5120 2024-06-29T16:01:30.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.089+0000 7f9b04ff9640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7f9af407d020 0x7f9af407f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.089+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 --> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b00022e50 con 0x7f9af407d020 2024-06-29T16:01:30.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.089+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f9b00099220 con 0x7f9b081f5120 2024-06-29T16:01:30.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.089+0000 7f9b07fff640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9af40763f0 0x7f9af4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.089+0000 7f9b077fe640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7f9af407d020 0x7f9af407f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.090+0000 7f9b077fe640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7f9af407d020 0x7f9af407f510 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.090+0000 7f9b07fff640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9af40763f0 0x7f9af4078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009bb0 tx=0x7f9af8005f90 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.090+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 <== osd.0 v2:172.21.15.59:6808/1075583359 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b00022e50 con 0x7f9af407d020 2024-06-29T16:01:30.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.104+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 --> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9b080ccae0 con 0x7f9af407d020 2024-06-29T16:01:30.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.104+0000 7f9b04ff9640 1 -- 172.21.15.59:0/3076285346 <== osd.0 v2:172.21.15.59:6808/1075583359 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9b080ccae0 con 0x7f9af407d020 2024-06-29T16:01:30.105 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T16:01:30.106 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:30.106 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T16:01:30.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.104+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7f9af407d020 msgr2=0x7f9af407f510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.104+0000 7f9b0e3fc640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.59:6808/1075583359,v1:172.21.15.59:6809/1075583359] conn(0x7f9af407d020 0x7f9af407f510 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9af40763f0 msgr2=0x7f9af4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0e3fc640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f9af40763f0 0x7f9af4078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009bb0 tx=0x7f9af8005f90 comp rx=0 tx=0).stop 2024-06-29T16:01:30.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b081f5120 msgr2=0x7f9b081607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0e3fc640 1 --2- 172.21.15.59:0/3076285346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b081f5120 0x7f9b081607a0 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f9b00005fd0 tx=0x7f9b00011ef0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0c999640 1 -- 172.21.15.59:0/3076285346 reap_dead start 2024-06-29T16:01:30.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 shutdown_connections 2024-06-29T16:01:30.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 >> 172.21.15.59:0/3076285346 conn(0x7f9b080d4070 msgr2=0x7f9b080d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 shutdown_connections 2024-06-29T16:01:30.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.105+0000 7f9b0e3fc640 1 -- 172.21.15.59:0/3076285346 wait complete. 2024-06-29T16:01:30.116 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-06-29T16:01:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.234+0000 7f4bafd5b640 1 -- 172.21.15.59:0/1655463077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80d1a60 msgr2=0x7f4ba80b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.234+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/1655463077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80d1a60 0x7f4ba80b5b30 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80b2d10 tx=0x7f4b9802a9e0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.234+0000 7f4bafd5b640 1 -- 172.21.15.59:0/1655463077 shutdown_connections 2024-06-29T16:01:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.234+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/1655463077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80d1a60 0x7f4ba80b5b30 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.234+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/1655463077 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ba80b2fa0 0x7f4ba80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.234+0000 7f4bafd5b640 1 -- 172.21.15.59:0/1655463077 >> 172.21.15.59:0/1655463077 conn(0x7f4ba8073e10 msgr2=0x7f4ba8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.234+0000 7f4bafd5b640 1 -- 172.21.15.59:0/1655463077 shutdown_connections 2024-06-29T16:01:30.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.234+0000 7f4bafd5b640 1 -- 172.21.15.59:0/1655463077 wait complete. 2024-06-29T16:01:30.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.236+0000 7f4bafd5b640 1 Processor -- start 2024-06-29T16:01:30.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.236+0000 7f4bafd5b640 1 -- start start 2024-06-29T16:01:30.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4bafd5b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80b2270 0x7f4ba815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4bafd5b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ba80b2fa0 0x7f4ba81f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4bafd5b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba80d1a60 0x7f4ba81471e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4bafd5b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba80b6400 con 0x7f4ba80b2270 2024-06-29T16:01:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4bafd5b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba80b6280 con 0x7f4ba80d1a60 2024-06-29T16:01:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4bafd5b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba80b6580 con 0x7f4ba80b2fa0 2024-06-29T16:01:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4badaf7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80b2270 0x7f4ba815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4badaf7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80b2270 0x7f4ba815ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44378/0 (socket says 172.21.15.59:44378) 2024-06-29T16:01:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4badaf7640 1 -- 172.21.15.59:0/836077333 learned_addr learned my addr 172.21.15.59:0/836077333 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4bae2f8640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba80d1a60 0x7f4ba81471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.237+0000 7f4bad2f6640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ba80b2fa0 0x7f4ba81f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4badaf7640 1 -- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ba80b2fa0 msgr2=0x7f4ba81f8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4badaf7640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ba80b2fa0 0x7f4ba81f8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4badaf7640 1 -- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba80d1a60 msgr2=0x7f4ba81471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4badaf7640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba80d1a60 0x7f4ba81471e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4badaf7640 1 -- 172.21.15.59:0/836077333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba80aeb10 con 0x7f4ba80b2270 2024-06-29T16:01:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4bad2f6640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ba80b2fa0 0x7f4ba81f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4bae2f8640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba80d1a60 0x7f4ba81471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4badaf7640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80b2270 0x7f4ba815ed70 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000b3f0 tx=0x7f4ba000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4ba67fc640 1 -- 172.21.15.59:0/836077333 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0015020 con 0x7f4ba80b2270 2024-06-29T16:01:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4ba67fc640 1 -- 172.21.15.59:0/836077333 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ba0011920 con 0x7f4ba80b2270 2024-06-29T16:01:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba80aedf0 con 0x7f4ba80b2270 2024-06-29T16:01:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4ba67fc640 1 -- 172.21.15.59:0/836077333 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0011d70 con 0x7f4ba80b2270 2024-06-29T16:01:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.238+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba815d9a0 con 0x7f4ba80b2270 2024-06-29T16:01:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.239+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba80b2680 con 0x7f4ba80b2270 2024-06-29T16:01:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.240+0000 7f4ba67fc640 1 -- 172.21.15.59:0/836077333 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f4ba0018020 con 0x7f4ba80b2270 2024-06-29T16:01:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.241+0000 7f4ba67fc640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f4b94076420 0x7f4b94078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.241+0000 7f4ba67fc640 1 -- 172.21.15.59:0/836077333 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f4ba00978e0 con 0x7f4ba80b2270 2024-06-29T16:01:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.241+0000 7f4bad2f6640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f4b94076420 0x7f4b94078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.241+0000 7f4bad2f6640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f4b94076420 0x7f4b94078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c005fd0 tx=0x7f4b9c005e00 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.245+0000 7f4ba67fc640 1 -- 172.21.15.59:0/836077333 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ba80b2680 con 0x7f4ba80b2270 2024-06-29T16:01:30.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.435+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f4ba815dc40 con 0x7f4b94076420 2024-06-29T16:01:30.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.436+0000 7f4ba67fc640 1 -- 172.21.15.59:0/836077333 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4ba815dc40 con 0x7f4b94076420 2024-06-29T16:01:30.437 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 0 to deep-scrub 2024-06-29T16:01:30.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f4b94076420 msgr2=0x7f4b94078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f4b94076420 0x7f4b94078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c005fd0 tx=0x7f4b9c005e00 comp rx=0 tx=0).stop 2024-06-29T16:01:30.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80b2270 msgr2=0x7f4ba815ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80b2270 0x7f4ba815ed70 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000b3f0 tx=0x7f4ba000b8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 shutdown_connections 2024-06-29T16:01:30.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f4b94076420 0x7f4b94078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba80d1a60 0x7f4ba81471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4ba80b2fa0 0x7f4ba81f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 --2- 172.21.15.59:0/836077333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ba80b2270 0x7f4ba815ed70 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 >> 172.21.15.59:0/836077333 conn(0x7f4ba8073e10 msgr2=0x7f4ba815ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 shutdown_connections 2024-06-29T16:01:30.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.438+0000 7f4bafd5b640 1 -- 172.21.15.59:0/836077333 wait complete. 2024-06-29T16:01:30.450 INFO:tasks.ceph:Scrubbing osd.1 2024-06-29T16:01:30.450 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:30.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.573+0000 7f30e712a640 1 -- 172.21.15.59:0/1839050727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00ce2c0 msgr2=0x7f30e01596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.573+0000 7f30e712a640 1 --2- 172.21.15.59:0/1839050727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00ce2c0 0x7f30e01596a0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009860 tx=0x7f30d002a4b0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.573+0000 7f30e712a640 1 -- 172.21.15.59:0/1839050727 shutdown_connections 2024-06-29T16:01:30.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.573+0000 7f30e712a640 1 --2- 172.21.15.59:0/1839050727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00ce2c0 0x7f30e01596a0 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.573+0000 7f30e712a640 1 -- 172.21.15.59:0/1839050727 >> 172.21.15.59:0/1839050727 conn(0x7f30e00b7690 msgr2=0x7f30e0073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.573+0000 7f30e712a640 1 -- 172.21.15.59:0/1839050727 shutdown_connections 2024-06-29T16:01:30.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.573+0000 7f30e712a640 1 -- 172.21.15.59:0/1839050727 wait complete. 2024-06-29T16:01:30.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.574+0000 7f30e712a640 1 Processor -- start 2024-06-29T16:01:30.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.575+0000 7f30e712a640 1 -- start start 2024-06-29T16:01:30.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e712a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30e00ce2c0 0x7f30e0151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e712a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e0160090 0x7f30e00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e712a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0153c10 0x7f30e0154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e712a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f30e00b5740 con 0x7f30e0160090 2024-06-29T16:01:30.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e712a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f30e00b55c0 con 0x7f30e0153c10 2024-06-29T16:01:30.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e712a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f30e00b58c0 con 0x7f30e00ce2c0 2024-06-29T16:01:30.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e0160090 0x7f30e00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e0160090 0x7f30e00b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44398/0 (socket says 172.21.15.59:44398) 2024-06-29T16:01:30.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30dffff640 1 -- 172.21.15.59:0/1233113383 learned_addr learned my addr 172.21.15.59:0/1233113383 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:30.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e56c7640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0153c10 0x7f30e0154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e4ec6640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30e00ce2c0 0x7f30e0151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30dffff640 1 -- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30e00ce2c0 msgr2=0x7f30e0151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30dffff640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30e00ce2c0 0x7f30e0151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30dffff640 1 -- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0153c10 msgr2=0x7f30e0154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30dffff640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0153c10 0x7f30e0154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30dffff640 1 -- 172.21.15.59:0/1233113383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e0154560 con 0x7f30e0160090 2024-06-29T16:01:30.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e56c7640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30e0153c10 0x7f30e0154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:30.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.576+0000 7f30e4ec6640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f30e00ce2c0 0x7f30e0151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:30.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.577+0000 7f30dffff640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e0160090 0x7f30e00b3da0 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f30d400c3f0 tx=0x7f30d400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.577+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4017070 con 0x7f30e0160090 2024-06-29T16:01:30.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.577+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30d40035f0 con 0x7f30e0160090 2024-06-29T16:01:30.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.577+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4003a40 con 0x7f30e0160090 2024-06-29T16:01:30.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.577+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e00b20e0 con 0x7f30e0160090 2024-06-29T16:01:30.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.577+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e00b26a0 con 0x7f30e0160090 2024-06-29T16:01:30.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.578+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f30e00b1e90 con 0x7f30e0160090 2024-06-29T16:01:30.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.579+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f30d401a070 con 0x7f30e0160090 2024-06-29T16:01:30.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.579+0000 7f30dd7fa640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f30cc0763f0 0x7f30cc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.579+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f30d4096da0 con 0x7f30e0160090 2024-06-29T16:01:30.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.579+0000 7f30e4ec6640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f30cc0763f0 0x7f30cc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.579+0000 7f30dd7fa640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f30cc07d020 0x7f30cc07f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.579+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 --> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30cc07fbe0 con 0x7f30cc07d020 2024-06-29T16:01:30.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.580+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f30d40994a0 con 0x7f30e0160090 2024-06-29T16:01:30.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.580+0000 7f30e56c7640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f30cc07d020 0x7f30cc07f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.580+0000 7f30e56c7640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f30cc07d020 0x7f30cc07f510 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.580+0000 7f30e4ec6640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f30cc0763f0 0x7f30cc078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30d0034000 tx=0x7f30d0004730 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.581+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 <== osd.1 v2:172.21.15.59:6824/1021101403 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30cc07fbe0 con 0x7f30cc07d020 2024-06-29T16:01:30.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.594+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 --> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f30e01596a0 con 0x7f30cc07d020 2024-06-29T16:01:30.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.594+0000 7f30dd7fa640 1 -- 172.21.15.59:0/1233113383 <== osd.1 v2:172.21.15.59:6824/1021101403 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f30e01596a0 con 0x7f30cc07d020 2024-06-29T16:01:30.595 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T16:01:30.595 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:30.595 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T16:01:30.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.594+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f30cc07d020 msgr2=0x7f30cc07f510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.594+0000 7f30e712a640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.59:6824/1021101403,v1:172.21.15.59:6825/1021101403] conn(0x7f30cc07d020 0x7f30cc07f510 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.594+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f30cc0763f0 msgr2=0x7f30cc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.594+0000 7f30e712a640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f30cc0763f0 0x7f30cc078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30d0034000 tx=0x7f30d0004730 comp rx=0 tx=0).stop 2024-06-29T16:01:30.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.594+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e0160090 msgr2=0x7f30e00b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.594+0000 7f30e712a640 1 --2- 172.21.15.59:0/1233113383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e0160090 0x7f30e00b3da0 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f30d400c3f0 tx=0x7f30d400c8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.595+0000 7f30e56c7640 1 -- 172.21.15.59:0/1233113383 reap_dead start 2024-06-29T16:01:30.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.595+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 shutdown_connections 2024-06-29T16:01:30.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.595+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 >> 172.21.15.59:0/1233113383 conn(0x7f30e00b7690 msgr2=0x7f30e006fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.595+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 shutdown_connections 2024-06-29T16:01:30.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.595+0000 7f30e712a640 1 -- 172.21.15.59:0/1233113383 wait complete. 2024-06-29T16:01:30.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-06-29T16:01:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.724+0000 7fc600bac640 1 -- 172.21.15.59:0/2181809579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5e4003ac0 msgr2=0x7fc5e4003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.724+0000 7fc600bac640 1 --2- 172.21.15.59:0/2181809579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5e4003ac0 0x7fc5e4003f30 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009b50 tx=0x7fc5e4005830 comp rx=0 tx=0).stop 2024-06-29T16:01:30.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.724+0000 7fc5fad9d640 1 -- 172.21.15.59:0/2181809579 reap_dead start 2024-06-29T16:01:30.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.724+0000 7fc600bac640 1 -- 172.21.15.59:0/2181809579 shutdown_connections 2024-06-29T16:01:30.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.724+0000 7fc600bac640 1 -- 172.21.15.59:0/2181809579 >> 172.21.15.59:0/2181809579 conn(0x7fc5fc0d45b0 msgr2=0x7fc5fc0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.724+0000 7fc600bac640 1 -- 172.21.15.59:0/2181809579 shutdown_connections 2024-06-29T16:01:30.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.724+0000 7fc600bac640 1 -- 172.21.15.59:0/2181809579 wait complete. 2024-06-29T16:01:30.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.726+0000 7fc600bac640 1 Processor -- start 2024-06-29T16:01:30.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.726+0000 7fc600bac640 1 -- start start 2024-06-29T16:01:30.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc600bac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5e4003ac0 0x7fc5fc0c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc600bac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5fc156840 0x7fc5fc155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc600bac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5fc156260 0x7fc5fc0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc600bac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc0b3e20 con 0x7fc5fc156840 2024-06-29T16:01:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc600bac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc0b3fa0 con 0x7fc5fc156260 2024-06-29T16:01:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc600bac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc5fc0b4120 con 0x7fc5e4003ac0 2024-06-29T16:01:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5fc156840 0x7fc5fc155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5fc156840 0x7fc5fc155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44418/0 (socket says 172.21.15.59:44418) 2024-06-29T16:01:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 -- 172.21.15.59:0/2904367662 learned_addr learned my addr 172.21.15.59:0/2904367662 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5fa59c640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5e4003ac0 0x7fc5fc0c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5fad9d640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5fc156260 0x7fc5fc0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 -- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5e4003ac0 msgr2=0x7fc5fc0c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5e4003ac0 0x7fc5fc0c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 -- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5fc156260 msgr2=0x7fc5fc0cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5fc156260 0x7fc5fc0cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 -- 172.21.15.59:0/2904367662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5fc0cdc80 con 0x7fc5fc156840 2024-06-29T16:01:30.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5f9d9b640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5fc156840 0x7fc5fc155b40 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000e570 tx=0x7fc5f0001840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5eaffd640 1 -- 172.21.15.59:0/2904367662 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0015070 con 0x7fc5fc156840 2024-06-29T16:01:30.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5eaffd640 1 -- 172.21.15.59:0/2904367662 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5f0010980 con 0x7fc5fc156840 2024-06-29T16:01:30.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5fc159a60 con 0x7fc5fc156840 2024-06-29T16:01:30.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.728+0000 7fc5eaffd640 1 -- 172.21.15.59:0/2904367662 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0010dd0 con 0x7fc5fc156840 2024-06-29T16:01:30.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.729+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5fc0cb850 con 0x7fc5fc156840 2024-06-29T16:01:30.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.729+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5fc0b38e0 con 0x7fc5fc156840 2024-06-29T16:01:30.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.730+0000 7fc5eaffd640 1 -- 172.21.15.59:0/2904367662 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7fc5f0018070 con 0x7fc5fc156840 2024-06-29T16:01:30.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.731+0000 7fc5eaffd640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc5dc076420 0x7fc5dc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.731+0000 7fc5eaffd640 1 -- 172.21.15.59:0/2904367662 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7fc5f0097e30 con 0x7fc5fc156840 2024-06-29T16:01:30.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.731+0000 7fc5fa59c640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc5dc076420 0x7fc5dc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.732+0000 7fc5fa59c640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc5dc076420 0x7fc5dc078930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec035000 tx=0x7fc5ec02d040 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.735+0000 7fc5eaffd640 1 -- 172.21.15.59:0/2904367662 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc5f0060d20 con 0x7fc5fc156840 2024-06-29T16:01:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.930+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fc5fc156080 con 0x7fc5dc076420 2024-06-29T16:01:30.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.931+0000 7fc5eaffd640 1 -- 172.21.15.59:0/2904367662 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc5ec02ba70 con 0x7fc5dc076420 2024-06-29T16:01:30.932 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 1 to deep-scrub 2024-06-29T16:01:30.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc5dc076420 msgr2=0x7fc5dc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc5dc076420 0x7fc5dc078930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec035000 tx=0x7fc5ec02d040 comp rx=0 tx=0).stop 2024-06-29T16:01:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5fc156840 msgr2=0x7fc5fc155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5fc156840 0x7fc5fc155b40 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000e570 tx=0x7fc5f0001840 comp rx=0 tx=0).stop 2024-06-29T16:01:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 shutdown_connections 2024-06-29T16:01:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc5dc076420 0x7fc5dc078930 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5fc156260 0x7fc5fc0cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5fc156840 0x7fc5fc155b40 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 --2- 172.21.15.59:0/2904367662 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc5e4003ac0 0x7fc5fc0c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 >> 172.21.15.59:0/2904367662 conn(0x7fc5fc0d45b0 msgr2=0x7fc5fc0b7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 shutdown_connections 2024-06-29T16:01:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:30.933+0000 7fc600bac640 1 -- 172.21.15.59:0/2904367662 wait complete. 2024-06-29T16:01:30.946 INFO:tasks.ceph:Scrubbing osd.2 2024-06-29T16:01:30.947 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.068+0000 7f002ea95640 1 -- 172.21.15.59:0/838949132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0014003a30 msgr2=0x7f0014003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.068+0000 7f002ea95640 1 --2- 172.21.15.59:0/838949132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0014003a30 0x7f0014003ea0 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f001000c3f0 tx=0x7f0014005830 comp rx=0 tx=0).stop 2024-06-29T16:01:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.068+0000 7f002d032640 1 -- 172.21.15.59:0/838949132 reap_dead start 2024-06-29T16:01:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.068+0000 7f002ea95640 1 -- 172.21.15.59:0/838949132 shutdown_connections 2024-06-29T16:01:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.068+0000 7f002ea95640 1 -- 172.21.15.59:0/838949132 >> 172.21.15.59:0/838949132 conn(0x7f00280d3e60 msgr2=0x7f00280d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.068+0000 7f002ea95640 1 -- 172.21.15.59:0/838949132 shutdown_connections 2024-06-29T16:01:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.068+0000 7f002ea95640 1 -- 172.21.15.59:0/838949132 wait complete. 2024-06-29T16:01:31.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.070+0000 7f002ea95640 1 Processor -- start 2024-06-29T16:01:31.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.070+0000 7f002ea95640 1 -- start start 2024-06-29T16:01:31.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002ea95640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0014003a30 0x7f002814f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002ea95640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002814eb40 0x7f0028158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002ea95640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002814e5f0 0x7f0028158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002ea95640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f0028166dd0 con 0x7f002814e5f0 2024-06-29T16:01:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002ea95640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0028166c50 con 0x7f002814eb40 2024-06-29T16:01:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002ea95640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0028166f50 con 0x7f0014003a30 2024-06-29T16:01:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002d032640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002814e5f0 0x7f0028158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002d032640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002814e5f0 0x7f0028158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44440/0 (socket says 172.21.15.59:44440) 2024-06-29T16:01:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002d032640 1 -- 172.21.15.59:0/1457911235 learned_addr learned my addr 172.21.15.59:0/1457911235 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f0027fff640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002814eb40 0x7f0028158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.071+0000 7f002c831640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0014003a30 0x7f002814f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002d032640 1 -- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0014003a30 msgr2=0x7f002814f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002d032640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0014003a30 0x7f002814f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002d032640 1 -- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002814eb40 msgr2=0x7f0028158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002d032640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002814eb40 0x7f0028158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002d032640 1 -- 172.21.15.59:0/1457911235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00281601e0 con 0x7f002814e5f0 2024-06-29T16:01:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002c831640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0014003a30 0x7f002814f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f0027fff640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f002814eb40 0x7f0028158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002d032640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002814e5f0 0x7f0028158f40 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f00280c7290 tx=0x7f0020010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000ee50 con 0x7f002814e5f0 2024-06-29T16:01:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0020015070 con 0x7f002814e5f0 2024-06-29T16:01:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00281604c0 con 0x7f002814e5f0 2024-06-29T16:01:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020003d00 con 0x7f002814e5f0 2024-06-29T16:01:31.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.072+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00281f56a0 con 0x7f002814e5f0 2024-06-29T16:01:31.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.073+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0028161ae0 con 0x7f002814e5f0 2024-06-29T16:01:31.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.074+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f0020004030 con 0x7f002814e5f0 2024-06-29T16:01:31.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.074+0000 7f00257fa640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f00080763f0 0x7f0008078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.075+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f0020098990 con 0x7f002814e5f0 2024-06-29T16:01:31.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.075+0000 7f00257fa640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f000807d020 0x7f000807f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.075+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 --> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f002000be50 con 0x7f000807d020 2024-06-29T16:01:31.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.075+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f0020098d40 con 0x7f002814e5f0 2024-06-29T16:01:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.075+0000 7f002c831640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f00080763f0 0x7f0008078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.075+0000 7f0027fff640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f000807d020 0x7f000807f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.075+0000 7f0027fff640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f000807d020 0x7f000807f510 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.075+0000 7f002c831640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f00080763f0 0x7f0008078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0010033000 tx=0x7f0010000fe0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.076+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 <== osd.2 v2:172.21.15.59:6800/2051957706 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f002000be50 con 0x7f000807d020 2024-06-29T16:01:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.089+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 --> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f00280d1610 con 0x7f000807d020 2024-06-29T16:01:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.089+0000 7f00257fa640 1 -- 172.21.15.59:0/1457911235 <== osd.2 v2:172.21.15.59:6800/2051957706 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0018006e10 con 0x7f000807d020 2024-06-29T16:01:31.090 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T16:01:31.091 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:31.091 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T16:01:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f000807d020 msgr2=0x7f000807f510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.59:6800/2051957706,v1:172.21.15.59:6801/2051957706] conn(0x7f000807d020 0x7f000807f510 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f00080763f0 msgr2=0x7f0008078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f00080763f0 0x7f0008078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0010033000 tx=0x7f0010000fe0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002814e5f0 msgr2=0x7f0028158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 --2- 172.21.15.59:0/1457911235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f002814e5f0 0x7f0028158f40 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f00280c7290 tx=0x7f0020010f40 comp rx=0 tx=0).stop 2024-06-29T16:01:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002d032640 1 -- 172.21.15.59:0/1457911235 reap_dead start 2024-06-29T16:01:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 shutdown_connections 2024-06-29T16:01:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 >> 172.21.15.59:0/1457911235 conn(0x7f00280d3e60 msgr2=0x7f00280d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 shutdown_connections 2024-06-29T16:01:31.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.090+0000 7f002ea95640 1 -- 172.21.15.59:0/1457911235 wait complete. 2024-06-29T16:01:31.101 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-06-29T16:01:31.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.219+0000 7ff1ad67f640 1 -- 172.21.15.59:0/2985477362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 msgr2=0x7ff1a80b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.219+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/2985477362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 0x7ff1a80b5b30 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7ff1a80b2680 tx=0x7ff19802aa30 comp rx=0 tx=0).stop 2024-06-29T16:01:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.219+0000 7ff1ad67f640 1 -- 172.21.15.59:0/2985477362 shutdown_connections 2024-06-29T16:01:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.219+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/2985477362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 0x7ff1a80b5b30 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.219+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/2985477362 >> v2:172.21.15.153:3300/0 conn(0x7ff1a80b2fa0 0x7ff1a80d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.219+0000 7ff1ad67f640 1 -- 172.21.15.59:0/2985477362 >> 172.21.15.59:0/2985477362 conn(0x7ff1a8073e10 msgr2=0x7ff1a8074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.219+0000 7ff1ad67f640 1 -- 172.21.15.59:0/2985477362 shutdown_connections 2024-06-29T16:01:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.219+0000 7ff1ad67f640 1 -- 172.21.15.59:0/2985477362 wait complete. 2024-06-29T16:01:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.221+0000 7ff1ad67f640 1 Processor -- start 2024-06-29T16:01:31.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.221+0000 7ff1ad67f640 1 -- start start 2024-06-29T16:01:31.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1ad67f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a80b2270 0x7ff1a80ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1ad67f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1a80b2fa0 0x7ff1a81478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1ad67f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 0x7ff1a8147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1ad67f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a80b6400 con 0x7ff1a80d1a60 2024-06-29T16:01:31.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1ad67f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a80b6280 con 0x7ff1a80b2270 2024-06-29T16:01:31.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1ad67f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff1a80b6580 con 0x7ff1a80b2fa0 2024-06-29T16:01:31.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 0x7ff1a8147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 0x7ff1a8147de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44454/0 (socket says 172.21.15.59:44454) 2024-06-29T16:01:31.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 -- 172.21.15.59:0/822516630 learned_addr learned my addr 172.21.15.59:0/822516630 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:31.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a67fc640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1a80b2fa0 0x7ff1a81478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a6ffd640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a80b2270 0x7ff1a80ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 -- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1a80b2fa0 msgr2=0x7ff1a81478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1a80b2fa0 0x7ff1a81478a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 -- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a80b2270 msgr2=0x7ff1a80ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a80b2270 0x7ff1a80ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 -- 172.21.15.59:0/822516630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a8145230 con 0x7ff1a80d1a60 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1a77fe640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 0x7ff1a8147de0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7ff19802b020 tx=0x7ff198003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff19f7fe640 1 -- 172.21.15.59:0/822516630 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198033050 con 0x7ff1a80d1a60 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff19f7fe640 1 -- 172.21.15.59:0/822516630 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff198004550 con 0x7ff1a80d1a60 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a8145510 con 0x7ff1a80d1a60 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.223+0000 7ff19f7fe640 1 -- 172.21.15.59:0/822516630 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198038450 con 0x7ff1a80d1a60 2024-06-29T16:01:31.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.224+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a8145ad0 con 0x7ff1a80d1a60 2024-06-29T16:01:31.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.224+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a80b6ae0 con 0x7ff1a80d1a60 2024-06-29T16:01:31.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.225+0000 7ff19f7fe640 1 -- 172.21.15.59:0/822516630 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7ff198036070 con 0x7ff1a80d1a60 2024-06-29T16:01:31.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.226+0000 7ff19f7fe640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7ff1880763f0 0x7ff188078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.226+0000 7ff19f7fe640 1 -- 172.21.15.59:0/822516630 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7ff1980b92e0 con 0x7ff1a80d1a60 2024-06-29T16:01:31.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.226+0000 7ff1a6ffd640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7ff1880763f0 0x7ff188078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.227+0000 7ff1a6ffd640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7ff1880763f0 0x7ff188078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0006fd0 tx=0x7ff1a0008040 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.230+0000 7ff19f7fe640 1 -- 172.21.15.59:0/822516630 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff1980828f0 con 0x7ff1a80d1a60 2024-06-29T16:01:31.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.419+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7ff1a80cebc0 con 0x7ff1880763f0 2024-06-29T16:01:31.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.420+0000 7ff19f7fe640 1 -- 172.21.15.59:0/822516630 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff1a80cebc0 con 0x7ff1880763f0 2024-06-29T16:01:31.421 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 2 to deep-scrub 2024-06-29T16:01:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7ff1880763f0 msgr2=0x7ff188078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7ff1880763f0 0x7ff188078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0006fd0 tx=0x7ff1a0008040 comp rx=0 tx=0).stop 2024-06-29T16:01:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 msgr2=0x7ff1a8147de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 0x7ff1a8147de0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7ff19802b020 tx=0x7ff198003f10 comp rx=0 tx=0).stop 2024-06-29T16:01:31.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 shutdown_connections 2024-06-29T16:01:31.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7ff1880763f0 0x7ff188078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a80d1a60 0x7ff1a8147de0 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff1a80b2fa0 0x7ff1a81478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 --2- 172.21.15.59:0/822516630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1a80b2270 0x7ff1a80ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 >> 172.21.15.59:0/822516630 conn(0x7ff1a8073e10 msgr2=0x7ff1a815bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 shutdown_connections 2024-06-29T16:01:31.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.422+0000 7ff1ad67f640 1 -- 172.21.15.59:0/822516630 wait complete. 2024-06-29T16:01:31.434 INFO:tasks.ceph:Scrubbing osd.3 2024-06-29T16:01:31.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.555+0000 7fc0564e4640 1 -- 172.21.15.59:0/3383215467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0500cec20 msgr2=0x7fc0500d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.555+0000 7fc0564e4640 1 --2- 172.21.15.59:0/3383215467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0500cec20 0x7fc0500d1790 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7fc044009860 tx=0x7fc04402a4b0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.555+0000 7fc0564e4640 1 -- 172.21.15.59:0/3383215467 shutdown_connections 2024-06-29T16:01:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.555+0000 7fc0564e4640 1 --2- 172.21.15.59:0/3383215467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0500cec20 0x7fc0500d1790 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.555+0000 7fc0564e4640 1 --2- 172.21.15.59:0/3383215467 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0501423c0 0x7fc0500b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.555+0000 7fc0564e4640 1 -- 172.21.15.59:0/3383215467 >> 172.21.15.59:0/3383215467 conn(0x7fc050073e10 msgr2=0x7fc050074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.555+0000 7fc0564e4640 1 -- 172.21.15.59:0/3383215467 shutdown_connections 2024-06-29T16:01:31.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.556+0000 7fc0564e4640 1 -- 172.21.15.59:0/3383215467 wait complete. 2024-06-29T16:01:31.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.557+0000 7fc0564e4640 1 Processor -- start 2024-06-29T16:01:31.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.557+0000 7fc0564e4640 1 -- start start 2024-06-29T16:01:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc0564e4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0500cec20 0x7fc05013fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc0564e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0500d1cd0 0x7fc0501e44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc0564e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0501423c0 0x7fc05015d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc0564e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fc0500b2bf0 con 0x7fc0501423c0 2024-06-29T16:01:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc0564e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc05015c4d0 con 0x7fc0500d1cd0 2024-06-29T16:01:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc0564e4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc0500b2d70 con 0x7fc0500cec20 2024-06-29T16:01:31.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0501423c0 0x7fc05015d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0501423c0 0x7fc05015d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44470/0 (socket says 172.21.15.59:44470) 2024-06-29T16:01:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 -- 172.21.15.59:0/2094512782 learned_addr learned my addr 172.21.15.59:0/2094512782 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc04ffff640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0500cec20 0x7fc05013fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc04f7fe640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0500d1cd0 0x7fc0501e44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 -- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0500cec20 msgr2=0x7fc05013fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc0500cec20 0x7fc05013fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 -- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0500d1cd0 msgr2=0x7fc0501e44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0500d1cd0 0x7fc0501e44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 -- 172.21.15.59:0/2094512782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0501e5120 con 0x7fc0501423c0 2024-06-29T16:01:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.559+0000 7fc054a81640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0501423c0 0x7fc05015d420 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7fc04800cb10 tx=0x7fc04800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.560+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048016020 con 0x7fc0501423c0 2024-06-29T16:01:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.560+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc04800ad60 con 0x7fc0501423c0 2024-06-29T16:01:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.560+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048012680 con 0x7fc0501423c0 2024-06-29T16:01:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.560+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0501e5400 con 0x7fc0501423c0 2024-06-29T16:01:31.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.560+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0501e5a60 con 0x7fc0501423c0 2024-06-29T16:01:31.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.560+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc050076540 con 0x7fc0501423c0 2024-06-29T16:01:31.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.561+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7fc048019020 con 0x7fc0501423c0 2024-06-29T16:01:31.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.562+0000 7fc04cff9640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc03c076420 0x7fc03c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.562+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7fc048098590 con 0x7fc0501423c0 2024-06-29T16:01:31.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.562+0000 7fc04cff9640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7fc03c07d050 0x7fc03c07f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.562+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 --> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc03c07fc10 con 0x7fc03c07d050 2024-06-29T16:01:31.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.562+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7fc048098940 con 0x7fc0501423c0 2024-06-29T16:01:31.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.562+0000 7fc04ffff640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc03c076420 0x7fc03c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.562+0000 7fc04f7fe640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7fc03c07d050 0x7fc03c07f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.563+0000 7fc04ffff640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc03c076420 0x7fc03c078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc04000bb50 tx=0x7fc04000b2e0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.563+0000 7fc04f7fe640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7fc03c07d050 0x7fc03c07f540 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.563+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 <== osd.3 v2:172.21.15.59:6816/2018446268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc03c07fc10 con 0x7fc03c07d050 2024-06-29T16:01:31.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 --> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc0501e4ed0 con 0x7fc03c07d050 2024-06-29T16:01:31.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc04cff9640 1 -- 172.21.15.59:0/2094512782 <== osd.3 v2:172.21.15.59:6816/2018446268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc0501e4ed0 con 0x7fc03c07d050 2024-06-29T16:01:31.578 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T16:01:31.579 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:31.579 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T16:01:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7fc03c07d050 msgr2=0x7fc03c07f540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc0564e4640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.59:6816/2018446268,v1:172.21.15.59:6817/2018446268] conn(0x7fc03c07d050 0x7fc03c07f540 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc03c076420 msgr2=0x7fc03c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc0564e4640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fc03c076420 0x7fc03c078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc04000bb50 tx=0x7fc04000b2e0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0501423c0 msgr2=0x7fc05015d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc0564e4640 1 --2- 172.21.15.59:0/2094512782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0501423c0 0x7fc05015d420 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7fc04800cb10 tx=0x7fc04800a060 comp rx=0 tx=0).stop 2024-06-29T16:01:31.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.577+0000 7fc054a81640 1 -- 172.21.15.59:0/2094512782 reap_dead start 2024-06-29T16:01:31.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.578+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 shutdown_connections 2024-06-29T16:01:31.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.578+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 >> 172.21.15.59:0/2094512782 conn(0x7fc050073e10 msgr2=0x7fc05006ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.578+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 shutdown_connections 2024-06-29T16:01:31.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.578+0000 7fc0564e4640 1 -- 172.21.15.59:0/2094512782 wait complete. 2024-06-29T16:01:31.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-06-29T16:01:31.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.708+0000 7f82b7bcf640 1 -- 172.21.15.59:0/1209271991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0162210 msgr2=0x7f82b00b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.708+0000 7f82b7bcf640 1 --2- 172.21.15.59:0/1209271991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0162210 0x7f82b00b1d10 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f82b0074d00 tx=0x7f82a802a4f0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.708+0000 7f82b7bcf640 1 -- 172.21.15.59:0/1209271991 shutdown_connections 2024-06-29T16:01:31.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.708+0000 7f82b7bcf640 1 --2- 172.21.15.59:0/1209271991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0162210 0x7f82b00b1d10 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.708+0000 7f82b7bcf640 1 -- 172.21.15.59:0/1209271991 >> 172.21.15.59:0/1209271991 conn(0x7f82b006c950 msgr2=0x7f82b0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.708+0000 7f82b7bcf640 1 -- 172.21.15.59:0/1209271991 shutdown_connections 2024-06-29T16:01:31.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.708+0000 7f82b7bcf640 1 -- 172.21.15.59:0/1209271991 wait complete. 2024-06-29T16:01:31.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.710+0000 7f82b7bcf640 1 Processor -- start 2024-06-29T16:01:31.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.710+0000 7f82b7bcf640 1 -- start start 2024-06-29T16:01:31.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b7bcf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b006f8a0 0x7f82b015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b7bcf640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b0147370 0x7f82b013e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b7bcf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0145250 0x7f82b0145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b7bcf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f82b0169340 con 0x7f82b0145250 2024-06-29T16:01:31.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b7bcf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f82b01691c0 con 0x7f82b006f8a0 2024-06-29T16:01:31.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b7bcf640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f82b01694c0 con 0x7f82b0147370 2024-06-29T16:01:31.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b616c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0145250 0x7f82b0145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b616c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0145250 0x7f82b0145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44490/0 (socket says 172.21.15.59:44490) 2024-06-29T16:01:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b616c640 1 -- 172.21.15.59:0/3967159059 learned_addr learned my addr 172.21.15.59:0/3967159059 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b596b640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b006f8a0 0x7f82b015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.711+0000 7f82b516a640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b0147370 0x7f82b013e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82b616c640 1 -- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b0147370 msgr2=0x7f82b013e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82b616c640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b0147370 0x7f82b013e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82b616c640 1 -- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b006f8a0 msgr2=0x7f82b015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82b616c640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b006f8a0 0x7f82b015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82b616c640 1 -- 172.21.15.59:0/3967159059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b0145ba0 con 0x7f82b0145250 2024-06-29T16:01:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82b616c640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0145250 0x7f82b0145660 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f82a802aae0 tx=0x7f82a8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3967159059 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a8031d60 con 0x7f82b0145250 2024-06-29T16:01:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3967159059 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82a8033e10 con 0x7f82b0145250 2024-06-29T16:01:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b01420d0 con 0x7f82b0145250 2024-06-29T16:01:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3967159059 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a8005260 con 0x7f82b0145250 2024-06-29T16:01:31.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.712+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b0142690 con 0x7f82b0145250 2024-06-29T16:01:31.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.713+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b00cfda0 con 0x7f82b0145250 2024-06-29T16:01:31.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.714+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3967159059 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f82a802ad10 con 0x7f82b0145250 2024-06-29T16:01:31.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.715+0000 7f82ae7fc640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f829c0763f0 0x7f829c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:31.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.715+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3967159059 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f82a80b8b70 con 0x7f82b0145250 2024-06-29T16:01:31.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.715+0000 7f82b596b640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f829c0763f0 0x7f829c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:31.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.715+0000 7f82b596b640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f829c0763f0 0x7f829c078900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82a000b450 tx=0x7f82a000d040 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:31.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.719+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3967159059 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f82b00cfda0 con 0x7f82b0145250 2024-06-29T16:01:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.911+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f82b0141e60 con 0x7f829c0763f0 2024-06-29T16:01:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.911+0000 7f82ae7fc640 1 -- 172.21.15.59:0/3967159059 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f82b0141e60 con 0x7f829c0763f0 2024-06-29T16:01:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 3 to deep-scrub 2024-06-29T16:01:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f829c0763f0 msgr2=0x7f829c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f829c0763f0 0x7f829c078900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f82a000b450 tx=0x7f82a000d040 comp rx=0 tx=0).stop 2024-06-29T16:01:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0145250 msgr2=0x7f82b0145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0145250 0x7f82b0145660 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f82a802aae0 tx=0x7f82a8033c20 comp rx=0 tx=0).stop 2024-06-29T16:01:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 shutdown_connections 2024-06-29T16:01:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f829c0763f0 0x7f829c078900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b0145250 0x7f82b0145660 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f82b0147370 0x7f82b013e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 --2- 172.21.15.59:0/3967159059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82b006f8a0 0x7f82b015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.913+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 >> 172.21.15.59:0/3967159059 conn(0x7f82b006c950 msgr2=0x7f82b00705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.914+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 shutdown_connections 2024-06-29T16:01:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:31.914+0000 7f82b7bcf640 1 -- 172.21.15.59:0/3967159059 wait complete. 2024-06-29T16:01:31.925 INFO:tasks.ceph:Scrubbing osd.4 2024-06-29T16:01:31.925 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:32.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.047+0000 7fe5f214b640 1 -- 172.21.15.59:0/2463831887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5ec0b2a50 msgr2=0x7fe5ec161380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.047+0000 7fe5f214b640 1 --2- 172.21.15.59:0/2463831887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5ec0b2a50 0x7fe5ec161380 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009830 tx=0x7fe5dc02a510 comp rx=0 tx=0).stop 2024-06-29T16:01:32.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.047+0000 7fe5f214b640 1 -- 172.21.15.59:0/2463831887 shutdown_connections 2024-06-29T16:01:32.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.047+0000 7fe5f214b640 1 --2- 172.21.15.59:0/2463831887 >> v2:172.21.15.153:3300/0 conn(0x7fe5ec0b7150 0x7fe5ec0b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.047+0000 7fe5f214b640 1 --2- 172.21.15.59:0/2463831887 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5ec0b2a50 0x7fe5ec161380 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.047+0000 7fe5f214b640 1 -- 172.21.15.59:0/2463831887 >> 172.21.15.59:0/2463831887 conn(0x7fe5ec0b5590 msgr2=0x7fe5ec0b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:32.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.047+0000 7fe5f214b640 1 -- 172.21.15.59:0/2463831887 shutdown_connections 2024-06-29T16:01:32.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.047+0000 7fe5f214b640 1 -- 172.21.15.59:0/2463831887 wait complete. 2024-06-29T16:01:32.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.049+0000 7fe5f214b640 1 Processor -- start 2024-06-29T16:01:32.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.049+0000 7fe5f214b640 1 -- start start 2024-06-29T16:01:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.050+0000 7fe5f214b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5ec0b2a50 0x7fe5ec14ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.050+0000 7fe5f214b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5ec0b7150 0x7fe5ec0c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5f214b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5ec0d2f70 0x7fe5ec0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5f214b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec0b4510 con 0x7fe5ec0b7150 2024-06-29T16:01:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5f214b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec0b53e0 con 0x7fe5ec0b2a50 2024-06-29T16:01:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5f214b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe5ec0b4690 con 0x7fe5ec0d2f70 2024-06-29T16:01:32.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5ec0b7150 0x7fe5ec0c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5ec0b7150 0x7fe5ec0c8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44502/0 (socket says 172.21.15.59:44502) 2024-06-29T16:01:32.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 -- 172.21.15.59:0/3219879335 learned_addr learned my addr 172.21.15.59:0/3219879335 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:32.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eb7fe640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5ec0b2a50 0x7fe5ec14ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5ebfff640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5ec0d2f70 0x7fe5ec0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 -- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5ec0d2f70 msgr2=0x7fe5ec0c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5ec0d2f70 0x7fe5ec0c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 -- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5ec0b2a50 msgr2=0x7fe5ec14ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5ec0b2a50 0x7fe5ec14ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 -- 172.21.15.59:0/3219879335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec157d20 con 0x7fe5ec0b7150 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5ebfff640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe5ec0d2f70 0x7fe5ec0c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5eaffd640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5ec0b7150 0x7fe5ec0c8c50 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000e6d0 tx=0x7fe5e000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e000cab0 con 0x7fe5ec0b7150 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5e000cd40 con 0x7fe5ec0b7150 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ec158000 con 0x7fe5ec0b7150 2024-06-29T16:01:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0010690 con 0x7fe5ec0b7150 2024-06-29T16:01:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.051+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ec1585c0 con 0x7fe5ec0b7150 2024-06-29T16:01:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.052+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe5ec0b4b10 con 0x7fe5ec0b7150 2024-06-29T16:01:32.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.053+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7fe5e0010830 con 0x7fe5ec0b7150 2024-06-29T16:01:32.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.054+0000 7fe5d3fff640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe5d80763f0 0x7fe5d8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.054+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7fe5e0099000 con 0x7fe5ec0b7150 2024-06-29T16:01:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.054+0000 7fe5d3fff640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7fe5d807cfc0 0x7fe5d807f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.054+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 --> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5d807fb80 con 0x7fe5d807cfc0 2024-06-29T16:01:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.054+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7fe5e0099490 con 0x7fe5ec0b7150 2024-06-29T16:01:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.054+0000 7fe5eb7fe640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe5d80763f0 0x7fe5d8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.054+0000 7fe5ebfff640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7fe5d807cfc0 0x7fe5d807f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.055+0000 7fe5eb7fe640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe5d80763f0 0x7fe5d8078900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc034000 tx=0x7fe5dc004730 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.055+0000 7fe5ebfff640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7fe5d807cfc0 0x7fe5d807f4b0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.056+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 <== osd.4 v2:172.21.15.153:6800/2158079959 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe5d807fb80 con 0x7fe5d807cfc0 2024-06-29T16:01:32.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.069+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 --> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe5ec08ccd0 con 0x7fe5d807cfc0 2024-06-29T16:01:32.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.069+0000 7fe5d3fff640 1 -- 172.21.15.59:0/3219879335 <== osd.4 v2:172.21.15.153:6800/2158079959 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe5ec08ccd0 con 0x7fe5d807cfc0 2024-06-29T16:01:32.071 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T16:01:32.071 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:32.071 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T16:01:32.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7fe5d807cfc0 msgr2=0x7fe5d807f4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6800/2158079959,v1:172.21.15.153:6801/2158079959] conn(0x7fe5d807cfc0 0x7fe5d807f4b0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe5d80763f0 msgr2=0x7fe5d8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe5d80763f0 0x7fe5d8078900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc034000 tx=0x7fe5dc004730 comp rx=0 tx=0).stop 2024-06-29T16:01:32.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5ec0b7150 msgr2=0x7fe5ec0c8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 --2- 172.21.15.59:0/3219879335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5ec0b7150 0x7fe5ec0c8c50 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000e6d0 tx=0x7fe5e000eb90 comp rx=0 tx=0).stop 2024-06-29T16:01:32.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5ebfff640 1 -- 172.21.15.59:0/3219879335 reap_dead start 2024-06-29T16:01:32.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 shutdown_connections 2024-06-29T16:01:32.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 >> 172.21.15.59:0/3219879335 conn(0x7fe5ec0b5590 msgr2=0x7fe5ec0d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:32.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 shutdown_connections 2024-06-29T16:01:32.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.070+0000 7fe5f214b640 1 -- 172.21.15.59:0/3219879335 wait complete. 2024-06-29T16:01:32.081 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-06-29T16:01:32.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.199+0000 7f0636919640 1 -- 172.21.15.59:0/461565008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0630162190 msgr2=0x7f06300b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.199+0000 7f0636919640 1 --2- 172.21.15.59:0/461565008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0630162190 0x7f06300b38e0 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f0628005fd0 tx=0x7f062802aa10 comp rx=0 tx=0).stop 2024-06-29T16:01:32.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.199+0000 7f0636919640 1 -- 172.21.15.59:0/461565008 shutdown_connections 2024-06-29T16:01:32.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.199+0000 7f0636919640 1 --2- 172.21.15.59:0/461565008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0630162190 0x7f06300b38e0 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.199+0000 7f0636919640 1 --2- 172.21.15.59:0/461565008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0630162770 0x7f06300ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.199+0000 7f0636919640 1 -- 172.21.15.59:0/461565008 >> 172.21.15.59:0/461565008 conn(0x7f06300d45b0 msgr2=0x7f06300d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:32.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.199+0000 7f0636919640 1 -- 172.21.15.59:0/461565008 shutdown_connections 2024-06-29T16:01:32.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.199+0000 7f0636919640 1 -- 172.21.15.59:0/461565008 wait complete. 2024-06-29T16:01:32.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.201+0000 7f0636919640 1 Processor -- start 2024-06-29T16:01:32.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.201+0000 7f0636919640 1 -- start start 2024-06-29T16:01:32.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.202+0000 7f0636919640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f063015e0a0 0x7f0630159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.202+0000 7f0636919640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630162770 0x7f06301531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.202+0000 7f0636919640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06300c6830 0x7f0630154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.202+0000 7f0636919640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f06300b43a0 con 0x7f063015e0a0 2024-06-29T16:01:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.202+0000 7f0636919640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06300b4220 con 0x7f0630162770 2024-06-29T16:01:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.202+0000 7f0636919640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f06300b4520 con 0x7f06300c6830 2024-06-29T16:01:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f063015e0a0 0x7f0630159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f063015e0a0 0x7f0630159b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44528/0 (socket says 172.21.15.59:44528) 2024-06-29T16:01:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 -- 172.21.15.59:0/3781555283 learned_addr learned my addr 172.21.15.59:0/3781555283 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f0634eb6640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06300c6830 0x7f0630154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062f7fe640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630162770 0x7f06301531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 -- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06300c6830 msgr2=0x7f0630154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06300c6830 0x7f0630154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 -- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630162770 msgr2=0x7f06301531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630162770 0x7f06301531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 -- 172.21.15.59:0/3781555283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0630155c40 con 0x7f063015e0a0 2024-06-29T16:01:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062f7fe640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630162770 0x7f06301531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062ffff640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f063015e0a0 0x7f0630159b70 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f062000d750 tx=0x7f062000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062cff9640 1 -- 172.21.15.59:0/3781555283 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620017070 con 0x7f063015e0a0 2024-06-29T16:01:32.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062cff9640 1 -- 172.21.15.59:0/3781555283 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0620004310 con 0x7f063015e0a0 2024-06-29T16:01:32.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f062cff9640 1 -- 172.21.15.59:0/3781555283 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062000de60 con 0x7f063015e0a0 2024-06-29T16:01:32.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.203+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0630155f20 con 0x7f063015e0a0 2024-06-29T16:01:32.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.204+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06301583f0 con 0x7f063015e0a0 2024-06-29T16:01:32.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.204+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0630158650 con 0x7f063015e0a0 2024-06-29T16:01:32.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.205+0000 7f062cff9640 1 -- 172.21.15.59:0/3781555283 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f0620009c30 con 0x7f063015e0a0 2024-06-29T16:01:32.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.206+0000 7f062cff9640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f061c076420 0x7f061c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.206+0000 7f062cff9640 1 -- 172.21.15.59:0/3781555283 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f0620097ee0 con 0x7f063015e0a0 2024-06-29T16:01:32.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.206+0000 7f062f7fe640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f061c076420 0x7f061c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.207+0000 7f062f7fe640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f061c076420 0x7f061c078930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0624007fd0 tx=0x7f0624009040 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.210+0000 7f062cff9640 1 -- 172.21.15.59:0/3781555283 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0620020020 con 0x7f063015e0a0 2024-06-29T16:01:32.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.400+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f06300b49d0 con 0x7f061c076420 2024-06-29T16:01:32.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.400+0000 7f062cff9640 1 -- 172.21.15.59:0/3781555283 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f06300b49d0 con 0x7f061c076420 2024-06-29T16:01:32.401 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 4 to deep-scrub 2024-06-29T16:01:32.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f061c076420 msgr2=0x7f061c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f061c076420 0x7f061c078930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0624007fd0 tx=0x7f0624009040 comp rx=0 tx=0).stop 2024-06-29T16:01:32.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f063015e0a0 msgr2=0x7f0630159b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f063015e0a0 0x7f0630159b70 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f062000d750 tx=0x7f062000dc10 comp rx=0 tx=0).stop 2024-06-29T16:01:32.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 shutdown_connections 2024-06-29T16:01:32.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f061c076420 0x7f061c078930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06300c6830 0x7f0630154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0630162770 0x7f06301531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 --2- 172.21.15.59:0/3781555283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f063015e0a0 0x7f0630159b70 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 >> 172.21.15.59:0/3781555283 conn(0x7f06300d45b0 msgr2=0x7f063006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:32.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 shutdown_connections 2024-06-29T16:01:32.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.402+0000 7f0636919640 1 -- 172.21.15.59:0/3781555283 wait complete. 2024-06-29T16:01:32.414 INFO:tasks.ceph:Scrubbing osd.5 2024-06-29T16:01:32.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:32.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.537+0000 7fdd98eac640 1 -- 172.21.15.59:0/131435408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940d2f70 msgr2=0x7fdd940d3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.537+0000 7fdd98eac640 1 --2- 172.21.15.59:0/131435408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940d2f70 0x7fdd940d3380 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c009830 tx=0x7fdd8c02a510 comp rx=0 tx=0).stop 2024-06-29T16:01:32.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.537+0000 7fdd98eac640 1 -- 172.21.15.59:0/131435408 shutdown_connections 2024-06-29T16:01:32.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.537+0000 7fdd98eac640 1 --2- 172.21.15.59:0/131435408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940d2f70 0x7fdd940d3380 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.537+0000 7fdd98eac640 1 --2- 172.21.15.59:0/131435408 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd940b7150 0x7fdd940b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.537+0000 7fdd98eac640 1 -- 172.21.15.59:0/131435408 >> 172.21.15.59:0/131435408 conn(0x7fdd940b5590 msgr2=0x7fdd940b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:32.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.537+0000 7fdd98eac640 1 -- 172.21.15.59:0/131435408 shutdown_connections 2024-06-29T16:01:32.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.537+0000 7fdd98eac640 1 -- 172.21.15.59:0/131435408 wait complete. 2024-06-29T16:01:32.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.539+0000 7fdd98eac640 1 Processor -- start 2024-06-29T16:01:32.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.539+0000 7fdd98eac640 1 -- start start 2024-06-29T16:01:32.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd98eac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940b2a50 0x7fdd941f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd98eac640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd940b7150 0x7fdd9414ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd98eac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd940d2f70 0x7fdd9406d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd98eac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fdd940b4510 con 0x7fdd940b2a50 2024-06-29T16:01:32.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd98eac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd940b53e0 con 0x7fdd940d2f70 2024-06-29T16:01:32.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd98eac640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd940b4690 con 0x7fdd940b7150 2024-06-29T16:01:32.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd9259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940b2a50 0x7fdd941f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd9259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940b2a50 0x7fdd941f4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44556/0 (socket says 172.21.15.59:44556) 2024-06-29T16:01:32.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd9259c640 1 -- 172.21.15.59:0/2788432372 learned_addr learned my addr 172.21.15.59:0/2788432372 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd91d9b640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd940b7150 0x7fdd9414ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.540+0000 7fdd92d9d640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd940d2f70 0x7fdd9406d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd9259c640 1 -- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd940b7150 msgr2=0x7fdd9414ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd9259c640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd940b7150 0x7fdd9414ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd9259c640 1 -- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd940d2f70 msgr2=0x7fdd9406d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd9259c640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd940d2f70 0x7fdd9406d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd9259c640 1 -- 172.21.15.59:0/2788432372 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9414d360 con 0x7fdd940b2a50 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd9259c640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940b2a50 0x7fdd941f4a80 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400cbc0 tx=0x7fdd8400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84016020 con 0x7fdd940b2a50 2024-06-29T16:01:32.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd8400ad90 con 0x7fdd940b2a50 2024-06-29T16:01:32.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd941f5340 con 0x7fdd940b2a50 2024-06-29T16:01:32.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84012680 con 0x7fdd940b2a50 2024-06-29T16:01:32.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.541+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd941f5900 con 0x7fdd940b2a50 2024-06-29T16:01:32.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.542+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdd940b7560 con 0x7fdd940b2a50 2024-06-29T16:01:32.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.543+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7fdd84019020 con 0x7fdd940b2a50 2024-06-29T16:01:32.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.543+0000 7fdd8affd640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fdd740763f0 0x7fdd74078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.544+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7fdd84098740 con 0x7fdd940b2a50 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.544+0000 7fdd8affd640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7fdd7407d020 0x7fdd7407f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.544+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 --> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd7407fbe0 con 0x7fdd7407d020 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.544+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7fdd84098bd0 con 0x7fdd940b2a50 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.544+0000 7fdd91d9b640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fdd740763f0 0x7fdd74078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.544+0000 7fdd92d9d640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7fdd7407d020 0x7fdd7407f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.544+0000 7fdd92d9d640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7fdd7407d020 0x7fdd7407f510 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.545+0000 7fdd91d9b640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fdd740763f0 0x7fdd74078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd9415dfb0 tx=0x7fdd7c00a250 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.545+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 <== osd.5 v2:172.21.15.153:6824/4044954575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdd7407fbe0 con 0x7fdd7407d020 2024-06-29T16:01:32.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.559+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 --> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdd9408ccd0 con 0x7fdd7407d020 2024-06-29T16:01:32.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.559+0000 7fdd8affd640 1 -- 172.21.15.59:0/2788432372 <== osd.5 v2:172.21.15.153:6824/4044954575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdd9408ccd0 con 0x7fdd7407d020 2024-06-29T16:01:32.561 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T16:01:32.561 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:32.561 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T16:01:32.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7fdd7407d020 msgr2=0x7fdd7407f510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6824/4044954575,v1:172.21.15.153:6825/4044954575] conn(0x7fdd7407d020 0x7fdd7407f510 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fdd740763f0 msgr2=0x7fdd74078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fdd740763f0 0x7fdd74078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd9415dfb0 tx=0x7fdd7c00a250 comp rx=0 tx=0).stop 2024-06-29T16:01:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940b2a50 msgr2=0x7fdd941f4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 --2- 172.21.15.59:0/2788432372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd940b2a50 0x7fdd941f4a80 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400cbc0 tx=0x7fdd8400a060 comp rx=0 tx=0).stop 2024-06-29T16:01:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd92d9d640 1 -- 172.21.15.59:0/2788432372 reap_dead start 2024-06-29T16:01:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 shutdown_connections 2024-06-29T16:01:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 >> 172.21.15.59:0/2788432372 conn(0x7fdd940b5590 msgr2=0x7fdd940d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 shutdown_connections 2024-06-29T16:01:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.560+0000 7fdd98eac640 1 -- 172.21.15.59:0/2788432372 wait complete. 2024-06-29T16:01:32.571 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-06-29T16:01:32.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.688+0000 7f874effc640 1 -- 172.21.15.59:0/3615929991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87480b40a0 msgr2=0x7f874815e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.688+0000 7f874effc640 1 --2- 172.21.15.59:0/3615929991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87480b40a0 0x7f874815e8d0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f8738005fd0 tx=0x7f873802a5c0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.688+0000 7f874effc640 1 -- 172.21.15.59:0/3615929991 shutdown_connections 2024-06-29T16:01:32.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.688+0000 7f874effc640 1 --2- 172.21.15.59:0/3615929991 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f874815ee10 0x7f874815f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.688+0000 7f874effc640 1 --2- 172.21.15.59:0/3615929991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87480b40a0 0x7f874815e8d0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.688+0000 7f874effc640 1 -- 172.21.15.59:0/3615929991 >> 172.21.15.59:0/3615929991 conn(0x7f87480d3ee0 msgr2=0x7f87480d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:32.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.689+0000 7f874effc640 1 -- 172.21.15.59:0/3615929991 shutdown_connections 2024-06-29T16:01:32.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.689+0000 7f874effc640 1 -- 172.21.15.59:0/3615929991 wait complete. 2024-06-29T16:01:32.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.690+0000 7f874effc640 1 Processor -- start 2024-06-29T16:01:32.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.690+0000 7f874effc640 1 -- start start 2024-06-29T16:01:32.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874effc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87480d23c0 0x7f87481f4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874effc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874815ee10 0x7f87481514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874effc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87481519f0 0x7f87481e7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874effc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f8748160280 con 0x7f87481519f0 2024-06-29T16:01:32.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874effc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8748160100 con 0x7f874815ee10 2024-06-29T16:01:32.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874effc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8748160400 con 0x7f87480d23c0 2024-06-29T16:01:32.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874d599640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87481519f0 0x7f87481e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874d599640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87481519f0 0x7f87481e7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37652/0 (socket says 172.21.15.59:37652) 2024-06-29T16:01:32.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874d599640 1 -- 172.21.15.59:0/2948555078 learned_addr learned my addr 172.21.15.59:0/2948555078 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:32.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f873ffff640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874815ee10 0x7f87481514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874d599640 1 -- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87480d23c0 msgr2=0x7f87481f4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874cd98640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87480d23c0 0x7f87481f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874d599640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87480d23c0 0x7f87481f4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874d599640 1 -- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874815ee10 msgr2=0x7f87481514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874d599640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874815ee10 0x7f87481514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874d599640 1 -- 172.21.15.59:0/2948555078 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8748151e00 con 0x7f87481519f0 2024-06-29T16:01:32.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f873ffff640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874815ee10 0x7f87481514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:32.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.692+0000 7f874cd98640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87480d23c0 0x7f87481f4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:32.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.693+0000 7f874d599640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87481519f0 0x7f87481e7170 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f874000ea40 tx=0x7f874000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.693+0000 7f873d7fa640 1 -- 172.21.15.59:0/2948555078 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874000ce40 con 0x7f87481519f0 2024-06-29T16:01:32.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.693+0000 7f873d7fa640 1 -- 172.21.15.59:0/2948555078 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8740004c10 con 0x7f87481519f0 2024-06-29T16:01:32.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.693+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8748154ef0 con 0x7f87481519f0 2024-06-29T16:01:32.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.693+0000 7f873d7fa640 1 -- 172.21.15.59:0/2948555078 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740005020 con 0x7f87481519f0 2024-06-29T16:01:32.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.693+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87481551d0 con 0x7f87481519f0 2024-06-29T16:01:32.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.694+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f874815e8d0 con 0x7f87481519f0 2024-06-29T16:01:32.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.695+0000 7f873d7fa640 1 -- 172.21.15.59:0/2948555078 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f87400051c0 con 0x7f87481519f0 2024-06-29T16:01:32.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.695+0000 7f873d7fa640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f87340763f0 0x7f8734078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:32.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.695+0000 7f873d7fa640 1 -- 172.21.15.59:0/2948555078 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f8740098a90 con 0x7f87481519f0 2024-06-29T16:01:32.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.695+0000 7f874cd98640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f87340763f0 0x7f8734078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:32.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.696+0000 7f874cd98640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f87340763f0 0x7f8734078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f87480d2e40 tx=0x7f8744009040 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:32.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.699+0000 7f873d7fa640 1 -- 172.21.15.59:0/2948555078 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8748151e00 con 0x7f87481519f0 2024-06-29T16:01:32.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.890+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f8748154c80 con 0x7f87340763f0 2024-06-29T16:01:32.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.891+0000 7f873d7fa640 1 -- 172.21.15.59:0/2948555078 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8748154c80 con 0x7f87340763f0 2024-06-29T16:01:32.892 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 5 to deep-scrub 2024-06-29T16:01:32.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f87340763f0 msgr2=0x7f8734078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f87340763f0 0x7f8734078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f87480d2e40 tx=0x7f8744009040 comp rx=0 tx=0).stop 2024-06-29T16:01:32.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87481519f0 msgr2=0x7f87481e7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:32.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87481519f0 0x7f87481e7170 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f874000ea40 tx=0x7f874000ef00 comp rx=0 tx=0).stop 2024-06-29T16:01:32.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 shutdown_connections 2024-06-29T16:01:32.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f87340763f0 0x7f8734078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87481519f0 0x7f87481e7170 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f874815ee10 0x7f87481514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 --2- 172.21.15.59:0/2948555078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f87480d23c0 0x7f87481f4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:32.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 >> 172.21.15.59:0/2948555078 conn(0x7f87480d3ee0 msgr2=0x7f87480b4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:32.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 shutdown_connections 2024-06-29T16:01:32.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:32.893+0000 7f874effc640 1 -- 172.21.15.59:0/2948555078 wait complete. 2024-06-29T16:01:32.905 INFO:tasks.ceph:Scrubbing osd.6 2024-06-29T16:01:32.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:33.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.027+0000 7f73e6e3c640 1 -- 172.21.15.59:0/3571077053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00d2eb0 msgr2=0x7f73e00b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.027+0000 7f73e6e3c640 1 --2- 172.21.15.59:0/3571077053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00d2eb0 0x7f73e00b4070 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f73e00b3d60 tx=0x7f73d802aa10 comp rx=0 tx=0).stop 2024-06-29T16:01:33.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.027+0000 7f73e6e3c640 1 -- 172.21.15.59:0/3571077053 shutdown_connections 2024-06-29T16:01:33.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.027+0000 7f73e6e3c640 1 --2- 172.21.15.59:0/3571077053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00d2eb0 0x7f73e00b4070 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.027+0000 7f73e6e3c640 1 --2- 172.21.15.59:0/3571077053 >> v2:172.21.15.153:3300/0 conn(0x7f73e00b2980 0x7f73e00b6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.027+0000 7f73e6e3c640 1 -- 172.21.15.59:0/3571077053 >> 172.21.15.59:0/3571077053 conn(0x7f73e00d0f90 msgr2=0x7f73e00d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:33.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.027+0000 7f73e6e3c640 1 -- 172.21.15.59:0/3571077053 shutdown_connections 2024-06-29T16:01:33.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.027+0000 7f73e6e3c640 1 -- 172.21.15.59:0/3571077053 wait complete. 2024-06-29T16:01:33.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.028+0000 7f73e6e3c640 1 Processor -- start 2024-06-29T16:01:33.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.028+0000 7f73e6e3c640 1 -- start start 2024-06-29T16:01:33.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e6e3c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00b2980 0x7f73e014dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e6e3c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e00d1c30 0x7f73e00c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e6e3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e00d2eb0 0x7f73e015cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e6e3c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f73e00b4730 con 0x7f73e00b2980 2024-06-29T16:01:33.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e6e3c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73e00b45b0 con 0x7f73e00d1c30 2024-06-29T16:01:33.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e6e3c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f73e00b48b0 con 0x7f73e00d2eb0 2024-06-29T16:01:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00b2980 0x7f73e014dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00b2980 0x7f73e014dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37678/0 (socket says 172.21.15.59:37678) 2024-06-29T16:01:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 -- 172.21.15.59:0/1415839251 learned_addr learned my addr 172.21.15.59:0/1415839251 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73d7fff640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e00d1c30 0x7f73e00c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e53d9640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e00d2eb0 0x7f73e015cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 -- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e00d2eb0 msgr2=0x7f73e015cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e00d2eb0 0x7f73e015cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 -- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e00d1c30 msgr2=0x7f73e00c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73e00d1c30 0x7f73e00c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 -- 172.21.15.59:0/1415839251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e00c93c0 con 0x7f73e00b2980 2024-06-29T16:01:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e53d9640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f73e00d2eb0 0x7f73e015cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.030+0000 7f73e4bd8640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00b2980 0x7f73e014dff0 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00c3a0 tx=0x7f73dc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.031+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc017070 con 0x7f73e00b2980 2024-06-29T16:01:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.031+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73dc0053c0 con 0x7f73e00b2980 2024-06-29T16:01:33.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.031+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e00cdc60 con 0x7f73e00b2980 2024-06-29T16:01:33.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.031+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc003850 con 0x7f73e00b2980 2024-06-29T16:01:33.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.031+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e00c78b0 con 0x7f73e00b2980 2024-06-29T16:01:33.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.032+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f73e00b4d60 con 0x7f73e00b2980 2024-06-29T16:01:33.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.032+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f73dc01a070 con 0x7f73e00b2980 2024-06-29T16:01:33.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.033+0000 7f73d57fa640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f73cc0763f0 0x7f73cc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.033+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f73dc096e30 con 0x7f73e00b2980 2024-06-29T16:01:33.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.033+0000 7f73d57fa640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7f73cc07d020 0x7f73cc07f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.033+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 --> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73cc07fbe0 con 0x7f73cc07d020 2024-06-29T16:01:33.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.033+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f73dc099530 con 0x7f73e00b2980 2024-06-29T16:01:33.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.033+0000 7f73d7fff640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f73cc0763f0 0x7f73cc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.033+0000 7f73e53d9640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7f73cc07d020 0x7f73cc07f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.034+0000 7f73d7fff640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f73cc0763f0 0x7f73cc078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f73e015db80 tx=0x7f73d000b2e0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.034+0000 7f73e53d9640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7f73cc07d020 0x7f73cc07f510 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.035+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 <== osd.6 v2:172.21.15.153:6808/4247178163 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73cc07fbe0 con 0x7f73cc07d020 2024-06-29T16:01:33.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.048+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 --> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f73e00d2b20 con 0x7f73cc07d020 2024-06-29T16:01:33.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.048+0000 7f73d57fa640 1 -- 172.21.15.59:0/1415839251 <== osd.6 v2:172.21.15.153:6808/4247178163 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f73e00d2b20 con 0x7f73cc07d020 2024-06-29T16:01:33.049 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T16:01:33.050 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:33.050 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T16:01:33.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7f73cc07d020 msgr2=0x7f73cc07f510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6808/4247178163,v1:172.21.15.153:6809/4247178163] conn(0x7f73cc07d020 0x7f73cc07f510 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f73cc0763f0 msgr2=0x7f73cc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f73cc0763f0 0x7f73cc078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f73e015db80 tx=0x7f73d000b2e0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00b2980 msgr2=0x7f73e014dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 --2- 172.21.15.59:0/1415839251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e00b2980 0x7f73e014dff0 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00c3a0 tx=0x7f73dc00c860 comp rx=0 tx=0).stop 2024-06-29T16:01:33.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e53d9640 1 -- 172.21.15.59:0/1415839251 reap_dead start 2024-06-29T16:01:33.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 shutdown_connections 2024-06-29T16:01:33.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 >> 172.21.15.59:0/1415839251 conn(0x7f73e00d0f90 msgr2=0x7f73e01604e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:33.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 shutdown_connections 2024-06-29T16:01:33.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.049+0000 7f73e6e3c640 1 -- 172.21.15.59:0/1415839251 wait complete. 2024-06-29T16:01:33.060 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-06-29T16:01:33.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.179+0000 7f759590a640 1 -- 172.21.15.59:0/362984274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f757c004070 msgr2=0x7f757c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.179+0000 7f759590a640 1 --2- 172.21.15.59:0/362984274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f757c004070 0x7f757c004800 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f75900cc4f0 tx=0x7f757c005830 comp rx=0 tx=0).stop 2024-06-29T16:01:33.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.179+0000 7f758f7fe640 1 -- 172.21.15.59:0/362984274 reap_dead start 2024-06-29T16:01:33.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.179+0000 7f759590a640 1 -- 172.21.15.59:0/362984274 shutdown_connections 2024-06-29T16:01:33.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.179+0000 7f759590a640 1 -- 172.21.15.59:0/362984274 >> 172.21.15.59:0/362984274 conn(0x7f75900b24c0 msgr2=0x7f75900cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:33.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.179+0000 7f759590a640 1 -- 172.21.15.59:0/362984274 shutdown_connections 2024-06-29T16:01:33.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.179+0000 7f759590a640 1 -- 172.21.15.59:0/362984274 wait complete. 2024-06-29T16:01:33.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.181+0000 7f759590a640 1 Processor -- start 2024-06-29T16:01:33.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.181+0000 7f759590a640 1 -- start start 2024-06-29T16:01:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.182+0000 7f759590a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f757c004070 0x7f759015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.182+0000 7f759590a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f759015de70 0x7f7590138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.182+0000 7f759590a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7590138bc0 0x7f759006c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.182+0000 7f759590a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f7590163c30 con 0x7f7590138bc0 2024-06-29T16:01:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.182+0000 7f759590a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7590144980 con 0x7f757c004070 2024-06-29T16:01:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.182+0000 7f759590a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7590163db0 con 0x7f759015de70 2024-06-29T16:01:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7590138bc0 0x7f759006c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7590138bc0 0x7f759006c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37688/0 (socket says 172.21.15.59:37688) 2024-06-29T16:01:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 -- 172.21.15.59:0/330006448 learned_addr learned my addr 172.21.15.59:0/330006448 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758effd640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f757c004070 0x7f759015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758e7fc640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f759015de70 0x7f7590138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 -- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f759015de70 msgr2=0x7f7590138680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f759015de70 0x7f7590138680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 -- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f757c004070 msgr2=0x7f759015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f757c004070 0x7f759015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 -- 172.21.15.59:0/330006448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75901375e0 con 0x7f7590138bc0 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f758f7fe640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7590138bc0 0x7f759006c1e0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f75900cd440 tx=0x7f7588011bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f7587fff640 1 -- 172.21.15.59:0/330006448 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75800019e0 con 0x7f7590138bc0 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f7587fff640 1 -- 172.21.15.59:0/330006448 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f758800f8f0 con 0x7f7590138bc0 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75901378c0 con 0x7f7590138bc0 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f7587fff640 1 -- 172.21.15.59:0/330006448 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758800fd40 con 0x7f7590138bc0 2024-06-29T16:01:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.183+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7590137e80 con 0x7f7590138bc0 2024-06-29T16:01:33.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.185+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f759015c890 con 0x7f7590138bc0 2024-06-29T16:01:33.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.185+0000 7f7587fff640 1 -- 172.21.15.59:0/330006448 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f75880035f0 con 0x7f7590138bc0 2024-06-29T16:01:33.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.186+0000 7f7587fff640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f75700763f0 0x7f7570078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.186+0000 7f7587fff640 1 -- 172.21.15.59:0/330006448 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f7588098240 con 0x7f7590138bc0 2024-06-29T16:01:33.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.186+0000 7f758effd640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f75700763f0 0x7f7570078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.187+0000 7f758effd640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f75700763f0 0x7f7570078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75800019b0 tx=0x7f7580006ef0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.190+0000 7f7587fff640 1 -- 172.21.15.59:0/330006448 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7588011e20 con 0x7f7590138bc0 2024-06-29T16:01:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.380+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f759006d700 con 0x7f75700763f0 2024-06-29T16:01:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.380+0000 7f7587fff640 1 -- 172.21.15.59:0/330006448 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f759006d700 con 0x7f75700763f0 2024-06-29T16:01:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 6 to deep-scrub 2024-06-29T16:01:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f75700763f0 msgr2=0x7f7570078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f75700763f0 0x7f7570078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75800019b0 tx=0x7f7580006ef0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7590138bc0 msgr2=0x7f759006c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7590138bc0 0x7f759006c1e0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f75900cd440 tx=0x7f7588011bd0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 shutdown_connections 2024-06-29T16:01:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f75700763f0 0x7f7570078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7590138bc0 0x7f759006c1e0 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f759015de70 0x7f7590138680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 --2- 172.21.15.59:0/330006448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f757c004070 0x7f759015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 >> 172.21.15.59:0/330006448 conn(0x7f75900b24c0 msgr2=0x7f75900af980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 shutdown_connections 2024-06-29T16:01:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.382+0000 7f759590a640 1 -- 172.21.15.59:0/330006448 wait complete. 2024-06-29T16:01:33.395 INFO:tasks.ceph:Scrubbing osd.7 2024-06-29T16:01:33.395 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:33.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.515+0000 7f287c3b2640 1 -- 172.21.15.59:0/2139624223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740d2f70 msgr2=0x7f28740d3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.515+0000 7f287c3b2640 1 --2- 172.21.15.59:0/2139624223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740d2f70 0x7f28740d3380 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f28740b34d0 tx=0x7f286c02aa30 comp rx=0 tx=0).stop 2024-06-29T16:01:33.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.515+0000 7f287c3b2640 1 -- 172.21.15.59:0/2139624223 shutdown_connections 2024-06-29T16:01:33.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.515+0000 7f287c3b2640 1 --2- 172.21.15.59:0/2139624223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740d2f70 0x7f28740d3380 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.516+0000 7f287c3b2640 1 --2- 172.21.15.59:0/2139624223 >> v2:172.21.15.153:3300/0 conn(0x7f28740b7150 0x7f28740b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.516+0000 7f287c3b2640 1 -- 172.21.15.59:0/2139624223 >> 172.21.15.59:0/2139624223 conn(0x7f28740b5590 msgr2=0x7f28740b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:33.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.516+0000 7f287c3b2640 1 -- 172.21.15.59:0/2139624223 shutdown_connections 2024-06-29T16:01:33.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.516+0000 7f287c3b2640 1 -- 172.21.15.59:0/2139624223 wait complete. 2024-06-29T16:01:33.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.517+0000 7f287c3b2640 1 Processor -- start 2024-06-29T16:01:33.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.517+0000 7f287c3b2640 1 -- start start 2024-06-29T16:01:33.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287c3b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28740b2a50 0x7f2874150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287c3b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740b7150 0x7f28740cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287c3b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28740d2f70 0x7f28740cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287c3b2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f28740b4510 con 0x7f28740b7150 2024-06-29T16:01:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287c3b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28740b53e0 con 0x7f28740d2f70 2024-06-29T16:01:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287c3b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f28740b4690 con 0x7f28740b2a50 2024-06-29T16:01:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740b7150 0x7f28740cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740b7150 0x7f28740cd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37708/0 (socket says 172.21.15.59:37708) 2024-06-29T16:01:33.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 -- 172.21.15.59:0/1756079235 learned_addr learned my addr 172.21.15.59:0/1756079235 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:33.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287a14e640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28740b2a50 0x7f2874150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287a94f640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28740d2f70 0x7f28740cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 -- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28740b2a50 msgr2=0x7f2874150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28740b2a50 0x7f2874150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 -- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28740d2f70 msgr2=0x7f28740cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28740d2f70 0x7f28740cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 -- 172.21.15.59:0/1756079235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28741f5a10 con 0x7f28740b7150 2024-06-29T16:01:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287a94f640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28740d2f70 0x7f28740cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.519+0000 7f287994d640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740b7150 0x7f28740cd7e0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f286400d700 tx=0x7f286400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.523+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2864017070 con 0x7f28740b7150 2024-06-29T16:01:33.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.523+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28740ce5a0 con 0x7f28740b7150 2024-06-29T16:01:33.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.523+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2864005310 con 0x7f28740b7150 2024-06-29T16:01:33.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.523+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2864009820 con 0x7f28740b7150 2024-06-29T16:01:33.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.523+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28740caf60 con 0x7f28740b7150 2024-06-29T16:01:33.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.524+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f28740ce880 con 0x7f28740b7150 2024-06-29T16:01:33.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.525+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f28640099c0 con 0x7f28740b7150 2024-06-29T16:01:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.525+0000 7f286affd640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28580763f0 0x7f2858078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.525+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f2864097bd0 con 0x7f28740b7150 2024-06-29T16:01:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.525+0000 7f286affd640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f285807cfc0 0x7f285807f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.525+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 --> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2864018ec0 con 0x7f285807cfc0 2024-06-29T16:01:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.525+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f286409d2a0 con 0x7f28740b7150 2024-06-29T16:01:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.525+0000 7f287a14e640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28580763f0 0x7f2858078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.526+0000 7f287a94f640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f285807cfc0 0x7f285807f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.526+0000 7f287a14e640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28580763f0 0x7f2858078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2870006fd0 tx=0x7f2870009060 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.526+0000 7f287a94f640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f285807cfc0 0x7f285807f4b0 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.527+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 <== osd.7 v2:172.21.15.153:6816/1688984748 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2864018ec0 con 0x7f285807cfc0 2024-06-29T16:01:33.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.540+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 --> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f28740b4b80 con 0x7f285807cfc0 2024-06-29T16:01:33.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f286affd640 1 -- 172.21.15.59:0/1756079235 <== osd.7 v2:172.21.15.153:6816/1688984748 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f28740b4b80 con 0x7f285807cfc0 2024-06-29T16:01:33.542 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-06-29T16:01:33.542 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:33.542 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-06-29T16:01:33.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f285807cfc0 msgr2=0x7f285807f4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6816/1688984748,v1:172.21.15.153:6817/1688984748] conn(0x7f285807cfc0 0x7f285807f4b0 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28580763f0 msgr2=0x7f2858078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f28580763f0 0x7f2858078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2870006fd0 tx=0x7f2870009060 comp rx=0 tx=0).stop 2024-06-29T16:01:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740b7150 msgr2=0x7f28740cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 --2- 172.21.15.59:0/1756079235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28740b7150 0x7f28740cd7e0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f286400d700 tx=0x7f286400dbc0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287a94f640 1 -- 172.21.15.59:0/1756079235 reap_dead start 2024-06-29T16:01:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 shutdown_connections 2024-06-29T16:01:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 >> 172.21.15.59:0/1756079235 conn(0x7f28740b5590 msgr2=0x7f28740ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 shutdown_connections 2024-06-29T16:01:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.541+0000 7f287c3b2640 1 -- 172.21.15.59:0/1756079235 wait complete. 2024-06-29T16:01:33.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-06-29T16:01:33.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.671+0000 7fe023478640 1 -- 172.21.15.59:0/2618060437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004004800 msgr2=0x7fe004004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.671+0000 7fe023478640 1 --2- 172.21.15.59:0/2618060437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004004800 0x7fe004004c10 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7fe010009cf0 tx=0x7fe004005830 comp rx=0 tx=0).stop 2024-06-29T16:01:33.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.671+0000 7fe023478640 1 -- 172.21.15.59:0/2618060437 shutdown_connections 2024-06-29T16:01:33.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.671+0000 7fe023478640 1 --2- 172.21.15.59:0/2618060437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe004004800 0x7fe004004c10 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.671+0000 7fe023478640 1 -- 172.21.15.59:0/2618060437 >> 172.21.15.59:0/2618060437 conn(0x7fe01c0b5610 msgr2=0x7fe01c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:33.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.671+0000 7fe023478640 1 -- 172.21.15.59:0/2618060437 shutdown_connections 2024-06-29T16:01:33.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.671+0000 7fe023478640 1 -- 172.21.15.59:0/2618060437 wait complete. 2024-06-29T16:01:33.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.673+0000 7fe023478640 1 Processor -- start 2024-06-29T16:01:33.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.673+0000 7fe023478640 1 -- start start 2024-06-29T16:01:33.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe023478640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe004004800 0x7fe01c151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe023478640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01c158770 0x7fe01c158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe023478640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe01c160400 0x7fe01c160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe023478640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c162d90 con 0x7fe01c158770 2024-06-29T16:01:33.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe023478640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c162c10 con 0x7fe01c160400 2024-06-29T16:01:33.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe023478640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe01c162f10 con 0x7fe004004800 2024-06-29T16:01:33.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe020a13640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01c158770 0x7fe01c158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe020a13640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01c158770 0x7fe01c158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37734/0 (socket says 172.21.15.59:37734) 2024-06-29T16:01:33.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe020a13640 1 -- 172.21.15.59:0/959882201 learned_addr learned my addr 172.21.15.59:0/959882201 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:33.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe021214640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe004004800 0x7fe01c151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe021a15640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe01c160400 0x7fe01c160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe020a13640 1 -- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe004004800 msgr2=0x7fe01c151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe020a13640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe004004800 0x7fe01c151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe020a13640 1 -- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe01c160400 msgr2=0x7fe01c160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe020a13640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe01c160400 0x7fe01c160810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe020a13640 1 -- 172.21.15.59:0/959882201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01c160d50 con 0x7fe01c158770 2024-06-29T16:01:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.674+0000 7fe021a15640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe01c160400 0x7fe01c160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.675+0000 7fe020a13640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01c158770 0x7fe01c158b80 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7fe010009c40 tx=0x7fe010009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.675+0000 7fe019ffb640 1 -- 172.21.15.59:0/959882201 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe010042020 con 0x7fe01c158770 2024-06-29T16:01:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.675+0000 7fe019ffb640 1 -- 172.21.15.59:0/959882201 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0100049b0 con 0x7fe01c158770 2024-06-29T16:01:33.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.675+0000 7fe019ffb640 1 -- 172.21.15.59:0/959882201 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe010004e00 con 0x7fe01c158770 2024-06-29T16:01:33.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.675+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe01c154cc0 con 0x7fe01c158770 2024-06-29T16:01:33.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.675+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe01c155280 con 0x7fe01c158770 2024-06-29T16:01:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.676+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe01c0b6400 con 0x7fe01c158770 2024-06-29T16:01:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.677+0000 7fe019ffb640 1 -- 172.21.15.59:0/959882201 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7fe010045020 con 0x7fe01c158770 2024-06-29T16:01:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.677+0000 7fe019ffb640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe0080763f0 0x7fe008078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.678+0000 7fe019ffb640 1 -- 172.21.15.59:0/959882201 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7fe0100b9040 con 0x7fe01c158770 2024-06-29T16:01:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.678+0000 7fe021214640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe0080763f0 0x7fe008078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.678+0000 7fe021214640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe0080763f0 0x7fe008078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe01c06ba10 tx=0x7fe014006b70 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:33.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.682+0000 7fe019ffb640 1 -- 172.21.15.59:0/959882201 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe01c0b6400 con 0x7fe01c158770 2024-06-29T16:01:33.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.874+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fe01c0cf1f0 con 0x7fe0080763f0 2024-06-29T16:01:33.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.875+0000 7fe019ffb640 1 -- 172.21.15.59:0/959882201 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe01c0cf1f0 con 0x7fe0080763f0 2024-06-29T16:01:33.876 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 7 to deep-scrub 2024-06-29T16:01:33.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe0080763f0 msgr2=0x7fe008078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe0080763f0 0x7fe008078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe01c06ba10 tx=0x7fe014006b70 comp rx=0 tx=0).stop 2024-06-29T16:01:33.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01c158770 msgr2=0x7fe01c158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:33.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01c158770 0x7fe01c158b80 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7fe010009c40 tx=0x7fe010009970 comp rx=0 tx=0).stop 2024-06-29T16:01:33.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 shutdown_connections 2024-06-29T16:01:33.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fe0080763f0 0x7fe008078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe01c160400 0x7fe01c160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01c158770 0x7fe01c158b80 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 --2- 172.21.15.59:0/959882201 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe004004800 0x7fe01c151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:33.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 >> 172.21.15.59:0/959882201 conn(0x7fe01c0b5610 msgr2=0x7fe01c0c9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:33.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 shutdown_connections 2024-06-29T16:01:33.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:33.877+0000 7fe023478640 1 -- 172.21.15.59:0/959882201 wait complete. 2024-06-29T16:01:33.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.006+0000 7f33b73a1640 1 -- 172.21.15.59:0/2647617132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33b00b3b50 msgr2=0x7f33b0166ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.006+0000 7f33b73a1640 1 --2- 172.21.15.59:0/2647617132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33b00b3b50 0x7f33b0166ca0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009be0 tx=0x7f33a802a920 comp rx=0 tx=0).stop 2024-06-29T16:01:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.006+0000 7f33b73a1640 1 -- 172.21.15.59:0/2647617132 shutdown_connections 2024-06-29T16:01:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.006+0000 7f33b73a1640 1 --2- 172.21.15.59:0/2647617132 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33b00b3b50 0x7f33b0166ca0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.006+0000 7f33b73a1640 1 -- 172.21.15.59:0/2647617132 >> 172.21.15.59:0/2647617132 conn(0x7f33b006c950 msgr2=0x7f33b0073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.006+0000 7f33b73a1640 1 -- 172.21.15.59:0/2647617132 shutdown_connections 2024-06-29T16:01:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.006+0000 7f33b73a1640 1 -- 172.21.15.59:0/2647617132 wait complete. 2024-06-29T16:01:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.008+0000 7f33b73a1640 1 Processor -- start 2024-06-29T16:01:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.008+0000 7f33b73a1640 1 -- start start 2024-06-29T16:01:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b73a1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33b00b3740 0x7f33b006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b73a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b00b3b50 0x7f33b015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:34.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b73a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33b015cd70 0x7f33b015d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:34.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b73a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f33b0167570 con 0x7f33b015cd70 2024-06-29T16:01:34.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b73a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f33b01673f0 con 0x7f33b00b3b50 2024-06-29T16:01:34.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b73a1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f33b01676f0 con 0x7f33b00b3740 2024-06-29T16:01:34.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b593e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33b015cd70 0x7f33b015d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:34.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b593e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33b015cd70 0x7f33b015d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37748/0 (socket says 172.21.15.59:37748) 2024-06-29T16:01:34.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b593e640 1 -- 172.21.15.59:0/3832641058 learned_addr learned my addr 172.21.15.59:0/3832641058 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:34.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b513d640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33b00b3740 0x7f33b006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.009+0000 7f33b493c640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b00b3b50 0x7f33b015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33b593e640 1 -- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33b00b3740 msgr2=0x7f33b006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33b593e640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33b00b3740 0x7f33b006e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33b593e640 1 -- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b00b3b50 msgr2=0x7f33b015b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33b593e640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b00b3b50 0x7f33b015b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33b593e640 1 -- 172.21.15.59:0/3832641058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b015d6c0 con 0x7f33b015cd70 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33b593e640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33b015cd70 0x7f33b015d180 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009d00 tx=0x7f33a8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33a5ffb640 1 -- 172.21.15.59:0/3832641058 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8030c50 con 0x7f33b015cd70 2024-06-29T16:01:34.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b006fb60 con 0x7f33b015cd70 2024-06-29T16:01:34.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33a5ffb640 1 -- 172.21.15.59:0/3832641058 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33a803e070 con 0x7f33b015cd70 2024-06-29T16:01:34.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33a5ffb640 1 -- 172.21.15.59:0/3832641058 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8032330 con 0x7f33b015cd70 2024-06-29T16:01:34.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.010+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b01392d0 con 0x7f33b015cd70 2024-06-29T16:01:34.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.011+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b01458a0 con 0x7f33b015cd70 2024-06-29T16:01:34.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.012+0000 7f33a5ffb640 1 -- 172.21.15.59:0/3832641058 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f33a80324d0 con 0x7f33b015cd70 2024-06-29T16:01:34.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.013+0000 7f33a5ffb640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f3394076420 0x7f3394078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:34.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.013+0000 7f33a5ffb640 1 -- 172.21.15.59:0/3832641058 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f33a80b9380 con 0x7f33b015cd70 2024-06-29T16:01:34.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.013+0000 7f33b513d640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f3394076420 0x7f3394078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:34.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.013+0000 7f33b513d640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f3394076420 0x7f3394078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f33ac007f30 tx=0x7f33ac007ec0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:34.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.017+0000 7f33a5ffb640 1 -- 172.21.15.59:0/3832641058 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f33b015d6c0 con 0x7f33b015cd70 2024-06-29T16:01:34.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.210+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f33b0138da0 con 0x7f3394076420 2024-06-29T16:01:34.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.211+0000 7f33a5ffb640 1 -- 172.21.15.59:0/3832641058 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13396 (secure 0 0 0) 0x7f33b0138da0 con 0x7f3394076420 2024-06-29T16:01:34.212 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:34.213 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T16:01:34.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f3394076420 msgr2=0x7f3394078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:34.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f3394076420 0x7f3394078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f33ac007f30 tx=0x7f33ac007ec0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33b015cd70 msgr2=0x7f33b015d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:34.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33b015cd70 0x7f33b015d180 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f33a8009d00 tx=0x7f33a8003ba0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 shutdown_connections 2024-06-29T16:01:34.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f3394076420 0x7f3394078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33b015cd70 0x7f33b015d180 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33b00b3b50 0x7f33b015b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 --2- 172.21.15.59:0/3832641058 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33b00b3740 0x7f33b006e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:34.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 >> 172.21.15.59:0/3832641058 conn(0x7f33b006c950 msgr2=0x7f33b015fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:34.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 shutdown_connections 2024-06-29T16:01:34.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:34.213+0000 7f33b73a1640 1 -- 172.21.15.59:0/3832641058 wait complete. 2024-06-29T16:01:34.225 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-06-29T16:01:34.114467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":432688,"kb_used_data":29128,"kb_used_omap":12,"kb_used_meta":403443,"kb_avail":754542032,"statfs":{"total":773094113280,"available":772651040768,"internally_reserved":0,"allocated":29827072,"data_stored":16070080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":413126194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-150,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":50,"num_read_kb":169,"num_write":11,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.617553"},"pg_stats":[{"pgid":"1.0","version":"280'287","reported_seq":1412,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-29T16:01:27.346332+0000","last_change":"2024-06-29T14:21:43.647578+0000","last_active":"2024-06-29T16:01:27.346332+0000","last_peered":"2024-06-29T16:01:27.346332+0000","last_clean":"2024-06-29T16:01:27.346332+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T16:01:27.346332+0000","last_undegraded":"2024-06-29T16:01:27.346332+0000","last_fullsized":"2024-06-29T16:01:27.346332+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:21:42.456603+0000","last_clean_scrub_stamp":"2024-06-29T14:21:42.456603+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:41:55.593222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543333,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53308,"kb_used_data":4076,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318532,"statfs":{"total":96636764160,"available":96582176768,"internally_reserved":0,"allocated":4173824,"data_stored":2451536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543334,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48564,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323276,"statfs":{"total":96636764160,"available":96587034624,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543333,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70772,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301068,"statfs":{"total":96636764160,"available":96564293632,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510630,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52728,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319112,"statfs":{"total":96636764160,"available":96582770688,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510631,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49144,"kb_used_data":4076,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322696,"statfs":{"total":96636764160,"available":96586440704,"internally_reserved":0,"allocated":4173824,"data_stored":2451536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:34.227 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-06-29T14:21:42.456603+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=29, tm_hour=14, tm_min=21, tm_sec=42, tm_wday=5, tm_yday=181, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=29, tm_hour=16, tm_min=1, tm_sec=28, tm_wday=5, tm_yday=181, tm_isdst=0) 2024-06-29T16:01:34.228 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-06-29T16:01:44.201 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:44.204 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:54.229 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:54.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.362+0000 7f0137f7f640 1 -- 172.21.15.59:0/2203555912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01300ce530 msgr2=0x7f01300b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.362+0000 7f0137f7f640 1 --2- 172.21.15.59:0/2203555912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01300ce530 0x7f01300b3800 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f0120009830 tx=0x7f012002a510 comp rx=0 tx=0).stop 2024-06-29T16:01:54.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.362+0000 7f0137f7f640 1 -- 172.21.15.59:0/2203555912 shutdown_connections 2024-06-29T16:01:54.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.362+0000 7f0137f7f640 1 --2- 172.21.15.59:0/2203555912 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01300b3d40 0x7f01300cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.362+0000 7f0137f7f640 1 --2- 172.21.15.59:0/2203555912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01300ce530 0x7f01300b3800 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.362+0000 7f0137f7f640 1 -- 172.21.15.59:0/2203555912 >> 172.21.15.59:0/2203555912 conn(0x7f01300b24c0 msgr2=0x7f01300cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:54.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.362+0000 7f0137f7f640 1 -- 172.21.15.59:0/2203555912 shutdown_connections 2024-06-29T16:01:54.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.362+0000 7f0137f7f640 1 -- 172.21.15.59:0/2203555912 wait complete. 2024-06-29T16:01:54.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.364+0000 7f0137f7f640 1 Processor -- start 2024-06-29T16:01:54.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.364+0000 7f0137f7f640 1 -- start start 2024-06-29T16:01:54.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0137f7f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f013006fea0 0x7f01301f13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:54.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0137f7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01300b3d40 0x7f01301507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:54.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0137f7f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01300ce530 0x7f013015c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:54.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0137f7f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f01300cd590 con 0x7f013006fea0 2024-06-29T16:01:54.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0137f7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01300cd410 con 0x7f01300b3d40 2024-06-29T16:01:54.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0137f7f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f013015dee0 con 0x7f01300ce530 2024-06-29T16:01:54.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0135d1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f013006fea0 0x7f01301f13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:54.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0135d1b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f013006fea0 0x7f01301f13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54986/0 (socket says 172.21.15.59:54986) 2024-06-29T16:01:54.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f0135d1b640 1 -- 172.21.15.59:0/3280174873 learned_addr learned my addr 172.21.15.59:0/3280174873 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:54.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f013651c640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01300ce530 0x7f013015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:54.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.365+0000 7f013551a640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01300b3d40 0x7f01301507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:54.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f0135d1b640 1 -- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01300ce530 msgr2=0x7f013015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f0135d1b640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01300ce530 0x7f013015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f0135d1b640 1 -- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01300b3d40 msgr2=0x7f01301507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f0135d1b640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01300b3d40 0x7f01301507b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f0135d1b640 1 -- 172.21.15.59:0/3280174873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01301513f0 con 0x7f013006fea0 2024-06-29T16:01:54.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f013551a640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01300b3d40 0x7f01301507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:54.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f0135d1b640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f013006fea0 0x7f01301f13b0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f012c00c3f0 tx=0x7f012c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:54.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f01267fc640 1 -- 172.21.15.59:0/3280174873 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c017070 con 0x7f013006fea0 2024-06-29T16:01:54.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f01267fc640 1 -- 172.21.15.59:0/3280174873 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f012c0035f0 con 0x7f013006fea0 2024-06-29T16:01:54.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01301516d0 con 0x7f013006fea0 2024-06-29T16:01:54.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f01267fc640 1 -- 172.21.15.59:0/3280174873 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c003a40 con 0x7f013006fea0 2024-06-29T16:01:54.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.366+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0130151c90 con 0x7f013006fea0 2024-06-29T16:01:54.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.367+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01300cccc0 con 0x7f013006fea0 2024-06-29T16:01:54.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.368+0000 7f01267fc640 1 -- 172.21.15.59:0/3280174873 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f012c01a070 con 0x7f013006fea0 2024-06-29T16:01:54.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.369+0000 7f01267fc640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0114076420 0x7f0114078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.369+0000 7f01267fc640 1 -- 172.21.15.59:0/3280174873 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f012c0970c0 con 0x7f013006fea0 2024-06-29T16:01:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.369+0000 7f013551a640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0114076420 0x7f0114078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.370+0000 7f013551a640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0114076420 0x7f0114078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0120034000 tx=0x7f0120004730 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.373+0000 7f01267fc640 1 -- 172.21.15.59:0/3280174873 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f012c021020 con 0x7f013006fea0 2024-06-29T16:01:54.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.564+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 --> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0130072500 con 0x7f0114076420 2024-06-29T16:01:54.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.565+0000 7f01267fc640 1 -- 172.21.15.59:0/3280174873 <== mgr.11045 v2:172.21.15.153:6832/2329467792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13392 (secure 0 0 0) 0x7f0130072500 con 0x7f0114076420 2024-06-29T16:01:54.567 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:54.567 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-06-29T16:01:54.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0114076420 msgr2=0x7f0114078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0114076420 0x7f0114078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0120034000 tx=0x7f0120004730 comp rx=0 tx=0).stop 2024-06-29T16:01:54.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f013006fea0 msgr2=0x7f01301f13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f013006fea0 0x7f01301f13b0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f012c00c3f0 tx=0x7f012c00c8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 shutdown_connections 2024-06-29T16:01:54.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f0114076420 0x7f0114078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f01300ce530 0x7f013015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01300b3d40 0x7f01301507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 --2- 172.21.15.59:0/3280174873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f013006fea0 0x7f01301f13b0 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 >> 172.21.15.59:0/3280174873 conn(0x7f01300b24c0 msgr2=0x7f013014f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:54.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.567+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 shutdown_connections 2024-06-29T16:01:54.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.568+0000 7f0137f7f640 1 -- 172.21.15.59:0/3280174873 wait complete. 2024-06-29T16:01:54.579 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T16:01:54.119021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":288,"ondisk_log_size":288,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":432676,"kb_used_data":29128,"kb_used_omap":12,"kb_used_meta":403443,"kb_avail":754542044,"statfs":{"total":773094113280,"available":772651053056,"internally_reserved":0,"allocated":29827072,"data_stored":16070080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":413126194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"1.0","version":"283'288","reported_seq":1422,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-29T16:01:34.102051+0000","last_change":"2024-06-29T16:01:34.102093+0000","last_active":"2024-06-29T16:01:34.102051+0000","last_peered":"2024-06-29T16:01:34.102051+0000","last_clean":"2024-06-29T16:01:34.102051+0000","last_became_active":"2024-06-29T14:21:43.646449+0000","last_became_peered":"2024-06-29T14:21:43.646449+0000","last_unstale":"2024-06-29T16:01:34.102051+0000","last_undegraded":"2024-06-29T16:01:34.102051+0000","last_fullsized":"2024-06-29T16:01:34.102051+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'288","last_scrub_stamp":"2024-06-29T16:01:34.101953+0000","last_deep_scrub":"283'288","last_deep_scrub_stamp":"2024-06-29T16:01:34.101953+0000","last_clean_scrub_stamp":"2024-06-29T16:01:34.101953+0000","objects_scrubbed":2,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:08:51.789060+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1908,"num_read_kb":5496,"num_write":469,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":288,"ondisk_log_size":288,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53300,"kb_used_data":4076,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318540,"statfs":{"total":96636764160,"available":96582184960,"internally_reserved":0,"allocated":4173824,"data_stored":2451536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48564,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323276,"statfs":{"total":96636764160,"available":96587034624,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543337,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70772,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301068,"statfs":{"total":96636764160,"available":96564293632,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543336,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424510634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52724,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319116,"statfs":{"total":96636764160,"available":96582774784,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424510633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52728,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319112,"statfs":{"total":96636764160,"available":96582770688,"internally_reserved":0,"allocated":3579904,"data_stored":1861168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424510635,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49140,"kb_used_data":4076,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322700,"statfs":{"total":96636764160,"available":96586444800,"internally_reserved":0,"allocated":4173824,"data_stored":2451536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:54.581 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-06-29T16:01:54.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.697+0000 7fcd1463d640 1 -- 172.21.15.59:0/3779974794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c145610 msgr2=0x7fcd0c073e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.697+0000 7fcd1463d640 1 --2- 172.21.15.59:0/3779974794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c145610 0x7fcd0c073e80 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fccfc005fd0 tx=0x7fccfc02a8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.698+0000 7fcd1463d640 1 -- 172.21.15.59:0/3779974794 shutdown_connections 2024-06-29T16:01:54.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.698+0000 7fcd1463d640 1 --2- 172.21.15.59:0/3779974794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c145610 0x7fcd0c073e80 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.698+0000 7fcd1463d640 1 -- 172.21.15.59:0/3779974794 >> 172.21.15.59:0/3779974794 conn(0x7fcd0c0b24c0 msgr2=0x7fcd0c0cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:54.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.698+0000 7fcd1463d640 1 -- 172.21.15.59:0/3779974794 shutdown_connections 2024-06-29T16:01:54.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.698+0000 7fcd1463d640 1 -- 172.21.15.59:0/3779974794 wait complete. 2024-06-29T16:01:54.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.699+0000 7fcd1463d640 1 Processor -- start 2024-06-29T16:01:54.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.700+0000 7fcd1463d640 1 -- start start 2024-06-29T16:01:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd1463d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c0cccc0 0x7fcd0c13e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd1463d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd0c145610 0x7fcd0c0a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd1463d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd0c15c320 0x7fcd0c15f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd1463d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0c0ceb40 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd1463d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0c0ce9c0 con 0x7fcd0c145610 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd1463d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcd0c0cecc0 con 0x7fcd0c15c320 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c0cccc0 0x7fcd0c13e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c0cccc0 0x7fcd0c13e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55010/0 (socket says 172.21.15.59:55010) 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 -- 172.21.15.59:0/3812630708 learned_addr learned my addr 172.21.15.59:0/3812630708 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 -- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd0c15c320 msgr2=0x7fcd0c15f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd0c15c320 0x7fcd0c15f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 -- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd0c145610 msgr2=0x7fcd0c0a96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd11bd8640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd0c145610 0x7fcd0c0a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:54.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd0c145610 0x7fcd0c0a96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 -- 172.21.15.59:0/3812630708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0c13ebe0 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd11bd8640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd0c145610 0x7fcd0c0a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:54.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.701+0000 7fcd123d9640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c0cccc0 0x7fcd0c13e6a0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009cd0 tx=0x7fccfc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:54.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.702+0000 7fcd0affd640 1 -- 172.21.15.59:0/3812630708 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc031da0 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.702+0000 7fcd0affd640 1 -- 172.21.15.59:0/3812630708 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fccfc003e10 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.702+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd0c1f4620 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.702+0000 7fcd0affd640 1 -- 172.21.15.59:0/3812630708 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc033330 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.702+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd0c1f48c0 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.703+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd0c0cd9b0 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.704+0000 7fcd0affd640 1 -- 172.21.15.59:0/3812630708 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7fccfc0334d0 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.704+0000 7fcd0affd640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fccf80763f0 0x7fccf8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.704+0000 7fcd0affd640 1 -- 172.21.15.59:0/3812630708 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7fccfc0b9280 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.705+0000 7fcd11bd8640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fccf80763f0 0x7fccf8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.705+0000 7fcd11bd8640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fccf80763f0 0x7fccf8078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c15d050 tx=0x7fcd00008040 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.709+0000 7fcd0affd640 1 -- 172.21.15.59:0/3812630708 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcd0c0cd9b0 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.899+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fcd0c0689f0 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.903+0000 7fcd0affd640 1 -- 172.21.15.59:0/3812630708 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v88)=0 v88) ==== 123+0+0 (secure 0 0 0) 0x7fcd0c0689f0 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.904+0000 7fcd0affd640 1 -- 172.21.15.59:0/3812630708 <== mon.0 v2:172.21.15.59:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fccfc081b30 con 0x7fcd0c0cccc0 2024-06-29T16:01:54.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fccf80763f0 msgr2=0x7fccf8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fccf80763f0 0x7fccf8078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c15d050 tx=0x7fcd00008040 comp rx=0 tx=0).stop 2024-06-29T16:01:54.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c0cccc0 msgr2=0x7fcd0c13e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:54.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c0cccc0 0x7fcd0c13e6a0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009cd0 tx=0x7fccfc003c20 comp rx=0 tx=0).stop 2024-06-29T16:01:54.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 shutdown_connections 2024-06-29T16:01:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7fccf80763f0 0x7fccf8078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcd0c15c320 0x7fcd0c15f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd0c145610 0x7fcd0c0a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 --2- 172.21.15.59:0/3812630708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd0c0cccc0 0x7fcd0c13e6a0 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 >> 172.21.15.59:0/3812630708 conn(0x7fcd0c0b24c0 msgr2=0x7fcd0c0720a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 shutdown_connections 2024-06-29T16:01:54.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:54.906+0000 7fcd1463d640 1 -- 172.21.15.59:0/3812630708 wait complete. 2024-06-29T16:01:54.917 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T16:01:54.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:55.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 -- 172.21.15.59:0/739916631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36201458a0 msgr2=0x7f36200cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:55.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 --2- 172.21.15.59:0/739916631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36201458a0 0x7f36200cf870 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f36200d0730 tx=0x7f361802a560 comp rx=0 tx=0).stop 2024-06-29T16:01:55.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 -- 172.21.15.59:0/739916631 shutdown_connections 2024-06-29T16:01:55.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 --2- 172.21.15.59:0/739916631 >> v2:172.21.15.153:3300/0 conn(0x7f362015c2f0 0x7f362015c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 --2- 172.21.15.59:0/739916631 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36200b3740 0x7f36200b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 --2- 172.21.15.59:0/739916631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36201458a0 0x7f36200cf870 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 -- 172.21.15.59:0/739916631 >> 172.21.15.59:0/739916631 conn(0x7f362006c950 msgr2=0x7f3620073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:55.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 -- 172.21.15.59:0/739916631 shutdown_connections 2024-06-29T16:01:55.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.034+0000 7f3624b54640 1 -- 172.21.15.59:0/739916631 wait complete. 2024-06-29T16:01:55.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.036+0000 7f3624b54640 1 Processor -- start 2024-06-29T16:01:55.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.036+0000 7f3624b54640 1 -- start start 2024-06-29T16:01:55.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f3624b54640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36200b3740 0x7f362015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:55.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f3624b54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f362015c2f0 0x7f36200aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:55.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f3624b54640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36200ab040 0x7f36200ab450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:55.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f3624b54640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 -- 0x7f362015fef0 con 0x7f362015c2f0 2024-06-29T16:01:55.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f3624b54640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f362015fd70 con 0x7f36200b3740 2024-06-29T16:01:55.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f3624b54640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3620160070 con 0x7f36200ab040 2024-06-29T16:01:55.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f361dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f362015c2f0 0x7f36200aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:55.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f361dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f362015c2f0 0x7f36200aab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55030/0 (socket says 172.21.15.59:55030) 2024-06-29T16:01:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.037+0000 7f361dd9b640 1 -- 172.21.15.59:0/1735521752 learned_addr learned my addr 172.21.15.59:0/1735521752 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-06-29T16:01:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f361ed9d640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36200ab040 0x7f36200ab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f361e59c640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36200b3740 0x7f362015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f361dd9b640 1 -- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36200ab040 msgr2=0x7f36200ab450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f361dd9b640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36200ab040 0x7f36200ab450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f361dd9b640 1 -- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36200b3740 msgr2=0x7f362015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f361dd9b640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36200b3740 0x7f362015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f361dd9b640 1 -- 172.21.15.59:0/1735521752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3620140ba0 con 0x7f362015c2f0 2024-06-29T16:01:55.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f361dd9b640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f362015c2f0 0x7f36200aab00 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f360800d690 tx=0x7f360800db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:55.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f3616ffd640 1 -- 172.21.15.59:0/1735521752 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608017070 con 0x7f362015c2f0 2024-06-29T16:01:55.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f3616ffd640 1 -- 172.21.15.59:0/1735521752 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f3608005330 con 0x7f362015c2f0 2024-06-29T16:01:55.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36200ab990 con 0x7f362015c2f0 2024-06-29T16:01:55.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f3616ffd640 1 -- 172.21.15.59:0/1735521752 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608009820 con 0x7f362015c2f0 2024-06-29T16:01:55.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.038+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36201408c0 con 0x7f362015c2f0 2024-06-29T16:01:55.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.039+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3620160650 con 0x7f362015c2f0 2024-06-29T16:01:55.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.040+0000 7f3616ffd640 1 -- 172.21.15.59:0/1735521752 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 165) ==== 98303+0+0 (secure 0 0 0) 0x7f36080099c0 con 0x7f362015c2f0 2024-06-29T16:01:55.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.040+0000 7f3616ffd640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f36000763f0 0x7f3600078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:55.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.041+0000 7f3616ffd640 1 -- 172.21.15.59:0/1735521752 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 6236+0+0 (secure 0 0 0) 0x7f3608097cf0 con 0x7f362015c2f0 2024-06-29T16:01:55.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.041+0000 7f361e59c640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f36000763f0 0x7f3600078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:55.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.041+0000 7f361e59c640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f36000763f0 0x7f3600078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3618033000 tx=0x7f36180046c0 comp rx=0 tx=0).ready entity=mgr.11045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:55.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.045+0000 7f3616ffd640 1 -- 172.21.15.59:0/1735521752 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3608060ab0 con 0x7f362015c2f0 2024-06-29T16:01:55.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.275+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3620070600 con 0x7f362015c2f0 2024-06-29T16:01:55.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.276+0000 7f3616ffd640 1 -- 172.21.15.59:0/1735521752 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+3923 (secure 0 0 0) 0x7f36080690f0 con 0x7f362015c2f0 2024-06-29T16:01:55.277 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-06-29T16:01:55.277 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":159,"btime":"2024-06-29T16:01:23:486230+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1039110220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1039110220},{"type":"v1","addr":"172.21.15.153:6835","nonce":1039110220}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":12583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/3996608135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":3996608135},{"type":"v1","addr":"172.21.15.153:6837","nonce":3996608135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152},{"gid":13620,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/952472867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":952472867},{"type":"v1","addr":"172.21.15.59:6834","nonce":952472867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151},{"gid":14070,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3782030803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3782030803},{"type":"v1","addr":"172.21.15.59:6835","nonce":3782030803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":159}],"filesystems":[]} 2024-06-29T16:01:55.278 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 159 2024-06-29T16:01:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f36000763f0 msgr2=0x7f3600078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f36000763f0 0x7f3600078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3618033000 tx=0x7f36180046c0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f362015c2f0 msgr2=0x7f36200aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:55.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f362015c2f0 0x7f36200aab00 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f360800d690 tx=0x7f360800db50 comp rx=0 tx=0).stop 2024-06-29T16:01:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 shutdown_connections 2024-06-29T16:01:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:6832/2329467792,v1:172.21.15.153:6833/2329467792] conn(0x7f36000763f0 0x7f3600078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f36200ab040 0x7f36200ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f362015c2f0 0x7f36200aab00 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 --2- 172.21.15.59:0/1735521752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36200b3740 0x7f362015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 >> 172.21.15.59:0/1735521752 conn(0x7f362006c950 msgr2=0x7f36200d0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:55.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 shutdown_connections 2024-06-29T16:01:55.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-06-29T16:01:55.278+0000 7f3624b54640 1 -- 172.21.15.59:0/1735521752 wait complete. 2024-06-29T16:01:55.290 DEBUG:tasks.cephfs.filesystem:already dead... 2024-06-29T16:01:55.290 INFO:teuthology.misc:Shutting down mds daemons... 2024-06-29T16:01:55.291 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T16:01:55.291 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.300 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T16:01:55.300 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T16:01:55.300 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.353 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T16:01:55.353 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T16:01:55.353 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.366 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T16:01:55.366 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T16:01:55.366 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.418 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T16:01:55.419 INFO:teuthology.misc:Shutting down osd daemons... 2024-06-29T16:01:55.419 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-06-29T16:01:55.420 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.462 INFO:tasks.ceph.osd.0:Stopped 2024-06-29T16:01:55.463 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-06-29T16:01:55.463 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.501 INFO:tasks.ceph.osd.1:Stopped 2024-06-29T16:01:55.502 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-06-29T16:01:55.502 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.580 INFO:tasks.ceph.osd.2:Stopped 2024-06-29T16:01:55.580 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-06-29T16:01:55.581 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.650 INFO:tasks.ceph.osd.3:Stopped 2024-06-29T16:01:55.651 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-06-29T16:01:55.651 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.690 INFO:tasks.ceph.osd.4:Stopped 2024-06-29T16:01:55.691 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-06-29T16:01:55.691 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.761 INFO:tasks.ceph.osd.5:Stopped 2024-06-29T16:01:55.762 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-06-29T16:01:55.762 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.835 INFO:tasks.ceph.osd.6:Stopped 2024-06-29T16:01:55.836 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-06-29T16:01:55.836 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.917 INFO:tasks.ceph.osd.7:Stopped 2024-06-29T16:01:55.918 INFO:teuthology.misc:Shutting down mgr daemons... 2024-06-29T16:01:55.918 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-06-29T16:01:55.919 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.952 INFO:tasks.ceph.mgr.y:Stopped 2024-06-29T16:01:55.952 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-06-29T16:01:55.953 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:55.994 INFO:tasks.ceph.mgr.x:Stopped 2024-06-29T16:01:55.994 INFO:teuthology.misc:Shutting down mon daemons... 2024-06-29T16:01:55.994 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-29T16:01:55.994 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:56.011 INFO:tasks.ceph.mon.a:Stopped 2024-06-29T16:01:56.011 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-29T16:01:56.012 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:56.051 INFO:tasks.ceph.mon.b:Stopped 2024-06-29T16:01:56.051 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-29T16:01:56.052 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:56.107 INFO:tasks.ceph.mon.c:Stopped 2024-06-29T16:01:56.107 INFO:tasks.ceph:Checking cluster log for badness... 2024-06-29T16:01:56.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-06-29T16:01:56.145 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T16:01:56.145 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-06-29T16:01:56.361 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T16:01:56.361 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-06-29T16:01:56.500 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T16:01:56.500 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-06-29T16:01:56.570 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T16:01:56.571 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-06-29T16:01:56.627 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T16:01:56.627 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-06-29T16:01:56.803 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T16:01:56.803 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-06-29T16:01:56.891 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T16:01:56.891 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-06-29T16:01:56.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T16:01:56.947 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-06-29T16:01:57.022 INFO:tasks.ceph:Archiving mon data... 2024-06-29T16:01:57.024 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/data/mon.a.tgz 2024-06-29T16:01:57.024 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-06-29T16:01:57.041 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.njt5VAZr5n 2024-06-29T16:01:57.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.njt5VAZr5n 2024-06-29T16:01:57.808 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.njt5VAZr5n 2024-06-29T16:01:57.893 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.njt5VAZr5n is 4MB 2024-06-29T16:01:58.070 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.njt5VAZr5n 2024-06-29T16:01:58.087 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/data/mon.b.tgz 2024-06-29T16:01:58.088 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-06-29T16:01:58.109 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.ZSKKiVYmTx 2024-06-29T16:01:58.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ZSKKiVYmTx 2024-06-29T16:01:58.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.ZSKKiVYmTx 2024-06-29T16:01:58.968 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.ZSKKiVYmTx is 4MB 2024-06-29T16:01:59.134 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.ZSKKiVYmTx 2024-06-29T16:01:59.149 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/data/mon.c.tgz 2024-06-29T16:01:59.150 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-06-29T16:01:59.204 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.5cCDjfuFgm 2024-06-29T16:01:59.204 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.5cCDjfuFgm 2024-06-29T16:01:59.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.5cCDjfuFgm 2024-06-29T16:02:00.010 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.5cCDjfuFgm is 4MB 2024-06-29T16:02:00.149 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.5cCDjfuFgm 2024-06-29T16:02:00.165 INFO:tasks.ceph:Cleaning ceph cluster... 2024-06-29T16:02:00.165 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-29T16:02:00.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-29T16:02:00.256 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-06-29T16:02:00.285 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-06-29T16:02:00.285 INFO:tasks.ceph:Archiving crash dumps... 2024-06-29T16:02:00.287 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/remote/smithi059/crash 2024-06-29T16:02:00.288 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-29T16:02:00.326 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/remote/smithi153/crash 2024-06-29T16:02:00.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-29T16:02:00.351 INFO:tasks.ceph:Shutting down logrotate 2024-06-29T16:02:00.351 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:02:00.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:02:00.449 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-06-29T16:02:00.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-06-29T16:02:00.504 INFO:tasks.ceph:Compressing logs... 2024-06-29T16:02:00.504 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-29T16:02:00.523 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-29T16:02:00.549 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53017.log 2024-06-29T16:02:00.551 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66677.log 2024-06-29T16:02:00.551 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45531.log 2024-06-29T16:02:00.551 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53017.log.gz 2024-06-29T16:02:00.551 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102103.log 2024-06-29T16:02:00.551 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66677.log.gz 2024-06-29T16:02:00.552 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110557.log 2024-06-29T16:02:00.552 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45531.log.gz 2024-06-29T16:02:00.552 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78718.log 2024-06-29T16:02:00.552 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102103.log.gz 2024-06-29T16:02:00.577 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43696.log 2024-06-29T16:02:00.577 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-29T16:02:00.577 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.1.43696.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-29T16:02:00.577 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43696.log.gz 2024-06-29T16:02:00.578 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37948.log 2024-06-29T16:02:00.578 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-06-29T16:02:00.578 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37948.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-06-29T16:02:00.578 INFO:teuthology.orchestra.run.smithi153.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37948.log.gz 2024-06-29T16:02:00.578 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-06-29T16:02:00.578 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37828.log 2024-06-29T16:02:00.578 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-06-29T16:02:00.579 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38068.log 2024-06-29T16:02:00.579 INFO:teuthology.orchestra.run.smithi153.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37828.log.gz 2024-06-29T16:02:00.579 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37988.log 2024-06-29T16:02:00.579 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38068.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38068.log.gz 2024-06-29T16:02:00.579 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37988.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-06-29T16:02:00.579 INFO:teuthology.orchestra.run.smithi153.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37988.log.gz 2024-06-29T16:02:00.579 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-06-29T16:02:00.579 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37908.log 2024-06-29T16:02:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37868.log 2024-06-29T16:02:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37908.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37908.log.gz 2024-06-29T16:02:00.592 INFO:teuthology.orchestra.run.smithi153.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-29T16:02:00.592 INFO:teuthology.orchestra.run.smithi153.stderr: 94.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-29T16:02:00.592 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-06-29T16:02:00.592 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37868.log: gzip -5 --verbose 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37868.log.gz 2024-06-29T16:02:00.592 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.38028.log 2024-06-29T16:02:00.592 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44100.log 2024-06-29T16:02:00.592 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.38028.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38028.log.gz 2024-06-29T16:02:00.608 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44096.log 2024-06-29T16:02:00.608 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.3.44100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.44100.log.gz 2024-06-29T16:02:00.617 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38108.log 2024-06-29T16:02:00.617 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.3.44096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.44096.log.gz 2024-06-29T16:02:00.617 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43701.log 2024-06-29T16:02:00.617 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38108.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38108.log.gz 2024-06-29T16:02:00.617 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-06-29T16:02:00.617 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.1.43701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43701.log.gz 2024-06-29T16:02:00.618 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-06-29T16:02:00.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: 92.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-06-29T16:02:00.819 INFO:teuthology.orchestra.run.smithi153.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-06-29T16:02:00.822 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98195.log 2024-06-29T16:02:00.822 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.110557.log.gz 2024-06-29T16:02:00.822 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110389.log 2024-06-29T16:02:00.823 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78718.log.gz 2024-06-29T16:02:00.823 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61658.log 2024-06-29T16:02:00.823 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98195.log.gz 2024-06-29T16:02:00.823 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81053.log 2024-06-29T16:02:00.823 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.110389.log.gz 2024-06-29T16:02:00.823 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95858.log 2024-06-29T16:02:00.823 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61658.log.gz 2024-06-29T16:02:00.823 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66155.log 2024-06-29T16:02:00.824 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81053.log.gz 2024-06-29T16:02:00.824 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37695.log 2024-06-29T16:02:00.824 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95858.log.gz 2024-06-29T16:02:00.824 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95438.log 2024-06-29T16:02:00.824 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.66155.log.gz 2024-06-29T16:02:00.824 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75505.log 2024-06-29T16:02:00.824 INFO:teuthology.orchestra.run.smithi059.stderr: 79.6% -- replaced with /var/log/ceph/ceph-client.admin.37695.log.gz 2024-06-29T16:02:00.824 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73910.log 2024-06-29T16:02:00.825 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102668.log 2024-06-29T16:02:00.825 INFO:teuthology.orchestra.run.smithi059.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.95438.log.gz 2024-06-29T16:02:00.825 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75505.log.gz 2024-06-29T16:02:00.825 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53147.log 2024-06-29T16:02:00.825 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73910.log.gz 2024-06-29T16:02:00.825 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58902.log 2024-06-29T16:02:00.825 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53147.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102668.log.gz 2024-06-29T16:02:00.825 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54815.log 2024-06-29T16:02:00.826 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53147.log.gz 2024-06-29T16:02:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46751.log 2024-06-29T16:02:00.826 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.58902.log.gz 2024-06-29T16:02:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54857.log 2024-06-29T16:02:00.826 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54815.log.gz 2024-06-29T16:02:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77477.log 2024-06-29T16:02:00.826 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46751.log.gz 2024-06-29T16:02:00.826 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52572.log 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54857.log.gz 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82774.log 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77477.log.gz 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54175.log 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52572.log.gz 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87532.log 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82774.log.gz 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47092.log 2024-06-29T16:02:00.827 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.54175.log.gz 2024-06-29T16:02:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69096.log 2024-06-29T16:02:00.828 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87532.log.gz 2024-06-29T16:02:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82227.log 2024-06-29T16:02:00.828 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47092.log.gz 2024-06-29T16:02:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.69096.log: /var/log/ceph/ceph-client.admin.96036.log 2024-06-29T16:02:00.828 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69096.log.gz 2024-06-29T16:02:00.828 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45045.log 2024-06-29T16:02:00.828 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.82227.log.gz 2024-06-29T16:02:00.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.96036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86661.log 2024-06-29T16:02:00.829 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.96036.log.gz 2024-06-29T16:02:00.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103256.log 2024-06-29T16:02:00.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70194.log 2024-06-29T16:02:00.829 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86661.log.gz 2024-06-29T16:02:00.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107500.log 2024-06-29T16:02:00.829 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.103256.log.gz 2024-06-29T16:02:00.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109906.log 2024-06-29T16:02:00.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107500.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.70194.log.gz 2024-06-29T16:02:00.830 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44457.log 2024-06-29T16:02:00.830 INFO:teuthology.orchestra.run.smithi059.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.45045.log.gz 2024-06-29T16:02:00.830 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107500.log.gz 2024-06-29T16:02:00.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111835.log 2024-06-29T16:02:00.830 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109906.log.gz 2024-06-29T16:02:00.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87906.log 2024-06-29T16:02:00.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111835.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44457.log.gz 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73658.log 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111835.log.gz 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57850.log 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.87906.log.gz 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68928.log 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73658.log.gz 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45903.log 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57850.log.gz 2024-06-29T16:02:00.831 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59894.log 2024-06-29T16:02:00.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68928.log: /var/log/ceph/ceph-client.admin.45903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93838.log 2024-06-29T16:02:00.832 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68928.log.gz 2024-06-29T16:02:00.832 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45903.log.gz 2024-06-29T16:02:00.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62137.log 2024-06-29T16:02:00.832 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59894.log.gz 2024-06-29T16:02:00.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95983.log 2024-06-29T16:02:00.832 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.93838.log.gz 2024-06-29T16:02:00.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61928.log 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62137.log.gz 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48642.log 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95983.log.gz 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101925.log 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61928.log.gz 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82311.log 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48642.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48642.log.gz 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52530.log 2024-06-29T16:02:00.833 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101925.log.gz 2024-06-29T16:02:00.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59936.log 2024-06-29T16:02:00.834 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82311.log.gz 2024-06-29T16:02:00.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52300.log 2024-06-29T16:02:00.834 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52530.log.gz 2024-06-29T16:02:00.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46416.log 2024-06-29T16:02:00.834 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59936.log.gz 2024-06-29T16:02:00.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82013.log 2024-06-29T16:02:00.834 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52300.log.gz 2024-06-29T16:02:00.835 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105654.log 2024-06-29T16:02:00.835 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46416.log.gz 2024-06-29T16:02:00.835 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.82013.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.61508.log 2024-06-29T16:02:00.835 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82013.log.gz 2024-06-29T16:02:00.835 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95168.log 2024-06-29T16:02:00.835 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109532.log 2024-06-29T16:02:00.835 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.105654.log.gz 2024-06-29T16:02:00.835 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61508.log.gz 2024-06-29T16:02:00.836 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.95168.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.76469.log 2024-06-29T16:02:00.836 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95168.log.gz 2024-06-29T16:02:00.836 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78327.log 2024-06-29T16:02:00.836 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76469.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109532.log.gz 2024-06-29T16:02:00.836 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113595.log 2024-06-29T16:02:00.836 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.76469.log.gz 2024-06-29T16:02:00.836 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46747.log 2024-06-29T16:02:00.836 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78327.log.gz 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102271.log 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.113595.log.gz 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50133.log 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.46747.log.gz 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49755.log 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102271.log.gz 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99188.log 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50133.log.gz 2024-06-29T16:02:00.837 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110744.log 2024-06-29T16:02:00.838 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49755.log.gz 2024-06-29T16:02:00.838 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49461.log 2024-06-29T16:02:00.838 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99188.log.gz 2024-06-29T16:02:00.838 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102878.log 2024-06-29T16:02:00.838 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110744.log.gz 2024-06-29T16:02:00.838 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66635.log 2024-06-29T16:02:00.838 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49461.log.gz 2024-06-29T16:02:00.838 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109323.log 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102878.log.gz 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51300.log 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66635.log.gz 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93964.log 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109323.log.gz 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54293.log 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51300.log.gz 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64174.log 2024-06-29T16:02:00.839 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93964.log.gz 2024-06-29T16:02:00.840 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47937.log 2024-06-29T16:02:00.840 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54293.log.gz 2024-06-29T16:02:00.840 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70236.log 2024-06-29T16:02:00.840 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64174.log.gz 2024-06-29T16:02:00.840 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61340.log 2024-06-29T16:02:00.840 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47937.log.gz 2024-06-29T16:02:00.840 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59768.log 2024-06-29T16:02:00.840 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70236.log.gz 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71872.log 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61340.log.gz 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72658.log 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.59768.log.gz 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94319.log 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71872.log.gz 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64783.log 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.72658.log.gz 2024-06-29T16:02:00.841 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77332.log 2024-06-29T16:02:00.842 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94319.log.gz 2024-06-29T16:02:00.842 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47415.log 2024-06-29T16:02:00.842 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64783.log.gz 2024-06-29T16:02:00.842 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114435.log 2024-06-29T16:02:00.842 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77332.log.gz 2024-06-29T16:02:00.842 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91188.log 2024-06-29T16:02:00.842 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47415.log.gz 2024-06-29T16:02:00.842 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50550.log 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.114435.log.gz 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46155.log 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91188.log.gz 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73555.log 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50550.log.gz 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103130.log 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46155.log.gz 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64447.log 2024-06-29T16:02:00.843 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73555.log.gz 2024-06-29T16:02:00.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52783.log 2024-06-29T16:02:00.844 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.103130.log.gz 2024-06-29T16:02:00.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107897.log 2024-06-29T16:02:00.844 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64447.log.gz 2024-06-29T16:02:00.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86787.log 2024-06-29T16:02:00.844 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52783.log.gz 2024-06-29T16:02:00.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94048.log 2024-06-29T16:02:00.844 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.107897.log.gz 2024-06-29T16:02:00.845 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69974.log 2024-06-29T16:02:00.845 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.86787.log.gz 2024-06-29T16:02:00.845 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93586.log 2024-06-29T16:02:00.845 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.94048.log.gz 2024-06-29T16:02:00.845 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65012.log 2024-06-29T16:02:00.845 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69974.log.gz 2024-06-29T16:02:00.845 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78886.log 2024-06-29T16:02:00.845 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93586.log.gz 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52488.log 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65012.log.gz 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80299.log 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78886.log.gz 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86913.log 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52488.log.gz 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89575.log 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.80299.log.gz 2024-06-29T16:02:00.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81522.log 2024-06-29T16:02:00.847 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86913.log.gz 2024-06-29T16:02:00.847 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93880.log 2024-06-29T16:02:00.847 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.89575.log.gz 2024-06-29T16:02:00.847 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103214.log 2024-06-29T16:02:00.847 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81522.log.gz 2024-06-29T16:02:00.847 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104880.log 2024-06-29T16:02:00.847 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93880.log.gz 2024-06-29T16:02:00.847 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49335.log 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.103214.log.gz 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.104880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100801.log 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.104880.log.gz 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105359.log 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49335.log.gz 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112003.log 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.100801.log.gz 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114267.log 2024-06-29T16:02:00.848 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105359.log.gz 2024-06-29T16:02:00.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76765.log 2024-06-29T16:02:00.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101746.log 2024-06-29T16:02:00.849 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.112003.log.gz 2024-06-29T16:02:00.849 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.114267.log.gz 2024-06-29T16:02:00.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-06-29T16:02:00.849 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76765.log.gz 2024-06-29T16:02:00.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46742.log 2024-06-29T16:02:00.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38170.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101746.log.gz 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92745.log 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72211.log 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46742.log.gz 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50301.log 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92745.log: /var/log/ceph/ceph-client.0.72211.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.92745.log.gz 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.72211.log.gz 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59810.log 2024-06-29T16:02:00.850 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94445.log 2024-06-29T16:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50301.log.gz 2024-06-29T16:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45657.log 2024-06-29T16:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.59810.log.gz 2024-06-29T16:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105444.log 2024-06-29T16:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.94445.log.gz 2024-06-29T16:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79730.log 2024-06-29T16:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45657.log.gz 2024-06-29T16:02:00.851 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38161.log 2024-06-29T16:02:00.852 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105444.log.gz 2024-06-29T16:02:00.852 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49671.log 2024-06-29T16:02:00.852 INFO:teuthology.orchestra.run.smithi059.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.79730.log.gz 2024-06-29T16:02:00.852 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97722.log 2024-06-29T16:02:00.852 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38161.log.gz 2024-06-29T16:02:00.852 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112423.log 2024-06-29T16:02:00.852 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49671.log.gz 2024-06-29T16:02:00.852 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89344.log 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97722.log.gz 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56316.log 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112423.log.gz 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104417.log 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.89344.log.gz 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90567.log 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56316.log.gz 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.104417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67211.log 2024-06-29T16:02:00.853 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.104417.log.gz 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52404.log 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.90567.log.gz 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108321.log 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67211.log.gz 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73513.log 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52404.log.gz 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73093.log 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108321.log.gz 2024-06-29T16:02:00.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67683.log 2024-06-29T16:02:00.855 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73513.log.gz 2024-06-29T16:02:00.855 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52933.log 2024-06-29T16:02:00.855 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73093.log.gz 2024-06-29T16:02:00.855 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112952.log 2024-06-29T16:02:00.855 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67683.log.gz 2024-06-29T16:02:00.855 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107939.log 2024-06-29T16:02:00.855 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52933.log.gz 2024-06-29T16:02:00.855 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105271.log 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.112952.log.gz 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82143.log 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107939.log.gz 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61424.log 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105271.log.gz 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81772.log 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82143.log.gz 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47793.log/var/log/ceph/ceph-client.admin.61424.log: 2024-06-29T16:02:00.856 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61424.log.gz 2024-06-29T16:02:00.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112069.log 2024-06-29T16:02:00.857 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81772.log.gz 2024-06-29T16:02:00.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113249.log 2024-06-29T16:02:00.857 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47793.log.gz 2024-06-29T16:02:00.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100633.log 2024-06-29T16:02:00.857 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112069.log.gz 2024-06-29T16:02:00.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68103.log 2024-06-29T16:02:00.857 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113249.log.gz 2024-06-29T16:02:00.858 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107729.log 2024-06-29T16:02:00.858 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.100633.log.gz 2024-06-29T16:02:00.858 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74377.log 2024-06-29T16:02:00.858 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68103.log.gz 2024-06-29T16:02:00.858 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51990.log 2024-06-29T16:02:00.858 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107729.log.gz 2024-06-29T16:02:00.858 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55591.log 2024-06-29T16:02:00.858 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74377.log.gz 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114813.log 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51990.log.gz 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101315.log 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55591.log.gz 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71375.log 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.114813.log.gz 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98931.log 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.101315.log.gz 2024-06-29T16:02:00.859 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115137.log 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71375.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.71375.log.gz 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98111.log 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98931.log.gz 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.115137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98549.log 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.115137.log.gz 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52032.log 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98111.log.gz 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98549.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-29T16:02:00.860 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98549.log.gz 2024-06-29T16:02:00.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60081.log 2024-06-29T16:02:00.861 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52032.log.gz 2024-06-29T16:02:00.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54411.log 2024-06-29T16:02:00.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45221.log 2024-06-29T16:02:00.861 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60081.log.gz 2024-06-29T16:02:00.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54983.log 2024-06-29T16:02:00.861 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54411.log.gz 2024-06-29T16:02:00.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49713.log 2024-06-29T16:02:00.862 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6%/var/log/ceph/ceph-client.admin.54983.log: -- replaced with /var/log/ceph/ceph-client.admin.45221.log.gz 2024-06-29T16:02:00.862 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78378.log 2024-06-29T16:02:00.862 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54983.log.gz 2024-06-29T16:02:00.862 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60949.log 2024-06-29T16:02:00.862 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49713.log.gz 2024-06-29T16:02:00.862 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53063.log 2024-06-29T16:02:00.862 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78378.log.gz 2024-06-29T16:02:00.862 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106618.log 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60949.log.gz 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111295.log 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53063.log.gz 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76681.log 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.106618.log.gz 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105929.log 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111295.log.gz 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97329.log 2024-06-29T16:02:00.863 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76681.log.gz 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69879.log 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97329.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.105929.log.gz 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74750.log 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97329.log.gz 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52362.log 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69879.log.gz 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110660.log 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74750.log.gz 2024-06-29T16:02:00.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87395.log 2024-06-29T16:02:00.865 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52362.log.gz 2024-06-29T16:02:00.865 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107248.log 2024-06-29T16:02:00.865 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110660.log.gz 2024-06-29T16:02:00.865 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93492.log 2024-06-29T16:02:00.865 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107248.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87395.log.gz 2024-06-29T16:02:00.865 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60650.log 2024-06-29T16:02:00.865 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107248.log.gz 2024-06-29T16:02:00.865 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109365.log 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93492.log.gz 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103526.log 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109365.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60650.log.gz 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99621.log 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109365.log.gz 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70278.log 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.103526.log.gz 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101883.log 2024-06-29T16:02:00.866 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99621.log.gz 2024-06-29T16:02:00.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80824.log 2024-06-29T16:02:00.867 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.70278.log.gz 2024-06-29T16:02:00.867 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-29T16:02:00.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101883.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-06-29T16:02:00.867 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101883.log.gz 2024-06-29T16:02:00.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52614.log 2024-06-29T16:02:00.867 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80824.log.gz 2024-06-29T16:02:00.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89041.log 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103568.log 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52614.log.gz 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89701.log 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.89041.log.gz 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49125.log 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103568.log.gz 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88317.log 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.89701.log.gz 2024-06-29T16:02:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88191.log 2024-06-29T16:02:00.869 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49125.log.gz 2024-06-29T16:02:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87612.log 2024-06-29T16:02:00.869 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.88317.log.gz 2024-06-29T16:02:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49377.log 2024-06-29T16:02:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87612.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88191.log.gz 2024-06-29T16:02:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111621.log 2024-06-29T16:02:00.869 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87612.log.gz 2024-06-29T16:02:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99537.log 2024-06-29T16:02:00.870 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49377.log.gz 2024-06-29T16:02:00.870 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86745.log 2024-06-29T16:02:00.870 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99537.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111621.log.gz 2024-06-29T16:02:00.870 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102836.log 2024-06-29T16:02:00.870 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.99537.log.gz 2024-06-29T16:02:00.870 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112754.log 2024-06-29T16:02:00.870 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.86745.log.gz 2024-06-29T16:02:00.870 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.102836.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.80866.log 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112754.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102836.log.gz 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54941.log 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.112754.log.gz 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89911.log 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80866.log.gz 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54899.log 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54941.log: /var/log/ceph/ceph-client.admin.89911.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54941.log.gz 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97900.log 2024-06-29T16:02:00.871 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89911.log.gz 2024-06-29T16:02:00.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90266.log 2024-06-29T16:02:00.872 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54899.log.gz 2024-06-29T16:02:00.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74960.log 2024-06-29T16:02:00.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90266.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97900.log.gz 2024-06-29T16:02:00.872 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89302.log 2024-06-29T16:02:00.872 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90266.log.gz 2024-06-29T16:02:00.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49209.log 2024-06-29T16:02:00.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68658.log 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.74960.log.gz 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89302.log.gz 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103004.log 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49209.log.gz 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67809.log 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68658.log.gz 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71045.log 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67809.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103004.log.gz 2024-06-29T16:02:00.873 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113291.log 2024-06-29T16:02:00.874 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.67809.log.gz 2024-06-29T16:02:00.874 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103172.log 2024-06-29T16:02:00.874 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71045.log.gz 2024-06-29T16:02:00.874 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106398.log 2024-06-29T16:02:00.874 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103172.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113291.log.gz 2024-06-29T16:02:00.874 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89743.log 2024-06-29T16:02:00.874 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.103172.log.gz 2024-06-29T16:02:00.874 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38174.log 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106398.log.gz 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90308.log 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38174.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.89743.log.gz 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45347.log 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38174.log.gz 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46755.log 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90308.log.gz 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52698.log 2024-06-29T16:02:00.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46755.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45347.log.gz 2024-06-29T16:02:00.876 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55549.log 2024-06-29T16:02:00.876 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46755.log.gz 2024-06-29T16:02:00.876 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68528.log 2024-06-29T16:02:00.876 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52698.log.gz 2024-06-29T16:02:00.876 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87142.log 2024-06-29T16:02:00.876 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68528.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55549.log.gz 2024-06-29T16:02:00.876 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51096.log 2024-06-29T16:02:00.876 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68528.log.gz 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108364.log 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87142.log.gz 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51096.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51096.log.gz 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94870.log 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108364.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31610.log 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108364.log.gz 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55843.log 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94870.log.gz 2024-06-29T16:02:00.877 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104291.log 2024-06-29T16:02:00.878 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.tmp-client.admin.31610.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31610.log.gz 2024-06-29T16:02:00.878 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70782.log 2024-06-29T16:02:00.878 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.55843.log.gz 2024-06-29T16:02:00.878 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.104291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50463.log 2024-06-29T16:02:00.878 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104291.log.gz 2024-06-29T16:02:00.878 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43502.log/var/log/ceph/ceph-client.admin.70782.log: 2024-06-29T16:02:00.878 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50463.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70782.log.gz 2024-06-29T16:02:00.878 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89995.log 2024-06-29T16:02:00.879 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50463.log.gz 2024-06-29T16:02:00.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79074.log 2024-06-29T16:02:00.879 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43502.log.gz 2024-06-29T16:02:00.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99100.log 2024-06-29T16:02:00.879 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89995.log.gz 2024-06-29T16:02:00.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.79074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73387.log 2024-06-29T16:02:00.879 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79074.log.gz 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101967.log 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99100.log.gz 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67389.log 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73387.log.gz 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61592.log 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101967.log.gz 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87437.log 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67389.log.gz 2024-06-29T16:02:00.880 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101526.log 2024-06-29T16:02:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67935.log 2024-06-29T16:02:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101526.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87437.log.gz 2024-06-29T16:02:00.881 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61592.log.gz 2024-06-29T16:02:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106272.log 2024-06-29T16:02:00.881 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.101526.log.gz 2024-06-29T16:02:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94958.log 2024-06-29T16:02:00.881 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67935.log.gz 2024-06-29T16:02:00.881 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47895.log 2024-06-29T16:02:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106272.log: /var/log/ceph/ceph-client.admin.94958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108735.log 2024-06-29T16:02:00.882 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106272.log.gz 2024-06-29T16:02:00.882 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.94958.log.gz 2024-06-29T16:02:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105486.log 2024-06-29T16:02:00.882 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47895.log.gz 2024-06-29T16:02:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64353.log 2024-06-29T16:02:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108735.log: /var/log/ceph/ceph-client.admin.105486.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108735.log.gz 2024-06-29T16:02:00.882 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95816.log 2024-06-29T16:02:00.883 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105486.log.gz 2024-06-29T16:02:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108406.log 2024-06-29T16:02:00.883 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64353.log.gz 2024-06-29T16:02:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90056.log 2024-06-29T16:02:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108406.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95816.log.gz 2024-06-29T16:02:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44415.log 2024-06-29T16:02:00.883 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.108406.log.gz 2024-06-29T16:02:00.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75214.log 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90056.log.gz 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99144.log 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.75214.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44415.log.gz 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.75214.log.gz 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77603.log 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99144.log.gz 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102187.log 2024-06-29T16:02:00.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70152.log 2024-06-29T16:02:00.885 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77603.log.gz 2024-06-29T16:02:00.885 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69396.log 2024-06-29T16:02:00.885 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.102187.log.gz 2024-06-29T16:02:00.885 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79306.log 2024-06-29T16:02:00.885 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70152.log: /var/log/ceph/ceph-client.0.69396.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70152.log.gz 2024-06-29T16:02:00.885 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69396.log.gz 2024-06-29T16:02:00.885 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106702.log 2024-06-29T16:02:00.885 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102145.log 2024-06-29T16:02:00.886 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79306.log: /var/log/ceph/ceph-client.admin.106702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95336.log 2024-06-29T16:02:00.886 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79306.log.gz 2024-06-29T16:02:00.886 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106702.log.gz 2024-06-29T16:02:00.886 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50754.log 2024-06-29T16:02:00.886 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.102145.log.gz 2024-06-29T16:02:00.886 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51738.log 2024-06-29T16:02:00.886 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.95336.log.gz 2024-06-29T16:02:00.886 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69624.log 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51012.log 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51738.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50754.log.gz 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69624.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51738.log.gz 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65775.log 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69624.log.gz 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89659.log 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51012.log: /var/log/ceph/ceph-client.admin.65775.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51012.log.gz 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64657.log 2024-06-29T16:02:00.887 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65775.log.gz 2024-06-29T16:02:00.888 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107416.log 2024-06-29T16:02:00.888 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89659.log: /var/log/ceph/ceph-client.admin.64657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64909.log 2024-06-29T16:02:00.888 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89659.log.gz 2024-06-29T16:02:00.888 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.64657.log.gz 2024-06-29T16:02:00.888 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70824.log 2024-06-29T16:02:00.888 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107416.log.gz 2024-06-29T16:02:00.888 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38175.log 2024-06-29T16:02:00.888 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64909.log.gz 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78820.log 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70824.log: /var/log/ceph/ceph-client.admin.38175.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70824.log.gz 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78634.log 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr: 90.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.38175.log.gz 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99323.log 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111793.log 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78634.log.gz 2024-06-29T16:02:00.889 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.78820.log.gz 2024-06-29T16:02:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44607.log 2024-06-29T16:02:00.890 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99323.log.gz 2024-06-29T16:02:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82608.log 2024-06-29T16:02:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111793.log: /var/log/ceph/ceph-client.admin.44607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101484.log 2024-06-29T16:02:00.890 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111793.log.gz 2024-06-29T16:02:00.890 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44607.log.gz 2024-06-29T16:02:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.82608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82608.log.gz 2024-06-29T16:02:00.890 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91314.log 2024-06-29T16:02:00.891 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95084.log 2024-06-29T16:02:00.891 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101484.log: /var/log/ceph/ceph-client.admin.91314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77248.log 2024-06-29T16:02:00.891 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.101484.log.gz 2024-06-29T16:02:00.891 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91314.log.gz 2024-06-29T16:02:00.891 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91398.log 2024-06-29T16:02:00.891 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.95084.log.gz 2024-06-29T16:02:00.891 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88275.log 2024-06-29T16:02:00.891 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77248.log.gz 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102710.log 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86871.log 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.91398.log.gz 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88275.log.gz 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102710.log.gz 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94109.log 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86871.log.gz 2024-06-29T16:02:00.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68019.log 2024-06-29T16:02:00.893 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2024-06-29T16:02:00.893 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65054.log 2024-06-29T16:02:00.893 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.94109.log.gz 2024-06-29T16:02:00.893 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61298.log 2024-06-29T16:02:00.893 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68019.log.gz 2024-06-29T16:02:00.893 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66553.log 2024-06-29T16:02:00.893 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65054.log.gz 2024-06-29T16:02:00.893 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50049.log 2024-06-29T16:02:00.894 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.61298.log.gz 2024-06-29T16:02:00.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.66553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78112.log 2024-06-29T16:02:00.894 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.66553.log.gz 2024-06-29T16:02:00.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113908.log 2024-06-29T16:02:00.894 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50049.log.gz 2024-06-29T16:02:00.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64699.log 2024-06-29T16:02:00.894 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78112.log.gz 2024-06-29T16:02:00.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105006.log 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113908.log.gz 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37916.log 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.64699.log.gz 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79894.log 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.105006.log.gz 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47083.log 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.37916.log.gz 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93754.log 2024-06-29T16:02:00.895 INFO:teuthology.orchestra.run.smithi059.stderr: 84.1% -- replaced with /var/log/ceph/ceph-client.admin.79894.log.gz 2024-06-29T16:02:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50091.log 2024-06-29T16:02:00.896 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47083.log.gz 2024-06-29T16:02:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99729.log 2024-06-29T16:02:00.896 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.93754.log.gz 2024-06-29T16:02:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107164.log 2024-06-29T16:02:00.896 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50091.log.gz 2024-06-29T16:02:00.896 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79198.log 2024-06-29T16:02:00.896 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99729.log.gz 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98770.log 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.107164.log.gz 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101400.log 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79198.log.gz 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61382.log 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101400.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98770.log.gz 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113196.log 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.101400.log.gz 2024-06-29T16:02:00.897 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100591.log 2024-06-29T16:02:00.898 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61382.log.gz 2024-06-29T16:02:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91230.log 2024-06-29T16:02:00.898 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113196.log.gz 2024-06-29T16:02:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109473.log 2024-06-29T16:02:00.898 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100591.log.gz 2024-06-29T16:02:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113511.log 2024-06-29T16:02:00.898 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.91230.log.gz 2024-06-29T16:02:00.898 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58376.log 2024-06-29T16:02:00.899 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109473.log: /var/log/ceph/ceph-client.admin.113511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103993.log 2024-06-29T16:02:00.899 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109473.log.gz 2024-06-29T16:02:00.899 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113511.log.gz 2024-06-29T16:02:00.899 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90835.log 2024-06-29T16:02:00.899 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.58376.log.gz 2024-06-29T16:02:00.899 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74665.log 2024-06-29T16:02:00.899 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103993.log.gz 2024-06-29T16:02:00.899 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70908.log 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90835.log.gz 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101161.log 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74665.log.gz 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70950.log 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70908.log.gz 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71290.log 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101161.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101161.log.gz 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113721.log 2024-06-29T16:02:00.900 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70950.log.gz 2024-06-29T16:02:00.901 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53273.log 2024-06-29T16:02:00.901 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71290.log.gz 2024-06-29T16:02:00.901 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70656.log 2024-06-29T16:02:00.901 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113721.log.gz 2024-06-29T16:02:00.901 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98973.log 2024-06-29T16:02:00.901 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53273.log.gz 2024-06-29T16:02:00.901 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107645.log 2024-06-29T16:02:00.901 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98973.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70656.log.gz 2024-06-29T16:02:00.902 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98973.log.gz 2024-06-29T16:02:00.902 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113763.log 2024-06-29T16:02:00.902 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65401.log 2024-06-29T16:02:00.902 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.107645.log.gz 2024-06-29T16:02:00.902 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79612.log 2024-06-29T16:02:00.902 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113763.log.gz 2024-06-29T16:02:00.902 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45945.log 2024-06-29T16:02:00.902 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65401.log.gz 2024-06-29T16:02:00.903 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107374.log 2024-06-29T16:02:00.903 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.79612.log.gz 2024-06-29T16:02:00.903 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53797.log 2024-06-29T16:02:00.903 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45945.log.gz 2024-06-29T16:02:00.903 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114477.log 2024-06-29T16:02:00.903 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107374.log.gz 2024-06-29T16:02:00.903 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.53797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91960.log 2024-06-29T16:02:00.903 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53797.log.gz 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100549.log 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.114477.log.gz 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90350.log 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100549.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91960.log.gz 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47088.log 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100549.log.gz 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52740.log 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90350.log: /var/log/ceph/ceph-client.admin.47088.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90350.log.gz 2024-06-29T16:02:00.904 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71662.log 2024-06-29T16:02:00.905 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.47088.log.gz/var/log/ceph/ceph-client.admin.52740.log: 2024-06-29T16:02:00.905 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104964.log 2024-06-29T16:02:00.905 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52740.log.gz 2024-06-29T16:02:00.905 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55633.log 2024-06-29T16:02:00.905 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71662.log.gz 2024-06-29T16:02:00.905 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.104964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46420.log 2024-06-29T16:02:00.905 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104964.log.gz 2024-06-29T16:02:00.905 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48509.log 2024-06-29T16:02:00.906 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55633.log.gz 2024-06-29T16:02:00.906 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77290.log 2024-06-29T16:02:00.906 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46420.log.gz 2024-06-29T16:02:00.906 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.48509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45489.log 2024-06-29T16:02:00.906 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48509.log.gz 2024-06-29T16:02:00.906 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74708.log 2024-06-29T16:02:00.906 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77290.log.gz 2024-06-29T16:02:00.906 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.45489.log: --verbose -- /var/log/ceph/ceph-client.admin.103442.log 2024-06-29T16:02:00.907 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45489.log.gz 2024-06-29T16:02:00.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71957.log 2024-06-29T16:02:00.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103442.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74708.log.gz 2024-06-29T16:02:00.907 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-06-29T16:02:00.907 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103442.log.gz 2024-06-29T16:02:00.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67557.log 2024-06-29T16:02:00.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98069.log 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.71957.log.gz 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58639.log 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.67557.log.gz 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114981.log 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98069.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.98069.log.gz 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111579.log 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.58639.log.gz 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60333.log 2024-06-29T16:02:00.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114981.log: /var/log/ceph/ceph-client.admin.111579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94828.log 2024-06-29T16:02:00.909 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.114981.log.gz 2024-06-29T16:02:00.909 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111579.log.gz 2024-06-29T16:02:00.909 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66844.log 2024-06-29T16:02:00.909 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60333.log.gz 2024-06-29T16:02:00.909 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66113.log 2024-06-29T16:02:00.909 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94828.log: /var/log/ceph/ceph-client.admin.66844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47457.log 2024-06-29T16:02:00.909 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.94828.log.gz 2024-06-29T16:02:00.909 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66844.log.gz 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104207.log 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66113.log.gz 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53963.log 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47457.log: /var/log/ceph/ceph-client.admin.104207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56062.log 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47457.log.gz 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104207.log.gz 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99239.log 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53963.log.gz 2024-06-29T16:02:00.910 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45573.log 2024-06-29T16:02:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56062.log: /var/log/ceph/ceph-client.admin.99239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54105.log 2024-06-29T16:02:00.911 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56062.log.gz 2024-06-29T16:02:00.911 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99239.log.gz 2024-06-29T16:02:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78283.log 2024-06-29T16:02:00.911 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45573.log.gz 2024-06-29T16:02:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56198.log 2024-06-29T16:02:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54105.log: /var/log/ceph/ceph-client.admin.78283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68700.log 2024-06-29T16:02:00.911 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54105.log.gz 2024-06-29T16:02:00.912 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.78283.log.gz 2024-06-29T16:02:00.912 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37779.log 2024-06-29T16:02:00.912 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56198.log.gz 2024-06-29T16:02:00.912 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74834.log 2024-06-29T16:02:00.912 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68700.log: /var/log/ceph/ceph-client.admin.37779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50922.log 2024-06-29T16:02:00.912 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68700.log.gz 2024-06-29T16:02:00.912 INFO:teuthology.orchestra.run.smithi059.stderr: 76.7% -- replaced with /var/log/ceph/ceph-client.admin.37779.log.gz 2024-06-29T16:02:00.912 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73784.log 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74834.log.gz 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79494.log 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50922.log.gz 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98850.log 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73784.log: /var/log/ceph/ceph-client.admin.79494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90923.log 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73784.log.gz 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79494.log.gz 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65903.log 2024-06-29T16:02:00.913 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78462.log 2024-06-29T16:02:00.914 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98850.log.gz 2024-06-29T16:02:00.914 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90923.log: /var/log/ceph/ceph-client.admin.65903.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.90923.log.gz 2024-06-29T16:02:00.914 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68886.log 2024-06-29T16:02:00.914 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65903.log.gz 2024-06-29T16:02:00.914 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77909.log 2024-06-29T16:02:00.914 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68886.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.78462.log.gz 2024-06-29T16:02:00.914 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.85173.log 2024-06-29T16:02:00.914 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68886.log.gz 2024-06-29T16:02:00.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68443.log 2024-06-29T16:02:00.915 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6%/var/log/ceph/ceph-client.admin.85173.log: -- replaced with /var/log/ceph/ceph-client.admin.77909.log.gz 2024-06-29T16:02:00.915 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52975.log 2024-06-29T16:02:00.915 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.85173.log.gz 2024-06-29T16:02:00.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114561.log 2024-06-29T16:02:00.915 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68443.log.gz 2024-06-29T16:02:00.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86441.log 2024-06-29T16:02:00.915 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52975.log.gz 2024-06-29T16:02:00.916 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61172.log 2024-06-29T16:02:00.916 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.114561.log.gz 2024-06-29T16:02:00.916 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38152.log 2024-06-29T16:02:00.916 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.86441.log.gz 2024-06-29T16:02:00.916 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114309.log 2024-06-29T16:02:00.916 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61172.log: /var/log/ceph/ceph-client.admin.38152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45777.log 2024-06-29T16:02:00.916 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61172.log.gz 2024-06-29T16:02:00.916 INFO:teuthology.orchestra.run.smithi059.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38152.log.gz 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89397.log 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.114309.log.gz 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80257.log 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45777.log: /var/log/ceph/ceph-client.admin.89397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49797.log 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45777.log.gz 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.89397.log.gz/var/log/ceph/ceph-client.admin.80257.log: 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98507.log 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80257.log.gz 2024-06-29T16:02:00.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78676.log 2024-06-29T16:02:00.918 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose 87.7% -- -- replaced with /var/log/ceph/ceph-client.admin.49797.log.gz /var/log/ceph/ceph-client.admin.110305.log 2024-06-29T16:02:00.918 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.918 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98507.log: /var/log/ceph/ceph-client.admin.78676.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98507.log.gz 2024-06-29T16:02:00.918 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100507.log 2024-06-29T16:02:00.918 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.78676.log.gz 2024-06-29T16:02:00.918 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106954.log 2024-06-29T16:02:00.918 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.110305.log.gz 2024-06-29T16:02:00.918 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73700.log 2024-06-29T16:02:00.919 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100507.log.gz 2024-06-29T16:02:00.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99921.log 2024-06-29T16:02:00.919 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106954.log.gz 2024-06-29T16:02:00.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76562.log 2024-06-29T16:02:00.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73700.log: /var/log/ceph/ceph-client.0.99921.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73700.log.gz 2024-06-29T16:02:00.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.0.99921.log.gz 2024-06-29T16:02:00.919 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.50634.log 2024-06-29T16:02:00.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51696.log 2024-06-29T16:02:00.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76562.log: /var/log/ceph/ceph-client.admin.50634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66029.log 2024-06-29T16:02:00.920 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76562.log.gz 2024-06-29T16:02:00.920 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50634.log.gz 2024-06-29T16:02:00.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46113.log 2024-06-29T16:02:00.920 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51696.log.gz 2024-06-29T16:02:00.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81929.log 2024-06-29T16:02:00.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74618.log 2024-06-29T16:02:00.920 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66029.log.gz 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46113.log: /var/log/ceph/ceph-client.admin.81929.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46113.log.gz 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72293.log 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81929.log.gz 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59852.log 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74618.log: /var/log/ceph/ceph-client.admin.72293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90182.log 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74618.log.gz 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72293.log.gz 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110515.log 2024-06-29T16:02:00.921 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59852.log.gz 2024-06-29T16:02:00.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79771.log 2024-06-29T16:02:00.922 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose 87.0% -- -- replaced with /var/log/ceph/ceph-client.admin.90182.log.gz /var/log/ceph/ceph-client.admin.77813.log 2024-06-29T16:02:00.922 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110515.log: /var/log/ceph/ceph-client.admin.79771.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110515.log.gz 2024-06-29T16:02:00.922 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103824.log 2024-06-29T16:02:00.922 INFO:teuthology.orchestra.run.smithi059.stderr: 84.0% -- replaced with /var/log/ceph/ceph-client.admin.79771.log.gz 2024-06-29T16:02:00.922 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.77813.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.101788.log 2024-06-29T16:02:00.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100167.log 2024-06-29T16:02:00.923 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77813.log.gz 2024-06-29T16:02:00.923 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.103824.log.gz 2024-06-29T16:02:00.923 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51612.log 2024-06-29T16:02:00.923 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101788.log.gz 2024-06-29T16:02:00.923 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44119.log 2024-06-29T16:02:00.923 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100167.log: /var/log/ceph/ceph-client.admin.51612.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-06-29T16:02:00.923 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100167.log.gz 2024-06-29T16:02:00.923 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51612.log.gz 2024-06-29T16:02:00.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110431.log 2024-06-29T16:02:00.924 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44119.log.gz 2024-06-29T16:02:00.924 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73303.log 2024-06-29T16:02:00.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.110431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74876.log 2024-06-29T16:02:00.924 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110431.log.gz 2024-06-29T16:02:00.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89533.log 2024-06-29T16:02:00.924 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.73303.log.gz 2024-06-29T16:02:00.924 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81687.log 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74876.log: /var/log/ceph/ceph-client.admin.89533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81603.log 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74876.log.gz 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.89533.log.gz 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68742.log 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81687.log.gz 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52074.log 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81603.log.gz 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77771.log 2024-06-29T16:02:00.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68742.log: /var/log/ceph/ceph-client.admin.52074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72862.log 2024-06-29T16:02:00.926 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68742.log.gz 2024-06-29T16:02:00.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77771.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52074.log.gz 2024-06-29T16:02:00.926 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113385.log 2024-06-29T16:02:00.926 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77771.log.gz 2024-06-29T16:02:00.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62190.log 2024-06-29T16:02:00.926 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72862.log.gz 2024-06-29T16:02:00.926 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98633.log 2024-06-29T16:02:00.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113385.log: /var/log/ceph/ceph-client.admin.62190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69054.log 2024-06-29T16:02:00.927 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113385.log.gz 2024-06-29T16:02:00.927 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62190.log.gz 2024-06-29T16:02:00.927 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43887.log 2024-06-29T16:02:00.927 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98633.log.gz 2024-06-29T16:02:00.927 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73471.log 2024-06-29T16:02:00.927 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69054.log: /var/log/ceph/ceph-client.admin.43887.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.69054.log.gz 2024-06-29T16:02:00.927 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73135.log 2024-06-29T16:02:00.927 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43887.log.gz 2024-06-29T16:02:00.928 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47709.log 2024-06-29T16:02:00.928 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73471.log: /var/log/ceph/ceph-client.admin.73135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95504.log 2024-06-29T16:02:00.928 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73471.log.gz 2024-06-29T16:02:00.928 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73135.log.gz 2024-06-29T16:02:00.928 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106660.log 2024-06-29T16:02:00.928 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44373.log 2024-06-29T16:02:00.928 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47709.log.gz 2024-06-29T16:02:00.928 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95504.log: /var/log/ceph/ceph-client.admin.106660.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71206.log 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.95504.log.gz 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.106660.log.gz 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45305.log 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71206.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.107206.log 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44373.log.gz 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71206.log.gz 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105762.log 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45305.log.gz 2024-06-29T16:02:00.929 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72957.log 2024-06-29T16:02:00.930 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107206.log.gz 2024-06-29T16:02:00.930 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60905.log 2024-06-29T16:02:00.930 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105762.log: /var/log/ceph/ceph-client.admin.72957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71419.log 2024-06-29T16:02:00.930 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105762.log.gz 2024-06-29T16:02:00.930 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72957.log.gz 2024-06-29T16:02:00.930 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71248.log 2024-06-29T16:02:00.930 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109627.log 2024-06-29T16:02:00.930 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60905.log.gz 2024-06-29T16:02:00.931 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71419.log: /var/log/ceph/ceph-client.admin.71248.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71419.log.gz 2024-06-29T16:02:00.931 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101053.log 2024-06-29T16:02:00.931 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71248.log.gz 2024-06-29T16:02:00.931 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72700.log 2024-06-29T16:02:00.931 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109627.log: /var/log/ceph/ceph-client.admin.101053.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-06-29T16:02:00.931 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.109627.log.gz 2024-06-29T16:02:00.931 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72700.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101053.log.gz 2024-06-29T16:02:00.931 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86483.log 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72700.log.gz 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44499.log 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47090.log 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86483.log: /var/log/ceph/ceph-client.admin.44499.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86483.log.gz 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44499.log.gz 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46418.log 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mgr.y.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47090.log.gz 2024-06-29T16:02:00.932 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50007.log 2024-06-29T16:02:00.933 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115177.log 2024-06-29T16:02:00.933 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46418.log.gz 2024-06-29T16:02:00.933 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50007.log: gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50007.log.gz 2024-06-29T16:02:00.933 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.101704.log 2024-06-29T16:02:00.933 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.115177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48504.log 2024-06-29T16:02:00.933 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.115177.log.gz 2024-06-29T16:02:00.933 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101704.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101704.log.gz 2024-06-29T16:02:00.933 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98153.log 2024-06-29T16:02:00.934 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.48504.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.60249.log 2024-06-29T16:02:00.934 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.0.48504.log.gz 2024-06-29T16:02:00.934 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65861.log 2024-06-29T16:02:00.934 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98153.log.gz 2024-06-29T16:02:00.934 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94151.log 2024-06-29T16:02:00.934 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82101.log 2024-06-29T16:02:00.934 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60249.log.gz/var/log/ceph/ceph-client.admin.65861.log: 2024-06-29T16:02:00.934 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94151.log: 87.5%gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.94151.log.gz 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.97585.log 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65861.log.gz/var/log/ceph/ceph-client.admin.82101.log: 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82101.log.gz 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.106219.log 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71830.log 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97585.log.gz 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93194.log 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106219.log.gz 2024-06-29T16:02:00.935 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69138.log 2024-06-29T16:02:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71830.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71830.log.gz 2024-06-29T16:02:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93194.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93194.log.gz 2024-06-29T16:02:00.936 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.106356.log 2024-06-29T16:02:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72616.log 2024-06-29T16:02:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106356.log: 91.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94620.log 2024-06-29T16:02:00.936 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69138.log.gz 2024-06-29T16:02:00.936 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106356.log.gz 2024-06-29T16:02:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72616.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.72616.log.gz 2024-06-29T16:02:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50259.log 2024-06-29T16:02:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54689.log 2024-06-29T16:02:00.937 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.94620.log.gz/var/log/ceph/ceph-client.admin.50259.log: 2024-06-29T16:02:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47751.log 2024-06-29T16:02:00.937 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50259.log.gz 2024-06-29T16:02:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51138.log 2024-06-29T16:02:00.937 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54689.log.gz 2024-06-29T16:02:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102920.log 2024-06-29T16:02:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47751.log: /var/log/ceph/ceph-client.admin.51138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54731.log 2024-06-29T16:02:00.938 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47751.log.gz 2024-06-29T16:02:00.938 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51138.log.gz 2024-06-29T16:02:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82841.log 2024-06-29T16:02:00.938 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.102920.log.gz 2024-06-29T16:02:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54731.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54731.log.gz 2024-06-29T16:02:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88061.log 2024-06-29T16:02:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67515.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82841.log.gz 2024-06-29T16:02:00.939 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.939 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47583.log 2024-06-29T16:02:00.939 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88061.log.gz 2024-06-29T16:02:00.939 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67515.log: gzip -5 86.8% -- replaced with /var/log/ceph/ceph-client.admin.67515.log.gz 2024-06-29T16:02:00.939 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.94540.log 2024-06-29T16:02:00.939 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59285.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47583.log.gz 2024-06-29T16:02:00.939 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.939 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107855.log 2024-06-29T16:02:00.940 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.94540.log.gz 2024-06-29T16:02:00.940 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59285.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.81011.log 2024-06-29T16:02:00.940 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59285.log.gz 2024-06-29T16:02:00.940 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111919.log 2024-06-29T16:02:00.940 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107855.log.gz 2024-06-29T16:02:00.940 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106870.log 2024-06-29T16:02:00.940 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81011.log: /var/log/ceph/ceph-client.admin.111919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38622.log 2024-06-29T16:02:00.940 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106870.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81011.log.gz 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.111919.log.gz 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59726.log 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106870.log.gz 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.38622.log: --verbose -- /var/log/ceph/ceph-client.admin.81095.log 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109864.log 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38622.log.gz 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.59726.log.gz 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59422.log 2024-06-29T16:02:00.941 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81095.log: /var/log/ceph/ceph-client.admin.109864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53189.log 2024-06-29T16:02:00.942 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4%/var/log/ceph/ceph-client.admin.59422.log: -- replaced with /var/log/ceph/ceph-client.admin.81095.log.gz 2024-06-29T16:02:00.942 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109864.log.gz 2024-06-29T16:02:00.942 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91652.log 2024-06-29T16:02:00.942 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59422.log.gz 2024-06-29T16:02:00.942 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53189.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-06-29T16:02:00.942 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.91652.log: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.53189.log.gz 2024-06-29T16:02:00.942 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53105.log 2024-06-29T16:02:00.942 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.91652.log.gz 2024-06-29T16:02:00.943 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92013.log 2024-06-29T16:02:00.943 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.53105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90098.log 2024-06-29T16:02:00.943 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92013.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53105.log.gz 2024-06-29T16:02:00.943 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90690.log 2024-06-29T16:02:00.943 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.92013.log.gz 2024-06-29T16:02:00.943 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114645.log 2024-06-29T16:02:00.943 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.90098.log.gz 2024-06-29T16:02:00.943 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90690.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90690.log.gz 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82185.log 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87311.log 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114645.log: /var/log/ceph/ceph-client.admin.82185.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.114645.log.gz 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.82185.log.gz 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50880.log 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87311.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87311.log.gz 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94006.log 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65945.log 2024-06-29T16:02:00.944 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50880.log: /var/log/ceph/ceph-client.admin.94006.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50880.log.gz 2024-06-29T16:02:00.945 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46408.log 2024-06-29T16:02:00.945 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.94006.log.gz 2024-06-29T16:02:00.945 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65945.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65945.log.gz 2024-06-29T16:02:00.945 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71514.log 2024-06-29T16:02:00.945 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94701.log 2024-06-29T16:02:00.945 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46408.log: /var/log/ceph/ceph-client.admin.71514.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46408.log.gz 2024-06-29T16:02:00.945 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71514.log.gz 2024-06-29T16:02:00.945 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103400.log 2024-06-29T16:02:00.946 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94701.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94701.log.gz 2024-06-29T16:02:00.946 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44161.log 2024-06-29T16:02:00.946 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81263.log 2024-06-29T16:02:00.946 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103400.log: /var/log/ceph/ceph-client.admin.44161.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103400.log.gz 2024-06-29T16:02:00.946 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44161.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95924.log 2024-06-29T16:02:00.946 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.946 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81263.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81263.log.gz 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107542.log 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51552.log 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95924.log: /var/log/ceph/ceph-client.admin.107542.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95924.log.gz 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.107542.log.gz 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46953.log 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51552.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51552.log.gz 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100903.log 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100861.log 2024-06-29T16:02:00.947 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46953.log: /var/log/ceph/ceph-client.admin.100903.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46953.log.gz 2024-06-29T16:02:00.948 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100903.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61466.log 2024-06-29T16:02:00.948 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.948 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100861.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.100861.log.gz 2024-06-29T16:02:00.948 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104543.log 2024-06-29T16:02:00.948 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97680.log 2024-06-29T16:02:00.948 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61466.log: /var/log/ceph/ceph-client.admin.104543.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61466.log.gz 2024-06-29T16:02:00.948 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50217.log 2024-06-29T16:02:00.948 INFO:teuthology.orchestra.run.smithi059.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.104543.log.gz 2024-06-29T16:02:00.949 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97680.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.97680.log.gz 2024-06-29T16:02:00.949 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108237.log 2024-06-29T16:02:00.949 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64531.log 2024-06-29T16:02:00.949 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50217.log: /var/log/ceph/ceph-client.admin.108237.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50217.log.gz 2024-06-29T16:02:00.949 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108237.log.gz 2024-06-29T16:02:00.949 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53879.log 2024-06-29T16:02:00.949 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64531.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64531.log.gz 2024-06-29T16:02:00.949 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56798.log 2024-06-29T16:02:00.950 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112381.log 2024-06-29T16:02:00.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53879.log: /var/log/ceph/ceph-client.admin.56798.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53879.log.gz 2024-06-29T16:02:00.950 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56798.log.gzgzip 2024-06-29T16:02:00.950 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65222.log 2024-06-29T16:02:00.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112381.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112381.log.gz 2024-06-29T16:02:00.950 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56121.log 2024-06-29T16:02:00.950 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109407.log 2024-06-29T16:02:00.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65222.log: /var/log/ceph/ceph-client.admin.56121.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65222.log.gz 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.56121.log.gz 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52656.log 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109407.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109407.log.gz 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78154.log 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105132.log 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52656.log: /var/log/ceph/ceph-client.admin.78154.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52656.log.gz 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78154.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60693.log 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.951 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105132.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.105132.log.gz 2024-06-29T16:02:00.952 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66197.log 2024-06-29T16:02:00.952 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86304.log 2024-06-29T16:02:00.952 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60693.log: /var/log/ceph/ceph-client.admin.66197.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60693.log.gz 2024-06-29T16:02:00.952 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69516.log 2024-06-29T16:02:00.952 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66197.log.gz 2024-06-29T16:02:00.952 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86304.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86304.log.gz 2024-06-29T16:02:00.952 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62078.log 2024-06-29T16:02:00.952 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69516.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69516.log.gz 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90648.log 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97422.log 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62078.log.gz 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99281.log 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97422.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90648.log.gz 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97422.log.gz 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114225.log 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108545.log 2024-06-29T16:02:00.953 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114225.log: /var/log/ceph/ceph-client.admin.99281.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99281.log.gz 2024-06-29T16:02:00.954 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.114225.log.gz 2024-06-29T16:02:00.954 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70404.log 2024-06-29T16:02:00.954 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38421.log 2024-06-29T16:02:00.954 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108545.log: /var/log/ceph/ceph-client.admin.70404.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108545.log.gz 2024-06-29T16:02:00.954 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70404.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60819.log 2024-06-29T16:02:00.954 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.954 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2024-06-29T16:02:00.954 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60819.log: /var/log/ceph/ceph-client.admin.38421.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38421.log.gz 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60819.log.gz 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113112.log 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.55675.log: -- /var/log/ceph/ceph-client.admin.44075.log 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79935.log 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44075.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.113112.log.gz 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44075.log.gz 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60020.log 2024-06-29T16:02:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100465.log 2024-06-29T16:02:00.956 INFO:teuthology.orchestra.run.smithi059.stderr: 83.4% -- replaced with /var/log/ceph/ceph-client.admin.79935.log.gz 2024-06-29T16:02:00.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92846.log 2024-06-29T16:02:00.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100465.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60020.log.gz 2024-06-29T16:02:00.956 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100465.log.gz 2024-06-29T16:02:00.956 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81347.log 2024-06-29T16:02:00.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111528.log 2024-06-29T16:02:00.956 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.92846.log.gz 2024-06-29T16:02:00.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48185.log 2024-06-29T16:02:00.957 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111528.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81347.log.gz 2024-06-29T16:02:00.957 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111528.log.gz 2024-06-29T16:02:00.957 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47087.log 2024-06-29T16:02:00.957 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81137.log 2024-06-29T16:02:00.957 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48185.log.gz 2024-06-29T16:02:00.957 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90224.log 2024-06-29T16:02:00.957 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81137.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47087.log.gz 2024-06-29T16:02:00.957 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81137.log.gz 2024-06-29T16:02:00.958 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50413.log 2024-06-29T16:02:00.958 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37750.log 2024-06-29T16:02:00.958 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90224.log.gz 2024-06-29T16:02:00.958 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51180.log 2024-06-29T16:02:00.958 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37750.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50413.log.gz 2024-06-29T16:02:00.958 INFO:teuthology.orchestra.run.smithi059.stderr: 77.3% -- replaced with /var/log/ceph/ceph-client.admin.37750.log.gz 2024-06-29T16:02:00.958 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80656.log 2024-06-29T16:02:00.958 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77561.log 2024-06-29T16:02:00.959 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51180.log.gz 2024-06-29T16:02:00.959 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79240.log 2024-06-29T16:02:00.959 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77561.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.80656.log.gz 2024-06-29T16:02:00.959 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77561.log.gz 2024-06-29T16:02:00.959 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54234.log 2024-06-29T16:02:00.959 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86997.log 2024-06-29T16:02:00.959 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79240.log.gz 2024-06-29T16:02:00.959 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50796.log 2024-06-29T16:02:00.960 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86997.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54234.log.gz 2024-06-29T16:02:00.960 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86997.log.gz 2024-06-29T16:02:00.960 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88988.log 2024-06-29T16:02:00.960 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103743.log 2024-06-29T16:02:00.960 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50796.log.gz 2024-06-29T16:02:00.960 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77374.log 2024-06-29T16:02:00.960 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103743.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88988.log.gz 2024-06-29T16:02:00.960 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.103743.log.gz 2024-06-29T16:02:00.961 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113950.log 2024-06-29T16:02:00.961 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108693.log 2024-06-29T16:02:00.961 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77374.log.gz 2024-06-29T16:02:00.961 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82353.log 2024-06-29T16:02:00.961 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108693.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113950.log.gz 2024-06-29T16:02:00.961 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108693.log.gz 2024-06-29T16:02:00.961 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64970.log 2024-06-29T16:02:00.961 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114351.log 2024-06-29T16:02:00.962 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64970.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.82353.log.gz 2024-06-29T16:02:00.962 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64970.log.gz 2024-06-29T16:02:00.962 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79156.log 2024-06-29T16:02:00.962 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114034.log 2024-06-29T16:02:00.962 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.114351.log.gz 2024-06-29T16:02:00.962 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98591.log 2024-06-29T16:02:00.962 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114034.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.79156.log.gz 2024-06-29T16:02:00.963 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114034.log.gz 2024-06-29T16:02:00.963 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98321.log 2024-06-29T16:02:00.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113427.log 2024-06-29T16:02:00.963 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98591.log.gz 2024-06-29T16:02:00.963 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78420.log 2024-06-29T16:02:00.963 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4%/var/log/ceph/ceph-client.admin.113427.log: -- replaced with /var/log/ceph/ceph-client.admin.98321.log.gz 2024-06-29T16:02:00.963 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113427.log.gz 2024-06-29T16:02:00.963 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47979.log 2024-06-29T16:02:00.964 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53501.log 2024-06-29T16:02:00.964 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78420.log.gz 2024-06-29T16:02:00.964 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85131.log 2024-06-29T16:02:00.964 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53501.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47979.log.gz 2024-06-29T16:02:00.964 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53501.log.gz 2024-06-29T16:02:00.964 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100717.log 2024-06-29T16:02:00.964 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95000.log 2024-06-29T16:02:00.964 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.85131.log.gz 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87226.log 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100717.log.gz 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95000.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95000.log.gz 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45263.log 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59558.log 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87226.log.gz 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105870.log 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59558.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45263.log.gz 2024-06-29T16:02:00.965 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59558.log.gz 2024-06-29T16:02:00.966 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100675.log 2024-06-29T16:02:00.966 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106177.log 2024-06-29T16:02:00.966 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105870.log.gz 2024-06-29T16:02:00.966 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45431.log 2024-06-29T16:02:00.966 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106177.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.100675.log.gz 2024-06-29T16:02:00.966 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106177.log.gz 2024-06-29T16:02:00.966 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86703.log/var/log/ceph/ceph-client.admin.45431.log: 2024-06-29T16:02:00.966 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45431.log.gz 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71125.log 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103046.log 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71125.log: /var/log/ceph/ceph-client.admin.86703.log: 87.5% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71125.log.gz 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86703.log.gz 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88359.log 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106314.log 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103046.log.gz 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88401.log 2024-06-29T16:02:00.967 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106314.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88359.log.gz 2024-06-29T16:02:00.968 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.106314.log.gz 2024-06-29T16:02:00.968 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74250.log 2024-06-29T16:02:00.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72904.log 2024-06-29T16:02:00.968 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88401.log.gz 2024-06-29T16:02:00.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70016.log 2024-06-29T16:02:00.968 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72904.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74250.log.gz 2024-06-29T16:02:00.968 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72904.log.gz 2024-06-29T16:02:00.968 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81971.log 2024-06-29T16:02:00.969 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88929.log 2024-06-29T16:02:00.969 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70016.log: /var/log/ceph/ceph-client.admin.81971.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70016.log.gz 2024-06-29T16:02:00.969 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81971.log.gz 2024-06-29T16:02:00.969 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91818.log 2024-06-29T16:02:00.969 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76860.log 2024-06-29T16:02:00.969 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88929.log.gz 2024-06-29T16:02:00.969 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68320.log 2024-06-29T16:02:00.970 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76860.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91818.log.gz 2024-06-29T16:02:00.970 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76860.log.gz 2024-06-29T16:02:00.970 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106744.log 2024-06-29T16:02:00.970 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104165.log 2024-06-29T16:02:00.970 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68320.log: /var/log/ceph/ceph-client.admin.106744.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68320.log.gz 2024-06-29T16:02:00.970 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106744.log.gz 2024-06-29T16:02:00.970 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73868.log 2024-06-29T16:02:00.970 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69204.log 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73868.log: /var/log/ceph/ceph-client.admin.104165.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73868.log.gz 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104165.log.gz 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53357.log 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65306.log 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69204.log.gz 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102355.log 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53357.log.gz 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47091.log 2024-06-29T16:02:00.971 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102355.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65306.log.gz 2024-06-29T16:02:00.972 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102355.log.gz 2024-06-29T16:02:00.972 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76996.log 2024-06-29T16:02:00.972 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47091.log: -- /var/log/ceph/ceph-client.admin.109053.log 2024-06-29T16:02:00.972 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.47091.log.gz 2024-06-29T16:02:00.972 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113553.log 2024-06-29T16:02:00.972 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109053.log: 87.2% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109053.log.gz 2024-06-29T16:02:00.972 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76996.log.gz 2024-06-29T16:02:00.972 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86829.log 2024-06-29T16:02:00.973 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70866.log 2024-06-29T16:02:00.973 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.113553.log.gz 2024-06-29T16:02:00.973 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87184.log 2024-06-29T16:02:00.973 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70866.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86829.log.gz 2024-06-29T16:02:00.973 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70866.log.gz 2024-06-29T16:02:00.973 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91356.log 2024-06-29T16:02:00.973 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87353.log 2024-06-29T16:02:00.973 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87184.log: /var/log/ceph/ceph-client.admin.91356.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87184.log.gz 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91356.log.gz 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55717.log 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103088.log 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87353.log: /var/log/ceph/ceph-client.admin.55717.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87353.log.gz 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.55717.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78508.log 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103088.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103088.log.gz 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61850.log 2024-06-29T16:02:00.974 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105528.log 2024-06-29T16:02:00.975 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78508.log: /var/log/ceph/ceph-client.0.61850.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78508.log.gz 2024-06-29T16:02:00.975 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61850.log.gz 2024-06-29T16:02:00.975 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97858.log 2024-06-29T16:02:00.975 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104459.log 2024-06-29T16:02:00.975 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105528.log: /var/log/ceph/ceph-client.admin.97858.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.105528.log.gz 2024-06-29T16:02:00.975 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97858.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68145.log 2024-06-29T16:02:00.975 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:00.975 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66071.log 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68145.log: /var/log/ceph/ceph-client.admin.104459.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104459.log.gz 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68145.log.gz 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114393.log 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59642.log 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66071.log: /var/log/ceph/ceph-client.admin.114393.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66071.log.gz 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.114393.log.gz 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77729.log 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104123.log 2024-06-29T16:02:00.976 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77729.log: /var/log/ceph/ceph-client.admin.59642.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59642.log.gz 2024-06-29T16:02:00.977 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.77729.log.gz 2024-06-29T16:02:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87862.log 2024-06-29T16:02:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74421.log 2024-06-29T16:02:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.104123.log: /var/log/ceph/ceph-client.admin.87862.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104123.log.gz 2024-06-29T16:02:00.977 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87862.log.gz 2024-06-29T16:02:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73826.log 2024-06-29T16:02:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-06-29T16:02:00.977 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73826.log: /var/log/ceph/ceph-client.admin.74421.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74421.log.gz 2024-06-29T16:02:00.978 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73826.log.gz 2024-06-29T16:02:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46071.log 2024-06-29T16:02:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-mon.a.log: -- /var/log/ceph/ceph-client.admin.98237.log 2024-06-29T16:02:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46071.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.46071.log.gz 2024-06-29T16:02:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65096.log 2024-06-29T16:02:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98237.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98237.log.gz 2024-06-29T16:02:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93628.log 2024-06-29T16:02:00.978 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65096.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65096.log.gz 2024-06-29T16:02:01.274 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81878.log 2024-06-29T16:02:01.274 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93628.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.93628.log.gz 2024-06-29T16:02:01.274 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47667.log 2024-06-29T16:02:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81878.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81878.log.gz 2024-06-29T16:02:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99495.log 2024-06-29T16:02:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47667.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47667.log.gz 2024-06-29T16:02:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56434.log 2024-06-29T16:02:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99495.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.99495.log.gz 2024-06-29T16:02:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67977.log 2024-06-29T16:02:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56434.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.56434.log.gz 2024-06-29T16:02:01.275 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80908.log 2024-06-29T16:02:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67977.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67977.log.gz 2024-06-29T16:02:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51468.log 2024-06-29T16:02:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80908.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80908.log.gz 2024-06-29T16:02:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112339.log 2024-06-29T16:02:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51468.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51468.log.gz 2024-06-29T16:02:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111253.log 2024-06-29T16:02:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112339.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112339.log.gz 2024-06-29T16:02:01.276 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102607.log 2024-06-29T16:02:01.277 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111253.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111253.log.gz 2024-06-29T16:02:01.277 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111751.log 2024-06-29T16:02:01.277 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102607.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102607.log.gz 2024-06-29T16:02:01.277 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103866.log 2024-06-29T16:02:01.277 INFO:teuthology.orchestra.run.smithi059.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-06-29T16:02:01.277 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111751.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.111751.log.gz 2024-06-29T16:02:01.277 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75218.log 2024-06-29T16:02:01.277 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.103866.log: /var/log/ceph/ceph-client.admin.88019.log 2024-06-29T16:02:01.278 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.103866.log.gz 2024-06-29T16:02:01.278 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.75218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.75218.log.gz 2024-06-29T16:02:01.278 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77519.log 2024-06-29T16:02:01.278 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74918.log 2024-06-29T16:02:01.278 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88019.log.gz 2024-06-29T16:02:01.278 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38172.log 2024-06-29T16:02:01.278 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74918.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.77519.log.gz 2024-06-29T16:02:01.278 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74918.log.gz 2024-06-29T16:02:01.279 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64741.log 2024-06-29T16:02:01.279 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86619.log 2024-06-29T16:02:01.279 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38172.log.gz 2024-06-29T16:02:01.279 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49881.log 2024-06-29T16:02:01.279 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86619.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64741.log.gz 2024-06-29T16:02:01.279 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86619.log.gz 2024-06-29T16:02:01.279 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70362.log 2024-06-29T16:02:01.279 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55759.log 2024-06-29T16:02:01.280 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49881.log.gz 2024-06-29T16:02:01.280 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51948.log 2024-06-29T16:02:01.280 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55759.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.70362.log.gz 2024-06-29T16:02:01.280 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55759.log.gz 2024-06-29T16:02:01.280 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89141.log 2024-06-29T16:02:01.280 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61256.log 2024-06-29T16:02:01.280 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51948.log.gz 2024-06-29T16:02:01.280 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78070.log 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61256.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89141.log.gz 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.61256.log.gz 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82953.log 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47331.log 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78070.log.gz 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60512.log 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47331.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.82953.log.gz 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47331.log.gz 2024-06-29T16:02:01.281 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78239.log 2024-06-29T16:02:01.282 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71556.log 2024-06-29T16:02:01.282 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60512.log.gz 2024-06-29T16:02:01.282 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54529.log 2024-06-29T16:02:01.282 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78239.log.gz 2024-06-29T16:02:01.282 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46029.log 2024-06-29T16:02:01.282 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54529.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71556.log.gz 2024-06-29T16:02:01.282 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54529.log.gz 2024-06-29T16:02:01.282 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95042.log 2024-06-29T16:02:01.285 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46029.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46029.log.gz 2024-06-29T16:02:01.286 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67641.log 2024-06-29T16:02:01.286 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67851.log 2024-06-29T16:02:01.286 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95042.log: /var/log/ceph/ceph-client.admin.67641.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.95042.log.gz 2024-06-29T16:02:01.286 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.67641.log.gz 2024-06-29T16:02:01.286 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60417.log 2024-06-29T16:02:01.286 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53921.log 2024-06-29T16:02:01.287 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67851.log.gz 2024-06-29T16:02:01.287 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80740.log 2024-06-29T16:02:01.287 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60417.log.gz 2024-06-29T16:02:01.287 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53921.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53921.log.gz 2024-06-29T16:02:01.287 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108777.log 2024-06-29T16:02:01.287 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72485.log 2024-06-29T16:02:01.287 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80740.log: /var/log/ceph/ceph-client.admin.108777.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.80740.log.gz 2024-06-29T16:02:01.287 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108777.log.gz 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102061.log 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51822.log 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72485.log: /var/log/ceph/ceph-client.admin.102061.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72485.log.gz 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102061.log.gz 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103298.log 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79853.log 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51822.log: /var/log/ceph/ceph-client.admin.103298.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51822.log.gz 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103298.log.gz 2024-06-29T16:02:01.288 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88103.log 2024-06-29T16:02:01.289 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49587.log 2024-06-29T16:02:01.289 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79853.log: /var/log/ceph/ceph-client.admin.88103.log: 84.0% -- replaced with /var/log/ceph/ceph-client.admin.79853.log.gz 2024-06-29T16:02:01.289 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.88103.log.gz 2024-06-29T16:02:01.289 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73219.log 2024-06-29T16:02:01.289 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49587.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49587.log.gz 2024-06-29T16:02:01.289 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98027.log 2024-06-29T16:02:01.289 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107290.log 2024-06-29T16:02:01.290 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73219.log: /var/log/ceph/ceph-client.admin.98027.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.73219.log.gz 2024-06-29T16:02:01.290 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.98027.log.gz 2024-06-29T16:02:01.290 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76954.log 2024-06-29T16:02:01.290 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90140.log 2024-06-29T16:02:01.290 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107290.log: /var/log/ceph/ceph-client.admin.76954.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107290.log.gz 2024-06-29T16:02:01.290 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76954.log.gz 2024-06-29T16:02:01.290 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104609.log 2024-06-29T16:02:01.290 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71598.log 2024-06-29T16:02:01.291 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90140.log: /var/log/ceph/ceph-client.admin.104609.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.90140.log.gz 2024-06-29T16:02:01.291 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104609.log.gz 2024-06-29T16:02:01.291 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113154.log 2024-06-29T16:02:01.291 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78760.log 2024-06-29T16:02:01.291 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71598.log: /var/log/ceph/ceph-client.admin.113154.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71598.log.gz 2024-06-29T16:02:01.291 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.113154.log.gz 2024-06-29T16:02:01.291 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110786.log 2024-06-29T16:02:01.291 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78760.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78760.log.gz 2024-06-29T16:02:01.292 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102229.log 2024-06-29T16:02:01.292 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89953.log 2024-06-29T16:02:01.292 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110786.log: /var/log/ceph/ceph-client.admin.102229.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.110786.log.gz 2024-06-29T16:02:01.292 INFO:teuthology.orchestra.run.smithi059.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.102229.log.gz 2024-06-29T16:02:01.292 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100423.log 2024-06-29T16:02:01.292 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89869.log 2024-06-29T16:02:01.292 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89953.log: /var/log/ceph/ceph-client.admin.100423.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.89953.log.gz 2024-06-29T16:02:01.292 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108945.log 2024-06-29T16:02:01.293 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.100423.log.gz 2024-06-29T16:02:01.293 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89869.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.89869.log.gz 2024-06-29T16:02:01.293 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49083.log 2024-06-29T16:02:01.293 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107080.log 2024-06-29T16:02:01.293 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108945.log: /var/log/ceph/ceph-client.admin.49083.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108945.log.gz 2024-06-29T16:02:01.293 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53543.log 2024-06-29T16:02:01.293 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49083.log.gz 2024-06-29T16:02:01.293 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107080.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107080.log.gz 2024-06-29T16:02:01.294 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86577.log 2024-06-29T16:02:01.294 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98279.log 2024-06-29T16:02:01.294 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53543.log: /var/log/ceph/ceph-client.admin.86577.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86577.log.gz 2024-06-29T16:02:01.294 INFO:teuthology.orchestra.run.smithi059.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.53543.log.gz 2024-06-29T16:02:01.294 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49419.log 2024-06-29T16:02:01.294 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69666.log 2024-06-29T16:02:01.294 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98279.log: /var/log/ceph/ceph-client.admin.49419.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.98279.log.gz 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49419.log.gzgzip 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87693.log 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69666.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69666.log.gz 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56535.log 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38092.log 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87693.log: /var/log/ceph/ceph-client.admin.56535.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87693.log.gz 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.56535.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72335.log 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.295 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38092.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38092.log.gz 2024-06-29T16:02:01.296 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46197.log 2024-06-29T16:02:01.296 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99663.log 2024-06-29T16:02:01.296 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72335.log: /var/log/ceph/ceph-client.admin.46197.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72335.log.gz 2024-06-29T16:02:01.296 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46197.log.gz 2024-06-29T16:02:01.296 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94403.log 2024-06-29T16:02:01.296 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99663.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.99663.log.gz 2024-06-29T16:02:01.296 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67767.log 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102752.log 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94403.log: /var/log/ceph/ceph-client.admin.67767.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94403.log.gz 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67767.log.gz 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111172.log 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102752.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102752.log.gz 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81816.log 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105804.log 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111172.log: /var/log/ceph/ceph-client.admin.81816.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111172.log.gz 2024-06-29T16:02:01.297 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81816.log.gz 2024-06-29T16:02:01.298 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106828.log 2024-06-29T16:02:01.298 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105804.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105804.log.gz 2024-06-29T16:02:01.298 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73616.log 2024-06-29T16:02:01.298 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91464.log 2024-06-29T16:02:01.298 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106828.log: /var/log/ceph/ceph-client.admin.73616.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106828.log.gz 2024-06-29T16:02:01.298 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73616.log.gz 2024-06-29T16:02:01.298 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52158.log 2024-06-29T16:02:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91464.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91464.log.gz 2024-06-29T16:02:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101442.log 2024-06-29T16:02:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46419.log 2024-06-29T16:02:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52158.log: /var/log/ceph/ceph-client.admin.101442.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52158.log.gz 2024-06-29T16:02:01.299 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.101442.log.gz 2024-06-29T16:02:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108494.log 2024-06-29T16:02:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46419.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.46419.log.gz 2024-06-29T16:02:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105720.log 2024-06-29T16:02:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52200.log 2024-06-29T16:02:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108494.log: /var/log/ceph/ceph-client.admin.105720.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108494.log.gz 2024-06-29T16:02:01.300 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105720.log.gz 2024-06-29T16:02:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114771.log 2024-06-29T16:02:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103908.log 2024-06-29T16:02:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52200.log: /var/log/ceph/ceph-client.admin.114771.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52200.log.gz 2024-06-29T16:02:01.300 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.114771.log.gz 2024-06-29T16:02:01.301 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.109245.log 2024-06-29T16:02:01.301 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103908.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103908.log.gz 2024-06-29T16:02:01.301 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106786.log 2024-06-29T16:02:01.301 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77080.log 2024-06-29T16:02:01.301 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.109245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.109245.log.gz 2024-06-29T16:02:01.301 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106016.log 2024-06-29T16:02:01.301 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106786.log.gz 2024-06-29T16:02:01.301 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77080.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77080.log.gz 2024-06-29T16:02:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44247.log 2024-06-29T16:02:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80950.log 2024-06-29T16:02:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106016.log: /var/log/ceph/ceph-client.admin.44247.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106016.log.gz 2024-06-29T16:02:01.302 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44247.log.gz 2024-06-29T16:02:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59165.log 2024-06-29T16:02:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62232.log 2024-06-29T16:02:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80950.log: /var/log/ceph/ceph-client.admin.59165.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.80950.log.gz 2024-06-29T16:02:01.303 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59165.log.gz 2024-06-29T16:02:01.303 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113805.log 2024-06-29T16:02:01.303 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62232.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62232.log.gz 2024-06-29T16:02:01.303 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104333.log 2024-06-29T16:02:01.303 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108076.log 2024-06-29T16:02:01.303 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113805.log: /var/log/ceph/ceph-client.admin.104333.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113805.log.gz 2024-06-29T16:02:01.303 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104333.log.gz 2024-06-29T16:02:01.303 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93712.log 2024-06-29T16:02:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77206.log 2024-06-29T16:02:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108076.log: /var/log/ceph/ceph-client.admin.93712.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108076.log.gz 2024-06-29T16:02:01.304 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.93712.log.gz 2024-06-29T16:02:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112993.log 2024-06-29T16:02:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77206.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.77206.log.gz 2024-06-29T16:02:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82732.log 2024-06-29T16:02:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94193.log 2024-06-29T16:02:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112993.log: /var/log/ceph/ceph-client.admin.82732.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112993.log.gz 2024-06-29T16:02:01.305 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82732.log.gz 2024-06-29T16:02:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60207.log 2024-06-29T16:02:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106135.log 2024-06-29T16:02:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94193.log: /var/log/ceph/ceph-client.admin.60207.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94193.log.gz 2024-06-29T16:02:01.305 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60207.log.gz 2024-06-29T16:02:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56493.log 2024-06-29T16:02:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99015.log 2024-06-29T16:02:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106135.log: /var/log/ceph/ceph-client.admin.56493.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106135.log.gz 2024-06-29T16:02:01.306 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56493.log.gz 2024-06-29T16:02:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69392.log 2024-06-29T16:02:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99015.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99015.log.gz 2024-06-29T16:02:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67431.log 2024-06-29T16:02:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89617.log 2024-06-29T16:02:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.69392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69392.log.gz 2024-06-29T16:02:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72215.log 2024-06-29T16:02:01.307 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.67431.log.gz 2024-06-29T16:02:01.307 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89617.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.89617.log.gz 2024-06-29T16:02:01.307 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54588.log 2024-06-29T16:02:01.307 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66299.log 2024-06-29T16:02:01.307 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.72215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.72215.log.gz 2024-06-29T16:02:01.307 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75380.log 2024-06-29T16:02:01.307 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.54588.log.gz 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66299.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.66299.log.gz 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64055.log 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67116.log 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75380.log: /var/log/ceph/ceph-client.admin.64055.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75380.log.gz 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64055.log.gz 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114855.log 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67116.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.67116.log.gz 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87100.log 2024-06-29T16:02:01.308 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51384.log 2024-06-29T16:02:01.309 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114855.log: /var/log/ceph/ceph-client.admin.87100.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.114855.log.gz 2024-06-29T16:02:01.309 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87100.log.gz 2024-06-29T16:02:01.309 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67074.log 2024-06-29T16:02:01.309 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51384.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51384.log.gz 2024-06-29T16:02:01.309 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49923.log 2024-06-29T16:02:01.309 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67725.log 2024-06-29T16:02:01.309 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67074.log: /var/log/ceph/ceph-client.admin.49923.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.67074.log.gz 2024-06-29T16:02:01.309 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49923.log.gz 2024-06-29T16:02:01.310 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53459.log 2024-06-29T16:02:01.310 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 2024-06-29T16:02:01.310 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67725.log: /var/log/ceph/ceph-client.admin.53459.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67725.log.gz 2024-06-29T16:02:01.310 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53459.log.gz 2024-06-29T16:02:01.310 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55801.log 2024-06-29T16:02:01.310 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44013.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2024-06-29T16:02:01.310 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65562.log 2024-06-29T16:02:01.310 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75026.log 2024-06-29T16:02:01.311 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55801.log: /var/log/ceph/ceph-client.admin.65562.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55801.log.gz 2024-06-29T16:02:01.311 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65562.log.gz 2024-06-29T16:02:01.311 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68485.log 2024-06-29T16:02:01.311 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75026.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75026.log.gz 2024-06-29T16:02:01.311 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102481.log 2024-06-29T16:02:01.311 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88233.log 2024-06-29T16:02:01.311 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68485.log: /var/log/ceph/ceph-client.admin.102481.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68485.log.gz 2024-06-29T16:02:01.312 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.102481.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37835.log 2024-06-29T16:02:01.312 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.312 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88233.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88233.log.gz 2024-06-29T16:02:01.312 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48021.log 2024-06-29T16:02:01.312 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55025.log 2024-06-29T16:02:01.312 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37835.log: /var/log/ceph/ceph-client.admin.48021.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.37835.log.gz 2024-06-29T16:02:01.312 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100381.log 2024-06-29T16:02:01.312 INFO:teuthology.orchestra.run.smithi059.stderr: 90.9%/var/log/ceph/ceph-client.admin.55025.log: -- replaced with /var/log/ceph/ceph-client.admin.48021.log.gz 2024-06-29T16:02:01.313 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55025.log.gz 2024-06-29T16:02:01.313 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64311.log 2024-06-29T16:02:01.313 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55067.log 2024-06-29T16:02:01.313 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100381.log: /var/log/ceph/ceph-client.admin.64311.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100381.log.gz 2024-06-29T16:02:01.313 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.64311.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.112634.log 2024-06-29T16:02:01.313 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.313 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55067.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.55067.log.gz 2024-06-29T16:02:01.313 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70446.log 2024-06-29T16:02:01.314 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50175.log 2024-06-29T16:02:01.314 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112634.log: /var/log/ceph/ceph-client.admin.70446.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.112634.log.gz 2024-06-29T16:02:01.314 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70446.log.gz 2024-06-29T16:02:01.314 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110599.log 2024-06-29T16:02:01.314 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50175.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50175.log.gz 2024-06-29T16:02:01.314 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64825.log 2024-06-29T16:02:01.314 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111484.log 2024-06-29T16:02:01.314 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110599.log: /var/log/ceph/ceph-client.admin.64825.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110599.log.gz 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64825.log.gz 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71788.log 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111484.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111484.log.gz 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46409.log 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95294.log 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71788.log: /var/log/ceph/ceph-client.admin.46409.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.71788.log.gz 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46409.log.gz 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96078.log 2024-06-29T16:02:01.315 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95294.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.95294.log.gz 2024-06-29T16:02:01.316 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98465.log 2024-06-29T16:02:01.316 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69837.log 2024-06-29T16:02:01.316 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.96078.log: /var/log/ceph/ceph-client.admin.98465.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.96078.log.gz 2024-06-29T16:02:01.316 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.98465.log.gz 2024-06-29T16:02:01.316 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92522.log 2024-06-29T16:02:01.316 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69837.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.69837.log.gz 2024-06-29T16:02:01.316 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92480.log 2024-06-29T16:02:01.316 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81645.log 2024-06-29T16:02:01.317 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92522.log: /var/log/ceph/ceph-client.admin.92480.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92522.log.gz 2024-06-29T16:02:01.317 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.92480.log.gz 2024-06-29T16:02:01.317 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46239.log 2024-06-29T16:02:01.317 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81645.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81645.log.gz 2024-06-29T16:02:01.317 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80572.log 2024-06-29T16:02:01.317 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110001.log 2024-06-29T16:02:01.317 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46239.log: /var/log/ceph/ceph-client.admin.80572.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46239.log.gz 2024-06-29T16:02:01.317 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80572.log.gz 2024-06-29T16:02:01.318 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66785.log 2024-06-29T16:02:01.318 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79078.log 2024-06-29T16:02:01.318 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110001.log: /var/log/ceph/ceph-client.admin.66785.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110001.log.gz 2024-06-29T16:02:01.318 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66785.log.gz 2024-06-29T16:02:01.318 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110179.log 2024-06-29T16:02:01.318 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.79078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79078.log.gz 2024-06-29T16:02:01.318 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100295.log 2024-06-29T16:02:01.318 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49251.log 2024-06-29T16:02:01.319 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110179.log: /var/log/ceph/ceph-client.admin.100295.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110179.log.gz 2024-06-29T16:02:01.319 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100295.log.gz 2024-06-29T16:02:01.319 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55885.log 2024-06-29T16:02:01.319 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49251.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49251.log.gz 2024-06-29T16:02:01.319 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93355.log 2024-06-29T16:02:01.319 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52837.log 2024-06-29T16:02:01.319 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55885.log: /var/log/ceph/ceph-client.admin.93355.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55885.log.gz 2024-06-29T16:02:01.319 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.93355.log.gz 2024-06-29T16:02:01.320 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.95696.log 2024-06-29T16:02:01.320 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52837.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52837.log.gz 2024-06-29T16:02:01.320 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53417.log 2024-06-29T16:02:01.320 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68570.log 2024-06-29T16:02:01.320 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.95696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.95696.log.gz 2024-06-29T16:02:01.320 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51510.log 2024-06-29T16:02:01.320 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53417.log.gz 2024-06-29T16:02:01.320 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68570.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68570.log.gz 2024-06-29T16:02:01.321 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66913.log 2024-06-29T16:02:01.321 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82690.log 2024-06-29T16:02:01.321 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51510.log: /var/log/ceph/ceph-client.admin.66913.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51510.log.gz 2024-06-29T16:02:01.321 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66913.log.gz 2024-06-29T16:02:01.321 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97627.log 2024-06-29T16:02:01.321 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82690.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82690.log.gz 2024-06-29T16:02:01.321 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80488.log 2024-06-29T16:02:01.321 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89083.log 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97627.log: /var/log/ceph/ceph-client.admin.80488.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97627.log.gz 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80488.log.gz 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91272.log 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99917.log 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89083.log: /var/log/ceph/ceph-client.admin.91272.log: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.89083.log.gz 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91272.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57061.log 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.99917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.99917.log.gz 2024-06-29T16:02:01.322 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78550.log 2024-06-29T16:02:01.323 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95126.log 2024-06-29T16:02:01.323 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57061.log: /var/log/ceph/ceph-client.admin.78550.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.57061.log.gz 2024-06-29T16:02:01.323 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78550.log.gz 2024-06-29T16:02:01.323 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45987.log 2024-06-29T16:02:01.323 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82995.log 2024-06-29T16:02:01.323 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95126.log: /var/log/ceph/ceph-client.admin.45987.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95126.log.gz 2024-06-29T16:02:01.323 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45987.log.gz 2024-06-29T16:02:01.323 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106534.log 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82995.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.82995.log.gz 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59327.log 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47093.log 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106534.log: /var/log/ceph/ceph-client.admin.59327.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106534.log.gz 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59327.log.gz 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79435.log 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47093.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47093.log.gz 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67305.log 2024-06-29T16:02:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110137.log 2024-06-29T16:02:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79435.log: /var/log/ceph/ceph-client.admin.67305.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79435.log.gz 2024-06-29T16:02:01.325 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67305.log.gz 2024-06-29T16:02:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110702.log 2024-06-29T16:02:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110137.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110137.log.gz 2024-06-29T16:02:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110473.log 2024-06-29T16:02:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52116.log 2024-06-29T16:02:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110702.log: /var/log/ceph/ceph-client.admin.110473.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.110702.log.gz 2024-06-29T16:02:01.325 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.110473.log.gz 2024-06-29T16:02:01.326 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93670.log 2024-06-29T16:02:01.326 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52116.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52116.log.gz 2024-06-29T16:02:01.326 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79671.log 2024-06-29T16:02:01.326 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45819.log 2024-06-29T16:02:01.326 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93670.log: /var/log/ceph/ceph-client.admin.79671.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93670.log.gz 2024-06-29T16:02:01.326 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79671.log.gz 2024-06-29T16:02:01.326 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65180.log 2024-06-29T16:02:01.326 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45819.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45819.log.gz 2024-06-29T16:02:01.327 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64573.log 2024-06-29T16:02:01.327 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104375.log 2024-06-29T16:02:01.327 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65180.log: /var/log/ceph/ceph-client.admin.64573.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65180.log.gz 2024-06-29T16:02:01.327 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64573.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61000.log 2024-06-29T16:02:01.327 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.327 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.104375.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.104375.log.gz 2024-06-29T16:02:01.327 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77038.log 2024-06-29T16:02:01.327 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108156.log 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61000.log: /var/log/ceph/ceph-client.admin.77038.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61000.log.gz 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77038.log.gz 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104249.log 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101011.log 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108156.log: /var/log/ceph/ceph-client.admin.104249.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108156.log.gz 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100251.log 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104249.log.gz 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101011.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101011.log.gz 2024-06-29T16:02:01.328 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77164.log 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100251.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100251.log.gz 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74047.log 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113992.log 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77164.log: /var/log/ceph/ceph-client.admin.74047.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.77164.log.gz 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74047.log.gz 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65138.log 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76818.log 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65138.log: /var/log/ceph/ceph-client.admin.113992.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113992.log.gz 2024-06-29T16:02:01.329 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65138.log.gz 2024-06-29T16:02:01.330 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111877.log 2024-06-29T16:02:01.330 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74127.log 2024-06-29T16:02:01.330 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76818.log: /var/log/ceph/ceph-client.admin.111877.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.76818.log.gz 2024-06-29T16:02:01.330 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111877.log.gz 2024-06-29T16:02:01.330 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66719.log 2024-06-29T16:02:01.330 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108647.log 2024-06-29T16:02:01.330 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74127.log: /var/log/ceph/ceph-client.admin.66719.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74127.log.gz 2024-06-29T16:02:01.330 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66719.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75338.log 2024-06-29T16:02:01.331 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.331 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110221.log 2024-06-29T16:02:01.331 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75338.log: /var/log/ceph/ceph-client.admin.108647.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108647.log.gz 2024-06-29T16:02:01.331 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75338.log.gz 2024-06-29T16:02:01.331 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107038.log 2024-06-29T16:02:01.331 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100083.log 2024-06-29T16:02:01.331 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110221.log: /var/log/ceph/ceph-client.admin.107038.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.110221.log.gz 2024-06-29T16:02:01.331 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107038.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57324.log 2024-06-29T16:02:01.332 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.332 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114939.log 2024-06-29T16:02:01.332 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57324.log: /var/log/ceph/ceph-client.admin.100083.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.100083.log.gz 2024-06-29T16:02:01.332 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.57324.log.gz 2024-06-29T16:02:01.332 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95396.log 2024-06-29T16:02:01.332 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.114939.log: --verbose -- /var/log/ceph/ceph-client.admin.56257.log 2024-06-29T16:02:01.332 INFO:teuthology.orchestra.run.smithi059.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.114939.log.gz 2024-06-29T16:02:01.332 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95396.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113847.log 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.95396.log.gz 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56257.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.56257.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51426.log 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50592.log 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113847.log: /var/log/ceph/ceph-client.admin.51426.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113847.log.gz 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59978.log 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51426.log.gz 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59684.log 2024-06-29T16:02:01.333 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59978.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50592.log.gz 2024-06-29T16:02:01.334 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59978.log.gz 2024-06-29T16:02:01.334 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106996.log 2024-06-29T16:02:01.334 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94785.log 2024-06-29T16:02:01.334 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59684.log: /var/log/ceph/ceph-client.admin.106996.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59684.log.gz 2024-06-29T16:02:01.334 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71333.log 2024-06-29T16:02:01.334 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106996.log.gz 2024-06-29T16:02:01.334 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68240.log 2024-06-29T16:02:01.335 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71333.log: /var/log/ceph/ceph-client.admin.94785.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94785.log.gz 2024-06-29T16:02:01.335 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71333.log.gz 2024-06-29T16:02:01.335 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64216.log 2024-06-29T16:02:01.335 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.68240.log: -- /var/log/ceph/ceph-client.admin.66257.log 2024-06-29T16:02:01.335 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68240.log.gz 2024-06-29T16:02:01.335 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99058.log 2024-06-29T16:02:01.335 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66257.log: 87.0% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66257.log.gz 2024-06-29T16:02:01.335 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64216.log.gz 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74292.log 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94277.log 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99058.log.gz 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75558.log 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94277.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74292.log.gz 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94277.log.gz 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102313.log 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106492.log 2024-06-29T16:02:01.336 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.75558.log.gz 2024-06-29T16:02:01.337 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51864.log 2024-06-29T16:02:01.337 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106492.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102313.log.gz 2024-06-29T16:02:01.337 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106492.log.gz 2024-06-29T16:02:01.337 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52446.log 2024-06-29T16:02:01.337 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.104797.log 2024-06-29T16:02:01.337 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51864.log: /var/log/ceph/ceph-client.admin.52446.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51864.log.gz 2024-06-29T16:02:01.337 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106912.log 2024-06-29T16:02:01.337 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52446.log.gz 2024-06-29T16:02:01.338 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94914.log 2024-06-29T16:02:01.338 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.104797.log: /var/log/ceph/ceph-client.admin.106912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.104797.log.gz 2024-06-29T16:02:01.338 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106912.log.gz 2024-06-29T16:02:01.338 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51222.log 2024-06-29T16:02:01.338 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59464.log 2024-06-29T16:02:01.338 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.94914.log: /var/log/ceph/ceph-client.admin.51222.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.94914.log.gz 2024-06-29T16:02:01.338 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51222.log.gz 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61550.log 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51906.log 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61550.log: /var/log/ceph/ceph-client.admin.59464.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59464.log.gz 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61550.log.gz 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104037.log 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.51906.log: -- /var/log/ceph/ceph-client.admin.72820.log 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51906.log.gz 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.104037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112855.log 2024-06-29T16:02:01.339 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72820.log: 86.9% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72820.log.gz 2024-06-29T16:02:01.340 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104037.log.gz 2024-06-29T16:02:01.340 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66365.log 2024-06-29T16:02:01.340 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70110.log 2024-06-29T16:02:01.340 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112855.log.gz 2024-06-29T16:02:01.340 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87269.log 2024-06-29T16:02:01.340 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70110.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66365.log.gz 2024-06-29T16:02:01.340 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70110.log.gz 2024-06-29T16:02:01.340 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115023.log 2024-06-29T16:02:01.341 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61214.log 2024-06-29T16:02:01.341 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87269.log: /var/log/ceph/ceph-client.admin.115023.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87269.log.gz 2024-06-29T16:02:01.341 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43679.log 2024-06-29T16:02:01.341 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.115023.log.gz 2024-06-29T16:02:01.341 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81730.log 2024-06-29T16:02:01.341 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43679.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61214.log.gz 2024-06-29T16:02:01.341 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43679.log.gz 2024-06-29T16:02:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60291.log 2024-06-29T16:02:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47625.log 2024-06-29T16:02:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81730.log: /var/log/ceph/ceph-client.admin.60291.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81730.log.gz 2024-06-29T16:02:01.342 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60291.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87735.log 2024-06-29T16:02:01.342 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.47625.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.115065.log 2024-06-29T16:02:01.342 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47625.log.gz 2024-06-29T16:02:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104922.log 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.115065.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87735.log.gz 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.115065.log.gz 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46752.log 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.104922.log: -- /var/log/ceph/ceph-client.admin.105048.log 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104922.log.gz 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46750.log 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105048.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46752.log.gz 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105048.log.gz 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.343 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95252.log 2024-06-29T16:02:01.344 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111337.log 2024-06-29T16:02:01.344 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46750.log.gz 2024-06-29T16:02:01.344 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.95252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45699.log 2024-06-29T16:02:01.344 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111337.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.95252.log.gz 2024-06-29T16:02:01.344 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111337.log.gz 2024-06-29T16:02:01.344 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79812.log 2024-06-29T16:02:01.344 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66557.log 2024-06-29T16:02:01.344 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45699.log.gz 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108819.log 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.66557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.66557.log.gz 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr: 83.8% -- replaced with /var/log/ceph/ceph-client.admin.79812.log.gz 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46413.log 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37974.log 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108819.log.gz 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73742.log 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37974.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46413.log.gz 86.1% -- replaced with /var/log/ceph/ceph-client.admin.37974.log.gz 2024-06-29T16:02:01.345 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.346 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43845.log 2024-06-29T16:02:01.346 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69718.log 2024-06-29T16:02:01.346 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73742.log.gz 2024-06-29T16:02:01.346 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98423.log 2024-06-29T16:02:01.346 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69718.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43845.log.gz 2024-06-29T16:02:01.346 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69718.log.gz 2024-06-29T16:02:01.346 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101585.log 2024-06-29T16:02:01.346 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111440.log 2024-06-29T16:02:01.347 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98423.log: /var/log/ceph/ceph-client.admin.101585.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98423.log.gz 2024-06-29T16:02:01.347 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101585.log.gz 2024-06-29T16:02:01.347 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102397.log 2024-06-29T16:02:01.347 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82269.log 2024-06-29T16:02:01.347 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102397.log: /var/log/ceph/ceph-client.admin.111440.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111440.log.gz 2024-06-29T16:02:01.347 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102397.log.gz 2024-06-29T16:02:01.347 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101220.log 2024-06-29T16:02:01.347 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70572.log 2024-06-29T16:02:01.348 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82269.log: /var/log/ceph/ceph-client.admin.101220.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.82269.log.gz 2024-06-29T16:02:01.348 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101220.log.gz 2024-06-29T16:02:01.348 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100759.log 2024-06-29T16:02:01.348 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.70572.log: --verbose -- /var/log/ceph/ceph-client.admin.108861.log 2024-06-29T16:02:01.348 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70572.log.gz 2024-06-29T16:02:01.348 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69474.log 2024-06-29T16:02:01.348 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108861.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.100759.log.gz 2024-06-29T16:02:01.348 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108861.log.gz 2024-06-29T16:02:01.349 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65731.log 2024-06-29T16:02:01.349 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107687.log 2024-06-29T16:02:01.349 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69474.log: /var/log/ceph/ceph-client.admin.65731.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69474.log.gz 2024-06-29T16:02:01.349 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72538.log 2024-06-29T16:02:01.349 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65731.log.gz 2024-06-29T16:02:01.349 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105313.log 2024-06-29T16:02:01.349 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107687.log: /var/log/ceph/ceph-client.admin.72538.log: 87.1% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107687.log.gz 2024-06-29T16:02:01.349 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72538.log.gz 2024-06-29T16:02:01.350 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53315.log 2024-06-29T16:02:01.350 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99453.log 2024-06-29T16:02:01.350 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105313.log: /var/log/ceph/ceph-client.admin.53315.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105313.log.gz 2024-06-29T16:02:01.350 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53609.log 2024-06-29T16:02:01.350 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53315.log.gz 2024-06-29T16:02:01.350 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102962.log 2024-06-29T16:02:01.350 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53609.log: /var/log/ceph/ceph-client.admin.99453.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99453.log.gz 2024-06-29T16:02:01.350 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53609.log.gz 2024-06-29T16:02:01.351 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108903.log 2024-06-29T16:02:01.351 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.102962.log: -- /var/log/ceph/ceph-client.admin.45389.log 2024-06-29T16:02:01.351 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102962.log.gz 2024-06-29T16:02:01.351 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87820.log 2024-06-29T16:02:01.351 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45389.log: 86.9% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45389.log.gz 2024-06-29T16:02:01.351 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108903.log.gz 2024-06-29T16:02:01.351 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74208.log 2024-06-29T16:02:01.351 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38725.log 2024-06-29T16:02:01.352 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87820.log.gz 2024-06-29T16:02:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102439.log 2024-06-29T16:02:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38725.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74208.log.gz 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38725.log.gz 2024-06-29T16:02:01.352 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47289.log 2024-06-29T16:02:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90732.log 2024-06-29T16:02:01.352 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.102439.log.gz 2024-06-29T16:02:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64489.log 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90732.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47289.log.gz 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90732.log.gz 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97942.log 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77122.log 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64489.log: /var/log/ceph/ceph-client.admin.97942.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64489.log.gz 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97942.log.gz 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86955.log 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67347.log 2024-06-29T16:02:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86955.log: /var/log/ceph/ceph-client.admin.77122.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.77122.log.gz 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86955.log.gz 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91656.log 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74792.log 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67347.log: /var/log/ceph/ceph-client.0.91656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.91656.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67347.log.gz 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49629.log 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50712.log 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74792.log: /var/log/ceph/ceph-client.admin.49629.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.74792.log.gz 2024-06-29T16:02:01.354 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49629.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101358.log 2024-06-29T16:02:01.355 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95210.log 2024-06-29T16:02:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50712.log: /var/log/ceph/ceph-client.admin.101358.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50712.log.gz 2024-06-29T16:02:01.355 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.101358.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93922.log 2024-06-29T16:02:01.355 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99369.log 2024-06-29T16:02:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93922.log: /var/log/ceph/ceph-client.admin.95210.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.95210.log.gz 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.93922.log.gz 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65604.log 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99369.log: /var/log/ceph/ceph-client.admin.65604.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.99369.log.gz 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65604.log.gz 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104501.log 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108987.log 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37638.log: /var/log/ceph/ceph-client.admin.104501.log: 79.0% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-06-29T16:02:01.356 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45615.log 2024-06-29T16:02:01.357 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104501.log.gz 2024-06-29T16:02:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110828.log 2024-06-29T16:02:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45615.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45615.log.gz 2024-06-29T16:02:01.357 INFO:teuthology.orchestra.run.smithi059.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.108987.log.gz 2024-06-29T16:02:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108279.log 2024-06-29T16:02:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110828.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68784.log 2024-06-29T16:02:01.357 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.110828.log.gz 2024-06-29T16:02:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81305.log 2024-06-29T16:02:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108279.log: /var/log/ceph/ceph-client.admin.68784.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68784.log.gz 2024-06-29T16:02:01.358 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108279.log.gz 2024-06-29T16:02:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107813.log 2024-06-29T16:02:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60735.log 2024-06-29T16:02:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81305.log: /var/log/ceph/ceph-client.admin.107813.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81305.log.gz 2024-06-29T16:02:01.358 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73345.log 2024-06-29T16:02:01.358 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107813.log.gz 2024-06-29T16:02:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58113.log 2024-06-29T16:02:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60735.log: /var/log/ceph/ceph-client.admin.73345.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60735.log.gz 2024-06-29T16:02:01.359 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.112261.log 2024-06-29T16:02:01.359 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73345.log.gz 2024-06-29T16:02:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71463.log 2024-06-29T16:02:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58113.log: /var/log/ceph/ceph-client.0.112261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.112261.log.gz 2024-06-29T16:02:01.359 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.58113.log.gz 2024-06-29T16:02:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99579.log 2024-06-29T16:02:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99411.log 2024-06-29T16:02:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71463.log: /var/log/ceph/ceph-client.admin.99579.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.71463.log.gz 2024-06-29T16:02:01.360 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.99579.log.gz 2024-06-29T16:02:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77645.log 2024-06-29T16:02:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89439.log 2024-06-29T16:02:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99411.log: /var/log/ceph/ceph-client.admin.77645.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99411.log.gz 2024-06-29T16:02:01.360 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77645.log.gz 2024-06-29T16:02:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56375.log 2024-06-29T16:02:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72377.log 2024-06-29T16:02:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89439.log: /var/log/ceph/ceph-client.admin.56375.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89439.log.gz 2024-06-29T16:02:01.361 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.56375.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107981.log 2024-06-29T16:02:01.361 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91146.log 2024-06-29T16:02:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107981.log: /var/log/ceph/ceph-client.admin.72377.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72377.log.gz 2024-06-29T16:02:01.361 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107981.log.gz 2024-06-29T16:02:01.361 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65987.log 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68826.log 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91146.log: /var/log/ceph/ceph-client.admin.65987.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91146.log.gz 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65987.log.gz 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111709.log 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67032.log 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68826.log.gz 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74335.log 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67032.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111709.log.gz 2024-06-29T16:02:01.362 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67032.log.gz 2024-06-29T16:02:01.363 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115219.log 2024-06-29T16:02:01.363 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49503.log 2024-06-29T16:02:01.363 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74335.log: /var/log/ceph/ceph-client.admin.115219.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74335.log.gz 2024-06-29T16:02:01.363 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.115219.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49293.log 2024-06-29T16:02:01.363 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.363 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47373.log 2024-06-29T16:02:01.363 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49293.log: /var/log/ceph/ceph-client.admin.49503.log: 87.4% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49503.log.gz 2024-06-29T16:02:01.363 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49293.log.gz 2024-06-29T16:02:01.364 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102794.log 2024-06-29T16:02:01.364 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52882.log 2024-06-29T16:02:01.364 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47373.log: /var/log/ceph/ceph-client.admin.102794.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47373.log.gz 2024-06-29T16:02:01.364 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102794.log.gz 2024-06-29T16:02:01.364 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107458.log 2024-06-29T16:02:01.364 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105612.log 2024-06-29T16:02:01.364 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107458.log: /var/log/ceph/ceph-client.admin.52882.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107458.log.gz 2024-06-29T16:02:01.364 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52882.log.gz 2024-06-29T16:02:01.365 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111092.log 2024-06-29T16:02:01.365 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70488.log 2024-06-29T16:02:01.365 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105612.log.gz 2024-06-29T16:02:01.365 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75446.log 2024-06-29T16:02:01.365 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70488.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111092.log.gz 2024-06-29T16:02:01.365 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70488.log.gz 2024-06-29T16:02:01.365 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80698.log 2024-06-29T16:02:01.365 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53231.log 2024-06-29T16:02:01.366 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.75446.log.gz 2024-06-29T16:02:01.366 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86388.log 2024-06-29T16:02:01.366 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53231.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.80698.log.gz 2024-06-29T16:02:01.366 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53231.log.gz 2024-06-29T16:02:01.366 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107122.log 2024-06-29T16:02:01.366 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81442.log 2024-06-29T16:02:01.366 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86388.log: /var/log/ceph/ceph-client.admin.107122.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86388.log.gz 2024-06-29T16:02:01.366 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94361.log 2024-06-29T16:02:01.367 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9%/var/log/ceph/ceph-client.admin.81442.log: -- replaced with /var/log/ceph/ceph-client.admin.107122.log.gz 2024-06-29T16:02:01.367 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81442.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2024-06-29T16:02:01.367 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.367 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100209.log 2024-06-29T16:02:01.367 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43583.log: /var/log/ceph/ceph-client.admin.94361.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.94361.log.gz 2024-06-29T16:02:01.367 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2024-06-29T16:02:01.368 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64258.log 2024-06-29T16:02:01.368 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.100209.log: -- /var/log/ceph/ceph-client.0.112257.log 2024-06-29T16:02:01.368 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.100209.log.gz 2024-06-29T16:02:01.368 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67599.log 2024-06-29T16:02:01.368 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.112257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.112257.log.gz 2024-06-29T16:02:01.368 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64258.log.gz 2024-06-29T16:02:01.368 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49839.log 2024-06-29T16:02:01.368 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87039.log 2024-06-29T16:02:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67599.log: /var/log/ceph/ceph-client.admin.49839.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.67599.log.gz 2024-06-29T16:02:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60861.log 2024-06-29T16:02:01.369 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5%/var/log/ceph/ceph-client.admin.87039.log: -- replaced with /var/log/ceph/ceph-client.admin.49839.log.gz 2024-06-29T16:02:01.369 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87039.log.gz 2024-06-29T16:02:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64615.log 2024-06-29T16:02:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95774.log 2024-06-29T16:02:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64615.log: /var/log/ceph/ceph-client.admin.60861.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60861.log.gz 2024-06-29T16:02:01.369 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.64615.log.gz 2024-06-29T16:02:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69921.log 2024-06-29T16:02:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.95774.log: -- /var/log/ceph/ceph-client.admin.68061.log 2024-06-29T16:02:01.370 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.95774.log.gz 2024-06-29T16:02:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93313.log 2024-06-29T16:02:01.370 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69921.log.gz 2024-06-29T16:02:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68061.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68061.log.gz 2024-06-29T16:02:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90974.log 2024-06-29T16:02:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82059.log 2024-06-29T16:02:01.371 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93313.log.gz 2024-06-29T16:02:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110912.log 2024-06-29T16:02:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82059.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90974.log.gz 2024-06-29T16:02:01.371 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82059.log.gz 2024-06-29T16:02:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114897.log 2024-06-29T16:02:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111380.log 2024-06-29T16:02:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110912.log: /var/log/ceph/ceph-client.admin.114897.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110912.log.gz 2024-06-29T16:02:01.372 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.114897.log.gz 2024-06-29T16:02:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61970.log 2024-06-29T16:02:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98675.log 2024-06-29T16:02:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61970.log: /var/log/ceph/ceph-client.admin.111380.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61970.log.gz 2024-06-29T16:02:01.372 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111380.log.gz 2024-06-29T16:02:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112813.log 2024-06-29T16:02:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111663.log 2024-06-29T16:02:01.372 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98675.log.gz 2024-06-29T16:02:01.373 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114519.log 2024-06-29T16:02:01.373 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111663.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.112813.log.gz 2024-06-29T16:02:01.373 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111663.log.gz 2024-06-29T16:02:01.373 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90487.log 2024-06-29T16:02:01.373 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43802.log 2024-06-29T16:02:01.373 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.114519.log.gz 2024-06-29T16:02:01.373 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54352.log 2024-06-29T16:02:01.373 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90487.log.gz 2024-06-29T16:02:01.374 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43802.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43802.log.gz 2024-06-29T16:02:01.374 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62012.log 2024-06-29T16:02:01.374 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54470.log 2024-06-29T16:02:01.374 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54352.log.gz 2024-06-29T16:02:01.374 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51054.log 2024-06-29T16:02:01.374 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54470.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62012.log.gz 2024-06-29T16:02:01.374 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54470.log.gz 2024-06-29T16:02:01.374 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76723.log 2024-06-29T16:02:01.375 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.51054.log: -- /var/log/ceph/ceph-client.admin.77416.log 2024-06-29T16:02:01.375 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51054.log.gz 2024-06-29T16:02:01.375 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86346.log 2024-06-29T16:02:01.375 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76723.log.gz 2024-06-29T16:02:01.375 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61084.log 2024-06-29T16:02:01.375 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86346.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77416.log.gz 2024-06-29T16:02:01.375 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86346.log.gz 2024-06-29T16:02:01.375 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77989.log 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79365.log 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.61084.log.gz 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101095.log 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77989.log.gz 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79365.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79365.log.gz 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59600.log 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101273.log 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101095.log.gz 2024-06-29T16:02:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65264.log 2024-06-29T16:02:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101273.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59600.log.gz 2024-06-29T16:02:01.377 INFO:teuthology.orchestra.run.smithi059.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.101273.log.gz 2024-06-29T16:02:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56003.log 2024-06-29T16:02:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104081.log 2024-06-29T16:02:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65264.log: /var/log/ceph/ceph-client.admin.56003.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65264.log.gz 2024-06-29T16:02:01.377 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.56003.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47499.log 2024-06-29T16:02:01.377 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109948.log 2024-06-29T16:02:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47499.log: /var/log/ceph/ceph-client.admin.104081.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47499.log.gz 2024-06-29T16:02:01.378 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104081.log.gz 2024-06-29T16:02:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73261.log 2024-06-29T16:02:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-06-29T16:02:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109948.log: /var/log/ceph/ceph-client.admin.73261.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.109948.log.gz 2024-06-29T16:02:01.378 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.73261.log.gz 2024-06-29T16:02:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91884.log 2024-06-29T16:02:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114729.log 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.91884.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91884.log.gz 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71704.log 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114729.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.114729.log.gz 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114687.log 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71704.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71704.log.gz 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44331.log 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114687.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.114687.log.gz 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93450.log 2024-06-29T16:02:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44331.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44331.log.gz 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67169.log 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93450.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93450.log.gz 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53801.log 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67169.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67169.log.gz 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.104801.log 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.53801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53801.log.gz 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46421.log 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.104801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.104801.log.gz 2024-06-29T16:02:01.380 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60123.log 2024-06-29T16:02:01.381 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46421.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.46421.log.gz 2024-06-29T16:02:01.381 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108605.log 2024-06-29T16:02:01.381 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60123.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60123.log.gz 2024-06-29T16:02:01.381 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43760.log 2024-06-29T16:02:01.381 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108605.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108605.log.gz 2024-06-29T16:02:01.381 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89827.log 2024-06-29T16:02:01.381 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43760.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43760.log.gz 2024-06-29T16:02:01.381 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89785.log 2024-06-29T16:02:01.382 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89827.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89827.log.gz 2024-06-29T16:02:01.382 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80782.log 2024-06-29T16:02:01.382 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89785.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.89785.log.gz 2024-06-29T16:02:01.382 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46749.log 2024-06-29T16:02:01.382 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80782.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80782.log.gz 2024-06-29T16:02:01.382 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78592.log 2024-06-29T16:02:01.382 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46749.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.46749.log.gz 2024-06-29T16:02:01.382 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97816.log 2024-06-29T16:02:01.383 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78592.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78592.log.gz 2024-06-29T16:02:01.383 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89260.log 2024-06-29T16:02:01.383 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97816.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97816.log.gz 2024-06-29T16:02:01.383 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51654.log 2024-06-29T16:02:01.383 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89260.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89260.log.gz 2024-06-29T16:02:01.383 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100339.log 2024-06-29T16:02:01.383 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51654.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51654.log.gz 2024-06-29T16:02:01.383 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-06-29T16:02:01.384 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.0.88697.log 2024-06-29T16:02:01.384 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44541.log 2024-06-29T16:02:01.384 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100339.log: /var/log/ceph/ceph-client.0.88697.log: 87.3% 0.0% -- replaced with /var/log/ceph/ceph-client.0.88697.log.gz 2024-06-29T16:02:01.384 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.100339.log.gz 2024-06-29T16:02:01.384 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90392.log 2024-06-29T16:02:01.384 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65689.log 2024-06-29T16:02:01.384 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90392.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44541.log.gz 2024-06-29T16:02:01.384 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90392.log.gz 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65819.log 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43971.log 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65689.log.gz 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61042.log 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43971.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65819.log.gz 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43971.log.gz 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47541.log 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105402.log 2024-06-29T16:02:01.385 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47541.log: /var/log/ceph/ceph-client.admin.61042.log: 87.2% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61042.log.gz 2024-06-29T16:02:01.386 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47541.log.gz 2024-06-29T16:02:01.386 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.95692.log 2024-06-29T16:02:01.386 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.105402.log: /var/log/ceph/ceph-client.admin.63945.log 2024-06-29T16:02:01.386 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105402.log.gz 2024-06-29T16:02:01.386 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.95692.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.0.95692.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.103340.log 2024-06-29T16:02:01.386 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.386 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50507.log 2024-06-29T16:02:01.386 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103340.log: /var/log/ceph/ceph-client.admin.63945.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63945.log.gz 2024-06-29T16:02:01.387 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103340.log.gz 2024-06-29T16:02:01.387 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70698.log 2024-06-29T16:02:01.387 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50507.log: -- /var/log/ceph/ceph-client.admin.87950.log 2024-06-29T16:02:01.387 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50507.log.gz 2024-06-29T16:02:01.387 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72023.log 2024-06-29T16:02:01.387 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87950.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70698.log.gz 2024-06-29T16:02:01.387 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87950.log.gz 2024-06-29T16:02:01.387 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90879.log 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44203.log 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72023.log: /var/log/ceph/ceph-client.admin.90879.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72023.log.gz 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.90879.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80352.log 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59369.log 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44203.log: /var/log/ceph/ceph-client.admin.80352.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44203.log.gz 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55507.log 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80352.log.gz 2024-06-29T16:02:01.388 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88509.log 2024-06-29T16:02:01.389 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55507.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59369.log.gz 2024-06-29T16:02:01.389 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.55507.log.gz 2024-06-29T16:02:01.389 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55944.log 2024-06-29T16:02:01.389 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60165.log 2024-06-29T16:02:01.389 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88509.log: /var/log/ceph/ceph-client.admin.55944.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88509.log.gz 2024-06-29T16:02:01.389 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.55944.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113637.log 2024-06-29T16:02:01.389 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80530.log 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60165.log: /var/log/ceph/ceph-client.admin.113637.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.113637.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60165.log.gz 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.109241.log 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.80530.log: -- /var/log/ceph/ceph-client.admin.65646.log 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.80530.log.gz/var/log/ceph/ceph-client.0.109241.log: 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.109241.log.gz 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80614.log 2024-06-29T16:02:01.390 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113679.log 2024-06-29T16:02:01.391 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65646.log.gz 2024-06-29T16:02:01.391 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99999.log 2024-06-29T16:02:01.391 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.113679.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.80614.log.gz 2024-06-29T16:02:01.391 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113679.log.gz 2024-06-29T16:02:01.391 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73429.log 2024-06-29T16:02:01.391 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.99999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98363.log 2024-06-29T16:02:01.391 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.99999.log.gz 2024-06-29T16:02:01.391 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51342.log 2024-06-29T16:02:01.392 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.98363.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73429.log.gz 2024-06-29T16:02:01.392 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98363.log.gz 2024-06-29T16:02:01.392 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102523.log 2024-06-29T16:02:01.392 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71914.log 2024-06-29T16:02:01.392 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102523.log: /var/log/ceph/ceph-client.admin.51342.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51342.log.gz 2024-06-29T16:02:01.392 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102523.log.gz 2024-06-29T16:02:01.392 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94743.log 2024-06-29T16:02:01.392 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91734.log 2024-06-29T16:02:01.393 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71914.log: /var/log/ceph/ceph-client.admin.94743.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71914.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94743.log.gz 2024-06-29T16:02:01.393 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.393 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93796.log 2024-06-29T16:02:01.393 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70530.log 2024-06-29T16:02:01.393 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91734.log.gz 2024-06-29T16:02:01.393 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.93796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112489.log 2024-06-29T16:02:01.393 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70530.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.93796.log.gz 2024-06-29T16:02:01.393 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.70530.log.gz 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64867.log 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97543.log 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.112489.log: /var/log/ceph/ceph-client.admin.64867.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112489.log.gz 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.64867.log.gz 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114603.log 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49545.log 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.114603.log: /var/log/ceph/ceph-client.admin.97543.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97543.log.gz 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.114603.log.gz 2024-06-29T16:02:01.394 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94235.log 2024-06-29T16:02:01.395 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.88701.log 2024-06-29T16:02:01.395 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49545.log: /var/log/ceph/ceph-client.admin.94235.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49545.log.gz 2024-06-29T16:02:01.395 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.94235.log.gz 2024-06-29T16:02:01.395 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87777.log 2024-06-29T16:02:01.395 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77687.log 2024-06-29T16:02:01.395 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87777.log: /var/log/ceph/ceph-client.0.88701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.88701.log.gz 2024-06-29T16:02:01.395 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87777.log.gz 2024-06-29T16:02:01.395 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48946.log 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.77687.log: --verbose -- /var/log/ceph/ceph-client.admin.60777.log 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77687.log.gz 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.2.48946.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81221.log 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.2.48946.log.gz 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60777.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93397.log 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60777.log.gz 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49965.log 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81221.log: /var/log/ceph/ceph-client.admin.93397.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81221.log.gz 2024-06-29T16:02:01.396 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91104.log 2024-06-29T16:02:01.397 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.93397.log.gz 2024-06-29T16:02:01.397 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100125.log 2024-06-29T16:02:01.397 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91104.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49965.log.gz 2024-06-29T16:02:01.397 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91104.log.gz 2024-06-29T16:02:01.397 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82419.log 2024-06-29T16:02:01.397 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68970.log 2024-06-29T16:02:01.397 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100125.log: /var/log/ceph/ceph-client.admin.82419.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.100125.log.gz 2024-06-29T16:02:01.397 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72999.log 2024-06-29T16:02:01.398 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82419.log.gz 2024-06-29T16:02:01.398 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103951.log 2024-06-29T16:02:01.398 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68970.log: /var/log/ceph/ceph-client.admin.72999.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68970.log.gz 2024-06-29T16:02:01.398 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72999.log.gz 2024-06-29T16:02:01.398 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105176.log 2024-06-29T16:02:01.398 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.103951.log: --verbose -- /var/log/ceph/ceph-client.admin.74576.log 2024-06-29T16:02:01.398 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.103951.log.gz 2024-06-29T16:02:01.398 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69558.log 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74576.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105176.log.gz 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74576.log.gz 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88779.log 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105570.log 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69558.log.gz 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80394.log 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105570.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88779.log.gz 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.105570.log.gz 2024-06-29T16:02:01.399 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73952.log 2024-06-29T16:02:01.400 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101841.log 2024-06-29T16:02:01.400 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80394.log: /var/log/ceph/ceph-client.admin.73952.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80394.log.gz 2024-06-29T16:02:01.400 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73952.log.gz 2024-06-29T16:02:01.400 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106576.log 2024-06-29T16:02:01.400 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.101841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71746.log 2024-06-29T16:02:01.400 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101841.log.gz 2024-06-29T16:02:01.400 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.106576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67473.log 2024-06-29T16:02:01.400 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71746.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.106576.log.gz 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91776.log 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71746.log.gz 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91016.log 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67473.log: /var/log/ceph/ceph-client.admin.91776.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.67473.log.gz 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91776.log.gz 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110043.log 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109585.log 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91016.log: /var/log/ceph/ceph-client.admin.110043.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91016.log.gz 2024-06-29T16:02:01.401 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110043.log.gz 2024-06-29T16:02:01.402 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88149.log 2024-06-29T16:02:01.402 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67893.log 2024-06-29T16:02:01.402 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109585.log: /var/log/ceph/ceph-client.admin.88149.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.109585.log.gz 2024-06-29T16:02:01.402 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37721.log 2024-06-29T16:02:01.402 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67893.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.88149.log.gz 2024-06-29T16:02:01.402 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74483.log 2024-06-29T16:02:01.402 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67893.log.gz 2024-06-29T16:02:01.402 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.37721.log: 76.8% -- replaced with /var/log/ceph/ceph-client.admin.37721.log.gz 2024-06-29T16:02:01.403 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110347.log 2024-06-29T16:02:01.403 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70740.log 2024-06-29T16:02:01.403 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74483.log: /var/log/ceph/ceph-client.admin.110347.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74483.log.gz 2024-06-29T16:02:01.403 INFO:teuthology.orchestra.run.smithi059.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.110347.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91058.log 2024-06-29T16:02:01.403 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.403 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107332.log 2024-06-29T16:02:01.403 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91058.log: /var/log/ceph/ceph-client.admin.70740.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.70740.log.gz 2024-06-29T16:02:01.403 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91058.log.gz 2024-06-29T16:02:01.404 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79553.log 2024-06-29T16:02:01.404 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110263.log 2024-06-29T16:02:01.404 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107332.log: /var/log/ceph/ceph-client.admin.79553.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.107332.log.gz 2024-06-29T16:02:01.404 INFO:teuthology.orchestra.run.smithi059.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.79553.log.gz 2024-06-29T16:02:01.404 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100041.log 2024-06-29T16:02:01.404 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.110263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88443.log 2024-06-29T16:02:01.404 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.110263.log.gz 2024-06-29T16:02:01.405 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103484.log 2024-06-29T16:02:01.405 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88443.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100041.log.gz 2024-06-29T16:02:01.405 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70614.log 2024-06-29T16:02:01.405 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.88443.log.gz 2024-06-29T16:02:01.405 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103484.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.103484.log.gz 2024-06-29T16:02:01.405 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86184.log 2024-06-29T16:02:01.405 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51780.log 2024-06-29T16:02:01.405 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70614.log: /var/log/ceph/ceph-client.admin.86184.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70614.log.gz 2024-06-29T16:02:01.406 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86184.log.gz 2024-06-29T16:02:01.406 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60375.log 2024-06-29T16:02:01.406 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72443.log 2024-06-29T16:02:01.406 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60375.log: /var/log/ceph/ceph-client.admin.51780.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51780.log.gz 2024-06-29T16:02:01.406 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60375.log.gz 2024-06-29T16:02:01.406 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110954.log 2024-06-29T16:02:01.406 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80215.log 2024-06-29T16:02:01.406 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72443.log: /var/log/ceph/ceph-client.admin.110954.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72443.log.gz 2024-06-29T16:02:01.407 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110954.log.gz 2024-06-29T16:02:01.407 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57587.log 2024-06-29T16:02:01.407 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61846.log 2024-06-29T16:02:01.407 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80215.log.gz 2024-06-29T16:02:01.407 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92804.log 2024-06-29T16:02:01.407 INFO:teuthology.orchestra.run.smithi059.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.57587.log.gz 2024-06-29T16:02:01.407 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.61846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80095.log 2024-06-29T16:02:01.407 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61846.log.gz 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92804.log: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45861.log 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92804.log.gz 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37995.log 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80095.log: /var/log/ceph/ceph-client.admin.45861.log: 87.5% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80095.log.gz 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45861.log.gz 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49167.log 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37995.log: -- /var/log/ceph/ceph-client.admin.74534.log 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37995.log.gz 2024-06-29T16:02:01.408 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107584.log 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74534.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49167.log.gz 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74534.log.gz 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88863.log 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50838.log 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107584.log: /var/log/ceph/ceph-client.admin.88863.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107584.log.gz 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88863.log.gz 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97984.log 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69012.log 2024-06-29T16:02:01.409 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50838.log.gz 2024-06-29T16:02:01.410 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.97984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109669.log 2024-06-29T16:02:01.410 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69012.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.97984.log.gz 2024-06-29T16:02:01.410 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69012.log.gz 2024-06-29T16:02:01.410 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61130.log 2024-06-29T16:02:01.410 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81179.log 2024-06-29T16:02:01.410 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61130.log: /var/log/ceph/ceph-client.admin.109669.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61130.log.gz 87.0% -- replaced with /var/log/ceph/ceph-client.admin.109669.log.gz 2024-06-29T16:02:01.410 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:01.411 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113469.log 2024-06-29T16:02:01.411 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111961.log 2024-06-29T16:02:01.411 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81179.log: /var/log/ceph/ceph-client.admin.113469.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81179.log.gz 2024-06-29T16:02:01.411 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113469.log.gz 2024-06-29T16:02:01.411 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48950.log 2024-06-29T16:02:01.411 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78197.log 2024-06-29T16:02:01.411 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.111961.log: /var/log/ceph/ceph-client.2.48950.log: 0.0% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111961.log.gz 2024-06-29T16:02:01.411 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.2.48950.log.gz 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82612.log 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75296.log 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78197.log: /var/log/ceph/ceph-client.0.82612.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78197.log.gz 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82612.log.gz 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54029.log 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37807.log 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75296.log: /var/log/ceph/ceph-client.admin.54029.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75296.log.gz 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54029.log.gzgzip 2024-06-29T16:02:01.412 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.90793.log 2024-06-29T16:02:01.413 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105220.log 2024-06-29T16:02:01.413 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90793.log: /var/log/ceph/ceph-client.admin.37807.log: 79.2% -- replaced with /var/log/ceph/ceph-client.admin.37807.log.gz 2024-06-29T16:02:01.413 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90793.log.gz 2024-06-29T16:02:01.413 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47853.log 2024-06-29T16:02:01.413 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44289.log 2024-06-29T16:02:01.413 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.105220.log: /var/log/ceph/ceph-client.admin.47853.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105220.log.gz 2024-06-29T16:02:01.413 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47853.log.gz 2024-06-29T16:02:01.413 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110870.log 2024-06-29T16:02:01.414 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110996.log 2024-06-29T16:02:01.414 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44289.log: /var/log/ceph/ceph-client.admin.110870.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44289.log.gz 2024-06-29T16:02:01.414 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110870.log.gzgzip 2024-06-29T16:02:01.414 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68614.log 2024-06-29T16:02:01.414 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107771.log 2024-06-29T16:02:01.414 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68614.log: /var/log/ceph/ceph-client.admin.110996.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.110996.log.gz 2024-06-29T16:02:01.414 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68614.log.gz 2024-06-29T16:02:01.414 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100945.log 2024-06-29T16:02:01.415 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.107771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65481.log 2024-06-29T16:02:01.415 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.107771.log.gz 2024-06-29T16:02:01.415 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.100945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82900.log 2024-06-29T16:02:01.415 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65481.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65481.log.gz 2024-06-29T16:02:01.415 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105090.log 2024-06-29T16:02:01.415 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.100945.log.gz/var/log/ceph/ceph-client.admin.82900.log: 2024-06-29T16:02:01.415 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54647.log 2024-06-29T16:02:01.415 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82900.log.gz 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102565.log 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54647.log: /var/log/ceph/ceph-client.admin.105090.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54647.log.gz 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105090.log.gz 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68401.log 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.102565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43929.log 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.102565.log.gz 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54773.log 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43929.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68401.log.gz 2024-06-29T16:02:01.416 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43929.log.gz 2024-06-29T16:02:01.417 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108450.log 2024-06-29T16:02:01.417 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.54773.log: -- /var/log/ceph/ceph-client.admin.46757.log 2024-06-29T16:02:01.417 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.54773.log.gz 2024-06-29T16:02:01.417 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.108450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88821.log 2024-06-29T16:02:01.417 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46757.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108450.log.gz 2024-06-29T16:02:01.417 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46757.log.gz 2024-06-29T16:02:01.417 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73177.log 2024-06-29T16:02:01.417 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70320.log 2024-06-29T16:02:01.418 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88821.log: /var/log/ceph/ceph-client.admin.73177.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88821.log.gz 2024-06-29T16:02:01.418 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73177.log.gz 2024-06-29T16:02:01.418 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109745.log 2024-06-29T16:02:01.418 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103663.log 2024-06-29T16:02:01.418 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.109745.log: /var/log/ceph/ceph-client.admin.70320.log: 87.8% 86.4% -- replaced with /var/log/ceph/ceph-client.admin.109745.log.gz 2024-06-29T16:02:01.418 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70320.log.gz 2024-06-29T16:02:01.418 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.103663.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103663.log.gz 2024-06-29T16:02:02.191 INFO:teuthology.orchestra.run.smithi153.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-06-29T16:02:02.421 INFO:teuthology.orchestra.run.smithi153.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-06-29T16:02:02.843 INFO:teuthology.orchestra.run.smithi153.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-06-29T16:02:03.058 INFO:teuthology.orchestra.run.smithi059.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-06-29T16:02:04.721 INFO:teuthology.orchestra.run.smithi059.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-06-29T16:02:05.261 INFO:teuthology.orchestra.run.smithi153.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-06-29T16:02:05.790 INFO:teuthology.orchestra.run.smithi153.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-06-29T16:02:05.833 INFO:teuthology.orchestra.run.smithi059.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-06-29T16:02:05.919 INFO:teuthology.orchestra.run.smithi153.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-06-29T16:02:06.229 INFO:teuthology.orchestra.run.smithi153.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-06-29T16:02:06.231 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-06-29T16:02:06.231 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m5.672s 2024-06-29T16:02:06.231 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m25.931s 2024-06-29T16:02:06.232 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.908s 2024-06-29T16:02:06.286 INFO:teuthology.orchestra.run.smithi059.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-06-29T16:02:06.609 INFO:teuthology.orchestra.run.smithi059.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-06-29T16:02:07.320 INFO:teuthology.orchestra.run.smithi059.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-06-29T16:02:07.322 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-06-29T16:02:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m6.784s 2024-06-29T16:02:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m29.410s 2024-06-29T16:02:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m1.619s 2024-06-29T16:02:07.323 INFO:tasks.ceph:Archiving logs... 2024-06-29T16:02:07.323 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/remote/smithi059/log 2024-06-29T16:02:07.324 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-29T16:02:12.127 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/remote/smithi153/log 2024-06-29T16:02:12.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-29T16:02:15.592 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-29T16:02:15.602 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-29T16:02:15.602 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-29T16:02:15.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-29T16:02:15.697 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-29T16:02:15.706 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-29T16:02:15.706 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-29T16:02:15.709 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-29T16:02:15.724 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-06-29T16:02:15.726 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-06-29T16:02:15.728 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-29T16:02:15.728 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-06-29T16:02:15.728 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 41 +1486us[+1501us] +/- 51ms 2024-06-29T16:02:15.728 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -1113us[-1099us] +/- 62ms 2024-06-29T16:02:15.729 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -893us[ -878us] +/- 74ms 2024-06-29T16:02:15.729 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T16:02:15.730 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-29T16:02:15.731 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-06-29T16:02:15.731 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv01.front.sepia.ceph.com 3 7 377 108 +1548us[+1557us] +/- 50ms 2024-06-29T16:02:15.731 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 44 -1075us[-1075us] +/- 62ms 2024-06-29T16:02:15.731 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 40 -826us[ -826us] +/- 74ms 2024-06-29T16:02:15.731 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T16:02:15.732 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-29T16:02:15.741 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-29T16:02:15.741 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-29T16:02:15.765 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-29T16:02:15.881 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-29T16:02:15.955 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-29T16:02:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:02:15.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-29T16:02:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:02:16.018 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-29T16:02:16.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-29T16:02:16.056 INFO:teuthology.task.internal:Duration was 6461.173733 seconds 2024-06-29T16:02:16.057 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-29T16:02:16.090 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-29T16:02:16.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-29T16:02:16.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-29T16:02:16.132 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-29T16:02:16.133 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-29T16:02:16.324 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-29T16:02:16.325 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-06-29T16:02:16.325 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-29T16:02:16.386 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-06-29T16:02:16.387 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-29T16:02:16.416 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-29T16:02:16.417 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-29T16:02:16.430 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-29T16:02:16.482 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-29T16:02:16.483 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-29T16:02:16.497 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-29T16:02:16.696 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-29T16:02:16.706 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-29T16:02:16.707 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-29T16:02:16.732 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-29T16:02:16.758 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-29T16:02:16.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-29T16:02:16.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-29T16:02:16.797 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-06-29T16:02:16.826 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-06-29T16:02:16.839 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-29T16:02:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:02:16.866 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-29T16:02:16.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:02:16.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-29T16:02:16.909 INFO:teuthology.task.internal:Transferring archived files... 2024-06-29T16:02:16.909 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/remote/smithi059 2024-06-29T16:02:16.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-29T16:02:16.948 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778923/remote/smithi153 2024-06-29T16:02:16.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-29T16:02:16.982 INFO:teuthology.task.internal:Removing archive directory... 2024-06-29T16:02:16.983 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-29T16:02:16.986 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-29T16:02:17.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-29T16:02:17.047 INFO:teuthology.task.internal:Not uploading archives. 2024-06-29T16:02:17.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-29T16:02:17.082 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-29T16:02:17.083 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-29T16:02:17.085 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-29T16:02:17.098 INFO:teuthology.orchestra.run.smithi059.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 29 16:02 /home/ubuntu/cephtest 2024-06-29T16:02:17.100 INFO:teuthology.orchestra.run.smithi153.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 29 16:02 /home/ubuntu/cephtest 2024-06-29T16:02:17.101 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-29T16:02:17.113 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-29T16:02:17.184 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} duration: 6461.173732995987 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: scheduled_mchangir@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=729a7f032ddd4e97bf5f1f5d056634a1 status: fail success: false 2024-06-29T16:02:17.184 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-29T16:02:17.265 INFO:teuthology.run:FAIL